Summary: A new stretchable, rechargeable sticker developed by researchers can detect authentic emotional states by measuring physiological signals like heart rate, skin temperature, and humidity, even when facial expressions are misleading. The wearable patch transmits real-time data to mobile devices, helping health providers assess mental health remotely.
Unlike traditional emotion recognition systems, this device integrates multiple sensors and facial analysis while preserving user privacy. With AI-powered accuracy and wireless functionality, it offers promise for applications in telehealth, early intervention, and monitoring emotional well-being.
Key Facts:
- Multi-Signal Detection: Measures skin temperature, humidity, heart rate, and oxygen independently without interference.
- AI Emotion Recognition: Achieved 96.28% accuracy for acted emotions and 88.83% for real ones.
- Remote Monitoring: Wirelessly transmits data for use in telemedicine and early mental health intervention.
Source: Penn State
Saying one thing while feeling another is part of being human, but bottling up emotions can have serious psychological consequences like anxiety or panic attacks.
To help health care providers tell the difference, a team led by scientists at Penn State has created a stretchable, rechargeable sticker that can detect real emotions — by measuring things like skin temperature and heart rate — even when users put on a brave face.
The researchers recently unveiled the wearable patch that can simultaneously and accurately track multiple emotional signals in a study published in the journal Nano Letters.
“This is a new and improved way to understand our emotions by looking at multiple body signals at once,” said Huanyu “Larry” Cheng, the James L. Henderson, Jr. Memorial Associate Professor of Engineering Science and Mechanics at Penn State and lead author on the paper.
“Relying only on facial expressions to understand emotions can be misleading. People often don’t visibly show how they truly feel, so that’s why we’re combining facial expression analysis with other important physiological signals, which will ultimately lead to better mental health monitoring and support.”
The sticker-like patch tracks a range of physiological responses, such as skin temperature, humidity, heart rate and blood oxygen levels, that are associated with emotional states.
Most importantly, Cheng explained, the device’s sensors are designed to work independently, minimizing any interference between the different measurements.
The device combines its analysis of the physiological signals with facial expression data to better distinguish between genuine emotions and acted ones. It then wirelessly transmits the real-time measured data to mobile devices and the cloud, where clinicians could potentially use it to better assess patients virtually.
The device does not record personal information, only signals, Cheng explained, meaning personal privacy is protected through the device’s design.
“This technology has the potential to help people who are struggling with their mental health, but maybe aren’t being fully honest with others or even themselves about how much they are struggling,” said Yangbo Yuan, co-author on the paper and doctoral student at Penn State studying engineering science and mechanics.
Cheng explained that the collected data could also help bridge cultural or social gaps, which can manifest as a person appearing more stoic or expressive to their health care providers. “By keeping track of these signals, it could be possible to detect problems like anxiety or depression earlier in its progression.”
The researchers built the stretchy, BandAid-sized device by folding together thin layers of flexible metals like platinum and gold and cutting them into wave-like shapes that maintain sensitivity even when pulled or twisted.
They also used layers of materials that change the flow of electrical current with temperature and built in hollow tubes made of carbon atoms, which absorb water and track humidity levels.
The multiple sensors were designed and placed on the device in such a way that they would not interfere with each other’s measurements. For example, the researchers put a rigid layer under the temperature and humidity sensors to protect them from the stretching that the facial expression sensors would experience.
They also used a waterproof layer to protect the temperature and strain sensors from humidity.
“We’ve engineered this device to measure these different signals independently, without them interfering with each other, providing a much clearer and more accurate picture of what’s happening beneath the surface,” said Libo Gao, co-corresponding author on the paper and associate professor at Xiamen University.
Next the team trained an artificial intelligence (AI) model to read and understand signs of performed and real human emotion. The researchers recruited eight people, a common sample size for pilot studies, to perform six common facial expressions: happiness, surprise, fear, sadness, anger and disgust.
The participants displayed each emotional expression 100 times while the device tracked their movement. The researchers then fed this data to an AI model, training it to correlate specific facial movements with different emotions.
They then recruited an additional three participants to further evaluate the model’s abilities. It classified performed facial expressions with 96.28% accuracy.
When it came to tracking real emotions, the researchers tested how well the device tracked the psychological responses of the same participants as they watched video clips designed to elicit emotions.
The device correctly identified emotions with 88.83% accuracy, with the sensors confirming that the psychological responses were consistent with known links between emotions and psychological reactions, such as increases in skin temperature and heart rate during surprise and anger.
Cheng noted that the ability to wirelessly transmit the data means that health care professionals could potentially monitor individuals remotely and provide timely emotional support through telemedicine.
“This sensor can serve a vital function in bridging gaps in access to care,” he said. “Given the rising stress levels in modern society, the ability to monitor emotions can provide early indicators of debilitating conditions and allow for proactive support.”
He explained that the device also opens the door for other systems for artificial intelligence (AI)-powered disease diagnostics and therapeutics beyond just emotion recognition.
He noted there may be potential applications for clinicians to better understand the mental and emotional state of non-verbal patients, better identify behavioral and psychological symptoms of dementia and recognize opioid overdose.
The technology could one day even be used for chronic wound monitoring and disease management, he added, as well as track neurodegenerative disease progression and athletic performance.
“While still in the research and development phase, this device is a significant step forward in our ability to monitor and understand human emotions, potentially paving the way for more proactive and personalized approaches to mental health care,” Cheng said.
Other contributors include Hongcheng Xu of Xi’an Jiaotong University.
Funding: The U.S. National Institutes of Health and the U.S. National Science Foundation funded the Penn State researchers’ contributions to this work.
About this emotion and neurotech research news
Author: Adrienne Berard
Source: Penn State
Contact: Adrienne Berard – Penn State
Image: The image is credited to Neuroscience News
Original Research: Closed access.
“Stretchable, Rechargeable, Multimodal Hybrid Electronics for Decoupled Sensing toward Emotion Detection” by Huanyu “Larry” Cheng et al. Nano Letters
Abstract
Stretchable, Rechargeable, Multimodal Hybrid Electronics for Decoupled Sensing toward Emotion Detection
Despite the rapid development of stretchable electronic devices for various applications in biomedicine and healthcare, the coupling between multiple input signals remains an obstacle in multimodal sensing before use in practical environments.
This work introduces a fully integrated stretchable, rechargeable, multimodal hybrid device that combines decoupled sensors with a flexible wireless powering and transmitting module for emotion recognition.
Through optimized structural design and material selection, the sensors can provide continuous real-time decoupled monitoring of biaxial strain, temperature, humidity, heart rate, and SpO2 levels.
With a stacked bilayer for both the sensors and the flexible circuit, the rechargeable system showcases a reduced device footprint and improved comfort. A neural network model is also demonstrated to allow for high-precision facial expression recognition.
By transmitting the real-time measured data to mobile devices and the cloud, the system can allow healthcare professionals to evaluate psychological health and provide emotional support through telemedicine when needed.
