Summary: A new report warns most software designed to read facial emotions are based on outdated scientific data. Researchers say the majority of emotional detection software applications do not capture real differences in the way people convey, or interpret, emotion on faces.
Source: APS
Software that purportedly reads emotions in faces is being deployed or tested for a variety of purposes, including surveillance, hiring, clinical diagnosis, and market research. But a new scientific report finds that facial movements are an inexact gauge of a person’s feelings, behaviors or intentions.
“It is not possible to confidently infer happiness from a smile, anger from a scowl or sadness from a frown, as much of current technology tries to do when applying what are mistakenly believed to be the scientific facts,” a group of leading experts in psychological science, neuroscience and computer science write in their comprehensive research review.
The report appears in Psychological Science in the Public Interest, a journal of the Association for Psychological Science, and is authored by Lisa Feldman Barrett of Northeastern University, Ralph Adolphs of the California Institute of Technology, Stacy Marsella of Northeastern University and the University of Glasgow, Aleix M. Martinez of The Ohio State University and Seth D. Pollak of the University of Wisconsin-Madison.
The authors note that the general public and some scientists believe that there are unique facial expressions that reliably indicate six emotion categories: anger, sadness, happiness, disgust, fear, and surprise. But in reviewing more than 1,000 published findings about facial movements and emotions, they found that typical study designs don’t capture the real-life differences in the way people convey and interpret emotions on faces. A scowl or a smile can express more than one emotion depending on the situation, the individual or the culture, they say.
“People scowl when angry, on average, approximately 25 percent of the time, but they move their faces in other meaningful ways when angry,” Barrett explains. “They might cry, or smile, or widen their eyes and gasp. And they also scowl when not angry, such as when they are concentrating or when they have a stomach ache. Similarly, most smiles don’t imply that a person is happy, and most of the time people who are happy do something other than smile.”
In a separate article in the journal, Alan Cowen and Dacher Keltner of the University of California, Berkeley; Disa Sauter, University of Amsterdam; and Jessica L. Tracy of the University of British Columbia note that most scientists agree that facial expressions are meaningful, even if they don’t follow a one-to-one match with six basic emotion categories. They propose a new model for studying emotion-related responses in all their complexity and variations. This approach would measure not only facial cues, but also body movements, voice fluctuations, head movements and other indicators to capture such nuanced responses as smiles of embarrassment or sympathetic vocalizations, they say.
The report’s conclusions have broad implications, according to the authorship team. The FBI and the Transportation Security Administration have trained agents in the past to assess smiling, scowling and other facial movements to identify and stop potential terrorists. Law enforcement agencies in the United States and Europe are now experimenting with technologies designed to automate emotion detection through facial scans. Some companies are experimenting with software to track the facial movements of job applicants during interviews. Such technology might be able to detect facial movements, but they do not detect the psychological meaning of those facial movements, Barrett and co-authors say.
“We thought this was an especially important issue to address because of the way so-called ‘facial expressions’ are being used in industry, educational and medical settings, and in national security,” said Barrett and her co-authors.
Source:
APS
Media Contacts:
Anna Mikulak – APS
Image Source:
The image is in the public domain.
Original Research: Open access
“Emotional Expressions Reconsidered: Challenges to Inferring Emotion From Human Facial Movements”. Lisa Feldman Barrett, Ralph Adolphs, Stacy Marsella, Aleix M. Martinez, Seth D. Pollak.
Psychological Science in the Public Interest. doi:10.1177/1529100619832930
Abstract
Emotional Expressions Reconsidered: Challenges to Inferring Emotion From Human Facial Movements
It is commonly assumed that a person’s emotional state can be readily inferred from his or her facial movements, typically called emotional expressions or facial expressions. This assumption influences legal judgments, policy decisions, national security protocols, and educational practices; guides the diagnosis and treatment of psychiatric illness, as well as the development of commercial applications; and pervades everyday social interactions as well as research in other scientific fields such as artificial intelligence, neuroscience, and computer vision. In this article, we survey examples of this widespread assumption, which we refer to as the common view, and we then examine the scientific evidence that tests this view, focusing on the six most popular emotion categories used by consumers of emotion research: anger, disgust, fear, happiness, sadness, and surprise. The available scientific evidence suggests that people do sometimes smile when happy, frown when sad, scowl when angry, and so on, as proposed by the common view, more than what would be expected by chance. Yet how people communicate anger, disgust, fear, happiness, sadness, and surprise varies substantially across cultures, situations, and even across people within a single situation. Furthermore, similar configurations of facial movements variably express instances of more than one emotion category. In fact, a given configuration of facial movements, such as a scowl, often communicates something other than an emotional state. Scientists agree that facial movements convey a range of information and are important for social communication, emotional or otherwise. But our review suggests an urgent need for research that examines how people actually move their faces to express emotions and other social information in the variety of contexts that make up everyday life, as well as careful study of the mechanisms by which people perceive instances of emotion in one another. We make specific research recommendations that will yield a more valid picture of how people move their faces to express emotions and how they infer emotional meaning from facial movements in situations of everyday life. This research is crucial to provide consumers of emotion research with the translational information they require.
