Reading Facial Expressions for Autism Symptoms

There’s an app for everything these days—from weight loss to working out. Now, thanks in part to support from the Office of Naval Research (ONR), there’s an app that may screen for autism by reading kids’ facial expressions for emotional cues.

“The long-term implications of this research are huge,” said Dr. Predrag Neskovic, a program officer in ONR’s Mathematical Data Science program. “Not only could the app be used to learn more about childhood autism, it could possibly reveal signs of post-traumatic stress disorder [PTSD] and mild traumatic brain injury [TBI] in warfighters—conditions that often have subtle symptoms and are difficult to diagnose.”

The app, called “Autism & Beyond,” was developed by researchers and software developers at Duke University and the Duke Medical Center. It currently is available for free download from the Apple App Store to families wishing to participate in a six-month medical research study by Duke University.

“Autism & Beyond” has children complete a series of questionnaires and watch short videos designed to make them smile, laugh and be surprised. Parents or caregivers use an iPhone’s user-facing “selfie” camera to record children’s facial movements for evaluation by doctors, researchers and software.

The app’s core technical component—the ONR-sponsored technology—is a complex mathematical algorithm that automatically maps key landmarks on children’s faces and assesses emotional responses based on movements of facial muscles.

“We analyze the video to track position and movement of the head and face, including the lips, eyes and nose—all of which indicate emotions,” said Dr. Guillermo Sapiro, a professor at Duke University, who developed the algorithm. “For example, while watching stimuli like a funny video, does the child smile, look toward the caregiver or ask the caregiver to view the video as well? We study all of that. Lack of emotion and social sharing are possible characteristics of childhood autism.”

Sapiro stresses the app isn’t a self-diagnosis resource, but is intended to serve as a potential screening tool for autism and other developmental challenges, and encourage users to contact a physician for specialized testing.

Once the autism studies are complete, Sapiro hopes to eventually expand and tailor the app to conditions suffered by warfighters, including PTSD, TBI and depression. As with autism, facial expressions can indicate the presence of such disorders. For example, many individuals suffering from PTSD or depression smile often, but their grins tend to be forced and short. Improved analysis of such facial expressions could lead to more accurate diagnoses by doctors and, consequently, enhanced treatment options for military personnel and veterans.

Photo of a child's face and the app output.
The app’s core technical component—the ONR-sponsored technology—is a complex mathematical algorithm that automatically maps key landmarks on children’s faces and assesses emotional responses based on movements of facial muscles. Credit: Office of Naval Research.

“The great thing about the algorithm and video analysis tool is they can be universally applied to various research apps and studies,” said Sapiro. “Today it’s autism, in the future it could be PTSD. How can we take ‘Autism & Beyond’ beyond its current state and to the next level? Autism is just the start.

“We hope to find the right partner and develop a research app studying PTSD,” he continued. “We’ve already received interest from a nearby Veterans Affairs center about using this technology in veterans’ homes to monitor behavior for signs of depression.”

About this autism research

Source: Warren Duffie – Office of Naval Research
Image Source: The image is credited to Office of Naval Research

Feel free to share this neuroscience article.
Join our Newsletter
I agree to have my personal information transferred to AWeber for Neuroscience Newsletter ( more information )
Sign up to receive our recent neuroscience headlines and summaries sent to your email once a day, totally free.
We hate spam and only use your email to contact you about newsletters. You can cancel your subscription any time.
  1. I have been told told that I have mild autism. At this point, I am a 71-yo female. I have known all my life that I didn’t fit in, that I am exceptionally intelligent, and that I am a lot more rational than emotional. Consequently, I am great to have around in an emergency or when things get gory ( the yuk part of my brain seems not to be present).

    The reason I am responding is that it would seem to me that it might prove beneficial to apply this app to people applying for specops. I don’t know why, but all my life I have understood the military mind; spent the summer of ’66 at WRAMC with soldiers from Nam debriefing on me in the gedunk without losing my lunch. Spent a lot of time over the years with GIs and vets listening to stories, knowing what, and what not, to say. There is something in the wiring in my brain that says had I been a boy and aimed toward the military, I would have understood.

Comments are closed.