A Technology That Reveals Your Feelings

Software sizes up your eyes, nose, mouth—even your wrinkles

141476482
Photographer: Getty Images

Memo to students: Think you can fool your teacher when you’re not paying attention? Think again. In the not-too-distant future, a laptop flashing a graph tracking classroom attention in real time could give you away. By the end of 2015, as many as 1,000 schools in the U.S. and Canada could be using a technology that monitors students, says Rich Cheston, chief solutions officer at Stoneware, a Lenovo unit that makes classroom management software. Stoneware will soon incorporate emotions analytics into one of its products to track attention in the classroom. A cousin of facial recognition, emotions analytics relies on video of facial expressions.

The teachers “can see it as they are teaching, so they can determine when to take corrective action,” Cheston says. His product will come out in September, and then he will start marketing it to schools.

Over the past few years, big companies, including Unilever and Coca-Cola, have used emotions analytics to better understand customers’ likes and dislikes and to tailor marketing and advertising campaigns. About a dozen companies are making and supporting such software, according to researcher Crone Consulting. The market leaders include Emotient, a startup in San Diego, and Affectiva in Waltham, Mass. Unilever relies on Affectiva’s emotions analysis to assess customer reactions to its ads. Emotient’s software will be used in Stoneware’s classroom product. And Emotient tested its software with the NBA’s Golden State Warriors to study how spectators respond to activities such as a dance cam.

The market for emotion-analyzing software could reach $10 billion worldwide in five years, up from more than $100 million in 2016 and less than $20 million this year, Crone Consulting says.

The software, which can be used in PCs, cars (to alert drivers when they get distracted), and smartphones, works with live or recorded video of facial expressions. How the different companies’ products work varies. Affectiva’s software homes in on points on a face, such as the corners of eyes or eyebrows. Algorithms also detect texture variations that occur when people laugh, frown, or smirk. Accessing its database of 3.2 million face videos, Affectiva says it can identify facial expressions and emotions with high accuracy. Its software processes video footage frame by frame to report the various emotional states of a user, from happiness to sadness, or from surprised to unfazed. It tests its emotion-recognition algorithms against images in its database.

“This is a passive way of not asking someone a question but simply observing their emotional response,” says Nicholas Langeveld, Affectiva’s chief executive officer. Affectiva says its analysis has a 90 percent-plus accuracy rate for some emotions.

Startup Beyond Verbal, founded three years ago in Tel Aviv, has a database of 1.5 million voices. The company analyzes vocal intonations to identify more than 300 mood variants in more than 40 languages with 80 percent accuracy.

Susan Etlinger, a data intelligence analyst at brand consultant Altimeter Group, says the software needs to be proven. “How can you verify whether someone was actually feeling joy or anger at the particular moment?” she asks. There also are ethical considerations, because those being studied may not know it, she says.

The field of facial-emotions analytics dates to the 1970s and the work of American psychologist Paul Ekman. “I thought it was going to be used in research,” says Ekman, who along with several colleagues developed a way to measure facial movements. Emotient has adapted his system. Ekman says he worries that some of the technology’s uses could infringe on privacy rights. After he threatened to resign from Emotient’s board, where he’s an adviser, Ekman said the company is now addressing his concerns.

Microsoft has developed prototype consumer apps that track emotions via a skin sensor on an activity band and a heart-rate monitor. They alert users when stress levels are high and offer tips for coping with the stress. Some employees in the research division developing the apps share their emotions through desk crystals, which change color when users are sad, happy, stressed, or bored. “Your colleague could help you out, make you feel better,” says Mary Czerwinski, a research manager at Microsoft. “But as soon as one of their bosses came in, they covered it up.”

The bottom line: The market for software that analyzes emotions could reach $10 billion worldwide by 2020.

(Corrects spelling of Nicholas Langeveld's last name in the sixth paragraph.)