They are a part of our lives – we invariably face them every day: emotions. The older people get, however, the more difficult it often becomes for them to recognize negative feelings. In addition to personal communication, interacting through videophones, avatars, and other digital media is becoming increasingly important. Older people and those with autism who have impaired social-cognitive abilities need special support. Potsdam computer scientists are attempting to meet this challenge by developing a training system that enables people to practice sending and understanding emotional-communicative signals.
The new training system relies on developing models that automatically recognize emotions – such as boredom, frustration, and interest – based on facial expressions, gestures, gaze direction, and physiological agitation. These signals are used both to receive direct feedback as well as to adjust task difficulty and context to the respective users during their training on the computer.
Users’ needs are crucial
“The training is intended to help people with autism as well as older people to better recognize and express emotions and improve their ability to deal with communication technologies,” says Prof. Ulrike Lucke. The goal is to significantly improve social participation in both groups.
“We have been using e-learning systems for some time now and have often contended with adaptivity in recent years,” she adds. The researcher and her team examine how such a system can be automatically adapted to individual needs. It may be useful, for example, that not all students work through their exercises in the same order but as it is most effective for them. “We adapt the system to the users’ needs and not vice versa.”
In previous projects, Lucke worked on software for people with disabilities, such as for blind people. The idea of creating emotionally adaptive software resulted from her collaboration with psychologists from Humboldt Universität Berlin (HU) who study the emotions of people with autism. The project “EMOTISK - Emotion Sensitive Systems for the Training of Social Cognition” is funded by the Federal Ministry of Education and Research and coordinated by HU. The Potsdam team includes the network’s only technology scientists. Collaborating with psychologists fascinates and inspires Lucke.
Training for people with autism
Relatively static training software is available from the previous project. Users reported being bored and frustrated, because the tasks were either too challenging or not challenging enough due to the same procedures being repeated again and again. The training was only partly successful because the software was not able to respond to the user’s individual needs.
The computer scientists are looking to change this with their new emotion training program. People with autism can train what is challenging for them: recognizing, expressing, and sending emotional signals. They learn to associate certain situations with their own emotions, deduce others’ feelings from facial expressions, and detect and react to emotions in social interaction. “Our goal is to individualize these exercises in such a way that their difficulty adapts to the reaction of the person in front of the screen,” she says.
Some researchers on the team specifically deal with frustration, an emotion that makes the whole body react albeit very individually. “We record the various signals sent by learners in front of the screen and adjust the software accordingly.” This involves determining whether more difficult or easier tasks are needed to avoid people being overstrained or underchallenged, which requires classifying and selecting task difficulty.
The training program does not use abstractly expressed emotions such as those in photographs or drawings but instead uses videos that show “real” people expressing their “real” feelings. This approach appears much more natural. The applicability to everyday life is, therefore, much easier and the learning effect much greater. The huge pool of emotion videos from the project partner Humboldt Universität has been invaluable to the development of the new system.
Exercising with Circus Empathico
The team developed a special program for children with autism, also in collaboration with psychologists from Humboldt Universität. Accompanied by their parents, the children receive training in emotion-based social behavior with a program that is suitable for them and does not use text. A fox explains the world and the Circus Empathico invites them to stay. Since the summer of 2015, an app has been available for use in clinical studies of children with autism. The initial results have been positive. “This, of course, makes me very happy,” says Lucke.
Lucke and her team want to have refined the existing static training system by the end of the EMOTISK program in 2018. The experience and insight gained during the project are to be transferrable to other software. It is not only users with autistic conditions or elderly people that sit frustrated in front of the computer but also often students. Transferring the results to other applications is, therefore, an important task for the team. Students are also involved in this through concrete programming tasks based on practical examples. “This creates exciting ideas and suggestions,” says Lucke. In addition, the future computer scientists not only develop but also test systems and gain experience in dealing with unfinished software. “This change of perspective is very helpful and important for computer scientists.”
The Researcher
Prof. Dr.-Ing. habil. Ulrike Lucke studied computer science at the University of Rostock. Since 2010, she has been Professor for Complex Multimedia Application Architectures and Chief Information Officer (CIO) at the University of Potsdam. Together with sociologist Alexander Knoth, she received the 2016 Teaching Award of the State of Brandenburg.
Universität Potsdam
Institut für Informatik und Computational Science
August-Bebel-Str. 89, 14482 Potsdam
Email: ulrike.luckeuuni-potsdampde
The Project
Emotion-Sensitive Systems for the Training of Social Cognition (EMOTISK)
Participants: Humboldt-Universität zu Berlin, Technische Universität Dresden, RWTH Aachen, University of Cologne, University of Potsdam
Duration: 2015–2018
Text: Dr. Barbara Eckardt
Published online by: Daniela Großmann
Contact for the editoral office: onlineredaktionuuni-potsdampde