Our eyes rarely remain still. When we talk to each other, go for a walk, drive, read, or look at a painting, the gaze always wanders from one point to the next, pausing in one spot for just a fraction of a second, only to focus on a new section the next moment. Six external ocular muscles ensure that our eyeballs can move in different directions. These eye movements are controlled by the central nervous system and can therefore reveal a lot about how the brain works. The AEye Junior Research Group keeps its eyes firmly on very practical applications and has high hopes for new tools of analysis.
Learning aids for artificial intelligence
Researchers at AEye must be prepared to think outside the box. The junior research group with its nine members is interdisciplinary: The researchers come from the fields of computer science, cognitive science, linguistics, and mathematics. Principal investigator Prof. Lena Jäger studied experimental and clinical linguistics after an undergraduate degree in sinology, earned a doctorate in cognitive science, and studied computer science in parallel. “Cognitive science and computer science are very closely connected,” she explains. “Many ideas influence each other. For example, the von Neumann architecture inspired cognitive models that try to explain human information processing. Conversely, the first artificial neural network was developed by a cognitive psychologist and mimics the function of a biological neuron.” The group leader is convinced that the two areas of research complement each other very well and that one helps understand the other better. Her group is primarily looking for methods to incorporate linguistic and cognitive expertise into the learning mechanisms of artificial neural networks.
The interaction between the different disciplines must work well for this to happen. Computer scientists and mathematicians think and work their way into the tasks of the linguists and vice versa. The PhD project of linguist Chiara Tschirner, for example, aims to use eye movement measurements – so-called eye tracking data – to develop a diagnostic screening system for language development disorders such as dyslexia.
Already diagnosing dyslexia at preschool age
Until now, dyslexia, which affects about one in ten children, has been diagnosed with extensive psychometric and linguistic tests. The children must already have a certain meta-knowledge about language to be able to answer the questions and must know, for example, what a rhyme is. “An early diagnosis or assessment of the risk of developing dyslexia is very important because the therapeutic success is much better then,” Jäger explains. If the risk of dyslexia is identified before children start school, it also spares them a lot of frustration. Teachers and parents can adjust to the situation from the outset and support the children accordingly. The AEye team is therefore working on the foundations for a test that will predict the risk of dyslexia in preschool children using eye movements and artificial intelligence.
The research project shows how well computer scientists on the one hand and linguists on the other are connected. “The linguists developed the experimental design in an ongoing dialogue with the computer scientists,” explains co-principal investigator Paul Prasse. “Then we use the data for our mathematical models. The models’ output is in turn analyzed and interpreted by the linguists.”
Tschirner and her colleague Maja Stegenwallner-Schütz are currently preparing the tests they want to carry out in kindergartens and schools with children from the age of five. In total, they will test 500 children. The children will see different objects on a monitor and are asked, for example, to look at the “bag” while their eye movements are recorded. “There is a connection between sensitivity to rhymes and dyslexia,” says Jäger, explaining the background of the test. Children with dyslexia find it more difficult to recognize rhymes or the same word beginnings. Children without dyslexia tend to be distracted more easily by similar-sounding words. If, for example, an “cat” is shown next to the “bag,” children without dyslexia switch back and forth between the two words, while the eyes of children with dyslexia tend to stay on the “mat”.
In addition to the eye tracker, Tschirner is also going to conduct classic diagnostic tests to assess the children’s awareness of linguistic sounds or their cognitive development. In parallel, mathematician David Reich will use the collected data to train a model. By means of machine learning, this model is designed to be able to analyze the children’s eye movements autonomously and predict the risk of dyslexia as precisely as possible.
Eye movements tell even much more
Other projects of the research group also focus on the interaction of eye movement measurements and artificial intelligence. Shuwen Deng uses eye movements to determine whether a child has attention deficit hyperactivity disorder (ADHD). The analysis of eye movements also reveals whether a person is attentive while driving and able to react quickly to dangerous situations or is already too tired to react. Assistance systems in vehicles already use eyelid closure speed, driving time, and sensors on the steering wheel to measure whether drivers should take a break because they are getting tired. The researchers are convinced that eye tracking data could predict this even more accurately.
Therefore, there is a driving simulator in the research group’s laboratory, which will be used to test numerous subjects in the future. The scenarios for these tests will be rather boring: for example, driving on a country road, with little traffic and in half-light. “We hope that our test subjects will tire quickly and that we will then be able to measure this,” explains Prasse. In other scenarios, the drivers will be distracted by the noise of children or a fellow passenger. The eye movements will also reveal whether this has an effect on, for example, reaction time.
David Reich uses this data to develop and implement machine learning models. This requires intuition and a lot of patience. “In about 95% of the cases, the model will not do what we hope for,” says Paul Prasse. In order to achieve the goal, it is necessary to develop a completely new type of model architecture. The researchers sit at the computer and tinker with their models for a long time, looking for problems and the right solutions. Especially where language or behavior meet eye movements, this requires not only computer know-how but also knowledge about cognitive processes. If sometimes, even after hours, something doesn't want to progress and the model gets stuck, talking to colleagues helps one find solutions, perhaps from a different perspective. “Sometimes it’s already enough that I as a mathematician explain to a linguist what I’m doing,” says Reich. “It often becomes clear very quickly what has been forgotten or ignored.”
Black box machine learning
When models are trained using machine learning methods, the idea is to enable them to recognize whether someone is tired, has dyslexia, or what their competence level of a foreign language is. In the process, the prediction errors are calculated retroactively, and the model parameters are improved in small steps. If it still doesn't fit, the computer scientists are called upon again. “Then the model architectures have to be adjusted again,” explains Prasse. “We build deeper networks, consider transformations that make sense for our problem or optimize the type of data input.” There are numerous relevant parameters to consider in order to make the model work as intended by the researchers. Daniel Krakowczyk is researching the interpretability of neural networks. He is interested, for example, in how exactly the models arrive at their results. After all, the decision processes of a neural network cannot be explained and interpreted straightaway. A look into this “black box” could reveal more about how an artificial intelligence gets to the solution of a problem, and from which properties in the data it derives a particular conclusion or prediction. The latter is a prerequisite for achieving transparency in AI.
The Research Group
AEye (Artificial Intelligence for Eye Tracking Data: Deep Learning Methods for the Automated Analysis of Cognitive Processes) is a junior research group at the University of Potsdam. The researchers develop machine learning methods for the analysis of eye tracking data to make inferences or predictions about the cognitive processes and psychological states of an individual.
Funding: Federal Ministry of Education and Research (BMBF)
Duration: 2020–2024
www.uni-potsdam.de/de/cs-ml/aeye
The Researchers
Prof. Dr. Lena Jäger studied sinology at the University of Freiburg, Tongji University Shanghai, and Université Paris 7 Denis-Diderot, and subsequently experimental and clinical linguistics at the University of Potsdam, where she received her PhD in cognitive science in 2015 while studying computer science at the same time. Since 2018, she has been conducting research in machine learning at the University of Potsdam. Since 2020, she has been principal investigator in the AEye Junior Research Group, for which she raised the funds. She has also been Professor at the Institute of Computational Linguistics of the University of Zurich since 2020.
Mail: lena.jaegeruuni-potsdampde
Shuwen Deng studied information and communication technology at Beijing University of Technology and Friedrich-Alexander-Universität Erlangen-Nürnberg. She is doing her PhD in the AEye Junior Research Group.
Mail: shuwen.denguuni-potsdampde
Dr. Maja Stegenwallner-Schütz studied patholinguistics and experimental and clinical linguistics at the University of Potsdam, where she received her PhD on language development in children in 2019. She is a postdoc in the AEye Junior Research Group as well as a researcher at the Department of Inclusion Education of the University of Potsdam.
Mail: stegenwauuni-potsdampde
David R. Reich studied mathematics at Freie Universität Berlin and Technische Universität Berlin. He is doing his PhD in the AEye Junior Research Group.
Mail: david.reichuuni-potsdampde
Daniel Krakowczyk studied computer science at Freie Universität Berlin and at the University of Potsdam. He is doing his PhD in the AEye Junior Research Group.
Mail: daniel.krakowczykuuni-potsdampde
Chiara Tschirner studied linguistics at Leipzig University and subsequently experimental and clinical linguistics at the University of Potsdam. She is doing her PhD in the AEye Junior Research Group.
Mail: chiara.tschirneruuni-potsdampde
Dr. Paul Prasse studied computer science at the University of Potsdam, earned a PhD there in 2016, and is now the co-principal investigator of the AEye Junior Research Group.
Mail: prasseuuni-potsdampde
This text was published in the university magazine Portal Wissen - Eins 2022 „Zusammen“.