Physiological Sensor-Based Real-Time Human Mental States Modeling – Toward Adaptive Human Machine Interaction

Existing human-machine interaction (HMI) development mainly undertakes either a human-centered or robot-centered manner reactively. As the collaborative-robots era emerges, current studies aim to design a machine that can be more responsive to humans’ states and exhibit behaviors that approximate human teammates. To enable such seamless HMI, it is critical to have a reliable, scalable, and easy-to-use technology that can assess the human cognitive states, so that the adaptive automation/interventions can be designed accordingly to augment HMI. In this talk, she will discuss research about real-time human cognitive states modeling using wearable sensors and machine learning models. In the first part of the presentation, she will present an advanced real-time sensing paradigm that monitors the cognitive load of the surgeon and provides adaptive assistance as needed during roboticassisted surgery. The second part of this talk will introduce a multimodal sensing approach to assessing human situation awareness, as well as an illustration of how such techniques can be applied in complex HMI. The results of this study will enable real-time performance enhancement mechanisms and lay the groundwork for future adaptive HMI.

Dr. Jing Yang is an Assistant Professor at Department of Industrial & Systems Engineering University at Buffalo. She received her PhD from School of Industrial Engineering at Purdue University and master from the University of Michigan. She focuses on developing real-time systems for human cognitive functions and behavior modeling, as well as designing interventions to enhance human performance in a variety of environments. She was a recipient of the 2021 Human Factors Award for Excellence in Human Factors Research for innovation in designing cognitive workload-triggered adaptive automation for future robotic-assisted surgery.