A Self Learning System for Emotion Awareness and Adaptation in Human Robots; Analysis of the Relationship of Computer Vision, Social Robotics, and Human Classification

Social Robotics aims to integrate robots into society by replicating and understanding human behavior. This endeavor has seen significant research growth in healthcare and human understanding, utilizing technologies like emotion and facial recognition to enable robots to respond naturally in social situations. These projects are largely dependent on machine learning and computer vision, which have grown exponentially over the past decade. However, these fields carry inherent biases and social issues that could infiltrate social robotics if not properly addressed by researchers, underscoring the need for vigilance in this field.
The technical component of my thesis explored a novel solution to the "cold start" problem in social robotics, a challenge where social robots grapple with initial interactions. By integrating Convolutional and Deep Neural Networks, we devised a study to establish a self-adaptive system for emotion recognition in NAO robots. The study amalgamates ResNet50 and Inception v3 models in a soft voting ensemble, which consolidates the predicted probabilities from both models for real-time application. Transfer learning is employed, leveraging pre-trained model weights of static facial expression images from the datasets. This weight initialization strategy enables the models to commence with insights gleaned from previous data and adapt to new users' facial expressions. The outcomes demonstrated an enhancement in accuracy, trust, and engagement scores, indicating the potential applicability of this technique in future research.
In my STS research, I saw a heavy reliance on machine learning and computer vision within social robotics. However, these technologies come with their own set of challenges. The datasets and models we employed in our technical project may contain inherent social biases, such as reduced accuracy for different skin tones and expressions that may not align with various cultures. Furthermore, machine learning and computer vision have a history of privacy concerns, which could lead to potential disasters, especially in sensitive areas like healthcare. The aim of our project was to eventually incorporate NAO robots into hospitals to aid children undergoing painful procedures. However, without a comprehensive understanding of the risks associated with these technologies, their application in social robotics could be problematic. Despite the challenges, these technologies are favored due to their ease of use and the rapid advancements in computer vision and machine learning. My research intends to highlight these issues and serve as a warning for future research involving these technologies.
This study provided an opportunity for reflection, allowing me to evaluate the work I conducted and assess whether I was happy with the results and my involvement. It is my hope that this research will help guide the domain of social robotics towards a safe and ethical trajectory. The goal is to enhance our decision-making process concerning the technologies we employ in our research. This is particularly crucial when our research directly translates into products designed for public interaction and by doing so, we can ensure a more informed and responsible approach to technological advancement in the field.
I want to acknowledge and that the Human-AI Robotics Lab within UVA’s Engineering Link Lab for giving me the opportunity to work on the technical project. The study was funded by a seed grant awarded by the Center for Engineering in Medicine at the University of Virginia. I would also like to thank my STS 4600 Professor William Davis and Capstone Advisor Rosanne Vrugtman for helping me through this research process.

School of Engineering and Applied Science

Bachelor of Science in Computer Science

Technical Advisor: Bianna Morrison

STS Advisor: William Davis

Technical Team Members: Sudhir Shenoy, Yusheng Jiang, Lauren Manuel, Afsaneh Doryab