A Framework for Developing Adaptive Social Robots: Integrating Theory of Mind and Model-Based Reinforcement Learning

A social robot is a physically embodied robot that can act (semi) autonomously to interact with humans while following social norms. As social robots become more commercially accessible, their ability to adapt to various social situations has become crucial for effective human-robot interactions. Recently, there has been great interest in using machine learning methods for developing adaptive social robots that assist in social rather than physical means. Interaction is a key component in both Reinforcement Learning (RL) and social robots. Therefore, RL may enable social robots to adapt their behaviors according to their human partners for natural human-robot interaction. However, current RL methods require many attempts to learn how to solve a task, which makes it challenging to apply RL in real-world scenarios with social robots.

In this dissertation, I present a new framework with multimodal inputs and outputs for modeling users for an RL application using the Theory of Mind to simulate a human's emotional transitions during the interaction. We use model-based learning to model the user's emotional transitions to improve the learning process's speed and adapt the multi-modal interaction channels and feedback types according to the social situations and user preferences integrated as prior knowledge. Two user studies were conducted as a part of this dissertation contributing to the framework, 1) The first study focuses on a self-learning system for emotion awareness and adaptation in humanoid robots. By combining a facial expression recognition system and personalized behavior adaptation, the robot improves its performance through interactions, eliciting ground-truth data from users and updating individualized models over time. 2) The second study delves into users' perception of a humanoid robot's emotional behaviors under social stress. The goal is to understand users' perception of the robot's verbal and non-verbal actions that evoke emotions, particularly ambiguous actions, and observe how users perceive them under social stress. This study focuses on the impact of a user's prior emotional state on their perception of the robot.

The key contributions of this work encompass three aspects: 1) a framework employing human and robot emotion states for emotion adaptation and personalized behaviors, 2) an implementation of the Theory of Mind approach to simulate users' emotional transitions, enabling the robot to reason its actions based on internal states and prior user knowledge, and 3) a Model-based RL model featuring principled transition probability equations based on the Geneva Emotion Wheel. This approach addresses the cold-start problem, facilitating interactions with new users by enhancing sampling efficiency and reducing the interactions needed to generate an optimal policy. Additionally, multi-modal feedback is utilized to improve the interaction experience for diverse user groups.

By integrating cognitive architecture, adaptive behaviors, Geneva Emotion Wheel based RL model, and empathy evoking actions, the proposed framework strives to enhance social robots' overall engagement and experience in Human-Robot Interaction (HRI) scenarios.