A Data Capture and Gesture Recognition System to Enable Human-Robot Collaboration; Robots Will Care For Us: The Need for Participatory Design in the Care Crisis

Traditionally, robots have been used in the place of human labor to automate routine or unsafe tasks in domains such as manufacturing, healthcare, and defense. As robotic technologies continue to be integrated in society, there is a shift towards robots collaboratively working alongside humans. Human-robot collaboration (HRC) proposes that the joint human-robot partnership will enable more efficient, productive, and safer operational outcomes. This will be achieved by complementing robotic precision with human decision making in various shared working environments and conditions. To achieve an efficacious collaboration, communication is vital to convey human intent and direction to the robot counterpart. Gesture-based communication is intuitive and adaptable to various environmental conditions.

Through my capstone project, we developed a gesture recognition system to enable human-robot collaboration through gesture-based communication. The system relies upon a vision-language model (VLM) to facilitate the gesture recognition and is informed by retrieval augmented generation (RAG) and chain-of-thought (CoT) prompting. VLMs offer a less computational and data intensive alternative compared to traditional vision-based machine and deep learning approaches, allowing for gesture understanding without prior training. RAG provides an additional knowledge base of domain-specific gestures to enhance the VLM’s contextual understanding, CoT prompting guides the VML through structured reasoning. The inclusion of RAG and CoT prompting improve the reliability of the VLM with limited computational cost. Our gesture recognition system was tested in a pilot study, which resulted in an accuracy score of 80% and an F1 score of 89.9%.

When examining the application of human-robot interaction (HRI), a field that encompasses HRC, there is a growing demand for care robots to be used for elderly care in the face of the current care crisis. Globally, the increasingly aging population requires both formal and informal, familial care to meet the complex needs of an aging demographic. Roboticists, technologists, corporations, and governments have been investing in care robots to reduce the scarcity of elderly care. However, the exclusion of key stakeholders, namely care partners and care receivers, in the design and integration of care robots has contributed to technological paternalism while still leaving the needs of care partners and care receivers unmet.

Through my STS research, I examined an ethnographic study conducted in Japan to analyze the state of elderly care, the sociopolitical and technological factors involved in the implementation of care robots, and the implications for key stakeholders. In my research, I found that human care partners are not unreceptive to care robots, but the unintentional decentering of human care partners and elderly care receivers in the design and implementation process by technologists and robotics resulted in a lack of usage and an increased workload for human care partners. The increased workload and discomfort faced by care receivers during the use of care robots has resulted in a lack of use. However, by partaking in a participatory design process, stakeholders will have increased agency in conveying their needs and ensuring the design is focused on those needs. The centering and involvement of stakeholders in the design process will help to shift away from technological paternalism and towards a design reflective of stakeholder needs.

School of Engineering and Applied Science
Bachelor of Science in Systems Engineering
Technical Advisor: Tariq Iqbal
STS Advisor: Richard Jacques
Capstone Team Members: Evan Smith, Camp Hagood, Aramis Rolly