A Data Capture and Gesture Recognition System to Enable Human-Robot Collaboration; Understanding Choice Architecture in the United States Healthcare System

Human-robot collaboration (HRC) can lead to safer outcomes and more efficient operations when completing dangerous tasks in a dynamic, noisy environment. However, current communication methods, including verbal or wearable sensor-based methods, may be unreliable under these conditions. The facilitation of gesture-based communication can bridge this gap and become an intuitive and non-intrusive facilitator of HRC. Currently, the United States healthcare system is falling short for countless patients, leaving critical needs unmet. To drive meaningful change, we must first identify and address the underlying causes of this problem from the perspective of key stakeholders—patients, providers, the government, pharmaceutical companies, and insurance companies—whose power imbalance and conflicting interests have hindered progress.
The technical portion of my capstone project produced a foundational gesture recognition framework, which leverages a vision-language-model (VLM) guided by retrieval-augmented generation (RAG) and chain-of-thought (CoT) prompting. The knowledge base provided by RAG provides context to the VLM while CoT introduces reasoning strategies for the model. An experiment validating our framework yielded an 80% accuracy and an 89.9% F1 score, highlighting the potential of VLMs for zero-shot gesture classification and robotic control, and providing a foundation for robust, field-deployable gesture-based HRC systems.
In my STS research, I analyzed the U.S. healthcare system through the lens of a choice architecture and incentives framework to understand how decisions are viewed from the perspectives of key stakeholders. Through this analysis, I found that the federal government does not properly balance the interests of pharmaceutical and insurance companies with the interests of patients. This is caused by the large influence pharmaceutical companies have gained through lobbying and their monopoly on the production of medicine, which is further enabled by insurance companies ‘covering’ the excessive prices set by the pharmaceutical companies. This ultimately affects patients the most as insurance remains the only viable option for patients to pay into, even though patients are still unable to afford and receive adequate care. This problem must be solved by incorporating multiple solutions, which include restrictions on lobbying expenditures and incentives for lowering medicinal and treatment costs.

School of Engineering and Applied Science

Bachelor of Science in Systems Engineering

Technical Advisor: Tariq Iqbal

STS Advisor: Richard Jacques

Technical Team Members: Camp Hagood, Sarah Naidu, Aramis Rolly