Developing a Multimodal Entertainment Tool with Intuitive Navigation, Hands-Free Control, and Avatar Features to Increase User Interactivity; Facial Recognition Software and the Politics of Design in Sociotechnical Systems

The growth of different technologies in recent years has raised questions about the extent of bias in data and the systems that surround it. Due to the long history of racism and oppression in America, there is a large amount of historical bias that exists in data collection methods and existing data files. Additionally, there are few common technologies that have been adapted for a wide range of users, including those who have disabilities or other impairments. The following technical and STS theses discuss two different technologies: facial recognition software and hands-free control and how bias can be eliminated from these technologies as well as how they can be expanded and improved to serve a larger user base.
The technical thesis discusses the creation of a multimodal entertainment tool that has three novel features: an intuitive tree-like map, hands-free voice controls, and avatars. All of these features seek to provide a hands on, active entertainment experience that caters to a wide user group. The research group focused on creating a cooking experience use case that would alleviate the troubles of cooking with messy hands while also providing different ways for the user to communicate with the system if they do not have use of their hands. The user is able to intuitively follow a recipe using the tree-like map that allows them easy navigation and the flexibility to do the recipe at their own pace. Next, the prototype includes the use of voice controls. The user can go through the entire recipe hands-free by only using the voice commands provided and taught by the system. Finally, there are two different avatars that add an entertainment aspect to the recipe, with one avatar telling jokes and the other asking trivia question.
The STS thesis pivots to discussing the current bias behind facial recognition technology. There have been many recent incidents of unethical usage of facial recognition technology, including unauthorized image capturing by law enforcement of protestors at Black Lives Matter events. This bias negatively affects all minorities, especially Black women. Through the thesis, case studies of different companies and their approach to dealing with the detrimental consequences of bias facial recognition are compared to create an approach to raise awareness and attempt eliminating racial bias in technology.
Through all of this research, my group and I were able to create an innovative new prototype that was unique from other cooking prototypes on the market. Additionally, I dived into research concerning facial recognition bias that is extremely prevalent in many large technology companies. Thank you to Professor Sean Ferguson, who helped guide me smoothly through my undergraduate thesis work. Additionally, thank you to all my friends, family, and mentors who have made my time at the university worthwhile.

School of Engineering and Applied Science
Bachelor of Science in Systems Engineering
Technical Advisor: Gregory Gerling
STS Advisor: Sean Ferguson
Technical Team Members: Nathaniel Barrington, Caton Gayle, Erin Hensien, Megan Lin, Saimanga Palnati