Developing a Multimodal Entertainment Tool with Intuitive Navigation, Hands-Free Control, and Avatar Features, to Increase User Interactivity; Rockwell’s Nightmare: The Digital War Between Smart Technology and Personal Privacy

Smart technologies will undoubtedly be a significant portion of the future of society, so we as a society must learn to live with them and optimize their capabilities. These gadgets are not simply Ring security cameras, iPhones, or TikTok. Streaming services recommend shows, movies, and other forms of entertainment to their users depending on what the users have previously watched or interacted with. Their data collection is imperative to recommend users new shows. To complement the research regarding the personal privacy question, creating a new multimedia experience was the primary focus for my capstone project. Because my capstone project interacts so closely with streaming services, the collection of data was and still is at the forefront of the project. The project utilizes a novel cooking experience to showcase a new type of interactivity in the digital entertainment industry. Three ways in which the project accomplishes this are through its mapping structure of navigation, its hands-free control, primarily through voice interaction, and the addition of avatars to engage the user as they progress through the multimedia experience. While the usage of data mining is not used in the first iteration of the prototype, it is very possible that the future prototypes could provide recommendations for other dishes given what the user has already cooked. Thus, the process of continually using user input to drive future recommendations stands at the forefront of this project, along with all other digital interactions done in modern-day society.

Digital entertainment has become the norm over the past decade, especially in recent years, due to the ease of at-home streaming. Individuals can now view movies and TV shows on demand and multiple entertainment companies are working on developing novel ways to keep their consumers engaged in a saturated industry. One major shortfall of on-demand streaming is that their services are mostly passive experiences for their users such as TV shows and movies where consumers perform no actions. In this project, a new type of entertainment experience that utilizes multi-media and multi-modal interaction to engage a diverse audience is designed for an active ‘edutainment’ how-to experience that differs from what is currently available. The specific use case this project focuses upon is an interactive cooking demonstration but could be outscoped for future development with industries such as auto-repair interactions like changing one’s oil. Specific novel features include an interactive map where users can navigate between various pages easily using a tree-like format. The map, and all the aspects of the design were built to be controlled optionally in a hands-free mode. This is achieved by specific voice and visual commands that the user can utilize to navigate through the interface. Additionally, an interactive avatar provides a humorous component by adding jokes and insight throughout various points. A second avatar serves as a probing tool that provides trivia questions to the users that relate to the content being shown. Wireframe prototypes have been created that include navigation, interaction, and presentation. We will also define and design for different user situations to accommodate varying user needs. Usability testing will be conducted on multiple versions of the prototype with users to determine what changes should be made to the design. First, the team will engage 3-6 different users in a full mock-up kitchen experience that allows them to use the prototype in a realistic situation. After that, the site UserBrain, an online user testing platform, will be used to engage around 30 different users to gather their feedback and opinions using the A/B testing method. The combined feedback will be used to make improvements to the prototype.

The Age of Digitization has brought countless advances in both society and technology, especially with regards to developing smart technologies, devices that are able to learn user patterns and adapt to help make that user’s life easier. This particular research will focus on the sociotechnical relationship between smart technologies such as TikTok that use data collection to provide a positive user experience and inherent concerns about personal privacy that exist when collecting data on those same users. To supplement the initial analysis, Actor Network Theory will be used as a framework to understand the intricacies of dealing with the competing stakeholders. Among these are companies, users, government officials, and the devices themselves. Connections between different groups expose the necessity for increased regulation between handling and misusing data. To argue against this, many users’ opinions about the handling of their personal data is relatively irrelevant to them, given some younger generations’ reliances on the Internet and other measures of interconnectivity to live how they desire. While there is no one solution to solve this problem, this research aims to find where lines have been drawn in the past and how can we as a society better draw them in the future to avoid the total loss of sovereignty over one’s personal data, if we have not lost that already.

Focusing on a human-centered design project while simultaneously writing about the societal implications of a similar concept has been tricky but exciting. Problems posed from writing a full-fledged research paper have helped develop the technical project further to recognize how companies can exploit users for their own benefit. This has caused the group to slightly adjust prototypes, an example being the way in which future media is recommended to the user. The group decided to not operate in that space, and to more solely focus on the navigational structure, hands-free control, and avatar entertainment that makes the project novel.

School of Engineering and Applied Science
Bachelor of Science in Systems Engineering
Technical Advisor: Gregory Gerling
STS Advisor: Bryn Seabrook
Technical Team Members: Caton Gayle, Erin Hensien, Grace Ko, Megan Lin, Sreya Palnati