Developing a ChatGPT-Based Moral Distress Consultant Assistant; Technological Momentum Analysis of the Failure Behind the Therac-25

Both my technical and STS research projects are connected by the technologies they focus on being employed in the healthcare industry, and the influence these technologies have, or had, on the industry. Where my two projects diverge is both the time and scale of the technologies that have been deployed and used. My technical project focuses on how to approach the problem of large language models (LLMs) influencing institutions in the healthcare industry through the deployment of chatbots, resulting in many unfortunate cases of harm coming to users. To better understand how technology can gain momentum to influence the practices and values of the very society that created and shaped it, my STS research examines how the Therac-25 gained momentum to influence the healthcare industry utilizing it, resulting in 6 unfortunate casualties. In my technical report, I explore the idea of using an LLM, primarily ChatGPT, to develop a chat assistant for use by moral distress consultants to provide additional information after each consultation. Through the use of prompt engineering, a GPT Assistant was created to assist moral distress consultants with giving alternative causes of moral distress, barriers to action, and strategies to tackle the situation at hand that caused the moral distress. In a Flutter application, the assistant is accessed through its unique ID and runs on a thread of messages between the user and the assistant, allowing for communication with the assistant in conversation context. The goal of this chat assistant is to provide use to moral distress consultants while avoiding the pitfalls of bad outputs that current chatbots deal with through the use of prompt engineering and professional oversight by the consultants themselves. While my technical project focuses on technology only recently being used in the healthcare industry, my STS research focuses on technology that has run its course. It examines the Therac-25 using technological momentum, a framework developed by Thomas P. Hughes, which is a framework that is used to analyze how society loses its influence over technology and is instead influenced by it. The Therac-25 was a radiation therapy machine that led to a series of 6 cases of radiation overdose, with common causes like software bugs and organizational failures being blamed. Using the technological framework, my STS research paper argues that the technological system surrounding the Therac-25 gained enough momentum to influence the society that once created and shaped it, creating the causes mentioned previously to exist in the first place. By analyzing how the Therac-25 gained enough momentum to influence the healthcare industry, I hope to present a new understanding of how society can be influenced by the technologies that it created. Working on my STS project informed and influenced the work I did on my technical project. Understanding how ChatGPT works and developing instructions for a GPT Assistant through prompt engineering allowed me to understand how LLMs work. Specifically, it allowed me to understand why chatbots built on LLMs like ChatGPT can give bad outputs. The work I did on my STS research paper informed me of how LLMs can influence institutions in the healthcare industry by giving bad outputs, leading to unfortunate cases of harm coming to users. In the end, the STS research paper motivated me to find a solution to avoid harm coming to end users of my chat assistant, resulting in a higher-quality technical project.

School of Engineering and Applied Science

Bachelor of Science in Computer Science

Technical Advisor: Kevin Sullivan

STS Advisor: Benjamin Laugelli

Technical Team Members: Steven Myung