Recommendations for User-Centered Design of Intelligent Systems in Healthcare; Perception is Reality: Significant and Novel Events Alter Patients' Attitudes Toward Electronic Health Records

Health-focused artificial intelligence is extremely beneficial to medicine, such as a Japanese AI system that analyzes endoscopic images for cancerous growths and is 94% effective. However, some AI models unintentionally misrepresent patient experiences, either through a lack of data or incorrect logic, which can seriously risk patient wellbeing. For example, many mobile applications that use image analysis to identify melanoma skin care are prone to false negatives, which delays diagnosis. Both projects examine factors that play a critical role in a medical AI system’s successfulness. My STS research examined an external factor: how patients perceive electronic health records (EHRs) and why they are reluctant to use this technology. If patients opt out of EHRs, AI models have less data on which to train and are less effective. In comparison, my technical project focused on an internal factor: how the development of a model’s logic can be a user-centered process. Developers who consistently engage with users (patients) are more likely to produce less biased systems.

My STS research demonstrated that since 2004, patient views of EHRs have shifted negative due to the increasing frequency of events that challenge the privacy and confidentiality of their medical data. For this work, I used Kerschner and Ehlers’ techno-attitudes framework to classify how patients have perceived EHRs over time. I gathered key statements from articles published around the time of significant events and classified them using the framework. The analysis suggested that when EHRs were still a novelty, patient attitudes were largely positive. However, as EHR adoption grew, so too did the prevalence of their harmful effects. Attitudes became more negative; patients lost trust in these systems and began to opt out of them. Therefore, patient trust, especially in the security and confidentiality of their data, is a major external factor that affects how effective medical AI can be.

My technical project also found that patients’ trust in medical technology is challenged when these systems reduce the quality of their care or risk their health. For instance, if the logic of an AI system represents biased assumptions that do not reflect users’ perspectives, patients may go undiagnosed and without necessary treatment. To address this issue, my technical project recommends incorporating user-centered design principles into all stages of the logic development process. First, team members and stakeholders should be identified with the goal of including all groups with insight into the target community’s experience. Next, the team should conduct initial user research with the identified stakeholders. Then, the team should develop a generalized proposal for the system’s logic and data and seek user feedback. Finally, the team should iteratively implement the proposed design until it reaches a final prototype. Throughout development, the team should maintain thorough documentation.

The results of my STS and technical projects both emphasize the importance of the end user. My technical project showed that a system can only be well-designed if it is built around the user’s needs and perspectives. My STS research demonstrated that even if a medical AI system is well-designed, it may not succeed due to patient apprehensions about EHRs, an adjacent technology. These results reinforce the idea of actor-network theory that is emphasized in sociotechnical systems thinking. To increase the likelihood of a system’s success, engineers should first understand the full network surrounding the project, from end users to related technology and social institutions. These projects also highlighted the engineer’s responsibility to ensure their project protects their users’ wellbeing. AI medical systems have a direct impact on a patient’s medical care. Therefore, engineers should ensure their projects empower the user by building systems with intention and care.

School of Engineering and Applied Science

Bachelor of Science in Computer Science

Technical Advisor: Daniel G. Graham

STS Advisor: Kathryn A. Neeley