Abstract
The growing deployment of electronic health record (EHR) systems in nationwide hospital systems coupled with recent breakthroughs in artificial intelligence present a tremendous opportunity for technological development in the healthcare industry. Predictive analytics, such as the HERO score deployed in neonatal intensive care units (NICUs) and the CoMET score in adult critical care wards, leverage the data collected at a patient’s bedside to offer clinicians insight into the patient’s risk of near-term instability. These insights are significant. In 2011, the HERO score was correlated with a 20% reduction of all-cause mortality in the NICU during a 3000 patient, multicenter, randomized clinical trial. In one adult Surgical Trauma ICU, the introduction of the CoMET score was similarly connected to a remarkable 50% reduction in incidence of septic shock, a leading cause of death in that care setting. These results validate many clinicians’ and researchers’ feeling that predictive analytics have the potential to revolutionize medicine. Many other new technologies seek to use EHR data to improve patient care; some examples include retrospective analysis of a patient’s complete medical history, AI for medical imaging and MRI data, and natural language models which learn from clinicians’ notes. Although these are interesting, they fall outside of the scope of my specific projects. This two-part capstone project investigates the following research problem: How can bedside predictive analytics be developed and implemented to improve critical care outcomes?
The technical project seeks to develop a predictive analytic algorithm to predict bloodstream infection, a precursor to sepsis with very high risk of mortality. The healthcare industry standard to detect bloodstream infection is the Sepsis Inflammatory Response Syndrome (SIRS) criteria, which is a simplistic rule-based algorithm easily memorized by clinical staff. Once a patient is identified as high-risk, the clinical team must order a blood culture with a typical response time of 1-3 days; before the blood culture returns, the clinicians must treat the patient without specific knowledge of the patient’s condition. This often leads to overuse of antibiotics or unnecessary intervention. These issues motivate the development of a predictive algorithm capable of predicting bloodstream infection with greater precision than existing clinical approaches. In the technical research project, we developed and leveraged a dataset of 40,000 patients from the UVA hospital system to develop deep learning algorithms which predict bloodstream infection in a critically ill population. One limitation facing these algorithms is the presence of missing data in the vital sign and lab data in a patient’s EHR. Therefore, another focus of the technical research was to understand and experiment with a variety of state-of-the-art techniques to impute, or fill in, missing data. Although the algorithms developed for this project are not yet ready for the clinical setting, they represent a significant step towards the long-term goal of predictive analytics which address bloodstream infection.
The STS research project takes a broader view and examines the sociotechnical system in which predictive analytics are embedded. It asks the research question, “How do nurses and physicians incorporate predictive analytics into their diagnosis and treatment of patients?” Due to the rapid pace of predictive analytics’ implementation in high-stakes critical care settings, there is a gap in understanding how clinicians use their predictions to improve patient care. To address this problem, I interviewed eight clinicians (four doctors, two nurses, and two research coordinators) about their experience during the recent 11,000 participant, randomized clinical trials of the CoMET score in fourth floor adult critical care wards. The interviews were focused on the expected and actual use cases clinicians discovered for the CoMET system, the ways in which clinicians calibrated their intuition and trust, and the educational and logistical systems surrounding CoMET’s implementation. I found clear consensus among interview subjects that one major issue facing CoMET is interpretability – clinicians generally appreciate an indication of rising risk, but they really want to know why the predicted risk is increasing. This gap in understanding colors the ways that clinicians use CoMET in their treatment of patients. I also found that the predictive analytic algorithm itself was only a tiny piece of the puzzle; a good algorithm is necessary but not sufficient for a good system. Such a system requires, minimally, the proper education and buy-in of attending physicians and nurses all the way down to residents and interns. Achieving this education and buy-in requires support and involvement from all aspects of medicine, from medical school education to employee training to experiential learning through real-life patient care. It is through this lens that the STS research project investigates the sociotechnical system of healthcare technology.
Both projects were fruitful in their own ways; the STS research project was especially exciting in that I had a unique opportunity to interview a large group of experts in my field of interest. I gleaned many valuable insights about the role of technology in healthcare which I will take with me into my career. I felt that this project went significantly better than expected. There are numerous opportunities for future research on this topic, such as investigating the effect of predictive analytics on the culture of accountability among clinical teams and examining predictive analytics’ role in aiding communication between clinicians and families.
The research capstone project was an extension of my research during my 3rd year, and I did not achieve the results I set out to achieve. Existing imputation techniques in the literature proved very difficult to replicate, and I was unable to substantively improve the performance of our predictive analytic tool. I also found myself pulled in many directions, with difficult classes and two different research projects, which limited my ability to explore new approaches which may have helped improve my project. Although I did not achieve my desired results, the research project was valuable and did achieve a significant result. Deriving insight from the STS research, future research on this topic should be directed towards interpretable machine learning models. This problem would also benefit from a standardized dataset for consistent comparison. A valuable project would indeed be to create a public standardized dataset for our task. Furthermore, I submit that any research conducted in this domain must be done in collaboration with clinicians spanning the healthcare system, from nurses to seasoned veterans of critical care.
