A Cognitive Assistant System for Context Inference and Decision Making in Emergency Medical Services

In emergency situations, the first responders need to collect, aggregate, filter and interpret information from different static and real-time sources and provide timely interventions and treatments to victims in a short period of time. Dealing with such a huge information load at the incident scene requires a significant amount of human cognitive effort. This thesis presents a cognitive assistant system for emergency medical services (EMS) that aims at improving situational awareness of the first responders by automated collection and analysis of data from the incident scene and providing suggestions on the most effective response actions to them. The proposed system relies on a Behavior Tree (BT) framework that combines the knowledge of EMS protocol guidelines with speech recognition, natural language processing, and machine learning methods to (i) extract critical information from responders' conversations and verbalized observations, (ii) infer the incident context, and (iii) decide on safe and effective response interventions to perform. We use a data-set of 8302 real EMS call records from an urban, high volume regional ambulance agency in the United States to evaluate the responsiveness and cognitive ability of the system and assess the safety of the suggestions provided to the responder. The experimental results show that the developed cognitive assistant achieves an average top-3 accuracy of 89% in selecting the correct EMS protocols and an average F1-score of 71% in suggesting the protocol specific interventions while providing transparency and evidence for the suggestions. We also simulate the streaming speech from emergency scenes to examine the effectiveness of the developed model in providing timely accurate suggestions. The simulation results show that the proposed cognitive assistant is able to achieve an average 70% F1-score in predicting correct interventions with only 45% of the input speech.