Driver State Modeling through Multimodal Naturalistic Driving Data

Effective shared-autonomy requires a clear understanding of driver's behavior, which is governed by multiple psychophysiological and environmental variables. Disentangling this intricate web of interactions requires understanding the driver's state and behaviors in different real-world scenarios, naturalistically, and longitudinally. Naturalistic Driving Studies (NDS) have shown to be an effective approach to understanding the driver's state and behavior in real-world scenarios. However, due to the lack of technological and computing capabilities, former NDS only focused on vision-based approaches, ignoring important psychophysiological factors such as cognition and emotion.

This dissertation first introduces the HARMONY framework, a human-centered multimodal naturalistic driving study, where driver's behaviors and states are monitored through (1) in-cabin and outside video streams (2) physiological signals including driver's heart rate and hand acceleration (IMU data), (3) ambient noise, light, and the vehicle’s GPS location, and (4) music logs, including song features such as tempo. HARMONY is the first study that collects long-term naturalistic facial, physiological, and environmental driving data simultaneously.

This dissertation then proposes different methods to analyze drivers’ states and their interaction with different contextual factors in-cabin and on the road, in the wild. More specifically, based on the HARMONY dataset collected from 22 participants, this dissertation (1) classifies drivers' state and behaviors through supervised learning using the wearable data in the wild; (2) identifies the psychophysiological response of drivers' within different driving behaviors and kinematic movements of the vehicle through unsupervised learning; (3) identifies the association of specific infrastructural elements (e.g., intersections), traffic objects (e.g., trucks and riders), and traffic conditions (e.g., distance to the lead vehicle, and weather condition) with the changes in driver's heart rate through computer vision techniques and Bayesian Change Point detection, and (4) analyzes the changes in driver's state (i.e., stress level and workload) under the dynamic perturbations of the external environment (i.e., changes in the number of vehicles as a proxy for traffic density and hand movement as a proxy for task demands) through state-space latent variable modeling framework.

In reference to IEEE copyrighted material which is used with permission in this thesis, the IEEE does not endorse any of University of Virginia products or services. Internal or personal use of this material is permitted. If interested in reprinting/republishing IEEE copyrighted material for advertising or promotional purposes or for creating new collective works for resale or redistribution, please go to http://www.ieee.org/publications_standards/publications/rights/rights_link.html to learn how to obtain a License from RightsLink. If applicable, University Microfilms and/or ProQuest Library, or the Archives of Canada may supply single copies of the dissertation.