Abstract
Multitasking is a core challenge in human-system interaction, particularly in safety-critical domains such as aviation and driving. As modern interfaces increase in complexity, human operators often face cognitive demands that exceed their attentional capacity. This leads to performance breakdowns. Thus, it is the interface designer’s responsibility to design displays that minimize dual-task interference—the performance cost incurred when two tasks are performed simultaneously compared to individually. Prior research has identified key factors that shape dual-task interference, including sensory modality, task load, and task type. However, the interaction between these factors remains relatively understudied, especially in immersive environments like virtual reality (VR).
This dissertation investigates how task load (low vs.high) and type (object-based vs. spatial) shape dual-task interference in visual-auditory and visual-tactile displays in VR. Across four experiments,
participants completed a dual-task paradigm, where they performed a continuous visual tracking task alongside detection tasks that varied in modality and task type. Results consistently showed that cross-
modal task pairings (i.e., visual-auditory, visual-tactile) reduced dual-task interference relative to intramodal pairings (i.e., visual-visual).However, task type had mixed findings: using distinct task types reduced dual-task costs in the visual-auditory combination but did not have a significant benefit in the visual-tactile combination. Task load impaired dual-tasking performance, but detection tasks were more
susceptible to load-induced costs than the tracking task, indicating prioritization of the latter. Expected crossmodal effects such as load-induced inattentional deafness and numbness were not observed, suggesting that these phenomena may be task-type dependent.
Based on the aforementioned results, this dissertation developed a computational model grounded in Multiple Resource Theory (MRT).This model simulated dual-task performance by incorporating task
demand, resource conflict, visual angle separation, and task priority.The model robustly accounted for most observed patterns, though MRT’s assumptions regarding the tactile modality may require further examination.
By integrating empirical findings with predictive modeling, this work contributes to the design of more effective multimodal systems for multitasking environments. It also provides actionable insights
into when multimodal displays reduce interference, when they do not, and why this is the case. Together, the findings advance our theoretical understanding of multimodal multitasking, challenge assumptions about sensory independence in multimodal displays, and inform human-centered interface design in complex, high-stakes settings.
