Bringing a Sociotechnical Lens to User Experience (UX) Research: Analyzing an AI-Based Simulation

Recent growth in artificial intelligence (AI) based technologies adds complexity to designing and evaluating human-computer interaction (HCI) and learning design. Many research studies on the usability and user experience (UX) of learning technologies follow evaluation approaches originally designed for HCI. However, due to numerous differences between the fields, the usability tools often do not properly align. This study outlines the need for expert and user evaluation tools following the sociotechnical theory to guide research on the usability evaluation of an AI-based Conversational Agent (CA) simulated experience for preservice teachers to practice teaching mathematics. This study combines the TPACK framework (technology, pedagogical, and content knowledge) and the Community of Inquiry framework to evaluate the social, pedagogical, technical, and content aspects of a simulated learning experience, creating the Social-TPAC heuristic evaluation tool. The study uncovered how the technical and content aspects of a user experience impacted the pedagogical and social aspects of learning, while also uncovering the need for human presence to facilitate AI-based simulation education.