Two Hands are Better Than None: Integrating Hand-Tracking into the Lumis inSight Platform; Understanding How an Online Setting Affects Learning and How to Adjust Classes Accordingly

It is fair to say that the 2020s began as a decade like no other. The emergence of the COVID-19 pandemic transformed the world almost instantly, putting countries on pause as people fought to save struggling businesses, provide adequate education, and preserve public health and welfare. For me as a college student, one of the most tangible effects of the pandemic was the transition from the traditional, classroom-based learning style that I had experienced all my life to online learning. As such, it was a natural decision to devote my fourth-year thesis to investigating online learning – how it affects students, how to improve it, and what it means for the future of learning in general. For the technical portion of this process, I built upon my work with Lumis Corporation to provide more accurate medical simulation software to nursing students. As a research project, I investigated how students learn differently in an online setting, and used that knowledge to predict ways both online and traditional learning will develop in the future. These two components shared a common thread; while my research explored the flaws in online learning and hypothesized the possibilities, my technical project worked to make those possibilities reality. Both left me excited for the future of online learning.

Colleges are known for offering a diverse array of classes. There are practical laboratories, massive lectures, intimate discussions, and of course, online courses. Each class format has strengths, and each has weaknesses, but they together they form a comprehensive learning experience. My research explored the strengths and weaknesses of online learning specifically, focusing on how different learner types respond to online learning, how the online setting affects the ability to comprehend and absorb information, how students react to online learning, and how online learning affects critical thinking and mental development. Utilizing Frank Geels’s multi-layer perspective, I analyzed how the COVID-19 virus initiated a transition to online learning. Under that framework, I reasoned that the new status quo – online courses – would generate niche-level innovations to improve online learning, such as tools that emphasize interaction, personalized feedback, and active learning.

A perfect example of the sort of personalized, practical online learning tool that I proposed will become more common in my research project is the inSight Platform offered by Lumis Corporation. Lumis aims to make medical simulations more accessible to students by offering a number of accurate, informative, and feedback-driven learning modules – such as teaching nursing students to administer medications or defibrillate a patient undergoing ventricular fibrillation. My project focused on the actual simulation process; when I began, users would interact with the platform by dragging a Bluetooth pen equipped with an infrared light over a mannequin. That light would be registered by a Nintendo Wiimote and converted to mouse coordinates, then sent to the corresponding point on a simulated body. However, in a real-world environment, medical staff do not drag pens across patients – they use their hands. In light of this, I worked to eliminate the need for a pen; I integrated hand-tracking software into the platform, mapped various hand gestures to different tasks, and ran test users through a demo defibrillation lesson. Users were able to click through various menus, drag and drop defibrillation pads, and navigate between different computer displays without every having to touch a mouse (or pen). This resulted in a user experience better afforded to medical treatment, and will enable more accurate simulation and a more immersive experience for all nursing students who use the platform.

I’ll be honest, when the transition to online learning was announced, like most students, I was not thrilled. I was a creature of habit when it came to education. Every year, I always had my color-coded notebooks and freshly sharpened pencils at the ready, and most importantly, I was always learning in person. I was scared – scared that after fifteen years of learning the same way, day in and day out, that my learning style and work ethic wouldn’t transfer to a new environment. And to be sure, it was hard at first. But as the year – and this project – progressed, I came to appreciate what online learning had to offer. I had more free time. I took better notes. I gained a better understanding of what works for me as a student. I owe a lot of that growth to this project; my research paper taught me different learning and teaching styles for all types of students, and encouraged me to seek out courses whose structure I felt best meshed to an online format. Meanwhile, my work with Lumis led to me putting those ideas into practice, and constantly working to develop software that would enhance the online learning experience. And while my technical project has yet to be tested in a real classroom, many of my other projects with this company have been used and enjoyed by students around the country. I’ve discovered that online learning isn’t perfect – but not many things are. Yet despite its defects, both aspects of my project show that there is a lot of potential for growth, and I am truly excited to see how it develops.

School of Engineering and Applied Science
Bachelor of Science in Computer Science
Technical Advisor: Raymond Pettit, Mark Floryan
STS Advisor: Kathryn Neeley