Abstract
Technical Project:
PERPETUALITY Multimodal Gesture Controlled Wearable Device
The technical problem of existing consumer gesture devices come from the reliance of accelerometers and gyroscopes as demonstrated by the discontinued Myo armband to the camera-based Leap. This limits them to having only broad hand motions with only 60-75% classification accuracy for individual digits. What we built, PERPETUALITY, is a wrist-form-factor gesture tracking device that fuses nine-axis inertial measurement unit (IMU) data with five-channel surface electromyography (sEMG) to achieve individual finger movement recognition as standard computer input. By monitoring tendon activity through sEMG electrodes strategically placed over the forearm's flexor digitorum superficialis and profundus, PERPETUALITY captures individual finger flexion intent with precision beyond IMU-only wearables. The idea is in targeting classification accuracies exceeding 95% at sub-50ms end-to-end latency. The system architecture centers: on a Nordic Semiconductor nRF52840 microcontroller interfacing with a TDK InvenSense ICM-20948 IMU via I2C and MyoWare 2.0 sEMG sensors via analog GPIO, with signal conditioning implementing fourth-order Butterworth bandpass filtering (50-450Hz for sEMG artifact rejection) and complementary filter fusion for IMU drift compensation. Sensor data streams to a host computer via USB where a machine learning classifier, a 1D Convolutional Neural Network with concurrent LSTM layers, inspired by CTRL-Labs at Reality Labs, does processing for the fused signal features and maps the recorded gesture predictions to Human Interface Device (HID) commands. The deliverable as discussed before is a functional wrist-worn prototype in this project’s custom 3D-printed enclosure with a hand-assembled multi-layer PCB. It demonstrates real-time gesture-controlled computer interaction including digital object manipulation and interface navigation. This has been recorded on a demonstrative video as well after completion. Edge inference with machine learning and HID makes it so that user intent with small wearables is decoded with low energy and no additional drivers, ensuring the device interfaces with standard HID protocols. This is an architecture designed for the future where users engage with AI and digital systems directly through the gesture of their wrist without opening a screen at the flick of their wrist.
STS Research Paper: The Fight for Legibility When Computation Determines Your Inclusion
The STS research paper investigates how gesture-controlled EMG devices, while promising to bridge the accessibility gap between able-bodied and disabled users, introduce a new form of segregation. This segregation defines the computationally legible versus the computationally illegible, defined in datasets that users had no part in defining but in every part being isolated from. The paper proceeds in three arguments. First, training datasets function as gatekeepers determining which bodies are eligible for recognition. The sEMG training data is overwhelmingly populated by recordings from able-bodied participants in controlled lab settings whose neuromuscular distributions define what the classifier learns to recognize as a "gesture," while users with spasticity, post-stroke hemiplegia, or peripheral neuropathy are structurally absent. Drawing on Kamikubo and team's meta-analysis of accessibility datasets and Simson and team's identification of "lazy data practices," the paper demonstrates that even when disability data is collected, the optimization objective structurally suppresses outlier contributions in favor of overall general user pattern performance. Second, the algorithm becomes the judge of ability, replacing relational human assessment with binary classification that lacks context, follow-up, or capacity to recognize intentional gestures outside the training distribution. Using Scully and team's three forms of "automated misrecognition" and Yamagami and team's empirical finding of an 18-percentage-point accuracy gap between disabled and non-disabled participants, the paper establishes that this exclusion is a design choice and not a technical limitation. Since personalized recognizers already achieve 91% accuracy across all populations, the exclusion is evident. Third, technoableism emerges when bodies are expected to conform to computational standards rather than vice versa. Drawing on Shew's framework and Bennett and Keyes' distinction between fairness and justice, the paper argues that even diversified datasets reproduce the power relation where technology defines the terms of legibility. The paper concludes with solutions: ability-based design from Wobbrock and team, co-constructed datasets following Costanza-Chock's design justice principles, and personalized classification architectures as demonstrated by Sîmpetru and team's MyoGestic framework.
How the Technical Project & the STS Research Paper Connect:
Building PERPETUALITY and writing the STS paper were not parallel projects they converged as investigations into the same fundamental question: who gets to be recognized by a gesture interface and who decides. The technical project forced direct confrontation with the assumptions embedded in every physical human centered engineering decision because electrode placement presumes particular forearm anatomies, signal filtering thresholds define what counts as "intent" versus "noise," and training datasets encode specific bodies as the universal standard. By testing PERPETUALITY across different users and different forearm positions, it revealed firsthand that signal amplitude, frequency content, and activation timing all vary with neuromuscular condition, confirmed in practice what the STS literature establishes in theory: that the feature space itself is constructed from bodies that do not represent the full range of users. Conversely, the STS research directly informed engineering choices in PERPETUALITY where the team adopted personalized classification from user-specific sEMG and IMU signals rather than deploying universal models, enabling individuals to grab digital objects, play Pacman, and control rotational sliders with the gesture wearable regardless of their particular motor profile. The technical work provided the empirical grounding that made the STS argument credible beyond theory, while the STS analysis provided the critical lens that prevented the technical work from reproducing the very exclusions it sought to overcome.
