ViWatch: Harness Vibrations for Fine-Grained On-Body Interfaces with a Commodity IMU Sensor

Wearable devices like smartwatches and smart wristbands have gained substantial popularity in recent years. While enabling a variety of computing applications, such devices are not always convenient to interact with because of the limited size of the touchscreen. A wide variety of approaches have been considered to improve their user experiences, ranging from using customized RF sensors, to multiple sensors in smartwatches. These solutions all have limitations related to the characteristics of their technology. We propose ViWatch (Vibration Watch), which harnesses vibrations with an IMU sensor on commodity smartwatches to provide for fine-grained on-body interfaces.
We build an on-body tapping system and a micro finger writing system to support finger movement recognition. Challenges addressed include difficulties collecting and labeling a large vibration dataset, filtering human activity noise from finger typing and writing vibration signals through signal processing, designing a novel adversarial neural network to overcome human variations including those of typing strength, writing style, hand shape, and smartwatch wrist position, and adopting a recurrent neural aligner for enabling both continuous and discrete finger movement recognition. We build a neural network with adequate regularization to mitigate over-fitting on different training users. Also, We design a refinement and calibration scheme with adversarial learning and transfer learning to improve the system performance during daily usage. To summarize, the thesis statement is: By detecting on-body vibrations and modeling continuous vibrations through large user studies, we build fine-grained finger vibration interfaces using a single IMU sensor, thus providing a new usable interaction capability for commodity wearable devices with tiny/no touchscreens. We have posted two demo videos on YouTube: Finger writing (https://youtu.be/aAEPv8KJ1Jk) and finger tapping (https://youtu.be/N5-ggvy2qfI)