A General-Purpose, Energy-Efficient, and Context-Aware Acoustic Event Detection Platform for Mobile Devices

Humans are extremely capable of remembering, recognizing, and acting upon hundreds of thousands of different types of acoustic events on a day-to-day basis. Decades of research on acoustic sensing have led to the creation of systems that now understand speech, recognizes the speaker, and finds a song. However, apart from speech, music, and some application specific sounds, the problem of recognizing varieties of general-purpose sounds that a mobile device encounters all the time has remained unsolved. The overarching goal of this research is to enable rapid development of mobile applications that recognize general-purpose acoustic events. As these applications are meant to run on a mobile device, the technical goals of this research include - energy-efficiency, communication-efficiency, leveraging the user contexts such as the location and position of the mobile device in order to improve the classification accuracy, and ease of application development.

With this goal in mind, we have built a general-purpose, energy-efficient, and context-aware acoustic event detection platform for mobile devices called - the Auditeur platform. Auditeur enables mobile application developers to have their application register for and get notified on a wide variety of acoustic events. Auditeur is backed by a cloud service to store crowd-contributed sound clips and to generate an energy-efficient and context-aware classification plan for the mobile device. When an acoustic event type has been registered, the mobile device instantiates the necessary acoustic processing modules and wires them together to dynamically form an acoustic processing pipeline in accordance with the classification plan. The mobile device then captures, processes, and classifies acoustic events locally and efficiently. Our analysis on user-contributed empirical data shows that Auditeur's energy-aware acoustic feature selection algorithm is capable of increasing the device-lifetime by 33.4%, sacrificing less than 2% of the maximum achievable accuracy. We implement seven applications with Auditeur, and deploy them in real-world scenarios to demonstrate that Auditeur is versatile, 11.04% - 441.42% less power hungry, and 10.71% - 13.86% more accurate in detecting acoustic events, compared to state-of-the-art techniques. We perform a user study involving 15 participants to demonstrate that even a novice programmer can implement the core logic of an interesting application with Auditeur in less than 30 minutes, using only 15 - 20 lines of Java code.

Besides Auditeur, three other systems have been developed in this research which empower some aspects of the platform. First, we introduce sMFCC, which is an approximation to a well-known acoustic feature called the Mel-frequency cepstral coefficient (MFCC). The motivation behind sMFCC is to enable faster extraction of MFCC features in applications that must sample the microphone at a very high rate and yet has to meet the real-time requirement. Second, we have developed the MultiNets, which is capable of switching wireless networking interfaces (WiFi and 3G) on a mobile device based on a predefined policy, such as saving energy during wireless data communication. The motivation behind MultiNets is to enable energy-efficient mobile-cloud communication in applications that must communicate quite frequently with a remote server over the Internet. Third, we have developed the Musical-Heart, which is a system that provides a biofeedback-based and context-aware music recommendation service to a mobile device. The motivation behind Musical-Heart is to build a convenient, non-invasive, personalized and low-cost wellness monitoring system that obtains heart rate and activity level information from a pair of specially designed earphones while a user is listening to the music on his mobile device.