Indoor Place Recognition for Situation Awareness Using Deep Learning Models and 3D Point Cloud Computation

With the increasing number of sensing devices in smart buildings for temperature, humidity, air quality, etc. acquiring information with regards to the state of the building and the location where a user is present has become very important recently in multiple domains including design, construction, facility management, and emergency response. This concept has been defined as Situation Awareness (SA) which can be obtained and improved by visualizing the information properly. The very first step to visualize such information is to identify the accurate location and orientation of the user in the indoor space. For accurate indoor localization, we are proposing to use a 3D point cloud model to train an end-to-end deep neural network which then is implemented on a head-mounted Augmented Reality (AR) platform (Microsoft HoloLens) to spatially localize users within an indoor space. To achieve this, a point cloud 3D model with more than 190 million points of a 17,000 sq. ft. open office space with diverse specifications such as office rooms, hallways, open areas, and crowded spaces were collected and registered by laser scanners. After pre-processing and generating submaps with different sizes, they are ultimately turned into global descriptor vectors to match with the input queries that come from the user’s ambient scanning with the HoloLens. Also, a real-time framework is designed to run on the HoloLens in which it captures the scanned 3D mesh around the user in real-time then sends the query to the neural network model by utilizing RESTful web service. Finally, the best sub-map candidates that match the queried 3D point cloud will be fetched to localize the user accurately. As result, we were able to reach 90% recall for the average top 1% of the results in the range of 1-meter threshold.