Abstract
Smart and automated management and maintenance of critical infrastructure is not only a concern for the departments of transportation and city officials, but also a necessity for the realization of next generation smart cities. The shortage of resources in the face of the size of the national infrastructure, and the subjectivity, limitations, and cost of traditional visual inspections demand innovative solutions that can reduce manual inspection and maintenance operations while enhancing infrastructure quality assurance. Examples of such solutions include robotic inspection and crowd-sourced monitoring. The intersection between such strategies is their use of images to record and document the state of health of a structure, and to allow for the detection of issues of interest using automated methods. In this regard, image-based automated structural inspection systems have gained traction and a variety of methods including those based on image processing and machine learning have been studied by the research community. Recently, visual defect recognition systems based on deep convolutional neural networks have shown great promise. However, a comprehensive study of the literature shows that there are numerous challenges facing the development of robust and efficient fully automated structural inspection systems.
This dissertation explores some of the challenges facing the use of deep learning for robust condition assessment of infrastructure. First, a framework for scalable multi-class urban defect recognition was proposed that leverages two cost-effective alternative data sources, namely web images and Google Street View scenes. Ten classes of urban defects were targeted and to minimize the need for human supervision, web images were used to train an initial model. This initial model then creates pseudo-labels for a massive pool of nearly 600,000 Google Street View images and a subset of the most confident pseudo-labels was used in conjunction with the web images to re-train the model. Data distillation using a set of geometric transformations on the unlabeled images was employed to increase the learning potential of the system. Results showed that the proposed self-training framework with data distillation helped increase the accuracy of the web-trained model on the Google Street View test set by nearly 20% to achieve a final accuracy of 80% over the ten categories. A sensitivity analysis and an error analysis also helped determine the influential factors affecting the performance of the model.
Next, one of the target classes of defects, namely cracks were selected for an in-depth analysis on detection and measurement algorithms. Unlike the majority of existing crack detection models that focus exclusively on one material, this work proposed three strategies to create robust models that can detect cracks in more than one material. These strategies include joint training, sequential training, and ensemble learning, each of which aim to enable a model to work on multiple domains of input data. For implementation, a patch-level crack detection model was created to demonstrate detection performance discrepancy across different materials (concrete vs asphalt). It was shown that using the proposed strategies, an existing pre-trained model can be adapted to work for other materials and a single model can be jointly-trained on different materials. Through the proposed solutions and comparisons with deep learning and edge detection baselines, the potential to increase the robustness and flexibility of deep learning crack detection models for practical real-world applications was demonstrated.
To improve the state of the art in defect detection in terms of the level of detail, a patch-level model was repurposed into a fully convolutional neural network, which was trained end-to-end on a dataset of high-resolution images manually annotated at the pixel-level. Aside from the fully convolutional architecture employed, three key techniques were leveraged to produce competitive performance. These include cost-sensitive learning to combat class imbalance, data augmentation with rescaling to diversify training data and cover multiple scales of cracks, and the inclusion of crack-like distractors to make the model robust to extraneous objects. The model was shown to be successful in correctly detecting over 92% of crack and 99.9% of non-crack pixels. Comparisons with the state-of-the-art patch-level and traditional edge detection and adaptive thresholding alternatives also highlighted the advantages of the proposed approach, in terms of level of details, image resolution independence, and computational efficiency.
Finally, the success of the proposed pixel-level crack detection model was then leveraged to quantify cracks, which is highly important in assessing the type and importance of the damage and the required maintenance measures. Pixel-level crack detections were subjected to a series of morphological operations and the width, length and directionality of cracks were determined. A definition that aligns well with the intuitive interpretation of a crack width was proposed and smoothing of width measurements was explored to increase robustness to jagged edge noise. Results show that the number of curve-fitting points employed in the estimation of crack orientation does not have a major effect on crack width estimation. Comparisons with both human crack width measurement and a number of state-of-the-art baselines demonstrated the consistency, usefulness and advantages of the method, which can be used to facilitate and accelerate manual structural inspections.
This dissertation is a collection of four manuscripts that describe the aforementioned research works. Through the presented results, this dissertation highlights the power of the emerging computer vision models based on deep learning in the field of automated structural inspections and introduces new opportunities for deployment using automated robotic and crowd-sourced inspection systems.
