Distribution-Aware Testing of Neural Networks Using Deep Generative Models

The reliability of software with a Deep Neural Network (DNN) as a component is urgently important today given the increasing number of critical applications deployed with DNNs. The need for reliability raises a need for rigorous testing of the safety and trustworthiness of these systems. In the last few years, there have been a number of research efforts focused on testing DNNs. However, the existing methods have several limitations. The methods that test DNNs with pixel-level manipulations on seed inputs do not consider the validity of the inputs which results in invalid inputs wasting testing time and inflating coverage values. The methods that use feature-level manipulations avoid invalid inputs. However, they either lack a mechanism to automatically learn the features which makes them costly and infeasible for complex datasets or they do not systematically test DNNs with feature-diverse and rare inputs because of which the methods are inadequate to ensure the reliability of DNNs. This thesis advances the state of DNN testing by proposing a DNN testing framework, DisTest, that addresses all these limitations.
A black-box test adequacy criterion, IDC, and a black-box test generation algorithm, CIT4DNN, are developed over the framework for measuring the feature diversity of test inputs and generating diverse and rare test inputs respectively. Experimental studies demonstrate that the test coverage measured by IDC reflects the feature interactions present in a test set and is more sensitive to feature diversity than the baseline test coverage metrics. CIT4DNN generates test inputs with 100% test adequacy and is cost-effective when compared to the baseline test generation algorithms.