Testing in Resource-Constrained Environments

In recent software development environments, resources including time, memory, and power constrain the execution of software test cases. Often the cost of running all test cases is prohibitively expensive, necessitating techniques to select test cases of the highest quality for test execution. While many test case selection and prioritization techniques exist to select test cases that are more likely to find faults, none take the resource constraints under which the tests will operate directly into consideration. Another cost in test case execution comes from structural testing. Structural testing is an important technique in which during execution test cases are also evaluated for structural coverage. This is typically performed through the use of code instrumentation. However, instrumentation incurs high overhead in terms of time and code growth, making it difficult to use in resource-constrained environments such as mobile device testing.

This thesis develops and evaluates techniques for efficient and effective execution of test cases in resource-constrained environments. The first set of techniques selects high quality test cases from test suites while taking resource constraints into account. The resulting test selection is guaranteed to execute within the specified resource constraints, and the resulting test cases have the highest possible potential for overall fault detection during their constrained execution. The selected tests are additionally ordered in a way that is likely to find faults earlier in test execution rather than later. These techniques represent the first to guarantee that test selections will execute within a given constraint, and they offer a spectrum of options for selecting tests to execute in resource-constrained environments.

The second set of techniques evaluates the quality of test cases during execution without the need of expensive and intrusive instrumentation. Instead, our quality evaluation techniques exploit recent hardware advances to monitor program execution. By leveraging these hardware advances, up to 90% of branch coverage and 79% of statement coverage can be determined with less time overhead compared to instrumentation. Because the techniques require little or no code modification or dynamic code insertion and can run on commodity machines, tablets, or mobile devices, the techniques also enable test case execution and evaluation on resource-constrained devices. No specialized hardware or components are required. The techniques are combined into a runtime and static testing system called THeME: Testing by Hardware Monitoring Events. THeME is a portable, extensible system, making it applicable for use in a wide range of resource-constrained environments.