Abstract
Complex hardware/software systems are ubiquitous, affecting every aspect of daily life. Software is integral to the normal functioning of critical systems such as power plants, financial systems, communication systems, modern medical systems and devices, and transportation systems to name a few. Because of society's increasing reliance on these systems, it is of paramount importance that software perform as intended, and not be subverted for malicious purposes. Consequently, techniques that thwart reverse engineering and tamper, (called tamper-resistance techniques), have become
increasingly important as a means to hinder malicious exploitation of software in critical systems.
Given the growing importance of preventing tampering with critical systems, research in this area has grown. Recently, software virtualization has been proposed as a suitable mechanism to impart tamper resistance to software applications. However, protections based on virtualization have not fully matured, which has led to successful attacks. This dissertation is the culmination of a detailed study examining the application of low-overhead process-level virtualization to protect software applications from reverse engineering and tamper. This research is structured as follows: First, a formal model describing virtualization is presented. The model is useful in describing general-purpose computing systems and the applicability of virtualization in protecting applications. Then we explored several novel tamper-resistance techniques that are based on process-level virtualization. Each technique was thoroughly evaluated in terms of performance overhead and protection. During the course of our investigation, a serious vulnerability in current process-level virtual machines was discovered. We modeled this vulnerability using our formal model and describe two attack implementations that successfully exploit this vulnerability. Finally, we conceptualize a revolutionary protection technique to compose an application with multiple virtual machines, providing robust program protection. The ideas presented in this dissertation are evaluated using current state-of-the-art attacks to gauge its effectiveness. The results of our investigation reveal that composable virtual machines are significantly more effective in thwarting reverse engineering and software tamper than current protection techniques.
