Transparent System Introspection in Support of Analyzing Stealthy Malware 1770 views

The proliferation of malware has increased dramatically and seriously degraded the privacy of users and the integrity of hosts. Millions of unique malware samples appear every year, which has driven the development of a vast array of analysis tools. Malware analysis is often performed with the assistance of virtualization or emulation for rapid deployment. Malware samples are run in an instrumented virtual machine or analysis tool, and existing introspection techniques help an analyst determine its behavior. Unfortunately, a growing body of malware samples has begun employing anti-debugging, anti-virtualization, and anti-emulation techniques to escape or otherwise subvert these analysis mechanisms. These anti-analysis techniques often require measuring differences between the analysis environment and the native environment (e.g., executing more slowly in a debugger). We call these measurable differences artifacts. Malware samples that use artifacts to exhibit stealthy behavior have increased the effort required to analyze and understand each stealthy sample. Additionally, traditional automated techniques fail against such samples because they produce measurable artifacts. We desire a transparent approach that produces no artifacts, thereby admitting the analysis of stealthy malware. We refer to this challenge as the debugging transparency problem. Solving this problem is thus concerned with reducing artifacts or permitting reliable analysis in the presence of artifacts. We present a system consisting of two approaches to address the debugging transparency problem and then demonstrate how these components can apply to currently available computer systems. We present two techniques capable of transparently acquiring snapshots of memory and disk activity that can be used to analyze stealthy malware. First, we discuss a novel use of a custom Field-Programmable Gate Array that provides snapshots of memory and disk activity with no measurable timing artifacts. Second, we present a novel use of System Management Mode on x86 platforms that produces no functional artifacts at the expense of producing timing artifacts. Finally, we present an approach to evaluating the tradeoff space that exists between analysis transparency and the fidelity of introspection data provided by such an analysis system. Together, these approaches form a cohesive solution to the debugging transparency problem that admits analyzing stealthy malware.

PHD (Doctor of Philosophy)

English

2016-12-06