Binary instrumentation and transformation for software security applications
Abstract
The capabilities of software analysis and manipulation are crucial to counter software security threats such as malware and vulnerabilities. Binary instrumentation and transformation are the essential techniques to enable software analysis and manipulation. However, existing approaches fail to meet requirements (e.g. flexibility, transparency) specific in software security applications. In this dissertation, we design and implement binary instrumentation and transformation systems specifically for software security applications. First, we present BISTRO, a static binary transformation framework that can extract/embed binary components from/into existing binaries without source code, symbolic or relocation information. We propose two algorithms to patch both direct and indirect control-flow transfer instructions when performing static binary transformation. Second, we present SPIDER, a dynamic binary instrumentation framework that enables efficient instruction-level instrumentation that is transparent to the instrumented binary program. In SPIDER, we propose a novel instrumentation primitive based on hardware virtualization called invisible breakpoint to replace traditional software breakpoint for better transparency, and design an algorithm to monitor the virtual-to-physical address mapping in hardware memory virtualization. Finally, we present iRiS, an iOS application vetting system for detecting private API uses. We propose a novel analysis of iOS applications using a combination of static analysis and dynamic binary instrumentation, and build iRiS on top of a dynamic binary instrumentation framework ported to iOS by us. We build the prototypes of the three aforementioned systems and evaluate their performance against real-world binary programs. BISTRO is able to transform large-scale binary programs such as Adobe Reader, and incurs trivial runtime overhead (1.9% on average) and small space overhead (11% on average). SPIDER remains transparent against all state-of-the-art anti-instrumentation detections, and incurs reasonable overhead which is similar to hardware breakpoint. We also apply BISTRO and SPIDER in five scenarios to demonstrate their effectiveness in software security applications. iRiS successfully identified 149 (7%) malicious applications from 2019 applications that have passed the official application vetting process of iOS. From these malicious applications, iRiS has found the usage of a total number of 153 different private APIs including 28 security-critical APIs that access sensitive user information such as device serial number.
Degree
Ph.D.
Advisors
Zhang, Purdue University.
Subject Area
Computer science
Off-Campus Purdue Users:
To access this dissertation, please log in to our
proxy server.