Secure In-VM Monitoring Using Hardware Virtualization
Monirul Sharif
Georgia Institute of Technology
Atlanta, GA, USA
msharif@cc.gatech.edu
Wenke Lee
Georgia Institute of Technology
Atlanta, GA, USA
wenke@cc.gatech.edu
Weidong Cui
Microsoft Research
Redmond, WA, USA
wdcui@microsoft.com
Andrea Lanzi
Institute Eurecom, Sophia Antipolis, France, lanzi@eurecom.fr
Georgia Institute of Technology,Atlanta, GA, USA, andrea@cc.gatech.edu
ABSTRACT
Kernel-level attacks or rootkits can compromise the security of
an operating system by executing with the privilege of the kernel.
Current approaches use virtualization to gain higher privilege over
these attacks, and isolate security tools from the untrusted guest
VM by moving them out and placing them in a separate trusted
VM. Although out-of-VM isolation can help ensure security, the
added overhead of world-switches between the guest VMs for each
invocation of the monitor makes this approach unsuitable for many
applications, especially fine-grained monitoring. In this paper, we
present Secure In-VM Monitoring (SIM), a general-purpose frame-
work that enables security monitoring applications to be placed
back in the untrusted guest VM for efficiency without sacrificing
the security guarantees provided by running them outside of the
VM. We utilize contemporary hardware memory protection and
hardware virtualization features available in recent processors to
create a hypervisor protected address space where a monitor can
execute and access data in native speeds and to which execution
is transferred in a controlled manner that does not require hyper-
visor involvement. We have developed a prototype into KVM uti-
lizing Intel VT hardware virtualization technology. We have also
developed two representative applications for the Windows OS that
monitor system calls and process creations. Our microbenchmarks
show at least 10 times performance improvement in invocation of a
monitor inside SIM over a monitor residing in another trusted VM.
With a systematic security analysis of SIM against a number of pos-
sible threats, we show that SIM provides at least the same security
guarantees as what can be achieved by out-of-VM monitors.
Categories and Subject Descriptors
D.4.6 [OPERATING SYSTEMS]: Security and Protection
General Terms
Security
Keywords
Virtual Machines, Secure Monitoring, Kernel Integrity, Malware
Permission to make digital or hard copies of all or part of this work for
personal or classroom use is granted without fee provided that copies are
not made or distributed for profit or commercial advantage and that copies
bear this notice and the full citation on the first page. To copy otherwise, to
republish, to post on servers or to redistribute to lists, requires prior specific
permission and/or a fee.
CCS’09, November 9–13, 2009, Chicago, Illinois, USA.
Copyright 2009 ACM 978-1-60558-352-5/09/11 ...$10.00.
1 Introduction
Kernel-level attacks or malicious programs such as rootkits that
compromise the kernel of an operating system are one of the most
important concerns in systems security at present. These attacks
can modify kernel-level code or sensitive data to hide various ma-
licious activities, change OS behavior or essentially take complete
control of the system. In addition, kernel-level security tools can
be crippled and made ineffective by these attacks. A large body of
research has adopted virtual machine monitor (VMM) technology
in an effort to mitigate such attacks because the higher privileged
hypervisor can enforce memory protections and preemptively in-
tercept events throughout the operating system environment.
A major reason for adopting virtualization is to isolate security
tools from an untrusted VM by moving them to a separate trusted
secure VM, and then use introspection [7,17] to monitor and protect
the untrusted system. Approaches that passively monitor various
security properties have been proposed [10, 11, 13, 19]. However,
passive monitoring can only detect remnants of an already success-
ful attack. Active monitoring from outside of the untrusted VM,
which has the advantage of detecting attacks earlier and preventing
certain attacks from succeeding, was enabled by Lares [16]. This is
achieved by placing secure hooks inside the guest VM that intercept
various events and invoke the security tool residing in a separate se-
cure VM. However, the large overhead for switching between the
guest VM, the hypervisor, and the secure VM makes this approach
suitable only for actively monitoring a few events that occur less
frequently during system execution.
Many security approaches require the ability to monitor frequently
executing events, such as host-based intrusion detection systems
(IDSs) that intercept every system call throughout the system, LSM
(Linux Security Module) [23] and SELinux that hook into a large
number of kernel events to enforce specific security policies, or
even instruction-level monitoring used by several offline analysis
approaches [4]. Due to the overhead involved in out-of-VM mon-
itoring, many such approaches either are not designed for produc-
tion systems, or are not created for VM’s. While keeping a monitor
inside the VM can be efficient, the key challenge is to ensure at least
the same level of security achieved by an out-of-VM approach.
In this paper, we present Secure In-VM Monitoring (SIM), a
general-purpose framework based on hardware virtualization fea-
tures that enables security monitors residing in the same VM it is
protecting to have the same level of security as residing in a sepa-
rate trusted or secured VM. A security monitor in our framework
retains the efficiency close to being inside the same VM by not re-
quiring any privilege transfers when switching to the monitor for
an intercepted event, and being able to access the system address
space at native speed. At the same time, isolation is achieved by
putting the monitor code along with its private data in a separate
