【免费】KVM/ARM:TheDesignandImplementationoftheLinuxARMHypervisor资源-CSDN文库

hypervisor

ARM

需积分: 0 70 浏览量 2016-11-25 15:23:01 上传评论 5 收藏 563KB PDF 举报

资源推荐

资源详情

资源评论

KVM/ARM: The Design and Implementation

of the Linux ARM Hypervisor

Christoffer Dall

Department of Computer Science

Columbia University

cdall@cs.columbia.edu

Jason Nieh

Department of Compouter Science

Columbia University

nieh@cs.columbia.edu

Abstract

As ARM CPUs become increasingly common in mobile devices

and servers, there is a growing demand for providing the ben-

efits of virtualization for ARM-based devices. We present our

experiences building the Linux ARM hypervisor, KVM/ARM,

the first full system ARM virtualization solution that can run

unmodified guest operating systems on ARM multicore hard-

ware. KVM/ARM introduces split-mode virtualization, allowing

a hypervisor to split its execution across CPU modes and be inte-

grated into the Linux kernel. This allows KVM/ARM to leverage

existing Linux hardware support and functionality to simplify hy-

pervisor development and maintainability while utilizing recent

ARM hardware virtualization extensions to run virtual machines

with comparable performance to native execution. KVM/ARM

has been successfully merged into the mainline Linux kernel,

ensuring that it will gain wide adoption as the virtualization

platform of choice for ARM. We provide the first measurements

on real hardware of a complete hypervisor using ARM hardware

virtualization support. Our results demonstrate that KVM/ARM

has modest virtualization performance and power costs, and

can achieve lower performance and power costs compared to

x86-based Linux virtualization on multicore hardware.

Categories and Subject Descriptors

C.0 [Computer Systems

Organization]: General–Hardware/software interface, System

architectures; D.4.7 [Operating Systems]: Organization and

Design; D.4.8 [Operating Systems]: Performance

Keywords

Virtualization, Hypervisors, Operating Systems,

Multicore, ARM, Linux

Permission to make digital or hard copies of all or part of this work for personal or

classroom use is granted without fee provided that copies are not made or distributed for

profit or commercial advantage and that copies bear this notice and the full citation on

the first page. Copyrights for components of this work owned by others than the author(s)

must be honored. Abstracting with credit is permitted. To copy otherwise, or republish,

to post on servers or to redistribute to lists, requires prior specific permission and/or a fee.

Request permissions from permissions@acm.org.

ASPLOS ’14, March 1–4, 2014, Salt Lake City, Utah, USA.

ACM 978-1-4503-2305-5/14/03.. . $15.00.

http://dx.doi.org/10.1145/2541940.2541946

1. Introduction

ARM-based devices are seeing tremendous growth across smart-

phones, netbooks, and embedded computers. While ARM CPUs

have benefited from their advantages in power efficiency in these

markets, ARM CPUs also continue to increase in performance

such that they are now within the range of x86 CPUs for many

classes of applications. This is spurring the development of new

ARM-based microservers and an upward push of ARM CPUs

into traditional server, PC, and network systems.

To help bring the benefits of virtualization to ARM-based

devices, ARM CPUs now include hardware support for virtual-

ization. Although virtualization is commonly used on x86-based

systems, there are key differences between ARM and x86 vir-

tualization. First, ARM and x86 virtualization extensions have

important differences such that x86 hypervisor designs are not

directly amenable to ARM. These differences impact hypervisor

performance and design, especially for multicore systems, but

have not been evaluated with real hardware. Second, unlike

x86-based systems, there is no PC-standard hardware equivalent

for ARM. The ARM market is fragmented with many different

vertically integrated ARM platforms with non-standard hardware.

Virtualizing ARM in a manner that works across the diversity

of ARM hardware in the absence of any real hardware standard

is a key challenge.

We describe our experiences building KVM/ARM [

the ARM hypervisor in the mainline Linux kernel. KVM/ARM

is the first hypervisor to leverage ARM hardware virtualization

support to run unmodified guest operating systems (OSes) on

ARM multicore hardware. Our work makes four main contri-

butions. First, we introduce split-mode virtualization, a new

approach to hypervisor design that splits the core hypervisor

so that it runs across different privileged CPU modes to take

advantage of the specific benefits and functionality offered by

each CPU mode. This approach provides key benefits in the

context of ARM virtualization. ARM introduced a new CPU

mode for running hypervisors called Hyp mode, but Hyp mode

has its own set of features distinct from existing kernel modes.

Hyp mode targets running a standalone hypervisor underneath

the OS kernel, and was not designed to work well with a hosted

hypervisor design, where the hypervisor is integrated with a host

333

kernel. For example, standard OS mechanisms in Linux would

have to be significantly redesigned to run in Hyp mode. Split-

mode virtualization makes it possible to take advantage of the

benefits of a hosted hypervisor design by running the hypervisor

in normal privileged CPU modes to leverage existing OS mech-

anisms without modification while at the same time still using

Hyp mode to leverage ARM hardware virtualization features.

Second, we designed and implemented KVM/ARM from the

ground up as an open source project that would be easy to main-

tain and integrate into the Linux kernel. This is especially impor-

tant in the context of ARM systems which lack standard ways to

integrate hardware components, features for hardware discovery

such as a standard BIOS or PCI bus, and standard mechanisms

for installing low-level software. A standalone bare metal hy-

pervisor would need to be ported to each and every supported

hardware platform, a huge maintenance and development burden.

Linux, however, is supported across almost all ARM platforms

and by integrating KVM/ARM with Linux, KVM/ARM is auto-

matically available on any device running a recent version of the

Linux kernel. By using split-mode virtualization, we can leverage

the existing Kernel-based Virtual Machine (KVM) [

] hypervi-

sor interface in Linux and can reuse substantial pieces of existing

kernel code and interfaces to reduce implementation complexity.

KVM/ARM requires adding less than 6,000 lines of ARM code

to Linux, a much smaller code base to maintain than standalone

hypervisors. KVM/ARM was accepted as the ARM hypervisor

of the mainline Linux kernel as of the Linux 3.9 kernel, ensuring

its wide adoption and use given the dominance of Linux on ARM

platforms. Based on our open source experiences, we offer some

useful hints on transferring research ideas into implementations

likely to be adopted by the open source community.

Third, we demonstrate the effectiveness of KVM/ARM on

real multicore ARM hardware. Our results are the first measure-

ments of a hypervisor using ARM virtualization support on real

hardware. We compare against the standard widely-used Linux

KVM x86 hypervisor and evaluate its performance overhead

for running application workloads in virtual machines (VMs)

versus native non-virtualized execution. Our results show that

KVM/ARM achieves comparable performance overhead in most

cases, and significantly lower performance overhead for two

important applications, Apache and MySQL, on multicore plat-

forms. These results provide the first comparison of ARM and

x86 virtualization extensions on real hardware to quantitatively

demonstrate how the different design choices affect virtualization

performance. We show that KVM/ARM also provides power

efficiency benefits over Linux KVM x86.

Finally, we make several recommendations regarding future

hardware support for virtualization based on our experiences

building and evaluating a complete ARM hypervisor. We identify

features that are important and helpful to reduce the software

complexity of hypervisor implementations, and discuss mech-

anisms useful to maximize hypervisor performance, especially

in the context of multicore systems.

This paper describes the design and implementation of

KVM/ARM. Section 2 presents an overview of the ARM vir-

tualization extensions and a comparison with x86. Section 3

describes the design of the KVM/ARM hypervisor. Section 4

discusses the implementation of KVM/ARM and our experi-

ences releasing it to the Linux community and having it adopted

into the mainline Linux kernel. Section 5 presents experimental

results quantifying the performance and energy efficiency of

KVM/ARM, as well as a quantitative comparison of real ARM

and x86 virtualization hardware. Section 6 makes recommen-

dations for designing future hardware virtualization support.

Section 7 discusses related work. Finally, we present some

concluding remarks.

2. ARM Virtualization Extensions

Because the ARM architecture is not classically virtualiz-

able [

], ARM introduced hardware virtualization support

as an optional extension in the latest ARMv7 [

] and ARMv8 [

]

architectures. For example, the Cortex-A15 [

] is a current

ARMv7 CPU with hardware virtualization support. We present

a brief overview of the ARM virtualization extensions.

CPU Virtualization

Figure 1 shows the CPU modes on the

ARMv7 architecture, including TrustZone (Security Extensions)

and a new CPU mode called Hyp mode. TrustZone splits the

modes into two worlds, secure and non-secure, which are or-

thogonal to the CPU modes. A special mode, monitor mode, is

provided to switch between the secure and non-secure worlds. Al-

though ARM CPUs always power up starting in the secure world,

ARM bootloaders typically transition to the non-secure world at

an early stage. The secure world is only used for specialized use

cases such as digital rights management. TrustZone may appear

useful for virtualization by using the secure world for hypervisor

execution, but this does not work because trap-and-emulate is not

supported. There is no means to trap operations executed in the

non-secure world to the secure world. Non-secure software can

therefore freely configure, for example, virtual memory. Any soft-

ware running at the highest non-secure privilege level therefore

has access to all non-secure physical memory, making it impos-

sible to isolate multiple VMs running in the non-secure world.

Hyp mode was introduced as a trap-and-emulate mechanism

to support virtualization in the non-secure world. Hyp mode is a

Non-Secure state

PL0 User

PL1 Kernel

PL2 Hyp

Monitor Mode (Secure PL1)

Secure state

PL0 User

PL1 Kernel

Figure 1: ARMv7 Processor Modes

334

CPU mode that is strictly more privileged than other CPU modes,

user and kernel modes. Software running in Hyp mode can con-

figure the hardware to trap from kernel mode into Hyp mode

on various sensitive instructions and hardware interrupts. To run

VMs, the hypervisor must at least partially reside in Hyp mode.

The VM will execute normally in user and kernel mode until

some condition is reached that requires intervention of the hyper-

visor. At this point, the hardware traps into Hyp mode giving con-

trol to the hypervisor, which can then manage the hardware and

provide the required isolation across VMs. Once the condition

is processed by the hypervisor, the CPU can be switched back

into user or kernel mode and the VM can continue executing.

The ARM architecture allows each trap to be configured to

trap directly into a VM’s kernel mode instead of going through

Hyp mode. For example, traps caused by system calls or page

faults from user mode can be configured to trap to a VM’s kernel

mode directly so that they are handled by the guest OS without

intervention of the hypervisor. This avoids going to Hyp mode on

each system call or page fault, reducing virtualization overhead.

Additionally, all traps into Hyp mode can be disabled and a

single non-virtualized kernel can run in kernel mode and have

complete control of the system.

ARM designed the virtualization support around a separate

CPU mode distinct from existing kernel mode, because they envi-

sioned a standalone hypervisor underneath a more complex rich

OS kernel [

]. They wanted to make it simpler for hypervisor

developers to implement the hypervisors, and therefore reduced

the number of control registers available in Hyp mode compared

to kernel mode. Similarly, they mandated certain bits to be set in

the page table entries, because they did not envision a hypervisor

sharing page tables with software running in user space, which

is for example what the Linux kernel does with kernel mode.

Memory Virtualization

ARM also provides hardware support

to virtualize physical memory. When running a VM, the physical

addresses managed by the VM are actually Intermediate Physical

Addresses (IPAs), also known as guest physical addresses, and

need to be translated into physical addresses (PAs), also known

as host physical addresses. Similarly to nested page tables on

x86, ARM provides a second set of page tables, Stage-2 page

tables, which translate from IPAs to PAs corresponding to guest

and host physical addresses, respectively. Stage-2 translation can

be completely disabled and enabled from Hyp mode. Stage-2

page tables use ARM’s new LPAE page table format, with subtle

differences from the page tables used by kernel mode.

Interrupt Virtualization

ARM defines the Generic Interrupt

Controller (GIC) architecture [

]. The GIC routes interrupts from

devices to CPUs and CPUs query the GIC to discover the source

of an interrupt. The GIC is especially important in multicore

configurations, because it is used to generate Inter-Processor

Interrupts (IPIs) from one CPU core to another. The GIC is split

in two parts, the distributor and the CPU interfaces. There is

only one distributor in a system, but each CPU core has a GIC

CPU interface. Both the CPU interfaces and the distributor are

accessed over a Memory-Mapped interface (MMIO). The dis-

tributor is used to configure the GIC, for example, to configure

the CPU core affinity of an interrupt, to completely enable or

disable interrupts on a system, or to send an IPI to another CPU

core. The CPU interface is used to acknowledge (ACK) and to

signal End-Of-Interrupt (EOI). For example, when a CPU core

receives an interrupt, it will read a special register on the GIC

CPU interface, which ACKs the interrupt and returns the number

of the interrupt. The interrupt will not be raised to the CPU again

before the CPU writes to the EOI register of the CPU interface

with the value retrieved from the ACK register.

Interrupts can be configured to trap to either Hyp or kernel

mode. Trapping all interrupts to kernel mode and letting OS

software running in kernel mode handle them directly is efficient,

but does not work in the context of VMs, because the hypervisor

loses control over the hardware. Trapping all interrupts to Hyp

mode ensures that the hypervisor retains control, but requires

emulating virtual interrupts in software to signal events to VMs.

This is cumbersome to manage and expensive because each step

of interrupt and virtual interrupt processing, such as ACKing and

EOIing, must go through the hypervisor.

The GIC v2.0 includes hardware virtualization support in the

form of a virtual GIC (VGIC) so that receiving virtual interrupts

does not need to be emulated in software by the hypervisor. The

VGIC introduces a VGIC CPU interface for each CPU and a

corresponding hypervisor control interface for each CPU. VMs

are configured to see the VGIC CPU interface instead of the

GIC CPU interface. Virtual interrupts are generated by writing to

special registers, the list registers, in the VGIC hypervisor control

interface, and the VGIC CPU interface raises the virtual inter-

rupts directly to a VM’s kernel mode. Because the VGIC CPU

interface includes support for ACK and EOI, these operations no

longer need to trap to the hypervisor to be emulated in software,

reducing overhead for receiving interrupts on a CPU. For ex-

ample, emulated virtual devices typically raise virtual interrupts

through a software API to the hypervisor, which can leverage the

VGIC by writing the virtual interrupt number for the emulated

device into the list registers. This causes the VGIC to interrupt the

VM directly to kernel mode and lets the guest OS ACK and EOI

the virtual interrupt without trapping to the hypervisor. Note that

the distributor must still be emulated in software and all accesses

to the distributor by a VM must still trap to the hypervisor. For

example, when a virtual CPU sends a virtual IPI to another vir-

tual CPU, this will cause a trap to the hypervisor, which emulates

the distributor access in software and programs the list registers

on the receiving CPU’s GIC hypervisor control interface.

Timer Virtualization

ARM defines the Generic Timer Archi-

tecture which includes support for timer virtualization. Generic

timers provide a counter that measures passing of time in real-

time, and a timer for each CPU, which is programmed to raise

an interrupt to the CPU after a certain amount of time has passed.

Timers are likely to be used by both hypervisors and guest OSes,

but to provide isolation and retain control, the timers used by

the hypervisor cannot be directly configured and manipulated by

guest OSes. Such timer accesses from a guest OS would need to

335

剩余14页未读，继续阅读

评论收藏

内容反馈

whut0000

粉丝: 2
资源: 22

KVM/ARM: The Design and Implementation of the Linux ARM Hypervis...

最新资源

KVM/ARM: The Design and Implementation of the Linux ARM Hypervis...

ARM_Hypervisor

Linux KVM on ARM64

kvm： the Linux Virtual Machine Monitor

嵌入式系统/ARM技术中的风河新版Hypervisor助虚拟化开发更上一层楼

嵌入式系统/ARM技术中的风河嵌入式Hypervisor拓展虚拟化支持

Design and Implementation of Linux Access Control Model.pdf

TCP IP Architecture, Design and Implementation in Linux

TCPIP Architecture, Design and Implementation in Linux 无水印pdf

TCP-IP Architecture Design and Implementation in Linux(2008).part7

查看opensuse硬件信息

ARM Virtualization: Performance and Architectural Implication

kvm-vmi:基于KVM的虚拟机自省

虚拟化KVM hypervisor

TCP-IP Architecture Design and Implementation in Linux(2008).part6

TCP-IP Architecture Design and Implementation in Linux(2008).part4

TCP-IP Architecture Design and Implementation in Linux(2008).part5

TCP-IP Architecture Design and Implementation in Linux(2008).part2

TCP-IP Architecture Design and Implementation in Linux(2008).part3

Cloud.Computing.Design.Patterns.9332557306.epub

嵌入式系统/ARM技术中的迁移Xen虚拟机到KVM的六大步骤

Mastering.KVM.Virtualization.1784399051

Mastering KVM Virtualization(PACKT,2016)

The Xen Hypervisor and its IO Subsystem

TCP-IP Architecture Design and Implementation in Linux(2008).part1

TCPIP Architecture Design and Implementation in Linux.djvu

TCPIP_Architecture_Design_And_Implementation_In_Linux.pdf

最新资源