HotSnap：虚拟机群集的热分布式快照系统资源-CSDN文库

研究论文

142 浏览量 2021-03-10 11:02:39 上传评论收藏 698KB PDF 举报

资源推荐

资源详情

资源评论

HotSnap: A Hot Distributed Snapshot System for Virtual Machine Cluster

Lei Cui, Bo Li, Yangyang Zhang, Jianxin Li

Beihang University, Beijing, China

{cuilei, libo, zhangyy, lijx}@act.buaa.edu.cn

Abstract

The management of virtual machine cluster (VMC) is

challenging owing to the reliability requirements, such

as non-stop service, failure tolerance, etc. Distributed s-

napshot of VMC is one promising approach to support

system reliability, it allows the system administrators of

data centers to recover the system from failure, and re-

sume the execution from a intermediate state rather than

the initial state. However, due to the heavyweight na-

ture of virtual machine (VM) technology, application-

s running in the VMC suffer from long downtime and

performance degradation during snapshot. Besides, the

discrepancy of snapshot completion times among VMs

brings the TCP backoff problem, resulting in network in-

terruption between two communicating VMs. This paper

proposes HotSnap, a VMC snapshot approach designed

to enable taking hot distributed snapshot with millisec-

onds system downtime and TCP backoff duration. At the

core of HotSnap is transient snapshot that saves the min-

imum instantaneous state in a short time, and full snap-

shot which saves the entire VM state during normal oper-

ation. We then design the snapshot protocol to coordinate

the individual VM snapshots into the global consistent

state of VMC. We have implemented HotSnap on QE-

MU/KVM, and conduct several experiments to show the

effectiveness and efﬁciency. Compared to the live migra-

tion based distributed snapshot technique which brings

seconds of system downtime and network interruption,

HotSnap only incurs tens of milliseconds.

1 Introduction

With the increasing prevalence of cloud computing and

IaaS paradigm, more and more distributed application-

s and systems are migrating to and running on virtual-

ization platform. In virtualized environments, distribut-

ed applications are encapsulated into virtual machines,

which are connected into virtual machine cluster (VM-

C) and coordinated to complete the heavy tasks. For

example, Amazon EC2 [1] offers load balancing web

farm which can dynamically add or remove virtual ma-

chine (VM) nodes to maximize resource utilization; Cy-

berGuarder [22] encapsulates security services such as

IDS and ﬁrewalls into VMs, and deploys them over a

virtual network to provide virtual network security ser-

vice; Emulab [12] leverages VMC to implement on-

demand virtual environments for developing and testing

networked applications; the parallel applications, such as

map-reduce jobs, scientiﬁc computing, client-server sys-

tems can also run on the virtual machine cluster which

provides an isolated, scaled and closed running environ-

ment.

Distributed snapshot [13, 27, 19] is a critical technique

to improve system reliability for distributed applications

and systems. It saves the running state of the application-

s periodically during the failure-free execution. Upon a

failure, the system can resume the computation from a

recorded intermediate state rather than the initial state,

thereby reducing the amount of lost computation [15]. It

provides the system administrators the ability to recover

the system from failure owing to hardware errors, soft-

ware errors or other reasons.

Since the snapshot process is always carried out peri-

odically during normal execution, transparency is a key

feature when taking distributed snapshot. In other word-

s, the users or applications should be unaware of the

snapshot process, neither the snapshot implementation

scheme nor the performance impact. However, the tra-

ditional distributed systems either implement snapshot

in OS kernel [11], or modify the MPI library to sup-

port snapshot function [17, 24]. Besides, many systems

even leave the job to developers to implement snapshot

on the application level [3, 25]. These technologies re-

quire modiﬁcation of OS code or recompilation of appli-

cations, thus violating the transparency from the view of

implementation schema.

The distributed snapshot of VMC seems to be an ef-

fective way to mitigate the transparency problem, since it

implements snapshot on virtual machine manager (VM-

M) layer which encapsulates the application’s running s-

tate and resources without modiﬁcation to target applica-

tions or the OS. Many systems such as VNSnap [18] and

Emulab [12] have been proposed to create the distributed

snapshot for a closed network of VMs. However, these

methods still have obvious shortcomings.

First, the snapshot should be non-disruptive to the up-

per applications, however the state-of-the-art VM snap-

shot technologies, either adopt stop-and-copy method

(e.g., Xen and KVM) which causes the service are com-

pletely unavailable, or leverage live migration based

schema which also causes long and unpredictable down-

time owing to the ﬁnal copy of dirty pages [26].

Second, the distributed snapshot should coordinate the

individual snapshots of VMs to maintain a global consis-

tent state. The global consistent state reﬂects the snap-

shot state in one virtual time epoch and regards causali-

ty, implying the VM before snapshot cannot receive the

packets send from the VM that has ﬁnished the snapshot

to keep the consistent state during distributed snapshot

(further explanations about global consistent state can be

referred in appendix A). However, due to the various VM

memory size, variety of workloads and parallel I/O oper-

ations to save the state, the snapshot start time, duration

time and completion time of different VMs are always

different, resulting in the TCP back-off issue [18], there-

by causing network interruption between the communi-

cating VMs. Figure 1 demonstrates one such case hap-

pened in TCP’s three-way handshake. Worse still, for

the master/slave style distributed applications, the mas-

ter always undertake heavier workloads so that cost more

time to ﬁnish the snapshot than the slaves, therefore, the

slaves which ﬁnish the snapshot ahead cannot commu-

nicate with the master until the master snapshot is over,

causing the whole system hung. As a result, the mas-

ter snapshot becomes the short-board during distributed

snapshot of master/slave systems.

Third, most distributed snapshot technologies adop-

t the coordinated snapshot protocol [13] to bring the

distributed applications into a consistent state. This re-

quires a coordinator to communicate snapshot-related

commands with other VMs during snapshot. In many

systems, the coordinator is setup in the customized mod-

ule such as VIOLIN switch in VNSnap [18] and XenBus

handler used in Emulab [12], thus lack of generality in

most virtualized environments.

To mitigate the problems above, we propose HotSnap,

a system capable of taking hot distributed snapshot that is

transparent to the upper applications. Once the snapshot

command is received, HotSnap ﬁrst suspends the VM,

freezes the memory state and disk state, creates a tran-

sient snapshot of VM, and then resumes the VM. The

SYN_RCVD

TIME_OUT

snapshot

SYN

SYN_RCVD

SYN/ACK

VM1

VM2

snapshot

TIME_OUT

TCP state

SYN

SYN/ACK

VM1

VM2

Figure 1: A TCP handshake case during distributed s-

napshot. V M

ﬁrst sends SYN to V M

to request a TCP

connection, at this moment V M

has not begin its snap-

shot; V M

receives this request, turn its own state into

SYN RCVD, and then sends SYN/ACK back to V M

We notice that now V M

has ﬁnished snapshot, and based

on the coordinated protocol, packets sent from VM

will

not be accepted by V M

until V M

has ﬁnished its own

snapshot. If VM

’s snapshot duration exceeds TCP time-

out, connection will fail.

transient snapshot only records the minimum instanta-

neous state, including CPU and device states, as well

as two bitmaps reserved for memory state and disk s-

tate, bringing only milliseconds of VM downtime, i.e.,

hot for upper applications. The full snapshot will be ac-

quired after resuming the VM, it saves the entire memory

state in a copy-on-write (COW) manner, and create the

disk snapshot in the redirect-on-write (ROW) schema;

the COW and ROW schemas enable creating the full s-

napshot without blocking the execution of VM, i.e., live

snapshot. Because the transient snapshot introduces on-

ly milliseconds of downtime, the discrepancy of down-

time among different VM snapshots will be minor, there-

by minimizing the TCP backoff duration.

HotSnap is completely implemented in VMM layer, it

requires no modiﬁcation to Guest OS or applications, and

can work without other additional modules. The major

contributions of the work are summarized as follows:

1) We propose a VM snapshot approach combined of

transient snapshot and full snapshot. The approach com-

pletes snapshot transiently, enables all VMs ﬁnish their

snapshots almost at the same time, which greatly reduces

the TCP backoff duration caused by the discrepancy of

VMs’ snapshot completion times.

2) A classic coordinated non-blocking protocol is sim-

pliﬁed and tailored to create the distributed snapshot of

the VMC in our virtualized environment.

3) We implement HotSnap on QEMU/KVM platform

[20]. Comprehensive experiments are conducted to eval-

uate the performance of HotSnap, and the results prove

the correctness and effectiveness of our system.

The rest of the paper is organized as follows. The

next section provides an analysis of the traditional dis-

b) VNSnap distributed snapshot.

a) VNSnap distributed snapshot.

pre-snapshot

live-snapshot downtime post-snapshot

VM1 VM2 VM1 VM2

b) HotSnap distributed snapshot.a) VNSnap distributed snapshot.

pre-snapshot

live-snapshot suspended post-snapshot

Arrow1

Arrow2

Arrow3

VM1 VM2 VM1 VM2

SNAPSHOT

Figure 2: Comparison of VNSnap and HotSnap.

tributed snapshot and their problems. Section 3 intro-

duces the HotSnap method, describes the transient snap-

shot, full snapshot and coordinated protocol. Section

4 describes the implementation-speciﬁc details on QE-

MU/KVM platform. The experimental results are shown

in Section 5. Finally we present the previous work re-

lated to HotSnap in section 6 and conclude our work in

Section 7.

2 An Analysis of Distributed Snapshot

The distributed snapshot includes independent VM s-

napshot and the coordinated protocol. Stop-and-copy

schema is a simple way to create snapshot of individ-

ual VM, but this schema introduces long downtime of

Guest OS and the upper applications running inside the

VM, thus is impractical in many scenarios that deliver

services to users. The live snapshot technologies lever-

age pre-copy based migration to achieve live snapshot

by iteratively saving the dirty pages to the snapshot ﬁle

[12, 18]. In this section, we will analyze the live mi-

gration based distributed snapshot proposed in VNSnap

[18], and explain how it results in TCP backoff problem.

Figure 2(a) demonstrates the procedure of VNSnap

distributed snapshot. Although VNSnap exploits the VI-

OLIN [12] switch to execute the coordinated protocol,

we treat V M

as the coordinator for clarity. Upon dis-

tributed snapshot, the coordinator, i.e., V M

, will send

SNAPSHOT command to VM

, and then create the snap-

shot of V M

itself. VNSnap leverages live migration to

iteratively save the dirtied pages into stable storage or re-

served memory region until some requirements are satis-

ﬁed, such as the amount of dirty pages are minor enough,

or the size cannot be further reduced even more iterations

are conducted. Then VNSnap suspends the VM, stores

the ﬁnal dirty memory pages, saves other devices’ state

and creates the disk snapshot. After these steps, the snap-

shot of VM

is over and V M

is resumed. Upon receiving

the SNAPSHOT command from VM

, V M

follows the

same procedure as V M

to create its own snapshot. VN-

Snap drops the packets send from the post-snapshot VM

to pre-snapshot VM, to keep the global state consistent.

Take this tiny cluster which consists of two VMs as

an example, the distributed snapshot duration time is

from the start time of V M

snapshot to the end time of

V M

snapshot (suppose V M

ﬁnishes snapshot later than

V M

), the TCP backoff duration is from the start of V M

suspend to the end of V M

suspend. The packets re-

sult in TCP backoff fall into three categories: 1) V M

is suspended while V M

is in live-snapshot, the packets

send from V M

to V M

will not arrive, as Arrow

illus-

trates; 2) V M

ﬁnishes snapshot and then turns into post-

snapshot state, but V M

is before or during snapshot. In

this situation, packets send from V M

will be dropped to

keep the consistent state of distributed snapshot. Arrow

shows such a case. 3) V M

is in post-snapshot, but V M

is suspended, V M

cannot receive the packets send from

V M

, as Arrow

shows.

Based on the three types of packets, we can conclude

that two aspects affect the TCP backoff duration in dis-

剩余14页未读，继续阅读

评论收藏

内容反馈

weixin_38682086

粉丝: 6
资源: 984

HotSnap：虚拟机群集的热分布式快照系统

ganeti_webmgr:Ganeti Web Manager是基于Django的Web应用程序，允许管理员和客户端访问其ganeti群集。 它包括一个权限和配额系统，管理员可以通过该系统授予对群集和虚拟机的访问权限。 它还包括用于组织对组织的访问的用户组

多智能体系统的自适应群集分布式优化（英文）.pdf

连通性保持下的多机器人系统分布式群集控制.pdf

VMWare虚拟机如何使用快照功能备份系统.docx

vSphere入门、使用和管理

VMware Infrastructure 3 入门

Vmware vSphere运维实录.胡嘉.pdf

RVTools4.2.2.msi

vmware vsphere 5安装配置图解

RVTools4.0.6.msi

vsan66安装与管理指南

vSphere基础

RVTools 4.5.1 2023最新版

RVTools 4.3.2

RVTools3.11.9.msi

基于CORDIC的反正弦和反余弦计算的FPGA实现

使用3DCNN和卷积LSTM进行手势识别学习时空特征

BA无标度网络中的SIR模型

基于三次贝塞尔曲线的类汽车曲率连续路径平滑

基于机器学习的设备剩余寿命预测方法综述

基于维纳过程的退化模型，具有递归过滤算法，可用于估计剩余使用寿命

基于FPGA的奇异值和特征值分解的快速实现。

磁悬浮系统自适应模糊PID控制器的设计

基于BP神经网络的人口预测

无人机协同目标的多无人机协同搜索方法

两轮平衡车的建模与控制研究

最新资源

ganeti_webmgr:Ganeti Web Manager是基于Django的Web应用程序，允许管理员和客户端访问其ganeti群集。它包括一个权限和配额系统，管理员可以通过该系统授予对群集和虚拟机的访问权限。它还包括用于组织对组织的访问的用户组