2322 IEEE TRANSACTIONS ON MULTIMEDIA, VOL. 19, NO. 10, OCTOBER 2017
Interactive Screen Video Streaming-Based
Pervasive Mobile Workstyle
Zhan Ma, Member, IEEE, Tao Yue, Xun Cao, Member, IEEE, Yiling Xu, Xin Li, and Yongjin Wang
Abstract—In this paper, we develop an interactive screen
video streaming-based system to enable the ubiquitous mobile
workstyle, which is referred to as personal computer to pervasive
computing (PC2PC). The desktop screens of virtualized systems
are compressed in the PC2PC servers and delivered to remote
end users for stream decoding, rendering, and interactions. We
have implemented a system from the scratch, where the emerging
screen content coding extension of high-efficiency video coding
is implemented to compress and stream the desktop screens of
the virtualized system in real time. Three core asset channels,
system, display, and inputs, are defined to enable systematic end-
to-end communication. Compared with Red Hat SPICE virtual
desktop infrastructure s cheme, the proposed PC2PC could save
network bandwidth consumption by a factor of 2, 7, and 4,
respectively, in terms of typical video streaming, web browsing, and
stationary office applications at the same visual quality. Meanwhile,
we have also measured the delays of the system and presented
preliminary results on the user experience aspect. A simple network
estimation is applied to optimize the quality bandwidth adaptation
for both single user and multiuser scenarios to consider the network
dynamics.
Index Terms—High-efficiency video coding, pervasive
computing, screen content coding, virtual desktop infrastructure.
I. INTRODUCTION
M
OBILE workstyle is an emerging office fashion that al-
lows employees to manage and operate their daily tasks
remotely. Such mobile workstyle is supported by the virtual
desktop infrastructure (VDI) where routine desktop operating
systems (OSs) are hosted at data centers (i.e., private or pub-
lic cloud) to provide the ubiquitous computing and storage. It
Manuscript received January 14, 2017; revised June 21, 2017 and August 3,
2017; accepted August 3, 2017. Date of publication August 9, 2017; date of
current version September 15, 2017. This work was supported in part by the
National Natural Science Foundation of China (NSFC) Projects under Grant
61371166 and Grant 61422107, in part by the NSFC Projects under Grant
61571215, Grant 61650101, Grant 61322112, and Grant 61531166004, in part
by the National Science Foundation for Young Scholar of Jiangsu Province,
China, under Grant BK20140610, and in part by the Natural Science Foundation
of Jiangsu Province under Grant BE2016186. The guest editor coordinating the
review of this manuscript and approving it for publication was Dr. Shiwen Mao.
(Corresponding author: Yiling Xu.)
Z. Ma, T. Yue, and X. Cao are with Nanjing University, Nanjing 210008,
China (e-mail: mazhan@nju.edu.cn; yuetao@nju.edu.cn; caoxun@nju.edu.cn).
Y. Xu is with Shanghai Jiaotong University, Shanghai 200240, China
(e-mail: yl.xu@sjtu.edu.cn).
X. Li is with Yun Ge Zhi Li, Inc., Richardson, TX 75082 USA (e-mail:
lixin@gwecom.com).
Y. Wang is with the Nanjing University of Posts and Telecommunications,
Nanjing 210028, China (e-mail: wangyj@njupt.edu.cn).
Color versions of one or more of the figrues in this paper are available online
at http//ieeexplore.ieee.org.
Digital Object Identifier 10.1109/TMM.2017.2737944
enables people to strike a better work-life balance and work
from anywhere, unrestricted by location, time or device [1], [2].
Meanwhile, it also provides better security by concealing sen-
sitive and confidential information processed and stored in data
centers.
Industry leaders in the area of information technology (IT)
actively embrace the mobile workstyles and have developed sev-
eral well-known platforms to enable such service, for instance,
VMware Horizon [3], Citrix XenDesktop [4], Microsoft Re-
mote Desktop Protocol (RDP) [5], Red Hat Simple Protocol for
Independent Computing Environments (SPICE) [6], etc. These
popular VDI technologies follow the similar procedure: an in-
stantaneous desktop screen is first analyzed to identify different
regions, such as text, icon, graphics, images, etc, and each region
will be processed individually using compression methods for
network delivery. They require complex protocols to maintain
the communication between the client and remote server. Be-
cause of the noticeable involvements from the client for desktop
rendering, it is also referred to as the “client rendering” solution.
Due to the advances in communication (such as 4G/LTE,
emerging 5G) and mobile computing capability, we envision the
increasing market demand of the mobile workstyle through the
VDI technologies. However, most existing technical solutions
are not well suited due to high complexity of “client rendering”
protocols. In this work, we propose a complete “server ren-
dering” solution where the server renders and compresses the
instantaneous desktop screen, and the client only decodes the
compressed screen stream and sends back mouse and keyboard
commands for user interaction. This is motivated by the fact
that instantaneous desktop screen rendering is actually a video
playback application refreshed at 60 frames per second (FPS)
1
for a typical display.
Note that offering the pervasive mobile workstyle requires
substantial effort in various aspects, such as OS virtualization,
cloud load balancing, VDI, etc. In this work we focus on VDI
that is applied to enable the interaction between hosted OSs
and remote clients. The overall PC2PC system is illustrated
in Fig. 1. A real-time HEVC based SCC compression engine
encodes instantaneous desktop screens of a virtualized OS at
the server side. This is referred to as the Display Channel to
deliver the compressed streams. To enable the systematic end-
to-end communication, we have also defined another two asset
channels, i.e., the System Channel to convey the messages so
as to initiate, maintain/monitor and destroy the communication
1
Different LCD/LED displays may have different refresh rate in practice.
1520-9210 © 2017 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.
See http://www.ieee.org/publications
standards/publications/rights/index.html for more information.