Abstract
Nowadays, the cloud platform is based on the traditional virtual machine technology (VM) to
achieve the hardware resources management and flexible scalability.There is a greater resource
overhead on the speed of starting and stopping, resource utilization, operational monitoring and
performance .The big data computing framework deployed in the cloud platform is a typical
application scenario.With the rapid growth of the amount of data,the traditional cloud platform
architecture and processing methods can not effectively adapt to the big data processing
environment.
With the advent of lightweight container technology, Docker Container provides developers
with the ability to rapidly build, deploy and migrate distributed applications ,and greatly simplifies
the deployment process and reduces the server costs. Kubernetes is an open source system for
automating the deployment and management of large-scale Docker container applications.It
provides the scheduling of resources, automatic deployment, service discovery, and flexible scaling
for the containerized applications, and it offers support for the big data distributed computing
frameworks Map-Reduce.Of course, Docker is deficiency in security, storage and other aspects, and
it is still in the stage of rapid development for the cloud platform.
This paper focuses on the realization of the deployment of Spark distributed computing
framework based on Docker containers, with the virtualization container Docker as the lowwer
bearing platform and the Kubernetes as a container management and scheduling system.The
containerized big data platform can greatly improve resource utilization and computational
parallelism, simplify operation and maintenance management costs, and be able to automatically
scales the Spark computing nodes according to the real-time load. For the deployment of Spark
clusters based on Kubernetes, the main research of this paper is as follows:
(1)To realize the communication between the Docker container of different host. Docker itself
does not have the communication capabilities between the host computer. The use of flannel build
an overlay network to achieve the communication capabilities between the different host computer
container.
(2)To design and implementation of the Spark cluster based on the Kubernetes system. This
paper analyzes the communication mechanism of Spark cluster, constructs Spark image using
dockerfile, designs and implements Spark cluster based on kubernetes system, which can rapidly
评论0