基于kubeadm的kubernetes高可用集群部署

# kubeadm-highavailiability (English / 中文) - kubernetes high availiability deployment based on kubeadm, for Kubernetes version v1.11.x/v1.9.x/v1.7.x/v1.6.x  - [中文文档(for v1.11.x版本)](README_CN.md) - [English document(for v1.11.x version)](README.md) - [中文文档(for v1.9.x版本)](v1.9/README_CN.md) - [English document(for v1.9.x version)](v1.9/README.md) - [中文文档(for v1.7.x版本)](v1.7/README_CN.md) - [English document(for v1.7.x version)](v1.7/README.md) - [中文文档(for v1.6.x版本)](v1.6/README_CN.md) - [English document(for v1.6.x version)](v1.6/README.md) --- - [GitHub project URL](https://github.com/cookeem/kubeadm-ha/) - [OSChina project URL](https://git.oschina.net/cookeem/kubeadm-ha/) --- - This operation instruction is for version v1.11.x kubernetes cluster > v1.11.x version now support deploy tls etcd cluster in control plane ### category 1. [deployment architecture](#deployment-architecture) 1. [deployment architecture summary](#deployment-architecture-summary) 1. [detail deployment architecture](#detail-deployment-architecture) 1. [hosts list](#hosts-list) 1. [prerequisites](#prerequisites) 1. [version info](#version-info) 1. [required docker images](#required-docker-images) 1. [system configuration](#system-configuration) 1. [kubernetes installation](#kubernetes-installation) 1. [firewalld and iptables settings](#firewalld-and-iptables-settings) 1. [kubernetes and related services installation](#kubernetes-and-related-services-installation) 1. [master hosts mutual trust](#master-hosts-mutual-trust) 1. [masters high availiability installation](#masters-high-availiability-installation) 1. [create configuration files](#create-configuration-files) 1. [kubeadm initialization](#kubeadm-initialization) 1. [high availiability configuration](#high-availiability-configuration) 1. [masters load balance settings](#masters-load-balance-settings) 1. [keepalived installation](#keepalived-installation) 1. [nginx load balance settings](#nginx-load-balance-settings) 1. [kube-proxy HA settings](#kube-proxy-ha-settings) 1. [high availiability verify](#high-availiability-verify) 1. [kubernetes addons installation](#kubernetes-addons-installation) 1. [workers join kubernetes cluster](#workers-join-kubernetes-cluster) 1. [workers join HA cluster](#workers-join-ha-cluster) 1. [verify kubernetes cluster installation](#verify-kubernetes-cluster-installation) 1. [verify kubernetes cluster high availiablity installation](#verify-kubernetes-cluster-high-availiablity-installation) 1. [cluster upgrade](#cluster-upgrade) 1. [kubernetes cluster upgrade from v1.11.1 to v1.11.5](#kubernetes-cluster-upgrade-from-v1-11-1-to-v1-11-5) ### deployment architecture #### deployment architecture summary  --- [category](#category) #### detail deployment architecture  - kubernetes components: > kube-apiserver: exposes the Kubernetes API. It is the front-end for the Kubernetes control plane. It is designed to scale horizontally – that is, it scales by deploying more instances. > etcd: is used as Kubernetes’ backing store. All cluster data is stored here. Always have a backup plan for etcd’s data for your Kubernetes cluster. > kube-scheduler: watches newly created pods that have no node assigned, and selects a node for them to run on. > kube-controller-manager: runs controllers, which are the background threads that handle routine tasks in the cluster. Logically, each controller is a separate process, but to reduce complexity, they are all compiled into a single binary and run in a single process. > kubelet: is the primary node agent. It watches for pods that have been assigned to its node (either by apiserver or via local configuration file) > kube-proxy: enables the Kubernetes service abstraction by maintaining network rules on the host and performing connection forwarding. - load balancer > keepalived cluster config a virtual IP address (192.168.20.10), this virtual IP address point to k8s-master01, k8s-master02, k8s-master03. > nginx service as the load balancer of k8s-master01, k8s-master02, k8s-master03's apiserver. The other nodes kubernetes services connect the keepalived virtual ip address (192.168.20.10) and nginx exposed port (16443) to communicate with the master cluster's apiservers. --- [category](#category) #### hosts list HostName | IPAddress | Notes | Components :--- | :--- | :--- | :--- k8s-master01 ~ 03 | 192.168.20.20 ~ 22 | master nodes * 3 | keepalived, nginx, etcd, kubelet, kube-apiserver k8s-master-lb | 192.168.20.10 | keepalived virtual IP | N/A k8s-node01 ~ 08 | 192.168.20.30 ~ 37 | worker nodes * 8 | kubelet --- [category](#category) ### prerequisites #### version info - Linux version: CentOS 7.4.1708 - Core version: 4.6.4-1.el7.elrepo.x86_64 ```sh $ cat /etc/redhat-release CentOS Linux release 7.4.1708 (Core) $ uname -r 4.6.4-1.el7.elrepo.x86_64 ``` - docker version: 17.12.0-ce-rc2 ```sh $ docker version Client: Version: 17.12.0-ce-rc2 API version: 1.35 Go version: go1.9.2 Git commit: f9cde63 Built: Tue Dec 12 06:42:20 2017 OS/Arch: linux/amd64 Server: Engine: Version: 17.12.0-ce-rc2 API version: 1.35 (minimum version 1.12) Go version: go1.9.2 Git commit: f9cde63 Built: Tue Dec 12 06:44:50 2017 OS/Arch: linux/amd64 Experimental: false ``` - kubeadm version: v1.11.1 ```sh $ kubeadm version kubeadm version: &version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.1", GitCommit:"b1b29978270dc22fecc592ac55d903350454310a", GitTreeState:"clean", BuildDate:"2018-07-17T18:50:16Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"linux/amd64"} ``` - kubelet version: v1.11.1 ```sh $ kubelet --version Kubernetes v1.11.1 ``` - networks addons > calico --- [category](#category) #### required docker images - required docker images and tags ```sh # kuberentes basic components # use kubeadm to list all required docker images $ kubeadm config images list --kubernetes-version=v1.11.1 k8s.gcr.io/kube-apiserver-amd64:v1.11.1 k8s.gcr.io/kube-controller-manager-amd64:v1.11.1 k8s.gcr.io/kube-scheduler-amd64:v1.11.1 k8s.gcr.io/kube-proxy-amd64:v1.11.1 k8s.gcr.io/pause:3.1 k8s.gcr.io/etcd-amd64:3.2.18 k8s.gcr.io/coredns:1.1.3 # use kubeadm to pull all required docker images $ kubeadm config images pull --kubernetes-version=v1.11.1 # kubernetes networks addons $ docker pull quay.io/calico/typha:v0.7.4 $ docker pull quay.io/calico/node:v3.1.3 $ docker pull quay.io/calico/cni:v3.1.3 # kubernetes metrics server $ docker pull gcr.io/google_containers/metrics-server-amd64:v0.2.1 # kubernetes dashboard $ docker pull gcr.io/google_containers/kubernetes-dashboard-amd64:v1.8.3 # kubernetes heapster $ docker pull k8s.gcr.io/heapster-amd64:v1.5.4 $ docker pull k8s.gcr.io/heapster-influxdb-amd64:v1.5.2 $ docker pull k8s.gcr.io/heapster-grafana-amd64:v5.0.4 # kubernetes apiserver load balancer $ docker pull nginx:latest # prometheus $ docker pull prom/prometheus:v2.3.1 # traefik $ docker pull traefik:v1.6.3 # istio $ docker pull docker.io/jaegertracing/all-in-one:1.5 $ docker pull docker.io/prom/prometheus:v2.3.1 $ docker pull docker.io/prom/statsd-exporter:v0.6.0 $ docker pull gcr.io/istio-release/citadel:1.0.0 $ docker pull gcr.io/istio-release/galley:1.0.0 $ docker pull gcr.io/istio-release/grafana:1.0.0 $ docker pull gcr.io/istio-release/mixer:1.0.0 $ docker pull gcr.io/istio-release/pilot:1.0.0 $ docker pull gcr.io/istio-release/proxy_init:1.0.0 $ docker pull gcr.io/istio-release/proxyv2:1.0.0 $ docker pull gcr.io/istio-release/servicegraph:1.0.0 $ docker pull gcr.io/istio-release/sidecar_injector:1.0.0 $ docker pull quay.io/coreos/hyperkube:v1.7.6_coreos.0 ``` --- [category](#category) #### system configuration - on all kubernetes nodes: add kubernetes' repository ```sh $ cat <<EOF > /etc/yum.repos.d/