k8s入门教程三:多主集群部署

高可用架构

kubernetes高可用集群通常由3或5个节点组成高可用集群,需要保障各个节点的高可用性

  • etcd 内置集群机制,保障数据持久存储
  • kube-apiserver 无状态api服务,有负载均衡调度器做负载分发,如haproxy或nginx
  • kube-scheduler 内置选举机制,保障调度器高可用,确保同个时刻一个leader节点工作,其他处于阻塞,防止脑裂
  • kube-controller-manager 内置的选举机制保障控制器高可用,机制和kube-scheduler一致。

如下图所示:

从上图可以得出:有多个worker node,那计算节点就是高可用。而master节点上面,由于etcd、scheduler、controller-manager由k8s系统自身实现了高可用。所以需要高用可的是apiserver这个组件。以下是以keepalived+haproxy实现apiserver的高可用。

环境准备

由于条件有限,拿三台机器实现master的高可用。环境如下:

名称 IP 系统版本 角色 安装软件
k8s01 192.168.1.90 centos 7.7 master kubeadm、kubelet、kubectl、docker、haproxy、keepalived
k8s02 192.168.1.91 centos 7.7 master kubeadm、kubelet、kubectl、docker、haproxy、keepalived
k8s03 192.168.1.92 centos 7.7 master kubeadm、kubelet、kubectl、docker
VIP 192.168.1.99

三台都准备安装kubelet、kubeadm、kubectl、docker,前一篇文章里面的 准备工作 都需要运行。

HA高可用

安装软件

在k8s01、k8s02直接使用yum安装 keepalived以及haproxy这2个软件。

1
yum install -y keepalived haproxy

keepalived配置

/etc/keepalived/keepalived.conf的配置如下:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
! Configuration File for keepalived

global_defs {
router_id lvs_vip
}

vrrp_instance VI_1 {
state MASTER #k8s02备用节点配置为:BACKUP
interface eth0
virtual_router_id 51
priority 100
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
192.168.1.99 dev eth0 label eth0:0
}
}

注意router_id、virtual_router_id以及state这几个配置项。virtual_router_id必须不一样,值越高,其优先级越高。

haproxy配置

/etc/haproxy/haproxy.cfg配置如下:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
global
log 127.0.0.1 local2

chroot /var/lib/haproxy
pidfile /var/run/haproxy.pid
maxconn 4000
user haproxy
group haproxy
daemon
stats socket /var/lib/haproxy/stats

defaults
mode http
log global
option httplog
option dontlognull
option http-server-close
option forwardfor except 127.0.0.0/8
option redispatch
retries 3
timeout http-request 10s
timeout queue 1m
timeout connect 10s
timeout client 1m
timeout server 1m
timeout http-keep-alive 10s
timeout check 10s
maxconn 3000

listen k8s-apiserver
bind *:6444
mode tcp
balance roundrobin
timeout server 900s
timeout connect 15s

server app1 192.168.1.90:6443 check port 6443 inter 5000 fall 5
server app2 192.168.1.91:6443 check port 6443 inter 5000 fall 5
server app3 192.168.1.92:6443 check port 6443 inter 5000 fall 5

frontend stats-front
bind *:8081
mode http
default_backend stats-back

backend stats-back
mode http
balance roundrobin
stats hide-version
stats uri /haproxy/stats
stats auth admin:admin

haproxy配置主要估了2个事情:

  • 监听6444端口,对应后端为三台master节点的6443端口
  • 开启 /haproxy/stats 状态查看后端服务是否有正常

完成之后,启动服务。观察VIP是否在master节点上,访问 http://192.168.1.99/haproxy/stats;如果正常,将keepalived停止,再访问URL,如果正常就继续下一步操作。

1
2
systemctl enable keepalived && systemctl start keepalived
systemctl enable haproxy && systemctl start haproxy

初始化master节点

  1. 使用 kubeadm config print init-defaults 2>/dev/null >kubeadm-init.yaml 生成配置,再修改相应的配置如下。主要是controlPlaneEndpoint配置 192.168.1.99:6444,这个VIP加haproxy监听的端口。
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
apiVersion: kubeadm.k8s.io/v1beta2
bootstrapTokens:
- groups:
- system:bootstrappers:kubeadm:default-node-token
token: abcdef.0123456789abcdef
ttl: 24h0m0s
usages:
- signing
- authentication
kind: InitConfiguration
localAPIEndpoint:
advertiseAddress: 192.168.1.90
bindPort: 6443
nodeRegistration:
criSocket: /var/run/dockershim.sock
name: k8s01
taints:
- effect: NoSchedule
key: node-role.kubernetes.io/master
---
apiServer:
timeoutForControlPlane: 4m0s
apiVersion: kubeadm.k8s.io/v1beta2
certificatesDir: /etc/kubernetes/pki
clusterName: kubernetes
controlPlaneEndpoint: "192.168.1.99:6444"
dns:
type: CoreDNS
etcd:
local:
dataDir: /var/lib/etcd
imageRepository: registry.cn-hangzhou.aliyuncs.com/google_containers
kind: ClusterConfiguration
kubernetesVersion: v1.17.3
networking:
dnsDomain: cluster.local
serviceSubnet: 10.96.0.0/16
podSubnet: 10.100.0.0/16
scheduler: {}
---
apiVersion: kubeproxy.config.k8s.io/v1alpha1
kind: KubeProxyConfiguration
featureGates:
SupportIPVSProxyMode: true
mode: ipvs
  1. 下载镜像,每一台都需要操作:
1
2
3
4
5
6
7
8
[root@k8s-1 ~]# kubeadm config images pull --config kubeadm-init.yaml
[config/images] Pulled registry.cn-hangzhou.aliyuncs.com/google_containers/kube-apiserver:v1.17.3
[config/images] Pulled registry.cn-hangzhou.aliyuncs.com/google_containers/kube-controller-manager:v1.17.3
[config/images] Pulled registry.cn-hangzhou.aliyuncs.com/google_containers/kube-scheduler:v1.17.3
[config/images] Pulled registry.cn-hangzhou.aliyuncs.com/google_containers/kube-proxy:v1.17.3
[config/images] Pulled registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.1
[config/images] Pulled registry.cn-hangzhou.aliyuncs.com/google_containers/etcd:3.4.3-0
[config/images] Pulled registry.cn-hangzhou.aliyuncs.com/google_containers/coredns:1.6.5
  1. 在k8s01上面做初始化
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
[root@k8s01 ~]# kubeadm init --config kubeadm-init.yaml
W0220 08:53:24.018854 15694 validation.go:28] Cannot validate kube-proxy config - no validator is available
W0220 08:53:24.019041 15694 validation.go:28] Cannot validate kubelet config - no validator is available
[init] Using Kubernetes version: v1.17.3
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Starting the kubelet
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [k8s01 kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 192.168.1.90 192.168.1.99]
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [k8s01 localhost] and IPs [192.168.1.90 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [k8s01 localhost] and IPs [192.168.1.90 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[endpoint] WARNING: port specified in controlPlaneEndpoint overrides bindPort in the controlplane address
[kubeconfig] Writing "admin.conf" kubeconfig file
[endpoint] WARNING: port specified in controlPlaneEndpoint overrides bindPort in the controlplane address
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[endpoint] WARNING: port specified in controlPlaneEndpoint overrides bindPort in the controlplane address
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[endpoint] WARNING: port specified in controlPlaneEndpoint overrides bindPort in the controlplane address
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
W0220 08:53:37.967621 15694 manifests.go:214] the default kube-apiserver authorization-mode is "Node,RBAC"; using "Node,RBAC"
[control-plane] Creating static Pod manifest for "kube-scheduler"
W0220 08:53:38.025082 15694 manifests.go:214] the default kube-apiserver authorization-mode is "Node,RBAC"; using "Node,RBAC"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[apiclient] All control plane components are healthy after 31.011867 seconds
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config-1.17" in namespace kube-system with the configuration for the kubelets in the cluster
[upload-certs] Skipping phase. Please see --upload-certs
[mark-control-plane] Marking the node k8s01 as control-plane by adding the label "node-role.kubernetes.io/master=''"
[mark-control-plane] Marking the node k8s01 as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule]
[bootstrap-token] Using token: abcdef.0123456789abcdef
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
[kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key
[addons] Applied essential addon: CoreDNS
[endpoint] WARNING: port specified in controlPlaneEndpoint overrides bindPort in the controlplane address
[addons] Applied essential addon: kube-proxy

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
https://kubernetes.io/docs/concepts/cluster-administration/addons/

You can now join any number of control-plane nodes by copying certificate authorities
and service account keys on each node and then running the following as root:

kubeadm join 192.168.1.99:6444 --token abcdef.0123456789abcdef \
--discovery-token-ca-cert-hash sha256:fe1e721f5627faeb20bc8ce85e806d1410ea67130bf7975ad9924835e3ac0e43 \
--control-plane

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 192.168.1.99:6444 --token abcdef.0123456789abcdef \
--discovery-token-ca-cert-hash sha256:fe1e721f5627faeb20bc8ce85e806d1410ea67130bf7975ad9924835e3ac0e43
[root@k8s-1 ~]# mkdir -p $HOME/.kube
[root@k8s-1 ~]# cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
[root@k8s-1 ~]# chown $(id -u):$(id -g) $HOME/.kube/config

上面有2个 kubeadm join ,之前在单主模式下,只会出现worker node的加入命令。

  1. 需要上传证书到其他的机器上,脚本参考了官方文档:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
[root@k8s01 ~]# cat upload_cert.sh
#!/bin/bash

USER=root # customizable
CONTROL_PLANE_IPS="k8s02 k8s03"
for host in ${CONTROL_PLANE_IPS}; do
ssh "${USER}"@$host "mkdir -p /etc/kubernetes/pki/etcd"
scp /etc/kubernetes/pki/ca.crt "${USER}"@$host:/etc/kubernetes/pki/
scp /etc/kubernetes/pki/ca.key "${USER}"@$host:/etc/kubernetes/pki/
scp /etc/kubernetes/pki/sa.key "${USER}"@$host:/etc/kubernetes/pki/
scp /etc/kubernetes/pki/sa.pub "${USER}"@$host:/etc/kubernetes/pki/
scp /etc/kubernetes/pki/front-proxy-ca.crt "${USER}"@$host:/etc/kubernetes/pki/
scp /etc/kubernetes/pki/front-proxy-ca.key "${USER}"@$host:/etc/kubernetes/pki/
scp /etc/kubernetes/pki/etcd/ca.crt "${USER}"@$host:/etc/kubernetes/pki/etcd/
# Quote this line if you are using external etcd
scp /etc/kubernetes/pki/etcd/ca.key "${USER}"@$host:/etc/kubernetes/pki/etcd/
done
  1. 必须保证证书已经上传到了节点,然后在k8s02 k8s03进行join的操作。
1
2
3
kubeadm join 192.168.1.99:6444 --token abcdef.0123456789abcdef \
--discovery-token-ca-cert-hash sha256:fe1e721f5627faeb20bc8ce85e806d1410ea67130bf7975ad9924835e3ac0e43 \
--control-plane
  1. 在k8s01上面查看状态:
1
2
3
4
5
6
7
8
9
10
[root@k8s01 ~]# kubectl get nodes
NAME STATUS ROLES AGE VERSION
k8s01 NotReady master 31m v1.17.3
k8s02 NotReady master 12m v1.17.3
k8s03 NotReady master 8m14s v1.17.3
[root@k8s01 ~]# kubectl get componentstatuses
NAME STATUS MESSAGE ERROR
controller-manager Healthy ok
scheduler Healthy ok
etcd-0 Healthy {"health":"true"}
  1. 在k8s01上面安装网络组件calico:
1
2
3
4
5
6
#https://docs.projectcalico.org/getting-started/kubernetes/quickstart
wget https://docs.projectcalico.org/manifests/calico.yaml
#将网段替换为podSubnet所设置的网段,calico默认值为192.168.0.0.16
sed -i 's#192.168.0.0/16#10.100.0.0/16#' calico.yaml
kubectl apply -f calico.yaml
kubectl taint nodes --all node-role.kubernetes.io/master-
  1. 查看状态
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
[root@k8s01 ~]# k get nodes
NAME STATUS ROLES AGE VERSION
k8s01 Ready master 43m v1.17.3
k8s02 Ready master 25m v1.17.3
k8s03 Ready master 20m v1.17.3
[root@k8s01 ~]# kubectl get pod -n kube-system
NAME READY STATUS RESTARTS AGE
calico-kube-controllers-77c4b7448-k5n72 1/1 Running 0 5m6s
calico-node-25gww 1/1 Running 0 5m7s
calico-node-s8rmm 1/1 Running 0 5m7s
calico-node-t5pw9 1/1 Running 0 5m7s
coredns-7f9c544f75-6w798 1/1 Running 0 46m
coredns-7f9c544f75-b745b 1/1 Running 0 46m
etcd-k8s01 1/1 Running 0 46m
etcd-k8s02 1/1 Running 0 28m
etcd-k8s03 1/1 Running 0 24m
kube-apiserver-k8s01 1/1 Running 0 46m
kube-apiserver-k8s02 1/1 Running 0 28m
kube-apiserver-k8s03 1/1 Running 0 24m
kube-controller-manager-k8s01 1/1 Running 1 46m
kube-controller-manager-k8s02 1/1 Running 0 28m
kube-controller-manager-k8s03 1/1 Running 0 24m
kube-proxy-76njt 1/1 Running 0 24m
kube-proxy-nl2kl 1/1 Running 0 28m
kube-proxy-sz6mw 1/1 Running 0 46m
kube-scheduler-k8s01 1/1 Running 1 46m
kube-scheduler-k8s02 1/1 Running 0 28m
kube-scheduler-k8s03 1/1 Running 0 24m
[root@k8s01 ~]#
[root@k8s01 ~]#
[root@k8s01 ~]# ipvsadm -L -n
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 10.96.0.1:443 rr
-> 192.168.1.90:6443 Masq 1 1 0
-> 192.168.1.91:6443 Masq 1 0 0
-> 192.168.1.92:6443 Masq 1 1 0
TCP 10.96.0.10:53 rr
-> 10.100.235.129:53 Masq 1 0 0
-> 10.100.235.131:53 Masq 1 0 0
TCP 10.96.0.10:9153 rr
-> 10.100.235.129:9153 Masq 1 0 0
-> 10.100.235.131:9153 Masq 1 0 0
UDP 10.96.0.10:53 rr
-> 10.100.235.129:53 Masq 1 0 0
-> 10.100.235.131:53 Masq 1 0 0

看到三个节点都是master时,说明部署已经成功了。

参考资料

0%