使用 kubeadm 部署 kubernetes(CRI 使用 containerd)
kubeadm 提供了 kubeadm init 以及 kubeadm join 这两个命令作为快速创建 kubernetes 集群的最佳实践。

安装 containerd

参考 安装 Docker 一节添加 apt/yum 源,之后执行如下命令。
1
# debian 系
2
$ sudo apt install containerd.io
3
4
# rhel 系
5
$ sudo yum install containerd.io
Copied!

配置 containerd

新建 /etc/systemd/system/cri-containerd.service 文件
1
[Unit]
2
Description=containerd container runtime for kubernetes
3
Documentation=https://containerd.io
4
After=network.target local-fs.target
5
6
[Service]
7
ExecStartPre=-/sbin/modprobe overlay
8
ExecStart=/usr/bin/containerd --config //etc/cri-containerd/config.toml
9
10
Type=notify
11
Delegate=yes
12
KillMode=process
13
Restart=always
14
RestartSec=5
15
# Having non-zero Limit*s causes performance problems due to accounting overhead
16
# in the kernel. We recommend using cgroups to do container-local accounting.
17
LimitNPROC=infinity
18
LimitCORE=infinity
19
LimitNOFILE=infinity
20
# Comment TasksMax if your systemd version does not supports it.
21
# Only systemd 226 and above support this version.
22
TasksMax=infinity
23
OOMScoreAdjust=-999
24
25
[Install]
26
WantedBy=multi-user.target
Copied!
新建 /etc/cri-containerd/config.toml containerd 配置文件
1
version = 2
2
# persistent data location
3
root = "/var/lib/cri-containerd"
4
# runtime state information
5
state = "/run/cri-containerd"
6
plugin_dir = ""
7
disabled_plugins = []
8
required_plugins = []
9
# set containerd's OOM score
10
oom_score = 0
11
12
[grpc]
13
address = "/run/cri-containerd/cri-containerd.sock"
14
tcp_address = ""
15
tcp_tls_cert = ""
16
tcp_tls_key = ""
17
# socket uid
18
uid = 0
19
# socket gid
20
gid = 0
21
max_recv_message_size = 16777216
22
max_send_message_size = 16777216
23
24
[debug]
25
address = ""
26
format = "json"
27
uid = 0
28
gid = 0
29
level = ""
30
31
[metrics]
32
address = "127.0.0.1:1338"
33
grpc_histogram = false
34
35
[cgroup]
36
path = ""
37
38
[timeouts]
39
"io.containerd.timeout.shim.cleanup" = "5s"
40
"io.containerd.timeout.shim.load" = "5s"
41
"io.containerd.timeout.shim.shutdown" = "3s"
42
"io.containerd.timeout.task.state" = "2s"
43
44
[plugins]
45
[plugins."io.containerd.gc.v1.scheduler"]
46
pause_threshold = 0.02
47
deletion_threshold = 0
48
mutation_threshold = 100
49
schedule_delay = "0s"
50
startup_delay = "100ms"
51
[plugins."io.containerd.grpc.v1.cri"]
52
disable_tcp_service = true
53
stream_server_address = "127.0.0.1"
54
stream_server_port = "0"
55
stream_idle_timeout = "4h0m0s"
56
enable_selinux = false
57
selinux_category_range = 1024
58
sandbox_image = "registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.5"
59
stats_collect_period = 10
60
# systemd_cgroup = false
61
enable_tls_streaming = false
62
max_container_log_line_size = 16384
63
disable_cgroup = false
64
disable_apparmor = false
65
restrict_oom_score_adj = false
66
max_concurrent_downloads = 3
67
disable_proc_mount = false
68
unset_seccomp_profile = ""
69
tolerate_missing_hugetlb_controller = true
70
disable_hugetlb_controller = true
71
ignore_image_defined_volumes = false
72
[plugins."io.containerd.grpc.v1.cri".containerd]
73
snapshotter = "overlayfs"
74
default_runtime_name = "runc"
75
no_pivot = false
76
disable_snapshot_annotations = false
77
discard_unpacked_layers = false
78
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes]
79
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
80
runtime_type = "io.containerd.runc.v2"
81
pod_annotations = []
82
container_annotations = []
83
privileged_without_host_devices = false
84
base_runtime_spec = ""
85
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
86
# SystemdCgroup enables systemd cgroups.
87
SystemdCgroup = true
88
# BinaryName is the binary name of the runc binary.
89
# BinaryName = "runc"
90
# BinaryName = "crun"
91
# NoPivotRoot disables pivot root when creating a container.
92
# NoPivotRoot = false
93
94
# NoNewKeyring disables new keyring for the container.
95
# NoNewKeyring = false
96
97
# ShimCgroup places the shim in a cgroup.
98
# ShimCgroup = ""
99
100
# IoUid sets the I/O's pipes uid.
101
# IoUid = 0
102
103
# IoGid sets the I/O's pipes gid.
104
# IoGid = 0
105
106
# Root is the runc root directory.
107
Root = ""
108
109
# CriuPath is the criu binary path.
110
# CriuPath = ""
111
112
# CriuImagePath is the criu image path
113
# CriuImagePath = ""
114
115
# CriuWorkPath is the criu work path.
116
# CriuWorkPath = ""
117
[plugins."io.containerd.grpc.v1.cri".cni]
118
bin_dir = "/opt/cni/bin"
119
conf_dir = "/etc/cni/net.d"
120
max_conf_num = 1
121
conf_template = ""
122
[plugins."io.containerd.grpc.v1.cri".registry]
123
config_path = "/etc/cri-containerd/certs.d"
124
[plugins."io.containerd.grpc.v1.cri".registry.headers]
125
# Foo = ["bar"]
126
[plugins."io.containerd.grpc.v1.cri".image_decryption]
127
key_model = ""
128
[plugins."io.containerd.grpc.v1.cri".x509_key_pair_streaming]
129
tls_cert_file = ""
130
tls_key_file = ""
131
[plugins."io.containerd.internal.v1.opt"]
132
path = "/opt/cri-containerd"
133
[plugins."io.containerd.internal.v1.restart"]
134
interval = "10s"
135
[plugins."io.containerd.metadata.v1.bolt"]
136
content_sharing_policy = "shared"
137
[plugins."io.containerd.monitor.v1.cgroups"]
138
no_prometheus = false
139
[plugins."io.containerd.runtime.v2.task"]
140
platforms = ["linux/amd64"]
141
[plugins."io.containerd.service.v1.diff-service"]
142
default = ["walking"]
143
[plugins."io.containerd.snapshotter.v1.devmapper"]
144
root_path = ""
145
pool_name = ""
146
base_image_size = ""
147
async_remove = false
Copied!

安装 kubelet kubeadm kubectl cri-tools kubernetes-cni

Ubuntu/Debian

1
$ apt-get update && apt-get install -y apt-transport-https
2
$ curl https://mirrors.aliyun.com/kubernetes/apt/doc/apt-key.gpg | apt-key add -
3
4
$ cat <<EOF | sudo tee /etc/apt/sources.list.d/kubernetes.list
5
deb https://mirrors.aliyun.com/kubernetes/apt/ kubernetes-xenial main
6
EOF
7
8
$ apt-get update
9
$ apt-get install -y kubelet kubeadm kubectl
Copied!

CentOS/Fedora

1
$ cat <<EOF | sudo tee /etc/yum.repos.d/kubernetes.repo
2
[kubernetes]
3
name=Kubernetes
4
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
5
enabled=1
6
gpgcheck=1
7
repo_gpgcheck=1
8
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
9
EOF
10
11
$ sudo yum install -y kubelet kubeadm kubectl
Copied!

修改内核的运行参数

1
$ cat <<EOF | sudo tee /etc/sysctl.d/99-kubernetes-cri.conf
2
net.bridge.bridge-nf-call-iptables = 1
3
net.ipv4.ip_forward = 1
4
net.bridge.bridge-nf-call-ip6tables = 1
5
EOF
6
7
# 应用配置
8
$ sysctl --system
Copied!

配置 kubelet

修改 kubelet.service

/etc/systemd/system/kubelet.service.d/10-proxy-ipvs.conf 写入以下内容
1
# 启用 ipvs 相关内核模块
2
[Service]
3
ExecStartPre=-/sbin/modprobe ip_vs
4
ExecStartPre=-/sbin/modprobe ip_vs_rr
5
ExecStartPre=-/sbin/modprobe ip_vs_wrr
6
ExecStartPre=-/sbin/modprobe ip_vs_sh
Copied!
执行以下命令应用配置。
1
$ sudo systemctl daemon-reload
Copied!

部署

master

1
$ systemctl enable cri-containerd
2
3
$ systemctl start cri-containerd
4
5
$ sudo kubeadm init \
6
--image-repository registry.cn-hangzhou.aliyuncs.com/google_containers \
7
--pod-network-cidr 10.244.0.0/16 \
8
--cri-socket /run/cri-containerd/cri-containerd.sock \
9
--v 5 \
10
--ignore-preflight-errors=all
Copied!
  • --pod-network-cidr 10.244.0.0/16 参数与后续 CNI 插件有关,这里以 flannel 为例,若后续部署其他类型的网络插件请更改此参数。
执行可能出现错误,例如缺少依赖包,根据提示安装即可。
执行成功会输出
1
...
2
[addons] Applied essential addon: CoreDNS
3
I1116 12:35:13.270407 86677 request.go:538] Throttling request took 181.409184ms, request: POST:https://192.168.199.100:6443/api/v1/namespaces/kube-system/serviceaccounts
4
I1116 12:35:13.470292 86677 request.go:538] Throttling request took 186.088112ms, request: POST:https://192.168.199.100:6443/api/v1/namespaces/kube-system/configmaps
5
[addons] Applied essential addon: kube-proxy
6
7
Your Kubernetes control-plane has initialized successfully!
8
9
To start using your cluster, you need to run the following as a regular user:
10
11
mkdir -p $HOME/.kube
12
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
13
sudo chown $(id -u):$(id -g) $HOME/.kube/config
14
15
You should now deploy a pod network to the cluster.
16
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
17
https://kubernetes.io/docs/concepts/cluster-administration/addons/
18
19
Then you can join any number of worker nodes by running the following on each as root:
20
21
kubeadm join 192.168.199.100:6443 --token cz81zt.orsy9gm9v649e5lf \
22
--discovery-token-ca-cert-hash sha256:5edb316fd0d8ea2792cba15cdf1c899a366f147aa03cba52d4e5c5884ad836fe
Copied!

node 工作节点

另一主机 重复 部署 小节以前的步骤,安装配置好 kubelet。根据提示,加入到集群。
1
$ systemctl enable cri-containerd
2
3
$ systemctl start cri-containerd
4
5
$ kubeadm join 192.168.199.100:6443 \
6
--token cz81zt.orsy9gm9v649e5lf \
7
--discovery-token-ca-cert-hash sha256:5edb316fd0d8ea2792cba15cdf1c899a366f147aa03cba52d4e5c5884ad836fe \
8
--cri-socket /run/cri-containerd/cri-containerd.sock
Copied!

查看服务

所有服务启动后,通过 crictl 查看本地实际运行的容器。这些服务大概分为三类:主节点服务、工作节点服务和其它服务。
1
CONTAINER_RUNTIME_ENDPOINT=/run/cri-containerd/cri-containerd.sock crictl ps -a
Copied!

主节点服务

  • apiserver 是整个系统的对外接口,提供 RESTful 方式供客户端和其它组件调用;
  • scheduler 负责对资源进行调度,分配某个 pod 到某个节点上;
  • controller-manager 负责管理控制器,包括 endpoint-controller(刷新服务和 pod 的关联信息)和 replication-controller(维护某个 pod 的复制为配置的数值)。

工作节点服务

  • proxy 为 pod 上的服务提供访问的代理。

其它服务

  • Etcd 是所有状态的存储数据库;

使用

/etc/kubernetes/admin.conf 复制到 ~/.kube/config
执行 $ kubectl get all -A 查看启动的服务。
由于未部署 CNI 插件,CoreDNS 未正常启动。如何使用 Kubernetes,请参考后续章节。

部署 CNI

这里以 flannel 为例进行介绍。

flannel

检查 podCIDR 设置
1
$ kubectl get node -o yaml | grep CIDR
2
3
# 输出
4
podCIDR: 10.244.0.0/16
5
podCIDRs:
Copied!
1
$ kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/v0.11.0/Documentation/kube-flannel.yml
Copied!

master 节点默认不能运行 pod

如果用 kubeadm 部署一个单节点集群,默认情况下无法使用,请执行以下命令解除限制
1
$ kubectl taint nodes --all node-role.kubernetes.io/master-
2
3
# 恢复默认值
4
# $ kubectl taint nodes NODE_NAME node-role.kubernetes.io/master=true:NoSchedule
Copied!

参考文档