目录
一.虚拟机准备
二.基础环境配置（各个节点都做）
1.IP和hosts解析
2.防火墙和selinux
3.安装基本软件
4.配置时间同步
5.禁用swap分区
6.修改内核参数并重载
7.配置ipvs
三.docker环境（各个节点都做）
1.配置软件源并安装docker-ce
2.配置docker加速
四.cri环境配置（各个节点都做）
1.下载查看版本
2.配置服务启动
harbor%E7%8E%AF%E5%A2%83%EF%BC%88%E5%8F%AA%E6%9C%89register%E8%8A%82%E7%82%B9%E5%81%9A%EF%BC%89-toc" style="margin-left:0px;">五.harbor环境（只有register节点做）
harbor%E9%85%8D%E7%BD%AE-toc" style="margin-left:40px;">1.安装harbor配置
2.验证
六.集群初始化
k8s%E6%89%80%E9%9C%80%E8%BD%AF%E4%BB%B6%EF%BC%88%E5%90%84%E4%B8%AA%E8%8A%82%E7%82%B9%E9%83%BD%E5%81%9A%EF%BC%89-toc" style="margin-left:40px;">1.下载k8s所需软件（各个节点都做）
harbor%E4%B8%8A%E6%96%B0%E5%BB%BA%E9%A1%B9%E7%9B%AE%EF%BC%8C%E5%B9%B6%E6%89%A7%E8%A1%8C%E8%84%9A%E6%9C%AC-toc" style="margin-left:40px;">2.在harbor上新建项目，并执行脚本
3.master节点初始化 （只在master节点做）
4.执行成功后出现此界面后进行权限设置
5.各个节点根据master初始化成功后的参数来加入（各个节点都做）
七.收尾工作（master节点做）
1.命令补全配置
2.网络配置
（1）下载
（2）执行脚本文件
（3）应用配置文件并验证
一.虚拟机准备
| 主机（我的主机名） | IP |
| master(node1-190.com) | 192.168.2.190 |
| node1(node2-191.com) | 192.168.2.191 |
| node2(node3-192.com) | 192.168.2.192 |
| node3(node4-193.com) | 192.168.2.193 |
| register(node5-196.com) | 192.168.2.196 |
二.基础环境配置（各个节点都做）
1.IP和hosts解析
[root@node1-190 ~]# vim /etc/hosts
192.168.2.190 node1-190.com
192.168.2.191 node2-191.com
192.168.2.192 node3-192.com
192.168.2.193 node4-193.com
192.168.2.196 node5-196.com2.防火墙和selinux
[root@node1-190 ~]# systemctl status firewalld.service;getenforce
● firewalld.service - firewalld - dynamic firewall daemon
Loaded: loaded (/usr/lib/systemd/system/firewalld.service; disabled; vendor preset: enabled)
Active: inactive (dead)
Docs: man:firewalld(1)
Disabled
#临时
systemctl stop firewalld
setenforce 0
#禁用
systemctl disable firewalld
sed -i '/^SELINUX=/ c SELINUX=disabled' /etc/selinux/config3.安装基本软件
yum install -y wget tree bash-completion lrzsz psmisc net-tools vim chrony4.配置时间同步
[root@node1-190 ~]# vim /etc/chrony.conf
:3,6 s/^/# #注释掉原有行
server ntp1.aliyun.com iburst
[root@node1-190 ~]# systemctl restart chronyd
[root@node1-190 ~]# chronyc sources
210 Number of sources = 1
MS Name/IP address Stratum Poll Reach LastRx Last sample
===============================================================================
^* 120.25.115.20 2 8 341 431 -357us[ -771us] +/- 20ms5.禁用swap分区
[root@node1-190 ~]# swapoff -a && sed -i 's/.*swap.*/#&/' /etc/fstab && free -m
total used free shared buff/cache available
Mem: 10376 943 8875 11 557 9178
Swap: 0 0 06.修改内核参数并重载
[root@node1-190 ~]# cat >> /etc/sysctl.d/k8s.conf << EOF
> vm.swappiness=0
> net.bridge.bridge-nf-call-ip6tables = 1
> net.bridge.bridge-nf-call-iptables = 1
> net.ipv4.ip_forward = 1
> EOF
[root@node1-190 ~]# modprobe br_netfilter && modprobe overlay && sysctl -p /etc/sysctl.d/k8s.conf
vm.swappiness = 0
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 17.配置ipvs
[root@node1-190 ~]# yum install ipset ipvsadm -y
[root@node1-190 ~]# cat <<EOF > /etc/sysconfig/modules/ipvs.modules
> #!/bin/bash
> modprobe -- ip_vs
> modprobe -- ip_vs_rr
> modprobe -- ip_vs_wrr
> modprobe -- ip_vs_sh
> modprobe -- nf_conntrack
> EOF
[root@node1-190 ~]# chmod +x /etc/sysconfig/modules/ipvs.modules && /bin/bash /etc/sysconfig/modules/ipvs.modules && lsmod | grep -e ip_vs -e nf_conntrack_ipv4
nf_conntrack_ipv4 15053 2
nf_defrag_ipv4 12729 1 nf_conntrack_ipv4
ip_vs_sh 12688 0
ip_vs_wrr 12697 0
ip_vs_rr 12600 0
ip_vs 145458 6 ip_vs_rr,ip_vs_sh,ip_vs_wrr
nf_conntrack 139264 7 ip_vs,nf_nat,nf_nat_ipv4,xt_conntrack,nf_nat_masquerade_ipv4,nf_conntrack_netlink,nf_conntrack_ipv4
libcrc32c 12644 4 xfs,ip_vs,nf_nat,nf_conntrack三.docker环境（各个节点都做）
1.配置软件源并安装docker-ce
[root@node1-190 ~]# yum install -y yum-utils device-mapper-persistent-data lvm2
[root@node1-190 ~]# yum-config-manager --add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
[root@node1-190 ~]# yum install -y docker-ce2.配置docker加速
[root@node1-190 ~]# cat >> /etc/docker/daemon.json <<-EOF
> {
> "registry-mirrors": [
> "http://74f21445.m.daocloud.io",
> "https://registry.docker-cn.com",
> "http://hub-mirror.c.163.com",
> "https://docker.mirrors.ustc.edu.cn"
> ],
> "insecure-registries": ["node5-196.com"], #写register节点的域名，各个节点的这个文件都写register的域名
> "exec-opts": ["native.cgroupdriver=systemd"]
> }
> EOF
[root@node1-190 ~]# systemctl daemon-reload && systemctl start docker四.cri环境配置（各个节点都做）
1.下载查看版本
链接：https://pan.baidu.com/s/1rRIqV-GO6ETe4N9glNs2cg
提取码：km17
[root@node1-190 ~]# tar -xf cri-dockerd-0.3.4.amd64.tgz -C /usr/local/
[root@node1-190 local]# ll
total 0
drwxr-xr-x. 2 root root 6 Apr 11 2018 bin
drwxrwxr-x 2 sulibao sulibao 25 Jun 30 03:31 cri-dockerd
drwxr-xr-x. 2 root root 6 Apr 11 2018 etc
drwxr-xr-x. 2 root root 6 Apr 11 2018 games
drwxr-xr-x. 2 root root 6 Apr 11 2018 include
drwxr-xr-x. 2 root root 6 Apr 11 2018 lib
drwxr-xr-x. 2 root root 6 Apr 11 2018 lib64
drwxr-xr-x. 2 root root 6 Apr 11 2018 libexec
drwxr-xr-x. 2 root root 6 Apr 11 2018 sbin
drwxr-xr-x. 5 root root 49 Jul 23 08:59 share
drwxr-xr-x. 2 root root 6 Apr 11 2018 src
[root@node1-190 local]# mv cri-dockerd/cri-dockerd /usr/local/bin/
[root@node1-190 local]# cri-dockerd --version
cri-dockerd 0.3.4 (e88b1605)2.配置服务启动
[root@node1-190 local]# vim /etc/systemd/system/cri-dockerd.service
[Unit]
Description=CRI Interface for Docker Application Container Engine
Documentation=https://docs.mirantis.com
After=network-online.target firewalld.service docker.service
Wants=network-online.target
[Service]
Type=notify
ExecStart=/usr/local/bin/cri-dockerd --pod-infra-container-image=registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.9 --network-plugin=cni --cni-conf-dir=/etc/cni/net.d --cni-bin-dir=/opt/cni/bin --container-runtime-endpoint=unix:///var/run/cri-dockerd.sock --cri-dockerd-root-directory=/var/lib/dockershim --docker-endpoint=unix:///var/run/docker.sock --cri-dockerd-root-directory=/var/lib/docker
ExecReload=/bin/kill -s HUP $MAINPID
TimeoutSec=0
RestartSec=2
Restart=always
StartLimitBurst=3
StartLimitInterval=60s
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity
TasksMax=infinity
Delegate=yes
KillMode=process
[Install]
WantedBy=multi-user.target
[root@node1-190 local]# vim /etc/systemd/system/cri-dockerd.socket
[Unit]
Description=CRI Docker Socket for the API
PartOf=cri-docker.service
[Socket]
ListenStream=/var/run/cri-dockerd.sock
SocketMode=0660
SocketUser=root
SocketGroup=docker
[Install]
WantedBy=sockets.target
[root@node1-190 ~]# systemctl daemon-reload
[root@node1-190 ~]# systemctl start cri-dockerd.service
[root@node1-190 ~]# systemctl enable cri-dockerd.serviceharbor%E7%8E%AF%E5%A2%83%EF%BC%88%E5%8F%AA%E6%9C%89register%E8%8A%82%E7%82%B9%E5%81%9A%EF%BC%89">五.harbor环境（只有register节点做）
harbor%E9%85%8D%E7%BD%AE">1.安装harbor配置
[root@node5-196 ~]# mv docker-compose-linux-x86_64 /usr/local/bin/docker-compose
[root@node5-196 ~]# chmod +x /usr/local/bin/docker-compose
[root@node5-196 ~]# tar -xf harbor-offline-installer-v2.8.4.tgz -C /usr/local/
[root@node5-196 ~]# cd /usr/local/
[root@node5-196 local]# ll
[root@node5-196 local]# cd harbor/
[root@node5-196 harbor]# ll
total 597536
-rw-r--r-- 1 root root 3639 Aug 15 17:53 common.sh
-rw-r--r-- 1 root root 611834153 Aug 15 17:54 harbor.v2.8.4.tar.gz
-rw-r--r-- 1 root root 12499 Aug 15 17:53 harbor.yml.tmpl
-rwxr-xr-x 1 root root 2725 Aug 15 17:53 install.sh
-rw-r--r-- 1 root root 11347 Aug 15 17:53 LICENSE
-rwxr-xr-x 1 root root 1881 Aug 15 17:53 prepare
[root@node5-196 harbor]# cp harbor.yml.tmpl harbor.yml
[root@node5-196 harbor]# vim harbor.yml
# Configuration file of Harbor
# The IP address or hostname to access admin UI and registry service.
# DO NOT use localhost or 127.0.0.1, because Harbor needs to be accessed by external clients.
hostname: node5-196.com #修改为自己的主机名
# http related config
http:
# port for http, default is 80. If https enabled, this port will redirect to https port
port: 80 #注意端口
# https related config
#https: #https区块全部注释
# https port for harbor, default is 443
#port: 443
# The path of cert and key files for nginx
#certificate: /your/certificate/path
#private_key: /your/private/key/path
[root@node5-196 harbor]# ./install.sh #安装
[root@node5-196 harbor]# docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
goharbor/harbor-exporter v2.8.4 b8d33e28ec68 2 weeks ago 97.7MB
goharbor/redis-photon v2.8.4 7b7324d651ca 2 weeks ago 120MB
goharbor/trivy-adapter-photon v2.8.4 91d8e9f0b21a 2 weeks ago 464MB
goharbor/notary-server-photon v2.8.4 a46f91560454 2 weeks ago 113MB
goharbor/notary-signer-photon v2.8.4 da66bd8d944b 2 weeks ago 110MB
goharbor/harbor-registryctl v2.8.4 805b38ca6bee 2 weeks ago 141MB
goharbor/registry-photon v2.8.4 756769e94123 2 weeks ago 79MB
goharbor/nginx-photon v2.8.4 375018db778b 2 weeks ago 116MB
goharbor/harbor-log v2.8.4 8a2045fb24d2 2 weeks ago 124MB
goharbor/harbor-jobservice v2.8.4 97808fc10f64 2 weeks ago 141MB
goharbor/harbor-core v2.8.4 c26fcd0714d8 2 weeks ago 164MB
goharbor/harbor-portal v2.8.4 4a8b0205c0f9 2 weeks ago 124MB
goharbor/harbor-db v2.8.4 5b8af16d7420 2 weeks ago 174MB
goharbor/prepare v2.8.4 bdbf974d86ce 2 weeks ago 166MB
[root@node5-196 harbor]# cat /etc/docker/daemon.json
{
"registry-mirrors": [
"http://74f21445.m.daocloud.io",
"https://registry.docker-cn.com",
"http://hub-mirror.c.163.com",
"https://docker.mirrors.ustc.edu.cn"
],
"insecure-registries": ["192.168.2.196"],
"insecure-registries": ["node5-196.com"],
"exec-opts": ["native.cgroupdriver=systemd"]
}
[root@node5-196 harbor]# tail -1 /etc/hosts
192.168.2.196 node5-196.com
[root@node5-196 harbor]# systemctl restart docker
[root@node5-196 harbor]# docker-compose up -d2.验证
[root@node5-196 harbor]# docker login http://node5-196.com #进行harbor上镜像的推送和拉取验证
Username: admin
Password:
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store
Login Succeeded
[root@node5-196 harbor]# docker tag busybox:latest node5-196.com/library/busybox:latest
[root@node5-196 harbor]# docker push node5-196.com/library/busybox:latest
The push refers to repository [node5-196.com/library/busybox]
3d24ee258efc: Layer already exists
latest: digest: sha256:023917ec6a886d0e8e15f28fb543515a5fcd8d938edb091e8147db4efed388ee size: 528
[root@node5-196 harbor]# docker pull node5-196.com/library/busybox:latest
latest: Pulling from library/busybox
Digest: sha256:023917ec6a886d0e8e15f28fb543515a5fcd8d938edb091e8147db4efed388ee
Status: Image is up to date for node5-196.com/library/busybox:latest
node5-196.com/library/busybox:latest
[root@node5-196 harbor]# docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
goharbor/harbor-exporter v2.8.4 b8d33e28ec68 2 weeks ago 97.7MB
goharbor/redis-photon v2.8.4 7b7324d651ca 2 weeks ago 120MB
goharbor/trivy-adapter-photon v2.8.4 91d8e9f0b21a 2 weeks ago 464MB
goharbor/notary-server-photon v2.8.4 a46f91560454 2 weeks ago 113MB
goharbor/notary-signer-photon v2.8.4 da66bd8d944b 2 weeks ago 110MB
goharbor/harbor-registryctl v2.8.4 805b38ca6bee 2 weeks ago 141MB
goharbor/registry-photon v2.8.4 756769e94123 2 weeks ago 79MB
goharbor/nginx-photon v2.8.4 375018db778b 2 weeks ago 116MB
goharbor/harbor-log v2.8.4 8a2045fb24d2 2 weeks ago 124MB
goharbor/harbor-jobservice v2.8.4 97808fc10f64 2 weeks ago 141MB
goharbor/harbor-core v2.8.4 c26fcd0714d8 2 weeks ago 164MB
goharbor/harbor-portal v2.8.4 4a8b0205c0f9 2 weeks ago 124MB
goharbor/harbor-db v2.8.4 5b8af16d7420 2 weeks ago 174MB
goharbor/prepare v2.8.4 bdbf974d86ce 2 weeks ago 166MB
busybox latest a416a98b71e2 6 weeks ago 4.26MB
node5-196.com/library/busybox latest a416a98b71e2 6 weeks ago 4.26MB
推送后的结果
六.集群初始化
k8s%E6%89%80%E9%9C%80%E8%BD%AF%E4%BB%B6%EF%BC%88%E5%90%84%E4%B8%AA%E8%8A%82%E7%82%B9%E9%83%BD%E5%81%9A%EF%BC%89">1.下载k8s所需软件（各个节点都做）
[root@node4-193 ~]# cat > /etc/yum.repos.d/kubernetes.repo << EOF
> [kubernetes]
> name=Kubernetes
> baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64
> enabled=1
> gpgcheck=0
> repo_gpgcheck=0
> gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg
> https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
> EOF
[root@node4-193 ~]#
[root@node4-193 ~]# yum install kubeadm kubectl kubelet -y
[root@node1-190 ~]# systemctl start kubeletharbor%E4%B8%8A%E6%96%B0%E5%BB%BA%E9%A1%B9%E7%9B%AE%EF%BC%8C%E5%B9%B6%E6%89%A7%E8%A1%8C%E8%84%9A%E6%9C%AC">2.在harbor上新建项目，并执行脚本
[root@node1-190 ~]# vim dockerimages.sh
#!/bin/bash
images=$(kubeadm config images list --kubernetes-version=1.28.0 | awk -F '/' '{print $NF}')
for i in ${images}
do
docker pull registry.aliyuncs.com/google_containers/$i
docker tag registry.aliyuncs.com/google_containers/$i node5-196.com/google_containers/$i #需要修改为刚创建的harbor的仓库地址
docker push node5-196.com/google_containers/$i
docker rmi registry.aliyuncs.com/google_containers/$i
done
[root@node1-190 ~]# docker login node5-196.com
Username: admin
Password:
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store
Login Succeeded
[root@node1-190 ~]# bash dockerimages.sh
[root@node1-190 ~]# docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
node5-196.com/google_containers/kube-apiserver v1.28.0 bb5e0dde9054 2 weeks ago 126MB
node5-196.com/google_containers/kube-scheduler v1.28.0 f6f496300a2a 2 weeks ago 60.1MB
node5-196.com/google_containers/kube-controller-manager v1.28.0 4be79c38a4ba 2 weeks ago 122MB
node5-196.com/google_containers/kube-proxy v1.28.0 ea1030da44aa 2 weeks ago 73.1MB
node5-196.com/google_containers/etcd 3.5.9-0 73deb9a3f702 3 months ago 294MB
node5-196.com/google_containers/coredns v1.10.1 ead0a4a53df8 6 months ago 53.6MB
node5-196.com/google_containers/pause 3.9 e6f181688397 10 months ago 744kB
3.master节点初始化 （只在master节点做）
执行完脚本无误后执行以下命令
kubeadm init --kubernetes-version=1.28.0 \
--apiserver-advertise-address=192.168.2.190 \ #修改为master节点的IP
--image-repository node5-196.com/google_containers \ #修改为harbor仓库的地址
--service-cidr=10.96.0.0/12 \
--pod-network-cidr=10.244.0.0/16 \
--ignore-preflight-errors=Swap \
--cri-socket=unix:///var/run/cri-dockerd.sock4.执行成功后出现此界面后进行权限设置
[root@node1-190 ~]# mkdir -p $HOME/.kube && cp -i /etc/kubernetes/admin.conf $HOME/.kube/config && chown $(id -u):$(id -g) $HOME/.kube/config && export KUBECONFIG=/etc/kubernetes/admin.conf5.各个节点根据master初始化成功后的参数来加入（各个节点都做）
kubeadm join 192.168.2.190:6443 --token m5mkth.rb2e9rb3892fsu37 \
--discovery-token-ca-cert-hash sha256:1209d580742d411fb4406953185ef9f27522b3ba150b8726fd39f364628d0942 \
--cri-socket=unix:///var/run/cri-dockerd.sock 出现此界面即成功
[root@node1-190 ~]# kubectl get nodes
NAME STATUS ROLES AGE VERSION
node1-190.com NotReady control-plane 121m v1.28.1
node2-191.com NotReady <none> 104m v1.28.1
node3-192.com NotReady <none> 96m v1.28.1
node4-193.com NotReady <none> 11m v1.28.1
node5-196.com NotReady <none> 9s v1.28.1七.收尾工作（master节点做）
1.命令补全配置
[root@node1-190 ~]# echo "source <(kubectl completion bash)" >> ~/.bashrc && echo "source <(kubeadm completion bash)" >> ~/.bashrc && source ~/.bashrc2.网络配置
（1）下载
链接：https://pan.baidu.com/s/1TRUyPxbBZRcyyyqojquhBw
提取码：wvap
（2）执行脚本文件
[root@node1-190 ~]# vim flannel.sh
#!/bin/bash
for i in $(grep image kube-flannel.yml | grep -v '#' | awk -F '/' '{print $NF}')
do
docker pull flannel/$i
docker tag flannel/$i node5-196.com/google_containers/$i #需要修改为刚创建的harbor的仓库地址
docker push node5-196.com/google_containers/$i
docker rmi flannel/$i
done
[root@node1-190 ~]# bash flannel.sh
[root@node1-190 ~]# docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
node5-196.com/google_containers/flannel v0.22.2 d73868a08083 13 days ago 70.2MB
node5-196.com/google_containers/kube-apiserver v1.28.0 bb5e0dde9054 2 weeks ago 126MB
node5-196.com/google_containers/kube-controller-manager v1.28.0 4be79c38a4ba 2 weeks ago 122MB
node5-196.com/google_containers/kube-scheduler v1.28.0 f6f496300a2a 2 weeks ago 60.1MB
node5-196.com/google_containers/kube-proxy v1.28.0 ea1030da44aa 2 weeks ago 73.1MB
node5-196.com/google_containers/flannel-cni-plugin v1.2.0 a55d1bad692b 5 weeks ago 8.04MB
node5-196.com/google_containers/etcd 3.5.9-0 73deb9a3f702 3 months ago 294MB
node5-196.com/google_containers/coredns v1.10.1 ead0a4a53df8 6 months ago 53.6MB
node5-196.com/google_containers/pause 3.9 e6f181688397 10 months ago 744kB
registry.cn-hangzhou.aliyuncs.com/google_containers/pause 3.9 e6f181688397 10 months ago 744kB
[root@node1-190 ~]# sed -i '/ image:/s#docker.io/flannel#node5-196.com/google_containers#' kube-flannel.yml（3）应用配置文件并验证
[root@node1-190 ~]# kubectl apply -f kube-flannel.yml #应用
[root@node1-190 ~]# kubectl get pod -n kube-system #验证
NAME READY STATUS RESTARTS AGE
coredns-6845c5578-g6dnk 1/1 Running 0 159m
coredns-6845c5578-z5cj4 1/1 Running 0 159m
etcd-node1-190.com 1/1 Running 0 160m
kube-apiserver-node1-190.com 1/1 Running 0 160m
kube-controller-manager-node1-190.com 1/1 Running 0 160m
kube-proxy-5k8gg 1/1 Running 0 134m
kube-proxy-b7kcg 1/1 Running 0 159m
kube-proxy-h5zrz 1/1 Running 0 142m
kube-proxy-h9gs5 1/1 Running 1 (35m ago) 38m
kube-proxy-vfx9x 1/1 Running 0 49m
kube-scheduler-node1-190.com 1/1 Running 0 160m
