• 对于注定会优秀的人来说,他所需要的,只是时间!
  • 手懒得,必受贫穷,手勤的,必得富足----《圣经》
  • 帮助别人,成就自己。愿君在本站能真正有所收获!
  • 如果你在本站中发现任何问题,欢迎留言指正!
  • 宝剑锋从磨砺出,梅花香自苦寒来!

<十八>Kubernetes学习笔记-手动搭建k8s-1.10.4之系统初始化

kubernetes eryajf 2年前 (2018-12-09) 2556°C 已收录 2个评论
本文预计阅读时间 27 分钟

1,集群机器

  • kube-node1:192.168.106.3
  • kube-node2:192.168.106.4
  • kube-node3:192.168.106.5
  • VIP:192.168.106.110

2,主机名

  • 设置永久主机名称,然后重新登录:
$ sudo hostnamectl set-hostname kube-node1 # 将 kube-node1 替换为当前主机名
$ sudo hostnamectl set-hostname kube-node2 # 将 kube-node2 替换为当前主机名
$ sudo hostnamectl set-hostname kube-node3 # 将 kube-node3 替换为当前主机名
  • 设置的主机名保存在 /etc/hosts 文件中;
$cat > /etc/hosts << EOF
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.106.3 kube-node1
192.168.106.4 kube-node2
192.168.106.5 kube-node3
EOF

3,添加 k8s 和 docker 账户

  • 在每台机器上添加 k8s 账户,可以无密码 sudo:
$ sudo useradd -m k8s
$ echo 123456 | passwd k8s --stdin # 为 k8s 账户设置密码
$ sudo visudo
$ sudo grep '%wheel.*NOPASSWD: ALL' /etc/sudoers
%wheel    ALL=(ALL)    NOPASSWD: ALL
$ sudo gpasswd -a k8s wheel
  • 在每台机器上添加 docker 账户,将 k8s 账户添加到 docker 组中,同时配置 dockerd 参数:
$ sudo useradd -m docker
$ sudo gpasswd -a k8s docker
$ sudo mkdir -p  /etc/docker/
$ cat > /etc/docker/daemon.json << EOF
{
    "registry-mirrors": ["https://hub-mirror.c.163.com","https://docker.mirrors.ustc.edu.cn"],
    "max-concurrent-downloads": 20
}
EOF

4,无密码 ssh 登录其它节点

如果没有特殊指明,本文档的所有操作均在 kube-node1 节点上执行,然后远程分发文件和执行命令。稍候会介绍一个神器,通过这个,就能够实现基本上全程在节点1中部署整个集群。

设置 kube-node1 可以无密码登录所有节点的 k8s 和 root 账户:

[k8s@kube-node1 k8s]$ ssh-keygen -t rsa
[k8s@kube-node1 k8s]$ ssh-copy-id root@kube-node1
[k8s@kube-node1 k8s]$ ssh-copy-id root@kube-node2
[k8s@kube-node1 k8s]$ ssh-copy-id root@kube-node3

[k8s@kube-node1 k8s]$ ssh-copy-id k8s@kube-node1
[k8s@kube-node1 k8s]$ ssh-copy-id k8s@kube-node2
[k8s@kube-node1 k8s]$ ssh-copy-id k8s@kube-node3

以后的部署操作,也就在kube-node1的k8s用户家目录下进行了,不要觉得不方便。

5,将可执行文件路径 /opt/k8s/bin 添加到 PATH 变量中

在每台机器上添加环境变量:

$ sudo sh -c "echo 'PATH=/opt/k8s/bin:$PATH:$HOME/bin:$JAVA_HOME/bin' >>/root/.bashrc"
$ echo 'PATH=/opt/k8s/bin:$PATH:$HOME/bin:$JAVA_HOME/bin' >>~/.bashrc

6,环境变量

现在将要引入一个环境变量,以便于整个流程的部署工作,一开始有点不大适应,但是习惯了这种操作之后,发现这个思路,结合在k8s的部署之上,是真的犀利。

定义变量之前,我们先定义所有这次部署k8s所工作的目录。

mkdir -p /opt/k8s/bin/

变量内容如下:

cat > /opt/k8s/bin/environment.sh << "EOF"

#!/usr/bin/bash

# 生成 EncryptionConfig 所需的加密 key
export ENCRYPTION_KEY=$(head -c 32 /dev/urandom | base64)

# 最好使用当前未用的网段来定义服务网段和Pod网段

# 服务网段,部署前路由不可达,部署后集群内路由可达(kube-proxy 和 ipvs 保证)
export SERVICE_CIDR="10.254.0.0/16"

# Pod 网段,建议 /16 段地址,部署前路由不可达,部署后集群内路由可达(flanneld 保证)
export CLUSTER_CIDR="172.30.0.0/16"

# 服务端口范围 (NodePort Range)
export NODE_PORT_RANGE="8400-9000"

# 集群各机器 IP 数组
export NODE_IPS=(192.168.106.3 192.168.106.4 192.168.106.5)

# 集群各 IP 对应的 主机名数组
export NODE_NAMES=(kube-node1 kube-node2 kube-node3)

# kube-apiserver 的 VIP(HA 组件 keepalived 发布的 IP)
export MASTER_VIP=192.168.106.110

# kube-apiserver VIP 地址(HA 组件 haproxy 监听 8443 端口)
export KUBE_APISERVER="https://${MASTER_VIP}:8443"

# HA 节点,配置 VIP 的网络接口名称
export VIP_IF="eth0"

# etcd 集群服务地址列表
export ETCD_ENDPOINTS="https://192.168.106.3:2379,https://192.168.106.4:2379,https://192.168.106.5:2379"

# etcd 集群间通信的 IP 和端口
export ETCD_NODES="kube-node1=https://192.168.106.3:2380,kube-node2=https://192.168.106.4:2380,kube-node3=https://192.168.106.5:2380"

# flanneld 网络配置前缀
export FLANNEL_ETCD_PREFIX="/kubernetes/network"

# kubernetes 服务 IP (一般是 SERVICE_CIDR 中第一个IP)
export CLUSTER_KUBERNETES_SVC_IP="10.254.0.1"

# 集群 DNS 服务 IP (从 SERVICE_CIDR 中预分配)
export CLUSTER_DNS_SVC_IP="10.254.0.2"

# 集群 DNS 域名
export CLUSTER_DNS_DOMAIN="cluster.local."

# 将二进制目录 /opt/k8s/bin 加到 PATH 中
export PATH=/opt/k8s/bin:$PATH
EOF

其中需要注意相关IP的设置对应,不要搞错了。

7,安装依赖包

在每台机器上安装依赖包:

$ sudo yum install -y epel-release
$ sudo yum install -y conntrack ipvsadm ipset jq sysstat curl iptables libseccomp

上边是传统方式的安装方法,现在可以采用一下刚刚定义的方式安装一下:

定义一个脚本

cat > magic.sh << "EOF"
#!/bin/bash

source /opt/k8s/bin/environment.sh

for node_ip in ${NODE_IPS[@]}
do
    echo ">>> ${node_ip}"
    ssh root@${node_ip} "yum install -y epel-release conntrack ipvsadm ipset jq sysstat curl iptables libseccomp"
done
EOF

注意:在第一个EOF中加引号,这样文本当中的变量就不会被替换。

执行这个脚本:

[k8s@kube-node1 ~]$ bash magic.sh
>>> 192.168.106.3
Loaded plugins: fastestmirror
Repository updates is listed more than once in the configuration
Repository extras is listed more than once in the configuration
Repository centosplus is listed more than once in the configuration
Loading mirror speeds from cached hostfile
 * base: mirrors.cn99.com
 * epel: mirrors.aliyun.com
 * extras: mirrors.cn99.com
 * updates: mirrors.aliyun.com
Package epel-release-7-11.noarch already installed and latest version
Package conntrack-tools-1.4.4-3.el7_3.x86_64 already installed and latest version
Package ipvsadm-1.27-7.el7.x86_64 already installed and latest version
Package ipset-6.29-1.el7.x86_64 already installed and latest version
Package jq-1.5-1.el7.x86_64 already installed and latest version
Package sysstat-10.1.5-13.el7.x86_64 already installed and latest version
Package curl-7.29.0-46.el7.x86_64 already installed and latest version
Package iptables-1.4.21-24.1.el7_5.x86_64 already installed and latest version
Package libseccomp-2.3.1-3.el7.x86_64 already installed and latest version
Nothing to do
>>> 192.168.106.4
Loaded plugins: fastestmirror
Repository updates is listed more than once in the configuration
Repository extras is listed more than once in the configuration
Repository centosplus is listed more than once in the configuration
Loading mirror speeds from cached hostfile
 * base: mirrors.cn99.com
 * epel: mirrors.aliyun.com
 * extras: mirrors.cn99.com
 * updates: mirrors.163.com
Package epel-release-7-11.noarch already installed and latest version
Package conntrack-tools-1.4.4-3.el7_3.x86_64 already installed and latest version
Package ipvsadm-1.27-7.el7.x86_64 already installed and latest version
Package ipset-6.29-1.el7.x86_64 already installed and latest version
Package jq-1.5-1.el7.x86_64 already installed and latest version
Package sysstat-10.1.5-13.el7.x86_64 already installed and latest version
Package curl-7.29.0-46.el7.x86_64 already installed and latest version
Package iptables-1.4.21-24.1.el7_5.x86_64 already installed and latest version
Package libseccomp-2.3.1-3.el7.x86_64 already installed and latest version
Nothing to do
>>> 192.168.106.5
Loaded plugins: fastestmirror
Repository updates is listed more than once in the configuration
Repository extras is listed more than once in the configuration
Repository centosplus is listed more than once in the configuration
Loading mirror speeds from cached hostfile
 * base: mirrors.cn99.com
 * epel: mirrors.aliyun.com
 * extras: mirrors.cn99.com
 * updates: mirrors.aliyun.com
Package epel-release-7-11.noarch already installed and latest version
Package conntrack-tools-1.4.4-3.el7_3.x86_64 already installed and latest version
Package ipvsadm-1.27-7.el7.x86_64 already installed and latest version
Package ipset-6.29-1.el7.x86_64 already installed and latest version
Package jq-1.5-1.el7.x86_64 already installed and latest version
Package sysstat-10.1.5-13.el7.x86_64 already installed and latest version
Package curl-7.29.0-46.el7.x86_64 already installed and latest version
Package iptables-1.4.21-24.1.el7_5.x86_64 already installed and latest version
Package libseccomp-2.3.1-3.el7.x86_64 already installed and latest version
Nothing to do

因为已经装过了,所以会报没什么可装的。

8,关闭防火墙

将新内容导入到对应脚本,而后就这样,一劳永逸的,一步一步搭建下去。

cat > magic.sh << "EOF"
#!/bin/bash

source /opt/k8s/bin/environment.sh

for node_ip in ${NODE_IPS[@]}
do
    echo ">>> ${node_ip}"
    ssh root@${node_ip} "systemctl stop firewalld && systemctl disable firewalld"
    ssh root@${node_ip} "iptables -F && sudo iptables -X && sudo iptables -F -t nat && sudo iptables -X -t nat && iptables -P FORWARD ACCEPT"
done
EOF

9,关闭 swap 分区

如果开启了 swap 分区,kubelet 会启动失败(可以通过将参数 –fail-swap-on 设置为 false 来忽略 swap on),故需要在每台机器上关闭 swap 分区,为了防止开机自动挂载 swap 分区,也要注释 /etc/fstab 中相应的条目。

cat > magic.sh << "EOF"
#!/bin/bash

source /opt/k8s/bin/environment.sh

for node_ip in ${NODE_IPS[@]}
do
    echo ">>> ${node_ip}"
    ssh root@${node_ip} "swapoff -a && sed -i '/ swap / s/^\(.*\)$/#\1/g' /etc/fstab"
done
EOF

10,关闭 SELinux

关闭 SELinux,否则后续 K8S 挂载目录时可能报错 Permission denied:

$ sudo setenforce 0
$ grep SELINUX /etc/selinux/config
SELINUX=disabled

11,加载内核模块

cat > magic.sh << "EOF"
#!/bin/bash

source /opt/k8s/bin/environment.sh

for node_ip in ${NODE_IPS[@]}
do
    echo ">>> ${node_ip}"
    ssh root@${node_ip} "modprobe br_netfilter && modprobe ip_vs"
done
EOF

12,设置系统参数

cat > kubernetes.conf <<EOF
net.bridge.bridge-nf-call-iptables=1
net.bridge.bridge-nf-call-ip6tables=1
net.ipv4.ip_forward=1
net.ipv4.tcp_tw_recycle=0
vm.swappiness=0
vm.overcommit_memory=1
vm.panic_on_oom=0
fs.inotify.max_user_watches=89100
fs.file-max=52706963
fs.nr_open=52706963
net.ipv6.conf.all.disable_ipv6=1
net.netfilter.nf_conntrack_max=2310720
EOF

分发给三台主机,并加载。

cat > magic.sh << "EOF"
#!/bin/bash

source /opt/k8s/bin/environment.sh

for node_ip in ${NODE_IPS[@]}
do
    echo ">>> ${node_ip}"
    scp kubernetes.conf root@{node_ip}:/etc/sysctl.d/kubernetes.conf
    ssh root@${node_ip} "sysctl -p /etc/sysctl.d/kubernetes.conf"
done
EOF

13,安装一些基础包

安装几个常用但系统未必安装的包,同时配置一下时间同步,并把时间同步写入到定时任务当中。

cat > magic.sh << "EOF"
#!/bin/bash

source /opt/k8s/bin/environment.sh

for node_ip in ${NODE_IPS[@]}
do
    echo ">>> ${node_ip}"
    ssh root@${node_ip} 'yum -y install wget ntpdate lrzsz curl rsync && ntpdate -u cn.pool.ntp.org && echo "* * * * * /usr/sbin/ntpdate -u cn.pool.ntp.org &> /dev/null" > /var/spool/cron/root'
done
EOF

14,创建目录

cat > magic.sh << "EOF"
#!/bin/bash

source /opt/k8s/bin/environment.sh

for node_ip in ${NODE_IPS[@]}
do
    echo ">>> ${node_ip}"
    ssh root@${node_ip} 'mkdir -p /opt/k8s/bin && chown -R k8s /opt/k8s && mkdir -p /etc/kubernetes/cert && chown -R k8s /etc/kubernetes'
    ssh root@${node_ip} 'mkdir -p /etc/etcd/cert && chown -R k8s /etc/etcd/cert && mkdir -p /var/lib/etcd && chown -R k8s /var/lib/etcd'
done
EOF

15,分发环境变量配置

cat > magic.sh << "EOF"
#!/bin/bash

source /opt/k8s/bin/environment.sh

for node_ip in ${NODE_IPS[@]}
do
    echo ">>> ${node_ip}"
    scp /opt/k8s/bin/environment.sh k8s@${node_ip}:/opt/k8s/bin/
    ssh k8s@${node_ip} "chmod +x /opt/k8s/bin/*"
done
EOF

16,参考

  • 系统内核相关参数参考:https://docs.openshift.com/enterprise/3.2/admin_guide/overcommit.html

weinxin
扫码订阅本站,第一时间获得更新
微信扫描二维码,订阅我们网站的动态,另外不定时发送WordPress小技巧,你可以随时退订,欢迎订阅哦~

二丫讲梵 , 版权所有丨如未注明 , 均为原创丨本网站采用BY-NC-SA协议进行授权 , 转载请注明<十八>Kubernetes学习笔记-手动搭建k8s-1.10.4之系统初始化
喜欢 (0)
[如果想支持本站,可支付宝赞助]
分享 (0)
eryajf
关于作者:
学无止境,我愿意无止境学。书山有路,我愿意举身投火,淬炼成金!永远不要忘记,激情的奋进,就是美好的未来!

您必须 登录 才能发表评论!

(2)个小伙伴在吐槽
  1. :grin: 国庆把k8s啃完。
    cpc2019-09-25 20:02 Windows 10 | Chrome 77.0.3865.90