大家好，我是小白。下面由我给白白们讲解一下云计算火热的容器编排之王Kubernetes…

上文咱们讲解了Linux容器技术docker，有些基础后理解Kubernetes后就更好理解啦~
简言之呢，docker虽然帮助我们实现了容器化，但仅仅是个使用工具，而我们并不仅仅只需要运行一个容器就能满足我们的需求，我们需要自动化部署，弹性伸缩，负载均衡，服务自动发现等等云原生的一整套的生态化需求，而这就是kubernetes能帮我们做到的。

一、何为Kubernetes

Kubernetes 是谷歌开源的容器集群管理系统，是 Google 多年大规模容器管理技术 Borg 的开源版本，主要功能包括：
基于容器的应用部署、维护和滚动升级
负载均衡和服务发现
跨机器和跨地区的集群调度
自动伸缩
无状态服务和有状态服务
广泛的 Volume 支持
插件机制保证扩展性
Kubernetes 发展非常迅速，已经成为容器编排领域的领导者。

Read More

背景: 虚拟化openstack测试使用nvmf
配置 NVMe/RDMA 客户端

安装mlnx_nvme-ofed driver

1
2

下载 MLNX_OFED_LINUX-5.4-1.0.3.0-rhel8.1-x86_64.iso
安装 kmod-mlnx-nvme-5.4-OFED.5.4.1.0.3.1.rhel8u1.x86_64

1

# yum install nvme-cli

1

# modprobe nvme-rdma

troubleshooting

1
2
3
4
5

# nvme connect -t rdma -n nqn.2021-10.io.spdk:cnode2 -a 10.224.129.226 -s 4420
Failed to open /dev/nvme-fabrics: No such file or directory

解决:
#modprobe nvme_fabrics

1
2
3
4
5
6
7

Failed to write to /dev/nvme-fabrics: Invalid argument
nvme_fabrics: no handler found for transport rdma.
解决: 安装mlx_ofd drvier
https://www.mellanox.com/products/infiniband-drivers/linux/mlnx_ofed
下载 MLNX_OFED_LINUX-5.4-1.0.3.0-rhel8.1-x86_64.iso
安装 kmod-mlnx-nvme-5.4-OFED.5.4.1.0.3.1.rhel8u1.x86_64

环境 : CentOS 8.1 x86_64

kata 2 做了很多优化, 故使用kata 2 版本, 操作系统选用高内核, CentOS 8 (社区CentOS 7只支持kata 1) , 同时操作系统开启 ipv6

一、部署kubernetes

使用 kubespray 部署生产kubernetes集群 , CNI插件使用了kube-ovn

遇到网络问题，需离线下载镜像,以及确保能访问github(github加速)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

images=(
  kube-apiserver:v1.18.10
  kube-controller-manager:v1.18.10
  kube-scheduler:v1.18.10
  kube-proxy:v1.18.10
  pause:3.2
  k8s-dns-node-cache:1.15.13
  cluster-proportional-autoscaler-amd64:1.8.1
)

for imageName in ${images[@]} ; do
    docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/$imageName
    docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/$imageName k8s.gcr.io/$imageName
    docker rmi registry.cn-hangzhou.aliyuncs.com/google_containers/$imageName
done

kubernetes部署步骤参考kubespray官网即可略

二、安装kata 2.0

2.1 安装kata 组件

Install the Kata Containers components with the following commands:

1
2
3
4
5
6
7
8
9
10
11
12

$ sudo -E dnf install -y centos-release-advanced-virtualization
$ sudo -E dnf module disable -y virt:rhel
$ source /etc/os-release
$ cat <<EOF | sudo -E tee /etc/yum.repos.d/kata-containers.repo
  [kata-containers]
  name=Kata Containers
  baseurl=http://mirror.centos.org/$contentdir/$releasever/virt/$basearch/kata-containers
  enabled=1
  gpgcheck=1
  skip_if_unavailable=1
  EOF
$ sudo -E dnf install -y kata-containers

2.2 检测硬件是否支持 Kata

Kata 对硬件的要求需要满足以下任意条件：

• Intel VT-x technology.
• ARM Hyp mode (virtualization extension).
• IBM Power Systems.
• IBM Z mainframes.

安装完 kata-runtime 之后，执行检测命令：

1
2
3
4

#kata-runtime kata-check

System is capable of running Kata Containers
System can currently create Kata Containers

这里的输出表示，运行环境支持 Kata Containers 。

2.3 配置集成kubelet

• 新增配置文件

1
2
3
4
5

mkdir -p  /etc/systemd/system/kubelet.service.d/
cat << EOF | sudo tee  /etc/systemd/system/kubelet.service.d/0-containerd.conf
[Service]                                                 
Environment="KUBELET_EXTRA_ARGS=--container-runtime=remote --runtime-request-timeout=15m --container-runtime-endpoint=unix:///run/containerd/containerd.sock"
EOF

• 重启生效

1
2

systemctl daemon-reload
systemctl restart kubelet

这里使用的是 containerd 。如果使用 CRI-O ，配置会不一样。

2.5 给 Kubernetes 提供 kata-runtime

通过直接创建 Container 可以使用 kata-runtime 。但在集群中，我们该如何告诉 Kubernetes 哪些负载需要使用 kata-runtime 呢？根据不同的版本，Kata 提供了不同的方式。

首先都需要生成 containerd 配置文件

1

containerd config default > /etc/containerd/config.toml

• RuntimeClass 的方式

这种方式对相关组件版本有要求：

1
2
3

Kata Containers v1.5.0 or above (including 1.5.0-rc)
Containerd v1.2.0 or above
Kubernetes v1.12.0 or above

我所使用的是

kubernetes: 1.18.10

containerd: 1.2.13

kata : 1.10

在 config.toml 配置文件中，“no_pivot = false”下增加如下内容：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27

[plugins.cri.containerd.runtimes]
  [plugins.cri.containerd.runtimes.runc]
     runtime_type = "io.containerd.runc.v1"
     [plugins.cri.containerd.runtimes.runc.options]
       NoPivotRoot = false
       NoNewKeyring = false
       ShimCgroup = ""
       IoUid = 0
       IoGid = 0
       BinaryName = "runc"
       Root = ""
       CriuPath = ""
       SystemdCgroup = false
  [plugins.cri.containerd.runtimes.kata]
     runtime_type = "io.containerd.kata.v2"
  [plugins.cri.containerd.runtimes.katacli]
     runtime_type = "io.containerd.runc.v1"
     [plugins.cri.containerd.runtimes.katacli.options]
       NoPivotRoot = false
       NoNewKeyring = false
       ShimCgroup = ""
       IoUid = 0
       IoGid = 0
       BinaryName = "/usr/bin/kata-runtime"
       Root = ""
       CriuPath = ""
       SystemdCgroup = false

这里 [plugins.cri.containerd.runtimes.kata] 中的 kata 将被作为 RuntimeClass handler 关键字。

• 使用 untrusted_workload_runtime 的方式

对于不符合上述版本要求的环境，可以使用之前的方式。

在配置文件中新增如下内容：

1
2
3

[plugins.cri.containerd.untrusted_workload_runtime]
  runtime_type = "io.containerd.runtime.v1.linux"
  runtime_engine = "/usr/bin/kata-runtime"

最后，都需要重启 containerd。

1
2

#systemctl daemon-reload
#systemctl restart containerd

三、使用kata-runtime

3.1 RuntimeClass 方式

• 创建 RuntimeClass

kata-runtime.yaml

1
2
3
4
5

kind: RuntimeClass
apiVersion: node.k8s.io/v1beta1
metadata:
  name: kata-containers
handler: kata

#kubectl create -f kata-runtime.yaml

也可以为 runc 创建 RuntimeClass

1
2
3
4

kubectl get runtimeclass

NAME              CREATED AT
kata-containers   2020-08-30

• 创建负载 kata-pod.yaml

1
2
3
4
5
6
7
8
9
10
11
12

apiVersion: v1
kind: Pod
metadata:
  name: kata-nginx
spec:
  runtimeClassName: kata-containers
  containers:
  - name: nginx
    image: nginx
    ports:
    - containerPort: 80

#kubectl apply -f kata-pod.yaml

• 查看负载

1

kata-runtime list

检查hypervisor是否运行

1

ps axu|grep qemu

国内访问github太慢，甚至超时，咋办？

第一步：找 IP
首先找到 github.com 的 IP 地址，可以在 IPAddress 网站上查找。还可以添加 github.global.ssl.fastly.net 的映射，这个有助于加速。

注意：隔段时间地址会变，网上找的有些都过期了，建议还是自己去查询一下。

第二步：修改 host
vim /etc/hosts
将找到的最新地址映射添加到 hosts 中。

亲测,确实有效果噢~

Post describes procedure to disable IPv6 on CentOS/RHEL 7. There are 2 ways to do this :

1. Disable IPv6 in kernel module (requires reboot)
2. Disable IPv6 using sysctl settings (no reboot required)

To verify if IPv6 is enabled or not, execute :

1

# ifconfig -a | grep inet6

    inet6 fe80::211:aff:fe6a:9de4  prefixlen 64  scopeid 0x20
    inet6 ::1  prefixlen 128  scopeid 0x10[host]

1. Disable IPv6 in kernel module (requires reboot)
2. Edit /etc/default/grub and add ipv6.disable=1 in line GRUB_CMDLINE_LINUX, e.g.:

1

# cat /etc/default/grub

GRUB_TIMEOUT=5
GRUB_DEFAULT=saved
GRUB_DISABLE_SUBMENU=true
GRUB_TERMINAL_OUTPUT=”console”
GRUB_CMDLINE_LINUX=”ipv6.disable=1 crashkernel=auto rhgb quiet”
GRUB_DISABLE_RECOVERY=”true”
2. Regenerate a GRUB configuration file and overwrite existing one:

1

# grub2-mkconfig -o /boot/grub2/grub.cfg

3. Restart system and verify no line “inet6” in “ip addr show” command output.

   1 2	# shutdown -r now # ip addr show | grep net6

4. Disable IPv6 using sysctl settings (no reboot required)
5. Append below lines in /etc/sysctl.conf:

net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1
NOTE : To disable IPv6 on a single interface add below lines to /etc/sysctl.conf :
net.ipv6.conf.[interface].disable_ipv6 = 1 ### put interface name here [interface]
net.ipv6.conf.default.disable_ipv6 = 1
2. To make the settings affective, execute :

1

# sysctl -p

NOTE : make sure the file /etc/ssh/sshd_config contains the line AddressFamily inet to avoid breaking SSH Xforwarding if you are using the sysctl method
3. Add the AddressFamily line to sshd_config :

1

# vi /etc/ssh/sshd_config

….
AddressFamily inet
….
Restart sshd for changes to get get effect :

1

# systemctl restart sshd

关于kata的介绍，网上资料很多，过大的官方话就不说了，下面说一些白话一点的，让小白能快速了解的。

1. 首先kata是什么

kata 本质上是和runc 平级的，不同的是runc启动的是容器，而kata 启动的是hypervisor(qemu-kvm/firecraker等)，因为公有云上安全性问题，所以衍生出了kata

2. kata 架构是什么样子的

kata 为了实现安全性，在容器外层包了一层虚拟化hypervisor(这里以qemu-kvm为例)，从而实现了隔离，安全性更高

基于CentOS 8 x86_64编译安装libvirt 7.2

1
2
3
4
5
6
7
8

#dnf --enablerepo=PowerTools install gcc cmake libxslt-devel
#dnf --enablerepo=PowerTools install meson rpcgen
#wget https://libvirt.org/sources/libvirt-7.2.0.tar.xz
#tar xvf libvirt-7.2.0.tar.xz
#cd libvirt-7.2.0
#meson build --prefix=/usr
#ninja -C build
#ninja -C build install

1

#libvirtd --version

启动测试

1

/usr/sbin/libvirtd --listen

报错

1
2
3
4
5

2021-09-28 04:27:03.665+0000: 41728: info : libvirt version: 7.2.0
2021-09-28 04:27:03.665+0000: 41728: info : hostname: openstack160.********
2021-09-28 04:27:03.665+0000: 41728: error : virNetTLSContextCheckCertFile:110 : Cannot read CA certificate '/etc/pki/CA/cacert.pem': No such file or directory

先临时修改了/etc/libvirt/libvirtd.conf 关闭tls

1
2
3
4
5

error: Failed to start domain 'instance-00000018'
error: can't connect to virtlogd: Failed to connect socket to '/var/run/libvirt/virtlogd-sock': No such file or directory

解决 ：virtlogd &

Troubleshooting

1. ERROR: Program ‘rpcgen portable-rpcgen’ not found

1

yum install rpcgen

2. ERROR: Dependency “gnutls” not found, tried pkgconfig

1

yum install gnutls-devel

3. ERROR: Dependency “libxml-2.0” not found, tried pkgconfig

1

yum install libxml2-devel

4. ERROR: Problem encountered: XDR is required for remote driver

1

yum install libtirpc-devel

5. meson.build:918:2: ERROR: Program ‘rst2html5 rst2html5.py rst2html5-3’ not found

   1	pip3 install rst2html5

6. ERROR: Dependency “libapparmor” not found, tried pkgconfig and cmake

   1 2 3

   wget https://rpmfind.net/linux/opensuse/distribution/leap/15.3/repo/oss/x86_64/libapparmor1-2.13.6-1.24.x86_64.rpm wget https://ftp.lysator.liu.se/pub/opensuse/distribution/leap/15.3/repo/oss/x86_64/libapparmor-devel-2.13.6-1.24.x86_64.rpm

7. ERROR: C shared or static library ‘attr’ not found

   1	dnf --enablerepo=powertools install libattr-devel

8. ERROR: C shared or static library ‘audit’ not found

   1	dnf --enablerepo=powertools install audit-libs audit-libs-devel

9. ERROR: Dependency “bash-completion” not found, tried pkgconfig and cmake

   1	dnf --enablerepo=powertools install bash-completion

10. ERROR: C shared or static library ‘cap-ng’ not found

    1	dnf --enablerepo=powertools install libcap-ng-devel

11. ERROR: Dependency “libcurl” not found, tried pkgconfig and cmake

    1	dnf --enablerepo=powertools install libcurl-devel

12. ERROR: Dependency “libfuse glusterfs-api libiscsi libnl pcap-config libssh libssh2 netcf numa openwsman pciaccess readline libsasl2 libudev wireshark yajl” not found, tried pkgconfig and cmake

    1	dnf --enablerepo=powertools install fuse-devel glusterfs-api-devel libiscsi-devel libnl3-devel libssh-devel libssh2-devel netcf-devel numactl-devel libwsman-devel libpciaccess-devel readline-devel cyrus-sasl-devel systemd-devel wireshark-devel yajl-devel

因原openstack 使用qemu 版本为2.0 太低，考虑升级qemu

1
2

#docker pull centos:8
#docker run --name spdk-vhost --privileged=true --network=host -v /opt/tmp:/opt/tmp -it centos:8 bash

Read More

qemu 启动基于vhost-user disk虚拟机 , 报错

1

(process:96671): GLib-WARNING **: 15:17:23.577: gmem.c:489: custom memory allocation vtable not supported

启动命令

Read More