nobug
首页 标签 归档

Kubernetes 1.6.3 安装GlusterFS+Heketi记录

2021-08-25

环境说明

1
2
3
4
5
6
7
Kubernetes：1.6.3
Heketi: 9
GlusterFS: latest(4.1.7)
节点：
node-01 172.22.101.111 /dev/sdb 2H4G
node-02 172.22.101.112 /dev/sdb 2H4G
node-03 172.22.101.113 /dev/sdb 2H4G

准备部署文件

使用 heketi 项目中的部署文件 glusterfs，下载以下文件到本地

1
2
3
4
5
6
wget https://raw.githubusercontent.com/heketi/heketi/master/extras/kubernetes/glusterfs-daemonset.json
wget https://raw.githubusercontent.com/heketi/heketi/master/extras/kubernetes/heketi-bootstrap.json
wget https://raw.githubusercontent.com/heketi/heketi/master/extras/kubernetes/heketi-deployment.json
wget https://raw.githubusercontent.com/heketi/heketi/master/extras/kubernetes/heketi-service-account.json
wget https://raw.githubusercontent.com/heketi/heketi/master/extras/kubernetes/heketi.json
wget https://raw.githubusercontent.com/heketi/heketi/master/extras/kubernetes/topology-sample.json

部署 glusterfs

在上面下载的文件中，glusterfs-daemonset.json 就是用于部署 glusterfs 的配置文件，将 glusterfs 作为 daemonset 的方式运行。可以通过将指定 stoargenode=glusterfs 标签来选择用于部署 glusterfs 的节点：

1
2
3
kubectl label node node-01 storagenode=glusterfs
kubectl label node node-02 storagenode=glusterfs
kubectl label node node-03 storagenode=glusterfs

在本集群中，下面用到的 daemonset 控制器及后面用到的 deployment 控制器的 api 版本均变为了 apps/v1，所以需要手动修改下载的 json 文件再进行部署，资源编排文件中需要指定 selector 声明。避免出现以下报错：

1
2
root@ubuntu heketi % kubectl create -f glusterfs-daemonset.json
error: unable to recognize "glusterfs-daemonset.json": no matches for kind "DaemonSet" in version "extensions/v1beta1"

修改 api 版本

1
"apiVersion": "extensions/v1beta1"

为 apps/v1

1
"apiVersion": "apps/v1",

报 selector 错

1
2
root@ubuntu heketi % kubectl create -f glusterfs-daemonset.json
error: error validating "glusterfs-daemonset.json": error validating data: ValidationError(DaemonSet.spec): missing required field "selector" in io.k8s.api.apps.v1.DaemonSetSpec; if you choose to ignore these errors, turn validation off with --validate=false

指定 selector 声明，spec 下面添加 selector

1
2
3
4
5
6
7
8
9
......
"spec": {
        "selector": {
            "matchLabels": {
                "glusterfs-node": "daemonset"
            }
        },
        "template":
......
1
2
[root@ubuntu heketi]# kubectl create -f glusterfs-daemonset.json
error: error validating "glusterfs-daemonset.json": error validating data: ValidationError(DaemonSet.spec): missing required field "selector" in io.k8s.api.apps.v1.DaemonSetSpec; if you choose to ignore these errors, turn validation off with --validate=false

部署

1
kubectl create -f glusterfs-daemonset.json

部署 heketi server 端

部署 heketi 之前，需要先为 heketi 创建 serviceaccount

1
kubectl create -f heketi-service-account.json

然后为该 serviceaccount 授权，为其绑定相应的权限来控制 gluster 的 pod，执行如下操作

1
kubectl create clusterrolebinding heketi-gluster-admin --clusterrole=edit --serviceaccount=default:heketi-service-account

接着，创建一个 Kubernetes secret 来保存我们 Heketi 实例的配置。必须将配置文件的执行程序设置为 kubernetes 才能让 Heketi server 控制 gluster pod。

heketi.json 的配置修改如下

1
2
3
4
5
6
7
8
9
10
11
12
......
#打开认证
"use_auth": true,
......
#修改admin用户的key
"key": "adminkey"
......
#修改执行插件为kubernetes
"executor": "kubernetes",
......
#备份heketi数据库
"backup_db_to_kube_secret": true

执行如下操作，将 heketi.json 创建为 kubernetes 的 secret

1
kubectl create secret generic heketi-config-secret --from-file=./heketi.json

接着部署 heketi 的运行容器，配置文件为 heketi-bootstrap.json，需要修改 image 为heketi/heketi:9
修改"apiVersion": "extensions/v1beta1"为"apiVersion": "apps/v1"
spec 下面添加 selector

1
2
3
4
5
6
7
8
......
"spec": {
    "selector": {
        "matchLabels": {
            "name": "deploy-heketi"
        }
    },
......

执行

1
kubectl create -f heketi-bootstrap.json

通过创建 ingress 的方式将 heketi 暴露出来，vim heketi.ingress.yaml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: traefik
  name: heketi
spec:
  rules:
  - host: heketi.default.172.22.101.113.xip.io
    http:
      paths:
      - path: /
        backend:
          serviceName: deploy-heketi
          servicePort: 8080
1
kubectl create -f hekti.ingress.yaml

以上完成 heketi server 端部署

配置 heketi client 端

需要说明的是，heketi 的客户端版本需要与 server 端对应，server 端我们使用的是 9.0，所以客户端也需要下载 9.0 版本

1
wget https://github.com/heketi/heketi/releases/download/v9.0.0/heketi-client-v9.0.0.linux.amd64.tar.gz

修改 topology-sample.json 文件，如下

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
{
  "clusters": [
    {
      "nodes": [
        {
          "node": {
            "hostnames": {
              "manage": [
                "node-01"
              ],
              "storage": [
                "172.22.101.111"
              ]
            },
            "zone": 1
          },
          "devices": [
            {
              "name": "/dev/sdb"
            }
          ]
        },
        {
          "node": {
            "hostnames": {
              "manage": [
                "node-02"
              ],
              "storage": [
                "172.22.101.112"
              ]
            },
            "zone": 1
          },
          "devices": [
            {
              "name": "/dev/sdb"
            }
          ]
        },
        {
          "node": {
            "hostnames": {
              "manage": [
                "node-03"
              ],
              "storage": [
                "172.22.101.113"
              ]
            },
            "zone": 1
          },
          "devices": [
            {
              "name": "/dev/sdb"
            }
          ]
        }
      ]
    }
  ]
}

修改node.hostnames.manage为主机名，storage为主机 ip，修改devices为挂载设备

初始化集群

创建集群前先设置环境变量

1
2
3
export HEKETI_CLI_SERVER=http://heketi.default.172.22.101.111.xip.io
export HEKETI_CLI_USER=admin
export HEKETI_CLI_KEY=adminkey
1
[root@ubuntu heketi]# ./heketi-cli  topology load --json=topology-sample.json

加载模块，在每台 glusterfs 节点上运行

1
2
3
4
5
modprobe dm_thin_pool
#不加载可能会报下面的错
[root@ubuntu heketi]# ./heketi-cli setup-openshift-heketi-storage
Error: /usr/sbin/modprobe failed: 1
  thin: Required device-mapper target(s) not detected in your kernel.

生成 heketi-storage.json 文件

1
2
[root@ubuntu heketi]# ./heketi-cli setup-openshift-heketi-storage
Saveing heketi-storage.json

修改 heketi-storage.json 镜像

1
"image": "heketi/heketi:dev" 为 "heketi/heketi:9"

执行

1
kubectl create -f heketi-storage.json

删除之前用于部署的 heketi

1
kubectl delete all,service,jobs,deployment,secret --selector="deploy-heketi"

部署 heketi，修改 heketi-deployment.json 中 image，添加 selector

1
2
3
4
5
6
7
8
9
10
......
"image": "heketi/heketi:9",
......
"spec": {
      	"selector": {
		    "matchLabels": {
		        "name": "heketi"
		    }
		},
......

查看集群列表，复制 id，用于storageclass-clusterid

1
2
3
[root@ubuntu heketi]# ./heketi-cli cluster list
Clusters:
Id:6dd0c671e45735a9c502b216ca2138b9 [file][block]

新建storageclass.yml

1
2
3
4
5
6
7
8
9
10
11
12
---
kind: StorageClass
apiVersion: storage.k8s.io/v1beta1
metadata:
  name: gluster-heketi
provisioner: kubernetes.io/glusterfs
parameters:
  resturl: "http://heketi-ip:8080"
  restuser: "admin"
  restuserkey: "adminkey"
  restauthenabled: "true"
  clusterid: "6dd0c671e45735a9c502b216ca2138b9"

创建存储类

1
kubectl create -f storageclass.yml

以上 glusterfs 部署完成

测试部署

创建 gluster-pvc.yml 文件，用于创建 pvc

1
2
3
4
5
6
7
8
9
10
11
12
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
 name: gluster1
 annotations:
   volume.beta.kubernetes.io/storage-class: gluster-heketi
spec:
 accessModes:
  - ReadWriteOnce
 resources:
   requests:
     storage: 1Gi
1
kubectl create -f gluster-pvc.yml

查看 PVC

1
2
3
kubectl get pvc
NAME       STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
gluster1   Bound    pvc-4aa45417-e35f-4a44-8360-3aa2fd5990e0   1Gi        RWO            gluster-heketi           8m39s

创建 nginx-pod.yml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
apiVersion: v1
kind: Pod
metadata:
  name: nginx-pod1
  labels:
    name: nginx-pod1
spec:
  containers:
  - name: nginx-pod1
    image: gcr.io/google_containers/nginx-slim:0.8
    ports:
    - name: web
      containerPort: 80
    volumeMounts:
    - name: gluster-vol1
      mountPath: /usr/share/nginx/html
  volumes:
  - name: gluster-vol1
    persistentVolumeClaim:
      claimName: gluster1
1
kubectl create -f nginx-pod.yml

查看 pod

1
kubectl get pods -o wide

启动成功后添加 index.html 到 nginx 容器中

1
2
3
4
5
6
7
kubectl exec -ti nginx-pod1 /bin/sh
$ cd /usr/share/nginx/html
$ echo 'Hello World from GlusterFS!!!' > index.html
$ ls
index.html
$ curl http://10.38.0.0
Hello World from GlusterFS!!!

最后检查 glusterfs pod 中挂载文件是否一致

1
2
3
4
5
6
7
8
9
10
kubectl exec -ti glusterfs-5ccwd /bin/sh
$ mount | grep heketi
/dev/xvda1 on /var/lib/heketi type ext4 (rw,relatime,discard,data=ordered)
/dev/mapper/vg_3e5321815938a95cba981bec1d81c4b0-brick_a3df6570e7a9d94739a4fe6d6491626d on /var/lib/heketi/mounts/vg_3e5321815938a95cba981bec1d81c4b0/brick_a3df6570e7a9d94739a4fe6d6491626d type xfs (rw,noatime,nouuid,attr2,inode64,logbsize=128k,sunit=256,swidth=512,noquota)
/dev/mapper/vg_3e5321815938a95cba981bec1d81c4b0-brick_f13e4577ad6ac6e22ec68f65a70908e0 on /var/lib/heketi/mounts/vg_3e5321815938a95cba981bec1d81c4b0/brick_f13e4577ad6ac6e22ec68f65a70908e0 type xfs (rw,noatime,nouuid,attr2,inode64,logbsize=128k,sunit=256,swidth=512,noquota)
cd /var/lib/heketi/mounts/vg_3e5321815938a95cba981bec1d81c4b0/brick_f13e4577ad6ac6e22ec68f65a70908e0/brick
$ ls
index.html
$ cat index.html
Hello World from GlusterFS!!!

参考

https://github.com/heketi/heketi
https://github.com/gluster/gluster-kubernetes/
https://www.cnblogs.com/breezey/p/9589047.html
https://www.cnblogs.com/ssgeek/p/11725648.html
https://blog.csdn.net/q1403539144/article/details/86614346

updated at 2023-04-07

Previous Post
群晖 Drive 忽略文件夹
Next Post
黑苹果启用hidpi