hello云胜

技术与生活

0%

发表于

使用ROOK部署Ceph集群

前置条件

需要服务器有裸盘

1
# lsblk -f

image-20210727135518965

裸盘，没有分区没有文件系统。

安装lvm2，ceph的osd需要依赖lvm2。k8s集群所有节点安装lvm2

1
yum install -y lvm2

准备rook包

1
2
3
wget https://github.com/rook/rook/archive/refs/tags/v1.6.8.tar.gz
tar -zxvf v1.6.8.tar.gz
cd rook-1.6.8/cluster/examples/kubernetes/ceph

我们需要的yaml都已经写好，放在rook-1.6.8/cluster/examples/kubernetes/ceph目录下

修改operator.yaml

CSI_PLUGIN需要在所有节点上启动，master默认被打上了NoSchedule的taint，需要在operator上面添加容忍

1
2
3
4
5
6
7
8
# vim operator.yaml
  CSI_PLUGIN_TOLERATIONS: |
    - effect: NoSchedule
      key: node-role.kubernetes.io/master
      operator: Exists
    - effect: NoExecute
      key: node-role.kubernetes.io/etcd
      operator: Exists

部署operator

1
kubectl create -f crds.yaml -f common.yaml -f operator.yaml

顺序不能变，crds.yaml和 common.yaml 必须在operator.yaml之前创建

查看pod状态

1
2
3
[root@paas-m-k8s-master-1 ceph]# kubectl -n rook-ceph get pod
NAME                                  READY   STATUS    RESTARTS   AGE
rook-ceph-operator-674c87d477-j8xwp   1/1     Running   0          5m34s

部署ceph cluster

1
kubectl create -f cluster.yaml

查看pod状态,发现很多失败。查看原因，是pull image失败。

1
2
3
4
5
awaiting headers)
  Normal   BackOff  22m (x4138 over 19h)    kubelet, paas-m-k8s-node-2  Back-off pulling image "k8s.gcr.io/sig-storage/csi-snapshotter:v4.1.1"
  Normal   BackOff  12m (x4176 over 19h)    kubelet, paas-m-k8s-node-2  Back-off pulling image "k8s.gcr.io/sig-storage/csi-resizer:v1.2.0"
  Normal   BackOff  7m38s (x4197 over 19h)  kubelet, paas-m-k8s-node-2  Back-off pulling image "k8s.gcr.io/sig-storage/csi-provisioner:v2.2.2"

describe看是从quay.io下载镜像。我们的网络即使配置了镜像加速也下载不了quay.io的image。

手动从quay.io下载镜像，传到k8s集群的所有主机上。

方法是可以借助阿里云构建镜像，然后从aliyun拉取镜像，再改tag

先去github建个仓库，里面建一个文件Dockerfile,就放一句FROM

1
FROM k8s.gcr.io/sig-storage/csi-provisioner:v2.2.2

然后使用阿里云的容器镜像服务，建空间，建仓库

仓库挂载上面创建github的仓库，注意选上海外机器构建

img

添加规则

img

master是我的分支的名字

img

成功后可以拉镜像了

mobaXterm有个批量执行命令的功能

在这里插入图片描述

在所有节点执行命令

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
docker login --username=yangyunsheng1989@126.com registry.cn-hangzhou.aliyuncs.com
docker pull registry.cn-hangzhou.aliyuncs.com/myk8s123/k8simg:v2.2.0
docker tag registry.cn-hangzhou.aliyuncs.com/myk8s123/k8simg:v2.2.0 k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.2.0
docker rmi registry.cn-hangzhou.aliyuncs.com/myk8s123/k8simg:v2.2.0


docker pull registry.cn-hangzhou.aliyuncs.com/myk8s123/k8simg:v3.2.1
docker tag registry.cn-hangzhou.aliyuncs.com/myk8s123/k8simg:v3.2.1 k8s.gcr.io/sig-storage/csi-attacher:v3.2.1
docker rmi registry.cn-hangzhou.aliyuncs.com/myk8s123/k8simg:v3.2.1

docker pull registry.cn-hangzhou.aliyuncs.com/myk8s123/k8simg:v4.1.1
docker tag registry.cn-hangzhou.aliyuncs.com/myk8s123/k8simg:v4.1.1 k8s.gcr.io/sig-storage/csi-snapshotter:v4.1.1
docker rmi registry.cn-hangzhou.aliyuncs.com/myk8s123/k8simg:v4.1.1


docker pull registry.cn-hangzhou.aliyuncs.com/myk8s123/k8simg:v1.2.0
docker tag registry.cn-hangzhou.aliyuncs.com/myk8s123/k8simg:v1.2.0 k8s.gcr.io/sig-storage/csi-resizer:v1.2.0
docker rmi registry.cn-hangzhou.aliyuncs.com/myk8s123/k8simg:v1.2.0

docker pull registry.cn-hangzhou.aliyuncs.com/myk8s123/k8simg:v2.2.2
docker tag registry.cn-hangzhou.aliyuncs.com/myk8s123/k8simg:v2.2.2 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.2
docker rmi registry.cn-hangzhou.aliyuncs.com/myk8s123/k8simg:v2.2.2

验证ceph的状态

1
2
3
kubectl create -f toolbox.yaml
#进入toolbox pod
kubectl -n rook-ceph exec -it $(kubectl -n rook-ceph get pod -l "app=rook-ceph-tools" -o jsonpath='{.items[0].metadata.name}') -- bash
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
[root@rook-ceph-tools-79fcdcf697-8nk7m /]# ceph status
  cluster:
    id:     c1797d91-e3ec-488c-bd59-32773a34cc1d
    health: HEALTH_OK

  services:
    mon: 3 daemons, quorum a,b,c (age 20h)
    mgr: a(active, since 20h)
    osd: 6 osds: 6 up (since 20h), 6 in (since 20h)   #集群内总共有几块裸盘就应该有几个osd。本例有6个node节点

  data:
    pools:   1 pools, 1 pgs
    objects: 0 objects, 0 B
    usage:   6.0 GiB used, 2.9 TiB / 2.9 TiB avail
    pgs:     1 active+clean

查看之前的裸盘

1
2
3
4
5
6
7
[root@paas-m-k8s-node-1 ~]# lsblk -f
NAME                                                                                       FSTYPE      LABEL UUID                                   MOUNTPOINT
vda
└─vda1                                                                                     ext4              207b19eb-8170-4983-acb5-9098af381e72   /
vdb                                                                                        LVM2_member       6nWGCq-SdMg-FdqH-fpFn-R6kc-T4gN-JKRjmJ
└─ceph--d4b6fe80--9e55--479a--874f--7fc49cf0ca2e-osd--block--f66a2af8--3a00--4bf1--8bde--be149f4d2e31

创建storageClass

1
2
3
4
5
6
[root@paas-m-k8s-master-1 rbd]# pwd
/root/rook/rook-1.6.8/cluster/examples/kubernetes/ceph/csi/rbd
[root@paas-m-k8s-master-1 rbd]# kubectl apply -f storageclass.yaml
cephblockpool.ceph.rook.io/replicapool created
storageclass.storage.k8s.io/rook-ceph-block created

注意这里进入的目录是rbd。也就是块存储，ceph还提供了文件存储的storage-class，是在cephfs目录下

rbd相较于cephfs性能要更优，但是不支持多mount，也就是每个pvc只能被一个pod挂载。

cephfs在读写大文件时性能比较优秀，读写小文件时性能较差，但是其支持多mount，所以在需要多个pod共享存储时就需要选用cephfs作为storage-class

1
2
3
4
5

[root@paas-m-k8s-master-1 ~]# kubectl get sc
NAME                   PROVISIONER                                       RECLAIMPOLICY   VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION   AGE
nfs-client (default)   cluster.local/nfs-client-nfs-client-provisioner   Delete          Immediate           true                   15d
rook-ceph-block        rook-ceph.rbd.csi.ceph.com                        Delete          Immediate           true                   174m

设置为默认storageclass

1
[root@d-paas-k8s-master-0 rbd]# kubectl patch storageclass rook-ceph-block -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'

创建pvc 验证csi

1
2
3
[root@paas-m-k8s-master-1 rbd]# pwd
/root/rook/rook-1.6.8/cluster/examples/kubernetes/ceph/csi/rbd
[root@paas-m-k8s-master-1 rbd]# kubectl apply -f pvc.yaml

查看在default namespace下是否生成了name为rbd-pvc的pvc，status为bound

1
2
3
[root@paas-m-k8s-master-1 rbd]# kubectl get pvc
NAME                                      STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS      AGE
rbd-pvc                                   Bound    pvc-4ef98f5f-462e-4834-bf3c-bd96f9b32b94   1Gi        RWO            rook-ceph-block   2m9s

验证完成删除rdb-pvc

1
2
[root@paas-m-k8s-master-1 rbd]# kubectl delete pvc rbd-pvc
persistentvolumeclaim "rbd-pvc" deleted

查看Dashboard

ceph提供了一个控制台服务。默认已经是开启的。

1
2
3
4
5
6
7
8
9
[root@paas-m-k8s-master-1 rbd]# kubectl get svc -n rook-ceph
NAME                       TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)             AGE
csi-cephfsplugin-metrics   ClusterIP   10.110.141.175   <none>        8080/TCP,8081/TCP   23h
csi-rbdplugin-metrics      ClusterIP   10.104.228.180   <none>        8080/TCP,8081/TCP   23h
rook-ceph-mgr              ClusterIP   10.102.85.146    <none>        9283/TCP            23h
rook-ceph-mgr-dashboard    ClusterIP   10.110.80.91     <none>        8443/TCP            23h
rook-ceph-mon-a            ClusterIP   10.110.117.117   <none>        6789/TCP,3300/TCP   23h
rook-ceph-mon-b            ClusterIP   10.102.192.7     <none>        6789/TCP,3300/TCP   23h
rook-ceph-mon-c            ClusterIP   10.98.8.230      <none>        6789/TCP,3300/TCP   23h

rook-ceph-mgr是暴露给Prometheus收集用的，rook-ceph-mgr-dashboard就是控制台服务。

然后创建个service暴露给集群外访问。dashboard-external-https.yaml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
apiVersion: v1
kind: Service
metadata:
  name: rook-ceph-mgr-dashboard-external-https
  namespace: rook-ceph
  labels:
    app: rook-ceph-mgr
    rook_cluster: rook-ceph
spec:
  ports:
  - name: dashboard
    port: 8443
    protocol: TCP
    targetPort: 8443
  selector:
    app: rook-ceph-mgr
    rook_cluster: rook-ceph
  sessionAffinity: None
  type: NodePort

其实这个yaml已经存在了。/root/rook/rook-1.6.8/cluster/examples/kubernetes/ceph下就有

1
2
3
4
5
6
7
8
9
10
11
12
[root@paas-m-k8s-master-1 ceph]# kubectl apply -f dashboard-external-https.yaml
service/rook-ceph-mgr-dashboard-external-https created
[root@paas-m-k8s-master-1 ceph]# kubectl -n rook-ceph get service
NAME                                     TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)             AGE
csi-cephfsplugin-metrics                 ClusterIP   10.110.141.175   <none>        8080/TCP,8081/TCP   23h
csi-rbdplugin-metrics                    ClusterIP   10.104.228.180   <none>        8080/TCP,8081/TCP   23h
rook-ceph-mgr                            ClusterIP   10.102.85.146    <none>        9283/TCP            23h
rook-ceph-mgr-dashboard                  ClusterIP   10.110.80.91     <none>        8443/TCP            23h
rook-ceph-mgr-dashboard-external-https   NodePort    10.104.156.86    <none>        8443:31867/TCP      26s
rook-ceph-mon-a                          ClusterIP   10.110.117.117   <none>        6789/TCP,3300/TCP   23h
rook-ceph-mon-b                          ClusterIP   10.102.192.7     <none>        6789/TCP,3300/TCP   23h
rook-ceph-mon-c                          ClusterIP   10.98.8.230      <none>        6789/TCP,3300/TCP   23h

https://1x.xxx.151.208:31867/

获取登录密码

用户名是：admin

密码获取：kubectl -n rook-ceph get secret rook-ceph-dashboard-password -o jsonpath=”{[‘data’][‘password’]}” | base64 –decode && echo

(],(-W|G^rJ’JO,2!<#R