Kubernetes集群实践（八）Kubernetes安装云原生存储系统Rook

发表于

本文主要介绍如何安装和使用云原生存储系统Rook。

关键词：k8s

准备工作

  • Ubuntu 20.04 LTS amd64，32GB，作为Master节点

假定已经安装了K8s集群，这里以单节点集群为例

Rook

  • Rook是一个开源的云原生存储协调器，提供平台、框架和对各种存储解决方案的支持，以与云原生环境进行本地集成。
  • Rook将存储软件转变为自我管理，自我扩展和自我修复的存储服务。它通过自动化部署、引导、配置、供应、扩展、升级、迁移、灾难恢复、监视和资源管理来实现。Rook使用底层云原生容器管理，调度和编排平台提供的功能来执行其职责。
  • Rook利用扩展点将其深度集成到云原生环境中，并为调度、生命周期管理、资源管理、安全性、监视和用户体验提供无缝的体验。

部署Rook

下载源码，项目地址：https://github.com/rook/rook/releases

最新源码为1.9.7

1
git clone --single-branch --branch v1.9.7 https://github.com/rook/rook.git

以块存储RBD为例

部署准备工作

1
2
cd rook/deploy/examples
kubectl create -f crds.yaml -f common.yaml -f operator.yaml

确保存储设备没有文件系统

1
sudo fdisk -l

如果有GPT或者dos等描述文字，说明存在分区表，需要把这个删掉

1
sudo wipefs /dev/sda

可以看到分区表

执行删除命令

1
sudo wipefs -a -f /dev/sda

根据需要修改一下集群

1
vim cluster.yaml

主要是指定设备，设定mon为1，mgr为1等操作

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
#################################################################################################################
# Define the settings for the rook-ceph cluster with common settings for a small test cluster.
# All nodes with available raw devices will be used for the Ceph cluster. One node is sufficient
# in this example.

# For example, to create the cluster:
#   kubectl create -f crds.yaml -f common.yaml -f operator.yaml
#   kubectl create -f cluster-test.yaml
#################################################################################################################
kind: ConfigMap
apiVersion: v1
metadata:
  name: rook-config-override
  namespace: rook-ceph # namespace:cluster
data:
  config: |
    [global]
    osd_pool_default_size = 1
    mon_warn_on_pool_no_redundancy = false
    bdev_flock_retry = 20
    bluefs_buffered_io = false
---
apiVersion: ceph.rook.io/v1
kind: CephCluster
metadata:
  name: my-cluster
  namespace: rook-ceph # namespace:cluster
spec:
  dataDirHostPath: /var/lib/rook
  cephVersion:
    image: quay.io/ceph/ceph:v17
    allowUnsupported: true
  mon:
    count: 1
    allowMultiplePerNode: true
  mgr:
    count: 1
    allowMultiplePerNode: true
  dashboard:
    enabled: true
  crashCollector:
    disable: true
  storage:
    useAllNodes: false
    useAllDevices: false
    #deviceFilter:
    config:
      # crushRoot: "custom-root" # specify a non-default root label for the CRUSH map
      # metadataDevice: "md0" # specify a non-rotational storage so ceph-volume will use it as block db device of bluestore.
      # databaseSizeMB: "1024" # uncomment if the disks are smaller than 100 GB
      # journalSizeMB: "1024"  # uncomment if the disks are 20 GB or smaller
      osdsPerDevice: "1" # this value can be overridden at the node or device level
      # encryptedDevice: "true" # the default value for this option is "false"
# Individual nodes and their config can be specified as well, but 'useAllNodes' above must be set to false. Then, only the named
# nodes below will be used as storage resources.  Each node's 'name' field should match their 'kubernetes.io/hostname' label.
    nodes:
      - name: "amd-5700g"
        devices: # specific devices to use for storage can be specified for each node
          - name: "sda"
    deviceFilter:
  healthCheck:
    daemonHealth:
      mon:
        interval: 45s
        timeout: 600s
  priorityClassNames:
    all: system-node-critical
    mgr: system-cluster-critical
  disruptionManagement:
    managePodBudgets: true
---
apiVersion: ceph.rook.io/v1
kind: CephBlockPool
metadata:
  name: builtin-mgr
  namespace: rook-ceph # namespace:cluster
spec:
  name: .mgr
  replicated:
    size: 1
    requireSafeReplicaSize: false

现在可以部署了

1
kubectl apply -f cluster.yaml

安装ceph工具

1
kubectl applt -f toolbox.yaml

等待相关操作执行完毕

此时进入ceph-tools，查看ceph集群工作状态

1
kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- bash

挂载Rook块存储到Pod容器

首先需要创建strongeClass，才可以让k8s控制器根据声明的PVC创建PV

1
2
3
4
5
6
7
8
9
10
11
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: mysql-standalone-pvc
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: rook-ceph-block
  resources:
    requests:
      storage: 1Gi

然后在Pod里创建PVC

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
apiVersion: v1
kind: Pod
metadata:
  name: mysql-standalone-pod
spec:
  containers:
  - name: mysql
    image: mysql:8-debian
    args:
    - --max_connections=10000
    - --character-set-server=utf8mb4
    - --collation-server=utf8mb4_general_ci
    - --default-authentication-plugin=mysql_native_password
    env:
    - name: MYSQL_ROOT_PASSWORD
      value: "root"
    volumeMounts:
    - name: mysql-standalone-data
      mountPath: /var/lib/mysql             # 容器挂载路径
      subPath: mysql
    - name: mysql-conf
      mountPath: /etc/mysql/conf.d
  volumes:
  - name: mysql-standalone-data
    persistentVolumeClaim:
      claimName: mysql-standalone-pvc
  - name: mysql-conf
    configMap:
      name: mysql-standalone-config