kubernetes持久化存储能力探测

jerry-jibu

kubernetes持久化存储能力探测

背景

Kubernetes (K8S) Container Storage Interface (CSI) 于2018年底在K8S v1.13 版本中提升为GA状态，为生产环境下部署有状态应用，比如数据库，提供了标准的持久卷(persistent volume, PV)配置接口。

持久卷快照(volume snapshot)接口在K8S v1.20版本中被提升为GA状态 (v1.17 - Beta)，该功能帮助用户以标准的接口集成快照这个非常重要的企业级存储能力，为保障核心应用的数据安全提供技术支撑。

问题

CSI 通过标准接口来暴露底层存储系统中的可用的存储能力。然而在实际部署过程中，不同K8S发行版叠加不同的存储系统能力，使得通过K8S来配置和使用持久卷变的更为复杂，经常需要运维人员手动对不同的存储能力进行标识，来找到满足应用的存储实例。

改进

本文介绍一个相关的开源工具: kubestr ，用于发现和验证K8S环境中CSI 存储能力，方便日常运维操作。

安装

[root@remote-dev ~] wget https://github.com/kastenhq/kubestr/releases/download/v0.4.17/kubestr-v0.4.17-linux-amd64.tar.gz

[root@remote-dev ~] tar -xvf kubestr-v0.4.17-linux-amd64.tar.gz

[root@remote-dev ~] ./kubestr --help
kubestr is a tool that will scan your k8s cluster
		and validate that the storage systems in place as well as run
		performance tests.

Usage:
  kubestr [flags]
  kubestr [command]

Available Commands:
  csicheck    Runs the CSI snapshot restore check
  fio         Runs an fio test
  help        Help about any command

Flags:
  -h, --help            help for kubestr
  -o, --output string   Options(json)

Use "kubestr [command] --help" for more information about a command.

CSI 存储能力探测

示例中通过运行./kubestr，查找K8S 集群中的 storage provider 以及对应的storage class

对于 rook-ceph 可以发现除了基本的PVC/PV能力外，还支持额外的多种存储功能 Raw Block, Snapshot, Expansion, Topology, Cloning

[root@remote-dev ~] ./kubestr

**************************************
  _  ___   _ ___ ___ ___ _____ ___
  | |/ / | | | _ ) __/ __|_   _| _ \
  | ' <| |_| | _ \ _|\__ \ | | |   /
  |_|\_\\___/|___/___|___/ |_| |_|_\

Explore your Kubernetes storage options
**************************************
Kubernetes Version Check:
  Valid kubernetes version (v1.19.5)  -  OK

RBAC Check:
  Kubernetes RBAC is enabled  -  OK

Aggregated Layer Check:
  The Kubernetes Aggregated Layer is enabled  -  OK

W0720 17:07:36.588200   10462 warnings.go:70] storage.k8s.io/v1beta1 CSIDriver is deprecated in v1.19+, unavailable in v1.22+; use storage.k8s.io/v1 CSIDriver
Available Storage Provisioners:

  fuseim.pri/ifs:
    Unknown driver type.

    Storage Classes:
      * managed-nfs-storage

    To perform a FIO test, run-
      ./kubestr fio -s <storage class>

  rook-ceph.rbd.csi.ceph.com:
    This is a CSI driver!
    (The following info may not be up to date. Please check with the provider for more information.)
    Provider:            Ceph RBD
    Website:             https://github.com/ceph/ceph-csi
    Description:         A Container Storage Interface (CSI)  Driver for Ceph RBD
    Additional Features: Raw Block, Snapshot, Expansion, Topology, Cloning

    Storage Classes:
      * rook-ceph-block
    Volume Snapshot Classes:
      * csi-rbdplugin-snapclass

    To perform a FIO test, run-
      ./kubestr fio -s <storage class>

    To test CSI snapshot/restore functionality, run-
      ./kubestr csicheck -s <storage class> -v <volume snapshot class>

持久卷快照和恢复功能验证

示例中通过 ./kubestr csicheck -s <storage class> -v <snapshot class> 命令， kubestr 基于指定的 storage class 创建PVC/PV 并结合 snapshot class 验证快照和恢复功能是否运行正常。

[root@remote-dev ~] ./kubestr csicheck -s rook-ceph-block -v csi-rbdplugin-snapclass
Creating application
  -> Created pod (kubestr-csi-original-podr22q8) and pvc (kubestr-csi-original-pvczbvjv)
Taking a snapshot
  -> Created snapshot (kubestr-snapshot-20210720170801)
Restoring application
  -> Restored pod (kubestr-csi-cloned-podvvm58) and pvc (kubestr-csi-cloned-pvcghp45)
Cleaning up resources
CSI checker test:
  CSI application successfully snapshotted and restored.  -  OK

基本的性能测试

示例中通过 ./kubestr fio -s <storage class> 命令， kubestr 基于指定的storage class 创建PVC/PV 并使用 fio 来对挂载的PVC 进行一个基本的IO性能测试

[root@remote-dev ~] ./kubestr fio -s rook-ceph-block
PVC created kubestr-fio-pvc-nkscx
Pod created kubestr-fio-pod-r8sxj
Running FIO test (default-fio) on StorageClass (rook-ceph-block) with a PVC of Size (100Gi)
Elapsed time- 43.356058723s
FIO test results:

FIO version - fio-3.20
Global options - ioengine=libaio verify=0 direct=1 gtod_reduce=1

JobName: read_iops
  blocksize=4K filesize=2G iodepth=64 rw=randread
read:
  IOPS=1371.392700 BW(KiB/s)=5502
  iops: min=1176 max=1668 avg=1381.166626
  bw(KiB/s): min=4704 max=6672 avg=5524.666504

JobName: write_iops
  blocksize=4K filesize=2G iodepth=64 rw=randwrite
write:
  IOPS=705.894043 BW(KiB/s)=2840
  iops: min=578 max=850 avg=709.266663
  bw(KiB/s): min=2312 max=3400 avg=2837.066650

JobName: read_bw
  blocksize=128K filesize=2G iodepth=64 rw=randread
read:
  IOPS=1376.654297 BW(KiB/s)=176745
  iops: min=1326 max=1436 avg=1386.633301
  bw(KiB/s): min=169728 max=183808 avg=177490.140625

JobName: write_bw
  blocksize=128k filesize=2G iodepth=64 rw=randwrite
write:
  IOPS=747.651184 BW(KiB/s)=96232
  iops: min=622 max=864 avg=750.533325
  bw(KiB/s): min=79616 max=110592 avg=96074.765625

Disk stats (read/write):
  rbd0: ios=47346/25116 merge=328/665 ticks=2157326/2151962 in_queue=2189862, util=99.720703%
  -  OK

参考

CSI Specification: https://github.com/container-storage-interface/spec/blob/master/spec.md

Kubernetes CSI Documentation: https://kubernetes-csi.github.io/docs/

kubestr: https://github.com/kastenhq/kubestr