Introduce
This plugin describe the deployment with containerized Ceph cluster by Rook operator and provide storage service in edge location node.
As a distributed backend storage, Ceph is widely used on Cloud and Edge solutions, it can provide Block, Filesystem and Object storage service. Traditional deployment for Ceph tools include: ceph-deploy, puppet, ansible etc.
Rook is a storage orchestrator for Kubernetes, which turns distributed storage system into self-managing, self-scaling, self-healing storage services, compared with traditional tools, there are several expectations by using Rook:
- Reduced deployment time for new clusters
- Simplified upgrades
- More agile horizontal scaling
- Better failure tolerance
- Reduced reliance on expert Ceph operators
CSI (Container Storage Interface) is a standard for exposing arbitrary block and file storage storage system to containerized workload on Container Orchestration Systems (COs) like Kubernetes. Also Ceph-CSI is supported by Rook from v1.0.
Implement
Rook implementation include two parts, Rook operator and Rook Ceph cluster.
Rook operator include CRDs and also including Ceph-CSI support for volume provisioning.
| apiVersion: apps/v1 kind: Deployment metadata: name: rook-ceph-operator namespace: rook-ceph labels: operator: rook storage-backend: ceph spec: selector: matchLabels: app: rook-ceph-operator replicas: 1 template: metadata: labels: app: rook-ceph-operator spec: serviceAccountName: rook-ceph-system containers: - name: rook-ceph-operator image: rook/ceph:v1.0.4 args: ["ceph", "operator"] volumeMounts: - mountPath: /var/lib/rook name: rook-config - mountPath: /etc/ceph name: default-config-dir env: - name: ROOK_CURRENT_NAMESPACE_ONLY value: "true" # CSI enablement - name: ROOK_CSI_ENABLE_CEPHFS value: "true" - name: ROOK_CSI_CEPHFS_IMAGE value: "quay.io/cephcsi/cephfsplugin:v1.0.0" - name: ROOK_CSI_ENABLE_RBD value: "true" - name: ROOK_CSI_RBD_IMAGE value: "quay.io/cephcsi/rbdplugin:v1.0.0" - name: ROOK_CSI_REGISTRAR_IMAGE value: "quay.io/k8scsi/csi-node-driver-registrar:v1.0.2" - name: ROOK_CSI_PROVISIONER_IMAGE value: "quay.io/k8scsi/csi-provisioner:v1.0.1" - name: ROOK_CSI_SNAPSHOTTER_IMAGE value: "quay.io/k8scsi/csi-snapshotter:v1.0.1" - name: ROOK_CSI_ATTACHER_IMAGE value: "quay.io/k8scsi/csi-attacher:v1.0.1" # The name of the node to pass with the downward API - name: NODE_NAME valueFrom: fieldRef: fieldPath: spec.nodeName # The pod name to pass with the downward API - name: POD_NAME valueFrom: fieldRef: fieldPath: metadata.name # The pod namespace to pass with the downward API - name: POD_NAMESPACE valueFrom: fieldRef: fieldPath: metadata.namespace volumes: - name: rook-config emptyDir: {} - name: default-config-dir emptyDir: {} |
|---|
Rook Ceph cluster include settings for Ceph monitor and storage, currently the setting as following:
| apiVersion: ceph.rook.io/v1 kind: CephCluster metadata: name: rook-ceph namespace: rook-ceph spec: cephVersion: image: ceph/ceph:v13.2.2-20190410 allowUnsupported: false dataDirHostPath: /var/lib/rook mon: count: 3 allowMultiplePerNode: true dashboard: enabled: true network: hostNetwork: false rbdMirroring: workers: 0 annotations: resources: storage: # cluster level storage configuration and selection useAllNodes: true useAllDevices: false deviceFilter: location: config: storeType: filestore metadataDevice: # "md0" specify a non-rotational storage so ceph-volume will use it as block db device of bluestore. databaseSizeMB: "10240" # uncomment if the disks are smaller than 100 GB journalSizeMB: "10240" # uncomment if the disks are 20 GB or smaller directories: - path: "/var/lib/rook/storage-dir" |
|---|