Cloud native Multi tenancy proposal

Motivation

In ONAP, we required to share resources with multiple users and/or application. In the web enterprise segment, it is like multiple deployments team sharing the Kubernetes(K8s) cluster. In the case of Telco or Cable segment, we have multiple end users sharing the same edge compute resource. This proposal refers to the Kubernetes Multi-tenancy options and how to utilize it in ONAP architecture and also to benefit Multi-tenancy use case in K8s

Goal(in Scope)

Focusing on the solution within the cluster for tenants, and working with Kubernetes SIG groups and adapt the solution in the ONAP

Goal(Out of Scope)

Working in Kubernetes core or API is clearly out of the scope of these documents. There are the solutions available to provide a separate control plane to each tenant in a cluster, it is quite expensive and hard to have such a solution in a cloud-native space.

Outline

In this section, we define Multi-Tenancy in general for the Orchestration engine. A tenant can be defined as a group of resources bounded and isolated amount of compute, storage, networking and control plane in a kubernetes cluster. A tenant can also be defined as a group of users slicing a bounded resource allocated for them. These resources can be as follows:

CPU, Memory, Extended Resources
Network bandwidth, I/O bandwidth, Kubernetes cluster resource
Resource reservation to provide the Guaranteed QoS in Kubernetes

Multi-tenancy can be distinguished as "Soft Multitenancy" and "Hard Multitenancy"

Soft Multitenancy tenants are trusted(means tenant obey the resource boundary between them). One tenant should not access the resource of another tenant
Hard Multitenancy tenants can't be trusted(means any tenant can be malicious, and there must be a strong security boundary between them), So one tenant should not have access to anything from other tenants.

Requirement

For a service provider, a tenant is basically a group of end-user sharing the same cluster, we have to make sure that the end user resources are tracked and accountable for their consumption in a cluster
In a few cases, admin or end-user application is shared among multiple tenants, in such case application resource should be tracked across the cluster
Centralization resource quota or the allocation limits record should be maintained by admin or for the end user. For example, just a kubectl "query" to Kubernetes API should display the resource quota and policy for each end-user or tenant
In Edge use case, the service orchestration like ONAP should get the resource details across multiple clusters by resource orchestration, should set the resource allocation for the cluster and decide the scheduling mechanism
User credential centralization with application orchestration

ONAP Cloud Native Multi-tenancy Proposal

ONAP Cloud Native Multi-tenancy proposal reuses the Kubernetes Multi-tenancy works to bind the tenant at the service orchestration and resource orchestration level.

Kubernetes Tenant project

All the materials discussed in the following section are documented in the reference section link, and the contents are belongs to authors to respective doc

Kubernetes community working on Tenant controller that define tenant as Custom resource definition, CRD, and define the following elements.

Kubernetes Tenant controller creates a multi-tenant ready kubernetes cluster that allows the creation of the following new types of kubernetes objects/ resources:

A tenant resource (referred as “Tenant-CR” for simplicity)
A namespace template resource (referred as “NamespaceTemplate-CR” for simplicity)

Tenant resource: Is a simple CRD object for the tenant with namespace associated with the tenant name

Namespace template resource: Define like Role, RoleBinding, ResourceQuota, network policy for the namespace associated with tenant-CR

Tenant controller set up

Build the Tenant controller

The following steps explain how to run the tenant controller in kubernetes

Tenant controller Build

$ go get github.com/kubernetes-sigs/multi-tenancy
$ cd $GOPATH/src/github.com/kubernetes-sigs/multi-tenancy
$ cat <<EOF > $PWD/.envrc
export PATH="`pwd`/tools/bin:$PATH"
EOF
$ source .envrc
<<Have to install additional few golang package, the project is not having vendor folder>>
$ go get github.com/golang/glog
$ go get k8s.io/client-go/kubernetes
$ go get k8s.io/client-go/kubernetes/scheme
$ go get k8s.io/client-go/plugin/pkg/client/auth/gcp
$ go get k8s.io/client-go/tools/clientcmd
$ go get github.com/hashicorp/golang-lru
$ devtk setup
$ devtk build
<<Running tenant controller>>
$ $PWD/out/tenant-controller/tenant-ctl -v=99 -kubeconfig=$HOME/.kube/config

Tenant CRD definitions

The Tenant CRD object is defined by following CRD objects:

Tenant-CRD

---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
name: tenants.tenants.k8s.io
spec:
group: tenants.k8s.io
versions:
- name: v1alpha1
served: true
storage: true
scope: Cluster
names:
plural: tenants
singular: tenant
kind: Tenant
---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
name: namespacetemplates.tenants.k8s.io
spec:
group: tenants.k8s.io
versions:
- name: v1alpha1
served: true
storage: true
scope: Cluster
names:
plural: namespacetemplates
singular: namespacetemplate
kind: NamespaceTemplate
shortNames:
- nstpl

Let's run the tenant controller and create a tenant object as follows:

Tenant object

$ kubectl create -f $GOPATH/src/github.com/kubernetes-sigs/multi-tenancy/poc/tenant-controller/data/manifests/crd.yaml
$ kubectl create -f $GOPATH/src/github.com/kubernetes-sigs/multi-tenancy/poc/tenant-controller/data/manifests/rbac.yaml
<<Running tenant controller>>
$ $GOPATH/src/github.com/kubernetes-sigs/multi-tenancy/out/tenant-controller/tenant-ctl -v=99 -kubeconfig=$HOME/.kube/config

$ # kubectl get crd
NAME                                             CREATED AT
namespacetemplates.tenants.k8s.io                2019-05-01T16:34:49Z
tenants.tenants.k8s.io                           2019-05-01T16:34:49Z

$ kubectl create -f $GOPATH/src/github.com/kubernetes-sigs/multi-tenancy/poc/tenant-controller/data/manifests/sample-nstemplate.yaml
$ kubectl create -f $GOPATH/src/github.com/kubernetes-sigs/multi-tenancy/poc/tenant-controller/data/manifests/sample-tenant.yaml
$ kubectl get tenants
NAME       AGE
tenant-a   7d

$ kubectl get ns
NAME            STATUS   AGE
default         Active   42d
kube-public     Active   42d
kube-system     Active   42d
tenant-a-ns-1   Active   7d18h
tenant-a-ns-2   Active   7d18h

A closer look into tenant Object

The tenant object looks like below:

tenant object

---
apiVersion: tenants.k8s.io/v1alpha1
kind: Tenant
metadata:
  name: tenant-a 
spec:
  namespaces:
      - name: ns-1
      - name: ns-2

The tenant controller takes this tenant spec as a template to creates the namespace for each namespace as tenant-a-ns1, tenant-a-ns2. In addition, the tenant object can also create an admin object for a tenant with a user for admin.

In addition, it creates a namespace template, it defines templates that define Rolebinding, ClusterRole, NetworkPolicy for the namespace tenant-a-ns1 and tenant-a-ns2.

namespaceTemplate

$ kubectl get namespacetemplate
NAME         AGE
restricted   7d

$ kubectl get namespacetemplate restricted -o yaml
apiVersion: tenants.k8s.io/v1alpha1
kind: NamespaceTemplate
metadata:
  creationTimestamp: "2019-05-01T17:37:11Z"
  generation: 1
  name: restricted
  resourceVersion: "3628408"
  selfLink: /apis/tenants.k8s.io/v1alpha1/namespacetemplates/restricted
  uid: bffbe9c8-6c37-11e9-91c3-a4bf014c3518
spec:
  templates:
  - apiVersion: rbac.authorization.k8s.io/v1
    kind: RoleBinding
    metadata:
      name: multitenancy:podsecuritypolicy
    roleRef:
      apiGroup: rbac.authorization.k8s.io
      kind: ClusterRole
      name: multitenancy:use-psp:restricted
    subjects:
    - apiGroup: rbac.authorization.k8s.io
      kind: Group
      name: system:serviceaccounts
  - apiVersion: networking.k8s.io/v1
    kind: NetworkPolicy
    metadata:
      name: multitenancy-default
    spec:
      podSelector: {}
      policyTypes:
      - Ingress
      - Egress

Resource quota proposal for the tenant CRD

A tenant-based resource quota is required to implement resource tracking in ONAP. The proposal here is to reuse the tenant controller work in Kubernetes and introduce the tenant resource quota CRD on the top of tenant controller

Tenant resource quota

apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: tenantresourcequota.tenants.k8s.io
spec:
  group: tenants.k8s.io
  versions:
    - name: v1alpha1
      served: true
      storage: true
  scope: Cluster
  names:
    plural: tenantresourcequotas
    singular: tenantresourcequota
    kind: TenantResourcequota
    shortNames:
    - trq

Before jumping into the definition of the tenant resource quota, refer Kubernetes resource quota - https://kubernetes.io/docs/concepts/policy/resource-quotas/ for more understanding on the resource quota

And tenant resource quota schema should be like this below

tenant resourcequota

// NamespaceTemplate defines a template of resources to be created inside a namespace.
type TenantResourceQuota struct {
    metav1.TypeMeta   `json:",inline"`
    metav1.ObjectMeta `json:"metadata,omitempty"`

    Spec TenantResourceQuotaSpec `json:"spec"`
}

// TenantResourceQuotaSpec defines the desired hard limits to enforce for Quota
type TenantResourceQuotaSpec struct {
	// Hard is the set of desired hard limits for each named resource
	// +optional
	Hard v1.ResourceList `json:"hard"`
	// A collection of filters that must match each object tracked by a quota.
	// If not specified, the quota matches all objects.
	// +optional
	Scopes []v1.ResourceQuotaScope `json:"scopes"`
	// ScopeSelector is also a collection of filters like Scopes that must match each object tracked by a quota
	// but expressed using ScopeSelectorOperator in combination with possible values.
	// +optional
	ScopeSelector *v1.ScopeSelector `json:"scopeSelector"`
    UserResourcequota  []string `json:"userResourcequota"`
}

And the example Tenant resource quota should be like this.