Versions Compared

Old Version 22

changes.mady.by.user cheng1.li

Saved on

compared with

New Version 23

changes.mady.by.user cheng1.li

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

Goal

Sdewan config agent is the controller of Sdewan CRDs. With the config agent, we are able to deploy CNFs. In this page, we have the following terms, let's define them here.

...

To deploy a CNF, user needs to create a CNF pod and some Sdewan rule CRs. In a Kubernetes namespace, there could be more than one CNF pod and many Sdewan rule CRs. We use label to correlate one CNF with some Sdewan rule CRs

We use label to correlate one CNF with some Sdewan rule CRs. The Sdewan controller watches Sdewan rule CRs and applies them onto the correlated CNF pod by calling CNF REST api.

Sdwan Design Principle

  • There could be multiple tenants/namespaces in a Kubernetes cluster. User may deploy multiple CNFs in any one or more tenants.
  • One Sdewan instance contains only one pod in this release. There could be two pods in future releases for active/backup case
  • CNF pod and Sdewan rule CRs can be created/updated/deleted in any order
  • The Sdewan controller and CNF pod could be down sometimes for some reasons. We need to handle these scenarios
  • Each Sdewan rule CR has labels to identify the type it belongs to. 3 types are available at this time: basic, app-intend and k8s-service. We extend k8s user role permission so that we can set user permission on type level of Sdewan rule CR
  • Sdewan rule CR dependencies are checked on creating/updating/deleting. For example, if we create a mwan3_rule CR which uses policy policy-x, but no mwan3_policy CR named policy-x exists. Then we block the request

CNF pod

In this section we describe what the CNF pod should be like.

...

Code Block
languageyml
titleCNF pod
apiVersion: v1
kind: Pod
metadata:
  annotations:
    k8s.plugin.opnfv.org/nfn-network: |-
      { "type": "ovn4nfv", "interface": [
        {
          "defaultGateway": "false",
          "interface": "net0",
          "name": "ovn-priv-net"
        }
      ]}
    k8s.plugin.opnfv.org/ovnInterfaces: '[{"ip_address":"172.16.44.2/24", "mac_address":"0a:00:00:00:00:01",
      "gateway_ip": "172.16.44.1","defaultGateway":"false","interface":"net0"}]'
    k8s.v1.cni.cncf.io/networks: '[{ "name": "ovn-networkobj"}]'
    k8s.v1.cni.cncf.io/networks-status: |-
      [{
          "name": "cni0",
          "interface": "eth0",
          "ips": [
              "10.244.64.26"
          ],
          "mac": "0a:58:0a:f4:40:1a",
          "default": true,
          "dns": {}
      },{
          "name": "ovn4nfv-k8s-plugin",
          "interface": "net0",
          "ips": [
              "172.16.44.2"
          ],
          "mac": "0a:00:00:00:00:01",
          "dns": {}
      }]
  name: cnf-pod-1
  namespace: default
  labels:
    sdewanPurpose: cnf-1
spec:
  containers:
  - command:
    - /bin/sh
    - /tmp/sdewan/entrypoint.sh
    image: integratedcloudnative/openwrt:dev
    name: sdewan
    readinessProbe:
      failureThreshold: 5
      httpGet:
        path: /
        port: 80
        scheme: HTTP
      initialDelaySeconds: 5
      periodSeconds: 5
      successThreshold: 1
      timeoutSeconds: 1
    securityContext:
      privileged: true
      procMount: Default
    volumeMounts:
    - mountPath: /tmp/sdewan
      name: example-sdewan
      readOnly: true
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: default-token-7t7fh
      readOnly: true
  dnsPolicy: ClusterFirst
  nodeName: ubuntu18
  nodeSelector:
    kubernetes.io/hostname: ubuntu18


Sdewan rule CRs

CRD defines all properties of a resource, but it's not human friendly. So we paste Sdewan rule CR samples instead of CRDs.

...

CR samples of IPSec type(ruoyu):

Sdewan rule CRD Reconcile Logic

As we have many kinds of CRDs, they have almost the same reconcile logic. So we only describe the Mwan3Rule logic.

...

Code Block
languagepy
def Mwan3RuleReconciler.Reconcile(req ctrl.Request):
  rule_cr = k8sClient.get(req.NamespacedName)
  cnf_pod = k8sClient.get_pod_with_label(rule_cr.labels.sdewanPurpose)
  if rule_cr DeletionTimestamp exists:
    # The CR is being deleted. finalizer on the CR
    if cnf_pod exists:
      if cnf_pod is ready:
        err = openwrt_client.delete_rule(cnf_pod_ip, rule_cr)
        if err:
          return "re-queue req"
        else:
          rule_cr.finalizer = nil
      else:
        return "re-queue req"
    else:
      # Just remove finalizer, because no CNF pod exists
      rule_cr.finalizer = nil
  else:
    # The CR is not being deleted
    if cnf_pod not exist:
      return
    else:
      if cnf_pod not ready:
        return "re-queue req"
      else:
        if dependencies mwan3_policy not applied:
          return "re-queue req"
        else:
          err = openwrt_client.add_or_update_rule(cnf_pod_ip, rule_cr)
          if not err:
            rule_cr.finalizer = new_finalizer
            rule_cr.status.appliedVersion = rule_cr.resourceVersion
          else:
            return "re-queue req"


Admission Webhook Usage

We use admission webhook to implemention several features.

  1. Prevent creating more than one CNF of the same lable and the same namespace
  2. Validate CR dependencies. For example, mwan3 rule depends on mwan3 policy
  3. Extend user permission to control the operations on rule CRs. For example, we can control that ONAP can't update/delete rule CRs created by platform.

Sdewan rule CR type level Permission Implementation

K8s support permission control on namespace level. For example, user1 may be able to create/update/delete one kind of resource(e.g. pod) in namespace ns1, but not namespace ns2. For Sdewan, this can't fit our requirement. We want label level control of Sdewan rule CRs. For example, user_onap can create/update/delete Mwan3Rule CR of label sdewan-bucket-type=app-intent, but not label sdewan-bucket-type=basic.

...

Code Block
languagepy
def mwan3rule_webhook_handle_permission(req admission.Request):
  userinfo = req["userInfo]
  mwan3rule_cr = decode(req)
  roles = k8s_client.get_role_from_user(userinfo)
  for role in roles:
    if mwan3rule_cr.labels.sdewan-bucket-type in role.annotation.sdewan-bucket-type-permission.mwan3rules:
      return {"allowd": True}
  return {"allowd": False}

ServiceRule controller (For next release)

We create a controller watches the services created in the cluster. For each service, it creates a FirewallDNAT CR. On controller startup, it makes a syncup to remove unused CRs.

References

...