Sdewan config agent is the controller of Sdewan(a CRD) instance. When a Sdewan instance is created, the agent creates the pod running OpenWRT. The OpenWRT could be a sdwan, an ipsec tunnel or a firewall, depends on the rules applied to the openwrt instand.

CRDs

Sdewan CRD depends on some CRDs. So we list the related CRDs here.

...

Goal

Sdewan config agent is the controller of Sdewan CRDs. With the config agent, we are able to deploy CNFs. In this page, we have the following terms, let's define them here.

• CNF pod: A pod running network function process(openWRT). The pod could be managed under deployment.
• Sdewan rule: The rule defines the CNF behaves. We have 3 classes of rules: mwan3, firewall, ipsec. Each class includes several kinds of rules. For example, mwan3 has 2 kinds: mwan3_policy and mwan3_rule. Firewall has 5 kinds: firewall_zone, firewall_snat, firewall_dnat, firewall_forwarding, firewall_rule. Ipsec has xx(ruoyu) kinds: xx, xx.
• Sdewan rule CRD: The CRD defines each kind of sdewan rule. For each kind of Sdewan rule, we have a Sdewan rule CRD. Sdewan rule CRD is namespaced resource.
• Sdewan rule CR: Instance of Sdewan rule CRD.
• Sdewan controller: The controller watching Sdewan rule CRs.
• CNF: A network function running in container.

To deploy a CNF, user needs to create a CNF pod and some Sdewan rule CRs. In a Kubernetes namespace, there could be more than one CNF pod and many Sdewan rule CRs. We use label to correlate one CNF with some Sdewan rule CRs. The Sdewan controller watches Sdewan rule CRs and applies them onto the correlated CNF pod by calling CNF REST api.

Sdwan Design Principle

• There could be multiple tenants/namespaces in a Kubernetes cluster. User may deploy multiple CNFs in any one or more tenants.
• One Sdewan instance contains only one pod in this release. There could be two pods in future releases for active/backup case
• CNF pod and Sdewan rule CRs can be created/updated/deleted in any order
• The Sdewan controller and CNF pod could be down sometimes for some reasons. We need to handle these scenarios
• Each Sdewan rule CR has labels to identify the type it belongs to. 3 types are available at this time: basic, app-intend and k8s-service. We extend k8s user role permission so that we can set user permission on type level of Sdewan rule CR
• Sdewan rule CR dependencies are checked on creating/updating/deleting. For example, if we create a mwan3_rule CR which uses policy policy-x, but no mwan3_policy CR named policy-x exists. Then we block the request

CNF pod

In this section we describe what the CNF pod should be like.

• CNF pod should has multiple network interfaces attached. We use multus and ovn4nfv CNIs to enable multiple interfaces. So in the CNF pod yaml, we set annotations: k8s.v1.cni.cncf.io/networks, k8s.plugin.opnfv.org/nfn-network.
• When user deploys a CNF, she/he most likely want to deploy the CNF on a specified node instead of a random node. Because some nodes may don't have provider network connected. So we set spec.nodeSelector in the pod yaml
• CNF pod runs openWRT in ICN. We use image integratedcloudnative/openwrt:dev
• CNF pod should setup with rediness probe. Sdewan controller would check pod readiness before calling CNF REST api.

apiVersion: v1
kind: Pod
metadata:
  annotations:
    k8s.plugin.opnfv.org/nfn-network: |-
      { "type": "ovn4nfv", "interface": [
        {
          "defaultGateway": "false",
          "interface": "net0",
          "name": "ovn-priv-net"
        }
      ]}
    k8s.plugin.opnfv.org/ovnInterfaces: '[{"ip_address":"172.16.44.2/24", "mac_address":"0a:00:00:00:00:01",
      "gateway_ip": "172.16.44.1","defaultGateway":"false","interface":"net0"}]'
    k8s.v1.cni.cncf.io/networks: '[{ "name": "ovn-networkobj"}]'
    k8s.v1.cni.cncf.io/networks-status: |-
      [{
          "name": "cni0",
          "interface": "eth0",
          "ips": [
              "10.244.64.26"
          ],
          "mac": "0a:58:0a:f4:40:1a",
          "default": true,
          "dns": {}
      },{
          "name": "ovn4nfv-k8s-plugin",
          "interface": "net0",
          "ips": [
              "172.16.44.2"
          ],
          "mac": "0a:00:00:00:00:01",
          "dns": {}
      }]
  name: cnf-pod-1
  namespace: default
  labels:
    sdewanPurpose: cnf-1
spec:
  containers:
  - command:
    - /bin/sh
    - /tmp/sdewan/entrypoint.sh
    image: integratedcloudnative/openwrt:dev
    name: sdewan
    readinessProbe:
      failureThreshold: 5
      httpGet:
        path: /
        port: 80
        scheme: HTTP
      initialDelaySeconds: 5
      periodSeconds: 5
      successThreshold: 1
      timeoutSeconds: 1
    securityContext:
      privileged: true
      procMount: Default
    volumeMounts:
    - mountPath: /tmp/sdewan
      name: example-sdewan
      readOnly: true
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: default-token-7t7fh
      readOnly: true
  dnsPolicy: ClusterFirst
  nodeName: ubuntu18
  nodeSelector:
    kubernetes.io/hostname: ubuntu18

Sdewan rule CRs

CRD defines all properties of a resource, but it's not human friendly. So we paste Sdewan rule CR samples instead of CRDs.

• Each Sdewan rule CR has a label named sdewanPurpose to indicate which CNF should the rule be applied onto
• Each Sdewan rule CR has the status field which indicates if the latest rule is applied and when it's applied
• Mwan3Policy.spec.members[].network should match the networks defined in CNF pod annotation k8s.plugin.opnfv.org/nfn-network. As well as FirewallZone.spec[].network

CR samples of Mwan3 type:

apiVersion: batch.sdewan.akraino.org/v1alpha1
kind: SdewanMwan3Policy
metadata:
  name: balance1
  namespace: default
  labels:
    sdewanPurpose: examplecnf-sdwan1
  specresourceVersion: "2"
spec: node: node1
  networksmembers:
    - namenetwork: ovn-net1
      isProviderweight: 2
      metric: true2
    - namenetwork: ovn-net2
      weight: 3
 isProvider     metric: false3
status:
  mwan3ConfappliedVersion: example1-conf"2"
  firewallConfappliedTime: example-firewall"2020-03-29T04:21:48Z"

apiVersion: batch.sdewan.akraino.org/v1alpha1
kind: Mwan3ConfMwan3Rule
metadata:
  name: mwan3rule-1
  namespace: default
  labels:
    sdewanPurpose: example1-conf
spec:
  policy:
    balance1:
      members:
      - nework: ovn-net1
        weight: 2
        metric: 2
      - network:cnf-1
  resourceVersion: "2"
spec:  
  name: http
  policy: balance1
  dest_ip: 0.0.0.0/0
  dest_port: 80
status:
  appliedVersion: "2"
  appliedTime: "2020-03-29T04:21:48Z"

CR samples of Firewall type:

apiVersion: batch.sdewan.akraino.org/v1alpha1
kind: FirewallZone
metadata:
  name: zone-1
  namespace: default
  labels:
    sdewanPurpose: cnf-1
  resourceVersion: "2"
spec:  
  - name: lan1
    newtork:
      - ovn-net1
    input: ACCEPT
    output: ACCEPT
  - name: wan1
    network:
      - ovn-net2
    input: REJECT
    output: ACCEPT
weightstatus:
3
  appliedVersion: "2"
  appliedTime: "2020-03-29T04:21:48Z"

apiVersion: batch.sdewan.akraino.org/v1alpha1
kind: FirewallRule
metricmetadata:
3   name: reject_80
  namespace: default
  rulelabels:
    sdewanPurpose:  httpscnf-1
  resourceVersion: "2"
spec:  
  policysrc: balance1lan1
  src_ip: 192.168.1.2
  dest_ip: 0.0.0.0/0
      dest_port: 443
    http:
      policy: balance1src_port: 80
  proto: tcp
  target: REJECT
status:
  appliedVersion: "2"
  appliedTime: "2020-03-29T04:21:48Z"

apiVersion: batch.sdewan.akraino.org/v1alpha1
kind: FirewallSNAT
metadata:
  name: snat_lan1
  namespace: default
  labels:
    sdewanPurpose: cnf-1
  resourceVersion: "2"
spec:  
  src: lan1
dest  src_ip: 0192.0168.01.0/02
  src_dip: 1.2.3.4
  dest_port: 80: wan1
  proto: icmp
status:
  appliedVersion: "2"
  appliedTime: "2020-03-29T04:21:48Z"

apiVersion: batch.sdewan.akraino.org/v1alpha1
kind: FirewallConfFirewallDNAT
metadata:
  name: dnat_wan1
  namespace: default
  labels:
    sdewanPurpose: example-firewallcnf-1
  resourceVersion: "2"
spec:  
zones  src: wan1
  src_dport: 19900
 - namedest: lan
 lan1
  dest_ip: 192.168.1.1
  dest_port: 22
 network proto: tcp
status:
  appliedVersion: "2"
  appliedTime: "2020- ovn-net2
      input: REJECT
      output: ACCEPT
    - name: wan
      network:
        - ovn-net1
      input: REJECT
      output: ACCEPT
  redirects:
    - name: DNAT-LAN03-29T04:21:48Z"

apiVersion: batch.sdewan.akraino.org/v1alpha1
kind: FirewallForwarding
metadata:
  name: forwarding_lan_to_wan
  namespace: default
  labels:
    sdewanPurpose: cnf-1
  resourceVersion: "2"
spec:  
  src: lan1
  dest: wan1
status:
  appliedVersion: "2"
  appliedTime: "2020-03-29T04:21:48Z"

CR samples of IPSec type(ruoyu):

Sdewan rule CRD Reconcile Logic

As we have many kinds of CRDs, they have almost the same reconcile logic. So we only describe the Mwan3Rule logic.

Mwan3Rule Reconcile could be triggered by the following cases:

• Create/Update/Delete Mwan3Rule CR
• CNF pod ready status change (With predicate feature, we can only watch .status.containerStatuses[0].ready field of CNF pod. With enqueueRequestsFromMapFunc, we can enqueue all Mwan3Rule CRs with specified labels.sdewanPurpose, if CNF pod's .status.containerStatuses[0].ready changes) [question for Srini: if CNF pod is created by user instead of our defined CR, how does the controller know which pod is CNF pod? By special label?]
  • CNF pod becomes ready after creating
  • CNF pod becomes ready after restart
  • CNF pod becomes not-ready after crash

Mwan3Rule Reconcile flow:

def Mwan3RuleReconciler.Reconcile(req ctrl.Request):
  rule_cr = k8sClient.get(req.NamespacedName)
  cnf_pod = k8sClient.get_pod_with_label(rule_cr.labels.sdewanPurpose)
  if rule_cr DeletionTimestamp exists:
    # The CR is being deleted. finalizer on the CR
    if cnf_pod exists:
      if cnf_pod is ready:
        err = openwrt_client.delete_rule(cnf_pod_ip, rule_cr)
        if err:
          return "re-queue req"
        srcelse:
 wan         src_dport: 1990
 rule_cr.finalizer = nil
      else:
        return "re-queue req"
    dest: lanelse:
      # Just remove finalizer, because no CNF pod exists
      dest_port: 22
 rule_cr.finalizer = nil
  else:
    # The CR is not being deleted
    proto: tcpif cnf_pod not exist:
      return
   target else:
DNAT       rules:if cnf_pod not ready:
        return "re- name: REJECT_LAN_80queue req"
      else:
        if dependencies mwan3_policy not applied:
          return "re-queue req"
        srcelse:
   lan       src_ip: 192.168.1.2err = openwrt_client.add_or_update_rule(cnf_pod_ip, rule_cr)
          if not err:
            rule_cr.finalizer = new_finalizer
            rule_cr.status.appliedVersion = rule_cr.resourceVersion
          else:
            return  src_port: 80
      proto: tcp
      target: REJECT
  forwardings:
    - name: lan-wan
      src: lan
      dest: wan

The Sdewan reconcile function logic

Image Removed

The Mwan3Conf Reconcile logic

Image Removed

The controller behave

...

• validate the instance Spec by API admission webhook 
• If this Conf instance is used by any Sdewan instance, then info the Sdewan controller to apply the conf.

...

• If this Conf instance is used by any Sdwan instance, prevent the delete action

...

• Creates sdewan deployment/pod if it does not exist
• Apply the Conf if the Conf instance exists

...

"re-queue req"

Admission Webhook Usage

We use admission webhook to implemention several features.

1. Prevent creating more than one CNF of the same lable and the same namespace
2. Validate CR dependencies. For example, mwan3 rule depends on mwan3 policy
3. Extend user permission to control the operations on rule CRs. For example, we can control that ONAP can't update/delete rule CRs created by platform.

Sdewan rule CR type level Permission Implementation

K8s support permission control on namespace level. For example, user1 may be able to create/update/delete one kind of resource(e.g. pod) in namespace ns1, but not namespace ns2. For Sdewan, this can't fit our requirement. We want label level control of Sdewan rule CRs. For example, user_onap can create/update/delete Mwan3Rule CR of label sdewan-bucket-type=app-intent, but not label sdewan-bucket-type=basic.

Let me first describe the extended permission system and then explain how we implement it. In k8s, user or serviceAccount could be bonded to one or more roles. The roles defines the permissions, for example the following role defines that sdewan-test role can create/update Mwan3Rule CRs in default namespace. Also sdewan-testrole can get Mwan3Policy CRs.

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  annotations:
  name: sdewan-test
  namespace: default
rules:
- apiGroups:
  - ""
  resources:
  - mwan3rules
  verbs:
  - create
  - update
- apiGroups:
  - ""
  resources:
  - mwan3policies
  verbs:
  - get

We extend the Role with annotations. In the annotation, we can define labled based permissions. For example, the following role extends sdewan-test role permission: sdewan-test can only create/update Mwan3Rule CRs with label sdewan-bucket-type=app-intent or sdewan-bucket-type=k8s-service. Also it can only get Mwan3Policy CR with label sdewan-bucket-type=app-intent.

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  annotations:
    sdewan-bucket-type-permission: |-
      { "mwan3rules": ["app-intent", "k8s-service"],
      "mwan3policies": ["app-intent"] }
  name: sdewan-test
  namespace: default
rules:
- apiGroups:
  - ""
  resources:
  - mwan3rules
  verbs:
  - create
  - update
- apiGroups:
  - ""
  resources:
  - mwan3policies
  verbs:
  - get

We use admission webhook to implement the type level permission control. Let me describe how admission webhook in simple words. When k8s api receives a request, kube-api call webhook API before save the object into etcd. If the webhook returns allowed=true, kube-api continues to persistent the object into etcd. Otherwise, kube-api reject the request. The webhook can optional tell kube-api to update the object together with allowed=true returned. Webhook request body has a field named userInfo, it indicates who is making the k8s api request. With this field, we can implement the extended permission in webhook.

def mwan3rule_webhook_handle_permission(req admission.Request):
  userinfo = req["userInfo]
  mwan3rule_cr = decode(req)
  roles = k8s_client.get_role_from_user(userinfo)
  for role in roles:
    if mwan3rule_cr.labels.sdewan-bucket-type in role.annotation.sdewan-bucket-type-permission.mwan3rules:
      return {"allowd": True}
  return {"allowd": False}

ServiceRule controller (For next release)

We create a controller watches the services created in the cluster. For each service, it creates a FirewallDNAT CR. On controller startup, it makes a syncup to remove unused CRs.

References

• https://github.com/kubernetes-sigs/controller-runtime/blob/master/pkg/doc.go
• https://book.kubebuilder.io/reference/using-finalizers.html
• https://godoc.org/sigs.k8s.io/controller-runtime/pkg/predicate#example-Funcs
• https://godoc.org/sigs.k8s.io/controller-runtime/pkg/handler#example-EnqueueRequestsFromMapFunc