Table of Contents
outline true

Introduction

...

To add more Jenkins slave nodes, please follow the akriano jenkins guide

To setup private jenkins, please refer to the README.md under icn/ci/

...

Virtual deployment

Test Framework

All components are tested with end-to-end testing

...

• The bpa_verifier.sh script get the MAC addresses and IP addresses of the 2 VMs provisioned by metal3, then creates a fake DHCP lease file using the IP address and MAC address information. It also creates a provisioning CR using the MAC address information
• The script the creates an ssh secret key using the ssh keys of the test host, applies the the provisioning CR
• The script busy loops till the KUD installation job completes or fails. If it completes successfully, it does a curl command using the authentication info of the new cluster to confirm if it was successful or not. On completing all the steps, it does a teardown where it deletes everything it created.

BPA

...

• Virtlet VM provisioning is tested as part of the 'verify_nestedk8s' testcase. K8s is first launched with Virtlet using KuD scripts after the prerequisite packages are installed.
• Next, BPA operator and multicloud-k8s docker images are built and BPA operator scripts including the provisioning_crd are deployed. Then, the Virtlet VM E2E script is launched which does the following:
• It creates a new flannel network definition for assigning mac address to VMs, creates test Virtlet VM, creates a provisioning CR for the same mac address. BPA operator then provisions the Virtlet VM by initiating the KuD installer job which installs K8s in the Virtlet VM.

BPA Rest Agent

• Test script, e2e_test.sh, creates dummy image file, creates test JSON file, checks bpa rest agent status, issues POST, GET, and PATCH requests sequentially.
• Next, e2e_test.sh checks uploaded MinIO image object size, and calls DELETE.
• If the script fails at any point then verification was unsuccessful.

Kubernetes Deployment (KuD)

KuD has test cases to verify if the add-ons are running correctly. All the test cases can be found in tests directory in the multicloud-k8s project. For each of these, we bring up the deployment that is specific to the addon, perform add-on specific actions on the pod related to the deployment

Multus:

• Multus CNI is a container network interface (CNI) plugin for Kubernetes that enables attaching multiple network interfaces to pods. This is accomplished by Multus acting as a "meta-plugin", a CNI plugin that can call multiple other CNI plugins.
• A 'NetworkAttachmentDefinition' is used to set up the network attachment, i.e. secondary interface for the pod. 
• A pod is created with requesting specific network annotations with bridge CNI to create multiple interfaces. When the pod is up and running, we can attach to it to check the network interfaces on it by running ip a command

Virtlet:

• Virtlet is a Kubernetes runtime server which allows you to run VM workloads, based on QCOW2 images.
• We create a Virtlet VM pod-spec file adhering to the standards for virtlet to create a VM in a K8S env. 
• The pod spec file is applied to bring up Virtlet deployment and make sure it is running. We attach to the pod and test to make sure the VM is running fine by connecting to it and checking details. 

OVN4NFV: 

• We use the Multus CNI container to create multiple ovn interfaces using OVN.
• After the pod is up and running we will be able to attach to the pod and check for multiple interfaces created inside the container. 

Node feature Discovery

• Node feature discovery for Kubernetes detects hardware features available on each node in a Kubernetes cluster and advertises those features using node labels.
• Create a pod with specific label information in the case the pods are scheduled only on nodes whose Major Kernal version is 3 and above. Since the NFD Master and worker Daemonset is already running, the master has all the label information about the nodes which is collected by the worker.
• If the O.S version matches, the PoD will be scheduled and up. Otherwise, the Pod will be in a pending state in case there are no nodes with matching labels that are requested by the pod

SRIOV

• The SRIOV network device plugin is Kubernetes device plugin for discovering and advertising SRIOV network virtual functions (VFs) in a Kubernetes host. 
• We first determine which hosts are SRIOV capable and install the drivers on them and run the DaemonSet and register Network attachment definition
• On an SRIOV capable hosts, we can get the resources for the node before we run the pod. When we run the test case, there is a request for a VF from the pod, therefore the number of resources for the node is increased.

QAT 

Le Yao (Deactivated)

CMK

Le Yao (Deactivated)

Optane PM

• The Optane PM plugin is Kubernetes CSI plugin and driver with storage volume provisioning for Kubernetes applications.
• Check whether the Optane PM hardware: NVDIMM is existed, if not skip the validation.
• Configure the Optane PM plugin in KUD, and create StorageClass and PersistentVolumeClaim which used by Kubernetes application, check whether the PVC is bound, if yes, the Optane PM volume created and bound to PVC and used by application, validation passed.

SDEWAN

• Use Kud to setup 3 clusters (traffic hub, edge1, edge2)
• Create SDEWAN CNF instance and dummy pod in edge1, SDEWAN CNF instance and httpbin pod in edge2
• Configure traffic hub as responder to provide IP addresses to any authenticated party requesting for IP addresses.
• Configure edge1 and edge2 IPSec configuration to get the IP addresses.
• Establish edge1 tunnel to traffic hub, edge2 tunnel to hub, and hub policy for traffic between edge1 and edge2
• Establish SNAT rule in edge1 and DNAT rule in edge2 to enable tcp connection from edge1 to edge2's httpbin service
• Verify curl command is successful from edge1 dummy pod to edge2's httpbin service

Openness

• Install EAA helm charts through ONAP4K8S in the edge location.
• Install Openness simple EAA producer and simple EAA consumer through ONAP4K8S
• Verify EAA consumer can consume the service provided by EAA producer.

ONAP4K8s:

• ONAP4K8s testing check the health connectivity ONAP Micro service, once it is installed

cFW:

• Cloud Native FW having multiple components such packetgen generator, sink and cFW
  • Packet generator: Sends packets to the packet sink through the firewall. This includes a script that periodically generates different volumes of traffic inside the container
  • Firewall: Reports the volume of traffic passing though to the ONAP DCAE collector.
  • Traffic sink: Displays the traffic volume that lands at the sink container using the link  node port through your browser and enable automatic page refresh by clicking the "Off" button. You can see the traffic volume in the charts.Image Removed

EdgeX Foundry:

EdgeX Foundry helm chart are installed through ONAP in the edge location. Test case ensure that all the EdgeX Framework containers are up and running

BluVal Testing

Igor D.C.

Rest Agent

• Test script, e2e_test.sh, creates dummy image file, creates test JSON file, checks bpa rest agent status, issues POST, GET, and PATCH requests sequentially.
• Next, e2e_test.sh checks uploaded MinIO image object size, and calls DELETE.
• If the script fails at any point then verification was unsuccessful.

Kubernetes Deployment (KuD)

KuD has test cases to verify if the add-ons are running correctly. All the test cases can be found in tests directory in the multicloud-k8s project. For each of these, we bring up the deployment that is specific to the addon, perform add-on specific actions on the pod related to the deployment

Multus:

• Multus CNI is a container network interface (CNI) plugin for Kubernetes that enables attaching multiple network interfaces to pods. This is accomplished by Multus acting as a "meta-plugin", a CNI plugin that can call multiple other CNI plugins.
• A 'NetworkAttachmentDefinition' is used to set up the network attachment, i.e. secondary interface for the pod. 
• A pod is created with requesting specific network annotations with bridge CNI to create multiple interfaces. When the pod is up and running, we can attach to it to check the network interfaces on it by running ip a command

Virtlet:

• Virtlet is a Kubernetes runtime server which allows you to run VM workloads, based on QCOW2 images.
• We create a Virtlet VM pod-spec file adhering to the standards for virtlet to create a VM in a K8S env. 
• The pod spec file is applied to bring up Virtlet deployment and make sure it is running. We attach to the pod and test to make sure the VM is running fine by connecting to it and checking details. 

OVN4NFV: 

• We use the Multus CNI container to create multiple ovn interfaces using OVN.
• After the pod is up and running we will be able to attach to the pod and check for multiple interfaces created inside the container. 

Node feature Discovery

• Node feature discovery for Kubernetes detects hardware features available on each node in a Kubernetes cluster and advertises those features using node labels.
• Create a pod with specific label information in the case the pods are scheduled only on nodes whose Major Kernal version is 3 and above. Since the NFD Master and worker Daemonset is already running, the master has all the label information about the nodes which is collected by the worker.
• If the O.S version matches, the PoD will be scheduled and up. Otherwise, the Pod will be in a pending state in case there are no nodes with matching labels that are requested by the pod

SRIOV

• The SRIOV network device plugin is Kubernetes device plugin for discovering and advertising SRIOV network virtual functions (VFs) in a Kubernetes host. 
• We first determine which hosts are SRIOV capable and install the drivers on them and run the DaemonSet and register Network attachment definition
• On an SRIOV capable hosts, we can get the resources for the node before we run the pod. When we run the test case, there is a request for a VF from the pod, therefore the number of resources for the node is increased.

QAT 

• KUD identify if there are QAT devices in the host and decide whether to deploy QAT device plugin into Kubernetes cluster.
• The QAT device plugin discovers and advertises QAT virtual functions (VFs) to Kubernetes cluster.
• KUD assign 1 QAT VF to the Kernel workloads. After the assginment finished, the Allocated resources in node description will increase.

CMK

• CPU Manager for Kubernetes provides cpu pinning for K8s workloads. In KUD, there are two test cases for the exclusive and shared cpu pools testing.

Optane PM

• The Optane PM plugin is Kubernetes CSI plugin and driver with storage volume provisioning for Kubernetes applications.
• Check whether the Optane PM hardware: NVDIMM is existed, if not skip the validation.
• Configure the Optane PM plugin in KUD, and create StorageClass and PersistentVolumeClaim which used by Kubernetes application, check whether the PVC is bound, if yes, the Optane PM volume created and bound to PVC and used by application, validation passed.

SDEWAN

• Use Kud to setup 3 clusters (sdewan-hub, edge-a, edge-b)
• Create SDEWAN CNF instance and dummy pod(using httpbin instead) in edge-a, SDEWAN CNF instance and httpbin pod in edge-b
• Configure sdewan-hub as responder to provide virtual IP addresses to any authenticated party requesting for IP addresses.
• Configure edge-a and edge-b IPSec configuration to get the IP addresses.
• Establish edge-a tunnel to sdewan-hub, edge-b tunnel to sdewan-hub, and hub XFRM policies will automatically route traffic between edge-a and edge-b
• Establish SNAT rule in edge-a and DNAT rule in edge-b to enable tcp connection from edge-a to edge-b's httpbin service.
• Verify curl command is successful from edge-a dummy pod(using httpbin instead) to edge-b's httpbin service. The function of the curl command is to return back the ip address of the requester.

Image Added

Openness

• Install EAA helm charts through ONAP4K8S in the edge location.
• Install Openness simple EAA producer and simple EAA consumer through ONAP4K8S
• Verify EAA consumer can consume the service provided by EAA producer.

ONAP4K8s:

• ONAP4K8s testing check the health connectivity ONAP Micro service, once it is installed

cFW:

• Cloud Native FW having multiple components such packetgen generator, sink and cFW
  • Packet generator: Sends packets to the packet sink through the firewall. This includes a script that periodically generates different volumes of traffic inside the container
  • Firewall: Reports the volume of traffic passing though to the ONAP DCAE collector.
  • Traffic sink: Displays the traffic volume that lands at the sink container using the link  node port through your browser and enable automatic page refresh by clicking the "Off" button. You can see the traffic volume in the charts.Image Added

EdgeX Foundry:

EdgeX Foundry helm chart are installed through ONAP in the edge location. Test case ensure that all the EdgeX Framework containers are up and running

BluVal Testing

Status as of May 28th 2020:

CI logs: 

The gerrit comments contains the CI log url. All the CI logs are under this folder ICN : https://jenkins.akraino.org/view/icn/job/icn-master-verify/

Latest CI logs

CD Logs:

ICN Master Baremetal Deployment Verifier

ICN Master Baremetal Virtual Deployment for Hardware verificationVerifer

ICN Master Hardware Baremetal Deployment Virtlet nested K8s VerifierVerifer

ICN SDEWAN Master Virtual Deployment VeriferEnd2End Testing

ICN Master Virtual Deployment Virtlet nested K8s Optane Hardware Baremetal Deployment Verifier

Test Dashboards

All the testing results are in logs

...