1 Introduction
2 Overall Test Architecture
- 2.1 Test Bed
- 2.2 Test Framework
- 2.3 Traffic Generator
3 Test API description
- 3.1 Akraino common tests
  - 3.1.1 The Test inputs
  - 3.1.2 Test Procedure
  - 3.1.3 Expected output
  - 3.1.4 Test Results
- 3.2 Blueprint extension tests
  - 3.2.1 The Test inputs
  - 3.2.2 Test Procedure
  - 3.2.3 Expected output
  - 3.2.4 Test Results
- 3.3 Feature Project Tests
  - 3.3.1 The Test inputs
  - 3.3.2 Test Procedure
  - 3.3.3 Expected output
  - 3.3.4 Test Results
- 3.4 BluVal Tests
  - 3.4.1 The Test inputs
  - 3.4.2 Test Procedure
  - 3.4.3 Expected output
  - 3.4.4 Test Results
    - 3.4.4.1 Vuls
    - 3.4.4.2 Lynis
    - 3.4.4.3 K8S Conformance
    - 3.4.4.4 Kube-Hunter
4 Test Dashboards
5 Additional Testing
6 Bottlenecks/Errata

Introduction

This document describes tests that were performed for PCEI R4:

PCEI Deployment Tests
PCEI End-to-End Validation Tests
BluVal Tests

Overall Test Architecture

Describe the components of Test set up

Test Bed

Test Framework

PCEI Deployment Tests

Described in the PCEI R4 Installation Guide

PCEI End-to-End Validation Tests

Described in the PCEI R4 End-to-End Validation Guide

BluVal Tests

Described in the BluVal Test section of this document.

Traffic Generator

Not used.

For end-to-end functional verification, a simulated IoT Client was provided. Please refer to PCEI R4 End-to-End Validation Guide.

Test API description

Test APIs NOT USED (except BluVal Robot)

Akraino common tests

NOT PERFORMED

The Test inputs

Test Procedure

Expected output

Test Results

Blueprint extension tests

The Test inputs

Test	Description	Result	Reference

Test	Description	Result	Reference
EMCO Deployment	Install EMCO Orchestrator	Pass	PCEI R4 Installation Guide
Edge Cluster Deployment	Deploy Edge K8S Clusters	Pass	PCEI R4 Installation Guide
EMCO UI Access	Access EMCO UI	Pass	PCEI R4 Installation Guide
Register Edge Cluster	Register Edge K8S Cluster with EMCO	Pass	PCEI R4 End-to-End Validation Guide
Create Service/App	Create Service/App in EMCO for Azure IoT Edge, AWS GGC and PCEI Location API App	All PASS	PCEI R4 End-to-End Validation Guide
Deploy Apps onto Edge Clusters	Deploy Azure IoT Edge, AWS GGC and PCEI Location API Apps onto Edge K8S Clusters	All PASS	PCEI R4 End-to-End Validation Guide
Verify Azure IoT Edge with IoT Client	Start IoT Client, send messages to Azure IoT Edge. Monitor IoT Edge receive and decode messages	PASS	PCEI R4 End-to-End Validation Guide
Verify AWS GGC App	Confirm AWS GGC App registers with AWS IoT Coire	PASS	PCEI R4 End-to-End Validation Guide
Verify PCEI Location API App	Confirm PCEI Location API App is running and responding to requests	PASS	PCEI R4 End-to-End Validation Guide

Test Procedure

PCEI R4 Installation Guide

PCEI R4 End-to-End Validation Guide

Expected output

All tests pass

Test Results

Refer to sections of the following documents for detailed test results:

PCEI R4 Installation Guide

PCEI R4 End-to-End Validation Guide

Feature Project Tests

NOT PERFORMED

The Test inputs

Test Procedure

Expected output

Test Results

BluVal Tests

The Test inputs

BluVal Test Environment setup according to:

Bluval User Guide

Test Procedure

Deploy a Test VM
Install Docker: https://docs.docker.com/engine/install/ubuntu/
Clone BluVal Validation Framework into the Test VM:
Copy SUT's .kube/config file and SSH key to the Test VM
Configure validation environment:

cd validation
vi bluval-pcei.yaml
blueprint:
    name: pcei 
    layers:
        - os
        - docker
        - k8s
    # Any hardware some basic tests

    os: &os_pcei
        -
            name: ltp
            what: ltp
            optional: "True"
        -
            name: cyclictest
            what: cyclictest
            optional: "True"
        -
            name: lynis
            what: lynis
            optional: "False"
        -
            name: vuls
            what: vuls
            optional: "False"

    docker: &docker_base
        -
            name: docker_bench
            what: docker_bench
            optional: "True"

    k8s: &k8s
        -
            name: conformance
            what: conformance
            optional: "False"
        -
            name: etcd_ha
            what: etcd_ha
            optional: "True"
        -
            name: kube-hunter
            what: kube-hunter
            optional: "False"
            
cd /home/onaplab/validation/bluval
vi volumes.yaml
volumes:
    # location of the ssh key to access the cluster
    ssh_key_dir:
        local: '/home/onaplab/.ssh'
        target: '/root/.ssh'
    # location of the k8s access files (config file, certificates, keys)
    kube_config_dir:
        local: '/home/onaplab/kube'
        target: '/root/.kube/'
    # location of the customized variables.yaml
    custom_variables_file:
        local: '/home/onaplab/validation/tests/variables.yaml'
        target: '/opt/akraino/validation/tests/variables.yaml'
    # location of the bluval-<blueprint>.yaml file
    blueprint_dir:
        local: '/home/onaplab/validation/bluval'
        target: '/opt/akraino/validation/bluval'
    # location on where to store the results on the local jumpserver
    results_dir:
        local: '/home/onaplab/results'
        target: '/opt/akraino/results'
    # location on where to store openrc file
    openrc:
        local: ''
        target: '/root/openrc'

# parameters that will be passed to the container at each layer
layers:
    # volumes mounted at all layers; volumes specific for a different layer are below
    common:
        - custom_variables_file
        - blueprint_dir
        - results_dir
    hardware:
        - ssh_key_dir
    os:
        - ssh_key_dir
    networking:
        - ssh_key_dir
    docker:
        - ssh_key_dir
    k8s:
        - ssh_key_dir
        - kube_config_dir
    k8s_networking:
        - ssh_key_dir
        - kube_config_dir
    openstack:
        - openrc
    sds:
    sdn:
    vim:

cd /home/onaplab/validation/tests
vi variables.yaml
### Input variables cluster's master host
host: 10.121.7.147             # cluster's master host address
username: onaplab            # login name to connect to cluster
password: onaplab         # login password to connect to cluster
ssh_keyfile: /root/.ssh/id_rsa        # Identity file for authentication

6. Run BluVal Robot:

cd
bash validation/bluval/blucon.sh pcei

7. Install LFTOOLS:

sudo apt install python3-pip
sudo python3 -m pip install -U pip
sudo python3 -m pip install -U setuptools
sudo -H pip3 install --ignore-installed PyYAML


pip3 install lftools

8. Push BluVal Results to Akraino Nexus

# Create .netrc file
vi .netrc
machine nexus.akraino.org
login <LF ID>
password <Password>

# Archive log files
zip -r results.zip ./results

# Push logs to Nexus
NEXUS_PATH="/pcei/job/v2"
NEXUS_URL="https://nexus.akraino.org/"
/home/onaplab/.local/bin/lftools deploy nexus-zip $NEXUS_URL logs $NEXUS_PATH results.zip

Expected output

Test Results

https://nexus.akraino.org/content/sites/logs/pcei/job/v2/results/

Vuls

CVEs Found:

CVE	CVSS	URL	Exception

CVE	CVSS	URL	Exception
CVE-2016-1585	9.8	https://nvd.nist.gov/vuln/detail/CVE-2016-1585	Requested by another BP
CVE-2017-18342	9.8	https://nvd.nist.gov/vuln/detail/CVE-2017-18342	Requested by another BP
CVE-2017-8283	9.8	https://nvd.nist.gov/vuln/detail/CVE-2017-8283	Requested by PCEI. Approved
CVE-2018-20839	9.8	https://nvd.nist.gov/vuln/detail/CVE-2018-20839	Requested by another BP
CVE-2019-17041	9.8	https://nvd.nist.gov/vuln/detail/CVE-2019-17041	Requested by another BP
CVE-2019-17042	9.8	https://nvd.nist.gov/vuln/detail/CVE-2019-17042	Requested by another BP
CVE-2019-19814	9.3	https://nvd.nist.gov/vuln/detail/CVE-2019-19814	Requested by PCEI. Approved

Lynis

Fixes for Lynis:

BOOT-5122
https://vineetcic.medium.com/how-to-set-grub-password-in-ubuntu-18-03-password-protect-boot-loader-ecb5db184054

PASS_MAX_DAYS
https://askubuntu.com/questions/424216/what-is-password-aging-limits

vi /etc/login.defs
change
PASS_MAX_DAYS 1500
UNMASK 027

NOTE: changing the UNMASK value from default 022 to 027 resulted in the Lynis test suite erroring out. Exception was granted.

KRNL-6000
https://linux-audit.com/understand-and-configure-core-dumps-work-on-linux/
echo "fs.suid_dumpable=0" >> /etc/sysctl.conf
sysctl -p

sysctl -w kernel.dmesg_restrict=1

sysctl -w net.ipv4.conf.all.accept_source_route=0

K8S Conformance

Exception Requested:

Kube-Hunter

Vulnerabilities found

ID	Status

ID	Status
KHV002	Fixed
KHV005	Fixed
KHV050	Fixed
CAP_NET_RAW	Pending

Fix for KHV002:

On SUT K8S Cluster:

kubectl replace -f - <<EOF
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  annotations:
    rbac.authorization.kubernetes.io/autoupdate: "false"
  labels:
    kubernetes.io/bootstrapping: rbac-defaults
  name: system:public-info-viewer
rules:
- nonResourceURLs:
  - /healthz
  - /livez
  - /readyz
  verbs:
  - get
EOF

Fix for KHV005, KHV050

On SUT K8S Cluster:

kubectl replace -f - <<EOF
apiVersion: v1
kind: ServiceAccount
metadata:
  name: default
  namespace: default
automountServiceAccountToken: false
EOF

Test Dashboards

Single pane view of how the test score looks like for the Blue print.

Test Group	Total Tests	Pass	Fail

Test Group	Total Tests	Pass	Fail
Blueprint Extension Tests	9	9	0
Vuls	1	1	0
Lynis	1	1	0
K8S Conformance	1	0	1
Kube-Hunter	1	1	0

Additional Testing

None

Bottlenecks/Errata

Please refer to PCEI R4 Release Notes

PCEI R4 Test Document

Introduction

Overall Test Architecture

Test Bed

Test Framework

Traffic Generator

Test API description

Akraino common tests

The Test inputs

Test Procedure

Expected output

Test Results

Blueprint extension tests

The Test inputs

Test Procedure

Expected output

Test Results

Feature Project Tests

The Test inputs

Test Procedure

Expected output

Test Results

BluVal Tests

The Test inputs

Test Procedure

Expected output

Test Results

Vuls

Lynis

K8S Conformance

Kube-Hunter

Test Dashboards

Additional Testing

Bottlenecks/Errata