PCEI R4 Test Document
Introduction
This document describes tests that were performed for PCEI R4:
- PCEI Deployment Tests
- PCEI End-to-End Validation Tests
- BluVal Tests
Overall Test Architecture
Describe the components of Test set up
Test Bed
Test Framework
- PCEI Deployment Tests
Described in the PCEI R4 Installation Guide
- PCEI End-to-End Validation Tests
Described in the PCEI R4 End-to-End Validation Guide
- BluVal Tests
Described in the BluVal Test section of this document.
Traffic Generator
Not used.
For end-to-end functional verification, a simulated IoT Client was provided. Please refer to PCEI R4 End-to-End Validation Guide.
Test API description
Test APIs NOT USED (except BluVal Robot)
Akraino common tests
NOT PERFORMED
The Test inputs
Test Procedure
Expected output
Test Results
Blueprint extension tests
The Test inputs
Test | Description | Result | Reference |
---|---|---|---|
EMCO Deployment | Install EMCO Orchestrator | Pass | |
Edge Cluster Deployment | Deploy Edge K8S Clusters | Pass | PCEI R4 Installation Guide |
EMCO UI Access | Access EMCO UI | Pass | PCEI R4 Installation Guide |
Register Edge Cluster | Register Edge K8S Cluster with EMCO | Pass | |
Create Service/App | Create Service/App in EMCO for Azure IoT Edge, AWS GGC and PCEI Location API App | All PASS | |
Deploy Apps onto Edge Clusters | Deploy Azure IoT Edge, AWS GGC and PCEI Location API Apps onto Edge K8S Clusters | All PASS | |
Verify Azure IoT Edge with IoT Client | Start IoT Client, send messages to Azure IoT Edge. Monitor IoT Edge receive and decode messages | PASS | |
Verify AWS GGC App | Confirm AWS GGC App registers with AWS IoT Coire | PASS | |
Verify PCEI Location API App | Confirm PCEI Location API App is running and responding to requests | PASS |
Test Procedure
PCEI R4 End-to-End Validation Guide
Expected output
All tests pass
Test Results
Refer to sections of the following documents for detailed test results:
PCEI R4 End-to-End Validation Guide
Feature Project Tests
NOT PERFORMED
The Test inputs
Test Procedure
Expected output
Test Results
BluVal Tests
The Test inputs
BluVal Test Environment setup according to:
Test Procedure
- Deploy a Test VM
- Install Docker: https://docs.docker.com/engine/install/ubuntu/
- Clone BluVal Validation Framework into the Test VM:
- Copy SUT's .kube/config file and SSH key to the Test VM
- Configure validation environment:
cd validation vi bluval-pcei.yaml blueprint: name: pcei layers: - os - docker - k8s # Any hardware some basic tests os: &os_pcei - name: ltp what: ltp optional: "True" - name: cyclictest what: cyclictest optional: "True" - name: lynis what: lynis optional: "False" - name: vuls what: vuls optional: "False" docker: &docker_base - name: docker_bench what: docker_bench optional: "True" k8s: &k8s - name: conformance what: conformance optional: "False" - name: etcd_ha what: etcd_ha optional: "True" - name: kube-hunter what: kube-hunter optional: "False" cd /home/onaplab/validation/bluval vi volumes.yaml volumes: # location of the ssh key to access the cluster ssh_key_dir: local: '/home/onaplab/.ssh' target: '/root/.ssh' # location of the k8s access files (config file, certificates, keys) kube_config_dir: local: '/home/onaplab/kube' target: '/root/.kube/' # location of the customized variables.yaml custom_variables_file: local: '/home/onaplab/validation/tests/variables.yaml' target: '/opt/akraino/validation/tests/variables.yaml' # location of the bluval-<blueprint>.yaml file blueprint_dir: local: '/home/onaplab/validation/bluval' target: '/opt/akraino/validation/bluval' # location on where to store the results on the local jumpserver results_dir: local: '/home/onaplab/results' target: '/opt/akraino/results' # location on where to store openrc file openrc: local: '' target: '/root/openrc' # parameters that will be passed to the container at each layer layers: # volumes mounted at all layers; volumes specific for a different layer are below common: - custom_variables_file - blueprint_dir - results_dir hardware: - ssh_key_dir os: - ssh_key_dir networking: - ssh_key_dir docker: - ssh_key_dir k8s: - ssh_key_dir - kube_config_dir k8s_networking: - ssh_key_dir - kube_config_dir openstack: - openrc sds: sdn: vim: cd /home/onaplab/validation/tests vi variables.yaml ### Input variables cluster's master host host: 10.121.7.147 # cluster's master host address username: onaplab # login name to connect to cluster password: onaplab # login password to connect to cluster ssh_keyfile: /root/.ssh/id_rsa # Identity file for authentication
6. Run BluVal Robot:
cd bash validation/bluval/blucon.sh pcei
7. Install LFTOOLS:
sudo apt install python3-pip sudo python3 -m pip install -U pip sudo python3 -m pip install -U setuptools sudo -H pip3 install --ignore-installed PyYAML pip3 install lftools
8. Push BluVal Results to Akraino Nexus
# Create .netrc file vi .netrc machine nexus.akraino.org login <LF ID> password <Password> # Archive log files zip -r results.zip ./results # Push logs to Nexus NEXUS_PATH="/pcei/job/v2" NEXUS_URL="https://nexus.akraino.org/" /home/onaplab/.local/bin/lftools deploy nexus-zip $NEXUS_URL logs $NEXUS_PATH results.zip
Expected output
Test Results
https://nexus.akraino.org/content/sites/logs/pcei/job/v2/results/
Vuls
CVEs Found:
CVE | CVSS | URL | Exception |
---|---|---|---|
CVE-2016-1585 | 9.8 | https://nvd.nist.gov/vuln/detail/CVE-2016-1585 | Requested by another BP |
CVE-2017-18342 | 9.8 | https://nvd.nist.gov/vuln/detail/CVE-2017-18342 | Requested by another BP |
CVE-2017-8283 | 9.8 | https://nvd.nist.gov/vuln/detail/CVE-2017-8283 | Requested by PCEI. Approved |
CVE-2018-20839 | 9.8 | https://nvd.nist.gov/vuln/detail/CVE-2018-20839 | Requested by another BP |
CVE-2019-17041 | 9.8 | https://nvd.nist.gov/vuln/detail/CVE-2019-17041 | Requested by another BP |
CVE-2019-17042 | 9.8 | https://nvd.nist.gov/vuln/detail/CVE-2019-17042 | Requested by another BP |
CVE-2019-19814 | 9.3 | https://nvd.nist.gov/vuln/detail/CVE-2019-19814 | Requested by PCEI. Approved |
Lynis
Fixes for Lynis:
PASS_MAX_DAYS
https://askubuntu.com/questions/424216/what-is-password-aging-limits
vi /etc/login.defs
change
PASS_MAX_DAYS 1500
UNMASK 027
NOTE: changing the UNMASK value from default 022 to 027 resulted in the Lynis test suite erroring out. Exception was granted.
KRNL-6000
https://linux-audit.com/understand-and-configure-core-dumps-work-on-linux/
echo "fs.suid_dumpable=0" >> /etc/sysctl.conf
sysctl -p
sysctl -w kernel.dmesg_restrict=1
sysctl -w net.ipv4.conf.all.accept_source_route=0
K8S Conformance
Exception Requested:
Kube-Hunter
Vulnerabilities found
ID | Status |
---|---|
KHV002 | Fixed |
KHV005 | Fixed |
KHV050 | Fixed |
CAP_NET_RAW | Pending |
Fix for KHV002:
On SUT K8S Cluster:
kubectl replace -f - <<EOF apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: annotations: rbac.authorization.kubernetes.io/autoupdate: "false" labels: kubernetes.io/bootstrapping: rbac-defaults name: system:public-info-viewer rules: - nonResourceURLs: - /healthz - /livez - /readyz verbs: - get EOF
Fix for KHV005, KHV050
On SUT K8S Cluster:
kubectl replace -f - <<EOF apiVersion: v1 kind: ServiceAccount metadata: name: default namespace: default automountServiceAccountToken: false EOF
Test Dashboards
Single pane view of how the test score looks like for the Blue print.
Test Group | Total Tests | Pass | Fail |
---|---|---|---|
Blueprint Extension Tests | 9 | 9 | 0 |
Vuls | 1 | 1 | 0 |
Lynis | 1 | 1 | 0 |
K8S Conformance | 1 | 0 | 1 |
Kube-Hunter | 1 | 1 | 0 |
Additional Testing
None
Bottlenecks/Errata
Please refer to PCEI R4 Release Notes