Page Comparison

Introduction

This document describes the blueprint test environment for the Smart Data Transaction for CPS blueprint. The test results and logs are posted in the Akraino Nexus at the link below:

https://nexus.akraino.org/content/sites/logs/fujitsu/job/

Akarino Test Group Information

N/A

Testing has been carried out at Fujitsu Limited labs without any Akraino Test Working Group resources.

Overall Test Architecture

Tests are carried out on the architecture  shown in the diagram below.

Image Removed

Test Bed

The test bed consists of 4 VMs running on x86 hardware, performing deploy and ci/cd and build and master node roles, two edge nodes on ARM64 (Jetson Nano) hardware, and two sensor nodes on ARM32 (Raspberry Pi) hardware.

...

CI/CD

...

Introduction

This document describes the blueprint test environment for the Smart Data Transaction for CPS blueprint. The test results and logs are posted in the Akraino Nexus at the link below:

https://nexus.akraino.org/content/sites/logs/fujitsu/job/sdt/r7

Akarino Test Group Information

N/A

Testing has been carried out at Fujitsu Limited labs without any Akraino Test Working Group resources.

Overall Test Architecture

Tests are carried out on the architecture  shown in the diagram below.

Image Added

Test Bed

The test bed consists of 4 VMs running on x86 hardware, performing deploy and ci/cd and build and master node roles, two edge nodes on ARM64 (Jetson Nano) hardware, and two sensor nodes on ARM32 (Raspberry Pi) hardware.

...

Traffic Generator

N/A

Test API description

Before running the tests below, ensure that the configuration in the chapter Verifying the Setup of Smart Data Transaction for CPS R7 Installation Guide has been implemented.

CI/CD Regression Tests: Node Setup

...

The robot command should report success for all test cases.

Test Results

...

Nexus URL: 

Image Removed

Pass (1/1 test case)

:  https://nexus.akraino.org/content/sites/logs/fujitsu/job/sdt/r7/lfedge-install/14/

Image Added

Pass (1/1 test case)

CI/CD Regression Tests: Images Build & Push

...

The robot command should report success for all test cases.

Test Results

Nexus URL: https://nexus.akraino.org/content/sites/logs/fujitsu/job/sdt/r7/lfedge-build/5

Pass (2/2 test cases)

CI/CD Regression Tests: Cluster Setup & Teardown

...

The robot command should report success for all test cases.

Test Results

Nexus URL: 

Image Removed

Pass (4/4 https://nexus.akraino.org/content/sites/logs/fujitsu/job/sdt/r7/lfedge-cluster/6

Image Added

Pass (4/4 test cases)

CI/CD Regression Tests: EdgeX Services

...

The robot command should report success for all test cases.

Test Results

Nexus URL: https://nexus.akraino.org/content/sites/logs/fujitsu/job/sdt/r7/edgex-install/7/

Pass (8/8 test cases)

CI/CD Regression Tests: Camera Device Service

...

The Robot Framework should report success for all test cases

Test Results

Nexus URL: 

Image Removed

Pass (9/9 test caseshttps://nexus.akraino.org/content/sites/logs/fujitsu/job/sdt/r7/camera/10

Image Added

Pass (9/9 test cases)

Feature Project Tests

N/A

...

1. Copy the folder ~/.kube from Kubernetes master node to Build VM
2. Create SSH Key on Build VM to access Kubernetes master node

Vuls

We use Ubuntu 20.04, and behind a proxy, so we run Vuls test as follows:

1. Create directory

   $ mkdir ~/vuls $ cd ~/vuls $ mkdir go-cve-dictionary-log goval-dictionary-log gost-log

   
   

2. Fetch NVD

   $ docker run --rm -it \ -v $PWD:/go-cve-dictionary \ -v $PWD/go-cve-dictionary-log:/var/log/go-cve-dictionary \ vuls/go-cve-dictionary fetch nvd --http-proxy $http_proxy

   
   

3. Fetch OVAL

   $ docker run --rm -it \ -v $PWD:/goval-dictionary \ -v $PWD/goval-dictionary-log:/var/log/goval-dictionary \ vuls/goval-dictionary fetch ubuntu 14 16 17 18 19 20 --http-proxy $http_proxy

   
   

4. Fetch gost

   $ docker run --rm -iit \ -v $PWD:/goste http_proxy=$http_proxy \ -e https_proxy=$https_proxy \ -v $PWD:/gost \ -v $PWD/gost-log:/var/log/gost \ vuls/gost fetch ubuntu --http-proxy $http_proxy

   
   

5. Create config.toml

   [servers] [servers.master] host = "192.168.51.22" port = "22" user = "test-user" keyPath = "/root/.ssh/id_rsa" # path to ssh private key in docker

   
   

6. Start vuls container to run tests

   $ docker run --rm -it \ -v ~/.ssh:/root/.ssh:ro \ -v $PWD:/vuls \ -v $PWD/vuls-log:/var/log/vuls \ -v /etc/localtime:/etc/localtime:ro \ -v /etc/timezone:/etc/timezone:ro \ vuls/vuls scan \ -config=./config.toml \    --http-proxy $http_proxy

   
   

7. Get the report

   $ docker run --rm -it \ -v ~/.ssh:/root/.ssh:ro \ -v $PWD:/vuls \ -v $PWD/vuls-log:/var/log/vuls \ -v /etc/localtime:/etc/localtime:ro \ vuls/vuls report \ -format-list \ -config=./config.toml \  --http-proxy $http_proxy

   
   

Lynis/Kube-Hunter

1. Create ~/validation/bluval/bluval-sdtfc.yaml to customize the Test

   blueprint: name: sdtfc layers: - osk8s - k8sos osk8s: &osk8s - name: lyniskube-hunter what: lyniskube-hunter optional: "False" k8sos: &k8sos - name: kube-hunterlynis what: kube-hunter lynis optional: "False"

   
   

2. Update ~/validation/bluval/volumes.yaml file

   volumes: # location of the ssh key to access the cluster ssh_key_dir: local: '/home/ubuntu/.ssh' target: '/root/.ssh' # location of the k8s access files (config file, certificates, keys) kube_config_dir: local: '/home/ubuntu/kube' target: '/root/.kube/' # location of the customized variables.yaml custom_variables_file: local: '/home/ubuntu/validation/tests/variables.yaml' target: '/opt/akraino/validation/tests/variables.yaml' # location of the bluval-<blueprint>.yaml file blueprint_dir: local: '/home/ubuntu/validation/bluval' target: '/opt/akraino/validation/bluval' # location on where to store the results on the local jumpserver results_dir: local: '/home/ubuntu/results' target: '/opt/akraino/results' # location on where to store openrc file openrc: local: '' target: '/root/openrc' # parameters that will be passed to the container at each layer layers: # volumes mounted at all layers; volumes specific for a different layer are below common: - custom_variables_file - blueprint_dir - results_dir hardware: - ssh_key_dir os: - ssh_key_dir networking: - ssh_key_dir docker: - ssh_key_dir k8s: - ssh_key_dir - kube_config_dir k8s_networking: - ssh_key_dir - kube_config_dir openstack: - openrc sds: sdn: vim:

   Update ~/validation/tests/variables.yaml file

   ### Input variables cluster's master host
   host: <IP Address>             # cluster's master host address
   username: <username>            # login name to connect to cluster
   password: <password>         # login password to connect to cluster
   ssh_keyfile: /root/.ssh/id_rsa        # Identity file for authentication
   

   Run Blucon

   $ bash validation/bluval/blucon.sh sdtfc
   

Expected output

BluVal tests should report success for all test cases.

Test Results

Vuls results (manual) Nexus URL: 

Lynis results (manual) Nexus URL: 

Kube-Hunter results Nexus URL: 

Vuls

Nexus URL: 

There are 8 CVEs with a CVSS score >= 9.0. These are exceptions requested here:

Release 5: Akraino CVE Vulnerability Exception Request

...

Lynis

Nexus URL (run via Bluval, without fixes): https://nexus.akraino.org/content/sites/logs/fujitsu/job/sdt-bluval/2/

Nexus URL (manual run, with fixes): https://nexus.akraino.org/content/sites/logs/fujitsu/job/sdt-lynis/2/

Image Removed

The initial results compare with the Lynis Incubation: PASS/FAIL Criteria, v1.0 as follows.

The Lynis Program Update test MUST pass with no errors.

2022-03-04 15:33:28 Test: Checking for program update...
2022-03-04 15:33:31 Current installed version  : 301
2022-03-04 15:33:31 Latest stable version      : 307
2022-03-04 15:33:31 Minimum required version   : 297
2022-03-04 15:33:31 Result: newer Lynis release available!
2022-03-04 15:33:31 Suggestion: Version of Lynis outdated, consider upgrading to the latest version [test:LYNIS] [details:-] [solution:-]

Fix: Download and run the latest Lynis directly on SUT.

Steps To Implement Security Scan Requirements#InstallandExecute

The following list of tests MUST complete as passing

...

Result: password minimum age is not configured

Suggestion: Configure minimum password age in /etc/login.defs [test:AUTH-9286]

...

Result: found umask 022, which could be improved

Suggestion: Default umask in /etc/login.defs could be more strict like 027 [test:AUTH-9328]

...

Result: SSH has no specific user or group limitation. Most likely all valid users can SSH to this machine.

Hardening: assigned partial number of hardening points (0 of 1).

...

Test: checking for file /etc/network/if-up.d/ntpdate

Result: file /etc/network/if-up.d/ntpdate does not exist

...

Hardening: assigned maximum number of hardening points for this item (3).

...

Set recommended value in /etc/sysctl.d/90-lynis-hardening.conf and disable apport in /etc/default/apport

...

Result: sysctl key net.ipv4.conf.default.accept_source_route has a different value than expected in scan profile. Expected=0, Real=1

...

Result: found installed compiler. See top of logfile which compilers have been found or use /usr/bin/grep to filter on 'compiler'

Hardening: assigned partial number of hardening points (1 of 3).

...

1. common: - custom_variables_file - blueprint_dir - results_dir hardware: - ssh_key_dir os: - ssh_key_dir networking: - ssh_key_dir docker: - ssh_key_dir k8s: - ssh_key_dir - kube_config_dir k8s_networking: - ssh_key_dir - kube_config_dir openstack: - openrc sds: sdn: vim:

   
   

2. Update ~/validation/tests/variables.yaml file

   ### Input variables cluster's master host host: <IP Address> # cluster's master host address username: <username> # login name to connect to cluster password: <password> # login password to connect to cluster ssh_keyfile: /root/.ssh/id_rsa # Identity file for authentication

   
   

3. Run Blucon

   $ bash validation/bluval/blucon.sh sdtfc

   
   

Expected output

BluVal tests should report success for all test cases.

Test Results

Vuls results (manual) Nexus URL: https://nexus.akraino.org/content/sites/logs/fujitsu/job/sdt/r7/sdt-vuls/2/

Lynis results (manual) Nexus URL: https://nexus.akraino.org/content/sites/logs/fujitsu/job/sdt/r7/sdt-lynis/2/

Kube-Hunter results Nexus URL: https://nexus.akraino.org/content/sites/logs/fujitsu/job/sdt/r7/sdt-bluval/1/

Vuls

Nexus URL: https://nexus.akraino.org/content/sites/logs/fujitsu/job/sdt/r7/sdt-vuls/2/

There are 4 CVEs with a CVSS score >= 9.0. These are exceptions requested here:

Release 7: Akraino CVE and KHV Vulnerability Exception Request

Lynis

Nexus URL (manual run, with fixes): https://nexus.akraino.org/content/sites/logs/fujitsu/job/sdt/r7/lynis/3/

The results compare with the Lynis Incubation: PASS/FAIL Criteria, v1.0 as follows.

The Lynis Program Update test MUST pass with no errors.

2022

...

The following list of tests MUST complete as passing

...

Result: max password age is 180 days
Hardening: assigned maximum number of hardening points for this item (3).

...

Result: umask is 027, which is fine
Hardening: assigned maximum number of hardening points for this item (2).

...

Result: SSH is limited to a specific set of users, which is good
Hardening: assigned maximum number of hardening points for this item (2).

...

-09-14 16:19:49 Test: Checking for program update...
2022-09-14 16:19:49 Result: Update check failed. No network connection?
2022-09-14 16:19:49 Info: to perform an automatic update check, outbound DNS connections should be allowed (TXT record).
2022-09-14 16:19:49 Suggestion: This release is more than 4 months old. Check the website or GitHub to see if there is an update available. [test:LYNIS] [details:-] [solution:-]

The test environment is a proxied private network inside the Fujitsu corporate network which does not allow direct DNS lookups using tools such as dig. Therefore the update check cannot be performed automatically.

The latest version of Lynis, 3.0.8 at time of execution, was downloaded and run directly on the SUT. See the link below:

Steps To Implement Security Scan Requirements#InstallandExecute

The following list of tests MUST complete as passing

The post-fix manual logs can be found at https://nexus.akraino.org/content/sites/logs/fujitsu/job/sdt-lynis/3/.

Kube-Hunter

Nexus URL (initial run without fixes): https://nexus.akraino.org/content/sites/logs/fujitsu/job/sdt-bluval/1/

Nexus URL (with fixes): https://nexus.akraino.org/content/sites/logs/fujitsu/job/sdt-bluval/2/

Image Removed

There are 5 Vulnerabilities.

• KHV002
• KHV005
• KHV050
• CAP_NET_RAW Enabled
• Access to pod's secrets

Fix for KHV002

...

$ kubectl replace -f - <<EOF
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  annotations:
    rbac.authorization.kubernetes.io/autoupdate: "false"
  labels:
    kubernetes.io/bootstrapping: rbac-defaults
  name: system:public-info-viewer
rules:
- nonResourceURLs:
  - /healthz
  - /livez
  - /readyz
  verbs:
  - get
EOF

Fix for KHV005, KHV050, Access to pod's secrets

...

$ kubectl replace -f - <<EOF
apiVersion: v1
kind: ServiceAccount
metadata:
  name: default
  namespace: default
automountServiceAccountToken: false
EOF

The above fixes are implemented in the Ansible playbook deploy/playbook/init_cluster.yml and configuration file deploy/playbook/k8s/fix.yml

Fix for CAP_NET_RAW Enabled:

Create a PodSecurityPolicy with requiredDropCapabilities: NET_RAW. The policy is shown below. The complete fix is implemented in the Ansible playbook deploy/playbook/init_cluster.yml and configuration files deploy/playbook/k8s/default-psp.yml and deploy/playbook/k8s/system-psp.yml, plus enabling PodSecurityPolicy checking in deploy/playbook/k8s/config.yml.

...

apiVersion: policy/v1beta1
kind: PodSecurityPolicy
metadata:
  name: psp-baseline
spec:
  privileged: true
  allowPrivilegeEscalation: true
  allowedCapabilities:
  - IPC_LOCK
  - NET_ADMIN
  requiredDropCapabilities:
  - NET_RAW
  hostIPC: true
  hostNetwork: true
  hostPID: true
  hostPorts:
  - max: 65535
    min: 0
  readOnlyRootFilesystem: false
  fsGroup:
    rule: 'RunAsAny'
  runAsUser:
    rule: 'RunAsAny'
  seLinux:
    rule: 'RunAsAny'
  supplementalGroups:
    rule: 'RunAsAny'
  volumes:
  - '*'

Results after fixes are shown below:

Image Removed

...

Kube-Hunter

Nexus URL: https://nexus.akraino.org/content/sites/logs/fujitsu/job/sdt/r7/sdt-bluval/1/

There are no reported vulnerabilities. Note, this release includes fixes for vulnerabilities found in release 6. See the release 6 test document for details on those vulnerabilities and the fixes.

Image Added

Note that the results still show one test failure. The "Inside-a-Pod Scanning" test case reports failure, apparently because the log ends with "Kube Hunter couldn't find any clusters" instead of "No vulnerabilities were found." This also occurred during release 6 testing. Because vulnerabilities were detected and reported earlier in release 6 by this test case, and those vulnerabilities are no longer reported, we believe this is a false negative, and may be caused by this issue:   https://github.com/aquasecurity/kube-hunter/issues/358

...

*Vuls is counted as one test case.

...