...

(optional) Update Jenkins to the latest version available (2.238 at time of writing):

sed -i "s/2.192/\"2.238241\"/" vars.yaml

Install Jenkins using Ansible playbook:

...

cp ~/.ssh/id_rsa /var/lib/jenkins/jenkins-rsa
chown jenkins:jenkins /var/lib/jenkins/jenkins-rsa
chmod 600 /var/lib/jenkins/.netrc

This keypair is also going to be used for accessing the Akraino and ONAP repositories as non-privileged accounts (for basic operations such as cloning a repository).
As such, the public key ~/.ssh/id_rsa.pub) just generated should be uploaded to the Akraino and ONAP accounts to be used.

...

echo "machine nexus.akraino.org login USERNAME password PASSWORD" | tee /var/lib/jenkins/.netrc
chown jenkins:jenkins /var/lib/jenkins/.netrc

Also needed to upload Bluval logs is the lftools python3 package, install it:

...

cd ~
git clone --recursive "https://gerrit.akraino.org/r/ci-management"

The following is temporary until patch https://gerrit.akraino.org/r/c/validation/+/3370 gets merged by the validation team and a fix is provided to allow Docker host network:

sed -i 's/ssh:\/\/akraino-jobbuilder@gerrit.akraino.org:29418/https:\/\/github.com\/igordcard/' ci-management/jjb/defaults.yaml

Let's finally get Jenkins to recognize the Bluval job (install JJB):

...

At the master node (where Jenkins is already installed at this point), download KUD source code with Kubernetes 1.16 patch (this guide should be update once this patch is merged):

cd ~
apt-get install -y git-review
git clone "https://gerrit.onap.org/r/multicloud/k8s"
cd k8sgit remote add gerrit https://GERRIT_USERNAME@gerrit.onap.org/r/a/multicloud/k8s
git review -s
git review -d 106869

Replace all localhost references with $HOSTNAME in KUD's aio.sh:

...

# install_addons
# if ${KUD_PLUGIN_ENABLED:-false}; then
#   install_plugin
# fi

Add ansible_user=root at the end of each host line in aio.sh [all], it should look like this (this is required when jenkins attempts to install KUD):

[all]
$HOSTNAME ansible_ssh_host=${OVN_CENTRAL_IP_ADDRESS} ansible_ssh_port=22 ansible_user=root
WORKER_NODE_HOSTNAME ansible_ssh_host=WORKER_NODE_IPADDR ansible_ssh_port=22 ansible_user=root

(optional) Finally install Kubernetes with KUD (ansible will automatically install it in the worker node too):

kud/hosting_providers/baremetal/aio.sh

The above step is optional because the ICN Jenkins Bluval job is now capable of installing and uninstalling KUD automatically. This is done before and after running the Bluval suite, respectively. However, what's mandatory is copying both aio.sh and installer.sh files above into /var/lib/jenkins:

cp kud/hosting_providers/baremetal/aio.sh /var/lib/jenkins/
cp kud/hosting_providers/vagrant/installer.sh /var/lib/jenkins/
chown jenkins:jenkins /var/lib/jenkins/aio.sh
chown jenkins:jenkins /var/lib/jenkins/installer.sh

Also necessary, for the time being, is copying the /var/lib/jenkins/jenkins-rsa private key into jenkins's own .ssh:

cd /var/lib/jenkins/.ssh
rm id_rsa*
cp ../jenkins-rsa id_rsa
chown jenkins:jenkins id_rsa

Remove libvirt and the virtual bridges it creates (this will be fixed in the future), as they create a security vulnerability in os/lynis:

apt-get purge -y $(apt-cache depends libvirt-bin qemu-kvm| awk '{ print $2 }' | tr '\n' ' ')
apt-get autoremove --purge -y
ip link delete dev virbr0
ip link delete dev virbr0-nic

(optional - Jenkins will take care of this too) A few fixes have to be applied to Kubernetes to address kube-hunter security vulnerabilities. Execute the following commands:

kubectl replace -f - << EOF
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: pod-reader
  annotations:
    rbac.authorization.kubernetes.io/autoupdate: "false"
  labels:
    kubernetes.io/bootstrapping: rbac-defaults
  name: system:public-info-viewer
rules:
- nonResourceURLs:
  - /livez
  - /readyz
  - /healthz
  verbs:
  - get
EOF

kubectl replace -f - << EOF
apiVersion: v1
kind: ServiceAccount
metadata:
  name: default
automountServiceAccountToken: false
EOF

At this point, everything is ready. Jump over to http://localhost:8080, log-in using admin/admin credentials and create a new build for icn-bluval-daily-master.

For the build, here are the recommended parameters to set according to the deployment herein outlined as well as to conform to upstream Bluval logging requirements:

DEPLOY_SCENARIO: master
CLUSTER_MASTER_IP: localhost
CLUSTER_SSH_USER: root
CLUSTER_SSH_PASSWORD: <empty>
CLUSTER_SSH_KEY: /var/lib/jenkins/jenkins-rsa
BLUEPRINT: icn
LAYER: <empty>
VERSION: master
OPTIONAL: nofalse
PULL: yestrue
LAB_SILO: intel
SETTINGS_FILE: <empty>

And pull the trigger.

Total time to run should be anywhere from 2 1.5 to 3 hours on an average server-grade dual-node with good Internet connection (~90% of the time will be spent running the k8s layer conformance testing [sonobuoy]).

The easiest way to check what logs have been uploaded to the Nexus is by opening loading the following URL:
https://logs.akraino.org/intel/bluval_results/icn/master/

Troubleshooting

Task download mitogen release failed

TASK [download mitogen release] ************************************************
task path: /opt/kubespray-2.12.6/mitogen.yaml:17
Thursday 29 October 2020 18:51:30 +0000 (0:00:00.385) 0:00:00.491 ****** 
An exception occurred during task execution. To see the full traceback, use -vvv. The error was: TypeError: load_file_common_arguments() got an unexpected keyword argument 'path'
fatal: [localhost]: FAILED! => {"changed": false, "module_stderr": "Traceback (most recent call last):\n File \"<stdin>\", line 113, in <module>\n File \"<stdin>\", line 105, in _ansiballz_main\n File \"<stdin>\", line 48, in invoke_module\n File \"/tmp/ansible_get_url_payload_40xmhT/__main__.py\", line 650, in <module>\n File \"/tmp/ansible_get_url_payload_40xmhT/__main__.py\", line 633, in main\nTypeError: load_file_common_arguments() got an unexpected keyword argument 'path'\n", "module_stdout": "", "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error", "rc": 1}
to retry, use: --limit @/opt/kubespray-2.12.6/mitogen.retry

This error occurs in the get_url module of ansible.  Purging the system of ansible resolved it.  Note that simply uninstalling ansible is insufficient, ansible-base must be uninstalled also.

pip uninstall ansible-base
pip uninstall --yes ansible