Weekly on Wednesdays at 8:00 AM PST / 11:00 AM EST.

Akraino Edge Stack 1 is inviting you to a scheduled Zoom meeting.

Topic: Blueprint Validation
Time: Sep 23, 2020 03:00 PM Universal Time UTC
Every week on Wed, until Sep 1, 2021, 50 occurrence(s)

Please download and import the following iCalendar (.ics) files to your calendar system.
Weekly: https://zoom.us/meeting/uZEkde6qrjwv3tbXTirGL_mMDLePAUznKw/ics?icsToken=98tyKu2tpzktGNSStVztd60tE9r8bPH2lCJaqJtplQ3CLx9eTyfaM9JjB6hxO8-B

Join Zoom Meeting
https://zoom.us/j/459272075?pwd=M3gwQ3JUaTYvZHhrVUpqL0ZzOTBudz09

Meeting ID: 459 272 075
Passcode: 311342
One tap mobile
+13462487799,,459272075# US (Houston)
+16699006833,,459272075# US (San Jose)

Dial by your location
+1 346 248 7799 US (Houston)
+1 669 900 6833 US (San Jose)
+1 253 215 8782 US (Tacoma)
+1 312 626 6799 US (Chicago)
+1 646 558 8656 US (New York)
+1 301 715 8592 US (Germantown)
877 369 0926 US Toll-free
855 880 1246 US Toll-free
+1 438 809 7799 Canada
+1 587 328 1099 Canada
+1 647 374 4685 Canada
+1 647 558 0588 Canada
+1 778 907 2071 Canada
+1 204 272 7920 Canada
855 703 8985 Canada Toll-free
Meeting ID: 459 272 075
Find your local number: https://zoom.us/u/alZMq7OBg

Introductory webcast recording

Notes

July 7, 2021

MOM:

Discussed for EALT-EDGE Blueprint, ELIOT IOTGateway and ELIOT uCPE Blueprint sonobuoy conformance test / results
Discussion → Sonobuoy will execute properly with K8s minimum 3 Node cluster (1 master + 2 worker nodes)
Discussion → Sonobuoy support for 1.17.2 → Need to be confirmed by validation team

June 16, 2021

Agenda:
- Making progress for BluVal to support K8s version v1.18, the patch set is under test
- Need to identify which R5 blueprints support K8s versions v1.19, v1.20
- Further discussed possibility for using multiple BluVal releases to support K8s v1.18, v1.19, v1.20
- Discussed testing compatibility of container run times such as Docker, Containerd, LXC, CRI-O
- Discussed supporting popular combinations first, so majority of BPs can be supported
- Need support for Debian Linux, in R4 we have 3 BPs which support Debian
- Discussed high level approach for integrating Anuket RC2 tests into BluVal using Robot layer
- Tina recommended we should provide an update to TSC meeting on June 17th
Participants: Tina Tsou (Deactivated) Deepak Kataria thorking Sirisha Gopigiri

June 9, 2021

Agenda:
- Discussed uplift of BluVal to support K8s version v1.18, Sirisha has submitted patch set for review
- Discussed possibility to support multiple BluVal versions to support v1.18, v1.19, v1.20
- Discussed support for Debian Linux
- Discussed Integrating Anuket RC2 tests into BluVal
Participants: Tina Tsou (Deactivated) Deepak Kataria thorking Sirisha Gopigiri, Jason Wen
Deepak Kataria presented BluVal at LFN Developer & Testing Forum on June 8th
- Presentation Slides, Recording and Meeting Notes are available at the link below
- https://wiki.lfnetworking.org/display/LN/2021-06-08+-+Anuket%3A+Cloud-Native+Full+Stack+Conformance+Validation+Framework

June 2, 2021

Agenda:
- Discuss uplift of BluVal to support K8s versions higher than v1.16
- Integrate Anuket RC2 tests into BluVal
- Integrate Chaos Tests in BluVal
- Leverage BluVal UI for improved user experience
- Integrate enhancements from security sub-committee
Participants: Tina Tsou (Deactivated) Deepak Kataria Arif khan Jason
Deepak Kataria presenting BluVal at LFN Developer & Testing Forum, taking place June 7-10, 2021. Presentation scheduled on June 8th
- What: 2021-06-08 - Anuket: Cloud-Native Full Stack Conformance Validation Framework
  When: Tue Jun 8 2021 from 8:00 to 9:00 (Time zone: Eastern Time US & Ca)
  Calendar: LFN Events > Virtual Events > Anuket
  Who: Deepak Kataria
  Meeting Link: https://zoom.us/j/94043909680
  Link: https://teamup.com/event/show/id/1TDWxFAH2Xrhr6B7seUCCEW6i9Q5uY

November 11, 2020

Agenda:
- Discuss how BluVal can help O-RAN and Akraino Alliance Paul Carver thorking Tina Tsou (Deactivated)
- Akraino BluVal Exception Request Srinivasan Selvam
Participants: Paul Carver thorking Tina Tsou (Deactivated)Srinivasan Selvam
ELIOT IotGateway Blueprint, ELIOT uCPE Blueprint, EALT-EDGE Blueprint due to the K8s conformance testing cannot support v1.17, these exceptions are accepted.
ORAN cooperation about BluVal will start from REC BP first and if we have any other BP want to integrate with ORAN we can also support them even if ORAN want to use this.(Because BluVal is also an open source for use )

October 28, 2020

Agenda:
- Discuss how BluVal can help O-RAN and Akraino Alliance Paul Carver thorking Suzy Gu jim.xu@zenlayer.com Hao Xu
- Akraino BluVal Exception Request Hao Xu Yin Ding
Participants:
Akraino BluVal Exception Request is added.

October 21, 2020

Participants: thorking Tina Tsou (Deactivated) Hao Xu
Akraino BluVal Exception Request is added.

July 29, 2020

Reviews, Jiras etc. - same as on July 15th;
Debugging a bluval issue with the IEC Type 4 Team - turns out there was some YAML formatting issue;
Proposal: Add a yamllint sanity check for `blueval-<blueprint>` files before actually running bluval, which should help new users figure out if they're running into YAML formatting issues or whether it's an actual bluval traceback (right now bluval throws a pretty cryptic error message with a yaml.safe_load traceback when bluval configuration files have formatting issues, e.g. wrong identation);

July 22, 2020

Reviews, Jiras etc. - same as on July 15th;
No participants, the usual participants being all on vacation currently;

July 15, 2020

Reviews:
- 3601 pending peer review
- 3604 pending peer review
Jiras:
- VAL-119 - Getting issue details... STATUS is ongoing
Tina Tsou (Deactivated) mentioned that the Bluval User Guide is not very friendly towards new users who haven't previously worked with LF infrastructure - particularly the LF specific terms could use a little more background (e.g. jumpserver, SUT etc.);
we agreed to open a new JIRA ticket for improving the Bluval User Guide with the used terminology / abbreviations;

July 8, 2020

Reviews:
- 3601 pending peer review
- 3604 pending peer review
Jiras:
- VAL-119 - Getting issue details... STATUS is ongoing
Cynthia Billovits is working with upstream developers - they said they fixed the issue we reported, but it turns out it still needs more work - we're waiting on them to get back to us hopefully later today.

July 1, 2020

Tapio Tallgren will be on holiday for next four weeks; Alex will host the meeting for the first two weeks
- Action: Get Zoom rights to Alexandru Avadanii
Jiras:
- VAL-117 - Getting issue details... STATUS is merged
- VAL-119 - Getting issue details... STATUS is ongoing

June 24, 2020

CentOS 8 is not supported in Bluval. We had a discussion about this topic last time and agreed that this should be documented somewhere
- VAL-117 - Getting issue details... STATUS is for CentOS 8 support
- VAL-119 - Getting issue details... STATUS is for the documentation issue
Bluval debugging capabilities can still be improved. Ideas from last time:
- Enable debugging with a single flag in variables.yaml VAL-120 - Getting issue details... STATUS
- Add "test" test cases which will validate that e.g the ssh connection works. This is often the first problem users have VAL-121 - Getting issue details... STATUS

June 10, 2020

Reviews:
- 3556: Enable mandatory tests for REC
- 3550: Use docker host network for containers
- 3558: Support password ssh auth for all robot tests
- 3561: Add a new variable Robot loglevel
- 3562: Remove inactive committers
Lynis/Vuls automation
Debugging support: new very simple test cases like
- Ping to host
- Ssh access to SUT
- Volume mount

June 3, 2020

Review:
- https://gerrit.akraino.org/r/c/validation/+/3543/1/tests/os/vuls/vuls.robot#87 - could add a global "debug" flag, that would be easier to use
New wiki page for debugging
Add comment about bluval.py vs. bluval.sh
Tickets:
- Followup on Vuls and Lynis automation: ongoing
- Followup on Redfish testing: password issue is now fixed, waiting for release. Power cycle issue is being worked on

May 13, 2020

Reviews:
- https://gerrit.akraino.org/r/c/validation/+/3398 is now merged
- Daniel is working on https://gerrit.akraino.org/r/c/validation/+/3421 Fix ssh cleanup issue for vuls test and https://gerrit.akraino.org/r/c/validation/+/3432 Fix blucon.py argument issue
- https://gerrit.akraino.org/r/c/validation/+/3370 Add ICN Bluval support needs to be reviewed and merged
Jiras:
- VAL-113 - Getting issue details... STATUS has now an upstream ticket: https://github.com/DMTF/Redfish-Usecase-Checkers/issues/39
- VAL-115 - Getting issue details... STATUS is a new one

May 06, 2020

Reviews:
- https://gerrit.akraino.org/r/c/validation/+/3398 bluval-eliot.yaml added for ELIOT BP
- https://gerrit.akraino.org/r/c/validation/+/3395 Support password-based ssh auth
Jira tickets:
- Kuralamudhan Ramakrishnan (Deactivated) / Igor D.C. Issue related conformance testing via a Jenkins job
  - There could be an old Docker image
  - Need to make sure that there is bluval-icn.yaml
- VAL-114 Sonobuyo fails in AWS with Network Cloud BP
- VAL-113 Redfish-Usecase-Checkers uses a weak password
Update from security committee:
- Lynis and kube-hunter draft mandatory test lists are ready, will be uploaded to wiki

April 29, 2020

New code
- https://gerrit.akraino.org/r/c/validation/+/3384 hardware: bios_version rework without dmidecode
- https://gerrit.akraino.org/r/c/validation/+/3395 Support password-based ssh auth
Open issues
- Redfish - ongoing
- Network cloud
Update from security group
- Lynis will have a list of mandatory tests, and this will be the pass/fail criteria
Remind PTLs to register the labs

April 22, 2020

Update from security group
- Lynis: Some of the tests have points attaches to them and some do not
- The points can be used to create some kind of metric
- Will try to create a list of acceptable and non-acceptable failures
- Kube-hunter is being studied
Two Jira tickets to be created:
- k8s conformance is mysteriously failing (from Srinivasan Selvam)
- KNI validation jobs are not running

April 15, 2020 (recording)

Security group update
- For Vuls and other tests that use CVE tagging:
  - High and medium vulnerabilities are mandatory, so the test fails if any of those fails
  - If the vulnerabilities cannot be fixed, they must be handled as exceptions and documented
  - The exceptions could have time limits
- For Lynis: gives a hardening index but it is difficult to use that
  - There could be low bar and any test score below it should just fail
  - Any test score above a high bar should pass
  - Test scores between the low and high would need to be analyzed
  - The high score could be the same as passing all high and medium priority tests
- Kube-hunter: work in progress
https://gerrit.akraino.org/r/c/ci-management/+/3356
https://gerrit.akraino.org/r/c/ci-management/+/3358
Tagging 3.0

April 8, 2020

Security group update: they now want examples of "Lynis, Vuls, or Sonar Cloud"
- Daniel will provide
https://gerrit.akraino.org/r/c/validation/+/3343
https://gerrit.akraino.org/r/c/validation/+/3340
Tagging 3.0 awaits this patch
Documentation review
- Test plan

April 1, 2020

Presentation now done, another coming
- Clarification: Redfish only has Use Case testing (which is what is packaged in Bluval)
What should kubehunter return?
- Robot has some third alternative between "pass" and "fail", Juha will investigate
- If a security tool finds vulnerabilities, it could be pass since the requirement is to run the tests but this could imply that there is nothing to investigate
- The vulnerabilities could be harmless in the end, so fail would also be misleading
- Later on, there can be a whitelist of "harmless" warnings
Tagging Release 3.0
- Can be done after the changes to security tests
- Cristina will do the tagging
Status
- Patches for CI integration have been merged
- Vuls fails when run after other tests; Daniel is investigating
- Juha will send the kubehunter sample report to security@lists.akraino.org

March 25, 2020

Presentation to TSC
Vuls and lynis on CI: https://gerrit.akraino.org/r/c/validation/+/3306
Discussion with Security Committee:
- Interpreting the results from the vulnerability tests will require understanding how the project is used
- Thus, the evaluation must be done together with the project PTL
- The Security Committee requested a sample document from the tests
Presentation to Akraino TSC+PTLs next week Tuesday:
- Start with the list of mandatory tests (Tapio)
- Show hands-on how to run the tests (Juha)
- Show how to run the tests in CI and copy the results (Cristina?)
- Show the results in UI (Ioakeim?)

March 18, 2020

Updates from Tapio:
- The mandatory tests presentation was approved by TSC
- I asked the Security team for input but got nothing this far
- I have not received any comments from Blueprints
- There is one Blueprint (Network Cloud) that is using OpenStack so we will keep Tempest in

There can be some tests coming from the O-RAN community
No progress on UI full control loop
KPI project might bring in some performance testing
Juha will update the tempest tests
Daniel is working on integrating the security tests in CI

March 11, 2020

We discussed the proposal for mandatory tests for Release 3:

March 4, 2020

Vuls improvements
- Ubuntu 18 is now running on ARM
- CentOS also seems to work
- The size is now 1.5 Gb, mainly due to (a) compressing the database and (b) doing a multi-stage build
Jenkins documentation: Running bluval in CI
Updates from TSC meeting:
- Bluval will be mandatory for all projects
- New project proposal: Kontour (KPI)
- CNTT/CNF-conformance?

February 26, 2020

Vuls is integrated for Ubuntu 16
- CentOS and Ubuntu 19 are WIP
- The images are getting big, ~7 Gb
Presentation for TSC planning meeting
- Need a definition of "mandatory tests" for Release 3
- Need to tag release 3.0 by end of April?
kubehunter is now integrated
Cristina is working on documenting how Bluval can integrated into a Jenkins
- Ioamkeim will test the instructions after they are done

February 19, 2020

The issue with the verify jobs caused by tox dependencies has been fixed upstream
The k8s layer container is failing to build, Juha is looking into it
The robot test for kubehunter is ready but it's blocked by the k8s container issue
A demo was presented to the TSC about the validation work, there were questions on how to integrate it in CI (Cristina needs to document this)
The vuls patch has been updated, waiting for review
A patch to reorganize the jobs and improve the relationship between them is in review https://gerrit.akraino.org/r/c/ci-management/+/2242

February 12, 2020

Demo to TSC+PTLs: let's try tomorrow in the TSC meeting
Juha K. is looking in kubehunter and kubestorage
Juha K. is evaluating the Vuls patch
CI jobs are failing with tox
- The workaround is manually marking patches as verified
- See https://github.com/pypa/virtualenv/issues/1551
Discussion about closed loop automation
- Proposal next week
CHOMP project to be followed up

January 29, 2020

Presentation to TSC
- Still some bugs remain
- Goal is now February 4th TSC+PTL meeting
Vuls currently only works on Ubuntu, cristinapauna and Daniel Stoica are working on supporting CentOS
The CHOMP project is interested in using Bluval
- Deepak Kataria will invite someone from the project to next meeting for a discussion
The right way to edit the committer list is to change it after half of committers have approved the change. Let's follow that rule unless the Technical Governance document says something else
We will make some proposals to ONES and then figure out later who will present them. At least cristinapauna will attend the event

January 22, 2020

Missing from Release 3:
- Redfish tests (fixes exist but not part of a release)
- Full control loop testing (UI can trigger tests in a lab, jjb in ci-integration is missing)
TSC presentation is planned for next week

January 15, 2020

The "push logs to Nexus from Jenkins" patch is still under work
The new bluval-fe repo is now ready

January 8, 2020

Presentation and demo to TSC
- Want to have the "push logs to Nexus from Jenkins" patch working before presentation
- Check again next week
The new repo was approved on December 19th by the TSC
- Need a record of the TSC meeting where this was approved
- Easiest is to have the topic on the agenda again tomorrow
Naga is leaving the project by the end of this week
- Will remain as a contributor with a new email address (INFO.yaml needs to be updated)
Security tests (Lynis and Vuls) may not work on ARM, will check this next
Discussion about installation logs and UI
- The Regional Controller is the only "official" (at least for TA) installer, so it only can tell if installation worked
- Could also enhance deployment logs so that they inform some UI when an installation has worked
Community lab is free to use for all!

December 18, 2019

New repo request went to wrong place, have tried again
Security tests:
- Lynis - merged
- Vuls - aim to have it for review this week. It is not support
The Jenkins job in CI-Integration is now pushing results from Enea lab to Nexus daily from the LF Jenkins master
Release 2.0.1 has been tagged
- Plan is to automate this in the future so after new tags, the Docker images are created automatically

December 4, 2019

Bluval containerization https://gerrit.akraino.org/r/c/validation/+/2089
- Goal is to make running the tests easier by running Bluval inside a container
- Cristina will test the patch and make comments
- The Jenkins job needs to be updated
New repo for UI (but only that, needs a new name)
Presentation to TSC
- Currently, pushing the results to LF Nexus does not work with the Jenkins job
- Should wait until everything is up and running
- Will check next week whether the presentation can be done on Dec 17 or Dec 19
UI launch
- Lab owners and TSC members can request access to the UI from Ioakim
LTP patch https://gerrit.akraino.org/r/c/validation/+/2075
- Some LTP tests stall which makes the test run very slowly
- This patch removes some of those tests
- Can be merged
- If someone tells that these tests should work, Juha will take another look at the removed tests

November 27, 2019

Release 2 update:
- Images are now in Docker hub
There were some changes to LTP, and we discussed whether to add a new tag. The conclusion was to create release 2.0.1, so blueprint projects can run the latest tagged version which has LTP working
We discussed the release 3 priorities. The current draft is here.
Action items:
- Cristina to tag release 2.0.1
- Deepak will think about more tests
- Ioakim will write a request for a new repo
- Naga will work on a proposal to refactor the tests

November 13, 2019

Topics:

Sonobuyo k8s version and Juha's patch https://gerrit.akraino.org/r/c/validation/+/1933
- Ok to go forward with this
Release tagging?
- Ok after merging a couple of patches
- Aiming for Friday
- "2.0"
LTP
- The test works now but it has not been merged or run in CI
- Will be included in "2.0" release but did not make it in time for mandatory
User Interface
- AWS instance

November 6, 2019

We had a short meeting today, due to small number of participants. News:

LTP patch is now done
Discussion about https://gerrit.akraino.org/r/c/validation/+/1919
Status from Cristina:

- We fixed VAL-69, VAL-72, VAL-73, VAL-86

- The CI jobs that build our containers were failing on arm due to the build server and that issue has been solved. There was another failure in the latest build and I am currently investigating it

- The CI jobs that run the validation k8s conformance test have not been able to run yet, we’re investigating the issue (VAL-81)

- A documentation draft for running validation in CI has been put on wiki, but I need to update it so that it’s more generic

October 30, 2019

Ken Yi wants to discuss the mandatory security testing requirements
Jenkins update https://gerrit.akraino.org/r/c/ci-management/+/1657
- Initial patch made but had some issues
- Needs some more testing and documentation
We need to check where we are rst. Release 2. Seems like Sonobuyo and Tempest are working fine, LTP and Redfish have issues
- Redfish tests have upstream bugs → propose to drop
- LTP testing requires sudo - is this acceptable?
- LTP is also somewhat broken at the moment
- Sonobuyo is failing to run on some blueprints
- WIll prioritize the Rel.2 bugs
UI update
Bug reports

October 23, 2019

Release 2: Redfish update https://gerrit.akraino.org/r/c/validation/+/1781
- Testing with OpenEdge is ongoing
- OpenStack integration: copy bluval-rec.yaml to bluval-unicycle.yaml and leave OpenStack tests out from bluval-rec.yaml
Add remote session to etcd HA testcases https://gerrit.akraino.org/r/c/validation/+/1793
- Will be modified to follow the The Principle: Whenever possible, use testing tools on the container to connect to the SUT. When this is not possible, ssh connection to the SUT can be used
- Indu will fix this so that the kubectl in the container will connect to the SUT using a modified kubeconfig file (replace localhost with real IP address)
Fix for LTP: https://gerrit.akraino.org/r/c/validation/+/1752 VAL-66 - Getting issue details... STATUS
- Will build binaries out of the source files and using those for testing
- The binaries will be copied to the SUT, run, and deleted afterwards
- The full LTP requires sudo rights which is a security risk. Need to investigate if there is a subset of LTP that can work without special rights
Versioning of Bluval code and Docker hub
- Use tags to mark releases
- Need separate Jenkins jobs for released/stable version and master
- The UI project needs separate versioning
- Could request a new repo for UI?
CI jobs: https://gerrit.akraino.org/r/c/ci-management/+/1657
- Still has a "-1" but getting there
- Need to create instructions once the patch is merged (AP: Vali)
- Does not cover uploading the results
Logging container: https://gerrit.akraino.org/r/c/validation/+/1767
- Will be part of service layer
NTP and DNS tests
UI status update
- Almost there, but some problems remain

October 16, 2019

LTP - no updates, will try install the tests through binaries built at the time when we build the container
- We need to keep in mind that there might be different OS installed on the cluster
CI jobs: https://gerrit.akraino.org/r/c/ci-management/+/1657
- We had a couple of sync sessions; work is still in progress
OpenStack Bluval integration https://gerrit.akraino.org/r/c/validation/+/1602 - no updates, patch needs to be tested on REC
Sonobuyo ARM64 patch has been merged: https://github.com/heptio/sonobuoy-plugin-systemd-logs/pull/5 - no updates, low priority
Logging container: patch was merged, test was moved to the validation project
- Discussion if this should be a separate layer, or include them in the existing layers (k8s, docker)
- The logging is an integrated system so it seems hard to separate it (elastic search logging)
NTP and DNS tests were developed for TA, but they seem generic tests. We decided to create a services layer to include those and we will move the logging inside this layer instead of having its own layer
- https://gerrit.akraino.org/r/c/ta/cloudtaf/+/1489
- https://gerrit.akraino.org/r/c/ta/cloudtaf/+/1728
UI status update: LF team informed us that the UI will be deployed in AWS by the end of this week

October 9, 2019

Release 2 table updates
LTP
- The test does not seem to work without having ltp installed on the target. Also needs to be updated to use ssh
- In order to do the installation, the test must be run as a user who can install the software
- Running as a non-root user: seems to work with sudo
- NB: CoreOS will not be able to run ltp. They need an exception
CI jobs: https://gerrit.akraino.org/r/c/ci-management/+/1657 and https://gerrit.akraino.org/r/c/validation/+/1647
- Separate discussion about how this will be finalized
OpenStack Bluval integration https://gerrit.akraino.org/r/c/validation/+/1602
Sonobuyo ARM64 patch has been merged: https://github.com/heptio/sonobuoy-plugin-systemd-logs/pull/5
- This means that one of the ARM specific patches can be removed
Logging container: https://gerrit.akraino.org/r/c/validation/+/1725
- This is currently used by a test in Telco Appliance blueprint family
- This will be a layer in Bluval and moved to validation

October 2, 2019

Updated "Release 2" table
The documentation on how to use Bluval is incomplete but it is now no longer "Work in Progress"
Cristina has written a blog post about how to do Kubernetes testing with Bluval; this should be linked to from somewhere (link is
https://www.lfedge.org/2019/09/26/running-k8s-conformance-tests-in-akraino-edge-stack/)
Jenkins job for running Bluval is being reviewed in the ci-management project (see https://gerrit.akraino.org/r/c/ci-management/+/1657)
The OpenStack + Bluval integration patch still needs end-to-end testing (https://gerrit.akraino.org/r/c/validation/+/1602)
VAl-53
- This will be a separate layer instead of part of Kubernetes
- Deepak will update the ticket

September 25, 2019

Plan is to use MySQL and AWS ECS for the Validation UI, Ioakeim will update the LF ticket IT-16700
Blueprint Validation pre-requisites were presented in the Akraino Technical Community Call held on 9/19; meeting was recorded
Naga is validating ETCD testing which was completed by Indu and will integrate with BluVal
Tempest robot test verified on Airship by Indu, Naga is testing integration with BluVal
Discussion on OpenStack layer: Tempest and Refstack are complementary. If projects don't support certain features the tests should detect the missing components and skip those tests (we need to test this functionality). As a backup we can also blacklist tests. Indu is working on VAL-51 which was created to test this functionality.
LTP tests need root access to run; options are either sudo or create a group with privileges as opposed to using root password. Question if they can be ran without root access; Ioakeim has been assigned VAL-50, and is investigating. One consideration is to create a special group with the right capabilities. This group must be created on every lab.
Discussion on security: we need a development guide documentation
Discussion on how to handle the situation where the nodes don't have access to internet and we need the test images to be in a local repository inside the cluster
A ticket was created for Cristina VAL-53, to create a Container for the Logging and Monitoring Layer

September 18, 2019

The UI VM discussion (MariaDB vs MySQL DB): waiting for LF to provide the quotation for MariaDB
Discussion on blueprint validation prerequisites (what we request from users with regards to infra and logs)
Update from Indu: completed the etcd testing, validated in Airship and REC (ready to merge). Tempest robot test verified on Airship and integrated with bluval
Discussion on OpenStack layer: Tempest and Refstack are complementary. If projects don't support certain features the tests should detect the missing components and skip those tests (we need to test this functionality). As a backup we can also blacklist tests.
LTP tests need root access to run; options are either sudo or create a group with privileges as opposed to using root password. Question if they can be ran without root access.
Discussion on security: we need a development guide documentation
Discussion on how to handle the situation where the nodes don't have access to internet and we need the test images to be in a local repository inside the cluster

September 11, 2019

"Release readiness" updated
Discussion about Blueprint family names and Blueprint names - it is best to use the same names that are used in repos
The UI VM discussion (MariaDB vs MySQL DB) will continue in https://jira.linuxfoundation.org/servicedesk/customer/portal/2/IT-16700
All Blueprints that want to have Maturity review before Akraino Release 2 must complete mandatory tests before Oct 31st

September 4, 2019

Need to have a discussion with LF about AWS database solution (ONAP SDK uses MariaDB, AWS provides an MySQL) AP: Tapio
Blueprint families: discussion about whether the UI and Nexus URL should use blueprint family names. Decision was Yes
Showing logs: there was a request to show the CI/CD logs with the UI. This is difficult to do and hence low priority
Plan is to show a demo to TSC next Tuesday September 10. Tapio will request a slot

August 28, 2019

Security tools discussion

August 21, 2019

Agenda:

Testing with µMEC
Mandatory tests:
- LTP
- Redfish
- Security
UI - issues with Nexus, getting fixes to ONAP SDK
https://gerrit.akraino.org/r/c/validation/+/1341

August 14, 2019

Update about the UI web page: biggest issue at the moment is that the SDK used has some bugs. These are being fixed upstream
ELIOT/Kumar has reported success in running Blueprint validation
Indu is planning to test REC with Bluval
The Security Sub-Committee/Ken Yi needs to be invited to meetings

July 31, 2019

News:

Mandatory tests and UI have been approved

Other items (in random order)

Security subcommittee is interested in using the Validation framework for two of their mandatory test sets. To be followed up
Next week Tapio will be on holiday, will ask Deepak to start the meeting
Naga has committed the patch to separate Docker and Robot parts (see https://gerrit.akraino.org/r/c/validation/+/1132). He will update the user documentation on wiki to match the latest

Fixing ARM64 support for Sonobuoy required moving to Kubernetes 1.15. The problem is that one test case (Aggregation) that comes with Kubernetes 1.15 works only with the Kubernetes 1.15 version, while almost all blueprints use an earlier Kubernetes version. So the plan is to blacklist that testcase in Blueval so that the same tests can be run everywhere
Indu is working to push etcd HA tests to Gerrit

July 24, 2019

News

Mandatory tests were discussed in PTL/TSC meeting on July 23 (but not approved for lack of quorum). Will be on the agenda again on July 25
Sonobuoy works also on ARM64 except for one test case
- Requires some patches to run
- Target is to check by end of week if that issue can be fixed

Public UI VM

LF has sent out a quote for two xlarge VMs on AWS, load balancer and 10G extra storage space
The extra storage will be used for database for UI
The two VMs are running the UI code
It is possible to scale down the proposal but let us go with this

Validation maintenance

We had a discussion about what code should be included in this projects repo
There is already blueprint related code: bluval-<blueprint name>.yaml
But we agreed that we do not need to allow blueprint specific code to the repo, at least for the time being

Demo next Tuesday

Will show UI in action with REC lab

Open gerrits

[UI] Support UI partial control - this is work in progress
Add resources for building cord-tester image - has a "-1"
sonobuoy testing – waiting for DANM4.0
Juha K will be back from holiday on August 5

Old APs - not discussed

Using k8s on our deployments
Using versioning for our releases
Moving the UI to a new git repo

July 17, 2019

Update about Sonobuoy testing on ARM64: with the latest Sonobuoy version, almost all test cases (except for one) seem to pass
Bluval UI:
- First item was about the Bluval UI implementation hosted at LF. LF has already a MySQL instance running in AWS as an elastic DB, and if it can be used, then the UI will only need to have the web server and the code to update the data in the database. This means that the UI can be stateless and all state will be in the DB. To be followed up
- Second issue was about how to check whether bluval has ran all the mandatory tests. The test log will have results of all tests that it has run, but the UI cannot know what tests it was supposed to run (since it is possible to disable tests). The tentative solution is to create a file from bluval.yaml that has all tests that were supposed to be run, and store it in the logs. Bluval.py needs to do that
We discussed the Redfish testing. Redfish testing is currently mainly checking that the implementations confirm to the schemas that have been defined. The DMTF has also defined a few interoperability profiles, such as for OCP. It would be useful to define inside Akraino a profile that the different hardwares could be tested against, to make sure that all implementations have common functionality implemented in the same way
Recording is here. Unfortunately, it only covers a part of the Redfish discussion

July 10, 2019

AP updates:
- AP: Kubernetes conformance tests failed on 12 test cases, moving to a newer version may fix this (https://gerrit.akraino.org/r/c/validation/+/1152)
- AP: Ioakeim to copy the developer info for the UI into the wiki: work in progress
- AP: Indu to take over Miguel's work on the etcd test: airship access is now ok, will check now if it works
Last week items - we did not cover these this time either
- Using k8s on our deployments
- Using versioning for our releases
- Moving the UI to a new git repo
Code badge feature (presentation)
- Code badge will show for a given combination of "blueprint + version + lab" whether the mandatory tests have passed
- We had a discussion about how this could be implemented in practice

July 3, 2019

Follow-ups on the APs from last meeting
- AP: Next week we can present in this meeting a demo of the UI: done
- AP: Ioakeim to copy the developer info for the UI into the wiki: work in progress
- AP: Cristina to start the user guide and then the team members complete it with the layers that they worked on: done
- AP: Indu to check the overlap between e2e test and conformance test that we now run with sonobuoy: done; conclusion is we will drop the e2e for the first validation release as there are too many tests and they take too long to execute
- AP: Indu to take over Miguel's work on the etcd test: need to get access to an airship deployment
- AP: Deepak to look into the options that the LTP test has: ran it internally and automated it in robot; conclusion is that it could be used by any distro
- AP: Andrew to upload the presentation to this wiki page: done Test Set Proposal to TSC
Status on the build jobs: the issue from last week with the UI was fixed; had one more issue with mariadb that was fixed today; overall state is good
Status on Sonobuoy testing on ARM: the patches for it have been merged;
- AP: Cristina to identify the conformance tests that don't have support for aarch64
Status on permanent lab resources for hosting the UI: Ioakeim sent email sent to LF; will open a formal ticket
Status on pending patches: discussion on patch #949; will make a new patch to incorporate the changes discussed
Demo of the UI; discussion on what it takes to consider a project mature
- AP: Ioakeim define the requirements better for the next week's meeting

June 26, 2019

Status on the build jobs: builds have been failing for the last week and a half; patch to fix it has been merged, we expect the jobs to pass on the tonight's run
Status on Sonobuoy testing on ARM: work is still in progress; we need to integrate the build and usage of these images in our own repo as up-streaming them is not feasible
Status on permanent lab resources for hosting the UI: [Ioakeim] request made at the TSC level; we need to check with LF first (Eric Ball)
- Successfully migrated to the ONAP portal SDK (Casablanca version); patch is work in progress
- AP: Next week we can present in this meeting a demo of the UI
Status on user documentation
- Discussion on what (user guide, developer guide) to store where (git, wiki)
- AP: Ioakeim to copy the developer info for the UI into the wiki
- AP: Cristina to start the user guide and then the team members complete it with the layers that they worked on
- Andrew promised to give us feedback on the user guide practicality
Status on Jenkins job that consumes the validation containers:
- This task will take a couple of weeks more until we are able to implement it; Cristina will integrate the k8s test with the IEC blueprint
Status on pending patches
- AP: Indu to check the overlap between e2e test and conformance test that we now run with sonobuoy
- AP: Indu to take over Miguel's work on the etcd test
Multiarch discussion: quick presentation on how to use the containers
Mandatory tests: presentation from Andrew on what tests should be mandatory for the respective layers
- AP: Deepak to look into the options that the LTP test has
- AP: Andrew to upload the presentation to this wiki page
AoB: None

June 19, 2019

Sonobuoy testing on ARM
- Alexandru Avadanii is working on this
Release 2 requirements
- What do we need for the first release?
  - Kubernetes conformance testing works
  - Docker containers are being built automatically every day
  - Need user documentation, starting from ready made containers
  - Need a Jenkins job to install
- When can we have it?
Meetings for next two weeks: Andrew Wilkinson can start the meetings in Zoom, cristinapauna will host
Need lab resources permanently for hosting the UI – topic for TSC meeting

June 12, 2019

New rules enforced: you need a "+2" in a review for merging, but it cannot be your own
- If you are a committer, you can give a "+2" to your own patch
- One "+2" from someone else than submitter is needed
- After this, anyone can merge the patch
Sonobuoy doesn’t work for arm, so we have 2 options→ postponed to next meeting
- call the tests directly without sonobuoy on arm hr
- adapt sonobuoy to work on arm (I incline more to this option but seems difficult to do it upstream so we’ll have to figure out a clever solution)
Documentation strategy
- We have projects lined up to test the Blueprint validation project
- We need to separate user documentation and developer documentation
- Follow Documentation Sub-Committee requirements (Architecture, User Guide and Release Notes)
Proposal to TSC about mandatory tests: https://docs.google.com/presentation/d/1ONU7jmeGVrhbJe2gKRtode1aHKH8teRgfN91LYAPBmo/edit?usp=sharing
We had a discussion about when a blueprint project can be "mature": is the maturation review before Release 2 or after it? Do we want to have "Mature" projects in Release 2? Assumption is yes
We need to define what a Blueprint Validation release is, since the different blueprints need to be able to test against a stable set of tests. It would make sense to call the first release "2", so that BluVal Rel.2 would be used to test blueprints for Release 2. Most likely will also need a BlueVal Rel. 2.1 to fix bugs, and then the blueprints could choose any version 2.x (the latest is probably the best choice but there is no pressure to upgrade if an earlier release works)

June 5, 2019

etcd ha test cases
- Issue is that there are two implementations, depending on how the k8s cluster has been deployed
- It would be better to have a single implementation that would work with both
- Juha and Cristina do not have access to an Airship deployment
Docker makefile bug
- Fixed now
UI implementation
- Discussion was about how to push jobs to validate different blueprints in different labs, since there are three kinds:
  - The public community lab that runs in UNH-IOL
  - 'Private' company labs which can operate in a 'master/agent' Jenkins model
  - Company labs that run in a peer model whereby the Edge lab jenkins only pulls and pushes but is not a slave to the LF jenkins.
- All of these will push the results in the LF Nexus, but only the two first ones can take jobs from LF Jenkins (and hence the UI)
- The UI will soon run in a VM with a public IP

May 29, 2019

Pass variables to robot testcases (838)
- As discussed earlier, we want to make a single file with all variables
- Decision: go with the YAML version
Organize file structure for tests (887)
- Proposal is to create a structure for storing the different tests
- Additional idea is to add "layer" to the structure
- Decision: Go ahead
New idea: there should be a single container per layer, use bluval to select what tests to run
Add container script for k8s ha (839): looks ok
UI discussion: the web UI can run in Docker containers, Ioakeim will make the instructions available on how to build them

May 22, 2019

May 8, 2019

Automatic signed-off-by and linting work now: when you check in code, you need to sign off the commit and you will get a result from automatic code syntax check
Building containers and uploading to hub.docker.com does not yet work, LF staff is investigating
There will be a presentation about this project in the Akraino TSC/PTL meeting next week. Tapio Tallgren will make some slides and cristinapauna will check if the Sonobuyo on Robot Framework in a container will work (dl: Monday)
We also had a discussion about the UI framework. The key part is reading the log files, parsing them, and rendering the result on a web page
We agreed on a new committer: Ioakeim Samaras

April 24

General complaints:
- Let's start having good commit messages to make it easier for outsiders to understand what is going on
- Let's also have signed-off-by lines in git commits
JJB for patch verification is now done (thanks Cristina!)
Container scripts are also working, will be automated soon
Test containers will be put to hub.docker.com under Akraino/validation (https://hub.docker.com/r/akraino/validation)
Dashboard discussion is postponed
There is a new Kanban board for the project: https://jira.akraino.org/secure/RapidBoard.jspa?projectKey=VAL&rapidView=5
We also had a discussion about a common "environment file" which would contain all the parameters that the different tests would need (examples: IP addresses, user names, Kubernetes admin file). Need to build from bottom up
We also discussed bringing a proposal to the TSC about what the mandatory tests will be. Propose to start with Kubernetes conformance tests and Linux Testing Project
Need to make sure that all mandatory tests work on different architectures

April 17, 2019

JJB for patch verification will be made by Cristina
Patch for creating the k8s conformance test docker container was merged. Miguel will try it out with the robot test he's working on. JJB for building the docker images automatically will be done by Cristina; currently waiting for LF to make the dockerhub repo available from the jenkins slaves
Robot tests for Cyclictest are developed by Naga
Documentation and robot tests for LTP and baremetal are developed by Miguel
Juha is looking into robot framework
Deepak presented a proposal for a Dashboard to trigger and view tests results in an user-friendly way; gathered input from the team and will continue the discussion next time.

April 10, 2019

Meeting time check - let's keep the current time, 11 AM Eastern/6 PM Eastern European/8 AM Pacific
New member introduction
New slack channels will be available for Akraino projects, we can use that instead of IRC
We should follow the LF best practices https://lf-releng-docs.readthedocs.io/en/latest/best-practices.html for this project
- We agreed that all patches need a +2 from someone else than the author, but the author can also commit her/himself
All key people have committer rights now
We also need to set up verification. Ideas are welcome, but the following come to mind:
- The best practices recommend Coala
We also need to agree on unit tests for the code. Agreed to make that a part of the reviews
- Later on, we should do more testing. TBD
Status of active code reviews:
- https://gerrit.akraino.org/r/#/c/518/ - will be committed
- https://gerrit.akraino.org/r/#/c/537/ - ongoing
Jira ticket status: https://jira.akraino.org/browse/VAL-4?jql=project%20%3D%20VAL%20AND%20resolution%20%3D%20Unresolved%20ORDER%20BY%20priority%20DESC%2C%20updated%20DESC

March 27, 2019

Agenda:

New committers to project
New meeting time. Propose Wednesdays 10-11 (an hour earlier)
Using IRC? Let's try the channel #akraino
Let's try to clarify the project scope: passing the mandatory validation tests is a part of the Akraino process to mature a process, so it is very important to agree in the Akraino TSC what the mandatory tests are. There can be tests that are supported by the framework but which are not mandatory. The logic should be:
- If the blueprint installs Kubernetes, Kubernetes e2e conformance tests are mandatory
- If the blueprint installs OpenStack, OpenStack Tempest and Rally are mandatory
We also discussed making some HA tests mandatory.
Then we need to agree on how to move forward. Let's say you run a test; what happens next? If you run the test from LF Jenkins, it would be great to sync the results to the LF Jenkins. This requires that you are running your lab in a DMZ or you are opening a lot of ports on the firewall to run a slave Jenkins. So the easier alternative is to copy the results of the tests to the LF Nexus. To make it easy (for scripts) to understand the logs, we need to agree on a common format. For this, we propose running all tests using the Robot Framework.
- Will BVal.yaml be an optional component to make it easy to chain together a number of tests?
- To run the tests, I want to run something like "docker run --env-file abcd.txt --volume output.log akraino_k8s_tests". For this we need
    - The docker container
    - Definition of the env-file format: we need to figure out what information the test will require and write it to the environment file
    - The actual code to read the environment file, launch the Robot Framework, run the tests with Robots, and copy the output to the output.log
- Once I have all this, I can make a Jenkins job in my lab which will launch the tests, and I can also configure it to copy the output somewhere.

March 6, 2019

Discussion about scope of testing + continue discussion about role of xtesting

Recording / chat

The plan is to use the Xtesting part of the OPNFV Functest project as the basis for the Akraino Blueprint Validation project. There will be separate containers for the different test sets, hosted at the Docker Hub. The "source code" (docker yaml etc) will be hosted in the Gerrit repository, a Jenkins job will build the images, and then they will be uploaded. We need to identify the mandatory tests that all Akraino Blueprint projects must pass. There will be a centralized database to store test results, to make it easy to follow the progress of different projects. A web interface to the database would be nice.

Specific notes:

The Nexus repository does not work well for storing Docker images that must support different architectures, as there is no support for manifest files. So it is best to use the Docker Hub for images
Xtesting has two different modes, a "tutorial" one that installs a Jenkins instance, and one for the "Functest" type, where the xtesting framework only creates the Jenkins jobs.
There is no need to have support for voting in the Jenkins jobs
Trevor Bramwell has set up the OPNFV test database, should ask him
Cedric is working on rewriting the web page code that shows the test results

February 27, 2019

Discussion about Xtesting.

Recording / chat

February 13, 2019

Agenda/notes

Weekly meetings
- Time is ok?
- Need to make official somehow
- Recording or not?
- Next week?
Will use Jira
Clarification of workflow
- Will BVF do testing or just provide the tools?
Next tasks
- Collection of test tools
- Define "MVP"
- Get an introduction from OPNFV team
- What are we missing with this approach?
  - ONAP VFN SDK project

Browser not supported