Linux

Linux Test Project

The Linux Test Project is a joint project started by SGI, developed and maintained by IBM, Cisco, Fujitsu, SUSE, Red Hat and others, that has a goal to deliver test suites to the open source community that validate the reliability, robustness, and stability of Linux. The LTP testsuite contains a collection of tools for testing the Linux kernel and related features.

Links: https://linux-test-project.github.io, https://github.com/linux-test-project/ltp

Cyclictest

Cyclictest accurately and repeatedly measures the difference between a thread's intended wake-up time and the time at which it actually wakes up in order to provide statistics about the system's latencies. It can measure latencies in real-time systems caused by the hardware, the firmware, and the operating system.

Links: https://wiki.linuxfoundation.org/realtime/documentation/howto/tools/cyclictest/start?s[]=cyclictest

Storage

Bonnie++

Bonnie++ is a benchmark suite that is aimed at performing a number of simple tests of hard drive and file system performance. Then you can decide which test is important and decide how to compare different systems after running it.

Links: https://www.coker.com.au/bonnie++/

Networking

Iperf3

iperf is a tool for active measurements of the maximum achievable bandwidth on IP networks. It supports tuning of various parameters related to timing, protocols, and buffers. For each test it reports the measured throughput / bitrate, loss, and other parameters.

Links: https://github.com/esnet/iperf

OpenStack

Tempest

This is a set of integration tests to be run against a live OpenStack cluster. Tempest has batteries of tests for OpenStack API validation, scenarios, and other specific tests useful in validating an OpenStack deployment.

Links: https://docs.openstack.org/tempest/latest/overview.html

Kubernetes

The Kubernetes integration testing only uses the local machine. The integration tests are run using make test-integration. The Kubernetes integration tests are written using the normal golang testing package but expect to have a running etcd instance to connect to.

End-to-end (e2e) tests for Kubernetes provide a mechanism to test end-to-end behavior of the system, and is the last signal to ensure end user operations match developer specifications. Although unit and integration tests provide a good signal, in a distributed system like Kubernetes it is not uncommon that a minor change may pass all unit and integration tests, but cause unforeseen changes at the system level. The primary objectives of the e2e tests are to ensure a consistent and reliable behavior of the kubernetes code base, and to catch hard-to-test bugs before users do, when unit and integration tests are insufficient.

Kubernetes has a large set of end-to-end tests which can be classified as follows:

• If a test has no labels, it is expected to run fast (under five minutes), be able to be run in parallel, and be consistent.

• [Slow]: If a test takes more than five minutes to run (by itself or in parallel with many other tests), it is labeled [Slow]. This partition allows us to run almost all of our tests quickly in parallel, without waiting for the stragglers to finish.

• [Serial]: If a test cannot be run in parallel with other tests (e.g. it takes too many resources or restarts nodes), it is labeled [Serial], and should be run in serial as part of a separate suite.

• [Disruptive]: If a test restarts components that might cause other tests to fail or break the cluster completely, it is labeled [Disruptive]. Any [Disruptive] test is also assumed to qualify for the [Serial] label, but need not be labeled as both. These tests are not run against soak clusters to avoid restarting components.

• [Conformance]: Designate that this test is included in the Conformance test suite for Conformance Testing. This test must meet a number of requirements to be eligible for this tag. This tag does not supersed any other labels.

• [LinuxOnly]: If a test is known to be using Linux-specific features (e.g.: seLinuxOptions) or is unable to run on Windows nodes, it is labeled [LinuxOnly]. When using Windows nodes, this tag should be added to the skipargument.

Link: https://github.com/kubernetes/community/blob/master/contributors/devel/sig-testing/e2e-tests.md, https://github.com/kubernetes/community/blob/master/contributors/devel/sig-testing/e2e-tests.md

Kubernetes performance testing can be done using the end-to-end tests by measuring the response time for different API calls after different operations (example: launch 100 containers, measure time until they respond).