| Table of Contents | ||
|---|---|---|
|
...
In details, KubeVirt technology addresses the needs of development teams that have adopted or want to adopt Kubernetes but possess existing Virtual Machine-based workloads that cannot be easily containerized. So KubeVirt extends Kubernetes by adding additional virtualization resource types (especially the VM/VMI type) through Kubernetes's Custom Resource Definitions API. By using this mechanism, the Kubernetes API can be used to manage these VM resources alongside all other resources Kubernetes provides. The resources themselves in Kubernetes are not enough to launch virtual machines. For this to happen, the functionality and business logic needs to be added to the cluster. Scheduling, networking, and storage are all delegated to Kubernetes, while KubeVirt provides the virtualization functionality. The functionality is not added to Kubernetes itself, but rather added to a Kubernetes cluster by running additional controllers and agents on an existing cluster. And these necessary controllers and agents are all provided by KubeVirt.
...
Intel QuickAssist Technology is developed by Intel and runs on the Intel Architecture to provide security and compression acceleration capabilities to improve performance and efficiency. It will offload the workloads like cryptography and compression from the CPU to handwarehardware. Server, networking, big data, and storage applications use Intel QuickAssist to offload compute-intensive operations, such as:
...
It has made great benefits in many areas, such as Hadoop AcceletationAcceleration, OpenSSL Integration, SDN and NFV Solutions Boost and so on.
...
With above support, QAT makes it easier for developers to integrate the accelerators in their designs and thus decrease the development time. And it can increases increase business flexibility by offering solutions that best fit the changing business requirements. It also frees up the valuable cycles on processors and allows it to perform value-added functionality.
What's more, QAT provides a uniform means of communication between accelerators, applications, and acceleration technologies. Due to this, the resources are managed more productively. Then It can boost application throughput, by reducing the demand on the platform and maximizes maximizing the CPU utilization.
Gaps
First of all, in a Kubernetes cluster, if we need utilize the QAT card and assign its vf to a container, we will compile the QAT driver on the host and deploy the QAT device plugin. After the QAT device plugin register successfully, a ListAndWatch function is for the Kubelet to Discover the devices and their properties as well as notify of any status change (devices become unhealthy). The list of devices is returned as an array of all devices description information (ID, health status) of the resource. Kubelet records this resource and its corresponding number of devices to node.status.capacity/allocable and updates it to apiserver.
...
So to enable QAT in Kubevirt, it is necessray to create related segments to the validation service and add the QAT feature Gate verified method.
- Add necessary information to swagger.json which is an add on for kubenetes API used in Kubevirt
... "qats": { ... "v1.QAT": { |
|---|
}, |
|---|
- Add Feature Gate to webhook validation service tp
- to config it in configmap
... if spec.Domain.Devices.QATs != nil && !config.QATPassthroughEnabled() { ... |
|---|
...
Non-privileged
To meet the Kubernetes and Kubevirt community specifications, the pod should be non-privileged. So we should mount the assigned QAT pci device to the VM through the interfaces Kubevirt provided.
| if util.IsQATVMI(vmi) { for _, qat := range vmi.Spec.Domain.Devices.QATs { requestResource(&resources, qat.DeviceName) } } |
|---|
PCI passthrough
...
This will call the Kubevirt to mount necessary resources for QAT into the VM, such as /sys/devices/.
PCI passthrough
Fundamentally, Kubevirt create a Kubernetes CRD to hold some resource configurations that fit a libvrit instance. So actually to assign a QAT vf into a VM is to create a libvirt and insert the specific device into hostdev block. (In libvirt, hostdev label means a plain host device assignment with all its limitations)
| <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x3d' slot='0x02' function='0x2'/> </source> <alias name |
|---|
<source>
<address domain='0x0000' bus='0x3d' slot='0x02' function='0x2'/>
</source>
<alias name='hostdev0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
</hostdev>
Example
...
| ='hostdev0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/> </hostdev> |
|---|
So when a QAT vf resource is required, I call the Kubevirt method to get the device id of the assigned QAT and integrate it into libvirt domain.
// Append HostDev to libvirt DomXML if QAT is required qatPCIAddresses := append([]string{}, c.QATDevices...) ... |
|---|
Example
Based on the Kubevirt official example VMI file, to make use of my QAT version, after building its docker images and uploading to docker, we should open the QAT feature gate through configmap.
| apiVersion: v1 kind: ConfigMap metadata: name: kubevirt-config namespace: kubevirt labels: kubevirt.io: "" data: feature-gates: "QAT" |
|---|
And the key point of QAT VMI:
... spec: ... |
|---|
Conclusion
The patch now has not been merged into the Kubevirt Project and is in review now. (PR: https://github.com/kubevirt/kubevirt/pull/2980)