Page Comparison

Table of Contents
maxLevel 3

Virtlet

...

To solve these problems, we should first have a clear knowledge of device plugin. A related concept for device plugin is Kubernetes extended-resources. In conclusion, By sending a patch node request to the Kubernetes apiserver, a custom resource type is added to the node, which is used for the quota statistics of the resource and the corresponding QoS configuration.

Example

To send a patch node request conveniently, request conveniently, start a proxy, so that you can easily send requests to the Kubernetes API server, we first execute kube proxy command to command to start it temporarily, then add six intel.com/devices resource to a node (~1 in the commands will automatically transform into /):

curl --header "Content-Type: application/json-patch+json" \
--request PATCH \
--data '[{"op": "add", "path": "/status/capacity/intel.com~1devices", "value": "6"}]' \
http://localhost:8001/api/v1/nodes/<your-node-name>/status

Now we extend 6 intel.com/devices resources for your node, then we can see 

kubectl describe node xxx
...
Capacity:
  ephemeral-storage:   3650656984Ki
  cpu:                 72
  memory:              263895388Ki
  intel.com/devices:   6
  pods:                110
...

Now we can use these resources in our pod by adding intel.com/devices: "1" to spec.containers.resources.requests/limits and the pod will be scheduled with statistics.

To clean up the extended resources, execute the following commands:

curl --header "Content-Type: application/json-patch+json" \
--request PATCH \
--data '[{"op": "remove", "path": "/status/capacity/intel.com~1devices"}]' \
http://localhost:8001/api/v1/nodes/<your-node-name>/status

Device plugin

Overview

Kubernetes provides to vendors a mechanism called device plugins to finish the following three tasks, device plugins are simple gRPC servers that may run in a container deployed through the pod mechanism or in bare metal mode.

service DevicePlugin {
	// returns a stream of []Device
	rpc ListAndWatch(Empty) returns (stream ListAndWatchResponse) {}
	rpc Allocate(AllocateRequest) returns (AllocateResponse) {}
}

• advertise devices.
• monitor devices (currently perform health checks).
• hook into the runtime to execute device specific instructions (e.g: Clean GPU memory) and to take in order to make the device available in the container.

...

Why device plugin

• Very few devices are handled natively by Kubelet (cpu and memory)
• Need a sustainable solution for vendors to be able to advertise their resources to Kubelet and monitor them without writing custom Kubernetes code
• A consistent and portable solution to consume hardware devices across k8s clusters to use a particular device type (GPU, QAT, FPGA, etc.) in pods
• ...

How it works

In Kubernetes, Kubelet will offer a register gRPC server which allows device plugin register itself to Kubelet. When starting, the device plugin will make a (client) gRPC call to the Register function that Kubelet exposes. The device plugins sends a RegisterRequest to Kubelet to notify Kubelet of the following informations, and Kubelet answers to the RegisterRequest with a RegisterResponse containing any error Kubelet might have encountered (api version not supported, resource name already register), then the device plugin start its gRPC server if it did not receive an error.

1. Its own unix socket name, which will receive the requests from Kubelet through the gRPC apis.
2. The api version of device plugin itself
3. The resource name they want to advertise. The resource name must follow a specified format (vendor-domain/vendor-device). such as intel.com/qat

After successful registration, Kubelet will call the ListAndWatch function from device plugin. A ListAndWatch function is for the Kubelet to Discover the devices and their properties as well as notify of any status change (devices become unhealthy). The list of devices is returned as an array of all devices description information (ID, health status) of the resource. Kubelet records this resource and its corresponding number of devices to node.status.capacity/allocable and updates it to apiserver. This function will always loop check, once the device is abnormal or unplugged from the machine, it will update and return the latest device list to Kubelet.

In this way, when creating a pod, fields such as intel.com/qat can be added to spec.containers.resource.limits/requests: "1" to inform Kubernetes to schedule the pod to nodes with more than one intel.com/qat resource allowance. When the pod is to run, Kubelet will call device plugin allocate function. Device plugin may do some initialization operations, such as QAT configuration or QRNG initialization. If initialization is successful, this function will return how to config the device assigned to the pod when the container is created, and this configuration will be passed to the container runtime as a parameter used to run container.

User Flow

...

To use the extend resource, we add intel.com/qat to spec.containers.resource.limits/requests, we expect the request to have limits == requests.

1. A user submits a pod spec requesting 1 QAT through intel.com/qat
2. The scheduler filters the nodes which do not match the resource requests
3. The pod lands on the node and Kubelet decides which device should be assigned to the pod
4. Kubelet calls Allocate on the matching Device Plugins
5. The user deletes the pod or the pod terminates

When receiving a pod which requests devices, Kubelet is in charge of:

• Deciding which device to assign to the pod's containers
• Calling the Allocate function with the list of devices

The Kubernetes scheduler is in charge of filtering the nodes which cannot satisfy the resource requests.

Enable QAT supported by virtlet

...

Gaps detection in source code

When testing the qat sriov support condition with the officer virtlet image, 

Fix

Example