Goals
This wiki describes the specifications for designing the Binary Provisioning Agent required for the Integrated Cloud Native Akraino project.
Overview of BPA
The BPA is part of the infra local controller which runs as a bootstrap k8s cluster in the ICN project. As described in Integrated Cloud Native Akraino project, the purpose of the BPA is to install packages that cannot be installed using kubectl. It will be called once the operating system (Linux) has been installed in the compute nodes by the baremetal operator. The Binary Provisioning Agent will carry out the following functions;
...
For more information on the BPA functions, check out the ICN Akraino project link above
Implementation
We do not intend to make any changes to the existing kubernetes API in order to implement the specifications described in this document. We will simply be extending the Kubernetes API using Custom Resource Definition as described here and then creating a custom controller that will handle the requirements of our provisioning Agent custom resource.
Overview of Proposed Workflow for provisioning CRD
Prerequisites: This workflow assumes that the baremetal CR and baremetal operator have been created and has successfully installed the compute nodes with Linux OS. It also assumes that the BPA controller is running.
...
Fig 1: Illustration of the proposed workflow
Workflow Summary/Description
- Create BPA CRD (Created only once and just creates the BPA resource kind)
Create the BPA Custom Resource
The BPA Operator continues to watch the k8s API server and once it sees that a new BPA CR object has been created, it queries the k8s API server for the Baremetal hosts lists. The baremetal hosts lists contains information about the compute nodes provisioned including the IP address, CPU, memory..etc of each host.
The BPA operator looks into the baremetal hosts list and knows which hosts should be master and which should be workers. As the master and worker fields have various parameters, it can do this in various ways;
- If the MAC address is provided in the BPA CR object, it compares that value with the value in the hosts list and assigns the roles. For example if a mac address of 00:c5:16:05:61:b2 is specified for master in the BPA CR spec, it checks the baremetal list for a host that has that MAC address and gives it the role of master.
- If there is no MAC address specified but just resources, it checks the baremetal list for hosts that meet the resource requirements
- If both MAC address and resource requirements are provided, it finds the host with the specified MAC address and confirms that the host meets the resource requirement provided in the BPA CR and then assigns the role.
- Using the MAC address of the host, the BPA operator looks in the DHCP file of the DHCP server running on the same host it is running and determines the IP address that corresponds to that MAC address
- The BPA operator reads a file containing the default username and password for the various hosts, copies its public key to those hosts in order to use kubespray later.
The BPA operator then creates the hosts.ini file using the assigned roles and their corresponding IP addresses.
The BPA operator then installs kubernetes using kubespray on the compute nodes thus creating an active kubernetes cluster. During installation, it would continue to check the status of the installation
On successful completion of the k8s cluster installation, the BPA operator would save the application-k8s kubeconfig file in order to access the k8s cluster and make changes such as software updates or add a worker node for future purposes.
BPA CRD
The BPA CRD tells the Kubernetes API how to expose the provisioning custom resource object. The CRD yaml file is applied using
...
“kubectl create -f bpa_v1alpha1_provisioning_crd.yaml” See below for the CRD definition.
BPA CRD Yaml File (*_crd.yaml)
| Code Block | ||
|---|---|---|
| ||
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
name: provisionings.bpa.akraino.org
spec:
group: bpa.akraino.org
names:
kind: Provisioning
listKind: ProvisioningList
plural: provisionings
singular: provisioning
shortNames:
- bpa
scope: Namespaced
subresources:
status: {}
validation:
openAPIV3Schema:
properties:
apiVersion:
description:
type: string
kind:
description:
type: string
metadata:
type: object
spec:
type: object
status:
type: object
version: v1alpha1
versions:
- name: v1alpha1
served: true
storage: true |
Provisioning Agent Object Definition( *_types,go)
The provisioning_types.go file is the API for the provisioning agent custom resource.
...
- Masters: This variable will contain an array of Master objects. The master struct as defined in the *-types.go file above contains CPU and memory information, this information would be used by the BPA operator to determine which compute nodes to assign the role of Master to when it gets the baremetal list from the API server.
- Workers: This variable will contain an array of Worker objects. Similar to the case of the Masters variables, the Worker struct will contain resource requirements for the Worker nodes and the BPA operator will use this information to determine which hosts to assign the role of worker.
- Replicas: An integer that defines the number of pods that should run when the CR is deployed.
Sample Provisioning CR YAML files
| Code Block | ||
|---|---|---|
| ||
apiVersion: bpa.akraino.org/v1alpha1
kind: Provisioning
metadata:
name: provisioning-sample
spec:
master:
mac-address: 00:c6:14:04:61:b2
worker:
mac-address: 00:c6:14:04:61:b2
|
...
Based on the values above, when the BPA operator gets the baremetal hosts object (Step 5in figure 1), it would assign hosts with 10 CPUs and 4Gi memory the role of master and it would assign hosts with 20CPUs and 8Gi memory the role of worker.
Sample Baremetal Lists from Query
| Code Block | ||
|---|---|---|
| ||
apiVersion: v1
items:
- apiVersion: metal3.io/v1alpha1
kind: BareMetalHost
metadata:
creationTimestamp: "2019-07-20T01:43:19Z"
finalizers:
- baremetalhost.metal3.io
generation: 2
name: demo-provisioning
namespace: metal3
resourceVersion: "35002"
selfLink: /apis/metal3.io/v1alpha1/namespaces/metal3/baremetalhosts/demo-provisioning
uid: 3b22014e-9252-4f15-89a5-67f96e1a07a2
spec:
bmc:
address: ipmi://172.31.1.17
credentialsName: demo-provisioning-bmc-secret
description: ""
externallyProvisioned: false
hardwareProfile: ""
image:
checksum: http://172.22.0.1/images/bionic-server-cloudimg-amd64.md5sum
url: http://172.22.0.1/images/bionic-server-cloudimg-amd64.qcow2
online: true
status:
errorMessage: ""
goodCredentials:
credentials:
name: demo-provisioning-bmc-secret
namespace: metal3
credentialsVersion: "30393"
hardware:
cpu:
arch: x86_64
clockMegahertz: 3700
count: 72
flags:
- ….
- xtopology
- xtpr
model: Intel(R) Xeon(R) Gold 6140M CPU @ 2.30GHz
firmware:
bios:
date: 11/07/2018
vendor: Intel Corporation
version: SE5C620.86B.00.01.0015.110720180833
hostname: localhost.localdomain
nics:
- ip: ""
mac: 3c:fd:fe:9c:88:60
model: 0x8086 0x1572
name: eth0
pxe: false
speedGbps: 0
vlanId: 0
- ip: 172.22.0.55
mac: a4:bf:01:64:86:6f
model: 0x8086 0x37d2
name: eth5
pxe: true
speedGbps: 0
vlanId: 0
…
ramMebibytes: 262144
storage:
- hctl: "6:0:0:0"
model: INTEL SSDSC2KB48
name: /dev/sda
rotational: false
serialNumber: BTYF8290022M480BGN
sizeBytes: 480103981056
vendor: ATA
wwn: "0x55cd2e414fc888c1"
wwnWithExtension: "0x55cd2e414fc888c1"
- hctl: "7:0:0:0"
model: INTEL SSDSC2KB48
name: /dev/sdb
rotational: false
serialNumber: BTYF83160FDB480BGN
sizeBytes: 480103981056
vendor: ATA
wwn: "0x55cd2e414fd7b5a3"
wwnWithExtension: "0x55cd2e414fd7b5a3"
systemVendor:
manufacturer: Intel Corporation
productName: S2600WFT (SKU Number)
serialNumber: BQPW84200264
hardwareProfile: unknown
lastUpdated: "2019-07-20T02:41:30Z"
operationalStatus: OK
poweredOn: false
provisioning:
ID: 94fa2511-3cb1-4372-ab42-9c377db8aeca
image:
checksum: ""
url: ""
state: provisioning
kind: List
metadata:
resourceVersion: ""
selfLink: ""
|
...
In addition, we would also have two other CRDs that the BPA would use to perform its functions;
- Software CRD
- Cluster CRD
Software CRD
The software CRD will install the required software, drivers and perform software updates
...
| Code Block | ||
|---|---|---|
| ||
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
name: cluster.bpa.akraino.org
spec:
group: bpa.akraino.org
names:
kind: software-updater
listKind: software-updaterList
plural: software-updaters
singular: software-updater
shortNames:
- su
scope: Namespaced
subresources:
status: {}
validation:
openAPIV3Schema:
properties:
apiVersion:
description:
type: string
kind:
description:
type: string
metadata:
type: object
spec:
type: object
status:
type: object
version: v1alpha1
versions:
- name: v1alpha1
served: true
storage: true |
Cluster CRD
The cluster CRD will have the Cluster name and contain the provisioning CR and/or the software CR for the specified cluster
...
| Code Block | ||
|---|---|---|
| ||
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
name: cluster.bpa.akraino.org
spec:
group: bpa.akraino.org
names:
kind: cluster
listKind: clusterList
plural: clusters
singular: cluster
shortNames:
- cl
scope: Namespaced
subresources:
status: {}
validation:
openAPIV3Schema:
properties:
apiVersion:
description:
type: string
kind:
description:
type: string
metadata:
type: object
spec:
type: object
status:
type: object
version: v1alpha1
versions:
- name: v1alpha1
served: true
storage: true |
Open Questions
- How does the BPA operator get the SSH information of the compute hosts ?
Future Work
This proposal would make it possible to assign roles to nodes based on the features discovered. Currently, the proposal makes use of CPU, memory, SRIOV and QAT. However the baremetal operator list returns much more information about the nodes, we would be able to extend this feature to allow the operator assign roles based on more complex requirements such as CPU model. This would feed into Hardware Platform Awareness (HPA)