Overview

This document describes how to deploy blueprints from Akraino's KNI Blueprint Family. It is common to all blueprints in that family, unless otherwise noted.

...

The resource requirements for deployment depend on the specific blueprint and deployment target. Please see:

...

The current KNI blueprints use the openshift-install tool from the OKD Kubernetes distro to stand up a minimal Kubernetes cluster. All other Day 1 and Day 2 operations are then driven purely through manipulation of declarative Kubernetes manifests. To use this in the context of Akraino KNI blueprints, the project has created a helper CLI tool that needs to be installed first.

If necessary, install golang binary (incl. GOPATH var).

Next, install the following dependencies:

sudo yum install -y make gcc libvirt-devel

Then install the knictl:

...

Secrets

Most secrets (TLS certificates, Kubernetes API keys, etc.) will be auto-generated for you, but you need to provide at least two secrets yourself:

a public SSH key
a pull secret

The public SSH key is automatically added to every machine provisioned into the cluster and allows remote access to that machine. In case you don't have / want to use an existing key, you can create a new key pair using:

ssh-keygen -t rsa -b 2048 -f ~/.ssh/id_rsa

The pull secret is used to download the container images used during cluster deployment. Unfortunately, the OKD Kubernetes distro used by the KNI blueprints does not (yet) provide pre-built container images for all of the deployed components. Instead of going through the hassle of building those from source, we use the ones made available by openshift.com. Therefore, you need to go to https://cloud.redhat.com/openshift/install/metal/user-provisioned, log in (creating a free account, if necessary), and hit "Download Pull Secret".

Create a $HOME/.kni folder and copy the following files:

id_rsa.pub → needs to contain the public key that you want to use to access your nodes
pull-secret.json → needs to contain the pull secret previously copied

Pre-Requisites for Deploying to AWS

For deploying a KNI blueprint to AWS, you need to

add a public hosted DNS zone for the cluster to Route53,
validate your AWS quota in the chosen region is sufficient,
set up an API user account with the necessarily IAM privileges.

Please see the upstream documentation for details.

Store the aws-access-key-id and aws-secret-access-key in a credentials file inside $HOME/.aws, with the following format:

Code Block

language	bash

[default]
aws_access_key_id=xxx
aws_secret_access_key=xxx

Pre-Requisites for Deploying to Bare Metal

The baremetal UPI install can be optionally automated when using knictl (see below). When attempting a manual baremetal UPI install, however, please be sure to read: https://docs.openshift.com/container-platform/4.1/installing/installing_bare_metal/installing-bare-metal.html

Pre-Requisites for Deploying to Libvirt

For deploying a KNI blueprint to VMs on KVM/libvirt, you need to

...

source utils/prep_host.sh

...

Please see the upstream documentation for details.

Structure of a site

In order to deploy a blueprint, you need to create a repository with a site. The site configuration is based in kustomize, and needs to use our blueprints as base, referencing that properly. Sample sites for deploying on libvirt, AWS and baremetal can be seen on: https://github.com/akraino-edge-stack/kni-blueprint-pae/tree/master/sites.
Site needs to have this structure:

.
├── 00_install-config
│   ├── install-config.name.patch.yaml
│   ├── install-config.patch.yaml
│   ├── kustomization.yaml
│   └── site-config.yaml
├── 01_cluster-mods
│   ├── kustomization.yaml
│   ├── manifests
│   └── openshift
├── 02_cluster-addons
│   └── kustomization.yaml
└── 03_services
└── kustomization.yaml

00_install-config

This folder will contain the basic settings for the site, including the base blueprint/profile, and the site name/domain. The following files are needed:

...

Code Block

language	yml

bases:
- git::https://gerrit.akraino.org/r/kni/blueprint-pae.git//profiles/production.aws/00_install-config

patches:
- install-config.patch.yaml

patchesJson6902:
- target:
version: v1
kind: InstallConfig
name: cluster
path: install-config.name.patch.yaml

transformers:
- site-config.yaml

...

install-config.patch.yaml: is a patch to modify the domain from the base blueprint. You need to customize with the domain you want to give to your site

Code Block

language	yml

apiVersion: v1
kind: InstallConfig
metadata:
name: cluster
baseDomain: devcluster.openshift.com

...

Code Block

language	yml

- op: replace
  path: "/metadata/name"
  value: kni-site

site-config.yaml: site configuration file, you can add entries in config to override behaviour of knictl (currently just releaseImageOverride is supported)

Code Block

language	yml

apiVersion: kni.akraino.org/v1alpha1
kind: SiteConfig
metadata:
 name: notImportantHere
 config:
   releaseImageOverride: registry.svc.ci.openshift.org/origin/release:4.1

NOTE: If you are deploying on baremetal, specific configuration needs to be set. This is going to be covered in an specific section for it

01_cluster_mods

This is the directory that will contain all the customizations for the basic cluster deployment. You could create patches for modifying number of masters/workers, network settings... everything that needs to be modified on cluster deployment time. It needs to have a basic kustomization.yaml file, that will reference the same level file for the blueprint. And you could create additional patches following kustomize syntax:

Code Block

language	yml

bases:
- git::https://gerrit.akraino.org/r/kni/blueprint-pae.git//profiles/production.aws/01_cluster-mods

02_cluster_addons and 03_services

Follow same structure as 01_cluster_mods, but in this case is for adding additional workloads after cluster deployment. They also need to have a kustomization.yaml file that references the file of the same level for the blueprint, and can include additional resources and patches.

How to deploy

...

Pre-Requisites for Deploying to AWS

For deploying a KNI blueprint to AWS, you need to

add a public hosted DNS zone for the cluster to Route53,
validate your AWS quota in the chosen region is sufficient,
set up an API user account with the necessarily IAM privileges.

Please see the upstream documentation for details.

Store the aws-access-key-id and aws-secret-access-key in a credentials file inside $HOME/.aws, with the following format:

Code Block

language	bash

[default]
aws_access_key_id=xxx
aws_secret_access_key=xxx

Pre-Requisites for Deploying to Bare Metal

The baremetal UPI install can be optionally automated when using knictl (see below). When attempting a manual baremetal UPI install, however, please be sure to read: https://docs.openshift.com/container-platform/4.1/installing/installing_bare_metal/installing-bare-metal.html

Pre-Requisites for Deploying to Libvirt

For deploying a KNI blueprint to VMs on KVM/libvirt, you need to

provision a machine with CentOS 1810 to serve as virthost and
prepare the virthost by running
source utils/prep_host.sh

from the kni-installer repo on that host.

Please see the upstream documentation for details.

Create site for AWS and libvirt

In order to deploy a blueprint, you need to create a repository with a site. The site configuration is based in kustomize, and needs to use our blueprints as base, referencing that properly. Sample sites for deploying on libvirt, AWS and baremetal can be seen on: https://github.com/akraino-edge-stack/kni-blueprint-pae/tree/master/sites.
Site needs to have this structure:

.
├── 00_install-config
│   ├── install-config.name.patch.yaml
│   ├── install-config.patch.yaml
│   ├── kustomization.yaml
│   └── site-config.yaml
├── 01_cluster-mods
│   ├── kustomization.yaml
│   ├── manifests
│   └── openshift
├── 02_cluster-addons
│   └── kustomization.yaml
└── 03_services
└── kustomization.yaml

00_install-config

This folder will contain the basic settings for the site, including the base blueprint/profile, and the site name/domain. The following files are needed:

kustomization.yaml: key file, where it will contain a link to the used blueprint/profile, and a reference to the used patches to customize the site bases:

Code Block

language	yml

bases:
- git::https://gerrit.akraino.org/r/kni/blueprint-pae.git//profiles/production.aws/00_install-config

patches:
- install-config.patch.yaml

patchesJson6902:
- target:
version: v1
kind: InstallConfig
name: cluster
path: install-config.name.patch.yaml

transformers:
- site-config.yaml

The entry in bases needs to reference the blueprint being used (in this case blueprint-pae), and the profile install-config file (in this case production.aws/00_install-config). The other entries need to be just written literally.

install-config.patch.yaml: is a patch to modify the domain from the base blueprint. You need to customize with the domain you want to give to your site
Code Block
language yml
apiVersion: v1 kind: InstallConfig metadata: name: cluster baseDomain: devcluster.openshift.com
install-config.name.patch.yaml: is a patch to modify the site name from the base blueprint. You need to customize with the name you want to give to your site

Code Block

language	yml

- op: replace
  path: "/metadata/name"
  value: kni-site

site-config.yaml: site configuration file, you can add entries in config to override behaviour of knictl (currently just releaseImageOverride is supported)

Code Block

language	yml

apiVersion: kni.akraino.org/v1alpha1
kind: SiteConfig
metadata:
 name: notImportantHere
 config:
   releaseImageOverride: registry.svc.ci.openshift.org/origin/release:4.1

NOTE: If you are deploying on baremetal, specific configuration needs to be set. This is going to be covered in an specific section for it

01_cluster_mods

This is the directory that will contain all the customizations for the basic cluster deployment. You could create patches for modifying number of masters/workers, network settings... everything that needs to be modified on cluster deployment time. It needs to have a basic kustomization.yaml file, that will reference the same level file for the blueprint. And you could create additional patches following kustomize syntax:

Code Block

language	yml

bases:
- git::https://gerrit.akraino.org/r/kni/blueprint-pae.git//profiles/production.aws/01_cluster-mods

02_cluster_addons and 03_services

Follow same structure as 01_cluster_mods, but in this case is for adding additional workloads after cluster deployment. They also need to have a kustomization.yaml file that references the file of the same level for the blueprint, and can include additional resources and patches.

How to deploy on AWS and libvirt

The whole deployment workflow is based on knictl CLI tool that this repository is providing.

CLI tool
Anchor
knictl
knictl

The current KNI blueprints use the openshift-install tool from the OKD Kubernetes distro to stand up a minimal Kubernetes cluster. All other Day 1 and Day 2 operations are then driven purely through manipulation of declarative Kubernetes manifests. To use this in the context of Akraino KNI blueprints, the project has created a helper CLI tool that needs to be installed first on Installer Node.

If necessary, install golang binary (incl. GOPATH var) using following steps, you can use latest version instead of the one given below.

wget https://golang.org/dl/go1.13.4.linux-amd64.tar.gz 

tar -C /usr/local -xzf go1.13.4.linux-amd64.tar.gz

export PATH=$PATH:/usr/local/go/bin

Next, install the following dependencies:

sudo yum install -y make gcc libvirt-devel

Then install the knictl:

mkdir -p $GOPATH/src/gerrit.akraino.org/kni
cd $GOPATH/src/gerrit.akraino.org/kni
git clone https://gerrit.akraino.org/r/kni/installer
cd installer
make build
mkdir -p $GOPATH/bin/
cp knictl $GOPATH/bin/

cp knictl /usr/local/go/bin/

Secrets

Most secrets (TLS certificates, Kubernetes API keys, etc.) will be auto-generated for you, but you need to provide at least two secrets yourself:

a public SSH key
a pull secret

The public SSH key is automatically added to every machine provisioned into the cluster and allows remote access to that machine. In case you don't have / want to use an existing key, you can create a new key pair using:

ssh-keygen -t rsa -b 2048 -f ~/.ssh/id_rsa

The pull secret is used to download the container images used during cluster deployment. Unfortunately, the OKD Kubernetes distro used by the KNI blueprints does not (yet) provide pre-built container images for all of the deployed components. Instead of going through the hassle of building those from source, we use the ones made available by openshift.com. Therefore, you need to go to https://cloud.redhat.com/openshift/install/metal/user-provisioned, log in (creating a free account, if necessary), and hit "Download Pull Secret".

Create a $HOME/.kni folder and copy the following files:

id_rsa.pub → needs to contain the public key that you want to use to access your nodes
pull-secret.json → needs to contain the pull secret previously copied

1. Fetch requirements for a site.

...

This will execute kustomize on the site manifests and will apply the output to the cluster. After that, the site deployment can be considered as finished.

...

How to Deploy on

...

Baremetal

Minimal hardware footprint needed

...

(where x.x.x.x is IPMI port IP of your master/worker node, this is followed by root username and password for IPMI e.g. iDRAC)

High level steps

Create

...

site for Baremetal

First step to start a baremetal deployment is to have a site defined, with all the network and baremetal settings defined in the yaml files. A sample of site using this baremetal automation can be seen here .
In order to define the settings for a site, the first section 00_install-config needs to be used.
Start by creating a kustomization file like the following: https://github.com/akraino-edge-stack/kni-blueprint-pae/blob/master/sites/community.baremetal.edge-sites.net/00_install-config/kustomization.yaml

...

In this kustomization file we are patching the default install-config, and also adding some extra files to define networking (site-config.yaml).

...

ha-lab-ipmi-creds.yaml:

This file is not shown on the site structure as it contains private content. This file should be present with given name in 00_install-config. It needs to have following structure:

...

apiVersion: v1
kind: InstallConfig
baseDomain: baremetal.edge-sites.net <- domain of your site
compute:
 - name: worker
   replicas: 2 <- number of needed workers
controlPlane:
   name: master
   platform: {}
   replicas: 1 <- number of needed masters (1/3)
metadata:
   name: cluster/3)
metadata:
   name: cluster  <- Do not change this value as this is not cluster name
networking:
  clusterNetworks:
  - cidr: 10.128.0.0/14
    hostPrefix: 23
  networkType: OpenShiftSDN
  serviceNetwork:
  - 172.30.0.0/16
platform:
   none: {}
   apiVIP: 192.168.111.4  <- IP for Kubernetes api endpoint, needs to be on the range of your baremetal network
   ingressVIP: 192.168.111.3 <- IP for the Kubernetes ingress endpoint, needs to be on the range of your baremetal network
   dnsVIP: 192.168.111.2 <- IP for the Kubernetes DNS endpoint, needs to be on the range of your baremetal network
   hosts:
      # Master nodes are always RHCOS
      -  name: master-0
         role: master
         bmc:
            address: ipmi://10.11.7.12 <- ipmi address for master
            credentialsName: community-lab-ipmi <- this needs to reference the name of the secret provided in credentials.yaml
         bootMACAddress: 3C:FD:FE:CD:98:C9  <- mac address for the provisioning interface of your master
         sdnMacAddress: 3C:FD:FE:CD:98:C8   <- mac address for the baremetal interface of your master
         # sdnIPAddress: 192.168.111.11     <- Optional -- Set static IP on your baremetal for your master
         hardwareProfile: default
         osProfile: 
            # With role == master, the osType is always rhcos
            # And with type rhcos, the following are settings are available
            type: rhcos
            pxe: bios         <- pxe boot type either bios (default if not specified) or uefi
            install_dev: sda  <- where to install the operating system (sda is the default)
      # Worker nodes can be either rhcos (default) || centos (7.x) || rhel (8.x)
      -  name: worker-0
         role: worker
         bmc: 
            address: ipmi://10.11.7.13
            credentialsName: community-lab-ipmi
         bootMACAddress: 3C:FD:FE:CD:9E:91
         sdnMacAddress: 3C:FD:FE:CD:9E:90
         hardwareProfile: default
         provisioning_interface: enp134s0f1 <- specify that if the provisioning interface is different than the one you will provide on next site-config.yaml
         baremetal_interface: enp134s0f0 <- specify that if the baremetal interface is different than the one you will provide on next site-config.yaml
         # If an osProfile/type is not defined, rhe node defaults to RHCOS
         osProfile: 
            type: centos7
            # With type: rhcos the following are settings are available
            pxe: bios    # pxe boot type either bios (default if not specified) or uefi
            install_dev: sda  # where to install the operating system (sda is the default)
      -  name: worker-1
         role: worker
         bmc: 
            address: ipmi://10.11.7.14
            credentialsName: community-lab-ipmi
         bootMACAddress: 3C:FD:FE:CD:9B:81
         sdnMacAddress: 3C:FD:FE:CD:9B:80
         hardwareProfile: default
         # If an osProfile/type is not defined, rhe node defaults to RHCOS
         # osProfile: 
            # type: rhcos
            # With type: rhcos the following are settings are available
            # pxe: bios|uefi    # pxe boot type either bios (default if not specified) or uefi
            # install_dev: sda  # where to install the operating system (sda is the default)
pullSecret: 'PULL_SECRET'
sshKey: |
   SSH_PUB_KEY)
pullSecret: 'PULL_SECRET'  <- Do not change anything here as this is automatically pulled from installer node
sshKey: |
   SSH_PUB_KEY             <- Do not change anything here as this is automatically pulled from installer node

site-config.yaml: https://github.com/akraino-edge-stack/kni-blueprint-pae/blob/master/sites/community.baremetal.edge-sites.net/00_install-config/site-config.yaml

...

./knictl fetch_requirements https://github.com/akraino-edge-stack/kni-blueprint-pae/tree/master/sites/community.baremetal.edge-sites.net

Prepare manifests

Run the prepare manifests command, using as a parameter the name of your site

...

It may happen that the monitor of this process stops at 93%-94%. This is fine, you can just launch again, or simply start using the cluster, as mostly all operators will come online over the timestops at 93%-94%. This is fine, you can just launch again, or simply start using the cluster, as mostly all operators will come online over the time. Follow

https://docs.openshift.com/container-platform/4.1/installing/installing_bare_metal/installing-bare-metal.html#installation-registry-storage-config_installing-bare-metal to fix image registry operator.

Prepare to deploy CentOS nodes

The default installation is totally automated for RHCOS. However, there is the possibility to deploy CentOS nodes, but this requires some specific preparation steps:

Download DVD iso from http://isoredirect.centos.org/centos/7/isos/x86_64/CentOS-7-x86_64-DVD-1908.iso , place it on /tmp

Mount it:

Code Block

language	bash

mount -o loop /tmp/CentOS-7-x86_64-DVD-18101908.iso /mnt/
mkdir -p $HOME/.kni/$SITE_NAME/baremetal_automation/matchbox-data/var/lib/matchbox/assets/centos7
cp -ar /mnt/. $HOME/.kni/$SITE_NAME/baremetal_automation/matchbox-data/var/lib/matchbox/assets/centos7/
umount /mnt

Prepare a $HOME/settings_upi.env file with the following parameters:

Code Block

language	bash

export CLUSTER_NAME="$CLUSTER_NAME"
export BASE_DOMAIN="$CLUSTER_DOMAIN"
export PULL_SECRET='your_pull_secret'
export KUBECONFIG_PATH=$HOME/.kni/$SITE_NAME/baremetal_automation/ocp/auth/kubeconfig
export OS_INSTALL_ENDPOINT=http://172.22.0.1<Installer node provisioning IP>:8080/assets/centos7
export ROOT_PASSWORD="pick_something"

Navigate to the kickstart script generation and execute it, copying the generated kickstart file:

Code Block

language	bash

cd $HOME/.kni/$SITE_NAME/baremetal_automation/kickstart/
bash add_kickstart_for_centos.sh
cp centos-worker-kickstart.cfg $HOME/.kni/$SITE_NAME/baremetal_automation/matchbox-data/var/lib/matchbox/assets/

After that, you are ready to deploy your CentOS workers with the usual procedure.

...

Code Block

language	bash

./knictl apply_workloads $SITE_NAME --kubeconfig /root$HOME/.kni/test.kni.home$SITE_NAME/baremetal_automation/ocp/auth/kubeconfig

...

Versions Compared

Old Version 33

New Version 34

Key

Overview

Secrets

Pre-Requisites for Deploying to AWS

Pre-Requisites for Deploying to Bare Metal

Pre-Requisites for Deploying to Libvirt

Structure of a site

00_install-config

01_cluster_mods

02_cluster_addons and 03_services

How to deploy

Pre-Requisites for Deploying to AWS

Pre-Requisites for Deploying to Bare Metal

Pre-Requisites for Deploying to Libvirt

Create site for AWS and libvirt

00_install-config

01_cluster_mods

02_cluster_addons and 03_services

How to deploy on AWS and libvirt

CLI tool
Anchor
knictl
knictl

Secrets

1. Fetch requirements for a site.

How to Deploy on

Baremetal

Minimal hardware footprint needed

High level steps

Create

site for Baremetal

ha-lab-ipmi-creds.yaml:

site-config.yaml: https://github.com/akraino-edge-stack/kni-blueprint-pae/blob/master/sites/community.baremetal.edge-sites.net/00_install-config/site-config.yaml

Prepare manifests

Page Comparison

Versions Compared

Old Version 33

New Version 34

Key

Secrets

Pre-Requisites for Deploying to AWS

Pre-Requisites for Deploying to Bare Metal

Pre-Requisites for Deploying to Libvirt

Structure of a site

00_install-config

01_cluster_mods

02_cluster_addons and 03_services

How to deploy

Pre-Requisites for Deploying to AWS

Pre-Requisites for Deploying to Bare Metal

Pre-Requisites for Deploying to Libvirt

Create site for AWS and libvirt

00_install-config

01_cluster_mods

02_cluster_addons and 03_services

How to deploy on AWS and libvirt

CLI tool Anchorknictlknictl

Secrets

1. Fetch requirements for a site.

How to Deploy on

Baremetal

Minimal hardware footprint needed

High level steps

Create

site for Baremetal

ha-lab-ipmi-creds.yaml:

site-config.yaml: https://github.com/akraino-edge-stack/kni-blueprint-pae/blob/master/sites/community.baremetal.edge-sites.net/00_install-config/site-config.yaml

Prepare manifests

CLI tool
Anchor
knictl
knictl