ELIOT Deployments in Virtual Environments Overview
This document provides a general approach to set up ELIOT ecosystem in virtual environments. The virtual environment taken are X86_64 , ARM64 and ARM32 servers.
X86 Servers with CentOS - 7 and ARM - 64 with Ubuntu 17.10 versions are chosen for setup.
ELIOT support Kubernetes and kubeedge orchestrator based two deployment model.
ELIOT Kubernetes based setup:
ELIOT Deployment in X86_64/AMD64 Server with CentOS - 7.5 version.
Pre-Installation steps to be executed on ELIOT Manager and ELIOT Node
Disable SELinux:
| Panel |
|---|
$ setenforce 0 $ sed -i --follow-symlinks 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/sysconfig/selinux |
Disable swap:
| Panel |
|---|
$ swapoff -a Note : To make sure on reboot of server the swap isn't re-enabled, comment the swap line UUID in /etc/fstab file. $ vi /etc/fstab # /dev/mapper/centos-swap swap swap defaults 0 0 |
Enable br_netfilter:
| Panel |
|---|
| $ modprobe br_netfilter $ echo '1' > /proc/sys/net/bridge/bridge-nf-call-iptables |
Install Docker:
Set up the repository
Install the required packages.
| Panel |
|---|
| $ sudo yum install -y yum-utils \ device-mapper-persistent-data \ lvm2 |
Set up stable repository
| Panel |
|---|
| $ sudo yum-config-manager \ -add-repo \ https://download.docker.com/linux/centos/docker-ce.repo |
Install Docker CE
To install latest execute step a or else if specific version has to be installed execute step b.
a) Install the latest version of Docker CE and containerd
| Panel |
|---|
| $ sudo yum install docker-ce docker-ce-cli containerd.io |
b) List and sort specific available in your repo.
| Panel |
|---|
| $ yum install docker-ce --showduplicates | sort -r $ sudo yum install docker-ce-<VERSION_STRING> docker-ce-cli-<VERSION_STRING> containerd.io Example : $ yum list docker-ce --showduplicates | sort -r docker-ce.x86_64 3:18.09.1-3.el7 docker-ce-stable docker-ce.x86_64 3:18.09.0-3.el7 docker-ce-stable docker-ce.x86_64 18.06.1.ce-3.el7 docker-ce-stable $ sudo yum install docker-ce-18.09.1 docker-ce-cli-18.09.1 containerd.io |
Start Docker
| Panel |
|---|
| $ sudo systemctl start docker |
Install Kubernetes
Add kubernetes repository in Cent OS system
| Panel |
|---|
| $ cat <<EOF > /etc/yum.repos.d/kubernetes.repo [kubernetes] name=Kubernetes baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64 enabled=1 gpgcheck=1 repo_gpgcheck=1 gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg EOF |
Install kubernetes packages kubeadm, kubelet and kubectl.
| Panel |
|---|
$ yum install -y kubelet kubeadm kubectl |
After installation is complete restart the ELIOT Manager and ELIOT Node
| Panel |
|---|
| $ sudo reboot |
Login to the servers and start the services, docker and kubelet
| Panel |
|---|
| $ systemctl start docker && systemctl enable docker $ systemctl start kubelet && systemctl enable kubelet |
Change the cgroup-driver (Need to make sure docker-ce and kubernetes are using same cgroup)
Check docker cgroup
| Panel |
|---|
| $ docker info | grep -i cgroup |
It will display docker is using 'cgroupfs' as a cgroup-driver
Run the below command to change the kubernetes cgroup-driver to 'cgroupfs' and Reload the systemd system and restart the kubelet service
| Panel |
|---|
| $ sed -i 's/cgroup-driver=systemd/cgroup-driver=cgroupfs/g' /etc/systemd/system/kubelet.service.d/10-kubeadm.conf $ systemctl daemon-reload $ systemctl restart kubelet |
Kubernetes Cluster Initialization
Login to ELIOT Manager Server ; Initialize the Kubernetes Master
| Panel |
|---|
| $ kubeadm init --apiserver-advertise-address=<ELIOT Manager Server IP Address> --pod-network-cidr=10.244.0.0/16 |
| Note |
|---|
Note : --apiserver-advertise-address = determines which IP Address Kubernetes should advertise its API server on. |
To start using your cluster, you need to run (as a regular user)
| Panel |
|---|
| $ mkdir -p $HOME/.kube $ sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config $ sudo chown $(id -u):$(id -g) $HOME/.kube/config or (for root user) $ export KUBECONFIG=/etc/kubernetes/admin.conf |
Adding ELIOT Node to the Cluster
Execute the command in the ELIOT Node
| Panel |
|---|
| $ kubeadm join --token <token> <master-ip>:6443 --discovery-token-ca-cert-hash sha256:<hash> [The Kuberadm join command string is displayed after successfull installation of Kubernetes Master (ELIOT Manager)] |
Deploy Flannel network in ELIOT Manager (Kubernetes Cluster)
| Panel |
|---|
| $ kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/a70459be0084506e4ec919aa1c114638878db11b/Documentation/kube-flannel.yml |
After join command is executed successfully , you could check the cluster node and pods
| Panel |
|---|
| $ kubectl get nodes $ kubectl get pods --all-namespaces |
| Note |
|---|
| Note : You will get the 'k8s-master' node is running as a 'master' cluster with status 'ready', and you will get all pods that are needed for the cluster, including the 'kube-flannel-ds' for network pod configuration Make sure all kube-system pod status is running. |
ELIOT KubeEdge based setup:
Eliot support Light weight orchestrator KubeEdge.
For more details refer
https://wiki.akraino.org/display/AK/KubeEdge
Kube Edge Building from source
| Panel |
|---|
|
Binary keadm is available in current path
Installing KubeEdge Master Node (on the Cloud) component
| Panel |
|---|
keadm init --docker-version=<expected version> --kubernetes-version=<expected version> --kubeedge-version=<expected version> Flags:
--docker-version string[="18.06.0"] Use this key to download and use the required Docker version (default "18.06.0")
-h, --help help for init
--kubeedge-version string[="0.3.0-beta.0"] Use this key to download and use the required KubeEdge version (default "0.3.0-beta.0")
--kubernetes-version string[="1.14.1"] Use this key to download and use the required Kubernetes version (default "1.14.1")
|
Installing KubeEdge Worker Node (at the Edge) component
| Panel |
|---|
keadm join --edgecontrollerip=<ip address> --edgenodeid=<unique string as edge identifier> Flags:
--docker-version string[="18.06.0"] Use this key to download and use the required Docker version (default "18.06.0")
-e, --edgecontrollerip string IP address of KubeEdge edgecontroller
-i, --edgenodeid string KubeEdge Node unique identification string, If flag not used then the command will generate a unique id on its own
-h, --help help for join
-k, --k8sserverip string IP:Port address of K8S API-Server
--kubeedge-version string[="0.3.0-beta.0"] Use this key to download and use the required KubeEdge version (default "0.3.0-beta.0") |
Reset KubeEdge Master and Worker nodes
| Panel |
|---|
Flags: -h, --help help for reset -k, --k8sserverip string IP:Port address of cloud components host/VM |
For more details, please follow below link:
https://kubeedge.readthedocs.io/en/latest/setup/installer_setup.html
ELIOT VM setup on ARM64
This section provides steps about setting the Virtual Machine with Ubuntu OS on ARM64 servers
Environment Details :
- ARM 64 servers with Host OS as Ubuntu 17.10 version.
- QEMU-KVM 2.10
Set up VM on ARM64 servers
Install QEMU-KVM 2.10
(Qemu-kvm is used for creating vms.)
| Panel |
|---|
| $ sudo apt-get install qemu-kvm libvirt-bin |
After qemu-kvm 2.10 is installed successful, install virtinst, followed by qemu-system-arm and qemu-efi
| Panel |
|---|
| $ sudo apt-get install virtinst |
Install QEMU and EFI image for QEMU
| Panel |
|---|
| $ sudo apt-get install qemu-system-arm qemu-efi |
After successfull instalation of qemu-efi, now need to create pflash volumes for UEFI. Two volumes are required, one static one for the UEFI firmware, and another dynamic one to store variables. Both need to be exactly 64M in size
| Panel |
|---|
$ dd if=/dev/zero of=flash0.img bs=1M count=64 |
Set Search Permissions for libvirt:
Change contents of qemu.conf found in below path /etc/libvirt/qemu.conf
uncomment user = "root" and group = "root"
Restart libvirtd
| Panel |
|---|
| $ service libvirtd restart |
| Note |
|---|
| Note: if you closed the terminal and reopened you need to repeat the previous step once. |
Start QEMU VMs with LIBVIRT:
Download the below image from the internet by executing the below command
| Panel |
|---|
| $ wget http://cdimage.ubuntu.com/releases/16.04/release/ubuntu-16.04.4-server-arm64.iso?_ga=2.265907727.1041693553.1521178200-1580291815.1512480357 |
Rename the ISO file to ubuntu-16.04.4-server-arm64.iso.
| Panel |
|---|
| $ mv ubuntu-16.04.4-server-arm64.iso?_ga=2.265907727.1041693553.1521178200-1580291815.1512480357 ubuntu-16.04.4-server-arm64.iso |
Execute the below command for VM creation:
| Panel |
|---|
$ sudo virt-install --name <VM Name> --ram 8192 --disk path=dut1.img,size=<Storage Space> --vcpus <Number of CPU's> --os-type linux --os-variant generic --cdrom './ubuntu-16.04.4-server-arm64.iso' --network default |
| Note |
|---|
| Note: In the above command VM name is dut1 , memory = 50 GB and CPUS = 4 After --name attribute will be your VM name. After --ram will be your ram size After --vcpus attribute will be your CPU?s for your VM. After --cdrom attribute within the single quotes will b path for .iso file. (The above step is also suitable with disk image also) Repeat the above step (virt-install) only if you need more than one VM. After creating the VM you can see your IP of your VM by executing command ifconfig. If you unexpectedly closes the terminal before knowing your IP address of VM , execute arp -n in terminal. It will give all the IP addresses of your main machine along with all the VM's you created. |