Introduction
The document provides guidance to install the ote-stack platform on AI Edge Blueprint, which covers the information of hardware requirement, prerequisite software and the detailed step. The ote-stack platform can be installed on Kubernetes or on a system with Docker support. The IEC cluster, which is based on Kubernetes, is chosen to be the edge infrastructure for AI Edge. The installation guide of IEC R2 can be got in here.
The guide shows step by step to deploy ote-stack platform with one edge cluster as well as add a new edge cluster to management.
License
Apache 2.0 license
How to use this document
The guide covers the details of pre-installation requirements, installation steps and uninstallation guide. The installation steps should be executed before the prerequisites and pre-installation requirements are ready. In addition, one IEC edge cluster with 3 nodes needs to be prepared in advance.
Deployment Architecture
The Deployment Architecture uses one IEC edge cluster with 3 nodes as the basic infrastructure and the ote-stack platform will installed on the cluster.
Pre-Installation Requirements
Hardware Requirements
This section describes the requirements for building IEC edge infrastructure and ote-stack platform. At least three edge nodes are needed to build the edge infrastructure.
Minimum Hardware Requirements
HW Aspect | Requirement |
---|---|
1 Jumpserver | A physical or virtualized machine that has direct network connectivity to the edge nodes. |
CPU | Minimum 1 socket for IEC and 2 socket for ote-stack components and usecase test (each cluster node) |
RAM | Minimum 4GB/server (Depending on usecase work load) |
Disk | Minimum 20GB (each cluster node) |
Networks | Mininum 1 |
Recommended Hardware Requirements
3 Huawei TaiShan 2280 Arm Server are required:
Processor | 2*Kunpeng 920 processor |
RAM | 12*32G-DDR4-2666MT/s |
Storage | 3*3.5 inch 4000G SATA and 1*2.5 inch 3200G NVMe SSD |
Network | 1 onboard network card, each card supports 4*GE port or 4*10GE port or 4*25GE port |
Refer TaiShan 2280 Server User Guide here
Software Prerequisites
- Available harbor (>=1.7, <2.0) with admin user and password
- At least one IEC edge cluster with healthy container network.
Database Prerequisites
Schema scripts (option)
It is not necessary to execute the below scripts. MySQL database will be deployed on the root edge cluster after executing the installation scripts. If there is an existing MySQL database, you can perform the below scripts so as to initialize the database for ote-stack platform.
CREATE DATABASE sys_ote_manage_platform; USE sys_ote_manage_platform; SET names utf8; CREATE TABLE IF NOT EXISTS `tb_system_config` ( `key` varchar(64) NOT NULL, `value` varchar(64) NOT NULL, PRIMARY KEY (`key`) ) ENGINE=InnoDB DEFAULT CHARSET=utf8 COMMENT='system configuration'; CREATE TABLE IF NOT EXISTS `tb_business_info` ( `id` bigint(20) unsigned NOT NULL AUTO_INCREMENT, `name` varchar(64) NOT NULL COMMENT 'business name', `namespace` varchar(64) NOT NULL DEFAULT '' COMMENT 'namespace', `user_id` bigint(20) unsigned NOT NULL COMMENT 'user id', `introduce` varchar(512) NOT NULL COMMENT 'business introduction', `objective` varchar(512) NOT NULL COMMENT 'resource requirements', `scale` varchar(512) NOT NULL COMMENT 'business scale', `comment` varchar(512) NOT NULL DEFAULT '' COMMENT 'review comments', `status` tinyint NOT NULL DEFAULT 0 COMMENT 'status', `update_time` TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP, `create_time` TIMESTAMP NOT NULL, PRIMARY KEY (`id`), UNIQUE KEY (`name`) ) ENGINE=InnoDB DEFAULT CHARSET=utf8; CREATE TABLE IF NOT EXISTS `tb_ote_web_users` ( `id` bigint(20) unsigned NOT NULL AUTO_INCREMENT, `uid` varchar(50) COLLATE utf8_unicode_ci NOT NULL COMMENT 'user id', `namespace` varchar(64) COLLATE utf8_unicode_ci DEFAULT '' COMMENT 'namespace', `user_name` varchar(50) COLLATE utf8_unicode_ci DEFAULT '' COMMENT 'username', `display_name` varchar(50) COLLATE utf8_unicode_ci DEFAULT '' COMMENT 'nickname', `real_name` varchar(50) COLLATE utf8_unicode_ci NOT NULL DEFAULT '' COMMENT 'real name', `password` varchar(60) COLLATE utf8_unicode_ci NOT NULL DEFAULT '' COMMENT 'password', `email` varchar(255) COLLATE utf8_unicode_ci NOT NULL DEFAULT '' COMMENT 'email', `phone` varchar(11) COLLATE utf8_unicode_ci NOT NULL DEFAULT '' COMMENT 'phone', `status` tinyint(4) unsigned NOT NULL DEFAULT '3' COMMENT 'status: 0:pending,1:reviewed,2:failed,3:forbidden,', `is_admin` tinyint(3) unsigned NOT NULL DEFAULT '0' COMMENT 'is admin', `role` tinyint unsigned NOT NULL DEFAULT '0' COMMENT 'role:0:unauthorized,1:general manager,2:admin,3:super admin', `updated_at` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP, `created_at` timestamp NOT NULL, PRIMARY KEY (`id`), UNIQUE KEY `tb_ote_web_users_uid_unique` (`uid`) USING BTREE, UNIQUE KEY `tb_ote_web_users_phone_unique` (`phone`) USING BTREE, UNIQUE KEY `tb_ote_web_users_user_name_unique` (`user_name`) USING BTREE ) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci; CREATE TABLE IF NOT EXISTS `tb_ote_web_repository_users` ( `id` bigint(20) unsigned NOT NULL AUTO_INCREMENT, `namespace` varchar(64) NOT NULL DEFAULT '' COMMENT 'namespace', `repository_uid` bigint(20) unsigned DEFAULT '0' COMMENT 'uid', `repository_username` varchar(50) COLLATE utf8_unicode_ci NOT NULL COMMENT 'username', `repository_email` varchar(255) COLLATE utf8_unicode_ci NOT NULL DEFAULT '' COMMENT 'email', `repository_password` varchar(255) COLLATE utf8_unicode_ci NOT NULL DEFAULT '' COMMENT 'password', `updated_at` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP, `created_at` timestamp NOT NULL, PRIMARY KEY (`id`), UNIQUE KEY (`namespace`) USING BTREE ) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci COMMENT='harbor user'; CREATE TABLE IF NOT EXISTS `tb_ote_web_third_repository` ( `id` bigint(20) unsigned NOT NULL AUTO_INCREMENT, `namespace` varchar(64) COLLATE utf8_unicode_ci NOT NULL DEFAULT '' COMMENT 'user namespace', `repository_id` varchar(50) COLLATE utf8_unicode_ci NOT NULL DEFAULT '' COMMENT 'repository id', `repository_url` varchar(500) COLLATE utf8_unicode_ci NOT NULL DEFAULT '' COMMENT 'repository addr', `repository_username` varchar(50) COLLATE utf8_unicode_ci NOT NULL DEFAULT '' COMMENT 'username, `repository_password` varchar(255) COLLATE utf8_unicode_ci NOT NULL COMMENT 'passwoed', `updated_at` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP, `created_at` timestamp NOT NULL, PRIMARY KEY (`id`), UNIQUE KEY (`namespace`, `repository_id`) USING BTREE ) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci COMMENT='third repo'; CREATE TABLE IF NOT EXISTS `tb_ote_cluster_label` ( `id` bigint(20) unsigned NOT NULL AUTO_INCREMENT, `cluster_name` varchar(64) COLLATE utf8_unicode_ci NOT NULL DEFAULT '' COMMENT 'cluster name', `label` varchar(64) COLLATE utf8_unicode_ci NOT NULL DEFAULT '' COMMENT 'label', `updated_at` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP, `created_at` timestamp NOT NULL, PRIMARY KEY (`id`), UNIQUE KEY (`cluster_name`, `label`) USING BTREE ) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci; CREATE TABLE IF NOT EXISTS `tb_ote_node_label` ( `id` bigint(20) unsigned NOT NULL AUTO_INCREMENT, `cluster_name` varchar(64) COLLATE utf8_unicode_ci NOT NULL DEFAULT '' COMMENT 'cluste', `node_name` varchar(64) COLLATE utf8_unicode_ci NOT NULL DEFAULT '' COMMENT 'nodename', `label` varchar(64) COLLATE utf8_unicode_ci NOT NULL DEFAULT '' COMMENT 'node label', `updated_at` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP, `created_at` timestamp NOT NULL, PRIMARY KEY (`id`), UNIQUE KEY (`cluster_name`, `node_name`, `label`) USING BTREE ) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci; CREATE TABLE IF NOT EXISTS `tb_app_info` ( `id` bigint(20) unsigned NOT NULL AUTO_INCREMENT, `namespace` varchar(64) NOT NULL COMMENT 'namespace', `app_name` varchar(64) NOT NULL COMMENT 'app_name', `main_version` varchar(32) NOT NULL COMMENT 'major version', `version_count` int(11) NOT NULL COMMENT 'minor version', `image` varchar(256) NOT NULL COMMENT 'image', `repository_id` varchar(64) NOT NULL DEFAULT '' COMMENT 'repository_id', `port` varchar(1024) NOT NULL DEFAULT '0' COMMENT 'port', `env` varchar(1024) NOT NULL DEFAULT '' COMMENT 'env', `volume` varchar(1024) NOT NULL DEFAULT '' COMMENT 'volume', `dependence` varchar(1024) NOT NULL DEFAULT '' COMMENT '', `replicas` int(10) NOT NULL DEFAULT '1' COMMENT 'replicas', `command` varchar(512) NOT NULL DEFAULT '' COMMENT 'command', `deploy_type` tinyint(4) NOT NULL DEFAULT '0' COMMENT 'deploy_type: 0:deployment, 1:daemonset', `gpu` int(10) NOT NULL DEFAULT '0', `request_cpu` int(10) NOT NULL DEFAULT '80', `request_mem` int(10) NOT NULL DEFAULT '4096', `limit_cpu` int(10) NOT NULL DEFAULT '80', `limit_mem` int(10) NOT NULL DEFAULT '4096', `min_replicas` int(10) NOT NULL DEFAULT '1', `max_replicas` int(10) NOT NULL DEFAULT '5', `is_hpa` tinyint(4) NOT NULL DEFAULT '0' COMMENT 'is_hpa:0:no,1:yes', `hpa_target_cpu` int(10) NOT NULL DEFAULT '80', `hpa_target_mem` int(10) NOT NULL DEFAULT '4096', `min_ready_seconds` int(10) NOT NULL DEFAULT '10', `max_surge` int(10) NOT NULL DEFAULT '1', `max_unavailable` int(10) NOT NULL DEFAULT '0', `status` tinyint(4) NOT NULL DEFAULT '0' COMMENT 'statur:0:init,1:success,2:fail,3:logical del,4:del', `update_time` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP COMMENT , `create_time` timestamp NOT NULL, PRIMARY KEY (`id`), UNIQUE KEY `namespace` (`namespace`,`app_name`,`main_version`,`version_count`) ) ENGINE=InnoDB DEFAULT CHARSET=utf8; CREATE TABLE IF NOT EXISTS `tb_deploy_info` ( `id` bigint(20) unsigned NOT NULL AUTO_INCREMENT, `name` varchar(128) NOT NULL COMMENT 'name', `namespace` varchar(64) NOT NULL COMMENT 'namespace', `app_name` varchar(64) NOT NULL COMMENT 'app_name', `version` varchar(48) NOT NULL COMMENT 'version', `cluster` varchar(64) NOT NULL COMMENT 'cluster', `node_label` varchar(64) NOT NULL DEFAULT 'all', `status` tinyint(4) NOT NULL DEFAULT '0' COMMENT 'status', `editable` tinyint(4) NOT NULL DEFAULT '1' COMMENT 'editable: 0:no,1:yes', `running` tinyint(4) NOT NULL DEFAULT '0' COMMENT 'running: 0:no,1:yes', `deploy_type` tinyint(4) NOT NULL DEFAULT '0' COMMENT 'deploy_type: 0:new,1:upgrade,2:rollback,3:del', `comment` varchar(256) NOT NULL DEFAULT '' COMMENT 'comment', `audit_comment` varchar(256) NOT NULL DEFAULT '' COMMENT 'audit_comment', `error_message` varchar(256) NOT NULL DEFAULT '' COMMENT 'error_message', `update_time` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP, `create_time` timestamp NOT NULL, `execute_time` timestamp NOT NULL, PRIMARY KEY (`id`), KEY `name` (`namespace`,`name`), KEY `namespace` (`namespace`,`app_name`,`cluster`,`version`) ) ENGINE=InnoDB DEFAULT CHARSET=utf8; CREATE TABLE IF NOT EXISTS tb_domain_info ( `id` BIGINT UNSIGNED NOT NULL AUTO_INCREMENT, `namespace` varchar(64) NOT NULL COMMENT 'namespace', `domain` varchar(128) NOT NULL COMMENT 'domain', `used_count` int NOT NULL DEFAULT 0 COMMENT 'used_count', `update_time` TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP, `create_time` TIMESTAMP NOT NULL, PRIMARY KEY(`id`), KEY(`namespace`), UNIQUE KEY(`domain`) ) ENGINE = InnoDB DEFAULT CHARACTER SET = utf8; CREATE TABLE IF NOT EXISTS tb_ingress_info ( `id` BIGINT UNSIGNED NOT NULL AUTO_INCREMENT, `namespace` varchar(64) NOT NULL COMMENT 'namespace', `domain` varchar(128) NOT NULL COMMENT 'domain', `uri` varchar(128) NOT NULL COMMENT 'uri', `is_rewrite` int8 NOT NULL DEFAULT 0 COMMENT '', `deploy_name` varchar(64) NOT NULL COMMENT 'deploy name', `status` tinyint(4) NOT NULL DEFAULT '0' COMMENT 'status', `unique_key` varchar(64) NOT NULL DEFAULT 'unique key', `update_time` TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP, `create_time` TIMESTAMP NOT NULL, PRIMARY KEY(`id`), UNIQUE KEY(`namespace`,`unique_key`) ) ENGINE = InnoDB DEFAULT CHARACTER SET = utf8; CREATE TABLE IF NOT EXISTS `tb_halo_service` ( `id` bigint(20) unsigned NOT NULL AUTO_INCREMENT, `service_name` varchar(64) NOT NULL COMMENT 'service_name', `package` varchar(64) NOT NULL COMMENT 'package', `update_time` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP, `create_time` timestamp NOT NULL, PRIMARY KEY (`id`), UNIQUE KEY `service_name` (`service_name`) ) ENGINE=InnoDB DEFAULT CHARSET=utf8; CREATE TABLE IF NOT EXISTS tb_halo_package ( id BIGINT UNSIGNED NOT NULL AUTO_INCREMENT, package_name varchar(64) NOT NULL COMMENT 'package_name', reserve_num int NOT NULL DEFAULT 2 COMMENT 'reserve_num', source_type varchar(16) NOT NULL COMMENT 'source_type', update_time TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP, create_time TIMESTAMP NOT NULL, PRIMARY KEY(`id`), UNIQUE KEY(`package_name`) ) ENGINE = InnoDB DEFAULT CHARACTER SET = utf8; CREATE TABLE IF NOT EXISTS tb_halo_command ( id BIGINT UNSIGNED NOT NULL AUTO_INCREMENT, name varchar(64) NOT NULL COMMENT 'jobName', job_id varchar(50) NOT NULL COMMENT 'jobID', command_info text NOT NULL COMMENT 'command_info with base64', create_time TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP, PRIMARY KEY(`id`), UNIQUE KEY `name` (`name`) ) ENGINE = InnoDB DEFAULT CHARACTER SET = utf8; CREATE TABLE IF NOT EXISTS `tb_alert_info` ( `id` bigint(20) unsigned NOT NULL AUTO_INCREMENT, `instance` varchar(100) COLLATE utf8_unicode_ci NOT NULL DEFAULT '' COMMENT 'alert host', `ipaddr` varchar(20) COLLATE utf8_unicode_ci NOT NULL DEFAULT '' COMMENT 'alert ip', `alert_name` varchar(100) COLLATE utf8_unicode_ci NOT NULL DEFAULT '' COMMENT 'alert name', `alert_user` varchar(50) COLLATE utf8_unicode_ci NOT NULL DEFAULT '' COMMENT 'alert user', `limit_value` varchar(10) COLLATE utf8_unicode_ci NOT NULL DEFAULT '' COMMENT 'alert limit value', `current_value` varchar(20) COLLATE utf8_unicode_ci NOT NULL DEFAULT '' COMMENT 'current alert value', `description` varchar(1000) COLLATE utf8_unicode_ci NOT NULL DEFAULT '' COMMENT 'description', `generator_url` varchar(500) COLLATE utf8_unicode_ci NOT NULL DEFAULT '' COMMENT '', `status` varchar(10) COLLATE utf8_unicode_ci NOT NULL DEFAULT '' COMMENT 'status', `update_time` TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP, `create_time` TIMESTAMP NOT NULL, PRIMARY KEY (`id`), UNIQUE KEY(`instance`, `alert_name`,`create_time`) ) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci;
Other Installation Requirements
Jump Host Requirements
Software requirements:
- git
- docker (version >= 1.13)
- golang (version >= 1.12)
- kubectl CLI
Network Requirements
- internet connectivity
Bare Metal Node Requirements
Execution Requirements (Bare Metal Only)
Installation High-Level Overview
Bare Metal Deployment Guide
Install Bare Metal Jump Host
The prerequisites software should be ready.
Creating a Node Inventory File
N/A
Creating the Settings Files
Copy the kubeconfig file of IEC cluster to directory /root/.kube/config.
Clone the AIEdge git repo and edit the configuration file (interface_conf) by setting:
- node_name in edge cluster which would deploy ote-stack
- docker repository for pulling images of ote-stack components
- informations of harbor repository such as admin password and domain
git clone https://gerrit.akraino.org/r/aiedge.git cd aiedge/deployment vim interface_conf
Running
Prerequisite
Compile cluster controller, controller manager and shim components following below scripts. Then docker push the output image to a docker repository which is set in the interface_conf file.
bash build/build.sh build_image
Other components are not source code released currently but released as docker images, and they can be pull from docker hub. Make sure that the docker repository set in the configuration file have all docker images required by ote-stack platform.
Deploy ote-stack on one IEC cluster
Simply start the installation script with default parameters in the same directory:
# create yaml file based on interface_conf sh create_yml.sh start # apply yaml file to cluster sh apply_yml.sh start
All components of ote-stack will be installed on the IEC edge. But some components can be replaced with external ones outside the cluster through changing the parameter of yaml file. For example, the open-api server can connect to an existing mysql server by modifying the parameters `mysqlHost` and `mysqlPort` in file deployments/open-api/open-api.yml and commenting out the line to deploy mysql.
After executing the installation script, the ote-stack web platform will be deployed on the node1_name set in the configuration file and expose port 8995. You can open the website through URL http://<node1_name_ip>:8995/, and the login page is as follows.
When ote-stack runs, a super administrator will be created. The username and password can be modified by changing parameters `superAdminName` and `superAdminPassword` in file deployments/open-api/open-api.yml. Note that the initial username and password of admin, only works for the first time when ote-stack starts.
Add a new cluster to ote-stack for management (option)
If there is another cluster that needs to be added to the ote-stack, some components should be deployed on the edge to connect with root cluster:
- tiller/ tiller-proxy
- ote-shim/ ote-cc
- node-agent/ nodes-server/ prometheus
- exporters like node-exporte (optional)
- fluent-bit (optional)
Virtual Deployment Guide
Standard Deployment Overview
Please refer section Bare Metal Deployment Guide for deployment, the installation can run successfully in virtual hardware too.
Snapshot Deployment Overview
N/A
Special Requirements for Virtual Deployments
Install Jump Host
N/A
Verifying the Setup - VMs
N/A
Upstream Deployment Guide
Upstream Deployment Key Features
- Provide the execution and network infrastructure for installing ote-stack and running the AI application.
Special Requirements for Upstream Deployments
Please refer the section "Pre-Installation Requirements" in wiki IEC Type1&2 Installation Guide for R2 and make sure the environment requirements met.
Scenarios and Deploy Settings for Upstream Deployments
Please prepare three arm servers and build a edge cluster according to wiki IEC Type1&2 Installation Guide for R2.
Including Upstream Patches with Deployment
N/A
Running
N/A
Interacting with Containerized Overcloud
N/A
Verifying the Setup
Verify manually:
- Login to the otestack web platform and create a simply application on the website
- Check the deployment status of application using kubectl cli on the IEC cluster
Or Run installation and verify scripts through CI/CD.
Developer Guide and Troubleshooting
Utilization of Images
Post-deployment Configuration
N/A
Debugging Failures
N/A
Reporting a Bug
N/A
Uninstall Guide
# remove all components deployed in edge cluster sh apply_yml.sh stop
MySQL data will be persisted on disk /home/work/ote/mysql-data/. You can use command `rm` to clean up the residual data.
Troubleshooting
Error Message Guide
N/A
Maintenance
Blue Print Package Maintenance
Software maintenance
N/A
Hardware maintenance
N/A
Blue Print Deployment Maintenance
N/A
Frequently Asked Questions
N/A
License
/*
Copyright 2019 Baidu, Inc.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
References
- Apache License 2.0: https://www.apache.org/licenses/LICENSE-2.0
- AI Edge Wiki: