Table of Contents

Introduction

<Details about Additional tests required for this Blue Print in addition to the Akraino Validation Feature Project>

Akarino Test Group Information

<The Testing Ecosystem>

Testing Working Group Resources

Overall Test Architecture

Describe  the components of Test set up

Test Bed

Test Framework

Traffic Generator

Test API description

<Akraino common tests>

The Test inputs

Test Procedure

Expected output

Test Results

<Blueprint extension tests>

The Test inputs

Test Procedure

Expected output

Test Results

<Feature Project Tests>

The Test inputs

Test Procedure

Expected output

Test Results

Test Dashboards

Single pane view of how the test score looks like for the Blue print.

...

Total Tests

...

Test Executed

...

Pass

...

Fail

...

In Progress

Additional Testing

...

Running Unit tests

A script to run all the unittests has been provided in ./python/federatedml/test folder.

Once FATE is installed, tests can be run using:

sh ./python/federatedml/test/run_test.sh

All the unit tests shall pass if FATE is installed properly.

Pipeline Examples

Introduction

We provide some example scripts of running FATE jobs with FATE-Pipeline.

Please refer to the document linked above for details on FATE-Pipeline and FATE-Flow CLI v2. DSL version of provided Pipeline examples can be found here.

Quick Start

Here is a general guide to quick start a FATE job.

1. (optional) create virtual env

   python -m venv venv
   source venv/bin/activate
   pip install -U pip
   
   

2. install fate_client

   # this step installs FATE-Pipeline, FATE-Flow CLI v2, and FATE-Flow SDK
   pip install fate_client
   pipeline init --help

3. configure server information

   # configure by conf file
   pipeline init -c pipeline/config.yaml
   # alternatively, input real ip address and port info to initialize pipeline
   # optionally, set log directory for Pipeline
   pipeline init --ip 127.0.0.1 --port 9380 --log-directory ./logs

4. upload data with FATE-Pipeline

   #  upload demo data to FATE data storage, optionally provide path to where deployed examples/data locates
   
   python demo/pipeline-upload.py --base /data/projects/fate

   If upload job is invoked correctly, job id will be printed to terminal and an upload bar is shown. If FATE-Board is available, job progress can be monitored on Board as well.

   UPLOADING:||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||100.00%
   2020-11-02 15:37:01.030 | INFO     | pipeline.utils.invoker.job_submitter:monitor_job_status:121 - Job id is 2020110215370091210977
   Job is still waiting, time elapse: 0:00:01
   Running component upload_0, time elapse: 0:00:09
   2020-11-02 15:37:13.410 | INFO     | pipeline.utils.invoker.job_submitter:monitor_job_status:129 - Job is success!!! Job id is 2020110215370091210977
   

1. run a FATE-Pipeline fit job

   python demo/pipeline-quick-demo.py

   This quick demo shows how to build to a heterogeneous SecureBoost job. Progress of job execution will be printed as modules run. A message indicating final status ("success") will be printed when job finishes. The script queries final model information when model training completes.

   2020-11-02 10:45:29.875 | INFO     | pipeline.utils.invoker.job_submitter:monitor_job_status:121 - Job id is 2020110210452959882932
   Job is still waiting, time elapse: 0:00:01
   Running component reader_0, time elapse: 0:00:07
   Running component dataio_0, time elapse: 0:00:10
   Running component intersection_0, time elapse: 0:00:14
   Running component hetero_secureboost_0, time elapse: 0:00:46
   Running component evaluation_0, time elapse: 0:00:50
   2020-11-02 10:46:21.889 | INFO     | pipeline.utils.invoker.job_submitter:monitor_job_status:129 - Job is success!!! Job id is 2020110210452959882932
   2020-11-02 10:46:21.890 | INFO     | pipeline.utils.invoker.job_submitter:monitor_job_status:130 - Total time: 0:00:52
   

2. (another example) run FATE-Pipeline fit and predict jobs

   python demo/pipeline-mini-demo.py

   This script trains a heterogeneous logistic regression model and then runs prediction with the trained model.

   2020-11-02 15:40:43.907 | INFO     | pipeline.utils.invoker.job_submitter:monitor_job_status:121 - Job id is 2020110215404362914679
   Job is still waiting, time elapse: 0:00:01
   Running component reader_0, time elapse: 0:00:08
   Running component dataio_0, time elapse: 0:00:10
   Running component intersection_0, time elapse: 0:00:15
   Running component hetero_lr_0, time elapse: 0:00:42
   2020-11-02 15:41:27.622 | INFO     | pipeline.utils.invoker.job_submitter:monitor_job_status:129 - Job is success!!! Job id is 2020110215404362914679
   2020-11-02 15:41:27.622 | INFO     | pipeline.utils.invoker.job_submitter:monitor_job_status:130 - Total time: 0:00:43
   

   Once fit job completes, demo script will print coefficients and training information of model.

   After having completed the fit job, script will invoke a predict job with the trained model. Note that Evaluation component is added to the prediction workflow. For more information on using FATE-Pipeline, please refer to this guide.

   2020-11-02 15:41:28.255 | INFO     | pipeline.utils.invoker.job_submitter:monitor_job_status:121 - Job id is 2020110215412764443280
   Job is still waiting, time elapse: 0:00:02
   Running component reader_1, time elapse: 0:00:08
   Running component dataio_0, time elapse: 0:00:11
   Running component intersection_0, time elapse: 0:00:15
   Running component hetero_lr_0, time elapse: 0:00:20
   Running component evaluation_0, time elapse: 0:00:25
   2020-11-02 15:41:54.605 | INFO     | pipeline.utils.invoker.job_submitter:monitor_job_status:129 - Job is success!!! Job id is 2020110215412764443280
   2020-11-02 15:41:54.605 | INFO     | pipeline.utils.invoker.job_submitter:monitor_job_status:130 - Total time: 0:00:26