Introduction

All test results and logs have been pushed to Akraino NEXUS:

https://nexus.akraino.org/content/sites/logs/fate/job/Fate_test/15/

Running Unit tests

A script to run all the unittests has been provided in ./python/federatedml/test folder.

Once FATE is installed, tests can be run using:

sh ./python/federatedml/test/run_test.sh

All the unit tests shall pass if FATE is installed properly.

Introduction

We provide some example scripts of running FATE jobs with FATE-Pipeline.

Please refer to the document linked above for details on FATE-Pipeline and FATE-Flow CLI v2. DSL version of provided Pipeline examples can be found here.

Quick Start

Here is a general guide to quick start a FATE job.

1. (optional) create virtual env

   python -m venv venv source venv/bin/activate pip install -U pip
   
   

2. install fate_client

   # this step installs FATE-Pipeline, FATE-Flow CLI v2, and FATE-Flow SDK pip install fate_client pipeline init --help

3. configure server information

   # configure by conf file pipeline init -c pipeline/config.yaml # alternatively, input real ip address and port info to initialize pipeline# optionally, set log directory for Pipeline pipeline init --ip 127.0.0.1 --port 9380 --log-directory ./logs

4. upload data with FATE-Pipeline

   # upload demo data to FATE data storage, optionally provide path to where deployed examples/data locates python demo/pipeline-upload.py --base /data/projects/fate

   If upload job is invoked correctly, job id will be printed to terminal and an upload bar is shown. If FATE-Board is available, job progress can be monitored on Board as well.

   UPLOADING:||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||100.00% 2020-11-02 15:37:01.030 | INFO | pipeline.utils.invoker.job_submitter:monitor_job_status:121 - Job id is 2020110215370091210977 Job is still waiting, time elapse: 0:00:01 Running component upload_0, time elapse: 0:00:09 2020-11-02 15:37:13.410 | INFO | pipeline.utils.invoker.job_submitter:monitor_job_status:129 - Job is success!!! Job id is 2020110215370091210977

1. run a FATE-Pipeline fit job

   python demo/pipeline-quick-demo.py

   This quick demo shows how to build to a heterogeneous SecureBoost job. Progress of job execution will be printed as modules run. A message indicating final status ("success") will be printed when job finishes. The script queries final model information when model training completes.

   2020-11-02 10:45:29.875 | INFO | pipeline.utils.invoker.job_submitter:monitor_job_status:121 - Job id is 2020110210452959882932 Job is still waiting, time elapse: 0:00:01 Running component reader_0, time elapse: 0:00:07 Running component dataio_0, time elapse: 0:00:10 Running component intersection_0, time elapse: 0:00:14 Running component hetero_secureboost_0, time elapse: 0:00:46 Running component evaluation_0, time elapse: 0:00:50 2020-11-02 10:46:21.889 | INFO | pipeline.utils.invoker.job_submitter:monitor_job_status:129 - Job is success!!! Job id is 2020110210452959882932 2020-11-02 10:46:21.890 | INFO | pipeline.utils.invoker.job_submitter:monitor_job_status:130 - Total time: 0:00:52

2. (another example) run FATE-Pipeline fit and predict jobs

   python demo/pipeline-mini-demo.py

   This script trains a heterogeneous logistic regression model and then runs prediction with the trained model.

   2020-11-02 15:40:43.907 | INFO | pipeline.utils.invoker.job_submitter:monitor_job_status:121 - Job id is 2020110215404362914679 Job is still waiting, time elapse: 0:00:01 Running component reader_0, time elapse: 0:00:08 Running component dataio_0, time elapse: 0:00:10 Running component intersection_0, time elapse: 0:00:15 Running component hetero_lr_0, time elapse: 0:00:42 2020-11-02 15:41:27.622 | INFO | pipeline.utils.invoker.job_submitter:monitor_job_status:129 - Job is success!!! Job id is 2020110215404362914679 2020-11-02 15:41:27.622 | INFO | pipeline.utils.invoker.job_submitter:monitor_job_status:130 - Total time: 0:00:43

   Once fit job completes, demo script will print coefficients and training information of model.

   After having completed the fit job, script will invoke a predict job with the trained model. Note that Evaluation component is added to the prediction workflow. For more information on using FATE-Pipeline, please refer to this guide.

   2020-11-02 15:41:28.255 | INFO | pipeline.utils.invoker.job_submitter:monitor_job_status:121 - Job id is 2020110215412764443280 Job is still waiting, time elapse: 0:00:02 Running component reader_1, time elapse: 0:00:08 Running component dataio_0, time elapse: 0:00:11 Running component intersection_0, time elapse: 0:00:15 Running component hetero_lr_0, time elapse: 0:00:20 Running component evaluation_0, time elapse: 0:00:25 2020-11-02 15:41:54.605 | INFO | pipeline.utils.invoker.job_submitter:monitor_job_status:129 - Job is success!!! Job id is 2020110215412764443280 2020-11-02 15:41:54.605 | INFO | pipeline.utils.invoker.job_submitter:monitor_job_status:130 - Total time: 0:00:26