Table of Contents

Introduction

FATE-Flow is the job scheduling system of the federated learning framework FATE, which realizes the complete management of the federated learning job life cycle, including data input, training job scheduling, indicator tracking, model center and other functions. It provides CLI and HTTP API for users to manage data and federated jobs.

FATE-Flow REST API

HTTP Method: POST
Content-Type: application/json

DataAccess

/v1/data/upload

request structure
- namespace: Required,String: upload data table namespace
- table_name: Required,String: upload data table name
- work_mode: Required,Integer: eggroll’s working mode
- file: Required, String: upload file location
- head: Required,Integer: determine if there is a data header
- partition: Required,Integer: set the number of partitions to save data
- module: Optional,String: If you need to use the data of the machine where the FATE-Flow server is located, this value is not empty.
- use_local_data: Optional,String: If you need to use the data of the machine where the FATE-Flow server is located, this value is 0.
- drop: Optional, Integer: When the cluster deployment uses the same table to upload data, it is necessary to carry the drop parameter,0 represents overwriting upload, 1 represents deleting the previous data and re-uploading
response structure
- job_id: upload job id,String
- data: return data for submitting job ,Object

...

request structure
- job_id: Optional,String:download job id
- limit: Optional, Integer:Limit quantity
response structure
- retcode: return code,Integer
- retmsg: return code description,String
- data: return data for submitting job ,Object

Job

/v1/job/submit

request structure
- job_runtime_conf: Required,Object: configuration information for the currently submitted job
- job_dsl: Required,Object: dsl of the currently submitted job
response structure
- job_id: job id of the currently submitted job,String
- data: return data for submitting job ,Object

...

request structure
- job_id: Optional,String: job id
- name: Optional,String: job name
- description: Optional,String: job description
- tag: Optional,String:Optional,String: job tag
- role: Optional,String: job role
- party_id: Optional,String: job party id
- roles: Optional,String: job roles
- initiator_party_id: Optional,String: initiator’s party id
- is_initiator: Optional,Integer: mark if it is the initiator
- dsl: Optional,String: job dsl
- runtime_conf : Optional,String: configuration information for the job
- run_ip: Optional,String: job run ip
- status: Optional,String: job status
- current_steps: Optional,String:record component id in DSL
- current_tasks: Optional,String: record task id
- progress: Optional,Integer: job progress
- create_time: Optional,Integer: job create time
- update_time: Optional,Integer:job update time
- start_time: Optional,Integer: job start time
- end_time: Optional,Integer: job end time
- elapsed: Optional,Integer: job elapsed time
response structure
- retcode: return code,Integer
- retmsg: return code description,String
- data: tasks data, Array

Tracking

/v1/tracking/job/data_view

...

request structure
- job_id: Required,String: job id
- role: Required,String: role information
- party_id: Required,Integer: party id
- component_name: Required,String: component name
response structure
- retcode: return code,Integer
- retmsg: return code description,String
- data: output data, Array
- meta: schema header information, Object

Pipeline

/v1/pipeline/dag/dependency

request structure
- job_id: Required,String:job id
- role: Required,String: role information
- party_id: Required,Integer: party id
response structure
- retcode: return code,Integer
- retmsg: return code description,String
- data: pipeline dag dependency data,Object

Model

/v1/model/load

request structure
- initiator: Required,Object: job initiator information, including party_id and role
- job_parameters: Required,Object: job parameters information, including work_mode, model_id and model_version
- role: Required,Object: role information of the parties
- servings: Optional,Array: fate serving address and port
response structure
- job_id:job id, String
- retcode: return code, Integer
- retmsg: return code description, String
- data: status info, Object

...

request structure
- name: Requied,String: model version
- namespace: Requied,String: model id
response structure
- retcode: return code, Integer
- retmsg: return code description, String
- data: model data, Object

Table

/v1/table/table_info

request structure
- create: Optional, Boolean: whether to create
- namespace: Optional,String: download data table namespace, need to be used with table_name
- table_name: Optional,String: download data table name, need to be used with namespace
- local: Optional,Object: local configuration
- role: Optional,Object: role information
- data_type: Optional,String: download file data type
- gen_table_info: Optional,Boolean: tag table information
response structure
- retcode: return code, Integer
- retmsg: return code description, String
- data: table information

...

request structure
- namespace: Optional,String: download data table namespace, need to be used with table_name
- table_name: Optional,String: download data table name, need to be used with namespace
response structure
- retcode: return code, Integer
- retmsg: return code description, String
- data: table information

FATE-Flow Client Command Line Interface

Usage

python fate_flow_client.py -f $command

JOB_OPERATE

submit_job

description: submit a pipeline job

parameter:

-c -config: runtime conf path, Required
-d -dsl: dsl path, Required

python fate_flow_client.py -f submit_job -c examples/test_hetero_lr_job_conf.json -d examples/test_hetero_lr_job_dsl.json

...

description: cancel job or stop job

parameter:

-j -job_id: job id, Required

python fate_flow_client.py -f stop_job -j $job_id

query_job

description: query job information by filters
parameter:
- -j -job_id: filter by job id, Optional
- -r -role : filter by role, Optional
- -p -party_id: filter by party id, Optional
- -s -status: filter by status, Optional
```
python fate_flow_client.py -f query_job -j $job_id
```

clean_job

description: clean processor,data table and metric data
parameter:
- -j -job_id: filter by job id, Optional
- -r -role : filter by role, Optional
- -p -party_id: filter by party id, Optional
- -cpn -component_name: component name, Optional
```
python fate_flow_client.py -f clean_job -j $job_id
```

data_view_query

description: query data view information by filters
parameter:j -job_id: filter by job id, Optionalr -role : filter by role, Optionalp -party_id: filter by party id, Optionals -status: filter by status, Optional

python fate_flow_client.py -f data_view_query -j $job_id

JOB

job_config

description: download the configuration of this job
parameter:
- -j -job_id: job id, Required
- -r -role : role, Required
- -p -party_id: party id, Required
- -o -output_path: config output directory path, Required
```
python fate_flow_client.py -f job_config -j $job_id -r $role -p $party_id -o $output_path
```

job_log

description: download the log of this job
parameter:
- -j -job_id: job id, Required
- -o -output_path: config output directory path, Required
```
python fate_flow_client.py -f job_log -j $job_id -o $output_path
```

TASK_OPERATE

query_task

description: query task information by filters
parameter:
- -j -job_id: filter by job id, Optional
- -cpn -component_name: filter by component name, Optional
- -r -role : filter by role, Optional
- -p -party_id: filter by party id, Optional
- -s -status: filter by status, Optional
```
python fate_flow_client.py -f query_task -j $job_id
```

TRACKING

component_parameters

description: query the parameters of this component
parameter:
- -j -job_id: job id, Required
- -cpn -component_name: component name, Required
- -r -role: role, Required
- -p -party_id: party id, Required
```
python fate_flow_client.py -f component_parameters -j $job_id -r $role -p $party_id -cpn $component_name
```

component_metric_all

description: query all metric data
parameter:
- -j –job_id: job id, Required
- -cpn –component_name: component name, Required
- -r –role: role, Required
- -p –party_id: party id, Required
```
python fate_flow_client.py -f component_metric_all -j $job_id -r $role -p $party_id -cpn $component_name
```

component_metrics

description: query the list of metrics
parameter:
- -j –job_id: job id, Required
- -cpn –component_name: component name, Required
- -r –role: role, Required
- -p –party_id: party id, Required
```
python fate_flow_client.py -f component_metrics -j $job_id -r $role -p $party_id -cpn $component_name
```

component_output_model

description: query this component model
parameter:
- -j –job_id: job id, Required
- -cpn –component_name: component name, Required
- -r –role: role, Required
- -p –party_id: party id, Required
```
python fate_flow_client.py -f component_output_model -j $job_id -r $role -p $party_id -cpn $component_name
```

component_output_data

description: download the output data of this component
parameter:
- -j –job_id: job id, Required
- -cpn –component_name: component name, Required
- -r –role: role, Required
- -p –party_id: party id, Required
- -o –output_path: config output path, Required
- -limit –limit: Limit quantity, Optional
```
python fate_flow_client.py -f component_output_data -j $job_id -r $role -p $party_id -cpn $component_name -o $output_path
```

component_output_data_table

description: view table name and namespace
parameter:
- -j –ob_id: job id, Required
- -cpn –component_name: component name, Required
- -r –role: role, Required
- -p –party_id: party id, Required
```
python fate_flow_client.py -f component_output_data_table -j $job_id -r $role -p $party_id -cpn $component_name
```

DATA

download

description: download table

parameter:

-c –config: config path, Required

python fate_flow_client.py -f download -c examples/download_guest.json

...

description: upload table

parameter:

-c –config: config path, Required
-drop –drop: Operations before file upload, Optional

python fate_flow_client.py -f upload -c examples/upload_guest.json
python fate_flow_client.py -f upload -c examples/upload_guest.json -drop 0(or1)

upload_history

description: upload table history

parameter:

-j –job_id: job id, Optional
-limit –limit: Limit quantity, Optional

python fate_flow_client.py -f upload_history -j $job_id
python fate_flow_client.py -f upload_history -limit 5

Table

table_info

description: query table information
parameter:
- -n –namespace: namespace, Required
- -t –table_name: table name, Required
```
python fate_flow_client.py -f table_info -n $namespace -t $table_name
```

table_delete

description: delete table
parameter:
- -n –namespace: namespace, Optional
- -t –table_name: table name, Optional
- -j –job_id: job id, Optional
- -cpn –component_name: component name, Optional
- -r –role: role, Optional
- -p –party_id: party id, Optional
```
python fate_flow_client.py -f table_delete -n $namespace -t $table_name
python fate_flow_client.py -f table_delete -j $job_id
```

Model

load

description: load model.

parameter:

-c –config: config path, Required

python fate_flow_client.py -f load -c $conf_path

response example:

{
    "data": {
        "detail": {
            "guest": {
                "9999": {
                    "retcode": 0,
                    "retmsg": "xxx"
                }
            },
            "host": {
                "10000": {
                    "retcode": 0,
                    "retmsg": "xxx"
                }
            }
        },
        "guest": {
            "9999": 0
        },
        "host": {
            "10000": 0
        }
    },
    "jobId": "xxxxxxxxxxxxxxxx",
    "retcode": 0,
    "retmsg": "success"
}

bind

description: bind model.

parameter:

-c –config: config path, Required

python fate_flow_client.py -f bind -c $conf_path

response example:

{
    "retcode": 0,
    "retmsg": "service id is xxx"
}

store

description: store model

parameter:

-c –config: config path, Required

python fate_flow_client.py -f store -c $conf_path

restore

description: restore mode

parameter:

-c –config: config path, Required

python fate_flow_client.py -f restore -c $conf_path

export

description: export model

parameter:

-c –config: config path, Required

python fate_flow_client.py -f export -c $conf_path

import

description: import model

parameter:

-c –config: config path, Required

python fate_flow_client.py -f import -c $conf_path

Next Previous

Versions Compared

Old Version 1

New Version Current

Key

Introduction

FATE-Flow REST API

DataAccess

Job

Tracking

Pipeline

Model

Table

FATE-Flow Client Command Line Interface

Usage

JOB_OPERATE

JOB

TASK_OPERATE

TRACKING

DATA

Table

Model

Page Comparison

Versions Compared

Old Version 1

New Version Current

Key

Introduction

FATE-Flow REST API

DataAccess

Job

Tracking

Pipeline

Model

Table

FATE-Flow Client Command Line Interface

Usage

JOB_OPERATE

JOB

TASK_OPERATE

TRACKING

DATA

Table

Model