PaddleSpeech/cloud/README.md

# Run DS2 on PaddleCloud

>Note:
>Make sure [PaddleCloud client](https://github.com/PaddlePaddle/cloud/blob/develop/doc/usage_cn.md#%E4%B8%8B%E8%BD%BD%E5%B9%B6%E9%85%8D%E7%BD%AEpaddlecloud) has be installed and current directory is `models/deep_speech_2/cloud/`

## Step-1  Configure data set

Configure your input data and output path in pcloud_submit.sh:

-  `TRAIN_MANIFEST`： Absolute path of train data manifest file in local file system.This file has format as bellow:

```
{"audio_filepath": "/home/disk1/LibriSpeech/dev-clean/1272/128104/1272-128104-0000.flac", "duration": 5.855, "text
": "mister quilter is the ..."}
{"audio_filepath": "/home/disk1/LibriSpeech/dev-clean/1272/128104/1272-128104-0001.flac", "duration": 4.815, "text
": "nor is mister ..."}
```

- `TEST_MANIFEST`: Absolute path of train data manifest file in local filesystem. This file has format like `TRAIN_MANIFEST`.
- `VOCAB_FILE`:  Absolute path of vocabulary file in local filesytem.
- `MEAN_STD_FILE`: Absolute path of normalizer's statistic file in local filesytem.
- `CLOUD_DATA_DIR:`  Absolute path in PaddleCloud filesystem. We will upload local train data to this directory.
- `CLOUD_MODEL_DIR`: Absolute path in PaddleCloud filesystem. PaddleCloud trainer will save model to this directory.

>Note: Upload will be skipped if target file has existed in `CLOUD_DATA_DIR`.

## Step-2  Configure computation resource

Configure computation resource in pcloud_submit.sh:

```
# Configure computation resource and submit job to PaddleCloud
 paddlecloud submit \
 -image wanghaoshuang/pcloud_ds2:latest \
 -jobname ${JOB_NAME} \
 -cpu 4 \
 -gpu 4 \
 -memory 10Gi \
 -parallelism 1 \
 -pscpu 1 \
 -pservers 1 \
 -psmemory 10Gi \
 -passes 1 \
 -entry "sh pcloud_train.sh ${CLOUD_DATA_DIR} ${CLOUD_MODEL_DIR}" \
 ${DS2_PATH}
```
For more information, please refer to [PaddleCloud](https://github.com/PaddlePaddle/cloud/blob/develop/doc/usage_cn.md#提交任务)

## Step-3  Configure algorithm options
Configure algorithm options in pcloud_train.sh:
```
python train.py \
--use_gpu=1 \
--trainer_count=4 \
--batch_size=256 \
--mean_std_filepath=$MEAN_STD_FILE \
--train_manifest_path='./local.train.manifest' \
--dev_manifest_path='./local.test.manifest' \
--vocab_filepath=$VOCAB_PATH \
--output_model_dir=${MODEL_PATH}
```
You can get more information about algorithm options by follow command:
```
cd ..
python train.py --help
```

## Step-4  Submit job
```
$ sh pcloud_submit.sh
```


## Step-5 Get logs
```
$ paddlecloud logs -n 10000 deepspeech20170727130129
```
For more information, please refer to [PaddleCloud client](https://github.com/PaddlePaddle/cloud/blob/develop/doc/usage_cn.md#下载并配置paddlecloud) or get help by follow command:
```
paddlecloud --help
```