From 861b946d7a33670efeb4692aade2a14465ec5649 Mon Sep 17 00:00:00 2001 From: Xinghai Sun Date: Thu, 7 Sep 2017 11:26:58 +0800 Subject: [PATCH 1/9] Re-design README.md doc structure and add table of contents. --- README.md | 53 ++++++++++++++++++++++++++++++++++++++--------------- 1 file changed, 38 insertions(+), 15 deletions(-) diff --git a/README.md b/README.md index db07d8c20..1962c1ccb 100644 --- a/README.md +++ b/README.md @@ -1,18 +1,39 @@ # DeepSpeech2 on PaddlePaddle ->TODO: to be updated, since the directory hierarchy was changed. +*DeepSpeech2 on PaddlePaddle* is an open-source implementation of end-to-end Automatic Speech Recognition (ASR) engine, based on [Baidu's Deep Speech 2 paper](http://proceedings.mlr.press/v48/amodei16.pdf), with [PaddlePaddle](https://github.com/PaddlePaddle/Paddle) platform. Our vision is to empower both industrial application and academic research on speech-to-text, via an easy-to-use, efficent and scalable integreted implementation, including training & inferencing module, distributed [PaddleCloud](https://github.com/PaddlePaddle/cloud) training, and demo deployment. Besides, several pre-trained models for both English and Mandarin speech are also released. + +## Table of Contents +- [Prerequisites](#prerequisites) +- [Installation](#installation) +- [Getting Started](#getting-started) +- [Data Preparation](#data-preparation) +- [Training a Model](#training-a-model) +- [Inference and Evaluation](#inference-and-evaluation) +- [Distributed Cloud Training](#distributed-cloud-training) +- [Hyper-parameters Tuning](#hyper-parameters-tuning) +- [Trying Live Demo with Your Own Voice](#trying-live-demo-with-your-own-voice) +- [Experiments and Benchmarks](#experiments-and-benchmarks) +- [Questions and Help](#questions-and-help) + +## Prerequisites +- Only support Python 2.7 +- PaddlePaddle the latest version (please refer to the [Installation Guide](https://github.com/PaddlePaddle/Paddle#installation)) ## Installation +Please install the [prerequisites](#prerequisites) above before moving on this. + ``` +git clone https://github.com/PaddlePaddle/models.git +cd models/deep_speech_2 sh setup.sh ``` -Please replace `$PADDLE_INSTALL_DIR` with your own paddle installation directory. +## Getting Started -## Usage +TODO -### Preparing Data +## Data Preparation ``` cd datasets @@ -31,7 +52,7 @@ More help for arguments: python datasets/librispeech/librispeech.py --help ``` -### Preparing for Training + ``` python tools/compute_mean_std.py @@ -51,7 +72,7 @@ More help for arguments: python tools/compute_mean_std.py --help ``` -### Training +## Training a model For GPU Training: @@ -71,7 +92,7 @@ More help for arguments: python train.py --help ``` -### Preparing language model +### Inference and Evaluation The following steps, inference, parameters tuning and evaluating, will require a language model during decoding. A compressed language model is provided and can be accessed by @@ -82,7 +103,7 @@ sh run.sh cd .. ``` -### Inference + For GPU inference @@ -102,7 +123,6 @@ More help for arguments: python infer.py --help ``` -### Evaluating ``` CUDA_VISIBLE_DEVICES=0 python evaluate.py @@ -114,7 +134,7 @@ More help for arguments: python evaluate.py --help ``` -### Parameters tuning +## Hyper-parameters Tuning Usually, the parameters $\alpha$ and $\beta$ for the CTC [prefix beam search](https://arxiv.org/abs/1408.2873) decoder need to be tuned after retraining the acoustic model. @@ -138,7 +158,12 @@ python tune.py --help Then reset parameters with the tuning result before inference or evaluating. -### Playing with the ASR Demo +## Distributed Cloud Training + +If you wish to train DeepSpeech2 on PaddleCloud, please refer to +[Train DeepSpeech2 on PaddleCloud](https://github.com/PaddlePaddle/models/tree/develop/deep_speech_2/cloud). + +## Trying Live Demo with Your Own Voice A real-time ASR demo is built for users to try out the ASR model with their own voice. Please do the following installation on the machine you'd like to run the demo's client (no need for the machine running the demo's server). @@ -163,8 +188,6 @@ On the client console, press and hold the "white-space" key on the keyboard to s It could be possible to start the server and the client in two seperate machines, e.g. `demo_client.py` is usually started in a machine with a microphone hardware, while `demo_server.py` is usually started in a remote server with powerful GPUs. Please first make sure that these two machines have network access to each other, and then use `--host_ip` and `--host_port` to indicate the server machine's actual IP address (instead of the `localhost` as default) and TCP port, in both `demo_server.py` and `demo_client.py`. +## Experiments and Benchmarks -## PaddleCloud Training - -If you wish to train DeepSpeech2 on PaddleCloud, please refer to -[Train DeepSpeech2 on PaddleCloud](https://github.com/PaddlePaddle/models/tree/develop/deep_speech_2/cloud). +## Questions and Help From a00a436b528d33cc8d6e8b78c9f801c635c6f62e Mon Sep 17 00:00:00 2001 From: Xinghai Sun Date: Sun, 10 Sep 2017 11:00:16 +0800 Subject: [PATCH 2/9] Rewrite README.md doc (50%) and correct some bugs. --- README.md | 274 ++++++++++++++---- examples/librispeech/prepare_data.sh | 9 +- .../librispeech/{generate.sh => run_infer.sh} | 1 - examples/librispeech/run_train.sh | 2 +- examples/librispeech_tiny/prepare_data.sh | 39 +++ examples/librispeech_tiny/run_infer.sh | 27 ++ examples/librispeech_tiny/run_test.sh | 28 ++ examples/librispeech_tiny/run_train.sh | 30 ++ examples/librispeech_tiny/run_tune.sh | 30 ++ tools/build_vocab.py | 8 +- tools/compute_mean_std.py | 4 +- 11 files changed, 388 insertions(+), 64 deletions(-) rename examples/librispeech/{generate.sh => run_infer.sh} (97%) create mode 100644 examples/librispeech_tiny/prepare_data.sh create mode 100644 examples/librispeech_tiny/run_infer.sh create mode 100644 examples/librispeech_tiny/run_test.sh create mode 100644 examples/librispeech_tiny/run_train.sh create mode 100644 examples/librispeech_tiny/run_tune.sh diff --git a/README.md b/README.md index 1962c1ccb..2f51a5fc5 100644 --- a/README.md +++ b/README.md @@ -11,6 +11,7 @@ - [Inference and Evaluation](#inference-and-evaluation) - [Distributed Cloud Training](#distributed-cloud-training) - [Hyper-parameters Tuning](#hyper-parameters-tuning) +- [Training for Mandarin Language](#training-for-mandarin-language) - [Trying Live Demo with Your Own Voice](#trying-live-demo-with-your-own-voice) - [Experiments and Benchmarks](#experiments-and-benchmarks) - [Questions and Help](#questions-and-help) @@ -21,7 +22,7 @@ ## Installation -Please install the [prerequisites](#prerequisites) above before moving on this. +Please install the [prerequisites](#prerequisites) above before moving onto this quick installation. ``` git clone https://github.com/PaddlePaddle/models.git @@ -31,138 +32,299 @@ sh setup.sh ## Getting Started -TODO +Several shell scripts provided in `./examples` will help us to quickly give it a try, including training, inferencing, evaluation and demo deployment. + +Most of the scripts in `./examples` are configured with 8 GPUs. If you don't have 8 GPUs available, please modify `CUDA_VISIBLE_DEVICE` and `--trainer_count`. If you don't have any GPU available, please set `--use_gpu` to False. + +Let's take a tiny sampled subset of [LibriSpeech dataset](http://www.openslr.org/12/) for instance. + +- Go to directory + + ``` + cd examples/librispeech_tiny + ``` + + Notice that this is only a toy example with a tiny sampled set of LibriSpeech. If we would like to try with the complete LibriSpeech (would take much a longer time for training), please go to `examples/librispeech` instead. +- Prepare the libripseech data + + ``` + sh preprare_data.sh + ``` + + `prepare_data.sh` downloads dataset, generates file manifests, collects normalizer' statitics and builds vocabulary for us. Once the running is done, we'll find our LibriSpeech data (not full in this "tiny" example) downloaded in `~/.cache/paddle/dataset/speech/Libri` and several manifest files as well as one mean stddev file generated in `./data/librispeech_tiny`, for the further model training. It needs to be run for only once. +- Train your own ASR model + + ``` + sh run_train.sh + ``` + + `run_train.sh` starts a training job, with training logs printed to stdout and model checkpoint of every pass/epoch saved to `./checkpoints`. We can resume the training from these checkpoints, or use them for inference, evalutiaton and deployment. +- Case inference with an existing model + + ``` + sh run_infer.sh + ``` + + `run_infer.sh` will quickly show us speech-to-text decoding results for several (default: 10) audio samples with an existing model. Since the model is only trained on a subset of LibriSpeech, the performance might not be very good. We can download a well-trained model and then do the inference: + + ``` + sh download_model_run_infer.sh + ``` +- Evaluate an existing model + + ``` + sh run_test.sh + ``` + + `run_test.sh` evaluates the model with Word Error Rate (or Character Error Rate) measurement. Similarly, we can also download a well-trained model and test its performance: + + ``` + sh download_model_run_test.sh + ``` +- Try out a live demo with your own voice + + Until now, we have trained and tested an ASR model quantitively and qualitatively with existing audios. But we haven't try the model with our own speech. `demo_server.sh` and `demo_client.sh` helps quickly build up a demo ASR engine with the trained model, enabling us to test and play around with the demo with our own voice. + + We start the server in one console by entering: + + ``` + sh run_demo_server.sh + ``` + + and start the client in another console by entering: + + ``` + sh run_demo_client.sh + ``` + + Then, in the client console, press the `whitespace` key, hold, and start speaking. Until we finish our ulterance, we release the key to let the speech-to-text results show in the console. + + Notice that `run_demo_client.sh` must be run in a machine with a microphone device, while `run_demo_server.sh` could be run in one without any audio recording device, e.g. any remote server. Just be careful to update `run_demo_server.sh` and `run_demo_client.sh` with the actual accessable IP address and port, if the server and client are running with two seperate machines. Nothing has to be done if running in one single machine. + + This demo will first download a pre-trained Mandarin model (trained with 3000 hours of internal speech data). If we would like to try some other model, just update `model_path` argument in the script.   +     +More detailed information are provided in the following sections. + +Wish you a happy journey with the DeepSpeech2 ASR engine! + ## Data Preparation +#### Generate Manifest + +*DeepSpeech2 on PaddlePaddle* accepts a textual **manifest** file as its data set interface. A manifest file summarizes a set of speech data, with each line containing the meta data (e.g. filepath, transcription, duration) of one audio clip, in [JSON](http://www.json.org/) format, just as: + ``` -cd datasets -sh run_all.sh -cd .. +{"audio_filepath": "/home/work/.cache/paddle/Libri/134686/1089-134686-0001.flac", "duration": 3.275, "text": "stuff it into you his belly counselled him"} +{"audio_filepath": "/home/work/.cache/paddle/Libri/134686/1089-134686-0007.flac", "duration": 4.275, "text": "a cold lucid indifference reigned in his soul"} ``` -`sh run_all.sh` prepares all ASR datasets (currently, only LibriSpeech available). After running, we have several summarization manifest files in json-format. +To use any custom data, we only need to generate such manifest files to summarize the dataset. Given such summarized manifests, training, inference and all other modules can be aware of where to access the audio files, as well as their meta data including the transcription labels. -A manifest file summarizes a speech data set, with each line containing the meta data (i.e. audio filepath, transcript text, audio duration) of each audio file within the data set, in json format. Manifest file serves as an interface informing our system of where and what to read the speech samples. +For example script to generate such manifest files, please refer to `data/librispeech/librispeech.py`, which download and generate manifests for LibriSpeech dataset. +#### Compute Mean & Stddev for Normalizer -More help for arguments: +To perform z-score normalization (zero-mean, unit stddev) upon audio features, we have to estimate in advance the mean and standard deviation of the features, with sampled training audios: ``` -python datasets/librispeech/librispeech.py --help +python tools/compute_mean_std.py \ +--num_samples 2000 \ +--specgram_type linear \ +--manifest_paths data/librispeech/manifest.train \ +--output_path data/librispeech/mean_std.npz ``` +It will compute the mean and standard deviation of power spectgram feature with 2000 random sampled audio clips listed in `data/librispeech/manifest.train` and save the results to `data/librispeech/mean_std.npz` for further usage. -``` -python tools/compute_mean_std.py -``` +#### Build Vocabulary -It will compute mean and stdandard deviation for audio features, and save them to a file with a default name `./mean_std.npz`. This file will be used in both training and inferencing. The default feature of audio data is power spectrum, and the mfcc feature is also supported. To train and infer based on mfcc feature, please generate this file by +A list of possible characters is required to convert the target transcription into list of token indices for training and in docoders convert from them back to text. Such a character-based vocabulary can be build with `tools/build_vocab.py`. ``` -python tools/compute_mean_std.py --specgram_type mfcc +python tools/build_vocab.py \ +--count_threshold 0 \ +--vocab_path data/librispeech/eng_vocab.txt \ +--manifest_paths data/librispeech/manifest.train ``` -and specify ```--specgram_type mfcc``` when running train.py, infer.py, evaluator.py or tune.py. +It will build a vocabuary file of `data/librispeeech/eng_vocab.txt` with all transcription text in `data/librispeech/manifest.train`, without character truncation. + +#### More Help -More help for arguments: +For more help on arguments: ``` +python data/librispeech/librispeech.py --help python tools/compute_mean_std.py --help +python tools/build_vocab.py --help ``` ## Training a model -For GPU Training: +`train.py` is the main caller of the training module. We list several usage below. -``` -CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 python train.py -``` +- Start training from scratch with 8 GPUs: -For CPU Training: + ``` + CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 python train.py --trainer_count 8 + ``` -``` -python train.py --use_gpu False -``` +- Start training from scratch with 16 CPUs: + + ``` + python train.py --use_gpu False --trainer_count 16 + ``` +- Resume training from a checkpoint (an existing model): + + ``` + CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 python train.py \ + --init_model_path CHECKPOINT_PATH_TO_RESUME_FROM + ``` -More help for arguments: +For more help on arguments: ``` python train.py --help ``` +or refer to `example/librispeech/run_train.sh. -### Inference and Evaluation +#### Augment the Dataset for Training -The following steps, inference, parameters tuning and evaluating, will require a language model during decoding. -A compressed language model is provided and can be accessed by +Data augmentation has often been a highly effective technique to boost the deep learning performance. We augment our speech data by synthesizing new audios with small random perterbation (label-invariant transformation) added upon raw audios. We don't have to do the syntheses by ourselves, as it is already embeded into the data provider and is done on the fly, randomly for each epoch. + +Six optional augmentation components are provided for us to configured and inserted into the processing pipeline. + + - Volume Perturbation + - Speed Perturbation + - Shifting Perturbation + - Online Beyesian normalization + - Noise Perturbation (need background noise audio files) + - Impulse Response (need impulse audio files) + +In order to inform the trainer of what augmentation components we need and what their processing orders are, we are required to prepare a *augmentation configuration file* in JSON format. For example: ``` -cd ./lm -sh run.sh -cd .. +[{ + "type": "speed", + "params": {"min_speed_rate": 0.95, + "max_speed_rate": 1.05}, + "prob": 0.6 +}, +{ + "type": "shift", + "params": {"min_shift_ms": -5, + "max_shift_ms": 5}, + "prob": 0.8 +}] ``` +When the `--augment_conf_file` argument of `trainer.py` is set to the path of the above example configuration file, each audio clip in each epoch will be processed: with 60% of chance, it will first be speed perturbed with a uniformly random sampled speed-rate between 0.95 and 1.05, and then with 80% of chance it will be shifted in time with a random sampled offset between -5 ms and 5 ms. Finally this newly synthesized audio clip will be feed into the feature extractor for further training. +For configuration examples, please refer to `conf/augmenatation.config.example`. -For GPU inference +Be careful when we are utilizing the data augmentation technique, as improper augmentation will instead do harm to the training, due to the enlarged train-test gap. -``` -CUDA_VISIBLE_DEVICES=0 python infer.py -``` +## Inference and Evaluation -For CPU inference +#### Prepare Language Model -``` -python infer.py --use_gpu=False -``` +A language model is required to improve the decoder's performance. We have prepared two language models (with lossy compression) for users to download and try. One is for English and the other is for Mandarin. Please refer to `examples/librispeech/download_model.sh` and `examples/mandarin_demo/download_model.sh` for their urls. If you wish to train your own better language model, please refer to [KenLM](https://github.com/kpu/kenlm) for tutorials. + +TODO: any other requirements or tips to add? + +#### Speech-to-text Inference + +We provide a inference module `infer.py` to infer, decode and visualize speech-to-text results for several given audio clips, which might help to have a intuitive and qualitative evaluation of the ASR model performance. + +- Inference with GPU: + + ``` + CUDA_VISIBLE_DEVICES=0 python infer.py --trainer_count 1 + ``` -More help for arguments: +- Inference with CPU: + + ``` + python infer.py --use_gpu False + ``` + +We provide two CTC decoders: *CTC greedy decoder* and *CTC beam search decoder*. The *CTC greedy decoder* is an implementation of the simple best-path decoding algorithm, selecting at each timestep the most likely token, thus being greedy and locally optimal. The [*CTC beam search decoder*](https://arxiv.org/abs/1408.2873) otherwise utilzied a heuristic breadth-first gragh search for arriving at a near global optimality; it requires a pre-trained KenLM language model for better scoring and ranking sentences. The decoder type can be set with argument `--decoding_method`. + +For more help on arguments: ``` python infer.py --help ``` +or refer to `example/librispeech/run_infer.sh. +#### Evaluate a Model -``` -CUDA_VISIBLE_DEVICES=0 python evaluate.py -``` +To evaluate a model quantitively, we can run: + +- Evaluation with GPU: + + ``` + CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 python test.py --trainer_count 8 + ``` + +- Evaluation with CPU: -More help for arguments: + ``` + python test.py --use_gpu False + ``` + +The error rate (default: word error rate, can be set with `--error_rate_type`) will be printed. + +For more help on arguments: ``` -python evaluate.py --help +python test.py --help ``` +or refer to `example/librispeech/run_test.sh. ## Hyper-parameters Tuning -Usually, the parameters $\alpha$ and $\beta$ for the CTC [prefix beam search](https://arxiv.org/abs/1408.2873) decoder need to be tuned after retraining the acoustic model. +The hyper-parameters $\alpha$ (coefficient for language model scorer) and $\beta$ (coefficient for word count scorer) for the [*CTC beam search decoder*](https://arxiv.org/abs/1408.2873) often have a significant impact on the decoder's performance. It'd be better to re-tune them on validation samples after the accustic model is renewed. -For GPU tuning +`tools/tune.py` performs a 2-D grid search over the hyper-parameter $\alpha$ and $\beta$. We have to provide the range of $\alpha$ and $\beta$, as well as the number of attempts. -``` -CUDA_VISIBLE_DEVICES=0 python tune.py -``` +- Tuning with GPU: -For CPU tuning + ``` + CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 python tools/tune.py \ + --trainer_count 8 \ + --alpha_from 0.1 \ + --alpha_to 0.36 \ + --num_alphas 14 \ + --beta_from 0.05 \ + --beta_to 1.0 \ + --num_betas 20 + ``` -``` -python tune.py --use_gpu=False -``` +- Tuning with CPU: -More help for arguments: + ``` + python tools/tune.py --use_gpu False + ``` + +After tuning, we can reset $\alpha$ and $\beta$ in the inference and evaluation modules to see if they can really improve the ASR performance. ``` python tune.py --help ``` +or refer to `example/librispeech/run_tune.sh. -Then reset parameters with the tuning result before inference or evaluating. +TODO: add figure. ## Distributed Cloud Training If you wish to train DeepSpeech2 on PaddleCloud, please refer to [Train DeepSpeech2 on PaddleCloud](https://github.com/PaddlePaddle/models/tree/develop/deep_speech_2/cloud). +## Training for Mandarin Language + ## Trying Live Demo with Your Own Voice A real-time ASR demo is built for users to try out the ASR model with their own voice. Please do the following installation on the machine you'd like to run the demo's client (no need for the machine running the demo's server). diff --git a/examples/librispeech/prepare_data.sh b/examples/librispeech/prepare_data.sh index 162a38c49..a18402ea3 100644 --- a/examples/librispeech/prepare_data.sh +++ b/examples/librispeech/prepare_data.sh @@ -13,7 +13,14 @@ if [ $? -ne 0 ]; then exit 1 fi -#cat data/librispeech/manifest.train* | shuf > data/librispeech/manifest.train +cat data/librispeech/manifest.train* | shuf > data/librispeech/manifest.train + + +# build vocabulary (for English data, we can just skip this) +# python tools/build_vocab.py \ +# --count_threshold=0 \ +# --vocab_path='data/librispeech/eng_vocab.txt' \ +# --manifest_paths='data/librispeech/manifeset.train' # compute mean and stddev for normalizer diff --git a/examples/librispeech/generate.sh b/examples/librispeech/run_infer.sh similarity index 97% rename from examples/librispeech/generate.sh rename to examples/librispeech/run_infer.sh index a34b7bc10..619d546e8 100644 --- a/examples/librispeech/generate.sh +++ b/examples/librispeech/run_infer.sh @@ -8,7 +8,6 @@ python -u infer.py \ --trainer_count=1 \ --beam_size=500 \ --num_proc_bsearch=12 \ ---num_proc_data=12 \ --num_conv_layers=2 \ --num_rnn_layers=3 \ --rnn_layer_size=2048 \ diff --git a/examples/librispeech/run_train.sh b/examples/librispeech/run_train.sh index 832838a81..14672167c 100644 --- a/examples/librispeech/run_train.sh +++ b/examples/librispeech/run_train.sh @@ -6,7 +6,7 @@ CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 \ python -u train.py \ --batch_size=256 \ --trainer_count=8 \ ---num_passes=200 \ +--num_passes=50 \ --num_proc_data=12 \ --num_conv_layers=2 \ --num_rnn_layers=3 \ diff --git a/examples/librispeech_tiny/prepare_data.sh b/examples/librispeech_tiny/prepare_data.sh new file mode 100644 index 000000000..a18402ea3 --- /dev/null +++ b/examples/librispeech_tiny/prepare_data.sh @@ -0,0 +1,39 @@ +#! /usr/bin/bash + +pushd ../.. + +# download data, generate manifests +python data/librispeech/librispeech.py \ +--manifest_prefix='data/librispeech/manifest' \ +--full_download='True' \ +--target_dir='~/.cache/paddle/dataset/speech/Libri' + +if [ $? -ne 0 ]; then + echo "Prepare LibriSpeech failed. Terminated." + exit 1 +fi + +cat data/librispeech/manifest.train* | shuf > data/librispeech/manifest.train + + +# build vocabulary (for English data, we can just skip this) +# python tools/build_vocab.py \ +# --count_threshold=0 \ +# --vocab_path='data/librispeech/eng_vocab.txt' \ +# --manifest_paths='data/librispeech/manifeset.train' + + +# compute mean and stddev for normalizer +python tools/compute_mean_std.py \ +--manifest_path='data/librispeech/manifest.train' \ +--num_samples=2000 \ +--specgram_type='linear' \ +--output_path='data/librispeech/mean_std.npz' + +if [ $? -ne 0 ]; then + echo "Compute mean and stddev failed. Terminated." + exit 1 +fi + + +echo "LibriSpeech Data preparation done." diff --git a/examples/librispeech_tiny/run_infer.sh b/examples/librispeech_tiny/run_infer.sh new file mode 100644 index 000000000..619d546e8 --- /dev/null +++ b/examples/librispeech_tiny/run_infer.sh @@ -0,0 +1,27 @@ +#! /usr/bin/bash + +pushd ../.. + +CUDA_VISIBLE_DEVICES=0 \ +python -u infer.py \ +--num_samples=10 \ +--trainer_count=1 \ +--beam_size=500 \ +--num_proc_bsearch=12 \ +--num_conv_layers=2 \ +--num_rnn_layers=3 \ +--rnn_layer_size=2048 \ +--alpha=0.36 \ +--beta=0.25 \ +--cutoff_prob=0.99 \ +--use_gru=False \ +--use_gpu=True \ +--share_rnn_weights=True \ +--infer_manifest='data/librispeech/manifest.dev-clean' \ +--mean_std_path='data/librispeech/mean_std.npz' \ +--vocab_path='data/librispeech/eng_vocab.txt' \ +--model_path='checkpoints/params.latest.tar.gz' \ +--lang_model_path='lm/data/common_crawl_00.prune01111.trie.klm' \ +--decoding_method='ctc_beam_search' \ +--error_rate_type='wer' \ +--specgram_type='linear' diff --git a/examples/librispeech_tiny/run_test.sh b/examples/librispeech_tiny/run_test.sh new file mode 100644 index 000000000..5a14cb682 --- /dev/null +++ b/examples/librispeech_tiny/run_test.sh @@ -0,0 +1,28 @@ +#! /usr/bin/bash + +pushd ../.. + +CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 \ +python -u evaluate.py \ +--batch_size=128 \ +--trainer_count=8 \ +--beam_size=500 \ +--num_proc_bsearch=12 \ +--num_proc_data=12 \ +--num_conv_layers=2 \ +--num_rnn_layers=3 \ +--rnn_layer_size=2048 \ +--alpha=0.36 \ +--beta=0.25 \ +--cutoff_prob=0.99 \ +--use_gru=False \ +--use_gpu=True \ +--share_rnn_weights=True \ +--test_manifest='data/librispeech/manifest.test-clean' \ +--mean_std_path='data/librispeech/mean_std.npz' \ +--vocab_path='data/librispeech/eng_vocab.txt' \ +--model_path='checkpoints/params.latest.tar.gz' \ +--lang_model_path='lm/data/common_crawl_00.prune01111.trie.klm' \ +--decoding_method='ctc_beam_search' \ +--error_rate_type='wer' \ +--specgram_type='linear' diff --git a/examples/librispeech_tiny/run_train.sh b/examples/librispeech_tiny/run_train.sh new file mode 100644 index 000000000..14672167c --- /dev/null +++ b/examples/librispeech_tiny/run_train.sh @@ -0,0 +1,30 @@ +#! /usr/bin/bash + +pushd ../.. + +CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 \ +python -u train.py \ +--batch_size=256 \ +--trainer_count=8 \ +--num_passes=50 \ +--num_proc_data=12 \ +--num_conv_layers=2 \ +--num_rnn_layers=3 \ +--rnn_layer_size=2048 \ +--num_iter_print=100 \ +--learning_rate=5e-4 \ +--max_duration=27.0 \ +--min_duration=0.0 \ +--use_sortagrad=True \ +--use_gru=False \ +--use_gpu=True \ +--is_local=True \ +--share_rnn_weights=True \ +--train_manifest='data/librispeech/manifest.train' \ +--dev_manifest='data/librispeech/manifest.dev' \ +--mean_std_path='data/librispeech/mean_std.npz' \ +--vocab_path='data/librispeech/eng_vocab.txt' \ +--output_model_dir='./checkpoints' \ +--augment_conf_path='conf/augmentation.config' \ +--specgram_type='linear' \ +--shuffle_method='batch_shuffle_clipped' diff --git a/examples/librispeech_tiny/run_tune.sh b/examples/librispeech_tiny/run_tune.sh new file mode 100644 index 000000000..9d992e884 --- /dev/null +++ b/examples/librispeech_tiny/run_tune.sh @@ -0,0 +1,30 @@ +#! /usr/bin/bash + +pushd ../.. + +CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 \ +python -u tools/tune.py \ +--num_samples=100 \ +--trainer_count=8 \ +--beam_size=500 \ +--num_proc_bsearch=12 \ +--num_conv_layers=2 \ +--num_rnn_layers=3 \ +--rnn_layer_size=2048 \ +--num_alphas=14 \ +--num_betas=20 \ +--alpha_from=0.1 \ +--alpha_to=0.36 \ +--beta_from=0.05 \ +--beta_to=1.0 \ +--cutoff_prob=0.99 \ +--use_gru=False \ +--use_gpu=True \ +--share_rnn_weights=True \ +--tune_manifest='data/librispeech/manifest.dev-clean' \ +--mean_std_path='data/librispeech/mean_std.npz' \ +--vocab_path='data/librispeech/eng_vocab.txt' \ +--model_path='checkpoints/params.latest.tar.gz' \ +--lang_model_path='lm/data/common_crawl_00.prune01111.trie.klm' \ +--error_rate_type='wer' \ +--specgram_type='linear' diff --git a/tools/build_vocab.py b/tools/build_vocab.py index 6fbb9bdfc..ef9bde49f 100644 --- a/tools/build_vocab.py +++ b/tools/build_vocab.py @@ -21,8 +21,10 @@ add_arg = functools.partial(add_arguments, argparser=parser) # yapf: disable add_arg('count_threshold', int, 0, "Truncation threshold for char counts.") add_arg('vocab_path', str, - 'datasets/vocab/zh_vocab.txt', - "Filepath to write the vocabulary.") + None, + "Filepath to write the vocabulary.", + nargs='+', + required=True) add_arg('manifest_paths', str, None, "Filepaths of manifests for building vocabulary. " @@ -34,7 +36,7 @@ args = parser.parse_args() def count_manifest(counter, manifest_path): - manifest_jsons = utils.read_manifest(manifest_path) + manifest_jsons = read_manifest(manifest_path) for line_json in manifest_jsons: for char in line_json['text']: counter.update(char) diff --git a/tools/compute_mean_std.py b/tools/compute_mean_std.py index 5bb6be39d..11aa856d7 100644 --- a/tools/compute_mean_std.py +++ b/tools/compute_mean_std.py @@ -20,10 +20,10 @@ add_arg('specgram_type', str, "Audio feature type. Options: linear, mfcc.", choices=['linear', 'mfcc']) add_arg('manifest_path', str, - 'datasets/manifest.train', + 'data/librispeech/manifest.train', "Filepath of manifest to compute normalizer's mean and stddev.") add_arg('output_path', str, - 'mean_std.npz', + 'data/librispeech/mean_std.npz', "Filepath of write mean and stddev to (.npz).") # yapf: disable args = parser.parse_args() From ae7ef7929a0bce79c5de03366840711e8e77f5b6 Mon Sep 17 00:00:00 2001 From: Xinghai Sun Date: Sun, 10 Sep 2017 20:36:38 +0800 Subject: [PATCH 3/9] Rename some folders and update examples. --- data/librispeech/librispeech.py | 2 +- data/tiny/tiny.py | 126 ++++++++++++++++++ examples/librispeech/prepare_data.sh | 2 +- examples/librispeech_tiny/prepare_data.sh | 39 ------ examples/tiny/run_data.sh | 45 +++++++ .../{librispeech_tiny => tiny}/run_infer.sh | 12 +- .../{librispeech_tiny => tiny}/run_test.sh | 0 .../{librispeech_tiny => tiny}/run_train.sh | 20 +-- .../{librispeech_tiny => tiny}/run_tune.sh | 0 infer.py | 6 +- {lm => model_utils}/__init__.py | 0 {models => model_utils}/decoder.py | 2 + {lm => model_utils}/lm_scorer.py | 0 {models => model_utils}/model.py | 7 +- {models => model_utils}/network.py | 0 .../tests/test_decoders.py | 2 +- models/__init__.py | 0 lm/run.sh => models/lm/download_en.sh | 3 - test.py | 6 +- tools/build_vocab.py | 6 +- tools/tune.py | 6 +- train.py | 4 +- 22 files changed, 209 insertions(+), 79 deletions(-) create mode 100644 data/tiny/tiny.py delete mode 100644 examples/librispeech_tiny/prepare_data.sh create mode 100644 examples/tiny/run_data.sh rename examples/{librispeech_tiny => tiny}/run_infer.sh (58%) rename examples/{librispeech_tiny => tiny}/run_test.sh (100%) rename examples/{librispeech_tiny => tiny}/run_train.sh (56%) rename examples/{librispeech_tiny => tiny}/run_tune.sh (100%) rename {lm => model_utils}/__init__.py (100%) rename {models => model_utils}/decoder.py (99%) rename {lm => model_utils}/lm_scorer.py (100%) rename {models => model_utils}/model.py (97%) rename {models => model_utils}/network.py (100%) rename {models => model_utils}/tests/test_decoders.py (99%) delete mode 100644 models/__init__.py rename lm/run.sh => models/lm/download_en.sh (99%) diff --git a/data/librispeech/librispeech.py b/data/librispeech/librispeech.py index d963a7d53..14a3804e2 100644 --- a/data/librispeech/librispeech.py +++ b/data/librispeech/librispeech.py @@ -41,7 +41,7 @@ MD5_TRAIN_OTHER_500 = "d1a0fd59409feb2c614ce4d30c387708" parser = argparse.ArgumentParser(description=__doc__) parser.add_argument( "--target_dir", - default=DATA_HOME + "/Libri", + default=DATA_HOME + "/libri", type=str, help="Directory to save the dataset. (default: %(default)s)") parser.add_argument( diff --git a/data/tiny/tiny.py b/data/tiny/tiny.py new file mode 100644 index 000000000..8ba2a13c5 --- /dev/null +++ b/data/tiny/tiny.py @@ -0,0 +1,126 @@ +"""Prepare Librispeech ASR datasets. + +Download, unpack and create manifest files. +Manifest file is a json-format file with each line containing the +meta data (i.e. audio filepath, transcript and audio duration) +of each audio file in the data set. +""" +from __future__ import absolute_import +from __future__ import division +from __future__ import print_function + +import distutils.util +import os +import sys +import tarfile +import argparse +import soundfile +import json +import codecs +from paddle.v2.dataset.common import md5file + +DATA_HOME = os.path.expanduser('~/.cache/paddle/dataset/speech') + +URL_ROOT = "http://www.openslr.org/resources/12" +URL_DEV_CLEAN = URL_ROOT + "/dev-clean.tar.gz" +MD5_DEV_CLEAN = "42e2234ba48799c1f50f24a7926300a1" + +parser = argparse.ArgumentParser(description=__doc__) +parser.add_argument( + "--target_dir", + default=DATA_HOME + "/tiny", + type=str, + help="Directory to save the dataset. (default: %(default)s)") +parser.add_argument( + "--manifest_prefix", + default="manifest", + type=str, + help="Filepath prefix for output manifests. (default: %(default)s)") +args = parser.parse_args() + + +def download(url, md5sum, target_dir): + """ + Download file from url to target_dir, and check md5sum. + """ + if not os.path.exists(target_dir): os.makedirs(target_dir) + filepath = os.path.join(target_dir, url.split("/")[-1]) + if not (os.path.exists(filepath) and md5file(filepath) == md5sum): + print("Downloading %s ..." % url) + os.system("wget -c " + url + " -P " + target_dir) + print("\nMD5 Chesksum %s ..." % filepath) + if not md5file(filepath) == md5sum: + raise RuntimeError("MD5 checksum failed.") + else: + print("File exists, skip downloading. (%s)" % filepath) + return filepath + + +def unpack(filepath, target_dir): + """ + Unpack the file to the target_dir. + """ + print("Unpacking %s ..." % filepath) + tar = tarfile.open(filepath) + tar.extractall(target_dir) + tar.close() + + +def create_manifest(data_dir, manifest_path): + """ + Create a manifest json file summarizing the data set, with each line + containing the meta data (i.e. audio filepath, transcription text, audio + duration) of each audio file within the data set. + """ + print("Creating manifest %s ..." % manifest_path) + json_lines = [] + for subfolder, _, filelist in sorted(os.walk(data_dir)): + text_filelist = [ + filename for filename in filelist if filename.endswith('trans.txt') + ] + if len(text_filelist) > 0: + text_filepath = os.path.join(data_dir, subfolder, text_filelist[0]) + for line in open(text_filepath): + segments = line.strip().split() + text = ' '.join(segments[1:]).lower() + audio_filepath = os.path.join(data_dir, subfolder, + segments[0] + '.flac') + audio_data, samplerate = soundfile.read(audio_filepath) + duration = float(len(audio_data)) / samplerate + json_lines.append( + json.dumps({ + 'audio_filepath': audio_filepath, + 'duration': duration, + 'text': text + })) + with codecs.open(manifest_path, 'w', 'utf-8') as out_file: + for line in json_lines: + out_file.write(line + '\n') + + +def prepare_dataset(url, md5sum, target_dir, manifest_path): + """ + Download, unpack and create summmary manifest file. + """ + if not os.path.exists(os.path.join(target_dir, "LibriSpeech")): + # download + filepath = download(url, md5sum, target_dir) + # unpack + unpack(filepath, target_dir) + else: + print("Skip downloading and unpacking. Data already exists in %s." % + target_dir) + # create manifest json file + create_manifest(target_dir, manifest_path) + + +def main(): + prepare_dataset( + url=URL_DEV_CLEAN, + md5sum=MD5_DEV_CLEAN, + target_dir=os.path.join(args.target_dir, "dev-clean"), + manifest_path=args.manifest_prefix + ".dev-clean") + + +if __name__ == '__main__': + main() diff --git a/examples/librispeech/prepare_data.sh b/examples/librispeech/prepare_data.sh index a18402ea3..6e9997703 100644 --- a/examples/librispeech/prepare_data.sh +++ b/examples/librispeech/prepare_data.sh @@ -16,7 +16,7 @@ fi cat data/librispeech/manifest.train* | shuf > data/librispeech/manifest.train -# build vocabulary (for English data, we can just skip this) +# build vocabulary (can be skipped for English, as already provided) # python tools/build_vocab.py \ # --count_threshold=0 \ # --vocab_path='data/librispeech/eng_vocab.txt' \ diff --git a/examples/librispeech_tiny/prepare_data.sh b/examples/librispeech_tiny/prepare_data.sh deleted file mode 100644 index a18402ea3..000000000 --- a/examples/librispeech_tiny/prepare_data.sh +++ /dev/null @@ -1,39 +0,0 @@ -#! /usr/bin/bash - -pushd ../.. - -# download data, generate manifests -python data/librispeech/librispeech.py \ ---manifest_prefix='data/librispeech/manifest' \ ---full_download='True' \ ---target_dir='~/.cache/paddle/dataset/speech/Libri' - -if [ $? -ne 0 ]; then - echo "Prepare LibriSpeech failed. Terminated." - exit 1 -fi - -cat data/librispeech/manifest.train* | shuf > data/librispeech/manifest.train - - -# build vocabulary (for English data, we can just skip this) -# python tools/build_vocab.py \ -# --count_threshold=0 \ -# --vocab_path='data/librispeech/eng_vocab.txt' \ -# --manifest_paths='data/librispeech/manifeset.train' - - -# compute mean and stddev for normalizer -python tools/compute_mean_std.py \ ---manifest_path='data/librispeech/manifest.train' \ ---num_samples=2000 \ ---specgram_type='linear' \ ---output_path='data/librispeech/mean_std.npz' - -if [ $? -ne 0 ]; then - echo "Compute mean and stddev failed. Terminated." - exit 1 -fi - - -echo "LibriSpeech Data preparation done." diff --git a/examples/tiny/run_data.sh b/examples/tiny/run_data.sh new file mode 100644 index 000000000..44345d8cc --- /dev/null +++ b/examples/tiny/run_data.sh @@ -0,0 +1,45 @@ +#! /usr/bin/bash + +pushd ../.. + +# download data, generate manifests +python data/tiny/tiny.py \ +--manifest_prefix='data/tiny/manifest' \ +--target_dir=$HOME'/.cache/paddle/dataset/speech/tiny' + +if [ $? -ne 0 ]; then + echo "Prepare LibriSpeech failed. Terminated." + exit 1 +fi + +cat data/tiny/manifest.dev-clean | head -n 32 > data/tiny/manifest.train +cat data/tiny/manifest.dev-clean | head -n 48 | tail -n 16 > data/tiny/manifest.dev +cat data/tiny/manifest.dev-clean | head -n 64 | tail -n 16 > data/tiny/manifest.test + + +# build vocabulary +python tools/build_vocab.py \ +--count_threshold=0 \ +--vocab_path='data/tiny/vocab.txt' \ +--manifest_paths='data/tiny/manifest.train' + +if [ $? -ne 0 ]; then + echo "Build vocabulary failed. Terminated." + exit 1 +fi + + +# compute mean and stddev for normalizer +python tools/compute_mean_std.py \ +--manifest_path='data/tiny/manifest.train' \ +--num_samples=32 \ +--specgram_type='linear' \ +--output_path='data/tiny/mean_std.npz' + +if [ $? -ne 0 ]; then + echo "Compute mean and stddev failed. Terminated." + exit 1 +fi + + +echo "Tiny data preparation done." diff --git a/examples/librispeech_tiny/run_infer.sh b/examples/tiny/run_infer.sh similarity index 58% rename from examples/librispeech_tiny/run_infer.sh rename to examples/tiny/run_infer.sh index 619d546e8..f09bc6638 100644 --- a/examples/librispeech_tiny/run_infer.sh +++ b/examples/tiny/run_infer.sh @@ -4,7 +4,7 @@ pushd ../.. CUDA_VISIBLE_DEVICES=0 \ python -u infer.py \ ---num_samples=10 \ +--num_samples=4 \ --trainer_count=1 \ --beam_size=500 \ --num_proc_bsearch=12 \ @@ -17,11 +17,11 @@ python -u infer.py \ --use_gru=False \ --use_gpu=True \ --share_rnn_weights=True \ ---infer_manifest='data/librispeech/manifest.dev-clean' \ ---mean_std_path='data/librispeech/mean_std.npz' \ ---vocab_path='data/librispeech/eng_vocab.txt' \ ---model_path='checkpoints/params.latest.tar.gz' \ ---lang_model_path='lm/data/common_crawl_00.prune01111.trie.klm' \ +--infer_manifest='data/tiny/manifest.train' \ +--mean_std_path='data/tiny/mean_std.npz' \ +--vocab_path='data/tiny/vocab.txt' \ +--model_path='checkpoints/params.pass-14.tar.gz' \ +--lang_model_path='models/lm/common_crawl_00.prune01111.trie.klm' \ --decoding_method='ctc_beam_search' \ --error_rate_type='wer' \ --specgram_type='linear' diff --git a/examples/librispeech_tiny/run_test.sh b/examples/tiny/run_test.sh similarity index 100% rename from examples/librispeech_tiny/run_test.sh rename to examples/tiny/run_test.sh diff --git a/examples/librispeech_tiny/run_train.sh b/examples/tiny/run_train.sh similarity index 56% rename from examples/librispeech_tiny/run_train.sh rename to examples/tiny/run_train.sh index 14672167c..7ca336876 100644 --- a/examples/librispeech_tiny/run_train.sh +++ b/examples/tiny/run_train.sh @@ -2,17 +2,17 @@ pushd ../.. -CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 \ +CUDA_VISIBLE_DEVICES=0,1 \ python -u train.py \ ---batch_size=256 \ ---trainer_count=8 \ ---num_passes=50 \ ---num_proc_data=12 \ +--batch_size=2 \ +--trainer_count=1 \ +--num_passes=10 \ +--num_proc_data=1 \ --num_conv_layers=2 \ --num_rnn_layers=3 \ --rnn_layer_size=2048 \ --num_iter_print=100 \ ---learning_rate=5e-4 \ +--learning_rate=5e-5 \ --max_duration=27.0 \ --min_duration=0.0 \ --use_sortagrad=True \ @@ -20,10 +20,10 @@ python -u train.py \ --use_gpu=True \ --is_local=True \ --share_rnn_weights=True \ ---train_manifest='data/librispeech/manifest.train' \ ---dev_manifest='data/librispeech/manifest.dev' \ ---mean_std_path='data/librispeech/mean_std.npz' \ ---vocab_path='data/librispeech/eng_vocab.txt' \ +--train_manifest='data/tiny/manifest.train' \ +--dev_manifest='data/tiny/manifest.train' \ +--mean_std_path='data/tiny/mean_std.npz' \ +--vocab_path='data/tiny/vocab.txt' \ --output_model_dir='./checkpoints' \ --augment_conf_path='conf/augmentation.config' \ --specgram_type='linear' \ diff --git a/examples/librispeech_tiny/run_tune.sh b/examples/tiny/run_tune.sh similarity index 100% rename from examples/librispeech_tiny/run_tune.sh rename to examples/tiny/run_tune.sh diff --git a/infer.py b/infer.py index 1ce969ae0..73e200b49 100644 --- a/infer.py +++ b/infer.py @@ -7,7 +7,7 @@ import argparse import functools import paddle.v2 as paddle from data_utils.data import DataGenerator -from models.model import DeepSpeech2Model +from model_utils.model import DeepSpeech2Model from utils.error_rate import wer, cer from utils.utility import add_arguments, print_arguments @@ -35,10 +35,10 @@ add_arg('mean_std_path', str, 'data/librispeech/mean_std.npz', "Filepath of normalizer's mean & std.") add_arg('vocab_path', str, - 'data/librispeech/eng_vocab.txt', + 'data/librispeech/vocab.txt', "Filepath of vocabulary.") add_arg('lang_model_path', str, - 'lm/data/common_crawl_00.prune01111.trie.klm', + 'model_zoo/lm/common_crawl_00.prune01111.trie.klm', "Filepath for language model.") add_arg('model_path', str, './checkpoints/params.latest.tar.gz', diff --git a/lm/__init__.py b/model_utils/__init__.py similarity index 100% rename from lm/__init__.py rename to model_utils/__init__.py diff --git a/models/decoder.py b/model_utils/decoder.py similarity index 99% rename from models/decoder.py rename to model_utils/decoder.py index 61ead25c8..ffba2731a 100644 --- a/models/decoder.py +++ b/model_utils/decoder.py @@ -180,6 +180,8 @@ def ctc_beam_search_decoder(probs_seq, prob = prob * ext_scoring_func(result) log_prob = log(prob) beam_result.append((log_prob, result)) + else: + beam_result.append((float('-inf'), '')) ## output top beam_size decoding results beam_result = sorted(beam_result, key=lambda asd: asd[0], reverse=True) diff --git a/lm/lm_scorer.py b/model_utils/lm_scorer.py similarity index 100% rename from lm/lm_scorer.py rename to model_utils/lm_scorer.py diff --git a/models/model.py b/model_utils/model.py similarity index 97% rename from models/model.py rename to model_utils/model.py index 93c4c41bf..cf146f8ce 100644 --- a/models/model.py +++ b/model_utils/model.py @@ -8,9 +8,10 @@ import os import time import gzip import paddle.v2 as paddle -from lm.lm_scorer import LmScorer -from models.decoder import ctc_greedy_decoder, ctc_beam_search_decoder -from models.network import deep_speech_v2_network +from model_utils.lm_scorer import LmScorer +from model_utils.decoder import ctc_greedy_decoder, ctc_beam_search_decoder +from model_utils.decoder import ctc_beam_search_decoder_batch +from model_utils.network import deep_speech_v2_network class DeepSpeech2Model(object): diff --git a/models/network.py b/model_utils/network.py similarity index 100% rename from models/network.py rename to model_utils/network.py diff --git a/models/tests/test_decoders.py b/model_utils/tests/test_decoders.py similarity index 99% rename from models/tests/test_decoders.py rename to model_utils/tests/test_decoders.py index acce46af8..adf36eefc 100644 --- a/models/tests/test_decoders.py +++ b/model_utils/tests/test_decoders.py @@ -4,7 +4,7 @@ from __future__ import division from __future__ import print_function import unittest -from models import decoder +from model_utils import decoder class TestDecoders(unittest.TestCase): diff --git a/models/__init__.py b/models/__init__.py deleted file mode 100644 index e69de29bb..000000000 diff --git a/lm/run.sh b/models/lm/download_en.sh similarity index 99% rename from lm/run.sh rename to models/lm/download_en.sh index 2108ea55f..5ca33c679 100644 --- a/lm/run.sh +++ b/models/lm/download_en.sh @@ -14,6 +14,3 @@ if [ $MD5 != $md5_tmp ]; then echo "Fail to download the language model!" exit 1 fi - - - diff --git a/test.py b/test.py index 747e40df8..791bfd585 100644 --- a/test.py +++ b/test.py @@ -7,7 +7,7 @@ import argparse import functools import paddle.v2 as paddle from data_utils.data import DataGenerator -from models.model import DeepSpeech2Model +from model_utils.model import DeepSpeech2Model from utils.error_rate import wer, cer from utils.utility import add_arguments, print_arguments @@ -36,14 +36,14 @@ add_arg('mean_std_path', str, 'data/librispeech/mean_std.npz', "Filepath of normalizer's mean & std.") add_arg('vocab_path', str, - 'data/librispeech/eng_vocab.txt', + 'data/librispeech/vocab.txt', "Filepath of vocabulary.") add_arg('model_path', str, './checkpoints/params.latest.tar.gz', "If None, the training starts from scratch, " "otherwise, it resumes from the pre-trained model.") add_arg('lang_model_path', str, - 'lm/data/common_crawl_00.prune01111.trie.klm', + 'model_zoo/lm/common_crawl_00.prune01111.trie.klm', "Filepath for language model.") add_arg('decoding_method', str, 'ctc_beam_search', diff --git a/tools/build_vocab.py b/tools/build_vocab.py index ef9bde49f..e167e92ad 100644 --- a/tools/build_vocab.py +++ b/tools/build_vocab.py @@ -21,10 +21,8 @@ add_arg = functools.partial(add_arguments, argparser=parser) # yapf: disable add_arg('count_threshold', int, 0, "Truncation threshold for char counts.") add_arg('vocab_path', str, - None, - "Filepath to write the vocabulary.", - nargs='+', - required=True) + 'data/librispeech/vocab.txt', + "Filepath to write the vocabulary.") add_arg('manifest_paths', str, None, "Filepaths of manifests for building vocabulary. " diff --git a/tools/tune.py b/tools/tune.py index 7a2379109..25e495f19 100644 --- a/tools/tune.py +++ b/tools/tune.py @@ -9,7 +9,7 @@ import functools import paddle.v2 as paddle import _init_paths from data_utils.data import DataGenerator -from models.model import DeepSpeech2Model +from model_utils.model import DeepSpeech2Model from utils.error_rate import wer from utils.utility import add_arguments, print_arguments @@ -41,10 +41,10 @@ add_arg('mean_std_path', str, 'data/librispeech/mean_std.npz', "Filepath of normalizer's mean & std.") add_arg('vocab_path', str, - 'data/librispeech/eng_vocab.txt', + 'data/librispeech/vocab.txt', "Filepath of vocabulary.") add_arg('lang_model_path', str, - 'lm/data/common_crawl_00.prune01111.trie.klm', + 'model_zoo/lm/common_crawl_00.prune01111.trie.klm', "Filepath for language model.") add_arg('model_path', str, './checkpoints/params.latest.tar.gz', diff --git a/train.py b/train.py index 4a7a0eda2..bbf1cd729 100644 --- a/train.py +++ b/train.py @@ -6,7 +6,7 @@ from __future__ import print_function import argparse import functools import paddle.v2 as paddle -from models.model import DeepSpeech2Model +from model_utils.model import DeepSpeech2Model from data_utils.data import DataGenerator from utils.utility import add_arguments, print_arguments @@ -41,7 +41,7 @@ add_arg('mean_std_path', str, 'data/librispeech/mean_std.npz', "Filepath of normalizer's mean & std.") add_arg('vocab_path', str, - 'data/librispeech/eng_vocab.txt', + 'data/librispeech/vocab.txt', "Filepath of vocabulary.") add_arg('init_model_path', str, None, From e11b735de5ba55f90f502c67026d94dd78e02226 Mon Sep 17 00:00:00 2001 From: Xinghai Sun Date: Tue, 12 Sep 2017 00:51:13 +0800 Subject: [PATCH 4/9] Update examples scripts and REAME.md for DS2. --- README.md | 96 ++++++++++--------- data/librispeech/eng_vocab.txt | 28 ------ data/librispeech/librispeech.py | 31 +++--- deploy/demo_server.py | 2 +- .../{prepare_data.sh => run_data.sh} | 24 +++-- examples/librispeech/run_infer.sh | 30 ++++-- examples/librispeech/run_infer_golden.sh | 54 +++++++++++ examples/librispeech/run_test.sh | 32 +++++-- examples/librispeech/run_test_golden.sh | 55 +++++++++++ examples/librispeech/run_train.sh | 17 +++- examples/librispeech/run_tune.sh | 17 +++- examples/mandarin/run_demo_client.sh | 17 ++++ examples/mandarin/run_demo_server.sh | 53 ++++++++++ examples/tiny/run_data.sh | 18 ++-- examples/tiny/run_infer.sh | 28 +++++- examples/tiny/run_infer_golden.sh | 54 +++++++++++ examples/tiny/run_test.sh | 38 ++++++-- examples/tiny/run_test_golden.sh | 55 +++++++++++ examples/tiny/run_train.sh | 27 ++++-- examples/tiny/run_tune.sh | 21 ++-- models/librispeech/download_model.sh | 20 ++++ models/lm/download_en.sh | 16 ---- models/lm/download_lm_en.sh | 18 ++++ utils/utility.sh | 20 ++++ 24 files changed, 594 insertions(+), 177 deletions(-) delete mode 100644 data/librispeech/eng_vocab.txt rename examples/librispeech/{prepare_data.sh => run_data.sh} (57%) create mode 100644 examples/librispeech/run_infer_golden.sh create mode 100644 examples/librispeech/run_test_golden.sh create mode 100644 examples/mandarin/run_demo_client.sh create mode 100644 examples/mandarin/run_demo_server.sh create mode 100644 examples/tiny/run_infer_golden.sh create mode 100644 examples/tiny/run_test_golden.sh create mode 100644 models/librispeech/download_model.sh delete mode 100644 models/lm/download_en.sh create mode 100644 models/lm/download_lm_en.sh create mode 100644 utils/utility.sh diff --git a/README.md b/README.md index 2f51a5fc5..aae0dc6d8 100644 --- a/README.md +++ b/README.md @@ -1,6 +1,6 @@ # DeepSpeech2 on PaddlePaddle -*DeepSpeech2 on PaddlePaddle* is an open-source implementation of end-to-end Automatic Speech Recognition (ASR) engine, based on [Baidu's Deep Speech 2 paper](http://proceedings.mlr.press/v48/amodei16.pdf), with [PaddlePaddle](https://github.com/PaddlePaddle/Paddle) platform. Our vision is to empower both industrial application and academic research on speech-to-text, via an easy-to-use, efficent and scalable integreted implementation, including training & inferencing module, distributed [PaddleCloud](https://github.com/PaddlePaddle/cloud) training, and demo deployment. Besides, several pre-trained models for both English and Mandarin speech are also released. +*DeepSpeech2 on PaddlePaddle* is an open-source implementation of end-to-end Automatic Speech Recognition (ASR) engine, based on [Baidu's Deep Speech 2 paper](http://proceedings.mlr.press/v48/amodei16.pdf), with [PaddlePaddle](https://github.com/PaddlePaddle/Paddle) platform. Our vision is to empower both industrial application and academic research on speech-to-text, via an easy-to-use, efficent and scalable integreted implementation, including training, inferencing & testing module, distributed [PaddleCloud](https://github.com/PaddlePaddle/cloud) training, and demo deployment. Besides, several pre-trained models for both English and Mandarin are also released. ## Table of Contents - [Prerequisites](#prerequisites) @@ -8,12 +8,14 @@ - [Getting Started](#getting-started) - [Data Preparation](#data-preparation) - [Training a Model](#training-a-model) +- [Data Augmentation Pipeline](#data-augmentation-pipeline) - [Inference and Evaluation](#inference-and-evaluation) - [Distributed Cloud Training](#distributed-cloud-training) - [Hyper-parameters Tuning](#hyper-parameters-tuning) - [Training for Mandarin Language](#training-for-mandarin-language) - [Trying Live Demo with Your Own Voice](#trying-live-demo-with-your-own-voice) - [Experiments and Benchmarks](#experiments-and-benchmarks) +- [Released Models](#released-models) - [Questions and Help](#questions-and-help) ## Prerequisites @@ -22,7 +24,7 @@ ## Installation -Please install the [prerequisites](#prerequisites) above before moving onto this quick installation. +Please install the [prerequisites](#prerequisites) above before moving on. ``` git clone https://github.com/PaddlePaddle/models.git @@ -32,43 +34,43 @@ sh setup.sh ## Getting Started -Several shell scripts provided in `./examples` will help us to quickly give it a try, including training, inferencing, evaluation and demo deployment. +Several shell scripts provided in `./examples` will help us to quickly give it a try, for most major modules, including data preparation, model training, case inference, model evaluation and demo deployment, with a few public dataset (e.g. [LibriSpeech](http://www.openslr.org/12/), [Aishell](https://github.com/kaldi-asr/kaldi/tree/master/egs/aishell)). Reading these examples will also help us understand how to make it work with our own data. -Most of the scripts in `./examples` are configured with 8 GPUs. If you don't have 8 GPUs available, please modify `CUDA_VISIBLE_DEVICE` and `--trainer_count`. If you don't have any GPU available, please set `--use_gpu` to False. +Some of the scripts in `./examples` are configured with 8 GPUs. If you don't have 8 GPUs available, please modify `CUDA_VISIBLE_DEVICE` and `--trainer_count`. If you don't have any GPU available, please set `--use_gpu` to False to use CPUs instead. Let's take a tiny sampled subset of [LibriSpeech dataset](http://www.openslr.org/12/) for instance. - Go to directory ``` - cd examples/librispeech_tiny + cd examples/tiny ``` - Notice that this is only a toy example with a tiny sampled set of LibriSpeech. If we would like to try with the complete LibriSpeech (would take much a longer time for training), please go to `examples/librispeech` instead. -- Prepare the libripseech data + Notice that this is only a toy example with a tiny sampled subset of LibriSpeech. If we would like to try with the complete dataset (would take several days for training), please go to `examples/librispeech` instead. +- Prepare the data ``` - sh preprare_data.sh + sh run_data.sh ``` - `prepare_data.sh` downloads dataset, generates file manifests, collects normalizer' statitics and builds vocabulary for us. Once the running is done, we'll find our LibriSpeech data (not full in this "tiny" example) downloaded in `~/.cache/paddle/dataset/speech/Libri` and several manifest files as well as one mean stddev file generated in `./data/librispeech_tiny`, for the further model training. It needs to be run for only once. + `run_data.sh` will download dataset, generate manifests, collect normalizer' statitics and build vocabulary. Once the data preparation is done, we will find the data (only part of LibriSpeech) downloaded in `~/.cache/paddle/dataset/speech/libri` and the corresponding manifest files generated in `./data/tiny` as well as a mean stddev file and a vocabulary file. It has to be run for the very first time we run this dataset and is reusable for all further experiments. - Train your own ASR model ``` sh run_train.sh ``` - `run_train.sh` starts a training job, with training logs printed to stdout and model checkpoint of every pass/epoch saved to `./checkpoints`. We can resume the training from these checkpoints, or use them for inference, evalutiaton and deployment. + `run_train.sh` will start a training job, with training logs printed to stdout and model checkpoint of every pass/epoch saved to `./checkpoints/tiny`. We can resume the training from these checkpoints, or use them for inference, evalutiaton and deployment. - Case inference with an existing model ``` sh run_infer.sh ``` - `run_infer.sh` will quickly show us speech-to-text decoding results for several (default: 10) audio samples with an existing model. Since the model is only trained on a subset of LibriSpeech, the performance might not be very good. We can download a well-trained model and then do the inference: + `run_infer.sh` will show us some speech-to-text decoding results for several (default: 10) samples with the trained model. The performance might not be good now as the current model is only trained with a toy subset of LibriSpeech. To see the results with a better model, we can download a well-trained (trained for several days, with the complete LibriSpeech) model and do the inference: ``` - sh download_model_run_infer.sh + sh run_infer_golden.sh ``` - Evaluate an existing model @@ -76,14 +78,14 @@ Let's take a tiny sampled subset of [LibriSpeech dataset](http://www.openslr.org sh run_test.sh ``` - `run_test.sh` evaluates the model with Word Error Rate (or Character Error Rate) measurement. Similarly, we can also download a well-trained model and test its performance: + `run_test.sh` will evaluate the model with Word Error Rate (or Character Error Rate) measurement. Similarly, we can also download a well-trained model and test its performance: ``` - sh download_model_run_test.sh + sh run_test_golden.sh ``` - Try out a live demo with your own voice - Until now, we have trained and tested an ASR model quantitively and qualitatively with existing audios. But we haven't try the model with our own speech. `demo_server.sh` and `demo_client.sh` helps quickly build up a demo ASR engine with the trained model, enabling us to test and play around with the demo with our own voice. + Until now, we have trained and tested our ASR model qualitatively (`run_infer.sh`) and quantitively (`run_test.sh`) with existing audio files. But we have not yet play the model with our own speech. `demo_server.sh` and `demo_client.sh` helps quickly build up a demo ASR engine with the trained model, enabling us to test and play around with the demo with our own voice. We start the server in one console by entering: @@ -112,20 +114,20 @@ Wish you a happy journey with the DeepSpeech2 ASR engine! #### Generate Manifest -*DeepSpeech2 on PaddlePaddle* accepts a textual **manifest** file as its data set interface. A manifest file summarizes a set of speech data, with each line containing the meta data (e.g. filepath, transcription, duration) of one audio clip, in [JSON](http://www.json.org/) format, just as: +*DeepSpeech2 on PaddlePaddle* accepts a textual **manifest** file as its data set interface. A manifest file summarizes a set of speech data, with each line containing some meta data (e.g. filepath, transcription, duration) of one audio clip, in [JSON](http://www.json.org/) format, such as: ``` {"audio_filepath": "/home/work/.cache/paddle/Libri/134686/1089-134686-0001.flac", "duration": 3.275, "text": "stuff it into you his belly counselled him"} {"audio_filepath": "/home/work/.cache/paddle/Libri/134686/1089-134686-0007.flac", "duration": 4.275, "text": "a cold lucid indifference reigned in his soul"} ``` -To use any custom data, we only need to generate such manifest files to summarize the dataset. Given such summarized manifests, training, inference and all other modules can be aware of where to access the audio files, as well as their meta data including the transcription labels. +To use your custom data, you only need to generate such manifest files to summarize the dataset. Given such summarized manifests, training, inference and all other modules can be aware of where to access the audio files, as well as their meta data including the transcription labels. -For example script to generate such manifest files, please refer to `data/librispeech/librispeech.py`, which download and generate manifests for LibriSpeech dataset. +For how to generate such manifest files, please refer to `data/librispeech/librispeech.py`, which download and generate manifests for LibriSpeech dataset. #### Compute Mean & Stddev for Normalizer -To perform z-score normalization (zero-mean, unit stddev) upon audio features, we have to estimate in advance the mean and standard deviation of the features, with sampled training audios: +To perform z-score normalization (zero-mean, unit stddev) upon audio features, we have to estimate in advance the mean and standard deviation of the features, with some training samples: ``` python tools/compute_mean_std.py \ @@ -140,7 +142,7 @@ It will compute the mean and standard deviation of power spectgram feature with #### Build Vocabulary -A list of possible characters is required to convert the target transcription into list of token indices for training and in docoders convert from them back to text. Such a character-based vocabulary can be build with `tools/build_vocab.py`. +A vocabulary of possible characters is required to convert the transcription into a list of token indices for training, and in docoding, to convert from a list of indices back to text again. Such a character-based vocabulary can be build with `tools/build_vocab.py`. ``` python tools/build_vocab.py \ @@ -149,7 +151,7 @@ python tools/build_vocab.py \ --manifest_paths data/librispeech/manifest.train ``` -It will build a vocabuary file of `data/librispeeech/eng_vocab.txt` with all transcription text in `data/librispeech/manifest.train`, without character truncation. +It will write a vocabuary file `data/librispeeech/eng_vocab.txt` with all transcription text in `data/librispeech/manifest.train`, without vocabulary truncation (`--count_threshold 0`). #### More Help @@ -163,7 +165,7 @@ python tools/build_vocab.py --help ## Training a model -`train.py` is the main caller of the training module. We list several usage below. +`train.py` is the main caller of the training module. We show several examples of usage below. - Start training from scratch with 8 GPUs: @@ -176,7 +178,7 @@ python tools/build_vocab.py --help ``` python train.py --use_gpu False --trainer_count 16 ``` -- Resume training from a checkpoint (an existing model): +- Resume training from a checkpoint: ``` CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 python train.py \ @@ -188,11 +190,11 @@ For more help on arguments: ``` python train.py --help ``` -or refer to `example/librispeech/run_train.sh. +or refer to `example/librispeech/run_train.sh`. -#### Augment the Dataset for Training +## Data Augmentation Pipeline -Data augmentation has often been a highly effective technique to boost the deep learning performance. We augment our speech data by synthesizing new audios with small random perterbation (label-invariant transformation) added upon raw audios. We don't have to do the syntheses by ourselves, as it is already embeded into the data provider and is done on the fly, randomly for each epoch. +Data augmentation has often been a highly effective technique to boost the deep learning performance. We augment our speech data by synthesizing new audios with small random perterbation (label-invariant transformation) added upon raw audios. We don't have to do the syntheses by ourselves, as it is already embeded into the data provider and is done on the fly, randomly for each epoch during training. Six optional augmentation components are provided for us to configured and inserted into the processing pipeline. @@ -203,7 +205,7 @@ Six optional augmentation components are provided for us to configured and inser - Noise Perturbation (need background noise audio files) - Impulse Response (need impulse audio files) -In order to inform the trainer of what augmentation components we need and what their processing orders are, we are required to prepare a *augmentation configuration file* in JSON format. For example: +In order to inform the trainer of what augmentation components we need and what their processing orders are, we are required to prepare a *augmentation configuration file* in [JSON](http://www.json.org/) format. For example: ``` [{ @@ -220,23 +222,23 @@ In order to inform the trainer of what augmentation components we need and what }] ``` -When the `--augment_conf_file` argument of `trainer.py` is set to the path of the above example configuration file, each audio clip in each epoch will be processed: with 60% of chance, it will first be speed perturbed with a uniformly random sampled speed-rate between 0.95 and 1.05, and then with 80% of chance it will be shifted in time with a random sampled offset between -5 ms and 5 ms. Finally this newly synthesized audio clip will be feed into the feature extractor for further training. +When the `--augment_conf_file` argument of `trainer.py` is set to the path of the above example configuration file, every audio clip in every epoch will be processed: with 60% of chance, it will first be speed perturbed with a uniformly random sampled speed-rate between 0.95 and 1.05, and then with 80% of chance it will be shifted in time with a random sampled offset between -5 ms and 5 ms. Finally this newly synthesized audio clip will be feed into the feature extractor for further training. -For configuration examples, please refer to `conf/augmenatation.config.example`. +For other configuration examples, please refer to `conf/augmenatation.config.example`. -Be careful when we are utilizing the data augmentation technique, as improper augmentation will instead do harm to the training, due to the enlarged train-test gap. +Be careful when we are utilizing the data augmentation technique, as improper augmentation will do harm to the training, due to the enlarged train-test gap. ## Inference and Evaluation -#### Prepare Language Model +### Prepare Language Model -A language model is required to improve the decoder's performance. We have prepared two language models (with lossy compression) for users to download and try. One is for English and the other is for Mandarin. Please refer to `examples/librispeech/download_model.sh` and `examples/mandarin_demo/download_model.sh` for their urls. If you wish to train your own better language model, please refer to [KenLM](https://github.com/kpu/kenlm) for tutorials. +A language model is required to improve the decoder's performance. We have prepared two language models (with lossy compression) for users to download and try. One is for English and the other is for Mandarin. Please refer to `models/lm/download_lm_en.sh` and `models/lm/download_lm_zh.sh` for their urls. If you wish to train your own better language model, please refer to [KenLM](https://github.com/kpu/kenlm) for tutorials. TODO: any other requirements or tips to add? -#### Speech-to-text Inference +### Speech-to-text Inference -We provide a inference module `infer.py` to infer, decode and visualize speech-to-text results for several given audio clips, which might help to have a intuitive and qualitative evaluation of the ASR model performance. +We provide a inference module `infer.py` to infer, decode and visualize speech-to-text results for several given audio clips. It might help us to have a intuitive and qualitative evaluation of the ASR model's performance. - Inference with GPU: @@ -247,21 +249,21 @@ We provide a inference module `infer.py` to infer, decode and visualize speech-t - Inference with CPU: ``` - python infer.py --use_gpu False + python infer.py --use_gpu False --trainer_count 12 ``` -We provide two CTC decoders: *CTC greedy decoder* and *CTC beam search decoder*. The *CTC greedy decoder* is an implementation of the simple best-path decoding algorithm, selecting at each timestep the most likely token, thus being greedy and locally optimal. The [*CTC beam search decoder*](https://arxiv.org/abs/1408.2873) otherwise utilzied a heuristic breadth-first gragh search for arriving at a near global optimality; it requires a pre-trained KenLM language model for better scoring and ranking sentences. The decoder type can be set with argument `--decoding_method`. +We provide two types of CTC decoders: *CTC greedy decoder* and *CTC beam search decoder*. The *CTC greedy decoder* is an implementation of the simple best-path decoding algorithm, selecting at each timestep the most likely token, thus being greedy and locally optimal. The [*CTC beam search decoder*](https://arxiv.org/abs/1408.2873) otherwise utilizes a heuristic breadth-first gragh search for reaching a near global optimality; it also requires a pre-trained KenLM language model for better scoring and ranking. The decoder type can be set with argument `--decoding_method`. For more help on arguments: ``` python infer.py --help ``` -or refer to `example/librispeech/run_infer.sh. +or refer to `example/librispeech/run_infer.sh`. -#### Evaluate a Model +### Evaluate a Model -To evaluate a model quantitively, we can run: +To evaluate a model's performance quantitively, we can run: - Evaluation with GPU: @@ -272,23 +274,23 @@ To evaluate a model quantitively, we can run: - Evaluation with CPU: ``` - python test.py --use_gpu False + python test.py --use_gpu False --trainer_count 12 ``` -The error rate (default: word error rate, can be set with `--error_rate_type`) will be printed. +The error rate (default: word error rate; can be set with `--error_rate_type`) will be printed. For more help on arguments: ``` python test.py --help ``` -or refer to `example/librispeech/run_test.sh. +or refer to `example/librispeech/run_test.sh`. ## Hyper-parameters Tuning -The hyper-parameters $\alpha$ (coefficient for language model scorer) and $\beta$ (coefficient for word count scorer) for the [*CTC beam search decoder*](https://arxiv.org/abs/1408.2873) often have a significant impact on the decoder's performance. It'd be better to re-tune them on validation samples after the accustic model is renewed. +The hyper-parameters $\alpha$ (coefficient for language model scorer) and $\beta$ (coefficient for word count scorer) for the [*CTC beam search decoder*](https://arxiv.org/abs/1408.2873) often have a significant impact on the decoder's performance. It would be better to re-tune them on a validation set when the accustic model is renewed. -`tools/tune.py` performs a 2-D grid search over the hyper-parameter $\alpha$ and $\beta$. We have to provide the range of $\alpha$ and $\beta$, as well as the number of attempts. +`tools/tune.py` performs a 2-D grid search over the hyper-parameter $\alpha$ and $\beta$. We have to provide the range of $\alpha$ and $\beta$, as well as the number of their attempts. - Tuning with GPU: @@ -309,12 +311,12 @@ The hyper-parameters $\alpha$ (coefficient for language model scorer) and $\beta python tools/tune.py --use_gpu False ``` -After tuning, we can reset $\alpha$ and $\beta$ in the inference and evaluation modules to see if they can really improve the ASR performance. +After tuning, we can reset $\alpha$ and $\beta$ in the inference and evaluation modules to see if they really help improve the ASR performance. ``` python tune.py --help ``` -or refer to `example/librispeech/run_tune.sh. +or refer to `example/librispeech/run_tune.sh`. TODO: add figure. @@ -352,4 +354,6 @@ It could be possible to start the server and the client in two seperate machines ## Experiments and Benchmarks +## Released Models + ## Questions and Help diff --git a/data/librispeech/eng_vocab.txt b/data/librispeech/eng_vocab.txt deleted file mode 100644 index 8268f3f33..000000000 --- a/data/librispeech/eng_vocab.txt +++ /dev/null @@ -1,28 +0,0 @@ -' - -a -b -c -d -e -f -g -h -i -j -k -l -m -n -o -p -q -r -s -t -u -v -w -x -y -z diff --git a/data/librispeech/librispeech.py b/data/librispeech/librispeech.py index 14a3804e2..e2ad8d413 100644 --- a/data/librispeech/librispeech.py +++ b/data/librispeech/librispeech.py @@ -19,8 +19,6 @@ import json import codecs from paddle.v2.dataset.common import md5file -DATA_HOME = os.path.expanduser('~/.cache/paddle/dataset/speech') - URL_ROOT = "http://www.openslr.org/resources/12" URL_TEST_CLEAN = URL_ROOT + "/test-clean.tar.gz" URL_TEST_OTHER = URL_ROOT + "/test-other.tar.gz" @@ -41,7 +39,7 @@ MD5_TRAIN_OTHER_500 = "d1a0fd59409feb2c614ce4d30c387708" parser = argparse.ArgumentParser(description=__doc__) parser.add_argument( "--target_dir", - default=DATA_HOME + "/libri", + default='~/.cache/paddle/dataset/speech/libri', type=str, help="Directory to save the dataset. (default: %(default)s)") parser.add_argument( @@ -60,14 +58,14 @@ args = parser.parse_args() def download(url, md5sum, target_dir): - """ - Download file from url to target_dir, and check md5sum. + """Download file from url to target_dir, and check md5sum. """ if not os.path.exists(target_dir): os.makedirs(target_dir) filepath = os.path.join(target_dir, url.split("/")[-1]) if not (os.path.exists(filepath) and md5file(filepath) == md5sum): print("Downloading %s ..." % url) - os.system("wget -c " + url + " -P " + target_dir) + ret = os.system("wget -c " + url + " -P " + target_dir) + print(ret) print("\nMD5 Chesksum %s ..." % filepath) if not md5file(filepath) == md5sum: raise RuntimeError("MD5 checksum failed.") @@ -77,8 +75,7 @@ def download(url, md5sum, target_dir): def unpack(filepath, target_dir): - """ - Unpack the file to the target_dir. + """Unpack the file to the target_dir. """ print("Unpacking %s ..." % filepath) tar = tarfile.open(filepath) @@ -87,8 +84,7 @@ def unpack(filepath, target_dir): def create_manifest(data_dir, manifest_path): - """ - Create a manifest json file summarizing the data set, with each line + """Create a manifest json file summarizing the data set, with each line containing the meta data (i.e. audio filepath, transcription text, audio duration) of each audio file within the data set. """ @@ -119,8 +115,7 @@ def create_manifest(data_dir, manifest_path): def prepare_dataset(url, md5sum, target_dir, manifest_path): - """ - Download, unpack and create summmary manifest file. + """Download, unpack and create summmary manifest file. """ if not os.path.exists(os.path.join(target_dir, "LibriSpeech")): # download @@ -135,6 +130,8 @@ def prepare_dataset(url, md5sum, target_dir, manifest_path): def main(): + args.target_dir = os.path.expanduser(args.target_dir) + prepare_dataset( url=URL_TEST_CLEAN, md5sum=MD5_TEST_CLEAN, @@ -145,12 +142,12 @@ def main(): md5sum=MD5_DEV_CLEAN, target_dir=os.path.join(args.target_dir, "dev-clean"), manifest_path=args.manifest_prefix + ".dev-clean") - prepare_dataset( - url=URL_TRAIN_CLEAN_100, - md5sum=MD5_TRAIN_CLEAN_100, - target_dir=os.path.join(args.target_dir, "train-clean-100"), - manifest_path=args.manifest_prefix + ".train-clean-100") if args.full_download: + prepare_dataset( + url=URL_TRAIN_CLEAN_100, + md5sum=MD5_TRAIN_CLEAN_100, + target_dir=os.path.join(args.target_dir, "train-clean-100"), + manifest_path=args.manifest_prefix + ".train-clean-100") prepare_dataset( url=URL_TEST_OTHER, md5sum=MD5_TEST_OTHER, diff --git a/deploy/demo_server.py b/deploy/demo_server.py index 658b14197..2d3931f74 100644 --- a/deploy/demo_server.py +++ b/deploy/demo_server.py @@ -11,7 +11,7 @@ import wave import paddle.v2 as paddle import _init_paths from data_utils.data import DataGenerator -from models.model import DeepSpeech2Model +from model_utils.model import DeepSpeech2Model from data_utils.utils import read_manifest from utils.utility import add_arguments, print_arguments diff --git a/examples/librispeech/prepare_data.sh b/examples/librispeech/run_data.sh similarity index 57% rename from examples/librispeech/prepare_data.sh rename to examples/librispeech/run_data.sh index 6e9997703..f65aa233b 100644 --- a/examples/librispeech/prepare_data.sh +++ b/examples/librispeech/run_data.sh @@ -1,26 +1,31 @@ #! /usr/bin/bash -pushd ../.. +pushd ../.. > /dev/null # download data, generate manifests python data/librispeech/librispeech.py \ --manifest_prefix='data/librispeech/manifest' \ ---full_download='True' \ ---target_dir='~/.cache/paddle/dataset/speech/Libri' +--target_dir='~/.cache/paddle/dataset/speech/Libri' \ +--full_download='True' if [ $? -ne 0 ]; then echo "Prepare LibriSpeech failed. Terminated." exit 1 fi -cat data/librispeech/manifest.train* | shuf > data/librispeech/manifest.train +cat data/librispeech/manifest.train-* | shuf > data/librispeech/manifest.train -# build vocabulary (can be skipped for English, as already provided) -# python tools/build_vocab.py \ -# --count_threshold=0 \ -# --vocab_path='data/librispeech/eng_vocab.txt' \ -# --manifest_paths='data/librispeech/manifeset.train' +# build vocabulary +python tools/build_vocab.py \ +--count_threshold=0 \ +--vocab_path='data/librispeech/vocab.txt' \ +--manifest_paths='data/librispeech/manifest.train' + +if [ $? -ne 0 ]; then + echo "Build vocabulary failed. Terminated." + exit 1 +fi # compute mean and stddev for normalizer @@ -37,3 +42,4 @@ fi echo "LibriSpeech Data preparation done." +exit 0 diff --git a/examples/librispeech/run_infer.sh b/examples/librispeech/run_infer.sh index 619d546e8..6b790502a 100644 --- a/examples/librispeech/run_infer.sh +++ b/examples/librispeech/run_infer.sh @@ -1,13 +1,23 @@ #! /usr/bin/bash -pushd ../.. +pushd ../.. > /dev/null +# download language model +pushd models/lm > /dev/null +sh download_lm_en.sh +if [ $? -ne 0 ]; then + exit 1 +fi +popd > /dev/null + + +# infer CUDA_VISIBLE_DEVICES=0 \ python -u infer.py \ --num_samples=10 \ --trainer_count=1 \ --beam_size=500 \ ---num_proc_bsearch=12 \ +--num_proc_bsearch=8 \ --num_conv_layers=2 \ --num_rnn_layers=3 \ --rnn_layer_size=2048 \ @@ -17,11 +27,19 @@ python -u infer.py \ --use_gru=False \ --use_gpu=True \ --share_rnn_weights=True \ ---infer_manifest='data/librispeech/manifest.dev-clean' \ +--infer_manifest='data/librispeech/manifest.test-clean' \ --mean_std_path='data/librispeech/mean_std.npz' \ ---vocab_path='data/librispeech/eng_vocab.txt' \ ---model_path='checkpoints/params.latest.tar.gz' \ ---lang_model_path='lm/data/common_crawl_00.prune01111.trie.klm' \ +--vocab_path='data/librispeech/vocab.txt' \ +--model_path='checkpoints/libri/params.latest.tar.gz' \ +--lang_model_path='models/lm/common_crawl_00.prune01111.trie.klm' \ --decoding_method='ctc_beam_search' \ --error_rate_type='wer' \ --specgram_type='linear' + +if [ $? -ne 0 ]; then + echo "Failed in inference!" + exit 1 +fi + + +exit 0 diff --git a/examples/librispeech/run_infer_golden.sh b/examples/librispeech/run_infer_golden.sh new file mode 100644 index 000000000..32e9d8623 --- /dev/null +++ b/examples/librispeech/run_infer_golden.sh @@ -0,0 +1,54 @@ +#! /usr/bin/bash + +pushd ../.. > /dev/null + +# download language model +pushd models/lm > /dev/null +sh download_lm_en.sh +if [ $? -ne 0 ]; then + exit 1 +fi +popd > /dev/null + + +# download well-trained model +pushd models/librispeech > /dev/null +sh download_model.sh +if [ $? -ne 0 ]; then + exit 1 +fi +popd > /dev/null + + +# infer +CUDA_VISIBLE_DEVICES=0 \ +python -u infer.py \ +--num_samples=10 \ +--trainer_count=1 \ +--beam_size=500 \ +--num_proc_bsearch=8 \ +--num_conv_layers=2 \ +--num_rnn_layers=3 \ +--rnn_layer_size=2048 \ +--alpha=0.36 \ +--beta=0.25 \ +--cutoff_prob=0.99 \ +--use_gru=False \ +--use_gpu=True \ +--share_rnn_weights=True \ +--infer_manifest='data/tiny/manifest.test-clean' \ +--mean_std_path='models/librispeech/mean_std.npz' \ +--vocab_path='models/librispeech/vocab.txt' \ +--model_path='models/librispeech/params.tar.gz' \ +--lang_model_path='models/lm/common_crawl_00.prune01111.trie.klm' \ +--decoding_method='ctc_beam_search' \ +--error_rate_type='wer' \ +--specgram_type='linear' + +if [ $? -ne 0 ]; then + echo "Failed in inference!" + exit 1 +fi + + +exit 0 diff --git a/examples/librispeech/run_test.sh b/examples/librispeech/run_test.sh index 5a14cb682..9709234ab 100644 --- a/examples/librispeech/run_test.sh +++ b/examples/librispeech/run_test.sh @@ -1,14 +1,24 @@ #! /usr/bin/bash -pushd ../.. +pushd ../.. > /dev/null +# download language model +pushd models/lm > /dev/null +sh download_lm_en.sh +if [ $? -ne 0 ]; then + exit 1 +fi +popd > /dev/null + + +# evaluate model CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 \ -python -u evaluate.py \ +python -u test.py \ --batch_size=128 \ --trainer_count=8 \ --beam_size=500 \ ---num_proc_bsearch=12 \ ---num_proc_data=12 \ +--num_proc_bsearch=8 \ +--num_proc_data=4 \ --num_conv_layers=2 \ --num_rnn_layers=3 \ --rnn_layer_size=2048 \ @@ -20,9 +30,17 @@ python -u evaluate.py \ --share_rnn_weights=True \ --test_manifest='data/librispeech/manifest.test-clean' \ --mean_std_path='data/librispeech/mean_std.npz' \ ---vocab_path='data/librispeech/eng_vocab.txt' \ ---model_path='checkpoints/params.latest.tar.gz' \ ---lang_model_path='lm/data/common_crawl_00.prune01111.trie.klm' \ +--vocab_path='data/librispeech/vocab.txt' \ +--model_path='checkpoints/libri/params.latest.tar.gz' \ +--lang_model_path='models/lm/common_crawl_00.prune01111.trie.klm' \ --decoding_method='ctc_beam_search' \ --error_rate_type='wer' \ --specgram_type='linear' + +if [ $? -ne 0 ]; then + echo "Failed in evaluation!" + exit 1 +fi + + +exit 0 diff --git a/examples/librispeech/run_test_golden.sh b/examples/librispeech/run_test_golden.sh new file mode 100644 index 000000000..080c3c062 --- /dev/null +++ b/examples/librispeech/run_test_golden.sh @@ -0,0 +1,55 @@ +#! /usr/bin/bash + +pushd ../.. > /dev/null + +# download language model +pushd models/lm > /dev/null +sh download_lm_en.sh +if [ $? -ne 0 ]; then + exit 1 +fi +popd > /dev/null + + +# download well-trained model +pushd models/librispeech > /dev/null +sh download_model.sh +if [ $? -ne 0 ]; then + exit 1 +fi +popd > /dev/null + + +# evaluate model +CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 \ +python -u test.py \ +--batch_size=128 \ +--trainer_count=8 \ +--beam_size=500 \ +--num_proc_bsearch=8 \ +--num_proc_data=4 \ +--num_conv_layers=2 \ +--num_rnn_layers=3 \ +--rnn_layer_size=2048 \ +--alpha=0.36 \ +--beta=0.25 \ +--cutoff_prob=0.99 \ +--use_gru=False \ +--use_gpu=True \ +--share_rnn_weights=True \ +--test_manifest='data/tiny/manifest.test-clean' \ +--mean_std_path='models/librispeech/mean_std.npz' \ +--vocab_path='models/librispeech/vocab.txt' \ +--model_path='models/librispeech/params.tar.gz' \ +--lang_model_path='models/lm/common_crawl_00.prune01111.trie.klm' \ +--decoding_method='ctc_beam_search' \ +--error_rate_type='wer' \ +--specgram_type='linear' + +if [ $? -ne 0 ]; then + echo "Failed in evaluation!" + exit 1 +fi + + +exit 0 diff --git a/examples/librispeech/run_train.sh b/examples/librispeech/run_train.sh index 14672167c..5485475e9 100644 --- a/examples/librispeech/run_train.sh +++ b/examples/librispeech/run_train.sh @@ -1,10 +1,11 @@ #! /usr/bin/bash -pushd ../.. +pushd ../.. > /dev/null +# train model CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 \ python -u train.py \ ---batch_size=256 \ +--batch_size=512 \ --trainer_count=8 \ --num_passes=50 \ --num_proc_data=12 \ @@ -23,8 +24,16 @@ python -u train.py \ --train_manifest='data/librispeech/manifest.train' \ --dev_manifest='data/librispeech/manifest.dev' \ --mean_std_path='data/librispeech/mean_std.npz' \ ---vocab_path='data/librispeech/eng_vocab.txt' \ ---output_model_dir='./checkpoints' \ +--vocab_path='data/librispeech/vocab.txt' \ +--output_model_dir='./checkpoints/libri' \ --augment_conf_path='conf/augmentation.config' \ --specgram_type='linear' \ --shuffle_method='batch_shuffle_clipped' + +if [ $? -ne 0 ]; then + echo "Failed in training!" + exit 1 +fi + + +exit 0 diff --git a/examples/librispeech/run_tune.sh b/examples/librispeech/run_tune.sh index 9d992e884..05c024bec 100644 --- a/examples/librispeech/run_tune.sh +++ b/examples/librispeech/run_tune.sh @@ -1,7 +1,8 @@ #! /usr/bin/bash -pushd ../.. +pushd ../.. > /dev/null +# grid-search for hyper-parameters in language model CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 \ python -u tools/tune.py \ --num_samples=100 \ @@ -23,8 +24,16 @@ python -u tools/tune.py \ --share_rnn_weights=True \ --tune_manifest='data/librispeech/manifest.dev-clean' \ --mean_std_path='data/librispeech/mean_std.npz' \ ---vocab_path='data/librispeech/eng_vocab.txt' \ ---model_path='checkpoints/params.latest.tar.gz' \ ---lang_model_path='lm/data/common_crawl_00.prune01111.trie.klm' \ +--vocab_path='data/librispeech/vocab.txt' \ +--model_path='checkpoints/libri/params.latest.tar.gz' \ +--lang_model_path='models/lm/common_crawl_00.prune01111.trie.klm' \ --error_rate_type='wer' \ --specgram_type='linear' + +if [ $? -ne 0 ]; then + echo "Failed in tuning!" + exit 1 +fi + + +exit 0 diff --git a/examples/mandarin/run_demo_client.sh b/examples/mandarin/run_demo_client.sh new file mode 100644 index 000000000..dfde20f88 --- /dev/null +++ b/examples/mandarin/run_demo_client.sh @@ -0,0 +1,17 @@ +#! /usr/bin/bash + +pushd ../.. > /dev/null + +# start demo client +CUDA_VISIBLE_DEVICES=0 \ +python -u deploy/demo_client.py \ +--host_ip='localhost' \ +--host_port=8086 \ + +if [ $? -ne 0 ]; then + echo "Failed in starting demo client!" + exit 1 +fi + + +exit 0 diff --git a/examples/mandarin/run_demo_server.sh b/examples/mandarin/run_demo_server.sh new file mode 100644 index 000000000..703184a6b --- /dev/null +++ b/examples/mandarin/run_demo_server.sh @@ -0,0 +1,53 @@ +#! /usr/bin/bash +# TODO: replace the model with a mandarin model + +pushd ../.. > /dev/null + +# download language model +pushd models/lm > /dev/null +sh download_lm_en.sh +if [ $? -ne 0 ]; then + exit 1 +fi +popd > /dev/null + + +# download well-trained model +pushd models/librispeech > /dev/null +sh download_model.sh +if [ $? -ne 0 ]; then + exit 1 +fi +popd > /dev/null + + +# start demo server +CUDA_VISIBLE_DEVICES=0 \ +python -u deploy/demo_server.py \ +--host_ip='localhost' \ +--host_port=8086 \ +--num_conv_layers=2 \ +--num_rnn_layers=3 \ +--rnn_layer_size=2048 \ +--alpha=0.36 \ +--beta=0.25 \ +--cutoff_prob=0.99 \ +--use_gru=False \ +--use_gpu=True \ +--share_rnn_weights=True \ +--speech_save_dir='demo_cache' \ +--warmup_manifest='data/tiny/manifest.test-clean' \ +--mean_std_path='models/librispeech/mean_std.npz' \ +--vocab_path='models/librispeech/vocab.txt' \ +--model_path='models/librispeech/params.tar.gz' \ +--lang_model_path='models/lm/common_crawl_00.prune01111.trie.klm' \ +--decoding_method='ctc_beam_search' \ +--specgram_type='linear' + +if [ $? -ne 0 ]; then + echo "Failed in starting demo server!" + exit 1 +fi + + +exit 0 diff --git a/examples/tiny/run_data.sh b/examples/tiny/run_data.sh index 44345d8cc..203d3e2c8 100644 --- a/examples/tiny/run_data.sh +++ b/examples/tiny/run_data.sh @@ -1,27 +1,26 @@ #! /usr/bin/bash -pushd ../.. +pushd ../.. > /dev/null # download data, generate manifests -python data/tiny/tiny.py \ +python data/librispeech/librispeech.py \ --manifest_prefix='data/tiny/manifest' \ ---target_dir=$HOME'/.cache/paddle/dataset/speech/tiny' +--target_dir='~/.cache/paddle/dataset/speech/libri' \ +--full_download='False' if [ $? -ne 0 ]; then echo "Prepare LibriSpeech failed. Terminated." exit 1 fi -cat data/tiny/manifest.dev-clean | head -n 32 > data/tiny/manifest.train -cat data/tiny/manifest.dev-clean | head -n 48 | tail -n 16 > data/tiny/manifest.dev -cat data/tiny/manifest.dev-clean | head -n 64 | tail -n 16 > data/tiny/manifest.test +head -n 64 data/tiny/manifest.dev-clean > data/tiny/manifest.tiny # build vocabulary python tools/build_vocab.py \ --count_threshold=0 \ --vocab_path='data/tiny/vocab.txt' \ ---manifest_paths='data/tiny/manifest.train' +--manifest_paths='data/tiny/manifest.dev' if [ $? -ne 0 ]; then echo "Build vocabulary failed. Terminated." @@ -31,8 +30,8 @@ fi # compute mean and stddev for normalizer python tools/compute_mean_std.py \ ---manifest_path='data/tiny/manifest.train' \ ---num_samples=32 \ +--manifest_path='data/tiny/manifest.tiny' \ +--num_samples=64 \ --specgram_type='linear' \ --output_path='data/tiny/mean_std.npz' @@ -43,3 +42,4 @@ fi echo "Tiny data preparation done." +exit 0 diff --git a/examples/tiny/run_infer.sh b/examples/tiny/run_infer.sh index f09bc6638..1d33bfbba 100644 --- a/examples/tiny/run_infer.sh +++ b/examples/tiny/run_infer.sh @@ -1,13 +1,23 @@ #! /usr/bin/bash -pushd ../.. +pushd ../.. > /dev/null +# download language model +pushd models/lm > /dev/null +sh download_lm_en.sh +if [ $? -ne 0 ]; then + exit 1 +fi +popd > /dev/null + + +# infer CUDA_VISIBLE_DEVICES=0 \ python -u infer.py \ ---num_samples=4 \ +--num_samples=10 \ --trainer_count=1 \ --beam_size=500 \ ---num_proc_bsearch=12 \ +--num_proc_bsearch=8 \ --num_conv_layers=2 \ --num_rnn_layers=3 \ --rnn_layer_size=2048 \ @@ -17,11 +27,19 @@ python -u infer.py \ --use_gru=False \ --use_gpu=True \ --share_rnn_weights=True \ ---infer_manifest='data/tiny/manifest.train' \ +--infer_manifest='data/tiny/manifest.tiny' \ --mean_std_path='data/tiny/mean_std.npz' \ --vocab_path='data/tiny/vocab.txt' \ ---model_path='checkpoints/params.pass-14.tar.gz' \ +--model_path='checkpoints/tiny/params.pass-19.tar.gz' \ --lang_model_path='models/lm/common_crawl_00.prune01111.trie.klm' \ --decoding_method='ctc_beam_search' \ --error_rate_type='wer' \ --specgram_type='linear' + +if [ $? -ne 0 ]; then + echo "Failed in inference!" + exit 1 +fi + + +exit 0 diff --git a/examples/tiny/run_infer_golden.sh b/examples/tiny/run_infer_golden.sh new file mode 100644 index 000000000..32e9d8623 --- /dev/null +++ b/examples/tiny/run_infer_golden.sh @@ -0,0 +1,54 @@ +#! /usr/bin/bash + +pushd ../.. > /dev/null + +# download language model +pushd models/lm > /dev/null +sh download_lm_en.sh +if [ $? -ne 0 ]; then + exit 1 +fi +popd > /dev/null + + +# download well-trained model +pushd models/librispeech > /dev/null +sh download_model.sh +if [ $? -ne 0 ]; then + exit 1 +fi +popd > /dev/null + + +# infer +CUDA_VISIBLE_DEVICES=0 \ +python -u infer.py \ +--num_samples=10 \ +--trainer_count=1 \ +--beam_size=500 \ +--num_proc_bsearch=8 \ +--num_conv_layers=2 \ +--num_rnn_layers=3 \ +--rnn_layer_size=2048 \ +--alpha=0.36 \ +--beta=0.25 \ +--cutoff_prob=0.99 \ +--use_gru=False \ +--use_gpu=True \ +--share_rnn_weights=True \ +--infer_manifest='data/tiny/manifest.test-clean' \ +--mean_std_path='models/librispeech/mean_std.npz' \ +--vocab_path='models/librispeech/vocab.txt' \ +--model_path='models/librispeech/params.tar.gz' \ +--lang_model_path='models/lm/common_crawl_00.prune01111.trie.klm' \ +--decoding_method='ctc_beam_search' \ +--error_rate_type='wer' \ +--specgram_type='linear' + +if [ $? -ne 0 ]; then + echo "Failed in inference!" + exit 1 +fi + + +exit 0 diff --git a/examples/tiny/run_test.sh b/examples/tiny/run_test.sh index 5a14cb682..f9c3cc11c 100644 --- a/examples/tiny/run_test.sh +++ b/examples/tiny/run_test.sh @@ -1,14 +1,24 @@ #! /usr/bin/bash -pushd ../.. +pushd ../.. > /dev/null +# download language model +pushd models/lm > /dev/null +sh download_lm_en.sh +if [ $? -ne 0 ]; then + exit 1 +fi +popd > /dev/null + + +# evaluate model CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 \ -python -u evaluate.py \ ---batch_size=128 \ +python -u test.py \ +--batch_size=16 \ --trainer_count=8 \ --beam_size=500 \ ---num_proc_bsearch=12 \ ---num_proc_data=12 \ +--num_proc_bsearch=8 \ +--num_proc_data=4 \ --num_conv_layers=2 \ --num_rnn_layers=3 \ --rnn_layer_size=2048 \ @@ -18,11 +28,19 @@ python -u evaluate.py \ --use_gru=False \ --use_gpu=True \ --share_rnn_weights=True \ ---test_manifest='data/librispeech/manifest.test-clean' \ ---mean_std_path='data/librispeech/mean_std.npz' \ ---vocab_path='data/librispeech/eng_vocab.txt' \ ---model_path='checkpoints/params.latest.tar.gz' \ ---lang_model_path='lm/data/common_crawl_00.prune01111.trie.klm' \ +--test_manifest='data/tiny/manifest.tiny' \ +--mean_std_path='data/tiny/mean_std.npz' \ +--vocab_path='data/tiny/vocab.txt' \ +--model_path='checkpoints/params.pass-19.tar.gz' \ +--lang_model_path='models/lm/common_crawl_00.prune01111.trie.klm' \ --decoding_method='ctc_beam_search' \ --error_rate_type='wer' \ --specgram_type='linear' + +if [ $? -ne 0 ]; then + echo "Failed in evaluation!" + exit 1 +fi + + +exit 0 diff --git a/examples/tiny/run_test_golden.sh b/examples/tiny/run_test_golden.sh new file mode 100644 index 000000000..080c3c062 --- /dev/null +++ b/examples/tiny/run_test_golden.sh @@ -0,0 +1,55 @@ +#! /usr/bin/bash + +pushd ../.. > /dev/null + +# download language model +pushd models/lm > /dev/null +sh download_lm_en.sh +if [ $? -ne 0 ]; then + exit 1 +fi +popd > /dev/null + + +# download well-trained model +pushd models/librispeech > /dev/null +sh download_model.sh +if [ $? -ne 0 ]; then + exit 1 +fi +popd > /dev/null + + +# evaluate model +CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 \ +python -u test.py \ +--batch_size=128 \ +--trainer_count=8 \ +--beam_size=500 \ +--num_proc_bsearch=8 \ +--num_proc_data=4 \ +--num_conv_layers=2 \ +--num_rnn_layers=3 \ +--rnn_layer_size=2048 \ +--alpha=0.36 \ +--beta=0.25 \ +--cutoff_prob=0.99 \ +--use_gru=False \ +--use_gpu=True \ +--share_rnn_weights=True \ +--test_manifest='data/tiny/manifest.test-clean' \ +--mean_std_path='models/librispeech/mean_std.npz' \ +--vocab_path='models/librispeech/vocab.txt' \ +--model_path='models/librispeech/params.tar.gz' \ +--lang_model_path='models/lm/common_crawl_00.prune01111.trie.klm' \ +--decoding_method='ctc_beam_search' \ +--error_rate_type='wer' \ +--specgram_type='linear' + +if [ $? -ne 0 ]; then + echo "Failed in evaluation!" + exit 1 +fi + + +exit 0 diff --git a/examples/tiny/run_train.sh b/examples/tiny/run_train.sh index 7ca336876..c66ec4e56 100644 --- a/examples/tiny/run_train.sh +++ b/examples/tiny/run_train.sh @@ -1,18 +1,19 @@ #! /usr/bin/bash -pushd ../.. +pushd ../.. > /dev/null -CUDA_VISIBLE_DEVICES=0,1 \ +# train model +CUDA_VISIBLE_DEVICES=0,1,2,3 \ python -u train.py \ ---batch_size=2 \ ---trainer_count=1 \ ---num_passes=10 \ +--batch_size=16 \ +--trainer_count=4 \ +--num_passes=20 \ --num_proc_data=1 \ --num_conv_layers=2 \ --num_rnn_layers=3 \ --rnn_layer_size=2048 \ --num_iter_print=100 \ ---learning_rate=5e-5 \ +--learning_rate=1e-5 \ --max_duration=27.0 \ --min_duration=0.0 \ --use_sortagrad=True \ @@ -20,11 +21,19 @@ python -u train.py \ --use_gpu=True \ --is_local=True \ --share_rnn_weights=True \ ---train_manifest='data/tiny/manifest.train' \ ---dev_manifest='data/tiny/manifest.train' \ +--train_manifest='data/tiny/manifest.tiny' \ +--dev_manifest='data/tiny/manifest.tiny' \ --mean_std_path='data/tiny/mean_std.npz' \ --vocab_path='data/tiny/vocab.txt' \ ---output_model_dir='./checkpoints' \ +--output_model_dir='./checkpoints/tiny' \ --augment_conf_path='conf/augmentation.config' \ --specgram_type='linear' \ --shuffle_method='batch_shuffle_clipped' + +if [ $? -ne 0 ]; then + echo "Fail to do inference!" + exit 1 +fi + + +exit 0 diff --git a/examples/tiny/run_tune.sh b/examples/tiny/run_tune.sh index 9d992e884..360c11d59 100644 --- a/examples/tiny/run_tune.sh +++ b/examples/tiny/run_tune.sh @@ -1,7 +1,8 @@ #! /usr/bin/bash -pushd ../.. +pushd ../.. > /dev/null +# grid-search for hyper-parameters in language model CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 \ python -u tools/tune.py \ --num_samples=100 \ @@ -21,10 +22,18 @@ python -u tools/tune.py \ --use_gru=False \ --use_gpu=True \ --share_rnn_weights=True \ ---tune_manifest='data/librispeech/manifest.dev-clean' \ ---mean_std_path='data/librispeech/mean_std.npz' \ ---vocab_path='data/librispeech/eng_vocab.txt' \ ---model_path='checkpoints/params.latest.tar.gz' \ ---lang_model_path='lm/data/common_crawl_00.prune01111.trie.klm' \ +--tune_manifest='data/tiny/manifest.tiny' \ +--mean_std_path='data/tiny/mean_std.npz' \ +--vocab_path='data/tiny/vocab.txt' \ +--model_path='checkpoints/params.pass-9.tar.gz' \ +--lang_model_path='models/lm/common_crawl_00.prune01111.trie.klm' \ --error_rate_type='wer' \ --specgram_type='linear' + +if [ $? -ne 0 ]; then + echo "Failed in tuning!" + exit 1 +fi + + +exit 0 diff --git a/models/librispeech/download_model.sh b/models/librispeech/download_model.sh new file mode 100644 index 000000000..4408f6c1c --- /dev/null +++ b/models/librispeech/download_model.sh @@ -0,0 +1,20 @@ +#! /usr/bin/bash + +source ../../utils/utility.sh + +# TODO: add urls +URL='to-be-added' +MD5=5b4af224b26c1dc4dd972b7d32f2f52a +TARGET=./librispeech_model.tar.gz + + +echo "Download LibriSpeech model ..." +download $URL $MD5 $TARGET +if [ $? -ne 0 ]; then + echo "Fail to download LibriSpeech model!" + exit 1 +fi +tar -zxvf $TARGET + + +exit 0 diff --git a/models/lm/download_en.sh b/models/lm/download_en.sh deleted file mode 100644 index 5ca33c679..000000000 --- a/models/lm/download_en.sh +++ /dev/null @@ -1,16 +0,0 @@ -echo "Downloading language model ..." - -mkdir data - -LM=common_crawl_00.prune01111.trie.klm -MD5="099a601759d467cd0a8523ff939819c5" - -wget -c http://paddlepaddle.bj.bcebos.com/model_zoo/speech/$LM -P ./data - -echo "Checking md5sum ..." -md5_tmp=`md5sum ./data/$LM | awk -F[' '] '{print $1}'` - -if [ $MD5 != $md5_tmp ]; then - echo "Fail to download the language model!" - exit 1 -fi diff --git a/models/lm/download_lm_en.sh b/models/lm/download_lm_en.sh new file mode 100644 index 000000000..e967e25dc --- /dev/null +++ b/models/lm/download_lm_en.sh @@ -0,0 +1,18 @@ +#! /usr/bin/bash + +source ../../utils/utility.sh + +URL=http://paddlepaddle.bj.bcebos.com/model_zoo/speech/common_crawl_00.prune01111.trie.klm +MD5="099a601759d467cd0a8523ff939819c5" +TARGET=./common_crawl_00.prune01111.trie.klm + + +echo "Download language model ..." +download $URL $MD5 $TARGET +if [ $? -ne 0 ]; then + echo "Fail to download the language model!" + exit 1 +fi + + +exit 0 diff --git a/utils/utility.sh b/utils/utility.sh new file mode 100644 index 000000000..4f617bfa9 --- /dev/null +++ b/utils/utility.sh @@ -0,0 +1,20 @@ +download() { + URL=$1 + MD5=$2 + TARGET=$3 + + if [ -e $TARGET ]; then + md5_result=`md5sum $TARGET | awk -F[' '] '{print $1}'` + if [ $MD5 == $md5_result ]; then + echo "$TARGET already exists, download skipped." + return 0 + fi + fi + + wget -c $URL -P `dirname "$TARGET"` + md5_result=`md5sum $TARGET | awk -F[' '] '{print $1}'` + if [ $MD5 == $md5_result ]; then + echo "Fail to download the language model!" + return 1 + fi +} From 87453365b2f24486e23763bd4baf0e31147de017 Mon Sep 17 00:00:00 2001 From: Xinghai Sun Date: Tue, 12 Sep 2017 14:12:14 +0800 Subject: [PATCH 5/9] Update REAME.md for DS2. --- .gitignore | 3 - README.md | 143 ++++++++++++++++++++++++++++---------- data/tiny/tiny.py | 126 --------------------------------- examples/tiny/run_data.sh | 6 ++ 4 files changed, 111 insertions(+), 167 deletions(-) delete mode 100644 .gitignore delete mode 100644 data/tiny/tiny.py diff --git a/.gitignore b/.gitignore deleted file mode 100644 index db0537f3b..000000000 --- a/.gitignore +++ /dev/null @@ -1,3 +0,0 @@ -manifest* -mean_std.npz -thirdparty/ diff --git a/README.md b/README.md index aae0dc6d8..afa6dd512 100644 --- a/README.md +++ b/README.md @@ -34,7 +34,7 @@ sh setup.sh ## Getting Started -Several shell scripts provided in `./examples` will help us to quickly give it a try, for most major modules, including data preparation, model training, case inference, model evaluation and demo deployment, with a few public dataset (e.g. [LibriSpeech](http://www.openslr.org/12/), [Aishell](https://github.com/kaldi-asr/kaldi/tree/master/egs/aishell)). Reading these examples will also help us understand how to make it work with our own data. +Several shell scripts provided in `./examples` will help us to quickly give it a try, for most major modules, including data preparation, model training, case inference and model evaluation, with a few public dataset (e.g. [LibriSpeech](http://www.openslr.org/12/), [Aishell](https://github.com/kaldi-asr/kaldi/tree/master/egs/aishell)). Reading these examples will also help us understand how to make it work with our own data. Some of the scripts in `./examples` are configured with 8 GPUs. If you don't have 8 GPUs available, please modify `CUDA_VISIBLE_DEVICE` and `--trainer_count`. If you don't have any GPU available, please set `--use_gpu` to False to use CPUs instead. @@ -83,27 +83,6 @@ Let's take a tiny sampled subset of [LibriSpeech dataset](http://www.openslr.org ``` sh run_test_golden.sh ``` -- Try out a live demo with your own voice - - Until now, we have trained and tested our ASR model qualitatively (`run_infer.sh`) and quantitively (`run_test.sh`) with existing audio files. But we have not yet play the model with our own speech. `demo_server.sh` and `demo_client.sh` helps quickly build up a demo ASR engine with the trained model, enabling us to test and play around with the demo with our own voice. - - We start the server in one console by entering: - - ``` - sh run_demo_server.sh - ``` - - and start the client in another console by entering: - - ``` - sh run_demo_client.sh - ``` - - Then, in the client console, press the `whitespace` key, hold, and start speaking. Until we finish our ulterance, we release the key to let the speech-to-text results show in the console. - - Notice that `run_demo_client.sh` must be run in a machine with a microphone device, while `run_demo_server.sh` could be run in one without any audio recording device, e.g. any remote server. Just be careful to update `run_demo_server.sh` and `run_demo_client.sh` with the actual accessable IP address and port, if the server and client are running with two seperate machines. Nothing has to be done if running in one single machine. - - This demo will first download a pre-trained Mandarin model (trained with 3000 hours of internal speech data). If we would like to try some other model, just update `model_path` argument in the script.       More detailed information are provided in the following sections. @@ -112,7 +91,7 @@ Wish you a happy journey with the DeepSpeech2 ASR engine! ## Data Preparation -#### Generate Manifest +### Generate Manifest *DeepSpeech2 on PaddlePaddle* accepts a textual **manifest** file as its data set interface. A manifest file summarizes a set of speech data, with each line containing some meta data (e.g. filepath, transcription, duration) of one audio clip, in [JSON](http://www.json.org/) format, such as: @@ -125,7 +104,7 @@ To use your custom data, you only need to generate such manifest files to summar For how to generate such manifest files, please refer to `data/librispeech/librispeech.py`, which download and generate manifests for LibriSpeech dataset. -#### Compute Mean & Stddev for Normalizer +### Compute Mean & Stddev for Normalizer To perform z-score normalization (zero-mean, unit stddev) upon audio features, we have to estimate in advance the mean and standard deviation of the features, with some training samples: @@ -139,8 +118,7 @@ python tools/compute_mean_std.py \ It will compute the mean and standard deviation of power spectgram feature with 2000 random sampled audio clips listed in `data/librispeech/manifest.train` and save the results to `data/librispeech/mean_std.npz` for further usage. - -#### Build Vocabulary +### Build Vocabulary A vocabulary of possible characters is required to convert the transcription into a list of token indices for training, and in docoding, to convert from a list of indices back to text again. Such a character-based vocabulary can be build with `tools/build_vocab.py`. @@ -153,7 +131,7 @@ python tools/build_vocab.py \ It will write a vocabuary file `data/librispeeech/eng_vocab.txt` with all transcription text in `data/librispeech/manifest.train`, without vocabulary truncation (`--count_threshold 0`). -#### More Help +### More Help For more help on arguments: @@ -181,7 +159,8 @@ python tools/build_vocab.py --help - Resume training from a checkpoint: ``` - CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 python train.py \ + CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 \ + python train.py \ --init_model_path CHECKPOINT_PATH_TO_RESUME_FROM ``` @@ -295,7 +274,8 @@ The hyper-parameters $\alpha$ (coefficient for language model scorer) and $\beta - Tuning with GPU: ``` - CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 python tools/tune.py \ + CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 \ + python tools/tune.py \ --trainer_count 8 \ --alpha_from 0.1 \ --alpha_to 0.36 \ @@ -322,14 +302,86 @@ TODO: add figure. ## Distributed Cloud Training -If you wish to train DeepSpeech2 on PaddleCloud, please refer to +We provide a cloud training module for users to do the distributed cluster training on [PaddleCloud](https://github.com/PaddlePaddle/cloud), to achieve a much faster training speed with multiple machines. To start with this, please first install PaddleCloud client and register a PaddleCloud account, as described in [PaddleCloud Usage](https://github.com/PaddlePaddle/cloud/blob/develop/doc/usage_cn.md#%E4%B8%8B%E8%BD%BD%E5%B9%B6%E9%85%8D%E7%BD%AEpaddlecloud). + +Then, we take the following steps to sumbit a training job: + +- go to directory: + + ``` + cd cloud + ``` +- Upload data: + + Data must be uploaded to PaddleCloud filesystem to be accessed from a cloud job. `pcloud_upload_data.sh` helps do the data packing and uploading: + + ``` + sh pcloud_upload_data.sh + ``` + + Given input manifests, `pcloud_upload_data.sh` will: + + - Extract the audio files listed in the input manifests. + - Pack them into a specified number of tar files. + - Upload these tar files to PaddleCloud filesystem. + - Create cloud manifests by replacing local filesystem paths with PaddleCloud filesystem paths. New manifests will be used to inform the cloud jobs of audio files' location and their meta information. + + It has to be done only once for the very first time we do the cloud training. Later on, the data is persisitent on the cloud filesystem and reusable for further job submissions. + + For argument details please refer to [Train DeepSpeech2 on PaddleCloud](https://github.com/PaddlePaddle/models/tree/develop/deep_speech_2/cloud). + + - Configure training arguments: + + Configure the cloud job parameters in `pcloud_submit.sh` (e.g. `NUM_NODES`, `NUM_GPUS`, `CLOUD_TRAIN_DIR`, `JOB_NAME` etc.) and then configure other hyper-parameters for training in `pcloud_train.sh` (just as what you do for local training). + + For argument details please refer to [Train DeepSpeech2 on PaddleCloud](https://github.com/PaddlePaddle/models/tree/develop/deep_speech_2/cloud). + + - Submit the job: + + By running: + + ``` + sh pcloud_submit.sh + ``` + we submit a training job to PaddleCloud. And we will see the job name when the submission is finished. Now our training job is running well on the PaddleCloud. + + - Get training logs + + Run this to list all the jobs you have submitted, as well as their running status: + + ``` + paddlecloud get jobs + ``` + + Run this, the corresponding job's logs will be printed. + ``` + paddlecloud logs -n 10000 $REPLACED_WITH_YOUR_ACTUAL_JOB_NAME + ``` + +For more information about the usage of PaddleCloud, please refer to [PaddleCloud Usage](https://github.com/PaddlePaddle/cloud/blob/develop/doc/usage_cn.md#提交任务). + +For more information about the DeepSpeech2 training on PaddleCloud, please refer to [Train DeepSpeech2 on PaddleCloud](https://github.com/PaddlePaddle/models/tree/develop/deep_speech_2/cloud). ## Training for Mandarin Language +TODO: to be added + ## Trying Live Demo with Your Own Voice -A real-time ASR demo is built for users to try out the ASR model with their own voice. Please do the following installation on the machine you'd like to run the demo's client (no need for the machine running the demo's server). +Until now, we have trained and tested our ASR model qualitatively (`infer.py`) and quantitively (`test.py`) with existing audio files. But we have not yet play the model with our own speech. `deploy/demo_server.py` and `deploy/demo_client.py` helps quickly build up a real-time demo ASR engine with the trained model, enabling us to test and play around with the demo, with our own voice. + +We start the demo's server in one console by: + +``` +CUDA_VISIBLE_DEVICES=0 \ +python deploy/demo_server.py \ +--trainer_count 1 \ +--host_ip localhost \ +--host_port 8086 +``` + +For the machine (might be the same or a different machine) to run the demo's client, we have to do the following installation before moving on. For example, on MAC OS X: @@ -338,22 +390,37 @@ brew install portaudio pip install pyaudio pip install pynput ``` -After a model and language model is prepared, we can first start the demo's server: + +Then we can start the client in another console by: ``` -CUDA_VISIBLE_DEVICES=0 python demo_server.py +CUDA_VISIBLE_DEVICES=0 \ +python -u deploy/demo_client.py \ +--host_ip 'localhost' \ +--host_port 8086 \ ``` -And then in another console, start the demo's client: + +Next, in the client console, press the `whitespace` key, hold, and start speaking. Until we finish our ulterance, we release the key to let the speech-to-text results shown in the console. To quit the client, just press `ESC` key. + +Notice that `deploy/demo_client.py` must be run in a machine with a microphone device, while `deploy/demo_server.py` could be run in one without any audio recording hardware, e.g. any remote server machine. Just be careful to set the `host_ip` and `host_port` argument with the actual accessable IP address and port, if the server and client are running with two seperate machines. Nothing has to be done if they are running in one single machine. + +We can also refer to `examples/mandarin/run_demo_server.sh` for example, which will first download a pre-trained Mandarin model (trained with 3000 hours of internal speech data) and then start the demo server with the model. With running `examples/mandarin/run_demo_client.sh`, we can speak Mandarin to test it. If we would like to try some other models, just update `--model_path` argument in the script.   + +For more help on arguments: ``` -python demo_client.py +python deploy/demo_server.py --help +python deploy/demo_client.py --help ``` -On the client console, press and hold the "white-space" key on the keyboard to start talking, until you finish your speech and then release the "white-space" key. The decoding results (infered transcription) will be displayed. - -It could be possible to start the server and the client in two seperate machines, e.g. `demo_client.py` is usually started in a machine with a microphone hardware, while `demo_server.py` is usually started in a remote server with powerful GPUs. Please first make sure that these two machines have network access to each other, and then use `--host_ip` and `--host_port` to indicate the server machine's actual IP address (instead of the `localhost` as default) and TCP port, in both `demo_server.py` and `demo_client.py`. ## Experiments and Benchmarks +TODO: to be added + ## Released Models +TODO: to be added + ## Questions and Help + +You are welcome to submit questions and bug reports in [Github Issues](https://github.com/PaddlePaddle/models/issues). You are also welcome to contribute to this project. diff --git a/data/tiny/tiny.py b/data/tiny/tiny.py deleted file mode 100644 index 8ba2a13c5..000000000 --- a/data/tiny/tiny.py +++ /dev/null @@ -1,126 +0,0 @@ -"""Prepare Librispeech ASR datasets. - -Download, unpack and create manifest files. -Manifest file is a json-format file with each line containing the -meta data (i.e. audio filepath, transcript and audio duration) -of each audio file in the data set. -""" -from __future__ import absolute_import -from __future__ import division -from __future__ import print_function - -import distutils.util -import os -import sys -import tarfile -import argparse -import soundfile -import json -import codecs -from paddle.v2.dataset.common import md5file - -DATA_HOME = os.path.expanduser('~/.cache/paddle/dataset/speech') - -URL_ROOT = "http://www.openslr.org/resources/12" -URL_DEV_CLEAN = URL_ROOT + "/dev-clean.tar.gz" -MD5_DEV_CLEAN = "42e2234ba48799c1f50f24a7926300a1" - -parser = argparse.ArgumentParser(description=__doc__) -parser.add_argument( - "--target_dir", - default=DATA_HOME + "/tiny", - type=str, - help="Directory to save the dataset. (default: %(default)s)") -parser.add_argument( - "--manifest_prefix", - default="manifest", - type=str, - help="Filepath prefix for output manifests. (default: %(default)s)") -args = parser.parse_args() - - -def download(url, md5sum, target_dir): - """ - Download file from url to target_dir, and check md5sum. - """ - if not os.path.exists(target_dir): os.makedirs(target_dir) - filepath = os.path.join(target_dir, url.split("/")[-1]) - if not (os.path.exists(filepath) and md5file(filepath) == md5sum): - print("Downloading %s ..." % url) - os.system("wget -c " + url + " -P " + target_dir) - print("\nMD5 Chesksum %s ..." % filepath) - if not md5file(filepath) == md5sum: - raise RuntimeError("MD5 checksum failed.") - else: - print("File exists, skip downloading. (%s)" % filepath) - return filepath - - -def unpack(filepath, target_dir): - """ - Unpack the file to the target_dir. - """ - print("Unpacking %s ..." % filepath) - tar = tarfile.open(filepath) - tar.extractall(target_dir) - tar.close() - - -def create_manifest(data_dir, manifest_path): - """ - Create a manifest json file summarizing the data set, with each line - containing the meta data (i.e. audio filepath, transcription text, audio - duration) of each audio file within the data set. - """ - print("Creating manifest %s ..." % manifest_path) - json_lines = [] - for subfolder, _, filelist in sorted(os.walk(data_dir)): - text_filelist = [ - filename for filename in filelist if filename.endswith('trans.txt') - ] - if len(text_filelist) > 0: - text_filepath = os.path.join(data_dir, subfolder, text_filelist[0]) - for line in open(text_filepath): - segments = line.strip().split() - text = ' '.join(segments[1:]).lower() - audio_filepath = os.path.join(data_dir, subfolder, - segments[0] + '.flac') - audio_data, samplerate = soundfile.read(audio_filepath) - duration = float(len(audio_data)) / samplerate - json_lines.append( - json.dumps({ - 'audio_filepath': audio_filepath, - 'duration': duration, - 'text': text - })) - with codecs.open(manifest_path, 'w', 'utf-8') as out_file: - for line in json_lines: - out_file.write(line + '\n') - - -def prepare_dataset(url, md5sum, target_dir, manifest_path): - """ - Download, unpack and create summmary manifest file. - """ - if not os.path.exists(os.path.join(target_dir, "LibriSpeech")): - # download - filepath = download(url, md5sum, target_dir) - # unpack - unpack(filepath, target_dir) - else: - print("Skip downloading and unpacking. Data already exists in %s." % - target_dir) - # create manifest json file - create_manifest(target_dir, manifest_path) - - -def main(): - prepare_dataset( - url=URL_DEV_CLEAN, - md5sum=MD5_DEV_CLEAN, - target_dir=os.path.join(args.target_dir, "dev-clean"), - manifest_path=args.manifest_prefix + ".dev-clean") - - -if __name__ == '__main__': - main() diff --git a/examples/tiny/run_data.sh b/examples/tiny/run_data.sh index 203d3e2c8..46266daaf 100644 --- a/examples/tiny/run_data.sh +++ b/examples/tiny/run_data.sh @@ -2,6 +2,12 @@ pushd ../.. > /dev/null +# prepare folder +if [ ! -e data/tiny ]; then + mkdir data/tiny +fi + + # download data, generate manifests python data/librispeech/librispeech.py \ --manifest_prefix='data/tiny/manifest' \ From 4969d297d8002de0c15d32342664cb5c756f628a Mon Sep 17 00:00:00 2001 From: Xinghai Sun Date: Tue, 12 Sep 2017 14:42:15 +0800 Subject: [PATCH 6/9] Correct typos for DS2 README.md. --- README.md | 64 ++++++++++++++++++++++++++++++------------------------- 1 file changed, 35 insertions(+), 29 deletions(-) diff --git a/README.md b/README.md index afa6dd512..7c176d8bb 100644 --- a/README.md +++ b/README.md @@ -1,6 +1,6 @@ # DeepSpeech2 on PaddlePaddle -*DeepSpeech2 on PaddlePaddle* is an open-source implementation of end-to-end Automatic Speech Recognition (ASR) engine, based on [Baidu's Deep Speech 2 paper](http://proceedings.mlr.press/v48/amodei16.pdf), with [PaddlePaddle](https://github.com/PaddlePaddle/Paddle) platform. Our vision is to empower both industrial application and academic research on speech-to-text, via an easy-to-use, efficent and scalable integreted implementation, including training, inferencing & testing module, distributed [PaddleCloud](https://github.com/PaddlePaddle/cloud) training, and demo deployment. Besides, several pre-trained models for both English and Mandarin are also released. +*DeepSpeech2 on PaddlePaddle* is an open-source implementation of end-to-end Automatic Speech Recognition (ASR) engine, based on [Baidu's Deep Speech 2 paper](http://proceedings.mlr.press/v48/amodei16.pdf), with [PaddlePaddle](https://github.com/PaddlePaddle/Paddle) platform. Our vision is to empower both industrial application and academic research on speech recognition, via an easy-to-use, efficient and scalable implementation, including training, inferencing & testing module, distributed [PaddleCloud](https://github.com/PaddlePaddle/cloud) training, and demo deployment. Besides, several pre-trained models for both English and Mandarin are also released. ## Table of Contents - [Prerequisites](#prerequisites) @@ -53,14 +53,14 @@ Let's take a tiny sampled subset of [LibriSpeech dataset](http://www.openslr.org sh run_data.sh ``` - `run_data.sh` will download dataset, generate manifests, collect normalizer' statitics and build vocabulary. Once the data preparation is done, we will find the data (only part of LibriSpeech) downloaded in `~/.cache/paddle/dataset/speech/libri` and the corresponding manifest files generated in `./data/tiny` as well as a mean stddev file and a vocabulary file. It has to be run for the very first time we run this dataset and is reusable for all further experiments. + `run_data.sh` will download dataset, generate manifests, collect normalizer' statistics and build vocabulary. Once the data preparation is done, we will find the data (only part of LibriSpeech) downloaded in `~/.cache/paddle/dataset/speech/libri` and the corresponding manifest files generated in `./data/tiny` as well as a mean stddev file and a vocabulary file. It has to be run for the very first time we run this dataset and is reusable for all further experiments. - Train your own ASR model ``` sh run_train.sh ``` - `run_train.sh` will start a training job, with training logs printed to stdout and model checkpoint of every pass/epoch saved to `./checkpoints/tiny`. We can resume the training from these checkpoints, or use them for inference, evalutiaton and deployment. + `run_train.sh` will start a training job, with training logs printed to stdout and model checkpoint of every pass/epoch saved to `./checkpoints/tiny`. We can resume the training from these checkpoints, or use them for inference, evaluation and deployment. - Case inference with an existing model ``` @@ -83,10 +83,8 @@ Let's take a tiny sampled subset of [LibriSpeech dataset](http://www.openslr.org ``` sh run_test_golden.sh ``` -     -More detailed information are provided in the following sections. -Wish you a happy journey with the DeepSpeech2 ASR engine! +More detailed information are provided in the following sections. Wish you a happy journey with the *DeepSpeech2 on PaddlePaddle* ASR engine! ## Data Preparation @@ -116,11 +114,12 @@ python tools/compute_mean_std.py \ --output_path data/librispeech/mean_std.npz ``` -It will compute the mean and standard deviation of power spectgram feature with 2000 random sampled audio clips listed in `data/librispeech/manifest.train` and save the results to `data/librispeech/mean_std.npz` for further usage. +It will compute the mean and standard deviation of power spectrum feature with 2000 random sampled audio clips listed in `data/librispeech/manifest.train` and save the results to `data/librispeech/mean_std.npz` for further usage. + ### Build Vocabulary -A vocabulary of possible characters is required to convert the transcription into a list of token indices for training, and in docoding, to convert from a list of indices back to text again. Such a character-based vocabulary can be build with `tools/build_vocab.py`. +A vocabulary of possible characters is required to convert the transcription into a list of token indices for training, and in decoding, to convert from a list of indices back to text again. Such a character-based vocabulary can be built with `tools/build_vocab.py`. ``` python tools/build_vocab.py \ @@ -173,14 +172,14 @@ or refer to `example/librispeech/run_train.sh`. ## Data Augmentation Pipeline -Data augmentation has often been a highly effective technique to boost the deep learning performance. We augment our speech data by synthesizing new audios with small random perterbation (label-invariant transformation) added upon raw audios. We don't have to do the syntheses by ourselves, as it is already embeded into the data provider and is done on the fly, randomly for each epoch during training. +Data augmentation has often been a highly effective technique to boost the deep learning performance. We augment our speech data by synthesizing new audios with small random perturbation (label-invariant transformation) added upon raw audios. We don't have to do the syntheses by ourselves, as it is already embedded into the data provider and is done on the fly, randomly for each epoch during training. Six optional augmentation components are provided for us to configured and inserted into the processing pipeline. - Volume Perturbation - Speed Perturbation - Shifting Perturbation - - Online Beyesian normalization + - Online Bayesian normalization - Noise Perturbation (need background noise audio files) - Impulse Response (need impulse audio files) @@ -211,13 +210,20 @@ Be careful when we are utilizing the data augmentation technique, as improper au ### Prepare Language Model -A language model is required to improve the decoder's performance. We have prepared two language models (with lossy compression) for users to download and try. One is for English and the other is for Mandarin. Please refer to `models/lm/download_lm_en.sh` and `models/lm/download_lm_zh.sh` for their urls. If you wish to train your own better language model, please refer to [KenLM](https://github.com/kpu/kenlm) for tutorials. +A language model is required to improve the decoder's performance. We have prepared two language models (with lossy compression) for users to download and try. One is for English and the other is for Mandarin. We can simply run this to download the preprared language models: + +``` +cd models/lm +sh download_lm_en.sh +sh download_lm_ch.sh +``` +If you wish to train your own better language model, please refer to [KenLM](https://github.com/kpu/kenlm) for tutorials. TODO: any other requirements or tips to add? ### Speech-to-text Inference -We provide a inference module `infer.py` to infer, decode and visualize speech-to-text results for several given audio clips. It might help us to have a intuitive and qualitative evaluation of the ASR model's performance. +An inference module caller `infer.py` is provided for us to infer, decode and visualize speech-to-text results for several given audio clips. It might help to have an intuitive and qualitative evaluation of the ASR model's performance. - Inference with GPU: @@ -225,13 +231,13 @@ We provide a inference module `infer.py` to infer, decode and visualize speech-t CUDA_VISIBLE_DEVICES=0 python infer.py --trainer_count 1 ``` -- Inference with CPU: +- Inference with CPUs: ``` python infer.py --use_gpu False --trainer_count 12 ``` -We provide two types of CTC decoders: *CTC greedy decoder* and *CTC beam search decoder*. The *CTC greedy decoder* is an implementation of the simple best-path decoding algorithm, selecting at each timestep the most likely token, thus being greedy and locally optimal. The [*CTC beam search decoder*](https://arxiv.org/abs/1408.2873) otherwise utilizes a heuristic breadth-first gragh search for reaching a near global optimality; it also requires a pre-trained KenLM language model for better scoring and ranking. The decoder type can be set with argument `--decoding_method`. +We provide two types of CTC decoders: *CTC greedy decoder* and *CTC beam search decoder*. The *CTC greedy decoder* is an implementation of the simple best-path decoding algorithm, selecting at each timestep the most likely token, thus being greedy and locally optimal. The [*CTC beam search decoder*](https://arxiv.org/abs/1408.2873) otherwise utilizes a heuristic breadth-first graph search for reaching a near global optimality; it also requires a pre-trained KenLM language model for better scoring and ranking. The decoder type can be set with argument `--decoding_method`. For more help on arguments: @@ -242,15 +248,15 @@ or refer to `example/librispeech/run_infer.sh`. ### Evaluate a Model -To evaluate a model's performance quantitively, we can run: +To evaluate a model's performance quantitatively, we can run: -- Evaluation with GPU: +- Evaluation with GPUs: ``` CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 python test.py --trainer_count 8 ``` -- Evaluation with CPU: +- Evaluation with CPUs: ``` python test.py --use_gpu False --trainer_count 12 @@ -267,9 +273,9 @@ or refer to `example/librispeech/run_test.sh`. ## Hyper-parameters Tuning -The hyper-parameters $\alpha$ (coefficient for language model scorer) and $\beta$ (coefficient for word count scorer) for the [*CTC beam search decoder*](https://arxiv.org/abs/1408.2873) often have a significant impact on the decoder's performance. It would be better to re-tune them on a validation set when the accustic model is renewed. +The hyper-parameters $\alpha$ (coefficient for language model scorer) and $\beta$ (coefficient for word count scorer) for the [*CTC beam search decoder*](https://arxiv.org/abs/1408.2873) often have a significant impact on the decoder's performance. It would be better to re-tune them on a validation set when the acoustic model is renewed. -`tools/tune.py` performs a 2-D grid search over the hyper-parameter $\alpha$ and $\beta$. We have to provide the range of $\alpha$ and $\beta$, as well as the number of their attempts. +`tools/tune.py` performs a 2-D grid search over the hyper-parameter $\alpha$ and $\beta$. We must provide the range of $\alpha$ and $\beta$, as well as the number of their attempts. - Tuning with GPU: @@ -304,16 +310,16 @@ TODO: add figure. We provide a cloud training module for users to do the distributed cluster training on [PaddleCloud](https://github.com/PaddlePaddle/cloud), to achieve a much faster training speed with multiple machines. To start with this, please first install PaddleCloud client and register a PaddleCloud account, as described in [PaddleCloud Usage](https://github.com/PaddlePaddle/cloud/blob/develop/doc/usage_cn.md#%E4%B8%8B%E8%BD%BD%E5%B9%B6%E9%85%8D%E7%BD%AEpaddlecloud). -Then, we take the following steps to sumbit a training job: +Then, we take the following steps to submit a training job: -- go to directory: +- Go to directory: ``` cd cloud ``` - Upload data: - Data must be uploaded to PaddleCloud filesystem to be accessed from a cloud job. `pcloud_upload_data.sh` helps do the data packing and uploading: + Data must be uploaded to PaddleCloud filesystem to be accessed within a cloud job. `pcloud_upload_data.sh` helps do the data packing and uploading: ``` sh pcloud_upload_data.sh @@ -326,7 +332,7 @@ Then, we take the following steps to sumbit a training job: - Upload these tar files to PaddleCloud filesystem. - Create cloud manifests by replacing local filesystem paths with PaddleCloud filesystem paths. New manifests will be used to inform the cloud jobs of audio files' location and their meta information. - It has to be done only once for the very first time we do the cloud training. Later on, the data is persisitent on the cloud filesystem and reusable for further job submissions. + It should be done only once for the very first time we do the cloud training. Later, the data is kept persisitent on the cloud filesystem and reusable for further job submissions. For argument details please refer to [Train DeepSpeech2 on PaddleCloud](https://github.com/PaddlePaddle/models/tree/develop/deep_speech_2/cloud). @@ -343,7 +349,7 @@ Then, we take the following steps to sumbit a training job: ``` sh pcloud_submit.sh ``` - we submit a training job to PaddleCloud. And we will see the job name when the submission is finished. Now our training job is running well on the PaddleCloud. + we submit a training job to PaddleCloud. And the job name will be printed when the submission is finished. Now our training job is running well on the PaddleCloud. - Get training logs @@ -369,7 +375,7 @@ TODO: to be added ## Trying Live Demo with Your Own Voice -Until now, we have trained and tested our ASR model qualitatively (`infer.py`) and quantitively (`test.py`) with existing audio files. But we have not yet play the model with our own speech. `deploy/demo_server.py` and `deploy/demo_client.py` helps quickly build up a real-time demo ASR engine with the trained model, enabling us to test and play around with the demo, with our own voice. +Until now, we have trained and tested our ASR model qualitatively (`infer.py`) and quantitatively (`test.py`) with existing audio files. But we have not yet try the model with our own speech. `deploy/demo_server.py` and `deploy/demo_client.py` helps quickly build up a real-time demo ASR engine with the trained model, enabling us to test and play around with the demo, with our own voice. We start the demo's server in one console by: @@ -381,7 +387,7 @@ python deploy/demo_server.py \ --host_port 8086 ``` -For the machine (might be the same or a different machine) to run the demo's client, we have to do the following installation before moving on. +For the machine (might not be the same machine) to run the demo's client, we have to do the following installation before moving on. For example, on MAC OS X: @@ -397,12 +403,12 @@ Then we can start the client in another console by: CUDA_VISIBLE_DEVICES=0 \ python -u deploy/demo_client.py \ --host_ip 'localhost' \ ---host_port 8086 \ +--host_port 8086 ``` -Next, in the client console, press the `whitespace` key, hold, and start speaking. Until we finish our ulterance, we release the key to let the speech-to-text results shown in the console. To quit the client, just press `ESC` key. +Now, in the client console, press the `whitespace` key, hold, and start speaking. Until we finish our utterance, we release the key to let the speech-to-text results shown in the console. To quit the client, just press `ESC` key. -Notice that `deploy/demo_client.py` must be run in a machine with a microphone device, while `deploy/demo_server.py` could be run in one without any audio recording hardware, e.g. any remote server machine. Just be careful to set the `host_ip` and `host_port` argument with the actual accessable IP address and port, if the server and client are running with two seperate machines. Nothing has to be done if they are running in one single machine. +Notice that `deploy/demo_client.py` must be run in a machine with a microphone device, while `deploy/demo_server.py` could be run in one without any audio recording hardware, e.g. any remote server machine. Just be careful to set the `host_ip` and `host_port` argument with the actual accessible IP address and port, if the server and client are running with two separate machines. Nothing should be done if they are running in one single machine. We can also refer to `examples/mandarin/run_demo_server.sh` for example, which will first download a pre-trained Mandarin model (trained with 3000 hours of internal speech data) and then start the demo server with the model. With running `examples/mandarin/run_demo_client.sh`, we can speak Mandarin to test it. If we would like to try some other models, just update `--model_path` argument in the script.   From 35caf5e0b744171634fbc2ea914e6f85a281718a Mon Sep 17 00:00:00 2001 From: Xinghai Sun Date: Tue, 12 Sep 2017 23:46:50 +0800 Subject: [PATCH 7/9] Add bash code highlight to README.md for DS2. --- README.md | 60 ++++++++++++++++----------------- data/librispeech/librispeech.py | 1 - 2 files changed, 30 insertions(+), 31 deletions(-) diff --git a/README.md b/README.md index 7c176d8bb..d9b989342 100644 --- a/README.md +++ b/README.md @@ -26,7 +26,7 @@ Please install the [prerequisites](#prerequisites) above before moving on. -``` +```bash git clone https://github.com/PaddlePaddle/models.git cd models/deep_speech_2 sh setup.sh @@ -42,45 +42,45 @@ Let's take a tiny sampled subset of [LibriSpeech dataset](http://www.openslr.org - Go to directory - ``` + ```bash cd examples/tiny ``` Notice that this is only a toy example with a tiny sampled subset of LibriSpeech. If we would like to try with the complete dataset (would take several days for training), please go to `examples/librispeech` instead. - Prepare the data - ``` + ```bash sh run_data.sh ``` `run_data.sh` will download dataset, generate manifests, collect normalizer' statistics and build vocabulary. Once the data preparation is done, we will find the data (only part of LibriSpeech) downloaded in `~/.cache/paddle/dataset/speech/libri` and the corresponding manifest files generated in `./data/tiny` as well as a mean stddev file and a vocabulary file. It has to be run for the very first time we run this dataset and is reusable for all further experiments. - Train your own ASR model - ``` + ```bash sh run_train.sh ``` `run_train.sh` will start a training job, with training logs printed to stdout and model checkpoint of every pass/epoch saved to `./checkpoints/tiny`. We can resume the training from these checkpoints, or use them for inference, evaluation and deployment. - Case inference with an existing model - ``` + ```bash sh run_infer.sh ``` `run_infer.sh` will show us some speech-to-text decoding results for several (default: 10) samples with the trained model. The performance might not be good now as the current model is only trained with a toy subset of LibriSpeech. To see the results with a better model, we can download a well-trained (trained for several days, with the complete LibriSpeech) model and do the inference: - ``` + ```bash sh run_infer_golden.sh ``` - Evaluate an existing model - ``` + ```bash sh run_test.sh ``` `run_test.sh` will evaluate the model with Word Error Rate (or Character Error Rate) measurement. Similarly, we can also download a well-trained model and test its performance: - ``` + ```bash sh run_test_golden.sh ``` @@ -106,7 +106,7 @@ For how to generate such manifest files, please refer to `data/librispeech/libri To perform z-score normalization (zero-mean, unit stddev) upon audio features, we have to estimate in advance the mean and standard deviation of the features, with some training samples: -``` +```bash python tools/compute_mean_std.py \ --num_samples 2000 \ --specgram_type linear \ @@ -121,7 +121,7 @@ It will compute the mean and standard deviation of power spectrum feature with 2 A vocabulary of possible characters is required to convert the transcription into a list of token indices for training, and in decoding, to convert from a list of indices back to text again. Such a character-based vocabulary can be built with `tools/build_vocab.py`. -``` +```bash python tools/build_vocab.py \ --count_threshold 0 \ --vocab_path data/librispeech/eng_vocab.txt \ @@ -134,7 +134,7 @@ It will write a vocabuary file `data/librispeeech/eng_vocab.txt` with all transc For more help on arguments: -``` +```bash python data/librispeech/librispeech.py --help python tools/compute_mean_std.py --help python tools/build_vocab.py --help @@ -165,7 +165,7 @@ python tools/build_vocab.py --help For more help on arguments: -``` +```bash python train.py --help ``` or refer to `example/librispeech/run_train.sh`. @@ -212,7 +212,7 @@ Be careful when we are utilizing the data augmentation technique, as improper au A language model is required to improve the decoder's performance. We have prepared two language models (with lossy compression) for users to download and try. One is for English and the other is for Mandarin. We can simply run this to download the preprared language models: -``` +```bash cd models/lm sh download_lm_en.sh sh download_lm_ch.sh @@ -227,13 +227,13 @@ An inference module caller `infer.py` is provided for us to infer, decode and vi - Inference with GPU: - ``` + ```bash CUDA_VISIBLE_DEVICES=0 python infer.py --trainer_count 1 ``` - Inference with CPUs: - ``` + ```bash python infer.py --use_gpu False --trainer_count 12 ``` @@ -252,13 +252,13 @@ To evaluate a model's performance quantitatively, we can run: - Evaluation with GPUs: - ``` + ```bash CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 python test.py --trainer_count 8 ``` - Evaluation with CPUs: - ``` + ```bash python test.py --use_gpu False --trainer_count 12 ``` @@ -266,7 +266,7 @@ The error rate (default: word error rate; can be set with `--error_rate_type`) w For more help on arguments: -``` +```bash python test.py --help ``` or refer to `example/librispeech/run_test.sh`. @@ -279,7 +279,7 @@ The hyper-parameters $\alpha$ (coefficient for language model scorer) and $\beta - Tuning with GPU: - ``` + ```bash CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 \ python tools/tune.py \ --trainer_count 8 \ @@ -293,13 +293,13 @@ The hyper-parameters $\alpha$ (coefficient for language model scorer) and $\beta - Tuning with CPU: - ``` + ```bash python tools/tune.py --use_gpu False ``` After tuning, we can reset $\alpha$ and $\beta$ in the inference and evaluation modules to see if they really help improve the ASR performance. -``` +```bash python tune.py --help ``` or refer to `example/librispeech/run_tune.sh`. @@ -314,14 +314,14 @@ Then, we take the following steps to submit a training job: - Go to directory: - ``` + ```bash cd cloud ``` - Upload data: Data must be uploaded to PaddleCloud filesystem to be accessed within a cloud job. `pcloud_upload_data.sh` helps do the data packing and uploading: - ``` + ```bash sh pcloud_upload_data.sh ``` @@ -346,7 +346,7 @@ Then, we take the following steps to submit a training job: By running: - ``` + ```bash sh pcloud_submit.sh ``` we submit a training job to PaddleCloud. And the job name will be printed when the submission is finished. Now our training job is running well on the PaddleCloud. @@ -355,12 +355,12 @@ Then, we take the following steps to submit a training job: Run this to list all the jobs you have submitted, as well as their running status: - ``` + ```bash paddlecloud get jobs ``` Run this, the corresponding job's logs will be printed. - ``` + ```bash paddlecloud logs -n 10000 $REPLACED_WITH_YOUR_ACTUAL_JOB_NAME ``` @@ -379,7 +379,7 @@ Until now, we have trained and tested our ASR model qualitatively (`infer.py`) a We start the demo's server in one console by: -``` +```bash CUDA_VISIBLE_DEVICES=0 \ python deploy/demo_server.py \ --trainer_count 1 \ @@ -391,7 +391,7 @@ For the machine (might not be the same machine) to run the demo's client, we hav For example, on MAC OS X: -``` +```bash brew install portaudio pip install pyaudio pip install pynput @@ -399,7 +399,7 @@ pip install pynput Then we can start the client in another console by: -``` +```bash CUDA_VISIBLE_DEVICES=0 \ python -u deploy/demo_client.py \ --host_ip 'localhost' \ @@ -414,7 +414,7 @@ We can also refer to `examples/mandarin/run_demo_server.sh` for example, which w For more help on arguments: -``` +```bash python deploy/demo_server.py --help python deploy/demo_client.py --help ``` diff --git a/data/librispeech/librispeech.py b/data/librispeech/librispeech.py index e2ad8d413..0709136e2 100644 --- a/data/librispeech/librispeech.py +++ b/data/librispeech/librispeech.py @@ -65,7 +65,6 @@ def download(url, md5sum, target_dir): if not (os.path.exists(filepath) and md5file(filepath) == md5sum): print("Downloading %s ..." % url) ret = os.system("wget -c " + url + " -P " + target_dir) - print(ret) print("\nMD5 Chesksum %s ..." % filepath) if not md5file(filepath) == md5sum: raise RuntimeError("MD5 checksum failed.") From ac56a2f249a853653e1d1fe7b173475c67c90a91 Mon Sep 17 00:00:00 2001 From: Xinghai Sun Date: Wed, 13 Sep 2017 15:36:34 +0800 Subject: [PATCH 8/9] Update READMD.md and other details by following reviewers comments. --- README.md | 64 +++++++++++++++---------------- deploy/demo_server.py | 2 +- examples/librispeech/run_train.sh | 1 + examples/tiny/run_train.sh | 1 + infer.py | 4 +- test.py | 4 +- tools/tune.py | 4 +- train.py | 2 +- 8 files changed, 42 insertions(+), 40 deletions(-) diff --git a/README.md b/README.md index d9b989342..055bd439e 100644 --- a/README.md +++ b/README.md @@ -1,6 +1,6 @@ # DeepSpeech2 on PaddlePaddle -*DeepSpeech2 on PaddlePaddle* is an open-source implementation of end-to-end Automatic Speech Recognition (ASR) engine, based on [Baidu's Deep Speech 2 paper](http://proceedings.mlr.press/v48/amodei16.pdf), with [PaddlePaddle](https://github.com/PaddlePaddle/Paddle) platform. Our vision is to empower both industrial application and academic research on speech recognition, via an easy-to-use, efficient and scalable implementation, including training, inferencing & testing module, distributed [PaddleCloud](https://github.com/PaddlePaddle/cloud) training, and demo deployment. Besides, several pre-trained models for both English and Mandarin are also released. +*DeepSpeech2 on PaddlePaddle* is an open-source implementation of end-to-end Automatic Speech Recognition (ASR) engine, based on [Baidu's Deep Speech 2 paper](http://proceedings.mlr.press/v48/amodei16.pdf), with [PaddlePaddle](https://github.com/PaddlePaddle/Paddle) platform. Our vision is to empower both industrial application and academic research on speech recognition, via an easy-to-use, efficient and scalable implementation, including training, inference & testing module, distributed [PaddleCloud](https://github.com/PaddlePaddle/cloud) training, and demo deployment. Besides, several pre-trained models for both English and Mandarin are also released. ## Table of Contents - [Prerequisites](#prerequisites) @@ -19,12 +19,12 @@ - [Questions and Help](#questions-and-help) ## Prerequisites -- Only support Python 2.7 +- Python 2.7 only supported - PaddlePaddle the latest version (please refer to the [Installation Guide](https://github.com/PaddlePaddle/Paddle#installation)) ## Installation -Please install the [prerequisites](#prerequisites) above before moving on. +Please make sure the above [prerequisites](#prerequisites) have been satisfied before moving on. ```bash git clone https://github.com/PaddlePaddle/models.git @@ -34,9 +34,9 @@ sh setup.sh ## Getting Started -Several shell scripts provided in `./examples` will help us to quickly give it a try, for most major modules, including data preparation, model training, case inference and model evaluation, with a few public dataset (e.g. [LibriSpeech](http://www.openslr.org/12/), [Aishell](https://github.com/kaldi-asr/kaldi/tree/master/egs/aishell)). Reading these examples will also help us understand how to make it work with our own data. +Several shell scripts provided in `./examples` will help us to quickly give it a try, for most major modules, including data preparation, model training, case inference and model evaluation, with a few public dataset (e.g. [LibriSpeech](http://www.openslr.org/12/), [Aishell](http://www.openslr.org/33)). Reading these examples will also help you to understand how to make it work with your own data. -Some of the scripts in `./examples` are configured with 8 GPUs. If you don't have 8 GPUs available, please modify `CUDA_VISIBLE_DEVICE` and `--trainer_count`. If you don't have any GPU available, please set `--use_gpu` to False to use CPUs instead. +Some of the scripts in `./examples` are configured with 8 GPUs. If you don't have 8 GPUs available, please modify `CUDA_VISIBLE_DEVICES` and `--trainer_count`. If you don't have any GPU available, please set `--use_gpu` to False to use CPUs instead. Let's take a tiny sampled subset of [LibriSpeech dataset](http://www.openslr.org/12/) for instance. @@ -46,28 +46,28 @@ Let's take a tiny sampled subset of [LibriSpeech dataset](http://www.openslr.org cd examples/tiny ``` - Notice that this is only a toy example with a tiny sampled subset of LibriSpeech. If we would like to try with the complete dataset (would take several days for training), please go to `examples/librispeech` instead. + Notice that this is only a toy example with a tiny sampled subset of LibriSpeech. If you would like to try with the complete dataset (would take several days for training), please go to `examples/librispeech` instead. - Prepare the data ```bash sh run_data.sh ``` - `run_data.sh` will download dataset, generate manifests, collect normalizer' statistics and build vocabulary. Once the data preparation is done, we will find the data (only part of LibriSpeech) downloaded in `~/.cache/paddle/dataset/speech/libri` and the corresponding manifest files generated in `./data/tiny` as well as a mean stddev file and a vocabulary file. It has to be run for the very first time we run this dataset and is reusable for all further experiments. + `run_data.sh` will download dataset, generate manifests, collect normalizer's statistics and build vocabulary. Once the data preparation is done, you will find the data (only part of LibriSpeech) downloaded in `~/.cache/paddle/dataset/speech/libri` and the corresponding manifest files generated in `./data/tiny` as well as a mean stddev file and a vocabulary file. It has to be run for the very first time you run this dataset and is reusable for all further experiments. - Train your own ASR model ```bash sh run_train.sh ``` - `run_train.sh` will start a training job, with training logs printed to stdout and model checkpoint of every pass/epoch saved to `./checkpoints/tiny`. We can resume the training from these checkpoints, or use them for inference, evaluation and deployment. + `run_train.sh` will start a training job, with training logs printed to stdout and model checkpoint of every pass/epoch saved to `./checkpoints/tiny`. These checkpoints could be used for training resuming, inference, evaluation and deployment. - Case inference with an existing model ```bash sh run_infer.sh ``` - `run_infer.sh` will show us some speech-to-text decoding results for several (default: 10) samples with the trained model. The performance might not be good now as the current model is only trained with a toy subset of LibriSpeech. To see the results with a better model, we can download a well-trained (trained for several days, with the complete LibriSpeech) model and do the inference: + `run_infer.sh` will show us some speech-to-text decoding results for several (default: 10) samples with the trained model. The performance might not be good now as the current model is only trained with a toy subset of LibriSpeech. To see the results with a better model, you can download a well-trained (trained for several days, with the complete LibriSpeech) model and do the inference: ```bash sh run_infer_golden.sh @@ -78,7 +78,7 @@ Let's take a tiny sampled subset of [LibriSpeech dataset](http://www.openslr.org sh run_test.sh ``` - `run_test.sh` will evaluate the model with Word Error Rate (or Character Error Rate) measurement. Similarly, we can also download a well-trained model and test its performance: + `run_test.sh` will evaluate the model with Word Error Rate (or Character Error Rate) measurement. Similarly, you can also download a well-trained model and test its performance: ```bash sh run_test_golden.sh @@ -100,7 +100,7 @@ More detailed information are provided in the following sections. Wish you a hap To use your custom data, you only need to generate such manifest files to summarize the dataset. Given such summarized manifests, training, inference and all other modules can be aware of where to access the audio files, as well as their meta data including the transcription labels. -For how to generate such manifest files, please refer to `data/librispeech/librispeech.py`, which download and generate manifests for LibriSpeech dataset. +For how to generate such manifest files, please refer to `data/librispeech/librispeech.py`, which will download data and generate manifest files for LibriSpeech dataset. ### Compute Mean & Stddev for Normalizer @@ -142,7 +142,7 @@ python tools/build_vocab.py --help ## Training a model -`train.py` is the main caller of the training module. We show several examples of usage below. +`train.py` is the main caller of the training module. Examples of usage are shown below. - Start training from scratch with 8 GPUs: @@ -172,9 +172,9 @@ or refer to `example/librispeech/run_train.sh`. ## Data Augmentation Pipeline -Data augmentation has often been a highly effective technique to boost the deep learning performance. We augment our speech data by synthesizing new audios with small random perturbation (label-invariant transformation) added upon raw audios. We don't have to do the syntheses by ourselves, as it is already embedded into the data provider and is done on the fly, randomly for each epoch during training. +Data augmentation has often been a highly effective technique to boost the deep learning performance. We augment our speech data by synthesizing new audios with small random perturbation (label-invariant transformation) added upon raw audios. You don't have to do the syntheses on your own, as it is already embedded into the data provider and is done on the fly, randomly for each epoch during training. -Six optional augmentation components are provided for us to configured and inserted into the processing pipeline. +Six optional augmentation components are provided to be selected, configured and inserted into the processing pipeline. - Volume Perturbation - Speed Perturbation @@ -183,7 +183,7 @@ Six optional augmentation components are provided for us to configured and inser - Noise Perturbation (need background noise audio files) - Impulse Response (need impulse audio files) -In order to inform the trainer of what augmentation components we need and what their processing orders are, we are required to prepare a *augmentation configuration file* in [JSON](http://www.json.org/) format. For example: +In order to inform the trainer of what augmentation components are needed and what their processing orders are, it is required to prepare in advance a *augmentation configuration file* in [JSON](http://www.json.org/) format. For example: ``` [{ @@ -204,13 +204,13 @@ When the `--augment_conf_file` argument of `trainer.py` is set to the path of th For other configuration examples, please refer to `conf/augmenatation.config.example`. -Be careful when we are utilizing the data augmentation technique, as improper augmentation will do harm to the training, due to the enlarged train-test gap. +Be careful when utilizing the data augmentation technique, as improper augmentation will do harm to the training, due to the enlarged train-test gap. ## Inference and Evaluation ### Prepare Language Model -A language model is required to improve the decoder's performance. We have prepared two language models (with lossy compression) for users to download and try. One is for English and the other is for Mandarin. We can simply run this to download the preprared language models: +A language model is required to improve the decoder's performance. We have prepared two language models (with lossy compression) for users to download and try. One is for English and the other is for Mandarin. Users can simply run this to download the preprared language models: ```bash cd models/lm @@ -223,7 +223,7 @@ TODO: any other requirements or tips to add? ### Speech-to-text Inference -An inference module caller `infer.py` is provided for us to infer, decode and visualize speech-to-text results for several given audio clips. It might help to have an intuitive and qualitative evaluation of the ASR model's performance. +An inference module caller `infer.py` is provided to infer, decode and visualize speech-to-text results for several given audio clips. It might help to have an intuitive and qualitative evaluation of the ASR model's performance. - Inference with GPU: @@ -248,7 +248,7 @@ or refer to `example/librispeech/run_infer.sh`. ### Evaluate a Model -To evaluate a model's performance quantitatively, we can run: +To evaluate a model's performance quantitatively, please run: - Evaluation with GPUs: @@ -275,7 +275,7 @@ or refer to `example/librispeech/run_test.sh`. The hyper-parameters $\alpha$ (coefficient for language model scorer) and $\beta$ (coefficient for word count scorer) for the [*CTC beam search decoder*](https://arxiv.org/abs/1408.2873) often have a significant impact on the decoder's performance. It would be better to re-tune them on a validation set when the acoustic model is renewed. -`tools/tune.py` performs a 2-D grid search over the hyper-parameter $\alpha$ and $\beta$. We must provide the range of $\alpha$ and $\beta$, as well as the number of their attempts. +`tools/tune.py` performs a 2-D grid search over the hyper-parameter $\alpha$ and $\beta$. You must provide the range of $\alpha$ and $\beta$, as well as the number of their attempts. - Tuning with GPU: @@ -297,7 +297,7 @@ The hyper-parameters $\alpha$ (coefficient for language model scorer) and $\beta python tools/tune.py --use_gpu False ``` -After tuning, we can reset $\alpha$ and $\beta$ in the inference and evaluation modules to see if they really help improve the ASR performance. +After tuning, you can reset $\alpha$ and $\beta$ in the inference and evaluation modules to see if they really help improve the ASR performance. ```bash python tune.py --help @@ -308,9 +308,9 @@ TODO: add figure. ## Distributed Cloud Training -We provide a cloud training module for users to do the distributed cluster training on [PaddleCloud](https://github.com/PaddlePaddle/cloud), to achieve a much faster training speed with multiple machines. To start with this, please first install PaddleCloud client and register a PaddleCloud account, as described in [PaddleCloud Usage](https://github.com/PaddlePaddle/cloud/blob/develop/doc/usage_cn.md#%E4%B8%8B%E8%BD%BD%E5%B9%B6%E9%85%8D%E7%BD%AEpaddlecloud). +We also provide a cloud training module for users to do the distributed cluster training on [PaddleCloud](https://github.com/PaddlePaddle/cloud), to achieve a much faster training speed with multiple machines. To start with this, please first install PaddleCloud client and register a PaddleCloud account, as described in [PaddleCloud Usage](https://github.com/PaddlePaddle/cloud/blob/develop/doc/usage_cn.md#%E4%B8%8B%E8%BD%BD%E5%B9%B6%E9%85%8D%E7%BD%AEpaddlecloud). -Then, we take the following steps to submit a training job: +Please take the following steps to submit a training job: - Go to directory: @@ -332,7 +332,7 @@ Then, we take the following steps to submit a training job: - Upload these tar files to PaddleCloud filesystem. - Create cloud manifests by replacing local filesystem paths with PaddleCloud filesystem paths. New manifests will be used to inform the cloud jobs of audio files' location and their meta information. - It should be done only once for the very first time we do the cloud training. Later, the data is kept persisitent on the cloud filesystem and reusable for further job submissions. + It should be done only once for the very first time to do the cloud training. Later, the data is kept persisitent on the cloud filesystem and reusable for further job submissions. For argument details please refer to [Train DeepSpeech2 on PaddleCloud](https://github.com/PaddlePaddle/models/tree/develop/deep_speech_2/cloud). @@ -349,7 +349,7 @@ Then, we take the following steps to submit a training job: ```bash sh pcloud_submit.sh ``` - we submit a training job to PaddleCloud. And the job name will be printed when the submission is finished. Now our training job is running well on the PaddleCloud. + a training job has been submitted to PaddleCloud, with the job name printed to the console. - Get training logs @@ -375,9 +375,9 @@ TODO: to be added ## Trying Live Demo with Your Own Voice -Until now, we have trained and tested our ASR model qualitatively (`infer.py`) and quantitatively (`test.py`) with existing audio files. But we have not yet try the model with our own speech. `deploy/demo_server.py` and `deploy/demo_client.py` helps quickly build up a real-time demo ASR engine with the trained model, enabling us to test and play around with the demo, with our own voice. +Until now, an ASR model is trained and tested qualitatively (`infer.py`) and quantitatively (`test.py`) with existing audio files. But it is not yet tested with your own speech. `deploy/demo_server.py` and `deploy/demo_client.py` helps quickly build up a real-time demo ASR engine with the trained model, enabling you to test and play around with the demo, with your own voice. -We start the demo's server in one console by: +To start the demo's server, please run this in one console: ```bash CUDA_VISIBLE_DEVICES=0 \ @@ -387,7 +387,7 @@ python deploy/demo_server.py \ --host_port 8086 ``` -For the machine (might not be the same machine) to run the demo's client, we have to do the following installation before moving on. +For the machine (might not be the same machine) to run the demo's client, please do the following installation before moving on. For example, on MAC OS X: @@ -397,7 +397,7 @@ pip install pyaudio pip install pynput ``` -Then we can start the client in another console by: +Then to start the client, please run this in another console: ```bash CUDA_VISIBLE_DEVICES=0 \ @@ -406,11 +406,11 @@ python -u deploy/demo_client.py \ --host_port 8086 ``` -Now, in the client console, press the `whitespace` key, hold, and start speaking. Until we finish our utterance, we release the key to let the speech-to-text results shown in the console. To quit the client, just press `ESC` key. +Now, in the client console, press the `whitespace` key, hold, and start speaking. Until finishing your utterance, release the key to let the speech-to-text results shown in the console. To quit the client, just press `ESC` key. -Notice that `deploy/demo_client.py` must be run in a machine with a microphone device, while `deploy/demo_server.py` could be run in one without any audio recording hardware, e.g. any remote server machine. Just be careful to set the `host_ip` and `host_port` argument with the actual accessible IP address and port, if the server and client are running with two separate machines. Nothing should be done if they are running in one single machine. +Notice that `deploy/demo_client.py` must be run on a machine with a microphone device, while `deploy/demo_server.py` could be run on one without any audio recording hardware, e.g. any remote server machine. Just be careful to set the `host_ip` and `host_port` argument with the actual accessible IP address and port, if the server and client are running with two separate machines. Nothing should be done if they are running on one single machine. -We can also refer to `examples/mandarin/run_demo_server.sh` for example, which will first download a pre-trained Mandarin model (trained with 3000 hours of internal speech data) and then start the demo server with the model. With running `examples/mandarin/run_demo_client.sh`, we can speak Mandarin to test it. If we would like to try some other models, just update `--model_path` argument in the script.   +Please also refer to `examples/mandarin/run_demo_server.sh`, which will first download a pre-trained Mandarin model (trained with 3000 hours of internal speech data) and then start the demo server with the model. With running `examples/mandarin/run_demo_client.sh`, you can speak Mandarin to test it. If you would like to try some other models, just update `--model_path` argument in the script.   For more help on arguments: diff --git a/deploy/demo_server.py b/deploy/demo_server.py index 2d3931f74..a7157001c 100644 --- a/deploy/demo_server.py +++ b/deploy/demo_server.py @@ -46,7 +46,7 @@ add_arg('vocab_path', str, 'data/librispeech/eng_vocab.txt', "Filepath of vocabulary.") add_arg('model_path', str, - './checkpoints/params.latest.tar.gz', + './checkpoints/libri/params.latest.tar.gz', "If None, the training starts from scratch, " "otherwise, it resumes from the pre-trained model.") add_arg('lang_model_path', str, diff --git a/examples/librispeech/run_train.sh b/examples/librispeech/run_train.sh index 5485475e9..07575dde1 100644 --- a/examples/librispeech/run_train.sh +++ b/examples/librispeech/run_train.sh @@ -3,6 +3,7 @@ pushd ../.. > /dev/null # train model +# if you wish to resume from an exists model, uncomment --init_model_path CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 \ python -u train.py \ --batch_size=512 \ diff --git a/examples/tiny/run_train.sh b/examples/tiny/run_train.sh index c66ec4e56..74d82712e 100644 --- a/examples/tiny/run_train.sh +++ b/examples/tiny/run_train.sh @@ -3,6 +3,7 @@ pushd ../.. > /dev/null # train model +# if you wish to resume from an exists model, uncomment --init_model_path CUDA_VISIBLE_DEVICES=0,1,2,3 \ python -u train.py \ --batch_size=16 \ diff --git a/infer.py b/infer.py index 73e200b49..d9c4c6776 100644 --- a/infer.py +++ b/infer.py @@ -38,10 +38,10 @@ add_arg('vocab_path', str, 'data/librispeech/vocab.txt', "Filepath of vocabulary.") add_arg('lang_model_path', str, - 'model_zoo/lm/common_crawl_00.prune01111.trie.klm', + 'models/lm/common_crawl_00.prune01111.trie.klm', "Filepath for language model.") add_arg('model_path', str, - './checkpoints/params.latest.tar.gz', + './checkpoints/libri/params.latest.tar.gz', "If None, the training starts from scratch, " "otherwise, it resumes from the pre-trained model.") add_arg('decoding_method', str, diff --git a/test.py b/test.py index 791bfd585..18089f332 100644 --- a/test.py +++ b/test.py @@ -39,11 +39,11 @@ add_arg('vocab_path', str, 'data/librispeech/vocab.txt', "Filepath of vocabulary.") add_arg('model_path', str, - './checkpoints/params.latest.tar.gz', + './checkpoints/libri/params.latest.tar.gz', "If None, the training starts from scratch, " "otherwise, it resumes from the pre-trained model.") add_arg('lang_model_path', str, - 'model_zoo/lm/common_crawl_00.prune01111.trie.klm', + 'models/lm/common_crawl_00.prune01111.trie.klm', "Filepath for language model.") add_arg('decoding_method', str, 'ctc_beam_search', diff --git a/tools/tune.py b/tools/tune.py index 25e495f19..96c25a3eb 100644 --- a/tools/tune.py +++ b/tools/tune.py @@ -44,10 +44,10 @@ add_arg('vocab_path', str, 'data/librispeech/vocab.txt', "Filepath of vocabulary.") add_arg('lang_model_path', str, - 'model_zoo/lm/common_crawl_00.prune01111.trie.klm', + 'models/lm/common_crawl_00.prune01111.trie.klm', "Filepath for language model.") add_arg('model_path', str, - './checkpoints/params.latest.tar.gz', + './checkpoints/libri/params.latest.tar.gz', "If None, the training starts from scratch, " "otherwise, it resumes from the pre-trained model.") add_arg('error_rate_type', str, diff --git a/train.py b/train.py index bbf1cd729..406484a18 100644 --- a/train.py +++ b/train.py @@ -48,7 +48,7 @@ add_arg('init_model_path', str, "If None, the training starts from scratch, " "otherwise, it resumes from the pre-trained model.") add_arg('output_model_dir', str, - "./checkpoints", + "./checkpoints/libri", "Directory for saving checkpoints.") add_arg('augment_conf_path',str, 'conf/augmentation.config', From 351f61e36664dd78b3100445c0c22151bf25129b Mon Sep 17 00:00:00 2001 From: Xinghai Sun Date: Wed, 13 Sep 2017 17:34:59 +0800 Subject: [PATCH 9/9] Update RAEDME.md and librispeech.py by following Yaming's review. --- README.md | 2 +- data/librispeech/librispeech.py | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 055bd439e..9d9d4c77e 100644 --- a/README.md +++ b/README.md @@ -36,7 +36,7 @@ sh setup.sh Several shell scripts provided in `./examples` will help us to quickly give it a try, for most major modules, including data preparation, model training, case inference and model evaluation, with a few public dataset (e.g. [LibriSpeech](http://www.openslr.org/12/), [Aishell](http://www.openslr.org/33)). Reading these examples will also help you to understand how to make it work with your own data. -Some of the scripts in `./examples` are configured with 8 GPUs. If you don't have 8 GPUs available, please modify `CUDA_VISIBLE_DEVICES` and `--trainer_count`. If you don't have any GPU available, please set `--use_gpu` to False to use CPUs instead. +Some of the scripts in `./examples` are configured with 8 GPUs. If you don't have 8 GPUs available, please modify `CUDA_VISIBLE_DEVICES` and `--trainer_count`. If you don't have any GPU available, please set `--use_gpu` to False to use CPUs instead. Besides, if out-of-memory problem occurs, just reduce `--batch_size` to fit. Let's take a tiny sampled subset of [LibriSpeech dataset](http://www.openslr.org/12/) for instance. diff --git a/data/librispeech/librispeech.py b/data/librispeech/librispeech.py index 0709136e2..8dce359a4 100644 --- a/data/librispeech/librispeech.py +++ b/data/librispeech/librispeech.py @@ -64,7 +64,7 @@ def download(url, md5sum, target_dir): filepath = os.path.join(target_dir, url.split("/")[-1]) if not (os.path.exists(filepath) and md5file(filepath) == md5sum): print("Downloading %s ..." % url) - ret = os.system("wget -c " + url + " -P " + target_dir) + os.system("wget -c " + url + " -P " + target_dir) print("\nMD5 Chesksum %s ..." % filepath) if not md5file(filepath) == md5sum: raise RuntimeError("MD5 checksum failed.")