-
-| # of GPU | Acceleration Rate |
-| -------- | --------------: |
-| 1 | 1.00 X |
-| 2 | 1.98 X |
-| 4 | 3.73 X |
-| 8 | 6.95 X |
-
-`utils/profile.sh` provides such a demo profiling tool, you can change it as need.
diff --git a/doc/images/ds2offlineModel.png b/docs/images/ds2offlineModel.png
similarity index 100%
rename from doc/images/ds2offlineModel.png
rename to docs/images/ds2offlineModel.png
diff --git a/doc/images/ds2onlineModel.png b/docs/images/ds2onlineModel.png
similarity index 100%
rename from doc/images/ds2onlineModel.png
rename to docs/images/ds2onlineModel.png
diff --git a/doc/src/augmentation.md b/docs/src/augmentation.md
similarity index 100%
rename from doc/src/augmentation.md
rename to docs/src/augmentation.md
diff --git a/doc/src/data_preparation.md b/docs/src/data_preparation.md
similarity index 100%
rename from doc/src/data_preparation.md
rename to docs/src/data_preparation.md
diff --git a/doc/src/deepspeech_architecture.md b/docs/src/deepspeech_architecture.md
similarity index 68%
rename from doc/src/deepspeech_architecture.md
rename to docs/src/deepspeech_architecture.md
index ffa37aff..580b1388 100644
--- a/doc/src/deepspeech_architecture.md
+++ b/docs/src/deepspeech_architecture.md
@@ -20,7 +20,7 @@ The arcitecture of the model is shown in Fig.1.
### Data Preparation
#### Vocabulary
-For English data, the vocabulary dictionary is composed of 26 English characters with " ' ", space, \
@@ -162,9 +167,9 @@ The arcitecture of the model is shown in Fig.2.
-For data preparation, decoder, the deepspeech2 offline model is same with the deepspeech2 online model.
+For data preparation and decoder, the deepspeech2 offline model is same with the deepspeech2 online model.
- The code of encoder and decoder for deepspeech2 offline model is in:
+The code of encoder and decoder for deepspeech2 offline model is in:
```
vi deepspeech/models/ds2/deepspeech2.py
```
diff --git a/doc/src/feature_list.md b/docs/src/feature_list.md
similarity index 100%
rename from doc/src/feature_list.md
rename to docs/src/feature_list.md
diff --git a/doc/src/getting_started.md b/docs/src/getting_started.md
similarity index 100%
rename from doc/src/getting_started.md
rename to docs/src/getting_started.md
diff --git a/doc/src/install.md b/docs/src/install.md
similarity index 95%
rename from doc/src/install.md
rename to docs/src/install.md
index 01049a2f..8cecba12 100644
--- a/doc/src/install.md
+++ b/docs/src/install.md
@@ -4,15 +4,16 @@ To avoid the trouble of environment setup, [running in Docker container](#runnin
## Prerequisites
- Python >= 3.7
-- PaddlePaddle 2.0.0 or later (please refer to the [Installation Guide](https://www.paddlepaddle.org.cn/documentation/docs/en/beginners_guide/index_en.html))
+- PaddlePaddle latest version (please refer to the [Installation Guide](https://www.paddlepaddle.org.cn/documentation/docs/en/beginners_guide/index_en.html))
-## Setup
+## Setup (Important)
- Make sure these libraries or tools installed: `pkg-config`, `flac`, `ogg`, `vorbis`, `boost`, `sox, and `swig`, e.g. installing them via `apt-get`:
```bash
sudo apt-get install -y sox pkg-config libflac-dev libogg-dev libvorbis-dev libboost-dev swig python3-dev
```
+The version of `swig` should >= 3.0
or, installing them via `yum`:
diff --git a/doc/src/ngram_lm.md b/docs/src/ngram_lm.md
similarity index 100%
rename from doc/src/ngram_lm.md
rename to docs/src/ngram_lm.md
diff --git a/doc/src/reference.md b/docs/src/reference.md
similarity index 79%
rename from doc/src/reference.md
rename to docs/src/reference.md
index 341e1361..d3676fff 100644
--- a/doc/src/reference.md
+++ b/docs/src/reference.md
@@ -1,5 +1,7 @@
# Reference
+We refer these repos to build `model` and `engine`:
+
* [delta](https://github.com/Delta-ML/delta.git)
* [espnet](https://github.com/espnet/espnet.git)
* [kaldi](https://github.com/kaldi-asr/kaldi.git)
diff --git a/doc/src/released_model.md b/docs/src/released_model.md
similarity index 100%
rename from doc/src/released_model.md
rename to docs/src/released_model.md
diff --git a/examples/aishell/s0/local/train.sh b/examples/aishell/s0/local/train.sh
index 85d1d42c..668ad0ea 100755
--- a/examples/aishell/s0/local/train.sh
+++ b/examples/aishell/s0/local/train.sh
@@ -19,6 +19,7 @@ fi
mkdir -p exp
+# seed may break model convergence
seed=10086
if [ ${seed} != 0 ]; then
export FLAGS_cudnn_deterministic=True
diff --git a/examples/aishell/s1/local/train.sh b/examples/aishell/s1/local/train.sh
index 2861e11e..5097d4d0 100755
--- a/examples/aishell/s1/local/train.sh
+++ b/examples/aishell/s1/local/train.sh
@@ -1,37 +1,49 @@
#!/bin/bash
-if [ $# != 2 ];then
- echo "usage: CUDA_VISIBLE_DEVICES=0 ${0} config_path ckpt_name"
- exit -1
-fi
+profiler_options=
+benchmark_batch_size=0
+benchmark_max_step=0
+
+# seed may break model convergence
+seed=0
+
+source ${MAIN_ROOT}/utils/parse_options.sh || exit 1;
ngpu=$(echo $CUDA_VISIBLE_DEVICES | awk -F "," '{print NF}')
echo "using $ngpu gpus..."
-config_path=$1
-ckpt_name=$2
-
device=gpu
if [ ${ngpu} == 0 ];then
device=cpu
fi
-echo "using ${device}..."
-
-mkdir -p exp
-seed=10086
-if [ ${seed} != 0]; then
+if [ ${seed} != 0 ]; then
export FLAGS_cudnn_deterministic=True
+ echo "using seed $seed & FLAGS_cudnn_deterministic=True ..."
fi
+if [ $# != 2 ];then
+ echo "usage: CUDA_VISIBLE_DEVICES=0 ${0} config_path ckpt_name"
+ exit -1
+fi
+
+config_path=$1
+ckpt_name=$2
+
+mkdir -p exp
+
python3 -u ${BIN_DIR}/train.py \
+--seed ${seed} \
--device ${device} \
--nproc ${ngpu} \
--config ${config_path} \
--output exp/${ckpt_name} \
---seed ${seed}
+--profiler-options "${profiler_options}" \
+--benchmark-batch-size ${benchmark_batch_size} \
+--benchmark-max-step ${benchmark_max_step}
+
-if [ ${seed} != 0 ]; then
+if [ ${seed} != 0 ]; then
unset FLAGS_cudnn_deterministic
fi
diff --git a/examples/callcenter/s1/local/train.sh b/examples/callcenter/s1/local/train.sh
index 6e63df83..d5dc15b0 100755
--- a/examples/callcenter/s1/local/train.sh
+++ b/examples/callcenter/s1/local/train.sh
@@ -19,8 +19,9 @@ echo "using ${device}..."
mkdir -p exp
-seed=10086
-if [ ${seed} != 0]; then
+# seed may break model convergence
+seed=0
+if [ ${seed} != 0 ]; then
export FLAGS_cudnn_deterministic=True
fi
diff --git a/examples/cc-cedict/README.md b/examples/cc-cedict/README.md
index e69de29b..513fca53 100644
--- a/examples/cc-cedict/README.md
+++ b/examples/cc-cedict/README.md
@@ -0,0 +1,58 @@
+# [CC-CEDICT](https://cc-cedict.org/wiki/)
+
+What is CC-CEDICT?
+CC-CEDICT is a continuation of the CEDICT project.
+The objective of the CEDICT project was to create an online, downloadable (as opposed to searchable-only) public-domain Chinese-English dictionary.
+CEDICT was started by Paul Andrew Denisowski in October 1997.
+For the most part, the project is modeled on Jim Breen's highly successful EDICT (Japanese-English dictionary) project and is intended to be a collaborative effort,
+with users providing entries and corrections to the main file.
+
+
+## Parse CC-CEDICT to Json format
+
+1. Parse to Json
+
+```
+run.sh
+```
+
+2. Result
+
+```
+exp/
+|-- cedict
+`-- cedict.json
+
+0 directories, 2 files
+```
+
+```
+4c4bffc84e24467fe1b2ea9ba37ed6b6 exp/cedict
+3adf504dacd13886f88cc9fe3b37c75d exp/cedict.json
+```
+
+```
+==> exp/cedict <==
+# CC-CEDICT
+# Community maintained free Chinese-English dictionary.
+#
+# Published by MDBG
+#
+# License:
+# Creative Commons Attribution-ShareAlike 4.0 International License
+# https://creativecommons.org/licenses/by-sa/4.0/
+#
+# Referenced works:
+
+==> exp/cedict.json <==
+{"traditional": "2019\u51a0\u72c0\u75c5\u6bd2\u75c5", "simplified": "2019\u51a0\u72b6\u75c5\u6bd2\u75c5", "pinyin": "er4 ling2 yi1 jiu3 guan1 zhuang4 bing4 du2 bing4", "english": "COVID-19, the coronavirus disease identified in 2019"}
+{"traditional": "21\u4e09\u9ad4\u7d9c\u5408\u75c7", "simplified": "21\u4e09\u4f53\u7efc\u5408\u75c7", "pinyin": "er4 shi2 yi1 san1 ti3 zong1 he2 zheng4", "english": "trisomy"}
+{"traditional": "3C", "simplified": "3C", "pinyin": "san1 C", "english": "abbr. for computers, communications, and consumer electronics"}
+{"traditional": "3P", "simplified": "3P", "pinyin": "san1 P", "english": "(slang) threesome"}
+{"traditional": "3Q", "simplified": "3Q", "pinyin": "san1 Q", "english": "(Internet slang) thank you (loanword)"}
+{"traditional": "421", "simplified": "421", "pinyin": "si4 er4 yi1", "english": "four grandparents, two parents and an only child"}
+{"traditional": "502\u81a0", "simplified": "502\u80f6", "pinyin": "wu3 ling2 er4 jiao1", "english": "cyanoacrylate glue"}
+{"traditional": "88", "simplified": "88", "pinyin": "ba1 ba1", "english": "(Internet slang) bye-bye (alternative for \u62dc\u62dc[bai2 bai2])"}
+{"traditional": "996", "simplified": "996", "pinyin": "jiu3 jiu3 liu4", "english": "9am-9pm, six days a week (work schedule)"}
+{"traditional": "A", "simplified": "A", "pinyin": "A", "english": "(slang) (Tw) to steal"}
+```
diff --git a/examples/chinese_g2p/README.md b/examples/chinese_g2p/README.md
deleted file mode 100644
index e3fdfe68..00000000
--- a/examples/chinese_g2p/README.md
+++ /dev/null
@@ -1,5 +0,0 @@
-# Download Baker dataset
-
-Baker dataset has to be downloaded mannually and moved to 'data/', because you will have to pass the CATTCHA from a browswe to download the dataset.
-
-Download URL https://test.data-baker.com/#/data/index/source.
diff --git a/examples/chinese_g2p/.gitignore b/examples/g2p/.gitignore
similarity index 100%
rename from examples/chinese_g2p/.gitignore
rename to examples/g2p/.gitignore
diff --git a/examples/g2p/README.md b/examples/g2p/README.md
new file mode 100644
index 00000000..4ec5922b
--- /dev/null
+++ b/examples/g2p/README.md
@@ -0,0 +1,3 @@
+# G2P
+
+* zh - Chinese G2P
diff --git a/examples/g2p/zh/README.md b/examples/g2p/zh/README.md
new file mode 100644
index 00000000..de557356
--- /dev/null
+++ b/examples/g2p/zh/README.md
@@ -0,0 +1,93 @@
+# G2P
+
+* WS
+jieba
+* G2P
+pypinyin
+* Tone sandhi
+simple
+
+We recommend using [Paraket](https://github.com/PaddlePaddle/Parakeet] [TextFrontEnd](https://github.com/PaddlePaddle/Parakeet/blob/develop/parakeet/frontend/__init__.py) to do G2P.
+The phoneme set should be changed, you can reference `examples/thchs30/a0/data/dict/syllable.lexicon`.
+
+## Download Baker dataset
+
+[Baker](https://test.data-baker.com/#/data/index/source) dataset has to be downloaded mannually and moved to './data',
+because you will have to pass the `CATTCHA` from a browswe to download the dataset.
+
+
+## RUN
+
+```
+. path.sh
+./run.sh
+```
+
+## Result
+
+```
+exp/
+|-- 000001-010000.txt
+|-- ref.pinyin
+|-- trans.jieba.pinyin
+`-- trans.pinyin
+
+0 directories, 4 files
+```
+
+```
+4f5a368441eb16aaf43dc1972f8b63dd exp/000001-010000.txt
+01707896391c2de9b6fc4a39654be942 exp/ref.pinyin
+43380ef160f65a23a3a0544700aa49b8 exp/trans.jieba.pinyin
+8e6ff1fc22d8e8584082e804e8bcdeb7 exp/trans.pinyin
+```
+
+```
+==> exp/000001-010000.txt <==
+000001 卡尔普#2陪外孙#1玩滑梯#4。
+ ka2 er2 pu3 pei2 wai4 sun1 wan2 hua2 ti1
+000002 假语村言#2别再#1拥抱我#4。
+ jia2 yu3 cun1 yan2 bie2 zai4 yong1 bao4 wo3
+000003 宝马#1配挂#1跛骡鞍#3，貂蝉#1怨枕#2董翁榻#4。
+ bao2 ma3 pei4 gua4 bo3 luo2 an1 diao1 chan2 yuan4 zhen3 dong3 weng1 ta4
+000004 邓小平#2与#1撒切尔#2会晤#4。
+ deng4 xiao3 ping2 yu3 sa4 qie4 er3 hui4 wu4
+000005 老虎#1幼崽#2与#1宠物犬#1玩耍#4。
+ lao2 hu3 you4 zai3 yu2 chong3 wu4 quan3 wan2 shua3
+
+==> exp/ref.pinyin <==
+000001 ka2 er2 pu3 pei2 wai4 sun1 wan2 hua2 ti1
+000002 jia2 yu3 cun1 yan2 bie2 zai4 yong1 bao4 wo3
+000003 bao2 ma3 pei4 gua4 bo3 luo2 an1 diao1 chan2 yuan4 zhen3 dong3 weng1 ta4
+000004 deng4 xiao3 ping2 yu3 sa4 qie4 er3 hui4 wu4
+000005 lao2 hu3 you4 zai3 yu2 chong3 wu4 quan3 wan2 shua3
+000006 shen1 chang2 yue1 wu2 chi3 er4 cun4 wu3 fen1 huo4 yi3 shang4
+000007 zhao4 di2 yue1 cao2 yun2 teng2 qu4 gui3 wu1
+000008 zhan2 pin3 sui1 you3 zhan3 yuan2 que4 tui2
+000009 yi2 san3 ju1 er2 tong2 he2 you4 tuo1 er2 tong2 wei2 zhu3
+000010 ke1 te4 ni1 shen1 chuan1 bao4 wen2 da4 yi1
+
+==> exp/trans.jieba.pinyin <==
+000001 ka3 er3 pu3 pei2 wai4 sun1 wan2 hua2 ti1
+000002 jia3 yu3 cun1 yan2 bie2 zai4 yong1 bao4 wo3
+000003 bao3 ma3 pei4 gua4 bo3 luo2 an1 diao1 chan2 yuan4 zhen3 dong3 weng1 ta4
+000004 deng4 xiao3 ping2 yu3 sa1 qie4 er3 hui4 wu4
+000005 lao3 hu3 you4 zai3 yu3 chong3 wu4 quan3 wan2 shua3
+000006 shen1 chang2 yue1 wu3 chi3 er4 cun4 wu3 fen1 huo4 yi3 shang4
+000007 zhao4 di2 yue1 cao2 yun2 teng2 qu4 gui3 wu1
+000008 zhan3 pin3 sui1 you3 zhan3 yuan2 que4 tui2
+000009 yi3 san3 ju1 er2 tong2 he2 you4 tuo1 er2 tong2 wei2 zhu3
+000010 ke1 te4 ni1 shen1 chuan1 bao4 wen2 da4 yi1
+
+==> exp/trans.pinyin <==
+000001 ka3 er3 pu3 pei2 wai4 sun1 wan2 hua2 ti1
+000002 jia3 yu3 cun1 yan2 bie2 zai4 yong1 bao4 wo3
+000003 bao3 ma3 pei4 gua4 bo3 luo2 an1 diao1 chan2 yuan4 zhen3 dong3 weng1 ta4
+000004 deng4 xiao3 ping2 yu3 sa1 qie4 er3 hui4 wu4
+000005 lao3 hu3 you4 zai3 yu3 chong3 wu4 quan3 wan2 shua3
+000006 shen1 chang2 yue1 wu3 chi3 er4 cun4 wu3 fen1 huo4 yi3 shang4
+000007 zhao4 di2 yue1 cao2 yun2 teng2 qu4 gui3 wu1
+000008 zhan3 pin3 sui1 you3 zhan3 yuan2 que4 tui2
+000009 yi3 san3 ju1 er2 tong2 he2 you4 tuo1 er2 tong2 wei2 zhu3
+000010 ke1 te4 ni1 shen1 chuan1 bao4 wen2 da4 yi1
+```
diff --git a/examples/chinese_g2p/local/convert_transcription.py b/examples/g2p/zh/local/convert_transcription.py
similarity index 100%
rename from examples/chinese_g2p/local/convert_transcription.py
rename to examples/g2p/zh/local/convert_transcription.py
diff --git a/examples/chinese_g2p/local/extract_pinyin_label.py b/examples/g2p/zh/local/extract_pinyin_label.py
similarity index 100%
rename from examples/chinese_g2p/local/extract_pinyin_label.py
rename to examples/g2p/zh/local/extract_pinyin_label.py
diff --git a/examples/chinese_g2p/local/ignore_sandhi.py b/examples/g2p/zh/local/ignore_sandhi.py
similarity index 100%
rename from examples/chinese_g2p/local/ignore_sandhi.py
rename to examples/g2p/zh/local/ignore_sandhi.py
diff --git a/examples/chinese_g2p/local/prepare_dataset.sh b/examples/g2p/zh/local/prepare_dataset.sh
similarity index 100%
rename from examples/chinese_g2p/local/prepare_dataset.sh
rename to examples/g2p/zh/local/prepare_dataset.sh
diff --git a/examples/chinese_g2p/path.sh b/examples/g2p/zh/path.sh
similarity index 82%
rename from examples/chinese_g2p/path.sh
rename to examples/g2p/zh/path.sh
index 482177dc..f475ed83 100644
--- a/examples/chinese_g2p/path.sh
+++ b/examples/g2p/zh/path.sh
@@ -1,4 +1,4 @@
-export MAIN_ROOT=`realpath ${PWD}/../../`
+export MAIN_ROOT=`realpath ${PWD}/../../../`
export PATH=${MAIN_ROOT}:${MAIN_ROOT}/utils:${PATH}
export LC_ALL=C
diff --git a/examples/chinese_g2p/requirements.txt b/examples/g2p/zh/requirements.txt
similarity index 100%
rename from examples/chinese_g2p/requirements.txt
rename to examples/g2p/zh/requirements.txt
diff --git a/examples/chinese_g2p/run.sh b/examples/g2p/zh/run.sh
similarity index 82%
rename from examples/chinese_g2p/run.sh
rename to examples/g2p/zh/run.sh
index 8197dce4..25b71311 100755
--- a/examples/chinese_g2p/run.sh
+++ b/examples/g2p/zh/run.sh
@@ -6,16 +6,19 @@ stage=-1
stop_stage=100
exp_dir=exp
-data_dir=data
+data=data
source ${MAIN_ROOT}/utils/parse_options.sh || exit -1
mkdir -p ${exp_dir}
+if [ $stage -le -1 ] && [ $stop_stage -ge -1 ];then
+ test -e ${data}/BZNSYP.rar || { echo "Please download BZNSYP.rar and put it in ${data}; exit -1; }
+fi
if [ $stage -le 0 ] && [ $stop_stage -ge 0 ];then
echo "stage 0: Extracting Prosody Labeling"
- bash local/prepare_dataset.sh --exp-dir ${exp_dir} --data-dir ${data_dir}
+ bash local/prepare_dataset.sh --exp-dir ${exp_dir} --data-dir ${data}
fi
# convert transcription in chinese into pinyin with pypinyin or jieba+pypinyin
diff --git a/examples/librispeech/s0/conf/deepspeech2.yaml b/examples/librispeech/s0/conf/deepspeech2.yaml
index d5b1ed91..3f1a376f 100644
--- a/examples/librispeech/s0/conf/deepspeech2.yaml
+++ b/examples/librispeech/s0/conf/deepspeech2.yaml
@@ -11,7 +11,7 @@ data:
max_output_input_ratio: .inf
collator:
- batch_size: 15
+ batch_size: 20
mean_std_filepath: data/mean_std.json
unit_type: char
vocab_filepath: data/vocab.txt
@@ -45,7 +45,7 @@ model:
training:
n_epoch: 50
- accum_grad: 4
+ accum_grad: 1
lr: 1e-3
lr_decay: 0.83
weight_decay: 1e-06
diff --git a/examples/librispeech/s0/local/train.sh b/examples/librispeech/s0/local/train.sh
index c95659ac..6aee372a 100755
--- a/examples/librispeech/s0/local/train.sh
+++ b/examples/librispeech/s0/local/train.sh
@@ -20,7 +20,8 @@ echo "using ${device}..."
mkdir -p exp
-seed=10086
+# seed may break model convergence
+seed=0
if [ ${seed} != 0 ]; then
export FLAGS_cudnn_deterministic=True
fi
diff --git a/examples/librispeech/s1/conf/augmentation.json b/examples/librispeech/s1/conf/augmentation.json
index 8e6e9704..40a5b790 100644
--- a/examples/librispeech/s1/conf/augmentation.json
+++ b/examples/librispeech/s1/conf/augmentation.json
@@ -19,17 +19,17 @@
{
"type": "specaug",
"params": {
+ "W": 0,
+ "warp_mode": "PIL",
"F": 10,
- "T": 50,
"n_freq_masks": 2,
+ "T": 50,
"n_time_masks": 2,
"p": 1.0,
- "W": 80,
"adaptive_number_ratio": 0,
"adaptive_size_ratio": 0,
"max_n_time_masks": 20,
- "replace_with_zero": true,
- "warp_mode": "PIL"
+ "replace_with_zero": true
},
"prob": 1.0
}
diff --git a/examples/librispeech/s1/conf/transformer.yaml b/examples/librispeech/s1/conf/transformer.yaml
index 4aa7b915..fe9cab06 100644
--- a/examples/librispeech/s1/conf/transformer.yaml
+++ b/examples/librispeech/s1/conf/transformer.yaml
@@ -33,7 +33,7 @@ collator:
keep_transcription_text: False
sortagrad: True
shuffle_method: batch_shuffle
- num_workers: 0
+ num_workers: 2
# network architecture
@@ -74,7 +74,7 @@ model:
training:
- n_epoch: 120
+ n_epoch: 120
accum_grad: 2
global_grad_clip: 5.0
optim: adam
diff --git a/examples/librispeech/s1/local/train.sh b/examples/librispeech/s1/local/train.sh
index 17a9e28d..f905b766 100755
--- a/examples/librispeech/s1/local/train.sh
+++ b/examples/librispeech/s1/local/train.sh
@@ -19,7 +19,8 @@ echo "using ${device}..."
mkdir -p exp
-seed=10086
+# seed may break model convergence
+seed=0
if [ ${seed} != 0 ]; then
export FLAGS_cudnn_deterministic=True
fi
@@ -31,7 +32,7 @@ python3 -u ${BIN_DIR}/train.py \
--output exp/${ckpt_name} \
--seed ${seed}
-if [ ${seed} != 0]; then
+if [ ${seed} != 0 ]; then
unset FLAGS_cudnn_deterministic
fi
diff --git a/examples/librispeech/s2/local/train.sh b/examples/librispeech/s2/local/train.sh
index a75e2bb2..66754201 100755
--- a/examples/librispeech/s2/local/train.sh
+++ b/examples/librispeech/s2/local/train.sh
@@ -19,7 +19,8 @@ echo "using ${device}..."
mkdir -p exp
-seed=10086
+# seed may break model convergence
+seed=0
if [ ${seed} != 0 ]; then
export FLAGS_cudnn_deterministic=True
fi
diff --git a/examples/ngram_lm/READEME.md b/examples/ngram_lm/READEME.md
new file mode 100644
index 00000000..84e1380c
--- /dev/null
+++ b/examples/ngram_lm/READEME.md
@@ -0,0 +1,3 @@
+# Ngram LM
+
+* s0 - kenlm ngram lm
diff --git a/examples/ngram_lm/s0/.gitignore b/examples/ngram_lm/s0/.gitignore
new file mode 100644
index 00000000..b20d93aa
--- /dev/null
+++ b/examples/ngram_lm/s0/.gitignore
@@ -0,0 +1 @@
+data/lm
diff --git a/examples/ngram_lm/s0/README.md b/examples/ngram_lm/s0/README.md
index 698d7c29..65916ec5 100644
--- a/examples/ngram_lm/s0/README.md
+++ b/examples/ngram_lm/s0/README.md
@@ -2,6 +2,95 @@
Train chinese chararctor ngram lm by [kenlm](https://github.com/kpu/kenlm).
+## Run
```
+. path.sh
bash run.sh
```
+
+## Results
+
+```
+exp/
+|-- text
+|-- text.char.tn
+|-- text.word.tn
+|-- text_zh_char_o5_p0_1_2_4_4_a22_q8_b8.arpa
+|-- text_zh_char_o5_p0_1_2_4_4_a22_q8_b8.arpa.klm.bin
+|-- text_zh_word_o3_p0_0_0_a22_q8_b8.arpa
+`-- text_zh_word_o3_p0_0_0_a22_q8_b8.arpa.klm.bin
+
+0 directories, 7 files
+```
+
+```
+3ae083627b9b6cef1a82d574d8483f97 exp/text
+d97da252d2a63a662af22f98af30cb8c exp/text.char.tn
+c18b03005bd094dbfd9b46442be361fd exp/text.word.tn
+73dbf50097896eda33985e11e1ba9a3a exp/text_zh_char_o5_p0_1_2_4_4_a22_q8_b8.arpa
+01334e2044c474b99c4f2ffbed790626 exp/text_zh_char_o5_p0_1_2_4_4_a22_q8_b8.arpa.klm.bin
+36a42de548045b54662411ae7982c77f exp/text_zh_word_o3_p0_0_0_a22_q8_b8.arpa
+332422803ffd73dd7ffd16cd2b0abcd5 exp/text_zh_word_o3_p0_0_0_a22_q8_b8.arpa.klm.bin
+```
+
+```
+==> exp/text <==
+少先队员因该为老人让坐
+祛痘印可以吗？有效果吗？
+不知这款牛奶口感怎样？ 小孩子喝行吗！
+是转基因油?
+我家宝宝13斤用多大码的
+会起坨吗？
+请问给送上楼吗？
+亲是送赁上门吗
+送货时候有外包装没有还是直接发货过来
+会不会有坏的？
+
+==> exp/text.char.tn <==
+少 先 队 员 因 该 为 老 人 让 坐
+祛 痘 印 可 以 吗 有 效 果 吗
+不 知 这 款 牛 奶 口 感 怎 样 小 孩 子 喝 行 吗
+是 转 基 因 油
+我 家 宝 宝 十 三 斤 用 多 大 码 的
+会 起 坨 吗
+请 问 给 送 上 楼 吗
+亲 是 送 赁 上 门 吗
+送 货 时 候 有 外 包 装 没 有 还 是 直 接 发 货 过 来
+会 不 会 有 坏 的
+
+==> exp/text.word.tn <==
+少先队员 因该 为 老人 让 坐
+祛痘 印 可以 吗 有 效果 吗
+不知 这 款 牛奶 口感 怎样 小孩子 喝行 吗
+是 转基因 油
+我家 宝宝 十三斤 用多大码 的
+会起 坨 吗
+请问 给 送 上楼 吗
+亲是 送赁 上门 吗
+送货 时候 有 外包装 没有 还是 直接 发货 过来
+会 不会 有坏 的
+
+==> exp/text_zh_char_o5_p0_1_2_4_4_a22_q8_b8.arpa <==
+\data\
+ngram 1=587
+ngram 2=395
+ngram 3=100
+ngram 4=2
+ngram 5=0
+
+\1-grams:
+-3.272324 -0.36706257
+
+==> exp/text_zh_word_o3_p0_0_0_a22_q8_b8.arpa <==
+\data\
+ngram 1=689
+ngram 2=1398
+ngram 3=1506
+
+\1-grams:
+-3.1755018 -0.23069073
+-1.2318869 0
+-3.067262 少先队员 -0.051341705
+```
diff --git a/examples/spm/README.md b/examples/spm/README.md
index 3109d3ff..fc4478eb 100644
--- a/examples/spm/README.md
+++ b/examples/spm/README.md
@@ -1,7 +1,96 @@
# [SentencePiece Model](https://github.com/google/sentencepiece)
+## Run
Train a `spm` model for English tokenizer.
```
+. path.sh
bash run.sh
```
+
+## Results
+
+```
+data/
+└── lang_char
+ ├── input.bpe
+ ├── input.decode
+ ├── input.txt
+ ├── train_unigram100.model
+ ├── train_unigram100_units.txt
+ └── train_unigram100.vocab
+
+1 directory, 6 files
+```
+
+```
+b5a230c26c61db5c36f34e503102f936 data/lang_char/input.bpe
+ec5a9b24acc35469229e41256ceaf77d data/lang_char/input.decode
+ec5a9b24acc35469229e41256ceaf77d data/lang_char/input.txt
+124bf3fe7ce3b73b1994234c15268577 data/lang_char/train_unigram100.model
+0df2488cc8eaace95eb12713facb5cf0 data/lang_char/train_unigram100_units.txt
+46360cac35c751310e8e8ffd3a034cb5 data/lang_char/train_unigram100.vocab
+```
+
+```
+==> data/lang_char/input.bpe <==
+▁mi ster ▁quilter ▁ is ▁the ▁a p ost le ▁o f ▁the ▁mi d d le ▁c las s es ▁ and ▁we ▁ar e ▁g l a d ▁ to ▁we l c om e ▁h is ▁g o s pe l
+▁ n or ▁ is ▁mi ster ▁quilter ' s ▁ma nne r ▁ l ess ▁in ter es t ing ▁tha n ▁h is ▁ma t ter
+▁h e ▁ t e ll s ▁us ▁tha t ▁ at ▁ t h is ▁f es t ive ▁ s e ason ▁o f ▁the ▁ y e ar ▁w ith ▁ ch r is t m a s ▁ and ▁ro a s t ▁be e f ▁ l o om ing ▁be fore ▁us ▁ s i mile s ▁d r a w n ▁f r om ▁ e at ing ▁ and ▁it s ▁re s u l t s ▁o c c ur ▁m ost ▁re a di l y ▁ to ▁the ▁ mind
+▁h e ▁ ha s ▁g r a v e ▁d o u b t s ▁w h e t h er ▁ s i r ▁f r e d er ic k ▁ l eig h to n ' s ▁w or k ▁ is ▁re all y ▁gre e k ▁a f ter ▁ all ▁ and ▁c a n ▁di s c o v er ▁in ▁it ▁b u t ▁li t t le ▁o f ▁ro ck y ▁it ha c a
+▁li nne ll ' s ▁ p ic tur es ▁ar e ▁a ▁ s or t ▁o f ▁ u p ▁g u ar d s ▁ and ▁ at ▁ em ▁painting s ▁ and ▁m ason ' s ▁ e x q u is i t e ▁ i d y ll s ▁ar e ▁a s ▁ n at ion a l ▁a s ▁a ▁ j ing o ▁ p o em ▁mi ster ▁b i r k e t ▁f o ster ' s ▁ l and s c a pe s ▁ s mile ▁ at ▁on e ▁m u ch ▁in ▁the ▁ s a m e ▁w a y ▁tha t ▁mi ster ▁c ar k er ▁us e d ▁ to ▁f las h ▁h is ▁ t e e t h ▁ and ▁mi ster ▁ j o h n ▁c o ll i er ▁g ive s ▁h is ▁ s i t ter ▁a ▁ ch e er f u l ▁ s l a p ▁on ▁the ▁b a ck ▁be fore ▁h
+e ▁ s a y s ▁li k e ▁a ▁ s ha m p o o er ▁in ▁a ▁ tur k is h ▁b at h ▁ n e x t ▁ma n
+▁it ▁ is ▁o b v i o u s l y ▁ u nne c ess ar y ▁for ▁us ▁ to ▁ p o i n t ▁o u t ▁h o w ▁ l u m i n o u s ▁the s e ▁c rit ic is m s ▁ar e ▁h o w ▁d e l ic at e ▁in ▁ e x p r ess ion
+▁on ▁the ▁g e n er a l ▁ p r i n c i p l es ▁o f ▁ar t ▁mi ster ▁quilter ▁w rit es ▁w ith ▁ e qual ▁ l u c i di t y
+▁painting ▁h e ▁ t e ll s ▁us ▁ is ▁o f ▁a ▁di f f er e n t ▁ qual i t y ▁ to ▁ma t h em at ic s ▁ and ▁f i nish ▁in ▁ar t ▁ is ▁a d d ing ▁m or e ▁f a c t
+▁a s ▁for ▁ e t ch ing s ▁the y ▁ar e ▁o f ▁ t w o ▁ k i n d s ▁b rit is h ▁ and ▁for eig n
+▁h e ▁ l a ment s ▁m ost ▁b i t ter l y ▁the ▁di v or c e ▁tha t ▁ ha s ▁be e n ▁ma d e ▁be t w e e n ▁d e c or at ive ▁ar t ▁ and ▁w ha t ▁we ▁us u all y ▁c all ▁ p ic tur es ▁ma k es ▁the ▁c u s t om ar y ▁a p pe a l ▁ to ▁the ▁ las t ▁ j u d g ment ▁ and ▁re mind s ▁us ▁tha t ▁in ▁the ▁gre at ▁d a y s ▁o f ▁ar t ▁mi c ha e l ▁a n g e l o ▁w a s ▁the ▁f ur nish ing ▁ u p h o l ster er
+
+==> data/lang_char/input.decode <==
+mister quilter is the apostle of the middle classes and we are glad to welcome his gospel
+nor is mister quilter's manner less interesting than his matter
+he tells us that at this festive season of the year with christmas and roast beef looming before us similes drawn from eating and its results occur most readily to the mind
+he has grave doubts whether sir frederick leighton's work is really greek after all and can discover in it but little of rocky ithaca
+linnell's pictures are a sort of up guards and at em paintings and mason's exquisite idylls are as national as a jingo poem mister birket foster's landscapes smile at one much in the same way that mister carker used to flash his teeth and mister john collier gives his sitter a cheerful slap on the back before he says like a shampooer in a turkish bath next man
+it is obviously unnecessary for us to point out how luminous these criticisms are how delicate in expression
+on the general principles of art mister quilter writes with equal lucidity
+painting he tells us is of a different quality to mathematics and finish in art is adding more fact
+as for etchings they are of two kinds british and foreign
+he laments most bitterly the divorce that has been made between decorative art and what we usually call pictures makes the customary appeal to the last judgment and reminds us that in the great days of art michael angelo was the furnishing upholsterer
+
+==> data/lang_char/input.txt <==
+mister quilter is the apostle of the middle classes and we are glad to welcome his gospel
+nor is mister quilter's manner less interesting than his matter
+he tells us that at this festive season of the year with christmas and roast beef looming before us similes drawn from eating and its results occur most readily to the mind
+he has grave doubts whether sir frederick leighton's work is really greek after all and can discover in it but little of rocky ithaca
+linnell's pictures are a sort of up guards and at em paintings and mason's exquisite idylls are as national as a jingo poem mister birket foster's landscapes smile at one much in the same way that mister carker used to flash his teeth and mister john collier gives his sitter a cheerful slap on the back before he says like a shampooer in a turkish bath next man
+it is obviously unnecessary for us to point out how luminous these criticisms are how delicate in expression
+on the general principles of art mister quilter writes with equal lucidity
+painting he tells us is of a different quality to mathematics and finish in art is adding more fact
+as for etchings they are of two kinds british and foreign
+he laments most bitterly the divorce that has been made between decorative art and what we usually call pictures makes the customary appeal to the last judgment and reminds us that in the great days of art michael angelo was the furnishing upholsterer
+
+==> data/lang_char/train_unigram100_units.txt <==
+ 0
+ 0
+▁ -2.01742
+e -2.7203
+s -2.82989
+t -2.99689
+l -3.53267
+n -3.84935
+o -3.88229
+```
diff --git a/examples/ted_en_zh/t0/local/train.sh b/examples/ted_en_zh/t0/local/train.sh
index 928356f9..f905b766 100755
--- a/examples/ted_en_zh/t0/local/train.sh
+++ b/examples/ted_en_zh/t0/local/train.sh
@@ -19,7 +19,8 @@ echo "using ${device}..."
mkdir -p exp
-seed=10086
+# seed may break model convergence
+seed=0
if [ ${seed} != 0 ]; then
export FLAGS_cudnn_deterministic=True
fi
@@ -31,7 +32,7 @@ python3 -u ${BIN_DIR}/train.py \
--output exp/${ckpt_name} \
--seed ${seed}
-if [ ${seed} != 0 ]; then
+if [ ${seed} != 0 ]; then
unset FLAGS_cudnn_deterministic
fi
diff --git a/examples/text_normalization/README.md b/examples/text_normalization/README.md
deleted file mode 100644
index dde0a557..00000000
--- a/examples/text_normalization/README.md
+++ /dev/null
@@ -1,3 +0,0 @@
-# Regular expression based text normalization for Chinese
-
-For simplicity and ease of implementation, text normalization is basically done by rules and dictionaries. Here's an example.
diff --git a/examples/timit/s1/local/train.sh b/examples/timit/s1/local/train.sh
index 3e2e4522..180d8b5a 100755
--- a/examples/timit/s1/local/train.sh
+++ b/examples/timit/s1/local/train.sh
@@ -19,7 +19,8 @@ echo "using ${device}..."
mkdir -p exp
-seed=10086
+# seed may break model convergence
+seed=0
if [ ${seed} != 0 ]; then
export FLAGS_cudnn_deterministic=True
fi
diff --git a/examples/tiny/s0/conf/deepspeech2.yaml b/examples/tiny/s0/conf/deepspeech2.yaml
index 64598b4b..40899655 100644
--- a/examples/tiny/s0/conf/deepspeech2.yaml
+++ b/examples/tiny/s0/conf/deepspeech2.yaml
@@ -48,7 +48,7 @@ training:
n_epoch: 10
accum_grad: 1
lr: 1e-5
- lr_decay: 1.0
+ lr_decay: 0.8
weight_decay: 1e-06
global_grad_clip: 5.0
log_interval: 1
diff --git a/examples/tiny/s0/local/train.sh b/examples/tiny/s0/local/train.sh
index bf4766ee..9a76c7ad 100755
--- a/examples/tiny/s0/local/train.sh
+++ b/examples/tiny/s0/local/train.sh
@@ -1,35 +1,44 @@
#!/bin/bash
-if [ $# != 3 ];then
- echo "usage: CUDA_VISIBLE_DEVICES=0 ${0} config_path ckpt_name model_type"
- exit -1
-fi
+profiler_options=
+
+# seed may break model convergence
+seed=0
+
+source ${MAIN_ROOT}/utils/parse_options.sh || exit 1;
ngpu=$(echo $CUDA_VISIBLE_DEVICES | awk -F "," '{print NF}')
echo "using $ngpu gpus..."
-config_path=$1
-ckpt_name=$2
-model_type=$3
-
device=gpu
if [ ${ngpu} == 0 ];then
device=cpu
fi
-mkdir -p exp
-
-seed=10086
if [ ${seed} != 0 ]; then
export FLAGS_cudnn_deterministic=True
+ echo "using seed $seed & FLAGS_cudnn_deterministic=True ..."
fi
+
+if [ $# != 3 ];then
+ echo "usage: CUDA_VISIBLE_DEVICES=0 ${0} config_path ckpt_name model_type"
+ exit -1
+fi
+
+config_path=$1
+ckpt_name=$2
+model_type=$3
+
+mkdir -p exp
+
python3 -u ${BIN_DIR}/train.py \
--device ${device} \
--nproc ${ngpu} \
--config ${config_path} \
--output exp/${ckpt_name} \
--model_type ${model_type} \
+--profiler-options "${profiler_options}" \
--seed ${seed}
if [ ${seed} != 0 ]; then
diff --git a/examples/tiny/s0/local/tune.sh b/examples/tiny/s0/local/tune.sh
deleted file mode 100755
index c344e77e..00000000
--- a/examples/tiny/s0/local/tune.sh
+++ /dev/null
@@ -1,33 +0,0 @@
-#!/bin/bash
-
-if [ $# != 1 ];then
- echo "usage: tune ckpt_path"
- exit 1
-fi
-
-# grid-search for hyper-parameters in language model
-python3 -u ${BIN_DIR}/tune.py \
---device 'gpu' \
---nproc 1 \
---config conf/deepspeech2.yaml \
---num_batches=-1 \
---batch_size=128 \
---beam_size=500 \
---num_proc_bsearch=12 \
---num_alphas=45 \
---num_betas=8 \
---alpha_from=1.0 \
---alpha_to=3.2 \
---beta_from=0.1 \
---beta_to=0.45 \
---cutoff_prob=1.0 \
---cutoff_top_n=40 \
---checkpoint_path ${1}
-
-if [ $? -ne 0 ]; then
- echo "Failed in tuning!"
- exit 1
-fi
-
-
-exit 0
diff --git a/examples/tiny/s1/conf/augmentation.json b/examples/tiny/s1/conf/augmentation.json
index 6010c2e4..9996cd4e 100644
--- a/examples/tiny/s1/conf/augmentation.json
+++ b/examples/tiny/s1/conf/augmentation.json
@@ -19,17 +19,17 @@
{
"type": "specaug",
"params": {
+ "W": 0,
+ "warp_mode": "PIL",
"F": 10,
- "T": 50,
"n_freq_masks": 2,
+ "T": 50,
"n_time_masks": 2,
"p": 1.0,
- "W": 80,
"adaptive_number_ratio": 0,
"adaptive_size_ratio": 0,
"max_n_time_masks": 20,
- "replace_with_zero": true,
- "warp_mode": "PIL"
+ "replace_with_zero": true
},
"prob": 1.0
}
diff --git a/examples/tiny/s1/local/train.sh b/examples/tiny/s1/local/train.sh
index 48968f63..5097d4d0 100755
--- a/examples/tiny/s1/local/train.sh
+++ b/examples/tiny/s1/local/train.sh
@@ -1,36 +1,49 @@
#!/bin/bash
-if [ $# != 2 ];then
- echo "usage: CUDA_VISIBLE_DEVICES=0 ${0} config_path ckpt_name"
- exit -1
-fi
+profiler_options=
+benchmark_batch_size=0
+benchmark_max_step=0
+
+# seed may break model convergence
+seed=0
+
+source ${MAIN_ROOT}/utils/parse_options.sh || exit 1;
ngpu=$(echo $CUDA_VISIBLE_DEVICES | awk -F "," '{print NF}')
echo "using $ngpu gpus..."
-config_path=$1
-ckpt_name=$2
-
device=gpu
if [ ${ngpu} == 0 ];then
device=cpu
fi
-mkdir -p exp
-
-seed=10086
-if [ ${seed} != 0 ]; then
+if [ ${seed} != 0 ]; then
export FLAGS_cudnn_deterministic=True
+ echo "using seed $seed & FLAGS_cudnn_deterministic=True ..."
fi
+if [ $# != 2 ];then
+ echo "usage: CUDA_VISIBLE_DEVICES=0 ${0} config_path ckpt_name"
+ exit -1
+fi
+
+config_path=$1
+ckpt_name=$2
+
+mkdir -p exp
+
python3 -u ${BIN_DIR}/train.py \
+--seed ${seed} \
--device ${device} \
--nproc ${ngpu} \
--config ${config_path} \
--output exp/${ckpt_name} \
---seed ${seed}
+--profiler-options "${profiler_options}" \
+--benchmark-batch-size ${benchmark_batch_size} \
+--benchmark-max-step ${benchmark_max_step}
+
-if [ ${seed} != 0 ]; then
+if [ ${seed} != 0 ]; then
unset FLAGS_cudnn_deterministic
fi
diff --git a/examples/tn/.gitignore b/examples/tn/.gitignore
new file mode 100644
index 00000000..0f250338
--- /dev/null
+++ b/examples/tn/.gitignore
@@ -0,0 +1 @@
+exp
diff --git a/examples/tn/README.md b/examples/tn/README.md
new file mode 100644
index 00000000..ff7be293
--- /dev/null
+++ b/examples/tn/README.md
@@ -0,0 +1,36 @@
+# Regular expression based text normalization for Chinese
+
+For simplicity and ease of implementation, text normalization is basically done by rules and dictionaries. Here's an example.
+
+## Run
+
+```
+. path.sh
+bash run.sh
+```
+
+## Results
+
+```
+exp/
+`-- normalized.txt
+
+0 directories, 1 file
+```
+
+```
+aff31f8aa08e2a7360228c9ce5886b98 exp/normalized.txt
+```
+
+```
+今天的最低气温达到零下十度.
+只要有四分之三十三的人同意，就可以通过决议。
+一九四五年五月二日，苏联士兵在德国国会大厦上升起了胜利旗，象征着攻占柏林并战胜了纳粹德国。
+四月十六日，清晨的战斗以炮击揭幕，数以千计的大炮和喀秋莎火箭炮开始炮轰德军阵地，炮击持续了数天之久。
+如果剩下的百分之三十点六是过去，那么还有百分之六十九点四.
+事情发生在二零二零年三月三十一日的上午八点.
+警方正在找一支点二二口径的手枪。
+欢迎致电中国联通，北京二零二二年冬奥会官方合作伙伴为您服务
+充值缴费请按一，查询话费及余量请按二，跳过本次提醒请按井号键。
+快速解除流量封顶请按星号键，腾讯王卡产品介绍、使用说明、特权及活动请按九，查询话费、套餐余量、积分及活动返款请按一，手机上网流量开通及取消请按二，查���本机号码及本号所使用套餐请按四，密码修改及重置请按五，紧急开机请按六，挂失请按七，查询充值记录请按八，其它自助服务及工服务请按零
+```
diff --git a/examples/text_normalization/data/sentences.txt b/examples/tn/data/sentences.txt
similarity index 100%
rename from examples/text_normalization/data/sentences.txt
rename to examples/tn/data/sentences.txt
diff --git a/examples/text_normalization/local/test_normalization.py b/examples/tn/local/test_normalization.py
similarity index 100%
rename from examples/text_normalization/local/test_normalization.py
rename to examples/tn/local/test_normalization.py
diff --git a/examples/text_normalization/path.sh b/examples/tn/path.sh
similarity index 100%
rename from examples/text_normalization/path.sh
rename to examples/tn/path.sh
diff --git a/examples/text_normalization/run.sh b/examples/tn/run.sh
similarity index 100%
rename from examples/text_normalization/run.sh
rename to examples/tn/run.sh
diff --git a/tests/benchmark/.gitignore b/tests/benchmark/.gitignore
new file mode 100644
index 00000000..7d166b06
--- /dev/null
+++ b/tests/benchmark/.gitignore
@@ -0,0 +1,2 @@
+old-pd_env.txt
+pd_env.txt
diff --git a/tests/benchmark/README.md b/tests/benchmark/README.md
new file mode 100644
index 00000000..d21999ab
--- /dev/null
+++ b/tests/benchmark/README.md
@@ -0,0 +1,11 @@
+# Benchmark Test
+
+## Data
+
+* Aishell
+
+## Docker
+
+```
+registry.baidubce.com/paddlepaddle/paddle 2.1.1-gpu-cuda10.2-cudnn7 59d5ec1de486
+```
diff --git a/tests/benchmark/run_all.sh b/tests/benchmark/run_all.sh
new file mode 100755
index 00000000..6f707cdc
--- /dev/null
+++ b/tests/benchmark/run_all.sh
@@ -0,0 +1,49 @@
+#!/bin/bash
+
+CUR_DIR=${PWD}
+ROOT_DIR=../../
+
+# 提供可稳定复现性能的脚本，默认在标准docker环境内py37执行：
+# collect env info
+bash ${ROOT_DIR}/utils/pd_env_collect.sh
+#cat pd_env.txt
+
+
+# 1 安装该模型需要的依赖 (如需开启优化策略请注明)
+#pushd ${ROOT_DIR}/tools; make; popd
+#source ${ROOT_DIR}/tools/venv/bin/activate
+#pushd ${ROOT_DIR}; bash setup.sh; popd
+
+
+# 2 拷贝该模型需要数据、预训练模型
+
+# 执行目录：需说明
+#pushd ${ROOT_DIR}/examples/aishell/s1
+pushd ${ROOT_DIR}/examples/tiny/s1
+
+mkdir -p exp/log
+. path.sh
+#bash local/data.sh &> exp/log/data.log
+
+# 3 批量运行（如不方便批量，1，2需放到单个模型中）
+
+model_mode_list=(conformer transformer)
+fp_item_list=(fp32)
+bs_item_list=(32 64 96)
+for model_mode in ${model_mode_list[@]}; do
+ for fp_item in ${fp_item_list[@]}; do
+ for bs_item in ${bs_item_list[@]}
+ do
+ echo "index is speed, 1gpus, begin, ${model_name}"
+ run_mode=sp
+ CUDA_VISIBLE_DEVICES=0 bash ${CUR_DIR}/run_benchmark.sh ${run_mode} ${bs_item} ${fp_item} 500 ${model_mode} # (5min)
+ sleep 60
+ echo "index is speed, 8gpus, run_mode is multi_process, begin, ${model_name}"
+ run_mode=mp
+ CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 bash ${CUR_DIR}/run_benchmark.sh ${run_mode} ${bs_item} ${fp_item} 500 ${model_mode}
+ sleep 60
+ done
+ done
+done
+
+popd # aishell/s1
diff --git a/tests/benchmark/run_benchmark.sh b/tests/benchmark/run_benchmark.sh
new file mode 100755
index 00000000..bd4655d1
--- /dev/null
+++ b/tests/benchmark/run_benchmark.sh
@@ -0,0 +1,57 @@
+#!/bin/bash
+
+set -xe
+
+# 运行示例：CUDA_VISIBLE_DEVICES=0 bash run_benchmark.sh ${run_mode} ${bs_item} ${fp_item} 500 ${model_mode}
+# 参数说明
+function _set_params(){
+ run_mode=${1:-"sp"} # 单卡sp|多卡mp
+ batch_size=${2:-"64"}
+ fp_item=${3:-"fp32"} # fp32|fp16
+ max_iter=${4:-"500"} # 可选，如果需要修改代码提前中断
+ model_name=${5:-"model_name"}
+ run_log_path=${TRAIN_LOG_DIR:-$(pwd)} # TRAIN_LOG_DIR 后续QA设置该参数
+
+# 以下不用修改
+ device=${CUDA_VISIBLE_DEVICES//,/ }
+ arr=(${device})
+ num_gpu_devices=${#arr[*]}
+ log_file=${run_log_path}/${model_name}_${run_mode}_bs${batch_size}_${fp_item}_${num_gpu_devices}
+}
+
+function _train(){
+ echo "Train on ${num_gpu_devices} GPUs"
+ echo "current CUDA_VISIBLE_DEVICES=$CUDA_VISIBLE_DEVICES, gpus=$num_gpu_devices, batch_size=$batch_size"
+
+ train_cmd="--benchmark-batch-size ${batch_size}
+ --benchmark-max-step ${max_iter}
+ conf/${model_name}.yaml ${model_name}"
+
+ case ${run_mode} in
+ sp) train_cmd="bash local/train.sh "${train_cmd}"" ;;
+ mp)
+ train_cmd="bash local/train.sh "${train_cmd}"" ;;
+ *) echo "choose run_mode(sp or mp)"; exit 1;
+ esac
+
+ # 以下不用修改
+ timeout 15m ${train_cmd} > ${log_file} 2>&1
+ if [ $? -ne 0 ];then
+ echo -e "${model_name}, FAIL"
+ export job_fail_flag=1
+ else
+ echo -e "${model_name}, SUCCESS"
+ export job_fail_flag=0
+ fi
+
+ trap 'for pid in $(jobs -pr); do kill -KILL $pid; done' INT QUIT TERM
+
+ if [ $run_mode = "mp" -a -d mylog ]; then
+ rm ${log_file}
+ cp mylog/workerlog.0 ${log_file}
+ fi
+}
+
+_set_params $@
+_train
+
diff --git a/third_party/__init__.py b/third_party/__init__.py
new file mode 100644
index 00000000..e69de29b
diff --git a/third_party/paddle_audio/__init__.py b/third_party/paddle_audio/__init__.py
new file mode 100644
index 00000000..e69de29b
diff --git a/third_party/paddle_audio/frontend.py b/third_party/paddle_audio/frontend.py
deleted file mode 100644
index 1b337732..00000000
--- a/third_party/paddle_audio/frontend.py
+++ /dev/null
@@ -1,146 +0,0 @@
-from typing import Tuple
-import numpy as np
-import paddle
-from paddle import Tensor
-from paddle import nn
-from paddle.nn import functional as F
-
-
-def frame(x: Tensor,
- num_samples: Tensor,
- win_length: int,
- hop_length: int,
- clip: bool = True) -> Tuple[Tensor, Tensor]:
- """Extract frames from audio.
-
- Parameters
- ----------
- x : Tensor
- Shape (N, T), batched waveform.
- num_samples : Tensor
- Shape (N, ), number of samples of each waveform.
- win_length : int
- Window length.
- hop_length : int
- Number of samples shifted between ajancent frames.
- clip : bool, optional
- Whether to clip audio that does not fit into the last frame, by
- default True
-
- Returns
- -------
- frames : Tensor
- Shape (N, T', win_length).
- num_frames : Tensor
- Shape (N, ) number of valid frames
- """
- assert hop_length <= win_length
- num_frames = (num_samples - win_length) // hop_length
- padding = (0, 0)
- if not clip:
- num_frames += 1
- # NOTE: pad hop_length - 1 to the right to ensure that there is at most
- # one frame dangling to the righe edge
- padding = (0, hop_length - 1)
-
- weight = paddle.eye(win_length).unsqueeze(1)
-
- frames = F.conv1d(x.unsqueeze(1),
- weight,
- padding=padding,
- stride=(hop_length, ))
- return frames, num_frames
-
-
-class STFT(nn.Layer):
- """A module for computing stft transformation in a differentiable way.
-
- Parameters
- ------------
- n_fft : int
- Number of samples in a frame.
-
- hop_length : int
- Number of samples shifted between adjacent frames.
-
- win_length : int
- Length of the window.
-
- clip: bool
- Whether to clip audio is necesaary.
- """
- def __init__(self,
- n_fft: int,
- hop_length: int,
- win_length: int,
- window_type: str = None,
- clip: bool = True):
- super().__init__()
-
- self.hop_length = hop_length
- self.n_bin = 1 + n_fft // 2
- self.n_fft = n_fft
- self.clip = clip
-
- # calculate window
- if window_type is None:
- window = np.ones(win_length)
- elif window_type == "hann":
- window = np.hanning(win_length)
- elif window_type == "hamming":
- window = np.hamming(win_length)
- else:
- raise ValueError("Not supported yet!")
-
- if win_length < n_fft:
- window = F.pad(window, (0, n_fft - win_length))
- elif win_length > n_fft:
- window = window[:n_fft]
-
- # (n_bins, n_fft) complex
- kernel_size = min(n_fft, win_length)
- weight = np.fft.fft(np.eye(n_fft))[:self.n_bin, :kernel_size]
- w_real = weight.real
- w_imag = weight.imag
-
- # (2 * n_bins, kernel_size)
- w = np.concatenate([w_real, w_imag], axis=0)
- w = w * window
-
- # (2 * n_bins, 1, kernel_size) # (C_out, C_in, kernel_size)
- w = np.expand_dims(w, 1)
- weight = paddle.cast(paddle.to_tensor(w), paddle.get_default_dtype())
- self.register_buffer("weight", weight)
-
- def forward(self, x: Tensor, num_samples: Tensor) -> Tuple[Tensor, Tensor]:
- """Compute the stft transform.
- Parameters
- ------------
- x : Tensor [shape=(B, T)]
- The input waveform.
- num_samples : Tensor
- Number of samples of each waveform.
- Returns
- ------------
- D : Tensor
- Shape(N, T', n_bins, 2) Spectrogram.
-
- num_frames: Tensor
- Shape (N,) number of samples of each spectrogram
- """
- num_frames = (num_samples - self.win_length) // self.hop_length
- padding = (0, 0)
- if not self.clip:
- num_frames += 1
- padding = (0, self.hop_length - 1)
-
- batch_size, _, _ = paddle.shape(x)
- x = x.unsqueeze(-1)
- D = F.conv1d(self.weight,
- x,
- stride=(self.hop_length, ),
- padding=padding,
- data_format="NLC")
- D = paddle.reshape(D, [batch_size, -1, self.n_bin, 2])
- return D, num_frames
-
diff --git a/third_party/paddle_audio/frontend/common.py b/third_party/paddle_audio/frontend/common.py
new file mode 100644
index 00000000..7638dae5
--- /dev/null
+++ b/third_party/paddle_audio/frontend/common.py
@@ -0,0 +1,201 @@
+import paddle
+import numpy as np
+from typing import Tuple, Optional, Union
+
+
+# https://github.com/kaldi-asr/kaldi/blob/cbed4ff688/src/feat/feature-window.cc#L109
+def povey_window(frame_len:int) -> np.ndarray:
+ win = np.empty(frame_len)
+ a = 2 * np.pi / (frame_len -1)
+ for i in range(frame_len):
+ win[i] = (0.5 - 0.5 * np.cos(a * i) )**0.85
+ return win
+
+def hann_window(frame_len:int) -> np.ndarray:
+ win = np.empty(frame_len)
+ a = 2 * np.pi / (frame_len -1)
+ for i in range(frame_len):
+ win[i] = 0.5 - 0.5 * np.cos(a * i)
+ return win
+
+def sine_window(frame_len:int) -> np.ndarray:
+ win = np.empty(frame_len)
+ a = 2 * np.pi / (frame_len -1)
+ for i in range(frame_len):
+ win[i] = np.sin(0.5 * a * i)
+ return win
+
+def hamm_window(frame_len:int) -> np.ndarray:
+ win = np.empty(frame_len)
+ a = 2 * np.pi / (frame_len -1)
+ for i in range(frame_len):
+ win[i] = 0.54 - 0.46 * np.cos(a * i)
+ return win
+
+def get_window(wintype:Optional[str], winlen:int) -> np.ndarray:
+ """get window function
+
+ Args:
+ wintype (Optional[str]): window type.
+ winlen (int): window length in samples.
+
+ Raises:
+ ValueError: not support window.
+
+ Returns:
+ np.ndarray: window coeffs.
+ """
+ # calculate window
+ if not wintype or wintype == 'rectangular':
+ window = np.ones(winlen)
+ elif wintype == "hann":
+ window = hann_window(winlen)
+ elif wintype == "hamm":
+ window = hamm_window(winlen)
+ elif wintype == "povey":
+ window = povey_window(winlen)
+ else:
+ msg = f"{wintype} Not supported yet!"
+ raise ValueError(msg)
+ return window
+
+
+def dft_matrix(n_fft:int, winlen:int=None, n_bin:int=None) -> Tuple[np.ndarray, np.ndarray, int]:
+ # https://en.wikipedia.org/wiki/Discrete_Fourier_transform
+ # (n_bins, n_fft) complex
+ if n_bin is None:
+ n_bin = 1 + n_fft // 2
+ if winlen is None:
+ winlen = n_bin
+ # https://github.com/numpy/numpy/blob/v1.20.0/numpy/fft/_pocketfft.py#L49
+ kernel_size = min(n_fft, winlen)
+
+ n = np.arange(0, n_fft, 1.)
+ wsin = np.empty((n_bin, kernel_size)) #[Cout, kernel_size]
+ wcos = np.empty((n_bin, kernel_size)) #[Cout, kernel_size]
+ for k in range(n_bin): # Only half of the bins contain useful info
+ wsin[k,:] = -np.sin(2*np.pi*k*n/n_fft)[:kernel_size]
+ wcos[k,:] = np.cos(2*np.pi*k*n/n_fft)[:kernel_size]
+ w_real = wcos
+ w_imag = wsin
+ return w_real, w_imag, kernel_size
+
+
+def dft_matrix_fast(n_fft:int, winlen:int=None, n_bin:int=None) -> Tuple[np.ndarray, np.ndarray, int]:
+ # (n_bins, n_fft) complex
+ if n_bin is None:
+ n_bin = 1 + n_fft // 2
+ if winlen is None:
+ winlen = n_bin
+ # https://github.com/numpy/numpy/blob/v1.20.0/numpy/fft/_pocketfft.py#L49
+ kernel_size = min(n_fft, winlen)
+
+ # https://en.wikipedia.org/wiki/DFT_matrix
+ # https://ccrma.stanford.edu/~jos/st/Matrix_Formulation_DFT.html
+ weight = np.fft.fft(np.eye(n_fft))[:self.n_bin, :kernel_size]
+ w_real = weight.real
+ w_imag = weight.imag
+ return w_real, w_imag, kernel_size
+
+
+def bin2hz(bin:Union[List[int], np.ndarray], N:int, sr:int)->List[float]:
+ """FFT bins to Hz.
+
+ http://practicalcryptography.com/miscellaneous/machine-learning/intuitive-guide-discrete-fourier-transform/
+
+ Args:
+ bins (List[int] or np.ndarray): bin index.
+ N (int): the number of samples, or FFT points.
+ sr (int): sampling rate.
+
+ Returns:
+ List[float]: Hz's.
+ """
+ hz = bin * float(sr) / N
+
+
+def hz2mel(hz):
+ """Convert a value in Hertz to Mels
+
+ :param hz: a value in Hz. This can also be a numpy array, conversion proceeds element-wise.
+ :returns: a value in Mels. If an array was passed in, an identical sized array is returned.
+ """
+ return 1127 * np.log(1+hz/700.0)
+
+
+def mel2hz(mel):
+ """Convert a value in Mels to Hertz
+
+ :param mel: a value in Mels. This can also be a numpy array, conversion proceeds element-wise.
+ :returns: a value in Hertz. If an array was passed in, an identical sized array is returned.
+ """
+ return 700 * (np.exp(mel/1127.0)-1)
+
+
+
+def rms_to_db(rms: float):
+ """Root Mean Square to dB.
+
+ Args:
+ rms ([float]): root mean square
+
+ Returns:
+ float: dB
+ """
+ return 20.0 * math.log10(max(1e-16, rms))
+
+
+def rms_to_dbfs(rms: float):
+ """Root Mean Square to dBFS.
+ https://fireattack.wordpress.com/2017/02/06/replaygain-loudness-normalization-and-applications/
+ Audio is mix of sine wave, so 1 amp sine wave's Full scale is 0.7071, equal to -3.0103dB.
+
+ dB = dBFS + 3.0103
+ dBFS = db - 3.0103
+ e.g. 0 dB = -3.0103 dBFS
+
+ Args:
+ rms ([float]): root mean square
+
+ Returns:
+ float: dBFS
+ """
+ return rms_to_db(rms) - 3.0103
+
+
+def max_dbfs(sample_data: np.ndarray):
+ """Peak dBFS based on the maximum energy sample.
+
+ Args:
+ sample_data ([np.ndarray]): float array, [-1, 1].
+
+ Returns:
+ float: dBFS
+ """
+ # Peak dBFS based on the maximum energy sample. Will prevent overdrive if used for normalization.
+ return rms_to_dbfs(max(abs(np.min(sample_data)), abs(np.max(sample_data))))
+
+
+def mean_dbfs(sample_data):
+ """Peak dBFS based on the RMS energy.
+
+ Args:
+ sample_data ([np.ndarray]): float array, [-1, 1].
+
+ Returns:
+ float: dBFS
+ """
+ return rms_to_dbfs(
+ math.sqrt(np.mean(np.square(sample_data, dtype=np.float64))))
+
+
+def gain_db_to_ratio(gain_db: float):
+ """dB to ratio
+
+ Args:
+ gain_db (float): gain in dB
+
+ Returns:
+ float: scale in amp
+ """
+ return math.pow(10.0, gain_db / 20.0)
\ No newline at end of file
diff --git a/third_party/paddle_audio/frontend/english.wav b/third_party/paddle_audio/frontend/english.wav
new file mode 100644
index 00000000..bb28291f
Binary files /dev/null and b/third_party/paddle_audio/frontend/english.wav differ
diff --git a/third_party/paddle_audio/frontend/kaldi.py b/third_party/paddle_audio/frontend/kaldi.py
new file mode 100644
index 00000000..d1c13fe3
--- /dev/null
+++ b/third_party/paddle_audio/frontend/kaldi.py
@@ -0,0 +1,266 @@
+from typing import Tuple
+import numpy as np
+import paddle
+from paddle import Tensor
+from paddle import nn
+from paddle.nn import functional as F
+import soundfile as sf
+
+from .common import get_window
+from .common import dft_matrix
+
+
+def read(wavpath:str, sr:int = None, start=0, stop=None, dtype='int16', always_2d=True)->Tuple[int, np.ndarray]:
+ """load wav file.
+
+ Args:
+ wavpath (str): wav path.
+ sr (int, optional): expect sample rate. Defaults to None.
+ dtype (str, optional): wav data bits. Defaults to 'int16'.
+
+ Returns:
+ Tuple[int, np.ndarray]: sr (int), wav (int16) [T, C].
+ """
+ wav, r_sr = sf.read(wavpath, start=start, stop=stop, dtype=dtype, always_2d=always_2d)
+ if sr:
+ assert sr == r_sr
+ return r_sr, wav
+
+
+def write(wavpath:str, wav:np.ndarray, sr:int, dtype='PCM_16'):
+ """write wav file.
+
+ Args:
+ wavpath (str): file path to save.
+ wav (np.ndarray): wav data.
+ sr (int): data samplerate.
+ dtype (str, optional): wav bit format. Defaults to 'PCM_16'.
+ """
+ sf.write(wavpath, wav, sr, subtype=dtype)
+
+
+def frames(x: Tensor,
+ num_samples: Tensor,
+ sr: int,
+ win_length: float,
+ stride_length: float,
+ clip: bool = False) -> Tuple[Tensor, Tensor]:
+ """Extract frames from audio.
+
+ Parameters
+ ----------
+ x : Tensor
+ Shape (B, T), batched waveform.
+ num_samples : Tensor
+ Shape (B, ), number of samples of each waveform.
+ sr: int
+ Sampling Rate.
+ win_length : float
+ Window length in ms.
+ stride_length : float
+ Stride length in ms.
+ clip : bool, optional
+ Whether to clip audio that does not fit into the last frame, by
+ default True
+
+ Returns
+ -------
+ frames : Tensor
+ Shape (B, T', win_length).
+ num_frames : Tensor
+ Shape (B, ) number of valid frames
+ """
+ assert stride_length <= win_length
+ stride_length = int(stride_length * sr)
+ win_length = int(win_length * sr)
+
+ num_frames = (num_samples - win_length) // stride_length
+ padding = (0, 0)
+ if not clip:
+ num_frames += 1
+ need_samples = num_frames * stride_length + win_length
+ padding = (0, need_samples - num_samples - 1)
+
+ weight = paddle.eye(win_length).unsqueeze(1) #[win_length, 1, win_length]
+
+ frames = F.conv1d(x.unsqueeze(-1),
+ weight,
+ padding=padding,
+ stride=(stride_length, ),
+ data_format='NLC')
+ return frames, num_frames
+
+
+def dither(signal:Tensor, dither_value=1.0)->Tensor:
+ """dither frames for log compute.
+
+ Args:
+ signal (Tensor): [B, T, D]
+ dither_value (float, optional): [scalar]. Defaults to 1.0.
+
+ Returns:
+ Tensor: [B, T, D]
+ """
+ D = paddle.shape(signal)[-1]
+ signal += paddle.normal(shape=[1, 1, D]) * dither_value
+ return signal
+
+
+def remove_dc_offset(signal:Tensor)->Tensor:
+ """remove dc.
+
+ Args:
+ signal (Tensor): [B, T, D]
+
+ Returns:
+ Tensor: [B, T, D]
+ """
+ signal -= paddle.mean(signal, axis=-1, keepdim=True)
+ return signal
+
+def preemphasis(signal:Tensor, coeff=0.97)->Tensor:
+ """perform preemphasis on the input signal.
+
+ Args:
+ signal (Tensor): [B, T, D], The signal to filter.
+ coeff (float, optional): [scalar].The preemphasis coefficient. 0 is no filter, Defaults to 0.97.
+
+ Returns:
+ Tensor: [B, T, D]
+ """
+ return paddle.concat([
+ (1-coeff)*signal[:, :, 0:1],
+ signal[:, :, 1:] - coeff * signal[:, :, :-1]
+ ], axis=-1)
+
+
+class STFT(nn.Layer):
+ """A module for computing stft transformation in a differentiable way.
+
+ http://practicalcryptography.com/miscellaneous/machine-learning/intuitive-guide-discrete-fourier-transform/
+
+ Parameters
+ ------------
+ n_fft : int
+ Number of samples in a frame.
+
+ sr: int
+ Number of Samplilng rate.
+
+ stride_length : float
+ Number of samples shifted between adjacent frames.
+
+ win_length : float
+ Length of the window.
+
+ clip: bool
+ Whether to clip audio is necesaary.
+ """
+ def __init__(self,
+ n_fft: int,
+ sr: int,
+ win_length: float,
+ stride_length: float,
+ dither:float=0.0,
+ preemph_coeff:float=0.97,
+ remove_dc_offset:bool=True,
+ window_type: str = 'povey',
+ clip: bool = False):
+ super().__init__()
+ self.sr = sr
+ self.win_length = win_length
+ self.stride_length = stride_length
+ self.dither = dither
+ self.preemph_coeff = preemph_coeff
+ self.remove_dc_offset = remove_dc_offset
+ self.window_type = window_type
+ self.clip = clip
+
+ self.n_fft = n_fft
+ self.n_bin = 1 + n_fft // 2
+
+ w_real, w_imag, kernel_size = dft_matrix(
+ self.n_fft, int(self.win_length * self.sr), self.n_bin
+ )
+
+ # calculate window
+ window = get_window(window_type, kernel_size)
+
+ # (2 * n_bins, kernel_size)
+ w = np.concatenate([w_real, w_imag], axis=0)
+ w = w * window
+ # (kernel_size, 2 * n_bins)
+ w = np.transpose(w)
+ weight = paddle.cast(paddle.to_tensor(w), paddle.get_default_dtype())
+ self.register_buffer("weight", weight)
+
+ def forward(self, x: Tensor, num_samples: Tensor) -> Tuple[Tensor, Tensor]:
+ """Compute the stft transform.
+ Parameters
+ ------------
+ x : Tensor [shape=(B, T)]
+ The input waveform.
+ num_samples : Tensor [shape=(B,)]
+ Number of samples of each waveform.
+ Returns
+ ------------
+ C : Tensor
+ Shape(B, T', n_bins, 2) Spectrogram.
+
+ num_frames: Tensor
+ Shape (B,) number of samples of each spectrogram
+ """
+ batch_size = paddle.shape(num_samples)
+ F, nframe = frames(x, num_samples, self.sr, self.win_length, self.stride_length, clip=self.clip)
+ if self.dither:
+ F = dither(F, self.dither)
+ if self.remove_dc_offset:
+ F = remove_dc_offset(F)
+ if self.preemph_coeff:
+ F = preemphasis(F)
+ C = paddle.matmul(F, self.weight) # [B, T, K] [K, 2 * n_bins]
+ C = paddle.reshape(C, [batch_size, -1, 2, self.n_bin])
+ C = C.transpose([0, 1, 3, 2])
+ return C, nframe
+
+
+def powspec(C:Tensor) -> Tensor:
+ """Compute the power spectrum |X_k|^2.
+
+ Args:
+ C (Tensor): [B, T, C, 2]
+
+ Returns:
+ Tensor: [B, T, C]
+ """
+ real, imag = paddle.chunk(C, 2, axis=-1)
+ return paddle.square(real.squeeze(-1)) + paddle.square(imag.squeeze(-1))
+
+
+def magspec(C: Tensor, eps=1e-10) -> Tensor:
+ """Compute the magnitude spectrum |X_k|.
+
+ Args:
+ C (Tensor): [B, T, C, 2]
+ eps (float): epsilon.
+
+ Returns:
+ Tensor: [B, T, C]
+ """
+ pspec = powspec(C)
+ return paddle.sqrt(pspec + eps)
+
+
+def logspec(C: Tensor, eps=1e-10) -> Tensor:
+ """Compute log-spectrum 20log10∣X_k∣.
+
+ Args:
+ C (Tensor): [description]
+ eps ([type], optional): [description]. Defaults to 1e-10.
+
+ Returns:
+ Tensor: [description]
+ """
+ spec = magspec(C)
+ return 20 * paddle.log10(spec + eps)
+
diff --git a/third_party/paddle_audio/frontend/kaldi_test.py b/third_party/paddle_audio/frontend/kaldi_test.py
new file mode 100644
index 00000000..34ff413c
--- /dev/null
+++ b/third_party/paddle_audio/frontend/kaldi_test.py
@@ -0,0 +1,533 @@
+from typing import Tuple
+import numpy as np
+import paddle
+import unittest
+
+import decimal
+import numpy
+import math
+import logging
+from pathlib import Path
+
+from scipy.fftpack import dct
+
+from third_party.paddle_audio.frontend import kaldi
+
+def round_half_up(number):
+ return int(decimal.Decimal(number).quantize(decimal.Decimal('1'), rounding=decimal.ROUND_HALF_UP))
+
+def rolling_window(a, window, step=1):
+ # http://ellisvalentiner.com/post/2017-03-21-np-strides-trick
+ shape = a.shape[:-1] + (a.shape[-1] - window + 1, window)
+ strides = a.strides + (a.strides[-1],)
+ return numpy.lib.stride_tricks.as_strided(a, shape=shape, strides=strides)[::step]
+
+
+def do_dither(signal, dither_value=1.0):
+ signal += numpy.random.normal(size=signal.shape) * dither_value
+ return signal
+
+def do_remove_dc_offset(signal):
+ signal -= numpy.mean(signal)
+ return signal
+
+def do_preemphasis(signal, coeff=0.97):
+ """perform preemphasis on the input signal.
+
+ :param signal: The signal to filter.
+ :param coeff: The preemphasis coefficient. 0 is no filter, default is 0.95.
+ :returns: the filtered signal.
+ """
+ return numpy.append((1-coeff)*signal[0], signal[1:] - coeff * signal[:-1])
+
+
+def framesig(sig, frame_len, frame_step, dither=1.0, preemph=0.97, remove_dc_offset=True, wintype='hamming', stride_trick=True):
+ """Frame a signal into overlapping frames.
+
+ :param sig: the audio signal to frame.
+ :param frame_len: length of each frame measured in samples.
+ :param frame_step: number of samples after the start of the previous frame that the next frame should begin.
+ :param winfunc: the analysis window to apply to each frame. By default no window is applied.
+ :param stride_trick: use stride trick to compute the rolling window and window multiplication faster
+ :returns: an array of frames. Size is NUMFRAMES by frame_len.
+ """
+ slen = len(sig)
+ frame_len = int(round_half_up(frame_len))
+ frame_step = int(round_half_up(frame_step))
+ if slen <= frame_len:
+ numframes = 1
+ else:
+ numframes = 1 + (( slen - frame_len) // frame_step)
+
+ # check kaldi/src/feat/feature-window.h
+ padsignal = sig[:(numframes-1)*frame_step+frame_len]
+ if wintype is 'povey':
+ win = numpy.empty(frame_len)
+ for i in range(frame_len):
+ win[i] = (0.5-0.5*numpy.cos(2*numpy.pi/(frame_len-1)*i))**0.85
+ else: # the hamming window
+ win = numpy.hamming(frame_len)
+
+ if stride_trick:
+ frames = rolling_window(padsignal, window=frame_len, step=frame_step)
+ else:
+ indices = numpy.tile(numpy.arange(0, frame_len), (numframes, 1)) + numpy.tile(
+ numpy.arange(0, numframes * frame_step, frame_step), (frame_len, 1)).T
+ indices = numpy.array(indices, dtype=numpy.int32)
+ frames = padsignal[indices]
+ win = numpy.tile(win, (numframes, 1))
+
+ frames = frames.astype(numpy.float32)
+ raw_frames = numpy.zeros(frames.shape)
+ for frm in range(frames.shape[0]):
+ frames[frm,:] = do_dither(frames[frm,:], dither) # dither
+ frames[frm,:] = do_remove_dc_offset(frames[frm,:]) # remove dc offset
+ raw_frames[frm,:] = frames[frm,:]
+ frames[frm,:] = do_preemphasis(frames[frm,:], preemph) # preemphasize
+
+ return frames * win, raw_frames
+
+
+def magspec(frames, NFFT):
+ """Compute the magnitude spectrum of each frame in frames. If frames is an NxD matrix, output will be Nx(NFFT/2+1).
+
+ :param frames: the array of frames. Each row is a frame.
+ :param NFFT: the FFT length to use. If NFFT > frame_len, the frames are zero-padded.
+ :returns: If frames is an NxD matrix, output will be Nx(NFFT/2+1). Each row will be the magnitude spectrum of the corresponding frame.
+ """
+ if numpy.shape(frames)[1] > NFFT:
+ logging.warn(
+ 'frame length (%d) is greater than FFT size (%d), frame will be truncated. Increase NFFT to avoid.',
+ numpy.shape(frames)[1], NFFT)
+ complex_spec = numpy.fft.rfft(frames, NFFT)
+ return numpy.absolute(complex_spec)
+
+
+def powspec(frames, NFFT):
+ """Compute the power spectrum of each frame in frames. If frames is an NxD matrix, output will be Nx(NFFT/2+1).
+
+ :param frames: the array of frames. Each row is a frame.
+ :param NFFT: the FFT length to use. If NFFT > frame_len, the frames are zero-padded.
+ :returns: If frames is an NxD matrix, output will be Nx(NFFT/2+1). Each row will be the power spectrum of the corresponding frame.
+ """
+ return numpy.square(magspec(frames, NFFT))
+
+
+
+def mfcc(signal,samplerate=16000,winlen=0.025,winstep=0.01,numcep=13,
+ nfilt=23,nfft=512,lowfreq=20,highfreq=None,dither=1.0,remove_dc_offset=True,preemph=0.97,
+ ceplifter=22,useEnergy=True,wintype='povey'):
+ """Compute MFCC features from an audio signal.
+
+ :param signal: the audio signal from which to compute features. Should be an N*1 array
+ :param samplerate: the samplerate of the signal we are working with.
+ :param winlen: the length of the analysis window in seconds. Default is 0.025s (25 milliseconds)
+ :param winstep: the step between successive windows in seconds. Default is 0.01s (10 milliseconds)
+ :param numcep: the number of cepstrum to return, default 13
+ :param nfilt: the number of filters in the filterbank, default 26.
+ :param nfft: the FFT size. Default is 512.
+ :param lowfreq: lowest band edge of mel filters. In Hz, default is 0.
+ :param highfreq: highest band edge of mel filters. In Hz, default is samplerate/2
+ :param preemph: apply preemphasis filter with preemph as coefficient. 0 is no filter. Default is 0.97.
+ :param ceplifter: apply a lifter to final cepstral coefficients. 0 is no lifter. Default is 22.
+ :param appendEnergy: if this is true, the zeroth cepstral coefficient is replaced with the log of the total frame energy.
+ :param winfunc: the analysis window to apply to each frame. By default no window is applied. You can use numpy window functions here e.g. winfunc=numpy.hamming
+ :returns: A numpy array of size (NUMFRAMES by numcep) containing features. Each row holds 1 feature vector.
+ """
+ feat,energy = fbank(signal,samplerate,winlen,winstep,nfilt,nfft,lowfreq,highfreq,dither,remove_dc_offset,preemph,wintype)
+ feat = numpy.log(feat)
+ feat = dct(feat, type=2, axis=1, norm='ortho')[:,:numcep]
+ feat = lifter(feat,ceplifter)
+ if useEnergy: feat[:,0] = numpy.log(energy) # replace first cepstral coefficient with log of frame energy
+ return feat
+
+def fbank(signal,samplerate=16000,winlen=0.025,winstep=0.01,
+ nfilt=40,nfft=512,lowfreq=0,highfreq=None,dither=1.0,remove_dc_offset=True, preemph=0.97,
+ wintype='hamming'):
+ """Compute Mel-filterbank energy features from an audio signal.
+
+ :param signal: the audio signal from which to compute features. Should be an N*1 array
+ :param samplerate: the samplerate of the signal we are working with.
+ :param winlen: the length of the analysis window in seconds. Default is 0.025s (25 milliseconds)
+ :param winstep: the step between successive windows in seconds. Default is 0.01s (10 milliseconds)
+ :param nfilt: the number of filters in the filterbank, default 26.
+ :param nfft: the FFT size. Default is 512.
+ :param lowfreq: lowest band edge of mel filters. In Hz, default is 0.
+ :param highfreq: highest band edge of mel filters. In Hz, default is samplerate/2
+ :param preemph: apply preemphasis filter with preemph as coefficient. 0 is no filter. Default is 0.97.
+ :param winfunc: the analysis window to apply to each frame. By default no window is applied. You can use numpy window functions here e.g. winfunc=numpy.hamming
+ winfunc=lambda x:numpy.ones((x,))
+ :returns: 2 values. The first is a numpy array of size (NUMFRAMES by nfilt) containing features. Each row holds 1 feature vector. The
+ second return value is the energy in each frame (total energy, unwindowed)
+ """
+ highfreq= highfreq or samplerate/2
+ frames,raw_frames = sigproc.framesig(signal, winlen*samplerate, winstep*samplerate, dither, preemph, remove_dc_offset, wintype)
+ pspec = sigproc.powspec(frames,nfft) # nearly the same until this part
+ energy = numpy.sum(raw_frames**2,1) # this stores the raw energy in each frame
+ energy = numpy.where(energy == 0,numpy.finfo(float).eps,energy) # if energy is zero, we get problems with log
+
+ fb = get_filterbanks(nfilt,nfft,samplerate,lowfreq,highfreq)
+ feat = numpy.dot(pspec,fb.T) # compute the filterbank energies
+ feat = numpy.where(feat == 0,numpy.finfo(float).eps,feat) # if feat is zero, we get problems with log
+
+ return feat,energy
+
+def logfbank(signal,samplerate=16000,winlen=0.025,winstep=0.01,
+ nfilt=40,nfft=512,lowfreq=64,highfreq=None,dither=1.0,remove_dc_offset=True,preemph=0.97,wintype='hamming'):
+ """Compute log Mel-filterbank energy features from an audio signal.
+
+ :param signal: the audio signal from which to compute features. Should be an N*1 array
+ :param samplerate: the samplerate of the signal we are working with.
+ :param winlen: the length of the analysis window in seconds. Default is 0.025s (25 milliseconds)
+ :param winstep: the step between successive windows in seconds. Default is 0.01s (10 milliseconds)
+ :param nfilt: the number of filters in the filterbank, default 26.
+ :param nfft: the FFT size. Default is 512.
+ :param lowfreq: lowest band edge of mel filters. In Hz, default is 0.
+ :param highfreq: highest band edge of mel filters. In Hz, default is samplerate/2
+ :param preemph: apply preemphasis filter with preemph as coefficient. 0 is no filter. Default is 0.97.
+ :returns: A numpy array of size (NUMFRAMES by nfilt) containing features. Each row holds 1 feature vector.
+ """
+ feat,energy = fbank(signal,samplerate,winlen,winstep,nfilt,nfft,lowfreq,highfreq,dither, remove_dc_offset,preemph,wintype)
+ return numpy.log(feat)
+
+def hz2mel(hz):
+ """Convert a value in Hertz to Mels
+
+ :param hz: a value in Hz. This can also be a numpy array, conversion proceeds element-wise.
+ :returns: a value in Mels. If an array was passed in, an identical sized array is returned.
+ """
+ return 1127 * numpy.log(1+hz/700.0)
+
+def mel2hz(mel):
+ """Convert a value in Mels to Hertz
+
+ :param mel: a value in Mels. This can also be a numpy array, conversion proceeds element-wise.
+ :returns: a value in Hertz. If an array was passed in, an identical sized array is returned.
+ """
+ return 700 * (numpy.exp(mel/1127.0)-1)
+
+def get_filterbanks(nfilt=26,nfft=512,samplerate=16000,lowfreq=0,highfreq=None):
+ """Compute a Mel-filterbank. The filters are stored in the rows, the columns correspond
+ to fft bins. The filters are returned as an array of size nfilt * (nfft/2 + 1)
+
+ :param nfilt: the number of filters in the filterbank, default 20.
+ :param nfft: the FFT size. Default is 512.
+ :param samplerate: the samplerate of the signal we are working with. Affects mel spacing.
+ :param lowfreq: lowest band edge of mel filters, default 0 Hz
+ :param highfreq: highest band edge of mel filters, default samplerate/2
+ :returns: A numpy array of size nfilt * (nfft/2 + 1) containing filterbank. Each row holds 1 filter.
+ """
+ highfreq= highfreq or samplerate/2
+ assert highfreq <= samplerate/2, "highfreq is greater than samplerate/2"
+
+ # compute points evenly spaced in mels
+ lowmel = hz2mel(lowfreq)
+ highmel = hz2mel(highfreq)
+
+ # check kaldi/src/feat/Mel-computations.h
+ fbank = numpy.zeros([nfilt,nfft//2+1])
+ mel_freq_delta = (highmel-lowmel)/(nfilt+1)
+ for j in range(0,nfilt):
+ leftmel = lowmel+j*mel_freq_delta
+ centermel = lowmel+(j+1)*mel_freq_delta
+ rightmel = lowmel+(j+2)*mel_freq_delta
+ for i in range(0,nfft//2):
+ mel=hz2mel(i*samplerate/nfft)
+ if mel>leftmel and mel