liangym
3 years ago
committed by
GitHub
commit
152ebcbce8
@ -0,0 +1,102 @@
|
||||
([简体中文](./README_cn.md)|English)
|
||||
# Speech SSL (Self-Supervised Learning)
|
||||
|
||||
## Introduction
|
||||
Speech SSL, or Self-Supervised Learning, refers to a training method on the large-scale unlabeled speech dataset. The model trained in this way can produce a good acoustic representation, and can be applied to other downstream speech tasks by fine-tuning on labeled datasets.
|
||||
|
||||
This demo is an implementation to recognize text or produce the acoustic representation from a specific audio file by speech ssl models. It can be done by a single command or a few lines in python using `PaddleSpeech`.
|
||||
|
||||
## Usage
|
||||
### 1. Installation
|
||||
see [installation](https://github.com/PaddlePaddle/PaddleSpeech/blob/develop/docs/source/install.md).
|
||||
|
||||
You can choose one way from easy, meduim and hard to install paddlespeech.
|
||||
|
||||
### 2. Prepare Input File
|
||||
The input of this demo should be a WAV file(`.wav`), and the sample rate must be the same as the model.
|
||||
|
||||
Here are sample files for this demo that can be downloaded:
|
||||
```bash
|
||||
wget -c https://paddlespeech.bj.bcebos.com/PaddleAudio/en.wav
|
||||
```
|
||||
|
||||
### 3. Usage
|
||||
- Command Line(Recommended)
|
||||
```bash
|
||||
# to recognize text
|
||||
paddlespeech ssl --task asr --lang en --input ./en.wav
|
||||
|
||||
# to get acoustic representation
|
||||
paddlespeech ssl --task vector --lang en --input ./en.wav
|
||||
```
|
||||
|
||||
Usage:
|
||||
```bash
|
||||
paddlespeech ssl --help
|
||||
```
|
||||
Arguments:
|
||||
- `input`(required): Audio file to recognize.
|
||||
- `model`: Model type of asr task. Default: `wav2vec2ASR_librispeech`.
|
||||
- `task`: Output type. Default: `asr`.
|
||||
- `lang`: Model language. Default: `en`.
|
||||
- `sample_rate`: Sample rate of the model. Default: `16000`.
|
||||
- `config`: Config of asr task. Use pretrained model when it is None. Default: `None`.
|
||||
- `ckpt_path`: Model checkpoint. Use pretrained model when it is None. Default: `None`.
|
||||
- `yes`: No additional parameters required. Once set this parameter, it means accepting the request of the program by default, which includes transforming the audio sample rate. Default: `False`.
|
||||
- `device`: Choose device to execute model inference. Default: default device of paddlepaddle in current environment.
|
||||
- `verbose`: Show the log information.
|
||||
|
||||
|
||||
- Python API
|
||||
```python
|
||||
import paddle
|
||||
from paddlespeech.cli.ssl import SSLExecutor
|
||||
|
||||
ssl_executor = SSLExecutor()
|
||||
|
||||
# to recognize text
|
||||
text = ssl_executor(
|
||||
model='wav2vec2ASR_librispeech',
|
||||
task='asr',
|
||||
lang='en',
|
||||
sample_rate=16000,
|
||||
config=None, # Set `config` and `ckpt_path` to None to use pretrained model.
|
||||
ckpt_path=None,
|
||||
audio_file='./en.wav',
|
||||
device=paddle.get_device())
|
||||
print('ASR Result: \n{}'.format(text))
|
||||
|
||||
# to get acoustic representation
|
||||
feature = ssl_executor(
|
||||
model='wav2vec2',
|
||||
task='vector',
|
||||
lang='en',
|
||||
sample_rate=16000,
|
||||
config=None, # Set `config` and `ckpt_path` to None to use pretrained model.
|
||||
ckpt_path=None,
|
||||
audio_file='./en.wav',
|
||||
device=paddle.get_device())
|
||||
print('Representation: \n{}'.format(feature))
|
||||
```
|
||||
|
||||
Output:
|
||||
```bash
|
||||
ASR Result:
|
||||
i knocked at the door on the ancient side of the building
|
||||
|
||||
Representation:
|
||||
Tensor(shape=[1, 164, 1024], dtype=float32, place=Place(gpu:0), stop_gradient=True,
|
||||
[[[ 0.02351918, -0.12980647, 0.17868176, ..., 0.10118122,
|
||||
-0.04614586, 0.17853957],
|
||||
[ 0.02361383, -0.12978461, 0.17870593, ..., 0.10103855,
|
||||
-0.04638699, 0.17855372],
|
||||
[ 0.02345137, -0.12982975, 0.17883906, ..., 0.10104341,
|
||||
-0.04643029, 0.17856732],
|
||||
...,
|
||||
[ 0.02313030, -0.12918393, 0.17845058, ..., 0.10073373,
|
||||
-0.04701405, 0.17862988],
|
||||
[ 0.02176583, -0.12929161, 0.17797582, ..., 0.10097728,
|
||||
-0.04687393, 0.17864393],
|
||||
[ 0.05269200, 0.01297141, -0.23336855, ..., -0.11257174,
|
||||
-0.17227529, 0.20338398]]])
|
||||
```
|
||||
@ -0,0 +1,10 @@
|
||||
#!/bin/bash
|
||||
|
||||
# audio download
|
||||
wget -c https://paddlespeech.bj.bcebos.com/PaddleAudio/en.wav
|
||||
|
||||
# to recognize text
|
||||
paddlespeech ssl --task asr --lang en --input ./en.wav
|
||||
|
||||
# to get acoustic representation
|
||||
paddlespeech ssl --task vector --lang en --input ./en.wav
|
||||
@ -0,0 +1,95 @@
|
||||
([简体中文](./README_cn.md)|English)
|
||||
|
||||
## Introduction
|
||||
Whisper is a general-purpose speech recognition model. It is trained on a large dataset of diverse audio and is also a multi-task model that can perform multilingual speech recognition as well as speech translation and language identification.
|
||||
|
||||
Whisper model trained by OpenAI whisper https://github.com/openai/whisper
|
||||
|
||||
## Usage
|
||||
### 1. Installation
|
||||
see [installation](https://github.com/PaddlePaddle/PaddleSpeech/blob/develop/docs/source/install.md).
|
||||
|
||||
You can choose one way from easy, meduim and hard to install paddlespeech.
|
||||
|
||||
### 2. Prepare Input File
|
||||
The input of this demo should be a WAV file(`.wav`), and the sample rate must be the same as the model.
|
||||
|
||||
Here are sample files for this demo that can be downloaded:
|
||||
```bash
|
||||
wget -c https://paddlespeech.bj.bcebos.com/PaddleAudio/zh.wav
|
||||
```
|
||||
|
||||
### 3. Usage
|
||||
- Command Line(Recommended)
|
||||
```bash
|
||||
# to recognize text
|
||||
paddlespeech whisper --task transcribe --input ./zh.wav
|
||||
|
||||
# to change model English-Only base size model
|
||||
paddlespeech whisper --lang en --size base --task transcribe --input ./en.wav
|
||||
|
||||
# to recognize text and translate to English
|
||||
paddlespeech whisper --task translate --input ./zh.wav
|
||||
|
||||
```
|
||||
|
||||
Usage:
|
||||
```bash
|
||||
paddlespeech whisper --help
|
||||
```
|
||||
Arguments:
|
||||
- `input`(required): Audio file to recognize.
|
||||
- `model`: Model type of asr task. Default: `whisper-large`.
|
||||
- `task`: Output type. Default: `transcribe`.
|
||||
- `lang`: Model language. Default: ``. Use `en` to choice English-only model. Now [medium,base,small,tiny] size can support English-only.
|
||||
- `size`: Model size for decode. Defalut: `large`. Now can support [large,medium,base,small,tiny].
|
||||
- `language`: Set decode language. Default: `None`. Forcibly set the recognized language, which is determined by the model itself by default.
|
||||
- `sample_rate`: Sample rate of the model. Default: `16000`. Other sampling rates are not supported now.
|
||||
- `config`: Config of asr task. Use pretrained model when it is None. Default: `None`.
|
||||
- `ckpt_path`: Model checkpoint. Use pretrained model when it is None. Default: `None`.
|
||||
- `yes`: No additional parameters required. Once set this parameter, it means accepting the request of the program by default, which includes transforming the audio sample rate. Default: `False`.
|
||||
- `device`: Choose device to execute model inference. Default: default device of paddlepaddle in current environment.
|
||||
- `verbose`: Show the log information.
|
||||
|
||||
|
||||
- Python API
|
||||
```python
|
||||
import paddle
|
||||
from paddlespeech.cli.whisper import WhisperExecutor
|
||||
|
||||
whisper_executor = WhisperExecutor()
|
||||
|
||||
# to recognize text
|
||||
text = whisper_executor(
|
||||
model='whisper',
|
||||
task='transcribe',
|
||||
sample_rate=16000,
|
||||
config=None, # Set `config` and `ckpt_path` to None to use pretrained model.
|
||||
ckpt_path=None,
|
||||
audio_file='./zh.wav',
|
||||
device=paddle.get_device())
|
||||
print('ASR Result: \n{}'.format(text))
|
||||
|
||||
# to recognize text and translate to English
|
||||
feature = whisper_executor(
|
||||
model='whisper',
|
||||
task='translate',
|
||||
sample_rate=16000,
|
||||
config=None, # Set `config` and `ckpt_path` to None to use pretrained model.
|
||||
ckpt_path=None,
|
||||
audio_file='./zh.wav',
|
||||
device=paddle.get_device())
|
||||
print('Representation: \n{}'.format(feature))
|
||||
```
|
||||
|
||||
Output:
|
||||
```bash
|
||||
Transcribe Result:
|
||||
Detected language: Chinese
|
||||
[00:00.000 --> 00:05.000] 我认为跑步最重要的就是给我带来了身体健康
|
||||
{'text': '我认为跑步最重要的就是给我带来了身体健康', 'segments': [{'id': 0, 'seek': 0, 'start': 0.0, 'end': 5.0, 'text': '我认为跑步最重要的就是给我带来了身体健康', 'tokens': [50364, 1654, 7422, 97, 13992, 32585, 31429, 8661, 24928, 1546, 5620, 49076, 4845, 99, 34912, 19847, 29485, 44201, 6346, 115, 50614], 'temperature': 0.0, 'avg_logprob': -0.23577967557040128, 'compression_ratio': 0.28169014084507044, 'no_speech_prob': 0.028302080929279327}], 'language': 'zh'}
|
||||
|
||||
Translate Result:
|
||||
Detected language: Chinese
|
||||
[00:00.000 --> 00:05.000] I think the most important thing about running is that it brings me good health.
|
||||
{'text': ' I think the most important thing about running is that it brings me good health.', 'segments': [{'id': 0, 'seek': 0, 'start': 0.0, 'end': 5.0, 'text': ' I think the most important thing about running is that it brings me good health.', 'tokens': [50364, 286, 519, 264, 881, 1021, 551, 466, 2614, 307, 300, 309, 5607, 385, 665, 1585, 13, 50614], 'temperature': 0.0, 'avg_logprob': -0.47945233395225123, 'compression_ratio': 1.095890410958904, 'no_speech_prob': 0.028302080929279327}], 'language': 'zh'}
|
||||
@ -0,0 +1,13 @@
|
||||
#!/bin/bash
|
||||
|
||||
# audio download
|
||||
wget -c https://paddlespeech.bj.bcebos.com/PaddleAudio/zh.wav https://paddlespeech.bj.bcebos.com/PaddleAudio/en.wav
|
||||
|
||||
# to recognize text
|
||||
paddlespeech whisper --task transcribe --input ./zh.wav
|
||||
|
||||
# to recognize text and translate to English
|
||||
paddlespeech whisper --task translate --input ./zh.wav
|
||||
|
||||
# to change model English-Only model
|
||||
paddlespeech whisper --lang en --size base --task transcribe --input ./en.wav
|
||||
@ -0,0 +1,32 @@
|
||||
#!/bin/bash
|
||||
|
||||
train_output_path=$1
|
||||
|
||||
stage=0
|
||||
stop_stage=0
|
||||
|
||||
# pwgan
|
||||
if [ ${stage} -le 0 ] && [ ${stop_stage} -ge 0 ]; then
|
||||
python3 ${BIN_DIR}/../lite_predict.py \
|
||||
--inference_dir=${train_output_path}/pdlite \
|
||||
--am=fastspeech2_aishell3 \
|
||||
--voc=pwgan_aishell3 \
|
||||
--text=${BIN_DIR}/../sentences.txt \
|
||||
--output_dir=${train_output_path}/lite_infer_out \
|
||||
--phones_dict=dump/phone_id_map.txt \
|
||||
--speaker_dict=dump/speaker_id_map.txt \
|
||||
--spk_id=0
|
||||
fi
|
||||
|
||||
# hifigan
|
||||
if [ ${stage} -le 1 ] && [ ${stop_stage} -ge 1 ]; then
|
||||
python3 ${BIN_DIR}/../lite_predict.py \
|
||||
--inference_dir=${train_output_path}/pdlite \
|
||||
--am=fastspeech2_aishell3 \
|
||||
--voc=hifigan_aishell3 \
|
||||
--text=${BIN_DIR}/../sentences.txt \
|
||||
--output_dir=${train_output_path}/lite_infer_out \
|
||||
--phones_dict=dump/phone_id_map.txt \
|
||||
--speaker_dict=dump/speaker_id_map.txt \
|
||||
--spk_id=0
|
||||
fi
|
||||
@ -0,0 +1,43 @@
|
||||
#!/bin/bash
|
||||
|
||||
train_output_path=$1
|
||||
|
||||
stage=0
|
||||
stop_stage=0
|
||||
|
||||
# pwgan
|
||||
if [ ${stage} -le 0 ] && [ ${stop_stage} -ge 0 ]; then
|
||||
python3 ${BIN_DIR}/../lite_predict.py \
|
||||
--inference_dir=${train_output_path}/pdlite \
|
||||
--am=speedyspeech_csmsc \
|
||||
--voc=pwgan_csmsc \
|
||||
--text=${BIN_DIR}/../sentences.txt \
|
||||
--output_dir=${train_output_path}/lite_infer_out \
|
||||
--phones_dict=dump/phone_id_map.txt \
|
||||
--tones_dict=dump/tone_id_map.txt
|
||||
fi
|
||||
|
||||
# for more GAN Vocoders
|
||||
# multi band melgan
|
||||
if [ ${stage} -le 1 ] && [ ${stop_stage} -ge 1 ]; then
|
||||
python3 ${BIN_DIR}/../lite_predict.py \
|
||||
--inference_dir=${train_output_path}/pdlite \
|
||||
--am=speedyspeech_csmsc \
|
||||
--voc=mb_melgan_csmsc \
|
||||
--text=${BIN_DIR}/../sentences.txt \
|
||||
--output_dir=${train_output_path}/lite_infer_out \
|
||||
--phones_dict=dump/phone_id_map.txt \
|
||||
--tones_dict=dump/tone_id_map.txt
|
||||
fi
|
||||
|
||||
# hifigan
|
||||
if [ ${stage} -le 2 ] && [ ${stop_stage} -ge 2 ]; then
|
||||
python3 ${BIN_DIR}/../lite_predict.py \
|
||||
--inference_dir=${train_output_path}/pdlite \
|
||||
--am=speedyspeech_csmsc \
|
||||
--voc=hifigan_csmsc \
|
||||
--text=${BIN_DIR}/../sentences.txt \
|
||||
--output_dir=${train_output_path}/lite_infer_out \
|
||||
--phones_dict=dump/phone_id_map.txt \
|
||||
--tones_dict=dump/tone_id_map.txt
|
||||
fi
|
||||
@ -0,0 +1,40 @@
|
||||
#!/bin/bash
|
||||
|
||||
train_output_path=$1
|
||||
|
||||
stage=0
|
||||
stop_stage=0
|
||||
|
||||
# pwgan
|
||||
if [ ${stage} -le 0 ] && [ ${stop_stage} -ge 0 ]; then
|
||||
python3 ${BIN_DIR}/../lite_predict.py \
|
||||
--inference_dir=${train_output_path}/pdlite \
|
||||
--am=fastspeech2_csmsc \
|
||||
--voc=pwgan_csmsc \
|
||||
--text=${BIN_DIR}/../sentences.txt \
|
||||
--output_dir=${train_output_path}/lite_infer_out \
|
||||
--phones_dict=dump/phone_id_map.txt
|
||||
fi
|
||||
|
||||
# for more GAN Vocoders
|
||||
# multi band melgan
|
||||
if [ ${stage} -le 1 ] && [ ${stop_stage} -ge 1 ]; then
|
||||
python3 ${BIN_DIR}/../lite_predict.py \
|
||||
--inference_dir=${train_output_path}/pdlite \
|
||||
--am=fastspeech2_csmsc \
|
||||
--voc=mb_melgan_csmsc \
|
||||
--text=${BIN_DIR}/../sentences.txt \
|
||||
--output_dir=${train_output_path}/lite_infer_out \
|
||||
--phones_dict=dump/phone_id_map.txt
|
||||
fi
|
||||
|
||||
# hifigan
|
||||
if [ ${stage} -le 2 ] && [ ${stop_stage} -ge 2 ]; then
|
||||
python3 ${BIN_DIR}/../lite_predict.py \
|
||||
--inference_dir=${train_output_path}/pdlite \
|
||||
--am=fastspeech2_csmsc \
|
||||
--voc=hifigan_csmsc \
|
||||
--text=${BIN_DIR}/../sentences.txt \
|
||||
--output_dir=${train_output_path}/lite_infer_out \
|
||||
--phones_dict=dump/phone_id_map.txt
|
||||
fi
|
||||
@ -0,0 +1,47 @@
|
||||
#!/bin/bash
|
||||
|
||||
train_output_path=$1
|
||||
|
||||
stage=0
|
||||
stop_stage=0
|
||||
|
||||
# pwgan
|
||||
if [ ${stage} -le 0 ] && [ ${stop_stage} -ge 0 ]; then
|
||||
python3 ${BIN_DIR}/../lite_predict_streaming.py \
|
||||
--inference_dir=${train_output_path}/pdlite_streaming \
|
||||
--am=fastspeech2_csmsc \
|
||||
--am_stat=dump/train/speech_stats.npy \
|
||||
--voc=pwgan_csmsc \
|
||||
--text=${BIN_DIR}/../sentences.txt \
|
||||
--output_dir=${train_output_path}/lite_infer_out_streaming \
|
||||
--phones_dict=dump/phone_id_map.txt \
|
||||
--am_streaming=True
|
||||
fi
|
||||
|
||||
# for more GAN Vocoders
|
||||
# multi band melgan
|
||||
if [ ${stage} -le 1 ] && [ ${stop_stage} -ge 1 ]; then
|
||||
python3 ${BIN_DIR}/../lite_predict_streaming.py \
|
||||
--inference_dir=${train_output_path}/pdlite_streaming \
|
||||
--am=fastspeech2_csmsc \
|
||||
--am_stat=dump/train/speech_stats.npy \
|
||||
--voc=mb_melgan_csmsc \
|
||||
--text=${BIN_DIR}/../sentences.txt \
|
||||
--output_dir=${train_output_path}/lite_infer_out_streaming \
|
||||
--phones_dict=dump/phone_id_map.txt \
|
||||
--am_streaming=True
|
||||
fi
|
||||
|
||||
# hifigan
|
||||
if [ ${stage} -le 2 ] && [ ${stop_stage} -ge 2 ]; then
|
||||
python3 ${BIN_DIR}/../lite_predict_streaming.py \
|
||||
--inference_dir=${train_output_path}/pdlite_streaming \
|
||||
--am=fastspeech2_csmsc \
|
||||
--am_stat=dump/train/speech_stats.npy \
|
||||
--voc=hifigan_csmsc \
|
||||
--text=${BIN_DIR}/../sentences.txt \
|
||||
--output_dir=${train_output_path}/lite_infer_out_streaming \
|
||||
--phones_dict=dump/phone_id_map.txt \
|
||||
--am_streaming=True
|
||||
fi
|
||||
|
||||
@ -1,8 +1,8 @@
|
||||
# LibriSpeech
|
||||
|
||||
## Wav2VecASR
|
||||
train: Epoch 1, 1*V100-32G, batchsize:10
|
||||
train: Epoch 1, 1*V100-32G, batchsize: 6
|
||||
|
||||
| Model | Params | Config | Augmentation| Test set | Decode method | WER |
|
||||
| --- | --- | --- | --- | --- | --- | --- |
|
||||
| wav2vec2ASR | 302.86 M | conf/wav2vec2ASR.yaml | spec_aug | test-clean | greedy search | 0.018887 |
|
||||
| wav2vec2ASR | 302.86 M | conf/wav2vec2ASR.yaml | spec_aug | test-clean | greedy search | 0.018906 |
|
||||
|
||||
@ -0,0 +1,30 @@
|
||||
#!/bin/bash
|
||||
|
||||
train_output_path=$1
|
||||
|
||||
stage=0
|
||||
stop_stage=0
|
||||
|
||||
# pwgan
|
||||
if [ ${stage} -le 0 ] && [ ${stop_stage} -ge 0 ]; then
|
||||
python3 ${BIN_DIR}/../lite_predict.py \
|
||||
--inference_dir=${train_output_path}/pdlite \
|
||||
--am=fastspeech2_ljspeech \
|
||||
--voc=pwgan_ljspeech \
|
||||
--text=${BIN_DIR}/../sentences_en.txt \
|
||||
--output_dir=${train_output_path}/lite_infer_out \
|
||||
--phones_dict=dump/phone_id_map.txt \
|
||||
--lang=en
|
||||
fi
|
||||
|
||||
# hifigan
|
||||
if [ ${stage} -le 1 ] && [ ${stop_stage} -ge 1 ]; then
|
||||
python3 ${BIN_DIR}/../lite_predict.py \
|
||||
--inference_dir=${train_output_path}/pdlite \
|
||||
--am=fastspeech2_ljspeech \
|
||||
--voc=hifigan_ljspeech \
|
||||
--text=${BIN_DIR}/../sentences_en.txt \
|
||||
--output_dir=${train_output_path}/lite_infer_out \
|
||||
--phones_dict=dump/phone_id_map.txt \
|
||||
--lang=en
|
||||
fi
|
||||
@ -0,0 +1,34 @@
|
||||
#!/bin/bash
|
||||
|
||||
train_output_path=$1
|
||||
|
||||
stage=0
|
||||
stop_stage=0
|
||||
|
||||
# pwgan
|
||||
if [ ${stage} -le 0 ] && [ ${stop_stage} -ge 0 ]; then
|
||||
python3 ${BIN_DIR}/../lite_predict.py \
|
||||
--inference_dir=${train_output_path}/pdlite \
|
||||
--am=fastspeech2_vctk \
|
||||
--voc=pwgan_vctk \
|
||||
--text=${BIN_DIR}/../sentences_en.txt \
|
||||
--output_dir=${train_output_path}/lite_infer_out \
|
||||
--phones_dict=dump/phone_id_map.txt \
|
||||
--speaker_dict=dump/speaker_id_map.txt \
|
||||
--spk_id=0 \
|
||||
--lang=en
|
||||
fi
|
||||
|
||||
# hifigan
|
||||
if [ ${stage} -le 1 ] && [ ${stop_stage} -ge 1 ]; then
|
||||
python3 ${BIN_DIR}/../lite_predict.py \
|
||||
--inference_dir=${train_output_path}/pdlite \
|
||||
--am=fastspeech2_vctk \
|
||||
--voc=hifigan_vctk \
|
||||
--text=${BIN_DIR}/../sentences_en.txt \
|
||||
--output_dir=${train_output_path}/lite_infer_out \
|
||||
--phones_dict=dump/phone_id_map.txt \
|
||||
--speaker_dict=dump/speaker_id_map.txt \
|
||||
--spk_id=0 \
|
||||
--lang=en
|
||||
fi
|
||||
@ -0,0 +1,14 @@
|
||||
# Copyright (c) 2021 PaddlePaddle Authors. All Rights Reserved.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
from .infer import SSLExecutor
|
||||
@ -0,0 +1,14 @@
|
||||
# Copyright (c) 2021 PaddlePaddle Authors. All Rights Reserved.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
from .infer import WhisperExecutor
|
||||
@ -0,0 +1,13 @@
|
||||
# Copyright (c) 2022 PaddlePaddle Authors. All Rights Reserved.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
@ -0,0 +1,123 @@
|
||||
# Copyright (c) 2022 PaddlePaddle Authors. All Rights Reserved.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.∏
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
# Modified from Whisper (https://github.com/openai/whisper/whisper/)
|
||||
import os.path
|
||||
import sys
|
||||
|
||||
import distutils
|
||||
import numpy as np
|
||||
import paddle
|
||||
import soundfile
|
||||
from yacs.config import CfgNode
|
||||
|
||||
from paddlespeech.s2t.models.whisper import log_mel_spectrogram
|
||||
from paddlespeech.s2t.models.whisper import ModelDimensions
|
||||
from paddlespeech.s2t.models.whisper import transcribe
|
||||
from paddlespeech.s2t.models.whisper import Whisper
|
||||
from paddlespeech.s2t.training.cli import default_argument_parser
|
||||
from paddlespeech.s2t.utils.log import Log
|
||||
|
||||
logger = Log(__name__).getlog()
|
||||
|
||||
|
||||
class WhisperInfer():
|
||||
def __init__(self, config, args):
|
||||
self.args = args
|
||||
self.config = config
|
||||
self.audio_file = args.audio_file
|
||||
|
||||
paddle.set_device('gpu' if self.args.ngpu > 0 else 'cpu')
|
||||
config.pop("ngpu")
|
||||
|
||||
#load_model
|
||||
model_dict = paddle.load(self.config.model_file)
|
||||
config.pop("model_file")
|
||||
dims = ModelDimensions(**model_dict["dims"])
|
||||
self.model = Whisper(dims)
|
||||
self.model.load_dict(model_dict)
|
||||
|
||||
def run(self):
|
||||
check(args.audio_file)
|
||||
|
||||
with paddle.no_grad():
|
||||
temperature = config.pop("temperature")
|
||||
temperature_increment_on_fallback = config.pop(
|
||||
"temperature_increment_on_fallback")
|
||||
if temperature_increment_on_fallback is not None:
|
||||
temperature = tuple(
|
||||
np.arange(temperature, 1.0 + 1e-6,
|
||||
temperature_increment_on_fallback))
|
||||
else:
|
||||
temperature = [temperature]
|
||||
|
||||
#load audio
|
||||
mel = log_mel_spectrogram(
|
||||
args.audio_file, resource_path=config.resource_path)
|
||||
|
||||
result = transcribe(
|
||||
self.model, mel, temperature=temperature, **config)
|
||||
if args.result_file is not None:
|
||||
with open(args.result_file, 'w') as f:
|
||||
f.write(str(result))
|
||||
return result
|
||||
|
||||
|
||||
def check(audio_file: str):
|
||||
if not os.path.isfile(audio_file):
|
||||
print("Please input the right audio file path")
|
||||
sys.exit(-1)
|
||||
|
||||
logger.info("checking the audio file format......")
|
||||
try:
|
||||
_, sample_rate = soundfile.read(audio_file)
|
||||
except Exception as e:
|
||||
logger.error(str(e))
|
||||
logger.error(
|
||||
"can not open the wav file, please check the audio file format")
|
||||
sys.exit(-1)
|
||||
logger.info("The sample rate is %d" % sample_rate)
|
||||
assert (sample_rate == 16000)
|
||||
logger.info("The audio file format is right")
|
||||
|
||||
|
||||
def main(config, args):
|
||||
WhisperInfer(config, args).run()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
parser = default_argument_parser()
|
||||
# save asr result to
|
||||
parser.add_argument(
|
||||
"--result_file", type=str, help="path of save the asr result")
|
||||
parser.add_argument(
|
||||
"--audio_file", type=str, help="path of the input audio file")
|
||||
parser.add_argument(
|
||||
"--debug",
|
||||
type=distutils.util.strtobool,
|
||||
default=False,
|
||||
help="for debug.")
|
||||
args = parser.parse_args()
|
||||
|
||||
config = CfgNode(new_allowed=True)
|
||||
|
||||
if args.config:
|
||||
config.merge_from_file(args.config)
|
||||
if args.decode_cfg:
|
||||
decode_confs = CfgNode(new_allowed=True)
|
||||
decode_confs.merge_from_file(args.decode_cfg)
|
||||
config.decode = decode_confs
|
||||
if args.opts:
|
||||
config.merge_from_list(args.opts)
|
||||
config.freeze()
|
||||
main(config, args)
|
||||
@ -0,0 +1,17 @@
|
||||
# Copyright (c) 2021 PaddlePaddle Authors. All Rights Reserved.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
from .wav2vec2_ASR import Wav2vec2ASR
|
||||
from .wav2vec2_ASR import Wav2vec2Base
|
||||
|
||||
__all__ = ["Wav2vec2ASR", "Wav2vec2Base"]
|
||||
@ -0,0 +1,13 @@
|
||||
# Copyright (c) 2022 PaddlePaddle Authors. All Rights Reserved.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
@ -0,0 +1,97 @@
|
||||
# Authors
|
||||
# * Mirco Ravanelli 2020
|
||||
# * Guillermo Cámbara 2021
|
||||
# * Sarthak Yadav 2022
|
||||
# Copyright (c) 2022 PaddlePaddle Authors. All Rights Reserved.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
# Modified from speechbrain(https://github.com/speechbrain/speechbrain/blob/develop/speechbrain/nnet/normalization.py)
|
||||
import paddle.nn as nn
|
||||
|
||||
from paddlespeech.s2t.modules.align import BatchNorm1D
|
||||
|
||||
|
||||
class BatchNorm1d(nn.Layer):
|
||||
"""Applies 1d batch normalization to the input tensor.
|
||||
Arguments
|
||||
---------
|
||||
input_shape : tuple
|
||||
The expected shape of the input. Alternatively, use ``input_size``.
|
||||
input_size : int
|
||||
The expected size of the input. Alternatively, use ``input_shape``.
|
||||
eps : float
|
||||
This value is added to std deviation estimation to improve the numerical
|
||||
stability.
|
||||
momentum : float
|
||||
It is a value used for the running_mean and running_var computation.
|
||||
affine : bool
|
||||
When set to True, the affine parameters are learned.
|
||||
track_running_stats : bool
|
||||
When set to True, this module tracks the running mean and variance,
|
||||
and when set to False, this module does not track such statistics.
|
||||
combine_batch_time : bool
|
||||
When true, it combines batch an time axis.
|
||||
Example
|
||||
-------
|
||||
>>> input = paddle.randn([100, 10])
|
||||
>>> norm = BatchNorm1d(input_shape=input.shape)
|
||||
>>> output = norm(input)
|
||||
>>> output.shape
|
||||
Paddle.Shape([100, 10])
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
input_shape=None,
|
||||
input_size=None,
|
||||
eps=1e-05,
|
||||
momentum=0.9,
|
||||
combine_batch_time=False,
|
||||
skip_transpose=False, ):
|
||||
super().__init__()
|
||||
self.combine_batch_time = combine_batch_time
|
||||
self.skip_transpose = skip_transpose
|
||||
|
||||
if input_size is None and skip_transpose:
|
||||
input_size = input_shape[1]
|
||||
elif input_size is None:
|
||||
input_size = input_shape[-1]
|
||||
|
||||
self.norm = BatchNorm1D(input_size, momentum=momentum, epsilon=eps)
|
||||
|
||||
def forward(self, x):
|
||||
"""Returns the normalized input tensor.
|
||||
Arguments
|
||||
---------
|
||||
x : paddle.Tensor (batch, time, [channels])
|
||||
input to normalize. 2d or 3d tensors are expected in input
|
||||
4d tensors can be used when combine_dims=True.
|
||||
"""
|
||||
shape_or = x.shape
|
||||
if self.combine_batch_time:
|
||||
if x.ndim == 3:
|
||||
x = x.reshape(shape_or[0] * shape_or[1], shape_or[2])
|
||||
else:
|
||||
x = x.reshape(shape_or[0] * shape_or[1], shape_or[3],
|
||||
shape_or[2])
|
||||
|
||||
elif not self.skip_transpose:
|
||||
x = x.transpose([0, 2, 1])
|
||||
|
||||
x_n = self.norm(x)
|
||||
if self.combine_batch_time:
|
||||
x_n = x_n.reshape(shape_or)
|
||||
elif not self.skip_transpose:
|
||||
x_n = x_n.transpose([0, 2, 1])
|
||||
|
||||
return x_n
|
||||
@ -0,0 +1,13 @@
|
||||
# Copyright (c) 2022 PaddlePaddle Authors. All Rights Reserved.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
@ -0,0 +1,12 @@
|
||||
# MIT License, Copyright (c) 2022 OpenAI.
|
||||
# Copyright (c) 2022 PaddlePaddle Authors and . All Rights Reserved.
|
||||
#
|
||||
# Modified from OpenAI Whisper 2022 (https://github.com/openai/whisper/whisper/__init__.py)
|
||||
from paddlespeech.s2t.models.whisper.whipser import decode
|
||||
from paddlespeech.s2t.models.whisper.whipser import DecodingOptions
|
||||
from paddlespeech.s2t.models.whisper.whipser import DecodingResult
|
||||
from paddlespeech.s2t.models.whisper.whipser import detect_language
|
||||
from paddlespeech.s2t.models.whisper.whipser import log_mel_spectrogram
|
||||
from paddlespeech.s2t.models.whisper.whipser import ModelDimensions
|
||||
from paddlespeech.s2t.models.whisper.whipser import transcribe
|
||||
from paddlespeech.s2t.models.whisper.whipser import Whisper
|
||||
@ -0,0 +1,362 @@
|
||||
# MIT License, Copyright (c) 2022 OpenAI.
|
||||
# Copyright (c) 2022 PaddlePaddle Authors and . All Rights Reserved.
|
||||
#
|
||||
# Modified from OpenAI Whisper 2022 (https://github.com/openai/whisper/whisper/tokenizer.py)
|
||||
import os
|
||||
from dataclasses import dataclass
|
||||
from functools import lru_cache
|
||||
from typing import List
|
||||
from typing import Optional
|
||||
from typing import Tuple
|
||||
from typing import Union
|
||||
|
||||
import numpy as np
|
||||
import paddle
|
||||
from paddlenlp.transformers import GPTTokenizer
|
||||
|
||||
LANGUAGES = {
|
||||
"en": "english",
|
||||
"zh": "chinese",
|
||||
"de": "german",
|
||||
"es": "spanish",
|
||||
"ru": "russian",
|
||||
"ko": "korean",
|
||||
"fr": "french",
|
||||
"ja": "japanese",
|
||||
"pt": "portuguese",
|
||||
"tr": "turkish",
|
||||
"pl": "polish",
|
||||
"ca": "catalan",
|
||||
"nl": "dutch",
|
||||
"ar": "arabic",
|
||||
"sv": "swedish",
|
||||
"it": "italian",
|
||||
"id": "indonesian",
|
||||
"hi": "hindi",
|
||||
"fi": "finnish",
|
||||
"vi": "vietnamese",
|
||||
"iw": "hebrew",
|
||||
"uk": "ukrainian",
|
||||
"el": "greek",
|
||||
"ms": "malay",
|
||||
"cs": "czech",
|
||||
"ro": "romanian",
|
||||
"da": "danish",
|
||||
"hu": "hungarian",
|
||||
"ta": "tamil",
|
||||
"no": "norwegian",
|
||||
"th": "thai",
|
||||
"ur": "urdu",
|
||||
"hr": "croatian",
|
||||
"bg": "bulgarian",
|
||||
"lt": "lithuanian",
|
||||
"la": "latin",
|
||||
"mi": "maori",
|
||||
"ml": "malayalam",
|
||||
"cy": "welsh",
|
||||
"sk": "slovak",
|
||||
"te": "telugu",
|
||||
"fa": "persian",
|
||||
"lv": "latvian",
|
||||
"bn": "bengali",
|
||||
"sr": "serbian",
|
||||
"az": "azerbaijani",
|
||||
"sl": "slovenian",
|
||||
"kn": "kannada",
|
||||
"et": "estonian",
|
||||
"mk": "macedonian",
|
||||
"br": "breton",
|
||||
"eu": "basque",
|
||||
"is": "icelandic",
|
||||
"hy": "armenian",
|
||||
"ne": "nepali",
|
||||
"mn": "mongolian",
|
||||
"bs": "bosnian",
|
||||
"kk": "kazakh",
|
||||
"sq": "albanian",
|
||||
"sw": "swahili",
|
||||
"gl": "galician",
|
||||
"mr": "marathi",
|
||||
"pa": "punjabi",
|
||||
"si": "sinhala",
|
||||
"km": "khmer",
|
||||
"sn": "shona",
|
||||
"yo": "yoruba",
|
||||
"so": "somali",
|
||||
"af": "afrikaans",
|
||||
"oc": "occitan",
|
||||
"ka": "georgian",
|
||||
"be": "belarusian",
|
||||
"tg": "tajik",
|
||||
"sd": "sindhi",
|
||||
"gu": "gujarati",
|
||||
"am": "amharic",
|
||||
"yi": "yiddish",
|
||||
"lo": "lao",
|
||||
"uz": "uzbek",
|
||||
"fo": "faroese",
|
||||
"ht": "haitian creole",
|
||||
"ps": "pashto",
|
||||
"tk": "turkmen",
|
||||
"nn": "nynorsk",
|
||||
"mt": "maltese",
|
||||
"sa": "sanskrit",
|
||||
"lb": "luxembourgish",
|
||||
"my": "myanmar",
|
||||
"bo": "tibetan",
|
||||
"tl": "tagalog",
|
||||
"mg": "malagasy",
|
||||
"as": "assamese",
|
||||
"tt": "tatar",
|
||||
"haw": "hawaiian",
|
||||
"ln": "lingala",
|
||||
"ha": "hausa",
|
||||
"ba": "bashkir",
|
||||
"jw": "javanese",
|
||||
"su": "sundanese",
|
||||
}
|
||||
|
||||
# language code lookup by name, with a few language aliases
|
||||
TO_LANGUAGE_CODE = {
|
||||
**{language: code for code, language in LANGUAGES.items()},
|
||||
"burmese": "my",
|
||||
"valencian": "ca",
|
||||
"flemish": "nl",
|
||||
"haitian": "ht",
|
||||
"letzeburgesch": "lb",
|
||||
"pushto": "ps",
|
||||
"panjabi": "pa",
|
||||
"moldavian": "ro",
|
||||
"moldovan": "ro",
|
||||
"sinhalese": "si",
|
||||
"castilian": "es",
|
||||
}
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class Tokenizer:
|
||||
"""A thin wrapper around `GPTTokenizer` providing quick access to special tokens"""
|
||||
|
||||
tokenizer: "GPTTokenizer"
|
||||
language: Optional[str]
|
||||
sot_sequence: Tuple[int]
|
||||
|
||||
def encode(self, text, **kwargs):
|
||||
return self.tokenizer.encode(text, **kwargs)
|
||||
|
||||
def decode(self,
|
||||
token_ids: Union[int, List[int], np.ndarray, paddle.Tensor],
|
||||
**kwargs):
|
||||
if len(token_ids) > 1:
|
||||
ids_list = []
|
||||
for ids in token_ids:
|
||||
if paddle.is_tensor(ids):
|
||||
ids = ids.item()
|
||||
if ids < len(self.tokenizer):
|
||||
ids_list.append(ids)
|
||||
token_ids = ids_list
|
||||
|
||||
return self.tokenizer.decode(token_ids, **kwargs)
|
||||
|
||||
def decode_with_timestamps(self, tokens) -> str:
|
||||
"""
|
||||
Timestamp tokens are above the special tokens' id range and are ignored by `decode()`.
|
||||
This method decodes given tokens with timestamps tokens annotated, e.g. "<|1.08|>".
|
||||
"""
|
||||
outputs = [[]]
|
||||
for token in tokens:
|
||||
if token >= self.timestamp_begin:
|
||||
timestamp = f"<|{(token - self.timestamp_begin) * 0.02:.2f}|>"
|
||||
outputs.append(timestamp)
|
||||
outputs.append([])
|
||||
else:
|
||||
outputs[-1].append(token)
|
||||
outputs = [
|
||||
s if isinstance(s, str) else self.tokenizer.decode(s)
|
||||
for s in outputs
|
||||
]
|
||||
return "".join(outputs)
|
||||
|
||||
@property
|
||||
@lru_cache()
|
||||
def eot(self) -> int:
|
||||
return self.tokenizer.eos_token_id
|
||||
|
||||
@property
|
||||
@lru_cache()
|
||||
def sot(self) -> int:
|
||||
return self._get_single_token_id("<|startoftranscript|>")
|
||||
|
||||
@property
|
||||
@lru_cache()
|
||||
def sot_lm(self) -> int:
|
||||
return self._get_single_token_id("<|startoflm|>")
|
||||
|
||||
@property
|
||||
@lru_cache()
|
||||
def sot_prev(self) -> int:
|
||||
return self._get_single_token_id("<|startofprev|>")
|
||||
|
||||
@property
|
||||
@lru_cache()
|
||||
def no_speech(self) -> int:
|
||||
return self._get_single_token_id("<|nospeech|>")
|
||||
|
||||
@property
|
||||
@lru_cache()
|
||||
def no_timestamps(self) -> int:
|
||||
return self._get_single_token_id("<|notimestamps|>")
|
||||
|
||||
@property
|
||||
@lru_cache()
|
||||
def timestamp_begin(self) -> int:
|
||||
return self.tokenizer.all_special_ids[-1] + 1
|
||||
|
||||
@property
|
||||
@lru_cache()
|
||||
def language_token(self) -> int:
|
||||
"""Returns the token id corresponding to the value of the `language` field"""
|
||||
if self.language is None:
|
||||
raise ValueError(
|
||||
"This tokenizer does not have language token configured")
|
||||
|
||||
additional_tokens = dict(
|
||||
zip(
|
||||
self.tokenizer.additional_special_tokens,
|
||||
self.tokenizer.additional_special_tokens_ids, ))
|
||||
candidate = f"<|{self.language}|>"
|
||||
if candidate in additional_tokens:
|
||||
return additional_tokens[candidate]
|
||||
|
||||
raise KeyError(f"Language {self.language} not found in tokenizer.")
|
||||
|
||||
@property
|
||||
@lru_cache()
|
||||
def all_language_tokens(self) -> Tuple[int]:
|
||||
result = []
|
||||
for token, token_id in zip(
|
||||
self.tokenizer.additional_special_tokens,
|
||||
self.tokenizer.additional_special_tokens_ids, ):
|
||||
if token.strip("<|>") in LANGUAGES:
|
||||
result.append(token_id)
|
||||
return tuple(result)
|
||||
|
||||
@property
|
||||
@lru_cache()
|
||||
def all_language_codes(self) -> Tuple[str]:
|
||||
return tuple(
|
||||
self.decode([l]).strip("<|>") for l in self.all_language_tokens)
|
||||
|
||||
@property
|
||||
@lru_cache()
|
||||
def sot_sequence_including_notimestamps(self) -> Tuple[int]:
|
||||
return tuple(list(self.sot_sequence) + [self.no_timestamps])
|
||||
|
||||
@property
|
||||
@lru_cache()
|
||||
def non_speech_tokens(self) -> Tuple[int]:
|
||||
"""
|
||||
Returns the list of tokens to suppress in order to avoid any speaker tags or non-speech
|
||||
annotations, to prevent sampling texts that are not actually spoken in the audio, e.g.
|
||||
|
||||
- ♪♪♪
|
||||
- ( SPEAKING FOREIGN LANGUAGE )
|
||||
- [DAVID] Hey there,
|
||||
|
||||
keeping basic punctuations like commas, periods, question marks, exclamation points, etc.
|
||||
"""
|
||||
symbols = list("\"#()*+/:;<=>@[\\]^_`{|}~「」『』")
|
||||
symbols += "<< >> <<< >>> -- --- -( -[ (' (\" (( )) ((( ))) [[ ]] {{ }} ♪♪ ♪♪♪".split(
|
||||
)
|
||||
|
||||
# symbols that may be a single token or multiple tokens depending on the tokenizer.
|
||||
# In case they're multiple tokens, suppress the first token, which is safe because:
|
||||
# These are between U+2640 and U+267F miscellaneous symbols that are okay to suppress
|
||||
# in generations, and in the 3-byte UTF-8 representation they share the first two bytes.
|
||||
miscellaneous = set("♩♪♫♬♭♮♯")
|
||||
assert all(0x2640 <= ord(c) <= 0x267F for c in miscellaneous)
|
||||
|
||||
# allow hyphens "-" and single quotes "'" between words, but not at the beginning of a word
|
||||
result = {
|
||||
self.tokenizer.encode(" -").input_ids[0],
|
||||
self.tokenizer.encode(" '").input_ids[0]
|
||||
}
|
||||
for symbol in symbols + list(miscellaneous):
|
||||
for tokens in [
|
||||
self.tokenizer.encode(symbol).input_ids,
|
||||
self.tokenizer.encode(" " + symbol).input_ids
|
||||
]:
|
||||
if len(tokens) == 1 or symbol in miscellaneous:
|
||||
result.add(tokens[0])
|
||||
|
||||
return tuple(sorted(result))
|
||||
|
||||
def _get_single_token_id(self, text) -> int:
|
||||
tokens = self.tokenizer.encode(text).input_ids
|
||||
assert len(tokens) == 1, f"{text} is not encoded as a single token"
|
||||
return tokens[0]
|
||||
|
||||
|
||||
@lru_cache(maxsize=None)
|
||||
def build_tokenizer(resource_path: str, name: str="gpt2"):
|
||||
os.environ["TOKENIZERS_PARALLELISM"] = "false"
|
||||
path = os.path.join(resource_path, "assets", name)
|
||||
tokenizer = GPTTokenizer.from_pretrained(path)
|
||||
|
||||
specials = [
|
||||
"<|startoftranscript|>",
|
||||
* [f"<|{lang}|>" for lang in LANGUAGES.keys()],
|
||||
"<|translate|>",
|
||||
"<|transcribe|>",
|
||||
"<|startoflm|>",
|
||||
"<|startofprev|>",
|
||||
"<|nospeech|>",
|
||||
"<|notimestamps|>",
|
||||
]
|
||||
|
||||
tokenizer.add_special_tokens(dict(additional_special_tokens=specials))
|
||||
return tokenizer
|
||||
|
||||
|
||||
@lru_cache(maxsize=None)
|
||||
def get_tokenizer(
|
||||
multilingual: bool,
|
||||
resource_path: str,
|
||||
*,
|
||||
task: Optional[str]=None, # Literal["transcribe", "translate", None]
|
||||
language: Optional[str]=None, ) -> Tokenizer:
|
||||
if language is not None:
|
||||
language = language.lower()
|
||||
if language not in LANGUAGES:
|
||||
if language in TO_LANGUAGE_CODE:
|
||||
language = TO_LANGUAGE_CODE[language]
|
||||
else:
|
||||
raise ValueError(f"Unsupported language: {language}")
|
||||
|
||||
if multilingual:
|
||||
tokenizer_name = "multilingual"
|
||||
task = task or "transcribe"
|
||||
language = language or "en"
|
||||
else:
|
||||
tokenizer_name = "gpt2"
|
||||
task = None
|
||||
language = None
|
||||
|
||||
tokenizer = build_tokenizer(
|
||||
resource_path=resource_path, name=tokenizer_name)
|
||||
all_special_ids: List[int] = tokenizer.all_special_ids
|
||||
sot: int = all_special_ids[1]
|
||||
translate: int = all_special_ids[-6]
|
||||
transcribe: int = all_special_ids[-5]
|
||||
|
||||
langs = tuple(LANGUAGES.keys())
|
||||
sot_sequence = [sot]
|
||||
if language is not None:
|
||||
sot_sequence.append(sot + 1 + langs.index(language))
|
||||
if task is not None:
|
||||
sot_sequence.append(transcribe if task == "transcribe" else translate)
|
||||
|
||||
return Tokenizer(
|
||||
tokenizer=tokenizer,
|
||||
language=language,
|
||||
sot_sequence=tuple(sot_sequence))
|
||||
@ -0,0 +1,92 @@
|
||||
# MIT License, Copyright (c) 2022 OpenAI.
|
||||
# Copyright (c) 2022 PaddlePaddle Authors and . All Rights Reserved.
|
||||
#
|
||||
# Modified from OpenAI Whisper 2022 (https://github.com/openai/whisper/whisper/utils.py)
|
||||
import zlib
|
||||
from typing import Iterator
|
||||
from typing import TextIO
|
||||
|
||||
|
||||
def exact_div(x, y):
|
||||
assert x % y == 0
|
||||
return x // y
|
||||
|
||||
|
||||
def str2bool(string):
|
||||
str2val = {"True": True, "False": False}
|
||||
if string in str2val:
|
||||
return str2val[string]
|
||||
else:
|
||||
raise ValueError(f"Expected one of {set(str2val.keys())}, got {string}")
|
||||
|
||||
|
||||
def optional_int(string):
|
||||
return None if string == "None" else int(string)
|
||||
|
||||
|
||||
def optional_float(string):
|
||||
return None if string == "None" else float(string)
|
||||
|
||||
|
||||
def compression_ratio(text) -> float:
|
||||
return len(text) / len(zlib.compress(text.encode("utf-8")))
|
||||
|
||||
|
||||
def format_timestamp(seconds: float,
|
||||
always_include_hours: bool=False,
|
||||
decimal_marker: str='.'):
|
||||
assert seconds >= 0, "non-negative timestamp expected"
|
||||
milliseconds = round(seconds * 1000.0)
|
||||
|
||||
hours = milliseconds // 3_600_000
|
||||
milliseconds -= hours * 3_600_000
|
||||
|
||||
minutes = milliseconds // 60_000
|
||||
milliseconds -= minutes * 60_000
|
||||
|
||||
seconds = milliseconds // 1_000
|
||||
milliseconds -= seconds * 1_000
|
||||
|
||||
hours_marker = f"{hours:02d}:" if always_include_hours or hours > 0 else ""
|
||||
return f"{hours_marker}{minutes:02d}:{seconds:02d}{decimal_marker}{milliseconds:03d}"
|
||||
|
||||
|
||||
def write_txt(transcript: Iterator[dict], file: TextIO):
|
||||
for segment in transcript:
|
||||
print(segment['text'].strip(), file=file, flush=True)
|
||||
|
||||
|
||||
def write_vtt(transcript: Iterator[dict], file: TextIO):
|
||||
print("WEBVTT\n", file=file)
|
||||
for segment in transcript:
|
||||
print(
|
||||
f"{format_timestamp(segment['start'])} --> {format_timestamp(segment['end'])}\n"
|
||||
f"{segment['text'].strip().replace('-->', '->')}\n",
|
||||
file=file,
|
||||
flush=True, )
|
||||
|
||||
|
||||
def write_srt(transcript: Iterator[dict], file: TextIO):
|
||||
"""
|
||||
Write a transcript to a file in SRT format.
|
||||
|
||||
Example usage:
|
||||
from pathlib import Path
|
||||
from whisper.utils import write_srt
|
||||
|
||||
result = transcribe(model, audio_path, temperature=temperature, **args)
|
||||
|
||||
# save SRT
|
||||
audio_basename = Path(audio_path).stem
|
||||
with open(Path(output_dir) / (audio_basename + ".srt"), "w", encoding="utf-8") as srt:
|
||||
write_srt(result["segments"], file=srt)
|
||||
"""
|
||||
for i, segment in enumerate(transcript, start=1):
|
||||
# write srt lines
|
||||
print(
|
||||
f"{i}\n"
|
||||
f"{format_timestamp(segment['start'], always_include_hours=True, decimal_marker=',')} --> "
|
||||
f"{format_timestamp(segment['end'], always_include_hours=True, decimal_marker=',')}\n"
|
||||
f"{segment['text'].strip().replace('-->', '->')}\n",
|
||||
file=file,
|
||||
flush=True, )
|
||||
File diff suppressed because it is too large
Load Diff
@ -0,0 +1,21 @@
|
||||
MIT License
|
||||
|
||||
Copyright (c) 2022 OpenAI
|
||||
|
||||
Permission is hereby granted, free of charge, to any person obtaining a copy
|
||||
of this software and associated documentation files (the "Software"), to deal
|
||||
in the Software without restriction, including without limitation the rights
|
||||
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
||||
copies of the Software, and to permit persons to whom the Software is
|
||||
furnished to do so, subject to the following conditions:
|
||||
|
||||
The above copyright notice and this permission notice shall be included in all
|
||||
copies or substantial portions of the Software.
|
||||
|
||||
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
||||
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
||||
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
||||
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
||||
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
||||
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
||||
SOFTWARE.
|
||||
@ -0,0 +1,168 @@
|
||||
# Copyright (c) 2022 PaddlePaddle Authors. All Rights Reserved.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
import argparse
|
||||
from pathlib import Path
|
||||
|
||||
import soundfile as sf
|
||||
from timer import timer
|
||||
|
||||
from paddlespeech.t2s.exps.syn_utils import get_frontend
|
||||
from paddlespeech.t2s.exps.syn_utils import get_lite_am_output
|
||||
from paddlespeech.t2s.exps.syn_utils import get_lite_predictor
|
||||
from paddlespeech.t2s.exps.syn_utils import get_lite_voc_output
|
||||
from paddlespeech.t2s.exps.syn_utils import get_sentences
|
||||
|
||||
|
||||
def parse_args():
|
||||
parser = argparse.ArgumentParser(
|
||||
description="Paddle Infernce with acoustic model & vocoder.")
|
||||
# acoustic model
|
||||
parser.add_argument(
|
||||
'--am',
|
||||
type=str,
|
||||
default='fastspeech2_csmsc',
|
||||
choices=[
|
||||
'speedyspeech_csmsc',
|
||||
'fastspeech2_csmsc',
|
||||
'fastspeech2_aishell3',
|
||||
'fastspeech2_ljspeech',
|
||||
'fastspeech2_vctk',
|
||||
'fastspeech2_mix',
|
||||
],
|
||||
help='Choose acoustic model type of tts task.')
|
||||
parser.add_argument(
|
||||
"--phones_dict", type=str, default=None, help="phone vocabulary file.")
|
||||
parser.add_argument(
|
||||
"--tones_dict", type=str, default=None, help="tone vocabulary file.")
|
||||
parser.add_argument(
|
||||
"--speaker_dict", type=str, default=None, help="speaker id map file.")
|
||||
parser.add_argument(
|
||||
'--spk_id',
|
||||
type=int,
|
||||
default=0,
|
||||
help='spk id for multi speaker acoustic model')
|
||||
# voc
|
||||
parser.add_argument(
|
||||
'--voc',
|
||||
type=str,
|
||||
default='pwgan_csmsc',
|
||||
choices=[
|
||||
'pwgan_csmsc',
|
||||
'pwgan_aishell3',
|
||||
'pwgan_ljspeech',
|
||||
'pwgan_vctk',
|
||||
'mb_melgan_csmsc',
|
||||
'hifigan_csmsc',
|
||||
'hifigan_aishell3',
|
||||
'hifigan_ljspeech',
|
||||
'hifigan_vctk',
|
||||
],
|
||||
help='Choose vocoder type of tts task.')
|
||||
# other
|
||||
parser.add_argument(
|
||||
'--lang',
|
||||
type=str,
|
||||
default='zh',
|
||||
help='Choose model language. zh or en or mix')
|
||||
parser.add_argument(
|
||||
"--text",
|
||||
type=str,
|
||||
help="text to synthesize, a 'utt_id sentence' pair per line")
|
||||
parser.add_argument(
|
||||
"--inference_dir", type=str, help="dir to save inference models")
|
||||
parser.add_argument("--output_dir", type=str, help="output dir")
|
||||
|
||||
args, _ = parser.parse_known_args()
|
||||
return args
|
||||
|
||||
|
||||
# only inference for models trained with csmsc now
|
||||
def main():
|
||||
args = parse_args()
|
||||
|
||||
# frontend
|
||||
frontend = get_frontend(
|
||||
lang=args.lang,
|
||||
phones_dict=args.phones_dict,
|
||||
tones_dict=args.tones_dict)
|
||||
|
||||
# am_predictor
|
||||
am_predictor = get_lite_predictor(
|
||||
model_dir=args.inference_dir, model_file=args.am + "_x86.nb")
|
||||
# model: {model_name}_{dataset}
|
||||
am_dataset = args.am[args.am.rindex('_') + 1:]
|
||||
|
||||
# voc_predictor
|
||||
voc_predictor = get_lite_predictor(
|
||||
model_dir=args.inference_dir, model_file=args.voc + "_x86.nb")
|
||||
|
||||
output_dir = Path(args.output_dir)
|
||||
output_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
sentences = get_sentences(text_file=args.text, lang=args.lang)
|
||||
|
||||
merge_sentences = True
|
||||
fs = 24000 if am_dataset != 'ljspeech' else 22050
|
||||
# warmup
|
||||
for utt_id, sentence in sentences[:3]:
|
||||
with timer() as t:
|
||||
mel = get_lite_am_output(
|
||||
input=sentence,
|
||||
am_predictor=am_predictor,
|
||||
am=args.am,
|
||||
frontend=frontend,
|
||||
lang=args.lang,
|
||||
merge_sentences=merge_sentences,
|
||||
speaker_dict=args.speaker_dict,
|
||||
spk_id=args.spk_id, )
|
||||
wav = get_lite_voc_output(voc_predictor=voc_predictor, input=mel)
|
||||
speed = wav.size / t.elapse
|
||||
rtf = fs / speed
|
||||
print(
|
||||
f"{utt_id}, mel: {mel.shape}, wave: {wav.shape}, time: {t.elapse}s, Hz: {speed}, RTF: {rtf}."
|
||||
)
|
||||
|
||||
print("warm up done!")
|
||||
|
||||
N = 0
|
||||
T = 0
|
||||
for utt_id, sentence in sentences:
|
||||
with timer() as t:
|
||||
mel = get_lite_am_output(
|
||||
input=sentence,
|
||||
am_predictor=am_predictor,
|
||||
am=args.am,
|
||||
frontend=frontend,
|
||||
lang=args.lang,
|
||||
merge_sentences=merge_sentences,
|
||||
speaker_dict=args.speaker_dict,
|
||||
spk_id=args.spk_id, )
|
||||
wav = get_lite_voc_output(voc_predictor=voc_predictor, input=mel)
|
||||
|
||||
N += wav.size
|
||||
T += t.elapse
|
||||
speed = wav.size / t.elapse
|
||||
rtf = fs / speed
|
||||
|
||||
sf.write(output_dir / (utt_id + ".wav"), wav, samplerate=fs)
|
||||
print(
|
||||
f"{utt_id}, mel: {mel.shape}, wave: {wav.shape}, time: {t.elapse}s, Hz: {speed}, RTF: {rtf}."
|
||||
)
|
||||
|
||||
print(f"{utt_id} done!")
|
||||
print(f"generation speed: {N / T}Hz, RTF: {fs / (N / T) }")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@ -0,0 +1,230 @@
|
||||
# Copyright (c) 2022 PaddlePaddle Authors. All Rights Reserved.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
import argparse
|
||||
from pathlib import Path
|
||||
|
||||
import numpy as np
|
||||
import soundfile as sf
|
||||
from timer import timer
|
||||
|
||||
from paddlespeech.t2s.exps.syn_utils import denorm
|
||||
from paddlespeech.t2s.exps.syn_utils import get_chunks
|
||||
from paddlespeech.t2s.exps.syn_utils import get_frontend
|
||||
from paddlespeech.t2s.exps.syn_utils import get_lite_am_sublayer_output
|
||||
from paddlespeech.t2s.exps.syn_utils import get_lite_predictor
|
||||
from paddlespeech.t2s.exps.syn_utils import get_lite_streaming_am_output
|
||||
from paddlespeech.t2s.exps.syn_utils import get_lite_voc_output
|
||||
from paddlespeech.t2s.exps.syn_utils import get_sentences
|
||||
from paddlespeech.t2s.exps.syn_utils import run_frontend
|
||||
from paddlespeech.t2s.utils import str2bool
|
||||
|
||||
|
||||
def parse_args():
|
||||
parser = argparse.ArgumentParser(
|
||||
description="Paddle Infernce with acoustic model & vocoder.")
|
||||
# acoustic model
|
||||
parser.add_argument(
|
||||
'--am',
|
||||
type=str,
|
||||
default='fastspeech2_csmsc',
|
||||
choices=['fastspeech2_csmsc'],
|
||||
help='Choose acoustic model type of tts task.')
|
||||
parser.add_argument(
|
||||
"--am_stat",
|
||||
type=str,
|
||||
default=None,
|
||||
help="mean and standard deviation used to normalize spectrogram when training acoustic model."
|
||||
)
|
||||
parser.add_argument(
|
||||
"--phones_dict", type=str, default=None, help="phone vocabulary file.")
|
||||
parser.add_argument(
|
||||
"--tones_dict", type=str, default=None, help="tone vocabulary file.")
|
||||
parser.add_argument(
|
||||
"--speaker_dict", type=str, default=None, help="speaker id map file.")
|
||||
parser.add_argument(
|
||||
'--spk_id',
|
||||
type=int,
|
||||
default=0,
|
||||
help='spk id for multi speaker acoustic model')
|
||||
# voc
|
||||
parser.add_argument(
|
||||
'--voc',
|
||||
type=str,
|
||||
default='pwgan_csmsc',
|
||||
choices=['pwgan_csmsc', 'mb_melgan_csmsc', 'hifigan_csmsc'],
|
||||
help='Choose vocoder type of tts task.')
|
||||
# other
|
||||
parser.add_argument(
|
||||
'--lang',
|
||||
type=str,
|
||||
default='zh',
|
||||
help='Choose model language. zh or en')
|
||||
parser.add_argument(
|
||||
"--text",
|
||||
type=str,
|
||||
help="text to synthesize, a 'utt_id sentence' pair per line")
|
||||
parser.add_argument(
|
||||
"--inference_dir", type=str, help="dir to save inference models")
|
||||
parser.add_argument("--output_dir", type=str, help="output dir")
|
||||
# inference
|
||||
|
||||
# streaming related
|
||||
parser.add_argument(
|
||||
"--am_streaming",
|
||||
type=str2bool,
|
||||
default=False,
|
||||
help="whether use streaming acoustic model")
|
||||
parser.add_argument(
|
||||
"--block_size", type=int, default=42, help="block size of am streaming")
|
||||
parser.add_argument(
|
||||
"--pad_size", type=int, default=12, help="pad size of am streaming")
|
||||
|
||||
args, _ = parser.parse_known_args()
|
||||
return args
|
||||
|
||||
|
||||
# only inference for models trained with csmsc now
|
||||
def main():
|
||||
args = parse_args()
|
||||
|
||||
# frontend
|
||||
frontend = get_frontend(
|
||||
lang=args.lang,
|
||||
phones_dict=args.phones_dict,
|
||||
tones_dict=args.tones_dict)
|
||||
|
||||
# am_predictor
|
||||
am_encoder_infer_predictor = get_lite_predictor(
|
||||
model_dir=args.inference_dir,
|
||||
model_file=args.am + "_am_encoder_infer" + "_x86.nb")
|
||||
am_decoder_predictor = get_lite_predictor(
|
||||
model_dir=args.inference_dir,
|
||||
model_file=args.am + "_am_decoder" + "_x86.nb")
|
||||
am_postnet_predictor = get_lite_predictor(
|
||||
model_dir=args.inference_dir,
|
||||
model_file=args.am + "_am_postnet" + "_x86.nb")
|
||||
am_mu, am_std = np.load(args.am_stat)
|
||||
# model: {model_name}_{dataset}
|
||||
am_dataset = args.am[args.am.rindex('_') + 1:]
|
||||
|
||||
# voc_predictor
|
||||
voc_predictor = get_lite_predictor(
|
||||
model_dir=args.inference_dir, model_file=args.voc + "_x86.nb")
|
||||
|
||||
output_dir = Path(args.output_dir)
|
||||
output_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
sentences = get_sentences(text_file=args.text, lang=args.lang)
|
||||
|
||||
merge_sentences = True
|
||||
|
||||
fs = 24000 if am_dataset != 'ljspeech' else 22050
|
||||
# warmup
|
||||
for utt_id, sentence in sentences[:3]:
|
||||
with timer() as t:
|
||||
normalized_mel = get_lite_streaming_am_output(
|
||||
input=sentence,
|
||||
am_encoder_infer_predictor=am_encoder_infer_predictor,
|
||||
am_decoder_predictor=am_decoder_predictor,
|
||||
am_postnet_predictor=am_postnet_predictor,
|
||||
frontend=frontend,
|
||||
lang=args.lang,
|
||||
merge_sentences=merge_sentences, )
|
||||
mel = denorm(normalized_mel, am_mu, am_std)
|
||||
wav = get_lite_voc_output(voc_predictor=voc_predictor, input=mel)
|
||||
speed = wav.size / t.elapse
|
||||
rtf = fs / speed
|
||||
print(
|
||||
f"{utt_id}, mel: {mel.shape}, wave: {wav.shape}, time: {t.elapse}s, Hz: {speed}, RTF: {rtf}."
|
||||
)
|
||||
|
||||
print("warm up done!")
|
||||
|
||||
N = 0
|
||||
T = 0
|
||||
block_size = args.block_size
|
||||
pad_size = args.pad_size
|
||||
get_tone_ids = False
|
||||
for utt_id, sentence in sentences:
|
||||
with timer() as t:
|
||||
# frontend
|
||||
frontend_dict = run_frontend(
|
||||
frontend=frontend,
|
||||
text=sentence,
|
||||
merge_sentences=merge_sentences,
|
||||
get_tone_ids=get_tone_ids,
|
||||
lang=args.lang)
|
||||
phone_ids = frontend_dict['phone_ids']
|
||||
phones = phone_ids[0].numpy()
|
||||
# acoustic model
|
||||
orig_hs = get_lite_am_sublayer_output(
|
||||
am_encoder_infer_predictor, input=phones)
|
||||
|
||||
if args.am_streaming:
|
||||
hss = get_chunks(orig_hs, block_size, pad_size)
|
||||
chunk_num = len(hss)
|
||||
mel_list = []
|
||||
for i, hs in enumerate(hss):
|
||||
am_decoder_output = get_lite_am_sublayer_output(
|
||||
am_decoder_predictor, input=hs)
|
||||
am_postnet_output = get_lite_am_sublayer_output(
|
||||
am_postnet_predictor,
|
||||
input=np.transpose(am_decoder_output, (0, 2, 1)))
|
||||
am_output_data = am_decoder_output + np.transpose(
|
||||
am_postnet_output, (0, 2, 1))
|
||||
normalized_mel = am_output_data[0]
|
||||
|
||||
sub_mel = denorm(normalized_mel, am_mu, am_std)
|
||||
# clip output part of pad
|
||||
if i == 0:
|
||||
sub_mel = sub_mel[:-pad_size]
|
||||
elif i == chunk_num - 1:
|
||||
# 最后一块的右侧一定没有 pad 够
|
||||
sub_mel = sub_mel[pad_size:]
|
||||
else:
|
||||
# 倒数几块的右侧也可能没有 pad 够
|
||||
sub_mel = sub_mel[pad_size:(block_size + pad_size) -
|
||||
sub_mel.shape[0]]
|
||||
mel_list.append(sub_mel)
|
||||
mel = np.concatenate(mel_list, axis=0)
|
||||
|
||||
else:
|
||||
am_decoder_output = get_lite_am_sublayer_output(
|
||||
am_decoder_predictor, input=orig_hs)
|
||||
am_postnet_output = get_lite_am_sublayer_output(
|
||||
am_postnet_predictor,
|
||||
input=np.transpose(am_decoder_output, (0, 2, 1)))
|
||||
am_output_data = am_decoder_output + np.transpose(
|
||||
am_postnet_output, (0, 2, 1))
|
||||
normalized_mel = am_output_data[0]
|
||||
mel = denorm(normalized_mel, am_mu, am_std)
|
||||
# vocoder
|
||||
wav = get_lite_voc_output(voc_predictor=voc_predictor, input=mel)
|
||||
|
||||
N += wav.size
|
||||
T += t.elapse
|
||||
speed = wav.size / t.elapse
|
||||
rtf = fs / speed
|
||||
|
||||
sf.write(output_dir / (utt_id + ".wav"), wav, samplerate=24000)
|
||||
print(
|
||||
f"{utt_id}, mel: {mel.shape}, wave: {wav.shape}, time: {t.elapse}s, Hz: {speed}, RTF: {rtf}."
|
||||
)
|
||||
|
||||
print(f"{utt_id} done!")
|
||||
print(f"generation speed: {N / T}Hz, RTF: {fs / (N / T) }")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@ -1,27 +1,77 @@
|
||||
# u2/u2pp Streaming ASR
|
||||
# U2/U2++ Streaming ASR
|
||||
|
||||
A C++ deployment example for `PaddleSpeech/examples/wenetspeech/asr1` recipe. The model is static model from `export`, how to export model please see [here](../../../../examples/wenetspeech/asr1/). If you want using exported model, `run.sh` will download it, for the model link please see `run.sh`.
|
||||
|
||||
This example will demonstrate how to using the u2/u2++ model to recognize `wav` and compute `CER`. We using AISHELL-1 as test data.
|
||||
|
||||
## Testing with Aishell Test Data
|
||||
|
||||
### Download wav and model
|
||||
### Source `path.sh` first
|
||||
|
||||
```bash
|
||||
source path.sh
|
||||
```
|
||||
|
||||
All bins are under `echo $SPEECHX_BUILD` dir.
|
||||
|
||||
### Download dataset and model
|
||||
|
||||
```
|
||||
./run.sh --stop_stage 0
|
||||
```
|
||||
|
||||
### compute feature
|
||||
### process `cmvn` and compute feature
|
||||
|
||||
```
|
||||
```bash
|
||||
./run.sh --stage 1 --stop_stage 1
|
||||
```
|
||||
|
||||
### decoding using feature
|
||||
If you only want to convert `cmvn` file format, can using this cmd:
|
||||
|
||||
```bash
|
||||
./local/feat.sh --stage 1 --stop_stage 1
|
||||
```
|
||||
|
||||
### Decoding using `feature` input
|
||||
|
||||
```
|
||||
./run.sh --stage 2 --stop_stage 2
|
||||
```
|
||||
|
||||
### decoding using wav
|
||||
### Decoding using `wav` input
|
||||
|
||||
```
|
||||
./run.sh --stage 3 --stop_stage 3
|
||||
```
|
||||
|
||||
This stage using `u2_recognizer_main` to recognize wav file.
|
||||
|
||||
The input is `scp` file which look like this:
|
||||
```text
|
||||
# head data/split1/1/aishell_test.scp
|
||||
BAC009S0764W0121 /workspace/PaddleSpeech/speechx/examples/u2pp_ol/wenetspeech/data/test/S0764/BAC009S0764W0121.wav
|
||||
BAC009S0764W0122 /workspace/PaddleSpeech/speechx/examples/u2pp_ol/wenetspeech/data/test/S0764/BAC009S0764W0122.wav
|
||||
...
|
||||
BAC009S0764W0125 /workspace/PaddleSpeech/speechx/examples/u2pp_ol/wenetspeech/data/test/S0764/BAC009S0764W0125.wav
|
||||
```
|
||||
|
||||
If you want to recognize one wav, you can make `scp` file like this:
|
||||
```text
|
||||
key path/to/wav/file
|
||||
```
|
||||
|
||||
Then specify `--wav_rspecifier=` param for `u2_recognizer_main` bin. For other flags meaning, please see `help`:
|
||||
```bash
|
||||
u2_recognizer_main --help
|
||||
```
|
||||
|
||||
The exmaple using `u2_recgonize_main` bin please see `local/recognizer.sh`.
|
||||
|
||||
### Decoding with `wav` using quant model
|
||||
|
||||
`local/recognizer_quant.sh` is same to `local/recognizer.sh`, but using quanted model.
|
||||
|
||||
|
||||
## Results
|
||||
|
||||
Please see [here](./RESULTS.md).
|
||||
|
||||
Loading…
Reference in new issue