diff --git a/docs/tutorial/st/st_tutorial.ipynb b/docs/tutorial/st/st_tutorial.ipynb new file mode 100644 index 000000000..2fb850535 --- /dev/null +++ b/docs/tutorial/st/st_tutorial.ipynb @@ -0,0 +1,529 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": { + "collapsed": false + }, + "source": [ + "\"Fork\n", + "\n", + "\n", + "# End-to-End Speech (to Text) Translation " + ] + }, + { + "cell_type": "markdown", + "metadata": { + "collapsed": false + }, + "source": [ + "# 前言" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "collapsed": false + }, + "source": [ + "## 背景知识\n", + "语音翻译（ST, Speech Translation）是一项从一段源语言音频中翻译出目标语言的任务。\n", + "本章主要针对语音到文本的翻译，比如，从一段英文语音中，得到中文的翻译文本。" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "collapsed": false + }, + "source": [ + "## 基本方法\n", + "### 级联模型（Cascaded）, ASR -> MT\n", + "级联模型由独立的两个模型，语音识别模型（ASR）和机器翻译模型（MT）组成。先通过 ASR 模型从语音中识别出源语言的相应文本，在利用 MT 模型将相应文本翻译成目标语言。\n", + "![cascaded](https://ai-studio-static-online.cdn.bcebos.com/af40e4d580764d1cb07e7f889d31e4e4e2f0839753a543bbad2e6334b48ad8cd)\n", + "\n", + "\n", + "### 端到端模型 （End-to-End） \n", + "端到端模型不显式对输入语音做文字识别，而直接生成翻译结果。\n", + "![e2e](https://ai-studio-static-online.cdn.bcebos.com/2043ffd3d9e34054b542dcd67ebb6e4441d91e8d4bf148a88fdc7ca452356c95)\n", + "\n", + "\n", + "相对于端到端模型，级联模型存在以下一些问题:\n", + "\n", + "1.错误传播（error propagation），由 ASR 识别错误所产生的错误文本，也会传递给 MT 模型，往往会导致生成更糟糕的翻译结果。\n", + "\n", + "2.时延叠加（latency accumulation），因为使用两个级联的模型，需要对输入数据进行多次处理，实际的时延是两个模型时延的累加，效率低于端到端模型。" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "collapsed": false + }, + "source": [ + "# 使用Transformer进行端到端语音翻译的的基本流程\n", + "## 基础模型\n", + "由于 ASR 章节已经介绍了 Transformer 以及语音特征抽取，在此便不做过多介绍，感兴趣的同学可以去相关章节进行了解。\n", + "\n", + "本小节，主要讨论利用 transformer（seq2seq）进行ST与ASR的异同。\n", + "\n", + "相似之处在于，两者都可以看做是从语音（speech）到文本（text）的任务。将语音作为输入，而将文字作为输出，区别只在于生成结果是对应语言的识别结果，还是另一语言的翻译结果。\n", + "\n", + "因此，我们只需要将数据中的目标文本替换为翻译文本（$Y$），便可利用 ASR 的模型结构实现语音翻译。\n", + "\n", + "规范化地讲，对于 ASR，利用包含语音（$S$）和转写文本（$X$）的数据集，训练得到一个模型 $M_{ASR}$，能对任意输入的源语言语音 $\\hat{S}$ 进行文字识别，输出结果 $\\hat{X}$。\n", + "\n", + "而ST的语料集，通常包含语音（$S$）、转写文本（$X$）以及翻译文本（$Y$），只需将ASR实践中的转写文本$X$替换为对应的翻译文本 $Y$，便可利用同样的流程得到一个翻译模型 $M_{ST}$，其能对任意输入的源语言语音 $\\hat{S}$ 进行翻译，输出结果 $\\hat{Y}$\n", + "\n", + "值得注意的是，相较于 ASR 任务而言，在 ST 中，因为翻译文本与源语音不存在单调对齐（monotonic aligned）的性质，因此 CTC 模块不能将翻译结果作为目标来使用，此处涉及一些学术细节，感兴趣的同学可以自行去了解 [CTC](https://github.com/PaddlePaddle/PaddleSpeech/blob/develop/docs/topic/ctc/ctc_loss.ipynb) 的具体内容。\n", + "\n", + "> 我们会在 [PaddleSpeech](https://github.com/PaddlePaddle/PaddleSpeech) 中放一些 Topic 的技术文章（如 [CTC](https://github.com/PaddlePaddle/PaddleSpeech/blob/develop/docs/topic/ctc/ctc_loss.ipynb) ），欢迎大家 star 关注。\n", + "\n", + "## 辅助任务训练，提升效果（ASR MTL）\n", + "\n", + "相比与 ASR 任务，ST 任务对于数据的标注和获取更加困难，通常很难获取大量的训练数据。\n", + "\n", + "因此，我们讲讨论如何更有效利用已有数据，提升 ST 模型的效果。\n", + "\n", + "1.先利用 ASR 对模型进行预训练，得到一个编码器能够有效的捕捉语音中的语义信息，在此基础上再进一步利用翻译数据训练ST模型。\n", + "\n", + "2.相较于 ASR 任务的二元组数据（$S$,$X$），通常包含三元组数据（$S$,$X$,$Y$）的ST任务能够自然有效的进行多任务学习。\n", + "顾名思义，我们可以将ASR任务作为辅助任务，将两个任务进行联合训练，利用ASR任务的辅助提升 ST 模型的效果。\n", + "具体上讲，如图所示，可以利用一个共享的编码器对语音进行编码，同时利用两个独立的解码器，分别执行 ASR 和 ST 任务。\n", + "![mtl](https://ai-studio-static-online.cdn.bcebos.com/3896f104527947cbba8f459d4477b0a8cb9e4d527d6f4da29bb7792d69b66d3c)\n", + "我们将实战中进行演示。\n", + "\n", + "## 引入预训练模型，提升效果 （FAT-ST PT）\n", + "\n", + "相比于文本到文本的机器翻译具有充足的语料（通常上百万条），语音到文本的翻译的语料很匮乏。那是否可以将文本到文本的翻译语料利用上来提升 ST 的模型效果呢？答案是肯定的。\n", + "\n", + "FAT 模型[1],借鉴了 Bert[2] 和 TLM[3]的 masked language model 预训练思路，并将其拓展到语音翻译的跨语言、跨模态（语音和文本）的场景。可以应对三元组（$S$,$X$,$Y$）中任意的单一或组合的数据类型。\n", + "举例来说，它可以利用纯语音或文本数据集（$S$|$X$|$Y$），也可以利用 ASR 数据集（$S$,$Y$），甚至文本翻译数据（$X$,$Y$)。在这种预训练模型的基础上进行 ST 的训练，能够有效解决训练数据匮乏的困境，提升最终的翻译效果。\n", + "![fat](https://ai-studio-static-online.cdn.bcebos.com/ef4b677e4cdf466fa5264d3b9dc976326e3c2046b1704c7da693015a71fc8a68)\n", + "\n" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "collapsed": false + }, + "source": [ + "# 实战\n", + "## ST 多任务学习，将 ASR 作为辅助任务\n", + "### 数据集: [Ted语音翻译数据集](http://www.nlpr.ia.ac.cn/cip/dataset.htm)（英文语音$\\rightarrow$中文文本）[4]\n", + "## 准备工作\n", + "## 特征抽取\n", + "参考语音识别的相关章节，略。\n", + "## 多任务模型\n", + "Transformer 内容参考语音识别的相关章节，略。\n" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "collapsed": false + }, + "source": [ + "## Stage 1 准备工作\n", + "### 安装 paddlespeech" + ] + }, + { + "cell_type": "code", + "execution_count": 1, + "metadata": { + "collapsed": false + }, + "outputs": [], + "source": [ + "!pip install -U pip\n", + "!pip install paddlespeech" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "collapsed": false + }, + "source": [ + "### 导入 python 包" + ] + }, + { + "cell_type": "code", + "execution_count": 2, + "metadata": { + "collapsed": false + }, + "outputs": [], + "source": [ + "import os\n", + "import paddle\n", + "import numpy as np\n", + "import kaldiio\n", + "import subprocess\n", + "from kaldiio import WriteHelper\n", + "from yacs.config import CfgNode\n", + "import IPython.display as dp\n", + "import warnings\n", + "warnings.filterwarnings(\"ignore\")\n", + "\n", + "from paddlespeech.s2t.frontend.featurizer.text_featurizer import TextFeaturizer\n", + "from paddlespeech.s2t.models.u2_st import U2STModel" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "collapsed": false + }, + "source": [ + "### 获取预训练模型和参数并配置" + ] + }, + { + "cell_type": "code", + "execution_count": 3, + "metadata": { + "collapsed": false + }, + "outputs": [], + "source": [ + "!wget -nc https://paddlespeech.bj.bcebos.com/s2t/ted_en_zh/st1/fat_st_ted-en-zh.tar.gz\n", + "!tar xzvf fat_st_ted-en-zh.tar.gz" + ] + }, + { + "cell_type": "code", + "execution_count": 4, + "metadata": { + "collapsed": false + }, + "outputs": [], + "source": [ + "config_path = \"conf/transformer_mtl_noam.yaml\" \n", + "\n", + "# 读取 conf 文件并结构化\n", + "st_config = CfgNode(new_allowed=True)\n", + "st_config.merge_from_file(config_path)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "collapsed": false + }, + "source": [ + "### 下载并配置 kaldi 环境" + ] + }, + { + "cell_type": "code", + "execution_count": 6, + "metadata": { + "collapsed": false + }, + "outputs": [], + "source": [ + "!wget -nc https://paddlespeech.bj.bcebos.com/s2t/ted_en_zh/st1/kaldi_bins.tar.gz\n", + "!tar xzvf kaldi_bins.tar.gz" + ] + }, + { + "cell_type": "code", + "execution_count": 7, + "metadata": { + "collapsed": false + }, + "outputs": [], + "source": [ + "kaldi_bins_path = os.path.abspath('kaldi_bins')\n", + "print(kaldi_bins_path)\n", + "if 'LD_LIBRARY_PATH' not in os.environ:\n", + " os.environ['LD_LIBRARY_PATH'] = f'{kaldi_bins_path}'\n", + "else:\n", + " os.environ['LD_LIBRARY_PATH'] += f':{kaldi_bins_path}'\n", + "os.environ['PATH'] += f':{kaldi_bins_path}'" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "collapsed": false + }, + "source": [ + "## Stage 2 获取特征\n", + "### 提取 kaldi 特征" + ] + }, + { + "cell_type": "code", + "execution_count": 8, + "metadata": { + "collapsed": false + }, + "outputs": [], + "source": [ + "def get_kaldi_feat(wav_path, config=st_config):\n", + " \"\"\"\n", + " Input preprocess and return paddle.Tensor stored in self.input.\n", + " Input content can be a file(wav).\n", + " \"\"\"\n", + " wav_file = os.path.abspath(wav_path)\n", + " cmvn = config.collator.cmvn_path\n", + " utt_name = '_tmp'\n", + "\n", + " # Get the object for feature extraction\n", + " fbank_extract_command = [\n", + " 'compute-fbank-feats', '--num-mel-bins=80', '--verbose=2',\n", + " '--sample-frequency=16000', 'scp:-', 'ark:-'\n", + " ]\n", + " fbank_extract_process = subprocess.Popen(fbank_extract_command,\n", + " stdin=subprocess.PIPE,\n", + " stdout=subprocess.PIPE,\n", + " stderr=subprocess.PIPE)\n", + " fbank_extract_process.stdin.write(\n", + " f'{utt_name} {wav_file}'.encode('utf8'))\n", + " fbank_extract_process.stdin.close()\n", + " fbank_feat = dict(kaldiio.load_ark(\n", + " fbank_extract_process.stdout))[utt_name]\n", + "\n", + " extract_command = ['compute-kaldi-pitch-feats', 'scp:-', 'ark:-']\n", + " pitch_extract_process = subprocess.Popen(extract_command,\n", + " stdin=subprocess.PIPE,\n", + " stdout=subprocess.PIPE,\n", + " stderr=subprocess.PIPE)\n", + " pitch_extract_process.stdin.write(\n", + " f'{utt_name} {wav_file}'.encode('utf8'))\n", + " process_command = ['process-kaldi-pitch-feats', 'ark:', 'ark:-']\n", + " pitch_process = subprocess.Popen(process_command,\n", + " stdin=pitch_extract_process.stdout,\n", + " stdout=subprocess.PIPE,\n", + " stderr=subprocess.PIPE)\n", + " pitch_extract_process.stdin.close()\n", + " pitch_feat = dict(kaldiio.load_ark(\n", + " pitch_process.stdout))[utt_name]\n", + " concated_feat = np.concatenate((fbank_feat, pitch_feat), axis=1)\n", + " raw_feat = f\"{utt_name}.raw\"\n", + " with WriteHelper(f'ark,scp:{raw_feat}.ark,{raw_feat}.scp') as writer:\n", + " writer(utt_name, concated_feat)\n", + " cmvn_command = [\n", + " \"apply-cmvn\", \"--norm-vars=true\", cmvn, f'scp:{raw_feat}.scp',\n", + " 'ark:-'\n", + " ]\n", + " cmvn_process = subprocess.Popen(cmvn_command,\n", + " stdout=subprocess.PIPE,\n", + " stderr=subprocess.PIPE)\n", + " process_command = ['copy-feats', '--compress=true', 'ark:-', 'ark:-']\n", + " process = subprocess.Popen(process_command,\n", + " stdin=cmvn_process.stdout,\n", + " stdout=subprocess.PIPE,\n", + " stderr=subprocess.PIPE)\n", + " norm_feat = dict(kaldiio.load_ark(process.stdout))[utt_name]\n", + " audio = paddle.to_tensor(norm_feat).unsqueeze(0)\n", + " audio_len = paddle.to_tensor(audio.shape[1], dtype='int64')\n", + " return audio, audio_len" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "collapsed": false + }, + "source": [ + "### 构建文本特征提取对象" + ] + }, + { + "cell_type": "code", + "execution_count": 9, + "metadata": { + "collapsed": false + }, + "outputs": [], + "source": [ + "text_feature = TextFeaturizer(\n", + " unit_type=st_config.collator.unit_type,\n", + " vocab=st_config.collator.vocab_filepath,\n", + " spm_model_prefix=st_config.collator.spm_model_prefix)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "collapsed": false + }, + "source": [ + "## Stage 3 使用模型获得结果" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "collapsed": false + }, + "source": [ + "### 构建 ST 模型" + ] + }, + { + "cell_type": "code", + "execution_count": 10, + "metadata": { + "collapsed": false + }, + "outputs": [], + "source": [ + "model_conf = st_config.model\n", + "model_conf.input_dim = st_config.collator.feat_dim\n", + "model_conf.output_dim = text_feature.vocab_size\n", + "print(model_conf)\n", + "model = U2STModel.from_config(model_conf)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "collapsed": false + }, + "source": [ + "### 加载预训练模型" + ] + }, + { + "cell_type": "code", + "execution_count": 11, + "metadata": { + "collapsed": false + }, + "outputs": [], + "source": [ + "params_path = \"exp/transformer_mtl_noam/checkpoints/fat_st_ted-en-zh.pdparams\"\n", + "model_dict = paddle.load(params_path)\n", + "model.set_state_dict(model_dict)\n", + "model.eval()" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "collapsed": false + }, + "source": [ + "### 预测" + ] + }, + { + "cell_type": "code", + "execution_count": 14, + "metadata": { + "collapsed": false + }, + "outputs": [], + "source": [ + "# 下载wav\n", + "!wget -nc https://paddlespeech.bj.bcebos.com/PaddleAudio/74109_0147917-0156334.wav\n", + "!wget -nc https://paddlespeech.bj.bcebos.com/PaddleAudio/120221_0278694-0283831.wav\n", + "!wget -nc https://paddlespeech.bj.bcebos.com/PaddleAudio/15427_0822000-0833000.wav\n", + "\n", + "wav_file = '74109_0147917-0156334.wav'\n", + "# wav_file = '120221_0278694-0283831.wav'\n", + "# wav_file = '15427_0822000-0833000.wav'\n", + "\n", + "transcript = \"my hair is short like a boy 's and i wear boy 's clothes but i 'm a girl and you know how sometimes you like to wear a pink dress and sometimes you like to wear your comfy jammies\"\n", + "dp.Audio(wav_file)" + ] + }, + { + "cell_type": "code", + "execution_count": 15, + "metadata": { + "collapsed": false + }, + "outputs": [], + "source": [ + "audio, audio_len = get_kaldi_feat(wav_file)\n", + "cfg = st_config.decoding\n", + "\n", + "\n", + "res = model.decode(audio,\n", + " audio_len,\n", + " text_feature=text_feature,\n", + " decoding_method=cfg.decoding_method,\n", + " beam_size=cfg.beam_size,\n", + " word_reward=cfg.word_reward,\n", + " decoding_chunk_size=cfg.decoding_chunk_size,\n", + " num_decoding_left_chunks=cfg.num_decoding_left_chunks,\n", + " simulate_streaming=cfg.simulate_streaming)\n", + "print(\"对应英文: {}\".format(transcript))\n", + "print(\"翻译结果: {}\".format(\"\".join(res[0].split())))\n" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "collapsed": false + }, + "source": [ + "# 参考文献\n", + "\n", + "1.Zheng, Renjie, Junkun Chen, Mingbo Ma, and Liang Huang. \"Fused acoustic and text encoding for multimodal bilingual pretraining and speech translation.\" ICML 2021.\n", + "\n", + "2.Devlin, Jacob, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. \"Bert: Pre-training of deep bidirectional transformers for language understanding.\" NAACL 2019.\n", + "\n", + "3.Conneau, Alexis, and Guillaume Lample. \"Cross-lingual language model pretraining.\" NIPS 2019.\n", + "\n", + "4.Liu, Yuchen, Hao Xiong, Zhongjun He, Jiajun Zhang, Hua Wu, Haifeng Wang, and Chengqing Zong. \"End-to-end speech translation with knowledge distillation.\" Interspeech 2019." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "collapsed": false + }, + "source": [ + "# PaddleSpeech\n", + "\n", + "请关注我们的 [Github Repo](https://github.com/PaddlePaddle/PaddleSpeech/)，非常欢迎加入以下微信群参与讨论：\n", + "- 扫描二维码\n", + "- 添加运营小姐姐微信\n", + "- 通过后回复【语音】\n", + "- 系统自动邀请加入技术群\n", + "\n", + "
" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "collapsed": false + }, + "outputs": [], + "source": [] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "py35-paddle1.2.0" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.7.4" + } + }, + "nbformat": 4, + "nbformat_minor": 1 +}