{ "cells": [ { "cell_type": "markdown", "metadata": { "collapsed": false }, "source": [ "\"Fork\n", "\n", "\n", "# End-to-End Speech (to Text) Translation " ] }, { "cell_type": "markdown", "metadata": { "collapsed": false }, "source": [ "# 前言" ] }, { "cell_type": "markdown", "metadata": { "collapsed": false }, "source": [ "## 背景知识\n", "语音翻译（ST, Speech Translation）是一项从一段源语言音频中翻译出目标语言的任务。\n", "本章主要针对语音到文本的翻译，比如，从一段英文语音中，得到中文的翻译文本。" ] }, { "cell_type": "markdown", "metadata": { "collapsed": false }, "source": [ "## 基本方法\n", "### 级联模型（Cascaded）, ASR -> MT\n", "级联模型由独立的两个模型，语音识别模型（ASR）和机器翻译模型（MT）组成。先通过 ASR 模型从语音中识别出源语言的相应文本，在利用 MT 模型将相应文本翻译成目标语言。\n", "![cascaded](https://ai-studio-static-online.cdn.bcebos.com/af40e4d580764d1cb07e7f889d31e4e4e2f0839753a543bbad2e6334b48ad8cd)\n", "\n", "\n", "### 端到端模型 （End-to-End） \n", "端到端模型不显式对输入语音做文字识别，而直接生成翻译结果。\n", "![e2e](https://ai-studio-static-online.cdn.bcebos.com/2043ffd3d9e34054b542dcd67ebb6e4441d91e8d4bf148a88fdc7ca452356c95)\n", "\n", "\n", "相对于端到端模型，级联模型存在以下一些问题:\n", "\n", "1.错误传播（error propagation），由 ASR 识别错误所产生的错误文本，也会传递给 MT 模型，往往会导致生成更糟糕的翻译结果。\n", "\n", "2.时延叠加（latency accumulation），因为使用两个级联的模型，需要对输入数据进行多次处理，实际的时延是两个模型时延的累加，效率低于端到端模型。" ] }, { "cell_type": "markdown", "metadata": { "collapsed": false }, "source": [ "# 使用Transformer进行端到端语音翻译的的基本流程\n", "## 基础模型\n", "由于 ASR 章节已经介绍了 Transformer 以及语音特征抽取，在此便不做过多介绍，感兴趣的同学可以去相关章节进行了解。\n", "\n", "本小节，主要讨论利用 transformer（seq2seq）进行ST与ASR的异同。\n", "\n", "相似之处在于，两者都可以看做是从语音（speech）到文本（text）的任务。将语音作为输入，而将文字作为输出，区别只在于生成结果是对应语言的识别结果，还是另一语言的翻译结果。\n", "\n", "因此，我们只需要将数据中的目标文本替换为翻译文本（$Y$），便可利用 ASR 的模型结构实现语音翻译。\n", "\n", "规范化地讲，对于 ASR，利用包含语音（$S$）和转写文本（$X$）的数据集，训练得到一个模型 $M_{ASR}$，能对任意输入的源语言语音 $\\hat{S}$ 进行文字识别，输出结果 $\\hat{X}$。\n", "\n", "而ST的语料集，通常包含语音（$S$）、转写文本（$X$）以及翻译文本（$Y$），只需将ASR实践中的转写文本$X$替换为对应的翻译文本 $Y$，便可利用同样的流程得到一个翻译模型 $M_{ST}$，其能对任意输入的源语言语音 $\\hat{S}$ 进行翻译，输出结果 $\\hat{Y}$\n", "\n", "值得注意的是，相较于 ASR 任务而言，在 ST 中，因为翻译文本与源语音不存在单调对齐（monotonic aligned）的性质，因此 CTC 模块不能将翻译结果作为目标来使用，此处涉及一些学术细节，感兴趣的同学可以自行去了解 [CTC](https://github.com/PaddlePaddle/PaddleSpeech/blob/develop/docs/topic/ctc/ctc_loss.ipynb) 的具体内容。\n", "\n", "> 我们会在 [PaddleSpeech](https://github.com/PaddlePaddle/PaddleSpeech) 中放一些 Topic 的技术文章（如 [CTC](https://github.com/PaddlePaddle/PaddleSpeech/blob/develop/docs/topic/ctc/ctc_loss.ipynb) ），欢迎大家 star 关注。\n", "\n", "## 辅助任务训练，提升效果（ASR MTL）\n", "\n", "相比与 ASR 任务，ST 任务对于数据的标注和获取更加困难，通常很难获取大量的训练数据。\n", "\n", "因此，我们讲讨论如何更有效利用已有数据，提升 ST 模型的效果。\n", "\n", "1.先利用 ASR 对模型进行预训练，得到一个编码器能够有效的捕捉语音中的语义信息，在此基础上再进一步利用翻译数据训练ST模型。\n", "\n", "2.相较于 ASR 任务的二元组数据（$S$,$X$），通常包含三元组数据（$S$,$X$,$Y$）的ST任务能够自然有效的进行多任务学习。\n", "顾名思义，我们可以将ASR任务作为辅助任务，将两个任务进行联合训练，利用ASR任务的辅助提升 ST 模型的效果。\n", "具体上讲，如图所示，可以利用一个共享的编码器对语音进行编码，同时利用两个独立的解码器，分别执行 ASR 和 ST 任务。\n", "![mtl](https://ai-studio-static-online.cdn.bcebos.com/3896f104527947cbba8f459d4477b0a8cb9e4d527d6f4da29bb7792d69b66d3c)\n", "我们将实战中进行演示。\n", "\n", "## 引入预训练模型，提升效果 （FAT-ST PT）\n", "\n", "相比于文本到文本的机器翻译具有充足的语料（通常上百万条），语音到文本的翻译的语料很匮乏。那是否可以将文本到文本的翻译语料利用上来提升 ST 的模型效果呢？答案是肯定的。\n", "\n", "FAT 模型[1],借鉴了 Bert[2] 和 TLM[3]的 masked language model 预训练思路，并将其拓展到语音翻译的跨语言、跨模态（语音和文本）的场景。可以应对三元组（$S$,$X$,$Y$）中任意的单一或组合的数据类型。\n", "举例来说，它可以利用纯语音或文本数据集（$S$|$X$|$Y$），也可以利用 ASR 数据集（$S$,$Y$），甚至文本翻译数据（$X$,$Y$)。在这种预训练模型的基础上进行 ST 的训练，能够有效解决训练数据匮乏的困境，提升最终的翻译效果。\n", "![fat](https://ai-studio-static-online.cdn.bcebos.com/ef4b677e4cdf466fa5264d3b9dc976326e3c2046b1704c7da693015a71fc8a68)\n", "\n" ] }, { "cell_type": "markdown", "metadata": { "collapsed": false }, "source": [ "# 实战\n", "## ST 多任务学习，将 ASR 作为辅助任务\n", "### 数据集: [Ted语音翻译数据集](http://www.nlpr.ia.ac.cn/cip/dataset.htm)（英文语音$\\rightarrow$中文文本）[4]\n", "## 准备工作\n", "## 特征抽取\n", "参考语音识别的相关章节，略。\n", "## 多任务模型\n", "Transformer 内容参考语音识别的相关章节，略。\n" ] }, { "cell_type": "markdown", "metadata": { "collapsed": false }, "source": [ "## Stage 1 准备工作\n", "### 安装 paddlespeech" ] }, { "cell_type": "code", "execution_count": 1, "metadata": { "collapsed": false }, "outputs": [], "source": [ "!pip install -U pip\n", "!pip install paddlespeech" ] }, { "cell_type": "markdown", "metadata": { "collapsed": false }, "source": [ "### 导入 python 包" ] }, { "cell_type": "code", "execution_count": 2, "metadata": { "collapsed": false }, "outputs": [], "source": [ "import os\n", "import paddle\n", "import numpy as np\n", "import kaldiio\n", "import subprocess\n", "from kaldiio import WriteHelper\n", "from yacs.config import CfgNode\n", "import IPython.display as dp\n", "import warnings\n", "warnings.filterwarnings(\"ignore\")\n", "\n", "from paddlespeech.s2t.frontend.featurizer.text_featurizer import TextFeaturizer\n", "from paddlespeech.s2t.models.u2_st import U2STModel" ] }, { "cell_type": "markdown", "metadata": { "collapsed": false }, "source": [ "### 获取预训练模型和参数并配置" ] }, { "cell_type": "code", "execution_count": 3, "metadata": { "collapsed": false }, "outputs": [], "source": [ "!wget -nc https://paddlespeech.bj.bcebos.com/s2t/ted_en_zh/st1/fat_st_ted-en-zh.tar.gz\n", "!tar xzvf fat_st_ted-en-zh.tar.gz" ] }, { "cell_type": "code", "execution_count": 4, "metadata": { "collapsed": false }, "outputs": [], "source": [ "config_path = \"conf/transformer_mtl_noam.yaml\" \n", "\n", "# 读取 conf 文件并结构化\n", "st_config = CfgNode(new_allowed=True)\n", "st_config.merge_from_file(config_path)" ] }, { "cell_type": "markdown", "metadata": { "collapsed": false }, "source": [ "### 下载并配置 kaldi 环境" ] }, { "cell_type": "code", "execution_count": 6, "metadata": { "collapsed": false }, "outputs": [], "source": [ "!wget -nc https://paddlespeech.bj.bcebos.com/s2t/ted_en_zh/st1/kaldi_bins.tar.gz\n", "!tar xzvf kaldi_bins.tar.gz" ] }, { "cell_type": "code", "execution_count": 7, "metadata": { "collapsed": false }, "outputs": [], "source": [ "kaldi_bins_path = os.path.abspath('kaldi_bins')\n", "print(kaldi_bins_path)\n", "if 'LD_LIBRARY_PATH' not in os.environ:\n", " os.environ['LD_LIBRARY_PATH'] = f'{kaldi_bins_path}'\n", "else:\n", " os.environ['LD_LIBRARY_PATH'] += f':{kaldi_bins_path}'\n", "os.environ['PATH'] += f':{kaldi_bins_path}'" ] }, { "cell_type": "markdown", "metadata": { "collapsed": false }, "source": [ "## Stage 2 获取特征\n", "### 提取 kaldi 特征" ] }, { "cell_type": "code", "execution_count": 8, "metadata": { "collapsed": false }, "outputs": [], "source": [ "def get_kaldi_feat(wav_path, config=st_config):\n", " \"\"\"\n", " Input preprocess and return paddle.Tensor stored in self.input.\n", " Input content can be a file(wav).\n", " \"\"\"\n", " wav_file = os.path.abspath(wav_path)\n", " cmvn = config.collator.cmvn_path\n", " utt_name = '_tmp'\n", "\n", " # Get the object for feature extraction\n", " fbank_extract_command = [\n", " 'compute-fbank-feats', '--num-mel-bins=80', '--verbose=2',\n", " '--sample-frequency=16000', 'scp:-', 'ark:-'\n", " ]\n", " fbank_extract_process = subprocess.Popen(fbank_extract_command,\n", " stdin=subprocess.PIPE,\n", " stdout=subprocess.PIPE,\n", " stderr=subprocess.PIPE)\n", " fbank_extract_process.stdin.write(\n", " f'{utt_name} {wav_file}'.encode('utf8'))\n", " fbank_extract_process.stdin.close()\n", " fbank_feat = dict(kaldiio.load_ark(\n", " fbank_extract_process.stdout))[utt_name]\n", "\n", " extract_command = ['compute-kaldi-pitch-feats', 'scp:-', 'ark:-']\n", " pitch_extract_process = subprocess.Popen(extract_command,\n", " stdin=subprocess.PIPE,\n", " stdout=subprocess.PIPE,\n", " stderr=subprocess.PIPE)\n", " pitch_extract_process.stdin.write(\n", " f'{utt_name} {wav_file}'.encode('utf8'))\n", " process_command = ['process-kaldi-pitch-feats', 'ark:', 'ark:-']\n", " pitch_process = subprocess.Popen(process_command,\n", " stdin=pitch_extract_process.stdout,\n", " stdout=subprocess.PIPE,\n", " stderr=subprocess.PIPE)\n", " pitch_extract_process.stdin.close()\n", " pitch_feat = dict(kaldiio.load_ark(\n", " pitch_process.stdout))[utt_name]\n", " concated_feat = np.concatenate((fbank_feat, pitch_feat), axis=1)\n", " raw_feat = f\"{utt_name}.raw\"\n", " with WriteHelper(f'ark,scp:{raw_feat}.ark,{raw_feat}.scp') as writer:\n", " writer(utt_name, concated_feat)\n", " cmvn_command = [\n", " \"apply-cmvn\", \"--norm-vars=true\", cmvn, f'scp:{raw_feat}.scp',\n", " 'ark:-'\n", " ]\n", " cmvn_process = subprocess.Popen(cmvn_command,\n", " stdout=subprocess.PIPE,\n", " stderr=subprocess.PIPE)\n", " process_command = ['copy-feats', '--compress=true', 'ark:-', 'ark:-']\n", " process = subprocess.Popen(process_command,\n", " stdin=cmvn_process.stdout,\n", " stdout=subprocess.PIPE,\n", " stderr=subprocess.PIPE)\n", " norm_feat = dict(kaldiio.load_ark(process.stdout))[utt_name]\n", " audio = paddle.to_tensor(norm_feat).unsqueeze(0)\n", " audio_len = paddle.to_tensor(audio.shape[1], dtype='int64')\n", " return audio, audio_len" ] }, { "cell_type": "markdown", "metadata": { "collapsed": false }, "source": [ "### 构建文本特征提取对象" ] }, { "cell_type": "code", "execution_count": 9, "metadata": { "collapsed": false }, "outputs": [], "source": [ "text_feature = TextFeaturizer(\n", " unit_type=st_config.collator.unit_type,\n", " vocab=st_config.collator.vocab_filepath,\n", " spm_model_prefix=st_config.collator.spm_model_prefix)" ] }, { "cell_type": "markdown", "metadata": { "collapsed": false }, "source": [ "## Stage 3 使用模型获得结果" ] }, { "cell_type": "markdown", "metadata": { "collapsed": false }, "source": [ "### 构建 ST 模型" ] }, { "cell_type": "code", "execution_count": 10, "metadata": { "collapsed": false }, "outputs": [], "source": [ "model_conf = st_config.model\n", "model_conf.input_dim = st_config.collator.feat_dim\n", "model_conf.output_dim = text_feature.vocab_size\n", "print(model_conf)\n", "model = U2STModel.from_config(model_conf)" ] }, { "cell_type": "markdown", "metadata": { "collapsed": false }, "source": [ "### 加载预训练模型" ] }, { "cell_type": "code", "execution_count": 11, "metadata": { "collapsed": false }, "outputs": [], "source": [ "params_path = \"exp/transformer_mtl_noam/checkpoints/fat_st_ted-en-zh.pdparams\"\n", "model_dict = paddle.load(params_path)\n", "model.set_state_dict(model_dict)\n", "model.eval()" ] }, { "cell_type": "markdown", "metadata": { "collapsed": false }, "source": [ "### 预测" ] }, { "cell_type": "code", "execution_count": 14, "metadata": { "collapsed": false }, "outputs": [], "source": [ "# 下载wav\n", "!wget -nc https://paddlespeech.bj.bcebos.com/PaddleAudio/74109_0147917-0156334.wav\n", "!wget -nc https://paddlespeech.bj.bcebos.com/PaddleAudio/120221_0278694-0283831.wav\n", "!wget -nc https://paddlespeech.bj.bcebos.com/PaddleAudio/15427_0822000-0833000.wav\n", "\n", "wav_file = '74109_0147917-0156334.wav'\n", "# wav_file = '120221_0278694-0283831.wav'\n", "# wav_file = '15427_0822000-0833000.wav'\n", "\n", "transcript = \"my hair is short like a boy 's and i wear boy 's clothes but i 'm a girl and you know how sometimes you like to wear a pink dress and sometimes you like to wear your comfy jammies\"\n", "dp.Audio(wav_file)" ] }, { "cell_type": "code", "execution_count": 15, "metadata": { "collapsed": false }, "outputs": [], "source": [ "audio, audio_len = get_kaldi_feat(wav_file)\n", "cfg = st_config.decoding\n", "\n", "\n", "res = model.decode(audio,\n", " audio_len,\n", " text_feature=text_feature,\n", " decoding_method=cfg.decoding_method,\n", " beam_size=cfg.beam_size,\n", " word_reward=cfg.word_reward,\n", " decoding_chunk_size=cfg.decoding_chunk_size,\n", " num_decoding_left_chunks=cfg.num_decoding_left_chunks,\n", " simulate_streaming=cfg.simulate_streaming)\n", "print(\"对应英文: {}\".format(transcript))\n", "print(\"翻译结果: {}\".format(\"\".join(res[0].split())))\n" ] }, { "cell_type": "markdown", "metadata": { "collapsed": false }, "source": [ "# 参考文献\n", "\n", "1.Zheng, Renjie, Junkun Chen, Mingbo Ma, and Liang Huang. \"Fused acoustic and text encoding for multimodal bilingual pretraining and speech translation.\" ICML 2021.\n", "\n", "2.Devlin, Jacob, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. \"Bert: Pre-training of deep bidirectional transformers for language understanding.\" NAACL 2019.\n", "\n", "3.Conneau, Alexis, and Guillaume Lample. \"Cross-lingual language model pretraining.\" NIPS 2019.\n", "\n", "4.Liu, Yuchen, Hao Xiong, Zhongjun He, Jiajun Zhang, Hua Wu, Haifeng Wang, and Chengqing Zong. \"End-to-end speech translation with knowledge distillation.\" Interspeech 2019." ] }, { "cell_type": "markdown", "metadata": { "collapsed": false }, "source": [ "# PaddleSpeech\n", "\n", "请关注我们的 [Github Repo](https://github.com/PaddlePaddle/PaddleSpeech/)，非常欢迎加入以下微信群参与讨论：\n", "- 扫描二维码\n", "- 添加运营小姐姐微信\n", "- 通过后回复【语音】\n", "- 系统自动邀请加入技术群\n", "\n", "
" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "py35-paddle1.2.0" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.7.4" } }, "nbformat": 4, "nbformat_minor": 1 }