PaddleSpeech/docs/tutorial/tts/tts_tutorial.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a href=\"https://github.com/PaddlePaddle/PaddleSpeech\"><img style=\"position: absolute; z-index: 999; top: 0; right: 0; border: 0; width: 128px; height: 128px;\" src=\"https://nosir.github.io/cleave.js/images/right-graphite@2x.png\" alt=\"Fork me on GitHub\"></a>\n",
    "\n",
    "# 『听』和『说』\n",
    "人类通过听觉获取的信息大约占所有感知信息的 20% ~ 30%。声音存储了丰富的语义以及时序信息，由专门负责听觉的器官接收信号，产生一系列连锁刺激后，在人类大脑的皮层听区进行处理分析，获取语义和知识。近年来，随着深度学习算法上的进步以及不断丰厚的硬件资源条件，**文本转语音（Text-to-Speech, TTS）** 技术在移动、虚拟娱乐等领域得到了广泛的应用。</font>\n",
    "## \"听\"书\n",
    "使用 [PaddleOCR](https://github.com/PaddlePaddle/PaddleOCR) 直接获取书籍上的文字。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--2021-12-03 12:10:59--  https://paddlespeech.bj.bcebos.com/tutorial/tts/ocr_result.jpg\n",
      "Connecting to 172.19.56.199:3128... connected.\n",
      "Proxy request sent, awaiting response... 200 OK\n",
      "Length: 110119 (108K) [image/jpeg]\n",
      "Saving to: 'download/ocr_result.jpg'\n",
      "\n",
      "ocr_result.jpg      100%[===================>] 107.54K  45.1KB/s    in 2.4s    \n",
      "\n",
      "2021-12-03 12:11:04 (45.1 KB/s) - 'download/ocr_result.jpg' saved [110119/110119]\n",
      "\n",
      "--2021-12-03 12:11:04--  https://paddlespeech.bj.bcebos.com/tutorial/tts/ocr.wav\n",
      "Connecting to 172.19.56.199:3128... connected.\n",
      "Proxy request sent, awaiting response... 200 OK\n",
      "Length: 360044 (352K) [audio/wav]\n",
      "Saving to: 'download/ocr.wav'\n",
      "\n",
      "ocr.wav             100%[===================>] 351.61K  85.2KB/s    in 4.1s    \n",
      "\n",
      "2021-12-03 12:11:10 (85.2 KB/s) - 'download/ocr.wav' saved [360044/360044]\n",
      "\n",
      "--2021-12-03 12:11:11--  https://paddlespeech.bj.bcebos.com/tutorial/tts/tts_lips.mp4\n",
      "Connecting to 172.19.56.199:3128... connected.\n",
      "Proxy request sent, awaiting response... 200 OK\n",
      "Length: 188071 (184K) [video/mp4]\n",
      "Saving to: 'download/tts_lips.mp4'\n",
      "\n",
      "tts_lips.mp4        100%[===================>] 183.66K  65.2KB/s    in 2.8s    \n",
      "\n",
      "2021-12-03 12:11:16 (65.2 KB/s) - 'download/tts_lips.mp4' saved [188071/188071]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# download demo sources\n",
    "!mkdir download\n",
    "!wget -P download https://paddlespeech.bj.bcebos.com/tutorial/tts/ocr_result.jpg\n",
    "!wget -P download https://paddlespeech.bj.bcebos.com/tutorial/tts/ocr.wav\n",
    "!wget -P download https://paddlespeech.bj.bcebos.com/tutorial/tts/tts_lips.mp4"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAABLAAAAHOCAIAAAAHU2eOAAEAAElEQVR4nOz9ebxlWVkfjH+fZ621hzPdqerW3NVz0wzdgIAaHPMmzrMiIhJfNQ44jyRGowlRE5VoRMUQX3EKKEZBVAxiHFFQFGiapumpurrmqjufaU9rref5/bHPuXWruhuFwPv+ovf7OZ+qc8/Zw9prr73O+j7D9yE/ep/38Wu+9hu2tqZv+ZM/92WwaZ/YAQwAChAAQBQASEAGHxHoE3/svTpHIIgIKDARIABCDETEbAkMWAAqUAV/qM1RAFCFCAAQPfERVLU9L+FxX9PuO9nz7+71fLAGqUgIgZmNtbN2EH2Q7Z/oEDxvhgARCFABqVQVk4F14AQwu80IhOtOwHveX/NVewU6BjPA0Pbf+Qgggkj0jYgQqbFEDBBF7uOJbmaQYIiZmAECJIIAYniqq6p2NnfWRYEBmAWoA6aMwJB2qCkIaomYZQyuFXVEEtAL6Cg8sJVjFDBNpMd0AOWO7rw96J+7zmnUdyHrIu3WNi+NU3aAd7HsxIrH2ygqiEWp2K6G21VTa16cKetQRxaTBqSTmsZVKBstvaysHFxY6DPDsObOijZ1WYx5JXhf17Wvyqaqo69j8BJ8ltiymPTyfHFhQYJnyPLy8uLi4rhPxlhyCcg0UZrax8ZL0Gra/Mkf/8XSyslbnvrcu5/3aZKsrN58V1UFl/UAZjAAmnWrB5XQKaSIfuzrMsZoTZJ2enAZQgeWABIJjSixc9xTJIBtB7kx7R2vymoECp3Lf1bunCv9lXxASZLUI4QiSbUXxqM0qc+euzdJcfbCxnjaD/HwZ3/nD6F7BGQBpzQb6AYeGMXpeYMxUE6uXLI2yTqLGvknfvKnv+3lv+sjkgxKGI2L2Iwunn3o6bcf37p0+uH33dPLs9Smt93xdD5wBMEiyf/kz1537OiNJ0/ekfYPQfMYDIjZAoBAgMiz0zKBAEbNMBF6SpoH6uKcxsYlSy45CrMsriPWAawgwDEyQhphGGAVmg1TBljBSkIUCCUwRlhHuYl6A6GEPBI9QuOI+km6BNeFMQqlbgom2AymByyClqBdIJWyTwyy7VFHiomiIgSCAYyCoQmJARnMHoXXA17iNErFFIzxoADEuLVlsg5GZdwpHnvw7HJ3SRvxdWimIDJVUe9sDZsmpFmWZJkne+Ntd+bLR/Kjt+Lg7bCrkEVoH4Yio525PBpCmaIx2PbFBRd2UG6gGUEjiKAEGJAJvYGxKdkebB/og3qgDtABpyC3+zQrgiAqotEKmmtMWBvRC7F5KFZnUUwyzQADjuDSW9J02aY3c3oEoQvOYBdq6kcYAlJ4DluQjft+7We5uNzNdLC0IFnWO3gUSVb1ummaO5uRJKpERKpRYuESQqxCU8TQIIKUHaVEvB2aLMvyLIEBtBFfxxgEmiSJkiVOYBLiTMmCHMDqltgUGjcoC6j9//r9dz7v475wsPQMZEdBiARFIEQDA7XQa6fLfexjH/vYxz7+z4eqEhEAEWHmGKNzzqpqCOHWW2/94z/+SzQNwDFGy+7vPNxHCdbuMhhhYkCjxLIssyzbs5UADHpSVvl3ggjGXCU7j4M86aGflL592K2RD3H7+QqlbbwqVAFVVYVCBBoJBDYzHvfEOz+OCqqiHSIqAMCtOaDlzQQiEIHZJImJMUavqgQCkUYFQERCINrtAUmZRUW8F4CUiYiJoBxDHZqaRBM7SHhmbAihJo5KQSgylMEAg9pLqyFKlDA5RjAYK7yiVhiDlFXgL6O86MMGcQJZhevAZjApwxgYAUEVIRbjcS8KGt9sbfgCFrlaLouqQR6cRuc46bh04VB3+Zal1e5gpRGKMVZV1VQTgqSJMdAQm+rCuNMzSZIY4hiacjIeD0dVMRkNd4Ka0keeVpaRpulwWq1tnVm67VCSZalx5DgBk0UEwcCXzeXLl7uDQ8xcVtN+7xAg/PdafQogIJmNUut8PRVomnYyZsAolAAvykTOAEAMTQgFIRiDyZVLXqu0k+ZdBzBbcWme2r40tWjt0swmsrKywpw89tgWqil61wyT3WHjfdPUk8zU1jFBm3KaZP077rjNWpDB5laxvNIZ9Dvf+s3f8aqf+QktN72Pa2sb08x9/PM+tqqKjjPb586++X/+wWe+8J92O4O0250Nyz1j6INDVUEf6rNzze4KIRLWABFogKpKoBg1QMQYI4C0nI7abp/ZwmZ/gxjCnLU9ExQNhEhzpg5ms60SRVANMwGmQKVoIEORoOKBBtpEaSANB2+SRIYT1FpMm8Tlly6u+8KPdsbL/SMxal3Wk0kdGl9MPcw0gDZHhe0uLh5bO3FrdeDGu9HPwCk0abxaxwKoBCVRUoBdlmEKBQNMpCCCAiCw0fZamGYfEqAM4vlUwbsTVDvBEdFsQgARiJmFCEQqSlCIKKmIqkBnU9Pjer4dwOPRZDKxVWFAZmrrqgmuYzs91+/RtYOgbYGEAPUgISKQEEhEiJBlmbUWUESJ0ccQACUmlWtMc0SzNld1bW0DDa7x0Xvvvari7/fg7WMf+9jHPvbxDxXWGBNjPHny5HT61qZpjOvQE/46Ej3hr/uHjychUK3DbE5eFYCIhBA+UqeNEdwueK6eC6rCs6sWzNyDbWvkSTx+8rg3j//qCSGAqM4sz6pC9L+xFtFZDyEqKQSRQrvIEyASGQVgBNeauRVKiqueSVURERFVJVGiyEqkEapQankiEc3WTMStc0VVWl8lUU1ETNf4WRVKIENgEhExxCBSCepjlCrUlYmRkwzGAQLEUBdJSlEqaIAh2ARgaEAETAVJoSkTW6oMlRE1g404EkbYkvJsUz+quplSB3EA1wWn0BRqjBIQET3KOodBVIgC7GNT+MJHK6ndrHtZr9dbXM67y3A9Srtu8RAWVxI122tr68UVL5mzlJBxRm3aXzjUBQCNqsrRZTZN+wsQXbtyqS6m5bTYHO3UZTXo93q9XtNE2trKuz0VSjMTgRgCokCo1+muXcKhw+Nulopvunmm1STJBhGztfhsTTwjYW3nCxHRzGfe+ocFgEtzIKqG2vumidZ288wkzCLUNB7q08xYa4G0nG73Fgd1HSLVddEE0WIK9paTZlo3sZ74IONiGDxFr5PhJrQBGCDQVT9426IkSao6QjXrdLTxRVUndvE5z3k2AMNYWeko8PXf8E0/+RM/ATCZ5OiREydvvmmytXHm7Ok7n/aMrYffd+r0+c/9/E8bLA8IFjFCIxhs5p68q7j2ASHsTkTtc8rtI/AhgURbPzR0ZlJRVRFV1Tj7XzWCFCRgASlIZz7uWYSCUVjAEp9RkMKSWGhK2oWmIJAHTA07BsbAFcFFjzXBxDYTQIiFOZIGCXWoK/Uh7y7sbA+t2OmwSly3LIbacFXQ2eF6XfnYBIZhIokxahBD1dZo4i+Yxy6ub02fGnD8ZkLPA87xAVIBKbFVhIhKtLFxZsABkSEGIEwgQ8YosbKhdtS1oQFXZ6Trp6aZiWk2CGn2riWEqoSZZWrXbqCqRAZk9h6KFBBFkiTOOHX9Xtbp9Ric53nW7zcwpExk2pvM2nrpKPoa1N4bIUA1RhBFSdMukQIaQ/DBq0RmYuLZ2ef/7p6bWJlJhaAqPoQQIE9ACJ/UVLiPfexjH/vYxz9EWCKqqmo8HjdNU5ZlCpc6an2I1287M7J+5NAeTJ/ow5kpWgEw2zRNiYgeF636YZApYiiAPaGaNFuCztaUiog5KYUCiI87RBtK+viuaHd7krVpexqVOQmcr/efbPu/GwKNkAAVaASElBVRJRjDNF/T0DVrZZ21UvVq34kgRoi0jSOjUGVVQEUV2i7+mKQ9oyoETAQWVRJhNASar/mojTIlIY2RjCFWQ4B4qIhvmqaxVp1GRwKaNRvqLZRjkNiINCTGEMCESIgKIwBDDCBsKtDIwEO68AkkoF6vqvujnMkTx+YY4gqSEjEBDISI1MQAX6OoVIqdtUta1ZlNSx/OXlr34pYOHIpLJ2gwsMur3F1ooq0ajErNUHf7/XVvtiQxmcsslTFwDI7N4mJe13U1LYpiIjEmxuZZL0vTVZcHX/u
      "text/plain": [
       "<PIL.JpegImagePlugin.JpegImageFile image mode=RGB size=1200x462 at 0x7FA3E4159A90>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import IPython.display as dp\n",
    "from PIL import Image\n",
    "img_path = 'download/ocr_result.jpg'\n",
    "im = Image.open(img_path)\n",
    "dp.display(im)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "使用 [PaddleSpeech](https://github.com/PaddlePaddle/PaddleSpeech)，阅读上一步识别出来的文字。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "                <audio controls=\"controls\" >\n",
       "                    <source src=\"data:audio/x-wav;base64,UklGRmR+BQBXQVZFZm10IBAAAAABAAEAwF0AAIC7AAACABAAZGF0YUB+BQDj/9//3//h/+D/5P/k/+T/5v/m/+b/5//r/+z/6//r/+r/7P/s/+7/7//v/+//7//v//H/8f/x//D/8P/x//P/9f/1//T/9P/2//f/9//3//j/9//3//n/+P/4//j/+P/6//r/+v/6//v/+v/6//n/+v/6//r/+//6//n/+P/6//3//P/7//v//P/8//v//P/7//r/+P/5//r/+f/6//r/+f/7//v/+v/7//z/+v/7//v//P/7//v/+//8//v/+//8//z//P/9//z//P/9//3//f/8//7//v/+//7/AAD//wAAAAABAAAAAQABAAIAAgAAAAIAAQACAAEAAgABAAEAAgABAAEAAQACAAIAAgABAAEAAQAAAAEAAAABAAAAAAD//wAA//8AAAAA//8AAP//AAD/////AAD///7////+//3//f/8//3//f/9//3//v/9//3//f/9//3//v/8//3//f/9//7//f/9//z//f/8//3//P/9//z//P/8//z//P/7//v/+v/6//r/+v/6//r/+v/6//n/+v/5//n/+f/5//j/+f/5//n/+f/5//n/+f/5//n/+f/5//n/+f/5//n/+f/5//n/+v/5//r/+v/6//r/+v/6//r/+//7//v/+//6//v/+v/7//v/+v/7//v/+v/7//v/+//7//r/+//7//3//f/+///////+/////v/+//7//v///////v///////////wAAAAD//wAAAAD//wAAAAAAAAAAAAABAAAAAAAAAAAAAAABAAAAAAAAAP//AAAAAAEAAAABAAAAAQAAAAAAAAAAAAAAAAABAAAAAQABAAEAAAABAAAA//8AAAAA//8AAAAAAQAAAAEA//8BAAAA//8AAAAAAgAAAAEAAAABAAEAAAABAAEAAQABAAAAAQABAAEAAQACAAIAAQABAAEAAQABAAIAAQABAAIAAQABAAAAAQACAAIAAgACAAEAAQABAAEAAgACAAMAAgACAAMAAgACAAIAAgACAAMABAACAAIAAgACAAIAAwADAAIAAwADAAIAAwADAAMAAwADAAIAAgADAAIAAwADAAMAAwAEAAQABAADAAQABAADAAMAAwADAAMAAQACAAMAAQACAAIAAgABAAIAAgABAAIAAQACAAIAAwACAAMAAgADAAMAAwACAAIAAgACAAMAAwADAAMAAwADAAQABAADAAMAAwACAAIAAgACAAIAAQACAAIAAQABAAEAAgABAAEAAgABAAIAAgABAAEAAAAAAAEAAgACAAMAAwADAAMAAgACAAIAAgACAAEAAgAEAAQAAgAEAAMAAwACAAEAAQABAAIAAgADAAIAAwABAAIAAwAEAAMAAwAEAAMAAwADAAQAAwAFAAMAAwACAAIAAQABAAEAAAACAAEAAQABAAEAAAABAAEAAAAAAAAAAAAAAAEAAQAAAP7//v///wEAAAACAAIAAAAAAP///////wAAAQAAAAEAAQAAAAAAAAAAAP/////+//7////+//7////+//3//f/9//7//f/9//7//v/8//3//v/9//z//f/8//z//f/9//7//f/9//3//v/9/////v/+//7//v/+//7//v/9//7//v////7////9/////v/9/////v/+//3//f/9//z//f/8//7//f/9//7//P/8//z//P/8//z//P/8//3//P/8/////v/+//7//v/+//3//P/+/////////wAA//////7//f/+//7//v/+/////v/+//7//v/+//7//v/9//3//v/+//7//v/+//3//f/9//3//v////z//f/9//z/+//+//z/+//7//r//P/8//z//f/8//v/+//7//v//P/7//v/+//7//v//P/8//z/+//7//r/+v/6//z//P/9//3/+//7//r/+//8//z/+//6//z/+v/8//v/+//6//r/+f/6//r/+v/5//n/+//7//v/+v/7//n/+P/4//j/+f/3//f/+P/5//v/+v/7//r/+v/6//r/+v/6//v//P/9//7//f////7//f/9//7//v/+/////v/9//////////7/AQABAAAA//////3//v/+//7//v/+/////v/////////////////+//3//v/+//3//////wAAAQADAAIAAAABAP///v8AAAIABAADAAMAAAAAAP//AAABAAIAAgABAAEAAAABAP/////+//7//////wAAAQAAAP///f/9//7//////wAA/////////v/+//3//f/9//7/AAABAAAAAAD+//3///8BAAEAAQACAAEAAAABAAAAAQAAAP/////+//7///8BAP//AQABAAIAAQABAAAAAAD/////AAD///7/////////AQACAAIAAAD///7//v///wAAAwADAAIAAgABAAAAAAABAAAA///9////AAABAAEAAgABAAAAAQABAAAAAQD///7///8AAAIAAwADAAIAAQABAAAAAAAAAAAAAQABAAQAAQACAP///////wAAAAABAAIABAAEAAQAAwADAAIAAQAFAAUABQADAAEAAQACAAMABgAFAAMAAwADAAQABQAGAAQABAADAAMABQAIAAgACQAIAAUABAACAAIAAwAEAAYABwAIAAkACgAJAAcAAwAAAP//AgAHAAoACgAKAAgABgAFAAQAAwABAAUACwANAAsACAAFAAUABgAHAAYAAgD7//z/AgAGAAcACwAJAP//+//9/wIABwALAAoACwAHAAEAAQAEAAYACAAJAAkACgALAA0ADAALAAsABgD/////BQAOABgAGQAUAAsAAAD6//z/AgAKABAAFAASABEACwADAPz//f/+//v/AgAOABAADgAOAAsABgAFAAcABgAMABAADQAHAAQAAwAFAA4ADgABAAAA/v/1//b///8VACIAKAAkABsACwD1/+z/6P/h/+f/CQAlADwAQwAyAAgA3v/R/+n/FAAmACwAJQALAAAAAgASABYAFAAIAP//AgAIAA4AFgARAAYABwAQAAoAAwALABcAKwAmABAA+//w/+H/8f8TADkAUgBUADsAAwDM/7v/0P/u/wIAJABbAGcARgAVAOj/v/+q/8D/+f9OAHIAXQBHABgA1v+w/8T/2f/z/zEAYwBkADwA9/+3/5v/tv/0/yoAdwCDADEA/f8FAPr/nf92/+L/IwAvAEAAVAAsAPD/4f/2/+//3P/l//z/BgDz/yMAXQBJAAMAzf+d/6P/7v8+AFYAMQAPAPv/2f+k/7X/9v8fAB8ARwBnABsA+v8EAA8AAgDu//b/CQDy/7b/lf/O/x4AQQBSAFMAUADn/5b/v/8IAPX/zv/k/yUAOAAmACsAHAADAAwAVQAtAH7/V/+4//f/NACTALQAhQAhAND/pf+Z//v/SgAIAPD/MABEABEAEQA9AB8A2//N/woAMQAoADcANQAKAB0AIADi/47/bv/M/20A/wDwAGUAuf8A/+v+zf8jAcMBVQEAAF3+9f3i/l8AlAHKAfQA3P9C/2z/4/8BAMD/of8LAKIAFQHVANT/Dv8w/+T/eQC4ANwAtwDO/9n+5P7c/60A0wDxALwAHgC//8f/XwAzAEL/PP+m/6//ZQAxASABuADQ/+f++P7e/90ABQF4AHz/D//Q/2MAswCqABwAg/9D/7L/4wAxAYAAGgDE/0H/Qv/0/2wAjgD7AO0A4/8X/1L/UwDAADcAqf/O/y4AGQBFAOYAbgCI/7j//v+R/2L/owAtAQYAwf84AFwA6v9g/zb/4f80AXYBOwBJ//f+Sf89ANYAjgEgAXf/a/4G/9z/SAAHAU4BwwC//yj/c/9kAOMA3P/N/h7/YgCdAZcBJwHIAOH/Bv9S/sv9x/7fAKAC9QKTAZf/ef5+/vb+kP8OAMUAjwFqAbYA+v/E//r/2f9b/+3+2f9cAUABFQASAA4APv9F/1oATgGHA
       "                    Your browser does not support the audio element.\n",
       "                </audio>\n",
       "              "
      ],
      "text/plain": [
       "<IPython.lib.display.Audio object>"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dp.Audio(\"download/ocr.wav\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "具体实现代码详见 [Story Talker](https://github.com/PaddlePaddle/PaddleSpeech/tree/develop/demos/story_talker)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 偶像开口说话\n",
    "*元宇宙来袭，构造你的虚拟人！* 看看 [PaddleGAN](https://github.com/PaddlePaddle/PaddleGAN) 怎样合成唇形，让WiFi之母——海蒂·拉玛说话。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "<video controls width=\"600\" height=\"360\" src=\"download/tts_lips.mp4\">animation</video>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from IPython.display import HTML\n",
    "html_str = '''\n",
    "<video controls width=\"600\" height=\"360\" src=\"{}\">animation</video>\n",
    "'''.format(\"download/tts_lips.mp4\")\n",
    "dp.display(HTML(html_str))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "具体实现代码请参考 [Metaverse](https://github.com/PaddlePaddle/PaddleSpeech/tree/develop/demos/metaverse)。\n",
    "\n",
    "下面让我们来系统地学习语音方面的知识，看看怎样使用 **PaddleSpeech** 实现基本的语音功能，以及怎样结合光学字符识别（Optical Character Recognition，OCR）、自然语言处理（Natural Language Processing，NLP）等技术“听”书、让名人开口说话。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 前言\n",
    "## 背景知识\n",
    "为了更好地了解文本转语音任务的要素，我们先简要地回顾一下文本转语音的发展历史。如果你对此已经有所了解，或希望能尽快使用代码实现，请直接跳至[实践](#实践)。\n",
    "### 定义\n",
    "<!----\n",
    "Note: \n",
    "1.此句抄自 [李沐Dive into Dive Learning](https://zh-v2.d2l.ai/chapter_introduction/index.html)\n",
    "2.修改参考A survey on Neural Speech Sysnthesis.\n",
    "---> \n",
    "文本转语音，又称语音合成（Speech Sysnthesis），指的是将一段文本按照一定需求转化成对应的音频，这种特性决定了的输出数据比输入输入长得多。文本转语音是一项包含了语义学、声学、数字信号处理以及机器学习的等多项学科的交叉任务。虽然辨识低质量音频文件的内容对人类来说很容易，但这对计算机来说并非易事。\n",
    "\n",
    "按照不同的应用需求，更广义的语音合成研究包括：*语音转换*，例如说话人转换、语音到歌唱转换、语音情感转换、口音转换等；*歌唱合成*，例如歌词到歌唱转换、可视语音合成等。\n",
    "\n",
    "### 发展历史\n",
    "\n",
    "<!--\n",
    "以下摘自维基百科 https://en.wikipedia.org/wiki/Speech_synthesis\n",
    "--->\n",
    "\n",
    "在第二次工业革命之前，语音的合成主要以机械式的音素合成为主。1779年，德裔丹麦科学家 Christian Gottlieb Kratzenstein 建造了人类的声道模型，使其可以产生五个长元音。1791年， Wolfgang von Kempelen 添加了唇和舌的模型，使其能够发出辅音和元音。贝尔实验室于20世纪30年代发明了声码器（Vocoder），将语音自动分解为音调和共振，此项技术由 Homer Dudley 改进为键盘式合成器并于 1939年纽约世界博览会展出。\n",
    "\n",
    "第一台基于计算机的语音合成系统起源于20世纪50年代。1961年，IBM 的 John Larry Kelly，以及 Louis Gerstman 使用 IBM 704 计算机合成语音，成为贝尔实验室最著名的成就之一。1975年，第一代语音合成系统之一 —— MUSA（MUltichannel Speaking Automation）问世，其由一个独立的硬件和配套的软件组成。1978年发行的第二个版本也可以进行无伴奏演唱。90 年代的主流是采用 MIT 和贝尔实验室的系统，并结合自然语言处理模型。\n",
    "\n",
    "<center><img src=\"https://ai-studio-static-online.cdn.bcebos.com/55035de353b042cd8c4468819b2d36e2fcc89bffdf2b442fa4c7b0b5499e1592\"></center>\n",
    "\n",
    "### 主流方法\n",
    "\n",
    "当前的主流方法分为**基于统计参数的语音合成**、**波形拼接语音合成**、**混合方法**以及**端到端神经网络语音合成**。基于参数的语音合成包含隐马尔可夫模型（Hidden Markov Model,HMM）以及深度学习网络（Deep Neural Network，DNN）。端到端的方法保函声学模型+声码器以及“完全”端到端方法。\n",
    "\n",
    "\n",
    "## 基于深度学习的语音合成技术\n",
    "\n",
    "### 语音合成基本知识\n",
    "\n",
    "<center><img src=\"https://ai-studio-static-online.cdn.bcebos.com/67900521529943b199d480386ca18138980bc9f68bd54cadbbaef84836c03b09\"></center>\n",
    "\n",
    "\n",
    "语音合成流水线包含 <font color=\"#ff0000\">**文本前端（Text Frontend）**</font> 、<font color=\"#ff0000\">**声学模型（Acoustic Model）**</font> 和 <font color=\"#ff0000\">**声码器（Vocoder）**</font> 三个主要模块:\n",
    "- 通过文本前端模块将原始文本转换为字符/音素。\n",
    "- 通过声学模型将字符/音素转换为声学特征，如线性频谱图、mel 频谱图、LPC 特征等。\n",
    "- 通过声码器将声学特征转换为波形。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 实践"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 安装 paddlespeech"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[33mWARNING: Ignoring invalid distribution -addlepaddle-gpu (/home/yuantian01/yt_py37/lib/python3.7/site-packages)\u001b[0m\n",
      "\u001b[33mWARNING: Ignoring invalid distribution -addlepaddle-gpu (/home/yuantian01/yt_py37/lib/python3.7/site-packages)\u001b[0m\n",
      "Looking in indexes: http://mirrors.aliyun.com/pypi/simple/\n",
      "Requirement already satisfied: pip in /home/yuantian01/yt_py37/lib/python3.7/site-packages (21.1.3)\n",
      "Collecting pip\n",
      "  Downloading http://mirrors.aliyun.com/pypi/packages/a4/6d/6463d49a933f547439d6b5b98b46af8742cc03ae83543e4d7688c2420f8b/pip-21.3.1-py3-none-any.whl (1.7 MB)\n",
      "\u001b[K     |████████████████████████████████| 1.7 MB 227 kB/s eta 0:00:01     |█████████████▌                  | 727 kB 227 kB/s eta 0:00:05\n",
      "\u001b[?25h\u001b[33mWARNING: Ignoring invalid distribution -addlepaddle-gpu (/home/yuantian01/yt_py37/lib/python3.7/site-packages)\u001b[0m\n",
      "Installing collected packages: pip\n",
      "  Attempting uninstall: pip\n",
      "\u001b[33m    WARNING: Ignoring invalid distribution -addlepaddle-gpu (/home/yuantian01/yt_py37/lib/python3.7/site-packages)\u001b[0m\n",
      "    Found existing installation: pip 21.1.3\n",
      "    Not uninstalling pip at /home/yuantian01/yt_py37/lib/python3.7/site-packages, outside environment /home/yuantian01/yt_py37_develop\n",
      "    Can't uninstall 'pip'. No files were found to uninstall.\n",
      "Successfully installed pip-21.3.1\n",
      "\u001b[33mWARNING: Ignoring invalid distribution -addlepaddle-gpu (/home/yuantian01/yt_py37/lib/python3.7/site-packages)\u001b[0m\n",
      "\u001b[33mWARNING: Ignoring invalid distribution -addlepaddle-gpu (/home/yuantian01/yt_py37/lib/python3.7/site-packages)\u001b[0m\n",
      "\u001b[33mWARNING: Ignoring invalid distribution -addlepaddle-gpu (/home/yuantian01/yt_py37/lib/python3.7/site-packages)\u001b[0m\n",
      "\u001b[33mWARNING: You are using pip version 21.1.3; however, version 21.3.1 is available.\n",
      "You should consider upgrading via the '/home/yuantian01/yt_py37_develop/bin/python -m pip install --upgrade pip' command.\u001b[0m\n",
      "\u001b[33mWARNING: Ignoring invalid distribution -addlepaddle-gpu (/home/yuantian01/yt_py37/lib/python3.7/site-packages)\u001b[0m\n",
      "\u001b[33mWARNING: Ignoring invalid distribution -addlepaddle-gpu (/home/yuantian01/yt_py37/lib/python3.7/site-packages)\u001b[0m\n",
      "Looking in indexes: http://mirrors.aliyun.com/pypi/simple/\n",
      "Requirement already satisfied: paddlespeech in /home/yuantian01/yt_py37_develop/lib/python3.7/site-packages (0.1.0a4)\n",
      "Requirement already satisfied: g2pM in /home/yuantian01/yt_py37/lib/python3.7/site-packages (from paddlespeech) (0.1.2.5)\n",
      "Requirement already satisfied: editdistance in /home/yuantian01/yt_py37/lib/python3.7/site-packages (from paddlespeech) (0.5.2)\n",
      "Requirement already satisfied: h5py in /home/yuantian01/yt_py37/lib/python3.7/site-packages (from paddlespeech) (3.4.0)\n",
      "Requirement already satisfied: scipy in /home/yuantian01/yt_py37/lib/python3.7/site-packages (from paddlespeech) (1.7.2)\n",
      "Requirement already satisfied: textgrid in /home/yuantian01/yt_py37/lib/python3.7/site-packages (from paddlespeech) (1.5)\n",
      "Requirement already satisfied: pypi-kenlm in /home/yuantian01/yt_py37_develop/lib/python3.7/site-packages (from paddlespeech) (0.1.20210121)\n",
      "Requirement already satisfied: paddlespeech-feat in /home/yuantian01/yt_py37_develop/lib/python3.7/site-packages (from paddlespeech) (0.0.1a0)\n",
      "Requirement already satisfied: tqdm in /home/yuantian01/yt_py37/lib/python3.7/site-packages (from paddlespeech) (4.62.2)\n",
      "Requirement already satisfied: jieba in /home/yuantian01/yt_py37/lib/python3.7/site-packages (from paddlespeech) (0.42.1)\n",
      "Requirement already satisfied: webrtcvad in /home/yuantian01/yt_py37/lib/python3.7/site-packages (from paddlespeech) (2.0.10)\n",
      "Requirement already satisfied: jsonlines in /home/yuantian01/yt_py37/lib/python3.7/site-packages (from paddlespeech) (2.0.0)\n",
      "Requirement already satisfied: pandas in /home/yuantian01/yt_py37/lib/python3.7/site-packages (from paddlespeech) (1.1.5)\n",
      "Collecting sentencepiece~=0.1.96\n",
      "  Downloading http://mirrors.aliyun.com/pypi/packages/ac/aa/1437691b0c7c83086ebb79ce2da16e00bef024f24fec2a5161c35476f499/sentencepiece-0.1.96-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.2 MB)\n",
      "\u001b[K     |████████████████████████████████| 1.2 MB 468 kB/s eta 0:00:01\n",
      "\u001b[?25hRequirement already satisfied: kaldiio in /home/yuantian01/yt_py37/lib/python3.7/site-packages (from paddlespeech) (2.17.2)\n",
      "Requirement already satisfied: nltk in /home/yuantian01/yt_py37/lib/python3.7/site-packages (from paddlespeech) (3.6.2)\n",
      "Requirement already satisfied: sox in /home/yuantian01/yt_py37/lib/python3.7/site-packages (from paddlespeech) (1.4.1)\n",
      "Requirement already satisfied: pypinyin in /home/yuantian01/yt_py37/lib/python3.7/site-packages (from paddlespeech) (0.42.0)\n",
      "Requirement already satisfied: praatio~=4.1 in /home/yuantian01/yt_py37/lib/python3.7/site-packages (from paddlespeech) (4.4.0)\n",
      "Requirement already satisfied: librosa in /home/yuantian01/yt_py37/lib/python3.7/site-packages (from paddlespeech) (0.8.1)\n",
      "Requirement already satisfied: paddlespeech-ctcdecoders in /home/yuantian01/yt_py37_develop/lib/python3.7/site-packages (from paddlespeech) (0.0.2a0)\n",
      "Requirement already satisfied: yacs in /home/yuantian01/yt_py37/lib/python3.7/site-packages (from paddlespeech) (0.1.8)\n",
      "Requirement already satisfied: python-dateutil in /usr/local/lib/python3.7/site-packages (from paddlespeech) (2.8.1)\n",
      "Requirement already satisfied: visualdl in /home/yuantian01/yt_py37/lib/python3.7/site-packages (from paddlespeech) (2.2.0)\n",
      "Requirement already satisfied: loguru in /home/yuantian01/yt_py37/lib/python3.7/site-packages (from paddlespeech) (0.5.3)\n",
      "Requirement already satisfied: resampy==0.2.2 in /home/yuantian01/yt_py37/lib/python3.7/site-packages (from paddlespeech) (0.2.2)\n",
      "Requirement already satisfied: timer in /home/yuantian01/yt_py37/lib/python3.7/site-packages (from paddlespeech) (0.2.2)\n",
      "Requirement already satisfied: sacrebleu in /home/yuantian01/yt_py37/lib/python3.7/site-packages (from paddlespeech) (2.0.0)\n",
      "Requirement already satisfied: matplotlib in /home/yuantian01/yt_py37/lib/python3.7/site-packages (from paddlespeech) (3.1.0)\n",
      "Requirement already satisfied: soundfile~=0.10 in /home/yuantian01/yt_py37/lib/python3.7/site-packages (from paddlespeech) (0.10.3.post1)\n",
      "Requirement already satisfied: soxbindings in /home/yuantian01/.local/lib/python3.7/site-packages (from paddlespeech) (1.2.3)\n",
      "Requirement already satisfied: pyworld in /home/yuantian01/yt_py37/lib/python3.7/site-packages (from paddlespeech) (0.2.12)\n",
      "Requirement already satisfied: nara-wpe in /home/yuantian01/yt_py37/lib/python3.7/site-packages (from paddlespeech) (0.0.7)\n",
      "Requirement already satisfied: typeguard in /home/yuantian01/yt_py37/lib/python3.7/site-packages (from paddlespeech) (2.12.1)\n",
      "Requirement already satisfied: inflect in /home/yuantian01/yt_py37/lib/python3.7/site-packages (from paddlespeech) (5.3.0)\n",
      "Requirement already satisfied: g2p-en in /home/yuantian01/yt_py37/lib/python3.7/site-packages (from paddlespeech) (2.1.0)\n",
      "Requirement already satisfied: six>=1.3 in /usr/local/lib/python3.7/site-packages (from resampy==0.2.2->paddlespeech) (1.15.0)\n",
      "Requirement already satisfied: numpy>=1.10 in /home/yuantian01/yt_py37/lib/python3.7/site-packages (from resampy==0.2.2->paddlespeech) (1.19.4)\n",
      "Requirement already satisfied: numba>=0.32 in /home/yuantian01/yt_py37/lib/python3.7/site-packages (from resampy==0.2.2->paddlespeech) (0.53.1)\n",
      "Requirement already satisfied: llvmlite<0.37,>=0.36.0rc1 in /home/yuantian01/yt_py37/lib/python3.7/site-packages (from numba>=0.32->resampy==0.2.2->paddlespeech) (0.36.0)\n",
      "Requirement already satisfied: setuptools in /home/yuantian01/yt_py37/lib/python3.7/site-packages (from numba>=0.32->resampy==0.2.2->paddlespeech) (57.4.0)\n",
      "Requirement already satisfied: cffi>=1.0 in /usr/local/lib/python3.7/site-packages (from soundfile~=0.10->paddlespeech) (1.14.3)\n",
      "Requirement already satisfied: pycparser in /usr/local/lib/python3.7/site-packages (from cffi>=1.0->soundfile~=0.10->paddlespeech) (2.20)\n",
      "Requirement already satisfied: distance>=0.1.3 in /home/yuantian01/yt_py37/lib/python3.7/site-packages (from g2p-en->paddlespeech) (0.1.3)\n",
      "Requirement already satisfied: joblib in /home/yuantian01/yt_py37/lib/python3.7/site-packages (from nltk->paddlespeech) (1.0.1)\n",
      "Requirement already satisfied: regex in /home/yuantian01/yt_py37/lib/python3.7/site-packages (from nltk->paddlespeech) (2021.8.28)\n",
      "Requirement already satisfied: click in /home/yuantian01/yt_py37/lib/python3.7/site-packages (from nltk->paddlespeech) (8.0.1)\n",
      "Requirement already satisfied: importlib-metadata in /usr/local/lib/python3.7/site-packages (from click->nltk->paddlespeech) (2.0.0)\n",
      "Requirement already satisfied: cached-property in /usr/local/lib/python3.7/site-packages (from h5py->paddlespeech) (1.5.2)\n",
      "Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.7/site-packages (from importlib-metadata->click->nltk->paddlespeech) (3.4.0)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: audioread>=2.0.0 in /home/yuantian01/yt_py37/lib/python3.7/site-packages (from librosa->paddlespeech) (2.1.9)\n",
      "Requirement already satisfied: scikit-learn!=0.19.0,>=0.14.0 in /home/yuantian01/yt_py37/lib/python3.7/site-packages (from librosa->paddlespeech) (0.24.2)\n",
      "Requirement already satisfied: decorator>=3.0.0 in /usr/local/lib/python3.7/site-packages (from librosa->paddlespeech) (4.4.2)\n",
      "Requirement already satisfied: pooch>=1.0 in /home/yuantian01/yt_py37/lib/python3.7/site-packages (from librosa->paddlespeech) (1.5.1)\n",
      "Requirement already satisfied: packaging>=20.0 in /usr/local/lib/python3.7/site-packages (from librosa->paddlespeech) (20.4)\n",
      "Requirement already satisfied: pyparsing>=2.0.2 in /usr/local/lib/python3.7/site-packages (from packaging>=20.0->librosa->paddlespeech) (2.4.7)\n",
      "Requirement already satisfied: requests in /usr/local/lib/python3.7/site-packages (from pooch>=1.0->librosa->paddlespeech) (2.25.0)\n",
      "Requirement already satisfied: appdirs in /usr/local/lib/python3.7/site-packages (from pooch>=1.0->librosa->paddlespeech) (1.4.4)\n",
      "Requirement already satisfied: threadpoolctl>=2.0.0 in /home/yuantian01/yt_py37/lib/python3.7/site-packages (from scikit-learn!=0.19.0,>=0.14.0->librosa->paddlespeech) (2.2.0)\n",
      "Requirement already satisfied: cycler>=0.10 in /home/yuantian01/yt_py37/lib/python3.7/site-packages (from matplotlib->paddlespeech) (0.10.0)\n",
      "Requirement already satisfied: kiwisolver>=1.0.1 in /home/yuantian01/yt_py37/lib/python3.7/site-packages (from matplotlib->paddlespeech) (1.3.2)\n",
      "Requirement already satisfied: bottleneck in /home/yuantian01/yt_py37/lib/python3.7/site-packages (from nara-wpe->paddlespeech) (1.2.1)\n",
      "Requirement already satisfied: mock in /home/yuantian01/yt_py37_develop/lib/python3.7/site-packages (from paddlespeech-feat->paddlespeech) (4.0.3)\n",
      "Requirement already satisfied: pytz>=2017.2 in /usr/local/lib/python3.7/site-packages (from pandas->paddlespeech) (2020.4)\n",
      "Requirement already satisfied: cython>=0.24.0 in /home/yuantian01/yt_py37/lib/python3.7/site-packages (from pyworld->paddlespeech) (0.29.24)\n",
      "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.7/site-packages (from requests->pooch>=1.0->librosa->paddlespeech) (2020.11.8)\n",
      "Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.7/site-packages (from requests->pooch>=1.0->librosa->paddlespeech) (2.10)\n",
      "Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.7/site-packages (from requests->pooch>=1.0->librosa->paddlespeech) (3.0.4)\n",
      "Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/local/lib/python3.7/site-packages (from requests->pooch>=1.0->librosa->paddlespeech) (1.26.2)\n",
      "Requirement already satisfied: portalocker in /home/yuantian01/yt_py37/lib/python3.7/site-packages (from sacrebleu->paddlespeech) (2.3.2)\n",
      "Requirement already satisfied: colorama in /home/yuantian01/yt_py37/lib/python3.7/site-packages (from sacrebleu->paddlespeech) (0.4.4)\n",
      "Requirement already satisfied: tabulate>=0.8.9 in /home/yuantian01/yt_py37/lib/python3.7/site-packages (from sacrebleu->paddlespeech) (0.8.9)\n",
      "Requirement already satisfied: protobuf>=3.11.0 in /usr/local/lib/python3.7/site-packages (from visualdl->paddlespeech) (3.13.0)\n",
      "Requirement already satisfied: bce-python-sdk in /home/yuantian01/yt_py37/lib/python3.7/site-packages (from visualdl->paddlespeech) (0.8.62)\n",
      "Requirement already satisfied: flask>=1.1.1 in /home/yuantian01/yt_py37/lib/python3.7/site-packages (from visualdl->paddlespeech) (2.0.1)\n",
      "Requirement already satisfied: Flask-Babel>=1.0.0 in /home/yuantian01/yt_py37/lib/python3.7/site-packages (from visualdl->paddlespeech) (2.0.0)\n",
      "Requirement already satisfied: pre-commit in /usr/local/lib/python3.7/site-packages (from visualdl->paddlespeech) (1.10.4)\n",
      "Requirement already satisfied: shellcheck-py in /home/yuantian01/yt_py37/lib/python3.7/site-packages (from visualdl->paddlespeech) (0.7.2.1)\n",
      "Requirement already satisfied: flake8>=3.7.9 in /home/yuantian01/yt_py37/lib/python3.7/site-packages (from visualdl->paddlespeech) (3.9.2)\n",
      "Requirement already satisfied: Pillow>=7.0.0 in /home/yuantian01/yt_py37_develop/lib/python3.7/site-packages (from visualdl->paddlespeech) (8.4.0)\n",
      "Requirement already satisfied: pyflakes<2.4.0,>=2.3.0 in /home/yuantian01/yt_py37/lib/python3.7/site-packages (from flake8>=3.7.9->visualdl->paddlespeech) (2.3.1)\n",
      "Requirement already satisfied: pycodestyle<2.8.0,>=2.7.0 in /home/yuantian01/yt_py37/lib/python3.7/site-packages (from flake8>=3.7.9->visualdl->paddlespeech) (2.7.0)\n",
      "Requirement already satisfied: mccabe<0.7.0,>=0.6.0 in /usr/local/lib/python3.7/site-packages (from flake8>=3.7.9->visualdl->paddlespeech) (0.6.1)\n",
      "Requirement already satisfied: itsdangerous>=2.0 in /home/yuantian01/yt_py37/lib/python3.7/site-packages (from flask>=1.1.1->visualdl->paddlespeech) (2.0.1)\n",
      "Requirement already satisfied: Jinja2>=3.0 in /home/yuantian01/yt_py37/lib/python3.7/site-packages (from flask>=1.1.1->visualdl->paddlespeech) (3.0.1)\n",
      "Requirement already satisfied: Werkzeug>=2.0 in /home/yuantian01/yt_py37/lib/python3.7/site-packages (from flask>=1.1.1->visualdl->paddlespeech) (2.0.1)\n",
      "Requirement already satisfied: Babel>=2.3 in /usr/local/lib/python3.7/site-packages (from Flask-Babel>=1.0.0->visualdl->paddlespeech) (2.9.0)\n",
      "Requirement already satisfied: MarkupSafe>=2.0 in /home/yuantian01/yt_py37/lib/python3.7/site-packages (from Jinja2>=3.0->flask>=1.1.1->visualdl->paddlespeech) (2.0.1)\n",
      "Requirement already satisfied: future>=0.6.0 in /home/yuantian01/yt_py37/lib/python3.7/site-packages (from bce-python-sdk->visualdl->paddlespeech) (0.18.2)\n",
      "Requirement already satisfied: pycryptodome>=3.8.0 in /home/yuantian01/yt_py37/lib/python3.7/site-packages (from bce-python-sdk->visualdl->paddlespeech) (3.10.1)\n",
      "Requirement already satisfied: nodeenv>=0.11.1 in /usr/local/lib/python3.7/site-packages (from pre-commit->visualdl->paddlespeech) (1.5.0)\n",
      "Requirement already satisfied: identify>=1.0.0 in /usr/local/lib/python3.7/site-packages (from pre-commit->visualdl->paddlespeech) (1.5.9)\n",
      "Requirement already satisfied: aspy.yaml in /usr/local/lib/python3.7/site-packages (from pre-commit->visualdl->paddlespeech) (1.3.0)\n",
      "Requirement already satisfied: cfgv>=1.0.0 in /usr/local/lib/python3.7/site-packages (from pre-commit->visualdl->paddlespeech) (3.2.0)\n",
      "Requirement already satisfied: toml in /usr/local/lib/python3.7/site-packages (from pre-commit->visualdl->paddlespeech) (0.10.2)\n",
      "Requirement already satisfied: pyyaml in /usr/local/lib/python3.7/site-packages (from pre-commit->visualdl->paddlespeech) (5.3.1)\n",
      "Requirement already satisfied: virtualenv in /usr/local/lib/python3.7/site-packages (from pre-commit->visualdl->paddlespeech) (20.1.0)\n",
      "Requirement already satisfied: filelock<4,>=3.0.0 in /usr/local/lib/python3.7/site-packages (from virtualenv->pre-commit->visualdl->paddlespeech) (3.0.12)\n",
      "Requirement already satisfied: distlib<1,>=0.3.1 in /usr/local/lib/python3.7/site-packages (from virtualenv->pre-commit->visualdl->paddlespeech) (0.3.1)\n",
      "\u001b[33mWARNING: Ignoring invalid distribution -addlepaddle-gpu (/home/yuantian01/yt_py37/lib/python3.7/site-packages)\u001b[0m\n",
      "Installing collected packages: sentencepiece\n",
      "  Attempting uninstall: sentencepiece\n",
      "\u001b[33m    WARNING: Ignoring invalid distribution -addlepaddle-gpu (/home/yuantian01/yt_py37/lib/python3.7/site-packages)\u001b[0m\n",
      "    Found existing installation: sentencepiece 0.1.86\n",
      "    Not uninstalling sentencepiece at /home/yuantian01/yt_py37/lib/python3.7/site-packages, outside environment /home/yuantian01/yt_py37_develop\n",
      "    Can't uninstall 'sentencepiece'. No files were found to uninstall.\n",
      "\u001b[31mERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.\n",
      "espnet 0.10.0 requires sentencepiece<0.1.90,>=0.1.82, but you have sentencepiece 0.1.96 which is incompatible.\u001b[0m\n",
      "Successfully installed sentencepiece-0.1.96\n",
      "\u001b[33mWARNING: Ignoring invalid distribution -addlepaddle-gpu (/home/yuantian01/yt_py37/lib/python3.7/site-packages)\u001b[0m\n",
      "\u001b[33mWARNING: Ignoring invalid distribution -addlepaddle-gpu (/home/yuantian01/yt_py37/lib/python3.7/site-packages)\u001b[0m\n",
      "\u001b[33mWARNING: Ignoring invalid distribution -addlepaddle-gpu (/home/yuantian01/yt_py37/lib/python3.7/site-packages)\u001b[0m\n",
      "\u001b[33mWARNING: You are using pip version 21.1.3; however, version 21.3.1 is available.\n",
      "You should consider upgrading via the '/home/yuantian01/yt_py37_develop/bin/python -m pip install --upgrade pip' command.\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "!pip install --upgrade pip && pip install paddlespeech -U"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "环境安装请参考 [Installation](https://github.com/PaddlePaddle/PaddleSpeech/blob/develop/docs/source/install.md) 教程。 \n",
    "下面使用 **PaddleSpeech** 提供的预训练模型合成中文语音。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 数据及模型准备"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 获取PaddlePaddle预训练模型"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--2021-12-03 12:11:28--  https://paddlespeech.bj.bcebos.com/Parakeet/released_models/pwgan/pwg_baker_ckpt_0.4.zip\n",
      "Connecting to 172.19.56.199:3128... connected.\n",
      "Proxy request sent, awaiting response... 200 OK\n",
      "Length: 15774206 (15M) [application/zip]\n",
      "Saving to: 'download/pwg_baker_ckpt_0.4.zip'\n",
      "\n",
      "pwg_baker_ckpt_0.4. 100%[===================>]  15.04M   455KB/s    in 26s     \n",
      "\n",
      "2021-12-03 12:11:56 (588 KB/s) - 'download/pwg_baker_ckpt_0.4.zip' saved [15774206/15774206]\n",
      "\n",
      "Archive:  download/pwg_baker_ckpt_0.4.zip\n",
      "   creating: download/pwg_baker_ckpt_0.4/\n",
      "  inflating: download/pwg_baker_ckpt_0.4/pwg_default.yaml  \n",
      "  inflating: download/pwg_baker_ckpt_0.4/pwg_snapshot_iter_400000.pdz  \n",
      "  inflating: download/pwg_baker_ckpt_0.4/pwg_stats.npy  \n",
      "--2021-12-03 12:11:57--  https://paddlespeech.bj.bcebos.com/Parakeet/released_models/fastspeech2/fastspeech2_nosil_baker_ckpt_0.4.zip\n",
      "Connecting to 172.19.56.199:3128... connected.\n",
      "Proxy request sent, awaiting response... 200 OK\n",
      "Length: 488623401 (466M) [application/octet-stream]\n",
      "Saving to: 'download/fastspeech2_nosil_baker_ckpt_0.4.zip'\n",
      "\n",
      "fastspeech2_nosil_b 100%[===================>] 465.99M   921KB/s    in 15m 1s  \n",
      "\n",
      "2021-12-03 12:27:00 (530 KB/s) - 'download/fastspeech2_nosil_baker_ckpt_0.4.zip' saved [488623401/488623401]\n",
      "\n",
      "Archive:  download/fastspeech2_nosil_baker_ckpt_0.4.zip\n",
      "   creating: download/fastspeech2_nosil_baker_ckpt_0.4/\n",
      "  inflating: download/fastspeech2_nosil_baker_ckpt_0.4/pitch_stats.npy  \n",
      "  inflating: download/fastspeech2_nosil_baker_ckpt_0.4/phone_id_map.txt  \n",
      "  inflating: download/fastspeech2_nosil_baker_ckpt_0.4/speech_stats.npy  \n",
      "  inflating: download/fastspeech2_nosil_baker_ckpt_0.4/energy_stats.npy  \n",
      "  inflating: download/fastspeech2_nosil_baker_ckpt_0.4/default.yaml  \n",
      "  inflating: download/fastspeech2_nosil_baker_ckpt_0.4/snapshot_iter_76000.pdz  \n"
     ]
    }
   ],
   "source": [
    "!wget -P download https://paddlespeech.bj.bcebos.com/Parakeet/released_models/pwgan/pwg_baker_ckpt_0.4.zip\n",
    "!unzip -d download download/pwg_baker_ckpt_0.4.zip\n",
    "!wget -P download https://paddlespeech.bj.bcebos.com/Parakeet/released_models/fastspeech2/fastspeech2_nosil_baker_ckpt_0.4.zip\n",
    "!unzip -d download download/fastspeech2_nosil_baker_ckpt_0.4.zip"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[01;34mdownload/pwg_baker_ckpt_0.4\u001b[00m\n",
      "|-- pwg_default.yaml\n",
      "|-- pwg_snapshot_iter_400000.pdz\n",
      "`-- pwg_stats.npy\n",
      "\n",
      "0 directories, 3 files\n",
      "\u001b[01;34mdownload/fastspeech2_nosil_baker_ckpt_0.4\u001b[00m\n",
      "|-- default.yaml\n",
      "|-- energy_stats.npy\n",
      "|-- phone_id_map.txt\n",
      "|-- pitch_stats.npy\n",
      "|-- snapshot_iter_76000.pdz\n",
      "`-- speech_stats.npy\n",
      "\n",
      "0 directories, 6 files\n"
     ]
    }
   ],
   "source": [
    "!tree download/pwg_baker_ckpt_0.4\n",
    "!tree download/fastspeech2_nosil_baker_ckpt_0.4"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 导入 Python 包"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--2021-12-03 12:27:12--  https://paddlespeech.bj.bcebos.com/Parakeet/tools/nltk_data.tar.gz\n",
      "Connecting to 172.19.56.199:3128... connected.\n",
      "Proxy request sent, awaiting response... 200 OK\n",
      "Length: 6850884 (6.5M) [application/x-gzip]\n",
      "Saving to: 'nltk_data.tar.gz'\n",
      "\n",
      "nltk_data.tar.gz    100%[===================>]   6.53M  1.07MB/s    in 13s     \n",
      "\n",
      "2021-12-03 12:27:27 (506 KB/s) - 'nltk_data.tar.gz' saved [6850884/6850884]\n",
      "\n",
      "nltk_data/\n",
      "nltk_data/taggers/\n",
      "nltk_data/taggers/averaged_perceptron_tagger/\n",
      "nltk_data/taggers/averaged_perceptron_tagger/averaged_perceptron_tagger.pickle\n",
      "nltk_data/taggers/averaged_perceptron_tagger.zip\n",
      "nltk_data/corpora/\n",
      "nltk_data/corpora/cmudict/\n",
      "nltk_data/corpora/cmudict/cmudict\n",
      "nltk_data/corpora/cmudict/README\n",
      "nltk_data/corpora/cmudict.zip\n"
     ]
    }
   ],
   "source": [
    "# 本项目的依赖需要用到 nltk 包，但是有时会因为网络原因导致不好下载，此处手动下载一下放到百度服务器的包\n",
    "!wget https://paddlespeech.bj.bcebos.com/Parakeet/tools/nltk_data.tar.gz\n",
    "!tar zxvf nltk_data.tar.gz"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "env: CUDA_VISIBLE_DEVICES=0\n"
     ]
    }
   ],
   "source": [
    "# 设置 gpu 环境\n",
    "%env CUDA_VISIBLE_DEVICES=0\n",
    "\n",
    "import logging\n",
    "import sys\n",
    "import warnings\n",
    "warnings.filterwarnings(\"ignore\", category=DeprecationWarning)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "import argparse\n",
    "import os\n",
    "from pathlib import Path\n",
    "import IPython.display as dp\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "import paddle\n",
    "import soundfile as sf\n",
    "import yaml\n",
    "from paddlespeech.t2s.frontend.zh_frontend import Frontend\n",
    "from paddlespeech.t2s.models.fastspeech2 import FastSpeech2\n",
    "from paddlespeech.t2s.models.fastspeech2 import FastSpeech2Inference\n",
    "from paddlespeech.t2s.models.parallel_wavegan import PWGGenerator\n",
    "from paddlespeech.t2s.models.parallel_wavegan import PWGInference\n",
    "from paddlespeech.t2s.modules.normalizer import ZScore\n",
    "from yacs.config import CfgNode"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 设置预训练模型的路径"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "========Config========\n",
      "batch_size: 64\n",
      "f0max: 400\n",
      "f0min: 80\n",
      "fmax: 7600\n",
      "fmin: 80\n",
      "fs: 24000\n",
      "max_epoch: 1000\n",
      "model:\n",
      "  adim: 384\n",
      "  aheads: 2\n",
      "  decoder_normalize_before: True\n",
      "  dlayers: 4\n",
      "  dunits: 1536\n",
      "  duration_predictor_chans: 256\n",
      "  duration_predictor_kernel_size: 3\n",
      "  duration_predictor_layers: 2\n",
      "  elayers: 4\n",
      "  encoder_normalize_before: True\n",
      "  energy_embed_dropout: 0.0\n",
      "  energy_embed_kernel_size: 1\n",
      "  energy_predictor_chans: 256\n",
      "  energy_predictor_dropout: 0.5\n",
      "  energy_predictor_kernel_size: 3\n",
      "  energy_predictor_layers: 2\n",
      "  eunits: 1536\n",
      "  init_dec_alpha: 1.0\n",
      "  init_enc_alpha: 1.0\n",
      "  init_type: xavier_uniform\n",
      "  pitch_embed_dropout: 0.0\n",
      "  pitch_embed_kernel_size: 1\n",
      "  pitch_predictor_chans: 256\n",
      "  pitch_predictor_dropout: 0.5\n",
      "  pitch_predictor_kernel_size: 5\n",
      "  pitch_predictor_layers: 5\n",
      "  positionwise_conv_kernel_size: 3\n",
      "  positionwise_layer_type: conv1d\n",
      "  postnet_chans: 256\n",
      "  postnet_filts: 5\n",
      "  postnet_layers: 5\n",
      "  reduction_factor: 1\n",
      "  stop_gradient_from_energy_predictor: False\n",
      "  stop_gradient_from_pitch_predictor: True\n",
      "  transformer_dec_attn_dropout_rate: 0.2\n",
      "  transformer_dec_dropout_rate: 0.2\n",
      "  transformer_dec_positional_dropout_rate: 0.2\n",
      "  transformer_enc_attn_dropout_rate: 0.2\n",
      "  transformer_enc_dropout_rate: 0.2\n",
      "  transformer_enc_positional_dropout_rate: 0.2\n",
      "  use_scaled_pos_enc: True\n",
      "n_fft: 2048\n",
      "n_mels: 80\n",
      "n_shift: 300\n",
      "num_snapshots: 5\n",
      "num_workers: 4\n",
      "optimizer:\n",
      "  learning_rate: 0.001\n",
      "  optim: adam\n",
      "seed: 10086\n",
      "updater:\n",
      "  use_masking: True\n",
      "win_length: 1200\n",
      "window: hann\n",
      "---------------------\n",
      "allow_cache: True\n",
      "batch_max_steps: 25500\n",
      "batch_size: 6\n",
      "discriminator_grad_norm: 1\n",
      "discriminator_optimizer_params:\n",
      "  epsilon: 1e-06\n",
      "  weight_decay: 0.0\n",
      "discriminator_params:\n",
      "  bias: True\n",
      "  conv_channels: 64\n",
      "  in_channels: 1\n",
      "  kernel_size: 3\n",
      "  layers: 10\n",
      "  nonlinear_activation: LeakyReLU\n",
      "  nonlinear_activation_params:\n",
      "    negative_slope: 0.2\n",
      "  out_channels: 1\n",
      "  use_weight_norm: True\n",
      "discriminator_scheduler_params:\n",
      "  gamma: 0.5\n",
      "  learning_rate: 5e-05\n",
      "  step_size: 200000\n",
      "discriminator_train_start_steps: 100000\n",
      "eval_interval_steps: 1000\n",
      "fmax: 7600\n",
      "fmin: 80\n",
      "fs: 24000\n",
      "generator_grad_norm: 10\n",
      "generator_optimizer_params:\n",
      "  epsilon: 1e-06\n",
      "  weight_decay: 0.0\n",
      "generator_params:\n",
      "  aux_channels: 80\n",
      "  aux_context_window: 2\n",
      "  bias: True\n",
      "  dropout: 0.0\n",
      "  freq_axis_kernel_size: 1\n",
      "  gate_channels: 128\n",
      "  in_channels: 1\n",
      "  interpolate_mode: nearest\n",
      "  kernel_size: 3\n",
      "  layers: 30\n",
      "  nonlinear_activation: None\n",
      "  nonlinear_activation_params:\n",
      "    \n",
      "  out_channels: 1\n",
      "  residual_channels: 64\n",
      "  skip_channels: 64\n",
      "  stacks: 3\n",
      "  upsample_scales: [4, 5, 3, 5]\n",
      "  use_causal_conv: False\n",
      "  use_weight_norm: True\n",
      "generator_scheduler_params:\n",
      "  gamma: 0.5\n",
      "  learning_rate: 0.0001\n",
      "  step_size: 200000\n",
      "lambda_adv: 4.0\n",
      "n_fft: 2048\n",
      "n_mels: 80\n",
      "n_shift: 300\n",
      "num_save_intermediate_results: 4\n",
      "num_snapshots: 10\n",
      "num_workers: 4\n",
      "pin_memory: True\n",
      "remove_short_samples: True\n",
      "save_interval_steps: 5000\n",
      "seed: 42\n",
      "stft_loss_params:\n",
      "  fft_sizes: [1024, 2048, 512]\n",
      "  hop_sizes: [120, 240, 50]\n",
      "  win_lengths: [600, 1200, 240]\n",
      "  window: hann\n",
      "top_db: 60\n",
      "train_max_steps: 400000\n",
      "trim_frame_length: 2048\n",
      "trim_hop_length: 512\n",
      "trim_silence: False\n",
      "win_length: 1200\n",
      "window: hann\n"
     ]
    }
   ],
   "source": [
    "fastspeech2_config = \"download/fastspeech2_nosil_baker_ckpt_0.4/default.yaml\"\n",
    "fastspeech2_checkpoint = \"download/fastspeech2_nosil_baker_ckpt_0.4/snapshot_iter_76000.pdz\"\n",
    "fastspeech2_stat = \"download/fastspeech2_nosil_baker_ckpt_0.4/speech_stats.npy\"\n",
    "pwg_config = \"download/pwg_baker_ckpt_0.4/pwg_default.yaml\"\n",
    "pwg_checkpoint = \"download/pwg_baker_ckpt_0.4/pwg_snapshot_iter_400000.pdz\"\n",
    "pwg_stat = \"download/pwg_baker_ckpt_0.4/pwg_stats.npy\"\n",
    "phones_dict = \"download/fastspeech2_nosil_baker_ckpt_0.4/phone_id_map.txt\"\n",
    "# 读取 conf 配置文件并结构化\n",
    "with open(fastspeech2_config) as f:\n",
    "    fastspeech2_config = CfgNode(yaml.safe_load(f))\n",
    "with open(pwg_config) as f:\n",
    "    pwg_config = CfgNode(yaml.safe_load(f))\n",
    "print(\"========Config========\")\n",
    "print(fastspeech2_config)\n",
    "print(\"---------------------\")\n",
    "print(pwg_config)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 文本前端（Text Frontend）\n",
    "\n",
    "一个文本前端模块主要包含:\n",
    "- 分段（Text Segmentation）\n",
    "- 文本正则化（Text Normalization, TN）\n",
    "- 分词（Word Segmentation, 主要是在中文中）\n",
    "- 词性标注（Part-of-Speech, PoS）\n",
    "- 韵律预测（Prosody）\n",
    "- 字音转换（Grapheme-to-Phoneme，G2P）\n",
    "\n",
    "<font size=2>（Grapheme: **语言**书写系统的最小有意义单位; Phoneme: 区分单词的最小**语音**单位）</font>\n",
    "    - 多音字（Polyphone）\n",
    "    - 变调（Tone Sandhi）\n",
    "        - “一”、“不”变调\n",
    "        - 三声变调\n",
    "        - 轻声变调\n",
    "        - 儿化音\n",
    "        - 方言\n",
    "- ...\n",
    "\n",
    "（输入给声学模型之前，还需要把音素序列转换为 id）\n",
    "\n",
    "\n",
    "其中最重要的模块是<font color=\"#ff0000\"> 文本正则化 </font>模块和<font color=\"#ff0000\"> 字音转换（TTS 中更常用 G2P 代指） </font>模块。\n",
    "\n",
    "\n",
    "各模块输出示例:\n",
    "```text\n",
    "• Text: 全国一共有112所211高校\n",
    "• Text Normalization: 全国一共有一百一十二所二一一高校\n",
    "• Word Segmentation: 全国/一共/有/一百一十二/所/二一一/高校/\n",
    "• G2P（注意此句中“一”的读音）:\n",
    "    quan2 guo2 yi2 gong4 you3 yi4 bai3 yi1 shi2 er4 suo3 er4 yao1 yao1 gao1 xiao4\n",
    "    （可以进一步把声母和韵母分开）\n",
    "    q uan2 g uo2 y i2 g ong4 y ou3 y i4 b ai3 y i1 sh i2 er4 s uo3 er4 y ao1 y ao1 g ao1 x iao4\n",
    "    （把音调和声韵母分开）\n",
    "    q uan g uo y i g ong y ou y i b ai y i sh i er s uo er y ao y ao g ao x iao\n",
    "    0 2 0 2 0 2 0 4 0 3 ...\n",
    "• Prosody (prosodic words #1, prosodic phrases #2, intonation phrases #3, sentence #4):\n",
    "    全国#2一共有#2一百#1一十二所#2二一一#1高校#4\n",
    "    （分词的结果一般是固定的，但是不同人习惯不同，可能有不同的韵律）\n",
    "```\n",
    "\n",
    "文本前端模块的设计需要结合很多专业的语义学知识和经验。人类在读文本的时候可以自然而然地读出正确的发音，但是这些先验知识计算机并不知晓。\n",
    "例如，对于一个句子的分词：\n",
    "\n",
    "```text\n",
    "我也想过过过儿过过的生活\n",
    "我也想/过过/过儿/过过的/生活\n",
    "\n",
    "货拉拉拉不拉拉布拉多\n",
    "货拉拉/拉不拉/拉布拉多\n",
    "\n",
    "南京市长江大桥\n",
    "南京市长/江大桥\n",
    "南京市/长江大桥\n",
    "```\n",
    "或者是词的变调和儿化音：\n",
    "```\n",
    "你要不要和我们一起出去玩？\n",
    "你要不（2声）要和我们一（4声）起出去玩（儿）？\n",
    "\n",
    "不好，我要一个人出去。\n",
    "不（4声）好，我要一（2声）个人出去。\n",
    "\n",
    "（以下每个词的所有字都是三声的，请你读一读，体会一下在读的时候，是否每个字都被读成了三声？）\n",
    "纸老虎、虎骨酒、展览馆、岂有此理、手表厂有五种好产品\n",
    "```\n",
    "又或是多音字，这类情况通常需要先正确分词：\n",
    "```text\n",
    "人要行，干一行行一行，一行行行行行;\n",
    "人要是不行，干一行不行一行，一行不行行行不行。\n",
    "\n",
    "佟大为妻子产下一女\n",
    "\n",
    "海水朝朝朝朝朝朝朝落\n",
    "浮云长长长长长长长消\n",
    "```\n",
    "\n",
    "PaddleSpeech Text-to-Speech的文本前端解决方案:\n",
    "- [文本正则](https://github.com/PaddlePaddle/PaddleSpeech/tree/develop/examples/other/tn)\n",
    "- [G2P](https://github.com/PaddlePaddle/PaddleSpeech/tree/develop/examples/other/g2p)\n",
    "    - 多音字模块: pypinyin/g2pM\n",
    "    - 变调模块: 用分词 + 规则"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 构造文本前端对象\n",
    "传入`phones_dict`，把相应的`phones`转换成`phone_ids`。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Frontend done!\n"
     ]
    }
   ],
   "source": [
    "# 传入 phones_dict 会把相应的 phones 转换成 phone_ids\n",
    "frontend = Frontend(phone_vocab_path=phones_dict)\n",
    "print(\"Frontend done!\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 调用文本前端\n",
    "\n",
    "文本前端对输入数据进行正则化时会进行分句，若`merge_sentences`设置为`False`，则所有分句的 `phone_ids` 构成一个 `List`；若设置为`True`，`input_ids[\"phone_ids\"][0]`则表示整句的`phone_ids`。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Building prefix dict from the default dictionary ...\n",
      "DEBUG:jieba:Building prefix dict from the default dictionary ...\n",
      "Loading model from cache /tmp/jieba.cache\n",
      "DEBUG:jieba:Loading model from cache /tmp/jieba.cache\n",
      "Loading model cost 1.135 seconds.\n",
      "DEBUG:jieba:Loading model cost 1.135 seconds.\n",
      "Prefix dict has been built successfully.\n",
      "DEBUG:jieba:Prefix dict has been built successfully.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "----------------------------\n",
      "text norm results:\n",
      "['你好，', '欢迎使用百度飞桨框架进行深度学习研究！']\n",
      "----------------------------\n",
      "g2p results:\n",
      "[['n', 'i2', 'h', 'ao3', 'sp', 'sp', 'h', 'uan1', 'ing2', 'sh', 'iii3', 'iong4', 'b', 'ai3', 'd', 'u4', 'f', 'ei1', 'j', 'iang3', 'k', 'uang1', 'j', 'ia4', 'j', 'in4', 'x', 'ing2', 'sh', 'en1', 'd', 'u4', 'x', 've2', 'x', 'i2', 'ian2', 'j', 'iou1', 'sp', 'sp']]\n",
      "----------------------------\n",
      "phone_ids:Tensor(shape=[41], dtype=int64, place=CUDAPlace(0), stop_gradient=True,\n",
      "       [155, 73 , 71 , 29 , 179, 179, 71 , 199, 126, 177, 115, 138, 37 , 9  ,\n",
      "        40 , 186, 69 , 46 , 151, 89 , 152, 204, 151, 80 , 151, 123, 260, 126,\n",
      "        177, 51 , 40 , 186, 260, 251, 260, 73 , 83 , 151, 140, 179, 179])\n"
     ]
    }
   ],
   "source": [
    "# input = \"我每天中午12:00起床\"\n",
    "# input = \"我出生于2005/11/08，那天的最低气温达到-10°C\"\n",
    "input = \"你好，欢迎使用百度飞桨框架进行深度学习研究！\"\n",
    "input_ids = frontend.get_input_ids(input, merge_sentences=True, print_info=True)\n",
    "phone_ids = input_ids[\"phone_ids\"][0]\n",
    "print(\"phone_ids:%s\"%phone_ids)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 用深度学习实现文本前端\n",
    "\n",
    "<center><img src=\"https://ai-studio-static-online.cdn.bcebos.com/0e89df202cd549c2a280cd3f1b95c76bec55ccb5398a489e906d93b6b4d4c886\"></center>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 声学模型（Acoustic Model）\n",
    "\n",
    "声学模型将字符/音素转换为声学特征，如线性频谱图、mel 频谱图、LPC 特征等，声学特征以 “帧” 为单位，一般一帧是 10ms 左右，一个音素一般对应 5~20 帧左右, 声学模型需要解决的是 <font color=\"#ff0000\">“不等长序列间的映射问题”</font>，“不等长”是指，同一个人发不同音素的持续时间不同，同一个人在不同时刻说同一句话的语速可能不同，对应各个音素的持续时间不同，不同人说话的特色不同，对应各个音素的持续时间不同。这是一个困难的“一对多”问题。\n",
    "```\n",
    "# 卡尔普陪外孙玩滑梯\n",
    "000001|baker_corpus|sil 20 k 12 a2 4 er2 10 p 12 u3 12 p 9 ei2 9 uai4 15 s 11 uen1 12 uan2 14 h 10 ua2 11 t 15 i1 16 sil 20\n",
    "```\n",
    "\n",
    "声学模型主要分为自回归模型和非自回归模型，其中自回归模型在 `t` 时刻的预测需要依赖 `t-1` 时刻的输出作为输入，预测时间长，但是音质相对较好，非自回归模型不存在预测上的依赖关系，预测时间快，音质相对较差。\n",
    "\n",
    "主流声学模型发展的脉络:\n",
    "- 自回归模型:\n",
    "    - Tacotron\n",
    "    - Tacotron2\n",
    "    - Transformer TTS\n",
    "- 非自回归模型:\n",
    "    - FastSpeech\n",
    "    - SpeedySpeech\n",
    "    - FastPitch\n",
    "    - FastSpeech2\n",
    "    - ...\n",
    " \n",
    "在本教程中，我们使用 `FastSpeech2` 作为声学模型。\n",
    "<center><img src=\"https://ai-studio-static-online.cdn.bcebos.com/6b6d671713ec4d20a0e60653c7a5d4ae3c35b1d1e58b4cc39e0bc82ad4a341d9\"></center>\n",
    "<br><center> FastSpeech2 网络结构图</center></br>\n",
    "\n",
    "\n",
    "PaddleSpeech TTS 实现的 FastSpeech2 与论文不同的地方在于，我们使用的的是 phone 级别的 `pitch` 和 `energy`(与 FastPitch 类似)。\n",
    "<center><img src=\"https://ai-studio-static-online.cdn.bcebos.com/862c21456c784c41a83a308b7d9707f0810cc3b3c6f94ed48c60f5d32d0072f0\"></center>\n",
    "<br><center> FastPitch 网络结构图</center></br>\n",
    "\n",
    "更多关于[语音合成模型的发展及改进](https://github.com/PaddlePaddle/PaddleSpeech/blob/develop/docs/source/tts/models_introduction.md)。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 初始化声学模型 FastSpeech2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vocab_size: 268\n",
      "encoder_type is transformer\n",
      "decoder_type is transformer\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/yuantian01/yt_py37/lib/python3.7/site-packages/paddle/framework/io.py:415: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working\n",
      "  if isinstance(obj, collections.Iterable) and not isinstance(obj, (\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "FastSpeech2 done!\n"
     ]
    }
   ],
   "source": [
    "with open(phones_dict, \"r\") as f:\n",
    "    phn_id = [line.strip().split() for line in f.readlines()]\n",
    "vocab_size = len(phn_id)\n",
    "print(\"vocab_size:\", vocab_size)\n",
    "odim = fastspeech2_config.n_mels\n",
    "model = FastSpeech2(\n",
    "    idim=vocab_size, odim=odim, **fastspeech2_config[\"model\"])\n",
    "# 加载预训练模型参数\n",
    "model.set_state_dict(paddle.load(fastspeech2_checkpoint)[\"main_params\"])\n",
    "# 推理阶段不启用 batch norm 和 dropout\n",
    "model.eval()\n",
    "stat = np.load(fastspeech2_stat)\n",
    "# 读取数据预处理阶段数据集的均值和标准差\n",
    "mu, std = stat\n",
    "mu, std = paddle.to_tensor(mu), paddle.to_tensor(std)\n",
    "# 构造归一化的新模型\n",
    "fastspeech2_normalizer = ZScore(mu, std)\n",
    "fastspeech2_inference = FastSpeech2Inference(fastspeech2_normalizer, model)\n",
    "fastspeech2_inference.eval()\n",
    "print(\"FastSpeech2 done!\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 调用声学模型"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "shepe of mel (n_frames x n_mels):\n",
      "[362, 80]\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/yuantian01/yt_py37/lib/python3.7/site-packages/paddle/fluid/dygraph/math_op_patch.py:253: UserWarning: The dtype of left and right variables are not the same, left dtype is paddle.int64, but right dtype is paddle.int32, the right dtype will convert to paddle.int64\n",
      "  format(lhs_dtype, rhs_dtype, lhs_dtype))\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAk8AAAGoCAYAAABfbgHJAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjAsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+17YcXAAAgAElEQVR4nOy9e7Bty17f9fl19xhzrrX3edzLheu9F8Lj8owgJIIGSyNJRCJlAhVLQipqNIn8oxVRKwXiC7WMlGUllcSohZCERE1IgBgsk2DARyyrIDwSohEpHpfHhfs6r/1ac84xuvvnH7/uHj3X3ufec7hnn33Wvv2tWrXWnHOM7h79mr3G+PT3J6rK0NDQ0NDQ0NDQa5N70gUYGhoaGhoaGrpJGounoaGhoaGhoaHXobF4GhoaGhoaGhp6HRqLp6GhoaGhoaGh16GxeBoaGhoaGhoaeh0ai6ehoaGhoaGhodehsXgaGnrKJCJfISLvf9LlGBoaGnpaNRZPQ0NvIYnIL4jIIiLvuPb+3xERFZHPeAPy+BoR+bsicldEXhCR/1VEPvPjTfdj5Kki8tmPM4+hoaGhN0tj8TQ09NbT+4DfU1+IyBcBl29EwmUB8+eAfxt4DvhM4E8B6Y1I/+MoV3grpzc0NDTUayyehobeevrzwL/Uvf592IKnSUR2IvJfiMgviciHROS/EZGL15D2lwDvU9UfUtM9Vf1eVf2lku63isj3iMh3i8g9EfkJEfniLt93i8j3ishHROR9IvKHus+8iHyLiPxcOffHReTTRORvlUN+UkTui8jvro8WReSbROSDwJ8pafyrIvKzIvKSiHy/iLy7S/+fFpGfFpE7IvJficj/ISJ/sHz2L4vI/yUif0xEXgS+VUTeW+6qvVjusP33IvJ8l94viMgfFpG/JyIPROQ7ReSdIvLXS/l/UETe9tqabGho6BNJY/E0NPTW0w8Dz4rIF4iIB74e+O+uHfNtwOdii6HPBt4D/AevIe2fAD6/LDJ+i4jcfsQxXwP8ZeDtwP8A/I8iMomIA/4n4CdLfr8N+EYR+apy3r+F3TH7auBZ4PcDV6r6m8vnX6yqt1X1u8vrf6Dk8enAN4jIbwX+M+DrgHcBvwj8RYDyGPN7gH8H+CTgp4F/7Fq5/1Hg54F3Av8pICW9dwNfAHwa8K3XzvnngK/E6vJ3AH8d+Bbgk7H58Q8xNDQ0dE1j8TQ09NZUvfv0lcBPAb9SPxARAb4B+DdV9SVVvQf8EWyR9VGlqj8PfAW2+PlLwAsi8mevLaJ+XFW/R1VX4I8Ce+A3AV8GfLKq/sequpS0/tsu3z8I/Huq+tPlrtZPquqLH6U4GfgPVfWkqgfg9wJ/WlV/QlVP2ELpywvn9dXA31fV71PVCPwJ4IPX0vtVVf2TqhpV9aCqP6uqf7Ok/5FyLf/ktXP+pKp+SFV/Bfg/gR9R1b+jqkfgrwC/4WPV6dDQ0CeeBhcwNPTW1J8H/hbGJP25a599MsZA/bitowC7y+JfS8Kq+sPY3R1E5MuA7wb+XWyxAvDL3bG57Nx7N6DAu0XklS45jy06wO7s/NxrKUPRR8oiperd2J2xmvf98gjuPeWzvlz6iB2Fv9y/EJF3An8c+CeAZ7B/Fl++ds6Hur8Pj3j9qDtzQ0NDn+Aai6ehobegVPUXReR92B2XP3Dt4xewL/Z/sNwx+Xjy+VER+T7gC7u3P63+UR7VfSrwq0DEeKnPeZXkfhl4L/D/vNbsr73+VewRXs37FvaI7leAD5Ry1M+kf/0q6f2R8t4XqepLIvK1wH/5Gss2NDQ09Koaj+2Ght66+gPAb1XVB/2bqpqxx2V/TEQ+BUBE3tOxR68qEfnHC5Rdz/t84HdinFXVPywiv6vsWPtG4FQ+/9vAvQJ5XxRA/AvL3SuA7wD+ExH5HDH9QyLySeWzDwGf9TGK9xeAf0VEvkREdtji50dU9ReA/xn4IhH52lKufw1jpj6angHuA3dE5D3AH/4Yxw8NDQ29Jo3F09DQW1Sq+nOq+mOv8vE3AT8L/LCI3AV+EPi815DsK9hi6f8WkfvA38DYnv+8O+avAr8be8T1LwK/S1VXVU3AP0vZsYfdAfsOzPIAjCn6S8D/AtwFvhOoOwC/FfguEXlFRL7uVa73B4F/H/he7E7Teyk8laq+APzzpZwvAr8e+DFsYfdq+o+A3wjcwRZf3/dRa2ZoaGjoNUpUr9/pHhoa+kSViHwr8Nmq+i886bJ8NJXHie8Hfq+q/m9PujxDQ0OfWBp3noaGhm6EROSrROT58kjvWzBI/oc/xmlDQ0NDb7jG4mloaOim6Mux3XwvYJ5MX1ssDoaGhobeVI3HdkNDQ0NDQ0NDr0PjztPQ0NDQ0NDQ0OvQjfB5mmWnF9UA2Tvwnjxb0dNOyBOoxwgIANHt7yoV8zOuEsCVu24OrlvEiIBqOa9+rDwi3UfnIQqSt88llx8Fsm7vJUVSOSll0FwyLnLerhlQJ1aw69KtIC19LWlmtfTqZcwTae/R0vLqtkts53eXY29ey0/KZx+lLuTaDU0VrJ6vf/CIc+sJkrsyTJkpJFw534miCOh2alYhZoeuVl9usTSuZ9EXu78OfUTVSmn3Vuxrr62NteUjWSErslqc3XRrJu0t7XrOQ3WnIIkujfNqkqy4JcMaAVif35H2WH2mrpxdmio83B59HV9vQykH5a7O1PLQ68f0ieQtDb+L7LyV8RgnAPLJ27V19Wd91F66VMdAHRN61v/VCXn2pJmzPtvKV7O/fn29avnrcE9W32kGubQK3AUr9ymGrdza5XV9TtGScH9d+eExj0Da2euwiy0fgFMKxNVD6gfgtTwU8EqYrJz7EHGS0dIoa/YsyZMX80eVxEdvd7V61EkRX+q8NE5NU6Mgq5z31/7f7AwulrywNqv5rLdLGkHtxEcNuu7azvKYFOetY3iX2xycsmWeo0OitHyvSx3orEyT1fHsE0KbbknqWJMnR9fqvOV/PbFHzAXt+EeMpT6RWjc17RwgXSjz3sq1c5GMcErW1+LqYZVXy7K7wFf5uy9bVemLLoHE0s4pW5+u811w5EnIZVytr7xEPD74mMV4o/VVv+WWvvjSxx+X/Mf/3ukHVPW3vwFFek26EYunPbf4TfuvBkBuXcLbn+f46Rav885nzRw+WVifU9LeBp7OCiFvnToLcnL4q20GSPuMXtjxskuIU6QspgSbULJKm5T0VL5F6oKrfjnlkkkGtzjcyV77E/iDEIp/sj8q4QDhqIRjmSAOmeneSrhjB8md++jh0L4kAeSZ2+iztyyL/YzuPNotoETLF08u17Im5Lgip8XKfTzBsoKzc/Jnvps7n3ObwzusLuItyJ0vdVv0QRtUebIP6heJBshBbcFa66LUAYCrE1wdzM6OzxeKTvUgPV/QXlu5yCKEB66VQd5z4FPefpdnZtuZPvtk7aPSJtf768zL9y85fOQSgMtfDkz3tjTV077kqvKELUS6z+1FyTeDW8Evpdir4pbttV+sPf3JTgiHiL+74D9iJtx3v+xTeekLPHmydGo+edpmMbfA9ACm+zUNxS/aJkN/VC5/6S7yoZcA+PDveC8v/3pFd4q/78o5Qp62NlHH2STvIvjTVt9pr9s/HUDeZXRS/JVjulcbGvJsx9k52b4UaxtlQRZp7z3/Ga/w2W9/AYD/74VPAeDqZ59jfsW1Lzy3QjiCP9g5831l90pkvmMV6u4vyBohWMHyxcSDT7vF3U/3HN9h59TFSEszgUTB1WFTvxBbf7W+68vYnF+B+a5y/9cJ8iV3APi8T/4wWR0/95LZUh1+7lncIqRLSyzvc+mzNQ9BkkCsbSj4E4QrIRQKyx8UDcKdz7WCvuO9L/HZz79ALgPmF+++jQ99+Dl4ZW51iu++AZNdU3om8cnvsf70uW/7CM9Ox/bF+4HDs/ziS2/j8Kv2z+X8kj9fiCdr+1Y3GZa3wfFdK/PzZSzNERFlXS3N44sX7H81tP6aZ0g7bQsofyXsX4D9y5bJdMi4xeahD365XcvxUyKSBLeUeWeqi2Np5ZIouKXL450Lzzx/BcD
      "text/plain": [
       "<Figure size 648x432 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "with paddle.no_grad():\n",
    "    mel = fastspeech2_inference(phone_ids)\n",
    "print(\"shepe of mel (n_frames x n_mels):\")\n",
    "print(mel.shape)\n",
    "# 绘制声学模型输出的 mel 频谱\n",
    "fig, ax = plt.subplots(figsize=(9, 6))\n",
    "im = ax.imshow(mel.T, aspect='auto',origin='lower')\n",
    "fig.colorbar(im, ax=ax)\n",
    "plt.title('Mel Spectrogram')\n",
    "plt.xlabel('Time')\n",
    "plt.ylabel('Frequency')\n",
    "plt.tight_layout()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 声码器（Vocoder）\n",
    "声码器将声学特征转换为波形。声码器需要解决的是 <font color=\"#ff0000\">“信息缺失的补全问题”</font>。信息缺失是指，在音频波形转换为频谱图的时候，存在**相位信息**的缺失，在频谱图转换为 mel 频谱图的时候，存在**频域压缩**导致的信息缺失；假设音频的采样率是16kHZ, 一帧的音频有 10ms，也就是说，1s 的音频有 16000 个采样点，而 1s 中包含 100 帧，每一帧有 160 个采样点，声码器的作用就是将一个频谱帧变成音频波形的 160 个采样点，所以声码器中一般会包含**上采样**模块。\n",
    "\n",
    "与声学模型类似，声码器也分为自回归模型和非自回归模型, 更细致的分类如下:\n",
    "\n",
    "- Autoregression\n",
    "    - WaveNet\n",
    "    - WaveRNN\n",
    "    - LPCNet\n",
    "- Flow\n",
    "    - WaveFlow\n",
    "    - WaveGlow\n",
    "    - FloWaveNet\n",
    "    - Parallel WaveNet\n",
    "- GAN\n",
    "    - WaveGAN\n",
    "    - Parallel WaveGAN\n",
    "    - MelGAN\n",
    "    - HiFi-GAN\n",
    "- VAE\n",
    "    - Wave-VAE\n",
    "- Diffusion\n",
    "    - WaveGrad\n",
    "    - DiffWave\n",
    "\n",
    "PaddleSpeech TTS 主要实现了百度的 `WaveFlow` 和一些主流的 GAN Vocoder, 在本教程中，我们使用 `Parallel WaveGAN` 作为声码器。\n",
    "\n",
    "<center><img src=\"https://ai-studio-static-online.cdn.bcebos.com/9eafa4e5642d45309e6e8883bff46380407b3858d0934bf5896868281316ce94\" ></center>\n",
    "<br><center>图1：Parallel WaveGAN 网络结构图</center></br>\n",
    "\n",
    "各 GAN Vocoder 的生成器和判别器的 Loss 的区别如下表格所示:\n",
    "\n",
    "Model  | Generator Loss |Discriminator Loss\n",
    ":-------------:| :------------:| :-----\n",
    "Parallel Wave GAN| adversial loss <br> Feature Matching  | Multi-Scale Discriminator |\n",
    "Mel GAN |adversial loss <br> Multi-resolution STFT loss  | adversial loss|\n",
    "Multi-Band Mel GAN | adversial loss <br> full band Multi-resolution STFT loss <br> sub band Multi-resolution STFT loss |Multi-Scale Discriminator|\n",
    "HiFi GAN |adversial loss <br> Feature Matching <br>  Mel-Spectrogram Loss | Multi-Scale Discriminator <br> Multi-Period Discriminator| \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 初始化声码器 Parallel WaveGAN"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Parallel WaveGAN done!\n"
     ]
    }
   ],
   "source": [
    "vocoder = PWGGenerator(**pwg_config[\"generator_params\"])\n",
    "# 模型加载预训练参数\n",
    "vocoder.set_state_dict(paddle.load(pwg_checkpoint)[\"generator_params\"])\n",
    "vocoder.remove_weight_norm()\n",
    "# 推理阶段不启用 batch norm 和 dropout\n",
    "vocoder.eval()\n",
    "# 读取数据预处理阶段数据集的均值和标准差\n",
    "stat = np.load(pwg_stat)\n",
    "mu, std = stat\n",
    "mu, std = paddle.to_tensor(mu), paddle.to_tensor(std)\n",
    "pwg_normalizer = ZScore(mu, std)\n",
    "# 构建归一化的模型\n",
    "pwg_inference = PWGInference(pwg_normalizer, vocoder)\n",
    "pwg_inference.eval()\n",
    "print(\"Parallel WaveGAN done!\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 调用声码器"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "shepe of wav (time x n_channels):[108600, 1]\n"
     ]
    }
   ],
   "source": [
    "with paddle.no_grad():\n",
    "    wav = pwg_inference(mel)\n",
    "print(\"shepe of wav (time x n_channels):%s\"%wav.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAoAAAAGoCAYAAADW2lTlAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjAsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+17YcXAAAgAElEQVR4nOzdd5gUVdYG8PcQh5xB8pCRHAaUKCIqiopZTGvOrnF1MbtGdHWNrGv6zDmjoChBUEBgJCeJI1FyjhPu90d3Q09Ph6qucKur3t/zzMNMT0/VYabDqRvOEaUUiIiIiCg4SukOgIiIiIjcxQSQiIiIKGCYABIREREFDBNAIiIiooBhAkhEREQUMEwAiYiIiAKGCSARkcNEpI+ILBORPSJypu54iIiYABKRb4nIPSLyfcxtyxLcNszBUB4B8LJSqrJS6msHz0NEZAgTQCLys8kAeotIaQAQkfoAygLoGnNby/B9ndIUwMJ0flBEytgcCxERE0Ai8rWZCCV8XcJf9wMwEcAfMbetUEqtF5EXRGSNiOwSkd9FpB8AiEgDEdkvIjUjBxaRriKyRUTKhr++UkQWi8h2ERkrIk3Dt68A0BzAt+Ep4PLh440SkW0islxErok67sMi8rmIvC8iuwBcHr7ts/Btu0Vkvoi0Do9wbgrHfJKjv0ki8hUmgETkW0qpQwCmA+gfvqk/gF8A/BpzW2T0byZCiWFNAB8C+ExEspRS6wFMA3BO1OEvAvC5UipfRIYCuBfA2QDqhM/xUTiGFgBWAzg9PAV8EMDHANYCaADgXABPiMjAqGMPBfA5gOoAPgjfdjqA9wDUADAbwFiEXsMbIjTF/Gp6vyUiCiImgETkd5NwJNnrh1By9kvMbZMAQCn1vlJqq1KqQCn1LIDyANqE7/chgAsBQEQEwLDwbQBwPYAnlVKLlVIFAJ4A0CUyChhNRBoD6APgn0qpA0qpOQDeAPC3qLtNU0p9rZQqUkrtD9/2i1JqbPj4nyGUaI5QSuUjlFBmi0j1dH9JRBQsTACJyO8mA+gbnr6to5RaBmAqQmsDawLoEL4PROQf4WncnSKyA0A1ALXDx/kCQK/wmsH+AIoQSiSB0Bq/F0RkR/jntgEQhEbnYjUAsE0ptTvqtj9j7rsmzs9tjPp8P4AtSqnCqK8BoHKyXwQRUQQXFxOR301DKJG7BsAUAFBK7RKR9eHb1iulVoXX+90N4AQAC5VSRSKyHaFEDkqp7SLyI4ALABwN4GOllAqfYw2Ax5VSHyC19QBqikiVqCSwCYB1UfdRJX+MiMg+HAEkIl8LT6HmArgDR0bsgNA6wDtwZP1fFQAFADYDKCMiDwKoGnO4DxGaqj0XR6Z/AeB/AO4RkfYAICLVROS8BPGsQWgE8kkRyRKRTgCuAvB+2v9JIiKTmAASURBMAlAXoaQv4pfwbZEEcCyAHwAsRWhK9gBKTsWOAtAKwF9KqbmRG5VSXwF4CsDH4Z27CwCckiSeCwFkIzQa+BWAh5RS49L5jxERpUOOzGAQERERURBwBJCIiIgoYJgAEhEREQUME0AiIiKigGECSERERBQwvqsDWLt2bZWdna07DCIiIiLtfv/99y1KqTqxt/suAczOzkZubq7uMIiIiIi0E5E/493OKWAiIiKigGECSERERBQwTACJiIiIAoYJIBEREVHAMAEkIiIiChgmgEREREQBwwSQiIiIKGCYABIREREFDBNAIiIiooBhAkhEREQUMEwAiYiIiAKGCSARERFRwDABJCIiIgoYJoBEREREAcMEkIgcs3zTbpzzylTsPVigOxQiIorCBJCIHDPi+z/w+5/bMWX5Ft2hEBFRFCaARERERAHDBJCIiIgoYJgAEhEREQUME0AiIiKigGEC6GNb9hzUHQIRERF5EBNAn/p12RbkPDYOPy3aqDsUIiIi8hgmgD41d+0OAMCs1ds1R0JBNm4xL0CIiLyICSARERFRwDAB9LmxC/7SHQIRERF5DBNAn1u5Za/uEIiIiMhjmAASERERBQwTwADYtOuA7hAo4JTuAIiIqBgmgAFw04ezdIdAREREHsIE0Kde/2Xl4c93HyjQGAkRERF5DRNAn9qxL193CESHPfXDEt0hEBFRFK0JoIgMFpE/RGS5iAxPcJ/zRWSRiCwUkQ/djpGIrFu5mbvRiYi8pIyuE4tIaQAjAZwIYC2AmSIySim1KOo+rQDcA6CPUmq7iNTVEy0RERGRf+gcAewJYLlSaqVS6hCAjwEMjbnPNQBGKqW2A4BSapPLMRIRERH5js4EsCGANVFfrw3fFq01gNYiMkVEfhORwfEOJCLXikiuiORu3rzZoXCJiIiI/MHrm0DKAGgFYACACwG8LiLVY++klHpNKZWjlMqpU6eOyyESERERZRadCeA6AI2jvm4Uvi3aWgCjlFL5SqlVAJYilBCSCUv+2q07BCIiIvIQnQngTACtRKSZiJQDMAzAqJj7fI3Q6B9EpDZCU8IrQURERERp05YAKqUKANwMYCyAxQA+VUotFJFHROSM8N3GAtgqIosATARwl1Jqq56IiYiIiPxBWxkYAFBKjQEwJua2B6M+VwDuCH8QERERkQ28vgmEiIiIiGzGBJCIiIgoYJgAEpEj1u/YrzsEIiJKgAkgETliwDM/6w6BiIgSYAIYEAvW7dQdAgXMoYKiYl/vOVigKRIiIorFBDAgTnvpV90hUMDd/OEs3SEQEVEYE0AicsXC9bt0h0BERGFMAImIiIgChgmgzyxavwvZw0fH/V5BYVHc24mIiChYmAD6zMsTlyX83lez17kYCREREXkVE8AA2Z9fqDsECohQF0ciIvIqJoAB8uA3C3WHQAExaelm3SEQEVESTACJyHY79+frDoGIiJJgAkhEREQUMEwAiYiIiAKGCSARuYL7QoiIvIMJIBG5Ysueg7pDICKiMCaAROSaJX+xHRwRkRcwASQi1+zaX6A7BCIiAhNAIiIiosBhAhgwq7bs1R0CERERacYEMGAuf2uG7hBcN37xRsxfu1N3GERERJ5RRncA5K6CwmDV4li2cTeueicXAJA3YojmaIJDRHSHQERESXAEkHztxOcm6w4hkG75aLbuEIiIKAkmgAGzbsd+7DnInZhERERBxgQwgDbuOqA7BCIiItKICSARuUaxHxwRkScwASQiIiIKGCaAPrNs4x7dIRAREZHHMQH0mWWbmAASERFRckwAiYiIiAKGCWAA/bXzAIqKuBifiIgoqJgABtDFb0zHK5NW6A6DiIiINGECGFBTlm/RHQIRERFpwgQwoHYfYDcQct/kZZt1h0BERGACGFjz1+3EgfxC3WFQwIycyKUHRERewAQwwA7mF+kOwVFrt+/THUIgLflrl+4QiIgoBSaA5Fux6xzZA9kdb/yySncIRESUAhNACownxizWHQIREZEnMAEkIiIiChgmgEREREQBwwSQiIiIKGCYABIREREFDBPAANt9MF93CERERKSB1gRQRAaLyB8islxEhie53zkiokQkx834/K7vUxN1h+AogegOgYiIyJO0JYAiUhrASACnAGgH4EIRaRfnflUA3ApgursREhEREfmTzhHAngCWK6VWKqUOAfgYwNA493sUwFMAWMWXLFFKdwTBwHFXIiLv05kANgSwJurrteHbDhORbgAaK6VGJzuQiFwrIrkikrt5M5vNExERESXj2U0gIlIKwH8A3Jnqvkqp15RSOUqpnDp16jgfHBEREVEG05kArgPQOOrrRuHbIqoA6ADgZxHJA3AsgFHcCEJERERkjc4EcCaAViLSTETKARgGYFTkm0qpnUqp2kqpbKVUNoDfAJyhlMrVEy5lurlrd+gOgYiIyBO0JYBKqQIANwMYC2AxgE+VUgtF5BEROUNXXOQjMbsR/ty6T08cVExhEXfjEBHppnUNoFJqjFKqtVKqhVLq8fBtDyqlRsW57wCO/pEZI75fojsEiuOeL+fpDkGLHfsO4Z4v5+FAfqHuUIiIvLsJhMiqbXsP6Q6B4vj897W6Q9DiuZ+W4qMZa/Bp7prUdyYichgTQCIihymlcKiQU99
      "text/plain": [
       "<Figure size 648x432 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# 绘制声码器输出的波形图\n",
    "wave_data = wav.numpy().T\n",
    "time = np.arange(0, wave_data.shape[1]) * (1.0 / fastspeech2_config.fs)\n",
    "fig, ax = plt.subplots(figsize=(9, 6))\n",
    "plt.plot(time, wave_data[0])\n",
    "plt.title('Waveform')\n",
    "plt.xlabel('Time (seconds)')\n",
    "plt.ylabel('Amplitude (normed)')\n",
    "plt.tight_layout()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 播放音频"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "                <audio controls=\"controls\" >\n",
       "                    <source src=\"data:audio/wav;base64,UklGRpRQAwBXQVZFZm10IBAAAAABAAEAwF0AAIC7AAACABAAZGF0YXBQAwDS/87/z//P/8//0//Y/9f/2f/Z/9f/2P/h/+P/4v/j/+L/4//m/+r/6//q/+3/7P/s/+r/6//s/+3/7f/t/+//8P/w/+//8f/x//H/8v/y//T/9P/0//T/9f/2//T/9f/2//b/9//2//b/9//2//b/9//3//f/9//4//f/9f/3/wAA///+//7//v////7////+//z//P/8//v/+//7//z/+//8//z/+//9//z//P/8//v//P/8//v//f/8//3//P/9//3//f/+//3//v/+/////v//////AAAAAAEAAQABAAEAAgACAAIAAwACAAIAAgACAAEAAgABAAIAAQACAAEAAAAAAAAA//8AAAAA/v/+//z//P/7//v/+v/7//j/+f/4//n/9//4//f/9//4//j/9//3//f/9//2//b/9v/2//X/9f/2//b/9P/2//X/9v/0//T/9P/0//T/9P/0//b/9P/z//X/9P/z//P/9P/z//L/8//z//T/8//y//P/8v/1//X/9f/2//T/8//z//L/8v/y//L/8v/y//L/8v/z//L/8//y//P/8f/y//L/8v/z//P/9P/0//T/8//y//T/8//0//T/9P/z//T/9f/0//X/9f/1//T/9v/1//b/9v/2//f/+P/4//r/+f/5//r/+//7//v/+//7//r//P/9//z//f/+////////////AAD9///////9/////v////7///8AAP//AAAAAAAAAAAAAAAAAAD//wAAAAABAAAAAAD//wAA//8AAP7////+/wAA/v/+/wAAAAD//wAA//8AAP//AAD+////AAD///////8AAP7//v////7//f/+//7//f/+//3//v/9//7//f/9//3//f/9//7//f/9//3//f/7//v/+//8//v/+//8//z/+v/8//r/+v/6//r/+v/5//r/+f/6//j/+v/5//r/+v/8//r/+//6//n/+v/6//n/+v/5//r/+v/6//v/+//5//v/+v/6//r/+f/5//r/+//6//v/+//6//v/+//6//r/+v/7//v/+v/6//r/+//6//r//P/8//z//P/7//z//f/8//7//v/+/////v8AAP3//v////3//v////////8AAAAAAAAAAAAAAAABAAAAAgABAAIAAQACAAEAAAABAAEAAAAAAAIAAQACAAEAAQABAAAAAAAAAAAAAAAAAAAAAAABAAAAAAABAAAAAAD//wAAAAAAAAAAAAAAAP//AAABAAEAAQAAAAAA//8BAAAAAQABAAEAAgACAAMAAwACAAMAAQADAAIAAgABAAEAAQACAAIAAgACAAEAAQABAAAAAAAAAAAAAAAAAAAAAAAAAP7//v/9//z//P/7//v/+f/6//f/+P/4//j/+P/5//f/9//5//r/+f/4//f/+P/3//j/+P/4//j/+f/5//j/9//5//b/+P/4//n/+P/4//j/+P/4//f/9//3//j/9//4//f/9//3//f/9v/0//X/9f/1//X/8//0//X/9f/1//b/9f/2//f/9//3//b/9//3//n/+P/5//n/+v/4//j/+f/5//n/+f/6//j/+v/5//n/+P/7//n/+//6//v/+f/5//j/+P/4//j/9//2//j/9//2//f/+P/4//f/+P/3//b/+P/0//b/9//2//b/9//4//f/9//5//f/9//4//f/+P/5//f/+f/5//n/+f/6//v/+//7//v/+//8//v//P/7//r//f/9//3//P/+//3/+//9//3//f/8//7//f/+//3//f/9//3//P/8//3//f/9//z//v/8//z//P/7//z/+//8//v//P/8//v//f/9//3/AAD+//7////8//3////9//v//v/+//z//P/+//7////+///////+//3//v/+/////////wAA///9///////9//7///////7///8AAAAAAAD////////9/wAA//8AAP//AAAAAP//AAAAAAAAAAAAAAAAAAABAAAAAAAAAAEAAAACAAEAAAABAAEAAQABAAEAAQAAAAIAAwACAAEAAwADAAMAAgADAAMAAwAEAAQABQADAAIABAACAAAAAAAAAAAAAAD//wEAAQAAAAAAAgABAAEAAQADAAIAAgABAAIAAwD//wAAAgACAAAAAAAEAAIAAgACAAIAAQACAAIAAgABAAQAAwACAAQAAgACAAMAAgACAAIAAAADAAMAAgACAAMAAgABAAAAAgACAAAAAQACAAAAAQADAAIAAQADAAMAAAAAAAAAAwABAAEABAADAAEABAABAAMAAwADAAMABAABAAQAAwACAAEAAwACAAIAAgABAAAAAQD/////AAD9/wAA/f/9//v//f/8//n/+//6//n/+f/5//j/+v/6//n//v/2//z/+//9//v/9//9//7/+v/5//r/+f/2//3/+v/4////9f/9//r/9v/6//r//f/5//n//P/9//r////7//3////6//r/+v/7//n/+f/3//f/+v/1//b/9//7//n/9//5//f/+f/3//X/9f/2//T/9//2//T/9//3//T/8//5//T/9v/2//P/+P/2//T/+P/3//P/+//3//b/+P/2//n/9P/3//X/+P/4//X/9v/4//j/8//3//T/9v/2//L/9v/1//T/9f/1//X/8v/1//P/8f/1//H/9P/x//L/8f/2//D/8//2//L/9v/x//P/8v/w//T/7P/1/+z/9f/2/+z/9v/y//j/7v/6//T/+f////T////7//n/+v8AAPz////+//z/AQD1/wEA+f/+/wQA/P8FAP//BAD6/wIAAQD5/wMA+P/8//r/9v/6//f/+v/0//7/+f/2////+P/9//n/+v/5//X/+//0//r/+v/1//z/9/8AAPf/9v////f/9v/6//f/8//8//H/+P////H//f/8//r/+f/8//z/+//8//X//f/3//j/9f/7//j/7/////D/9//2/+///f/z//X/9f/6//b/9f/4//T/9v/0//f/9P/3//D/9//2/+7/9//y//T/9//y//X/+P/y//T/9//w//n/9P/y//b/8P/6//X/9v/4//H/+P/y/+//9P/y/+//8f/z/+//9f/v/+//+f/y//X//P/1////9//5//3/9P8CAP//BAACAAIACQAHAAEADAAOAAIADwACAAsADgAAABUACQAKAA4ABAARAAwACwARABIAFQAQABUAHAAPABkAGQATACAACgATABgAEAAYAAoAEwAPAA0ACwAKABMACAANABAADAAMAAkADAADAAYAAAAJAAcA+/8HAAEABgABAAIAAAADAAAA/P8LAPn/BQABAP//CAD+/wYAAAAHAAYAAwAFAAcACQAFAAsACgAIABAADgAKABIADQAVABQADgANAA8ADAAKABMACQASABMAEwANAA4AEQAAABcA/P8EAAYA7v8XAO7/BwAAAPH/BwDx/wAA9P8BAPj/+f/6//D/9f/y//j/9P/4//T/+P/4/+7/+P/1/+z/+//q//j//v/n/wkA9f/5/wUA8f8IAPz/+v////b/AADv//P/+v/r//r/8v/v//z/5f/7//f/8v/8//L/9P/v/+//8f/x/+n/7f/n/+3/5v/q/+//6P/q/+j/5//l/+f/5P/n/+H/3P/q/+D/4f/d/+H/5f/h/+H/3v/k/+D/2//h/+L/2v/i/+f/1f/i/9n/1P/j/7//zP/R/8j/yf/O/8P/qf+m/6f/kf+c/4P/iv95/3v/fv91/3//c/+A/33/ef+I/3//lv+T/5v/nf+t/8j/w//a/+H/+/8IABYAKgA9AFIAXgB0AJAAkQC3ALYAyADdANYA+AD5AAQBEwEfASsBOAE1AT8BWgFQAV0BUQFNAVgBKwFIARsBGAH+ANEA2QCTAI4ATgAjAAAApv+R/0//EP/w/pr+fP5Q/v798f2i/XD9Sv3p/Mr
       "                    Your browser does not support the audio element.\n",
       "                </audio>\n",
       "              "
      ],
      "text/plain": [
       "<IPython.lib.display.Audio object>"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dp.Audio(wav.numpy().T, rate=fastspeech2_config.fs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 保存音频"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "mkdir: cannot create directory 'output': File exists\r\n"
     ]
    }
   ],
   "source": [
    "!mkdir output\n",
    "sf.write(\n",
    "    \"output/output.wav\",\n",
    "    wav.numpy(),\n",
    "    samplerate=fastspeech2_config.fs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 进阶 —— 个性化调节\n",
    "FastSpeech2 模型可以个性化地调节音素时长、音调和能量，通过一些简单的调节就可以获得一些有意思的效果。\n",
    "\n",
    "例如对于以下的原始音频`\"凯莫瑞安联合体的经济崩溃，迫在眉睫\"`。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "                <audio controls=\"controls\" >\n",
       "                    <source src=\"https://paddlespeech.bj.bcebos.com/Parakeet/docs/demos/speed/x1_001.wav\" type=\"audio/x-wav\" />\n",
       "                    Your browser does not support the audio element.\n",
       "                </audio>\n",
       "              "
      ],
      "text/plain": [
       "<IPython.lib.display.Audio object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# 原始音频\n",
    "dp.display(dp.Audio(url=\"https://paddlespeech.bj.bcebos.com/Parakeet/docs/demos/speed/x1_001.wav\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "                <audio controls=\"controls\" >\n",
       "                    <source src=\"https://paddlespeech.bj.bcebos.com/Parakeet/docs/demos/speed/x1.2_001.wav\" type=\"audio/x-wav\" />\n",
       "                    Your browser does not support the audio element.\n",
       "                </audio>\n",
       "              "
      ],
      "text/plain": [
       "<IPython.lib.display.Audio object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# speed x 1.2\n",
    "dp.display(dp.Audio(url=\"https://paddlespeech.bj.bcebos.com/Parakeet/docs/demos/speed/x1.2_001.wav\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "                <audio controls=\"controls\" >\n",
       "                    <source src=\"https://paddlespeech.bj.bcebos.com/Parakeet/docs/demos/speed/x0.8_001.wav\" type=\"audio/x-wav\" />\n",
       "                    Your browser does not support the audio element.\n",
       "                </audio>\n",
       "              "
      ],
      "text/plain": [
       "<IPython.lib.display.Audio object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# speed x 0.8\n",
    "dp.display(dp.Audio(url=\"https://paddlespeech.bj.bcebos.com/Parakeet/docs/demos/speed/x0.8_001.wav\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "                <audio controls=\"controls\" >\n",
       "                    <source src=\"https://paddlespeech.bj.bcebos.com/Parakeet/docs/demos/child_voice/001.wav\" type=\"audio/x-wav\" />\n",
       "                    Your browser does not support the audio element.\n",
       "                </audio>\n",
       "              "
      ],
      "text/plain": [
       "<IPython.lib.display.Audio object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# pitch x 1.3(童声)\n",
    "dp.display(dp.Audio(url=\"https://paddlespeech.bj.bcebos.com/Parakeet/docs/demos/child_voice/001.wav\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "                <audio controls=\"controls\" >\n",
       "                    <source src=\"https://paddlespeech.bj.bcebos.com/Parakeet/docs/demos/robot/001.wav\" type=\"audio/x-wav\" />\n",
       "                    Your browser does not support the audio element.\n",
       "                </audio>\n",
       "              "
      ],
      "text/plain": [
       "<IPython.lib.display.Audio object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# robot\n",
    "dp.display(dp.Audio(url=\"https://paddlespeech.bj.bcebos.com/Parakeet/docs/demos/robot/001.wav\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "具体实现代码请参考 [Style FastSpeech2](https://github.com/PaddlePaddle/PaddleSpeech/tree/develop/demos/style_fs2)。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 用 PaddleSpeech 训练 TTS 模型\n",
    "PaddleSpeech 的 examples 是按照 数据集/模型 的结构安排的:\n",
    "```text\n",
    "examples   \n",
    "├── aishell3\n",
    "│   ├── README.md\n",
    "│   ├── tts3\n",
    "│   └── vc0\n",
    "├── csmsc\n",
    "│   ├── README.md\n",
    "│   ├── tts2\n",
    "│   ├── tts3\n",
    "│   ├── voc1\n",
    "│   └── voc3\n",
    "├── ...\n",
    "└── ...\n",
    "```\n",
    "我们在每个数据集的 README.md 介绍了子目录和模型的对应关系, 在 TTS 中有如下对应关系:\n",
    "```text\n",
    "tts0 - Tactron2\n",
    "tts1 - TransformerTTS\n",
    "tts2 - SpeedySpeech\n",
    "tts3 - FastSpeech2\n",
    "voc0 - WaveFlow\n",
    "voc1 - Parallel WaveGAN\n",
    "voc2 - MelGAN\n",
    "voc3 - MultiBand MelGAN\n",
    "```\n",
    "### 基于 CSMCS 数据集训练 FastSpeech2 模型\n",
    "```bash\n",
    "git clone https://github.com/PaddlePaddle/PaddleSpeech.git\n",
    "cd examples/csmsc/tts\n",
    "```\n",
    "根据 README.md, 下载 CSMCS 数据集和其对应的强制对齐文件, 并放置在对应的位置\n",
    "```bash\n",
    "./run.sh\n",
    "```\n",
    "`run.sh` 中包含预处理、训练、合成、静态图推理等步骤:\n",
    "\n",
    "```bash\n",
    "#!/bin/bash\n",
    "set -e\n",
    "source path.sh\n",
    "gpus=0,1\n",
    "stage=0\n",
    "stop_stage=100\n",
    "conf_path=conf/default.yaml\n",
    "train_output_path=exp/default\n",
    "ckpt_name=snapshot_iter_153.pdz\n",
    "\n",
    "# with the following command, you can choice the stage range you want to run\n",
    "# such as `./run.sh --stage 0 --stop-stage 0`\n",
    "# this can not be mixed use with `$1`, `$2` ...\n",
    "source ${MAIN_ROOT}/utils/parse_options.sh || exit 1\n",
    "\n",
    "if [ ${stage} -le 0 ] && [ ${stop_stage} -ge 0 ]; then\n",
    "    # prepare data\n",
    "    bash ./local/preprocess.sh ${conf_path} || exit -1\n",
    "fi\n",
    "if [ ${stage} -le 1 ] && [ ${stop_stage} -ge 1 ]; then\n",
    "    # train model, all `ckpt` under `train_output_path/checkpoints/` dir\n",
    "    CUDA_VISIBLE_DEVICES=${gpus} ./local/train.sh ${conf_path} ${train_output_path} || exit -1\n",
    "fi\n",
    "if [ ${stage} -le 2 ] && [ ${stop_stage} -ge 2 ]; then\n",
    "    # synthesize, vocoder is pwgan\n",
    "    CUDA_VISIBLE_DEVICES=${gpus} ./local/synthesize.sh ${conf_path} ${train_output_path} ${ckpt_name} || exit -1\n",
    "fi\n",
    "if [ ${stage} -le 3 ] && [ ${stop_stage} -ge 3 ]; then\n",
    "    # synthesize_e2e, vocoder is pwgan\n",
    "    CUDA_VISIBLE_DEVICES=${gpus} ./local/synthesize_e2e.sh ${conf_path} ${train_output_path} ${ckpt_name} || exit -1\n",
    "fi\n",
    "if [ ${stage} -le 4 ] && [ ${stop_stage} -ge 4 ]; then\n",
    "    # inference with static model\n",
    "    CUDA_VISIBLE_DEVICES=${gpus} ./local/inference.sh ${train_output_path} || exit -1\n",
    "fi\n",
    "```\n",
    "\n",
    "### 基于 CSMCS 数据集训练 Parallel WaveGAN 模型\n",
    "```bash\n",
    "git clone https://github.com/PaddlePaddle/PaddleSpeech.git\n",
    "cd examples/csmsc/voc1\n",
    "```\n",
    "根据 README.md, 下载 CSMCS 数据集和其对应的强制对齐文件, 并放置在对应的位置\n",
    "```bash\n",
    "./run.sh\n",
    "```\n",
    "`run.sh` 中包含预处理、训练、合成等步骤:\n",
    "```bash\n",
    "#!/bin/bash\n",
    "set -e\n",
    "source path.sh\n",
    "gpus=0,1\n",
    "stage=0\n",
    "stop_stage=100\n",
    "conf_path=conf/default.yaml\n",
    "train_output_path=exp/default\n",
    "ckpt_name=snapshot_iter_5000.pdz\n",
    "\n",
    "# with the following command, you can choice the stage range you want to run\n",
    "# such as `./run.sh --stage 0 --stop-stage 0`\n",
    "# this can not be mixed use with `$1`, `$2` ...\n",
    "source ${MAIN_ROOT}/utils/parse_options.sh || exit 1\n",
    "\n",
    "if [ ${stage} -le 0 ] && [ ${stop_stage} -ge 0 ]; then\n",
    "    # prepare data\n",
    "    ./local/preprocess.sh ${conf_path} || exit -1\n",
    "fi\n",
    "if [ ${stage} -le 1 ] && [ ${stop_stage} -ge 1 ]; then\n",
    "    # train model, all `ckpt` under `train_output_path/checkpoints/` dir\n",
    "    CUDA_VISIBLE_DEVICES=${gpus} ./local/train.sh ${conf_path} ${train_output_path} || exit -1\n",
    "fi\n",
    "if [ ${stage} -le 2 ] && [ ${stop_stage} -ge 2 ]; then\n",
    "    # synthesize\n",
    "    CUDA_VISIBLE_DEVICES=${gpus} ./local/synthesize.sh ${conf_path} ${train_output_path} ${ckpt_name} || exit -1\n",
    "fi\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# FAQ\n",
    "\n",
    "- 需要注意的问题\n",
    "- 经验与分享\n",
    "- 用户的其他问题"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 作业\n",
    "在 CSMSC 数据集上利用 FastSpeech2 和 Parallel WaveGAN 实现一个中文 TTS 系统。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 关注 PaddleSpeech\n",
    "请关注我们的 [Github Repo](https://github.com/PaddlePaddle/PaddleSpeech/)，非常欢迎加入以下微信群参与讨论：\n",
    "- 扫描右侧二维码\n",
    "- 添加运营小姐姐微信\n",
    "- 通过后回复【S】\n",
    "- 系统自动邀请加入技术群\n",
    "<center><img src=\"https://ai-studio-static-online.cdn.bcebos.com/bca0bc75dce14b53af44e374e64fc91aeeb13c075c894d6aabed033148f65377\" ></center>\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.7.0 64-bit ('yt_py37_develop': venv)",
   "language": "python",
   "name": "python37064bitytpy37developvenv88cd689abeac41d886f9210a708a170b"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.0"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}