23 KiB

Raw Permalink Blame History

Text to speech - Wio Terminal

For dis part of di lesson, you go change text to speech so e go fit give spoken feedback.

Text to speech

Di speech services SDK wey you use for di last lesson to change speech to text fit also change text back to speech.

Get list of voices

When you dey request speech, you go need provide di voice wey you wan use because speech fit dey generate with different voices. Each language get different voices wey e support, and you fit get di list of voices wey dey available for each language from di speech services SDK. But di microcontroller get limitation - di call wey go get di list of voices wey text to speech services support na JSON document wey big reach 77KB, e too big to process for Wio Terminal. As at di time wey dem write dis, di full list get 215 voices, each one dey define with JSON document like dis:

{
    "Name": "Microsoft Server Speech Text to Speech Voice (en-US, AriaNeural)",
    "DisplayName": "Aria",
    "LocalName": "Aria",
    "ShortName": "en-US-AriaNeural",
    "Gender": "Female",
    "Locale": "en-US",
    "StyleList": [
        "chat",
        "customerservice",
        "narration-professional",
        "newscast-casual",
        "newscast-formal",
        "cheerful",
        "empathetic"
    ],
    "SampleRateHertz": "24000",
    "VoiceType": "Neural",
    "Status": "GA"
}

Dis JSON na for di Aria voice, wey get plenty voice styles. Wetin you need when you dey change text to speech na di shortname, en-US-AriaNeural.

Instead make you download and decode di whole list for your microcontroller, you go need write serverless code wey go retrieve di list of voices for di language wey you dey use, and call am from your Wio Terminal. Your code fit then pick di correct voice from di list, like di first one wey e see.

Task - create serverless function to get list of voices

Open your smart-timer-trigger project for VS Code, and open di terminal make sure say di virtual environment dey activated. If e no dey, kill di terminal and create am again.
Open di local.settings.json file and add di settings for di speech API key and location:
```
"SPEECH_KEY": "<key>",
"SPEECH_LOCATION": "<location>"
```
Replace <key> with di API key for your speech service resource. Replace <location> with di location wey you use when you create di speech service resource.
Add new HTTP trigger to dis app wey dem call get-voices using dis command from inside di VS Code terminal for di root folder of di functions app project:
```
func new --name get-voices --template "HTTP trigger"
```
Dis one go create HTTP trigger wey dem call get-voices.

Replace di content of di __init__.py file for di get-voices folder with dis:

import json
import os
import requests

import azure.functions as func

def main(req: func.HttpRequest) -> func.HttpResponse:
    location = os.environ['SPEECH_LOCATION']
    speech_key = os.environ['SPEECH_KEY']

    req_body = req.get_json()
    language = req_body['language']

    url = f'https://{location}.tts.speech.microsoft.com/cognitiveservices/voices/list'

    headers = {
        'Ocp-Apim-Subscription-Key': speech_key
    }

    response = requests.get(url, headers=headers)
    voices_json = json.loads(response.text)

    voices = filter(lambda x: x['Locale'].lower() == language.lower(), voices_json)
    voices = map(lambda x: x['ShortName'], voices)

    return func.HttpResponse(json.dumps(list(voices)), status_code=200)

Dis code dey make HTTP request to di endpoint wey go get di voices. Di voices list na big JSON block wey get voices for all languages, so di voices for di language wey dem pass for di request body go dey filter out, then di shortname go dey extract and return as JSON list. Di shortname na di value wey you need to change text to speech, so na only dis value dem go return.

💁 You fit change di filter as you like to select di voices wey you want.

Dis one go reduce di size of di data from 77KB (as at di time wey dem write am), to smaller JSON document. Example, for US voices na 408 bytes.

Run your function app locally. You fit then call am using tool like curl di same way wey you test your text-to-timer HTTP trigger. Make sure say you pass your language as JSON body:
```
{
    "language":"<language>"
}
```
Replace <language> with your language, like en-GB, or zh-CN.

💁 You fit find dis code for di code-spoken-response/functions folder.

Task - retrieve di voice from your Wio Terminal

Open di smart-timer project for VS Code if e never open.
Open di config.h header file and add di URL for your function app:
```
const char *GET_VOICES_FUNCTION_URL = "<URL>";
```
Replace <URL> with di URL for di get-voices HTTP trigger for your function app. Dis one go be di same as di value for TEXT_TO_TIMER_FUNCTION_URL, but di function name go be get-voices instead of text-to-timer.
Create new file for di src folder wey dem call text_to_speech.h. Dis one go dey use to define class wey go change text to speech.

Add di following include directives to di top of di new text_to_speech.h file:

#pragma once

#include <Arduino.h>
#include <ArduinoJson.h>
#include <HTTPClient.h>
#include <Seeed_FS.h>
#include <SD/Seeed_SD.h>
#include <WiFiClient.h>
#include <WiFiClientSecure.h>

#include "config.h"
#include "speech_to_text.h"

Add di following code below dis to declare di TextToSpeech class, along with instance wey fit dey use for di rest of di application:
```
class TextToSpeech
{
public:
private:
};

TextToSpeech textToSpeech;
```
To call your functions app, you go need declare WiFi client. Add dis one to di private section of di class:
```
WiFiClient _client;
```
For di private section, add field for di selected voice:
```
String _voice;
```
For di public section, add init function wey go get di first voice:
```
void init()
{
}
```
To get di voices, JSON document go need dey send to di function app with di language. Add dis code to di init function to create dis JSON document:
```
DynamicJsonDocument doc(1024);
doc["language"] = LANGUAGE;

String body;
serializeJson(doc, body);
```

Next create HTTPClient, then use am call di functions app to get di voices, post di JSON document:

HTTPClient httpClient;
httpClient.begin(_client, GET_VOICES_FUNCTION_URL);

int httpResponseCode = httpClient.POST(body);

Below dis add code to check di response code, and if e be 200 (success), then extract di list of voices, retrieve di first one from di list:

if (httpResponseCode == 200)
{
    String result = httpClient.getString();
    Serial.println(result);

    DynamicJsonDocument doc(1024);
    deserializeJson(doc, result.c_str());

    JsonArray obj = doc.as<JsonArray>();
    _voice = obj[0].as<String>();

    Serial.print("Using voice ");
    Serial.println(_voice);
}
else
{
    Serial.print("Failed to get voices - error ");
    Serial.println(httpResponseCode);
}

After dis, end di HTTP client connection:
```
httpClient.end();
```
Open di main.cpp file, and add di following include directive for di top to include dis new header file:
```
#include "text_to_speech.h"
```
For di setup function, under di call to speechToText.init();, add dis one to initialize di TextToSpeech class:
```
textToSpeech.init();
```

Build dis code, upload am to your Wio Terminal and test am through di serial monitor. Make sure say your function app dey run.

You go see di list of available voices wey di function app return, along with di selected voice.

--- Available filters and text transformations: colorize, debug, default, direct, hexlify, log2file, nocontrol, printable, send_on_enter, time
--- More details at http://bit.ly/pio-monitor-filters
--- Miniterm on /dev/cu.usbmodem1101  9600,8,N,1 ---
--- Quit: Ctrl+C | Menu: Ctrl+T | Help: Ctrl+T followed by Ctrl+H ---
Connecting to WiFi..
Connected!
Got access token.
["en-US-JennyNeural", "en-US-JennyMultilingualNeural", "en-US-GuyNeural", "en-US-AriaNeural", "en-US-AmberNeural", "en-US-AnaNeural", "en-US-AshleyNeural", "en-US-BrandonNeural", "en-US-ChristopherNeural", "en-US-CoraNeural", "en-US-ElizabethNeural", "en-US-EricNeural", "en-US-JacobNeural", "en-US-MichelleNeural", "en-US-MonicaNeural", "en-US-AriaRUS", "en-US-BenjaminRUS", "en-US-GuyRUS", "en-US-ZiraRUS"]
Using voice en-US-JennyNeural
Ready.

Change text to speech

Once you get voice wey you wan use, e fit dey use to change text to speech. Di same memory limitation wey dey with voices also dey apply when you dey change speech to text, so you go need write di speech to SD card make e ready to play over di ReSpeaker.

💁 For di earlier lessons for dis project, you use flash memory to store speech wey microphone capture. Dis lesson dey use SD card because e dey easier to play audio from am using di Seeed audio libraries.

Another limitation dey wey you go need consider, na di available audio data from di speech service, and di formats wey di Wio Terminal dey support. Unlike full computers, audio libraries for microcontrollers fit dey very limited for di audio formats wey dem dey support. Example, di Seeed Arduino Audio library wey fit play sound over di ReSpeaker only dey support audio wey get 44.1KHz sample rate. Di Azure speech services fit provide audio for plenty formats, but none of dem dey use dis sample rate, dem only dey provide 8KHz, 16KHz, 24KHz and 48KHz. Dis mean say di audio go need re-sample to 44.1KHz, something wey go need more resources wey di Wio Terminal no get, especially memory.

When you need manipulate data like dis, e better make you use serverless code, especially if di data dey come through web call. Di Wio Terminal fit call serverless function, pass di text wey e wan change, and di serverless function fit call di speech service to change text to speech, as well as re-sample di audio to di sample rate wey e need. E fit then return di audio for di form wey di Wio Terminal need to store am for di SD card and play am over di ReSpeaker.

Task - create serverless function to change text to speech

Open your smart-timer-trigger project for VS Code, and open di terminal make sure say di virtual environment dey activated. If e no dey, kill di terminal and create am again.
Add new HTTP trigger to dis app wey dem call text-to-speech using dis command from inside di VS Code terminal for di root folder of di functions app project:
```
func new --name text-to-speech --template "HTTP trigger"
```
Dis one go create HTTP trigger wey dem call text-to-speech.
Di librosa Pip package get functions to re-sample audio, so add dis one to di requirements.txt file:
```
librosa
```
Once you don add am, install di Pip packages using dis command from di VS Code terminal:
```
pip install -r requirements.txt
```
⚠️ If you dey use Linux, including Raspberry Pi OS, you fit need install libsndfile with dis command:
```
sudo apt update
sudo apt install libsndfile1-dev
```

To change text to speech, you no fit use di speech API key directly, instead you go need request access token, use di API key to authenticate di access token request. Open di __init__.py file from di text-to-speech folder and replace all di code inside am with dis:

import io
import os
import requests

import librosa
import soundfile as sf
import azure.functions as func

location = os.environ['SPEECH_LOCATION']
speech_key = os.environ['SPEECH_KEY']

def get_access_token():
    headers = {
        'Ocp-Apim-Subscription-Key': speech_key
    }

    token_endpoint = f'https://{location}.api.cognitive.microsoft.com/sts/v1.0/issuetoken'
    response = requests.post(token_endpoint, headers=headers)
    return str(response.text)

Dis one dey define constants for di location and speech key wey dem go read from di settings. E then dey define di get_access_token function wey go retrieve access token for di speech service.

Below dis code, add di following:

playback_format = 'riff-48khz-16bit-mono-pcm'

def main(req: func.HttpRequest) -> func.HttpResponse:
    req_body = req.get_json()
    language = req_body['language']
    voice = req_body['voice']
    text = req_body['text']

    url = f'https://{location}.tts.speech.microsoft.com/cognitiveservices/v1'

    headers = {
        'Authorization': 'Bearer ' + get_access_token(),
        'Content-Type': 'application/ssml+xml',
        'X-Microsoft-OutputFormat': playback_format
    }

    ssml =  f'<speak version=\'1.0\' xml:lang=\'{language}\'>'
    ssml += f'<voice xml:lang=\'{language}\' name=\'{voice}\'>'
    ssml += text
    ssml += '</voice>'
    ssml += '</speak>'

    response = requests.post(url, headers=headers, data=ssml.encode('utf-8'))

    raw_audio, sample_rate = librosa.load(io.BytesIO(response.content), sr=48000)
    resampled = librosa.resample(raw_audio, sample_rate, 44100)

    output_buffer = io.BytesIO()
    sf.write(output_buffer, resampled, 44100, 'PCM_16', format='wav')
    output_buffer.seek(0)

    return func.HttpResponse(output_buffer.read(), status_code=200)

Dis one dey define di HTTP trigger wey dey change di text to speech. E dey extract di text wey e wan change, di language and di voice from di JSON body wey dem set for di request, build some SSML to request di speech, then call di REST API wey dey authenticate using di access token. Dis REST API call dey return di audio wey dem encode as 16-bit, 48KHz mono WAV file, wey di value of playback_format dey define, wey dem send to di REST API call.

Dis one then dey re-sample by librosa from sample rate of 48KHz to sample rate of 44.1KHz, then dis audio dey save to binary buffer wey dem go return.

Run your function app locally, or deploy am to di cloud. You fit then call am using tool like curl di same way wey you test your text-to-timer HTTP trigger. Make sure say you pass di language, voice and text as di JSON body:
```
{
    "language": "<language>",
    "voice": "<voice>",
    "text": "<text>"
}
```
Replace <language> with your language, like en-GB, or zh-CN. Replace <voice> with di voice wey you wan use. Replace <text> with di text wey you wan change to speech. You fit save di output to file and play am with any audio player wey fit play WAV files.

Example, to change "Hello" to speech using US English with di Jenny Neural voice, with di function app wey dey run locally, you fit use dis curl command:
```
curl -X GET 'http://localhost:7071/api/text-to-speech' \
     -H 'Content-Type: application/json' \
     -o hello.wav \
     -d '{
       "language":"en-US",
       "voice": "en-US-JennyNeural",
       "text": "Hello"
     }'
```
Dis one go save di audio to hello.wav for di current directory.

💁 You fit find dis code for di code-spoken-response/functions folder.

Task - retrieve di speech from your Wio Terminal

Open di smart-timer project for VS Code if e never open.
Open di config.h header file and add di URL for your function app:
```
const char *TEXT_TO_SPEECH_FUNCTION_URL = "<URL>";
```
Replace <URL> with di URL for di text-to-speech HTTP trigger for your function app. Dis one go be di same as di value for TEXT_TO_TIMER_FUNCTION_URL, but di function name go be text-to-speech instead of text-to-timer.
Open di text_to_speech.h header file, and add di following method to di public section of di TextToSpeech class:
```
void convertTextToSpeech(String text)
{
}
```
To di convertTextToSpeech method, add di following code to create di JSON wey go send to di function app:
```
DynamicJsonDocument doc(1024);
doc["language"] = LANGUAGE;
doc["voice"] = _voice;
doc["text"] = text;

String body;
serializeJson(doc, body);
```
Dis one dey write di language, voice and text to di JSON document, then serialize am to string.
Below dis, add di following code to call di function app:
```
HTTPClient httpClient;
httpClient.begin(_client, TEXT_TO_SPEECH_FUNCTION_URL);

int httpResponseCode = httpClient.POST(body);
```
Dis one dey create HTTPClient, then make POST request using di JSON document to di text to speech HTTP trigger.

If di call work, di raw binary data wey di function app call return fit dey stream to file for di SD card. Add dis code to do dis:

if (httpResponseCode == 200)
{            
    File wav_file = SD.open("SPEECH.WAV", FILE_WRITE);
    httpClient.writeToStream(&wav_file);
    wav_file.close();
}
else
{
    Serial.print("Failed to get speech - error ");
    Serial.println(httpResponseCode);
}

Dis code dey check di response, and if e be 200 (success), di binary data dey stream to file for di root of di SD Card wey dem call SPEECH.WAV.

For di end of dis method, close di HTTP connection:
```
httpClient.end();
```
Di text wey you wan talk fit now turn to audio. For di main.cpp file, add dis line for di end of di say function to turn di text wey you wan talk to audio:
```
textToSpeech.convertTextToSpeech(text);
```

Task - play audio from your Wio Terminal

E go soon ready

How to carry your functions app go cloud

Di reason why we dey run di functions app for local na because di librosa Pip package for linux get one dependency wey dey need one library wey no dey install by default, and you go need install am before di function app fit run. Function apps no dey use server - you no fit manage server by yourself, so you no fit install dis library before di app go run.

Di way to do am na to carry your functions app go cloud by using Docker container. Dis container go dey deploy by di cloud anytime e need to start new instance of your function app (like if di demand don pass di resources wey dey available, or if di function app never dey use for some time and dem don close am).

You fit find di instructions to set up function app and deploy am with Docker for di create a function on Linux using a custom container documentation on Microsoft Docs.

Once you don deploy am, you fit move your Wio Terminal code to fit access dis function:

Add di Azure Functions certificate to config.h:

const char *FUNCTIONS_CERTIFICATE =
    "-----BEGIN CERTIFICATE-----\r\n"
    "MIIFWjCCBEKgAwIBAgIQDxSWXyAgaZlP1ceseIlB4jANBgkqhkiG9w0BAQsFADBa\r\n"
    "MQswCQYDVQQGEwJJRTESMBAGA1UEChMJQmFsdGltb3JlMRMwEQYDVQQLEwpDeWJl\r\n"
    "clRydXN0MSIwIAYDVQQDExlCYWx0aW1vcmUgQ3liZXJUcnVzdCBSb290MB4XDTIw\r\n"
    "MDcyMTIzMDAwMFoXDTI0MTAwODA3MDAwMFowTzELMAkGA1UEBhMCVVMxHjAcBgNV\r\n"
    "BAoTFU1pY3Jvc29mdCBDb3Jwb3JhdGlvbjEgMB4GA1UEAxMXTWljcm9zb2Z0IFJT\r\n"
    "QSBUTFMgQ0EgMDEwggIiMA0GCSqGSIb3DQEBAQUAA4ICDwAwggIKAoICAQCqYnfP\r\n"
    "mmOyBoTzkDb0mfMUUavqlQo7Rgb9EUEf/lsGWMk4bgj8T0RIzTqk970eouKVuL5R\r\n"
    "IMW/snBjXXgMQ8ApzWRJCZbar879BV8rKpHoAW4uGJssnNABf2n17j9TiFy6BWy+\r\n"
    "IhVnFILyLNK+W2M3zK9gheiWa2uACKhuvgCca5Vw/OQYErEdG7LBEzFnMzTmJcli\r\n"
    "W1iCdXby/vI/OxbfqkKD4zJtm45DJvC9Dh+hpzqvLMiK5uo/+aXSJY+SqhoIEpz+\r\n"
    "rErHw+uAlKuHFtEjSeeku8eR3+Z5ND9BSqc6JtLqb0bjOHPm5dSRrgt4nnil75bj\r\n"
    "c9j3lWXpBb9PXP9Sp/nPCK+nTQmZwHGjUnqlO9ebAVQD47ZisFonnDAmjrZNVqEX\r\n"
    "F3p7laEHrFMxttYuD81BdOzxAbL9Rb/8MeFGQjE2Qx65qgVfhH+RsYuuD9dUw/3w\r\n"
    "ZAhq05yO6nk07AM9c+AbNtRoEcdZcLCHfMDcbkXKNs5DJncCqXAN6LhXVERCw/us\r\n"
    "G2MmCMLSIx9/kwt8bwhUmitOXc6fpT7SmFvRAtvxg84wUkg4Y/Gx++0j0z6StSeN\r\n"
    "0EJz150jaHG6WV4HUqaWTb98Tm90IgXAU4AW2GBOlzFPiU5IY9jt+eXC2Q6yC/Zp\r\n"
    "TL1LAcnL3Qa/OgLrHN0wiw1KFGD51WRPQ0Sh7QIDAQABo4IBJTCCASEwHQYDVR0O\r\n"
    "BBYEFLV2DDARzseSQk1Mx1wsyKkM6AtkMB8GA1UdIwQYMBaAFOWdWTCCR1jMrPoI\r\n"
    "VDaGezq1BE3wMA4GA1UdDwEB/wQEAwIBhjAdBgNVHSUEFjAUBggrBgEFBQcDAQYI\r\n"
    "KwYBBQUHAwIwEgYDVR0TAQH/BAgwBgEB/wIBADA0BggrBgEFBQcBAQQoMCYwJAYI\r\n"
    "KwYBBQUHMAGGGGh0dHA6Ly9vY3NwLmRpZ2ljZXJ0LmNvbTA6BgNVHR8EMzAxMC+g\r\n"
    "LaArhilodHRwOi8vY3JsMy5kaWdpY2VydC5jb20vT21uaXJvb3QyMDI1LmNybDAq\r\n"
    "BgNVHSAEIzAhMAgGBmeBDAECATAIBgZngQwBAgIwCwYJKwYBBAGCNyoBMA0GCSqG\r\n"
    "SIb3DQEBCwUAA4IBAQCfK76SZ1vae4qt6P+dTQUO7bYNFUHR5hXcA2D59CJWnEj5\r\n"
    "na7aKzyowKvQupW4yMH9fGNxtsh6iJswRqOOfZYC4/giBO/gNsBvwr8uDW7t1nYo\r\n"
    "DYGHPpvnpxCM2mYfQFHq576/TmeYu1RZY29C4w8xYBlkAA8mDJfRhMCmehk7cN5F\r\n"
    "JtyWRj2cZj/hOoI45TYDBChXpOlLZKIYiG1giY16vhCRi6zmPzEwv+tk156N6cGS\r\n"
    "Vm44jTQ/rs1sa0JSYjzUaYngoFdZC4OfxnIkQvUIA4TOFmPzNPEFdjcZsgbeEz4T\r\n"
    "cGHTBPK4R28F44qIMCtHRV55VMX53ev6P3hRddJb\r\n"
    "-----END CERTIFICATE-----\r\n";

Change all di <WiFiClient.h> to <WiFiClientSecure.h>.
Change all di WiFiClient fields to WiFiClientSecure.
For every class wey get WiFiClientSecure field, add one constructor and set di certificate for dat constructor:
```
_client.setCACert(FUNCTIONS_CERTIFICATE);
```

Disclaimer:
Dis dokyument don translate wit AI translation service Co-op Translator. Even though we dey try make am accurate, abeg make you sabi say machine translation fit get mistake or no dey correct well. Di original dokyument for di language wey dem write am first na di main source wey you go trust. For important information, e better make professional human translator check am. We no go fit take blame for any misunderstanding or wrong interpretation wey fit happen because you use dis translation.

23 KiB Raw Permalink Blame History