22 KiB

Raw Permalink Blame History

Tok to Text - Wio Terminal

For dis part of di lesson, you go write code wey go change di tok wey dey di audio wey you capture to text using di speech service.

Send di audio go di speech service

You fit send di audio go di speech service using di REST API. To use di speech service, first you go need request access token, den use dat token take access di REST API. Dis access tokens dey expire after 10 minutes, so your code suppose dey request dem regularly to make sure say dem dey always up to date.

Task - get access token

Open di smart-timer project if e never open.

Add di library dependencies wey dey below to di platformio.ini file to access WiFi and handle JSON:

seeed-studio/Seeed Arduino rpcWiFi @ 1.0.5
seeed-studio/Seeed Arduino rpcUnified @ 2.1.3
seeed-studio/Seeed_Arduino_mbedtls @ 3.0.1
seeed-studio/Seeed Arduino RTC @ 2.0.0
bblanchon/ArduinoJson @ 6.17.3

Add di code wey dey below to di config.h header file:
```
const char *SSID = "<SSID>";
const char *PASSWORD = "<PASSWORD>";

const char *SPEECH_API_KEY = "<API_KEY>";
const char *SPEECH_LOCATION = "<LOCATION>";
const char *LANGUAGE = "<LANGUAGE>";

const char *TOKEN_URL = "https://%s.api.cognitive.microsoft.com/sts/v1.0/issuetoken";
```
Change <SSID> and <PASSWORD> to di correct values for your WiFi.

Change <API_KEY> to di API key for your speech service resource. Change <LOCATION> to di location wey you use when you create di speech service resource.

Change <LANGUAGE> to di locale name for di language wey you go dey tok, for example en-GB for English, or zn-HK for Cantonese. You fit find di list of di supported languages and dia locale names for di Language and voice support documentation on Microsoft docs.

Di TOKEN_URL constant na di URL of di token issuer without di location. Later, dem go combine am with di location to get di full URL.

Just like how you take connect to Custom Vision, you go need use HTTPS connection to connect to di token issuing service. For di end of config.h, add di code wey dey below:

const char *TOKEN_CERTIFICATE =
    "-----BEGIN CERTIFICATE-----\r\n"
    "MIIF8zCCBNugAwIBAgIQAueRcfuAIek/4tmDg0xQwDANBgkqhkiG9w0BAQwFADBh\r\n"
    "MQswCQYDVQQGEwJVUzEVMBMGA1UEChMMRGlnaUNlcnQgSW5jMRkwFwYDVQQLExB3\r\n"
    "d3cuZGlnaWNlcnQuY29tMSAwHgYDVQQDExdEaWdpQ2VydCBHbG9iYWwgUm9vdCBH\r\n"
    "MjAeFw0yMDA3MjkxMjMwMDBaFw0yNDA2MjcyMzU5NTlaMFkxCzAJBgNVBAYTAlVT\r\n"
    "MR4wHAYDVQQKExVNaWNyb3NvZnQgQ29ycG9yYXRpb24xKjAoBgNVBAMTIU1pY3Jv\r\n"
    "c29mdCBBenVyZSBUTFMgSXNzdWluZyBDQSAwNjCCAiIwDQYJKoZIhvcNAQEBBQAD\r\n"
    "ggIPADCCAgoCggIBALVGARl56bx3KBUSGuPc4H5uoNFkFH4e7pvTCxRi4j/+z+Xb\r\n"
    "wjEz+5CipDOqjx9/jWjskL5dk7PaQkzItidsAAnDCW1leZBOIi68Lff1bjTeZgMY\r\n"
    "iwdRd3Y39b/lcGpiuP2d23W95YHkMMT8IlWosYIX0f4kYb62rphyfnAjYb/4Od99\r\n"
    "ThnhlAxGtfvSbXcBVIKCYfZgqRvV+5lReUnd1aNjRYVzPOoifgSx2fRyy1+pO1Uz\r\n"
    "aMMNnIOE71bVYW0A1hr19w7kOb0KkJXoALTDDj1ukUEDqQuBfBxReL5mXiu1O7WG\r\n"
    "0vltg0VZ/SZzctBsdBlx1BkmWYBW261KZgBivrql5ELTKKd8qgtHcLQA5fl6JB0Q\r\n"
    "gs5XDaWehN86Gps5JW8ArjGtjcWAIP+X8CQaWfaCnuRm6Bk/03PQWhgdi84qwA0s\r\n"
    "sRfFJwHUPTNSnE8EiGVk2frt0u8PG1pwSQsFuNJfcYIHEv1vOzP7uEOuDydsmCjh\r\n"
    "lxuoK2n5/2aVR3BMTu+p4+gl8alXoBycyLmj3J/PUgqD8SL5fTCUegGsdia/Sa60\r\n"
    "N2oV7vQ17wjMN+LXa2rjj/b4ZlZgXVojDmAjDwIRdDUujQu0RVsJqFLMzSIHpp2C\r\n"
    "Zp7mIoLrySay2YYBu7SiNwL95X6He2kS8eefBBHjzwW/9FxGqry57i71c2cDAgMB\r\n"
    "AAGjggGtMIIBqTAdBgNVHQ4EFgQU1cFnOsKjnfR3UltZEjgp5lVou6UwHwYDVR0j\r\n"
    "BBgwFoAUTiJUIBiV5uNu5g/6+rkS7QYXjzkwDgYDVR0PAQH/BAQDAgGGMB0GA1Ud\r\n"
    "JQQWMBQGCCsGAQUFBwMBBggrBgEFBQcDAjASBgNVHRMBAf8ECDAGAQH/AgEAMHYG\r\n"
    "CCsGAQUFBwEBBGowaDAkBggrBgEFBQcwAYYYaHR0cDovL29jc3AuZGlnaWNlcnQu\r\n"
    "Y29tMEAGCCsGAQUFBzAChjRodHRwOi8vY2FjZXJ0cy5kaWdpY2VydC5jb20vRGln\r\n"
    "aUNlcnRHbG9iYWxSb290RzIuY3J0MHsGA1UdHwR0MHIwN6A1oDOGMWh0dHA6Ly9j\r\n"
    "cmwzLmRpZ2ljZXJ0LmNvbS9EaWdpQ2VydEdsb2JhbFJvb3RHMi5jcmwwN6A1oDOG\r\n"
    "MWh0dHA6Ly9jcmw0LmRpZ2ljZXJ0LmNvbS9EaWdpQ2VydEdsb2JhbFJvb3RHMi5j\r\n"
    "cmwwHQYDVR0gBBYwFDAIBgZngQwBAgEwCAYGZ4EMAQICMBAGCSsGAQQBgjcVAQQD\r\n"
    "AgEAMA0GCSqGSIb3DQEBDAUAA4IBAQB2oWc93fB8esci/8esixj++N22meiGDjgF\r\n"
    "+rA2LUK5IOQOgcUSTGKSqF9lYfAxPjrqPjDCUPHCURv+26ad5P/BYtXtbmtxJWu+\r\n"
    "cS5BhMDPPeG3oPZwXRHBJFAkY4O4AF7RIAAUW6EzDflUoDHKv83zOiPfYGcpHc9s\r\n"
    "kxAInCedk7QSgXvMARjjOqdakor21DTmNIUotxo8kHv5hwRlGhBJwps6fEVi1Bt0\r\n"
    "trpM/3wYxlr473WSPUFZPgP1j519kLpWOJ8z09wxay+Br29irPcBYv0GMXlHqThy\r\n"
    "8y4m/HyTQeI2IMvMrQnwqPpY+rLIXyviI2vLoI+4xKE4Rn38ZZ8m\r\n"
    "-----END CERTIFICATE-----\r\n";

Na di same certificate wey you use when you dey connect to Custom Vision.

Add include for di WiFi header file and di config header file to di top of di main.cpp file:
```
#include <rpcWiFi.h>

#include "config.h"
```

Add code to connect to WiFi for main.cpp above di setup function:

void connectWiFi()
{
    while (WiFi.status() != WL_CONNECTED)
    {
        Serial.println("Connecting to WiFi..");
        WiFi.begin(SSID, PASSWORD);
        delay(500);
    }

    Serial.println("Connected!");
}

Call dis function from di setup function after di serial connection don establish:
```
connectWiFi();
```
Create new header file for di src folder wey you go call speech_to_text.h. For dis header file, add di code wey dey below:
```
#pragma once

#include <Arduino.h>
#include <ArduinoJson.h>
#include <HTTPClient.h>
#include <WiFiClientSecure.h>

#include "config.h"
#include "mic.h"

class SpeechToText
{
public:

private:

};

SpeechToText speechToText;
```
Dis one include some necessary header files for HTTP connection, configuration and di mic.h header file, and e define one class wey dem call SpeechToText, before e declare one instance of dat class wey you fit use later.
Add di 2 fields wey dey below to di private section of dis class:
```
WiFiClientSecure _token_client;
String _access_token;
```
Di _token_client na WiFi Client wey dey use HTTPS and e go dey used to get di access token. Dis token go later dey store for _access_token.

Add di method wey dey below to di private section:

String getAccessToken()
{
    char url[128];
    sprintf(url, TOKEN_URL, SPEECH_LOCATION);

    HTTPClient httpClient;
    httpClient.begin(_token_client, url);

    httpClient.addHeader("Ocp-Apim-Subscription-Key", SPEECH_API_KEY);
    int httpResultCode = httpClient.POST("{}");

    if (httpResultCode != 200)
    {
        Serial.println("Error getting access token, trying again...");
        delay(10000);
        return getAccessToken();
    }

    Serial.println("Got access token.");
    String result = httpClient.getString();

    httpClient.end();

    return result;
}

Dis code dey build di URL for di token issuer API using di location of di speech resource. E go then create HTTPClient to make di web request, set am up to use di WiFi client wey dem configure with di token endpoints certificate. E go set di API key as header for di call. E go then make POST request to get di certificate, retry if e get any error. Finally, di access token go return.

For di public section, add method to get di access token. Dis one go dey needed for later lessons to change text to speech.
```
String AccessToken()
{
    return _access_token;
}
```
For di public section, add init method wey go set up di token client:
```
void init()
{
    _token_client.setCACert(TOKEN_CERTIFICATE);
    _access_token = getAccessToken();
}
```
Dis one go set di certificate for di WiFi client, then e go get di access token.
For main.cpp, add dis new header file to di include directives:
```
#include "speech_to_text.h"
```
Initialize di SpeechToText class for di end of di setup function, after di mic.init call but before Ready go show for di serial monitor:
```
speechToText.init();
```

Task - read audio from flash memory

For one earlier part of dis lesson, di audio don record to di flash memory. Dis audio go need to dey send to di Speech Services REST API, so e need to dey read from di flash memory. E no fit load am into in-memory buffer because e go too big. Di HTTPClient class wey dey make REST calls fit stream data using Arduino Stream - na class wey fit load data in small chunks, dey send di chunks one by one as part of di request. Anytime you call read for stream, e go return di next block of data. Arduino stream fit dey created wey fit read from di flash memory. Create new file wey you go call flash_stream.h for di src folder, and add di code wey dey below to am:
```
#pragma once

#include <Arduino.h>
#include <HTTPClient.h>
#include <sfud.h>

#include "config.h"

class FlashStream : public Stream
{
public:
    virtual size_t write(uint8_t val)
    {    
    }

    virtual int available()
    {
    }

    virtual int read()
    {
    }

    virtual int peek()
    {
    }
private:

};
```
Dis one dey declare di FlashStream class, wey dey derive from di Arduino Stream class. Dis na abstract class - di classes wey dey derive from am go need implement some methods before di class fit dey instantiated, and dem don define dis methods for dis class.

✅ Read more about Arduino Streams for di Arduino Stream documentation
Add di fields wey dey below to di private section:
```
size_t _pos;
size_t _flash_address;
const sfud_flash *_flash;

byte _buffer[HTTP_TCP_BUFFER_SIZE];
```
Dis one dey define temporary buffer to store data wey dem read from di flash memory, along with fields to store di current position when dem dey read from di buffer, di current address to read from di flash memory, and di flash memory device.
For di private section, add di method wey dey below:
```
void populateBuffer()
{
    sfud_read(_flash, _flash_address, HTTP_TCP_BUFFER_SIZE, _buffer);
    _flash_address += HTTP_TCP_BUFFER_SIZE;
    _pos = 0;
}
```
Dis code dey read from di flash memory for di current address and e dey store di data for buffer. E go then increase di address, so di next call go read di next block of memory. Di buffer size dey based on di largest chunk wey di HTTPClient go send to di REST API at one time.

💁 To erase flash memory, you go need use di grain size, but to read no need.
For di public section of dis class, add constructor:
```
FlashStream()
{
    _pos = 0;
    _flash_address = 0;
    _flash = sfud_get_device_table() + 0;

    populateBuffer();
}
```
Dis constructor dey set up all di fields to start to dey read from di start of di flash memory block, and e dey load di first chunk of data into di buffer.
Implement di write method. Dis stream go only dey read data, so dis one no go do anything and e go return 0:
```
virtual size_t write(uint8_t val)
{
    return 0;
}
```
Implement di peek method. Dis one dey return di data for di current position without moving di stream go front. If you call peek many times, e go always return di same data as long as dem never read data from di stream.
```
virtual int peek()
{
    return _buffer[_pos];
}
```
Implement di available function. Dis one dey return how many bytes fit dey read from di stream, or -1 if di stream don complete. For dis class, di maximum wey fit dey available no go pass di HTTPClient chunk size. When dis stream dey used for di HTTP client, e dey call dis function to see how much data dey available, then e go request dat amount of data to send to di REST API. We no want make each chunk pass di HTTP client chunk size, so if more than dat dey available, na di chunk size e go return. If e small, na wetin dey available e go return. Once all di data don stream finish, e go return -1.
```
virtual int available()
{
    int remaining = BUFFER_SIZE - ((_flash_address - HTTP_TCP_BUFFER_SIZE) + _pos);
    int bytes_available = min(HTTP_TCP_BUFFER_SIZE, remaining);

    if (bytes_available == 0)
    {
        bytes_available = -1;
    }

    return bytes_available;
}
```
Implement di read method to return di next byte from di buffer, dey increase di position. If di position pass di size of di buffer, e go populate di buffer with di next block from di flash memory and reset di position.
```
virtual int read()
{
    int retVal = _buffer[_pos++];

    if (_pos == HTTP_TCP_BUFFER_SIZE)
    {
        populateBuffer();
    }

    return retVal;
}
```
For di speech_to_text.h header file, add include directive for dis new header file:
```
#include "flash_stream.h"
```

Task - change di tok to text

Di tok fit change to text by sending di audio go di Speech Service via REST API. Dis REST API get different certificate from di token issuer, so add di code wey dey below to di config.h header file to define dis certificate:

const char *SPEECH_CERTIFICATE =
    "-----BEGIN CERTIFICATE-----\r\n"
    "MIIF8zCCBNugAwIBAgIQCq+mxcpjxFFB6jvh98dTFzANBgkqhkiG9w0BAQwFADBh\r\n"
    "MQswCQYDVQQGEwJVUzEVMBMGA1UEChMMRGlnaUNlcnQgSW5jMRkwFwYDVQQLExB3\r\n"
    "d3cuZGlnaWNlcnQuY29tMSAwHgYDVQQDExdEaWdpQ2VydCBHbG9iYWwgUm9vdCBH\r\n"
    "MjAeFw0yMDA3MjkxMjMwMDBaFw0yNDA2MjcyMzU5NTlaMFkxCzAJBgNVBAYTAlVT\r\n"
    "MR4wHAYDVQQKExVNaWNyb3NvZnQgQ29ycG9yYXRpb24xKjAoBgNVBAMTIU1pY3Jv\r\n"
    "c29mdCBBenVyZSBUTFMgSXNzdWluZyBDQSAwMTCCAiIwDQYJKoZIhvcNAQEBBQAD\r\n"
    "ggIPADCCAgoCggIBAMedcDrkXufP7pxVm1FHLDNA9IjwHaMoaY8arqqZ4Gff4xyr\r\n"
    "RygnavXL7g12MPAx8Q6Dd9hfBzrfWxkF0Br2wIvlvkzW01naNVSkHp+OS3hL3W6n\r\n"
    "l/jYvZnVeJXjtsKYcXIf/6WtspcF5awlQ9LZJcjwaH7KoZuK+THpXCMtzD8XNVdm\r\n"
    "GW/JI0C/7U/E7evXn9XDio8SYkGSM63aLO5BtLCv092+1d4GGBSQYolRq+7Pd1kR\r\n"
    "EkWBPm0ywZ2Vb8GIS5DLrjelEkBnKCyy3B0yQud9dpVsiUeE7F5sY8Me96WVxQcb\r\n"
    "OyYdEY/j/9UpDlOG+vA+YgOvBhkKEjiqygVpP8EZoMMijephzg43b5Qi9r5UrvYo\r\n"
    "o19oR/8pf4HJNDPF0/FJwFVMW8PmCBLGstin3NE1+NeWTkGt0TzpHjgKyfaDP2tO\r\n"
    "4bCk1G7pP2kDFT7SYfc8xbgCkFQ2UCEXsaH/f5YmpLn4YPiNFCeeIida7xnfTvc4\r\n"
    "7IxyVccHHq1FzGygOqemrxEETKh8hvDR6eBdrBwmCHVgZrnAqnn93JtGyPLi6+cj\r\n"
    "WGVGtMZHwzVvX1HvSFG771sskcEjJxiQNQDQRWHEh3NxvNb7kFlAXnVdRkkvhjpR\r\n"
    "GchFhTAzqmwltdWhWDEyCMKC2x/mSZvZtlZGY+g37Y72qHzidwtyW7rBetZJAgMB\r\n"
    "AAGjggGtMIIBqTAdBgNVHQ4EFgQUDyBd16FXlduSzyvQx8J3BM5ygHYwHwYDVR0j\r\n"
    "BBgwFoAUTiJUIBiV5uNu5g/6+rkS7QYXjzkwDgYDVR0PAQH/BAQDAgGGMB0GA1Ud\r\n"
    "JQQWMBQGCCsGAQUFBwMBBggrBgEFBQcDAjASBgNVHRMBAf8ECDAGAQH/AgEAMHYG\r\n"
    "CCsGAQUFBwEBBGowaDAkBggrBgEFBQcwAYYYaHR0cDovL29jc3AuZGlnaWNlcnQu\r\n"
    "Y29tMEAGCCsGAQUFBzAChjRodHRwOi8vY2FjZXJ0cy5kaWdpY2VydC5jb20vRGln\r\n"
    "aUNlcnRHbG9iYWxSb290RzIuY3J0MHsGA1UdHwR0MHIwN6A1oDOGMWh0dHA6Ly9j\r\n"
    "cmwzLmRpZ2ljZXJ0LmNvbS9EaWdpQ2VydEdsb2JhbFJvb3RHMi5jcmwwN6A1oDOG\r\n"
    "MWh0dHA6Ly9jcmw0LmRpZ2ljZXJ0LmNvbS9EaWdpQ2VydEdsb2JhbFJvb3RHMi5j\r\n"
    "cmwwHQYDVR0gBBYwFDAIBgZngQwBAgEwCAYGZ4EMAQICMBAGCSsGAQQBgjcVAQQD\r\n"
    "AgEAMA0GCSqGSIb3DQEBDAUAA4IBAQAlFvNh7QgXVLAZSsNR2XRmIn9iS8OHFCBA\r\n"
    "WxKJoi8YYQafpMTkMqeuzoL3HWb1pYEipsDkhiMnrpfeYZEA7Lz7yqEEtfgHcEBs\r\n"
    "K9KcStQGGZRfmWU07hPXHnFz+5gTXqzCE2PBMlRgVUYJiA25mJPXfB00gDvGhtYa\r\n"
    "+mENwM9Bq1B9YYLyLjRtUz8cyGsdyTIG/bBM/Q9jcV8JGqMU/UjAdh1pFyTnnHEl\r\n"
    "Y59Npi7F87ZqYYJEHJM2LGD+le8VsHjgeWX2CJQko7klXvcizuZvUEDTjHaQcs2J\r\n"
    "+kPgfyMIOY1DMJ21NxOJ2xPRC/wAh/hzSBRVtoAnyuxtkZ4VjIOh\r\n"
    "-----END CERTIFICATE-----\r\n";

Add constant to dis file for di speech URL without di location. Later, dem go combine am with di location and language to get di full URL.
```
const char *SPEECH_URL = "https://%s.stt.speech.microsoft.com/speech/recognition/conversation/cognitiveservices/v1?language=%s";
```
For di speech_to_text.h header file, for di private section of di SpeechToText class, define field for WiFi Client wey dey use di speech certificate:
```
WiFiClientSecure _speech_client;
```
For di init method, set di certificate for dis WiFi Client:
```
_speech_client.setCACert(SPEECH_CERTIFICATE);
```
Add di code wey dey below to di public section of di SpeechToText class to define method to change tok to text:
```
String convertSpeechToText()
{

}
```
Add di code wey dey below to dis method to create HTTP client wey dey use di WiFi client wey dem configure with di speech certificate, and dey use di speech URL wey dem set with di location and language:
```
char url[128];
sprintf(url, SPEECH_URL, SPEECH_LOCATION, LANGUAGE);

HTTPClient httpClient;
httpClient.begin(_speech_client, url);
```

Some headers go need dey set for di connection:

httpClient.addHeader("Authorization", String("Bearer ") + _access_token);
httpClient.addHeader("Content-Type", String("audio/wav; codecs=audio/pcm; samplerate=") + String(RATE));
httpClient.addHeader("Accept", "application/json;text/xml");

Dis one dey set headers for di authorization using di access token, di audio format using di sample rate, and e dey set say di client dey expect di result as JSON.

After dis one, add di code wey dey below to make di REST API call:

Serial.println("Sending speech...");

FlashStream stream;
int httpResponseCode = httpClient.sendRequest("POST", &stream, BUFFER_SIZE);

Serial.println("Speech sent!");

Dis one dey create FlashStream and e dey use am to stream data go di REST API.

Under dis one, add di code wey dey below:

String text = "";

if (httpResponseCode == 200)
{
    String result = httpClient.getString();
    Serial.println(result);

    DynamicJsonDocument doc(1024);
    deserializeJson(doc, result.c_str());

    JsonObject obj = doc.as<JsonObject>();
    text = obj["DisplayText"].as<String>();
}
else if (httpResponseCode == 401)
{
    Serial.println("Access token expired, trying again with a new token");
    _access_token = getAccessToken();
    return convertSpeechToText();
}
else
{
    Serial.print("Failed to convert text to speech - error ");
    Serial.println(httpResponseCode);
}

Dis code dey check di response code.

If e be 200, di code for success, di result go dey retrieve, decode from JSON, and di DisplayText property go dey set into di text variable. Na dis property dem dey use return di text version of di tok.

If di response code na 401, di access token don expire (dis tokens dey last only 10 minutes). New access token go dey request, and dem go make di call again.

If na any other thing, error go dey send to di serial monitor, and di text go dey blank.

Add di code wey dey below to di end of dis method to close di HTTP client and return di text:
```
httpClient.end();

return text;
```
For main.cpp call dis new convertSpeechToText method for di processAudio function, then log di tok to di serial monitor:
```
String text = speechToText.convertSpeechToText();
Serial.println(text);
```

Build dis code, upload am to your Wio Terminal and test am through di serial monitor. Once you see Ready for di serial monitor, press di C button (di one wey dey for di left-hand side, close to di power switch), and tok. 4 seconds of audio go dey capture, then e go change to text.

--- Available filters and text transformations: colorize, debug, default, direct, hexlify, log2file, nocontrol, printable, send_on_enter, time
--- More details at http://bit.ly/pio-monitor-filters
--- Miniterm on /dev/cu.usbmodem1101  9600,8,N,1 ---
--- Quit: Ctrl+C | Menu: Ctrl+T | Help: Ctrl+T followed by Ctrl+H ---
Connecting to WiFi..
Connected!
Got access token.
Ready.
Starting recording...
Finished recording
Sending speech...
Speech sent!
{"RecognitionStatus":"Success","DisplayText":"Set a 2 minute and 27 second timer.","Offset":4700000,"Duration":35300000}
Set a 2 minute and 27 second timer.

💁 You fit find dis code for di code-speech-to-text/wio-terminal folder.

😀 Your tok to text program don work well!

Disclaimer:
Dis dokyument don use AI transle-shon service Co-op Translator do di transle-shon. Even as we dey try make am accurate, abeg make you sabi say transle-shon wey machine do fit get mistake or no dey correct well. Di original dokyument for im native language na di one wey you go take as di correct source. For important mata, e good make you use professional human transle-shon. We no go fit take blame for any misunderstanding or wrong interpretation wey fit happen because you use dis transle-shon.

22 KiB Raw Permalink Blame History