diff --git a/6-consumer/README.md b/6-consumer/README.md index 4b986ad9..d3c8c9d8 100644 --- a/6-consumer/README.md +++ b/6-consumer/README.md @@ -6,6 +6,8 @@ The latest iterations are now part of our smart devices. In kitchens in homes al In these 4 lessons you'll learn how to build a smart timer, using AI to recognize your voice, understand what you are asking for, and reply with information about your timer. You'll also add support for multiple languages. +> ⚠️ Working with speech and microphone data uses a lot of memory, meaning it is easy to hit limits on microcontrollers. The project here works around these issues, but be aware the Wio Terminal labs are complex and may take more time that other labs in this curriculum. + > 💁 These lessons will use some cloud resources. If you don't complete all the lessons in this project, make sure you [clean up your project](../clean-up.md). ## Topics diff --git a/6-consumer/lessons/2-language-understanding/README.md b/6-consumer/lessons/2-language-understanding/README.md index 92ca888a..0c3a5082 100644 --- a/6-consumer/lessons/2-language-understanding/README.md +++ b/6-consumer/lessons/2-language-understanding/README.md @@ -493,7 +493,9 @@ Rather than calling LUIS from the IoT device, you can use serverless code with a ### Task - make your function available to your IoT device -1. For your IoT device to call your REST endpoint, it will need to know the URL. When you accessed it earlier, you used `localhost`, which is a shortcut to access REST endpoints on your local machine. To allow you IoT device to get access, you need to either: +1. For your IoT device to call your REST endpoint, it will need to know the URL. When you accessed it earlier, you used `localhost`, which is a shortcut to access REST endpoints on your local machine. To allow you IoT device to get access, you need to either publish to the cloud, or get your IP address to access it locally. + + > ⚠️ If you are using a Wio Terminal, it is easier to run the function app locally, as there will be a dependency on libraries that mean you cannot deploy the function app in the same way as you have done before. Run the function app locally and access it via your computers IP address. If you do want to deploy to the cloud, information will be provided in a later lesson on the way to do this. * Publish the Functions app - follow the instructions in earlier lessons to publish your functions app to the cloud. Once published, the URL will be `https://.azurewebsites.net/api/text-to-timer`, where `` will be the name of your functions app. Make sure to also publish your local settings. @@ -530,6 +532,8 @@ Rather than calling LUIS from the IoT device, you can use serverless code with a Once you have your IP address, you will able to access the function at `http://:7071/api/text-to-timer`, where `` will be your IP address, for example `http://192.168.1.10:7071/api/text-to-timer`. + > 💁 Not that this uses port 7071, so after the IP address you will need to have `:7071`. + > 💁 This will only work if your IoT device is on the same network as your computer. 1. Test the endpoint by accessing it using curl. diff --git a/6-consumer/lessons/3-spoken-feedback/code-spoken-response/functions/smart-timer-trigger/host.json b/6-consumer/lessons/3-spoken-feedback/code-spoken-response/functions/smart-timer-trigger/host.json new file mode 100644 index 00000000..291065f8 --- /dev/null +++ b/6-consumer/lessons/3-spoken-feedback/code-spoken-response/functions/smart-timer-trigger/host.json @@ -0,0 +1,15 @@ +{ + "version": "2.0", + "logging": { + "applicationInsights": { + "samplingSettings": { + "isEnabled": true, + "excludedTypes": "Request" + } + } + }, + "extensionBundle": { + "id": "Microsoft.Azure.Functions.ExtensionBundle", + "version": "[2.*, 3.0.0)" + } +} \ No newline at end of file diff --git a/6-consumer/lessons/3-spoken-feedback/code-spoken-response/functions/smart-timer-trigger/local.settings.json b/6-consumer/lessons/3-spoken-feedback/code-spoken-response/functions/smart-timer-trigger/local.settings.json new file mode 100644 index 00000000..ee6b34ce --- /dev/null +++ b/6-consumer/lessons/3-spoken-feedback/code-spoken-response/functions/smart-timer-trigger/local.settings.json @@ -0,0 +1,10 @@ +{ + "IsEncrypted": false, + "Values": { + "FUNCTIONS_WORKER_RUNTIME": "python", + "AzureWebJobsStorage": "", + "LUIS_KEY": "", + "LUIS_ENDPOINT_URL": "", + "LUIS_APP_ID": "" + } +} \ No newline at end of file diff --git a/6-consumer/lessons/3-spoken-feedback/code-spoken-response/functions/smart-timer-trigger/requirements.txt b/6-consumer/lessons/3-spoken-feedback/code-spoken-response/functions/smart-timer-trigger/requirements.txt new file mode 100644 index 00000000..d0405a38 --- /dev/null +++ b/6-consumer/lessons/3-spoken-feedback/code-spoken-response/functions/smart-timer-trigger/requirements.txt @@ -0,0 +1,4 @@ +# Do not include azure-functions-worker as it may conflict with the Azure Functions platform + +azure-functions +azure-cognitiveservices-language-luis \ No newline at end of file diff --git a/6-consumer/lessons/3-spoken-feedback/code-spoken-response/functions/smart-timer-trigger/text-to-timer/__init__.py b/6-consumer/lessons/3-spoken-feedback/code-spoken-response/functions/smart-timer-trigger/text-to-timer/__init__.py new file mode 100644 index 00000000..d15d6e68 --- /dev/null +++ b/6-consumer/lessons/3-spoken-feedback/code-spoken-response/functions/smart-timer-trigger/text-to-timer/__init__.py @@ -0,0 +1,46 @@ +import logging + +import azure.functions as func +import json +import os +from azure.cognitiveservices.language.luis.runtime import LUISRuntimeClient +from msrest.authentication import CognitiveServicesCredentials + + +def main(req: func.HttpRequest) -> func.HttpResponse: + luis_key = os.environ['LUIS_KEY'] + endpoint_url = os.environ['LUIS_ENDPOINT_URL'] + app_id = os.environ['LUIS_APP_ID'] + + credentials = CognitiveServicesCredentials(luis_key) + client = LUISRuntimeClient(endpoint=endpoint_url, credentials=credentials) + + req_body = req.get_json() + text = req_body['text'] + logging.info(f'Request - {text}') + prediction_request = { 'query' : text } + + prediction_response = client.prediction.get_slot_prediction(app_id, 'Staging', prediction_request) + + if prediction_response.prediction.top_intent == 'set timer': + numbers = prediction_response.prediction.entities['number'] + time_units = prediction_response.prediction.entities['time unit'] + total_seconds = 0 + + for i in range(0, len(numbers)): + number = numbers[i] + time_unit = time_units[i][0] + + if time_unit == 'minute': + total_seconds += number * 60 + else: + total_seconds += number + + logging.info(f'Timer required for {total_seconds} seconds') + + payload = { + 'seconds': total_seconds + } + return func.HttpResponse(json.dumps(payload), status_code=200) + + return func.HttpResponse(status_code=404) \ No newline at end of file diff --git a/6-consumer/lessons/3-spoken-feedback/code-spoken-response/functions/smart-timer-trigger/text-to-timer/function.json b/6-consumer/lessons/3-spoken-feedback/code-spoken-response/functions/smart-timer-trigger/text-to-timer/function.json new file mode 100644 index 00000000..d9019652 --- /dev/null +++ b/6-consumer/lessons/3-spoken-feedback/code-spoken-response/functions/smart-timer-trigger/text-to-timer/function.json @@ -0,0 +1,20 @@ +{ + "scriptFile": "__init__.py", + "bindings": [ + { + "authLevel": "function", + "type": "httpTrigger", + "direction": "in", + "name": "req", + "methods": [ + "get", + "post" + ] + }, + { + "type": "http", + "direction": "out", + "name": "$return" + } + ] +} \ No newline at end of file diff --git a/6-consumer/lessons/3-spoken-feedback/code-timer/wio-terminal/smart-timer/platformio.ini b/6-consumer/lessons/3-spoken-feedback/code-timer/wio-terminal/smart-timer/platformio.ini index a2d49d0f..8836ab42 100644 --- a/6-consumer/lessons/3-spoken-feedback/code-timer/wio-terminal/smart-timer/platformio.ini +++ b/6-consumer/lessons/3-spoken-feedback/code-timer/wio-terminal/smart-timer/platformio.ini @@ -13,8 +13,8 @@ platform = atmelsam board = seeed_wio_terminal framework = arduino lib_deps = - seeed-studio/Seeed Arduino FS @ 2.0.3 - seeed-studio/Seeed Arduino SFUD @ 2.0.1 + seeed-studio/Seeed Arduino FS @ 2.1.1 + seeed-studio/Seeed Arduino SFUD @ 2.0.2 seeed-studio/Seeed Arduino rpcWiFi @ 1.0.5 seeed-studio/Seeed Arduino rpcUnified @ 2.1.3 seeed-studio/Seeed_Arduino_mbedtls @ 3.0.1 diff --git a/6-consumer/lessons/3-spoken-feedback/code-timer/wio-terminal/smart-timer/src/config.h b/6-consumer/lessons/3-spoken-feedback/code-timer/wio-terminal/smart-timer/src/config.h index 4863d689..5dc05537 100644 --- a/6-consumer/lessons/3-spoken-feedback/code-timer/wio-terminal/smart-timer/src/config.h +++ b/6-consumer/lessons/3-spoken-feedback/code-timer/wio-terminal/smart-timer/src/config.h @@ -16,7 +16,7 @@ const char *LANGUAGE = ""; const char *TOKEN_URL = "https://%s.api.cognitive.microsoft.com/sts/v1.0/issuetoken"; const char *SPEECH_URL = "https://%s.stt.speech.microsoft.com/speech/recognition/conversation/cognitiveservices/v1?language=%s"; -const char *FUNCTION_URL = ""; +const char *TEXT_TO_TIMER_FUNCTION_URL = ""; const char *TOKEN_CERTIFICATE = "-----BEGIN CERTIFICATE-----\r\n" @@ -89,36 +89,3 @@ const char *SPEECH_CERTIFICATE = "Y59Npi7F87ZqYYJEHJM2LGD+le8VsHjgeWX2CJQko7klXvcizuZvUEDTjHaQcs2J\r\n" "+kPgfyMIOY1DMJ21NxOJ2xPRC/wAh/hzSBRVtoAnyuxtkZ4VjIOh\r\n" "-----END CERTIFICATE-----\r\n"; - -const char *FUNCTIONS_CERTIFICATE = - "-----BEGIN CERTIFICATE-----\r\n" - "MIIFWjCCBEKgAwIBAgIQDxSWXyAgaZlP1ceseIlB4jANBgkqhkiG9w0BAQsFADBa\r\n" - "MQswCQYDVQQGEwJJRTESMBAGA1UEChMJQmFsdGltb3JlMRMwEQYDVQQLEwpDeWJl\r\n" - "clRydXN0MSIwIAYDVQQDExlCYWx0aW1vcmUgQ3liZXJUcnVzdCBSb290MB4XDTIw\r\n" - "MDcyMTIzMDAwMFoXDTI0MTAwODA3MDAwMFowTzELMAkGA1UEBhMCVVMxHjAcBgNV\r\n" - "BAoTFU1pY3Jvc29mdCBDb3Jwb3JhdGlvbjEgMB4GA1UEAxMXTWljcm9zb2Z0IFJT\r\n" - "QSBUTFMgQ0EgMDEwggIiMA0GCSqGSIb3DQEBAQUAA4ICDwAwggIKAoICAQCqYnfP\r\n" - "mmOyBoTzkDb0mfMUUavqlQo7Rgb9EUEf/lsGWMk4bgj8T0RIzTqk970eouKVuL5R\r\n" - "IMW/snBjXXgMQ8ApzWRJCZbar879BV8rKpHoAW4uGJssnNABf2n17j9TiFy6BWy+\r\n" - "IhVnFILyLNK+W2M3zK9gheiWa2uACKhuvgCca5Vw/OQYErEdG7LBEzFnMzTmJcli\r\n" - "W1iCdXby/vI/OxbfqkKD4zJtm45DJvC9Dh+hpzqvLMiK5uo/+aXSJY+SqhoIEpz+\r\n" - "rErHw+uAlKuHFtEjSeeku8eR3+Z5ND9BSqc6JtLqb0bjOHPm5dSRrgt4nnil75bj\r\n" - "c9j3lWXpBb9PXP9Sp/nPCK+nTQmZwHGjUnqlO9ebAVQD47ZisFonnDAmjrZNVqEX\r\n" - "F3p7laEHrFMxttYuD81BdOzxAbL9Rb/8MeFGQjE2Qx65qgVfhH+RsYuuD9dUw/3w\r\n" - "ZAhq05yO6nk07AM9c+AbNtRoEcdZcLCHfMDcbkXKNs5DJncCqXAN6LhXVERCw/us\r\n" - "G2MmCMLSIx9/kwt8bwhUmitOXc6fpT7SmFvRAtvxg84wUkg4Y/Gx++0j0z6StSeN\r\n" - "0EJz150jaHG6WV4HUqaWTb98Tm90IgXAU4AW2GBOlzFPiU5IY9jt+eXC2Q6yC/Zp\r\n" - "TL1LAcnL3Qa/OgLrHN0wiw1KFGD51WRPQ0Sh7QIDAQABo4IBJTCCASEwHQYDVR0O\r\n" - "BBYEFLV2DDARzseSQk1Mx1wsyKkM6AtkMB8GA1UdIwQYMBaAFOWdWTCCR1jMrPoI\r\n" - "VDaGezq1BE3wMA4GA1UdDwEB/wQEAwIBhjAdBgNVHSUEFjAUBggrBgEFBQcDAQYI\r\n" - "KwYBBQUHAwIwEgYDVR0TAQH/BAgwBgEB/wIBADA0BggrBgEFBQcBAQQoMCYwJAYI\r\n" - "KwYBBQUHMAGGGGh0dHA6Ly9vY3NwLmRpZ2ljZXJ0LmNvbTA6BgNVHR8EMzAxMC+g\r\n" - "LaArhilodHRwOi8vY3JsMy5kaWdpY2VydC5jb20vT21uaXJvb3QyMDI1LmNybDAq\r\n" - "BgNVHSAEIzAhMAgGBmeBDAECATAIBgZngQwBAgIwCwYJKwYBBAGCNyoBMA0GCSqG\r\n" - "SIb3DQEBCwUAA4IBAQCfK76SZ1vae4qt6P+dTQUO7bYNFUHR5hXcA2D59CJWnEj5\r\n" - "na7aKzyowKvQupW4yMH9fGNxtsh6iJswRqOOfZYC4/giBO/gNsBvwr8uDW7t1nYo\r\n" - "DYGHPpvnpxCM2mYfQFHq576/TmeYu1RZY29C4w8xYBlkAA8mDJfRhMCmehk7cN5F\r\n" - "JtyWRj2cZj/hOoI45TYDBChXpOlLZKIYiG1giY16vhCRi6zmPzEwv+tk156N6cGS\r\n" - "Vm44jTQ/rs1sa0JSYjzUaYngoFdZC4OfxnIkQvUIA4TOFmPzNPEFdjcZsgbeEz4T\r\n" - "cGHTBPK4R28F44qIMCtHRV55VMX53ev6P3hRddJb\r\n" - "-----END CERTIFICATE-----\r\n"; diff --git a/6-consumer/lessons/3-spoken-feedback/code-timer/wio-terminal/smart-timer/src/language_understanding.h b/6-consumer/lessons/3-spoken-feedback/code-timer/wio-terminal/smart-timer/src/language_understanding.h index ddabb4fb..9d6227a0 100644 --- a/6-consumer/lessons/3-spoken-feedback/code-timer/wio-terminal/smart-timer/src/language_understanding.h +++ b/6-consumer/lessons/3-spoken-feedback/code-timer/wio-terminal/smart-timer/src/language_understanding.h @@ -4,18 +4,12 @@ #include #include #include -#include #include "config.h" class LanguageUnderstanding { public: - LanguageUnderstanding() - { - _client.setCACert(FUNCTIONS_CERTIFICATE); - } - int GetTimerDuration(String text) { DynamicJsonDocument doc(1024); @@ -26,9 +20,9 @@ public: serializeJson(obj, body); HTTPClient httpClient; - httpClient.begin(_client, FUNCTION_URL); + httpClient.begin(_client, TEXT_TO_TIMER_FUNCTION_URL); - int httpResponseCode = httpClient.sendRequest("POST", body); + int httpResponseCode = httpClient.POST(body); int seconds = 0; if (httpResponseCode == 200) @@ -54,8 +48,7 @@ public: } private: - // WiFiClient _client; - WiFiClientSecure _client; + WiFiClient _client; }; LanguageUnderstanding languageUnderstanding; \ No newline at end of file diff --git a/6-consumer/lessons/3-spoken-feedback/wio-terminal-set-timer.md b/6-consumer/lessons/3-spoken-feedback/wio-terminal-set-timer.md index 4b818398..1a1b4299 100644 --- a/6-consumer/lessons/3-spoken-feedback/wio-terminal-set-timer.md +++ b/6-consumer/lessons/3-spoken-feedback/wio-terminal-set-timer.md @@ -15,49 +15,10 @@ Microcontrollers don't natively have support for multiple threads in Arduino, so 1. Open the `config.h` header file and add the URL for your function app: ```cpp - const char *FUNCTION_URL = ""; + const char *TEXT_TO_TIMER_FUNCTION_URL = ""; ``` - Replace `` with the URL for your function app that you obtained in the last step of the last lesson, either pointing to the IP address of your local machine that is running the function app, or the function app deployed to the cloud. - -1. If you have deployed your functions app to the cloud, add the following certificate for `azurewebsites.net` to the `config.h` file. - - ```cpp - const char *FUNCTIONS_CERTIFICATE = - "-----BEGIN CERTIFICATE-----\r\n" - "MIIFWjCCBEKgAwIBAgIQDxSWXyAgaZlP1ceseIlB4jANBgkqhkiG9w0BAQsFADBa\r\n" - "MQswCQYDVQQGEwJJRTESMBAGA1UEChMJQmFsdGltb3JlMRMwEQYDVQQLEwpDeWJl\r\n" - "clRydXN0MSIwIAYDVQQDExlCYWx0aW1vcmUgQ3liZXJUcnVzdCBSb290MB4XDTIw\r\n" - "MDcyMTIzMDAwMFoXDTI0MTAwODA3MDAwMFowTzELMAkGA1UEBhMCVVMxHjAcBgNV\r\n" - "BAoTFU1pY3Jvc29mdCBDb3Jwb3JhdGlvbjEgMB4GA1UEAxMXTWljcm9zb2Z0IFJT\r\n" - "QSBUTFMgQ0EgMDEwggIiMA0GCSqGSIb3DQEBAQUAA4ICDwAwggIKAoICAQCqYnfP\r\n" - "mmOyBoTzkDb0mfMUUavqlQo7Rgb9EUEf/lsGWMk4bgj8T0RIzTqk970eouKVuL5R\r\n" - "IMW/snBjXXgMQ8ApzWRJCZbar879BV8rKpHoAW4uGJssnNABf2n17j9TiFy6BWy+\r\n" - "IhVnFILyLNK+W2M3zK9gheiWa2uACKhuvgCca5Vw/OQYErEdG7LBEzFnMzTmJcli\r\n" - "W1iCdXby/vI/OxbfqkKD4zJtm45DJvC9Dh+hpzqvLMiK5uo/+aXSJY+SqhoIEpz+\r\n" - "rErHw+uAlKuHFtEjSeeku8eR3+Z5ND9BSqc6JtLqb0bjOHPm5dSRrgt4nnil75bj\r\n" - "c9j3lWXpBb9PXP9Sp/nPCK+nTQmZwHGjUnqlO9ebAVQD47ZisFonnDAmjrZNVqEX\r\n" - "F3p7laEHrFMxttYuD81BdOzxAbL9Rb/8MeFGQjE2Qx65qgVfhH+RsYuuD9dUw/3w\r\n" - "ZAhq05yO6nk07AM9c+AbNtRoEcdZcLCHfMDcbkXKNs5DJncCqXAN6LhXVERCw/us\r\n" - "G2MmCMLSIx9/kwt8bwhUmitOXc6fpT7SmFvRAtvxg84wUkg4Y/Gx++0j0z6StSeN\r\n" - "0EJz150jaHG6WV4HUqaWTb98Tm90IgXAU4AW2GBOlzFPiU5IY9jt+eXC2Q6yC/Zp\r\n" - "TL1LAcnL3Qa/OgLrHN0wiw1KFGD51WRPQ0Sh7QIDAQABo4IBJTCCASEwHQYDVR0O\r\n" - "BBYEFLV2DDARzseSQk1Mx1wsyKkM6AtkMB8GA1UdIwQYMBaAFOWdWTCCR1jMrPoI\r\n" - "VDaGezq1BE3wMA4GA1UdDwEB/wQEAwIBhjAdBgNVHSUEFjAUBggrBgEFBQcDAQYI\r\n" - "KwYBBQUHAwIwEgYDVR0TAQH/BAgwBgEB/wIBADA0BggrBgEFBQcBAQQoMCYwJAYI\r\n" - "KwYBBQUHMAGGGGh0dHA6Ly9vY3NwLmRpZ2ljZXJ0LmNvbTA6BgNVHR8EMzAxMC+g\r\n" - "LaArhilodHRwOi8vY3JsMy5kaWdpY2VydC5jb20vT21uaXJvb3QyMDI1LmNybDAq\r\n" - "BgNVHSAEIzAhMAgGBmeBDAECATAIBgZngQwBAgIwCwYJKwYBBAGCNyoBMA0GCSqG\r\n" - "SIb3DQEBCwUAA4IBAQCfK76SZ1vae4qt6P+dTQUO7bYNFUHR5hXcA2D59CJWnEj5\r\n" - "na7aKzyowKvQupW4yMH9fGNxtsh6iJswRqOOfZYC4/giBO/gNsBvwr8uDW7t1nYo\r\n" - "DYGHPpvnpxCM2mYfQFHq576/TmeYu1RZY29C4w8xYBlkAA8mDJfRhMCmehk7cN5F\r\n" - "JtyWRj2cZj/hOoI45TYDBChXpOlLZKIYiG1giY16vhCRi6zmPzEwv+tk156N6cGS\r\n" - "Vm44jTQ/rs1sa0JSYjzUaYngoFdZC4OfxnIkQvUIA4TOFmPzNPEFdjcZsgbeEz4T\r\n" - "cGHTBPK4R28F44qIMCtHRV55VMX53ev6P3hRddJb\r\n" - "-----END CERTIFICATE-----\r\n"; - ``` - - > 💁 If you are accessing your functions app locally, you don't need to do this. + Replace `` with the URL for your function app that you obtained in the last step of the last lesson, pointing to the IP address of your local machine that is running the function app. 1. Create a new file in the `src` folder called `language_understanding.h`. This will be used to define a class to send the recognized speech to your function app to be converted to seconds using LUIS. @@ -70,7 +31,6 @@ Microcontrollers don't natively have support for multiple threads in Arduino, so #include #include #include - #include #include "config.h" ``` @@ -89,7 +49,7 @@ Microcontrollers don't natively have support for multiple threads in Arduino, so LanguageUnderstanding languageUnderstanding; ``` -1. To call your functions app, you need to declare a WiFi client. The type you need depends on whether you are accessing the function app locally or deployed to the cloud. +1. To call your functions app, you need to declare a WiFi client. * If you are running the function app locally, add the following to the `private` section of the class: @@ -97,21 +57,6 @@ Microcontrollers don't natively have support for multiple threads in Arduino, so WiFiClient _client; ``` - * If you are running the function app in the cloud, add the following to the `private` section of the class: - - ```cpp - WiFiClientSecure _client; - ``` - - You will also need to set the certificate on this class, so add the following constructor to the `public` section: - - ```cpp - LanguageUnderstanding() - { - _client.setCACert(FUNCTIONS_CERTIFICATE); - } - ``` - 1. In the `public` section, declare a method called `GetTimerDuration` to call the functions app: ```cpp @@ -145,9 +90,9 @@ Microcontrollers don't natively have support for multiple threads in Arduino, so ```cpp HTTPClient httpClient; - httpClient.begin(_client, FUNCTION_URL); + httpClient.begin(_client, TEXT_TO_TIMER_FUNCTION_URL); - int httpResponseCode = httpClient.sendRequest("POST", body); + int httpResponseCode = httpClient.POST(body); ``` This makes a POST request to the functions app, passing the JSON body and getting the response code. @@ -227,6 +172,7 @@ The number of seconds can be used to set a timer. ```cpp void say(String text) { + Serial.print("Saying "); Serial.println(text); } ``` @@ -315,7 +261,7 @@ The number of seconds can be used to set a timer. timer.tick(); ``` -1. Build this code, upload it to your Wio Terminal and test it out through the serial monitor. Press the C button (the one on the left-hand side, closest to the power switch), and speak. 4 seconds of audio will be captured, converted to text, then sent to your function app, and a timer will be set. If you are running the functions app locally, make sure it is running. +1. Build this code, upload it to your Wio Terminal and test it out through the serial monitor. Once you see `Ready` in the serial monitor, press the C button (the one on the left-hand side, closest to the power switch), and speak. 4 seconds of audio will be captured, converted to text, then sent to your function app, and a timer will be set. Make sure your functions app is running locally. You will see when the timer starts, and when it ends. diff --git a/6-consumer/lessons/3-spoken-feedback/wio-terminal-text-to-speech.md b/6-consumer/lessons/3-spoken-feedback/wio-terminal-text-to-speech.md index e27369e6..133d5f1d 100644 --- a/6-consumer/lessons/3-spoken-feedback/wio-terminal-text-to-speech.md +++ b/6-consumer/lessons/3-spoken-feedback/wio-terminal-text-to-speech.md @@ -1,3 +1,546 @@ # Text to speech - Wio Terminal -Coming soon +In this part of the lesson, you will convert text to speech to provide spoken feedback. + +## Text to speech + +The speech services SDK that you used in the last lesson to convert speech to text can be used to convert text back to speech. + +## Get a list of voices + +When requesting speech, you need to provide the voice to use as speech can be generated using a variety of different voices. Each language supports a range of different voices, and you can get the list of supported voices for each language from the speech services SDK. The limitations of microcontrollers come into play here - the call to get the list of voices supported by the text to speech services is a JSON document of over 77KB in size, far to large to be processed by the Wio Terminal. At the time of writing, the full list contains 215 voices, each defined by a JSON document like the following: + +```json +{ + "Name": "Microsoft Server Speech Text to Speech Voice (en-US, AriaNeural)", + "DisplayName": "Aria", + "LocalName": "Aria", + "ShortName": "en-US-AriaNeural", + "Gender": "Female", + "Locale": "en-US", + "StyleList": [ + "chat", + "customerservice", + "narration-professional", + "newscast-casual", + "newscast-formal", + "cheerful", + "empathetic" + ], + "SampleRateHertz": "24000", + "VoiceType": "Neural", + "Status": "GA" +} +``` + +This JSON is for the **Aria** voice, which has multiple voice styles. All that is needed when converting text to speech is the shortname, `en-US-AriaNeural`. + +Instead of downloading and decoding this entire list on your microcontroller, you will need to write some more serverless code to retrieve the list of voices for the language you are using, and call this from your Wio Terminal. Your code can then pick an appropriate voice from the list, such as the first one it finds. + +### Task - create a serverless function to get a list of voices + +1. Open your `smart-timer-trigger` project in VS Code, and open the terminal ensuring the virtual environment is activated. If not, kill and re-create the terminal. + +1. Open the `local.settings.json` file and add settings for the speech API key and location: + + ```json + "SPEECH_KEY": "`", + "SPEECH_LOCATION": "" + ``` + + Replace `` with the API key for your speech service resource. Replace `` with the location you used when you created the speech service resource. + +1. Add a new HTTP trigger to this app called `get-voices` using the following command from inside the VS Code terminal in the root folder of the functions app project: + + ```sh + func new --name get-voices --template "HTTP trigger" + ``` + + This will create an HTTP trigger called `get-voices`. + +1. Replace the contents of the `__init__.py` file in the `get-voices` folder with the following: + + ```python + import json + import os + import requests + + import azure.functions as func + + def main(req: func.HttpRequest) -> func.HttpResponse: + location = os.environ['SPEECH_LOCATION'] + speech_key = os.environ['SPEECH_KEY'] + + req_body = req.get_json() + language = req_body['language'] + + url = f'https://{location}.tts.speech.microsoft.com/cognitiveservices/voices/list' + + headers = { + 'Ocp-Apim-Subscription-Key': speech_key + } + + response = requests.get(url, headers=headers) + voices_json = json.loads(response.text) + + voices = filter(lambda x: x['Locale'].lower() == language.lower(), voices_json) + voices = map(lambda x: x['ShortName'], voices) + + return func.HttpResponse(json.dumps(list(voices)), status_code=200) + ``` + + This code makes an HTTP request to the endpoint to get the voices. This voices list is a large block of JSON with voices for all languages, so the voices for the language passed in the request body are filtered out, then the shortname is extracted and returned as a JSON list. The shortname is the value needed to convert text to speech, so only this value is returned. + + > 💁 You can change the filter as necessary to select just the voices you want. + + This reduces the size of the data from 77KB (at the time of writing), to a much smaller JSON document. For example, for US voices this is 408 bytes. + +1. Run your function app locally. You can then call this using a tool like curl in the same way that you tested your `text-to-timer` HTTP trigger. Make sure to pass your language as a JSON body: + + ```json + {"language":""} + ``` + + Replace `` with your language, such as `en-GB`, or `zh-CN`. + +### Task - retrieve the voice from your Wio Terminal + +1. Open the `smart-timer` project in VS Code if it is not already open. + +1. Open the `config.h` header file and add the URL for your function app: + + ```cpp + const char *GET_VOICES_FUNCTION_URL = ""; + ``` + + Replace `` with the URL for the `get-voices` HTTP trigger on your function app. This will be the same as the value for `TEXT_TO_TIMER_FUNCTION_URL`, except with a function name of `get-voices` instead of `text-to-timer`. + +1. Create a new file in the `src` folder called `text_to_speech.h`. This will be used to define a class to convert from text to speech. + +1. Add the following include directives to the top of the new `text_to_speech.h` file: + + ```cpp + #pragma once + + #include + #include + #include + #include + #include + #include + #include + + #include "config.h" + #include "speech_to_text.h" + ``` + +1. Add the following code below this to declare the `TextToSpeech` class, along with an instance that can be used in the rest of the application: + + ```cpp + class TextToSpeech + { + public: + private: + }; + + TextToSpeech textToSpeech; + ``` + +1. To call your functions app, you need to declare a WiFi client. The type you need depends on whether you are accessing the function app locally or deployed to the cloud. + + * If you are running the function app locally, add the following to the `private` section of the class: + + ```cpp + WiFiClient _client; + ``` + + * If you are running the function app in the cloud, add the following to the `private` section of the class: + + ```cpp + WiFiClientSecure _client; + ``` + + You will also need to set the certificate on this class, so add the following constructor to the `public` section: + + ```cpp + TextToSpeech() + { + _client.setCACert(FUNCTIONS_CERTIFICATE); + } + ``` + +1. In the `private` section, add a field for the selected voice: + + ```cpp + String _voice; + ``` + +1. To the `public` section, add an `init` function that will get the first voice: + + ```cpp + void init() + { + } + ``` + +1. To get the voices, a JSON document needs to be sent to the function app with the language. Add the following code to the `init` function to create this JSON document: + + ```cpp + DynamicJsonDocument doc(1024); + doc["language"] = LANGUAGE; + + String body; + JsonObject obj = doc.as(); + serializeJson(obj, body); + ``` + +1. Next create an `HTTPClient`, then use it to call the functions app to get the voices, posting the JSON document: + + ```cpp + HTTPClient httpClient; + httpClient.begin(_client, GET_VOICES_FUNCTION_URL); + + int httpResponseCode = httpClient.POST(body); + ``` + +1. Below this add code to check the response code, and if it is 200 (success), then extract the list of voices, retrieving the first one from the list: + + ```cpp + if (httpResponseCode == 200) + { + String result = httpClient.getString(); + Serial.println(result); + + DynamicJsonDocument doc(1024); + deserializeJson(doc, result.c_str()); + + JsonArray obj = doc.as(); + _voice = obj[0].as(); + + Serial.print("Using voice "); + Serial.println(_voice); + } + else + { + Serial.print("Failed to get voices - error "); + Serial.println(httpResponseCode); + } + ``` + +1. After this, end the HTTP client connection: + + ```cpp + httpClient.end(); + ``` + +1. Open the `main.cpp` file, and add the following include directive at the top to include this new header file: + + ```cpp + #include "text_to_speech.h" + ``` + +1. In the `setup` function, underneath the call to `speechToText.init();`, add the following to initialize the `TextToSpeech` class: + + ```cpp + textToSpeech.init(); + ``` + +1. Build this code, upload it to your Wio Terminal and test it out through the serial monitor. Make sure your function app is running. + + You will see the list of available voices returned from the function app, along with the selected voice. + + ```output + --- Available filters and text transformations: colorize, debug, default, direct, hexlify, log2file, nocontrol, printable, send_on_enter, time + --- More details at http://bit.ly/pio-monitor-filters + --- Miniterm on /dev/cu.usbmodem1101 9600,8,N,1 --- + --- Quit: Ctrl+C | Menu: Ctrl+T | Help: Ctrl+T followed by Ctrl+H --- + Connecting to WiFi.. + Connected! + Got access token. + ["en-US-JennyNeural", "en-US-JennyMultilingualNeural", "en-US-GuyNeural", "en-US-AriaNeural", "en-US-AmberNeural", "en-US-AnaNeural", "en-US-AshleyNeural", "en-US-BrandonNeural", "en-US-ChristopherNeural", "en-US-CoraNeural", "en-US-ElizabethNeural", "en-US-EricNeural", "en-US-JacobNeural", "en-US-MichelleNeural", "en-US-MonicaNeural", "en-US-AriaRUS", "en-US-BenjaminRUS", "en-US-GuyRUS", "en-US-ZiraRUS"] + Using voice en-US-JennyNeural + Ready. + ``` + +## Convert text to speech + +Once you have a voice to use, it can be used to convert text to speech. The same memory limitations with voices also apply when converting speech to text, so you will need to write the speech to an SD card ready to be played over the ReSpeaker. + +> 💁 In earlier lessons in this project you used flash memory to store speech captured from the microphone. This lesson uses an SD card as is it easier to play audio from it using the Seeed audio libraries. + +There is also another limitation to consider, the available audio data from the speech service, and the formats that the Wio Terminal supports. Unlike full computers, audio libraries for microcontrollers can be very limited in the audio formats they support. For example, the Seeed Arduino Audio library that can play sound over the ReSpeaker only supports audio at a 44.1KHz sample rate. The Azure speech services can provide audio in a number of formats, but none of them use this sample rate, they only provide 8KHz, 16KHz, 24KHz and 48KHz. This means the audio needs to be re-sampled to 44.1KHz, something that would need more resources that the Wio Terminal has, especially memory. + +When needing to manipulate data like this, it is often better to use serverless code, especially if the data is sourced via a web call. The Wio Terminal can call a serverless function, passing in the text to convert, and the serverless function can both call the speech service to convert text to speech, as well as re-sample the audio to the required sample rate. It can then return the audio in the form the Wio Terminal needs to be stored on the SD card and played over the ReSpeaker. + +### Task - create a serverless function to convert text to speech + +1. Open your `smart-timer-trigger` project in VS Code, and open the terminal ensuring the virtual environment is activated. If not, kill and re-create the terminal. + +1. Add a new HTTP trigger to this app called `text-to-speech` using the following command from inside the VS Code terminal in the root folder of the functions app project: + + ```sh + func new --name text-to-speech --template "HTTP trigger" + ``` + + This will create an HTTP trigger called `text-to-speech`. + +1. The [librosa](https://librosa.org) Pip package has functions to re-sample audio, so add this to the `requirements.txt` file: + + ```sh + librosa + ``` + + Once this has been added, install the Pip packages using the following command from the VS Code terminal: + + ```sh + pip install -r requirements.txt + ``` + + > ⚠️ If you are using Linux, including Raspberry Pi OS, you may need to install `libsndfile` with the following command: + > + > ```sh + > sudo apt update + > sudo apt install libsndfile1-dev + > ``` + +1. To convert text to speech, you cannot use the speech API key directly, instead you need to request an access token, using the API key to authenticate the access token request. Open the `__init__.py` file from the `text-to-speech` folder and replace all the code in it with the following: + + ```python + import io + import os + import requests + + import librosa + import soundfile as sf + import azure.functions as func + + location = os.environ['SPEECH_LOCATION'] + speech_key = os.environ['SPEECH_KEY'] + + def get_access_token(): + headers = { + 'Ocp-Apim-Subscription-Key': speech_key + } + + token_endpoint = f'https://{location}.api.cognitive.microsoft.com/sts/v1.0/issuetoken' + response = requests.post(token_endpoint, headers=headers) + return str(response.text) + ``` + + This defines constants for the location and speech key that will be read from the settings. It then defines the `get_access_token` function that will retrieve an access token for the speech service. + +1. Below this code, add the following: + + ```python + playback_format = 'riff-48khz-16bit-mono-pcm' + + def main(req: func.HttpRequest) -> func.HttpResponse: + req_body = req.get_json() + language = req_body['language'] + voice = req_body['voice'] + text = req_body['text'] + + url = f'https://{location}.tts.speech.microsoft.com/cognitiveservices/v1' + + headers = { + 'Authorization': 'Bearer ' + get_access_token(), + 'Content-Type': 'application/ssml+xml', + 'X-Microsoft-OutputFormat': playback_format + } + + ssml = f'' + ssml += f'' + ssml += text + ssml += '' + ssml += '' + + response = requests.post(url, headers=headers, data=ssml.encode('utf-8')) + + raw_audio, sample_rate = librosa.load(io.BytesIO(response.content), sr=48000) + resampled = librosa.resample(raw_audio, sample_rate, 44100) + + output_buffer = io.BytesIO() + sf.write(output_buffer, resampled, 44100, 'PCM_16', format='wav') + output_buffer.seek(0) + + return func.HttpResponse(output_buffer.read(), status_code=200) + ``` + + This defines the HTTP trigger that converts the text to speech. It extracts the text to convert, the language and the voice from the JSON body set to the request, builds some SSML to request the speech, then calls the relevant REST API authenticating using the access token. This REST API call returns the audio encoded as 16-bit, 48KHz mono WAV file, defined by the value of `playback_format`, which is sent to the REST API call. + + This is then re-sampled by `librosa` from a sample rate of 48KHz to a sample rate of 44.1KHz, then this audio is saved to a binary buffer that is then returned. + +1. Run your function app locally, or deploy it to the cloud. You can then call this using a tool like curl in the same way that you tested your `text-to-timer` HTTP trigger. Make sure to pass the language, voice and text as the JSON body: + + ```json + { + "language": "", + "voice": "", + "text": "" + } + ``` + + Replace `` with your language, such as `en-GB`, or `zh-CN`. Replace `` with the voice you want to use. Replace `` with the text you want to convert to speech. You can save the output to a file and play it with any audio player that can play WAV files. + + For example, to convert "Hello" to speech using US English with the Jenny Neural voice, with the function app running locally, you can use the following curl command: + + ```sh + curl -X GET 'http://localhost:7071/api/text-to-speech' \ + -H 'Content-Type: application/json' \ + -o hello.wav \ + -d '{ + "language":"en-US", + "voice": "en-US-JennyNeural", + "text": "Hello" + }' + ``` + + This will save the audio to `hello.wav` in the current directory. + +### Task - retrieve the speech from your Wio Terminal + +1. Open the `smart-timer` project in VS Code if it is not already open. + +1. Open the `config.h` header file and add the URL for your function app: + + ```cpp + const char *TEXT_TO_SPEECH_FUNCTION_URL = ""; + ``` + + Replace `` with the URL for the `text-to-speech` HTTP trigger on your function app. This will be the same as the value for `TEXT_TO_TIMER_FUNCTION_URL`, except with a function name of `text-to-speech` instead of `text-to-timer`. + +1. Open the `text_to_speech.h` header file, and add the following method to the `public` section of the `TextToSpeech` class: + + ```cpp + void convertTextToSpeech(String text) + { + } + ``` + +1. To the `convertTextToSpeech` method, add the following code to create the JSON to send to the function app: + + ```cpp + DynamicJsonDocument doc(1024); + doc["language"] = LANGUAGE; + doc["voice"] = _voice; + doc["text"] = text; + + String body; + JsonObject obj = doc.as(); + serializeJson(obj, body); + ``` + + This writes the language, voice and text to the JSON document, then serializes it to a string. + +1. Below this, add the following code to call the function app: + + ```cpp + HTTPClient httpClient; + httpClient.begin(_client, TEXT_TO_SPEECH_FUNCTION_URL); + + int httpResponseCode = httpClient.POST(body); + ``` + + This creates an HTTPClient, then makes a POST request using the JSON document to the text to speech HTTP trigger. + +1. If the call works, the raw binary data returned from the function app call can be streamed to a file on the SD card. Add the following code to do this: + + ```cpp + if (httpResponseCode == 200) + { + File wav_file = SD.open("SPEECH.WAV", FILE_WRITE); + httpClient.writeToStream(&wav_file); + wav_file.close(); + } + else + { + Serial.print("Failed to get speech - error "); + Serial.println(httpResponseCode); + } + ``` + + This code checks the response, and if it is 200 (success), the binary data is streamed to a file in the root of the SD Card called `SPEECH.WAV`. + +1. At the end of this method, close the HTTP connection: + + ```cpp + httpClient.end(); + ``` + +### Task - play audio from your Wio Terminal + + + + + + + + + + + + + + + + +## Deploying your functions app to the cloud + + + #include + + + + * If you are running the function app in the cloud, add the following to the `private` section of the class: + + ```cpp + WiFiClientSecure _client; + ``` + + You will also need to set the certificate on this class, so add the following constructor to the `public` section: + + ```cpp + LanguageUnderstanding() + { + _client.setCACert(FUNCTIONS_CERTIFICATE); + } + ``` + +1. If you have deployed your functions app to the cloud, add the following certificate for `azurewebsites.net` to the `config.h` file. + + ```cpp + const char *FUNCTIONS_CERTIFICATE = + "-----BEGIN CERTIFICATE-----\r\n" + "MIIFWjCCBEKgAwIBAgIQDxSWXyAgaZlP1ceseIlB4jANBgkqhkiG9w0BAQsFADBa\r\n" + "MQswCQYDVQQGEwJJRTESMBAGA1UEChMJQmFsdGltb3JlMRMwEQYDVQQLEwpDeWJl\r\n" + "clRydXN0MSIwIAYDVQQDExlCYWx0aW1vcmUgQ3liZXJUcnVzdCBSb290MB4XDTIw\r\n" + "MDcyMTIzMDAwMFoXDTI0MTAwODA3MDAwMFowTzELMAkGA1UEBhMCVVMxHjAcBgNV\r\n" + "BAoTFU1pY3Jvc29mdCBDb3Jwb3JhdGlvbjEgMB4GA1UEAxMXTWljcm9zb2Z0IFJT\r\n" + "QSBUTFMgQ0EgMDEwggIiMA0GCSqGSIb3DQEBAQUAA4ICDwAwggIKAoICAQCqYnfP\r\n" + "mmOyBoTzkDb0mfMUUavqlQo7Rgb9EUEf/lsGWMk4bgj8T0RIzTqk970eouKVuL5R\r\n" + "IMW/snBjXXgMQ8ApzWRJCZbar879BV8rKpHoAW4uGJssnNABf2n17j9TiFy6BWy+\r\n" + "IhVnFILyLNK+W2M3zK9gheiWa2uACKhuvgCca5Vw/OQYErEdG7LBEzFnMzTmJcli\r\n" + "W1iCdXby/vI/OxbfqkKD4zJtm45DJvC9Dh+hpzqvLMiK5uo/+aXSJY+SqhoIEpz+\r\n" + "rErHw+uAlKuHFtEjSeeku8eR3+Z5ND9BSqc6JtLqb0bjOHPm5dSRrgt4nnil75bj\r\n" + "c9j3lWXpBb9PXP9Sp/nPCK+nTQmZwHGjUnqlO9ebAVQD47ZisFonnDAmjrZNVqEX\r\n" + "F3p7laEHrFMxttYuD81BdOzxAbL9Rb/8MeFGQjE2Qx65qgVfhH+RsYuuD9dUw/3w\r\n" + "ZAhq05yO6nk07AM9c+AbNtRoEcdZcLCHfMDcbkXKNs5DJncCqXAN6LhXVERCw/us\r\n" + "G2MmCMLSIx9/kwt8bwhUmitOXc6fpT7SmFvRAtvxg84wUkg4Y/Gx++0j0z6StSeN\r\n" + "0EJz150jaHG6WV4HUqaWTb98Tm90IgXAU4AW2GBOlzFPiU5IY9jt+eXC2Q6yC/Zp\r\n" + "TL1LAcnL3Qa/OgLrHN0wiw1KFGD51WRPQ0Sh7QIDAQABo4IBJTCCASEwHQYDVR0O\r\n" + "BBYEFLV2DDARzseSQk1Mx1wsyKkM6AtkMB8GA1UdIwQYMBaAFOWdWTCCR1jMrPoI\r\n" + "VDaGezq1BE3wMA4GA1UdDwEB/wQEAwIBhjAdBgNVHSUEFjAUBggrBgEFBQcDAQYI\r\n" + "KwYBBQUHAwIwEgYDVR0TAQH/BAgwBgEB/wIBADA0BggrBgEFBQcBAQQoMCYwJAYI\r\n" + "KwYBBQUHMAGGGGh0dHA6Ly9vY3NwLmRpZ2ljZXJ0LmNvbTA6BgNVHR8EMzAxMC+g\r\n" + "LaArhilodHRwOi8vY3JsMy5kaWdpY2VydC5jb20vT21uaXJvb3QyMDI1LmNybDAq\r\n" + "BgNVHSAEIzAhMAgGBmeBDAECATAIBgZngQwBAgIwCwYJKwYBBAGCNyoBMA0GCSqG\r\n" + "SIb3DQEBCwUAA4IBAQCfK76SZ1vae4qt6P+dTQUO7bYNFUHR5hXcA2D59CJWnEj5\r\n" + "na7aKzyowKvQupW4yMH9fGNxtsh6iJswRqOOfZYC4/giBO/gNsBvwr8uDW7t1nYo\r\n" + "DYGHPpvnpxCM2mYfQFHq576/TmeYu1RZY29C4w8xYBlkAA8mDJfRhMCmehk7cN5F\r\n" + "JtyWRj2cZj/hOoI45TYDBChXpOlLZKIYiG1giY16vhCRi6zmPzEwv+tk156N6cGS\r\n" + "Vm44jTQ/rs1sa0JSYjzUaYngoFdZC4OfxnIkQvUIA4TOFmPzNPEFdjcZsgbeEz4T\r\n" + "cGHTBPK4R28F44qIMCtHRV55VMX53ev6P3hRddJb\r\n" + "-----END CERTIFICATE-----\r\n"; + ``` + + > 💁 If you are accessing your functions app locally, you don't need to do this. \ No newline at end of file diff --git a/6-consumer/lessons/4-multiple-language-support/wio-terminal-translate-speech.md b/6-consumer/lessons/4-multiple-language-support/wio-terminal-translate-speech.md index 7d25fa0f..11a7085d 100644 --- a/6-consumer/lessons/4-multiple-language-support/wio-terminal-translate-speech.md +++ b/6-consumer/lessons/4-multiple-language-support/wio-terminal-translate-speech.md @@ -1,3 +1,181 @@ # Translate speech - Wio Terminal -Coming soon! +In this part of the lesson, you will write code to translate text using the translator service. + +## Convert text to speech using the translator service + +The speech service REST API doesn't support direct translations, instead you can use the Translator service to translate the text generated by the speech to text service, and the text of the spoken response. This service has a REST API you can use to translate the text. + +### Task - use the translator resource to translate text + +1. Open the `smart-timer` project in VS Code if it is not already open. + +1. Your smart timer will have 2 languages set - the language of the server that was used to train LUIS, and the language spoken by the user. Update the `LANGUAGE` constant in the `config.h` header file to be the language that will be spoken by the used, and add a new constant called `SERVER_LANGUAGE` for the language used to train LUIS: + + ```cpp + const char *LANGUAGE = ""; + const char *SERVER_LANGUAGE = ""; + ``` + + Replace `` with the locale name for language you will be speaking in, for example `fr-FR` for French, or `zn-HK` for Cantonese. + + Replace `` with the locale name for language used to train LUIS. + + You can find a list of the supported languages and their locale names in the [Language and voice support documentation on Microsoft docs](https://docs.microsoft.com/azure/cognitive-services/speech-service/language-support?WT.mc_id=academic-17441-jabenn#speech-to-text). + + > 💁 If you don't speak multiple languages you can use a service like [Bing Translate](https://www.bing.com/translator) or [Google Translate](https://translate.google.com) to translate from your preferred language to a language of your choice. These services can then play audio of the translated text. + > + > For example, if you train LUIS in English, but want to use French as the user language, you can translate sentences like "set a 2 minute and 27 second timer" from English into French using Bing Translate, then use the **Listen translation** button to speak the translation into your microphone. + > + > ![The listen translation button on Bing translate](../../../images/bing-translate.png) + +1. Add the translator API key below the `SPEECH_API_KEY`: + + ```cpp + const char *TRANSLATOR_API_KEY = ""; + ``` + + Replace `` with the API key for your translator service resource. + +1. Add the translator URL below the `VOICE_URL`: + + ```cpp + const char *TRANSLATE_URL = "https://api.cognitive.microsofttranslator.com/translate?api-version=3.0&from=%s&to=%s"; + ``` + + This is a global URL and doesn't need your location in it, instead the location of the translator resource is passed as a header. This URL will need the from and to languages - so the language that the text is in and the language you want it translated to. + + + + + + + + + + + + + + + + + + + + + +1. Above the `say` function, define a `translate_text` function that will translate text from the server language to the user language: + + ```python + def translate_text(text, from_language, to_language): + ``` + + The from and to languages are passed to this function - your app needs to convert from user language to server language when recognizing speech, and from server language to user language when provided spoken feedback. + +1. Inside this function, define the URL and headers for the REST API call: + + ```python + url = f'https://api.cognitive.microsofttranslator.com/translate?api-version=3.0' + + headers = { + 'Ocp-Apim-Subscription-Key': translator_api_key, + 'Ocp-Apim-Subscription-Region': location, + 'Content-type': 'application/json' + } + ``` + + The URL for this API is not location specific, instead the location is passed in as a header. The API key is used directly, so unlike the speech service there is no need to get an access token from the token issuer API. + +1. Below this define the parameters and body for the call: + + ```python + params = { + 'from': from_language, + 'to': to_language + } + + body = [{ + 'text' : text + }] + ``` + + The `params` defines the parameters to pass to the API call, passing the from and to languages. This call will translate text in the `from` language into the `to` language. + + The `body` contains the text to translate. This is an array, as multiple blocks of text can be translated in the same call. + +1. Make the call the REST API, and get the response: + + ```python + response = requests.post(url, headers=headers, params=params, json=body) + ``` + + The response that comes back is a JSON array, with one item that contains the translations. This item has an array for translations of all the items passed in the body. + + ```json + [ + { + "translations": [ + { + "text": "Set a 2 minute 27 second timer.", + "to": "en" + } + ] + } + ] + ``` + +1. Return the `test` property from the first translation from the first item in the array: + + ```python + return response.json()[0]['translations'][0]['text'] + ``` + +1. Update the `while True` loop to translate the text from the call to `convert_speech_to_text` from the user language to the server language: + + ```python + if len(text) > 0: + print('Original:', text) + text = translate_text(text, language, server_language) + print('Translated:', text) + + message = Message(json.dumps({ 'speech': text })) + device_client.send_message(message) + ``` + + This code also prints the original and translated versions of the text to the console. + +1. Update the `say` function to translate the text to say from the server language to the user language: + + ```python + def say(text): + print('Original:', text) + text = translate_text(text, server_language, language) + print('Translated:', text) + speech = get_speech(text) + play_speech(speech) + ``` + + This code also prints the original and translated versions of the text to the console. + +1. Run your code. Ensure your function app is running, and request a timer in the user language, either by speaking that language yourself, or using a translation app. + + ```output + pi@raspberrypi:~/smart-timer $ python3 app.py + Connecting + Connected + Using voice fr-FR-DeniseNeural + Original: Définir une minuterie de 2 minutes et 27 secondes. + Translated: Set a timer of 2 minutes and 27 seconds. + Original: 2 minute 27 second timer started. + Translated: 2 minute 27 seconde minute a commencé. + Original: Times up on your 2 minute 27 second timer. + Translated: Chronométrant votre minuterie de 2 minutes 27 secondes. + ``` + + > 💁 Due to the different ways of saying something in different languages, you may get translations that are slightly different to the examples you gave LUIS. If this is the case, add more examples to LUIS, retrain then re-publish the model. + +> 💁 You can find this code in the [code/pi](code/pi) folder. + +😀 Your multi-lingual timer program was a success! +