Lesson 23 (#103)
* Adding content * Update en.json * Update README.md * Update TRANSLATIONS.md * Adding lesson tempolates * Fixing code files with each others code in * Update README.md * Adding lesson 16 * Adding virtual camera * Adding Wio Terminal camera capture * Adding wio terminal code * Adding SBC classification to lesson 16 * Adding challenge, review and assignment * Adding images and using new Azure icons * Update README.md * Update iot-reference-architecture.png * Adding structure for JulyOT links * Removing icons * Sketchnotes! * Create lesson-1.png * Starting on lesson 18 * Updated sketch * Adding virtual distance sensor * Adding Wio Terminal image classification * Update README.md * Adding structure for project 6 and wio terminal distance sensor * Adding some of the smart timer stuff * Updating sketchnotes * Adding virtual device speech to text * Adding chapter 21 * Language tweaks * Lesson 22 stuff * Update en.json * Bumping seeed libraries * Adding functions lab to lesson 22 * Almost done with LUIS * Update README.md * Reverting sunlight sensor change Fixes #88 * Structure * Adding speech to text lab for Pi * Adding virtual device text to speech lab * Finishing lesson 23pull/116/head
Jim Bennett
3 years ago
committed by
GitHub
parent
63a0723186
commit
6a7159ebce
@ -1,9 +1,12 @@
|
|||||||
#
|
# Cancel the timer
|
||||||
|
|
||||||
## Instructions
|
## Instructions
|
||||||
|
|
||||||
|
In the assignment for the last lesson, you added a cancel timer intent to LUIS. For this assignment you need to handle this intent in the serverless code, send a command to the IoT device, then cancel the timer.
|
||||||
|
|
||||||
## Rubric
|
## Rubric
|
||||||
|
|
||||||
| Criteria | Exemplary | Adequate | Needs Improvement |
|
| Criteria | Exemplary | Adequate | Needs Improvement |
|
||||||
| -------- | --------- | -------- | ----------------- |
|
| -------- | --------- | -------- | ----------------- |
|
||||||
| | | | |
|
| Handle the intent in serverless code and send a command | Was able to handle the intent and send a command to the device | Was able to handle the intent but was unable to send the command to the device | Was unable to handle the intent |
|
||||||
|
| Cancel the timer on the device | Was able to receive the command and cancel the timer | Was able to receive the command but not cancel the timer | Was unable to receive the command |
|
||||||
|
|||||||
@ -0,0 +1,15 @@
|
|||||||
|
{
|
||||||
|
"version": "2.0",
|
||||||
|
"logging": {
|
||||||
|
"applicationInsights": {
|
||||||
|
"samplingSettings": {
|
||||||
|
"isEnabled": true,
|
||||||
|
"excludedTypes": "Request"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"extensionBundle": {
|
||||||
|
"id": "Microsoft.Azure.Functions.ExtensionBundle",
|
||||||
|
"version": "[2.*, 3.0.0)"
|
||||||
|
}
|
||||||
|
}
|
||||||
@ -0,0 +1,12 @@
|
|||||||
|
{
|
||||||
|
"IsEncrypted": false,
|
||||||
|
"Values": {
|
||||||
|
"FUNCTIONS_WORKER_RUNTIME": "python",
|
||||||
|
"AzureWebJobsStorage": "UseDevelopmentStorage=true",
|
||||||
|
"IOT_HUB_CONNECTION_STRING": "<connection string>",
|
||||||
|
"LUIS_KEY": "<primary key>",
|
||||||
|
"LUIS_ENDPOINT_URL": "<endpoint url>",
|
||||||
|
"LUIS_APP_ID": "<app id>",
|
||||||
|
"REGISTRY_MANAGER_CONNECTION_STRING": "<connection string>"
|
||||||
|
}
|
||||||
|
}
|
||||||
@ -0,0 +1,4 @@
|
|||||||
|
# Do not include azure-functions-worker as it may conflict with the Azure Functions platform
|
||||||
|
|
||||||
|
azure-functions
|
||||||
|
azure-cognitiveservices-language-luis
|
||||||
@ -0,0 +1,60 @@
|
|||||||
|
from typing import List
|
||||||
|
import logging
|
||||||
|
|
||||||
|
import azure.functions as func
|
||||||
|
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
from azure.cognitiveservices.language.luis.runtime import LUISRuntimeClient
|
||||||
|
from msrest.authentication import CognitiveServicesCredentials
|
||||||
|
|
||||||
|
from azure.iot.hub import IoTHubRegistryManager
|
||||||
|
from azure.iot.hub.models import CloudToDeviceMethod
|
||||||
|
|
||||||
|
def main(events: List[func.EventHubEvent]):
|
||||||
|
luis_key = os.environ['LUIS_KEY']
|
||||||
|
endpoint_url = os.environ['LUIS_ENDPOINT_URL']
|
||||||
|
app_id = os.environ['LUIS_APP_ID']
|
||||||
|
registry_manager_connection_string = os.environ['REGISTRY_MANAGER_CONNECTION_STRING']
|
||||||
|
|
||||||
|
credentials = CognitiveServicesCredentials(luis_key)
|
||||||
|
client = LUISRuntimeClient(endpoint=endpoint_url, credentials=credentials)
|
||||||
|
|
||||||
|
for event in events:
|
||||||
|
logging.info('Python EventHub trigger processed an event: %s',
|
||||||
|
event.get_body().decode('utf-8'))
|
||||||
|
|
||||||
|
device_id = event.iothub_metadata['connection-device-id']
|
||||||
|
|
||||||
|
event_body = json.loads(event.get_body().decode('utf-8'))
|
||||||
|
prediction_request = { 'query' : event_body['speech'] }
|
||||||
|
|
||||||
|
prediction_response = client.prediction.get_slot_prediction(app_id, 'Staging', prediction_request)
|
||||||
|
|
||||||
|
if prediction_response.prediction.top_intent == 'set timer':
|
||||||
|
numbers = prediction_response.prediction.entities['number']
|
||||||
|
time_units = prediction_response.prediction.entities['time unit']
|
||||||
|
total_seconds = 0
|
||||||
|
|
||||||
|
for i in range(0, len(numbers)):
|
||||||
|
number = numbers[i]
|
||||||
|
time_unit = time_units[i][0]
|
||||||
|
|
||||||
|
if time_unit == 'minute':
|
||||||
|
total_seconds += number * 60
|
||||||
|
else:
|
||||||
|
total_seconds += number
|
||||||
|
|
||||||
|
logging.info(f'Timer required for {total_seconds} seconds')
|
||||||
|
|
||||||
|
payload = {
|
||||||
|
'seconds': total_seconds
|
||||||
|
}
|
||||||
|
direct_method = CloudToDeviceMethod(method_name='set-timer', payload=json.dumps(payload))
|
||||||
|
|
||||||
|
registry_manager_connection_string = os.environ['REGISTRY_MANAGER_CONNECTION_STRING']
|
||||||
|
registry_manager = IoTHubRegistryManager(registry_manager_connection_string)
|
||||||
|
|
||||||
|
registry_manager.invoke_device_method(device_id, direct_method)
|
||||||
|
|
||||||
|
|
||||||
@ -0,0 +1,15 @@
|
|||||||
|
{
|
||||||
|
"scriptFile": "__init__.py",
|
||||||
|
"bindings": [
|
||||||
|
{
|
||||||
|
"type": "eventHubTrigger",
|
||||||
|
"name": "events",
|
||||||
|
"direction": "in",
|
||||||
|
"eventHubName": "samples-workitems",
|
||||||
|
"connection": "IOT_HUB_CONNECTION_STRING",
|
||||||
|
"cardinality": "many",
|
||||||
|
"consumerGroup": "$Default",
|
||||||
|
"dataType": "binary"
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
@ -0,0 +1,184 @@
|
|||||||
|
import io
|
||||||
|
import json
|
||||||
|
import pyaudio
|
||||||
|
import requests
|
||||||
|
import time
|
||||||
|
import wave
|
||||||
|
import threading
|
||||||
|
|
||||||
|
from azure.iot.device import IoTHubDeviceClient, Message, MethodResponse
|
||||||
|
|
||||||
|
from grove.factory import Factory
|
||||||
|
button = Factory.getButton('GPIO-HIGH', 5)
|
||||||
|
|
||||||
|
audio = pyaudio.PyAudio()
|
||||||
|
microphone_card_number = 1
|
||||||
|
speaker_card_number = 1
|
||||||
|
rate = 16000
|
||||||
|
|
||||||
|
def capture_audio():
|
||||||
|
stream = audio.open(format = pyaudio.paInt16,
|
||||||
|
rate = rate,
|
||||||
|
channels = 1,
|
||||||
|
input_device_index = microphone_card_number,
|
||||||
|
input = True,
|
||||||
|
frames_per_buffer = 4096)
|
||||||
|
|
||||||
|
frames = []
|
||||||
|
|
||||||
|
while button.is_pressed():
|
||||||
|
frames.append(stream.read(4096))
|
||||||
|
|
||||||
|
stream.stop_stream()
|
||||||
|
stream.close()
|
||||||
|
|
||||||
|
wav_buffer = io.BytesIO()
|
||||||
|
with wave.open(wav_buffer, 'wb') as wavefile:
|
||||||
|
wavefile.setnchannels(1)
|
||||||
|
wavefile.setsampwidth(audio.get_sample_size(pyaudio.paInt16))
|
||||||
|
wavefile.setframerate(rate)
|
||||||
|
wavefile.writeframes(b''.join(frames))
|
||||||
|
wav_buffer.seek(0)
|
||||||
|
|
||||||
|
return wav_buffer
|
||||||
|
|
||||||
|
api_key = '<key>'
|
||||||
|
location = '<location>'
|
||||||
|
language = '<language>'
|
||||||
|
connection_string = '<connection_string>'
|
||||||
|
|
||||||
|
device_client = IoTHubDeviceClient.create_from_connection_string(connection_string)
|
||||||
|
|
||||||
|
print('Connecting')
|
||||||
|
device_client.connect()
|
||||||
|
print('Connected')
|
||||||
|
|
||||||
|
def get_access_token():
|
||||||
|
headers = {
|
||||||
|
'Ocp-Apim-Subscription-Key': api_key
|
||||||
|
}
|
||||||
|
|
||||||
|
token_endpoint = f'https://{location}.api.cognitive.microsoft.com/sts/v1.0/issuetoken'
|
||||||
|
response = requests.post(token_endpoint, headers=headers)
|
||||||
|
return str(response.text)
|
||||||
|
|
||||||
|
def convert_speech_to_text(buffer):
|
||||||
|
url = f'https://{location}.stt.speech.microsoft.com/speech/recognition/conversation/cognitiveservices/v1'
|
||||||
|
|
||||||
|
headers = {
|
||||||
|
'Authorization': 'Bearer ' + get_access_token(),
|
||||||
|
'Content-Type': f'audio/wav; codecs=audio/pcm; samplerate={rate}',
|
||||||
|
'Accept': 'application/json;text/xml'
|
||||||
|
}
|
||||||
|
|
||||||
|
params = {
|
||||||
|
'language': language
|
||||||
|
}
|
||||||
|
|
||||||
|
response = requests.post(url, headers=headers, params=params, data=buffer)
|
||||||
|
response_json = json.loads(response.text)
|
||||||
|
|
||||||
|
if response_json['RecognitionStatus'] == 'Success':
|
||||||
|
return response_json['DisplayText']
|
||||||
|
else:
|
||||||
|
return ''
|
||||||
|
|
||||||
|
def get_voice():
|
||||||
|
url = f'https://{location}.tts.speech.microsoft.com/cognitiveservices/voices/list'
|
||||||
|
|
||||||
|
headers = {
|
||||||
|
'Authorization': 'Bearer ' + get_access_token()
|
||||||
|
}
|
||||||
|
|
||||||
|
response = requests.get(url, headers=headers)
|
||||||
|
voices_json = json.loads(response.text)
|
||||||
|
|
||||||
|
first_voice = next(x for x in voices_json if x['Locale'].lower() == language.lower())
|
||||||
|
return first_voice['ShortName']
|
||||||
|
|
||||||
|
voice = get_voice()
|
||||||
|
print(f"Using voice {voice}")
|
||||||
|
|
||||||
|
playback_format = 'riff-48khz-16bit-mono-pcm'
|
||||||
|
|
||||||
|
def get_speech(text):
|
||||||
|
url = f'https://{location}.tts.speech.microsoft.com/cognitiveservices/v1'
|
||||||
|
|
||||||
|
headers = {
|
||||||
|
'Authorization': 'Bearer ' + get_access_token(),
|
||||||
|
'Content-Type': 'application/ssml+xml',
|
||||||
|
'X-Microsoft-OutputFormat': playback_format
|
||||||
|
}
|
||||||
|
|
||||||
|
ssml = f'<speak version=\'1.0\' xml:lang=\'{language}\'>'
|
||||||
|
ssml += f'<voice xml:lang=\'{language}\' name=\'{voice}\'>'
|
||||||
|
ssml += text
|
||||||
|
ssml += '</voice>'
|
||||||
|
ssml += '</speak>'
|
||||||
|
|
||||||
|
response = requests.post(url, headers=headers, data=ssml.encode('utf-8'))
|
||||||
|
return io.BytesIO(response.content)
|
||||||
|
|
||||||
|
def play_speech(speech):
|
||||||
|
with wave.open(speech, 'rb') as wave_file:
|
||||||
|
stream = audio.open(format=audio.get_format_from_width(wave_file.getsampwidth()),
|
||||||
|
channels=wave_file.getnchannels(),
|
||||||
|
rate=wave_file.getframerate(),
|
||||||
|
output_device_index=speaker_card_number,
|
||||||
|
output=True)
|
||||||
|
|
||||||
|
data = wave_file.readframes(4096)
|
||||||
|
|
||||||
|
while len(data) > 0:
|
||||||
|
stream.write(data)
|
||||||
|
data = wave_file.readframes(4096)
|
||||||
|
|
||||||
|
stream.stop_stream()
|
||||||
|
stream.close()
|
||||||
|
|
||||||
|
def say(text):
|
||||||
|
speech = get_speech(text)
|
||||||
|
play_speech(speech)
|
||||||
|
|
||||||
|
def announce_timer(minutes, seconds):
|
||||||
|
announcement = 'Times up on your '
|
||||||
|
if minutes > 0:
|
||||||
|
announcement += f'{minutes} minute'
|
||||||
|
if seconds > 0:
|
||||||
|
announcement += f'{seconds} second'
|
||||||
|
announcement += ' timer.'
|
||||||
|
say(announcement)
|
||||||
|
|
||||||
|
def create_timer(total_seconds):
|
||||||
|
minutes, seconds = divmod(total_seconds, 60)
|
||||||
|
threading.Timer(total_seconds, announce_timer, args=[minutes, seconds]).start()
|
||||||
|
announcement = ''
|
||||||
|
if minutes > 0:
|
||||||
|
announcement += f'{minutes} minute'
|
||||||
|
if seconds > 0:
|
||||||
|
announcement += f'{seconds} second'
|
||||||
|
announcement += ' timer started.'
|
||||||
|
say(announcement)
|
||||||
|
|
||||||
|
def handle_method_request(request):
|
||||||
|
if request.name == 'set-timer':
|
||||||
|
payload = json.loads(request.payload)
|
||||||
|
seconds = payload['seconds']
|
||||||
|
if seconds > 0:
|
||||||
|
create_timer(payload['seconds'])
|
||||||
|
|
||||||
|
method_response = MethodResponse.create_from_method_request(request, 200)
|
||||||
|
device_client.send_method_response(method_response)
|
||||||
|
|
||||||
|
device_client.on_method_request_received = handle_method_request
|
||||||
|
|
||||||
|
while True:
|
||||||
|
while not button.is_pressed():
|
||||||
|
time.sleep(.1)
|
||||||
|
|
||||||
|
buffer = capture_audio()
|
||||||
|
text = convert_speech_to_text(buffer)
|
||||||
|
if len(text) > 0:
|
||||||
|
print(text)
|
||||||
|
message = Message(json.dumps({ 'speech': text }))
|
||||||
|
device_client.send_message(message)
|
||||||
@ -0,0 +1,86 @@
|
|||||||
|
import json
|
||||||
|
import threading
|
||||||
|
import time
|
||||||
|
from azure.cognitiveservices.speech import SpeechConfig, SpeechRecognizer, SpeechSynthesizer
|
||||||
|
from azure.iot.device import IoTHubDeviceClient, Message, MethodResponse
|
||||||
|
|
||||||
|
api_key = '<key>'
|
||||||
|
location = '<location>'
|
||||||
|
language = '<language>'
|
||||||
|
connection_string = '<connection_string>'
|
||||||
|
|
||||||
|
device_client = IoTHubDeviceClient.create_from_connection_string(connection_string)
|
||||||
|
|
||||||
|
print('Connecting')
|
||||||
|
device_client.connect()
|
||||||
|
print('Connected')
|
||||||
|
|
||||||
|
recognizer_config = SpeechConfig(subscription=api_key,
|
||||||
|
region=location,
|
||||||
|
speech_recognition_language=language)
|
||||||
|
|
||||||
|
recognizer = SpeechRecognizer(speech_config=recognizer_config)
|
||||||
|
|
||||||
|
def recognized(args):
|
||||||
|
if len(args.result.text) > 0:
|
||||||
|
message = Message(json.dumps({ 'speech': args.result.text }))
|
||||||
|
device_client.send_message(message)
|
||||||
|
|
||||||
|
recognizer.recognized.connect(recognized)
|
||||||
|
|
||||||
|
recognizer.start_continuous_recognition()
|
||||||
|
|
||||||
|
speech_config = SpeechConfig(subscription=api_key,
|
||||||
|
region=location)
|
||||||
|
speech_config.speech_synthesis_language = language
|
||||||
|
speech_synthesizer = SpeechSynthesizer(speech_config=speech_config)
|
||||||
|
|
||||||
|
voices = speech_synthesizer.get_voices_async().get().voices
|
||||||
|
first_voice = next(x for x in voices if x.locale.lower() == language.lower())
|
||||||
|
speech_config.speech_synthesis_voice_name = first_voice.short_name
|
||||||
|
|
||||||
|
def say(text):
|
||||||
|
ssml = f'<speak version=\'1.0\' xml:lang=\'{language}\'>'
|
||||||
|
ssml += f'<voice xml:lang=\'{language}\' name=\'{first_voice.short_name}\'>'
|
||||||
|
ssml += text
|
||||||
|
ssml += '</voice>'
|
||||||
|
ssml += '</speak>'
|
||||||
|
|
||||||
|
recognizer.stop_continuous_recognition()
|
||||||
|
speech_synthesizer.speak_ssml(ssml)
|
||||||
|
recognizer.start_continuous_recognition()
|
||||||
|
|
||||||
|
def announce_timer(minutes, seconds):
|
||||||
|
announcement = 'Times up on your '
|
||||||
|
if minutes > 0:
|
||||||
|
announcement += f'{minutes} minute'
|
||||||
|
if seconds > 0:
|
||||||
|
announcement += f'{seconds} second'
|
||||||
|
announcement += ' timer.'
|
||||||
|
say(announcement)
|
||||||
|
|
||||||
|
def create_timer(total_seconds):
|
||||||
|
minutes, seconds = divmod(total_seconds, 60)
|
||||||
|
threading.Timer(total_seconds, announce_timer, args=[minutes, seconds]).start()
|
||||||
|
announcement = ''
|
||||||
|
if minutes > 0:
|
||||||
|
announcement += f'{minutes} minute'
|
||||||
|
if seconds > 0:
|
||||||
|
announcement += f'{seconds} second'
|
||||||
|
announcement += ' timer started.'
|
||||||
|
say(announcement)
|
||||||
|
|
||||||
|
def handle_method_request(request):
|
||||||
|
if request.name == 'set-timer':
|
||||||
|
payload = json.loads(request.payload)
|
||||||
|
seconds = payload['seconds']
|
||||||
|
if seconds > 0:
|
||||||
|
create_timer(payload['seconds'])
|
||||||
|
|
||||||
|
method_response = MethodResponse.create_from_method_request(request, 200)
|
||||||
|
device_client.send_method_response(method_response)
|
||||||
|
|
||||||
|
device_client.on_method_request_received = handle_method_request
|
||||||
|
|
||||||
|
while True:
|
||||||
|
time.sleep(1)
|
||||||
@ -0,0 +1,130 @@
|
|||||||
|
import io
|
||||||
|
import json
|
||||||
|
import pyaudio
|
||||||
|
import requests
|
||||||
|
import time
|
||||||
|
import wave
|
||||||
|
import threading
|
||||||
|
|
||||||
|
from azure.iot.device import IoTHubDeviceClient, Message, MethodResponse
|
||||||
|
|
||||||
|
from grove.factory import Factory
|
||||||
|
button = Factory.getButton('GPIO-HIGH', 5)
|
||||||
|
|
||||||
|
audio = pyaudio.PyAudio()
|
||||||
|
microphone_card_number = 1
|
||||||
|
speaker_card_number = 1
|
||||||
|
rate = 16000
|
||||||
|
|
||||||
|
def capture_audio():
|
||||||
|
stream = audio.open(format = pyaudio.paInt16,
|
||||||
|
rate = rate,
|
||||||
|
channels = 1,
|
||||||
|
input_device_index = microphone_card_number,
|
||||||
|
input = True,
|
||||||
|
frames_per_buffer = 4096)
|
||||||
|
|
||||||
|
frames = []
|
||||||
|
|
||||||
|
while button.is_pressed():
|
||||||
|
frames.append(stream.read(4096))
|
||||||
|
|
||||||
|
stream.stop_stream()
|
||||||
|
stream.close()
|
||||||
|
|
||||||
|
wav_buffer = io.BytesIO()
|
||||||
|
with wave.open(wav_buffer, 'wb') as wavefile:
|
||||||
|
wavefile.setnchannels(1)
|
||||||
|
wavefile.setsampwidth(audio.get_sample_size(pyaudio.paInt16))
|
||||||
|
wavefile.setframerate(rate)
|
||||||
|
wavefile.writeframes(b''.join(frames))
|
||||||
|
wav_buffer.seek(0)
|
||||||
|
|
||||||
|
return wav_buffer
|
||||||
|
|
||||||
|
api_key = '<key>'
|
||||||
|
location = '<location>'
|
||||||
|
language = '<language>'
|
||||||
|
connection_string = '<connection_string>'
|
||||||
|
|
||||||
|
device_client = IoTHubDeviceClient.create_from_connection_string(connection_string)
|
||||||
|
|
||||||
|
print('Connecting')
|
||||||
|
device_client.connect()
|
||||||
|
print('Connected')
|
||||||
|
|
||||||
|
def get_access_token():
|
||||||
|
headers = {
|
||||||
|
'Ocp-Apim-Subscription-Key': api_key
|
||||||
|
}
|
||||||
|
|
||||||
|
token_endpoint = f'https://{location}.api.cognitive.microsoft.com/sts/v1.0/issuetoken'
|
||||||
|
response = requests.post(token_endpoint, headers=headers)
|
||||||
|
return str(response.text)
|
||||||
|
|
||||||
|
def convert_speech_to_text(buffer):
|
||||||
|
url = f'https://{location}.stt.speech.microsoft.com/speech/recognition/conversation/cognitiveservices/v1'
|
||||||
|
|
||||||
|
headers = {
|
||||||
|
'Authorization': 'Bearer ' + get_access_token(),
|
||||||
|
'Content-Type': f'audio/wav; codecs=audio/pcm; samplerate={rate}',
|
||||||
|
'Accept': 'application/json;text/xml'
|
||||||
|
}
|
||||||
|
|
||||||
|
params = {
|
||||||
|
'language': language
|
||||||
|
}
|
||||||
|
|
||||||
|
response = requests.post(url, headers=headers, params=params, data=buffer)
|
||||||
|
response_json = json.loads(response.text)
|
||||||
|
|
||||||
|
if response_json['RecognitionStatus'] == 'Success':
|
||||||
|
return response_json['DisplayText']
|
||||||
|
else:
|
||||||
|
return ''
|
||||||
|
|
||||||
|
def say(text):
|
||||||
|
print(text)
|
||||||
|
|
||||||
|
def announce_timer(minutes, seconds):
|
||||||
|
announcement = 'Times up on your '
|
||||||
|
if minutes > 0:
|
||||||
|
announcement += f'{minutes} minute'
|
||||||
|
if seconds > 0:
|
||||||
|
announcement += f'{seconds} second'
|
||||||
|
announcement += ' timer.'
|
||||||
|
say(announcement)
|
||||||
|
|
||||||
|
def create_timer(total_seconds):
|
||||||
|
minutes, seconds = divmod(total_seconds, 60)
|
||||||
|
threading.Timer(total_seconds, announce_timer, args=[minutes, seconds]).start()
|
||||||
|
announcement = ''
|
||||||
|
if minutes > 0:
|
||||||
|
announcement += f'{minutes} minute'
|
||||||
|
if seconds > 0:
|
||||||
|
announcement += f'{seconds} second'
|
||||||
|
announcement += ' timer started.'
|
||||||
|
say(announcement)
|
||||||
|
|
||||||
|
def handle_method_request(request):
|
||||||
|
if request.name == 'set-timer':
|
||||||
|
payload = json.loads(request.payload)
|
||||||
|
seconds = payload['seconds']
|
||||||
|
if seconds > 0:
|
||||||
|
create_timer(payload['seconds'])
|
||||||
|
|
||||||
|
method_response = MethodResponse.create_from_method_request(request, 200)
|
||||||
|
device_client.send_method_response(method_response)
|
||||||
|
|
||||||
|
device_client.on_method_request_received = handle_method_request
|
||||||
|
|
||||||
|
while True:
|
||||||
|
while not button.is_pressed():
|
||||||
|
time.sleep(.1)
|
||||||
|
|
||||||
|
buffer = capture_audio()
|
||||||
|
text = convert_speech_to_text(buffer)
|
||||||
|
if len(text) > 0:
|
||||||
|
print(text)
|
||||||
|
message = Message(json.dumps({ 'speech': text }))
|
||||||
|
device_client.send_message(message)
|
||||||
@ -0,0 +1,69 @@
|
|||||||
|
import json
|
||||||
|
import threading
|
||||||
|
import time
|
||||||
|
from azure.cognitiveservices.speech import SpeechConfig, SpeechRecognizer
|
||||||
|
from azure.iot.device import IoTHubDeviceClient, Message, MethodResponse
|
||||||
|
|
||||||
|
api_key = '<key>'
|
||||||
|
location = '<location>'
|
||||||
|
language = '<language>'
|
||||||
|
connection_string = '<connection_string>'
|
||||||
|
|
||||||
|
device_client = IoTHubDeviceClient.create_from_connection_string(connection_string)
|
||||||
|
|
||||||
|
print('Connecting')
|
||||||
|
device_client.connect()
|
||||||
|
print('Connected')
|
||||||
|
|
||||||
|
recognizer_config = SpeechConfig(subscription=api_key,
|
||||||
|
region=location,
|
||||||
|
speech_recognition_language=language)
|
||||||
|
|
||||||
|
recognizer = SpeechRecognizer(speech_config=recognizer_config)
|
||||||
|
|
||||||
|
def recognized(args):
|
||||||
|
if len(args.result.text) > 0:
|
||||||
|
message = Message(json.dumps({ 'speech': args.result.text }))
|
||||||
|
device_client.send_message(message)
|
||||||
|
|
||||||
|
recognizer.recognized.connect(recognized)
|
||||||
|
|
||||||
|
recognizer.start_continuous_recognition()
|
||||||
|
|
||||||
|
def say(text):
|
||||||
|
print(text)
|
||||||
|
|
||||||
|
def announce_timer(minutes, seconds):
|
||||||
|
announcement = 'Times up on your '
|
||||||
|
if minutes > 0:
|
||||||
|
announcement += f'{minutes} minute'
|
||||||
|
if seconds > 0:
|
||||||
|
announcement += f'{seconds} second'
|
||||||
|
announcement += ' timer.'
|
||||||
|
say(announcement)
|
||||||
|
|
||||||
|
def create_timer(total_seconds):
|
||||||
|
minutes, seconds = divmod(total_seconds, 60)
|
||||||
|
threading.Timer(total_seconds, announce_timer, args=[minutes, seconds]).start()
|
||||||
|
announcement = ''
|
||||||
|
if minutes > 0:
|
||||||
|
announcement += f'{minutes} minute'
|
||||||
|
if seconds > 0:
|
||||||
|
announcement += f'{seconds} second'
|
||||||
|
announcement += ' timer started.'
|
||||||
|
say(announcement)
|
||||||
|
|
||||||
|
def handle_method_request(request):
|
||||||
|
if request.name == 'set-timer':
|
||||||
|
payload = json.loads(request.payload)
|
||||||
|
seconds = payload['seconds']
|
||||||
|
if seconds > 0:
|
||||||
|
create_timer(payload['seconds'])
|
||||||
|
|
||||||
|
method_response = MethodResponse.create_from_method_request(request, 200)
|
||||||
|
device_client.send_method_response(method_response)
|
||||||
|
|
||||||
|
device_client.on_method_request_received = handle_method_request
|
||||||
|
|
||||||
|
while True:
|
||||||
|
time.sleep(1)
|
||||||
@ -0,0 +1,140 @@
|
|||||||
|
# Text to speech - Raspberry Pi
|
||||||
|
|
||||||
|
In this part of the lesson, you will write code to convert text to speech using the speech service.
|
||||||
|
|
||||||
|
## Convert text to speech using the speech service
|
||||||
|
|
||||||
|
The text can be sent to the speech service using the REST API to get speech as an audio file that can be played back on your IoT device. When requesting speech, you need to provide the voice to use as speech can be generated using a variety of different voices.
|
||||||
|
|
||||||
|
Each language supports a range of different voices, and you can make a REST request against the speech service to get the list of supported voices for each language.
|
||||||
|
|
||||||
|
### Task - get a voice
|
||||||
|
|
||||||
|
1. Add the following code above the `say` function to request the list of voices for a language:
|
||||||
|
|
||||||
|
```python
|
||||||
|
def get_voice():
|
||||||
|
url = f'https://{location}.tts.speech.microsoft.com/cognitiveservices/voices/list'
|
||||||
|
|
||||||
|
headers = {
|
||||||
|
'Authorization': 'Bearer ' + get_access_token()
|
||||||
|
}
|
||||||
|
|
||||||
|
response = requests.get(url, headers=headers)
|
||||||
|
voices_json = json.loads(response.text)
|
||||||
|
|
||||||
|
first_voice = next(x for x in voices_json if x['Locale'].lower() == language.lower() and x['VoiceType'] == 'Neural')
|
||||||
|
return first_voice['ShortName']
|
||||||
|
|
||||||
|
voice = get_voice()
|
||||||
|
print(f"Using voice {voice}")
|
||||||
|
```
|
||||||
|
|
||||||
|
This code defines a function called `get_voice` that uses the speech service to get a list of voices. It then finds the first voice that matches the language that is being used.
|
||||||
|
|
||||||
|
This function is then called to store the first voice, and the voice name is printed to the console. This voice can be requested once and the value used for every call to convert text to speech.
|
||||||
|
|
||||||
|
> 💁 You can get the full list of supported voices from the [Language and voice support documentation on Microsoft Docs](https://docs.microsoft.com/azure/cognitive-services/speech-service/language-support?WT.mc_id=academic-17441-jabenn#text-to-speech). If you want to use a specific voice, then you can remove this function and hard code the voice to the voice name from this documentation. For example:
|
||||||
|
>
|
||||||
|
> ```python
|
||||||
|
> voice = 'hi-IN-SwaraNeural'
|
||||||
|
> ```
|
||||||
|
|
||||||
|
### Task - convert text to speech
|
||||||
|
|
||||||
|
1. Below this, define a constant for the audio format to be retrieved from the speech services. When you request audio, you can do it in a range of different formats.
|
||||||
|
|
||||||
|
```python
|
||||||
|
playback_format = 'riff-48khz-16bit-mono-pcm'
|
||||||
|
```
|
||||||
|
|
||||||
|
The format you can use depends on your hardware. If you get `Invalid sample rate` errors when playing the audio then change this to another value. You can find the list of supported values in the [Text to speech REST API documentation on Microsoft Docs](https://docs.microsoft.com/azure/cognitive-services/speech-service/rest-text-to-speech?WT.mc_id=academic-17441-jabenn#audio-outputs). You will need to use `riff` format audio, and the values to try are `riff-16khz-16bit-mono-pcm`, `riff-24khz-16bit-mono-pcm` and `riff-48khz-16bit-mono-pcm`.
|
||||||
|
|
||||||
|
1. Below this, declare a function called `get_speech` that will convert the text to speech using the speech service REST API:
|
||||||
|
|
||||||
|
```python
|
||||||
|
def get_speech(text):
|
||||||
|
```
|
||||||
|
|
||||||
|
1. In the `get_speech` function, define the URL to call and the headers to pass:
|
||||||
|
|
||||||
|
```python
|
||||||
|
url = f'https://{location}.tts.speech.microsoft.com/cognitiveservices/v1'
|
||||||
|
|
||||||
|
headers = {
|
||||||
|
'Authorization': 'Bearer ' + get_access_token(),
|
||||||
|
'Content-Type': 'application/ssml+xml',
|
||||||
|
'X-Microsoft-OutputFormat': playback_format
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
This set the headers to use a generated access token, set the content to SSML and define the audio format needed.
|
||||||
|
|
||||||
|
1. Below this, define the SSML to send to the REST API:
|
||||||
|
|
||||||
|
```python
|
||||||
|
ssml = f'<speak version=\'1.0\' xml:lang=\'{language}\'>'
|
||||||
|
ssml += f'<voice xml:lang=\'{language}\' name=\'{voice}\'>'
|
||||||
|
ssml += text
|
||||||
|
ssml += '</voice>'
|
||||||
|
ssml += '</speak>'
|
||||||
|
```
|
||||||
|
|
||||||
|
This SSML sets the language and the voice to use, along with the text to convert.
|
||||||
|
|
||||||
|
1. Finally, add code in this function to make the REST request and return the binary audio data:
|
||||||
|
|
||||||
|
```python
|
||||||
|
response = requests.post(url, headers=headers, data=ssml.encode('utf-8'))
|
||||||
|
return io.BytesIO(response.content)
|
||||||
|
```
|
||||||
|
|
||||||
|
### Task - play the audio
|
||||||
|
|
||||||
|
1. Below the `get_speech` function, define a new function to play the audio returned by the REST API call:
|
||||||
|
|
||||||
|
```python
|
||||||
|
def play_speech(speech):
|
||||||
|
```
|
||||||
|
|
||||||
|
1. The `speech` passed to this function will be the binary audio data returned from the REST API. Use the following code to open this as a wave file and pass it to PyAudio to play the audio:
|
||||||
|
|
||||||
|
```python
|
||||||
|
def play_speech(speech):
|
||||||
|
with wave.open(speech, 'rb') as wave_file:
|
||||||
|
stream = audio.open(format=audio.get_format_from_width(wave_file.getsampwidth()),
|
||||||
|
channels=wave_file.getnchannels(),
|
||||||
|
rate=wave_file.getframerate(),
|
||||||
|
output_device_index=speaker_card_number,
|
||||||
|
output=True)
|
||||||
|
|
||||||
|
data = wave_file.readframes(4096)
|
||||||
|
|
||||||
|
while len(data) > 0:
|
||||||
|
stream.write(data)
|
||||||
|
data = wave_file.readframes(4096)
|
||||||
|
|
||||||
|
stream.stop_stream()
|
||||||
|
stream.close()
|
||||||
|
```
|
||||||
|
|
||||||
|
This code uses a PyAudio stream, the same as capturing audio. The difference here is the stream is set as an output stream, and data is read from the audio data and pushed to the stream.
|
||||||
|
|
||||||
|
Rather than hard coding the stream details such as the sample rate, it is read from the audio data.
|
||||||
|
|
||||||
|
1. Replace the contents of the `say` function to the following:
|
||||||
|
|
||||||
|
```python
|
||||||
|
speech = get_speech(text)
|
||||||
|
play_speech(speech)
|
||||||
|
```
|
||||||
|
|
||||||
|
This code converts the text to speech as binary audio data, and plays the audio.
|
||||||
|
|
||||||
|
1. Run the app, and ensure the function app is also running. Set some timers, and you will hear a spoken response saying that your timer has been set, then another spoken response when the timer is complete.
|
||||||
|
|
||||||
|
If you get `Invalid sample rate` errors, change the `playback_format` as described above.
|
||||||
|
|
||||||
|
> 💁 You can find this code in the [code-spoken-response/pi](code-spoken-response/pi) folder.
|
||||||
|
|
||||||
|
😀 Your timer program was a success!
|
||||||
@ -0,0 +1,97 @@
|
|||||||
|
# Set a timer - Virtual IoT Hardware and Raspberry Pi
|
||||||
|
|
||||||
|
In this part of the lesson, you will set a timer on your virtual IoT device or Raspberry Pi based off a command from the IoT Hub.
|
||||||
|
|
||||||
|
## Set a timer
|
||||||
|
|
||||||
|
The command sent from the serverless function contains the time for the timer in seconds as the payload. This time can be used to set a timer.
|
||||||
|
|
||||||
|
Timers can be set using the Python `threading.Timer` class. This class takes a delay time and a function, and after the delay time, the function is executed.
|
||||||
|
|
||||||
|
### Task - set a timer
|
||||||
|
|
||||||
|
1. Open the `smart-timer` project in VS Code, and ensure the virtual environment is loaded in the terminal if you are using a virtual IoT device.
|
||||||
|
|
||||||
|
1. Add the following import statement at the top of the file to import the threading Python library:
|
||||||
|
|
||||||
|
```python
|
||||||
|
import threading
|
||||||
|
```
|
||||||
|
|
||||||
|
1. Above the `handle_method_request` function that handles the method request, add a function to speak a response. Fow now this will just write to the console, but later in this lesson this will speak the text.
|
||||||
|
|
||||||
|
```python
|
||||||
|
def say(text):
|
||||||
|
print(text)
|
||||||
|
```
|
||||||
|
|
||||||
|
1. Below this add a function that will be called by a timer to announce that the timer is complete:
|
||||||
|
|
||||||
|
```python
|
||||||
|
def announce_timer(minutes, seconds):
|
||||||
|
announcement = 'Times up on your '
|
||||||
|
if minutes > 0:
|
||||||
|
announcement += f'{minutes} minute'
|
||||||
|
if seconds > 0:
|
||||||
|
announcement += f'{seconds} second'
|
||||||
|
announcement += ' timer.'
|
||||||
|
say(announcement)
|
||||||
|
```
|
||||||
|
|
||||||
|
This function takes the number of minutes and seconds for the timer, and builds a sentence to say that the timer is complete. It will check the number of minutes and seconds, and only include each time unit if it has a number. For example, if the number of minutes is 0 then only seconds are included in the message. This sentence is then sent to the `say` function.
|
||||||
|
|
||||||
|
1. Below this, add the following `create_timer` function to create a timer:
|
||||||
|
|
||||||
|
```python
|
||||||
|
def create_timer(seconds):
|
||||||
|
minutes, seconds = divmod(seconds, 60)
|
||||||
|
threading.Timer(seconds, announce_timer, args=[minutes, seconds]).start()
|
||||||
|
```
|
||||||
|
|
||||||
|
This function takes the total number of seconds for the timer that will be sent in the command, and converts this to minutes and seconds. It then creates and starts a timer object using the total number of seconds, passing in the `announce_timer` function and a list containing the minutes and seconds. When the timer elapses, it will call the `announce_timer` function, and pass the contents of this list as the parameters - so the first item in the list gets passes as the `minutes` parameter, and the second item as the `seconds` parameter.
|
||||||
|
|
||||||
|
1. To the end of the `create_timer` function, add some code to build a message to be spoken to the user to announce that the timer is starting:
|
||||||
|
|
||||||
|
```python
|
||||||
|
announcement = ''
|
||||||
|
if minutes > 0:
|
||||||
|
announcement += f'{minutes} minute'
|
||||||
|
if seconds > 0:
|
||||||
|
announcement += f'{seconds} second'
|
||||||
|
announcement += ' timer started.'
|
||||||
|
say(announcement)
|
||||||
|
```
|
||||||
|
|
||||||
|
Again, this only includes the time unit that has a value. This sentence is then sent to the `say` function.
|
||||||
|
|
||||||
|
1. At the start of the `handle_method_request` function, add the following code to check that the `set-timer` direct method was requested:
|
||||||
|
|
||||||
|
```python
|
||||||
|
if request.name == 'set-timer':
|
||||||
|
```
|
||||||
|
|
||||||
|
1. Inside this `if` statement, extract the timer time in seconds from the payload and use this to create a timer:
|
||||||
|
|
||||||
|
```python
|
||||||
|
payload = json.loads(request.payload)
|
||||||
|
seconds = payload['seconds']
|
||||||
|
if seconds > 0:
|
||||||
|
create_timer(payload['seconds'])
|
||||||
|
```
|
||||||
|
|
||||||
|
The timer is only created if the number of seconds is greater than 0
|
||||||
|
|
||||||
|
1. Run the app, and ensure the function app is also running. Set some timers, and the output will show the timer being set, and then will show when it elapses:
|
||||||
|
|
||||||
|
```output
|
||||||
|
pi@raspberrypi:~/smart-timer $ python3 app.py
|
||||||
|
Connecting
|
||||||
|
Connected
|
||||||
|
Set a one minute 4 second timer.
|
||||||
|
1 minute, 4 second timer started
|
||||||
|
Times up on your 1 minute, 4 second timer
|
||||||
|
```
|
||||||
|
|
||||||
|
> 💁 You can find this code in the [code-timer/pi](code-timer/pi) or [code-timer/virtual-iot-device](code-timer/virtual-iot-device) folder.
|
||||||
|
|
||||||
|
😀 Your timer program was a success!
|
||||||
@ -0,0 +1,72 @@
|
|||||||
|
# Text to speech - Virtual IoT device
|
||||||
|
|
||||||
|
In this part of the lesson, you will write code to convert text to speech using the speech service.
|
||||||
|
|
||||||
|
## Convert text to speech
|
||||||
|
|
||||||
|
The speech services SDK that you used in the last lesson to convert speech to text can be used to convert text back to speech. When requesting speech, you need to provide the voice to use as speech can be generated using a variety of different voices.
|
||||||
|
|
||||||
|
Each language supports a range of different voices, and you can get the list of supported voices for each language from the speech services SDK.
|
||||||
|
|
||||||
|
### Task - convert text to speech
|
||||||
|
|
||||||
|
1. Import the `SpeechSynthesizer` from the `azure.cognitiveservices.speech` package by adding it to the existing imports:
|
||||||
|
|
||||||
|
```python
|
||||||
|
from azure.cognitiveservices.speech import SpeechConfig, SpeechRecognizer, SpeechSynthesizer
|
||||||
|
```
|
||||||
|
|
||||||
|
1. Above the `say` function, create a speech configuration to use with the speech synthesizer:
|
||||||
|
|
||||||
|
```python
|
||||||
|
speech_config = SpeechConfig(subscription=api_key,
|
||||||
|
region=location)
|
||||||
|
speech_config.speech_synthesis_language = language
|
||||||
|
speech_synthesizer = SpeechSynthesizer(speech_config=speech_config)
|
||||||
|
```
|
||||||
|
|
||||||
|
This uses the same API key, location and language that was used by the recognizer.
|
||||||
|
|
||||||
|
1. Below this, add the following code to get a voice and set it on the speech config:
|
||||||
|
|
||||||
|
```python
|
||||||
|
voices = speech_synthesizer.get_voices_async().get().voices
|
||||||
|
first_voice = next(x for x in voices if x.locale.lower() == language.lower())
|
||||||
|
speech_config.speech_synthesis_voice_name = first_voice.short_name
|
||||||
|
```
|
||||||
|
|
||||||
|
This retrieves a list of all the available voices, then finds the first voice that matches the language that is being used.
|
||||||
|
|
||||||
|
> 💁 You can get the full list of supported voices from the [Language and voice support documentation on Microsoft Docs](https://docs.microsoft.com/azure/cognitive-services/speech-service/language-support?WT.mc_id=academic-17441-jabenn#text-to-speech). If you want to use a specific voice, then you can remove this function and hard code the voice to the voice name from this documentation. For example:
|
||||||
|
>
|
||||||
|
> ```python
|
||||||
|
> speech_config.speech_synthesis_voice_name = 'hi-IN-SwaraNeural'
|
||||||
|
> ```
|
||||||
|
|
||||||
|
1. Update the contents of the `say` function to generate SSML for the response:
|
||||||
|
|
||||||
|
```python
|
||||||
|
ssml = f'<speak version=\'1.0\' xml:lang=\'{language}\'>'
|
||||||
|
ssml += f'<voice xml:lang=\'{language}\' name=\'{first_voice.short_name}\'>'
|
||||||
|
ssml += text
|
||||||
|
ssml += '</voice>'
|
||||||
|
ssml += '</speak>'
|
||||||
|
```
|
||||||
|
|
||||||
|
1. Below this, stop the speech recognition, speak the SSML, then start the recognition again:
|
||||||
|
|
||||||
|
```python
|
||||||
|
recognizer.stop_continuous_recognition()
|
||||||
|
speech_synthesizer.speak_ssml(ssml)
|
||||||
|
recognizer.start_continuous_recognition()
|
||||||
|
```
|
||||||
|
|
||||||
|
The recognition is stopped whilst the text is spoken to avoid the announcement of the timer starting being detected, sent to LUIS and possibly interpreted as a request to set a new timer.
|
||||||
|
|
||||||
|
> 💁 You can test this out by commenting out the lines to stop and restart the recognition. Set one timer, and you may find the announcement sets a new timer, which causes a new announcement, leading to a new timer, and so on for ever!
|
||||||
|
|
||||||
|
1. Run the app, and ensure the function app is also running. Set some timers, and you will hear a spoken response saying that your timer has been set, then another spoken response when the timer is complete.
|
||||||
|
|
||||||
|
> 💁 You can find this code in the [code-spoken-response/virtual-iot-device](code-spoken-response/virtual-iot-device) folder.
|
||||||
|
|
||||||
|
😀 Your timer program was a success!
|
||||||
@ -0,0 +1,3 @@
|
|||||||
|
# Set a timer - Wio Terminal
|
||||||
|
|
||||||
|
Coming soon
|
||||||
@ -0,0 +1,3 @@
|
|||||||
|
# Text to speech - Wio Terminal
|
||||||
|
|
||||||
|
Coming soon
|
||||||
Binary file not shown.
|
After Width: | Height: | Size: 11 KiB |
Loading…
Reference in new issue