You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
IoT-For-Beginners/5-retail/lessons/2-check-stock-device/wio-terminal-object-detecto...

103 lines
4.3 KiB

Lesson 20 (#140) * Adding content * Update en.json * Update README.md * Update TRANSLATIONS.md * Adding lesson tempolates * Fixing code files with each others code in * Update README.md * Adding lesson 16 * Adding virtual camera * Adding Wio Terminal camera capture * Adding wio terminal code * Adding SBC classification to lesson 16 * Adding challenge, review and assignment * Adding images and using new Azure icons * Update README.md * Update iot-reference-architecture.png * Adding structure for JulyOT links * Removing icons * Sketchnotes! * Create lesson-1.png * Starting on lesson 18 * Updated sketch * Adding virtual distance sensor * Adding Wio Terminal image classification * Update README.md * Adding structure for project 6 and wio terminal distance sensor * Adding some of the smart timer stuff * Updating sketchnotes * Adding virtual device speech to text * Adding chapter 21 * Language tweaks * Lesson 22 stuff * Update en.json * Bumping seeed libraries * Adding functions lab to lesson 22 * Almost done with LUIS * Update README.md * Reverting sunlight sensor change Fixes #88 * Structure * Adding speech to text lab for Pi * Adding virtual device text to speech lab * Finishing lesson 23 * Clarifying privacy Fixes #99 * Update README.md * Update hardware.md * Update README.md * Fixing some code samples that were wrong * Adding more on translation * Adding more on translator * Update README.md * Update README.md * Adding public access to the container * First part of retail object detection * More on stock lesson * Tweaks to maps lesson * Update README.md * Update pi-sensor.md * IoT Edge install stuffs * Notes on consumer groups and not running the event monitor at the same time * Assignment for object detector * Memory notes for speech to text * Migrating LUIS to an HTTP trigger * Adding Wio Terminal speech to text * Changing smart timer to functions from hub * Changing a param to body to avoid URL encoding * Update README.md * Tweaks before IoT Show * Adding sketchnote links * Adding object detection labs * Adding more on object detection * More on stock detection * Finishing stock counting
3 years ago
# Call your object detector from your IoT device - Wio Terminal
Once your object detector has been published, it can be used from your IoT device.
## Copy the image classifier project
The majority of your stock detector is the same as the image classifier you created in a previous lesson.
### Task - copy the image classifier project
1. Connect your ArduCam your Wio Terminal, following the steps from [lesson 2 of the manufacturing project](../../../4-manufacturing/lessons/2-check-fruit-from-device/wio-terminal-camera.md#task---connect-the-camera).
You might also want to fix the camera in a single position, for example, by hanging the cable over a box or can, or fixing the camera to a box with double-sided tape.
1. Create a brand new Wio Terminal project using PlatformIO. Call this project `stock-counter`.
1. Replicate the steps from [lesson 2 of the manufacturing project](../../../4-manufacturing/lessons/2-check-fruit-from-device/README.md#task---capture-an-image-using-an-iot-device) to capture images from the camera.
1. Replicate the steps from [lesson 2 of the manufacturing project](../../../4-manufacturing/lessons/2-check-fruit-from-device/README.md#task---classify-images-from-your-iot-device) to call the image classifier. The majority of this code will be re-used to detect objects.
## Change the code from a classifier to an image detector
The code you used to classify images is very similar to the code to detect objects. The main difference is the URL that is called that you obtained from Custom Vision, and the results of the call.
### Task - change the code from a classifier to an image detector
1. Add the following include directive to the top of the `main.cpp` file:
```cpp
#include <vector>
```
1. Rename the `classifyImage` function to `detectStock`, both the name of the function and the call in the `buttonPressed` function.
1. Above the `detectStock` function, declare a threshold to filter out any detections that have a low probability:
```cpp
const float threshold = 0.3f;
```
Unlike an image classifier that only returns one result per tag, the object detector will return multiple results, so any with a low probability need to be filtered out.
1. Above the `detectStock` function, declare a function to process the predictions:
```cpp
void processPredictions(std::vector<JsonVariant> &predictions)
{
for(JsonVariant prediction : predictions)
{
String tag = prediction["tagName"].as<String>();
float probability = prediction["probability"].as<float>();
char buff[32];
sprintf(buff, "%s:\t%.2f%%", tag.c_str(), probability * 100.0);
Serial.println(buff);
}
}
```
This takes a list of predictions and prints them to the serial monitor.
1. In the `detectStock` function, replace the contents of the `for` loop that loops through the predictions with the following:
```cpp
std::vector<JsonVariant> passed_predictions;
for(JsonVariant prediction : predictions)
{
float probability = prediction["probability"].as<float>();
if (probability > threshold)
{
passed_predictions.push_back(prediction);
}
}
processPredictions(passed_predictions);
```
This loops through the predictions, comparing the probability to the threshold. All predictions that have a probability higher than the threshold are added to a `list` and passed to the `processPredictions` function.
1. Upload and run your code. Point the camera at objects on a shelf and press the C button. You will see the output in the serial monitor:
```output
Connecting to WiFi..
Connected!
Image captured
Image read to buffer with length 17416
tomato paste: 35.84%
tomato paste: 35.87%
tomato paste: 34.11%
tomato paste: 35.16%
```
> 💁 You may need to adjust the `threshold` to an appropriate value for your images.
You will be able to see the image that was taken, and these values in the **Predictions** tab in Custom Vision.
![4 cans of tomato paste on a shelf with predictions for the 4 detections of 35.8%, 33.5%, 25.7% and 16.6%](../../../images/custom-vision-stock-prediction.png)
> 💁 You can find this code in the [code-detect/wio-terminal](code-detect/wio-terminal) folder.
😀 Your stock counter program was a success!