This article is a tutorial on how to use NanoEdge™ AI Studio to create a dual tone multi frequency (DTMF)[1] classifier running on the STWIN SensorTile Wireless Industrial Node development kit (STEVAL-STWINKT1B).
This tutorial shows you how to program the STWIN for datalogging; how to use NanoEdge™ AI Studio to collect data, label data, find the best algorithm, run the Emulator, generate the library files for deployment; how to use STM32CubeIDE to build project and program the STWIN, and finally run the classifier on STWIN. The main process of this tutorial is shown in the flowchart below:
1. Prerequisites
1.1. Hardware
- STEVAL-STWINKT1B, the STWIN development kit,
- Windows® powered laptop/PC (Windows® 7, 8, or 10),
- Two Micro-USB cables (one to connect the STWIN to PC, and another one for the STLINK-V3MINI)
- STLINK-V3MINI, which is included in the dev kit.
1.2. Software
1.2.1. Source Code
- A firmware package
STWIN-DTMF-Classifier-wiki-package.zip
is ready to download from st.com, please download the zip file here. The package includes:
- Project source code in subfolder
STWIN_DTMF_Classifier
- Two Pre-compiled binary files:
STWIN_DTMF_Classifier-datalog.bin
for dataloggingSTWIN_DTMF_Classifier-detection.bin
for tone detection
- Library files generated by NanoEdge™ AI Studio
libneai_STWIN-DTMF-Classifier.zip
- Project source code in subfolder
1.2.2. NanoEdge™ AI Studio
- NanoEdge™ AI Studio is a new Machine Learning (ML) technology to bring true innovation easily to the end-users. In just a few steps, developers can create an optimal ML library for their project, based on a minimal amount of data.
- Download NanoEdge™ AI Studio as a free trial version for the STMicroelectronics boards.
1.2.3. STM32CubeIDE
- STM32CubeIDE is an all-in-one multi-OS development tool, which is part of the STM32Cube software ecosystem. It is an advanced C/C++ development platform with peripheral configuration, code generation, code compilation, and debug features for STM32 microcontrollers and microprocessors.
- Download STM32CubeIDE from st.com
1.2.4. Tera Term
- Tera Term is an open-source and freely available software terminal emulator, which is used to display the classification results.
- Users can download and install the latest version available from Tera Term.
1.2.5. Tone Generation App
Install the corresponding app on your smart phone for DTMF tone generation:
- iOS: Dialer by DigitalSand.
- Android DTMF Tone Generator by Guillaume Desbieys.
2. Step 1 – Program STWIN for datalogging
The first step is to connect the STWIN to your PC and program it for datalogging mode with the provided binary file.
2.1. Board connection
There are three connections to make on the STWIN, as shown in the picture:
- Connect the STDC14 cable between the STWIN and STLINK-V3MINI.
- Plug the USB cable micro-B side into the STWIN.
- Plug the USB cable micro-B side into the STLINK-V3MINI.
- Plug the STWIN USB cable into PC.
- Plug the STLINK-V3MINI USB cable into PC.
2.2. Program STWIN using binary file
The easiest way to program the STWIN is using the drag-and-drop. Once you have the pre-built binary file in your local disk, the STWIN can be very easily programmed with the binary by simply performing a drag-and-drop action:
- In Windows File Explorer, find the binary file
..\STWIN-DTMF-Classifier-wiki-package\binary\STWIN_DTMF_Classifier-datalog.bin
- Left click on the file and hold the left button on your mouse
- Find the STLINK_V3M disk in the Navigation pane on the left hand side, drag the binary to that disk and release the left button
- When the transfer is done, the STWIN is programmed successfully.
3. Step 2 – Create a Classification Project in NanoEdge™ AI Studio
In this step, we will create a classfication project in NanoEdge™ AI Studio, and configure the project settings.
Open NanoEdge™ AI Studio on PC, you should see the welcome interface below and then follow the steps:
- Click on Create a new project.
- Choose Classification for Project type.
- Give a project name.
- Add a brief description for the project.
- Click on the drop down menu for Target to open the pop-up window, then:
- Click on STMicroelectronics tab
- Choose STEVAL-STWINKT1B Cortex-M4 as target
- Click on Validate to go back to the main interface.
- Leave the Max RAM and Max Flash as default.
- Choose Microphone sensor for Sensor type.
- Finally, click on Create to create the project.
4. Step 3 – Datalogging in NanoEdge™ AI Studio
In this step, we will use the programmed STWIN and NanoEdge™ AI Studio to collect the data for each class. The tone generation app will be used to generate the sound when each key is pressed.
Once the project was created, you should see the window as shown below, and you are ready for datalogging.
- By default, there is already a Class1 added.
- Click on Choose Signals to open the "Import your signal" window for this class. Then choose From serial (USB). You should see the "Import your signal" window as shown below:
- Select your Serial / COM port. Refresh if needed.
- Select 921600 for Baudrate
- Check the box to define Max lines of 200 signals. By doing this, the data logger will automatically stop when there are 200 signals collected.
- Leave the Delimiter as default. Open the Tone Generation app on your smart phone, adjust phone volume to about 50%, let the phone speaker face to the STWIN, and press on key "1". Keep pressing the key during data logging. Then click on the Record button to start data logging. The real-time data and plotting will be shown.
- Once 200 signals are collected, the data logger will stop and you can release the key on the smart phone. click on the Validate import for to validate the data.
- Then you will return to the main interface, Click on "pencil" icon to rename the class, e.g. "key1". The information of data is shown in the middle column, and the plotting of data is shown on the right column. You can also turn on "Show FFT" to show the FFT plotting.
- Click on Add Class and repeat 2-8 to add classes for all the keys, as well as a class for "no key is pressed", so you should have 13 classes in total. The final result is shown below and you are ready for next step.
5. Step 4 – Benchmarking in NanoEdge™ AI Studio
In this step, we will use NanoEdge™ AI Studio to quickly and automatically find the optimized algorithm for our use case.
After collecting the data for all classes, click on "Optimize and Benchmark" in NanoEdge™ AI Studio. You should see the welcome interface as shown below, with the explanations for each measuring metrics.
- Click on the START button to initiate the library selection process
- A "Select your Class" window will pop up. Select All classes and click on Validate.
- After validation, the Studio will start the searching, you should see the window below:
- This area indicates the benchmark process:
- Progress bar
- Elapsed time
- Number of libraries tested
- You can pause the benchmark and resume later
- You can stop the benchmark when the desired performance is reached
- This area shows the performance indicators. The higher balanced accuracy and confidence level are better, while the lower RAM and Flash sizes are better.
- This small log window will display notable information and events such as benchmark status, search speed per thread, new libraries found, etc.
- This graph shows the evolution of the performance indicators over time, as thousands of candidate libraries are tested.
- This area indicates the benchmark process:
- Once you are satisfied with the performance during the library selection process, click on "Stop" to end the benchmarking. The optimized algorithm will be selected for deployment later.
6. Step 5 – Emulator in NanoEdge™ AI Studio
In this step, we will use the Emulator in NanoEdge™ AI Studio to verify the selected library before we generate and deploy the library files on STWIN.
After finish the benchmarking, click on "Emulator" in NanoEdge™ AI Studio to open the Emulator interface.
- Click on the Initialize Emulator button to open the Emulator, as shown below and then the process is similar to Step 3
- Select Serial data as we did in Step 3.
- Select the same COM port and Baudrate, check Max lines for 200 lines.
- Press a key in the tone generation app on your smart phone, and click on the "Record" button. The classification result should be shown in the interface.
- Try not pressing any key to see if the "no-key-pressed" is correctly classified, and you can try all other keys.
- If the classification results are not accute, you should go back to Step 3 to start over the datalogging and benchmarking process.
- Otherwise you can go to the next step to generate the libarary files for deployment.
7. Step 6 – Deploy in NanoEdge™ AI Studio
In this step, we will generate the library files using NanoEdge™ AI Studio.
7.1. Generate library files in NanoEdge™ AI Studio
When you finish the Emulator, click on "Deploy" in NanoEdge™ AI Studio to open the Deploy interface for library generation.
- Check the box for
-mfloat-abi (hard)
to enable the hardware FPU operations. - Click on the Compile button, and there will be a pop-up window as shown below:
- Click on Development version, then there will be a pop-up window to let you choose a place to save the libraries:
- Locate to a local directory you prefer, rename the file if you want, then click on Save.
- Go to the directory you chose, make sure the file
libneai_*.zip
is successfully saved there, extract the file, and you are ready for the next step.
7.2. Library files explanation
The NanoEdge AI static library for classification is the code that contains an AI model (for example, as a bundle of signal treatment, machine learning model, optimally tuned hyperparameters, etc.) designed to identify a sensor pattern in a class. All classes are defined by the user in NanoEdge™ AI Studio and are used during the training process of the AI model.
After the zip file is extracted, you should find these folders and files included:
There are three files we will need for next step: libneai.a
, knowledge.h
and NanoEdgeAI.h
.
libneai.a
is the C/C++ static library for the classifer.knowledge.h
is the header file that contains the knowledge buffer declaration.NanoEdgeAI.h
is the header file that contains functions and variable definitions, as well as the class number and names. An example of the code is shown below:
8. Step 7 – Program STWIN for Detection
In this step, we will add the library files into the project, import the project into STM32CubeIDE, modify the source code, then program the STWIN for detection.
- Replace the old library files by the new ones
- Go to the library folder, copy the file
libneai.a
- Go to the project folder
..\STWIN_DTMF_Classifier\Core\Src
, paste the file here, and click on Replace the file in the destination in the pop-up window. - Go back to the library folder, copy the files
knowledge.h
andNanoEdgeAI.h
. - Go back to the project folder
..\STWIN_DTMF_Classifier\Core\Inc
, paste the files here, and click on Replace the files in the destination in the pop-up window.
- Go to the library folder, copy the file
- Import project into STM32CubeIDE workspace
- Open STM32CubeIDE on your PC, create or choose a workspace and click Launch to launch your workspace.
- In the menu bar, click File -> Import...
- In the pop-up window, select Existing Projects into Workspace, and click Next, as shown below:
- In the next interface, click Browse..., find the project folder where you saved and then click Select Folder, then click Finish and wait for the project to be imported, as shown in the picture below:
- Build Project and Program the STWIN
- In STM32CubeIDE, unfold the project in Project Explorer, find
..\Core\Src\main.c
and double click to open it in the Editor, as shown below:
- In STM32CubeIDE, unfold the project in Project Explorer, find
- In
main.c
, go to line 48 to find#define NEAI_MODE
. Change the value to "1" if it is not, as shown below:
- In
- Press
CTRL + S
to save the change. Make sure the STWIN and STLINK_V3MINI are connected to PC via USB cables, and then click on the "Run" button in the Toolbar to build and program the project:
- Press
- You can see the log information in the Console window. When the project is successfully built and programmed on the STWIN, you should see the following information:
- Now you are ready for the final step - to run the DTMF classifier on STWIN.
Download verified successfully Debugger connection lost. Shutting down...
9. Step 8 – Run DTMF classifier on STWIN
In this step, we will run the DTMF classifier on STWIN and use Tera Term to display the classification results.
- Once the STWIN is successfully programmed for detection, you can open the Tera Term on PC.
- In Tera Term's menu bar, click on Setup -> Serial port.... The serial port setup window will pop up as shown below:
- Select the same COM port as we did in Step 3 for datalogging.
- Select the baudrate value "921600" for Speed.
- Leave other settings as default, and click on New open.
- Bring out your smart phone, and press any key in the tone generation app. The classification results will show in the interface, along with the probability of each class. When there's a key pressed, it will show the corresponding class. When the key is released, it will show the "no-key-pressed" class, as shown below:
10. Resources
- Firmware Package - Project source code and pre-compiled binary files
- NanoEdge™ AI Studio Getting Started Guide
- NanoEdge™ AI Library documentation
- UM2777 - How to use the STEVAL-STWINKT1B SensorTile Wireless Industrial Node for condition monitoring and predictive maintenance applications
- UM2237 STM32CubeProgrammer software description
- UM2609 STM32CubeIDE user guide
11. References