How to create a multistate vibrations classifier using NanoEdge AI Studio (using STEVAL-MKBOXPRO)

This article is a tutorial on how to use NanoEdge AI Studio to create a multistate vibrations classifier running on the SensorTile.box PRO with multisensors and wireless connectivity development kit (STEVAL-MKBOXPRO).

This tutorial shows you how to program the SensorTile.box PRO for data logging; how to use NanoEdge AI Studio to collect data, label data, find the best algorithm, run the emulator, generate the library files for deployment; how to use STM32CubeIDE to modify codes, build the project, program the SensorTile.box PRO; and finally run the classifier on SensorTile.box PRO. The main process of this tutorial is shown in the flowchart below:

1. Prerequisites

1.1. Hardware

• An STEVAL-MKBOXPRO development kit board,
• One USB Type-C^® cable (to connect the SensorTile.box PRO to the PC),
• A Windows^® powered laptop/PC (Windows^® 10 or 11),
• A multispeed USB powered fan (can be ordered from this link), which has three speed mode: low, medium, and high.

1.2. Software

1.2.1. Source Code

• A firmware package mkboxpro-fan-classifier-wiki-package.zip is ready to download from st.com. Please download the zip file here.

The package includes:

• Project source code in subfolder STEVAL-MKBOXPRO_LSM6DSV16X
• Two Pre-compiled binary files in subfolder Binary:

• MKBOXPRO_Fan_Classifier-datalog.elf for datalogging
• MKBOXPRO_Fan_Classifier-detection.elf for detection

• Library files generated by NanoEdge AI Studio in subfolder libneai_mkboxpro-fan-nclassifier.zip

1.2.2. NanoEdge AI Studio

• NanoEdge AI Studio is a new machine learning (ML) technology that brings true innovation easily to end users. In just a few steps, developers can create an optimal ML library for their project based on a minimal amount of data.
• Download NanoEdge AI Studio as a free trial version for the STMicroelectronics boards.
• After registration, you will receive the license key in your registered email, which will be used to activate NanoEdge AI Studio when you launch it for the first time.

Information

If you have any issues during installation or activation process, please follow this Troubleshooting Guide to configure proxy or other settings. If you need to run NanoEdge AI Studio on a virtual machine, please submit a request on our Online Support platform. To ensure your request is quickly routed to the correct support team, please select Product / Application for Category, and NanoEdgeAIStudio for Product Category/Part Number.

1.2.3. STM32CubeIDE

• STM32CubeIDE is an all-in-one multi-OS development tool, which is part of the STM32Cube software ecosystem. It is an advanced C/C++ development platform with peripheral configuration, code generation, code compilation, and debug features for STM32 microcontrollers and microprocessors.
• Download STM32CubeIDE from st.com

1.2.4. Tera Term

• Tera Term is an open-source and freely available software terminal emulator, which is used to display the classification results.
• Users can download and install the latest version available from Tera Term.

2. Program SensorTile.box PRO for data logging

The first step is to connect the SensorTile.box PRO to your PC and program it for datalogging mode with the provided binary file.

2.1. Board connection

There is a single connection to make on the SensorTile.box PRO, as shown in the picture:

1. Plug the USB Type-C^® cable side into the SensorTile.box PRO
2. Plug the SensorTile USB cable into PC

2.2. Program SensorTile.box PRO using binary file

The easiest way to program the SensorTile.box PRO is using the STM32CubeProgrammer. Once you have the pre-built binary file on your local disk, use CubeProgrammer to flash it to SensorTile.box PRO:

1. On SensorTile, slide the power button to OFF position
2. Press and hold button 2 and slide the power button to ON position

3. Launch CubeProgrammer and select USB from the dropdown menu
4. Refresh the Port
5. Connect using USB

6. Under the Erasing & Programming tab, press Browse and select the MKBOXPRO_Fan_Classifier-datalog.elf binary file
7. Click Start Programming
You will see a success prompt once programming is complete. Then, close CubeProgrammer.

3. Create an n-Class classification project in NanoEdge AI Studio

In this step, we will create an n-Class classification project in NanoEdge AI Studio, and configure the project settings.

Open NanoEdge AI Studio on PC, you should see the welcome interface below and then follow the steps:

1. Click on n-Class Classification to choose it as project type.
2. Click on CREATE NEW PROJECT to create the project.

You should see the Project settings page as below:

1. Give the project a name, for example, MKBOXPRO_Fan_Classifier
2. Add a brief description of the project.
3. Choose 12 as the RAM limit.
4. Choose 32 as the flash limit.
5. Choose STEVAL-MKBOXPRO for Target. On the target selection screen, scroll down to STM32U5 series, select STEVAL-MKBOXPRO, and click Confirm.

   

6. Choose Accelerometer 3 Axes for Sensor type.
7. Finally, click on NEXT to go to the next step.

4. Data logging in NanoEdge AI Studio

In this step, alter the speed of the fan by pressing the button on the back, and apply clog to the fan to collect data for seven (7) classes: stop, low speed, medium speed, high speed, low speed clogged, medium speed clogged, and high speed clogged.

4.1. Hardware setup before data logging

Before starting data logging, it is recommended to disconnect SensorTile.box PRO from the PC and power cycle the board.

1. Slide the power switch to OFF position
2. Stick the SensorTile on top of the fan with double-sided tape. You can also use normal tape as double-sided tape by making a ring, but this may cause reduced detection accuracy due to poor mechanical coupling of the fan and SensorTile.

3. Connect the USB cable and slide the power switch to ON position
4. Plug the USB cable of the fan into a PC or USB charger. Leave the fan in stop mode

4.2. Start data logging

Next, we use the programmed SensorTile and NanoEdge AI Studio to collect the data for each class via USB streaming. Go back to NanoEdge AI Studio, once the project was created, you should see the window as shown below:

1. Click on ADD SIGNAL to open the "Import signal" window.


Select signal source from USB
2. In the pop-up window, choose FROM SERIAL (USB) as the Type of signal source.


Configuration of data logging via USB for SensorTile.box PRO fan classifier
3. In the next window, select your Serial / COM port. Refresh if needed.

Information

COM port number varies on different PCs. In this example, it is COM63. You can go to Windows Device Manager to look for USB Serial Device under Ports (COM & LPT)

4. Select 115200 for baud rate
5. You can check the box to define max. lines of 100 signals. By doing this, the data logger will automatically stop when 100 lines of signal are collected. However, this should be an absolute minimum for any application, feel free to get more.
6. Give a class name, for example "stop", which will be used to shown as the classification result later.
7. Make sure that the fan is initially turned off. Then, click the START/STOP button to start data logging. The real-time data and plotting will be shown.

8. Once 100 lines of signal are collected, the data logger stops. Click on CONTINUE to the next step.


Preview collected data and finish import for one class
9. You can preview the data collected. If no problem, click on IMPORT to finish data logging for this class.
10. Repeat step 1-9 to add all classes

    • Press the button on the back of the fan to switch its speed between low, medium, and high
    • Use a sticky note or a notebook (stick the sticky note or put the notebook closely at the front side of the fan) to clog the fan for clogged classes.
    • You should finally have 7 classes in total, including stop, low speed, medium speed, high speed, low speed clogged, medium speed clogged, and high speed clogged. The final result is shown below, and you are ready for the next step.

5. Benchmarking in NanoEdge AI Studio

In this step, we will use NanoEdge AI Studio to quickly and automatically find the optimized algorithm for our use case.

After collecting the data for all classes, click on "Benchmark" in NanoEdge AI Studio. You should see the welcome interface as shown below:

1. Click on RUN NEW BENCHMARK
2. A "Select your Classes" window will pop up. Select All classes and click on START.

3. Then the NanoEdge AI Studio starts the benchmarking. You should see the window below:



1. This area shows the benchmark status and its timestamp
2. Performance indicators: higher balanced accuracy and score are better, while lower RAM and flash sizes are better
3. Elapsed time and number of libraries tested
4. You can pause the benchmark and resume later using this button
5. The graph shows the evolution of the performance indicators over time, as thousands of candidate libraries are tested.

Information

Benchmarking can run for hours or even days, so you do not need to wait for it to completely finish. Regarding the performance indicators, you can refer to this NanoEdge AI Studio wiki for more details.



4. Once you are satisfied with the performance during the library selection process, click on the "PAUSE" button to end the benchmarking. The optimized algorithm is selected for deployment later. You should see the details of this benchmark as below:

NanoEdge AI Studio allows for the estimation of execution time for any library encountered during the benchmark with the STM32F411 at 84 MHz simulator provided by Arm, utilizing a hardware floating-point unit. The estimation is an average of multiple calls to the NanoEdge AI library functions tested. The tool does not use directly the user data but data of a similar range to make the estimation. Thus, this emulator does not reflect the board or MCU selected in step 1 but is a useful baseline or to compare solutions.

6. Validation in NanoEdge AI Studio

The validation screen is used to summarize any selected benchmark or to compare the metrics of two or more benchmarks. Further, for each library displayed, the user can click on the blank sheet of paper to open the validation report. It includes: a data summary, an in-depth performance review, and the algorithm flow chart.

The user can also run the emulator with real data to test a library before deployment, as shown in the next section. Moreover, the "New Experiment" option lets you compare libraries on different datasets to better understand which ones are best for your application.

Click on "Validation" in NanoEdge AI Studio to open the validation interface:

6.1. Emulator in NanoEdge AI Studio

In this step, we use the emulator in NanoEdge AI Studio to verify the selected library before we generate and deploy the library files on SensorTile.box PRO.

To run the emulator for any given library, click on the "Serial emulator" button corresponding to the library to be tested.

1. Select the same COM port as we did in the data logging step.
2. Select the same baud rate (115200).
3. Get the fan ready in any mode, and click on the "START/STOP" button.
4. The real-time classification result should be shown in the Class Detected area.
5. Switch between different speeds, and add clogs. Check the classification results.
6. Click on the "START/STOP" button to stop the emulation.

• If the classification results are not accurate, you should go back to the data logging step to start over the data logging and benchmarking process.
• Otherwise, you can go to the next step to generate the library files for deployment.

A sample emulator output is shown below.

The emulator also allows you to export both the serial data and library results as a csv file for in-depth evaluation. You can use this data to analyze the library output and to improve your dataset/training.

Information

For more information about the Emulator, please refer to this NanoEdge AI Studio wiki

7. Deployment in NanoEdge AI Studio

In this step, we generate the library files using NanoEdge AI Studio.

7.1. Generate library files in NanoEdge AI Studio

When you finish the emulator, click on "Deployment" in NanoEdge AI Studio to open the Deploy interface for library generation.

1. If you have multiple benchmarks, choose the one you want to compile. Otherwise, leave it default.
2. Check the box for Float abi to enable the hardware FPU operations.
3. Click on the COMPILE LIBRARY button, and there is a pop-up window as shown below:

• Click Get Library and fill the survey or press skip, then there is another pop-up window letting you to choose a local directory to save the libraries.

• Locate to a local directory that you prefer, rename the file as you want, then click on Save.
• Go to the directory that you chose, make sure the file libneai_*.zip is successfully saved there, extract the file, and you are ready for the next step.

7.2. Library files explanation

The NanoEdge AI static library for classification is the code that contains an AI model (for example, as a bundle of signal treatment, machine learning model, optimally tuned hyperparameters, etc.) designed to identify a sensor pattern in a class. All classes are defined by the user in NanoEdge AI Studio and are used during the training process of the AI model.

After the zip file is extracted, you should find these folders and files included:

There are three files that we will need for the next step: libneai.a, knowledge.h and NanoEdgeAI.h.

• libneai.a is the C/C++ static library for the classifier.
• knowledge.h is the header file that contains the knowledge buffer declaration.
• NanoEdgeAI.h is the header file that contains function prototypes and sample declarations. An example of the code is shown below:

Information

The class number may not be in the same order as you recorded in datalogging step. There is an automatically generated "unknown" class, which indicates when the probability is less than 50% for each known class. For more details, refer to this NanoEdge AI Library for n-class classification wiki.

8. Program SensorTile.box PRO for detection

In this step, we add the library files into the project, import the project into STM32CubeIDE, modify the source code, then program the SensorTile.box PRO for detection.

1. Replace the old library files with the new ones

• Go to the library folder, copy the file libneai.a
• Go to the project folder ..\STEVAL-MKBOXPRO_LSM6DSV16X\Core\Src, paste the file here, and click on Replace the file in the destination in the pop-up window.
• Go back to the library folder, copy the files knowledge.h and NanoEdgeAI.h.
• Go back to the project folder ..\STEVAL-MKBOXPRO_LSM6DSV16X\Core\Inc, paste the files here, and click on Replace the files in the destination in the pop-up window.

2. Import project into STM32CubeIDE workspace

• Open STM32CubeIDE on your PC, create or choose a workspace and click Launch to launch your workspace.
• In the menu bar, click File -> Import...
• In the pop-up window, select Existing Projects into Workspace, and click Next, as shown below:





• In the next interface, click Browse..., find the project folder where you saved and then click Select Folder, then click Finish and wait for the project to be imported, as shown in the picture below:




3. Build project and program the SensorTile.box PRO

• In STM32CubeIDE, unfold the project in Project Explorer, find ..\Core\Src\main.c and ..\Core\Inc\NanoEdgeAI.h. Double-click on the two files separately to open them in the editor, as shown below:





• In main.c, go to line 46 to find #define NEAI_MODE. Change the value to "1" if it is not, as shown below:





• Then go to NanoEdgeAI.h, find the buffer declarations in the commented part underneath, select the codes of "buffer for mapping class id to class name", line 84 to 93 const char *id2class[CLASS_NUMBER + 1]. Right-click and choose "Copy" or simply press Ctrl+C on the keyboard to copy the codes:





• Go back to main.c, paste the copied codes below float output_class_buffer[CLASS_NUMBER]; to replace the existing declarations. As shown below:





• Press CTRL + S to save the changes.
• Click on the “hammer” button to build the project:
• After built successfully, you should see:

  Build Finished. 0 errors, 0 warnings. (took 2s.866ms)

  in the console.
• There is a Debug folder generated, in which you can find the STEVAL-MKBOXPRO_LSM6DSV16X.elf binary file that we will use for the next step




4. Replace the old library files with the new ones
1. Once you have generated the binary, use CubeProgrammer to flash it to SensorTile
2. On SensorTile, slide the power button to OFF position

1. Press and hold button 2 and slide the power button to the ON position
2. Launch CubeProgrammer and select USB from the dropdown menu
3. Refresh the Port
4. Connect using USB

1. Under the Erasing & Programming tab, press Browse and select the binary file we just built, which can be found in this path:

..\STEVAL-MKBOXPRO_LSM6DSV16X\Debug\ STEVAL-MKBOXPRO_LSM6DSV16X.elf


1. Click Start Programming
2. Close CubeProgrammer once you see a success prompt indicating that programming is complete
3. Restart the board by sliding the power button off position, and then back to on position.

9. Run the Classifier on SensorTile.box PRO

In this step, we will run the fan classifier on SensorTile.box PRO and use Tera Term to display the classification results.

• Once the SensorTile.box PRO is successfully programmed for detection, you can open the Tera Term on PC.
• In Tera Term menu bar, click on Setup -> Terminal.... In the New-line setting, change Receive to AUTO and Transmit to CR:

• Go back to Tera Term's menu bar, click on Setup -> Serial port.... The serial port setup window pops up as shown below:

• Select the same COM port as we did in the data logging step for data logging.
• Select the baud rate value "115200" for speed.
• Leave other settings as default, and click on New open.

• Get the fan ready and make sure SensorTile.box PRO is firmly attached at the top of the fan. The classification results appear in the interface, along with the probability of each class, as shown below:

10. Resources

• Firmware package - Project source code and precompiled binary files
• NanoEdge AI Studio wiki
• NanoEdge AI Library for n-class classification (nCC) wiki
• UM2237 STM32CubeProgrammer software description
• UM2609 STM32CubeIDE user guide
• How to troubleshoot common NanoEdge AI Studio installation / activation problems