Last edited one month ago

Image classification


This article explains how to use the stai_mpu API for image classification applications supporting OpenVX [1], TensorFlow Lite[2] or ONNX [3] back-ends.

1. Description[edit | edit source]

The image classification neural network model allows identification of the subject represented by an image. It classifies an image into various classes.

Image classification application

The application demonstrates a computer vision use case for image classification where frames are grabbed from a camera input (/dev/videox) and analyzed by a neural network model interpreted by OpenVX, TFLite or ONNX framework.
A Gstreamer pipeline is used to stream camera frames (using v4l2src), to display a preview (using gtkwaylandsink) and to execute neural network inference (using appsink).
The result of the inference is displayed in the preview. The overlay is done using GtkWidget with cairo.

The models used with this application are the MobileNet v1 downloaded from the TensorFlow Lite Hub[4] and the MobileNet v2 downloaded from the ST model zoo[5]

2. Installation[edit | edit source]

2.1. Install from the OpenSTLinux AI package repository[edit | edit source]

After having configured the AI OpenSTLinux package you can install X-LINUX-AI components for image classification application:

Collapse

2.1.1. Install on STM32MP2x board[edit | edit source]

The OpenVX application will be installed to take advantage of the neural processing unit (NPU) and graphics processing unit (GPU).

  • To install this application, please use the following command:
x-linux-ai -i stai-mpu-image-classification-cpp-ovx


  • Then, restart the demo launcher:
systemctl restart weston-graphical-session.service

2.1.2. Install on STM32MP1x board[edit | edit source]

2.2. Source code location[edit | edit source]

  • in the OpenSTLinux Distribution with X-LINUX-AI Expansion Package:
<Distribution Package installation directory>/layers/meta-st/meta-st-x-linux-ai/recipes-samples/image-classification/files/stai_mpu
  • on GitHub:
recipes-samples/image-classification/files/stai_mpu

2.3. Regenerate the package from OpenSTLinux Distribution (optional)[edit | edit source]

Using the OpenSTLinux Distribution, you are able to rebuild the application.


  • Set up the build environment:
cd <Distribution Package installation directory>
source layers/meta-st/scripts/envsetup.sh
Warning.png Be careful to select the right board in the script to setup the correct environment
Collapse
  • Rebuild the application on STM32MP2x:
bitbake stai-mpu-image-classification-cpp-ovx -c compile

The generated binary is available here:

<Distribution Package installation directory>/<build directory>/tmp-glibc/work/cortexa35-ostl-linux/stai-mpu-image-classification-cpp/5.0.0-r0/stai-mpu-image-classification-cpp-5.0.0/stai_mpu


  • Rebuild the application on STM32MP1x:

For more information about building the application for STM32MP1x, please expand this section.

3. How to use the application[edit | edit source]

3.1. Launching via the demo launcher[edit | edit source]

You can click on the icon to run either the C/C++ or Python application. By default the OpenVX applications will be installed on STM32MP2x, and the TFLite applications will be installed on STM32MP1x.

Demo launcher

3.2. Executing with the command line[edit | edit source]

The image classification C++ & Python applications are located in the userfs partition:

/usr/local/x-linux-ai/image-classification/stai_mpu_image_classification
/usr/local/x-linux-ai/image-classification/stai_mpu_image_classification.py

It accepts the following input parameters:

  • In C/C++ application:
 
Usage: stai_mpu_image_classification -m <model .tflite> -l <label .txt file>

-m --model_file <.tflite file path>:  .tflite model to be executed
-l --label_file <label file path>:    name of file containing labels
-i --image <directory path>:          image directory with image to be classified
-v --video_device <n>:                video device is automatically detected but can be set (example video0)
--frame_width  <val>:                 width of the camera frame (default is 640)
--frame_height <val>:                 height of the camera frame (default is 480)
--framerate <val>:                    framerate of the camera (default is 15fps)
--input_mean <val>:                   model input mean (default is 127.5)
--input_std  <val>:                   model input standard deviation (default is 127.5)
--dual_camera_pipeline                Use dual camera post-processing pipeline, one dedicated for the NN and the other dedicated for the display 
--verbose:                            enable verbose mode
--validation:                         enable the validation mode
--val_run:                            set the number of draws in the validation mode
--help:                               show this help
  • In Python application:

4. Testing with MobileNet[edit | edit source]

4.1. Test with MobileNetV2 on STM32MP2x[edit | edit source]

The model used for testing is the mobilenet_v2_1.0_224_int8_per_tensor.nb

To ease launching of the application, two shell scripts are available for both C/C++ and Python applications on the board:

  • For C/C++ application:
    • launch image classification based on camera frame inputs:
    /usr/local/x-linux-ai/image-classification/launch_bin_image_classification.sh
    • launch image classification based on the pictures located in /usr/local/demo-ai/image-classification/models/mobilenet/testdata directory
    /usr/local/x-linux-ai/image-classification/launch_bin_image_classification_testdata.sh
  • For Python application:
    • launch image classification based on camera frame inputs:
    /usr/local/x-linux-ai/image-classification/launch_python_image_classification.sh
    • launch image classification based on the pictures located in /usr/local/demo-ai/image-classification/models/mobilenet/testdata directory
    /usr/local/x-linux-ai/image-classification/launch_python_image_classification_testdata.sh

4.2. Test with MobileNetV1 on STM32MP1x[edit | edit source]

5. Going further[edit | edit source]

5.1. Install all image classification packages[edit | edit source]

After having configured the AI OpenSTLinux package you can install directly all the X-LINUX-AI packages related to image classification by using the following command::

x-linux-ai -i stai-mpu-image-classification-*

Then restart the demo launcher:

systemctl restart weston-graphical-session.service

5.2. Launching via the demo launcher[edit | edit source]

Now, it's possible to launch all the image classification application on your board by clicking on it in the demo launcher. The Python & C/C++ application are now available using TFLite and ONNX backend. Moreover, on STM32MP2x the OpenVX backend is also available.

Demo launcher

5.3. Launching via the command line interface (CLI)[edit | edit source]

5.3.1. CLI on STM32MP2x[edit | edit source]

The model used for testing is the MobilenetV2.

The two shell scripts described before offers the possibility to select the framework directly between TFlite, ONNX and OpenVX. To be able to run the application using all supported frameworks, the models for each frameworks must be available in the /usr/local/x-linux-ai/image-classification/models/mobilenet/ directory. Then, you will need to specify the framework as an argument of the launch scripts as follow.

  • For C/C++ application:
    • Run image classification based on camera input with the chosen framework. Available framework options are: tflite, onnx, nbg.
    /usr/local/x-linux-ai/image-classification/launch_bin_image_classification.sh nbg
    • Run image classification based on picture located in the /usr/local/demo-ai/image-classification/models/mobilenet/testdata directory with the chosen framework. Available framework options are: tflite, onnx, nbg.
    /usr/local/x-linux-ai/image-classification/launch_bin_image_classification_testdata.sh nbg
  • For Python application:
    • Run image classification based on camera input with the chosen framework. Available framework options are: tflite, onnx, nbg.
    /usr/local/x-linux-ai/image-classification/launch_python_image_classification.sh nbg
    • Run image classification based on picture located in the /usr/local/demo-ai/image-classification/models/mobilenet/testdata directory with the chosen framework. Available framework options are: tflite, onnx, nbg.
    /usr/local/x-linux-ai/image-classification/launch_python_image_classification_testdata.sh nbg

5.3.2. CLI on STM32MP1x[edit | edit source]

6. References[edit | edit source]