STM32 CV hands-on project

Agenda

1 Preparation and introduction

2 How to install and use X-CUBE-AI

3 Integrate IPL

4 Integrate AI model

2
 Agenda

1 Preparation and introduction

2 How to install and use X-CUBE-AI

3 Integrate IPL

4 Integrate AI model

3
 Objectives of the hands-on seminar​
• According to ABI research, global shipments of Deep Edge AI devices will reach 2.5 billion units by
2030. ST has observed a growing community and ecosystem of Deep Edge AI technology focused
on independent, low-power, and cost-effective embedded solutions.

• ST is offering a set of AI tools accelerating the development cycle and the optimization of trained AI
models (STM32Cube.AI) for the STM32 family and Motion MEMS sensors with a Machine Learning
Core integrated

• The seminar is intended to provide a hands-on experience on ST ecosystem for AI Deep Learning
solutions.(computer vision)

• The NN model is already provided, but the objective of the seminar is not to go deeply in the
explanation of a NN design or fine tuning.​

• Even if basic concepts of AI will be touched, the seminar is not intended to give a detailed training on
AI, Deep Learning or Python programming.​

4
 The 5 key steps behind AI projects
Machine Learning Model Creation Operating Mode

Process & analyze new data

Capture data Train ML model
using trained model

2 4

1 3 5
Clean, label data
Embed ML model
Select ML model
in real system 5
architecture
 Starting from Template project

• This project realizes the display of the camera on LCD. On the basis of this project, we will
show you step by step how to deploy the trained model to the board.

6
 Program logic
• The following figure describes the running logic of the entire program.
• We only need to add the function of AI to the original program.

RGB565
DCMI LTDC
Camera Buffer 1 LCD

Interrupt
IPL

grayscale

Buffer 2 AI model

7
 Person presence detection model
• This model based on a low-complexity CNN model (so-called Google_Model) working on
grayscale images (8 bits per pixel) with a resolution of 96 × 96 pixels.
• The person presence detection models contain three output classes: Person ， Not Person
and Unknow

Unknow

Person
Not-person

8
 Wio lite AI Dev board

• Wio Lite AI Single Board is a microcontroller vision development board based on the STM32H725AE
chip which is an Arm® Cortex®-M7 32-bit 550 MHz high-performance microprocessor. It is designed
in feather development board form factor for greater simplicity and rapid project development
assistance.

Features
•Based on STM32H725AE Chip, an Arm® Cortex®-M7 32-bit 550
MHz high-performance microprocessor
•16MB SPI NOR flash and 8MB PSRAM
•Include a 40-Pin FPC port for LCD RGB565 or RGB888 and a 24-
Pin FPC port for a DCMI camera
•Wireless connectivity with dual-band 2.4GHz / 5GHz Wi-Fi and
Bluetooth 5.1
•One USB Type-C for power supply and one micro-SD card slot for
extra storage
•Provide software support for a quick start of vision projects

9
 HW and SW setup checklist
• We have some hardware and software to prepare before
we actually start.

HW SW version
Wio lite AI Dev board STM32CubeIDE 1.6.1

ST-LINK V2/V3 STM32CubeMX 6.3.0 or later

USB - type C X-CUBE-AI 6.0.0 and 7.0.0

 Connection
• WIO Lite AI board is connected to ST-LINK through the
SWD protocol, which is used to download code and debug.

ST-LINK V3 Wio Lite AI Single Board

Type-C USB

SWD
CLK

DIO
GND

RST

optional
11
 Connection
• 5 pins correspond to signals with the same color:
• SWCLK
• SWDIO
• GND
• RST
• VDD (3.3V)

optional

Wio Lite AI Single Board ST-LINK V2 12

 Connection
• Using ST-LINK to connect to the development board is recommended, but it is optional.
• You can upgrade BootLoader first, and then download binary through Type-C.
• But it is impossible to debug in this way.

Type- C

13
 Agenda

1 Preparation and introduction

2 How to install and use X-CUBE-AI

3 Integrate IPL

4 Integrate AI model

14
Install X-CUBE-AI

Click
【INSTALL/REMOVE】

15
 Install X-CUBE-AI

1.Select
【STMicroelectonics】

2.Check the
corresponding version

3.Click 【install now】

16
 Enabling the X-CUBE-AI component

From the menu, select [software packs]>[select components]

17
 Enabling the X-CUBE-AI component
• From the Additional Software Component Selection window, the X-CUBE-AI(6.0.0)/core bundle must
be checked to be able to upload the NN models and generate the associated STM32 NN library, In this
case, as the library is fully integrated as a static library, the user only needs to implement his AI-based
application/middleware on top of the generated well-defined NN API

18
 CLI environment settings

• After installing X-Cube-AI 6.0.0, you will find the CLI application « stm32ai » in the following
repository: C:\Users\<username>\STM32Cube\Repository\Packs\STMicroelectronics\X-CUBE-
AI\6.0.0\Utilities\windows (or mac or linux)

• Windows®, Linux®, macOS® (with GCC tool-chain) are supported and user should update his
environment variables by adding the previous repository in the system path

• Validate your settings by opening a terminal and trying the following commands
• stm32ai --version
• stm32ai --tools_version

19
 « stm32ai » for Windows®
• Make sure X-Cube-AI 6.0.0 is installed
• https://www.st.com/content/st_com/en/products/embedded-software/mcu-mpu-embedded-software/stm32-embedded-software/stm32cube-expansion-
packages/x-cube-ai.html

• Use a windows command prompt (or bash)

• Add stm32ai application in the system PATH variable:
• set CUBE_FW_DIR=C:\Users\<user_name>\STM32Cube\Repository

• set X_CUBE_AI_DIR=%CUBE_FW_DIR%\Packs\STMicroelectronics\X-CUBE-AI\6.0.0
• set PATH=%X_CUBE_AI_DIR%\Utilities\windows;%PATH%

• Verify the environment

• > stm32ai --version

• stm32ai - Neural Network Tools for STM32AI v1.4.1 (STM.ai v6.0.0)

20
 How-to update your PATH variable (1/3)

• If the previous steps don’t work, you

can use graphical interface to update
the PATH environment variable:

• Right click on « This PC » (main root)

• Click on « Properties »
1

21
 How-to update your PATH variable (2/3)

Click on « Advanced system settings »

Then, click on « Environment Variables »

22
 How-to update your PATH variable (3/3)

Click on « Path » and then, click on « Edit… »

4
Add the repository of stm32ai
(C:\Users\<username>\STM32Cube\Repository\Packs\STMicroel
ectronics\X-CUBE-AI\6.0.0\Utilities\windows (or mac or linux)

3 23
 Generate command
• The 'generate' command is used to generate the specialized network and data C-
files. They are generated in the specified output directory.

$ stm32ai generate -m <model_file_path> -o <output-directory-path>

...
Generated files (5)
-----------------------------------------------------------
<output-directory-path>\<name>_config.h
<output-directory-path>\<name>.c
<output-directory-path>\<name>_data.c
<output-directory-path>\<name>.h
<output-directory-path>\<name>_data.h

Creating report file <output-directory-path>\network_generate_report.txt

24
 Generate command
• The 'validate' command allows to import, to render and to validate the generated C-files “Validation
engine” and “AI validation application” sections to have an overview of the underlying process. In
particular for the validation on target ('--mode stm32'), the STM32 board should be flashed with a
validation firmware including the model.

Evaluation report (summary)

----------------------------------------------------------------------------------------------------------
Mode acc rmse mae l2r tensor
----------------------------------------------------------------------------------------------------------
x86 C-model #1 92.68% 0.053623 0.005785 0.340042 dense_4_nl [ai_float, [(1, 1, 36)],
m_id=[10]]
original model #1 92.68% 0.053623 0.005785 0.340042 dense_4_nl [ai_float, [(1, 1, 36)],
m_id=[10]]
X-cross #1 100.00% 0.000000 0.000000 0.000000 dense_4_nl [ai_float, [(1, 1, 36)], m_id=[10]]
----------------------------------------------------------------------------------------------------------

25
 Generate command
• The 'analyze' command is the primary command to import, to parse, to check and to render an uploaded
pre-trained model. Detailed report provides the main system metrics to know if the generated code can
be deployed on a STM32 device. It includes also rendering information by layer or/and operator After
completion, the user can be fully confident on the imported model in term of supported layer/operators.

• Analyze a model (simple model file)

$ stm32ai analyze -m <model_file_path>\person_detect.tflite

• Analyze a multiple model files (caffe type example)

$ stm32ai analyze -m ./caffe/lenet.prototxt -m ./caffe/lenet.caffemodel

• Analyze a 32b float model with compression request

$ stm32ai analyze -m <model_file_path> -c 8

26
 Supported-ops command
• The 'suppoorted-ops' command is used to display the list of the supported operators for a given deep
learning framework ('-t/--type' option).

• Generate the list of the supported Keras operators (default)

$ stm32ai supported-ops
Parsing operators for KERAS toolbox
53 operators found
Activation, ActivityRegularization, Add, AlphaDropout, Average, AveragePooling1D, AveragePooling2D,
ZeroPadding2D
23 custom operators found
Abs, Acos, Acosh, Atan, Atanh, Ceil, Clip, Cos, Exp, FloorDiv, FloorMod, Gather, Lambda, Log,
Pow, Reshape, Round, Shape, Sign, Sin, Sqrt, Square, Tanh

• Generate the list of the supported tflite operators

$ stm32ai supported-ops -t tflite

27
 Agenda

1 Preparation and introduction

2 How to install and use X-CUBE-AI

3 Integrate IPL

4 Integrate AI model

28
 Introduction

• STM32IPL => STM32 Image Processing Library

• A software library, written in C
• Offers image processing & computer vision functionalities for visual analysis applications on STM32
• Wraps some functionalities offered by OpenMV library.

• Organized in 2 separate parts:

• Core => core image processing part independent of HW platform
• Extended => functions that exploit HW accelerators for speed-up purpose (e.g., DMA2D, JPEG CODEC …)

29
STM32IPL SW architecture

STM32IPL

30
 STM32IPL Package

• Drivers => includes HAL, BSP & Arm CMSIS drivers

• Middlewares => ST & Third-party libraries
• STM32 Image Processing Library
• FatFs library
• JPEG library

• Projects => demo project on STM32H747I-DISCO board

• Utilities => some other useful functions

The “DSP” folder in CMSIS & the “core” folder

in STM32_ImageProcessing_Library will be
used in Seeed project

31
 STM32IPL Integration

• To make the integration more general, only

the “Core” part of STM32IPL is integrated.
• Add folder
“STM32ImageProcessingLibrary\Middlewares\S
T\STM32_ImageProcessing_Library\Core” into
user project.
• Add folder
“STM32ImageProcessingLibrary\Drivers\CMSIS\
DSP” into user project, as the dependency
of IPL library. Keep only the “Include” &
“Source” sub-folders.
Add the “DSP” folder • In
to user project, keep “Middlewares\ST\STM32_ImageProcessing_Libr
Add the “Core” only the “Include” & ary\Core\Inc\stm32ipl.h”, comment the line
folder to user “Source” sub-folders “#include “stm32ipl_conf.h””
project
32
STM32IPL Integration

Make sure that these 2

paths are added in
Includes path

33
 STM32IPL Integration

Add these 2 paths in

Source Location

34
STM32IPL Integration

Make sure these

symbols are added

35
 Initialization of the library

• Include the following header file to use STM32IPL API:

• Initialize the library by calling the following function:

• When STM32IPL not needed any more, un-initialize the library to release the allocated memory:

36
 Memory initialization

• A block of memory must be assigned to STM32IPL library in initialization phase, the size & kind of
memory is decided by developer.
• Case 1 => 256 KB of SRAM is reserved to STM32IPL:

• Case 2 => 512 KB of SDRAM is reserved to STM32IPL:

37
 Memory buffer management

• In many IPL functions, the buffers are allocated, used and then automatically released when
function ends.
• In other cases, the function allocates a buffer, uses it, fills it with results and then returns it to callers
which must manage the proper release when done with it.

Here the STM32Ipl_ReadImage()

gets the amount of memory
buffer to store image data, which
can be used by algorithm. After
being processed by algorithm,
this buffer is released by calling
STM32Ipl_ReleaseData()

38
 Agenda

1 Preparation and introduction

2 How to install and use X-CUBE-AI

3 Integrate IPL

4 Integrate AI model

39
Enabling the X-CUBE-AI component

1. Click on [STMicroelectronics X-CUBE-AI X.Y.Z] to open the

initial AI configuration window.

2. Check artificial intelligence X-CUBE-AI

3. Click on add network

4. Select the model type

5. Click on the [Browse..] button to upload the DL file(s) from the

host file system.

42
 Integrate the AI model
Loading model
3

Generate code
1 （convert model
to C code）

Analysis model

43
 New project

The '<name>.c/.h' files contain the topology of the C-model

(C-struct definition of the tensors and the operators),
including the embedded inference client API (refer to [API]) to
use the generated model on the top of the optimized
inference runtime library. '<name>_data.c/.h' files are by
default a simple C-array with the data of the weight/bias
tensors.

Libraries required for cube.ai

44
 Code snippet
• The following code snippet provides a typical and minimal example using the API for a 32b float
model. The pre-trained model is generated with the default options i.e. input buffer is not allocated
in the “activations” buffer and default 'network' c-name is used. Note that all AI requested client
resources (activations buffer and data buffers for the IO) are allocated at compile time thanks the
generated macros: AI_NETWORK_XXX_SIZE allowing a minimalist, easier and quick integration.

46
 Code snippet
• Define initialization function

47
 Code snippet

• Define inference function

48
 Integration
• According to the previous introduction, we will integrate the functions of the
AI part into the original project step by step
1.Include the necessary header and defines related variables and macro to main.c

49
 Integration
• Buffer for IPL

50
 Integration

• Initialization IPL and AI function

51
 Integration

• Inference

52
 Integration

Refresh the inference results of AI after each frame is collected

53
 Integration
• Update compilation path

54
 Integration
• For the version of cubeIDE 6.0.0, you need to remove this compilation option

delete

55
 Integration
• In order to display floating-point numbers properly, you
need to check this compilation option

56
 Model test
• After the integration of the model is completed, it can be downloaded to the development board,
and the subtitle "person" can be seen on the screen when there is someone in the camera screen.

57
 How to upgrade a STM32 project
• For the interface between them, backward or/and forward compatibility is never considered, if
the generated c-model files are updated with a new version of the X-CUBE-AI tools, the X-
CUBE-AI library files should be also aligned. Without specific notification/advice, at source
level, no breaking change to the embedded inference API that can break the client’s
application is expected.

58
 Summary
• we are starting from a template project with only basic driver of Camera and LCD panel, and
we will see the step by step process to integrate a reference AI NN model for People
Detection
• There are several key points in this training：

• Basic knowledge of AI

• Learn what STM32cube.AI is and how to use STM32cube.AI

• How to deploy the AI model to STM32

59
想了解更多?
www.st.com/ai

60
 Appendix
update BootLoader
• Extract the wio_lite_ai_upgrade.7z file

• And then follow the prompts from the readme.txt file

61