Open in app Sign up Sign In

Search Medium

Ultra TinyML: Machine Learning for 8-bit

Microcontroller
How to create a gesture recognition system with Arduino and Neuton TinyML

Leonardo Cavagnis · Follow

Published in Towards Data Science
11 min read · Feb 21, 2022

Listen Share

“A mistake repeated more than once is a decision” — P. Coelho

TinyML is a sub-field of Machine Learning that studies the way to run ML models on small and
low-powered devices.

In this article, I will show an easy way to get started with TinyML: implementing a Machine
Learning model on an Arduino board while creating something cool: a gesture recognition
system based on an accelerometer.

Gesture recognition is a process that attempts to recognize human gestures through the use of
mathematical algorithms.

To make the experiment simpler, the system is designed to recognize only two gestures: a punch
and a flex movement.
This, in the data science field, is called binary classification.

“Punch” gesture. Image by author.

 “Flex” gesture. Image by author.

But… Why “Ultra” TinyML?

The biggest challenge of this experiment is trying to run the prediction model on a very tiny
device: an 8-bit microcontroller.
To achieve this, you can use Neuton.

Neuton is a TinyML framework. It allows to automatically build neural networks without any
coding and with little machine learning experience and embed them into small compute
devices.
It supports 8, 16, and 32-bit microcontrollers, unlike TensorFlow Lite TinyML framework which
only supports 32-bit. You don’t need a powerful machine to use Neuton, it is an online software
tool which runs on a web browser.
Using the free plan, you can train and download an unlimited number of models.

…Let’s start!

The experiment is divided in three steps:

1. Capture the training dataset

2. Train the model using Neuton

3. Deploy and run the model on Arduino

 Experiment process flow. Image by author.

The hardware system

The gesture recognition system is composed by:

A microcontroller: Arduino Mega 2560

An accelerometer sensor: GY-521 Module

Arduino Mega 2560 is a microcontroller board based on the ATmega2560: a low-power 8-bit
MCU with 256KB flash memory, 32 general purpose working registers, UART interfaces, 10-bit
A/D converter, and many other peripherals.
Mega 2560 board is designed for complex projects: it has a larger space than other Arduino
boards (Uno, Nano, Micro, etc.). This makes it ideal for machine learning applications where
there are large amounts of data to process.

The GY-521 module is built around the InvenSense MPU-6050: a sensor containing, in a single IC,
a 3-axis MEMS accelerometer and 3-axis MEMS gyroscope.
Its operation is very precise since it contains for each channel an accurate digital converter. It is
able to capture the values of the X, Y and Z axes at the same time. Communication with MCU
takes place using the I2C interface.

Below, the connection circuit designed with Fritzing software:

Connection circuit: Arduino Mega2560 and GY-521. Image by author.

GY-521 is powered with the 5V and GND pins of Arduino Mega power section, while for data
communication the I2C pins are used (Pin 20 and Pin 21).
The remaining pins are optional and not useful for this application.

To verify if the GY-521 module is correctly supplied, connect the USB cable of the Arduino board
and check if the LED mounted on the sensor board is turning on.

GY-521: LED position. Image by author.

After verifying the sensor power supply, check if the I2C communication is working properly by
downloading the Adafruit MPU6050 Arduino library and opening the “plotter” example.

Add a library in Arduino IDE. Image by author.

 Adafruit MPU6050 Library. Image by author.

MPU6050 Examples. Image by author.

Upload the example sketch on the Arduino board, open the “Serial Plotter” within the Tools menu,
set 115200 in the baud drop-down menu and “shake” the sensor board. The expected result will
be the following:
 Serial Plotter of MPU6050 plotter example. Image by author.

Now, the system is ready to collect accelerometer and gyroscope data.

Capture training data

The first step to build the predictive model is to collect enough motion measurements.
This set of measures is called training dataset and it will be used to train the Neuton neural
network builder.

The easiest way to achieve this is to repeat several times the same two motions (punch and flex),
by capturing acceleration and gyroscope measurements and storing the result in a file.
To do this, you create an Arduino sketch dedicated to sensor data acquisition. The program will
acquire the measurements of each motion and will print the sensor measurements output on
the serial port console.

You will perform at least 60 motions: 30 for the first movement (punch) and 30 for the second
one (flex). For each motion, you will acquire 50 acceleration and 50 gyroscope measures in a 1
second time window (Sampling time: 20ms —50Hz). In this experiment, 60 motions are enough.
By increasing the number of motion measurements, you can improve the predictive power of
the model. However, a large dataset can lead to an over-fitted model. There is no “correct” dataset
size, but a “trial and error” approach is recommended.

The serial port output of the Arduino sketch will be formatted according to Neuton training
dataset requirements:

CSV format: it is a database file format. Each line of the file is a data record consisting of one
or more fields, separated by commas.

At least 50 data records (for binary classification task means at least 25 data records per
group).

First row must contain the column names (e.g., ax0, ay0, az0, gx0, gy0, gz0, …).
 A target variable column must be present. Each row must be assigned to a specific target
group (in this experiment: ‘0’ for punch, ‘1’ for flex).

Use a dot as a decimal separator.

Below, Arduino program for dataset creation:

IMU Sensor initialization and CSV header generation:

#define NUM_SAMPLES 50

Adafruit_MPU6050 mpu;
void setup() {
// init serial port
Serial.begin(115200);
while (!Serial) {
delay(10);
}
// init IMU sensor
if (!mpu.begin()) {
while (1) {
delay(10);
}
}

// configure IMU sensor

// [...]
// print the CSV header (ax0,ay0,az0,...,gx49,gy49,gz49,target)
for (int i=0; i<NUM_SAMPLES; i++) {
Serial.print("aX");
Serial.print(i);
Serial.print(",aY");
Serial.print(i);
Serial.print(",aZ");
Serial.print(i);
Serial.print(",gX");
Serial.print(i);
Serial.print(",gY");
Serial.print(i);
Serial.print(",gZ");
Serial.print(i);
Serial.print(",");
}
Serial.println("target");
}
 Acquisition of 30 consecutive motions. The start of a motion is detected if the acceleration
sum is above a certain threshold (e.g., 2.5 G).

#define NUM_GESTURES 30
#define GESTURE_0 0
#define GESTURE_1 1
#define GESTURE_TARGET GESTURE_0
//#define GESTURE_TARGET GESTURE_1
void loop() {
sensors_event_t a, g, temp;

while(gesturesRead < NUM_GESTURES) {

// wait for significant motion
while (samplesRead == NUM_SAMPLES) {
// read the acceleration data
mpu.getEvent(&a, &g, &temp);

// sum up the absolutes

float aSum = fabs(a.acceleration.x) +
fabs(a.acceleration.y) +
fabs(a.acceleration.z);

// check if it's above the threshold

if (aSum >= ACC_THRESHOLD) {
// reset the sample read count
samplesRead = 0;
break;
}
}

// read samples of the detected motion

while (samplesRead < NUM_SAMPLES) {
// read the acceleration and gyroscope data
mpu.getEvent(&a, &g, &temp);

samplesRead++;

// print the sensor data in CSV format

Serial.print(a.acceleration.x, 3);
Serial.print(',');
Serial.print(a.acceleration.y, 3);
Serial.print(',');
Serial.print(a.acceleration.z, 3);
Serial.print(',');
Serial.print(g.gyro.x, 3);
Serial.print(',');
Serial.print(g.gyro.y, 3);
Serial.print(',');
Serial.print(g.gyro.z, 3);
Serial.print(',');
 // print target at the end of samples acquisition
if (samplesRead == NUM_SAMPLES) {
Serial.println(GESTURE_TARGET);
}

delay(10);
}
gesturesRead++;
}
}

Firstly, run the above sketch with the serial monitor opened and GESTURE_TARGET set to
GESTURE_0. Then, run with GESTURE_TARGET set to GESTURE_1. For each execution, perform
the same motion 30 times, ensuring, as far as possible, that the motion is performed in the same
way.

Copy the serial monitor output of the two motions in a text file and rename it to
“trainingdata.csv”.

Example of training dataset in CSV format. Image by author.

Train the model with Neuton TinyML

The process of training a model involves providing a Machine Learning algorithm with training data to
learn from. It is the phase where you try to fit the best combination of weights and bias to a ML
algorithm to minimize a loss function.
Neuton performs training automatically and without any user interaction.
Train a Neural Network with Neuton is quick and easy and is divided into three phases:

1. Dataset: Upload and validation

2. Training: Auto ML

3. Prediction: Result analysis and model download

Dataset: Upload and validation

First, create a new Neuton solution and name it (e.g., Gesture Recognition).

Neuton: add new solution. Image by author.

Upload CSV training dataset file.

 Neuton: upload CSV file. Image by author.

Neuton validates CSV file according to dataset requirements.

Neuton: dataset validation. Image by author.

If the CSV file meets the requirements a green check will appear, otherwise an error
message will be shown.
 Neuton: validated dataset. Image by author.

Select the column name of the target variable (e.g., target) and click “Next”.

Neuton: target variable. Image by author.

 Neuton: dataset content preview. Image by author.

Training: Auto ML
Now, let’s get to the heart of training!

Neuton analyzes the content of the training dataset and defines the ML task type. With this
dataset, the binary classification task is automatically detected.

Neuton: Task type. Image by author.

Metric is used to monitor and measure the performance of a model during training. For this
experiment, you use the Accuracy metric: it represents how accurately class is predicted.
The higher the value, the better the model.

Neuton: Metric. Image by author.

Enable TinyML option to allow Neuton to build a tiny model for microcontroller.

Neuton: TinyML option. Image by author.

In the TinyML settings page, select “8-bit” in the drop-down menu and enable “Float
datatype support” option. This, because the microcontroller used in the experiment is an 8-
bit MCU with floating point number support.
 Neuton: TinyML settings. Image by author.

After pressing the “Start Training” button, you will see the process progress bar and the
percentage of completion.

Neuton: Training started. Image by author.

The first step is the data preprocessing. It is the process of preparing (cleaning, organizing,
transforming, etc.) the raw dataset to make it suitable for training and building ML models.
After data preprocessing completion, model training starts. Process can take a long time;
you can close the window and come back when the process is finished. During training you
can monitor the real-time model performance by observing model status (“consistent” or “not
consistent”) and Target metric value.

Neuton: Data preprocessing complete. Image by author.

Upon training is complete, the “Status” will change to “Training completed”. Model is
consistent and has reached the best predictive power.
 Neuton: Training completed. Image by author.

Prediction: Result analysis and model download

That’s all… Model is ready!

Neuton: Training completed. Image by author.

After the training procedure is completed, you will be redirected to the «Prediction» section.
In this experiment, the model has reached an accuracy of 98%. It means that from 100 predicted
records, 98 had been assigned to the correct class… that’s impressive!
Moreover, the size of the model to embed is less than 3KB.
This is a very small size, considering that the Arduino board in use is 256KB memory size and a
typical memory size for an 8-bit microcontroller is 64KB÷256KB.

Neuton: Metrics. Image by author.

To download the model archive, click on the “Download” button.

Neuton: Prediction tab. Image by author.

Deploy the model on Arduino
Now, it’s time to embed the resulting model into the microcontroller.

The model archive downloaded from Neuton includes the following files and folders:

/model: the neural network model in a compact form (HEX and binary).

/neuton: a set of functions used to perform predictions, calculation, data transferring, result
management, etc.

user_app.c: a file in which you can set the logic of your application to manage the
predictions.

Neuton model archive. Image by author.

First, you modify the user_app.c file adding functions to initialize model and run inference.

/*
* Function: model_init
* ----------------------------
*
* returns: result of initialization (bool)
*/
uint8_t model_init() {
uint8_t res;

res = CalculatorInit(&neuralNet, NULL);

return (ERR_NO_ERROR == res);

}
/*
* Function: model_run_inference
* ----------------------------
*
 * sample: input array to make prediction
* size_in: size of input array
* size_out: size of result array
*
* returns: result of prediction
*/
float* model_run_inference(float* sample,
uint32_t size_in,
uint32_t *size_out) {
if (!sample || !size_out)
return NULL;

if (size_in != neuralNet.inputsDim)
return NULL;
*size_out = neuralNet.outputsDim;

return CalculatorRunInference(&neuralNet, sample);

}

After that, you create the user_app.h header file to allow the main application using the user
functions.

uint8_t model_init();
float* model_run_inference(float* sample,
uint32_t size_in,
uint32_t* size_out);

Below, the Arduino sketch of main application:

Model initialization

#include "src/Gesture Recognition_v1/user_app.h"

void setup() {
// init serial port and IMU sensor
// [...]
// init Neuton neural network model
if (!model_init()) {
Serial.print("Failed to initialize Neuton model!");
while (1) {
delay(10);
}
}
}
 Model inference

#define GESTURE_ARRAY_SIZE (6*NUM_SAMPLES+1)

void loop() {
sensors_event_t a, g, temp;
float gestureArray[GESTURE_ARRAY_SIZE] = {0};
// wait for significant motion
// [...]
// read samples of the detected motion
while (samplesRead < NUM_SAMPLES) {
// read the acceleration and gyroscope data
mpu.getEvent(&a, &g, &temp);

// fill gesture array (model input)

gestureArray[samplesRead*6 +
gestureArray[samplesRead*6 +
gestureArray[samplesRead*6 +
gestureArray[samplesRead*6 +
gestureArray[samplesRead*6 +
gestureArray[samplesRead*6 +
0] = a.acceleration.x;
1] = a.acceleration.y;
2] = a.acceleration.z;
3] = g.gyro.x;
4] = g.gyro.y;
5] = g.gyro.z;

samplesRead++;

delay(10);
// check the end of gesture acquisition
if (samplesRead == NUM_SAMPLES) {
uint32_t size_out = 0;

// run model inference

float* result = model_run_inference(gestureArray,
GESTURE_ARRAY_SIZE,
&size_out);
// check if model inference result is valid
if (result && size_out) {
// check if problem is binary classification
if (size_out >= 2) {
// check if one of the result has >50% of accuracy
if (result[0] > 0.5) {
Serial.print("Detected gesture: 0");
// [...]
} else if (result[1] > 0.5) {
Serial.print("Detected gesture: 1");
// [...]
} else {
// solution is not reliable
Serial.println("Detected gesture: NONE");
}
}
}
 }
}
}

Model in action!
Project and code are ready!

/neuton_gesturerecognition
|- /src
| |- /Gesture Recognition_v1
| |- /model
| |- /neuton
| |- user_app.c
| |- user_app.h
|- neuton_gesturerecognition.ino

Now, it’s time to see the predictive model in action!

Verify that the hardware system is correctly setup

Open the main application file

Click on the “Verify” button and then on the “Upload” one

Open the Serial Monitor

Grab your hardware system in the hand and perform some motions.

For each detected motion, the model will try to guess what type of movement is (0-punch or 1-
flex) and how accurate the prediction is. If the accuracy of the prediction is low (0.5), the model
does not make a decision.

Below, an example of model inference execution:

 Serial monitor output of Neuton gesture recognition system. Image by author.

…Already done?
Doing Machine learning with Neuton is simple and fast. Model accuracy and performance
achieved on a low-power 8-bit microcontroller are impressive!
Neuton is suitable for fast-prototyping development. It allows user to focus on the application,
avoiding wasting time in complex and manual statistical analysis.

Here, you can find all Arduino sketches described in this article!

Tinyml AI Arduino Machine Learning Neuton

Follow

Written by Leonardo Cavagnis

199 Followers · Writer for Towards Data Science

Passionate Embedded Software Engineer, IOT Enthusiast and AI addicted.

More from Leonardo Cavagnis and Towards Data Science

Leonardo Cavagnis in The Startup

Android and MQTT: A Simple Guide

How to develop an MQTT Client with Android

8 min read · Dec 4, 2020

555 5
 Dominik Polzer in Towards Data Science

All You Need to Know to Build Your First LLM App

A step-by-step tutorial to document loaders, embeddings, vector stores and prompt templates

· 26 min read · Jun 21

4K 38
 Kenneth Leung in Towards Data Science

Running Llama 2 on CPU Inference Locally for Document Q&A

Clearly explained guide for running quantized open-source LLM applications on CPUs using LLama 2, C
Transformers, GGML, and LangChain

· 11 min read · Jul 18

1.92K 25

Leonardo Cavagnis

Firmware 101: STM32 Quickstart guide

A ready-to-use tutorial to start with firmware development on STM32 board

8 min read · Oct 13, 2021

134

See all from Leonardo Cavagnis

See all from Towards Data Science

Recommended from Medium

Simsangcheol

ONNX
ONNX (Open Neural Network Exchange) is an open-source format for representing machine learning
models. It was developed by Microsoft and…

2 min read · Mar 23

3
 Sheeza Shabbir

Deep Learning and Computer Vision-Deep Learning Architectures

Deep learning has become one of the most popular areas of machine learning in recent years, and it’s not
hard to see why. Deep learning…

5 min read · May 2

Lists

Predictive Modeling w/ Python

18 stories · 229 saves

Practical Guides to Machine Learning

10 stories · 239 saves

The New Chatbots: ChatGPT, Bard, and Beyond

13 stories · 76 saves

Natural Language Processing

480 stories · 103 saves
 Fateh Ali Shahrukh Khan

How to connect ESP32 to WiFi using ESP-IDF (IOT Development FrameWork)

Yes I know its example is available on esp-idf/examples/wifi/getting_started/station in your machine or on
Espressif git hub but running an…

6 min read · Jul 9

 Irene Aldridge

Quant Trading (and HFT): Performance 101

Understanding the Core Principles of Trading Performance.

5 min read · Jun 19

14

Juan F. Palomeque-Gonzalez

Arduino: 100 weekend projects

Arduino is a versatile microcontroller that has captured the hearts of hobbyists, makers, and engineers alike.
It is an excellent platform…

· 2 min read · Mar 3

15
 Denaya

Usage of Placeholders in Tensorflow

The purpose of this article is to explore what a placeholderis in Tensorflow.

5 min read · Jun 2

See more recommendations