FACE RECOGNITION ON THE EDGE – REDUCE COST,

COMPLEXITY, AND POWER CONSUMPTION OF YOUR DESIGN

COOPER CARNAHAN
APRIL 2020

PUBLIC
Face Recognition Target Applications
• Anywhere you want to embed user personalization and identification
for advanced human/machine interfaces and access control

• Smart home devices

− Thermostats, HVAC and lighting controls, smart mirrors
• Smart Safety/Security/Alarm devices
− Alarm panels, access control, smart locks
• Smart appliances
− Washing machines, dryers, ovens, refrigerators, stoves and cook tops,
microwave ovens, vent hoods, dishwashers
− Countertop appliances (coffee machines, multi-cookers, microwave ovens)
• Smart toys
− Educational, Games & puzzles, Plush, App-enabled
− Interactive playmates/robotics
• Smart industrial devices
− Power tools, ergonomic stations, industrial workstations
− Industrial automation and access control
1 PUBLIC
MPU vs. MCU
MPU MCU
(Application Processor) (Embedded Processor)
Smart phones & tablets, Switches,
Smart screens, Appliances,
Applications
Routers/gateways, Power tools,
Robo-cleaners, etc. Toys, etc.
Graphics Complex UI None, or basic bit mapped
Operating System Linux, Android, iOS, Windows, … FreeRTOS, MicroC/OS, Mbed OS, …
Virtual Memory Support Memory Management Unit No
Storage Giga Bytes (NAND Flash) Mega Bytes (NOR Flash – often on chip)
Memory Giga Bytes (SDRAM) Mega Byte (SRAM – usually all on chip)
Arm Cortex-A Arm Cortex-M
CPU
Intel Core i7 Intel 8051
Number of cores 1, 2, 4, 8, … 1 (or 2)
NXP Example Family i.MX 8M i.MX RT1060

2 PUBLIC
Minimize Cost, Power Requirements, and Design Complexity

• Replace expensive and power

hungry “MPU+PMIC+MCU” design
with simple MCU-only design
− Drastic cost system cost reduction
− Drastic simplification of design
− Reduce system power
− Reduce PCB footprint, improve form factor
flexibility
• Reduce memory subsystem
− Replace huge expensive DDR and Flash
with low cost SDRAM and QSPI Flash
− Additional power reduction
− Further PCB footprint reduction

3 PUBLIC
4 PUBLIC
Agenda
• i.MXRT106F Solution Overview
− Features
− Use-cases
− What is a “Solution?”
• Oasis-Lite Inference Engine Overview
− Inference Engine Highlights
• Out-of-Box Demonstration
− Detecting + Registering Faces
− Emotion Recognition
− Verbose Mode Debug Logging
− Detection Resolution
• Q&A
5 PUBLIC
i.MX RT106F – MCU-Based Solution for Face Recognition
• MCU-based face & emotion
recognition
− Arm Cortex-M7, 600 MHz, 1 MB
SRAM
− Lowers BOM cost ~50% compared
to MPU implementations
• Offline
− Eliminates cost, complexity & privacy
issues of cloud-based solutions
• Slashes time-to-market
− Fullreference design, software
source, schematics, BOM & layout
− From concept to launch in <6 months

6 PUBLIC
SLN-VIZN-IOT Dual Camera Adapter
• Uses combination IR + RGB camera
− Cheaper than 3D time-of-flight solutions
• Liveness/anti-spoofing capable
- Designed for consumer security apps
− Defends against color and grayscale pictures
as well as phone displays
• Low-light face recognition
− Good for nighttime applications
• **3D Time-of-flight in development**

7 PUBLIC
MCU-based Face Recognition – Example Use Cases
• Secure Access Hurry up, Dad, get in
− Use: Replace key, card, BLE/phone with user face range with your
− Example Application: Residential Smart Lock phone so I can get in
the house…
• User Authorization
− Use: Control equipment access levels / authorities by
user face
− Example Application: Cooktop, Forklift

• User Identification
− Use: Identify and catalog users (by name or
anonymously – Friction Free Identification)
− Example Application: Retail loyalty program at POS

• User Customization • Emotion Detection

− Use: Customize environmental or equipment controls − Use: Identify user mood (and react)
based on user − Example Application: Toys, Guest Ratings at POS
− Example Application: Thermostat, Lighting Control

8 PUBLIC
What is a “Solution?” – Certified, Production Grade HW & SW
OOB HW/SW Software Source
• Production-ready hardware
− Cost and form-factor optimized

• Pre-integrated production-ready software,

fully tested & FCC/CE certified
− Source code provided for everything but
inference engine
Schematics Layout Certifications
• NXP provides a single point of contact
for support, licensing and procurement

• Easy security enablement

BOMs Documentation
• Plug & play out-of-the-box-experience U1
U3
MIMXRT1052DVL6B i.MXRT1050 Cross Processor
W9812G6JB-6I 128Mbit SDRAM 3V 166MHz
U4 IS26KL256S-DABLI00256Mb Hyperflash 3V 100MHz Flash
U5 A7101CHTK2 Secure IoT/Authentication Conn IC
U6 LBEE5KL1DX-883 IC WIFI BT/BLE B/G/N 3-4.8V LGA46
U7 MKW21Z512VHT4 KINETIS L 32-BIT MCU CORTEX-M0
U8,U9 NX3L2267GM,115 IC ANALOG SWITCH SPDT 10XQFN
U10 XCL214B333DR DC/DC CONVERTER 3.3V 5W

9 PUBLIC
i.MX RT106F MCU based solution for face recognition – HW

Parallel

SPI • Dev kit: SLN-VIZN-IOT

− On sale now
− $199 (US MSRP)

• i.MX RT106F includes

license for all face
recognition software
− <$4.00 (100k resale)

• More information
http://www.nxp.com/mcu-vision

10 PUBLIC
i.MX RT106F MCU solution for face recognition – SW

• Pre-integrated software
minimizes development time

• Out of the box automatic

registration & recognition
− Name not required

• <1 second inference time

− Scalable, power vs speed

• Register up to 200 faces

locally

11 PUBLIC
Jumpstart Your Design

Pre-compiled binaries programmed at factory

Automated Flashing Tools

MCUXpresso-Compatible SDK

12 PUBLIC
Import Example Projects and Real Source Code

13 PUBLIC
For more information… http://www.nxp.com/mcu-vision

CC [2]5

• Buy SLN-VIZN-IOT kits CC [2]6

• View getting started pages and videos

• Download schematics, BOM, layout

− Kit comes with software download code
 Source, binaries, documentation
 Available in MCUXpresso SDK in 3Q20

14 PUBLIC
Slide 15

CC [2]5 Replace with arrow landing page

Cooper Carnahan, 4/6/2020

CC [2]6 Add link to arrow buy now

Cooper Carnahan, 4/6/2020
CC14

15 PUBLIC
Slide 16

CC14 Figure out what to do with this section

Cooper Carnahan, 4/9/2020
Oasis-Lite Face Detection & Recognition Pipeline
• Multiple distinct AI models involved in face detection and recognition CC [2]3
− Allows user to choose the features they want
 Updates to different stages of the pipeline easier to make
− New features continuously being added

Face Rec
Image Liveness Face Quality Face Face Face Feature
Image Capturing Face Detection Display
Preprocessing Detection Classification Alignment Recognition Algorithm
Update

16 PUBLIC
Slide 17

CC [2]3 Not sure we're even supposed to say this now

Cooper Carnahan, 4/6/2020
Oasis-Lite Inference Engine
• High Performance
− Model quantized can best use the DMIPS performance from an Arm Cortex-M core.
− NXP provided optimization dedicated for 32bit MCUs.
− Better performance than any available open-source engine (ex. 1/5 inference time and 1/10th memory cost of TensorFlow
Lite)

• High Accuracy
- The quantization algorithm keeps the accuracy same or very close to raw floating-point data model.
- Benchmark test result confirmed the same accuracy with raw model for facial recognition.

• Low Memory Footprint ( compare with 3rd party open source )

− Specially designed for mobile devices and fit for MPUs & MCUs with quick model deployment.
− NXP-dedicated code optimization for single Arm Cortex-M core and low runtime memory applications.

17 PUBLIC
18 PUBLIC
Out of Box Experience
• Face Recognition
− Detection + recognition
− Registering a face
• Emotion Recognition
−3 different configurations enabled by
default
• Debug Logging
−3 different configurations
− Useful for communication with external
devices
• Detection Resolution
− Configureresolution for higher range or
higher performance
• Additional Features

19 PUBLIC
Detecting and Registering Faces
• Point-and-click
• Detection indicated by red LED and box around user’s face
• Press button on-board or use serial command to begin registering a face

20 PUBLIC
21 PUBLIC
“Friction Free” Interface
• Automatically registers unrecognized with unique identifier
• Useful for applications where user input is limited

22 PUBLIC
Enable Emotion Recognition SHELL>> emotion 0
SHELL>> emotion recognition is disabled
Command: “emotion <0|2|4|7>”
SHELL>> emotion 4
2
2 Emotion 4 Emotion 7 Emotion SHELL>> 4
2 types emotion recognition is set
Emotion
Mode Mode Mode
SHELL>> emotion 4
Neutral X X X SHELL>> 4 types emotion recognition is set

Happy X X X SHELL>> emotion 7

SHELL>> 7 types emotion recognition is set
Angry X X

Surprised X X

Sad X

Fear X

Disgust X

Accuracy 92% 81% 62%

23 PUBLIC
24 PUBLIC
 Debug Messages
• Command: “verbose <0|1|2|3>” Ex. Message:

Time Stamp Debug Message

Message
Importance “verbose 0” “verbose 1” “verbose 2” “verbose 3”
Type

Critical [L1] High X X X

Detailed [L2] Medium X X

Priority Level
Abbreviation Definition
Dt Time taken to detect face
Rt Time taken to recognize face
Misc. [L3] Low X Predictive accuracy/confidence value of
Sim
face rec
Face_id Internal face database identifier

25 PUBLIC
26 PUBLIC
Configuring Detection Resolution – VGA vs QVGA
VGA Command: “detection resolution <vga|qvga>”
(Face detected
at ~1.4m)
Detection & Detection
Requires
Mode Resolution Recognition Inference
SDRAM?
Range Speed

VGA 640x480 Yes .3m ~1.4m 100ms

QVGA
(default)
(Face detected QVGA
at ~.7m) 320x240 No .3m ~ .7m 80ms
(default)

**Does not affect video output resolution, just resolution of image

passed to inference engine allowing increased range**

27 PUBLIC
Additional Out of Box Features
SHELL>> help “camera white_pwm <value>”: PWM pulse width for white led, value should
be between 0 (inactive) to 100
“help”: List all the registered commands
“version”: The version information
“exit”: Exit program
“save n”: Save database into flash memory
“list”: list all registered users

“mode <auto|manual>”: User add mode “updateotw”: Reboot the board and start the OTW firmware update

“add username”: Add user “reset”: Reset the MCU

“add –s”: Stop the former add command issued “emotion”: Emotion recognition types <0|2|4|7>

“del username”: Delete user “liveness”: liveness <on|off>

“del –a”: Delete all users “detection resolution <qvga|vga>”: Set detection resolution and reset
“rename oldusername newusername”: Rename user
“display”: display output_mode <rgb|ir>
“verbose <0|1|2|3> (none-all)”: Debug message detail level setting
“display”: display output_device <usb|riverdi>
“camera ir_pwm <value>”: PWM pulse width for IR led, value should be between 0
(inactive) to 100 “display”: display interface <loopback|infobar>

28 PUBLIC
 Arrow Engineering Services
Expertise where you need it – across the entire design cycle
Design & Prototype &
Ideation Validation Testing
I N N O VAT E
New Product Certification Software/Cloud
Introduction Services Implementation

Product Business Model Process

Transformation Transformation Transformation
TRANSFORM
Value Engineering Legacy
Cloud Migration
(Sustenance & Reengineering) Modernization

Remote Device CloudOps &

SCALE Management DevOps Manufacturing

Supply, Demand & Order Management,

Support Services
Inventory Management Fulfilment & Logistics

29 PUBLIC
29
 Learn, Design & Buy – arrow.com

BUY

Purchase: https://www.arrow.com/en/products/sln-vizn-iot/nxp-semiconductors
Support: Contact your local Arrow FAE for support or send email to: NXPFaceRecognition@arrow.com

30 PUBLIC
30