Design of Real-Time Video Processing Systems in Simulink With the Video & Image Processing Blockset

David Jackson Product Marketing Manager Video & Signal Processing The MathWorks dave.jackson@mathworks.com

Agenda
! Video & Imaging System Design
! ! ! ! Imaging with MATLAB Key Challenges Model-Based Design Simulink

! Featured Product:
! Video & Image Processing Blockset

! Executable Specifications
! Benefits, ROI Study

! Video Design Example

! Edge Detection

! Q&A
2005 Altera Corporation 2

Intelligent Video Demonstration

! Traffic Sign Detection Example

2005 Altera Corporation 3

The MathWorks at a Glance

! Headquarters: Natick, Massachusetts, USA ! European Offices: UK, France, Germany, Spain, Benelux, Italy, Switzerland, Nordic ! North American Offices: CA, MI, Washington DC, TX ! Asia/Pacific Office: Korea ! Worldwide Consulting ! Distributors in 20 Countries
Earths Topography on an Equidistant Cylindrical Projection, Created with the MATLAB Mapping Toolbox

! Revenues of ~$300M in 2004 ! Privately Held ! Over 1,000 Employees Worldwide, 1/3 in Product Development ! Worldwide Revenue Balance: 50% North America, 50% International ! More than 1,000,000 Users in 175+ Countries
2005 Altera Corporation 4

Video System Design

Some Image & Video Processing Systems

UAV Automotive Safety Communications Defense

Office Equipment

Security Cameras

Semiconductor

Medical Devices

2005 Altera Corporation 6

Video System Design Challenges

! Extreme Computation Demands ! Embedded System Resource Constraints
! Real-Time Requirements

! Designing for a Target Hardware

! DSP, FPGA

! End-Product Performance, Power, Size

! Roadmap for Adding Features, Performance

! Testing & Validating Results

2005 Altera Corporation 7

Simulink
! Foundation for Model-Based Design, Automatic Code Generation, & Verification & Validation ! Open Architecture for Integrating Models From Other Tools ! Applications in Controls, Signal Processing, Communications, & Other System Engineering Areas The Leading Environment for Modeling, Simulating, & Implementing Dynamic & Embedded Systems
2005 Altera Corporation 8

Simulink Key Features

! Hierarchical, Component-Based Modeling ! Open Application Program Interface (API) ! Hybrid (Mixed-Signal), Multirate & Multitasking System Simulation ! MATLAB Integration ! Extensive Library of Predefined Blocks ! Application-Specific Libraries Available
! ! ! ! ! Video & Image Processing Signal Processing Communications Control Physical Modeling

2005 Altera Corporation 9

Model-Based Design
Requirements & Specifications Design Implementation Test & Verification

Continuous Verification Model Elaboration

Executable Specifications ! Reduce Ambiguity Avoid Re-Work

Design With Simulation ! Rapid Design Iterations

Automatic Code Generation ! Minimizes Coding Errors

Continuous Verification ! Detect Errors Earlier

2005 Altera Corporation 10

Featured Product Highlight Video & Image Processing Blockset

! Provides Over 60 Components & 100s of Algorithms Focused on Implementation of Embedded Systems

Streaming Video In/Out

Detection, Thresholding

Tracking, Counting

Background Estimation

2005 Altera Corporation 11

Video & Image Processing Blockset Libraries:

! ! ! ! ! ! ! ! ! ! !
12

Analysis & Enhancement Conversions Filtering Geometric Transforms Morphological Operations Sinks Sources Statistics Text & Graphics Transforms Utilities

2005 Altera Corporation

Signal Processing Blockset Libraries:

! ! ! ! ! ! ! ! ! !
13

Estimation Filtering Math Functions Quantizers Signal Management Signal Operations Sinks Sources Statistics Transforms

2005 Altera Corporation

Fixed-Point & Integer Modeling*

! Optimize Performance & Numerics Due to Finite Word Effects ! Built-In Tools for Scaling & Modeling Finite Word Effects ! Easy to Change Parameters to Simulate Impact of Rounding, Overflow, Etc.

* Signal Processing & Video & Image Processing Blocksets require Simulink Fixed-Point for Integer & Fixed-Point Data Types 2005 Altera Corporation 14

Real-Time Workshop & RTW Embedded Coder Automatic C-Code Generation

! Generate C-Code from Video & Image Processing Models
! Rapidly Prototype Your Algorithms & Systems ! Test & Verify With Target Hardware ! Easily Retarget Your Model to Different Processors

2005 Altera Corporation 15

Application Example

Video & Image Processing Blockset Edge Detection

Live Simulink Demo

2005 Altera Corporation 17

Edge Detection
Easy to Import Streaming Video Into the Simulation Handy Viewers for Inspecting Video at Any Point in the Algorithm

Options Important to Embedded System Designers

2005 Altera Corporation 18

Model & Simulate Double, Single, Fixed-Point Data Types

Executable Specifications
Accelerating Your Video & Image Processing System Designs

Benefits of Executable Specifications

! Explore Design Trade-Offs in Software Before Costly Hardware/Prototyping Stage ! Simulate With Different Data Sets (Videos) to Test Against Multiple Environments, Situations, Etc. ! Block Diagram Makes It Easy to Understand Layout of Algorithm/Design
! Reduces Specification Ambiguity ! Increases Communication Between Teams/Design Stages

! Facilitates IP-Reuse Between Team Members & Projects ! Copies of return on investment (ROI) white paper available at: www.mathworks.com/roi
2005 Altera Corporation 20

Summary

Video & Image Processing Blockset Design & Simulation for Video & Image Processing Systems
! Tools for Model-Based Design:
! Create Executable Specification ! Model Floating- & Fixed-Point Data Types ! Trade-Off Analysis & Visualization ! Construct Test Harnesses for Verification ! Automatic C-Code Generation for DSP, U

2005 Altera Corporation 22

Model-Based Design Tutorial

Simulink, DSP Builder & SOPC Builder
Alex Soohoo DSP Product Marketing Manager Altera Corporation

Alteras Model-Based DSP Design Flow

SOPC Builder

DSP Builder

DSP System Development

2005 Altera Corporation 24

DSP Builder Design Flow

MATLAB/Simulink Domain (System Simulation & Verification)

HDL Domain ( Hardware Implementation/ RTL Simulation)

2005 Altera Corporation 25

DSP Builder Overview

DSP Builder

2005 Altera Corporation 26

DSP Builder Overview

Creates Creates HDL HDL Code Code Place Place & & Route Route Creates Creates Simulation Simulation Testbench Testbench

HDL HDL Synthesis Synthesis

Creates SOPC Builder Component

Verify Verify in in Hardware Hardware

2005 Altera Corporation 27

Download Download Design Design Hardware-in-the-Loop Hardware-in-the-Loop

DSP Builder Library Components

! Arithmetic ! Bus Manipulation ! Complex Signals ! Logical Components ! SOPC Ports ! Storage ! MegaCore Functions ! Rate Change ! State Machine ! Altera Library ! DSP Board Components
2005 Altera Corporation 28

SOPC Builder: System Integration Tool

Altera Value: Standard IP Functions & Interconnect Fabric
System on a Programmable Chip
Standard Standard IP IP Function Function I/O I/O Channel Channel Control Control On-Chip On-Chip Memory Memory Flash Flash Memory Memory Interface Interface SDRAM SDRAM

Custom Custom Logic Logic

Your Value: Custom Logic

2005 Altera Corporation 29

Nios II Processor Overview

! Family of 32-Bit RISC Processors ! Library Peripherals & Interfaces ! Import Custom Peripherals ! Royalty-Free ! IDE/Debugger ! RTOS ! TCP/IP Stack Nios II ! Performance over 200 DMIPs ! Cost As Low as 35 of Logic
2005 Altera Corporation 30

Avalon Switch Fabric

Cache

Nios II CPU Debug

UART GPIO Timer SPI SDRAM

Controller

On-Chip ROM On-Chip ROM

Your Design Here

Nios II IDE: Editor & Software Tool-Chain

Project Management
Project Navigation Source Files Features: Project Manager Editor & Compiler Debugger Flash Programmer

Source Editor
Basic Editing Capabilities C/C++ Syntax Highlighting Comprehensive Search Help Feature

Compiler
Compiler Based on GNU Tool Chain Command-Line Operation Possible

Integrated Editor & Software Tool-chain Functionality

2005 Altera Corporation 31

Building a Real-Time Video System

! Target Application:
! Video Edge Detection

! Design Steps
! ! ! ! ! Simulink Testbench DSP Builder Implementation SOPC Integration Control Software Development Program Hardware Platform

! Hardware Demo

2005 Altera Corporation 32

Video Hardware System

2 3 1 RAM
Avalon DMA DMA DMA DMA DMA DMA DMA DMA

Edge Edge Detection Detection Co-Processor Co-Processor

SOPC SOPC Builder Builder

2005 Altera Corporation 33

DSP DSP Builder Builder

Application Specification
! Prewitt Edge Detection Algorithm ! VGA Resolution 16-bit RGB(5:6:5) Format Video
! 15 Hz Input Via Camera ! 60 Hz Output Via VGA Display Controller ! One Pixel per Clock

! Sample-Based Algorithm ! Target Hardware: Stratix II EP2S60 DSP Development Board

2005 Altera Corporation 34

Simulink Testbench
! Build Model Using Video & Imaging Processing (VIP) Blockset ! Read Video From a File to Matrices Representing Image (One Matrix per Color Channel) ! Outputs Three Displays
! Original Image ! Edge Detected Image ! Edge Detected Image Overlaid on Original ! Convert Floating-Point to 5:6:5 RGB ! Requires Matrix to Sample Conversion Blocks

! Testbench Designed to Match HW Specification

2005 Altera Corporation 35

Testbench Top Level

2005 Altera Corporation 36

Testbench 5:6:5 Sampler

! AVI Reader Outputs RGB Float Matrices for Image
! Specification Requires 5:6:5 Sample-Based Data ! Convert Format Using Simulink Blocks

2005 Altera Corporation 37

Custom Edge Detection Test Core

! Place Wrapper Around Block From VIP Blockset to Match HW Specification & Testbench

2005 Altera Corporation 38

Testbench Output

2005 Altera Corporation 39

DSP Builder Implementation

! Substitute Simulink Test Core With DSP Builder Blockset

2005 Altera Corporation 40

Building the Edge Detector

! Consider Vertical Edge Detection
! For Each Pixel in the Image Apply the Following 3x3 Mask on the Intensity I of Adjacent Pixels

-1 -1 -1

0 0 0

1 1 1

2005 Altera Corporation 41

Vertical Edge Detection

! Have to Identify 3x3 Pixel Blocks
I I(x-1,y+1) (x-1,y+1) Target Pixel (x,y) I I(x-1,y) (x-1,y) I I(x-1,y-1) (x-1,y-1) I I(x+1,y+1) (x+1,y+1) I I(x+1,y) (x+1,y) I I(x+1,y-1) (x+1,y-1)

! Use Delay & Memory Delay Blocks

! Delay Uses Registers for Pixel Identification ! Memory Delay Uses RAM for Line Storage
2005 Altera Corporation 42

1 Z Delay

1 Z100 Memory Delay

Vertical Edge Detection

! Use DSP Builder Blocks to Carry Out Arithmetic on Pixel Intensities
' I ( x ( 1, y ( 1) $ % " ( % " Edge( x, y ) ) % I ( x ( 1, y ) " % " ( % " % I ( x ( 1, y ! 1) " & # ' I ( x ! 1, y ( 1) $ % " ( % " % I ( x ! 1, y ) " % " ( % " % I ( x ! 1, y ! 1) " & #

! Use Pipelined Adder Blocks

A(7:0) + R(7:0) B(7:0) Pipelined Adder
2005 Altera Corporation 43

Adding DSP Builder Blocks

2005 Altera Corporation 44

Vertical Edge Detection Block

! Horizontal Edge Detection Works Using a Similar 3x3 Mask

Add x+1 Inputs Subtract to Determine Edges

Add x-1 Inputs

2005 Altera Corporation 45

DSP Builder Core Top Level

2005 Altera Corporation 46

System Integration SOPC Builder

! DSP Builder Contains Avalon Port Library
! Provides All Signals Required for Avalon Slave
Write Data 7:0 Datain Address Address

! Other DSP Builder Blocks Used

! FIFO Blocks for DMA Performance ! Comparison & AND Gates for Control
d(7:0) rreq wreq FIFO q(7:0) full empty Usdw(5:0) FIFO
2005 Altera Corporation 47

a == b Comparator

AND

Logical Bit Operator

Creating the Co-Processor

! SOPC Slave Block Built Within DSP Builder
! Connected to Tested Edge Detection Unit

2005 Altera Corporation 48

Building the SOPC System

1. Generate SOPC Component Inside DSP Builder
! Using Signal Compiler Block

2. Add Edge Detection Co-Processor Component With SOPC Builder System Editor 3. Add Additional SOPC Components
! Nios II Processor ! Camera Capture ! VGA Device Controller ! Memory Controller

2005 Altera Corporation 49

SOPC Builder System Editor

Nios II Edge Detector + DMAs VGA Out Camera

2005 Altera Corporation 50

Nios II Processor Control Code

! Simple C Language Program
! Rotate Frame Buffers " 1 Buffer for Camera to Write To " 1 Buffer for Edge Detector Block to Work On " 1 Buffer for Display ! Setup DMA Transfers to Processing Module ! Push Button Controls " Toggle On/Off, Video Display

2005 Altera Corporation 51

Program Hardware: Stratix II DSP Board + Camera Module

VGA Output Connector VGA Resolution CMOS-Sensor Camera-on-a-Chip Module

Peripheral Expansion Daughter Card Interface

2005 Altera Corporation 52

Demo