Computer Vision System Toolbox 4.

0
Design and simulate computer vision and video processing systems
Computer Vision System Toolbox provides algorithms and tools for the design and simulation of computer vision and video processing systems. These capabilities are provided as MATLAB functions, MATLAB System objects, and Simulink blocks. The system toolbox includes algorithms for feature extraction, motion detection, object tracking, stereo vision, video processing, and video analysis. Tools include video file I/O, video display, drawing graphics, and compositing. For rapid prototyping and embedded system design, the system toolbox supports fixed-point arithmetic and C-code generation. Key Features Algorithms available as MATLAB functions, MATLAB System objects, and Simulink blocks Feature detection, extraction, and matching for applications such as automatic image registration, video mosaicking, and video stabilization Motion estimation, including block matching, optical flow, and template matching Stereo vision, including fundamental matrix estimation and stereo image rectification Video processing, including chroma resampling and deinterlacing Image processing, including edge detection, morphology, and filtering Video file I/O, video display, graphic overlays, and compositing Algorithm support for floating-point, integer, and fixed-point data types Automatic C-code generation (with MATLAB Coder or Simulink Coder) Feature-Based Registration Computer Vision System Toolbox supports automatic image registration by providing algorithms that use features to estimate the geometric relationships between images or video frames. Typical uses include video mosaicking, video stabilization, image fusion, and stereo vision. Feature Detection, Extraction, and Matching Feature detection, extraction, and matching are the first steps in the feature-based registration workflow. Features are a set of interest points that are likely to be common across a pair of related images. Feature extraction enables you to derive a set of feature vectors, also called descriptors, from pixels surrounding each interest point. The system toolbox offers capabilities to detect features that include corners, edges, and lines; extract descriptors from the features; and find the most likely paired matches of descriptors. Estimating Geometric Relationships With a set of matched interest points, you can infer the geometric relationship between two images or video frames. The feature detection, extraction, and matching workflow produces many interest points and typically includes outliers. To estimate the geometric relationship with this set of interest points, you can exclude outliers with statistically robust methods such as RANSAC and least median of squares. With this workflow, the system toolbox can produce a projective or affine transformation that describes the geometric relationship between a pair of images or video frames.

Feature-based registration, as used for video stabilization. Interest points are detected in two sequential video frames (top) using corner features; the putative matches are determined with numerous outliers (bottom left); and outliers are removed using the RANSAC method (bottom right).

Motion Estimation and Tracking The system toolbox provides a variety of motion estimation algorithms such as optical flow, block matching, template matching, and background estimation using Gaussian mixture models (GMM). Evaluation metrics for finding the best block match include MSE, MAD, MaxAD, SAD, and SSD. These algorithms create motion vectors, which relate to the whole image, blocks, arbitrary patches, or individual pixels, depending upon the algorithm.

Detecting moving objects using a stationary camera. In this series of video frames, optical flow is calculated and detected motion is shown by overlaying the flow field on top of each frame.

Video tracking is a common use for motion estimation algorithms. You can use calculated motion to identify a moving object or measure the movement of a detected object over consecutive video frames. The system toolbox also provides Kalman filtering to predict the movement of an object in upcoming video frames.

Estimation of camera motion by template matching to calculate a motion vector (left). Using the motion vector, the video is stabilized for camera motion (right).

Stereo Vision Stereo vision is the process of reconstructing a 3D scene from two or more views of the scene. Using the system toolbox, you can: Perform uncalibrated stereo image rectification on a pair of stereo images Match individual pixels along epipolar lines to compute the disparity map Calculate the distance from the camera for each point in the disparity map using knowledge about the intrinsic parameters of the camera Backproject the disparity map with the original stereo images to create a 3D reconstruction of the scene

Reconstructing a scene using a pair of stereo images. To visualize the disparity, the right channel is combined with the left channel to create a composite (top left). A disparity map of the scene (top right) is then calculated by solving the point correspondence problem, and a 3D rendering of the scene (bottom) is reconstructed.

Stereo Image Rectification Stereo image rectification transforms a pair of stereo images so that a corresponding point in one image can be found in the corresponding row in the other image. This process reduces the 2D stereo correspondence problem to a 1D problem, which simplifies the process of determining the depth of each point in the scene from the camera. You can rectify a pair of stereo images with the system toolbox by determining a set of matched interest points, estimating the fundamental matrix, and then deriving two projective transformations.

Results from stereo image rectification. Non overlapping areas are show in red and cyan.

Video Processing, Visualization, and Graphics Computer Vision System Toolbox provides algorithms and tools for video processing workflows. You can read and write from common video formats, perform common video processing algorithms such as deinterlacing and chroma-resampling, and display results with text and graphics burnt in to the video. Video processing in MATLAB uses System objects, which avoids excessive memory use by streaming data to and from video files.

Video deinterlacing in MATLAB.

Video I/O Computer Vision System Toolbox can read and write multimedia files in a wide range of formats, such as AVI, MPEG, and WMV. You can stream video to and from MMS sources over the Internet or a local network. You can

acquire video directly from Web cameras, frame grabbers, DCAM-compatible cameras, and other imaging devices using Image Acquisition Toolbox. Simulink users can also use the MATLAB workspace as a video source or sink. Visualization The system toolbox includes a video viewer with many features. You can: View video streams in-the-loop as the data is being processed View any video signal within your code or block diagram Use multiple video viewers at the same time Freeze the display and evaluate the current frame Display pixel information for a region in the frame Pan and zoom for closer inspection as the simulation is running Start, stop, pause, and step through Simulink simulations one frame at a time

Model with viewers for four videos: (from left) original, estimated background, foreground pixels, and results of tracking.

Graphics Adding graphics to video helps with visualizing extracted information or debugging problems with a system design. You can insert text in order to display the number of objects or keep track of other key information. You can insert graphics, such as markers, lines, and polygons to mark found features, delineate objects, or highlight other key features. Inserted text and graphics are incorporated into the data itself rather than as a separate layer.

You can also combine two video sources in a composite that can highlight objects or focus attention on a key region.

Images with text and graphics inserted. Adding these elements can help you visualize extracted information and debug your design.

Stream Processing in MATLAB and Simulink Computer Vision System Toolbox supports a stream processing architecture in both MATLAB and Simulink. In a stream processing architecture, one or more video frames from a continuous stream are processed at a time. This type of processing is appropriate for analysis of large video files or systems with live video. In MATLAB, stream processing is enabled by System objects, which use MATLAB objects to represent time-based and data-driven algorithms, sources, and sinks. System objects implicitly manage many details of stream processing, such as data indexing, buffering, and management of algorithm state. You can mix System objects with standard MATLAB functions and operators. Most System objects have a corresponding Simulink block with the same capabilities. Simulink handles stream processing implicitly by managing the flow of data through the blocks that make up a Simulink model. Simulink is an interactive graphical environment for modeling and simulating dynamic systems that uses hierarchical diagrams to represent a system model. It includes a library of general-purpose, predefined blocks to represent algorithms, sources, sinks, and system hierarchy. Computer Vision System Toolbox provides a library of blocks specifically for the design of computer vision and video processing systems.

An abandoned object detection model (top). The three viewers (bottom) show steps in the process of detecting and tracking an abandoned object in a live video stream from a camera in a train station.

System Design and Implementation With MATLAB and Simulink your workflow for rapid prototyping, verification, and implementation can be integrated with algorithm development. You can: Convert floating-point algorithms into fixed-point representations to perform bit-true simulations Create system-level test benches to verify system behavior against requirements before implementing hardware and software Generate real-time C code from your MATLAB code or Simulink model and then download it onto a supported DSP board for real-time evaluation Fixed-Point Modeling Many real-time systems use hardware that requires fixed-point representation of your algorithm. Computer Vision System Toolbox supports fixed-point modeling in all relevant blocks and System objects with dialogs boxes and object properties that help you with configuration. Support for fixed point in the system toolbox includes: Word sizes from 1 to 128 bits Arbitrary binary-point placement Overflow handling methods (wrap or saturation) Rounding methods, including ceiling, convergent, floor, nearest, round, simplest, and zero

The Fixed-Point Tool in Simulink Fixed Point facilitates the conversion of floating-point data types to fixed point. The tool tracks overflows and maxima and minima, helping you to configure fixed-point properties. Code Generation Support Once you have developed your algorithm or system model, you can automatically generate C code from it for verification, rapid prototyping, and implementation. Most System objects, functions, and blocks in Computer Vision System Toolbox can generate ANSI/ISO C code using MATLAB Coder, Simulink Coder, or Embedded Coder. You can select optimizations for specific processor architectures and integrate legacy C code with the generated code to leverage existing intellectual property. You can generate C code for both floating-point and fixed-point data types.

Simulink model designed to create code for a specific hardware target. This model generates C code for a video stabilization system and embeds the algorithm into a digital signal processor (DSP).

Image Processing Primitives Computer Vision System Toolbox includes image processing primitives for solving frequent system problems, such as interfering noise, low dynamic range, and out-of-focus optics. Unlike similar functions in Image Processing Toolbox, these System objects and Simulink blocks support fixed-point data types and C-code generation. These primitives include: 2D spatial and frequency filtering Image pre- and post-processing algorithms Morphological operators Geometric transformations Color space conversions

Resources
Product Details, Demos, and System Requirements www.mathworks.com/products/computer-vision Trial Software www.mathworks.com/trialrequest Sales www.mathworks.com/contactsales Technical Support www.mathworks.com/support Online User Community www.mathworks.com/matlabcentral Training Services www.mathworks.com/training Third-Party Products and Services www.mathworks.com/connections Worldwide Contacts www.mathworks.com/contact

2011 The MathWorks, Inc. MATLAB and Simulink are registered trademarks of The MathWorks, Inc. See www.mathworks.com/trademarks for a list of additional trademarks. Other product or brand names may be trademarks or registered trademarks of their respective holders.

10