NJIT Capstone Project

Team 9

Visual and Auditory Recognition System for Autistic

Children

Steffen Pharai
Alan Singh
Maciej Sprycha

Advisors: Dr. Sahin (NJIT), Dr. Seyffert (JFK MC), Dr. Rosenberg (JFK MC)
 Product Description
• What is autism?
– Developmental disorder that usually appears within
first 3 years of an individual’s life
– Affects brain’s normal development of social and
communication skills
• Symptoms
– Impaired communication skills
– Reduced social interaction capabilities
– Withdrawn behavior
– Heightened/lowered senses
 Product Description
• AKA “Eureka” Moment Monitor
– Essentially measures when an autistic child is able to
recognize an onscreen stimulus (picture, video, shape,
audio, etc.) after it is slowly re-constructed
• Complete software + hardware package
– Main use in research at JFK MC
• Software
– Provides adjustable visual and auditory stimuli
• Hardware
– Trigger system for recognition feedback
– Digital (0/1) and analog (eg. Optical flow)
 Customer Needs

• Customer: NJ Neuroscience Institute at JFK

Medical Center
• Main Use: Internal research
• General requirements
– Software package with adjustable inputs/outputs
– Host PC environment flexibility
– Versatile trigger system
– Adjustable stimuli type/difficulty
 Customer Needs
• Compact GUI System
– Can run in windowed and full screen modes
– Drop down menus for quick control of options
– GUI must contain all adjustable options for each trial
• Versatile Trigger System
– Must be comfortable and easy to use by autistic
children
– Contain two triggers for optical flow stimuli
– Force feedback (accidental button depression, set
threshold)
 Customer Needs
• Visual/Auditory Stimuli
– Adjustable complexity settings
– Time/speed settings
– Shapes
– Optical Flow
– Images
– Audio clips w/ noise (cocktail scenario)
• Flexible video/audio outputs
– Must be viewable on projector, LCD screen, & TV
– Stereo audio for compatibility with equipment
• Synchronization with EEG signal
– Not critical & time permitting
 Background
• No known device or software comparable to this
project
• Closest find
– Brain wave-directed amusement device (created by
Brotz, Gregory R.)
– United States Patent 5213338
– Idea specifically designed as a sort of “game” but for
autistic children and children with disabilities.
– By reading the brain waves from a person, one can
create visual stimuli that affect those brain waves.
– Example: Relaxing visual stimuli, exciting visual stimuli
 Design Concepts

• Coded in Matlab (2009b)

• Simulink
• Image Acquisition Toolbox
• Image/Video Processing
Blockset
• Virtual Reality Toolbox
Hybrid Block Diagram

• Threshold feedback system for controller

– Prevent accidental triggering of system
 Software Features
• KEY: Adjustable options
• Time
• A/V Quality
• Noise type (blur, salt/pepper, cocktail scenario, etc.)
• Numerous available presets
• Threshold depression feedback for controller
• Reaction time calculation
• Delay between recognition and controller input (How?)
• EEG wave flagging (tentative)
• Trigger confirmation (pop-up dialog Y/N)
 Hardware Features
• Using readily available XBOX 360 Controller
• 2 Analog Triggers
• 4 Buttons
• D-Pad
• 2 Analog Sticks
• Force Feedback
• Interface over USB/RF-USB to Matlab program
 Hardware Features
• Will need to be interfaced via USB
• RAW data output interpreted by Matlab
 Software Methodology
• 1. Open GUI (Initialize Trial Settings)
– Panel for every adjustable program parameter
– Time, FPS, Frames, Noise Type, Noise Area, Noise
Strength, Optical Flow, Shape Selection, EEG
flagging, etc.
 Software Methodology
• 2. Generate SD/HD AVI File (Background
Process)
– Matlab will generate appropriate stimuli (A or V)
based on configured parameters.
– Generate individual frames and append to AVI file
 Software Methodology
• 3. Display stimuli
– Matlab will play newly generated custom stimulus
to the appropriately selected output device
 Software Methodology
• 4. Hardware Interaction
– Autistic child will press analog trigger(s) at which
point trial will pause (confirmation required)
– After confirming, program will calculate recognition
time and trigger depression characteristics
 Design Concepts
• Trigger System

Design Element Design 1 Design 2 Design 3

Connectivity USB USB Wireless (RF-USB)
Triggers 1 (Digital) 2 (Analog) 2 (Analog)
Buttons 2-4 2-4 2-4
D-Pad No Yes Yes
Feedback No Yes (Vibration) Yes (Vibration)
 Design Concepts
• Graphical User Interface (GUI)

Design Element Design 1 Design 2 Design 3

A/V Preview Yes Yes Yes
Control Drop Down I/D/S* I/D/S*
Custom Input No Yes Yes
Difficulty Levels 1 3 (E/M/H) 4 (E/M/H/C)
Visual Yes Yes Yes
Audio No No Yes
Interface Mode No No Yes (Easy/Advanced)
 Design Concepts
• Method of Stimuli Generation (Matlab)

Design Description
Design 1 Generate figures in real time and display sequentially (inefficient/slow)
– dependant on host PC speed/configuration.
Design 2 Generate figures in real time and create an SD/HD AVI file to play
afterwards. – Eliminates potential for slowdowns during stimuli
display.
Design 3 Use pre-made stimuli files with no real time processing/creation. –
Fast, limits custom stimuli profile ability.
• Examples of stimuli
– Blur, noise, pixelization, optical flow, shape distortion, etc.
 Evaluation of Designs
• Trigger System
Design Element Weight Design 1 Design 2 Design 3
Connectivity 5 7 7 10
Triggers 10 5 10 10
Buttons 5 10 10 10
D-Pad 2 0 10 10
Feedback 5 0 10 10
Adjustable Threshold 6 0 10 10
Cost 4 10 7 5
Totals 175 343 350
 Evaluation of Designs
• Graphical User Interface (GUI)
Design Element Weight Design 1 Design 2 Design 3
A/V Preview 7 10 10 10
Control 5 3 10 10
Custom Input 10 0 10 10
Difficulty Levels 10 3 10 10
Visual 10 10 10 10
Audio 8 0 0 10
Interface Mode 5 0 0 10
W/FS Output 10 10 10 10
Totals 315 520 650
 Evaluation of Designs
• Method of Stimuli Generation (Matlab)

Design Description
Design 1 Generate figures in real time and display sequentially (inefficient/slow)
– dependant on host PC speed/configuration.
Design 2 Generate figures in real time and create an SD/HD AVI file to play
afterwards. – Eliminates potential for slowdowns during stimuli
display.
Design 3 Use pre-made stimuli files with no real time processing/creation. –
Fast, limits custom stimuli profile ability.
• Design 2 gives the best speed and largest flexibility on
relatively modern PC
 Final Design (Overview)
• Software
– Matlab w/ Simulink, Image Acqusition Toolbox, &
Image/Video Processing Blockset
• Hardware
– XBOX 360 Controller

System Requirements:

Dual Core CPU

4GB RAM
20GB Free Space
1280x1024 Resolution
USB Connection
 Final Design (Software)
• Stimuli Generation Method
– Revised on 11/2009
• Store parameters needed for specific stimuli
• Create AVI object file
• Generate individual frames and write as JPG to
HDD
– Adjustable compression
• Load JPG’s and append to AVI in reverse order
 Final Design (Software)
• How exactly are the frames generated?
– Combination of built-in filtering and newly written
filter models based on existing mathematical
models
• Blur
– Gaussian blur; low pass filter that attenuates high
frequencies
– Adjustable matrix dimensions and sigma (how much
are the pixels altered?)
– Convolves original image with a matrix of Gaussian
values (depending on symmetry, strength, etc.)
 Final Design (Software)
• Blur (contd.)
– Two scans done (vertical and horizontal) to fully
blur image
– Eliminates aliasing (that’s why its used in
compressing/smoothing an image)
– Algorithm written in such a way that blurring is as
linear as possible
• No bias towards initial blurring stages
– IMG1 <BLUR> IMG2 :: IMG2 <BLUR> IMG3 …
Final Design (Software)

• Steepness of curves relies on sigma value

 Final Design (Software)

0.00000067 0.00002292 0.00019117 0.00038771 0.00019117 0.00002292 0.00000067

0.00002292 0.00078633 0.00655965 0.01330373 0.00655965 0.00078633 0.00002292

0.00019117 0.00655965 0.05472157 0.11098164 0.05472157 0.00655965 0.00019117

0.00038771 0.01330373 0.11098164 0.22508352 0.11098164 0.01330373 0.00038771

0.00019117 0.00655965 0.05472157 0.11098164 0.05472157 0.00655965 0.00019117

0.00002292 0.00078633 0.00655965 0.01330373 0.00655965 0.00078633 0.00002292

0.00000067 0.00002292 0.00019117 0.00038771 0.00019117 0.00002292 0.00000067

 Final Design (Software)
• Salt & Pepper
– Uses on/off pixel mapping
– Relies on random pixel number generator
– Converts 8-bit pixel value into white/black
• 000/255 on colormap; 00000000 /11111111
• Pixel can be changed to any other color as well
(red/blue/white combination for example)
– Algorithm selects more and more pixels to convert
with each frame pass resulting in more noise
being generated
Final Design (Software)
 Final Design (Software)
• Optical Flow
– Perceived pattern of apparent motion of objects
– Great for object and motion direction detection
– Utilizes vector maps of 64x64, 128x128, 256x256
or 512x512
• Can also use 4:3, or 16:9 vector maps depending on
output video resolution and aspect ratio
– Based off of motion video with large objects
• Vectors created based on acceleration of individual
pixels
 Final Design (Software)

• Image on the left: Somewhat unclear where the object is heading towards
• Image on the right: Much clearer towards which direction the object is heading
 Final Design (Software)
• Auditory
– Cocktail scenario principal
– Background noise WAV interleaved with phrase
repetition or music WAV
– Slowly decrease volume of noise until the target
audio clip is recognizable
– Allows for preset file selection as well as custom
file input
– 2 layer audio (one fixed; one variable)
 Final Design (Software)

• Multiply desired waveform by a decaying exponential

or triangle as a method of decreasing volume
 Final Design (Software)
• GUI
– Custom made interface
– “Backbone” of software package
– Supports all control parameters for stimuli
generation
– Allows for basic patient/trial tracking
• Patient name, DOB, trial identifiers, etc.
– Includes timing for image recognition
– Controls stimuli generator scripts, stimuli display,
and incorporates toolbox written for hardware
interface
 Final Design (Software)
• GUI (contd.)
– Will contain diagnostic menu
– Allow for reaction time measurement using simple
traffic light test
 Final Design (Hardware)
• Controller
– Xbox 360 Wireless Controller w/ RF-USB Receiver
• Input interpreter toolbox
– Custom made applet to register controller inputs
• Button presses
• Analog trigger position
• POV/D-Pad direction
– Registers inputs and sends them to backbone
structure for analysis/action determination
– Controller input stream stored in live matrix
 Final Design (Hardware)
• Windows Interface
– Uses default Windows 7 x64 compatible driver
– DirectX Input for Force Feedback
• Matlab Interface
– Uses pass-through from Windows driver into
Matlab
– Virtual Reality Toolbox for controller input
readings
– Create small GUI for controller diagnostics and
debugging
 Final Design (Customer Needs)
• Software
– Customizable stimuli settings via GUI
– Package includes all requested algorithms
– Allows output to virtually any video/audio device
• Hardware
– Ergonomic controller for use by children
– Adjustable threshold settings to prevent accidental
triggering
 High Level Requirements
• Architecture
– The system will run on Windows XP, Vista, and 7
operating systems (x86 and x64)
– The software package will be able to function in
both windowed and full screen modes
– The software is capable of functioning on any
system which meets the previously mentioned
minimum system requirements
– The software package will be natively coded in
Matlab (with tentative plans for C++ compile)
 High Level Requirements
• Algorithms
– The algorithm package will include blur, salt &
pepper, optical flow, and auditory stimuli
– The stimuli are all independently adjustable via
GUI
– The algorithms are all able to render frames with
an emphasis on minimizing processing time and
hard drive space
 High Level Requirements
• GUI
– The GUI will include basic patient/trial tracking
options
– The GUI will be responsible for tracking stimuli
generation time as well as patient reaction time
– The GUI will be responsible for channeling
communication between scripts and outside
functions
 Test Results (Timing)
• Accuracy of timing
– Can we rely on the built in tic/toc performance
stopwatch timer within Matlab?
– How exactly does it function?
– What are the alternatives?
• Tic/toc has some variation
– However, it is very insignificant; in the range of
about +/- 0.033 ms!
– Variation is due to the amount of CPU time spent
on fetching initial function and argument calls
 Test Results (Timing)
• Determining recognition time of child
– Typical reaction time: 200-400 ms
– Delay between button press and timer start/stop
action
• 0.020 to 0.045 ms
• The delay is insignificant compared to desired
reaction time resolution
– If it was larger, pre-fetch could be used to warm
up scripts and eliminate CPU time wasted on
function calls
 Test Results (Parallel Processing)
• What is multi core
processing?
– Spreading several
instruction sets over
multiple processing cores
within a CPU
– Instructions must not rely
on each other
(sequentially)
 Test Results (Parallel Processing)
• Test system specifications
– Core2Quad Q9400
– 8GB DDR3 1333 RAM
– Western Digital Caviar Black 7200RPM SATAII
– Windows 7 x64 Professional w/ Matlab x64 2009b
• 1 vs. 2 vs. 4 core performance
– Rendered 100 frames @ 10fps using blur script
– 4 core: 31.456 seconds (average)
– 2 core: 27.089 seconds (average)
– 1 core: 24.477 seconds (average)
 Known/Expected Issues
• All problems encountered were resolved
• Initial algorithm
– Slow due to no code optimization
– Pre-allocation of matrices and functions sped
things up considerably but at a cost of memory
– Rewritten due to poor performance and scalability
on different platforms
– New algorithms rely more on HDD performance
now, but require very little RAM to render
 Known/Expected Issues
• Have scheduled testing of controller interface
update speed for Q1 2010
– How fast can we read off changing inputs while
the video is playing back to child?
– Delay between button press and program
interpretation in live environment?
– Expected result is that the delay is insignificant
 High Level Schedule
• Project Management via “Microsoft Project
Server 2007”
– Desktop and web-based project manager
– Accessible from anywhere
– Allows for project collaboration
– Precise time/issue tracking
– Bug Tracking
– Internal e-mail system (addon)
High Level Schedule
 High Level Schedule
• Software Development
– Blur: 11/20/2009
– Salt & Pepper: 12/21/2009
– Optical Flow: 1/26/2010
– Auditory: 3/3/2010
– GUI: 3/17/2010
• Hardware Integration
– Input interpreters: 1/6/2010
– Integration w/ scripts: 3/29/2010
 High Level Schedule
• Milestones Reached
– Research & Analysis
• Customer Needs
• Three Design Concepts
• Design Evaluations
• Mid-Term Presentation
– Blur Algorithm
• Noise
• Sanity Checks
• Testing
 High Level Schedule
• Milestones Reached
– GUI
• Appearance
• Video Preview
• Initial integration with scripts
– Hardware
• Order controller and receiver
• Interface controller with Matlab
 Financials
• Hardware
– XBOX 360 Controller: $49.99 MSRP
– RF-USB Adapter: $39.99 MSRP
• Software
– Matlab 2009b: Free (NJIT IST)
– Matlab Add-ons: $399 MSRP
– Microsoft Project Server: $50/Mo
 Roles & Responsibilities
• Steffen Pharai (Software developer)
– GUI
– Stimuli algorithms
– Timing synchronization
• Alan Singh (Hardware)
– Controller interface
– Reaction time test applet
– Beta tester
• Maciej Sprycha (Software developer)
– GUI
– Stimuli algorithms
– Controller interface
 Questions?
• Open to any questions.
 References
"Autism." Google Health. Web. 22 Oct. 2009.
<https://health.google.com/health/ref/Autism>.

"Brain wave-directed amusement device - Patent 5213338." Patent Searching and Inventing
Resources. Web. 22 Oct. 2009. <http://www.freepatentsonline.com/5213338.html>.

"Image Acquisition Toolbox - MATLAB." The MathWorks - MATLAB and Simulink for Technical
Computing. Web. 24 Oct. 2009. <http://www.mathworks.com/products/imaq/>.

"Image Processing - Image and video processing, analysis, algorithm development, and
system design." The MathWorks - MATLAB and Simulink for Technical Computing. Web. 24
Oct. 2009. <http://www.mathworks.com/applications/imageprocessing/>.

“Xbox 360 Digital Controls.” Richard Burke, 22 Apr. 2009. Web. 24 Oct. 2009. <http://richard-
burke.dyndns.org/wordpress/2009/04/xbox-360-first-person-shooter-robot-part-ii/>.