Real-Time Traffic Sign Detection

using Hierarchical Distance Matching

by

Craig Northway

A thesis submitted to the

School of Information Technology and Electrical Engineering The University of Queensland

for the degree of

BACHELOR OF ENGINEERING

October 2002

ii

Statement of originality
I declare that the work presented in the thesis is, to the best of my knowledge and belief, original and my own work, except as acknowledged in the text, and that the material has not been submitted, either in whole or in part, for a degree at this or any other university.

Craig Northway

iii

iv .

ELEC3600. Shane Goodwin deserves mention for his help as the lab supervisor organising cameras and facilities. now its handed in you can contact me again. I’d like to thank all of my friends and family. To all the engineers: Hope you enjoyed your degree as much as I have! Special thanks goes to Nia. To Jenna and Ben Appleton and Simon Long for their signals related technical insights. particularly my girlfriend and best friend. gave me in this area was invaluable. Unfortunately its impossible to mention them all! From an academic perspective I must thank my supervisor Brian Lovell for his technical help when I was struggling. “Get Naked for SEES 2002!” v . thanks. The excellent background Vaughan Clarkson’s course. Vivien. Jesse and Jon. To my ”non-engineering” friends Michael. excellent view of the big picture and promotion of my work. I’ll also single out Toby. for the use of her laptop. Leon and the rest of the SEES exec. Sarah Adsett and my parents Bruce and Rosalie Northway for their support.Acknowledgments There are many people who deserve acknowledgment for the help I have received while working on this thesis and during my 16 years of study. Scott.

vi .

rotational and scale invariant matching. Other matching examples demonstrated include “Letter” matching. and remind you of the speed limits? Vehicles will soon have the ability to warn drivers of pending situations or automatically take evasive action. A prototype Matching system also built in MATLAB 3. The prototype matching system uses static images and was designed to explore the matching algorithm. There are four deliverables for the thesis: 1. vii . 4. Due to the visual nature of existing infrastructure. signs and line markings. A hierarchy creation system built in MATLAB 2. A real-time application in Visual C++ using the DirectShow SDK and IPL Image Processing and OpenCV libraries. Matching of up to 20 frames per second using a 30+ leaf hierarchy was achieved in the real-time with a few false matches. Examples of other uses for the matching algorithm. The hierarchy creation system is based on simple graph theory and creates small (< 50) hierarchies of templates.Abstract Smart Cars that avoid pedestrians. Future work on this thesis would include the development of a final verification stage to eliminate the false matches. increasing the robustness and applications of the algorithm. This thesis will develop a real-time traffic sign detection algorithm using hierarchical distance matching. image processing will play a large part in these systems. Refactoring of the systems design would also allow for larger hierarchies to be created and searched.

viii

Contents

Statement of originality

iii

Acknowledgments

v

Abstract

vii

1 Introduction

1

2 Topic 2.1 2.2 Extensions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Deliverables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

3 4 4

3 Assumptions

5

4 Specification 4.1 4.2 4.3 4.4 MATLAB Hierarchy Creation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . MATLAB Matching Prototype . . . . . . . . . . . . . . . . . . . . . . . . . . . . Real-Time Implementation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Smart Vehicle System . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

7 7 8 8 8

ix

x

CONTENTS

5 Literature Review 5.1 5.2 Historical Work: Chamfer Matching . . . . . . . . . . . . . . . . . . . . . . . . . Current Matching Algorithms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.2.1 5.2.2 5.3 Current Hierarchical Distance Matching Applications . . . . . . . . . . .

11 11 12 12 14 15 16 17 17

Other Possible Techniques . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Hierarchies and Trees 5.3.1 5.3.2 5.3.3

Graph Theoretic Approach . . . . . . . . . . . . . . . . . . . . . . . . . . Nearest Neighbour . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Colour Information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

6 Theory 6.1 6.2 Chamfer Matching . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Feature Extraction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.2.1 6.3 6.4 Edge detection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

19 19 20 20 22 23 25 26 26 27 28 28

Distance Transform . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Distance Matching . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.4.1 6.4.2 6.4.3 6.4.4 Reverse Matching . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Oriented Edge Matching . . . . . . . . . . . . . . . . . . . . . . . . . . . . Coarse/Fine Search . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Hierarchy Search . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

6.5

Tree/Hierarchy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.5.1 Graph Theory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

1 8. . . . . . . . .5. . . . . . . . . . . . . . . . . . . . . . . . . . . .2 6. . . . . . . . . . . . . Masking Reverse Search . . . . . . . Directional Matching . . . . . . . . . . . . . . Hierarchy Optimisation . . . . . . . . . .2 Image Acquisition . . . . . . . . . . . . . .3 8. . . Hierarchy Creation . . . . . . . . . . . . . . . . .2 8. . . . . . . .3. . . . . . . . .1 8. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Final Implementation . . . . . . Pyramid Search . . . . . Final Implementation . . . .2. . . . . . . . . . . . . . . . . .2. . . . . . . . . . . Rejected Refinements . . . . 37 37 38 40 41 42 43 44 45 46 46 47 48 49 52 55 55 Group Creation . .3. .3 MATLAB Prototype Matching . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .3. . . . . . . . Multi-Level Hierarchy . . . . . . 8. . . . . . . . . . . . . . . . . . . . . . . . .6 Basic System . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .setup. . . . . . . . . . . . . . . . . . . . 8. . . .1. . . . . . . . . . . . . . .2 8. . . . . . . . . . .3. . .2.6 Finding groups . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .5 8. . . . . . .1 8. . . . . . . . . . . . . . . . . . . . . . . . . 7 Hardware Design and Implementation 7. . . 8. . . . . . . . . . . . . . . . . . . .3. 8. . . . . . . . . . . . . . . . . . . . . . . . . . . .m . .3. . . . . . . Score Calculation . . . . . . . . .m . .4 8. . . . . . .4 8.1 Camera . . . . 35 36 8 Software Design and Implementation 8. . . . . . . . . .2. .createtemps. . . . .CONTENTS xi 6. . . . . . . . . . . .3 8. . . . 29 33 Programming . .6 Trees .5 8. . . . . . . . . . . . . . . . . . . . . . . .2. . . . . . . . . .2. .1 Hierarchy Creation . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . Results . . . . . . .5 Performance . . . . . . . . . . . . . . . . .2 Hierarchies .5 8. . . . . . . . . . . . . . . . . . . . . . . . . . .4 8. . . . . . Strengths/Weaknesses . . .4. . . . . . . . . . . . . . . . . . . . . . . . . . . . .1. . . . . . . .6 8. . . . . . . . . . . . . .1 Matlab Matching Results . . . . . 63 63 63 65 65 65 66 66 67 67 67 68 68 68 Matlab Matching . . . . .4 Real-Time . . . . . . . . . . . . .2 8. . . . . . . . . . . . . . .1 9. . . . .7 Matching Process . . . . . . . . . . . . . 9. . . . . . . . . . . . . .1 9. . .3. . . . . . . . . . Object Oriented Design . . . . . . . . . . .3 9. . . . . . . . . . .4. . . . . . . . .2 9. . . . . 9. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9. 9. . . Enhancements/Refinements . . . . . . . . . . . .1 8. . . .3 8. . . . . . .4 My Performance . . . . . . . . . . . . . . . . . . . . . . . . . . Actual Design . . . . . . . . . . . . 9. . . . . . . . .1 Hierarchy Creation . . .4. . . . . . . . . . . . . . . . .3. . Further Information . . . . . . . . . . . . 8. . . . . . . . . . .3. . . . . . .4. . . . .2. . . . .3. . .4. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Size Variant Matching . . . . . . . . . .4. . Rotational Matching . . . . . . 9. . . . . . .xii CONTENTS 8. . .2 Skills Learnt . . . Letter Matching . . Final Matching Algorithm Used. . . . . . . . . . . . . Further Examples . . . .4. . . . . . . . . . . . . . . . . . .4.3 Real-Time Matching . . . . . . . . . . . . . . . . . . . . . . . . . . . 56 56 56 58 59 59 60 60 9 Results 9. . . . . . . . . . .4 9. . . . . . . . . . . . . . . . . . .4. . . . . . . . . . . . .1 9. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .3. . . .

. . . . .1. . . . . .3 Better OO Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .5 Optimisation . . . . . . A. . . . . . . . . . . . . . . . . . . . . . . A. . . . 79 79 79 79 79 80 80 80 81 81 82 . . . . . . . . . . . . . . . . . . 10. . . . . . . . . . . . . . . . .4 Improved Hierarchy Generation . . . . . . . . . . 10. . .2 Lighting . . . . . . . . . . . . . . . . . . . . . . . . . . . .1. . . . . . . . .6 Final Verification Stage . . .3 Position .1. . . . .4 Angle . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . A. . . . . . . . . . . . . . . . . . . . 69 69 70 70 70 70 70 11 Conclusions 71 12 Publication 12. . . . . . A. . . . . . . . .2 Temporal Information . .8 Objects . .CONTENTS xiii 10 Future Development 10. . .5 Damage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10. . . . . . . . . . . . .1 Video Footage . . . . . . . . . . . . . . . . . . . . .1 Australia’s Innovators Of The Future . . A. . . . . . .1. . . . . . . . . . . . . . . . . . . .2 Programming . . . . . . . .1 Assumptions . . . . . . .1. . . . . . . . . . . . . . . . . . .1 Speed . . . . . . . . . . . . . . . . . . . . . . . . .7 Computer Vision Functions . . . . . A. . . . . . . . . . . . .1. . . . . . A. . . . . . . . . . . . . . . . . . . . . . . . . . . 73 73 A A. . . A. . . . . . . . . . . . . . . . . . . . .6 Size . . . . . . . . . . . . . . . . . . . . . . .1. . . . . . . . . . . . A. . . . . . . . . .1. . . . . . . . . . . . . . . . . . 10. . . . . . . . . . . . . . . .

111 . . . . . . . . . . . . . . . . . . . . . . . . . . A. . . . . . . . .2 Circular Signs Scores . . . . . . . . . A. . . . . . . . . . . . . .5. . . . . . . . . . . . . . . . . . . A. . . . . . . . . . . . . .9 Hierarchy Results . . . . . A. . . . . . . . . . . . .8 Tested Enhancements/Refinements to Real-Time System . . . .4. . . . . . . . . . . . . . .3 IPL Image Processing Library . . . . .2 Different Feature Extractions . .7.4 Template Format . . . . . . . . . . . . . . . . . . A. . . . . .2 Direct Show . . . . . . . . . .2 Deallocation . . . . . . . . . .7. . . . . . . . . . A. . . . . . . . . . . . . . . . A. . 82 82 82 82 84 87 87 87 89 89 91 91 92 95 95 95 96 97 98 A. . . . . . . . . . . . . . . . .9. . . A. . . . . . . .2.7. . . .4 Prototype Orientation Code . . . . .9. . . .2. . . . . . . . . . . . . . . . . .5. . . . . . . . . . . . . . . . .3 mytree . . . . . .6 UML of Real-Time System . . . . . . . . 102 A. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . A. . . . .3 Sub-Sampling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . A. . . . . . . . . . . . . . . . .5 Rejected Prototype Implementations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .2. .xiv CONTENTS A. . 102 A. . A.1 Diamond Signs . A. . . . . . . . . . . . .2 Orientation Map . . . . . . .1 Orientated Edge Transform . . . . .1 Distance Transform . . . . . . . . . . A. . . . . . . . . . . . . . . . . . . . . . A. . . . . . . . . . . . . . . . . . . . . . . A. . . . . . . . . . . . . . . . . . . .5. . . . . . .1 MATLAB . . . . . . . . . . . . . . . . . . . . .7. . . . .7 Code Details of Real-Time System . . . . . . . . .4 Open CV . . . . . . . . . . .3 Extra Hierarchy Implementation Flowcharts . . . . A. . . .2. . . . . .4. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . A. . . . . . . . . . . . . . .1 Localised Tresholding . . . . . . . . . . . . . A. . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . .11 CD . .10 Matlab Matching Results . . 117 . . . . . . .12 Code . . . . . . . . . . . . . . . . . . . . . . . . . . . .CONTENTS xv A. . . . . . . . . . . . . . 115 A. . . . . . . . . . . . . . . . . . . . . . . . . . . 115 A. . . . . . . .12. . . . . . . . . . . . .1 Listing . . .12. . 112 A. . . . . . . . . 115 A. . . . . .2 Compilation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

xvi CONTENTS .

. . . .1 System Output . 14 16 6. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .7 6. . . . . . . . . .9 Canny Edge Detection Process . . . . . . . . . . .1 5. . . . . . . . . . . . . . . . . . . . . . . . . . . . .6 6. . . 9 5. . . . . . . Original Image . . 21 23 23 24 24 24 25 25 27 xvii . . . . 3-4 Distance Transform (not divided by 3) . . . . . . . . . . Template . . . . . . . . . . . . . . . . Distance Image . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Matching Techniques [4] . . . . . . . . . . . . . . . . . . . . . . . . . . . Search Expansion . . . . . . . . . . . . Overlaying of Edge Image [3] . . . . . . . . . . . . . . .2 6. . .1 Likely Sign Position . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 3. . . . . . . . . . . . . . . . . . . .2 Binary Target Hierarchy [1] . . . . . . . . . . . . . . . . . . . . . . . .1 6. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .4 6.8 6. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Image Clustering with Graph Theory [2] . . . . . . . . . . . . . . . . . . . . . . . .3 6. . . . . . . . . . . . . . . .1 Real Time Traffic Sign System . . . . . . . . . 6 4. Template Distance Transform . . . . . .List of Figures 1. . . . . . . . . .5 6. . . . .

. . . . 64 66 67 . . . . . . . .1 Block Diagram from GraphEdit . . 9. . . . 36 8. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .14 My Size Variant Hierarchy . . . . . . . . . . 8. . 50 Sign 60 Sign . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .11 Tree . . . . . . . . 6. . . .6 8. . . . . . . . . . . . . . . . . . . . . .4 8. . . . . . . . . Reverse Matching Mask . . . . . . . . .1 9. . . . . . . . . . . . . . . . .10 Pyramid Search . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .2 8. . . . .1 8. . . . . . . . . . . . .3 Circular Sign Hierarchy . . . . . . . . . . . . . . . . . . .7 8. . . . . . . . . . . . . . . . . Simple Matching System . . . . combinegroups. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .13 Intended Class Diagram . . . . . .12 Breadth First Search . . . . . . . . . . .8 8. . . . . . . . . . 8.5 8. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .11 Oriented Edge Detection .2 9. . . . . . . . . . . . . . . . . . . . . . . . . .10 Simple Graph [2] . . . 6. . . .xviii LIST OF FIGURES 6. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8. . . . 28 30 32 7. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Hierarchy Creation Flowchart . . . . . . . . . . . . . . . . . . . . . .9 Hierarchy Creation Block Diagram . . . . . . . . . . . . . . . . . . . . . . Noise Behind Sign . . . . . .m Flowchart . . . . . . . . . . . . . . . . . . . . . . 38 39 40 41 44 45 47 48 49 50 52 53 57 61 8. Group Creation Block Diagram . . . . . . . . . . .3 8. . . My Chamfer Transform . . . . . . . . . . . . . . . .12 Orientation Map . . 8. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Image Acquisition Flowchart . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . .2 findbestnotin. . . . . 103 A. . 105 A. . . . . . . . . . .21 Fourth Group Template . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .m Flowchart . . . . . . . . . . . A. . . . . . . . . A. . . . . . . . . . A. . . . . . . . . . . . . . . . . . . . . . . . . . . . 104 A. . A. . . .20 Fourth Group . . . . . . . . 102 A. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .m Flowchart . . . . . . . . . . . A. . . . . . . . . . . . .8 Actual Class Diagram . . . . .19 Third Group. . . . . . . 105 A. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .18 Second Group. . . . . . . . . . . . . . . . . . . A. . . . . . . . . . . . .10 Spiral Search Pattern . . . .5 Simple Image . . . . . . . . . . . . . . . . .11 Straight Search Pattern . .6 Localised Thesholding . . . . . . A. . A. . .15 Optimised Scores . . . . A. . . . . . . . . . . . . .4 remove. . . . . . . . . . . . . . . template = self . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .9 Actual Sequence Diagram . . . . . .m Flowchart . . . . . . . . . . . . . . . . . .13 Truncated Distance Transform . . . . . . . . . . . . . . . 106 . . . . . . . . . . . . . . . . . . . . . . . . . . . . .17 First Group Template . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105 A. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .22 Fifth Group . . . . . . . . . . . . . . A. . . . . . . . . . . . 80 84 85 86 89 90 92 93 94 98 98 A. 100 A. . . . . . . . . . . . . . . . . . . . . . .LIST OF FIGURES xix A. . . . . .7 Intended Sequence Diagram . . . . . .16 First Group . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .14 Original Scores . . . . . . .3 anneal. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 100 A. 104 A. . .1 Multi-Resolution Hierarchy [4] . . . . . . . . . template = self . . . . . . . . 104 A. . . . . . . . . . . . . . . . . . .12 Untruncated Distance Transform . . . . . . . . . . . . . . . .

. . . . 106 A. . 112 A. . . . . 107 A. . . . . . . .27 Seventh Group Template . . . . . 110 A. . . . . . . . . . . . . . . . . . . . 106 A. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 110 A. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114 . . . . . . . . .41 Closer View of scores . . . . . 107 A. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .30 First Template Group . . . . . . . . . . . . . . . . . . . . . . . . . . . 113 A. . . . . . . . . . . . . . . . . . . 109 A. . . . . . . . . . . . . . . . .36 Second Level Optimisation . . . . . . . . . . .31 First Template Group Combinational Template . . . . .33 Second Template Group Combinational Template . . . . . . 108 A. . . . . . . . . . . . .39 Distance Transform Image . . . . . . . . . . . . . . . .32 Second Template Group . . . . . . . .38 Oriented Edge Image . . . .42 Match . . . .29 Eight Group Template . . . . . . . .37 Original Image . . . 110 A. . . . . . . . . . . . . . . . . . . . .34 Last Template Group . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .25 Sixth Group Template . . . . . . . . . . . 107 A. . . .23 Fifth Group Template . . . . . .40 Scores . . . . . .26 Seventh Group . . . . . .35 Last Template Group Combinational Template . . . . . . . . . . . . . . 107 A. . . . . . . . . . . . . . . . . . . 112 A. . . . . . . . . . . . . . . . . . . . . . . . . . . 111 A. . 110 A. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114 A. . 114 A. . . . .xx LIST OF FIGURES A.24 Sixth Group . . . . . . . . . . . . . . . . . . . 109 A. . . . . . . . . . . . . . . . . . . . . . . . . .28 Eigth Group . . . . . . . . . . . . . . . . . . . . . .

. 116 xxi . . . . . . . . . . . . . . . . . . . . .3 Real-Time . . . . . . .List of Tables 8. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54 A. 115 A. . . . . . . . . . . .2 MATLAB Matching . .1 Directional Scoring . . . . . . . . . . . . . . . . 116 A. . . . . . . . . . . . . . . . .1 Hierarchy . . . .

xxii LIST OF TABLES .

A Traffic Sign Recognition system has the potential to reduce the road toll. The system will be useful for autonomous vehicles and smart cars.Chapter 1 Introduction This thesis will develop a real-time traffic sign matching application.1: System Output 1 . There also exists the possibility for computer control of vehicles and prompting for pedestrians and hazardous road situations. After testing the matching on traffic signs. By “highlighting” signs and recording signs that have been past. other implementations shall further demonstrate the effectiveness of the algorithm. Figure 1. the system would help keep the driver aware of the traffic situation.

Some systems already developed by vehicle manufacturers include a night vision system which Cadillac have introduced into their “Deville” vehicles. INTRODUCTION If reliable smart vehicle systems can be established on PC platforms upgrading and producing cars as smart vehicles would be cheap and practical. tools. It uses a vision system to detect pedestrians and traffic signs. possibly due to driver error or fatigue. cyclists. vehicle identification and mobile robots. etc. This system projects an image of the road with obstacles highlighted onto the windscreen. Examples of these include pedestrians. These recognition cases could be used in applications such as autonomous vehicles. The European Union are heavily sponsoring research into this technology through a smart vehicle initiative with a view to decreasing the road toll.2 CHAPTER 1. . motorcyclists. If the goals of the project can be met the developed application (C++) and associated utilities (MATLABT M ) will form a general solution for hierarchy creation and implementation. car models. Daimler Chrysler have produced a prototype autonomous vehicle capable of handling many and varied traffic scenarios. This thesis will help establish a working knowledge of such systems and demonstrate the simplicity of algorithm development on a PC platform. Hierarchical Distance Matching could be applied to a range of other object detection problems. Mercedes-Benz’ Freightliner’s Lane Departure System uses a video camera to monitor lane changes alerting the driver to lane changes without the use of indicators. known local landmarks. text of known font. military targets.

The topic for this thesis is Real-time Traffic Sign Matching. Other methods used for traffic sign detection have included colour detection[6. 9]. Another factor contributing to the speed of the algorithm is the coarse/fine and hierarchical nature allowing significant speed-ups (without sacrificing accuracy) when compared to exhaustive matching. 5] are superior to other approaches for implementation on a general purpose PC platform. 7]. simulated annealing[10] and neural networks [11]. It can be mathematical shown [4.Chapter 2 Topic From the background research into shape based object recognition it was obvious that Gavrila’s Chamfer methods [4. multiplication) on the data set which is efficient when computed in this manner. Gavrila’s success defined the topic and prompted further research into hierarchies and distance matching. Thus video footage can be searched for N objects simultaneously without the extensive calculations necessary for an exhaustive search in real-time on a general purpose platform. None of these have been able to produce an accurate real-time system. This thesis intends to prove the hypothesis that multiple object detection. using Hierarchical Distance (Chamfer) Matching. can be successful in real-time using a hierarchy of images. It involves repeated simple operations (such as addition. Gavrila’s success is due to the simplicity of his algorithm and its suitability to standard computation and the SIMD instructions. 3 . colour then shape [8. 5] that this pyramid style search will not miss a match. such as traffic signs.

2.2 Deliverables Based on these goals a MATLABT M based Hierarchy creation implementation. alpha-numeric characters. the independent development and implementation of Hierarchical Chamfer Matching (HCM) and the evaluation of HCM as a method for object detection. The development of this algorithm will allow hierarchies other than the initially intended Traffic Sign’s to be used. a prototype static matching implementation in MATLABT M and a real-time HCM Object Detection implementation will be delivered. 5] and thus the original contribution presented in this thesis will be the automated hierarchy creation. TOPIC 2. e.1 Extensions The extensions to previous work [4. car models (from outline/badge).4 CHAPTER 2.g. hand gestures. . pedestrians.

1 5 . The following assumptions exist: • Camera should be of a high enough quality to resolve signs at speed. • Due to the size invariance of the method the sign should pass through the specific size(s) without being obscured. • Lighting must be such that the camera can produce a reasonable image. • Angle of the signs in relation to the car’s positions should not be extreme. These assumptions must be reasonable for the thesis to be successful. • Signs should be positioned consistently in the footage. Many of these assumptions are for the specific task of traffic sign detection. • Objects being detected must be similar in shape for the hierarchy to be effective. Further Details of these are in Appendix A.Chapter 3 Assumptions Before commencement of this project some of the assumptions were identified. • The Computer Vision functions should operate as specified. • Signs should not be damaged.

6 CHAPTER 3. ASSUMPTIONS Figure 3.1: Likely Sign Position .

The following specifies clearly the input and output of each deliverable. 2.1 MATLAB Hierarchy Creation The hierarchy creation system should be able to synthesise an image hierarchy without user input into the classification. This system should work on image databases of reasonable. This implementation will be the problem of traffic sign recognition. A brief specification of what a Smart Vehicle System may do is included.Chapter 4 Specification The goals of this project are: 1. 4. Demonstrate the algorithm on other matching problems. 7 . This system will initially be based in MATLAB. 4. Program an implementation of this object detection in C++ using the Single-Instruction Multiple Data (SIMD) instructions created for the Intel range of processors. A prototype matching system for static images also built in MATLAB 3. Establish an automated method of hierarchy creation that can be generalised to any database of objects with similar features.

1).3 Real-Time Implementation This Real-Time Implementation should match objects at over 5 frames per second in reasonable circumstances. Output A hierarchy of images and combinational templates. It will be written in Visual C++ based upon the DirectShow streaming media architecture. developed using the EZRGB24 example (from Microsoft DirectShow SDK A. Output The image overlayed with matches. 4.2 MATLAB Matching Prototype This system should match traffic signs/objects on still images accurately. This unit would attach to the car either at manufacture or by “retro-fitting”. It .8 CHAPTER 4. SPECIFICATION Input A directory of images (which share similarity). It is not required to meet any time constraints Input An image hierarchy. and a threshold for the similarity. Input Image hierarchy and video stream. The image processing operations will be performed by the IPL Image Processing and Open CV Libraries. 4.2).4 Smart Vehicle System A smart vehicle system for driver aid would be a self contained unit. Output Video Stream overlayed with matches. 4. and image to be matched. shown in the block diagram (Figure 4.

. the system may be able to control the car.1: Real Time Traffic Sign System block). People should be wary of the systems ability.4. eg. etc.. if the car is in a 100km zone. With the use of radar and other visual clues. The system must be careful not to lure the driver into a false sense of security. snow. The system may have higher intelligence allowing it to tailor the hierarchy or matching chances to the situation. . a 30km speed sign would be unlikely.4. or a heads-up display (output block). such as storms. allowing the driver to check their speed between signs. It would be able to keep track of the current speed limit. particularly in extreme situations. The system would recognise all common warning and speed signs (real-time detection Figure 4. SMART VEHICLE SYSTEM 9 would provide the driver with details via either verbal comments. This would avoid possible collisions and keep within the speed limits.

10 CHAPTER 4. SPECIFICATION .

along with several mathematical texts to understand the concepts. That is minimising the generalised distance between two sets of edge points. let alone real-time video. The first major work on chamfer matching was the 1977 paper “Parametric Correspondence and chamfer matching: Two new techniques for image matching” by H. This work discussed the general concept of chamfer matching. justifying the choice of Hierarchical Based Chamfer Matching. These papers form the basis of current matching techniques. research into state of the art traffic sign and shape based object detection applications is reviewed. Current research into image classification and grouping for search and retrieval is therefore also applicable to this topic. all relevant to the project. 11 . The algorithm was investigated again throughout the mid 90’s when implementations on specific hardware became practical. Basic works on trees and graph theory were examined briefly. The topic was revisited in the late 80’s by Gunilla Borgefors [3]. Secondly. This is well before HCMA systems would have been practical for fast static matching.1 Historical Work: Chamfer Matching Background work on HCMA (Hierarchical Chamfer Matching Algorithm) was started in the late 70’s. It was initially an algorithm only suitable to fine-matching. 5. an authority on Distance Matching and Transforms.G. Firstly several historically significant papers are reviewed. Barrow et al.Chapter 5 Literature Review The review of background material for this thesis will cover several topics.

as with all distance matching techniques. The systems for both traffic signs and pedestrian detection are based on . The most successful work presented in this area is from Dairu Gavrila and associates at Daimler Chrysler using HCMA.” [3]. Tools can be recognised based solely on the outline in this situation hence are perfect for HCM. The mid-nineties saw several uses of distance transforms as matching algorithms. the 3-4 DT. This solved the major problem of the first proposal. Borgefors also proposed the use of the technique for aerial image registration.2 Current Matching Algorithms Current approaches to shape based real-time object detection. This was on static images.2. This was in 1993. requires the image to be overlayed over each template to score each match. The paper demonstrated the algorithms object detecting effectiveness on images of tools on a plain background. its limitation to fine matching. even surprisingly good. it should be possible in well under half a second. The conclusions reached were that the results were “good. then in 2002 with only hardware improvements. this is a computationally expensive operation. 5. Orientated Pixel Matching and Neural Networks. This algorithm used a distance transform proposed by its author. Considerable work done on Hausdorff matching by Huttenlocher and Rucklidge in [13. assuming Moore’s Law holds. LITERATURE REVIEW Borgefors [3] extended this early work to present the idea of using a coarse/fine resolution search. Thus the HCMA is an excellent tool for edge matching. This idea was later presented in [12].12 CHAPTER 5. as long as it is used for matching task with in capability(sic). 14] showed that the Hausdorff distance could be used as a matching metric between two edge images.1 Current Hierarchical Distance Matching Applications Daimler-Chrysler Autonomous Vehicle Hierarchical Chamfer Matching (HCM) is currently being used in automated vehicle systems at Daimler-Chrysler. These include Hierarchical Chamfer Matching. Their best results required at least 20 seconds to compute on a binary image of 360x240 pixels. 5. Hausdorff matching.

e. The hierarchy creation in this system is not fully automated. The optimisation is done with simulated annealing.will not miss a solution” in their hierarchies of resolution/template. Their matching technique employs “oriented edge” pixels. As suggested in these papers this matching technique is similar to Hausdorff distance methods. labelling each edge pixel with a direction.2.5. pedestrian outlines. The overall technique proposed by Diamler-Chrysler shows excellent results and is worthy of further development. Worst case measurements of matching templates are then considered to determine minimum thresholds that “assure [the algorithm]. They go onto prove that distance transforms provide a smoother similarity measure than correlations which “enables the use of various efficient search algorithms to lock onto the correct solution”. Their experiments show that the traffic sign detection could be run at 10-15 HZ and the pedestrian detection at 1-5 Hz on a dual processor 450MHz Pentium system. Chamfer measures. Their method of creating the hierarchy automatically uses a “bottom-up approach and applies a “K-means”like algorithm at each level” where K is the desired partition size. This creates a binary search tree (Figure 5. though not employed in the matching (Hausdorff matching) are used to cluster the edge maps into groups of two. Target Recognition An Automatic Target recognition system developed by Olson and Huttenlocher [1] uses a hierarchical search tree. A new template for each set of pairs is generated and the clustering continued until all templates belong to a single hierarchy. Translated.1). The hierarchy creation presented is a simple approach that may present good results. scale and rotation invariant template based matching for a deformable contour i. This approach may be unique. The disadvantage of their early approaches was in the one-level tree created. They have designed algorithms using the SIMD instruction sets provided by Intel for their MMX architecture. . There is no mention of the real-time performance of the oriented edge pixel algorithm. These efficient algorithms are coarse-fine searches and multiple template hierarchies. The most surprising result of this work is the success of rigid. It may not be quick enough for traffic sign recognition. rotated and scaled views are incorporated into a hierarchy. CURRENT MATCHING ALGORITHMS 13 HCM. The clustering is achieved by trying various combinations of templates from a random starting point and minimising the maximum distance between the templates in a group and their chosen prototype.

If the score was less then the rejection value (max − (max × percent 100 )) the pixel was expanded to more positions in the next pyramid level for matching. except the distance measure cannot be pre-processed.1: Binary Target Hierarchy [1] Planar Image Mosaicing Hierarchical Chamfer Matching has been used successfully for Planar-Image Mosaicing. 14. Dhanaraks and Covaisaruch [12] used HCMA to “find the best matching position from edge feature (sic) in multi resolution pyramid”.14 CHAPTER 5. LITERATURE REVIEW Figure 5. 15] for object detection. 5.2 Other Possible Techniques Hausdorff Matching Many researchers have considered Hausdorff matching [13. They chose one image to be the “distance” image and another to be the “polygon” image. It is a valid approach for this application and will be considered as a possible matching strategy. This is an interesting thresholding concept based on the maximum values rather than absolute. . The interesting concept used in this work was the thresholding for taking a match to the next level.2. The resolution pyramids are built and matching is carried out by translating the polygon image across the distance image. It is a similar algorithm to chamfer matching.

HIERARCHIES AND TREES 15 Neural Networks Work by Daniel Rogahn [11] and papers such as [16] are example of neural network techniques for traffic sign recognition. Image classification techniques have been examined in multimedia retrieval systems [20. 19. Some traffic sign detection algorithms use it as a cue [6. 2.3 Hierarchies and Trees The results shown by [4. 7. It is an excellent technique for situations where the colours are constant and illumination can be controlled. 10.5.3. many unrelated areas of ground and trees were also highlighted. 9. Their features were not perceptible accurately from colour data alone. 5. I do not have the necessary background knowledge to explore this properly. Previous work by myself on traffic sign recognition has attempted to incorporate colour data into the matching process. it is difficult to define exact colours for matching. 19]. Further research was therefore carried out into tree structures and image grouping and classification. 13. Yellow diamond warning signs were quiet easily detectable as present in an image. 11. It was impossible to determine if this circle was red or brown. 26] into real time object identification. Thus identification of signs was not plausible from colour detection alone. 21. Hierarchies and trees have also been investigated . 8. It may be more effective in streamed uncompressed video. 5] have been far superior to other research [6. Due to most colour representation schemes not being perceptively true. 22]. Signs such as those indicating speed limits have a thin red circle surrounding the details. 9]. 7. The overhead of detecting the colour (even with a look up table) and the varying illumination made it difficult in this real-time scenario. though it may still be a useful procedure for masking areas of interest. Colour detection Colour data has been used for matching in scenarios such as face detection [18. 8. By including it’s colour in the detection. My previous results have shown that on compressed video this red circle is destroyed by artifacts.

The algorithm they proposed considers retrieving groups of images which not only match the template. but are also similar to each other.” A graph is constructed of this set (each template is represented as a node) with the edges representing the distance measures between each template (Figure 5. The technique demonstrated in [2] was considered a simple and effective starting point for hierarchy creation in this thesis. They “query the database and get back the best N matches. The measure of inter-cluster similarity is established to determine which cluster should be returned. Connected clusters that include the original query image are then found.1 Graph Theoretic Approach Selim Askoy [2] used distance measures to obtain similarities between the images.2: Image Clustering with Graph Theory [2] [23. The hierarchy creation was then looked upon as a “graph clustering problem”. 5.3. . With the increasing electronic availability of large amounts of multimedia material high speed retrieval systems (such as trees) have been the subject of significant research. Define S as the set containing the original query image and the images that are retrieved as the results of the above queries. S will contain N 2 + 1 images in the worst case. 24].16 CHAPTER 5. This approach sounds similar to that used in [4. This has application to object recognition hierarchies.2). The clustering algorithm used is presented in the paper. For each of those N matches we do a query and get back the best N matches again. LITERATURE REVIEW Figure 5. 5] in assuring that each grouping was the closest.

of clusters. 5. or similarity measure threshold) is reached.5. reducing the number of unmerged clusters. The N images are placed into distinct clusters using their similarity measure. after the tree has been created a cluster centre is established.3. This continues until a bounding parameter (no. Various methods such as linear regression and boolean features can be used for this.2 Nearest Neighbour Huang et al [23] used trees established by the nearest neighbour algorithm and built using normalised cuts (partitions of a weighted graph that minimise the dissociation with other groups and maximises the association within the group) in a recursive nature. They select a representative image of the cluster rather than compute a new composite image (first proposed in [23]). with a potentially useful clustering technique. The similarity of all the clusters is then computed again. Their method of construction also allows trees to be created with uneven distances to leaves.3. This might provide a speed-up in matching where some images in the hierarchy are relatively unique. HIERARCHIES AND TREES 17 5. This technique proved effective in the paper. with the added complexity of many leaves.3. This allows the simplicity of a binary tree. but was too complicated to pursue in an undergraduate thesis on image matching. “Two clusters are picked such that their similarity measure is the largest” these then form a new cluster. .3 Colour Information To group images [20] uses colour information. This creates a tree that can at most have two clusters branching off a parent cluster. providing a short and certain path to them. yet each leaf cluster could contain more than two images. To represent each cluster.

18 CHAPTER 5. LITERATURE REVIEW .

This is followed by the details of the graph theory and hierarchy basics necessary to understand and develop this work. but there are still many basic theories and methods that are accepted as the “way” of doing things. Score the template at “all” locations 19 . The steps required are: 1. 6. If this distance measure is below or above a certain threshold it signifies a match. but those relating to the hierarchical search are relatively new to the image processing field. Some elements of the HCM algorithm use this type of method. Lastly some programming libraries that may not be familiar to all electrical engineers are mentioned. Relevant image processing theories and techniques are explained first.Chapter 6 Theory The theory behind this thesis is split into 3 main sections. Feature Extraction 2.1 Chamfer Matching The basic idea of Chamfer (or Distance) Matching is to measure the distance between the features of an image and a template. Image processing is a relatively dynamic field where many problems are yet to have optimal solutions. Distance Transform 3.

before going onto explain the theory of Gavrila’s HCMA. . those that use first order derivatives and second order derivatives. By matching with prototypes first a significant speed-up can be observed compared to an exhaustive search for each template. a hierarchical approach can be used. Images can be grouped into a tree and represented by prototype templates that combine their similar features. and areas of constant intensity will be ignored. These features are usually corners and edges. The following section describes the theory behind each of the steps in simple Distance Matching. Determine whether the scores indicate a match In Gavrila’s Hierarchical Chamfer Matching Algorithm (HCMA). There are algorithms that vary from simple to complex.2. 6. THEORY 4. 6. Ideally all edges of objects and changes in colour should be represented by a single line.1 Edge detection The goal of edge detection is to produce a “line drawing”. To find changes in intensity we need to examine the difference between adjacent points. The boundary of an object is generally a change in image intensity. Of the edge detectors that use gradient approximation there are two types. Using a first order gradient approximation changes in intensity will be highlighted. Standard edge and corner detection algorithms such as Sobel filtering and Canny edge detection can be applied to colour/gray images to generate binary feature maps. distance matching is applied to the scenario of matching multiple objects.20 CHAPTER 6.2 Feature Extraction Shape based objection recognition starts with feature extraction representations of images. The generalised form of edge detection is gradient approximation and thresholding. When trying to match a set of images with sufficient similarity.

which should be maximum at the peak of the edge data where the gradient of the original image is sharpest.1: Canny Edge Detection Process . Optimal detection with no spurious responses 2. The third aim relates to locating single edge points in response to a change in brightness. Figure 6. It is used in a wide range of applications with successful results. 1986) is currently the most popular technique for image processing. Canny demonstrated that Gaussian filtering was optimal for his criteria. Non-maximum suppression (peak detection) is used for this. The second aim is for accuracy. This requires getting a first derivative normal to the edge.1. Calculating this normal is usually considered too difficult and the actual implementation of the edge detection is as follows in figure 6.2. It was formulated with 3 objectives: 1. Good localisation with minimal distance between detected and true edge position 3. FEATURE EXTRACTION 21 Canny Edge Detector The Canny Edge detector (Canny. It retains all the maximum pixels in a ridge of data resulting in a thin line of edge points. Single response to eliminate multiple response to a single edge The first aim was reached by optimal smoothing.6.

j+1 + 4. vi. 6. 12. on each pass. vi−1.j+1 + 4. 3-4 transforms and other more complicated 4 3 4 3 1 approximations.j−1 + 3. Set all feature pixels to zero and others to “infinity” before the first pass. The points either side of it on the edge are established with the direction information. THEORY Non maximal suppression This essentially locates the highest points in edge magnitude. The value of the distance transform increases as the distance is further from a feature pixel in the original image. These include 1-2. Some papers [3. The diagonals are represented by 4 3 4 3.22 CHAPTER 6. A 3-4 transform uses the following distance operator: 1 0 1 This matrix 4 3 1 shows why the transform is named as such.3 Distance Transform Distance transforms are applied to binary feature images. set it k−1 k−1 k−1 k−1 k−1 k−1 k as the following value: vi. The real Euclidean distance to pixels is too expensive to calculate and for most applications an estimate can be used. 25] have gone on to prove that approximations were sufficient 3 for the purposes of distance matching. Given a 3x3 region a point is considered maximum if its value is greater than those either side of it. vi+1. vi+1. vi+1. The following images show a feature image and its corresponding distance transform.j−1 + 4.j+1 + 4) Complete sufficient passes (k represents the pass number) .j+1 + k−1 k−1 k−1 3.j + 3. Then for each pixel. Each pixel is labelled with a number to represent its distance from the nearest feature pixel. If there is an adjacent pixel labelled as 1 a lower threshold must be met to set a pixel to 1. vi. vi. Hysteresis Thresholding Hysteresis thresholding allows pixels near edge to be considered as edges for a lower threshold.j + 3. a high threshold must be met to set a pixel to 1. A simple way to calculate a distance transform is to iterate over a feature image using to distance operator to find the minimum distance value for each pixel. vi−1.j . and adjacent distances 3 . such as those resulting from edge detection. If no adjacent pixels are 1 (edge).j = min(vi−1.

T represents the number of features in T and dI (t) is the distance between the template and image feature.6. I) = 1 |T | t∈T dI (t) [5] where T and I are the features of the template and Image respectively. One simply scores each position by overlaying the edge data of the template as shown in figure 6. This gives a matching score. 6. DISTANCE MATCHING 23 0 0 1 0 1 0 0 0 0 1 0 0 0 0 0 1 0 0 0 0 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Feature Image 6 3 0 3 0 3 8 6 3 0 3 6 7 4 3 0 3 6 6 3 0 0 3 6 7 4 3 3 4 7 8 7 6 6 7 8 Figure 6.3: Overlaying of Edge Image [3] calculated with Dchamf er (T.4.3.4 Distance Matching Distance matching of a single template on an image is a simple process after a feature extraction and distance transform. Her two pass algorithm [25] is popular and is used by the Open CV library. By completing this process for every image position in the region of interest .2: 3-4 Distance Transform (not divided by 3) Distance Transform until you have calculated the maximum distance that is necessary for implementation of the matching or other algorithm you intend to use. the better the match. The mean distance of the edge pixels to template is then Figure 6. Borgefors is responsible for much of the early work on distance transforms. The lower the score. More complicated faster methods exist.

6) illustrate how an incorrect match could occur due to these circumstances. The template figure 6.6. If any of these scores fall below the matching threshold.24 CHAPTER 6. if the template points are a subset of the image points it may score as highly as an exact match. Figure 6. where the distance transform of the image. THEORY a score is generated for each location. Thus the forward distance matching confirms the presence of template features in the image.4 . If the template is missing features that are present in the image ie.4: Original Image Figure 6.5: Distance Image Figure 6.6: Template . The template can be considered found. but doesn’t confirm the presence of image features in the template. is correlated against the feature extraction of the template.6 is a sub-set of the image and fits the distance transform. The following example (figures 6. There is one problem with the simple “forward” distance matching.

7 demonstrates the relationship between a forward match (Feature Template to DT Image) and a reverse match (DT Template to Feature Image). When the Figure 6. This should be eliminated by forward matching with a “sensible” template.1 Reverse Matching A reverse match is often used to preclude these false matches. Figure 6.8: Template Distance Transform template of the cross is overlayed on this distance transform (Figure 6. DISTANCE MATCHING 25 6.4.8) the score will be high.6. The only problem is if there are not sufficient pixels in the edge image. When we combine the forward and reverse match we can use the resulting score to reject or accept matches.7: Matching Techniques [4] we can see that its reverse matching score will be significantly lower.4. If we revisit the example that caused errors in forward matching Figure 6. .

Thus each template edge point is assigned to one of the M templates. Gavrila et al [5] used a similar technique to increase matching accuracy of chamfer matching.4. Huttenlocher [1] and Johnson [26] have published papers describing the use of oriented edge matching in an image hierarchy. mx − ix . are useful in shape based matching. The individual distance measures for each M type can be combined later. Each pixel now has a distance from the nearest feature pixel and an orientation distance from the nearest feature pixel. and similar techniques. Their formula for calculating the Hausdorff distance took this extra orientation parameter and normalised it to be comparable with the location distance ))) [26. THEORY 6. thus M templates and M feature images.2 Oriented Edge Matching Oriented Edge Matching. the scaling of the template can .3 Coarse/Fine Search A Coarse/Fine Search. When using edge points the orientation can be binned into M segments of the unit circle. It has the same general form as their definition of a Hausdorff measure. mo the orientation and similar measures. Templates are then matched with this extra parameter.26 CHAPTER 6. Generally a coarse fine search involves decreasing the “steps” of the template search over the image if matching scores dictate. mo − io| 6. Though the calculation of the score for each position in a pyramid search requires less computational expense (less pixels). 1] Where: m is the α my − iy template. or pyramid resolution search is a popular method for increasing the speed of a search based image recognition technique. By splitting the features detected from the extraction into types and matching them separately. This orientation match generalised Hausdorff Matching to oriented pixels. Conversely a pyramid resolution search scales (smaller) the image and template. They further clarify that features in the template are present in the image. I) = maxm∈M (mini∈I (max( for ix . Gavrila suggests having M feature types. the “chance” that you are measure the distance between the same features of the image and template increase. mx my represent the x and y coordinates of that pixel. Hα (M. iy .4. io of the image. Oriented Edge Matching can evaluate to a distance measure between orientations. increasing the scale if the scores are sufficient. therefore can be substituted into a Hausdorff matching algorithm.

Figure 6. In a distance based search the smooth results (compared to feature to feature matching). as in HCM.4. at a particular search step. and the furthest possible matching location.4. the details of the signs are quiet fine. the current threshold. Hence reducing their size can cause these details to be destroyed. In this search they use a depth first tree search. In a matching scenario such as traffic signs. To not miss this possibility the threshold must be set according to: Tσ = θ − 2 ∗ ( σ )2 . Thus 2 HCM has the excellent property that in a coarse fine search a “match” cannot be missed. DISTANCE MATCHING 27 Figure 6. Thus the threshold for this point of the search is now Tpσ = θ − 2 ∗ ( σ )2 − worstchild. if the score is below a threshold. and the threshold defining a match is θ. can be set such that a match “cannot” be missed.9: Search Expansion create difficulties. At each point the image is searched with prototype template p. Where worst child = 2 .4 Hierarchy Search The approach proposed by Gavrila [4. To ensure that Tpσ will not reject any possible matches two factors must now be taken into account: the distance between the location of the score. 6. the search is expanded at that point with the children nodes being scored. Then when using a distance measure. Tσ . The thresholds can once again be set using a mathematical equation to ensure that templates are not missed. mean that a reasonable match at a coarse search level might indicate an “exact” match at a finer level. and the distance between the prototype template and its children. 5] is to combine a coarse/fine search with a hierarchical search. In this scenario a number of resolution levels is covered concurrently with the levels of the search tree. Tpσ . If the current resolution of the search is σ.9 shows the furthest the actual location (the cross) can be from the search (squares).6.

The vision most people have of graphs is a diagrammatic representation such as figure 6.10: Simple Graph [2] vertices and edges respectively.10. THEORY maxti of C Dp (T. . Only the applicable properties will be discussed here. and a list of unordered pairs of these elements. are said to be adjacent if they are joined by edge. .28 CHAPTER 6. where C is the set of children of prototype p C = t1 . I). Tree’s are a specific type of graph fulfilling certain mathematical properties. Once again a match cannot be missed. . 6. . but would obviously . u and v.5 Tree/Hierarchy The hypothesis of this thesis is to prove that creating a hierarchy of templates will allow the matching process described above to be carried out in real-time on multiple objects. which are Figure 6. 6.1 Graph Theory “A graph consists of a non-empty set of elements. Adjacency: Vertices. u and v are said to be incident with e and correspondingly e is incident with u and v. called vertices. Graphs come in many different forms and have numerous properties and definitions associated with them. This is useful for small and simple graphs.” [27] This statement defines a graph. e.5. called edges. Where points are joined by lines. tc .

with n vertices labelled 1. TREE/HIERARCHY 29 be confusing for larger representations.3.11) are connected graphs which contain no cycles.n. The significance of trees has increased in recent years due to modern computers. Multimedia and Internet based storage and search research is at the “cutting edge” of tree development.2 Trees Trees (Figure 6. These systems are required to store large amounts of data and search them very quickly. called a weight” [27]. “A complete graph is a graph in which every two distinct vertices are joined by exactly on edge. This tree will then be searched using feature information extracted from an image. In this definition a directed graph refers to a set of vertices with edges that infer adjacency in only one direction..” [27] A dissimilarity matrix is and adjacency matrix of a weighted directed graph. A weighted graph by definition is “a graph to each edge of which has been assigned a positive number.2. Each edge is weighted with the similarity in that direction. One type of these are called . An adjacency matrix is defined as such: “Let G be a graph without loops.5.5. This form is more suitable to mathematical and computational manipulation. Let G be a weighted. Processing graphs in a computer in this form is also generally inappropriate.”[27] 6. This thesis will create a tree using traffic sign templates based on their feature similarity.2. Trees were first used in a modern mathematical context by Kirchoff during his work on electrical networks during the 1840’s. directed graph without loops. Another necessary definition ia a complete graph.6. Some tree properties can be used to construct trees from graphs. with n vertices labelled 1.3n. they were revisited during work on chemical models in the 1870’s [28]. Increasingly tree structures are being used to store and organise data. A dissimilarity matrix is the n x n matrix in which the entry in row i and column j is a measure of the dissimilarity between vertices i and j. It is possible to take each vertex and list those that are adjacent to it in the column or row of a matrix. The adjacency matrix M(G) is the n x n matrix in which the entry in row i and column j is the number of edges joining the vertices i and j.

An easier approach is to build the tree with a bottomup approach.e. i. This allows trees to be created easily. 10 is greater than 9.e. Features The features of a tree are usually the attributes used to split the tree. image . features. Values are constant in relation to each other. because the templates to represent higher levels in the tree are not yet established. therefore also greater than 7. Data such as image templates which are not ordered. image 3 matches image 5 well. These are not applicable in this application. the size of the desired tree and tree quality measures. combining the templates at each level. Multivariate trees require combinational features to be evaluated at each node. creating all combinations of these and finding spanning graphs would be computationally too expensive. From here the tree is constructed by moving up levels. the features are obviously the value of the number.30 CHAPTER 6. or the lowest level of the tree which will have only one edge connected to them.11: Tree Finding Splits When building a tree it is necessary to split the data at each node. i. for instance 9 is greater than 7. THEORY minimum spanning trees. Figure 6. Trees can be multivariate or univariate. Finding a split amounts to determining attributes that are “useful” and creating a decision rule based on these [29]. A bottom-up approach to growing a tree starts with the leaves. These are the criteria for finding splits. ordered. When creating a tree the “programmer/user” needs to determine several variables/concepts before commencing. In a simple tree of integers. There are systematic methods for finding spanning trees from graphs.

especially considering cases where the sample size can affect necessary thresholds. Simulated Annealing The optimisation technique used by Gavrila [4.6. but deeper trees can be more accurate (note: that is a very general statement). These include restrictions on node size. Multi-stage searches are perhaps beyond the scope of this thesis. resulting in a more efficient search because less paths are tested. 5] to optimise hierarchies (Maximise the tree quality) was simulated annealing. The name originates . TREE/HIERARCHY 31 7 matches image 5 doesn’t imply that image 7 matches image 3 more or less. Shallow trees can be computationally efficient. are more difficult to place into trees. This will often be application dependant. classification of test cases and testing cost [29]. A single threshold will not necessarily be possible in most situations. This should ensure that the images within a group are similar and groups are dissimilar. multi-stage searches and thresholds on impurity [29]. Thresholds on impurity allow only groups/spilts to exist that are above or below a certain value when the splitting criterion is used. Features used to find splits and create an image tree are in this thesis likely to be distance measures between images. A simple method proposed by Gavrila [4. It is a process of stochastic optimisation.5. Size of Trees Obtaining trees of the correct size can be a complex issue. Restrictions on node-size allow the “user” to control the maximum size of a node. The effect will be to decrease the threshold used to determine whether to expand a search. There are many options for deciding the quality of a tree. Tree Quality measures Tree quality could depend on size. 5] for a distance matching image tree was to minimise the distance between images of the same group and maximising the distance between different groups. optimisation of splitting criteria. Some techniques for obtaining correctly sized trees exist.

12: Breadth First Search . It allows the search to jump out of local minimum by allowing “backwards” steps. Figure 6. where if the “backwards” change is not too expensive given the current “temperature” it will be accepted.32 CHAPTER 6. A BFS visits all the vertices adjacent to a node before going onto another one. A DFS works “down” the tree checking each path to the leaves before moving across. hence would not require this list of locations. A depth first search can be seen as working down the levels before going across. The cooling process and the search algorithm are iterative procedures controlled by a decreasing parameter. Every node on the current level must be searched before we can move onto the next level. two are depth first search (DFS) and breadth first search (BFS. This works on an exponential decay like temperature. figure 6. Simulated Annealing was also used by [10] to recognise objects. THEORY from the process of slowly cooling molecules to form a perfect crystal. The next level below must be searched before the search can move horizontally to the next template on the same level.12). Gavrila [4. “temperature” or if no better combination is possible. 5] used a depth first search which requires a list of node locations to visit. Searching Trees There are several well-known search methods for trees. They differ in their direction of search. Searches with simulated annealing can be stopped based on search length. A BFS checks across the tree first. A good way to visualise a breadth first search this is laying the nodes out onto horizontal levels.

Algorithms and data structures are important.6. the IPL Image Processing and Open CV libraries is also necessary. PROGRAMMING 33 6. Specific knowledge of MATLAB. as are concepts of Object Orientated programming.2. Details of these are included in Appendix A.6.6 Programming This thesis requires a good knowledge of programming concepts and topics. .

THEORY .34 CHAPTER 6.

• Video streaming from a USB/Fire-wire device Where the video has been pre-recorded the only hardware required is the computer. Asynchronous File Source and WDM Streaming Capture Device respectively for the two example sources above. This proves the value of the IPL and Open CV libraries used. Suitable devices were already available in the laboratory 35 .Chapter 7 Hardware Design and Implementation The hardware for this thesis should be simple and “off the shelf”. Where it is being streamed the camera must be plugged into a port/card on the computer. Showing that these libraries allow image processing on a general purpose platform. Microsoft DirectShow. providing this platform is of a comparatively good standard. The two practical medias are: • Video recorded on a digital camera and written to an MPG/AVI. allows to filter that is built to work on any streaming media source. This will allow Directshow to access the video with a suitable object. No design decisions were required for the hardware.

36 CHAPTER 7. It would also be fitted with a telephoto lens to allow high resolution at a fair distance from the sign. Due to the camera being pointed down the road the automatic focus would often blur the traffic sign. The focus would be fixed to the expected distance of sign detection.1: Block Diagram from GraphEdit 7. This thesis used a standard digital camcorder. which are difficult to adjust “on the fly”. A purpose built camera would be made to adjust automatically when set at high shutter speeds.1 Camera In a commercial application a purpose built camera would be used. or adjusted to focus on the region of interest. HARDWARE DESIGN AND IMPLEMENTATION Figure 7. Several problems are evident with standard camcorders. forcing manual settings. . The automatic settings do not cater for high shutter speeds necessary.

Briefly. 3 and 4 vertices. It is then annealed until further optimisation is not possible. The description explains inputs and outputs to most major functions. the algorithm involves grouping the images into complete graphs of 2.1 Hierarchy Creation The method of hierarchy creation is based on the graph theoretical approach outlined in [2] and the traffic sign specific application in [5]. The process tests several orders and optimises hierarchy solutions for each of these. This seems the logical feature in a hierarchy for 37 . The appendix A. and shows the procedural design of the functions. Each complete graph forms a group which can be added to a hierarchy. 8. It is a bottom-up approach and can be applied recursively with varying thresholds to generate a multilevel approach. The hierarchy is constructed taking groups in an arbitrary order based on weightings of group similarities. The best hierarchy is chosen by the best optimised scores. Optimisation is defined as minimising intragroup scores and maximising intergroup scores. The features considered in this tree are the image similarities and dissimilarities. The technique outlined produces a single level hierarchy. These help to find splits based on thresholding these values.12 is simply a listing of directories. The similarities and dissimilarities are based on distance matching scores between templates.Chapter 8 Software Design and Implementation All the code for this thesis is included on the CD attached to this document and not as an appendix. files and their contents. describes the abstract data types.

The images used for hierarchy creation were taken from websites of sign distributors. SOFTWARE DESIGN AND IMPLEMENTATION distance matching.) This allowed quality pictures of signs to be included. resulting in the following implementation. The list is then iterated again zero padding any smaller images to the maximum size found in the last iteration and adding them all into a three dimensional vector. Before the process commenced. The size of the tree has been limited only by restricting node size to lesser the complication of application.e.jpg.38 CHAPTER 8.1. etc) if so it is loaded and the size tested. 8.1 represents the process.2 is the design for the process used. all the diamond signs were made to be the same size. checking if the extension is an image (. Figure 8. i. . similar sign types were resized.1 Image Acquisition The flowchart in figure 8.bmp. The block diagram in figure 8.1: Hierarchy Creation Block Diagram Refinements were made to the exact methods of each sub-process during construction. . Matlab provides a simple files command to retrieve a list of files from a directory.m. The list of files is iterated through. Images were then acquired from a directory with a Matlab script. This is to find the maximum image size in the directory. (For other matching applications the images can be generated appropriately. The distance transform of each image can be calculated using chamfer. This was the initial design.

Entry (i. vi+1. Dissimilarity Matrix The dissimilarity matrix was calculated using average chamfer distance Dchamf er (T. vi+1.j+1 +3. values are approximated for corner pixels. vi. T represents the number of features in T and dI (t) is the distance between the template and image feature.2: Image Acquisition Flowchart Chamfer. and all non-feature pixels to an effective “infinity” or maximum value greater than the maximum distance to be iterated too. The algorithm iterates over the k image a certain number of times.8. This is a very inefficient. vi.j+1 +4. vi−1.j +3.j) in the dissimilarity matrix represents the distance measure between template i and image j. vi.j +3.1. vi−1.j+1 +4. vi+1.j . Firstly all feature pixels are set to 0.j−1 +4. (Both being templates from the database) This .m The chamfer routine written for template distance transforms was inefficient but simple. but simple calculation of the “chamfer” or distance transform (Figure 8. I) = 1 |T | t∈T dI (t) [5] where T and I are the features of the template and image respectively.3).j+1 +4) After this is complete. HIERARCHY CREATION 39 Figure 8. The time taken for off-line distance transforms is not related to the speed of the matching. each time labelling each pixel with the result of: vi.j−1 +3.j = k−1 k−1 k−1 k−1 k−1 k−1 k−1 k−1 k−1 min(vi−1.

chamdata) From this point on in the software each image is referred to by its position in the images struct. 8.4.2 Group Creation A Design Diagram is shown in figure 8. . SOFTWARE DESIGN AND IMPLEMENTATION Figure 8. The adjacency matrix was formed by thresholding the dissimilarity matrix.3: My Chamfer Transform initialisation script is called createTree. All values below the maximum distance are set to 1 indicating the images are similar (adjacent given this threshold). This was effectively setting a threshold on impurity to control the properties of the tree. The graph of images was represented by an adjacency matrix. Groups were created by finding complete graphs within the set of images. These positions were allocated by the order files were retrieved from MATLAB’s files structure.m Inputs: • (optional) directory Output: • dissimilarity matrix • Images (MATLAB structure with fields edgedata.40 CHAPTER 8.

8.setup. This also requires the adjacency matrix to be cubed. instead of the unthresholded dissimilarity matrix to create these groups we ensure any similarities are of sufficient quality.m. The pairs can be used to find the third image. The pairs found are used to find complete graphs of triplets. Effectively all the closed walks of length 2.m The MATLAB script for finding the groups.4: Group Creation Block Diagram Using the properties of adjacency matrices complete graphs of pairs can be found from the diagonal of the adjacency matrix squared. setup.1 Finding groups . By using the adjacency matrix.2. Then in the same way triplets are used to find the quads. The adjacency matrix can be searched to find the product terms contributing to the entries. is specified as follows: Input: • Dissimilarity Matrix • Images (structure with fields edgedata and chamdata) • THRES a threshold of the average chamfer distance . Once again the diagonal shows if a triplet is present. 3 and 4 through the connected sub-graphs of the set of images have been found.2. GROUP CREATION 41 Figure 8.8.

createtemps. The worst (maximum) score of the templates against combinational template. The minimum score is taken to be the intragroup score because it represents the best match.m The createmps script.2. as already mentioned.m. SOFTWARE DESIGN AND IMPLEMENTATION Output: • Imagestruct (structure with many fields representing groups and their intergroup and intragroup scores) • Adjsquare (the square of the adjacency matrix) 8. The combinational templates are formed using the createtemps script by creating a distance transform that is the mean of the images’ distance transforms. The scores of similarity and dissimilarity are calculated using distance matching because this is the method of matching to be used. Intergroup scores are calculated by comparing a combinational template to all the images not included in that template. Intragroup scores are found by testing all the templates in a group against their combinational template distance transform. • Images . Output: . is the intragroup score.the structure containing all the image data.42 CHAPTER 8. hence make the best match (minimum) as bad as possible. The arguments are as follows: Input: • Image Number for “root” image. Thus each group is scored by how badly it matches its template. • Vector of Image Numbers for all in group.2 Score Calculation . The goal is to reduce the intergroup similarity. creates a combinational template. Createtemps.m The scores referred to in the previous list are calculated by createtemps.

Firstly. Once this has been completed all images with no possible groups. intergroup and intragroup scores. It should ensure that combinational templates are subsets (or close to) of the templates.2).the template intergscore The combinational template is later thresholded to create the feature template for this possible tree node. 8. iterate through the order to find the highest scoring match that will fit into the hierarchy for each image not already included (findbestnotin.m. Once the hierarchy is finished the groups scores and added together to form a hierarchy score. GROUP CREATION 43 • temps .5 shows the procedural design of the script. The flowchart (Figure 8. The templates are not saved at this time simply the scores. and templates recreated later. The features being used to create and optimise the hierarchy are the group size.the structure containing the template data (distance and edge) and scores • Tempscore . one to represent the intragroup scores.8.m. This process reveals the common features of the template. If all templates for each group were saved the memory necessary would start to become ridiculously large. Hierarchy Creation is performed by the MATLAB script createhier. The scores are stored. Inputs: • allpairs (vector containing all pairs already in hierarchy) • Imagestruct (same as before) • Order (arbitrary order of construction) . or those that haven’t already been included are added as single images.3 Hierarchy Creation An arbitrary order is used to guide the initial selection of the groups.2. These scores are a measure of hierarchy quality. and one for the intergroup scores.2. The hierarchy has two scores. Thus as explained when discussing “reverse” matching they match as well as the actual templates. a recursive implementation see figure A. It is morphologically thinned to ensure that all lines are of single width.

To avoid local minima the optimisation is allowed to take one step backwards. SOFTWARE DESIGN AND IMPLEMENTATION Figure 8. if partially built) • NOIMAGES (number of images) Outputs: • hierarchy (structure with groups. otherwise the annealing process is finished.5: Hierarchy Creation Flowchart • Hierarchy (the existing hierarchy. 5]. If the resulting hierarchy has a better score.2. The quality of the trees is measured by the similarity of images within a group (intragroup scores) and their dissimilarity to other groups (intergroup scores). scores and intergroup scores) • Scorevect (the cumulative totals for inter and intra group scores) 8. A group is not allowed to be removed if it has no pairs. to a higher score. it is kept and the annealing process is continued. it has already been removed or the last step was backwards. . “Backwards” steps are not dependant on a “temperature” factor.44 CHAPTER 8.4 Hierarchy Optimisation The hierarchies are optimised with a simple method similar to the simulated annealing used by Gavrila [4. if the next score is then lower than the previous best the change is accepted. In this case it has been greatly simplified due to limited understanding of the mathematical concepts and time restraints.

A script temps2images. The anneal function calls the remove function each pass. 8.m Flowchart the ”best” hierarchy the templates are regenerated. . The optimisation is attempted for a variety of orders.8. as only template scores were saved the first time they were created.5 Multi-Level Hierarchy To create a multi-level hierarchy the same functions are applied to the templates resulting from the combination of the leaf level images.m was programmed to automate the creation of multilevel hierarchies.2. For Figure 8. This is all represented by the flowcharts in figures 8.6.4 and A. A.2. This is done for multiple arbitrary orders to show the effectiveness of the optimisation.m. Alternatively if all the template images are written to files they can be accessed with createTree to form another hierarchy.6: combinegroups. The hierarchy is created for each order using createhier. it is then optimised with the anneal function. GROUP CREATION 45 This optimisation takes place in the combinegroups.m script.3.

2. This template could then be selected to test varying combinations of template and image. This led to the notion of masking the reverse search to avoid non-sign details affecting the match. Run combinegroups to combine and optimise these groups into a hierarchy. 4.2. Combinegroups will show you each group. This helped to refine several techniques and test possible approaches to speeding up the matching process. Use setup1 to create the groups that are used to optimise the hierarchy. 8. or there are too many images.46 CHAPTER 8. and output as files the images of the combinational templates. 3. Place the image files into the same directory. in an easy development environment. 5. The final MATLAB matching system was different than the eventual Visual C++ real-time system. SOFTWARE DESIGN AND IMPLEMENTATION 8. but was an excellent learning experience. The basic chamfer matching algorithm was implemented using simple forward and reverse matching. even on static images. Repeat this on the combinational templates for the next hierarchy level and so on. you must recursively apply it to each level of the hierarchy. .3 MATLAB Prototype Matching A prototype matching system was created in MATLAB to help understand and refine the algorithm. They will be named based on the number of “root” of each group. involving both forward and reverse matching. Note: The algorithm is written recursively to make it easy (not efficient). To use this bottom-up hierarchy creation. The first approach taken was a simple distance match of one template to images. which means it will fail if there are too many groups.6 Final Implementation The final implementation has been submitted with this thesis. 1. Run the createtree script (edited to use that directory) to get the images and dissimilarity matrix into the workspace. Which happens if your threshold is too low. It was always destined to be slow and unusable.

7: Simple Matching System Effective matching was stifled greatly by trees. This was an attempt to reduce detail in areas of trees. MATLAB PROTOTYPE MATCHING 47 Figure 8.3)of the edge detection also attempted to remove the tree data.7): . so this approach was discontinued. Sub-sampling (A.5.1 Basic System The design of the basic fine/coarse single template distance matching system in MATLAB is as follows (Figure 8.5. Localised thresholding (A. sub-sampling of the edge detection and oriented edge detection. Further methods tried to improve the matching included Localised thresholding. The thesis was not meant use colour information.2) were also tried for different levels of the search. Oriented edge detection did not remove the trees as possible matches but increased their “random” appearance when compared to the well directed outline of traffic signs. but at a fine level helped reduce false matching.1)used simple statistical methods.3.3.5.8. 8. They still caused unnecessary expansion of the search. Different thresholds (A. due to previous work proving it to be unreliable. Simple colour detection and subsequent masking of the edge detection image by the colour information was tried. first encountered during ELEC4600 to increase the threshold in areas with high edge content.

Following the theory relating to coarse fine distance matching the threshold is reduced as a function of the step. SOFTWARE DESIGN AND IMPLEMENTATION R10simplepyroverlay.48 CHAPTER 8. For example if a sign is surrounded by Trees the edge detection may look as in figure 8. By masking the edge detection with figure 8. and gave some false matches. named expand. If the step is one a reverse and forward score is computed for both locations. The reverse search is only appropriate for areas included within the boundaries of a sign. As shown it takes a recursive approach to searching each of the starting locations.8: Noise Behind Sign shape of the matrix will cause the tree edge detection to inflate the reverse score. like trees. The search still expanded unnecessarily on areas of noise. Ensuring the only features considered are those of the sign. For each search position based on the step a forward score is calculated. Setting the region of interest to the square Figure 8. else the search is terminated. If there were sufficient pixels in the edge image to indicate a “sensible” reverse match and the product of the forward and reverse threshold a match is considered to be found. by recurring with a smaller step. After the initialisation of variables the iterative for loop steps through each of the starting positions separated by 8 pixels vertically and horizontally. The region of interest is changed to include solely inside the boundaries of the sign. which are iterated over.8.3.7. This matched individual templates well. 8.2 Masking Reverse Search This simple search was improved by masking the reverse search.m This file implements the design in figure 8. . If these conditions are not met the search is terminated without a match being found.9. This location is passed to a recursive loop. not the background. which searches this sub-area. the search is expanded further on this location. If this forward score is below a threshold.

This implementation is much more complicated than the simple search as the hierarchy must be searched concurrently with the coarse/fine matching.3.10) was used to search each group for a match: . MATLAB PROTOTYPE MATCHING 49 Figure 8.3 Pyramid Search This pyramid search used the hierarchy object created by the MATLAB script described in the previous section.3.9: Reverse Matching Mask 8.8. The following design (Figure 8.

50 CHAPTER 8.10: Pyramid Search . SOFTWARE DESIGN AND IMPLEMENTATION Figure 8.

The edge pixels for each direction are placed in another matrix (three dimensional) directionmatrix. The planned implementation was to modify the existing canny edge detection algorithm in MATLAB to produce a binned orientation map. MATLAB PROTOTYPE MATCHING 51 It was implemented in pyroverlay.m and was a simple iteration through each member of the group. Shown in A. and in the other it is more vertical. finding the maximums for each.3. By binning the values during this calculation the output from the canny edge detector could be scaled with different magnitudes representing orientations. and each of the quadrants for the gradient vector fall into two cases. O----O----O (2) (3) (From MATLAB image processing toolbox edge function) The edge function iterates over the directions. Work on the pyramid search was very brief due to the poor results of the simple one template fine coarse search. Another function simply called each group.11).1 is the section of code changed. There are eight divisions.4.8. Directions are binned based on the following diagram: % % % % % % % % % % 3 2 The X marks the pixel in question. so the expectations were high. . Oriented Edge Detection Oriented edge detection has been used by other researchers in matching problems. including hierarchical searches. I can then output a matrix with edges directionally coded into the magnitude (Figure 8. The canny edge detector already estimates the direction of edges for use in the non-maximal suppression. divided by the 45 degree line. Refinements were need to the matching design to improve accuracy and precision. In one case the gradient O----0----0 4 | | O | (1)| X | 1 | O | |(4) vector is more horizontal. but for the non-maximum supression we are only worried about 4 of them since we use symmetric points about the center pixel.

vi+1. The oriented distance matching was implemented for single image coarse fine matching. but confirmation of matches used the forward.j−1 + 3.j+1 + 4. but as this was a prototype implementation designed to test the algorithm it was not attempted.m.j+1 + 4)) Results of this for the image shown in figure 8.j + 3. reverse and orientation matching scores.j−1 + k−1 k−1 k−1 k−1 k−1 k−1 k−1 k−1 4. This was too expensive to perform on the entire image.52 CHAPTER 8.4 Directional Matching To match the oriented edge detection to the template requires a “oriented edge map” of the template.11: Oriented Edge Detection 8. This rejected almost all of the false matches.12. vi.4. so it was only implemented for forward matching against the distance transform of the template. vi.3.j . Directionchamfer.m was the script to perform this function. vi−1. a position then inherited the direction of the pixel that its minimum distance was calculated from. By extending the distance transform to produce a matrix labelling every position with the direction of the closest template feature pixel allows a comparison of the distance between image and template pixels and a “distance” between their direction. vi−1.j = dir(min(vi−1. The script written was simplepyrdirectedoverly. vi. vi+1.j+1 + 4.j + 3. Positions were expanded based on the forward matching as before.2 was iterated over the template image.11 are shown in figure 8. allowing the results presented in my thesis seminar. vi+1. SOFTWARE DESIGN AND IMPLEMENTATION Figure 8. . The code in k−1 A. after calculating the result of the minimum distance (split into positions which have 4 added to them and 3). A more efficient distance transform method may have been possible. Equating the following: diri.j+1 + 3.

3. This results in the following scores: .12: Orientation Map The orientation matching score was calculated using the following formula: For every pixel in the directed edge image subtract the value of the corresponding pixel in the direction map then Mod that with three. MATLAB PROTOTYPE MATCHING 53 Figure 8.8.

1: Directional Scoring . Orientation Map 1 2 3 4 1 2 3 4 Matching Score 0 1 2 1 1 0 1 2 Table 8. SOFTWARE DESIGN AND IMPLEMENTATION Edge Score 1 1 1 1 2 2 2 2 etc.54 CHAPTER 8...

5 Rejected Refinements Some rejected refinements to the system are present in Appendix A. The work was always intended to aid in understanding of the Algorithm. 8.3.3.6 Final Implementation A final implementation for the MATLAB prototype Matching was not delivered. .5. The work was left “unfinished” and implementation of the real-time system was started. MATLAB PROTOTYPE MATCHING 55 8.8.3.

2 Object Oriented Design The system was designed in an object oriented environment.4. SOFTWARE DESIGN AND IMPLEMENTATION 8.1 Matching Process Figure 8. based on the prototyping. to be implemented was as figure 8.4 Real-Time The design for the real-time implementation would reflect the properties of the matching algorithm discovered in the prototype implementation.4. Reverse matching cannot be used until the leaf level of the matching process because combinational templates may not include every feature of the leaf template they represent. The basic forward and reverse matching design would be implemented first. Hence Object Oriented design concepts were used. They only contain the common features of their leaves. These intended designs are the ideal situation where the search is handled within the template classes. The class diagram (Figure 8. EZrgb24 . Based on this score the position is expanded to include sub-positions.10 shows the initial procedural design for the matching algorithm simplified to the main processes.13) shows the main classes and their main features important to the matching algorithm. This includes the edge image.13) was established prior to implementation. The following initial design (Figure 8. 8. If the leaf level of the tree is reached. If the matching reaches the minimum step the hierarchy search is enacted.56 CHAPTER 8. The procedural diagram of the matching algorithm. distance image and the output image. a reverse match is calculated confirming the presence. It is executed on each frame. If additional accuracy proved necessary this could be expanded to include orientation matching. Each template root is forward scored against positions. and is where the edge detection and distance transform will be effected. expanding on the best match above the threshold. 8. The transform method is part of the original example code. The EZrgb24 class contains the image data. This will search the children of each root. An abstract builder pattern is used to create the trees.10.

This design is simplified due to my limited knowledge of UML. and can use the same interface. Their are methods for EZrgb24 to access the data.7) shows the flow of control between process and objects. The builders are invoked to create the tree of templates.4. The EZrgb24 object is responsible for creating the IplImage objects to represent the images. It has subclasses that are concrete builders. It allows other classes access to the hierarchy through the array of root templates. It . Sequence Diagram The sequence diagram figure (A.8. Each template contains its distance and edge data and a pointer to its array of children. It has methods to score and search through the image hierarchy. The mytree class allows the polymorphism to be used. Any of the concrete builders can be implemented at run-time.13: Intended Class Diagram will also write the output to the stream. The constructor creates the mytemplatev objects in the appropriate hierarchy. REAL-TIME 57 Figure 8. The mytree class is an abstract builder. The mytemplatev class contains and operate on the template data. actual implementations.

This (not shown in diagram) allocates the images with the calculated edge and distance transforms. The IplImage objects require careful maintenance to prevent memory leaks. The diagram also shows more of the .) 8. The template class then takes care of forward and reverse scoring appropriately through the hierarchy (not shown in detail). For each frame the transform method is executed. Obviously at the completion of each frame the transform filter will be run again. it could have been added later. The basics of tree building are kept the same in this design. Ideally in object oriented design the analysis phase should ensure that the classes are minimal and do not implement too much functionality. The classes have also become too big. even as a reference to a static attribute.8 of the actual implementation shows that the EZrgb24 class has responsibility for most of the scoring and searching methods. but memory deallocation cannot be properly controlled without causing errors. Due to the limited nature of reverse scoring this memory leak is allowed to continue to demonstrate the intended design. Separating the searches simplified the programming task. It was much simpler to keep all references and use of this data within the transform method and not “pass” it to the template objects. SOFTWARE DESIGN AND IMPLEMENTATION also creates a mytree object and instantiates it with whichever tree is necessary for the matching task. Reverse scoring is still run in this method.58 CHAPTER 8. Scores and details of the found image are returned to the Ezrgb24 object to be written to the output. (Not Shown in diagram. As can be seen.4. The class diagram A. This design allows the template details to be completely hidden from the EZrgb24 object by encapsulation within the mytemplatev object. until a match is found. the threshold must be constantly modified by step and template to combination template parameters. This change in design was necessary during the programming of the thesis. If this were included as indicated in the theory. For each position to be search. This builder class creates the template hierarchy.3 Actual Design By reverse engineering the code actually written I am able to present the actual design implemented. The mytemplatev class has almost become an abstract data type. If the results were poor. transform runs the hierarchy search. this design does not merge the coarse/fine search into the hierarchy search.

8. This may be expanded to the hierarchical search based on the scores. Firstly a coarse/fine search is executed with the root templates at each position.4 Further Information Appendix A. The function is greatly simplified because the transform method knows which template matches (because it generated the scores). The output is written similar to before. private variables. but without showing all the private methods needed. and control resides mainly within the transform filter. Sequence Diagram The sequence diagram A.8 . It can be easily seen it is overcomplicated.4. REAL-TIME 59 implementation details ie.9 reflects these changes made to the class diagram. This forward searches the root templates until the step is one.7 explains in more detail how some difficult parts of the implementation were achieved.5 Enhancements/Refinements The following enhancements and refinements were implemented on the real-time system: • Spiralling out from the centre of the ROI • Temporal Filtering to remove trees • Oriented Edges • Expanding all possibilities or best • Reverse Scoring • Truncating the Distance Transform These are explained in detail in Appendix A.4. 8.4. 8. At the leaf nodes a reverse search is executed.

such as lighting. truncated and scaled distance matching approaches. in a very large font. Oriented edge detection was not fully implemented in the real-time environment as match accuracy was reasonable. . The hierarchy is interchangeable with the traffic sign hierarchy thanks to polymorphism. By creating a hierarchy of differently sized objects they can all be searched for simultaneously.4.6 Final Matching Algorithm Used. creating the templates using bitmaps of the letters.7 Further Examples Some further examples were programmed to prove the possibilities of the matching algorithm. car movements.60 CHAPTER 8. occlusions. This system allowed the results of many of the refinements to be tested. 8. Size Variance Matching Hierarchical chamfer matching can be employed to create a size variant matching system. Traffic sign footage contains many uncontrollable variables. that of letters. And some of the refinements/bugs were discovered by preparing this easier case. due to false matches. The spiral search pattern was rejected as the matching may need to find multiple objects. and providing the letter images. trees. Temporal filtering was also unnecessary. • Letter Matching • Size Variance Matching • Deformable Contour Matching Letter Matching Many of the improvements mentioned were discovered by using a simplified matching case. by printed out copies. The final matching algorithm implemented used the maximum. The hierarchy works on text of a known font.4. By creating a hierarchy of letters. Only the best match at each level of the hierarchy was expanded. the matching can be demonstrated and tested in real-time in the lab. SOFTWARE DESIGN AND IMPLEMENTATION 8. damage.

. are rotations of objects. where masking of the reverse search is used. etc. this would also require a different mask for each template. diamond.14: My Size Variant Hierarchy Rotational Matching Another possible scenario.. Most rotations bare a similarity to the previous and next rotation.4. as already mentioned.8.. Figure 8. for example (Figure A. Due to this added complication. REAL-TIME 61 The hierarchy is generally formed by grouping similarly sized objects. the example application realised used simple circles on a plain background.1. instead of different masks for subtrees (circle. In this application. a hierarchy can be created of similar rotational shapes.). By exploiting this.

62

CHAPTER 8. SOFTWARE DESIGN AND IMPLEMENTATION

Chapter 9

Results
9.1 Hierarchy Creation

The hierarchy creation code could successfully optimise small hierarchies. When given a relatively low threshold, the results were not affected by the order of images therefore were optimised. If the threshold was high, the results were the same for groups of similar orders. The creation system was designed around recursive programming. This allowed problems to be simplified, however, it did not create an efficient system. Due to memory constraints large hierarchies and those with many combinations (i.e. low threshold) should be avoided.

9.1.1

Hierarchies

Selected results generated by the hierarchy creation are included.

Diamond Signs The hierarchy included in A.9.1 is of a sub-set of diamond sign templates to demonstrate the effectiveness of the automated hierarchy creation. The following commands were used: 63

64

CHAPTER 9. RESULTS

[diss, images]= createTree;

[imagestruct, adjsquare]= setup1(diss, images, 0.5);

[hierarchy, temps, options] = combinegroups(images, imagestruct, 0.5, adjsquare);

The scores achieved by each order before optimisation are presented in figure A.14. After optimisation it can be seen (figure A.15) that groups with similar starting orders have given the same score, hence some “optimisation” has been achieved.

Circular Signs

A Hierarchy of Circular signs was generated in a similar manner. The following figure represents the hierarchy created. The scores achieved are represented in figure A.36. They demonstrate

Figure 9.1: Circular Sign Hierarchy an “optimal” solution has probably been achieved, because no matter what order the hierarchy was created in the result was the same.

Others

Hierarchies for letters, multi-resolution and deformable contours are not included due to size restrictions on this document.

9. due to the exploitation of the canny edge detection. The first major improvement to matching accuracy was made by the masking of reverse scoring (8.2).10 detail the results achieved in the MATLAB matching prototype. increased the accuracy and precision of the match.9. This refined the sign matching but still allowed unnecessary search expansion. It produced similar scores for an exact match as it did for noise-like patterns created by the edge detection of trees. Localised thresholding as a means to limit unnecessary expansion of the search. Using a Hierarchical search in this situation did not improve the matching. Oriented edge were not implemented but results indicate that this or another stage may still be necessary for match verification.6). Which was able to match signs in static images. So did matching in different feature extractions. simple temporal filtering.3. MATLAB MATCHING 65 9. such as trees.3 Real-Time Matching The results presented on traffic sign detection show that a real-time detection system based on Hierarchical Chamfer Matching built for a general purpose platform is a realistic goal. 9. The use of truncated distances was retained.2 Matlab Matching The basic system of single template matching has limited possibilities. The development of the algorithm provided some insights into valuable enhancements. In much of the footage recorded . The matching algorithm is intolerant to poor edge detection. The addition of reverse matching had limited success. Using additional oriented edge information. The ideas rejected include the spiral search pattern. and expanding “all” matches below the threshold.1 Matlab Matching Results The diagrams in A. proved computationally expensive (Figure A. There was only a slight increase in computational expense.2.2. This still did not limit the search’s tendency to expand in places that contained dense edge information.

9. which means features are missing from the image and a good forward match is not possible. This blurring reduces the quality of the edge detection. the results are much better due to the controlled environment.3) show the system output. Figure 9. The biggest problem affecting matching was the poor edge detection resulting from blurred footage. probably due to automatic focussing of the camera. In other examples.2 and 9.1 Performance On a 1. The images here (figures 9. These results were on video that was 360 × 288 pixels.2 Results Virtually all traffic signs that were edge detected without distortion were found.66 CHAPTER 9.3. False matches were infrequent and limited to noisy sections. RESULTS the sign is blurred as it nears the correct size for matching.3. to under 10 frames a second for more difficult scenes. such as letter matching.2: 50 Sign . 9.6Ghz Pentium 4 with 256 Meg of RAM the frame rates varied bfrom over 20 frames/second in scenes were there was little noise to cause unnecessary expansion.

3. a very robust detection system would be possible.4 Size Variant Matching The demonstration of matching over 20 different sized circles. 9. This was once again in real-time.5 Rotational Matching A simple cross pattern was sampled at varying rotations and placed into a hierarchy. then using an object type hierarchy a very large number of shapes and sizes could be recognised.3. REAL-TIME MATCHING 67 Figure 9.3. By creating a size hierarchy.3 Letter Matching A hierarchy was created using the alphabet in a known font. Results proved that print-outs of letters could be matched when held in front of a USB camera. 9.3. with virtually no false matches in an “office” environment. By combining rotations with scaling and skews in a large hierarchy. This further demonstrated hierarchical matching. This demonstrated the algorithm’s ability to match rotations of objects.3: 60 Sign 9. has shown that this algorithm could be suitable for matching an object of unknown size.9. . at a high frame rate.

68 CHAPTER 9. My Visual C++ programming skills were improved due to the complicated nature of the matching algorithm.4 9. 9.4. My strength is knowledge. distance transforms and matching metrics. This allows me to quickly evaluate possible approaches based on results of prototyping. With the knowledge gained during the thesis I could perform much better if the application were to be programmed again. I greatly expanded my mathematical knowledge of image processing.2 Strengths/Weaknesses My main weakness was object orientated design and the ability to realise that design. RESULTS 9.1 My Performance Skills Learnt During the course of this thesis I have learnt many new skills. I learnt several graph theory concepts and the basics of constructing a tree. . image processing and object oriented programming. particularly edge detection. I also benefitted from investigating the OpenCV library in great detail. These included graph theory.4. experience and understanding of Image Processing Algorithms. I was unable to build a well structured program.

1 Video Footage The fixation of a camera to the vehicle and the use of a more suitable camera would increase the consistency of footage. A clearer image would allow better edge detection and therefore better matching. • More consistent video footage • Temporal Information included • Better Object Oriented Design • Improved Hierarchy Generation • Optimisation • Final verification stage 10. 69 .Chapter 10 Future Development Several aspects of this thesis could be improved if future work was conducted.

the quality of code may be increased. and automatically generate the concrete builder in c++/pseudocode the application would be easier for developers to use. Tracking of potential signs from frame to frame and use of the expected paths of traffic signs could help to speed up matching by pin-pointing likely locations. such as trees. 5]. could be stopped. FUTURE DEVELOPMENT 10.6 Final Verification Stage A final verification stage such as using orientation information (similar to the MATLAB prototype). Further improvements would be possible if unnecessary expansion caused by noise. Changes in hierarchy would be simpler.3 Better OO Design If the design were more complete and could be effectively realised. 10. .4 Improved Hierarchy Generation If the MATLABtm system was able to handle larger image databases.70 CHAPTER 10. This would not necessarily make it faster. 10. colour or neural network stage [4. A faster application would allow larger hierarchies and hence more robust matching. but would increase the readability. 10.5 Optimisation After the design was improved and the code simplified effort could be spent improving the efficiency. Larger hierarchies could also be created. 10.2 Temporal Information The use of temporal information in the application would increase the quality and speed of matching. Design methods for speed of applications in the environment should be studied further.

As are results. This document contains the assumptions and a brief specification of the recognition system. Very few false matches are detected. A hierarchy creation system was implemented in MATLAB. but can produce excellent results. This allowed various parameters and properties of the metric to be explored. The real-time system is dependant on quality video footage. It uses graph theory concepts derived from [2].Chapter 11 Conclusions This thesis proved that the hierarchical distance matching algorithm is effective for many image processing scenarios. recommendations 71 . It is worthy of further investigation and development. A simple static matching system was developed to prototype the algorithm outlined in [4] and [5]. It is incomplete but matches single templates on images with high accuracy yet poor time performance. The matching was then prototyped. The output of this can be used in both the static and real-time matching systems. in particular traffic sign recognition. again in MATLAB. The traffic sign matching application developed has proven that “smart” vehicle systems are not far away from mass production. The hierarchy creation system that has been developed will create and optimise a structure. The goals of the thesis were achieved. A real-time matching application was built utilising the IPL Image Processing and OpenCV libraries. Relevant literature has been reviewed to provide theoretical basis for this thesis. It is capable of recognition up to 20 frames per second. Implementation and design details of both hardware and software are included. It was also expanded to other matching scenarios.

72 CHAPTER 11. . CONCLUSIONS for future work and conclusions.

It involves the use of a camera mounted in the car.1 Australia’s Innovators Of The Future Craig Northway’s Real-Time Traffic Sign Recognition project is based on the work of Daimler Chrysler Research. vehicles could soon have the ability to warn drivers of pending situations or automatically take evasion action. he said. which hopes to develop Smart Cars that avoid pedestrians and remind you of the speed limits. He also told how the project would be demonstrated at a transport mission in Ireland later in the year. 73 . Craig’s supervisor Associate Professor Brian Lovell said he hopes the device will become marketable. A computer processes the information into (sic) real time.Chapter 12 Publication Extract from article to be published in UQ News on the 20th of October. 12. If the project is developed further.

PUBLICATION .74 CHAPTER 12.

pp. “Visual routines for autonomouis driving. [10] M.Bibliography [1] C. “Graph-Theoretic Clustering for Image Grouping and Retrieval. [3] G. M. Philomin. H.” IEEE Transactions on Pattern Analysis and Machine Intelligence. 6. “Fast Object Recognition in Noisy Images using Simulated Annealing. “Automatic target recognition by matching oriented edge pixels. “Realtime traffic sign recognition. on Computer Vision and Pattern Recognition. 1988. 1996. et. [7] J. 1. 849–865. “Hierarchical chamfer matching: A parametric edge matching algorithm.” Image and Vision Computing.. Huttenlocher. vol. 103–113. P.” IEEE Transactions on Image Processing. Borgefors. [4] D. “Multi-feature hierarchical template matching using distance transforms. al. Ballard. 1997. pp. no. vol. Aksoy and R. [2] S.” 2000. 87–93. 10. 75 . 1999.. pp. Haralick.” 1998. et al. of the International Conference on Pattern Recognition. pp. [6] G. Markis. Betke and N.” 1993. pp. [9] G. Gavrila. [8] J. last viewed on 30/03/02. Gavrila and V. “Real-time object detection for ”smart” vehicles. 1998.” Web Site. 14. 439–444. [5] D. vol. M.” In International Conference on Computer Vision. Saligan and D.” In Proc. Logemann. 109–223. “An active vision system for real-time traffic sign recognition. C. “Robust method for road sign detection and recognition. M. 1999.” IEEE Conf. Olson and D.

pp. Dhanaraks and N. K. pp. [17] P. 15.” Third IEEE Computer Society Workshop on Perceptual Organization in Computer Vision (POCV01).” Proceedings of the IEEE Southwest Symposium on Image Analysis and Interpretation. O. Jain. Rucklidge.” 1995. “A probabilistic formulation for hausdorff matching. C. and F. E. “Face Detection in Color Images. 2000. of the IEEE Conference on Computer Vision and Pattern Recognition. Iqbal and J. Jr. “Hierarchical Artificial Neural Networks for Edge Enhancement. Poggio. 193–199. A. [12] P. “Perceptual Grouping for Image Retrieval and Classification. A. 1996.” 1998. 1993. 26. Rucklidge.” IEEE Transactions on Computers. Huttenlocher and W. last viewed on 23/03/02.” In Proc. Papageorgiou and T.” IEEE Transactions on Pattern Analysis and Machine Intelligence. Turcajova and J.76 BIBLIOGRAPHY [11] D. “Pedestrian detection using wavelet templates. of the International Conference on Computer Vision. “Road sign detection and recognition. no. 457–464. “PLANAR IMAGE MOSAICING BY HIERARCHICAL CHAMFER MATCHING ALGORITHM. Lu and A.” In Proc. S. Olson. H. M. 1998. 1995. vol. Rogahn. [16] S.” Pattern Recognition. M. [18] R. of the IEEE Conference on ComputerVision and Pattern Recognition. pp. [13] G. D. “Locating objects using the hausdorff distance. 2001. Aggarwal. K. 1997. [21] R. 427–435. 19–1–19–4. K. vol.” In Proc. H.-L. . Oren. 1993. King Sun. “A non-parametric positioning procedure for pattern classification. pp. Szeto. [22] E. Katsky. [20] Q. [19] Estevez and Kehtarnavaz. “Comparing images using the hausdorff distance. Covavisaruch.” 2001. 7.-M. pp. “A hierarchical multiresolution technique for image registration.” Web Site. C. 614–624. [15] C. [14] W. pp. 850–863. 8. 1969. vol. “A real-time histographic approach to road sign recognition.

Venkata and S. Graphs An Introductory Approach.” IEEE Computer Vision and Pattern Recognition. New Jersey. 1997. 1990. [25] G. K. Logic and Discrete Mathematics. Z. J. [27] R. Watkins. Abdel-Mottaleb. 427–435. Jing Huang. 1986. 344–371.” Computer Vision. On Growing Better Decision Trees from Data.BIBLIOGRAPHY 77 [23] R. E. Graphics and Image Processing.” 1999. vol. Hebert. S Ravi Kumar. [29] K. 1999. CVPR ’97. 34. [26] A. pp. Wilson and J. New York. M. [28] W. K. . 1990. “Recognzing objects by matchin oriented points. Borgefors. “Distance transforms in digital images. [24] S. Prentice Hall. John Wiley and Sons. Grassmann and J. Johns Hopkins University. pp. 1997.” Part of the IS and T SPIE Conference on Storage and Retrieval for Image and Video Databases VII. “Hierarchical clustering algorithm for fast image retrieval. paul Tremblay. “An Automatic Hierarchical Image Classification Scheme. Murthy. Johnson and M. PhD thesis. J.

78 BIBLIOGRAPHY .

1 A. This can prove the potential of the algorithm. Due to the reflective nature of traffic signs. 79 .1. A. 5]. A.2 Lighting The lighting during the filming should be reasonable. In a smart vehicle system the camera must be able to resolve images at speeds of up to 110km/h.1.Appendix A A. For demonstration purposes footage can be taken from a slow moving vehicle. the signs would tend to pass through the same area of the footage (Upper Left in Australia).1. Once again in a smart vehicle system a high quality camera capable of low light filming would be used. If the camera was mounted on the dash of a vehicle.3 Position It is reasonable to assume that the traffic signs consistently appear in a similar region of the video footage. such that once again the camera can resolve the image. detection can be performed at night using this method [4.1 Assumptions Speed The camera must be able to resolve a sharp image from a fast moving vehicle.

A. Figure A. Small amounts of damage should not affect the matching. A. twisted or missing sections.80 APPENDIX A. The HCM algorithm is unable to rectify this situation. without incorporating these skewed images into the hierarchy.4 Angle The relative angle between the car and the sign is close to perpendicular.1. In the Daimler Chrysler system [4. Signs that have suffered damage may be bent.1: Multi-Resolution Hierarchy [4] .6 Size Invariance Due to the Size invariant nature of the algorithm it must be assumed that the signs pass through this size(s) as the car approaches them without being obscured.1. It is fair to assume that most signs are relatively undamaged. 5] sign templates are two sizes.5 Damage Signs must be assumed to be undamaged.1. reducing the chance of them being ”missed”. A. If the car were in an extreme right lane. the image of the sign would be skewed severely. as they are regularly maintained by local governments.

7 Computer Vision Functions The following assumption relates to the Computer Vision functions present in the IPL and Open CV libraries. Thus Traffic signs. ASSUMPTIONS 81 A. . The Edge Detection should give a reliable single line outline of the signs. are predictable shaped similar objects suitable for HCM.8 Objects One major assumption must be made about the shapes to be detected.1.1. The comparative shapes of bananas. text of known font. Traffic signs fulfil this assumption. Even within one type of fruit. such as bananas there is sufficient variation with similarities to create a hierarchy of bananas alone. oranges and potatoes are dissimilar. as there are a limited number of signs that are easily grouped into basic outline shapes. car outlines.1. For HCM to be effective the shape of the objects should be similar. For example a hierarchy of fruits and vegetables might be unsuccessful. If the functions are correct they should be able to produce an edge detection of the signs in most circumstances based on set thresholds providing the video meets the previous assumptions.A. A.

com/software/products/perflib/ijl/index..1 Programming MATLAB MATLAB will not be new to most electrical engineers.2 A. File Source. are operated on and end in filters.htm A.82 APPENDIX A. It is untyped allowing fast prototyping of algorithms. but not with sufficient structure or speed for extensive programming. For more information see: http://msdn. decompressors etc. A. TV tuner. It is a numerical mathematics package. Image data structures . are operated on by filters such as splitters.htm.4 Open CV The open source computer vision library is available free of charge for research purposes from http://www.2. This could be hard coded or. Filter graphs start with a source. It can be compiled at run-time or pre-compiled into dlls.intel. able to be programmed using m-files in language similar to C or Java. and displayed by renderers or written to files by writers. This uses SIMD instruction sets to perform efficient operations on media such as audio and video. This has since been discontinued as a free download.microsoft.asp A. USB Camera. e.com/software/products/opensource/libraries/cvfl. Filters are joined by COM objects. A graph of filters is created.2 Direct Show Direct show is Microsoft’s architecture for streaming media.2.g.com/default. A.2. In this system streams originate. by using an application from the SDK.2.intel.3 IPL Image Processing Library The IPL Image Processing library was created by Intel to use their extended MMX instruction set. For more information see: http://www. designed graphically. SIMD stands for single instruction multiple data and is advantageous in situations where recurring operations are made to large amounts of data such as in signal processing.

2. PROGRAMMING 83 from the IPL imaging library are used extensively.A. It is a set of more complex image processing functions compared to the IPL library. .

A.m Flowchart .84 APPENDIX A.3 Extra Hierarchy Implementation Flowcharts Figure A.2: findbestnotin.

3: anneal. EXTRA HIERARCHY IMPLEMENTATION FLOWCHARTS 85 Figure A.m Flowchart .A.3.

4: remove.m Flowchart .86 APPENDIX A. Figure A.

threes = ([edge(i-1. j+1) edge(i+1. cstrong = floor((idxStrong-1)/m)+1. e2(idxWeak) = 1.ay. ’thin’.2 Orientation Map fours = ([edge(i-1.ax. 8). 1).4.j) = threemin + 3.:. idxWeak = idxLocalMax(mag(idxLocalMax) > lowThresh). if (fourmin > threemin) if (threemin < edge(i.1 Prototype Orientation Code Orientated Edge Transform for dir = 1:4 e2 = repmat(logical(uint8(0)). j-1) edge(i+1.mag). j-1) edge(i. j) edge(i.3)*2)). fourpos] = min(fours). m. if (min([threes fours])<40) [fourmin.4. [threemin. end A. idxWeak(mag(idxWeak) > highThresh)]. e2 = bwmorph(e2.4 A. e(idxWeak) = 1.j)) newedge(i. (j-1+(threepos . j) edge(i+1. else . idxLocalMax = cannyFindLocalMaxima(dir.A. m)+1. threepos] = min(threes).*(im2double(e2)). j+1)]). e2 = bwselect(e2. cstrong. idxStrong = [idxStrong. n). %this should create a direction map. j-1) edge(i-1.4.j) = direction(i. % Thin double (or triple) pixel wide contours directionmatrix(:.dir) = dir. rstrong = rem(idxStrong-1. PROTOTYPE ORIENTATION CODE 87 A.. rstrong.. if (threepos > 2) direction(i. j+1)]).

else direction(i. (j-1+(fourpos .1)*2)).j)) newedge(i.j) = direction(i-1. direction(i.3)*2)). (j-1+(fourpos . if (fourpos > 2) direction(i. j). end end end end .j) = direction((i-1+(threepos-1)*2).j) = fourmin + 4. end end else if(fourmin < edge(i.j) = direction(i+1.88 APPENDIX A.

5 A. whereas in higher gradient areas the threshold could be increased to show fewer edges. where there are many different areas. This would allow edges to be found. with many different textures and lighting conditions is not appropriate. This is shown in this command: highThresh = min(find(cumsum(counts) > PercentOfPixelsNotEdges*m*n)) / 64. In areas of low gradients.5.A. A localised threshold Figure A.7. even if the maximum gradients were very low. Where the image contains few “textures” A global threshold is excellent. where PercentOfPixelsNotEdges = 0. An informal statistical study of sections of tree image was conducted to see if any recognisable . The standard MATLAB edge detection command when used without parameters adjusts the threshold such that 70% of the pixels are registered as on. Assigning one global threshold for the whole image in a situation such as traffic sign recognition.1 Rejected Prototype Implementations Localised Tresholding Localised Tresholding was investigated as a technique to remove the noise caused by trees.5. The localised thresholds were to be applied using the canny edge detector. the threshold could be lowered to detect fine details. REJECTED PROTOTYPE IMPLEMENTATIONS 89 A.5: Simple Image for each section of the image would provide better edge detection. In examples such as this classic figure A.5.

This would result in only major edges of the trees being found. This was achieved by setting an average threshold.n] = size(image).6: Localised Thesholding characteristics of tree gradient detection could be found to raise the threshold in these areas. As expected. areas of tree contained high average gradients with high standard deviations.1*sigma/max(col). By localising the threshold based on the mean and/or standard deviation of the gradient image in that region it was possible to keep only the major features of the trees and signs. The following code demonstrates the thresholding: sigma = std2(image). EX = median(image(:)). but raising it if the mean/standard deviation passed a certain threshold. if (lowthres >= thres) %std is really low . There were many edges and hence variations in gradients. the major features were kept. and rough position. and this feature detection could have been used in the first stage of a matching hierarchy to determine sign type.6. similar to the MATLAB default. lowthres = EX/max(image(:)). thres = thres/max(image(:)). Figure A.%thres . Unfortunately the same was true of the “inside” details of signs. thres = EX + 1*sigma.90 APPENDIX A. [m. As the results of this thresholding method show (figure A.

5.05. standard deviation and mean. end Even though this provided a reasonable starting point for the search these calculations. lowthres = 0. .98. It becomes expensive to produce multiple feature extractions of the same image in MATLAB. It proved very difficult to sub-sample and retain the shape of the sign. lowthres = minthres . Even when a custom canny edge detection was implemented to allow the use of the same gradient image every time.0. REJECTED PROTOTYPE IMPLEMENTATIONS 91 thres = 0. A.5. The custom canny edge detection did not perform well and was rejected in favour of other possible solutions.99.3 Sub-Sampling By sub-sampling the edge detection it was hoped the general shape of the sign would remain.2 Different Feature Extractions A simpler method of implementing the same concept could have been using different levels of feature extraction for each level of the hierarchy. as they were being detected at the smallest possible size at which the features could be resolved with the edge detection. were computationally expensive.A. so a quicker method was sought. A. end if thres < minthres thres = minthres. This was to ensure they are detected as early as possible.5. but the trees would “disappear” or become more random compared to the distinct outline shape of the traffic signs.

A.7: Intended Sequence Diagram .92 APPENDIX A.6 UML of Real-Time System Figure A.

UML OF REAL-TIME SYSTEM 93 Figure A.6.8: Actual Class Diagram .A.

Figure A.94 APPENDIX A.9: Actual Sequence Diagram .

and as presented later. 255). destroying the header has caused problems. 1 ≈ 50. A. iplAdd(imghtempdist. Thus 5 = 50. A. imghtempdist.1 Distance Transform When using a distance transform. In some instances of referencing. This second example from the template creation scales the distances such that all values that could be represented by an 8-bit unsigned integer are output.imghgray32F.2 Deallocation Deallocation when using IplImage objects seems difficult. imghtempdist). iplScaleFP(imghgray32F. iplThreshold(imghtempdist. 10). 0. This can be done when scaling the image type from floating point to integer...7. Other complex sections of code where commenting may not be sufficient are shown.A.. etc. NULL). CODE DETAILS OF REAL-TIME SYSTEM 95 A. and scales it such that each distance is ten times its value. In this first example the data is scaled such that only distances rom 0-5 are included in the output. CV_DIST_MASK_5. 5).7 Code Details of Real-Time System Some of the functions from the IPL Image Processing Library were used and should be documented. imghtempdist. 255). So in these situations only the image data is deallocated. It then truncated this at 5.7. iplScaleFP(imghgray32F.. imghtempdist. . 0 = 0. 4 = 40. imghmult. it is necessary to truncate the values. imghmult. NULL). cvDistTransform(imghinv. CV_DIST_L2. CV_DIST_L2. 4 ≈ 200 .. iplMultiplyS(imghtempdist. Especially when referencing them across classes. 0. Thus 5 will be ≈ 255. The structure and header must always be deallocate. This will evenly place these values from zero to the maximum of the data type being scaled to. cvDistTransform(imghinv. CV_DIST_MASK_5.imghgray32F.7.

pointing to the array of combinational templates as its child 7. iplDeallocateImage(imgh). The variable names help describe the process as each template is named after the letters it represented. Repeat 2-3 for each leaf group 5. Repeat 5-6 as necessary 8.7. It is possible to use IPL IMAGE ALL as a parameter. Point each of the root templates to the appropriate combinational template For examples see the letter hierarchies mytreea. To code this the following procedure should be used: 1. Create the arrays of leave templates 3. Various constructors are available for the leaf and node templates. . It must be constructed in a bottom up approach.96 APPENDIX A. Create the combinational template for this group with the array of leaves as its children array 4.3 mytree The method for creating a mytree concrete builder is not automated from the image hierarchy. but if objects share IplROI’s this will cause errors as they are also deallocated. mytreel. IPL_IMAGE_HEADER). A. For each intermediate stage create an array of combination templates 6. Create the root array 2. Create the combinational template of the previous combinational template. iplDeallocate(imgh.

m converts the images to the *. p_filemask). . TEMPX*3). p_datamask.7. The pixel ordering is different. cvSetData(imghmask. FILE *p_filemask.7. TEMPY). CODE DETAILS OF REAL-TIME SYSTEM 97 A. fclose(p_filemask). 3. Use cvSetData to set and IplImage object to point to the data. "rb"). IPL_DEPTH_8U. 3). imghmask = cvCreateImageHeader(cvSize(TEMPX. BYTE *p_datamask = new BYTE[TEMPX*TEMPY*3].tmp format.4 Template Format The necessary format for images to be included in the template is an unsigned character file of each pixel (much like a bitmap).A. TEMPX*TEMPY. p_filemask = fopen(maskname. The MATLAB file templatecreate. Then they can just be read with a FILE pointer into a BYTE array. fread(p_datamask.

98

APPENDIX A.

A.8

Tested Enhancements/Refinements to Real-Time System

Spiral Design

In a traffic sign matching scenario, there are particular assumptions (already stated) that can be made about the location of a traffic sign. The sign is more likely to be at a particular height, in a particular horizontal area. This property can be used to increase the speed of the search. The following search pattern was designed:

Figure A.10: Spiral Search Pattern Compared to a simple search following this or similar pattern: It can be faster if stopped when

Figure A.11: Straight Search Pattern a match is found. Thus this search would only be of advantage if only one sign was assumed present and false matches could be guaranteed not to occur. If there are multiple signs to be detected, or false matches are likely the entire area should be searched and any advantage of the spiralling search is lost. This design possibility was ruled out due to the likely occurrences of false matches shown by the prototype matching application.

A.8. TESTED ENHANCEMENTS/REFINEMENTS TO REAL-TIME SYSTEM

99

Temporal Filtering to Remove Trees

Temporal Filtering was briefly investigated to remove trees. A simple subtraction of background from frame to frame would remove objects that change little. Due to the noise like nature of the tree edge detections, they are likely to change slightly regardless. The possibility of a sign being surrounded by tree, hence affect by this subtraction is also too great. Testing showed, that signs weren’t affected and trees were thinned of noise, but not sufficiently to make the overhead of background temporal filtering worthwhile.

Oriented Edges

Implementing the oriented edge algorithm prototyped in MATLAB, was briefly attempted. Modifying the Open CV source code proved difficult and time consuming due to the poor documentation and commenting. When the forward and reverse matching was completed, it was demonstration that orientation information was not necessary with other refinements.

Expanding all Possibilities or Best

At each level of the hierarchical search there are multiple methods of expanding the tree. The two main possibilities in this design are: ” Taking the best match above the thresholds at each level ” Taking every match above the thresholds at each level The initial design takes the best match at each level. Due to the small (< 50) nature of the hierarchies in this example it is unlikely the incorrect path will be chosen by taking the best option. In practise this appeared to provide sufficiently accurate results. This will also improve the efficiency of the matching. Taking every match would require a global knowledge of the results of each “thread” of the recursive search. At the end of the matching process, if multiple signs (leaf nodes) had been detected, they would need to be compared. By allowing only one “thread” at each level the leaf template found can be displayed (by copying onto the output), and then the process can iterate to the next position, with no knowledge of which template was found.

100

APPENDIX A.

Reducing the truncated distance

Initially the design used the distance transform straight from the cvDistTransform function. As opposed to the MATLAB implementation this calculated, using the two pass method [25], the distance to “inifinity” i.e. the furthest distance in that image. The MATLAB implementation only used a set number of iterations. Comparative one dimensional cross sections of these distance transforms of a point would be as follows: It can be seen that points a fair way from

Figure A.12: Untruncated Distance Transform

Figure A.13: Truncated Distance Transform the edge detection are still given a high value in the untruncated situation. In the truncated case, pixels beyond a certain distance are discounted, similar to a weighted hausdorff matching technique. By only allowing pixels close to the object to score, poor matching features get ignored, increasing the accuracy of the matching.

Maximum vs. Minimum

The search was tested as a search for the maximum match and minimum. Maximum matching inverting the distance transform. This has the benefit of not weighting pixels that don’t match as the truncated pixels are zero.

I can control the weighting of pixels that are scored.A. Missing features can destroy a match. 230. . TESTED ENHANCEMENTS/REFINEMENTS TO REAL-TIME SYSTEM 101 The distance matching scores are an average per pixel. Scaling the distance By scaling the inverted truncated distance I can control the ”weighting” given to pixels relative to the threshold..8. 253. 240. or other similar amounts. If they are given zero. 2. they will contribute nothing but their presence in the average.. 255. due to the poor resolution.e... If pixels outside the truncated distance are scored as the maximum of the image type. They can still weight the score. If I were to use 255.. Non-linear functions (including the truncation mentioned earlier) could be applied to effect the matching. 1. No noticeable difference could be seen between either.. By scaling this to 250. i. and little accuracy over the scale is given. to represent 0.. 254.

9 A.9. A.1 Hierarchy Results Diamond Signs Figure A.14: Original Scores .102 APPENDIX A.

HIERARCHY RESULTS 103 Figure A.15: Optimised Scores .A.9.

template = self .104 APPENDIX A.17: First Group Template Figure A. Figure A.18: Second Group.29.16: First Group Figure A. The following leaf level groupings were generated figures A.16-A.

9. HIERARCHY RESULTS 105 Figure A. template = self Figure A.A.20: Fourth Group Figure A.19: Third Group.21: Fourth Group Template .

22: Fifth Group Figure A.23: Fifth Group Template Figure A.24: Sixth Group . Figure A.106 APPENDIX A.

26: Seventh Group Figure A. HIERARCHY RESULTS 107 Figure A.9.27: Seventh Group Template Figure A.28: Eigth Group .25: Sixth Group Template Figure A.A.

29: Eight Group Template .108 APPENDIX A. Figure A.

34).The second group was of the 3rd and 5th groups (figure A. HIERARCHY RESULTS 109 It is apparent from inspection of the groups that the optimisation is “sensible’. 2nd and 4th groups (figure A.A.30). By applying the same commands on the template images. upon closer inspection it can be seen that despite their likeness neither have similar features aligned with the sign outline.9. Figure A. Though the crossroad image has not been placed in a group with the left side road image. The Crossroad sign was still “by itself” and the last grouping was of the 7th and 8th groups (figure A. The first grouping was of the 1st.31: First Template Group Combinational Template .30: First Template Group Figure A. the next level of the hierarchy is generated.32).

34: Last Template Group Figure A.35: Last Template Group Combinational Template .32: Second Template Group Figure A.110 APPENDIX A.33: Second Template Group Combinational Template Figure A. Figure A.

HIERARCHY RESULTS 111 A.9.2 Circular Signs Scores Figure A.A.36: Second Level Optimisation .9.

40 A.37: Original Image Figure A. The unnecessary expansion of the search over noisy areas of the edge detection can be seen (figures A.38) and distance transform (figure A.39).41) .37).38: Oriented Edge Image This diagram shows the scores achieved at points throughout the image.10 Matlab Matching Results This is the original image (figure A.112 APPENDIX A. A. Figure A. the oriented edge detection (figure A.

. MATLAB MATCHING RESULTS 113 Figure A.A.39: Distance Transform Image and the match (figure A.10.42).

42: Match .114 APPENDIX A.40: Scores Figure A.41: Closer View of scores Figure A. Figure A.

the MATLAB matching code in the “matching” directory.A.12. retaining the directory name of the example it is based on.11.11 CD Included on the CD (with the code) are a PDF version of this document and demonstration footage of matching. CD 115 File alreadyin chamfer combinegroups createhier createtemp createTree directionchamfer. The MATLAB Hierarchy creation code is in the “hierarchy” directory.1 Code Listing All code is in the directory “codelisting”.1: Hierarchy A. and the real-time code in “EZRGB24”.12 A. A. . triplets and quads Table A.m Description Checks if the value is already in an array Performs a 3-4 chamfer Transform Combines the groups Creates the hierarchy Creates templates Imports files from directory 3-4 Chamfer Transform also produces direction information findbestnotin remove setup1 Finds the best group not in the hierarchy Removes a group from a hierarchy makes all the pairs.

116 APPENDIX A. .m) is in the “coexisting” directory.2: MATLAB Matching File EZRGB24 mytree mytemplatev mytreev mytreel Description The Filter Abstract Builder for Trees Template Class A Concrete Builder Another Concrete Builder Table A.m simplepyroverlay . Templates have been included in the “templates” directory under coexisting but not described. Real-Time Header files are also included. File nonmaxsuppression R10Simplepyroverlay overlay Description Simple Non-maximal Suppression fine/coarse search using one template takes an image and a template and simply translates the template across the image simplepyrdirectedoverlay.m simple pyramid overlay with edge direction simplepyroverlay. The file for creating templates (templatecreate.3: Real-Time Hierarchy MATLAB Matching This code is “unfinished” hence only useful files are described.simple finecoarse overlaying Table A.

A. CODE 117 A.au) notes from ELEC4600 on compiling direct show applications for more information. It must also be able to find the *.dlls at run time. Include paths must be set. The EZRGB24 example which this is based on needs the baseclasses as referenced in the project settings.12. To run the code templates must be found (Hidden in codelisting/templates/temps).2 Compilation Compilation of the Visual C++ code requires the installation of Microsoft’s DirectX SDK. . See Brian Lovell’s (lovell@itee.uq.edu.lib files for IPL and OpenCV at compile and the appropriate *.12. IPL Image Processing Library and OpenCV.

Sign up to vote on this title
UsefulNot useful

Master Your Semester with Scribd & The New York Times

Special offer: Get 4 months of Scribd and The New York Times for just $1.87 per week!

Master Your Semester with a Special Offer from Scribd & The New York Times