You are on page 1of 48

i

Real Time Distance Determination for an Automobile Environment using Inverse Perspective Mapping in Open CV
Shane Tuohy B.E in Electronic and Computer Engineering Supervisor – Dr. Martin Glavin Co Supervisor – Dr. Fearghal Morgan

24 March 2010

ii

Abstract
This project aims to develop a real time distance determination algorithm for use in an automobile environment. Increasingly modern cars are being fitted with image sensors, these image sensors can be used to obtain large amounts of information about the surrounding area. From a single front facing image, it is difficult to determine distances to objects in front of the vehicle with any degree of certainty. There is a non linear relationship between the height of an object in a front facing image, and its distance from the camera. This project aims to use Inverse Perspective Mapping to overcome this problem. Using Inverse Perspective Mapping, we can transform the front facing image to a top down bird’s eye view, in which there is a linear relationship between distances in the image and in the real world. The aim of the project is to implement the algorithm in the C language using the OpenCV libraries. Working in this way provides for a high performance, low overhead system that will be possible to implement and run on a low power embedded device in an automobile environment.

iii

Acknowledgements
I would like to acknowledge the help and support I received throughout the project from my project supervisor Dr. Martin Glavin and postgraduate researchers in the CAR lab, NUIG. In particular I would like to thank Diarmaid O'Cualain for his constant support and patience. This project would not have been possible without debugging help, discussion and encouragement received from my fellow 4th EE/ECE classmates. Finally, I’d like to thank my parents for their continued support over the last 4 years.

iv

Declaration of Originality

I declare that this thesis is my original work except where stated. Date: ___________________________________ Signature: ___________________________________

.................. 24 Finding Threshold Range.......................................................................................... 21 Processing Real Time Images (from camera) .............. ii Acknowledgements............................ iv Table of Figures ....................................................... 16 Warp Perspective ...............................3....................................................................2...............3....................................................................................................... 7 cvSetImageROI .......................................... 23 5............................... 6 OpenCV ............3 4 Inverse Perspective Mapping ......................... 12 4..................................................................................................................................................... iii Declaration of Originality .......3 3...........................................2 4....................................................................................................................................... 15 Threshold Image ............................2 3...................................3 Overall System Flowchart ............................................................................3.................................................................................3 4....................................................................................................4 4.................................................................................3................................................................................................ 15 Capture Video Frame ....... 9 Project Structure ...................... 13 First Frame Operations Flowchart ..........................................1 4...........................................5 Generate lookup array ...1 4.............................................. 8 cvWarpPerspective .................................. 20 Provide Graphical Overlay.......................................................................... 14 Process Specifics ..................................................... 22 4............................................................................ 15 First Frame Operations .....................................................................................................................................................................................1 3...............................................................................5 4..........................6 4................................. 6 3..........................................................................................7 5 Optimisation and Testing .......................................3.................................. 18 Distance Determination ....................................................................................................................................................................................................2 Computer Vision .........2..............................................................................................................2.......................................................................v 1 Contents Abstract ..................... 8 3................................2 4.......... vii 1 2 3 Glossary ............ 2 Background Technologies ..............4 3............ 8 Drawing Functions ....................3........................................................2.....................................4 5............ 25 Using Performance Primitives......................................................................................................................................................1 3................................................................................. 27 Level to Trigger Detection ..............................................1 5............3 5......................................3.......2 5................................................................................................................................................. 1 System Overview......................................................................... 28 ................................................................................................................. 23 Sampling Rate ................................................................... 6 Useful OpenCV Functions .............

...............................................................................................................................................................................................................3 7............................................On the CD ...................... 38 8 9 10 Conclusion .................................................................................................................................... 32 Performance of Algorithms ............................................................6 5....................................................................................................vi 5.................................... 29 Calibration ................2 7.............................................................................................................................................................. 37 Environmental Conditions...2 Selection of Sampling Rate ...................... 34 7 Further Work...................... 41 ...............1........ 32 6......................7 6 Memory Management .......................................................................................................................................................................1 6................................... 37 Tracking .......................................1 7..1........................ 37 7......................................................................................1........................1.......... 39 References .......................................4 Processing Time ............................... 38 Embedded Implementation .............. 40 Appendix A .......................................................................................................................................................................................................................................... 30 Results ..

..................................................................................Inverse Perspective Mapped view of road scene....... 36 ............................ 24 Figure 16 .................... 9 Figure 5 . 35 Figure 26 .....Samples of successful distance determination ..............................Comparison of computation times in seconds .............................................. ............................ 19 Figure 12 – Previous figure after perspective has been warped using transformation matrix ........No Thresholding Applied .............................................................................Overview of proposed system ..........................................................Plot of computation times for different sampling rates ............................................ 17 Figure 9 ...................... 20 Figure 14 – Source image with overlay of rectangular box and distance value ........... 27 Figure 19 . 4 Figure 2 .................................................... and real world distance ............ 28 Figure 20 .................................................... Vertical distance now corresponds linearly to real distance .............................Graph of processing time for each of 3 major constituents of algorithm .............. 14 Figure 8 ........ 31 Figure 22 ..............Inverse Perspective Mapped image with distance indicated by white arrow....... 11 Figure 6 ................................................................................................................... 32 Figure 23 ...Thresholded image from figure above with road removed and non road objects highlighted18 Figure 11 – Sample road scene image before perspective is warped using transformation matrix ............... 26 Figure 18 ..........................................................................................Thresholded image of same scene as figure above with road pixels removed ...............................................................Original Image ................................................... 5 Figure 4 – Illustration of camera position and coordinate systems in use ............................ ready for object detection................................................................. 5 Figure 3 ............Overall System Flowchart ....................... 33 Figure 24 ..............................................................Illustrates mapping of points in source top down image to points in front facing image.vii Table of Figures Figure 1 ......... 22 Figure 15 ............................................. Distorted image of car ahead can be seen in red box at top of image.................................Thresholding with range of ±35 ..........comparison between sampling rate of 1 and 10................Original image before thresholding to remove road pixels .........Example of distance detection performed with small trigger value ........Graph illustrating memory use for several video samples .......... 19 Figure 13 – Transformed image of sample road scene................. 17 Figure 10 ....................................Images illustrating differences between vertical distance on camera image.................First Frame Operations Flowchart ............................................................ 30 Figure 21 – Example of ‘known square’ method of calibration ...... 13 Figure 7 .......................................... 34 Figure 25 .................................... 26 Figure 17 -Small threshold value applied to scene Large threshold value applied to scene .................Measured computation time ...........

1 1 Glossary IPM – Inverse Perspective Mapping OpenCV – Open Computer Vision Thresholding – Process by which pixels above or below a certain intensity are removed from an image C – Low level. compiled programming language gcc – Open source C compiler ROI – Region of Interest .

An increasingly common application of computer vision systems is in the field of safety. . is the car. Current systems on the market from manufacturers such as Mercedes [2] pre-charge brakes and tighten slack on seatbelts if an imminent collision is detected. The Opel Insignia uses a front mounted camera to detect road signs and monitor lane departures providing increased levels of information to drivers. With an effective distance determination algorithm. Computer vision is fundamentally the process by which we can allow machines to ‘see’ the world and react to it. Distance determination in an automobile environment is understandably a worthwhile undertaking. based on the interpretation of the world around them. ESP etc. Using computer vision techniques and optical cameras. For this reason. ABS. have become standard on many car models. Safety is a primary concern for all car manufacturers. One such machine. we can create powerful and helpful real world applications which incorporate real world conditions. were killed in road traffic accidents [1]. It is becoming increasingly common for modern automobiles to be fitted with camera systems to aid in driver awareness and safety.2 2 System Overview In 2005. in theory. Its importance cannot be overstated in fields such as manufacturing. drive itself. useful information to machine operators or users. stability controls. In fact. safety systems can be vastly improved. Using the techniques of computer vision. Cars in the near future will be able to intelligently analyze their environment and react accordingly to improve driver safety. which senses upcoming obstacles and adjusts the speed of the vehicle accordingly. In recent years. it is entirely possible to envision a car that could. Machines can be programmed to detect and respond to dangerous conditions automatically. more than one per day. collision avoidance or prevention systems are in the best interests of car safety. Computer vision can be used to provide accurate. Systems such as those found in the Opal Insignia are becoming more and more popular. combined with lane detection algorithms. potentially lifesaving information. surveillance and environment detection. 396 people. airbags. where computer vision can be leveraged to provide useful. Distance data from a system similar to the one proposed could be applied to an adaptive cruise control system. steps can be taken to alert drivers to potential hazards and unsafe driving.

The proposed system consists of a single front facing video camera mounted on a vehicle capturing video images at 30 frames per second. this leaves them susceptible to interference. a front facing optical camera fitted to a car can have several uses. than a single camera system. embedded devices typically found in automobiles. These systems are also expensive to implement. the passive nature. which are very capable. These can work well when the RADAR signals reflect from a metal object. the same camera can be used for lane detection and road sign detection as well as distance determination. however. it also carries severe processing and configuration overheads. can make implementation of these types of algorithm in real time a reality. A single camera system is more reliable and simpler than any of these methods. yet consume very little power. A dual (or more) camera setup as employed by Toyota in some Lexus models. low resource. they do not. providing a comprehensive security package using a single camera. which provide more information about a scene and are cheap and reliable. more accurate results. Devices such as the Intel Atom processor. and therefore. It is also a much more expensive system to implement. provides more data to process. The proposed system is capable of providing life saving information to drivers. In conclusion. for obvious reasons. Mercedes Benz. flexibility and simplicity of a single camera set up makes it well suited to implementation in an automobile environment. Current research into the area of collision detection is in the field of forward facing cameras. computer vision algorithms were much too computationally heavy to implement in real time on low power devices. possibly from other identical systems approaching them. which render it unsuitable for use in low power. Previously. and Nissan use RADAR or LIDAR sensors to implement collision detection. active devices such as RADAR or LADAR. Mounting these active systems to cars can difficult. and are therefore a sub optimal solution. However. Often. alternatively. This setup distinguishes the system from similar systems which use either a multi camera setup or.3 Currently available systems on the market from manufacturers such as Audi. Active systems such as RADAR or LIDAR require signals to be reflected from targets. carry a significant financial expense. detect pedestrians or animals on the road. and most importantly. .

As the vehicle in front approaches the camera. as indicated by the white lines. vertical distance in the image.4 Figure 1 . does not vary in a linear fashion. now distance on the image is linearly related to distance in the real world. The third image has been transformed to a top down view.Overview of proposed system The two figures below illustrate the problem explicitly. .

Vertical distance now corresponds linearly to real distance .5 Figure 2 .Inverse Perspective Mapped image with distance indicated by white arrow. and real world distance Figure 3 .Images illustrating differences between vertical distance on camera image.

the programmer doesn’t have to continually “reinvent the wheel”. would not have been capable of handling the processing overhead of such a system. Although OpenCV itself is an open source project. There are many commonly used techniques in image processing for computer vision applications. which uses computer vision techniques on a portable device to ‘understand’ a scene and overlay useful information. Vehicles are required to merge into two way traffic and carry out other complex driving manoeuvres autonomously.willowgarage. 1 2 http://www. Without advanced computer vision algorithms.6 3 Background Technologies This chapter illustrates the technology background of the project. It is often found in mobile phones. OpenCV provides implementations of many of these functions allowing for rapid algorithm implementation.1 Computer Vision Computer vision is a rapidly growing field.2 OpenCV2 OpenCV stands for “Open Computer Vision”. more powerful. A recently popular example is the field of augmented reality. quick routines for standard image processing techniques. Originally developed by Intel in 1999. One area in which computer visions are an integral part is the DARPA Urban Challenge1 in which students attempt to create a vehicle that will drive itself. It provides a brief overview of the field of computer vision and the technologies being used to implement the system. This opens up exciting possibilities for innovative computer vision implementations in everyday live. Intel provides a product named “Integrated Performance Primitives”. these challenges would be impossible. 3. While implementing algorithms. 3. The purpose of OpenCV is to provide robust.com/wiki/ . The purpose of OpenCV is to provide open and optimized code to support the advancement of The functions themselves are mostly written in C. microprocessors are increasingly able to handle the computationally expensive overhead of working with images. which prior to now.asp http://opencv. a commercial package of highly optimized routines which OpenCV can use in place of its own routines to speed up computation.mil/grandchallenge/index. OpenCV is a library of functions for image processing.darpa. Modern. it is now an Open Source project released under the BSD license.

leading to a real time. if a pixel is above the threshold value. code for this system was written solely in C. CV_THRESH_TOZERO – If a pixel is below the threshold value. it is assigned a value of 0. Several different types of thresholding are available. it is given an intensity value of 0. generate homography matrices and sample regions of images. CV_THRESH_TOZERO_INV and CV_THRESH_BINARY.1. • • • • • CV_THRESH_BINARY CV_THRESH_BINARY_INV CV_THRESH_TRUNC CV_THRESH_TOZERO_INV CV_THRESH_TOZERO Most pertinent to this project are CV_THRESH_TOZERO. OpenCV provides extensive thresholding options via the cvThreshold function. and compiled using the standard gcc compiler.1 Useful OpenCV Functions 3. or 255. depending on which side of the threshold value the pixel intensity value lies on.2. OpenCV greatly accelerated development by providing routines to threshold images. or close to real time implementation.7 For this particular project.1 cvThreshold Performing a threshold of an image is a fundamental image processing technique. . and performing a particular operation depending on this value. Using a lower level language like C increases performance. Although wrappers for the OpenCV libraries have been developed for high level languages such as C# and Python.2. Otherwise it is not affected. 3. CV_THRESH_BINARY – As the name suggests. it is given an intensity of 0. CV_THRESH_TOZERO_INV – The opposite of above. Otherwise it is not affected. It involves examining the intensity values of each pixel in an image.

this function has proven very useful throughout the project. while leaving the rest of the image the same. averaging.2. 3.2. It allows points to be mapped from one perspective to another through the use of a single function. .3 cvWarpPerspective cvWarpPerspective. smoothing etc to be applied to one part of an image.4 Drawing Functions OpenCV provides numerous drawing functions which allow feedback to be given to the user easily by overlaying shapes and text onto images. This allows functions such as thresholding. This has been the most important function provided by OpenCV for this project.8 3. 3. allows the perspective of an image be warped based on a transformation matrix.2.2 cvSetImageROI cvSetImageROI allows the programmer to perform operations on specified areas of an image. greatly reducing programmer overhead. as the name suggest.

Figure 4 – Illustration of camera position and coordinate systems in use . useless for distance determination. since the relative position of the objects will change in a linear fashion. Vehicles that are far away from the camera appear high in the image. it is however. To overcome this issue. Using this homography matrix. we use Inverse Perspective Mapping [3][4]. The problem is that this does not occur in a linear fashion. Inverse Perspective Mapping (IPM) uses a 3 by 3 homography matrix to translate points from the image plane to those found in the real world. we can transform our image so that we are looking directly at the true road plane.3 Inverse Perspective Mapping A front facing image from a car is useful for many applications. and their relation to one another. The figure below illustrates the different coordinate systems that are employed. They appear progressively lower as they approach the camera.9 3. We can use this to measure the distance between objects in the image with a degree of certainty. therefore there is no simple way to discern distance information based on the position of a vehicle in a front facing image.

vanishing points.v) to its corresponding point in the real world plane W(X.Demonstration of benefits of homogenous coordinates Using the Transformation Matrix above.Illustrating transformation matrix Transformation Matrix We use homogenous co ordinate systems as it allows us to represent points at infinity. e. Z) with Z = 0. resulting in distortion of the image. For our purposes.v) involves a rotation about the angle θ and a translation along the line of sight of the camera.10 As can be seen from the figure above. as can be seen in the image below. as well as constants. mapping a point from I(u.g. The transformation results in images in which any point with a Z coordinate greater than zero is incorrectly mapped. . an image similar to the one in the figure below is perfectly sufficient to determine distances. They also allow us to include division into linear equations. Y. The matrix mathematics are shown below in Equation 1 [5] Equation 1 – Matrix Mathematics for transforming point from image coordinates to world coordinates Simplifying results in Equation 2: Equation 2 . we can map any point in the plane I(u.[6] Equation 3 .

image.Inverse Perspective Mapped view of road scene.11 Figure 5 . . Distorted image of car ahead can be seen in red box at top of scene.

Goal 1 • • • Commission the OpenCV system to load frames of video into memory. Sample the image pixels of the road directly in front of the vehicle. It describes an overall flow for the system before explaining in detail the logic and implementation behind each step in the system. . it is then possible to calculate the distance. Threshold based on these sample pixels to remove the road surface from the image. the ground plane). This project was split into 5 distinct goals to measure progress. The only remaining pixels are that of the sky. by looking for the first pixels of “non-road” directly in front of the vehicle. • Generate the IPM characteristic matrix which is developed from the height of the camera from the ground. This is done by applying the IPM matrix developed in the previous milestone. Goal 4 • Display this information to the driver by overlaying graphics on the original image to clearly indicate the distance to the vehicle in front. Those goals were.t.12 4 Project Structure This chapter describes the structure of work carried out during the course of the project and the development of the system. edges of the road (such as trees and buildings). Goal 2 • Transform the image from the original input view to the IPM transformed view. and the angle at which the camera is mounted (w. One the position is determined. Goal 3 • Determine the distance to the vehicle in front.r. road markings and other vehicles.

1 Overall System Flowchart The following flowchart illustrates general overall system operation.13 Goal 5 • Modify the system to run in real time with a video stream. Figure 6 . this could be achieved in real-time using a video camera and a laptop (or other available embedded signal processing hardware). If possible.Overall System Flowchart . 4.

14 4.First Frame Operations Flowchart .2 First Frame Operations Flowchart The following flowchart illustrates operations carried out on receipt of first frame of video. Figure 7 .

which need only be carried out once.3. map to a square in the destination image. Helpfully. 4. OpenCV provides a function to generate this matrix without needing to manually carry out the mathematical operations listed in the Inverse Perspective Mapping section above. 4 source points are captured from the user.2. we can use the corners of the square to generate the appropriate transformation matrix for the current environmental conditions. If we lay a square shaped object on the road in front of the camera. The source points that are chosen by the user by clicking points in the image.2 First Frame Operations This section of the system consists of several operations. generation of the transformation matrix.3 Process Specifics 4. The function cvGeneratePerspectiveTransform takes an array of points from the source image.15 4. These points are stored in an array. Using these functions we create a CvCapture object from which we can query for frames from either the video.3. The simplified matrix mathematics is illustrated in the equation below. . saving on computation time. 4.3.2 Generate transformation matrix The transformation matrix is the key to the Inverse Perspective Mapping algorithm. so as much work as possible is done at the beginning of the program.1 Capture Video Frame Capturing of video frames is done using OpenCVs file capture or camera capture functions. The data generated through these operations can then be used over and over for each frame of video that is sampled. This is done using a mouse handler to return the position of user selected points in the image. Many of the methods used in this project are computationally heavy to do on the fly. and used for the next step.3.2. 4. This way the stored values can be used repeatedly for each frame.1 Capture source points Firstly. and generates the transformation matrix that maps them to an array of points in the destination (top down) image. or camera connected to the laptop.

16 Importantly. and could be improved by instead taking the median value for all pixels. 9 shows the result of . That way. given a certain image. We threshold based on these values. This gives a good value for the RGB characteristics of the road surface. G and B images from the images containing just the road surface and merge the result • Finally we perform a binary threshold to leave any non road values with a high value to aid in distance determination An example of the thresholding algorithm at work can be seen in the following three figures. This allows us to map points back from the destination image back to the source image. anything in front of the car with an RGB intensity value greater than zero is an object in front of the car. • • Split image into its constituent channels (R. A solution to this problem is to threshold the image to remove road pixels. G. Thresholding works as follows. that is. Fig.3. To obtain a value for the particular scene we are working on. could be implemented to improve the process further. we need to be able to discern what part of an image is the road. B) Use built in OpenCV threshold functions CV_THRESH_TOZERO to remove pixels above and below the threshold value • • This leaves us with images containing just the road surface in the R. The major difficulty that this presents is. Adaptive thresholding. G and B planes We subtract the original R. and what part is a vehicle. how can one detect what is a road pixel and what is not? Roads vary in shade depending on the time of day. This process is somewhat rough. thus eliminating noise values generated by the system. we can sample the value of pixels directly in front of the vehicle. weather etc. applying this operation in the other direction gives us the inverse transformation matrix. We take a small patch of road slightly in front of the car and obtain an average value for the pixels across that patch. using different thresholding values for different areas of the image. 8 shows a sample source image prior to application of the algorithm. 4. Fig.3 Threshold Image In order to detect a vehicle on the road in front of us.

10 shows how non road objects are highlighted to allow for easier detection.Original image before thresholding to remove road pixels Figure 9 . Fig. Figure 8 . Finally.Thresholded image of same scene as figure above with road pixels removed .17 removal of road pixels from the image.

Thresholded image from figure above with road removed and non road objects highlighted As can be seen from the example above. leaving the object in front clearly detectable due to its bright colouring.3. we generated the transformation matrix based on 4 source points in the image mapped to 4 destination points in a transformed image. Now we need to transform the video frame based on this transformation matrix. 4.18 Figure 10 . the thresholding algorithm is very effective. Warping the perspective of an image involves considerable computational overhead. Most of the road surface has been removed. This is the matrix we generated earlier. . It is therefore pertinent to use the operation as sparingly as possible. OpenCV provides a function to transform an image based on a 3x3 homography matrix.4 Warp Perspective As part of the first frame operations. Application of this function to an image results in a transformation similar to the one shown in the figures below.

They are given to illustrate clearly the effects of the perspective transform.19 Figure 11 – Sample road scene image before perspective is warped using transformation matrix Figure 12 – Previous figure after perspective has been warped using transformation matrix Note – Image samples above are not thresholded as they would be during real operation of the system. .

thanks to the threshold applied earlier. Figure 13 – Transformed image of sample road scene.5 Distance Determination Now that we have warped the image to a top down view. We know the coordinates of the front of our car and so.20 4. so we increment the position of the rectangle until the average function returns a non-zero . we can measure the distance to the vehicle in front linearly. we loop vertically upwards through the image. working on a small rectangle in the area in front of the car. This consists of several steps. we average the pixel values across the rectangle. ready for object detection.3. We know that all road pixels are zero. For each small rectangle.

we can overlay graphics and text for display to the user. The figure below shows a sample overlay of information to the user. e.3. The value that we have calculated for distance is in pixels. This value will stay the same as long as the characteristics of the camera (its height from the ground. For the purposes of testing using videos from several different sources. This value will change in a linear fashion as the real world distance changes. It is this linearity that is the strength of the Inverse Perspective Mapping algorithm.5m. Now we have the position of the object directly in front of the car in its transformed image coordinates. This is the value in pixels that corresponds to one meter. It is the distance in pixels from the base of the image (front of our vehicle) to the next object in front of our vehicle.6 Provide Graphical Overlay Now that we have a value for the position of the object in co ordinates relevant to the original image. Graphical information provided to the user is in the form of the distance figure being displayed in the upper left quadrant of the image. This value can calibrated for a particular camera configuration by placing a meter stick in front of the car flat on the road surface and measuring the number of pixels that this corresponds to in the top down view. In order to display an accurate value for distance. The value was calibrated based on the fact that the standard for Irish lane markings is approximately 1. A rectangular box is also drawn around the detected area to verify that the correct object has been detected. focal length etc) remain the same. an approximate scaling factor was chosen based on analysis of several sample top down images.21 value. We transform this point back to the original coordinate system in order to overlay feedback to the user. typically where very little activity takes place. we need to know the ‘scaling factor’.g. a value of 700 pixels will equate to twice the distance 350 pixels equates to. This value is stored in a global variable that is accessible from the main body of code and corresponds to the distance between our vehicle and the object directly in front. . 4.

22 Figure 14 – Source image with overlay of rectangular box and distance value 4. put live real time data from a camera also.3. . This involved implementation of a mechanism by which the program would read frames from a camera rather than a file.7 Processing Real Time Images (from camera) While most design. If the system is started without a command line argument. it had to be made possible to not only work on pre recorded video. testing and processing was carried out on pre recorded video. it attempts to load the video at the path specified by the argument. For the algorithm to function in a real time system. a very important goal had also to be realised. the program will attempt to query frames from a camera attached to the system. If the system is invoked with one command line argument.

we must perform non trivial matrix multiplication. when we need to map a point to the original image from the transformed view. back to the source image we must perform a perspective transform on this point. If we are running the algorithm on a real time system. to its equivalent in the front facing image. and store the values in an array. This allows us to map the point where the object is detected back to the original image with ease. So. IPM view. this overhead may be unacceptable for real time operation. instead of performing the actual transformation. As was explored in the section on Inverse Perspective Mapping.1 Generate lookup array One way of optimising an algorithm is to generate a lookup table. we simply need to reference a value in a pre computed array.23 5 Optimisation and Testing Computer vision is inherently computationally heavy. In order to translate a single point from the top down. . This way we can map a point to its equivalent vertical co-ordinate by simply referencing the lookup array at that point. in order to transform an image from the IPM coordinate system to the image coordinate system. Using the look up table. Algorithms which employ many computer vision techniques can take prohibitively long to compute for real time video. 5. The solution is to use the inverse transformation matrix to map all vertical points back to their equivalent point in the source image in one single operation. The figure below illustrates how points are not linearly mapped back to the original image. we can map any points’ vertical coordinate in the top down image. This section explores some ways in which the system was optimised for maximum accuracy with minimum computation overhead.

Performing all operations on each frame equates to 30 sets of operations per second.046m. 30 frames per second. Sampling and computing distance information every 33 milliseconds provides very little extra information.Illustrates mapping of points in source top down image to points in front facing image 5. instead we perform the operations once every x frames. Over 33ms the difference in distance between our vehicle and the vehicle in front of us will be negligible. the rate of change of distance between the two cars is 1. the difference in distance is 0. This project is targeted to function in an automobile environment using a front mounted camera. usually cars on the same road are travelling at roughly similar speeds. A video file contains. This is much too frequent for our needs. So.38m/s. Most of the time. This value is small enough as to .2 Sampling Rate Image processing operations are computationally heavy operations.24 Figure 15 . the rate of change of distance between our object and the object in front of us is relatively small. in general. For a car travelling 5km/hr faster than another car. In 33ms.

while not removing much of other objects. Enough allowance must be given to allow all road pixels to be removed. Choosing an appropriate level above and below the threshold value is very important. This results in a less accurate thresholding value. we must know something about the characteristics of the road we are travelling on. If the value is too large. This is done by sampling a small box in front of the car and extracting average colour data from this area.3 Finding Threshold Range In order to remove the road surface from the input image. The larger the sampling box used. A larger sample area gives us a more reliable and accurate reading of the road surface. the more likely that road markings or other objects will be included in the averaging process.25 be imperceptible to a user. but there is a trade off in computation time and levels of noise in the image. which when used in thresholding. This involves sampling an area of the road to determine appropriate RGB values for the colour of the road surface. 5. leading to detection of the road as an object. too much of the destination object will be removed. leading to difficulty in distance determination. Therefore we can alter the sampling rate to calculate distance less frequently. and therefore save on resources with little to no visible difference during operation. An example of incorrect values in use can be seen below. decreases reliability and performance. If the value is too small. not all of the road surface will be removed. .

.No Thresholding Applied Figure 17 -Small threshold value applied to scene Large threshold value applied to scene As can be seen from the image above on the left.Original Image . which is clearly too large as we have removed much of the vehicle in front along with the road. The image on the right applied a threshold range of ±100. we need a relatively large range to accurately remove the entire road surface. in which a range of values ±1 the threshold values were applied.26 Figure 16 .

but there are available more optimized implementations of some core functions. . OpenCV is able to take advantage of these performance routines to greatly accelerate the execution of code. Since an intended target platform for this system is the Intel Atom [8] processor.Thresholding with range of ±35 In the figure above some of the detail in the bottom part of the vehicle in front of ours has been removed along with the road. OpenCV provided somewhat optimized routines for image processing. When deploying on an Intel processor. These are available in a commercial package called “Intel Performance Primitives” [7]. This package is a commercial product.4 Using Performance Primitives Performance is a major concern in this project. use of these performance primitives could greatly accelerate execution of the algorithm. which can be seen below. and for cost reasons its use was not explored during the course of the project. 5.27 A range of ±35 gives satisfactory results. This is not a problem as we will now apply a binary threshold to highlight any values that are above zero. Figure 18 .

which would only be reached by an actual object. averaging a small rectangle directly in front of the vehicle. we loop horizontally through the frame. We cannot simply check for the first non zero pixel value in the image. as shown above. results in very small levels of noise triggering object detection. Conversely. The figure below shows the effect of noise on distance detection.5 Level to Trigger Detection While carrying out distance detection. An appropriate value was found through measuring average values across the detection rectangle and discerning a threshold value from these measurements. The frame that we are scanning has been thresholded. Testing needed to be done to select a value to ‘trigger’ object detection. Figure 19 .Example of distance detection performed with small trigger value Using a very low value. but that does not definitely mean that all road pixels have successfully been removed. using a very high value leads to no object detection at all.28 5. and filter out any noise. There may be noise or artifacts left in the image. .

Primary among these is. There are several ways to ensure that memory used by an OpenCV program is kept in check. The following table illustrates common memory allocation methods and their equivalent de allocation methods.29 5. While processing a 30 second video. when allocating memory structures. one must be vigilant in de allocating it when finished. cvCreateImage cvCreateImageHeader cvCreateMat cvCreateMatND cvCreateData cvCreateSparseMat cvCreateMemStorage cvCreateGraphScanner cvOpenFileStorage cvAlloc cvReleaseImage cvReleaseImageHeader cvReleaseMat cvReleaseMatND cvReleaseData cvReleaseSparseMat cvReleaseMemStorage cvReleaseGraphScanner cvReleaseFileStorage cvFree Once these rules were observed strictly. Below is a table illustrating measurements of memory use for several sample videos. Memory is not in ample supply in embedded systems and therefore must be strictly monitored. after memory use was cut. . memory use became a big concern. Clearly this was unacceptable. During early testing of the finished algorithm.6 Memory Management Given that the majority of the project was completed using the C programming language. system memory use peaked at over 1GB. memory usage declined drastically. it was of paramount importance to de allocate any memory that was allocated. which does not include automatic memory management.

we can employ a more accurate method of calibration. There are several ways in which this can be done. by clicking the corners of this rectangle. When the system is installed in a vehicle. calibration for a particular environment needs to be carried out.7 Calibration In order to obtain accurate distances from the system. This method provides a somewhat rough value.30 Memory Usage 16 14 12 10 Memory (MB) 8 6 4 2 0 1 2 3 4 5 Figure 20 . which means that instead of the approximate method employed for testing in the current implementation. The figure below shows a scene in which a rectangular shape has been overlain. the position and angle of the camera will not change.Graph illustrating memory use for several video samples 5. Using simple mouse clicks. Due to the wide variety of samples from different environments being tested as part of this project. the transformation matrix for this environment can be found. this is the method that has been implemented. One such method of calibration is to calibrate the camera by placing a square object of known size in front of the camera on the road plane. . which will give more accurate distance values. but is satisfactory for testing purposes. the transformation matrix for that environment can be obtained.

the system detects the checkerboard pattern of known size [11]. Finally. Firstly. height above the ground. From this information the transformation matrix can be generated. . it is possible to perform automatic calibration of the camera using a checkerboard patterned square placed in front of the car.31 Figure 21 – Example of ‘known square’ method of calibration A second method that can be employed to calibrate the camera and generate the transformation matrix is through the use of the cameras intrinsic and extrinsic characteristics. and viewing angle can be used. A cameras focal length. This technique has not been implemented as part of this system. Exploration of this method of calibration is outside the scope of this project. This is the preferred option. providing simple and reliable calibration.

32 6 Results This section illustrates results obtained by the system as implemented. performance in a very important concern. It was found that the algorithm functioned as expected for most all sample videos. 6. . Since the goal for this algorithm is to implement it in real time on a video stream. Below are screenshots of several videos in which the algorithm functioned as expected. as a general rule. very resource intensive.Samples of successful distance determination Areas in which the algorithm did not perform as expected will be explored in the “Further Work” chapter. Naturally the easiest way to evaluate the effectiveness of a system is to test it on a variety of different conditions.1 Selection of Sampling Rate Computer vision algorithms are. Figure 22 . Insight is given into situations in which the algorithm is effective and where it can be improved.

To improve performance and lessen the load on the system. at different sampling rates. 30 times per second. It is not feasible to run the entire algorithm on each frame. Each frame is displayed for 33 milliseconds.33 A standard camera captures frames at a rate of 30 frames per second. .Plot of computation times for different sampling rates A sampling rate of 10 frames was chosen as a good tradeoff between accuracy and updating frequency. Below is a chart of computation times for the algorithm on a 386 frame video stream. we run the algorithm less than 30 times per second. Total Computation Time 25 20 15 Time (s) 10 5 0 0 2 4 6 8 Sampling Rate 10 12 14 16 Figure 23 . every x frames.

Firstly measurements were taken of the total time to perform all 3 of the major operations that must be repeatedly carried out.comparison between sampling rate of 1 and 10 6. Warping perspective to create top down view. • • • Thresholding image to remove road surface. namely.2 Performance of Algorithms Analysis was carried out on time taken to carry out each step of the system as described in the overall flowchart of the system. Distance determination. .34 Average Total Computation Time 25 20 15 Time 10 5 0 1 Sampling Rate 10 Figure 24 . the results of this analysis can be seen below. The chart below shows a comparison on the time taken by each step to process all frames of a 386 frame video.Measured computation time .

Timing was then carried out using the built in Linux command ‘time’ [9]. Below is an illustration of the percentage of total execution time taken up by each of the 3 major parts of the algorithm.8 1. Testing the algorithm on a dedicated microprocessor would give more quantitative bench marks. which measures real time taken as well as user and kernel time taken to execute programs.6 0. The results were corroborated with a second timing method.8 0. This figure is not inherently very useful to know. To generate these values. It is greatly affected by the presence of other processes running on a system. as it is relative to the processor on which the test is being carried out. using built in clock functionality in the C language [10]. .Comparison of computation times in seconds Carrying out all operations on each frame was found to take a total of 0.2 1 Time (s) 0.6 1.4 1.2 0 1 2 3 4 5 6 7 8 Distance Determination Thresholding Warp Perspective Figure 25 .053 seconds or 53 milliseconds. What can be inferred from the figures obtained are figures for the percentage of total time taken up by each part of the system.35 Processing Time in Seconds 1. the algorithm was modified to run only one of the 3 major operations listed above.4 0.

again as expected. . In this system. altering each pixel based on a rule.Graph of processing time for each of 3 major constituents of algorithm As can be seen from the figure above. resulting in significant processing time. this is done several times. warping perspective of the image in order to generate the top down view of the scene is by far the most time consuming part of the algorithm. The second most computationally expensive operation is thresholding of the image. Thresholding an image works on the whole image.36 Processing Time for Each Section as % 70% 60% 50% 40% 30% 20% 10% 0% Warp Perspective Thresholding Distance Determination Figure 26 . as predicted.

This figure can be improved upon in a number of ways.05 seconds of processor time. the horizon and the bonnet of the car are irrelevant and unimportant. as part of the thresholding operation. going from bright areas to dim areas. • Reduce number of channels in image to be transformed. only threshold the portion required by the algorithm. e.37 7 Further Work While the system is very successful in determining distances and detecting objects in front of a vehicle. In the system. Currently thresholding is applied to the whole image.. .g.g. A more efficient algorithm using fewer resources would improve overall processing time. to one.1. This slows down computation and increases the amount of memory used.1. • When thresholding image. e. • Change thresholding algorithm to use less memory. This aspect of the system could be improved using adaptive thresholding. Cropping these areas out will increase efficiency.1 Processing Time Currently each processed frame requires on average. 7. 0. this is not required as some parts of the image. it stands to be improved in several areas. several extra data structures are allocated and de allocated. and should provide a drastic increase in performance. • Implement tracking algorithm The sample rate could be further reduced from 3 times per second with the help of a tracking algorithm.2 Environmental Conditions The algorithm in its current form is quite susceptible to changes in environment. This will have the effect of reducing the computation required to transform the (3 channel) image being transformed to 1.. 7.

38 Secondly.1. which interferes with distance detection. the system detects road markings in the middle of the road as objects.3 Tracking Implementing tracking as part of the system would greatly improve the algorithm in several ways. 7. Manufacturer specific high performance C libraries such as the Intel Performance Primitives could be employed to greatly increase performance.4 Embedded Implementation It is very much hoped that the system will be ported to an embedded processor in the near future where it can be properly tested and benchmarked for use in an actual vehicle. A tracking algorithm could assist in situations where the algorithm has lost the object or has been compromised by noise conditions on the road. . there is little change in the location of the detected object from one frame to another.1. By the nature of the environment where the system operates. The system could be improved to intelligently filter out these markings and improve reliability of the algorithm 7.

and by performing pre crash safety procedures if an impending collision. makes it an attractive proposition for car manufacturers due to its low cost and simple configuration. For ‘normal’ road conditions the algorithm was found to function very well. This system offers benefits over similar active systems in terms of both cost and functionality.39 8 Conclusion As can be seen from the successful implementation of this algorithm in the C language. but indications are positive that this will be possible. in that its object detection is not solely limited to metal. . single camera. A real time. reflective objects. which can have many purposes in a single installation. a real time distance determination system using OpenCV is clearly achievable. alerting a user that they are not maintaining a safe following distance in relation to the car in front. This information could then be integrated into the vehicles operation in several ways. by alerting a user of imminent danger. OpenCV has proven a powerful and lightweight computer vision framework and greatly assisted in the development of the project. The system as it stands is functional and complete. All of these benefits combine to make a vehicle which implements this system a safer one which ought to lead to fewer road accidents and fewer injuries or fatalities. The use of a single optical camera. providing useful information to the user. passive distance determination algorithm as implemented here could have a positive effect on road safety and avoidance of road collisions. Refinements are needed before the system can be deployed with confidence to an actual embedded device.

Maud. 5. 2004. Samad et al. Gary Bradski. Learning OpenCV – Computer Vision with the OpenCV libraries. Implementation of Inverse Perspective Mapping Algorithm For The Development Of An Automatic Lane Tracking System 4.uk/content/unitedkingdom/mpc/mpc_unitedkingdom_website/en/home_mpc/passengercars/ho me/new_cars/models/cls-class/c219/overview/safety. Reilly Media.40 9 References 1. C. 2008.pdf) 2. O. Adrian Kaehler.com/library/cmd/blcmdl1_time. Hussain. .intel. H. Road Safety Authority – Road Collision Facts 2005 (http://www. 2009. Mallot et al. Applications of Linear Algebra: Computer Vision in Sports Intel Performance Primitives (http://software. 7.about. Timing in C (http://beige.edu/B673/node104.html) 11.indiana.com/en-us/intel-ipp/) Intel Atom processor (http://www. NUIG Guest Lecture. 8. 6.mercedesbenz.ie/publication/publication/upload/2005%20Road%20Collision%20Facts. Inverse perspective mapping simplifies optical flow computation and obstacle detection D. 9.rsa. 1991. Mercedes Pre Safe (http://www2.co.intel.ucs.html) 3. O Cualain. Lane Departure Detection Using Subtractive Clustering in the Hough Domain.htm) 10.com/technology/atom/) ‘time’ command (http://linux. Paul Smith.

On the CD Included on the submitted CD is the entirety of the Subversion repository of code developed throughout the course of the project.41 10 Appendix A . Some sample images and videos are included for testing purposes. The code is split into various folders with snippets to carry out different parts of the algorithm. . which incorporates many of the separate parts can be found in the ‘Final Implementation’ folder. The final implementation.