3d Machinevision Final Project Report

TOOL IDENTIFICATION SYSTEM
USING 3D MACHINE VISION

By,
Prasanna Iyer (101172980)
Tushar Vora (300120192)
Nirmal Dawda (101135054)
Moose Hijazi (101102624)
MECH5801F [34141] 3D Machine Vision (LEC) Fall 2019
Final Project
Submitted To,
Prof. Paul Harrison
Carleton University
Department of Mechanical and Aerospace Engineering
14/12/2019
1
Abstract
The project aim was to identify the object like hammer and screw driver automatically based on the
stereo images provided and to calculate the 3d depth of the objects. Its application is in space for robot to
automatically identify tools for maintenance purpose. In this report we have implemented the automatic
approach to solve the above mentioned task. Camera calibration was performed manually by selecting
similar points on both the stereo images. Object detection and object identification was performed by
blob analysis technique. The points in one stereo image was matched on the another image by local stereo
matching technique i.e Sum of square difference of intensities and the points were matched along the
epipolar line of the respective points. The depth of the object points was identified using stereo 3d
reconstruction technique and the pose and orientation of the tools were identified. The result obtained
from fully automatic process was almost as expected with some minor errors.
2
Acknowledgements
We would like to extend our sincere gratitude to professor Paul Harrison for the guidance he has
provided to us throughout this semester, in cementing our knowledge base in 3D Machine Vision.
This project would not be accomplished, if not for good teaching he has been passing to us on a
consistent basis.
We also want to thank the department of mechanical and Aerospace engineering for making courses
like wind engineering available in the university, giving us a chance of gaining anticipated
knowledge which would be crucial to us in the future.
Lastly, thanks to our classmates for being a constant source of motivation throughout the course
period; they bestowed confidence in us, and that confidence was crucial in driving us to complete
this project.
3
Table of Contents
Abstract .................................................................................................................................................................................................. 2
Acknowledgements ................................................................................................................................................................................ 3
Table of Contents................................................................................................................................................................................... 4
List of Tables.......................................................................................................................................................................................... 5
List of Figures ........................................................................................................................................................................................ 6
List of Appendices ................................................................................................................................................................................. 7
1 Chapter: Introduction – Stereo Images……………………………………………………………………………………………..8

1.1 Objective of Project ......................................................................................................................................................... 8
1.2 Stereo Images................................................................................................................................................................... 9
2 Chapter: Epipolar Geometry .................................................................................................................................................... …11
3 Chapter: Camera Calibration……………………………………………………………………………………………………..14
4 Chapter: Object Detection and Object Point Detection ………………………………………………………………………...18

4.1 Introduction .......................................................................................................................................................................
4.1.1 Various Object Detection Techniques
4.1.1.1 RGB to Binary Image
4.1.1.2 Edge Detection Technique

4.1.1.3 Template Matching
4.2 Morphological Operation ...................................................................................................................................................
4.2.1 Dilation
4.2.2 Erosion
5 Chapter: Introduction – Rectificaton & Stereo Matching………………………………………………………………………26

5.1 Rectification ......................................................................................................................................................................
5.2 Stereo Matching................................................................................................................................................................
5.3 Alternatice Method...........................................................................................................................................................
5.4 Stereo Matching along epipolar line: Alternative Method .............................................................................................
6 Chapter: Introduction – Rectificaton & Stereo Matching……………………………………………………………………….31

6.1 Introduction......................................................................................................................................................................
6.2 Method Used .....................................................................................................................................................................
7 Chapter: Pose Estimation………………………………….………………………………………………………………………..33
8 Chapter: Results……………………………………………………………………………………………………………………..35
9 Chapter: Discussions & Conclusions……………………………………………………………………………………………….38

References………………………………………………………………………………………………………………………………40
Appendices ...................................................................................................................................................................................... 41-96
T
a
b
l
e
o
f
4
List of Tables
Table 1- Projection matrix ofcamera1……………………………………………………….………..17

Table 2- Projection matrix of camera 2…………………………….………….………………………17
Table 3- Intrinsic matrix of camera 1………………………………………………………………….17
Table 4- Intrinsic matrix of camera 2…………………………………………………………………17
Table 5- Rotation matrix of camera 1………………………………..………………………………..17
Table 6- Rotation matrix of camera 2………………………………………..………………………..17
Table 7- Origin of camera 1…………………………………………..……………………………….17
Table 8- Origin of camera 2…………………………………..……………………………………….17
Table 9- Rotation matrix of Hammer……………………….…………………………………………35
Table 10- Rotation matrix of Screw Driver……………………………………………………………35
Table 11- Origin of Hammer……………………………..……………………………………………35
Table 12- Origin of Screw Driver……………...……………………………………………………..35

Table 13- Final result and comparison………..……………………………………………………….36
Table 14. Tick-Tock Runtime…..……………………………………………………………………..36
5
List of Figures
Figure 1. Comparison of Methodologies ......................................................................................................8

Figure 2. Original stereo-image 1 .................................................................................................................9
Figure 3. Original stereo-image 2 .................................................................................................................9
Figure 4. Epipolar Geometry ......................................................................................................................11
Figure 5. Selecting points using fzoom.......................................................................................................13
Figure 6. Generated epipolar lines (2) ........................................................................................................13
Figure 7. Generated epipolar lines ..............................................................................................................13
Figure 8. Perspective Projection .................................................................................................................14
Figure 9. Orientation of Calibrated cameras ...............................................................................................16
Figure 10. 2D Calibrated Camera (blue) and coins (red)............................................................................16
Figure 11. Grey Level thresholding ............................................................................................................18
Figure 12. Edge Detection Methods ...........................................................................................................19
Figure 13. Template Matching....................................................................................................................19
Figure 14. Modified Blob Analysis.............................................................................................................20
Figure 15. Plane Separation ........................................................................................................................20
Figure 16. Morphological Operation ..........................................................................................................21
Figure 17. Binary Morphology ...................................................................................................................21
Figure 18. Dilation of A by B ....................................................................................................................22
Figure 19. Binary erosion ...........................................................................................................................22
Figure 20. Erosion of A by B......................................................................................................................23
Figure 21. Implemented Morphology .........................................................................................................23
Figure 22. Morphological Code ..................................................................................................................23
Figure 23. Blob Separation using labels .....................................................................................................24
Figure 24. Prior to Area classifier ...............................................................................................................24
Figure 25. Result of Area classifier ............................................................................................................24
Figure 26. Screw located based on area ......................................................................................................25
Figure 27. Hammer located based on area ..................................................................................................25
Figure 28. Extreme point method ...............................................................................................................25
Figure 29. Automatically Identified objects using regionprops..................................................................25
Figure 30. Template specification code ......................................................................................................28
Figure 31. Template dimensions.................................................................................................................28
Figure 32. Screwdriver epipolar line 2........................................................................................................29
Figure 35. Hammer epipolar line 2 .............................................................................................................29
Figure 36. Hammer epipolar line 1 .............................................................................................................29
Figure 37. Hammer epipolar line ................................................................................................................29
Figure 38. Matched hammer point..............................................................................................................30
Figure 39. Anomalous Hammer point.........................................................................................................30
Figure 40. Reconstruction of the object ......................................................................................................31
Figure 41. Hammer Schematic....................................................................................................................33
Figure 42. Mathematics of Pose estimation ................................................................................................33
Figure 43. 3D Plot of cameras and tools in world co-ordinates..................................................................34
Figure 44. Succesfull automatic point selection .........................................................................................35
Figure 45. Run-time profile summary.........................................................................................................37
Figure 46. Excess shadow causing distortion in the Blob Analysis............................................................38
6
List of Appendices
Main Code
i. Final_3d_code
User Defined Functions
i. zoom_point
ii. plotting_epipolar_lines
iii. DecomposePpm
iv. blob_object_detection
v. epipolar_line_objectpoints
vi. stereo_matching
vii. reconstruction
viii. pose_and_orientation
ix. rectification_1
x. edgedetectionusinginbuiltfunction
xi. templatematchingusingmatlabinbuiltcode
7
Chapter 1: Introduction - Stereo Images
1.1 Objective of Project:-

In the modern era, the robotic systems have been workhorses from decades for industrial automation as well as space
exploration. In space identification of particular objects and their positions in space coordinates need to be considered.
Keeping this in mind, project is divided into two phases,
Phase 1: Constructing Rectified Images and Epipolar Geometry
Performing analysis on the given test stereo images as shown below, conducting several procedure as mention below in the
report. To obtained rectified images, concept of epipolar geometry is applied to obtained epipolar lines by selecting several
points in the image and rectifying that points by decomposing PPM with pixel points with reference to world coordinate
systems.
Phase 2: Object identification and 3D Reconstruction

After completing Phase 1, we need to locate particular object(hammer and screw-driver) in the image and plot its 3D point
with respect to world coordinates and camera coordinates. We have standard procedure to find pose estimation of the object
but due to longer computation time in the process we opted to merge stereo rectification and stereo matching together and
searched the points along epipolar lines as flowchart is mentioned below:
Figure 1. Comparison of Methodologies
8
1.2 Stereo Images:-
Stereo vision is motivated from the human vision system, which can perceive depth properties of an image. The human
vision system obtains two different images of an image from slightly different views from the human position and interprets
the images for depth perception. Because the human has more vision abilities in depth perception, stereo vision can be
applied in many applications, such as 3D scene reconstruction, object recognition, entertainment, robot navigation, and also
in space applications, etc. To obtain the depth information of a scene from a pair of images, it needs to solve one of the
inherent problems in stereo vision, called the correspondence problem. Because two different image planes are defined in
stereo vision, the projections of an object planes are represented w.r.t. different image coordinates. Solving the stereo
correspondence is called the stereo matching and results of stereo matching is represented by disparity map whose intensity
represents the coordinate difference between the two stereo rectified images ,called disparity.
In general, a stereo vision system consists of two identical vision cameras, whichcapturetheleftandrightimagesofanobject.
For our, illustration for a stereo system we have below images for stereo vision system:
Figure 2. Original stereo-image 1
Figure 3. Original stereo-image 2
9
To remove the computation complexity of the stereo matching problem, the stereo geometry of a stereo vision system
should be considered so that stereo matching is restricted to particular image portion. To understand the stereo geometry,
we need to perform calibration of a stereo vision w.r.t. to reference coordinate system is needed. A direct calibration
method is presented to obtain geometric relationship between two stereo cameras, which can be done by 3D Euclidean
transformations, rotation and translation. From the calibration, we come to know that stereo correspondence are related by a
2D point-to-line projection, called epipolar geometry.
The epipolar geometry between the stereo images provides a very strong constraint to find stereo correspondences. The
epipolar constraint is essential in stereo matching because an image point in one image has its conjugate point on an
epipolar line in the other image, while the epipolar line is derived by the original point. Therefore, computation complexity
and time of the stereo correspondences can be reduced. To get effective and fast stereo matching, we need to get epipolar
lines parallel to the horizontal image axis as the point’s lies on the same horizontal line. These whole process of making non
epipolar lines to parallel ones it is called stereo rectification. These process will transforms all epipolar lines in stereo
images parallel to the horizontal image axis.
After stereo rectification we switch on to the stereo matching part in which it is categorized in two methods in terms of
matching cost and energy aggregation, they are Local stereo matching and Global stereo matching. In Local stereo
matching techniques use image templates defined in both stereo images to measure their correlation. Common techniques
are SAD (sum of absolute difference), SSD (sum of squared difference), and NCC (normalized crosscorrelation). . The
global approaches produce more accurate results than the local methods. However, high computation time is needed.
Examples of global matching are Belief Propagation (BP), Semi Global Matching (SGM), Graph Cut (GC), and Cooperative
Region (CR), etc.
10
Chapter 2: Epipolar Geometry
Figure 4. Epipolar Geometry
Epipolar geometry is the geometry of stereo vision system. When two cameras view a 3D scene from two
different positions, there are a number of geometric relations between the 3D points and their projections
onto the 2D images that lead to constraints between the image points. The relation between the cameras are
approximated by the pinhole camera model.
The figure shows two pinhole cameras looking at point x. In real cameras, the image plane is actually behind
the focal center, produce an image that is symmetric about the focal center of the lens. We can simplify
problem by putting a virtual image plane in front of the focal center i.e. optical center of each camera lens to
produce an image not transformed by the symmetry. O 1 and O 2 represents the centres of the two cameras
lenses. W shows the point of interest in both cameras. Point’sp1 and p2 are the projected points in the image planes
of W. Each camera captures a 2D image of the 3D world. The conversion from 3D to 2D is defined as
perspective projection and is described by the pinhole camera model. These model helps in emanating rays from
the camera, passing through its focal center. It should be noted that each emanating rays corresponds to a single
point in the image.
e1 and e2are the epipolar points which lies on the 3D line of the each center projects onto a distinct point
into the other cameras image plane. The line O1–X is seen by the left camera as a point because it is directly
in line with that camera's optical center. However, the right camera sees this line as a line in its image plane.
The line (e2–x2) in the right camera is called an epipolar line. And the plane in which O1, O2, and X lies is
called epipolar plane.
Now to get epipolar lines we need to follow below steps as mentioned (all the coding is done in MatLab):
11
Plot both Stereo Images
Selecting point for both images

[u,v] = ginput(nPoints)
Define point vectors

p = [u, v, 1]
Centroids for the stereo images

C = mean (u(1:(npoints-1)))
To find normalization scale factor

S = (sqrt(2))*8/sf
Defining normalization matrices

norm = [s,0,-s*cx;0,s,-s*cy;0,0,1]
Normalization of points vectors and

Extracting the points
p_nor=p*(norm)'
Computing normalized y matrix

Y = zeros(npoints-1,9)
Obtain the SVD of the normalized Y

matrix
[U, S, V] = svd(Y)
Define normalized F based on the last

column of Vy
Compute the SVD of the normalized F

matrix
Zero the smallest singular value of F and

recomputed the normalized F matrix
Un-normalize the recomputed F matrix
Determine the normal vectors, a1 and a2,

for Images 1 and 2
Obtaining Epipolar Lines
12
Figure 5. Selecting points using fzoom
Figure 7. Generated epipolar lines Figure 6. Generated epipolar lines (2)
13
Chapter 3: Camera Calibration
It is used to determine the 3D Euclidean transformations between the stereo images of camera coordinates systems of
stereo vision system. In order to calibrate a stereo vision system, it is mandatory to define a reference coordinate
system, usually called as the world coordinate system, so that w.r.t. to reference coordinates system, camera coordinates
can be determined.
Figure 8. Perspective Projection
As shown in fig. , it is a Direct Camera Calibration. A 3D point W is projected on the image plane as point p, which is
the intersecting point of O and W. The mapping from 3D coordinates to 2D coordinates is called as perspective
projection matrix, which is represented by a linear transformation in homogeneous coordinates.
Let p = [u v 1], 1p1 = [u’ v’ 1]T and W = [x y z 1]T be the homogeneous coordinates of p, 1p1 and W, respectively.
Then the perspective transformation is given by the matrix M:
1p 1 = M wp
Here, M is the 3 x 4 perspective projection matrix, which can be decomposed, using the QR factorization, into the
product as,
M=KT
T represents a 3 x 4 matrix that depicts the extrinsic properties of Cameras (rotation and position).
T = [R12|t12] = T2T1-1
K is 3 x 3 matrix captures intrinsic camera properties (focal length and pixel scaling), upper right triangular, R is
orthonormal.
To calculate K, R, and O value, in MatLab code we have used function flipud for R to make orthonormal and then
making K and R diagonal components positive. To obtain origin offset we need to extract points from rotation matrix.
14
 Calibration of PPM:
Individual projections are represented as follows:
Atleast 6 points with known 3D world coordinates are mapped to 2D pixel locations in the image, the 12
components of the PPM can be established(i.e. calibrated).If there is any calibration pattern, like a
checkerboard pattern – M can be estimated through a least squares minimization technique. Assume there
are N points for a homogeneous linear system like equation; then linear equation can be given by
Am=0
With
And
Since the rank of A is 11, m (or M) can be recovered from an SVD-based technique as the column of V
corresponds to the smallest singular value of A, with A = UDV T.The column of V is a unit vector with a
length 12. It is to be noted that z point is divided out for defining u and v.
15
Calibrated camera points
As seen the figure blue marks represents camera world coordinates and red markers represents the points
selected manually for calibration i.e coins.
In the figure Camera 1 and 2 axes are plotted with respect to world coordinates. Red axis represents x axis of
camera, Green axis represents y axis of camera, and Blue axis represents z axis of camera.
Figure 9. Orientation of Calibrated cameras
Figure 10. 2D Calibrated Camera (blue) and coins (red)
16
17
Chapter 4: Object Detection and Object Point Detection
4.1 Introduction
For object detection and finding relevant regions in which object is lying can be done with few of methods
such as blob analysis, edge detection method and RGB Plane separation, etc. We have used blob analysis for
our object detection part and done hands on with other methods, but we did not get effective results.
Considering all the above technique we opted blob analysis for our project, we implemented RGB plane to
Binary image function, for removing shadows and other colours to depict objects precisely. For removal of
noise in such cases, a morphological operation is performed with dilation and erosion function which helps
in filling up holes in the objects and reduces noise level gradually.
4.1.1 Various Object Detection Techniques:
4.1.1.1 RGB to binary image –

Firstly, in the object identification RGB to Grayscale and Gray scale to binary(0 & 1) based on the
Threshold Gray value, the following steps are conducted and analysed the results, the noise amount in the
image is considerably high.
As observed in the graph of the Gray level thresholding, background and object needs to be separate based
on values of RGB to gray and intensity of the colours.
Figure 11. Grey Level thresholding
18
4.1.1.2 Edge Detection Technique
Other option was Edge Detection Technique, we applied Roberts, Sobel, Prewitt, Canny and Log methods as
images are shown below. But the results were not as expected, no proper edges can be found and makes it
harder to find particular object. In canny and log methods, images have more dots(noise) which is not
acceptable.
Figure 12. Edge Detection Methods
4.1.1.3 Template Matching –

One of the method is Template matching, in which a template with some pixels has been made, it is run
along the target image for the same energy cost function from the reference image. In these method, a
template of approx. 50 pixels is taken red dots are points selected on hammer.
During analysis, when the alignment of one object let’s say hammer template is equivalent to matching
points in the target image, then points are matched effectively. However, when objects is rotated from their
original position or alignment of that object is changed then resultant output will be different as only few
points will be matched in the target image(as shown in below image). Matching of the template was easy in
some points but it differs as the minimum cost function values changes in some pixels we gives an
unexpected results, which are not relevant for further stereo reconstruction and pose estimation. We have
results as mentioned below,
Figure 13. Template Matching
19
4.1.2 Final Method Modified Blob Analysis
Observing above failure methods, we have decided to use other technique which separate RGB plane and
gives less noise in the image. RGB Plane separation was one of the method, which proved quite useful in the
plane separation and also getting proper objects after applying and reversing planes based on threshold
values, as shown in below image
Figure 14. Modified Blob Analysis
From the above image, we have separated RGB planes and convertinginto binary plane. After, converting,
the images are as shown below, it was observed that because of the blue plane Blue plane is totally black, as
it is seen in the fig noise was occurring. So now blue is inverted and green plane is neglected. Merging red
plane and inverted blue plane gives best possible output as seen in
Figure 15. Plane Separation
20
4.2 Morphological Operation –
After combining red and inverted blue plane, the result is shown in the fig. In the fig, red circles are marked
which indicates holes beacause of this object is not properly identified. In order to perfectly identify object,
we need to fill these holes and reduce the noise. These can be achieved by morphological operation (dilation
and erosion).
Figure 16. Morphological Operation
4.2.1 Dilation
Well in morphological operation, there are two terms binary dilation and binary erosion. In binary dilation,
the holes are filled to get better image and it helps in filling holes with maximum value in the image i.e. 1
which makes object visible.
In binary morphology, dilation is shift invariant functions. A binary image is observed in mathematical
morphology as a subset of a Euclidean space. Let say E be a Euclidean space or an integer grid, A is a binary
image in E, and B is a structuring element regarded as a subset of R.
For Example, Suppose A is 11 x 11 matrix and B is 3 x 3 matrix
Figure 17. Binary Morphology
21
For each pixel of A that has a value of 1, superimpose B, with the center of B aligned with the corresponding
pixel in A. In dilation of A by B, each pixel is included in every superimposed B. The dilation of A by B is
given by,
Figure 18. Dilation of A by B
4.2.1 Erosion –
The basic idea in binary morphology is to probe an image with a pre-defined shape and drawing conclusions
on how it fits or misses the space in the image. This is called structuring element, and is itself binary image.
Let E be Euclidean space, binary image A in E. The erosion of A by structuring element is defined by B,
Where Bz is the translation of B by the vector z.

For Example, Suppose A is 11 x 11 matrix and B is 3 x 3 matrix
Figure 19. Binary erosion
22
Assuming that the origin B at its center, for each pixel in A superimpose the origin in B, if B is completely
contained by A the pixel is retained or it will be deleted. . Therefore, Erosion of A by B is given by this
13 x13 matrix.
Figure 20. Erosion of A by B
Seeing both morphological operations, we can construct image which is noise free and that can be observed
in the below image. In Matlab code(inbuilt function), we have performed this processed twice for better
image accuracy, can be seen below.
Figure 21. Implemented Morphology
Figure 22. Morphological Code
23
In these, we have a disk size of pixel say 10, has been moved along each pixels in the binary image to
remove the noise with morphological operations.
Imopen function is used in opening of grayscale or binary image with structuring element. The argument in
the structuring element must be single as opposed to an array of objects. The morphological open operation
is an erosion followed by a dilation.
Imclose function used in closing of grayscale or binary image returning the closed image. The morphological
close operation is a dilation followed by an erosion.
4.3 Labelling Objects –

Now objects are identify, so need to separate each object to get hammer and screw-driver.
In the given below image, individual images are obtained by analysing the area and region in which they are
lying. Inbuilt function bwlabel has been used to separate each object.
Function bwlabel returns the label matrix, that contains labels for the 8 connected points found in bw. Where
nearest 8 points with their edges or corners are considered for finding suitable points for particular objects.
To extract features from binary image using regionprops which is another inbuilt function, can be used
directly as it is connected with the bw label. In Matlab, both these are implemented to achieve desired
results.
Figure 23. Blob Separation using labels
Based upon the area of objects, each object is classified for example coins are having the least area and scale
contains the maximum area then hammer and screw driver.
Figure 25. Result of AreaFigureclassifier24. Prior to Area classifier
24
Figure 26. Screw located based on area Figure 27. Hammer located based on area
Points
Selection
Similarly, like object detection can be done using blob analysis, we can also use blob analysis in finding and
locating objects with extreme point method which defines point location and edges also. Selecting points on
the edge and corner for operations this method proved handy.
Figure 28. Extreme point method
As shown in the regionprops extrema property was used to find the automatic points and these function gives
8 points as shown in the fig. Hammer head and tip was identified using distance formula of the points.
Average points were selected for hammer head and tip, Hammer head two points and hammer tip were was
found by averaging, and similarly the process is applied to screw driver as shown in the image. And as
redcross (+) is the center of the points calculated. Mostly average points are calculated with distance formula
for more precise points. Similarly, points are taken for screw-driver.
Figure 29. Automatically Identified objects using regionprops
25
Chapter 5: Rectification and Stereo-Matching
5.1 Rectification
Rectification is a process that aids object matching by transforming images such that epipolar lines are
parallel with image rows. This makes it easier to search each image matrix for corresponding objects,
known as object detection.
Old PPM:
, = , , = , , − , = , − ,
New PPM:
, = , , = , , − , = , − ,
Kc,new can be defined based on the average intrinsic properties while Rc,new can be defined based on the
following:
New image X axis is parallel to the line connecting the two focal points (“baseline”)
New image Y axis is perpendicular to the baseline
New image Z axis should be close to original Z axes
New and old PPM are related as follows:
Points defined in the old pixel frame are now analogously transformed to the new frame. As the final step
in rectification, image interpolation needs to be performed to re-align the transformed image with integer
pixel locations.
However, the main ideology of this report is to combine rectification and stereo-matching into a single
process in the code. This was implemented after several attempts at rectification which required high
computational power to individually rectify point by point of the image, which also turns out to be time-
consuming. To counter these detrimental factors, SSD is implemented by the use of template matching
along epipolar lines. The key difference is the value of disparity parameter “d”. Details on this code can
be found in the appendix
26
5.2 Stereo Matching
Stereo matching is a technique to find corresponding points between stereo images captured from a pair of
cameras. In general, most investigations in stereo matching are categorized as one of two methods. One is
local matchingandtheotherisglobalmatching.
The local matching method finds corresponding points by comparing image properties only. If the
corresponding point in the left image is found similar in the right image, the centre point of the both images
template is considered to be matched. Therefore,inthelocalmethod,acostfunctionfor measuring the similarity
of two templates is designed to use their image properties, such as intensity, colour, sharpness, etc.
In the global matching techniques, the cost function is the primary element which measure the energy for a
correspondence matched point in terms of other image and continuity properties. Global Matching
Techniques helped a lot in the recent researches as the energy cost of pair of image points in terms not only
of image properties but also of disparity differences between neighbouring pixels.
Ingeneral,thedisparityvaluefpofa pixel p is determined in a way to minimize an energyfunction,
Where N is a set of neighbouring pixels and P is a set of pixels in an image. V(fp,fq) is the cost of assigning
disparity values fpand fqto two neighbouring pixels, called the discontinuity cost. The second term is the cost
of assigning disparity value fpto pixel p, which is the data cost. Thus, correspondence is defined by locating for
a point-pair that yields the smallest global energy.
As per above discussion, we have implemented Sum of Squared Difference method for stereo matching along
epipolar line. This method uses the intensity information around a pixel (x, y) in a reference image to find its
matching conjugate in a target image. It is one type of techniques in which area matching is done using a
template window. Then, this template window searches correspondence points in the target image with the help
of cost function which measures image difference with another template.
In the figure, two images, g (l) and g (r), are left and right stereo images, respectively. In the left image, an
image template at g(l) (x, y) is defined with the width and height of size m. By assuming that two images are
obtained from a parallel stereo vision system, epipolar lines in both images are on the same horizontal line.
Therefore, correspondence searching can be done in the horizontal epipolar lines. Now let us define a
template at g(r) (x + d, y) in the right image, where d is a disparity value. Then, the sum of squared
difference between two templates is defined as
27
5.3 Alternative Method
The alternate method proposed in this project involves combining stereo matching and rectification into a
single procedure. By considering only the epi-polar lines within the template, as opposed to searching all
epi-polar lines in the entire image, there is a discernible reduction in computational time. Stereo-matched
points are searched and located along those epi-polar lines falling within the templates shown below.
Figure 30. Template specification code
Figure 31. Template dimensions
Fig. shows the template dimensions taken into consideration while performing SSD. Template dimensions
are shown for a single point. Intensity of template was matched with all points along epipolar line. After
matching, minimum cost function is obtained and results are shown in the subsequent figures. This is the
purpose of the user defined function stereo-matching, which can be found in detail in the appendix. By using
the template, the need to search for matching points along the entire epipolar line is eliminated as only the
portion of the epipolar line within the template boundary box is under consideration. Using this rectification
+ stereo-matching combined method, computational time is reduced from the order of minutes to seconds.
28
Screwdriver
Figure 33. Screwdriver epipolar line 1 Figure 32. Screwdriver epipolar line 2
Figure 34. Screwdriver epipolar line 3
Hammer
Figure 36. Hammer epipolar line 1 Figure 35. Hammer epipolar line 2
Figure 37. Hammer epipolar line
29
The figures shown above are the images containing the epipolar lines passing through the matched points
that lie within the boundary box. From the three lines passing through each tool, a depiction is made, of 3
points on each tool, two on each side of the top portion of the tools and one to determine the centre-point of
the base. These 3 points are required to obtain the Pose Estimation.
5.3 Stereo-matching Along Epipolar Line: An alternate strategy
Figure 39. Anomalous Hammer point Figure 38. Matched hammer point
 From epipolar stereo-matched image it is clear that the screwdriver points have been matched
successfully while all but one of the hammer points have been matched.
 This final hammer point, as shown in the figure, is a discrepancy, and will not produce the correct
bounding box if used.
 To match the final hammer point, the minimum distance to the nearest stereo-matched point along the
epi-polar line is searched for and the corresponding distance is measured in the object-detected image.
 In this way a distinction between hammer left and right points can be made between a pair of matched
images. This is particularly useful to automatically determine which point represents which part of the
hammer.
 The adjoining figure displays the result of this method. Clearly the hammer points are now correctly
matched and the bounding box generated encapsulates the exact proportions of the hammer.
 Upon closer inspection, 3 red crosses can be observed, which are essential to the process of Pose and
orientation estimation in the following section
30
Chapter 6: Stereo Reconstruction
6.1 Introduction –
After obtaining rectified stereo images, a stereo matching technique can be applied to find correspondence in
the same horizontal lines, as we have performed with SSD method mentioned above. Once the disparity of
matching image points is determined, the coordinates of a 3D point associated to the matching pixels is
computed. Two simple methods are used for reconstruction,
a) Simple Triangulation
Because stereo images are rectified already, depth computation uses a simple equation for a parallel stereo
camera. When there is a disparity du in x direction of the pictures plane, the depth D to a 3D point from the
stereo camera is
Where B = || O1 – O2 || is the length of a baseline of the cameras. For the focal length f of the camera, the
intrinsic properties can be used.
b) Solution of a Linear Equation:
The range of pair of conjugate points is also reconstructed by the two pixel coordinates p1 and p2, the two
projection matrix M1 and M2, an over constrained linear system is written by,
AW = y
Where W gives the position of the 3D point projected to the conjugated points. Coordinates of W are
presented with respect to world coordinate system.
6.2 Method Used –
The two point vector should intersect at a point which is the 3d point , but in reality the two point vectors in
rectified planes doesn’t intersect .There is minor offset due to numerical precision, pixel offset, and many
more.
The average 3d point can be obtained by taking midpoint of the shortest distance (perpendicular to both
point vector lines) as shown in the figure
Figure 40. Reconstruction of the object
31
Calculation of 3d point:
 The midpoint of the two lines is related by the following expression having scale factor
k1,k2,and k
1- +2 = +1 +
2- ×
=
×
Where 1- wo2, wo1 is camera 1 and 2 world to camera coordinates

wR1, wR2 is camera 1 and 2 rotation matrix world to camera
‘j ‘vector is the cross product of two-point vectors which gives shortest vector between two lines
 The midpoint can be obtained by following linear equations:
1- scalar matrix ‘k’

≜ 1 − 2
2- ≜
3- Vector from origin of cameras
≜ −
4-
= −
5- = +1 +2 ‘wP’ is the final 3d point
The reconstruction of hammer and screw driver points is calculated using above method and it is shown
in the result table.
32
Chapter 7: Pose Estimation
Estimating the pose and orientation involves accurately describing the heights and spatial rotation matrixes
of the tools involved. This methodology is explained in the following section. The same schematic shown
for the hammer is followed for the screwdriver, only on a smaller scale.
Figure 41. Hammer Schematic
Figure 42. Mathematics of Pose estimation
Shown above is a sample mathematical diagram of how pose estimation is enforced. Taking the measured
perpendicular distance of the base to the line joining the two hammer tip points, the effective length of the
hammer can be reproduced. The hammer points are taken with the world co-ordinate frame of reference in
mind. Distance is mathematically calculated by using the relation described above. Error in length was found
to be less than 5% of the actual length.
To obtain the rotation matrix, the two vectors, in this case, AB and CD are taken as X and Y axis
respectively. The elements along the Z axis are therefore the cross product of both X and Y, i.e. the cross
product of AB and CD. Together these form the rotation matrix. As a result, the intersection between AB
and CD at a point, is taken as the origin of the hammer from which the orientation is described.
33
Figure 43. 3D Plot of cameras and tools in world co-ordinates
As shown above, a plot of the tools and respective camera frames in world co-ordinates is presented. The
colours RGB represent the XYZ axis frame respectively, of the individual objects present.
These pose and orientation results are discussed in detail in the results section.
34
Chapter 8: Results
This section highlights the results obtained from running the code presented in the appendix. The results
obtained are through automatic point selection. Possible discrepancies in the automatic selection were
verified with manual point selection wherever needed.
Figure 44. Successful automatic point selection
Object Hammer Object Screw-driver

0.86 -0.34 -0.37 0.64 0.33 -0.68
Rot. Matrix -0.14 -0.53 -0.83 Rot. Matrix 0.24 -0.94 -0.24
-0.48 0.77 -0.41 -0.72 -0.008 -0.69
Table 9 Rotation matrix of hammer Table 10 Rotation matrix of screw driver
Shown above is the Orientation and Rotation matrix of the Hammer and Screw driver wrt world co-
ordinates. The rotation matrix is represented as the unit vectors of the tool origin. Each row corresponds to
an axis of the hammer, defined in the pose estimation section. The last row represents the z axis, taken as the
cross product of x and y axis.
To check the accuracy of obtained unit vectors from the individual points, a table is prepared to compare the
results of the automatic point selection and the actual co-ordinates of the same points present in the world
co-ordinate frame.
Object Hammer Object Screw-driver

Center-point 186.1 566.98 275.95 Center-point 264.7 273.66 33.1
Table 11 Origin of HammerTable 12 Origin of screw driver
The points shown above are considered as the central points of the respective tools, from which the
orientation and rotation matrixes originate. They are the obtained by the intersection of two perpendicular
vectors as described in the Pose Estimation section.
35
Hammer Screw-driver
Base 136.92 383.68 -11.21 264.7 273.66 33.1
(Handle)
Original Base 120 380 0 290 290 20
Tip 186.1 566.98 275.95 291.72 165.81 5.88

Original Tip 130 620 230 290 160 5
Height 344 114

Original 332.565 130.865
Height
Table 13 Final Result and comparison
Shown above is the comparison between original data results and the result obtained from automatic point
selection. All measurements are in mm, implying a very minute error. Original heights were calculated
manually using a simple 3D distance formula between two arbitrary points. This calculation is compared to
the distance between the points obtained through automatic selection to generate a height for comparison.
From simply observing the table, it can be noted that although error exists, it is not very large despite the
inherent inaccuracies of the system model. By comparing the values with their original counterparts we can
obtain the differences in measurement and thereby find the accuracy of the system. As measurements are
being made on the mm scale, it is quite apparent that even the slightest deviation in obtaining points
automatically will generate a large difference in height value. So the obtained result is appreciable in that
respect.
Another important area within our results is computational time. Object detection systems in real-life must
be able to function at a high pace with equally high accuracy to enable the best results. Therefore an estimate
of running time and execution time becomes necessary. Verification of the run-time is performed within the
MATLAB environment, by using the tick-tock function and the ‘run and time’ script.
Table 14.Tick-Tock Runtime
36
Figure 45. Run-time profile summary
The run-times are clearly in the order of seconds. This is largely due to the alternate strategy of combining
the rectification and stereo-matching into one common step, as previously outlined. In the case of the
profiler, time taken to select the test points (i.e the coins) is also taken under consideration.
37
Chapter 9: Discussions and Conclusions
Phase 1:
Accuracy of the Epipolar lines depend on the F matrix or fundamental matrix, which depends on the points
selected. Accuracy greatly increases when points are selected with a zoom function. A better projection
matrix causes the extrinsic and intrinsic properties of the camera to be calculated more accurately, enabling a
better camera calibration. Sometimes an axis of the camera lies in the opposite direction of camera, which
can be manually corrected after observation.
Phase 2:
Object detection
Completing automatic object detection through the use of blob analysis, is much more viable provided that
the image contains a lesser number of objects and the background colour is uniform to an extent. If the
shadows are greater, then accuracy of blob decreases and alternate algorithms must be used. e.g CNN and
Machine Learning.
Figure 46. Excess shadow causing distortion in the Blob Analysis
Rectification
Traditional rectification even on modern PCs would be expected to have a computing time in the range of a
few minutes, whereas in the code referenced in the appendix, run-time is exponentially reduced. This is why
rectification is generally done on the required section of the image only. The alternate method implemented
here, was to instead apply stereo-matching along the epipolar line.
Stereo-matching and Pose estimation
Technique of local matching is used, which is however not the most accurate method as seen with the
bounding boxes. Further accuracy in matching can be achieved by using semi global matching techniques
like Speckle imaging, space time- stereo etc. The resultant estimation of pose is highly dependent on the
initial test points as well as the accuracy of stereo matched points. Matching and identification is also
dependent on the amount of noise that can be filtered out of the image under consideration.
Reconstruction
It is observed that accuracy of reconstruction increases when selection of points is highly accurate.
Reconstructed 3D point cloud can also be generated.
38
Conclusion:
In this project, a system to automatically identify tools present in an image was realized, with suitable
accuracy in results. The orientation and pose of the tools were also found, tabulated, and compared to the
real-world values in order to determine the accuracy of the system.
39
References
1. MECH 5801 LEC 2 PPT - Triangulation and Stereo Imaging, Paul Harrison
2. Handbook of 3D Machine Vision Optical Metrology And Imaging, Song Zhang
3. https://www.mathworks.com/help/images/ref/bwlabel.html#d117e25274
4. https://en.wikipedia.org/wiki/Dilation_(morphology)
5. Camera and stereo, Linda Shapiro
6. The Epipolar Geometry Toolbox for Matlab, Gian Luca Mariottini, Eleonora and Domenico
Prattichizzo, University of Sienna, July 2004
7. Epipolar Geometry and Stereo Vision, Derek Hoiem, University of Illinois
8. A Compact Algorithm for Rectification of Stereo Pairs, Andrea Fusiello, Emanuele, Alessandro
Verri, January 1999
9. Tutorial on Rectification of Stereo Images, Andrea Fusiello
10. https://www.google.co.in/url?sa=i&url=https%3A%2F%2Fwww.slideshare.net%2Ftawosetimothy%
2Fimage-segmentation-34430371&psig=AOvVaw02FwPAtQDHhJqXs2y-
e6LB&ust=1576418203870000&source=images&cd=vfe&ved=0CAIQjRxqFwoTCPCKgdOlteYCF
QAAAAAdAAAAABAD
11. https://www.qc.edu.hk/math/Advanced%20Level/Point_to_line.htm
40
12/14/2019 Final_3d_code
Contents
Clearing and closing everything
Reading Images
Calculating Fundamental matrix and plotiing epipolar lines
Calibrating the camera
Automatic Object Detection and object points detection Image one and two
Epipolar lines of object points selected automatically for stereo matching
Stereo matching along epipolar line to find point in image 1 in to image 2
Rectification process/parameters
3D Reconstruction of points in image 1 and 2
Pose estimation and orientaion
Clearing and closing everything
clear all
close all
Reading Images
image_Matrix1 = imread ('image5','jpg'); % Image 1 reading

image_Matrix2 = imread ('image4','jpg'); % Image 2 reading
gray_mat1=rgb2gray(image_Matrix1);gray_doub1=double(gray_mat1); % Converting image 1 into graysacle
gray_mat2=rgb2gray(image_Matrix2);gray_doub2=double(gray_mat2); % Converting image 2 into graysacle
Calculating Fundamental matrix and plotiing epipolar lines
% Image 1 and 2 selecting test points npoints=9;
% Number of points to be selected

[u1,v1]=zoom_point(image_Matrix1); % Calling zoom_point function for selecting points with zooming in image 1
% Define the point vectors
tic % calculating run time

x = ones(npoints,1); % column vector with values one
p1=[u1,v1,x]; % point vectors 1
p2=[u2,v2,x]; % Point vectors 2
% Calculating the centroids of the test points for Image 1 and 2
cx1=mean(u1(1:(npoints-1))); cy1=mean(v1(1:(npoints-1)));% Image 1 centroid point cordinates

cx2 = mean(u2(1:(npoints-1)));cy2 = mean(v2(1:(npoints-1)));% Image 2 centroid point cordinates
% Calculating the normalization scale factors for Image 1 and 2
sf1=0;sf2=0;
for i=1:(npoints-1) a=((u1(i)-
cx1)^2+(v1(i)-cy1)^2)^(0.5);
sf1=sf1+a; b=((u2(i)-cx2)^2+(v2(i)-
cy2)^2)^(0.5); sf2=sf2+b;
end
s1=(sqrt(2))*8/sf1; % scale factor of image 1

s2=(sqrt(2))*8/sf2; % scale factor of image 2
% Normalization matrices
norm_1=[s1,0,-s1*cx1;0,s1,-s1*cy1;0,0,1];
norm_2=[s2,0,-s2*cx2;0,s2,-s2*cy2;0,0,1];
% Normalizing the point vectors and extracting its coordinates
p1_nor=p1*(norm_1)';p2_nor=p2*(norm_2)';
% Computing the normalized Y matrix
Yn=zeros(npoints-1,9);
for i=1:(npoints-1)
Yn(i,:)= [p1_nor(i,1)*p2_nor(i,1),p1_nor(i,1)*p2_nor(i,2),p1_nor(i,1)*p2_nor(i,3),p1_nor(i,2)*p2_nor(i,1), ...
p1_nor(i,2)*p2_nor(i,2),p1_nor(i,2)*p2_nor(i,3),p1_nor(i,3)*p2_nor(i,1),p1_nor(i,3)*p2_nor(i,2),p1_nor(i,3)*p2_nor(i,3)];
end
% Solving homogenous equation by SVD method to get F matrix elements
[Uy,Sy,Vy] = svd(Yn);
% Defining normalized F based on the last column of Vy
41
file:///G:/Matlab cousre/3d machine vision/html/Final_3d_code.html 1/5
42
F_no=Vy(:,end);
F_nor=[F_no(1),F_no(2),F_no(3);F_no(4),F_no(5),F_no(6);F_no(7),F_no(8),F_no(9)];
% Computing the SVD of the normalized F matrix
[Uf,Sf,Vf] = svd(F_nor);
% Zero the smallest singular value (Sf) of F and recomputed the normalized F matrix
Sf(3,3)=0;
F_nor_rec=Uf*Sf*Vf';
% Un-normalizing the recomputed F matrix
F_unorm=(norm_1)'* F_nor_rec * norm_2;
% Determine the normal vectors, a1 and a2, for Images 1 and 2
a1=(F_unorm*(p2)')';a2=(F_unorm'*(p1)')'; at=(F_unorm'*[1490,2010,1]')';
% plotting epipolar lines
[x1,y1] = plotting_epipolar_lines( image_Matrix1,npoints,u1,v1,a1);% Calling fuction to plot epipolar line of image 1

[x2,y2] = plotting_epipolar_lines( image_Matrix2,npoints,u2,v2,a2);% Calling fuction to plot epipolar line of image 2
[x3,y3] = plotting_epipolar_lines( image_Matrix2,1,1490,2010,at);% image 2 test point epipolar line
Error using ginput (line 84)

Interrupted by figure deletion
Error in zoom_point (line 11)

[x,y,b] = ginput(1);
Error in Final_3d_code (line 18)

Calibrating the camera
% Known world origin
% Lab Test images world cordinate

%points=[0,0,0;45,214,0;94,461,0;168,134,0;234,364,0;485,160,240;570,270,240;670,240,240;680,330,240];
% Final images world cordinate points=[0,0,0;47,429,0;245,346,0;350,206,0;106,112,0;228,- 140,60;112,-

175,60;84,-290,100;220,-360,100]; x_points=points(:,1);y_points=points(:,2);z_points=points(:,3);
a1=zeros(npoints-1,12);b1=zeros(npoints-1,12);a2=zeros(npoints- 1,12);b2=zeros(npoints- 1,12);
A1=[];A2=[];
for i=1:(npoints-1) a1(i,:)=[x_points(i),y_points(i),z_points(i),1,0,0,0,0,(- (u1(i))).*x_points(i),(-(u1(i))).*y_points(i),(-
(u1(i))).*z_points(i),(-(u1(i)))]; b1(i,:)=[0,0,0,0,x_points(i),y_points(i),z_points(i),1,(- (v1(i))).*x_points(i),(-(v1(i))).*y_points(i),(-
(v1(i))).*z_points(i),(-(v1(i)))];
a2(i,:)=[x_points(i),y_points(i),z_points(i),1,0,0,0,0,(- (u2(i))).*x_points(i),(-(u2(i))).*y_points(i),(-(u2(i))).*z_points(i),(-(u2(i)))];
b2(i,:)=[0,0,0,0,x_points(i),y_points(i),z_points(i),1,(-(v2(i))).*x_points(i),(-(v2(i))).*y_points(i),(- (v2(i))).*z_points(i),(-(v2(i)))];
B1=[ a1(i,:); b1(i,:)];B2=[ a2(i,:); b2(i,:)];
A1=[A1;B1];A2=[A2;B2];
end
% solving A matrix to get values of m by SVD
[Ua1,Sa1,Va1] = svd(A1);[Ua2,Sa2,Va2] = svd(A2);
% M (intrinsic and extrinsic properties) elements and matrix
M1= vec2mat(Va1(:,end),4);M2= vec2mat(Va2(:,end),4);
% 8 th coin as a test coin to check M matrix
tp=[points(1,:),1]';
x=M1*tp;y=M2*tp;
x=x./x(3);y=y./y(3);
% Decomposing PPM to obtain intrinsic matrix,Rotaion and origin of cameras
[ K1 , R1 , o1 ] = DecomposePpm(M1); % Calling Decompose function for Camera 1

[ K2 , R2 , o2 ] = DecomposePpm(M2); % Calling Decompose function for Camera 2
% Plotting 3d world cordinates of origin and the points selected
world_ax=[0,0,0;100,0,0;0,100,0;0,0,100];
figure();
hold on;
plot3(x_points(1:npoints),y_points(1:npoints),z_points(1:npoints),'+','markersize',15,'linewidth',2,'color','r');
plot3(o1(1),o1(2),o1(3),'+','markersize',15,'linewidth',2,'color','b');
plot3(o2(1),o2(2),o2(3),'+','markersize',15,'linewidth',2,'color','b');
% plotting the axes pf the camera origin in world cordinates
43
44
figure()
hold on
grid on
title('Camera 1 and 2')
xlabel('x')
ylabel('y')
zlabel('z')
quiver3(o1(1),o1(2),o1(3),R1(1,1),R1(2,1),R1(3,1),100,'r') % camera 1 x -axis

quiver3(o1(1),o1(2),o1(3),R1(1,2),R1(2,2),R1(3,2),100,'g') % camera 2 y -axis
quiver3(o1(1),o1(2),o1(3),R1(1,3),R1(2,3),R1(3,3),100,'b') % camera 3 z -axis
quiver3(o2(1),o2(2),o2(3),R2(1,1),R2(2,1),R2(3,1),100,'r') % camera 1 x -axis

quiver3(o2(1),o2(2),o2(3),R2(1,2),R2(2,2),R2(3,2),100,'g') % camera 2 y -axis
quiver3(o2(1),o2(2),o2(3),R2(1,3),R2(2,3),R2(3,3),100,'b') % camera 3 z -axis
hold off
Automatic Object Detection and object points detection Image one and two
thresh_2=0.08;d_size_2=5;sd_high_2=80000;
thresh_1=0.1;d_size_1=10;sd_high_1=70000;
% Calling blob_object_detection for Automatic object and object points detection for image 1
[sdbboxes1,hmbboxes1,hasm_top1,hasm_bottom1,scs_top1,scs_bottom1] = blob_object_detection(image_Matrix1,thresh_1,d_size_1,sd_high_1);
% Calling blob_object_detection for Automatic object and object points detection for image 2
[sdbboxes2,hmbboxes2,hasm_top2,hasm_bottom2,scs_top2,scs_bottom2] = blob_object_detection(image_Matrix2,thresh_2,d_size_2,sd_high_2);
Epipolar lines of object points selected automatically for stereo matching
% Epipolar line for the hammer points
hm_points1=[hasm_top1;hasm_bottom1]; hm_points2=[hasm_top2;hasm_bottom2]; % hammer top and bottom points of image 1

hm_ones= ones(size( hm_points1,1),1);
hammer_points_1=[hm_points1,hm_ones]; hammer_points_2=[hm_points2,hm_ones]; % Hammer top and bottom points vector of image 1
hammer_epi2=(F_unorm'*(hammer_points_1)')'; % Determining the normal vector of hammer points by multiplying with fundamental matrix
% Calling epipolar_line_objectpoints function for plotting epipolar line in image 2 of hammer top/head point 1
[hx21,hy21] = epipolar_line_objectpoints(image_Matrix2,hmbboxes2(1),hmbboxes2(3),hammer_epi2(1,:));
% Calling epipolar_line_objectpoints function for plotting epipolar line in image 2 of hammer top/head point 2
% Calling epipolar_line_objectpoints function for plotting epipolar line in image 2 of hammer bottom/tip point
% Epipolar line for the screw driver points
sd_points1=[scs_top1;scs_bottom1]; sd_points2=[scs_top2;scs_bottom2]; % Screw Driver top and bottom points of image 1

sd_ones= ones(size( sd_points1,1),1);
screw_points_1=[sd_points1,sd_ones]; screw_points_2=[sd_points2,sd_ones]; % Screw Driver top and bottom points vector of image 1
screw_epi2=(F_unorm'*(screw_points_1)')'; % Determining the normal vector of Screw Driver points by multiplying with fundamental matrix
% Calling epipolar_line_objectpoints function for plotting epipolar line in image 2 of Screw Driver top/head point 1
[sx21,sy21] = epipolar_line_objectpoints(image_Matrix2,sdbboxes2(1),sdbboxes2(3),screw_epi2(1,:));
% Calling epipolar_line_objectpoints function for plotting epipolar line in image 2 of Screw Driver top/head point 2
% Calling epipolar_line_objectpoints function for plotting epipolar line in image 2 of Screw Driver bottom/tip point 1
Stereo matching along epipolar line to find point in image 1 in to image 2
template_width=10; % Template size around the point selected for matching
% Hammer points template of image 1 for matching in image 2
hm_points1_temp_1 =gray_doub1((floor(hm_points1(1,2)-template_width):floor(hm_points1(1,2)+template_width)),...
(floor(hm_points1(1,1)-template_width):floor(hm_points1(1,1)+template_width)));
hm_points1_temp_2 =gray_doub1(floor(hm_points1(2,2)+50-template_width):floor(hm_points1(2,2)+50+template_width),...
floor(hm_points1(2,1)+50-template_width):floor(hm_points1(2,1)+50+template_width));
hm_points1_temp_3 =gray_doub1(floor(hm_points1(3,2)-template_width):floor(hm_points1(3,2)+template_width),...
floor(hm_points1(3,1)-template_width):floor(hm_points1(3,1)+template_width));
% Screw Driver points template of image 1 for matching in image 2
sd_points1_temp_1 =gray_doub1((floor(sd_points1(1,2)-template_width):floor(sd_points1(1,2)+template_width)),...
(floor(sd_points1(1,1)-template_width):floor(sd_points1(1,1)+template_width)));
sd_points1_temp_2 =gray_doub1(floor(sd_points1(2,2)- template_width):floor(sd_points1(2,2)+template_width),...
45
46
floor(sd_points1(2,1)-template_width):floor(sd_points1(2,1)+template_width));
sd_points1_temp_3 =gray_doub1(floor(sd_points1(3,2)-template_width):floor(sd_points1(3,2)+template_width),...
floor(sd_points1(3,1)-template_width):floor(sd_points1(3,1)+template_width));
% Stereo matching the points
% Hammer head and tip points stereo matching
hm1_st_x2=stereo_matching(gray_doub2,hx21,hy21,hmbboxes2,hm_points1_temp_1,template_width); % Calling stereo_matching function to match hammer head point 1

hm2_st_x2=stereo_matching(gray_doub2,hx22,hy22,hmbboxes2,hm_points1_temp_2,template_width); % Calling stereo_matching function to match hammer head point 2
hm3_st_x2=stereo_matching(gray_doub2,hx23,hy23,hmbboxes2,hm_points1_temp_3,template_width) ; % Calling stereo_matching function to match hammer Tip point 1
% Screw Driver head and tip points stereo matching
sd1_st_x2=stereo_matching(gray_doub2,sx21,sy21,sdbboxes2,sd_points1_temp_1,template_width); % Calling stereo_matching function to match Screw Driver head point 1

sd2_st_x2=stereo_matching(gray_doub2,sx22,sy22,sdbboxes2,sd_points1_temp_2,template_width); % Calling stereo_matching function to match Screw Driver head point 1
sd3_st_x2=stereo_matching(gray_doub2,sx23,sy23,sdbboxes2,sd_points1_temp_3,template_width); % Calling stereo_matching function to match Screw Driver tip point 1
% Plotting Stereo matched points of image 2
figure()
imagesc(image_Matrix2)
hold on
plot(hx21(hm1_st_x2),hy21(hm1_st_x2),'+','markersize',15,'linewidth',2,'color','r');
plot(sx21(sd1_st_x2),sy21(sd1_st_x2),'+','markersize',15,'linewidth',2,'color','r');
hold off
Rectification process/parameters
to = (o2-o1);% Distance between two optical centre of camera

r1=to/norm(to); % New 'x' axis of image plane
r21=cross(R1(3,:)',to);r22=cross(R2(3,:)',to);% New 'y' axis rotation component
r31=cross(r1,r21);r32=cross(r1,r22);% New 'z' axis rotation component
R1_new=[r1';r21'/norm(r21);r31'/norm(r31)];R2_new=[r1';r22'/norm(r22);r32'/norm(r32)];% New rotation matrix of camera 1 and 2
K_new=(K1+K2)/2;% New intrinsic property will be mean of the K1,and K2

Q1_old=K1*R1;Q2_old=K2*R2; % Old Q
Q1_new=K_new*R1_new;Q2_new=K_new*R2_new; % New Q
T1=Q1_new*(Q1_old)^(-1);T2=Q2_new*(Q2_old)^(-1); % Transformation matrix of camera 1 and 2 in to rectified plane
%[rectifiedleft, rectifiedright] = rectifyimage(image_Matrix1, image_Matrix2, T1, T2);
3D Reconstruction of points in image 1 and 2
% 3d world cordinarte of manually selected points in both images 1 and 2
P1_new_1=(T1*p1')';P2_new_2=(T2*p2')';
P_world = reconstruction(o1,to,P1_new_1,P2_new_2 ,R1_new,R2_new,K_new);
% 3d world cordinarte of Automatically selected hammer points in both images 1 and 2
hammer_points_2_ste=[hx21(hm1_st_x2),hy21(hm1_st_x2),1;hx22(hm2_st_x2),hy22(hm2_st_x2),1;hx23(hm3_st_x2),hy23(hm3_st_x2),1];
hammer_points_11=(T1*hammer_points_1')';hammer_points_22_ste=(T2*hammer_points_2_ste')';
hammer_points_22=(T2*hammer_points_2')';
% Finding Correct hammer points of first image into other image either automatically detected point or stereo matched points based on the Euclidesn distance
hd11=pdist([hx21(hm1_st_x2),hy21(hm1_st_x2);hm_points2(1,:)],'euclidean');
hd12=pdist([hx21(hm2_st_x2),hy21(hm2_st_x2); hm_points2(2,:)],'euclidean');
P_world_1 = reconstruction(o1,to,hammer_points_11,hammer_points_22_ste ,R1_new,R2_new,K_new);
if hd11<hd12
Ph_world_2 = reconstruction(o1,to,hammer_points_11,hammer_points_22 ,R1_new,R2_new,K_new);

else
hammer_points_22_new=[hammer_points_22(2,:);hammer_points_22(1,:);hammer_points_22(3,:)];
Ph_world_2 = reconstruction(o1,to,hammer_points_11,hammer_points_22_new ,R1_new,R2_new,K_new);
end
Ph_world_2(1,1)= Ph_world_2(1,1)*(-1);Ph_world_2(1,2)= Ph_world_2(1,2)*(-1);
% 3d world cordinarte of Automatically selected screw Driver points in both images 1 and 2
screw_points_2_ste=[sx21(sd1_st_x2),sy21(sd1_st_x2),1;sx22(sd2_st_x2),sy22(sd2_st_x2),1;sx23(sd3_st_x2),sy23(sd3_st_x2),1];
screw_points_11=(T1*screw_points_1')';screw_points_22_ste=(T2*screw_points_2_ste')';
screw_points_22=(T2*screw_points_2')';
Ps_world_1 = reconstruction(o1,to,screw_points_11,screw_points_22_ste ,R1_new,R2_new,K_new); % Reconstruction of stereo matched points
Ps_world_2 = reconstruction(o1,to,screw_points_11,screw_points_22 ,R1_new,R2_new,K_new); % Reconstruction of automatic points selected
47
48
p111=(T1*[365,906,1]')';p222=(T2*[554,1762,1]')'; % Reconstruction of manually selected points
Ps_world_4=reconstruction(o1,to,p111,p222 ,R1_new,R2_new,K_new);
Pose estimation and orientaion
% HAmmer
% calling pose_and_orientation function for finding leght of hammer and rotation matrix
[ HD,H_DA,H_DC,H_ZD,H_dis ] = pose_and_orientation( Ph_world_2(:,1)',Ph_world_2(:,2)',Ph_world_2(:,3)' );
Rot_ham=[H_DA;H_DC;H_ZD]; % Rotation matrix of hammer in terms of wolrd to hammer
% Screw Driver
% calling pose_and_orientation function for finding leght of Screw driver and rotation matrix
[ SD,S_DA,S_DC,S_ZD,S_dis ] = pose_and_orientation( Ps_world_1(:,1)',Ps_world_1(:,2)',Ps_world_1(:,3)' );
Rot_scw=[S_DA;S_DC;S_ZD]; % Rotation matrix of Screw driver in terms of wolrd to Screw driver
% Plotting the Axis of Camera 1,Camera 2,Hammer and the screw Driver with respect to world.
figure()
hold on
grid on
title('Camera 1')
xlabel('x')
ylabel('y')
zlabel('z')
% Hammer x ,y and z axis respectively
quiver3(HD(1),HD(2),HD(3),Rot_ham(1,1),Rot_ham(2,1),Rot_ham(3,1),100,' r ' )
quiver3(HD(1),HD(2),HD(3),Rot_ham(1,2),Rot_ham(2,2),Rot_ham(3,2),100,'g')
quiver3(HD(1),HD(2),HD(3),Rot_ham(1,3),Rot_ham(2,3),Rot_ham(3,3),100,'b ')
% Screw Driver x,y, and z axis respectively
quiver3(SD(1),SD(2),SD(3),Rot_scw(1,1),Rot_scw(2,1),Rot_scw(3,1),100,'r')
quiver3(SD(1),SD(2),SD(3),Rot_scw(1,2),Rot_scw(2,2),Rot_scw(3,2),100,'g')
quiver3(SD(1),SD(2),SD(3),Rot_scw(1,3),Rot_scw(2,3),Rot_scw(3,3),100,'b')
% Camera 1 x,y, and z axis respectively
quiver3(o1(1),o1(2),o1(3),R1(1,1),R1(2,1),R1(3,1),100,'r')
quiver3(o1(1),o1(2),o1(3),R1(1,2),R1(2,2),R1(3,2),100,'g')
quiver3(o1(1),o1(2),o1(3),R1(1,3),R1(2,3),R1(3,3),100,'b')
% Camera 2 x,y, and z axis respectively
quiver3(o2(1),o2(2),o2(3),R2(1,1),R2(2,1),R2(3,1),100,'r')
quiver3(o2(1),o2(2),o2(3),R2(1,2),R2(2,2),R2(3,2),100,'g')
quiver3(o2(1),o2(2),o2(3),R2(1,3),R2(2,3),R2(3,3),100,'b')
hold off
toc
run_time=toc;
Published with MATLAB® R2013a
49
50
12/13/2019 Zoom_ point function
Zoom_ point function
% This function takes image as an input and provide x and y cordinate of...
... points selected in the u and v vector.
function [ u,v ] = zoom_point( image )

imagesc(image)
axis('equal')
u = []; v = [];
while 0<1
[x,y,b] = ginput(1);
if isempty(b)
break;
elseif b==91
ax = axis; width=ax(2)-ax(1); height=ax(4)-ax(3);
axis([x-width/2 x+width/2 y-height/2 y+height/2]);
zoom(1/2);
elseif b==93
ax = axis; width=ax(2)-ax(1); height=ax(4)-ax(3);
axis([x-width/2 x+width/2 y-height/2 y+height/2]);
zoom(2);
else
u=[u;x];
v=[v;y];
end
end
end
Error using zoom_point (line 7)

Not enough input arguments.
51
file:///G:/Matlab cousre/3d machine vision/html/zoom_point.html 1/2
52
12/13/2019 Zoom_ point function
53
file:///G:/Matlab cousre/3d machine vision/html/zoom_point.html 2/2
54
12/13/2019 plotting_epipolar_lines
plotting_epipolar_lines
% This function takes image ,number of points and normal vector ontained from the fundametal matrix as an input and provides...
%... the x and y cordinate of epipolar line as an output.
% This fucntion plots the epipolar line
function [x_ax,y_ax] = plotting_epipolar_lines( image,npoints,u,v,a)
figure();
imagesc(image)
axis('equal')
axis([0 6000 0 4000])
hold on;
%plot(u(1:(npoints-1)),v(1:(npoints-1)),'+','markersize',15,'linewidth',2,'color','r');% Ploting non test points
%plot(u((npoints-1):npoints),v((npoints-1):npoints),'+','markersize',15,'linewidth',2,'color','g');%Ploting test points
for i=1:npoints
x_ax=0:size(image,2);
y_ax=(-(a(i,1)/a(i,2)))*x_ax +(-(a(i,3)/a(i,2)));
plot(x_ax,y_ax,'color',[i/10 i/20 i/30]);

xlim auto;
ylim auto;
end
end
Error using plotting_epipolar_lines (line 11)

55
file:///G:/Matlab cousre/3d machine vision/html/plotting_epipolar_lines.html 1/2
56
12/13/2019 plotting_epipolar_lines
57
file:///G:/Matlab cousre/3d machine vision/html/plotting_epipolar_lines.html 2/2
58
12/13/2019 DecomposePPm function
DecomposePPm function
This fucntion takes projection matrix as an input and provides intrinsic...
%... matrix, rotation matrix and origin of camera as an output
function [ K , R , o ] = DecomposePpm(M)
% Computes the intrinsic matrix (K), rotation matrix (R), and origin offset (o) for the given perspective projection matrix (M)
[Temp1,Temp2] = qr(flipud(M(:,1:3))');
% Calculate the intrinsic matrix (right-upper-rectangular)

%K = fliplr( flipud( Temp2') );
K = rot90( Temp2',2 );
% Calculate the rotation matrix (orthonormal)
R = flipud( Temp1');
% Define K and R such that all diagonal components of K are positive
D = diag(sign(diag(K)));
K = K * D;
R = D * R;
% Use the rotation matrix to extract the origin offset
o = -R'* K^(-1) * M(:,4);

end
Error using DecomposePpm (line 9)

59
file:///G:/Matlab cousre/3d machine vision/html/DecomposePpm.html 1/2
60
12/13/2019 DecomposePPm function
61
file:///G:/Matlab cousre/3d machine vision/html/DecomposePpm.html 2/2
62
12/13/2019 blob_object_detection
Contents
blob_object_detection funtion
Reading image
Segmenting Red,Blue,Green plane of image
Plotting R,G,B planes and original image
Converting each plane segmented into binary images based on threshold gray value
Neglecting green plane and inverting the blue plane and the plotting
Applying morphological functions (Erotion and Dialation) to get rid of noise in binary plane
Labeling/seperating each blob by using bw label function

Measuring the area of each blob to identify hammer and screw driver
Classifying objects according to the pixels areas.

Plotting bounding box of Hammer and Screw Driver and selecting automatic points
blob_object_detection funtion
% This fucntion takes image ,threshold values as an input and provide ...
%... Boundind boxes/object detection of Hammer and screw driver
% Hammer and screw driver automatic selected points are also the ouput.
function [sdbboxes1,hmbboxes1,hasm_top,hasm_bottom,scs_top,scs_bottom ] = blob_object_detection( image,thresh,d_size,sd_high)
Reading image
A=image;
Error using blob_object_detection (line 9)

Segmenting Red,Blue,Green plane of image
rmat=A(:,:,1); % Red plane

gmat=A(:,:,2); % Green plane
bmat=A(:,:,3); % Blue plane
Plotting R,G,B planes and original image
figure(1);
subplot(2,2,1),imshow(rmat);title('Red plane');
subplot(2,2,2),imshow(gmat);title('Green plane');
subplot(2,2,3),imshow(bmat);title('Blue plane');
subplot(2,2,4),imshow(A);title('original image');
Converting each plane segmented into binary images based on threshold gray value
lr=0.3;lg=0.2; % threshhold binary level of R,G,B plane

%lb=0.1;
Br=im2bw(rmat,lr);Bg=im2bw(gmat,lg);Bb=im2bw(bmat,thresh); % Binary image planes
Neglecting green plane and inverting the blue plane and the plotting
Bsum=(Br&(~Bb));
figure(2);
subplot(2,2,1),imshow(Br);title('Red plane');
subplot(2,2,2),imshow(Bg);title('Green plane');
subplot(2,2,3),imshow(Bb);title('Blue plane');
subplot(2,2,4),imshow(Bsum);title('Combined image (Green plane neglected and Blue plane inverted');
Applying morphological functions (Erotion and Dialation) to get rid of noise in binary plane
se=strel('disk',d_size); % Disk of size 8 will clear noise

afterOpening=imopen(Bsum,se);figure();%Morphological open function performs erosion follwed by dialation
imshow(afterOpening);
afterClosing=imclose(afterOpening,se);figure();% Morphological close function performs dialation follwed by erosion
63
file:///G:/Matlab cousre/3d machine vision/html/blob_object_detection.html 1/3
64
imshow(afterClosing);
afterClosing1=imclearborder(~afterClosing);figure();imshow(afterClosing1); % Clears noise atttached at the boundaries of the image.
% Applying again to filter more noise
se1=strel('disk',d_size);
afterOpening2=imopen(afterClosing1,se1);figure();
imshow(afterOpening2);
afterClosing2=imclose(afterOpening2,se1);figure();
imshow(afterClosing2);
Labeling/seperating each blob by using bw label function
[l,num]=bwlabel(afterClosing2,8); % bwlabel function assign each object an different gray value for differentiating object
im1=(l==5); % plotting only one blobobject for testing
figure();imshow(im1);
Measuring the area of each blob to identify hammer and screw driver
q=regionprops(l,'Area'); % regionprops function identified the area /no of pixels the blob contains
blobMeasurements=struct2cell(q); % converting structured varirable to cell
Classifying objects according to the pixels areas.
for i=1:num
if blobMeasurements{i}<7000
continue
end
% Screw Driver detection
if blobMeasurements{i}>50000 && blobMeasurements{i}<sd_high

sd=(l==i);
sdbboxes=regionprops(sd,'BoundingBox'); % regionprops (BoundingBox) provides the cordinated of the box
sdep=regionprops(sd,'Extrema'); % regionprops (Extrema) provides the extreme 8 points of the blob
sdbboxes1=sdbboxes.BoundingBox;
figure();imshow(sd);title('screw driver') ;
% Hammer detection
elseif (blobMeasurements{i}>112000 && blobMeasurements{i}<800000)

hm=(l==i);
hmbboxes=regionprops(hm,'BoundingBox'); % regionprops (BoundingBox) provides the cordinated of the box
hmep=regionprops(hm,'Extrema'); % regionprops (Extrema) provides the extreme 8 points of the blob
hmbboxes1=hmbboxes.BoundingBox;
figure();imshow(hm);title('Hammer') ;
% Scale/world refrence frame detection
% elseif blobMeasurements{i}>800000
% sc=(l==i);
% figure();imshow(sc);title('scale');
% scbboxes=regionprops(sc,'BoundingBox');
% scep=regionprops(sc,'Extrema');
% scbboxes1=scbboxes.BoundingBox;
% coins detection
% elseif blobMeasurements{i}>7000 && blobMeasurements{i}<40000
% co=(l==i);
% figure();imshow(co);title('coin')
end
end
Plotting bounding box of Hammer and Screw Driver and selecting automatic points
% Hammer head points and tip points segregation
has=struct2cell(hmep); % Converting sturcture to cell variable

%sac=struct2cell(scep);
hasm=cell2mat(has); % converting cell to matrix variable
%sasm=cell2mat(sac);
hasmx=hasm(:,1);hasmy=hasm(:,2);
hasmx_top_leftx=(hasmx(1)+hasmx(2))/2;hasmy_top_lefty=(hasmy(1)+hasmy(2))/2;hasm_top_left=[hasmx_top_leftx,hasmy_top_lefty];
hasmx_top_rightx=(hasmx(7)+hasmx(8))/2;hasmy_top_righty=(hasmy(7)+hasmy(8))/2;hasm_top_right=[hasmx_top_rightx,hasmy_top_righ ty];
hasmx_bottom_leftx=(hasmx(3)+hasmx(4))/2;hasmy_bottom_lefty=(hasmy(3)+hasmy(4))/2;hasm_bottom_left=[hasmx_bottom_leftx,hasmy_bottom_lefty];
65
66
hasmx_bottom_rightx=(hasmx(5)+hasmx(6))/2;hasmy_bottom_righty=(hasmy(5)+hasmy(6))/2;hasm_bottom_right=[hasmx_bottom_rightx,ha smy_bottom_righty];
% Based on the distance the tip and head is segregrated

hasm_top_dist=pdist([hasm_top_left;hasm_top_right],'euclidean');
hasm_bottom_dist=pdist([hasm_bottom_left;hasm_bottom_right],'euclidean');
if hasm_top_dist<hasm_bottom_dist
hasm_top=(hasm_top_left+hasm_top_right)/2;
hasm_bottom=[hasm_bottom_left;hasm_bottom_right];
else
hasm_bottom=(hasm_bottom_left+hasm_bottom_right)/2;
hasm_top=[hasm_top_left;hasm_top_right];
end
% Screw driver head points and tip points segregation
scs=struct2cell(sdep); % Converting sturcture to cell variable

scsm=cell2mat(scs); % converting cell to matrix variable
scsx=scsm(:,1);scsy=scsm(:,2);
scsx_top_leftx=(scsx(1)+scsx(2))/2;scsy_top_lefty=(s csy(1)+scsy(2))/2;scs_top_left=[scsx_top_leftx,scsy_top_lefty];
scsx_top_rightx=(scsx(7)+scsx(8))/2;scsy_top_righty=(scsy (7)+scsy(8))/2;scs_top_right=[scsx_top_rightx,scsy_top_righty];
scsx_bottom_leftx=(scsx(3)+scsx(4))/2;scsy_bottom_lefty=(scsy(3)+scsy(4))/2;scs_bottom_left=[scsx_bottom_leftx,scsy_bottom_lefty];
%scsx_bottom_rightx=(scsx(5)+scsx(6))/2;scsy_bottom_righty=(scsy(5)+scsy(6))/2;scs_bottom_right=[scsx_bottom_rightx,scsy_bott om_righty];
scs_top=[scs_top_left;scs_top_right]; % Screw driver head points

scs_bottom=scs_bottom_left; % Screw Driver tip points
% Plotting the bounding box and points
figure();
imagesc(A)
axis('equal')
axis([0 6000 0 4000])
hold on
% Screw Driver
rectangle('position',[sdbboxes1(1),sdbboxes1(2),sdbboxes1(3),sdbboxes1(4)],'Edgecolor','r','Linewidth',2,'Tag','Screw Driver')
plot(scsx,scsy,'+','markersize',15,'linewidth',2,'color','g');
plot([scs_top_left(1),scs_top_right(1)],[scs_top_left(2),scs_top_right(2)],'+','markersize',15,'linewidth',2,'color','r');
plot( scs_bottom(1),scs_bottom(2),'+','markersize',15,'linewidth',2,'color','r');
% Hammer
rectangle('position',[hmbboxes1(1),hmbboxes1(2),hmbboxes1(3),hmbboxes1(4)],'Edgecolor','g','Linewidth',2)
plot(hasmx,hasmy,'+','markersize',15,'linewidth',2,'color','b');
if hasm_top_dist < hasm_bottom_dist

plot([hasm_bottom_left(1),hasm_bottom_right(1)],[hasm_bottom_left(2),hasm_bottom_right(2)], '+','markersize',15,'linewidth',2,'color','r');
plot( hasm_top(1),hasm_top(2),'+','markersize',15,'linewidth',2,'color','r');
else
plot([hasm_top_left(1),hasm_top_right(1)],[hasm_top_left(2),hasm_top_right(2)],'+','markersize',15,'linewidth',2,'color','r');
plot( hasm_bottom(1),hasm_bottom(2),'+','markersize',15,'linewidth',2,'color','r');
end
hold off
end
67
68
12/13/2019 epipolar_line_objectpoints
epipolar_line_objectpoints
% This function takes image , the bounding box x limits and the normal...
% ...vector as an input and provide epipolar lines inside the bounding box of...
% ...the object in order to reduce the computational time for stereo matching
function [x_ax,y_ax] = epipolar_line_objectpoints(image,x_min,x_max ,a)
figure();
imagesc(image)
axis('equal')
axis([0 6000 0 4000])
hold on; x_ax=(floor(x_min)-
50):(floor(x_max)+floor(x_min)+50); y_ax=(-
(a(1)/a(2)))*x_ax +(-(a(3)/a(2)));
plot(x_ax,y_ax,'color','b');
xlim auto;
ylim auto;
end
Error using epipolar_line_objectpoints (line 12)

69
file:///G:/Matlab cousre/3d machine vision/html/epipolar_line_objectpoints.html 1/2
70
12/13/2019 epipolar_line_objectpoints
file
:///
G:/
M
atl
ab
co
us
re/
3d
m
ac
hin
e
vis
ion
/ht
ml/
epi
pol
ar
_li
ne
_o
bje
ctp
oin
ts.
ht
ml
71
2/2
72
12/14/2019 stereo_matching
stereo_matching
% This function takes image , x adn y cordinate of epipolar lines,bounding...

% ...box cordinates, Template points cordinates and Template width of point to...
%... be stereo matched usind Sum of square diffence method
% Output of this function is the minimized cost function output which the...
%...stereomatched points in the other image
function D = stereo_matching( image,x_2,y_2,box,T1,tem)
new_min=100000;locax=1;
for ix=1:size(x_2,2)
%if (y_2(ix)<(box(2)-50)|| y_2(ix)>(box(4)+50))&&y_2(ix)<0&&x_2(ix)<0
%if (floor(y_2(ix))<floor(box(2)-100)|| floor(y_2(ix))>floor(box(4)+100))
if (floor(y_2(ix))<floor(box(2))|| floor(y_2(ix))>(floor(box(4))+floor(box(2)))) % points along the epipolar lines
%if (floor(y_2(ix))>4000 || floor(y_2(ix))<0||floor(x_2(ix))>6000 || floor(x_2(ix))<0)
continue
else
% Image 2 point template created as of same size for intensity matching hm_points_temp=image((floor(y_2(ix)-
tem):floor(y_2(ix)+tem)),(floor(x_2(ix)- tem):floor(x_2(ix)+tem)));
% sum of square Intensity difference of point in Image 1 and the point along epipolar line in the other image A=(T1-
hm_points_temp).^2;
B=sum(sum(A)); % sum of all points inside templates
if B<new_min % Finding Minimum cost function
new_min=B;
locax=ix;
end
end
end
D=locax; % Minimized cost function output / Stereo matched point
end
Error using stereo_matching (line 12)

73
file:///G:/Matlab cousre/3d machine vision/html/stereo_matching.html 1/2
74
12/14/2019 stereo_matching
75
file:///G:/Matlab cousre/3d machine vision/html/stereo_matching.html 2/2
76
12/13/2019 reconstruction
reconstruction
% T h i s f u n ct i o n t a k e s o r i g in o f c a m e r a 1 , d i s t a n c e b e t w e e n t w o o r ig i n , Ne w r e ct if ie d p o i n t s , N e w r e ct if ie d r o t a t io n m a t r i x , a n d N e w a v e r a g e in tr in sic m a t r i x
% Pr o vi d e s f inal 3 d p o in t a s a n o u t p ut b y t ra ingu la t ion t h e o r y
function P_ wo r ld = r e co n str u ct io n( o 1 , to , P1 _ n e w, P2 _ n e w, R1 _n e w, R2 _ n e w, K_ n e w )
f or i=1 : si ze ( P1 _ n e w , 1 )
p 1 1 =( R 1 _ n e w) ' * i n v ( K _ n e w ) *( ( P 1 _ n e w ( i , : ) ) / ( P 1 _ n e w( i , 3 ) ) ) ' ; % I m a g e 1 p o in t in r e ct if ie d p l a n e is c o n v e r t e d t o w o l r d p o i n t v e c t o r
p 2 2 =( R 2 _ n e w) ' * i n v ( K _ n e w ) *( ( P 2 _ n e w ( i , : ) ) / ( P 2 _ n e w( i , 3 ) ) ) ' ; % I m a g e 2 po in t i n r e ct if ie d p l a n e is c o n v e r t e d t o w o l r d p o i n t v e c t o r
j=cr o ss( p 1 1 , p2 2 ) ; % Pe r p e n d icu la r v e ct o r f r o m t w o i m a g e p o in t ve ct o r s

j1=j/norm(j); % unit ve ct o r
P=[p11,p22,j1];
k a =in v( P) ;
ka1 =ka * to;
P_world(:,i)=o1 +ka1(1,1)*p11+(ka1(3,1 )/2)*j1; % final m i d point o f t he t wo ve ct or p erp end icu lar ve cto r
e nd
Er r o r u s in g r e con st r u ct ion ( lin e 7 )

N o t e n o u g h in p u t a r g u m e n t s .
77
file:///G:/Matlab cousre/3d machine vision/html/reconstruction.html 1/2
78
12/13/2019 reconstruction
79
file:///G:/Matlab cousre/3d machine vision/html/reconstruction.html 2/2
80
12/13/2019 pose_and_orientation
pose_and_orientation
% This function takes three world cordinates points ( for e g Ham m er two head pooints and hamm er tip point)as a n input
% P ro v i d e s O rig i n of object,unit v e c t o r X -axis,unit v ect or Y -axis,unit v ec t or Z -axis a n d l ength of obj ect a s a n output.
function [ d,DA_norm,DC_norm,ZD_norm,dis ] = pose_and_orientation( a,b,c )
syms t
% A, B object top points and C is the bottom points and D is the perpendicular intersecting point from point C To line AB. A B= b -
a;
r=a+t*AB; % line equation
D C = c -r;
SD=DC(1)*AB(1)+DC(2)*AB(2)+DC(3)*AB(3);
t=solve(SD==0,t);
t = v p a(t);
d=a+t*AB; % intersecting point D lies on the line AB
DA=d-a;
DC=c-d;
AC=c-a;
dis=norm(cross(AB,AC)/norm(AB)); % Length of object
ZD=cross(DA,DC);
DC_norm=DC/norm(DC); % Y- axis od object
DA_norm=DA/norm(DA); % X- axis of object
ZD_norm=ZD/norm(ZD); % Z- axis of object
end
Error using pose_and_orientation (line 10)

81
file:///G:/Matlab cousre/3d machine vision/html/pose_and_orientation.html 1/2
82
12/13/2019 pose_and_orientation
83
file:///G:/Matlab cousre/3d machine vision/html/pose_and_orientation.html 2/2
84
12/13/2019 rectification_1
rectification_1
% This function takes image, Rectification Transformation matrix, minimum row and colum,maximum row and column of rectified image
% Provide Rectified image as an output but not accurate
function image_2=rectification_1(image,T,min_ro,max_co,max_ro,min_co)
% Convert the color image into gray scale image.

image = rgb2gray(image);
% Convert the data type from uint8 to double.
image = double(image);
nRow = size(image, 1);

nCol = size(image, 2);
for row2 = min_ro:max_ro

for col2 = min_co:max_co
R_p1=T*[;xx;1];
R_p=R_p1/R_p1(3);
x=R_p(2);
y=R_p(1);
row1 = y + half_nRow;
col1 = x + half_nCol;
% Bilinear interpolation
if 1 <= col1 && col1 <= nCol && 1 <= row1 && row1 <= nRow
% ..When inside the image 1,
% ..linearly interpolate to get sub_pixel gray level.
up = floor(row1);% convert to integer
down = up + 1;
left = floor(col1);
right = left + 1;
if down <= nRow && right <= nCol

intensity_1 = image(up, left);
intensity_2 = image(down, left);
leftIntensity = (row1 - up) * ...
(intensity_2 - intensity_1) + intensity_1;
intensity_1 = image(up, right);

intensity_2 = image(down, right);
rightIntensity = (row1 - up) * ...
(intensity_2 - intensity_1) + intensity_1;
intensity = (col1 - left) * ...

(rightIntensity - leftIntensity) + leftIntensity;
else
intensity = image(round(row1), round(col2));
end
else
% ..When ouside the image 1,
% ..set the gray_level to be white.
intensity = 0;
end
image_2(row2, col2) = intensity;

end
end
% Display the result.
figure;
subplot(1, 2, 1);
imshow(image ./ 255);
title('Original image');
axis on;
axis image;
subplot(1, 2, 2);
imshow(image_2 ./ 255);
85
file:///G:/Matlab cousre/3d machine vision/html/rectification_1.html 1/2
86
12/13/2019 rectification_1
hold on
title('Rectified image');
axis on;
axis image;
end
Error using rectification_1 (line 9)

87
file:///G:/Matlab cousre/3d machine vision/html/rectification_1.html 2/2
88
12/13/2019 Edge detection using in built function
Edge detection using in built function
close all;
A=imread('image1','jpg'); % reading image
Bsum=rgb2gray(A); % converting rgb to gray scale

figure(),imshow(Bsum);
a=edge(Bsum,'roberts'); % Roberts edge detection method

b=edge(Bsum,'sobel'); % Sobel edge detection method
c=edge(Bsum,'prewitt'); % Prewitt edge detection method
d=edge(Bsum,'canny'); % Canny edge detection method
e=edge(Bsum,'log'); % Log edge detection method
figure(3);
subplot(2,3,1),imshow(A),title('Original Image');
subplot(2,3,2),imshow(a),title('roberts');
subplot(2,3,3),imshow(b),title('sobel');
subplot(2,3,4),imshow(c),title('prewitt');
subplot(2,3,5),imshow(d),title('canny Image');
subplot(2,3,6),imshow(e),title('log Image');
Warning: Image is too big to fit on screen; displaying at 13%
89
file:///G:/Matlab cousre/3d machine vision/html/edge_detection_method.html 1/2
90
12/13/2019 Edge detection using in built function
91
file:///G:/Matlab cousre/3d machine vision/html/edge_detection_method.html 2/2
92
12/13/2019 Template matching using matlab inbuilt code
Template matching using matlab inbuilt code
Contents
Reading image and template and extracting feature points for matching
Matching feature in image
Reading image and template and extracting feature points for matching
s
boxImage1 = imread('hammer1.jpg'); % Reading Template image
figure(1);
i m sho w (boxIm age1 );
title('Image of a hammer ');
boxImage=rgb2gray(boxImage1); % Converting image to gray scale
sceneImage1 = imread('image1.jpg'); % Reading image

figure(2);
imshow(sceneImage1);
title('Image of a Cluttered Scene');
sceneImage=rgb2gray(sceneImage1); % Converting image to gray scale
% Selecting feature points on image and template
boxPoints = detectSURFFeatures(boxImage); % Template feature points

scenePoints = detectSURFFeatures(sceneImage); % Image feature points
% plotting feature points
figure(3);
imshow(boxImage);
title('100 Strongest Feature Points f rom B o x Image');
hold o n;
p l o t( s el ec t S t ro ng es t ( bo xP o i n ts , 1 0 0 ) ) ;
figure(4);
imshow(sceneImage);
title('300 Strongest Feature Points f rom S c e n e Image');
hold o n ;
p l o t ( s e l e c t S t r o n g e s t ( s c e n e P o i n t s , 3 0 0 ));
U n d e f i n e d func tion o r var iable 's'.
E r r o r in t e m p l a t e _ m a t c h i n g (l in e 3 )
s
Matching feature in image
[boxFeatures, boxPoints] = extractFeatures(boxImage, boxPoints);

[sceneFeatures, scenePoints] = extractFeatures(sceneImage, scenePoints);
boxPairs = matchFeatures(boxFeatures, sceneFeatures);
m a t c h e d B o x P o i n t s = b ox P o i n t s ( b o x P a i r s ( : , 1 ) , :);
matchedScenePoints = scenePoints(boxPairs(:, 2 ), :);
93
file:///G:/Matlab cousre/3d machine vision/html/template_matching.html 1/2
94
12/13/2019 Template matching using matlab inbuilt code
figure(5);
showMatchedFeatures(boxImage, sceneImage, matchedBoxPoints, ...
matchedScenePoints, 'montage');
title('Putatively Matched Points (Including Outliers)');
[tform, inlierBoxPoints, inlierScenePoints] = ...

estimateGeometricTransform(matchedBoxPoints, matchedScenePoints, 'affine');
figure(6);
showMatchedFeatures(boxImage, sceneImage, inlierBoxPoints, ...
inlierScenePoints, 'montage');
title('Matched Points (Inliers Only)');
boxPolygon = [1, 1;... % top-left

size(boxImage, 2), 1;... % top-right
size(boxImage, 2), size(boxImage, 1);... % bottom-right
1, size(boxImage, 1);... %bottom-left
1, 1]; % top-left again to close the polygon
newBoxPolygon = transformPointsForward(tform, boxPolygon);
figure(7);
imshow(sceneImage);
hold on;
line(newBoxPolygon(:, 1), newBoxPolygon(:, 2), 'Color', 'y');
title('Detected Box');
95
file:///G:/Matlab cousre/3d machine vision/html/template_matching.html 2/2
96

3d Machinevision Final Project Report

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

3d Machinevision Final Project Report

Uploaded by

Copyright:

Available Formats

TOOL IDENTIFICATION SYSTEM

USING 3D MACHINE VISION

Prasanna Iyer (101172980)

Tushar Vora (300120192)

Nirmal Dawda (101135054)

Moose Hijazi (101102624)

MECH5801F [34141] 3D Machine Vision (LEC) Fall 2019

Prof. Paul Harrison

Department of Mechanical and Aerospace Engineering

List of Figures ........................................................................................................................................................................................ 6

List of Appendices ................................................................................................................................................................................. 7

1 Chapter: Introduction – Stereo Images……………………………………………………………………………………………..8

1.2 Stereo Images................................................................................................................................................................... 9

2 Chapter: Epipolar Geometry .................................................................................................................................................... …11

3 Chapter: Camera Calibration……………………………………………………………………………………………………..14

4 Chapter: Object Detection and Object Point Detection ………………………………………………………………………...18

4.1.1 Various Object Detection Techniques

4.1.1.1 RGB to Binary Image

4.1.1.2 Edge Detection Technique

4.2 Morphological Operation ...................................................................................................................................................

5 Chapter: Introduction – Rectificaton & Stereo Matching………………………………………………………………………26

5.2 Stereo Matching................................................................................................................................................................

5.3 Alternatice Method...........................................................................................................................................................

5.4 Stereo Matching along epipolar line: Alternative Method .............................................................................................

6 Chapter: Introduction – Rectificaton & Stereo Matching……………………………………………………………………….31

6.2 Method Used .....................................................................................................................................................................

7 Chapter: Pose Estimation………………………………….………………………………………………………………………..33

9 Chapter: Discussions & Conclusions……………………………………………………………………………………………….38

Table 1- Projection matrix ofcamera1……………………………………………………….………..17

Table 12- Origin of Screw Driver……………...……………………………………………………..35

Figure 1. Comparison of Methodologies ......................................................................................................8

User Defined Functions

1.1 Objective of Project:-

Phase 2: Object identification and 3D Reconstruction

Figure 1. Comparison of Methodologies

Figure 2. Original stereo-image 1

Figure 3. Original stereo-image 2

Figure 4. Epipolar Geometry

Selecting point for both images

Define point vectors

Centroids for the stereo images

To find normalization scale factor

Defining normalization matrices

Normalization of points vectors and

Computing normalized y matrix

Obtain the SVD of the normalized Y

Define normalized F based on the last

Compute the SVD of the normalized F

Zero the smallest singular value of F and

Un-normalize the recomputed F matrix

Determine the normal vectors, a1 and a2,

Obtaining Epipolar Lines

Figure 7. Generated epipolar lines Figure 6. Generated epipolar lines (2)

Figure 8. Perspective Projection

Figure 9. Orientation of Calibrated cameras

Figure 10. 2D Calibrated Camera (blue) and coins (red)

4.1.1 Various Object Detection Techniques:

4.1.1.1 RGB to binary image –

Figure 11. Grey Level thresholding

Figure 12. Edge Detection Methods

4.1.1.3 Template Matching –

Figure 13. Template Matching

Figure 14. Modified Blob Analysis