Vision 1

Outline of the vision part

of the course

Introduction

Sensors, cameras etc

Image pre-processing
• Histogram
• Binarization
• Filters

Separate different objects in an image

Position and rotation of objects

Templates
Background
Why machine vision?
Image processing

1970s: viewed as the visual perception component of an ambitious agenda to mimic human's
intelligence to endow the robots with intelligent behaviour

1980s: a lot of attention was focused on more sophisticated mathematical techniques for
performing quantitative image and scene analysis.

1990: Increased interaction with computer graphics, especially in the cross-disciplinary area of
image-based modeling and rendering.

2000: One trend, which now dominates a lot of the visual recognition research, is the application
of sophisticated machine learning techniques to computer vision problems

2010: Increased use and integration with AI and machine learning

Examples of machine
vision application
Visual guided robot application
Bin-picking applications
Control of processes
Identification

Shape and dimension check

Completeness check

Image and object comparison

Bin picking
Completeness of putting
chocolate in a box
Principle of eye
Image processing
system
• Three main parts
• Sensors
• Computers
• Communication interface

https://www.mvtec.com/company/reference/detail/industrial-inspection-line-scan-based-vision-system-tackles-
color-print-inspection-daheng-image-v/
Image system
Sensors
Single image sensor

Line sensor

Array sensor
Cameras

• CCD
• Charge-coupled devices
• Has been used for 30 years
• 2009 års Nobel winner in physics, Willard Boyle and George Smith,
• CMOS
• Complementary metal-oxide semiconductor
• Used since 90’s
• Less power when CCD cameras
 Principle CCD

Cross-section of a CCD element

Principle structure of a
interline transfer CCD
CC 2.5 Michael Schmid, wiki
 Principle CMOS

CC 2.5 Michael Schmid, wiki

Image processing &
analysis

• Image processing image in->image out

• Image analysis image in->measurements out (features)
• Image description Features in->high-level description out
Steps in an Image
systems
Sensor
Intermediate results
Raw image data
Preprocessing
Image data
Segmentation
Segmented image
Feature extraction
Features
Classification
Object classes Image description
Interpretation
Image definitions

• 2-dimension array of pixels

P[0,0]
• P[0,0] top left corner
• i, j specifies row and column (i points down and j right).
Specific pixel value can be found at P[i,j]
• Image size m*n.
• P[m,n] last pixel in the array
P[i,j]

P[m,n]
Pixel resolution
2 ways to define pixel resolution
• With the set of two positive integer
numbers, the number of pixel columns
(width) and the number of pixel rows
(height), for example as 2048 × 1536.
• With the total number of pixels in the
image, typically given as the number of
megapixels, calculated by multiplying
pixel columns by pixel rows and dividing
by one million.
• Example: An image that is 2048 pixels in
width and 1536 pixels in height has a total
of 2048×1536 = 3,145,728 pixels or 3.1
megapixels. Image resolution can be
either 2048 × 1536 or a 3.1-megapixel.
Light
Spectroscopy of
welding

Laser welding Results from spectroscopy of the

welding process
Colour Theory
RGB Cube
RGB colour system
RGB colour system

• Bit depth
• The bit depth of a image quantifies how many unique colors are available in a
image color pallet. For a gray scale image it determines how many shades of
gray.
• Image Channels
• The available color channels for a image, normally 3 red, green and blue.
• Fourth channels handles transparacy
RGB colour system

• Different depths and channels

• 8-bit RGB image gives 2^8 = 256 unique colour values.
• 16-bit RGB image gives 2^16 = 65536 unique colour values
• 24-bit RGB image gives 2^24 ≈ 16.7 million unique colour values.

• Images
• Normal bit depth for bmp, jpg and other images is 24. This gives 8-bits per
channel and is often referred to as “true colour”
Other color models

• CMY and CMYK

• Cyan, Magenta, Yellow, Black
• Primary used in printing applications

• HSI
• Hue, Saturation, Intensity
• Describing colors practical human interpretation
Grey scale transformation

• Average
Colour image

Average grey scale

• Weighted average
Weighted grey scale (Red)
g = .7*r + .15*g + 0.15*b

Can be customized
Red
Colour to
Green
grey scale
Blue

= 122
120

Grey
Red
Colour to
Green
grey scale
Blue

Grey
Example grey scale
transformation
131 243 80 58
254 193 73 199
242 218 51 157
128 140 102 74
239 112 117 227

255 105 252 172

197 201 132 32
34 196 73 130
154 46 34 225
145 164 190 66

189
217
63
245
87
228
169
137
Use both weighted
244 69 150 62 and average grey
113 225 22 159
159 252 34 153 scale transformation
Pre-processing
Type of operation

• The types of operations that can be applied to digital images to

transform an input image a[m,n] into an output image b[m,n] can be
classified into three categories:
Type of operation

input output
• Point operation:
• the output value at a specific
coordinate is dependent only on the
input value at that same coordinate.
• Local operation:
• the output value at a specific input output
coordinate is dependent on the input
values in the neighbourhoods of that
same coordinate.
Type of operation

Global operation: input output

• The output value at a specific

coordinate is dependent on all
the values in the input image.
Histogram
Black and white

• Object separated from background

• Object pixels have value 1
• Background pixels have value 0
• However
• White is 255
• Black is 0
• Normally based on experience
• First runs can be to select threshold
• Automatic methods can be used
Thresholds

• Global thresholds works extremely well except

when the 2 peaks in the histogram is violated.
• Dynamic thresholds is based on current pixels
brightness compared to the local background
Global Thresholds
Global Thresholds,
example

13 10 18 10 9 9
10 10 100 100 10 8
1 10 100 102 1 10
12 100 98 103 12 11
14 103 99 100 14 40
12 104 105 101 12 60
10 10 10 10 65 70
Global Threshold,
example

0-25 18 20
18
26-50 1 16
51-75 3 14
12
76-100 6 10
101-125 6 8
6
125-150 0 4
151-175 0 2
0
176-200 0
- 25 -50 -75 100 125 150 175 200 225 255
201-225 0 0 26 51 6- 1- 5- 1- 6- 1- 6-
7 10 12 15 17 20 22
226-255 0
Global Threshold,
example
T=50
0 0 0 0 0 0
0 0 1 1 0 0
0 0 1 1 0 0
0 1 1 1 0 0
0 1 1 1 0 0
0 1 1 1 0 1
0 0 0 0 1 1
T=80
0 0 0 0 0 0
0 0 1 1 0 0
0 0 1 1 0 0
0 1 1 1 0 0
0 1 1 1 0 0
0 1 1 1 0 0
0 0 0 0 0 0
Example
microstructure
imgo = imread('TM3000_1332(x7,0k).tif');
imgc = imcrop(imgo, [1 1 1275 947]);
histogram(imgc)
imgb = imbinarize(imgc);
figure
imshowpair(imgc,imgb,'montage')
%%
thresh = multithresh(imgc,1);
seg_I = imquantize(imgc,thresh);
RGB = label2rgb(seg_I);
figure;
imshow(RGB)
axis off
title('RGB Segmented Image')

(1-sum(sum(imgb))/948/1276)/…
(sum(sum(imgb))/948/1276)

Matlab code
Example
microstructure
10 4
2.5

1.5

0.5

0
0 50 100 150 200 250
Converted grey scale image
Histogram of the image to black and white image
Example
microstructure
RGB Segmented Image

Converted to RGB segments for better understanding

Noise

The image can be corrupted by

• Random variations intensity (noise)
• Variations in illuminations
• Poor contrast
Three types of intensity variations
• Salt and pepper (random, both black and white intensity)
• Impulse (random, white intensity)
• Gaussian (intensity variation follows the Gaussian distribution
 Salt and pepper noise

Original image Salt and pepper noise

https://en.wikipedia.org/wiki/Lenna
Filter

• Linear filter
• Weighted sums
• Gaussian smoothing filter
• Non linear filter
• Everything else such as Median filter
Types of neibourhood

4 neighbours 8 neighbours
Moving window

Input image

Filter kernel

Output image
Moving window
Border effects

• Images borders are a problem. How to handle [0,0]?

• Image cropping. Problem reduced output image
• Set border pixel, usually 0 (black). Problem creating false edges
• Use available pixels. Working fine for rectangular filters
• Repeat valid border pixels: pixel [1,1] are used for [0,0], [1,0], [0,1] and pixel
[2,0] are used for [2.0]
Border effects,
repeated values
0 1 2 3 4 5 6 7 8 9
0

9
Smoothing filter

• Low pass filter used to suppress noise

• Are based on average
• Scaling factor h(m,n) sets to 1
• Destroys edges
• Can be customized
Moving window

Input image
Input image Output image
Output image
19 9 12 19 1 7 19
5 24 20 93 1 128 20 25 35 21 33 24
16 105 17 18 12 19 10 26 36 23
23 6 16 22 8 15 20
24 22 5 3 17 12 20
6 18 214 213 218 19 11
7 6 222 223 210 9 4
16 2 214 211 212 24 21
15 14 221 209 215 5 1
18 19 212 221 221 2 6
5 16 221 213 203 16 1
12 19 20 19 17 116 23
18 19 9 8 14 11 0
20 21 21 15 2 12 3
Smoothing filter
Colour 2 grey
Original image 3x3 filter

10x10 filter 20x20 filter Filter sizes

matters!
Original image
Noise

Original image Image after

with noise filtering with
3x3 pixel
Smoothing filter

Original image Filtered image

Gaussian filter

• Optimal smoothing filter

• Using integral filter weights
• Kernel sizes 3x3 or 5x5
Gaussian
Gaussianfilter
distribution

Gaussian
3x3 filter

Gaussian
5x5 filter
Gaussian filter

Original image

Filtered image
Median filter

• Main problem with mean filters is that they tend to blur sharp
discontinuities in the image
• Median filter are effective in removing salt and pepper noise
• Sort the pixels into ascending order by grey level
• Select the value of the middle pixel as the new value for pixel [i,j]
• Time consuming due to the sorting of all pixels
Median filter

215 204 90 83 166 131

206 216 109 46 60 100 7
110 234 161 34 226 230 30
204 7 172 77 233 138 61
183 130 61 216 19 122 77
178 250 215 30 155 127 130
67 105 216 91 93 159 172
92 186 28 56 184 238 215
20 128 69 30 120 97 216
178 29 251 51 166 34 250

130
New pixel value
Median filter

Original image Filtered image

Different types of
filters

Mean filter

Original image

Gaussian filter

Median filter
Example

19 9 12 19 1 7 19
5 24 20 93 1 128 20
16 105 17 18 12 19 10
23 6 16 22 8 15 20
24 22 5 3 17 12 8
6 18 214 213 218 19 11
7 6 222 223 210 9 4
16 2 214 211 212 24 21
15 14 221 209 215 5 1
18 19 212 221 221 2 6
5 16 221 213 203 16 1
12 19 20 19 17 116 23
18 19 9 8 14 11 0
20 21 21 15 2 12 3
Image arithmetic

• General problem. Keep the interval of 0-255

• Addition and averaging
• Clipping: Cut the maximum value to 255
• Division by the number of images (average)
• Scaling to minimum and maximum of the pixel value
• Subtraction
• Absolute value
• Positive clipping: Set all negative value to 0
• Negative clipping: Set all positive values to 0 and use absolute negative values
for grey scale information

• Linear scaling
Example subtraction

Reference image Misprint Absolute difference

Positive clipping Negative clipping Linear scaling

Example subtraction

Effect of uneven illuminance

Microscope image Reference image Results is an image

of collenchyma cells that has the same with more uniform
illumination illumination
variation
 Averaging using
addition

One image 500 images

Source: Image Arithmetic in DSP: Image Averaging and Image Subtraction - Technical Articles (allaboutcircuits.com)
Image arithmetic
Combining two images
• Minimum: Using the darker pixel
• Maximum: Using the lighter pixel
Input image 1 Maximum of two pixels
28 76 37 72 50 50 76 37 72 50
30 55 100 100 8 30 55 100 100 10
91 77 100 102 39 91 77 100 102 39
29 100 98 103 4 29 100 98 103 12
92 103 99 100 89 92 103 99 100 89
22 104 105 101 54 22 104 105 101 54
56 99 36 4 95 56 99 36 10 95

Input image 2 Minimum of two pixels

50 10 18 10 9 28 10 18 10 9
10 10 100 100 10 10 10 100 100 8
1 10 100 102 1 1 10 100 102 1
12 100 98 103 12 12 100 98 103 4
14 103 99 100 14 14 103 99 100 14
12 104 105 101 12 12 104 105 101 12
10 10 10 10 65 10 10 10 4 65
Linear gray level
scaling
C1 > brightness increased
C1 < brightness reduced
| c2| > contrast enhanced
| c2| < contrast diminished

Simple but effective. Remember white = 255 and black is 0

Linear gray level
scaling
Too high brightness Too high contrast

Correct

Too low brightness Too low contrast

Contrast stretching

• Point operation
• Using the full range of the grey scale
• Can remove effects from the camera system

• Example image with min = 85 and max = 225

Contrast stretching

Analysing the grey scale contents in the image

using histogram.
Contrast stretching
Contrast stretching