Robot Vision SS21

Robotic Vision
SS21
Prof. Dr.-Ing. Maria Schweigel
SS 2021 Robotic Vision – Prof. Dr.-Ing. Maria Schweigel 1

organizational things 2
meetings:
Lesson every Monday 8:00-9:30 and
10:00 – 11:30 online in BBB
Exam:
written exam at end of semester about 120 min
SS 2021 Robotic Vision – Prof. Dr.-Ing. Maria Schweigel

content 3
Chapter 1 – Introduction
1. Development of Robotic, Robotic today (Industry robotic, Service
robotic, Self driving cars, Medical robots, Aeronautic and military
robots)
2. Autonomous robotics: Definition Robot
3. Artificial Intelligence – AI: Neural Networks, classification flow –
superviced learning, over and under fitting,
4. Ethic Aspects
5. Vision based Robotics: Main components of vision based robots,
Classification of a camera as a sensor, Why vision in robotics?,
Challenges of robot vision, Structure of vision sensing
Chapter 2 – Coordinate systems and transformations

1. Mathematical representation of a robot
2. Rotation Matrix: In 2D, In 3D, Other representations of rotation matrix
in 3D
3. Transformation Matrix: In 2D, In 3D
4. Center of Rotation
content 4
Chapter 3 – Physics of Light and color

1. Radiometry (irradiance, radiance, radiant intensity)
2. Photometry: physical values (luminous flux, luminous intensity,
illuminance), physics of photometry (Absorbtion, Reflectance, Luminance),
Effects of Photometry (Photometric Stereo, Shadows, Polarization)
3. Colorimetry: Trichromatic theory and Opponent color theory, Physic of
colors (math. Description, reproducing color, chromaticity space, hue and
saturation)
4. Problems in Robot Vision because of light and color: Color temperature
and white balancing, Dichromatic refection, Gamma and gamma
correction, Shadows and shadow removal
Chapter 4 - Image formation

1. Perspective projection: Projecting Points with the Camera Matrix, Other
mathematical descriptions of perspective projection, Distortions
2. Camera calibration: Classical calibration techniques, Bouguet’s Calibration
3. Types of wide-view cameras: Fisheye Lens Camera, Catadioptric Camera,
Spherical Camera, Transformation of image models
content 5
Chapter 5 – Image processing

1. Histogram
2. Monadic operations
3. Diadic Operations
4. Spatial Operations: Linear Spatial filtering, Smoothing, Edge detection,
Template Matching, other Spatial filtering
5. Mathematical Morphology
6. Shape Changing: Cropping/ROI, Resizing, Warping
Chapter 6 – Image feature extraction

1. Classification: Binary classification via grey level, Color classification
2. Representation
3. Features: Boundary Box, Moments: Calculating Area, Center from
Moments, Orientation and Shape from moments, Shape, Character
recognition, Line features, Point features: Moravec’s interest operator,
Harris-Stephens Edge Detection, SIFT - Scale-invariant feature
transform, SURF - Speeded Up Robust Feature

content 6
Chapter7 - Vision-Based Control

1. Multi vision: Correspondence, Geometry of Multi Vision:
Epipolargeometry, Planar Homography, Photogrammetry
2. Visual servoing: Position-Based Visual Servo (PBVS), Image-Based
Visual Servo (IBVS)

literature 7
‘Vision Based Autonomous Robot Navigation’,

Chatterjee, Rakshit, Singh, Springer 2013
‘Robotics – Vision and Control‘, Peter Corke,

Springer 2017

Chapter 1 –
Introduction

Development of Robotic 9
Electro the Smoking Robot

(1939)
1. Introduction
https://www.pinterest.de/pin/409616528588699674/ https://www.pinterest.de/pin/311381761708387505/

1. Introduction

1. Introduction
https://www.zdnet.com/pictures/15-vintage-robots-that-definitely-show-their-age/10/

Robotic today 12
1. Introduction

Robotic today 13
1. Introduction
https://www.abb-kundenmagazin.de/fertigungsautomatisierung/trommelwirbel/

Industry robotic 14
1. Introduction

Service robotic 15
Care-O-bot
by
Fraunhofer
1. Introduction
IPA
https://automationline.rbsonline.de/multiturn-sensor-fuer-smarten-serviceroboter/

Service robotic 16
ARMA by KIT (Karlsruher

Instutut of Technology)
• size: 1,7m
• weight: 70kg
1. Introduction
• energy supply: battery

• 63 actuators, 214 sensors,
76 microkontroller, 3PCs for
perzeption
• balance
→ARMA should be able to act

in normal home and help
human with his daily work. https://his.anthropomatik.kit.edu/241.php

Self driving cars 17
1. Introduction

Self driving cars 18
https://www.wired.com/story/waymo-launches-self-driving-minivans-fiat-chrysler/
1. Introduction
Waymo
https://www.buyherepayhereusa.com/blo
g/safety-tips-learn-from-self-driving-cars/

Medical robots 19
1. Introduction
https://intelligente-welt.de/medizinroboter/
Medizin robot Da Vinci, the only system with a wide

international certification
Medical robots 20
https://intelligente-welt.de/medizinroboter/
1. Introduction
Miro by German Aerospace Center (DLR)


Agriculture robots 21
http://www.agriexpo.online/prod/bosch-deepfield-robotics/product-168586-1199.html
1. Introduction
BoniRob by University Osnabrück and BOSCH


Aeronautic and military robots 22
Mars Science Lander by NASA

1. Introduction
https://www.bostondynamics.com/wildcat
https://mars.nasa.gov/news/8371/curio WildCat by Boston

sity-rover-to-temporarily-switch-brains/
Dynamics

Aeronautic and military robots 23
1. Introduction

Elektro the Smoking Robot

(1939) Terminator (2029)
1. Introduction
https://www.pinterest.de/pin/409616528588699674/ http://de.moviepedia.wikia.com/wiki/Terminator_Genisys/Wikia-Kritik

Definition 25
1. Introduction
„A Robot is a goal oriented machine

that can sense, plan and act.“

Autonomous robitics 26
- from greek word:

„autós“ = self „nomos“ = law
1. Introduction
- autonomous: self-sufficient, independent, act

to own laws
→ An autonomous system (AS) acts

according to its own laws, which are
determined by software or acquired
through learning processes.

Artificial Intelligence - AI 27
•
1. Introduction
neural networks
• support vector machines
• decision trees
• fuzzy logic
• deep learning
• ….

Neural Networks 28
1. Introduction

Neural Networks 29
1. Introduction

Neural Networks 30
1. Introduction

classification flow – supervised learning
1. acquisition of data, determining features and related
class of data , e.g. with help of expert
2. training of classifier so that classifier can classify as much
as possible training data with the correct class =
1. Introduction
optimizing process (step by step adaption of classifier

model till results with testing data satified
3. determining quality of classificator with help of an
independent data set (validation data), How good
classifator can handle with unknown data?
→ tree data sets necessary:

• training data
• testing data
• validation data (independent)

over and under fitting
e
Optimum
1. Introduction
e – mean square error
N – number of training cycle
training data
validation data
over fitting
under fitting
Optimum
under fitting over fitting
N

example over and under fitting
1. Introduction
? ? ?

Ethic Aspects 34
What and from

What acceptance Can ethical
whom can the AS
is given to an AS? instructions be
learn?
1. Introduction
translated into the AS

language?
Ethic Aspects
Will the AS replace Who ultimately bears
humans? the responsibility?
Which security gaps

With whom can the How much is a life
are hidden in an AS?
AS connect and worth?
communicate?

Main components of vision based robots 35
Maps Path planning

1. Introduction
Data Acquisition Self-

Localisation
Feature Landmark
Extraction Recognition

Classification of a camera as a sensor 37
passive
1. Introduction
aktive
sensors
proprioceptive exteroceptive

Why vision in robotics? 38
• Most important sense for humans

• conventional sensors, like ultrasonic, IR, GPS,
laser sensors etc. have physical limitations or
1. Introduction
high cost!
→ Vision based robotic:
• Reduce cost
• Do not emit light or waves
• Maintaining of high degree of intelligence,
flexibility and robustness

Challenges of robot vision 39
• Handling with hugh data

• Vision processing is computationally
1. Introduction
intensive
→ Increasing of computation resources

provide usage of vision based robotics

Structure of vision sensing 40
1. Introduction

Chapter 2 –
Coordinate systems and
transformations

Mathematical representation of a robot 44
2. Coordinate systems and transformations
Representation as a Point or a Vector

y axis
P
PQ Q
P
Q
origin x axis

Representation as a relative Pose and a object frame

y axis
yR
P2
P1
{R}
ξR
P
xR
origin x axis

Relations of Pose and Points

y axis
P yR
RP
OP
Oξ
{R}
R xR
{O} x axis

Relations of Pose and Points

y axis
yC
P CP
xC
yR
RP
{C}
OP Rξ
C
Oξ
{R}
R xR
{O} x axis

Relations of Pose and Points in 3 Dimensions
[Corke, Robotics – Vision and Control]

Algebraic rules

Rotation Matrix in 2D 50
Rotation of coordinate system about origin in 2D

Ay
By
P
AP
y
cos ϕ
BP Bx
y
BP sin ϕ
x ϕ
-sin ϕ {A} {B} AP cos ϕ Ax

x

Euler’s rotation theorem:

“Any rotation can be considered as a sequence of rotations about

different coordinate axes”
[Corke, Robotics – Vision

and Control]

Rotation in 3 Dimensions is not commutative!


and Control]

Chapter 3 –
Physics of Light and color

3. Physics of Light and color

Photometry 56

Photometry 57
Emission
spectra for
sun, tungsten
lamp and
response of
human eye

and Control]

Absorption 58
Transmittance
of sun light in
5 m water

and Control]

Procedure of illumination 59
illuminant
illuminance luminance
E(λ) L(λ)
reflectance
R(λ)


Colorimetry 61
Spectra
response of
three
different cone
cells of
human eye
(normalized)

and Control]

Lab color space 62
https://wisotop.
de/lab-
farbmodell.php

Procedure of image formation 63
illuminant response
M(λ)
illuminance luminance
E(λ) L(λ)
reflectance
R(λ)

Colorimetry 64
Spectra of
light from
source till
human eye

and Control]

CIE primaries 65
Red: λr = 700nm
Green: λg = 546.1nm
Blue: λb = 435.8nm

CIE color matching functions 66
Color
matching
functions
for
standard
observer

and Control]

Chromaticity space 67
r-g-chro-
maticity
plane

and Control]

Chromaticity space – imaginary nonphysical primaries 68
Color
matching
functions
for standard
observer
based on
imaginary
primaries
X,Y,Z

and Control]

Chromaticity space – imaginary nonphysical primaries 69
x,y-chromaticity plane
[https://en.wikipedia.org/wiki/Impossible_c
olor]

HSV color space 70
https://en.wi
kipedia.org/w
iki/HSL_and_
HSV#/media/
File:HSV_colo
r_solid_cylind
er_saturation
_gray.png

LCh color space 72
https://sensing.konicaminolta.
us/blog/understanding-the-
cie-lch-color-space/

Color temperature 73
Light source Color temperature (K)

Candle light 1 900

Dawn/dusk sky 2 000
40W tungsten lamp 2 600
2.5W(25W) LED lamp (warm white) 2 700
100w tungsten lamp 2 850
Tungsten halogen lamp 3 200
Direct sunlight 5 800
Overcast sky 6 000-7 000
Standard daylight 6 500
Hazy sky 8 000
Clear blue sky 10 000-30 000
and Control]

Dichromatic reflection model 74
Dichromatic
reflection on a
surface
a) Molecular
view
b) Theoretical
view
https://www.researchgate.ne
t/figure/The-dichromatic-
reflection-model-of-surface-
reflectance-in-
inhomogeneous-materials-
a_fig1_293121272

Dichromatic reflection model 75
http://time.com/5234787/di
rty-dozen-pesticides/
Picture of strawberries: Colored areas because of body

reflection, shiny areas because of specular reflection

Gamma correction 76
a)
b)
c)
a) without gamma correction b) with gamma encoding c) with gamma decoding

https://www.elmar-baumann.de/fotografie/lexikon/gammakorrektur.html

Gamma correction 77
Gamma correction of grey steps

Top: without gamma correction (linear grey steps)
Bottom: with gamma correction about γ=2.2 (linear sensing of steps)
https://www.elmar-
baumann.de/fotogra
fie/lexikon/gammak
orrektur.html

Shadow removal 78

and Control]

Chapter 4 –
Image formation

4. Image formation

Perspective Projection 81
4. Image formation
Perspective Projection of an object by pin hole and by lense [Corke, Robotics – Vision
and Control]

Characteristics of Perspective Projection 82
https://www.arthipo.com/artblog/en
/drawing-techniques/basic-
4. Image formation
principles-of-perspective-drawing-1-
diminution.html
Objects appear smaller the farther they are way

https://en.wikipedia.org/wiki/London_Eye#/m
edia/File:Ojo_de_Londres,_Londres,_Inglaterr
Circular Objects appear elliptic a,_2014-08-07,_DD_028.JPG

Central-projection model 83
P = (X, Y, Z)
4. Image formation
z
x
P‘ = (X‘, Y‘ ,Z‘)
xC
zC {C}
Camera origin
y yC

Camera Sensors 84
https://de.wikipedia.org/wiki/CCD-Sensor#/media/File:Delta-
https://de.wikipedia.org/wiki/Active_Pixel_Sensor#/media/File:Ac Doped_Charged_Coupled_Devices_(CCD)_for_Ultra-
4. Image formation
tive_pixel_sensor_prototype.jpg Violet_and_Visible_Detection.jpg
Camera Sensors: CCD-Sensor (left), CMOS-Sensor (right)

Discrete image plane 85
P = (X, Y, Z)
4. Image formation
z
x
P‘ = (X‘, Y‘ ,Z‘)
H
xC
zC {C}
Camera origin
y W yC

Perspective Projection with Camera Matrix C 86
4. Image formation

Distortions - Chromatic aberration 91
4. Image formation
https://de.wikipedia.org/wiki/Chromatische_Aberration

Distortions - Spherical aberration 92
4. Image formation
https://en.wikipedia.org/wiki/Spherical_aberration

Distortions - Astigmatism 93
4. Image formation
http://hyperphysics.phy-astr.gsu.edu/hbase/geoopt/coma.html

Geometric Distortions – tangential distortion 94
4. Image formation
https://de.mathworks.com/help/vision/ug/camera-calibration.html

Geometric Distortions – radial distortion 95
4. Image formation
Without distortion (left), Pincushion distortion (mid) and Barrel distortion (right)
http://vrguy.blogspot.com/2013/07/what-is-geometric-distortion-and-why.html

Geometric Distortions – radial distortion 96
4. Image formation
Barrel distortion
http://www.scandig.eu/Kameraverzerrung.html

Geometric Distortions 97
Distortion of point (u,v) is:

4. Image formation
 u   u ( k1 r 2
+ k 2 r 4
+ k 3 r 6
+ ... )  
+
2 p1uv + p2 ( r 2
+ 2u 2
) 

 =
  v   v ( k1r 2 + k2 r 4 + k3r 6 + ...)   p1 ( r 2 + 2v 2 ) + 2 p2uv 
radial tangential

Geometric Distortions 98
Tangential and radial distortion

4. Image formation
https://www.google.com/url?sa=i&rct=j&q=&esrc=s&source=i
mages&cd=&cad=rja&uact=8&ved=2ahUKEwiy5qyJqPLgAhUGJ
1AKHaxVC44Qjhx6BAgBEAI&url=https%3A%2F%2Fwww2.infor
matik.hu-
berlin.de%2F~stuebing%2Fpublic%2FStudienarbeit_Sascha_Stu
ebing_2010.pdf&psig=AOvVaw1bTNeojvyH8AGSAGz_Ywkg&us
t=1552126514887415

Classical camera calibration 99
4. Image formation
http://www.signkwik.com/security
-camera-calibration.html

Bouguet’s camera calibration 100
Detailed examples of camera calibration in matlab on:

http://www.vision.caltech.edu/bouguetj/calib_doc/#examples
4. Image formation
http://www.vision.caltech
.edu/bouguetj/calib_doc/
htmls/example.html

Fisheye lens camera 101
4. Image formation
https://upload.wikimedia.
org/wikipedia/commons/
c/c3/The_Squirrels_0048.
jpg

P = (X, Y, Z)
4. Image formation
z R x
Θ Φ
u
P‘ = (u,v)
{C}
Camera origin r
Φ
v
Mapping of projection types

4. Image formation
Stereographic r = 2 f tan ( )
Equidistant (Equiangular) r = f
Equisolid  
r = 2 f sin  
2
orthographic r = f sin ( )

4. Image formation
Gnomonic stereographic equidistant

equisolid orthographic
http://www.fisheyelens.de/html/def.html
4. Image formation
Gnomonic stereographic equidistant

equisolid orthographic
http://www.fisheyelens.de/html/def.html
Catadioptric camera 106
4. Image formation
https://commons.wikimedia.org/wiki/File:Phoe
nix_500mm_f8_mirror.jpg
https://www.math.drexel.edu/~
ahicks/design/what-are.html

Catadioptric camera 107
4. Image formation

and Control]

Spherical camera 108
4. Image formation

and Control]

Unified imaging model 109
4. Image formation

and Control]

Unified imaging model 110
4. Image formation
Imaging L m
Perspective 0 f
Stereographic 1 f
Fisheye >1 f
Catadioptic (elliptical, 0<ε<1) 2 2 ( 2 p − 1)
1+  2 1+  2
Catadioptic (parabolic, ε=1) 1 2p-1

Catadioptic (hyperbolic, ε>1) 2 2 ( 2 p − 1)
1+  2 1+  2

Chapter 5 –
Image processing

5. Image processing

Some monadic image operations 113

and Control]
5. Image processing
thresholding histogram normalized
Gramma correction brightness increase posterization

diadic image operations 114
Chroma-keying:
5. Image processing

Smoothing kernels 115
5. Image processing

and Control]

Edge detection 116
Iu = Ku * I Iv = Kv * I
5. Image processing
Maginitude of gradient: Direction of gradient

m = Iu 2 + Iv 2  = a tan 2 ( I u , I v )
 0.125 0 −0.125   0.125 0.25 −0.125 

   
=  0.25 0 −0.25  K S , xT = 0 0 0  = KS , y
Sobel: KS ,x
 0.125 0.25 −0.125 
 0.125 0 −0.125   

1 0 −1 1 1 1
Prewitt:   
= 0 0 0 

K P,x = 1 0 −1 K P, y
 −1 −1 −1
1 0 −1  

1 0   0 1
Roberts: K R, x =  K R, y =  
 0 −1   −1 0 
Edge detection with smoothing 117
Derivate of
I u = K * G ( ) * I
Gaussian:
5. Image processing
DoG (Derivative of gaussian)

u ²+v ²
u −
DoGu ( u , v ) = − e 2 ²
2 4
Additional information for edge detection with matlab on:

https://de.mathworks.com/help/images/ref/edge.html#d117e60716

Similarity functions for template matching 118
SAD (sum of absolute differences):

s =  I1 ( u, v ) − I 2 ( u, v )
5. Image processing
ZSAD (zero-offset SAD):

( ) (
s =  I1 ( u, v ) − I 1 − I 2 ( u, v ) − I 2 )
SSD (squared differences): (
s =  I1 ( u, v ) − I 2 ( u, v )
2
)
(( ) (
s =  I1 ( u, v ) − I 1 − I 2 ( u, v ) − I ))
2
ZSSD (zero-offset SSD):
NCC (normalized cross-correlation): s=

 I ( u, v ) I ( u, v )
1 2
 I ( u, v )  I ( u, v )
2 2
1 2
ZNCC (zero-offset NCC):

s=
 ( I ( u, v ) − I ) ( I ( u, v ) − I )
1 1 2 2
 ( I ( u, v ) − I )  ( I ( u, v ) − I )
2 2
1 1 1 1

Important Morphological operations 119
Morphological operation Matlab function Realization

Erosion imerode O=IӨS
5. Image processing
f(·)=min(·)
Dilatation imdilate O=IꚚS
f(·)=max(·)
Open imopen O = I ○ S = (I Ө S) Ꚛ S
Erosion then dilatation
Close imclose O = I ● S = (I Ꚛ S) Ө S
Dilatation then erosion

Usage of Morphological operations 120
Example Skeleton operation with bwskel:

5. Image processing
original binary image skeleton detection
https://de.mathworks.com/help/images/ref/bwskel.html

Chapter 6 –
Image feature extraction

6. Image feature extraction

Image segmentation 125
classification
representation
features

Binary classification via grey level 126
0, if I u , v   t
C u , v  = 
1, if I u , v   t
t..threshold, C..Class, I…intensity
Local threshold by niblack:
t u , v  =  (W ) − k (W )

Moments of an image 127
The Moment m is:

mpq =  u v I u , v 
p q
( u ,v )I
where p,q are order of moment
0. moment:
m00 =  I u, v 
( u ,v )I
Center of mass:
m10 m01
uc = vc =
m00 m00
Central Moments of an image 128
 pq =  ( u − uc ) ( v − vc ) I u, v
p q
( u ,v )I
→independent from translation

Moments of inertia:
m10 2 m012
20 = m20 − 02 = m02 −
m00 m00
Symmetric matrix:
 20 11 
J =
 11 02 
Central Moments of an image 129
Sides of ellipse: 2 1
a=2 b=2
m00 m00
λ..eigenvalue of J
Aspect ration: a
r=
b
Angle of orientation:
x22
 = tan −1
x21
x2… last column of eigenvector
Other Moments 130
ij
normalized central moments: ij =  i+ j 
1+ 
  2 
00
Hu-moments:
H1 =  20 + 02 H 2 = (20 −02 ) + ( 211 )
2 2
H3 = (30 − 312 ) + ( 321 −03 ) H 4 = (30 +12 ) + (21 +03 )

2 2 2 2
H5 = (30 − 312 )(30 + 12 ) (30 + 12 ) − 3 (21 + 03 )  + ( 321 −03 )(21 + 03 ) 3 (30 + 12 ) − (21 + 03 ) 
2 2 2 2
   
H 6 = (20 −02 ) (30 + 12 ) − (21 + 03 )  + ( 411 )(30 +12 )(21 +03 )
2 2
 
H 7 = ( 321 −03 )(30 −12 ) (30 + 12 ) − 3 (21 + 03 )  − (30 − 312 )(21 + 03 ) 3 (30 + 12 ) − (21 + 03 ) 
2 2 2 2
   
Ming-Kuei Hu, "Visual Pattern Recognition by Moment Invariants",

doi:10.1109/TIT.1962.1057692
Shape from Perimeter 131
Chain code Freeman chain Crack code
p=14 p=12.2 p=18

Shape from circularity 132
Circularity:
4 m00
= 2
p
p…perimeter length
m00 …0.order moment (area for binary images)

Detecting lines 133

v = −u arctan ( ) +
sin ( )
 max = −  min = max (W , H )    −  max ,  min )   

  − , 
 2 2
θ η u
η θ
ρ
η
θ
v
Detecting points 134
interest point
interest point
any point on line (corner point)

Moravec‘s corner detection 135
 ( I u +  + i , v +  v + j  − I u + i , v + j  )
s ( u , v,  u ,  v ) =
2
u
( i , j )W
C ( u, v ) = min s ( u, v, u ,  v )
u ,v

Harris corner detection 136
s ( u , v,  u ,  v ) =  W ( i , j ) ( I  u +  u + i , v +  v + j  − I u + i , v + j  )
2
( i , j )W
with
I  u +  u + i , v +  v + j   I  u + i , v + j  + I u  u + i , v + j   u + I v u + i , v + j   v
results
s ( u , v,  u ,  v ) =  W ( i, j ) ( I u u + i, v + j   u + I v u + i , v + j   v )
2
( i , j )W
s ( u, v,  u ,  v ) =  u2  W ( i, j ) I u 2 u + i, v + j  +  v2  W ( i, j ) I v 2 u + i, v + j  +  u  v  W (i, j ) I u + i, v + j I u + i, v + j 
u v
( i , j )W ( i , j )W ( i , j )W
 u 
s ( u , v,  u ,  v ) = (  u v ) A 
 v 
  W ( i, j ) I u 2 u + i, v + j   W ( i, j ) I u u + i, v + j I v u + i , v + j  
 ( i , j )W ( i , j )W 
A= 
  W ( i, j ) I u u + i, v + j I v u + i, v + j   W ( i , j ) I v 2 u + i , v + j  
 (i , j )W ( i , j )W 
Harris corner detection (2) 137
can convert to
 G ( I  I u2 ) G ( I  I u I v ) 
A= 
 G ( I  I u I v ) G ( I  I v2 ) 
 
CH ( u , v ) = det ( A ) − k tr ( A )
with
 n

det ( A) = 12 =   sgn (  )  ai ,  (i ) 
 i =1 
tr ( A ) = 1 + 2 =  ( a jj )
n
j =1

SIFT- Scale-invariant feature transform 138
Gaussian sequence
G(σ) G(σ) G(σ)
- - -
Laplacian of Gaussian (LoG) sequence

SIFT- Scale-invariant feature transform (2) 139
844x1087 422x544 211x272 106x136

Point features by matlab 140
https://de.mathworks.com/help/vision/object-detection-using-features.html?s_tid=CRUX_lftnav

Chapter 7 –
Multi Vision and Vision-
Based Control

7. Multi Vision and VBC

Geometry of stereo vision 143
P = (X, Y, Z)
z
x
P1‘ = (X1‘, Y1‘ ,Z1‘)
P2‘ = (X2‘, Y2‘ ,Z2‘)

H
xC
zC {C1}
y W xC
yC
{C2}
yC
Corresponding Points 144
Detect distinctive points

Simplification of problem
Describe points
Descriptor vector
Find similar points

Epipolar geometry 145
T
P2 ' F P1 ' = 0
F K 2 −T t  X R K1−1
l1 F T P2 '
l2 F P1 '

Epipolar geometry – two cameras parallel 146

Epipolar geometry – two cameras in series 147

Planar homography 148

Sparse stereo 149

Sparse stereo 150
https://www.uio.no/studier/emner/matnat/its/nedlagte-emner/UNIK4690/v16/forelesninger/lecture_6_3_stereo_processing.pdf

Dense stereo 151

Dense stereo 152
https://www.uio.no/studier/emner/matnat/its/nedlagte-emner/UNIK4690/v16/forelesninger/lecture_6_3_stereo_processing.pdf
bx bx
Z= f X = xL
bx
Y = yL
d d d

Bundle Adjustment 153

Structured light 155

and Control]

Mosaicing 156
https://de.mathworks.com/matlabcentral/fileexchange/46148-image-mosaicing
http://leblogdex.overblog.com/image-mosaicing

Visual odometry (VO) 158
Camera motion
Video (Sequence of images)
https://de.mathworks.com/help/vision/ug/monocular-visual-odometry.html

Visual servoing 159
Eye in hand Eye to hand
„Vision Based Attitude Control“ , Maroš Hladký, Luleå University of Technology

Position-Based Visual Servoing 160

and Control]

Position-Based Visual Servoing 162

Image-Based Visual Servoing 163
Observing camera motion Controlling feature motion

P = J P ( P, K , C )
P = J P ( P ', Z ) ( ) 
+
 J P , Z
 P 1 1
 =   
 ( P '* − P')
 vx 
 fˆ
−
u   Z
0
u uv
Z fˆ
−


u 2

fˆ +  v   
fˆ 
  vy 
  vz 
 (
 J P , Z
 P n n ) 


 =  w 
v   fˆ v ˆ v 2  x
−u   
uv
 0 − Z Z f + fˆ −
fˆ
   wy 
w 
 z

Image-Based Visual Servoing 164

Depth estimation 165
Z=3 (right depth)

Z=1 (too low depth)

and Control]

Depth estimation 166
Z=3 (right depth)

Z=10 (too big depth)

and Control]

Holonomic mobile robot 167
 fˆ 
− 0 v   vx 
P =  Z   vy 
 fˆ   
 0 − −u   wz 
 Z 
https://www.servomagazine.com/magazine/article/a-look-at-holonomic-
locomotion


Robot Vision SS21

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Robot Vision SS21

Uploaded by

Copyright:

Available Formats

Robotic Vision

Prof. Dr.-Ing. Maria Schweigel

SS 2021 Robotic Vision – Prof. Dr.-Ing. Maria Schweigel 1

SS 2021 Robotic Vision – Prof. Dr.-Ing. Maria Schweigel

Chapter 2 – Coordinate systems and transformations

Chapter 3 – Physics of Light and color

Chapter 4 - Image formation

Chapter 5 – Image processing

Chapter 6 – Image feature extraction

SS 2021 Robotic Vision – Prof. Dr.-Ing. Maria Schweigel

Chapter7 - Vision-Based Control

SS 2021 Robotic Vision – Prof. Dr.-Ing. Maria Schweigel

‘Vision Based Autonomous Robot Navigation’,

‘Robotics – Vision and Control‘, Peter Corke,

SS 2021 Robotic Vision – Prof. Dr.-Ing. Maria Schweigel

SS 2021 Robotic Vision – Prof. Dr.-Ing. Maria Schweigel 8

Electro the Smoking Robot

SS 2021 Robotic Vision – Prof. Dr.-Ing. Maria Schweigel

SS 2021 Robotic Vision – Prof. Dr.-Ing. Maria Schweigel

SS 2021 Robotic Vision – Prof. Dr.-Ing. Maria Schweigel

SS 2021 Robotic Vision – Prof. Dr.-Ing. Maria Schweigel

SS 2021 Robotic Vision – Prof. Dr.-Ing. Maria Schweigel

SS 2021 Robotic Vision – Prof. Dr.-Ing. Maria Schweigel

SS 2021 Robotic Vision – Prof. Dr.-Ing. Maria Schweigel

ARMA by KIT (Karlsruher

• energy supply: battery

→ARMA should be able to act

SS 2021 Robotic Vision – Prof. Dr.-Ing. Maria Schweigel

SS 2021 Robotic Vision – Prof. Dr.-Ing. Maria Schweigel

SS 2021 Robotic Vision – Prof. Dr.-Ing. Maria Schweigel

Medizin robot Da Vinci, the only system with a wide

Miro by German Aerospace Center (DLR)

SS 2021 Robotic Vision – Prof. Dr.-Ing. Maria Schweigel

BoniRob by University Osnabrück and BOSCH

SS 2021 Robotic Vision – Prof. Dr.-Ing. Maria Schweigel

Mars Science Lander by NASA

https://mars.nasa.gov/news/8371/curio WildCat by Boston

SS 2021 Robotic Vision – Prof. Dr.-Ing. Maria Schweigel

SS 2021 Robotic Vision – Prof. Dr.-Ing. Maria Schweigel

Elektro the Smoking Robot

SS 2021 Robotic Vision – Prof. Dr.-Ing. Maria Schweigel

„A Robot is a goal oriented machine

SS 2021 Robotic Vision – Prof. Dr.-Ing. Maria Schweigel

- from greek word:

- autonomous: self-sufficient, independent, act

→ An autonomous system (AS) acts

SS 2021 Robotic Vision – Prof. Dr.-Ing. Maria Schweigel

SS 2021 Robotic Vision – Prof. Dr.-Ing. Maria Schweigel

SS 2021 Robotic Vision – Prof. Dr.-Ing. Maria Schweigel

SS 2021 Robotic Vision – Prof. Dr.-Ing. Maria Schweigel

SS 2021 Robotic Vision – Prof. Dr.-Ing. Maria Schweigel

optimizing process (step by step adaption of classifier

→ tree data sets necessary:

SS 2021 Robotic Vision – Prof. Dr.-Ing. Maria Schweigel

e – mean square error

N – number of training cycle

SS 2021 Robotic Vision – Prof. Dr.-Ing. Maria Schweigel

SS 2021 Robotic Vision – Prof. Dr.-Ing. Maria Schweigel

What and from

translated into the AS

Which security gaps

SS 2021 Robotic Vision – Prof. Dr.-Ing. Maria Schweigel

Maps Path planning