Computer Vision - Mosig

Light, colour, camera

Sergi Pujades

Original slides by : Jean-Sebastian Franco

 What is an image?

• An image is a 2D signal
▪ Usually this image depicts a 3D scene

• From a mathematical perspective

▪ A matrix of values that represent a signal
▪ One can manipulate a signal with tools

• From a human perspective (perception)

▪ Has semantic information
▪ We interpret the information rather than focus on the magnitudes

“A picture is worth a thousand words" - para-phrased from Henrik Ibsen 1906

Slice of an image - 1D signal
Image as a 2D surface

becomes

becomes
Image as pixel matrix

Gray levels - 8 bits:

0 - black
255 - white
Image formation process
Image formation process
 Observed phenomena

• Light: plays a fundamental role in 3D perception

• No light - no image!

Diffuse
reflection
Specular
reflection
Shadow
 Light source

• Characteristics
▪ Point light source or extended light source
▪ Spectrum : amplitude (photons) for each wavelength

Source : Wikimedia
 Light source :

some spectrums

Sodium vapor lamp

(monochromatic) Solar spectrum

Mercure vapor lamp White light : continuous spectrum, energy

evenly distributed among the wavelengths
 Light source :

some spectrums

Sodium vapor lamp

(monochromatic)
 Light source :

some spectrums

Sodium vapor lamp

(monochromatic)

Two Honda Fits under low-pressure sodium lamps. Both appear black,
even though the car on the left is bright red, while the car on the right is actually black.
“Sodium-vapor lamp” Wikipedia article
 Light sources

• Ray characterisation
▪ One direction
▪ Amount of energy + spectrum
Surface reflectance
 Surface reflectance

• What happens when a light ray hits the surface

▪ absorption (ex. black surface)
▪ reflexion - refraction (ex. mirror, reflector)
▪ diffusion (ex. milk)
▪ fluorescence (ex. some powder - “glow in the dark”)
▪ transmission and emission (ex. Human skin)

• Complex phenomena, many unknown parameters by the observer, 


difficult to estimate them

• Most surfaces can be “fairly well” approximated by simple(r) models

 Surface reflectance

• First simplified hypothesis:

▪ No fluorescent surfaces
▪ “cold” objects (they do not emit light)
▪ All light emitted from a surface point is formed solely by the light
arriving at this point

Not always true !

 Standard model : BRDF

• Bi-directional Reflectance Distribution Function

• Models the ratio of energy for each wavelength 𝜆
▪ Incoming from direction 𝑣^𝑖
▪ Emitted towards direction 𝑣^𝑟
Local frame on the surface

on a point

( 𝑥 𝑦 )
^ , 𝑑^ , 𝑛^
𝑑
𝑓𝑟(𝜃𝑖, 𝜙𝑖, 𝜃𝑟, 𝜙𝑟, 𝜆)
 BRDF - reciprocity

• Bi-directional Reflectance Distribution Function

• Reciprocity :
𝑓𝑟(𝜃𝑖, 𝜙𝑖, 𝜃𝑟, 𝜙𝑟, 𝜆) = 𝑓𝑖(𝜃𝑟, 𝜙𝑟, 𝜃𝑖, 𝜙𝑖, 𝜆)

Local frame on the surface


on a point

( 𝑥 𝑦 )
^ , 𝑑^ , 𝑛^
𝑑
 BRDF - isotropy

• Additional simplification: isotropy

▪ No “favourite” direction, invariance around 𝑛^

𝑓𝑟(𝜃𝑖, 𝜙𝑖, 𝜃𝑟, 𝜙𝑟, 𝜆) = 𝑓𝑟(𝑣^𝑖, 𝑣^𝑟, 𝑛,

^ 𝜆
)
Local frame on the surface

on a point

( 𝑥 𝑦 )
^ , 𝑑^ , 𝑛^
𝑑
 BRDF - isotropy

• Hypothesis does not hold for some particular surfaces

 BRDF - energy

• How much energy leaves the surface?

▪ Integral over all directions

𝐿𝑟(𝑣^𝑟, 𝜆) = 𝐿𝑖(𝑣^𝑖, 𝜆)𝑓𝑟(𝑣^𝑖, 𝑣^𝑟, 𝑛, )

∫
^ 𝜆 𝑐𝑜𝑠𝜃𝑖𝑑𝑣^𝑖
𝑖
 BRDF - energy

• How much energy leaves the surface?

▪ Integral over all directions

𝐿𝑟(𝑣^𝑟, 𝜆) = 𝐿𝑖(𝑣^𝑖, 𝜆)𝑓𝑟(𝑣^𝑖, 𝑣^𝑟, 𝑛, )

∫
^ 𝜆 𝑐𝑜𝑠𝜃𝑖𝑑𝑣^𝑖
𝑖

Foreshortening
 BRDF - energy

• Particular case: set of point sources

▪ Discrete sum

𝐿𝑟(𝑣^𝑟, 𝜆) = 𝐿𝑖(𝜆)𝑓𝑟(𝑣^𝑖, 𝑣^𝑟, 𝑛, )

^ 𝜆 𝑐𝑜𝑠𝜃𝑖
∑
𝑖
 BRDF classic 1 : Lambert assumption

• Diffuse surface: uniform in all directions

▪ paper, milk, matt paint...

• The BRDF is a constant function

𝑓𝑑(𝑣^𝑖, 𝑣^𝑟, 𝑛, )
^ 𝜆 = 𝑓𝑑(𝜆)
 BRDF classic 2 : specular

• A completely reflective material (mirror)

) {
^𝑟 = 𝑠^𝑖
𝑓𝑟(𝑣^𝑖, 𝑣^𝑟, 𝑛,
^ 𝜆 = 1 if 𝑣
0 else
 BRDF classique 2 : specular

• For a specular material: central lobe on 𝑠^𝑖

▪ Phong
𝑓𝑠(𝜃𝑠, 𝜆) = 𝑘𝑠(𝜆)𝑐𝑜𝑠𝑘𝑒𝜃𝑠

▪ Torrence-Sparrow

𝑓𝑠(𝜃𝑠, 𝜆) = 𝑘𝑠(𝜆)𝑒 −𝑐𝑠2𝜃𝑠2

 Di-chromatic BRDF : diffuse + specular

𝐿𝑟(𝑣^𝑟, 𝜆) = 𝐿𝑠(𝑣^𝑖, 𝑣^𝑟, 𝑛, ) (

^ 𝜆 + 𝐿𝑑 𝑣^𝑖, 𝑣^𝑟, 𝑛,
^ 𝜆
)
  = 𝑐𝑠(𝜆)𝑚𝑠(𝑣^𝑖, 𝑣^𝑟, 𝑛^) + 𝑐𝑑(𝜆)𝑚𝑑(𝑣^𝑖, 𝑣^𝑟, 𝑛^)

Specular component Diffuse component

• specular component: same light spectrum 𝑐𝑠(𝜆)

• diffuse component: “filtered” spectrum (by the material pigments) 𝑐𝑑(𝜆)
• geometric terms ms and md
Image formation process
Pipeline in a digital camera
Image formation
 How to create images?

• Image = 2D projection of a 3D world

 Image acquisition

• Visual perception

Source : Wikimedia
 Image acquisition

• Optical role : isolate the light rays

Brunelleschi, XVth Century; architect, Florence.

Camera obscura: XVIth Century.
Limits of pinhole cameras
 Limits of pinhole cameras

• Rayleigh
▪ Ideal diameter d
▪ Focal distance f
▪ Wavelength 𝜆

𝑑 = 1.9 𝑓𝜆

Source : Dominic Alves

 Limits of pinhole cameras

Source : Leica
 Optical system

• Groups of multiple lenses

Source : diglloyd.com
 Optical system

• Groups of multiple lenses

• Can be described as a convergent lens

1 1 1
+ =
𝑧𝑜 𝑧𝑖 𝑓
Descartes relationship
 Optical system

• Perfect lens: hypothesis

▪ A point in the scene corresponds to a point in the image
▪ All image points are in the same plane
▪ Constant scaling

• Classic challenges: hypothesis not fulfilled

▪ Chromatic artefacts (fringing, vignetting)
▪ Geometric artefacts
 Optical system: classic challenges

• Chromatic artefacts (fringing)

▪ diffraction: wavelength dependant

Source : wikimedia
 Optical system: classic challenges
Szeliski
• Vignetting chap. 2.2
▪ energy = fonction of the ray length

4(𝑓)
2
𝜋 𝑑
𝐸=𝐿 𝑐𝑜𝑠4𝛼
 Optical system: classic challenges

• Geometric Distorsion

Barrel

Pincushion

“Straight lines have to be straight: …” Devernay, Faugeras.

Machine Vision and applications 2001
Image formation
 Photo sensitive surfaces

• “Argentique” - “silver process”

▪ First photo : Nicéphore Niépce (1826)
▪ Daguerreotypes (1839)
▪ Color photo (Cros, Ducos de Hauron, 1869, and Lumière Brothers 1903)
▪ Photographic film (Eastman, 1889)
▪ Cinema (Lumière Brothers, 1895)

• Analog
▪ Television (Baird, Farnsworth, Zworykin, 1920s)

• Digital
▪ CCD (1970s)
▪ CMOS (1980s)
Point de vue du Gras, Saint-Loup de Varenne, 1826

oldest surviving camera photograph

Pipeline in a digital camera
 CCD (charged coupled device)

CMOS (complementary metal oxide semi-conductor)

• Array of photo-sites
• Electric charge is proportional to the number of received photons
• Received charge is transferred towards a bus
• CMOS : circuit with amplification and obturation per photo-site: “more
intelligent sensor”

Source : gatan.com
 CCD / CMOS

Source : Sony
 Rolling shutter

• Each line of the image is read sequentially in time

▪ Usually in inexpensive CMOS
▪ Distortion appears if the camera / object is moving
 Color perception

• RGB: additive color space, CMY : subtractive color space

 Color perception

• Why 3 primary colors?

Source : wikimedia Retina Fovea

 Color perception

• Rods:
▪ Achromatic perception of lights
▪ Peripheral retina
▪ Pigmentation: rhodopsin, sensitive to all visible spectrum (pic on green)

• Cones
▪ Chromatic vision
▪ Located on the fovea
▪ 3 types de cônes/pigmentation
• Cyanolabe (pic 445nm)
• Chlorolabe (pic 535nm)
• Erytholabe (pic 570nm)
 Color perception: the mantis shrimp

(most complex visual system ever discovered)

• 16 types of cones :
▪ 12 for the color (6 in UV range)
▪ 4 for the polarisation (among circular)
• Trinocular eye: depth perception

Source : wikimedia
 Color specification

• Indirect perception of the light spectrum through 3 chromatic stimuli

• First verified and quantified by the CIE (Commission Internationale de l’Éclairage -
International Commission on Illumination) on an a high number of subjects:
• R=700nm, G=546.1nm, B=435.9nm
• Correspondance functions r(λ), g(λ), b(λ)
 Color specification

• CIE givres an XYZ space to avoid the red negative addition

𝑋 0.49 0.31 0.20 𝑅
[𝑍 ] 0.17697 [𝐵]
1
𝑌 = 0.17697 0.81240 0.01063 𝐺
0.00 0.01 0.99

• This is the base for all “standard” color spaces: RGB, YUV...
 Color specification

• Chromatic coordinates
▪ Coordinates without intensity
▪ Only chromatic component
𝑋
𝑥 = 𝑋+𝑌+𝑍
𝑌
𝑦 = 𝑋+𝑌+𝑍
𝑍
𝑧 = 𝑋+𝑌+𝑍
 Color perception in a camera

• Deviation / dispersion prism

▪ 3 CCD sensors (or CMOS - historically tubes)
▪ Precise alignment, high quality filter
▪ Expensive

Source : wikimedia
 Color perception in a camera

• Bayer filter
▪ Individual filter for each pixel (RGGB, RGCB)
▪ Photo- sites not co-localized
▪ Color in each pixel: interpolation (demosaicing)

Source : wikimedia
 Color perception in a camera

• How does it work? Integration over spectrums

𝑅 = ∫ 𝐿(𝜆)𝑆𝑅(𝜆)𝑑𝜆

𝐺 = ∫ 𝐿(𝜆)𝑆𝐺(𝜆)𝑑𝜆

𝐵 = ∫ 𝐿(𝜆)𝑆𝐵(𝜆)𝑑𝜆

• Filters do not need to exactly match the XYZ signals. Each

manufacturer has it’s own filters and provides the conversion towards
the standard RGB space.
 Sensor artefacts

• Noise
▪ Salt and pepper
▪ Thermic noise: Gauss, Poisson
▪ Amplified by ISO gain

No filter With filter

• Aliasing
Pipeline in a digital camera
 Gamma transformation

• Human perception - non-linear with respect to number of photons

▪ Linear intensity: how many photos reached the sensor?

▪ Gamma correction allows to have more bits in the dark areas and
less in the bright areas
 Gamma transformation

• Transformation of the intensity

▪ Use of a power function
 Gamma transformation

By X-romix 10:00, 7 June 2008 (UTC), Updated by --Rubybrian (talk) 14:25, 14 September 2010 (UTC);
Photographer: Toni Frissell - This file was derived from: Weeki Wachee spring 10079u.jpg, CC BY-SA 4.0, https://
commons.wikimedia.org/w/index.php?curid=4176109
 JPEG compression

• Compression using 8x8 blocks

▪ Lossy compression
▪ Adds compression noise
 Color models: RGB

Cyan
(0,255,255)

White
Magenta (255,255,255)
Blue (255,0,255)
(0,0,255)

Yellow
Green (255,255,0)
Black (0,255,0)
(0,0,0)
Red
(255,0,0)
 Color models: RGB
• Additive model
▪ Mixing wavelengths
▪ Designed for graphical screens

• Some colours can not be represented

Source : wikimedia
 Color models: CMY

Cyan

White
(0,0,0)
Magenta
Blue

Yellow
Green
Black

Red
 Color models: CMY
• Subtractive model
▪ Mixing of pigments
▪ Used in printers (CMYK)

• Why K (key) ?
▪ Opacity of black

• Some colors can not be represented 𝐶 1 𝑅

[ 𝑌 ] [1] [𝐵]
𝑀 = 1 − 𝐺
 Color models: HSV (or HSB)
• Hue (“intrinsic color”) – Saturation - Value (or brightness)
▪ decomposition between luminance / chrominance
▪ V = max (R,G,B)

Source : wikimedia
 Color models: YUV / YIQ

• 1 luma component + 2 chroma components

• Used for TV broadcasting
▪ YUV (PAL / SECAM)
▪ YIQ (NTSC)
▪ Y only for the black & white TV

𝑌 0.299 0.587 0.114 𝑅

[𝐵]
𝐼 = 0.596 −0.274 −0.322 𝐺
𝑄 0.211 −0.523 0.312

𝑌′ 0.299 0.587 0.114 𝑅

[𝑉] [𝐵]
𝑈 = −0.14713 −0.28886 0.436 𝐺
0.615 −0.51499 −0.10001

Source : wikimedia
 Application: 

histogram based object detection
• Histogram h(R,G,B)
▪ Number of occurrences of a R,G,B triplet
▪ example : 32 x 32 x 32 = 32k cells

Light source color

Background 

color
Object 

Transition area color

Object color

Limitations, possible improvements?

 Application: 

histogram based object detection
• The histogram can be seen as an approximation of the 

distribution P(R,G,B|objet)

1
• P(R,G,B | objet) ~ N h(R, G, B)
pixels

▪ The bigger Npixels, the better the approximation

• Ideally:
▪ Robust to illumination changes
▪ Reduction of the size of the histogram

• Focus on chromatic components

▪ For instance UV from YUV
 Application: 

histogram based object detection
• Two distributions :
1
• Pixels of the object: P(U, V | object) ≈ h
Nobject object
(U, V )
1
• Pixels of the image: P(U, V ) ≈ h
Nimage image
(U, V )

• What we are looking for: P(object | U, V )

• Using Bayes rule:
P(object)
• P(object | U, V ) = P(U, V | object) P(U, V)
Nobject
• P(object) = Nimage

hobject(U, V)
• P(object | U, V ) = himage(U, V)
 Application: robotic detection

• Exercise
▪ Suggest a protocol and algorithm to detect the ball

Source : RoboCup 2009

 Application: robotic detection

• Segmentation HSV

balle

sol

Hue
robot bleu

blanc ...

Saturation
 Application: robotic detection

• Segmentation YUV

sol

V
robot bleu
balle

blanc ...