Image Representation and

Formation
Instructor
Dr. Nguyen Duc Dung
nddung@hcmut.edu.vn
 Acknowledgements
• This lecture reused materials from
– Image Representation and Formation (by Dr. Nguyen Ho Man
Rang)
– CO3057: Image Processing and Computer Vision (by Dr. Le
Thanh Sach from HCMUT)
– CS194-26: Image Manipulation and Computational Photography
(by Prof. Alexei Efros from Berkeley)

Advanced Image/Video Processing: Chapter 01 – Image Representation and Formation Slide: 2

 Images: What are about?
• Images:
– Rectangular shapes of shades or colors

Advanced Image/Video Processing: Chapter 01 – Image Representation and Formation Slide: 3

 Digital Images: What are about?
• Digital Images: Point/Pixel
– Space (rectangular shape): is digitalized into
matrix of rectangular cells which is called
pixels.

Advanced Image/Video Processing: Chapter 01 – Image Representation and Formation Slide: 4

 Digital Image Representation
Width
• An image f(x,y) is represented
ROI
as an 2D Array, matrix

Height
• Width =
– number of pixels in x-direction
• Height =
– number of pixels in y-direction
• Size (width ´ height)
• ROI = region of interest
– To reduce the amount of data

Advanced Image/Video Processing: Chapter 01 – Image Representation and Formation Slide: 5

 Digital Image Representation
• Pixel representation:
– Gray images:
• Pixel’s values are on or inside of [0,255]
• Pixel’s value = 0: black
• Pixel’s value = 255: white

Advanced Image/Video Processing: Chapter 01 – Image Representation and Formation Slide: 6

 Digital Image Representation

Advanced Image/Video Processing: Chapter 01 – Image Representation and Formation Slide: 7

 Digital Image Representation
• Pixel representation:
– Color images:
• A color = a combination of a red, a green, and a blue shade
• An pixel is represented by a triple [R,G,B].
• An image has 3 planes: R, G, and B-plane

Advanced Image/Video Processing: Chapter 01 – Image Representation and Formation Slide: 8

 Spatial Image Resolution:
• Resolution
– The size of an area in a scene that is represented
by one pixel in the image
• Different Resolutions are possible (256x256….16x16)

• Lower resolution leads to data reduction!

Advanced Image/Video Processing: Chapter 01 – Image Representation and Formation Slide: 9

 Image/Video Processing: What is about?

• Multi-kinds of processing tasks:

– Simple: Scaling, rotating an image
– Others:
• Enhancement:
– Obtain a better image from a bad image.
• Restoration
– Recover from damaged images.
• Compression
– Reduce the size of images with/without loss of the quality.
• Segmentation
– Obtain interested objects from the whole image.
• Recognition
– Recognize, classify, or identify objects.
• …

Advanced Image/Video Processing: Chapter 01 – Image Representation and Formation Slide: 10

 Why should students study
image/video processing?
• Why should students study image/video
processing?
– Usefulness/Applications
– Challenge
– Active research area

Advanced Image/Video Processing: Chapter 01 – Image Representation and Formation Slide: 11

Applications
 Examples: Image Correction

• Needed when image data is erroneous:

– Bad transmission
– Bits are missing: Salt & Pepper Noise

Advanced Image/Video Processing: Chapter 01 – Image Representation and Formation Slide: 13

 Image Deblurring: Motion Blur

• Can be used when a camera or object is

moved during exposure

Advanced Image/Video Processing: Chapter 01 – Image Representation and Formation Slide: 14

 Deblurring

• Can be used when the camera was not

focused properly!!

Advanced Image/Video Processing: Chapter 01 – Image Representation and Formation Slide: 15

 Image manipulation
• Image improvement, e.g. too dark image

• Rotate + scale

Advanced Image/Video Processing: Chapter 01 – Image Representation and Formation Slide: 16

 Medical Image Processing

• Image Processing is widely used

• E.g. Analysis of microscopic images

Advanced Image/Video Processing: Chapter 01 – Image Representation and Formation Slide: 17

 Medical Image Processing

• MR/CT Imaging of a human body

• Use for Brain Surgery

Advanced Image/Video Processing: Chapter 01 – Image Representation and Formation Slide: 18

 PhD 2003 on Texture and Action Synthesis
Inspired Photoshop’s “Context-aware Fill” and
Microsoft “Smart Erase” buttons:

© Antonio Criminisi
 Colorizing Black and White Photos

Zhang et al, “Colorful Image Colorization", ECCV 2016

CycleGAN
Photo to Painting
And back…
Horse to zebra
Failure case
 Image/Video Processing: Active
research area
• Many research groups around the world working
on Image Processing and Computer Vision
• Many conferences and transactions organized
for the field
– Conferences: CVPR, ICCV, ECCV, ICIP
– Journals: TIP, TPAMI
– …

Advanced Image/Video Processing: Chapter 01 – Image Representation and Formation Slide: 31

 • How can we acquire images?

Advanced Image/Video Processing: Chapter 01 – Image Representation and Formation Slide: 32

 Where does an image come from?

Advanced Image/Video Processing: Chapter 01 – Image Representation and Formation Slide: 33

 Where does an image come from?
Electromagnetic spectrum

Advanced Image/Video Processing: Chapter 01 – Image Representation and Formation Slide: 34

 Where does an image come from?

Charged coupled device

CCD-chip

Advanced Image/Video Processing: Chapter 01 – Image Representation and Formation Slide: 35

 Where does an image come from?

• Integration over
time
– Exposure time
• Shutter time
– Motion blur
– Maximum charge
• Saturation
Advanced Image/Video Processing: Chapter 01 – Image Representation and Formation Slide: 36
 The lens

• A lens focuses a bundle of rays to one point

• Parallel rays pass through a focal point F at a distance f beyond
the plane of the lens. f is the focal length
• O is the optical center
• F and O span the optical axis/principal axis
Advanced Image/Video Processing: Chapter 01 – Image Representation and Formation Slide: 37
 Focus

1 1 1
+ =
g b f

In focus Not in focus

(vis på oh)
Advanced Image/Video Processing: Chapter 01 – Image Representation and Formation Slide: 38
 Focus and depth-of-field
• Depth-of-field is the distance from the nearest to the farthest object
in front of the camera that can be captured sharply in the result image,
i.e., the blur does not exceed a certain value

1 1 1
+ =
g b f
Advanced Image/Video Processing: Chapter 01 – Image Representation and Formation Slide: 39
 Aperture

Depth-of-field

• The smaller aperture, the longer the depth of field.

• Downside: smaller aperture Þ less light enters the camera
Þ increase exposure time
Þ risk of blur due to motion
Advanced Image/Video Processing: Chapter 01 – Image Representation and Formation Slide: 40
 Field-of-view

• Field-of-view, v, depends on size of the sensor and focal length

• ”Fisheye” lens Þ small focal length and large field-of-view
• Horizontal FOV and Vertical FOV
Advanced Image/Video Processing: Chapter 01 – Image Representation and Formation Slide: 41
Computer Graphics vs Image
Processing
 Creating Realistic Imagery

Computer Graphics Computational Photography

Photography

Realism
Manipulation
Ease of capture

+ great creative possibilities + instantly realistic

+ easy to manipulate + easy to aquire
objects/viewpoint - very hard to manipulate
-Tremendous expertise and objects/viewpoint
effort to obtain realism
Relationship to Vision and Graphics

3D Model 2D image

Computer Graphics: Models to Images

Computer Vision: Images to Models
Comp. Photography: Images to Images
Google Photosphere

https://www.youtube.com/watch?v=ZIsRPqcv0Cw
Course objectives
1. You will have new abilities for visual
creation.

Graphic by James Hays

Course objectives
2. You will get a foundation in computer vision.

Safety Health Security

Comfort Fun Access

 Course objectives
3. You’ll better appreciate your own visual ability.

Is that a
queen or a
bishop?
Seeing less than you think…
Seeing less than you think…
Video by Antonio Torralba (starring Rob Fergus)
 But actually…

Video by Antonio Torralba (starring Rob Fergus)

Course objectives
4. You will get a more intuitive understanding
of important mathematical and
computational concepts
• Convolutions, filtering
• Gradients
• Change of basis, interpolation, extrapolation, PCA
• FFT
• Dynamic programming, recursion
• …
Course objectives
5. You’ll have fun doing cool stuff!
Programming Project #1
Project 2: Fun with frequencies
Project 2: Fun with frequencies

Prof. Christos Papadimalik

© Riyaz Faizullabhoy
Project 3: Gradient Domain Editing
Project 3: Gradient Domain Editing

© Weilun Sun
Project 4: Face morphing and caricatures
Project 4: Face morphing and caricatures

The Beatles

Polyjuice Potion

© Rachel Albert
Project 5: Automatic Mosaic Stitching
Photo Mosaics
Project 5(g)
Tour Into the Picture
 Paper Pop-up

Step 1: define planes

Step 2: rectify each plane

Step 3: compute 3D box coords

For each project:

Derive the math, implement stuff from scratch,

and apply it to your own photos

Every person does their own project

Programming Language:
– Python or Matlab
– you can use other languages, but you are on your
own
Textbook

http://szeliski.org/Book/
General Comments
Prerequisites
• Linear algebra!!!
• Good programming skills
• Some computer graphics, computer vision, or image
processing is useful, but not required.

Emphasis on programming projects!

• Building something from scratch
Getting help outside of class
Course Web Page
• BKEL website

Question:
• nddung@hcmut.edu.vn
Administrative Stuff
Grading
• Programming Project (40%)
• Final Exam (60%)
Academic Integrity

• Can discuss projects, but don’t share code

• Don’t look up code or copy from a friend

• If you’re not sure if it’s allowed, ask

• Acknowledge any inspirations

• If you get stuck, come talk to us