DATA REPRESENTATION (PART 3)

MULTIMEDIA
MULTIMEDIA DATA

• Multimedia means different types of data

• Text
• Images
• Audio
• Video
• Graphics

• Already seen how text is represented (ASCII)

• Image vs. vector graphics

CONTINUOUS VS. DISCRETE

• Most ‘real life’ data is continuous

• Takes on any value in a range of values (i.e. infinite possibilities)
• How loud is a piece of music?
• How bright is a light?

• Computers represent data as discrete values

• Values are separate and distinct with gaps between them
• 1, 2, 3, 4, 5, 6, …
DIGITALIZATION, QUANTIZATION, AND SAMPLING

• Digitization – converting from continuous data to discrete data

• Quantization – Digitization of amplitude – the size (loudness, brightness, etc.)

• Sampling – How often measurements are made (sounds) or how close the
measurements are (images)
AUDIO DATA

• Sounds are produced by vibrations

• Frequency – how often vibration
• Amplitude – Size of vibration

• Sounds are sampled at discrete times and amplitude is measured

• Amplitude is quantized, usually an integer

• CD quality
• 44100 samples/second (44Hz)
• Sample represented as 16 bits (65536 values)
• 1 second audio = 44100 16-bit numbers (88 kB)
IMAGE DATA

• A digital camera samples an image with light sensors.

• Each sample is a pixel.
• Pixels are arranged in a rectangular array, M rows and N columns.
• The measurement at each pixel is the brightness for black and white images.
• Measurements are usually an integer in the range 0–255 (8 bit). 0 means
black and 255 means white.
IMAGE DATA

• For colour images, three measurements are made:

• How much red, green, and blue light is in that position (RGB).
• An image is represented in a computer as an array of pixels (usually one row
at a time). Each pixel is represented by either one number (black and white)
or three numbers (colour).
• Colours are commonly defined by the RGB values in hexadecimal (e.g. HTML).
e.g. Yellow is #FFFF00
• Equal amounts of R, G, B gives black (#000000), white (#FFFFFF), and various
shades of gray.
VIDEO DATA

• Videos are essentially a sequence of images sampled close in time.

• Each image is called a frame.
• Traditional film movies have 24 frames each second.
• HDTV at 1080p may have 24, 50, 60 frames each second.
• A video is represented in a computer as a sequence of images.
SAMPLING RATE

• If we sample too often, there is too much processing and more data is
produced.

• If we don’t sample enough, the audio/video may be “choppy.”

• We may even be tricked into seeing something else (aliasing).

ALIASING
Original Scaled down by a factor of 2

Notice the direction of the stripes when the image is scaled down.
UNDERSAMPLED: ALIASING

• Top-secret helicopter?

• Car chases in movies

• Explanation
SAMPLING

• Nyquist theorem: we must sample at least twice as fast as the maximum

frequency to obtain an accurate representation.
• Intuitively, how often you sample depends on how quickly the data change.
• Sampling at least twice as fast as the maximum frequency ensures that a
change is not missed.
• Conversely a signal that does not change (constant) only has to be sampled
once.
• Human can hear sounds up to about 20000Hz, hence the sampling rate of CD
audio is about twice as much.
HOW MUCH DATA?

• 2 minutes of CD quality audio: 10.6 MB

• iPhone 6 picture (8 megapixels): 24 MB
• 1 hour of HDTV 1080p video (24fps): 501 GB

• That’s a lot of data!

HOW MUCH DATA?

• Capacity of CD: about 700MB—a little over 2 hours

• Capacity of DVD: 4.7GB—33 seconds of 1080p video, with no audio!

• DVD movies are generally lower quality than 1080p. Luckily the data can be
compressed.
DATA VS. INFORMATION

• Although there is a lot of data, there may not be much information.

• Intuitively, a piece of data has a lot of information if we learn something
unexpected from the data.
• “Everyone who writes the final exam will get at least 0%.” is not very informative—you
already know this. The data is redundant.
• “It snows in August.” is very informative because it is not expected.
• Compression works by removing redundancy, in order to obtain a more
compact representation.
COMPRESSION

• Compression works by removing certain redundancies. A certain context is needed

(e.g. what you already know).
• Images: the colour of a pixel is likely similar to those close by. If you know one pixel it is likely
you know what is close by.
• Videos: the colour of a pixel is likely similar to those in a previous frame.

• Many compression algorithms work by “prediction”: only record prediction errors.

• Run length encoding is a method to compress data with many consecutive repeated
values.
LOSSY COMPRESSION

• Most practical compression algorithms are lossy - the data stored is a close
approximation of the original.
• By accepting possible loss, we may be able to compress more.
• Instead of using 8 bits to represent 256 possible levels, use 7 bits to represent 128 levels.
• We save 12.5% space but the data is not as accurate.
• Why does this work? We may not be able to tell the difference!

• There is a trade-off between quality and space usage. That is often controlled by a
“quality” setting in your digital camera, for example.
JPEG COMPRESSION

• Many images are compressed using the JPEG standard (.jpg).

• Developed by the Joint Photographic Experts Group in 1992.
• Designed for photographs, lossy.
• A quality setting is used to control space and quality.
• It uses a variety of techniques, including run-length encoding.
• There are many newer algorithms but JPEG is still one of the most widely
used.
DIFFERENT JPG QUALITIES

8 KB 14 KB 21 KB 144 KB
BASIC IMAGE PROCESSING

• Colour balance: the three colour components (R,G,B) can be manipulated

independently to change the colour balance.
• If there is too much yellow, we can decrease the R and G values, or we can also increase the B
values.
• What is the difference between the two approaches above?

• Contrast enhancement
• Blur out certain parts—tend to remove details.
• Sharpening certain parts—tend to highlight details
CONTRAST ENHANCEMENT

• Contrast: relative difference between two tones

BLUR / SHARPEN

• Blur out certain parts—tend to remove details.

• Sharpening certain parts—tend to highlight details
MPG3

• Lossy compression of audio files

• Standard for portable digital players

• An MP3 file created using 128 kbits/s results in a file 1/11 the size of the CD source

• Uses perceptual coding. Removes:

• Sounds (most) humans can’t hear
• Quieter of two sounds that happen at the same time
VIDEO CODECS AND CONTAINERS

• Video compression predicts differences between frames

• Many different video codecs (encoder/decoder)
• MPEG-4 and H.264 most popular video codecs

• Containers
• Combination of video codec and audio codec in one package
• Also contains other information (e.g. titles/chapters, subtitles, other languages for
DVD/Blu-ray)
• For many applications (e.g. Facebook photos), lossy compression is
acceptable.

• Are there applications in which lossy compression is not acceptable?

• What are the implications of using lossy compression?