Multimedia Communication

Technology
(EC314), 3-0-0, credits - 3
Prepared by
Dr Shweta Shah
Syllabus

 MULTIMEDIA COMMUNICATIONS
 Introduction, Revision Of Fourier Transform, OFT and DCT, Multimedia
Communication Model, Elements Of Multimedia Systems, User
Requirements, Multimedia Terminals, Input And Output Devices.
 AUDIO-VISUAL INTEGRATION
 Human Speech Generation Model, Synthetic Speech Generation,
Media Interaction, Bimodality Of Human Speech, Speech Signal
Properties, Visual Properties, Lip Reading, Speech-Driven Talking
Heads, Lip Synchronization, Lip Tracking, Audio-To-Visual Mapping,
Bimodal Person Verification, Joint Audio-Video Coding.
 MULTIMEDIA PROCESSING IN COMMUNICATIONS
 Digital Media, Digital Audio, Digital Image And Video, DPI And PPI, Signal
Processing Elements, Challenges Of Multimedia Information Processing,
Lossless and Lossy Coding Techniques, Perceptual Coding, Perceptual
Coding of Digital Audio Signals, Transform Audio Coders, Audio Sub-Band
Coders, Speech Coder Attributes, CD Audio Coding For Multimedia
Applications, Image Coding, Video Coding, Water Marking, Organization,
Storage And Retrieval Issues, Signal Processing For Network Multimedia,
Multimedia Processors.
 MULTIMEDIA COMMUNICATION STANDARDS
 JPEG, MPEG And ITU standards.
 MULTIMEDIA COMMUNICATIONS ACROSS NETWORKS
 Network Requirements, Real Time Packet Transfer Concept, Multimedia
Requirements And ATM Networks, Packet Audio/video in The Network
Environment, Video Transport Across Generic Networks, VOIP Application.
  Total: 42 hours

BOOKS RECOMMENDED

1. Rao K. R., Bojkovic Zoran S. and Milovanovic Dragorad A. "Multimedia

Communication Systems: Techniques, Standard And Networks", PHI, 1st Ed., 2002.

2. Vaseghi Saeed V., "Multimedia Signal Processing Theory And Application In Speech,
Music And Communications", Wiley, 1st Ed., 2007.

3. Rao Kamisetty, Bojkovic Zoras and Dragorad, "Introduction To Multimedia

Communications", Wiley, 1st Ed., 2006.

4. Ohm and Jens R., "Multimedia Communication Technology", Springer, 1st Ed., 2004.

5. Mihaela Vander Scharr and Chow Philip A., "Multimedia Over 1P And Wireless
Networks— Compression, Networking And Systems", Academic Press, 1st Ed., 2007.
 Course outcome
 After completion of the course student will be able to:
 CO-1: To acquire the basic knowledge of multimedia communication technologies
including audio, image, video, text compression techniques and distributed multimedia
system
 CO-2: Explanation about signal processing aspects involved in multimedia including signal
properties
 CO-3: Application of coding techniques in recent applications for data storage and
communication of multimedia
 CO-4: Analysis/comparison of various coding techniques, case study and problem solving
as per given data
 CO-5: Evaluation of standard source coding protocols for multimedia and protocols for
distributed networks
 CO-6: Design and optimum parameter selection criteria for multimedia communication
requirements
Course Evolution

 Class Tests: 20 Marks (Mostly surprise tests, be prepared!!!)

 Mid Sem: 30 Marks
 End Sem: 50 Marks
The Term “Multimedia”

Composed of 2 parts:
Multi (multus) : “numerous, multiple”
Media (medium): “middle, center” – agent for
something. Used for dissemination (distribute) and
representation of information.
In general, multimedia could be defined as the usage
of multiple agent (text, audio, video, images) for
disseminating and presenting information to audience
(target user)
The Term “Media”

Can be categorized based on a

few criteria:
Perception media
Representation media
Presentation media
Storage media
Transmission media
Perception Media

“How do humans perceive information”

We perceive information from what we see
and what we hear
Visual media:
Text, graphics, images, video
Auditory media:
Music, sound and voice
Representation Media

“How in information encoded in the computer”

 Referring to how the information is represented internally
to the computer.
 The encoding used is of essential importance.
 Several options:
 Text is encoded in ASCII
 An audio data stream in PCM (Pulse Coded Modulation)
 Image in JPEG format (JPEG: Joint Photographic Experts Group)
 Video in MPEG format (MPEG: Moving Picture Experts Group)
Presentation Media

“Which medium is used to output information from

the computer or input in the computer”
 Refers to physical means used by systems to reproduce
information for humans, e.g: audio and visual devices
 Input:
 Keyboards, cameras, microphone, Head Mounted Device
(for VR input)
 Output:
 Paper, monitors, loudspeakers
Storage Media

“Where is information stored”

Refer to various physical means for storing
computer data, such as magnetic tapes,
magnetic disks, or digital optical disks (CD-
ROM, CD, DVD), drives, etc..
Transmission Media

“Which medium is used to transmit data”

 Refers to the physical means – cable of various type
(coaxial cable, twisted pair, fiber optics), radio
tower, satellite – that allow the transmission of
telecommunication signals.
 The difference between transmission media and
storage media is the capability of transferring data
continuously over networked computers.
 Definition
 Multimedia is the presentation of a (usually interactive)
computer application, incorporating media elements
such as text graphics, video, animation, and sound, on a
computer.
• Multimedia melds the sensory power of television with the data
manipulation and interactive powers of computer.
 What is Multimedia Communication?
 Multimedia communication is the field referring to the representation,
storage, retrieval and dissemination of machine-processable information
expressed in multiple media, such as text, image, graphics, speech, audio,
video, animation, handwriting and data files.
 Definition: Multimedia Systems

• A multimedia system is characterized by computer-controlled,

integrated production, manipulation, presentation, storage
and communication of independent information, which is
encoded at least through continuous (time-dependent) and a
discrete (time-independent) medium.
 History of Multimedia
 1. Newspaper: perhaps the first mass communication medium
using text, graphics, and images
 2. Motion pictures: conceived of in 1830's in order to observe
motion too rapid for perception by the human eye
 3. Wireless radio transmission: Guglielmo Marconi, at
Pontecchio, Italy, in 1895
 4. Television: the new medium for the 20th century, established
video as a commonly available medium
 History of Multimedia
 The connection between computers and ideas about multimedia covers what is actually only a
short period:
 1945 - Vannevar Bush wrote a landmark article describing what amounts to a hypermedia system
called Memex
 1960 - Ted Nelson coined the term hypertext
 1967 - Nicholas Negroponte formed the Architecture Machine Group
 1968 - Douglas Engelbart demonstrated the On-Line System (NLS) another very early hypertext
program
 1969- Nelson and van Dam at Brown University created an early hypertext editor called FRESS
 1972 – A Game of Pong (first commercial multimedia product)
 1973 – ATARI (laser disc, used in game cartridges)
 1973 – IBM Discovision (first multimedia interactive kiosk products)
 1978 – Apple II (with floppy drive)
 1980s – mouse was invented by Xerox Corp.
 1981 – Microsoft and IBM (IBM PC) Christmas 1981 – Nintendo hit the market (30 million machines)
 1982 – Apple II had voice synthesis capability
 The History of Multimedia
 1984 – Macintosh using mouse
 1984 – Virtual Reality was invented by NASA, input devices using HMD (Head-Mounted Display)
and Dataglove
 1985 – Macromind (Macromedia) produced VideoWorks, later changed to Director (the most
widely used cross-platform multimedia authoring tool)
 1986 – first electronic encyclopedia, first international conference on multimedia, first CD-ROM
 1987 – Mac II (first color GUI)
 1988 – CD-R (CD-Record)
 1989 – Creative Labs (Sound Blaster sound card)
 1990s – Adobe released Photoshop.
 1990s – Windows 3.0 multimedia enabled by Microsoft
 1992 – first children ‘s interactive book title “Just Grandma and Me”
 1993 – double speed CD-ROM drives as a multimedia standard
 1993 – Web Browser Mosaic
 1994 – Web Browser Netscape
 1995 – Multimedia PC, 32 bit, Windows 95.
 Later followed by Windows 98, Windows 2000
 …….. And multimedia continues to evolve
 Hypermedia

 Hypermedia is a way of organizing multimedia information by

linking media elements.
 Hypermedia has grown out of a fusion between hypertext and
multimedia.
 Hypertext was developed to provide a different structure for basic
text in computer systems :
text is essentially sequential in nature, even though its structure is
hierarchical (chapters, sections, subsections, paragraphs)
 hypertext was developed to permit more random access between
components of text documents, or between documents, to allow a
greater degree of flexibility and cross-referencing than a purely linear or
sequential model would allow.
 Hypermedia
– The structure of a hypermedia organizations is called a
hypermedia web, which consists of a number of multimedia
elements or nodes with links between them.

– Links represent semantic relationships, thus when a link exists

between two nodes they must be related in some fashion :
a digital image linked to a textual description of it
a slide-show linked to an audio commentary

– Most widely used hypermedia tools are hypermedia browsers,

which let users view nodes and traverse links between them,
and mark-up languages, such as HTML, which allow users to
create hypermedia webs as structured documents.
Hypermedia

Image Node

Text Node

Audio Node

A Simple Hypermedia Web

 Hypermedia and Multimedia
 A hypertext system: meant to be read nonlinearly, by
following links that point to other parts of the document,
or to other documents
 Hyper Media: not constrained to be text-based, can
include other media, e.g., graphics, images, and
especially the continuous media , sound and video(e.g.
the WWW)
 Multimedia means that computer information can be
represented through audio, graphics, images, video,
and animation in addition to traditional media.
Multimedia Representation

 Text -
 Each character (letters and symbols) is represented by a fixed
 number of bits. In ASCII code ‘A’ is 01000001
Multimedia Representation

 Image –
 Represented as 2-dimensional signal
 Arranged in units called pixels
 The value of each pixel represents the radiance intensity of the
scene. For color image, there are usually three components, e.g.
red, green, and blue. Each range from 0 to 255
 Image compression method can dramatically reduce the size of
the image data. Some standards include JPEG, JPEG2000
 Multimedia Representation
 Audio –
 Represented as 1 dimensional
signal
 Continuous acoustic wave is
digitized into a sequence of bytes
 Audio compression method is
used to reduce the amount of
data (MP3 e.g. 10:1 ratio)
Multimedia Representation

 Video –
 Video is represented as
sequence of images called
frames
 NTSC video has roughly 30
frames per second
 Video compression standard
such as MPEG2(DVD) and
MPEG4 can significantly r
reduce the size of video files
Key Properties of a Multimedia
Systems
Discrete and Continuous Media
At least one discrete and continuous media
Independent Media
Separate each media independently
Computer-Controlled Systems
Integration
 Interactive Multimedia Systems
 Interaction
 Thought - Something that you do to yourself ~ internal process (mental state)
 Action - Something you do to an object in the world. The effect is that something in the
world has changed. E.g.: pressing key, clicking mouse button
 Interaction - Involves the participants action and reaction. Two way process. E.g.: give a
query to search engine which can return an outcome of my search.

 Properties
 Various media integration
 High level degree of interactivity between
user and computer
 Digital environment
Application Areas of IMS

 Education
 Computer Aided Learning (CAL)
 E-Learning (World Wide Web)
 Training
 Point of Sales Information (Kiosk)
 Direct visitors around the large complexes
 News Delivery, Broadcasting and Advertising
 Commerce and Business Applications
 Virtual Reality (Games)
Multimedia- Applications

In Medicine

Source:
Cardiac Imaging,
YALE centre for
advanced cardiac
imaging

Multimedia I -
Mohammed Dwikat
2009
Multimedia- Applications

In training

Multimedia I -
Mohammed Dwikat
2009
Multimedia- Applications

Public awareness
campaign

Source
Interactive Multimedia Project
Department of food science&
nutrition, Colorado State Univ

Multimedia I -
Mohammed Dwikat
2009
WE SHALL DEAL WITH DIGITAL MEDIA BECAUSE OF
Robustness
Seamless Integration
Reusability and interchangeability
Ease of distributed potential
What are the media signals?

 MEDIA SIGNALS ARE THOSE SIGNALS WHICH ARE

ACCESSIBLE TO THE HUMAN INFORMATION PROCESSING
SYSTEM (Brain)..
 They are normally analog (continuous) in nature
 We convert those signals to digital format and use
 E.g., audio, speech, image, video signals, speech with
video, music with images etc.
 WHY IS DIGITAL MULTIMEDIA SIGNAL PROCESSING POSSIBLE?

 This is because we have means DIGITAL REPRESENTATION AND

PROCESSING OF ANY TYPE OF information.
 We have studied PCM, DPCM, ADPCM, DM schemes which
ultimately represents digital form of analog signals without removal
of redundancy.
 INFORMATION IS REPRESENTED BY BITS AND PROCESSED DIGITALLY
Multimedia Processing
 Audio -
 Equalization
 Denoise
 Voice removal (for karaoke)
 Image -
 Histogram equalization
 Denoise
 Deblur/Sharpen/Edge enhancement … and many more
 Video
 Change frame rate
 Blending
 stabilization
MEDIA SIGNAL PROCESSING

 BASIC CONCEPTS REQUIRED

 Fourier spectra and Spectral analysis
 DCT, DFT
 Sampling theorem
 Partial differential equations and difference equations
 Source/entropy coding –waveform based, transform
based etc.
 Spectral analysis
 Spectral analysis is one of several statistical techniques necessary for characterizing and
analyzing sequenced data.
 There are two reasons for calculating the spectrum.
 First, the periodic spectrum, regardless of whether the data are periodic or not, may be used to
reduce empirical data to a simple mathematical expression, thus saving storage space and
facilitating analytical manipulation. This kind of process is used for reducing the size of
computer image files.
 Secondly, the spectrum is a fundamental component for describing statistically a single set of
sequenced nonperiodic data: it is a scale expansion of the variance. Similarly it is necessary for
investigating possible connections between several sets of similarly sequenced data: it
provides scale expansions of regression parameters.
 Spectral analysis is based on the Fourier theorem, which states that any waveform can
be decomposed into a sum of sine waves at different frequencies with different
amplitudes and different phase relationships.
Fourier Analysis

 The spectral analysis is based on Fourier theorem

 The Fourier transform (FT) decomposes a function of time
(a signal) into its constituent frequencies.
 The term Fourier transform refers to both the frequency
domain representation and the mathematical operation
that associates the frequency domain representation to
a function of time.
 DCT: Discrete Cosine Transform
 A discrete cosine transform (DCT) expresses a finite sequence of data points in terms of a
sum of cosine functions oscillating at different frequencies.
 The DCT transforms a signal or image from the spatial domain to the frequency domain.
 The DCT helps separate the image into parts (or spectral sub-bands) of differing
importance (with respect to the image's visual quality).
 DFT: Discrete Fourier Transform
 the discrete Fourier transform (DFT) converts
a finite sequence of equally-
spaced samples of a function into a same-
length sequence of equally-spaced samples
of the discrete-time Fourier transform (DTFT),
which is a complex-valued function of
frequency.
 The DFT is the most important discrete
transform, used to perform Fourier analysis in
many practical applications.
 In digital signa processing, the function is any
quantity or signal that varies over time, such
as the pressure of a sound wave,
a radio signal, or daily temperature readings,
sampled over a finite time interval
Sampling theorem

 the sampling theorem is a fundamental bridge

between continuous-time signals and discrete-time
signals.
 It establishes a sufficient condition for a sample rate that
permits a discrete sequence of samples to capture all
the information from a continuous-time signal of
finite bandwidth.
 Sampling is a process of converting a signal (for
example, a function of continuous time and/or space)
into a sequence of values (a function of discrete time
and/or space).
Partial differential equation

a partial differential equation (PDE) is

a differential equation that contains
unknown multivariable functions and
their partial derivatives.
PDEs are used to formulate problems
involving functions of several variables,
and are either solved by hand, or used to
create a computer model.
Difference equations

An equation that shows the

relationship between consecutive
values of a sequence and
the differences among them.
They are often rearranged as a
recursive formula so that a systems
output can be computed from the
input signal and past outputs.
Source/Entropy Coding

In information theory, Shannon's source

coding theorem (or noiseless coding theorem)
establishes the limits to possible data
compression, and the operational meaning of
the Shannon entropy.
an entropy encoding is a lossless data
compression scheme that is independent of the
specific characteristics of the medium
 Data Compression
How does compression work ?
Assume that we have 4 letters in a language viz. “A”
“B” “S” “Z”
To uniquely encode each letter, let us assign two bits
each :
A-00, B-01, S-10, Z – 11. Message “AAABSAAAAZ” is
encoded with 20 bits
Now let us assign: A- 0, B-100, S-101, Z – 111. The same
message can be encoded using 16 bits
The basic idea behind data compression – assign
shorter codeword to more frequently used symbols
Multimedia Compression Standards
 In order to enable communication between different
media source compression standards are established
 Audio –
 Differential pulse code modulation (DPCM), adaptive predictive
coding (APC), linear predictive coding (LPC), MPEG audio
coders (e.g MP3)
 Image –
 Graphics interchange format (GIF), Joint Photographic Experts
Group (JPEG 10:1 to 20:1), JPEG2000
 Video –
 Moving JPEG (MJPEG), ITU-T H.261, H.263, Motion Picture Expert
Group (MPEG-1, MPEG-2, MPEG- 4,…)
 Multimedia Communication Model
 A layered model of multimedia comprises of five components:
 Partitioning of complex information objects into distinct information types for
the purpose of easier communicating, storing and processing.
 Standardization of service components per information type,
 For level of quality per information
 Creation of two levels
 A network service platform
 It hides the transport network and its building blocks from applicant’s as well as user’s point
of view
 A multimedia communication platform
 Provides communication support on the basis of information structure and information
exchange among the large applications

 Definition of generic application for multiple use in various environments

 Specific application support…
 Elements of multimedia systems
Two key communication models:
Person to person communication
A user interface that provides mechanisms for all users to interact with
each other
Transport layer allows signal to move from one user location to some or
other user locations by providing interface and maintaining the quality
of multimedia signal
E.g. teleconferencing, video phone, distance learning, etc…
Person to machine communication
A user interface for interacting with machine, along with transport
layer for moving multimedia signal from the storage location to user
Mechanism of storage and retrieval
E.g. business meeting notes, access of broadcast video, digital library
 Communication modes
 Simplex
 Only one direction
 Half duplex
 Two way alternate
 Duplex
 Two way simultaneous
 Broadcast
 One to many
 Multicast
 Many to many
Network Requirements

High speed and changing bit rates

Several virtual connections using the same
access
Synchronization of different information types
Suitable standardized service and
supplementary services supporting multimedia
applications
Network types

Circuit mode
Also known as synchronous communication channel

Packet mode
Also known as asynchronous communication channel
Connection oriented
Connection less
Circuit switched network
Packet mode: connection oriented
Packet mode: connectionless
 Network QoS
 The operational parameters associated with a communication
channel through a network are known as the network Quality of
Service (QoS) parameters.
 QoS for a constant bit rate channel that is setup through a circuits
switched network include:
 The bit rate
 The mean bit error rate
 The mean BER of a channel is the probability of a bit being corrupted during its
transmission across the channel in a defined time interval.
 If the BER prob is P and the no of bits in a block is N, then assuming random errors,
the prob of a block containing a bit error, PB , is given by : PB = 1-(1-P)N
 The transmission delay
 The QoS parameters associated with a packet switched
network include:
 The maximum packet size
 The mean packet transfer rate
 The mean packet error rate
 The mean packet transfer delay
 The worst case jitter
 The transmission delay
 Packet transfer concept
 Offers a transmission capability that can be handle high bit rates
to support video transmission.
 packet switching is flexible than circuit switching which can
emulate the different data rates multiplex together by creating a
network.
 A network on packet switching services, can be dealt with in a
common format.
 It offers dynamic allocation of bandwidth and switching
resources as well as the elimination of channel structure
 can carry variable bit rate signals directly
 no buffering or rate control feedback is necessary at the
transmitter
 There are different protocols used like ATM (Asynchronous
Transfer Mode), IP (Internet Protocol), etc…
 Multimedia Requirements and ATM
networks
Constant variable or burst oriented bitstreams
virtual connections or virtual parts through the
subscriber access depending on instantaneous
needs with the total capacity of about 150 or 600
MB/s
uniform bitrate
independent transmission and Switching Systems
 Multimedia Terminal
 To achieve the vision of the current communication revolution
and to become available to everyone, a number of
technological issues must be addressed and put into a framework
that leads to integration, ease of use, and high quality outputs.
 Issues to address in common are:
 Technique for compression and coding
 Technique for organizing, storing and retrieving multimedia content,
including both downloading and streaming techniques,..
 Techniques for accessing the multimedia signals
 Techniques for searching in order to find multimedia sources
 Techniques for browsing individual multimedia documents and
libraries
 User requirements
- from multimedia communication system
 User needs
 Fast preparation and presentation of the different information of
interest
 Dynamic control of multimedia applications w.r.t. connection and
quality on demand
 Intelligent support
 Standardization
 For modern Multimedia services:
 Instant availability
 Real time information transfer
 Service always online
 Access their service from any terminal(e.g. mobile)
 Input Devices

 Keyboard, mouse (track balls, joysticks, etc)

 graphics tablets - drawing
 Scanner – capture image from printed material
 Digital camera - capture and transform image into digital
form
 Touch screen for kiosk application
 Analogue audio input from microphone and audio player
 Networking support for fail distribution
 Modem
 Etc…
 Output Devices

High resolution screen, 256 colors (at least) –

output video
Speakers, amplifier or tape devices - Output audio
Network with capacity at 10 millions bit/second
Analog modem at 28800 bit/second speed. ISDN
digital modem at speed 128000 bit/second.
Printer, etc…
Storage Requirements

At least 32MB RAM to produce good quality of

graphics, audio, video, etc
VRAM (Video Random Access Memory) to
support high color definition
Hard disk at high volume capacity with good
drive system speed to support graphics, video,
audio, and animation processing.
Secondary storage CD-ROM, Magnetic Tape, etc
Types of media used in a multimedia
applications
Multimedia Communication Networks
Multimedia communication networks
and their services
Why is multimedia so hot?
Technology Thrust
More processing power per chip
Progress in storage capacity
Personal computer revolution
Progress in networking
Progress in user interfaces, and software
Progress in compression techniques
Market Thrust
Large market - Revolutionizing film/video industry
Application challenges and competition
Market opportunities
Customer demand and comfort
END OF INTRODUCTION CHAPTER