<Properties of Emerging Computing and Communication Applications>
Spatiotemporal and live data streams are norm rather than the exception;
A holistic picture of an event or situation is more important than silos of
isolated data; Users want insights and information that are independent of
the medium and data source; Only relevant information counts;
Exploration (browsing) is the predominant mode of interaction (Instead of
relational database querying is the primary interaction in previous
generation application).
<Difference of human and computer in percept> Current information envir
onments often work against the human-machine synergy: Humans are goo
d at conceptual and perceptual analysis; Computers are exactly the opposit
e, they are efficient at mathematical and logical analysis.
<Experiential environment> is an environment where users directly use th
eir senses to observe data and information of interest related to an event. N
atural interaction with the data based on the users’ particular set of interes
ts in the context of that event.
<Properties of Experiential Environments> Direct and Intuitive, Provides
a holistic picture of an event without using unfamiliar metaphors and com
mands; Provide the same query and presentation spaces, Query and pre
sentation spaces are the same; Consider both the User State and Context,
People operate best when they are in known contexts. Good experiential e
nvironment should promote perceptual analysis and exploration.
<Why Multimedia Computing> We need a good experiential system to impr
ove human-machine synergy for current information environments; Experi
ential Systems are technically implemented as Multimedia Systems (Integra
te, Process and Output data from different sensory modalities).
<Formal Definition of Multimedia Computing> Consider a system equipped
with multiple sensors working in a physical environment; Multimedia diffe
rs from monomedia fields like computer vision or audio processing in the w
ay that Multimedia is closely related to how humans experience the world;
Partial information from multiple media sources is correlated and combine
d to get complete information about the environment; Multimedia computi
ng and communication is fundamentally about combining information from
multiple sources in the context of the problem being solved.
<Objective of Multimedia System> Process different types of data streams s
imultaneously as one correlated set of streams that represent information a
nd knowledge of interest for solving a problem.
<Challenge of Multimedia System> Discover correlations that exist in the se
t of multimedia data; Combine partial information from disparate sources t
o build holistic information in a given context.
<Experience and Information> Experience is the direct observation of or p
articipation in events as a basis of knowledge (Webster’s dictionary). Infor
mation is an efficient but abstract communication of experience. Communic
ation is the process of sharing experiences and information with others.
<Perception> is the process of understanding sensory signals to recover us
eful information; Understanding of sensory information is an important ste
p in many multimedia systems; Without any knowledge, the system cannot
produce any information; Perception is usually considered as controlled ha
llucination process.
a finger touches the screen, it disturbs the screen's electrostatic field,
measurable as a change in capacitance.
<Types of microphones> Dynamic microphones operate on the principle
of electromagnetic induction. They consist of a diaphragm attached to a coil
of wire, placed within the magnetic field of a magnet. When sound waves
<Neisser’s Perceptual Cycle> An agent is continuously interacting with the hit the diaphragm, it moves, causing the coil to move within the magnetic
environment using its sensory mechanisms; Build the model of the environ field. This movement generates an electrical current that is analogous to
ment; The system decides what is further required to complete the task an the sound waves; Condenser microphones, also known as capacitor
d how that information could be acquired.Agent:{Environment(available in microphones, use a capacitor to convert acoustic energy into electrical
formation)-modifies->Schema-Directs->Exploration-samples->Env.} energy. The main components are a diaphragm and a backplate, which
<Semantic gap> The semantic gap is the difference between the together create a capacitor. When sound waves hit the diaphragm, the
information that a machine can extract from (perceptual) data and the distance between the diaphragm and the backplate changes, causing
interpretation that a user in a given situation has for the same data. variations in capacitance. These changes are translated into an electrical
signal. To function, condenser microphones require power, either from a
battery or phantom power provided by a mixer or audio interface.
<Digitization> is a process for converting continuous signal into a discrete
set of its samples. Digitized sample values are much less susceptible to
distortions than analog signals. The error the quantization introduces is the
quantization noise. The error the sampling rate introduces is the
discretization error.
<Nyquist frequency> If a function x(t) contains no frequencies higher than
B hertz, x(t) is completely determined by its ordinates at a series of points
spaced 1/2B seconds apart.
<Sound Pressure Level> Sound Pressure Level (SPL), or L p , is defined as:
Lp=20lg(p/p_ref) dB. Measurement on logarithmic scale using decibels,
<Masking in hearing> Frequency Masking: The phenomenon where a loud where p is the RMS sound pressure and p ref is a reference sound pressure.
sound in one frequency band masks a softer sound in another frequency The sound pressure perceived by human ear is non-linear.
band, which would have been audible without the loud sound, is known as <A-weighting scheme> Sound-pressure measurements are often frequency
auditory masking. weighted to match perception. A-weighting is applied to instrument-
Temporal Masking: After a loud sound ceases, a quiet sound may remain measured sound levels in an effort to account for the relative loudness
inaudible for a brief duration as the ear temporarily reduces its sensitivity, perceived by the human ear, as the ear is less sensitive to low audio
requiring time to readjust its gain to normal levels. frequencies. It is employed by arithmetically adding a table of values, listed
<Types of Sensors> Sensors that imitate human sensors; Sensors that do by octave or third-octave bands, to the measured sound pressure levels in
not imitate human sensor; Sensors that measure physical facts about the dB. Sound-pressure levels weighted by A-weighting scheme are usually
environment. <Sensors Limitations> Limited dynamic range; Offset, bias; labeled as dBA or dB(A).
Nonlinearity; Hysteresis(deviation over time).
<Sinusoid Signal> x(t)=Acos(2πBt+θ), B=1/T.
<Human Audible Frequency Range> Hearing range is from 20 Hz to 22 kHz;
Professional audio-recording equipment usually employs sampling
frequencies of at least 44 kHz; Digital filtering and machine learning, might
use these higher frequencies. <Speaker Characteristics> Rated power;
Impedance; Crossover frequencies; Frequency range.
<Types of Touch Sensors> Resistive touch screens work on the principle of <Produce Virtual Surround Sound> Interaural Level Difference (volume,
pressure. They consist of two conductive layers separated by a small gap: a ILD), Interaural Time Difference(ITD, delay); ILD and ITD can be analysed
flexible top layer (typically made of polyester coated with a conductive by our brain to determine whether the sound came from the left or right; In
material like indium tin oxide) and a rigid bottom layer. When a user addition, human brain can use a sound wave’s reflection off the pinna, or
presses on the screen, the two layers make contact at the point of touch, auricle, or the ear to determine the sound’s location. The volume and sound
creating an electrical circuit; Capacitive touch screens work by sensing the quality change due to these reflections are known as Head-Related
electrical properties of the human body. They are constructed with a layer Transfer Functions (HRTFs); To produce virtual surround sound, Recreate
of glass coated with a transparent conductor (like indium tin oxide). When the ILD, ITD, HRTFs based on the speakers available.
<Music> Overtone; Timbre; Spectrum; ADSR envelope.
<Units of Light> Candela: Measures luminous intensity, which is defined as
power emitted by a light source in a particular direction; Lumen: measures
luminous flux, i.e. the total light an object emits; Lux: luminous power per The compression obtained based on these entropy compression routines is Chrominance (1 Cr & 1 Cb) 8x8 block. Explore only spatial
area. Steradian. Lumen, lm, cd*sr; Lux, lx, lm/m2. Nit, Lv, cd/m2.
<CCD/CMOS> <Lights> Subtractive, CYMK; Additive, RGB.
<Transformation in documents> Audio and video technologies changed the metric> Nonnegativity, Reflexivity ,Symmetry, Triangle inequality. <X-
book metaphor for documents; Hyperlinks in electronic documents and the means> Good scoring mechanism is difficult to determine. Traverse X for K. Compression can be achieved by: Lossy, DCT + Quantization; Ent., RLE +
introduction on hyperlinked pages on web allow documents to be read in
an order that a reader finds appropriate rather than what the author
intended; Now a document is no longer a compact and closed physical
artifact, but a dynamically and organically growing result of collaborative
authoring that presents multimedia in all its forms.
<Evolution of Documents>
Type of Media: Text remains an important component of documents, but
now documents use different media as the author sees fit; Author can
combine different media in space and time to communicate his ideas in the
most compelling manner; The same content segments in electronic form
can be used frequently as necessary. Non-Linear Flow: With the advent of <Differential coding> leverages global knowledge about the properties of
electronic media and the ability to create links, the limitation of linearity the signal to encode; widely used for compression; scales very well from
can be easily overcome; This provides a very flexible method of authoring
documents that may be customized for different types of audiences.
<Stages in Document Creation> Data acquisition and organization; Data
selection; Editing.<Synchronization> means establishing, specifying and
then performing coordination of multiple media sources precisely in space
and time. <Desirable Encoding Properties> The code must be
unambiguously decodable: One coded message corresponds to exactly one
decoded message; The code should be easily decodable: Able to find symbol algorithm predicts the color of each pixel based on the colors of previous
endings and the end of message easily, Able to decode online, that is as
symbol comes in, it can be decoded without knowing the entire coded
message; The code should be compact.
<Huffman code> A block code: each source symbol is mapped into a fixed
sequence of code symbols; Instantaneous decodable : each code word in a
string of code symbols can be decoded without referencing succeeding
symbols; Uniquely decodable: any string of code symbols can be decoded in
only one way by examining the individual symbols of the string in a left to
right manner. Limitation: only be encoded using an integer number of bits
per symbol. <VLC vs. FLC> Huffman Code is a common form of Variable
Length Coding: Assign shorter code word to higher probability symbol, and
vice versa. <Coding Algorithms> Truncated Huffman, Shift, Lempel-Ziv,
Arithmetic.<Variants of LZ77> LZR, LZH, DEFLATE (gzip).<78> LZC, LZW.
<Arithmetic coding> Unlike the variable-length codes described previously,
Arithmetic Coding generates nonblock codes: A one-to-one correspondence
between source symbols and code words does not exist; An entire
sequence of source symbols is assigned a single arithmetic code word; The
code word itself defines an interval or real numbers between 0 and 1; As
the number of symbols in the message increases, the interval used to
represent it becomes smaller and smaller; A termination symbol is usually
required. Technical Challenge: Precision problem, Accurate
representation of interval boundaries may not be possible due to limited forward/rewind operations. <I-Frame Coding> Block-based: Each frame
number of bits in representing numbers. 32/64 bits. is partitioned into non-overlapping blocks, Each 16x16 block is called
<Weakness of Entropy-based Compression Methods for Multimedia data> Macroblock. Each Macroblock contains: 4 Luminance (Y) 8x8 bloc, 2
usually a factor of two or less because: Entropy is usually higher for redundancies: Each frame is coded independently.
multimedia signals than for text data; Sampled signals contain noise, the
possibility for finding ideal repeating patterns is low; Multimedia data is
usually multidimensional.
<Lossy compression> leverages the fact that multimedia data can be
gracefully degraded in quality by increasingly losing more information.
Challenge: Tradeoff between quality and cost.
<Vector Quantization> is similar to quantisation, but instead of working on
a single-valued function, it works on tuples, triples, or higher dimensional
vectors. Common approach: Linear Quant, KX means. <Linear
Quantization> The easiest way to map n vectors to k vectors. <Distance
<Perceptual Quantization> Even though it is highly desirable to reproduce Huffman Encoding. DCT transforms the digitized signal in spatial domain
content as accurate as possible, but “accuracy” is not defined based on into a representation in frequency domain. Quantization removes some
simple mathematical distance; Two signals are accurately reproduced if “details” in the frequency domain, with graceful degradation in spatial
they percept the same, even they have many differences bitwise. domain. RLE transforms the quantized coefs. into a short sequence of
The simplest and yet most widely used perceptual quantization techniques (run, level) pairs. The (run, level) pairs can then be compressed easier by
for audio data include the <A/µ-law> µ=255, A=87.6. Both are conceptually Huffman Encoder. <Quantization in H.261> DC coef. quantized uniformly
similar, µ-law become a quasi-standard for low-bandwidth voice. with step size 8 with no deadzone, All other AC coefs. quantized uniformly
<Visual Quantization> The coexistence of a lip-synchronous audio track with deadzone and with the same step size. Zigzag Scanning. Scan
generally allows for lower frame rates because human brain is good at quantized coefficients in zig-zag order, Non-zero coefficients tend to be
filling missing information across modalities. grouped together. More compression, more quality loss. Blocky effect ,
Mosquito noise. <P-Frame Coding> block-based, same Macroblock
structure as in I-frame coding Frame predicted from previous I/P-frame.
lossless to entirely lossy. The main component of a differential encoder is P-frame explore both the spatial and temporal redundancies. Motion
the predictor, Lossy differential coding can be achieved by quantizing the Estimation+Motion Compensation. Residual: Motion Compensated diff.
difference values, Difference encoding is usually combined with entropy
encoding for better compression performance. The idea behind the
difference encoding of audio is that sound other than noise follows a
rather predictable wave pattern. PNG image format uses a differential
encoding step before an LZ-derivate entropy encoder is used. The
neighboring pixels. Adaptive Differential Pulse Code Modulation.
<MPEG-1> IBBPBBPBB. Encoding: IPBBPBBP. Increase memory
requirement: Two reference frames for B-predictions, Holding pending
frame for predictions. Increase computation requirement: Two motion
estimations for each block in a B-frame. B-Frames allows higher
compression ratio, B-Frames induces higher video latency. NG 4 live.
<Video compression> H.26_13456, MPEG-_124. How to achieve? Explore
redundancies in video sequences: Spatial, Temporal. Reduce irrelevant
data according to “Psycho Visual” Model.
<I/P-Frames> I-Frames provide access points. Serve as recovery points
when some frames are lost. Serve as probe points to support fast-