Chapter1

I t d ti t M lti di
IntroductiontoMultimedia

1.1WhatisMultimedia?
y Whendifferentpeople mentionthetermmultimedia,they
often have quite different, or even opposing, viewpoints.
oftenhavequitedifferent,orevenopposing,viewpoints.

1.1WhatisMultimedia?

APCvendor:aPCthathassoundcapability,aDVDROMdrive,and
perhapsthesuperiorityofmultimediaenabledmicroprocessors
that understand additional multimedia instructions
thatunderstandadditionalmultimediainstructions.

1 2 Multimedia and Hypermedia

1.2MultimediaandHypermedia

Aconsumerentertainmentvendor:interactivecableTVwith
hundredsofdigitalchannelsavailable,oracableTVlikeservice
delivered over a highspeed
deliveredoverahigh
speedInternetconnection.
Internet connection
AComputerScience(CS)student:applicationsthatusemultiple
modalities,includingtext,images,drawings(graphics),animation,
video sound including speech and interactivity
video,soundincludingspeech,andinteractivity.

1.3WorldWideWeb
1.4OverviewofMultimediaSoftwareTools

y MultimediaandComputerScience:
p
Graphics,HCI,visualization,computervision,datacompression,
graphtheory,networking,databasesystems.

1 5 F th E l ti
1.5FurtherExploration
1

ComponentsofMultimedia
p
y Multimediainvolvesmultiplemodalitiesoftext,
audio,images,drawings,animation,andvideo.
di i
d
i
i ti
d id
Examplesofhowthesemodalitiesareputto
use:
use:

Videoteleconferencing.
Di t ib t d l t
Distributedlecturesforhighereducation.
f hi h
d ti
Telemedicine.
Cooperativeworkenvironments.
Searching in(very)largevideoandimagedatabases
f t
fortargetvisualobjects.
t i l bj t
"Augmented"reality:placingrealappearing
computer graphics and video objects into scenes
computergraphicsandvideoobjectsintoscenes.
3

y Includingaudiocuesforwhere videoconference
participants are located
participantsarelocated.
y Buildingsearchablefeaturesintonewvideo,and
enabling very highto very lowbitrate use of new
enablingveryhightoverylowbitrateuseofnew,
scalablemultimediaproducts.
y Makingmultimediacomponentseditable.
Making multimedia components editable
y Building"inverseHollywood"applications thatcan
recreate the process by which a video was made.
recreatetheprocessbywhichavideowasmade.
Videounderstandinghasalsobeencalledaninverse
Hollywoodproblem.

y Usingvoicerecognition tobuildaninteractive
environment,sayakitchenwallwebbrowser.
4

MultimediaResearchTopicsandProjects
p
j

CurrentMultimediaProjects
j
y Manyexcitingresearchprojectsarecurrently
underway.Hereareafewofthem:
d
H
f
f th

y To
Tothecomputerscienceresearcher,multimedia
the computer science researcher, multimedia
consistsofawidevarietyoftopics:
Multimediaprocessingandcoding:multimediacontent
analysis,contentbasedmultimediaretrieval,multimedia
l i
t tb d
lti di
ti l
lti di
security,audio/image/videoprocessing,compression,etc.
Multimediasystemsupportandnetworking:network
y
pp
g
protocols,Internet, operatingsystems,serversandclients,
qualityofservice(QoS),anddatabases.
Multimediatools,end
Multimedia tools endsystems
systemsandapplications:
and applications:
hypermediasystems,userinterfaces,authoringsystems.
Multimodalinteractionandintegration:"ubiquity"
webeverywheredevices,multimediaeducationincluding
b
h
d i
lti di d ti i l di
ComputerSupportedCollaborativeLearning,anddesign
andapplicationsofvirtualenvironments.

1. Camerabasedobjecttrackingtechnology:trackingof
th
thecontrolobjectsprovidesusercontroloftheprocess.
t l bj t
id
t l f th
2. 3Dmotioncapture:usedformultipleactorcaptureso
that multiple real actors in a virtual studio can be used
thatmultiplerealactorsinavirtualstudiocanbeused
toautomaticallyproducerealisticanimatedmodels with
naturalmovement.
3. Multipleviews:allowingphotorealistic(videoquality)
synthesisofvirtualactorsfromseveralcamerasorfrom
asinglecameraunderdiering
i l
d di i lighting.
li h i
4. 3Dcapturetechnology:allowsynthesisofhighly
realistic facial animation from speech
realisticfacialanimationfromspeech.

1.2MultimediaandHypermedia
yp
5. Specific multimediaapplications:aimedat
handicapped persons with low vision capability
handicappedpersonswithlowvisioncapability
andtheelderly arichfield ofendeavor.
6. Digitalfashion:aimstodevelopsmartclothingthat
6.
Digital fashion: aims to develop smart clothing that
cancommunicatewithothersuchenhanced
clothingusingwirelesscommunication,soasto
artificially enhancehumaninteractioninasocial
enhance human interaction in a social
setting.
7 ElectronicHousecall
7.
Electronic Housecall system:aninitiativefor
system: an initiative for
providinginteractivehealthmonitoringservicesto
patientsintheirhomes
8. AugmentedInteractionapplications:usedto
developinterfacesbetweenrealandvirtual
humans for tasks such as augmented storytelling
humansfortaskssuchasaugmentedstorytelling.
7

y HistoryofMultimedia:
1. Newspaper:perhapsthefirst masscommunication
medium,usestext,graphics,andimages.
2. Motionpictures:conceivedofin1830'sinorderto
observemotiontoorapidforperceptionbythe
humaneye.
3. Wirelessradiotransmission:GuglielmoMarconi,at
Pontecchio,Italy,in1895.
4. Television:thenewmediumforthe20thcentury,
establishedvideoasacommonlyavailablemedium
andhassincechangedtheworldofmass
communications.
communications
8

5. Theconnectionbetweencomputersandideasaboutmultimediacovers
what is actually only a short period:
whatisactuallyonlyashortperiod:
ComponentsofMultimedia

1985 NegroponteandWiesner cofoundedtheMITMediaLab.

1989 TimBernersLeeproposedtheWorldWideWeb

MultimediaResearchTopicsandProjects

1990 KristinaHooperWoolseyheadedtheAppleMultimediaLab.

CurrentMultimediaProjects

1991 MPEG1
1991
MPEG 1 wasapprovedasaninternationalstandardfordigital
d
i
i
l
d d f di i l
video ledtothenewerstandards,MPEG2,MPEG4,and
furtherMPEGsinthe1990s.

1945 Vannevar Bushwrotealandmarkarticledescribingwhatamountsto

ahypermediasystemcalledMemex.

1991 TheintroductionofPDAs
h
d
f
in1991begananewperiodinthe
b
d
h
useofcomputersinmultimedia.

LinktofullV.Bush1945Memex article,"AsWeMayThink"

1960 TedNelsoncoinedthetermhypertext.
1967 NicholasNegroponteformedtheArchitectureMachineGroup.
g p
p
1968 DouglasEngelbart demonstratedtheOnLineSystem(NLS),another
veryearlyhypertextprogram.
1969 NelsonandvanDamatBrownUniversitycreatedanearlyhypertext
1969
Nelson and van Dam at Brown University created an early hypertext
editorcalledFRESS.
1976 TheMITArchitectureMachineGroupproposedaprojectentitled
MultipleMedia
p
resultedintheAspenMovieMap,thefirst
p
p,
hypermediavideodisk,in1978.

1992 JPEG wasacceptedastheinternationalstandardfordigital

imagecompression ledtothenewJPEG2000standard.
1992 Thefirst MBone audiomulticastontheNetwasmade.
1993 TheUniversityofIllinoisNationalCenterforSupercomputing
1993
The University of Illinois National Center for Supercomputing
ApplicationsproducedNCSAMosaic thefirst fullfledged
browser.

10

HypermediaandMultimedia
yp
1994 JimClarkandMarcAndreessencreatedthe
Netscape program
Netscapeprogram.
1995 TheJAVA languagewascreatedforplatform
independentapplicationdevelopment.
p
pp
p
1996 DVDvideowasintroduced;highqualityfulllength
moviesweredistributedonasingledisk.
1998 XML 1.0wasannouncedasaW3C
Recommendation.
1998 HandheldMP3devicesfirst madeinroadsinto
consumeristtastesinthefallof1998,withthe
introduction of devices holding 32MB of flash
introductionofdevicesholding32MBofflash
memory.
2000 WWWsizewasestimatedatover1billionpages.
11

y Ahypertextsystem:meanttobereadnonlinearly,
b f ll i li k thatpointtootherpartsofthe
byfollowinglinks
th t i t t th
t f th
document,ortootherdocuments(Fig.1.1)
y HyperMedia:notconstrainedtobetextbased, can
includeothermedia,e.g.,graphics,images,and
especially the continuous media soundandvideo.
especiallythecontinuousmedia
sound and video
TheWorldWideWeb(WWW) thebestexampleofa
hypermedia application
hypermediaapplication.

y Multimediameansthatcomputerinformationcan
be represented through audio graphics images
berepresentedthroughaudio,graphics,images,
video,andanimationinadditiontotraditional
ed a
media.
12

y Examplesoftypicalpresentmultimediaapplications
include:
include:

Digitalvideoeditingandproduction systems.
El t i
Electronicnewspapers/magazines.
/
i
WorldWideWeb.
O li
Onlinereferenceworks:e.g.encyclopedias,games,etc.
f
k
l
di
Homeshopping.
InteractiveTV.
Multimediacourseware.
Videoconferencing.
Videoondemand.
Interactivemovies.

13

14

1.3WorldWideWeb
y TheW3Chaslistedthefollowinggoalsforthe
WWW:
WWW

1990 TimBernersLeeinventedtheHyperText
M k L
MarkupLanguage(HTML),andtheHyperText
(HTML) d th H
T t
TransferProtocol(HTTP).

1. Universalaccessofwebresources(byeveryone
everywhere).
h )

1993 NCSAreleasedanalphaversionofMosaic
basedontheversionbyMarcAndreessenfor
X Windows thefirst
XWindows
the first popularbrowser.
popular browser

2. Eectiveness ofnavigatingavailableinformation.
3. Responsibleuseofpostedmaterial.

y HistoryofWWW
1960s CharlesGoldfarbetal.developedthe
p
g g (
)
GeneralizedMarkupLanguage(GML)forIBM.
1986 TheISOreleasedafinal versionofthe
Standard Generalized Markup Language
StandardGeneralizedMarkupLanguage
(SGML).
15

1994 MarcAndreessenetal.formedMosaic
C
CommunicationsCorporation
i ti
C
ti
l t th
laterthe
NetscapeCommunicationsCorporation.
1998 TheW3CacceptedXML
1998
Th W3C
d XML version1.0
i 10
specifications asaRecommendation the
main focus of the W3C and supersedes HTML
mainfocusoftheW3CandsupersedesHTML.
16

HTTP(HyperTextTransferProtocol)
( yp
)
y HTTP:aprotocolthatwasoriginallydesignedfor
transmitting hypermedia but can also support the
transmittinghypermedia,butcanalsosupportthe
transmissionofanyfile type.
y HTTPisastateless
HTTP is a stateless request/responseprotocol:no
request/response protocol: no
informationcarriedoverforthenextrequest.
y Thebasicrequestformat:
The basic request format:
MethodURIVersion
AdditionalHeaders:
Messagebody

Twopopularmethods:GETandPOST.
Thebasicresponseformat:
VersionStatusCodeStatusPhrase
AdditionalHeaders
Additional Headers
Messagebody

Twocommonlyseenstatuscodes:

y TheURI(UniformResourceIdentifier):anidentifier
The URI (Uniform Resource Identifier): an identifier
fortheresourceaccessed,e.g.thehostname,
y p
y
p
alwaysprecededbythetoken"http://".

1.200OK
1 200 OK therequestwasprocessed
th
t
d
successfully.
2.404NotFound theURIdoesnot exist.

17

18

HTML(HyperTextMarkupLanguage)
( yp
p
g g )
y HTML:alanguageforpublishingHypermediaonthe
W ld Wid W b defined
WorldWideWeb
d fi d usingSGML:
i SGML

y AverysimpleHTMLpageisasfollows:

1.HTMLusesASCII,itisportabletoalldierent (possibly
bi
binaryincompatible)computerhardware.
i
tibl )
t h d
2.ThecurrentversionofHTMLisversion4.01.
3.ThenextgenerationofHTMLisXHTML a
reformulationofHTMLusingXML.

y HTMLusestagstodescribedocumentelements:
<tokenparams> defining astartingpoint,
</token> theendingpointoftheelement.

y Naturally,HTMLhasmorecomplexstructuresand
canbemixedinwithotherstandards.
b
d
h h
d d

g g
Someelementshavenoendingtags.
19

20

XML(ExtensibleMarkupLanguage)
(
p
g g )
y XML:amarkuplanguagefortheWWWinwhich
th
thereismodularityofdata,structureandviewso
i
d l it f d t t t
d i
thatuserorapplicationcanbeabletodefine the
tags (structure)
tags(structure).
y ExampleofusingXMLtoretrievestockinformation
from a database according to a user query:
fromadatabaseaccordingtoauserquery:
1. FirstuseaglobalDocumentTypeDefinition (DTD) that
is already defined
isalreadydefined.
2. TheserversidescriptwillabidebytheDTDrulesto
generate an XML document according to the query
generateanXMLdocumentaccordingtothequery
usingdatafromyourdatabase.
y
y
( ) p
g
3. FinallysendusertheXMLStyleSheet(XSL)depending
onthetypeofdeviceusedtodisplaytheinformation.

y ThecurrentXMLversionisXML1.0,approved
bytheW3CinFeb.1998.
y
y
,
g
y XMLsyntaxlookslikeHTMLsyntax,althoughit
ismuchmorestrict:
All
Alltagsareinlowercase,andatagthathasonly
tags are in lower case and a tag that has only
inlinedatahastoterminateitself,i.e.,<token
params />.
UsesnamespacessothatmultipleDTDsdeclaring
dierent elementsbutwithsimilartagnamescan
havetheirelementsdistinguished.
p
DTDscanbeimportedfromURIsaswell.

21

22

y ThefollowingXMLrelatedspecifications are
alsostandardized:
l
t d di d

AnexampleofanXMLdocument
structure thedefinitionforasmall
XHTML document:
XHTMLdocument:

XMLProtocol:usedtoexchangeXMLinformation
between processes
betweenprocesses.
XMLSchema:amorestructuredandpowerful
language for defining XMLdatatypes(tags).
languagefordefining
XML data types (tags)
XSL: basicallyCSSforXML.
SMIL:synchronizedMultimediaIntegration
SMIL synchronized Multimedia Integration
Language,pronounced"smile" aparticular
applicationofXML(globallypredefined DTD)that
allowsforspecification ofinteractionamongany
mediatypesanduserinput,inatemporally
scripted manner.
scriptedmanner.
23

24

SMIL (SynchronizedMultimedia
I t
IntegrationLanguage)
ti L
)
y PurposeofSMIL:itisalsodesirabletobeableto
publish multimedia presentations using a markup
publishmultimediapresentationsusingamarkup
language.
y Amultimediamarkuplanguageneedstoenable
A multimedia markup language needs to enable
scheduling andsynchronization ofdierent
multimediaelements,anddefine theirinteractivity
withtheuser.
ih h
y TheW3CestablishedaWorkingGroupin1997to
comeupwithspecifications
ith
ifi ti
f
foramultimedia
lti di
synchronizationlanguage
SMIL2.0wasacceptedinAugust2001.
SMIL 2 0 was accepted in August 2001

y SMIL2.0isspecified inXMLusingamodularization
approach similar to the one used in xhtml:
approachsimilartotheoneusedinxhtml:
25

1. AllSMILelementsaredividedintomodules sets
ofXMLelements,attributesandvaluesthatdefine
f XML l
t tt ib t
d l
th t d fi
oneconceptualfunctionality.
2 IIntheinterestofmodularization,notallavailable
2.
th i t
t f
d l i ti
t ll
il bl
modulesneedtobeincludedforallapplications.
3 LLanguageProfiles:specifies
3.
P fil
ifi aparticulargrouping
i l
i
ofmodules,andparticularmodulesmayhave
integration requirements that a profile must
integrationrequirementsthataprofile
must
follow.
x SMIL
SMIL2.0hasamainlanguageprofile
2.0 has a main language profile thatincludesalmost
that includes almost
allSMILmodules.

y BasicelementsofSMILasshowninthe
Basic elements of SMIL as shown in the
followingexample:

26

1.4OverviewofMultimediaSoftware
T l
Tools
Thecategoriesofsoftwaretoolsbriefly
examinedhereare:
1 Music Sequencing and Notation
1.MusicSequencingandNotation
2.DigitalAudio
3.GraphicsandImageEditing
4 Vid Edi i
4.VideoEditing
5.Animation
6.MultimediaAuthoring
27

28

MusicSequencingandNotation
q
g

DigitalAudio
g

y Cakewalk: nowcalledProAudio.
Thetermsequencercomesfromolderdevicesthat
storedsequencesofnotes("events",inMIDI).
ItisalsopossibletoinsertWAVfiles
It is also possible to insert WAV files andWindowsMCI
and Windows MCI
commands(foranimationandvideo)intomusictracks
(MCIisaubiquitouscomponentoftheWindowsAPI.)

y Cubase:anothersequencing/editingprogram,with
capabilitiessimilartothoseofCakewalk.Itincludes
some digital audio editing tools
somedigitalaudioeditingtools.
y MacromediaSoundedit:matureprogramfor
creating audio for multimedia projects and the web
creatingaudioformultimediaprojectsandtheweb
thatintegrateswellwithotherMacromedia
productssuchasFlashandDirector.

y DigitalAudiotoolsdealwithaccessingand
editingtheactualsampledsoundsthatmake
diti th
t l
l d
d th t k
upaudio:
C
CoolEdit:avery
l Edit
powerfulandpopulardigitalaudio
f l d
l di it l di
toolkit;emulatesaprofessionalaudiostudio
multitrack p
productionsandsoundfile editing
g
includingdigitalsignalprocessingeects.
SoundForge:asophisticatedPCbasedprogramfor
editingaudioWAVfiles.
di i
di WAV fil
ProTools:ahighendintegratedaudioproduction
and editing environment MIDIcreationand
andeditingenvironment
MIDI creation and
manipulation;powerfulaudiomixing,recording,
andeditingsoftware.

29

GraphicsandImageEditing
p
g
g

30

VideoEditingg

y AdobeIllustrator:apowerfulpublishingtoolfromAdobe.
Uses vector graphics; graphics can be exported to Web
Usesvectorgraphics;graphicscanbeexportedtoWeb.
y AdobePhotoshop:thestandardinagraphics,image
p
processingandmanipulationtool.
g
p
Allowslayersofimages,graphics,andtextthatcanbe
separatelymanipulatedformaximumflexibility.
Filterfactorypermitscreationofsophisticatedlightingeects
filters.

y Macromedia
MacromediaFireworks:softwareformakinggraphics
Fireworks: software for making graphics
specificallyfortheweb.
y MacromediaFreehand:atextandwebgraphicsediting
g p
g
toolthatsupportsmanybitmapformatssuchasGIF,
PNG,andJPEG.
31

y AdobePremiere:anintuitive,simplevideoediting
tool for nonlinear editing i e putting video clips
toolfornonlinearediting,i.e.,puttingvideoclips
intoanyorder:
Video
Videoandaudioarearrangedin
and audio are arranged in "tracks"
tracks .
Providesalargenumberofvideoandaudiotracks,
superimpositionsandvirtualclips.=>effective
multimediaproductionswithlittleeffort.

y AdobeAfterEffects:apowerfulvideoeditingtool
th t
thatenablesuserstoaddandchangeexisting
bl
t dd d h
i ti
movies.Canaddmanyeects:lighting,shadows,
g; y
motionblurring;layers.
y FinalCutPro:avideoeditingtoolbyApple;
Macintoshonly.
32

Animation
y RenderingTools:

y MultimediaAPIs:
Java3D:APIusedbyJavatoconstructandrender3D
graphics, similar to the way in which the Java Media
graphics,similartothewayinwhichtheJavaMedia
Frameworkisusedforhandlingmediafiles.
1 Provides
1.
Providesabasicsetofobjectprimitives(cube,splines,
a basic set of object primitives (cube splines
etc.)forbuildingscenes.
y
p
p
2. ItisanabstractionlayerbuiltontopofOpenGLorDirectX
(theusercanselectwhich).

DirectX :WindowsAPIthatsupportsvideo,images,
: Windows API that supports video, images,
audioand3Danimation
OpenGL:thehighlyportable,mostpopular3DAPI.
OpenGL: the highly portable most popular 3 D API

3DStudioMax:renderingtoolthatincludesanumberof
veryhighendprofessionaltoolsforcharacteranimation,
ggamedevelopment,andvisualeects
p
,
production.
p
SoftimageXSI:apowerfulmodeling,animation,and
renderingpackageusedforanimationandspecial
eects infilms
in films andgames.
and games
Maya:competingproducttoSoftimage;aswell,itisa
complete modeling package.
completemodelingpackage.
RenderMan:renderingpackagecreatedbyPixar.

y GIFAnimationPackages:asimplerapproachto
GIF Animation Packages: a simpler approach to
animation,allowsveryquickdevelopmentof
eective smallanimationsfortheweb.

33

34

MultimediaAuthoringg

1.5FurtherExploration
p

y MacromediaFlash:allowsuserstocreateinteractive
movies by using the score metaphor i e a timeline
moviesbyusingthescoremetaphor,i.e.,atimeline
arrangedinparalleleventsequences.
y MacromediaDirector:usesamoviemetaphorto
Macromedia Director: uses a movie metaphor to
createinteractivepresentations verypowerful
andincludesabuiltinscriptinglanguage,Lingo,that
allowscreationofcomplexinteractivemovies.
ll
i
f
l i
i
i
y Authorware:amature,wellsupportedauthoring
productbasedontheIconic/Flowcontrol
d t b d th I i /Fl
t l metaphor.
t h
y Quest: similartoAuthorware inmanyways,usesa
type of flowcharting metaphor.However,the
typeofflowcharting
metaphor However the
flowchart nodescanencapsulateinformationina
moreabstractway(calledframes)thansimply
subroutinelevels.

LinktoFurtherExplorationforChapter1.

35

y InChapter1oftheFurtherExplorationdirectory,the
website provides links to much of the history of
websiteprovideslinkstomuchofthehistoryof
multimedia.
Other links in the text website include information on:
y Otherlinksinthetextwebsiteincludeinformationon:

TedNelsonandtheXanadu project.
Nicholas Negroponte'ssworkattheMITMediaLab.
NicholasNegroponte
work at the MIT Media Lab.
DouglasEngelbart,andthehistoryofthe"OnLineSystem".
TheMITMediaLab
Clientsideexecution.

36