Evaluation of available MPEG-7

Annotation Tools

Mario Döller, Nikolaus Lefin, Harald Kosch

University of Passau, Germany
 Content

{ Motivation to this work

{ Annotation of multimedia data
z Facts and Problems
z Approach
z Standards
{ Evaluation Criteria
{ Comparison of MPEG-7 Annotation Tools
{ Conclusion
 Motivation
The effective (competitve) use of a meta data standard
shows in the:
z Number of working and maintained (further developed) tools
which allow the annotation and processing and exchanging of
meta data
{ For MPEG-7: MPEGIF provided a rather good list
(http://www.mpegif.org/resources.php#section37), but it
is outdated.
{ Reference SW is rather incomplete and complex to install,
but it is maintained: http://www.lis.e-technik.tu-
muenchen.de/research/bv/topics/mmdb/mpeg7.html
{ Better version on the way to commerce (Stephan
Herrmann is not joinable)
{ Therefore we examined several available indexing tools !
{ This talk presents actual implementations
(and supported ones)
 Annotation
Facts and Problems

{ Low-Level Features:
z Automatic extractable attributes stored in feature
vectors (e.g., color, texture, etc.).

{ High-Level Features:
z Semantic description of the content (e.g.,
handshake of two presidents, etc.).

{ Mid-Level Features:
(Mathias Gruhne 06)
z Attributes which can further processed to a
„meaningful“ or „semantic“ description (mostly
domain dependent and contour-based)

{ Abstract view of human beings can not be

detected automatically by computers.
z -> well known Semantic Gap
Annotation
Approach

{ manuel
z Annotation by human beings.
z Highest accuracy concerning on the description
of semantics.

{ automatic
z Calculated by machines based on specific
algorithms
z Object recognition based on various techniques
(e.g., segmentation)

{ semi-automatic
z Combination of manual and automatic
annotation.
z Combine efficiency and speed of the automatic
approach with the accuracy of the manual
approach.
 Annotation
Standards (update of a similar slide from 05 and 06)
Dublin
{ SMEF – Standard Media Exchange Core
Framework:
z BBC developed data models (to be DIG-35,
exchanged to TVA) P/Meta EXIF
{ P/Meta – EBU P/Meta Project:
z Exchange of program content
MPEG-7
Harmonized
between high-level business functions Model
(about 80% has been adapted to
TVA)
{ SMPTE – Metadata dictionary & MXF:
SMEF SMPTE

z Addresses Program Interchange

independent of format (MXF no longer
supported)
{ Dublin Core Metadata Initiative:
z Interoperable online metadata
standards was used as input to
P/Meta
{ TV-Anytime – TV-Anytime Metadata:
z Attractors/descriptors used e.g. in
Electronic Program Guides (EPG), or
in Web pages to describe content.
Controlled terms are available.
{ DIG-35, NISO Standards, EXIF
TV
Anytime
{ MPEG-7 Harmonized Model:
z Harmonized elements with
other standards and existing
practices
-> Mapping is possible!
z Registration authority for
classification schemes,
controlled terms, ontologies
(no real work)
Evaluation Criteria
{ Media
z media formats (e.g., video, audio, image)
z compression formats (e.g., JPEG,
{ Domain
z application dependent
z extensibility
{ Annotation
z manual, semi-, and/or automatic.
z correctness of the resulting MPEG-7 descriptions
{ Development Status
{ User Interface
z usability
z Documentation
{ Ability to integrate
 MPEG-7 Annotation Tools
Caliph & Emir

{ Caliph: well engineered tool for

semi-automatic annotation

{ Highlights:
z Drawing panel supporting semantic
description based on directed graphs
z Autopilot allows Batch-Annotation

Mathias Lux and other developer (greetings to him)

MPEG-7 Annotation Tools
Caliph & Emir
MPEG-7 Annotation Tools
Caliph & Emir
 MPEG-7 Annotation Tools
M-Ontomat-Annotizer
{ provides means for the annotation
of images and videos

{ Highlights:
z Supports ontologies for the annotation
of high level features.
z Visual Descriptior Extractor (VDE)
{ Contour detection and annotation
z fitting concepts can be selected within
an ontology browser

aceMedia project: Jonathon S. Hare, Patrick A. S. Sinclair, Paul H. Lewis,

Kirk Martinez, Peter G. B. Enser, and Christine J. Sandom.
MPEG-7 Annotation Tools
M-Ontomat-Annotizer
 MPEG-7 Annotation Tools
IBM VideoAnnEx
{ Annotation of video shots
{ Highlights:
z supports automatic shot detection
z Descriptive elements are organized as
lists
z Region Annotation: manuel tagging of
objects with bounding boxes.

IBM Watson: John R. Smith team

MPEG-7 Annotation Tools
IBM VideoAnnEx
MPEG-7 Annotation Tools
IBM VideoAnnEx
 MPEG-7 Annotation Tools
VIZARD – Video Wizard
{VIZARD is a Video Publishing Tool
well engineered annotation facilities

{ Highlights:
z VAnnotator supports automatic shot
detection
z VAnnotator supports manuel
segmentation of videos
z Lens Factory allows the assignment of
tags
Joanneum Research: H. Rehatschek group –
is there a newer free version available?
MPEG-7 Annotation Tools
VIZARD – Video Wizard
MPEG-7 Annotation Tools
VIZARD – Video Wizard
 MPEG-7 Annotation Tools
Marvel
{ Image and Video Retrieval Tool
{ Automatic extraction of low level features
on local multimedia collections
{ produces a HTML-CGI-Archiv database
(no access to MPEG-7 instances)
{ Highlights:
z Indexing, categorization and retrieval in large
multimedia collections
z uses content analyze techniques for
categorization
z Uses an internal concept archiv for indexing.

IBM Watson: John R. Smith team, used for Video TREC

MPEG-7 Annotation Tools
Marvel
 MPEG-7 Annotation Tools
MPEG-7 Audio Encoder
{ Java-library for annotation of audio
data
{ provides command line tool and GUI

{ Highlights:
z Einstellung der Deskriptoren möglich

Holger Crysandt RWTH Aachen, T-Systems and Universität Ancona

MPEG-7 Annotation Tools
MPEG-7 Audio Encoder
Short Overall Evaluation
 MPEG-7 Annotation Tools
Caliph & Emir - Evaluation
Media { media formats { images
{ compression { JPEG, PNG
formats

Domain { Area { no restriction *

{ Extensibility { arbitrary extensible **

Descriptors { validation { well formed, not valid 0

{ application { supports semantic and low level *
feature descriptors

Development { version { Version 0.9.23 *

{ system JAVA 5.0, Windows oder Linux **
Status
{
requirements

User Interface { interface { Java Swing, intuitiv *

{ documentation { adequate available *

Integration { data transfer { MPEG-7 instances *

{ interface { LIRE library available and **
documented

Annotation { manuel, semi-, { manuel for semantic, automatic **

automatic for EXIF and IPTC IIM data
{ feature extration { automatic extraction **
 MPEG-7 Annotation Tools
M-Ontomat-Annotizer- Evaluation
Media { media formats { Bilder, Videos
{ compression formats { JPEG, GIF, PNG, TIFF, MPEG, MOV

Domain { Area { via ontologies **

{ Extensibility { via ontologies **

Descriptors { validation { well formed, 6/8 valid 0

{ application { every D in a separate file 0

Development { version { Version 0.52 0

system requirements { Java 1.4.1, JMF for videos 0
Status
{
(Windows)

User Interface { interface { Java Swing, complex 0

{ documentation { adequate available *

Integration { data transfer { 8 different MPEG-7 instance files 0

{ interface { plugins, no documentation 0

Annotation { manuel, semi-, { Ds automatic, description of -

automatic ontology browser
{ feature extration { automatic, every D separat 0
 MPEG-7 Annotation Tools
IBM VideoAnnEx - Evaluation
Media { media formats { video
{ compression { MPEG
formats

Domain { Area { no restriction **

{ Extensibility { abitrarily extensible **

Descriptors { validation { well formed and valid **

{ application { no feature descriptors -

Development { version { Version 1.5. **

{ system Windows -
Status
{
requirements

User Interface { interface { well arranged GUI **

{ documentation { tutorial web site -

Integration { data transfer { MPEG-7 instance files and Frame *

{ interface Map File including shot detection
{ none available -

Annotation { manuel, semi-, { automatic shot detection, **

automatic description manuel
{ feature extration { only shot detection 0
 MPEG-7 Annotation Tools
VIZARD - Evaluation
Media { media formats { video *
{ compression { MPEG, AVI *
formats

Domain { Area { no restriction **

{ Extensibility { abitrary extensible **

Descriptors { validation { well formed, not valid 0

{ application { no feature extraction -

Development { version { beta version, 2003 -

{ system Windows 0
Status
{
requirements

User Interface { interface { GUI, clear **

{ documentation { no documentation 0

Integration { data transfer { MPEG-7 instance files, annotator *

{ interface XML file 0
{ no interface

Annotation { manuel, semi-, { semi-automatic *

automatic { no feature extraction 0
{ feature extration
 MPEG-7 Annotation Tools
IBM MARVel - Evaluation

Media { media formats { image, video **

{ compression { JPEG, PNG, GIF, PNM, TIFF, MPEG **
formats

Domain { Area { available concepts *

{ Extensibility { not documented 0

Descriptors { validation { MPEG-7 not available 0

{ application { MPEG-7 not available 0

Development { version { version 1.2, 2005 *

{ system Windows 0
Status
{
requirements

User Interface { interface { intuitive GUI *

{ documentation { none available 0

Integration { data transfer { internal database 0

{ interface { no interfaces available 0

Annotation { manuel, semi-, { automatic **

automatic { not documented 0
{ feature extration
 MPEG-7 Annotation Tools
MPEG-7 Audio Encoder - Evaluation

Media { media formats { Audio *

{ compression { WAVE, AU, MP3, AIFF **
formats

Domain { Area { no restriction **

{ Extensibility { no restriction **

Descriptors { validation { well formed, valid **

{ application { supports all 17 Audio D/DS **

Development { version { version 0.42, 2006, active **

{ system Java 1.4, JLayer for MP3 **
Status
{
requirements

User Interface { interface { command line tool, Java Webstart **

{ documentation GUI *
{ web site

Integration { data transfer { MPEG-7 instance files **

{ interface { JAR file **

Annotation { manuel, semi-, { automatic **

automatic { automatic, features selectable **
{ feature extration
Conclusion
{ the support for automatic or at least semi-
automatic annotation of low- and high level
features is rather poor.
{ only few tools are able to produce valid MPEG-7
descriptions which limit their usability.
{ highlights present
z Caliph in annotating images and
z VideoAnnEx for videos, whereas the annotation of
video is in most cases limited to simple shot
detection
z MPEG-7 Audio Encoder for audio is most
sophisticated
{ development of MPEG-7 frameworks promise to
improve the current situation as they can serve as
basis for more advanced tools