You are on page 1of 36

Cross Language Information Technologies

Free University of Bozen, Italy

Multi-media Retrieval Systems


based on
User Centered Design Methodology:
from the Content-based to the Semantic-based approach
dott. ing.Tania Di Mascio
EURAC RESEARCH BOZEN

Tania Di Mascio, Eurac Research

Bozen (Italy), 22 december

Outline
Outline

Introduction
Definitions
Open issues

Multi-media Retrieval Systems [20,21]


Content Based Image Retrieval Systems for Vector Images [4,7,9,22]
Ontology Based Text Retrieval Systems for the Semantic Web [4,7,9,22]
Ontology Based Multimedia Retrieval Systems for LIS [22, 23]

Human Computer Interaction [1,3]


Information Visualization

Vector Images Visualization [2,9,22]

Ontology Visualization [5,6]


Usability Evaluation

Usability Evaluation for software system [10,13]

Usability Evaluation for web site [11]

Publications

Tania Di Mascio, Eurac Research

Bozen (Italy), 22 december

Introduction
Introduction definitions

Multi-media Information
Text

Retrieval Systems
Content-Based

Images

Logical-Based

Sounds
Combinations (e.g. paint)

Ontology-Based

Ontology Based Text Retrieval Systems


Content Based Image Retrieval Systems
Ontology Based Multimedia Retrieval Systems
Tania Di Mascio, Eurac Research

Bozen (Italy), 22 december

Open Issues
Introduction open issues

Multi-media Information Retrieval Systems [20, 21]


Content Based
Ontology Based
Ontology Based
Image
Multimedia
Text
Retrieval Systems Retrieval Systems Retrieval Systems
vector images

LIS

Semantic web

Content Based Image Retrieval Systems for Vector images [4, 7, 9, 22]
Ontology Based Text Retrieval Systems for Semantic Web [8, 12]
Ontology Based Multimedia Retrieval Systems for LIS [22, 23]

Retrieval Issue
Tania Di Mascio, Eurac Research

Bozen (Italy), 22 december

Open Issues
Introduction open issues

Human Computer Interaction [1, 3]


visualization
usability

Vector images Visualization [2, 9, 22]


Ontology Visualization [5, 6]
Usability Evaluation for software systems [10, 13]
Usability Evaluation for software Web systems [11]

Interaction Issue
Tania Di Mascio, Eurac Research

Bozen (Italy), 22 december

Outline
Outline

Introduction

Multi-media Retrieval Systems [20,21]


Content Based Image Retrieval Systems for Vector Images [4,7,9,22]
Ontology Based Text Retrieval Systems for the Semantic Web [4,7,9,22]
Ontology Based Multimedia Retrieval Systems LIS [22, 23]

Human Computer Interaction [1,3]


Information Visualization

Vector Images Visualization [2,9,22]

Ontology Visualization [5,6]


Usability Evaluation

Usability Evaluation for software system [10,13]

Usability Evaluation for web site [11]

Publications

Tania Di Mascio, Eurac Research

Bozen (Italy), 22 december

Content-Based Image Retrieval Systems


Multimedia Retrieval Systems Content Based Image Retrieval Systems
Users

Users Interface

Query Resut
Module

Query Interface
Module

Engine System
Query
Image

Query Image
Indexed
Feature
Extraction
Module

Ranking
Module

Feature
Representation
Module

Similarity
Computation
Module

Feature Process

DB Images

Tania Di Mascio, Eurac Research

Indexing
Repository

Compairison
Process
Bozen (Italy), 22 december

Content-Based Image Retrieval Systems


Multimedia Retrieval Systems Content Based Image Retrieval Systems Retrieval Issue

CBIR Engine
Query
Image

Query Image
indexed
Feature
Extraction
Module

Ranking
Module

Feature
Representation
Module

Similarity
Computation
Module

Feature Process

DB Images

Indexing
Repository

Compairison
Process

Mathematic and Physic approach to solve the Retrieval Issue


Tania Di Mascio, Eurac Research

Bozen (Italy), 22 december

Content-Based Image Retrieval Systems


Multimedia Retrieval Systems Content Based Image Retrieval Systems Retrieval Issue

Feature Extraction Module

m
i =0

* xi

= xCM

mi
i =0

m * y
i

i= 0
n

mi

= yCM

i= 0

Center of Mass
Calculation
Process

System Origin
Moving
Process

Discretizing
Process

Tania Di Mascio, Eurac Research

Bozen (Italy), 22 december

Content-Based Image Retrieval Systems


Multimedia Retrieval Systems Content Based Image Retrieval Systems Retrieval Issue

Feature Representation Module

Feature Descriptor
9 Hu Moments
Regular Moments
9 Zernike Moments

mi ; j =

i
j
(
)
(
)
x

CM
CM
i; j

9 Bamieh Moments
Tania Di Mascio, Eurac Research

Bozen (Italy), 22 december

Content-Based Image Retrieval Systems


Multimedia Retrieval Systems Content Based Image Retrieval Systems Retrieval Issue

Similarity Computation Module

Similarity Measures
9Euclidean Distance
9 Cross Correlation

DE =
D=

wi (FP FI
N

w F
i =0

Tania Di Mascio, Eurac Research

Pi

FI i

w F w F
max ( p , x )
D(i, v ) =
1
(
)
min
,
p
x

J =1

9 Discrimination Cost

i =1 N

j =1

Pj

k =1

Ik

vj

ij

vj

ij

Bozen (Italy), 22 december

Content-Based Image Retrieval Systems


Multimedia Retrieval Systems Content Based Image Retrieval Systems Interaction Issue

Tuning Interface

Tania Di Mascio, Eurac Research

Bozen (Italy), 22 december

Content-Based Image Retrieval Systems


Multimedia Retrieval Systems Content Based Image Retrieval Systems Interaction Issue

Search Interface
VISTO (Visual Image Search TOol)

Tania Di Mascio, Eurac Research

Bozen (Italy), 22 december

Ontology-Based Text Retrieval Systems


Multimedia Retrieval Systems Ontology Based Text Retrieval Systems Retrieval Issue

Sewasie Interaction Layer


Ontology
Editor
Tool

OLAP
Tool

Query
Tool

Negotiation
Tool

Sewasie
User
Profile

Data

MetaData

Sewasie Core System

Query
Agent

Brokering
Agent

Internet
Internet

SINode
Repository

Tania Di Mascio, Eurac Research

SINode
Repository

..

SINode

Repository

Bozen (Italy), 22 december

Ontology-Based Text Retrieval Systems


Multimedia Retrieval Systems Ontology Based Text Retrieval Systems Retrieval Issue

The intelligence of the Sewasie Query tool is driven by an


ontology.
the User perspective

describes the domain of the


data in the information
system;
defines a vocabulary which
is:
richer than the logical
schema of data,
closer than the users
rich vocabulary.
Tania Di Mascio, Eurac Research

the System perspective

enables only the


generation of precise and
unambiguous query
expressions.

Bozen (Italy), 22 december

Ontology-Based Text Retrieval Systems


Multimedia Retrieval Systems Ontology Based Text Retrieval Systems Interaction Issue

Retrieval
Data

Query Tool
Ontology
Editor
Tool

MetaData

Tania Di Mascio, Eurac Research

Query
Building
Module

Query
Result
Module

Data

OLAP
Visualization
Tool

Sewasie
User
Profile

Single
Item
Negotiation
Communication
Tool

Bozen (Italy), 22 december

Ontology-Based Text Retrieval Systems


Multimedia Retrieval Systems Ontology Based Text Retrieval Systems Interaction Issue

Query Tool Interface


the User perspective

to support the user in


formulating a precise
query in order to best
captures users
information needs;
complete ignorance
Tania Di Mascio, Eurac Research

the System perspective

to generate a query ready


to be executed by some
evaluation engine
associated to the
information system;
conjunctive query
Bozen (Italy), 22 december

Ontology-Based Text Retrieval Systems


Multimedia Retrieval Systems Ontology Based Text Retrieval Systems Interaction Issue

Query Tool Interface

Information Domains;
Query start;
Query Compose;
Query result;

Tania Di Mascio, Eurac Research

Bozen (Italy), 22 december

Ontology-Based Multimedia Retrieval Systems


Multimedia Retrieval Systems Ontology Based Multimedia Retrieval Systems e-LIS

Tania Di Mascio, Eurac Research

Bozen (Italy), 22 december

Ontology-Based Multimedia Retrieval Systems


Multimedia Retrieval Systems Ontology Based Multimedia Retrieval Systems Retrieval Issues

1) String
2) Image
ES2:

ES1:(Con
Tania Di Mascio, Eurac Research

Ori

Loc

Mov not )
Bozen (Italy), 22 december

Ontology-Based Multimedia Retrieval Systems


Multimedia Retrieval Systems Ontology Based Multimedia Retrieval Systems Interaction Issue

static Interface

dynamic Interface
based on
constraint visualisation

Users can select


parameters

Tania Di Mascio, Eurac Research

Users are guided in parameters


choosing i.e.

Bozen (Italy), 22 december

Outline
Outline

Introduzione

Definizione

Problemi Aperti

Multimedia Retrieval Systems [20,21]

Content Based Image Retrieval Systems for Vector Images [4,7,9,22]

Ontology Based Text Retrieval Systems for the Semantic Web [4,7,9,22]

Ontology Based Multimedia Retrieval Systems [22, 23]

Human Computer Interaction [1,3]

Information Visualization

Vector Images Visualization [2,9,22]

Ontology Visualization [5,6]

Usability Evaluation

Usability Evaluation for Software system [10,13]

Usability Evaluation for Web site [11]

Publications

Tania Di Mascio, Eurac Research

Bozen (Italy), 22 december

Vector Images Visualization


Human Computer Interaction Information Visualization Ontology Visualization e-LIS

Movement Parameters as animation SVG

Tania Di Mascio, Eurac Research

Bozen (Italy), 22 december

Vector Images Visualization


Human Computer Interaction Information Visualization Vector Images Visualization VISTO

Result as Similarity Ranking List

Tania Di Mascio, Eurac Research

Bozen (Italy), 22 december

Outline
Outline

Introduzione

Definizione

Problemi Aperti

Multimedia Retrieval Systems [20,21]

Content Based Image Retrieval Systems for Vector Images [4,7,9,22]

Ontology Based Text Retrieval Systems for the Semantic Web [4,7,9,22]

Ontology Based Multimedia Retrieval Systems [22, 23]

Human Computer Interaction [1,3]

Information Visualization

Vector Images Visualization [2,9,22]

Ontology Visualization [5,6]

Usability Evaluation

Usability Evaluation for Software system [10,13]

Usability Evaluation for Web site [11]

Publications

Tania Di Mascio, Eurac Research

Bozen (Italy), 22 december

User Centered Design Methodology


Human Computer Interaction Usability Evaluation

Context analysis

survey evaluation

Usability goals

Tania Di Mascio, Eurac Research

I prototype

survey evaluation
observational evaluation
(Think Aloud and Verbal Protocols)

II prototype

survey evaluation
observational evaluation
(Think Aloud and Verbal Protocols)
controlled experiment

Final prototype

survey evaluation
observational evaluation
(Think Aloud and Verbal Protocols)
controlled experiment

Product

survey evaluation
observational evaluation
(Think Aloud and Verbal Protocols)
controlled experiment
Bozen (Italy), 22 december

User Centered Design Methodology


Human Computer Interaction Usability Evaluation

User-Based Criteria
A common Think Aloud Protocol method schema
Users

Analysing Users

Designing
the experiment

Performing
the experiment

Teaching
Users

Analysing the
usability

Tania Di Mascio, Eurac Research

Engineers

Bozen (Italy), 22 december

Ontology-Based Text Retrieval Systems


Human Computer Interaction Usability Evaluation Software System

Usability tests of the Query tool aims at perform:


The effectiveness: it is evaluated by relating the goals or
sub-goals of using the Query tool to the accuracy and
completeness with which these goals can be achieved.
The efficiency: it is evaluated by relating the time spent
to show query result.
The satisfaction: it is evaluated by relating the comfort
and acceptability of the Query tool interface.

Tania Di Mascio, Eurac Research

Bozen (Italy), 22 december

Us
ers

Analy
sing
User
s

Perfor
ming
the
experi
ment

Ontology-Based Text Retrieval Systems

Teach
ing
Users

Analysi
ng the
usabilit
y

Human Computer Interaction Usability Evaluation Software Systems

For each query

A complexity tree

Level
l=1

P1
C1

l=2

P1
C1
P1

l=3

l=4

Tania Di Mascio, Eurac Research

C2
P2

C1

C1

P1

P2

Desig
ning
the
exper
iment

C1
P3
C2
P3

Bozen (Italy), 22 december

En
gin
eer
s

Us
ers

Ontology-Based Text Retrieval Systems

Desig
ning
the
exper
iment

Analy
sing
User
s

Perfor
ming
the
experi
ment
Teach
ing
Users
Analysi
ng the
usabilit
y

En
gin
eer
s

Human Computer Interaction Usability Evaluation Software Systems

C1 = 0,1

if edge = add proprierty

P1
C1

C2 = 0,2

P1

if edge = add concept

C1

C2

P1

Pn = n

num _ lel

l =1

l * c1; 2

C1

for each node of level

Tania Di Mascio, Eurac Research

num _ nod _ per _ level


n =1

pn (l ) * f (n, l )

P1

C1

P2
C1

P3
C2

P2

P3

evr _ num _ suc _ level If l < lmax


f (n, l ) =
1
If l=lmax

Bozen (Italy), 22 december

Us
ers

Ontology-Based Text Retrieval Systems

Analy
sing
User
s

Perfor
ming
the
experi
ment
Teach
ing
Users
Analysi
ng the
usabilit
y

En
gin
eer
s

Human Computer Interaction Usability Evaluation Software Systems

For example:
Supplier located in Europe which is a Multinational selling on
the Italian Market
Supplier

1
0,2

Which is a

Multinational

Europe
0,1

selling_on

0,1

l=1

located_in

l=2

l=3

Italian_market

The complexity is: 1*[0.1*1]*2+2*[0.2*1+0.1*2]*0.5+3*[0.1*1]*1 = 0.9


Tania Di Mascio, Eurac Research

Desig
ning
the
exper
iment

Bozen (Italy), 22 december

DIEL web-site Systems


Human Computer Interaction Usability Evaluation DIEL Web Systems

Tania Di Mascio, Eurac Research

Effectiveness measure (*)

TRAD

BOX

Domain Expert

33,4%

20,07%

Non Domain Expert

29,7%

17,02%

Computer Skilled

11,7%

5,2%

Non Computer Skilled

35,3%

21,6%
Bozen (Italy), 22 december

Outline
Outline

Introduzione

Multimedia Retrieval Systems [20,21]


Content Based Image Retrieval Systems for Vector Images [4,7,9,22]
Ontology Based Text Retrieval Systems for the Semantic Web [4,7,9,22]

Human Computer Interaction [1,3]


Information Visualization

Vector Images Visualization [2,9,22]

Ontology Visualization [5,6]


Usability Evaluation

Usability Evaluation for Software system [10,13]

Usability Evaluation for Web site [11]

Publications

Tania Di Mascio, Eurac Research

Bozen (Italy), 22 december

Publications
Publications

Libri Editati

[1] L. Tarantino and T. Di Mascio. Atti di HCItaly Symposium 2005, September 13 2005, Rome (Italy). Finito di stampare in
Luglio 2005. Studio Grafico Pierpaolo Ceccarelli.
Riviste Internazionali

[2] D. Frigioni, L. Tarantino, T. Di Mascio. On Formal Aspects of Zooming in Geographic Maps. Informatica, Vol. 26, n. 3, pp.
309319, 2002. Slovene Society Informatika.

[3] L. Tarantino and T. Di Mascio. Advances Visual Interfaces: the Focus is on the User. The Knowledge Engineering
Review, Vol. 18:2, pp. 175181, 2003. Cambridge University Press, printed in the United Kingdom.
Conferenze Internazionali

[4] T. Di Mascio and L. Tarantino. Main Features of a CBIR prototype supporting cartoon production. In Proceedings of the
10th Human Computer Interaction International Conference (HCI2003), June 2227 2003, Crete (Greece), Vol. 1, pp. 921
925, 2003. Lawrence Erlbaum Associates.

[5] T. Di Mascio. Assisting User of Semantic Web: the Sewasie Proposal. In Proceedings of the 10th Human Computer
Interaction International Conference (HCI2003), June 2227 2003, Crete (Greece), Adjunct Proceedings, pp. 157159,
2003. Lawrence Erlbaum Associates.

[6] T. Catarci, T. Di Mascio, E. Franconi, G. Santucci and S. Tessaris. An Ontology-based Visual Interface for query
generation support. In Proceedings of International Conference on Ontologies, Databases and Applications of SEmantics
(ODBASE2003), November 0307 2003, Catania (Italy), Adjunct Proceedings, pp. 3234, 2003. Springer-Verlag.

[7] T. Di Mascio, M. Francesconi, D. Frigioni, and L. Tarantino. Tuning a CBIR System for vector images: the interface
support. In Proceedings of the Advanced Visual Interface (AVI2004), May 2528 2004, Gallipoli (Italy), pp. 425428,
2004. ACM Press.

[8] T. Catarci, P. Dongilli, T. Di Mascio, E. Franconi, G. Santucci and S. Tessaris. An Ontology based visual tool for query
formulation support. In Proceedings of European Conference on Artificial Intelligence (ECAI2004), August 2427 2004,
Valencia (Spain), pp. 308312, 2004. IOS Press.

[9] T. Di Mascio, D. Frigioni, and L. Tarantino. A visual environment for tuning a contentbased nvector image retrieval
engine. In Proceedings of the 11th Human Computer Interaction International Conference (HCI2005), June 2227 2005,
Las Vegas (USA), Adjunct Proceedings, 2005. Lawrence Erlbaum Associates.

[10] T. Catarci, T. Di Mascio, P. Dongilli, E. Franconi, G. Santucci, and S. Tessaris. Usability evaluation in the SEWASIE
(SEmantic Webs and AgentS in Integrated Economies) project. In Proceedings of the 11th Human Computer Interaction
International Conference (HCI2005), June 2227 2005, Las Vegas (USA), Vol. 2, 2005. Lawrence Erlbaum Associates.

[11] T. Di Mascio, I. Salvatore, and L. Tarantino. Navigation by zooming in boxes: preliminary evaluation. In
Proceedings of the Advanced Visual Interface (AVI2006), May 2326 2006, Venice (Italy), pp. 262265. ACM
Press.

Tania Di Mascio, Eurac Research

Bozen (Italy), 22 december

Publications
Publications
[12] R. Gennari and T. Di Mascio. The e-LIS Project: an Ontology based Approach to an Electronic Dictionary for
Italian Sign Language. In Proceedings of the second Workshop on Formal Ontology (FOMI2006), December 1415
2006, Trento (Italy). IOS Press.
Conferenze Nazionali

[13] T. Catarci, T. Di Mascio, P. Dongilli, E. Franconi, G. Santucci, and S. Tessaris. An Ontology- Based Query Manager:
Usability Evaluation Proceedings of the 4th Italian Symposium on Human Computer Interaction (HCItaly2005), September
13 2003, Roma (Italy), pp.95100.
Rapporti Tecnici di Progetto

[14] E. Franconi, S. Tessaris, T. Catarci, T. Di Mascio, G. Santucci, and G. Vetere. Specification of the tool for the design,
management and storage of the Semantic Information related to the data stored. Technical Report D6.2 of Sewasie
project (IST FP5 Programme-IST-2001-34825), February 2003.

[15] S. Tessaris, R. Bernardi, T. Catarci, T. Di Mascio, E. Franconi, G. Santucci, and G. Vetere. Specification of the tool for
end-user query management. Technical Report D6.3 of Sewasie project (IST FP5 Programme-IST-2001-34825), March
2003.

[16] T. Di Mascio, T. Catarci, G. Santucci, E. Franconi, S. Tessaris, and G. Vetere. Evaluation of existing tools, description
of the user centered design. Technical Report D6.1 of Sewasie project (IST FP5 Programme-IST-2001-34825), May 2003.

[17] D. Montanari, A. Tavernari, T. Di Mascio, A. Jertila, S. Tessaris, H. Roberts, A. Becks and A. Brunner. First tests and
analysys of test results. Technical Report D9.3 of Sewasie project (IST FP5 Programme-IST-2001-34825), May 2004.

[18] A. Tavernari, D. Montanari, T. Di Mascio, A. Jertila, R. Offermanns, A. Becks and C. Seeling. Second tests and analysys
of test results. Technical Report D9.4 of Sewasie project (IST FP5 Programme-IST-2001-34825), February 2005.

[19] T. Di Mascio, T. Catarci, G. Santucci, D. Montanari, A. Tavernari, A. Jertila, E. Franconi, P. Dongilli, J. Heguiabehere
and A. Becks. Evaluation of the system usability. Technical Report D9.5 of Sewasie project (IST FP5 Programme-IST2001-34825), March 2005.

Lavori sottomessi a Riviste Internazionali

[20] T. Di Mascio, D. Frigioni, and L. Tarantino. A CBIR system for vector images: the tuning interface.

[21] T. Di Mascio, S. Tessaris, P. Dongilli, T. Catarci, G. Santucci and E. Franconi. Browsing and Querying in the
Sewasie Environment.
Lavori sottomessi Conferenze Internazionali

[22] T. Di Mascio, L. Laura, V. Mirabella, and L. Tarantino. The interface of VISTO: a CBIR system for vector
images.

[23] G. Marcelli, T. Di Mascio MDSS Discovery Support System.

Tania Di Mascio, Eurac Research

Bozen (Italy), 22 december

The End

Grazie!

tania@ing.univaq.it

Tania Di Mascio, Eurac Research

Bozen (Italy), 22 december