Align, Disambiguate, and Walk

A Unified Approach for Measuring Semantic Similarity

Semantic Similarity; how similar are a pair of lexical items?

Semantic Similarity

Semantic Similarity

Semantic Similarity
Sentence level

(Tsatsaronis et al., 2010)

(Kauchak and Barzilay, 2006)

(Surdeanu et al., 2011)

(Dagan et al., 2006)

Semantic Similarity

Semantic Similarity
Word level

(Biran et al., 2011)

Locuacious Talkative

Lexical substitution
(McCarthy and Navigli, 2009)

Semantic Similarity

Semantic Similarity
Sense level

Semantic Similarity
Sense level

(Snow et al., 2007)

(Neely et al., 1989)

Exisiting Similarity Measures

Exisiting Similarity Measures

Allison and Dix (1986) Gusfield (1997) Wise (1996) Keselj et al. (2003)

Exisiting Similarity Measures

Allison and Dix (1986) Gusfield (1997) Wise (1996) Keselj et al. (2003)
Salton and McGill (1983) Gabrilovich and Markovitch (2007)

Radinsky et al. (2011)

Ramage et al. (2009) Yeh et al., (2009) Turney (2007) Landauer et al. (1998)

Exisiting Similarity Measures

Allison and Dix (1986) Gusfield (1997) Wise (1996) Keselj et al. (2003)
Salton and McGill (1983) Gabrilovich and Markovitch (2007)

Patwardan (2003) Banerjee and Pederson (2003)

Hirst and St-Onge (1998)

Lin (1998) Jiang and Conrath (1997)

Radinsky et al. (2011)

Ramage et al. (2009) Yeh et al., (2009) Turney (2007) Landauer et al. (1998)

Resnik (1995)
Sussna (1993, 1997) Wu and Palmer (1994)

Leacock and Chodorow (1998)

Exisiting Similarity Measures

Allison and Dix (1986) Gusfield (1997) Wise (1996) Keselj et al. (2003)
Salton and McGill (1983) Gabrilovich and Markovitch (2007)

Patwardan (2003) Banerjee and Pederson (2003)

Hirst and St-Onge (1998)

Lin (1998) Jiang and Conrath (1997)

Radinsky et al. (2011)

Ramage et al. (2009) Yeh et al., (2009) Turney (2007) Landauer et al. (1998)

Resnik (1995)
Sussna (1993, 1997) Wu and Palmer (1994)

Leacock and Chodorow (1998)

Exisiting Similarity Measures

Allison and Dix (1986) Gusfield (1997) Wise (1996) Keselj et al. (2003)
Salton and McGill (1983) Gabrilovich and Markovitch (2007)

Patwardan (2003) Banerjee and Pederson (2003)

Hirst and St-Onge (1998)

Lin (1998) Jiang and Conrath (1997)

Radinsky et al. (2011)

Ramage et al. (2009) Yeh et al., (2009) Turney (2007) Landauer et al. (1998)

Resnik (1995)
Sussna (1993, 1997) Wu and Palmer (1994)

Leacock and Chodorow (1998)

Exisiting Similarity Measures

Allison and Dix (1986) Gusfield (1997) Wise (1996) Keselj et al. (2003)
Salton and McGill (1983) Gabrilovich and Markovitch (2007)

Patwardan (2003) Banerjee and Pederson (2003)

Hirst and St-Onge (1998)

Lin (1998) Jiang and Conrath (1997)

Radinsky et al. (2011)

Ramage et al. (2009) Yeh et al., (2009) Turney (2007) Landauer et al. (1998)

Resnik (1995)
Sussna (1993, 1997) Wu and Palmer (1994)

Leacock and Chodorow (1998)

Exisiting Similarity Measures

Allison and Dix (1986) Gusfield (1997) Wise (1996) Keselj et al. (2003)
Salton and McGill (1983) Gabrilovich and Markovitch (2007)

Patwardan (2003) Banerjee and Pederson (2003)

Hirst and St-Onge (1998)

Lin (1998) Jiang and Conrath (1997)

Radinsky et al. (2011)

Ramage et al. (2009) Yeh et al., (2009) Turney (2007) Landauer et al. (1998)

Resnik (1995)
Sussna (1993, 1997) Wu and Palmer (1994)

Leacock and Chodorow (1998)

Contribution

Contribution

Contribution

Contribution

Advantage 1
Unified representation

Cross-level semantic similarity

Advantage 2

Advantage 3
Sense-level operation

Outline

How Does it work?

How Does it work?

How Does it work?

Semantic Signature

Semantic Signature

Semantic Signature

Semantic Signature

Semantic Signature

Semantic Signature

Semantic Signature

Semantic Signature

Semantic Signature

Semantic Signature

Semantic Signature

Semantic Signature
over all synsets in WordNet

. . .

Semantic Signature
over all synsets in WordNet

. . .

Semantic Signature
over all synsets in WordNet

. . .

Semantic Signature

Semantic Signature

Semantic Signature

Semantic Signature

Semantic Signature

Semantic Signature

Personalized PageRank

Personalized PageRank

Personalized PageRank

Personalized PageRank
1 2 woman n fry v

food n

Personalized PageRank
1 2 woman n fry v

food n

Personalized PageRank
1 2 woman n fry v

food n

2 fry v

Personalized PageRank
1 2 woman n fry v

food n

2 fry v 1

food n

Personalized PageRank
1 2 woman n fry v

food n

2 fry v 1 1 woman n

food n

Personalized PageRank
1 2 woman n fry v

food n

2 fry v 1 1 woman n

food n

1 food n

1 beveragen 2 food n

1 fat n

1 nutriment n 2 dish n 1 cooking n 1 french_fries n 3 cook v

fry v

1 food n

1 beveragen 2 food n

1 fat n

1 nutriment n 2 dish n 1 cooking n 1 french_fries n 3 cook v

fry v

Comparing Semantic Signatures

Comparing Semantic Signatures

Cosine

Weighted Overlap Top-k Jaccard

Comparing Semantic Signatures

Weighted Overlap

Comparing Semantic Signatures

Weighted Overlap

Comparing Semantic Signatures

Weighted Overlap

Comparing Semantic Signatures

Weighted Overlap

Comparing Semantic Signatures

Weighted Overlap

Comparing Semantic Signatures

Top- Jaccard

Comparing Semantic Signatures

Top- Jaccard

= 4

Comparing Semantic Signatures

Top- Jaccard

= 4

Comparing Semantic Signatures

Top- Jaccard

= 4

Alignment-based disambiguation

Alignment-based disambiguation

Why is disambiguation needed?

Why is disambiguation needed?

Why is disambiguation needed?

Why is disambiguation needed?

Why is disambiguation needed?

Alignment-based disambiguation

Alignment-based disambiguation

Alignment-based disambiguation

Alignment-based disambiguation

1 manager n 2 manager n

fire 1 v fire 2 v fire 3 v

. . .

worker n

1 employee n

terminate 1 v terminate 2 v terminate v terminate 4 v

3

1 work n 2 work n 3 work n

boss 1 n boss 2 n
. . .

worker n
. . .

. . .

Alignment-based disambiguation

1 manager n 2 manager n

fire 1 v fire 2 v fire 3 v

. . .

worker n

1 employee n

terminate 1 v terminate 2 v terminate v terminate 4 v

3

1 work n 2 work n 3 work n

boss 1 n boss 2 n
. . .

worker n
. . .

. . .

Alignment-based disambiguation

1 employee n

terminate 1 v terminate 2 v terminate v

3

1 work n 2 work n 3 work n

boss 1 n boss 2 n
. . .

manager n

. . .

manager n

terminate 4 v

Alignment-based disambiguation

1 employee n

terminate 1 v terminate 2 v terminate v

3

1 work n 2 work n 3 work n

boss 1 n boss 2 n
. . .

manager n

. . .

manager n

terminate 4 v

Alignment-based disambiguation
0.5

1 employee n

terminate 1 v terminate 2 v terminate v

3

1 work n 2 work n 3 work n

boss 1 n boss 2 n
. . .

manager n

. . .

manager n

terminate 4 v

Alignment-based disambiguation
0.5

1 employee n

terminate 1 v terminate 2 v terminate v

3

1 work n 2 work n 3 work n

boss 1 n boss 2 n
. . .

manager n

. . .

manager n

terminate 4 v

Tversky (1977) Markman and Gentner (1993)

Alignment-based disambiguation
0.5 0.3
1 employee n

terminate 1 v terminate 2 v terminate v

3

1 work n 2 work n 3 work n

boss 1 n boss 2 n
. . .

manager n

. . .

manager n

terminate 4 v

Tversky (1977) Markman and Gentner (1993)

Alignment-based disambiguation
0.5 0.3 0.3
1 employee n

terminate 1 v terminate 2 v terminate v

3

1 work n 2 work n 3 work n

boss 1 n boss 2 n
. . .

manager n

. . .

manager n

terminate 4 v

Alignment-based disambiguation

1 manager n 2 manager n

fire 1 v fire 2 v fire 3 v fire 4 v

. . .

worker n

1 employee n

terminate 1 v terminate 2 v terminate v terminate 4 v

3

1 work n 2 work n 3 work n

boss 1 n boss 2 n
. . .

worker n
. . .

. . .

Alignment-based disambiguation

1 manager n 2 manager n

fire 1 v fire 2 v fire 3 v fire 4 v

. . .

worker n

1 employee n

terminate 1 v terminate 2 v terminate v terminate 4 v

3

1 work n 2 work n 3 work n

boss 1 n boss 2 n
. . .

worker n
. . .

. . .

Outline

Experiments

Semantic Textual Similarity (SemEval-2012)

Experiments

Semantic Textual Similarity (SemEval-2012)

Synonymy recognition (TOEFL dataset) Correlation-based (RG-65 dataset)

Experiments

Semantic Textual Similarity (SemEval-2012)

Synonymy recognition (TOEFL dataset) Correlation-based (RG-65 dataset)

Coarsening WordNet sense inventory

Experiment 1

Similarity at Sentence level

5 datasets Three evaluation measures

ALL, ALLnrm, and Mean

Experiment 1

Similarity at Sentence level

5 datasets Three evaluation measures

ALL, ALLnrm, and Mean

UKP2 (Br et al., 2012) TLSim and TLSyn (ari et al., 2012)

Experiment 1

Similarity at Sentence level

Features
Main features
Cosine Weighted Overlap Top-k Jaccard

Experiment 1

Similarity at Sentence level

Features
Main features
Cosine Weighted Overlap Top-k Jaccard

String-based features
Longest common substring Longest common subsequence Greedy string tiling Character/word n-grams

Experiment 1

Similarity at Sentence level

STS Results
TLsim TLsyn UKP2 ADW TLsim TLsyn UKP2 ADW TLsim TLsyn UKP2 ADW

Experiment 1

Similarity at Sentence level

STS Results
TLsim TLsyn UKP2 ADW TLsim TLsyn UKP2 ADW TLsim TLsyn UKP2 ADW

Experiments

Semantic Textual Similarity (Semeval-12)

Synonymy recognition (TOEFL dataset) Correlation-based (RG-65 dataset)

Coarsening WordNet sense inventory

Experiment 2

Similarity at Word Level

Experiment 2

Similarity at Word Level

Experiment 2

Similarity at Word Level

Similarity at Word Level

Synonym Recognition
TOEFL dataset (Landauer and Dumais, 1997)
80 multiple choice questions Human test takers: 64.5% only

Similarity at Word Level

Synonym Recognition
Accuracy on TOEFL dataset

75

80

85

90

95

100

Similarity at Word Level

Judgment Correlation Dataset: RG-65 (Rubenstein and Goodenough, 1965)
65 word pairs
judged by 51 human subjects

Scale of 0 4

Similarity at Word Level

Judgment Correlation
Spearman correlation, RG-65 dataset

Similarity at Word Level

Judgment Correlation
Spearman correlation, RG-65 dataset

Experiments

Semantic Textual Similarity (Semeval-12)

Synonymy recognition (TOEFL dataset) Correlation-based (RG-65 dataset)

Coarsening WordNet sense inventory

Experiment 3

Similarity at Sense Level

Coarse-graining WordNet

Experiment 3

Similarity at Sense Level

Coarse-graining WordNet Navigli (2006)

Experiment 3

Similarity at Sense Level

Coarse-graining WordNet Navigli (2006) Snow et al. (2007)

Experiment 3

Similarity at Sense Level

Binary classification: Merged or not-merged

Experiment 3

Similarity at Sense Level

Binary classification: Merged or not-merged

(Hovy et al., 2006)

about 3500 sense pairs (Noun) about 5000 sense pairs (Verb)

(Kilgarriff, 2001)
about 16000 sense pairs (Noun) about 31000 sense pairs (Verb)

Experiment 3

Similarity at Sense Level

Binary classification: Merge or not-merged

if similarity

Experiment 3

Similarity at Sense Level

Binary classification: Merge or not-merged

if similarity

Experiment 3

Similarity at Sense Level

F-score on OntoNotes dataset
Noun
0.22 0.37 0.42 0.42

Verb
0.37 0.45 0.53

0.54
0.52

0.41

Experiment 3

Similarity at Sense Level

F-score on Senseval-2 dataset
Noun Verb

0.37

0.29 0.43 0.49 0.5 0.49

0.42
0.47 0.47 0.44

Conclusions
A unified approach for computing semantic similarity for any pair of lexical items
Experiments with SOA performance
Sense level (sense coarsening) Word level (synonymy recognition and judgment) Sentence level (Semantic Textual Similarity)

Future Direction
Larger sense inventories (e.g., BabelNet)

Future Direction
Larger sense inventories (e.g., BabelNet)
Cross-level semantic similarity

Future Direction
Larger sense inventories (e.g., BabelNet)
Cross-level semantic similarity Create datasets for cross-level similarity
Future Semeval task?

expression

1 thank_you n

2 sensing n

thanks
1

1 heed v

gratitude

listen audio-lingual
1

v
2 ear n

listening

1 auscultation n

listen

1 v

hearing

STS-13
System Dkpro TakeLab ADW (STS-13) ADW (All) GP ADW (All) LR HDL 0.735 0.486 0.621 0.717 0.667 OnWN 0.735 0.633 0.511 0.697 0.735 FNWN 0.341 0.269 0.446 0.411 0.409 SMT 0.323 0.279 0.384 0.272 0.374 mean 0.565 0.434 0.502 0.538 0.565 Rank 6 58 34 20 6