ATLANTIS T HINKING M ACHINES VOLUME 3 S ERIES E DITOR : K AI -U WE K UHNBERGER

Atlantis Thinking Machines

Series Editor: Kai-Uwe K hnberger u Institute of Cognitive Science University of Osnabr ck, Germany u (ISSN: 1877-3273)

Aims and scope of the series This series publishes books resulting from theoretical research on and reproductions of general Articial Intelligence (AI). The book series focuses on the establishment of new theories and paradigms in AI. At the same time, the series aims at exploring multiple scientic angles and methodologies, including results from research in cognitive science, neuroscience, theoretical and experimental AI, biology and from innovative interdisciplinary methodologies. All books in this series are co-published with Springer. For more information on this series and our other book series, please visit our website at: www.atlantis-press.com/publications/books

A MSTERDAM PARIS B EIJING c ATLANTIS PRESS

Integration of World Knowledge for Natural Language Understanding

Ekaterina Ovchinnikova
USC ISI 4676 Admiralty Way Marina del Rey, CA 90292 USA

A MSTERDAM PARIS B EIJING

Atlantis Press 8, square des Bouleaux 75019 Paris, France For information on all Atlantis Press publications, visit our website at: www.atlantis-press.com Copyright This book, or any parts thereof, may not be reproduced for commercial purposes in any form or by any means, electronic or mechanical, including photocopying, recording or any information storage and retrieval system known or to be invented, without prior permission from the Publisher.

Atlantis Thinking Machines Volume 1: Enaction, Embodiment, Evolutionary Robotics. Simulation Models for a Post-Cognitivist Science of Mind - Marieke Rohde, Ezequiel A. Di Paolo Volume 2: Real-World Reasoning: Toward Scalable, Uncertain Spatiotemporal, Contextual and Causal Inference - Ben Goertzel, Nil Geisweiller, L cio Coelho, Predrag Jani i , Cassio Pennachin u cc

ISBNs Print: E-Book: ISSN:

978-94-91216-52-7 978-94-91216-53-4 1877-3273

c 2012 ATLANTIS PRESS

Foreword

Inference-based natural language understanding (NLU) was a thriving area of research in the 1970s and 1980s. It resulted in good theoretical work and in interesting small-scale systems. But in the early 1990s it foundered on three difculties: Parsers were not accurate enough to produce predicate-argument relations reliably, so that inference had no place to start. Inference processes were not efcient enough nor accurate enough. There was no large knowledge base designed for NLU applications. The rst of these difculties has been overcome by progress in statistical parsing. The second problem is one that many people, including Ekaterina Ovchinnikova, are working on now. The research described in this volume addresses the third difculty, and indeed shows considerable promise in overcoming it. For this reason, I believe Dr. Ovchinnikovas work has a real potential to reignite interest in inference-based NLU in the computational linguistics community. A key notion in her work is that there already exists sufcient world knowledge in a variety of resources, at a level of precision that enables their translation into formal logic. To my mind, the most important of these are WordNet and FrameNet, especially the latter, and she describes the kind of information one can get out of these resources. She exploits in particular the hierarchical information and the glosses in WordNet, generating 600,000 axioms. She also describes how one can utilize FrameNet to generate about 50,000 axioms representing relations between words and frames, and about 5000 axioms representing frameframe relations. Her analysis of FrameNet is quite thorough, and I found this part of her work inspiring. She also critically discusses foundational ontologies such as DOLCE, SUMO. and OpenCyc, and domain-specic ontologies of the sort being constructed for the Semantic Web.
v

vi

Integration of World Knowledge for Natural Language Understanding

She examines the problems raised by semi-formal ontologies, like YAGO and ConceptNet, which have been gleaned from text or Netizens and which may be more difcult to translate into formal logic. She also shows how to use distributional data for a default mode of processing when the required knowledge is not available. Her use of knowledge from a variety of resources, combined into a single system, leads to the very hard problem of ensuring consistency in such a knowledge base. She engages in a very close study of the kinds of conceptual inconsistencies that occur in FrameNet and in Description Logic ontologies. She then provides algorithms for nding and resolving inconsistencies in these resources. I found this part of her work especially impressive. She examines three forms of inference standard deduction, weighted abduction, and reasoning in description logics, explicating the strengths and weaknesses of each. Finally she evaluates her work on inference-based NLU by applying her reasoning engines to the Recognizing Textual Entailment problem. She uses the RTE-2 test set and shows that her approach, with no special tuning to the RTE task, achieves state-of-the-art performance. She also evaluates her approach, with similarly excellent results, on the Semantic Role Labeling task and on paraphrasing noun-noun dependencies, both of which fall out as a by-product of weighted abduction. So the research described here is very exciting indeed. It is a solid achievement on its own and it promises to open doors to much greater progress in automatic natural language understanding in the very near future. Jerry R. Hobbs Information Sciences Institute University of Southern California Marina del Rey, California

Acknowledgments

The research presented in this book is based on my PhD thesis, that would not have been possible without the help of many people. In the rst place I would like to thank my thesis advisor Kai-Uwe K hnberger who has scientically and organizationally supported all my u research adventures giving me freedom to try whatever I thought was interesting. I am indebted to Jerry Hobbs who has invited me to visit the Information Sciences Institute where I have spent the most productive six months of my dissertation work. Jerry has introduced me to the exciting eld of abductive reasoning and encouraged me to combine this approach with my research efforts, which turned to be highly successful. I owe my deepest gratitude to Frank Richter who has supported me from the very beginning of my research career. Whenever I needed scientic advice or organizational support, Frank was always there to help. My gratitude especially goes to the ISI colleagues. I particularly beneted from discussions with Eduard Hovy. Thanks to Rutu Mulkar-Mehta who has developed and supported the Mini-TACITUS system, I managed to implement the extensions to the system that many of my research results are based upon. I very much thank Niloofar Montazeri who shared with me the tedious work on recognizing textual entailment challenge. I thank Nicola Guarino for giving me an opportunity to spend a couple of weeks at the Laboratory of Applied Ontology. Many thanks to the LOA colleagues Laure Vieu, Alessandro Oltramari, and Stefano Borgo for a fruitful collaboration on the topic of conceptual consistency. The following gratitudes go to the researchers from all around the world who have directly contributed to this work. I very much thank Tonio Wandmacher for being my guide into the world of distributional semantics and for constantly challenging my trust in inference-based approaches. I am grateful to Johan Bos, the developer of the Boxer and Nutcracker systems, who helped me to organize experiments involving these systems. I would like to thank
vii

viii

Integration of World Knowledge for Natural Language Understanding

Anselmo Pe as for collaborating with me on the issue of paraphrasing noun dependencies. n I thank Michael McCord for making the ESG semantic parser available for my experiments. I thank Helmar Gust who agreed to write a review of my thesis. Concerning the nancial side, I would like to thank the German Academic Exchange service (DAAD) for according me a three year graduate scholarship. I also thank the Doctorate Programme at the University of Osnabr ck for supporting my conference and scientic u trips nancially. I would like to thank Johannes Dellert, Ilya Oparin, Ulf Krumnack, Konstantin Todorov, and Sascha Alexeyenko for valuable comments, hints, and discussions. Special thanks to Ilya for keeping asking me when my thesis was going to be nished. I am grateful to Irina V. Azarova who gave me a feeling of what computational linguistics really is. I express my particular gratitude to my parents Andrey and Elena for their continued support and encouragement, which I was always able to count on. Finally, I sincerely thank my husband Fedor who has greatly contributed to the realization of this book. Thank you for valuable discussions, introduction into statistical data processing, manifold technical and software support, cluster programming necessary for large-scale experiments, and all other things, which cannot be expressed by words. E. O., November 2011, Los Angeles

Contents

Foreword Acknowledgments List of Figures List of Tables List of Algorithms 1. Preliminaries 1.1 1.2 1.3 2. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Objectives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . How to Read This Book . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

v vii xiii xv xvii 1 1 4 13 15 15 18 19 26 30 31 33 36 39 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40 46 53 56 59 64 65 65 66 67 68

Natural Language Understanding and World Knowledge 2.1 2.2 What is Natural Language Understanding? . . . . . . . . . . . . . . . . . . . . . . Representation of Meaning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.2.1 Meaning Representation in Linguistic Theories . . . . . . . . . . . . . . . 2.2.2 Linguistic Meaning in Articial Intelligence . . . . . . . . . . . . . . . . Shared Word Knowledge for Natural Language Understanding . . . . . . . . . . . 2.3.1 Linguistic vs. World Knowledge . . . . . . . . . . . . . . . . . . . . . . 2.3.2 Natural Language Phenomena Requiring Word Knowledge to be Resolved Concluding Remarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

2.3

2.4

3. Sources of World Knowledge 3.1 Lexical-semantic Resources . . . . . . . . . . . . . . . . . . . . . . 3.1.1 Hand-crafted Electronic Dictionaries . . . . . . . . . . . . . 3.1.2 Automatically Generated Lexical-semantic Databases . . . . Ontologies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.2.1 Foundational Ontologies . . . . . . . . . . . . . . . . . . . 3.2.2 Domain-specic Ontologies . . . . . . . . . . . . . . . . . . Mixed Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.3.1 Ontologies Learned from Text . . . . . . . . . . . . . . . . . 3.3.2 Ontologies Learned from Structured Sources: YAGO . . . . 3.3.3 Ontologies Generated Using Community Efforts: ConceptNet Concluding Remarks . . . . . . . . . . . . . . . . . . . . . . . . . .

3.2

3.3

3.4

ix

Integration of World Knowledge for Natural Language Understanding

4.

Reasoning for Natural Language Understanding 4.1 Semantic Parsers . . . . . . . . . . . . . . . . 4.1.1 English Slot Grammar . . . . . . . . . 4.1.2 Boxer . . . . . . . . . . . . . . . . . Deduction for Natural Language Understanding Abduction for Natural Language Understanding Reasoning with Description Logics . . . . . . . Concluding Remarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

73 74 74 76 77 81 86 90 93 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95 95 95 96 96 101 114 116 118 121 123 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 124 126 128 131 133 134 136 139 141 145 147 151 153 155 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 155 162 166 171 173 177 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 179 179 182 185 187

4.2 4.3 4.4 4.5 5.

Knowledge Base Construction 5.1 Preliminaries . . . . . . . . . . . . . . . . . . . . 5.1.1 Notation . . . . . . . . . . . . . . . . . . 5.1.2 Axiom Weights . . . . . . . . . . . . . . Axioms derived from Lexical-Semantic Resources 5.2.1 Axioms derived from WordNet . . . . . . 5.2.2 Axioms derived from FrameNet . . . . . . 5.2.3 Axioms derived from Proposition Store . . Ontology . . . . . . . . . . . . . . . . . . . . . . Similarity Space . . . . . . . . . . . . . . . . . . . Concluding Remarks . . . . . . . . . . . . . . . .

5.2

5.3 5.4 5.5 6.

Ensuring Consistency 6.1 Conceptual Inconsistency of Frame Relations . . . . . . . . . 6.1.1 Ontological Status of Frames . . . . . . . . . . . . . 6.1.2 Constraints on Frame Relations . . . . . . . . . . . . 6.1.3 Reasoning-Related Conceptual Problems in FrameNet 6.1.4 Case Study . . . . . . . . . . . . . . . . . . . . . . . Logical Inconsistency in Ontologies . . . . . . . . . . . . . . 6.2.1 Resolving Logical Inconsistencies . . . . . . . . . . 6.2.2 Tracing Clashes . . . . . . . . . . . . . . . . . . . . 6.2.3 Resolving Overgeneralization . . . . . . . . . . . . . 6.2.4 Root and Derived Concepts . . . . . . . . . . . . . . 6.2.5 Rewriting Algorithm . . . . . . . . . . . . . . . . . . 6.2.6 Prototypical Implementation . . . . . . . . . . . . . Concluding Remarks . . . . . . . . . . . . . . . . . . . . . .

6.2

6.3 7.

Abductive Reasoning with the Integrative Knowledge Base 7.1 7.2 7.3 7.4 7.5 Adapting Mini-TACITUS to a Large Knowledge Base Rening Abductive Reasoning Procedure for NLU . Reasoning with Ontologies . . . . . . . . . . . . . . Reasoning with Similarity Space . . . . . . . . . . . Concluding Remarks . . . . . . . . . . . . . . . . . . . . . .

8.

Evaluation 8.1 Natural Language Understanding Tasks . . . 8.1.1 Recognizing Textual Entailment . . 8.1.2 Semantic Role Labeling . . . . . . . 8.1.3 Paraphrasing of Noun Dependencies Experiments with Boxer and Nutcracker . . .

8.2

Contents

xi

8.3

8.4

8.2.1 Recognizing Textual Entailment . . 8.2.2 Semantic Role Labeling . . . . . . . Experiments with Mini-TACITUS . . . . . . 8.3.1 Recognizing Textual Entailment . . 8.3.2 Semantic Role Labeling . . . . . . . 8.3.3 Paraphrasing of Noun Dependencies 8.3.4 Domain Text Interpretation . . . . . Concluding Remarks . . . . . . . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

189 194 195 196 201 202 207 212 215 221 225 241

9. Conclusion Appendix A Bibliography Index

List of Figures

1.1 2.1 2.2 2.3 3.1 4.1 4.2 4.3 4.4

Inference-based NLU pipeline. . . . . . . . . . . . . . . . . . . . . . . . . . .

Human natural language understanding. . . . . . . . . . . . . . . . . . . . . . 17 Computational natural language understanding. . . . . . . . . . . . . . . . . . 18 Meaning of John gave Mary a book as a conceptual dependency graph. . . . . 28 Illustration of the denition for the term ontology given by Guarino (1998). . 57 Inference-based NLP pipeline . . . . . . . . . . . . . . . . . . . . . . . . . . 73 The LF produced by ESG for the sentence If Mary gives John a book, then he reads it. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75 The LF produced by ESG for the sentence The book is given to John by Mary. . 75 The DRS produced by Boxer for the sentence If Mary gives John a book, then he reads it. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76

5.1 5.2 6.1 6.2

Modules of the proposed integrative knowledge base. . . . . . . . . . . . . . . 94 Schematic overview on the generation of an LSA-based semantic space. . . . . 120 DOLCE basic categories. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 127 Medical cluster: frame relations from FrameNet (top) enriched and cleaned up (bottom). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 133

7.1 8.1 8.2

Abductive reasoning pipeline. . . . . . . . . . . . . . . . . . . . . . . . . . . 175 NLU pipeline based on Boxer and Nutcracker. . . . . . . . . . . . . . . . . . . 190 NLU pipeline based on ESG and Mini-TACITUS. . . . . . . . . . . . . . . . . 196

xiii

List of Tables

3.1 3.2 3.3 4.1 4.2 5.1 5.2 5.3 5.4 5.5 6.1 8.1 8.2 8.3 8.4 8.5 8.6 8.7 8.8 8.9

English WordNet 1.3 statistics . . . . . . . . . . . . . . . . . . . . . . . . . . 48 English FrameNet 1.5 statistics. . . . . . . . . . . . . . . . . . . . . . . . . . 52 VerbOcean statistics. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55 Syntax and semantics of A L C N DL. . . . . . . . . . . . . . . . . . . . . . 87 Tableau expansion rules for A L C N satisability. . . . . . . . . . . . . . . . 88 Direction of entailment for WordNet relations. . . . . . . . . . . . . . . . . . . 99 Axioms derived from WordNet morphosemantic relations. . . . . . . . . . . . 100 Statistics for axioms extracted from WordNet. . . . . . . . . . . . . . . . . . . 101 Results of FrameNet frame clustering. . . . . . . . . . . . . . . . . . . . . . . 113 Statistics for axioms extracted from FrameNet. . . . . . . . . . . . . . . . . . 113 Tableau expansion rules for tracing clashes in A L C N terminology. . . . . . 140 Examples of text-hypothesis pairs from the RTE-2 development set. . . . . . . 181 Evaluation of the Shalmaneser system towards FATE. . . . . . . . . . . . . . . 184 Results of recognizing textual entailment by the Nutcracker system for the 39 RTE-2 pairs annotated with medical frames in FATE. . . . . . . . . . . . . . 194 Evaluation of the Boxer system performing SRL towards FATE. . . . . . . . . 194 Evaluation of the Mini-TACITUS system performing RTE using RTE-2 test dataset. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 201 Evaluation of the Mini-TACITUS system performing SRL towards FATE. . . . 202 Paraphrasing of noun dependencies in the RTE-2 development set. . . . . . . . 205 Paraphrasing of noun dependencies in the RTE-2 test set. . . . . . . . . . . . . 205 Paraphrasing axioms used to resolve RTE-2 development set pairs. . . . . . . . 205

8.10 Paraphrasing axioms used to resolve RTE-2 test set pairs. . . . . . . . . . . . . 208
xv

List of Algorithms

6.1 7.1 7.2

Repair terminology T containing unsatisable concept C. . . . . . . . . . . . 148 Mini-TACITUS reasoning algorithm: interaction of the time and depth parameters. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 158 Algorithm for selecting axioms, which can be evoked given a logical form. . . . 159

xvii