Fundamental Text Analysis Pipeline and Recognizing Textual Entailment
Sunanda Bansal
sunanda.basnal@mail.concordia.ca
Abstract
In this paper I present the details of my
work for Project #1 and the experiments
performed for the Project #2 for the par-
tial fulfillment of Natural Language Anal-
ysis Course. I developed a fundamental
pipeline for textual analysis, on the basis
of which I analyzed the possible options
to recognize named entities in the Movie
Reviews Corpus provided in the Natural
Language Toolkit (NLTK). Using the Rec-
ognizing Textual Entailment (RTE) Chal-
lenge Corpora provided for RTE tasks in
NLTK, I trained classifiers, using some ad-
ditional features which were based on the
textual pipeline in the first project. I tested
for all the possible combinations of fea-
tures and present here the best perform-
ing models that received 8.64% increase
of performance on average on the RTE test
sets.
Introduction
Natural Language Processing deals with under-
standing and generation of natural language in
both formats - text and speech. Today, in 2018,
there is large amount of textual data present in
digital format which can be great source of ex-
tracting information. Attempting to understand
text is one of the various major studies in natu-
ral language processing. The fundamental steps to
analyzing text in natural language processing are,
generally speaking, tokenization, Part of Speech
tagging, parsing and Named Entities Recognition.
I analyze the fundamental pipeline, the challenges
and possible solutions while dealing with datasets
like Movie Corpus Corpus, which is casefolded.
The objective is to also enhance the performance
of each module of this fundamental pipeline. This
pipeline is further to be used for analyzing the text
and hypothesis and improving the performance
above a baseline for the task of Recognizing Tex-
tual Entailment.
Recognizing Textual Entailment is the task to
determine whether given a text and hypothesis, a
human would generally consider the hypothesis an
entailment of text or not. Using the Recogniz-
ing Textual Entailment (RTE) Challenge Corpora
provided for RTE tasks in NLTK, I try to engi-
neer some features and then compare and analyze
the performance of models and classifiers. I used
the the fundamental textual pipeline developed in
the initial phases of project #1 to facilitate fea-
ture engineering in RTE. I compared the perfor-
mance of Naive Bayes classifier and Support Vec-
tor Machine over a feature space of 12 features, 4
of which form the baseline. the feature space is
discussed in details in the Approach sub-section
of Recognizing Textual Entailment section of this
paper.
1 Fundamental Textual Pipeline
The fundamental steps to process a text so that it
can be analyzed to extract some useful informa-
tion require that it be broken down into small units,
while still preserving the sequence and grouping
of these units within, as a sentence. Then these
units need at least the very basic grammar infor-
mation as to which part of speech role the unit
might be playing in the sentence. Moving after-
wards to the step of looking at its syntactic struc-
ture to identify constituents and dependencies, we
get a starting point to analyze and find patterns to
extract the information we need.
This pipeline was to be tested on the Movie
Reviews Corpus of NLTK during development.
Movie Review Corpus is entirely casefolded and
each token in this corpus is separated by at least a
space, and sometimes a newline character. These
two features of the corpus proved challenging for
the task of improvement of pipeline performance.
1.1 Design
The fundamental textual pipeline that I created,
uses the following sequence of NLTK modules -
Sentence Splitting - Using sentence tokenizer,
nltk.tokenize.sent_tokenize, I split
the whole text into sentences.
Tokenization - Using the word tokenizer,
nltk.tokenize.word_tokenize, I split
each sentence into tokens.
Part of Speech Tagging - nltk.pos_tag
was used to tag each token of each sentence with
part of speech. I also used Stanford POS tagger to
compare the tagger outputs.
Parsing - I used Stanford parser for parsing
each sentence for both constituencies and depen-
dencies.
Named Entities Recognition - I used a sim-
ple grammar and captured the consequent tokens
tagged proper nouns as Name Entities.
NE → (<NNP>|<NNPS>)+
1.2 Approach
I focus my work at improving the performance of
the pipeline at the initial levels since the error in
the initial modules propagated all the way to later
modules that gave bizarre results. There was a dire
need of processing after sentence splitting since
the sentence splitting tool provided by NLTK is
very basic.
Periods : There was poor Sentence across the
acronyms which would consider the period in
acronyms to be a sign of sentence termination. It
was incorrect because the sentence does not split
that way and hence, it affects what the structure
provided by the parser, for that split half of the
sentence. It is possible that the space between ev-
ery token, even the letters of acronym and periods
was the cause of sentence splitter tripping up. I
used the following in the post-processing to sen-
tence splitting step -
Step 1 - Find if a single character [a-z] followed by
a period is found at at the end of the sentence. This
most likely means that the period was not used as
a sentence terminator.
Step 2 - So, while the pattern continues, concate-
nate the strings and add the final string where the
pattern has just stopped continuing to the string as
the trailing part of the sentence.
Parenthesis : The corpora exhibits inconsistent
use of parenthesis rampant through the file with
some issues concerning regarding the sentence
splitting within nested parenthesis, parenthesis
with multiple lines and sentence terminator sym-
bols ( . ? ! ), dangling parenthesis - where
the author started a sentence in a a parenthesis
but forgot to close it and using parenthesis to
state bullet points like a), 3), c.), (i). I
created the following fix in the post-processing of
sentence splitting step-
Step 1 - Identifying only bullet points of the
pattern a), b) ) and split the sentence right
before [0-9a-b] \). Now you shouldn’t have
any parenthesis that are mistaken to be dangling.
Step 2 - Identify Dangling Parenthesis - Count
the number of opening parenthesis and closing
parenthesis. If the count is not equal then there
is an extra opening parenthesis. Append the next
sentence (upto 5 sentences) until you find a dan-
gling closing parenthesis. If a closing parenthesis
not found, then we have a dangling parenthesis. I
split the sentence at this parenthesis.
1.3 Analysis
Many errors were resolved with my fixes and im-
provement on Tagger level and Parser level could
clearly be observed. The project required that our
pipeline be tested on Nokia Reviews Dataset pro-
vided a few days before the deadline for analy-
sis. This data set was smaller, and was casefolded
just like movie reviews data set. Hence, similarly,
most of the proper nouns can not be recognized as
proper nouns and therefore also impacts the recog-
nition of named entities.
This data also presented a new issue where the
reviews are being enumerate and then separate by
two newlines, it hould be considered a separate be-
ginning of the sentence. For this data set, may be
we could first divide the data set into paragraphs,
separate by newlines, and then split those para-
graphs into sentences at the sentence terminator
symbols. This way we should be able split the Re-
view 2 part from the beginning sentence of the re-
view. I analysed the above but did not implement
the fix since this dataset was just meant to test the
performance of pipeline. I observed that -
1. The file has header that is taken as a com-
plete sentence. It does not parse well at all.
It needs to be addressed by removing the se-
quence of non-informative characters that are
there for the sake of design.
2. Data has some gibberish in between, possibly
 because of encodings, which does not parse
well.
3. i is taken to be a List Item Marker or a plu-
ral/common noun, which is wrong, because it
should be tagged Pronoun.
2 Named Entity Recognition
The second part of Project #1 required Named En-
tity Recognition to be implemented on three levels
- after tokenization, after POS Tagging and after
parsing. I found most scope of precise work to be
done at the tokenization level, for my case. Most
of my improvement and fixes from first part fo-
cused on tokenization and sentence splitting, so
this module gave me accurate clues at this level
in comparison to other levels. It required that
POS tagger performance be improved and simi-
larly Parser output be improved before any useful
pattern could be identified at those levels.
I fixed some issues at POS tagging level, which
improved the performance of parser, but I didnt
delve too much into improvement of tagger results
or parser results, given the time limit, for that was
not the focus of this task, Named Entity Recogni-
tion was.
2.1 Approach and Analysis
Grammar: I used a simple grammar and cap-
tured the consequent tokens tagged singular/plural
proper nouns as Name Entities.
NE → (<NNP>|<NNPS>)+
Acronyms : I found acronyms to be a short-
ened form of a named entity, so I identified the
acronyms and tagged them Proper Nouns. This
worked at various levels, part of which was done
for first part of Project #1. Firstly, I fixed the sen-
tence splitting for acronyms, by using the algo-
rithms already discussed in previous section. For
the second part, I combined the acronym as one
token and tagged it as a proper noun (NNP). Since
my grammar simple picks up sequence Proper
Nouns, it picks up the acronyms as a named en-
tity.
Though a lot of named entities have been cap-
tured well, many-a-times my regular expression
captures partial named entities. For example, in
the cases where the names are abbreviated like
E. B. White my regular expression only cap-
tures E. B. as a named entity. Which is true, but
not complete. It would be easier to capture the
whole name, if there were more clues that asso-
ciate them together. I found POS tags to be incon-
sistent in this case, so they could not be used. If
the Names were properly capitalized, I may have
been able to capture the latter part of the name
with POS tag NNP, which takes capitalization into
account, according to my observation. In support
of my observation, my simple rule with Stanford
POS tagger on the test set captures almost all the
named entities, simply because the names are cap-
italized and tagged as NNP. Whereas, in the case-
folded texts in doesnt often give accurate proper
noun tags.
Titles Followed by Person Names : I found the
names of people to be frequently preceded with ti-
tles out of which the three that I found to be fairly
common and easy to sight on a cursory look are
- mr., mrs., dr. . I identify named entities
associated with titles at tokenization level. I cap-
ture one word that follows these titles. With better
tagger performance we might be able to look at the
tags of the words following so that we can capture
named entities spanning 2-3 words or more. But,
my POS tagger didnt give me tags good enough
to make a pattern out of, for the sake of capturing
more than 1 words following the titles.
Quotes and Parenthesis : In the Movie Re-
views Dataset, I often found names of movies or
people in the parenthesis or quotes. I have ex-
tracted the phrases where there are up to 3 words
within the scope of parenthesis and quotes. I
noticed that the names are often no longer than
3 words. If a sequence in the parenthesis is
more than 3 words, it was most likely a com-
ment by the user, as per few reviews I superficially
scanned. There is further scope of work in iden-
tifying names that are longer than 3 words but are
present in a comma separated format within paren-
thesis.
It is difficult to capture the words between sin-
gle quotes , because they are used as apostrophe
in the text. Special cases need to be constructed
in order to avoid capturing the sequences between
apostrophes.
3 Recognizing Textual Entailment
In Project #2, I developed features for the task of
recognizing textual entailment in the RTE dataset
provided in Natural Language Processing Toolkit.
The goal was to conduct different experiments and
analyze the results as well as the errors. In order to
analyze and compare the results a baseline model
was used.
3.1 Baseline
The baseline considered was taken from Chapter 6
of (Bird et al., 2009)
Word Overlap - Returns the number of words
that are common in the text and hypothesis.
Word Hypothesis Extra - Returns the the num-
ber of words that are present in hypothesis but not
in the text.
Named Entity Overlap - Returns the number
of Named Entities that are common in text and hy-
pothesis.
Named Entity Hypothesis Extra - Returns in
the number of Named entities that are present in
the Hypothesis but not in the text.
3.2 Approach
Overlapping words and Named Entities are good
features for baseline because the more common
words do signify to certain extent that the content
of the text and hypothesis is similar. Similarly, the
extra word and named entities in the hypothesis
are a good indication of having more content in
hypothesis which may not be entailed by the text.
Though these features are good for baseline, they
may not be good enough since they don’t look at
other important aspects of the text and hypothesis.
Therefore, extending the intuitions from baseline,
I added the following features:
Overlapping lemmas - Many a times, words
with same lemma appear in the text and hypoth-
esis. This feature calculates such overlappings, if
any.
Overlapping POS Tuples - With the intuition
that word as well as the role of the word in the
sentence is important, I drew overlapping count of
(word,pos_tag) tuples in the text and hypoth-
esis.
Overlapping Dependency Triples - With the
intuition and intention of taking dependencies into
account as a form of syntactic analysis, I consid-
ered the overlapping count of dependency triples
in text and hypothesis, provided by Stanford’s de-
pendency parser .
Ambiguous Words - I observed, in the train-
ing corpus, that there were instances where text
and hypothesis used same word for in completely
different senses, which could be somewhat disam-
biguated with looking at Part of Speech tags. In
this feature I count the number of words, if any,
which are present in both text and hypothesis, but
with different Part of Speech tags.
VERBS AND ADJECTIVES: I was interested
in the role of verb and adjective in the determi-
nation of entailment, on a shallow level, so I de-
veloped 4 features that individually focus on the
following aspects of both.
Similarity : It is not necessary for the hypothe-
sis to use the same exact verb or adjective as in
text. Many a times, a close synonym suffices.
Therefore, in order to use this similarity which
many increase the chances of entailment, I use
path_similarity score, of all senses for all
Verbs and Adjectives respectively as VERB and
ADJ from the Wordnet.
Lemma Overlap : Though Word Lemma
Overlap captures lemma overlap for all words, I
specifically wanted to experiment with the useful-
ness of verb and adjective lemma overlap in pre-
dicting entailments.
In order to develop these features I used used
fundamental textual analysis pipeline and received
the tagged, constituency parsed and dependency
parsed results for each of the text and hypothesis
pair.
3.3 Evaluation
I ran experiments for all possible combinations
of the feature space with three classifiers - Naive
Bayes Classifier and Support Vector Machine. I
observed that a lot of them had 100% precision
and a lot had 100% recall for one or the other test
sets. I removed all such experiments because in
those cases all the test instances were being indis-
criminately predicted to be not an entailment and
all to be an entailment respectively. I have perfor-
mance data of 5681 experimental models out of
which I selected the few that outdid others in one
test case or other for analysis and the results can
be seen in Table 1. The models that were selected
on the basis of their RTE 1, RTE 2 and RTE 3 per-
formances increased the baseline performance by
6-8%.
3.4 Analysis
I analyzed the results for the rest of
the models. I observed using the
most_informative_feature function
of NLTK library that NE Hypothesis Extra feature
is consistently most informative feature in most
of the cases it is a part of the feature set of the
model. A close second are Adjective Similarity,
 Classifier RTE 1 RTE 2 RTE 3 RTE 10 RTE 30 Features
Word Overlap, Word Hypothesis Extra,
Baselines Naive Bayes 0.62 0.61 0.66 0.67 0.5
NE Overlap, NE Hypothesis Extra
Word Overlap, Word Hypothesis Extra,
Best RTE
Naive Bayes 0.63 0.64 0.64 0.83 0.51 NE Hypothesis Extra, Verb Lemma Overlap,
10
Adjective Similarity, Adjective lemma overlap
NE Overlap, Overlapping POS Tags,
Best RTE
Naive Bayes 0.57 0.57 0.57 0.25 0.67 Ambiguous Words, Adjective Similarity,
30
Adjective Lemma Overlap
Word Overlap,Word Hypothesis Extra,
Best RTE 1 SVM 0.68 0.66 0.68 0.71 0.54 Overlapping lemmas, Verb similarity,
Verb Lemma Overlap, Adjective Similarity
Word Overlap, Word Hypothesis Extra,
Overlapping POS Tags, Overlapping Dependency Triples,
Best RTE 2 SVM 0.67 0.67 0.68 0.71 0.55
Overlapping lemmas, Verb similarity,
Verb Lemma Overlap, Adjective Lemma Overlap
Word Overlap, NE Overlap,
NE Hypothesis Extra, Overlapping POS Tags,
Best RTE 3 SVM 0.66 0.66 0.7 0.71 0.54 Overlapping Dependency Triples, Ambiguous Words,
Overlapping lemmas, Verb similarity, Adjective Similarity,
Adjective lemma overlap

Table 1: Best Performing Models

Overlapping POS tags, Verb Similarity and Word

Hypothesis Extra. Overlapping Dependency
Triples didn’t turn out to be as informative as was
expected.

Future Work
There is scope of improvement in the Tagger and
Parser. For NER task, the temporal and numerical
expressions can be identified by using regular ex-
pressions. In RTE task, the parse tree information
to include, or match constituents could be useful.
Using much more data to train on, a neural net-
work based classifier could be trained to predict
entailments. Most of my work in RTE was based
on extending the intuition of overlapping words.
The intuition of words and NEs extra in hypoth-
esis could be extended to expand and experiment
with the feature space.

References
Bird, Steven, Edward Loper and Ewan Klein 2009.
Natural Language Processing with Python, vol-
ume 1. OReilly Media Inc.