Professional Documents
Culture Documents
et
Indexing: Basic Concepts
Indexing is an arrangement of index terms to
permit fast searching and reading memory space
requirement
used to speed up access to desired information
from document collection as per users query such
that
It enhances efficiency in terms of time for retrieval.
Relevant documents are searched and retrieved quick
Index file usually has index terms in a sorted order.
Which list is easier to search?
fox pig zebra hen ant cat dog lion ox
ant cat dog fox hen lion ox pig zebra
…continued
An index file consists of records, called
index entries. Index files are much smaller
than the original file.
Remember Heaps Law: in 1 GB of text
collection the vocabulary has a size of only 5
MB. This size may be further reduced by
Linguistic pre-processing (or text operations).
The usual unit for indexing is the word
Index terms - are used to look up records in a
file.
Major Steps in Index Construction
Source file: Collection of text document
A document can be described by a set of representative
keywords called index terms.
Index Terms Selection: apply text operations or preprocessing
Tokenize: identify words in a document,
Stop words removal
Word stem: reduce words with similar meaning into their
stem/root word
Term relevance weight: Different index terms have varying
relevance when used to describe document contents. This
effect is captured through the assignment of numerical
weights to each index term of a document.
Indexing structure: a set of index terms (vocabulary) are
organized in Index File to easily identify documents in which
each term occurs in.
Basic Indexing Process
Documents to
be indexed. Friends, Romans, countrymen.
Token Tokenizer
stream. Friends Romans countrymen
roman 1 2
So let it be with
Doc 2 Caesar. The noble
Brutus has told you
Caesar was ambitious
Sorting the Vocabulary
Term Doc #
I 1
After all did
enact
1
1
Sequential file
documents have julius 1
Doc
caesar 1
been tokenized, I 1 Term No.
stop words are was
killed
1
1 1 ambition 2
removed, and I 1
2 brutus 1
the 1
normalization capitol 1 3 brutus 2
and stemming are brutus
killed
1
1 4 capitol 1
applied, to me 1
5 caesar 1
generate index so 2
let 2
6 caesar 2
terms it 2
be 2 7 caesar 2
These index with 2
8 enact 1
caesar 2
terms in the 2
9 julius 1
sequential file are noble
brutus
2
2 10 kill 1
sorted in hath
told
2
2 11 kill 1
alphabetical you 2
12 noble 2
order caesar
was
2
2
ambitious 2
Sequential File
To access records search serially;
starting at the first record read and investigate all the
succeeding records until the required record is found or end of
the file is reached.
Its main advantages:
easy to implement;
provides fast access to the next record using lexicographic
order.
Can be searched quickly, using binary search, O(log n)
Its disadvantages:
No weights attached to terms.
Random access is slow: since similar terms are indexed
individually, we need to find all terms that match with the
query.
Inverted file
A word oriented indexing mechanism based on sorted list
of keywords, with each keyword having links to the
documents containing it.
Building and maintaining an inverted index is a relatively low cost
risk. On a text of n words an inverted index can be built in O(n)
time
This list is inverted from a list of terms in location order to a list of
terms in alphabetical order.
Original
Documents •W1:d1,d2,d3
•W2:d2,d4,d7,d9
•…
•Wn :di,…dn
Document IDs
•Inverted Files
Inverted file
Data to be held in the inverted file includes
The vocabulary (List of terms): is the set of all distinct
words (index terms) in the text collection.
Having information about vocabulary (list of terms) speeds
searching for relevant documents
For each term: it contains information related to
Location: all the text locations/positions where the word occurs
frequency of occurrence of terms in a document collection
TF , number of occurrences of term tj in document di
ij
DF , number of documents containing t
j j
TCF, total frequency of t in the corpus n
j
m , maximum frequency of any term in d
i i
n, total number of documents in a collection
….
Inverted file
Having information about the location of each term
within the document helps for:
user interface design: highlight location of search term
proximity based ranking: adjacency and near operators (in
Boolean searching)
Having information about frequency is used for:
calculatingterm weighting (like TF, TF*IDF, …)
optimizing query processing
Inverted File
Documents are organized by the terms/words they contain
Term CF Doc ID TF Location
term 1 3 2 1 66
This is called
19 1 213 an index file.
29 1 45
term 2 4 3 1 94
19 2 7, 212 Text operations
are performed
22 1 56
before building
term 3 1 5 1 43 the index.
term 4 3 11 2 3, 70
34 1 40
Records kept for each term j in the word list contains the
following:
term j
number of documents in which term j occurs (DF )
j
Collection frequency of term j
pointer to inverted (postings) list for term j
Postings File (Inverted List)
For each distinct term in the vocabulary, stores a list of
pointers to the documents that contain that term.
Each element in an inverted list is called a posting, i.e.,
the occurrence of a term in a document
It is stored as a separate inverted list for each column, i.e.,
a list corresponding to each term in the index file.
Each list consists of one or many individual postings
term 1 3 3 Inverted
term 2 3 4 lists
term 3 1 1
term 4 2 3
Example:
Given a collection of documents, they are parsed
to extract words and these are saved with the
Document ID.
So let it be with
Doc 2 Caesar. The noble
Brutus has told you
Caesar was ambitious
Sorting the Vocabulary Term Doc #
Term Doc # ambitious 2
I 1 be 2
After all documents did 1 brutus 1
enact 1 brutus 2
have been tokenized julius 1 capitol 1
the inverted file is caesar
I
1
1
caesar
caesar
1
2
sorted by terms was
killed
1
1
caesar 2
did 1
I 1 enact 1
the 1 has 1
capitol 1 I 1
brutus 1 I 1
killed 1 I 1
me 1 it 2
so 2
julius 1
let 2
killed 1
it 2
killed 1
be 2
let 2
with 2
caesar 2 me 1
the 2 noble 2
noble 2 so 2
brutus 2 the 1
hath 2 the 2
told 2 told 2
you 2 you 2
caesar 2 was 1
was 2 was 2
ambitious 2 with 2
Remove stop words, stemming & compute
frequency
Multiple term Term Doc #
entries in a single ambition 2
Term Doc # TF
document are brutus 1 ambition 2 1
merged and brutus 2 brutus 1 1
frequency capitol 1 brutus 2 1
information added caesar 1 capitol 1 1
Counting number caesar 2
caesar 1 1
of occurrence of caesar 2
terms in the enact 1
caesar 2 2
collections helps to julius 1 enact 1 1
compute TF kill 1 julius 1 1
kill 1 kill 1 2
noble 2 noble 2 1
Vocabulary and postings file
The file is commonly split into a Dictionary and a Posting file
vocabulary posting
Term Doc # TF Term DF CF Doc # TF
ambition 2 1
ambitious 1 1 2 1
brutus 1 1 1 1
brutus 2 1 brutus 2 2
2 1
capitol 1 1 capitol 1 1
1 1
caesar 1 1 caesar 2 3 1 1
caesar 2 2 enact 1 1 2 2
enact 1 1 julius 1 1 1 1
julius 1 1 kill 1 2 1 1
kill 1 2 1 2
noble 1 1
noble 2 1 2 1
Pointers
Searching on Inverted File
Since the whole index file is divided into two,
searching can be done faster by loading vocabulary
list which takes less memory even for large document
collection
Using binary Search the searching takes logarithmic
time
The search is in the vocabulary lists
• This structure is
particularly useful for
any application
requiring prefix
based ("starts with")
pattern matching.
Suffix tree
A suffix tree is an extension of
suffix trie that construct a Trie of
all the proper suffixes of S
The suffix tree is created by •O
compacting unary nodes of the
suffix TRIE.
We store pointers rather
than words in the leaves.
It is also possible to replace
strings in every edge by a pair
(a,b), where a & b are the
beginning and end index of the
string. i.e.
(3,7) for OGOL$
(1,2) for GO
(7,7) for $
Example: Suffix tree
•Let s=abab, a suffix tree of s is a
compressed trie of all suffixes of s=abab$
• We label each leaf with the
starting point of the
•{ corresponding suffix.
• $ •$
• b$ •ab
•b •5
• ab$ •$
• bab$
• •$ •ab$ •4
•ab$
abab$ } •3
•2
•1
Generalized suffix tree
• Given a set of strings S, a generalized suffix tree of S is a
compressed trie of all suffixes of s S
• To make suffixes prefix-free we add a special char, $, at the end of
s. To associate each suffix with a unique string in S add a different
special symbol to each s
• Build a suffix tree for the string s1$s2#, where `$' and `#'
are a special terminator for s1,s2.
• Ex.: Let s1=abab & s2=aab, a generalized suffix tree for s1 & s2 is:
•a •$ •#
•{ •b
• $ # •# •5 •4
• b$ b# •b
• ab$ •ab$ •ab$ •$
ab#
•3
• bab$ aab#
•ab$ • •# •4
• abab$ $ •1
•2
•} •1 •3 •2
Search in suffix tree
Searching for all instances of a substring S in a suffix tree is easy
since any substring of S is the prefix of some suffix.
Pseudo-code for searching in suffix tree:
Start at root
Go down the tree by taking each time the corresponding path
If S correspond to a node then return all leaves in sub-tree
the places where S can be found are given by the pointers in all the
leaves in the sub tree rooted at x.
If S encountered a NIL pointer before reaching the end, then
S is not in the tree
Example:
If S = "GO" we take the GO path and return:
GOOGOL$,GOL$.
If S = "OR" we take the O path and then we hit a NIL pointer so
"OR" is not in the tree.
Exercise
Given the following index terms:
worker, word, world, run & information
construct index file using suffix tree?
Suffix Tree Applications
Suffix Tree can be used to solve a large number of
string problems that occur in:
text-editing,
free-text search,
etc.
Some examples of string problems are given below.
String matching
Longest Common Substring
Longest Repeated Substring
Palindromes
etc..
Complexity Analysis
The suffix tree for a string can be built in O(n2) time.
Searching is very fast: The search time is linear in the
length of string S.
The number of leaves is n+1, where n is the number of
input strings.
Furthermore, in the leaves, we may store either the
strings themselves or pointers to the strings (that is,
integers).
Searching for a substring[1..m], in string[1..n], can be
solved in O(m) time.
Expensive memory-wise
Suffix trees consume a lot of space
How many bytes required to store MISSISSIPI ?