P. 1
NLP Python Intro 1-3

NLP Python Intro 1-3

|Views: 376|Likes:
Published by alebx

More info:

Published by: alebx on Jun 29, 2012
Copyright:Attribution Non-commercial


Read on Scribd mobile: iPhone, iPad and Android.
download as PDF, TXT or read online from Scribd
See more
See less





We can use a frequency distribution to examine the distribution of word lengths in a corpus. For each
word, we find its length, and increment the count for words of this length.

>>> def print_length_dist(text):

fd = nltk.FreqDist(len(token) for token in text if re.match(r’\w+$’, token))


for i in range(1,15):


print "%2d" % int(100*fd.freq(i)),



Now we can call print_length_dist on a text to print the distribution of word lengths. We
see that the most frequent word length for the English sample is 3 characters, while the most frequent
length for the Finnish sample is 5-6 characters.

>>> print_length_dist(nltk.corpus.genesis.words(’english-kjv.txt’))
2 15 30 23 12 6 4 2 1 0 0 0 0 0
>>> print_length_dist(nltk.corpus.genesis.words(’finnish.txt’))
0 12 6 10 17 17 11 9 5 3 2 1 0 0

This is an intriguing area for exploration, and so in Listing 3.4 we look at it on a larger scale using
the Universal Declaration of Human Rights corpus, which has text samples from over 300 languages.
(Note that the names of the files in this corpus include information about character encoding; here we
will use texts in ISO Latin-1.) The output is shown in Figure 3.3 (a color figure in the online version).

Listing 3.4 Cumulative Word Length Distributions for Several Languages

import pylab

def cld(lang):

text = nltk.corpus.udhr.words(lang)
fd = nltk.FreqDist(len(token) for token in text)
ld = [100*fd.freq(i) for i in range(36)]
return [sum(ld[0:i+1]) for i in range(len(ld))]

>>> langs = [’Chickasaw-Latin1’, ’English-Latin1’,

’German_Deutsch-Latin1’, ’Greenlandic_Inuktikut-Latin1’,


’Hungarian_Magyar-Latin1’, ’Ibibio_Efik-Latin1’]
>>> dists = [pylab.plot(cld(l), label=l[:-7], linewidth=2) for l in langs]
>>> pylab.title(’Cumulative Word Length Distributions for Several Languages’)
>>> pylab.legend(loc=’lower right’)
>>> pylab.show()

You're Reading a Free Preview

/*********** DO NOT ALTER ANYTHING BELOW THIS LINE ! ************/ var s_code=s.t();if(s_code)document.write(s_code)//-->