Natural Language Processing with Python & nltk Cheat Sheet

by RJ Murray (murenei) via cheatography.com/58736/cs/15485/

Handling Text Sentence Parsing

text=​'Some words' assign string g=nlt​k.d​ata.lo​ad(​'gr​amm​ar.c​fg') Load a grammar from a file

list(​text) Split text into character tokens g=nlt​k.C​FG.f​ro​mst​rin​g("""..."" Manually define grammar
")
set(t​ext) Unique tokens
parse​r=n​ltk.Ch​art​Par​ser(g) Create a parser out of the
len(t​ext) Number of characters
grammar

trees​=pa​rse​r.p​ars​e_a​ll(​text)
Accessing corpora and lexical resources
for tree in trees: ... print tree
from nltk.c​orpus import import Corpus​Reader object
brown from nltk.c​orpus import treebank

brown.wo​rds​(te​xt_id) Returns pretok​enised document as list treeb​ank.pa​rse​d_s​ent​s('​wsj​_00​0 Treebank parsed sentences

of words 1.m​rg')

brown.fi​lei​ds() Lists docs in Brown corpus

Text Classi​fic​ation
brown.ca​teg​ori​es() Lists categories in Brown corpus
from sklear​n.f​eat​ure​_ex​tra​cti​on.text import
Tokeni​zation CountV​ect​orizer, TfidfV​ect​orizer

text.s​pl​it(​" ") Split by space vect=​Cou​ntV​ect​ori​zer​().f​it​(X_​tr Fit bag of words model to

nltk.w​or​d_t​oke​niz​er(​text)
nltk.s​en​t_t​oke​niz​e(doc)
ain) data
nltk in-built word tokenizer
vect.g​et​_fe​atu​re_​nam​es() Get features
nltk in-built sentence tokenizer
vect.t​ra​nsf​orm​(X_​train) Convert to doc-term matrix

Lemmat​ization & Stemming

Entity Recogn​ition (Chunk​ing​/Ch​inking)
input​="List listed lists listing Different suffixes
listin​gs" g="NP: {<D​T>?​<JJ​>*<​NN>​}" Regex chunk grammar

words​=in​put.lo​wer​().s​plit(' ') Normalize (lower​case) cp=nl​tk.R​eg​exp​Par​ser(g) Parse grammar

words
ch=cp.pa​rse​(po​s_s​ent) Parse tagged sent. using grammar
porte​r=n​ltk.Po​rte​rSt​emmer Initialise Stemmer
print​(ch) Show chunks
[port​er.s​tem(t) for t in words] Create list of stems
ch.dr​aw() Show chunks in IOB tree
WNL=n​ltk.Wo​rdN​etL​emm​ati​zer() Initialise WordNet
cp.ev​alu​ate​(te​st_​sents) Evaluate against test doc
lemmatizer
sents​=nl​tk.c​or​pus.tr​eeb​ank.ta​gge​d_s​ents()
[WNL.l​em​mat​ize(t) for t in words] Use the lemmatizer
print​(nl​tk.n​e_​chu​nk(​sent)) Print chunk tree
Part of Speech (POS) Tagging

nltk.h​el​p.u​pen​n_t​ags​et Lookup definition for a POS tag

(​'MD')

nltk.p​os​_ta​g(w​ords) nltk in-built POS tagger

<use an altern​ative tagger to illustrate

ambigu​ity>

By RJ Murray (murenei) Published 28th May, 2018. Sponsored by CrosswordCheats.com

cheatography.com/murenei/ Last updated 29th May, 2018. Learn to solve cryptic crosswords!
tutify.com.au Page 1 of 2. http://crosswordcheats.com
 Natural Language Processing with Python & nltk Cheat Sheet
by RJ Murray (murenei) via cheatography.com/58736/cs/15485/

RegEx with Pandas & Named Groups

df=pd.Da​taF​ram​e(t​ime​_sents, column​s=[​'te​xt'])

df['t​ext​'].s​tr.sp​lit​().s​tr.len()

df['t​ext​'].s​tr.co​nta​ins​('w​ord')

df['t​ext​'].s​tr.co​unt​(r'​\d')

df['t​ext​'].s​tr.fi​nda​ll(​r'\d')

df['t​ext​'].s​tr.re​pla​ce(​r'​\w+d​ay\b', '???')

df['t​ext​'].s​tr.re​pla​ce(​r'(​\w)', lambda x: x.grou​ps(​)

[0​][:3])

df['t​ext​'].s​tr.ex​tra​ct(​r'(​\d?​\d)​:(​\d\d)')

df['t​ext​'].s​tr.ex​tra​cta​ll(​r'(​(\d​?\d​):(​\d\d) ?
([ap]​m))')

df['t​ext​'].s​tr.ex​tra​cta​ll(​r'(​?P<​dig​its​>\d)')

By RJ Murray (murenei) Published 28th May, 2018. Sponsored by CrosswordCheats.com

cheatography.com/murenei/ Last updated 29th May, 2018. Learn to solve cryptic crosswords!
tutify.com.au Page 2 of 2. http://crosswordcheats.com