INTRODUCTION TO

ARTIFICIAL INTELLIGENCE
FOR IT & NON-IT PROFESSIONALS
HOW TO BUILD AN NLP PIPELINE
HOW TO BUILD A NLP PIPELINE

• NLP system is often referred to as a pipeline because it usually

involves several stages of processing
• In NLP pipeline, natural language flows in at one end and the
processed output flows out the other
STEPS TO BUILD AN NLP PIPELINE

• Nine steps are used to build an NLP pipeline:

STEP1:SENTENCE SEGMENTATION

• It breaks the paragraph into separate sentences.

• Example: Independence Day is one of the important festivals for every
Pakistani citizen
• produces the result: Independence Day is one of the important
festivals for every Pakistani citizen
STEP2: WORD TOKENIZATION

• is used to break the sentence into separate words or tokens.

• Example: DigiSkills offers Online Training certification courses.
• Output of tokenization: DigiSkills | offers | Online | Training certification
| courses | .
STEP3: STEMMING

• is used to normalize each word into its base form or root form
• Example: celebrates, celebrated and celebrating originated from a
single root word celebrate
• problem with stemming: sometimes it produces a root word which may
not have any meaning
STEP 4: LEMMATIZATION

• Used to group different inflected forms of a word, called Lemma

• refers to the ups/downs of a language, like pitch & tone in a person’s
speech
• lemmatization produces the root word, which has a meaning, and tone
STEP 5: IDENTIFYING STOP WORDS

• In English, there are several words that appear very frequently like
"is", "and", "the", and "a“
• NLP pipelines will flag these words as stop words.
• Stop words might be filtered out before doing any statistical analysis
STEP 6: DEPENDENCY PARSING

• is used to find that how all the words in the sentence are related to
each other
STEP7:PARTS OF SPEECH (POS) TAGS

• include Noun, Verb, Adverb, and Adjective

• Example: Google something on the Internet
STEP 8: NAMED ENTITY RECOGNITION
(NER)
• NER is the process of detecting the named entity such as person
name, movie name, organization name, or location
• Example: Steve Jobs (name) introduced iPhone at the Macworld
Conference (organization) in San Francisco, California (Location).
STEP 9: CHUNKING

• Chunking is used to collect individual pieces of information and

grouping them into bigger chunks, or pieces of sentences.