CSI 411 - Compiler - Lecture 2 PDF

Lecture 2 - Introduction to Lexical Analyzer
Adnan Ferdous Ashrafi
September 22, 2020
Adnan Ferdous Ashrafi Lecture 2 - Introduction to Lexical Analyzer September 22, 2020 1 / 22
1 Lexical Analyzer
What is a Lexical Analyzer?
Why do we need a lexical analyzer?
Interaction of Lexical Analyzer and Parser
Tokens, Patterns, and Lexemes
Application of tokens
Lexical Errors
2 Specification of Tokens
Strings and Languages
Operations on Language
3 Regular Expressions (REGEX)

Rules of Regular Expressions
Example of Regular Expressions
4 Conclusion
What is a Lexical Analyzer?
Definition of a Lexical Analyzer

In computer science, lexical analysis, lexing or tokenization is the process of
converting a sequence of characters into a sequence of tokens. A program
that performs lexical analysis may be termed a lexer, tokenizer, or scanner.
A lexer is generally combined with a parser, which together analyze the

syntax of programming languages, web pages, and so forth.
Why do we need a lexical analyzer?
Role of a Lexical Analyzer
As the first phase of a compiler, the main task of the lexical analyzer is to:
read the input characters of the source program
group them into lexemes
produce as output a sequence of tokens for each lexeme in the source
program
During and after the lexical analysis, :
It is common for the lexical analyzer to interact with the symbol table
If the lexical analyzer discovers a lexeme constituting an identifier, it
needs to enter that lexeme into the symbol table
In some cases, information regarding the kind of identifier may be
read from the symbol table by the lexical analyzer to assist it in
determining the proper token it must pass to the parser
The stream of tokens is sent to the parser for syntax analysis
Lexical Analyzers and Parsers
Figure 1: Interactions between the lexical analyzer and the parser
Lexical Analysis - Tokens, Patterns, and Lexemes
When discussing lexical analysis, we use three related but distinct terms:
Token
Pattern
Lexeme
Lexical Analysis - Token
Token
A token is a pair consisting of a token name and an optional attribute
value. The token name is an abstract symbol representing a kind of lexical
unit, e.g., a particular keyword, or a sequence of input characters denoting
an identifier. The token names are the input symbols that the parser
processes. In what follows, we shall generally write the name of a token in
boldface. We will often refer to a token by its token name.
Lexical Analysis - Pattern
Pattern
A pattern is a description of the form that the lexemes of a token may
take. In the case of a keyword as a token, the pattern is just the sequence
of characters that form the keyword. For identifiers and some other
tokens, the pattern is a more complex structure that is matched by many
strings.
Lexical Analysis - Lexeme
Lexeme
A lexeme is a sequence of characters in the source program that matches
the pattern for a token and is identified by the lexical analyzer as an
instance of that token.
Example of tokens
Figure 2: Examples of tokens
Application of tokens
In many programming languages, the following classes cover most or all of

the tokens:
1 One token for each keyword. The pattern for a keyword is the same
as the keyword itself.
2 Tokens for the operators, either individually or in classes such as the
token comparison mentioned in Fig. 2.
3 One token representing all identifiers.
4 One or more tokens representing constants, such as numbers and
literal strings.
5 Tokens for each punctuation symbol, such as left and right
parentheses, comma, and semicolon.
Lexical Errors
It is hard for a lexical analyzer to tell, without the aid of other components,
that there is a source-code error.
The simplest recovery strategy is ”panic mode” recovery. We delete suc-

cessive characters from the remaining input, until the lexical analyzer can
find a well-formed token at the beginning of what input is left. This re-
covery technique may confuse the parser, but in an interactive computing
environment it may be quite adequate.
Other possible error-recovery actions are:
Delete one character from the remaining input.
Insert a missing character into the remaining input.
Replace a character by another character.
Transpose two adjacent characters.
Specification of Tokens - Alphabet
Alphabet
An alphabet is any finite set of symbols. Typical examples of symbols are
letters, digits, and punctuation. The set {0, 1} is the binary alphabet. ASCII
is an important example of an alphabet; it is used in many software systems.
Unicode, which includes approximately 100,000 characters from alphabets
around the world, is another important example of an alphabet.
Specification of Tokens - Strings
String
A string over an alphabet is a finite sequence of symbols drawn from that
alphabet. In language theory, the terms ”sentence” and ”word” are often
used as synonyms for ”string.” The length of a string s , usually written
|s|, is the number of occurrences of symbols in s. For example, banana is
a string of length six. The empty string, denoted , is the string of length
zero.
Terms for Parts of Strings
The following string-related terms are commonly used:
A prefix of string s is any string obtained by removing zero or more
symbols from the end of s. For example, ban, banana, and are
prefixes of banana.
A suffix of string s is any string obtained by removing zero or more
symbols from the beginning of s. For example, nana, banana, and
are suffixes of banana.
A substring of s is obtained by deleting any prefix and any suffix from
s. For instance, banana, nan, and are substrings of banana.
The proper prefixes, suffixes, and substrings of a string s are those,
prefixes, suffixes, and substrings, respectively, of s that are not or
not equal to s itself.
A subsequence of s is any string formed by deleting zero or more not
necessarily consecutive positions of s. For example, baan is a
subsequence of banana.
Specification of Tokens - Language
Language
A language is any countable set of strings over some fixed alphabet. This
definition is very broad. Abstract languages like 0, the empty set, or {}, the
set containing only the empty string, are languages under this definition. So
too are the set of all syntactically well-formed C programs and the set of all
grammatically correct English sentences, although the latter two languages
are difficult to specify exactly. Note that the definition of ”language” does
not require that any meaning be ascribed to the strings in the language.
Operations on Language
In lexical analysis, the most important operations on languages are union,

concatenation, and closure.
Figure 3: Definitions of operations on languages
Regular Expressions (REGEX)
The process of describing identifiers by giving names to sets of letters and

digits and using the language operators union, concatenation, and closure
is called Regular Expressions.
Regular expressions are used for describing all the languages that can be
built from these operators applied to the symbols of some alphabet.
The regular expressions are built recursively out of smaller regular expres-
sions, using the rules described below. Each regular expression r denotes a
language L(r ), which is also defined recursively from the languages denoted
by r ’s subexpressions.
PHere are the rules that define the regular expressions
over some alphabet and the languages that those expressions denote.
Rules of Regular Expressions
Basic Rules
1 is a regular expression, and L () is {} , that is, the language
whose sole member is the empty string.
2 If a is a symbol in C, then a is a regular expression, and L(a)= {a},
that is, the language with one string, of length one, with a in its one
position. Note that by convention, we use italics for symbols, and
boldface for their corresponding regular expression.
Inductive Rules
S
1 (r )|(s) is a regular expression denoting the language L(r ) L(s).
2 (r )(s) is a regular expression denoting the language L(r )L(s) .
3 (r )∗ is a regular expression denoting (L(r ))∗ .
4 (r ) is a regular expression denoting L(r ). This last rule says that we
can add additional pairs of parentheses around expressions without
changing the language they denote.
Example of Regular Expressions
Let C = {a, b}.

1 The regular expression a|b denotes the language {a, b}.
2 (a|b)(a|b) denotes {aa, ab, ba, bb}, the language of all strings of
length two over the alphabet C. Another regular expression for the
same language is aa|ab|ba|bb.
3 a∗ denotes the language consisting of all strings of zero or more a’s,
that is,{, a, aa, aaa, ...}.
4 (a|b)∗ denotes the set of all strings consisting of zero or more
instances of a or b, that is, all strings of a’s and b’s:
{, a, b, aa, ab, ba, bb, aaa, ...}. Another regular expression for the
same language is (a ∗ b∗)∗.
5 a|a ∗ b denotes the language {a, b, ab, aab, aaab, ...}, that is, the
string a and all strings consisting of zero or more a’s and ending in b.
More examples of Regular Expressions
Figure 4: Lex regular expressions
Thank you.
Any Questions?

CSI 411 - Compiler - Lecture 2 PDF

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

CSI 411 - Compiler - Lecture 2 PDF

Uploaded by

Copyright:

Available Formats

Lecture 2 - Introduction to Lexical Analyzer

Adnan Ferdous Ashrafi

September 22, 2020

3 Regular Expressions (REGEX)

Definition of a Lexical Analyzer

A lexer is generally combined with a parser, which together analyze the

Figure 1: Interactions between the lexical analyzer and the parser

Figure 2: Examples of tokens

In many programming languages, the following classes cover most or all of

The simplest recovery strategy is ”panic mode” recovery. We delete suc-

In lexical analysis, the most important operations on languages are union,

Figure 3: Definitions of operations on languages

The process of describing identifiers by giving names to sets of letters and

Let C = {a, b}.

Figure 4: Lex regular expressions

You might also like