Professional Documents
Culture Documents
GTU # 2170701
Unit – 2
Lexical Analyzer
Symbol Table
Upon receiving a “Get next token” command from parser, the lexical analyzer reads the input
character until it can identify the next token.
Lexical analyzer also stripping out comments and white space in the form of blanks, tabs, and
newline characters from the source program.
= Operator1
+ Operator2
45 Constant1
Lexemes
Lexemes of identifier: total, sum
Lexemes of operator: =, +
Lexemes of constant: 45
Prof. Dixita B Kagathara #2170701 (CD) Unit 2 – Lexical Analyzer 8
Input buffering
There are mainly two techniques for input buffering:
1. Buffer pairs
2. Sentinels
Buffer Pair
The lexical analysis scans the input string from left to right one character at a time.
Buffer divided into two N-character halves, where N is the number of character on one disk
block.
: : : E : : = : : Mi : * : : : C: * : * : 2 : eof : : :
forward forward
lexeme_beginnig
forward
lexeme_beginnig
In buffer pairs we must check, each time we move the forward pointer that we have not moved
off one of the buffers.
Thus, for each character read, we make two tests.
We can combine the buffer-end test with the test for the current character.
We can reduce the two tests to one if we extend each buffer to hold a sentinel character at the
end.
The sentinel is a special character that cannot be part of the source program, and a natural
choice is the character EOF.
𝜖
a
aa
aaa Infinite …..
aaaa
aaaaa…..
a
aa
aaa Infinite …..
aaaa
aaaaa…..
2. 0 or 11 or 111
𝐒𝐭𝐫𝐢𝐧𝐠𝐬: 𝟎, 𝟏𝟏, 𝟏𝟏𝟏 𝐑. 𝐄. = 𝟎 𝟏𝟏 𝟏𝟏𝟏
17. All binary string with at least 3 characters and 3rd character should be zero
𝑺𝒕𝒓𝒊𝒏𝒈𝒔: 𝟎𝟎𝟎, 𝟏𝟎𝟎, 𝟏𝟏𝟎𝟎, 𝟏𝟎𝟎𝟏… 𝑹. 𝑬. = 𝟎 𝟏 𝟎 𝟏 𝟎(𝟎 | 𝟏) ∗
18. Language which consist of exactly two b’s over the set Σ = {𝑎, 𝑏}
∗ ∗ ∗
𝑺𝒕𝒓𝒊𝒏𝒈𝒔: 𝒃𝒃, 𝒃𝒂𝒃, 𝒂𝒂𝒃𝒃, 𝒂𝒃𝒃𝒂… 𝑹. 𝑬. = 𝒂 𝒃 𝒂 𝒃 𝒂
is a state
is a transition
is a start state
is a final state
< =
0 1 2 return (relop,LE)
>
3 return (relop,NE)
=
other
5
4 return (relop,LT)
return (relop,EQ)
>
=
6 7 return (relop,GE)
other
8 return (relop,GT)
start
1
digit . digit E +or - digit other
8
2 3 4 5 6 7
E digit
3
5280
39.37
1.894 E - 4
2.56 E + 7
45 E + 6
96 E 2
Prof. Dixita B Kagathara #2170701 (CD) Unit 2 – Lexical Analyzer 36
Hard coding and automatic generation lexical analyzers
Lexical analysis is about identifying the pattern from the input.
To recognize the pattern, transition diagram is constructed.
It is known as hard coding lexical analyzer.
Example: to represent identifier in ‘C’, the first character must be letter and other characters are
either letter or digits.
To recognize this pattern, hard coding lexical analyzer will work with a transition diagram.
The automatic generation lexical analyzer takes special notation as input.
For example, lex compiler tool will take regular expression as input and finds out the pattern
matching to that regular expression.
Letter or digit
Start Letter
1 2 3
start
start 𝜖 𝑖 N(s) N(t) 𝑓
𝑖 𝑓
start a a b
𝑖 𝑓 1 2 3
𝜖 N(t) 𝜖 𝜖
𝜖 𝜖 𝜖
4 5
b
Prof. Dixita B Kagathara #2170701 (CD) Unit 2 – Lexical Analyzer 44
Regular expression to NFA using Thompson's rule
a*b
𝜖 𝑎 𝜖 𝑏
1 2 3 4 5
b*ab
𝜖
𝜖 𝑏 𝜖 𝑎 𝑏
1 2 3 4 5 6
OPERATION DESCRIPTION
− 𝑐𝑙𝑜𝑠𝑢𝑟𝑒(𝑠) Set of NFA states reachable from NFA state 𝑠 on
– transition alone.
− 𝑐𝑙𝑜𝑠𝑢𝑟𝑒(𝑇) Set of NFA states reachable from some NFA state
𝑠 in 𝑇 on – transition alone.
M𝒐𝒗𝒆 (𝑇, 𝑎) Set of NFA states to which there is a transition on
input symbol 𝑎 from some NFA state 𝑠 in 𝑇.
(a|b)*abb 𝜖
a
2 3
𝜖 𝜖
𝜖 𝜖 a b b
0 1 6 7 8 9 10
𝜖 𝜖
4 5
b
a
2 3
𝜖 𝜖
𝜖 𝜖 a b b
0 1 6 7 8 9 10
𝜖 𝜖
4 5
b
𝜖- Closure(0)= {0, 1, 7, 2, 4}
= {0,1,2,4,7} ---- A
a
2 3 States a b
𝜖 𝜖
A = {0,1,2,4,7} B
𝜖 𝜖 a b b
0 1 6 7 8 9 10 B = {1,2,3,4,6,7,8}
𝜖 𝜖
4 5
b
𝜖
A= {0, 1, 2, 4, 7}
Move(A,a) = {3,8}
𝜖- Closure(Move(A,a)) = {3, 6, 7, 1, 2, 4, 8}
= {1,2,3,4,6,7,8} ---- B
Prof. Dixita B Kagathara #2170701 (CD) Unit 2 – Lexical Analyzer 52
Conversion from NFA to DFA
a
2 3 States a b
𝜖 𝜖
A = {0,1,2,4,7} B C
𝜖 𝜖 a b b
0 1 6 7 8 9 10 B = {1,2,3,4,6,7,8}
C = {1,2,4,5,6,7}
𝜖 𝜖
4 5
b
𝜖
A= {0, 1, 2, 4, 7}
Move(A,b) = {5}
𝜖- Closure(Move(A,b)) = {5, 6, 7, 1, 2, 4}
= {1,2,4,5,6,7} ---- C
Prof. Dixita B Kagathara #2170701 (CD) Unit 2 – Lexical Analyzer 53
Conversion from NFA to DFA
a
2 3 States a b
𝜖 𝜖
A = {0,1,2,4,7} B C
𝜖 𝜖 a b b
0 1 6 7 8 9 10 B = {1,2,3,4,6,7,8} B
C = {1,2,4,5,6,7}
𝜖 𝜖
4 5
b
𝜖
B = {1, 2, 3, 4, 6, 7, 8}
Move(B,a) = {3,8}
𝜖- Closure(Move(B,a)) = {3, 6, 7, 1, 2, 4, 8}
= {1,2,3,4,6,7,8} ---- B
Prof. Dixita B Kagathara #2170701 (CD) Unit 2 – Lexical Analyzer 54
Conversion from NFA to DFA
a
2 3 States a b
𝜖 𝜖
A = {0,1,2,4,7} B C
𝜖 𝜖 a b b
0 1 6 7 8 9 10 B = {1,2,3,4,6,7,8} B D
C = {1,2,4,5,6,7}
𝜖 𝜖
4 5 D = {1,2,4,5,6,7,9}
b
B= {1, 2, 3, 4, 6, 7, 8}
Move(B,b) = {5,9}
𝜖- Closure(Move(B,b)) = {5, 6, 7, 1, 2, 4, 9}
= {1,2,4,5,6,7,9} ---- D
Prof. Dixita B Kagathara #2170701 (CD) Unit 2 – Lexical Analyzer 55
Conversion from NFA to DFA
a
2 3 States a b
𝜖 𝜖
A = {0,1,2,4,7} B C
𝜖 𝜖 a b b
0 1 6 7 8 9 10 B = {1,2,3,4,6,7,8} B D
C = {1,2,4,5,6,7} B
𝜖 𝜖
4 5 D = {1,2,4,5,6,7,9}
b
C= {1, 2, 4, 5, 6 ,7}
Move(C,a) = {3,8}
𝜖- Closure(Move(C,a)) = {3, 6, 7, 1, 2, 4, 8}
= {1,2,3,4,6,7,8} ---- B
Prof. Dixita B Kagathara #2170701 (CD) Unit 2 – Lexical Analyzer 56
Conversion from NFA to DFA
a
2 3 States a b
𝜖 𝜖
A = {0,1,2,4,7} B C
𝜖 𝜖 a b b
0 1 6 7 8 9 10 B = {1,2,3,4,6,7,8} B D
C = {1,2,4,5,6,7} B C
𝜖 𝜖
4 5 D = {1,2,4,5,6,7,9}
b
𝜖
C= {1, 2, 4, 5, 6, 7}
Move(C,b) = {5}
𝜖- Closure(Move(C,b))= {5, 6, 7, 1, 2, 4}
= {1,2,4,5,6,7} ---- C
Prof. Dixita B Kagathara #2170701 (CD) Unit 2 – Lexical Analyzer 57
Conversion from NFA to DFA
a
2 3 States a b
𝜖 𝜖
A = {0,1,2,4,7} B C
𝜖 𝜖 a b b
0 1 6 7 8 9 10 B = {1,2,3,4,6,7,8} B D
C = {1,2,4,5,6,7} B C
𝜖 𝜖
4 5 D = {1,2,4,5,6,7,9} B
b
D= {1, 2, 4, 5, 6, 7, 9}
Move(D,a) = {3,8}
𝜖- Closure(Move(D,a)) = {3, 6, 7, 1, 2, 4, 8}
= {1,2,3,4,6,7,8} ---- B
Prof. Dixita B Kagathara #2170701 (CD) Unit 2 – Lexical Analyzer 58
Conversion from NFA to DFA
a
2 3 States a b
𝜖 𝜖
A = {0,1,2,4,7} B C
𝜖 𝜖 a b b
0 1 6 7 8 9 10 B = {1,2,3,4,6,7,8} B D
C = {1,2,4,5,6,7} B C
𝜖 𝜖
4 5 D = {1,2,4,5,6,7,9} B E
b
E = {1,2,4,5,6,7,10}
𝜖
D= {1, 2, 4, 5, 6, 7, 9}
Move(D,b) = {5,10}
𝜖- Closure(Move(D,b)) = {5, 6, 7, 1, 2, 4, 10}
= {1,2,4,5,6,7,10} ---- E
Prof. Dixita B Kagathara #2170701 (CD) Unit 2 – Lexical Analyzer 59
Conversion from NFA to DFA
a
2 3 States a b
𝜖 𝜖
A = {0,1,2,4,7} B C
𝜖 𝜖 a b b
0 1 6 7 8 9 10 B = {1,2,3,4,6,7,8} B D
C = {1,2,4,5,6,7} B C
𝜖 𝜖
4 5 D = {1,2,4,5,6,7,9} B E
b
E = {1,2,4,5,6,7,10} B
𝜖
E= {1, 2, 4, 5, 6, 7, 10}
Move(E,a) = {3,8}
𝜖- Closure(Move(E,a)) = {3, 6, 7, 1, 2, 4, 8}
= {1,2,3,4,6,7,8} ---- B
Prof. Dixita B Kagathara #2170701 (CD) Unit 2 – Lexical Analyzer 60
Conversion from NFA to DFA
a
2 3 States a b
𝜖 𝜖
A = {0,1,2,4,7} B C
𝜖 𝜖 a b b
0 1 6 7 8 9 10 B = {1,2,3,4,6,7,8} B D
C = {1,2,4,5,6,7} B C
𝜖 𝜖
4 5 D = {1,2,4,5,6,7,9} B E
b
E = {1,2,4,5,6,7,10} B C
𝜖
E= {1, 2, 4, 5, 6, 7, 10}
Move(E,b)= {5}
𝜖- Closure(Move(E,b))= {5,6,7,1,2,4}
= {1,2,4,5,6,7} ---- C
Prof. Dixita B Kagathara #2170701 (CD) Unit 2 – Lexical Analyzer 61
Conversion from NFA to DFA
b
States a b B D
a
A = {0,1,2,4,7} B C a
B = {1,2,3,4,6,7,8} B D
A a a b
C = {1,2,4,5,6,7} B C
D = {1,2,4,5,6,7,9} B E b
C E
E = {1,2,4,5,6,7,10} B C b
Transition Table
b
Note:
• Accepting state in NFA is 10
DFA
• 10 is element of E
• So, E is acceptance state in DFA
States a b
{𝐴, 𝐵, 𝐶, 𝐷, 𝐸}
A B C
B B D
Nonaccepting States Accepting States
{𝐴, 𝐵, 𝐶, 𝐷} {𝐸} C B C
D B E
E B C
{𝐴, 𝐵, 𝐶} {𝐷}
States a b
{𝐴, 𝐶} {𝐵}
A B A
B B D
firstpos(n)
The set of positions that can match the first symbol of a string generated by the subtree at node 𝑛.
lastpos(n)
The set of positions that can match the last symbol of a string generated be the subtree at node 𝑛.
followpos(i)
The set of positions that can follow position 𝑖 in the tree.
2. If n is * node and i is position in lastpos(n), then all position in firstpos(n) are in followpos(i)
𝟒
∗ 𝑎
𝟑
𝑎 𝑏
𝟏 𝟐
{1,2} | {1,2}
𝑖 = 𝑙𝑎𝑠𝑡𝑝𝑜𝑠(𝑐1) = {5}
𝑓𝑖𝑟𝑠𝑡𝑝𝑜𝑠 𝑐2 = 6
𝑎 𝑏 𝑓𝑜𝑙𝑙𝑜𝑤𝑝𝑜𝑠 5 = 6
{1} {1} {2} {2}
𝟏 𝟐
{1,2} | {1,2}
𝑖 = 𝑙𝑎𝑠𝑡𝑝𝑜𝑠(𝑐1) = {4}
𝑓𝑖𝑟𝑠𝑡𝑝𝑜𝑠 𝑐2 = 5
𝑎 𝑏 𝑓𝑜𝑙𝑙𝑜𝑤𝑝𝑜𝑠 4 = 5
{1} {1} {2} {2}
𝟏 𝟐
{1,2} | {1,2}
𝑖 = 𝑙𝑎𝑠𝑡𝑝𝑜𝑠(𝑐1) = {3}
𝑓𝑖𝑟𝑠𝑡𝑝𝑜𝑠 𝑐2 = 4
𝑎 𝑏 𝑓𝑜𝑙𝑙𝑜𝑤𝑝𝑜𝑠 3 = 4
{1} {1} {2} {2}
𝟏 𝟐
{1,2} | {1,2}
𝑖 = 𝑙𝑎𝑠𝑡𝑝𝑜𝑠(𝑐1) = {1,2}
𝑓𝑖𝑟𝑠𝑡𝑝𝑜𝑠 𝑐2 = 3
𝑎 𝑏 𝑓𝑜𝑙𝑙𝑜𝑤𝑝𝑜𝑠 1 = 3
{1} {1} {2} {2}
𝟏 𝟐 𝑓𝑜𝑙𝑙𝑜𝑤𝑝𝑜𝑠 2 = 3
{1,2} | {1,2}
𝑖 = 𝑙𝑎𝑠𝑡𝑝𝑜𝑠(𝑛) = {1,2}
𝑓𝑖𝑟𝑠𝑡𝑝𝑜𝑠 𝑛 = 1,2
𝑎 𝑏 𝑓𝑜𝑙𝑙𝑜𝑤𝑝𝑜𝑠 1 = 1,2
{1} {1} {2} {2}
𝟏 𝟐 𝑓𝑜𝑙𝑙𝑜𝑤𝑝𝑜𝑠 2 = 1,2
=(1,2,3) ----- A
States a b
A={1,2,3} B A
B={1,2,3,4}
State C
δ( (1,2,3,5),a) = followpos(1) U followpos(3) States a b
A={1,2,3} B A
=(1,2,3) U (4) = {1,2,3,4} ----- B
B={1,2,3,4} B C
C={1,2,3,5} B D
δ( (1,2,3,5),b) = followpos(2) U followpos(5) D={1,2,3,6}
b
a States a b
A={1,2,3} B A
a b b B={1,2,3,4} B C
A B C D
C={1,2,3,5} B D
a
a D={1,2,3,6} B A
b
DFA
Prof. Dixita B Kagathara #2170701 (CD) Unit 2 – Lexical Analyzer 82
Conversion from regular expression to DFA
Construct DFA for following regular expression:
1. (c | d)*c#