Compiler Construction

CS-4207

Instructor Name: Atif Ishaq

Lecture 6
 Today’s Lecture

 Recognition of Token

 Regular Expression and FSM

 Transition Diagram Construction

2
 Recognition of Token : Transition Diagram

 A language defined by a grammar is a (possibly infinite) set of strings

 An automation is a device that determines, by reading a string (word) one

character at a time, whether the string belongs to a special language

 A finite state automata (FSA, NFA) is an automaton that recognizes regular

languages (regular expressions)

 Simplest automaton : memory is an element of a finite set

3
 Recognition of Token : Transition Diagram

 Graphically a Finite State Automata are represented by

 A set of labeled states, represented as nodes in a digraph

 Directed edges labelled with a character are drawn between states

 One or more states designated as terminal (accepting)

 One or more state designated as initial

 On reading character a ∈ ∑ , automaton may move from state S1 to state S2 if

there exists an a-labled edge connecting S1 to S2.

 A string belongs to the language if, while reading the string, the automaton

may move from an initial state to an accepting state.

4
 Recognition of Token : Transition Diagram

Following diagram is an NFA which recognizes the language of all string over ∑ :

{a , b} which have an even number of a’s and b’s

For even a’s and b’s

5
Recognition of Token : Transition Diagram

6
Recognition of Token : Transition Diagram

7
Recognition of Token : Transition Diagram

8
Recognition of Token : Transition Diagram

9
Recognition of Tokens : Transition Diagram

relop  < | > | <= | >= | <> | =

id  letter (letter | digit )*

10
Recognition of Tokens : Transition Diagram

A transition diagram for unsigned digits

A transition diagram for white spaces

11
 What else a Lexical Analyze Do?

 All keyword / reserve word are matched as ids

 After the match, symbol table or special keyword table is consulted
 Keywords table contains string version of all keywords along with the
associated token value
 When a match is found the token is returned along with its symbolic
value, i.e, “then”,16
 If match is not found then it is assumed that an id has been discovered

if 15
then 16
begin 17
... ...
12
 Transition Diagram : Code
token nexttoken()
{ while (1) {
switch (state) {
case 0: c = nextchar();
if (c==blank || c==tab || c==newline) { Decides the
state = 0;
lexeme_beginning++; next start state
}
else if (c==‘<’) state = 1; to check
else if (c==‘=’) state = 5;
else if (c==‘>’) state = 6;
else state = fail();
break; int fail()
case 1: { forward = token_beginning;
… swith (start) {
case 9: c = nextchar(); case 0: start = 9; break;
if (isletter(c)) state = 10; case 9: start = 12; break;
else state = fail(); case 12: start = 20; break;
break; case 20: start = 25; break;
case 10: c = nextchar(); case 25: recover(); break;
if (isletter(c)) state = 10; default: /* error */
else if (isdigit(c)) state = 10; }
else state = 11; return start;
break; }
13
…
Transition Diagram : Code

14
 Lecture Outcome

 Significance of context free grammar in compiler construction

 How to resolve associativity and precedence issues in arithmetic

expressions

 Focusing on unambiguous grammar for parsing

15
 Lecture Outcome

 Token Recognition

 Transition Diagram Construction

16
17