Syntax Analysis

CS2210
Lecture 4

CS2210 Compiler Design 2004/05

Parser
source lexical analyzer token parser get next token symbol table parse tree rest of frontend IR

Parsing = determining whether a string of tokens can be generated by a grammar

CS2210 Compiler Design 2004/05

Grammars

Precise, easy-to understand description of syntax Context-free grammars -> efficient parsers (automatically!) Help in translation and error detection

Eg. Attribute grammars Can add new constructs systematically

CS2210 Compiler Design 2004/05

Easier language evolution

Syntax Errors

Many errors are syntactic or exposed by parsing

eg. Unbalanced () Report errors quickly & accurately Recover quickly (continue parsing after error) Little overhead on parse time
CS2210 Compiler Design 2004/05

Error handling goals:

Error Recovery

Panic mode

Discard tokens until synchronization token found (often ;)

Phrase level

Local correction: replace a token by another and continue Encode commonly expected errors in grammar Find closest input string that is in L(G)

Error productions

Global correction

Too costly in practice

CS2210 Compiler Design 2004/05

Context-free Grammars

Precise and easy way to specify the syntactical structure of a programming language Efficient recognition methods exist Natural specification of many recursive constructs:

expr -> expr + expr | term

CS2210 Compiler Design 2004/05

Context-free Grammar Definition

Terminals T

Symbols which form strings of L(G), G a CFG (= tokens in the scanner), e.g. if, else, id Syntactic variables denoting sets of strings of L(G) Impose hierarchical structure (e.g., precedence rules) Denotes the set of strings of L(G) Rules that determine how strings are formed N -> (N|T) *
CS2210 Compiler Design 2004/05

Nonterminals N

Start symbol S ( N)

Productions P

Example: Expression Grammar

expr -> expr op expr expr -> (expr) expr -> - expr expr -> id op -> + op -> op -> * op -> / op -> ^

Terminals:

{id, +, -, *, /, ^} {expr, op,} Expr

Nonterminals

Start symbol

CS2210 Compiler Design 2004/05

Notational Conventions

Terminals

Nonterminals

a,b,c.. +,-,.. ,.; etc 0..9 expr or <expr>

A, B, C .. S start symbol (if present) or first nonterminal in production list u,v,..

Terminal strings

Grammar symbol strings

, A ->

Productions

CS2210 Compiler Design 2004/05

Shorthands & Derivations

E -> E + E | E * E | (E) | - E | <id>

E => - E E derives -E => derives in 1 step =>* derive in n (0..) steps

CS2210 Compiler Design 2004/05

More Definitions

L(G) language generated by G = set of strings derived from S S =>+ w : w sentence of G (w string of terminals) S =>+ : sentential form of G (string can contain nonterminals) G and G are equivalent : L(G) = L(G) A language generated by a grammar (of the form shown) is called a context-free language
CS2210 Compiler Design 2004/05

Example
G = ({-,*,(,),<id>}, {E}, E, {E -> E + E, E-> E * E , E -> (E) , E-> - E, E -> <id>})
Sentence: -(<id> + <id>) Derivation: E => -E => -(E) => -(E+E)=>-(<id>+E) => -(<id> + <id>)

Leftmost derivation i.e. always replace leftmost nonterminal Rightmost derivation analogously Left /right sentential form

CS2210 Compiler Design 2004/05

Parse Trees
E E => -E => -(E) => -(E+E)=> -(<id>+E) => -(<id> + <id>) ( E <id> Parse tree = graphical representation of a derivation ignoring replacement order E E + ) E <id>

CS2210 Compiler Design 2004/05

Ambiguous Grammars

>=2 different parse trees for some sentence >= 2 leftmost/rightmost derivations Usually want to have unambiguous grammars

E.g. want to just one evaluation order: <id> + <id> * <id> to be parsed as <id> + (<id> * <id>) not (<id>+<id>)*<id> To keep grammars simple accept ambiguity and resolve separately (outside of grammar)

CS2210 Compiler Design 2004/05

Expressive Power

CFGs are more powerful than REs

Can express matching () with CFGs Can express most properties desired for programming languages Identifiers declared before used L = {wcw|w is in (a|b) *} Parameter checking (#formals = #actuals) L ={a nbmcndm|n 1, m 1}

CFGs cannot express:

CS2210 Compiler Design 2004/05

Eliminating Ambiguity (1)

Grammar stmt -> if expr then stmt | if expr then stmt else stmt | other is ambiguous: Sentence: if E1 then if E2 then S1 else S2

stmt => if expr then stmt => if E1 then stmt => if E1 then if expr then stmt else stmt => if E1 then if E2 then stmt else stmt => if E1 then if E2 then S1 else stmt => if E1 then if E2 then S1 else S 2

stmt => if expr then stmt else stmt => if E1 then stmt else stmt => if E1 then if expr then stmt else stmt => if E1 then if E2 then stmt else stmt => if E1 then if E2 then S1 else stmt => if E1 then if E2 then S1 else S 2

Which one do we prefer?

CS2210 Compiler Design 2004/05

Eliminating Ambiguity (2)

Grammar stmt -> if expr then stmt | if expr then stmt else stmt | other is ambiguous: Sentence: if E1 then if E2 then S1 else S2
stmt -> matchted_stmt | unmatched_stmt matched_stmt -> if expr then matched_stmt else matched_stmt | other unmatched_stmt -> if expr then stmt | if expr then matched_stmt else unmatched_stmt

CS2210 Compiler Design 2004/05

Left Recursion
If for grammar G there is a derivation A =>+ A, for some string then G is left recursive Example: S -> Aa | b A -> Ac | Sd |

CS2210 Compiler Design 2004/05

Parsing

= determining whether a string of tokens can be generated by a grammar Two classes based on order in which parse tree is constructed:

Top-down parsing

Start construction at root of parse tree Start at leaves and proceed to root
CS2210 Compiler Design 2004/05

Bottom-up parsing

Recursive Descent Parsing

A top-down method based on recursive procedures (one for each nonterminal typically)

May have to backtrack when wrong production was picked

Predictive parsing = a recursive descent parsing approach that avoids backtracking

More efficient Uses (limited) lookahead to decide what productions to use

CS2210 Compiler Design 2004/05

Predictive Parser

Program with a (parsing) procedure for each nonterminal which

Decides what production to use (based on lookahead in the input) Uses a production by mimicking the right side

CS2210 Compiler Design 2004/05

Predictive Parser Example

type -> simple | ^id | array [simple] of type simple -> integer | char | num dotdot num
procedure match(t:token); begin if lookahead = t then lookahead = nexttoken; else error; end; procedure type; begin if lookahead is in {integer,char,num) then simple else if lookakead = ^ then begin match(^);match(id) end else if lookahead = array then begin match(array);match([); simple; match(]);match(of); type end else error; end

CS2210 Compiler Design 2004/05

Predictive Parsing Obstacles

expr -> expr + term

expr; match(+); term; Infinite recursion (left recursion)

stmt -> if expr then stmt else stmt | if expr then stmt

Common prefix

Cant predict production

Solution

Eliminate left recursion Left factoring

CS2210 Compiler Design 2004/05

Eliminating Left Recursion (1)

Simple case: immediate left recursion: Replace A -> A | with A -> A A -> A |

CS2210 Compiler Design 2004/05

Eliminating Left Recursion (2)

Order the nonterminals A 1 .. A n for i := 1 to n do begin for j := 1 to i-1 do begin replace each production of the form Ai -> Aj by the productions Ai -> 1 | 2 || k where A i -> 1 | 2 | | k are all current A j productions end eliminate immediate left recursion among the A i productions end
CS2210 Compiler Design 2004/05

Example Eliminating Left Recursion

S -> Aa | b A -> Ac | Sd | Order: S,A
for i := 1 to n do begin for j := 1 to i-1 do begin replace each production of the form Ai -> A j by the productions Ai -> 1 | 2 || k where Ai -> 1 | 2 | | k are all current A j productions end eliminate immediate left recursion among the A i productions end

i=2,j=1: Eliminate A->S Replace A->Sd with A->Ac|Aad|bd|

Eliminate immediate left recursion: S->Aa|b A -> bdA|A A ->cA | adA |

CS2210 Compiler Design 2004/05

Left Factoring

Find longest common prefix and turn into new nonterminal

stmt -> if expr then stmt stmt stmt -> else stmt |

CS2210 Compiler Design 2004/05

Transition Diagrams

Create initial and final state For each production A -> X1X2Xn create a path from the initial to the final state, with edges labeled X1, X2, Xn
0 T + 3 6

E:

CS2210 Compiler Design 2004/05

Non-recursive Predictive Parsers

Avoid recursion for efficiency reasons Typically built automatically by tools

Input X Y Z $ a + b $
Predictive Parsing Program

Stack

output M[A,a]gives production A symbol on stack a input symbol (and $)

Parsing Table M

CS2210 Compiler Design 2004/05

Parsing Algorithm

X symbol on top of stack, a current input symbol

Stack contents and remaining input called parser configuration (initially $S on stack and complete input string)
If X=a=$ halt and announce success If X=a $ pop X off stack advance input to next symbol If X is a nonterminal use M[X,a] which contains production X->rhs or error replace X on stack with rhs or call error routine, respectively, e.g. X->UVW replace X with WVU (U on top) output the production (or augment parse tree)
CS2210 Compiler Design 2004/05

1. 2. 3.

10

Construction of Parsing Table Helpers (1)

First() : =set of terminals that begin strings derived from

First(X) = {X} for terminal X If X-> a production add to First(X) For X->Y1Yk place a in First(X) if a in First(Y i) and First(Yj) for j=1i-1, if First(Yj) j=1k add to First(X)

CS2210 Compiler Design 2004/05

Construction of Parsing Table Helpers (2)

Follow(A) := set of terminals a that can appear immediately to the right of A in some sentential form i.e., S =>* Aa for some , (a can include $)

Place $ in Follow(S), S start symbol, $ right end marker If there is a production A-> B put everything in First() except in Follow(B) If there is a production A-> B or A->B where is in First() then everything in Follow(A) is in Follow(B)

CS2210 Compiler Design 2004/05

Construction Algorithm
Input: Grammar G Output: Parsing table M For each production A -> do For each terminal a in FIRST() add A-> to M[A, a] If is in FIRST() add A-> to M[A,b] for each terminal b in FOLLOW(A). ($ counts as a terminal in this step) Make each undefined entry in M to error
CS2210 Compiler Design 2004/05

11

Example
E -> TE E -> +TE | T ->FT T -> *FT | F -> (E) | id FIRST(E) = FIRST(T) = FIRST(F) ={(,id} FIRST(E) = {+, } FIRST(T) = {*, } FOLLOW(E)=FOLLOW(E)={),$} FOLLOW(T)=FOLLOW(T)={+.),$} FOLLOW(F) ={+.*,),$} I + d

* (

E E T T F

CS2210 Compiler Design 2004/05

LL(1)

A grammar whose parsing table has no multiply defined entries is said to be LL(1)

First L = left to right input scanning Second L = leftmost derivation (1) = 1 token lookahead

Not all grammars can be brought to LL(1) form, i.e., there are languages that do not fall into the LL(1) class

CS2210 Compiler Design 2004/05

12