mI ;~ if.:B ~'~ !

fiG fi

.tEa :£~I I~ 1:=". -H: .. ' , ~ ~ :* ~ H ... "111=1 iC:;~J ~ ~ ~J IX '1" -"LJb :IS' ~ 'f/J

tilt Jjj(JI ft~~ IJt

Cumptlers:

'Principles, Techniques, and Tools

.iI.1I HlI!:SI 'Six II)

Compilers: Principles, Techniques, and Tools

Alfred V. Abo RaviSethi Jeffrey D. Ullman

Pearson Education lli Jti_ m

~#.t •• ~Nffl.~$~tt*ftft~U f,fi ilmUI ti7!tl:i Ill(~:)lJti}

• ~ AlfreiJ V. Aho Ravi Sethi J~ffrey D. Ullman

~ifMm *' !J,.'F

• A~ib~~allit±HHl!i~fi ~I:Jtm*;tl3:~ !!«4fm 14 4}

~r.lIii 100061 It!. "f E8 n 31 S @ pptph , t.>QIJI • en

JI.-j bJ:. hU(J : 1/ www . pptph . (:j,IIll • (;0

i~t-~MJ. 010·67129212 010--61129211( -fUn ~tJji;IY..~rn;t ilit1ii IlR 0 PJikllfF

JLl'f, M lIB ~ m $Iii~ r ru IN

fi$ '* 16 .~,rn .itJi( ~ fj 1Ji!£~

• Jf*~720~980 1/16 (11* ;50.75

'¥:I: ; 1 009 -f¥ qlt!l:1-4000M

2002 ¥ 2 JJ ~ I AA

2002 ~ 2}J jt~~ I /X(jJIjfIJ

itF~trJFijiHc OOf:OI-2001-4831 ~ ISBN 7 -llS-0991o-2/TP' 2649

Ii 1ft :63.00 :7t

* ~:tn1i ro !lilt. itt. I iJl1i **111 ~ * if: (010 }67 ]29223

m-=KUfi § (ctP) ftM

ffUitJj,lI~* ~IJtI (~) 1Wnl~: (Aho,A.V.), (~) ~IDj (Sethi, RJ, (~) IE )j\~ (Ullman. lD.) ~. -:i~~: )J.~;;'*E§llif#i.;f±, 2002.2

~*.~~.~~.AM~~tt*~*~rt

ISBN 7-115-09916-2

I .~ ... II .[l~iiJ ... ® •... ®I[ ... III.tjif&1F-m1Fl9:it-~~$:"~-~;f1- ~:t N.TP314

~mr&*oo~it CIP ~~~* (2001) ~089601 ~ mifiJlfIM

English Reprint Edition Copyright e 2001 by PEARSON EDUCATION NORTH ASIA LIMITED and PEOPLE'S POSTS & TELECOMMUNICATIONS PUBUSIDNG HOUSE.

Compilers: Principles. Techniques, and Tools

By Alfred V. Abo, Ravi Sethi, Jeffrey D. Ullman Copyright © 1986

All Rights Reserved,

Published by arrangement with Addison Wesley; Pearson Education,lnc.

This edition is authorized for sale only in People's Republic of China (excluding the Special

Administrative Region of Hong Kong and Macau). ~~ViC lI*: Ol.21M1-4831lt

~~~w3~~~~~,*~.~~~~Kw~a~~*~n w~tt~~~~~oom~-~M8Lffi#~~~~~~~~§~ m.o*~~12.J .-.km7~~.~&*M~; .=.~ ~1-~~oo~~~~~~~m~~~~~~watI~~ffl*~ ~-~~*a~*~.;.=.~~Yw~~~,£m~~~,~ ~§~m~am~~~IA,~.~~~*}~~ffl~~*~~; m~.w~~~T~~~~~k*,A~~~I~m~~~r~. it~U{f7t;tJT~J£~~1!l!ffl B9 LR .t!; m1iltJrm 1 *¥!$IJ~liw ~~±~mm,*~~~B$#~m*.*~~~~~fi~~~A ~~~7~~.~*~~~~~~m~,~~~~~~~~~~~ ~*fi7w~~M~;~~.M~7~~~fflN~~ff~~m~~ fit~ll tJ!.; ~ )\.f:JA ~ fB]* -:; l¥Ji"1~3T~, iJt R~ T ~f1~ ~~~:Wl W ~fiw~~~ftW; ~n.~~J~~ft~~~~, §*~*~ on-the-fly f-~lij:£J£1.f¥!, ~~i5~~~ftliI!~1ftit1irt;~ Peephole

~~~~~~~B;.+.~~~~~OO~~o~7*~.~.~ mn~~W~~~,~fl~r~~~tr~~tt~~~*~~;~+ -.M~7~~~.~mrlfi~m~~~~~-~~~~B;.+ =.~lli-~~ffl*~~~~*~~~~~~~MOO~~ffl~o

*~~~~~~~.m~~*~~~~~~~~~~~~~, ~~~~$*.m~~ffR~A~~~o

. .... ....- ... _'.......--._- _'.- _, .....

~l~,ft.$~~T(*~"+~~~~.~ •• ~.~n •• ~~~~ ~~»e ~~~~~m~,~hli"M~.m •• ~a"~~.~~Mt ~m.~ m."h~m~T, ~~7fr~oo~~mfit~~7~*mA~~¥~e~~~ M~~~~.~~~*~~~~mt ~M~~~~~~I~m*~T~~~¥~ ~8;, -®tt*tJ*J~,*1S I ~~gjlJi~iTa91.t~ § ~tilTe<Jf&~it-~JlOiI;fifiJt ~flJ!" ~Jit~XftfIlJHdHiiJll, ~:1m~9Hi9~~;f1tf.J'J-I:i1I:I ~() ~mr t ~1~tfJ~£ ~m.M~~tt*~~_M~~~*~*~M~.n~~.~.~(~~)@. B91.W~, lmj1~A~1f._ ~*~ijE, 51J!I~lJ'Ht~tt;tt ~ JHi.i}I*fd;fatJ~!Jt nc:g I 1m ~~~I mfj:;ft(rti[ it I ~i! ~ 1J£tU:t~Htm;fn.r- ()

~ lJ\ Sf ~ ~.IF;ff 1977 ~ ~ 19HJD m tB: (I ~ ij I j£j~ m f;Jf t ~ 'll&H~ t'itlrp~ MB'!J!.J\fio" M~ftOOtoA WTO, ffiL~F~~IE~jJt."'~B~.~L ain~~~J.t-. ~._W*mA~m~ •• ~b~.*.M.~#A~o~M~-~m~.~~ B. ~*~-~tt*~~~@~3a, fiH~~~1.~~~m~~~~m~~ rt., 51j£~9~J.AA1itt, 1iJ ~~j£il~JI$i~*~~tf!ifti, !CeJi~1:.m~iti-*.i{t~ *~~~,.~ftID.#~~~~A.~~*~o

~7.m~~uM~~~~~~*, ~*OO~~.~w~n •• OO.~,A ~1II~lt!;te~;f±~il1f~~*&~~iiE, !;j1E~~~~ I¥.rttttlliri~a]~f1::1 Mitti11 .-~.AW.~~*«~.Mo~-~~M*m*.fi~~~~~~~~W, .OO*.~.~~~m*m~~#,.~~~.~~affi*.~* •• ~~.* ~~G~~~M~~~t &~T~.mW~~*~.~~~t~~mm~~mffi AW~~tt*~~~*.~.m~§*~~~~m~

~.~*.~ •• ~~m.M'~~*«~~~~I~W~-~*M~._~ ~::f1WHfJttm, ft*±~ft6 (www.pptph.com.cn) J:.fitBTJt1n1tJtt~tj:mtI:HRm ~ BtItF ~ tR rn.. 115 ~$jJHs Ii I Jll;fij ttl JW.ti!i(St~1f1*, & B1tE ~ -t1t1fj, ttil~lt 0 * mr- 7;: f(~f!j1O ~ 1:~{tm ~ rfl;l' ~ftrit W.R 111 &. tfttEt !tin, airJ #.ftlU@i~ a<J.& ~ ~ IT t&:i£ R1fJ aq IfF t fi ttl J! ~ J!*l¥.J i1111UW.f~Lm.*4~.!:j tt*fl*f ()

Alfred V. Aho;fO Jeffrey D. Ullman ~rr.mt-4~~~AlR+?t~~~f'F1tQ Alfred V. Abo ffi.~ AT&T .!JUF~!j1[it.m~lIWf~$, Jeffrey D. UUman Jl/9~ IEWTifI;fI*~it.mf4~~~*~·" liLl(: 1977 $, fmir}E;<~7 «Principles of CompilerDesign» ~~, 1986 ~ Alfred V. Abo .. Ravi Sethi::to Jeffrey D. Ullman ~

AY.. ir~ 1 {Compilers: Principles, Techniques. and Tools), X1fJwHW:it mi31 ~tr7~~~.~*~7~~~~~~~~*~$~*X~~~W*~fflm~M .~ 1R1ltfie<Jtirf!J1tmiamflM..it.ml4~~A9L ~1ft, t;ffl~~~9~ • .g~ ~~~.~~~M~o~~*~~~*.~~~~oo~~~~~a~~.m.~ l~j;.*~fij*~ftQ ».. 20 fitre 80 ~1ttp.M¥. 4', f\tmMi~ffiJW 7 ~.®:JI~ti~~ ••• ~~~~~~#~~nM~.M~.~3~~~.~~~~*~o

*-=f=;~ 12.;

~~.ftmT~w.~~*~~, ~~*~~~~$~~~~~ m=.~lliT~~~~~~~A~~~~~*~~~~~B,~~~ffl*~

~-@~*~~*~.o~~.~~-@W~~~~~T~~~~o .=.~.T~~*m,~,.~~ .. ~~~~m~a~~dftIAQ~. 9=' ~tt*riE@fflTJc* ~m! ~

.tm.WiiH~HtT ±:~a<J*tfTtt*, JMa.g.=f I~ijllf.Jii~3r~.$;j\J:{£)t

fJi~Jt.fttf~fflag LR .$"

m~.~MT •• ~~ft~~~~~.~c*~~X~.».ffl*.*~~

lll~Jllflit~ .

mA.mwl~~~$m~~~~~~~~Q#~~~~~~~ru~~-* ff 1 if:fm I¥.J ltit Q

.~.~~T~»emm~~tr~~~~HMm~Q .)\"~$ra]1H~B9i>t~3f~~ ~.mnJJT.~iI't~#;J~fPJ18(Ri¥:Jt~rB] {~~Q

m1L.~l;£T §:lfj~~tr<J~~o ~~~*I¥J on-the-fly ~Jij1:Jil1J~~ E!fl":1;} .~~~~~R~tt~~~~*.~~~.«~8rt~4*B&~M.fl1~ ~"

;

~+.~~~~~~~~~~T*T~mft*m~~~w~~~, ffifl*r

1mW j£iT~fiUtttt~£*litt: ~ .+~.M~7~~~B~.~N~~~~~~~-~~~ijeo.#Ifi

I

~~~~~~ft~~~.M~m~~o m+=~~~~~~ffl*~~~~*~~~~~~wB~~~mmQ ~~~~-*.~tt.~, ~~*~~ •• mf~~~~B~~tt~~m, ~~m~tt~~~~~§~mBLo~~12~~~~W~~~T~~~~rr~~ fl*~~,~E~~~~~ftw~~~~fi~~rr~ff~mm,~~~~~~~ 1t{to

~mR~~m~~~~~~~~.*~~~~~~~~Et~-~~m7~ ~~H~~*~~~@ffl"~~a~~.w~~~~~~tttt~~~~~ffi~:

"~m~*~~~*.~Am~, .7~~~~ffl?"~~m~.~~~.*~ ~~~,~*@fflT~~,*~~~~T~~E,~ml~~~~; ~=~*~ aq~J~~31B~~'M'$ I' ~ 4if.gSfi 30---40 1'"1;tJT-:f1];:)~, ;lt~~IIY1\t~1i 521"-

~T.65~~~Z$!*~~m~~~~~~~~*Y~~, ~T~~~ffl9* ~~~~~.ili,.*tt~,~fi~a~m~~~*m¥#.~~~~ft-.% ~~&~~ff*.~.~.M~~~.~T~*.~~~~o

A~IbU ~tHf&l±~!Il:;t.~FL ~1i!l:~*qli~ifl'mB9~ .. R;ffiWf~~tElE P3 ~ ttl ~,~~*.m~~~~~~~&~~F~~m~~~*mo *~~~~~~rr.m~~*~~~~~~W~m~~W~.mm~M$rr .tn tt {tt: ff 1t I¥J A9t~~ ~

m-r*}·}~if.t~ttt atl'aJ-ftift, *Btff~OO~. ~J1tt;JLMm~!tS;fl&mt ,~~ fPJ 11 ~ :fr1~ _iE e

?~1t~ ~-& jffi ~3Cjj*~it .tfttj~~~~·~\ ± ff ••• a~*.~#~~~*e.m~~~~~~ff~m

2

Preface

This book is a descendant of Principles of Compile,. Design by Alfred V + Aho and Jeffrey D. Unman. like its ancestor ~ it is intended as a text for a first course in compiler design. The emphasis is on solving problems universally encountered in designing a language- translator, regardless of the source or target machine.

Although few people are likely to build or even maintain a compiler for a major programming language~ the reader can profitably apply the ideas and techniques discussed in this book to general software design. For example, the string matching techniques for building lexical analyzers have also been used in text editors, information retrieval systems, and pattern recognition programs. Context-free grammars and syntax-directed definitions have been used to build many little languages such as the typesetting and figme drawing systems that produced this book. The techniques of code optimization have been used in program verifiers and in programs that produce "structured" programs from unstructured ones.

Use of Ole Beok

The major topics' in compiler design are covered in depth. The first chapter introduces the basic structure of a compiler and is essential to the rest of the book.

Chapter 2 presents a translator from infix to postfix expressions. built using some of the basic techniques described in this book, Many of the remaining chapters amplify the material in Chapter 2.

Chapter 3 covers lexical analysis, regular expressions, fin ite-state machines. and scanner-generator tools. The material in this chapter is broadly applicable to text-processing.

Chapter 4 covers the major parsing techniques in depth, ranging from the recursive-descent methods that are suitable for hand implementation to the computational'y more intensive LR techniques that have been used in parser generators.

Chapter S introou"s the principal ideas in syntax-directed translation. This chapter is used in the remainder of the book for both specifying and implementing translaticas.

Chapter 6 presents the maio ideas for performing static semantic checting ...

Type checking and unification are discussed in detail.

2

PREFACE

Chapter 7 discusses storage organizations used to support the run-time environment of a program.

Chapter 8 begins with a discussion of intermediate languages and then shows how common programming language constructs can be trllnslated into inter~iate C()de.

Chapter 9 covers target code generation. Included are the basic "on-thefly' code generation methods, as well as optimaJ methods for generating code for expressions. Peephole optimization and code-generator generators are also covered.

Chapter 10 is a comprehensive trcatment of code optimization. Data-flow analysis methods are covered in detail. as well as the principal methods for global optimization.

Chapter t 1 discusses some pragmatic issues that arise in implementing a compiler. Software engineering and testing are particularly important in compiler construction.

Chapter 12 presents case studies of compilers that have been constructed using some of the techniques presented in this book.

AppcndiJ A describes a simple language., a "subset" of Pascal, that can be used as the basis of an implementation project.

The authors have taught both introductory and advanced courses. at tbe undergraduate and graduate levels, from the material in this book at AT&T Bell Laboratories. Columbia, Princeton, and Stanford.

AD introductory compiler course might cover material from the following sections of this book:

introduction lexical analysis symbol tables parsmg syntax-directed

translation

type cbecking run-time organization intermediate

code generation code generation code optimization

Chapter 1 and Sections 2.1-2.5 2.6. 3.1-3.4

2,7,7.6

2.4, 4.1-4.4

2.5. 5.1·5.5 6.1-6.2 7.1~7.3

8.1-S.3 9.1-9.4 10.1-10.2

Information needed for a programming project like tbe one in Appendix A is introduced in Chapter 2.

A course stressing tools in compiler construction might include tbe discussion of lexical analyzer generators in Sections 3.5. of parser generators in Sections 4.8 and 4.9, of code-generator generators in Section 9.12. and material on techniques for compiler construction from Chapter 11.

An advanced course might stress tbe algorithms used in lexical anaJyzer generators and parser generators discussed in Chapters 3 and 4. the material

PREFACE

3

on type eq u iva lence, over load ing, pol ymorph ism, and un i ficat ion in Chapter 6, [he material on run-rime storage organization in Chapter 7, the patternd ireeted code generat ion methods d IsCU ssed in Chapte r 9, a nd materia I on code optimization from Chapter W.

Exercises

A'S before 1 we rate exercises with stars. Exercises without stars [est underst a nd ing of de fin itions, sin gly starred exercises are intended for more advanced courses, and doubly starred exercises are food for thought.

Acknowledgments

At various stages in the writing of this book, a number of people have given us invaluable comments on the manuscript. In this regard we owe a debt of gratitude Co Bill A ppelbe , Nelson Beebe, Jon Bentley, Lois Bogess, Rodney Farrow, Stu Feldman, Charles Fischer. Ch ris Fraser + A r( G ittelm an, Eric Grosse, Dave Hanson. Fritz Henglein, Robert Henry, Gerard Holzmann. Steve Johnson, Brian Kernighan, Ken Kubota, Daniel Lehmann, Dave MacQueen, Dianne Maki, Alan Martin, Doug Mcl lroy , Charles McLaughlin, John Mitchell, Elliott Organick , Robert Paige. Phil Pfeiffer, Rob Pike, Kari-Jouko R aiha, Denn is Ritchie, Sriram Sankar, Paul Stoecker, Bjarne Strousrrup, Tom Szymanski. Kim Tracy, Peter Weinberger, Jennifer Widom , and Reinhard Wilhelm.

This book was phototypeset by the authors using the excellent software available on the UNIX system. The typesetting command read

pic flIes : tbl : eqn : troff -ms

pic is Brian Kernighan's language for typesetting figures; we owe Brian a special debt of gratuude for accommodating our special and extensive figuredrawing needs so cheerfully. tbl is Mike Lesk's language for laying out tables. eqn is Brian Kernighan and Lorinda Cherry's language for typesetting mathematics. troff is Joe Ossana's program for formarnng text for a phototypesetter, which in our case was a Mergenthaler Linotron 202/N. The ms package of troff macros was written by Mike l.esk . to addition, we managed the (ext using make due to Stu Feldman. Cross references within the text-were maintained using awk created by A I Aho, Brian Kernighan, and Peter Weinberger. and sed created by Lee McMahon.

The authors would particularly like '0 acknowledge Patricia Solomon for helping prepare the manuscript for photocomposition. Her cheerfulness and expert typing were greatly appreciated. 1. D. Ullman was supported by an Einstein Fellowship of the Israeli Academy of Arts and Sciences during part of the lime in which this book was written. Finally, the authors would like to thank AT&T Bell Laboratories for its support during the preparation of the

rna n uscript .

A. V. A., R. S., J. D. U.

Contents

Chapter I Introduction to Compiling I

1.1 Compilers H '· • .

1.2 Analysis of the source program .,............................ 4

1.3 The phases of a compiler 10

1.4 Cousins of the compiler 16

1.5 The grouping of phases 20

1.6 Compiler-construction tools 22

Bibliographic notes 23

Cbapter 2; A Simple Ose-Pass Cotnpiler 25

..

2.1 Overview 25

2.2 Syntax definition ,........................ 26

2.3 Syntax-directed translation...................... 33

2.4 Parsing . 40

2.5 A translator for simple expressions 48

2.6 Lexical analysis , H .. .. .. ••• • 54

2.7 Incorporating a symbol table 60

2.8 Abstract stack machines _.............. 62

2.9 Putting the techniques together 69

Exercises .. "........................................................... 78

Bibliographic notes 81

Chapter 3 Lexical Analysis 8J

3.1 The role of the lexical analyzer 84

3.2 Input buffering ..... ,................................................. 88

3.3 Specification of tokens 92

3.4 Recognition of tokens 98

3.5 A language for specifying lexical analyzers .. 105

3.6 Finite automata........ l13

3.7 From a regular expression to an NFA 121

3.8 Design of a lexical analyzer generator 128

3.9 Optimization of DFA~based pattern matchers 134

Exercises 146

Bibliographic notes 157

.... _ ... _..._..,..__ .... _ .... --- .. .. . .. .

2

CONTENTS

Chapter 4 Sy.tax Analysis

159

4.1 The role of the parser 160

4.2 Context-free grammars 165

4.3 Writing a grammar 172

4.4 Top-down parsing' 181

4.5 Bottom-up parsing ;................. 195

4.6 Operator-precedence parsing 203

4.7 LR parsers 215

4.8 Using ambiguous grammars 241

4.9 Parser generators 257

Exercises 267

Bibliographic notes 277

Chapter 5 Syntax-Di~ Translation 179

5.1 Syntax-directed definitions.......................... 280

5.2 Construction of syntax trees 287

5.3 Bottom-up evaluation of S-attributed definitions 293

5,4 L-attributed definitions 296

5.5 Top-down translation .. 302

5.6 Bottom-up evaluation of inherited attributes 308

5.7 Recursive evaluators 316

5.8 Space for attribute values at compile time 320

5 .9 Assigning space at compiler-construction time 323

5.l0 Analysis of syntax-directed definitions 329

Exercises :........... 336

Bibliographic notes 340

Chapter 6 Type Cheeklng 343

6.1 Type systems 344

6.2 Specification of a simple type checker 348

6.3 Equivalence of type expressions 352

6.4 Type conversions 359

6.5 Overloading of functions and operators . 36.

6.6 Polymorphic functions 364

6.7 An algorithm for unification . . .. 376

Exercises 381

Bibliographic notes 386

Copter 7 Run-Time En,lronments 389

7.1 Source language issues 389

7.2 Storage organization.................................. 396

7.3 Storage-anocation strategies 401

7.4 Access to nonloca1 names 4lt

CONTENTS 3

7.5 Parameter passing ,... 424

7.6 Symbol tables , ,....... 429

7.7 Language facilities for dynamic storage allocation ..... 440

1.8 Dynamic storage allocation techniques ,....... 442

7.9 Storage allocation in Fortran ", "................ 446

Exercises , ,........... 455

Bibliographic notes 461

Chapter 8 Intermediate Code Generation 463

8. I Intermediate languages .. , , ,................ 464

8.2 Declarations ,.................................................. 473

8.3 Assignment statements 478

8.4 Boolean expressions , 4B8

8.5 Case statements , - , , , 497

8.6 Backpatching , , , 500

8.7 Procedure cans , , , , , . . . . .. . . 506

Exercises , , ".. 508

Bibliographic notes , ., . . . . .. . . . . . . su

Chapter 9 Code Generation 513

9.1 Issues in the design of a code generator _ . 5]4

9.2 The target machiee .; 519

9.3 R ua-time storage management :........ 522

9.4 Basic blocks and flow graphs 528

95 Next-use information , ,........... 534

9.6 A. simple code generator . . .. . . .. .. .. 535

9.7 Register allocation and assignment :.......... 541

9.8 The dag representation of basic blocks ~............... 546

9.9 Peephole optimization ,............................. 554

9.lO Generating code from dags 557

9.11 Dynamic programming code-generation algorithm 567

9.12 Code-generator generators 512

Exercises , , ,........... 580

Bibliographic notes 583

Chapter II Code Optimization 585

10.1 Introduction ~ ,.... 586

10.2 The principal sources of optimization ,..... 592

10.3 Optim ization of basic blocks ,........... 598

10.4 Loops in flow graphs , , .. "........... 602

10.5 Introduction to global data-flow analysis , "......... 608

10.6 Iterative solution of data-flow equations 624

10.7 Code-improving transformations 633

10.8 DeaUng with aliases 648

4

CONTENTS

10.9 Data-flow analysis of structured flow graphs 660

10.10 Efficient data-flow algorithms 671

10.ll A tool for data-flow analysis 680

10.12 Estimation of types , 694

10.13 Symbolic debugging of optimized code 703

Exercises .... . .. . . . . . . .. .. . . . . . . .. . . . . . . . .. . . . . . . . .. . . .. . . . . . . . . . . . . . . . . 111

Bibliographic notes . 118

Chapter 11 Want to Write I Compiler? 713

11. 1 Planning a compiler " .. . .. . 723

11.2 Approaches to compiler development 125

11.3 The compiler-development environment 729

I 1.4 Testing and maintenance 731

Chapter 12 A Look at Some CompUers 733

12.1 EQN. a preprocessor for typesetting mathematics 733

12.2 Compilers for Pascal ,............................. 734

12.3 The C compilers 735

12.4 The Fortran H compilers 737

12.5 The Blissll1 compiler 740

12.6 Modula-Z optimizing compiler 742

Appeadix A Compiler Project 745

A.l Introduction 745

A.2 A Pascal subset 745

A.3 Program structure 745

A.4 Lexical conventions 748

A.5 Suggested exercises ,.............................. 749

A.6 Evolution of the interpreter 150

A.7 Extensions :...... 751

Bibliography 751

Index

I

788

CHAPTER 1

Introduction to Compiling

The principles and techniques of compiler writing are so pervasive that the ideas found in this book will be used many times in (he career of a computer scientist. Compiler writing spans programming languages. machine arehitecture, language theory. algorithms t and software engineering. Fortunately, a few basic compiler-writing techniques can be used to construct translators for a wide variety of languages and machines. In this chapter. we introduce the subject of compiling by describing the components of a compiler t the environment in which compilers do their job, and some software tools that make it easier to build compilers.

1. t COMPILERS

Simply stated, a compiler is a program that reads a program written in ORe language - the source language - and translates il into an equivalent program in another language - the targe/langlJage (see Fig. 1.1). As an important part of tbis translation process. the compiler reports to its user the presence of errors in the source program.

source program

larget program

error messages

Fig. 1.1. A compiler.

At first glance, the variety of compilers may appear overwhelming. There are thousands of source languages, ranging from traditional programming languages such as Fortran and Pascal to specialized languages Chat have arisen in virtually every area of computer application. Target languages are equally as varied; a target language may be another programming language, or the machine language of any computer between a microprocessor and a

2 INTRODLlCTION TO COMPIUNG

SEC. 1.1

supercomputer. Compilers arc sometimes classified as single-pass. multi-pass, Joad-and-go, debuggmg, or optimizing. depending on how they have been constructed or on what funcrion they arc supposed to perform. Despite chis apparent complexity. the basic tasks that any compiler must perform are essentially the same. By understanding these tasks, we can construct compilers for a wide variety or source languages and target machines using the same basic techniques.

Our knowledge about how to organize and write compilers has increased vastly since the first compilers started to appear in the early I 9SO's < It is difficult to give an exact dale for the firsl compiler because io itially a great deal of experimentation and implementation was done independently by several groups. Much of the early work. on compiling dealt with the translation or arithmetic formulas into machine code.

Throughout the 1950's, compilers were considered notoriously difficult pmgrams to write. Tile first Fortran compiler, for example. took (g staff-years to implement (Backus et al. [1957]). We have since discovered systematic techniques for handling many of the important tasks that occur during compilation. Good implementation languages. programming environments. and software tools have also beell developed. With these advances. a substantial compiler call be implemented even as a student project in a one-semester compiler-design course.

TIt! Analysis..s,nthesis Model of Compilation

There are two parts to compilation: analysis and synthesis. The analysis part breaks up the source program into constituent pieces and creates an intermediate representat ion of the source program. The synthesis part constructs the desired target program from the intermediate representation. Of the two parts, synthesis requires the most specialized techniques. We shall consider analysis informally ill Section 1.2 and outline the way target code is synthesized in a standard compiler in Section 1.3.

During analysis, the operations implied by the source program are determined and recorded in a hierarchical structure called a tree. Often, a special kind of tree caned a syntax tree is used. in which each node represents an operation and the children of a node represent the arguments of the operation. For example. a syntax tree for an assignment statement is shown in Fig. 1.2 .

. - .-

/'

position

<, -: initial.

+

<, /"

ra.te

..

<, 60

.... 1.2 .. Syntax tree for position:. initial ... rate. 60.

SEC. 1.1

COMPILERS 3

Many software tools that manipulate source programs first perform some kind of analysis. Some examples of such tools inc1ude:

l , Structure editors. A structure editor takes as input a sequence of commands to build a source program. The structure editor not only performs the text-creation and modification functions of an ordinary text editor, but it also analyzes the program text, putting an appropriate hierarchical structure on the source program. Thus, tne structure editor can perform additional tasks that are useful in tbe preparanon of programs. For example, it can check. that the input is correctly formed, can supply keywords automatically (e.g .• when the user types while. the editor supplies the matching do and reminds the user thac a conditional must come between them), and can jump from a begin or left parenthesis 10 its matching end or right parenthesis. Further. the output of such an editor is often similar to the output of the analysis phase of a compiler.

2. Prelty printers. A pretty printer analyzes a program and prints it in. suth a way that the structure of the program becomes clearly visible. For example. comments may appear in a special ron., and statements may appear with an amount of indentation proportional to the depth of their nesting in the hierarchical organization of the statements.

3. Stot;c checkers. A static checker reads a program, analyzes it, and attempts to discover potential bugs without running tbe program. The analysis portion is often similar to that found in optimizing compilers of the type discussed in Chapter 10. For example" a static checler may detect that parts of the source program can never be executed, or that a certain variable might be used before being defined. In addition, it can catch logical errors such as trying to use a real variable as a pointer. employing the type-checking techniques discussed in Cbapter 6.

4. lmerpreters, Instead of producing a target program as a translatioD, an interpreter performs the operations implied by the source program. For an assignment statement, for example. an interpreter might build a tree tike Fig. 1.2, and then carry out the operations at the nodes as it "walks" the tree. At the root it would discover it had an assignment to perform. so it would call a routine to evaluate the expression on the rightt and then store the resu Iting value in the location associated with the identifier pasi tion. At the right child of the root, the routine would discover it had to compute the su m of two expressions. It would caU itself recursively to compute the value of the expression rate * 60. It would then add that value to the value of the va.riable initial.

Interpreters are frequenlly used to execute command languages. since each operator executed in a command language is usually an invocation of a complex routine such as an editor or compiler. Similarly. some "very high-kve!" languages, like A PL~ are normally interpreted because there are many things about the data, such. as the size and shape of arrays. that

4 INTRODUCTJON TO COMPILING

SEC. 1.1

cannot be deduced at compile time.

Traditionally, we think of a compiler as a program that translates a source language like Fortran into the assembly or machine language of some computer. However, there arc seemingly unrelated places where compiler technology is regularly used. The analysis portion in each of the following examples is similar to that of a conventional compiler,

J. 'rea formatters. A text forrnatter takes input Iha( is a stream of characters, most of which is text to he typeset, but some of which includes commands to indicate paragraphs. figures. or mathematical structures like subscripts and superscripts. We mention some of the analysis done by text formatters in [he next section.

2, Silicon compilers, A silicon compiIer has a source language that is similar or identical to a conventional programming language. However. the variables of the language represent) not locations in memory, but, logical signals (0 or I) or groups of signals in a switching circuit. The output is a circuit design in an appropriate language. See Johnson 119831. Ullman 119841, or Trickey 11985 J for a d iscussion of silicon compi lat ion.

3. Query inierpreters. A query interpreter translates a pred icate conta i n ing relational and boolean operators into commands to search a database for records satisfying that predicate. (See Ullman [19821 or Date 119861.)

The Context 6f a Compiler

In addition to a compiler. several other programs may be required to create an executable target program, A source program may be divided into modules stored in separate files. The task of collecting the source program is sometimes entrusted to a distinct program. called a preprocessor. The preprocessor may also expand shorthands called macros. into source language statements.

Figure t. 3 shows a typical "corn pilation ." The target program created by the compiler may require further processing before it can be run. The compiler in Fig. 1.3 creates assembly code that is translated by an assembler into machine code and then linked together with some library routines into [he code that actually runs on the machine.

We shall consider the components of a compiler in the next two sections; the remaining programs in Fig. 1.3 are discussed in Section 1.4.

1.1 ANALYSIS OF THE SOURCE PROGRAM

In this section, we introduce analysis and illustrate its use in some textformatting languages. The subject is treated in more detail in Chapters 2-4 and 6. In compiling, analysis consists of three phases:

I. Linear analysis, in which the stream of characters making up the source program is read from left-to-right and grouped into tokens that are sequences of characters having a collective meaning.

SEC, 1.2

ANALYSIS OF THE SOURCE PROGRAM 5

skeletal source program

source program

target assembly program

rclocatublc maehiac code

library. rclocatablc object tiles

absolute machine code

Fig. 1.3. A language-precessing system.

2. Hierarchicat analysis, in which characters or tokens are grouped hierarchically into nested collections with collective meaning.

3, Semantic analysis, in which certain checks are performed to ensure that the components of a program fit together meaningfully.

Lexical AaaJ)sis

In a compiler, linear analysis is called lexical analysis or scanning. For example, in lexical analysis the characters in the assignment statement

'position := initial + ra~e 4 60 wouJd be grouped into the following tokens;

1. The identifier position.

2. The assignment symbol :-,

3. The identifier initial.

4. The plus sign.

S. The identifier ra teo

6. The multiplication sign.

7. The number 60.

The blanks separating the characters of these tokens would normally be eliminated during lexical analysis.

6 INTRODUCTION TO COMPILING

SEC. 1.2

Syntax Analysis

Hierarchical analysis is called parsing or syntax anatysis. II involves grouping the tokens of the source program into grammatical phrases that are used by the compiler to synthesize output. Usually, the grammatical phrases of the source program are represented by a parse tree such as the one shown in Fig. 1.4.

assignment statement

/

idemifier

I

position

::~

expression

/!~

e xpression e xpression

iden!ifier .> 1 <.

I expression apressitm

ini tia,l I I

identijier number

I I

rate 60

Fig. 1.4. Parse tree for position::: ini tial + rate * 60.

In the expression initial + rate .. 60, the phrase rate * 60 is a logical unit because the usual conventions of arithmetic expressions tell us that multiplication is performed before addition. Because the expression initial + rate is followed by a *. it is no! grouped into a single phrase by itself in Fig. 1.4.

The hierarchical structure of a program is usually expressed by recursive rules. For example, we might have the following rules as part of the definition of expressions:

I. A ny identifier is an expression.

2. A ny number is an expression.

3. If expres ... ion I and expression 2 are expressions, then so are

expression 1 + expression 2 expression I * expre,o;; s ion ; ( expression I )

Rules (I) and (2) are (nonrecursive) basis rules, while (3) defines expressions in terms of operators applied (0 other expressions. Thus. by rule (1). initial and rate are expressions. By rule (2). 60 is an expression. while by rule (3). we can first infer that rate * 60 is an expression and finally [hat initial +rate" 60 is an expression.

Similarly, many languages define statements recursively by rules such as:

SEC. 1.2

ANALYSIS OF THE SOURCE PROGRAM 7

I. If identifier I is an identifier, and expression 2 is an ex pression, then identijier, : = expr(,."O.fiOn2

is a statement.

2. If expression I is an expression and statement 2 is a staremen t. then while { expression I } do statement 2

if ( expression I ) then statemem 1

are statements.

The division between lexical and syntactic analysis is somewhat arbitrary.

We usually choose 3 division that simplifies the overall task of analysis. One factor \0 determining the division ;s whether a source tanguage construct is inherently recursive or not. Lexical constructs do not require recursion, while syntactic constructs often do. Context-free grammars are a formalization of recursive rules that can be used to guide syntactic analysis. They are introduced in Chapter 2 and studied extensively in Chapter 4.

For example. recursion is not required to recognize identifiers, which are typically strings of letters and digits beginning with a letter. We would normally recognize identifiers by a simple scan of the input stream. waiting unli1 a character that was neither a letter nor a digit was found. and then grouping an the letters and digits found up to that point into an identifier token. The characters so grouped are recorded in a table. called a symbol table. and removed from the input so that processing of the next token can begin.

On the other hand. this kind of linear scan is not powerful enough to analyze expressions or statements. For example" we cannot properly match parentheses in expressions. or begin and end in statements. without putting some kind of hierarchical or nesting structure on the input.

.-

/' "-

position ..

/ "

initial ..

/' "'"

position

/

initi.l

<, /

rate

*

(a)

(b)

<, "'Uoreal I

60

/'

rate

<, 60

Fig. 1.5. Semantic analysis inserts a conversion from integer 10 real,

The parse tree in Fig. 1.4 describes the syntactic structure of the input. A more common internal representation of this syntactic structure is given by the syntax tree in Fig. L5(a). A syntax tree is a compressed representation of the parse tree in which the operators appear as the interior nodes. and the operands of an operator are the children of t he node for that operator. The construction of trees such as the one in Fig. 1.5(3) is discussed in Section 5.2.

8 INTRODUCTION TO COMPIUNG

SEC. 1.2

We shall take up in Chapter 2, and in more detail in Chapter 5, the subject of syntax-directed translation. in which the compiler uses the hierarchical structure Or! the input to help generate the output.

SelD8Dtic Analysis

The semantic analysis phase checks the source program for semantic errors and gathers type information for the subsequent code-generation phase. It uses the hierarchical structure determined by the syntax-analysis phase to identjfy the operators and operands of expressions and statements.

An important component of semantic analysis is type checking. Here (he compiler checks that each operator has operands that are permitted by the source language specification. For example, many programming language definitions require a compiler to report an error every time a real number is used to index an array. However. the language specification may permit some operand coercions, for example, when a binary arithmetic operator is applied to an integer and real. [0 this case; the compiler may need to convert the integer to 3 real. Type checking and semantic analysis are discussed in Chapter 6.

Example 1.1. Inside a machine, the bit pattern representing an integer is generally different from the bit pattern for a real, even if the integer and the real number happen to have the same value. Suppose I for example. that all identifiers in Fig. 1.5 have been declared to be rea Is and that 60 by itself is assumed to be an integer. Type checking of Fig. 1.5(a) reveals that • is applied to a real, rate, and an integer, 60. The general approach is to CORvert the integer into a reaL This has been achieved in Fig. t.5(b) by creating an extra node for the operator inttoreal that explicitly converts an integer into a real. Alternatively. since the operand of Inttoreal is a constant, the compiler may instead replace the integer constant by an equivalent real constant. 0

Analysis in Text Formatters

It is useful to regard (he input to a text formatter as specifying a hierarchy of boxes that are rectangular regions to be filled by some bit pattern, representing light and dark. pixels to be printed by the output device.

For example, the lEX system (Knuth (1984a n views its input this way.

Each character that is not part of a command represents a box containing the bit pattern for that character in the appropriate font and size. Consecutive characters not separated by "white space" (blanks or newline characters) are grouped into words. consisting of a sequence of horizontally arranged boxes. shown schematically in Fig. 1.6. The grouping of characters into words (or commands) is the linear or lexical aspect of analysis in a text formatter,

Boxes in lEX may be built from smaller boxes by arbitrary horizontal and vertical combinations. For example.

\hbox{ <list of boxes> }

SEC. 1.2

ANALYSIS OF THE SOURCE PROGRAM 9

Fig. 1.6. Grouping of characters and words into boxes,

groups the list of boxes by juxtaposing them horizontally. while the \ vbox operator sirn ilarly groups a list of boxes by vert ical juxtaposit ion. Thus. if we say in lEX

'hbox{\vbox{! 1} 'vbox{@ 2}}

we get the arrangement of boxes shown In Fig. 1.7. Determining the hierarchical arrangement of boxes implied by the input is part of syntax analysis in lEX.

Fig. 1.7. Hierarchy of boxes in TEX.

As another example, the preprocessor EQN for mathematics (Kernighan and Cherry 1197SJ). or the mathematical processor in lEX. builds mathematical expressions from operators like sub and sup for subscripts and super· scripts. If EQN encounters an input text of the form

BOX sub box

it shrinks the size of box and attaches it to BOX near the lower right corner. as illustrated in Fig. 1.8. The sup operator similarly attaches box at (he upper right.

Fie. 1.8. Building the subscript structure in mathematjcal text,

These operators can be applied recursively, so, for example. the EQN text

10 INTRODUCTION TO COMP1L1NG

SEC. 1.2

a sub {i sup 2}

results in Cli~' Grouping the operators sub and su.p into tokens is part of the lexlcal analysis of EQN text. However, the syntactic structure of the text is needed to determine the size and placement of a box.

1.3 THE PHASES OF A COMPILER

Conceptually, a compiler operates in phases. each of which fran sforrns the source program from one representat ion to another. A typical decomposition of a compiler is shown in Fig. 1.9. In practice. some of the phases may be grouped together, as mentioned in Section 1.5, and the intermediate representations between the grouped phases need not be explicitly constructed.

source program

lexical

generator

target program

Fig. 1.9. Phases of a compiler.

The first three phases. forming the bulk of the analysis portion of a compiler, were introduced in the last section. Two other activities. symbol-table management and error handling, are sbown interacting with the six phases of lexical analysis. syntax an alysis, semantic analysis, intermediate code generation, code optimization. and code generation. Informally, we shall also call the symbol-table manager and the error handler "phases."

sec. ).3

THE PHASES OF A COMPILER II

Symbol .. Table Management

An essential function of a compiler is to record the identifiers used in the source program and collect information about various attributes of each identifier. These attributes may provide information about the storage allocated for an identifier. irs type, it s scope (where in the program it is valid), and. in the case of procedure names, such things as the number and types of its arguments. the method of passing each argument (e.g .• by reference), and the type returned. if any.

A .~ymbol table is a data structure containing a record for each identifier. with fields for the attributes of the identifier. The data structure allows. us to find the record for each identifier qu ick Iy and to store or retrieve data from 'hat record quickly. Symbol tables are discussed in Chapters 2 and 7.

When an identifier in the source program is detected by the Ie xica I analyzer. the identifier is entered into the symbol table. However, the attributes of an identifier cannot normally be determined during lexical analysis. For example. in a Pascal declaration like

var position, initial, rate : real ;

the type real is not known when position, initial, and rate are seen by the lexical analyzer.

The remaining phases enter information about identifiers into the symbol table and then use this information in various ways. For example. when doing semantic analysis and intermediate code generation, we need to know what the types of identifiers are, so we can check that the source program uses them in valid ways, and so that we can generate the proper operations on them. The code generator typically enters and uses detailed information about rhe storage assigned to identifiers.

Error Detection and Reporting

Each phase can encounter errors. However, after detecting an error, a phase must somehow deal with that error. SO that compilation can proceed, allowing further errors in the source program to be detected. A compiler that stops when it finds the first error is not as helpful as it could be.

The syntax and semantic analysis phases usually handle a large fraction of the errors detectable by the compiler, The lexical phase can detect errors where the characters remaining in the input do not form any token of the language. Errors where the token stream violates the structure rules (syntax) of the language are determined by the syntax analysis phase. During semantic analysis the compiler tries to detect constructs that have the right syntactic structure but no meaning to the operation involved, e.g., if we try to add two identifiers. one of which is the name of an array, and the other the name of a procedure. We discuss the handling of errors by each phase in the pari of the book devoted to that phase.

12 INTRODUCTION TO COM~ll.ING

SEC.!.3

The Anal)'sis Phases

I

As translation progresses, the compiler's internal representation of the source program changes. We illustrate these representations by considering the translation of the statement

position ;~ initial + rate * 60

( 1.1)

Figure 1.10 shows the representation of this statemem after each phase.

The lexical analysis phase reads the characters in the source program and groups them into a stream of tokens in which each token represents a logically cohesive sequence of characters, such as an identifier, a keyword (if. while, etc.) , a pu net uat ion character, or a rnu hi-character operator like : = . The character seq uence form ing a token is CII. ned the lexeme for the token.

Certain tokens will be augmented by a "lexical value. " For example. when an identifier like rate is found) the lexical analyzer not only generates a token. say id, bu. also enters the lexerne rate into the symbol table. if it is not already there. The lex ical value associated with this occurrence of id points to Ihe symbol-table entry for rate.

In th is section, we shall use id I, Id2• and id) for pos i tion, ini tial, and rate. respectively. to emphasize that the internal representation of an identifier is different from the character sequence forming the identifier. The representation of ( 1.1) after lexical analysis is therefore suggested by:

( 1.2)

We should also make up tokens for the multi-character operator : =: and the number 60 to reflect their mrernal representation. but we defer that until Chapter 2. Lexical analysis is covered in detail in Chapter 3.

The second and third phases, syntax and semantic analysis, have also been introduced in Section 1.2. Syntax analysis imposes a hierarchical structure on the token stream, which we shan portray by syntax. trees as in Fig. L 1 I(a). A typical data structure for the tree IS shown in Fig. 1.1l(b) in which an interior node is a record with a field for the operator and two fields containing pointers to (be records for t he left and right ch ildren. A leaf is a record with two or more fields. onc to identify the token at the leaf', and the others to record information about the token. Additional information about language constructs can be kept by adding more' fields to the' records for nodes. We discuss syntax and semantic analysis in Chapters 4 and 6, respectively.

Intermediate Code Generation

A fter syntax and semantic analysis. some compilers generate an explicit intermediate representation of the source program. We can think of this intermediate representation as a program for an abstract machine. This intermediate representation should have two important properties; it should be easy to produce. and easy to translate into the target program.

The intermediate representation can have a variety of forms. In Chapter 8,

SEC. 1.3

SYMBOL ThBLE

position

2 initial 3 rate

4

THE PHASES OF A COMPllER t3

position :a initial + rate * 60

lexical analyzer

syntax analyzer

._ .-

intermediate code. generator

temp1 .- inttoreal(60) temp2 := id3 * temp1 temp3 :z id2 + temp2 id1 :. temp3

code o~timiz.r

temp1 ~~ id3 * 60.0 id1 ;= id2 + temp1

code generator

MOVF id3, R2
MULF 160.0, R2
MOVF id2, R1
ADDF R,2, R1
MOVF R 1 t id 1 Fig. 1.10. Translation of a statement

14 lNTRODUCTION TO COMPIUNG

SEC. l.3

:-
/' <,
id, +-
/' <,
id1 •
-: <,
ld) 60 (a)

(b)

Fig. 1.11. The data structure in (b) is for the tree in (8).

we consider an intermediate form called "three-address code," which is like the assembly language for a machine in which every memory location can act like a register. Th tee-address code consists of a sequence of instructions, each of which has at most three operands. The source program in (I. &) might appear in three-address code as

temp 1 :~ inttoreal(60)

temp2 :- id3 * temp1 temp3 :~ id2 + temp2

( 1.3)

id 1 :. temp3

This intermediate form has several propernes. First. each three-address instruction has at most one operator in addition to the assignment. Thus, when generating these instructions. the compiler has to decide on the order in which operations are to be done; the multiplication precedes the addition in the source program of (1.1). Second. the compiler must generate a temporary name to hold the value computed by each instruction, Third. some "threeaddress" instructions have fewer than three operands. e.g .• the first and last instructions in (1.3).

In Chapter 8t we cover the principal intermediate representations used in compilers. In general. these representations must do more than compute expressions; they must also handle flow-of-control constructs and procedure calls. Chapters Sand 8 present algorithms for generating intermediate code for typical programming language constructs.

Code Optimization

The code optimization phase attempts to improve the intermediate code ~ so that faster-running machine code wiIJ result. Some optimizations are trivial, For example, a natural algorithm generates the intermediate code (1.3), using an instruction for each operator in me tree representation after semantic analysis. even though there is a better way to perform the same calculation. using the two, instructions

SEC. 1.3 .

THE PHASES OF A COMPILER 15

temp 1 : = id3 -* 60.0 id1 := id2 + temp1

There is nothing wrong with this simple algorithm. since the problem can be fixed during the code-oprimization phase. That is, the compiler can deduce that the conversion of 60 from integer to real representation can be done once and for all at compile time. so the inttoreal operation can be eliminated. Besides, temp3 is used only once, to transmit its value to id 1. It then becomes safe to substitute id 1 for temp3, whereupon the last statement of

( l , 3) is not needed and the code of (1.4) resu lis.

There is great variation in the amount of code optimization different compilers perform. In those that do the most. called "optimizing compilers." a significant fraction of the time of the compiler is spent on this phase. However, (here are simple optimizations that significantly improve the running time of the target program without slowing down compilation too much. Many of these are discussed in Chapter 9, while Chapter 10 gives (he technology used by the most powerful optimizing compilers.

( l.4)

Code Genttation

The final phase of the compiler is the generation of target code, cons.sung normally of relocatable machine code or assembly code. Memory locations are selected for each of the variables used by the program. Then. intermediate insrructions are each translated into a sequence of machine instructions that perform 'he same task. A crucial aspect is the assignment of variables to registers.

For example, using registers I and 2. the translation of the code of ( 1.4) might become

MOVF id3, R2
MULF #60.0, R2
MOVF id2, R1 ( 1.5)
ADD!' R2, R1
MOVF R 1 t id1 The first and second operands of each instruction specify a source and destination, respectively. The F in each instruction tells us that instructions deal with floating-point numbers. This code moves the contents of the address' id3 into register 2, then multiplies it with the real-constant 60.0. The # signifies that 60.0 is to be treated as a constant. The third instruction moves id2 imo register I and adds to it the value previously computed in register 2. Finally, the value in register I is moved into the address of id 1, so the code implements the assignme nt in Fig. I. 10. Ch apten 9 covers code generation.

I We have side- stepped the important issue of stor a g~ allocation for the identifiers in the slIUH'C program - A:> We shall see In Ch a ptcr 7. the organizat ion of storage at run-t irne depends (If! the laAguagc bciJlg compiled, SIl)ragc-allocalion decision» arc made either (luring intcrmcdiatl: code generation or during code generation.

16 INTROOUCTION TO COMPILING

SEC. 1.4

1.4 COUSINS OF THE COMPILER

As we saw in Fig. 1.3, the input to a compiler may be produced by one or more preprocessors, and further processing of the compiler's output may be needed before running machine code is obtained. En this section. we discuss the context in which a compiler typically operates.

Preprocessors

Preprocessors produce input to compilers. They may perform the following functions:

l. Macro processing. A preprocessor may allow a user to define macros that are shorthands for longer constructs.

2. File inclusion. A preprocessor may include header files into the program text. For example. the C preprocessor cau ses the contents of the fi Ie <global. h> to replace the statement 'include <global. h> when it processes a file containing this statement.

3. "Rational" preprocessors. These processors augment older languages with more modern flow-or-control and data-structuring facilities. For example, such a preprocessor might provide the user with built-in macros for constructs like while-statements or if-statements, where none exist in the programming language itself.

4. Language extensions. These processors attempt to add capabilities to the language by what amounts to built-in macros. For example. the language Equel (Stonebraker er al. (19761) is a database query language embedded in C. Statements beginning with ## are taken by the preprocessor to be database-access statements, unrelated to C. and are translated into procedure calls on routines that perform the database access.

Macro processors deal with two kinds of statement: macro definition and macro use. Definitions are normaUy indicated by some unique character or keyword, like define or macr-o, They consist of a name for the macro being defined and a body, forming its definition. Often. macro processors permit [ormul parameters in their definition. that is, symbols to be replaced by values (a "value" is a string of characters, in this context). The use of a macro consists of naming the macro and supplying actual parameters, that is. values for its formal parameters. The macro processor substitutes the actual parameters for the formal parameters in the body of the macro; the transformed body then replaces the macro use itself.

Example 1.2. The lEX typesetting system mentioned in Section 1.2 contains a general macro facility. Macro definitions take the form

\.define <macro name> <template> {<body>}

A macro name is any string of letters preceded by a backslash. The template

SEC. 1.4

COUSINS OF THE COMPilER 17

is any string of cha racters, wit h strings of the form # 1. #2, ' . . • #9 regarded as formal parameters. These symbols may also appear in the body, any number of times. For example, Ihe following macro defines a citation for the Journal of the ACM.

\define'JACM #1;#2;13.

{{\s! J. ACM} {\bf #1}:#2, pp. #3.}

The macro name is \ JACM , and the template is •• # 1; #2 ;#3."; semicolons separate the parameters and the last parameter is followed by a period. A use of this macro must take the form of the template, except that arbitrary strings may be substituted for the formal parameters.? Thus. we may write

\JACM 17;4;715-728.

and expect to see

J. ACM 17:4. pp. 715-728.

The port ion of the body {\ s 1 J. ACtO calls for an italicized ("slanted ") "J. ACM", Expression {\bf # 1} says that the first actual parameter is to be made boldface; this parameter is intended to be the volume number.

TEX allows any punctuation or string of text to separate the volume. issue, and page numbers in the definition of the \3ACM macro. We could even have used no punctuation at all. in which case TEX would take each actual parame· ter 10 be a single character or a string surrounded by { } 0

Assemblers

Some compilers produce assembly code, as in (1.5). thet is passed to an assembler for further processing. Other compilers perform the job of the assembler, producing relocatable machine code that can be passed directly to the loader/link-editor. We assume the reader has some familiarity with what an assembly language looks like and what an assembler does; here we shall review the relationship between assembly and machine code,

Assembly code is a mnemonic version of machine code, in which names are used insread of binary codes for operations. and names are also given to memory addresses. A typical sequence of assembly instructions might be

MOV a" R1 ADD '2, Rl MOV R1~ b

( 1,6)

This code moves the contents of the address a into register I, then adds the constant 2 to it, treating the contents of register t as a fixed-point number.

2 Well. almost arbu rar)' ~Iring~, SIRCC a simple kfHo-rigl1l scan of the macro usc is made. and a,~ lR"JoOn as a symbol matching the Ii,':X! folhtw,"~ a ti l'ymbol in the template is found. the preceding string is deemed to match #i, Thus, if we tried 10 subsruutc ab; cd for # 1, we would fintJ that only ab matched 11 and cd was matched to 12.

18 INTRODUCTION TO COMPIUNC

SEC. 1.4

and finally stores the result in the location named by b. Thus, it computes b : == a + 2.

It is customary for assembly languages to have macro facilities that are similar to those in the macro preprocessors discussed above.

Two--Pass Assembly

The simplest form of assembler makes two passes over tile input. where a pass consists of reading an input file once. In the first pass. all the identifiers thai denote storage locations are found and stored in a symbol table (separate from that of (he compiler). ldentifiers are assigned storage locations as they are encountered for the first time. so after reading (1.6), for example, the symbol table might contain the entries shown in Fig. 1.12. In that figure, we have assumed that a word. consisting of four bytes. is set aside for each identifier. and that addresses are assigned starting from byte O.

lDEtHIFIER ADDRESS

a 0

b 4

Fig. 1.12. All assembler's symbol table with identifiers of (1.6).

In the second pass, the assembler scans the input again. This time, it translates each operation code into the sequence of bits representing that operation in machine langu age. and it translates each ident iCier represent ing a location into the address given for that identifier in Ihe symbol table.

The output of the second pass is usually relocatable mach ine code. meaning that it can be loaded starting at any location L in memory; i.e., if L is added to all addresses in the code, then all references win be correct. Thus, [he output of the assembler must distinguish those portions of instructions that refer to addresses thai can be relocated.

Example 1.3. The following is a hyporhetical machine code into which the assembly instructions ( 1.6) might be translated.

0001 01 00 00000000 • 0011 01 10 00000010 0010 01 00 00000100 •

( 1.7)

We envision a tiny instruction word, in which the first four bits are the instruction code. with 000 1, 00 1 0, and 00 11 standing for load. store. and add, respectively, By load and store we mean moves from memory into a register and vice versa. The next two bits designate a register, and 01 refers to register ) in each of t he three above instructions. The two bits after that represent a "tag," with 00 standing for the ordinary address mode, where the

SEC. 1.4

COUSlNS OF THE COMPILER 19

last eight bits refer to a memory address. The tag 10 stands For the "irnmediate" mode, where the last eight bits are taken literally as the operand. This mode appears in the second instruction of (1.7).

We also see in (1.7) a * associated with the first and third instructions> This * represents tbe relocation bit that is associated with each operand In relocatable machine code. Suppose that the address space containing the data is to be loaded starting at location L. The presence of the ... means that L must be added to the address of the instruction. Thus, if L = 00001111, i.e., 15. then a and b wou ld be at locations ) 5 and 19, respectively. and the instructions of (1.7) would appear as

0001 01 00 00001111 0011 01 10 00000010 0010 01 00 00010011

( 1.8)

in absolute. or unrelocata hie. machine code. N ole that there is no * associated with the second instruct ion in (1.7). so L has not been added to its address in ( 1.8). which is exactly right because the bits represents the constant 2, not the location 2> 0

Loaders and Link.-Edltors

Usually, a program called a loader performs the two functions of loading and link-editing. The process of loading consists of taking relocatable machine code, altering the relocatable addresses as discussed in Example 1.3, and placing the altered instructions and data in memory at the proper locations.

The link -editor allows us to make a single program from several files of relocatable machine code. These files may have been the result of several different compilations, and one or more may be library files of routines provided by 'he system and available to any program that needs them.

If the files are to be used together in a useful way, there may be some external references. in wh ich the code of one file refers to a location in another file. This reference may be to a data location defined in one file and used ill another, or it may be to the entry point of a procedure that appears in the code for one file and is called from another file. The relocatable machine code file must retain the information in the symbol table for each data location or instruct ion label that is referred to externa l1y . If we do not k now in advance what might be referred to. we in effect must include the entire assembler symbol rable as part of the relocatable machine code.

For example. the code of (1.7) would be preceded by

a 0

b 4

If a file loaded with (1.1) referred to b. then that reference would 00 replaced by 4 plu s the offset by wh ich the data locations in file ( I .7) were relocated,

20 INTRODUCTION TO COMPILING

SEC. 1.5

1.5 THE GROUPING OF PHASES

The discussion of phases in Section 1.3 deals with the logical organ ization of a compi ler. 1 n an implemen tati on. act iv it ies from more than one phase are often grouped together.

Front and Back Ends

Often, the phases are collected into a front end and a btICK. end. The front end consists of those phases, or parts of phases, that depend primarily on the source language and are largely independent of the target machine. These normally include lexical and syntactic analysis. the creation of the symbol table, semantic analysis, and the generation or intermediate code. A certain amount of code optimization can be done by the front end as well. The front end also Includes the error handling that goes along with each of these phases.

The back end includes those portions of the compiler that depend on the target machine, and generally. these portions do not depend on the source language, just the intermediate language. In the back end, we find aspects of the code optimization phase, and we find code generation. along with the necessary error handling and symbol-table operations.

It has become fairly routine to take the front end of a compiler and redo its associated back end to produce a compiler for the same source language on a different machine. If the back end is designed carefully. it may not even be necessary to redesign too much of the back end; this matter is discussed in Chapter 9. It is also tempting to compile several different languages into the same intermediate language and use a common back end for the different fron tends. thereby obtain ing several compi lers for one mach ine . However. because of subtle differences in the viewpoints of different languages, there has been only limited success in this direction.

Passes

Several phases of compilation are usually implemented in a single pass consisting of reading an input file and writing an output file. In practice. there is great variation in the way the phases of a compiler are grouped into passes, so we prefer to organize our discussion of compiling around phases rather than passes. Chapter 12 discusses some representative compilers and mentions the way they have structured the phases into passes.

As we have mentioned, il is common for several phases to be grouped into one pass, and for the activity of these phases to be interleaved during the pass. For example, lexical analysis, syntax analysis, semantic analysis. and intermediate code generation might be grouped into one pass. If so, the token stream after lexical analysis may be translated directly Into intermediate code. I n more deta il, we rna y th ink of the s ynta x analyzer as be ing "in ch arge. " It attempts to discover the grammatical structure on the tokens it sees; it obtains tokens as ;t needs them, by caning the lexical analyzer to find the next token. As the grammatical structure is discovered, the parser calls the intermediate

SEC. 1.5

THE GROUPING m; PHASES 21

code generator to perform semantic analysis and generate a portion of the code. A compiler organized this way is presented in Chapter 2.

Redudlg the Number of Passes

If is desirable 10 have relarivcly few passes, since it takes time 10 read and write intermediate files. On the other hand. if we group several phases into one pass, we may be forced to keep the entire program in memory. because one phase may need information in a different order than a previous phase produces it. The internal form of the program may be considerably larger than either (he source program or the target program, SO this space may not be a trivial matter.

For some phases. grouping into one pass presents few problems. For exampie. as we mentioned above, the interface between (he lexical and syntactic analyzers can often be limited to a single token. On the other hand, it is often very hard 10 perform code generation until the intermediate representation has been completely generated. For example. languages like PLiI and Algol 68 permit variables (0 be used before they are declared. We cannot generate the target code for a construct if we do not know the types of variables involved in that construct. Similarly. most languages allow goro's that jump forward in the code. We cannot determine the target address of such a jump until we have seen [he intervening source code and generated target code for it.

In some cases, it is possible to leave a blank slot for missing information. and fill in [he slot when the information becomes available. In particular, intermediate and target code generation can often be merged into one pass using a technique called "backpatching." While we cannot explain all the details until we have seen intermediate-code generation in Chapter 8, we can illustrate backpatch ing in terms of an assembler. Recall that in the previous sect ion we discussed a two-pass assem bler. where the first pass discovered all the identifiers {hat represent memory locations and deduced their addresses as they were discovered. Then a second pass substituted addresses for identifiers.

We can combine the action of the passes as follows. On encountering an assembly statement thelt is a forward reference. say

GOTO target

we generate a skeletal instruction, with the machine operation code for GOTO and blanks for the address. All instructions with blanks for the address or target are kept in a list associated with the symbol-table entry for target. The blanks are filled in when we finally encounter an instruction such as

target: MOV foobart R1

and determine the value of tarqet~ it is the address of the current instruction. We then "backpatch ," by going down the list for target of all the instructions that need its address, substituting the address of target for the

22 INTR ODUCflON TO COMPlUN{J

SEC. 1.5

blanks in the address fields of those instructions. This approach is easy to implement if the instructions can be kept in memory until all target addresses can be determined.

This approach is a reasonable one for an assembler that can keep all its output in memory. Since the intermediate and final representations of code for an assembler are rough1y the same. and surely of approximately the same length. backpatching over the length of the entire assembly program is not infeasible, However, in a compiler. with a space-con sum ing intermediate code, we may need to be careful about the distance over which backpatching occurs.

1.6 COMPILER-CONSTRUCTION TOOLS

The compiler writer. like any programmer, can profitably use software tools such as debuggers, version managers. profilers, and so on. In Chapter t l , we shan see how some of these tools can be used to i!Dplement a_ compiler. In addition to these software-development tools •. other more specialized tools have been developed for helping implement various phases of a compiler. We mention them briefly in this section; they are covered in detail in the appropri-

ate chapters. '

Shortly after the first compilers were written, systems to. help with the compiler-writ ing process appeared. These systems have of len been referred to as compiler-compilers, compiler-generators; or translator-writing systems, Largely) they are oriented around a particular model of languages, and they are most suitable for generating compilers of languages similar to the model.

For. example, it is tempting to assume that lexical analyzers for all languages are essentially the same, except for the particular keywords and signs recognized. Many compiler-compilers do in fact produce fixed lexical analysis routines for use in the generated compiler. These routines differ only in the list of keywords recognized, and this Jist is all that needs to be supplied by the user. Tile approach is valid. but may be unworkable if it is required to recognize nonstandard tokens. such as identifiers that may include certain

characters other than letters and digits. _

Some genera) tools have been created for the automatic design of specific compiler components. These tools use specia1ized languages for specifying and implementing the component, and many use algorithms tnat are quite sophisticated. The most successful tools are those that hide the details of the generation algorithm and produce components that can be easily integrated into the remainder of a compiler. The following is a list of some useful compiler-construcuon tools:

I. Parser generators. These produce syntax. analyzers, normally from input that is based on a context-free grammar. In early compilers. syntax analysisconsumed not only a large fraction of the running time of a compiler, but a large fraction of the intellectual effort of writing a compiler. This phase is now considered one of the easiest to implement. Many of

CHAPTER I

BIBLlOGR APHJC NOTES 23

the "lurle languages" used to typeset this book) such as PIC (Kernighan (1982)) and EQN. were implemented in a few days using the parser generator described in Section 4.7. Many parser generators utilize powerful parsing algorithms that are too complex to be carried out by hand.

2. Scanner generators. These automatically generate lexical analyzers. normally from a specification based on regular expressions, discussed in Chapter 3. The basic organization of the resulting lexical analyzer is in effect a finite au tomaton . A t ypical scali ner generator and its implernentation are discussed in Sections 3.5 and 3-8.

3. Symax-directed translation engines. These produce collections of routines that walk the parse tree, such as Fig> 1.4, generating intermediate code. The basic idea is that one or more "translations" are associated with each node of the parse tree, and each translation is defined in terms of translations at its neighbor nodes in the tree. Such engines are discussed in Chapter 5.

4. Automatic code- generators. Such a fool takes a collection of rules that define the translation of each operation of the intermediate language into the machine lang .. age for the target machine. The rules must include sufficient detail that we can handle the different possible access methods for data; e.g .• variables may be in registers I in a fixed (static) location in memory, or may be allocated a position on a stack. The basic tech n ique is "template matching." The intermediate code statements are replaced by "templates" that represent sequences of machine instructions. ill such a way that the assumptions about storage of variables match from template to template. Since there are usually many options regarding where variables are to be placed (e.g., in one of several registers or in memory), there are many possible ways to "tile" intermediate code with a given set of templates, and it is necessary to select a good tiling without a combinatorial explosion in running time of the compiler. Tools of this nature are covered in Chapter 9.

5. Data-flow engines, Much of the information needed to perform good code optim izat ion involves "data-now analysis," the gathering of in formation about how values are transmitted from one part of a program to each other part. Different tasks of this nature can be performed by essentially the same routine, with the user supplying details of the relationship between intermediate code statements and the information being gathered. A tool of this nature is described in Section 10.11.

BIBLIOGRAPHIC NOTES

Writing In 1962 on the history of compiler wntmg, Knuth ll9621 observed that, "In this field there has been an unusual amount of parallel discovery of the same technique by people work ing independently." He contin ued by observing that several individuals had in fact discovered "various aspects of a

24 INTRODUCTION TO COMPILING

CHAPTER 1

techn ique, and it has been pol ished up through t he years into a very pretty algorithm, which none of the originators fully realized." Ascribing credit for techniques remains a perilous task; the bibliographic notes in th is book are intended merely as an aid for further study of the literature.

H istorical notes on the development of programming languages and compilers until the arrival of Fortran may be found in Knuth and Trabb Pardo 11977] . Wexetblat 119811 contain s historical recollections about several programming languages. by participants in their development.

Some fundamental early papers on compiling have been collected in Rosen tI9671 and Pollack [19721. The January 1961 issue of the Communications of the ACM provides a snapshot of the state of compiler writing at the time. A , detailed account of an early Algol 60 compiler is given by Randell and Russell [19641,

Beginning in the early '960~s with the study of syntax; theoretical studies have had a profound influence on the development of compiler technology, perhaps, at least as much influence as in any other area of computer science. The fascination with syntax has long since waned, but compiling as a whole continues to be the subject of lively research. The fruits of this research will become evident when we examine compiling in more detail in the following chapters.

CHAPTER 2

A Simple One-Pass Compiler

This chapter is an introduction to the material in Chapters 3 through 8 of this book. It presen ts a number of basic com piling techniques th at are ill u strated . by developing a working C program that translates infix expressions into postfix form. Here. the .emphasis is on the front end of a compiler. that is, on lexical analysis. parsing, and intermediate code generation. Chapters 9 and 10 cover code generation and optimization.

2.1 OVERVIEW

A programming language can be defined by describing what its programs. look like (the syntax of the language) and what its programs mean (the semantics of the language). For specifying the syntax. of a language, we present a widely Llsed notation, called con text - free grammars or BN F (for Back u s-N aur Form). With the notations curren1ly available, the semantics of a language is much more difficult to describe than the syntax. Consequently, for specifying the semantics of a language we shall use informal descriptions and suggestive

"

examples.

Besides specifying (he syntax of a language. a context-free grammar can be used to help guide 'he translation of programs. A grammar-oriented cornpiling technique, known as syntax-directed translation, is very helpful for organizing a compiler front end and wiIJ be used extensively throughout this chapter.

In the course of discussing syntax-directed translation, we shall construct a compiler that translates infix expressions into postfix form I a notation in which the operators appear after their operands. For example, the postfix form of the expression 9-5+2 is 95-2+. Postfix notation can be converted directly into code for a computer that performs all its computations using a stack. We begin by constructing a simple program to translate expressions consisting of digits separated by plus and minus signs into postfix form. As the basic ideas become clear; we extend the program to handle more general programming language constructs. Each of our translators is formed by systernatically extending the previous one.

26 A SIMPLE COMPILER

SEC. 2.2

In our compiler, the lexical analyzer converts the stream of input characters into a stream of tokens that becomes the input to the following phase, as shown in Fig. 2.1. The "syntax-directed translator" in the figure is a combination of a syntax analyzer and an intermediate-code generator. One reason for starting with expressions consisting of digits and operators is to make lexical analysis initially very easy; each input character forms a single token. Later, we extend the language to include lexical constructs such as numbers, identifiers, and keywords. For this extended language we shall construct a lexical analyzer that collects consecutive input characters into the appropriate tokens. The construction of lexical analyzers will be discussed in detail in Chapter 3.

lexical analyzer

character stream

token stream

syntaxdirected translator

intermediate reprcscntat ion

Fig. 2.1. Structure of our compiler front end.

2.2 SYNTAX DEFINITION

In this section. we introduce a notation, called a context-free grammar (grammar. for short), for specifying the syntax of a language. It will be used throughout (his book as part of the specification of the front end of a compiler.

A grammar naturally describes the hierarchical structure of many programming language constructs. For example. an if-else statement in C has the form

if ( expression) statement etse statement

That is, the statement is the concatenation of the keyword if, an opening parenthesis. an expression. a closing parenthesis, a statement. [he keyword else. and another statement. (In C, there is no keyword then.) Using the variable expr to denote an expression and the variable stmt to denote a statement, this structuring rule can be expressed as

stmt -+ if ( expr ) stnu else stmt

(2. I)

in which the arrow may be read as "can have the form." Such a rule is called a production. In a production lexica) elements like {he keyword if and the parentheses are called tokens. Variables like expr and SImi represent sequences of tokens and are caned nonterminals.

A context-free grammar has four components:

I. A set of tokens, known as terminal symbols.

SEC. 2.2

SYNTAX DEFlNlTlON 27

2. A set of nonterrninals.

"3. A set of productions where each production consists of a nonterminal, called the left side of the production, an arrow, and a sequence of tokens and/or nonterrninals, called the right side of the production.

4. A designation of one of the nonterrninals as the start symbol.

We follow the convention of specifying grammars by listing their prodeclions, with the productions for the start symbol listed first. We assume that digits, signs such as <=, and boldface strings such as while are terminals. An italicized name is a nonterminal and any non italicized name or symbol may be assumed to be a token. I For notational convenience, productions with the same nonterminal on the left can have their right sides grou ped , wuh the alternative right sides separated by the symbol I, which we read as "or ."

Example 2. t . Several examples in th is chapter use expressions consisting (If digits and plus and minus signs, e.g., 9-5+2, 3-1, and 7. Since a plus or minus sign must appear between two digits. we refer to such expressions as "lists of digits separated by plus or minus signs. to The following grammar describes the syntax of these expressions. The productions are:

list - list + digit list -I> list - digit lisr -I> digit

digit - 0 I 1 I 2 I 3 I 4 I 5 I 6 I 7 I 8 I 9

(2.2) (2.3) (2.4) (2.5)

The righ t sides of the th ree productions with nonterminal list on the left side can equivalently be grouped:

list -+ Jist + digit I list - dig;' I digi;

According to our conventions, the tokens of the grammar are the symbols + - 0 1 234 567 e 9

The nonterminals are the italicized names lis: and dig;'. with lis: being the starting nonterminal because its productions are given first. 0

We say a production is for a nonterminal if the nonterminal appears on the left side of the production. A string of tokens is a sequence of zero or more tokens. The string containing zero tokens, written as E. is called the empty string.

A grammar derives strings by beginning with the start symbol and repeatedly replacing a nonterminal by the right side of a production for that

I lndividual italic letters will be used for adduional purposes when grammars arc studied in dc:t~il in Chapter 4. For example. We shall use X. Y. and Z (0 talk about a symbol that is either It I~kcn or a nonrerminal. However. an)' italicized name containirag IWI) or more characters will continue to represent a nonrerminat.

28 A SIMPLE COMPILER

SEC. 2.2

nonterrninal. The token strings that can be derived from the start symbol form the language defined by the grammar.

Example 2.2. The language defined by the grammar of Example 2.1 consists of lists of digits separated by plus and minus signs.

The ten productions for the nonterminal digit allow it (0 stand for any of the tokens 0, 1, ...• 9, From production (2.4), a single digit by itself is a list. Productions (2.2) and (2.3) express the fact that if we take any list and follow it by a plus or minus sign and then another digit we have a new list.

It turns out that productions {2.2) to (2.5) are aU we need to define the language we are interested in. For example, we can deduce that 9-5+2 is a list as follows.

a) 9 is a list b)' production (2.4), since 9 is a digit.

b) 9-5 is a list by production (2.3). since 9 is a list and 5 is a digit.

c) 9-5+2 is a fist by production (2.2), since 9-5 is a list and 2 is a digit.

This reasoning is illustrated by the tree in Fig. 2.2. Each node in the tree is labeled by a grammar symbol. An interior node and its children correspond to a prod uct ion; the interior node corresponds to the left side of the production, the children to the right side. Such trees are called parse trees and are discussed below. 0

lis:
-> -.
lis: digit
/ -.
list digit
digit
I
9 5 + 2 Fig.2.2. Parse tree for 9- 5+-2 according to the grammar in Example 2. L

Example 2.3. A somewhat different sort of list Is the sequence of statements separated by semicolons found in Pascal begin-end blocks. One nuance of such lists is that an empty list of statements may be found between the tokens begin and end. We may starr to develop a grammar for begin-end blocks by including the productions:

block: begin opLstmts end

opl_<;tmIJ stmt.Jist I E

stmt.Iist stmt.Iis: ; stmt I stmt

SEC. 2.2

SYNTAX DEFINITION 29

Note that (he second possible right side for opLstmts ("optional statement list") is e , which stands for the empty string of symbols. That is, opt.xtmts can be replaced by the empty string, so a block can consist of the two-token string begin end. Notice that the productions for stmt.Jist are analogous to those for list in Exam ple 2.1. with sern icolon in place of (he ar it h metic operator and stmt in place of dixit. We have not shown the productions for stmt, Shortly, we shall discuss the appropriate productions for the various kinds of statements, such as it-statements, assignment statements. and so on. 0

Parse Trees

A parse tree pictorially shows how (he stare symbol or a grammar derives a string in the language, If nonterrninal A has a production A - XYZ, then a parse tree may have an interior node labeled A with three children labeled X, Y, and Z, from left to right:

A

/1'"

X Y Z

Formally, given a context-free grammar, a parse tree is a tree with the following properties:

I. The root is labeled by the start symbol-

2. Each leaf is labeled by a token or by e ,

3. Each interior node is labeled by a nontermmal.

4. If A is the nonrerrninal labeling some interior node and X ,. X 2 •... , XII are the labels of the children of that node from left to right. the" A ._,. X I X 2 ' .• XII is a production. Here, X I' X 2' . • • ,X/I stand for a symbol that is either a terminal or a nonterminal. As a special case, if A -+ f. then a node labeled A may have a single child labeled €.

Example 2.4. In Fig. 2.2. the root is labeled list, the start symbol of the grammar in Example 2.1. The children of the root are labeled, from left to right. JL~I, +, and digit. Note th at

lis - list + dil<il

is a production in the grammar of Example 2. l , The same pattern with - is repeated at the left child of the root, and the three nodes labeled digit each have one child that is labeled by a digit. 0

The leaves of a parse tree read from left to right form the yield of the tree. which is the string generated or derived from the nonterminal at the root of the parse (fee. In Fig. 2.2. the generated string is 9-5+2. ln that figure. all the leaves are shown at the bottom level. Henceforth, we shall not necessarily

30 A SIMPLE COMPILER

SEC. 2.2

iine up the leaves in this way. Any tree imparts a natural left-to-right order to its leaves) based on the idea that if a and b are two children with the same parent, and a is to the left of h. then all descendants of a are to the left of descendants of h.

Another definition of the language generated by a grammar is as the set of strings that can be generated by some parse tree. The process of finding a parse tree for a given string of tokens i~ called parsing that string.

Ambiguity

We have to be careful in talking about the structure of a string according to a grammar. While it is clear that each parse tree derives exactly the string read off its leaves, a grammar can have more than one parse tree generating a given string of tokens. Sucb a grammar is said to ,be ambiguous. To show that a grammar is ambiguous, all we need to do is find a token string that has more than one parse tree. Since a string with more than one parse tree usually has more than one meaning. for compiting applications we need to design unambiguous grammars, or to use ambiguous grammars with additional rules to resolve the ambiguities.

E1t:ample 2.5. Suppose we did not distinguish between digits and lists as in Example 2.t. We could have written the grammar

string -- string + string I siring - string I 0 I 1 I 2 I 3 I 4 I 5 I 6 I 7 I 8 I 9 Merging the not ion of digit and list into the nonterminal string makes superficial sense. because a single digit is a special case of a list.

However, Fig. 2.3 shows that an expression lite 9-5+2 nQW has more than one parse tree, The two trees for 9-5+2 correspond to the two ways of parenthesizing the expression: (9-5) +2 and 9- (5+2). This second parenthesization gives. the expression the value 2 rather than the customary value 6. The grammar 9fExample 2.1 did not permit this interpretation. 0

Associativity of Operators

By con vent ion I 9+ 5 + 2 is equ ivalent to (9+ 5 ) + 2 and 9 - 5 - 2 is equ ivalent to ( 9 - 5 ) - 2. W hen an operand I ike 5 has operators to it s left and right, conventions are needed for deciding which operator takes that operand. We say that the operator + associates to the left because an operand with plus signs on both sides of it is taken by the operator to its left. In most programming languages the four arithmetic operators, addition I su btraction, mu ltiplication , and division are left associative.

Some common operators such as exponentiation are right associative. As another example, the assign rnent operator =: in C is right associative; in C, the expression a=b;c is treated in the same way as the expression a= (b=c } .

Strings like a=b=c with a right-associative operator are generated by the following grammar:

SEC. 2.2

SYNTAX DEFINITION 31

.'flring slri,,!:
/1"'" /1""
S";"1: + ,'irr;nR string string
/I~ 1 I /1"'-
slrirag :;Iring :2 9 string + string
1 I 1 I
9 S 5 2 Fig. 2.3, Two parse trees for 9-5+2,

right - ieuer = right I leuer letter - a I b I ... I z

The contrast between a parse tree for a left-associative operator like - and a parse tree for a right-associative operator like :r;: is shown by Fig. 2.4. Note that the parse tree for 9-5-2 grows down towards the left. whereas the parse tree for a=bzc grows down towards the right.

nSf

/1'"

USf digit

/1'" J

:2

right

/1"'-

letter • right

, /1""

letter = right

I

a

list

dixit

1

5

b

letter

I

I

digit

I

9

c

Fie. 2.4. Parse frees for left- and right-associative operators.

Precedence of Operators

Consider (he expression 9+5 .. 2. There are two possible interpretations of this expression: (9 + 5 ) .. 2 or 9 + ( 5 It 2 ) . The associativity of + and • do not resolve this ambiguity. For this reason, we need to know the relative precedence of operators when more than one kind of operator is, present.

We say that * has higher precedence than + if * takes its operands before + does. In ordinary arithmetic, multiplication and division have higher precedence than addition and subtraction. Therefore, 5 is taken- by it in both 9+5.2 and 9*5+2; i.e., the expressions are equivalent to 9+( 5*2) and (9*5) +2, respectively.

Syntax of expressions. A grammar for arithmetic expressions can be

32 A SIMPLE COMPILER

SEC. 2.2

constructed from a table showing the associativity and precedence of operators. We start with. the four common arithmetic operators and a precedence table, showing (he operators in order of increasing precedence with operators at the same precedence level on the same line:

left associative: .. -

left associative: * /

We create two nonterminals expr and term for the two levels of precedence. and an extra nonterrnjnal [actor for generating basic units in expressions. The basic units in expressions are presently digits and parenthesized expressions.

factor -+ digit ~ ( expr )

Now consider the binary operators. • and /', that have the highest precedence. Since these operators associate to the left, the productions are similar to those for lists that associate to the left.

term - term .. factor I term I factor I factor

Similarly, expr generates lists of terms separated by the additive operators.

expr - r!xpr + term I expr - term l term

The resulting grammar is therefore

expr - expr + term

term -+ term * [actor I

factor - digit I ( expr )

This grammar treats an expression as a list of terms separated by either + or - signs, and a term as. a list of factors separated by * or / signs. Nonce that any parenthesized expression is a factor. so with parentheses we can develop expressions that have arbitrarily deep nesting (and also arbitrarily deep trees).

expr - term term I .factor

term factor

Syntax of statements. Keywords allow us to recognize statements in most languages. A n Pascal statements begin wit h a keyword except assignments and procedure calls. Some Pascal statements are defined by the following (ambiguous) grammar in which the token id represents an. identifier.

stmt ~ id .: = expr

I ir expr then stmt

1 if expr lhen stmt else stmt I while apr do stnu

I begin opt.stmts end

The nonterminal opt.smu« generates a possibly empty list of statements separated by semicolons using the productions in Example 2.3,

SEC. 2.3

SYNTAX·DIRECfED TRANSLATION 33

1..3 SYNTAX·DIRECTED TRANSLATION

To translate a programming language construct, a compiler may need to keep track of many quantities besides tne code generated for the construct. For example, the compiler may need to know the type of tbe construct, or the location of fhe first instruction in the target code. or the number of instructions generated. We therefore talk abstractly about auributes associated with constructs. An attribute may represent any quantity. e.g., a type. a string, a memory location, or whatever.

In this section, we present a formalism called a syntax-directed definuion for specifying translations for programming language constructs. A syntaxdirected definition specifies the translation of a construct ill terms of attributes associated with its syntactic components. In later chapters, syntax-directed definitions are used to specify many of the translations that take place in the front end of a compiler.

We also introduce a more procedural notation. caUed a translation scheme, for specifying translations. Throughout this chapter, we use translation schemes for translating infix expressions into postfix notation. A more detailed d lscussion of syntax-directed definitions and their implementation is contained in Chapter 5.

Postlb Notation

The postfu notouo« for an expression E can be defined inductively as follows; t. 1£ E is a variable or constant, cben the postfix notation for E is E itself.

2. If E is an expression of the form E, op EZi where op is any binary operator. then the postfix notal ion for E is E I' E 2' op, where E,' and E 2' are the postfix notations for Eland E 2> respectively,

3. Jf E is an expression of the form ( E I ), then the postfix notation for E 1 is also the postfix notation for E.

No parentheses are needed in postfix notation because the position and arity (number of arguments) of the operators permits only one decoding of a postfIX expression. For example, the postfix notation for (9-5) +2 is 95-2+ and the postfix notation for 9-(S+2} is 952+-.

Synlax .. Directe4 Deftnitlons

A syntax-directed definition uses a context-free grammar to specify the syntactic structure of the input. With each grammar symbol. it associates a set of attributes. and with each production, a set of semanuc rules for computing values of the attributes associated with the symbols appearing in that productjon, The grammar and the set of semantic rules constitute the syntaxdirected definition.

A IrllDslation is an input-output mapping. The output for each input x is specified in the following manner. First ~ construct a parse tree for x. Suppose

34 A SIMPLE COMPILER

SEC. 2.3

a node n in the parse tree is labeled by the grammar symbol X. We write X.a to denote the vaJue of attribute a of X at that node. The value of X,« at n is computed using the semantic rule for attribute a associated with the X· production used at node n. A parse tree showing the attribute values at each node is called an annotated parse tree.

SYllthesized Attributes

An attribute is said to be synthesized if its value at a parse- tree node is determined from attribute values at the children of the node. Synthesized attributes have the desirable property that they can be evaluated during a single bottom-up traversal of the parse tree. In this chapter, only synthesized attributes are used; "inherited" attributes are considered in Chapter 5.

Example 2.6. A syntax-directed definition for translating expressions consisting of digits separated by plus or minus signs into postfix notation is shown in Fig. 2.5. Associated with each nonterminal is a string-valued attribute I that represents the postfix notation for the expression generated by that nonterminal in a parse tree.

PRODUCTION SEMANTtC RULE
r!xpr -" exprl ... term expr.t - t'xprl . t ~ IHm.1 ~ J."
espr ..... expr I term expr.t txpr,.t I term.t II , - ,
-
apr ... term expr.t - term.t
term 0 term. I - '0'
term ..... 1 term.' . - • 1 r
term -0 9 term.t - '9' Fig. 1.S. Syntax-directed definition for infix to pOiitfix translation.

The postfix form of a digit is the digic itself; e.g., the semantic rule associated with the production term ... 9 defines term.t to be 9 whenever this production is used at a node in a parse tree. When the production expr - term is applied. the value of term.t becomes the value of expr.t,

The prod uction expr "* expr l + term derives an expression containing a plus operator (the subscript in expr I d istingu ishes the instance of expr on the right from that on the Jeft side). The left operand of the plus operator is given b)' expr 1 and the right operand by term. The semantic rule

expr.t ;= expr l.t II term.t II '+'

associated with this production defines the value of attribute expr . t by concatenating the postfix forms expr 1./ and term. t of the left and right operands. respectively, and then appending the plus sign. The operator II in semantic

SEC. 2.3

SYNTAX·DIRECTED TRANSLATION 35

rules represents string concatenation.

Figure 2.6 contains the annotated parse tree corresponding to the tree of Fig. 2.2. The value of the r-attribute at each node has been computed using the semantic rule associated with the production used at that node. The value of the attribute at the root is the postfix notation for the string generated by the parse tree. 0

expr.t = 95-2+-

---- <,

expr.t ;;;::: 9

I

term.' =' 9

expr.t ~ 95-

.> "-

term.t"" :2,

term.t "" 5

9

5

+-

2

Fig. 2.6. Attribute values at nodes in a parse tree.

Example 2.7. Suppose a robot can be instructed to move one step east, north. west, or south from its current position. A sequence of such instructions is generated by the following gram mar:

seq -+ seq instr I begin

instr - east I north ) west ! south

Changes in the position of the robot on input begin west south east east east north north. are shown in Fig, 2.7.

(2.1)
oorth
(- 1,0) west begin
(0,0)
south north
(-J,-I) easl east east (2. -1) Fig.1.7. Keeping Irack of a robot's position.

In the figure. a position is marked by a pair (x.y), wbere x and y represent the nu mber of steps to the east and north. respectively; from the starting

36 A SIMPLE COMPILER

SEC. 2.3

position. (If x is negative. then the robot is to the west of the starling position; similarly. if y is negative. then the robot is to the south of the starting position.)

Let us construct a synrax-d irected definition to translate' an inst ruct ion sequence into a robot position. We shall use two attributes, seq.x and seq.y ; to keep track of the position resulting from an instruction sequence generated by the nonterrninal seq. Initially, W'I generates begin, and seq.x and S(;''I.Y are both initialized to 0, as shown at the leftmost interior node of the parse tree for begin west south shown in Fig. 2.8.

seq.x = - I '~('4'Y = -1

/ <.

seq.x .= - 1 instr.dx = 0

seq.y ;;;;; 0 iR,~/r.dy -= - I

/ <. I

seq.x= 0 s ea.» = 0

I

instr.dx= -1 inslr.dy = 0

I

south

begin

west

Fig. 2.8. Annotated parse tree fur begin west south.

The change in position due to an individual instruction derived from instr is given by attributes instr.dx and lnstr.dy . For example. if instr derives west, then in ... tr.dx = - I and !Msrr.dy =" O. Suppose a sequence seq is formed by following a sequence seq I by a new instruction instr. The new position of the robot is then given by the ru les

seq.x : = seq I'X + instr.dx .w:q.y ;= seq i.y + instr.dy

A syntax-directed definition for translating an instruction sequence into a robot position is shown in Fig. 2.9. 0

Depth-First Traversals

A syntax-directed definition does not impose any specific order for the evaluation of attributes on a parse tree; any evaluation order that computes an attribute a after all the other attributes that a depends on is acceptable. In genera I, we may have to evaluate some attributes when a node is first reached during a walk of the parse tree, others after all its children have been visited, or at some point in between visits to (he children of the node. Suitable evaluation orders are discussed in more detail in Chapter S.

The tram-lations in this chapter can all be implemented by evaluating the semantic rules for the attributes in a parse tree in a predetermined order. A traversal of a tree starts at (he root and visits each node of the tree in some

SEC. 2.3

SYNTAX-DIRECTED TRANSLATION 37

PRODUCTION SEMANTIC RULES
seq.x - 0
seq -+ begin
seq.y - 0
seq.x - seq I'X + instr.dx
Sfq - ,leq I instr
seq.y - :it'q 1 .y + in.\·tl".dy
instr.dx - I
hafr - east
instr.dy . - 0
instr.dx - 0
instr -+ north
instr.dv - I
instr.dx - -I
instr -+ west
instr.dy - 0
instr. dx ~ 0
instr -+ south
instr.dy - -I Fig. 2.9. Syntax-directed definition of the robot's position.

order. In [his chapter, semantic rules wiJl be evaluated using the depth-first traversal defined in Fig. 2. to. h starts at the root and recursively visits the children of each node in left-to-right order, as shown in Fig. 2. II, The semantic rules at a given node are evaluated once all descendants of (hat node have been visited. It is caned "depth-first" because it visits an unvisited child of a node whenever it can. so it tries 10 visit nodes as far away from the root as qu ick Iy as it can.

procedure visit (n: node); begin

for each child m of n, from left tn right do visi: (m);

evaluate semantic rules at node n

Fig. 2.10. A depth- first t r avcrsai of a tree,

Translation Schemes

1n the remainder of this chapter, we use a procedural specification for defining a translation. A translation scheme is a context-free grammar In which program fragments called semantic actions are embedded within the right sides of productions, A translation scheme is like a syntax-directed definition, except that the order of evaluation of (he semantic rules is explicitly shown. The

38 A SI MPLE COMPILER

SEC. 2.3

FiC. 2.11. Example of a depih-fjrst traversal of a [fee.

position at which an action is to be executed is shown by enclosing it between braces and writing it within the right side of a production, as in

rest r+e term {print('+')} re-stl

A translation scheme generates an output for each sentence x generated by the underlying grammar by executing the actions in the order they appear during a depth-first traversal of a parse tree for x. For example; consider a parse tree with a node labeled rest representing this production. The action { prin: ( , + ') } will be performed after the subtree for term is traversed but before the ch lid for rest I is visited,

rest

------ / ' , :-----_

... term (print(' +-')) rest,

Fig. 2.12. A n extra leaf is constructed for a semantic action.

When drawing a parse tree for a translation scheme, we indicate an action by constructing for it an extra child, connected by a dashed line to the node for its production < For example, the portion of the parse tree for the above production and action is drawn as in Fig. 2. l2. The node for a semantic action has no children, 80 the action is performed when (hat node is first seen.

Emitting a Translation

In this chapter. the semantic actions in translations schemes will write the output of a translation into a file, a string OT character at a time. For example, we translate 9-5+2 into 95-2+ by printing each character in 9-5+2 exactly once. without using any storage fOT the translation of subexpressions. When the output is created incrementally in this fashion. the order in which the characters are printed is important.

Notice that the syntax-directed ~efinilions mentioned so far have the following important property: the siring representing the translation of the nonterminal on the left side of each production is the concatenation of the translations

SEC. 2.3

SYNTAX-OIRECTED TRANSLATION 39

of tbe nonrermmals on the right. in the same order as in the production, with some additional strings (perhaps none) interleaved. A syntax-directed definition with this property is termed simple. For example, consider the firs. production and semantic rule from the syntax-directed definition of Fig. 2.5:

PRODUCTION expr .... expr l + term

SEMANTlC RULE

expr.t := expr 1.1 II term.t II ' +'

(2.6)

Here the translation expr.t is the concatenation of the translations of expr I and term. followed by the symbol +. Notice that expr 1 appears before term on the right side of the production.

An additional string appears between term.t and rest r- t in

PkODUCTION rest - + term rest,

SEMANTIC RULE

rest.t .= term.t 1\ t +' "rest}.t

(2.1)

but, again, the nonterminal tum appears before rest. on the right side.

Simple syntax-d irected definitions can be implemented with translation schemes ill which actions print the additional strings in the order they appear in the definition. The actions in the following productions print the additional strings in (2.6) and (2.7). respectively:

expr - expr I + term {print (' + ') } rest .... + term {print(' +')} rest.

Example 2~8~ Figure 2.5 contained a simple definition for translating expressions into postfix form. A translation scheme derived from this definition is given in Fig. 2.13 and a parse tree with actions for 9-5+2 is shown in Fig. 2.14. Note that although Figures 2.6 and 2.14 represent the same inputoutput mapping. the translation in the two cases is constructed differently; Fig. 2.6 attaches the output to the root of the parse tree. while Fig. 2.14 prints the output in crement a l1y .

expr - expr + term { pril1t('.') }
expr -;. expr - term { prjnr(' - ') }
expr - term
term - 0 ( pr;tU(' 0 ') }
term - 1 { print(' 1') }
term - 9 { prim('g') }
FiI.2.13. Actions translaling expressions Into postfix. lI:otatioo. The root of Fig. 2.14 represents the first production in Fig. 2.13. In a depth-firse traversal, we first perform all the actions in the subtree for the left operand apr when we traverse the leftmost subtree of the root. We then visit the leaf. at which there is no action. We neJlt perform the actions ift the

40 A SIMPLE COMPILER

SEC. 2.3

subtree for the right operand term and. finally, the semant ie action {print('+') } at the extra node.

Since the productions for term have only a digit on the right side. that digit is printed by the actions for the productions. No output is necessary for the production expr -. term, and only the operator needs to be printed in the action for tne first two productions. When executed during a depth-first traversal of the parse tree, the actions in Fig. 2.14 print 95-2+. 0

expr

~ I ,,---_

~ + " {prinl('+')}

expr I ttrm

/ 9

expr

~_/ ~{Pi'int('-'»

term

I 5

lam I

2

...

{prifl/('2')}

....

{print ('5')}

...

{prinl{,g')l

Fjg. 2.14. Action S translating 9 - 5+ 2 into 95- 2 +,

As a general rule, most parsing methods process their input from left to right in a "greedy" fashion; that is, they construct as much of a parse tree as possible before reading the next input token. In a simple translation scheme (one derived from a simple syntax.-directed definition), actions are also done in a left-to-right order. Therefore, to implement a simple translation scheme we can execute the semantic actions while we parse; it is not necessary to construct the parse tree at all.

2.4 PARSING

Parsing is the process of determining if a string of tokens can be generated by 20 grammar. 10 discussing this problem, it is helpful to think of a parse tree being constructed, even though a compiler may not actually construct such a tree. However. a parser must be capable of constructing the tree, or else the translation cannot be guaranteed correct.

This section introduces a parsing method that can be applied to construct syntax-directed translators. A complete C program, implementing the translation scheme of Fig. 2.13, appears in the next section. A viable alternative is to use a software tool to generate a translator directly from a translation scheme. See Section 4.9 for the description of such a tool; it can implement the tran slat ion scheme of Fig. 2. 13 without mod ificat ion.

A parser can be constructed for any grammar. Grammars used in practice, however. have a special form. For any context-free grammar there is a parser that takes at most 0(113) time to parse a string of n tokens. But cubic time is

SEC. 2.4

PARSING 41

too expensive. Given a programming language, we can generally construct a grammar that can be parsed quickly. Linear algorithms suffice to parse essentially all languages that arise in pract ice. Programming language parsers almost always make a single left-to-right scan over the input. look ing ahead one token at a time.

Most parsing methods fall into one of two classes, called the top-down and bottom-up methods. Tbese terms refer to the order in which nodes in the parse tree are constructed. In the former. construction starts at the root and proceeds towards the leaves. while, in the latter, construction starts at the leaves and proceeds towards the root. The popularity of top-down parsers is due to the fact that efficient parsers can be constructed more easily by hand using top-down methods. Bottom-up parsing. however, can handle a larger class of grammars and translation' schemes. so software tools for generating parsers directly from grammars have tended to use bottom-up methods.

Top-Down Parsing

We introduce top-down parsing by considering a grammar that is well-suited for this class of methods. Later in th ill sect ion. we consider the construct ion of top-down parsers in genera]. The following grammar generates a subset of the types of Pascal. We use the token dotdot for ••..• , to emphasize that th~ character sequence is treated as a unit.

type -r . simple I t id

I array [ simple ] or type simple ...... integer

I char

I num dotdot DUm.

(2.8)

The top-down construction of a parse tree is done by starting with the root, labeled with the starting nonterminal, and repeatedly performiog the following two steps (see Fig. 2.15 for an example •.

l. At node n. labeled with nonterminal A. select one of the productions for A and construct children at n for the symbols on the right side of the production.

2. Find t he next node at wh ich a subtree is to be consrructed.

For some grammars, the above steps can be implemented during a single leftto-right scan of the input string. The current token being scanned in the input is frequently referred to as the tookaneod symbol. Initially, the Iookahead symbol is the first, i.e .. leftmost. token of the input string. Figure 2.16 illustrates the parsing of the Siring

array [ num dotdol num ] of integer

Initially, the token array is the lookahead symbol and the known part of the

42 A SIMPLE COMPILER

SEC. 2.4

(a) type
type
(b) ~/\~
artay [ simple 1 01 lYpe
rype
~/\~
(c) array [ s;mple ] of type
/1'"
Rum dotdot num
type
~/\~
(d) array E simple 1 of type
/I~ I
num dotdot num .o;imple
type
~/\~
array ( s imple ) of typ~
(c) /I~ I
num dotdot num ,~;",p'e
I
integer
Fig. 2.15. Steps in the top-down construction of a parse tree, parse tree consists of the root, labeled with the starting noncermioal type in Fig. 2.16(a). The objective is to construct the remainder of the parse tree ill such a way that Ihe string generated by the parse tree matches the ill put string.

For a match to occur, nontermmal type in Fig. 2.16(a) must derive a string that starts with the lookahead symbol array. ln grammar (2.8). there is just one production for type that can derive such a string. so we select it. and construct the children of the root labeled with the symbols on the right side of the production.

Each of the three snapshots in Fig. 2.16 has arrows marking the lookahead sym bol in the input and the node in the parse tree that is being considered. When children are constructed at a node, we next consider the leftmost child. In Fig. 2. &6(b). children have just been constructed at the root, .and the leftmost child labeled with array is being considered.

When the node being considered in the parse tree is for a terminal and the

sse. 2.4

PARSING 43

PARSE type
TREE t
(a)
INPUT array ( num dotdot n.m } of integer
t
typt
PARSE ~/\~
TREE array [ simple ] of type
, t
(b)
INPUT array [ num dotdot num ] of' integer
t
type
PARSE ~/\~
TREE array ( .dmpJe ) of type
t
(c)
INPUT arra), ( num dotdot nun. ] or integer
t Fig. 2.1~. Top-down parsing while scanning the input from left to right.

terminal matches the lookahead symbol, then we advance in both the parse Cree and the input. The next token in the input becomes the new lookahead symbol and the next child in the parse tree is considered. In Fig. 2.I6(c). the arrow in the parse tree has advanced to the next child or the root and the arrow in the input has advanced to the next token [. After the next advance, the arrow in the parse tree will point to the child labeled with nonterminal simple. Wilen a node labeled with a nonterminal is considered. we repeat the process of selecting a production for the nonterminal.

In general, the selection of a production for a nonterminal may involve trial-and-error; that is, we may have to try a production and backtrack to try another production if the first is found to be unsu it able > A production is unsuitable if, after using the production, we cannot complete the tree to match the input string. There is an important special case, however, called predictive parsing, in which backtracking does not occur.

44 A SIMPLE COMPILER

SEC. 2.4

Predictive Parsing

Recursive-descent parsing is a top-down method of syntax analysis in which we execute a set of recursive procedures to process the input. A procedure is associated with each nonterminal of a grammar. Here. we consider a special form of recursive-descent parsing, called predictive parsing, in which the lookahead symbol unambiguously determines the procedure selected for each nonterminal. The sequence of procedures called in processing the input implicitly defines a parse free for the input.

procedure motet: (I: token): begin

if lookahead = t then hmk"h('(tJ : = nexttoken else error

end;

procedure type; begin

if lookaheod is. in { integer. char. num ~ then ,~imph

else if /OOkllhf'Cld "" 't' then begin moldl ( , t'); match (id)

end

else jf /oahlh('dd "" array then begin

murd!(array}; matche{'); ,\·im"fe~ mtll~'h(']'); m(It('h(of); type

end

else error

end;

pro«dure simple; begin

If Iookahead i;: integer tben match ( i nteRer)

else if lo(}k(rh(lUJ = thar then m<uch (char)

elst if look"hfOd = numthen begin

mlttch (num); match (dotdol); nwt('h (num) end

else error

end;

Fig. 2_17. Pseudo-code for a predictive parser.

The predictive par~er in Fig. 2.17 consists of procedures (or the nonterminaJs type and simple of grammar (2.8) and an additional procedure match, We

SEC. 2.4

PARSING 45

use match to simplify the code for type and simple; it advances to the next input token if its argument t matches tile look ahead symbol. Thus match changes the variable lookahead, which is the currently scanned input token.

Paning begins with a cal] of the procedure for the starting nonterminal type in our grammar. With the same input as in Fig. 2.16, lookaheod is initially the first token array. Procedure type executes the code

match (array); matcht' l '): simple; match {'} '); match (of); type (2.9)

corresponding 10 the right side of the production

type -+ array [ simple ] or type

Note [nat each terminal in the right side is matched with the lookahead symbol and that each nonterminal in the right side leads to a can of its procedure.

W i(h the in put of Fig. 2. 16, after the tokens array and [ are matched, • he look ahead symbol is Dum. At this point procedure simple is called and the code

match (num); match (dotdot); match (Dum)

in its bod y is executed.

The look ahead symbol guides the selection of the production to' be used. If t he right side of a product ion starts with a token. then the prod uct ion can be used when the look ahead symbol matches the token, Now consider a right side starting with a nonterminal. as in

type - simple

(2.10)

Th is production is used if the lookahead symbol can be generated from simple. For example, during the execution of the code fragment (2.9), suppose the lookahead symbol is integer when control reaches the procedure call type. There is no prod uction for type (hat starts with token integer. However, a production for simple does, so production (2.10) is used by having Iype call procedure simple on lookahead integer.

Predictive parsing relies on information about what first symbols can be generated by the right side of a production. More precisely. let « be the right side of a production for nonterminal A. We define FIRSf(o.) to be the set of tokens that appear as the first symbols of one or more strings generated from ot. If a is f_ or carl generate e, then E. is also in PI RST( ex). 2 For exa mple,

FlRST(simple) - {integer. char •• urn } FIRST( tid) = { t}

FIRST{array ( simple ) of type) = {array}

In practice, many production right sides start with tokens. simplifying the

~ Productions with Ii on the right siuc complicate the determination of the first symbols generated by i.I nonterminal, Fur example, if aontcrminal B can derive the empty !ilring and there is a production A --.. Be, then ihc first symbol !enera1cd by C CaR a~M) be (he fifS~ symbol generated by A. If C can also generate E, then both fIRST(A} and FIR Sf(BC) conta in fi..

46 A SIMPLE COMPILER

SEC. 2.4

construction of FIRST" sets. An algorithm for computing FIRST's is given in Section 4.4.

The FIRST sets must be considered if there are two productions A -+ Cl and A -+ 13. Recursive-descent parsing without backtracking requires FIR ST( a) and FIRsr(~) to be disjoint. The Iookahead symbol can then be used (0 decide which production to use; if the lookahead symbol is in FIRST(o.). th.en a is used. Otherwise, if the lookahead symbol is in FIRsr(~), then ~ is used.

When to Use E .. Productions

Productions with £. on the right side require special treatment. The recursivedescent parser will use an e-production as a default when no other production can be used. For example. consider:

stmt begin opt-stmts end

opt.stmu stmulis: I E

While parsing oputmts, if the lookahead symbol is not in FIRST(stmLJi.tt») then the a-production is used. This choice is exactly right if the tookabead symbol is end. Any lookahead symboJ other than end will result in an error! detected during the parsing of stmt.

Designing a Predictive Parser

A predictive parser is a program consisting of a procedure for every nonterminat. Each procedure does two things,

1. It decides which production to use by looking at the look ahead symbol.

The production witb right side, o is used if the lookahead symbol is in FIRSf(cc). If there is a conflict between two right sides for any lookahead symbol, then we cannot use this parsing method on th is grammar. A production with E on the right side is used if the lookahead symbol is not in the FIRST set for any other right hand side.

2. The procedure uses a production by mimicking the right side. A nontermina 1 results in a call to Ihe procedure for the nonterminal, and a token matching the lookahead symbol results in the next input token being read. If at some point the token in the production does not matcb the lookahead symbol. an error is declared. Figure 2.17 is the result of applying these rules to grammar (2.&).

Just as a translation scheme is formed by extending a grammar. a syntaxdirected translator can be formed by extending a predictive parser. An algorithm for this purpose is given in Section 5.S. The following limited construelion suffices for the present because the translation schemes implemented in this chapter do not associate attributes with nonterminals:

I. Construct a predictive parser. ignoring the actions in productions.

SEC. 2.4

PARSING 47

2. Copy the actions from the translation scheme into the parser. I f an act ion appears after grammar symbol X in production p, then it is copied after the code implementing X, Otherwise, if it appears at the beginning of the production, then it is copied just before the code implementing the production.

We shall construct such a translator in the next section.

Left Recursion

It is possible for a recursive-descent parser to loop forever. A problem arises with left-recursive productions like

expr -+ expr ... term

in which the leftmost symbol on the right side is the same as the nonterminal on the left side of the product ion. Suppose the procedure for expr decides to apply this production, The right side begins with expr so the procedure for expr is called recursively" and the parser loops forever. Note that the lookahead symbol changes only when a terminal in the' right side is matched. Since the production begins with the nonterminal upr> no changes to the input take place between recursive calls, causing the infinite loop.

A
.......
I A R
/' .......
R
<,
»:
A
./ <,
A R
/' 1 I <,
A R
I
(l I a I Cl I a] [} [a I ~ I ... [ cc l E.

(a) (b) Fig. 2.18. Lcfl- and right-rccunivc ways of generating a string.

A left-recursive production can be eliminated by rewriting the offending production. Consider a nonterrninal A with two productions

A - Ao; I 13

where a. and 13 are sequences of terminals and nonterminals that do not start with A. For example, in

expr - expr + term I term

48 A SJMPLE COMPILER

SEC. 2.4

A :;:;: expr, o ~ + term, and J3 = term.

The nonterminal A is left recursive because the production A .... An has A itself as the leftmost symbol on the right side. Repeated application of this production builds up a sequence of CltS to the right of A. as in Fig. 2.18(a). When A is finally replaced by ~, we have a tl followed by a sequence of zero or more a's.

The same effect can be achieved, as in Fig. 2.18(b), by rewriting the productions for A in the following manner.

A 13 R

R a R I E

(2.11)

Here R is a new nonterminal. The production R ..... ct R is right recursive becau se tll is production for R has R itself as the last symbol on the right side. Right-recursive productions lead to trees that grow down towards the right, as in Fig. 2, 18(b). Trees growing down to the right make it harder to translate expressions containing left-associative operators, such as minus. In the next section. however, we shall see that the proper translation of expressions into postfix. notation can still be attained by a careful design of the translation scheme based on a right-recursive grammar.

In Chapter 4. we consider more general forms of left recursion and show how all left recursion can be eliminated from a grammar,

2.S A TRANSLATOR FOR SIMPLE EXPRESSIONS

Using the techniques of the last three sections, we now construct a syntaxdirected translator. in the form of a working C program, that translates arithmetic expressions into postfix form. To keep the initial program manageably small, we start off with expressions consisting of digits separated by plus and minus signs. The langu age is extended in the next two sections to include numbers, identifiers. and other operators. Since expressions appear as a construct in so many languages, it is worth studying their translation in detail.

expr ..... expr + term { pr;ru(,._f) }
expr ..... expr - term { print('-') }
apr -0- term
II!rm .... 0 { prin; ( , 0 ' ) }
term .... 1 { prim (' 1 ' ) ~
term ~ 9 { prillt (' 9') } rig. 2.19. Initial specification of infix-to-postfix translator.

A syntax-directed translation scheme can often serve as the specification for a translator. We use the scheme in Fig. 2.19 (repeated from Fig. 2.13) as the

SEC. 2.5

A TRANSLATOR FOR SIMPLE EXPRESSIONS 49

definition of the translation to be performed. As is often the case, the underlying grammar of a given scheme has to be modified before it can be parsed with a predictive parser. In particelar, the grammar underlying the scheme in Fig. 2.19 is left-recursive, and as we saw in the last section, a predictive parser cannot handle a 'eft-recursive grammar. By eliminating the leftrecursion. we can obtain a grammar suitable for use in a predicrive recursivedescent translator.

Abstract and Concrete Syntax

A useful starting point for thinking about rbe translation of an input string is an abstract syntax tree in which each node represents an operator and the children of the node represent the operands. By contrast. a parse tree is called a concrete syntax tree, and the underlying grammar is caUed a concrete syntax for the language. Abstract syntax trees. or simply syntax trees, differ from parse trees because superficial distinctions of form. unimportant for translation. do not appear in syntax trees.

/'"

2

/'"

9 S

FIg~ 2.10. Syo tax tree for 9 - 5 +- 2 .

For example, the syntax. tree for 9-5+2· is shown in Fig. 2.20. Since + and - have the same precedence level. and operators at the same precedence level are evaluated left to right. the tree shows 9-5 grouped as a subexpression. Comparing Fig. 2.20 with the corresponding parse tree of Fig. 2.2. we note that the syntax tree associates an operator with an interior node. rather titan making the operator be one of the children.

It is desirable for a translation scheme to be based on a grammar whose parse (Tees are as close to syntax trees as possible. The grouping of subexpressions by the grammar in Fig. 2. 19 is similar to their grouping in syntax trees. Unfortunately, the grammar of Fig. 2.]9 is left-recursive, and hence not suitable for predictive parsing. It appears there is a conflict; on the one hand we need a grammar that facilitates parsing, on the other hand we need a radically different grammar for easy translation. The obvious solution is to eliminate the left-recursion. However ~ this must be done carefully as the following example shows.

Example 1.'. The following grammar is unsuitable for translat ing expressions into postfix form, even though it generates exactly the same language as the grammar in Fig. 2.l9 and can be used for recursive-descent parsing.

50 A SIMPLE COMPILER

SEC. 2.5

expr - term rest

rest ... + expr I - expr

term - 0 I 1 I ..' I 9

E

This grammar has the problem that the operands of the operators generated by rest - + expr and rest - - expr are not obvious from the productions. Neither of the following choices for forming the translation rest.' from that of expr. t is acceptable:

rest -t _ expr {resl.t:= ' -' II expr.t } rest -t - expr {rest./:= expr.t I ,_, }

(2.12) (2.13)

(We have only shown the production and semantic action for the minus operator.) The lranslation of 9-5 is 95-. However. if we use the action in (2.12), then the minus sign appears before expr.t and 9-5 incorrectly remains 9-5 in translation.

On the other hand. if we use (2.13) and the analogous rule for plus, the operators consistently move to the right end and 9-5+ 2 is translated incorrectly into 952+~ (the correct translation is 95-2+). 0

Adapting the Translation Scheme

The left-recursion elimination technique sketched in Fig. 2.18 can also be appJied to productions containing semantic actions. We extend the transformarion in Section 5.5 to take synthesized attributes into account. The technique transforms the productions A ~ Au I A 13 I 'Y into

A .... ..,R

R - a.R I ~R ~ E

When semantic actions are embedded in the productions, we carry them along in the transformat ion. Here t jf we let A = exp«, 01 = + term {print ( , .. ,I) }. 13 = _ term {print(,-t)}. and "V = term. the transformation. above produces the translation scheme (2.14). The expr productions in Fig. 2.19 have been transformed into the productions for expr and the new nonterm ina' rest in (2.14). The productions for term are repeated from Fig. 2.19. Notice that the underlying grammar is different from the one in Example 2.9 and the difference makes the desired translation possible.

expr rest term term

-+ term rest

- + term { print('+')

-+ 0 { prim('Ot) }

-+ 1 { prim r 1 ') }

} rest ~ - term { print ( , .. ") } rest I t (2.14)

term -+ 9 { prim('9t) }

Figure 2.21 shows bow 9-5+2 is translated using the above grammar.

SEC. 2.5

A TRANSLATOR FOR SIMPLE EXPRESSIONS 51

expr

/

term

/ \~

9 {pr;nt( '9') }

~

/I '~, _______

term {print('-')}

/ '~, // ~~~

5 {print('S')} + term {prinr('+')} rest

/ '\ I

2 {prinf(' 2'} } t.

rest

Fig.2.21. Translation of 9-5+2 into 95-2+.

Procedures for the Nonterminals expr, term, and rest

We now implement a translator in C using the syntax-directed translation scheme (2. ]4). The essence of the translator IS the C code in Fig. 2.22 for the functions expr. term, and rest. These functions implement the corresponding nonterminals in (2.14).

expr( ) {

term(); rest()t

}

rest( ) {

if (lookahead ~a '+') {

match('+'); term(); putchar(;T'); rest();

}

else if (lookahead •• '_'J (

match('-'); term(); putchar(i_'); rest();

}

else ;

}

term() {

if (isdiqit(lookahead}) { putchar(lciokahead); match(lookahead);

}

else er ror ( ) ;

}

Fig.2.22. Functions for the nonterminals espr; re.~f.1and term.

The function ma.tch. presented later. is the C counterpart of the code m

52 A SIMPLE COMPILER

SEC 2.5

Fig. 2.17 to match a token with the lookahead symbol and advance through the input. Since each token is a sjng)e character in our language. match can be implemented by comparing and reading characters.

For those unfamiliar witb the programming language C. we mention the salient differences between C and other Algol derivatives such as Pascal. as we find uses for those features of C. A program in C consists of a sequence of function definitions. with execution starting at a distinguished function called mai.n. Function definitions cannot be nested. Parentheses enclosing function parameter lists are needed even if there are no parameters; hence we write expz t }, term(), and rest(). Functions communicate either by passing parameters "by value" or by accessing data gJobal (0 all functions. For example, the functions term{) and rest () examine the lookahead symbol using the global identifier lookahead.

C and Pascal use the following symbols for assignments and equality tests:

OPERATION C PMiCl\l
assignment = . -
.-
equality test .:11:= :
inequality test I;; <> The functions fOf 'he nonrerminals mrmrc the right sides of productions.

For example. the production expr - term rest is implemented by the calls term ( } and rest ( ) in the function expr ( ).

As another example, function rest () uses the first production for rest in (2.14) if the lookahead symbol is a plus sign, the second production if the lookahead symbol is a minus sign, and the production rest ~ E by default. The first production for rest is implemented by the first if-statement in Fig. 2.22. If the lookahead symbol is +. the plus sign is matched by the call ma t.c h ( '+ ' }. A fter the call term ( ). the C standard library rou tine putchar ( , + ") implements the semantic action by printing a plus character. Since the third production for rest has e as its right side, the last else in rest ( ) does noth ing.

The ten productions for term generate the ten digits. In Fig. 2.22. (he routine isdigi t tests if the lookahead symbol is a digit. The digit is primed and matched if the test succeeds; otherwise, an error occurs. (Note that match changes the lookahead symbol, so the printing must occur before the digit is rnatched.) Before showing a complete program, we shall make one speed-improving transformation to the code in Fig. 2.22.

Optimizing the Translator

Certain recursive calls can be replaced by iterations. When the las. statement executed in a procedure body is a recursive call of the same procedure. the can is said to be tail recur.~·iVt'. For example. the caUs of rest () at (he end .f the fourth and seventh lines of the function rest () are tail recursive

SEC. 2.5

A TRANSLATOR FOR SIMPLE EXPRESSIONS 53

because control flows to the end of the function body after each of these calk

We can speed up a program by replacing tail recursion by iteration. For a procedure without parameters. a tail-recursive call can be simply replaced by a jump tothe beginning of [he procedure. The code for rest can be rewritten as:

rest ( ) {

L: if (lookahead == '+;)" {

match{'+'); terro(); putchar(;+'); goto L;

}

else if (lookahead == ~-~) {

match(;-;). term(); putchar(~-'); goto L;

} else

}

As long as the lookahead symbol IS a plus or minus sign, procedure rest matches the sign, calls term to match a digit, and repeats the process. Note that since match removes the sign each time it is called. this cycle occurs only on alternating sequences of signs and digits. If this change is made in Fig. 2.22. the only remaining call of rest is from expr (see line 3). The two functions can therefore be integrated into one, as shown in Fig. 2.23. In C. a statement stmt can be repeatedly executed by writing

while (1) stmt

because the condition 1 is always true. We can exit from a loop by execunng a break-statement. The stylized form of the code in Fig. 2.23 allows other operators to be added conveniently.

expr( ) {

term( ) i while ( 1 }

if llookahead a: '+') {

matchl'~'); ter~(lj putchar('+');

else if (lookahead == '_'. { match('~/); term(); putchar('-');

}

else break.

Fia. 2.13. Replacement for functions expr and rest of Fig. 2.22.

54 A SlMPLE COMPILER

SEC. 2.6

The Complete Program

The complete C program for our translator is shown in Fig. 2.24. The first tine, beginning with #inelude t loads <ctype. h>, a file of standard routines that contains the code for the predicate isdigi t,

Tokens. consisting of single characters. are supplied by the standard library routine getchar that reads the next character from the input file. However. lookahead is declared to be an integer on line 2 of Fig. 2.24 to anticipate the additional tokens that are not single characters that will be introduced in later sections, Since lookahead is declared outside any of the functions. it is global to any functions that are defined after line 2 of Fig. 2.24.

The function match checks tokens: it reads the next input token if the lookahead symbol is matched and calls the error routine otherwise.

The function error uses the standard library function printf to print the message "synt.ax error" and then terminates execution by the call exi t ( 1 ) to another standard library function.

2.6 LEXICAL ANALYSIS

We shall now add 10 the translator of the previous section a lexical analyzer that reads and con verts the input into a stream of tokens to be analyzed by the parser. Recall from the definition of a grammar in Section 2.2 that the sentences of a language consist of strings of tokens. A sequence of input characters that comprises a single token is called a lexeme. A lexical analyzer can insulate a parser from the lexeme representation of tokens. We begin by listing some of the functions we might want a lexical analyzer to perform.

Removal of White Space and Comments

The expression translator in the last section sees every character in the input. so extraneous characters, such as blanks. will cause it to fail. Many languages allow "white space" (blanks, tabs. and newlines) to appear between tokens. Comments can likewise be ignored by the parser and translator, so they may also be treated as wh ite space.

If white space is eliminated by the lexical analyzer, the parser will never have to con sider it. The alternative of modifying the grammar to incorporate white space into the syntax is not nearly as easy to implement.

Constants

Anytime a single digit appears in an expression. it seems reasonable to allow an arbitrary integer constant \n its place. Since an integer constant is a sequence of digits. integer constants can be allowed either ~y adding productions to the grammar for expressions, or b)l creating a token for such constants. The job of collecting digits into integers is generally given to a lexical analyzer because numbers can be treated as single units during translation.

Let num be the token representing an integer. When a sequence of digits

SEC. 2.6

LEXICAL ANALYSJS 55

#include <ctype.h> /. loads file with predicate isdigit ./ int lookabead.;

main( ) {

lookahead • getohar(); expr{ ),

putchar(~'n'); /. adds trailinq newline character ./

expr{ ) {

term( ); while(1)

if (lookahead ., '+1) {

match{'+' J; term(}; putohar('.'J;

}

else if tlookahead •• '-~) ( match('-'); term(); putchar"-');

}

else :preak;

term( ) {

if (isdigit(lookahead)> { putchartlookahead); match(lookah~ad)t

}

else error ( );

Jlatch( t ) int t;

{

if (lookahead ~= t) lookahead : getchar(); else error( J;

}

error( » (

printf("syntax error\nM); /. print error .essage */

exit(1); /. then halt *1

}

Fla. 2+Z4. C program to translate an infix expression iaro postfix form.

56 A SJMPLE COMPILER

SEC. 2.6

appears in the input stream, the lexical analyzer will pass num to the parser. The value of the integer will be passed along as an attribute of the token num, Logically, the lexical analyzer passes both the token and the attribute to the parser. If we write a token and its attribute as a tuple enclosed between < >. the input

31 + 28 + 59

is transformed into the sequence of tuples

<num, 3.> <+, > <RUin, 28> <+. > <Bum, 59>

The token + has no attribute. The second components of the tuples. the attributes. play no role during parsing, but are needed during translation.

Recognizing Identifiers and Keywords

Languages use identifiers as names of variables, arrays, functions. and the like. A grammar for a language often treats an identifier as a token. A parser based on such a grammar wants to see the same token, say id. each time an identifier appears in the input. For example, the input

count = count + increment;

(2.15)

would be converted by the lexical analyzer into the token stream HI = id + id ;

(2.16)

This token scream is used for parsing.

When talking about the teucal analysis of the input line (2.15). it is. useful to distinguish between the token id and th.e lexemes count and increment associated with instances of th is token. The translator needs to know that the lexeme count forms the first two instances of id in (2.16) and that the lexerne increment forms the third instance of Id.

When a lexeme forming an identifier is seen in the input. some mechanism is needed to determine if the lexeme has been seen before. As mentioned in Chapter I, a symbol table is used as such a mechanism. The lexerne is stored in the symbol table and a pointer to this symbol-table entry becomes an attribute of the token Jd.

Many languages use fixed character strings such as begin~ end, if. and so on, as punctuation marks or to identify certain constructs. These character strings. called keywords, generally satisfy the rules for forming identifiers, so a mechan ism is needed for deciding when a lexeme forms a keyword and when it forms an identifier. The - problem is easier to resolve if keywords are reserved. i.e., if they cannot be used as identifiers. Then a character string forms an identifier only if it is not a keyword.

The problem of isolatin g tokens also arises if the same characters appear in the lexemes of more than one token, as in -c , <=, and <> in Pascal. Techniques for recognizing such tokens efficiently are discussed in Chapter 3.

SEC. 2.6

LEXICAL ANALYSiS 57

Interface to the Lexkal Analyzer

When a lexical analyzer is; inserted between the parser and the input stream, it interacts with the two in the manner shown in Fig. 2.25. It reads characters from the input, groups tbem into lexemes, and passes the tokens formed by the lexemes, together with their attribute values, to the later stages of the compiler. In some situations. the lexical analyzer has 10 read some characters ahead before it can decide on the token to be returned to the parser. For example, a lexical analyzer for Pascal must read ahead after it sees the character >. If the next character is a. tben the character sequence > = is the lexerne forming the token for the "greater than or equaJ to" operator. Otherwise » is the lexeme forming the "greater than" operator. and the lexical analyzer has read one character too many. The extra character bas to be pushed back onto the input. because it can be the beginning of the next lexeme in the input.

read

pass

t k d

~ character o en an
2 lexical its attribates
analyzer parlier
- push back Character

Fig. Z.ZS. Inserting a lexical analyzer between tnc input and the parser.

The lexical analyzer and parser form a producer-consumer pair. The lex.ical analyzer produces tokens and the parser consumes them. Produced tokens can be held in a tok.en buffer unt it they are consumed. The interaction between the two is constrained only by the size of the buffer. because the lexical analyzer cannot proceed when the buffer is fun and the parser can not proceed when the buffer is empty. Commonly, the buffer holds just one token. In this case, the interaction can be implemented simply by making the lexical analyzer be a procedure caned by the parser. returning tokens on demand.

The implementation of reading and pushing back characters is usuaUy done by sett ing up an input bu ffer. A bJock of characters is read into [he buffer at a time; a pointer keeps track of the portion of the input that has been analyzed. Pushing back a character is implemented by moving back the pointer. Input characters may also need to be saved for error repornng, since some indication has to be given of where in the input text the error occurred. The buffering of input characters can be justified on efficiency grounds alone. Fetching a block of characters is usually more efficient than fetching one character at a time. Techniques for input buffering are discussed in Section 3.2.

58 A SIMPLE COMPILER

SEC. 2.6

A Lexical Analyzer

We now construct a rud imentary lexical analyzer for the expression translator of Seenon 2.5. The purpose of the lexical analyzer is to allow white space and numbers to appear within expressions. In the next section, we extend the lexica) analyzer to allow identifiers as wei L

USCs getchar ( ) to read character

returns token
lexan( ) to caller
lexical
analyzer
)

I tokenvall sets global variable to attribute value

pushes back c using UDqetc(c,stdin)

Fig. 1.26. Implementing the interactions in Fig> 2.25,

Figure 2.26 suggests how the lexical analyzer. written as the function lexan in C, implements the interactions in Fig. 2.25. The routines getchar and ungetc from the standard include-file <stdio, h> take care of ill put buffering: lexan reads and pushes back input characters by calling the routines getchar and ungetc, respectively. With c declared to be a character, the pair of statements

c ~ getchar(); unqetc(c, stdin);

leaves the input stream undisturbed. The call of getchar assigns the next input character to e: the caU of ungetc pushes back the value of c onto the standard input stdin.

J f· the implementat ion language does not allow data structures to be returned from functions. then tokens and their attributes have to be passed separately. The function lexan returns an integer encoding of a token. The token for a character can be any conventional integer encoding of that character. A token, such as n.m. can then be encoded by an integer larger m an any integer encoding a character, say 256. To allow the encoding to be changed easily, we use a symbolic constant NOM to refer to the integer encoding of num. In Pascal, the association between NUM and the encoding can be done by a const declaration; in C. NUM can be made to stand for 256 using a defi ne-statement:

#define NUM 256

The function 1 exan returns NOM when a sequence of digits is seen in the input. A global variable token"1fal is set to the value of the sequence of digits. Thus, if a 7 is followed im med iately by a 6 In the inpu I. tokenva 1 is assigned the integer value 76.

SEC. 2.6

Allowing numbers within expressions requires a change 1[1 the grammar in Fig. 2.19. We replace the individual digits by the nonterminal factor and introduce the foilowing productions and semantic actions:

factor -+ I expr )

I .um { prinJ(num.vaiue) }

The C code for factor in Fig. 2.27 is a direct implementation of the produclions above. When looka.head equals NUM, the value of attribute num.lIalue is given by the global variable tokenval. The action of printing this value is done by the standard library function printf. The first argument of printf is a string between double quotes specifying the format to be used for printing the remaining arguments; Where %d appears in the string, the decimal representation of the next argument is printed. Thus, the printf statement in Fig. 2.27 prints a blank followed by the decimal representation of tokenval followed by another bJank.

factQr( ) {

if (lookahead ae '(~) {

match('('); expr(); match{')');

else if (looxahead •• NUM) t

Pl;" int f ( " %d ", tokenval); match ( NOM) ;

}

else errori);

}

F\:. 2.17. C code for factor when operands can be numbers.

The implementation of function lexan is shown in Fig. 2.28. Every time the body of the while statement on lines 8·28 is executed, a character is read into t on line 9. If the character is a blank or a tab (written " t ~) .• hen no token is returned to the parser; we merely go around the while loop again. If the character is 3 newline (written "vn "), then a global variable lineno is incremented. thereby keeping track of line bU mbers in the input. but again no token is returned. Supplying a line number with an error message helps pinpoint errors.

The code for reading a sequence of digits is on lines '4-23. The predicate isdigi t (t) from the include-file <ctype. h> is used on lines 14 and 17 to determine if an incoming character t is a digit. If it is. then its integer value is given by the expression t - ; 0 r in both ASCII and ESC Die. With other character sets, the conversion may need to be done differently. In Section 2.9. we incorporate this lexical analyzer into our expression translator.

60 A SIMPLE COMPILER

SEC. 2.6

(I) (2) (3) (4)

(5) (6) (7) ( 8) (9)

( 10) no

(12)

( 13)

( 14) (15)

( 16) (17)

( 18)

( 19) (20)

(21 } (22) (23, (24) (25) (26) (27) (28) (29) }

#inclu~e <stdio.h>
#inelucle <ctype.h>
int lineno • 1 i
int tokenval = NONE;
int lexan( )
{
int t;
while(1) { t =- 9'etchar ( ) ;

if (t ;;::; ~ ; :: t ;;a ~ \t; )

/4 strip out blanks and tabs * / else if (t ::;::; ~\n')

lineno = lineno + 1;

else if (isdigit(t)) { tokenval = t - ~O'; t .9'etchar();

while (isdiqit(t» {

tokenval = tokenval*1D + t-~O'; t R getchar();

ungetc(t, stdin); return NUM;

}

else {

tokenval a NONE; return ti

}

}

Fig. 2.28. C code for lexical analyzer eliminating white space and collect ing numbers.

2.7 INCORPORATING A SYMBOL TABLE

A data structure called a symbol table is generally used to store information about various source language constructs. The information is collected by the analysis phases of the compiler and used by the synthesis phases to generate the target code. For example. during lexical analysis, the character string, or lexeme, forming an identifier is saved in a symbol-table entry. Later phases of the compiler might add to this entry information such as the type of the identifier, its usage (e.g., procedure. variable, or label), and its posit ion in storage. The code generation phase would then use the information to generate the proper code to store aod access this variable. In Section 7.6. we discuss the implementation and use of symbol tables in detail. In this section, we

SEC. 2.7

INCORPORATINQ A SYMBOL TABLE 61

illustrate how the lexical analyzer of the previous section might interact with a symbol table.

The Symbol-Table Int.erlaee

The symbol-table routines are concerned primarily with saving and retrieving lexemes. When a lexeme is saved, we also save the token associated with the lexeme, The following operations will be performed on the symbol table.

insert(st t}: Returns index of new entry for string s, token t. lookup ( s}; Returns index of the entry for string s ,

or 0 if s is not found.

The lexical analyzer uses the lookup operation to determine whether there is an entry for a lexeme in the symbol table. If no entry exists, then it uses the insert operation to create one. We shall discuss an implementation in which the lexical analyzer and parser both know about the format of symbol-table entries.

Handliag Reserved. Keywords

The symbol-table routines above can handle any collection of reserved keywords. For example, consider tokens div and mod with lexemes div arid mod. respectively. We can initialize the symbol table using the calls

insert' "div" ~ div); insert ( "mod 11., mod);

Any subsequent call lookup { Itdiv") returns the token div, so div cannot be used as an identifier.

Any collection of reserved keywords can be handled in this way by appropriately initializing the symbol table.

A Symbot- Table Implementation

The data structure for a particular implementation of a symbol table is sketched in Fig. 2.29. We do not wish to set aside a fixed amount of space to hold lexemes forming identifiers; a fixed amount of space may not be large enough to hold a very long identifier and may be wastefully large for a short identifier, such as L In Fig. 2.29, a separate array lexemes holds the character string forming an identifier. The string is terminated by an end-of-srring character I denoted by EOSt that may Rot appear in identifiers. Each entry in the symbol-table array symtable is a record consisting of two fields, lexptr, pointing to the beginning of a lexeme, and token. Additional fields can hold attribute values, although we shan nOI do so here.

In Fig. 2.29, the Oth entry is 'eft empty, because Lookup returns 0 to indicate that there is no entry for a string. The I st and 2nd entries are for [he keywords div and mod, The 3rd and 4th entries are for identifiers count and i.

62 A SIMPLE COMPILER

SEC. 2.7

ARRAY symtable

artrjbutes

-------'0

1------+--d=I"--+ - - - - - - - -I I

----~-------j~ mod - - - - - - - -l 2

,....._,~---+-~-::-ld=----+ - - - - - - - -l 3

~-r-- id ---~~~~~4

~~~__t----+ - - - - - - - ~

lex.ptr

token

F"1g. 2.19. Symbol table and array for storing strings.

Pseudo-code for a lexical analyzer that handles identifiers is shown in Pig. 2.30; a C implementation appears in Section 2.9. White space and integer constants are handled by the lexical analyzer in the same manner as in Fig. 2.28 in the last section.

When our present lexical analyzer reads a letter, it starts saving letters and digits in a buffer lexbuf. The string collected in lexbuf is then looked up in the symbol table, using the lookup operation. Since the symbol table is initialized with entries for the keywords di v and mod, as shown in Fig. 2.29 t the lookup operation will find these entries if lexbuf contains either di v or mod. If there is no entry for the string in lexbuf, i.e., lookup returns 0, then lexbuf contains a lexeme for a new identifier. An entry for the new identifier is created using insert. After the insertion is made, p is the index of the symbol-table entry for the string in lexbuf. This index is communicated to the parser by setting tokenval to p, and the token in the token field of the entry is returned.

The default action is to return the integer encoding of the character as a token. Since the single character tokens here have no attributes, tokenval is set to NONE.

2.8 ABSTRACT STACK MACHINES

The front end of a compiler constructs an intermediate representation of the source program from which the back end generates the target program. One popular form of intermediate representation is code for an abstract stack machine. As mentioned in Chapter 1. partitioning a compiler into a front end and a back end makes it easier to modify a compiler to run on a new machine.

In this section, we present an abstract stack machine and show bow code

SEC. 2.8

ABSTRACT MACHINES 63

funetion texan: integ«;

val' lexb#4: array lO .. IOO] 01 ch.r;

c: char;

begln

loot bt&ln

read a character into c;

ir c is a blank or a tab then do nothing

ellie if c is a newline thea tineno : = lineno + 1 else it c is a digit then begin

set k>kenval to 'he value of this and following digits; retum HUM

I

end

.. if c is a Jetter then lM8:in

place c and successive letters and digits into le.rbuJ; p r= lookup (lexbuJ);

if p = 0 then

P : = inserr(lexbuj. 10); tokenvtll := tr;

retdl'n the loken field of table entry p

end

else begi. / * token is a single character ./

set wkeRllallo NONE; ! * there is no attribute • / retbrn integer encoding of character c

end

Fi&. 2.30. Pseudo-code for a Ini~ll an.alyzer.

$

(

can be generated for it. The machine has separate instruction and data memories and all arithmetic operations are performed on values on a stack. The instructions are quite limited and fall into three classes: integer arithmetic, stack manipuh'tion. and control flow. Figure 2.31 illustrates the machine. The pointer pc indicates the instruction we are about to execute. The meanings of the- instructions shown will be discussed shortly.

Arithmetic IBStructions

The abstract machine must impiement each operator in the intermediate language. A basic operation. such as addition or subtraction, is supported directly by the abstract machine. A more complex operation, however, may need to be implemented as a sequence of abstract machine instructions. We simplify the description of the machine by assuming that there is an

64 A SIMPLE! COMPILER

SEC. 2.8

INSTRUCTIONS STACK DATil.
1 push 5 ,~~ == ~16 0
2 rvalue 2 1 11 2
3 ... 7 ~
4 rva.lue 3 ! 4
;5 • pc
6 Fig. 2.31. Snapsho( of the slack machine after (he flrst four instructions are executed.

instruction for each arithmetic operator.

The abstract machine code for an arithmetic expression simulates the evaluation of a postfix representation for that expression using a stack. The evaluation proceeds by processing the postfix representation from left to right, pushing each operand onto the stack as it is encountered. When a k~ary operator is. encountered, its leftmost argument is k - I positions below the top of the stack and us rightmost argument is at the top. The evaluation applies the operator to the top k values on the stack, pops the operands, and pushes the result onto the stack. For example, in the evaluation of the postfix expression 1 3 + 5 *, the following actions are performed:·

1. Stack I.

2. Stack 3.

3. Add the two topmost elements. pop them, and stack the result 4.

4. Stack 5.

5. Multiply the tW9 topmost elements, pop them. and stack the result 20.

The value on top of the slack at the end (here 20) is the value of the entire expression.

In the intermediate language, all values win be integers. with 0 corresponding to false and nonzero integers corresponding to true. The boolean operators and and or require both their arguments to be evaluated.

L·values and R-values

There is a distinction between the meaning of identifiers on the left and right sides of an assignment. In each of the assignments

i ::= 5;

i :- i + 1.

the right side specifies an integer value, while the left side specifies where the value is to be stored. Similarly, if p and q are pointers to characters. and

pt :a qt;

SEC. 2,8

ABSTRACT MACHINES 65

the right side q t specifies a character. wh ile p t specifies where the character is to be stored. The terms l-value and r-value refer to values that are appropriate on the left and right sides of an assignment. respectively. That is. r-values are what we usually think. of as "values," while I-values are locations.

Stack Manipulation

Besides the obvious instruction for pushing an integer constant onto [he slack and popping a value from the top of the stack, there are instructions to access data memory;

push v
rvalue I
Ivalue [
pop
:31:
copy push v onto the stack

push contents of data location I

push address of data location [ throwaway value on top of the stack

the r-value on top is placed in the l-value below it and both are popped

push a copy of the top value on the stack

Transladon of Expressions

Code to evaluate an expression on a stack machine is closely related to postfix notation for that expression. By definition, the postfix form of expression E + F is the concatenation of the postfix. form of E, the postfix form of F. and +. Similarly, stack-machine code to evaluate E + F is the concatenation of the code to evaluate E. the code to evaluate F. and the instruction to add their values. The translation of expressions into stack-machine code can therefore be done by adapting the translators in Sections 2.6 and 2,7.

Here we generate stack code for expressions in which data locations are addressed symbolically. (The allocation of data locations for identifiers is discussed in Chapter 7,) The expression a+b translates into:

r"alue a

rvalue b

+

In words: push the contents of the data locations for a and b onto the stack; then pop the top two values on the stack. add them, and push the result onto the stack.

The translation of assignments into stack-machine code is done as follows: the I-value of the Identifier assigned to is pushed onto the stack. the expression is evaluated, and its r-value is assigned to the identifier. For example, the assignment

day :~ (1461*y) div 4 + (153*m + 2) div 5 + d translates into the code in Fig. 2.32.

(2.11)

66 A SIMPLE COMPILER

SEC. 2.8

Ivalue day push 2
push 1461 of-
rvalue y push 5
* div
push 4 +
div rvalue d
push 153 -+-
rvalue m :e
* Fig. 2.31. Translation of day; -= ( 1461"y) di"of 4 ... ( 153*m -+- 2) di v 5 -+- d_

These remarks can be expressed formally as follows. Each nonterminal has an attribute I giving lts translation. Attribute lexeme of id gives the string representation of the ident ifier _

stmt -+ hi : = expr

{ stnu.t := tlvalue' II id.lexeme ij expr.t lit:.' }

Control Flow

The stack. machine executes instructions in numerical sequence unless told to do otherwise by a conditional or unconditional jump statement. Several options exist for specifying the targets of jumps:

I. The instruction operand gives the target location.

2. The instruction operand specifies the relative distance. positive or nega-

tive, to be jumped.

3. The target is specified symbolically; i.e .• the machine supports labels.

With the first two options there is the additional possibility of taking the operand from the top of the stack _

We choose the third option for the abstract machine because it is easier to generate such jumps. Moreover. symbolic addresses need not be changed if, after we generate code for the abstract machine, we make certain improvements in the code that result in the insertion or de1etion of instructions.

The control-flow instructions for the stack machine are:

labe 1 I target o-f jumps to l; has no other effect

goto I next instruction is taken from statement with label I

gofalse I pop the top value: jump if it is zero

got:tue I pop the top value; jump if it is nonzero

hal t stop execution

SEC. 2.8

ABSTRACT MACHINES 67

Translation of Statements

The layout in Fig. 2.33 sketches the abstract-machine code for conditional and while statements. The following discussion concentrates on creating labels.

Consider the code layout for if-statements in Fig. 2.33. There can only be one label out instruction in the translation of a source program; otherwise. there will be confusion about where control flows to from a qoto out statement. We therefore need some mechanism for consistently replacing out in [he code layout by a unique label every time an if-statement is translated.

Suppose newlabel is a procedure that returns a fresh label every time it is called. I,. the following semantic action, the label returned by a can of newlabel is recorded using a local variable out:

SImi -.. if expr then stm: I

{ out: = newlaoel :

stmt.t := expr.t II 'qofalse' out II SlmI,.t I 'la.bel' out}

(2.18)

IF

WHILE

label out

label test
code for expr
gofalse out
code for stmt.
goto test
label out code for expr

gofalse out

code for sImI.

Fig. 2.33. Code layout for conditional and while statements.

Emitting a Translation

The expression translators in Section 2.5 used print statements to incrementally generate the translation of an expression. Similar print statements can be used to emit the translation of statements. Instead of print statements, we use a procedure emil to hide printing details. For example, emit can worry about whether each abstract-machine instruction needs to be on a separate line. U sing the procedure emit. we can write the following instead of (2. 18):

stmt - if

expr then

{ out := newtabet; emit ("gofa.lse' , QUI);}

stnu ,

{ emit ( , label', Qut); }

When semantic actions appear within a production, we consider the elements

68 A SIMPLE COMPILER

SEC. 2.8

on the right side of the production in a left-to-right order. For the above product ion • the order of actions is as follows: actions during the parsing of expr are done, out is set to tne label returned by new/abel and the qofalse in struet ion is emitted I actions during the parsi ng of stmi , are done. and. finally, the label instruction is emitted. Assuming the actions during the parsing of expr and stnu, emit the code for these nonterminals, the above production implements the code layout of Fig. 2.33.

procedun stmt;

var test, out: integer; I.. for labels .. I begin

ir looknhead = id then begin

emil (' 1 val ue '. tokenval): /'tklt('h (ld); n1(lf('h (' ~,.'); expr end

else if /rWiwM{ld ::: 'if' then begin mlrl('h (' if');

expr ;

out ;= newlClbel;

emil ('qofalse' . oul); mat('h (' then'):

stmt;

ttmil (' labeY. Ollt) end

/. code for tcmail\ing statements goes here *1 else error ~

end

FIg. 2.34. Pseudo-code for translating statements.

Pseudo-code for translating assignment and conditional statements is shown in Fig. 2.34. Since variable out is local to procedure stmt, its value is not affected by Ihe calls to procedures expr and stmt, The generation of labels requires some thought. Suppose that the labels in the translation are of the form L 1, L2, . .. . The pseudo-code manlpulates such labels using the integer following L. Th us, out is declared to be an integer. newlabel retu rns an integer that becomes the value of out, and emit must be written to print a label given an integer.

Tbe code layout for while statements in Fig. 2.33 call be converted into code in a similar fashion. The translation of a sequence of statements is simply the concatenation of the statements in the sequence. and is left to the reader.

The translation of most single-entry single-exit constructs \8 similar to that of while statements. We illustrate by considering control flow in expressions.

Example 2.10. The lexical analyzer in Sect ion 2.7 contains a cond itional of

SEC. 2.9

PU1TING THE TECHNIQUES TOGETHER 69

the form:

if t = blank or t = tab then

1ft is a blank. then clearly it is not necessary to test if t is a tab. becau se the first equality implies that the condition is true. The expression

expr I or expr 2

can therefore be implemented as

if expr I then true else expr2

The reader can verify that the following code implements th.e or operator: code for apr I copy

gotrue out pop

code for expr2 label out

/-J. pop value of expr , *1

Recall that the gotrue and 90false instructions pop the value on top of the stack to simplify code generation for conditional and while statements. By copying the value of expr I we ensure that the value on top of the stack is true if the gotrue instruction leads to a jump. 0

2.9 PUTTING THE TECHNIQUES TOGETHER

In this chapter. we have presented a number of syntax-directed techniques for constructing a compiler front end. To summarize these techniques. in this section we PU( together a C program thai functions as an infix-to-postfix translator for a language consisting of sequences of expressions termmared by semicolons. The expressions consist of numbers, identifiers, and the operators +, -. ... /. di v, and mod. The output of the translator is a posrfi x representation for each expression. Th.e translator is an extension of the programs developed in Sections 2.5-2.7. A listing of the complete C program is given at the end of this section.

DeKription of the Translator

The translator is designed using the syntax-directed translation scheme in Fig. 2.35. The token id represents a nonempty sequence of letters and digits begin ning with a letter, Dum a sequence of digits, and eor an end-of-file character. Tokens are separated by sequences of blanks, tabs, and newlines ("white space"). The attribute lexeme of the token id gives the character string forming the token; the attribute value of the token num gives the integer represented by the Dum.

The code for the translator is arranged into seven modules. each stored in a separate file. Execution begins in the module ma.in. e that consists of a can

70 A SIMPLE COMPILER

SEC. 2.9

start .... USI eof

hst - e:lpr list

I E

expr .... apr + tam I expr - term I term

term - term • la('tor term I factor term dl" [actor term mod JQ~·tOT fiu>lOr

[actor - ( expr

lid

I num

{ prin/(' + ') ~ { prim(' _') }

{ printC .') } { prim(' /') }

{ print ( I DIV') } { pritrt(' MOD') }

( prinl(id.lexeme) } { prinl(num.value) }

Fig. 1.35. Specification for infix-to-pestfjx translator.

infi" expressions

l
init.c / lex.r.c
J t
symbol.c ~ parse:t'.c I-- error.c
\ ~
e.ittel:.c
~ postfix expressions

Fig. 1.36. ModlJles of infix-to-postfix translator.

to in it ( ) for in itialjzalion followed by a call to par Be ( ) for the translation. The remaining six modules are shown in Fig. 2.36. There is also a global header fiJe global.h that contains definitions common to more than one module; the first statement ill every module

#include "global.b"

causes this header file to be included as part of the module. Before showing the code for the translator. we briefly describe each module and how it was constructed.

SEC. 2.9

PUTT1NG THE TECHN1QUES TOGETHER 71

The Le~iaal Analysis Module lexer . C

The lexical analyzer is a routine called lexan{ } that is called by the parser to find tokens. Implemented from the pseudo-code in Fig. 2.30, the routine reads the input one character at a time and returns to the parser the token it found. The value of the attribute associated with the token is assigned to a global variable tokenval.

The tonowing tokens are expected by the parser:

+ - * / DIV MOD ( ) ID NOM DONE

Here ID represents an identifier, HUM a number. and DONE the end-of-file character. White space is silently stripped out by the lexical analyzer. The table in Fig. 2.37 shows the token and attribute value produced by tbe lexical analyzer for each source language lexeme.

LeXEME TOKEN ATTRIBUTE VALUE

white space " .

sequence of digits NOM numeric value of sequence

div............................ OIV

mod ...... <..................... MOD

other sequences of a letter

then letters and di,its ID index into s}r.II'Itable

end-of-file character DONE

any other character........ that character NONE

Fig. 1.37. Description of tokens.

The lexical analyzer uses the symbol-table routine lookup to determine whether an identifier lexeme nas been previously seen and the routine insert to store a new lexeme into the symbol table. It also increments a global variable lineno every time it sees a newline character.

The Parser Module parser. e

The parser is constructed using the techniques of Section 2.5. We first eliminate left-recursion from the translation scheme of Fig. 2.35 so that the underlying grammar can be parsed with a recursive-descent parser. The

transformed scheme is shown in Fig. 2.38. _

We then construct functions for the nonterminals expr, term, and factor as we did in Fig. 2.24. The function parse( ) implements Ihe start 'symbol of the grammar: it calls lexan whenever it needs a new token. The parser uses the function emit to generate the output and the function error to report a syntax error.

72 A SIMPLE COMPILER

SEC. 2.9'

starr USf eo'

hst expr ; lis:

I E.

txpr term moreterms

more terms ... term { prim ( • .. ') } moreterms

I - term { print (' -') } moreterms I E,

term - jac·tor more/actors

morefactors ...... * factor 1 print (. * J) } morefactors j / factor { pr;nt('/') } morefactor«

I div factor { prim ('DIVI) } more/actors

I mod factor { prinr('MOD') } morefQ(·tors I E

j(J("tor .... ( expr 1

I id { print(ld.le.ume) }

I num { print (nom. Wllue) }

Fig. 2.38. Syntax-directed. translation scheme after eliminating left-recursion.

The Emitter Module emitter. c

Tbe emitter module consists of a single function emit (t, tval) that generates the output for token t with attribute value tval.

The Symbol .. Table Modules symbol. c and ini t . c

The symbol-table mod ule symbol. c implements the data structure shown in Fig. 2.29 of Section 2.7. The entries in the array sym.table are pairs consisting of a pointer to the 1 exemes array and an integer denoting the token stored there. The operation insert (s , t) returns the syrntable index for the lexeme s forming the token t. The function lookup ( s) returns the index of the entry in syrntable for the lexeme s or 0 if s is not there.

The module ini t . c is used to preload syrntable with keywords. The lexeme and token representations for all the keywords are stored in the array keywords, which has the same type as the symtable array. The function init<) goes sequentially through the keyword array, using the function insert to put the keywords in the symbol table. This arrangement allows us to change the representation of rhe tokens for keywords in a convenient way.

The Error Module error. e

The error moduJe manages the error reporting. which is extremely primitive. On encountering a syntax error. the compiler prints a message saying that an error has occurred on the current input I ine and then halts. A better error recovery technique might skip to the next semicolon and continue parsing; the

SEC. 2.9

PUTTlNO THE TECHNIQUES TOGETHER 73

reader is encouraged to make this modification to the translator. More sophisticated error recovery techniques are presented in Chapter 4.

Creating the Compiler

The code for the modules appears in seven files: lexer. c. parser. c. emitter. c, symbol. c. init. c, error. e, and main.e. The file main. e contains the main routine in the C program that calls init ( ~, then

paX'se ( ) , and upon successful completion exit ( O).

Under the UNIX operating system, the compiler can be created by executing the command

cc lexer.c pa.:t'Sitr.o emitte-r.c liIymbol.c init.~ error.c main.e

or by separately compiling the files. using

cc -c filename. c

and linking the resulting filename .0 files:

co lexer.o parser.o emitter.o symbol.o init.o error.o aain.o

The cc com mand creates a file a. out that contains the translator. The transtaror can then be exercised by typing a • out followed by the expressions to be translated; e. g .•

2+3 .. 5;

12 div 5 mod 2;

or whatever other expressions you like- Try it.

The Listing

Here is a listing of the C program implementing the translator. Si'Jown is the global header file qlobal. h. followed by tile seven source files. For clarlty, (he program has been written in an eJementary C style.

#include <stdio.h>
#include <ctype.h>
Idefine BSIZE 128
'define NONE -1
Idefine EOS ",0'
"define NOM 256
Idefine DIV 257
Idefine MOD 258
,define 1D 259
Idefine DONE 260
int tokenva.l; I. /. load ilo routine. -I

1* load character test routines *1

1* buffer size *1

v.lue of token attribute ./

74 A SIMPLE COMPlLER

SEC. 2.9

int lineno;

struct entry {

1* form of symbol table entry ./

char *lexptr; int token;

L

struet entry symtable{];

lexer.c

#inelude Wqlobal.h~ char lexbuf{BSIZE1; int lineno = 1;

int tokenval. NONE;

int lex~n() {

1* lexical analyzer ./

int t; while{1}

t =:- getchar();

iflt-:" ::t,..~\t')

/* strip out white space */ else if (t .= '\n;)

lineno a lineno + 1;

else if (isdiqit(t)) ~ /* t is a di~it ./

ungetc(t, stdinl;

seanf(~%d-; &tokenval);

return HUM;

}

else if (isalpha(t) { int p~ b = 0;

while (isalnum(t)) ( lexbuf[b} .. t; t • geotchar ( } ;

b = b + 1;

if (b >: BSIZE) error("compiler error~);

/. t is a letter */

J* t is alphanumeric ./

lexbuf(b] :s- lOBi

if (t !e EOF)

unqetc(t, stdin); p = lookup(lexbuf);

if (p a. 0)

p = insert(lezbuf, ID); tokenval • p;

return symtable[p].token;

}

else if (t S~ EOF) return DONE;

SEC. 2.9

PUTIING THE TECHNIQUES TOGETHER 75

else {

tokenval = NONE; return t;

}

}

}

parser.c

#include ~global.h· int lookahe .. d.

parse ( ) {

;. parses and ~ranslates expression list .1

lookahead. lexan();

while (lookahead 1= DONE) { expr{); match(';')i

}

}

expr{ ) (

int t; terD() ; whi le ( 1)

switch (lookahead) { case '+1, case '_":

t :I! look.head;

match(lQokahead}; term(); eMit(t, NONB); continue;

default: return.

}

}

term() {

int t; factor ( ) ; "hile( 1)

switch (lookahead) {

case '.': case 'I'! case ntv: ease MOO: t :!:: lookahe.ad.

mateb(lookahead); factor()i emit(t, NONE); continue;

default: return;

}

}

76 A SIMPLE COMPILER

SEC. 2.9

factori) {

switch(lookahead) { case '(I:

match('('); expr(); match(')'); break; case HUM:

eait(NUM, tokenval.; match{NUM); break; case ID:

emit(ID, tokenval). match{XO); break; default:

error(~synt.x error");

}

}

match(t)

int t;

{

if (lookahead ~: t. lookahead·lexan(); else error(~syntax error");

eDitter.c

#include "global.hR

emit (tt tval)

I. generates output *1

int e , tval;

8viteh(t) {
case i +' : cas. , _ t : case ' ... .;: ease ' /" :
printf (~"c\n";. t); break;
case DIV:
printf (tlDIV'\n" ) ; break;
case MOD:
printf ( MMOD'\n" ) ; break;
ease NOM! printf (""d\n" t tval); break; case ID:

printf(~%s\n". symtableltval).lexptr); break; default:

ptint£(~tOken %do tokenval Xd\n", t, tv.l);

}

s}'1l'Ibol.e

#include ftglobal.h~

Idefine STRMAX 999 #define SYMMAX 100

I. size of lexemes array -/ 1* size Qf symtable *1

SEC. 2.9

PUTTING THE TECHNIQUES TOGETHER 77

char lexemes[STRMAX];

int lastchar = - 1; 1* last used position in lexemes *1 struct entry symtable[SYMMAX};

int lookup(s) ohar silt

1* last used position in symtable *1 1* returns position of entry for $ *1

int lastentry ~ 0;

{

int p;

for (p = lastentry; p > 0; P • P - 1)

if (strcmp(symtable(p].lexptr~ 8) :2 0) return p;

return 0;

}

int insert(s~ tok) char s[];

int tok;

;- returns position of entry for s *1

(

int len;

len. strlen(s); 1* strlen c~p~tes length of s *J if (lastentry + 1 >= SYMMAX)

error("symbol table full-); if (lastcbar + lep + , >. STRMAX)

error{Ulexemes a.rray full"~;

lastentry. lastentry + 1t

symtable{lastent~y}.token • tok; symtable[lastentry].lexptr a &lexemes[lastchar + 1}; lastchar a lastehar + len + 1; strcpy{symtable[lastentry].lexptr, sIt

return lastentry;

}

init. c

#include Ilgloba1.h"

struct entry keywords[] z ( IOdivll, DIV,

"mod", MOD~

0, 0

) ;

init() {

J* loads keywords into symtable ./

struct entry *P.

for (p ~ keywords; p->token; p*+) insert(p->lexptr, p->token);

78 A SIMPLE COMPILER

SEC. 2.9

error.C:

#include -global.h"

error{m) /. generate. all error messages _;

ohar *m;

{

fprintf(atderr, -line %d: ~a'nft, lineno, .);

exlt(1); ;. unsuccessful termination ./

}

mo.in.c

#include ~91obal.h· main( }

(

init(); parse( ) i exiteD);

1* successful t.r~in.tiQn *1

}

EXERCISES

2.1 Consider the context-free grammar

S~SS+ ISS* la

a) Show how the string aa+a.* can be generated by this grammar.

b) Construct a parse tree for this string.

c) What language is generated by this grammar? Justify your answer.

2.2 What language is generated by the following grammars? In each case justify your answer.

a) S - 0 S 1 I 0 1

b) S - + S S I - S S I a

c) S - S (S ) S I E;

d) S a S b SIb S a S I E

e) S a I s + sis s t s • I (S )

2.3 Which of the grammars in Exercise 2.2 are ambiguous?

2.4 Construct unambiguous context-free grammars for each of the following languages. In each case show that your grammar is correct.

CHAPTER 2

EX~RCISES 79

I

a) Arithmetic expressions in postfix notation.

b) Left-associative lists of identifiers separated by commas.

c) Rlght-associarive lists of identifiers separated by commas.

d) Arithmetic expressions of integers and identifiers with the four binary operators +, -; '*, I.

e) Add unary plus and minus to the arithmetic operators of (d).

*%..5 a) Show that all binary strings generated by the following grammar have values divisible by 3. Him. Use induction on the number of nodes in a parse tree.

num .... 11 I 1001 I num 0 ~ num num

b) Does the grammar generate all binary strings with values divisible by 3'!

2.6 Construct a context-free grammar for roman numerals.

2.7 Construct a syntax-directed translation scheme that translates arunmerle expressions from infix notation into prefix notation in which an operator appears before its operands ~ e. g.. - xy is the prefix notation for x - y. Give annotated parse trees for the inputs 9 - 5 + 2 and 9- 5*2.

1.8 Construct a syntax-directed translation scheme that translates arithmetic expression s from postfix notation into in fix notation. Give annotated parse trees for the inputs 95-2* and 952*-.

2.9 Construct a syntax-directed translation scheme that translates integers into roman numerals.

2.10 Construct a syntax-directed translation scheme that translates roman numerals into integers.

2.11 Construct recursive-descent parsers for the grammars in Exercise 2.2 (a). (b). and (c).

2.12 Construct a syntax-d irected translator that verifies that the parentheses in an input string are properly balanced.

2.13 The following ru les define the translation of an E ngl ish word into pig Latin:

a) If the word begins with a nonempty string of consonants, move [he initial consonant string to the back of the word and add the su ffix AY~ e.g., pig becomes iqpay.

b) If the word begins with a vowel. add the suffix YAY; e.g .. owl . becomes owlyay.

c) U following a Q is a consonant.

d) Y at the beginning of a word is a vowel if it is nOI followed by a vowel.

so A SlMPLE COMPILER

CHAPTER 2

e) One-letter words are not changed.

Construct a syntax-directed translation scheme for pig Latin. 2.14 In the programming language C the for-statement has the form:

for ( expr I ; expr-, ; expr 3 ) stmt

The first expression is executed before the loop; it is typica By used for initializing the loop index. The second expression is a test made before each iteration of the loop; the loop is exited if the expression becomes O. The loop itself consists of the statement {stmt expr 3 ;}. The third expression is executed at the end of each iteration; it is typically used to increment the loop index. The meaning of the forstatement is similar to

expr I ; ",hile ( expr2 ) { stmt expr3; }

Construct a syntax-directed translation scheme to translate C forstatements into stack-machine code.

*2.15 Consider the following for-statement:

for i : = I step 10 - j until '0 * j do j: = j +]

Three semantic definitions can be given for this statement. One possible meaning is that the limit 10 * j and increment 10 - j are to be evaluated once before the loop. as in PLll. For example. if j = 5 before the loop, we would run through the loop ten times and exit. A second, completely different, meaning would ensue if we are required to evaluate the limit and increment every lime through the loop. For example, if j = 5 before the loop. the loop would never terminate. A third 'meaning is given by languages such as Algol. When the increment is negative. the test made for termination of the loop is i < 10 * j, rather than; > 10 * j. For each of these three semantic definitions construct a syntax-directed translation scheme to translate these for-loops into stack-machine code.

2.16 Consider the following gram mar fragment for if-then- and if -t henelse-statements:

stmt -+ if expr then stmt

I if expr tben stmt else SImI I other

where oCher stands for th.e other statements in the language.

a) Show that this grammar is ambiguous.

b) Construct an equivalent unambiguous grammar that associates each else with the closest previous unmatched then.

CHAPTER 2

BIBLIOGRAPHIC NOTES 81

c) Construct a syntax-directed translation scheme based on this grammar to translate conditional statements into stack machine code.

*2.17 Construct a syntax-directed translation scheme that translates arithmetic expressions in infix notation into arithmetic expressions in infix. notation having no redundant parentheses. Show the annotated parse tree for the input ( ( (1 + 2) * (3 * 4)) + 5).

PROGRAMMING EXERCISES

Pl.. Implement a translator from integers to roman numerals based on the syntax-directed translation scheme developed in Exercise 2.9.

P1.1 Modify the translator in Section 2.9 to produce as output code for the abstract stack machine of Section 2.8.

'2.3 Modify the error recovery module of the translator in Section 2.9 to sk ip to the next input expression on encountering an error.

P2.4 Extend the translator in Section 2.9 to handle all Pascal expressions. Pl.S Extend the com piler of Section 2.9 to translate into stack -machine code statements generated by the following grammar;

stmt -. id ~. e:cpr

! if expr then stm:

I while expr do stmt I begin opt....stmts end

Qpt....stmts - slmLlist I _ IE

srmt..Jist .... stmLlist ; stmt stmt

*Pl.6 Construct a set of test expressions for the compiler in Section 2.9, so that each production is used at least once in deriving some tese: expression. Construct a testing program that can be used as a general compiler testing tool. Use your testing program to run your compiler on these test expressions.

P2.7 Construct a set of test statements for your compi ler of Exercise PZ. 5 so that each production is used at least once to generate some lest statement. Use the testing program of Exercise P2.6 (0 run your compiler on these test expressions.

BIBLIOGRAPHIC NOTES

This introductory chapter touches on a number of subjects that are treated in more detail in the rest of the book. Pointers (0 the literature appear in the chapters containing further material.

Context-free grammars were introduced by Chomsky t 1956) as part of a

-

...._

82 A SIM PLE COMPILER

CHAPTBR 2

study of natural languages. Their use in specifying the syntax of programming languages arose independently. While working with a draft of Algol 60, John Backus "hastily adapted [Emil Post's productions] to that use" (Wexelblat 11981, p.162)), The resulting notation was a variant of context-free grammars. The scholar Panini devised an equivalent syntactic notation to specify the rules of Sanskrit grammar between 400 B.C. and 200 B.C. (lngerman 119671).

The proposal that BNF, which began as an abbreviation of Backus Normal Form, be read as Backus-Naur Form, to recognize Naur's contributions as editor of the Algol 60 report (Naur (t963H. is contained in a letter by Knuth (19641.

Syntax-directed definitions are a form of inductive definitions in which the induction is on the syntact ie structure. As such they have long been used informally In mathematics. Their application to programming languages came with the use of a grammar to structure the Algol 60 report. Shortly thereafter, Irons ll96l) constructed a syntax-directed compiler.

Recursive-descent parsing has been used since the early 1960's. Bauer 119761 attributes (he method to Lucas ll96ll. Hoare 1'962b, p.12SI describes an Algol compiler organized as "3 set of procedures. each of which is capable of processing one of the syntactic units of the Algol 60 report." Foster [19681 discusses the elimination of left recursion from productions containing semantic actions that do nor affecc anribute values.

McCartb y tl963 1 advocated that the translation of a language be based on abstract syntax. In the same paper McCarthy 11963, p.24] left "the reader to convince himself" that a tail-recursive formulation of the factorial function is equivalent to an iterative program.

The benefits of partitioning a cornpiler into a front end and a back end were explored In a committee report by Strong et al. l1958}. The report coined the name UNCOl (from universal computer oriented language) for a universal intermediate language. The concept has remained an ideal.

A good way to learn about implementation techniques is to read the code of existing compilers. Unfortunately, code is nOI often published. RandeU and Russell (19641 give a comprehensive account of an early Algol compiler. Compiler code may also be seen in McKeeman. Horning. and Wortman 1 1970}. Barron ll9811 is a collection of papers on Pascal implementation. including implementation notes distributed with the Pascal P compiler (Nori et al. tl981 D, code generation details (Amman n ll977 D. and the code for an implementation of Pascal S, a Pascal subset designed by Wirth 11981 J for student lise. Knuth 11985) gives an unusually dear and derailed description of the lEX translator.

Kern ighan and Pike [1984] describe in detail how to build a desk calculator program around a syntax-directed translation scheme using the compilerconstruct Ion tools available on the UNIX operating system. Equation (2.17) is from Tantzen l19631.

CHAPTER 3

Lexical Analysis

This chapter deals with techniques for specifying and implementing lexical analyzers. A simple way to buHd a 'exica' analyzer is to construct a diagram that illustrates tbe structure of the tokens of the source language, and then to hand-translate the diagram into a program for finding tokens. Efficient lexical analyzers can be produced in this manner.

The techniques used to implement lexical analyzers can also be applied to other areas such as query languages and information retrieval systems. In each application. the underlying problem is the specification and design of programs that execute actions triggered by patterns in strings. Since patterndirected programming is widely useful, we introduce a pattern-action language called Lex for specifying lexical analyzers. In this language, patterns are specified by regular expressions. and a compiler for LeI!. can generate all efficient finite-automaton recognizer for the regular expressions.

Several other languages use regular expressions to describe patterns. For example. the pattern-scanning language A WK uses regular expressions to select input lines for processing and the UN]X system shell allows a user to refer to a set of file names by writing a regular expression, The UNIX command rm * . o, for instance t removes all files with names end ing in ". 0" , 1

A software tool that automates the construction of lexical analyzers allows people with different backgrounds to use pattern matching in their own application areas. For example, Janis 11976) used a lexical-analyzer generator to create a program that recognizes imperfections in printed circuit boards. The circuits are dlgitally scanned and converted into "strings" of line segments at different angles. The "lexical analyzer" looked fOT patterns corresponding to imperfections in the string of line segments. A major advantage of a lexicalanalyzer generator is that it can utilize (he best-known pattern-matching algorithms and thereby create efficient lexical analyzers for people who are not experts in pattern-matching techniques.

\ The expression ... 0 is a vanant of the usual notation for regular exprc:l-'iion:<i. Exercises 3- 10 and 3. '4 mention some commonly used variants or regular expression notanons.

84 LEXICAL A.NALYSIS

SEC. 3.1

3+ 1 THE ROLE OF THE LEXICAL ANALYZER

The lexical analyzer is the first phase of a compiler. Its main task is to read the input characters and produce as output a sequence of tokens that the parser uses for syntax analysis. This interaction, summarized schematically in Fig. 3.1, is commonly implemented by making the lexical analyzer be a subrout ine or a coroutine of the parser. Upon rece iving a "get next token" command from the parser, the lexical analyzer reads input characters until it can identify the next token.

source program

lexical token

anelyzer parser
gt'l next
~ token /
symbol
table ->

Fig. 3.1. I ntcract ion of lexica! analyzer with parser.

Since the lexical analyzer is the part of the compiler that reads the source text, it may also perform certain secondary tasks at the user interface. One such task is stripping Qui from the source program comments and white space in I he form o-f blank, tab. and new line characters. A nother is corre taring error messages from the compiler with the source program. For example, the lexical analyzer may keep track of (he number of newline characters seen, so that a line number can be associated with an error message. In some compilers. the lexical analyzer is in charge of making a copy of the source program with the error messages marked in it. If the source Janguage supports some macro preprocessor functions; then these preprocess.or functions may also be implemented as lexical analysis takes place.

Sometimes. lexical analyzers are divided into a cascade of two phases. the first called "scann ing," and the second "lexical analysis." The scanner is responsible for doing simple tasks, whi Ie the lexical analyzer proper does the more complex operations. For example, a Fortran compiler might use a scanner to eliminate blanks from the input.

Issues in 'Lexical Analysis

There are several reasons for separating the analysis phase of compiling into lexical analysis and parsing.

I . Simpler design is perhaps the most import ant consideration. The separation of lexical analysis from syntax analysis often allows us to simplify

SEC 3.1

THE ROLE OF THE LEXlCAL ANALYZER 85

one or the other of these phases. For example. a parser embodying the conventions for comments and white space is significantly more complex than one that can assume comments and white space have already been removed by a lexical analyzer. If we are designing a new language, separating the lexical and syntactic conventions can lead to a cleaner overall language design.

2. Compiler efficiency is improved. A separate lexical analyz.er allows Us to construct a specialized and potentially more efficient processor for the task. A large amount of time is spent reading the source program and partitioning it into tokens. Specialized buffering techniques for reading input characters and processing tokens can significantly speed up the performance of a compiler.

3. Compiler portability is enhanced. Input alphabet peculiarities and other de v ice-sped fie anomalies can be restricted to the lexical analyzer. The representation of special or non-standard symbols, such as t in Pascal. can be isolated in the lexical analyzer.

Specialized tools have been designed to help automate the construction of lexical analyzers and parsers when they are separated. We shan see several examples of such tools in this book.

Tokens, Patterns, Lexemes

When talking about lexical analysis, we use the terms "token." "pattern," and "lexeme " with specific meal] ings. Examples of their use are shown in Fig. 3.2. In general, there is a set of strings in the input for which the same token is produced as output. This set of strings is described by a rule called a pattern associated with the token. The pattern is said to mat(-/l each string in the set. A lexeme is a sequence of characters in the source program that is matched by the pattern for a token. For example. in the Pascal statement

canst pi = 3.1416;

the substring pi is a lexerne for the token "identifier."

TOKEN SAMPLE LEXEMES INFORMi\L DESCRIPTION OF P,\TTERN
cons! const const
ir if if
relation <, <=, ~, <)0,\ >t >= <; or <;If or lit or < > or >= or >
id pi, count, 02 letter followed by leiters and digits
num 3.1416,Q,6.02E23 any numeric cot)}tant
IiteTa! "core dumped" any characters between .. and .. except 01 Fig. 3.2. Examples of tokens.

Sign up to vote on this title
UsefulNot useful