Lecture 4

Lexical Analysis
 Front-End

IR
Tokens (Intermediate
Source Code
Representation)
Scanner Parser

Errors
Modules
- Scanner
- Parser
Lexical Analysis
The task of the scanner is to take a program written in some
programming language as a stream of characters and break it into a
stream of tokens. This activity is called lexical analysis. A token,
however, contains more than just the words extracted from the input.
The lexical analyzer partition input string into substrings, called words,
and classifies them according to their role.
Goal of Lexical Analysis
Goal:
• Partition input string into substrings
• Classify according to their role
Tokens
• A token is a syntactic category
• Natural language
“ He wrote the program”’
• Words: “He” , “wrote”, “, “the”, “program”
Tokens

If ( i == j )
x = 0;
else
x = 1;
Tokens

if ( \b i \b = = \b j ) \n … …
Tokens
• Programming language
• “if(b==0) a= b”

Words: “if” , “(“ , “b” , “==“ , “0” , “)” , “a” , “=“ ,”b”