Lexical Analysis
The lexical analyzer (scanner) reads characters and groups them into tokens. It also strips whitespace and comments and enters identifiers into the symbol table.
| Term | Meaning | Example |
|---|---|---|
| Token | A category: a pair (token-name, attribute) | <id, 1>, <num, 60> |
| Lexeme | The actual character sequence matched | rate, 60 |
| Pattern | A rule (usually a regular expression) describing a token | letter(letter | digit)* |
Counting Tokens
printf("sum = %d", a + b);Tokens: printf, (, "sum = %d", ,, a, +, b, ), ; → 9 tokens. A whole string literal counts as one token.
Input Buffering
A two-buffer scheme uses lexemeBegin and forward pointers. Sentinels (an EOF character at the end of each buffer half) reduce the number of tests per character from two to one.
A Lex Program
%{
#include <stdio.h>
int words = 0, nums = 0;
%}
%%
[0-9]+ { nums++; printf("NUMBER(%s) ", yytext); }
[a-zA-Z_][a-zA-Z0-9_]* { words++; printf("ID(%s) ", yytext); }
[ \t\n]+ ;
. { printf("SYM(%s) ", yytext); }
%%
int main() { yylex(); printf("\nids=%d nums=%d\n", words, nums); }
int yywrap() { return 1; }$ echo "rate = rate + 60;" | ./a.out
ID(rate) SYM(=) ID(rate) SYM(+) NUMBER(60) SYM(;)
ids=2 nums=1
Lexical Errors and Recovery
Errors include illegal characters and unterminated strings. Panic-mode recovery deletes characters until a valid token can be formed.
Track your progress through the Compiler Design course