DEPARTMENT OF COMPUTER SCIENCE AND ENGINEERING

Compiler Design
KCS502
by
DR PARUL YADAV

Institute of Engineering and Technology, Lucknow

Lecture 5:
Input buffering, Token Specification
and Recognition
Contents
• Input buggering
• Token specification
• Token Recognition
 Input Buffering

• Way for reading source program can be speeded

• 2 buffers that are alternatively reloaded
• Each of size N (size of a disk block eg. 4096 bytes)
• Using one system load command, can read N characters into a buffer,
rather than using one system call per character
Input Buffering
 Sentinels

• We must check, each time we advance forward, that we have not moved off
one of the buffers.
• If we do, then reload the other buffer.
• Make 2 tests for each for each character read:
1. one for the end of the buffer
2.One to determine what character is read
• Both tests can be combined, extending each buffer to hold a sentinel character
at the end
Sentinels at the End of Each Buffer
Lookahead code with sentinels
Token Specification: Regular Expression
Regular Expression
Regular Expression: Operator
Precedence
Regular Expression: Example
Regular Expression: Algebraic Laws
Regular Definitions
Regular Definitions
Regular Expression: Extension
Regular Expression: Extension
Regular Expression: Extension
Rewriting Regular Definitions
Rewriting Regular Definitions
Recognition of Tokens: Grammar
Recognition of Tokens: Patterns for
Tokens

No token
Tokens, Patterns and Attribute Values
Transition diagram for
Transition Diagram for ’s and
Keywords
Transition Diagram for Keywords
Transition Diagram for Unsigned
Numbers
Transition Diagram for Whitespace
Implementation of
Thank you