ROLE OF LEXICAL ANALYZER

1.Lexical anylazer functions

2.Separation of lexical analyzer from syntex analyzer
3.Tokens, Lexemes, Patterns
4.Lexical Errors
 Lexical Analyzer Functions
 It produces a stream of Tokens.
It removes comments from the source program.
 It removes white space characters such as blank spaces,tab spaces
and new line characters.
 It counts line numbers of source program.
It generates a symbol table which stores information about
identifiers.
 It provides error messages with correspnding number lines.
 Interaction between lexical analyzer and syntex analyzer.
Separation of lexical analyzer from syntex analyzer

1. To simplify the design of a complier.

2. To increase the efficiency of a complier.

3. To enhance the portability of a complier.

 Token, Lexeme, Ptterns
Token: Token is a sequence of characters that can be treated as a single
logical entity. Typical tokens are: Identifiers, keywords, operators, special
symbols and Constants.

Lexeme:A lexeme is a sequence of characters in the source program.

Pattern: patterns are mainly useful for define regular expressions.

 Lexical Errors
Types of lexical errors:

1.Spelling errors
2.Unmatched string
3.Apperance of illigal characters
4.Exceeding length of identifier
Error Recovery mechanisms:
1.Delete a single character from the remaining input
2.Insert a missing character into the remaining input
3.Replace one character by another character
4.Transpose to adjacent characters
5.Delete successive characters from the remaining input until lexical
analyzer recognizes a well formed token.This apporch is called as panic
mode error recovery.