Lexical Analyzer Lexical Analyzer reads the source program character by character and returns the tokens of the source program. A token describes a pattern of characters having same meaning in the source program.(such as identifiers,operators,keywords,numbers, delimeters and so on) Ex: newval :oldval 12 => tokens: newval identifier = assignment operator oldval identifier + add operator 12 a number Puts information about identifiers into the symbol table. Regular expressions are used to describe tokens(lexical constructs). A (Deterministic)Finite State Automaton can be used in the implementation of a lexical analyzer. CS308 Compiler Theory 5Lexical Analyzer • Lexical Analyzer reads the source program character by character and ret rns the returns the t ko ens of the so rce program of the source program. • A token describes a pattern of characters having same meaning in the source program. (such as identifiers, operators, keywords, numbers, source program. (such as identifiers, operators, keywords, numbers, delimeters and so on) Ex: newval := oldval + 12 => tokens: newval identifier := assignment operator oldval identifier + add operator 12 a number • Puts information about identifiers into the symbol table. • R l i d t d ib t k (l i l t t ) Regular expressions are used to describe tokens (lexical constructs). • A (Deterministic) Finite State Automaton can be used in the implementation of a lexical analyzer. 5 implementation of a lexical analyzer. CS308 Compiler Theory