LEX.pdf 有關LEX的詳細介紹

docs.sun.com · Programming Utilities Guide

Search this Book

Tips

Search

lex-Lexical Analysis

2

characters that satisfy certain characteristics. Hence the name lex. You don’t have to use lex

to handle problems of this kind. You could write programs in a standard language like C to

handle them, too. In fact, what lex does is produce such C programs. (lex is therefore called

a program generator.)

What lex offers you is typically a faster, easier way to create programs that perform these

tasks. Its weakness is that it often produces C programs that are longer than necessary for the

task at hand and that execute more slowly than they otherwise might. In many applications

this is a minor consideration, and the advantages of using lex considerably outweigh it.

lex can also be used to collect statistical data on features of an input text, such as character

count, word length, number of occurrences of a word, and so forth. In the remaining sections

For information about using lex to develop applications in languages other than English, see

lex (1).

Generating a Lexical Analyzer Program

lex generates a C-language scanner from a source specification that you write. This

specification contains a list of rules indicating sequences of characters -- expressions -- to be

searched for in an input text, and the actions to take when an expression is found. To see how

to write a lex specification see the section "Writing lex Source" on page 54.

The C source code for the lexical analyzer is generated when you enter

$ lex lex.l

where lex.l is the file containing your lex specification. (The name lex.l is conventionally

Note that running lex on a specification that is spread across several files, as in the following

example, produces one lex.yy.c:

$ lex lex1.l lex2.l lex3.l

Invoking lex with the -t option causes it to write its output to stdout rather than lex.yy.c,

so that it can be redirected:

$ lex -t lex.l > lex.c

Options to lex must appear between the command name and the filename argument.

The lexical analyzer code stored in lex.yy.c (or the .c file to which it was redirected) must

be compiled to generate the executable object program, or scanner, that performs the lexical

analysis of an input text.

int yywrap()

{

return(1);

}

main()

{

while (yylex())

;

how lex works.

Writing lex Source

lex source consists of at most three sections: definitions, rules, and user-defined routines. The

rules section is mandatory. Sections for definitions and user routines are optional, but must

appear in the indicated order if present:

definitions

%%

rules

%%

Lines in the rules section that begin with white space and that appear before the first rule are

copied to the beginning of the function yylex(), immediately after the first brace. You might

use this feature to declare local variables for yylex().

Each rule specifies the pattern sought and the actions to take on finding it. The pattern

specification must be entered at the beginning of a line. The scanner writes input that does not

match a pattern directly to the output file. So the simplest lexical analyzer program is just the

Regular Expressions

You specify the patterns you are interested in with a notation called a regular expression. A

regular expression is formed by stringing together characters with or without operators. The

simplest regular expressions are strings of text characters with no operators at all:

orange ;

Because you specified a null action on the right with the semicolon, the scanner does nothing

but print the original input text with every occurrence of this regular expression removed, that

is, without any occurrence of the string orange at all.

Operators

Unlike orange above, most expressions cannot be specified so easily. The expression itself

might be too long, or, more commonly, the class of desired expressions is too large; it might,

in fact, be infinite.

Using operators -- summarized in Table 2-1 on page 58 -- you can form regular expressions

for any expression of a certain class. The + operator, for instance, means one or more

mmmmm

and 7* is a regular expression that matches any string of zero or more 7s:

77

77777

777

The empty third line matches because it has no 7s in it at all.

The | operator indicates alternation, so that ab|cd matches either ab or cd. The operators {}

Brackets, [], indicate any one character from the string of characters specified between the

brackets. Thus, [dgka] matches a single d, g, k, or a.

[A-Za-z0-9*&#]

This is a regular expression that matches any letter (whether upper or lowercase), any digit, an

asterisk, an ampersand, or a #.

Given the following input text, the lexical analyzer with the previous specification in one of

its rules will recognize *, &, r, and #, perform on each recognition whatever action the rule

specifies (we have not indicated an action here), and print the rest of the text as it stands:

$$$$?? ????!!!*$$ $$$$$$&+====r~~# ((

To include the hyphen character in the class, have it appear as the first or last character in the

brackets: [-A-Z] or [A-Z-].

The operators become especially powerful in combination. For example, the regular

expression to recognize an identifier in many programming languages is:

R2D2