[No value for “title”]
Edition 2.5.35, 10 September 2007
Vern Paxson, Will Estes and John Millaway
The flex manual is placed under the same licensing conditions as the rest of flex:
Copyright
c
2001, 2002, 2003, 2004, 2005, 2006, 2007 The Flex Project.
Copyright
c
1990, 1997 The Regents of the University of California. All rights reserved.
This code is derived from software contributed to Berkeley by Vern Paxson.
The United States Government has rights in this work pursuant to contract no. DE-
AC03-76SF00098 between the United States Department of Energy and the University of
California.
Redistribution and use in source and binary forms, with or without modification, are per-
mitted provided that the following conditions are met:
1. Redistributions of source code must retain the above copyright notice, this list of con-
ditions and the following disclaimer.
2. Redistributions in binary form must reproduce the above copyright notice, this list of
conditions and the following disclaimer in the documentation and/or other materials
provided with the distribution.
Neither the name of the University nor the names of its contributors may be used to endorse
or promote products derived from this software without specific prior written permission.
THIS SOFTWARE IS PROVIDED “AS IS” AND WITHOUT ANY EXPRESS OR IM-
PLIED WARRANTIES, INCLUDING, WITHOUT LIMITATION, THE IMPLIED WAR-
RANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE.
i
Table of Contents
1 Copyright . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
2 Reporting Bugs . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
3 Introduction. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
4 Some Simple Examples. . . . . . . . . . . . . . . . . . . . . 4
5 Format of the Input File . . . . . . . . . . . . . . . . . . . 6
5.1 Format of the Definitions Section . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
5.2 Format of the Rules Section . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
5.3 Format of the User Code Section . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
5.4 Comments in the Input . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
6 Patterns . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
7 How the Input Is Matched . . . . . . . . . . . . . . . . 14
8 Actions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
9 The Generated Scanner . . . . . . . . . . . . . . . . . . . 19
10 Start Conditions . . . . . . . . . . . . . . . . . . . . . . . . . 21
11 Multiple Input Buffers . . . . . . . . . . . . . . . . . . . 27
12 End-of-File Rules . . . . . . . . . . . . . . . . . . . . . . . . 31
13 Miscellaneous Macros . . . . . . . . . . . . . . . . . . . . 32
14 Values Available To the User . . . . . . . . . . . . . 33
15 Interfacing with Yacc . . . . . . . . . . . . . . . . . . . . 34
ii
16 Scanner Options . . . . . . . . . . . . . . . . . . . . . . . . . 35
16.1 Options for Specifying Filenames . . . . . . . . . . . . . . . . . . . . . . . . . . . 35
16.2 Options Affecting Scanner Behavior . . . . . . . . . . . . . . . . . . . . . . . . 36
16.3 Code-Level And API Options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39
16.4 Options for Scanner Speed and Size. . . . . . . . . . . . . . . . . . . . . . . . . 41
16.5 Debugging Options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
16.6 Miscellaneous Options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44
17 Performance C onsi der atio ns . . . . . . . . . . . . . . 45
18 Generating C++ Scanners . . . . . . . . . . . . . . . 5 0
19 Reentrant C Scanners. . . . . . . . . . . . . . . . . . . . 54
19.1 Uses for Reentrant Scanners. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54
19.2 An Overview of the Reentrant API . . . . . . . . . . . . . . . . . . . . . . . . . 55
19.3 Reentrant Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55
19.4 The Reentrant API in De tail . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56
19.4.1 Declaring a Scanner As Reentrant. . . . . . . . . . . . . . . . . . . . . . 56
19.4.2 The Extra Argument. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56
19.4.3 Global Variables Replaced By Macros . . . . . . . . . . . . . . . . . . 56
19.4.4 Init and Destroy Functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56
19.4.5 Accessing Variables with Reentrant Scanners. . . . . . . . . . . . 57
19.4.6 Extra Data. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58
19.4.7 About yyscan t. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59
19.5 Functions and Macros Available in Reentrant C Scanners. . . . . 59
20 Incompatibilities with Lex and Posix. . . . . . 61
21 Memory Management. . . . . . . . . . . . . . . . . . . . 64
21.1 The Default Memory Management. . . . . . . . . . . . . . . . . . . . . . . . . . 64
21.2 Overriding The Default Memory Management . . . . . . . . . . . . . . . 65
21.3 A Note About yytext And Memory . . . . . . . . . . . . . . . . . . . . . . . . . 66
22 Serialized Tables. . . . . . . . . . . . . . . . . . . . . . . . . 67
22.1 Creating Serialized Tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67
22.2 Loading and Unloading Serialized Tables . . . . . . . . . . . . . . . . . . . . 67
22.3 Tables File Format . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68
23 Diagnostics. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72
24 Limitations. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74
25 Additional Reading . . . . . . . . . . . . . . . . . . . . . . 75
iii
FAQ . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76
When was flex born? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76
How do I expand backslash-escape sequences in C-style quoted strings?
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76
Why do flex scanners call fileno if it is not ANSI compatible? . . . . . . . 76
Does flex support recursive pattern definitions? . . . . . . . . . . . . . . . . . . . . 76
How do I skip huge chunks of input (tens of megabytes) while using
flex? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77
Flex is not matching my patterns in the same order that I defined them.
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77
My actions are executing out of order or sometimes not at all. . . . . . . 77
How can I have multiple input sources feed into the same scanner at the
same time? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77
Can I build nested parsers that work with the same input file?. . . . . . 78
How can I match text only at the end of a file? . . . . . . . . . . . . . . . . . . . . 78
How can I make REJECT cascade across start condition boundaries?
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78
Why can’t I use fast or full tables with interactive mode? . . . . . . . . . . 79
How much faster is -F or -f than -C?. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79
If I have a simple grammar can’t I just parse it with flex? . . . . . . . . . . 79
Why doesn’t yyrestart() set the start state back to INITIAL? . . . . . . 79
How can I match C-style comments?. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79
The ’.’ isn’t working the way I expected. . . . . . . . . . . . . . . . . . . . . . . . . . . 80
Can I get the flex manual in another format? . . . . . . . . . . . . . . . . . . . . . . 80
Does there exist a "faster" NDFA->DFA algorithm? . . . . . . . . . . . . . . . 80
How doe s flex compile the DFA so quickly? . . . . . . . . . . . . . . . . . . . . . . . 80
How can I use more than 8192 rules? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81
How do I abandon a file in the middle of a scan and switch to a new
file?. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81
How do I execute code only during initialization (only before the first
scan)? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81
How do I execute code at termination? . . . . . . . . . . . . . . . . . . . . . . . . . . . 82
Where else can I find help? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82
Can I include comments in the "rules" section of the file? . . . . . . . . . . 82
I get an error about undefined yywrap(). . . . . . . . . . . . . . . . . . . . . . . . . . . 82
How can I change the matching pattern at run time? . . . . . . . . . . . . . . 82
How can I expand macros in the input? . . . . . . . . . . . . . . . . . . . . . . . . . . . 82
How can I build a two-pass scanner? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83
How do I match any string not matched in the preceding rules? . . . . . 83
I am trying to port code from AT&T lex that uses yysptr and yysbuf.
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83
Is there a way to make flex treat NULL like a regular character? . . . . 83
Whenever flex can not match the input it says "flex scanner jammed".
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
84
Why doesn’t flex have non-greedy operators like perl does? . . . . . . . . . 84
Memory leak - 16386 bytes allocated by malloc. . . . . . . . . . . . . . . . . . . . 84
How do I track the byte offset for lseek()? . . . . . . . . . . . . . . . . . . . . . . . . . 85
How do I use my own I/O classes in a C++ scanner? . . . . . . . . . . . . . . 85
