.TH FLEX 1 "26 May 1990" "Version 2.3"
.SH NAME
-flex - fast lexical analyzer generator
+flexdoc - documentation for flex, fast lexical analyzer generator
.SH SYNOPSIS
.B flex
.B [-bcdfinpstvFILT8 -C[efmF] -Sskeleton]
of the regular expressions. Whenever it finds one, it executes
the corresponding C code.
.SH SOME SIMPLE EXAMPLES
-.LP
+.PP
First some simple examples to get the flavor of how one uses
.I flex.
The following
and the "printf" is the
.I action.
The "%%" marks the beginning of the rules.
-.LP
+.PP
Here's another simple example:
.nf
- int num_lines = 0, num_chars = 0;
+ int num_lines = 0, num_chars = 0;
%%
- \\n ++num_lines; ++num_chars;
- . ++num_chars;
+ \\n ++num_lines; ++num_chars;
+ . ++num_chars;
%%
main()
- {
- yylex();
- printf( "# of lines = %d, # of chars = %d\\n",
- num_lines, num_chars );
- }
+ {
+ yylex();
+ printf( "# of lines = %d, # of chars = %d\\n",
+ num_lines, num_chars );
+ }
.fi
This scanner counts the number of characters and the number
which matches a newline ("\\n") and increments both the line count and
the character count, and one which matches any character other than
a newline (indicated by the "." regular expression).
-.LP
+.PP
A somewhat more complicated example:
.nf
Pascal. It identifies different types of
.I tokens
and reports on what it has seen.
-.LP
+.PP
The details of this example will be explained in the following
sections.
.SH FORMAT OF THE INPUT FILE
definitions to simplify the scanner specification, and declarations of
.I start conditions,
which are explained in a later section.
-.LP
+.PP
Name definitions have the form:
.nf
.fi
and matches one-or-more digits followed by a '.' followed
by zero-or-more digits.
-.LP
+.PP
The
.I rules
section of the
.fi
where the pattern must be unindented and the action must begin
on the same line.
-.LP
+.PP
See below for a further description of patterns and actions.
-.LP
+.PP
Finally, the user code section is simply copied to
.B lex.yy.c
verbatim.
if it is missing, the second
.B %%
in the input file may be skipped, too.
-.LP
+.PP
In the definitions and rules sections, any
.I indented
text or text enclosed in
.B %}
is copied verbatim to the output (with the %{}'s removed).
The %{}'s must appear unindented on lines by themselves.
-.LP
+.PP
In the rules section,
any indented or %{} text appearing before the
first rule may be used to declare variables
errors (this feature is present for
.I POSIX
compliance; see below for other such features).
-.LP
+.PP
In the definitions section, an unindented comment (i.e., a line
beginning with "/*") is also copied verbatim to the output up
to the next "*/". Also, any line in the definitions section
(foo|bar)*
.fi
-.LP
+.PP
Some notes on patterns:
.IP -
A negated character class such as the example "[^A-Z]"
rule listed first in the
.I flex
input file is chosen.
-.LP
+.PP
Once the match is determined, the text corresponding to the match
(called the
.I token)
corresponding to the matched pattern is then executed (a more
detailed description of actions follows), and then the remaining
input is scanned for another match.
-.LP
+.PP
If no match is found, then the
.I default rule
is executed: the next character in the input is considered matched and
.fi
(It will copy all other characters in the input to the output since
they will be matched by the default rule.)
-.LP
+.PP
Here is a program which compresses multiple blanks and tabs down to
a single blank, and throws away whitespace found at the end of a line:
.nf
[ \\t]+$ /* ignore this token */
.fi
-.LP
+.PP
If the action contains a '{', then the action spans till the balancing '}'
is found, and the action may cross multiple lines.
.I flex
and will consider the action to be all the text up to the next
.B %}
(regardless of ordinary braces inside the action).
-.LP
+.PP
An action consisting solely of a vertical bar ('|') means "same as
the action for the next rule." See below for an illustration.
-.LP
+.PP
Actions can include arbitrary C code, including
.B return
statements to return a value to whatever routine called
will simply immediately return, unless
.B yyrestart()
is first called (see below).
-.LP
+.PP
Actions are not allowed to modify yytext or yyleng.
-.LP
+.PP
There are a number of special directives which can be included within
an action:
.IP -
if you give arguments to the scanning routine using a
K&R-style/non-prototyped function declaration, you must terminate
the definition with a semi-colon (;).
-.LP
+.PP
Whenever
.B yylex()
is called, it scans tokens from the global input file
In the latter case (i.e., when an action
executes a return), the scanner may then be called again and it
will resume scanning where it left off.
-.LP
+.PP
By default (and for purposes of efficiency), the scanner uses
block-reads rather than simple
.I getc()
number of characters read or the constant YY_NULL (0 on Unix systems)
to indicate EOF. The default YY_INPUT reads from the
global file-pointer "yyin".
-.LP
+.PP
A sample redefinition of YY_INPUT (in the definitions
section of the input file):
.nf
.fi
This definition will change the input processing to occur
one character at a time.
-.LP
+.PP
You also can add in things like keeping track of the
input line number this way; but don't expect your scanner to
go very fast.
-.LP
+.PP
When the scanner receives an end-of-file indication from YY_INPUT,
it then checks the
.B yywrap()
to point to another input file, and scanning continues. If it returns
true (non-zero), then the scanner terminates, returning 0 to its
caller.
-.LP
+.PP
The default
.B yywrap()
always returns 1. Presently, to redefine it you must first
"#undef yywrap", as it is currently implemented as a macro. As indicated
by the hedging in the previous sentence, it may be changed to
a true function in the near future.
-.LP
+.PP
The scanner writes its
.B ECHO
output to the
.fi
will be active only when the current start condition is
either "INITIAL", "STRING", or "QUOTE".
-.LP
+.PP
Start conditions
are declared in the definitions (first) section of the input
using unindented lines beginning with either
exclusive start conditions make it easy to specify "mini-scanners"
which scan portions of the input that are syntactically different
from the rest (e.g., comments).
-.LP
+.PP
If the distinction between inclusive and exclusive start conditions
is still a little vague, here's a simple example illustrating the
connection between the two. The set of rules:
<INITIAL,example>foo /* do something */
.fi
-.LP
+.PP
The default rule (to
.B ECHO
any unmatched character) remains active in start conditions.
-.LP
+.PP
.B BEGIN(0)
returns to the original state where only the rules with
no start conditions are active. This state can also be
.B BEGIN(0).
(The parentheses around the start condition name are not required but
are considered good style.)
-.LP
+.PP
.B BEGIN
actions can also be given as indented code at the beginning
of the rules section. For example, the following will cause
...more rules follow...
.fi
-.LP
+.PP
To illustrate the uses of start conditions,
here is a scanner which provides two different interpretations
of a string like "123.456". By default it will treat it as
is only called when the scanner reaches the end of its buffer, which
may be a long time after scanning a statement such as an "include"
which requires switching the input source.
-.LP
+.PP
To negotiate these sorts of problems,
.I flex
provides a mechanism for creating and switching between multiple
.I new_buffer.
Note that
.B yy_switch_to_buffer()
-may be used by yywrap() to sets things up for continued scanning, instead
+may be used by yywrap() to set things up for continued scanning, instead
of opening a new file and pointing
.I yyin
at it.
.fi
is used to reclaim the storage associated with a buffer.
-.LP
+.PP
.B yy_new_buffer()
is an alias for
.B yy_create_buffer(),
and
.I delete
for creating and destroying dynamic objects.
-.LP
+.PP
Finally, the
.B YY_CURRENT_BUFFER
macro returns a
.B YY_BUFFER_STATE
handle to the current buffer.
-.LP
+.PP
Here is an example of using these features for writing a scanner
which expands include files (the
.B <<EOF>>
or, switching to a new buffer using
.B yy_switch_to_buffer()
as shown in the example above.
-.LP
+.PP
<<EOF>> rules may not be used with other
patterns; they may only be qualified with a list of start
conditions. If an unqualified <<EOF>> rule is given, it
<INITIAL><<EOF>>
.fi
-.LP
+.PP
These rules are useful for catching things like unclosed comments.
An example:
.nf
can be redefined to provide an action
which is always executed prior to the matched rule's action. For example,
it could be #define'd to call a routine to convert yytext to lower-case.
-.LP
+.PP
The macro
.B YY_USER_INIT
may be redefined to provide an action which is always executed before
the first scan (and before the scanner's internal initializations are done).
For example, it could be used to call a routine to read
in a data table or open a logging file.
-.LP
+.PP
In the generated scanner, the actions are all gathered in one large
switch statement and separated using
.B YY_BREAK,
.B %t
provides a crude way for introducing equivalence classes into
the scanner specification.
-.LP
+.PP
Note that the
.B -i
option (see below) coupled with the equivalence classes which
run in
.I trace
mode. It will generate a lot of messages to
-.I stdout
+.I stderr
concerning
the form of the input and the resultant non-deterministic and deterministic
finite automata. This option is mostly for use in maintaining
pattern sets that require backtracking
arbitrary trailing context
- '^' beginning-of-line operator
yymore()
+ '^' beginning-of-line operator
.fi
with the first three all being quite expensive and the last two
being quite cheap.
-.LP
+.PP
.B REJECT
should be avoided at all costs when performance is important.
It is a particularly expensive option.
-.LP
+.PP
Getting rid of backtracking is messy and often may be an enormous
amount of work for a complicated scanner. In principal, one begins
by using the
state it's in when it has seen "fo". When this has happened,
if anything other than another 'o' is seen, the scanner will
have to back up to simply match the 'f' (by the default rule).
-.LP
+.PP
The comment regarding State #8 indicates there's a problem
when "foob" has been scanned. Indeed, on any character other
than a 'b', the scanner will have to back up to accept "foo".
Similarly, the comment for State #9 concerns when "fooba" has
been scanned.
-.LP
+.PP
The final comment reminds us that there's no point going to
all the trouble of removing backtracking from the rules unless
we're using
or
.B -F,
since there's no performance gain doing so with compressed scanners.
-.LP
+.PP
The way to remove the backtracking is to add "error" rules:
.nf
}
.fi
-.LP
+.PP
Eliminating backtracking among a list of keywords can also be
done using a "catch-all" rule:
.nf
.fi
This is usually the best solution when appropriate.
-.LP
+.PP
Backtracking messages tend to cascade.
With a complicated set of rules it's not uncommon to get hundreds
of messages. If one can decipher them, though, it often
a valid token. A possible future
.I flex
feature will be to automatically add rules to eliminate backtracking).
-.LP
+.PP
.I Variable
trailing context (where both the leading and trailing parts do not have
a fixed length) entails almost the same performance loss as
provide any savings, and can even make things worse (see
.B BUGS
in flex(1)).
-.LP
+.PP
Another area where the user can increase a scanner's performance
(and one that's easier to implement) arises from the fact that
the longer the tokens matched, the faster the scanner will run.
of the number of rules or (modulo the considerations given at the
beginning of this section) how complicated the rules are with
regard to operators such as '*' and '|'.
-.LP
+.PP
A final example in speeding up a scanner: suppose you want to scan
through a file containing identifiers and keywords, one per line
and with no other extraneous characters, and recognize all the
this is about as fast as one can get a
.I flex
scanner to go for this particular problem.
-.LP
+.PP
A final note:
.I flex
is slow when matching NUL's, particularly when a token contains
.I flex
may in the near future undergo changes incompatible with
its current definition.
-.LP
+.PP
.I flex
is fully compatible with
.I lex
.I flex,
the rule will be expanded to
"foo([A-Z][A-Z0-9]*)?" and so the string "foo" will match.
-Note that because of this, the
-.B ^, $, <s>, /,
+.PP
+Note that if the definition begins with
+.B ^
+or ends with
+.B $
+then it is
+.I not
+expanded with parentheses, to allow these operators to appear in
+definitions without losing their special meanings. But the
+.B <s>, /,
and
.B <<EOF>>
operators cannot be used in a
.I flex
or
.I lex.
-.LP
+.PP
The following
.I flex
features are not included in
does not truncate the action. Actions that are not enclosed in
braces are simply terminated at the end of the line.
.SH DIAGNOSTICS
+.I warning, rule cannot be matched
+indicates that the given rule
+cannot be matched because it follows other rules that will
+always match the same text as it. For
+example, in the following "foo" cannot be matched because it comes after
+an identifier "catch-all" rule:
+.nf
+
+ [a-z]+ got_identifier();
+ foo got_foo();
+
+.fi
+Using
+.B REJECT
+in a scanner suppresses this warning.
+.PP
+.I warning,
+.B -s
+.I option given but default rule
+.I can be matched
+means that it is possible (perhaps only in a particular start condition)
+that the default rule (match any single character) is the only one
+that will match a particular input. Since
+.B -s
+was given, presumably this is not intended.
+.PP
.I reject_used_but_not_detected undefined
or
.I yymore_used_but_not_detected undefined -
mechanism for dealing with this problem; this feature is still supported
but now deprecated, and will go away soon unless the author hears from
people who can argue compellingly that they need it.)
-.LP
+.PP
.I flex scanner jammed -
a scanner compiled with
.B -s
has encountered an input string which wasn't matched by
any of its rules.
-.LP
+.PP
.I flex input buffer overflowed -
a scanner rule matched a string long enough to overflow the
scanner's internal input buffer (16K bytes by default - controlled by
.B YY_BUF_SIZE
in "flex.skel". Note that to redefine this macro, you must first
-.B #undefine
+.B #undef
it).
-.LP
+.PP
.I scanner requires -8 flag -
Your scanner specification includes recognizing 8-bit characters and
you did not specify the -8 flag (and your site has not installed flex
with -8 as the default).
-.LP
+.PP
.I
fatal flex scanner internal error--end of buffer missed -
This can occur in an scanner which is reentered after a long-jump
yyrestart( yyin );
.fi
-.LP
+.PP
.I too many %t classes! -
You managed to put every single character into its own %t class.
.I flex
.SH DEFICIENCIES / BUGS
See flex(1).
.SH "SEE ALSO"
-.LP
+.PP
flex(1), lex(1), yacc(1), sed(1), awk(1).
-.LP
+.PP
M. E. Lesk and E. Schmidt,
.I LEX - Lexical Analyzer Generator
.SH AUTHOR
Van Jacobson. Original version by Jef Poskanzer. The fast table
representation is a partial implementation of a design done by Van
Jacobson. The implementation was done by Kevin Gong and Vern Paxson.
-.LP
+.PP
Thanks to the many
.I flex
beta-testers, feedbackers, and contributors, especially Casey
-Leedom, benson@odi.com, Keith Bostic,
+Leedom, benson@odi.com, Peter A. Bigot, Keith Bostic,
Frederic Brehm, Nick Christopher, Jason Coughlin,
Scott David Daniels, Leo Eskin,
Chris Faylor, Eric Goldman, Eric
Hughes, Jeffrey R. Jones, Kevin B. Kenny, Ronald Lamprecht,
-Greg Lee, Craig Leres, Mohamed el Lozy, Jim Meyering, Marc Nozell, Esmond Pitt,
-Jef Poskanzer, Jim Roskind,
+Greg Lee, Craig Leres, Mohamed el Lozy, Jim Meyering, Marc Nozell,
+Walter Pelissero, Francois Pinard, Esmond Pitt, Jef Poskanzer, Jim Roskind,
Dave Tallman, Frank Whaley, Ken Yap, and those whose names
have slipped my marginal mail-archiving skills but whose contributions
are appreciated all the same.
-.LP
+.PP
Thanks to Keith Bostic, John Gilmore, Craig Leres, Bob
Mulcahy, Rich Salz, and Richard Stallman for help with various distribution
headaches.
-.LP
+.PP
Thanks to Esmond Pitt and Earle Horton for 8-bit character support;
to Benson Margulies and Fred
Burke for C++ support; to Ove Ewerlid for the basics of support for
NUL's; and to Eric Hughes for the basics of support for multiple buffers.
-.LP
+.PP
Work is being done on extending
.I flex
to generate scanners in which the
using -f or -F. If you are working in this area and are interested
in comparing notes and seeing whether redundant work can be avoided,
contact Ove Ewerlid (ewerlid@mizar.DoCS.UU.SE).
-.LP
+.PP
This work was primarily done when I was at the Real Time Systems Group
at the Lawrence Berkeley Laboratory in Berkeley, CA. Many thanks to all there
for the support I received.
-.LP
+.PP
Send comments to:
.nf
Vern Paxson
- Computer Science Department
- 4126 Upson Hall
- Cornell University
- Ithaca, NY 14853-7501
-
- vern@cs.cornell.edu
- decvax!cornell!vern
+ Computer Systems Engineering
+ Bldg. 46A, Room 1123
+ Lawrence Berkeley Laboratory
+ University of California
+ Berkeley, CA 94720
+
+ vern@ee.lbl.gov
+ ucbvax!ee.lbl.gov!vern
.fi
projects / flex / commitdiff