--- /dev/null
+.TH FLEX 1 "24 February 1990" "Version 2.2"
+.SH NAME
+flex - fast lexical analyzer generator
+.SH SYNOPSIS
+.B flex
+.B [-bcdfinpstvFILT -C[efmF] -Sskeleton]
+.I [filename ...]
+.SH DESCRIPTION
+.I flex
+is a tool for generating
+.I scanners:
+programs which recognized lexical patterns in text.
+.I flex
+reads
+the given input files, or its standard input if no file names are given,
+for a description of a scanner to generate. The description is in
+the form of pairs
+of regular expressions and C code, called
+.I rules. flex
+generates as output a C source file,
+.B lex.yy.c,
+which defines a routine
+.B yylex().
+This file is compiled and linked with the
+.B -ll
+library to produce an executable. When the executable is run,
+it analyzes its input for occurrences
+of the regular expressions. Whenever it finds one, it executes
+the corresponding C code.
+.SH SOME SIMPLE EXAMPLES
+.LP
+First some simple examples to get the flavor of how one uses
+.I flex.
+The following
+.I flex
+input specifies a scanner which whenever it encounters a tab
+will print eight blanks to its standard output:
+.nf
+
+ %%
+ \t printf( " " );
+
+.fi
+By default, any text not matched by a
+.I flex
+scanner
+is copied to the output, so the net effect of this scanner is
+to copy its input file to its output with each tab expanded
+into eight blanks.
+In this input, there is just one rule. "\t" is the
+.I pattern
+(it's a regular expression specifying a tab) and the "printf" is the
+.I action. The "%%" marks the beginning of the rules.
+.LP
+Here's another simple example:
+.nf
+
+ int num_lines = 0, num_chars = 0;
+
+ %%
+ \n ++num_lines; ++num_chars;
+ . ++num_chars;
+
+ %%
+ main()
+ {
+ yylex();
+ printf( "# of lines = %d, # of chars = %d\n",
+ num_lines, num_chars );
+ }
+
+.fi
+This scanner counts the number of characters and the number
+of lines in its input (it produces no output other than the
+final report on the counts). The first line
+declares two globals, "num_lines" and "num_chars", which are accessible
+both inside
+.B yylex()
+and in the
+.B main()
+routine declared after the second "%%". There are two rules, one
+which matches a newline ("\\n") and increments both the line count and
+the character count, and one which matches any character other than
+a newline (indicated by the "." regular expression).
+.LP
+A somewhat more complicated example:
+.nf
+
+ /* scanner for a toy Pascal-like language */
+
+ %{
+ /* need this for the call to atof() below */
+ #include <math.h>
+ %}
+
+ DIGIT [0-9]
+ ID [a-z][a-z0-9]*
+
+ %%
+
+ {DIGIT}+ {
+ printf( "An integer: %s (%d)\\n", yytext,
+ atoi( yytext ) );
+ }
+
+ {DIGIT}+"."{DIGIT}* {
+ printf( "A float: %s (%d)\\n", yytext,
+ atof( yytext ) );
+ }
+
+ if|then|begin|end|procedure|function {
+ printf( "A keyword: %s\\n", yytext );
+ }
+
+ {ID} printf( "An identifier: %s\\n", yytext );
+
+ "+"|"-"|"*"|"/" printf( "An operator: %s\\n", yytext );
+
+ "{"[^}\\n]*"}" /* eat up one-line comments */
+
+ [ \\t\\n]+ /* eat up whitespace */
+
+ . printf( "Unrecognized character: %s\\n", yytext );
+
+ %%
+
+ main( argc, argv )
+ int argc;
+ char **argv;
+ {
+ ++argv, --argc;
+ if ( argc > 0 )
+ yyin = fopen( argv[0], "r" );
+ else
+ yyin = stdin;
+
+ yylex();
+ }
+
+.fi
+This is the beginnings of a simple scanner for a language like
+Pascal. It identifies different types of
+.I tokens
+and reports on what it has seen.
+.LP
+The details of the example will be explained in the following
+sections.
+.SH FORMAT OF THE INPUT FILE
+The
+.I flex
+input file consists of three sections, separated by
+.B %%:
+.nf
+
+ definitions
+ %%
+ rules
+ %%
+ user code
+
+.fi
+The
+.I definitions
+section contains declarations of simple
+.I name
+definitions to simplify the scanner specification and of
+.I start conditions,
+which are explained in a later section.
+.LP
+Name definitions have the form:
+.nf
+
+ name definition
+
+.fi
+The "name" is a word beginning with a letter or a '_'
+followed by zero or more letters, digits, '_', or '-'.
+The definition is taken to begin at the first non-white-space
+following the name and continue to the end of the line.
+Definition can subsequently be referred to using "{name}", which
+will expand to "(definition)". For example,
+.nf
+
+ DIGIT [0-9]
+ ID [a-z][a-z0-9]*
+
+.fi
+defines "DIGIT" to be a regular expression which matches a
+single digit, and
+"ID" to be a regular expression which matches a letter
+followed by zero-or-more letters or digits.
+A subsequent reference to
+.nf
+
+ {DIGIT}+"."{DIGIT}*
+
+.fi
+is identical to
+.nf
+
+ ([0-9])+"."([0-9])*
+
+.fi
+and matches one-or-more digits followed by a '.' followed
+by zero-or-more digits.
+.LP
+The
+.I rules
+section of the
+.I flex
+input contains a series of rules of the form:
+.nf
+
+ pattern action
+
+.fi
+where the pattern must be unindented and the action must begin
+on the same line.
+.LP
+See below for a further description of patterns and actions.
+.LP
+Finally, the user code section is simply copied to
+.B lex.yy.c
+verbatim.
+It is used for companion routines which call or are called
+by the scanner. The presence of this section is optional;
+if it is missing, the second
+.B %%
+in the input file may be skipped, too.
+.LP
+In the definitions and rule sections, any
+.I indented
+text or text enclosed in
+.B %{
+and
+.B %}
+is copied verbatim to the output (with the %{}'s removed).
+The %{}'s must appear unindented on lines by themselves.
+.LP
+In the rules section,
+any indented or %{} text appearing before the
+first rule may be used to declare variables
+which are local to the scanning routine, and, after the declarations,
+code which is to be executed whenever the scanning routine is entered.
+Other indented or %{} text in the rule section is still copied to the output,
+but its meaning is not well-defined and it may well cause compile-time
+errors (this feature is present for
+.I POSIX
+compliance; see below for other such features).
+.LP
+In the definitions section, an unindented comment (i.e., a line
+beginning with "/*") is also copied verbatim to the output up
+to the next "*/". Also, any line beginning with '#' is ignored.
+.SH PATTERNS
+The patterns in the input are written using an extended set of regular
+expressions. These are:
+.nf
+
+ x match the character 'x'
+ . any character except newline
+ [xyz] an 'x', a 'y', or a 'z'
+ [abj-oZ] an 'a', a 'b', any letter
+ from 'j' through 'o', or a 'Z'
+ [^A-Z] any character EXCEPT an uppercase letter,
+ including a newline (unlike how many other
+ regular expression tools treat the '^'!).
+ This means that a pattern like [^"]* will
+ match an entire file (overflowing the input
+ buffer) unless there's another quote in
+ the input.
+ [^A-Z\\n] any character EXCEPT an uppercase letter or
+ a newline
+ r* zero or more r's, where r is any regular expression
+ r+ one or more r's
+ r? zero or one r's (that is, "an optional r")
+ r{2,5} anywhere from two to five r's
+ r{2,} two or more r's
+ r{4} exactly 4 r's
+ {name} the expansion of the "name" definition
+ (see above)
+ "[xyz]\\"foo"
+ the literal string: [xyz]"foo
+ \\x if x is an 'a', 'b', 'f', 'n', 'r',
+ 't', or 'v', then the ANSI-C
+ interpretation of \\x. Otherwise,
+ a literal 'x' (used to escape
+ operators such as '*')
+ \\123 the character with octal value 123
+ \\x2a the character with hexadecimal value 2a
+ (r) match an r; parentheses are used
+ to override precedence (see below)
+
+
+ rs the regular expression r followed
+ by the regular expression s; called
+ "concatenation"
+
+
+ r|s either an r or an s
+
+
+ r/s an r but only if it is followed by
+ an s. The s is not part of the
+ matched text. This type of
+ pattern is known as "trailing context".
+ ^r an r, but only at the beginning of a line
+ r$ an r, but only at the end of a line
+ (r must not use trailing context)
+
+
+ <s>r an r, but only in start condition s (see
+ below for discussion of start conditions)
+ <s1,s2,s3>r
+ same, but in any of start conditions s1,
+ s2, or s3
+
+
+ <<EOF>> an end-of-file
+ <s1,s2><<EOF>>
+ an end-of-file when in start condition s1 or s2
+
+.fi
+The regular expressions listed above are grouped according to
+precedence, from highest precedence at the top to lowest at the bottom.
+Those grouped together have equal precedence. For example,
+.nf
+
+ foo|bar*
+
+.fi
+is the same as
+.nf
+
+ (foo)|(ba(r*))
+
+.fi
+since the '*' operator has higher precedence than concatenation,
+and concatenation higher than alternation ('|'). This pattern
+therefore matches
+.I either
+the string "foo"
+.I or
+the string "ba" followed by zero-or-more r's.
+To match "foo" or zero-or-more "bar"'s, use:
+.nf
+
+ foo|(bar)*
+
+.fi
+and to match zero-or-more "foo"'s or "bar"'s:
+.nf
+
+ (foo|bar)*
+
+.fi
+.SH HOW THE INPUT IS MATCHED
+When the generated scanner is run, it analyzes its input looking
+for strings which match any of its patterns. If it finds more than
+one match, it takes the one matching the most text (for trailing
+context rules, this includes the length of the trailing part, even
+though it will then be returned to the input). If it finds two
+or more matches of the same length, the
+rule listed first in the
+.I flex
+input file is chosen.
+.LP
+Once the match is determined, the text corresponding to the match
+(called the
+.I token)
+is made available in the global character pointer
+.B yytext,
+and its length in the global integer
+.B yyleng.
+The
+.I action
+corresponding to the matched pattern is then executed (a more
+detailed description of actions follows), and then the remaining
+input is scanned for another match.
+.LP
+If no match is found, then the
+.I default rule
+is executed: the next character in the input is matched and
+copied to the standard output. Thus, the simplest legal
+.I flex
+input is:
+.nf
+
+ %%
+
+.fi
+which generates a scanner that simply copies its input (one character
+at a time) to its output.
+.SH ACTIONS
+Each pattern in a rule has a corresponding action, which can be any
+arbitrary C statement. The pattern ends at the first non-escaped
+whitespace character; the remainder of the line is its action. If the
+action is empty, then when the pattern is matched the input token
+is simply discarded. For example, here is the specification for a program
+which deletes all occurrences of "zap me" from its input:
+.nf
+
+ %%
+ "zap me"
+
+.fi
+Here is a program which compresses multiple blanks and tabs down to
+a single blank, and throws away whitespace found at the end of a line:
+.nf
+
+ %%
+ [ \t]+ putchar( ' ' );
+ [ \t]+$ /* ignore this token */
+
+.fi
+.LP
+If the action contains a '{', then the action spans till the balancing
+'}' is found, and the action may cross multiple lines.
+.I flex
+knows about C strings and comments and won't be fooled by braces found
+within them, but also allows actions to begin with
+.B %{
+and will consider the action to be all the text up to the next
+.B %}.
+.LP
+An action consisting solely of a vertical bar ('|') means "same as
+the action for the next rule. See below for an illustration.
+.LP
+Actions can include arbitrary C code, including
+.B return
+statements to return a value whatever routine called
+.B yylex().
+Each time
+.B yylex()
+is called it continues processing tokens from where it last left
+off until it either reaches
+the end of the file or executes a return.
+.LP
+Actions are not allowed to modify yytext or yyleng.
+.LP
+There are a number of special directives which can be included within
+an action:
+.IP -
+.B ECHO
+copies yytext to the scanner's output.
+.IP -
+.B BEGIN
+followed by the name of a start condition places the scanner in the
+corresponding start condition (see below).
+.IP -
+.B REJECT
+directs the scanner to proceed on to the "second best" rule which matched the
+input (or a prefix of the input). The rule is chosen as described
+above in "How the Input is Matched", and
+.B yytext
+and
+.B yyleng
+set up appropriately.
+It may either be one which matched as much text
+as the originally chosen rule but came later in the
+.I flex
+input file, or one which matched less text.
+For example, the following will both count the
+words in the input and call the routine special() whenever "frob" is seen:
+.nf
+
+ %{
+ int word_count = 0;
+ %}
+ %%
+
+ frob special(); REJECT;
+ [^ \t\n]+ ++word_count;
+
+.fi
+Without the
+.B REJECT,
+any "frob"'s in the input would not be counted as words, since the
+scanner normally executes only one action per token.
+Multiple
+.B REJECT's
+are allowed, each one finding the next best choice to the currently
+active rule. For example, when the following scanner scans the token
+"abcd", it will write "abcdabcaba" to the output:
+.nf
+
+ %%
+ a |
+ ab |
+ abc |
+ abcd ECHO; REJECT;
+ .|\n /* eat up any unmatched character */
+
+.fi
+(The first three rules share the fourth's action since they use
+the special '|' action.)
+.B REJECT
+is a particularly expensive feature in terms scanner performance;
+if it is used in
+.I any
+of the scanner's actions it will slow down
+.I all
+of the scanner's matching.
+.IP -
+.B yymore()
+tells the scanner that the next time it matches a rule, the corresponding
+token should be
+.I appended
+onto the current value of
+.B yytext
+rather than replacing it. For example, given the input "mega-kludge"
+the following will write "mega-mega-kludge" to the output:
+.nf
+
+ %%
+ mega- ECHO; yymore();
+ kludge ECHO;
+
+.fi
+First "mega-" is matched and echoed to the output. Then "kludge"
+is matched, but the previous "mega-" is still hanging around at the
+beginning of
+.B yytext
+so the ECHO for the "kludge" rule will actually write "mega-kludge".
+The presence of
+.B yymore()
+in the scanner's action entails a minor performance penalty in the
+scanner's matching speed.
+.IP -
+.B yyless(n)
+returns all but the first
+.I n
+characters of the current token back to the input stream, where they
+will be rescanned when the scanner looks for the next match.
+.B yytext
+and
+.B yyleng
+are adjusted appropriately (e.g.,
+.B yyleng
+will now be equal to
+.I n
+). For example, on the input "foobar" the following will write out
+"foobarbar":
+.nf
+
+ foobar ECHO; yyless(3);
+ [a-z]+ ECHO;
+
+.fi
+An argument of 0 to
+.B yyless
+will cause the current input string to be scanned again. Unless you've
+changed how the scanner will subsequently process its input (using
+.B BEGIN,
+for example), this will result in an endless loop.
+.IP -
+.B unput(c)
+puts the character
+.I c
+back onto the input stream. It will be the next character scanned.
+The following action will take the current token and cause it
+to be rescanned enclosed in parentheses.
+.nf
+
+ {
+ int i;
+ unput( ')' );
+ for ( i = yyleng - 1; i >= 0; --i )
+ unput( yytext[i] );
+ unput( '(' );
+ }
+
+.fi
+Note that since each
+.B unput()
+puts the given character back at the
+.I beginning
+of the input stream, pushing back strings must be done back-to-front.
+.IP -
+.B input()
+reads the next character from the input stream. For example,
+the following is one way to eat up C comments:
+.nf
+
+ %%
+ "/*" {
+ register int c;
+
+ for ( ; ; )
+ {
+ while ( (c = input()) != '*' &&
+ c != EOF )
+ ; /* eat up text of comment */
+
+ if ( c == '*' )
+ {
+ while ( (c = input()) == '*' )
+ ;
+ if ( c == '/' )
+ break; /* found the end */
+ }
+
+ if ( c == EOF )
+ {
+ error( "EOF in comment" );
+ break;
+ }
+ }
+ }
+
+.fi
+.IP -
+.I yyterminate()
+can be used in lieu of a return statement in an action. It terminates
+the scanner and returns a 0 to the scanner's caller, indicating "all done".
+By default,
+.I yyterminate()
+is also called when an end-of-file is encountered. It is a macro and
+may be redefined.
+.SH THE GENERATED SCANNER
+The output of
+.I flex
+is the file
+.B lex.yy.c,
+which contains the scanning routine
+.B yylex(),
+a number of tables used by it for matching tokens, and a number
+of auxilliary routines and macros. By default,
+.B yylex()
+is declared as follows:
+.nf
+
+ int yylex()
+ {
+ ... various definitions and the actions in here ...
+ }
+
+.fi
+(If your environment supports function prototypes, then it will
+be "int yylex( void )".) This definition may be changed by redefining
+the "YY_DECL" macro. For example, you could use:
+.nf
+
+ #undef YY_DECL
+ #define YY_DECL float lexscan( a, b ) float a, b;
+
+.fi
+to give it the the scanning routine the name
+.I lexscan,
+returning a float, and taking two floats as arguments. Note that
+if you give arguments to the scanning routine using a
+K&R-style/non-prototyped function declaration, you must terminate
+the definition with a semi-colon (;).
+.LP
+Whenever
+.B yylex()
+is called, it scans tokens from the global input file
+.I yyin
+(default, stdin). It continues until it either reaches
+an end-of-file (at which point it returns the value 0) or
+one of its actions executes a
+.I return
+statement.
+In the former case, the scanner may not be called again unless
+.B void yyrestart( FILE *input_file )
+is called, to point
+.I yyin
+at the new input_file. In the latter case (i.e., when an action
+executes a return), the scanner may then be called again and it
+will resume scanning where it left off.
+.LP
+By default (and for purposes of efficiency), the scanner uses
+block-reads rather than simple
+.I getc()
+calls to read characters from
+.I yyin.
+The nature of how it gets its input can be controlled by redefining the
+.B YY_INPUT
+macro.
+YY_INPUT's calling sequence is "YY_INPUT(buf,result,max_size)". Its
+action is to place up to
+.I max_size
+characters in the character array
+.I buf
+and return in the integer variable
+.I result
+either the
+number of characters read or the constant YY_NULL (0 on Unix systems)
+to indicate EOF. The default YY_INPUT reads from the
+global file-pointer "yyin" (which is by default
+.I stdin),
+so if you
+just want to change the input file, you needn't redefine
+YY_INPUT - just point yyin at the input file.
+.LP
+A sample redefinition of YY_INPUT (in the definitions section of the input
+file):
+.nf
+
+ %{
+ #undef YY_INPUT
+ #define YY_INPUT(buf,result,max_size) \\
+ result = ((buf[0] = getchar()) == EOF) ? YY_NULL : 1;
+ %}
+
+.fi
+You also can add in things like keeping track of the
+input line number this way; but don't expect your scanner to
+go very fast.
+.LP
+When the scanner receives an end-of-file indication from YY_INPUT,
+it then checks the
+.B yywrap()
+function. If it returns false (zero), then it is assumed that the
+function has gone ahead and set up
+.I yyin
+to point to another input file, and scanning continues. If it returns
+true (non-zero), then the scanner terminates, returning 0 to its
+caller.
+.LP
+The default
+.B yywrap()
+always returns 1. Presently, to redefine it you must first
+"#undef yywrap", as it is currently implemented as a macro. As noted
+by the hedging in the previous sentence, it may be changed to
+a true function in the near future.
+.LP
+The scanner writes its
+.B ECHO
+output to the
+.I yyout
+global (default, stdout), which may be redefined by the user simply
+by assigning it to some other FILE*.
+.SH START CONDITIONS
+.I flex
+provides a mechanism for conditionally activating rules. Any rule
+whose pattern is prefixed with "<sc>" will only be active when
+the scanner is in the start condition named "sc". For example,
+.nf
+
+ <STRING>[^"]* { /* eat up the string body ... */
+ ...
+ }
+
+.fi
+will be active only when the scanner is in the "STRING" start
+condition, and
+.nf
+
+ <INITIAL,STRING,QUOTE>\\. { /* handle an escape ... */
+ ...
+ }
+
+.fi
+will be active only when the current start condition is
+either "INITIAL", "STRING", or "QUOTE".
+.LP
+Start conditions
+are declared in the definitions (first) section of the input
+using unindented lines beginning with either
+.B %s
+or
+.B %x
+followed by a list of names.
+The former declares
+.I inclusive
+start conditions, the latter
+.I exclusive
+start conditions. A start condition is activated using the
+.B BEGIN
+action. Until the next
+.B BEGIN
+action is executed, rules with the given start
+condition will be active and
+rules with other start conditions will be inactive.
+If the start condition is
+.I inclusive,
+then rules with no start conditions at all will also be active.
+If it is
+.I exclusive,
+then
+.I only
+rules qualified with the start condition will be active.
+So a set of rules conditioned on the same exclusive start condition
+describe a scanner which is independent of any of the other rules in the
+.I flex
+input. Because of this,
+exclusive start conditions make it easy to specify "mini-scanners"
+which scan portions of the input that are syntactically different
+from the rest (e.g., comments).
+.LP
+.B BEGIN(0)
+returns to the original state where only the rules with
+no start conditions are active. This state can also be
+referred to as the start-condition "INITIAL", so
+.B BEGIN(INITIAL)
+is equivalent to
+.B BEGIN(0).
+.LP
+Here is a scanner which will recognize numbers only if they
+are preceded earlier in the line by the string "expect-number":
+.nf
+
+ %s expect
+
+ %%
+ expect-number BEGIN(expect);
+
+ <expect>[0-9]+ printf( "found a number\n" );
+ <expect>\n {
+ /* that's the end of the line, so
+ * we need another "expect-number"
+ * before we'll recognize any more
+ * numbers
+ */
+ BEGIN(INITIAL);
+ }
+
+.fi
+Here is a scanner which recognizes (and discards) C comments while
+maintaining a count of the current input line.
+.nf
+
+ %x comment
+ %%
+ int line_num = 1;
+
+ <comment>[^*\n]*
+ <comment>"*"+[^*/\n]*
+ <comment>\n ++line_num;
+ <comment>"*"+"/" BEGIN(INITIAL);
+
+ "/*" BEGIN(comment);
+
+.fi
+Note that start-conditions names are really integer values and
+can be stored as such. Thus, the above could be extended in the
+following fashion:
+.nf
+
+ %x comment
+ %%
+ int line_num = 1;
+ int comment_caller;
+
+ <comment>[^*\n]*
+ <comment>"*"+[^*/\n]*
+ <comment>\n ++line_num;
+ <comment>"*"+"/" BEGIN(comment_caller);
+
+ "/*" {
+ comment_caller = INTIIAL;
+ BEGIN(comment);
+ }
+
+ ...
+
+ <foo>"/*" {
+ comment_caller = foo;
+ BEGIN(comment);
+ }
+.fi
+One can then implement a "stack" of start conditions using an
+array of integers. (It is likely that such stacks will become
+a full-fledged
+.I flex
+feature in the future.) Note, though, that
+start conditions do not have their own name-space; %s's and %x's
+declare names effectively the same as #define's.
+.SH END-OF-FILE RULES
+The special rule "<<EOF>>" indicates
+actions which are to be taken when an end-of-file is
+encountered and yywrap() returns non-zero (i.e., indicates
+no further files to process). The action can either
+point yyin at a new file to process, in which case the
+action
+.I must
+finish with the special
+.I YY_NEW_FILE
+action
+(this is a branch, so subsequent code in the action won't
+be executed), or the action must finish with a
+.I return
+or
+.I yyterminate()
+statement. <<EOF>> rules may not be used with other
+patterns; they may only be qualified with a list of start
+conditions. If an unqualified <<EOF>> rule is given, it
+applies only to the
+.B INITIAL
+start condition, and
+.I not
+to
+.B %s
+(or
+.B %x)
+start conditions.
+.LP
+These rules are useful for catching things like unclosed comments.
+An example:
+.nf
+
+ %x quote
+ %%
+ ...
+ <quote><<EOF>> {
+ error( "unterminated quote" );
+ yyterminate();
+ }
+ <<EOF>> {
+ if ( *++filelist )
+ {
+ yyin = fopen( *filelist, "r" );
+ YY_NEW_FILE;
+ }
+ else
+ yyterminate();
+ }
+
+.fi
+.SH MISCELLANEOUS MACROS
+The macro
+.bd
+YY_USER_ACTION
+can be redefined to provide an action
+which is always executed prior to the matched rule's action. For example,
+it could be #define'd to call a routine to convert yytext to lower-case.
+.LP
+In the generated scanner, the actions are all gathered in one large
+switch statement and separated using
+.B YY_BREAK,
+which may be redefined.
+This allows, for example, C++ users to
+#define YY_BREAK to do nothing (while being very careful that every
+rule ends with a "break" or a "return"!) to avoid suffering from
+unreachable statement warnings where a rule's action ends with "return".
+.SH INTERFACING WITH YACC
+One of the main uses of
+.I flex
+is as a companion to the
+.I yacc
+parser-generator.
+.I yacc
+parsers expect to call the
+.B yylex()
+routine to find the next input token. The routine is supposed to
+return the type of the next token as well as putting any associated
+value in the global
+.B yylval.
+To use
+.I flex
+with
+.I yacc,
+one specifies the
+.B -d
+option to
+.I yacc
+to instruct it to generate the file
+.B y.tab.h
+containing definitions of all the
+.B %tokens
+appearing in the
+.I yacc
+input. This file is then included in the
+.I flex
+scanner. For example, if one of the tokens is "TOK_NUMBER",
+part of the scanner might look like:
+.nf
+
+ %{
+ #include "y.tab.h"
+ %}
+
+ %%
+
+ [0-9]+ yylval = atoi( yytext ); return TOK_NUMBER;
+
+.fi
+.SH TRANSLATION TABLE
+In the name of POSIX compliance,
+.I flex
+supports a
+.I translation table
+for mapping input characters together into specified sets.
+The table is specified in the first section, and its format looks like:
+.nf
+
+ %t
+ 1 abcd
+ 2 ABCDEFGHIJKLMNOPQRSTUVWXYZ
+ 52 0123456789
+ %t
+
+.fi
+This example specifies that the characters 'a', 'b', 'c', and 'd'
+are to all be lumped into group #1, the upper-case letters are
+to be in group #2, and digits in group #52, and
+.I no other characters will appear in the patterns
+(note that characters can also be specified in a
+.B %t
+table using escape sequences).
+The group numbers are actually disregarded by
+.I flex;
+.B %t
+serves, though, to lump characters together. Given the above
+table, for example, the pattern "aAA*5" is equivalent to "dZQ*0".
+They both say, "match any character in group #1, followed by
+a character from group #2, followed by zero-or-more characters
+from group #2, followed by a character from group #52." Thus
+.B %t
+provides a crude way for introducing equivalence classes into
+the scanner specification. It is the author's belief that the
+.B -i
+option coupled with the equivalence classes which
+.I flex
+automatically generates take care of virtually all the instances
+when one might consider using
+.B %t.
+But what the hell, it's there if you want it.
+.so options.man
+.SH PERFORMANCE CONSIDERATIONS
+The main design goal of
+.I flex
+is that it generate high-performance scanners. It has been optimized
+for dealing well with large sets of rules. Aside from the effects
+outlined above of table compression on scanner speed,
+there are a number of options/actions which degrade performance. These
+are, in decreasing order of performance impact:
+.nf
+
+ REJECT
+ pattern sets that require backtracking
+ arbitrary trailing context
+ %T
+ '^' beginning-of-line operator
+ yymore()
+ start conditions
+
+.fi
+.LP
+Getting rid of backtracking is messy and often may be too much
+work for a complicated scanner's rules. In principal, one begins
+by using the
+.B -b
+flag to generate a
+.I lex.backtrack
+file. For example, on the input
+.nf
+
+ %%
+ foo return TOK_KEYWORD;
+ foobar return TOK_KEYWORD;
+
+.fi
+the file looks like:
+.nf
+
+ State #6 is non-accepting -
+ associated rules:
+ 2 3
+ out-transitions: [ o ]
+ jam-transitions: EOF [ \001-n p-\177 ]
+
+ State #8 is non-accepting -
+ associated rules:
+ 3
+ out-transitions: [ a ]
+ jam-transitions: EOF [ \001-` b-\177 ]
+
+ State #9 is non-accepting -
+ associated rules:
+ 3
+ out-transitions: [ r ]
+ jam-transitions: EOF [ \001-q s-\177 ]
+
+ Compressed tables always backtrack.
+
+.fi
+The first few lines tell us that there's a scanner state in
+which it can make a transition on an 'o' but not on any other
+character, and the currently scanned text does not match any rule.
+If the scanner is in that state and then reads
+something other than an 'o', it will have to backtrack to find
+a rule which is matched. With
+a bit of headscratching one can see that this must be the
+state it's in when it has seen "fo". When this has happened,
+if anything other than another 'o' is seen, the scanner will
+have to back up to simply match the 'f' (by the default rule).
+.LP
+The comment regarding State #8 indicates there's a problem
+when "foob" has been scanned. Indeed, on any character other
+than a 'b', the scanner will have to back up to accept "foo".
+Similarly, the comment for State #9 concerns when "fooba" has
+been scanned.
+.LP
+The final comment reminds us that there's no point going to
+all the trouble of removing backtracking from the rules unless
+we're using
+.B -f
+or
+.B -F,
+since there's no performance gain doing so with compressed scanners.
+.LP
+The way to remove the backtracking is to add "error" rules:
+.nf
+
+ %%
+ foo return TOK_KEYWORD;
+ foobar return TOK_KEYWORD;
+
+ fooba |
+ foob |
+ fo {
+ /* false alarm, not really a keyword */
+ return TOK_ID;
+ }
+
+.fi
+.LP
+Unfortunately backtracking messages tend to cascade and
+with a complicated input set it's not uncommon to get hundreds
+of messages. If one can decipher them, though, it often
+only takes a dozen or so rules to eliminate the backtracking.
+(A possible future
+.I flex
+feature will be to automatically add rules to eliminate backtracking.
+The problem is that while it's easy for
+.I flex
+to figure out what rules are needed, it's very hard for it to
+know what the proper action is. Currently I'm thinking that it
+will simply invoke a user-redefinable macro and that's it ...)
+.LP
+Another area where the user can increase a scanner's performance
+(and one that's easier to implement) arises from the fact that
+the longer the tokens matched, the faster the scanner will run.
+This is because with long tokens the processing of most input
+characters takes place in the (short) inner scanning loop, and
+does not often have to go through the additional work of setting up
+the scanning environment (e.g.,
+.B yytext)
+for the action. Recall the scanner for C comments:
+.nf
+
+ %x comment
+ %%
+ int line_num = 1;
+
+ <comment>[^*\n]*
+ <comment>"*"+[^*/\n]*
+ <comment>\n ++line_num;
+ <comment>"*"+"/" BEGIN(INITIAL);
+
+ "/*" BEGIN(comment);
+
+.fi
+This could be sped up by writing it as:
+.nf
+
+ %x comment
+ %%
+ int line_num = 1;
+
+ <comment>[^*\n]*
+ <comment>[^*\n]*\n ++line_num;
+ <comment>"*"+[^*/\n]*
+ <comment>"*"+[^*/\n]*\n ++line_num;
+ <comment>"*"+"/" BEGIN(INITIAL);
+
+ "/*" BEGIN(comment);
+
+.fi
+Now instead of each newline requiring the processing of another
+action, recognizing the newlines is "distributed" over the other rules
+to keep the matched text as long as possible. Note that
+.I adding
+rules does
+.I not
+slow down the scanner! The speed of the scanner is independent
+of the number of rules or (modulo the considerations given at the
+beginning of this section) how complicated the rules are with
+regard to operators such as '*' and '|'.
+.SH INCOMPATIBILITIES WITH LEX AND POSIX
+.I flex
+is a rewrite of the Unix
+.I lex
+tool (the two implementations do not share any code, though),
+which dates to the late 1970's. There are some incompatibilities
+which are of concern to those who wish to write scanners acceptable
+to either implementation. At present, the POSIX lex draft is
+very close to the original lex implementation, so some of these
+incompatibilities are also in conflict with the POSIX draft. But
+the intent is that except as noted below,
+.I flex
+as it presently stands will
+ultimately be POSIX comformant (i.e., that those areas of conflict with
+the POSIX draft will be resolved in
+.I flex's
+favor). Please bare in
+mind that all the comments are with regard to the POSIX
+.I draft
+standard and not the final document; they are included so
+.I flex
+users can be aware of the standardization issues and those areas where
+.I flex
+may in the near future be incompatibly changed with
+its current definition.
+.LP
+.I flex
+is fully compatible with
+.I lex
+with the following exceptions:
+.IP -
+When definitions are expanded,
+.I flex
+encloses them in parentheses.
+With lex, the following
+.nf
+
+ NAME [A-Z][A-Z0-9]*
+ %%
+ foo{NAME}? printf( "Found it\\n" );
+ %%
+
+.fi
+will not match the string "foo" because when the macro
+is expanded the rule is equivalent to "foo[A-Z][A-Z0-9]*?"
+and the precedence is such that the '?' is associated with
+"[A-Z0-9]*". With
+.I flex,
+the rule will be expanded to
+"foo([A-z][A-Z0-9]*)?" and so the string "foo" will match.
+Note that because of this, the
+.B ^, $, <s>,
+and
+.B /
+operators cannot be used in a definition.
+.IP
+Note that the POSIX draft interpretation here is the same as
+.I flex's.
+.IP -
+The undocumented lex-scanner internal variable
+.B yylineno
+is not supported. (The variable is not part of the POSIX draft.)
+.IP -
+The
+.B input()
+routine is not redefinable, though may be called to read characters
+following whatever has been matched by a rule. If
+.B input()
+encounters an end-of-file the normal
+.B yywrap()
+processing is done. A ``real'' end-of-file is returned as
+.I EOF.
+.IP
+Input is instead controlled by redefining the
+.B YY_INPUT
+macro.
+.IP
+The
+.I flex
+restriction that
+.B input()
+cannot be redefined is in accordance with the POSIX draft, but
+.B YY_INPUT
+has not yet been accepted into the draft.
+.IP -
+.B output()
+is not supported.
+Output from the ECHO macro is done to the file-pointer
+"yyout" (default
+.I stdout).
+.IP
+The POSIX draft mentions that an
+.B output()
+routine exists but currently gives no details as to what it does.
+.IP -
+If you are providing your own yywrap() routine, you must include a
+"#undef yywrap" in the definitions section (section 1). Note that
+the "#undef" will have to be enclosed in %{}'s.
+.IP
+The POSIX draft
+specifies that yywrap() is a function and this is unlikely to change; so
+.I flex users are warned
+that
+.I yywrap()
+is likely to be changed to a function in the near future.
+.IP -
+The precedence of the
+.B {}
+operator is different. lex interprets "abc{1,3}" as "match one, two, or
+three occurrences of 'abc'", whereas
+.I flex
+interprets it as "match 'ab'
+followed by one, two, or three occurrences of 'c'". The latter is
+in agreement with the current POSIX draft.
+.IP -
+To refer to yytext outside of your scanner source file, use
+"extern char *yytext;" rather than "extern char yytext[];".
+This is contrary to the POSIX draft but a point on which I refuse
+to budge, as the array representation entails a serious performance penalty.
+.IP -
+The name
+.bd
+FLEX_SCANNER
+is #define'd so scanners may be written for use with either
+.I flex
+or
+.I lex.
+.SH DEFICIENCES / BUGS
+.LP
+Some trailing context
+patterns cannot be properly matched and generate
+warning messages ("Dangerous trailing context"). These are
+patterns where the ending of the
+first part of the rule matches the beginning of the second
+part, such as "zx*/xy*", where the 'x*' matches the 'x' at
+the beginning of the trailing context. (Note that the POSIX draft
+states that the text matched by such patterns is undefined.)
+If desperate, you can use
+.B yyless()
+to effect arbitrary trailing context.
+.LP
+.I variable
+trailing context (where both the leading and trailing parts do not have
+a fixed length) entails the same performance loss as
+.I REJECT
+(i.e., substantial).
+.LP
+For some trailing context rules, parts which are actually fixed-length are
+not recognized as such, leading to the abovementioned performance loss.
+In particular, parts using '|' or {n} are always considered variable-length.
+.LP
+Use of unput() or input() invalidates yytext and yyleng.
+.LP
+Use of unput() to push back more text than was matched can
+result in the pushed-back text matching a beginning-of-line ('^')
+rule even though it didn't come at the beginning of the line
+(though this is rare!).
+.LP
+Nulls are not allowed in
+.I flex
+inputs or in the inputs to
+scanners generated by
+.I flex.
+Their presence generates fatal errors.
+.LP
+.I flex
+does not generate correct #line directives for code internal
+to the scanner; thus, bugs in
+.I
+flex.skel
+yield bogus line numbers.
+.LP
+Due to both buffering of input and read-ahead, you cannot intermix
+calls to stdio routines, such as, for example,
+.B getchar()
+with
+.I flex
+rules and expect it to work. Call
+.B input()
+instead.
+.LP
+The total table entries listed by the
+.B -v
+flag excludes the number of table entries needed to determine
+what rule has been matched. The number of entries is equal
+to the number of DFA states if the scanner does not use REJECT,
+and somewhat greater than the number of states if it does.
+.LP
+It would be useful if
+.I flex
+wrote to lex.yy.c a summary of the flags used in
+its generation (such as which table compression options).
+.LP
+Some of the macros, such as
+.B yywrap(),
+may in the future become functions which live in the
+.B -ll
+library. This will doubtless break a lot of code, but may be
+required for POSIX-compliance.
+.LP
+The
+.I flex
+internal algorithms need documentation.
+.SH "SEE ALSO"
+.LP
+lex(1), yacc(1), sed(1), awk(1).
+.LP
+M. E. Lesk and E. Schmidt,
+.I LEX - Lexical Analyzer Generator
+.SH AUTHOR
+Vern Paxson, with the help of many ideas and much inspiration from
+Van Jacobson. Original version by Jef Poskanzer. Fast table
+representation is a partial implementation of a design done by Van
+Jacobson. The implementation was done by Kevin Gong and Vern Paxson.
+.LP
+Thanks to the many
+.I flex
+beta-testers and feedbackers, especially Casey
+Leedom, benson@odi.com,
+Frederic Brehm, Nick Christopher, Jason Coughlin,
+Chris Faylor, Eric Goldman, Eric
+Hughes, Jeffrey R. Jones, Kevin B. Kenny, Ronald Lamprecht,
+Greg Lee, Craig Leres, Mohamed el Lozy, Jim Meyering, Marc Nozell, Esmond Pitt,
+Jef Poskanzer, Dave Tallman, Frank Whaley, Ken Yap, and others whose names
+have slipped my marginal mail-archiving skills but whose contributions
+are appreciated all the same.
+.LP
+Thanks to Keith Bostic, John Gilmore, Bob
+Mulcahy, Rich Salz, and Richard Stallman for help with various distribution
+headaches.
+.LP
+Thanks to Esmond Pitt for 8-bit character support, Benson Margulies and Fred
+Burke for C++ support, and Ove Ewerlid for supporting NUL's (as well as for
+impressive efforts regarding generating extremely high-performance
+scanners, which with luck will be soon forthcoming).
+.LP
+This work was primarily done when I was a member of the Real Time System Group
+at the Lawrence Berkeley Laboratory in Berkeley, CA. Many thanks to all there
+for the support I received.
+.LP
+Send comments to:
+.nf
+
+ Vern Paxson
+ Computer Science Department
+ 4126 Upson Hall
+ Cornell University
+ Ithaca, NY 14853-7501
+
+ vern@cs.cornell.edu
+ decvax!cornell!vern
+ vern@LBL (bitnet)
+
+.fi
projects / flex / commitdiff