From: Vern Paxson Date: Sat, 6 Feb 1993 21:08:06 +0000 (+0000) Subject: Mostly .LP -> .PP X-Git-Tag: flex-2-5-5b~454 X-Git-Url: https://granicus.if.org/sourcecode?a=commitdiff_plain;h=69e8b486e485c04f4d04708f0b163c3473af63ed;p=flex Mostly .LP -> .PP --- diff --git a/flex.1 b/flex.1 index cef3d7d..3ed69be 100644 --- a/flex.1 +++ b/flex.1 @@ -1,6 +1,6 @@ .TH FLEX 1 "26 May 1990" "Version 2.3" .SH NAME -flex - fast lexical analyzer generator +flexdoc - documentation for flex, fast lexical analyzer generator .SH SYNOPSIS .B flex .B [-bcdfinpstvFILT8 -C[efmF] -Sskeleton] @@ -28,7 +28,7 @@ it analyzes its input for occurrences of the regular expressions. Whenever it finds one, it executes the corresponding C code. .SH SOME SIMPLE EXAMPLES -.LP +.PP First some simple examples to get the flavor of how one uses .I flex. The following @@ -52,23 +52,23 @@ In this input, there is just one rule. "username" is the and the "printf" is the .I action. The "%%" marks the beginning of the rules. -.LP +.PP Here's another simple example: .nf - int num_lines = 0, num_chars = 0; + int num_lines = 0, num_chars = 0; %% - \\n ++num_lines; ++num_chars; - . ++num_chars; + \\n ++num_lines; ++num_chars; + . ++num_chars; %% main() - { - yylex(); - printf( "# of lines = %d, # of chars = %d\\n", - num_lines, num_chars ); - } + { + yylex(); + printf( "# of lines = %d, # of chars = %d\\n", + num_lines, num_chars ); + } .fi This scanner counts the number of characters and the number @@ -83,7 +83,7 @@ routine declared after the second "%%". There are two rules, one which matches a newline ("\\n") and increments both the line count and the character count, and one which matches any character other than a newline (indicated by the "." regular expression). -.LP +.PP A somewhat more complicated example: .nf @@ -143,7 +143,7 @@ This is the beginnings of a simple scanner for a language like Pascal. It identifies different types of .I tokens and reports on what it has seen. -.LP +.PP The details of this example will be explained in the following sections. .SH FORMAT OF THE INPUT FILE @@ -168,7 +168,7 @@ section contains declarations of simple definitions to simplify the scanner specification, and declarations of .I start conditions, which are explained in a later section. -.LP +.PP Name definitions have the form: .nf @@ -205,7 +205,7 @@ is identical to .fi and matches one-or-more digits followed by a '.' followed by zero-or-more digits. -.LP +.PP The .I rules section of the @@ -218,9 +218,9 @@ input contains a series of rules of the form: .fi where the pattern must be unindented and the action must begin on the same line. -.LP +.PP See below for a further description of patterns and actions. -.LP +.PP Finally, the user code section is simply copied to .B lex.yy.c verbatim. @@ -229,7 +229,7 @@ by the scanner. The presence of this section is optional; if it is missing, the second .B %% in the input file may be skipped, too. -.LP +.PP In the definitions and rules sections, any .I indented text or text enclosed in @@ -238,7 +238,7 @@ and .B %} is copied verbatim to the output (with the %{}'s removed). The %{}'s must appear unindented on lines by themselves. -.LP +.PP In the rules section, any indented or %{} text appearing before the first rule may be used to declare variables @@ -249,7 +249,7 @@ but its meaning is not well-defined and it may well cause compile-time errors (this feature is present for .I POSIX compliance; see below for other such features). -.LP +.PP In the definitions section, an unindented comment (i.e., a line beginning with "/*") is also copied verbatim to the output up to the next "*/". Also, any line in the definitions section @@ -352,7 +352,7 @@ and to match zero-or-more "foo"'s-or-"bar"'s: (foo|bar)* .fi -.LP +.PP Some notes on patterns: .IP - A negated character class such as the example "[^A-Z]" @@ -410,7 +410,7 @@ or more matches of the same length, the rule listed first in the .I flex input file is chosen. -.LP +.PP Once the match is determined, the text corresponding to the match (called the .I token) @@ -423,7 +423,7 @@ The corresponding to the matched pattern is then executed (a more detailed description of actions follows), and then the remaining input is scanned for another match. -.LP +.PP If no match is found, then the .I default rule is executed: the next character in the input is considered matched and @@ -452,7 +452,7 @@ which deletes all occurrences of "zap me" from its input: .fi (It will copy all other characters in the input to the output since they will be matched by the default rule.) -.LP +.PP Here is a program which compresses multiple blanks and tabs down to a single blank, and throws away whitespace found at the end of a line: .nf @@ -462,7 +462,7 @@ a single blank, and throws away whitespace found at the end of a line: [ \\t]+$ /* ignore this token */ .fi -.LP +.PP If the action contains a '{', then the action spans till the balancing '}' is found, and the action may cross multiple lines. .I flex @@ -472,10 +472,10 @@ within them, but also allows actions to begin with and will consider the action to be all the text up to the next .B %} (regardless of ordinary braces inside the action). -.LP +.PP An action consisting solely of a vertical bar ('|') means "same as the action for the next rule." See below for an illustration. -.LP +.PP Actions can include arbitrary C code, including .B return statements to return a value to whatever routine called @@ -490,9 +490,9 @@ however, then any subsequent call to will simply immediately return, unless .B yyrestart() is first called (see below). -.LP +.PP Actions are not allowed to modify yytext or yyleng. -.LP +.PP There are a number of special directives which can be included within an action: .IP - @@ -735,7 +735,7 @@ returning a float, and taking two floats as arguments. Note that if you give arguments to the scanning routine using a K&R-style/non-prototyped function declaration, you must terminate the definition with a semi-colon (;). -.LP +.PP Whenever .B yylex() is called, it scans tokens from the global input file @@ -758,7 +758,7 @@ pointer.) In the latter case (i.e., when an action executes a return), the scanner may then be called again and it will resume scanning where it left off. -.LP +.PP By default (and for purposes of efficiency), the scanner uses block-reads rather than simple .I getc() @@ -778,7 +778,7 @@ either the number of characters read or the constant YY_NULL (0 on Unix systems) to indicate EOF. The default YY_INPUT reads from the global file-pointer "yyin". -.LP +.PP A sample redefinition of YY_INPUT (in the definitions section of the input file): .nf @@ -795,11 +795,11 @@ section of the input file): .fi This definition will change the input processing to occur one character at a time. -.LP +.PP You also can add in things like keeping track of the input line number this way; but don't expect your scanner to go very fast. -.LP +.PP When the scanner receives an end-of-file indication from YY_INPUT, it then checks the .B yywrap() @@ -811,14 +811,14 @@ function has gone ahead and set up to point to another input file, and scanning continues. If it returns true (non-zero), then the scanner terminates, returning 0 to its caller. -.LP +.PP The default .B yywrap() always returns 1. Presently, to redefine it you must first "#undef yywrap", as it is currently implemented as a macro. As indicated by the hedging in the previous sentence, it may be changed to a true function in the near future. -.LP +.PP The scanner writes its .B ECHO output to the @@ -850,7 +850,7 @@ condition, and .fi will be active only when the current start condition is either "INITIAL", "STRING", or "QUOTE". -.LP +.PP Start conditions are declared in the definitions (first) section of the input using unindented lines beginning with either @@ -884,7 +884,7 @@ input. Because of this, exclusive start conditions make it easy to specify "mini-scanners" which scan portions of the input that are syntactically different from the rest (e.g., comments). -.LP +.PP If the distinction between inclusive and exclusive start conditions is still a little vague, here's a simple example illustrating the connection between the two. The set of rules: @@ -903,11 +903,11 @@ is equivalent to foo /* do something */ .fi -.LP +.PP The default rule (to .B ECHO any unmatched character) remains active in start conditions. -.LP +.PP .B BEGIN(0) returns to the original state where only the rules with no start conditions are active. This state can also be @@ -917,7 +917,7 @@ is equivalent to .B BEGIN(0). (The parentheses around the start condition name are not required but are considered good style.) -.LP +.PP .B BEGIN actions can also be given as indented code at the beginning of the rules section. For example, the following will cause @@ -939,7 +939,7 @@ is true: ...more rules follow... .fi -.LP +.PP To illustrate the uses of start conditions, here is a scanner which provides two different interpretations of a string like "123.456". By default it will treat it as @@ -1042,7 +1042,7 @@ which is sensitive to the scanning context. is only called when the scanner reaches the end of its buffer, which may be a long time after scanning a statement such as an "include" which requires switching the input source. -.LP +.PP To negotiate these sorts of problems, .I flex provides a mechanism for creating and switching between multiple @@ -1072,7 +1072,7 @@ come from .I new_buffer. Note that .B yy_switch_to_buffer() -may be used by yywrap() to sets things up for continued scanning, instead +may be used by yywrap() to set things up for continued scanning, instead of opening a new file and pointing .I yyin at it. @@ -1082,7 +1082,7 @@ at it. .fi is used to reclaim the storage associated with a buffer. -.LP +.PP .B yy_new_buffer() is an alias for .B yy_create_buffer(), @@ -1091,13 +1091,13 @@ provided for compatibility with the C++ use of and .I delete for creating and destroying dynamic objects. -.LP +.PP Finally, the .B YY_CURRENT_BUFFER macro returns a .B YY_BUFFER_STATE handle to the current buffer. -.LP +.PP Here is an example of using these features for writing a scanner which expands include files (the .B <> @@ -1179,7 +1179,7 @@ action; or, switching to a new buffer using .B yy_switch_to_buffer() as shown in the example above. -.LP +.PP <> rules may not be used with other patterns; they may only be qualified with a list of start conditions. If an unqualified <> rule is given, it @@ -1192,7 +1192,7 @@ specify an <> rule for only the initial start condition, use <> .fi -.LP +.PP These rules are useful for catching things like unclosed comments. An example: .nf @@ -1224,14 +1224,14 @@ YY_USER_ACTION can be redefined to provide an action which is always executed prior to the matched rule's action. For example, it could be #define'd to call a routine to convert yytext to lower-case. -.LP +.PP The macro .B YY_USER_INIT may be redefined to provide an action which is always executed before the first scan (and before the scanner's internal initializations are done). For example, it could be used to call a routine to read in a data table or open a logging file. -.LP +.PP In the generated scanner, the actions are all gathered in one large switch statement and separated using .B YY_BREAK, @@ -1322,7 +1322,7 @@ from group #2, followed by a character from group #52." Thus .B %t provides a crude way for introducing equivalence classes into the scanner specification. -.LP +.PP Note that the .B -i option (see below) coupled with the equivalence classes which @@ -1572,7 +1572,7 @@ makes run in .I trace mode. It will generate a lot of messages to -.I stdout +.I stderr concerning the form of the input and the resultant non-deterministic and deterministic finite automata. This option is mostly for use in maintaining @@ -1718,17 +1718,17 @@ are, from most expensive to least: pattern sets that require backtracking arbitrary trailing context - '^' beginning-of-line operator yymore() + '^' beginning-of-line operator .fi with the first three all being quite expensive and the last two being quite cheap. -.LP +.PP .B REJECT should be avoided at all costs when performance is important. It is a particularly expensive option. -.LP +.PP Getting rid of backtracking is messy and often may be an enormous amount of work for a complicated scanner. In principal, one begins by using the @@ -1779,13 +1779,13 @@ a bit of headscratching one can see that this must be the state it's in when it has seen "fo". When this has happened, if anything other than another 'o' is seen, the scanner will have to back up to simply match the 'f' (by the default rule). -.LP +.PP The comment regarding State #8 indicates there's a problem when "foob" has been scanned. Indeed, on any character other than a 'b', the scanner will have to back up to accept "foo". Similarly, the comment for State #9 concerns when "fooba" has been scanned. -.LP +.PP The final comment reminds us that there's no point going to all the trouble of removing backtracking from the rules unless we're using @@ -1793,7 +1793,7 @@ we're using or .B -F, since there's no performance gain doing so with compressed scanners. -.LP +.PP The way to remove the backtracking is to add "error" rules: .nf @@ -1809,7 +1809,7 @@ The way to remove the backtracking is to add "error" rules: } .fi -.LP +.PP Eliminating backtracking among a list of keywords can also be done using a "catch-all" rule: .nf @@ -1822,7 +1822,7 @@ done using a "catch-all" rule: .fi This is usually the best solution when appropriate. -.LP +.PP Backtracking messages tend to cascade. With a complicated set of rules it's not uncommon to get hundreds of messages. If one can decipher them, though, it often @@ -1831,7 +1831,7 @@ it's easy to make a mistake and have an error rule accidentally match a valid token. A possible future .I flex feature will be to automatically add rules to eliminate backtracking). -.LP +.PP .I Variable trailing context (where both the leading and trailing parts do not have a fixed length) entails almost the same performance loss as @@ -1864,7 +1864,7 @@ Note that here the special '|' action does provide any savings, and can even make things worse (see .B BUGS in flex(1)). -.LP +.PP Another area where the user can increase a scanner's performance (and one that's easier to implement) arises from the fact that the longer the tokens matched, the faster the scanner will run. @@ -1914,7 +1914,7 @@ slow down the scanner! The speed of the scanner is independent of the number of rules or (modulo the considerations given at the beginning of this section) how complicated the rules are with regard to operators such as '*' and '|'. -.LP +.PP A final example in speeding up a scanner: suppose you want to scan through a file containing identifiers and keywords, one per line and with no other extraneous characters, and recognize all the @@ -2000,7 +2000,7 @@ Compiled with this is about as fast as one can get a .I flex scanner to go for this particular problem. -.LP +.PP A final note: .I flex is slow when matching NUL's, particularly when a token contains @@ -2038,7 +2038,7 @@ users can be aware of the standardization issues and those areas where .I flex may in the near future undergo changes incompatible with its current definition. -.LP +.PP .I flex is fully compatible with .I lex @@ -2160,8 +2160,16 @@ and the precedence is such that the '?' is associated with .I flex, the rule will be expanded to "foo([A-Z][A-Z0-9]*)?" and so the string "foo" will match. -Note that because of this, the -.B ^, $, , /, +.PP +Note that if the definition begins with +.B ^ +or ends with +.B $ +then it is +.I not +expanded with parentheses, to allow these operators to appear in +definitions without losing their special meanings. But the +.B , /, and .B <> operators cannot be used in a @@ -2284,7 +2292,7 @@ is #define'd so scanners may be written for use with either .I flex or .I lex. -.LP +.PP The following .I flex features are not included in @@ -2323,6 +2331,32 @@ is (rather surprisingly) truncated to does not truncate the action. Actions that are not enclosed in braces are simply terminated at the end of the line. .SH DIAGNOSTICS +.I warning, rule cannot be matched +indicates that the given rule +cannot be matched because it follows other rules that will +always match the same text as it. For +example, in the following "foo" cannot be matched because it comes after +an identifier "catch-all" rule: +.nf + + [a-z]+ got_identifier(); + foo got_foo(); + +.fi +Using +.B REJECT +in a scanner suppresses this warning. +.PP +.I warning, +.B -s +.I option given but default rule +.I can be matched +means that it is possible (perhaps only in a particular start condition) +that the default rule (match any single character) is the only one +that will match a particular input. Since +.B -s +was given, presumably this is not intended. +.PP .I reject_used_but_not_detected undefined or .I yymore_used_but_not_detected undefined - @@ -2346,26 +2380,26 @@ supported a mechanism for dealing with this problem; this feature is still supported but now deprecated, and will go away soon unless the author hears from people who can argue compellingly that they need it.) -.LP +.PP .I flex scanner jammed - a scanner compiled with .B -s has encountered an input string which wasn't matched by any of its rules. -.LP +.PP .I flex input buffer overflowed - a scanner rule matched a string long enough to overflow the scanner's internal input buffer (16K bytes by default - controlled by .B YY_BUF_SIZE in "flex.skel". Note that to redefine this macro, you must first -.B #undefine +.B #undef it). -.LP +.PP .I scanner requires -8 flag - Your scanner specification includes recognizing 8-bit characters and you did not specify the -8 flag (and your site has not installed flex with -8 as the default). -.LP +.PP .I fatal flex scanner internal error--end of buffer missed - This can occur in an scanner which is reentered after a long-jump @@ -2376,7 +2410,7 @@ reentering the scanner, use: yyrestart( yyin ); .fi -.LP +.PP .I too many %t classes! - You managed to put every single character into its own %t class. .I flex @@ -2384,9 +2418,9 @@ requires that at least one of the classes share characters. .SH DEFICIENCIES / BUGS See flex(1). .SH "SEE ALSO" -.LP +.PP flex(1), lex(1), yacc(1), sed(1), awk(1). -.LP +.PP M. E. Lesk and E. Schmidt, .I LEX - Lexical Analyzer Generator .SH AUTHOR @@ -2394,30 +2428,30 @@ Vern Paxson, with the help of many ideas and much inspiration from Van Jacobson. Original version by Jef Poskanzer. The fast table representation is a partial implementation of a design done by Van Jacobson. The implementation was done by Kevin Gong and Vern Paxson. -.LP +.PP Thanks to the many .I flex beta-testers, feedbackers, and contributors, especially Casey -Leedom, benson@odi.com, Keith Bostic, +Leedom, benson@odi.com, Peter A. Bigot, Keith Bostic, Frederic Brehm, Nick Christopher, Jason Coughlin, Scott David Daniels, Leo Eskin, Chris Faylor, Eric Goldman, Eric Hughes, Jeffrey R. Jones, Kevin B. Kenny, Ronald Lamprecht, -Greg Lee, Craig Leres, Mohamed el Lozy, Jim Meyering, Marc Nozell, Esmond Pitt, -Jef Poskanzer, Jim Roskind, +Greg Lee, Craig Leres, Mohamed el Lozy, Jim Meyering, Marc Nozell, +Walter Pelissero, Francois Pinard, Esmond Pitt, Jef Poskanzer, Jim Roskind, Dave Tallman, Frank Whaley, Ken Yap, and those whose names have slipped my marginal mail-archiving skills but whose contributions are appreciated all the same. -.LP +.PP Thanks to Keith Bostic, John Gilmore, Craig Leres, Bob Mulcahy, Rich Salz, and Richard Stallman for help with various distribution headaches. -.LP +.PP Thanks to Esmond Pitt and Earle Horton for 8-bit character support; to Benson Margulies and Fred Burke for C++ support; to Ove Ewerlid for the basics of support for NUL's; and to Eric Hughes for the basics of support for multiple buffers. -.LP +.PP Work is being done on extending .I flex to generate scanners in which the @@ -2426,21 +2460,22 @@ These scanners may well be substantially faster than those generated using -f or -F. If you are working in this area and are interested in comparing notes and seeing whether redundant work can be avoided, contact Ove Ewerlid (ewerlid@mizar.DoCS.UU.SE). -.LP +.PP This work was primarily done when I was at the Real Time Systems Group at the Lawrence Berkeley Laboratory in Berkeley, CA. Many thanks to all there for the support I received. -.LP +.PP Send comments to: .nf Vern Paxson - Computer Science Department - 4126 Upson Hall - Cornell University - Ithaca, NY 14853-7501 - - vern@cs.cornell.edu - decvax!cornell!vern + Computer Systems Engineering + Bldg. 46A, Room 1123 + Lawrence Berkeley Laboratory + University of California + Berkeley, CA 94720 + + vern@ee.lbl.gov + ucbvax!ee.lbl.gov!vern .fi