Fixed bug in mini-scanner examle

Fixed bug in YY_INPUT redefinition
yylineno defense
reentrancy documentation
Something else which I forget.

diff --git a/flex.1 b/flex.1
index ffa589a189d4663546532531c3c1cb5fca2e6288..cef3d7d858ba758cb06878a5f040ca2febaa9d98 100644 (file)
--- a/flex.1
+++ b/flex.1
@@ -105,7 +105,7 @@ A somewhat more complicated example:
                 }
 
     {DIGIT}+"."{DIGIT}*        {
-                printf( "A float: %s (%d)\\n", yytext,
+                printf( "A float: %s (%g)\\n", yytext,
                         atof( yytext ) );
                 }
 
@@ -786,7 +786,10 @@ section of the input file):
     %{
     #undef YY_INPUT
     #define YY_INPUT(buf,result,max_size) \\
-        result = ((buf[0] = getchar()) == EOF) ? YY_NULL : 1;
+        { \\
+        int c = getchar(); \\
+        result = (c == EOF) ? YY_NULL : (buf[0] = c, 1); \\
+        }
     %}
 
 .fi
@@ -2041,6 +2044,99 @@ is fully compatible with
 .I lex
 with the following exceptions:
 .IP -
+The undocumented
+.I lex
+scanner internal variable
+.B yylineno
+is not supported.  It is difficult to support this option efficiently,
+since it requires examining every character scanned and reexamining
+the characters when the scanner backs up.
+Things get more complicated when the end of buffer or file is reached or a
+NUL is scanned (since the scan must then be restarted with the proper line
+number count), or the user uses the yyless(), unput(), or REJECT actions,
+or the multiple input buffer functions.
+.IP
+The fix is to add rules which, upon seeing a newline, increment
+yylineno.  This is usually an easy process, though it can be a drag if some
+of the patterns can match multiple newlines along with other characters.
+.IP
+yylineno is not part of the POSIX draft.
+.IP -
+The
+.B input()
+routine is not redefinable, though it may be called to read characters
+following whatever has been matched by a rule.  If
+.B input()
+encounters an end-of-file the normal
+.B yywrap()
+processing is done.  A ``real'' end-of-file is returned by
+.B input()
+as
+.I EOF.
+.IP
+Input is instead controlled by redefining the
+.B YY_INPUT
+macro.
+.IP
+The
+.I flex
+restriction that
+.B input()
+cannot be redefined is in accordance with the POSIX draft, but
+.B YY_INPUT
+has not yet been accepted into the draft (and probably won't; it looks
+like the draft will simply not specify any way of controlling the
+scanner's input other than by making an initial assignment to
+.I yyin).
+.IP -
+.I flex
+scanners do not use stdio for input.  Because of this, when writing an
+interactive scanner one must explicitly call fflush() on the
+stream associated with the terminal after writing out a prompt.
+With
+.I lex
+such writes are automatically flushed since
+.I lex
+scanners use
+.B getchar()
+for their input.  Also, when writing interactive scanners with
+.I flex,
+the
+.B -I
+flag must be used.
+.IP -
+.I flex
+scanners are not as reentrant as
+.I lex
+scanners.  In particular, if you have an interactive scanner and
+an interrupt handler which long-jumps out of the scanner, and
+the scanner is subsequently called again, you may get the following
+message:
+.nf
+
+    fatal flex scanner internal error--end of buffer missed
+
+.fi
+To reenter the scanner, first use
+.nf
+
+    yyrestart( yyin );
+
+.fi
+.IP -
+.B output()
+is not supported.
+Output from the
+.B ECHO
+macro is done to the file-pointer
+.I yyout
+(default
+.I stdout).
+.IP
+The POSIX draft mentions that an
+.B output()
+routine exists but currently gives no details as to what it does.
+.IP -
 .I lex
 does not support exclusive start conditions (%x), though they
 are in the current POSIX draft.
@@ -2084,49 +2180,6 @@ one must use "[^\\]]".  The latter works with
 .I lex,
 too.
 .IP -
-The undocumented
-.I lex
-scanner internal variable
-.B yylineno
-is not supported.  (The variable is not part of the POSIX draft.)
-.IP -
-The
-.B input()
-routine is not redefinable, though it may be called to read characters
-following whatever has been matched by a rule.  If
-.B input()
-encounters an end-of-file the normal
-.B yywrap()
-processing is done.  A ``real'' end-of-file is returned by
-.B input()
-as
-.I EOF.
-.IP
-Input is instead controlled by redefining the
-.B YY_INPUT
-macro.
-.IP
-The
-.I flex
-restriction that
-.B input()
-cannot be redefined is in accordance with the POSIX draft, but
-.B YY_INPUT
-has not yet been accepted into the draft.
-.IP -
-.B output()
-is not supported.
-Output from the
-.B ECHO
-macro is done to the file-pointer
-.I yyout
-(default
-.I stdout).
-.IP
-The POSIX draft mentions that an
-.B output()
-routine exists but currently gives no details as to what it does.
-.IP -
 The
 .I lex
 .B %r
@@ -2138,7 +2191,7 @@ If you are providing your own yywrap() routine, you must include a
 the "#undef" will have to be enclosed in %{}'s.
 .IP
 The POSIX draft
-specifies that yywrap() is a function and this is unlikely to change; so
+specifies that yywrap() is a function and this is very unlikely to change; so
 .I flex users are warned
 that
 .B yywrap()
@@ -2313,6 +2366,17 @@ Your scanner specification includes recognizing 8-bit characters and
 you did not specify the -8 flag (and your site has not installed flex
 with -8 as the default).
 .LP
+.I
+fatal flex scanner internal error--end of buffer missed -
+This can occur in an scanner which is reentered after a long-jump
+has jumped out (or over) the scanner's activation frame.  Before
+reentering the scanner, use:
+.nf
+
+    yyrestart( yyin );
+
+.fi
+.LP
 .I too many %t classes! -
 You managed to put every single character into its own %t class.
 .I flex
@@ -2334,7 +2398,7 @@ Jacobson.  The implementation was done by Kevin Gong and Vern Paxson.
 Thanks to the many
 .I flex
 beta-testers, feedbackers, and contributors, especially Casey
-Leedom, benson@odi.com,
+Leedom, benson@odi.com, Keith Bostic,
 Frederic Brehm, Nick Christopher, Jason Coughlin,
 Scott David Daniels, Leo Eskin,
 Chris Faylor, Eric Goldman, Eric