-Changes between release 2.5.4 (11Sep96) and release 2.5.3:
+This is the file NEWS for the flex package. It records user -visible
+changes between releases of flex.
- - Fixed a bug introduced in 2.5.3 that blew it when a call
- to input() occurred at the end of an input file.
+See the file COPYING for copying conditions.
- - Fixed scanner skeleton so the example in the man page of
- scanning strings using exclusive start conditions works.
+* version 2.5.6
- - Minor Makefile tweaks.
+** Support for reentrant C scanners has been added.
+** Developer test suite added.
-Changes between release 2.5.3 (29May96) and release 2.5.2:
+** Two new options %option reentrant (-R) and
+%option reentrant-bison (-Rb).
- - Some serious bugs in yymore() have been fixed. In particular,
- when using AT&T-lex-compatibility or %array, you can intermix
- calls to input(), unput(), and yymore(). (This still doesn't
- work for %pointer, and isn't likely to in the future.)
+** All globals optionally placed into struct yyglobals_t.
- - A bug in handling NUL's in the input stream of scanners using
- REJECT has been fixed.
+** All access to globals replaced by macro invocations.
- - The default main() in libfl.a now repeatedly calls yylex() until
- it returns 0, rather than just calling it once.
+** All functions optionally take one additional
+argument, yy_globals.
- - Minor tweak for Windows NT Makefile, MISC/NT/Makefile.
+** New style for invoking reentrant scanner:
+yylex_init(void** scanner );
+yylex( scanner );
+yylex_destroy( scanner );
+** Added get/set functions for members of struct yy_globals_t.
+e.g., yyget_text, yyget_leng, etc.
-Changes between release 2.5.2 (25Apr95) and release 2.5.1:
+** Prefix substitution added for new functions.
- - The --prefix configuration option now works.
+** Macro shortcuts to the lengthy get/set functions
+provided for use in actions. e.g., yytext_r, yyleng_r, etc.
- - A bug that completely broke the "-Cf" table compression
- option has been fixed.
+** Arbitrary, user-defined data, "yyextra", may be added to scanner.
- - A major headache involving "const" declarators and Solaris
- systems has been fixed.
+** %option nomain no longer implies %option yywrap.
+But the inverse is still true.
- - An octal escape sequence in a flex regular expression must
- now contain only the digits 0-7.
+** TESTS/ directory has been added. Users can
+'make check' in the TESTS directory to execute the test suite.
- - You can now use "--" on the flex command line to mark the
- end of flex options.
+** Support for bison variables yylval and yylloc added.
- - You can now specify the filename '-' as a synonym for stdin.
+** Updated the man page with the new reentrant API.
- - By default, the scanners generated by flex no longer
- statically initialize yyin and yyout to stdin and stdout.
- This change is necessary because in some ANSI environments,
- stdin and stdout are not compile-time constant. You can
- force the initialization using "%option stdinit" in the first
- section of your flex input.
-
- - "%option nounput" now correctly omits the unput() routine
- from the output.
-
- - "make clean" now removes config.log, config.cache, and the
- flex binary. The fact that it removes the flex binary means
- you should take care if making changes to scan.l, to make
- sure you don't wind up in a bootstrap problem.
-
- - In general, the Makefile has been reworked somewhat (thanks
- to Francois Pinard) for added flexibility - more changes will
- follow in subsequent releases.
-
- - The .texi and .info files in MISC/texinfo/ have been updated,
- thanks also to Francois Pinard.
-
- - The FlexLexer::yylex(istream* new_in, ostream* new_out) method
- now does not have a default for the first argument, to disambiguate
- it from FlexLexer::yylex().
-
- - A bug in destructing a FlexLexer object before doing any scanning
- with it has been fixed.
-
- - A problem with including FlexLexer.h multiple times has been fixed.
-
- - The alloca() chud necessary to accommodate bison has grown
- even uglier, but hopefully more correct.
-
- - A portability tweak has been added to accommodate compilers that
- use char* generic pointers.
-
- - EBCDIC contact information in the file MISC/EBCDIC has been updated.
-
- - An OS/2 Makefile and config.h for flex 2.5 is now available in
- MISC/OS2/, contributed by Kai Uwe Rommel.
-
- - The descrip.mms file for building flex under VMS has been updated,
- thanks to Pat Rankin.
-
- - The notes on building flex for the Amiga have been updated for
- flex 2.5, contributed by Andreas Scherer.
-
-
-Changes between release 2.5.1 (28Mar95) and release 2.4.7:
-
- - A new concept of "start condition" scope has been introduced.
- A start condition scope is begun with:
-
- <SCs>{
-
- where SCs is a list of one or more start conditions. Inside
- the start condition scope, every rule automatically has the
- prefix <SCs> applied to it, until a '}' which matches the
- initial '{'. So, for example:
-
- <ESC>{
- "\\n" return '\n';
- "\\r" return '\r';
- "\\f" return '\f';
- "\\0" return '\0';
- }
-
- is equivalent to:
-
- <ESC>"\\n" return '\n';
- <ESC>"\\r" return '\r';
- <ESC>"\\f" return '\f';
- <ESC>"\\0" return '\0';
-
- As indicated in this example, rules inside start condition scopes
- (and any rule, actually, other than the first) can be indented,
- to better show the extent of the scope.
-
- Start condition scopes may be nested.
-
- - The new %option directive can be used in the first section of
- a flex scanner to control scanner-generation options. Most
- options are given simply as names, optionally preceded by the
- word "no" (with no intervening whitespace) to negate their
- meaning. Some are equivalent to flex flags, so putting them
- in your scanner source is equivalent to always specifying
- the flag (%option's take precedence over flags):
-
- 7bit -7 option
- 8bit -8 option
- align -Ca option
- backup -b option
- batch -B option
- c++ -+ option
- caseful opposite of -i option (caseful is the default);
- case-sensitive same as above
- caseless -i option;
- case-insensitive same as above
- debug -d option
- default opposite of -s option
- ecs -Ce option
- fast -F option
- full -f option
- interactive -I option
- lex-compat -l option
- meta-ecs -Cm option
- perf-report -p option
- read -Cr option
- stdout -t option
- verbose -v option
- warn opposite of -w option (so use "%option nowarn" for -w)
-
- array equivalent to "%array"
- pointer equivalent to "%pointer" (default)
-
- Some provide new features:
-
- always-interactive generate a scanner which always
- considers its input "interactive" (no call to isatty()
- will be made when the scanner runs)
- main supply a main program for the scanner, which
- simply calls yylex(). Implies %option noyywrap.
- never-interactive generate a scanner which never
- considers its input "interactive" (no call to isatty()
- will be made when the scanner runs)
- stack if set, enable start condition stacks (see below)
- stdinit if unset ("%option nostdinit"), initialize yyin
- and yyout statically to nil FILE* pointers, instead
- of stdin and stdout
- yylineno if set, keep track of the current line
- number in global yylineno (this option is expensive
- in terms of performance). The line number is available
- to C++ scanning objects via the new member function
- lineno().
- yywrap if unset ("%option noyywrap"), scanner does not
- call yywrap() upon EOF but simply assumes there
- are no more files to scan
-
- Flex scans your rule actions to determine whether you use the
- REJECT or yymore features (this is not new). Two %options can be
- used to override its decision, either by setting them to indicate
- the feature is indeed used, or unsetting them to indicate it
- actually is not used:
-
- reject
- yymore
-
- Three %option's take string-delimited values, offset with '=':
-
- outfile="<name>" equivalent to -o<name>
- prefix="<name>" equivalent to -P<name>
- yyclass="<name>" set the name of the C++ scanning class
- (see below)
-
- A number of %option's are available for lint purists who
- want to suppress the appearance of unneeded routines in
- the generated scanner. Each of the following, if unset,
- results in the corresponding routine not appearing in the
- generated scanner:
-
- input, unput
- yy_push_state, yy_pop_state, yy_top_state
- yy_scan_buffer, yy_scan_bytes, yy_scan_string
-
- You can specify multiple options with a single %option directive,
- and multiple directives in the first section of your flex input file.
-
- - The new function:
-
- YY_BUFFER_STATE yy_scan_string( const char *str )
-
- returns a YY_BUFFER_STATE (which also becomes the current input
- buffer) for scanning the given string, which occurs starting
- with the next call to yylex(). The string must be NUL-terminated.
- A related function:
-
- YY_BUFFER_STATE yy_scan_bytes( const char *bytes, int len )
-
- creates a buffer for scanning "len" bytes (including possibly NUL's)
- starting at location "bytes".
-
- Note that both of these functions create and scan a *copy* of
- the string/bytes. (This may be desirable, since yylex() modifies
- the contents of the buffer it is scanning.) You can avoid the
- copy by using:
-
- YY_BUFFER_STATE yy_scan_buffer( char *base, yy_size_t size )
-
- which scans in place the buffer starting at "base", consisting
- of "size" bytes, the last two bytes of which *must* be
- YY_END_OF_BUFFER_CHAR (these bytes are not scanned; thus, scanning
- consists of base[0] through base[size-2], inclusive). If you
- fail to set up "base" in this manner, yy_scan_buffer returns a
- nil pointer instead of creating a new input buffer.
-
- The type yy_size_t is an integral type to which you can cast
- an integer expression reflecting the size of the buffer.
-
- - Three new routines are available for manipulating stacks of
- start conditions:
-
- void yy_push_state( int new_state )
-
- pushes the current start condition onto the top of the stack
- and BEGIN's "new_state" (recall that start condition names are
- also integers).
-
- void yy_pop_state()
-
- pops the top of the stack and BEGIN's to it, and
-
- int yy_top_state()
-
- returns the top of the stack without altering the stack's
- contents.
-
- The start condition stack grows dynamically and so has no built-in
- size limitation. If memory is exhausted, program execution
- is aborted.
-
- To use start condition stacks, your scanner must include
- a "%option stack" directive.
-
- - flex now supports POSIX character class expressions. These
- are expressions enclosed inside "[:" and ":]" delimiters (which
- themselves must appear between the '[' and ']' of a character
- class; other elements may occur inside the character class, too).
- The expressions flex recognizes are:
-
- [:alnum:] [:alpha:] [:blank:] [:cntrl:] [:digit:] [:graph:]
- [:lower:] [:print:] [:punct:] [:space:] [:upper:] [:xdigit:]
-
- These expressions all designate a set of characters equivalent to
- the corresponding isXXX function (for example, [:alnum:] designates
- those characters for which isalnum() returns true - i.e., any
- alphabetic or numeric). Some systems don't provide isblank(),
- so flex defines [:blank:] as a blank or a tab.
-
- For example, the following character classes are all equivalent:
-
- [[:alnum:]]
- [[:alpha:][:digit:]
- [[:alpha:]0-9]
- [a-zA-Z0-9]
-
- If your scanner is case-insensitive (-i flag), then [:upper:]
- and [:lower:] are equivalent to [:alpha:].
-
- - The promised rewrite of the C++ FlexLexer class has not yet
- been done. Support for FlexLexer is limited at the moment to
- fixing show-stopper bugs, so, for example, the new functions
- yy_scan_string() & friends are not available to FlexLexer
- objects.
-
- - The new macro
-
- yy_set_interactive(is_interactive)
-
- can be used to control whether the current buffer is considered
- "interactive". An interactive buffer is processed more slowly,
- but must be used when the scanner's input source is indeed
- interactive to avoid problems due to waiting to fill buffers
- (see the discussion of the -I flag in flex.1). A non-zero value
- in the macro invocation marks the buffer as interactive, a zero
- value as non-interactive. Note that use of this macro overrides
- "%option always-interactive" or "%option never-interactive".
-
- yy_set_interactive() must be invoked prior to beginning to
- scan the buffer.
-
- - The new macro
-
- yy_set_bol(at_bol)
-
- can be used to control whether the current buffer's scanning
- context for the next token match is done as though at the
- beginning of a line (non-zero macro argument; makes '^' anchored
- rules active) or not at the beginning of a line (zero argument,
- '^' rules inactive).
-
- - Related to this change, the mechanism for determining when a scan is
- starting at the beginning of a line has changed. It used to be
- that '^' was active iff the character prior to that at which the
- scan started was a newline. The mechanism now is that '^' is
- active iff the last token ended in a newline (or the last call to
- input() returned a newline). For most users, the difference in
- mechanisms is negligible. Where it will make a difference,
- however, is if unput() or yyless() is used to alter the input
- stream. When in doubt, use yy_set_bol().
-
- - The new beginning-of-line mechanism involved changing some fairly
- twisted code, so it may have introduced bugs - beware ...
-
- - The macro YY_AT_BOL() returns true if the next token scanned from
- the current buffer will have '^' rules active, false otherwise.
-
- - The new function
-
- void yy_flush_buffer( struct yy_buffer_state* b )
-
- flushes the contents of the current buffer (i.e., next time
- the scanner attempts to match a token using b as the current
- buffer, it will begin by invoking YY_INPUT to fill the buffer).
- This routine is also available to C++ scanners (unlike some
- of the other new routines).
-
- The related macro
-
- YY_FLUSH_BUFFER
-
- flushes the contents of the current buffer.
-
- - A new "-ooutput" option writes the generated scanner to "output".
- If used with -t, the scanner is still written to stdout, but
- its internal #line directives (see previous item) use "output".
-
- - Flex now generates #line directives relating the code it
- produces to the output file; this means that error messages
- in the flex-generated code should be correctly pinpointed.
-
- - When generating #line directives, filenames with embedded '\'s
- have those characters escaped (i.e., turned into '\\'). This
- feature helps with reporting filenames for some MS-DOS and OS/2
- systems.
-
- - The FlexLexer class includes two new public member functions:
-
- virtual void switch_streams( istream* new_in = 0,
- ostream* new_out = 0 )
-
- reassigns yyin to new_in (if non-nil) and yyout to new_out
- (ditto), deleting the previous input buffer if yyin is
- reassigned. It is used by:
-
- int yylex( istream* new_in = 0, ostream* new_out = 0 )
-
- which first calls switch_streams() and then returns the value
- of calling yylex().
-
- - C++ scanners now have yy_flex_debug as a member variable of
- FlexLexer rather than a global, and member functions for testing
- and setting it.
-
- - When generating a C++ scanning class, you can now use
-
- %option yyclass="foo"
-
- to inform flex that you have derived "foo" as a subclass of
- yyFlexLexer, so flex will place your actions in the member
- function foo::yylex() instead of yyFlexLexer::yylex(). It also
- generates a yyFlexLexer::yylex() member function that generates a
- run-time error if called (by invoking yyFlexLexer::LexerError()).
- This feature is necessary if your subclass "foo" introduces some
- additional member functions or variables that you need to access
- from yylex().
-
- - Current texinfo files in MISC/texinfo, contributed by Francois
- Pinard.
-
- - You can now change the name "flex" to something else (e.g., "lex")
- by redefining $(FLEX) in the Makefile.
-
- - Two bugs (one serious) that could cause "bigcheck" to fail have
- been fixed.
-
- - A number of portability/configuration changes have been made
- for easier portability.
-
- - You can use "YYSTATE" in your scanner as an alias for YY_START
- (for AT&T lex compatibility).
-
- - input() now maintains yylineno.
-
- - input() no longer trashes yytext.
-
- - interactive scanners now read characters in YY_INPUT up to a
- newline, a large performance gain.
-
- - C++ scanner objects now work with the -P option. You include
- <FlexLexer.h> once per scanner - see comments in <FlexLexer.h>
- (or flex.1) for details.
-
- - C++ FlexLexer objects now use the "cerr" stream to report -d output
- instead of stdio.
-
- - The -c flag now has its full glorious POSIX interpretation (do
- nothing), rather than being interpreted as an old-style -C flag.
-
- - Scanners generated by flex now include two #define's giving
- the major and minor version numbers (YY_FLEX_MAJOR_VERSION,
- YY_FLEX_MINOR_VERSION). These can then be tested to see
- whether certain flex features are available.
-
- - Scanners generated using -l lex compatibility now have the symbol
- YY_FLEX_LEX_COMPAT #define'd.
-
- - When initializing (i.e., yy_init is non-zero on entry to yylex()),
- generated scanners now set yy_init to zero before executing
- YY_USER_INIT. This means that you can set yy_init back to a
- non-zero value in YY_USER_INIT if you need the scanner to be
- reinitialized on the next call.
-
- - You can now use "#line" directives in the first section of your
- scanner specification.
-
- - When generating full-table scanners (-Cf), flex now puts braces
- around each row of the 2-d array initialization, to silence warnings
- on over-zealous compilers.
-
- - Improved support for MS-DOS. The flex sources have been successfully
- built, unmodified, for Borland 4.02 (all that's required is a
- Borland Makefile and config.h file, which are supplied in
- MISC/Borland - contributed by Terrence O Kane).
-
- - Improved support for Macintosh using Think C - the sources should
- build for this platform "out of the box". Contributed by Scott
- Hofmann.
-
- - Improved support for VMS, in MISC/VMS/, contributed by Pat Rankin.
-
- - Support for the Amiga, in MISC/Amiga/, contributed by Andreas
- Scherer. Note that the contributed files were developed for
- flex 2.4 and have not been tested with flex 2.5.
-
- - Some notes on support for the NeXT, in MISC/NeXT, contributed
- by Raf Schietekat.
-
- - The MISC/ directory now includes a preformatted version of flex.1
- in flex.man, and pre-yacc'd versions of parse.y in parse.{c,h}.
-
- - The flex.1 and flexdoc.1 manual pages have been merged. There
- is now just one document, flex.1, which includes an overview
- at the beginning to help you find the section you need.
-
- - Documentation now clarifies that start conditions persist across
- switches to new input files or different input buffers. If you
- want to e.g., return to INITIAL, you must explicitly do so.
-
- - The "Performance Considerations" section of the manual has been
- updated.
-
- - Documented the "yy_act" variable, which when YY_USER_ACTION is
- invoked holds the number of the matched rule, and added an
- example of using yy_act to profile how often each rule is matched.
-
- - Added YY_NUM_RULES, a definition that gives the total number
- of rules in the file, including the default rule (even if you
- use -s).
-
- - Documentation now clarifies that you can pass a nil FILE* pointer
- to yy_create_buffer() or yyrestart() if you've arrange YY_INPUT
- to not need yyin.
-
- - Documentation now clarifies that YY_BUFFER_STATE is a pointer to
- an opaque "struct yy_buffer_state".
-
- - Documentation now stresses that you gain the benefits of removing
- backing-up states only if you remove *all* of them.
-
- - Documentation now points out that traditional lex allows you
- to put the action on a separate line from the rule pattern if
- the pattern has trailing whitespace (ugh!), but flex doesn't
- support this.
-
- - A broken example in documentation of the difference between
- inclusive and exclusive start conditions is now fixed.
-
- - Usage (-h) report now goes to stdout.
-
- - Version (-V) info now goes to stdout.
-
- - More #ifdef chud has been added to the parser in attempt to
- deal with bison's use of alloca().
-
- - "make clean" no longer deletes emacs backup files (*~).
-
- - Some memory leaks have been fixed.
-
- - A bug was fixed in which dynamically-expanded buffers were
- reallocated a couple of bytes too small.
-
- - A bug was fixed which could cause flex to read and write beyond
- the end of the input buffer.
-
- - -S will not be going away.
-
-
-Changes between release 2.4.7 (03Aug94) and release 2.4.6:
-
- - Fixed serious bug in reading multiple files.
-
- - Fixed bug in scanning NUL's.
-
- - Fixed bug in input() returning 8-bit characters.
-
- - Fixed bug in matching text with embedded NUL's when
- using %array or lex compatibility.
-
- - Fixed multiple invocations of YY_USER_ACTION when using '|'
- continuation action.
-
- - Minor prototyping fixes.
-
-Changes between release 2.4.6 (04Jan94) and release 2.4.5:
-
- - Linking with -lfl no longer required if your program includes
- its own yywrap() and main() functions. (This change will cause
- problems if you have a non-ANSI compiler on a system for which
- sizeof(int) != sizeof(void*) or sizeof(int) != sizeof(size_t).)
-
- - The use of 'extern "C++"' in FlexLexer.h has been modified to
- get around an incompatibility with g++'s header files.
-
-Changes between release 2.4.5 (11Dec93) and release 2.4.4:
-
- - Fixed bug breaking C++ scanners that use REJECT or variable
- trailing context.
-
- - Fixed serious input problem for interactive scanners on
- systems for which char is unsigned.
-
- - Fixed bug in incorrectly treating '$' operator as variable
- trailing context.
-
- - Fixed bug in -CF table representation that could lead to
- corrupt tables.
-
- - Fixed fairly benign memory leak.
-
- - Added `extern "C++"' wrapper to FlexLexer.h header. This
- should overcome the g++ 2.5.X problems mentioned in the
- NEWS for release 2.4.3.
-
- - Changed #include of FlexLexer.h to use <> instead of "".
-
- - Added feature to control whether the scanner attempts to
- refill the input buffer once it's exhausted. This feature
- will be documented in the 2.5 release.
-
-
-Changes between release 2.4.4 (07Dec93) and release 2.4.3:
-
- - Fixed two serious bugs in scanning 8-bit characters.
-
- - Fixed bug in YY_USER_ACTION that caused it to be executed
- inappropriately (on the scanner's own internal actions, and
- with incorrect yytext/yyleng values).
-
- - Fixed bug in pointing yyin at a new file and resuming scanning.
-
- - Portability fix regarding min/max/abs macros conflicting with
- function definitions in standard header files.
-
- - Added a virtual LexerError() method to the C++ yyFlexLexer class
- for reporting error messages instead of always using cerr.
-
- - Added warning in flexdoc that the C++ scanning class is presently
- experimental and subject to considerable change between major
- releases.
-
-
-Changes between release 2.4.3 (03Dec93) and release 2.4.2:
-
- - Fixed bug causing fatal scanner messages to fail to print.
-
- - Fixed things so FlexLexer.h can be included in other C++
- sources. One side-effect of this change is that -+ and -CF
- are now incompatible.
-
- - libfl.a now supplies private versions of the the <string.h>/
- <strings.h> string routines needed by flex and the scanners
- it generates, to enhance portability to some BSD systems.
-
- - More robust solution to 2.4.2's flexfatal() bug fix.
-
- - Added ranlib of installed libfl.a.
-
- - Some lint tweaks.
-
- - NOTE: problems have been encountered attempting to build flex
- C++ scanners using g++ version 2.5.X. The problem is due to an
- unfortunate heuristic in g++ 2.5.X that attempts to discern between
- C and C++ headers. Because FlexLexer.h is installed (by default)
- in /usr/local/include and not /usr/local/lib/g++-include, g++ 2.5.X
- decides that it's a C header :-(. So if you have problems, install
- the header in /usr/local/lib/g++-include instead.
-
-
-Changes between release 2.4.2 (01Dec93) and release 2.4.1:
-
- - Fixed bug in libfl.a referring to non-existent "flexfatal" function.
-
- - Modified to produce both compress'd and gzip'd tar files for
- distributions (you probably don't care about this change!).
-
-
-Changes between release 2.4.1 (30Nov93) and release 2.3.8:
-
- - The new '-+' flag instructs flex to generate a C++ scanner class
- (thanks to Kent Williams). flex writes an implementation of the
- class defined in FlexLexer.h to lex.yy.cc. You may include
- multiple scanner classes in your program using the -P flag. Note
- that the scanner class also provides a mechanism for creating
- reentrant scanners. The scanner class uses C++ streams for I/O
- instead of FILE*'s (thanks to Tom Epperly). If the flex executable's
- name ends in '+' then the '-+' flag is automatically on, so creating
- a symlink or copy of "flex" to "flex++" results in a version of
- flex that can be used exclusively for C++ scanners.
-
- Note that without the '-+' flag, flex-generated scanners can still
- be compiled using C++ compilers, though they use FILE*'s for I/O
- instead of streams.
-
- See the "GENERATING C++ SCANNERS" section of flexdoc for details.
-
- - The new '-l' flag turns on maximum AT&T lex compatibility. In
- particular, -l includes support for "yylineno" and makes yytext
- be an array instead of a pointer. It does not, however, do away
- with all incompatibilities. See the "INCOMPATIBILITIES WITH LEX
- AND POSIX" section of flexdoc for details.
-
- - The new '-P' option specifies a prefix to use other than "yy"
- for the scanner's globally-visible variables, and for the
- "lex.yy.c" filename. Using -P you can link together multiple
- flex scanners in the same executable.
-
- - The distribution includes a "texinfo" version of flexdoc.1,
- contributed by Roland Pesch (thanks also to Marq Kole, who
- contributed another version). It has not been brought up to
- date, but reflects version 2.3. See MISC/flex.texinfo.
-
- The flex distribution will soon include G.T. Nicol's flex
- manual; he is presently bringing it up-to-date for version 2.4.
-
- - yywrap() is now a function, and you now *must* link flex scanners
- with libfl.a.
-
- - Site-configuration is now done via an autoconf-generated
- "configure" script contributed by Francois Pinard.
-
- - Scanners now use fread() (or getc(), if interactive) and not
- read() for input. A new "table compression" option, -Cr,
- overrides this change and causes the scanner to use read()
- (because read() is a bit faster than fread()). -f and -F
- are now equivalent to -Cfr and -CFr; i.e., they imply the
- -Cr option.
-
- - In the blessed name of POSIX compliance, flex supports "%array"
- and "%pointer" directives in the definitions (first) section of
- the scanner specification. The former specifies that yytext
- should be an array (of size YYLMAX), the latter, that it should
- be a pointer. The array version of yytext is universally slower
- than the pointer version, but has the advantage that its contents
- remain unmodified across calls to input() and unput() (the pointer
- version of yytext is, still, trashed by such calls).
-
- "%array" cannot be used with the '-+' C++ scanner class option.
-
- - The new '-Ca' option directs flex to trade off memory for
- natural alignment when generating a scanner's tables. In
- particular, table entries that would otherwise be "short"
- become "long".
-
- - The new '-h' option produces a summary of the flex flags.
-
- - The new '-V' option reports the flex version number and exits.
-
- - The new scanner macro YY_START returns an integer value
- corresponding to the current start condition. You can return
- to that start condition by passing the value to a subsequent
- "BEGIN" action. You also can implement "start condition stacks"
- by storing the values in an integer stack.
-
- - You can now redefine macros such as YY_INPUT by just #define'ing
- them to some other value in the first section of the flex input;
- no need to first #undef them.
-
- - flex now generates warnings for rules that can't be matched.
- These warnings can be turned off using the new '-w' flag. If
- your scanner uses REJECT then you will not get these warnings.
-
- - If you specify the '-s' flag but the default rule can be matched,
- flex now generates a warning.
-
- - "yyleng" is now a global, and may be modified by the user (though
- doing so and then using yymore() will yield weird results).
-
- - Name definitions in the first section of a scanner specification
- can now include a leading '^' or trailing '$' operator. In this
- case, the definition is *not* pushed back inside of parentheses.
-
- - Scanners with compressed tables are now "interactive" (-I option)
- by default. You can suppress this attribute (which makes them
- run slightly slower) using the new '-B' flag.
-
- - Flex now generates 8-bit scanners by default, unless you use the
- -Cf or -CF compression options (-Cfe and -CFe result in 8-bit
- scanners). You can force it to generate a 7-bit scanner using
- the new '-7' flag. You can build flex to generate 8-bit scanners
- for -Cf and -CF, too, by adding -DDEFAULT_CSIZE=256 to CFLAGS
- in the Makefile.
-
- - You no longer need to call the scanner routine yyrestart() to
- inform the scanner that you have switched to a new file after
- having seen an EOF on the current input file. Instead, just
- point yyin at the new file and continue scanning.
-
- - You no longer need to invoke YY_NEW_FILE in an <<EOF>> action
- to indicate you wish to continue scanning. Simply point yyin
- at a new file.
-
- - A leading '#' no longer introduces a comment in a flex input.
-
- - flex no longer considers formfeed ('\f') a whitespace character.
-
- - %t, I'm happy to report, has been nuked.
-
- - The '-p' option may be given twice ('-pp') to instruct flex to
- report minor performance problems as well as major ones.
-
- - The '-v' verbose output no longer includes start/finish time
- information.
-
- - Newlines in flex inputs can optionally include leading or
- trailing carriage-returns ('\r'), in support of several PC/Mac
- run-time libraries that automatically include these.
-
- - A start condition of the form "<*>" makes the following rule
- active in every start condition, whether exclusive or inclusive.
-
- - The following items have been corrected in the flex documentation:
-
- - '-C' table compression options *are* cumulative.
-
- - You may modify yytext but not lengthen it by appending
- characters to the end. Modifying its final character
- will affect '^' anchoring for the next rule matched
- if the character is changed to or from a newline.
-
- - The term "backtracking" has been renamed "backing up",
- since it is a one-time repositioning and not a repeated
- search. What used to be the "lex.backtrack" file is now
- "lex.backup".
-
- - Unindented "/* ... */" comments are allowed in the first
- flex input section, but not in the second.
-
- - yyless() can only be used in the flex input source, not
- externally.
-
- - You can use "yyrestart(yyin)" to throw away the
- current contents of the input buffer.
-
- - To write high-speed scanners, attempt to match as much
- text as possible with each rule. See MISC/fastwc/README
- for more information.
-
- - Using the beginning-of-line operator ('^') is fairly
- cheap. Using unput() is expensive. Using yyless() is
- cheap.
-
- - An example of scanning strings with embedded escape
- sequences has been added.
-
- - The example of backing-up in flexdoc was erroneous; it
- has been corrected.
-
- - A flex scanner's internal buffer now dynamically grows if needed
- to match large tokens. Note that growing the buffer presently
- requires rescanning the (large) token, so consuming a lot of
- text this way is a slow process. Also note that presently the
- buffer does *not* grow if you unput() more text than can fit
- into the buffer.
-
- - The MISC/ directory has been reorganized; see MISC/README for
- details.
-
- - yyless() can now be used in the third (user action) section
- of a scanner specification, thanks to Ceriel Jacobs. yyless()
- remains a macro and cannot be used outside of the scanner source.
-
- - The skeleton file is no longer opened at run-time, but instead
- compiled into a large string array (thanks to John Gilmore and
- friends at Cygnus). You can still use the -S flag to point flex
- at a different skeleton file.
-
- - flex no longer uses a temporary file to store the scanner's
- actions.
-
- - A number of changes have been made to decrease porting headaches.
- In particular, flex no longer uses memset() or ctime(), and
- provides a single simple mechanism for dealing with C compilers
- that still define malloc() as returning char* instead of void*.
-
- - Flex now detects if the scanner specification requires the -8 flag
- but the flag was not given or on by default.
-
- - A number of table-expansion fencepost bugs have been fixed,
- making flex more robust for generating large scanners.
-
- - flex more consistently identifies the location of errors in
- its input.
-
- - YY_USER_ACTION is now invoked only for "real" actions, not for
- internal actions used by the scanner for things like filling
- the buffer or handling EOF.
-
- - The rule "[^]]" now matches any character other than a ']';
- formerly it matched any character at all followed by a ']'.
- This change was made for compatibility with AT&T lex.
-
- - A large number of miscellaneous bugs have been found and fixed
- thanks to Gerhard Wilhelms.
-
- - The source code has been heavily reformatted, making patches
- relative to previous flex releases no longer accurate.
-
-
-Changes between 2.3 Patch #8 (21Feb93) and 2.3 Patch #7:
-
- - Fixed bugs in dynamic memory allocation leading to grievous
- fencepost problems when generating large scanners.
- - Fixed bug causing infinite loops on character classes with 8-bit
- characters in them.
- - Fixed bug in matching repetitions with a lower bound of 0.
- - Fixed bug in scanning NUL characters using an "interactive" scanner.
- - Fixed bug in using yymore() at the end of a file.
- - Fixed bug in misrecognizing rules with variable trailing context.
- - Fixed bug compiling flex on Suns using gcc 2.
- - Fixed bug in not recognizing that input files with the character
- ASCII 128 in them require the -8 flag.
- - Fixed bug that could cause an infinite loop writing out
- error messages.
- - Fixed bug in not recognizing old-style lex % declarations if
- followed by a tab instead of a space.
- - Fixed potential crash when flex terminated early (usually due
- to a bad flag) and the -v flag had been given.
- - Added some missing declarations of void functions.
- - Changed to only use '\a' for __STDC__ compilers.
- - Updated mailing addresses.
-
-
-Changes between 2.3 Patch #7 (28Mar91) and 2.3 Patch #6:
-
- - Fixed out-of-bounds array access that caused bad tables
- to be produced on machines where the bad reference happened
- to yield a 1. This caused problems installing or running
- flex on some Suns, in particular.
-
-
-Changes between 2.3 Patch #6 (29Aug90) and 2.3 Patch #5:
-
- - Fixed a serious bug in yymore() which basically made it
- completely broken. Thanks goes to Jean Christophe of
- the Nethack development team for finding the problem
- and passing along the fix.
-
-
-Changes between 2.3 Patch #5 (16Aug90) and 2.3 Patch #4:
-
- - An up-to-date version of initscan.c so "make test" will
- work after applying the previous patches
-
-
-Changes between 2.3 Patch #4 (14Aug90) and 2.3 Patch #3:
-
- - Fixed bug in hexadecimal escapes which allowed only digits,
- not letters, in escapes
- - Fixed bug in previous "Changes" file!
-
-
-Changes between 2.3 Patch #3 (03Aug90) and 2.3 Patch #2:
-
- - Correction to patch #2 for gcc compilation; thanks goes to
- Paul Eggert for catching this.
-
-
-Changes between 2.3 Patch #2 (02Aug90) and original 2.3 release:
-
- - Fixed (hopefully) headaches involving declaring malloc()
- and free() for gcc, which defines __STDC__ but (often) doesn't
- come with the standard include files such as <stdlib.h>.
- Reordered #ifdef maze in the scanner skeleton in the hope of
- getting the declarations right for cfront and g++, too.
-
- - Note that this patch supercedes patch #1 for release 2.3,
- which was never announced but was available briefly for
- anonymous ftp.
-
-
-Changes between 2.3 (full) release of 28Jun90 and 2.2 (alpha) release:
-
- User-visible:
-
- - A lone <<EOF>> rule (that is, one which is not qualified with
- a list of start conditions) now specifies the EOF action for
- *all* start conditions which haven't already had <<EOF>> actions
- given. To specify an end-of-file action for just the initial
- state, use <INITIAL><<EOF>>.
-
- - -d debug output is now contigent on the global yy_flex_debug
- being set to a non-zero value, which it is by default.
-
- - A new macro, YY_USER_INIT, is provided for the user to specify
- initialization action to be taken on the first call to the
- scanner. This action is done before the scanner does its
- own initialization.
-
- - yy_new_buffer() has been added as an alias for yy_create_buffer()
-
- - Comments beginning with '#' and extending to the end of the line
- now work, but have been deprecated (in anticipation of making
- flex recognize #line directives).
-
- - The funky restrictions on when semi-colons could follow the
- YY_NEW_FILE and yyless macros have been removed. They now
- behave identically to functions.
-
- - A bug in the sample redefinition of YY_INPUT in the documentation
- has been corrected.
-
- - A bug in the sample simple tokener in the documentation has
- been corrected.
-
- - The documentation on the incompatibilities between flex and
- lex has been reordered so that the discussion of yylineno
- and input() come first, as it's anticipated that these will
- be the most common source of headaches.
-
-
- Things which didn't used to be documented but now are:
-
- - flex interprets "^foo|bar" differently from lex. flex interprets
- it as "match either a 'foo' or a 'bar', providing it comes at the
- beginning of a line", whereas lex interprets it as "match either
- a 'foo' at the beginning of a line, or a 'bar' anywhere".
-
- - flex initializes the global "yyin" on the first call to the
- scanner, while lex initializes it at compile-time.
-
- - yy_switch_to_buffer() can be used in the yywrap() macro/routine.
-
- - flex scanners do not use stdio for their input, and hence when
- writing an interactive scanner one must explictly call fflush()
- after writing out a prompt.
-
- - flex scanner can be made reentrant (after a fashion) by using
- "yyrestart( yyin );". This is useful for interactive scanners
- which have interrupt handlers that long-jump out of the scanner.
-
- - a defense of why yylineno is not supported is included, along
- with a suggestion on how to convert scanners which rely on it.
-
-
- Other changes:
-
- - Prototypes and proper declarations of void routines have
- been added to the flex source code, courtesy of Kevin B. Kenny.
-
- - Routines dealing with memory allocation now use void* pointers
- instead of char* - see Makefile for porting implications.
-
- - Error-checking is now done when flex closes a file.
-
- - Various lint tweaks were added to reduce the number of gripes.
-
- - Makefile has been further parameterized to aid in porting.
-
- - Support for SCO Unix added.
-
- - Flex now sports the latest & greatest UC copyright notice
- (which is only slightly different from the previous one).
-
- - A note has been added to flexdoc.1 mentioning work in progress
- on modifying flex to generate straight C code rather than a
- table-driven automaton, with an email address of whom to contact
- if you are working along similar lines.
-
-
-Changes between 2.2 Patch #3 (30Mar90) and 2.2 Patch #2:
-
- - fixed bug which caused -I scanners to bomb
-
-
-Changes between 2.2 Patch #2 (27Mar90) and 2.2 Patch #1:
-
- - fixed bug writing past end of input buffer in yyunput()
- - fixed bug detecting NUL's at the end of a buffer
-
-
-Changes between 2.2 Patch #1 (23Mar90) and 2.2 (alpha) release:
-
- - Makefile fixes: definition of MAKE variable for systems
- which don't have it; installation of flexdoc.1 along with
- flex.1; fixed two bugs which could cause "bigtest" to fail.
-
- - flex.skel fix for compiling with g++.
-
- - README and flexdoc.1 no longer list an out-of-date BITNET address
- for contacting me.
-
- - minor typos and formatting changes to flex.1 and flexdoc.1.
-
-
-Changes between 2.2 (alpha) release of March '90 and previous release:
-
- User-visible:
-
- - Full user documentation now available.
-
- - Support for 8-bit scanners.
-
- - Scanners now accept NUL's.
-
- - A facility has been added for dealing with multiple
- input buffers.
-
- - Two manual entries now. One which fully describes flex
- (rather than just its differences from lex), and the
- other for quick(er) reference.
-
- - A number of changes to bring flex closer into compliance
- with the latest POSIX lex draft:
-
- %t support
- flex now accepts multiple input files and concatenates
- them together to form its input
- previous -c (compress) flag renamed -C
- do-nothing -c and -n flags added
- Any indented code or code within %{}'s in section 2 is
- now copied to the output
-
- - yyleng is now a bona fide global integer.
-
- - -d debug information now gives the line number of the
- matched rule instead of which number rule it was from
- the beginning of the file.
-
- - -v output now includes a summary of the flags used to generate
- the scanner.
-
- - unput() and yyrestart() are now globally callable.
-
- - yyrestart() no longer closes the previous value of yyin.
-
- - C++ support; generated scanners can be compiled with C++ compiler.
-
- - Primitive -lfl library added, containing default main()
- which calls yylex(). A number of routines currently living
- in the scanner skeleton will probably migrate to here
- in the future (in particular, yywrap() will probably cease
- to be a macro and instead be a function in the -lfl library).
-
- - Hexadecimal (\x) escape sequences added.
-
- - Support for MS-DOS, VMS, and Turbo-C integrated.
-
- - The %used/%unused operators have been deprecated. They
- may go away soon.
-
-
- Other changes:
-
- - Makefile enhanced for easier testing and installation.
- - The parser has been tweaked to detect some erroneous
- constructions which previously were missed.
- - Scanner input buffer overflow is now detected.
- - Bugs with missing "const" declarations fixed.
- - Out-of-date Minix/Atari patches provided.
- - Scanners no longer require printf() unless FLEX_DEBUG is being used.
- - A subtle input() bug has been fixed.
- - Line numbers for "continued action" rules (those following
- the special '|' action) are now correct.
- - unput() bug fixed; had been causing problems porting flex to VMS.
- - yymore() handling rewritten to fix bug with interaction
- between yymore() and trailing context.
- - EOF in actions now generates an error message.
- - Bug involving -CFe and generating equivalence classes fixed.
- - Bug which made -CF be treated as -Cf fixed.
- - Support for SysV tmpnam() added.
- - Unused #define's for scanner no longer generated.
- - Error messages which are associated with a particular input
- line are now all identified with their input line in standard
- format.
- - % directives which are valid to lex but not to flex are
- now ignored instead of generating warnings.
- - -DSYS_V flag can now also be specified -DUSG for System V
- compilation.
-
-
-Changes between 2.1 beta-test release of June '89 and previous release:
-
- User-visible:
-
- - -p flag generates a performance report to stderr. The report
- consists of comments regarding features of the scanner rules
- which result in slower scanners.
-
- - -b flag generates backtracking information to lex.backtrack.
- This is a list of scanner states which require backtracking
- and the characters on which they do so. By adding rules
- one can remove backtracking states. If all backtracking states
- are eliminated, the generated scanner will run faster.
- Backtracking is not yet documented in the manual entry.
-
- - Variable trailing context now works, i.e., one can have
- rules like "(foo)*/[ \t]*bletch". Some trailing context
- patterns still cannot be properly matched and generate
- error messages. These are patterns where the ending of the
- first part of the rule matches the beginning of the second
- part, such as "zx*/xy*", where the 'x*' matches the 'x' at
- the beginning of the trailing context. Lex won't get these
- patterns right either.
-
- - Faster scanners.
-
- - End-of-file rules. The special rule "<<EOF>>" indicates
- actions which are to be taken when an end-of-file is
- encountered and yywrap() returns non-zero (i.e., indicates
- no further files to process). See manual entry for example.
-
- - The -r (reject used) flag is gone. flex now scans the input
- for occurrences of the string "REJECT" to determine if the
- action is needed. It tries to be intelligent about this but
- can be fooled. One can force the presence or absence of
- REJECT by adding a line in the first section of the form
- "%used REJECT" or "%unused REJECT".
-
- - yymore() has been implemented. Similarly to REJECT, flex
- detects the use of yymore(), which can be overridden using
- "%used" or "%unused".
-
- - Patterns like "x{0,3}" now work (i.e., with lower-limit == 0).
-
- - Removed '\^x' for ctrl-x misfeature.
-
- - Added '\a' and '\v' escape sequences.
-
- - \<digits> now works for octal escape sequences; previously
- \0<digits> was required.
-
- - Better error reporting; line numbers are associated with rules.
-
- - yyleng is a macro; it cannot be accessed outside of the
- scanner source file.
-
- - yytext and yyleng should not be modified within a flex action.
-
- - Generated scanners #define the name FLEX_SCANNER.
-
- - Rules are internally separated by YY_BREAK in lex.yy.c rather
- than break, to allow redefinition.
-
- - The macro YY_USER_ACTION can be redefined to provide an action
- which is always executed prior to the matched rule's action.
-
- - yyrestart() is a new action which can be used to restart
- the scanner after it has seen an end-of-file (a "real" one,
- that is, one for which yywrap() returned non-zero). It takes
- a FILE* argument indicating a new file to scan and sets
- things up so that a subsequent call to yylex() will start
- scanning that file.
-
- - Internal scanner names all preceded by "yy_"
-
- - lex.yy.c is deleted if errors are encountered during processing.
-
- - Comments may be put in the first section of the input by preceding
- them with '#'.
-
-
-
- Other changes:
-
- - Some portability-related bugs fixed, in particular for machines
- with unsigned characters or sizeof( int* ) != sizeof( int ).
- Also, tweaks for VMS and Microsoft C (MS-DOS), and identifiers all
- trimmed to be 31 or fewer characters. Shortened file names
- for dinosaur OS's. Checks for allocating > 64K memory
- on 16 bit'ers. Amiga tweaks. Compiles using gcc on a Sun-3.
- - Compressed and fast scanner skeletons merged.
- - Skeleton header files done away with.
- - Generated scanner uses prototypes and "const" for __STDC__.
- - -DSV flag is now -DSYS_V for System V compilation.
- - Removed all references to FTL language.
- - Software now covered by BSD Copyright.
- - flex will replace lex in subsequent BSD releases.
+See the file ONEWS for changes in earlier releases.
projects / flex / commitdiff