]> granicus.if.org Git - re2c/log
re2c
9 years agoMakefile.am: dropped bison flags: -y -d --no-lines.
Ulya Trofimovich [Mon, 10 Aug 2015 12:39:57 +0000 (13:39 +0100)]
Makefile.am: dropped bison flags: -y -d --no-lines.

-y stands for yacc compatibility, we don't need it since we no longer
accept yacc instead of bison.

-d stands for defines file, we already specify that in makefile rule
for parser.

--no-lines is generally useless and didn't have any effect for some
reason (#line directives are present).

9 years agorun_tests.sh: separate stdout and stderr, diff with -b on wine.
Ulya Trofimovich [Mon, 10 Aug 2015 12:16:56 +0000 (13:16 +0100)]
run_tests.sh: separate stdout and stderr, diff with -b on wine.

Found some errors while testing mingw-built re2c.exe on wine:
    1. redirecting stderr to stdout resulted resulted in random
       mix in tests that triggered re2c warnings
    2. removing all '\r' symbols in the generated files didn't
       work anymore since some tests that contain '\r' were added

Fixed:
    1. redirect both to different files and cat manually
    2. diff with -b flag on wine

9 years agoStop loosing arcs in DFA loops when building path cover with --skeleton.
Ulya Trofimovich [Mon, 10 Aug 2015 11:03:07 +0000 (12:03 +0100)]
Stop loosing arcs in DFA loops when building path cover with --skeleton.

The problem with the algorithm was that when one and the same skeleton
node is visited for the 3rd time, the algorithm would abandon this
branch and drop all the path prefixes that lead to it. It's ok when
bruteforcing all paths (the arcs in the dropped prefixes will occur in
other paths that lead to final states), but it's not ok for path cover:
some arcs may have occured only in these dropped prefixes).

The fix is easy: just check if the branch was cancelled and in case it
was try the same prefixes with another branch.

This small example demonstrates the problem:

    /*!re2c
        [^\x00]* [\x00] {}
    */

DFA skeleton looks like this:

    state 0:
        arcs to state 0: [0x01, 0xFF]
        arcs to state 1: [0x00]
    state 1: final

Before the fix, re2c would generate the following paths (provided
path cover construction is forced instead of generating all paths):

    - 0x01,0x00
    - 0x00

Clearly the [0xFF] arc looping from state 0 to state 0 is missing
(uncovered). After the fix:

    - 0x01,0x00
    - 0xFF,0x01,0x00
    - 0x01,0xFF,0x00
    - 0x00

Not a minimal path cover, but we don't need it. We just need a small
enough one.

9 years agoYet one more explicit cast of pointer difference to uint32_t.
Ulya Trofimovich [Mon, 10 Aug 2015 10:48:46 +0000 (11:48 +0100)]
Yet one more explicit cast of pointer difference to uint32_t.

Much the same as previous commit: array of length bounded with 32 bits,
first pointer points to some random element. second pointer points to
the beginning of array (unless there were some errors in calculatins
that would have likely crashed re2c anyway).

Fixes [-Wconversion] warning.

9 years agoYet another explicit cast of pointer difference to uint32_t.
Ulya Trofimovich [Mon, 10 Aug 2015 10:41:51 +0000 (11:41 +0100)]
Yet another explicit cast of pointer difference to uint32_t.

This time we deal with pointers in 'Ins' array (the length of array
equals the calculated size of regular expression, which is currently
of type uint32_t: larger regular expressions will surely crash re2c).

So we can guarantee that, if regexp was correctly compiled to instructions,
than the first pointer points to some instuction in the array and second
pointer points to the beginning of array, so this cast is safe.

If compilation was incorrect than re2c will fail anyway.

Fixes [-Wconversion] warning.

9 years agoAnother explicit cast of pointer difference to uint32_t that seems to be safe.
Ulya Trofimovich [Mon, 10 Aug 2015 09:51:11 +0000 (10:51 +0100)]
Another explicit cast of pointer difference to uint32_t that seems to be safe.

In theory, 'top' points at the top of stack and it's value is always greater
than or equal to 'stk' that points to stack bottom. Total stack size is
equal to 'DFA::nStates', which is of type uint32_t (DFAs larger than
2^32 states are currently not supported an will crash re2c).

In practice, 'stk' is not changed and 'top' is only incremented
before the cast (it's decremented afterwards. Note that the function
is self-recursive).

Fixes [-Wconversion] warning.

9 years agoExplicit cast of pointer difference to uint32_t: it seems to be safe from the code.
Ulya Trofimovich [Mon, 10 Aug 2015 09:32:31 +0000 (10:32 +0100)]
Explicit cast of pointer difference to uint32_t: it seems to be safe from the code.

In theory, this function returns the number of 'Span's in the newly
constructed 'Span' array, so it must be a non-negative integer number
that fits into 32 bits and the cast is safe.

In practice, 'x0' variable stays unchanged in this function, while
'x' variable's value can only increase: whenever it can be (conditionally)
decremented, it is always unconditionally incremented. So 'x - x0'
must be non-negative.

Fixes [-Wconversion] warning.

9 years agoPrint single character as char rather than convert it to one-symbol string.
Ulya Trofimovich [Mon, 10 Aug 2015 09:26:14 +0000 (10:26 +0100)]
Print single character as char rather than convert it to one-symbol string.

9 years agoFixes some hidden NULL pointer dereferencing.
Ulya Trofimovich [Mon, 10 Aug 2015 09:19:00 +0000 (10:19 +0100)]
Fixes some hidden NULL pointer dereferencing.

'specMap' parameter can sometimes be NULL (when not in '-c' mode).
In code it was dereferenced before the check for '-c', but due to
compiler optimizations this was never revealed.

I found the bug while trying to measure the size of 'specMap'
before the check (caught segfault). Fixed by moving the check prior
to dereferencing.

9 years agoExplicit cast of pointer difference to uint32_t: it's obviously safe from the code.
Ulya Trofimovich [Mon, 10 Aug 2015 09:12:20 +0000 (10:12 +0100)]
Explicit cast of pointer difference to uint32_t: it's obviously safe from the code.

Fixes some [-Wconversion] warnings.

9 years ago'DFA::kCount' type should be ptrdiff_t as it's involved in pointer arithmetics.
Ulya Trofimovich [Mon, 10 Aug 2015 09:07:23 +0000 (10:07 +0100)]
'DFA::kCount' type should be ptrdiff_t as it's involved in pointer arithmetics.

9 years agoAllow generic container to have size_t elements rather than uint32_t.
Ulya Trofimovich [Mon, 10 Aug 2015 08:58:10 +0000 (09:58 +0100)]
Allow generic container to have size_t elements rather than uint32_t.

Cast to uint32_t in those use cases when we are sure that container
only contains some few elements.

9 years agoUse 32 bits insted of 8 for warning status.
Ulya Trofimovich [Sun, 9 Aug 2015 19:31:00 +0000 (20:31 +0100)]
Use 32 bits insted of 8 for warning status.

Warning status can be in fact represented with only 2 bits, but
since it's engaged in arithmetic operations it gets promoted and
special precautions are needed. Much easier to ure 32 bits:
warnings are nowhere near performance/memory bottleneck.

9 years agoUse ptrdiff_t instead of uint32_t to represent offset in buffer.
Ulya Trofimovich [Sun, 9 Aug 2015 19:27:10 +0000 (20:27 +0100)]
Use ptrdiff_t instead of uint32_t to represent offset in buffer.

Found with [-Wconversion].

9 years agoRemoved useless piece of code (pretty-printing octal characters).
Ulya Trofimovich [Sun, 9 Aug 2015 19:11:20 +0000 (20:11 +0100)]
Removed useless piece of code (pretty-printing octal characters).

This piece of code could (and should) never be executed: EBCDIC and
non-pritables are handled prior to calling 'prtCh'.

9 years agoUse size_t to store the length of path in skeleton.
Ulya Trofimovich [Sun, 9 Aug 2015 19:01:52 +0000 (20:01 +0100)]
Use size_t to store the length of path in skeleton.

Though this length shouldn't actually excedd 32 bits (othewise
too much data will be generated, and that is checked), it's better
to use size_t and don't care about the order of checks.

9 years agoAdded comment.
Ulya Trofimovich [Sun, 9 Aug 2015 19:01:17 +0000 (20:01 +0100)]
Added comment.

9 years agoA special truncated unsigned 32-bit type for overflow-sensitive calculations.
Ulya Trofimovich [Sun, 9 Aug 2015 19:00:20 +0000 (20:00 +0100)]
A special truncated unsigned 32-bit type for overflow-sensitive calculations.

With --skeleton switch we need to generate lots of data: strings that
correspond to various paths in DFA and match given regular expression.
For small graphs we can afford to generate all paths, for large graphs
we can only generate path cover. Anyway we need to be able to estimate
the amount of data to be generated (measured in skeleton arcs). Since
it can easily exceed 32 bits (and 64 as well), calculations must stop
as soon as certain limit is reached.

9 years agoEncodings: use 32-bit unsigned arithmetics instead of 8-bit and 16-bit.
Ulya Trofimovich [Sun, 9 Aug 2015 18:07:10 +0000 (19:07 +0100)]
Encodings: use 32-bit unsigned arithmetics instead of 8-bit and 16-bit.

8-bit and 16-bit unsigned integers used in arithmetic operations
are promoted to 32 bits before operation and then truncated back.
Theoretically this may change their value.

This fixes a lot of [-Wconversion] warnings.

9 years agoconfigure.ac: added warning to CXXFLAGS: -Wconversion
Ulya Trofimovich [Sun, 9 Aug 2015 18:01:00 +0000 (19:01 +0100)]
configure.ac: added warning to CXXFLAGS: -Wconversion

9 years agoMakefile.am: dropped re2c flag '-i'.
Ulya Trofimovich [Thu, 6 Aug 2015 11:04:21 +0000 (12:04 +0100)]
Makefile.am: dropped re2c flag '-i'.

Line information is useful.

9 years agoForgot to update bootstrap lexer (changed by commit 1d4462d5d531dc707a442bc984283ea51...
Ulya Trofimovich [Thu, 6 Aug 2015 09:57:46 +0000 (10:57 +0100)]
Forgot to update bootstrap lexer (changed by commit 1d4462d5d531dc707a442bc984283ea51f77338c).

9 years agoNow -Werror-<warning> turns on <warning> (unlike -Werror in general).
Ulya Trofimovich [Wed, 5 Aug 2015 09:33:55 +0000 (10:33 +0100)]
Now -Werror-<warning> turns on <warning> (unlike -Werror in general).

At least GCC does so.

9 years agoBetter representation for rule actions; omit line info for autogenerated actions.
Ulya Trofimovich [Tue, 4 Aug 2015 12:38:57 +0000 (13:38 +0100)]
Better representation for rule actions; omit line info for autogenerated actions.

Since there's no such code in source file, there's no sense in
pointing into it. Updated test.

9 years agoFree memory allocated for range suffies at the same time as everything else.
Ulya Trofimovich [Tue, 4 Aug 2015 10:36:13 +0000 (11:36 +0100)]
Free memory allocated for range suffies at the same time as everything else.

Moved static member definition closer to class.

9 years agoAdded simple struct to store locations of parsed elements in source file.
Ulya Trofimovich [Tue, 4 Aug 2015 10:10:54 +0000 (11:10 +0100)]
Added simple struct to store locations of parsed elements in source file.

9 years agoNew condition belongs to the whole rule rather than to rule's code.
Ulya Trofimovich [Tue, 4 Aug 2015 09:29:47 +0000 (10:29 +0100)]
New condition belongs to the whole rule rather than to rule's code.

9 years agoRemoved unused 'strdup' function and autoconf check.
Ulya Trofimovich [Tue, 4 Aug 2015 08:53:52 +0000 (09:53 +0100)]
Removed unused 'strdup' function and autoconf check.

Function left unused by commit 00b14f309b5da3917d62f7d98a727290eaee6ea2.

9 years agoSome tests for windows-style newlines (CR LF).
Ulya Trofimovich [Thu, 30 Jul 2015 22:10:58 +0000 (23:10 +0100)]
Some tests for windows-style newlines (CR LF).

Copied two existing large tests and patched newlines: LF -> CR LF.

9 years agoFixed bug #115 "flex-style named definitions cause ambiguity in re2c grammar".
Ulya Trofimovich [Thu, 30 Jul 2015 14:11:03 +0000 (15:11 +0100)]
Fixed bug #115 "flex-style named definitions cause ambiguity in re2c grammar".

This commit removes 10 shift/reduce conflicts in bison grammar for re2c.
These conflicts are caused by allowing flex-style named definitions
    name regular-expression
to contain newlines and to be mixed with rules. It's not just some
conflicts in LALR(1) grammar, it is genuine ambiguity as can be observed
from the following example:
    /*!re2c
        name "a"
        "b" "c" {}
    */
which can be parsed in two ways:
    definition -> name "a"
    rule -> "b" "c" {}
and
    definition -> name "a" "b"
    rule -> "c" {}
, both ways being perfectly valid.

This commit resolves ambiguity by forbidding newlines in flex-style
named definitions (conforming to flex syntax). Newline in these
definitions is treated in a special way: lexer emits token 'FID_END',
which marks the end of flex-style named definition in parser.

9 years agoFixed segfault on options that expect an argument but are passed none.
Ulya Trofimovich [Wed, 29 Jul 2015 13:46:19 +0000 (14:46 +0100)]
Fixed segfault on options that expect an argument but are passed none.

Example of commands that triggered segfault:
    $ re2c -o
    $ re2c --type-header
    $ re2c --input

9 years agoForce custom input API with "--skeleton" option.
Ulya Trofimovich [Wed, 29 Jul 2015 11:54:50 +0000 (12:54 +0100)]
Force custom input API with "--skeleton" option.

Dropped this line while rewriting options parser.

9 years agoRemoved extra newline at the end of some error messages.
Ulya Trofimovich [Wed, 29 Jul 2015 11:37:51 +0000 (12:37 +0100)]
Removed extra newline at the end of some error messages.

9 years agoOutput error message when passed multiple arguments after "--" option.
Ulya Trofimovich [Wed, 29 Jul 2015 11:32:20 +0000 (12:32 +0100)]
Output error message when passed multiple arguments after "--" option.

All arguments after "--" are treated as input files, but re2c expects
exactly one input file.

9 years agoCommented parser conflict.
Ulya Trofimovich [Wed, 29 Jul 2015 11:19:40 +0000 (12:19 +0100)]
Commented parser conflict.

See https://github.com/skvadrik/re2c/issues/115

orts the commit.

9 years agoImproved warning messages with '-Werror'.
Ulya Trofimovich [Mon, 27 Jul 2015 20:29:03 +0000 (21:29 +0100)]
Improved warning messages with '-Werror'.

Report an error rather than a warning (at least GCC does so).

9 years agoOutput condition name with '-Wnaked-default' in '-c' mode.
Ulya Trofimovich [Mon, 27 Jul 2015 14:54:57 +0000 (15:54 +0100)]
Output condition name with '-Wnaked-default' in '-c' mode.

9 years agoUpdated tests.
Ulya Trofimovich [Mon, 27 Jul 2015 14:33:29 +0000 (15:33 +0100)]
Updated tests.

Line info changed in warning messages since commit
929c87fdf3a3de5f08206a258dfabbc0a068c561:
"Unified location output in warning messages.

9 years agoUnified location output in warning messages.
Ulya Trofimovich [Mon, 27 Jul 2015 14:12:49 +0000 (15:12 +0100)]
Unified location output in warning messages.

9 years agoRenames '-Wempty-rule' -> '-Wmatch-empty-string' and improved message.
Ulya Trofimovich [Mon, 27 Jul 2015 11:57:46 +0000 (12:57 +0100)]
Renames '-Wempty-rule' -> '-Wmatch-empty-string' and improved message.

9 years agoEnable warnings globally for tests.
Ulya Trofimovich [Mon, 27 Jul 2015 11:45:36 +0000 (12:45 +0100)]
Enable warnings globally for tests.

Fixed reference results for tests that trigger warnings.

9 years agoDon't lose stdout and stderr when testing with -o.
Ulya Trofimovich [Mon, 27 Jul 2015 10:45:30 +0000 (11:45 +0100)]
Don't lose stdout and stderr when testing with -o.

9 years agoDon't output generated code when failed with -Werror.
Ulya Trofimovich [Mon, 27 Jul 2015 10:41:23 +0000 (11:41 +0100)]
Don't output generated code when failed with -Werror.

9 years agoAdded warinigs: -W, -Werror.
Ulya Trofimovich [Mon, 27 Jul 2015 08:00:33 +0000 (09:00 +0100)]
Added warinigs: -W, -Werror.

    -W: enable all warnings
    -Werror: turn all enabled (hereafter) warnings into errors

Specific warnings added by this commit:
    -Wempty-character-class
    -Wempty-rule
    -Wnaked-default

For each warning there are -Wno-, -Werror-, -Wno-error- options.

9 years agoFixed typos in release script, updated release guidelines.
Ulya Trofimovich [Fri, 24 Jul 2015 19:28:20 +0000 (20:28 +0100)]
Fixed typos in release script, updated release guidelines.

9 years agoSplit release script into tw parts: distcheck and everything else.
Ulya Trofimovich [Tue, 21 Jul 2015 18:06:07 +0000 (19:06 +0100)]
Split release script into tw parts: distcheck and everything else.

9 years agoMakefile.am: remved nonexistent file 'doc/index.html'.
Ulya Trofimovich [Tue, 21 Jul 2015 17:17:54 +0000 (18:17 +0100)]
Makefile.am: remved nonexistent file 'doc/index.html'.

9 years agoSite front page 'doc/index.html' gone to branch 'gh-pages'.
Ulya Trofimovich [Tue, 21 Jul 2015 16:46:09 +0000 (17:46 +0100)]
Site front page 'doc/index.html' gone to branch 'gh-pages'.

Note that anther site page, manual.html, is autogenerated from
'doc/re2c.ad.in' which is also used to generate documentation.

9 years agoMakefile.am: added forgotten header.
Ulya Trofimovich [Tue, 21 Jul 2015 16:22:11 +0000 (17:22 +0100)]
Makefile.am: added forgotten header.

9 years agoFixed re2c behaviour with "--" command line option.
Ulya Trofimovich [Tue, 21 Jul 2015 12:47:31 +0000 (13:47 +0100)]
Fixed re2c behaviour with "--" command line option.

All arguments after "--" are treated as non-options. This can be
used to handle option-like filenames (those starting with "-" and "--").

9 years agoAllow to specify exacly one input file and at most one output and header file.
Ulya Trofimovich [Tue, 21 Jul 2015 11:15:01 +0000 (12:15 +0100)]
Allow to specify exacly one input file and at most one output and header file.

9 years agoFixed GXX warning '-Wsuggest-attribute=format'.
Ulya Trofimovich [Mon, 20 Jul 2015 21:14:12 +0000 (22:14 +0100)]
Fixed GXX warning '-Wsuggest-attribute=format'.

g++ warns if function with '__attribute__((format (printf, ...)))'
has wrong number of arguments or argument type is wrong.

9 years agoUse 'vfprintf' instead of 'fprintf' to print variadic arguments.
Ulya Trofimovich [Mon, 20 Jul 2015 18:45:54 +0000 (19:45 +0100)]
Use 'vfprintf' instead of 'fprintf' to print variadic arguments.

Added GCC warninigs -Wformat=2 (reveals this error) and
-Wsuggest-attribute=format.

9 years agoModified release script to try build with bmake as well as make.
Ulya Trofimovich [Mon, 20 Jul 2015 14:37:05 +0000 (15:37 +0100)]
Modified release script to try build with bmake as well as make.

9 years agoMakefile.am: drop non-POSIX make feature: '$<' in target rules.
Ulya Trofimovich [Mon, 20 Jul 2015 14:24:46 +0000 (15:24 +0100)]
Makefile.am: drop non-POSIX make feature: '$<' in target rules.

POSIX make allows '$<' only in inferencee rule.
Found while trying to build with bmake.

9 years agoMakefile.am: respect various 'make' alternatives: use '$(MAKE)'.
Ulya Trofimovich [Mon, 20 Jul 2015 13:42:32 +0000 (14:42 +0100)]
Makefile.am: respect various 'make' alternatives: use '$(MAKE)'.

9 years agoMakefile.am: use inference rule '.re.cc' to avoid rule duplication.
Ulya Trofimovich [Mon, 20 Jul 2015 13:39:54 +0000 (14:39 +0100)]
Makefile.am: use inference rule '.re.cc' to avoid rule duplication.

Inference rules is a standard POSIX make feature.

9 years agoNew re2c-based parser for command line options.
Ulya Trofimovich [Sun, 19 Jul 2015 14:22:47 +0000 (15:22 +0100)]
New re2c-based parser for command line options.

9 years agoRemoved unused class variable.
Ulya Trofimovich [Mon, 29 Jun 2015 20:51:02 +0000 (21:51 +0100)]
Removed unused class variable.

9 years agoDon't check range bounds in pretty-printing function.
Ulya Trofimovich [Fri, 26 Jun 2015 13:09:51 +0000 (14:09 +0100)]
Don't check range bounds in pretty-printing function.

9 years agoRemoved unused function.
Ulya Trofimovich [Fri, 26 Jun 2015 13:01:56 +0000 (14:01 +0100)]
Removed unused function.

9 years agoComment on test for character class operations.
Ulya Trofimovich [Tue, 23 Jun 2015 21:02:27 +0000 (22:02 +0100)]
Comment on test for character class operations.

9 years agoAdded test for operations on character classes (addition and subtraction).
Ulya Trofimovich [Tue, 23 Jun 2015 20:44:13 +0000 (21:44 +0100)]
Added test for operations on character classes (addition and subtraction).

Test is built into a separate binary. Both test building and running
is triggered by `make check`.

9 years agoOperations on character classes: fixed subtraction, simplified addition.
Ulya Trofimovich [Mon, 22 Jun 2015 12:50:17 +0000 (13:50 +0100)]
Operations on character classes: fixed subtraction, simplified addition.

Subtraction was broken by commit f56196d29f6c29b37e3e95a6777714c237e1c71c:
"Simplified implementation of range union and difference."

9 years agoMore tests for "--empty-class" option.
Ulya Trofimovich [Tue, 16 Jun 2015 15:09:32 +0000 (16:09 +0100)]
More tests for "--empty-class" option.

This time I found all tests that are affected by this option
(that is, contain empty ranges) and added explicit option variants
for most of them (excluding 'test/empty_range.*' group, as we already
have 'test/bug61_positive.*' group that checks the same thing).

9 years agoUpdated docs (added new options an long variants for all options).
Ulya Trofimovich [Tue, 16 Jun 2015 14:48:55 +0000 (15:48 +0100)]
Updated docs (added new options an long variants for all options).

9 years agoMakefile.am: added forgotten header.
Ulya Trofimovich [Tue, 16 Jun 2015 14:20:02 +0000 (15:20 +0100)]
Makefile.am: added forgotten header.

9 years agoAdded cmd option "--empty-class <match-empty|match-none|error>".
Ulya Trofimovich [Tue, 16 Jun 2015 13:17:17 +0000 (14:17 +0100)]
Added cmd option "--empty-class <match-empty|match-none|error>".

This option controls re2c actions when it encounters empty character
class (e.g. [], [^\0x00-\xFF] or [\0x00-\xFF]\[\0x00-\xFF]):
    match-empty (default) - match on empty input
    match-none - fail to match on any input
    error - compilation error

This is a final fix for bug #61 "empty character class [] matches empty string".

9 years agoPartial fix for bug #61 "empty character class [] matches empty string".
Ulya Trofimovich [Tue, 16 Jun 2015 11:19:03 +0000 (12:19 +0100)]
Partial fix for bug #61 "empty character class [] matches empty string".

Given the following code:
    /*!re2c
        [] {}
    */

    /*!re2c
        [^\x00-\xFF] {}
    */

    /*!re2c
        [\x00-\xFF]\[\x00-\xFF] {}
    */
re2c versions <=0.13.6 and >=0.13.7 behaved differently.
0.13.6 consistently considered that empty range should match empty string.
Since 0.13.7 empty positive range [] and empty difference (e.g. [a-z][a-z])
still match empty string, but empty negative range (e.g. [^\x00-\xFF])
matches nothing (always fails). The faulty commit is
28ee7c95bca46ad3cdb965741c5c29e21c50df14
"Added UTF-8 encoding support and tests for it."

This commit brings back consistent behaviour of 0.13.6: empty range,
however it was constructed, always matches empty string. Whether this
behaviour is sane or not is another question.

9 years agoConstruct non-NULL regexps from NULL ranges (for variable-length encodings).
Ulya Trofimovich [Mon, 15 Jun 2015 14:39:29 +0000 (15:39 +0100)]
Construct non-NULL regexps from NULL ranges (for variable-length encodings).

NULL range represents empty range: range union and difference functions
return NULL for empty ranges. Thus NULL can be passed to functions
that construct regexp from range ('MatchOp', 'UTF8Range' and 'UTF16Range').
All these functions must behave return non-NULL for NULL ranges, since
further code relies on this.

9 years agoCrash on attempt to create range with lower bound greater or equal to lower bound.
Ulya Trofimovich [Mon, 15 Jun 2015 14:09:02 +0000 (15:09 +0100)]
Crash on attempt to create range with lower bound greater or equal to lower bound.

Better have an assert than nothing until we handle such cases properly.
As for now, if the user inputs range like [9-0], it will be tranformed
to [0-9]. Later re2c should at least warn about such cases.

9 years agoNow range internals are only visible to union/difference functions.
Ulya Trofimovich [Mon, 15 Jun 2015 13:31:26 +0000 (14:31 +0100)]
Now range internals are only visible to union/difference functions.

Ranges must be constructed so that linked ranges don't overlap and
are monotonous. This is always true for one-link ranges created by
range constructor, and we construct larger ranges from them using
union and difference functions (that maintain the invariant).

9 years agoSimplified implementation of range union and difference.
Ulya Trofimovich [Mon, 15 Jun 2015 13:09:08 +0000 (14:09 +0100)]
Simplified implementation of range union and difference.

9 years agoExplicitly shallow-copy ranges instead of using obscure copy constructor.
Ulya Trofimovich [Sat, 13 Jun 2015 10:51:19 +0000 (11:51 +0100)]
Explicitly shallow-copy ranges instead of using obscure copy constructor.

9 years agoRestructured sources layout, improved formatting.
Ulya Trofimovich [Wed, 10 Jun 2015 16:30:15 +0000 (17:30 +0100)]
Restructured sources layout, improved formatting.

9 years agoUnified macro names in header guards.
Ulya Trofimovich [Tue, 9 Jun 2015 16:36:36 +0000 (17:36 +0100)]
Unified macro names in header guards.

Inspired by sudden collision with '__COUNTER__' I occasionally got
while guarding 'src/util/counter.h'.

Now all headers use guards of the form '_RE2C_PATH_TO_HEADER_BASENAME_'.
Some compilers (e.g. clang++) would warn about [-Wreserved-id-macro],
but their reasonong about reserved macro names is quite crude and re2c
is not able to confirm their standards anyway (e.g. autoconf-generated
macro name 'SIZEOF___INT64').

9 years agoDistinct restricted type for rule priority.
Ulya Trofimovich [Tue, 9 Jun 2015 14:28:58 +0000 (15:28 +0100)]
Distinct restricted type for rule priority.

As with labels, try to control how rule priorities are created:
make a special counter that creates new priorities and disallow
everyone but this counter do it.

9 years agoSimplified creation of rule states and backup states.
Ulya Trofimovich [Mon, 8 Jun 2015 21:25:32 +0000 (22:25 +0100)]
Simplified creation of rule states and backup states.

New simpler implementation uses STL containers instaed of sparse
array. It's less efficient, but the place is not a bottleneck and
simplicity is more important than efficiency.

9 years agoA better data structure for mapping 'yyaccept' values to DFA rule states.
Ulya Trofimovich [Mon, 8 Jun 2015 14:06:56 +0000 (15:06 +0100)]
A better data structure for mapping 'yyaccept' values to DFA rule states.

'yyaccept' values must be continuous natural numbers starting from 0,
so array indices represent them ideally.

9 years agoTried to improve code readability.
Ulya Trofimovich [Mon, 8 Jun 2015 13:41:55 +0000 (14:41 +0100)]
Tried to improve code readability.

Now rule states and default state are added to DFA in separate
independent passes.

Removed 'bSaveOnHead' DFA property as it can be calculated right
before setting initial state action.

9 years agoTrack backup states only if DFA has default state.
Ulya Trofimovich [Mon, 8 Jun 2015 10:27:14 +0000 (11:27 +0100)]
Track backup states only if DFA has default state.

9 years agoReduced redundant variable.
Ulya Trofimovich [Sun, 7 Jun 2015 12:59:48 +0000 (13:59 +0100)]
Reduced redundant variable.

Variables 'accept' and 'accfixup' were assigned both only once and
in the same place and used to do the same thing.

9 years agoFixed bug #60 "redundant use of YYMARKER".
Ulya Trofimovich [Sat, 6 Jun 2015 17:51:27 +0000 (18:51 +0100)]
Fixed bug #60 "redundant use of YYMARKER".

Bug description: sometimes re2c would generate code that backups
current input position (e.g. 'YYMARKER = YYCURSOR'), but wouldn't
generate code that restores backuped position (e.g. 'YYCURSOR =
YYMARKER').

Analyses: DFA may have overlapping rules (e.g. "a" and "aaa").
In such cases, if the shorter rule matched, lexer must attempt to
match the longer one. If the longer rule also mathed, then lexer
prefers it to the shorter rule. If the longer rule didn't match,
lexer must backtrack input position to the point when the shorter
rule matched. In order to be able to backtrack, re2c must generate
backup code (e.g. 'YYMARKER = YYCURSOR') and restore code (e.g.
'YYCURSOR = YYMARKER').
In some rare cases DFA has overlapping rules, but if the shorter rule
matched, then the longer rule will always match (perhaps on an
arbitrary long input string), e.g.:
    /*!re2c
        [^]+ "a" { 1st }
        "b"      { 2nd }
    */
In this cases there's no need to generate backup code for 2nd rule:
lexer will either encounter final "a" and the 1st rule will match
or YYFILL will not return; anyway, restore code will never be run.
re2c used to output backup code but not restore code in such cases.
This is the bug: backup code is useless without restore code and
should be omitted.

In future re2c should warn about such cases (when the shorter of
two overlapping rules is shadowed by the longer one).

The fix: postpone insertion of save actions (those with backup code)
untill it is known if restore code will be generated.
I also removed obsolete global variable 'bUsedYYMarker', which was
always set to 'true' (it should be per-DFA, not per-block configuration
anyway).

9 years agoFixed bug #59 "bogus 'yyaccept' in '-c' mode".
Ulya Trofimovich [Thu, 4 Jun 2015 21:25:03 +0000 (22:25 +0100)]
Fixed bug #59 "bogus 'yyaccept' in '-c' mode".

We have one 'yyaccept' initialization per re2c block. Each block
consists of one or more DFA (multiple DFA in '-c' mode in case of
multiple conditions). Each DFA may or may not use 'yyaccept'
(that is, save 'yyaccept' in some states and have a dispatch state
based on saved 'yyaccept' value).

Description of the bug: in '-c' mode, sometimes a DFA would have
states that save 'yyaccept', but no dispatch state that uses that
saved values. DFA didn't actually need 'yyaccept' (all the
assignments vanished if other conditions that need 'yyaccept' were
removed).

The essence of the bug: re2c decided whether to output 'yyaccept'
related stuff on a per-block basis: for multiple conditions in the
same block, the same decision was made (if any condition needed
'yyaccept', all of them would to output it).

The fix: 'yyaccept' initialization should be done on a per-block
basis, while assignments to 'yyaccept' should be done on a per-DFA
basis. Also, 'yyaccept' initialization must be delayed, while
assignments to 'yyaccept' must not.

Note: we may consider per-DFA 'yyaccept' initialization (have a
local 'yyaccept' variable per DFA). This wouldn't conflict with '-f'
switch (as it might seem) as long as we name all the variables
'yyaccept' and don't generate any 'yyaccept' initializations with '-f'.

9 years agoUse autoconf to locate bison.
Ulya Trofimovich [Tue, 2 Jun 2015 21:54:01 +0000 (22:54 +0100)]
Use autoconf to locate bison.

9 years agoRespect user CXXFLAGS.
Ulya Trofimovich [Tue, 2 Jun 2015 21:29:19 +0000 (22:29 +0100)]
Respect user CXXFLAGS.

As automake manual (chapter 27.6 "Flag Variables Ordering") states,
CXXFLAGS is a user variable and should be left for users to override
C++ compiler flags. Thus we should leave CXXFLAGS as is and modify
AM_CXXFLAGS insted.

9 years agoAdded missing header (revealed by mingw build).
Ulya Trofimovich [Tue, 2 Jun 2015 16:16:27 +0000 (17:16 +0100)]
Added missing header (revealed by mingw build).

9 years agoCheck CXXFLAGS in configure.
Ulya Trofimovich [Tue, 2 Jun 2015 15:58:53 +0000 (16:58 +0100)]
Check CXXFLAGS in configure.

Instead of unconditionally setting CXXFLAGS in Makefile.am,
check the presence of a flag in configure.ac. If the flag is
present (that is, an attempt to compile an empty C++ program with
this flag is successful), then it is added to CXXFLAGS.

Now one can add *any* compiler flag in configure.ac without
worrying about portability.

9 years agoSupport 'make distcheck'.
Ulya Trofimovich [Tue, 2 Jun 2015 12:19:50 +0000 (13:19 +0100)]
Support 'make distcheck'.

The only problem with 'make distcheck' was that we needed write
access to top source directory ('make' wanted to overwrite bootstrap
parser if it was built with bison and 'make check' wanted to create
temporary files in 'test/' directory).

This commit fixes it:
    - 'make' doesn't try to overwrite bootstrap if it is identical
      to the existing one (must always be true for 'make distcheck')
    - testing script makes a temporary directory and keeps all
      temporary files there. If some tests failed, temporary files
      for them are left and test sources and reference results are
      copied into temporary directory to make debug more convenient.

This commit makes use of 'make distcheck' in release script.

9 years agoFixed build system to support automake's 'subdir-objects' feature.
Ulya Trofimovich [Tue, 2 Jun 2015 11:34:09 +0000 (12:34 +0100)]
Fixed build system to support automake's 'subdir-objects' feature.

As I updated automake to version 1.15 it began to produce lots of
warnings about 'subdir-objects' not used when it should have been.
Turns out that 'subdir-objects' will be on by default in 2.0.

So I tried to turn on 'subdir-objects' and builds began to fail:
automake didn't expand '$(sourcedir)' and '$(builddir)' prefixes.
I erroneously prepended these prefixes in commit
38f526d04415adb7b5e6bca228fc26409833f5c3 "Updated build system.",
as commit message says:
...
    Makefile.am:
        - explicitly prefixed all file names with $(srcdir) or $(builddir)
...

But automake prepends these prefixes already where necessary, except
for custom rules with side effects: if a custom rule touches some
files that are not explicit targets or dependencies of this rule,
then automake won't find these files unless they are in build directory.
We have such side-effectful custom rules:
    - parser rule produces multiple files and touches bootstrap files
    - scanner rule touches bootstrap file
    - doc rules touch bootstrap files
Multiple files is a common problem of make. Bootstrap introduces
circular dependency, since bootstrap files need to be updated after
they've been used. So it's hard to get rid of side effects in these
rules.

This commit enabels 'subdir-objects' feature and removes all prefixes
in variables and adds them in side-effectful custom rules (for files
from source directory, not for files from build directory). It also
makes use of '$@' and '$<' special variables in custom rules (which
makes side effects more explicit).

Still I don't yet fully understand how automake uses things like
'$(sourcedir)' and '$(builddir)' and their relation with 'subdir-objects'
(it's probably related with non-recursive makefiles).

9 years agoEnable silent builds by default.
Ulya Trofimovich [Mon, 1 Jun 2015 21:59:08 +0000 (22:59 +0100)]
Enable silent builds by default.

9 years agoIgnore one more autoconf-generated script.
Ulya Trofimovich [Mon, 1 Jun 2015 10:43:26 +0000 (11:43 +0100)]
Ignore one more autoconf-generated script.

9 years agoRemoved remnants of CVS ("# $Id$" in the beginning of some files)
Ulya Trofimovich [Mon, 1 Jun 2015 10:42:23 +0000 (11:42 +0100)]
Removed remnants of CVS ("# $Id$" in the beginning of some files)

9 years agoUse autoreconf.
Ulya Trofimovich [Mon, 1 Jun 2015 10:23:38 +0000 (11:23 +0100)]
Use autoreconf.

As stated by autoconf manual, autoreconf is more portable than
self-written scripts because aclocal may be removed in future.

9 years agoSplit header and appropriate source file into smaller parts.
Ulya Trofimovich [Thu, 28 May 2015 20:10:45 +0000 (21:10 +0100)]
Split header and appropriate source file into smaller parts.

9 years agoMoved another utility class to a separate file in 'src/util/'.
Ulya Trofimovich [Thu, 28 May 2015 17:40:41 +0000 (18:40 +0100)]
Moved another utility class to a separate file in 'src/util/'.

9 years agoMoved utility class to a separate file in 'src/util/'.
Ulya Trofimovich [Thu, 28 May 2015 17:26:10 +0000 (18:26 +0100)]
Moved utility class to a separate file in 'src/util/'.

9 years agoDon't output newline instead if label in initial DFA state.
Ulya Trofimovich [Thu, 28 May 2015 11:16:06 +0000 (12:16 +0100)]
Don't output newline instead if label in initial DFA state.

Rationale: the equivalence of initial label to
're2c::label_counter_t::FIRST' is NOT a proper criterion
and pretty-printing shouldn't rely on it. The real criterion
is something like "(first re2c block OR any use block in '-r'
mode) AND first condition in'-c' mode", but it's spurious and
introduces unnecessary complications.

Droping this newline allows us drop equivalence operator for
labels.

Used the following bash script to ensure that all the changes
in tests are caused by missing newline(s):

    #!/bin/bash

    for f2 in *.temp
    do
        f1=${f2%.temp}

        diff1=`diff $f1 $f2 | grep '^< ' | wc -l`
        diff1_line=`diff $f1 $f2 | grep '^< #line' | wc -l`
        diff1_newline=`diff $f1 $f2 | grep '^< $' | wc -l`
        diff2=`diff $f1 $f2 | grep '^> ' | wc -l`
        diff2_line=`diff $f1 $f2 | grep '^> #line' | wc -l`

        # missing: only newlines and line directives
        if [[ $diff1 -ne $((diff1_line + diff1_newline)) ]]
        then
            echo "FAIL1: $f1"
            exit 1
        fi

        # added: only line directives
        if [[ $diff2 -ne $diff2_line ]]
        then
            echo "FAIL2: $f1"
            exit 1
        fi

        # the number of missing line directives
        # equals to the number of added line directives
        if [[ $diff1_line -ne $diff2_line ]]
        then
            echo "FAIL4: $f1"
            exit 1
        fi
    done

    echo "OK"

9 years agoIntroduced a distinct type for labels.
Ulya Trofimovich [Thu, 28 May 2015 10:45:11 +0000 (11:45 +0100)]
Introduced a distinct type for labels.

9 years agoMoved label assignment out of global scope to 're2c::OutputFile'.
Ulya Trofimovich [Wed, 27 May 2015 21:24:19 +0000 (22:24 +0100)]
Moved label assignment out of global scope to 're2c::OutputFile'.

Now label assignment is done with the help of a simple class
're2c::Label' that has very few operations.