Explorer09 [Mon, 19 Mar 2018 14:13:25 +0000 (22:13 +0800)]
scanner: Include-guard flexint_shared.h
This prevents compiler's "redefinition of typedef" warnings or errors
when we could include flexint_shared.h twice, when building flex's own
scanner (scan.c) or in the "multiple_scanners" test in the testsuite.
(Depending on the compiler's flags this might or might not produce
errors, but it's good to guard it anyway.)
Will Estes [Mon, 7 May 2018 19:27:14 +0000 (15:27 -0400)]
po: include update_linguas script in distribution.
Since automake takes a hands off approach to the po/ directory, we
need to list update_linguas.sh in the top Makefile.am. We list it as a
dist_noinst_SCRIPT so that it will get distributed with executeable
permissions. We also list autogen.sh in the same way for the same
reason.
Explorer09 [Thu, 8 Mar 2018 01:59:35 +0000 (09:59 +0800)]
scanner: Skeleton no longer includes integer limit macros.
The [U]INT{8,16,32}_{MIN,MAX} macros are never used in skeleton code.
Having them in skeleton just increases the chance of conflicts in case
that user defines them in non-C99 environment (see issue #307, when
flex code is built in Visual C++ (before VS2013)).
flexint.h is now split in two files. Only "flexint_shared.h" will be
included in skeleton now, which defines flex integral types.
flexint.h contains integer limits macros that would be used in flex
only.
Explorer09 [Thu, 8 Mar 2018 02:04:36 +0000 (10:04 +0800)]
scanner: Fix glibc features.h dependency in skeleton.
Commit a17d79e9c722a6735b6d2a8f152287404f27df32 defines _POSIX_C_SOURCE
to the minimum of 1 if it's not defined in the user's scanner code or
the compiling environment. However in glibc the macros are not yet set
up until one of the libc headers is included. This unfortunately have
made us overwrite the default _POSIX_C_SOURCE value that would be
defined by glibc (200809L at the time of writing), causing regressions
in user code.
Now in this patch:
1. Ensure feature test macros have been set up in glibc before checking
or defining any of them in our skeleton code.
2. Have a more conservative logic when determining the need to define
_POSIX_C_SOURCE (required for fileno()).
Fixes: #313
Note:
It could be tricky for application code to ensure feature test macros
have been set up in glibc, since <features.h> is no portable header and
not meant to be included directly by applications. The way to do it is
to include a libc header which in turn includes <features.h>. However,
many of the glibc headers check __USE_POSIX (a glibc internal macro
defined if _POSIX_C_SOURCE is defined) and determine which interfaces
to expose already, making the headers inappropriate for our goal.
Those which don't depend on _POSIX_C_SOURCE, and are also available
since ANSI C89, are only <assert.h>, <errno.h> and <math.h>.
<assert.h> is finally favored due to other considerations:
- <math.h> check for __USE_XOPEN in glibc, making a dependency on
_XOPEN_SOURCE, besides it exposes much more interfaces than we need.
- In djgpp, <errno.h> depends on _POSIX_SOURCE to hide definitions of
some errno values when it's defined.
- <assert.h> exposes the fewest interfaces among the 3 headers and, at
the time of writing, checks for only C99 (for __func__), C11 (for
_Static_assert), and _GNU_SOURCE when needed.
Explorer09 [Sun, 4 Mar 2018 11:59:42 +0000 (19:59 +0800)]
scanner: Fix scan.l xgettext warnings.
xgettext was not very clever at interpreting lex patterns and can get
confused when seeing unquoted quotation marks, and emit warnings for
them. Now fix the warnings by properly quoting the quotation marks in
lex regex patterns.
Example
Original: \"[^"\n]*\" -> "warning: unterminated string literal"
Fixed: "\""[^""\n]*"\"" -> OK
My basic build test shows that the generated stage1scan.c is
bit-identical to the original.
Kang-Che Sung [Fri, 23 Feb 2018 03:07:29 +0000 (11:07 +0800)]
travis: Hard-code hashes for verifying tarballs
GPG signatures require an external keyserver which might be offline,
which is undesirable for build server use. It's equally secure to just
hard-code the hashes, provided they're trusted (i.e. you verify a hash
against a GPG signature once).
Fixes: #311.
As a side note: the original two signatures
(gettext-0.19.8.1.tar.lz.sig and automake-1.15.1.tar.gz.sig) are signed
against the files' SHA-1 hash.
Explorer09 [Tue, 27 Feb 2018 01:10:12 +0000 (09:10 +0800)]
build: Move dnl comments out of AC_CHECK_FUNCS
Due to a bug, autoheader (2.69) will treat M4 dnl comments in a quoted
argument of AC_CHECK_FUNCS as function tokens and generate a lot of
redundant and useless HAVE_* macros in config.h.in.
(Examples: HAVE_DNL, HAVE_AVAILABLE_, HAVE_BY)
It seems to be this commit dbb4e94dc7bacbcfd4acef4f085ef752fe1aa03f of
mine that revealed this autoheader bug, and the affected config.h.in
had been shipped within flex-2.6.4 release tarball.
I have reported the autoheader bug here:
<https://lists.gnu.org/archive/html/bug-autoconf/2018-02/msg00005.html>
As a workaround, let's move comments out of AC_CHECK_FUNCS.
Explorer09 [Mon, 30 Oct 2017 05:29:26 +0000 (13:29 +0800)]
Obsolete yypad64() macro.
Slightly rewrite the logic in yytbl_data_load() and yytbl_write_pad64()
so they simply check if the bytes read/written are in 8-byte boundary.
No need to calculate how many bytes we need to pad. (Incidentally this
makes smaller code in x86_64.)
For yytbl_hdr_init(), just expand the calculation from the macro.
Explorer09 [Fri, 13 Oct 2017 16:31:01 +0000 (00:31 +0800)]
scanner: temporarily protect against ccl overflow & overwriting.
For ccladd(), if cclp given is a non-last ccl, adding a char into it
will overflow the buffer and overwrite the first char in the next ccl.
For now, add a temporary detection and protection code. (Not sure if
this could happen in user input, but if it could, then you can expect
some "corrupted" behavior for generated scanners.)
scanner: Include flexdef.h at %top block of scan.l
config.h may define macros that alter the API of the standard library
funtions, and so it should be included before any other standard
header, even before the skeleton's standard header inclusion.
For example: config.h may #define _GNU_SOURCE that would expose the
reallocarray() prototype from <stdlib.h> on glibc 2.26+ systems. If we
include <stdlib.h> before config.h, reallocarray() would not be
available for use in lex file since the second include doesn't help
due to header guard.
For now our config.h might `#define malloc rpl_malloc` -- this
substitution must work before including stdlib.h, or else the compiler
will complain about missing prototypes, and may result in incorrect
code in scan.l (gcc warning: return makes pointer from integer without
a cast [-Wint-conversion]).
Don't use program_name in the description of -T/--trace or -V/--version
option. It's ugly when user invokes flex with a long path like
"/home/username/tools/bin/my-custom-built-flex".
This solution is not long term. If possible, the help text should be
modified so that the "flex" name is no longer needed below the first
"Usage:" line. All translations of help text will need to be updated
as well.
scanner: remove BASENAME(); don't strip path from program_name
There's no technical need of stripping path from program_name. I think
the users should be fine if they see the path they use to invoke flex
is diagnostic messages and help texts.
Yes, users will see "Usage: ../flex [OPTIONS]..." now if they invoke
flex with the path "../flex".
The --version output has been changed so that the name field will be
always "flex" or "flex++". If the flex program has been renamed to
"lex" (for compatibility or other reason) this will allow identifying
the implementation name ("flex"). (And it's a recommended practice in
GNU Coding Standards)
In skeleton files comments are indicated by leading `%#` and when
directly read in using `flex -S <skeleton.skl>` they should be
ignored. Example: `flex.skl`.
It's simply to return (regexec(®ex_blank_line, str, 0, NULL, 0) == 0);
The reason for encapsulation is to allow replacing this with a
non-regex method if necessary.
viktor.shepel [Tue, 20 Jun 2017 14:03:42 +0000 (17:03 +0300)]
filter: memory leak free scanner postprocessing.
**Issue:**
Scanner postprocessing leaks memory during correction of `#line`
directives values and generation of C header file.
**Root cause:**
`filter_fix_linedirs` and `filter_tee_header` functions do not
dispose allocated memory.
**Solution:**
Automatically reclaim affected memory by allocating it on stack
insted of heap. Stack allocation should not be a problem as its
only 512 bytes and there is no recursive calls.
Thomas Klausner [Fri, 19 May 2017 08:22:44 +0000 (10:22 +0200)]
scanner: Use reallocarr() when available.
NetBSD had a crash during build. Since the provided substitute for
reallocarray() wasn't working, use NetBSD's reallocarr(). Let
configure choose that function whenever it is available. Use
reallocarray if available. Still fallback if neither is available.
Jeff Smith [Sat, 1 Apr 2017 06:18:12 +0000 (01:18 -0500)]
filter: Output correct #line value for current file.
A #line pre-processor directive specifies the line number and source
file of the following lines. If the source file _is_ the current file,
the line number should be that of the line following the directive. So
the specified line number should be the current line number plus 1.
test: rename some files for non-case-sensitive filesystems.
-C*f and -C*F option-specific test files collide on case-insensitive
file systems, and cause tests to either not be run at all, or to
overwrite each other's files at build time. So rename -C*F ones to
-C*_F.
projects / flex / log