# docs
SRC_DOC = doc/manpage.rst
SRC_DOC_EXT = \
+ doc/manual/contributors.rst_ \
doc/manual/syntax/rules.rst_ \
doc/manual/syntax/interface.rst_ \
doc/manual/syntax/configurations.rst_ \
+ doc/manual/syntax/named_definitions.rst_ \
doc/manual/syntax/regular_expressions.rst_ \
doc/manual/warnings/warnings_list.rst \
doc/manual/warnings/warnings_general.rst \
.INDENT 0.0
.TP
.B \fB\-? \-h \-\-help\fP
-Show a short help screen:
+Show help message.
.TP
.B \fB\-b \-\-bit\-vectors\fP
-Implies \fB\-s\fP\&. Use bit vectors as well to try to
-coax better code out of the compiler. Most useful for
-specifications with more than a few keywords (e.g., for most programming
-languages).
+Optimize conditional jumps using bit masks. Implies \fB\-s\fP\&.
.TP
.B \fB\-c \-\-conditions\fP
-Used for (f)lex\-like condition support.
+Enable support of Flex\-like "conditions": multiple interrelated lexers within one block.
.TP
.B \fB\-d \-\-debug\-output\fP
-Creates a parser that dumps information about
-the current position and the state the parser is in.
-This is useful for debugging parser issues and states. If you use this
-switch, you need to define a \fBYYDEBUG\fP macro, which will be called like a
-function with two parameters: \fBvoid YYDEBUG (int state, char current)\fP\&.
-The first parameter receives the state or \fB\-1\fP and the second parameter
-receives the input at the current cursor.
+Emit \fBYYDEBUG\fP in the generated code.
+\fBYYDEBUG\fP should be defined by the user in the form of a void function with two parameters:
+\fBstate\fP (lexer state or \-1) and \fBsymbol\fP (current input symbol of type \fBYYCTYPE\fP).
.TP
.B \fB\-D \-\-emit\-dot\fP
-Emit Graphviz dot data, which can then be processed
-with e.g., \fBdot \-Tpng input.dot > output.png\fP\&. Please note that
-scanners with many states may crash dot.
+Instead of normal output generate lexer graph in DOT format.
+The output can be converted to PNG with the help of Graphviz (something like \fBdot \-Tpng \-odfa.png dfa.dot\fP).
+Note that large graphs may crash Graphviz.
.TP
.B \fB\-e \-\-ecb\fP
-Generate a parser that supports EBCDIC. The generated
-code can deal with any character up to 0xFF. In this mode, \fBre2c\fP assumes
-an input character size of 1 byte. This switch is incompatible with
-\fB\-w\fP, \fB\-x\fP, \fB\-u\fP, and \fB\-8\fP\&.
+Generate a lexer that reads input in EBCDIC encoding.
+\fBre2c\fP assumes that character range is 0 \-\- 0xFF an character size is 1 byte.
.TP
.B \fB\-f \-\-storable\-state\fP
-Generate a scanner with support for storable state.
+Generate a lexer which can store its inner state.
+This is useful in push\-model lexers which are stopped by an outer program when there is not enough input,
+and then resumed when more input becomes available.
+In this mode users should additionally define
+\fBYYGETSTATE ()\fP and \fBYYSETSTATE (state)\fP macros
+and variables \fByych\fP, \fByyaccept\fP and the \fBstate\fP as part of the lexer state.
.TP
.B \fB\-F \-\-flex\-syntax\fP
-Partial support for flex syntax. When this flag
-is active, named definitions must be surrounded by curly braces and
-can be defined without an equal sign and the terminating semicolon.
-Instead, names are treated as direct double quoted strings.
+Partial support for Flex syntax:
+in this mode named definitions don\(aqt need the equal sign and the terminating semicolon,
+and when used they must be surrounded by curly braces.
+Names without curly braces are treated as double\-quoted strings.
.TP
.B \fB\-g \-\-computed\-gotos\fP
-Generate a scanner that utilizes GCC\(aqs
-computed\-goto feature. That is, \fBre2c\fP generates jump tables whenever a
-decision is of certain complexity (e.g., a lot of if conditions would be
-otherwise necessary). This is only usable with compilers that support this feature.
-Note that this implies \fB\-b\fP and that the complexity threshold can be configured
-using the \fBcgoto:threshold\fP inplace configuration.
+Optimize conditional jumps using non\-standard "computed goto" extension (must be supported by C/C++ compiler).
+\fBre2c\fP generates jump tables only in complex cases with a lot of conditional branches.
+Complexity threshold can be configured with \fBcgoto:threshold\fP configuration.
+This option implies \fB\-b\fP\&.
.TP
.B \fB\-i \-\-no\-debug\-info\fP
-Do not output \fB#line\fP information. This is
-useful when you want use a CMS tool with \fBre2c\fP\(aqs output. You might
-want to do this if you do not want to impose re2c as a build requirement
-for your source.
+Do not output \fB#line\fP information.
+This is useful when the generated code is tracked by some version control system.
.TP
.B \fB\-o OUTPUT \-\-output=OUTPUT\fP
Specify the \fBOUTPUT\fP file.
.TP
.B \fB\-r \-\-reusable\fP
-Allows reuse of scanner definitions with \fB/*!use:re2c */\fP after \fB/*!rules:re2c */\fP\&.
-In this mode, no \fB/*!re2c */\fP block and exactly one \fB/*!rules:re2c */\fP must be present.
-The rules are saved and used by every \fB/*!use:re2c */\fP block that follows.
-These blocks can contain inplace configurations, especially \fBre2c:flags:e\fP,
-\fBre2c:flags:w\fP, \fBre2c:flags:x\fP, \fBre2c:flags:u\fP, and \fBre2c:flags:8\fP\&.
-That way it is possible to create the same scanner multiple times for
-different character types, different input mechanisms, or different output mechanisms.
-The \fB/*!use:re2c */\fP blocks can also contain additional rules that will be appended
-to the set of rules in \fB/*!rules:re2c */\fP\&.
+Allows reuse of \fBre2c\fP rules with \fB/*!rules:re2c */\fP and \fB/*!use:re2c */\fP blocks.
+In this mode simple \fB/*!re2c */\fP blocks are not allowed
+and exactly one \fB/*!rules:re2c */\fP block must be present.
+The rules are saved and used by every \fB/*!use:re2c */\fP block that follows (which may add rules of their own).
+This option allows to reuse the same set of rules with different configurations.
.TP
.B \fB\-s \-\-nested\-ifs\fP
-Generate nested ifs for some switches. Many
-compilers need this assist to generate better code.
+Use nested \fBif\fP statements instead of \fBswitch\fP statements in conditional jumps.
+This usually results in more efficient code with non\-optimizing C/C++ compilers.
.TP
.B \fB\-t HEADER \-\-type\-header=HEADER\fP
-Create a \fBHEADER\fP file that
-contains types for the (f)lex\-like condition support. This can only be
-activated when \fB\-c\fP is in use.
+Generate a \fBHEADER\fP file that contains enum with condition names.
+Requires \fB\-c\fP option.
.TP
.B \fB\-T \-\-tags\fP
Enable submatch extraction with tags.
Enable submatch extraction with POSIX\-style capturing groups.
.TP
.B \fB\-u \-\-unicode\fP
-Generate a parser that supports UTF\-32. The generated
-code can deal with any valid Unicode character up to 0x10FFFF. In this
-mode, \fBre2c\fP assumes an input character size of 4 bytes. This switch is
-incompatible with \fB\-e\fP, \fB\-w\fP, \fB\-x\fP, and \fB\-8\fP\&. This implies \fB\-s\fP\&.
+Generate a lexer that reads input in UTF\-32 encoding.
+\fBre2c\fP assumes that character range is 0 \-\- 0x10FFFF and character size is 4 bytes.
+Implies \fB\-s\fP\&.
.TP
.B \fB\-v \-\-version\fP
Show version information.
.TP
.B \fB\-V \-\-vernum\fP
-Show the version as a number in the MMmmpp (Majorm, minor, patch) format.
+Show version information in \fBMMmmpp\fP format (major, minor, patch).
.TP
.B \fB\-w \-\-wide\-chars\fP
-Generate a parser that supports UCS\-2. The
-generated code can deal with any valid Unicode character up to 0xFFFF.
-In this mode, \fBre2c\fP assumes an input character size of 2 bytes. This
-switch is incompatible with \fB\-e\fP, \fB\-x\fP, \fB\-u\fP, and \fB\-8\fP\&. This implies
-\fB\-s\fP\&.
+Generate a lexer that reads input in UCS\-2 encoding.
+\fBre2c\fP assumes that character range is 0 \-\- 0xFFFF and character size is 2 bytes.
+Implies \fB\-s\fP\&.
.TP
.B \fB\-x \-\-utf\-16\fP
-Generate a parser that supports UTF\-16. The generated
-code can deal with any valid Unicode character up to 0x10FFFF. In this
-mode, \fBre2c\fP assumes an input character size of 2 bytes. This switch is
-incompatible with \fB\-e\fP, \fB\-w\fP, \fB\-u\fP, and \fB\-8\fP\&. This implies \fB\-s\fP\&.
+Generate a lexer that reads input in UTF\-16 encoding.
+\fBre2c\fP assumes that character range is 0 \-\- 0x10FFFF and character size is 2 bytes.
+Implies \fB\-s\fP\&.
.TP
.B \fB\-8 \-\-utf\-8\fP
-Generate a parser that supports UTF\-8. The generated
-code can deal with any valid Unicode character up to 0x10FFFF. In this
-mode, \fBre2c\fP assumes an input character size of 1 byte. This switch is
-incompatible with \fB\-e\fP, \fB\-w\fP, \fB\-x\fP, and \fB\-u\fP\&.
+Generate a lexer that reads input in UTF\-8 encoding.
+\fBre2c\fP assumes that character range is 0 \-\- 0x10FFFF and character size is 1 byte.
.TP
.B \fB\-\-case\-insensitive\fP
-Makes all strings case insensitive. This makes
-"\-quoted expressions behave as \(aq\-quoted expressions.
+Treat single\-quoted and double\-quoted strings as case\-insensitive.
.TP
.B \fB\-\-case\-inverted\fP
-Invert the meaning of single and double quoted
-strings. With this switch, single quotes are case sensitive and double
-quotes are case insensitive.
+Invert the meaning of single\-quoted and double\-quoted strings:
+treat single\-quoted strings as case\-sensitive and double\-quoted strings as case\-insensitive.
.TP
.B \fB\-\-no\-generation\-date\fP
Suppress date output in the generated file.
This option only has effect with \fB\-\-tags\fP or \fB\-\-posix\-captures\fP options.
.TP
.B \fB\-\-no\-optimize\-tags\fP
-Suppress optimization of tag variables (mostly used for debugging).
+Suppress optimization of tag variables (useful for debugging or benchmarking).
.TP
.B \fB\-\-no\-version\fP
Suppress version output in the generated file.
.TP
-.B \fB\-\-no\-generation\-date\fP
-Suppress version output in the generated file.
-.TP
.B \fB\-\-encoding\-policy POLICY\fP
-Specify how \fBre2c\fP must treat Unicode
-surrogates. \fBPOLICY\fP can be one of the following: \fBfail\fP (abort with
-an error when a surrogate is encountered), \fBsubstitute\fP (silently replace
-surrogates with the error code point 0xFFFD), \fBignore\fP (treat surrogates as
-normal code points). By default, \fBre2c\fP ignores surrogates (for backward
-compatibility). The Unicode standard says that standalone surrogates are
-invalid code points, but different libraries and programs treat them
-differently.
+Define the way \fBre2c\fP treats Unicode surrogates.
+\fBPOLICY\fP can be one of the following: \fBfail\fP (abort with an error when a surrogate is encountered),
+\fBsubstitute\fP (silently replace surrogates with the error code point 0xFFFD),
+\fBignore\fP (default, treat surrogates as normal code points).
+The Unicode standard says that standalone surrogates are invalid,
+but real\-world libraries and programs behave in different ways.
.TP
.B \fB\-\-input INPUT\fP
-Specify re2c\(aqs input API. \fBINPUT\fP can be either \fBdefault\fP or \fBcustom\fP\&.
+Specify \fBre2c\fP input API. \fBINPUT\fP can be either \fBdefault\fP or \fBcustom\fP (enables the use of generic API).
.TP
.B \fB\-S \-\-skeleton\fP
-Instead of embedding re2c\-generated code into C/C++
-source, generate a self\-contained program for the same DFA. Most useful
-for correctness and performance testing.
+Ignore user\-defined interface code and generate a self\-contained "skeleton" program.
+Additionally, generate input files with strings derived from the regular grammar
+and compressed match results that are used to verify "skeleton" behavior on all inputs.
+This option is useful for finding bugs in optimizations and code generation.
.TP
.B \fB\-\-empty\-class POLICY\fP
-What to do if the user uses an empty character
-class. \fBPOLICY\fP can be one of the following: \fBmatch\-empty\fP (match empty
-input: pretty illogical, but this is the default for backwards
-compatibility reasons), \fBmatch\-none\fP (fail to match on any input),
-\fBerror\fP (compilation error). Note that there are various ways to
-construct an empty class, e.g., [], [^\ex00\-\exFF],
-[\ex00\-\exFF][\ex00\-\exFF].
-.TP
-.B \fB\-\-dfa\-minimization <table | moore>\fP
-The internal algorithm used by re2c to minimize the DFA (defaults to \fBmoore\fP).
-Both the table filling algorithm and the Moore algorithm should produce the same DFA (up to states relabeling).
-The table filling algorithm is much simpler and slower; it serves as a reference implementation.
+Define the way \fBre2c\fP treats empty character classes.
+\fBPOLICY\fP can be one of the following: \fBmatch\-empty\fP (match empty input: illogical, but default behavior for backwards compatibility reasons),
+\fBmatch\-none\fP (fail to match on any input),
+\fBerror\fP (compilation error).
+.TP
+.B \fB\-\-dfa\-minimization ALGORITHM\fP
+The internal algorithm used by re2c to minimize the DFA.
+\fBALGORITHM\fP can be either \fBmoore\fP (Moore algorithm, the default) or \fBtable\fP (table filling algorithm).
+Both algorithms should produce the same DFA up to states relabeling;
+table filling is much slower and serves as a reference implementation.
.TP
.B \fB\-\-eager\-skip\fP
-This option controls when the generated lexer advances to the next input symbol
-(that is, increments YYCURSOR or invokes YYSKIP).
-By default this happens after transition to the next state,
-but \fB\-\-eager\-skip\fP option allows to override default behavior
-and advance input position immediately after reading input symbol.
-This option is implied by \fB\-\-no\-lookahead\fP\&.
+Make the generated lexer advance the input position "eagerly":
+immediately after reading input symbol.
+By default this happens after transition to the next state.
+Implied by \fB\-\-no\-lookahead\fP\&.
.TP
.B \fB\-\-dump\-nfa\fP
-Generate .dot representation of NFA and dump it on stderr.
+Generate representation of NFA in DOT format and dump it on stderr.
.TP
.B \fB\-\-dump\-dfa\-raw\fP
-Generate .dot representation of DFA under construction and dump it on stderr.
+Generate representation of DFA in DOT format under construction and dump it on stderr.
.TP
.B \fB\-\-dump\-dfa\-det\fP
-Generate .dot representation of DFA immediately after determinization and dump it on stderr.
+Generate representation of DFA in DOT format immediately after determinization and dump it on stderr.
.TP
.B \fB\-\-dump\-dfa\-tagopt\fP
-Generate .dot representation of DFA after tag optimizations and dump it on stderr.
+Generate representation of DFA in DOT format after tag optimizations and dump it on stderr.
.TP
.B \fB\-\-dump\-dfa\-min\fP
-Generate .dot representation of DFA after minimization and dump it on stderr.
+Generate representation of DFA in DOT format after minimization and dump it on stderr.
.TP
.B \fB\-\-dump\-adfa\fP
-Generate .dot representation of DFA after tunneling and dump it on stderr.
+Generate representation of DFA in DOT format after tunneling and dump it on stderr.
.TP
.B \fB\-1 \-\-single\-pass\fP
Deprecated. Does nothing (single pass is the default now).
been turned on so far or will be turned on later.
.TP
.B \fB\-W<warning>\fP
-Turn on a \fBwarning\fP\&.
+Turn on \fBwarning\fP\&.
.TP
.B \fB\-Wno\-<warning>\fP
-Turn off a \fBwarning\fP\&.
+Turn off \fBwarning\fP\&.
.TP
.B \fB\-Werror\-<warning>\fP
-Turn on a \fBwarning\fP and treat it as an error (this implies \fB\-W<warning>\fP).
+Turn on \fBwarning\fP and treat it as an error (this implies \fB\-W<warning>\fP).
.TP
.B \fB\-Wno\-error\-<warning>\fP
Don\(aqt treat this particular \fBwarning\fP as an error. This doesn\(aqt turn off
.TP
.B \fB\-Wcondition\-order\fP
Warn if the generated program makes implicit
-assumptions about condition numbering. You should use either the \fB\-t, \-\-type\-header\fP option or
+assumptions about condition numbering. One should use either the \fB\-t, \-\-type\-header\fP option or
the \fB/*!types:re2c*/\fP directive to generate a mapping of condition names to numbers and then use
the autogenerated condition names.
.TP
.B \fB\-Wempty\-character\-class\fP
Warn if a regular expression contains an empty
-character class. Rationally, trying to match an empty
-character class makes no sense: it should always fail. However, for
-backwards compatibility reasons, \fBre2c\fP allows empty character classes and
+character class. Trying to match an empty character class makes no sense: it should always fail.
+However, for backwards compatibility reasons \fBre2c\fP allows empty character classes and
treats them as empty strings. Use the \fB\-\-empty\-class\fP option to change the default
behavior.
.TP
.B \fB\-Wmatch\-empty\-string\fP
-Warn if a regular expression in a rule is
-nullable (matches an empty string). If the DFA runs in a loop and an empty match
-is unintentional (the input position in not advanced manually), the lexer may
-get stuck in an infinite loop.
+Warn if a rule is nullable (matches an empty string).
+If the lexer runs in a loop and the empty match is unintentional,
+the lexer may unexpectedly hang in an infinite loop.
.TP
.B \fB\-Wswapped\-range\fP
Warn if the lower bound of a range is greater than its upper
Warn if some input strings cause undefined
control flow in the lexer (the faulty patterns are reported). This is the
most dangerous and most common mistake. It can be easily fixed by adding
-the default rule (\fB*\fP) (this rule has the lowest priority, matches any code unit, and consumes
-exactly one code unit).
+the default rule \fB*\fP which has the lowest priority, matches any code unit, and consumes
+exactly one code unit.
.TP
.B \fB\-Wunreachable\-rules\fP
Warn about rules that are shadowed by other rules and will never match.
.B \fB\-Wuseless\-escape\fP
Warn if a symbol is escaped when it shouldn\(aqt be.
By default, re2c silently ignores such escapes, but this may as well indicate a
-typo or error in the escape sequence.
+typo or an error in the escape sequence.
.TP
.B \fB\-Wnondeterministic\-tags\fP
-Warn if tag has \fBn\fP\-th degree of nondeterminism, where \fBn\fP is greater than 1.
+Warn if a tag has \fBn\fP\-th degree of nondeterminism, where \fBn\fP is greater than 1.
.UNINDENT
.SH INTERFACE CODE
.sp
-The user must supply interface code either in the form of C/C++ code
-(macros, functions, variables, etc.) or in the form of \fBINPLACE CONFIGURATIONS\fP\&.
-Which symbols must be defined and which are optional
-depends on the particular use case.
+Below is the list of all symbols which may be used by the lexer in order to interact with the outer world.
+These symbols should be defined by the user,
+either in the form of inplace configurations,
+or as C/C++ variables, functions, macros and other language constructs.
+Which primitives are necessary depends on the particular use case.
.INDENT 0.0
.TP
+.B \fByyaccept\fP
+L\-value of unsigned integral type that is used to hold the number of the last matched rule.
+Explicit definition by the user is necessary only with \fB\-f\fP \fB\-\-storable\-state\fP option.
+.TP
.B \fBYYBACKUP ()\fP
-Backup current input position (used only with generic API).
+Backup current input position (used only with \fB\-\-input custom\fP option).
.TP
.B \fBYYBACKUPCTX ()\fP
-Backup current input position for trailing context (used only with generic API).
+Backup current input position for trailing context (used only with \fB\-\-input custom\fP option).
+.TP
+.B \fByych\fP
+L\-value of type \fBYYCTYPE\fP that is used to hold current input character.
+Explicit definition by the user is necessary only with \fB\-f\fP \fB\-\-storable\-state\fP option.
.TP
.B \fBYYCONDTYPE\fP
-In \fB\-c\fP mode, you can use \fB\-t\fP to generate a file that
-contains the enumeration used as conditions. Each of the values refers
-to a condition of a rule set.
+The type of \fIcondition\fP identifiers (used only with \fB\-c\fP \fB\-\-conditions\fP option).
+Should be generated either with \fB/*!types:re2c*/\fP directive, or with \fB\-t\fP \fB\-\-type\-header\fP option.
.TP
.B \fBYYCTXMARKER\fP
-l\-value of type \fBYYCTYPE *\fP\&.
-The generated code saves trailing context backtracking information in
-\fBYYCTXMARKER\fP\&. The user only needs to define this macro if a scanner
-specification uses trailing context in one or more of its regular
-expressions.
+L\-value of type \fBYYCTYPE *\fP that is used to backup input position of trailing context.
+It is needed only if regular expressions use the lookahead operator \fB/\fP\&.
.TP
.B \fBYYCTYPE\fP
-Type used to hold an input symbol (code unit). Usually
-\fBchar\fP or \fBunsigned char\fP for ASCII, EBCDIC or UTF\-8, or \fIunsigned short\fP
-for UTF\-16 or UCS\-2, or \fBunsigned int\fP for UTF\-32.
+The type of the input characters (\fIcode units\fP).
+Usually it should be \fBunsigned char\fP for ASCII, EBCDIC and UTF\-8 encodings,
+\fBunsigned short\fP for UTF\-16 or UCS\-2 encodings,
+and \fBunsigned int\fP for UTF\-32 encoding.
.TP
.B \fBYYCURSOR\fP
-l\-value of type \fBYYCTYPE *\fP that points to the current input symbol. The generated code advances
-\fBYYCURSOR\fP as symbols are matched. On entry, \fBYYCURSOR\fP is assumed to
-point to the first character of the current token. On exit, \fBYYCURSOR\fP
-will point to the first character of the following token.
-.TP
-.B \fBYYDEBUG (state, current)\fP
-This is only needed if the \fB\-d\fP flag was
-specified. It allows easy debugging of the generated parser by calling a
-user defined function for every state. The function should have the
-following signature: \fBvoid YYDEBUG (int state, char current)\fP\&. The first
-parameter receives the state or \-1 and the second parameter receives the
-input at the current cursor.
+L\-value of type \fBYYCTYPE *\fP that is used as a pointer to the current input symbol.
+Initially \fBYYCURSOR\fP points to the first character and is advanced by the lexer during matching.
+When a rule matches, \fBYYCURSOR\fP points past the last character of the matched string.
+.TP
+.B \fBYYDEBUG (state, symbol)\fP
+A function\-like primitive that is used to dump debug information (only used with \fB\-d\fP \fB\-\-debug\-output\fP option).
+\fBYYDEBUG\fP should return no value and accept two arguments:
+\fBstate\fP (either lexer state or \fB\-1\fP) and \fBsymbol\fP (current input symbol).
.TP
.B \fBYYFILL (n)\fP
-The generated code "calls"" \fBYYFILL (n)\fP when the
-buffer needs (re)filling: at least \fBn\fP additional characters should be
-provided. \fBYYFILL (n)\fP should adjust \fBYYCURSOR\fP, \fBYYLIMIT\fP, \fBYYMARKER\fP,
-and \fBYYCTXMARKER\fP as needed. Note that for typical programming languages
-\fBn\fP will be the length of the longest keyword plus one. The user can
-place a comment of the form \fB/*!max:re2c*/\fP to insert a \fBYYMAXFILL\fP define set to the maximum
-length value.
+A function\-like primitive that is called by the lexer when there is not enough input.
+\fBYYFILL\fP should return no value and supply at least \fBn\fP additional characters.
+Maximal value of \fBn\fP equals \fBYYMAXFILL\fP, which can be obtained with the \fB/*!max:re2c*/\fP directive.
.TP
.B \fBYYGETCONDITION ()\fP
-This define is used to get the condition prior to
-entering the scanner code when using the \fB\-c\fP switch. The value must be
-initialized with a value from the \fBYYCONDTYPE\fP enumeration type.
+R\-value of type \fBYYCONDTYPE\fP that represents current \fIcondition\fP identifier (used only with \fB\-c\fP \fB\-\-conditions\fP option).
.TP
.B \fBYYGETSTATE ()\fP
-The user only needs to define this macro if the \fB\-f\fP
-flag was specified. In that case, the generated code "calls"
-\fBYYGETSTATE ()\fP at the very beginning of the scanner in order to obtain
-the saved state. \fBYYGETSTATE ()\fP must return a signed integer. The value
-must be either \-1, indicating that the scanner is entered for the first
-time, or a value previously saved by \fBYYSETSTATE (s)\fP\&. In the second
-case, the scanner will resume operations right after where the last
-\fBYYFILL (n)\fP was called.
+R\-value of signed integral type that represents current lexer state (used only with \fB\-f\fP \fB\-\-storable\-state\fP option).
+Initial value of lexer state should be \fB\-1\fP\&.
.TP
.B \fBYYLESSTHAN (n)\fP
-Check if less than \fBn\fP input characters are left (used only with generic API).
+R\-value of boolean type that is \fBtrue\fP if and only if there is less than \fBn\fP input characters left (used only with \fB\-\-input custom\fP option).
.TP
.B \fBYYLIMIT\fP
-An expression of type \fBYYCTYPE *\fP that marks the end of the buffer \fBYYLIMIT[\-1]\fP
-is the last character in the buffer). The generated code repeatedly
-compares \fBYYCURSOR\fP to \fBYYLIMIT\fP to determine when the buffer needs
-(re)filling.
+R\-value of type \fBYYCTYPE *\fP that marks the end of input (\fBYYLIMIT[\-1]\fP should be the last input character).
+Lexer compares \fBYYCURSOR\fP and \fBYYLIMIT\fP in order to determine if there is enough input characters left.
.TP
.B \fBYYMARKER\fP
-An l\-value of type \fBYYCTYPE *\fP\&.
-The generated code saves backtracking information in \fBYYMARKER\fP\&. Some
-simple scanners might not use this.
+L\-value of type \fBYYCTYPE *\fP used to backup input position of successful match.
+This might be necessary if there is an overlapping longer rule that might also match.
.TP
.B \fBYYMTAGP (t)\fP
-Append current input position to the history of tag \fBt\fP\&.
+Append current input position to the history of \fIm\-tag\fP \fBt\fP (used only with \fB\-T\fP \fB\-\-tags\fP option).
.TP
.B \fBYYMTAGN (t)\fP
-Append default value to the history of tag \fBt\fP\&.
+Append default value to the history of \fIm\-tag\fP \fBt\fP (used only with \fB\-T\fP \fB\-\-tags\fP option).
.TP
.B \fBYYMAXFILL\fP
-This will be automatically defined by \fB/*!max:re2c*/\fP blocks as explained above.
+Integral constant that denotes maximal value of \fBYYFILL\fP argument
+and is autogenerated by \fB/*!max:re2c*/\fP directive.
.TP
.B \fBYYMAXNMATCH\fP
-This will be automatically defined by \fB/*!maxnmatch:re2c*/\fP\&.
+Integral constant that denotes maximal number of capturing groups in a rule
+and is autogenerated by \fB/*!maxnmatch:re2c*/\fP directive (used only with \fB\-\-posix\-captures\fP option).
+.TP
+.B \fByynmatch\fP
+L\-value of unsigned integral type that is used to hold the number of capturing groups in the matching rule.
+Used only with \fB\-P\fP \fB\-\-posix\-captures\fP option.
.TP
.B \fBYYPEEK ()\fP
-Get current input character (used only with generic API).
+R\-value of type \fBYYCTYPE\fP that denotes current input character (used only with \fB\-\-input custom\fP option).
+.TP
+.B \fByypmatch\fP
+An array of l\-values that are used to hold the values of \fIs\-tags\fP
+corresponding to the capturing parentheses in the matching rule.
+The length of array must be at least \fByynmatch * 2\fP (ideally \fBYYMAXNMATCH * 2\fP).
+Used only with \fB\-P\fP \fB\-\-posix\-captures\fP option.
.TP
.B \fBYYRESTORE ()\fP
-Restore input position (used only with generic API).
+Restore input position (used only with \fB\-\-input custom\fP option).
.TP
.B \fBYYRESTORECTX ()\fP
-Restore input position from the value of trailing context (used only with generic API).
+Restore input position from the value of trailing context (used only with \fB\-\-input custom\fP option).
.TP
.B \fBYYRESTORETAG (t)\fP
-Restore input position from the value of tag \fBt\fP (used only with generic API).
-.TP
-.B \fBYYSETCONDITION (c)\fP
-This define is used to set the condition in
-transition rules. This is only being used when \fB\-c\fP is active and
-transition rules are being used.
-.TP
-.B \fBYYSETSTATE (s)\fP
-The user only needs to define this macro if the \fB\-f\fP
-flag was specified. In that case, the generated code "calls"
-\fBYYSETSTATE\fP just before calling \fBYYFILL (n)\fP\&. The parameter to
-\fBYYSETSTATE\fP is a signed integer that uniquely identifies the specific
-instance of \fBYYFILL (n)\fP that is about to be called. Should the user
-wish to save the state of the scanner and have \fBYYFILL (n)\fP return to
-the caller, all he has to do is store that unique identifier in a
-variable. Later, when the scanner is called again, it will call
-\fBYYGETSTATE ()\fP and resume execution right where it left off. The
-generated code will contain both \fBYYSETSTATE (s)\fP and \fBYYGETSTATE\fP even
-if \fBYYFILL (n)\fP is disabled.
+Restore input position from the value of \fIs\-tag\fP \fBt\fP (used only with \fB\-\-input custom\fP option).
+.TP
+.B \fBYYSETCONDITION (condition)\fP
+Set current \fIcondition\fP identifier to \fBcondition\fP (used only with \fB\-c\fP \fB\-\-conditions\fP option).
+.TP
+.B \fBYYSETSTATE (state)\fP
+Set current lexer state to \fBstate\fP (used only with \fB\-f\fP \fB\-\-storable\-state\fP option).
+Parameter \fBstate\fP is of signed integral type.
.TP
.B \fBYYSKIP ()\fP
Advance input position to the next character (used only with generic API).
.TP
.B \fBYYSTAGP (t)\fP
-Save current input position to tag \fBt\fP (used only with generic API).
+Save current input position to \fIs\-tag\fP \fBt\fP (used only with \fB\-T\fP \fB\-\-tags\fP and \fB\-\-input custom\fP option).
.TP
.B \fBYYSTAGN (t)\fP
-Save default value to tag \fBt\fP (used only with generic API).
+Save default value to \fIs\-tag\fP \fBt\fP (used only with \fB\-T\fP \fB\-\-tags\fP and \fB\-\-input custom\fP options).
.UNINDENT
.SH SYNTAX
.sp
-Code for \fBre2c\fP consists of a set of \fBRULES\fP, \fBNAMED DEFINITIONS\fP, \fBCODE\fP and
-\fBINPLACE CONFIGURATIONS\fP\&.
+A program can contain any number of \fBre2c\fP blocks.
+Each block consists of a sequence of \fBRULES\fP, \fBNAMED DEFINITIONS\fP and \fBINPLACE CONFIGURATIONS\fP\&.
.SS RULES
.sp
-Each rule consist of a regular expression (see \fBREGULAR EXPRESSIONS\fP) accompanied with a block of C/C++ code
-which is to be executed when the associated regular expression is
-matched. You can either start the code with an opening curly brace or
-the sequence \fB:=\fP\&. If you use an opening curly brace, \fBre2c\fP will count brace depth
-and stop looking for code automatically. Otherwise, curly braces are not
-allowed and \fBre2c\fP stops looking for code at the first line that does
-not begin with whitespace. If two or more rules overlap, the first rule
-is preferred.
-.sp
-There is one special rule that can be used instead of regular expression: the default rule \fB*\fP\&.
-Note that the default rule \fB*\fP differs from \fB[^]\fP: the default rule has the lowest priority,
-matches any code unit (either valid or invalid) and always consumes exactly one character.
-\fB[^]\fP, on the other hand, matches any valid code point (not the same as a code unit) and can consume multiple
-code units. In fact, when a variable\-length encoding is used, \fB*\fP
-is the only possible way to match an invalid input character.
-.sp
-In general, all rules have the form:
-.INDENT 0.0
-.INDENT 3.5
-\fBregular\-expression\-or\-* code\fP
-.UNINDENT
-.UNINDENT
-.sp
-If \fB\-c\fP is active, then each regular expression is preceded by a list
-of comma\-separated condition names. Besides the normal naming rules, there
-are two special cases: \fB<*>\fP (these rules are merged to all conditions)
-and \fB<>\fP (these rules cannot have an associated regular expression;
-their code is merged to all actions). Non\-empty rules may furthermore specify the new
-condition. In that case, \fBre2c\fP will generate the necessary code to
-change the condition automatically. Rules can use \fB:=>\fP as a shortcut
-to automatically generate code that not only sets the
-new condition state but also continues execution with the new state. A
-shortcut rule should not be used in a loop where there is code between
-the start of the loop and the \fBre2c\fP block unless \fBre2c:cond:goto\fP
-is changed to \fBcontinue\fP\&. If some code is needed before all rules (though not before simple jumps), you
-can insert it with \fB<!>\fP pseudo\-rules.
-.INDENT 0.0
-.INDENT 3.5
-\fB<condition\-list\-or\-*> regular\-expression\-or\-* code\fP
+Rules consist of a regular expression followed by a user\-defined action:
+a block of C/C++ code) that is executed in case of sucessful match.
+Action can be either an arbitrary block of code enclosed in curly braces \fB{\fP and \fB}\fP,
+or a block of code without curly braces preceded with \fB:=\fP and ended with a newline that is not followed by a whitespace.
.sp
-\fB<condition\-list\-or\-*> regular\-expression\-or\-* => condition code\fP
+If multiple rules match, \fBre2c\fP prefers the longest match.
+If rules match the same string, the earlier rule has priority.
.sp
-\fB<condition\-list\-or\-*> regular\-expression\-or\-* :=> condition\fP
+There is one special kind of rule: the \fIdefault rule\fP with \fB*\fP instaed of the regular expression.
+It always has the lowest priority, matches any \fIcode unit\fP (either valid or invalid) and consumes exactly one \fIcode unit\fP\&.
+Note that \fIdefault rule\fP is not the same as \fB[^]\fP, which
+matches any valid \fIcode point\fP and can consume multiple \fIcode units\fP\&.
+In case of variable\-length encodings \fB*\fP is the only possible way to match invalid input character.
.sp
-\fB<> code\fP
-.sp
-\fB<> => condition code\fP
-.sp
-\fB<> :=> condition\fP
-.sp
-\fB<!condition\-list> code\fP
-.sp
-\fB<!> code\fP
-.UNINDENT
-.UNINDENT
+If \fB\-c\fP \fB\-\-conditions\fP option is used, then rules have more complex form
+described in the section about conditions.
.SS NAMED DEFINITIONS
.sp
-Named definitions are of the form:
-.INDENT 0.0
-.INDENT 3.5
-\fBname = regular\-expression;\fP
-.UNINDENT
-.UNINDENT
-.sp
-If \fB\-F\fP is active, then named definitions are also of the form:
-.INDENT 0.0
-.INDENT 3.5
-\fBname { regular\-expression }\fP
-.UNINDENT
-.UNINDENT
+Named definitions are of the form \fBname = regexp ;\fP
+where \fBname\fP is an identifier that consists of letters, digits and underscores,
+and \fBregexp\fP is a regular expression.
+With \fB\-F\fP \fB\-\-flex\-syntax\fP option named definitions are also of the form \fBname regexp\fP\&.
+Each name should be defined before it is used.
.SS INPLACE CONFIGURATIONS
.INDENT 0.0
.TP
If non\-zero, the argument is generated unless \fBre2c:define:YYFILL:naked\fP is set to non\-zero.
.UNINDENT
.SS REGULAR EXPRESSIONS
+.sp
+re2c uses the following syntax for regular expressions:
.INDENT 0.0
-.TP
-.B \fB"foo"\fP
-literal string \fB"foo"\fP\&. ANSI\-C escape sequences can be used.
-.TP
-.B \fB\(aqfoo\(aq\fP
-literal string \fB"foo"\fP (case insensitive for characters [a\-zA\-Z]).
-ANSI\-C escape sequences can be used.
-.TP
-.B \fB[xyz]\fP
-character class; in this case, the regular expression matches \fBx\fP, \fBy\fP, or \fBz\fP\&.
-.TP
-.B \fB[abj\-oZ]\fP
-character class with a range in it; matches \fBa\fP, \fBb\fP, any letter from \fBj\fP through \fBo\fP, or \fBZ\fP\&.
-.TP
-.B \fB[^class]\fP
-inverted character class.
-.TP
-.B \fBr \e s\fP
-match any \fBr\fP which isn\(aqt \fBs\fP\&. \fBr\fP and \fBs\fP must be regular expressions
-which can be expressed as character classes.
-.TP
-.B \fBr*\fP
-zero or more occurrences of \fBr\fP\&.
-.TP
-.B \fBr+\fP
-one or more occurrences of \fBr\fP\&.
-.TP
-.B \fBr?\fP
-optional \fBr\fP\&.
-.TP
-.B \fB(r)\fP
-\fBr\fP; parentheses are used to override precedence.
-.TP
-.B \fBr s\fP
-\fBr\fP followed by \fBs\fP (concatenation).
-.TP
-.B \fBr | s\fP
-\fBr\fP or \fBs\fP (alternative).
-.TP
-.B \fBr\fP / \fBs\fP
-\fBr\fP but only if it is followed by \fBs\fP\&. Note that \fBs\fP is not
-part of the matched text. This type of regular expression is called
-"trailing context". Trailing context can only be at the end of a rule
-and cannot be part of a named definition.
-.TP
-.B \fBr{n}\fP
-matches \fBr\fP exactly \fBn\fP times.
-.TP
-.B \fBr{n,}\fP
-matches \fBr\fP at least \fBn\fP times.
-.TP
-.B \fBr{n,m}\fP
-matches \fBr\fP at least \fBn\fP times, but not more than \fBm\fP times.
-.TP
-.B \fB\&.\fP
-match any character except newline.
-.TP
-.B \fBname\fP
-matches a named definition as specified by \fBname\fP only if \fB\-F\fP is
-off. If \fB\-F\fP is active then this behaves like it was enclosed in double
-quotes and matches the string "name".
-.TP
-.B \fB@stag\fP
-save input position at which \fB@stag\fP matches in a variable named \fBstag\fP
-.TP
-.B \fB#mtag\fP
-save all input positions at which \fB#mtag\fP matches in a variable named \fBmtag\fP
-(multiple positions are possible if \fB#mtag\fP is enclosed in a repetition subexpression that matches several times)
+.IP \(bu 2
+\fB"foo"\fP case\-sensitive string literal
+.IP \(bu 2
+\fB\(aqfoo\(aq\fP case\-insensitive string literal
+.IP \(bu 2
+\fB[a\-xyz]\fP, \fB[^a\-xyz]\fP character class (possibly negated)
+.IP \(bu 2
+\fB\&.\fP any character except newline
+.IP \(bu 2
+\fBR \e S\fP difference of character classes \fBR\fP and \fBS\fP
+.IP \(bu 2
+\fBR*\fP zero or more occurrences of \fBR\fP
+.IP \(bu 2
+\fBR+\fP zero or more occurrences of \fBR\fP
+.IP \(bu 2
+\fBR?\fP optional \fBR\fP
+.IP \(bu 2
+\fBR{n}\fP repetition of \fBR\fP exactly \fBn\fP times
+.IP \(bu 2
+\fBR{n,}\fP repetition of \fBR\fP at least \fBn\fP times
+.IP \(bu 2
+\fBR{n,m}\fP repetition of \fBR\fP from \fBn\fP to \fBm\fP times
+.IP \(bu 2
+\fB(R)\fP just \fBR\fP; parentheses are used to override precedence or for POSIX\-style submatch
+.IP \(bu 2
+\fBR S\fP concatenation: \fBR\fP followed by \fBS\fP
+.IP \(bu 2
+\fBR | S\fP alternative: \fBR or S\fP
+.IP \(bu 2
+\fBR / S\fP loohakead: \fBR\fP followed by \fBS\fP, but \fBS\fP is not consumed
+.IP \(bu 2
+\fBname\fP the regular expression defined as \fBname\fP (or literal string \fB"name"\fP in Flex compatibility mode)
+.IP \(bu 2
+\fB{name}\fP the regular expression defined as \fBname\fP in Flex compatibility mode
+.IP \(bu 2
+\fB@stag\fP an \fIs\-tag\fP: saves the last input position at which \fB@stag\fP matches in a variable named \fBstag\fP
+.IP \(bu 2
+\fB#mtag\fP an \fIm\-tag\fP: saves all input positions at which \fB#mtag\fP matches in a variable named \fBmtag\fP
.UNINDENT
.sp
-Character classes and string literals may contain octal or hexadecimal
-character definitions and the following set of escape sequences:
-\fB\ea\fP, \fB\eb\fP, \fB\ef\fP, \fB\en\fP, \fB\er\fP, \fB\et\fP, \fB\ev\fP, \fB\e\e\fP\&. An octal character is defined by a backslash
-followed by its three octal digits (e.g., \fB\e377\fP).
-Hexadecimal characters from 0 to 0xFF are defined by a backslash, a lower
-case \fBx\fP and two hexadecimal digits (e.g., \fB\ex12\fP). Hexadecimal characters from 0x100 to 0xFFFF are defined by a backslash, a lower case
-\fB\eu\(ga\(gaor an upper case \(ga\(ga\eX\fP, and four hexadecimal digits (e.g., \fB\eu1234\fP).
-Hexadecimal characters from 0x10000 to 0xFFFFffff are defined by a backslash, an upper case \fB\eU\fP,
-and eight hexadecimal digits (e.g., \fB\eU12345678\fP).
-.sp
-The only portable "any" rule is the default rule, \fB*\fP\&.
+Character classes and string literals may contain the following escape sequences:
+\fB\ea\fP, \fB\eb\fP, \fB\ef\fP, \fB\en\fP, \fB\er\fP, \fB\et\fP, \fB\ev\fP, \fB\e\e\fP, octal escapes \fB\eooo\fP and hexadecimal escapes \fB\exhh\fP, \fB\euhhhh\fP and \fB\eUhhhhhhhh\fP\&.
.SH SUBMATCH EXTRACTION
.sp
\fBre2c\fP supports two kinds of submatch extraction.
.sp
For further details see \fBhttp://re2c.org/examples/examples.html\fP page on the website
or \fBre2c/examples/\fP subdirectory of \fBre2c\fP distribution.
-.SH SCANNER WITH STORABLE STATES
-.sp
-When the \fB\-f\fP flag is specified, \fBre2c\fP generates a scanner that can
-store its current state, return to its caller, and later resume
-operations exactly where it left off.
-.sp
-The default mode of operation in \fBre2c\fP is a
-"pull" model, where the scanner asks for extra input whenever it needs it. However, this mode of operation assumes that the scanner is the "owner" of the parsing loop, and that may not always be convenient.
+.SH STORABLE STATE
.sp
-Typically, if there is a preprocessor ahead of the scanner in the
-stream, or for that matter, any other procedural source of data, the
-scanner cannot "ask" for more data unless both the scanner and the source
-live in separate threads.
+With \fB\-f\fP \fB\-\-storable\-state\fP option re2c generates a lexer that can
+store its current state, return to the caller, and later resume operations exactly where it left off.
+The default mode of operation in re2c is a "pull" model, where the lexer "pulls" more input whenever it needs it.
+However, this mode of operation assumes that the lexer is the owner of the parsing loop, and that may not always be convenient.
.sp
-The \fB\-f\fP flag is useful exactly for situations like that: it lets users design
-scanners that work in a "push" model, i.e., a model where data is fed to the
-scanner chunk by chunk. When the scanner runs out of data to consume, it
-stores its state and returns to the caller. When more input data is
-fed to the scanner, it resumes operations exactly where it left off.
+Storable state is useful exactly for situations like that: it allows to construct
+lexers that work in a "push" model, where data is fed to the lexer chunk by chunk.
+When the lexer needs more input, it stores its state and returns to the caller.
+Later, when more input becomes available, it resumes operations exactly where it stopped.
.sp
Changes needed compared to the "pull" model:
.INDENT 0.0
.IP \(bu 2
-The user has to supply macros named \fBYYSETSTATE ()\fP and \fBYYGETSTATE (state)\fP\&.
-.IP \(bu 2
-The \fB\-f\fP option inhibits declaration of \fByych\fP and \fByyaccept\fP, so the
-user has to declare them and save and restore them where required.
-In the \fBexamples/push_model/push.re\fP example, these are declared as
-fields of a (C++) class of which the scanner is a method, so they do
-not need to be saved/restored explicitly. For C, they could, e.g., be made
-macros that select fields from a structure passed in as a parameter.
-Alternatively, they could be declared as local variables, saved with
-\fBYYFILL (n)\fP when it decides to return and restored upon entering the
-function. Also, it could be more efficient to save the state from
-\fBYYFILL (n)\fP because \fBYYSETSTATE (state)\fP is called unconditionally.
-\fBYYFILL (n)\fP however does not get \fBstate\fP as a parameter, so we would have
-to store state in a local variable by \fBYYSETSTATE (state)\fP\&.
-.IP \(bu 2
-Modify \fBYYFILL (n)\fP to return (from the function calling it) if more input is needed.
-.IP \(bu 2
-Modify the caller to recognize if more input is needed and respond appropriately.
-.IP \(bu 2
-The generated code will contain a switch block that is used to
-restore the last state by jumping behind the corresponding \fBYYFILL (n)\fP
-call. This code is automatically generated in the epilogue of the first \fB/*!re2c */\fP
-block. It is possible to trigger generation of the \fBYYGETSTATE ()\fP
-block earlier by placing a \fB/*!getstate:re2c*/\fP comment. This is especially useful when the scanner code should be
-wrapped inside a loop.
+Define \fBYYSETSTATE ()\fP and \fBYYGETSTATE (state)\fP\&.
+.IP \(bu 2
+Define \fByych\fP, \fByyaccept\fP and \fBstate\fP variables as a part of persistent lexer state.
+\fBstate\fP should be initialized to \fB\-1\fP\&.
+.IP \(bu 2
+\fBYYFILL\fP should return to the outer program instead of trying to supply more input.
+Return code should indicate that lexer needs more input.
+.IP \(bu 2
+The outer program should recognize situations when lexer needs more input
+and respond appropriately.
+.IP \(bu 2
+Use \fB/*!getstate:re2c*/\fP directive if it is necessary to execute any code
+before entering the lexer.
+.IP \(bu 2
+Use configurations \fBstate:abort\fP and \fBstate:nextlabel\fP to tweak the generated code.
.UNINDENT
+.SH CONDITIONS
+.sp
+\fIConditions\fP are enabled with \fB\-c\fP \fB\-\-conditions\fP\&.
+This option allows to encode multiple interrelated lexers within the same re2c block.
.sp
-Please see \fBexamples/push_model/push.re\fP for an example of a "push" model scanner. The
-generated code can be tweaked with inplace configurations \fBstate:abort\fP
-and \fBstate:nextlabel\fP\&.
-.SH SCANNER WITH CONDITION SUPPORT
+Each lexer corresponds to a single \fIcondition\fP\&.
+It starts with a label of the form \fByyc_name\fP,
+where \fBname\fP is \fIcondition\fP name
+and \fByyc\fP prefix can be adjusted with configuration \fBre2c:condprefix\fP\&.
+Different lexers are separated with a comment \fB/* *********************************** */\fP
+which can be adjusted with configuration \fBre2c:cond:divider\fP\&.
.sp
-You can precede regular expressions with a list of condition names when
-using the \fB\-c\fP switch. \fBre2c\fP will then generate a scanner block for
-each condition, and each of the generated blocks will have its own
-precondition. The precondition is given by the interface define
-\fBYYGETCONDITON()\fP and must be of type \fBYYCONDTYPE\fP\&.
+Furthermore, each \fIcondition\fP has a unique identifier of the form \fByycname\fP,
+where \fBname\fP is condition name
+and \fByyc\fP prefix can be adjusted with configuration \fBre2c:condenumprefix\fP\&.
+Identifiers have the type \fBYYCONDTYPE\fP and should be generated with \fB/*!types:re2c*/\fP directive or \fB\-t\fP \fB\-\-type\-header\fP option.
+Users shouldn\(aqt define these identifiers manually, as the order of \fIconditions\fP is not specified.
.sp
-There are two special rule types. First, the rules of the condition \fB<*>\fP
-are merged to all conditions (note that they have a lower priority than
-other rules of that condition). And second, the empty condition list
-allows to provide a code block that does not have a scanner part,
-meaning it does not allow any regular expressions. The condition value
-referring to this special block is always the one with the enumeration
-value 0. This way the code of this special rule can be used to
-initialize a scanner. It is in no way necessary to have these rules: but
-sometimes it is helpful to have a dedicated uninitialized condition
-state.
+Before all \fIconditions\fP re2c generates entry code that checks the current \fIcondition\fP identifier
+and transfers control flow to the start label of the active \fIcondition\fP\&.
+After matching some rule of this \fIcondition\fP,
+lexer may either transfer control flow back to the entry code (after executing the associated action and optionally setting another \fIcondition\fP with \fB=>\fP),
+or use \fB:=>\fP shortcut and transition directly to the start label of another \fIcondition\fP (skipping the action and the entry code).
+Configuration \fBre2c:cond:goto\fP allows to change the default behavior.
+.sp
+Syntactically each rule must be preceded with a list of comma\-separated \fIcondition\fP names or a wildcard \fB*\fP
+enclosed in angle brackets \fB<\fP and \fB>\fP\&.
+Wildcard means "any condition" and is semantically equivalent to listing all condition names.
+Here \fBregexp\fP is a regular expression, \fBdefault\fP refers to the \fIdefault rule\fP \fB*\fP,
+and \fBaction\fP is a block of C/C++ code.
+.INDENT 0.0
+.IP \(bu 2
+\fB<conditions\-or\-wildcard> regexp\-or\-default action\fP
+.IP \(bu 2
+\fB<conditions\-or\-wildcard> regexp\-or\-default => condition action\fP
+.IP \(bu 2
+\fB<conditions\-or\-wildcard> regexp\-or\-default :=> condition\fP
+.UNINDENT
.sp
-Non empty rules allow to specify the new condition, which makes them
-transition rules. Besides generating calls for the
-\fBYYSETCONDTITION\fP define, no other special code is generated.
+Rules with an exclamation mark \fB!\fP in front of condition list have a special meaning:
+they have no regular expression,
+and the associated action is merged as an entry code to actions of normal rules.
+This might be a convenient place to peform a routine task that is common to all rules.
+.INDENT 0.0
+.IP \(bu 2
+\fB<!conditions\-or\-wildcard> action\fP
+.UNINDENT
.sp
-There is another kind of special rule that allows to prepend code to any
-code block of all rules of a certain set of conditions or to all code
-blocks of all rules. This can be helpful when some operation is common
-among rules. For instance, this can be used to store the length of the
-scanned string. These special setup rules start with an exclamation mark
-followed by either a list of conditions \fB<! condition, ... >\fP or a star
-\fB<!*>\fP\&. When \fBre2c\fP generates the code for a rule whose state does not have a
-setup rule and a starred setup rule is present, the starred setup code will be
-used as setup code.
+Another special form of rules with an empty condition list \fB<>\fP and no regular expression
+allows to specify an "entry condition" that can be used to execute code before entering the lexer.
+It is semantically equivalent to a condition with number zero, name \fB0\fP and an empty regular expression.
+.INDENT 0.0
+.IP \(bu 2
+\fB<> action\fP
+.IP \(bu 2
+\fB<> => condition action\fP
+.IP \(bu 2
+\fB<> :=> condition\fP
+.UNINDENT
.SH ENCODINGS
.sp
\fBre2c\fP supports the following encodings: ASCII (default), EBCDIC (\fB\-e\fP),
check for invalid input, the only correct way to do so is to use the default
rule (\fB*\fP). Note that the full range rule (\fB[^]\fP) won\(aqt catch invalid code units when a variable\-length encoding is used
(\fB[^]\fP means "any valid code point", whereas the default rule (\fB*\fP) means "any possible code unit").
-.SH GENERIC INPUT API
+.SH GENERIC API
.sp
-\fBre2c\fP usually operates on input with pointer\-like primitives
+By default \fBre2c\fP operates on input using pointer\-like primitives
\fBYYCURSOR\fP, \fBYYMARKER\fP, \fBYYCTXMARKER\fP, and \fBYYLIMIT\fP\&.
+Normally pointer\-like primitives are defined as variables of type \fBYYCTYPE*\fP,
+but it is possible to use STL iterators or any other abstraction as long as it syntactically fits into the following use cases:
+.INDENT 0.0
+.IP \(bu 2
+\fB++YYCURSOR;\fP
+.IP \(bu 2
+\fByych = *YYCURSOR;\fP
+.IP \(bu 2
+\fByych = *++YYCURSOR;\fP
+.IP \(bu 2
+\fByych = *(YYMARKER = YYCURSOR);\fP
+.IP \(bu 2
+\fByych = *(YYMARKER = ++YCURSOR);\fP
+.IP \(bu 2
+\fBYYMARKER = YYCURSOR;\fP
+.IP \(bu 2
+\fBYYMARKER = ++YYCURSOR;\fP
+.IP \(bu 2
+\fBYYCURSOR = YYMARKER;\fP
+.IP \(bu 2
+\fBYYCTXMARKER = YYCURSOR + 1;\fP
+.IP \(bu 2
+\fBYYCURSOR = YYCTXMARKER;\fP
+.IP \(bu 2
+\fBif (YYLIMIT <= YYCURSOR) ...\fP
+.IP \(bu 2
+\fBif ((YYLIMIT \- YYCURSOR) < n) ...\fP
+.IP \(bu 2
+\fBYYDEBUG (label, *YYCURSOR);\fP
+.UNINDENT
.sp
-The generic input API (enabled with the \fB\-\-input custom\fP switch) allows
-customizing input operations. In this mode, \fBre2c\fP will express all
-operations on input in terms of the following primitives:
+If this input model is too restrictive, then it is possible to use generic input API enabled with \fB\-\-input custom\fP option.
+In this mode all input operations are expressed in terms of the primitives below.
+These primitives can be defined in any suitable way; one doesn\(aqt have to stick to the pointer semantics.
+For example, it is possible to read input directly from file without any buffering,
+or to disable \fBYYFILL\fP mechanism and perform end\-of\-input checking on each input character from inside of \fBYYPEEK\fP or \fBYYSKIP\fP\&.
.INDENT 0.0
-.INDENT 3.5
-.TS
-center;
-|l|l|.
-_
-T{
+.IP \(bu 2
\fBYYPEEK ()\fP
-T} T{
-get current input character
-T}
-_
-T{
+.IP \(bu 2
\fBYYSKIP ()\fP
-T} T{
-advance to next character
-T}
-_
-T{
+.IP \(bu 2
\fBYYBACKUP ()\fP
-T} T{
-backup current input position
-T}
-_
-T{
+.IP \(bu 2
\fBYYBACKUPCTX ()\fP
-T} T{
-backup current input position for trailing context
-T}
-_
-T{
+.IP \(bu 2
\fBYYSTAGP (t)\fP
-T} T{
-save current input position to tag \fBt\fP
-T}
-_
-T{
+.IP \(bu 2
\fBYYSTAGN (t)\fP
-T} T{
-save default value to tag \fBt\fP
-T}
-_
-T{
+.IP \(bu 2
\fBYYMTAGP (t)\fP
-T} T{
-append input position to the history of tag \fBt\fP
-T}
-_
-T{
+.IP \(bu 2
\fBYYMTAGN (t)\fP
-T} T{
-append default value to the history of tag \fBt\fP
-T}
-_
-T{
+.IP \(bu 2
\fBYYRESTORE ()\fP
-T} T{
-restore current input position
-T}
-_
-T{
+.IP \(bu 2
\fBYYRESTORECTX ()\fP
-T} T{
-restore current input position for trailing context
-T}
-_
-T{
+.IP \(bu 2
\fBYYRESTORETAG (t)\fP
-T} T{
-restore current input position from tag \fBt\fP
-T}
-_
-T{
+.IP \(bu 2
\fBYYLESSTHAN (n)\fP
-T} T{
-check if less than \fBn\fP input characters are left
-T}
-_
-.TE
-.UNINDENT
.UNINDENT
.sp
-A couple of useful links that provide some examples:
+Default input model can be expressed in terms of generic API as follows
+(except for \fBYMTAGP\fP and \fBYYMTAGN\fP, which have no default implementation):
.INDENT 0.0
-.IP 1. 3
-\fI\%http://skvadrik.github.io/aleph_null/posts/re2c/2015\-01\-13\-input_model.html\fP
-.IP 2. 3
-\fI\%http://skvadrik.github.io/aleph_null/posts/re2c/2015\-01\-15\-input_model_custom.html\fP
+.IP \(bu 2
+\fB#define YYPEEK () *YYCURSOR\fP
+.IP \(bu 2
+\fB#define YYSKIP () ++YYCURSOR\fP
+.IP \(bu 2
+\fB#define YYBACKUP () YYMARKER = YYCURSOR\fP
+.IP \(bu 2
+\fB#define YYBACKUPCTX () YYCTXMARKER = YYCURSOR\fP
+.IP \(bu 2
+\fB#define YYRESTORE () YYCURSOR = YYMARKER\fP
+.IP \(bu 2
+\fB#define YYRESTORECTX () YYCURSOR = YYCTXMARKER\fP
+.IP \(bu 2
+\fB#define YYRESTORERAG (t) YYCURSOR = t\fP
+.IP \(bu 2
+\fB#define YYLESSTHAN (n) YYLIMIT \- YYCURSOR < n\fP
+.IP \(bu 2
+\fB#define YYSTAGP (t) t = YYCURSOR\fP
+.IP \(bu 2
+\fB#define YYSTAGN (t) t = NULL\fP
.UNINDENT
.SH SEE ALSO
.sp
See also: flex(1), lex(1), quex (\fI\%http://quex.sourceforge.net\fP).
.SH AUTHORS
.sp
-Peter Bumbulis \fI\%peter@csg.uwaterloo.ca\fP
-.sp
-Brian Young \fI\%bayoung@acm.org\fP
-.sp
-Dan Nuffer \fI\%nuffer@users.sourceforge.net\fP
-.sp
-Marcus Boerger \fI\%helly@users.sourceforge.net\fP
-.sp
-Hartmut Kaiser \fI\%hkaiser@users.sourceforge.net\fP
-.sp
-Emmanuel Mogenet \fI\%mgix@mgix.com\fP
+Originaly written by Peter Bumbulis in 1993;
+developed and maintained by Brain Young, Marcus Boerger, Dan Nuffer and Ulya Trofimovich.
+Below is a (more or less) full list of contributors retrieved from the Git history and mailing lists:
.sp
-Ulya Trofimovich \fI\%skvadrik@gmail.com\fP
+Abs62,
+asmwarrior,
+Ben Smith,
+Brian Young,
+CRCinAU,
+Dan Nuffer,
+Derick Rethans,
+Dimitri John Ledkov,
+Eldar Zakirov,
+Emmanuel Mogenet,
+Hartmut Kaiser,
+jcfp,
+Jean\-Claude Wippler,
+Jeff Trull,
+Jérôme Dumesnil,
+Jesse Buesking,
+joscherl,
+Julian Andres Klode,
+Marcus Boerger,
+Mike Gilbert,
+nuno\-lopes,
+Oleksii Taran,
+paulmcq,
+Paulo Custodio,
+Perry E. Metzger,
+philippschaefer,
+Ross Burton,
+Rui Maciel,
+Ryan Mast,
+Samuel006,
+Sergei Trofimovich,
+sirzooro,
+Tim Kelly,
+Ulya Trofimovich
.SH VERSION INFORMATION
.sp
-This manpage describes \fBre2c\fP version 1.0.1, package date 11 Aug 2017.
+This manpage describes \fBre2c\fP version 1.0.1, package date 25 Aug 2017.
.\" Generated by docutils manpage writer.
.
const char *help =
"\n"
" -? -h --help\n"
-" Show a short help screen:\n"
+" Show this help message.\n"
"\n"
" -b --bit-vectors\n"
-" Implies -s. Use bit vectors as well to try to coax better code\n"
-" out of the compiler. Most useful for specifications with more\n"
-" than a few keywords (e.g., for most programming languages).\n"
+" Optimize conditional jumps using bit masks. Implies -s.\n"
"\n"
" -c --conditions\n"
-" Used for (f)lex-like condition support.\n"
+" Enable support of Flex-like \"conditions\" which allow to specify\n"
+" multiple interrelated lexer blocks.\n"
"\n"
" -d --debug-output\n"
-" Creates a parser that dumps information about the current posi‐\n"
-" tion and the state the parser is in. This is useful for debug‐\n"
-" ging parser issues and states. If you use this switch, you need\n"
-" to define a YYDEBUG macro, which will be called like a function\n"
-" with two parameters: void YYDEBUG (int state, char current).\n"
-" The first parameter receives the state or -1 and the second\n"
-" parameter receives the input at the current cursor.\n"
+" Emit YYDEBUG calls in the generated code. YYDEBUG should be\n"
+" defined like a void function with two parameters: void YYDEBUG\n"
+" (int state, YYCTYPE symbol), The first parameter is lexer state\n"
+" or -1, the second parameter is current input symbol.\n"
"\n"
" -D --emit-dot\n"
-" Emit Graphviz dot data, which can then be processed with e.g.,\n"
-" dot -Tpng input.dot > output.png. Please note that scanners with\n"
-" many states may crash dot.\n"
+" Instead of generating code, generate DFA graph in DOT format\n"
+" (used by Graphviz). The output can be converted to PNG format\n"
+" with a command similar to dot -Tpng dfa.dot -odfa.png. Note\n"
+" that large graphs may crash Graphviz.\n"
"\n"
" -e --ecb\n"
-" Generate a parser that supports EBCDIC. The generated code can\n"
-" deal with any character up to 0xFF. In this mode, re2c assumes\n"
-" an input character size of 1 byte. This switch is incompatible\n"
-" with -w, -x, -u, and -8.\n"
+" Generate a parser that reads input in EBCDIC encoding. In this\n"
+" mode re2c assumes that character range is 0 -- 0xFF an character\n"
+" size is 1 byte.\n"
"\n"
" -f --storable-state\n"
-" Generate a scanner with support for storable state.\n"
+" Generate a scanner which can store its inner state. This is\n"
+" useful in push-model lexers which are stopped when there is not\n"
+" enough input data and resumed by an outer program when more data\n"
+" becomes available. In this mode users should additionally\n"
+" define YYGETSTATE () and YYSETSTATE (state) macros and variables\n"
+" yych, yyaccept and the state as part of the lexer state.\n"
"\n"
" -F --flex-syntax\n"
-" Partial support for flex syntax. When this flag is active, named\n"
-" definitions must be surrounded by curly braces and can be\n"
-" defined without an equal sign and the terminating semicolon.\n"
-" Instead, names are treated as direct double quoted strings.\n"
+" Partial support for Flex syntax: in this mode named definitions\n"
+" don't need the equal sign and the terminating semicolon, and\n"
+" they must be surrounded by curly braces when used. Names with‐\n"
+" out curly braces are treated as double quoted strings.\n"
"\n"
" -g --computed-gotos\n"
-" Generate a scanner that utilizes GCC's computed-goto feature.\n"
-" That is, re2c generates jump tables whenever a decision is of\n"
-" certain complexity (e.g., a lot of if conditions would be other‐\n"
-" wise necessary). This is only usable with compilers that support\n"
-" this feature. Note that this implies -b and that the complexity\n"
-" threshold can be configured using the cgoto:threshold inplace\n"
-" configuration.\n"
+" Optimize conditional jumps using non-standard \"computed goto\"\n"
+" extension (must be supported by C/C++ compiler). re2c generates\n"
+" jump tables only in complex cases with a lot of conditional\n"
+" branches. Complexity threshold can be configured with\n"
+" cgoto:threshold configuration. This option implies -b.\n"
"\n"
" -i --no-debug-info\n"
-" Do not output #line information. This is useful when you want\n"
-" use a CMS tool with re2c's output. You might want to do this if\n"
-" you do not want to impose re2c as a build requirement for your\n"
-" source.\n"
+" Do not output #line information. This is useful when the gener‐\n"
+" ated code is tracked by some version control system.\n"
"\n"
" -o OUTPUT --output=OUTPUT\n"
" Specify the OUTPUT file.\n"
"\n"
" -r --reusable\n"
-" Allows reuse of scanner definitions with /*!use:re2c */ after\n"
-" /*!rules:re2c */. In this mode, no /*!re2c */ block and exactly\n"
-" one /*!rules:re2c */ must be present. The rules are saved and\n"
-" used by every /*!use:re2c */ block that follows. These blocks\n"
-" can contain inplace configurations, especially re2c:flags:e,\n"
-" re2c:flags:w, re2c:flags:x, re2c:flags:u, and re2c:flags:8.\n"
-" That way it is possible to create the same scanner multiple\n"
-" times for different character types, different input mechanisms,\n"
-" or different output mechanisms. The /*!use:re2c */ blocks can\n"
-" also contain additional rules that will be appended to the set\n"
-" of rules in /*!rules:re2c */.\n"
+" Allows reuse of re2c rules with /*!rules:re2c */ and /*!use:re2c\n"
+" */ blocks. In this mode simple /*!re2c */ blocks are not\n"
+" allowed and exactly one /*!rules:re2c */ block must be present.\n"
+" The rules are saved and used by every /*!use:re2c */ block that\n"
+" follows (which may add rules of their own). This option allows\n"
+" to reuse the same set of rules with different configurations.\n"
"\n"
" -s --nested-ifs\n"
-" Generate nested ifs for some switches. Many compilers need this\n"
-" assist to generate better code.\n"
+" Use nested if statements instead of switch statements in condi‐\n"
+" tional jumps. Non-optimizing C/C++ compilers generally generate\n"
+" better code with this option.\n"
"\n"
" -t HEADER --type-header=HEADER\n"
-" Create a HEADER file that contains types for the (f)lex-like\n"
-" condition support. This can only be activated when -c is in use.\n"
+" Generate a HEADER file that contains enum with condition names.\n"
+" Requires -c option.\n"
"\n"
" -T --tags\n"
" Enable submatch extraction with tags.\n"
" Enable submatch extraction with POSIX-style capturing groups.\n"
"\n"
" -u --unicode\n"
-" Generate a parser that supports UTF-32. The generated code can\n"
-" deal with any valid Unicode character up to 0x10FFFF. In this\n"
-" mode, re2c assumes an input character size of 4 bytes. This\n"
-" switch is incompatible with -e, -w, -x, and -8. This implies -s.\n"
+" Generate a parser that reads input in UTF-32 encoding. re2c\n"
+" assumes that character range is 0 -- 0x10FFFF and character size\n"
+" is 4 bytes. Implies -s.\n"
"\n"
" -v --version\n"
" Show version information.\n"
"\n"
" -V --vernum\n"
-" Show the version as a number in the MMmmpp (Majorm, minor,\n"
-" patch) format.\n"
+" Show version information in MMmmpp format (major, minor, patch).\n"
"\n"
" -w --wide-chars\n"
-" Generate a parser that supports UCS-2. The generated code can\n"
-" deal with any valid Unicode character up to 0xFFFF. In this\n"
-" mode, re2c assumes an input character size of 2 bytes. This\n"
-" switch is incompatible with -e, -x, -u, and -8. This implies -s.\n"
+" Generate a parser that reads input in UCS-2 encoding. re2c\n"
+" assumes that character range is 0 -- 0xFFFF and character size\n"
+" is 2 bytes. Implies -s.\n"
"\n"
" -x --utf-16\n"
-" Generate a parser that supports UTF-16. The generated code can\n"
-" deal with any valid Unicode character up to 0x10FFFF. In this\n"
-" mode, re2c assumes an input character size of 2 bytes. This\n"
-" switch is incompatible with -e, -w, -u, and -8. This implies -s.\n"
+" Generate a parser that reads input in UTF-16 encoding. re2c\n"
+" assumes that character range is 0 -- 0x10FFFF and character size\n"
+" is 2 bytes. Implies -s.\n"
"\n"
" -8 --utf-8\n"
-" Generate a parser that supports UTF-8. The generated code can\n"
-" deal with any valid Unicode character up to 0x10FFFF. In this\n"
-" mode, re2c assumes an input character size of 1 byte. This\n"
-" switch is incompatible with -e, -w, -x, and -u.\n"
+" Generate a parser that reads input in UTF-8 encoding. re2c\n"
+" assumes that character range is 0 -- 0x10FFFF and character size\n"
+" is 1 byte.\n"
"\n"
" --case-insensitive\n"
-" Makes all strings case insensitive. This makes \"-quoted expres‐\n"
-" sions behave as '-quoted expressions.\n"
+" Treat single-quoted and double-quoted strings as case-insensi‐\n"
+" tive.\n"
"\n"
" --case-inverted\n"
-" Invert the meaning of single and double quoted strings. With\n"
-" this switch, single quotes are case sensitive and double quotes\n"
-" are case insensitive.\n"
+" Invert the meaning of single-quoted and double-quoted strings:\n"
+" treat single-quoted strings as case-sensitive and double-quoted\n"
+" strings as case-insensitive.\n"
"\n"
" --no-generation-date\n"
" Suppress date output in the generated file.\n"
" with --tags or --posix-captures options.\n"
"\n"
" --no-optimize-tags\n"
-" Suppress optimization of tag variables (mostly used for debug‐\n"
-" ging).\n"
+" Suppress optimization of tag variables (useful for debugging or\n"
+" benchmarking).\n"
"\n"
" --no-version\n"
" Suppress version output in the generated file.\n"
" Suppress version output in the generated file.\n"
"\n"
" --encoding-policy POLICY\n"
-" Specify how re2c must treat Unicode surrogates. POLICY can be\n"
+" Define the way re2c treats Unicode surrogates. POLICY can be\n"
" one of the following: fail (abort with an error when a surrogate\n"
" is encountered), substitute (silently replace surrogates with\n"
-" the error code point 0xFFFD), ignore (treat surrogates as normal\n"
-" code points). By default, re2c ignores surrogates (for backward\n"
-" compatibility). The Unicode standard says that standalone surro‐\n"
-" gates are invalid code points, but different libraries and pro‐\n"
-" grams treat them differently.\n"
+" the error code point 0xFFFD), ignore (default, treat surrogates\n"
+" as normal code points). The Unicode standard says that stand‐\n"
+" alone surrogates are invalid, but real-world libraries and pro‐\n"
+" grams behave in different ways.\n"
"\n"
" --input INPUT\n"
-" Specify re2c's input API. INPUT can be either default or custom.\n"
+" Specify re2c input API. INPUT can be either default or custom\n"
+" (enables the use of generic API).\n"
"\n"
" -S --skeleton\n"
-" Instead of embedding re2c-generated code into C/C++ source, gen‐\n"
-" erate a self-contained program for the same DFA. Most useful for\n"
-" correctness and performance testing.\n"
+" Ignore user-defined interface code and generate a self-contained\n"
+" \"skeleton\" program. Additionally, generate input files with\n"
+" strings derived from the regular grammar and compressed match\n"
+" results that are used to verify \"skeleton\" behavior on all\n"
+" inputs. This option is useful for finding bugs in optimizations\n"
+" and code generation.\n"
"\n"
" --empty-class POLICY\n"
-" What to do if the user uses an empty character class. POLICY can\n"
-" be one of the following: match-empty (match empty input: pretty\n"
-" illogical, but this is the default for backwards compatibility\n"
-" reasons), match-none (fail to match on any input), error (compi‐\n"
-" lation error). Note that there are various ways to construct an\n"
-" empty class, e.g., [], [^\\x00-\\xFF], [\\x00-\\xFF][\\x00-\\xFF].\n"
-"\n"
-" --dfa-minimization <table | moore>\n"
-" The internal algorithm used by re2c to minimize the DFA\n"
-" (defaults to moore). Both the table filling algorithm and the\n"
-" Moore algorithm should produce the same DFA (up to states rela‐\n"
-" beling). The table filling algorithm is much simpler and\n"
-" slower; it serves as a reference implementation.\n"
+" Define the way re2c treats empty character classes. POLICY can\n"
+" be one of the following: match-empty (match empty input: illogi‐\n"
+" cal, but default behavior for backwards compatibility reasons),\n"
+" match-none (fail to match on any input), error (compilation\n"
+" error).\n"
+"\n"
+" --dfa-minimization ALGORITHM\n"
+" The internal algorithm used by re2c to minimize the DFA. ALGO‐\n"
+" RITHM can be either moore (Moore algorithm, the default) or ta‐\n"
+" ble (table filling algorithm). Both algorithms should produce\n"
+" the same DFA up to states relabeling; table filling is much\n"
+" slower and serves as a reference implementation.\n"
"\n"
" --eager-skip\n"
-" This option controls when the generated lexer advances to the\n"
-" next input symbol (that is, increments YYCURSOR or invokes\n"
-" YYSKIP). By default this happens after transition to the next\n"
-" state, but --eager-skip option allows to override default behav‐\n"
-" ior and advance input position immediately after reading input\n"
-" symbol. This option is implied by --no-lookahead.\n"
+" Make the generated lexer advance the input position \"eagerly\":\n"
+" immediately after reading input symbol. By default this happens\n"
+" after transition to the next state. Implied by --no-lookahead.\n"
"\n"
" --dump-nfa\n"
-" Generate .dot representation of NFA and dump it on stderr.\n"
+" Generate representation of NFA in DOT format and dump it on\n"
+" stderr.\n"
"\n"
" --dump-dfa-raw\n"
-" Generate .dot representation of DFA under construction and dump\n"
-" it on stderr.\n"
+" Generate representation of DFA in DOT format under construction\n"
+" and dump it on stderr.\n"
"\n"
" --dump-dfa-det\n"
-" Generate .dot representation of DFA immediately after deter‐\n"
-" minization and dump it on stderr.\n"
+" Generate representation of DFA in DOT format immediately after\n"
+" determinization and dump it on stderr.\n"
"\n"
" --dump-dfa-tagopt\n"
-" Generate .dot representation of DFA after tag optimizations and\n"
-" dump it on stderr.\n"
+" Generate representation of DFA in DOT format after tag optimiza‐\n"
+" tions and dump it on stderr.\n"
"\n"
" --dump-dfa-min\n"
-" Generate .dot representation of DFA after minimization and dump\n"
-" it on stderr.\n"
+" Generate representation of DFA in DOT format after minimization\n"
+" and dump it on stderr.\n"
"\n"
" --dump-adfa\n"
-" Generate .dot representation of DFA after tunneling and dump it\n"
-" on stderr.\n"
+" Generate representation of DFA in DOT format after tunneling and\n"
+" dump it on stderr.\n"
"\n"
" -1 --single-pass\n"
" Deprecated. Does nothing (single pass is the default now).\n"
INTERFACE CODE
--------------
-The user must supply interface code either in the form of C/C++ code
-(macros, functions, variables, etc.) or in the form of ``INPLACE CONFIGURATIONS``.
-Which symbols must be defined and which are optional
-depends on the particular use case.
-
.. include:: @top_srcdir@/doc/manual/syntax/interface.rst_
SYNTAX
------
-Code for ``re2c`` consists of a set of ``RULES``, ``NAMED DEFINITIONS``, ``CODE`` and
-``INPLACE CONFIGURATIONS``.
+A program can contain any number of ``re2c`` blocks.
+Each block consists of a sequence of ``RULES``, ``NAMED DEFINITIONS`` and ``INPLACE CONFIGURATIONS``.
RULES
~~~~~
-
-Each rule consist of a regular expression (see ``REGULAR EXPRESSIONS``) accompanied with a block of C/C++ code
-which is to be executed when the associated regular expression is
-matched. You can either start the code with an opening curly brace or
-the sequence ``:=``. If you use an opening curly brace, ``re2c`` will count brace depth
-and stop looking for code automatically. Otherwise, curly braces are not
-allowed and ``re2c`` stops looking for code at the first line that does
-not begin with whitespace. If two or more rules overlap, the first rule
-is preferred.
-
.. include:: @top_srcdir@/doc/manual/syntax/rules.rst_
NAMED DEFINITIONS
~~~~~~~~~~~~~~~~~
-Named definitions are of the form:
-
- ``name = regular-expression;``
-
-If ``-F`` is active, then named definitions are also of the form:
-
- ``name { regular-expression }``
+.. include:: @top_srcdir@/doc/manual/syntax/named_definitions.rst_
.. include:: @top_srcdir@/doc/manual/features/submatch/submatch.rst_
-SCANNER WITH STORABLE STATES
-----------------------------
+STORABLE STATE
+--------------
.. include:: @top_srcdir@/doc/manual/features/state/state.rst_
-SCANNER WITH CONDITION SUPPORT
-------------------------------
+CONDITIONS
+----------
.. include:: @top_srcdir@/doc/manual/features/conditions/conditions.rst_
.. include:: @top_srcdir@/doc/manual/features/encodings/encodings.rst_
-GENERIC INPUT API
------------------
+GENERIC API
+-----------
.. include:: @top_srcdir@/doc/manual/features/generic_api/generic_api.rst_
-A couple of useful links that provide some examples:
-
-1. http://skvadrik.github.io/aleph_null/posts/re2c/2015-01-13-input_model.html
-2. http://skvadrik.github.io/aleph_null/posts/re2c/2015-01-15-input_model_custom.html
-
-
SEE ALSO
--------
See also: flex(1), lex(1), quex (http://quex.sourceforge.net).
-
AUTHORS
-------
-Peter Bumbulis peter@csg.uwaterloo.ca
-
-Brian Young bayoung@acm.org
-
-Dan Nuffer nuffer@users.sourceforge.net
-
-Marcus Boerger helly@users.sourceforge.net
-
-Hartmut Kaiser hkaiser@users.sourceforge.net
-
-Emmanuel Mogenet mgix@mgix.com
-
-Ulya Trofimovich skvadrik@gmail.com
+Originaly written by Peter Bumbulis in 1993;
+developed and maintained by Brain Young, Marcus Boerger, Dan Nuffer and Ulya Trofimovich.
+Below is a (more or less) full list of contributors retrieved from the Git history and mailing lists:
+.. include:: @top_srcdir@/doc/manual/contributors.rst_
VERSION INFORMATION
--- /dev/null
+Abs62,
+asmwarrior,
+Ben Smith,
+Brian Young,
+CRCinAU,
+Dan Nuffer,
+Derick Rethans,
+Dimitri John Ledkov,
+Eldar Zakirov,
+Emmanuel Mogenet,
+Hartmut Kaiser,
+jcfp,
+Jean-Claude Wippler,
+Jeff Trull,
+Jérôme Dumesnil,
+Jesse Buesking,
+joscherl,
+Julian Andres Klode,
+Marcus Boerger,
+Mike Gilbert,
+nuno-lopes,
+Oleksii Taran,
+paulmcq,
+Paulo Custodio,
+Perry E. Metzger,
+philippschaefer,
+Ross Burton,
+Rui Maciel,
+Ryan Mast,
+Samuel006,
+Sergei Trofimovich,
+sirzooro,
+Tim Kelly,
+Ulya Trofimovich
-You can precede regular expressions with a list of condition names when
-using the ``-c`` switch. ``re2c`` will then generate a scanner block for
-each condition, and each of the generated blocks will have its own
-precondition. The precondition is given by the interface define
-``YYGETCONDITON()`` and must be of type ``YYCONDTYPE``.
-
-There are two special rule types. First, the rules of the condition ``<*>``
-are merged to all conditions (note that they have a lower priority than
-other rules of that condition). And second, the empty condition list
-allows to provide a code block that does not have a scanner part,
-meaning it does not allow any regular expressions. The condition value
-referring to this special block is always the one with the enumeration
-value 0. This way the code of this special rule can be used to
-initialize a scanner. It is in no way necessary to have these rules: but
-sometimes it is helpful to have a dedicated uninitialized condition
-state.
-
-Non empty rules allow to specify the new condition, which makes them
-transition rules. Besides generating calls for the
-``YYSETCONDTITION`` define, no other special code is generated.
-
-There is another kind of special rule that allows to prepend code to any
-code block of all rules of a certain set of conditions or to all code
-blocks of all rules. This can be helpful when some operation is common
-among rules. For instance, this can be used to store the length of the
-scanned string. These special setup rules start with an exclamation mark
-followed by either a list of conditions ``<! condition, ... >`` or a star
-``<!*>``. When ``re2c`` generates the code for a rule whose state does not have a
-setup rule and a starred setup rule is present, the starred setup code will be
-used as setup code.
+*Conditions* are enabled with ``-c`` ``--conditions``.
+This option allows to encode multiple interrelated lexers within the same re2c block.
+
+Each lexer corresponds to a single *condition*.
+It starts with a label of the form ``yyc_name``,
+where ``name`` is *condition* name
+and ``yyc`` prefix can be adjusted with configuration ``re2c:condprefix``.
+Different lexers are separated with a comment ``/* *********************************** */``
+which can be adjusted with configuration ``re2c:cond:divider``.
+
+Furthermore, each *condition* has a unique identifier of the form ``yycname``,
+where ``name`` is condition name
+and ``yyc`` prefix can be adjusted with configuration ``re2c:condenumprefix``.
+Identifiers have the type ``YYCONDTYPE`` and should be generated with ``/*!types:re2c*/`` directive or ``-t`` ``--type-header`` option.
+Users shouldn't define these identifiers manually, as the order of *conditions* is not specified.
+
+Before all *conditions* re2c generates entry code that checks the current *condition* identifier
+and transfers control flow to the start label of the active *condition*.
+After matching some rule of this *condition*,
+lexer may either transfer control flow back to the entry code (after executing the associated action and optionally setting another *condition* with ``=>``),
+or use ``:=>`` shortcut and transition directly to the start label of another *condition* (skipping the action and the entry code).
+Configuration ``re2c:cond:goto`` allows to change the default behavior.
+
+Syntactically each rule must be preceded with a list of comma-separated *condition* names or a wildcard ``*``
+enclosed in angle brackets ``<`` and ``>``.
+Wildcard means "any condition" and is semantically equivalent to listing all condition names.
+Here ``regexp`` is a regular expression, ``default`` refers to the *default rule* ``*``,
+and ``action`` is a block of C/C++ code.
+
+* ``<conditions-or-wildcard> regexp-or-default action``
+
+* ``<conditions-or-wildcard> regexp-or-default => condition action``
+
+* ``<conditions-or-wildcard> regexp-or-default :=> condition``
+
+Rules with an exclamation mark ``!`` in front of condition list have a special meaning:
+they have no regular expression,
+and the associated action is merged as an entry code to actions of normal rules.
+This might be a convenient place to peform a routine task that is common to all rules.
+
+* ``<!conditions-or-wildcard> action``
+
+Another special form of rules with an empty condition list ``<>`` and no regular expression
+allows to specify an "entry condition" that can be used to execute code before entering the lexer.
+It is semantically equivalent to a condition with number zero, name ``0`` and an empty regular expression.
+
+* ``<> action``
+
+* ``<> => condition action``
+
+* ``<> :=> condition``
-``re2c`` usually operates on input with pointer-like primitives
+By default ``re2c`` operates on input using pointer-like primitives
``YYCURSOR``, ``YYMARKER``, ``YYCTXMARKER``, and ``YYLIMIT``.
+Normally pointer-like primitives are defined as variables of type ``YYCTYPE*``,
+but it is possible to use STL iterators or any other abstraction as long as it syntactically fits into the following use cases:
-The generic input API (enabled with the ``--input custom`` switch) allows
-customizing input operations. In this mode, ``re2c`` will express all
-operations on input in terms of the following primitives:
-
- +----------------------+-----------------------------------------------------------+
- | ``YYPEEK ()`` | get current input character |
- +----------------------+-----------------------------------------------------------+
- | ``YYSKIP ()`` | advance to next character |
- +----------------------+-----------------------------------------------------------+
- | ``YYBACKUP ()`` | backup current input position |
- +----------------------+-----------------------------------------------------------+
- | ``YYBACKUPCTX ()`` | backup current input position for trailing context |
- +----------------------+-----------------------------------------------------------+
- | ``YYSTAGP (t)`` | save current input position to tag ``t`` |
- +----------------------+-----------------------------------------------------------+
- | ``YYSTAGN (t)`` | save default value to tag ``t`` |
- +----------------------+-----------------------------------------------------------+
- | ``YYMTAGP (t)`` | append input position to the history of tag ``t`` |
- +----------------------+-----------------------------------------------------------+
- | ``YYMTAGN (t)`` | append default value to the history of tag ``t`` |
- +----------------------+-----------------------------------------------------------+
- | ``YYRESTORE ()`` | restore current input position |
- +----------------------+-----------------------------------------------------------+
- | ``YYRESTORECTX ()`` | restore current input position for trailing context |
- +----------------------+-----------------------------------------------------------+
- | ``YYRESTORETAG (t)`` | restore current input position from tag ``t`` |
- +----------------------+-----------------------------------------------------------+
- | ``YYLESSTHAN (n)`` | check if less than ``n`` input characters are left |
- +----------------------+-----------------------------------------------------------+
+* ``++YYCURSOR;``
+* ``yych = *YYCURSOR;``
+* ``yych = *++YYCURSOR;``
+* ``yych = *(YYMARKER = YYCURSOR);``
+* ``yych = *(YYMARKER = ++YCURSOR);``
+* ``YYMARKER = YYCURSOR;``
+* ``YYMARKER = ++YYCURSOR;``
+* ``YYCURSOR = YYMARKER;``
+* ``YYCTXMARKER = YYCURSOR + 1;``
+* ``YYCURSOR = YYCTXMARKER;``
+* ``if (YYLIMIT <= YYCURSOR) ...``
+* ``if ((YYLIMIT - YYCURSOR) < n) ...``
+* ``YYDEBUG (label, *YYCURSOR);``
+
+
+If this input model is too restrictive, then it is possible to use generic input API enabled with ``--input custom`` option.
+In this mode all input operations are expressed in terms of the primitives below.
+These primitives can be defined in any suitable way; one doesn't have to stick to the pointer semantics.
+For example, it is possible to read input directly from file without any buffering,
+or to disable ``YYFILL`` mechanism and perform end-of-input checking on each input character from inside of ``YYPEEK`` or ``YYSKIP``.
+
+* ``YYPEEK ()``
+* ``YYSKIP ()``
+* ``YYBACKUP ()``
+* ``YYBACKUPCTX ()``
+* ``YYSTAGP (t)``
+* ``YYSTAGN (t)``
+* ``YYMTAGP (t)``
+* ``YYMTAGN (t)``
+* ``YYRESTORE ()``
+* ``YYRESTORECTX ()``
+* ``YYRESTORETAG (t)``
+* ``YYLESSTHAN (n)``
+
+Default input model can be expressed in terms of generic API as follows
+(except for ``YMTAGP`` and ``YYMTAGN``, which have no default implementation):
+
+* ``#define YYPEEK () *YYCURSOR``
+* ``#define YYSKIP () ++YYCURSOR``
+* ``#define YYBACKUP () YYMARKER = YYCURSOR``
+* ``#define YYBACKUPCTX () YYCTXMARKER = YYCURSOR``
+* ``#define YYRESTORE () YYCURSOR = YYMARKER``
+* ``#define YYRESTORECTX () YYCURSOR = YYCTXMARKER``
+* ``#define YYRESTORERAG (t) YYCURSOR = t``
+* ``#define YYLESSTHAN (n) YYLIMIT - YYCURSOR < n``
+* ``#define YYSTAGP (t) t = YYCURSOR``
+* ``#define YYSTAGN (t) t = NULL``
-When the ``-f`` flag is specified, ``re2c`` generates a scanner that can
-store its current state, return to its caller, and later resume
-operations exactly where it left off.
+With ``-f`` ``--storable-state`` option re2c generates a lexer that can
+store its current state, return to the caller, and later resume operations exactly where it left off.
+The default mode of operation in re2c is a "pull" model, where the lexer "pulls" more input whenever it needs it.
+However, this mode of operation assumes that the lexer is the owner of the parsing loop, and that may not always be convenient.
-The default mode of operation in ``re2c`` is a
-"pull" model, where the scanner asks for extra input whenever it needs it. However, this mode of operation assumes that the scanner is the "owner" of the parsing loop, and that may not always be convenient.
+Storable state is useful exactly for situations like that: it allows to construct
+lexers that work in a "push" model, where data is fed to the lexer chunk by chunk.
+When the lexer needs more input, it stores its state and returns to the caller.
+Later, when more input becomes available, it resumes operations exactly where it stopped.
-Typically, if there is a preprocessor ahead of the scanner in the
-stream, or for that matter, any other procedural source of data, the
-scanner cannot "ask" for more data unless both the scanner and the source
-live in separate threads.
+Changes needed compared to the "pull" model:
-The ``-f`` flag is useful exactly for situations like that: it lets users design
-scanners that work in a "push" model, i.e., a model where data is fed to the
-scanner chunk by chunk. When the scanner runs out of data to consume, it
-stores its state and returns to the caller. When more input data is
-fed to the scanner, it resumes operations exactly where it left off.
+* Define ``YYSETSTATE ()`` and ``YYGETSTATE (state)``.
-Changes needed compared to the "pull" model:
+* Define ``yych``, ``yyaccept`` and ``state`` variables as a part of persistent lexer state.
+ ``state`` should be initialized to ``-1``.
+
+* ``YYFILL`` should return to the outer program instead of trying to supply more input.
+ Return code should indicate that lexer needs more input.
+
+* The outer program should recognize situations when lexer needs more input
+ and respond appropriately.
+
+* Use ``/*!getstate:re2c*/`` directive if it is necessary to execute any code
+ before entering the lexer.
+
+* Use configurations ``state:abort`` and ``state:nextlabel`` to tweak the generated code.
-* The user has to supply macros named ``YYSETSTATE ()`` and ``YYGETSTATE (state)``.
-
-* The ``-f`` option inhibits declaration of ``yych`` and ``yyaccept``, so the
- user has to declare them and save and restore them where required.
- In the ``examples/push_model/push.re`` example, these are declared as
- fields of a (C++) class of which the scanner is a method, so they do
- not need to be saved/restored explicitly. For C, they could, e.g., be made
- macros that select fields from a structure passed in as a parameter.
- Alternatively, they could be declared as local variables, saved with
- ``YYFILL (n)`` when it decides to return and restored upon entering the
- function. Also, it could be more efficient to save the state from
- ``YYFILL (n)`` because ``YYSETSTATE (state)`` is called unconditionally.
- ``YYFILL (n)`` however does not get ``state`` as a parameter, so we would have
- to store state in a local variable by ``YYSETSTATE (state)``.
-
-* Modify ``YYFILL (n)`` to return (from the function calling it) if more input is needed.
-
-* Modify the caller to recognize if more input is needed and respond appropriately.
-
-* The generated code will contain a switch block that is used to
- restore the last state by jumping behind the corresponding ``YYFILL (n)``
- call. This code is automatically generated in the epilogue of the first ``/*!re2c */``
- block. It is possible to trigger generation of the ``YYGETSTATE ()``
- block earlier by placing a ``/*!getstate:re2c*/`` comment. This is especially useful when the scanner code should be
- wrapped inside a loop.
-
-Please see ``examples/push_model/push.re`` for an example of a "push" model scanner. The
-generated code can be tweaked with inplace configurations ``state:abort``
-and ``state:nextlabel``.
``-? -h --help``
- Show a short help screen:
+ Show help message.
``-b --bit-vectors``
- Implies ``-s``. Use bit vectors as well to try to
- coax better code out of the compiler. Most useful for
- specifications with more than a few keywords (e.g., for most programming
- languages).
+ Optimize conditional jumps using bit masks. Implies ``-s``.
``-c --conditions``
- Used for (f)lex-like condition support.
+ Enable support of Flex-like "conditions": multiple interrelated lexers within one block.
``-d --debug-output``
- Creates a parser that dumps information about
- the current position and the state the parser is in.
- This is useful for debugging parser issues and states. If you use this
- switch, you need to define a ``YYDEBUG`` macro, which will be called like a
- function with two parameters: ``void YYDEBUG (int state, char current)``.
- The first parameter receives the state or ``-1`` and the second parameter
- receives the input at the current cursor.
+ Emit ``YYDEBUG`` in the generated code.
+ ``YYDEBUG`` should be defined by the user in the form of a void function with two parameters:
+ ``state`` (lexer state or -1) and ``symbol`` (current input symbol of type ``YYCTYPE``).
``-D --emit-dot``
- Emit Graphviz dot data, which can then be processed
- with e.g., ``dot -Tpng input.dot > output.png``. Please note that
- scanners with many states may crash dot.
+ Instead of normal output generate lexer graph in DOT format.
+ The output can be converted to PNG with the help of Graphviz (something like ``dot -Tpng -odfa.png dfa.dot``).
+ Note that large graphs may crash Graphviz.
``-e --ecb``
- Generate a parser that supports EBCDIC. The generated
- code can deal with any character up to 0xFF. In this mode, ``re2c`` assumes
- an input character size of 1 byte. This switch is incompatible with
- ``-w``, ``-x``, ``-u``, and ``-8``.
+ Generate a lexer that reads input in EBCDIC encoding.
+ ``re2c`` assumes that character range is 0 -- 0xFF an character size is 1 byte.
``-f --storable-state``
- Generate a scanner with support for storable state.
+ Generate a lexer which can store its inner state.
+ This is useful in push-model lexers which are stopped by an outer program when there is not enough input,
+ and then resumed when more input becomes available.
+ In this mode users should additionally define
+ ``YYGETSTATE ()`` and ``YYSETSTATE (state)`` macros
+ and variables ``yych``, ``yyaccept`` and the ``state`` as part of the lexer state.
``-F --flex-syntax``
- Partial support for flex syntax. When this flag
- is active, named definitions must be surrounded by curly braces and
- can be defined without an equal sign and the terminating semicolon.
- Instead, names are treated as direct double quoted strings.
+ Partial support for Flex syntax:
+ in this mode named definitions don't need the equal sign and the terminating semicolon,
+ and when used they must be surrounded by curly braces.
+ Names without curly braces are treated as double-quoted strings.
``-g --computed-gotos``
- Generate a scanner that utilizes GCC's
- computed-goto feature. That is, ``re2c`` generates jump tables whenever a
- decision is of certain complexity (e.g., a lot of if conditions would be
- otherwise necessary). This is only usable with compilers that support this feature.
- Note that this implies ``-b`` and that the complexity threshold can be configured
- using the ``cgoto:threshold`` inplace configuration.
+ Optimize conditional jumps using non-standard "computed goto" extension (must be supported by C/C++ compiler).
+ ``re2c`` generates jump tables only in complex cases with a lot of conditional branches.
+ Complexity threshold can be configured with ``cgoto:threshold`` configuration.
+ This option implies ``-b``.
``-i --no-debug-info``
- Do not output ``#line`` information. This is
- useful when you want use a CMS tool with ``re2c``'s output. You might
- want to do this if you do not want to impose re2c as a build requirement
- for your source.
+ Do not output ``#line`` information.
+ This is useful when the generated code is tracked by some version control system.
``-o OUTPUT --output=OUTPUT``
Specify the ``OUTPUT`` file.
``-r --reusable``
- Allows reuse of scanner definitions with ``/*!use:re2c */`` after ``/*!rules:re2c */``.
- In this mode, no ``/*!re2c */`` block and exactly one ``/*!rules:re2c */`` must be present.
- The rules are saved and used by every ``/*!use:re2c */`` block that follows.
- These blocks can contain inplace configurations, especially ``re2c:flags:e``,
- ``re2c:flags:w``, ``re2c:flags:x``, ``re2c:flags:u``, and ``re2c:flags:8``.
- That way it is possible to create the same scanner multiple times for
- different character types, different input mechanisms, or different output mechanisms.
- The ``/*!use:re2c */`` blocks can also contain additional rules that will be appended
- to the set of rules in ``/*!rules:re2c */``.
+ Allows reuse of ``re2c`` rules with ``/*!rules:re2c */`` and ``/*!use:re2c */`` blocks.
+ In this mode simple ``/*!re2c */`` blocks are not allowed
+ and exactly one ``/*!rules:re2c */`` block must be present.
+ The rules are saved and used by every ``/*!use:re2c */`` block that follows (which may add rules of their own).
+ This option allows to reuse the same set of rules with different configurations.
``-s --nested-ifs``
- Generate nested ifs for some switches. Many
- compilers need this assist to generate better code.
+ Use nested ``if`` statements instead of ``switch`` statements in conditional jumps.
+ This usually results in more efficient code with non-optimizing C/C++ compilers.
``-t HEADER --type-header=HEADER``
- Create a ``HEADER`` file that
- contains types for the (f)lex-like condition support. This can only be
- activated when ``-c`` is in use.
+ Generate a ``HEADER`` file that contains enum with condition names.
+ Requires ``-c`` option.
``-T --tags``
Enable submatch extraction with tags.
Enable submatch extraction with POSIX-style capturing groups.
``-u --unicode``
- Generate a parser that supports UTF-32. The generated
- code can deal with any valid Unicode character up to 0x10FFFF. In this
- mode, ``re2c`` assumes an input character size of 4 bytes. This switch is
- incompatible with ``-e``, ``-w``, ``-x``, and ``-8``. This implies ``-s``.
+ Generate a lexer that reads input in UTF-32 encoding.
+ ``re2c`` assumes that character range is 0 -- 0x10FFFF and character size is 4 bytes.
+ Implies ``-s``.
``-v --version``
Show version information.
``-V --vernum``
- Show the version as a number in the MMmmpp (Majorm, minor, patch) format.
+ Show version information in ``MMmmpp`` format (major, minor, patch).
``-w --wide-chars``
- Generate a parser that supports UCS-2. The
- generated code can deal with any valid Unicode character up to 0xFFFF.
- In this mode, ``re2c`` assumes an input character size of 2 bytes. This
- switch is incompatible with ``-e``, ``-x``, ``-u``, and ``-8``. This implies
- ``-s``.
+ Generate a lexer that reads input in UCS-2 encoding.
+ ``re2c`` assumes that character range is 0 -- 0xFFFF and character size is 2 bytes.
+ Implies ``-s``.
``-x --utf-16``
- Generate a parser that supports UTF-16. The generated
- code can deal with any valid Unicode character up to 0x10FFFF. In this
- mode, ``re2c`` assumes an input character size of 2 bytes. This switch is
- incompatible with ``-e``, ``-w``, ``-u``, and ``-8``. This implies ``-s``.
+ Generate a lexer that reads input in UTF-16 encoding.
+ ``re2c`` assumes that character range is 0 -- 0x10FFFF and character size is 2 bytes.
+ Implies ``-s``.
``-8 --utf-8``
- Generate a parser that supports UTF-8. The generated
- code can deal with any valid Unicode character up to 0x10FFFF. In this
- mode, ``re2c`` assumes an input character size of 1 byte. This switch is
- incompatible with ``-e``, ``-w``, ``-x``, and ``-u``.
+ Generate a lexer that reads input in UTF-8 encoding.
+ ``re2c`` assumes that character range is 0 -- 0x10FFFF and character size is 1 byte.
``--case-insensitive``
- Makes all strings case insensitive. This makes
- "-quoted expressions behave as '-quoted expressions.
+ Treat single-quoted and double-quoted strings as case-insensitive.
``--case-inverted``
- Invert the meaning of single and double quoted
- strings. With this switch, single quotes are case sensitive and double
- quotes are case insensitive.
+ Invert the meaning of single-quoted and double-quoted strings:
+ treat single-quoted strings as case-sensitive and double-quoted strings as case-insensitive.
``--no-generation-date``
Suppress date output in the generated file.
This option only has effect with ``--tags`` or ``--posix-captures`` options.
``--no-optimize-tags``
- Suppress optimization of tag variables (mostly used for debugging).
+ Suppress optimization of tag variables (useful for debugging or benchmarking).
``--no-version``
Suppress version output in the generated file.
-``--no-generation-date``
- Suppress version output in the generated file.
-
``--encoding-policy POLICY``
- Specify how ``re2c`` must treat Unicode
- surrogates. ``POLICY`` can be one of the following: ``fail`` (abort with
- an error when a surrogate is encountered), ``substitute`` (silently replace
- surrogates with the error code point 0xFFFD), ``ignore`` (treat surrogates as
- normal code points). By default, ``re2c`` ignores surrogates (for backward
- compatibility). The Unicode standard says that standalone surrogates are
- invalid code points, but different libraries and programs treat them
- differently.
+ Define the way ``re2c`` treats Unicode surrogates.
+ ``POLICY`` can be one of the following: ``fail`` (abort with an error when a surrogate is encountered),
+ ``substitute`` (silently replace surrogates with the error code point 0xFFFD),
+ ``ignore`` (default, treat surrogates as normal code points).
+ The Unicode standard says that standalone surrogates are invalid,
+ but real-world libraries and programs behave in different ways.
``--input INPUT``
- Specify re2c's input API. ``INPUT`` can be either ``default`` or ``custom``.
+ Specify ``re2c`` input API. ``INPUT`` can be either ``default`` or ``custom`` (enables the use of generic API).
``-S --skeleton``
- Instead of embedding re2c-generated code into C/C++
- source, generate a self-contained program for the same DFA. Most useful
- for correctness and performance testing.
+ Ignore user-defined interface code and generate a self-contained "skeleton" program.
+ Additionally, generate input files with strings derived from the regular grammar
+ and compressed match results that are used to verify "skeleton" behavior on all inputs.
+ This option is useful for finding bugs in optimizations and code generation.
``--empty-class POLICY``
- What to do if the user uses an empty character
- class. ``POLICY`` can be one of the following: ``match-empty`` (match empty
- input: pretty illogical, but this is the default for backwards
- compatibility reasons), ``match-none`` (fail to match on any input),
- ``error`` (compilation error). Note that there are various ways to
- construct an empty class, e.g., [], [^\\x00-\\xFF],
- [\\x00-\\xFF][\\x00-\\xFF].
-
-``--dfa-minimization <table | moore>``
- The internal algorithm used by re2c to minimize the DFA (defaults to ``moore``).
- Both the table filling algorithm and the Moore algorithm should produce the same DFA (up to states relabeling).
- The table filling algorithm is much simpler and slower; it serves as a reference implementation.
+ Define the way ``re2c`` treats empty character classes.
+ ``POLICY`` can be one of the following: ``match-empty`` (match empty input: illogical, but default behavior for backwards compatibility reasons),
+ ``match-none`` (fail to match on any input),
+ ``error`` (compilation error).
+
+``--dfa-minimization ALGORITHM``
+ The internal algorithm used by re2c to minimize the DFA.
+ ``ALGORITHM`` can be either ``moore`` (Moore algorithm, the default) or ``table`` (table filling algorithm).
+ Both algorithms should produce the same DFA up to states relabeling;
+ table filling is much slower and serves as a reference implementation.
``--eager-skip``
- This option controls when the generated lexer advances to the next input symbol
- (that is, increments YYCURSOR or invokes YYSKIP).
- By default this happens after transition to the next state,
- but ``--eager-skip`` option allows to override default behavior
- and advance input position immediately after reading input symbol.
- This option is implied by ``--no-lookahead``.
+ Make the generated lexer advance the input position "eagerly":
+ immediately after reading input symbol.
+ By default this happens after transition to the next state.
+ Implied by ``--no-lookahead``.
``--dump-nfa``
- Generate .dot representation of NFA and dump it on stderr.
+ Generate representation of NFA in DOT format and dump it on stderr.
``--dump-dfa-raw``
- Generate .dot representation of DFA under construction and dump it on stderr.
+ Generate representation of DFA in DOT format under construction and dump it on stderr.
``--dump-dfa-det``
- Generate .dot representation of DFA immediately after determinization and dump it on stderr.
+ Generate representation of DFA in DOT format immediately after determinization and dump it on stderr.
``--dump-dfa-tagopt``
- Generate .dot representation of DFA after tag optimizations and dump it on stderr.
+ Generate representation of DFA in DOT format after tag optimizations and dump it on stderr.
``--dump-dfa-min``
- Generate .dot representation of DFA after minimization and dump it on stderr.
+ Generate representation of DFA in DOT format after minimization and dump it on stderr.
``--dump-adfa``
- Generate .dot representation of DFA after tunneling and dump it on stderr.
+ Generate representation of DFA in DOT format after tunneling and dump it on stderr.
``-1 --single-pass``
Deprecated. Does nothing (single pass is the default now).
+Below is the list of all symbols which may be used by the lexer in order to interact with the outer world.
+These symbols should be defined by the user,
+either in the form of inplace configurations,
+or as C/C++ variables, functions, macros and other language constructs.
+Which primitives are necessary depends on the particular use case.
+
+``yyaccept``
+ L-value of unsigned integral type that is used to hold the number of the last matched rule.
+ Explicit definition by the user is necessary only with ``-f`` ``--storable-state`` option.
``YYBACKUP ()``
- Backup current input position (used only with generic API).
+ Backup current input position (used only with ``--input custom`` option).
``YYBACKUPCTX ()``
- Backup current input position for trailing context (used only with generic API).
+ Backup current input position for trailing context (used only with ``--input custom`` option).
+
+``yych``
+ L-value of type ``YYCTYPE`` that is used to hold current input character.
+ Explicit definition by the user is necessary only with ``-f`` ``--storable-state`` option.
``YYCONDTYPE``
- In ``-c`` mode, you can use ``-t`` to generate a file that
- contains the enumeration used as conditions. Each of the values refers
- to a condition of a rule set.
+ The type of *condition* identifiers (used only with ``-c`` ``--conditions`` option).
+ Should be generated either with ``/*!types:re2c*/`` directive, or with ``-t`` ``--type-header`` option.
``YYCTXMARKER``
- l-value of type ``YYCTYPE *``.
- The generated code saves trailing context backtracking information in
- ``YYCTXMARKER``. The user only needs to define this macro if a scanner
- specification uses trailing context in one or more of its regular
- expressions.
+ L-value of type ``YYCTYPE *`` that is used to backup input position of trailing context.
+ It is needed only if regular expressions use the lookahead operator ``/``.
``YYCTYPE``
- Type used to hold an input symbol (code unit). Usually
- ``char`` or ``unsigned char`` for ASCII, EBCDIC or UTF-8, or *unsigned short*
- for UTF-16 or UCS-2, or ``unsigned int`` for UTF-32.
+ The type of the input characters (*code units*).
+ Usually it should be ``unsigned char`` for ASCII, EBCDIC and UTF-8 encodings,
+ ``unsigned short`` for UTF-16 or UCS-2 encodings,
+ and ``unsigned int`` for UTF-32 encoding.
``YYCURSOR``
- l-value of type ``YYCTYPE *`` that points to the current input symbol. The generated code advances
- ``YYCURSOR`` as symbols are matched. On entry, ``YYCURSOR`` is assumed to
- point to the first character of the current token. On exit, ``YYCURSOR``
- will point to the first character of the following token.
-
-``YYDEBUG (state, current)``
- This is only needed if the ``-d`` flag was
- specified. It allows easy debugging of the generated parser by calling a
- user defined function for every state. The function should have the
- following signature: ``void YYDEBUG (int state, char current)``. The first
- parameter receives the state or -1 and the second parameter receives the
- input at the current cursor.
+ L-value of type ``YYCTYPE *`` that is used as a pointer to the current input symbol.
+ Initially ``YYCURSOR`` points to the first character and is advanced by the lexer during matching.
+ When a rule matches, ``YYCURSOR`` points past the last character of the matched string.
+
+``YYDEBUG (state, symbol)``
+ A function-like primitive that is used to dump debug information (only used with ``-d`` ``--debug-output`` option).
+ ``YYDEBUG`` should return no value and accept two arguments:
+ ``state`` (either lexer state or ``-1``) and ``symbol`` (current input symbol).
``YYFILL (n)``
- The generated code "calls"" ``YYFILL (n)`` when the
- buffer needs (re)filling: at least ``n`` additional characters should be
- provided. ``YYFILL (n)`` should adjust ``YYCURSOR``, ``YYLIMIT``, ``YYMARKER``,
- and ``YYCTXMARKER`` as needed. Note that for typical programming languages
- ``n`` will be the length of the longest keyword plus one. The user can
- place a comment of the form ``/*!max:re2c*/`` to insert a ``YYMAXFILL`` define set to the maximum
- length value.
+ A function-like primitive that is called by the lexer when there is not enough input.
+ ``YYFILL`` should return no value and supply at least ``n`` additional characters.
+ Maximal value of ``n`` equals ``YYMAXFILL``, which can be obtained with the ``/*!max:re2c*/`` directive.
``YYGETCONDITION ()``
- This define is used to get the condition prior to
- entering the scanner code when using the ``-c`` switch. The value must be
- initialized with a value from the ``YYCONDTYPE`` enumeration type.
+ R-value of type ``YYCONDTYPE`` that represents current *condition* identifier (used only with ``-c`` ``--conditions`` option).
``YYGETSTATE ()``
- The user only needs to define this macro if the ``-f``
- flag was specified. In that case, the generated code "calls"
- ``YYGETSTATE ()`` at the very beginning of the scanner in order to obtain
- the saved state. ``YYGETSTATE ()`` must return a signed integer. The value
- must be either -1, indicating that the scanner is entered for the first
- time, or a value previously saved by ``YYSETSTATE (s)``. In the second
- case, the scanner will resume operations right after where the last
- ``YYFILL (n)`` was called.
+ R-value of signed integral type that represents current lexer state (used only with ``-f`` ``--storable-state`` option).
+ Initial value of lexer state should be ``-1``.
``YYLESSTHAN (n)``
- Check if less than ``n`` input characters are left (used only with generic API).
+ R-value of boolean type that is ``true`` if and only if there is less than ``n`` input characters left (used only with ``--input custom`` option).
``YYLIMIT``
- An expression of type ``YYCTYPE *`` that marks the end of the buffer ``YYLIMIT[-1]``
- is the last character in the buffer). The generated code repeatedly
- compares ``YYCURSOR`` to ``YYLIMIT`` to determine when the buffer needs
- (re)filling.
+ R-value of type ``YYCTYPE *`` that marks the end of input (``YYLIMIT[-1]`` should be the last input character).
+ Lexer compares ``YYCURSOR`` and ``YYLIMIT`` in order to determine if there is enough input characters left.
``YYMARKER``
- An l-value of type ``YYCTYPE *``.
- The generated code saves backtracking information in ``YYMARKER``. Some
- simple scanners might not use this.
+ L-value of type ``YYCTYPE *`` used to backup input position of successful match.
+ This might be necessary if there is an overlapping longer rule that might also match.
``YYMTAGP (t)``
- Append current input position to the history of tag ``t``.
+ Append current input position to the history of *m-tag* ``t`` (used only with ``-T`` ``--tags`` option).
``YYMTAGN (t)``
- Append default value to the history of tag ``t``.
+ Append default value to the history of *m-tag* ``t`` (used only with ``-T`` ``--tags`` option).
``YYMAXFILL``
- This will be automatically defined by ``/*!max:re2c*/`` blocks as explained above.
+ Integral constant that denotes maximal value of ``YYFILL`` argument
+ and is autogenerated by ``/*!max:re2c*/`` directive.
``YYMAXNMATCH``
- This will be automatically defined by ``/*!maxnmatch:re2c*/``.
+ Integral constant that denotes maximal number of capturing groups in a rule
+ and is autogenerated by ``/*!maxnmatch:re2c*/`` directive (used only with ``--posix-captures`` option).
+
+``yynmatch``
+ L-value of unsigned integral type that is used to hold the number of capturing groups in the matching rule.
+ Used only with ``-P`` ``--posix-captures`` option.
``YYPEEK ()``
- Get current input character (used only with generic API).
+ R-value of type ``YYCTYPE`` that denotes current input character (used only with ``--input custom`` option).
+
+``yypmatch``
+ An array of l-values that are used to hold the values of *s-tags*
+ corresponding to the capturing parentheses in the matching rule.
+ The length of array must be at least ``yynmatch * 2`` (ideally ``YYMAXNMATCH * 2``).
+ Used only with ``-P`` ``--posix-captures`` option.
``YYRESTORE ()``
- Restore input position (used only with generic API).
+ Restore input position (used only with ``--input custom`` option).
``YYRESTORECTX ()``
- Restore input position from the value of trailing context (used only with generic API).
+ Restore input position from the value of trailing context (used only with ``--input custom`` option).
``YYRESTORETAG (t)``
- Restore input position from the value of tag ``t`` (used only with generic API).
-
-``YYSETCONDITION (c)``
- This define is used to set the condition in
- transition rules. This is only being used when ``-c`` is active and
- transition rules are being used.
-
-``YYSETSTATE (s)``
- The user only needs to define this macro if the ``-f``
- flag was specified. In that case, the generated code "calls"
- ``YYSETSTATE`` just before calling ``YYFILL (n)``. The parameter to
- ``YYSETSTATE`` is a signed integer that uniquely identifies the specific
- instance of ``YYFILL (n)`` that is about to be called. Should the user
- wish to save the state of the scanner and have ``YYFILL (n)`` return to
- the caller, all he has to do is store that unique identifier in a
- variable. Later, when the scanner is called again, it will call
- ``YYGETSTATE ()`` and resume execution right where it left off. The
- generated code will contain both ``YYSETSTATE (s)`` and ``YYGETSTATE`` even
- if ``YYFILL (n)`` is disabled.
+ Restore input position from the value of *s-tag* ``t`` (used only with ``--input custom`` option).
+
+``YYSETCONDITION (condition)``
+ Set current *condition* identifier to ``condition`` (used only with ``-c`` ``--conditions`` option).
+
+``YYSETSTATE (state)``
+ Set current lexer state to ``state`` (used only with ``-f`` ``--storable-state`` option).
+ Parameter ``state`` is of signed integral type.
``YYSKIP ()``
Advance input position to the next character (used only with generic API).
``YYSTAGP (t)``
- Save current input position to tag ``t`` (used only with generic API).
+ Save current input position to *s-tag* ``t`` (used only with ``-T`` ``--tags`` and ``--input custom`` option).
``YYSTAGN (t)``
- Save default value to tag ``t`` (used only with generic API).
+ Save default value to *s-tag* ``t`` (used only with ``-T`` ``--tags`` and ``--input custom`` options).
--- /dev/null
+
+Named definitions are of the form ``name = regexp ;``
+where ``name`` is an identifier that consists of letters, digits and underscores,
+and ``regexp`` is a regular expression.
+With ``-F`` ``--flex-syntax`` option named definitions are also of the form ``name regexp``.
+Each name should be defined before it is used.
-``"foo"``
- literal string ``"foo"``. ANSI-C escape sequences can be used.
+re2c uses the following syntax for regular expressions:
-``'foo'``
- literal string ``"foo"`` (case insensitive for characters [a-zA-Z]).
- ANSI-C escape sequences can be used.
+* ``"foo"`` case-sensitive string literal
-``[xyz]``
- character class; in this case, the regular expression matches ``x``, ``y``, or ``z``.
+* ``'foo'`` case-insensitive string literal
-``[abj-oZ]``
- character class with a range in it; matches ``a``, ``b``, any letter from ``j`` through ``o``, or ``Z``.
+* ``[a-xyz]``, ``[^a-xyz]`` character class (possibly negated)
-``[^class]``
- inverted character class.
+* ``.`` any character except newline
-``r \ s``
- match any ``r`` which isn't ``s``. ``r`` and ``s`` must be regular expressions
- which can be expressed as character classes.
+* ``R \ S`` difference of character classes ``R`` and ``S``
-``r*``
- zero or more occurrences of ``r``.
+* ``R*`` zero or more occurrences of ``R``
-``r+``
- one or more occurrences of ``r``.
+* ``R+`` zero or more occurrences of ``R``
-``r?``
- optional ``r``.
+* ``R?`` optional ``R``
-``(r)``
- ``r``; parentheses are used to override precedence.
+* ``R{n}`` repetition of ``R`` exactly ``n`` times
-``r s``
- ``r`` followed by ``s`` (concatenation).
+* ``R{n,}`` repetition of ``R`` at least ``n`` times
-``r | s``
- ``r`` or ``s`` (alternative).
+* ``R{n,m}`` repetition of ``R`` from ``n`` to ``m`` times
-``r`` / ``s``
- ``r`` but only if it is followed by ``s``. Note that ``s`` is not
- part of the matched text. This type of regular expression is called
- "trailing context". Trailing context can only be at the end of a rule
- and cannot be part of a named definition.
+* ``(R)`` just ``R``; parentheses are used to override precedence or for POSIX-style submatch
-``r{n}``
- matches ``r`` exactly ``n`` times.
+* ``R S`` concatenation: ``R`` followed by ``S``
-``r{n,}``
- matches ``r`` at least ``n`` times.
+* ``R | S`` alternative: ``R or S``
-``r{n,m}``
- matches ``r`` at least ``n`` times, but not more than ``m`` times.
+* ``R / S`` loohakead: ``R`` followed by ``S``, but ``S`` is not consumed
-``.``
- match any character except newline.
+* ``name`` the regular expression defined as ``name`` (or literal string ``"name"`` in Flex compatibility mode)
-``name``
- matches a named definition as specified by ``name`` only if ``-F`` is
- off. If ``-F`` is active then this behaves like it was enclosed in double
- quotes and matches the string "name".
+* ``{name}`` the regular expression defined as ``name`` in Flex compatibility mode
-``@stag``
- save input position at which ``@stag`` matches in a variable named ``stag``
+* ``@stag`` an *s-tag*: saves the last input position at which ``@stag`` matches in a variable named ``stag``
-``#mtag``
- save all input positions at which ``#mtag`` matches in a variable named ``mtag``
- (multiple positions are possible if ``#mtag`` is enclosed in a repetition subexpression that matches several times)
+* ``#mtag`` an *m-tag*: saves all input positions at which ``#mtag`` matches in a variable named ``mtag``
-Character classes and string literals may contain octal or hexadecimal
-character definitions and the following set of escape sequences:
-``\a``, ``\b``, ``\f``, ``\n``, ``\r``, ``\t``, ``\v``, ``\\``. An octal character is defined by a backslash
-followed by its three octal digits (e.g., ``\377``).
-Hexadecimal characters from 0 to 0xFF are defined by a backslash, a lower
-case ``x`` and two hexadecimal digits (e.g., ``\x12``). Hexadecimal characters from 0x100 to 0xFFFF are defined by a backslash, a lower case
-``\u``or an upper case ``\X``, and four hexadecimal digits (e.g., ``\u1234``).
-Hexadecimal characters from 0x10000 to 0xFFFFffff are defined by a backslash, an upper case ``\U``,
-and eight hexadecimal digits (e.g., ``\U12345678``).
-
-The only portable "any" rule is the default rule, ``*``.
+Character classes and string literals may contain the following escape sequences:
+``\a``, ``\b``, ``\f``, ``\n``, ``\r``, ``\t``, ``\v``, ``\\``, octal escapes ``\ooo`` and hexadecimal escapes ``\xhh``, ``\uhhhh`` and ``\Uhhhhhhhh``.
-There is one special rule that can be used instead of regular expression: the default rule ``*``.
-Note that the default rule ``*`` differs from ``[^]``: the default rule has the lowest priority,
-matches any code unit (either valid or invalid) and always consumes exactly one character.
-``[^]``, on the other hand, matches any valid code point (not the same as a code unit) and can consume multiple
-code units. In fact, when a variable-length encoding is used, ``*``
-is the only possible way to match an invalid input character.
-
-In general, all rules have the form:
-
- ``regular-expression-or-* code``
-
-
-If ``-c`` is active, then each regular expression is preceded by a list
-of comma-separated condition names. Besides the normal naming rules, there
-are two special cases: ``<*>`` (these rules are merged to all conditions)
-and ``<>`` (these rules cannot have an associated regular expression;
-their code is merged to all actions). Non-empty rules may furthermore specify the new
-condition. In that case, ``re2c`` will generate the necessary code to
-change the condition automatically. Rules can use ``:=>`` as a shortcut
-to automatically generate code that not only sets the
-new condition state but also continues execution with the new state. A
-shortcut rule should not be used in a loop where there is code between
-the start of the loop and the ``re2c`` block unless ``re2c:cond:goto``
-is changed to ``continue``. If some code is needed before all rules (though not before simple jumps), you
-can insert it with ``<!>`` pseudo-rules.
-
- ``<condition-list-or-*> regular-expression-or-* code``
-
- ``<condition-list-or-*> regular-expression-or-* => condition code``
-
- ``<condition-list-or-*> regular-expression-or-* :=> condition``
-
-
- ``<> code``
-
- ``<> => condition code``
-
- ``<> :=> condition``
-
-
- ``<!condition-list> code``
-
- ``<!> code``
-
+Rules consist of a regular expression followed by a user-defined action:
+a block of C/C++ code) that is executed in case of sucessful match.
+Action can be either an arbitrary block of code enclosed in curly braces ``{`` and ``}``,
+or a block of code without curly braces preceded with ``:=`` and ended with a newline that is not followed by a whitespace.
+
+If multiple rules match, ``re2c`` prefers the longest match.
+If rules match the same string, the earlier rule has priority.
+
+There is one special kind of rule: the *default rule* with ``*`` instaed of the regular expression.
+It always has the lowest priority, matches any *code unit* (either valid or invalid) and consumes exactly one *code unit*.
+Note that *default rule* is not the same as ``[^]``, which
+matches any valid *code point* and can consume multiple *code units*.
+In case of variable-length encodings ``*`` is the only possible way to match invalid input character.
+
+If ``-c`` ``--conditions`` option is used, then rules have more complex form
+described in the section about conditions.
been turned on so far or will be turned on later.
``-W<warning>``
- Turn on a ``warning``.
+ Turn on ``warning``.
``-Wno-<warning>``
- Turn off a ``warning``.
+ Turn off ``warning``.
``-Werror-<warning>``
- Turn on a ``warning`` and treat it as an error (this implies ``-W<warning>``).
+ Turn on ``warning`` and treat it as an error (this implies ``-W<warning>``).
``-Wno-error-<warning>``
Don't treat this particular ``warning`` as an error. This doesn't turn off
``-Wcondition-order``
Warn if the generated program makes implicit
- assumptions about condition numbering. You should use either the ``-t, --type-header`` option or
+ assumptions about condition numbering. One should use either the ``-t, --type-header`` option or
the ``/*!types:re2c*/`` directive to generate a mapping of condition names to numbers and then use
the autogenerated condition names.
``-Wempty-character-class``
Warn if a regular expression contains an empty
- character class. Rationally, trying to match an empty
- character class makes no sense: it should always fail. However, for
- backwards compatibility reasons, ``re2c`` allows empty character classes and
+ character class. Trying to match an empty character class makes no sense: it should always fail.
+ However, for backwards compatibility reasons ``re2c`` allows empty character classes and
treats them as empty strings. Use the ``--empty-class`` option to change the default
behavior.
``-Wmatch-empty-string``
- Warn if a regular expression in a rule is
- nullable (matches an empty string). If the DFA runs in a loop and an empty match
- is unintentional (the input position in not advanced manually), the lexer may
- get stuck in an infinite loop.
+ Warn if a rule is nullable (matches an empty string).
+ If the lexer runs in a loop and the empty match is unintentional,
+ the lexer may unexpectedly hang in an infinite loop.
``-Wswapped-range``
Warn if the lower bound of a range is greater than its upper
Warn if some input strings cause undefined
control flow in the lexer (the faulty patterns are reported). This is the
most dangerous and most common mistake. It can be easily fixed by adding
- the default rule (``*``) (this rule has the lowest priority, matches any code unit, and consumes
- exactly one code unit).
+ the default rule ``*`` which has the lowest priority, matches any code unit, and consumes
+ exactly one code unit.
``-Wunreachable-rules``
Warn about rules that are shadowed by other rules and will never match.
``-Wuseless-escape``
Warn if a symbol is escaped when it shouldn't be.
By default, re2c silently ignores such escapes, but this may as well indicate a
- typo or error in the escape sequence.
+ typo or an error in the escape sequence.
``-Wnondeterministic-tags``
- Warn if tag has ``n``-th degree of nondeterminism, where ``n`` is greater than 1.
+ Warn if a tag has ``n``-th degree of nondeterminism, where ``n`` is greater than 1.
projects / re2c / commitdiff