Lexer: unified handling of various re2c directives.

Lexer: unified handling of various re2c directives.

Now when lexer encounters the beginning of a new directive, it
dumps all the intermediate code to the output.

Before this commit re2c lexer dumped intermediate code on each
newline that occured in the input.

So this commit affects two aspects:
    - Intermediate code is dumped much less often: it's good for
      performance, but it becomes more probable that the intermediate
      code will occasionally occupy too much buffer space and incur
      buffer resize.
    - Some re2c directives ignored characters right before the
      (on the same line). These unfortunate characters are no longer
      ignored.