]> granicus.if.org Git - python/commit
bpo-30455: Generate all token related code and docs from Grammar/Tokens. (GH-10370)
authorSerhiy Storchaka <storchaka@gmail.com>
Sat, 22 Dec 2018 09:18:40 +0000 (11:18 +0200)
committerGitHub <noreply@github.com>
Sat, 22 Dec 2018 09:18:40 +0000 (11:18 +0200)
commit8ac658114dec4964479baecfbc439fceb40eaa79
treee66c4c3beda293a6fdf01763306697d15d0af157
parentc1b4b0f6160e1919394586f44b12538505fed300
bpo-30455: Generate all token related code and docs from Grammar/Tokens. (GH-10370)

"Include/token.h", "Lib/token.py" (containing now some data moved from
"Lib/tokenize.py") and new files "Parser/token.c" (containing the code
moved from "Parser/tokenizer.c") and "Doc/library/token-list.inc" (included
in "Doc/library/token.rst") are now generated from "Grammar/Tokens" by
"Tools/scripts/generate_token.py". The script overwrites files only if
needed and can be used on the read-only sources tree.

"Lib/symbol.py" is now generated by "Tools/scripts/generate_symbol_py.py"
instead of been executable itself.

Added new make targets "regen-token" and "regen-symbol" which are now
dependencies of "regen-all".

The documentation contains now strings for operators and punctuation tokens.
18 files changed:
.gitattributes
Doc/library/token-list.inc [new file with mode: 0644]
Doc/library/token.rst
Grammar/Tokens [new file with mode: 0644]
Include/token.h
Lib/symbol.py [changed mode: 0755->0644]
Lib/test/test_symbol.py
Lib/test/test_tokenize.py
Lib/token.py
Lib/tokenize.py
Makefile.pre.in
Misc/NEWS.d/next/Core and Builtins/2018-04-14-11-02-57.bpo-30455.ANRwjo.rst [new file with mode: 0644]
PCbuild/pythoncore.vcxproj
PCbuild/pythoncore.vcxproj.filters
Parser/token.c [new file with mode: 0644]
Parser/tokenizer.c
Tools/scripts/generate_symbol_py.py [new file with mode: 0755]
Tools/scripts/generate_token.py [new file with mode: 0644]