From: Ezio Melotti Date: Sun, 20 Dec 2009 12:26:45 +0000 (+0000) Subject: Merged revisions 76914 via svnmerge from X-Git-Tag: v3.1.2rc1~243 X-Git-Url: https://granicus.if.org/sourcecode?a=commitdiff_plain;h=c5c8ff962fab3a454d7618689936512a76da18bf;p=python Merged revisions 76914 via svnmerge from svn+ssh://pythondev@svn.python.org/python/branches/py3k ........ r76914 | ezio.melotti | 2009-12-20 14:24:57 +0200 (Sun, 20 Dec 2009) | 1 line Fixed capitalization and markup; added imports in the example ........ --- diff --git a/Doc/library/tokenize.rst b/Doc/library/tokenize.rst index deb7826393..d66b07c62d 100644 --- a/Doc/library/tokenize.rst +++ b/Doc/library/tokenize.rst @@ -94,16 +94,19 @@ function it uses to do this is available: (as a string) and a list of any lines (not decoded from bytes) it has read in. - It detects the encoding from the presence of a utf-8 bom or an encoding - cookie as specified in pep-0263. If both a bom and a cookie are present, + It detects the encoding from the presence of a UTF-8 BOM or an encoding + cookie as specified in :pep:`263`. If both a BOM and a cookie are present, but disagree, a SyntaxError will be raised. - If no encoding is specified, then the default of 'utf-8' will be returned. + If no encoding is specified, then the default of ``'utf-8'`` will be returned. Example of a script re-writer that transforms float literals into Decimal objects:: + from tokenize import tokenize, untokenize, NUMBER, STRING, NAME, OP + from io import BytesIO + def decistmt(s): """Substitute Decimals for floats in a string of statements.