bpo-38469: Handle named expression scope with global/nonlocal keywords (GH-16755) (#16760)
The symbol table handing of PEP572's assignment expressions is not resolving correctly the scope of some variables in presence of global/nonlocal keywords in conjunction with comprehensions.
(cherry picked from commit fd5c414880b2e05720b9cf14ab0b0d7ae2b7d925)
Co-authored-by: Pablo Galindo <Pablogsal@gmail.com>
Rebased version of what's new PR (GH-16745) (#16748)
* Use Unicode character for accent
* Various grammar fixes
* Sort library modules alphabetically; remove duplicated idlelib/IDLE section
(cherry picked from commit bb78f6cfa6f2b84fa4611d39c35baf1c7dce7f8d)
Announce the change in the CancelledError inheritance (GH-16730)
This is a fairly noticeable change that requires adjustments in
existing asyncio code. It should therefore be announced.
(cherry picked from commit e634da27471a76f4abe685e86e6e302e0ed6b553)
Co-authored-by: Phil Jones <philip.graham.jones@googlemail.com>
bpo-38332: Catch KeyError from unknown cte in encoded-word. (GH-16503)
KeyError should cause a failure in parsing the encoded word and should be caught and raised as a _InvalidEWError instead.
(cherry picked from commit 65dcc8a8dc41d3453fd6b987073a5f1b30c5c0fd)
Co-authored-by: Andrei Troie <andreitroie90@gmail.com>
bpo-38347: find pathfix for Python scripts whose name contain a '-' (GH-16536)
pathfix.py: Assume all files that end on '.py' are Python scripts when working recursively.
(cherry picked from commit 2b7dc40b2af6578181808ba73c1533fc114e55df)
bpo-38109: Add missing constants to Lib/stat.py (GH-16665) (GH-16690)
Add missing stat.S_IFDOOR, stat.S_IFPORT, stat.S_IFWHT,
stat.S_ISDOOR, stat.S_ISPORT, and stat.S_ISWHT values to
the Python implementation of the stat module.
(cherry picked from commit 7bb14316b8ceddb813f31040a299af94a57ab339)
PR GH-4906 changed the typing.Generic class hierarchy, leaving an
outdated comment in the library reference. User-defined Generic ABCs now
must get a abc.ABCMeta metaclass from something other than typing.Generic
inheritance.
(cherry picked from commit d47f0dd2e85ce032aebfedbde18cdb2e728fa79f)
Co-authored-by: M. Eric Irrgang <mei2n@virginia.edu>
Pablo Galindo [Wed, 9 Oct 2019 21:25:06 +0000 (22:25 +0100)]
[3.8] bpo-38379: don't claim objects are collected when they aren't (GH-16658) (GH-16683)
* [bpo-38379](https://bugs.python.org/issue38379): when a finalizer resurrects an object,
nothing is actually collected in this run of gc.
Change the stats to relect that truth.
(cherry picked from commit ecbf35f9335b0420cb8adfda6f299d6747a16515)
Co-authored-by: Tim Peters <tim.peters@gmail.com>
https://bugs.python.org/issue38379
bpo-37531: regrtest ignores output on timeout (GH-16659)
bpo-37531, bpo-38207: On timeout, regrtest no longer attempts to call
`popen.communicate() again: it can hang until all child processes
using stdout and stderr pipes completes. Kill the worker process and
ignores its output.
bpo-37531: Change also the faulthandler timeout of the main process
from 1 minute to 5 minutes, for Python slowest buildbots.
(cherry picked from commit 0ec618af98ac250a91ee9c91f8569e6df6772758)
Co-authored-by: Victor Stinner <vstinner@python.org>
Pablo Galindo [Tue, 8 Oct 2019 20:38:11 +0000 (21:38 +0100)]
[3.8] bpo-38395: Fix ownership in weakref.proxy methods (GH-16632) (GH-16662)
The implementation of weakref.proxy's methods call back into the Python
API using a borrowed references of the weakly referenced object
(acquired via PyWeakref_GET_OBJECT). This API call may delete the last
reference to the object (either directly or via GC), leaving a dangling
pointer, which can be subsequently dereferenced.
To fix this, claim a temporary ownership of the referenced object when
calling the appropriate method. Some functions because at the moment they
do not need to access the borrowed referent, but to protect against
future changes to these functions, ownership need to be fixed in
all potentially affected methods..
(cherry picked from commit 10cd00a9e3c22af37c748ea5a417f6fb66601e21)
Co-authored-by: Pablo Galindo <Pablogsal@gmail.com>
bpo-36698: IDLE no longer fails when write non-encodable characters to stderr. (GH-16583)
It now escapes them with a backslash, as the regular Python interpreter.
Added the "errors" field to the standard streams.
(cherry picked from commit b690a2759e62d9ee0b6ea1b20e8f7e4b2cdbf8bb)
bpo-38294: Add list of no-longer-escaped chars to re.escape documentation. (GH-16442)
Prior to 3.7, re.escape escaped many characters that don't have
special meaning in Python, but that use to require escaping in other
tools and languages. This commit aims to make it clear which characters
were, but are no longer escaped.
(cherry picked from commit 15ae75d660befe643ed42eb2707a557cea97256c)
* Add log() method: add timestamp and load average prefixes
to main messages.
* WindowsLoadTracker:
* LOAD_FACTOR_1 is now computed using SAMPLING_INTERVAL
* Initialize the load to the arithmetic mean of the first 5 values
of the Processor Queue Length value (so over 5 seconds), rather
than 0.0.
* Handle BrokenPipeError and when typeperf exit.
* format_duration(1.5) now returns '1.5 sec', rather than
'1 sec 500 ms'
bpo-38338, test.pythoninfo: add more ssl infos (GH-16539)
test.pythoninfo now logs environment variables used by OpenSSL and
Python ssl modules, and logs attributes of 3 SSL contexts
(SSLContext, default HTTPS context, stdlib context).
(cherry picked from commit b3e7045f8314e7b62cd95861d207fe2f97e47198)
Co-authored-by: Victor Stinner <vstinner@python.org>
* Fix TestWorkerProcess.__repr__(): start_time is only valid
if _popen is not None.
* Fix _kill(): don't set _killed to True if _popen is None.
* _run_process(): only set _killed to False after calling
run_test_in_subprocess().
(cherry picked from commit 2ea71a07d0a720707094ee55f78fd232c40724bc)
Co-authored-by: Victor Stinner <vstinner@python.org>
* Windows: Fix counter name in WindowsLoadTracker. Counter names are
localized: use the registry to get the counter name. Original
change written by Lorenz Mende.
* Regrtest.main() now ensures that the Windows load tracker is also
killed if an exception is raised
* TestWorkerProcess now ensures that worker processes are no longer
running before exiting: kill also worker processes when an
exception is raised.
* Enhance regrtest messages and warnings: include test name,
duration, add a worker identifier, etc.
* Rename MultiprocessRunner to TestWorkerProcess
* Use print_warning() to display warnings.
PyConfig_InitPythonConfig() and PyConfig_InitIsolatedConfig() no
longer return PyStatus: they cannot fail anymore.
(cherry picked from commit 8462a4936b3a551dc546a6adea04a70b0a07ca67)
Co-authored-by: Victor Stinner <vstinner@redhat.com>
Clear weakrefs in garbage found by the GC (GH-16495) (#16499)
Fix a bug due to the interaction of weakrefs and the cyclic garbage
collector. We must clear any weakrefs in garbage in order to prevent
their callbacks from executing and causing a crash.
(cherry picked from commit bcda460baf25062ab68622b3f043f52b9db4d21d)
Co-authored-by: Neil Schemenauer <nas-github@arctrix.com>
Victor Stinner [Mon, 30 Sep 2019 12:49:42 +0000 (14:49 +0200)]
bpo-38322: Fix gotlandmark() of PC/getpathp.c (GH-16490)
Write the filename into a temporary buffer instead of reusing prefix.
The problem is that join() modifies prefix inplace. If prefix is not
normalized, join() can make prefix shorter and so gotlandmark()
does modify prefix instead of returning it unmodified.
Christian Heimes [Mon, 30 Sep 2019 07:10:38 +0000 (09:10 +0200)]
[3.8] bpo-38270: More fixes for strict crypto policy (GH-16418) (#16437)
test_hmac and test_hashlib test built-in hashing implementations and
OpenSSL-based hashing implementations. Add more checks to skip OpenSSL
implementations when a strict crypto policy is active.
Use EVP_DigestInit_ex() instead of EVP_DigestInit() to initialize the
EVP context. The EVP_DigestInit() function clears alls flags and breaks
usedforsecurity flag again.
Gregory P. Smith [Sat, 28 Sep 2019 15:22:00 +0000 (08:22 -0700)]
bpo-38115: Deal with invalid bytecode offsets in lnotab (GH-16079) (GH-16464)
Document that lnotab can contain invalid bytecode offsets (because of
terrible reasons that are difficult to fix). Make dis.findlinestarts()
ignore invalid offsets in lnotab. All other uses of lnotab in CPython
(various reimplementations of addr2line or line2addr in Python, C and gdb)
already ignore this, because they take an address to look for, instead.
Add tests for the result of dis.findlinestarts() on wacky constructs in
test_peepholer.py, because it's the easiest place to add them.
bpo-38216, bpo-36274: Allow subclasses to separately override validation and encoding behavior (GH-16448)
* bpo-38216: Allow bypassing input validation
* bpo-36274: Also allow the URL encoding to be overridden.
* bpo-38216, bpo-36274: Add tests demonstrating a hook for overriding validation, test demonstrating override encoding, and a test to capture expectation of the interface for the URL.
* Call with skip_host to avoid tripping on the host checking in the URL.
* Remove obsolete comment.
* Make _prepare_path_encoding its own attr.
This makes overriding just that simpler.
Also, don't use the := operator to make backporting easier.
* Once again separate the path validation and request encoding, drastically simplifying the behavior. Drop the guarantee that all processing happens in _prepare_path.
(cherry picked from commit 7774d7831e8809795c64ce27f7df52674581d298)
Co-authored-by: Jason R. Coombs <jaraco@jaraco.com>
Add a new struct_size field to PyPreConfig and PyConfig structures to
allow to modify these structures in the future without breaking the
backward compatibility.
* Replace private _config_version field with public struct_size field
in PyPreConfig and PyConfig.
* Public PyPreConfig_InitIsolatedConfig() and
PyPreConfig_InitPythonConfig()
return type becomes PyStatus, instead of void.
* Internal _PyConfig_InitCompatConfig(),
_PyPreConfig_InitCompatConfig(), _PyPreConfig_InitFromConfig(),
_PyPreConfig_InitFromPreConfig() return type becomes PyStatus,
instead of void.
* Remove _Py_CONFIG_VERSION
* Update the Initialization Configuration documentation.
bpo-38243, xmlrpc.server: Escape the server_title (GH-16373)
Escape the server title of xmlrpc.server.DocXMLRPCServer
when rendering the document page as HTML.
(cherry picked from commit e8650a4f8c7fb76f570d4ca9c1fbe44e91c8dfaa)
Co-authored-by: Dong-hee Na <donghee.na92@gmail.com>
bpo-38206: Clarify tp_dealloc requirements for heap allocated types. (GH-16248)
As mentioned in the bpo ticket, this mistake came up on two reviews:
- https://github.com/python/cpython/pull/16127GH-pullrequestreview-288312751
- https://github.com/python/cpython/pull/16071GH-pullrequestreview-287819525
Would be nice to have it documented in a more permanent place than 3.8's whatsnew entry.
Christian Heimes [Thu, 26 Sep 2019 20:53:09 +0000 (22:53 +0200)]
[3.8] bpo-38275: Skip ssl tests for disabled versions (GH-16386) (GH-16425)
test_ssl now handles disabled TLS/SSL versions better. OpenSSL's crypto
policy and run-time settings are recognized and tests for disabled versions
are skipped.
* search_for_prefix() directly calls reduce() if found is greater
than 0.
* Add calculate_pybuilddir() subfunction.
* search_for_prefix(): add path string buffer for readability.
* Fix some error handling code paths: release resources on error.
* calculate_read_pyenv(): rename tmpbuffer to filename.
* test.pythoninfo now also logs windows.dll_path
bpo-38234: On macOS and FreeBSD, the temporary directory can be
symbolic link. For example, /tmp can be a symbolic link to /var/tmp.
Call realpath() to resolve all symbolic links.
* bpo-38234: Add test_init_setpath_config() to test_embed (GH-16402)
* Add test_embed.test_init_setpath_config(): test Py_SetPath()
with PyConfig.
* test_init_setpath() and test_init_setpythonhome() no longer call
Py_SetProgramName(), but use the default program name.
* _PyPathConfig: isolated, site_import and base_executable
fields are now only available on Windows.
* If executable is set explicitly in the configuration, ignore
calculated base_executable: _PyConfig_InitPathConfig() copies
executable to base_executable.
* Complete path config documentation.
* bpo-38234: Fix test_embed.test_init_setpath_config() on FreeBSD (GH-16406)
Explicitly preinitializes with a Python preconfiguration to avoid
Py_SetPath() implicit preinitialization with a compat
preconfiguration.
Fix also test_init_setpath() and test_init_setpythonhome() on macOS:
use self.test_exe as the executable (and base_executable), rather
than shutil.which('python3').
* bpo-38234: Py_Initialize() sets global path configuration (GH-16421)
* Py_InitializeFromConfig() now writes PyConfig path configuration to
the global path configuration (_Py_path_config).
* Add test_embed.test_get_pathconfig().
* Fix typo in _PyWideStringList_Join().
[3.8] closes bpo-38174: Update vendored expat library to 2.2.8. (GH-16409)
Fixes CVE-2019-15903. See full changelog at https://github.com/libexpat/libexpat/blob/R_2_2_8/expat/Changes.
(cherry picked from commit 52b940803860e37bcc3f6096b2d24e7c20a0e807)
bpo-38271: encrypt private key test files with AES256 (GH-16385)
The private keys for test_ssl were encrypted with 3DES in traditional
PKCSGH-5 format. 3DES and the digest algorithm of PKCSGH-5 are blocked by
some strict crypto policies. Use PKCSGH-8 format with AES256 encryption
instead.
Signed-off-by: Christian Heimes <christian@python.org>
https://bugs.python.org/issue38271
[3.8] bpo-38270: Check for hash digest algorithms and avoid MD5 (GH-16382) (GH-16393)
Make it easier to run and test Python on systems with restrict crypto policies:
* add requires_hashdigest to test.support to check if a hash digest algorithm is available and working
* avoid MD5 in test_hmac
* replace MD5 with SHA256 in test_tarfile
* mark network tests that require MD5 for MD5-based digest auth or CRAM-MD5
projects / python / log