]> granicus.if.org Git - python/log
python
4 years agobpo-38304: Remove PyConfig.struct_size (GH-16500) (GH-16508)
Victor Stinner [Tue, 1 Oct 2019 08:56:37 +0000 (10:56 +0200)]
bpo-38304: Remove PyConfig.struct_size (GH-16500) (GH-16508)

For now, we'll rely on the fact that the config structures aren't covered by the stable ABI.

We may revisit this in the future if we further explore the idea of offering a stable embedding API.

(cherry picked from commit bdace21b769998396d0ccc8da99a8ca9b507bfdf)

4 years agobpo-38319: Fix shutil._fastcopy_sendfile(): set sendfile() max block size (GH-16491)
Giampaolo Rodola [Tue, 1 Oct 2019 03:40:54 +0000 (11:40 +0800)]
bpo-38319: Fix shutil._fastcopy_sendfile(): set sendfile() max block size (GH-16491)

4 years agobpo-32689: Updates shutil.move to allow for Path objects to be used as source arg...
Maxwell A McKinnon [Tue, 1 Oct 2019 02:41:16 +0000 (19:41 -0700)]
bpo-32689: Updates shutil.move to allow for Path objects to be used as source arg (GH-15326)

Important work originally done by @emilyemorehouse two years ago and nearly ready to go in.

This bug has affected many people and in some cases has been a dealbreaker to the adoption of the otherwise wonderful pathlib and PEP519. https://stackoverflow.com/questions/33625931/copy-file-with-pathlib-in-python.

This adds the outstanding test request from that PR @vstinner (https://github.com/python/cpython/pull/5393).

Test fails without the change, passes with it, along with every other test in test_shutil.

Some variants were experimented with to make the one line change and the most performant one was picked.

# Added Test for PathLike directory destination, the current fail case

```
Lib/test/test_shutil.py::TestMove::test_move_file_pathlike FAILED                                                               [100%]

============================================================== FAILURES ===============================================================
__________________________________________________ TestMove.test_move_file_pathlike ___________________________________________________

self = <test.test_shutil.TestMove testMethod=test_move_file_pathlike>

    def test_move_file_pathlike(self):
        # Move a file to another location on the same filesystem.
        src = pathlib.Path(self.src_file)
>       self._check_move_file(src, self.dst_dir, self.dst_file)

Lib/test/test_shutil.py:1563:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
Lib/test/test_shutil.py:1545: in _check_move_file
    shutil.move(src, dst)
/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/shutil.py:562: in move
    real_dst = os.path.join(dst, _basename(src))
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

path = PosixPath('/var/folders/r2/psq74t5x3nbfzlph8bh2pvdw0000gn/T/tmp9ie0wh9_/foo')

    def _basename(path):
        # A basename() variant which first strips the trailing slash, if present.
        # Thus we always get the last component of the path, even for directories.
        sep = os.path.sep + (os.path.altsep or '')
>       return os.path.basename(path.rstrip(sep))
E       AttributeError: 'PosixPath' object has no attribute 'rstrip'

/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/shutil.py:526: AttributeError
============================================== 1 failed, 102 deselected in 0.30 seconds ===============================================
```

After change:

```
========================================================= test session starts =========================================================
platform darwin -- Python 3.7.4, pytest-5.0.1, py-1.8.0, pluggy-0.12.0 -- /Users/maxwellmckinnon/.venvs/TA3.7/bin/python3.7
cachedir: .pytest_cache
rootdir: /Users/maxwellmckinnon/dev/cpython
plugins: cov-2.7.1, mock-1.10.4
collected 103 items / 102 deselected / 1 selected

Lib/test/test_shutil.py::TestMove::test_move_file_pathlike PASSED                                                               [100%]

============================================== 1 passed, 102 deselected in 0.06 seconds ===============================================
```

Running all the tests in test_shutil.py
```
╰─ pytest Lib/test/test_shutil.py -v
========================================================= test session starts =========================================================
platform darwin -- Python 3.7.4, pytest-5.0.1, py-1.8.0, pluggy-0.12.0 -- /Users/maxwellmckinnon/.venvs/TA3.7/bin/python3.7
cachedir: .pytest_cache
rootdir: /Users/maxwellmckinnon/dev/cpython
plugins: cov-2.7.1, mock-1.10.4
collected 103 items

Lib/test/test_shutil.py::TestShutil::test_chown PASSED                                                                          [  0%]
Lib/test/test_shutil.py::TestShutil::test_copy PASSED                                                                           [  1%]
...
Lib/test/test_shutil.py::TermsizeTests::test_stty_match SKIPPED                                                                 [ 99%]
Lib/test/test_shutil.py::PublicAPITests::test_module_all_attribute PASSED                                                       [100%]

================================================ 96 passed, 7 skipped in 1.25 seconds =================================================
```

# Performance Considerations
Is it considered poor form to get rid of _basename altogether and make use of pathlib in the move function? I'm not sure if the idea is for all these modules to strictly avoid circular dependencies. They are already using os.path which is just as much a citizen in 3.8 as pathlib right?

e.g.

`real_dst = os.path.join(dst, _basename(src))`
becomes
`real_dst = Path(dst) / Path(src).name`

I've looked around and familiarized myself, and I now think importing pathlib here is fine. My only remaining concern is that of performance.

Here's the performance difference for this step.

```
In [46]: %timeit real_dst = os.path.join("a/b/c", _basename('b/'))
2.71 µs ± 62.6 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)

In [47]: %timeit real_dst = Path("a/b/c") / Path('b/').name
12.4 µs ± 65.3 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)
```

Is 10us significant or insignificant compared to the least expensive operation this function will do? I don't know. Let's find out.

```
In [55]: %timeit os.rename('/tmp/a/a.txt', '/tmp/a/b.txt'); os.rename('/tmp/a/b.txt', '/tmp/a/a.txt')
124 µs ± 2.18 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each)
```
62us to rename. 10us seems significant enough that we wouldn't want to favor the Path sugar suggestion. 16% speed decrease from adding the 10us.

What do people think? I was hoping to get to use pathlib.Path here, but I suspect for this low level move, it should be as fast as possible, and 16% is not worth one line of sugary code to me.

https://bugs.python.org/issue32689

Automerge-Triggered-By: @gvanrossum
4 years agoFix and improve `asyncio.run()` docs (GH-16403)
Kyle Stanley [Tue, 1 Oct 2019 00:12:21 +0000 (20:12 -0400)]
Fix and improve `asyncio.run()` docs (GH-16403)

4 years agobpo-38321: Fix _testcapimodule.c warning (GH-16494)
Victor Stinner [Mon, 30 Sep 2019 19:14:26 +0000 (21:14 +0200)]
bpo-38321: Fix _testcapimodule.c warning (GH-16494)

Fix the following warning:

    modules\_testcapimodule.c(6409):
    warning C4146: unary minus operator applied to unsigned type,
    result still unsigned

4 years agoClear weakrefs in garbage found by the GC (#16495)
Neil Schemenauer [Mon, 30 Sep 2019 17:06:45 +0000 (10:06 -0700)]
Clear weakrefs in garbage found by the GC (#16495)

Fix a bug due to the interaction of weakrefs and the cyclic garbage
collector. We must clear any weakrefs in garbage in order to prevent
their callbacks from executing and causing a crash.

4 years agobpo-38321: Fix PyCStructUnionType_update_stgdict() warning (GH-16492)
Victor Stinner [Mon, 30 Sep 2019 15:49:48 +0000 (17:49 +0200)]
bpo-38321: Fix PyCStructUnionType_update_stgdict() warning (GH-16492)

bpo-22273, bpo-38321: Fix following warning:

    modules\_ctypes\stgdict.c(704):
    warning C4244: 'initializing': conversion from 'Py_ssize_t' to 'int', possible loss of data

4 years agobpo-38321: Fix _asynciomodule.c compiler warning (GH-16493)
Victor Stinner [Mon, 30 Sep 2019 14:41:34 +0000 (16:41 +0200)]
bpo-38321: Fix _asynciomodule.c compiler warning (GH-16493)

bpo-38248, bpo-38321: Fix warning:

    modules\_asynciomodule.c(2667):
    warning C4102: 'set_exception': unreferenced label

The related goto has been removed by
commit edad4d89e357c92f70c0324b937845d652b20afd.

4 years agobpo-38322: Fix gotlandmark() of PC/getpathp.c (GH-16489)
Victor Stinner [Mon, 30 Sep 2019 12:49:34 +0000 (14:49 +0200)]
bpo-38322: Fix gotlandmark() of PC/getpathp.c (GH-16489)

Write the filename into a temporary buffer instead of reusing prefix.
The problem is that join() modifies prefix inplace. If prefix is not
normalized, join() can make prefix shorter and so gotlandmark()
does modify prefix instead of returning it unmodified.

4 years agobpo-38304: Fix PyConfig usage in python_uwp.cpp (GH-16487)
Victor Stinner [Mon, 30 Sep 2019 10:52:29 +0000 (12:52 +0200)]
bpo-38304: Fix PyConfig usage in python_uwp.cpp (GH-16487)

* Set PyPreConfig.struct_size and PyConfig.struct_size as required by
  the API.
* PyPreConfig_InitPythonConfig() can now fail: check PyStatus result.

4 years agobpo-30773: Fix ag_running; prohibit running athrow/asend/aclose in parallel (#7468)
Yury Selivanov [Mon, 30 Sep 2019 05:59:11 +0000 (22:59 -0700)]
bpo-30773: Fix ag_running; prohibit running athrow/asend/aclose in parallel (#7468)

4 years agobpo-38242: Revert "bpo-36889: Merge asyncio streams (GH-13251)" (#16482)
Yury Selivanov [Mon, 30 Sep 2019 04:59:55 +0000 (21:59 -0700)]
bpo-38242: Revert "bpo-36889: Merge asyncio streams (GH-13251)" (#16482)

See https://bugs.python.org/issue38242 for more details

4 years agobpo-38163: Child mocks detect their type as sync or async (GH-16471)
Lisa Roach [Mon, 30 Sep 2019 04:56:47 +0000 (21:56 -0700)]
bpo-38163: Child mocks detect their type as sync or async (GH-16471)

4 years agobpo-37096: Add large-file tests for modules using sendfile(2) (GH-13676)
Giampaolo Rodola [Mon, 30 Sep 2019 04:51:55 +0000 (12:51 +0800)]
bpo-37096: Add large-file tests for modules using sendfile(2) (GH-13676)

4 years agobpo-38161: Removes _AwaitEvent from AsyncMock. (GH-16443)
Lisa Roach [Mon, 30 Sep 2019 04:01:28 +0000 (21:01 -0700)]
bpo-38161: Removes _AwaitEvent from AsyncMock. (GH-16443)

4 years agobpo-38317: Fix PyConfig.warnoptions priority (GH-16478)
Victor Stinner [Sun, 29 Sep 2019 23:40:17 +0000 (01:40 +0200)]
bpo-38317: Fix PyConfig.warnoptions priority (GH-16478)

Fix warnings options priority: PyConfig.warnoptions has the highest
priority, as stated in the PEP 587.

* Document options order in PyConfig.warnoptions documentation.
* Make PyWideStringList_INIT macro private: replace "Py" prefix
  with "_Py".
* test_embed: add test_init_warnoptions().

4 years agobpo-38019: correctly handle pause/resume reading of closed asyncio unix pipe (GH...
Andrew Svetlov [Sun, 29 Sep 2019 12:00:35 +0000 (15:00 +0300)]
bpo-38019: correctly handle pause/resume reading of closed asyncio unix pipe (GH-16472)

5 years agobpo-38108: Makes mock objects inherit from Base (GH-16060)
Lisa Roach [Sun, 29 Sep 2019 01:42:44 +0000 (18:42 -0700)]
bpo-38108: Makes mock objects inherit from Base (GH-16060)

5 years agobpo-38310: Predict BUILD_MAP_UNPACK_WITH_CALL -> CALL_FUNCTION_EX. (GH-16467)
Brandt Bucher [Sun, 29 Sep 2019 00:12:49 +0000 (17:12 -0700)]
bpo-38310: Predict BUILD_MAP_UNPACK_WITH_CALL -> CALL_FUNCTION_EX. (GH-16467)

5 years agobpo-37408: Precise that Tarfile "format" argument only concerns writing. (GH-14389)
Pascal Chambon [Sat, 28 Sep 2019 15:04:44 +0000 (17:04 +0200)]
bpo-37408: Precise that Tarfile "format" argument only concerns writing. (GH-14389)

5 years agobpo-38115: Deal with invalid bytecode offsets in lnotab (GH-16079)
T. Wouters [Sat, 28 Sep 2019 14:49:15 +0000 (16:49 +0200)]
bpo-38115: Deal with invalid bytecode offsets in lnotab (GH-16079)

Document that lnotab can contain invalid bytecode offsets (because of
terrible reasons that are difficult to fix). Make dis.findlinestarts()
ignore invalid offsets in lnotab. All other uses of lnotab in CPython
(various reimplementations of addr2line or line2addr in Python, C and gdb)
already ignore this, because they take an address to look for, instead.

Add tests for the result of dis.findlinestarts() on wacky constructs in
test_peepholer.py, because it's the easiest place to add them.

5 years agobpo-38216, bpo-36274: Allow subclasses to separately override validation and encoding...
Jason R. Coombs [Sat, 28 Sep 2019 12:32:01 +0000 (08:32 -0400)]
bpo-38216, bpo-36274: Allow subclasses to separately override validation and encoding behavior (GH-16448)

* bpo-38216: Allow bypassing input validation

* bpo-36274: Also allow the URL encoding to be overridden.

* bpo-38216, bpo-36274: Add tests demonstrating a hook for overriding validation, test demonstrating override encoding, and a test to capture expectation of the interface for the URL.

* Call with skip_host to avoid tripping on the host checking in the URL.

* Remove obsolete comment.

* Make _prepare_path_encoding its own attr.

This makes overriding just that simpler.

Also, don't use the := operator to make backporting easier.

* Add a news entry.

* _prepare_path_encoding -> _encode_prepared_path()

* Once again separate the path validation and request encoding, drastically simplifying the behavior. Drop the guarantee that all processing happens in _prepare_path.

5 years agobpo-38304: Add PyConfig.struct_size (GH-16451)
Victor Stinner [Sat, 28 Sep 2019 02:28:35 +0000 (04:28 +0200)]
bpo-38304: Add PyConfig.struct_size (GH-16451)

Add a new struct_size field to PyPreConfig and PyConfig structures to
allow to modify these structures in the future without breaking the
backward compatibility.

* Replace private _config_version field with public struct_size field
  in PyPreConfig and PyConfig.
* Public PyPreConfig_InitIsolatedConfig() and
  PyPreConfig_InitPythonConfig()
  return type becomes PyStatus, instead of void.
* Internal _PyConfig_InitCompatConfig(),
  _PyPreConfig_InitCompatConfig(), _PyPreConfig_InitFromConfig(),
  _PyPreConfig_InitFromPreConfig() return type becomes PyStatus,
  instead of void.
* Remove _Py_CONFIG_VERSION
* Update the Initialization Configuration documentation.

5 years agobpo-38301: In Solaris family, we must be sure to use '-D_REENTRANT' (#16446)
Jesús Cea [Sat, 28 Sep 2019 01:44:32 +0000 (03:44 +0200)]
bpo-38301: In Solaris family, we must be sure to use '-D_REENTRANT' (#16446)

5 years agobpo-38243, xmlrpc.server: Escape the server_title (GH-16373)
Dong-hee Na [Fri, 27 Sep 2019 19:59:37 +0000 (04:59 +0900)]
bpo-38243, xmlrpc.server: Escape the server_title (GH-16373)

Escape the server title of xmlrpc.server.DocXMLRPCServer
when rendering the document page as HTML.

5 years agoFix typo in the "Porting to Python 3.8" section. (GH-16435)
bariod [Fri, 27 Sep 2019 18:01:33 +0000 (20:01 +0200)]
Fix typo in the "Porting to Python 3.8" section. (GH-16435)

5 years agobpo-32820: Simplify __format__ implementation for ipaddress. (GH-16378)
Serhiy Storchaka [Fri, 27 Sep 2019 17:02:58 +0000 (20:02 +0300)]
bpo-32820: Simplify __format__ implementation for ipaddress. (GH-16378)

Also cache the compiled RE for parsing the format specifier.

5 years agobpo-38187: Fix a refleak in Tools/c-analyzer. (gh-16304)
Eric Snow [Fri, 27 Sep 2019 14:53:34 +0000 (15:53 +0100)]
bpo-38187: Fix a refleak in Tools/c-analyzer. (gh-16304)

The "Slot" helper (descriptor) is leaking references due to its caching mechanism. The change includes a partial fix to Slot, but also adds Variable.storage to replace the problematic use of Slot.

https://bugs.python.org/issue38187

5 years agobpo-38270: More fixes for strict crypto policy (GH-16418)
Christian Heimes [Fri, 27 Sep 2019 13:03:53 +0000 (15:03 +0200)]
bpo-38270: More fixes for strict crypto policy (GH-16418)

test_hmac and test_hashlib test built-in hashing implementations and
OpenSSL-based hashing implementations. Add more checks to skip OpenSSL
implementations when a strict crypto policy is active.

Use EVP_DigestInit_ex() instead of EVP_DigestInit() to initialize the
EVP context. The EVP_DigestInit() function clears alls flags and breaks
usedforsecurity flag again.

Signed-off-by: Christian Heimes <christian@python.org>
https://bugs.python.org/issue38270

5 years agobpo-38206: Clarify tp_dealloc requirements for heap allocated types. (GH-16248)
Ammar Askar [Fri, 27 Sep 2019 11:11:27 +0000 (07:11 -0400)]
bpo-38206: Clarify tp_dealloc requirements for heap allocated types. (GH-16248)

As mentioned in the bpo ticket, this mistake came up on two reviews:
- https://github.com/python/cpython/pull/16127#pullrequestreview-288312751
- https://github.com/python/cpython/pull/16071#pullrequestreview-287819525

Would be nice to have it documented in a more permanent place than 3.8's whatsnew entry.

https://bugs.python.org/issue38206

Automerge-Triggered-By: @encukou
5 years agocleanup ababstractproperty in typing.py (GH-16432)
HongWeipeng [Fri, 27 Sep 2019 07:54:26 +0000 (15:54 +0800)]
cleanup ababstractproperty in typing.py (GH-16432)

5 years agobpo-28009: Fix uuid.uuid1() and uuid.get_node() on AIX (GH-8672)
Michael Felt [Thu, 26 Sep 2019 19:43:15 +0000 (20:43 +0100)]
bpo-28009: Fix uuid.uuid1() and uuid.get_node() on AIX (GH-8672)

5 years agobpo-38275: Fix test_ssl issue caused by GH-16386 (#16428)
Christian Heimes [Thu, 26 Sep 2019 16:23:17 +0000 (18:23 +0200)]
bpo-38275: Fix test_ssl issue caused by GH-16386 (#16428)

Check presence of SSLContext.minimum_version to make tests pass with
old versions of OpenSSL.

Signed-off-by: Christian Heimes <christian@python.org>
5 years agobpo-38275: Skip ssl tests for disabled versions (GH-16386)
Christian Heimes [Thu, 26 Sep 2019 15:02:59 +0000 (17:02 +0200)]
bpo-38275: Skip ssl tests for disabled versions (GH-16386)

test_ssl now handles disabled TLS/SSL versions better. OpenSSL's crypto
policy and run-time settings are recognized and tests for disabled versions
are skipped.

Signed-off-by: Christian Heimes <christian@python.org>
https://bugs.python.org/issue38275

5 years agobpo-38239: Fix test_gdb for Link Time Optimization (LTO) (GH-16422)
Victor Stinner [Thu, 26 Sep 2019 14:54:13 +0000 (16:54 +0200)]
bpo-38239: Fix test_gdb for Link Time Optimization (LTO) (GH-16422)

5 years agobpo-38234: Py_Initialize() sets global path configuration (GH-16421)
Victor Stinner [Thu, 26 Sep 2019 13:51:50 +0000 (15:51 +0200)]
bpo-38234: Py_Initialize() sets global path configuration (GH-16421)

* Py_InitializeFromConfig() now writes PyConfig path configuration to
  the global path configuration (_Py_path_config).
* Add test_embed.test_get_pathconfig().
* Fix typo in _PyWideStringList_Join().

5 years agocompileall tests: Use shorter name for long_path test (GH-16419)
Petr Viktorin [Thu, 26 Sep 2019 13:16:32 +0000 (15:16 +0200)]
compileall tests: Use shorter name for long_path test (GH-16419)

Apparently, the path needs to be limited to 260 characters on
(some versions of) Windows.

5 years agobpo-38223: Reorganize test_shutil. (GH-16281)
Serhiy Storchaka [Thu, 26 Sep 2019 10:15:08 +0000 (13:15 +0300)]
bpo-38223: Reorganize test_shutil. (GH-16281)

* Group tests for specific functions and groups of related functions
into separate classes.
* Clean up creating and cleaning up temporary directories.
* Simplify and make more robust monkey patching of shutil.open.

5 years agobpo-38112: compileall: Skip long path path on Windows if the path can't be created...
Petr Viktorin [Thu, 26 Sep 2019 09:53:51 +0000 (11:53 +0200)]
bpo-38112: compileall: Skip long path path on Windows if the path can't be created (GH-16414)

This avoids the buildbot failure on Windows:
```
FileNotFoundError: [WinError 206] The filename or extension is too long: 'd:\\temp\\tmp5r3z438t\\long\\1\\2\\3\\4\\5\\6\\7\\8\\9\\10\\11\\12\\13\\14\\15\\16\\17\\18\\19\\20\\21\\22\\23\\24\\25\\26\\27\\28\\29\\30\\31\\32\\33\\34\\35\\36\\37\\38\\39\\40\\41\\42\\43\\44\\45\\46\\47\\48\\49\\50\\51\\52\\53\\54\\55\\56\\57\\58\\59\\60\\61\\62\\63\\64\\65\\66\\67\\68\\69\\70\\71\\72\\73\\74\\75\\76\\77\\78'
```
Creates a path that's long but avoids OS restrictions.

https://bugs.python.org/issue38112

5 years agobpo-38244: [Documentation] Fix unreliable link to sever in ftplib.rst (#16319)
Prateek Nayak [Thu, 26 Sep 2019 09:22:31 +0000 (14:52 +0530)]
bpo-38244: [Documentation] Fix unreliable link to sever in ftplib.rst (#16319)

5 years agobpo-38130: Fix error in explaining when an exception is re-raised (GH-16016)
Mohammad Dehghan [Thu, 26 Sep 2019 09:09:05 +0000 (12:39 +0330)]
bpo-38130: Fix error in explaining when an exception is re-raised (GH-16016)

Co-Authored-By: Ashwin Ramaswami <aramaswamis@gmail.com>
5 years agoDoc: Several fixes and improvements for 3.9 whatsnew (GH-16375)
Kyle Stanley [Thu, 26 Sep 2019 07:06:46 +0000 (03:06 -0400)]
Doc: Several fixes and improvements for 3.9 whatsnew (GH-16375)

5 years agoDoc: Use the `with` statement in the first example of the ftplib doc. (GH-16271)
Stéphane Wirtel [Thu, 26 Sep 2019 07:01:18 +0000 (09:01 +0200)]
Doc: Use the `with` statement in the first example of the ftplib doc. (GH-16271)

5 years agobpo-38112: Compileall improvements (GH-16012)
Lumír 'Frenzy' Balhar [Thu, 26 Sep 2019 06:28:26 +0000 (08:28 +0200)]
bpo-38112: Compileall improvements (GH-16012)

* Raise the limit of maximum path depth to actual  recursion limit

* Add posibilities to adjust a path compiled in .pyc  file.

Now, you can:
- Strip a part of path from a beggining of path into compiled file
   example "-s /test /test/build/real/test.py" → "build/real/test.py"
- Append some new path to a beggining of path into compiled file
   example "-p /boo real/test.py" → "/boo/real/test.py"

You can also use both options in the same time. In that case,
striping is done before appending.

* Add a possibility to specify multiple optimization levels

Each optimization level then leads to separated compiled file.
Use `action='append'` instead of `nargs='+'` for the -o option.
Instead of `-o 0 1 2`, specify `-o 0 -o 1 -o 2`. It's more to type,
but much more explicit.

* Add a symlinks limitation feature

This feature allows us to limit byte-compilation of symbolic
links if they are pointing outside specified dir (build root
for example).

5 years agocloses bpo-38174: Update vendored expat library to 2.2.8. (GH-16346)
Benjamin Peterson [Thu, 26 Sep 2019 04:33:58 +0000 (21:33 -0700)]
closes bpo-38174: Update vendored expat library to 2.2.8. (GH-16346)

Fixes CVE-2019-15903. See full changelog at https://github.com/libexpat/libexpat/blob/R_2_2_8/expat/Changes.

5 years agobpo-38234: Fix test_embed.test_init_setpath_config() on FreeBSD (GH-16406)
Victor Stinner [Thu, 26 Sep 2019 02:01:49 +0000 (04:01 +0200)]
bpo-38234: Fix test_embed.test_init_setpath_config() on FreeBSD (GH-16406)

Explicitly preinitializes with a Python preconfiguration to avoid
Py_SetPath() implicit preinitialization with a compat
preconfiguration.

Fix also test_init_setpath() and test_init_setpythonhome() on macOS:
use self.test_exe as the executable (and base_executable), rather
than shutil.which('python3').

5 years agobpo-38234: Complete init config documentation (GH-16404)
Victor Stinner [Thu, 26 Sep 2019 01:15:07 +0000 (03:15 +0200)]
bpo-38234: Complete init config documentation (GH-16404)

5 years agobpo-38234: Add test_init_setpath_config() to test_embed (GH-16402)
Victor Stinner [Thu, 26 Sep 2019 00:22:35 +0000 (02:22 +0200)]
bpo-38234: Add test_init_setpath_config() to test_embed (GH-16402)

* Add test_embed.test_init_setpath_config(): test Py_SetPath()
  with PyConfig.
* test_init_setpath() and test_init_setpythonhome() no longer call
  Py_SetProgramName(), but use the default program name.
* _PyPathConfig: isolated, site_import  and base_executable
  fields are now only available on Windows.
* If executable is set explicitly in the configuration, ignore
  calculated base_executable: _PyConfig_InitPathConfig() copies
  executable to base_executable.
* Complete path config documentation.

5 years agobpo-38142: Updated _hashopenssl.c to be PEP 384 compliant (#16071)
Christian Heimes [Wed, 25 Sep 2019 21:03:30 +0000 (23:03 +0200)]
bpo-38142: Updated _hashopenssl.c to be PEP 384 compliant (#16071)

* Updated _hashopenssl.c to be PEP 384 compliant
* Remove refleak test from test_hashlib. The updated type no longer accepts random arguments to __init__.

5 years agobpo-22273: Removed temporary test skipping on PPC platforms. (GH-16399)
Vinay Sajip [Wed, 25 Sep 2019 19:57:20 +0000 (20:57 +0100)]
bpo-22273: Removed temporary test skipping on PPC platforms. (GH-16399)

5 years agobpo-38271: encrypt private key test files with AES256 (GH-16385)
Christian Heimes [Wed, 25 Sep 2019 15:55:02 +0000 (17:55 +0200)]
bpo-38271: encrypt private key test files with AES256 (GH-16385)

The private keys for test_ssl were encrypted with 3DES in traditional
PKCS#5 format. 3DES and the digest algorithm of PKCS#5 are blocked by
some strict crypto policies. Use PKCS#8 format with AES256 encryption
instead.

Signed-off-by: Christian Heimes <christian@python.org>
https://bugs.python.org/issue38271

Automerge-Triggered-By: @tiran
5 years agobpo-38005: Remove support of string argument in InterpreterID(). (GH-16227)
Serhiy Storchaka [Wed, 25 Sep 2019 15:35:57 +0000 (18:35 +0300)]
bpo-38005: Remove support of string argument in InterpreterID(). (GH-16227)

Make negative interpreter id to raise ValueError instead of RuntimeError.

5 years agobpo-38255: super() can search attributes as well as methods (GH-16368)
Raymond Hettinger [Wed, 25 Sep 2019 15:13:19 +0000 (08:13 -0700)]
bpo-38255: super() can search attributes as well as methods (GH-16368)

Improvement suggested by Géry Ogam.

5 years agobpo-38234: Fix test_embed pathconfig tests (GH-16390)
Victor Stinner [Wed, 25 Sep 2019 14:30:36 +0000 (16:30 +0200)]
bpo-38234: Fix test_embed pathconfig tests (GH-16390)

bpo-38234: On macOS and FreeBSD, the temporary directory can be
symbolic link. For example, /tmp can be a symbolic link to /var/tmp.
Call realpath() to resolve all symbolic links.

5 years agobpo-38270: Check for hash digest algorithms and avoid MD5 (GH-16382)
Christian Heimes [Wed, 25 Sep 2019 14:30:20 +0000 (16:30 +0200)]
bpo-38270: Check for hash digest algorithms and avoid MD5 (GH-16382)

Make it easier to run and test Python on systems with restrict crypto policies:

* add requires_hashdigest to test.support to check if a hash digest algorithm is available and working
* avoid MD5 in test_hmac
* replace MD5 with SHA256 in test_tarfile
* mark network tests that require MD5 for MD5-based digest auth or CRAM-MD5

https://bugs.python.org/issue38270

5 years agobpo-22273: Re-enabled ctypes test on ARM machines. (GH-16388)
Vinay Sajip [Wed, 25 Sep 2019 14:05:55 +0000 (15:05 +0100)]
bpo-22273: Re-enabled ctypes test on ARM machines. (GH-16388)

5 years agobpo-36046: posix_spawn() doesn't support uid/gid (GH-16384)
Victor Stinner [Wed, 25 Sep 2019 13:52:49 +0000 (15:52 +0200)]
bpo-36046: posix_spawn() doesn't support uid/gid (GH-16384)

* subprocess.Popen now longer uses posix_spawn() if uid, gid or gids are set.
* test_subprocess: add "nobody" and "nfsnobody" group names for test_group().
* test_subprocess: test_user() and test_group() are now also tested with close_fds=False.

5 years agobpo-37064: Add option -a to pathfix.py tool (GH-15717)
PatrikKopkan [Wed, 25 Sep 2019 12:26:28 +0000 (14:26 +0200)]
bpo-37064: Add option -a to pathfix.py tool (GH-15717)

Add option -a to Tools/Scripts/pathfix.py script: add flags.

5 years agobpo-38140: Document offsets in PyMemberDef (GH-16354)
Petr Viktorin [Wed, 25 Sep 2019 11:06:16 +0000 (13:06 +0200)]
bpo-38140: Document offsets in PyMemberDef (GH-16354)

bpo-38140: Document offsets in PyMemberDef

Co-Authored-By: Ammar Askar <ammar_askar@hotmail.com>
5 years agobpo-38248: Fix inconsistent immediate asyncio.Task cancellation (GH-16330)
Yury Selivanov [Wed, 25 Sep 2019 10:32:08 +0000 (03:32 -0700)]
bpo-38248: Fix inconsistent immediate asyncio.Task cancellation (GH-16330)

5 years agoChanged conditions for ctypes array-in-struct handling. (GH-16381)
Vinay Sajip [Wed, 25 Sep 2019 10:11:57 +0000 (11:11 +0100)]
Changed conditions for ctypes array-in-struct handling. (GH-16381)

5 years agobpo-38260: Add Docs on asyncio.run (GH-16337)
Emmanuel Arias [Wed, 25 Sep 2019 08:53:49 +0000 (05:53 -0300)]
bpo-38260: Add Docs on asyncio.run (GH-16337)

Add docs about return and raise exception on asyncio.run

https://bugs.python.org/issue38260

Automerge-Triggered-By: @asvetlov
5 years agobpo-22273: Disabled tests while investigating buildbot failures on ARM7L/PPC64. ...
Vinay Sajip [Wed, 25 Sep 2019 06:58:32 +0000 (07:58 +0100)]
bpo-22273: Disabled tests while investigating buildbot failures on ARM7L/PPC64. (GH-16377)

5 years agobpo-38219: Optimize dict creating and updating by a dict. (GH-16268)
Serhiy Storchaka [Wed, 25 Sep 2019 06:47:00 +0000 (09:47 +0300)]
bpo-38219: Optimize dict creating and updating by a dict. (GH-16268)

5 years agobpo-38265: Update os.pread to accept the length type as Py_ssize_t. (GH-16359)
Dong-hee Na [Wed, 25 Sep 2019 05:47:04 +0000 (14:47 +0900)]
bpo-38265: Update os.pread to accept the length type as Py_ssize_t. (GH-16359)

5 years agobpo-22273: Update ctypes to correctly handle arrays in small structur… (GH-15839)
Vinay Sajip [Wed, 25 Sep 2019 03:38:44 +0000 (04:38 +0100)]
bpo-22273: Update ctypes to correctly handle arrays in small structur… (GH-15839)

5 years agobpo-38234: Cleanup getpath.c (GH-16367)
Victor Stinner [Wed, 25 Sep 2019 00:54:25 +0000 (02:54 +0200)]
bpo-38234: Cleanup getpath.c (GH-16367)

* search_for_prefix() directly calls reduce() if found is greater
  than 0.
* Add calculate_pybuilddir() subfunction.
* search_for_prefix(): add path string buffer for readability.
* Fix some error handling code paths: release resources on error.
* calculate_read_pyenv(): rename tmpbuffer to filename.
* test.pythoninfo now also logs windows.dll_path

5 years agobpo-38234: test_embed: test pyvenv.cfg and pybuilddir.txt (GH-16366)
Victor Stinner [Wed, 25 Sep 2019 00:10:35 +0000 (02:10 +0200)]
bpo-38234: test_embed: test pyvenv.cfg and pybuilddir.txt (GH-16366)

Add test_init_pybuilddir() and test_init_pyvenv_cfg() to test_embed
to test pyvenv.cfg and pybuilddir.txt configuration files.

Fix sysconfig._generate_posix_vars(): pybuilddir.txt uses UTF-8
encoding, not ASCII.

5 years agobpo-36871: Avoid duplicated 'Actual:' in assertion message (GH-16361)
Samuel Freilich [Tue, 24 Sep 2019 22:04:29 +0000 (18:04 -0400)]
bpo-36871: Avoid duplicated 'Actual:' in assertion message (GH-16361)

Fixes an issue caught after merge of PR 16005.

Tightened test assertions to check the entire assertion message.

5 years agobpo-36871: Handle spec errors in assert_has_calls (GH-16005)
Samuel Freilich [Tue, 24 Sep 2019 19:08:31 +0000 (15:08 -0400)]
bpo-36871: Handle spec errors in assert_has_calls (GH-16005)

The fix in PR 13261 handled the underlying issue about the spec for specific methods not being applied correctly, but it didn't fix the issue that was causing the misleading error message.

The code currently grabs a list of responses from _call_matcher (which may include exceptions). But it doesn't reach inside the list when checking if the result is an exception. This results in a misleading error message when one of the provided calls does not match the spec.

https://bugs.python.org/issue36871

Automerge-Triggered-By: @gpshead
5 years agobpo-38234: Add tests for Python init path config (GH-16358)
Victor Stinner [Tue, 24 Sep 2019 16:21:02 +0000 (18:21 +0200)]
bpo-38234: Add tests for Python init path config (GH-16358)

5 years agobpo-38234: Py_SetPath() uses the program full path (GH-16357)
Victor Stinner [Tue, 24 Sep 2019 15:44:15 +0000 (17:44 +0200)]
bpo-38234: Py_SetPath() uses the program full path (GH-16357)

Py_SetPath() now sets sys.executable to the program full path
(Py_GetProgramFullPath()), rather than to the program name
(Py_GetProgramName()).

Fix also memory leaks in pathconfig_set_from_config().

5 years agobpo-37123: multiprocessing test_mymanager() accepts SIGTERM (GH-16349)
Victor Stinner [Tue, 24 Sep 2019 12:19:48 +0000 (14:19 +0200)]
bpo-37123: multiprocessing test_mymanager() accepts SIGTERM (GH-16349)

Multiprocessing test test_mymanager() now also expects -SIGTERM, not
only exitcode 0.

bpo-30356: BaseManager._finalize_manager() sends SIGTERM to the
manager process if it takes longer than 1 second to stop, which
happens on slow buildbots.

5 years agobpo-38212: Increase MP test_queue_feeder_donot_stop_onexc() timeout (GH-16348)
Victor Stinner [Tue, 24 Sep 2019 10:47:49 +0000 (12:47 +0200)]
bpo-38212: Increase MP test_queue_feeder_donot_stop_onexc() timeout (GH-16348)

Multiprocessing tests: increase test_queue_feeder_donot_stop_onexc()
timeout from 1 to 60 seconds.

5 years agobpo-38136: Updates await_count and call_count to be different things (GH-16192)
Lisa Roach [Tue, 24 Sep 2019 03:49:40 +0000 (20:49 -0700)]
bpo-38136: Updates await_count and call_count to be different things (GH-16192)

5 years agocloses bpo-16637: libpython: construct integer object directly from gdbvalue (GH...
Marc Hartmayer [Tue, 24 Sep 2019 03:34:12 +0000 (05:34 +0200)]
closes bpo-16637: libpython: construct integer object directly from gdbvalue (GH-15232)

This fixes the exception '`ValueError: invalid literal for int() with base 10`
if `str(gdbval)` returns a hexadecimal value (e.g. '0xa0'). This is the case if
the output-radix is set to 16 in gdb. See
https://sourceware.org/gdb/onlinedocs/gdb/Numbers.html for more information.

5 years agocloses bpo-38253: Fix typo of Py_SET_ERANGE_IF_OVERFLOW in pyport.h. (GH-16230)
Hai Shi [Tue, 24 Sep 2019 02:20:47 +0000 (21:20 -0500)]
closes bpo-38253: Fix typo of Py_SET_ERANGE_IF_OVERFLOW in pyport.h. (GH-16230)

5 years agobpo-38234: read_pth_file() now returns PyStatus (GH-16338)
Victor Stinner [Mon, 23 Sep 2019 22:55:48 +0000 (00:55 +0200)]
bpo-38234: read_pth_file() now returns PyStatus (GH-16338)

Refactor path configuration code:

* read_pth_file() now returns PyStatus to report errors, rather than
  calling Py_FatalError().
* Move argv0_path and zip_path buffers out of PyCalculatePath
  structures.
* On Windows, _PyPathConfig.home is now preferred over PyConfig.home.

5 years agobpo-38234: Fix _PyConfig_InitPathConfig() (GH-16335)
Victor Stinner [Mon, 23 Sep 2019 16:47:29 +0000 (18:47 +0200)]
bpo-38234: Fix _PyConfig_InitPathConfig() (GH-16335)

* _PyConfig_InitPathConfig() now starts by copying the global path
  configuration, and then override values set in PyConfig.
* _PyPathConfig_Calculate() implementations no longer override
  _PyPathConfig fields which are already computed. For example,
  if _PyPathConfig.prefix is not NULL, leave it unchanged.
* If Py_SetPath() has been called, _PyConfig_InitPathConfig() doesn't
  call _PyPathConfig_Calculate() anymore.
* _PyPathConfig_Calculate() no longer uses PyConfig,
  except to initialize PyCalculatePath structure.
* pathconfig_calculate(): remove useless temporary
  "_PyPathConfig new_config" variable.
* calculate_module_search_path(): remove hack to workaround memory
  allocation failure, call Py_FatalError() instead.
* Fix get_program_full_path(): handle memory allocation failure.

5 years agobpo-38236: Fix init_dump_ascii_wstr() (GH-16333)
Victor Stinner [Mon, 23 Sep 2019 13:35:46 +0000 (15:35 +0200)]
bpo-38236: Fix init_dump_ascii_wstr() (GH-16333)

Add missing "return;" (to not dereference NULL pointer).

5 years agobpo-38236: Dump path config at first import error (GH-16300)
Victor Stinner [Mon, 23 Sep 2019 12:45:47 +0000 (14:45 +0200)]
bpo-38236: Dump path config at first import error (GH-16300)

Python now dumps path configuration if it fails to import the Python
codecs of the filesystem and stdio encodings.

5 years agobpo-38209: Simplify dataclasses.InitVar by using __class_getitem__(). (GH-16255)
Serhiy Storchaka [Sun, 22 Sep 2019 10:32:41 +0000 (13:32 +0300)]
bpo-38209: Simplify dataclasses.InitVar by using __class_getitem__(). (GH-16255)

5 years agoUpdated incorrect level-setting code to use setLevel(). (GH-16325)
Vinay Sajip [Sun, 22 Sep 2019 02:51:51 +0000 (03:51 +0100)]
Updated incorrect level-setting code to use setLevel(). (GH-16325)

5 years agoShorter docstring (GH-16322)
Raymond Hettinger [Sat, 21 Sep 2019 19:57:44 +0000 (12:57 -0700)]
Shorter docstring (GH-16322)

5 years agoDoc: Add whatsnew entry for loop.shutdown_default_executor() (GH-16308)
Kyle Stanley [Sat, 21 Sep 2019 06:18:40 +0000 (02:18 -0400)]
Doc: Add whatsnew entry for loop.shutdown_default_executor() (GH-16308)

5 years agoDoc: Fix spelling errors of 'initial' in enum.py (GH-16314)
HongWeipeng [Sat, 21 Sep 2019 05:22:54 +0000 (13:22 +0800)]
Doc: Fix spelling errors of 'initial' in enum.py (GH-16314)

5 years agoMinor code and comment cleanup (GH-16315)
Raymond Hettinger [Sat, 21 Sep 2019 04:46:52 +0000 (21:46 -0700)]
Minor code and comment cleanup (GH-16315)

5 years agobpo-38237: Make pow's arguments have more descriptive names and be keyword passable...
Ammar Askar [Sat, 21 Sep 2019 04:28:49 +0000 (00:28 -0400)]
bpo-38237: Make pow's arguments have more descriptive names and be keyword passable (GH-16302)

Edit: `math.pow` changes removed on Mark's request.

https://bugs.python.org/issue38237

Automerge-Triggered-By: @rhettinger
5 years agobpo-38234: Fix PyConfig_Read() when Py_SetPath() was called (GH-16298)
Victor Stinner [Fri, 20 Sep 2019 23:50:16 +0000 (01:50 +0200)]
bpo-38234: Fix PyConfig_Read() when Py_SetPath() was called (GH-16298)

* If Py_SetPath() has been called, _PyConfig_InitPathConfig() now
  uses its value.
* Py_Initialize() now longer copies path configuration from PyConfig
  to the global path configuration (_Py_path_config).

5 years agoDoc: Remove provisional note for asyncio.run() (GH-16310)
Kyle Stanley [Fri, 20 Sep 2019 23:46:21 +0000 (19:46 -0400)]
Doc: Remove provisional note for asyncio.run() (GH-16310)

Based on a comment from @asvetlov https://github.com/python/cpython/pull/15735#discussion_r323619076, this removes the provisional note for ``asyncio.run()`` in the documentation.

Automerge-Triggered-By: @1st1
5 years agobpo-38234: Remove _PyPathConfig.dll_path (GH-16307)
Victor Stinner [Fri, 20 Sep 2019 23:02:56 +0000 (01:02 +0200)]
bpo-38234: Remove _PyPathConfig.dll_path (GH-16307)

The DLL path is not computed from any user configuration and cannot
be configured by PyConfig. Instead, add a new _Py_dll_path global variable.

Remove _PyConfig_SetPathConfig(): replaced with _PyPathConfig_Init().

Py_Initialize() now longer sets the "global path configuration",
but only initialize _Py_dll_path.

5 years agobpo-38205: Py_UNREACHABLE() calls Py_FatalError() (GH-16290)
Victor Stinner [Fri, 20 Sep 2019 21:10:16 +0000 (23:10 +0200)]
bpo-38205: Py_UNREACHABLE() calls Py_FatalError() (GH-16290)

5 years agobpo-33694: Remove test_asyncio ProactorDatagramTransportTests (GH-16288)
Victor Stinner [Fri, 20 Sep 2019 21:07:59 +0000 (23:07 +0200)]
bpo-33694: Remove test_asyncio ProactorDatagramTransportTests (GH-16288)

ProactorDatagramTransportTests tests are too close to the
implementation.

5 years agobpo-6559: Update _posixsubprocess.fork_exec doc (GH-16283)
Orivej Desh [Fri, 20 Sep 2019 17:01:10 +0000 (17:01 +0000)]
bpo-6559: Update _posixsubprocess.fork_exec doc (GH-16283)

It did not list the argument added in d4cc7bf993eda4149a05ed25f2f33e18e43fd7c1.

https://bugs.python.org/issue6559

Automerge-Triggered-By: @gpshead
5 years agobpo-37937: Mention frame.f_trace in sys.settrace docs (GH-15439)
Ram Rachum [Fri, 20 Sep 2019 15:19:02 +0000 (18:19 +0300)]
bpo-37937: Mention frame.f_trace in sys.settrace docs (GH-15439)

Mention frame.f_trace in sys.settrace docs, as well as the fact you still
need to call `sys.settrace` to enable the tracing machinery before setting
`frame.f_trace` will have any effect.

5 years agobpo-37353: Updated parser note about source code compatibility(GH-14277)
Prateek Nayak [Fri, 20 Sep 2019 13:55:26 +0000 (19:25 +0530)]
bpo-37353: Updated parser note about source code compatibility(GH-14277)

5 years agobpo-38093: Correctly returns AsyncMock for async subclasses. (GH-15947)
Lisa Roach [Fri, 20 Sep 2019 04:04:18 +0000 (21:04 -0700)]
bpo-38093: Correctly returns AsyncMock for async subclasses. (GH-15947)

5 years agobpo-34002: Minor efficiency and clarity improvements in email package. (GH-7999)
Michael Selik [Fri, 20 Sep 2019 03:25:55 +0000 (20:25 -0700)]
bpo-34002: Minor efficiency and clarity improvements in email package. (GH-7999)

* Check intersection of two sets explicitly

Comparing ``len(a) > ``len(a - b)`` is essentially looking for an
intersection between the two sets. If set ``b`` does not intersect ``a``
then ``len(a - b)`` will be equal to ``len(a)``. This logic is more
clearly expressed as ``a & b``.

* Change while/pop to a for-loop

Copying the list, then repeatedly popping the first element was
unnecessarily slow. I also cleaned up a couple other inefficiencies.
There's no need to unpack a tuple, then re-pack and append it. The list
can be created with the first element instead of empty. Secondly, the
``endswith`` method returns a bool, so there's no need for an if-
statement to set ``encoding`` to True or False.

* Use set.intersection to check for intersections

``a.intersection(b)`` method is more clear of purpose than ``not
a.isdisjoint(b)`` and avoids an unnecessary set construction that ``a &
set(b)`` performs.

* Use not isdisjoint instead of intersection

While it reads slightly worse, the isdisjoint method will stop when it
finds a counterexample and returns a bool, rather than looping over the
entire iterable and constructing a new set.

5 years agobpo-38140: Make dict and weakref offsets opaque for C heap types (#16076)
Eddie Elizondo [Thu, 19 Sep 2019 16:29:05 +0000 (09:29 -0700)]
bpo-38140: Make dict and weakref offsets opaque for C heap types (#16076)

* Make dict and weakref offsets opaque for C heap types

* Add news

5 years agobpo-34037: test_asyncio uses shutdown_default_executor() (GH-16284)
Victor Stinner [Thu, 19 Sep 2019 14:45:06 +0000 (16:45 +0200)]
bpo-34037: test_asyncio uses shutdown_default_executor() (GH-16284)