* _PyPreConfig_InitCompatConfig() sets utf8_mode to 0.
* Change Py_UTF8Mode default value to 0.
* Fix _PyPreConfig_Copy(): copy also _config_init attrbibute.
* _PyPreConfig_AsDict() exports _config_init
* Fix _PyPreConfig_GetGlobalConfig(): use Py_UTF8Mode if it's greater
than 0, even if utf8_mode >= 0.
* Add unit tests on environment variables using Python API.
In development (-X dev) mode and in a debug build, IOBase finalizer
of the _pyio module now logs the exception if the close() method
fails. The exception is ignored silently by default in release build.
test_io: test_error_through_destructor() now uses
support.catch_unraisable_exception() rather than capturing stderr.
Victor Stinner [Thu, 23 May 2019 01:30:23 +0000 (03:30 +0200)]
bpo-36721: Add --embed option to python-config (GH-13500)
To embed Python into an application, a new --embed option must be
passed to "python3-config --libs --embed" to get "-lpython3.8" (link
the application to libpython). To support both 3.8 and older, try
"python3-config --libs --embed" first and fallback to "python3-config
--libs" (without --embed) if the previous command fails.
Add a pkg-config "python-3.8-embed" module to embed Python into an
application: "pkg-config python-3.8-embed --libs" includes
"-lpython3.8". To support both 3.8 and older, try "pkg-config
python-X.Y-embed --libs" first and fallback to "pkg-config python-X.Y
--libs" (without --embed) if the previous command fails (replace
"X.Y" with the Python version).
On the other hand, "pkg-config python3.8 --libs" no longer contains
"-lpython3.8". C extensions must not be linked to libpython (except
on Android, case handled by the script); this change is backward
incompatible on purpose.
"make install" now also installs "python-3.8-embed.pc".
Alexey Izbyshev [Thu, 23 May 2019 00:01:08 +0000 (03:01 +0300)]
bpo-35091: Objects/listobject.c: Replace overflow checks in gallop fu… (GH-10202)
…nctions with asserts
The actual overflow can never happen because of the following:
* The size of a list can't be greater than PY_SSIZE_T_MAX / sizeof(PyObject*).
* The size of a pointer on all supported plaftorms is at least 4 bytes.
* ofs is positive and less than the list size at the beginning of each iteration.
PyErr_WriteUnraisable() now creates a traceback object if there is no
current traceback. Moreover, call PyErr_NormalizeException() and
PyException_SetTraceback() to normalize the exception value. Ignore
silently any error.
* _PyPreConfig_GetGlobalConfig() and _PyCoreConfig_GetGlobalConfig()
now do nothing if the configuration was not initialized with
_PyPreConfig_InitCompatConfig() and _PyCoreConfig_InitCompatConfig()
* Remove utf8_mode=-2 special case: use utf8_mode=-1 instead.
* Fix _PyPreConfig_InitPythonConfig():
* isolated = 0 instead of -1
* use_environment = 1 instead of -1
* Rename _PyConfig_INIT to _PyConfig_INIT_COMPAT
* Rename _PyPreConfig_Init() to _PyPreConfig_InitCompatConfig()
* Rename _PyCoreConfig_Init() to _PyCoreConfig_InitCompatConfig()
* PyInterpreterState_New() now uses _PyCoreConfig_InitPythonConfig()
as default configuration, but it's very quickly overriden anyway.
* _freeze_importlib.c uses _PyCoreConfig_SetString() to set
program_name.
* Cleanup preconfig_init_utf8_mode(): cmdline is always non-NULL.
* Copy test_exceptions.test_unraisable() to
test_sys.UnraisableHookTest().
* Use catch_unraisable_exception() in test_coroutines,
test_exceptions, test_generators.
Adjust the shutdown test so that, after submitting three jobs
to the executor, the test checks for less than three threads,
instead of looking for exactly three threads.
If idle threads are being recycled properly, then we should have
less than three threads.
* Switched idle count to semaphor, Updated tests
As suggested by reviewer tomMoral, swapped lock-protected counter
with a semaphore to track the number of unused threads.
Adjusted test_threads_terminate to wait for completiton of the
previous future before submitting a new one (and checking the
number of threads used).
Also added a new test to confirm the thread pool can be saturated.
Sam Martin [Wed, 22 May 2019 21:29:02 +0000 (22:29 +0100)]
bpo-33110: Catch errors raised when running add_done_callback on already completed futures (GH-13141)
Wrap the callback call within the `add_done_callback` function within concurrent.futures, in order to behave in an identical manner to callbacks added to a running future are triggered once it has completed.
bpo-36878: Track extra text added to 'type: ignore' in the AST (GH-13479)
GH-13238 made extra text after a # type: ignore accepted by the parser.
This finishes the job and actually plumbs the extra text through the
parser and makes it available in the AST.
Paul Dagnelie [Wed, 22 May 2019 14:23:01 +0000 (07:23 -0700)]
bpo-36972: Add SupportsIndex (GH-13448)
In order to support typing checks calling hex(), oct() and bin() on user-defined classes, a SupportIndex protocol is required. The ability to check these at runtime would be good to add for completeness sake. This is pretty much just a copy of SupportsInt with the names tweaked.
Jeroen Demeyer [Wed, 22 May 2019 10:05:02 +0000 (12:05 +0200)]
bpo-36994: add test for profiling method_descriptor with **kwargs (GH-13461)
It adds a missing testcase for bpo-34125. This is testing code which is
affected by PEP 590, so missing this test might accidentally break
CPython if we screw up with implementing PEP 590.
Victor Stinner [Wed, 22 May 2019 09:28:22 +0000 (11:28 +0200)]
bpo-36829: Add sys.unraisablehook() (GH-13187)
Add new sys.unraisablehook() function which can be overridden to
control how "unraisable exceptions" are handled. It is called when an
exception has occurred but there is no way for Python to handle it.
For example, when a destructor raises an exception or during garbage
collection (gc.collect()).
Changes:
* Add an internal UnraisableHookArgs type used to pass arguments to
sys.unraisablehook.
* Add _PyErr_WriteUnraisableDefaultHook().
* The default hook now ignores exception on writing the traceback.
* test_sys now uses unittest.main() to automatically discover tests:
remove test_main().
* Add _PyErr_Init().
* Fix PyErr_WriteUnraisable(): hold a strong reference to sys.stderr
while using it
Max Bernstein [Tue, 21 May 2019 17:09:21 +0000 (10:09 -0700)]
bpo-36929: Modify io/re tests to allow for missing mod name (#13392)
* bpo-36929: Modify io/re tests to allow for missing mod name
For a vanishingly small number of internal types, CPython sets the
tp_name slot to mod_name.type_name, either in the PyTypeObject or the
PyType_Spec. There are a few minor places where this surfaces:
* Custom repr functions for those types (some of which ignore the
tp_name in favor of using a string literal, such as _io.TextIOWrapper)
* Pickling error messages
The test suite only tests the former. This commit modifies the test
suite to allow Python implementations to omit the module prefix.
Lihua Zhao [Tue, 21 May 2019 10:50:14 +0000 (18:50 +0800)]
bpo-36648: fix mmap issue for VxWorks (GH-12394)
The mmap module set MAP_SHARED flag when map anonymous memory, however VxWorks
only support MAP_PRIVATE when map anonymous memory, this commit clear MAP_SHARED
and set MAP_PRIVATE.
_PyRuntime.preconfig.coerce_c_locale can now be used to
check if the C locale has been coerced.
* Fix _Py_LegacyLocaleDetected(): don't attempt to coerce the
C locale if LC_ALL environment variable is set. Add 'warn'
parameter: emit_stderr_warning_for_legacy_locale() must not
the LC_ALL env var.
* _PyPreConfig_Write() sets coerce_c_locale to 0 if
_Py_CoerceLegacyLocale() fails.
Victor Stinner [Mon, 20 May 2019 14:38:48 +0000 (16:38 +0200)]
bpo-36763: Fix encoding/locale tests in test_embed (GH-13443)
* Fix encoding and locale tests in test_embed.InitConfigTests.
* InitConfigTests now only computes EXPECTED_CONFIG once.
* Add tests for PYTHONWARNINGS and PYTHONPATH env vars
* Add _PyPreConfig.parse_argv
* Add _PyCoreConfig._config_init field and _PyCoreConfigInitEnum enum
type
* Initialization functions: reject preconfig=NULL and config=NULL
* Add config parameter to _PyCoreConfig_DecodeLocaleErr(): pass
config->argv to _Py_PreInitializeFromPyArgv(), to parse config
command line arguments in preinitialization.
* Add config parameter to _PyCoreConfig_SetString(). It now
preinitializes Python.
* _PyCoreConfig_SetPyArgv() now also preinitializes Python for wide
argv
* Fix _Py_PreInitializeFromCoreConfig(): don't pass args to
_Py_PreInitializeFromPyArgv() if config.parse_argv=0.
* Use "char * const *" and "wchar_t * const *" types for 'argv'
parameters and _PyArgv.argv.
* Add unit test on preinitialization from argv.
* _PyPreConfig.allocator type becomes int
* Add _PyPreConfig_InitFromPreConfig() and
_PyPreConfig_InitFromCoreConfig() helper functions
Victor Stinner [Sat, 18 May 2019 01:21:27 +0000 (03:21 +0200)]
bpo-36763: Use _PyCoreConfig_InitPythonConfig() (GH-13398)
_PyPreConfig_InitPythonConfig() and _PyCoreConfig_InitPythonConfig()
no longer inherit their values from global configuration variables.
Changes:
* _PyPreCmdline_Read() now ignores -X dev and PYTHONDEVMODE
if dev_mode is already set.
* Inline _PyPreConfig_INIT macro into _PyPreConfig_Init() function.
* Inline _PyCoreConfig_INIT macro into _PyCoreConfig_Init() function.
* Replace _PyCoreConfig_Init() with _PyCoreConfig_InitPythonConfig()
in most tests of _testembed.c.
* Replace _PyCoreConfig_Init() with _PyCoreConfig_InitIsolatedConfig()
in _freeze_importlib.c.
* Move some initialization functions from the internal
to the private API.
* The type of initlization function 'argc' parameters becomes
Py_ssize_t, instead of int.
* Change _PyPreConfig_Copy() return type to void, instead of int.
The function cannot fail anymore.
* Fix compilation warnings on Windows.
* frozenmain.c and _testembed.c now use functions rather than macros.
* Move _Py_INIT_xxx() macros to the internal API.
* Move _PyWstrList_INIT macro to the internal API.
Py_Main() and _Py_RunMain() now return the exitcode rather than
calling Py_Exit(exitcode) when calling PyErr_Print() if the current
exception type is SystemExit.
_PyPreConfig_InitIsolatedConfig() sets configure_locale to 0 to
prevent Python to modify the LC_CTYPE locale. In that case,
coerce_c_locale an coerce_c_locale_warn are set to 0 as well.
Replace _PyPreConfig_INIT and _PyCoreConfig_INIT with new functions
_PyPreConfig_Init() and _PyCoreConfig_Init().
_PyCoreConfig: set configure_c_stdio and parse_argv to 0 by default
to behave as Python 3.6 in the default configuration.
_PyCoreConfig_Read() no longer sets coerce_c_locale_warn to 1 if it's
equal to 0. coerce_c_locale_warn must now be set to -1 (ex: using
_PyCoreConfig_InitPythonConfig()) to enable C locale coercion
warning.
Add unit tests for _PyCoreConfig_InitPythonConfig()
and _PyCoreConfig_InitIsolatedConfig().
Changes:
* Rename _PyCoreConfig_GetCoreConfig() to _PyPreConfig_GetCoreConfig()
* Fix core_read_precmdline(): handle parse_argv=0
* Fix _Py_PreInitializeFromCoreConfig(): pass coreconfig.argv
to _Py_PreInitializeFromPyArgv(), except if parse_argv=0
Victor Stinner [Fri, 17 May 2019 13:20:52 +0000 (15:20 +0200)]
bpo-36763: Add PyMemAllocatorName (GH-13387)
* Add PyMemAllocatorName enum
* _PyPreConfig.allocator type becomes PyMemAllocatorName, instead of
char*
* Remove _PyPreConfig_Clear()
* Add _PyMem_GetAllocatorName()
* Rename _PyMem_GetAllocatorsName() to
_PyMem_GetCurrentAllocatorName()
* Remove _PyPreConfig_SetAllocator(): just call
_PyMem_SetupAllocators() directly, we don't have do reallocate the
configuration with the new allocator anymore!
* _PyPreConfig_Write() parameter becomes const, as it should be in
the first place!
Pablo Galindo [Fri, 17 May 2019 10:37:08 +0000 (11:37 +0100)]
bpo-1875: Raise SyntaxError in invalid blocks that will be optimised away (GH-13332)
Move the check for dead conditionals (if 0) to the peephole optimizer
and make sure that the code block is still compiled to report any
existing syntax errors within.