From: R David Murray Date: Mon, 10 Mar 2014 18:27:21 +0000 (-0400) Subject: whatsnew: multiprocessing start methods and context (#8713 and #18999) X-Git-Tag: v3.4.1rc1~233^2~72 X-Git-Url: https://granicus.if.org/sourcecode?a=commitdiff_plain;h=d840b8d6139ed0784d7fea1293a6b7d88a1d8a4f;p=python whatsnew: multiprocessing start methods and context (#8713 and #18999) Also tweaked the docs a bit to use our standard style for versionadded/changed. (I'm guessing there are other places in the multiprocessing docs where similar tweaks should be made.) --- diff --git a/Doc/library/multiprocessing.rst b/Doc/library/multiprocessing.rst index f757b312d7..16aabd5002 100644 --- a/Doc/library/multiprocessing.rst +++ b/Doc/library/multiprocessing.rst @@ -136,9 +136,11 @@ to start a process. These *start methods* are Available on Unix platforms which support passing file descriptors over Unix pipes. -Before Python 3.4 *fork* was the only option available on Unix. Also, -prior to Python 3.4, child processes would inherit all the parents -inheritable handles on Windows. +.. versionchanged:: 3.4 + *span* added on all unix platforms, and *forkserver* added for + some unix platforms. + Child processes no longer inherit all of the parents inheritable + handles on Windows. On Unix using the *spawn* or *forkserver* start methods will also start a *semaphore tracker* process which tracks the unlinked named @@ -1853,25 +1855,30 @@ with the :class:`Pool` class. callbacks and has a parallel map implementation. *processes* is the number of worker processes to use. If *processes* is - ``None`` then the number returned by :func:`os.cpu_count` is used. If - *initializer* is not ``None`` then each worker process will call + ``None`` then the number returned by :func:`os.cpu_count` is used. + + If *initializer* is not ``None`` then each worker process will call ``initializer(*initargs)`` when it starts. + *maxtasksperchild* is the number of tasks a worker process can complete + before it will exit and be replaced with a fresh worker process, to enable + unused resources to be freed. The default *maxtasksperchild* is None, which + means worker processes will live as long as the pool. + + *context* can be used to specify the context used for starting + the worker processes. Usually a pool is created using the + function :func:`multiprocessing.Pool` or the :meth:`Pool` method + of a context object. In both cases *context* is set + appropriately. + Note that the methods of the pool object should only be called by the process which created the pool. .. versionadded:: 3.2 - *maxtasksperchild* is the number of tasks a worker process can complete - before it will exit and be replaced with a fresh worker process, to enable - unused resources to be freed. The default *maxtasksperchild* is None, which - means worker processes will live as long as the pool. + *maxtasksperchild* .. versionadded:: 3.4 - *context* can be used to specify the context used for starting - the worker processes. Usually a pool is created using the - function :func:`multiprocessing.Pool` or the :meth:`Pool` method - of a context object. In both cases *context* is set - appropriately. + *context* .. note:: diff --git a/Doc/whatsnew/3.4.rst b/Doc/whatsnew/3.4.rst index 38c025015b..8cf9165517 100644 --- a/Doc/whatsnew/3.4.rst +++ b/Doc/whatsnew/3.4.rst @@ -1063,11 +1063,25 @@ On Unix, two new :ref:`start methods ` (``spawn`` and ``forkserver``) have been added for starting processes using :mod:`multiprocessing`. These make the mixing of processes with threads more robust, and the ``spawn`` method matches the semantics that multiprocessing has -always used on Windows. (Contributed by Richard Oudkerk in :issue:`8713`). - -Also, except when using the old *fork* start method, child processes -will no longer inherit unneeded handles/file descriptors from their parents -(part of :issue:`8713`). +always used on Windows. New function +:func:`~multiprocessing.get_all_start_methods` reports all start methods +available on the platform, :func:`~multiprocessing.get_start_method` reports +the current start method, and :func:`~multiprocessing.set_start_method` sets +the start method. (Contributed by Richard Oudkerk in :issue:`8713`). + +:mod:`multiprocessing` also now has the concept of a ``context``, which +determines how child processes are created. New function +:func:`~multiprocessing.get_context` returns a context that uses a specified +start method. It has the same API as the :mod:`multiprocessing` module itself, +so you can use it to create :class:`~multiprocessing.pool.Pool`\ s and other +objects that will operate within that context. This allows a framework and an +application or different parts of the same application to use multiprocessing +without interfering with each other. (Contributed by Richard Oudkerk in +:issue:`18999`.) + +Except when using the old *fork* start method, child processes no longer +inherit unneeded handles/file descriptors from their parents (part of +:issue:`8713`). :mod:`multiprocessing` now relies on :mod:`runpy` (which implements the ``-m`` switch) to initialise ``__main__`` appropriately in child processes diff --git a/Misc/NEWS b/Misc/NEWS index cda67d6611..1271bfe9af 100644 --- a/Misc/NEWS +++ b/Misc/NEWS @@ -1636,6 +1636,9 @@ Library - Issue #18281: Unused stat constants removed from `tarfile`. +- Issue #18999: Multiprocessing now supports 'contexts' with the same API + as the module, but bound to specified start methods. + - Issue #18468: The re.split, re.findall, and re.sub functions and the group() and groups() methods of match object now always return a string or a bytes object. @@ -2051,6 +2054,10 @@ Library - Issue #18532: Change the builtin hash algorithms' names to lower case names as promised by hashlib's documentation. +- Issue #8713: add new spwan and forkserver start methods, and new functions + get_all_start_methods, get_start_method, and set_start_method, to + multiprocessing. + - Issue #18405: Improve the entropy of crypt.mksalt(). - Issue #12015: The tempfile module now uses a suffix of 8 random characters