[3.6] bpo-30414: multiprocessing.Queue._feed do not break from main loop on exc ...

* bpo-30414: multiprocesing.Queue._feed do not break from main loop on exc

Queue background running thread was not handling exceptions correctly.
Any exception occurred inside thread (putting unpickable object) cause
feeder to finish running. After that every message put into queue is
silently ignored.

* bpo-30414: multiprocesing.Queue._feed do not break from main loop on exc

Queue background running thread was not handling exceptions correctly.
Any exception occurred inside thread (putting unpickable object) cause
feeder to finish running. After that every message put into queue is
silently ignored.
(cherry picked from commit bc50f03db4f58c869b78e98468e374d7e61f1227)

diff --git a/Lib/multiprocessing/queues.py b/Lib/multiprocessing/queues.py
index a4f4ef8b7b5334ea88efb97690de8abe59cd4b75..7f77837a74acb6809e248f253f26a32a216c60e5 100644 (file)
--- a/Lib/multiprocessing/queues.py
+++ b/Lib/multiprocessing/queues.py
@@ -221,8 +221,8 @@ class Queue(object):
         else:
             wacquire = None
 
-        try:
-            while 1:
+        while 1:
+            try:
                 nacquire()
                 try:
                     if not buffer:
@@ -249,21 +249,19 @@ class Queue(object):
                                 wrelease()
                 except IndexError:
                     pass
-        except Exception as e:
-            if ignore_epipe and getattr(e, 'errno', 0) == errno.EPIPE:
-                return
-            # Since this runs in a daemon thread the resources it uses
-            # may be become unusable while the process is cleaning up.
-            # We ignore errors which happen after the process has
-            # started to cleanup.
-            try:
+            except Exception as e:
+                if ignore_epipe and getattr(e, 'errno', 0) == errno.EPIPE:
+                    return
+                # Since this runs in a daemon thread the resources it uses
+                # may be become unusable while the process is cleaning up.
+                # We ignore errors which happen after the process has
+                # started to cleanup.
                 if is_exiting():
                     info('error in queue thread: %s', e)
+                    return
                 else:
                     import traceback
                     traceback.print_exc()
-            except Exception:
-                pass
 
 _sentinel = object()
 

diff --git a/Lib/test/_test_multiprocessing.py b/Lib/test/_test_multiprocessing.py
index 3eb83c59ab17d70c62957cc623083728d55eeeb9..f1f93674935e7ab13d80aba6d8922355c867066c 100644 (file)
--- a/Lib/test/_test_multiprocessing.py
+++ b/Lib/test/_test_multiprocessing.py
@@ -752,6 +752,20 @@ class _TestQueue(BaseTestCase):
         # Windows (usually 15.6 ms)
         self.assertGreaterEqual(delta, 0.170)
 
+    def test_queue_feeder_donot_stop_onexc(self):
+        # bpo-30414: verify feeder handles exceptions correctly
+        if self.TYPE != 'processes':
+            self.skipTest('test not appropriate for {}'.format(self.TYPE))
+
+        class NotSerializable(object):
+            def __reduce__(self):
+                raise AttributeError
+        with test.support.captured_stderr():
+            q = self.Queue()
+            q.put(NotSerializable())
+            q.put(True)
+            self.assertTrue(q.get(timeout=0.1))
+
 #
 #
 #

diff --git a/Misc/NEWS b/Misc/NEWS
index 58a2ee0c3e53a744ee9a26a855531e16b5802afb..bdaf7aaf70a0bfde6a9fc7ed82b6b7b28f35fc96 100644 (file)
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -45,6 +45,9 @@ Core and Builtins
 Library
 -------
 
+- bpo-30414: multiprocessing.Queue._feed background running
+  thread do not break from main loop on exception.
+
 - bpo-30003: Fix handling escape characters in HZ codec.  Based on patch
   by Ma Lin.