[2.7] bpo-30414: multiprocessing.Queue._feed do not break from main loop on exc ...

* bpo-30414: multiprocesing.Queue._feed do not break from main loop on exc

Queue background running thread was not handling exceptions correctly.
Any exception occurred inside thread (putting unpickable object) cause
feeder to finish running. After that every message put into queue is
silently ignored.

* bpo-30414: multiprocesing.Queue._feed do not break from main loop on exc

Queue background running thread was not handling exceptions correctly.
Any exception occurred inside thread (putting unpickable object) cause
feeder to finish running. After that every message put into queue is
silently ignored..
(cherry picked from commit bc50f03db4f58c869b78e98468e374d7e61f1227)

diff --git a/Lib/multiprocessing/queues.py b/Lib/multiprocessing/queues.py
index a88e298973ca8782779451eff5345f48310a184a..bb1e7282db8e3db2953039b8e63f9536480a45ca 100644 (file)
--- a/Lib/multiprocessing/queues.py
+++ b/Lib/multiprocessing/queues.py
@@ -244,8 +244,8 @@ class Queue(object):
         else:
             wacquire = None
 
-        try:
-            while 1:
+        while 1:
+            try:
                 nacquire()
                 try:
                     if not buffer:
@@ -270,19 +270,17 @@ class Queue(object):
                                 wrelease()
                 except IndexError:
                     pass
-        except Exception, e:
-            # Since this runs in a daemon thread the resources it uses
-            # may be become unusable while the process is cleaning up.
-            # We ignore errors which happen after the process has
-            # started to cleanup.
-            try:
+            except Exception as e:
+                # Since this runs in a daemon thread the resources it uses
+                # may be become unusable while the process is cleaning up.
+                # We ignore errors which happen after the process has
+                # started to cleanup.
                 if is_exiting():
                     info('error in queue thread: %s', e)
+                    return
                 else:
                     import traceback
                     traceback.print_exc()
-            except Exception:
-                pass
 
 _sentinel = object()
 

diff --git a/Lib/test/test_multiprocessing.py b/Lib/test/test_multiprocessing.py
index 4d39501ccb073ab1c4816ffb6c6c4df2b876a99e..8a6bb780f540ae0d895e2fe620962a2286829a8d 100644 (file)
--- a/Lib/test/test_multiprocessing.py
+++ b/Lib/test/test_multiprocessing.py
@@ -641,6 +641,21 @@ class _TestQueue(BaseTestCase):
                     self.fail("Probable regression on import lock contention;"
                               " see Issue #22853")
 
+    def test_queue_feeder_donot_stop_onexc(self):
+        # bpo-30414: verify feeder handles exceptions correctly
+        if self.TYPE != 'processes':
+            self.skipTest('test not appropriate for {}'.format(self.TYPE))
+
+        class NotSerializable(object):
+            def __reduce__(self):
+                raise AttributeError
+        with test.support.captured_stderr():
+            q = self.Queue()
+            q.put(NotSerializable())
+            q.put(True)
+            self.assertTrue(q.get(timeout=0.1))
+
+
 #
 #
 #

diff --git a/Misc/NEWS b/Misc/NEWS
index 3c6f829f051443975563cc0e46c38229e25421dc..ba5d5d17828890ea8d5089848900a411080b2254 100644 (file)
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -49,6 +49,9 @@ Extension Modules
 Library
 -------
 
+- bpo-30414: multiprocessing.Queue._feed background running
+  thread do not break from main loop on exception.
+
 - bpo-30003: Fix handling escape characters in HZ codec.  Based on patch
   by Ma Lin.