From: Berker Peksag <berker.peksag@gmail.com>
Date: Tue, 29 Dec 2015 23:42:43 +0000 (+0200)
Subject: Issue #25977: Fix typos in Lib/tokenize.py
X-Git-Tag: v3.6.0a1~839
X-Git-Url: https://granicus.if.org/sourcecode?a=commitdiff_plain;h=a7161e7facdfa1d6f673beb16a95a647ce764b32;p=python

Issue #25977: Fix typos in Lib/tokenize.py

Patch by John Walker.
---

a7161e7facdfa1d6f673beb16a95a647ce764b32
diff --cc Lib/tokenize.py
index 2237c3a554,9fd676c5b2..7a003580a4
--- a/Lib/tokenize.py
+++ b/Lib/tokenize.py
@@@ -636,31 -638,14 +636,31 @@@ def _tokenize(readline, encoding)
                          contstr = line[start:]
                          contline = line
                          break
 -                elif initial in single_quoted or \
 -                    token[:2] in single_quoted or \
 -                    token[:3] in single_quoted:
 +
 +                # Check up to the first 3 chars of the token to see if
 +                #  they're in the single_quoted set. If so, they start
 +                #  a string.
 +                # We're using the first 3, because we're looking for
 +                #  "rb'" (for example) at the start of the token. If
 +                #  we switch to longer prefixes, this needs to be
 +                #  adjusted.
 +                # Note that initial == token[:1].
-                 # Also note that single quote checking must come afer
++                # Also note that single quote checking must come after
 +                #  triple quote checking (above).
 +                elif (initial in single_quoted or
 +                      token[:2] in single_quoted or
 +                      token[:3] in single_quoted):
                      if token[-1] == '\n':                  # continued string
                          strstart = (lnum, start)
 -                        endprog = _compile(endpats[initial] or
 -                                           endpats[token[1]] or
 -                                           endpats[token[2]])
 +                        # Again, using the first 3 chars of the
 +                        #  token. This is looking for the matching end
 +                        #  regex for the correct type of quote
 +                        #  character. So it's really looking for
 +                        #  endpats["'"] or endpats['"'], by trying to
 +                        #  skip string prefix characters, if any.
 +                        endprog = _compile(endpats.get(initial) or
 +                                           endpats.get(token[1]) or
 +                                           endpats.get(token[2]))
                          contstr, needcont = line[start:], 1
                          contline = line
                          break