Remove unexpected code completion handling from ConsumeToken()

With this change tok::code_completion is finally handled exclusively as a
special token kind like other tokens that need special treatment.

All callers have been updated to use the specific token consumption methods and
the parser has a clear idea the current token isn't special by the time
ConsumeToken() gets called, so this has been unreachable for some time.

ConsumeAnyToken() behaviour is unchanged and will continue to support
unexpected code completion as part of the special token path.

This survived an amount of fuzzing and validation, but please ping the list if
you hit a code path that previously relied on the old unexpected handler and
now asserts.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@198942 91177308-0d34-0410-b5e6-96231b3b80d8

diff --git a/include/clang/Parse/Parser.h b/include/clang/Parse/Parser.h
index adb633a81da8117e6978edd1a04ecaad3ef16e67..b0dda6385d4723919e17bb334b5afb09dec547da 100644 (file)
--- a/include/clang/Parse/Parser.h
+++ b/include/clang/Parse/Parser.h
@@ -273,16 +273,12 @@ public:
   bool ParseTopLevelDecl(DeclGroupPtrTy &Result);
 
   /// ConsumeToken - Consume the current 'peek token' and lex the next one.
-  /// This does not work with all kinds of tokens: strings and specific other
-  /// tokens must be consumed with custom methods below.  This returns the
-  /// location of the consumed token.
+  /// This does not work with special tokens: string literals, code completion
+  /// and balanced tokens must be handled using the specific consume methods.
+  /// Returns the location of the consumed token.
   SourceLocation ConsumeToken() {
     assert(!isTokenSpecial() &&
            "Should consume special tokens with Consume*Token");
-
-    if (LLVM_UNLIKELY(Tok.is(tok::code_completion)))
-      return handleUnexpectedCodeCompletionToken();
-
     PrevTokLocation = Tok.getLocation();
     PP.Lex(Tok);
     return PrevTokLocation;
@@ -329,7 +325,7 @@ private:
   /// isTokenSpecial - True if this token requires special consumption methods.
   bool isTokenSpecial() const {
     return isTokenStringLiteral() || isTokenParen() || isTokenBracket() ||
-           isTokenBrace();
+           isTokenBrace() || Tok.is(tok::code_completion);
   }
 
   /// \brief Returns true if the current token is '=' or is a type of '='.