bpo-31394: Clarify documentation about token type attribute (GH-3469)

Make it clear that Ellipsis tokens also have type attribute set to token.OP.

diff --git a/Doc/library/tokenize.rst b/Doc/library/tokenize.rst
index cd27a101a8fef2c217bc839dca99a3698b213511..e6ddc0579217b0f27315060cdab2b3102ec080b1 100644 (file)
--- a/Doc/library/tokenize.rst
+++ b/Doc/library/tokenize.rst
@@ -16,8 +16,8 @@ implemented in Python.  The scanner in this module returns comments as tokens
 as well, making it useful for implementing "pretty-printers," including
 colorizers for on-screen displays.
 
-To simplify token stream handling, all :ref:`operators` and :ref:`delimiters`
-tokens are returned using the generic :data:`~token.OP` token type.  The exact
+To simplify token stream handling, all :ref:`operator <operators>` and :ref:`delimiter <delimiters>`
+tokens and :data:`Ellipsis` are returned using the generic :data:`~token.OP` token type.  The exact
 type can be determined by checking the ``exact_type`` property on the
 :term:`named tuple` returned from :func:`tokenize.tokenize`.