[3.6] bpo-33899: Make tokenize module mirror end-of-file is end-of-line behavior...

[3.6] bpo-33899: Make tokenize module mirror end-of-file is end-of-line behavior (GH-7891) (GH-8134)

Most of the change involves fixing up the test suite, which previously made
the assumption that there wouldn't be a new line if the input didn't end in
one.

Contributed by Ammar Askar.

(cherry picked from commit c4ef4896eac86a6759901c8546e26de4695a1389)