[lit] Protect full test suite from FILECHECK_OPTS

lit's test suite calls lit multiple times for various sample test
suites.  `FILECHECK_OPTS` is safe for FileCheck calls in lit's test
suite.  It's not safe for FileCheck calls in the sample test suites,
whose output affects the results of lit's test suite.

Without this patch, only one such sample test suite is protected from
`FILECHECK_OPTS`, and I admit I haven't discovered other cases for
which I can produce false failures using `FILECHECK_OPTS`.  However,
it's hard to predict the future, especially false passes.  Thus, this
patch protects all existing and future sample test suites from
`FILECHECK_OPTS` (and the deprecated
`FILECHECK_DUMP_INPUT_ON_FAILURE`).

Reviewed By: probinson

Differential Revision: https://reviews.llvm.org/D65156

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@366980 91177308-0d34-0410-b5e6-96231b3b80d8

diff --git a/utils/lit/tests/lit.cfg b/utils/lit/tests/lit.cfg
index ebdcb5000b9a2fd6944022c45b2ff0e68d3d1229..2830956f80dd26c25eff5f252ff37cd4fd4fbe47 100644 (file)
--- a/utils/lit/tests/lit.cfg
+++ b/utils/lit/tests/lit.cfg
@@ -44,10 +44,17 @@ for attribute in ('llvm_tools_dir', 'lit_tools_dir'):
     if directory:
         llvm_config.with_environment('PATH', directory, append_path=True)
 
+# This test suite calls %{lit} to test lit's behavior for the sample test
+# suites in %{inputs}.  This test suite's results are then determined in part
+# by %{lit}'s textual output, which includes the output of FileCheck calls
+# within %{inputs}'s test suites.  Thus, %{lit} clears environment variables
+# that can affect FileCheck's output.
 config.substitutions.append(('%{inputs}', os.path.join(
     config.test_source_root, 'Inputs')))
-config.substitutions.append(('%{lit}', "%%{python} %s" % (
-    os.path.join(lit_path, 'lit.py'),)))
+config.substitutions.append(('%{lit}',
+    "{env} %{{python}} {lit}".format(
+        env="env -u FILECHECK_OPTS -u FILECHECK_DUMP_INPUT_ON_FAILURE",
+        lit=os.path.join(lit_path, 'lit.py'))))
 config.substitutions.append(('%{python}', '"%s"' % (sys.executable)))
 
 # Enable coverage.py reporting, assuming the coverage module has been installed

diff --git a/utils/lit/tests/shtest-run-at-line.py b/utils/lit/tests/shtest-run-at-line.py
index 7e5d53b8e26ceea37a1c6c1e4fa135c18469cae6..cd0e08137ee589b841df478f335fb9e16ae39ddb 100644 (file)
--- a/utils/lit/tests/shtest-run-at-line.py
+++ b/utils/lit/tests/shtest-run-at-line.py
@@ -1,7 +1,7 @@
 # Check that -vv makes the line number of the failing RUN command clear.
 # (-v is actually sufficient in the case of the internal shell.)
 #
-# RUN: env -u FILECHECK_OPTS not %{lit} -j 1 -vv %{inputs}/shtest-run-at-line > %t.out
+# RUN: not %{lit} -j 1 -vv %{inputs}/shtest-run-at-line > %t.out
 # RUN: FileCheck --input-file %t.out %s
 #
 # END.