Reapply "Modules: Cache PCMs in memory and avoid a use-after-free"
This reverts commit r298185, effectively reapplying r298165, after fixing the
new unit tests (PR32338). The memory buffer generator doesn't null-terminate
the MemoryBuffer it creates; this version of the commit informs getMemBuffer
about that to avoid the assert.
Original commit message follows:
----
Clang's internal build system for implicit modules uses lock files to
ensure that after a process writes a PCM it will read the same one back
in (without contention from other -cc1 commands). Since PCMs are read
from disk repeatedly while invalidating, building, and importing, the
lock is not released quickly. Furthermore, the LockFileManager is not
robust in every environment. Other -cc1 commands can stall until
timeout (after about eight minutes).
This commit changes the lock file from being necessary for correctness
to a (possibly dubious) performance hack. The remaining benefit is to
reduce duplicate work in competing -cc1 commands which depend on the
same module. Follow-up commits will change the internal build system to
continue after a timeout, and reduce the timeout. Perhaps we should
reconsider blocking at all.
This also fixes a use-after-free, when one part of a compilation
validates a PCM and starts using it, and another tries to swap out the
PCM for something new.
The PCMCache is a new type called MemoryBufferCache, which saves memory
buffers based on their filename. Its ownership is shared by the
CompilerInstance and ModuleManager.
- The ModuleManager stores PCMs there that it loads from disk, never
touching the disk if the cache is hot.
- When modules fail to validate, they're removed from the cache.
- When a CompilerInstance is spawned to build a new module, each
already-loaded PCM is assumed to be valid, and is frozen to avoid
the use-after-free.
- Any newly-built module is written directly to the cache to avoid the
round-trip to the filesystem, making lock files unnecessary for
correctness.
Original patch by Manman Ren; most testcases by Adrian Prantl!
Aaron Ballman [Sun, 19 Mar 2017 15:15:28 +0000 (15:15 +0000)]
Cleaning up the IdentifierResolver::iterator class a bit; NFC.
The comment about there being three different forms that Ptr represents was stale. Also, the opaque value does not need to be exposed (these functions are unused).
[Modules] In case of lock timeout, fallback and build module
Duncan's r298165 introduced the PCMCache mechanism, which guarantees
that locks aren't necessary anymore for correctness but only for
performance, by avoiding building it twice when possible.
Change the logic to avoid an error but actually build the module in case
the timeout happens. Instead of an error, still emit a remark for
debugging purposes.
Modules: Cache PCMs in memory and avoid a use-after-free
Clang's internal build system for implicit modules uses lock files to
ensure that after a process writes a PCM it will read the same one back
in (without contention from other -cc1 commands). Since PCMs are read
from disk repeatedly while invalidating, building, and importing, the
lock is not released quickly. Furthermore, the LockFileManager is not
robust in every environment. Other -cc1 commands can stall until
timeout (after about eight minutes).
This commit changes the lock file from being necessary for correctness
to a (possibly dubious) performance hack. The remaining benefit is to
reduce duplicate work in competing -cc1 commands which depend on the
same module. Follow-up commits will change the internal build system to
continue after a timeout, and reduce the timeout. Perhaps we should
reconsider blocking at all.
This also fixes a use-after-free, when one part of a compilation
validates a PCM and starts using it, and another tries to swap out the
PCM for something new.
The PCMCache is a new type called MemoryBufferCache, which saves memory
buffers based on their filename. Its ownership is shared by the
CompilerInstance and ModuleManager.
- The ModuleManager stores PCMs there that it loads from disk, never
touching the disk if the cache is hot.
- When modules fail to validate, they're removed from the cache.
- When a CompilerInstance is spawned to build a new module, each
already-loaded PCM is assumed to be valid, and is frozen to avoid
the use-after-free.
- Any newly-built module is written directly to the cache to avoid the
round-trip to the filesystem, making lock files unnecessary for
correctness.
Original patch by Manman Ren; most testcases by Adrian Prantl!
Matthias Gehre [Fri, 17 Mar 2017 21:41:20 +0000 (21:41 +0000)]
Implement DR 373 "Lookup on namespace qualified name in using-directive"
Summary:
3.4.6 [basic.lookup.udir] paragraph 1:
In a using-directive or namespace-alias-definition, during the lookup for a namespace-name or for a name in a nested-name-specifier, only namespace names are considered.
Zachary Turner [Fri, 17 Mar 2017 16:24:34 +0000 (16:24 +0000)]
[clang-cl] Fix cross-compilation with MSVC 2017.
clang-cl works best when the user runs vcvarsall to set up
an environment before running, but even this is not enough
on VC 2017 when cross compiling (e.g. using an x64 toolchain
to target x86, or vice versa).
The reason is that although clang-cl itself will have a
valid environment, it will shell out to other tools (such
as link.exe) which may not. Generally we solve this through
adding the appropriate linker flags, but this is not enough
in VC 2017.
The cross-linker and the regular linker both link against
some common DLLs, but these DLLs live in the binary directory
of the native linker. When setting up a cross-compilation
environment through vcvarsall, it will add *both* directories
to %PATH%, so that when cl shells out to any of the associated
tools, those tools will be able to find all of the dependencies
that it links against. If you don't do this, link.exe will
fail to run because the loader won't be able to find all of
the required DLLs that it links against.
To solve this we teach the driver how to spawn a process with
an explicitly specified environment. Then we modify the
PATH before shelling out to subtools and run with the modified
PATH.
Patch by Hamza Sood
Differential Revision: https://reviews.llvm.org/D30991
Craig Topper [Fri, 17 Mar 2017 05:14:37 +0000 (05:14 +0000)]
[AVX-512] Add missing typecasts and parentheses to _mm512_mask_i64gather_ps. My macro cleanup script I used on the others last year must have missed it.
Eric Christopher [Wed, 15 Mar 2017 23:41:58 +0000 (23:41 +0000)]
Fix up testcase to:
a) avoid actually compiling anything
b) avoid depositing files in the test directory
c) use a check line to make sure that we're looking for the correct behavior
Akira Hatanaka [Wed, 15 Mar 2017 18:04:13 +0000 (18:04 +0000)]
[Driver] Define macro __APPLE_EMBEDDED_SIMULATOR__ when option
-m(i|tv|watch)os-simulator-version-min is on the command line.
Previously the driver would treat -m(i|tv|watch)os-simulator-version-min
as an alias of -m(i|tv|watch)os-version-min. This no longer works since
we now need to distinguish between the two options (the latter is used
for iOS running in a VM, for example).
This commit stops making the simulator options the aliases of the OS
options and defines a macro to differentiate between the two groups of
options.
Alex Lorenz [Wed, 15 Mar 2017 17:16:41 +0000 (17:16 +0000)]
[ObjC][Sema] Avoid warning about a call to an instance method on an
instance of a qualified Class object when that instance method comes from
a protocol that's implemented by NSObject
Instance methods from a root class like NSObject are also class methods because
the metaclass of root class derives from that root class. Therefore, we can
avoid the warning for instances of qualified Class objects that point to classes
that derive from NSObject. Note that we actually don't know if a Class instance
points to a class that derives from NSObject at compile-time, so we have to
make a reasonable assumption that the majority of instances will do so.
Zachary Turner [Wed, 15 Mar 2017 16:35:13 +0000 (16:35 +0000)]
Disable warning about MSVC not found.
When this test runs on bots that are configured to default
to MSVC, but MSVC isn't actually installed, we can emit a
warning that MSVC is not found. Since MSVC isn't actually
needed for this test to succeed, just disable this warning.
Zachary Turner [Wed, 15 Mar 2017 16:07:35 +0000 (16:07 +0000)]
Update clang-cl driver for MSVC 2017.
2017 changes the way you find an installed copy of
Visual Studio as well as its internal directory layout.
As a result, clang-cl was unable to find VS2017 even
when you had run vcvarsall to set up a toolchain
environment. This patch updates everything for 2017
and cleans up the way we handle a tiered search a la
environment -> installation -> PATH for which copy
of Visual Studio to bind to.
Patch originally by Hamza Sood, with some fixups for landing.
John Brawn [Wed, 15 Mar 2017 14:03:32 +0000 (14:03 +0000)]
[Driver] Restructure handling of -ffast-math and similar options
The way -ffast-math and the various related options to tweak floating-point
handling are handled is inflexible and rather confusing. This patch restructures
things so that we go through the options adjusting our idea of what's enabled as
we go, instead of trying to figure each individual thing out by working
backwards from the end, as this makes the behaviour of each individual option
more clear.
Doing it this way also means we get gcc-compatible behaviour for when the
__FAST_MATH__ and __FINITE_MATH_ONLY__ macros are defined, as they should depend
on the final set of features that are enabled and not just on -ffast-math and
-ffinite-math-only specifically.
Modules: Optimize bitcode encoding of diagnostic state
Since bitcode uses VBR encoding, large numbers are more expensive than
small ones. Instead of emitting a UINT_MAX sentinel after each sequence
of state-change pairs, emit the size of the sequence as a prefix.
This should have no functionality change besides saving bits from the
encoding.
Reid Kleckner [Tue, 14 Mar 2017 18:01:02 +0000 (18:01 +0000)]
Warn on enum assignment to bitfields that can't fit all values
This adds -Wbitfield-enum-conversion, which warns on implicit
conversions that happen on bitfield assignment that change the value of
some enumerators.
Values of enum type typically take on a very small range of values, so
they are frequently stored in bitfields. Unfortunately, there is no
convenient way to calculate the minimum number of bits necessary to
store all possible values at compile time, so users usually hard code a
bitwidth that works today and widen it as necessary to pass basic
testing and validation. This is very error-prone, and leads to stale
widths as enums grow. This warning aims to catch such bugs.
This would have found two real bugs in clang and two instances of
questionable code. See r297680 and r297654 for the full description of
the issues.
This warning is currently disabled by default while we investigate its
usefulness outside of LLVM.
The major cause of false positives with this warning is this kind of
enum:
enum E { W, X, Y, Z, SENTINEL_LAST };
The last enumerator is an invalid value used to validate inputs or size
an array. Depending on the prevalance of this style of enum across a
codebase, this warning may be more or less feasible to deploy. It also
has trouble on sentinel values such as ~0U.
[Driver] Add flag to request arch-specific-subdir in -rpath
Summary:
This patch adds -f[no-]rtlib-add-rpath, which if enabled, embeds the
arch-specific subdirectory in resource directory using -rpath (instead
of doing so only during native compilation).
This patch also re-enables test arch-specific-libdir.c which was
silently unsupported because of the REQUIRES tag 'linux'.
Vedant Kumar [Tue, 14 Mar 2017 01:56:34 +0000 (01:56 +0000)]
[ubsan] Add a nullability sanitizer
Teach UBSan to detect when a value with the _Nonnull type annotation
assumes a null value. Call expressions, initializers, assignments, and
return statements are all checked.
Because _Nonnull does not affect IRGen, the new checks are disabled by
default. The new driver flags are:
-fsanitize=nullability-arg (_Nonnull violation in call)
-fsanitize=nullability-assign (_Nonnull violation in assignment)
-fsanitize=nullability-return (_Nonnull violation in return stmt)
-fsanitize=nullability (all of the above)
This patch builds on top of UBSan's existing support for detecting
violations of the nonnull attributes ('nonnull' and 'returns_nonnull'),
and relies on the compiler-rt support for those checks. Eventually we
will need to update the diagnostic messages in compiler-rt (there are
FIXME's for this, which will be addressed in a follow-up).
One point of note is that the nullability-return check is only allowed
to kick in if all arguments to the function satisfy their nullability
preconditions. This makes it necessary to emit some null checks in the
function body itself.
Testing: check-clang and check-ubsan. I also built some Apple ObjC
frameworks with an asserts-enabled compiler, and verified that we get
valid reports.
Juergen Ributzka [Tue, 14 Mar 2017 00:14:40 +0000 (00:14 +0000)]
Reapply [VFS] Ignore broken symlinks in the directory iterator.
Modified the tests to accept any iteration order, to run only on Unix, and added
additional error reporting to investigate SystemZ bot issue.
The VFS directory iterator and recursive directory iterator behave differently
from the LLVM counterparts. Once the VFS iterators hit a broken symlink they
immediately abort. The LLVM counterparts don't stat entries unless they have to
descend into the next directory, which allows to recover from this issue by
clearing the error code and skipping to the next entry.
This change adds similar behavior to the VFS iterators. There should be no
change in current behavior in the current CLANG source base, because all
clients have loop exit conditions that also check the error code.
Reid Kleckner [Mon, 13 Mar 2017 22:33:04 +0000 (22:33 +0000)]
Widen AST bitfields too small to represent all enumerators
All of these were found by a new warning that I am prototyping,
-Wbitfield-enum-conversion.
Stmt::ExprBits::ObjectKind - This was not wide enough to represent
OK_ObjSubscript, so this was a real, true positive bug.
ObjCDeclSpec::objCDeclQualifier - On Windows, setting DQ_CSNullability
would cause the bitfield to become negative because enum types are
always implicitly 'int' there. This would probably never be noticed
because this is a flag-style enum, so we only ever test one bit at a
time. Switching to 'unsigned' also makes this type pack smaller on
Windows.
FunctionDecl::SClass - Technically, we only need two bits for all valid
function storage classes. Functions can never have automatic or register
storage class. This seems a bit too clever, and we have a bit to spare,
so widening the bitfield seems like the best way to pacify the warning.
You could classify this as a false positive, but widening the bitfield
defends us from invalid ASTs.
Matt Arsenault [Mon, 13 Mar 2017 19:47:53 +0000 (19:47 +0000)]
AMDGPU: Make 0 the private nullptr value
We can't actually pretend that 0 is valid for address space 0.
r295877 added a workaround to stop allocating user objects
there, so we can use 0 as the invalid pointer.
Some of the tests seemed to be using private as the non-0 null
test address space, so add copies using local to make sure
this is still stressed.
Change ASTFileSignature from a random 32-bit number to the hash of the
PCM content.
- Move definition ASTFileSignature to Basic/Module.h so Module and
ASTSourceDescriptor can use it.
- Change the signature from uint64_t to std::array<uint32_t,5>.
- Stop using (saving/reading) the size and modification time of PCM
files when there is a valid SIGNATURE.
- Add UNHASHED_CONTROL_BLOCK, and use it to store the SIGNATURE record
and other records that shouldn't affect the hash. Because implicit
modules reuses the same file for multiple levels of -Werror, this
includes DIAGNOSTIC_OPTIONS and DIAG_PRAGMA_MAPPINGS.
This helps to solve a PCH + implicit Modules dependency issue: PCH files
are handled by the external build system, whereas implicit modules are
handled by internal compiler build system. This prevents invalidating a
PCH when the compiler overwrites a PCM file with the same content
(modulo the diagnostic differences).
Craig Topper [Mon, 13 Mar 2017 17:16:50 +0000 (17:16 +0000)]
[X86] Add checking of the scale argument to scatter/gather builtins
The only valid values for scale immediate of scatter/gather builtins are 1, 2, 4, or 8. This patch enforces this in the frontend otherwise we generate invalid instruction encodings in the backend.
When diagnosing taking address of packed members skip __unaligned-qualified expressions
Given that we have already explicitly stated in the qualifier that the
expression is __unaligned, it makes little sense to diagnose that the address
of the packed member may not be aligned.
Alex Lorenz [Mon, 13 Mar 2017 11:17:41 +0000 (11:17 +0000)]
Add -iframeworkwithsysroot compiler option
This commit adds support for a new -iframeworkwithsysroot compiler option which
allows the user to specify a framework path that can be prefixed with the
sysroot. This option is similar to the -iwithsysroot option that exists to
supplement -isystem.
Martin Probst [Mon, 13 Mar 2017 09:39:23 +0000 (09:39 +0000)]
clang-format: [JS] do not wrap @see tags.
Summary:
@see is special among JSDoc tags in that it is commonly followed by URLs. The JSDoc spec suggests that users should wrap URLs in an additional {@link url...} tag (@see http://usejsdoc.org/tags-see.html), but this is very commonly violated, with @see being followed by a "naked" URL.
This change special cases all JSDoc lines that contain an @see not to be wrapped to account for that.
Martin Probst [Mon, 13 Mar 2017 09:14:23 +0000 (09:14 +0000)]
clang-format: [JS] allow breaking after non-null assertions.
Summary:
Previously clang-format would not break after any !. However in TypeScript, ! can be used as a post fix operator for non-nullability:
x.foo()!.bar()!;
With this change, clang-format will wrap after the ! if it is likely a post-fix non null operator.
Sanjay Patel [Sun, 12 Mar 2017 19:15:10 +0000 (19:15 +0000)]
[x86] these aren't the undefs you're looking for (PR32176)
x86 has undef SSE/AVX intrinsics that should represent a bogus register operand.
This is not the same as LLVM's undef value which can take on multiple bit patterns.
There are better solutions / follow-ups to this discussed here:
https://bugs.llvm.org/show_bug.cgi?id=32176
...but this should prevent miscompiles with a one-line code change.
Eric Fiselier [Sat, 11 Mar 2017 02:35:37 +0000 (02:35 +0000)]
[coroutines] Fix diagnostics depending on the first coroutine statement.
Summary:
Some coroutine diagnostics need to point to the location of the first coroutine keyword in the function, like when diagnosing a `return` inside a coroutine. Previously we did this by storing each *valid* coroutine statement in a list and select the first one to use in diagnostics. However if every coroutine statement is invalid we would have no location to point to.
This patch fixes the storage of the first coroutine statement location, ensuring that it gets stored even when the resulting AST node would be invalid.
This patch also removes the `CoroutineStmts` list in `FunctionScopeInfo` because it was unused.
Gor Nishanov [Sat, 11 Mar 2017 01:30:17 +0000 (01:30 +0000)]
[coroutines] Refactor SuspendExpr to create just one OpaqueValue (almost NFC)
Summary:
Create only one OpaqueValue for await_ready/await_suspend/await_resume.
Store OpaqueValue used in the CoroutineSuspendExpr node, so that CodeGen does not have to hunt looking for it.
Juergen Ributzka [Fri, 10 Mar 2017 22:49:04 +0000 (22:49 +0000)]
Reapply [VFS] Ignore broken symlinks in the directory iterator.
Modified the tests to accept any iteration order.
The VFS directory iterator and recursive directory iterator behave differently
from the LLVM counterparts. Once the VFS iterators hit a broken symlink they
immediately abort. The LLVM counterparts allow to recover from this issue by
clearing the error code and skipping to the next entry.
This change adds the same functionality to the VFS iterators. There should be
no change in current behavior in the current CLANG source base, because all
clients have loop exit conditions that also check the error code.
Juergen Ributzka [Fri, 10 Mar 2017 21:23:27 +0000 (21:23 +0000)]
[VFS] Ignore broken symlinks in the directory iterator.
The VFS directory iterator and recursive directory iterator behave differently
from the LLVM counterparts. Once the VFS iterators hit a broken symlink they
immediately abort. The LLVM counterparts allow to recover from this issue by
clearing the error code and skipping to the next entry.
This change adds the same functionality to the VFS iterators. There should be
no change in current behavior in the current CLANG source base, because all
clients have loop exit conditions that also check the error code.
projects / clang / log