+<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN"
+ "http://www.w3.org/TR/html4/strict.dtd">
<html>
<head>
<title>Pretokenized Headers (PTH)</title>
- <link type="text/css" rel="stylesheet" href="../menu.css" />
- <link type="text/css" rel="stylesheet" href="../content.css" />
+ <link type="text/css" rel="stylesheet" href="../menu.css">
+ <link type="text/css" rel="stylesheet" href="../content.css">
<style type="text/css">
td {
vertical-align: top;
<!--#include virtual="../menu.html.incl"-->
<div id="content">
-<h1>Pretokenized Headers</h1>
-
-<p> <a href="http://en.wikipedia.org/wiki/Precompiled_header">Precompiled
-headers</a> is a general approach employed by many compilers to reduce
-compilation time. The underlying motivation of the approach is that within a
-codebase frequently the same (and often large) header files are included by
-multiple source files. Consequently, compile times can often be greatly improved
-by caching some of the (redundant) work done by a compiler to process headers.
-Precompiled header files, which represent one of possibly many ways to implement
-this optimization, are literally files that represent an on-disk cache that
-contains the vital information necessary to reduce some (or all) of the work
-needed to process a corresponding header file. While details of precompiled
-headers vary between compilers, precompiled headers have been shown to be a
-highly effective at speeding up program compilation on systems with very large
-system headers (e.g., Mac OS X).</p>
-
-<p>Clang supports an implementation of precompiled headers known as
-<em>pre-tokenized headers</em> (PTH). Clang's pre-tokenized headers support most
-of same interfaces as GCC's pre-compiled headers (as well as others) but are
-completely different in their implementation. This pages first describes the
-interface for using PTH and then briefly elaborates on their design and
-implementation.</p>
-
-
-<h2>Using Pretokenized Headers (High-level Interface)</h2>
-
-<p>The high-level interface to generate a PTH file is the same as GCC's:</p>
-<pre>
- $ gcc -x c-header test.h -o test.h.gch
- $ clang -x c-header test.h -o test.h.pth
-</pre>
-
-<p>A PTH file can then be used as a prefix header when a <tt>-include</tt>
-option is passed to <tt>clang</tt>:</p>
+<h1>Pretokenized Headers (PTH)</h1>
-<pre>
- $ clang -include test.h test.c -o test
-</pre>
+<p>This document first describes the low-level
+interface for using PTH and then briefly elaborates on its design and
+implementation. If you are interested in the end-user view, please see the
+<a href="UsersManual.html#precompiledheaders">User's Manual</a>.</p>
-<p>The <tt>clang</tt> driver will first check if a PTH file for <tt>test.h</tt>
-is available; if so, the contents of <tt>test.h</tt> (and the files it includes)
-will be processed from the PTH file. Otherwise, <tt>clang</tt> falls back to
-directly processing the content of <tt>test.h</tt>. This mirrors the behavior of
-GCC.</p>
-<p><b>NOTE:</b> <tt>clang</tt> does <em>not</em> automatically used PTH files
-for headers that are directly included within a source file. For example:</p>
+<h2>Using Pretokenized Headers with <tt>clang</tt> (Low-level Interface)</h2>
-<pre>
- $ clang -x c-header test.h -o test.h.pth
- $ cat test.c
- #include "test.h"
- $ clang test.c -o test
-</pre>
-
-<p>In this example, <tt>clang</tt> will not automatically use the PTH file for
-<tt>test.h</tt> since <tt>test.h</tt> was included directly in the source file
-and not specified on the command line using <tt>-include</tt>.</p>
-
-<h2>Using Pretokenized Headers (Low-level Interface)</h2>
-
-<p>The low-level Clang driver, <tt>clang-cc</tt>, supports three command line
+<p>The Clang compiler frontend, <tt>clang -cc1</tt>, supports three command line
options for generating and using PTH files.<p>
-<p>To generate PTH files using <tt>clang-cc</tt>, use the option <tt>-emit-pth</tt>:
-
-<pre>
- $ clang-cc test.h -emit-pth -o test.h.pth
-</pre>
+<p>To generate PTH files using <tt>clang -cc1</tt>, use the option
+<b><tt>-emit-pth</tt></b>:
+
+<pre> $ clang -cc1 test.h -emit-pth -o test.h.pth </pre>
<p>This option is transparently used by <tt>clang</tt> when generating PTH
-files. Similarly, PTH files can be used as prefix headers using the <tt>-include-pth</tt> option:</p>
+files. Similarly, PTH files can be used as prefix headers using the
+<b><tt>-include-pth</tt></b> option:</p>
<pre>
- $ clang-cc -include-pth test.h.pth test.c -o test.s
+ $ clang -cc1 -include-pth test.h.pth test.c -o test.s
</pre>
<p>Alternatively, Clang's PTH files can be used as a raw "token-cache"
(or "content" cache) of the source included by the original header
file. This means that the contents of the PTH file are searched as substitutes
-for <em>any</em> source files that are used by <tt>clang-cc</tt> to process a
-source file. This is done by specifying the <tt>-token-cache</tt> option:</p>
+for <em>any</em> source files that are used by <tt>clang -cc1</tt> to process a
+source file. This is done by specifying the <b><tt>-token-cache</tt></b>
+option:</p>
<pre>
$ cat test.h
- #include<stdio.h>
- $ clang-cc -emit-pth test.h -o test.h.pth
+ #include <stdio.h>
+ $ clang -cc1 -emit-pth test.h -o test.h.pth
$ cat test.c
#include "test.h"
- $ clang-cc test.c -o test -token-cache test.h.pth
+ $ clang -cc1 test.c -o test -token-cache test.h.pth
</pre>
<p>In this example the contents of <tt>stdio.h</tt> (and the files it includes)
header files was motivated by the following factors:<p>
<ul>
-<li><p><em>Language independence</em>: PTH files are (roughly) language
-independent. They work with any language that Clang's lexer can handle,
-including C, Objective-C, and (in the early stages) C++. This means development
-on language features at the parsing level or above (which is basically almost
-all interesting pieces) does not require PTH to be modified.</p></li>
-<li><em>Simple design</em>: Relatively speaking, PTH has a simple design and
+<li><p><b>Language independence</b>: PTH files work with any language that
+Clang's lexer can handle, including C, Objective-C, and (in the early stages)
+C++. This means development on language features at the parsing level or above
+(which is basically almost all interesting pieces) does not require PTH to be
+modified.</p></li>
+
+<li><b>Simple design</b>: Relatively speaking, PTH has a simple design and
implementation, making it easy to test. Further, because the machinery for PTH
resides at the lower-levels of the Clang library stack it is fairly
straightforward to profile and optimize.</li>
<ul>
-<li><p><em>Architecture independence</em>: In contrast to GCC's PCH files (and
+<li><p><b>Architecture independence</b>: In contrast to GCC's PCH files (and
those of several other compilers), Clang's PTH files are architecture
independent, requiring only a single PTH file when building an program for
multiple architectures.</p>
multiple architectures. This can also reduce compile times because only a single
PTH file needs to be generated during a build instead of several.</p></li>
-<li><p><em>Reduced memory pressure</em>: Similar to GCC,
+<li><p><b>Reduced memory pressure</b>: Similar to GCC,
Clang reads PTH files via the use of memory mapping (i.e., <tt>mmap</tt>).
Clang, however, memory maps PTH files as read-only, meaning that multiple
-invocations of <tt>clang-cc</tt> can share the same pages in memory from a
+invocations of <tt>clang -cc1</tt> can share the same pages in memory from a
memory-mapped PTH file. In comparison, GCC also memory maps its PCH files but
also modifies those pages in memory, incurring the copy-on-write costs. The
read-only nature of PTH can greatly reduce memory pressure for builds involving
multiple cores, thus improving overall scalability.</p></li>
+<li><p><b>Fast generation</b>: PTH files can be generated in a small fraction
+of the time needed to generate GCC's PCH files. Since PTH/PCH generation is a
+serial operation that typically blocks progress during a build, faster
+generation time leads to improved processor utilization with parallel builds on
+multicore machines.</p></li>
+
</ul>
<p>Despite these strengths, PTH's simple design suffers some algorithmic
fundamentally dominate at an algorithmic level, especially when one considers
header files of arbitrary size.</p>
-<p>Consequently, as alluded earlier, there are plans to potentially implement an
-alternative PCH implementation for Clang based on the lazy deserialization of
-ASTs. This approach would theoretically have the same constant-time algorithmic
-advantages just mentioned but would also retain some of the strengths of PTH
-such as reduced memory pressure (ideal for multi-core builds).</p>
+<p>There are plans to potentially implement an complementary PCH implementation
+for Clang based on the lazy deserialization of ASTs. This approach would
+theoretically have the same constant-time algorithmic advantages just mentioned
+but would also retain some of the strengths of PTH such as reduced memory
+pressure (ideal for multi-core builds).</p>
<h3>Internal PTH Optimizations</h3>
<ul>
<li><p><em><tt>stat</tt> caching</em>: PTH files cache information obtained via
-calls to <tt>stat</tt> that <tt>clang-cc</tt> uses to resolve which files are
+calls to <tt>stat</tt> that <tt>clang -cc1</tt> uses to resolve which files are
included by <tt>#include</tt> directives. This greatly reduces the overhead
involved in context-switching to the kernel to resolve included files.</p></li>
projects / clang / blobdiff