1 <?xml version="1.0" encoding="ISO-8859-1"?>
2 <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
3 <html xmlns="http://www.w3.org/1999/xhtml" lang="en" xml:lang="en"><head><!--
4 XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
5 This file is generated from xml source: DO NOT EDIT
6 XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
8 <title>Apache Performance Tuning - Apache HTTP Server</title>
9 <link href="../style/css/manual.css" rel="stylesheet" media="all" type="text/css" title="Main stylesheet" />
10 <link href="../style/css/manual-loose-100pc.css" rel="alternate stylesheet" media="all" type="text/css" title="No Sidebar - Default font size" />
11 <link href="../style/css/manual-print.css" rel="stylesheet" media="print" type="text/css" />
12 <link href="../images/favicon.ico" rel="shortcut icon" /></head>
13 <body id="manual-page"><div id="page-header">
14 <p class="menu"><a href="../mod/">Modules</a> | <a href="../mod/directives.html">Directives</a> | <a href="../faq/">FAQ</a> | <a href="../glossary.html">Glossary</a> | <a href="../sitemap.html">Sitemap</a></p>
15 <p class="apache">Apache HTTP Server Version 2.3</p>
16 <img alt="" src="../images/feather.gif" /></div>
17 <div class="up"><a href="./"><img title="<-" alt="<-" src="../images/left.gif" /></a></div>
19 <a href="http://www.apache.org/">Apache</a> > <a href="http://httpd.apache.org/">HTTP Server</a> > <a href="http://httpd.apache.org/docs/">Documentation</a> > <a href="../">Version 2.3</a> > <a href="./">Miscellaneous Documentation</a></div><div id="page-content"><div id="preamble"><h1>Apache Performance Tuning</h1>
21 <p><span>Available Languages: </span><a href="../en/misc/perf-tuning.html" title="English"> en </a> |
22 <a href="../fr/misc/perf-tuning.html" hreflang="fr" rel="alternate" title="Français"> fr </a> |
23 <a href="../ko/misc/perf-tuning.html" hreflang="ko" rel="alternate" title="Korean"> ko </a> |
24 <a href="../tr/misc/perf-tuning.html" hreflang="tr" rel="alternate" title="Türkçe"> tr </a></p>
28 <p>Apache 2.x is a general-purpose webserver, designed to
29 provide a balance of flexibility, portability, and performance.
30 Although it has not been designed specifically to set benchmark
31 records, Apache 2.x is capable of high performance in many
32 real-world situations.</p>
34 <p>Compared to Apache 1.3, release 2.x contains many additional
35 optimizations to increase throughput and scalability. Most of
36 these improvements are enabled by default. However, there are
37 compile-time and run-time configuration choices that can
38 significantly affect performance. This document describes the
39 options that a server administrator can configure to tune the
40 performance of an Apache 2.x installation. Some of these
41 configuration options enable the httpd to better take advantage
42 of the capabilities of the hardware and OS, while others allow
43 the administrator to trade functionality for speed.</p>
46 <div id="quickview"><ul id="toc"><li><img alt="" src="../images/down.gif" /> <a href="#hardware">Hardware and Operating System Issues</a></li>
47 <li><img alt="" src="../images/down.gif" /> <a href="#runtime">Run-Time Configuration Issues</a></li>
48 <li><img alt="" src="../images/down.gif" /> <a href="#compiletime">Compile-Time Configuration Issues</a></li>
49 <li><img alt="" src="../images/down.gif" /> <a href="#trace">Appendix: Detailed Analysis of a Trace</a></li>
51 <div class="top"><a href="#page-header"><img alt="top" src="../images/up.gif" /></a></div>
53 <h2><a name="hardware" id="hardware">Hardware and Operating System Issues</a></h2>
57 <p>The single biggest hardware issue affecting webserver
58 performance is RAM. A webserver should never ever have to swap,
59 as swapping increases the latency of each request beyond a point
60 that users consider "fast enough". This causes users to hit
61 stop and reload, further increasing the load. You can, and
62 should, control the <code class="directive"><a href="../mod/mpm_common.html#maxclients">MaxClients</a></code> setting so that your server
63 does not spawn so many children it starts swapping. This procedure
64 for doing this is simple: determine the size of your average Apache
65 process, by looking at your process list via a tool such as
66 <code>top</code>, and divide this into your total available memory,
67 leaving some room for other processes.</p>
69 <p>Beyond that the rest is mundane: get a fast enough CPU, a
70 fast enough network card, and fast enough disks, where "fast
71 enough" is something that needs to be determined by
74 <p>Operating system choice is largely a matter of local
75 concerns. But some guidelines that have proven generally
80 <p>Run the latest stable release and patchlevel of the
81 operating system that you choose. Many OS suppliers have
82 introduced significant performance improvements to their
83 TCP stacks and thread libraries in recent years.</p>
87 <p>If your OS supports a <code>sendfile(2)</code> system
88 call, make sure you install the release and/or patches
89 needed to enable it. (With Linux, for example, this means
90 using Linux 2.4 or later. For early releases of Solaris 8,
91 you may need to apply a patch.) On systems where it is
92 available, <code>sendfile</code> enables Apache 2 to deliver
93 static content faster and with lower CPU utilization.</p>
97 </div><div class="top"><a href="#page-header"><img alt="top" src="../images/up.gif" /></a></div>
99 <h2><a name="runtime" id="runtime">Run-Time Configuration Issues</a></h2>
103 <table class="related"><tr><th>Related Modules</th><th>Related Directives</th></tr><tr><td><ul><li><code class="module"><a href="../mod/mod_dir.html">mod_dir</a></code></li><li><code class="module"><a href="../mod/mpm_common.html">mpm_common</a></code></li><li><code class="module"><a href="../mod/mod_status.html">mod_status</a></code></li></ul></td><td><ul><li><code class="directive"><a href="../mod/core.html#allowoverride">AllowOverride</a></code></li><li><code class="directive"><a href="../mod/mod_dir.html#directoryindex">DirectoryIndex</a></code></li><li><code class="directive"><a href="../mod/core.html#hostnamelookups">HostnameLookups</a></code></li><li><code class="directive"><a href="../mod/core.html#enablemmap">EnableMMAP</a></code></li><li><code class="directive"><a href="../mod/core.html#enablesendfile">EnableSendfile</a></code></li><li><code class="directive"><a href="../mod/core.html#keepalivetimeout">KeepAliveTimeout</a></code></li><li><code class="directive"><a href="../mod/prefork.html#maxspareservers">MaxSpareServers</a></code></li><li><code class="directive"><a href="../mod/prefork.html#minspareservers">MinSpareServers</a></code></li><li><code class="directive"><a href="../mod/core.html#options">Options</a></code></li><li><code class="directive"><a href="../mod/mpm_common.html#startservers">StartServers</a></code></li></ul></td></tr></table>
105 <h3><a name="dns" id="dns">HostnameLookups and other DNS considerations</a></h3>
109 <p>Prior to Apache 1.3, <code class="directive"><a href="../mod/core.html#hostnamelookups">HostnameLookups</a></code> defaulted to <code>On</code>.
110 This adds latency to every request because it requires a
111 DNS lookup to complete before the request is finished. In
112 Apache 1.3 this setting defaults to <code>Off</code>. If you need
113 to have addresses in your log files resolved to hostnames, use the
114 <code class="program"><a href="../programs/logresolve.html">logresolve</a></code>
115 program that comes with Apache, or one of the numerous log
116 reporting packages which are available.</p>
118 <p>It is recommended that you do this sort of postprocessing of
119 your log files on some machine other than the production web
120 server machine, in order that this activity not adversely affect
121 server performance.</p>
123 <p>If you use any <code><code class="directive"><a href="../mod/mod_access_compat.html#allow">Allow</a></code> from domain</code> or <code><code class="directive"><a href="../mod/mod_access_compat.html#deny">Deny</a></code> from domain</code>
124 directives (i.e., using a hostname, or a domain name, rather than
125 an IP address) then you will pay for
126 two DNS lookups (a reverse, followed by a forward lookup
127 to make sure that the reverse is not being spoofed). For best
128 performance, therefore, use IP addresses, rather than names, when
129 using these directives, if possible.</p>
131 <p>Note that it's possible to scope the directives, such as
132 within a <code><Location /server-status></code> section.
133 In this case the DNS lookups are only performed on requests
134 matching the criteria. Here's an example which disables lookups
135 except for <code>.html</code> and <code>.cgi</code> files:</p>
137 <div class="example"><p><code>
138 HostnameLookups off<br />
139 <Files ~ "\.(html|cgi)$"><br />
140 <span class="indent">
141 HostnameLookups on<br />
146 <p>But even still, if you just need DNS names in some CGIs you
147 could consider doing the <code>gethostbyname</code> call in the
148 specific CGIs that need it.</p>
152 <h3><a name="symlinks" id="symlinks">FollowSymLinks and SymLinksIfOwnerMatch</a></h3>
156 <p>Wherever in your URL-space you do not have an <code>Options
157 FollowSymLinks</code>, or you do have an <code>Options
158 SymLinksIfOwnerMatch</code> Apache will have to issue extra
159 system calls to check up on symlinks. One extra call per
160 filename component. For example, if you had:</p>
162 <div class="example"><p><code>
163 DocumentRoot /www/htdocs<br />
164 <Directory /><br />
165 <span class="indent">
166 Options SymLinksIfOwnerMatch<br />
171 <p>and a request is made for the URI <code>/index.html</code>.
172 Then Apache will perform <code>lstat(2)</code> on
173 <code>/www</code>, <code>/www/htdocs</code>, and
174 <code>/www/htdocs/index.html</code>. The results of these
175 <code>lstats</code> are never cached, so they will occur on
176 every single request. If you really desire the symlinks
177 security checking you can do something like this:</p>
179 <div class="example"><p><code>
180 DocumentRoot /www/htdocs<br />
181 <Directory /><br />
182 <span class="indent">
183 Options FollowSymLinks<br />
185 </Directory><br />
187 <Directory /www/htdocs><br />
188 <span class="indent">
189 Options -FollowSymLinks +SymLinksIfOwnerMatch<br />
194 <p>This at least avoids the extra checks for the
195 <code class="directive"><a href="../mod/core.html#documentroot">DocumentRoot</a></code> path.
196 Note that you'll need to add similar sections if you
197 have any <code class="directive"><a href="../mod/mod_alias.html#alias">Alias</a></code> or
198 <code class="directive"><a href="../mod/mod_rewrite.html#rewriterule">RewriteRule</a></code> paths
199 outside of your document root. For highest performance,
200 and no symlink protection, set <code>FollowSymLinks</code>
201 everywhere, and never set <code>SymLinksIfOwnerMatch</code>.</p>
205 <h3><a name="htacess" id="htacess">AllowOverride</a></h3>
209 <p>Wherever in your URL-space you allow overrides (typically
210 <code>.htaccess</code> files) Apache will attempt to open
211 <code>.htaccess</code> for each filename component. For
214 <div class="example"><p><code>
215 DocumentRoot /www/htdocs<br />
216 <Directory /><br />
217 <span class="indent">
218 AllowOverride all<br />
223 <p>and a request is made for the URI <code>/index.html</code>.
224 Then Apache will attempt to open <code>/.htaccess</code>,
225 <code>/www/.htaccess</code>, and
226 <code>/www/htdocs/.htaccess</code>. The solutions are similar
227 to the previous case of <code>Options FollowSymLinks</code>.
228 For highest performance use <code>AllowOverride None</code>
229 everywhere in your filesystem.</p>
233 <h3><a name="negotiation" id="negotiation">Negotiation</a></h3>
237 <p>If at all possible, avoid content-negotiation if you're
238 really interested in every last ounce of performance. In
239 practice the benefits of negotiation outweigh the performance
240 penalties. There's one case where you can speed up the server.
241 Instead of using a wildcard such as:</p>
243 <div class="example"><p><code>
247 <p>Use a complete list of options:</p>
249 <div class="example"><p><code>
250 DirectoryIndex index.cgi index.pl index.shtml index.html
253 <p>where you list the most common choice first.</p>
255 <p>Also note that explicitly creating a <code>type-map</code>
256 file provides better performance than using
257 <code>MultiViews</code>, as the necessary information can be
258 determined by reading this single file, rather than having to
259 scan the directory for files.</p>
261 <p>If your site needs content negotiation consider using
262 <code>type-map</code> files, rather than the <code>Options
263 MultiViews</code> directive to accomplish the negotiation. See the
264 <a href="../content-negotiation.html">Content Negotiation</a>
265 documentation for a full discussion of the methods of negotiation,
266 and instructions for creating <code>type-map</code> files.</p>
270 <h3>Memory-mapping</h3>
274 <p>In situations where Apache 2.x needs to look at the contents
275 of a file being delivered--for example, when doing server-side-include
276 processing--it normally memory-maps the file if the OS supports
277 some form of <code>mmap(2)</code>.</p>
279 <p>On some platforms, this memory-mapping improves performance.
280 However, there are cases where memory-mapping can hurt the performance
281 or even the stability of the httpd:</p>
285 <p>On some operating systems, <code>mmap</code> does not scale
286 as well as <code>read(2)</code> when the number of CPUs increases.
287 On multiprocessor Solaris servers, for example, Apache 2.x sometimes
288 delivers server-parsed files faster when <code>mmap</code> is disabled.</p>
292 <p>If you memory-map a file located on an NFS-mounted filesystem
293 and a process on another NFS client machine deletes or truncates
294 the file, your process may get a bus error the next time it tries
295 to access the mapped file content.</p>
299 <p>For installations where either of these factors applies, you
300 should use <code>EnableMMAP off</code> to disable the memory-mapping
301 of delivered files. (Note: This directive can be overridden on
302 a per-directory basis.)</p>
310 <p>In situations where Apache 2.x can ignore the contents of the file
311 to be delivered -- for example, when serving static file content --
312 it normally uses the kernel sendfile support the file if the OS
313 supports the <code>sendfile(2)</code> operation.</p>
315 <p>On most platforms, using sendfile improves performance by eliminating
316 separate read and send mechanics. However, there are cases where using
317 sendfile can harm the stability of the httpd:</p>
321 <p>Some platforms may have broken sendfile support that the build
322 system did not detect, especially if the binaries were built on
323 another box and moved to such a machine with broken sendfile support.</p>
326 <p>With an NFS-mounted files, the kernel may be unable
327 to reliably serve the network file through it's own cache.</p>
331 <p>For installations where either of these factors applies, you
332 should use <code>EnableSendfile off</code> to disable sendfile
333 delivery of file contents. (Note: This directive can be overridden
334 on a per-directory basis.)</p>
338 <h3><a name="process" id="process">Process Creation</a></h3>
342 <p>Prior to Apache 1.3 the <code class="directive"><a href="../mod/prefork.html#minspareservers">MinSpareServers</a></code>, <code class="directive"><a href="../mod/prefork.html#maxspareservers">MaxSpareServers</a></code>, and <code class="directive"><a href="../mod/mpm_common.html#startservers">StartServers</a></code> settings all had drastic effects on
343 benchmark results. In particular, Apache required a "ramp-up"
344 period in order to reach a number of children sufficient to serve
345 the load being applied. After the initial spawning of
346 <code class="directive"><a href="../mod/mpm_common.html#startservers">StartServers</a></code> children,
347 only one child per second would be created to satisfy the
348 <code class="directive"><a href="../mod/prefork.html#minspareservers">MinSpareServers</a></code>
349 setting. So a server being accessed by 100 simultaneous
350 clients, using the default <code class="directive"><a href="../mod/mpm_common.html#startservers">StartServers</a></code> of <code>5</code> would take on
351 the order 95 seconds to spawn enough children to handle
352 the load. This works fine in practice on real-life servers,
353 because they aren't restarted frequently. But does really
354 poorly on benchmarks which might only run for ten minutes.</p>
356 <p>The one-per-second rule was implemented in an effort to
357 avoid swamping the machine with the startup of new children. If
358 the machine is busy spawning children it can't service
359 requests. But it has such a drastic effect on the perceived
360 performance of Apache that it had to be replaced. As of Apache
361 1.3, the code will relax the one-per-second rule. It will spawn
362 one, wait a second, then spawn two, wait a second, then spawn
363 four, and it will continue exponentially until it is spawning
364 32 children per second. It will stop whenever it satisfies the
365 <code class="directive"><a href="../mod/prefork.html#minspareservers">MinSpareServers</a></code>
368 <p>This appears to be responsive enough that it's almost
369 unnecessary to twiddle the <code class="directive"><a href="../mod/prefork.html#minspareservers">MinSpareServers</a></code>, <code class="directive"><a href="../mod/prefork.html#maxspareservers">MaxSpareServers</a></code> and <code class="directive"><a href="../mod/mpm_common.html#startservers">StartServers</a></code> knobs. When more than 4 children are
370 spawned per second, a message will be emitted to the
371 <code class="directive"><a href="../mod/core.html#errorlog">ErrorLog</a></code>. If you
372 see a lot of these errors then consider tuning these settings.
373 Use the <code class="module"><a href="../mod/mod_status.html">mod_status</a></code> output as a guide.</p>
375 <p>Related to process creation is process death induced by the
376 <code class="directive"><a href="../mod/mpm_common.html#maxrequestsperchild">MaxRequestsPerChild</a></code>
377 setting. By default this is <code>0</code>,
378 which means that there is no limit to the number of requests
379 handled per child. If your configuration currently has this set
380 to some very low number, such as <code>30</code>, you may want to bump this
381 up significantly. If you are running SunOS or an old version of
382 Solaris, limit this to <code>10000</code> or so because of memory leaks.</p>
384 <p>When keep-alives are in use, children will be kept busy
385 doing nothing waiting for more requests on the already open
386 connection. The default <code class="directive"><a href="../mod/core.html#keepalivetimeout">KeepAliveTimeout</a></code> of <code>5</code>
387 seconds attempts to minimize this effect. The tradeoff here is
388 between network bandwidth and server resources. In no event
389 should you raise this above about <code>60</code> seconds, as <a href="http://www.research.digital.com/wrl/techreports/abstracts/95.4.html">
390 most of the benefits are lost</a>.</p>
394 </div><div class="top"><a href="#page-header"><img alt="top" src="../images/up.gif" /></a></div>
395 <div class="section">
396 <h2><a name="compiletime" id="compiletime">Compile-Time Configuration Issues</a></h2>
400 <h3>Choosing an MPM</h3>
404 <p>Apache 2.x supports pluggable concurrency models, called
405 <a href="../mpm.html">Multi-Processing Modules</a> (MPMs).
406 When building Apache, you must choose an MPM to use. There
407 are platform-specific MPMs for some platforms:
408 <code class="module"><a href="../mod/mpm_netware.html">mpm_netware</a></code> and <code class="module"><a href="../mod/mpm_winnt.html">mpm_winnt</a></code>. For
409 general Unix-type systems, there are several MPMs from which
410 to choose. The choice of MPM can affect the speed and scalability
415 <li>The <code class="module"><a href="../mod/worker.html">worker</a></code> MPM uses multiple child
416 processes with many threads each. Each thread handles
417 one connection at a time. Worker generally is a good
418 choice for high-traffic servers because it has a smaller
419 memory footprint than the prefork MPM.</li>
421 <li>The <code class="module"><a href="../mod/prefork.html">prefork</a></code> MPM uses multiple child
422 processes with one thread each. Each process handles
423 one connection at a time. On many systems, prefork is
424 comparable in speed to worker, but it uses more memory.
425 Prefork's threadless design has advantages over worker
426 in some situations: it can be used with non-thread-safe
427 third-party modules, and it is easier to debug on platforms
428 with poor thread debugging support.</li>
432 <p>For more information on these and other MPMs, please
433 see the MPM <a href="../mpm.html">documentation</a>.</p>
437 <h3><a name="modules" id="modules">Modules</a></h3>
441 <p>Since memory usage is such an important consideration in
442 performance, you should attempt to eliminate modules that you are
443 not actually using. If you have built the modules as <a href="../dso.html">DSOs</a>, eliminating modules is a simple
444 matter of commenting out the associated <code class="directive"><a href="../mod/mod_so.html#loadmodule">LoadModule</a></code> directive for that module.
445 This allows you to experiment with removing modules, and seeing
446 if your site still functions in their absense.</p>
448 <p>If, on the other hand, you have modules statically linked
449 into your Apache binary, you will need to recompile Apache in
450 order to remove unwanted modules.</p>
452 <p>An associated question that arises here is, of course, what
453 modules you need, and which ones you don't. The answer here
454 will, of course, vary from one web site to another. However, the
455 <em>minimal</em> list of modules which you can get by with tends
456 to include <code class="module"><a href="../mod/mod_mime.html">mod_mime</a></code>, <code class="module"><a href="../mod/mod_dir.html">mod_dir</a></code>,
457 and <code class="module"><a href="../mod/mod_log_config.html">mod_log_config</a></code>. <code>mod_log_config</code> is,
458 of course, optional, as you can run a web site without log
459 files. This is, however, not recommended.</p>
463 <h3>Atomic Operations</h3>
467 <p>Some modules, such as <code class="module"><a href="../mod/mod_cache.html">mod_cache</a></code> and
468 recent development builds of the worker MPM, use APR's
469 atomic API. This API provides atomic operations that can
470 be used for lightweight thread synchronization.</p>
472 <p>By default, APR implements these operations using the
473 most efficient mechanism available on each target
474 OS/CPU platform. Many modern CPUs, for example, have
475 an instruction that does an atomic compare-and-swap (CAS)
476 operation in hardware. On some platforms, however, APR
477 defaults to a slower, mutex-based implementation of the
478 atomic API in order to ensure compatibility with older
479 CPU models that lack such instructions. If you are
480 building Apache for one of these platforms, and you plan
481 to run only on newer CPUs, you can select a faster atomic
482 implementation at build time by configuring Apache with
483 the <code>--enable-nonportable-atomics</code> option:</p>
485 <div class="example"><p><code>
487 ./configure --with-mpm=worker --enable-nonportable-atomics=yes
490 <p>The <code>--enable-nonportable-atomics</code> option is
491 relevant for the following platforms:</p>
495 <li>Solaris on SPARC<br />
496 By default, APR uses mutex-based atomics on Solaris/SPARC.
497 If you configure with <code>--enable-nonportable-atomics</code>,
498 however, APR generates code that uses a SPARC v8plus opcode for
499 fast hardware compare-and-swap. If you configure Apache with
500 this option, the atomic operations will be more efficient
501 (allowing for lower CPU utilization and higher concurrency),
502 but the resulting executable will run only on UltraSPARC
506 <li>Linux on x86<br />
507 By default, APR uses mutex-based atomics on Linux. If you
508 configure with <code>--enable-nonportable-atomics</code>,
509 however, APR generates code that uses a 486 opcode for fast
510 hardware compare-and-swap. This will result in more efficient
511 atomic operations, but the resulting executable will run only
512 on 486 and later chips (and not on 386).
519 <h3>mod_status and ExtendedStatus On</h3>
523 <p>If you include <code class="module"><a href="../mod/mod_status.html">mod_status</a></code> and you also set
524 <code>ExtendedStatus On</code> when building and running
525 Apache, then on every request Apache will perform two calls to
526 <code>gettimeofday(2)</code> (or <code>times(2)</code>
527 depending on your operating system), and (pre-1.3) several
528 extra calls to <code>time(2)</code>. This is all done so that
529 the status report contains timing indications. For highest
530 performance, set <code>ExtendedStatus off</code> (which is the
535 <h3>accept Serialization - multiple sockets</h3>
539 <div class="warning"><h3>Warning:</h3>
540 <p>This section has not been fully updated
541 to take into account changes made in the 2.x version of the
542 Apache HTTP Server. Some of the information may still be
543 relevant, but please use it with care.</p>
546 <p>This discusses a shortcoming in the Unix socket API. Suppose
547 your web server uses multiple <code class="directive"><a href="../mod/mpm_common.html#listen">Listen</a></code> statements to listen on either multiple
548 ports or multiple addresses. In order to test each socket
549 to see if a connection is ready Apache uses
550 <code>select(2)</code>. <code>select(2)</code> indicates that a
551 socket has <em>zero</em> or <em>at least one</em> connection
552 waiting on it. Apache's model includes multiple children, and
553 all the idle ones test for new connections at the same time. A
554 naive implementation looks something like this (these examples
555 do not match the code, they're contrived for pedagogical
558 <div class="example"><p><code>
560 <span class="indent">
562 <span class="indent">
563 fd_set accept_fds;<br />
565 FD_ZERO (&accept_fds);<br />
566 for (i = first_socket; i <= last_socket; ++i) {<br />
567 <span class="indent">
568 FD_SET (i, &accept_fds);<br />
571 rc = select (last_socket+1, &accept_fds, NULL, NULL, NULL);<br />
572 if (rc < 1) continue;<br />
573 new_connection = -1;<br />
574 for (i = first_socket; i <= last_socket; ++i) {<br />
575 <span class="indent">
576 if (FD_ISSET (i, &accept_fds)) {<br />
577 <span class="indent">
578 new_connection = accept (i, NULL, NULL);<br />
579 if (new_connection != -1) break;<br />
584 if (new_connection != -1) break;<br />
587 process the new_connection;<br />
592 <p>But this naive implementation has a serious starvation problem.
593 Recall that multiple children execute this loop at the same
594 time, and so multiple children will block at
595 <code>select</code> when they are in between requests. All
596 those blocked children will awaken and return from
597 <code>select</code> when a single request appears on any socket
598 (the number of children which awaken varies depending on the
599 operating system and timing issues). They will all then fall
600 down into the loop and try to <code>accept</code> the
601 connection. But only one will succeed (assuming there's still
602 only one connection ready), the rest will be <em>blocked</em>
603 in <code>accept</code>. This effectively locks those children
604 into serving requests from that one socket and no other
605 sockets, and they'll be stuck there until enough new requests
606 appear on that socket to wake them all up. This starvation
607 problem was first documented in <a href="http://bugs.apache.org/index/full/467">PR#467</a>. There
608 are at least two solutions.</p>
610 <p>One solution is to make the sockets non-blocking. In this
611 case the <code>accept</code> won't block the children, and they
612 will be allowed to continue immediately. But this wastes CPU
613 time. Suppose you have ten idle children in
614 <code>select</code>, and one connection arrives. Then nine of
615 those children will wake up, try to <code>accept</code> the
616 connection, fail, and loop back into <code>select</code>,
617 accomplishing nothing. Meanwhile none of those children are
618 servicing requests that occurred on other sockets until they
619 get back up to the <code>select</code> again. Overall this
620 solution does not seem very fruitful unless you have as many
621 idle CPUs (in a multiprocessor box) as you have idle children,
622 not a very likely situation.</p>
624 <p>Another solution, the one used by Apache, is to serialize
625 entry into the inner loop. The loop looks like this
626 (differences highlighted):</p>
628 <div class="example"><p><code>
630 <span class="indent">
631 <strong>accept_mutex_on ();</strong><br />
633 <span class="indent">
634 fd_set accept_fds;<br />
636 FD_ZERO (&accept_fds);<br />
637 for (i = first_socket; i <= last_socket; ++i) {<br />
638 <span class="indent">
639 FD_SET (i, &accept_fds);<br />
642 rc = select (last_socket+1, &accept_fds, NULL, NULL, NULL);<br />
643 if (rc < 1) continue;<br />
644 new_connection = -1;<br />
645 for (i = first_socket; i <= last_socket; ++i) {<br />
646 <span class="indent">
647 if (FD_ISSET (i, &accept_fds)) {<br />
648 <span class="indent">
649 new_connection = accept (i, NULL, NULL);<br />
650 if (new_connection != -1) break;<br />
655 if (new_connection != -1) break;<br />
658 <strong>accept_mutex_off ();</strong><br />
659 process the new_connection;<br />
664 <p><a id="serialize" name="serialize">The functions</a>
665 <code>accept_mutex_on</code> and <code>accept_mutex_off</code>
666 implement a mutual exclusion semaphore. Only one child can have
667 the mutex at any time. There are several choices for
668 implementing these mutexes. The choice is defined in
669 <code>src/conf.h</code> (pre-1.3) or
670 <code>src/include/ap_config.h</code> (1.3 or later). Some
671 architectures do not have any locking choice made, on these
672 architectures it is unsafe to use multiple
673 <code class="directive"><a href="../mod/mpm_common.html#listen">Listen</a></code>
676 <p>The directive <code class="directive"><a href="../mod/mpm_common.html#acceptmutex">AcceptMutex</a></code> can be used to
677 change the selected mutex implementation at run-time.</p>
680 <dt><code>AcceptMutex flock</code></dt>
683 <p>This method uses the <code>flock(2)</code> system call to
684 lock a lock file (located by the <code class="directive"><a href="../mod/mpm_common.html#lockfile">LockFile</a></code> directive).</p>
687 <dt><code>AcceptMutex fcntl</code></dt>
690 <p>This method uses the <code>fcntl(2)</code> system call to
691 lock a lock file (located by the <code class="directive"><a href="../mod/mpm_common.html#lockfile">LockFile</a></code> directive).</p>
694 <dt><code>AcceptMutex sysvsem</code></dt>
697 <p>(1.3 or later) This method uses SysV-style semaphores to
698 implement the mutex. Unfortunately SysV-style semaphores have
699 some bad side-effects. One is that it's possible Apache will
700 die without cleaning up the semaphore (see the
701 <code>ipcs(8)</code> man page). The other is that the
702 semaphore API allows for a denial of service attack by any
703 CGIs running under the same uid as the webserver
704 (<em>i.e.</em>, all CGIs, unless you use something like
705 <code class="program"><a href="../programs/suexec.html">suexec</a></code> or <code>cgiwrapper</code>).</p>
708 <dt><code>AcceptMutex pthread</code></dt>
711 <p>(1.3 or later) This method uses POSIX mutexes and should
712 work on any architecture implementing the full POSIX threads
713 specification, however appears to only work on Solaris (2.5
714 or later), and even then only in certain configurations. If
715 you experiment with this you should watch out for your server
716 hanging and not responding. Static content only servers may
720 <dt><code>AcceptMutex posixsem</code></dt>
723 <p>(2.0 or later) This method uses POSIX semaphores. The
724 semaphore ownership is not recovered if a thread in the process
725 holding the mutex segfaults, resulting in a hang of the web
731 <p>If your system has another method of serialization which
732 isn't in the above list then it may be worthwhile adding code
735 <p>Another solution that has been considered but never
736 implemented is to partially serialize the loop -- that is, let
737 in a certain number of processes. This would only be of
738 interest on multiprocessor boxes where it's possible multiple
739 children could run simultaneously, and the serialization
740 actually doesn't take advantage of the full bandwidth. This is
741 a possible area of future investigation, but priority remains
742 low because highly parallel web servers are not the norm.</p>
744 <p>Ideally you should run servers without multiple
745 <code class="directive"><a href="../mod/mpm_common.html#listen">Listen</a></code>
746 statements if you want the highest performance.
751 <h3>accept Serialization - single socket</h3>
755 <p>The above is fine and dandy for multiple socket servers, but
756 what about single socket servers? In theory they shouldn't
757 experience any of these same problems because all children can
758 just block in <code>accept(2)</code> until a connection
759 arrives, and no starvation results. In practice this hides
760 almost the same "spinning" behaviour discussed above in the
761 non-blocking solution. The way that most TCP stacks are
762 implemented, the kernel actually wakes up all processes blocked
763 in <code>accept</code> when a single connection arrives. One of
764 those processes gets the connection and returns to user-space,
765 the rest spin in the kernel and go back to sleep when they
766 discover there's no connection for them. This spinning is
767 hidden from the user-land code, but it's there nonetheless.
768 This can result in the same load-spiking wasteful behaviour
769 that a non-blocking solution to the multiple sockets case
772 <p>For this reason we have found that many architectures behave
773 more "nicely" if we serialize even the single socket case. So
774 this is actually the default in almost all cases. Crude
775 experiments under Linux (2.0.30 on a dual Pentium pro 166
776 w/128Mb RAM) have shown that the serialization of the single
777 socket case causes less than a 3% decrease in requests per
778 second over unserialized single-socket. But unserialized
779 single-socket showed an extra 100ms latency on each request.
780 This latency is probably a wash on long haul lines, and only an
781 issue on LANs. If you want to override the single socket
782 serialization you can define
783 <code>SINGLE_LISTEN_UNSERIALIZED_ACCEPT</code> and then
784 single-socket servers will not serialize at all.</p>
788 <h3>Lingering Close</h3>
792 <p>As discussed in <a href="http://www.ics.uci.edu/pub/ietf/http/draft-ietf-http-connection-00.txt">
793 draft-ietf-http-connection-00.txt</a> section 8, in order for
794 an HTTP server to <strong>reliably</strong> implement the
795 protocol it needs to shutdown each direction of the
796 communication independently (recall that a TCP connection is
797 bi-directional, each half is independent of the other). This
798 fact is often overlooked by other servers, but is correctly
799 implemented in Apache as of 1.2.</p>
801 <p>When this feature was added to Apache it caused a flurry of
802 problems on various versions of Unix because of a
803 shortsightedness. The TCP specification does not state that the
804 <code>FIN_WAIT_2</code> state has a timeout, but it doesn't prohibit it.
805 On systems without the timeout, Apache 1.2 induces many sockets
806 stuck forever in the <code>FIN_WAIT_2</code> state. In many cases this
807 can be avoided by simply upgrading to the latest TCP/IP patches
808 supplied by the vendor. In cases where the vendor has never
809 released patches (<em>i.e.</em>, SunOS4 -- although folks with
810 a source license can patch it themselves) we have decided to
811 disable this feature.</p>
813 <p>There are two ways of accomplishing this. One is the socket
814 option <code>SO_LINGER</code>. But as fate would have it, this
815 has never been implemented properly in most TCP/IP stacks. Even
816 on those stacks with a proper implementation (<em>i.e.</em>,
817 Linux 2.0.31) this method proves to be more expensive (cputime)
818 than the next solution.</p>
820 <p>For the most part, Apache implements this in a function
821 called <code>lingering_close</code> (in
822 <code>http_main.c</code>). The function looks roughly like
825 <div class="example"><p><code>
826 void lingering_close (int s)<br />
828 <span class="indent">
829 char junk_buffer[2048];<br />
831 /* shutdown the sending side */<br />
832 shutdown (s, 1);<br />
834 signal (SIGALRM, lingering_death);<br />
838 <span class="indent">
839 select (s for reading, 2 second timeout);<br />
840 if (error) break;<br />
841 if (s is ready for reading) {<br />
842 <span class="indent">
843 if (read (s, junk_buffer, sizeof (junk_buffer)) <= 0) {<br />
844 <span class="indent">
848 /* just toss away whatever is here */<br />
859 <p>This naturally adds some expense at the end of a connection,
860 but it is required for a reliable implementation. As HTTP/1.1
861 becomes more prevalent, and all connections are persistent,
862 this expense will be amortized over more requests. If you want
863 to play with fire and disable this feature you can define
864 <code>NO_LINGCLOSE</code>, but this is not recommended at all.
865 In particular, as HTTP/1.1 pipelined persistent connections
866 come into use <code>lingering_close</code> is an absolute
867 necessity (and <a href="http://www.w3.org/Protocols/HTTP/Performance/Pipeline.html">
868 pipelined connections are faster</a>, so you want to support
873 <h3>Scoreboard File</h3>
877 <p>Apache's parent and children communicate with each other
878 through something called the scoreboard. Ideally this should be
879 implemented in shared memory. For those operating systems that
880 we either have access to, or have been given detailed ports
881 for, it typically is implemented using shared memory. The rest
882 default to using an on-disk file. The on-disk file is not only
883 slow, but it is unreliable (and less featured). Peruse the
884 <code>src/main/conf.h</code> file for your architecture and
885 look for either <code>USE_MMAP_SCOREBOARD</code> or
886 <code>USE_SHMGET_SCOREBOARD</code>. Defining one of those two
887 (as well as their companions <code>HAVE_MMAP</code> and
888 <code>HAVE_SHMGET</code> respectively) enables the supplied
889 shared memory code. If your system has another type of shared
890 memory, edit the file <code>src/main/http_main.c</code> and add
891 the hooks necessary to use it in Apache. (Send us back a patch
894 <div class="note">Historical note: The Linux port of Apache didn't start to
895 use shared memory until version 1.2 of Apache. This oversight
896 resulted in really poor and unreliable behaviour of earlier
897 versions of Apache on Linux.</div>
901 <h3>DYNAMIC_MODULE_LIMIT</h3>
905 <p>If you have no intention of using dynamically loaded modules
906 (you probably don't if you're reading this and tuning your
907 server for every last ounce of performance) then you should add
908 <code>-DDYNAMIC_MODULE_LIMIT=0</code> when building your
909 server. This will save RAM that's allocated only for supporting
910 dynamically loaded modules.</p>
914 </div><div class="top"><a href="#page-header"><img alt="top" src="../images/up.gif" /></a></div>
915 <div class="section">
916 <h2><a name="trace" id="trace">Appendix: Detailed Analysis of a Trace</a></h2>
920 <p>Here is a system call trace of Apache 2.0.38 with the worker MPM
921 on Solaris 8. This trace was collected using:</p>
923 <div class="example"><p><code>
924 truss -l -p <var>httpd_child_pid</var>.
927 <p>The <code>-l</code> option tells truss to log the ID of the
928 LWP (lightweight process--Solaris's form of kernel-level thread)
929 that invokes each system call.</p>
931 <p>Other systems may have different system call tracing utilities
932 such as <code>strace</code>, <code>ktrace</code>, or <code>par</code>.
933 They all produce similar output.</p>
935 <p>In this trace, a client has requested a 10KB static file
936 from the httpd. Traces of non-static requests or requests
937 with content negotiation look wildly different (and quite ugly
940 <div class="example"><pre>/67: accept(3, 0x00200BEC, 0x00200C0C, 1) (sleeping...)
941 /67: accept(3, 0x00200BEC, 0x00200C0C, 1) = 9</pre></div>
943 <p>In this trace, the listener thread is running within LWP #67.</p>
945 <div class="note">Note the lack of <code>accept(2)</code> serialization. On this
946 particular platform, the worker MPM uses an unserialized accept by
947 default unless it is listening on multiple ports.</div>
949 <div class="example"><pre>/65: lwp_park(0x00000000, 0) = 0
950 /67: lwp_unpark(65, 1) = 0</pre></div>
952 <p>Upon accepting the connection, the listener thread wakes up
953 a worker thread to do the request processing. In this trace,
954 the worker thread that handles the request is mapped to LWP #65.</p>
956 <div class="example"><pre>/65: getsockname(9, 0x00200BA4, 0x00200BC4, 1) = 0</pre></div>
958 <p>In order to implement virtual hosts, Apache needs to know
959 the local socket address used to accept the connection. It
960 is possible to eliminate this call in many situations (such
961 as when there are no virtual hosts, or when
962 <code class="directive"><a href="../mod/mpm_common.html#listen">Listen</a></code> directives
963 are used which do not have wildcard addresses). But
964 no effort has yet been made to do these optimizations. </p>
966 <div class="example"><pre>/65: brk(0x002170E8) = 0
967 /65: brk(0x002190E8) = 0</pre></div>
969 <p>The <code>brk(2)</code> calls allocate memory from the heap.
970 It is rare to see these in a system call trace, because the httpd
971 uses custom memory allocators (<code>apr_pool</code> and
972 <code>apr_bucket_alloc</code>) for most request processing.
973 In this trace, the httpd has just been started, so it must
974 call <code>malloc(3)</code> to get the blocks of raw memory
975 with which to create the custom memory allocators.</p>
977 <div class="example"><pre>/65: fcntl(9, F_GETFL, 0x00000000) = 2
978 /65: fstat64(9, 0xFAF7B818) = 0
979 /65: getsockopt(9, 65535, 8192, 0xFAF7B918, 0xFAF7B910, 2190656) = 0
980 /65: fstat64(9, 0xFAF7B818) = 0
981 /65: getsockopt(9, 65535, 8192, 0xFAF7B918, 0xFAF7B914, 2190656) = 0
982 /65: setsockopt(9, 65535, 8192, 0xFAF7B918, 4, 2190656) = 0
983 /65: fcntl(9, F_SETFL, 0x00000082) = 0</pre></div>
985 <p>Next, the worker thread puts the connection to the client (file
986 descriptor 9) in non-blocking mode. The <code>setsockopt(2)</code>
987 and <code>getsockopt(2)</code> calls are a side-effect of how
988 Solaris's libc handles <code>fcntl(2)</code> on sockets.</p>
990 <div class="example"><pre>/65: read(9, " G E T / 1 0 k . h t m".., 8000) = 97</pre></div>
992 <p>The worker thread reads the request from the client.</p>
994 <div class="example"><pre>/65: stat("/var/httpd/apache/httpd-8999/htdocs/10k.html", 0xFAF7B978) = 0
995 /65: open("/var/httpd/apache/httpd-8999/htdocs/10k.html", O_RDONLY) = 10</pre></div>
997 <p>This httpd has been configured with <code>Options FollowSymLinks</code>
998 and <code>AllowOverride None</code>. Thus it doesn't need to
999 <code>lstat(2)</code> each directory in the path leading up to the
1000 requested file, nor check for <code>.htaccess</code> files.
1001 It simply calls <code>stat(2)</code> to verify that the file:
1002 1) exists, and 2) is a regular file, not a directory.</p>
1004 <div class="example"><pre>/65: sendfilev(0, 9, 0x00200F90, 2, 0xFAF7B53C) = 10269</pre></div>
1006 <p>In this example, the httpd is able to send the HTTP response
1007 header and the requested file with a single <code>sendfilev(2)</code>
1008 system call. Sendfile semantics vary among operating systems. On some other
1009 systems, it is necessary to do a <code>write(2)</code> or
1010 <code>writev(2)</code> call to send the headers before calling
1011 <code>sendfile(2)</code>.</p>
1013 <div class="example"><pre>/65: write(4, " 1 2 7 . 0 . 0 . 1 - ".., 78) = 78</pre></div>
1015 <p>This <code>write(2)</code> call records the request in the
1016 access log. Note that one thing missing from this trace is a
1017 <code>time(2)</code> call. Unlike Apache 1.3, Apache 2.x uses
1018 <code>gettimeofday(3)</code> to look up the time. On some operating
1019 systems, like Linux or Solaris, <code>gettimeofday</code> has an
1020 optimized implementation that doesn't require as much overhead
1021 as a typical system call.</p>
1023 <div class="example"><pre>/65: shutdown(9, 1, 1) = 0
1024 /65: poll(0xFAF7B980, 1, 2000) = 1
1025 /65: read(9, 0xFAF7BC20, 512) = 0
1026 /65: close(9) = 0</pre></div>
1028 <p>The worker thread does a lingering close of the connection.</p>
1030 <div class="example"><pre>/65: close(10) = 0
1031 /65: lwp_park(0x00000000, 0) (sleeping...)</pre></div>
1033 <p>Finally the worker thread closes the file that it has just delivered
1034 and blocks until the listener assigns it another connection.</p>
1036 <div class="example"><pre>/67: accept(3, 0x001FEB74, 0x001FEB94, 1) (sleeping...)</pre></div>
1038 <p>Meanwhile, the listener thread is able to accept another connection
1039 as soon as it has dispatched this connection to a worker thread (subject
1040 to some flow-control logic in the worker MPM that throttles the listener
1041 if all the available workers are busy). Though it isn't apparent from
1042 this trace, the next <code>accept(2)</code> can (and usually does, under
1043 high load conditions) occur in parallel with the worker thread's handling
1044 of the just-accepted connection.</p>
1047 <div class="bottomlang">
1048 <p><span>Available Languages: </span><a href="../en/misc/perf-tuning.html" title="English"> en </a> |
1049 <a href="../fr/misc/perf-tuning.html" hreflang="fr" rel="alternate" title="Français"> fr </a> |
1050 <a href="../ko/misc/perf-tuning.html" hreflang="ko" rel="alternate" title="Korean"> ko </a> |
1051 <a href="../tr/misc/perf-tuning.html" hreflang="tr" rel="alternate" title="Türkçe"> tr </a></p>
1052 </div><div id="footer">
1053 <p class="apache">Copyright 2009 The Apache Software Foundation.<br />Licensed under the <a href="http://www.apache.org/licenses/LICENSE-2.0">Apache License, Version 2.0</a>.</p>
1054 <p class="menu"><a href="../mod/">Modules</a> | <a href="../mod/directives.html">Directives</a> | <a href="../faq/">FAQ</a> | <a href="../glossary.html">Glossary</a> | <a href="../sitemap.html">Sitemap</a></p></div>
projects / apache / blob