1 <?xml version="1.0" encoding="UTF-8" ?>
2 <!DOCTYPE manualpage SYSTEM "../style/manualpage.dtd">
3 <?xml-stylesheet type="text/xsl" href="../style/manual.en.xsl"?>
4 <!-- $LastChangedRevision$ -->
7 Licensed to the Apache Software Foundation (ASF) under one or more
8 contributor license agreements. See the NOTICE file distributed with
9 this work for additional information regarding copyright ownership.
10 The ASF licenses this file to You under the Apache License, Version 2.0
11 (the "License"); you may not use this file except in compliance with
12 the License. You may obtain a copy of the License at
14 http://www.apache.org/licenses/LICENSE-2.0
16 Unless required by applicable law or agreed to in writing, software
17 distributed under the License is distributed on an "AS IS" BASIS,
18 WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
19 See the License for the specific language governing permissions and
20 limitations under the License.
23 <manualpage metafile="perf-tuning.xml.meta">
24 <parentdocument href="./">Miscellaneous Documentation</parentdocument>
26 <title>Apache Performance Tuning</title>
30 <p>Apache 2.x is a general-purpose webserver, designed to
31 provide a balance of flexibility, portability, and performance.
32 Although it has not been designed specifically to set benchmark
33 records, Apache 2.x is capable of high performance in many
34 real-world situations.</p>
36 <p>Compared to Apache 1.3, release 2.x contains many additional
37 optimizations to increase throughput and scalability. Most of
38 these improvements are enabled by default. However, there are
39 compile-time and run-time configuration choices that can
40 significantly affect performance. This document describes the
41 options that a server administrator can configure to tune the
42 performance of an Apache 2.x installation. Some of these
43 configuration options enable the httpd to better take advantage
44 of the capabilities of the hardware and OS, while others allow
45 the administrator to trade functionality for speed.</p>
49 <section id="hardware">
51 <title>Hardware and Operating System Issues</title>
53 <p>The single biggest hardware issue affecting webserver
54 performance is RAM. A webserver should never ever have to swap,
55 as swapping increases the latency of each request beyond a point
56 that users consider "fast enough". This causes users to hit
57 stop and reload, further increasing the load. You can, and
58 should, control the <directive module="mpm_common"
59 >MaxRequestWorkers</directive> setting so that your server
60 does not spawn so many children it starts swapping. This procedure
61 for doing this is simple: determine the size of your average Apache
62 process, by looking at your process list via a tool such as
63 <code>top</code>, and divide this into your total available memory,
64 leaving some room for other processes.</p>
66 <p>Beyond that the rest is mundane: get a fast enough CPU, a
67 fast enough network card, and fast enough disks, where "fast
68 enough" is something that needs to be determined by
71 <p>Operating system choice is largely a matter of local
72 concerns. But some guidelines that have proven generally
77 <p>Run the latest stable release and patchlevel of the
78 operating system that you choose. Many OS suppliers have
79 introduced significant performance improvements to their
80 TCP stacks and thread libraries in recent years.</p>
84 <p>If your OS supports a <code>sendfile(2)</code> system
85 call, make sure you install the release and/or patches
86 needed to enable it. (With Linux, for example, this means
87 using Linux 2.4 or later. For early releases of Solaris 8,
88 you may need to apply a patch.) On systems where it is
89 available, <code>sendfile</code> enables Apache 2 to deliver
90 static content faster and with lower CPU utilization.</p>
96 <section id="runtime">
98 <title>Run-Time Configuration Issues</title>
102 <module>mod_dir</module>
103 <module>mpm_common</module>
104 <module>mod_status</module>
107 <directive module="core">AllowOverride</directive>
108 <directive module="mod_dir">DirectoryIndex</directive>
109 <directive module="core">HostnameLookups</directive>
110 <directive module="core">EnableMMAP</directive>
111 <directive module="core">EnableSendfile</directive>
112 <directive module="core">KeepAliveTimeout</directive>
113 <directive module="prefork">MaxSpareServers</directive>
114 <directive module="prefork">MinSpareServers</directive>
115 <directive module="core">Options</directive>
116 <directive module="mpm_common">StartServers</directive>
122 <title>HostnameLookups and other DNS considerations</title>
124 <p>Prior to Apache 1.3, <directive module="core"
125 >HostnameLookups</directive> defaulted to <code>On</code>.
126 This adds latency to every request because it requires a
127 DNS lookup to complete before the request is finished. In
128 Apache 1.3 this setting defaults to <code>Off</code>. If you need
129 to have addresses in your log files resolved to hostnames, use the
130 <program>logresolve</program>
131 program that comes with Apache, or one of the numerous log
132 reporting packages which are available.</p>
134 <p>It is recommended that you do this sort of postprocessing of
135 your log files on some machine other than the production web
136 server machine, in order that this activity not adversely affect
137 server performance.</p>
139 <p>If you use any <code><directive module="mod_access_compat"
140 >Allow</directive> from domain</code> or <code><directive
141 module="mod_access_compat">Deny</directive> from domain</code>
142 directives (i.e., using a hostname, or a domain name, rather than
143 an IP address) then you will pay for
144 two DNS lookups (a reverse, followed by a forward lookup
145 to make sure that the reverse is not being spoofed). For best
146 performance, therefore, use IP addresses, rather than names, when
147 using these directives, if possible.</p>
149 <p>Note that it's possible to scope the directives, such as
150 within a <code><Location /server-status></code> section.
151 In this case the DNS lookups are only performed on requests
152 matching the criteria. Here's an example which disables lookups
153 except for <code>.html</code> and <code>.cgi</code> files:</p>
155 <highlight language="config">
157 <Files ~ "\.(html|cgi)$">
162 <p>But even still, if you just need DNS names in some CGIs you
163 could consider doing the <code>gethostbyname</code> call in the
164 specific CGIs that need it.</p>
168 <section id="symlinks">
170 <title>FollowSymLinks and SymLinksIfOwnerMatch</title>
172 <p>Wherever in your URL-space you do not have an <code>Options
173 FollowSymLinks</code>, or you do have an <code>Options
174 SymLinksIfOwnerMatch</code> Apache will have to issue extra
175 system calls to check up on symlinks. One extra call per
176 filename component. For example, if you had:</p>
178 <highlight language="config">
179 DocumentRoot /www/htdocs
181 Options SymLinksIfOwnerMatch
185 <p>and a request is made for the URI <code>/index.html</code>.
186 Then Apache will perform <code>lstat(2)</code> on
187 <code>/www</code>, <code>/www/htdocs</code>, and
188 <code>/www/htdocs/index.html</code>. The results of these
189 <code>lstats</code> are never cached, so they will occur on
190 every single request. If you really desire the symlinks
191 security checking you can do something like this:</p>
193 <highlight language="config">
194 DocumentRoot /www/htdocs
196 Options FollowSymLinks
199 <Directory /www/htdocs>
200 Options -FollowSymLinks +SymLinksIfOwnerMatch
204 <p>This at least avoids the extra checks for the
205 <directive module="core">DocumentRoot</directive> path.
206 Note that you'll need to add similar sections if you
207 have any <directive module="mod_alias">Alias</directive> or
208 <directive module="mod_rewrite">RewriteRule</directive> paths
209 outside of your document root. For highest performance,
210 and no symlink protection, set <code>FollowSymLinks</code>
211 everywhere, and never set <code>SymLinksIfOwnerMatch</code>.</p>
215 <section id="htaccess">
217 <title>AllowOverride</title>
219 <p>Wherever in your URL-space you allow overrides (typically
220 <code>.htaccess</code> files) Apache will attempt to open
221 <code>.htaccess</code> for each filename component. For
224 <highlight language="config">
225 DocumentRoot /www/htdocs
231 <p>and a request is made for the URI <code>/index.html</code>.
232 Then Apache will attempt to open <code>/.htaccess</code>,
233 <code>/www/.htaccess</code>, and
234 <code>/www/htdocs/.htaccess</code>. The solutions are similar
235 to the previous case of <code>Options FollowSymLinks</code>.
236 For highest performance use <code>AllowOverride None</code>
237 everywhere in your filesystem.</p>
241 <section id="negotiation">
243 <title>Negotiation</title>
245 <p>If at all possible, avoid content-negotiation if you're
246 really interested in every last ounce of performance. In
247 practice the benefits of negotiation outweigh the performance
248 penalties. There's one case where you can speed up the server.
249 Instead of using a wildcard such as:</p>
251 <highlight language="config">DirectoryIndex index</highlight>
253 <p>Use a complete list of options:</p>
255 <highlight language="config">DirectoryIndex index.cgi index.pl index.shtml index.html</highlight>
257 <p>where you list the most common choice first.</p>
259 <p>Also note that explicitly creating a <code>type-map</code>
260 file provides better performance than using
261 <code>MultiViews</code>, as the necessary information can be
262 determined by reading this single file, rather than having to
263 scan the directory for files.</p>
265 <p>If your site needs content negotiation consider using
266 <code>type-map</code> files, rather than the <code>Options
267 MultiViews</code> directive to accomplish the negotiation. See the
268 <a href="../content-negotiation.html">Content Negotiation</a>
269 documentation for a full discussion of the methods of negotiation,
270 and instructions for creating <code>type-map</code> files.</p>
276 <title>Memory-mapping</title>
278 <p>In situations where Apache 2.x needs to look at the contents
279 of a file being delivered--for example, when doing server-side-include
280 processing--it normally memory-maps the file if the OS supports
281 some form of <code>mmap(2)</code>.</p>
283 <p>On some platforms, this memory-mapping improves performance.
284 However, there are cases where memory-mapping can hurt the performance
285 or even the stability of the httpd:</p>
289 <p>On some operating systems, <code>mmap</code> does not scale
290 as well as <code>read(2)</code> when the number of CPUs increases.
291 On multiprocessor Solaris servers, for example, Apache 2.x sometimes
292 delivers server-parsed files faster when <code>mmap</code> is disabled.</p>
296 <p>If you memory-map a file located on an NFS-mounted filesystem
297 and a process on another NFS client machine deletes or truncates
298 the file, your process may get a bus error the next time it tries
299 to access the mapped file content.</p>
303 <p>For installations where either of these factors applies, you
304 should use <code>EnableMMAP off</code> to disable the memory-mapping
305 of delivered files. (Note: This directive can be overridden on
306 a per-directory basis.)</p>
312 <title>Sendfile</title>
314 <p>In situations where Apache 2.x can ignore the contents of the file
315 to be delivered -- for example, when serving static file content --
316 it normally uses the kernel sendfile support the file if the OS
317 supports the <code>sendfile(2)</code> operation.</p>
319 <p>On most platforms, using sendfile improves performance by eliminating
320 separate read and send mechanics. However, there are cases where using
321 sendfile can harm the stability of the httpd:</p>
325 <p>Some platforms may have broken sendfile support that the build
326 system did not detect, especially if the binaries were built on
327 another box and moved to such a machine with broken sendfile support.</p>
330 <p>With an NFS-mounted filesystem, the kernel may be unable
331 to reliably serve the network file through its own cache.</p>
335 <p>For installations where either of these factors applies, you
336 should use <code>EnableSendfile off</code> to disable sendfile
337 delivery of file contents. (Note: This directive can be overridden
338 on a per-directory basis.)</p>
342 <section id="process">
344 <title>Process Creation</title>
346 <p>Prior to Apache 1.3 the <directive module="prefork"
347 >MinSpareServers</directive>, <directive module="prefork"
348 >MaxSpareServers</directive>, and <directive module="mpm_common"
349 >StartServers</directive> settings all had drastic effects on
350 benchmark results. In particular, Apache required a "ramp-up"
351 period in order to reach a number of children sufficient to serve
352 the load being applied. After the initial spawning of
353 <directive module="mpm_common">StartServers</directive> children,
354 only one child per second would be created to satisfy the
355 <directive module="prefork">MinSpareServers</directive>
356 setting. So a server being accessed by 100 simultaneous
357 clients, using the default <directive module="mpm_common"
358 >StartServers</directive> of <code>5</code> would take on
359 the order 95 seconds to spawn enough children to handle
360 the load. This works fine in practice on real-life servers,
361 because they aren't restarted frequently. But does really
362 poorly on benchmarks which might only run for ten minutes.</p>
364 <p>The one-per-second rule was implemented in an effort to
365 avoid swamping the machine with the startup of new children. If
366 the machine is busy spawning children it can't service
367 requests. But it has such a drastic effect on the perceived
368 performance of Apache that it had to be replaced. As of Apache
369 1.3, the code will relax the one-per-second rule. It will spawn
370 one, wait a second, then spawn two, wait a second, then spawn
371 four, and it will continue exponentially until it is spawning
372 32 children per second. It will stop whenever it satisfies the
373 <directive module="prefork">MinSpareServers</directive>
376 <p>This appears to be responsive enough that it's almost
377 unnecessary to twiddle the <directive module="prefork"
378 >MinSpareServers</directive>, <directive module="prefork"
379 >MaxSpareServers</directive> and <directive module="mpm_common"
380 >StartServers</directive> knobs. When more than 4 children are
381 spawned per second, a message will be emitted to the
382 <directive module="core">ErrorLog</directive>. If you
383 see a lot of these errors then consider tuning these settings.
384 Use the <module>mod_status</module> output as a guide.</p>
386 <p>Related to process creation is process death induced by the
387 <directive module="mpm_common">MaxConnectionsPerChild</directive>
388 setting. By default this is <code>0</code>,
389 which means that there is no limit to the number of connections
390 handled per child. If your configuration currently has this set
391 to some very low number, such as <code>30</code>, you may want to bump this
392 up significantly. If you are running SunOS or an old version of
393 Solaris, limit this to <code>10000</code> or so because of memory leaks.</p>
395 <p>When keep-alives are in use, children will be kept busy
396 doing nothing waiting for more requests on the already open
397 connection. The default <directive module="core"
398 >KeepAliveTimeout</directive> of <code>5</code>
399 seconds attempts to minimize this effect. The tradeoff here is
400 between network bandwidth and server resources. In no event
401 should you raise this above about <code>60</code> seconds, as <a
402 href="http://www.hpl.hp.com/techreports/Compaq-DEC/WRL-95-4.html">
403 most of the benefits are lost</a>.</p>
409 <section id="compiletime">
411 <title>Compile-Time Configuration Issues</title>
415 <title>Choosing an MPM</title>
417 <p>Apache 2.x supports pluggable concurrency models, called
418 <a href="../mpm.html">Multi-Processing Modules</a> (MPMs).
419 When building Apache, you must choose an MPM to use. There
420 are platform-specific MPMs for some platforms:
421 <module>mpm_netware</module>,
422 <module>mpmt_os2</module>, and <module>mpm_winnt</module>. For
423 general Unix-type systems, there are several MPMs from which
424 to choose. The choice of MPM can affect the speed and scalability
429 <li>The <module>worker</module> MPM uses multiple child
430 processes with many threads each. Each thread handles
431 one connection at a time. Worker generally is a good
432 choice for high-traffic servers because it has a smaller
433 memory footprint than the prefork MPM.</li>
435 <li>The <module>event</module> MPM is threaded like the
436 Worker MPM, but is designed to allow more requests to be
437 served simultaneously by passing off some processing work
438 to supporting threads, freeing up the main threads to work
439 on new requests.</li>
441 <li>The <module>prefork</module> MPM uses multiple child
442 processes with one thread each. Each process handles
443 one connection at a time. On many systems, prefork is
444 comparable in speed to worker, but it uses more memory.
445 Prefork's threadless design has advantages over worker
446 in some situations: it can be used with non-thread-safe
447 third-party modules, and it is easier to debug on platforms
448 with poor thread debugging support.</li>
452 <p>For more information on these and other MPMs, please
453 see the MPM <a href="../mpm.html">documentation</a>.</p>
457 <section id="modules">
459 <title>Modules</title>
461 <p>Since memory usage is such an important consideration in
462 performance, you should attempt to eliminate modules that you are
463 not actually using. If you have built the modules as <a
464 href="../dso.html">DSOs</a>, eliminating modules is a simple
465 matter of commenting out the associated <directive
466 module="mod_so">LoadModule</directive> directive for that module.
467 This allows you to experiment with removing modules, and seeing
468 if your site still functions in their absence.</p>
470 <p>If, on the other hand, you have modules statically linked
471 into your Apache binary, you will need to recompile Apache in
472 order to remove unwanted modules.</p>
474 <p>An associated question that arises here is, of course, what
475 modules you need, and which ones you don't. The answer here
476 will, of course, vary from one web site to another. However, the
477 <em>minimal</em> list of modules which you can get by with tends
478 to include <module>mod_mime</module>, <module>mod_dir</module>,
479 and <module>mod_log_config</module>. <code>mod_log_config</code> is,
480 of course, optional, as you can run a web site without log
481 files. This is, however, not recommended.</p>
487 <title>Atomic Operations</title>
489 <p>Some modules, such as <module>mod_cache</module> and
490 recent development builds of the worker MPM, use APR's
491 atomic API. This API provides atomic operations that can
492 be used for lightweight thread synchronization.</p>
494 <p>By default, APR implements these operations using the
495 most efficient mechanism available on each target
496 OS/CPU platform. Many modern CPUs, for example, have
497 an instruction that does an atomic compare-and-swap (CAS)
498 operation in hardware. On some platforms, however, APR
499 defaults to a slower, mutex-based implementation of the
500 atomic API in order to ensure compatibility with older
501 CPU models that lack such instructions. If you are
502 building Apache for one of these platforms, and you plan
503 to run only on newer CPUs, you can select a faster atomic
504 implementation at build time by configuring Apache with
505 the <code>--enable-nonportable-atomics</code> option:</p>
509 ./configure --with-mpm=worker --enable-nonportable-atomics=yes
512 <p>The <code>--enable-nonportable-atomics</code> option is
513 relevant for the following platforms:</p>
517 <li>Solaris on SPARC<br />
518 By default, APR uses mutex-based atomics on Solaris/SPARC.
519 If you configure with <code>--enable-nonportable-atomics</code>,
520 however, APR generates code that uses a SPARC v8plus opcode for
521 fast hardware compare-and-swap. If you configure Apache with
522 this option, the atomic operations will be more efficient
523 (allowing for lower CPU utilization and higher concurrency),
524 but the resulting executable will run only on UltraSPARC
528 <li>Linux on x86<br />
529 By default, APR uses mutex-based atomics on Linux. If you
530 configure with <code>--enable-nonportable-atomics</code>,
531 however, APR generates code that uses a 486 opcode for fast
532 hardware compare-and-swap. This will result in more efficient
533 atomic operations, but the resulting executable will run only
534 on 486 and later chips (and not on 386).
543 <title>mod_status and ExtendedStatus On</title>
545 <p>If you include <module>mod_status</module> and you also set
546 <code>ExtendedStatus On</code> when building and running
547 Apache, then on every request Apache will perform two calls to
548 <code>gettimeofday(2)</code> (or <code>times(2)</code>
549 depending on your operating system), and (pre-1.3) several
550 extra calls to <code>time(2)</code>. This is all done so that
551 the status report contains timing indications. For highest
552 performance, set <code>ExtendedStatus off</code> (which is the
559 <title>accept Serialization - multiple sockets</title>
561 <note type="warning"><title>Warning:</title>
562 <p>This section has not been fully updated
563 to take into account changes made in the 2.x version of the
564 Apache HTTP Server. Some of the information may still be
565 relevant, but please use it with care.</p>
568 <p>This discusses a shortcoming in the Unix socket API. Suppose
569 your web server uses multiple <directive module="mpm_common"
570 >Listen</directive> statements to listen on either multiple
571 ports or multiple addresses. In order to test each socket
572 to see if a connection is ready Apache uses
573 <code>select(2)</code>. <code>select(2)</code> indicates that a
574 socket has <em>zero</em> or <em>at least one</em> connection
575 waiting on it. Apache's model includes multiple children, and
576 all the idle ones test for new connections at the same time. A
577 naive implementation looks something like this (these examples
578 do not match the code, they're contrived for pedagogical
581 <highlight language="c">
586 FD_ZERO (&accept_fds);
587 for (i = first_socket; i <= last_socket; ++i) {
588 FD_SET (i, &accept_fds);
590 rc = select (last_socket+1, &accept_fds, NULL, NULL, NULL);
591 if (rc < 1) continue;
593 for (i = first_socket; i <= last_socket; ++i) {
594 if (FD_ISSET (i, &accept_fds)) {
595 new_connection = accept (i, NULL, NULL);
596 if (new_connection != -1) break;
599 if (new_connection != -1) break;
601 process_the(new_connection);
605 <p>But this naive implementation has a serious starvation problem.
606 Recall that multiple children execute this loop at the same
607 time, and so multiple children will block at
608 <code>select</code> when they are in between requests. All
609 those blocked children will awaken and return from
610 <code>select</code> when a single request appears on any socket
611 (the number of children which awaken varies depending on the
612 operating system and timing issues). They will all then fall
613 down into the loop and try to <code>accept</code> the
614 connection. But only one will succeed (assuming there's still
615 only one connection ready), the rest will be <em>blocked</em>
616 in <code>accept</code>. This effectively locks those children
617 into serving requests from that one socket and no other
618 sockets, and they'll be stuck there until enough new requests
619 appear on that socket to wake them all up. This starvation
620 problem was first documented in <a
621 href="http://bugs.apache.org/index/full/467">PR#467</a>. There
622 are at least two solutions.</p>
624 <p>One solution is to make the sockets non-blocking. In this
625 case the <code>accept</code> won't block the children, and they
626 will be allowed to continue immediately. But this wastes CPU
627 time. Suppose you have ten idle children in
628 <code>select</code>, and one connection arrives. Then nine of
629 those children will wake up, try to <code>accept</code> the
630 connection, fail, and loop back into <code>select</code>,
631 accomplishing nothing. Meanwhile none of those children are
632 servicing requests that occurred on other sockets until they
633 get back up to the <code>select</code> again. Overall this
634 solution does not seem very fruitful unless you have as many
635 idle CPUs (in a multiprocessor box) as you have idle children,
636 not a very likely situation.</p>
638 <p>Another solution, the one used by Apache, is to serialize
639 entry into the inner loop. The loop looks like this
640 (differences highlighted):</p>
642 <highlight language="c">
644 <strong>accept_mutex_on ();</strong>
648 FD_ZERO (&accept_fds);
649 for (i = first_socket; i <= last_socket; ++i) {
650 FD_SET (i, &accept_fds);
652 rc = select (last_socket+1, &accept_fds, NULL, NULL, NULL);
653 if (rc < 1) continue;
655 for (i = first_socket; i <= last_socket; ++i) {
656 if (FD_ISSET (i, &accept_fds)) {
657 new_connection = accept (i, NULL, NULL);
658 if (new_connection != -1) break;
661 if (new_connection != -1) break;
663 <strong>accept_mutex_off ();</strong>
664 process the new_connection;
668 <p><a id="serialize" name="serialize">The functions</a>
669 <code>accept_mutex_on</code> and <code>accept_mutex_off</code>
670 implement a mutual exclusion semaphore. Only one child can have
671 the mutex at any time. There are several choices for
672 implementing these mutexes. The choice is defined in
673 <code>src/conf.h</code> (pre-1.3) or
674 <code>src/include/ap_config.h</code> (1.3 or later). Some
675 architectures do not have any locking choice made, on these
676 architectures it is unsafe to use multiple
677 <directive module="mpm_common">Listen</directive>
680 <p>The <directive module="core">Mutex</directive> directive can
681 be used to change the mutex implementation of the
682 <code>mpm-accept</code> mutex at run-time. Special considerations
683 for different mutex implementations are documented with that
686 <p>Another solution that has been considered but never
687 implemented is to partially serialize the loop -- that is, let
688 in a certain number of processes. This would only be of
689 interest on multiprocessor boxes where it's possible multiple
690 children could run simultaneously, and the serialization
691 actually doesn't take advantage of the full bandwidth. This is
692 a possible area of future investigation, but priority remains
693 low because highly parallel web servers are not the norm.</p>
695 <p>Ideally you should run servers without multiple
696 <directive module="mpm_common">Listen</directive>
697 statements if you want the highest performance.
704 <title>accept Serialization - single socket</title>
706 <p>The above is fine and dandy for multiple socket servers, but
707 what about single socket servers? In theory they shouldn't
708 experience any of these same problems because all children can
709 just block in <code>accept(2)</code> until a connection
710 arrives, and no starvation results. In practice this hides
711 almost the same "spinning" behaviour discussed above in the
712 non-blocking solution. The way that most TCP stacks are
713 implemented, the kernel actually wakes up all processes blocked
714 in <code>accept</code> when a single connection arrives. One of
715 those processes gets the connection and returns to user-space,
716 the rest spin in the kernel and go back to sleep when they
717 discover there's no connection for them. This spinning is
718 hidden from the user-land code, but it's there nonetheless.
719 This can result in the same load-spiking wasteful behaviour
720 that a non-blocking solution to the multiple sockets case
723 <p>For this reason we have found that many architectures behave
724 more "nicely" if we serialize even the single socket case. So
725 this is actually the default in almost all cases. Crude
726 experiments under Linux (2.0.30 on a dual Pentium pro 166
727 w/128Mb RAM) have shown that the serialization of the single
728 socket case causes less than a 3% decrease in requests per
729 second over unserialized single-socket. But unserialized
730 single-socket showed an extra 100ms latency on each request.
731 This latency is probably a wash on long haul lines, and only an
732 issue on LANs. If you want to override the single socket
733 serialization you can define
734 <code>SINGLE_LISTEN_UNSERIALIZED_ACCEPT</code> and then
735 single-socket servers will not serialize at all.</p>
741 <title>Lingering Close</title>
743 <p>As discussed in <a
744 href="http://www.ics.uci.edu/pub/ietf/http/draft-ietf-http-connection-00.txt">
745 draft-ietf-http-connection-00.txt</a> section 8, in order for
746 an HTTP server to <strong>reliably</strong> implement the
747 protocol it needs to shutdown each direction of the
748 communication independently (recall that a TCP connection is
749 bi-directional, each half is independent of the other).</p>
751 <p>When this feature was added to Apache it caused a flurry of
752 problems on various versions of Unix because of a
753 shortsightedness. The TCP specification does not state that the
754 <code>FIN_WAIT_2</code> state has a timeout, but it doesn't prohibit it.
755 On systems without the timeout, Apache 1.2 induces many sockets
756 stuck forever in the <code>FIN_WAIT_2</code> state. In many cases this
757 can be avoided by simply upgrading to the latest TCP/IP patches
758 supplied by the vendor. In cases where the vendor has never
759 released patches (<em>i.e.</em>, SunOS4 -- although folks with
760 a source license can patch it themselves) we have decided to
761 disable this feature.</p>
763 <p>There are two ways of accomplishing this. One is the socket
764 option <code>SO_LINGER</code>. But as fate would have it, this
765 has never been implemented properly in most TCP/IP stacks. Even
766 on those stacks with a proper implementation (<em>i.e.</em>,
767 Linux 2.0.31) this method proves to be more expensive (cputime)
768 than the next solution.</p>
770 <p>For the most part, Apache implements this in a function
771 called <code>lingering_close</code> (in
772 <code>http_main.c</code>). The function looks roughly like
775 <highlight language="c">
776 void lingering_close (int s)
778 char junk_buffer[2048];
780 /* shutdown the sending side */
783 signal (SIGALRM, lingering_death);
787 select (s for reading, 2 second timeout);
789 if (s is ready for reading) {
790 if (read (s, junk_buffer, sizeof (junk_buffer)) <= 0) {
793 /* just toss away whatever is here */
801 <p>This naturally adds some expense at the end of a connection,
802 but it is required for a reliable implementation. As HTTP/1.1
803 becomes more prevalent, and all connections are persistent,
804 this expense will be amortized over more requests. If you want
805 to play with fire and disable this feature you can define
806 <code>NO_LINGCLOSE</code>, but this is not recommended at all.
807 In particular, as HTTP/1.1 pipelined persistent connections
808 come into use <code>lingering_close</code> is an absolute
810 href="http://www.w3.org/Protocols/HTTP/Performance/Pipeline.html">
811 pipelined connections are faster</a>, so you want to support
818 <title>Scoreboard File</title>
820 <p>Apache's parent and children communicate with each other
821 through something called the scoreboard. Ideally this should be
822 implemented in shared memory. For those operating systems that
823 we either have access to, or have been given detailed ports
824 for, it typically is implemented using shared memory. The rest
825 default to using an on-disk file. The on-disk file is not only
826 slow, but it is unreliable (and less featured). Peruse the
827 <code>src/main/conf.h</code> file for your architecture and
828 look for either <code>USE_MMAP_SCOREBOARD</code> or
829 <code>USE_SHMGET_SCOREBOARD</code>. Defining one of those two
830 (as well as their companions <code>HAVE_MMAP</code> and
831 <code>HAVE_SHMGET</code> respectively) enables the supplied
832 shared memory code. If your system has another type of shared
833 memory, edit the file <code>src/main/http_main.c</code> and add
834 the hooks necessary to use it in Apache. (Send us back a patch
837 <note>Historical note: The Linux port of Apache didn't start to
838 use shared memory until version 1.2 of Apache. This oversight
839 resulted in really poor and unreliable behaviour of earlier
840 versions of Apache on Linux.</note>
846 <title>DYNAMIC_MODULE_LIMIT</title>
848 <p>If you have no intention of using dynamically loaded modules
849 (you probably don't if you're reading this and tuning your
850 server for every last ounce of performance) then you should add
851 <code>-DDYNAMIC_MODULE_LIMIT=0</code> when building your
852 server. This will save RAM that's allocated only for supporting
853 dynamically loaded modules.</p>
861 <title>Appendix: Detailed Analysis of a Trace</title>
863 <p>Here is a system call trace of Apache 2.0.38 with the worker MPM
864 on Solaris 8. This trace was collected using:</p>
867 truss -l -p <var>httpd_child_pid</var>.
870 <p>The <code>-l</code> option tells truss to log the ID of the
871 LWP (lightweight process--Solaris' form of kernel-level thread)
872 that invokes each system call.</p>
874 <p>Other systems may have different system call tracing utilities
875 such as <code>strace</code>, <code>ktrace</code>, or <code>par</code>.
876 They all produce similar output.</p>
878 <p>In this trace, a client has requested a 10KB static file
879 from the httpd. Traces of non-static requests or requests
880 with content negotiation look wildly different (and quite ugly
884 <pre>/67: accept(3, 0x00200BEC, 0x00200C0C, 1) (sleeping...)
885 /67: accept(3, 0x00200BEC, 0x00200C0C, 1) = 9</pre>
888 <p>In this trace, the listener thread is running within LWP #67.</p>
890 <note>Note the lack of <code>accept(2)</code> serialization. On this
891 particular platform, the worker MPM uses an unserialized accept by
892 default unless it is listening on multiple ports.</note>
895 <pre>/65: lwp_park(0x00000000, 0) = 0
896 /67: lwp_unpark(65, 1) = 0</pre>
899 <p>Upon accepting the connection, the listener thread wakes up
900 a worker thread to do the request processing. In this trace,
901 the worker thread that handles the request is mapped to LWP #65.</p>
904 <pre>/65: getsockname(9, 0x00200BA4, 0x00200BC4, 1) = 0</pre>
907 <p>In order to implement virtual hosts, Apache needs to know
908 the local socket address used to accept the connection. It
909 is possible to eliminate this call in many situations (such
910 as when there are no virtual hosts, or when
911 <directive module="mpm_common">Listen</directive> directives
912 are used which do not have wildcard addresses). But
913 no effort has yet been made to do these optimizations. </p>
916 <pre>/65: brk(0x002170E8) = 0
917 /65: brk(0x002190E8) = 0</pre>
920 <p>The <code>brk(2)</code> calls allocate memory from the heap.
921 It is rare to see these in a system call trace, because the httpd
922 uses custom memory allocators (<code>apr_pool</code> and
923 <code>apr_bucket_alloc</code>) for most request processing.
924 In this trace, the httpd has just been started, so it must
925 call <code>malloc(3)</code> to get the blocks of raw memory
926 with which to create the custom memory allocators.</p>
929 <pre>/65: fcntl(9, F_GETFL, 0x00000000) = 2
930 /65: fstat64(9, 0xFAF7B818) = 0
931 /65: getsockopt(9, 65535, 8192, 0xFAF7B918, 0xFAF7B910, 2190656) = 0
932 /65: fstat64(9, 0xFAF7B818) = 0
933 /65: getsockopt(9, 65535, 8192, 0xFAF7B918, 0xFAF7B914, 2190656) = 0
934 /65: setsockopt(9, 65535, 8192, 0xFAF7B918, 4, 2190656) = 0
935 /65: fcntl(9, F_SETFL, 0x00000082) = 0</pre>
938 <p>Next, the worker thread puts the connection to the client (file
939 descriptor 9) in non-blocking mode. The <code>setsockopt(2)</code>
940 and <code>getsockopt(2)</code> calls are a side-effect of how
941 Solaris' libc handles <code>fcntl(2)</code> on sockets.</p>
944 <pre>/65: read(9, " G E T / 1 0 k . h t m".., 8000) = 97</pre>
947 <p>The worker thread reads the request from the client.</p>
950 <pre>/65: stat("/var/httpd/apache/httpd-8999/htdocs/10k.html", 0xFAF7B978) = 0
951 /65: open("/var/httpd/apache/httpd-8999/htdocs/10k.html", O_RDONLY) = 10</pre>
954 <p>This httpd has been configured with <code>Options FollowSymLinks</code>
955 and <code>AllowOverride None</code>. Thus it doesn't need to
956 <code>lstat(2)</code> each directory in the path leading up to the
957 requested file, nor check for <code>.htaccess</code> files.
958 It simply calls <code>stat(2)</code> to verify that the file:
959 1) exists, and 2) is a regular file, not a directory.</p>
962 <pre>/65: sendfilev(0, 9, 0x00200F90, 2, 0xFAF7B53C) = 10269</pre>
965 <p>In this example, the httpd is able to send the HTTP response
966 header and the requested file with a single <code>sendfilev(2)</code>
967 system call. Sendfile semantics vary among operating systems. On some other
968 systems, it is necessary to do a <code>write(2)</code> or
969 <code>writev(2)</code> call to send the headers before calling
970 <code>sendfile(2)</code>.</p>
973 <pre>/65: write(4, " 1 2 7 . 0 . 0 . 1 - ".., 78) = 78</pre>
976 <p>This <code>write(2)</code> call records the request in the
977 access log. Note that one thing missing from this trace is a
978 <code>time(2)</code> call. Unlike Apache 1.3, Apache 2.x uses
979 <code>gettimeofday(3)</code> to look up the time. On some operating
980 systems, like Linux or Solaris, <code>gettimeofday</code> has an
981 optimized implementation that doesn't require as much overhead
982 as a typical system call.</p>
985 <pre>/65: shutdown(9, 1, 1) = 0
986 /65: poll(0xFAF7B980, 1, 2000) = 1
987 /65: read(9, 0xFAF7BC20, 512) = 0
988 /65: close(9) = 0</pre>
991 <p>The worker thread does a lingering close of the connection.</p>
994 <pre>/65: close(10) = 0
995 /65: lwp_park(0x00000000, 0) (sleeping...)</pre>
998 <p>Finally the worker thread closes the file that it has just delivered
999 and blocks until the listener assigns it another connection.</p>
1002 <pre>/67: accept(3, 0x001FEB74, 0x001FEB94, 1) (sleeping...)</pre>
1005 <p>Meanwhile, the listener thread is able to accept another connection
1006 as soon as it has dispatched this connection to a worker thread (subject
1007 to some flow-control logic in the worker MPM that throttles the listener
1008 if all the available workers are busy). Though it isn't apparent from
1009 this trace, the next <code>accept(2)</code> can (and usually does, under
1010 high load conditions) occur in parallel with the worker thread's handling
1011 of the just-accepted connection.</p>