1 <?xml version="1.0" encoding="UTF-8" ?>
2 <!DOCTYPE manualpage SYSTEM "./style/manualpage.dtd">
3 <?xml-stylesheet type="text/xsl" href="./style/manual.en.xsl"?>
4 <!-- $LastChangedRevision$ -->
7 Licensed to the Apache Software Foundation (ASF) under one or more
8 contributor license agreements. See the NOTICE file distributed with
9 this work for additional information regarding copyright ownership.
10 The ASF licenses this file to You under the Apache License, Version 2.0
11 (the "License"); you may not use this file except in compliance with
12 the License. You may obtain a copy of the License at
14 http://www.apache.org/licenses/LICENSE-2.0
16 Unless required by applicable law or agreed to in writing, software
17 distributed under the License is distributed on an "AS IS" BASIS,
18 WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
19 See the License for the specific language governing permissions and
20 limitations under the License.
23 <manualpage metafile="logs.xml.meta">
25 <title>Log Files</title>
28 <p>In order to effectively manage a web server, it is necessary
29 to get feedback about the activity and performance of the
30 server as well as any problems that may be occurring. The Apache HTTP Server
31 provides very comprehensive and flexible logging
32 capabilities. This document describes how to configure its
33 logging capabilities, and how to understand what the logs
37 <section id="overview">
38 <title>Overview</title>
42 <module>mod_log_config</module>
43 <module>mod_log_forensic</module>
44 <module>mod_logio</module>
45 <module>mod_rewrite</module>
46 <module>mod_cgi</module>
51 The Apache HTTP Server provides a variety of different mechanisms for
52 logging everything that happens on your server, from the initial
53 request, through the URL mapping process, to the final resolution of
54 the connection, including any errors that may have occurred in the
55 process. In addition to this, third-party modules may provide logging
56 capabilities, or inject entries into the existing log files, and
57 applications such as CGI programs, or PHP scripts, or other handlers,
58 may send messages to the server error log.
62 In this document we discuss the logging modules that are a standard
63 part of the http server.
68 <section id="security">
69 <title>Security Warning</title>
71 <p>Anyone who can write to the directory where Apache httpd is
72 writing a log file can almost certainly gain access to the uid
73 that the server is started as, which is normally root. Do
74 <em>NOT</em> give people write access to the directory the logs
75 are stored in without being aware of the consequences; see the
76 <a href="misc/security_tips.html">security tips</a> document
79 <p>In addition, log files may contain information supplied
80 directly by the client, without escaping. Therefore, it is
81 possible for malicious clients to insert control-characters in
82 the log files, so care must be taken in dealing with raw
86 <section id="errorlog">
87 <title>Error Log</title>
94 <directive module="core">ErrorLog</directive>
95 <directive module="core">LogLevel</directive>
99 <p>The server error log, whose name and location is set by the
100 <directive module="core">ErrorLog</directive> directive, is the
101 most important log file. This is the place where Apache httpd
102 will send diagnostic information and record any errors that it
103 encounters in processing requests. It is the first place to
104 look when a problem occurs with starting the server or with the
105 operation of the server, since it will often contain details of
106 what went wrong and how to fix it.</p>
108 <p>The error log is usually written to a file (typically
109 <code>error_log</code> on Unix systems and
110 <code>error.log</code> on Windows and OS/2). On Unix systems it
111 is also possible to have the server send errors to
112 <code>syslog</code> or <a href="#piped">pipe them to a
115 <p>The format of the error log is relatively free-form and
116 descriptive. But there is certain information that is contained
117 in most error log entries. For example, here is a typical
121 [Wed Oct 11 14:32:52 2000] [error] [client 127.0.0.1]
122 client denied by server configuration:
123 /export/home/live/ap/htdocs/test
126 <p>The first item in the log entry is the date and time of the
127 message. The second item lists the severity of the error being
128 reported. The <directive module="core">LogLevel</directive>
129 directive is used to control the types of errors that are sent
130 to the error log by restricting the severity level. The third
131 item gives the IP address of the client that generated the
132 error. Beyond that is the message itself, which in this case
133 indicates that the server has been configured to deny the
134 client access. The server reports the file-system path (as
135 opposed to the web path) of the requested document.</p>
137 <p>A very wide variety of different messages can appear in the
138 error log. Most look similar to the example above. The error
139 log will also contain debugging output from CGI scripts. Any
140 information written to <code>stderr</code> by a CGI script will
141 be copied directly to the error log.</p>
143 <p>It is not possible to customize the error log by adding or
144 removing information. However, error log entries dealing with
145 particular requests have corresponding entries in the <a
146 href="#accesslog">access log</a>. For example, the above example
147 entry corresponds to an access log entry with status code 403.
148 Since it is possible to customize the access log, you can
149 obtain more information about error conditions using that log
152 <p>During testing, it is often useful to continuously monitor
153 the error log for any problems. On Unix systems, you can
154 accomplish this using:</p>
161 <section id="accesslog">
162 <title>Access Log</title>
166 <module>mod_log_config</module>
167 <module>mod_setenvif</module>
170 <directive module="mod_log_config">CustomLog</directive>
171 <directive module="mod_log_config">LogFormat</directive>
172 <directive module="mod_setenvif">SetEnvIf</directive>
176 <p>The server access log records all requests processed by the
177 server. The location and content of the access log are
178 controlled by the <directive module="mod_log_config">CustomLog</directive>
179 directive. The <directive module="mod_log_config">LogFormat</directive>
180 directive can be used to simplify the selection of
181 the contents of the logs. This section describes how to configure the server
182 to record information in the access log.</p>
184 <p>Of course, storing the information in the access log is only
185 the start of log management. The next step is to analyze this
186 information to produce useful statistics. Log analysis in
187 general is beyond the scope of this document, and not really
188 part of the job of the web server itself. For more information
189 about this topic, and for applications which perform log
190 analysis, check the <a
191 href="http://dmoz.org/Computers/Software/Internet/Site_Management/Log_analysis/">
192 Open Directory</a> or <a
193 href="http://dir.yahoo.com/Computers_and_Internet/Software/Internet/World_Wide_Web/Servers/Log_Analysis_Tools/">
196 <p>Various versions of Apache httpd have used other modules and
197 directives to control access logging, including
198 mod_log_referer, mod_log_agent, and the
199 <code>TransferLog</code> directive. The <directive
200 module="mod_log_config">CustomLog</directive> directive now subsumes
201 the functionality of all the older directives.</p>
203 <p>The format of the access log is highly configurable. The format
204 is specified using a format string that looks much like a C-style
205 printf(1) format string. Some examples are presented in the next
206 sections. For a complete list of the possible contents of the
207 format string, see the <module>mod_log_config</module> <a
208 href="mod/mod_log_config.html#formats">format strings</a>.</p>
210 <section id="common">
211 <title>Common Log Format</title>
213 <p>A typical configuration for the access log might look as
217 LogFormat "%h %l %u %t \"%r\" %>s %b" common<br />
218 CustomLog logs/access_log common
221 <p>This defines the <em>nickname</em> <code>common</code> and
222 associates it with a particular log format string. The format
223 string consists of percent directives, each of which tell the
224 server to log a particular piece of information. Literal
225 characters may also be placed in the format string and will be
226 copied directly into the log output. The quote character
227 (<code>"</code>) must be escaped by placing a back-slash before
228 it to prevent it from being interpreted as the end of the
229 format string. The format string may also contain the special
230 control characters "<code>\n</code>" for new-line and
231 "<code>\t</code>" for tab.</p>
233 <p>The <directive module="mod_log_config">CustomLog</directive>
234 directive sets up a new log file using the defined
235 <em>nickname</em>. The filename for the access log is relative to
236 the <directive module="core">ServerRoot</directive> unless it
237 begins with a slash.</p>
239 <p>The above configuration will write log entries in a format
240 known as the Common Log Format (CLF). This standard format can
241 be produced by many different web servers and read by many log
242 analysis programs. The log file entries produced in CLF will
243 look something like this:</p>
246 127.0.0.1 - frank [10/Oct/2000:13:55:36 -0700] "GET
247 /apache_pb.gif HTTP/1.0" 200 2326
250 <p>Each part of this log entry is described below.</p>
253 <dt><code>127.0.0.1</code> (<code>%h</code>)</dt>
255 <dd>This is the IP address of the client (remote host) which
256 made the request to the server. If <directive
257 module="core">HostnameLookups</directive> is
258 set to <code>On</code>, then the server will try to determine
259 the hostname and log it in place of the IP address. However,
260 this configuration is not recommended since it can
261 significantly slow the server. Instead, it is best to use a
262 log post-processor such as <program>logresolve</program> to determine
263 the hostnames. The IP address reported here is not
264 necessarily the address of the machine at which the user is
265 sitting. If a proxy server exists between the user and the
266 server, this address will be the address of the proxy, rather
267 than the originating machine.</dd>
269 <dt><code>-</code> (<code>%l</code>)</dt>
271 <dd>The "hyphen" in the output indicates that the requested
272 piece of information is not available. In this case, the
273 information that is not available is the RFC 1413 identity of
274 the client determined by <code>identd</code> on the clients
275 machine. This information is highly unreliable and should
276 almost never be used except on tightly controlled internal
277 networks. Apache httpd will not even attempt to determine
278 this information unless <directive
279 module="core">IdentityCheck</directive> is set
280 to <code>On</code>.</dd>
282 <dt><code>frank</code> (<code>%u</code>)</dt>
284 <dd>This is the userid of the person requesting the document
285 as determined by HTTP authentication. The same value is
286 typically provided to CGI scripts in the
287 <code>REMOTE_USER</code> environment variable. If the status
288 code for the request (see below) is 401, then this value
289 should not be trusted because the user is not yet
290 authenticated. If the document is not password protected,
291 this part will be "<code>-</code>" just like the previous
294 <dt><code>[10/Oct/2000:13:55:36 -0700]</code>
295 (<code>%t</code>)</dt>
298 The time that the request was received.
302 <code>[day/month/year:hour:minute:second zone]<br />
304 month = 3*letter<br />
307 minute = 2*digit<br />
308 second = 2*digit<br />
309 zone = (`+' | `-') 4*digit</code>
311 It is possible to have the time displayed in another format
312 by specifying <code>%{format}t</code> in the log format
313 string, where <code>format</code> is as in
314 <code>strftime(3)</code> from the C standard library.
317 <dt><code>"GET /apache_pb.gif HTTP/1.0"</code>
318 (<code>\"%r\"</code>)</dt>
320 <dd>The request line from the client is given in double
321 quotes. The request line contains a great deal of useful
322 information. First, the method used by the client is
323 <code>GET</code>. Second, the client requested the resource
324 <code>/apache_pb.gif</code>, and third, the client used the
325 protocol <code>HTTP/1.0</code>. It is also possible to log
326 one or more parts of the request line independently. For
327 example, the format string "<code>%m %U%q %H</code>" will log
328 the method, path, query-string, and protocol, resulting in
329 exactly the same output as "<code>%r</code>".</dd>
331 <dt><code>200</code> (<code>%>s</code>)</dt>
333 <dd>This is the status code that the server sends back to the
334 client. This information is very valuable, because it reveals
335 whether the request resulted in a successful response (codes
336 beginning in 2), a redirection (codes beginning in 3), an
337 error caused by the client (codes beginning in 4), or an
338 error in the server (codes beginning in 5). The full list of
339 possible status codes can be found in the <a
340 href="http://www.w3.org/Protocols/rfc2616/rfc2616.txt">HTTP
341 specification</a> (RFC2616 section 10).</dd>
343 <dt><code>2326</code> (<code>%b</code>)</dt>
345 <dd>The last part indicates the size of the object returned
346 to the client, not including the response headers. If no
347 content was returned to the client, this value will be
348 "<code>-</code>". To log "<code>0</code>" for no content, use
349 <code>%B</code> instead.</dd>
353 <section id="combined">
354 <title>Combined Log Format</title>
356 <p>Another commonly used format string is called the Combined
357 Log Format. It can be used as follows.</p>
360 LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\"
361 \"%{User-agent}i\"" combined<br />
362 CustomLog log/access_log combined
365 <p>This format is exactly the same as the Common Log Format,
366 with the addition of two more fields. Each of the additional
367 fields uses the percent-directive
368 <code>%{<em>header</em>}i</code>, where <em>header</em> can be
369 any HTTP request header. The access log under this format will
373 127.0.0.1 - frank [10/Oct/2000:13:55:36 -0700] "GET
374 /apache_pb.gif HTTP/1.0" 200 2326
375 "http://www.example.com/start.html" "Mozilla/4.08 [en]
379 <p>The additional fields are:</p>
382 <dt><code>"http://www.example.com/start.html"</code>
383 (<code>\"%{Referer}i\"</code>)</dt>
385 <dd>The "Referer" (sic) HTTP request header. This gives the
386 site that the client reports having been referred from. (This
387 should be the page that links to or includes
388 <code>/apache_pb.gif</code>).</dd>
390 <dt><code>"Mozilla/4.08 [en] (Win98; I ;Nav)"</code>
391 (<code>\"%{User-agent}i\"</code>)</dt>
393 <dd>The User-Agent HTTP request header. This is the
394 identifying information that the client browser reports about
399 <section id="multiple">
400 <title>Multiple Access Logs</title>
402 <p>Multiple access logs can be created simply by specifying
403 multiple <directive module="mod_log_config">CustomLog</directive>
404 directives in the configuration
405 file. For example, the following directives will create three
406 access logs. The first contains the basic CLF information,
407 while the second and third contain referer and browser
408 information. The last two <directive
409 module="mod_log_config">CustomLog</directive> lines show how
410 to mimic the effects of the <code>ReferLog</code> and <code
411 >AgentLog</code> directives.</p>
414 LogFormat "%h %l %u %t \"%r\" %>s %b" common<br />
415 CustomLog logs/access_log common<br />
416 CustomLog logs/referer_log "%{Referer}i -> %U"<br />
417 CustomLog logs/agent_log "%{User-agent}i"
420 <p>This example also shows that it is not necessary to define a
421 nickname with the <directive
422 module="mod_log_config">LogFormat</directive> directive. Instead,
423 the log format can be specified directly in the <directive
424 module="mod_log_config">CustomLog</directive> directive.</p>
427 <section id="conditional">
428 <title>Conditional Logs</title>
430 <p>There are times when it is convenient to exclude certain
431 entries from the access logs based on characteristics of the
432 client request. This is easily accomplished with the help of <a
433 href="env.html">environment variables</a>. First, an
434 environment variable must be set to indicate that the request
435 meets certain conditions. This is usually accomplished with
436 <directive module="mod_setenvif">SetEnvIf</directive>. Then the
437 <code>env=</code> clause of the <directive
438 module="mod_log_config">CustomLog</directive> directive is used to
439 include or exclude requests where the environment variable is
440 set. Some examples:</p>
443 # Mark requests from the loop-back interface<br />
444 SetEnvIf Remote_Addr "127\.0\.0\.1" dontlog<br />
445 # Mark requests for the robots.txt file<br />
446 SetEnvIf Request_URI "^/robots\.txt$" dontlog<br />
447 # Log what remains<br />
448 CustomLog logs/access_log common env=!dontlog
451 <p>As another example, consider logging requests from
452 english-speakers to one log file, and non-english speakers to a
453 different log file.</p>
456 SetEnvIf Accept-Language "en" english<br />
457 CustomLog logs/english_log common env=english<br />
458 CustomLog logs/non_english_log common env=!english
461 <p>In a caching scenario one would want to know about
462 the efficiency of the cache. A very simple method to
463 find this out would be:</p>
466 SetEnv CACHE_MISS 1<br />
467 LogFormat "%h %l %u %t "%r " %>s %b %{CACHE_MISS}e" common-cache<br />
468 CustomLog logs/access_log common-cache
471 <p><module>mod_cache</module> will run before
472 <module>mod_env</module> and when successfull will deliver the
473 content without it. In that case a cache hit will log
474 <code>-</code>, while a cache miss will log <code>1</code>.</p>
476 <p>Although we have just shown that conditional logging is very
477 powerful and flexible, it is not the only way to control the
478 contents of the logs. Log files are more useful when they
479 contain a complete record of server activity. It is often
480 easier to simply post-process the log files to remove requests
481 that you do not want to consider.</p>
485 <section id="rotation">
486 <title>Log Rotation</title>
488 <p>On even a moderately busy server, the quantity of
489 information stored in the log files is very large. The access
490 log file typically grows 1 MB or more per 10,000 requests. It
491 will consequently be necessary to periodically rotate the log
492 files by moving or deleting the existing logs. This cannot be
493 done while the server is running, because Apache httpd will continue
494 writing to the old log file as long as it holds the file open.
495 Instead, the server must be <a
496 href="stopping.html">restarted</a> after the log files are
497 moved or deleted so that it will open new log files.</p>
499 <p>By using a <em>graceful</em> restart, the server can be
500 instructed to open new log files without losing any existing or
501 pending connections from clients. However, in order to
502 accomplish this, the server must continue to write to the old
503 log files while it finishes serving old requests. It is
504 therefore necessary to wait for some time after the restart
505 before doing any processing on the log files. A typical
506 scenario that simply rotates the logs and compresses the old
507 logs to save space is:</p>
510 mv access_log access_log.old<br />
511 mv error_log error_log.old<br />
512 apachectl graceful<br />
514 gzip access_log.old error_log.old
517 <p>Another way to perform log rotation is using <a
518 href="#piped">piped logs</a> as discussed in the next
523 <title>Piped Logs</title>
525 <p>Apache httpd is capable of writing error and access log
526 files through a pipe to another process, rather than directly
527 to a file. This capability dramatically increases the
528 flexibility of logging, without adding code to the main server.
529 In order to write logs to a pipe, simply replace the filename
530 with the pipe character "<code>|</code>", followed by the name
531 of the executable which should accept log entries on its
532 standard input. The server will start the piped-log process when
533 the server starts, and will restart it if it crashes while the
534 server is running. (This last feature is why we can refer to
535 this technique as "reliable piped logging".)</p>
537 <p>Piped log processes are spawned by the parent Apache httpd
538 process, and inherit the userid of that process. This means
539 that piped log programs usually run as root. It is therefore
540 very important to keep the programs simple and secure.</p>
542 <p>One important use of piped logs is to allow log rotation
543 without having to restart the server. The Apache HTTP Server
544 includes a simple program called <program>rotatelogs</program>
545 for this purpose. For example, to rotate the logs every 24 hours, you
549 CustomLog "|/usr/local/apache/bin/rotatelogs
550 /var/log/access_log 86400" common
553 <p>Notice that quotes are used to enclose the entire command
554 that will be called for the pipe. Although these examples are
555 for the access log, the same technique can be used for the
558 <p>A similar but much more flexible log rotation program
559 called <a href="http://www.cronolog.org/">cronolog</a>
560 is available at an external site.</p>
562 <p>As with conditional logging, piped logs are a very powerful
563 tool, but they should not be used where a simpler solution like
564 off-line post-processing is available.</p>
567 <section id="virtualhost">
568 <title>Virtual Hosts</title>
570 <p>When running a server with many <a href="vhosts/">virtual
571 hosts</a>, there are several options for dealing with log
572 files. First, it is possible to use logs exactly as in a
573 single-host server. Simply by placing the logging directives
574 outside the <directive module="core"
575 type="section">VirtualHost</directive> sections in the
576 main server context, it is possible to log all requests in the
577 same access log and error log. This technique does not allow
578 for easy collection of statistics on individual virtual
581 <p>If <directive module="mod_log_config">CustomLog</directive>
582 or <directive module="core">ErrorLog</directive>
583 directives are placed inside a
584 <directive module="core" type="section">VirtualHost</directive>
585 section, all requests or errors for that virtual host will be
586 logged only to the specified file. Any virtual host which does
587 not have logging directives will still have its requests sent
588 to the main server logs. This technique is very useful for a
589 small number of virtual hosts, but if the number of hosts is
590 very large, it can be complicated to manage. In addition, it
591 can often create problems with <a
592 href="vhosts/fd-limits.html">insufficient file
595 <p>For the access log, there is a very good compromise. By
596 adding information on the virtual host to the log format
597 string, it is possible to log all hosts to the same log, and
598 later split the log into individual files. For example,
599 consider the following directives.</p>
602 LogFormat "%v %l %u %t \"%r\" %>s %b"
604 CustomLog logs/access_log comonvhost
607 <p>The <code>%v</code> is used to log the name of the virtual
608 host that is serving the request. Then a program like <a
609 href="programs/other.html">split-logfile</a> can be used to
610 post-process the access log in order to split it into one file
611 per virtual host.</p>
615 <title>Other Log Files</title>
619 <module>mod_logio</module>
620 <module>mod_log_forensic</module>
621 <module>mod_cgi</module>
622 <module>mod_rewrite</module>
626 <directive module="mod_log_config">LogFormat</directive>
627 <directive module="mod_log_forensic">ForensicLog</directive>
628 <directive module="mpm_common">PidFile</directive>
629 <directive module="mod_rewrite">RewriteLog</directive>
630 <directive module="mod_rewrite">RewriteLogLevel</directive>
631 <directive module="mod_cgi">ScriptLog</directive>
632 <directive module="mod_cgi">ScriptLogBuffer</directive>
633 <directive module="mod_cgi">ScriptLogLength</directive>
638 <title>Logging actual bytes sent and received</title>
640 <p><module>mod_logio</module> adds in two additional
641 <directive module="mod_log_config">LogFormat</directive> fields
642 (%I and %O) that log the actual number of bytes received and sent
647 <title>Forensic Logging</title>
649 <p><module>mod_log_forensic</module> provides for forensic logging of
650 client requests. Logging is done before and after processing a
651 request, so the forensic log contains two log lines for each
652 request. The forensic logger is very strict with no customizations.
653 It can be an invaluable debugging and security tool.</p>
656 <section id="pidfile">
657 <title>PID File</title>
659 <p>On startup, Apache httpd saves the process id of the parent
660 httpd process to the file <code>logs/httpd.pid</code>. This
661 filename can be changed with the <directive
662 module="mpm_common">PidFile</directive> directive. The
663 process-id is for use by the administrator in restarting and
664 terminating the daemon by sending signals to the parent
665 process; on Windows, use the -k command line option instead.
666 For more information see the <a href="stopping.html">Stopping
667 and Restarting</a> page.</p>
670 <section id="scriptlog">
671 <title>Script Log</title>
673 <p>In order to aid in debugging, the
674 <directive module="mod_cgi">ScriptLog</directive> directive
675 allows you to record the input to and output from CGI scripts.
676 This should only be used in testing - not for live servers.
677 More information is available in the <a
678 href="mod/mod_cgi.html">mod_cgi</a> documentation.</p>
681 <section id="rewritelog">
682 <title>Rewrite Log</title>
684 <p>When using the powerful and complex features of <a
685 href="mod/mod_rewrite.html">mod_rewrite</a>, it is almost
686 always necessary to use the <directive
687 module="mod_rewrite">RewriteLog</directive> to help
688 in debugging. This log file produces a detailed analysis of how
689 the rewriting engine transforms requests. The level of detail
690 is controlled by the <directive
691 module="mod_rewrite">RewriteLogLevel</directive> directive.</p>