1 <?xml version="1.0" encoding="UTF-8" ?>
2 <!DOCTYPE manualpage SYSTEM "./style/manualpage.dtd">
3 <?xml-stylesheet type="text/xsl" href="./style/manual.en.xsl"?>
4 <!-- $LastChangedRevision$ -->
7 Copyright 2002-2004 The Apache Software Foundation
9 Licensed under the Apache License, Version 2.0 (the "License");
10 you may not use this file except in compliance with the License.
11 You may obtain a copy of the License at
13 http://www.apache.org/licenses/LICENSE-2.0
15 Unless required by applicable law or agreed to in writing, software
16 distributed under the License is distributed on an "AS IS" BASIS,
17 WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
18 See the License for the specific language governing permissions and
19 limitations under the License.
22 <manualpage metafile="logs.xml.meta">
24 <title>Log Files</title>
27 <p>In order to effectively manage a web server, it is necessary
28 to get feedback about the activity and performance of the
29 server as well as any problems that may be occurring. The Apache
30 HTTP Server provides very comprehensive and flexible logging
31 capabilities. This document describes how to configure its
32 logging capabilities, and how to understand what the logs
36 <section id="security">
37 <title>Security Warning</title>
39 <p>Anyone who can write to the directory where Apache is
40 writing a log file can almost certainly gain access to the uid
41 that the server is started as, which is normally root. Do
42 <em>NOT</em> give people write access to the directory the logs
43 are stored in without being aware of the consequences; see the
44 <a href="misc/security_tips.html">security tips</a> document
47 <p>In addition, log files may contain information supplied
48 directly by the client, without escaping. Therefore, it is
49 possible for malicious clients to insert control-characters in
50 the log files, so care must be taken in dealing with raw
54 <section id="errorlog">
55 <title>Error Log</title>
59 <directive module="core">ErrorLog</directive>
60 <directive module="core">LogLevel</directive>
64 <p>The server error log, whose name and location is set by the
65 <directive module="core">ErrorLog</directive> directive, is the
66 most important log file. This is the place where Apache httpd
67 will send diagnostic information and record any errors that it
68 encounters in processing requests. It is the first place to
69 look when a problem occurs with starting the server or with the
70 operation of the server, since it will often contain details of
71 what went wrong and how to fix it.</p>
73 <p>The error log is usually written to a file (typically
74 <code>error_log</code> on unix systems and
75 <code>error.log</code> on Windows and OS/2). On unix systems it
76 is also possible to have the server send errors to
77 <code>syslog</code> or <a href="#piped">pipe them to a
80 <p>The format of the error log is relatively free-form and
81 descriptive. But there is certain information that is contained
82 in most error log entries. For example, here is a typical
86 [Wed Oct 11 14:32:52 2000] [error] [client 127.0.0.1]
87 client denied by server configuration:
88 /export/home/live/ap/htdocs/test
91 <p>The first item in the log entry is the date and time of the
92 message. The second entry lists the severity of the error being
93 reported. The <directive module="core">LogLevel</directive>
94 directive is used to control the types of errors that are sent
95 to the error log by restricting the severity level. The third
96 entry gives the IP address of the client that generated the
97 error. Beyond that is the message itself, which in this case
98 indicates that the server has been configured to deny the
99 client access. The server reports the file-system path (as
100 opposed to the web path) of the requested document.</p>
102 <p>A very wide variety of different messages can appear in the
103 error log. Most look similar to the example above. The error
104 log will also contain debugging output from CGI scripts. Any
105 information written to <code>stderr</code> by a CGI script will
106 be copied directly to the error log.</p>
108 <p>It is not possible to customize the error log by adding or
109 removing information. However, error log entries dealing with
110 particular requests have corresponding entries in the <a
111 href="#accesslog">access log</a>. For example, the above example
112 entry corresponds to an access log entry with status code 403.
113 Since it is possible to customize the access log, you can
114 obtain more information about error conditions using that log
117 <p>During testing, it is often useful to continuously monitor
118 the error log for any problems. On unix systems, you can
119 accomplish this using:</p>
126 <section id="accesslog">
127 <title>Access Log</title>
131 <module>mod_log_config</module>
132 <module>mod_setenvif</module>
135 <directive module="mod_log_config">CustomLog</directive>
136 <directive module="mod_log_config">LogFormat</directive>
137 <directive module="mod_setenvif">SetEnvIf</directive>
141 <p>The server access log records all requests processed by the
142 server. The location and content of the access log are
143 controlled by the <directive module="mod_log_config">CustomLog</directive>
144 directive. The <directive module="mod_log_config">LogFormat</directive>
145 directive can be used to simplify the selection of
146 the contents of the logs. This section describes how to configure the server
147 to record information in the access log.</p>
149 <p>Of course, storing the information in the access log is only
150 the start of log management. The next step is to analyze this
151 information to produce useful statistics. Log analysis in
152 general is beyond the scope of this document, and not really
153 part of the job of the web server itself. For more information
154 about this topic, and for applications which perform log
155 analysis, check the <a
156 href="http://dmoz.org/Computers/Software/Internet/Site_Management/Log_analysis/">
157 Open Directory</a> or <a
158 href="http://dir.yahoo.com/Computers_and_Internet/Software/Internet/World_Wide_Web/Servers/Log_Analysis_Tools/">
161 <p>Various versions of Apache httpd have used other modules and
162 directives to control access logging, including
163 mod_log_referer, mod_log_agent, and the
164 <code>TransferLog</code> directive. The <directive
165 module="mod_log_config">CustomLog</directive> directive now subsumes
166 the functionality of all the older directives.</p>
168 <p>The format of the access log is highly configurable. The format
169 is specified using a format string that looks much like a C-style
170 printf(1) format string. Some examples are presented in the next
171 sections. For a complete list of the possible contents of the
172 format string, see the <module>mod_log_config</module> <a
173 href="mod/mod_log_config.html#formats">format strings</a>.</p>
175 <section id="common">
176 <title>Common Log Format</title>
178 <p>A typical configuration for the access log might look as
182 LogFormat "%h %l %u %t \"%r\" %>s %b" common<br />
183 CustomLog logs/access_log common
186 <p>This defines the <em>nickname</em> <code>common</code> and
187 associates it with a particular log format string. The format
188 string consists of percent directives, each of which tell the
189 server to log a particular piece of information. Literal
190 characters may also be placed in the format string and will be
191 copied directly into the log output. The quote character
192 (<code>"</code>) must be escaped by placing a back-slash before
193 it to prevent it from being interpreted as the end of the
194 format string. The format string may also contain the special
195 control characters "<code>\n</code>" for new-line and
196 "<code>\t</code>" for tab.</p>
198 <p>The <directive module="mod_log_config">CustomLog</directive>
199 directive sets up a new log file using the defined
200 <em>nickname</em>. The filename for the access log is relative to
201 the <directive module="core">ServerRoot</directive> unless it
202 begins with a slash.</p>
204 <p>The above configuration will write log entries in a format
205 known as the Common Log Format (CLF). This standard format can
206 be produced by many different web servers and read by many log
207 analysis programs. The log file entries produced in CLF will
208 look something like this:</p>
211 127.0.0.1 - frank [10/Oct/2000:13:55:36 -0700] "GET
212 /apache_pb.gif HTTP/1.0" 200 2326
215 <p>Each part of this log entry is described below.</p>
218 <dt><code>127.0.0.1</code> (<code>%h</code>)</dt>
220 <dd>This is the IP address of the client (remote host) which
221 made the request to the server. If <directive
222 module="core">HostnameLookups</directive> is
223 set to <code>On</code>, then the server will try to determine
224 the hostname and log it in place of the IP address. However,
225 this configuration is not recommended since it can
226 significantly slow the server. Instead, it is best to use a
227 log post-processor such as <program>logresolve</program> to determine
228 the hostnames. The IP address reported here is not
229 necessarily the address of the machine at which the user is
230 sitting. If a proxy server exists between the user and the
231 server, this address will be the address of the proxy, rather
232 than the originating machine.</dd>
234 <dt><code>-</code> (<code>%l</code>)</dt>
236 <dd>The "hyphen" in the output indicates that the requested
237 piece of information is not available. In this case, the
238 information that is not available is the RFC 1413 identity of
239 the client determined by <code>identd</code> on the clients
240 machine. This information is highly unreliable and should
241 almost never be used except on tightly controlled internal
242 networks. Apache httpd will not even attempt to determine
243 this information unless <directive
244 module="core">IdentityCheck</directive> is set
245 to <code>On</code>.</dd>
247 <dt><code>frank</code> (<code>%u</code>)</dt>
249 <dd>This is the userid of the person requesting the document
250 as determined by HTTP authentication. The same value is
251 typically provided to CGI scripts in the
252 <code>REMOTE_USER</code> environment variable. If the status
253 code for the request (see below) is 401, then this value
254 should not be trusted because the user is not yet
255 authenticated. If the document is not password protected,
256 this entry will be "<code>-</code>" just like the previous
259 <dt><code>[10/Oct/2000:13:55:36 -0700]</code>
260 (<code>%t</code>)</dt>
263 The time that the server finished processing the request.
267 <code>[day/month/year:hour:minute:second zone]<br />
269 month = 3*letter<br />
272 minute = 2*digit<br />
273 second = 2*digit<br />
274 zone = (`+' | `-') 4*digit</code>
276 It is possible to have the time displayed in another format
277 by specifying <code>%{format}t</code> in the log format
278 string, where <code>format</code> is as in
279 <code>strftime(3)</code> from the C standard library.
282 <dt><code>"GET /apache_pb.gif HTTP/1.0"</code>
283 (<code>\"%r\"</code>)</dt>
285 <dd>The request line from the client is given in double
286 quotes. The request line contains a great deal of useful
287 information. First, the method used by the client is
288 <code>GET</code>. Second, the client requested the resource
289 <code>/apache_pb.gif</code>, and third, the client used the
290 protocol <code>HTTP/1.0</code>. It is also possible to log
291 one or more parts of the request line independently. For
292 example, the format string "<code>%m %U%q %H</code>" will log
293 the method, path, query-string, and protocol, resulting in
294 exactly the same output as "<code>%r</code>".</dd>
296 <dt><code>200</code> (<code>%>s</code>)</dt>
298 <dd>This is the status code that the server sends back to the
299 client. This information is very valuable, because it reveals
300 whether the request resulted in a successful response (codes
301 beginning in 2), a redirection (codes beginning in 3), an
302 error caused by the client (codes beginning in 4), or an
303 error in the server (codes beginning in 5). The full list of
304 possible status codes can be found in the <a
305 href="http://www.w3.org/Protocols/rfc2616/rfc2616.txt">HTTP
306 specification</a> (RFC2616 section 10).</dd>
308 <dt><code>2326</code> (<code>%b</code>)</dt>
310 <dd>The last entry indicates the size of the object returned
311 to the client, not including the response headers. If no
312 content was returned to the client, this value will be
313 "<code>-</code>". To log "<code>0</code>" for no content, use
314 <code>%B</code> instead.</dd>
318 <section id="combined">
319 <title>Combined Log Format</title>
321 <p>Another commonly used format string is called the Combined
322 Log Format. It can be used as follows.</p>
325 LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\"
326 \"%{User-agent}i\"" combined<br />
327 CustomLog log/access_log combined
330 <p>This format is exactly the same as the Common Log Format,
331 with the addition of two more fields. Each of the additional
332 fields uses the percent-directive
333 <code>%{<em>header</em>}i</code>, where <em>header</em> can be
334 any HTTP request header. The access log under this format will
338 127.0.0.1 - frank [10/Oct/2000:13:55:36 -0700] "GET
339 /apache_pb.gif HTTP/1.0" 200 2326
340 "http://www.example.com/start.html" "Mozilla/4.08 [en]
344 <p>The additional fields are:</p>
347 <dt><code>"http://www.example.com/start.html"</code>
348 (<code>\"%{Referer}i\"</code>)</dt>
350 <dd>The "Referer" (sic) HTTP request header. This gives the
351 site that the client reports having been referred from. (This
352 should be the page that links to or includes
353 <code>/apache_pb.gif</code>).</dd>
355 <dt><code>"Mozilla/4.08 [en] (Win98; I ;Nav)"</code>
356 (<code>\"%{User-agent}i\"</code>)</dt>
358 <dd>The User-Agent HTTP request header. This is the
359 identifying information that the client browser reports about
364 <section id="multiple">
365 <title>Multiple Access Logs</title>
367 <p>Multiple access logs can be created simply by specifying
368 multiple <directive module="mod_log_config">CustomLog</directive>
369 directives in the configuration
370 file. For example, the following directives will create three
371 access logs. The first contains the basic CLF information,
372 while the second and third contain referer and browser
373 information. The last two <directive
374 module="mod_log_config">CustomLog</directive> lines show how
375 to mimic the effects of the <code>ReferLog</code> and <code
376 >AgentLog</code> directives.</p>
379 LogFormat "%h %l %u %t \"%r\" %>s %b" common<br />
380 CustomLog logs/access_log common<br />
381 CustomLog logs/referer_log "%{Referer}i -> %U"<br />
382 CustomLog logs/agent_log "%{User-agent}i"
385 <p>This example also shows that it is not necessary to define a
386 nickname with the <directive
387 module="mod_log_config">LogFormat</directive> directive. Instead,
388 the log format can be specified directly in the <directive
389 module="mod_log_config">CustomLog</directive> directive.</p>
392 <section id="conditional">
393 <title>Conditional Logs</title>
395 <p>There are times when it is convenient to exclude certain
396 entries from the access logs based on characteristics of the
397 client request. This is easily accomplished with the help of <a
398 href="env.html">environment variables</a>. First, an
399 environment variable must be set to indicate that the request
400 meets certain conditions. This is usually accomplished with
401 <directive module="mod_setenvif">SetEnvIf</directive>. Then the
402 <code>env=</code> clause of the <directive
403 module="mod_log_config">CustomLog</directive> directive is used to
404 include or exclude requests where the environment variable is
405 set. Some examples:</p>
408 # Mark requests from the loop-back interface<br />
409 SetEnvIf Remote_Addr "127\.0\.0\.1" dontlog<br />
410 # Mark requests for the robots.txt file<br />
411 SetEnvIf Request_URI "^/robots\.txt$" dontlog<br />
412 # Log what remains<br />
413 CustomLog logs/access_log common env=!dontlog
416 <p>As another example, consider logging requests from
417 english-speakers to one log file, and non-english speakers to a
418 different log file.</p>
421 SetEnvIf Accept-Language "en" english<br />
422 CustomLog logs/english_log common env=english<br />
423 CustomLog logs/non_english_log common env=!english
426 <p>Although we have just shown that conditional logging is very
427 powerful and flexibly, it is not the only way to control the
428 contents of the logs. Log files are more useful when they
429 contain a complete record of server activity. It is often
430 easier to simply post-process the log files to remove requests
431 that you do not want to consider.</p>
435 <section id="rotation">
436 <title>Log Rotation</title>
438 <p>On even a moderately busy server, the quantity of
439 information stored in the log files is very large. The access
440 log file typically grows 1 MB or more per 10,000 requests. It
441 will consequently be necessary to periodically rotate the log
442 files by moving or deleting the existing logs. This cannot be
443 done while the server is running, because Apache will continue
444 writing to the old log file as long as it holds the file open.
445 Instead, the server must be <a
446 href="stopping.html">restarted</a> after the log files are
447 moved or deleted so that it will open new log files.</p>
449 <p>By using a <em>graceful</em> restart, the server can be
450 instructed to open new log files without losing any existing or
451 pending connections from clients. However, in order to
452 accomplish this, the server must continue to write to the old
453 log files while it finishes serving old requests. It is
454 therefore necessary to wait for some time after the restart
455 before doing any processing on the log files. A typical
456 scenario that simply rotates the logs and compresses the old
457 logs to save space is:</p>
460 mv access_log access_log.old<br />
461 mv error_log error_log.old<br />
462 apachectl graceful<br />
464 gzip access_log.old error_log.old
467 <p>Another way to perform log rotation is using <a
468 href="#piped">piped logs</a> as discussed in the next
473 <title>Piped Logs</title>
475 <p>Apache httpd is capable of writing error and access log
476 files through a pipe to another process, rather than directly
477 to a file. This capability dramatically increases the
478 flexibility of logging, without adding code to the main server.
479 In order to write logs to a pipe, simply replace the filename
480 with the pipe character "<code>|</code>", followed by the name
481 of the executable which should accept log entries on its
482 standard input. Apache will start the piped-log process when
483 the server starts, and will restart it if it crashes while the
484 server is running. (This last feature is why we can refer to
485 this technique as "reliable piped logging".)</p>
487 <p>Piped log processes are spawned by the parent Apache httpd
488 process, and inherit the userid of that process. This means
489 that piped log programs usually run as root. It is therefore
490 very important to keep the programs simple and secure.</p>
492 <p>One important use of piped logs is to allow log rotation
493 without having to restart the server. The Apache HTTP Server
494 includes a simple program called <program>rotatelogs</program>
495 for this purpose. For example, to rotate the logs every 24 hours, you
499 CustomLog "|/usr/local/apache/bin/rotatelogs
500 /var/log/access_log 86400" common
503 <p>Notice that quotes are used to enclose the entire command
504 that will be called for the pipe. Although these examples are
505 for the access log, the same technique can be used for the
508 <p>A similar but much more flexible log rotation program
509 called <a href="http://www.cronolog.org/">cronolog</a>
510 is available at an external site.</p>
512 <p>As with conditional logging, piped logs are a very powerful
513 tool, but they should not be used where a simpler solution like
514 off-line post-processing is available.</p>
517 <section id="virtualhost">
518 <title>Virtual Hosts</title>
520 <p>When running a server with many <a href="vhosts/">virtual
521 hosts</a>, there are several options for dealing with log
522 files. First, it is possible to use logs exactly as in a
523 single-host server. Simply by placing the logging directives
524 outside the <directive module="core"
525 type="section">VirtualHost</directive> sections in the
526 main server context, it is possible to log all requests in the
527 same access log and error log. This technique does not allow
528 for easy collection of statistics on individual virtual
531 <p>If <directive module="mod_log_config">CustomLog</directive>
532 or <directive module="core">ErrorLog</directive>
533 directives are placed inside a
534 <directive module="core" type="section">VirtualHost</directive>
535 section, all requests or errors for that virtual host will be
536 logged only to the specified file. Any virtual host which does
537 not have logging directives will still have its requests sent
538 to the main server logs. This technique is very useful for a
539 small number of virtual hosts, but if the number of hosts is
540 very large, it can be complicated to manage. In addition, it
541 can often create problems with <a
542 href="vhosts/fd-limits.html">insufficient file
545 <p>For the access log, there is a very good compromise. By
546 adding information on the virtual host to the log format
547 string, it is possible to log all hosts to the same log, and
548 later split the log into individual files. For example,
549 consider the following directives.</p>
552 LogFormat "%v %l %u %t \"%r\" %>s %b"
554 CustomLog logs/access_log comonvhost
557 <p>The <code>%v</code> is used to log the name of the virtual
558 host that is serving the request. Then a program like <a
559 href="programs/other.html">split-logfile</a> can be used to
560 post-process the access log in order to split it into one file
561 per virtual host.</p>
565 <title>Other Log Files</title>
569 <module>mod_cgi</module>
570 <module>mod_rewrite</module>
573 <directive module="mpm_common">PidFile</directive>
574 <directive module="mod_rewrite">RewriteLog</directive>
575 <directive module="mod_rewrite">RewriteLogLevel</directive>
576 <directive module="mod_cgi">ScriptLog</directive>
577 <directive module="mod_cgi">ScriptLogBuffer</directive>
578 <directive module="mod_cgi">ScriptLogLength</directive>
582 <section id="pidfile">
583 <title>PID File</title>
585 <p>On startup, Apache httpd saves the process id of the parent
586 httpd process to the file <code>logs/httpd.pid</code>. This
587 filename can be changed with the <directive
588 module="mpm_common">PidFile</directive> directive. The
589 process-id is for use by the administrator in restarting and
590 terminating the daemon by sending signals to the parent
591 process; on Windows, use the -k command line option instead.
592 For more information see the <a href="stopping.html">Stopping
593 and Restarting</a> page.</p>
596 <section id="scriptlog">
597 <title>Script Log</title>
599 <p>In order to aid in debugging, the
600 <directive module="mod_cgi">ScriptLog</directive> directive
601 allows you to record the input to and output from CGI scripts.
602 This should only be used in testing - not for live servers.
603 More information is available in the <a
604 href="mod/mod_cgi.html">mod_cgi</a> documentation.</p>
607 <section id="rewritelog">
608 <title>Rewrite Log</title>
610 <p>When using the powerful and complex features of <a
611 href="mod/mod_rewrite.html">mod_rewrite</a>, it is almost
612 always necessary to use the <directive
613 module="mod_rewrite">RewriteLog</directive> to help
614 in debugging. This log file produces a detailed analysis of how
615 the rewriting engine transforms requests. The level of detail
616 is controlled by the <directive
617 module="mod_rewrite">RewriteLogLevel</directive> directive.</p>