1 <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
2 "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
4 <html xmlns="http://www.w3.org/1999/xhtml">
6 <meta name="generator" content="HTML Tidy, see www.w3.org" />
8 <title>Log Files - Apache HTTP Server</title>
10 <!-- Background white, links blue (unvisited), navy (visited), red (active) -->
12 <body bgcolor="#FFFFFF" text="#000000" link="#0000FF"
13 vlink="#000080" alink="#FF0000">
14 <!--#include virtual="header.html" -->
16 <h1 align="center">Log Files</h1>
18 <p>In order to effectively manage a web server, it is necessary
19 to get feedback about the activity and performance of the
20 server as well as any problems that may be occuring. The Apache
21 HTTP Server provides very comprehensive and flexible logging
22 capabilities. This document describes how to configure its
23 logging capabilities, and how to understand what the logs
27 <li><a href="#security">Security Warning</a></li>
29 <li><a href="#errorlog">Error Log</a></li>
32 <a href="#accesslog">Access Log</a>
35 <li><a href="#common">Common Log Format</a></li>
37 <li><a href="#combined">Combined Log Format</a></li>
39 <li><a href="#multiple">Multiple Access Logs</a></li>
41 <li><a href="#conditional">Conditional Logging</a></li>
45 <li><a href="#rotation">Log Rotation</a></li>
47 <li><a href="#piped">Piped Logs</a></li>
49 <li><a href="#virtualhosts">Virtual Hosts</a></li>
52 <a href="#other">Other Log Files</a>
55 <li><a href="#pidfile">PID File</a></li>
57 <li><a href="#scriptlog">Script Log</a></li>
59 <li><a href="#rewritelog">Rewrite Log</a></li>
65 <h2><a id="security" name="security">Security Warning</a></h2>
67 <p>Anyone who can write to the directory where Apache is
68 writing a log file can almost certainly gain access to the uid
69 that the server is started as, which is normally root. Do
70 <em>NOT</em> give people write access to the directory the logs
71 are stored in without being aware of the consequences; see the
72 <a href="misc/security_tips.html">security tips</a> document
75 <p>In addition, log files may contain information supplied
76 directly by the client, without escaping. Therefore, it is
77 possible for malicious clients to insert control-characters in
78 the log files, so care must be taken in dealing with raw
82 <h2><a id="errorlog" name="errorlog">Error Log</a></h2>
86 <td valign="top"><strong>Related Directives</strong><br />
88 <a href="mod/core.html#errorlog">ErrorLog</a><br />
89 <a href="mod/core.html#loglevel">LogLevel</a> </td>
93 <p>The server error log, whose name and location is set by the
94 <a href="mod/core.html#errorlog">ErrorLog</a> directive, is the
95 most important log file. This is the place where Apache httpd
96 will send diagnostic information and record any errors that it
97 encounters in processing requests. It is the first place to
98 look when a problem occurs with starting the server or with the
99 operation of the server, since it will often contain details of
100 what went wrong and how to fix it.</p>
102 <p>The error log is usually written to a file (typically
103 <code>error_log</code> on unix systems and
104 <code>error.log</code> on Windows and OS/2). On unix systems it
105 is also possible to have the server send errors to
106 <code>syslog</code> or <a href="#pipe">pipe them to a
109 <p>The format of the error log is relatively free-form and
110 descriptive. But there is certain information that is contained
111 in most error log entries. For example, here is a typical
115 <code>[Wed Oct 11 14:32:52 2000] [error] [client 127.0.0.1]
116 client denied by server configuration:
117 /export/home/live/ap/htdocs/test</code>
120 <p>The first item in the log entry is the date and time of the
121 message. The second entry lists the severity of the error being
122 reported. The <a href="mod/core.html#loglevel">LogLevel</a>
123 directive is used to control the types of errors that are sent
124 to the error log by restricting the severity level. The third
125 entry gives the IP address of the client that generated the
126 error. Beyond that is the message itself, which in this case
127 indicates that the server has been configured to deny the
128 client access. The server reports the file-system path (as
129 opposed to the web path) of the requested document.</p>
131 <p>A very wide variety of different messages can appear in the
132 error log. Most look similar to the example above. The error
133 log will also contain debugging output from CGI scripts. Any
134 information written to <code>stderr</code> by a CGI script will
135 be copied directly to the error log.</p>
137 <p>It is not possible to customize the error log by adding or
138 removing information. However, error log entries dealing with
139 particular requests have corresponding entries in the <a
140 href="#accesslog">access log</a>. For example, the above example
141 entry corresponds to an access log entry with status code 403.
142 Since it is possible to customize the access log, you can
143 obtain more information about error conditions using that log
146 <p>During testing, it is often useful to continuously monitor
147 the error log for any problems. On unix systems, you can
148 accomplish this using:</p>
151 <code>tail -f error_log</code>
155 <h2><a id="accesslog" name="accesslog">Access Log</a></h2>
159 <td valign="top"><strong>Related Modules</strong><br />
161 <a href="mod/mod_log_config.html">mod_log_config</a><br />
164 <td valign="top"><strong>Related Directives</strong><br />
167 href="mod/mod_log_config.html#customlog">CustomLog</a><br />
169 href="mod/mod_log_config.html#logformat">LogFormat</a><br />
170 <a href="mod/mod_setenvif.html#setenvif">SetEnvIf</a>
175 <p>The server access log records all requests processed by the
176 server. The location and content of the access log are
178 href="mod/mod_log_config.html#customlog">CustomLog</a>
180 href="mod/mod_log_config.html#logformat">LogFormat</a>
181 directive can be used to simplify the selection of the contents
182 of the logs. This section describes how to configure the server
183 to record information in the access log.</p>
185 <p>Of course, storing the information in the access log is only
186 the start of log management. The next step is to analyze this
187 information to produce useful statistics. Log analysis in
188 general is beyond the scope of this document, and not really
189 part of the job of the web server itself. For more information
190 about this topic, and for applications which perform log
191 analysis, check the <a
192 href="http://dmoz.org/Computers/Software/Internet/Site_Management/Log_analysis/">
193 Open Directory</a> or <a
194 href="http://dir.yahoo.com/Computers_and_Internet/Software/Internet/World_Wide_Web/Servers/Log_Analysis_Tools/">
197 <p>Various versions of Apache httpd have used other modules and
198 directives to control access logging, including
199 mod_log_referer, mod_log_agent, and the
200 <code>TransferLog</code> directive. The <code>CustomLog</code>
201 directive now subsumes the functionality of all the older
204 <p>The format of the access log is highly configurable. The
205 format is specified using a <a
206 href="mod/mod_log_config.html#format">format string</a> that
207 looks much like a C-style printf(1) format string. Some
208 examples are presented in the next sections. For a complete
209 list of the possible contents of the format string, see the <a
210 href="mod/mod_log_config.html">mod_log_config
211 documentation</a>.</p>
213 <h3><a id="common" name="common">Common Log Format</a></h3>
215 <p>A typical configuration for the access log might look as
219 <code>LogFormat "%h %l %u %t \"%r\" %>s %b" common<br />
220 CustomLog logs/access_log common</code>
223 <p>This defines the <em>nickname</em> <code>common</code> and
224 associates it with a particular log format string. The format
225 string consists of percent directives, each of which tell the
226 server to log a particular piece of information. Literal
227 characters may also be placed in the format string and will be
228 copied directly into the log output. The quote character
229 (<code>"</code>) must be escaped by placing a back-slash before
230 it to prevent it from being interpreted as the end of the
231 format string. The format string may also contain the special
232 control characters "<code>\n</code>" for new-line and
233 "<code>\t</code>" for tab.</p>
235 <p>The <code>CustomLog</code> directive sets up a new log file
236 using the defined <em>nickname</em>. The filename for the
237 access log is relative to the <a
238 href="mod/core.html#serverroot">ServerRoot</a> unless it begins
241 <p>The above configuration will write log entries in a format
242 known as the Common Log Format (CLF). This standard format can
243 be produced by many different web servers and read by many log
244 analysis programs. The log file entries produced in CLF will
245 look something like this:</p>
248 <code>127.0.0.1 - frank [10/Oct/2000:13:55:36 -0700] "GET
249 /apache_pb.gif HTTP/1.0" 200 2326</code>
252 <p>Each part of this log entry is described below.</p>
255 <dt><code>127.0.0.1</code> (<code>%h</code>)</dt>
257 <dd>This is the IP address of the client (remote host) which
258 made the request to the server. If <a
259 href="mod/core.html#hostnamelookups">HostnameLookups</a> is
260 set to <code>On</code>, then the server will try to determine
261 the hostname and log it in place of the IP address. However,
262 this configuration is not recommended since it can
263 significantly slow the server. Instead, it is best to use a
264 log post-processor such as <a
265 href="programs/logresolve.html">logresolve</a> to determine
266 the hostnames. The IP address reported here is not
267 necessarily the address of the machine at which the user is
268 sitting. If a proxy server exists between the user and the
269 server, this address will be the address of the proxy, rather
270 than the originating machine.</dd>
272 <dt><code>-</code> (<code>%l</code>)</dt>
274 <dd>The "hyphen" in the output indicates that the requested
275 piece of information is not available. In this case, the
276 information that is not available is the RFC 1413 identity of
277 the client determined by <code>identd</code> on the clients
278 machine. This information is highly unreliable and should
279 almost never be used except on tightly controlled internal
280 networks. Apache httpd will not even attempt to determine
281 this information unless <a
282 href="mod/core.html#identitycheck">IdentityCheck</a> is set
283 to <code>On</code>.</dd>
285 <dt><code>frank</code> (<code>%u</code>)</dt>
287 <dd>This is the userid of the person requesting the document
288 as determined by HTTP authentication. The same value is
289 typically provided to CGI scripts in the
290 <code>REMOTE_USER</code> environment variable. If the status
291 code for the request (see below) is 401, then this value
292 should not be trusted because the user is not yet
293 authenticated. If the document is not password protected,
294 this entry will be "<code>-</code>" just like the previous
297 <dt><code>[10/Oct/2000:13:55:36 -0700]</code>
298 (<code>%t</code>)</dt>
301 The time that the server finished processing the request.
305 <code>[day/month/year:hour:minute:second zone]<br />
307 month = 3*letter<br />
310 minute = 2*digit<br />
311 second = 2*digit<br />
312 zone = (`+' | `-') 4*digit</code>
314 It is possible to have the time displayed in another format
315 by specifying <code>%{format}t</code> in the log format
316 string, where <code>format</code> is as in
317 <code>strftime(3)</code> from the C standard library.
320 <dt><code>"GET /apache_pb.gif HTTP/1.0"</code>
321 (<code>\"%r\"</code>)</dt>
323 <dd>The request line from the client is given in double
324 quotes. The request line contains a great deal of useful
325 information. First, the method used by the client is
326 <code>GET</code>. Second, the client requested the resource
327 <code>/apache_pb.gif</code>, and third, the client used the
328 protocol <code>HTTP/1.0</code>. It is also possible to log
329 one or more parts of the request line independently. For
330 example, the format string "<code>%m %U%q %H</code>" will log
331 the method, path, query-string, and protocol, resulting in
332 exactly the same output as "<code>%r</code>".</dd>
334 <dt><code>200</code> (<code>%>s</code>)</dt>
336 <dd>This is the status code that the server sends back to the
337 client. This information is very valuable, because it reveals
338 whether the request resulted in a successful response (codes
339 beginning in 2), a redirection (codes beginning in 3), an
340 error caused by the client (codes beginning in 4), or an
341 error in the server (codes beginning in 5). The full list of
342 possible status codes can be found in the <a
343 href="http://www.w3.org/Protocols/rfc2616/rfc2616.txt">HTTP
344 specification</a> (RFC2616 section 10).</dd>
346 <dt><code>2326</code> (<code>%b</code>)</dt>
348 <dd>The last entry indicates the size of the object returned
349 to the client, not including the response headers. If no
350 content was returned to the client, this value will be
351 "<code>-</code>". To log "<code>0</code>" for no content, use
352 <code>%B</code> instead.</dd>
355 <h4><a id="combined" name="combined">Combined Log
358 <p>Another commonly used format string is called the Combined
359 Log Format. It can be used as follows.</p>
362 <code>LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\"
363 \"%{User-agent}i\"" combined<br />
364 CustomLog log/acces_log combined</code>
367 <p>This format is exactly the same as the Common Log Format,
368 with the addition of two more fields. Each of the additional
369 fields uses the percent-directive
370 <code>%{<em>header</em>}i</code>, where <em>header</em> can be
371 any HTTP request header. The access log under this format will
375 <code>127.0.0.1 - frank [10/Oct/2000:13:55:36 -0700] "GET
376 /apache_pb.gif HTTP/1.0" 200 2326
377 "http://www.example.com/start.html" "Mozilla/4.08 [en]
378 (Win98; I ;Nav)"</code>
381 <p>The additional fields are:</p>
384 <dt><code>"http://www.example.com/start.html"</code>
385 (<code>\"%{Referer}i\"</code>)</dt>
387 <dd>The "Referer" (sic) HTTP request header. This gives the
388 site that the client reports having been referred from. (This
389 should be the page that links to or includes
390 <code>/apache_pb.gif</code>).</dd>
392 <dt><code>"Mozilla/4.08 [en] (Win98; I ;Nav)"</code>
393 (<code>\"%{User-agent}i\"</code>)</dt>
395 <dd>The User-Agent HTTP request header. This is the
396 identifying information that the client browser reports about
400 <h3><a id="multiple" name="multiple">Multiple Access
403 <p>Multiple access logs can be created simply by specifying
404 multiple <code>CustomLog</code> directives in the configuration
405 file. For example, the following directives will create three
406 access logs. The first contains the basic CLF information,
407 while the second and third contain referer and browser
408 information. The last two <code>CustomLog</code> lines show how
409 to mimic the effects of the <code>ReferLog</code> and
410 <code>AgentLog</code> directives.</p>
413 <code>LogFormat "%h %l %u %t \"%r\" %>s %b" common<br />
414 CustomLog logs/access_log common<br />
415 CustomLog logs/referer_log "%{Referer}i -> %U"<br />
416 CustomLog logs/agent_log "%{User-agent}i"</code>
419 <p>This example also shows that it is not necessary to define a
420 nickname with the <code>LogFormat</code> directive. Instead,
421 the log format can be specified directly in the
422 <code>CustomLog</code> directive.</p>
424 <h3><a id="conditional" name="conditional">Conditional
427 <p>There are times when it is convenient to exclude certain
428 entries from the access logs based on characteristics of the
429 client request. This is easily accomplished with the help of <a
430 href="env.html">environment variables</a>. First, an
431 environment variable must be set to indicate that the request
432 meets certain conditions. This is usually accomplished with <a
433 href="mod/mod_setenvif.html#setenvif">SetEnvIf</a>. Then the
434 <code>env=</code> clause of the <code>CustomLog</code>
435 directive is used to include or exclude requests where the
436 environment variable is set. Some examples:</p>
439 <code># Mark requests from the loop-back interface<br />
440 SetEnvIf Remote_Addr "127\.0\.0\.1" dontlog<br />
441 # Mark requests for the robots.txt file<br />
442 SetEnvIf Request_URI "^/robots\.txt$" dontlog<br />
443 # Log what remains<br />
444 CustomLog logs/access_log common env=!dontlog</code>
447 <p>As another example, consider logging requests from
448 english-speakers to one log file, and non-english speakers to a
449 different log file.</p>
452 <code>SetEnvIf Accept-Language "en" english<br />
453 CustomLog logs/english_log common env=english<br />
454 CustomLog logs/non_english_log common env=!english</code>
457 <p>Although we have just shown that conditional logging is very
458 powerful and flexibly, it is not the only way to control the
459 contents of the logs. Log files are more useful when they
460 contain a complete record of server activity. It is often
461 easier to simply post-process the log files to remove requests
462 that you do not want to consider.</p>
465 <h2><a id="rotation" name="rotation">Log Rotation</a></h2>
467 <p>On even a moderately busy server, the quantity of
468 information stored in the log files is very large. The access
469 log file typically grows 1 MB or more per 10,000 requests. It
470 will consequently be necessary to periodically rotate the log
471 files by moving or deleting the existing logs. This cannot be
472 done while the server is running, because Apache will continue
473 writing to the old log file as long as it holds the file open.
474 Instead, the server must be <a
475 href="stopping.html">restarted</a> after the log files are
476 moved or deleted so that it will open new log files.</p>
478 <p>By using a <em>graceful</em> restart, the server can be
479 instructed to open new log files without losing any existing or
480 pending connections from clients. However, in order to
481 accomplish this, the server must continue to write to the old
482 log files while it finishes serving old requests. It is
483 therefore necessary to wait for some time after the restart
484 before doing any processing on the log files. A typical
485 scenario that simply rotates the logs and compresses the old
486 logs to save space is:</p>
489 <code>mv access_log access_log.old<br />
490 mv error_log error_log.old<br />
491 apachectl graceful<br />
493 gzip access_log.old error_log.old</code>
496 <p>Another way to perform log rotation is using <a
497 href="#piped">piped logs</a> as discussed in the next
501 <h2><a id="piped" name="piped">Piped Logs</a></h2>
503 <p>Apache httpd is capable of writing error and access log
504 files through a pipe to another process, rather than directly
505 to a file. This capability dramatically increases the
506 flexibility of logging, without adding code to the main server.
507 In order to write logs to a pipe, simply replace the filename
508 with the pipe character "<code>|</code>", followed by the name
509 of the executable which should accept log entries on its
510 standard input. Apache will start the piped-log process when
511 the server starts, and will restart it if it crashes while the
512 server is running. (This last feature is why we can refer to
513 this technique as "reliable piped logging".)</p>
515 <p>Piped log processes are spawned by the parent Apache httpd
516 process, and inherit the userid of that process. This means
517 that piped log programs usually run as root. It is therefore
518 very important to keep the programs simple and secure.</p>
520 <p>Some simple examples using piped logs:</p>
523 <code># compressed logs<br />
524 CustomLog "|/usr/bin/gzip -c >>
525 /var/log/access_log.gz" common<br />
526 # almost-real-time name resolution<br />
527 CustomLog "|/usr/local/apache/bin/logresolve >>
528 /var/log/access_log" common</code>
531 <p>Notice that quotes are used to enclose the entire command
532 that will be called for the pipe. Although these examples are
533 for the access log, the same technique can be used for the
536 <p>One important use of piped logs is to allow log rotation
537 without having to restart the server. The Apache HTTP Server
538 includes a simple program called <a
539 href="programs/rotatelogs.html">rotatelogs</a> for this
540 purpose. For example, to rotate the logs every 24 hours, you
544 <code>CustomLog "|/usr/local/apache/bin/rotatelogs
545 /var/log/access_log 86400" common</code>
548 <p>A similar, but much more flexible log rotation program
549 called <a href="http://www.cronolog.org/">cronolog</a>
550 is available at an external site.</p>
552 <p>As with conditional logging, piped logs are a very powerful
553 tool, but they should not be used where a simpler solution like
554 off-line post-processing is available.</p>
557 <h2><a id="virtualhosts" name="virtualhosts">Virtual
560 <p>When running a server with many <a href="vhosts/">virtual
561 hosts</a>, there are several options for dealing with log
562 files. First, it is possible to use logs exactly as in a
563 single-host server. Simply by placing the logging directives
564 outside the <code><VirtualHost></code> sections in the
565 main server context, it is possible to log all requests in the
566 same access log and error log. This technique does not allow
567 for easy collection of statistics on individual virtual
570 <p>If <code>CustomLog</code> or <code>ErrorLog</code>
571 directives are placed inside a <code><VirtualHost></code>
572 section, all requests or errors for that virtual host will be
573 logged only to the specified file. Any virtual host which does
574 not have logging directives will still have its requests sent
575 to the main server logs. This technique is very useful for a
576 small number of virtual hosts, but if the number of hosts is
577 very large, it can be complicated to manage. In addition, it
578 can often create problems with <a
579 href="vhosts/fd-limits.html">insufficient file
582 <p>For the access log, there is a very good compromise. By
583 adding information on the virtual host to the log format
584 string, it is possible to log all hosts to the same log, and
585 later split the log into individual files. For example,
586 consider the following directives.</p>
589 <code>LogFormat "%v %l %u %t \"%r\" %>s %b"
591 CustomLog logs/access_log comonvhost</code>
594 <p>The <code>%v</code> is used to log the name of the virtual
595 host that is serving the request. Then a program like <a
596 href="programs/other.html">split-logfile</a> can be used to
597 post-process the access log in order to split it into one file
598 per virtual host.</p>
600 <p>Unfortunately, no similar technique is available for the
601 error log, so you must choose between mixing all virtual hosts
602 in the same error log and using one error log per virtual
606 <h2><a id="other" name="other">Other Log Files</a></h2>
610 <td valign="top"><strong>Related Modules</strong><br />
612 <a href="mod/mod_cgi.html">mod_cgi</a><br />
613 <a href="mod/mod_rewrite.html">mod_rewrite</a> </td>
615 <td valign="top"><strong>Related Directives</strong><br />
617 <a href="mod/core.html#pidfile">PidFile</a><br />
619 href="mod/mod_rewrite.html#RewriteLog">RewriteLog</a><br />
621 href="mod/mod_rewrite.html#RewriteLogLevel">RewriteLogLevel</a><br />
622 <a href="mod/mod_cgi.html#scriptlog">ScriptLog</a><br />
624 href="mod/mod_cgi.html#scriptloglength">ScriptLogLength</a><br />
626 href="mod/mod_cgi.html#scriptlogbuffer">ScriptLogBuffer</a>
631 <h3><a id="pidfile" name="pidfile">PID File</a></h3>
633 <p>On startup, Apache httpd saves the process id of the parent
634 httpd process to the file <code>logs/httpd.pid</code>. This
635 filename can be changed with the <a
636 href="mod/core.html#pidfile">PidFile</a> directive. The
637 process-id is for use by the administrator in restarting and
638 terminating the daemon by sending signals to the parent
639 process; on Windows, use the -k command line option instead.
640 For more information see the <a href="stopping.html">Stopping
641 and Restarting</a> page.</p>
643 <h3><a id="scriptlog" name="scriptlog">Script Log</a></h3>
645 <p>In order to aid in debugging, the <a
646 href="mod/mod_cgi.html#scriptlog">ScriptLog</a> directive
647 allows you to record the input to and output from CGI scripts.
648 This should only be used in testing - not for live servers.
649 More information is available in the <a
650 href="mod/mod_cgi.html">mod_cgi documentation</a>.</p>
652 <h3><a id="rewritelog" name="rewritelog">Rewrite Log</a></h3>
654 <p>When using the powerful and complex features of <a
655 href="mod/mod_rewrite.html">mod_rewrite</a>, it is almost
656 always necessary to use the <a
657 href="mod/mod_rewrite.html#RewriteLog">RewriteLog</a> to help
658 in debugging. This log file produces a detailed analysis of how
659 the rewriting engine transforms requests. The level of detail
660 is controlled by the <a
661 href="mod/mod_rewrite.html#RewriteLogLevel">RewriteLogLevel</a>
663 <!--#include virtual="footer.html" -->