1 <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">
4 <TITLE>Log Files - Apache HTTP Server</TITLE>
7 <!-- Background white, links blue (unvisited), navy (visited), red (active) -->
15 <!--#include virtual="header.html" -->
16 <h1 align="center">Log Files</h1>
18 <p>In order to effectively manage a web server, it is necessary to get
19 feedback about the activity and performance of the server as well as
20 any problems that may be occuring. The Apache HTTP Server provides
21 very comprehensive and flexible logging capabilities. This document
22 describes how to configure the various log files, and how to
23 understand what the logs contain.</p>
26 <li><a href="#security">Security Warning</a></li>
27 <li><a href="#errorlog">Error Log</a></li>
28 <li><a href="#accesslog">Access Log</a>
30 <li><a href="#common">Common Log Format</a></li>
31 <li><a href="#combined">Combined Log Format</a></li>
32 <li><a href="#multiple">Multiple Access Logs</a></li>
33 <li><a href="#conditional">Conditional Logging</a></li>
35 <li><a href="#rotation">Log Rotation</a></li>
36 <li><a href="#piped">Piped Logs</a></li>
37 <li><a href="#virtualhosts">VirtualHosts</a>
38 <li><a href="#other">Other Log Files</a>
40 <li><a href="#pidfile">PID File</a></li>
41 <li><a href="#scriptlog">Script Log</a></li>
42 <li><a href="#rewritelog">Rewrite Log</a></li>
48 <h2><a name="security">Security Warning</a></h2>
50 <p>Anyone who can write to the directory where Apache is writing a
51 log file can almost certainly gain access to the uid that the server is
52 started as, which is normally root. Do <EM>NOT</EM> give people write
53 access to the directory the logs are stored in without being aware of
54 the consequences; see the <A HREF="misc/security_tips.html">security tips</A>
55 document for details.</p>
57 <p>In addition, log files may contain information supplied directly
58 by the client, without escaping. Therefore, it is possible for
59 malicious clients to insert control-characters in the log files, so
60 care must be taken in dealing with raw logs.</p>
64 <h2><a name="errorlog">Error Log</a></h2>
68 <strong>Related Directives</strong><br><br>
70 <a href="mod/core.html#errorlog">ErrorLog</a><br>
71 <a href="mod/core.html#loglevel">LogLevel</a>
74 <p>The server error log, whose name and location is set by the <a
75 href="mod/core.html#errorlog">ErrorLog</a> directive, is the most
76 important log file. This is the place where Apache HTTPD will send
77 diagnostic information and record any errors that it encounters in
78 processing requests. It is the first place to look when a problem
79 occurs with starting the server or with the operation of the server,
80 since it will often contain details of what went wrong and how to fix
83 <p>The error log is usually written to a file (typically
84 <code>error_log</code> on unix systems and <code>error.log</code> on
85 Windows and OS/2). On unix systems it is also possible to have the
86 server send errors to the <code>syslog</code> or <a href="#pipe">pipe
87 them through a program</a>.</p>
89 <p>The format of the error log is relatively free-form and
90 descriptive. But there is certain information that is contained
91 in most error log entries. For example, here is a typical message.</p>
94 [Wed Oct 11 14:32:52 2000] [error] [client 127.0.0.1] client denied by server configuration: /export/home/live/ap/htdocs/test
97 <p>The first item in the log entry is the date and time of the
98 message. The second entry lists the severity of the error being
99 reported. The <a href="mod/core.html#loglevel">LogLevel</a> directive
100 is used to control the types of errors that are sent to the error log
101 by restricting the severity level. The third entry gives the IP
102 address of the client that generated the error. Beyond that is the
103 message itself, which in this case indicates that the server has been
104 configured to deny the client access. The server reports the
105 file-system path (as opposed to the web path) of the requested
108 <p>A very wide variety of different messages can appear in the error
109 log. Most look similar to the example above. The error log will also
110 contain debugging output from CGI scripts. Any information written to
111 <code>stderr</code> by a CGI script will be copied directly to the
114 <p>It is not possible to customize the error log by adding or removing
115 information. However, error log entries dealing with particular
116 requests have corresponding entries in the <a href="accesslog">access
117 log</a>. For example, the above example entry corresponds to an
118 access log entry with status code 403. Since it is possible to
119 customize the access log, you can obtain more information about error
120 conditions using that log file.</p>
122 <p>During testing, it is often useful to continuously monitor the
123 error log for any problems. On unix systems, you can accomplish this
131 <h2><a name="accesslog">Access Log</a></h2>
133 <table border=1><tr><td valign="top">
134 <strong>Related Modules</strong><br><br>
136 <a href="mod/mod_log_config.html">mod_log_config</a><br>
138 </td><td valign="top">
139 <strong>Related Directives</strong><br><br>
141 <a href="mod/mod_log_config.html#customlog">CustomLog</a><br>
142 <a href="mod/mod_log_config.html#logformat">LogFormat</a><br>
143 <a href="mod/mod_setenvif.html#setenvif">SetEnvIf</a>
147 <p>The server access log records all requests processed by the server.
148 The location of the access log as well as its contents are controlled
149 by the <a href="mod/mod_log_config.html#customlog">CustomLog</a>
151 href="mod/mod_log_config.html#logformat">LogFormat</a> directive can
152 be used to simplify the selection of the contents of the logs.
153 This section describes how to configure the server to record
154 information in the access log.</p>
156 <p>Of course, storing the information in the access log is only the
157 start of log management. The next step is to analyze this information
158 to produce useful statistics. Log analysis in general is beyond the
159 scope of this document, and not really part of the job of the
160 web server itself. For more information about this topic, and for
161 applications which perform log analysis, check the <a
162 href="http://dmoz.org/Computers/Software/Internet/Site_Management/Log_analysis/"
163 >Open Directory</a> or <a
164 href="http://dir.yahoo.com/Computers_and_Internet/Software/Internet/World_Wide_Web/Servers/Log_Analysis_Tools/"
167 <p>Various versions of Apache HTTPD have used other modules and
168 directives to control access logging, including mod_log_referer,
169 mod_log_agent, and the <code>TransferLog</code> directive. The
170 <code>CustomLog</code> directive now subsumes the functionality of all
171 the older directives.</p>
173 <p>The format of the access log is highly configurable. The format is
174 specified using a <a href="mod/mod_log_config.html#format">format
175 string</a> that looks much like a C-style printf(1) format string.
176 Some examples are presented in the next sections. For a complete list
177 of the possible contents of the format string, see the <a
178 href="mod/mod_log_config.html">mod_log_config documentation</a>.</p>
180 <h3><a name="common">Common Log Format</a></h3>
182 <p>A typical configuration for the access log might look
186 LogFormat "%h %l %u %t \"%r\" %>s %b" common<br>
187 CustomLog logs/access_log common
190 <p>This defines the <em>nickname</em> <code>common</code> and
191 associates it with a particular log format string. The format string
192 consists of percent directives, each of which tell the server to log a
193 particular piece of information. Literal characters may also be
194 placed in the format string and will be copied directly into the log
195 output. The quote character (<code>"</code>) must be escaped by
196 placing a back-slash before it to prevent it from being interpreted as
197 the end of the format string. The format string may also contain the
198 special control characters "<code>\n</code>" for new-line and
199 "<code>\t</code>" for tab.</p>
201 <p>The <code>CustomLog</code> directive sets up a new log file using
202 the defined <em>nickname</em>. The filename for the access log is
203 relative to the <a href="mod/core.html#serverroot">ServerRoot</a>
204 unless it begins with a slash.</p>
206 <p>The above configuration will write log entries in a format known as
207 the Common Log Format (CLF). This standard format can be produced by
208 many different web servers and read by many log analysis programs.
209 The log file entries produced in CLF will look something like
213 127.0.0.1 - frank [10/Oct/2000:13:55:36 -0700] "GET /apache_pb.gif HTTP/1.0" 200 2326
216 <p>Each part of this log entry is described below.</p>
219 <dt><code>127.0.0.1</code> (<code>%h</code>)</dt> <dd>This is the IP
220 address of the client (remote host) which made the request to the
222 href="mod/core.html#hostnamelookups">HostNameLookups</a> is set to
223 <code>On</code>, then the server will try to determine the hostname
224 and log it in place of the IP address. However, this configuration is
225 not recommended since it can significantly slow the server. Instead,
226 it is best to use a log post-processor such as <a
227 href="programs/logresolve.html">logresolve</a> to determine the
228 hostnames. The IP address reported here is not necessarily the
229 address of the machine at which the user is sitting. If a proxy
230 server exists between the user and the server, this address will be
231 the address of the proxy, rather than the originating machine.</dd>
233 <dt><code>-</code> (<code>%l</code>)</dt> <dd>The "hyphen" in the
234 output indicates that the requested piece of information is not
235 available. In this case, the information that is not available is the
236 RFC 1413 identity of the client determined by <code>identd</code> on
237 the clients machine. This information is highly unreliable and should
238 almost never be used except on tightly controlled internal networks.
239 Apache HTTPD will not even attempt to determine this information
240 unless <a href="mod/core.html#identitycheck">IdentityCheck</a> is set
241 to <code>On</code>.</dd>
243 <dt><code>frank</code> (<code>%u</code>)</dt> <dd>This is the userid
244 of the person requesting the document as determined by HTTP
245 authentication. The same value is typically provided to CGI scripts
246 in the <code>REMOTE_USER</code> environment variable. If the status
247 code for the request (see below) is 401, then this value should not be
248 trusted because the user is not yet authenticated. If the document is
249 not password protected, this entry will be "<code>-</code>" just like
250 the previous one.</dd>
252 <dt><code>[10/Oct/2000:13:55:36 -0700]</code> (<code>%t</code>)</dt>
253 <dd>The time that the server finished processing the request. The
255 <BLOCKQUOTE><CODE> [day/month/year:hour:minute:second zone] <BR>
262 zone = (`+' | `-') 4*digit</CODE></BLOCKQUOTE>
263 It is possible to have the time displayed in another format
264 by specifying <code>%{format}t</code> in the log format string, where
265 <code>format</code> is as in <code>strftime(3)</code> from the C
269 <dt><code>"GET /apache_pb.gif HTTP/1.0"</code>
270 (<code>\"%r\"</code>)</dt> <dd>The request line from the client is
271 given in double quotes. The request line contains a great deal of
272 useful information. First, the method used by the client is
273 <code>GET</code>. Second, the client requested the resource
274 <code>/apache_pb.gif</code>, and third, the client used the protocol
275 <code>HTTP/1.0</code>.</dd> It is also possible to log one or more
276 parts of the request line independently. For example, the format
277 string "<code>%m %U%q %H</code>" will log the method, path,
278 query-string, and protocol, resulting in exactly the same output as
279 "<code>%r</code>".</dd>
281 <dt><code>200</code></dt> (<code>%>s</code>) <dd>This is the status
282 code that the server sends back to the client. This information is
283 very valuable, because it reveals whether the request resulted in a
284 successful response (codes beginning in 2), a redirection (codes
285 beginning in 3), an error caused by the client (codes beginning in 4),
286 or an error in the server (codes beginning in 5).
287 The full list of possible status codes can be
288 found in the <a href="http://www.w3.org/Protocols/rfc2616/rfc2616.txt"
289 >HTTP specification</a> (RFC2616 section 10).</dd>
291 <dt><code>2326</code> (<code>%b</code>)
292 <dd>The last entry indicates the size of the object returned to
293 the client, not including the response headers. If no content
294 was returned to the client, this value will be "<code>-</code>".
295 To log "<code>0</code>" for no content, use <code>%B</code>
300 <h4><a name="combined">Combined Log Format</a></h4>
302 <p>Another commonly used format string is called the
303 Combined Log Format. It can be used as follows.</p>
306 LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-agent}i\"" combined<br>
307 CustomLog log/acces_log combined
310 <p>This format is exactly the same as the Common Log Format,
311 with the addition of two more fields. The access log under this
312 format will look like:</p>
315 127.0.0.1 - frank [10/Oct/2000:13:55:36 -0700] "GET /apache_pb.gif HTTP/1.0" 200 2326 "http://www.example.com/start.html" "Mozilla/4.08 [en] (Win98; I ;Nav)"
318 <p>The additional fields are:</p>
322 <dt><code>"http://www.example.com/start.html"</code>
323 (<code>\"%{Referer}i\"</code>)</dt> <dd>The "Referer" (sic) HTTP
324 request header. This gives the site that the client reports having
325 been referred from. (This should be the page that links to or includes
326 <code>/apache_pb.gif</code>).
328 <dt><code>"Mozilla/4.08 [en] (Win98; I ;Nav)"</code>
329 (<code>\"%{User-agent}i\"</code>)</dt> <dd>The User-Agent HTTP request
330 header. This is the identifying information that the client browser
331 reports about itself.</dd>
335 <h3><a name="multiple">Multiple Access Logs</a></h3>
337 <p>Multiple access logs can be created simply by specifying multiple
338 <code>CustomLog</code> directives in the configuration file. For
339 example, the following directives will create three access logs. The
340 first contains the basic CLF information, while the second and third
341 contain referer and browser information. The last two
342 <code>CustomLog</code> lines show how to mimic the effects of the
343 <code>ReferLog</code> and <code>AgentLog</code> directives.</p>
346 LogFormat "%h %l %u %t \"%r\" %>s %b" common<br>
347 CustomLog logs/access_log common<br>
348 CustomLog logs/referer_log "%{Referer}i -> %U"<br>
349 CustomLog logs/agent_log "%{User-agent}i"
352 <p>This example also shows that it is not necessary to define a
353 nickname with the <code>LogFormat</code> directive. Instead, the log
354 format can be specified directly in the <code>CustomLog</code>
357 <h3><a name="conditional">Conditional Logging</a></h3>
359 <p>There are times when it is convenient to exclude certain entries
360 from the access logs based on characteristics of the client request.
361 This is easily accomplished with the help of <a
362 href="env.html">environment variables</a>. First, an environment
363 variable must be set to indicate that the request meets certain
364 conditions. This is usually accomplished with <a
365 href="mod/mod_setenvif.html#setenvif">SetEnvIf</a>. Then the
366 <code>env=</code> clause of the <code>CustomLog</code> directive is
367 used to include or exclude requests where the environment variable is
368 set. Some examples:</p>
371 # Mark requests from the loop-back interface<br>
372 SetEnvIf Remote_Addr "127\.0\.0\.1" dontlog<br>
373 # Mark requests for the robots.txt file<br>
374 SetEnvIf Request_URI "^/robots\.txt$" dontlog<br>
375 # Log what remains<br>
376 CustomLog logs/access_log common env=!dontlog
379 <p>As another example, consider logging requests from english-speakers
380 to one log file, and non-english speakers to a different log file.</p>
383 SetEnvIf Accept-Language "en" english<br>
384 CustomLog logs/english_log common env=english<br>
385 CustomLog logs/non_english_log common env=!english
388 <p>Although we have just shown that conditional logging is very
389 powerful and flexibly, it is not the only way to control the contents
390 of the logs. Log files are more useful when they contain a complete
391 record of server activity. It is often easier to simply post-process
392 the log files to remove requests that you do not want to consider.</p>
396 <h2><a name="rotation">Log Rotation</a></h2>
398 <p>On even a moderately busy server, the quantity of information
399 stored in the log files is very large. The access log file typically
400 grows 1 MB or more per 10,000 requests. It will consequently be
401 necessary to periodically rotate the log files by moving or deleting
402 the existing logs. This cannot be done while the server is running,
403 because Apache will continue writing to the old log file as long as it
404 holds the file open. Instead, the server must be <a
405 href="stopping.html">restarted</a> after the log files are moved or
406 deleted so that it will open new log files.</p>
408 <p>By using a <em>graceful</em> restart, the server can be instructed
409 to open new log files without losing any existing or pending
410 connections from clients. However, in order to accomplish this, the
411 server must continue to write to the old log files while it finishes
412 serving old requests. It is therefore necessary to wait for some time
413 after the restart before doing any processing on the log files. A
414 typical scenario that simply rotates the logs and compresses the old
415 logs to save space is:</p>
418 mv access_log access_log.old<br>
419 mv error_log error_log.old<br>
420 apachectl graceful<br>
422 gzip access_log.old error_log.old
425 <p>Another way to perform log rotation is using <a href="#piped">piped
426 logs</a> as discussed in the next section.</p>
430 <h2><a name="piped">Piped Logs</a></h2>
432 <p>Apache HTTPD is capable of writing error and access log files
433 through a pipe to another process, rather than directly to a file.
434 This capability dramatically increases the flexibility of logging,
435 without adding code to the main server. In order to write logs to a
436 pipe, simply replace the filename with the pipe character
437 "<code>|</code>", followed by the name of the executable which should
438 accept log entries on its standard input. Apache will start the
439 piped-log process when the server starts, and will restart it if it
440 crashes while the server is running. (This last feature is why we can
441 refer to this technique as "reliable piped logging".)</p>
443 <p>Piped log processes are spawned by the parent Apache HTTPD process,
444 and inherit the userid of that process. This means that piped log
445 programs usually run as root. It is therefore very important to keep
446 the programs simple and secure.</p>
448 <p>Some simple examples using piped logs:</p>
451 # compressed logs<br>
452 CustomLog "|/usr/bin/gzip -c >> /var/log/access_log.gz" common<br>
453 # almost-real-time name resolution<br>
454 CustomLog "|/usr/local/apache/bin/logresolve >> /var/log/access_log" common
457 <p>Notice that quotes are used to enclose the entire command
458 that will be called for the pipe. Although these examples are
459 for the access log, the same technique can be used for the
462 <p>One important use of piped logs is to allow log rotation without
463 having to restart the server. Apache HTTPD includes a simple program
464 called <a href="programs/rotatelogs.html">rotatelogs</a> for this
465 purpose. For example, to rotate the logs every 24 hours, you can
469 CustomLog "|/usr/local/apache/bin/rotatelogs /var/log/access_log 86400" common
472 <p>A similar, but much more flexible log rotation program
473 called <a href="http://www.ford-mason.co.uk/resources/cronolog/">cronolog</a>
474 is available at an external site.</p>
476 <p>As with conditional logging, piped logs are a very powerful tool,
477 but they should not be used where a simpler solution like
478 off-line post-processing is available.</p>
482 <h2><a name="virtualhosts">Virtual Hosts</a></h2>
484 <p>When running a server with many <a href="vhosts/">virtual
485 hosts</a>, there are several options for dealing with log files.
486 First, it is possible to use logs exactly as in a single-host server.
487 Simply by placing the logging directives outside the
488 <code><VirtualHost></code> sections in the main server context,
489 it is possible to log all requests in the same access log and error
490 log. This technique does not allow for easy collection of statistics
491 on individual virtual hosts.</p>
493 <p>If <code>CustomLog</code> or <code>ErrorLog</code> directives are
494 placed inside a <code><VirtualHost></code> section, all requests
495 or errors for that virtual host will be logged only to the specified
496 file. Any virtual host which does not have logging directives will
497 still have its requests sent to the main server logs. This technique
498 is very useful for a small number of virtual hosts, but if the number
499 of hosts is very large, it can be complicated to manage. In addition,
500 it can often create problems with <a
501 href="vhosts/fd-limits.html">insufficient file descriptors</a>.</p>
503 <p>For the access log, there is a very good compromise. By adding
504 information on the virtual host to the log format string,
505 it is possible to log all hosts to the same log, and later
506 split the log into individual files. For example, consider the
507 following directives.</p>
510 LogFormat "%v %l %u %t \"%r\" %>s %b" comonvhost<br>
511 CustomLog logs/access_log comonvhost
514 <p>The <code>%v</code> is used to log the name of the virtual host
515 that is serving the request. Then a program like <a
516 href="programs/other.html">split-logfile</a> can be used to
517 post-process the access log in order to split it into one file per
520 <p>Unfortunately, no similar technique is available for the error log,
521 so you must choose between mixing all virtual hosts in the same error
522 log and using one error log per virtual host.</p>
526 <h2><a name="other">Other Log Files</a></h2>
528 <table border=1><tr><td valign="top">
529 <strong>Related Modules</strong><br><br>
531 <a href="mod/mod_cgi.html">mod_cgi</a><br>
532 <a href="mod/mod_rewrite.html">mod_rewrite</a>
534 </td><td valign="top">
535 <strong>Related Directives</strong><br><br>
537 <a href="mod/core.html#pidfile">PidFile</a><br>
538 <a href="mod/mod_rewrite.html#RewriteLog">RewriteLog</a></br>
539 <a href="mod/mod_rewrite.html#RewriteLogLevel">RewriteLogLevel</a></br>
540 <a href="mod/mod_cgi.html#scriptlog">ScriptLog</a><br>
541 <a href="mod/mod_cgi.html#scriptloglength">ScriptLogLength</a><br>
542 <a href="mod/mod_cgi.html#scriptlogbuffer">ScriptLogBuffer</a>
546 <h3><a name="pidfile">PID File</a></h3>
548 <p>On startup, Apache HTTPD saves the process id of the parent httpd
549 process to the file <code>logs/httpd.pid</code>. This filename can be
550 changed with the <A HREF="mod/core.html#pidfile">PidFile</A>
551 directive. The process-id is for use by the administrator in
552 restarting and terminating the daemon by sending signals
553 to the parent process; on Windows, use the -k command line
554 option instead. For more information see the <A
555 HREF="stopping.html">Stopping and Restarting</A> page.
557 <h3><a name="scriptlog">Script Log</a></h3>
559 <p>In order to aid in debugging, the
560 <a href="mod/mod_cgi.html#scriptlog">ScriptLog</a>
561 directive allows you to record the input to and output from
562 CGI scripts. This should only be used in testing - not for
563 live servers. More information is available in the
564 <a href="mod/mod_cgi.html">mod_cgi documentation</a>.
566 <h3><a name="rewritelog">Rewrite Log</a></h3>
568 <p>When using the powerful and complex features of <a
569 href="mod/mod_rewrite.html">mod_rewrite</a>, it is almost always
570 necessary to use the <a
571 href="mod/mod_rewrite.html#RewriteLog">RewriteLog</a> to help in
572 debugging. This log file produces a detailed analysis of how the
573 rewriting engine transforms requests. The level of detail is
575 href="mod/mod_rewrite.html#RewriteLogLevel">RewriteLogLevel</a>
578 <!--#include virtual="footer.html" -->