1 <?xml version="1.0" encoding="UTF-8" ?>
2 <!DOCTYPE manualpage SYSTEM "./style/manualpage.dtd">
3 <?xml-stylesheet type="text/xsl" href="./style/manual.en.xsl"?>
4 <!-- $LastChangedRevision$ -->
7 Licensed to the Apache Software Foundation (ASF) under one or more
8 contributor license agreements. See the NOTICE file distributed with
9 this work for additional information regarding copyright ownership.
10 The ASF licenses this file to You under the Apache License, Version 2.0
11 (the "License"); you may not use this file except in compliance with
12 the License. You may obtain a copy of the License at
14 http://www.apache.org/licenses/LICENSE-2.0
16 Unless required by applicable law or agreed to in writing, software
17 distributed under the License is distributed on an "AS IS" BASIS,
18 WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
19 See the License for the specific language governing permissions and
20 limitations under the License.
23 <manualpage metafile="logs.xml.meta">
25 <title>Log Files</title>
28 <p>In order to effectively manage a web server, it is necessary
29 to get feedback about the activity and performance of the
30 server as well as any problems that may be occurring. The Apache
31 HTTP Server provides very comprehensive and flexible logging
32 capabilities. This document describes how to configure its
33 logging capabilities, and how to understand what the logs
37 <section id="security">
38 <title>Security Warning</title>
40 <p>Anyone who can write to the directory where Apache is
41 writing a log file can almost certainly gain access to the uid
42 that the server is started as, which is normally root. Do
43 <em>NOT</em> give people write access to the directory the logs
44 are stored in without being aware of the consequences; see the
45 <a href="misc/security_tips.html">security tips</a> document
48 <p>In addition, log files may contain information supplied
49 directly by the client, without escaping. Therefore, it is
50 possible for malicious clients to insert control-characters in
51 the log files, so care must be taken in dealing with raw
55 <section id="errorlog">
56 <title>Error Log</title>
60 <directive module="core">ErrorLog</directive>
61 <directive module="core">LogLevel</directive>
65 <p>The server error log, whose name and location is set by the
66 <directive module="core">ErrorLog</directive> directive, is the
67 most important log file. This is the place where Apache httpd
68 will send diagnostic information and record any errors that it
69 encounters in processing requests. It is the first place to
70 look when a problem occurs with starting the server or with the
71 operation of the server, since it will often contain details of
72 what went wrong and how to fix it.</p>
74 <p>The error log is usually written to a file (typically
75 <code>error_log</code> on Unix systems and
76 <code>error.log</code> on Windows). On Unix systems it
77 is also possible to have the server send errors to
78 <code>syslog</code> or <a href="#piped">pipe them to a
81 <p>The format of the error log is relatively free-form and
82 descriptive. But there is certain information that is contained
83 in most error log entries. For example, here is a typical
87 [Wed Oct 11 14:32:52 2000] [error] [client 127.0.0.1]
88 client denied by server configuration:
89 /export/home/live/ap/htdocs/test
92 <p>The first item in the log entry is the date and time of the
93 message. The second item lists the severity of the error being
94 reported. The <directive module="core">LogLevel</directive>
95 directive is used to control the types of errors that are sent
96 to the error log by restricting the severity level. The third
97 item gives the IP address of the client that generated the
98 error. Beyond that is the message itself, which in this case
99 indicates that the server has been configured to deny the
100 client access. The server reports the file-system path (as
101 opposed to the web path) of the requested document.</p>
103 <p>A very wide variety of different messages can appear in the
104 error log. Most look similar to the example above. The error
105 log will also contain debugging output from CGI scripts. Any
106 information written to <code>stderr</code> by a CGI script will
107 be copied directly to the error log.</p>
109 <p>It is not possible to customize the error log by adding or
110 removing information. However, error log entries dealing with
111 particular requests have corresponding entries in the <a
112 href="#accesslog">access log</a>. For example, the above example
113 entry corresponds to an access log entry with status code 403.
114 Since it is possible to customize the access log, you can
115 obtain more information about error conditions using that log
118 <p>During testing, it is often useful to continuously monitor
119 the error log for any problems. On Unix systems, you can
120 accomplish this using:</p>
127 <section id="accesslog">
128 <title>Access Log</title>
132 <module>mod_log_config</module>
133 <module>mod_setenvif</module>
136 <directive module="mod_log_config">CustomLog</directive>
137 <directive module="mod_log_config">LogFormat</directive>
138 <directive module="mod_setenvif">SetEnvIf</directive>
142 <p>The server access log records all requests processed by the
143 server. The location and content of the access log are
144 controlled by the <directive module="mod_log_config">CustomLog</directive>
145 directive. The <directive module="mod_log_config">LogFormat</directive>
146 directive can be used to simplify the selection of
147 the contents of the logs. This section describes how to configure the server
148 to record information in the access log.</p>
150 <p>Of course, storing the information in the access log is only
151 the start of log management. The next step is to analyze this
152 information to produce useful statistics. Log analysis in
153 general is beyond the scope of this document, and not really
154 part of the job of the web server itself. For more information
155 about this topic, and for applications which perform log
156 analysis, check the <a
157 href="http://dmoz.org/Computers/Software/Internet/Site_Management/Log_analysis/">
158 Open Directory</a> or <a
159 href="http://dir.yahoo.com/Computers_and_Internet/Software/Internet/World_Wide_Web/Servers/Log_Analysis_Tools/">
162 <p>Various versions of Apache httpd have used other modules and
163 directives to control access logging, including
164 mod_log_referer, mod_log_agent, and the
165 <code>TransferLog</code> directive. The <directive
166 module="mod_log_config">CustomLog</directive> directive now subsumes
167 the functionality of all the older directives.</p>
169 <p>The format of the access log is highly configurable. The format
170 is specified using a format string that looks much like a C-style
171 printf(1) format string. Some examples are presented in the next
172 sections. For a complete list of the possible contents of the
173 format string, see the <module>mod_log_config</module> <a
174 href="mod/mod_log_config.html#formats">format strings</a>.</p>
176 <section id="common">
177 <title>Common Log Format</title>
179 <p>A typical configuration for the access log might look as
183 LogFormat "%h %l %u %t \"%r\" %>s %b" common<br />
184 CustomLog logs/access_log common
187 <p>This defines the <em>nickname</em> <code>common</code> and
188 associates it with a particular log format string. The format
189 string consists of percent directives, each of which tell the
190 server to log a particular piece of information. Literal
191 characters may also be placed in the format string and will be
192 copied directly into the log output. The quote character
193 (<code>"</code>) must be escaped by placing a back-slash before
194 it to prevent it from being interpreted as the end of the
195 format string. The format string may also contain the special
196 control characters "<code>\n</code>" for new-line and
197 "<code>\t</code>" for tab.</p>
199 <p>The <directive module="mod_log_config">CustomLog</directive>
200 directive sets up a new log file using the defined
201 <em>nickname</em>. The filename for the access log is relative to
202 the <directive module="core">ServerRoot</directive> unless it
203 begins with a slash.</p>
205 <p>The above configuration will write log entries in a format
206 known as the Common Log Format (CLF). This standard format can
207 be produced by many different web servers and read by many log
208 analysis programs. The log file entries produced in CLF will
209 look something like this:</p>
212 127.0.0.1 - frank [10/Oct/2000:13:55:36 -0700] "GET
213 /apache_pb.gif HTTP/1.0" 200 2326
216 <p>Each part of this log entry is described below.</p>
219 <dt><code>127.0.0.1</code> (<code>%h</code>)</dt>
221 <dd>This is the IP address of the client (remote host) which
222 made the request to the server. If <directive
223 module="core">HostnameLookups</directive> is
224 set to <code>On</code>, then the server will try to determine
225 the hostname and log it in place of the IP address. However,
226 this configuration is not recommended since it can
227 significantly slow the server. Instead, it is best to use a
228 log post-processor such as <program>logresolve</program> to determine
229 the hostnames. The IP address reported here is not
230 necessarily the address of the machine at which the user is
231 sitting. If a proxy server exists between the user and the
232 server, this address will be the address of the proxy, rather
233 than the originating machine.</dd>
235 <dt><code>-</code> (<code>%l</code>)</dt>
237 <dd>The "hyphen" in the output indicates that the requested
238 piece of information is not available. In this case, the
239 information that is not available is the RFC 1413 identity of
240 the client determined by <code>identd</code> on the clients
241 machine. This information is highly unreliable and should
242 almost never be used except on tightly controlled internal
243 networks. Apache httpd will not even attempt to determine
244 this information unless <directive
245 module="core">IdentityCheck</directive> is set
246 to <code>On</code>.</dd>
248 <dt><code>frank</code> (<code>%u</code>)</dt>
250 <dd>This is the userid of the person requesting the document
251 as determined by HTTP authentication. The same value is
252 typically provided to CGI scripts in the
253 <code>REMOTE_USER</code> environment variable. If the status
254 code for the request (see below) is 401, then this value
255 should not be trusted because the user is not yet
256 authenticated. If the document is not password protected,
257 this part will be "<code>-</code>" just like the previous
260 <dt><code>[10/Oct/2000:13:55:36 -0700]</code>
261 (<code>%t</code>)</dt>
264 The time that the request was received.
268 <code>[day/month/year:hour:minute:second zone]<br />
270 month = 3*letter<br />
273 minute = 2*digit<br />
274 second = 2*digit<br />
275 zone = (`+' | `-') 4*digit</code>
277 It is possible to have the time displayed in another format
278 by specifying <code>%{format}t</code> in the log format
279 string, where <code>format</code> is as in
280 <code>strftime(3)</code> from the C standard library.
283 <dt><code>"GET /apache_pb.gif HTTP/1.0"</code>
284 (<code>\"%r\"</code>)</dt>
286 <dd>The request line from the client is given in double
287 quotes. The request line contains a great deal of useful
288 information. First, the method used by the client is
289 <code>GET</code>. Second, the client requested the resource
290 <code>/apache_pb.gif</code>, and third, the client used the
291 protocol <code>HTTP/1.0</code>. It is also possible to log
292 one or more parts of the request line independently. For
293 example, the format string "<code>%m %U%q %H</code>" will log
294 the method, path, query-string, and protocol, resulting in
295 exactly the same output as "<code>%r</code>".</dd>
297 <dt><code>200</code> (<code>%>s</code>)</dt>
299 <dd>This is the status code that the server sends back to the
300 client. This information is very valuable, because it reveals
301 whether the request resulted in a successful response (codes
302 beginning in 2), a redirection (codes beginning in 3), an
303 error caused by the client (codes beginning in 4), or an
304 error in the server (codes beginning in 5). The full list of
305 possible status codes can be found in the <a
306 href="http://www.w3.org/Protocols/rfc2616/rfc2616.txt">HTTP
307 specification</a> (RFC2616 section 10).</dd>
309 <dt><code>2326</code> (<code>%b</code>)</dt>
311 <dd>The last part indicates the size of the object returned
312 to the client, not including the response headers. If no
313 content was returned to the client, this value will be
314 "<code>-</code>". To log "<code>0</code>" for no content, use
315 <code>%B</code> instead.</dd>
319 <section id="combined">
320 <title>Combined Log Format</title>
322 <p>Another commonly used format string is called the Combined
323 Log Format. It can be used as follows.</p>
326 LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\"
327 \"%{User-agent}i\"" combined<br />
328 CustomLog log/access_log combined
331 <p>This format is exactly the same as the Common Log Format,
332 with the addition of two more fields. Each of the additional
333 fields uses the percent-directive
334 <code>%{<em>header</em>}i</code>, where <em>header</em> can be
335 any HTTP request header. The access log under this format will
339 127.0.0.1 - frank [10/Oct/2000:13:55:36 -0700] "GET
340 /apache_pb.gif HTTP/1.0" 200 2326
341 "http://www.example.com/start.html" "Mozilla/4.08 [en]
345 <p>The additional fields are:</p>
348 <dt><code>"http://www.example.com/start.html"</code>
349 (<code>\"%{Referer}i\"</code>)</dt>
351 <dd>The "Referer" (sic) HTTP request header. This gives the
352 site that the client reports having been referred from. (This
353 should be the page that links to or includes
354 <code>/apache_pb.gif</code>).</dd>
356 <dt><code>"Mozilla/4.08 [en] (Win98; I ;Nav)"</code>
357 (<code>\"%{User-agent}i\"</code>)</dt>
359 <dd>The User-Agent HTTP request header. This is the
360 identifying information that the client browser reports about
365 <section id="multiple">
366 <title>Multiple Access Logs</title>
368 <p>Multiple access logs can be created simply by specifying
369 multiple <directive module="mod_log_config">CustomLog</directive>
370 directives in the configuration
371 file. For example, the following directives will create three
372 access logs. The first contains the basic CLF information,
373 while the second and third contain referer and browser
374 information. The last two <directive
375 module="mod_log_config">CustomLog</directive> lines show how
376 to mimic the effects of the <code>ReferLog</code> and <code
377 >AgentLog</code> directives.</p>
380 LogFormat "%h %l %u %t \"%r\" %>s %b" common<br />
381 CustomLog logs/access_log common<br />
382 CustomLog logs/referer_log "%{Referer}i -> %U"<br />
383 CustomLog logs/agent_log "%{User-agent}i"
386 <p>This example also shows that it is not necessary to define a
387 nickname with the <directive
388 module="mod_log_config">LogFormat</directive> directive. Instead,
389 the log format can be specified directly in the <directive
390 module="mod_log_config">CustomLog</directive> directive.</p>
393 <section id="conditional">
394 <title>Conditional Logs</title>
396 <p>There are times when it is convenient to exclude certain
397 entries from the access logs based on characteristics of the
398 client request. This is easily accomplished with the help of <a
399 href="env.html">environment variables</a>. First, an
400 environment variable must be set to indicate that the request
401 meets certain conditions. This is usually accomplished with
402 <directive module="mod_setenvif">SetEnvIf</directive>. Then the
403 <code>env=</code> clause of the <directive
404 module="mod_log_config">CustomLog</directive> directive is used to
405 include or exclude requests where the environment variable is
406 set. Some examples:</p>
409 # Mark requests from the loop-back interface<br />
410 SetEnvIf Remote_Addr "127\.0\.0\.1" dontlog<br />
411 # Mark requests for the robots.txt file<br />
412 SetEnvIf Request_URI "^/robots\.txt$" dontlog<br />
413 # Log what remains<br />
414 CustomLog logs/access_log common env=!dontlog
417 <p>As another example, consider logging requests from
418 english-speakers to one log file, and non-english speakers to a
419 different log file.</p>
422 SetEnvIf Accept-Language "en" english<br />
423 CustomLog logs/english_log common env=english<br />
424 CustomLog logs/non_english_log common env=!english
427 <p>Although we have just shown that conditional logging is very
428 powerful and flexible, it is not the only way to control the
429 contents of the logs. Log files are more useful when they
430 contain a complete record of server activity. It is often
431 easier to simply post-process the log files to remove requests
432 that you do not want to consider.</p>
436 <section id="rotation">
437 <title>Log Rotation</title>
439 <p>On even a moderately busy server, the quantity of
440 information stored in the log files is very large. The access
441 log file typically grows 1 MB or more per 10,000 requests. It
442 will consequently be necessary to periodically rotate the log
443 files by moving or deleting the existing logs. This cannot be
444 done while the server is running, because Apache will continue
445 writing to the old log file as long as it holds the file open.
446 Instead, the server must be <a
447 href="stopping.html">restarted</a> after the log files are
448 moved or deleted so that it will open new log files.</p>
450 <p>By using a <em>graceful</em> restart, the server can be
451 instructed to open new log files without losing any existing or
452 pending connections from clients. However, in order to
453 accomplish this, the server must continue to write to the old
454 log files while it finishes serving old requests. It is
455 therefore necessary to wait for some time after the restart
456 before doing any processing on the log files. A typical
457 scenario that simply rotates the logs and compresses the old
458 logs to save space is:</p>
461 mv access_log access_log.old<br />
462 mv error_log error_log.old<br />
463 apachectl graceful<br />
465 gzip access_log.old error_log.old
468 <p>Another way to perform log rotation is using <a
469 href="#piped">piped logs</a> as discussed in the next
474 <title>Piped Logs</title>
476 <p>Apache httpd is capable of writing error and access log
477 files through a pipe to another process, rather than directly
478 to a file. This capability dramatically increases the
479 flexibility of logging, without adding code to the main server.
480 In order to write logs to a pipe, simply replace the filename
481 with the pipe character "<code>|</code>", followed by the name
482 of the executable which should accept log entries on its
483 standard input. Apache will start the piped-log process when
484 the server starts, and will restart it if it crashes while the
485 server is running. (This last feature is why we can refer to
486 this technique as "reliable piped logging".)</p>
488 <p>Piped log processes are spawned by the parent Apache httpd
489 process, and inherit the userid of that process. This means
490 that piped log programs usually run as root. It is therefore
491 very important to keep the programs simple and secure.</p>
493 <p>One important use of piped logs is to allow log rotation
494 without having to restart the server. The Apache HTTP Server
495 includes a simple program called <program>rotatelogs</program>
496 for this purpose. For example, to rotate the logs every 24 hours, you
500 CustomLog "|/usr/local/apache/bin/rotatelogs
501 /var/log/access_log 86400" common
504 <p>Notice that quotes are used to enclose the entire command
505 that will be called for the pipe. Although these examples are
506 for the access log, the same technique can be used for the
509 <p>A similar but much more flexible log rotation program
510 called <a href="http://www.cronolog.org/">cronolog</a>
511 is available at an external site.</p>
513 <p>As with conditional logging, piped logs are a very powerful
514 tool, but they should not be used where a simpler solution like
515 off-line post-processing is available.</p>
518 <section id="virtualhost">
519 <title>Virtual Hosts</title>
521 <p>When running a server with many <a href="vhosts/">virtual
522 hosts</a>, there are several options for dealing with log
523 files. First, it is possible to use logs exactly as in a
524 single-host server. Simply by placing the logging directives
525 outside the <directive module="core"
526 type="section">VirtualHost</directive> sections in the
527 main server context, it is possible to log all requests in the
528 same access log and error log. This technique does not allow
529 for easy collection of statistics on individual virtual
532 <p>If <directive module="mod_log_config">CustomLog</directive>
533 or <directive module="core">ErrorLog</directive>
534 directives are placed inside a
535 <directive module="core" type="section">VirtualHost</directive>
536 section, all requests or errors for that virtual host will be
537 logged only to the specified file. Any virtual host which does
538 not have logging directives will still have its requests sent
539 to the main server logs. This technique is very useful for a
540 small number of virtual hosts, but if the number of hosts is
541 very large, it can be complicated to manage. In addition, it
542 can often create problems with <a
543 href="vhosts/fd-limits.html">insufficient file
546 <p>For the access log, there is a very good compromise. By
547 adding information on the virtual host to the log format
548 string, it is possible to log all hosts to the same log, and
549 later split the log into individual files. For example,
550 consider the following directives.</p>
553 LogFormat "%v %l %u %t \"%r\" %>s %b"
555 CustomLog logs/access_log comonvhost
558 <p>The <code>%v</code> is used to log the name of the virtual
559 host that is serving the request. Then a program like <a
560 href="programs/other.html">split-logfile</a> can be used to
561 post-process the access log in order to split it into one file
562 per virtual host.</p>
566 <title>Other Log Files</title>
570 <module>mod_logio</module>
571 <module>mod_log_forensic</module>
572 <module>mod_cgi</module>
573 <module>mod_rewrite</module>
576 <directive module="mod_log_config">LogFormat</directive>
577 <directive module="mod_log_forensic">ForensicLog</directive>
578 <directive module="mpm_common">PidFile</directive>
579 <directive module="mod_rewrite">RewriteLog</directive>
580 <directive module="mod_rewrite">RewriteLogLevel</directive>
581 <directive module="mod_cgi">ScriptLog</directive>
582 <directive module="mod_cgi">ScriptLogBuffer</directive>
583 <directive module="mod_cgi">ScriptLogLength</directive>
588 <title>Logging actual bytes sent and received</title>
590 <p><module>mod_logio</module> adds in two additional
591 <directive module="mod_log_config">LogFormat</directive> fields
592 (%I and %O) that log the actual number of bytes received and sent
597 <title>Forensic Logging</title>
599 <p><module>mod_log_forensic</module> provides for forensic logging of
600 client requests. Logging is done before and after processing a
601 request, so the forensic log contains two log lines for each
602 request. The forensic logger is very strict with no customizations.
603 It can be an invaluable debugging and security tool.</p>
606 <section id="pidfile">
607 <title>PID File</title>
609 <p>On startup, Apache httpd saves the process id of the parent
610 httpd process to the file <code>logs/httpd.pid</code>. This
611 filename can be changed with the <directive
612 module="mpm_common">PidFile</directive> directive. The
613 process-id is for use by the administrator in restarting and
614 terminating the daemon by sending signals to the parent
615 process; on Windows, use the -k command line option instead.
616 For more information see the <a href="stopping.html">Stopping
617 and Restarting</a> page.</p>
620 <section id="scriptlog">
621 <title>Script Log</title>
623 <p>In order to aid in debugging, the
624 <directive module="mod_cgi">ScriptLog</directive> directive
625 allows you to record the input to and output from CGI scripts.
626 This should only be used in testing - not for live servers.
627 More information is available in the <a
628 href="mod/mod_cgi.html">mod_cgi</a> documentation.</p>
631 <section id="rewritelog">
632 <title>Rewrite Log</title>
634 <p>When using the powerful and complex features of <a
635 href="mod/mod_rewrite.html">mod_rewrite</a>, it is almost
636 always necessary to use the <directive
637 module="mod_rewrite">RewriteLog</directive> to help
638 in debugging. This log file produces a detailed analysis of how
639 the rewriting engine transforms requests. The level of detail
640 is controlled by the <directive
641 module="mod_rewrite">RewriteLogLevel</directive> directive.</p>