From: Darold Gilles Date: Tue, 6 Jan 2015 21:54:03 +0000 (+0100) Subject: Reformat usage output and add explanation of the --noreport command line option. X-Git-Tag: v6.3~15 X-Git-Url: https://granicus.if.org/sourcecode?a=commitdiff_plain;h=b705535540d2317176eb13a9091b8b2c407c182c;p=pgbadger Reformat usage output and add explanation of the --noreport command line option. --- diff --git a/README b/README index 282e7b5..2a7f4a1 100644 --- a/README +++ b/README @@ -26,7 +26,8 @@ SYNOPSIS -D | --dns-resolv : client ip adresses are replaced by their DNS name. Be warned that this can really slow down pgBadger. -e | --end datetime : end date/time for the data to be parsed in log. - -f | --format logtype : possible values: syslog,stderr,csv. Default: stderr. + -f | --format logtype : possible values: syslog, syslog2, stderr and csv. + Default: stderr. -G | --nograph : disable graphs on HTML output. Enabled by default. -h | --help : show this message and exit. -i | --ident name : programname used as syslog ident. Default: postgres @@ -56,46 +57,50 @@ SYNOPSIS -p | --prefix string : the value of your custom log_line_prefix as defined in your postgresql.conf. Only use it if you aren't using one of the standard prefixes specified - in the pgBadger documentation, such as if your prefix - includes additional variables like client ip or - application name. See examples below. + in the pgBadger documentation, such as if your + prefix includes additional variables like client ip + or application name. See examples below. -P | --no-prettify : disable SQL queries prettify formatter. - -q | --quiet : don't print anything to stdout, not even a progress bar. - -r | --remote-host ip : set the host where to execute the cat command on remote - logfile to parse localy the file. - -R | --retention N : number of week to keep in incremental mode. Default 0, - disabled. Used to set the number of weel to keep in - output directory. Older week end day directory are - automatically removed. - -s | --sample number : number of query samples to store/display. Default: 3 + -q | --quiet : don't print anything to stdout, not even a progress bar. + -r | --remote-host ip : set the host where to execute the cat command on + remote logfile to parse localy the file. + -R | --retention N : number of week to keep in incremental mode. Default + to 0, disabled. Used to set the number of weel to + keep in output directory. Older weeks and days + directory are automatically removed. + -s | --sample number : number of query samples to store. Default: 3. -S | --select-only : only report SELECT queries. - -t | --top number : number of queries to store/display. Default: 20 + -t | --top number : number of queries to store/display. Default: 20. -T | --title string : change title of the HTML page report. -u | --dbuser username : only report on entries for the given user. - -U | --exclude-user username : exclude entries for the specified user from report. + -U | --exclude-user username : exclude entries for the specified user from + report. -v | --verbose : enable verbose or debug mode. Disabled by default. -V | --version : show pgBadger version and exit. -w | --watch-mode : only report errors just like logwatch could do. -x | --extension : output format. Values: text, html, bin, json or tsung. Default: html - -X | --extra-files : in incremetal mode allow pgbadger to write CSS and JS - files in the output directory as separate files. + -X | --extra-files : in incremetal mode allow pgbadger to write CSS and + JS files in the output directory as separate files. -z | --zcat exec_path : set the full path to the zcat program. Use it if zcat or bzcat or unzip is not in your path. --pie-limit num : pie data lower than num% will show a sum instead. --exclude-query regex : any query matching the given regex will be excluded from the report. For example: "^(VACUUM|COMMIT)" You can use this option multiple times. - --exclude-file filename: path of the file which contains all the regex to use - to exclude queries from the report. One regex per line. - --include-query regex : any query that does not match the given regex will be - excluded from the report. For example: "(table_1|table_2)" - You can use this option multiple times. - --include-file filename: path of the file which contains all the regex of the - queries to include from the report. One regex per line. + --exclude-file filename: path of the file which contains all the regex to + use to exclude queries from the report. One regex + per line. + --include-query regex : any query that does not match the given regex will + be excluded from the report. You can use this + option multiple times. For example: "(tbl1|tbl2)". + --include-file filename: path of the file which contains all the regex of + the queries to include from the report. One regex + per line. --disable-error : do not generate error report. --disable-hourly : do not generate hourly report. - --disable-type : do not generate report of queries by type, database... + --disable-type : do not generate report of queries by type, database + or user. --disable-query : do not generate query reports (slowest, most frequent, queries by users, by database, ...). --disable-session : do not generate session report. @@ -104,7 +109,8 @@ SYNOPSIS --disable-temporary : do not generate temporary report. --disable-checkpoint : do not generate checkpoint/restartpoint report. --disable-autovacuum : do not generate autovacuum report. - --charset : used to set the HTML charset to be used. Default: utf-8. + --charset : used to set the HTML charset to be used. + Default: utf-8. --csv-separator : used to set the CSV field separator, default: , --exclude-time regex : any timestamp matching the given regex will be excluded from the report. Example: "2013-04-12 .*" @@ -112,8 +118,11 @@ SYNOPSIS --exclude-appname name : exclude entries for the specified application name from report. Example: "pg_dump". --exclude-line regex : pgbadger will start to exclude any log entry that - will match the given regex. Can be used multiple time. + will match the given regex. Can be used multiple + time. --anonymize : obscure all literals in queries to hide confidential data. + --noreport : prevent pgbadger to create reports in incremental + mode. pgBadger is able to parse a remote log file using a passwordless ssh connection. Use the -r or --remote-host to set the host ip address or @@ -123,9 +132,9 @@ SYNOPSIS --ssh-program ssh path to the ssh program to use. Default: ssh. --ssh-user username connection login name. Default to running user. --ssh-identity file path to the identity file to use. - --ssh-timeout second timeout to ssh connection failure. Default 10 seconds. - --ssh-options options list of -o options to use for the ssh connection. Options - always used: + --ssh-timeout second timeout to ssh connection failure. Default 10 secs. + --ssh-options options list of -o options to use for the ssh connection. + Options always used: -o ConnectTimeout=$ssh_timeout -o PreferredAuthentications=hostbased,publickey @@ -211,7 +220,7 @@ DESCRIPTION the report using command line options. pgBadger supports any custom format set into the log_line_prefix - directive of your postgresql.conf file as long as it at least specifies + directive of your postgresql.conf file as long as it at least specify the %t and %p patterns. pgBadger allow parallel processing on a single log file and multiple @@ -235,7 +244,7 @@ FEATURE The most frequent errors. Histogram of query times. - The following reports are also available with hourly charts divided by + The following reports are also available with hourly charts divide by periods of five minutes: SQL queries statistics. @@ -266,8 +275,8 @@ FEATURE per hour, but you can specify the granularity down to the minute. pgBadger can also be used in a central place to parse remote log files - using a password-less SSH connection. This mode can be used with - compressed files and in multiprocess per file mode (-J) but can not be + using a password less SSH connection. This mode can be used with + compressed files and in mode multiprocess per file (-J) but can not be used with CSV log format. REQUIREMENT @@ -292,7 +301,7 @@ REQUIREMENT don't need to install it. Compressed log file format is autodetected from the file exension. If - pgBadger finds a gz extension it will use the zcat utility, with a bz2 + pgBadger find a gz extension it will use the zcat utility, with a bz2 extension it will use bzcat and if the file extension is zip or xz then the unzip or xz utilities will be used. @@ -303,8 +312,8 @@ REQUIREMENT --zcat="C:\tools\unzip -p" By default pgBadger will use the zcat, bzcat and unzip utilities - following the file extension. If you use the default (autodetect - compress format) you can mixed gz, bz2, xz or zip files. Specifying a + following the file extension. If you use the default autodetection + compress format you can mixed gz, bz2, xz or zip files. Specifying a custom value to --zcat option will remove this feature of mixed compressed format. @@ -342,7 +351,7 @@ POSTGRESQL CONFIGURATION log_min_duration_statement = 0 - Here every statement will be logged, on a busy server you may want to + Here every statement will be logged, on busy server you may want to increase this value to only log queries with a higher duration time. Note that if you have log_statement set to 'all' nothing will be logged through log_min_duration_statement. See next chapter for more @@ -428,9 +437,9 @@ PARALLEL PROCESSING memory to build the html output. With that method, at start/end of chunks pgbadger may truncate or omit a - maximum of N queries per log file which is an insignificant gap if you + maximum of N queries perl log file which is an insignificant gap if you have millions of queries in your log file. The chance that the query - that you were looking for is lost is near 0, this is why I think this + that you were looking for is loose is near 0, this is why I think this gap is livable. Most of the time the query is counted twice but truncated. @@ -438,9 +447,9 @@ PARALLEL PROCESSING dedicate one core to one log file at a time. To enable this behavior you have to use option -J N instead. With 200 log files of 10MB each the use of the -J option start being really interesting with 8 Cores. Using this - method you will be sure to not lose any queries in the reports. + method you will be sure to not loose any queries in the reports. - Here is a benchmark done on a server with 8 CPUs and a single file of + He are a benchmarck done on a server with 8 CPUs and a single file of 9.5GB. Option | 1 CPU | 2 CPU | 4 CPU | 8 CPU @@ -448,7 +457,7 @@ PARALLEL PROCESSING -j | 1h41m18 | 50m25 | 25m39 | 15m58 -J | 1h41m18 | 54m28 | 41m16 | 34m45 - With 200 log files of 10MB each and a total of 2GB the results are + With 200 log files of 10MB each and a total og 2GB the results are slightly different: Option | 1 CPU | 2 CPU | 4 CPU | 8 CPU @@ -476,7 +485,7 @@ INCREMENTAL REPORTS The main index file will show a dropdown menu per week with a link to the week report and links to daily reports of this week. - For example, if you run pgBadger as follows, based on a daily rotated + For example, if you run pgBadger as follow based on a daily rotated file: 0 4 * * * /usr/bin/pgbadger -I -q /var/log/postgresql/postgresql.log.1 \ diff --git a/doc/pgBadger.pod b/doc/pgBadger.pod index 42ccb27..1ad4025 100644 --- a/doc/pgBadger.pod +++ b/doc/pgBadger.pod @@ -28,7 +28,8 @@ Options: -D | --dns-resolv : client ip adresses are replaced by their DNS name. Be warned that this can really slow down pgBadger. -e | --end datetime : end date/time for the data to be parsed in log. - -f | --format logtype : possible values: syslog,stderr,csv. Default: stderr. + -f | --format logtype : possible values: syslog, syslog2, stderr and csv. + Default: stderr. -G | --nograph : disable graphs on HTML output. Enabled by default. -h | --help : show this message and exit. -i | --ident name : programname used as syslog ident. Default: postgres @@ -58,46 +59,50 @@ Options: -p | --prefix string : the value of your custom log_line_prefix as defined in your postgresql.conf. Only use it if you aren't using one of the standard prefixes specified - in the pgBadger documentation, such as if your prefix - includes additional variables like client ip or - application name. See examples below. + in the pgBadger documentation, such as if your + prefix includes additional variables like client ip + or application name. See examples below. -P | --no-prettify : disable SQL queries prettify formatter. - -q | --quiet : don't print anything to stdout, not even a progress bar. - -r | --remote-host ip : set the host where to execute the cat command on remote - logfile to parse localy the file. - -R | --retention N : number of week to keep in incremental mode. Default 0, - disabled. Used to set the number of weel to keep in - output directory. Older week end day directory are - automatically removed. - -s | --sample number : number of query samples to store/display. Default: 3 + -q | --quiet : don't print anything to stdout, not even a progress bar. + -r | --remote-host ip : set the host where to execute the cat command on + remote logfile to parse localy the file. + -R | --retention N : number of week to keep in incremental mode. Default + to 0, disabled. Used to set the number of weel to + keep in output directory. Older weeks and days + directory are automatically removed. + -s | --sample number : number of query samples to store. Default: 3. -S | --select-only : only report SELECT queries. - -t | --top number : number of queries to store/display. Default: 20 + -t | --top number : number of queries to store/display. Default: 20. -T | --title string : change title of the HTML page report. -u | --dbuser username : only report on entries for the given user. - -U | --exclude-user username : exclude entries for the specified user from report. + -U | --exclude-user username : exclude entries for the specified user from + report. -v | --verbose : enable verbose or debug mode. Disabled by default. -V | --version : show pgBadger version and exit. -w | --watch-mode : only report errors just like logwatch could do. -x | --extension : output format. Values: text, html, bin, json or tsung. Default: html - -X | --extra-files : in incremetal mode allow pgbadger to write CSS and JS - files in the output directory as separate files. + -X | --extra-files : in incremetal mode allow pgbadger to write CSS and + JS files in the output directory as separate files. -z | --zcat exec_path : set the full path to the zcat program. Use it if zcat or bzcat or unzip is not in your path. --pie-limit num : pie data lower than num% will show a sum instead. --exclude-query regex : any query matching the given regex will be excluded from the report. For example: "^(VACUUM|COMMIT)" You can use this option multiple times. - --exclude-file filename: path of the file which contains all the regex to use - to exclude queries from the report. One regex per line. - --include-query regex : any query that does not match the given regex will be - excluded from the report. For example: "(table_1|table_2)" - You can use this option multiple times. - --include-file filename: path of the file which contains all the regex of the - queries to include from the report. One regex per line. + --exclude-file filename: path of the file which contains all the regex to + use to exclude queries from the report. One regex + per line. + --include-query regex : any query that does not match the given regex will + be excluded from the report. You can use this + option multiple times. For example: "(tbl1|tbl2)". + --include-file filename: path of the file which contains all the regex of + the queries to include from the report. One regex + per line. --disable-error : do not generate error report. --disable-hourly : do not generate hourly report. - --disable-type : do not generate report of queries by type, database... + --disable-type : do not generate report of queries by type, database + or user. --disable-query : do not generate query reports (slowest, most frequent, queries by users, by database, ...). --disable-session : do not generate session report. @@ -106,7 +111,8 @@ Options: --disable-temporary : do not generate temporary report. --disable-checkpoint : do not generate checkpoint/restartpoint report. --disable-autovacuum : do not generate autovacuum report. - --charset : used to set the HTML charset to be used. Default: utf-8. + --charset : used to set the HTML charset to be used. + Default: utf-8. --csv-separator : used to set the CSV field separator, default: , --exclude-time regex : any timestamp matching the given regex will be excluded from the report. Example: "2013-04-12 .*" @@ -114,8 +120,11 @@ Options: --exclude-appname name : exclude entries for the specified application name from report. Example: "pg_dump". --exclude-line regex : pgbadger will start to exclude any log entry that - will match the given regex. Can be used multiple time. + will match the given regex. Can be used multiple + time. --anonymize : obscure all literals in queries to hide confidential data. + --noreport : prevent pgbadger to create reports in incremental + mode. pgBadger is able to parse a remote log file using a passwordless ssh connection. Use the -r or --remote-host to set the host ip address or hostname. There's also @@ -124,9 +133,9 @@ some additional options to fully control the ssh connection. --ssh-program ssh path to the ssh program to use. Default: ssh. --ssh-user username connection login name. Default to running user. --ssh-identity file path to the identity file to use. - --ssh-timeout second timeout to ssh connection failure. Default 10 seconds. - --ssh-options options list of -o options to use for the ssh connection. Options - always used: + --ssh-timeout second timeout to ssh connection failure. Default 10 secs. + --ssh-options options list of -o options to use for the ssh connection. + Options always used: -o ConnectTimeout=$ssh_timeout -o PreferredAuthentications=hostbased,publickey @@ -173,7 +182,8 @@ Or better, use the auto-generated incremental reports: will generate a report per day and per week. -In incremental mode, you can also specify the number of week to keep in the reports: +In incremental mode, you can also specify the number of week to keep in the +reports: /usr/bin/pgbadger --retention 2 -I -q /var/log/postgresql/postgresql.log.1 -O /var/www/pg_reports/ @@ -187,6 +197,7 @@ This will help avoid having COPY statements, as generated by pg_dump, on top of the list of slowest queries. You can also use --exclude-appname "pg_dump" to solve this problem in a simpler way. + =head1 DESCRIPTION pgBadger is a PostgreSQL log analyzer build for speed with fully detailed diff --git a/pgbadger b/pgbadger index 25d6fae..08b035b 100755 --- a/pgbadger +++ b/pgbadger @@ -1576,7 +1576,8 @@ Options: -D | --dns-resolv : client ip adresses are replaced by their DNS name. Be warned that this can really slow down pgBadger. -e | --end datetime : end date/time for the data to be parsed in log. - -f | --format logtype : possible values: syslog,stderr,csv. Default: stderr. + -f | --format logtype : possible values: syslog, syslog2, stderr and csv. + Default: stderr. -G | --nograph : disable graphs on HTML output. Enabled by default. -h | --help : show this message and exit. -i | --ident name : programname used as syslog ident. Default: postgres @@ -1606,46 +1607,50 @@ Options: -p | --prefix string : the value of your custom log_line_prefix as defined in your postgresql.conf. Only use it if you aren't using one of the standard prefixes specified - in the pgBadger documentation, such as if your prefix - includes additional variables like client ip or - application name. See examples below. + in the pgBadger documentation, such as if your + prefix includes additional variables like client ip + or application name. See examples below. -P | --no-prettify : disable SQL queries prettify formatter. - -q | --quiet : don't print anything to stdout, not even a progress bar. - -r | --remote-host ip : set the host where to execute the cat command on remote - logfile to parse localy the file. - -R | --retention N : number of week to keep in incremental mode. Default 0, - disabled. Used to set the number of weel to keep in - output directory. Older week end day directory are - automatically removed. - -s | --sample number : number of query samples to store/display. Default: 3 + -q | --quiet : don't print anything to stdout, not even a progress bar. + -r | --remote-host ip : set the host where to execute the cat command on + remote logfile to parse localy the file. + -R | --retention N : number of week to keep in incremental mode. Default + to 0, disabled. Used to set the number of weel to + keep in output directory. Older weeks and days + directory are automatically removed. + -s | --sample number : number of query samples to store. Default: 3. -S | --select-only : only report SELECT queries. - -t | --top number : number of queries to store/display. Default: 20 + -t | --top number : number of queries to store/display. Default: 20. -T | --title string : change title of the HTML page report. -u | --dbuser username : only report on entries for the given user. - -U | --exclude-user username : exclude entries for the specified user from report. + -U | --exclude-user username : exclude entries for the specified user from + report. -v | --verbose : enable verbose or debug mode. Disabled by default. -V | --version : show pgBadger version and exit. -w | --watch-mode : only report errors just like logwatch could do. -x | --extension : output format. Values: text, html, bin, json or tsung. Default: html - -X | --extra-files : in incremetal mode allow pgbadger to write CSS and JS - files in the output directory as separate files. + -X | --extra-files : in incremetal mode allow pgbadger to write CSS and + JS files in the output directory as separate files. -z | --zcat exec_path : set the full path to the zcat program. Use it if zcat or bzcat or unzip is not in your path. --pie-limit num : pie data lower than num% will show a sum instead. --exclude-query regex : any query matching the given regex will be excluded from the report. For example: "^(VACUUM|COMMIT)" You can use this option multiple times. - --exclude-file filename: path of the file which contains all the regex to use - to exclude queries from the report. One regex per line. - --include-query regex : any query that does not match the given regex will be - excluded from the report. For example: "(table_1|table_2)" - You can use this option multiple times. - --include-file filename: path of the file which contains all the regex of the - queries to include from the report. One regex per line. + --exclude-file filename: path of the file which contains all the regex to + use to exclude queries from the report. One regex + per line. + --include-query regex : any query that does not match the given regex will + be excluded from the report. You can use this + option multiple times. For example: "(tbl1|tbl2)". + --include-file filename: path of the file which contains all the regex of + the queries to include from the report. One regex + per line. --disable-error : do not generate error report. --disable-hourly : do not generate hourly report. - --disable-type : do not generate report of queries by type, database... + --disable-type : do not generate report of queries by type, database + or user. --disable-query : do not generate query reports (slowest, most frequent, queries by users, by database, ...). --disable-session : do not generate session report. @@ -1654,7 +1659,8 @@ Options: --disable-temporary : do not generate temporary report. --disable-checkpoint : do not generate checkpoint/restartpoint report. --disable-autovacuum : do not generate autovacuum report. - --charset : used to set the HTML charset to be used. Default: utf-8. + --charset : used to set the HTML charset to be used. + Default: utf-8. --csv-separator : used to set the CSV field separator, default: , --exclude-time regex : any timestamp matching the given regex will be excluded from the report. Example: "2013-04-12 .*" @@ -1662,8 +1668,11 @@ Options: --exclude-appname name : exclude entries for the specified application name from report. Example: "pg_dump". --exclude-line regex : pgbadger will start to exclude any log entry that - will match the given regex. Can be used multiple time. + will match the given regex. Can be used multiple + time. --anonymize : obscure all literals in queries to hide confidential data. + --noreport : prevent pgbadger to create reports in incremental + mode. pgBadger is able to parse a remote log file using a passwordless ssh connection. Use the -r or --remote-host to set the host ip address or hostname. There's also @@ -1672,9 +1681,9 @@ some additional options to fully control the ssh connection. --ssh-program ssh path to the ssh program to use. Default: ssh. --ssh-user username connection login name. Default to running user. --ssh-identity file path to the identity file to use. - --ssh-timeout second timeout to ssh connection failure. Default 10 seconds. - --ssh-options options list of -o options to use for the ssh connection. Options - always used: + --ssh-timeout second timeout to ssh connection failure. Default 10 secs. + --ssh-options options list of -o options to use for the ssh connection. + Options always used: -o ConnectTimeout=\$ssh_timeout -o PreferredAuthentications=hostbased,publickey @@ -1721,7 +1730,8 @@ Or better, use the auto-generated incremental reports: will generate a report per day and per week. -In incremental mode, you can also specify the number of week to keep in the reports: +In incremental mode, you can also specify the number of week to keep in the +reports: /usr/bin/pgbadger --retention 2 -I -q /var/log/postgresql/postgresql.log.1 \ -O /var/www/pg_reports/