Sebastien GODARD [Fri, 14 Dec 2018 14:36:19 +0000 (15:36 +0100)]
sysstat-12.1.2
sysstat version 12.1.2 final packaging.
lsm and spec files updated.
Changelog added.
This version adds support for discard I/O statistics to iostat and sar
commands. Discard I/Os have been separated from writes in Linux kernel
4.19.
This version also fixes several important bugs.
sadf: Add discard I/O metric to all "sar -b" output formats
In addition to "sar -d" statistics, new discard I/O metrics should also be
available for "sar -b" statistics. Add these metrics to sadf output
formats.
"sar -b" statistics are calculated as the sum of individual
statistics for each whole device in the system. Since discard I/Os are
now separated from writes in the kernel, take them into account when
calculating "sar -b" statistics.
New fields are added to "sar -b" output: dtps and bdscd/s.
Check that structure's size is greater than 0 even for
unknown activities.
Else trying to skip this unknown activity in read_file_stat_bunch()
function will decrease offset pointer in file and result in an
infinite loop:
rd_stats.h: Reorder fields in stats_disk structure
New field dc_ticks must be added *after* all other fields of same type
("unsigned int" here). This is compulsory to make older versions of
sar/sadf able to read this structure.
Todd Walton [Tue, 27 Nov 2018 19:24:40 +0000 (14:24 -0500)]
Clarify sadc(8) on using multiple -S keywords
I modified man/sadc.in to show the -S option being followed by
"keyword", and then listed the keywords. This allows the syntax to more
clearly show that multiple keywords can be specified by comma-separating
them. It also reflects the language used in sar(1) for the -n option.
Also, I modified the FAQ to call out that one can specify multiple
keywords for -S by comma-separating them.
Sebastien GODARD [Mon, 26 Nov 2018 13:32:05 +0000 (14:32 +0100)]
Fix #196 and #199: Out of bound reads security issues
Check args before calling memmove() and memset() in remap_struct()
function to avoid out of bound reads which would possibly lead to
unknown code execution and/or sadf command crash.
Sebastien GODARD [Fri, 16 Nov 2018 16:15:11 +0000 (17:15 +0100)]
iostat: Add initial support for discard I/O
Statistics for discard operations have been recently added to Linux
kernel 4.19. Previous kernels didn't separate discards from writes
in the block layer statistics.
This patch adds initial support for discard I/O tracking to iostat.
It adds several new metrics which are similar to those already existing
for reads and writes.
Reports are also modified so that the user can better choose how things
will be displayed, using switches like -s (to get a narrow output
fitting on a 80-characters wide screen) or -h (to get a human-friendly
output): See below.
This patch still needs to be tested (I don't have a 4.19 kernel yet!),
completed (e.g. JSON output) and documented (manual pages will be
updated in a coming patch).
Service time value (displayed as svctm by sar and iostat) could no
longer be trusted and has been made obsolete for several years. So
remove it now from sar (sar -d) and iostat output.
This patch adds new non regression tests for:
* sadf JSON and XML outputs: The output for sar -u is a bit different
than that for sar -u ALL and thus also needs to be tested,
* mpstat JSON output,
* iostat JSON output.
Sebastien GODARD [Sun, 14 Oct 2018 15:02:28 +0000 (17:02 +0200)]
Add new non regression tests for XML output
Add two new non regression tests that check the XML output of
sadf -x against DTD and XSD documents.
The tests are run only if the xmllint command exists.
Sebastien GODARD [Sat, 13 Oct 2018 07:49:36 +0000 (09:49 +0200)]
sysstat-12.1.1
sysstat version 12.1.1 final packaging.
lsm and spec files updated.
Changelog added.
This version mainly includes bugfixes.
The only noteworthy new feature is the addition of two new metrics to
sar -H (Hugepage activity): HugePages_Rsvd and HugePages_Surp.
Also sadf has been made more robust to corrupted datafiles, enabling it
to complete its job even if an unexpected end of datafile is met.
Make sure multiplication result won't overflow integer type
Make sure that multiplication result won't overflow int type before it
is converted to size_t (which is unsigned int or unsigned long according
to GNU C library documentation)
Use thread-safe versions of gmtime() and localtime()
It is safer to use gmtime_r() and localtime_r() functions (which are
thread-safe) instead of gmtime() and localtime() ones, even though
sysstat's code doesn't use multiple threads.
When a system activity datafile has been corrupted, sadf may not be able
to do its job completely. For example reaching the end of file
unexpectedly will make sadf stop, preventing it from, e.g. displaying
its SVG output (sadf -g) or terminating its JSON or XML file properly
(sadf -j or sadf -x).
This patch tells sadf to keep on working even if the end of file has
been reached unexpectedly, enabling it to finish its job.
Note: sar is not really concerned by this problem because you can tell
it to stop before the problem occurs using its option -e.
Sebastien GODARD [Mon, 27 Aug 2018 08:24:16 +0000 (10:24 +0200)]
sadf: Take into account new huge pages metrics
Update various sadf formats (JSON, XML, raw, etc.) to take into account
new HugePages_Rsvd and HugePages_Surp metrics.
SVG output format will be updated later.
Sebastien GODARD [Sun, 26 Aug 2018 13:54:33 +0000 (15:54 +0200)]
sar: Fix timestamp format in report output
The contents of environment variable S_TIME_FORMAT was not taken into
account when displaying a RESTART or a COMMENT message if this message
was the first line displayed in the report by sar:
$ sar
Linux 4.17.11-200.fc28.x86_64 (xxx) 08/25/2018 _x86_64_ (8 CPU)
14:10:50 LINUX RESTART (8 CPU)
02:15:47 PM CPU %user %nice %system %iowait %steal %idle
02:20:00 PM all 6.27 0.00 2.17 3.85 0.00 87.71
...
This patch fixes the problem by testing the environment variable sooner.
Sebastien GODARD [Sat, 25 Aug 2018 13:53:39 +0000 (15:53 +0200)]
sar: Fortify remap_struct() function
New metrics can be added to sar with new sysstat versions. These new
metrics correspond to new fields added to existing statistics
structures.
For compatibility reasons, sysstat assumes that these structures
(identified by their number of fields of different types: long long,
long and int) can only grow with newer sysstat versions. Yet commit c2f9d16
has fixed a longstanding bug regarding a wrong size used for structure
stats_huge. With this bugfix structure stats_huge has now a size smaller
than before. We need to update remap_struct() function to deal with
this. Else adding new fields to structure stats_huge will make sar
unable to read older binary datafiles and crash:
The idea here is to make it possible for the user to *unselect* an
activity which is collected by default by sadc.
For example, you can now choose to collect all
possible statistics except those for interrupts:
sysstat version 12.0.0 final packaging.
lsm file updated.
Typo fixed in sadf.c.
Changelog added.
Version 12.0.0 mainly focuses on SVG output from the sadf command,
with a lot of bugfixes and the addition of a new option ("showtoc")
that will enable the user to quickly navigate through graphs.
Version 12.0.0 is also the first stable version that includes the new,
compact and flexible binary format for sar.
Update spec file to version 12.0.0.
Add an entry to prevent from creating a debug package (that we don't
need). Else we have the following error when creating the RPM packages:
iostat.c: In function ‘read_sysfs_dlist_part_stat’:
iostat.c:576:39: warning: ‘%s’ directive output may be truncated writing up to 255 bytes into a region of size between 0 and 1023 [-Wformat-truncation=]
snprintf(filename, MAX_PF_NAME, "%s/%s/%s", dfile, drd->d_name, S_STAT);
^~
iostat.c:576:3: note: ‘snprintf’ output between 7 and 1285 bytes into a destination of size 1024
snprintf(filename, MAX_PF_NAME, "%s/%s/%s", dfile, drd->d_name, S_STAT);
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
sadf_misc.c: In function ‘print_dbppc_timestamp’:
sadf_misc.c:486:28: warning: ‘snprintf’ output may be truncated before the last format character [-Wformat-truncation=]
snprintf(temp, 80, "%s%s ", pre, cur_date);
^
sadf_misc.c:486:3: note: ‘snprintf’ output 2 or more bytes (assuming 81) into a destination of size 80
snprintf(temp, 80, "%s%s ", pre, cur_date);
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
sadf_misc.c:491:25: warning: ‘%s’ directive output may be truncated writing up to 4 bytes into a region of size between 1 and 80 [-Wformat-truncation=]
snprintf(pre, 80, "%s%s%s", temp, cur_time,
^~
sadf_misc.c:491:2: note: ‘snprintf’ output 1 or more bytes (assuming 84) into a destination of size 80
snprintf(pre, 80, "%s%s%s", temp, cur_time,
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
strlen(cur_date) && utc ? " UTC" : "");
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
sa_common.c: Fix pointer signedness difference warning from gcc
Remove "unsigned" attribute to avoid warning from gcc:
sa_common.c: In function ‘add_list_item’:
sa_common.c:2516:20: warning: pointer targets in assignment from ‘unsigned char *’ to ‘char *’ differ in signedness [-Wpointer-sign]
if ((e->item_name = (unsigned char *) malloc(len + 1)) == NULL) {
^
svg_stats.c: In function ‘recappend’:
svg_stats.c:549:25: warning: ‘ L’ directive output may be truncated writing 2 bytes into a region of size between 1 and 128 [-Wformat-truncation=]
snprintf(data, 128, "%s L%llu,%.2f%s", restart ? data1 : "", timetag, p_value,
^~
svg_stats.c:549:2: note: ‘snprintf’ output between 9 and 591 bytes into a destination of size 128
snprintf(data, 128, "%s L%llu,%.2f%s", restart ? data1 : "", timetag, p_value,
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
p_value != value ? data2 : "");
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
sar/sadf: Devices list management code refactoring
No longer use separate lists for devices entered on the command line
with options --dev, --iface and --fs.
Devices are now saved in the list @item_list which is part of structure
activity. This is a linked list whose number of elements is given by
@item_list_sz (also part of structure activity).
Change constant values to avoid defining arrays size as
type array[CONSTANT_VALUE + 1];
This is error-prone and not consistent with the rest of sysstat code.
Remove assignment from incompatible pointer type warnings
Remove warnings given by gcc, e.g.:
sadf_misc.c: In function ‘count_new_net_dev’:
sadf_misc.c:1033:6: warning: assignment to ‘void **’ from incompatible pointer type ‘struct interface_lst **’ [-Wincompatible-pointer-types]
p = &(e->next);
^
sadf: SVG: Fix function used to assess canvas height
Make sure that we count all the items once and only once for those
which can be registered or unregistered by the system (network
interfaces, block devices...). This is needed to assess SVG canvas
height since each of them will have its own graph.
Explicitly set "item_name" variable before using it to avoid some
warnings by the compiler, e.g.:
svg_stats.c: In function ‘svg_print_net_dev_stats’:
svg_stats.c:2409:5: warning: ‘item_name’ may be used uninitialized in this function [-Wmaybe-uninitialized]
strncpy(item_name, sndc->interface, CHUNKSIZE);
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
sadf: SVG: Fix core dumped on buffers reallocation
When buffers used to save graphs data needed to be extended to
accomodate for new devices, the new size was wrongly calculated
and a core dump was then created.
Check return code of some snprintf() functions to quiet gcc about
possible truncated output, e.g.:
12,18d11
< common.c: In function ‘get_dev_part_nr’:
< common.c:341:35: warning: ‘%s’ directive output may be truncated writing up to 255 bytes into a region of size between 0 and 1023 [-Wformat-truncation=]
< snprintf(line, MAX_PF_NAME, "%s/%s/%s", dfile, drd->d_name, S_STAT);
< ^~
< common.c:341:3: note: ‘snprintf’ output between 7 and 1285 bytes into a destination of size 1024
< snprintf(line, MAX_PF_NAME, "%s/%s/%s", dfile, drd->d_name, S_STAT);
< ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Recent gcc versions think the code used in SREALLOC() macro might be
wrong because we use the result of a multiplication as a boolean value:
In file included from sa.h:11,
from sadc.c:37:
sadc.c: In function ‘sa_sys_init’:
sadc.c:336:55: warning: ‘*’ in boolean context, suggest ‘&&’ instead [-Wint-in-bool-context]
(size_t) act[i]->msize * (size_t) act[i]->nr_ini * (size_t) act[i]->nr2);
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~
common.h:125:16: note: in definition of macro ‘SREALLOC’
if (SIZE) { \
^~~~
Add a comparison to zero to make clear to gcc that the code is actually
fine.
sadf: SVG: Don't display "No data" over the table of contents
Sometimes, when all the selected activities have only zero values and
the skipempty option has been used with "sadf -g", the "No data" message
is displayed instead of the graphs.
In these conditions make sure that it doesn't get printed over the table
of contents entries.
Here is how to reproduce the problem on my system (I don't have any
network filesystems):
sadf: SVG: Make sure gXX-0 id exists for memory activity
A_MEMORY activity (which has two possible outputs: memory statistics and
swap statistics) might have no "gXX-0" id in SVG file if memory
statistics hadn't been selected. This would break corresponding link in
the table of contents displayed by sadf with its option "showtoc", e.g.:
A_MEMORY activity may consist only of memory *or* swap statistics, not
necessarily both of them.
Description field in struct activity is used as an entry in the table of
contents displayed by "sadf -g" (SVG output).
sadf: SVG: Don't insert a gap in SVG output if an activity is not
displayed
In "packed" mode, we have to make sure that at least one graph for
current activity has been displayed before moving to next row.
Without this patch problem could happen with a command like:
For actvities with multiple items, make sure that the graph id number
for the first item being displayed is zero ("gXX-0").
If this is not the case then the link from the table of contents will
not work. This could happen when "skipempty" option was used and the
first item in the list didn't have a graph because all its stats were
zero.
projects / sysstat / log