Sebastien GODARD [Mon, 27 Aug 2018 08:24:16 +0000 (10:24 +0200)]
sadf: Take into account new huge pages metrics
Update various sadf formats (JSON, XML, raw, etc.) to take into account
new HugePages_Rsvd and HugePages_Surp metrics.
SVG output format will be updated later.
Sebastien GODARD [Sun, 26 Aug 2018 13:54:33 +0000 (15:54 +0200)]
sar: Fix timestamp format in report output
The contents of environment variable S_TIME_FORMAT was not taken into
account when displaying a RESTART or a COMMENT message if this message
was the first line displayed in the report by sar:
$ sar
Linux 4.17.11-200.fc28.x86_64 (xxx) 08/25/2018 _x86_64_ (8 CPU)
14:10:50 LINUX RESTART (8 CPU)
02:15:47 PM CPU %user %nice %system %iowait %steal %idle
02:20:00 PM all 6.27 0.00 2.17 3.85 0.00 87.71
...
This patch fixes the problem by testing the environment variable sooner.
Sebastien GODARD [Sat, 25 Aug 2018 13:53:39 +0000 (15:53 +0200)]
sar: Fortify remap_struct() function
New metrics can be added to sar with new sysstat versions. These new
metrics correspond to new fields added to existing statistics
structures.
For compatibility reasons, sysstat assumes that these structures
(identified by their number of fields of different types: long long,
long and int) can only grow with newer sysstat versions. Yet commit c2f9d16
has fixed a longstanding bug regarding a wrong size used for structure
stats_huge. With this bugfix structure stats_huge has now a size smaller
than before. We need to update remap_struct() function to deal with
this. Else adding new fields to structure stats_huge will make sar
unable to read older binary datafiles and crash:
The idea here is to make it possible for the user to *unselect* an
activity which is collected by default by sadc.
For example, you can now choose to collect all
possible statistics except those for interrupts:
sysstat version 12.0.0 final packaging.
lsm file updated.
Typo fixed in sadf.c.
Changelog added.
Version 12.0.0 mainly focuses on SVG output from the sadf command,
with a lot of bugfixes and the addition of a new option ("showtoc")
that will enable the user to quickly navigate through graphs.
Version 12.0.0 is also the first stable version that includes the new,
compact and flexible binary format for sar.
Update spec file to version 12.0.0.
Add an entry to prevent from creating a debug package (that we don't
need). Else we have the following error when creating the RPM packages:
iostat.c: In function ‘read_sysfs_dlist_part_stat’:
iostat.c:576:39: warning: ‘%s’ directive output may be truncated writing up to 255 bytes into a region of size between 0 and 1023 [-Wformat-truncation=]
snprintf(filename, MAX_PF_NAME, "%s/%s/%s", dfile, drd->d_name, S_STAT);
^~
iostat.c:576:3: note: ‘snprintf’ output between 7 and 1285 bytes into a destination of size 1024
snprintf(filename, MAX_PF_NAME, "%s/%s/%s", dfile, drd->d_name, S_STAT);
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
sadf_misc.c: In function ‘print_dbppc_timestamp’:
sadf_misc.c:486:28: warning: ‘snprintf’ output may be truncated before the last format character [-Wformat-truncation=]
snprintf(temp, 80, "%s%s ", pre, cur_date);
^
sadf_misc.c:486:3: note: ‘snprintf’ output 2 or more bytes (assuming 81) into a destination of size 80
snprintf(temp, 80, "%s%s ", pre, cur_date);
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
sadf_misc.c:491:25: warning: ‘%s’ directive output may be truncated writing up to 4 bytes into a region of size between 1 and 80 [-Wformat-truncation=]
snprintf(pre, 80, "%s%s%s", temp, cur_time,
^~
sadf_misc.c:491:2: note: ‘snprintf’ output 1 or more bytes (assuming 84) into a destination of size 80
snprintf(pre, 80, "%s%s%s", temp, cur_time,
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
strlen(cur_date) && utc ? " UTC" : "");
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
sa_common.c: Fix pointer signedness difference warning from gcc
Remove "unsigned" attribute to avoid warning from gcc:
sa_common.c: In function ‘add_list_item’:
sa_common.c:2516:20: warning: pointer targets in assignment from ‘unsigned char *’ to ‘char *’ differ in signedness [-Wpointer-sign]
if ((e->item_name = (unsigned char *) malloc(len + 1)) == NULL) {
^
svg_stats.c: In function ‘recappend’:
svg_stats.c:549:25: warning: ‘ L’ directive output may be truncated writing 2 bytes into a region of size between 1 and 128 [-Wformat-truncation=]
snprintf(data, 128, "%s L%llu,%.2f%s", restart ? data1 : "", timetag, p_value,
^~
svg_stats.c:549:2: note: ‘snprintf’ output between 9 and 591 bytes into a destination of size 128
snprintf(data, 128, "%s L%llu,%.2f%s", restart ? data1 : "", timetag, p_value,
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
p_value != value ? data2 : "");
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
sar/sadf: Devices list management code refactoring
No longer use separate lists for devices entered on the command line
with options --dev, --iface and --fs.
Devices are now saved in the list @item_list which is part of structure
activity. This is a linked list whose number of elements is given by
@item_list_sz (also part of structure activity).
Change constant values to avoid defining arrays size as
type array[CONSTANT_VALUE + 1];
This is error-prone and not consistent with the rest of sysstat code.
Remove assignment from incompatible pointer type warnings
Remove warnings given by gcc, e.g.:
sadf_misc.c: In function ‘count_new_net_dev’:
sadf_misc.c:1033:6: warning: assignment to ‘void **’ from incompatible pointer type ‘struct interface_lst **’ [-Wincompatible-pointer-types]
p = &(e->next);
^
sadf: SVG: Fix function used to assess canvas height
Make sure that we count all the items once and only once for those
which can be registered or unregistered by the system (network
interfaces, block devices...). This is needed to assess SVG canvas
height since each of them will have its own graph.
Explicitly set "item_name" variable before using it to avoid some
warnings by the compiler, e.g.:
svg_stats.c: In function ‘svg_print_net_dev_stats’:
svg_stats.c:2409:5: warning: ‘item_name’ may be used uninitialized in this function [-Wmaybe-uninitialized]
strncpy(item_name, sndc->interface, CHUNKSIZE);
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
sadf: SVG: Fix core dumped on buffers reallocation
When buffers used to save graphs data needed to be extended to
accomodate for new devices, the new size was wrongly calculated
and a core dump was then created.
Check return code of some snprintf() functions to quiet gcc about
possible truncated output, e.g.:
12,18d11
< common.c: In function ‘get_dev_part_nr’:
< common.c:341:35: warning: ‘%s’ directive output may be truncated writing up to 255 bytes into a region of size between 0 and 1023 [-Wformat-truncation=]
< snprintf(line, MAX_PF_NAME, "%s/%s/%s", dfile, drd->d_name, S_STAT);
< ^~
< common.c:341:3: note: ‘snprintf’ output between 7 and 1285 bytes into a destination of size 1024
< snprintf(line, MAX_PF_NAME, "%s/%s/%s", dfile, drd->d_name, S_STAT);
< ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Recent gcc versions think the code used in SREALLOC() macro might be
wrong because we use the result of a multiplication as a boolean value:
In file included from sa.h:11,
from sadc.c:37:
sadc.c: In function ‘sa_sys_init’:
sadc.c:336:55: warning: ‘*’ in boolean context, suggest ‘&&’ instead [-Wint-in-bool-context]
(size_t) act[i]->msize * (size_t) act[i]->nr_ini * (size_t) act[i]->nr2);
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~
common.h:125:16: note: in definition of macro ‘SREALLOC’
if (SIZE) { \
^~~~
Add a comparison to zero to make clear to gcc that the code is actually
fine.
sadf: SVG: Don't display "No data" over the table of contents
Sometimes, when all the selected activities have only zero values and
the skipempty option has been used with "sadf -g", the "No data" message
is displayed instead of the graphs.
In these conditions make sure that it doesn't get printed over the table
of contents entries.
Here is how to reproduce the problem on my system (I don't have any
network filesystems):
sadf: SVG: Make sure gXX-0 id exists for memory activity
A_MEMORY activity (which has two possible outputs: memory statistics and
swap statistics) might have no "gXX-0" id in SVG file if memory
statistics hadn't been selected. This would break corresponding link in
the table of contents displayed by sadf with its option "showtoc", e.g.:
A_MEMORY activity may consist only of memory *or* swap statistics, not
necessarily both of them.
Description field in struct activity is used as an entry in the table of
contents displayed by "sadf -g" (SVG output).
sadf: SVG: Don't insert a gap in SVG output if an activity is not
displayed
In "packed" mode, we have to make sure that at least one graph for
current activity has been displayed before moving to next row.
Without this patch problem could happen with a command like:
For actvities with multiple items, make sure that the graph id number
for the first item being displayed is zero ("gXX-0").
If this is not the case then the link from the table of contents will
not work. This could happen when "skipempty" option was used and the
first item in the list didn't have a graph because all its stats were
zero.
sadf: SVG: Don't insert id tag if activity not displayed
When using both "showtoc" and "skipempty" options, e.g.:
$ sadf -g -O showtoc,skipempty -- -A > output.svg
clicking on the link pointing at an activity with no graphs displayed
(because all its statistics were zero) would move to the first
following graph... which is not the graph corresponding to the link!
This was because even if the graphs don't get displayed, sadf inserted
the id tag in the SVG file, e.g.:
Sebastien GODARD [Sat, 16 Jun 2018 15:52:06 +0000 (17:52 +0200)]
sadf: SVG: Add new option "showtoc"
Add new option "showtoc" aimed at displaying, at the beginning of the
SVG file, the list of activities for which there are graphs ("table of
contents"). Each entry consists of a link pointing at the corresponding
activity in the SVG file, enabling the user to quickly navigate through
graphs.
Sebastien GODARD [Fri, 15 Jun 2018 14:25:10 +0000 (16:25 +0200)]
sadf: SVG: Don't take into account activites with no graphs
In "packed" mode, activities which don't have a graph (e.g. USB devices)
should not be taken into account when calculating SVG canvas height.
Without this patch, a command like:
$ sadf -g -O packed -- -A > output.svg
created a SVG file with some unused white space at its end.
Sebastien GODARD [Fri, 15 Jun 2018 14:11:36 +0000 (16:11 +0200)]
sadf: SVG: Display graphs for swap utilization in packed mode
When both memory and swap utilization were selected, graphs for swap
utilization weren't displayed by sadf if option "packed" had been
entered, e.g.:
$ sadf -g -O packed -- -rS > output.svg
This was because SVG canvas height was not big enough. Indeed,
sadf assumed that each activity only needed one row in "packed" mode,
which is wrong for A_MEMORY activity (which requires two rows: One for
memory utilization and one for swap utilization).
This patch fixes the problem.
sysstat version 11.7.4 final packaging.
lsm and spec files updated.
Changelog added.
This version adds several new options to control:
o the number of decimal places used by sysstat commands (sar,
cifsiostat, iostat, mpstat and pidstat),
o the network interfaces, block devices and filesystems for which
statistics shall be displayed by sar or sadf.
Note that this version also redefines the metric kbmemused (amount of
used memory displayed by sar) to be consistent with the output of
commands such as top(1) and free(1). See sar's manual page.
Sebastien GODARD [Sat, 26 May 2018 15:47:18 +0000 (17:47 +0200)]
sadf: SVG: Fix coredump problem when --dev option used
Function svg_print_disk_stats() should allocate only the necessary
number of slots when a list of devices has been entered on the
command line with option "--dev=".
Sebastien GODARD [Sun, 20 May 2018 14:54:25 +0000 (16:54 +0200)]
sadf: Allow user to select filesystems to display
Add new option "--fs=<fs_list>". It has the same meaning as for
sar. This may be especially useful for SVG output for which it is
difficult to "grep" a particular device. Using this new option makes
it possible for the user to draw graphs only for the desired
block device(s) and not for all of them.
Sebastien GODARD [Wed, 16 May 2018 12:50:29 +0000 (14:50 +0200)]
sadf: Allow user to select block devices to display
Add new option "--dev=<dev_list>". It has the same meaning as for
sar. This may be especially useful for SVG output for which it is
difficult to "grep" a particular device. Using this new option makes
it possible for the user to draw graphs only for the desired
block device(s) and not for all of them.
Sebastien GODARD [Wed, 16 May 2018 12:41:55 +0000 (14:41 +0200)]
sar: Allow user to select block devices to display
Add new option "--dev=<dev_list>" to allow the user to specify which
block devices to display. This option is to be used with option -d.
Sample output:
$ sar -dp
Linux 4.4.14-200.fc22.x86_64 (home) 05/12/18 _x86_64_ (8 CPU)
Sebastien GODARD [Sun, 13 May 2018 13:58:33 +0000 (15:58 +0200)]
sadf: Allow user to select interfaces to display
Add new option "--iface=<iface_list>". It has the same meaning as for
sar. This may be especially useful for SVG output for which it is
difficult to "grep" a particular device. Using this new option makes
it possible for the user to draw graphs only for the desired
interface(s) and not for all of them.
Sebastien GODARD [Sat, 12 May 2018 08:31:50 +0000 (10:31 +0200)]
sar: Allow user to select interfaces to display
Add new option "--iface=<iface_list>" to allow the user to specify which
network interfaces to display. This option is to be used with options
"-n DEV" and "-n EDEV".
Sample output:
$ sar -n DEV
Linux 4.4.14-200.fc22.x86_64 (home) 05/12/18 _x86_64_ (8 CPU)
Sebastien GODARD [Thu, 10 May 2018 12:23:22 +0000 (14:23 +0200)]
Allow user to select number of decimal places
Add new option: "--dec={0|1|2}" which can be used to specify the number
of decimal places.
This option can be used with sar, mpstat, iostat, pidstat and
cifsiostat.
Sample output:
$ sar -b
Linux 4.4.14-200.fc22.x86_64 (home) 05/10/18 _x86_64_ (8 CPU)
sysstat version 11.7.3 final packaging.
lsm and spec files updated.
Changelog added.
This version has improved support for offline/online CPU (sar/sadf and
mpstat). It also adds a new option to sar (option "-z") to omit output
for any devices for which there was no activity during the sample
period (this option already existed for iostat).
Last it contains several important bug fixes, among them:
o A possible crash related to the new binary format used by sar,
which could happen when buffers needed to be reallocated to accomodate
more devices added to the system,
o The %CPU value displayed by pidstat for processes using more than
100% of a core.
Reassure Coverity by making sure that the total number of jiffies spent
by all processors is not zero.
This would be possible only if all processors were tickless.
Reassure Coverity by making sure that the total number of jiffies spent
by all processors/nodes is not zero.
This would be possible only if all processors were tickless.
Sebastien GODARD [Sat, 31 Mar 2018 08:15:04 +0000 (10:15 +0200)]
pidstat: Report CPU usage for a process as the sum of all threads
Commit 52977c4 introduced a new macro (SP_VALUE_100()) to limit CPU
usage value displayed by pidstat to 100%. This was intended to be a
workaround for some corner cases where inaccurate statistics could be
provided by the kernel.
Yet this macro also limits the value to 100% even when a process uses
more than 100% of a core, e.g. when it starts several threads loading
the system. This patch reverts commit 52977c4 and reports CPU usage
for a process as the sum of CPU usage for all its threads.
Sebastien GODARD [Sat, 24 Mar 2018 09:33:20 +0000 (10:33 +0100)]
sar: Softnet stats: Improve support for offline/online CPUs
When a CPU goes offline, the corresponding line in the
/proc/net/softnet_stat file disappears. The problem is that there is no
immediate solution to know which line goes with which CPU.
To fix this, we now use the /proc/stat file to know which CPU are online
and which ones are not.
Moreover, when a CPU comes back online, its counters get their original
values, which makes sar think they have just jumped from 0:
This is what sar displayed before (in this example, CPU 5 goes offline):
Sebastien GODARD [Fri, 23 Mar 2018 11:25:44 +0000 (12:25 +0100)]
mpstat: Remove option "-P ON"
mpstat now doesn't display offline CPUs. So option "-P ON" used to
tell mpstat to display only online CPUs is no longer needed.
This is the same behavior as sar.
Sebastien GODARD [Fri, 23 Mar 2018 11:14:49 +0000 (12:14 +0100)]
mpstat: Compute stats for node "all" as the sum of individual CPU
Don't use global CPU stats from /proc/stat file for node "all".
Compute stats for node "all" as the sum of individual ones.
Also better handle CPUs that go offline or back online.
projects / sysstat / log