Sebastien GODARD [Sun, 20 May 2018 14:54:25 +0000 (16:54 +0200)]
sadf: Allow user to select filesystems to display
Add new option "--fs=<fs_list>". It has the same meaning as for
sar. This may be especially useful for SVG output for which it is
difficult to "grep" a particular device. Using this new option makes
it possible for the user to draw graphs only for the desired
block device(s) and not for all of them.
Sebastien GODARD [Wed, 16 May 2018 12:50:29 +0000 (14:50 +0200)]
sadf: Allow user to select block devices to display
Add new option "--dev=<dev_list>". It has the same meaning as for
sar. This may be especially useful for SVG output for which it is
difficult to "grep" a particular device. Using this new option makes
it possible for the user to draw graphs only for the desired
block device(s) and not for all of them.
Sebastien GODARD [Wed, 16 May 2018 12:41:55 +0000 (14:41 +0200)]
sar: Allow user to select block devices to display
Add new option "--dev=<dev_list>" to allow the user to specify which
block devices to display. This option is to be used with option -d.
Sample output:
$ sar -dp
Linux 4.4.14-200.fc22.x86_64 (home) 05/12/18 _x86_64_ (8 CPU)
Sebastien GODARD [Sun, 13 May 2018 13:58:33 +0000 (15:58 +0200)]
sadf: Allow user to select interfaces to display
Add new option "--iface=<iface_list>". It has the same meaning as for
sar. This may be especially useful for SVG output for which it is
difficult to "grep" a particular device. Using this new option makes
it possible for the user to draw graphs only for the desired
interface(s) and not for all of them.
Sebastien GODARD [Sat, 12 May 2018 08:31:50 +0000 (10:31 +0200)]
sar: Allow user to select interfaces to display
Add new option "--iface=<iface_list>" to allow the user to specify which
network interfaces to display. This option is to be used with options
"-n DEV" and "-n EDEV".
Sample output:
$ sar -n DEV
Linux 4.4.14-200.fc22.x86_64 (home) 05/12/18 _x86_64_ (8 CPU)
Sebastien GODARD [Thu, 10 May 2018 12:23:22 +0000 (14:23 +0200)]
Allow user to select number of decimal places
Add new option: "--dec={0|1|2}" which can be used to specify the number
of decimal places.
This option can be used with sar, mpstat, iostat, pidstat and
cifsiostat.
Sample output:
$ sar -b
Linux 4.4.14-200.fc22.x86_64 (home) 05/10/18 _x86_64_ (8 CPU)
sysstat version 11.7.3 final packaging.
lsm and spec files updated.
Changelog added.
This version has improved support for offline/online CPU (sar/sadf and
mpstat). It also adds a new option to sar (option "-z") to omit output
for any devices for which there was no activity during the sample
period (this option already existed for iostat).
Last it contains several important bug fixes, among them:
o A possible crash related to the new binary format used by sar,
which could happen when buffers needed to be reallocated to accomodate
more devices added to the system,
o The %CPU value displayed by pidstat for processes using more than
100% of a core.
Reassure Coverity by making sure that the total number of jiffies spent
by all processors is not zero.
This would be possible only if all processors were tickless.
Reassure Coverity by making sure that the total number of jiffies spent
by all processors/nodes is not zero.
This would be possible only if all processors were tickless.
Sebastien GODARD [Sat, 31 Mar 2018 08:15:04 +0000 (10:15 +0200)]
pidstat: Report CPU usage for a process as the sum of all threads
Commit 52977c4 introduced a new macro (SP_VALUE_100()) to limit CPU
usage value displayed by pidstat to 100%. This was intended to be a
workaround for some corner cases where inaccurate statistics could be
provided by the kernel.
Yet this macro also limits the value to 100% even when a process uses
more than 100% of a core, e.g. when it starts several threads loading
the system. This patch reverts commit 52977c4 and reports CPU usage
for a process as the sum of CPU usage for all its threads.
Sebastien GODARD [Sat, 24 Mar 2018 09:33:20 +0000 (10:33 +0100)]
sar: Softnet stats: Improve support for offline/online CPUs
When a CPU goes offline, the corresponding line in the
/proc/net/softnet_stat file disappears. The problem is that there is no
immediate solution to know which line goes with which CPU.
To fix this, we now use the /proc/stat file to know which CPU are online
and which ones are not.
Moreover, when a CPU comes back online, its counters get their original
values, which makes sar think they have just jumped from 0:
This is what sar displayed before (in this example, CPU 5 goes offline):
Sebastien GODARD [Fri, 23 Mar 2018 11:25:44 +0000 (12:25 +0100)]
mpstat: Remove option "-P ON"
mpstat now doesn't display offline CPUs. So option "-P ON" used to
tell mpstat to display only online CPUs is no longer needed.
This is the same behavior as sar.
Sebastien GODARD [Fri, 23 Mar 2018 11:14:49 +0000 (12:14 +0100)]
mpstat: Compute stats for node "all" as the sum of individual CPU
Don't use global CPU stats from /proc/stat file for node "all".
Compute stats for node "all" as the sum of individual ones.
Also better handle CPUs that go offline or back online.
Sebastien GODARD [Fri, 16 Mar 2018 09:58:37 +0000 (10:58 +0100)]
sar: Compute global CPU stats as the sum of individual ones
sar used to get statistics for CPU "all" from the first line of the
/proc/stat file giving global CPU utilization.
There are several problems with this:
1) With recent kernels (problem detected on a 4.4.14 kernel), the number
of jiffies spent in idle and iowait modes given by this file for global
CPU utilization goes crazy when a CPU is set offline or comes back
online. These counters may even not be monotonic, resulting in wrong
results being displayed by sar.
E.g.:
2) The updating of the /proc/stat global and individual values is not
done atomically. As a result there can be skew between the global and
individual values reported by sar.
E.g.:
sar: Better assess size of buffers that need to be reallocated
When a buffer needs to be reallocated, doubling its size may not be
enough to contain all the additional items.
Assess the needed size based on a value giving the minimum number of
items the buffer should be able to contain.
sar: Test for zero value when reallocating all the buffers
When sar reads the contents of a file and meets a LINUX RESTART, it
may have to reallocate the buffers used to save CPU statistics to match
the new number of CPU. If the number of CPU has increased it doubles the
size of its buffers.
There is a problem though if CPU activity was not included in file (or
if its format was unknown to current sysstat version). In this case the
buffer size was zero. So test this before reallocating buffers.
Below is a sample output before the patch was applied.
The file contains only A_PCSW activity then a LINUX RESTART message
(which cannot be displayed):
$ sar -f data0 -w
Linux 4.4.14-200.fc22.x86_64 (test.home) 03/04/18 _x86_64_
(8 CPU)
Sebastien GODARD [Sun, 18 Feb 2018 15:18:58 +0000 (16:18 +0100)]
sar: Add new -z option
Add new option to sar (-z). This option tells sar to omit output for any
devices for which there was no activity during the sample period.
This option applies for network interfaces, interrupts, block devices,
serial lines, filesystems, CPU (in softnet statistics).
Note: This option already existed for iostat.
Sebastien GODARD [Mon, 12 Feb 2018 14:54:01 +0000 (15:54 +0100)]
sysstat-11.7.2
sysstat version 11.7.2 final packaging.
lsm and spec files updated.
Changelog added.
Year updated in (C) message.
After one additional month of hard work, sysstat version 11.7.2 seems to
be quite stable on my machine. It should now be possible to use it
safely on production systems.
BTW I told you that sar's data files new binary format would be at least
25% smaller than with previous versions. I could have even said 30% to
45% smaller!
Version 11.7.2 also includes a rewritten function that will enable you
to convert your old data files (from versions 9.1.6 and later) to the
up-to-date format (11.7.2).
Please upload, test and tell me if anything goes wrong.
Sebastien GODARD [Sun, 11 Feb 2018 10:43:20 +0000 (11:43 +0100)]
sar/sadf: Add checks on file's header size read from file
sa_open_read_magic() reads file's magic structure and in particular the
size of the header structure to come.
header_size field in file_magic structure exists only with versions
10.3.1 and later. So checking bounds for header_size is done only for
those versions, based on the values of sysstat_version and
sysstat_patchlevel.
With a corrupted datafile (i.e. a file having the right FORMAT_MAGIC
value but values corresponding to older sysstat versions in
sysstat_version and sysstat_patchlevel), the test is not done.
So do it again in check_file_actlst() function.
Offline CPU should be ignored only when graphs are drawn and if they
have been offline on the whole period.
When graphs are created, offline CPU are useful: Their frequency value
(0) is used to make the graph go through 0.
Don't display a graph for CPUs which have been offline on the whole
period.
A CPU whose state alternates between offline and online still gets a
graph.
For very small files with just a few samples taken at a few seconds apart
(e.g. 15 samples taken at 1 second interval) the background grid was not
drawn on the whole width of the graph.
Fix this.
SVG: Make sure that "No data" is displayed when no graphs are created
If no graphs are detected (e.g. the file contains only a LINUX RESTART
message) then SVG file is empty and an error is displayed by the browser.
Make sure that "No data" is displayed in the browser window instead.
Add several tests for continuous integration.
In particular some tests, which previously failed because of endianness
mismatch (see #145), are restored (sar should now be able to process
both endian formats).
@nr_ini shouldn't be modified by upgrade_stats_serial() function because
it is used to kwnow how many structures have to be read from the
original file.
sadf: Properly convert structures with compatible changes
Some activity structures may have gained additional fields while keeping
their original magic number. This is because these changes were seen as
compatible for sar.
Anyway these structures have to be handled in a specific way when they
are converted since these additional fields didn't exist for all the
versions since 9.1.6.
Update sadf function used to convert an old datafile (from version 9.1.6
and later) to the up-to-date format (11.7.2). This function corresponds
to sadf option -c ("sadf -c old_datafile > new_datafile").
It should work on both little endian and big endian data files.
The original endianness of the file is preserved.
Sebastien GODARD [Sat, 27 Jan 2018 06:47:55 +0000 (07:47 +0100)]
Save HZ as an unsigned long integer
Number of clock ticks per second is given as a long integer by sysconf()
function. It is also saved as an unsigned long integer in sar's data
files (sa_hz).
Sebastien GODARD [Mon, 22 Jan 2018 16:06:21 +0000 (17:06 +0100)]
iostat: Refresh device list properly
When running iostat to monitor disk activity,
disconnecting a USB drive from the system then reconnecting another one
didn't make the new one appear on the list.
This patch fixes the problem.
Reported-by: Robert Hoffmann <robert@noreply.servermasters.com> Signed-off-by: Sebastien GODARD <sysstat@users.noreply.github.com>
Sebastien GODARD [Sat, 20 Jan 2018 19:54:41 +0000 (20:54 +0100)]
RAW: Check return code from check_*_reg functions
Distinguish between new devices and devices that have been unregistered
then registered again (this indication is displayed by "sadf -r" with
showhints option).
Sebastien GODARD [Fri, 19 Jan 2018 15:52:15 +0000 (16:52 +0100)]
sar: Don't read statistics twice when displaying average since system
startup
Entering something like "sar 0" to display statistics (here CPU) since
system startup made sadc read the statistics twice. One reading is
enough.
Before this patch, sar called sadc with interval=1 and count=-1.
Now sadc is called with interval=1 and count=1.
Sebastien GODARD [Fri, 19 Jan 2018 15:10:14 +0000 (16:10 +0100)]
sar: Display all items for USB and filesystems activities
The summary displayed by sar for USB and filesystems activities didn't
include all the items. In pacticular, filesystems or USB devices that
had been unmounted were not displayed in the summary ending the report.
This patch fixes the problem.
Sebastien GODARD [Wed, 17 Jan 2018 20:26:56 +0000 (21:26 +0100)]
sadc: Select activities by name
This patch enables the user to select exactly which activities will be
collected by sadc and saved into the binary data file.
Selected activities are entered following sadc's option -S using their
formal report name (these names are displayed by "sar --help").
Sebastien GODARD [Wed, 17 Jan 2018 19:49:49 +0000 (20:49 +0100)]
iostat: Display device name at the end of line when option -h used
iostat's option -h was intended to make the output more easily readable
by a human, especially when some devices had a long name that would make
the whole output shift around. In fact it added a '\n' to split the line
and the result was not particularly easier to read.
This patch moves the device name at the end of the line when option -h
is used without inserting more lines.
Option -h still sets option --human to automatically select the right
unit to display.
projects / sysstat / log