Sebastien GODARD [Sat, 27 Jan 2018 06:47:55 +0000 (07:47 +0100)]
Save HZ as an unsigned long integer
Number of clock ticks per second is given as a long integer by sysconf()
function. It is also saved as an unsigned long integer in sar's data
files (sa_hz).
Sebastien GODARD [Mon, 22 Jan 2018 16:06:21 +0000 (17:06 +0100)]
iostat: Refresh device list properly
When running iostat to monitor disk activity,
disconnecting a USB drive from the system then reconnecting another one
didn't make the new one appear on the list.
This patch fixes the problem.
Reported-by: Robert Hoffmann <robert@noreply.servermasters.com> Signed-off-by: Sebastien GODARD <sysstat@users.noreply.github.com>
Sebastien GODARD [Sat, 20 Jan 2018 19:54:41 +0000 (20:54 +0100)]
RAW: Check return code from check_*_reg functions
Distinguish between new devices and devices that have been unregistered
then registered again (this indication is displayed by "sadf -r" with
showhints option).
Sebastien GODARD [Fri, 19 Jan 2018 15:52:15 +0000 (16:52 +0100)]
sar: Don't read statistics twice when displaying average since system
startup
Entering something like "sar 0" to display statistics (here CPU) since
system startup made sadc read the statistics twice. One reading is
enough.
Before this patch, sar called sadc with interval=1 and count=-1.
Now sadc is called with interval=1 and count=1.
Sebastien GODARD [Fri, 19 Jan 2018 15:10:14 +0000 (16:10 +0100)]
sar: Display all items for USB and filesystems activities
The summary displayed by sar for USB and filesystems activities didn't
include all the items. In pacticular, filesystems or USB devices that
had been unmounted were not displayed in the summary ending the report.
This patch fixes the problem.
Sebastien GODARD [Wed, 17 Jan 2018 20:26:56 +0000 (21:26 +0100)]
sadc: Select activities by name
This patch enables the user to select exactly which activities will be
collected by sadc and saved into the binary data file.
Selected activities are entered following sadc's option -S using their
formal report name (these names are displayed by "sar --help").
Sebastien GODARD [Wed, 17 Jan 2018 19:49:49 +0000 (20:49 +0100)]
iostat: Display device name at the end of line when option -h used
iostat's option -h was intended to make the output more easily readable
by a human, especially when some devices had a long name that would make
the whole output shift around. In fact it added a '\n' to split the line
and the result was not particularly easier to read.
This patch moves the device name at the end of the line when option -h
is used without inserting more lines.
Option -h still sets option --human to automatically select the right
unit to display.
Sebastien GODARD [Mon, 15 Jan 2018 16:38:27 +0000 (17:38 +0100)]
sadf: RAW: Display activity name when showhints option used
"sadf -r -O showhints..." now displays the name for each activity.
Also move the code displaying the number of structures used and allocated
from raw_stats.c to sadf.c.
Sebastien GODARD [Fri, 12 Jan 2018 10:56:29 +0000 (11:56 +0100)]
sysstat-11.7.1
sysstat version 11.7.1 final packaging.
lsm and spec files updated.
Changelog added.
NOTE: Stable versions (11.2.14 [EOL], 11.4.8 and 11.6.2) also exist
and will be available for download from my web site:
http://pagesperso-orange.fr/sebastien.godard/
The stable versions include only the bug fixes added in sysstat 11.7.1
but not the new features.
Version 11.7.1 includes major changes concerning sar's binary data
file format and how sar works. Among them:
* A binary data file should take much less space on disk thanks
to new optimizations on how values are stowed in memory. The space saved
is estimated at least at 25% compared to previous format.
* Structures are no longer statically allocated, meaning that
the system can now register as many new devices (disks, network
interfaces, etc.) as needed and you will find all of them saved in your
saXX data file (provided that you have selected the corresponding
activities to collect). Previous version could lead to some devices
being ignored and going unnoticed if no free structures were left.
* Sar (and sadf) will now be able to read a binary data file
whatever its endianness is: Both big-endian and little-endian files can
be read by the same sar or sadf executable.
* More flexibility has been added to sar's binary data file
format. I cannot promise that the format won't change again in the
future but it should now be possible to add new metrics or activities
without making the format unreadable by older sar versions (starting
with version 11.7.1). Older versions will be able to read newer formats
but will display only the metrics they currently know.
Version 11.7.1 is a development version. As such, it is still NOT ready
for use in production. Yet I would really like to have it tested by the
maximum number of users and get feedback! :-)
NOTE: The function provided by sadf (option -c) to convert an old
data file to the up-to-date format has been temporarily inhibited. This
function will be working again in next sysstat version (11.7.2).
Sebastien GODARD [Fri, 12 Jan 2018 09:15:40 +0000 (10:15 +0100)]
sar: Add new option -h
Add option -h to make sar's output easier to read by a human.
This option tells sar to display {devices, network interfaces} names at
the end of the line to make sure the output won't shift. This option
also enables implicitly --human and -p (pretty-print).
SVG: Put queue and load average metrics back in order
Commit 4a15362 changed the order of metrics in stats_queue structure.
This new order has also to be taken into account by corresdponding
rendering function in svg_stats.c.
File activity.c should depend on rd_stats.h and rd_sensors.h
for sadf and sar in addition to sadc.
Structures size (STATS_*_SIZE) is defined in these include files and
used in activity.c for all commands.
sadf: Update functions to display unsigned unsigned long values
Some metrics's type is now unsigned long long instead of unsigned long.
So update functions used by sadf accordingly. Other metrics whose type
remains unsigned long can still be displayed.
sar: Update functions to be consistent with new stats structures
Update functions used to read or write statistics to remain consistent
with structures' new types.
Some work has still to be done in rndr_stats.c and svg_stats.c
Sebastien GODARD [Sun, 31 Dec 2017 14:35:00 +0000 (15:35 +0100)]
sar: Fix code trying to guess when a header line needs to be displayed
sar should repeat the header line for each activity at regular intervals
when displaying stats from file.
The code previously used needs to be updated as the number of items may
now vary with each sample.
Sebastien GODARD [Sun, 31 Dec 2017 08:32:33 +0000 (09:32 +0100)]
sar: Print items' name on the right of the report
Some items' name (devices, network interfaces) may sometimes be long
enough to create a shift in the statistics displayed by sar which are no
longer in their original column.
So display them at the end of the line on the right.
Sebastien GODARD [Tue, 26 Dec 2017 15:56:15 +0000 (16:56 +0100)]
sadf: Don't test for activities available in file if only the header
needs to be displayed
When "sadf -H" is used, there is no need to check that activities exist
in file. This is all the more true now as even CPU statistics (A_CPU
activity) don't necessarily exist in file.
E.g.:
sadc -S 0,36 datafile 1 2
Then "sadf -H datafile" should not display error message "Requested
activities not available in file".
Sebastien GODARD [Tue, 26 Dec 2017 15:53:27 +0000 (16:53 +0100)]
sadf: RAW: Display number of allocated slots for current activity
Enhance output displayed by sadf in raw mode (sadf -r) again: If
showhints option is used, display number of allocated slots for items
for current activity sample.
Mainly useful for debugging purpose, as this value is not saved in file.
Sebastien GODARD [Tue, 26 Dec 2017 15:27:23 +0000 (16:27 +0100)]
sadc: Fix segfault happening when buffer needs to be reallocated
reallocate_buffer() in sa_wrap.c assigns a new address to a->_buf0
buffer. This new address needs also to be set to the variable pointing
to the statistics structure in parent function.
Sebastien GODARD [Sun, 24 Dec 2017 15:14:01 +0000 (16:14 +0100)]
sadf: ppc: Remove duplicate % sign
In some cases, the % sign was printed twice by sadf -p, e.g.:
$ sadf -p test_file
test.home 1 2017-12-24 14:52:39 UTC all %%user 0.25
test.home 1 2017-12-24 14:52:39 UTC all %%nice 0.00
test.home 1 2017-12-24 14:52:39 UTC all %%system 0.25
test.home 1 2017-12-24 14:52:39 UTC all %%iowait 0.00
test.home 1 2017-12-24 14:52:39 UTC all %%steal 0.00
test.home 1 2017-12-24 14:52:39 UTC all %%idle 99.50
Expected result:
$ sadf -p test_file
test.home 1 2017-12-24 14:52:39 UTC all %user 0.25
test.home 1 2017-12-24 14:52:39 UTC all %nice 0.00
test.home 1 2017-12-24 14:52:39 UTC all %system 0.25
test.home 1 2017-12-24 14:52:39 UTC all %iowait 0.00
test.home 1 2017-12-24 14:52:39 UTC all %steal 0.00
test.home 1 2017-12-24 14:52:39 UTC all %idle 99.50
Problem was triggered by commit 5431b6c.
This patch fixes it.
Sebastien GODARD [Sat, 23 Dec 2017 16:47:19 +0000 (17:47 +0100)]
SVG: Display time as xx:00 instead of xxH when oneday option used
Timestamps were displayed as 0H 2H ... 22H for graphs created with a
command like "sadf -g -O oneday...".
They are now displayed as 0:00 2:00 ... 22:00
Sebastien GODARD [Sat, 23 Dec 2017 16:04:21 +0000 (17:04 +0100)]
A_SERIAL: Don't increment serial line number before saving
The line number, as saved in the stats_serial structure, was used to
determine if the structure was empty or not: A value of 0 used to mean
that the structure was actually empty. As a consequence line #0 was
saved with the value 1, line #1 was saved with the value 2, etc.
This is no longer useful as structures are now dynamically allocated and
we know exactly how many of them we have.
So save now the real line number.
Sebastien GODARD [Fri, 22 Dec 2017 17:07:36 +0000 (18:07 +0100)]
sa_conv.c: Inhibit conversion function
sadf's option -c, used to convert an old datafile to up-to-date format,
is temporarily inhibited.
sysstat 11.7.1 datafiles have a new binary format and this function
needs to be rewritten. Yet I don't want to delay 11.7.1 release too
much.
This function will be restored in next sysstat release.
Sebastien GODARD [Fri, 22 Dec 2017 07:49:56 +0000 (08:49 +0100)]
sadc/sar: New format (part 7): Allocate structures dynamically
The goal of this patch is to make sar and sadc able to deal with any
number of devices which may be registered by the system, even if this
number is much higher than the intial one when the datafile was created.
Sebastien GODARD [Sat, 16 Dec 2017 09:41:10 +0000 (10:41 +0100)]
Use ULLONG_MAX/2 to check if network device counters have overflown
Counters's type for stats_net_dev structure is "unsigned long long"
(it used to be "unsigned long" before). So use ULLONG_MAX instead of
ULONG_MAX to try to guess if counters have overflown.
Sebastien GODARD [Sat, 16 Dec 2017 08:29:21 +0000 (09:29 +0100)]
Workaround for offline CPU coming back online
I noticed that whith recent kernels, when a CPU comes back online, some
fields (user, nice, system) restart from their previous value, whereas
others (idle, iowait) restart from zero. For values restarting from
zero, we need to set their previous value to zero to avoid getting an
interval value < 0.
We try also to guess if the counters have overflown: If the previous
value was greater than ULLONG_MAX & 0x80000, then the CPU was probably
not set offline but the counter overflew (the value 0x80000 is an
arbitrary limit that I decided to use).
In the example above, idle value for CPU#3 is 95477 before CPU is set
offline, then 677 when it has come back online, and iowait value goes
from 25 to 0.
On the other hand, user value for example goes from 1208 to 1213...
(We can also notice that values given for global CPU usage on the first
line have a really strange behavior...)
Sebastien GODARD [Sun, 10 Dec 2017 08:35:16 +0000 (09:35 +0100)]
sar/sadc: Make use of __nr_t type
__nr_t type (which defaults to int) is used to define a number of items
(devices, network interfaces, etc.) So use it instead of int in various
places.
sar: No need to init previous iteration structure when a device has been
registered again
Simplify code a bit: Don't reset previous iteration's statistics when a
known device has been registered again, since this is already done when
stats are displayed.
Sebastien GODARD [Fri, 17 Nov 2017 13:37:09 +0000 (14:37 +0100)]
sar/sadc: Get rid of volatile activities
So called "volatile activities" were activities for which the number of
items could vary and corresponding structures were reallocated
accordingly.
In fact only A_CPU activity (CPU statistics) was volatile, to take into
account a possible restart of a VM machine including more CPU than when
the datafile was created.
The goal is now to allocate dynamically all the structures used by
activities, so we no longer have to distinguish between activities which
are volatile or not.
Sebastien GODARD [Fri, 17 Nov 2017 08:33:50 +0000 (09:33 +0100)]
sar/sadc: Don't assume CPU statistics are always recorded in file
CPU statistics were always collected and saved when a saDD datafile was
created. Those statistics were in particular used to compute time
interval in jiffies, and to give the number of processors of the machine
(which was displayed in the report header).
We no longer use CPU jiffies to compute time interval (which is now
expressed in 1/100th of a second and calculated using the /proc/uptime
file). So CPU statistics are no longer compulsory: They may not be
collected and available in file.
Since we always need a way to know the number of CPU of the machine
where the data file has been created, we add a new field (sa_cpu_nr) in
the file_header structure.
Sebastien GODARD [Sun, 12 Nov 2017 09:14:20 +0000 (10:14 +0100)]
sar/sadc: Increase devices upper limits
We found that previous upper limits values were too low sometimes (e.g.
512 for the maximum number of network interfaces). So increase them.
(Upper limits exist to make sure we don't allocate unreasonably high
amount of memory to save devices statistics, in particular when we read
data from a -possibly- corrupted datafile).
Sebastien GODARD [Sun, 12 Nov 2017 08:17:55 +0000 (09:17 +0100)]
sar/sadc: Remove preallocation constants
Next sysstat version will dynamically allocate records for new devices
when they are registered by the system. So we no longer need to
preallocate those records.
This patch removes the constants and files that were used for this
purpose.
Sebastien GODARD [Fri, 13 Oct 2017 12:20:30 +0000 (14:20 +0200)]
Don't compute global system uptime when reading CPU stats
Don't compute "global system uptime" (i.e. total number of jiffies spent
by all processors) in read_stat_cpu() function so that we don't have to
pass the result from one function to another.
Global system uptime is used only when displaying CPU statistics, so
this is now done locally in the functions displaying CPU statistics.
Also as a consequence, field uptime has been removed from the
record_header structure associated with every sample statistics in
sar/sadc.
Sebastien GODARD [Fri, 13 Oct 2017 07:29:19 +0000 (09:29 +0200)]
No longer use /proc/stat to calculate system uptime
sysstat commands used the /proc/stat file to compute system uptime based
on the number of jiffies spent by CPU#0 (which cannot be set offline).
Now only use the /proc/uptime file which we assume always exists on a
Linux machine.
activity structure's f_count2() member (used to count the number of
sub-items) was tested in sadf.c but never initialized (its value is
initialized only when sadc.c is compiled, not sadf.c). So use a specific
flag to indicate if activity has sub-items instead of testing f_count2()
value.
Add extra checks on *_types_nr[] values read from a file to cope with
possible corrupted files.
*_types_nr[] values give the number of fields of each type (long long,
long, int) composing a structure.
sadc/sar: New format (part 4): Update file_magic structure
Use integer type for upgraded field instead of unsigned char in
file_magic structure. Thus we are no longer limited to 15 for the
patchlevel and sublevel version numbers.
sadc/sar: New format (part 3): Use 64-bit time fields
Now use unsigned long long type for time fields (instead of
unsigned long, which are only 32-bit wide on 32-bit systems).
This is to avoid any future problems (like year 2038 integer
overflow).
Also code the year number as an integer instead of a char in datafile's
header.
This function will be used to map two structures from two
different sysstat versions.
It will thus be possible to add new fields, e.g., in structures
containing statistics and still be possible for another sysstat version
to read its contents properly.
This patch aims at making sar and sadf able to read and process files
created on machines with a different endian type. It will now be
possible e.g., to read a file with sar on a little-endian machine even
if this has been created on a big-endian machine.
Yet it will still not be allowed to append (write) data with sar to a
file with a different endianness type.
projects / sysstat / log