]> granicus.if.org Git - zfs/log
zfs
5 years agoLinux 5.0 compat: ASM_BUG macro
Brian Behlendorf [Wed, 8 May 2019 17:18:40 +0000 (10:18 -0700)]
Linux 5.0 compat: ASM_BUG macro

The 5.0 kernel defines the macro ASM_BUG.  In order to prevent a
conflict and build failure rename ASM_BUG to ZFS_ASM_BUG.  This
is currently only an issue on aarch64 but all instances of
ASM_BUG we're renamed to avoid any future conflict on x86_64.

Reviewed-by: Tomohiro Kusumi <kusumi.tomohiro@gmail.com>
Reviewed-by: Richard Laager <rlaager@wiktel.com>
Reviewed-by: Chris Dunlop <chris@onthe.net.au>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #8725
Issue #8545

5 years agoFix errant EFAULT during writes (#8719)
Brian Behlendorf [Wed, 8 May 2019 17:04:04 +0000 (10:04 -0700)]
Fix errant EFAULT during writes (#8719)

Commit 98bb45e resolved a deadlock which could occur when
handling a page fault in zfs_write().  This change added
the uio_fault_disable field to the uio structure but failed
to initialize it to B_FALSE.  This uninitialized field would
cause uiomove_iov() to call __copy_from_user_inatomic()
instead of copy_from_user() resulting in unexpected EFAULTs.

Resolve the issue by fully initializing the uio, and clearing
the uio_fault_disable flags after it's used in zfs_write().

Additionally, reorder the uio_t field assignments to match
the order the fields are declared in the  structure.

Reviewed-by: Chunwei Chen <tuxoko@gmail.com>
Reviewed-by: Richard Laager <rlaager@wiktel.com>
Reviewed-by: Tim Chase <tim@chase2k.com>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #8640
Closes #8719

5 years agoMake zfs_special_class_metadata_reserve_pct into a parameter
DeHackEd [Tue, 7 May 2019 22:34:42 +0000 (18:34 -0400)]
Make zfs_special_class_metadata_reserve_pct into a parameter

Exported and documented a new module parameter.

Reviewed-by: Richard Laager <rlaager@wiktel.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: DHE <git@dehacked.net>
Closes #8706

5 years agoFix send/recv lost spill block
Brian Behlendorf [Tue, 7 May 2019 22:18:44 +0000 (15:18 -0700)]
Fix send/recv lost spill block

When receiving a DRR_OBJECT record the receive_object() function
needs to determine how to handle a spill block associated with the
object.  It may need to be removed or kept depending on how the
object was modified at the source.

This determination is currently accomplished using a heuristic which
takes in to account the DRR_OBJECT record and the existing object
properties.  This is a problem because there isn't quite enough
information available to do the right thing under all circumstances.
For example, when only the block size changes the spill block is
removed when it should be kept.

What's needed to resolve this is an additional flag in the DRR_OBJECT
which indicates if the object being received references a spill block.
The DRR_OBJECT_SPILL flag was added for this purpose.  When set then
the object references a spill block and it must be kept.  Either
it is update to date, or it will be replaced by a subsequent DRR_SPILL
record.  Conversely, if the object being received doesn't reference
a spill block then any existing spill block should always be removed.

Since previous versions of ZFS do not understand this new flag
additional DRR_SPILL records will be inserted in to the stream.
This has the advantage of being fully backward compatible.  Existing
ZFS systems receiving this stream will recreate the spill block if
it was incorrectly removed.  Updated ZFS versions will correctly
ignore the additional spill blocks which can be identified by
checking for the DRR_SPILL_UNMODIFIED flag.

The small downside to this approach is that is may increase the size
of the stream and of the received snapshot on previous versions of
ZFS.  Additionally, when receiving streams generated by previous
unpatched versions of ZFS spill blocks may still be lost.

OpenZFS-issue: https://www.illumos.org/issues/9952
FreeBSD-issue: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=233277

Reviewed-by: Paul Dagnelie <pcd@delphix.com>
Reviewed-by: Matt Ahrens <mahrens@delphix.com>
Reviewed-by: Tom Caputi <tcaputi@datto.com>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #8668

5 years agoFix `zfs set atime|relatime=off|on` behavior on inherited datasets
Tomohiro Kusumi [Tue, 7 May 2019 17:06:30 +0000 (02:06 +0900)]
Fix `zfs set atime|relatime=off|on` behavior on inherited datasets

`zfs set atime|relatime=off|on` doesn't disable or enable the property
on read for datasets whose property was inherited from parent, until
a dataset is once unmounted and mounted again.

(The properties start to work properly if a dataset is once unmounted
and mounted again. The difference comes from regular mount process,
e.g. via zpool import, uses mount options based on properties read
from ondisk layout for each dataset, whereas
`zfs set atime|relatime=off|on` just remounts a specified dataset.)

--
 # zpool create p1 <device>
 # zfs create p1/f1
 # zfs set atime=off p1
 # echo test > /p1/f1/test
 # sync
 # zfs list
 NAME    USED  AVAIL     REFER  MOUNTPOINT
 p1      176K  18.9G     25.5K  /p1
 p1/f1    26K  18.9G       26K  /p1/f1
 # zfs get atime
 NAME   PROPERTY  VALUE  SOURCE
 p1     atime     off    local
 p1/f1  atime     off    inherited from p1
 # stat /p1/f1/test | grep Access | tail -1
 Access: 2019-04-26 23:32:33.741205192 +0900
 # cat /p1/f1/test
 test
 # stat /p1/f1/test | grep Access | tail -1
 Access: 2019-04-26 23:32:50.173231861 +0900
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ changed by read(2)
--

The problem is that zfsvfs::z_atime which was probably intended to keep
incore atime state just gets updated by a callback function of "atime"
property change, atime_changed_cb(), and never used for anything else.

Since now that all file read and atime update use a common function
zpl_iter_read_common() -> file_accessed(), and whether to update atime
via ->dirty_inode() is determined by atime_needs_update(),
atime_needs_update() needs to return false once atime is turned off.
It currently continues to return true on `zfs set atime=off`.

Fix atime_changed_cb() by setting or dropping SB_NOATIME in VFS super
block depending on a new atime value, so that atime_needs_update() works
as expected after property change.

The same problem applies to "relatime" except that a self contained
relatime test is needed. This is because relatime_need_update() is based
on a mount option flag MNT_RELATIME, which doesn't exist in datasets
with inherited "relatime" property via `zfs set relatime=...`, hence it
needs its own relatime test zfs_relatime_need_update().

Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Tomohiro Kusumi <kusumi.tomohiro@gmail.com>
Closes #8674
Closes #8675

5 years agoLinux 5.1 compat: Drop ULLONG_MAX and LLONG_MAX definitions
Tomohiro Kusumi [Tue, 7 May 2019 16:55:40 +0000 (01:55 +0900)]
Linux 5.1 compat: Drop ULLONG_MAX and LLONG_MAX definitions

Linux kernel commit 54d50897d544c874562253e2a8f70dfcad22afe8
"linux/kernel.h: split *_MAX and *_MIN macros into <linux/limits.h>"

which first appeared in 5.1 has moved several macros from
<linux/kernel.h> to <linux/limits.h>. This broke compilation due to
header inclusion order against the local header include/spl/sys/types.h
which also defines ULLONG_MAX and LLONG_MAX if undefined.

It looks like local ULLONG_MAX and LLONG_MAX were never needed
(or after spl integration ?) as <linux/kernel.h> has had the same
definitions since an upstream commit
111ebb6e6f7bd7de6d722c5848e95621f43700d9 in 2.6.18, so drop them.

--
linux/include/linux/limits.h:17: error: "LLONG_MAX" redefined [-Werror]
 #define LLONG_MAX ((long long)(~0ULL >> 1))
zfs/include/spl/sys/types.h:35: note: this is the location of the previous definition
 #define LLONG_MAX  ((long long)(~0ULL>>1))

Reviewed-by: Richard Laager <rlaager@wiktel.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Tomohiro Kusumi <kusumi.tomohiro@gmail.com>
Closes #8714

5 years agoCleanup special/dedup language
Richard Laager [Tue, 7 May 2019 16:51:09 +0000 (11:51 -0500)]
Cleanup special/dedup language

This standardizes the language on "deduplication tables" rather than
"dedup data" (which might be read as the data blocks rather than the
DDT).  Likewise, it standardizes on "small file blocks".  It also
standardizes on "normal" rather than using both "normal" and "general"
in the same paragraph. I also replaced "non-specified" with the more
explicit "non-dedup/special".

Reviewed-by: George Melikov <mail@gmelikov.ru>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Richard Laager <rlaager@wiktel.com>
Closes #8713

5 years agoFix zfs-mount-generator for datasets with spaces
Antonio Russo [Tue, 7 May 2019 16:32:23 +0000 (12:32 -0400)]
Fix zfs-mount-generator for datasets with spaces

Alternative implementation of @rlaager's original modification
of zfs-mount-generator fix, with @chrisrd's comments. Set
IFS to be only the tab character, matching our `-H` call in
`zfs list`, allowing spaces to appear in dataset names (and
mountpoints).

Also adds comments explaining our rationale.

Reviewed-by: Chris Dunlop <chris@onthe.net.au>
Reviewed-by: Richard Laager <rlaager@wiktel.com>
Reviewed-by: George Melikov <mail@gmelikov.ru>
Signed-off-by: Antonio Russo <antonio.e.russo@gmail.com>
Closes #8708
Closes #8718

5 years agoOpenZFS 10473 - zfs(1M) missing cross-reference to zfs-program(1M)
Richard Laager [Mon, 6 May 2019 17:37:18 +0000 (12:37 -0500)]
OpenZFS 10473 - zfs(1M) missing cross-reference to zfs-program(1M)

Authored by: Jason King <jason.king@joyent.com>
Reviewed by: Toomas Soome <tsoome@me.com>
Reviewed by: Andy Fiddaman <andy@omniosce.org>
Reviewed by: Peter Tribble <peter.tribble@gmail.com>
Reviewed by: Gergő Mihály Doma <domag02@gmail.com>
Reviewed-by: George Melikov <mail@gmelikov.ru>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Approved by: Dan McDonald <danmcd@joyent.com>
Ported-by: Richard Laager <rlaager@wiktel.com>
OpenZFS-issue: https://www.illumos.org/issues/10473
OpenZFS-commit: https://github.com/illumos/illumos-gate/commit/736e67003
Closes #8711

5 years agoFix typo/etc in module/zfs/zfs_ctldir.c
Tomohiro Kusumi [Sun, 5 May 2019 22:57:23 +0000 (07:57 +0900)]
Fix typo/etc in module/zfs/zfs_ctldir.c

Drop duplicated phrases in comments.

Also drop an obsolete comment "Perform a mount of the associated...",
as all it does now is get objid from DMU and lookup incore inode.

Reviewed-by: Giuseppe Di Natale <guss80@gmail.com>
Reviewed-by: Richard Laager <rlaager@wiktel.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Tomohiro Kusumi <kusumi.tomohiro@gmail.com>
Closes #8707

5 years agoLinux 5.0 compat: Use totalhigh_pages()
Tomohiro Kusumi [Sat, 4 May 2019 23:40:48 +0000 (08:40 +0900)]
Linux 5.0 compat: Use totalhigh_pages()

Linux kernel commit ca79b0c211af63fa3276f0e3fd7dd9ada2439839
"mm: convert totalram_pages and totalhigh_pages variables to atomic"

replaced `totalhigh_pages` with an inline function `totalhigh_pages()`.
This broke compilation on IA32, etc, as ZoL uses `totalhigh_pages`
on archs with highmem. Confirmed on Fedora 30 (5.0.9-301.fc30.i686).

Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Tomohiro Kusumi <kusumi.tomohiro@gmail.com>
Closes #8677
Closes #8701

5 years agoImprove rate at which new zvols are processed
John Gallagher [Sat, 4 May 2019 23:39:10 +0000 (16:39 -0700)]
Improve rate at which new zvols are processed

The kernel function which adds new zvols as disks to the system,
add_disk(), briefly opens and closes the zvol as part of its work.
Closing a zvol involves waiting for two txgs to sync. This, combined
with the fact that the taskq processing new zvols is single threaded,
makes this processing new zvols slow.

Waiting for these txgs to sync is only necessary if the zvol has been
written to, which is not the case during add_disk(). This change adds
tracking of whether a zvol has been written to so that we can skip the
txg_wait_synced() calls when they are unnecessary.

This change also fixes the flags passed to blkdev_get_by_path() by
vdev_disk_open() to be FMODE_READ | FMODE_WRITE | FMODE_EXCL instead of
just FMODE_EXCL. The flags were being incorrectly calculated because
we were using the wrong version of vdev_bdev_mode().

Reviewed-by: George Wilson <george.wilson@delphix.com>
Reviewed-by: Serapheim Dimitropoulos <serapheim@delphix.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: John Gallagher <john.gallagher@delphix.com>
Closes #8526
Closes #8615

5 years agoClearer wording on Errata #4
JMoVS [Thu, 2 May 2019 23:52:57 +0000 (01:52 +0200)]
Clearer wording on Errata #4

Users of existing pools, especially pools with top-level encrypted
datasets, could run into trouble trying to work around Errata #4.
Clarify that removing encrypted snapshots and bookmarks is enough
to clear the errata.

Reviewed-by: George Melikov <mail@gmelikov.ru>
Reviewed-by: Richard Laager <rlaager@wiktel.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: Tom Caputi <tcaputi@datto.com>
Signed-off-by: Justin Scholz <git@justinscholz.de>
Closes #8682
Closes #8683

5 years agoReword comment in lz4_compress_zfs
Matthew Ahrens [Thu, 2 May 2019 23:46:04 +0000 (16:46 -0700)]
Reword comment in lz4_compress_zfs

The comment in lz4_compress_zfs could be more clear and specific.  It
also contains needlessly strong language.

Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: Serapheim Dimitropoulos <serapheim@delphix.com>
Signed-off-by: Matthew Ahrens <mahrens@delphix.com>
Closes: #8702
Closes: #8703
5 years agoAdd feature check for 'zpool resilver' command
Tom Caputi [Thu, 2 May 2019 23:42:31 +0000 (19:42 -0400)]
Add feature check for 'zpool resilver' command

The 'zpool resilver' command requires that the resilver_defer
feature is active on the pool. Unfortunately, the check for
this was left out of the original patch. This commit simply
corrects this so that the command properly returns an error
in this case.

Reviewed by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: Igor Kozhukhov <igor@dilos.org>
Signed-off-by: Tom Caputi <tcaputi@datto.com>
Closes #8700

5 years agoFix estimated scrub completion time
Tom Caputi [Thu, 2 May 2019 00:34:24 +0000 (20:34 -0400)]
Fix estimated scrub completion time

Currently, it is possible for the 'zpool scrub' command to
progress slightly beyond 100% due to concurrent changes
happening on the live pool. This behavior is expected, but
the userspace code for 'zpool status' would subtract the
expected amount of data from the amount of data already
scrubbed, resulting in a negative integer being casted to a
large positive one. This number was then used to calculate
the estimated completion time, resulting in wildly wrong
results. This code changes the behavior so that 'zpool status'
does not attempt to report an estimate during this period.

Reviewed by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: Igor Kozhukhov <igor@dilos.org>
Reviewed-by: George Melikov <mail@gmelikov.ru>
Signed-off-by: Tom Caputi <tcaputi@datto.com>
Closes #8611
Closes #8687

5 years agoRemove incorrect (and inappropriate) comment in dprintf_dnode
Matthew Ahrens [Thu, 2 May 2019 00:32:54 +0000 (17:32 -0700)]
Remove incorrect (and inappropriate) comment in dprintf_dnode

This comment seems to misunderstand the ## preprocessor token, which
does token concatenation.  It is not needed here, since we are
concatenating string literals, which is performed by putting the
literals next to each other.

Additionally, the comment uses offensive language.

Reviewed-by: Igor Kozhukhov <igor@dilos.org>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: George Melikov <mail@gmelikov.ru>
Reviewed-by: Tony Nguyen <tony.nguyen@delphix.com>
Signed-off-by: Matthew Ahrens <mahrens@delphix.com>
Closes #8698
Closes #8699

5 years agoUse sigaction(2) instead of sigignore(3) for portability
Tomohiro Kusumi [Tue, 9 Apr 2019 16:58:28 +0000 (01:58 +0900)]
Use sigaction(2) instead of sigignore(3) for portability

sigignore(3) isn't portable.
This code fails to compile on platforms without sigignore(3).
Use sigaction(2).

--
zfs_main.c: In function 'zfs_do_diff':
zfs_main.c:7178:9: error: implicit declaration of function 'sigignore' [-Werror=implicit-function-declaration]
  (void) sigignore(SIGPIPE);
         ^~~~~~~~~

Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Tomohiro Kusumi <kusumi.tomohiro@gmail.com>
Closes #8593

5 years agoUse sigaction(2) instead of sigset(3) for portability
Tomohiro Kusumi [Tue, 9 Apr 2019 16:58:03 +0000 (01:58 +0900)]
Use sigaction(2) instead of sigset(3) for portability

sigset(3) isn't portable.
This code fails to compile on platforms without sigset(3).
Use sigaction(2).

--
largest_file.c: In function 'main':
largest_file.c:75:9: error: implicit declaration of function 'sigset'; did you mean 'sigvec'? [-Werror=implicit-function-declaration]
  (void) sigset(SIGXFSZ, sigxfsz);
         ^~~~~~
         sigvec

Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Tomohiro Kusumi <kusumi.tomohiro@gmail.com>
Closes #8593

5 years agoCorrect snprintf() size argument
Tomohiro Kusumi [Wed, 1 May 2019 02:41:12 +0000 (11:41 +0900)]
Correct snprintf() size argument

The size argument of snprintf(3) in glibc and snprintf() in Linux
kernel includes trailing \0, as snprintf(3) man page explains it as
"write at most size bytes (including the trailing null byte ('\0'))",
i.e. snprintf() can just take buffer size.

e.g. For snprintf() in module/zfs/zfs_ctldir.c, a buffer size is
MAXPATHLEN, and a caller is passing MAXPATHLEN to snprintf(), so size
should just be `path_len` to do what the caller is trying to do.

Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: George Melikov <mail@gmelikov.ru>
Reviewed-by: Richard Laager <rlaager@wiktel.com>
Signed-off-by: Tomohiro Kusumi <kusumi.tomohiro@gmail.com>
Closes #8692

5 years agoClarify that deduped data is encrypted
Richard Laager [Tue, 30 Apr 2019 20:53:54 +0000 (15:53 -0500)]
Clarify that deduped data is encrypted

Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: George Melikov <mail@gmelikov.ru>
Reviewed-by: Tom Caputi <tcaputi@datto.com>
Signed-off-by: Richard Laager <rlaager@wiktel.com>
Closes #8691

5 years agoAdd CODE_OF_CONDUCT.md
Brian Behlendorf [Tue, 30 Apr 2019 17:58:45 +0000 (10:58 -0700)]
Add CODE_OF_CONDUCT.md

Signed-off-by: Matthew Ahrens <mahrens@delphix.com>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #8660

5 years agoLinux 5.0 compat: Remove incorrect ASSERT
Brian Behlendorf [Tue, 30 Apr 2019 01:20:21 +0000 (18:20 -0700)]
Linux 5.0 compat: Remove incorrect ASSERT

Not all block devices, notably scsi_debug, set a root_blkg on the
request queue.  Remove this assertion and allow the the existing
call to blkg_tryget() to gracefully handle the NULL (which it does).

Reviewed-by: Tomohiro Kusumi <kusumi.tomohiro@gmail.com>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #8678

5 years agoUse NV_ENCODE_NATIVE for nvlist encoding variable
Tomohiro Kusumi [Fri, 26 Apr 2019 18:24:31 +0000 (03:24 +0900)]
Use NV_ENCODE_NATIVE for nvlist encoding variable

Use NV_ENCODE_NATIVE for nvlist encoding variable instead of 0.

Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: Chris Dunlop <chris@onthe.net.au>
Signed-off-by: Tomohiro Kusumi <kusumi.tomohiro@gmail.com>
Closes #8653

5 years agoPrevent `make distclean` removing config/config.rpath
Tomohiro Kusumi [Fri, 26 Apr 2019 18:22:14 +0000 (03:22 +0900)]
Prevent `make distclean` removing config/config.rpath

`make distclean` removes an empty file config/config.rpath.
Avoid that by adding some text.

Also see e1245d83e9("Prevent `make distclean` removing 0 sized file").

--
 # find . -size 0
 ./config/config.rpath
 # ./autogen.sh && ./configure
 # git diff
 # make distclean
 # git diff
 diff --git a/config/config.rpath b/config/config.rpath
 deleted file mode 100644
 index e69de29bb..000000000

Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Tomohiro Kusumi <kusumi.tomohiro@gmail.com>
Closes #8665

5 years agoUse SEEK_{SET,CUR,END} for file seek "whence"
Tomohiro Kusumi [Thu, 25 Apr 2019 17:17:28 +0000 (02:17 +0900)]
Use SEEK_{SET,CUR,END} for file seek "whence"

Use either SEEK_* or 0,1,2..., but not both.

Reviewed-by: George Melikov <mail@gmelikov.ru>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Tomohiro Kusumi <kusumi.tomohiro@gmail.com>
Closes #8656

5 years agoFixes for the DMU free throttle
Tom Caputi [Thu, 25 Apr 2019 17:16:24 +0000 (13:16 -0400)]
Fixes for the DMU free throttle

This patch fixes 2 issues with the DMU free throttle implemented
in dmu_free_long_range(). The first issue is that get_next_chunk()
was calculating the number of L1 blocks the free would dirty
incorrectly. In some cases involving extremely large files, this
code would greatly overestimate the number of effected L1 blocks,
causing excessive calls to txg_wait_open(). This patch corrects
the calculation.

The second issue is that the free throttle uses the total number
of free'd blocks in all (open, quiescing, and syncing) txgs to
determine whether to throttle. This causes large frees (such as
those created by the first issue) to cause 4 txg syncs before
any further frees were allowed to proceed. This patch ensures
that the accounting is done entirely in a per-txg fashion, so
that frees from a given txg don't affect those that immediately
follow it.

Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: Matthew Ahrens <mahrens@delphix.com>
Signed-off-by: Tom Caputi <tcaputi@datto.com>
Closes #8655

5 years agoClarify and improve encryption documentation
Richard Laager [Thu, 25 Apr 2019 00:14:25 +0000 (19:14 -0500)]
Clarify and improve encryption documentation

- Remove the language that "all user data" is encrypted.  This is to
  avoid misunderstandings or arguments about what is "user data",
  especially in light of "user properties".
- Document that properties are unencrypted.
- Document that snapshot names are unencrypted.
- For consistency with the rest of the zfs.8 man page, use "ZFS" as the
  generic noun, not (bolded) "zfs".  The latter refers to the command.
  Likewise, use "ZFS" instead of "the kernel module".
- Give "a passphrase" as an example of a "user's key".

Reviewed-by: Tom Caputi <tcaputi@datto.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: George Melikov <mail@gmelikov.ru>
Signed-off-by: Richard Laager <rlaager@wiktel.com>
Closes #8652

5 years agoDuplicate the encryption copies=3 limitation
Richard Laager [Thu, 25 Apr 2019 00:12:14 +0000 (19:12 -0500)]
Duplicate the encryption copies=3 limitation

This adds the encryption copies=3 limitation language into the copies
property section.

Reviewed-by: George Melikov <mail@gmelikov.ru>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Richard Laager <rlaager@wiktel.com>
Closes #8651

5 years agoDeprecate dedupditto
Richard Laager [Thu, 25 Apr 2019 00:10:47 +0000 (19:10 -0500)]
Deprecate dedupditto

This documents, in zpool.8, that dedupditto is deprecated and will be
made to have no effect in a future release.

Reviewed-by: George Melikov <mail@gmelikov.ru>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: Matthew Ahrens <mahrens@delphix.com>
Signed-off-by: Richard Laager <rlaager@wiktel.com>
Closes #8650

5 years agoEliminate useless double-bolding in man pages
Richard Laager [Sun, 21 Apr 2019 20:56:33 +0000 (15:56 -0500)]
Eliminate useless double-bolding in man pages

As far as I know and can tell from testing, \fB\fB...\fR\fR is exactly
equivalent to \fB...\fR.

Reviewed-by: Tom Caputi <tcaputi@datto.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Richard Laager <rlaager@wiktel.com>
Closes #8641

5 years agoAlphabetize zpool-features.5 by short name
Richard Laager [Sun, 21 Apr 2019 20:47:23 +0000 (15:47 -0500)]
Alphabetize zpool-features.5 by short name

The features are sorted in the en_US locale, not the C locale.
Specifically, that means that bookmark_v2 comes _after_ bookmarks.

Reviewed-by: Tom Caputi <tcaputi@datto.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Richard Laager <rlaager@wiktel.com>
Closes #8641

5 years agoStandardize .RE placement in zpool-features.5
Richard Laager [Sun, 21 Apr 2019 20:46:21 +0000 (15:46 -0500)]
Standardize .RE placement in zpool-features.5

This command is being used to unindent, so it should be at the end of
each block.  This is consistent with the other man pages.

Reviewed-by: Tom Caputi <tcaputi@datto.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Richard Laager <rlaager@wiktel.com>
Closes #8641

5 years agoAdd missing formatting to sha512 in zpool-features.5
Richard Laager [Sun, 21 Apr 2019 20:24:52 +0000 (15:24 -0500)]
Add missing formatting to sha512 in zpool-features.5

Reviewed-by: Tom Caputi <tcaputi@datto.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Richard Laager <rlaager@wiktel.com>
Closes #8641

5 years agoCorrect GUID of large_blocks in zpool-features.5
Richard Laager [Sun, 21 Apr 2019 20:23:52 +0000 (15:23 -0500)]
Correct GUID of large_blocks in zpool-features.5

It is org.open-zfs:large_blocks (plural).

Reviewed-by: Tom Caputi <tcaputi@datto.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Richard Laager <rlaager@wiktel.com>
Closes #8641

5 years agoChange wording in zfs-module-parameters.5
Tom Caputi [Sun, 21 Apr 2019 20:12:30 +0000 (15:12 -0500)]
Change wording in zfs-module-parameters.5

Reviewed-by: Tom Caputi <tcaputi@datto.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Richard Laager <rlaager@wiktel.com>
Closes #8641

5 years agoDocument that hole_birth is effectively useless
Richard Laager [Fri, 19 Apr 2019 01:21:09 +0000 (20:21 -0500)]
Document that hole_birth is effectively useless

The first sentence of this commit comes from the wiki, and was
originally written by:
Rich Ercolani <rincebrain@gmail.com>
with changes by:
Tom Caputi <tcaputi@datto.com>

Reviewed-by: Tom Caputi <tcaputi@datto.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Richard Laager <rlaager@wiktel.com>
Closes #8641
Closes #8642

5 years agoDocument send_holes_without_birth_time
Richard Laager [Fri, 19 Apr 2019 01:15:58 +0000 (20:15 -0500)]
Document send_holes_without_birth_time

Reviewed-by: Tom Caputi <tcaputi@datto.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Richard Laager <rlaager@wiktel.com>
Closes #8641

5 years agoCorrect bookmark_v2 dependencies
Richard Laager [Fri, 19 Apr 2019 00:09:11 +0000 (19:09 -0500)]
Correct bookmark_v2 dependencies

encryption depends on bookmark_v2.

bookmark_v2 depends on bookmarks.

Reviewed-by: Tom Caputi <tcaputi@datto.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Richard Laager <rlaager@wiktel.com>
Closes #8641

5 years agoFix formatting for multi_vdev_crash_dump in zpool-features.5
Richard Laager [Fri, 19 Apr 2019 00:08:32 +0000 (19:08 -0500)]
Fix formatting for multi_vdev_crash_dump in zpool-features.5

This needs to use tabs instead of spaces to display correctly (i.e. with
things lined up).

Reviewed-by: Tom Caputi <tcaputi@datto.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Richard Laager <rlaager@wiktel.com>
Closes #8641

5 years agoFix incorrect use of .Nm directive for ZPOOL_VDEV_NAME_GUID in zpool(8)
Tomohiro Kusumi [Tue, 23 Apr 2019 17:23:00 +0000 (02:23 +0900)]
Fix incorrect use of .Nm directive for ZPOOL_VDEV_NAME_GUID in zpool(8)

It should only affect "zpool".

Reviewed-by: George Melikov <mail@gmelikov.ru>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: Richard Laager <rlaager@wiktel.com>
Signed-off-by: Tomohiro Kusumi <kusumi.tomohiro@gmail.com>
Closes #8644

5 years agoconfig: libintl/libiconv for gettext() detection
Rafael Kitover [Fri, 19 Apr 2019 19:09:29 +0000 (19:09 +0000)]
config: libintl/libiconv for gettext() detection

Detect in autoconf whether `-lintl` and possibly `-liconv` are necessary
for translation functions like `gettext()`.

The actual autoconf code is just:

```
AM_ICONV
AM_GNU_GETTEXT([external])
LIBS="$LIBS $LTLIBINTL $LTLIBICONV"
```

References:

https://www.gnu.org/software/gettext/manual/html_node/AM_005fGNU_005fGETTEXT.html
https://www.gnu.org/software/gettext/manual/html_node/AM_005fICONV.html

The reason to check for `libiconv` and add it separately is that this is
sometimes necessary if users are linking statically.

The `config/*.m4` files were added by running `gettextize` and removing
everything else.

The empty file `config/config.rpath` is necessary to avoid an error with
some versions of autotools, see:

http://ramblingfoo.blogspot.com/2007/07/required-file-configrpath-not-found.html

The `config.rpath` copied by `gettextize` does not currently work, there
is some kind of missing interaction with `libtool` and it tries to apply
`libtool` flags to the compiler.

Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: Richard Laager <rlaager@wiktel.com>
Signed-off-by: Rafael Kitover <rkitover@gmail.com>
Closes #8554

5 years agoDrop unused ZNODE_STATS and ZNODE_STAT_ADD()
Tomohiro Kusumi [Fri, 19 Apr 2019 19:05:15 +0000 (04:05 +0900)]
Drop unused ZNODE_STATS and ZNODE_STAT_ADD()

Unused since 5649246dd3("Remove znode move functionality"),
and ZNODE_STAT_ADD() will never be needed.

Reviewed-by: Richard Elling <Richard.Elling@RichardElling.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: George Melikov <mail@gmelikov.ru>
Signed-off-by: Tomohiro Kusumi <kusumi.tomohiro@gmail.com>
Closes #8636

5 years agoFix typo "/zbin/zpool" -> "/sbin/zpool"
Tomohiro Kusumi [Fri, 19 Apr 2019 19:04:21 +0000 (04:04 +0900)]
Fix typo "/zbin/zpool" -> "/sbin/zpool"

Reviewed-by: Richard Elling <Richard.Elling@RichardElling.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: George Melikov <mail@gmelikov.ru>
Signed-off-by: Tomohiro Kusumi <kusumi.tomohiro@gmail.com>
Closes #8643

5 years agoFix incorrect "[UNUSED]" comments
Tomohiro Kusumi [Fri, 19 Apr 2019 19:03:32 +0000 (04:03 +0900)]
Fix incorrect "[UNUSED]" comments

These aren't unused.
`flag` in zfs_create() also isn't to indicate large file.

Reviewed-by: Richard Elling <Richard.Elling@RichardElling.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: George Melikov <mail@gmelikov.ru>
Reviewed-by: Richard Laager <rlaager@wiktel.com>
Signed-off-by: Tomohiro Kusumi <kusumi.tomohiro@gmail.com>
Closes #8635

5 years agoTag 0.8.0-rc4 zfs-0.8.0-rc4
Brian Behlendorf [Tue, 16 Apr 2019 20:24:49 +0000 (13:24 -0700)]
Tag 0.8.0-rc4

Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
5 years agoCode improvement and bug fixes for QAT support
cfzhu [Tue, 16 Apr 2019 19:38:36 +0000 (03:38 +0800)]
Code improvement and bug fixes for QAT support

1. Support QAT when ZFS is root file-system:
   When ZFS module is loaded before QAT started, the QAT can
   be started again in post-process, e.g.:
   echo 0 > /sys/module/zfs/parameters/zfs_qat_compress_disable
   echo 0 > /sys/module/zfs/parameters/zfs_qat_encrypt_disable
   echo 0 > /sys/module/zfs/parameters/zfs_qat_checksum_disable
2. Verify alder checksum of the de-compress result
3. Allocate Digest, IV and AAD buffer in physical contiguous
   memory by QAT_PHYS_CONTIG_ALLOC.
4. Update the documentation for zfs_qat_compress_disable,
   zfs_qat_checksum_disable, zfs_qat_encrypt_disable.

Reviewed-by: Tom Caputi <tcaputi@datto.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Weigang Li <weigang.li@intel.com>
Signed-off-by: Chengfeix Zhu <chengfeix.zhu@intel.com>
Closes #8323
Closes #8610

5 years agoRestructure vec_idx loops
Richard Laager [Tue, 16 Apr 2019 19:34:06 +0000 (14:34 -0500)]
Restructure vec_idx loops

This replaces empty for loops with while loops to make the code easier
to read.

Reviewed-by: Tom Caputi <tcaputi@datto.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reported-by: github.com/dcb314
Signed-off-by: Richard Laager <rlaager@wiktel.com>
Closes #6681
Closes #6682
Closes #6683
Closes #8623

5 years agoAdd option [-V|--version] to emit version string
TerraTech [Wed, 10 Apr 2019 07:43:28 +0000 (00:43 -0700)]
Add option [-V|--version] to emit version string

Add the 'zfs version' and 'zpool version' subcommands to display
the version of the user space utilities and loaded zfs kernel
module.  For example:

$ zfs version
zfs-0.8.0-rc3_169_g67e0366b88
zfs-kmod-0.8.0-rc3_169_g67e0366b88

The '-V' and '--version' aliases were added to support the
common convention of using 'zfs --version` to obtain the version
information.

Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: Matthew Ahrens <mahrens@delphix.com>
Reviewed-by: Richard Laager <rlaager@wiktel.com>
Signed-off-by: TerraTech <1118433+TerraTech@users.noreply.github.com>
Closes #2501
Closes #8567

5 years agozfs allow refreservation needed for zfs create -V
Richard Laager [Tue, 16 Apr 2019 17:12:08 +0000 (12:12 -0500)]
zfs allow refreservation needed for zfs create -V

When creating a non-sparse volume, zfs create sets a refreservation.
Accordingly, one needs the "refreservation" ability in addition to the
"create" ability in order to create a non-sparse volume.

Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reported-by: github.com/homerlinux
Reported-by: Matthew Ahrens <mahrens@delphix.com>
Signed-off-by: Richard Laager <rlaager@wiktel.com>
Closes #8531
Closes #8624

5 years agoClarify GRUB's lack of support for sha512, skein, edonr
Richard Laager [Mon, 15 Apr 2019 02:30:13 +0000 (21:30 -0500)]
Clarify GRUB's lack of support for sha512, skein, edonr

zfs.8 correctly said that GRUB did not support them, but
zpool-features.5 said that "Booting off pools...is supported."  Now,
zpool-features.5 discusses GRUB specifically and indicates its lack of
support for these features.  Also, I have clarified the wording in both
places to indicate that the pool feature cannot be used.  It's not a
filesystem dataset thing, but pool-wide.

I described this as "cannot be used".  I think technically the feature
can be enabled, just not active.  However, the effect is essentially the
same: you cannot enable those checksum algorithms on any dataset in the
pool, so you might as well not enable the feature (which is just
pointing a loaded gun at your foot).  In the past, an argument could be
made that having all the features enabled was useful for simplicity, as
long as you didn't activate the GRUB-incompatible features, but that's
getting less and less realistic over time.  A user can still do that,
but we should not encourage that.

Reviewed-by: George Melikov <mail@gmelikov.ru>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Richard Laager <rlaager@wiktel.com>
Closes #8626
Closes #8446

5 years agoUpdate a comment to match the code
Richard Laager [Mon, 15 Apr 2019 02:23:26 +0000 (21:23 -0500)]
Update a comment to match the code

GRUB supports large_blocks.

Reviewed-by: George Melikov <mail@gmelikov.ru>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Richard Laager <rlaager@wiktel.com>
Closes #8626

5 years agoReference zfeature.c in a SPA_VERSION comment
Richard Laager [Tue, 16 Apr 2019 04:59:37 +0000 (23:59 -0500)]
Reference zfeature.c in a SPA_VERSION comment

Reviewed-by: George Melikov <mail@gmelikov.ru>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Richard Laager <rlaager@wiktel.com>
Closes #8626

5 years agoRemove zfs.h comments about GRUB
Richard Laager [Mon, 15 Apr 2019 02:22:38 +0000 (21:22 -0500)]
Remove zfs.h comments about GRUB

Nobody is going to be bumping SPA_VERSION again, as OpenZFS has moved on
to feature flags.  Also, there is no requirement to keep GRUB
up-to-date, nor has that been happening.

The ZPL_VERSION could be bumped, but that would likely be handled in a
similar way, by adding filesystem feature flags.  In any event, we do
not need this comment, and we certainly don't need a reference to the
GRUB 0.97 source code in a Solaris tree.

Reviewed-by: George Melikov <mail@gmelikov.ru>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Richard Laager <rlaager@wiktel.com>
Closes #8626

5 years agoReword the dedup limitation for Edon-R
Richard Laager [Mon, 15 Apr 2019 02:19:36 +0000 (21:19 -0500)]
Reword the dedup limitation for Edon-R

The old wording was effectively "You can not use this (except you can)",
which just seems confusing.

Reviewed-by: George Melikov <mail@gmelikov.ru>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Richard Laager <rlaager@wiktel.com>
Closes #8626

5 years agoConsistently captialize GUID for features
Richard Laager [Mon, 15 Apr 2019 02:15:04 +0000 (21:15 -0500)]
Consistently captialize GUID for features

Reviewed-by: George Melikov <mail@gmelikov.ru>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Richard Laager <rlaager@wiktel.com>
Closes #8626

5 years agoFix the spelling of deferred
Richard Laager [Mon, 15 Apr 2019 02:07:36 +0000 (21:07 -0500)]
Fix the spelling of deferred

Reviewed-by: George Melikov <mail@gmelikov.ru>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Richard Laager <rlaager@wiktel.com>
Closes #8626

5 years agoUpdate zdb.8's fsck reference
Richard Laager [Mon, 15 Apr 2019 02:06:34 +0000 (21:06 -0500)]
Update zdb.8's fsck reference

On Linux, this is in man section 8, not 1M.  Also, there is no fsdb on
Linux, so I removed that.

Reviewed-by: George Melikov <mail@gmelikov.ru>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Richard Laager <rlaager@wiktel.com>
Closes #8626

5 years agoRefer to commands consistently in zpool-features.5
Richard Laager [Mon, 15 Apr 2019 01:52:34 +0000 (20:52 -0500)]
Refer to commands consistently in zpool-features.5

This had a mix of command vs subcommand, quoted vs not quoted, and
bolded vs. not bolded command names.

Also, fix man page sections from 1M (Solaris) to 8 (Linux).

Reviewed-by: George Melikov <mail@gmelikov.ru>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Richard Laager <rlaager@wiktel.com>
Closes #8626

5 years agoEliminate most mentions of "special"
Richard Laager [Mon, 15 Apr 2019 01:46:19 +0000 (20:46 -0500)]
Eliminate most mentions of "special"

Previously, the "spare" vdev type was described as "A special
pseudo-vdev which...".  I wanted to eliminate the word "special" from
that, now that the allocation_classes feature exists and there is such a
thing as a "special vdev".  I ended up eliminating almost all instances
of the word "special" that are not referencing the allocation_classes
feature.

Reviewed-by: George Melikov <mail@gmelikov.ru>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Richard Laager <rlaager@wiktel.com>
Closes #8626

5 years agoFix issues with truncated files in raw sends
Tom Caputi [Wed, 10 Apr 2019 17:17:36 +0000 (10:17 -0700)]
Fix issues with truncated files in raw sends

When receiving a raw send stream only reallocated objects
whose contents were not freed by the standard indicators
should call dmu_free_long_range().

Furthermore, if calling dmu_free_long_range() is required
then the objects current block size must be used and not
the new block size.

Two additional test cases were added to provided realistic
test coverage for processing reallocated objects which are
part of a raw receive.

Reviewed-by: Olaf Faaland <faaland1@llnl.gov>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Tom Caputi <tcaputi@datto.com>
Closes #8528
Closes #8607

5 years agoFix hierarchy misspellings
Richard Laager [Mon, 15 Apr 2019 02:06:34 +0000 (21:06 -0500)]
Fix hierarchy misspellings

Reviewed-by: George Melikov <mail@gmelikov.ru>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reported-by: Matthew Ahrens <mahrens@delphix.com>
Signed-off-by: Richard Laager <rlaager@wiktel.com>
Closes #8563
Closes #8622

5 years agoDon't hard-code number of ioctls for portability
Tomohiro Kusumi [Sun, 7 Apr 2019 17:18:01 +0000 (02:18 +0900)]
Don't hard-code number of ioctls for portability

Use (ZFS_IOC_LAST - ZFS_IOC_FIRST) instead of 256.
It seems 256 is just a number large enough to hold ioctls
at the moment.

Using 256 also causes compile-time warning or error
on platfoms whose enum zfs_ioc definition differs.

Reviewed-by: Olaf Faaland <faaland1@llnl.gov>
Reviewed by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: Igor Kozhukhov <igor@dilos.org>
Signed-off-by: Tomohiro Kusumi <kusumi.tomohiro@gmail.com>
Closes #8598

5 years agoSync reserved Illumos ioctl comment with actual number
Tomohiro Kusumi [Sun, 7 Apr 2019 17:35:28 +0000 (02:35 +0900)]
Sync reserved Illumos ioctl comment with actual number

It's 81 now.

Reviewed-by: Olaf Faaland <faaland1@llnl.gov>
Reviewed by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: Igor Kozhukhov <igor@dilos.org>
Signed-off-by: Tomohiro Kusumi <kusumi.tomohiro@gmail.com>
Closes #8598

5 years agocompile with -fno-omit-frame-pointer
Richard Elling [Sun, 14 Apr 2019 18:04:54 +0000 (11:04 -0700)]
compile with -fno-omit-frame-pointer

By default, depending on the version, gcc can reuse the frame pointer register.
This is a micro-optimization that might help on some very old x86 processors.
However, it also makes dynamic tracing less useful because the stacks cannot
be easily observed.

This rule change instructs gcc to use the -fno-omit-frame-pointer option
when compiling.

Reviewed-by: Olaf Faaland <faaland1@llnl.gov>
Reviewed by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Richard Elling <Richard.Elling@RichardElling.com>
Closes #8617

5 years agoCleanup nits from ab7615d92
Tom Caputi [Sun, 14 Apr 2019 18:03:06 +0000 (14:03 -0400)]
Cleanup nits from ab7615d92

This patch simply up cleans up a nit and corrects an error message
issue that were introduced in the Multiple DVA scrub patch.

Reviewed-by: George Melikov <mail@gmelikov.ru>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Tom Caputi <tcaputi@datto.com>
Closes #8619

5 years agoFix issue in receive_object() during reallocation
Brian Behlendorf [Fri, 12 Apr 2019 21:28:04 +0000 (14:28 -0700)]
Fix issue in receive_object() during reallocation

When receiving an object to a previously allocated interior slot
the new object should be "allocated" by setting DMU_NEW_OBJECT,
not "reallocated" with dnode_reallocate().  For resilience verify
the slot is free as required in case the stream is malformed.

Add a test case to generate more realistic incremental send streams
that force reallocation to occur during the receive.

Reviewed-by: Olaf Faaland <faaland1@llnl.gov>
Reviewed-by: Tom Caputi <tcaputi@datto.com>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #8067
Closes #8614

5 years agoFix TXG_MASK cstyle
Brian Behlendorf [Fri, 12 Apr 2019 18:30:59 +0000 (11:30 -0700)]
Fix TXG_MASK cstyle

Fix style issue for 'tx->tx_txg&TXG_MASK'.  There should be white
space around the '&' character.  Split the dnode_reallocate() ASSERT
to make it more readable to clearly separate the checks.

Reviewed-by: George Melikov <mail@gmelikov.ru>
Reviewed-by: Tom Caputi <tcaputi@datto.com>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #8606

5 years agoZTS: Make fault cleanup function more robust
John Wren Kennedy [Fri, 12 Apr 2019 17:07:20 +0000 (11:07 -0600)]
ZTS: Make fault cleanup function more robust

The cleanup function of auto_online_001_pos does not account for the
possibility that the test may fail while a disk is still removed. If
the test run is using real disks, cleanup should involve restoring any
that are missing.

Reviewed-by: Giuseppe Di Natale <guss80@gmail.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: John Kennedy <john.kennedy@delphix.com>
Closes #8579

5 years agoAllow zfs-tests to recover from hibernation
Alek P [Thu, 11 Apr 2019 17:20:37 +0000 (10:20 -0700)]
Allow zfs-tests to recover from hibernation

When a system sleeps during a zfs-test, the time spent
hibernating is counted against the test's runtime even
though the test can't and isn't running.
This patch tries to detect timeouts due to hibernation and
reruns tests that timed out due to system sleeping.
In this version of the patch, the existing behavior of returning
non-zero when a test was killed is preserved. With this patch applied
we still return nonzero and we also automatically rerun the test we
suspect of being killed due to system hibernation.

Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed by: John Kennedy <john.kennedy@delphix.com>
Signed-off-by: Alek Pinchuk <apinchuk@datto.com>
Closes #8575

5 years agoAlways call rw_init in zio_crypt_key_unwrap
Jorgen Lundman [Wed, 10 Apr 2019 22:39:40 +0000 (07:39 +0900)]
Always call rw_init in zio_crypt_key_unwrap

The error path in zio_crypt_key_unwrap would call zio_crypt_key_destroy which
calls rw_destroy(&key->zk_salt_lock); which has not yet been initialized.

We move the rw_init() call to the start of zio_crypt_key_unwrap instead.

Reviewed-by: Tom Caputi <tcaputi@datto.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Jorgen Lundman <lundman@lundman.net>
Closes #8604
Closes #8605

5 years agoAvoid stack overwrite in zfs_setattr_dir()
Tim Chase [Wed, 10 Apr 2019 22:38:21 +0000 (17:38 -0500)]
Avoid stack overwrite in zfs_setattr_dir()

The bulk[] array index, count, must be reset per-iteration in order to
not overwrite the stack.

Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: Chris Dunlop <chris@onthe.net.au>
Reviewed-by: Tom Caputi <tcaputi@datto.com>
Signed-off-by: Tim Chase <tim@chase2k.com>
Closes #8072
Closes #8597
Closes #8601

5 years agoDon't assume pthread_t is uint_t for portability
Tomohiro Kusumi [Wed, 10 Apr 2019 03:49:03 +0000 (12:49 +0900)]
Don't assume pthread_t is uint_t for portability

POSIX doesn't define pthread_t as uint_t. It could be a pointer.
This code causes below compile error on a platform using pointer
for pthread_t.

--
kernel.c:815:25: error: cast from pointer to integer of different size [-Werror=pointer-to-int-cast]
    (void) printf("%u ", (uint_t)pthread_self());

Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: Igor Kozhukhov <igor@dilos.org>
Signed-off-by: Tomohiro Kusumi <kusumi.tomohiro@gmail.com>
Closes #8558

5 years agoUnbreak build on Linux kernel < 3.10
Tomohiro Kusumi [Mon, 8 Apr 2019 21:59:24 +0000 (06:59 +0900)]
Unbreak build on Linux kernel < 3.10

d12614521a("Fixes for procfs files backed by linked lists")
uses PDE_DATA(), but since PDE_DATA() (public interface which
replaced old public interface PDE()) first appeared in upstream
kernel 3.10, it lacks visible local definition for kernel < 3.10.

Move the local PDE_DATA() definition to a ZoL header, to unbreak
build on kernel < 3.10.

--
module/spl/spl-procfs-list.c: In function 'procfs_list_open':
module/spl/spl-procfs-list.c:166: error: implicit declaration of function 'PDE_DATA'
module/spl/spl-procfs-list.c:166: warning: assignment makes pointer from integer without a cast

Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: John Gallagher <john.gallagher@delphix.com>
Signed-off-by: Tomohiro Kusumi <kusumi.tomohiro@gmail.com>
Closes #8599

5 years agoFix 'zfs list -t snapshot' depth
Brian Behlendorf [Mon, 8 Apr 2019 16:14:45 +0000 (09:14 -0700)]
Fix 'zfs list -t snapshot' depth

Commit df583073 introduced the ability to list the snapshots for a
specified dataset.  This change inadvertently resulted in only the top-
level snapshots being listed when no dataset was specified.  Fix this
issue by adding an additional check to determine if a dataset was
provided to avoid incorrectly restricting the depth.

Reviewed-by: George Melikov <mail@gmelikov.ru>
Reviewed-by: Tom Caputi <tcaputi@datto.com>
Reviewed-by: Alek Pinchuk <apinchuk@datto.com>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #8591
Closes #8594

5 years agoFix buffer length in strlcpy()
Brian Behlendorf [Mon, 8 Apr 2019 16:10:59 +0000 (09:10 -0700)]
Fix buffer length in strlcpy()

The length used for the strlcpy() used the size of zv_value
when it should have used the size of zc_name.  Correct this
typo.

Reviewed-by: George Melikov <mail@gmelikov.ru>
Reviewed-by: Jorgen Lundman <lundman@lundman.net>
Reviewed-by: Igor Kozhukhov <igor@dilos.org>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #8595
Closes #8596

5 years agoRevert "Fix issues with truncated files in raw sends"
Brian Behlendorf [Sat, 6 Apr 2019 00:32:56 +0000 (17:32 -0700)]
Revert "Fix issues with truncated files in raw sends"

This partially reverts commit 5dbf8b4ed.  This change resolved
the issues observed with truncated files in raw sends.  However,
the required changes to dnode_allocate() introduced a regression
for non-raw streams which needs to be understood.

The additional debugging improvements from the original patch
were not reverted.

Reviewed-by: Tom Caputi <tcaputi@datto.com>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Issue #7378
Issue #8528
Issue #8540
Issue #8565
Close #8584

5 years agopredictive prefetch disabled on new pools until export/reboot
Matthew Ahrens [Fri, 5 Apr 2019 17:12:02 +0000 (10:12 -0700)]
predictive prefetch disabled on new pools until export/reboot

When a pool is initially created (by `zpool create`), predictive
prefetch is inadvertently disabled, until the pool is export/import-ed,
or the machine is rebooted.

When device removal was introduced, we added some code to disable
predictive prefetching until indirect vdevs have been loaded.  This
resulted in the "default state" of prefetch being disabled, until we
proactively enable it after indirect vdevs are loaded.  Unfortunately
this resulted in a few bugs where in some code paths we neglect to
enable predictive prefetch.  The first of these was fixed by
https://github.com/zfsonlinux/zfs/commit/20507534d4ede14d4dd82c99fc8d461704ce7419

This commit fixes another case where we also need to explicitly enable
predictive prefetch, when the pool is initially created.

Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: Richard Elling <Richard.Elling@RichardElling.com>
Reviewed-by: George Melikov <mail@gmelikov.ru>
Signed-off-by: Matthew Ahrens <mahrens@delphix.com>
Closes #8577

5 years agofeatures.kernel layout should match features.pool
Don Brady [Fri, 5 Apr 2019 02:00:55 +0000 (20:00 -0600)]
features.kernel layout should match features.pool

The features.kernel layout should match features.pool.

Reviewed-by: Sara Hartse <sara.hartse@delphix.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Don Brady <don.brady@delphix.com>
Closes #8566

5 years agoRestrict kstats and print real pointers
Sara Hartse [Fri, 5 Apr 2019 01:57:06 +0000 (18:57 -0700)]
Restrict kstats and print real pointers

There are several places where we use zfs_dbgmsg and %p to
print pointers. In the Linux kernel, these values obfuscated
to prevent information leaks which means the pointers aren't
very useful for debugging crash dumps. We decided to restrict
the permissions of dbgmsg (and some other kstats while we were
at it) and print pointers with %px in zfs_dbgmsg as well as
spl_dumpstack

Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: John Gallagher <john.gallagher@delphix.com>
Signed-off-by: sara hartse <sara.hartse@delphix.com>
Closes #8467
Closes #8476

5 years agoHint about zpool free vs zfs available
Josh Soref [Thu, 4 Apr 2019 17:00:16 +0000 (13:00 -0400)]
Hint about zpool free vs zfs available

Also describe free/allocated/fragmentation

Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: Matt Ahrens <mahrens@delphix.com>
Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>
Closes #7565
Closes #8483

5 years agoFix txg_wait_open() load average inflation
Brian Behlendorf [Thu, 4 Apr 2019 16:44:46 +0000 (09:44 -0700)]
Fix txg_wait_open() load average inflation

Callers of txg_wait_open() which set should_quiesce=B_TRUE should be
accounted for as iowait time.  Otherwise, the caller is understood
to be idle and cv_wait_sig() is used to prevent incorrectly inflating
the system load average.

Similarly txg_wait_wait() has been updated to use cv_wait_io() to
be accounted against iowait.

Reviewed-by: Tim Chase <tim@chase2k.com>
Reviewed-by: Olaf Faaland <faaland1@llnl.gov>
Reviewed-by: Matt Ahrens <mahrens@delphix.com>
Reviewed-by: George Melikov <mail@gmelikov.ru>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #8550
Closes #8558

5 years agoMove dracut specifics to dracut module
Michael Niewöhner [Wed, 3 Apr 2019 00:14:39 +0000 (02:14 +0200)]
Move dracut specifics to dracut module

Dracut depends on the environment variable BOOTFS to be set after pool
import. This dracut specific systemd ExecStartPost command should not be
called for any non-dracut systems, so let's move it to a static systemd
unit that.

Reviewed-by: Manuel Amador (Rudd-O) <rudd-o@rudd-o.com>
Reviewed-by: Matthew Thode <prometheanfire@gentoo.org>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Michael Niewöhner <foss@mniewoehner.de>
Closes #8510

5 years agoFix man(1) warnings
Josh Soref [Tue, 2 Apr 2019 18:05:09 +0000 (14:05 -0400)]
Fix man(1) warnings

The macOS man app strenuously objects to blank lines in man files.

mdoc warning: Empty input line #xyz

Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: bunder2015 <omfgbunder@gmail.com>
Reviewed-by: George Melikov <mail@gmelikov.ru>
Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>
Closes #8559

5 years agoAppend snapshot name to "TIME SENT SNAPSHOT" output
TerraTech [Mon, 1 Apr 2019 19:25:17 +0000 (12:25 -0700)]
Append snapshot name to "TIME SENT SNAPSHOT" output

Simply appends zhp->zfs_name to the "TIME SENT SNAPSHOT" output.

Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: Matt Ahrens <mahrens@delphix.com>
Reviewed-by: Richard Elling <Richard.Elling@RichardElling.com>
Signed-off-by: TerraTech <TerraTech@users.noreply.github.com>
Closes #8543

5 years agoDo not iterate through filesystems unnecessarily
Tom Caputi [Mon, 1 Apr 2019 18:58:59 +0000 (14:58 -0400)]
Do not iterate through filesystems unnecessarily

Currently, when attempting to list snapshots ZFS may do a lot of
extra work checking child datasets. This is because the code does
not realize that it will not be able to reach any snapshots
contained within snapshots that are at the depth limit since the
snapshots of those datasets are counted as an additional layer
deeper. This patch corrects this issue.

In addition, this patch adds the ability to do perform the commands:

$ zfs list -t snapshot <dataset>
$ zfs get -t snapshot <prop> <dataset>

as a convenient way to list out properties of all snapshots of a
given dataset without having to use the depth limit.

Reviewed-by: Alek Pinchuk <apinchuk@datto.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: Richard Laager <rlaager@wiktel.com>
Reviewed-by: Matt Ahrens <mahrens@delphix.com>
Reviewed-by: George Melikov <mail@gmelikov.ru>
Signed-off-by: Tom Caputi <tcaputi@datto.com>
Closes #8539

5 years agoFix systemd-import services
Michael Niewöhner [Sun, 17 Mar 2019 21:47:15 +0000 (22:47 +0100)]
Fix systemd-import services

On debian, systemd complains about missing /bin/awk because it
actually is located at /usr/bin/awk. It is not a good idea to
hardcode binary paths because different linux distros use different
paths. According to systemd's man page it is absolutely safe to
miss paths for binaries located at standard locations (/bin,
/sbin, /usr/bin, ...).

Further, replace this more or less complicated awk command by
grep.

Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Michael Niewöhner <foss@mniewoehner.de>
Issue #8510

5 years agoRemove hard dependency on bash
Michael Niewöhner [Sun, 17 Mar 2019 21:44:57 +0000 (22:44 +0100)]
Remove hard dependency on bash

zfs-import-* services have a hard dependency on bash while not
everyone has bash installed. At this point /bin/sh is sufficient,
so use that.

Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Michael Niewöhner <foss@mniewoehner.de>
Issue #8510

5 years agoUpdate raw send documentation
Tom Caputi [Fri, 29 Mar 2019 22:05:55 +0000 (18:05 -0400)]
Update raw send documentation

This patch simply clarifies some of the limitations related to
raw sends in the man page. No functional changes.

Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: Jason Cohen <jwittlincohen@gmail.com>
Signed-off-by: Tom Caputi <tcaputi@datto.com>
Closes #8503
Closes #8544

5 years agoAdd TRIM support
Brian Behlendorf [Fri, 29 Mar 2019 16:13:20 +0000 (09:13 -0700)]
Add TRIM support

UNMAP/TRIM support is a frequently-requested feature to help
prevent performance from degrading on SSDs and on various other
SAN-like storage back-ends.  By issuing UNMAP/TRIM commands for
sectors which are no longer allocated the underlying device can
often more efficiently manage itself.

This TRIM implementation is modeled on the `zpool initialize`
feature which writes a pattern to all unallocated space in the
pool.  The new `zpool trim` command uses the same vdev_xlate()
code to calculate what sectors are unallocated, the same per-
vdev TRIM thread model and locking, and the same basic CLI for
a consistent user experience.  The core difference is that
instead of writing a pattern it will issue UNMAP/TRIM commands
for those extents.

The zio pipeline was updated to accommodate this by adding a new
ZIO_TYPE_TRIM type and associated spa taskq.  This new type makes
is straight forward to add the platform specific TRIM/UNMAP calls
to vdev_disk.c and vdev_file.c.  These new ZIO_TYPE_TRIM zios are
handled largely the same way as ZIO_TYPE_READs or ZIO_TYPE_WRITEs.
This makes it possible to largely avoid changing the pipieline,
one exception is that TRIM zio's may exceed the 16M block size
limit since they contain no data.

In addition to the manual `zpool trim` command, a background
automatic TRIM was added and is controlled by the 'autotrim'
property.  It relies on the exact same infrastructure as the
manual TRIM.  However, instead of relying on the extents in a
metaslab's ms_allocatable range tree, a ms_trim tree is kept
per metaslab.  When 'autotrim=on', ranges added back to the
ms_allocatable tree are also added to the ms_free tree.  The
ms_free tree is then periodically consumed by an autotrim
thread which systematically walks a top level vdev's metaslabs.

Since the automatic TRIM will skip ranges it considers too small
there is value in occasionally running a full `zpool trim`.  This
may occur when the freed blocks are small and not enough time
was allowed to aggregate them.  An automatic TRIM and a manual
`zpool trim` may be run concurrently, in which case the automatic
TRIM will yield to the manual TRIM.

Reviewed-by: Jorgen Lundman <lundman@lundman.net>
Reviewed-by: Tim Chase <tim@chase2k.com>
Reviewed-by: Matt Ahrens <mahrens@delphix.com>
Reviewed-by: George Wilson <george.wilson@delphix.com>
Reviewed-by: Serapheim Dimitropoulos <serapheim@delphix.com>
Contributions-by: Saso Kiselkov <saso.kiselkov@nexenta.com>
Contributions-by: Tim Chase <tim@chase2k.com>
Contributions-by: Chunwei Chen <tuxoko@gmail.com>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #8419
Closes #598

5 years agoSend stream should only list included snaps
Tom Caputi [Thu, 28 Mar 2019 22:48:58 +0000 (18:48 -0400)]
Send stream should only list included snaps

Currently, zfs send streams will include a list of all snapshots
on the source side if the '-p' option is provided. This can cause
performance problems on the receive side, especially if those
snapshots aren't present on the destination. These problems arise
because guid_to_name(), which is used for several receive side
functions, will search the entire receive-side pool if it can't
find a snapshot with a matching guid. This patch corrects the
issue by ensuring only streams that require this list of snapshots
include them.

Reviewed-by: Alek Pinchuk <apinchuk@datto.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: Matt Ahrens <mahrens@delphix.com>
Signed-off-by: Tom Caputi <tcaputi@datto.com>
Closes #8533

5 years agoFix issues with truncated files in raw sends
Tom Caputi [Wed, 27 Mar 2019 18:30:48 +0000 (14:30 -0400)]
Fix issues with truncated files in raw sends

This patch fixes a few issues with raw receives involving
truncated files:

* dnode_reallocate() now calls dnode_set_blksz() instead of
  dnode_setdblksz(). This ensures that any remaining dbufs with
  blkid 0 are resized along with their containing dnode upon
  reallocation.

* One of the calls to dmu_free_long_range() in receive_object()
  needs to check that the object it is about to free some contents
  or hasn't been completely removed already by a previous call to
  dmu_free_long_object() in the same function.

* The same call to dmu_free_long_range() in the previous point
  needs to ensure it uses the object's current block size and
  not the new block size. This ensures the blocks of the object
  that are supposed to be freed are completely removed and not
  simply partially zeroed out.

This patch also adds handling for DRR_OBJECT_RANGE records to
dprintf_drr() for debugging purposes.

Reviewed-by: Matt Ahrens <mahrens@delphix.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Tom Caputi <tcaputi@datto.com>
Closes #7378
Closes #8528

5 years agoUpdate valid vdev types for get_disklist
Richard Elling [Tue, 26 Mar 2019 21:18:58 +0000 (14:18 -0700)]
Update valid vdev types for get_disklist

Reviewed-by: George Melikov <mail@gmelikov.ru>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed by: John Kennedy <john.kennedy@delphix.com>
Signed-off-by: Richard Elling <Richard.Elling@RichardElling.com>
Closes #8532

5 years agoZTS: Detect e2fsprogs verity issue
Brian Behlendorf [Tue, 26 Mar 2019 20:57:40 +0000 (13:57 -0700)]
ZTS: Detect e2fsprogs verity issue

The projectid_001_pos and projecttree_001_pos test cases use the lsattr
command to detect that the project quota bit is set correctly.  Due to
a bug in e2fsprogs-1.44.4 setting the Project 'P' bit also results in
the Verity 'V' bit being reported as set.  This will result in the test
case failing.

The issue has been resolved in e2fsprogs but in order to avoid testing
failures these two test cases are skipped when e2fsprogs-1.44.4 is
installed.

https://github.com/tytso/e2fsprogs/commit/7e5a95e3d

Reviewed-by: Richard Elling <Richard.Elling@RichardElling.com>
Reviewed-by: George Melikov <mail@gmelikov.ru>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #8534

5 years agoCorrect a very minor grammar issue
Evan Allrich [Tue, 26 Mar 2019 19:27:29 +0000 (14:27 -0500)]
Correct a very minor grammar issue

Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: George Melikov <mail@gmelikov.ru>
Signed-off-by: Evan Allrich <evan@unguku.com>
Closes #8535

5 years agogit ignore python 3.7 virtual environment directories
Richard Elling [Mon, 25 Mar 2019 22:05:26 +0000 (15:05 -0700)]
git ignore python 3.7 virtual environment directories

Reviewed-by: George Melikov <mail@gmelikov.ru>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Richard Elling <Richard.Elling@RichardElling.com>
Closes #8525

5 years agoFix vd_path and error in spa_vdev_remove()
Igor K [Fri, 22 Mar 2019 20:25:07 +0000 (23:25 +0300)]
Fix vd_path and error in spa_vdev_remove()

Make a local copy of the vd_path and preserve the removal error
for use in spa_history_log_internal().  This is required because
after spa_vdev_exit() there is nothing preventing the vdev state
from changing.

Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: Tom Caputi <tcaputi@datto.com>
Signed-off-by: Igor Kozhukhov <igor@dilos.org>
Closes #8522

5 years agoPanic when running 'zpool split'
Roman Strashkin [Fri, 22 Mar 2019 20:11:36 +0000 (23:11 +0300)]
Panic when running 'zpool split'

Added missing remove of detachable VDEV from txg's DTL list
to avoid use-after-free for the split VDEV

Reviewed by: Pavel Zakharov <pavel.zakharov@delphix.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: Jorgen Lundman <lundman@lundman.net>
Signed-off-by: Roman Strashkin <roman.strashkin@nexenta.com>
Closes #5565
Closes #7856

5 years agoZFS Reads may result in unneccesary calls to zil_commit
George Wilson [Fri, 22 Mar 2019 20:09:11 +0000 (16:09 -0400)]
ZFS Reads may result in unneccesary calls to zil_commit

ZFS supports O_RSYNC for read operations and when specified will ensure
the same level of data integrity that O_DSYNC and O_SYNC provides for
writes. O_RSYNC by itself has no effect so it must be combined with
either O_DSYNC or O_SYNC. However, many platforms don't support O_RSYNC
and have mapped O_SYNC to mean O_RSYNC within ZFS. This is incorrect
and causes unnecessary calls to zil_commit. Only platforms which
support O_RSYNC should implement the zil_commit functionality in the
read code path.

Reviewed-by: Matt Ahrens <mahrens@delphix.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: George Wilson <george.wilson@delphix.com>
Closes #8523

5 years agoMMP interval and fail_intervals in uberblock
Olaf Faaland [Thu, 21 Mar 2019 19:47:57 +0000 (12:47 -0700)]
MMP interval and fail_intervals in uberblock

When Multihost is enabled, and a pool is imported, uberblock writes
include ub_mmp_delay to allow an importing node to calculate the
duration of an activity test.  This value, is not enough information.

If zfs_multihost_fail_intervals > 0 on the node with the pool imported,
the safe minimum duration of the activity test is well defined, but does
not depend on ub_mmp_delay:

zfs_multihost_fail_intervals * zfs_multihost_interval

and if zfs_multihost_fail_intervals == 0 on that node, there is no such
well defined safe duration, but the importing host cannot tell whether
mmp_delay is high due to I/O delays, or due to a very large
zfs_multihost_interval setting on the host which last imported the pool.
As a result, it may use a far longer period for the activity test than
is necessary.

This patch renames ub_mmp_sequence to ub_mmp_config and uses it to
record the zfs_multihost_interval and zfs_multihost_fail_intervals
values, as well as the mmp sequence.  This allows a shorter activity
test duration to be calculated by the importing host in most situations.
These values are also added to the multihost_history kstat records.

It calculates the activity test duration differently depending on
whether the new fields are present or not; for importing pools with
only ub_mmp_delay, it uses

(zfs_multihost_interval + ub_mmp_delay) * zfs_multihost_import_intervals

Which results in an activity test duration less sensitive to the leaf
count.

In addition, it makes a few other improvements:
* It updates the "sequence" part of ub_mmp_config when MMP writes
  in between syncs occur.  This allows an importing host to detect MMP
  on the remote host sooner, when the pool is idle, as it is not limited
  to the granularity of ub_timestamp (1 second).
* It issues writes immediately when zfs_multihost_interval is changed
  so remote hosts see the updated value as soon as possible.
* It fixes a bug where setting zfs_multihost_fail_intervals = 1 results
  in immediate pool suspension.
* Update tests to verify activity check duration is based on recorded
  tunable values, not tunable values on importing host.
* Update tests to verify the expected number of uberblocks have valid
  MMP fields - fail_intervals, mmp_interval, mmp_seq (sequence number),
  that sequence number is incrementing, and that uberblock values match
  tunable settings.

Reviewed-by: Andreas Dilger <andreas.dilger@whamcloud.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: Tony Hutter <hutter2@llnl.gov>
Signed-off-by: Olaf Faaland <faaland1@llnl.gov>
Closes #7842