]> granicus.if.org Git - zfs/log
zfs
7 years agoFix column alignment with long zpool names
George G [Sun, 5 Nov 2017 21:09:56 +0000 (21:09 +0000)]
Fix column alignment with long zpool names

`zpool status` normally aligns NAME/STATE/etc columns:

    NAME                       STATE     READ WRITE CKSUM
    dummy                      ONLINE       0     0     0
      mirror-0                 ONLINE       0     0     0
        /tmp/dummy-long-1.bin  ONLINE       0     0     0
        /tmp/dummy-long-2.bin  ONLINE       0     0     0
      mirror-1                 ONLINE       0     0     0
        /tmp/dummy-long-3.bin  ONLINE       0     0     0
        /tmp/dummy-long-4.bin  ONLINE       0     0     0

However, if the zpool name is longer than the zvol names, alignment
issues arise:

    NAME                  STATE     READ WRITE CKSUM
    dummy-very-very-long-zpool-name  ONLINE       0     0     0
      mirror-0            ONLINE       0     0     0
        /tmp/dummy-1.bin  ONLINE       0     0     0
        /tmp/dummy-2.bin  ONLINE       0     0     0
      mirror-1            ONLINE       0     0     0
        /tmp/dummy-3.bin  ONLINE       0     0     0
        /tmp/dummy-4.bin  ONLINE       0     0     0

`zpool iostat` and `zpool import` are also affected:

                  capacity     operations     bandwidth
    pool        alloc   free   read  write   read  write
    ----------  -----  -----  -----  -----  -----  -----
    dummy        104K  1.97G      0      0    152  9.84K
    dummy-very-very-long-zpool-name   152K  1.97G      0      1    144  13.1K
    ----------  -----  -----  -----  -----  -----  -----

    dummy-very-very-long-zpool-name  ONLINE
      mirror-0            ONLINE
        /tmp/dummy-1.bin  ONLINE
        /tmp/dummy-2.bin  ONLINE
      mirror-1            ONLINE
        /tmp/dummy-3.bin  ONLINE
        /tmp/dummy-4.bin  ONLINE

Reviewed-by: George Melikov <mail@gmelikov.ru>
Reviewed-by: Tony Hutter <hutter2@llnl.gov>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: George Gaydarov <git@gg7.io>
Closes #6786

7 years agoRewrite fHits() in arc_summary.py with SI units
Scot W. Stevenson [Sat, 4 Nov 2017 20:33:28 +0000 (21:33 +0100)]
Rewrite fHits() in arc_summary.py with SI units

Complete rewrite of fHits(). Move units from non-standard English
abbreviations to SI units, thereby avoiding confusion because of
"long scale" and "short scale" numbers. Remove unused parameter
"Decimal". Add function string. Aim to confirm to PEP8.

Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: George Melikov <mail@gmelikov.ru>
Reviewed-by: Giuseppe Di Natale <dinatale2@llnl.gov>
Signed-off-by: Scot W. Stevenson <scot.stevenson@gmail.com>
Closes #6815

7 years agoUndo c89 workarounds to match with upstream
Don Brady [Sat, 4 Nov 2017 20:25:13 +0000 (14:25 -0600)]
Undo c89 workarounds to match with upstream

With PR 5756 the zfs module now supports c99 and the
remaining past c89 workarounds can be undone.

Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: George Melikov <mail@gmelikov.ru>
Signed-off-by: Don Brady <don.brady@delphix.com>
Closes #6816

7 years agoMinor code cleanup in arc_summary.py
Scot W. Stevenson [Fri, 3 Nov 2017 22:43:53 +0000 (23:43 +0100)]
Minor code cleanup in arc_summary.py

Simplify and inline single-use function div1(); inline twice-used
function div2(); add function comment to zfs_header(); replace
variable "unused" in get_Kstat() with "_" following convention.

Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: George Melikov <mail@gmelikov.ru>
Reviewed-by: Giuseppe Di Natale <dinatale2@llnl.gov>
Signed-off-by: Scot W. Stevenson <scot.stevenson@gmail.com>
Closes #6802

7 years agoInitramfs fixes
Brian Behlendorf [Fri, 3 Nov 2017 22:38:52 +0000 (15:38 -0700)]
Initramfs fixes

* initramfs: Fix inconsistent whitespace
* initramfs: Fix a spelling error
* initramfs: Set elevator=noop on the rpool's disks

Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: George Melikov <mail@gmelikov.ru>
Signed-off-by: Richard Laager <rlaager@wiktel.com>
Closes #6807

7 years agoAllow test-runner to filter test groups by tag
Giuseppe Di Natale [Fri, 3 Nov 2017 16:53:32 +0000 (09:53 -0700)]
Allow test-runner to filter test groups by tag

Enable test-runner to accept a list of tags to identify
which test groups the user wishes to run.

Also allow test-runner to perform multiple iterations
of a test run.

Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: John Wren Kennedy <john.kennedy@delphix.com>
Reviewed-by: George Melikov <mail@gmelikov.ru>
Signed-off-by: Giuseppe Di Natale <dinatale2@llnl.gov>
Closes #6788

7 years agoinitramfs: Set elevator=noop on the rpool's disks
Richard Laager [Thu, 2 Nov 2017 02:54:56 +0000 (21:54 -0500)]
initramfs: Set elevator=noop on the rpool's disks

ZFS already sets elevator=noop for wholedisk vdevs (for all pools), but
typical root-on-ZFS installations use partitions.  This sets
elevator=noop on the disks in the root pool.

Ubuntu 16.04 and 16.10 had this.  It was lost in 17.04 due to Debian
switching to this upstream initramfs script.

Signed-off-by: Richard Laager <rlaager@wiktel.com>
7 years agoinitramfs: Fix a spelling error
Richard Laager [Thu, 2 Nov 2017 02:54:28 +0000 (21:54 -0500)]
initramfs: Fix a spelling error

This fixes a typo in a comment.

Signed-off-by: Richard Laager <rlaager@wiktel.com>
7 years agoinitramfs: Fix inconsistent whitespace
Richard Laager [Thu, 2 Nov 2017 02:53:22 +0000 (21:53 -0500)]
initramfs: Fix inconsistent whitespace

This fixes one instance of inconsistent whitespace.

Signed-off-by: Richard Laager <rlaager@wiktel.com>
7 years agoAdd scan.coverity.com badge to README
Brian Behlendorf [Mon, 30 Oct 2017 23:21:24 +0000 (16:21 -0700)]
Add scan.coverity.com badge to README

Include the scan.coverity.com status badge in the top level README.

Reviewed-by: Giuseppe Di Natale <dinatale2@llnl.gov>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #6801

7 years agoOpenZFS 640 - number_to_scaled_string is duplicated in several commands
Jason King [Tue, 13 Jun 2017 09:16:45 +0000 (04:16 -0500)]
OpenZFS 640 - number_to_scaled_string is duplicated in several commands

Porting Notes:
- The OpenZFS patch added nicenum_scale() and nicenum() to a
  library not used by ZFS.  Rather than pull in a new dependency
  the version of nicenum in lib/libzpool/util.c was simply
  replaced with the new one.

Reviewed by: Sebastian Wiedenroth <wiedi@frubar.net>
Reviewed by: Robert Mustacchi <rm@joyent.com>
Reviewed by: Yuri Pankov <yuripv@gmx.com>
Approved by: Dan McDonald <danmcd@joyent.com>
Authored by: Jason King <jason.brian.king@gmail.com>
Ported-by: Brian Behlendorf <behlendorf1@llnl.gov>
OpenZFS-issue: https://www.illumos.org/issues/640
OpenZFS-commit: https://github.com/openzfs/openzfs/commit/0a055120
Closes #6796

7 years agoRewrite of function fBytes() in arc_summary.py
Scot W. Stevenson [Wed, 25 Oct 2017 06:29:02 +0000 (08:29 +0200)]
Rewrite of function fBytes() in arc_summary.py

Replace if-elif-else construction with shorter loop;
remove unused parameter "Decimal"; centralize format
string; add function documentation string; conform to
PEP8.

Reviewed-by: Giuseppe Di Natale <dinatale2@llnl.gov>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Scot W. Stevenson <scot.stevenson@gmail.com>
Closes #6784

7 years agosystemd zfs-import.target and documentation
Antonio Russo [Mon, 30 Oct 2017 20:18:26 +0000 (16:18 -0400)]
systemd zfs-import.target and documentation

zfs-import-{cache,scan}.service must complete before any mounting of
filesystems can occur. To simplify this dependency, create a target
that is reached After (in the systemd sense) the pool is imported.

Additionally, recommend that legacy zfs mounts use the option

x-systemd.requires=zfs-import.target

to codify this requirement.

Reviewed-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: Giuseppe Di Natale <dinatale2@llnl.gov>
Signed-off-by: Antonio Russo <antonio.e.russo@gmail.com>
Closes #6764

7 years agoUpdate zfs module parameters man5
abraunegg [Mon, 30 Oct 2017 20:15:10 +0000 (07:15 +1100)]
Update zfs module parameters man5

Update zfs module parameters man5 with missing parameter details
for multiple tunings.

Reviewed-by: George Melikov <mail@gmelikov.ru>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: Giuseppe Di Natale <dinatale2@llnl.gov>
Signed-off-by: Alex Braunegg <alex.braunegg@gmail.com>
Closes #6785

7 years agoFix status command options in zpool(8)
Brian Behlendorf [Fri, 27 Oct 2017 22:52:03 +0000 (15:52 -0700)]
Fix status command options in zpool(8)

The 'zpool status' command supports the -P option for printing full
path names.  It does not support the -p parsable option for printing
exact values.

Reviewed-by: George Melikov <mail@gmelikov.ru>
Reviewed-by: loli10K <ezomori.nozomu@gmail.com>
Reviewed-by: Giuseppe Di Natale <dinatale2@llnl.gov>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #6792
Closes #6794

7 years agoOpenZFS 8081 - Compiler warnings in zdb
Brian Behlendorf [Fri, 27 Oct 2017 19:46:35 +0000 (12:46 -0700)]
OpenZFS 8081 - Compiler warnings in zdb

Fix compiler warnings in zdb.  With these changes, FreeBSD can compile
zdb with all compiler warnings enabled save -Wunused-parameter.

usr/src/cmd/zdb/zdb.c
usr/src/cmd/zdb/zdb_il.c
usr/src/uts/common/fs/zfs/sys/sa.h
usr/src/uts/common/fs/zfs/sys/spa.h
Fix numerous warnings, including:
* const-correctness
* shadowing global definitions
* signed vs unsigned comparisons
* missing prototypes, or missing static declarations
* unused variables and functions
* Unreadable array initializations
* Missing struct initializers

usr/src/cmd/zdb/zdb.h
Add a header file to declare common symbols

usr/src/lib/libzpool/common/sys/zfs_context.h
usr/src/uts/common/fs/zfs/arc.c
usr/src/uts/common/fs/zfs/dbuf.c
usr/src/uts/common/fs/zfs/spa.c
usr/src/uts/common/fs/zfs/txg.c
Add a function prototype for zk_thread_create, and ensure that every
callback supplied to this function actually matches the prototype.

usr/src/cmd/ztest/ztest.c
usr/src/uts/common/fs/zfs/sys/zil.h
usr/src/uts/common/fs/zfs/zfs_replay.c
usr/src/uts/common/fs/zfs/zvol.c
Add a function prototype for zil_replay_func_t, and ensure that
every function of this type actually matches the prototype.

usr/src/uts/common/fs/zfs/sys/refcount.h
Change FTAG so it discards any constness of __func__, necessary
since existing APIs expect it passed as void *.

Porting Notes:
- Many of these fixes have already been applied to Linux.  For
  consistency the OpenZFS version of a change was applied if the
  warning was addressed in an equivalent but different fashion.

Reviewed by: Matthew Ahrens <mahrens@delphix.com>
Reviewed by: Prakash Surya <prakash.surya@delphix.com>
Authored by: Alan Somers <asomers@gmail.com>
Approved by: Richard Lowe <richlowe@richlowe.net>
Ported-by: Brian Behlendorf <behlendorf1@llnl.gov>
OpenZFS-issue: https://www.illumos.org/issues/8081
OpenZFS-commit: https://github.com/openzfs/openzfs/commit/843abe1b8a
Closes #6787

7 years agoCorrect make mancheck recipe
Giuseppe Di Natale [Fri, 27 Oct 2017 16:52:18 +0000 (09:52 -0700)]
Correct make mancheck recipe

The current make recipe for mancheck silently ignores errors. Correct
the recipe so errors cause the mancheck recipe fail.

The zpool reopen command in the zpool.8 manpage had a bullet list
without an .El.

Reviewed-by: George Melikov <mail@gmelikov.ru>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: loli10K <ezomori.nozomu@gmail.com>
Signed-off-by: Giuseppe Di Natale <dinatale2@llnl.gov>
Closes #6790

7 years agoZFS send fails to dump objects larger than 128PiB
LOLi [Thu, 26 Oct 2017 23:58:38 +0000 (01:58 +0200)]
ZFS send fails to dump objects larger than 128PiB

When dumping objects larger than 128PiB it's possible for do_dump() to
miscalculate the FREE_RECORD offset due to an integer overflow
condition: this prevents the receiving end from correctly restoring
the dumped object.

Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
Signed-off-by: loli10K <ezomori.nozomu@gmail.com>
Closes #6760

7 years agoAllow 'zpool events' filtering by pool name
LOLi [Thu, 26 Oct 2017 23:49:33 +0000 (01:49 +0200)]
Allow 'zpool events' filtering by pool name

Additionally add four new tests:

 * zpool_events_clear: verify 'zpool events -c' functionality
 * zpool_events_cliargs: verify command line options and arguments
 * zpool_events_follow: verify 'zpool events -f'
 * zpool_events_poolname: verify events filtering by pool name

Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: Giuseppe Di Natale <dinatale2@llnl.gov>
Signed-off-by: loli10K <ezomori.nozomu@gmail.com>
Closes #3285
Closes #6762

7 years agoOpenZFS 8558, 8602 - lwp_create() returns EAGAIN
Brian Behlendorf [Thu, 26 Oct 2017 19:57:53 +0000 (12:57 -0700)]
OpenZFS 8558, 8602 - lwp_create() returns EAGAIN

8558 lwp_create() returns EAGAIN on system with more than 80K ZFS filesystems

On a system with more than 80K ZFS filesystems, we've seen cases
where lwp_create() will start to fail by returning EAGAIN. The
problem being, for each of those 80K ZFS filesystems, a taskq will
be created for each dataset as part of the ZIL for each dataset.

Porting Notes:
- The new nomem taskq kstat was dropped.
- Added module options and documentation for new tunings
  zfs_zil_clean_taskq_nthr_pct, zfs_zil_clean_taskq_minalloc,
  zfs_zil_clean_taskq_maxalloc, and zfs_sync_taskq_batch_pct.

Reviewed by: George Wilson <george.wilson@delphix.com>
Reviewed by: Sebastien Roy <sebastien.roy@delphix.com>
Approved by: Robert Mustacchi <rm@joyent.com>
Authored by: Prakash Surya <prakash.surya@delphix.com>
Reviewed-by: George Melikov <mail@gmelikov.ru>
Reviewed-by: Chris Dunlop <chris@onthe.net.au>
Ported-by: Brian Behlendorf <behlendorf1@llnl.gov>
OpenZFS-issue: https://www.illumos.org/issues/8558
OpenZFS-commit: https://github.com/openzfs/openzfs/commit/216d772

8602 remove unused "dp_early_sync_tasks" field from "dsl_pool" structure

Reviewed by: Serapheim Dimitropoulos <serapheim@delphix.com>
Reviewed by: Matthew Ahrens <mahrens@delphix.com>
Approved by: Robert Mustacchi <rm@joyent.com>
Authored by: Prakash Surya <prakash.surya@delphix.com>
Reviewed-by: George Melikov <mail@gmelikov.ru>
Reviewed-by: Chris Dunlop <chris@onthe.net.au>
Ported-by: Brian Behlendorf <behlendorf1@llnl.gov>
OpenZFS-issue: https://www.illumos.org/issues/8602
OpenZFS-commit: https://github.com/openzfs/openzfs/commit/2bcb545
Closes #6779

7 years agoAdded no_scrub_restart flag to zpool reopen
Arkadiusz Bubała [Thu, 26 Oct 2017 19:26:09 +0000 (21:26 +0200)]
Added no_scrub_restart flag to zpool reopen

Added -n flag to zpool reopen that allows a running scrub
operation to continue if there is a device with Dirty Time Log.

By default if a component device has a DTL and zpool reopen
is executed all running scan operations will be restarted.

Added functional tests for `zpool reopen`

Tests covers following scenarios:
* `zpool reopen` without arguments,
* `zpool reopen` with pool name as argument,
* `zpool reopen` while scrubbing,
* `zpool reopen -n` while scrubbing,
* `zpool reopen -n` while resilvering,
* `zpool reopen` with bad arguments.

Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: Tom Caputi <tcaputi@datto.com>
Signed-off-by: Arkadiusz Bubała <arkadiusz.bubala@open-e.com>
Closes #6076
Closes #6746

7 years agoarcstat: flush stdout / outfile after each line
Fabian-Gruenbichler [Thu, 26 Oct 2017 19:18:49 +0000 (21:18 +0200)]
arcstat: flush stdout / outfile after each line

Otherwise, if arcstat gets interrupted before the desired number of
iterations is reached, the output file will be empty (both if set via
'-o' or via shell redirection).

Reviewed-by: George Melikov <mail@gmelikov.ru>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: Giuseppe Di Natale <dinatale2@llnl.gov>
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
Closes #6775

7 years agocommitcheck: Multiple OpenZFS ports in commit
Giuseppe Di Natale [Thu, 26 Oct 2017 17:23:58 +0000 (10:23 -0700)]
commitcheck: Multiple OpenZFS ports in commit

Allow commitcheck.sh to handle multiple OpenZFS ports in
a single commit. This is useful in the cases when a change
upstream has bug fixes and it makes sense to port them with
the original patch.

Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: George Melikov <mail@gmelikov.ru>
Reviewed-by: Chris Dunlop <chris@onthe.net.au>
Signed-off-by: Giuseppe Di Natale <dinatale2@llnl.gov>
Closes #6780

7 years agoAdd Coverity defect fix commit checker support
Giuseppe Di Natale [Thu, 26 Oct 2017 17:17:00 +0000 (10:17 -0700)]
Add Coverity defect fix commit checker support

Enable commitcheck.sh to test if a commit message is
in the expected format for a coverity defect fix.

Reviewed-by: George Melikov <mail@gmelikov.ru>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Giuseppe Di Natale <dinatale2@llnl.gov>
Closes #6777

7 years agoEnsure arc_size_break is filled in arc_summary.py
Giuseppe Di Natale [Mon, 23 Oct 2017 21:18:12 +0000 (14:18 -0700)]
Ensure arc_size_break is filled in arc_summary.py

Use mfu_size and mru_size pulled from the arcstats
kstat file to calculate the mfu and mru percentages
for arc size breakdown.

Reviewed-by: George Melikov <mail@gmelikov.ru>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: Richard Elling <Richard.Elling@RichardElling.com>
Reviewed-by: AndCycle <andcycle@andcycle.idv.tw>
Signed-off-by: Giuseppe Di Natale <dinatale2@llnl.gov>
Closes #5526
Closes #6770

7 years agoCorrect flake8 errors after STYLE builder update
Giuseppe Di Natale [Mon, 23 Oct 2017 21:01:43 +0000 (14:01 -0700)]
Correct flake8 errors after STYLE builder update

Fix new flake8 errors related to bare excepts and ambiguous
variable names due to a STYLE builder update.

Reviewed-by: George Melikov <mail@gmelikov.ru>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Giuseppe Di Natale <dinatale2@llnl.gov>
Closes #6776

7 years agoZTS: Add auto-spare tests
David Quigley [Mon, 23 Oct 2017 18:42:37 +0000 (12:42 -0600)]
ZTS: Add auto-spare tests

The ZED is expected to automatically kick in a hot spare device
when there's one available in the pool and a sufficient number of
read errors have been encountered.  Use zinject to simulate the
failure condition and verify the hot spare is used.

auto_spare_001_pos.ksh: read IO errors, the vdev is FAULTED
auto_spare_002_pos.ksh: read CHECKSUM errors, the vdev is DEGRADE

Reviewed by: Richard Elling <Richard.Elling@RichardElling.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: David Quigley <david.quigley@intel.com>
Closes #6280

7 years agoUse ashift=12 by default on SSDSC2BW48 disks
adisbladis [Mon, 23 Oct 2017 18:00:45 +0000 (02:00 +0800)]
Use ashift=12 by default on SSDSC2BW48 disks

Currently the 480GB models of this disk do not use ashift=12 by
default.  SSDSC2BW48 is also optimized for 4k blocks.

Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: George Melikov <mail@gmelikov.ru>
Signed-off-by: adisbladis <adis@blad.is>
Closes #6774

7 years agoProvide commit message format for Coverity defects
Giuseppe Di Natale [Mon, 23 Oct 2017 16:47:16 +0000 (09:47 -0700)]
Provide commit message format for Coverity defects

Provide details about the commit message format for Coverity defect
fixes submitted.

Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: George Melikov <mail@gmelikov.ru>
Signed-off-by: Giuseppe Di Natale <dinatale2@llnl.gov>
Closes #6771

7 years agoEmit history events for 'zpool create'
Brian Behlendorf [Mon, 23 Oct 2017 16:45:59 +0000 (09:45 -0700)]
Emit history events for 'zpool create'

History commands and events were being suppressed for the
'zpool create' command since the history object did not
yet exist.  Create the object earlier so this history
doesn't get lost.

Split the pool_destroy event in to pool_destroy and
pool_export so they may be distinguished.

Updated events_001_pos and events_002_pos test cases.  They
now check for the expected history events and were reworked
to be more reliable.

Reviewed-by: Nathaniel Clark <nathaniel.l.clark@intel.com>
Reviewed-by: Giuseppe Di Natale <dinatale2@llnl.gov>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #6712
Closes #6486

7 years agoSupport integration with new QAT products
wli5 [Fri, 20 Oct 2017 18:11:25 +0000 (02:11 +0800)]
Support integration with new QAT products

Support integration with new QAT products: Intel(R) C62x Chipset,
or Atom(R) C3000 Processor Product Family SoC:
1. Detect new file name in auto-conf.
2. Change MAX_INSTANCES to 48.
3. Change "num_inst" to U16 to clean a build warning.

Reviewed-by: George Melikov <mail@gmelikov.ru>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Weigang Li <weigang.li@intel.com>
Closes #6767

7 years agoAdd convenience 'zfs_get' functions
John [Thu, 19 Oct 2017 18:18:42 +0000 (11:18 -0700)]
Add convenience 'zfs_get' functions

Add get functions to match existing ones.

Reviewed-by: Giuseppe Di Natale <dinatale2@llnl.gov>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: John Ramsden <johnramsden@riseup.net>
Closes #6308

7 years agoRemove vn_rename and vn_remove dependency
Brian Behlendorf [Thu, 19 Oct 2017 17:06:55 +0000 (10:06 -0700)]
Remove vn_rename and vn_remove dependency

The only place vn_rename and vn_remove are used is when writing
out an updated pool configuration file.  By truncating the file
instead of renaming and removing it we can avoid having to implement
these interfaces entirely.  Functionally an empty cache file is
treated the same as a missing cache file.  This is particularly
advantageous because the Linux kernel has never provided a way
to reliably implement vn_rename and vn_remove.

The cachefile_004_pos.ksh test case was updated to understand
that an empty cache file is the same as a missing one.

The zfs-import-* systemd service files were not updated to use
ConditionFileNotEmpty in place of ConditionPathExists.  This
means that after exporting all pools and rebooting new pools
will not the scanned for on the next boot.  This small change
should not impact normal usage since pools are not exported
as part of a normal shutdown.

Documentation was updated accordingly.

Reviewed-by: George Melikov <mail@gmelikov.ru>
Reviewed-by: Arkadiusz Bubała <arkadiusz.bubala@open-e.com>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes zfsonlinux/spl#648
Closes #6753

7 years agoFix ASSERT in dmu_free_long_object_raw()
Tom Caputi [Wed, 18 Oct 2017 17:08:36 +0000 (13:08 -0400)]
Fix ASSERT in dmu_free_long_object_raw()

This small patch fixes an issue where dmu_free_long_object_raw()
calls dnode_hold() after freeing the dnode a line above.

Reviewed-by: Jorgen Lundman <lundman@lundman.net>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Tom Caputi <tcaputi@datto.com>
Closes #6766

7 years agoUpdate codecov.io behavior
Brian Behlendorf [Wed, 18 Oct 2017 17:07:02 +0000 (10:07 -0700)]
Update codecov.io behavior

Update the codecov.yml included in the repository to behave as
originally intended.  This can be refined as needed.

* Always post coverage results to the GitHub PR after two builds
  have been uploaded.  This is the normal case since there will
  be a build uploaded for both kernel and user coverage results.

* Adjust red -> yellow -> green coloring in the web interface.
  Due to the number of unlikely error conditions which are hard
  to force consider 90% coverage an excellent level of coverage.

* Allow a 1% variance in coverage between test runs.  This is
  approximately 10x larger than the typical variance observed
  which leaves us a reasonable margin to prevent false positives.

* Always post a new smaller comment to PRs which does not include
  a file list.  Old coverage reports are removed.

Reviewed by: Prakash Surya <prakash.surya@delphix.com>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #6765

7 years agoFix coverity defects: CID 161388
Tobin Harding [Tue, 17 Oct 2017 16:37:50 +0000 (03:37 +1100)]
Fix coverity defects: CID 161388

CID 161388: Resource Leak (REASOURCE_LEAK)

Jump to errout so that file descriptor gets closed before returning
from function.

Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Tobin C. Harding <me@tobin.cc>
Closes #6755

7 years agoFix coverity defects: 147480, 147584
Tobin Harding [Mon, 16 Oct 2017 22:32:48 +0000 (09:32 +1100)]
Fix coverity defects: 147480, 147584

CID 147480: Logically dead code (DEADCODE)

Remove non-null check and subsequent function call. Add ASSERT to future
proof the code.

usage label is only jumped to before `zhp` is initialized.

CID 147584: Out-of-bounds access (OVERRUN)

Subtract length of current string from buffer length for `size` argument
to `snprintf`.

Starting address for the write is the start of the buffer + the current
string length. We need to subtract this string length else risk a buffer
overflow.

Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Tobin C. Harding <me@tobin.cc>
Closes #6745

7 years agoAdd DKMS package on Debian-based distributions
Neal Gompa (ニール・ゴンパ) [Sun, 15 Oct 2017 20:00:44 +0000 (16:00 -0400)]
Add DKMS package on Debian-based distributions

* config/deb.am: Enable building DKMS packages for Debian
* rpm/generic/zfs-dkms.spec.in: Adjust spec to be Debian-compatible
  * Condition kernel-devel Req to RPM distros
  * Adjust the DKMS Req to have a minimum of a version only
  * Ensure that --rpm_safe_upgrade isn't used on non-RPM distros
* config/deb.am: Drop CONFIG_KERNEL and CONFIG_USER guards
* Makefile.am: Add pkg-dkms target

Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Neal Gompa <ngompa@datto.com>
Closes #6044
Closes #6731

7 years agoFix function documentation to correctly mirror code
Tobin Harding [Fri, 13 Oct 2017 19:42:04 +0000 (06:42 +1100)]
Fix function documentation to correctly mirror code

Currently the function documentation states that two strings are
allocated, this is outdated. Only one char ** parameter is passed
into the function now, clearly only a pointer to a single string
is returned and needs to be free'd.

Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Tobin C. Harding <me@tobin.cc>
Closes #6754

7 years agoIncrease default zloop.sh vdev size
Brian Behlendorf [Fri, 13 Oct 2017 19:39:39 +0000 (12:39 -0700)]
Increase default zloop.sh vdev size

The default 128M vdev size used by zloop.sh isn't always large
enough and can result in ENOSPC failures which suspend the pool.
Increase the default size to 512M and provide a -s option which
can be used to specify an alternate size.

This does increase the free space requirements to run zloop.sh.
However, since the vdevs are sparse 4x the space is not required.

Reviewed-by: Don Brady <don.brady@delphix.com>
Reviewed-by: George Melikov <mail@gmelikov.ru>
Reviewed-by: Giuseppe Di Natale <dinatale2@llnl.gov>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #6758

7 years agoPost-Encryption Followup
Brian Behlendorf [Fri, 13 Oct 2017 17:02:39 +0000 (10:02 -0700)]
Post-Encryption Followup

This PR includes fixes for bugs and documentation issues found
after the encryption patch was merged and general code improvements
for long-term maintainability.

Reviewed-by: Jorgen Lundman <lundman@lundman.net>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Tom Caputi <tcaputi@datto.com>
Issue #6526
Closes #6639
Closes #6703
Cloese #6706
Closes #6714
Closes #6595

7 years agoTypo in dsl_dataset.h
Damian Wojsław [Fri, 13 Oct 2017 00:10:38 +0000 (02:10 +0200)]
Typo in dsl_dataset.h

The parameters dsl_dataset_t *os in function prototype should be
renamed to dsl_dataset_t *ds.

Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Damian Wojsław <damian@wojslaw.pl>
Closes #6756
Closes #6273

7 years agoFixes for SPARC support
Brian Behlendorf [Thu, 12 Oct 2017 16:51:56 +0000 (09:51 -0700)]
Fixes for SPARC support

The current code base almost compiles on SPARC, but a few fixes are
required for the code to compile (and work efficiently). Code in this
PR comes from OpenZFS project which was initially dropped when porting
the crypto framework.

Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Pengcheng Xu <i@jsteward.moe>
Closes #6733
Closes #6738
Closes #6750

7 years agoExplicitly depend on icp module in initramfs hook
Antonio Russo [Thu, 12 Oct 2017 16:39:45 +0000 (12:39 -0400)]
Explicitly depend on icp module in initramfs hook

Automatic dependency resolution is unreliable on many systems.
Follow suit with existing code, and explicitly include icp
in module dependencies.

Reviewed-by: George Melikov <mail@gmelikov.ru>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Antonio Russo <antonio.e.russo@gmail.com>
Closes #6751

7 years agoFix for #6714
Tom Caputi [Thu, 5 Oct 2017 17:43:34 +0000 (13:43 -0400)]
Fix for #6714

This 2 line patch fixes a possible integer overflow reported by grsec.

Signed-off-by: Tom Caputi <tcaputi@datto.com>
7 years agoFix for #6706
Tom Caputi [Tue, 3 Oct 2017 17:18:45 +0000 (13:18 -0400)]
Fix for #6706

This patch resolves an issue where raw sends would fail to send
encryption parameters if the wrapping key was unloaded and reloaded
before the data was sent and the dataset wass not an encryption root.
The code attempted to lookup the values from the wrapping key which
was not being initialized upon reload. This change forces the code to
lookup the correct value from the encryption root's DSL Crypto Key.
Unfortunately, this issue led to the on-disk DSL Crypto Key for some
non-encryption root datasets being left with zeroed out encryption
parameters. However, this should not present a problem since these
values are never looked at and are overrwritten upon changing keys.

This patch also fixes an issue where raw, resumable sends were not
being cleaned up appropriately if an invalid DSL Crypto Key was
received.

Signed-off-by: Tom Caputi <tcaputi@datto.com>
7 years agoFix for #6703
Tom Caputi [Tue, 3 Oct 2017 01:55:39 +0000 (21:55 -0400)]
Fix for #6703

This patch resolves an issue where spa_keystore_change_key_sync_impl()
incorrectly recursed into clone DSL Directories while recursively
rewrapping encryption keys. Clones share keys with their origins, so
this logic was incorrect.

Signed-off-by: Tom Caputi <tcaputi@datto.com>
7 years agoFixes for #6639
Tom Caputi [Thu, 28 Sep 2017 15:49:13 +0000 (11:49 -0400)]
Fixes for #6639

Several issues were uncovered by running stress tests with zfs
encryption and raw sends in particular. The issues and their
associated fixes are as follows:

* arc_read_done() has the ability to chain several requests for
  the same block of data via the arc_callback_t struct. In these
  cases, the ARC would only use the first request's dsobj from
  the bookmark to decrypt the data. This is problematic because
  the first request might be a prefetch zio which is able to
  handle the key not being loaded, while the second might use a
  different key that it is sure will work. The fix here is to
  pass the dsobj with each individual arc_callback_t so that each
  request can attempt to decrypt the data separately.

* DRR_FREE and DRR_FREEOBJECT records in a send file were not
  having their transactions properly tagged as raw during raw
  sends, which caused a panic when the dbuf code attempted to
  decrypt these blocks.

* traverse_prefetch_metadata() did not properly set
  ZIO_FLAG_SPECULATIVE when issuing prefetch IOs.

* Added a few asserts and code cleanups to ensure these issues
  are more detectable in the future.

Signed-off-by: Tom Caputi <tcaputi@datto.com>
7 years agoEncryption patch follow-up
Tom Caputi [Tue, 12 Sep 2017 20:15:11 +0000 (16:15 -0400)]
Encryption patch follow-up

* PBKDF2 implementation changed to OpenSSL implementation.

* HKDF implementation moved to its own file and tests
  added to ensure correctness.

* Removed libzfs's now unnecessary dependency on libzpool
  and libicp.

* Ztest can now create and test encrypted datasets. This is
  currently disabled until issue #6526 is resolved, but
  otherwise functions as advertised.

* Several small bug fixes discovered after enabling ztest
  to run on encrypted datasets.

* Fixed coverity defects added by the encryption patch.

* Updated man pages for encrypted send / receive behavior.

* Fixed a bug where encrypted datasets could receive
  DRR_WRITE_EMBEDDED records.

* Minor code cleanups / consolidation.

Signed-off-by: Tom Caputi <tcaputi@datto.com>
7 years agoRelax ASSERT for #6526
Tom Caputi [Wed, 11 Oct 2017 16:12:48 +0000 (12:12 -0400)]
Relax ASSERT for #6526

This patch resolves a minor issue where an ASSERT in
metaslab_passivate() that only applies to non weight-based
metaslabs was erroneously applied to all metaslabs.

Signed-off-by: Tom Caputi <tcaputi@datto.com>
7 years agoRemove useless DEFAULT_INCLUDES in AM_CCASFLAGS
KireinaHoro [Wed, 11 Oct 2017 17:42:05 +0000 (01:42 +0800)]
Remove useless DEFAULT_INCLUDES in AM_CCASFLAGS

CPPASCOMPILE and LTCPPASCOMPILE all include DEFAULT_INCLUDES,
hence it's unnecessary to add the includes again.

Signed-off-by: Pengcheng Xu <i@jsteward.moe>
7 years agoFix libspl assembler flags to respect cpu type
KireinaHoro [Wed, 11 Oct 2017 05:36:26 +0000 (13:36 +0800)]
Fix libspl assembler flags to respect cpu type

It's important to respect the user's CFLAGS as mismatched -mcpu
will directly result in the assembler not able to produce correct
code. Fixes #6733.

Signed-off-by: Pengcheng Xu <i@jsteward.moe>
7 years agoSPARC optimizations for Encode()
KireinaHoro [Wed, 11 Oct 2017 05:16:46 +0000 (13:16 +0800)]
SPARC optimizations for Encode()

Normally a SPARC processor runs in big endian mode. Save the extra labor
needed for little endian machines when the target is a big endian one
(sparc).

Signed-off-by: Pengcheng Xu <i@jsteward.moe>
7 years agoSPARC optimizations for SHA1Transform()
KireinaHoro [Wed, 11 Oct 2017 05:09:43 +0000 (13:09 +0800)]
SPARC optimizations for SHA1Transform()

Passing arguments explicitly into SHA1Transform() increases the number of
registers abailable to the compiler, hence leaving more local and out registers
available. The missing symbol of sha1_consts[], which prevents compiling on
SPARC, is added back, which speeds up the process of utilizing the relative
constants.
This should fix #6738.

Signed-off-by: Pengcheng Xu <i@jsteward.moe>
7 years agoFix boot from ZFS issues
aun [Wed, 11 Oct 2017 17:06:20 +0000 (20:06 +0300)]
Fix boot from ZFS issues

* Correct ZFS snapshot listing
* Disable "lvm is not available" message on quiet boot

Reviewed-by: George Melikov <mail@gmelikov.ru>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Alar Aun <spamtoaun@gmail.com>
Closes #6700
Closes #6747

7 years agoFix chattr/cleanup failure
Brian Behlendorf [Wed, 11 Oct 2017 16:15:44 +0000 (09:15 -0700)]
Fix chattr/cleanup failure

The chattr cleanup step may fail to delete the user if there is still
an active process running as that user.  Retry the userdel when this
occurs to eliminate spurious false positves.

  ERROR: userdel quser1 exited 8
  userdel: user quser1 is currently used by process 26814

Reviewed-by: George Melikov <mail@gmelikov.ru>
Reviewed-by: Giuseppe Di Natale <dinatale2@llnl.gov>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #6749

7 years agoFix coverity defects: CID 147474
Tobin Harding [Tue, 10 Oct 2017 23:41:47 +0000 (10:41 +1100)]
Fix coverity defects: CID 147474

CID 147474: Logically dead code (DEADCODE)

Remove ternary operator and return `error` directly.

Currently return value is derived from a ternary operator. The
conditional is always true. The ternary operator is therefore
redundant i.e dead code.

Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Tobin C. Harding <me@tobin.cc>
Closes #6723

7 years agoSkip FREEOBJECTS for objects which can't exist
Fabian Grünbichler [Tue, 26 Sep 2017 12:03:21 +0000 (14:03 +0200)]
Skip FREEOBJECTS for objects which can't exist

When sending an incremental stream based on a snapshot, the receiving
side must have the same base snapshot.  Thus we do not need to send
FREEOBJECTS records for any objects past the maximum one which exists
locally.

This allows us to send incremental streams (again) to older ZFS
implementations (e.g. ZoL < 0.7) which actually try to free all objects
in a FREEOBJECTS record, instead of bailing out early.

Reviewed by: Paul Dagnelie <pcd@delphix.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
Closes #5699
Closes #6507
Closes #6616

7 years agoFree objects when receiving full stream as clone
Fabian Grünbichler [Fri, 29 Sep 2017 10:00:29 +0000 (12:00 +0200)]
Free objects when receiving full stream as clone

All objects after the last written or freed object are not supposed to
exist after receiving the stream.  Free them accordingly, as if a
freeobjects record for them had been included in the stream.

Reviewed by: Paul Dagnelie <pcd@delphix.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
Closes #5699
Closes #6507
Closes #6616

7 years agoFix intra-pool resumable 'zfs send -t <token>'
LOLi [Tue, 10 Oct 2017 22:22:05 +0000 (00:22 +0200)]
Fix intra-pool resumable 'zfs send -t <token>'

Because resuming from a token requires "guid" -> "snapshot" mapping
we have to walk the whole dataset hierarchy to find the right snapshot
to send; when both source and destination exists, for an incremental
resumable stream, libzfs gets confused and picks up the wrong snapshot
to send from: this results in attempting to send

   "destination@snap1 -> source@snap2"

instead of

   "source@snap1 -> source@snap2"

which fails with a "Invalid cross-device link" error (EXDEV).

Fix this by adjusting the logic behind dataset traversal in
zfs_iter_children() to pick the right snapshot to send from.

Additionally update dry-run 'zfs send -t' to print its output to
stderr: this is consistent with other dry-run commands.

Reviewed-by: George Melikov <mail@gmelikov.ru>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: loli10K <ezomori.nozomu@gmail.com>
Closes #6618
Closes #6619
Closes #6623

7 years agoFix ARC behavior on 32-bit systems
Brian Behlendorf [Tue, 10 Oct 2017 22:19:19 +0000 (15:19 -0700)]
Fix ARC behavior on 32-bit systems

With the addition of the ABD changes consumption of the virtual
address space has been greatly reduced.  This exposed an issue on
CONFIG_HIGHMEM systems where free memory was being calculated
incorrectly.  Functionally this didn't cause any major problems
prior to ABD because a lack of available virtual address space
was used as an indicator of low memory.

This patch makes the following changes to address the issue and
in the process realigns the code further with OpenZFS.  There
are no substantive changes in behavior for 64-bit systems.

* Added CONFIG_HIGHMEM case to the arc_all_memory() and
  arc_free_memory() functions to only consider low memory pages
  on CONFIG_HIGHMEM systems.

* The arc_free_memory() function was updated to return bytes
  instead of pages to be consistent with the other helper
  functions.  In user space we make up some reasonable values
  since currently only testing is performed in this context.

* Adds three new values to the arcstats kstat to provide visibility
  in to the ARC's assessment of the memory situation:
  memory_all_bytes, memory_free_bytes, and memory_available_bytes.

* Added kmem_reap() call to arc_available_memory() for 32-bit
  builds to realign code with OpenZFS.

* Reduced size of test file in /async_destroy_001_pos.ksh to
  speed up test case.  Multiple txgs are still required.

* Move vdevs used by zpool_clear_001_pos and zpool_upgrade_002_pos
  to TEST_BASE_DIR location to speed up test cases.

Reviewed-by: David Quigley <david.quigley@intel.com>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #5352
Closes #6734

7 years agoFix inclusion of libgcc_s.so on Void
privb0x23 [Mon, 9 Oct 2017 21:34:26 +0000 (21:34 +0000)]
Fix inclusion of libgcc_s.so on Void

On Void Linux (x86_64 musl) libgcc_s.so is located in "/usr/lib"
so it is not found by dracut and it produces an error.

Add a simple additional path check for "/usr/lib/libgcc_s.so*"
and install it in the initramfs.

Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: privb0x23 <privb0x23@users.noreply.github.com>
Closes #6715

7 years agoFix abdstats kstat on 32-bit systems
Brian Behlendorf [Fri, 6 Oct 2017 18:23:12 +0000 (11:23 -0700)]
Fix abdstats kstat on 32-bit systems

When decrementing the struct_size and scatter_chunk_waste kstats
the value needs to be cast to an int on 32-bit systems.

Reviewed-by: George Melikov <mail@gmelikov.ru>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #6721

7 years agoUse bitwise '&' instead of logical '&&'
Tobin Harding [Fri, 6 Oct 2017 02:38:55 +0000 (13:38 +1100)]
Use bitwise '&' instead of logical '&&'

Make two instances of the same change. Change bitwise AND (&) to logical
AND (&&).

Currently the code uses a bitwise AND between two boolean values.

In the first instance;

The first operand is a flag that has been bitwise combined with a bit
mask to get a boolean value as to whether a file has group write
permissions set.

The second operand used is a struct member that is intended as a
boolean flag not a bit mask.

In the second instance the argument is the same except with world write
permissions instead of group write (S_IWOTH, S_IWGRP).

Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: George Melikov <mail@gmelikov.ru>
Reviewed-by: Chris Dunlop <chris@onthe.net.au>
Signed-off-by: Tobin C. Harding <me@tobin.cc>
Closes #6684
Closes #6722

7 years agoRemove unnecessary equality check
Tobin Harding [Fri, 6 Oct 2017 02:33:44 +0000 (13:33 +1100)]
Remove unnecessary equality check

Currently `if` statement includes an assignment (from a function return
value) and a equality check. The parenthesis are in the incorrect place,
currently the code clobbers the function return value because of this.

We can fix this by simplifying the `if` statement.

`if (foo != 0)`

can be more succinctly expressed as

`if (foo)`

Remove the equality check, add parenthesis to correct the statement.

Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: George Melikov <mail@gmelikov.ru>
Reviewed-by: Chris Dunlop <chris@onthe.net.au>
Signed-off-by: Tobin C. Harding <me@tobin.cc>
Closes #6685
Close #6719

7 years agoUse linear abd in vdev_copy_uberblocks()
Isaac Huang [Fri, 6 Oct 2017 02:30:02 +0000 (20:30 -0600)]
Use linear abd in vdev_copy_uberblocks()

The vdev_copy_uberblocks() function should use abd_alloc_linear() to
allocate ub_abd, because abd_to_buf(ub_abd)) is used later.

Reviewed-by: George Melikov <mail@gmelikov.ru>
Reviewed-by: Giuseppe Di Natale <dinatale2@llnl.gov>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Isaac Huang <he.huang@intel.com>
Closes #6718
Closes #6713

7 years agoRemove dead code from AVL tree
Brian Behlendorf [Fri, 6 Oct 2017 02:28:00 +0000 (19:28 -0700)]
Remove dead code from AVL tree

The avl_update_* functions are never used by ZFS and are therefore
being removed.  They're barely even used in Illumos.  Additionally,
simplify avl_add() by using a VERIFY which produces exactly the same
behavior under Linux.

Reviewed-by: George Melikov <mail@gmelikov.ru>
Reviewed-by: Giuseppe Di Natale <dinatale2@llnl.gov>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #6716

7 years agoreceive_freeobjects() skips freeing some objects
Ned Bass [Mon, 2 Oct 2017 22:36:04 +0000 (15:36 -0700)]
receive_freeobjects() skips freeing some objects

When receiving a FREEOBJECTS record, receive_freeobjects()
incorrectly skips a freed object in some cases. Specifically, this
happens when the first object in the range to be freed doesn't exist,
but the second object does. This leaves an object allocated on disk
on the receiving side which is unallocated on the sending side, which
may cause receiving subsequent incremental streams to fail.

The bug was caused by an incorrect increment of the object index
variable when current object being freed doesn't exist.  The
increment is incorrect because incrementing the object index is
handled by a call to dmu_object_next() in the increment portion of
the for loop statement.

Add test case that exposes this bug.

Reviewed-by: George Melikov <mail@gmelikov.ru>
Reviewed-by: Giuseppe Di Natale <dinatale2@llnl.gov>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Ned Bass <bass6@llnl.gov>
Closes #6694
Closes #6695

7 years agoUpdate the default for zfs_txg_history
Alek P [Fri, 29 Sep 2017 22:58:52 +0000 (15:58 -0700)]
Update the default for zfs_txg_history

It's often useful to have access to txg history for debugging
purposes. This patch changes the default from 0 to 100 TXGs
worth of history preserved.

Reviewed by: Matthew Ahrens <mahrens@delphix.com>
Reviewed-by: Giuseppe Di Natale <dinatale2@llnl.gov>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed by: Richard Elling <Richard.Elling@RichardElling.com>
Reviewed by: Prakash Surya <prakash.surya@delphix.com>
Reviewed-by: George Melikov <mail@gmelikov.ru>
Signed-off-by: Alek Pinchuk <apinchuk@datto.com>
Closes #6691

7 years agoScale the dbuf cache with arc_c
chrisrd [Fri, 29 Sep 2017 22:49:19 +0000 (08:49 +1000)]
Scale the dbuf cache with arc_c

Commit d3c2ae1 introduced a dbuf cache with a default size of the
minimum of 100M or 1/32 maximum ARC size. (These figures may be adjusted
using dbuf_cache_max_bytes and dbuf_cache_max_shift.) The dbuf cache
is counted as metadata for the purposes of ARC size calculations.

On a 1GB box the ARC maximum size defaults to c_max 493M which gives a
dbuf cache default minimum size of 15.4M, and the ARC metadata defaults
to minimum 16M. I.e. the dbuf cache is an significant proportion of the
minimum metadata size. With other overheads involved this actually means
the ARC metadata doesn't get down to the minimum.

This patch dynamically scales the dbuf cache to the target ARC size
instead of statically scaling it to the maximum ARC size. (The scale is
still set by dbuf_cache_max_shift and the maximum size is still fixed by
dbuf_cache_max_bytes.) Using the target ARC size rather than the current
ARC size is done to help the ARC reach the target rather than simply
focusing on the current size.

Reviewed-by: Chunwei Chen <tuxoko@gmail.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: George Melikov <mail@gmelikov.ru>
Signed-off-by: Chris Dunlop <chris@onthe.net.au>
Issue #6506
Closes #6561

7 years agoAdd 'zfs diff' coverage to the ZFS Test Suite
LOLi [Thu, 28 Sep 2017 20:04:14 +0000 (22:04 +0200)]
Add 'zfs diff' coverage to the ZFS Test Suite

This change adds four new tests to the ZTS:

 * zfs_diff_changes: verify type of changes diplayed (-, +, R and M)
 * zfs_diff_cliargs: verify command line options and arguments
 * zfs_diff_timestamp: verify 'zfs diff -t'
 * zfs_diff_types: verify type of objects (files, dirs, pipes...)

Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: John Wren Kennedy <john.kennedy@delphix.com>
Signed-off-by: loli10K <ezomori.nozomu@gmail.com>
Closes #6686

7 years agovdev_id: extension for new scsi topology
Simon Guest [Wed, 27 Sep 2017 17:39:47 +0000 (06:39 +1300)]
vdev_id: extension for new scsi topology

On systems with SCSI rather than SAS disk topology, this change enables
the vdev_id script to match against the block device path, and therefore
create a vdev alias in /dev/disk/by-vdev.

Reviewed-by: Tony Hutter <hutter2@llnl.gov>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Simon Guest <simon.guest@tesujimath.org>
Closes #6592

7 years agoRun ztest for longer on "Coverage" builders
Prakash Surya [Tue, 26 Sep 2017 19:29:32 +0000 (12:29 -0700)]
Run ztest for longer on "Coverage" builders

Reviewed-by: George Melikov <mail@gmelikov.ru>
Reviewed-by: Giuseppe Di Natale <dinatale2@llnl.gov>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Prakash Surya <prakash.surya@delphix.com>
Closes #6675

7 years agoFix printk() calls missing log level
DeHackEd [Mon, 25 Sep 2017 17:38:27 +0000 (13:38 -0400)]
Fix printk() calls missing log level

Reviewed-by: George Melikov <mail@gmelikov.ru>
Reviewed-by: Giuseppe Di Natale <dinatale2@llnl.gov>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: DHE <git@dehacked.net>
Closes #6672

7 years agoFix some ZFS Test Suite issues
LOLi [Mon, 25 Sep 2017 17:32:34 +0000 (19:32 +0200)]
Fix some ZFS Test Suite issues

* Add 'zfs bookmark' coverage (zfs_bookmark_cliargs)

 * Add OpenZFS 8166 coverage (zpool_scrub_offline_device)

 * Fix "busy" zfs_mount_remount failures

 * Fix bootfs_003_pos, bootfs_004_neg, zdb_005_pos local cleanup

 * Update usage of $KEEP variable, add get_all_pools() function

 * Enable history_008_pos and rsend_019_pos (non-32bit builders)

 * Enable zfs_copies_005_neg, update local cleanup

 * Fix zfs_send_007_pos (large_dnode + OpenZFS 8199)

 * Fix rollback_003_pos (use dataset name, not mountpoint, to unmount)

 * Update default_raidz_setup() to work properly with more than 3 disks

 * Use $TEST_BASE_DIR instead of hardcoded (/var)/tmp for file VDEVs

 * Update usage of /dev/random to /dev/urandom

Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: loli10K <ezomori.nozomu@gmail.com>
Issue #6086
Closes #5658
Closes #6143
Closes #6421
Closes #6627
Closes #6632

7 years agoFix ZTS MMP tests and ztest -M behavior
Olaf Faaland [Sat, 23 Sep 2017 16:28:18 +0000 (09:28 -0700)]
Fix ZTS MMP tests and ztest -M behavior

Quote "$MMP_IMPORT_MSG" when it is passed as an argument, as it is a
multi-word string.  Some tests were passing when they should not have,
because the grep was only testing for the first word.

Correct the message expected when no hostid is set and the test attempts
to enable multihost.  It did not match the actual output in that
situation.

Disable ztest_reguid() when ztest is invoked with the -M option.  If
ztest performs a reguid, a concurrent import attempt may fail with the
error "one or more devices is currently unavailable" if the guid sum is
calculated on the original device guids but compared against the guid
sum ztest wrote based on the new device guids.

Reviewed-by: George Melikov <mail@gmelikov.ru>
Reviewed-by: Giuseppe Di Natale <dinatale2@llnl.gov>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Olaf Faaland <faaland1@llnl.gov>
Closes #6666

7 years agoFix "--enable-code-coverage" debug build
Brian Behlendorf [Sat, 23 Sep 2017 05:16:18 +0000 (22:16 -0700)]
Fix "--enable-code-coverage" debug build

When --enable-code-coverage is provided it should not result
in NDEBUG being defined.  This is controlled by --enable-debug.

Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #6674

7 years agoUpdate codecov.yml
Brian Behlendorf [Sat, 23 Sep 2017 01:54:34 +0000 (18:54 -0700)]
Update codecov.yml

Update the codecov.yml to make the following functional changes.

* Do not require the CI testing to pass before posting results.
* Set red-yellow-green coverage percent from 50%-100%
* Allow a 1% drop in coverage to still be considered a pass.
* Reduce the size of the comment posted to the issue.

Additionally, the top level README.markdown has been updated
to include the codecov.io badge and the project summary reworded.

Reviewed-by: Prakash Surya <prakash.surya@delphix.com>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #6669

7 years agoAdd support for "--enable-code-coverage" option
Prakash Surya [Sat, 23 Sep 2017 01:49:57 +0000 (18:49 -0700)]
Add support for "--enable-code-coverage" option

This change adds support for a new option that can be passed to the
configure script: "--enable-code-coverage". Further, the "--enable-gcov"
option has been removed, as this new option provides the same
functionality (plus more).

When using this new option the following make targets are available:

 * check-code-coverage
 * code-coverage-capture
 * code-coverage-clean

Note: these make targets can only be run from the root of the project.

Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Prakash Surya <prakash.surya@delphix.com>
Closes #6670

7 years agoReimplement vdev_random_leaf and rename it
Olaf Faaland [Fri, 22 Sep 2017 21:29:26 +0000 (14:29 -0700)]
Reimplement vdev_random_leaf and rename it

Rename it as mmp_random_leaf() since it is defined in mmp.c.

The earlier implementation could end up spinning forever if a pool had a
vdev marked writeable, none of whose children were writeable.  It also
did not guarantee that if a writeable leaf vdev existed, it would be
found.

Reimplement to recursively walk the device tree to select the leaf.  It
searches the entire tree, so that a return value of (NULL) indicates
there were no usable leaves in the pool; all were either not writeable
or had pending mmp writes.

It still chooses the starting child randomly at each level of the tree,
so if the pool's devices are healthy, the mmp writes go to random leaves
with an even distribution.  This was verified by testing using
zfs_multihost_history enabled.

Reviewed by: Thomas Caputi <tcaputi@datto.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: Giuseppe Di Natale <dinatale2@llnl.gov>
Signed-off-by: Olaf Faaland <faaland1@llnl.gov>
Closes #6631
Closes #6665

7 years agoCleanup zloop working directory after each pass
Don Brady [Thu, 21 Sep 2017 17:17:56 +0000 (11:17 -0600)]
Cleanup zloop working directory after each pass

Reviewed-by: George Melikov <mail@gmelikov.ru>
Reviewed-by: Giuseppe Di Natale <dinatale2@llnl.gov>
Reviewed by: John Kennedy <jwk404@gmail.com>
Reviewed-by: Olaf Faaland <faaland1@llnl.gov>
Signed-off-by: Don Brady <don.brady@delphix.com>
Issue #6595
Closes #6663

7 years agoIncrease default arc_c_min
Brian Behlendorf [Wed, 20 Sep 2017 16:36:17 +0000 (09:36 -0700)]
Increase default arc_c_min

Increase the default arc_c_min value to which whichever is larger,
either 32M or 1/32 of total system memory.  This is advantageous for
systems with more than 1G of memory where performance issues may
occur when the ARC is allowed to collapse below a minimum size.
At the same time we want to use the bare minimum value which is
still functional so the filesystem can be used in very low memory
environments.

Reviewed-by: Tim Chase <tim@chase2k.com>
Reviewed-by: George Melikov <mail@gmelikov.ru>
Reviewed-by: Giuseppe Di Natale <dinatale2@llnl.gov>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #6659

7 years agoExport symbol dmu_tx_mark_netfree()
Brian Behlendorf [Wed, 20 Sep 2017 16:30:24 +0000 (09:30 -0700)]
Export symbol dmu_tx_mark_netfree()

This symbol is needed by Lustre for the same reason it was needed
by the ZPL.  It should have been exported when the original patch
was merged.

Reviewed-by: George Melikov <mail@gmelikov.ru>
Reviewed-by: Giuseppe Di Natale <dinatale2@llnl.gov>
Reviewed-by: Tony Hutter <hutter2@llnl.gov>
Reviewed-by: Alex Zhuravlev <bzzz@whamcloud.com>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #6660

7 years agomisc: fix meaningless values
Feng Sun [Tue, 19 Sep 2017 19:19:08 +0000 (03:19 +0800)]
misc: fix meaningless values

Reviewed-by: George Melikov <mail@gmelikov.ru>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: Tim Chase <tim@chase2k.com>
Signed-off-by: Feng Sun <loyou85@gmail.com>
Closes #6658

7 years agoCorrect cppcheck errors
Giuseppe Di Natale [Tue, 19 Sep 2017 19:17:29 +0000 (12:17 -0700)]
Correct cppcheck errors

ZFS buildbot STYLE builder was moved to Ubuntu 17.04
which has a newer version of cppcheck. Handle the
new cppcheck errors.

uu_* functions removed in this commit were unused
and effectively dead code. They are now retired.

Reviewed-by: George Melikov <mail@gmelikov.ru>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Giuseppe Di Natale <dinatale2@llnl.gov>
Closes #6653

7 years agoZTS fix slog_replay_volume.ksh failure
Brian Behlendorf [Tue, 19 Sep 2017 17:09:37 +0000 (10:09 -0700)]
ZTS fix slog_replay_volume.ksh failure

The slog_replay_volume.ksh test case will fail when the pool is
layered on files in a filesystem which does not support discard.
Avoid this issue by creating the pool using DISKS which will
either be loopback device or real disk.

Reviewed-by: George Melikov <mail@gmelikov.ru>
Reviewed-by: Giuseppe Di Natale <dinatale2@llnl.gov>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #6654

7 years agoRemove FRU and LIBTOPO Support
David Quigley [Tue, 19 Sep 2017 00:06:40 +0000 (18:06 -0600)]
Remove FRU and LIBTOPO Support

FRU and LIBTOPO support are illumos only features that will not be ported to
Linux and make the code more complicated than necessary. This commit
makes way for further cleanups of the zed/FMA code.

Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: David Quigley <david.quigley@intel.com>
Closes #6641

7 years agoCorrect shellcheck errors
Giuseppe Di Natale [Mon, 18 Sep 2017 21:23:09 +0000 (14:23 -0700)]
Correct shellcheck errors

The ZFS buildbot moved to using Ubuntu 17.04 for the
STYLE builder which has a newer version of shellcheck.
Correct the new issues it discovers.

Reviewed-by: loli10K <ezomori.nozomu@gmail.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Giuseppe Di Natale <dinatale2@llnl.gov>
Closes #6647

7 years agoZTS fix events_002_pos.sh failure
Brian Behlendorf [Sun, 17 Sep 2017 02:36:44 +0000 (19:36 -0700)]
ZTS fix events_002_pos.sh failure

Fix spurious events_002_pos failures by waiting longer before
grabbing the log to check for the resilver_finish event.  It
would be better to rework this logic to wait only as long as
needed rather than a fixed timeout.

Reviewed-by: George Melikov <mail@gmelikov.ru>
Reviewed-by: Giuseppe Di Natale <dinatale2@llnl.gov>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #6651

7 years agoLinux 3.14 compat: IO acct, global_page_state, etc
Giuseppe Di Natale [Sat, 16 Sep 2017 18:00:19 +0000 (11:00 -0700)]
Linux 3.14 compat: IO acct, global_page_state, etc

generic_start_io_acct/generic_end_io_acct in the master
branch of the linux kernel requires that the request_queue
be provided.

Move the logic from freemem in the spl to arc_free_memory
in arc.c. Do this so we can take advantage of global_page_state
interface checks in zfs.

Upstream kernel replaced struct block_device with
struct gendisk in struct bio. Determine if the
function bio_set_dev exists during configure
and have zfs use that if it exists.

bio_set_dev https://github.com/torvalds/linux/commit/74d4699
global_node_page_state https://github.com/torvalds/linux/commit/75ef718
io acct https://github.com/torvalds/linux/commit/d62e26b

Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Giuseppe Di Natale <dinatale2@llnl.gov>
Closes #6635

7 years agoAdd mdoc style checker
LOLi [Sat, 16 Sep 2017 17:51:24 +0000 (19:51 +0200)]
Add mdoc style checker

Add a new make 'mancheck' target which uses mandoc -Tlint to verify
manpage files: currently only zfs(8), zpool(8) zdb(8) and zgenhostid(8)
are supported.

Additionally fix some outstanding manpage formatting issues.

Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: loli10K <ezomori.nozomu@gmail.com>
Closes #6646

7 years agoZTEST: Always enable asserts
David Quigley [Wed, 13 Sep 2017 18:48:05 +0000 (14:48 -0400)]
ZTEST: Always enable asserts

The build for ztest always enabled debug information but does not enable
asserts unless --enable-debug is used. This will always enable asserts
in the ztest code.

Reviewed-by: George Melikov <mail@gmelikov.ru>
Reviewed-by Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: David Quigley <david.quigley@intel.com>
Closes #6640

7 years agoOpenZFS 8435 - zpool.1m and zfs.1m: minor cleanup
George Melikov [Fri, 15 Sep 2017 20:13:52 +0000 (23:13 +0300)]
OpenZFS 8435 - zpool.1m and zfs.1m: minor cleanup

3796 listsnapshots not documented in zpool man page

Authored by: George Melikov <mail@gmelikov.ru>
Reviewed by: Matt Ahrens <mahrens@delphix.com>
Reviewed by: Yuri Pankov <yuripv@gmx.com>
Approved by: Dan McDonald <danmcd@joyent.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Ported-by: Giuseppe Di Natale <dinatale2@llnl.gov>
Ported-by: George Melikov mail@gmelikov.ru
OpenZFS-issue: https://www.illumos.org/issues/8435
OpenZFS-commit: openzfs/openzfs@a058d1c

Porting notes: OpenZFS review applied,
some ZoL changes were reverted.
See https://github.com/openzfs/openzfs/pull/415

7 years agoMake "-fno-inline" compile option more accessible
Prakash Surya [Fri, 15 Sep 2017 18:47:11 +0000 (11:47 -0700)]
Make "-fno-inline" compile option more accessible

When functions are inlined, it can make the system much more difficult
to instrument using tools such as ftrace, BPF, crash, etc. Thus, to aid
development and increase the system's observability, when the
"--enable-debuginfo" flag is specified, the "-fno-inline" compilation
option will be used for both userspace and kernel modules.

Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Prakash Surya <prakash.surya@delphix.com>
Closes #6605

7 years agoAdd configure option to enable gcov analysis
Brian Behlendorf [Fri, 15 Sep 2017 17:24:13 +0000 (10:24 -0700)]
Add configure option to enable gcov analysis

* Add configure option to enable gcov analysis.
* Includes a few minor ctime fixes.
* Add codecov.yml configuration.

Reviewed-by: Prakash Surya <prakash.surya@delphix.com>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #6642

7 years agoModifying XATTRs doesnt change the ctime
Gaurav Kumar [Wed, 13 Sep 2017 19:20:07 +0000 (12:20 -0700)]
Modifying XATTRs doesnt change the ctime

Changing any metadata, should modify the ctime.

Reviewed-by: Chunwei Chen <tuxoko@gmail.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: gaurkuma <gauravk.18@gmail.com>
Closes #3644
Closes #6586

7 years agoFix bug in distclean which removes needed files
David Quigley [Wed, 13 Sep 2017 18:45:04 +0000 (12:45 -0600)]
Fix bug in distclean which removes needed files

Running distclean removes the following files because of an error
in Makefile.am

deleted:    tests/zfs-tests/include/commands.cfg
deleted:    tests/zfs-tests/include/libtest.shlib
deleted:    tests/zfs-tests/include/math.shlib
deleted:    tests/zfs-tests/include/properties.shlib
deleted:    tests/zfs-tests/include/zpool_script.shlib

Reviewed-by: George Melikov <mail@gmelikov.ru>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: Giuseppe Di Natale <dinatale2@llnl.gov>
Signed-off-by: David Quigley <david.quigley@intel.com>
Closes #6636

7 years agoRelax (ref)reservation constraints on ZVOLs
LOLi [Tue, 12 Sep 2017 18:33:22 +0000 (20:33 +0200)]
Relax (ref)reservation constraints on ZVOLs

This change allow (ref)reservation to be set larger than the current
ZVOL size: this is safe as we normally set refreservation > volsize
at ZVOL creation time when we account for metadata.

Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: George Melikov <mail@gmelikov.ru>
Reviewed by: Richard Elling <Richard.Elling@RichardElling.com>
Reviewed by: Matthew Ahrens <mahrens@delphix.com>
Signed-off-by: loli10K <ezomori.nozomu@gmail.com>
Closes #2468
Closes #6610

7 years agoFix false config_cache_write events
Arkadiusz Bubała [Mon, 11 Sep 2017 17:25:01 +0000 (19:25 +0200)]
Fix false config_cache_write events

On pool import when the old cache file is removed
the ereport.fs.zfs.config_cache_write event is generated.
Because zpool export always removes cache file it happens
every export - import sequence.

Reviewed-by: George Melikov <mail@gmelikov.ru>
Reviewed-by: loli10K <ezomori.nozomu@gmail.com>
Reviewed-by: Giuseppe Di Natale <dinatale2@llnl.gov>
Signed-off-by: Arkadiusz Bubała <arkadiusz.bubala@open-e.com>
Closes #6617

7 years agoAdd -vnP support to 'zfs send' for bookmarks
LOLi [Fri, 8 Sep 2017 22:24:31 +0000 (00:24 +0200)]
Add -vnP support to 'zfs send' for bookmarks

This leverages the functionality introduced in cf7684b to expose
verbose, dry-run and parsable 'zfs send' options for bookmarks.

Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: loli10K <ezomori.nozomu@gmail.com>
Closes #3666
Closes #6601