Don Brady [Mon, 7 Nov 2016 23:01:38 +0000 (16:01 -0700)]
Add illumos FMD ZFS logic to ZED -- phase 2
The phase 2 work primarily entails the Diagnosis Engine and
the Retire Agent modules. It also includes infrastructure
to support a crude FMD environment to host these modules.
The Diagnosis Engine consumes I/O and checksum ereports and
feeds them into a SERD engine which will generate a corres-
ponding fault diagnosis when the SERD engine fires. All the
diagnosis state data is collected into cases, one case per
vdev being tracked.
The Retire Agent responds to diagnosed faults by isolating
the faulty VDEV. It will notify the ZFS kernel module of
the new VDEV state (degraded or faulted). This agent is
also responsible for managing hot spares across pools.
When it encounters a device fault or a device removal it
replaces the device with an appropriate spare if available.
Reviewed-by: Tony Hutter <hutter2@llnl.gov> Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov> Signed-off-by: Don Brady <don.brady@intel.com>
Closes #5343
Chunwei Chen [Wed, 2 Nov 2016 00:19:52 +0000 (17:19 -0700)]
Use set_cached_acl and forget_cached_acl when possible
Originally, these two function are inline, so their usability is tied to
posix_acl_release. However, since Linux 3.14, they became EXPORT_SYMBOL, so we
can always use them. In this patch, we create an independent test for these
two functions so we can use them when possible.
Chunwei Chen [Fri, 28 Oct 2016 20:37:00 +0000 (13:37 -0700)]
Batch free zpl_posix_acl_release
Currently every calls to zpl_posix_acl_release will schedule a delayed task,
and each delayed task will add a timer. This used to be fine except for
possibly bad performance impact.
However, in Linux 4.8, a new timer wheel implementation[1] is introduced. In
this new implementation, the larger the delay, the less accuracy the timer is.
So when we have a flood of timer from zpl_posix_acl_release, they will expire
at the same time. Couple with the fact that task_expire will do linear search
with lock held. This causes an extreme amount of contention inside interrupt
and would actually lockup the system.
We fix this by doing batch free to prevent a flood of delayed task. Every call
to zpl_posix_acl_release will put the posix_acl to be freed on a lockless
list. Every batch window, 1 sec, the zpl_posix_acl_free will fire up and free
every posix_acl that passed the grace period on the list. This way, we only
have one delayed task every second.
This patch addresses multiple 'zpool import' block device
indentification problems which are most likely to occur on a
system configured to use blkid, by_vdev paths, multipath and
failover. The symptom most commonly observed is the import
uses different path names to import the pool than would
normally be expected.
* When using blkid to identify vdevs the listed devices may
be added to the cache in any order. In order to apply the
preferred search order heuristic a zfs_path_order() function
was added to calculate the order given full path names.
* Since it's possible to have multiple block devices with
different vdev guids which refer to the same ZPOOL_CONFIG_PATH
the slice cache must be indexed by guid and name. By avoiding
collisions the preferred ordering can be maintaining even
when multiple block devices claim the same ZPOOL_CONFIG_PATH.
The preferred sorting by partition was never benefitial for
a Linux system and was removed as part of this change.
* When adding entries to the blkid cache avl_find/avl_insert
are used instead of avl_add because collisions are possible
and must be handled gracefully.
* For pools using multipath devices there are, at a minimum,
three devices where a vdev label may be read. They are the
dm-* device and each underlying /dev/sd* device. Due to the
way the block cache is implemented each of these devices may
have a different cached copy of the vdev label. This can
result in "ghost pools" which appear to persist even after
a 'zpool labelclear' has been done to the dm-* device. In
order to prevent this the vdev label is read with O_DIRECT
in order to bypass any caching to get the on-disk version.
* When opening a block device verify that vdev guid read from
the disk matches the expected vdev guid. This allows for bad
labels to be filtered out.
Reviewed-by: Tony Hutter <hutter2@llnl.gov> Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #5359
Only restrict the maximum zio alloc size to 32-bit kernel space.
The same virtual address space limitations don't apply to user
space. This resolves a memory allocation failure in raidz_test
where it expects to be able to exercises all valid zio sizes.
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
The isainfo(1) utility was used by the ZFS Test Suite to determine
when running on a 32-bit platform. This non-portable check has been
replaced with an is_32bit helper function which uses getconf(1).
The getconf(1) utility is available for Linux, FreeBSD, and Illumos.
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Tony Hutter [Fri, 4 Nov 2016 20:34:13 +0000 (13:34 -0700)]
Allow autoreplace even when enclosure LED sysfs entries don't exist
The previous autoreplace code assumed that if you were using autoreplace, then
you also had the enclosure SES driver loaded. This could lead to autoreplace
not working if the SES driver wasn't loaded, or if it wasn't creating the
proper enclosure_device symlinks (which has happened). This patch removes
that assumption.
Reviewed by: Don Brady <don.brady@intel.com> Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov> Signed-off-by: Tony Hutter <hutter2@llnl.gov>
Closes #5363
Chunwei Chen [Tue, 26 Jan 2016 20:29:46 +0000 (12:29 -0800)]
Add support for O_TMPFILE
Linux 3.11 add O_TMPFILE to open(2), which allow creating an unlinked file on
supported filesystem. It's basically doing open(2) and unlink(2) atomically.
The filesystem support is added through i_op->tmpfile. We basically copy the
create operation except we get rid of the link and name related stuff and add
the new node to unlinked set.
We also add support for linkat(2) to link tmpfile. However, since all previous
file operation will skip ZIL, we force a txg_wait_synced to make sure we are
sync safe.
Chunwei Chen [Thu, 13 Oct 2016 00:30:46 +0000 (17:30 -0700)]
Fix unlinked file cannot do xattr operations
Currently, doing things like fsetxattr(2) on an unlinked file will result in
ENODATA. There's two places that cause this: zfs_dirent_lock and zfs_zget.
The fix in zfs_dirent_lock is pretty straightforward. In zfs_zget though, we
need it to not return error when the zp is unlinked. This is a pretty big
change in behavior, but skimming through all the callers, I don't think this
change would cause any problem. Also there's nothing preventing z_unlinked
from being set after the z_lock mutex is dropped before but before zfs_zget
returns anyway.
The rest of the stuff is to make sure we don't log xattr stuff when owner is
unlinked.
Romain Dolbeau [Wed, 2 Nov 2016 19:40:23 +0000 (20:40 +0100)]
Add parity generation/rebuild using AVX-512 for x86-64
avx512f should work on all AVX512 hardware, since it only uses
Foundation instructions.
avx512bw should be faster on hardware supporting the AVW512BW
extension. We can use full-width pshufb (instead of relying on the 256
bits AVX2 pshufb). As a side-effect, the code is also unrolled more.
BearBabyLiu [Wed, 2 Nov 2016 19:34:10 +0000 (03:34 +0800)]
Fix dsl_prop_get_all_dsl() memory leak
On error dsl_prop_get_all_ds() does not free the nvlist it allocates.
This behavior may have been intentional when originally written
but is atypical and often confusing. Since no callers rely on this
behavior the function has been updated to always free the nvlist
on error.
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov> Signed-off-by: BearBabyLiu <liu.huang@zte.com.cn>
Closes #5320
Brian Behlendorf [Mon, 31 Oct 2016 21:16:37 +0000 (21:16 +0000)]
Skip async_destroy_001_pos on 32-bit systems
The async_destroy_001_pos test case currently hangs when testing on
a 32-bit system. Conditionally skip this test case on 32-bit
systems until the root cause is identified and resolved.
Reviewed-by: Tom Caputi <tcaputi@datto.com> Reviewed-by: Tim Chase <tim@chase2k.com> Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Issue #5352
Issue #5347
Brian Behlendorf [Mon, 31 Oct 2016 19:24:54 +0000 (19:24 +0000)]
Use vmem_size() for 32-bit systems
On 32-bit Linux systems use vmem_size() to correctly size the ARC
and better determine when IO should be throttle due to low memory.
On 64-bit systems this change has no effect since the virtual
address space available far exceeds the physical memory available.
Reviewed-by: Tom Caputi <tcaputi@datto.com> Reviewed-by: Tim Chase <tim@chase2k.com> Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Issue #5347
Brian Behlendorf [Fri, 28 Oct 2016 23:53:24 +0000 (23:53 +0000)]
Fix 32-bit maximum volume size
A limit of 1TB exists for zvols on 32-bit systems. Update the code
to correctly reflect this limitation in a similar manor as the
OpenZFS implementation.
Reviewed-by: Tom Caputi <tcaputi@datto.com> Reviewed-by: Tim Chase <tim@chase2k.com> Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Issue #5347
Brian Behlendorf [Fri, 28 Oct 2016 22:42:56 +0000 (22:42 +0000)]
Enable .zfs/snapshot for 32-bit systems
Originally the .zfs/snapshot directory was disabled for 32-bit systems
because 64-bit inode numbers were not supported. This is no longer
the case and this functionality can be enabled by default.
Reviewed-by: Tom Caputi <tcaputi@datto.com> Reviewed-by: Tim Chase <tim@chase2k.com> Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Issue #5347
Closes #2002
Brian Behlendorf [Fri, 28 Oct 2016 22:40:14 +0000 (22:40 +0000)]
Add TASKQID_INVALID
Add the TASKQID_INVALID macros and update callers to use the macro
instead of testing against 0. There is no functional change
even though the functions in zfs_ctldir.c incorrectly used -1
instead of 0.
Reviewed-by: Tom Caputi <tcaputi@datto.com> Reviewed-by: Tim Chase <tim@chase2k.com> Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Issue #5347
Process all systemd services through the systemd scriptlets
This patch ensures that all systemd services are processed through the
systemd scriptlets, so that services are properly configured per the
preset file installed by the package.
Without this, zfs.target is set, but none of the services are enabled per
the preset file, meaning automounting filesystems and such won't work
out of the box.
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov> Signed-off-by: Neal Gompa <ngompa13@gmail.com>
Closes #5356
legend-hua [Mon, 31 Oct 2016 21:55:40 +0000 (05:55 +0800)]
Update migration tests
Due to the instability of the migration tests, the test will skip.
The migration tests focus on migrating test file from fs to ZFS fs.
We can create zpool and ext2 directly by loop device, rather than
by set_partition
Reviewed-by: Sydney Vanda <sydney.m.vanda@intel.com> Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov> Signed-off-by: legend-hua <liu.hua130@zte.com.cn>
Closes #5315
Jason Zaman [Fri, 28 Oct 2016 23:10:00 +0000 (07:10 +0800)]
Add paxcheck make lint target
This uses scanelf (from pax-utils) to check for any issues with the
binaries. It currently checks for executable stacks and textrels.
The checks are in a script so can be extended easily in the future for
more checks.
Executable stacks and textrels are frequently caused by issues in asm
files and lead to security and perf problems.
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov> Signed-off-by: Jason Zaman <jason@perfinion.com>
Closes #5338
Hajo Möller [Wed, 26 Oct 2016 17:30:43 +0000 (19:30 +0200)]
Fix lookup_bdev() on Ubuntu
Ubuntu added support for checking inode permissions to lookup_bdev() in kernel
commit 193fb6a2c94fab8eb8ce70a5da4d21c7d4023bee (merged in 4.4.0-6.21).
Upstream bug: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1636517
This patch adds a test for Ubuntu's variant of lookup_bdev() to configure and
calls the function in the correct way.
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov> Signed-off-by: Hajo Möller <dasjoe@gmail.com>
Closes #5336
Brian Behlendorf [Wed, 26 Oct 2016 16:13:43 +0000 (09:13 -0700)]
Disable zio_dva_throttle_enabled by default
Until it can be determined definitively that a performance
regression wasn't introduced accidentally by 3dfb57a this
functionality is being disabled by default. It can be re-
enabled by setting zio_dva_throttle_enabled=1.
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #5335
Issue #5289
jxiong [Tue, 25 Oct 2016 20:21:05 +0000 (04:21 +0800)]
Do not upgrade userobj accounting for snapshot dataset
'zfs recv' could disown a living objset without calling
dmu_objset_disown(). This will cause the problem that the objset
would be released while the upgrading thread is still running.
This patch avoids the problem by checking if a dataset is a snapshot
before calling dmu_objset_userobjspace_upgrade(). Snapshots
are immutable and therefore it doesn't make sense to update them.
- Fix autoreplace behaviour on statechange-led.sh script.
ZED sends the following events on an auto-replace:
1. statechange: Disk goes UNAVAIL->ONLINE
2. statechange: Disk goes ONLINE->UNAVAIL
3. vdev_attach: Disk goes ONLINE
Events 1-2 happen when ZED first attempts to do an auto-online. When that
fails, ZED then tries an auto-replace, generating the vdev_attach event in #3.
In the previous code, statechange-led was only looking at the UNAVAIL->ONLINE
transition to turn off the LED. It ignored the #2 ONLINE->UNAVAIL transition,
assuming it was just the "old" VDEV going offline. This is problematic, as
a drive can go from ONLINE->UNAVAIL when it's malfunctioning, and we don't want
to ignore that.
This new patch correctly turns on the fault LED every time a drive becomes
UNAVAIL. It also monitors vdev_attach events to trigger turning off the LED
when an auto-replaced disk comes online.
- Remove unnecessary libdevmapper warning with --with-config=kernel
This fixes an unnecessary libdevmapper warning when building
--with-config=kernel. Kernel code does not use libdevmapper, so the warning
is not needed.
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov> Signed-off-by: Tony Hutter <hutter2@llnl.gov>
Closes #2375
Closes #5312
Closes #5331
Brian Behlendorf [Mon, 24 Oct 2016 23:39:47 +0000 (16:39 -0700)]
Disable zpool_upgrade_002_pos test case
This test case frequently triggers issue #4034. There exists a
fix for this which is in the process of being upstreamed. Until
that fix is available disable the test case.
Reviewed by: George Wilson <george.wilson@delphix.com> Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #5329
Issue #4034
Brian Behlendorf [Mon, 24 Oct 2016 20:28:58 +0000 (13:28 -0700)]
Fix taskq creation failure in vdev_open_children()
When creating and destroying pools in tight loop it's possible to
exhaust the number of allowed threads on a system. This results
in taskq_create() failling and a NULL dereference.
Resolve the issue by falling back to opening the vdevs all
synchronously.
Reviewed-by: Denys Rtveliashvili <denys@rtveliashvili.name> Reviewed-by: Håkan Johansson <f96hajo@chalmers.se> Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes zfsonlinux/spl#521
Closes #4637
Tony Hutter [Mon, 24 Oct 2016 17:45:59 +0000 (10:45 -0700)]
Turn on/off enclosure slot fault LED even when disk isn't present
Previously when a drive faulted, the statechange-led.sh script would lookup
the drive's LED sysfs entry in /sys/block/sd*/device/enclosure_device, and
turn it on. During testing we noticed that if you pulled out a drive, or if
the drive was so badly broken that it no longer appeared to Linux, that the
/sys/block/sd* path would be removed, and the script could not lookup the
LED entry.
To fix this, this patch looks up the disks's more persistent
"/sys/class/enclosure/X:X:X:X/Slot N" LED sysfs path at pool import. It then
passes that path to the statechange-led script to use, rather than having the
script look it up on the fly. This allows the script to turn on/off the slot
LEDs even when the drive is missing.
Change location of current symlink created by test-runner
test-runner should be creating the current symlink in the
directory above the output directory. In a previous commit,
the current symlink was placed in the current working
directory, which could be inaccessible. It is more likely
that the output directory is always accessible.
This is needed because without this there's no deterministic
way to get the path to ZFS Test Suite results until after the
test suite has started. This makes it difficult for buildbot to
follow the log file.
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov> Signed-off-by: Giuseppe Di Natale <dinatale2@llnl.gov>
Closes #5314
Romain Dolbeau [Fri, 21 Oct 2016 17:55:49 +0000 (19:55 +0200)]
Fletcher4 algorithm implemented in pure NEON for Aarch64 / ARMv8 64 bits
This is not useful on micro-architecture with a weak NEON
implementation (only 64 bits); the native version is slower &
the byteswap barely faster than scalar. On A53 or A57, it's
a small improvement on scalar but OK for byteswap.
Brian Behlendorf [Fri, 21 Oct 2016 15:23:27 +0000 (08:23 -0700)]
Fix userquota_compare() function
The AVL tree compare function requires that either -1, 0, or 1 be
returned. However the strcmp() function only guarantees that a
negative, zero, or positive value is returned. Therefore, the
return value of strcmp() needs to be sanitized with AVL_ISIGN.
This was initially overlooked because the x86_64 implementation
of strcmp() happens to only returns the allowed values. This
was observed on an aarch64 platform which behaves correctly but
differently as described above.
Reviewed-by: Jinshan Xiong <jinshan.xiong@intel.com> Reviewed-by: Richard Laager <rlaager@wiktel.com> Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #5311
Closes #5313
Brian Behlendorf [Thu, 20 Oct 2016 16:55:03 +0000 (09:55 -0700)]
Fix make distclean Makefile.am removal
The file tests/zfs-tests/tests/stress/Makefile.am gets mistakenly
removed by the distclean target because it's empty. Adding a
`SUBDIRS =` line prevents the removal.
This directory is being preserved as the location to add assorted
stress tests. These may include but are not limited to.
Brian Behlendorf [Tue, 18 Oct 2016 23:49:23 +0000 (23:49 +0000)]
Linux 4.9 compat: inode_change_ok() renamed setattr_prepare()
In torvalds/linux@31051c8 the inode_change_ok() function was
renamed setattr_prepare() and updated to take a dentry ratheri
than an inode. Update the code to call the setattr_prepare()
and add a wrapper function which call inode_change_ok() for
older kernels.
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov> Signed-off-by: Chunwei Chen <david.chen@osnexus.com>
Requires-spl: refs/pull/581/head
Chunwei Chen [Wed, 19 Oct 2016 18:19:17 +0000 (11:19 -0700)]
Linux 4.9 compat: remove iops->{set,get,remove}xattr
In Linux 4.9, torvalds/linux@fd50eca, iops->{set,get,remove}xattr and
generic_{set,get,remove}xattr are removed. xattr operations will directly
go through sb->s_xattr.
Tony Hutter [Wed, 19 Oct 2016 19:55:59 +0000 (12:55 -0700)]
Multipath autoreplace, control enclosure LEDs, event rate limiting
1. Enable multipath autoreplace support for FMA.
This extends FMA autoreplace to work with multipath disks. This
requires libdevmapper to be installed at build time.
2. Turn on/off fault LEDs when VDEVs become degraded/faulted/online
Set ZED_USE_ENCLOSURE_LEDS=1 in zed.rc to have ZED turn on/off the enclosure
LED for a drive when a drive becomes FAULTED/DEGRADED. Your enclosure must
be supported by the Linux SES driver for this to work. The enclosure LED
scripts work for multipath devices as well. The scripts will clear the LED
when the fault is cleared.
3. Rate limit ZIO delay and checksum events so as not to flood ZED
ZIO delay and checksum events are rate limited to 5/sec in the zfs module.
Reviewed-by: Richard Laager <rlaager@wiktel.com>
Reviewed by: Don Brady <don.brady@intel.com> Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov> Signed-off-by: Tony Hutter <hutter2@llnl.gov>
Closes #2449
Closes #3017
Closes #5159
luozhengzheng [Tue, 18 Oct 2016 18:32:59 +0000 (02:32 +0800)]
Fix coverity defects: CID 150926
CID 150926: Unchecked return value (CHECKED_RETURN)
- This case cannot occur given the existing taskq implementation
and flags passed to task_dispatch().
Brian Behlendorf [Tue, 18 Oct 2016 17:44:44 +0000 (10:44 -0700)]
Fix unused variable
Accidentally introduced by 3dfb57a, when building with debugging
disabled several variables are unused. Resolve this by wrapping
them in ASSERTV to remove them for non-debug builds.
Reviewed by: Don Brady <don.brady@intel.com> Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #5284
Generate a symlink in the current working directory to
test-runner.py output. This will make it easier for the
ZFS buildbot to collect logs.
Reviewed by: John Kennedy <john.kennedy@delphix.com> Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov> Signed-off-by: Giuseppe Di Natale <dinatale2@llnl.gov>
Closes #5293
luozhengzheng [Mon, 17 Oct 2016 19:03:52 +0000 (03:03 +0800)]
Fix coverity defects: CID 150924
CID 150924: Unchecked return value (CHECKED_RETURN)
- On taskq_dispatch failure the reference must be dropped and
this entry can be safely skipped. This case should be impossible
in the existing implementation but should be handled regardless.
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov> Signed-off-by: luozhengzheng <luo.zhengzheng@zte.com.cn>
Closes #5278
Rudd-O [Mon, 17 Oct 2016 18:51:15 +0000 (18:51 +0000)]
Properly use the Dracut cleanup hook to order pool shutdown
When Dracut starts up, it needs to determine whether a pool will remain
"hanging open" before the system shuts off. In such a case, then the
code to clean up the pool (using the previous export -F work) must
be invoked. Since Dracut has had a recent change that makes
mount-zfs.sh simply not run when the root dataset is already mounted,
we must use the cleanup hook to order Dracut to do shutdown cleanup.
Important note: this code will not accomplish its stated goal until this
bug is fixed: https://bugzilla.redhat.com/show_bug.cgi?id=1385432
That bug impacts more than just ZFS. It impacts LUKS, dmraid, and
unmount during poweroff. It is a Fedora-wide bug.
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov> Signed-off-by: Manuel Amador (Rudd-O) <rudd-o@rudd-o.com>
Closes #5287
Håkan Johansson [Mon, 17 Oct 2016 18:46:35 +0000 (20:46 +0200)]
Pass status_cbdata_t to print_status_config() and friends
First rename spare_cbdata_t cb -> spare_cb in print_status_config(),
to free up cb.
Using the structure removes the explicit parameters namewidth
and name_flags from several functions. Also use status_cbdata_t
for print_import_config(). This simplifies print_logs().
Remove the parameter 'verbose' for print_logs(). It does not really
mean verbose, it selected between the print_status_config and
print_import_config() paths. This selection is now done by
cb_print_config of spare_cbdata_t.
Reviewed-by: Tony Hutter <hutter2@llnl.gov> Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov> Signed-off-by: Håkan Johansson <f96hajo@chalmers.se>
Closes #5259
Brian Behlendorf [Fri, 14 Oct 2016 23:11:16 +0000 (16:11 -0700)]
Allow partition aliases in vdev_id.conf (#5266)
When pools are assembled from partitions, vdev_id.conf aliases
do not work. The directory /dev/disk/by-vdev is not created because
the associated udev rule for parsing vdev_id.conf is never called.
Extend to logic to match "disk" and "partition".
Patch-proposed-by: @sparksh Reviewed-by: Ned Bass <bass6@llnl.gov> Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #3859
Closes #5266
Reviewed-by: Giuseppe Di Natale <dinatale2@llnl.gov> Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov> Signed-off-by: cao.xuewen <cao.xuewen@zte.com.cn>
Closes #5237
OpenZFS 6877 - zfs_rename_006_pos fails due to missing zvol snapshot device file
Authored by: Akash Ayare <aayare@delphix.com>
Reviewed by: John Kennedy <john.kennedy@delphix.com>
Reviewed by: Matthew Ahrens <mahrens@delphix.com>
Reviewed by: Dan Kimmel <dan.kimmel@delphix.com> Reviewed-by: luozhengzheng <luo.zhengzheng@zte.com.cn> Reviewed-by: yuxiang <guo.yong33@zte.com.cn> Ported-by: Brian Behlendorf <behlendorf1@llnl.gov>
Bug was caused due to a change in functionality. At some point, ZFS
snapshots no longer created associated device files which were being
used in the test. To resolve this issue, a clone of the snapshot can be
produced which will also create the expected device files; then, the
test will behave as it did historically.
liaoyuxiangqin [Fri, 14 Oct 2016 16:33:51 +0000 (00:33 +0800)]
Enable quota_002_pos, quota_004_pos and quota_005_pos
In this test the 'ls -ls' command was used to print testfile size in
blocks. Because the environment variable BLOCK_SIZE was set
the 'ls -ls' command detected this and output its block count as the
number of 8192 blocks. Rather than change the variable name
the -k was was added to force ls to return 1k blocks. This has the
additional advantage of behaving consistently across platforms.
For additional details on GNU 'ls' behavior regarding block size:
Don Brady [Fri, 14 Oct 2016 00:59:18 +0000 (18:59 -0600)]
OpenZFS 7090 - zfs should throttle allocations
OpenZFS 7090 - zfs should throttle allocations
Authored by: George Wilson <george.wilson@delphix.com>
Reviewed by: Alex Reece <alex@delphix.com>
Reviewed by: Christopher Siden <christopher.siden@delphix.com>
Reviewed by: Dan Kimmel <dan.kimmel@delphix.com>
Reviewed by: Matthew Ahrens <mahrens@delphix.com>
Reviewed by: Paul Dagnelie <paul.dagnelie@delphix.com>
Reviewed by: Prakash Surya <prakash.surya@delphix.com>
Reviewed by: Sebastien Roy <sebastien.roy@delphix.com>
Approved by: Matthew Ahrens <mahrens@delphix.com> Ported-by: Don Brady <don.brady@intel.com> Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
When write I/Os are issued, they are issued in block order but the ZIO
pipeline will drive them asynchronously through the allocation stage
which can result in blocks being allocated out-of-order. It would be
nice to preserve as much of the logical order as possible.
In addition, the allocations are equally scattered across all top-level
VDEVs but not all top-level VDEVs are created equally. The pipeline
should be able to detect devices that are more capable of handling
allocations and should allocate more blocks to those devices. This
allows for dynamic allocation distribution when devices are imbalanced
as fuller devices will tend to be slower than empty devices.
The change includes a new pool-wide allocation queue which would
throttle and order allocations in the ZIO pipeline. The queue would be
ordered by issued time and offset and would provide an initial amount of
allocation of work to each top-level vdev. The allocation logic utilizes
a reservation system to reserve allocations that will be performed by
the allocator. Once an allocation is successfully completed it's
scheduled on a given top-level vdev. Each top-level vdev maintains a
maximum number of allocations that it can handle (mg_alloc_queue_depth).
The pool-wide reserved allocations (top-levels * mg_alloc_queue_depth)
are distributed across the top-level vdevs metaslab groups and round
robin across all eligible metaslab groups to distribute the work. As
top-levels complete their work, they receive additional work from the
pool-wide allocation queue until the allocation queue is emptied.
GeLiXin [Thu, 13 Oct 2016 21:02:07 +0000 (05:02 +0800)]
Fix coverity defects: CID 147654, 147690
coverity scan CID:147654,type: Copy into fixed size buffer
- string operation may write past the end of the fixed-size
destination buffer
coverity scan CID:147690,type: Uninitialized scalar variable
- call zfs_prop_get first in case we use sourcetype and
share_sourcetype without initialization
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov> Signed-off-by: GeLiXin <ge.lixin@zte.com.cn>
Closes #5253
Tom Caputi [Wed, 12 Oct 2016 19:52:30 +0000 (15:52 -0400)]
Fix ICP memleak introduced in #4760
The ICP requires destructors to for each crypto module that is added.
These do not necessarily exist in Illumos because they assume that
these modules can never be unloaded from the kernel. Some of this
cleanup code was missed when #4760 was merged, resulting in leaks.
This patch simply fixes that.
Reviewed-by: Tony Hutter <hutter2@llnl.gov> Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov> Signed-off-by: Tom Caputi <tcaputi@datto.com>
Issue #4760
Closes #5265
cao [Tue, 11 Oct 2016 18:32:34 +0000 (02:32 +0800)]
Fix coverity defects: CID 147452, 147447, 147446
coverity scan CID:147452, Type:Unchecked return value from library
coverity scan CID:147447, Type:Unchecked return value from library
coverity scan CID:147446, Type:Unchecked return value from library
Reviewed-by: Giuseppe Di Natale <dinatale2@llnl.gov> Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov> Signed-off-by: cao.xuewen <cao.xuewen@zte.com.cn>
Closes #5264
Brian Behlendorf [Tue, 11 Oct 2016 16:32:31 +0000 (09:32 -0700)]
Enable zfs_snapshot_008_neg and zfs_snapshot_009_pos (#5260)
The zfs_snapshot_008_neg test case does not use nested pools and
can be safely enabled. The zfs_snapshot_009_pos test case is
also passing without modification.
Reviewed-by: Richard Laager <rlaager@wiktel.com> Reviewed-by: Giuseppe Di Natale <dinatale2@llnl.gov> Reviewed-by: ChaoyuZhang <zhang.chaoyu@zte.com.cn> Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #5260
GeLiXin [Mon, 10 Oct 2016 22:30:22 +0000 (06:30 +0800)]
Fix coverity defects: CID 147639
When array is passed as a parameter it degenerates into a
pointer so the sizeof(path) in is_shorthand_path() and always
get return value of 8, instead of the string length we want.
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov> Signed-off-by: GeLiXin <ge.lixin@zte.com.cn>
Closes #5198
Tim Chase [Mon, 10 Oct 2016 22:19:14 +0000 (17:19 -0500)]
Write issue taskq shouldn't be dynamic
This is as much an upstream compatibility as it's a bit of a performance
gain.
The illumos taskq implemention doesn't allow a TASKQ_THREADS_CPU_PCT type
to be dynamic and in fact enforces as much with an ASSERT.
As to performance, if this taskq is dynamic, it can cause excessive
contention on tq_lock as the threads are created and destroyed because it
can see bursts of many thousands of tasks in a short time, particularly
in heavy high-concurrency zvol write workloads.
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov> Signed-off-by: Tim Chase <tim@chase2k.com>
Closes #5236
Tom Caputi [Mon, 10 Oct 2016 18:34:57 +0000 (14:34 -0400)]
Porting over some ICP code that was missed in #4760
When #4760 was merged tests were added to ensure that the new checksums
were working properly. However, some of the functionality for sha2
functions were not ported over, resulting in some Coverity defects and
code that would be unstable when needed in the future. This patch
simply ports over the missing code and fixes the defects in the
process.
Reviewed by: Brian Behlendorf <behlendorf1@llnl.gov> Signed-off-by: Tom Caputi <tcaputi@datto.com>
Issue #4760
Closes #5251
Stian Ellingsen [Thu, 6 Oct 2016 18:03:41 +0000 (20:03 +0200)]
Use env, not sh in zfsctl_snapshot_{,un}mount()
Call mount and umount via /usr/bin/env instead of /bin/sh in
zfsctl_snapshot_mount() and zfsctl_snapshot_unmount().
This change fixes a shell code injection flaw. The call to /bin/sh
passed the mountpoint unescaped, only surrounded by single quotes. A
mountpoint containing one or more single quotes would cause the command
to fail or potentially execute arbitrary shell code.
This change also provides compatibility with grsecurity patches.
Grsecurity only allows call_usermodehelper() to use helper binaries in
certain paths. /usr/bin/* is allowed, /bin/* is not.
OpenZFS decided that ignore_hole_birth was too imprecise and
incorrect a name (and went with send_holes_without_birth_time).
Rename it in ZoL too, while keeping the name "ignore_hole_birth"
pointing to the same variable for existing consumers.
Reviewed-by: Giuseppe Di Natale <dinatale2@llnl.gov> Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov> Signed-off-by: Rich Ercolani <rincebrain@gmail.com>
Closes #5239
Håkan Johansson [Fri, 7 Oct 2016 20:25:35 +0000 (22:25 +0200)]
Fix vdev_open_child() race on updating vdev_parent->vdev_nonrot
Updating vd->vdev_parent->vdev_nonrot in vdev_open_child()
is a race when vdev_open_child is called for many children
from a task queue.
vdev_open_child() is only called by vdev_open_children(), let
the latter update the parent vdev_nonrot member. The update
was already there, so done twice previously. Thus using the
same logic at the end in vdev_open_children() to update
vdev_nonrot, either we are vdev_uses_zvols() or not.
Reviewed-by: Richard Elling <Richard.Elling@RichardElling.com> Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov> Signed-off-by: Haakan T Johansson <f96hajo@chalmers.se>
Closes #5162
Introduce a make recipe for flake8 to enable python
style checking. Ensure all python scripts pass flake8.
Return an error code of 0 for arcstat.py -v and
dbufstat.py -v. Add test cases for python scripts.
Reviewed by: Richard Laager <rlaager@wiktel.com> Reviewed-by: Richard Elling <Richard.Elling@RichardElling.com> Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov> Reviewed-by: Ian Lee <IanLee1521@gmail.com>
Closes #5230
OpenZFS 6988 spa_sync() spends half its time in dmu_objset_do_userquota_updates
Using a benchmark which creates 2 million files in one TXG, I observe
that the thread running spa_sync() is on CPU almost the entire time we
are syncing, and therefore can be a performance bottleneck. About 50% of
the time in spa_sync() is in dmu_objset_do_userquota_updates().
The problem is that dmu_objset_do_userquota_updates() calls
zap_increment_int(DMU_USERUSED_OBJECT) once for every file that was
modified (or created). In this benchmark, all the files are owned by the
same user/group, so all 2 million calls to zap_increment_int() are
modifying the same entry in the zap. The same issue exists for the
DMU_GROUPUSED_OBJECT.
We should keep an in-memory map from user to space delta while we are
syncing, and when we finish, iterate over the in-memory map and modify
the ZAP once per entry. This reduces the number of calls to
zap_increment_int() from "number of objects modified" to "number of
owners/groups of modified files".
This reduced the time spent in spa_sync() in the file create benchmark
by ~33%, from 11 seconds to 7 seconds.
Upstream bugs: DLPX-44799
Ported by: Ned Bass <bass6@llnl.gov>
Porting notes:
- Added curly braces around declaration of userquota_cache_t cache to
quiet compiler warning;
- Handled the userobj accounting the same way it proposed in this path.
Jinshan Xiong [Tue, 4 Oct 2016 18:46:10 +0000 (11:46 -0700)]
Add support for user/group dnode accounting & quota
This patch tracks dnode usage for each user/group in the
DMU_USER/GROUPUSED_OBJECT ZAPs. ZAP entries dedicated to dnode
accounting have the key prefixed with "obj-" followed by the UID/GID
in string format (as done for the block accounting).
A new SPA feature has been added for dnode accounting as well as
a new ZPL version. The SPA feature must be enabled in the pool
before upgrading the zfs filesystem. During the zfs version upgrade,
a "quotacheck" will be executed by marking all dnode as dirty.
projects / zfs / log