Fix lockdep recursive locking false positive in dbuf_destroy

lockdep reports a possible recursive lock in dbuf_destroy.

It is true that dbuf_destroy is acquiring the dn_dbufs_mtx
on one dnode while holding it on another dnode.  However,
it is impossible for these to be the same dnode because,
among other things,dbuf_destroy checks MUTEX_HELD before
acquiring the mutex.

This fix defines a class NESTED_SINGLE == 1 and changes
that lock to call mutex_enter_nested with a subclass of
NESTED_SINGLE.

In order to make the userspace code compile,
include/sys/zfs_context.h now defines mutex_enter_nested and
NESTED_SINGLE.

This is the lockdep report:

[  122.950921] ============================================
[  122.950921] WARNING: possible recursive locking detected
[  122.950921] 4.19.29-4.19.0-debug-d69edad5368c1166 #1 Tainted: G           O
[  122.950921] --------------------------------------------
[  122.950921] dbu_evict/1457 is trying to acquire lock:
[  122.950921] 0000000083e9cbcf (&dn->dn_dbufs_mtx){+.+.}, at: dbuf_destroy+0x3c0/0xdb0 [zfs]
[  122.950921]
               but task is already holding lock:
[  122.950921] 0000000055523987 (&dn->dn_dbufs_mtx){+.+.}, at: dnode_evict_dbufs+0x90/0x740 [zfs]
[  122.950921]
               other info that might help us debug this:
[  122.950921]  Possible unsafe locking scenario:

[  122.950921]        CPU0
[  122.950921]        ----
[  122.950921]   lock(&dn->dn_dbufs_mtx);
[  122.950921]   lock(&dn->dn_dbufs_mtx);
[  122.950921]
                *** DEADLOCK ***

[  122.950921]  May be due to missing lock nesting notation

[  122.950921] 1 lock held by dbu_evict/1457:
[  122.950921]  #0: 0000000055523987 (&dn->dn_dbufs_mtx){+.+.}, at: dnode_evict_dbufs+0x90/0x740 [zfs]
[  122.950921]
               stack backtrace:
[  122.950921] CPU: 0 PID: 1457 Comm: dbu_evict Tainted: G           O      4.19.29-4.19.0-debug-d69edad5368c1166 #1
[  122.950921] Hardware name: Supermicro H8SSL-I2/H8SSL-I2, BIOS 080011  03/13/2009
[  122.950921] Call Trace:
[  122.950921]  dump_stack+0x91/0xeb
[  122.950921]  __lock_acquire+0x2ca7/0x4f10
[  122.950921]  lock_acquire+0x153/0x330
[  122.950921]  dbuf_destroy+0x3c0/0xdb0 [zfs]
[  122.950921]  dbuf_evict_one+0x1cc/0x3d0 [zfs]
[  122.950921]  dbuf_rele_and_unlock+0xb84/0xd60 [zfs]
[  122.950921]  dnode_evict_dbufs+0x3a6/0x740 [zfs]
[  122.950921]  dmu_objset_evict+0x7a/0x500 [zfs]
[  122.950921]  dsl_dataset_evict_async+0x70/0x480 [zfs]
[  122.950921]  taskq_thread+0x979/0x1480 [spl]
[  122.950921]  kthread+0x2e7/0x3e0
[  122.950921]  ret_from_fork+0x27/0x50

Reviewed-by: Tony Hutter <hutter2@llnl.gov>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Jeff Dike <jdike@akamai.com>
Closes #8984

diff --git a/include/spl/sys/mutex.h b/include/spl/sys/mutex.h
index ed0cd4932cfa823702b0a903a8be1cc79d27f687..a61f35c61eb118457cd9e2a8a83b6e2bdef2060f 100644 (file)
--- a/include/spl/sys/mutex.h
+++ b/include/spl/sys/mutex.h
@@ -127,6 +127,8 @@ spl_mutex_lockdep_on_maybe(kmutex_t *mp)                    \
 })
 /* END CSTYLED */
 
+#define        NESTED_SINGLE 1
+
 #ifdef CONFIG_DEBUG_LOCK_ALLOC
 #define        mutex_enter_nested(mp, subclass)                        \
 {                                                              \

diff --git a/include/sys/zfs_context.h b/include/sys/zfs_context.h
index 224f5cb83177143a91feb71100241470a7498939..60a2484867447371393af1d531e97597b64fc742 100644 (file)
--- a/include/sys/zfs_context.h
+++ b/include/sys/zfs_context.h
@@ -258,6 +258,8 @@ extern void mutex_enter(kmutex_t *mp);
 extern void mutex_exit(kmutex_t *mp);
 extern int mutex_tryenter(kmutex_t *mp);
 
+#define        NESTED_SINGLE 1
+#define        mutex_enter_nested(mp, class) mutex_enter(mp)
 /*
  * RW locks
  */

diff --git a/module/zfs/dbuf.c b/module/zfs/dbuf.c
index 31b9b1481b86fccbfe3bc93f73f9674b2f4a6c29..4d347b6f42ca9ebbea0c8f5f4a01a146392dc7d2 100644 (file)
--- a/module/zfs/dbuf.c
+++ b/module/zfs/dbuf.c
@@ -2690,7 +2690,8 @@ dbuf_destroy(dmu_buf_impl_t *db)
        if (db->db_blkid != DMU_BONUS_BLKID) {
                boolean_t needlock = !MUTEX_HELD(&dn->dn_dbufs_mtx);
                if (needlock)
-                       mutex_enter(&dn->dn_dbufs_mtx);
+                       mutex_enter_nested(&dn->dn_dbufs_mtx,
+                           NESTED_SINGLE);
                avl_remove(&dn->dn_dbufs, db);
                atomic_dec_32(&dn->dn_dbufs_count);
                membar_producer();