From: wujing <realwujing@qq.com>
To: jgg@ziepe.ca
Cc: leon@kernel.org, linux-kernel@vger.kernel.org,
linux-rdma@vger.kernel.org, realwujing@qq.com,
yuanql9@chinatelecom.cn
Subject: Re: [PATCH] IB/core: Fix ABBA deadlock in rdma_dev_exit_net
Date: Tue, 16 Dec 2025 17:59:52 +0800 [thread overview]
Message-ID: <tencent_713807A8D67394A5D8339F8AD33FCCBFCE07@qq.com> (raw)
In-Reply-To: <20251216005705.GB31492@ziepe.ca>
[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset=y, Size: 2174 bytes --]
Hi Jason,
You're right that the locks aren't nested in rdma_dev_exit_net() - it does release
rdma_nets_rwsem before acquiring devices_rwsem. However, this is still an ABBA deadlock,
just not the trivial nested kind. The issue is caused by **rwsem writer priority**
and lock ordering inconsistency.
Here's the actual deadlock scenario:
**Thread A (rdma_dev_exit_net - cleanup_net workqueue):**
```
down_write(&rdma_nets_rwsem); // Acquired
xa_store(&rdma_nets, ...);
up_write(&rdma_nets_rwsem); // Released
down_read(&devices_rwsem); // Waiting here <-- BLOCKED
```
**Thread B (rdma_dev_init_net - stress-ng-clone):**
```
down_read(&devices_rwsem); // Acquired
down_read(&rdma_nets_rwsem); // Waiting here <-- BLOCKED
```
The deadlock happens because:
1. Thread A releases rdma_nets_rwsem as a **writer**
2. Thread B (and many others) are waiting to acquire rdma_nets_rwsem as **readers**
3. Thread A then tries to acquire devices_rwsem as a reader
4. BUT: rwsem gives priority to pending writers over new readers
5. Since Thread A was a pending writer on rdma_nets_rwsem, Thread B's read request is blocked
6. Thread B holds devices_rwsem, which Thread A needs
7. Thread A holds the "writer priority slot" on rdma_nets_rwsem, which Thread B needs
This is a **priority inversion deadlock**, not a simple nested lock deadlock.
The production crash log shows exactly this:
- Thread A: `rdma_dev_exit_net+0x60` stuck in `rwsem_down_write_slowpath` trying to get devices_rwsem
- Thread B: `rdma_dev_init_net+0x120` stuck in `rwsem_down_read_slowpath` trying to get rdma_nets_rwsem
Lockdep doesn't catch this because:
1. The locks aren't held simultaneously (no nested locking)
2. It's a reader-writer priority issue, not a simple lock ordering issue
3. It requires specific timing: writer releases lock, then tries to acquire another
lock that readers (waiting for the first lock) already hold
The fix ensures both paths acquire locks in the same order:
- rdma_dev_init_net: devices_rwsem → rdma_nets_rwsem
- rdma_dev_exit_net: devices_rwsem → rdma_nets_rwsem (was reversed)
This eliminates the priority inversion scenario.
Best regards
next prev parent reply other threads:[~2025-12-16 10:00 UTC|newest]
Thread overview: 5+ messages / expand[flat|nested] mbox.gz Atom feed top
2025-12-11 8:08 wujing
2025-12-16 0:57 ` Jason Gunthorpe
2025-12-16 9:59 ` wujing [this message]
2025-12-16 13:59 ` Michael Gur
2025-12-16 14:22 ` Jason Gunthorpe
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=tencent_713807A8D67394A5D8339F8AD33FCCBFCE07@qq.com \
--to=realwujing@qq.com \
--cc=jgg@ziepe.ca \
--cc=leon@kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-rdma@vger.kernel.org \
--cc=yuanql9@chinatelecom.cn \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®