From: Ben Roberts <ben.roberts@gsacapital.com>
To: Trond Myklebust <trondmy@kernel.org>, Anna Schumaker <anna@kernel.org>
Cc: linux-nfs@vger.kernel.org, linux-kernel@vger.kernel.org,
Ben Roberts <ben.roberts@gsacapital.com>
Subject: [PATCH v2] pNFS: deadlock in pnfs_send_layoutreturn
Date: Wed, 8 Apr 2026 13:25:34 +0100 [thread overview]
Message-ID: <20260408122534.537816-1-ben.roberts@gsacapital.com> (raw)
On a HPC cluster running 5.14.0-611.9.1.el9.x86_64, regular deadlocks were
seen within pnfs_send_layoutreturn leading to userspace processes stuck in
uninterruptible sleep, ultimately requiring reboots to clear. This was
occurring frequently, sometimes multiple times per day on specific hosts
with heavy load. Claude code was tasked with hunting down any potential
deadlocks within pnfs_send_layoutreturn, and identified the following
condition. This patch has been running in production on top of the EL9
kernel for over three months without any reoccurrence of the deadlock.
The pnfs_send_layoutreturn() function can deadlock when memory
allocation fails. The issue occurs in the error path where
pnfs_put_layout_hdr() is called, which may trigger
pnfs_layoutreturn_before_put_layout_hdr(), potentially causing
a recursive call back to pnfs_send_layoutreturn().
Call chain that triggers the deadlock:
1. pnfs_send_layoutreturn() - kzalloc() fails
2. Error path calls pnfs_put_layout_hdr(lo)
3. pnfs_put_layout_hdr() calls pnfs_layoutreturn_before_put_layout_hdr()
4. If NFS_LAYOUT_RETURN_REQUESTED is still set, attempts another
layoutreturn, creating recursion/deadlock
The fix ensures that NFS_LAYOUT_RETURN_REQUESTED is cleared in the
allocation failure path before calling pnfs_put_layout_hdr(). This
prevents pnfs_layoutreturn_before_put_layout_hdr() from attempting
another layout return, breaking the recursion cycle.
v2 fixes a syntax error introduced while composing the original mail.
Signed-off-by: Ben Roberts <ben.roberts@gsacapital.com>
Assisted-by: Claude:claude-sonnet-4-5
---
fs/nfs/pnfs.c | 1 +
1 file changed, 1 insertion(+)
diff --git a/fs/nfs/pnfs.c b/fs/nfs/pnfs.c
index bc13d1e69449..47bda53b2b3a 100644
--- a/fs/nfs/pnfs.c
+++ b/fs/nfs/pnfs.c
@@ -1361,6 +1361,7 @@ pnfs_send_layoutreturn(struct pnfs_layout_hdr *lo,
if (unlikely(lrp == NULL)) {
status = -ENOMEM;
spin_lock(&ino->i_lock);
+ pnfs_clear_layoutreturn_info(lo);
pnfs_clear_layoutreturn_waitbit(lo);
spin_unlock(&ino->i_lock);
put_cred(cred);
--
2.43.0
For details of how GSA uses your personal information, please see our Privacy Notice here: https://www.gsacapital.com/privacy-notice
This email and any files transmitted with it contain confidential and proprietary information and is solely for the use of the intended recipient.
If you are not the intended recipient please return the email to the sender and delete it from your computer and you must not use, disclose, distribute, copy, print or rely on this email or its contents.
This communication is for informational purposes only.
It is not intended as an offer or solicitation for the purchase or sale of any financial instrument or as an official confirmation of any transaction.
Any comments or statements made herein do not necessarily reflect those of GSA Capital.
GSA Capital Partners LLP is authorised and regulated by the Financial Conduct Authority and is registered in England and Wales at Stratton House, 5 Stratton Street, London W1J 8LA, number OC309261.
GSA Capital Services Limited is registered in England and Wales at the same address, number 5320529.
next reply other threads:[~2026-04-08 12:25 UTC|newest]
Thread overview: 6+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-04-08 12:25 Ben Roberts [this message]
2026-04-10 14:30 ` Benjamin Coddington
2026-04-17 10:17 ` Ben Roberts
2026-04-17 16:19 ` Benjamin Coddington
2026-04-22 10:46 ` Ben Roberts
2026-04-22 12:52 ` Trond Myklebust
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20260408122534.537816-1-ben.roberts@gsacapital.com \
--to=ben.roberts@gsacapital.com \
--cc=anna@kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-nfs@vger.kernel.org \
--cc=trondmy@kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®