From: Benjamin Coddington <ben.coddington@hammerspace.com>
To: Ben Roberts <ben.roberts@gsacapital.com>
Cc: Benjamin Coddington <ben.coddington@hammerspace.com>,
Trond Myklebust <trondmy@kernel.org>,
Anna Schumaker <anna@kernel.org>,
linux-nfs@vger.kernel.org, linux-kernel@vger.kernel.org
Subject: Re: [PATCH v2] pNFS: deadlock in pnfs_send_layoutreturn
Date: Fri, 17 Apr 2026 09:19:25 -0700 [thread overview]
Message-ID: <14A7BAF5-7D96-4C34-9FEE-CA929267A174@hammerspace.com> (raw)
In-Reply-To: <AM8PR04MB7764059A4CC2893C16A3180E8A202@AM8PR04MB7764.eurprd04.prod.outlook.com>
On 17 Apr 2026, at 3:17, Ben Roberts wrote:
> Hi Ben,
>
>> Did you reproduce and diagnose this problem on a recent upstream kernel
>> version?
>
> I'm not able to switch the production systems to a more recent kernel at
> this time, and don't have a reliable way to reproduce the issue in the
> wild without risking production systems back on an unpatched kernel. The
> best evidence I have that this patch is needed, is that we were seeing
> this deadlock occur repeatedly under high loads and memory pressure last
> year before applying this patch locally. It was rolled out to all
> production systems in early Jan and we have not seen a single reoccurrence
> since. The relevant code paths look similar between the modified EL9
> kernel and the current git HEAD.
>
> I've spent all this week trying to devise a precise reproduction (with a
> lot of help from an LLM since I'm not that familar with kernel
> development) on both 7.0.0-rc7 (9a9c8ce300cd) and 5.14.0-611.9.1 kernels
> to definitively prove this patch is needed, but without success. The
> initial analysis suggested the deadlock might be triggered from a single
> process via a recursive call. This theory has been ruled out; all calling
> paths triggered by a single process are guarded in such a way that a
> recursive call does not happen. Revised theory for why this patch helps
> is that the deadlock is subject to an inter-process race, where one
> process suffers a memory allocation failure, causing a second process to
> become a victim of bad state, leading to an unserviceable RPC request.
> The timing appears to be very sensitive and hard to reproduce on demand.
>
> The LLM-assisted analysis follows, and to my non-expert eyes seems
> compelling.
>
> Thanks,
> Ben
The kzalloc failure is definitely a rarely-used/tested path, so its possible
there's an issue there no one has seen yet, but from what I can see it looks
like every call to pnfs_send_layoutreturn() first calls
pnfs_prepare_layoutreturn(), which already clears
NFS_LAYOUT_RETURN_REQUESTED. I don't see how you can end up with another
proccess seeing the flag.
There's at least one body of work in this area that your systems
don't yet have:
https://lore.kernel.org/linux-nfs/20240613050055.854323-1-trond.myklebust@hammerspace.com/
Again, I strongly recommend engaging RH support here.
Ben
next prev parent reply other threads:[~2026-04-17 16:19 UTC|newest]
Thread overview: 6+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-04-08 12:25 Ben Roberts
2026-04-10 14:30 ` Benjamin Coddington
2026-04-17 10:17 ` Ben Roberts
2026-04-17 16:19 ` Benjamin Coddington [this message]
2026-04-22 10:46 ` Ben Roberts
2026-04-22 12:52 ` Trond Myklebust
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=14A7BAF5-7D96-4C34-9FEE-CA929267A174@hammerspace.com \
--to=ben.coddington@hammerspace.com \
--cc=anna@kernel.org \
--cc=ben.roberts@gsacapital.com \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-nfs@vger.kernel.org \
--cc=trondmy@kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®