mirror of https://lore.kernel.org/lkml/
 help / color / mirror / Atom feed
* [PATCH 0/6] nfsd: layout recall and lease break fixes
@ 2026-10-06  4:29 Daejun Park via B4 Relay
  2026-10-06  4:29 ` [PATCH 1/6] nfsd: make the time limit of a layout recall work Daejun Park via B4 Relay
                   ` (7 more replies)
  0 siblings, 8 replies; 9+ messages in thread
From: Daejun Park via B4 Relay @ 2026-10-06  4:29 UTC (permalink / raw)
  To: Chuck Lever, Jeff Layton, NeilBrown, Olga Kornievskaia, Dai Ngo,
	Tom Talpey
  Cc: Christoph Hellwig, Benjamin Coddington, Trond Myklebust,
	Anna Schumaker, linux-nfs, linux-kernel, Daejun Park

nfsd polls a client that has accepted a CB_LAYOUTRECALL by sending the
callback again every 10 ms, with the stateid of the first callback.  A
Linux client from v6.8 on answers the repeat with NFS4ERR_OLD_STATEID,
and nfsd takes that as a failed recall.  So a client that needs more
than 10 ms to return a layout has the layout revoked and the fence path
run.  With the block layout that client loses data it has written and
synced, with the SCSI layout it is cut off from the device.

Patch 3 lets nfsd keep polling instead.  It needs the two patches
before it.  Patch 1: the limit of two lease periods on that poll has
never triggered, and without it the poll would have no end.  Patch 2:
for the block layout a lease break gives up after fs.lease-break-time
(45 s) without running /sbin/nfsd-recall-failed.  Until now the recall
had failed long before that; with patch 3 it is polled for two lease
periods (180 s).  Patches 1 to 3 have to go together, also to stable
kernels.  Kernels before v7.1 do not have the lease break timeout that
patch 2 fixes; there the operation on the server waits for the recall,
for up to two lease periods, and that operation can be an nfsd thread
serving a WRITE of another client.  Whether that is wanted in older
stable kernels is for you to say; it is how they already treat a client
that keeps answering NFS4ERR_DELAY, then without any limit.

Patch 2 fences the client at fs.lease-break-time, as the SCSI layout
does since v7.1.  By default that is shorter than the one lease period
RFC 8881 section 12.5.5 has the server wait after a recall.

Patch 4 fixes a lease break that hits a LAYOUTGET in progress and then
waits for a layout that is never recalled.  It showed up as writes on
the server that hung for seconds in the tests for the other patches.

Patches 5 and 6 deal with two answers that the polls handle badly.  A
client that a LAYOUTRETURN has moved past the seqid of the recall
answers every poll NFS4ERR_OLD_STATEID and is never asked for the
layouts it still holds; patch 5 polls such a client, one that has not
answered NFS4_OK, with a new seqid.  A client that answers
NFS4ERR_BAD_STATEID after returning its last layout is fenced; patch 6
looks for layouts first.  Neither showed up in the runs below.  Patch 5
builds on patches 3 and 4 and should go to stable kernels with them;
patch 6 stands on its own.

Tests.  QEMU guests on one host: an nvmet-tcp target, the server (nfsd,
XFS on the nvmet namespace, pnfs export, 8 nfsd threads) and three
clients, all on nfsd-testing 32eb1a60b456 with KASAN and lockdep, NFS
v4.2, block layout, except in 5.  The clients do O_DIRECT I/O in 1 MiB
requests on one file for 30 seconds.  Each case was run once.  "before"
is that commit, "after" is that commit with patches 1 to 4.  With all
six, the block layout runs give the same results: no recall failed,
and no block read back as zeroes.

1. Recalls that ended in the failure path, out of the recalls sent, on
   a file that was written beforehand:

                                                    before      after
   2 clients read (different halves of the file)    7 / 9       0 / 9
   2 clients write                                  8 / 9       0 / 9
   3 clients write                                  19 / 20     0 / 47
   2 clients write, the server writes 4 KiB to
   the file every 50 ms                             216 / 229   0 / 275

   Over all cases of a run the server logged "failed to respond to
   layout recall" 362 times before and not at all after.

2. Blocks that read back as zeroes on the server, out of the 1 MiB
   blocks the clients wrote to a new file with O_DIRECT, followed by
   one successful fsync():

                                                    before      after
   2 clients write                                  6 / 564     0 / 475
   3 clients write                                  11 / 527    0 / 552
   1 client writes, the server sets the file size
   every second                                     21 / 469    0 / 481
   2 clients write, file preallocated               8 / 601     0 / 575
   1 client writes, 1 client reads                  4 / 493     0 / 37

   (In the last case the reader holds the layout after the patches and
   the writer makes little progress; see the end of this letter.  The
   commit message of patch 3 has the numbers with patches 1 and 3
   only.)

3. One client writing through a block layout whose path to the device
   is cut while it can still talk to the server, and a 4 KiB write to
   the file on the server.  The client's write that is in flight stays
   stuck until the path is back.

                                                    before      after
   path cut for 15 s: the server write takes        0.05 s      15.5 s
     the recall                                     fails       completes
     /sbin/nfsd-recall-failed is run                at once     no
   path cut for 70 s: the server write takes        0.05 s      47.7 s
     /sbin/nfsd-recall-failed is run                at once     at 47.7 s

   Before, the write of the client reaches the device after the write
   on the server has returned, 15 and 70 seconds later.  After, the
   server waits for the client, and when the lease break times out it
   runs the script first (patch 2; with patches 1 and 3 only, the write
   on the server went on after 47.0 s without the script).  With a lease
   time of 15 seconds and the path cut for 70 seconds, the recall fails
   30 seconds after the first answer of the client and the script is
   run (patch 1).

4. A lease break during a LAYOUTGET (patch 4).  Two clients writing
   while the server writes 4 KiB to the file every 50 ms.  One write on
   the server took 9.7 seconds before and 13.9 seconds with patches 1
   to 3, in both runs until the clients were done.  With patch 4 the
   longest write took 56 ms.  With a delay of 100 ms added to LAYOUTGET
   for this test, between the lookup of the layout stateid and the
   insertion of the layout: the second write on the server took 30
   seconds, the whole run, without patch 4, and the longest write took
   108 ms with it.

5. SCSI layout.  The target runs a kernel with "nvmet: fix preempt of
   another registrant by the reservation holder" [1], so that the
   preempt removes the registration.

                                                    before      after
   2 clients write: recalls that fenced the client  2 / 2       0 / 9
   registrants on the namespace afterwards (of 4)   2           4
   2 clients write a new file (fresh boot): blocks
   that read back as zeroes                         1 / 630     0 / 540
   path of a client cut for 15 seconds: the client
   is fenced                                        yes         no

   The fenced clients logged reservation conflicts and did the rest of
   the run through the server.

pynfs BLOCK1, BLOCK2 and BLOCK4 pass before and after (BLOCK3 fails in
both with a TypeError in the test).  The commit messages of patches 5
and 6 have pynfs clients that answer as described there.  No KASAN or
lockdep report on the server, the clients or the target in any run.

Rebased onto nfsd-testing 56589cdb5881 for posting; the numbers above
are from 32eb1a60b456.  The one conflict was with "nfsd: drain pNFS
fence work during state teardown", in
nfsd4_layout_lm_breaker_timedout(): patch 2 now drops only the early
return for a layout type without ->fence_client, and the fenced and
stopped checks under ls_lock stay.  On 56589cdb5881 with these patches
and the two series that go on top of them, case 3 with the path cut for
70 s gives 48.7 s and the script run at that point (0.05 s and at once
without the patches), and the SCSI case with the path cut for 15 s gives
14.9 s with no fencing.  With the SCSI path cut for 70 s, the write on
the server waits 48.3 s and the fence worker fences the client
(registrants 2 to 1).  pynfs gives the same as above.  No KASAN or
lockdep report.

Not changed by these patches:

- nfsd still polls a client that has answered NFS4_OK with the seqid it
  has used, where RFC 8881 section 12.5.3 has the server increment the
  seqid in each request.  Patch 3 only makes nfsd bear the answer
  (patch 5 gives a new seqid only to a client that has not answered
  NFS4_OK), and while it polls, nfsd sends a callback every 10 ms for
  up to two lease periods.  Ending the callback on NFS4_OK and letting the
  LAYOUTRETURN finish the recall would be the cleaner fix; I can look
  at that next.

- CB_LAYOUTRECALL carries no referring call, as CB_RECALL does since
  "NFSD: Send referring calls with CB_RECALL".  When a recall overtakes
  the reply to the LAYOUTGET that created the layout stateid, the Linux
  client answers NFS4ERR_NOMATCHING_LAYOUT, nfsd drops the layout and
  the client uses it afterwards.  With these patches a client that
  writes through a layout still loses a few blocks that way when
  another client writes to the file through the MDS (2 to 3 of about
  165 in 30 seconds, where it lost 35 to 217 before).  I am sending a
  series for that separately, on top of this one.

- Which client makes progress when several clients use one file.  With
  these patches the client that holds the layout still wins, and with
  every recall done properly it wins more thoroughly than before.

[1] https://lore.kernel.org/r/20261002065931epcms2p1cb2c16f98cf3b05f8c859ad92b0ad465@epcms2p1

---
Daejun Park (6):
      nfsd: make the time limit of a layout recall work
      nfsd: run the fence script when a block layout lease break times out
      nfsd: keep polling a layout recall answered with NFS4ERR_OLD_STATEID
      nfsd: fail a LAYOUTGET in progress when the layout lease is broken
      nfsd: poll a layout recall with a new seqid when the client is past it
      nfsd: do not fence a client for a layout recall with no layouts left

 fs/nfsd/nfs4layouts.c | 122 ++++++++++++++++++++++++++++++++++++++++++++------
 fs/nfsd/state.h       |   3 ++
 fs/nfsd/trace.h       |   2 +
 3 files changed, 113 insertions(+), 14 deletions(-)
---
base-commit: 56589cdb58819ce54decedbbfddf231d94b5ce41
change-id: 20261006-nfsd-layout-recall-ba868d33371e

Best regards,
-- 
Daejun Park <daejun7.park@samsung.com>



^ permalink raw reply	[flat|nested] 9+ messages in thread

* [PATCH 1/6] nfsd: make the time limit of a layout recall work
  2026-10-06  4:29 [PATCH 0/6] nfsd: layout recall and lease break fixes Daejun Park via B4 Relay
@ 2026-10-06  4:29 ` Daejun Park via B4 Relay
  2026-10-06  4:29 ` [PATCH 2/6] nfsd: run the fence script when a block layout lease break times out Daejun Park via B4 Relay
                   ` (6 subsequent siblings)
  7 siblings, 0 replies; 9+ messages in thread
From: Daejun Park via B4 Relay @ 2026-10-06  4:29 UTC (permalink / raw)
  To: Chuck Lever, Jeff Layton, NeilBrown, Olga Kornievskaia, Dai Ngo,
	Tom Talpey
  Cc: Christoph Hellwig, Benjamin Coddington, Trond Myklebust,
	Anna Schumaker, linux-nfs, linux-kernel, Daejun Park

From: Daejun Park <daejun7.park@samsung.com>

nfsd4_cb_layout_done() polls a client that has answered CB_LAYOUTRECALL
and still holds layouts, and is meant to give up after two lease
periods: the recall then fails and the client is fenced.  The limit is
computed from task->tk_start.  The poll restarts the RPC task from
->rpc_call_done(), and rpc_exit_task() resets the statistics of a task
that was restarted there, tk_start included.  So tk_start is the time
of the last poll, and the limit is never reached: a client that keeps
answering is polled every 10 ms for as long as it holds the layout.

Count the two lease periods from the first answer of the client, so
that the time in which a callback could not be delivered is not taken
from it.  A layout stateid is recalled once in its life (ls_recalled is
not cleared), so the time needs no reset.

When the limit is reached, the existing failure path is taken.  It does
not look at the result of ->fence_client(), so a SCSI layout client
whose fencing fails is treated as fenced there.

A Linux client before v6.8 answers every poll with NFS4_OK until its
LAYOUTRETURN is prepared (read from the code, not run).  If such a
client does not return the layout, it was polled without end and is now
fenced after two lease periods, which is what commit 6b9b21073d3b
("nfsd: give up on CB_LAYOUTRECALLs after two lease periods") and
commit 851238a22f3b ("nfsd: fix error handling for clients that fail to
return the layout") set out to do.

Found by reading the code.  The limit could only be tested together
with "nfsd: keep polling a layout recall answered with
NFS4ERR_OLD_STATEID", because a current Linux client does not keep
answering a recall with NFS4_OK or NFS4ERR_DELAY: a client writing
through a block layout whose path to the device is cut for 70 seconds
while it can still talk to the server, a lease time of 15 seconds, and
a write to the file on the server.  The recall fails 30 seconds after
the first answer of the client and /sbin/nfsd-recall-failed is run.
With the default lease time the limit is 180 seconds; a recall that
comes from a lease break is not waited for that long, the breaker gives
up after fs.lease-break-time.

Fixes: 6b9b21073d3b ("nfsd: give up on CB_LAYOUTRECALLs after two lease periods")

Signed-off-by: Daejun Park <daejun7.park@samsung.com>
---
 fs/nfsd/nfs4layouts.c | 11 +++++++++--
 fs/nfsd/state.h       |  1 +
 2 files changed, 10 insertions(+), 2 deletions(-)

diff --git a/fs/nfsd/nfs4layouts.c b/fs/nfsd/nfs4layouts.c
index 898776c9d..3731b4db7 100644
--- a/fs/nfsd/nfs4layouts.c
+++ b/fs/nfsd/nfs4layouts.c
@@ -257,6 +257,7 @@ nfsd4_alloc_layout_stateid(struct nfsd4_compound_state *cstate,
 		return NULL;
 	}
 
+	ls->ls_recall_start = 0;
 	ls->ls_fenced = false;
 	ls->ls_fence_inflight = false;
 	ls->ls_fence_stopped = false;
@@ -718,8 +719,14 @@ nfsd4_cb_layout_done(struct nfsd4_callback *cb, struct rpc_task *task)
 		now = ktime_get();
 		nn = net_generic(ls->ls_stid.sc_client->net, nfsd_net_id);
 
-		/* Client gets 2 lease periods to return it */
-		cutoff = ktime_add_ns(task->tk_start,
+		/*
+		 * Client gets 2 lease periods to return it, counted from its
+		 * first answer.  task->tk_start cannot be used for this, it
+		 * is set anew each time the callback is restarted.
+		 */
+		if (!ls->ls_recall_start)
+			ls->ls_recall_start = now;
+		cutoff = ktime_add_ns(ls->ls_recall_start,
 					 (u64)nn->nfsd4_lease * NSEC_PER_SEC * 2);
 
 		if (ktime_before(now, cutoff)) {
diff --git a/fs/nfsd/state.h b/fs/nfsd/state.h
index 910f4eac7..ff840a311 100644
--- a/fs/nfsd/state.h
+++ b/fs/nfsd/state.h
@@ -852,6 +852,7 @@ struct nfs4_layout_stateid {
 	struct nfsd_file		*ls_file;
 	struct nfsd4_callback		ls_recall;
 	stateid_t			ls_recall_sid;
+	ktime_t				ls_recall_start;
 	bool				ls_recalled;
 	struct mutex			ls_mutex;
 

-- 
2.43.0



^ permalink raw reply	[flat|nested] 9+ messages in thread

* [PATCH 2/6] nfsd: run the fence script when a block layout lease break times out
  2026-10-06  4:29 [PATCH 0/6] nfsd: layout recall and lease break fixes Daejun Park via B4 Relay
  2026-10-06  4:29 ` [PATCH 1/6] nfsd: make the time limit of a layout recall work Daejun Park via B4 Relay
@ 2026-10-06  4:29 ` Daejun Park via B4 Relay
  2026-10-06  4:29 ` [PATCH 3/6] nfsd: keep polling a layout recall answered with NFS4ERR_OLD_STATEID Daejun Park via B4 Relay
                   ` (5 subsequent siblings)
  7 siblings, 0 replies; 9+ messages in thread
From: Daejun Park via B4 Relay @ 2026-10-06  4:29 UTC (permalink / raw)
  To: Chuck Lever, Jeff Layton, NeilBrown, Olga Kornievskaia, Dai Ngo,
	Tom Talpey
  Cc: Christoph Hellwig, Benjamin Coddington, Trond Myklebust,
	Anna Schumaker, linux-nfs, linux-kernel, Daejun Park

From: Daejun Park <daejun7.park@samsung.com>

When a lease break has waited for fs.lease-break-time,
nfsd4_layout_lm_breaker_timedout() has the client that holds the layout
fenced before the breaker goes on.  For a layout type without
->fence_client, which is the block layout, it returns true at once: the
lease is removed and the operation on the server goes on while the
client still holds the layout.  /sbin/nfsd-recall-failed, which
Documentation/admin-guide/nfs/pnfs-block-server.rst names as the way to
fence such a client, is only run when the CB_LAYOUTRECALL fails.

This hits a client that keeps answering the recall with NFS4_OK or
NFS4ERR_DELAY for longer than fs.lease-break-time, 45 seconds by
default, for example because its path to the device is cut.  With the
previous patch the recall is polled for two lease periods, 180 seconds
by default, so the lease break times out first.  A Linux client before
v6.8 answers that way (read from the code, not run), and with the next
patch every Linux client is polled that way; until now a client from
v6.8 on has the recall fail after 10 ms.

Let the fence worker run the script for a layout type without
->fence_client, and give up the lease after that.  As on the recall
failure path, the exit status of the script is not looked at.  The
same document says that the server retries a failed fence and keeps
the file from other clients meanwhile; that is what the fence worker
does with ->fence_client().  Doing it for the script would block the
file for good on a server that has no /sbin/nfsd-recall-failed, so
this patch leaves it as the recall failure path has it.

As for the SCSI layout since the commit this fixes, the client is then
fenced after fs.lease-break-time.  That is shorter than one lease
period by default, while RFC 8881 section 12.5.5 has the server wait
one lease period after a recall before it takes further action.
Setting fs.lease-break-time to at least the lease time gives that wait
for both layout types.

The fence worker now runs the script with UMH_WAIT_PROC, and since
"nfsd: drain pNFS fence work during state teardown" tearing down the
client waits for the fence worker.  A script that expires the client it
fences, through /proc/fs/nfsd/clients/*/ctl, would therefore wait for
itself.  That is already so on the recall failure path, where the script
runs while the callback is in flight and the teardown waits for the
callbacks (read from the code, not run).

Tested together with the next patch: a client writing through a block
layout whose path to the device is cut for 70 seconds while it can
still talk to the server, default lease and lease break times, and a
4 KiB write to the file on the server.  Without this patch the write
on the server goes on after 47 seconds and /sbin/nfsd-recall-failed is
not run.  With it the script is run after 47 seconds and the write goes
on then.  In both runs the client returns the layout when its path is
back, and the recall completes.

Fixes: f52792f484ba ("NFSD: Enforce timeout on layout recall and integrate lease manager fencing")

Signed-off-by: Daejun Park <daejun7.park@samsung.com>
---
 fs/nfsd/nfs4layouts.c | 20 +++++++++++++++-----
 1 file changed, 15 insertions(+), 5 deletions(-)

diff --git a/fs/nfsd/nfs4layouts.c b/fs/nfsd/nfs4layouts.c
index 3731b4db7..088e26e8e 100644
--- a/fs/nfsd/nfs4layouts.c
+++ b/fs/nfsd/nfs4layouts.c
@@ -861,6 +861,7 @@ static void nfsd4_layout_fence_worker(struct work_struct *work)
 	struct delayed_work *dwork = to_delayed_work(work);
 	struct nfs4_layout_stateid *ls = container_of(dwork,
 			struct nfs4_layout_stateid, ls_fence_work);
+	const struct nfsd4_layout_ops *ops;
 	struct nfsd_file *nf;
 	struct block_device *bdev;
 	struct nfs4_client *clp;
@@ -884,7 +885,17 @@ static void nfsd4_layout_fence_worker(struct work_struct *work)
 	clp = ls->ls_stid.sc_client;
 	nn = net_generic(clp->net, nfsd_net_id);
 	bdev = nf->nf_file->f_path.mnt->mnt_sb->s_bdev;
-	if (nfsd4_layout_ops[ls->ls_layout_type]->fence_client(ls, nf)) {
+	ops = nfsd4_layout_ops[ls->ls_layout_type];
+	if (!ops->fence_client) {
+		/*
+		 * The layout type cannot fence.  Run the fence script, as a
+		 * failed recall does, before the lease is given up.
+		 */
+		nfsd4_cb_layout_fail(ls, nf);
+		nfsd_file_put(nf);
+		goto dispose;
+	}
+	if (ops->fence_client(ls, nf)) {
 		/* fenced ok */
 		nfsd_file_put(nf);
 		pr_warn("%s: FENCED client[%pISpc] clid[%d] to device[%s]\n",
@@ -936,8 +947,9 @@ static void nfsd4_layout_fence_worker(struct work_struct *work)
  * nfsd4_layout_lm_breaker_timedout - The layout recall has timed out.
  * @fl: file to check
  *
- * If the layout type supports a fence operation, schedule a worker to
- * fence the client from accessing the block device.
+ * Schedule a worker to fence the client from accessing the block device.
+ * If the layout type has no fence operation, the worker runs the fence
+ * script instead.
  *
  * This function runs under the protection of the spin_lock flc_lock.
  * At this time, the file_lease associated with the layout stateid is
@@ -959,8 +971,6 @@ nfsd4_layout_lm_breaker_timedout(struct file_lease *fl)
 {
 	struct nfs4_layout_stateid *ls = fl->c.flc_owner;
 
-	if (!nfsd4_layout_ops[ls->ls_layout_type]->fence_client)
-		return true;
 	/*
 	 * Make sure layout has not been returned yet before
 	 * taking a reference count on the layout stateid. The

-- 
2.43.0



^ permalink raw reply	[flat|nested] 9+ messages in thread

* [PATCH 3/6] nfsd: keep polling a layout recall answered with NFS4ERR_OLD_STATEID
  2026-10-06  4:29 [PATCH 0/6] nfsd: layout recall and lease break fixes Daejun Park via B4 Relay
  2026-10-06  4:29 ` [PATCH 1/6] nfsd: make the time limit of a layout recall work Daejun Park via B4 Relay
  2026-10-06  4:29 ` [PATCH 2/6] nfsd: run the fence script when a block layout lease break times out Daejun Park via B4 Relay
@ 2026-10-06  4:29 ` Daejun Park via B4 Relay
  2026-10-06  4:29 ` [PATCH 4/6] nfsd: fail a LAYOUTGET in progress when the layout lease is broken Daejun Park via B4 Relay
                   ` (4 subsequent siblings)
  7 siblings, 0 replies; 9+ messages in thread
From: Daejun Park via B4 Relay @ 2026-10-06  4:29 UTC (permalink / raw)
  To: Chuck Lever, Jeff Layton, NeilBrown, Olga Kornievskaia, Dai Ngo,
	Tom Talpey
  Cc: Christoph Hellwig, Benjamin Coddington, Trond Myklebust,
	Anna Schumaker, linux-nfs, linux-kernel, Daejun Park

From: Daejun Park <daejun7.park@samsung.com>

When a client answers CB_LAYOUTRECALL with NFS4_OK and still holds
layouts, nfsd4_cb_layout_done() sends the callback again after 10 ms to
see if the client is done.  The repeat carries the stateid of the first
callback, although RFC 8881 section 12.5.3 has the server increment the
seqid in each CB_LAYOUTRECALL request.

The same section makes the seqid of a processed CB_LAYOUTRECALL the
current seqid of the client.  Since commit dce72920c81b ("NFSv4.1: if
referring calls are complete, trust the stateid argument"), which is in
v6.8, the Linux client answers a recall whose seqid is not newer than
its own with NFS4ERR_OLD_STATEID, unless a LAYOUTRETURN is on the wire
(NFS4ERR_DELAY) or the layouts are gone (NFS4ERR_NOMATCHING_LAYOUT).
Before that commit it answered a repeat of the recall it was working
on with NFS4_OK until its LAYOUTRETURN was prepared, and with
NFS4ERR_OLD_STATEID from then until nfsd had processed the LAYOUTRETURN
(read from the code, not run).  So a client from v6.8 on that needs
more than 10 ms to finish its I/O and to send LAYOUTRETURN answers the
second callback with NFS4ERR_OLD_STATEID, and an older client can
answer a later one that way.

nfsd4_cb_layout_done() takes that as a failed recall.  With the block
layout it logs "client ... failed to respond to layout recall.
Fencing.." and runs /sbin/nfsd-recall-failed, with the SCSI layout it
fences the client with a persistent reservation preempt, and in both
cases nfsd4_cb_layout_release() drops the layouts.  Section 12.5.5
requires the server to wait one lease period after a recall before it
takes further action.  Here the client is still using the layout: its
I/O can be in flight when the conflicting operation goes on, and what
it has written to newly allocated blocks stays unwritten on the server,
because the layout is gone when the client wants to commit it.

Clients doing O_DIRECT I/O in 1 MiB requests on one file for 30
seconds, block layout on XFS.

Recalls that ended in the failure path, out of the recalls sent:

                                                   before      after
  2 clients read (different halves of the file)    7 / 9       0 / 9
  2 clients write                                  8 / 9       0 / 9
  3 clients write                                  19 / 20     0 / 20
  2 clients write, the server writes 4 KiB to
  the file every 50 ms                             216 / 229   0 / 250

Blocks that read back as zeroes on the server, out of the 1 MiB blocks
that the clients wrote to a new file with O_DIRECT, followed by one
successful fsync():

                                                   before      after
  2 clients write                                  6 / 564     0 / 481
  3 clients write                                  11 / 527    0 / 569
  1 client writes, the server sets the file size
  every second                                     21 / 469    0 / 504
  2 clients write, file preallocated               8 / 601     0 / 602
  1 client writes, 1 client reads                  4 / 493     0 / 508

One client writing through a block layout whose path to the device is
cut for 15 seconds while it can still talk to the server, and a 4 KiB
write to the file on the server:

                                                   before      after
  the write on the server takes                    0.05 s      15.6 s
  the recall                                       fails       completes
  /sbin/nfsd-recall-failed is run                  yes         no

Before, the write of the client that was in flight when the layout was
recalled was stuck for 18.5 seconds and no WRITE was sent to the
server, so it reached the device about 15 seconds after the write on
the server had returned.  After, the write on the server waits for it.

Two clients writing one file with the SCSI layout on an nvmet-tcp
namespace: before, both recalls sent ended in nfsd4_scsi_fence_client(),
the number of registrants on the namespace went from 4 to 2, and both
clients logged reservation conflicts and did the rest of the run
through the server.  The same with a new file, after a fresh boot: both
clients were fenced, and one of the 630 blocks they had written read
back as zeroes.  After, no client is fenced and no block is lost.

Handle NFS4ERR_OLD_STATEID like NFS4ERR_DELAY: the client has seen the
recall, so keep polling until the layouts are returned or the time
limit is reached.  This patch must not be applied without the two
before it: the limit only works with "nfsd: make the time limit of a
layout recall work", and with "nfsd: run the fence script when a block
layout lease break times out" a block layout client is fenced when the
lease break gives up.  On kernels before v7.1 a lease break does not
time out; there the operation on the server waits until the recall
ends, for at most two lease periods.
That operation can be an nfsd thread serving a WRITE or SETATTR of
another client (xfs_break_layouts()).  It is what such a kernel does
today for a client that keeps answering NFS4ERR_DELAY, without the
limit of the first patch; RFC 8881 section 12.5.5 asks the server to
wait at least one lease period before it fences.

The poll sends the callback every 10 ms for as long as the client
holds the layout, as it already does for a client that answers
NFS4ERR_DELAY.  With a client that needs long to return the layout
that is about 100 callbacks a second, for up to two lease periods.

A client can also answer this way without having processed the recall,
when a LAYOUTRETURN that left layouts behind has moved its seqid past
the seqid of the recall.  With this patch alone it is then polled until
the time limit and fenced, where it used to be fenced at once, and
LAYOUTGETs for the file fail with NFS4ERR_RECALLCONFLICT meanwhile;
"nfsd: poll a layout recall with a new seqid when the client is past
it", later in this series, gives such a client a new seqid.

This keeps the poll as it is for a client that has answered NFS4_OK.
Sending each repeat with a new seqid, or ending the callback on NFS4_OK
and letting the LAYOUTRETURN finish the recall, would follow the RFC;
both are larger changes than a fix for stable kernels should be.

Fixes: 6b9b21073d3b ("nfsd: give up on CB_LAYOUTRECALLs after two lease periods")

Signed-off-by: Daejun Park <daejun7.park@samsung.com>
---
 fs/nfsd/nfs4layouts.c | 8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/fs/nfsd/nfs4layouts.c b/fs/nfsd/nfs4layouts.c
index 088e26e8e..3d288649c 100644
--- a/fs/nfsd/nfs4layouts.c
+++ b/fs/nfsd/nfs4layouts.c
@@ -707,7 +707,15 @@ nfsd4_cb_layout_done(struct nfsd4_callback *cb, struct rpc_task *task)
 	switch (task->tk_status) {
 	case 0:
 	case -NFS4ERR_DELAY:
+	case -NFS4ERR_OLD_STATEID:
 		/*
+		 * RFC 8881 section 12.5.3 wants a new seqid in each
+		 * CB_LAYOUTRECALL request.  The poll below sends the request
+		 * again with the seqid of the first one.  A client that has
+		 * processed that one is at its seqid or past it, and the
+		 * Linux client answers the repeat with NFS4ERR_OLD_STATEID
+		 * while it is returning the layouts.
+		 *
 		 * Anything left? If not, then call it done. Note that we don't
 		 * take the spinlock since this is an optimization and nothing
 		 * should get added until the cb counter goes to zero.

-- 
2.43.0



^ permalink raw reply	[flat|nested] 9+ messages in thread

* [PATCH 4/6] nfsd: fail a LAYOUTGET in progress when the layout lease is broken
  2026-10-06  4:29 [PATCH 0/6] nfsd: layout recall and lease break fixes Daejun Park via B4 Relay
                   ` (2 preceding siblings ...)
  2026-10-06  4:29 ` [PATCH 3/6] nfsd: keep polling a layout recall answered with NFS4ERR_OLD_STATEID Daejun Park via B4 Relay
@ 2026-10-06  4:29 ` Daejun Park via B4 Relay
  2026-10-06  4:29 ` [PATCH 5/6] nfsd: poll a layout recall with a new seqid when the client is past it Daejun Park via B4 Relay
                   ` (3 subsequent siblings)
  7 siblings, 0 replies; 9+ messages in thread
From: Daejun Park via B4 Relay @ 2026-10-06  4:29 UTC (permalink / raw)
  To: Chuck Lever, Jeff Layton, NeilBrown, Olga Kornievskaia, Dai Ngo,
	Tom Talpey
  Cc: Christoph Hellwig, Benjamin Coddington, Trond Myklebust,
	Anna Schumaker, linux-nfs, linux-kernel, Daejun Park

From: Daejun Park <daejun7.park@samsung.com>

A layout stateid takes its FL_LAYOUT lease when it is allocated, before
the first layout is added to it.  When the lease is broken in between,
nfsd4_recall_file_layout() finds no layouts and does nothing.  The
LAYOUTGET then hands out a layout that is not recalled: the lease is
already marked as breaking, so ->lm_break() is not called for it again.
The breaker waits until the client gives the layout up by itself or the
lease break time is over; with the SCSI layout the client is then
fenced although it was never sent a recall.  While the breaker waits,
kernel_setlease() fails for new layout stateids and the LAYOUTGETs of
other clients get NFS4ERR_DELAY.

Commit c5c707f96fc9 ("nfsd: implement pNFS layout recalls") marked such
a layout stateid as recalled, and since nothing kept the LAYOUTGET from
adding its layout afterwards, that layout was never recalled.  Commit
fb610c4dbc99 ("nfsd: fix race to check ls_layouts") therefore made the
recall skip a layout stateid without layouts, which left the case
above.

Let the lease break mark such a layout stateid as recalled again, and
make nfsd4_insert_layout() refuse to add layouts to a recalled layout
stateid, which is what was missing the first time.  The LAYOUTGET fails
with NFS4ERR_RECALLCONFLICT, the layout stateid is freed and the lease
goes away.  No callback is sent for it; the new
nfsd_layout_recall_empty tracepoint shows where fi_lo_recalls was
raised without one.  NFS4ERR_RECALLCONFLICT is what nfsd already answers
to a LAYOUTGET for a file with a recall in progress, also when the
recall went to another client (RFC 8881 section 15.1.14.3: "a
conflicting recall operation that is currently in progress for that
object").  The check in nfsd4_insert_layout() also covers a
layout stateid with layouts that is recalled between the checks in
nfsd4_layoutget() and nfsd4_insert_layout().

Only a lease break does this.  A LAYOUTGET that finds a layout stateid
without layouts of another client leaves it alone, as before.

The window is the time a LAYOUTGET needs from allocating the layout
stateid to inserting the layout.

Two clients writing to one file while the server writes 4 KiB to it
every 50 ms, for 30 seconds, block layout.  Without this patch one
write on the server took 9.7 seconds in one run and 13.9 seconds in
another, in both until the clients were done; in a third run it did not
happen.  With this patch the longest write on the server took 56 ms.

To see the new path at work, a delay of 100 ms was added to LAYOUTGET
between the two steps for one pair of runs.  Without the patch the
second write on the server took 30 seconds, the whole run.  With it the
longest write took 108 ms and nfsd_layout_recall_empty fired 16 times.
(With that delay every LAYOUTGET meets a write on the server, so the
clients got no layout in that run.  It shows the path and nothing else.)

Fixes: c5c707f96fc9 ("nfsd: implement pNFS layout recalls")

Signed-off-by: Daejun Park <daejun7.park@samsung.com>
---
 fs/nfsd/nfs4layouts.c | 49 +++++++++++++++++++++++++++++++++++++++++++------
 fs/nfsd/trace.h       |  1 +
 2 files changed, 44 insertions(+), 6 deletions(-)

diff --git a/fs/nfsd/nfs4layouts.c b/fs/nfsd/nfs4layouts.c
index 3d288649c..66f79e4e4 100644
--- a/fs/nfsd/nfs4layouts.c
+++ b/fs/nfsd/nfs4layouts.c
@@ -337,14 +337,15 @@ nfsd4_preprocess_layout_stateid(struct svc_rqst *rqstp,
 }
 
 static void
-nfsd4_recall_file_layout(struct nfs4_layout_stateid *ls)
+nfsd4_recall_file_layout_locked(struct nfs4_layout_stateid *ls)
 {
-	spin_lock(&ls->ls_lock);
+	lockdep_assert_held(&ls->ls_lock);
+
 	if (ls->ls_recalled)
-		goto out_unlock;
+		return;
 
 	if (list_empty(&ls->ls_layouts))
-		goto out_unlock;
+		return;
 
 	ls->ls_recalled = true;
 	atomic_inc(&ls->ls_stid.sc_file->fi_lo_recalls);
@@ -354,7 +355,34 @@ nfsd4_recall_file_layout(struct nfs4_layout_stateid *ls)
 		refcount_inc(&ls->ls_stid.sc_count);
 		nfsd4_run_cb(&ls->ls_recall);
 	}
-out_unlock:
+}
+
+static void
+nfsd4_recall_file_layout(struct nfs4_layout_stateid *ls)
+{
+	spin_lock(&ls->ls_lock);
+	nfsd4_recall_file_layout_locked(ls);
+	spin_unlock(&ls->ls_lock);
+}
+
+/*
+ * A lease break also has to stop a LAYOUTGET that is in progress.  Its
+ * layout stateid has the lease and no layouts yet, so there is nothing to
+ * recall, and the breaker goes on once that layout stateid is gone.  Mark it
+ * as recalled without sending a callback: nfsd4_insert_layout() then fails
+ * the LAYOUTGET instead of handing out a layout after the break.
+ */
+static void
+nfsd4_break_file_layout(struct nfs4_layout_stateid *ls)
+{
+	spin_lock(&ls->ls_lock);
+	if (!ls->ls_recalled && list_empty(&ls->ls_layouts)) {
+		ls->ls_recalled = true;
+		atomic_inc(&ls->ls_stid.sc_file->fi_lo_recalls);
+		trace_nfsd_layout_recall_empty(&ls->ls_stid.sc_stateid);
+	} else {
+		nfsd4_recall_file_layout_locked(ls);
+	}
 	spin_unlock(&ls->ls_lock);
 }
 
@@ -433,6 +461,10 @@ nfsd4_insert_layout(struct nfsd4_layoutget *lgp, struct nfs4_layout_stateid *ls)
 	if (nfserr)
 		goto out;
 	spin_lock(&ls->ls_lock);
+	if (ls->ls_recalled) {
+		nfserr = nfserr_recallconflict;
+		goto out_unlock_ls;
+	}
 	list_for_each_entry(lp, &ls->ls_layouts, lo_perstate) {
 		if (layouts_try_merge(&lp->lo_seg, seg))
 			goto done;
@@ -451,6 +483,10 @@ nfsd4_insert_layout(struct nfsd4_layoutget *lgp, struct nfs4_layout_stateid *ls)
 	if (nfserr)
 		goto out;
 	spin_lock(&ls->ls_lock);
+	if (ls->ls_recalled) {
+		nfserr = nfserr_recallconflict;
+		goto out_unlock_ls;
+	}
 	list_for_each_entry(lp, &ls->ls_layouts, lo_perstate) {
 		if (layouts_try_merge(&lp->lo_seg, seg))
 			goto done;
@@ -461,6 +497,7 @@ nfsd4_insert_layout(struct nfsd4_layoutget *lgp, struct nfs4_layout_stateid *ls)
 	new = NULL;
 done:
 	nfs4_inc_and_copy_stateid(&lgp->lg_sid, &ls->ls_stid);
+out_unlock_ls:
 	spin_unlock(&ls->ls_lock);
 out:
 	spin_unlock(&fp->fi_lock);
@@ -794,7 +831,7 @@ nfsd4_layout_lm_break(struct file_lease *fl)
 	 * Enforce break lease timeout to prevent NFSD
 	 * thread from hanging in __break_lease.
 	 */
-	nfsd4_recall_file_layout(fl->c.flc_owner);
+	nfsd4_break_file_layout(fl->c.flc_owner);
 	return false;
 }
 
diff --git a/fs/nfsd/trace.h b/fs/nfsd/trace.h
index ad106d627..78f682bfd 100644
--- a/fs/nfsd/trace.h
+++ b/fs/nfsd/trace.h
@@ -679,6 +679,7 @@ DEFINE_STATEID_EVENT(layout_get_lookup_fail);
 DEFINE_STATEID_EVENT(layout_commit_lookup_fail);
 DEFINE_STATEID_EVENT(layout_return_lookup_fail);
 DEFINE_STATEID_EVENT(layout_recall);
+DEFINE_STATEID_EVENT(layout_recall_empty);
 DEFINE_STATEID_EVENT(layout_recall_done);
 DEFINE_STATEID_EVENT(layout_recall_fail);
 DEFINE_STATEID_EVENT(layout_recall_release);

-- 
2.43.0



^ permalink raw reply	[flat|nested] 9+ messages in thread

* [PATCH 5/6] nfsd: poll a layout recall with a new seqid when the client is past it
  2026-10-06  4:29 [PATCH 0/6] nfsd: layout recall and lease break fixes Daejun Park via B4 Relay
                   ` (3 preceding siblings ...)
  2026-10-06  4:29 ` [PATCH 4/6] nfsd: fail a LAYOUTGET in progress when the layout lease is broken Daejun Park via B4 Relay
@ 2026-10-06  4:29 ` Daejun Park via B4 Relay
  2026-10-06  4:29 ` [PATCH 6/6] nfsd: do not fence a client for a layout recall with no layouts left Daejun Park via B4 Relay
                   ` (2 subsequent siblings)
  7 siblings, 0 replies; 9+ messages in thread
From: Daejun Park via B4 Relay @ 2026-10-06  4:29 UTC (permalink / raw)
  To: Chuck Lever, Jeff Layton, NeilBrown, Olga Kornievskaia, Dai Ngo,
	Tom Talpey
  Cc: Christoph Hellwig, Benjamin Coddington, Trond Myklebust,
	Anna Schumaker, linux-nfs, linux-kernel, Daejun Park

From: Daejun Park <daejun7.park@samsung.com>

nfsd4_cb_layout_done() polls a client that answers a CB_LAYOUTRECALL
with NFS4ERR_OLD_STATEID, as "nfsd: keep polling a layout recall
answered with NFS4ERR_OLD_STATEID" has it do, and each poll repeats the
seqid that ->prepare gave the recall.  NFS4ERR_OLD_STATEID says that
the client is at or past that seqid.  A client that has answered
NFS4_OK to the recall is returning its layouts, and the polls end when
it has.  A client that has not is past the seqid without having taken
the recall up.  A LAYOUTRETURN that returns some layouts and leaves
others gets a new seqid; if nfsd processes it after ->prepare, the
client takes in a seqid above the recall's from its reply.  Such a
client answers every poll with NFS4ERR_OLD_STATEID.  The recall goes on
for two lease periods, a LAYOUTGET for the file from another client
gets NFS4ERR_RECALLCONFLICT meanwhile, and the client is then fenced,
although it was never asked for the layouts it still holds.

RFC 8881 section 12.5.3 has a new seqid for each CB_LAYOUTRECALL
request.  When a client that has not answered NFS4_OK answers
NFS4ERR_OLD_STATEID, give the next poll a new seqid, which the client
takes up.  ->prepare clears ls_recall_accepted, and ->done sets it when
the client answers NFS4_OK.  The new seqid is traced as
nfsd_layout_recall_new_seqid.

->done runs in rpciod and does not take ls_mutex, which could block it.
While a recall is out, only a LAYOUTRETURN changes the seqid besides,
since "nfsd: fail a LAYOUTGET in progress when the layout lease is
broken" keeps LAYOUTGET from adding layouts to a recalled layout
stateid, and nfsd4_preprocess_layout_stateid() only checks that the
seqid a client presents is not above the current one.

A pynfs client that holds a layout answers a recall whose seqid is not
above the first seqid of the recall with NFS4ERR_OLD_STATEID, and
returns the layout on a later one.  Without this patch, nfsd sent the
recall 501 times in six seconds, all with the first seqid, and a
LAYOUTGET of another client still got NFS4ERR_RECALLCONFLICT.  With it,
the second recall carried a new seqid, 10 ms later, the client returned
the layout, and the LAYOUTGET succeeded.  The block layout runs of the
earlier patches give the same results with this one and the next: no
recall failed, and every block written to a new file was found.

This builds on "nfsd: keep polling a layout recall answered with
NFS4ERR_OLD_STATEID" and "nfsd: fail a LAYOUTGET in progress when the
layout lease is broken", earlier in this series, and should go to
stable kernels with them.

Fixes: c5c707f96fc9 ("nfsd: implement pNFS layout recalls")

Signed-off-by: Daejun Park <daejun7.park@samsung.com>
---
 fs/nfsd/nfs4layouts.c | 29 ++++++++++++++++++++++++++---
 fs/nfsd/state.h       |  2 ++
 fs/nfsd/trace.h       |  1 +
 3 files changed, 29 insertions(+), 3 deletions(-)

diff --git a/fs/nfsd/nfs4layouts.c b/fs/nfsd/nfs4layouts.c
index 66f79e4e4..c140c9bde 100644
--- a/fs/nfsd/nfs4layouts.c
+++ b/fs/nfsd/nfs4layouts.c
@@ -727,6 +727,7 @@ nfsd4_cb_layout_prepare(struct nfsd4_callback *cb)
 	mutex_lock(&ls->ls_mutex);
 	nfs4_inc_and_copy_stateid(&ls->ls_recall_sid, &ls->ls_stid);
 	mutex_unlock(&ls->ls_mutex);
+	ls->ls_recall_accepted = false;
 	return true;
 }
 
@@ -742,13 +743,35 @@ nfsd4_cb_layout_done(struct nfsd4_callback *cb, struct rpc_task *task)
 
 	trace_nfsd_cb_layout_done(&ls->ls_stid.sc_stateid, task);
 	switch (task->tk_status) {
-	case 0:
-	case -NFS4ERR_DELAY:
 	case -NFS4ERR_OLD_STATEID:
+		/*
+		 * The client is at or past the seqid of the request.  Unless
+		 * it has answered NFS4_OK to this recall and is returning
+		 * layouts, it has not taken the recall up, for instance
+		 * because a LAYOUTRETURN moved it past the seqid.  Poll with a
+		 * new seqid then (RFC 8881 section 12.5.3), which such a
+		 * client answers afresh.
+		 *
+		 * ->done runs in rpciod and does not take ls_mutex, which
+		 * could block it.  While the recall is out, only a
+		 * LAYOUTRETURN changes the seqid besides, and the check of the
+		 * seqid a client presents only asks whether it is above the
+		 * current one.
+		 */
+		if (!ls->ls_recall_accepted) {
+			nfs4_inc_and_copy_stateid(&ls->ls_recall_sid,
+						  &ls->ls_stid);
+			trace_nfsd_layout_recall_new_seqid(&ls->ls_recall_sid);
+		}
+		fallthrough;
+	case -NFS4ERR_DELAY:
+	case 0:
+		if (!task->tk_status)
+			ls->ls_recall_accepted = true;
 		/*
 		 * RFC 8881 section 12.5.3 wants a new seqid in each
 		 * CB_LAYOUTRECALL request.  The poll below sends the request
-		 * again with the seqid of the first one.  A client that has
+		 * again with the seqid it last had.  A client that has
 		 * processed that one is at its seqid or past it, and the
 		 * Linux client answers the repeat with NFS4ERR_OLD_STATEID
 		 * while it is returning the layouts.
diff --git a/fs/nfsd/state.h b/fs/nfsd/state.h
index ff840a311..3ad87c85a 100644
--- a/fs/nfsd/state.h
+++ b/fs/nfsd/state.h
@@ -853,6 +853,8 @@ struct nfs4_layout_stateid {
 	struct nfsd4_callback		ls_recall;
 	stateid_t			ls_recall_sid;
 	ktime_t				ls_recall_start;
+	/* The client has answered NFS4_OK to the recall */
+	bool				ls_recall_accepted;
 	bool				ls_recalled;
 	struct mutex			ls_mutex;
 
diff --git a/fs/nfsd/trace.h b/fs/nfsd/trace.h
index 78f682bfd..f6bdf69d4 100644
--- a/fs/nfsd/trace.h
+++ b/fs/nfsd/trace.h
@@ -681,6 +681,7 @@ DEFINE_STATEID_EVENT(layout_return_lookup_fail);
 DEFINE_STATEID_EVENT(layout_recall);
 DEFINE_STATEID_EVENT(layout_recall_empty);
 DEFINE_STATEID_EVENT(layout_recall_done);
+DEFINE_STATEID_EVENT(layout_recall_new_seqid);
 DEFINE_STATEID_EVENT(layout_recall_fail);
 DEFINE_STATEID_EVENT(layout_recall_release);
 

-- 
2.43.0



^ permalink raw reply	[flat|nested] 9+ messages in thread

* [PATCH 6/6] nfsd: do not fence a client for a layout recall with no layouts left
  2026-10-06  4:29 [PATCH 0/6] nfsd: layout recall and lease break fixes Daejun Park via B4 Relay
                   ` (4 preceding siblings ...)
  2026-10-06  4:29 ` [PATCH 5/6] nfsd: poll a layout recall with a new seqid when the client is past it Daejun Park via B4 Relay
@ 2026-10-06  4:29 ` Daejun Park via B4 Relay
  2026-10-06 19:46 ` [PATCH 0/6] nfsd: layout recall and lease break fixes Chuck Lever
       [not found] ` <CGME20261006194708epcas2p2df158d7a6f0f9eb57cf1f6c3a0ec0c4b@epcms2p1>
  7 siblings, 0 replies; 9+ messages in thread
From: Daejun Park via B4 Relay @ 2026-10-06  4:29 UTC (permalink / raw)
  To: Chuck Lever, Jeff Layton, NeilBrown, Olga Kornievskaia, Dai Ngo,
	Tom Talpey
  Cc: Christoph Hellwig, Benjamin Coddington, Trond Myklebust,
	Anna Schumaker, linux-nfs, linux-kernel, Daejun Park

From: Daejun Park <daejun7.park@samsung.com>

nfsd4_cb_layout_done() fences the client for any answer to a
CB_LAYOUTRECALL that it does not handle, without looking at whether the
layout stateid still has layouts.  A LAYOUTRETURN can return the last
layout while the recall is still out.  The Linux client answers
NFS4ERR_NOMATCHING_LAYOUT then, which RFC 8881 section 20.3.3 allows
for a client that holds no layouts for the file.  A client that answers
NFS4ERR_BAD_STATEID is right too, since section 12.5.3 makes a layout
stateid invalid once the client has no layouts for the file, but it is
fenced for nothing.

Check for layouts before fencing, as the polls already do before their
time limit, and end the recall when none are left.  This does not depend
on the earlier patches in this series.

A pynfs client that returns its layout on the recall and then answers
it NFS4ERR_BAD_STATEID: without this patch, nfsd logged "failed to
respond to layout recall" and ran the fence for it; with it, the recall
ended without either.

Fixes: c5c707f96fc9 ("nfsd: implement pNFS layout recalls")

Signed-off-by: Daejun Park <daejun7.park@samsung.com>
---
 fs/nfsd/nfs4layouts.c | 9 +++++++++
 1 file changed, 9 insertions(+)

diff --git a/fs/nfsd/nfs4layouts.c b/fs/nfsd/nfs4layouts.c
index c140c9bde..345890add 100644
--- a/fs/nfsd/nfs4layouts.c
+++ b/fs/nfsd/nfs4layouts.c
@@ -803,6 +803,15 @@ nfsd4_cb_layout_done(struct nfsd4_callback *cb, struct rpc_task *task)
 		}
 		fallthrough;
 	default:
+		/*
+		 * Nothing to fence for once the layouts are gone: a
+		 * LAYOUTRETURN can return the last one while the recall is
+		 * still out.
+		 */
+		if (list_empty(&ls->ls_layouts)) {
+			trace_nfsd_layout_recall_done(&ls->ls_stid.sc_stateid);
+			return 1;
+		}
 		/*
 		 * Unknown error or non-responding client, we'll need to fence.
 		 */

-- 
2.43.0



^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: [PATCH 0/6] nfsd: layout recall and lease break fixes
  2026-10-06  4:29 [PATCH 0/6] nfsd: layout recall and lease break fixes Daejun Park via B4 Relay
                   ` (5 preceding siblings ...)
  2026-10-06  4:29 ` [PATCH 6/6] nfsd: do not fence a client for a layout recall with no layouts left Daejun Park via B4 Relay
@ 2026-10-06 19:46 ` Chuck Lever
       [not found] ` <CGME20261006194708epcas2p2df158d7a6f0f9eb57cf1f6c3a0ec0c4b@epcms2p1>
  7 siblings, 0 replies; 9+ messages in thread
From: Chuck Lever @ 2026-10-06 19:46 UTC (permalink / raw)
  To: daejun7.park
  Cc: Jeff Layton, NeilBrown, Olga Kornievskaia, Dai Ngo, Tom Talpey,
	Christoph Hellwig, Benjamin Coddington, Trond Myklebust,
	Anna Schumaker, linux-nfs, linux-kernel

Hi Daejun,

An nfsd thread must not block for more than a few milliseconds, and
never on something a client controls. A client can tie up thread
after thread until the server has no more threads to respond to
LAYOUTRETURN or DELEGRETURN, and then the server deadlocks.

For delegations we are safe. nfsd_open_break_lease() passes
O_NONBLOCK and the operation returns NFS4ERR_DELAY.

For layouts we are not. A WRITE or SETATTR goes into XFS,
xfs_break_leased_layouts() calls break_layout() blocking, and the
nfsd thread sleeps in __break_lease() until the lease is dropped or
the breaker times out. That dates from f52792f484ba, but it was
bounded by fs.lease-break-time, and a Linux client from v6.8 on
rarely hit it. It responded OLD_STATEID, the callback failed, the
lease was dropped, and the thread was back within a round trip.

This series makes that wait the common case and then unbounded.

- Patch 3 polls on OLD_STATEID, so the thread sleeps for as long as
  the client takes to drain its I/O, up to lease-break-time.

- Patch 2 waits on the fence script with UMH_WAIT_PROC while
  lm_breaker_timedout() keeps returning false, so the thread waits
  lease-break-time plus however long the script takes to reach the
  client.

- Patch 5 extends the wait of patch 3 to a client past the seqid.

I think the fix has to come first. Before the WRITE or SETATTR, call
break_layout() non-blocking to start the recall, and return
NFS4ERR_DELAY on -EWOULDBLOCK, as we do for delegations. Then the
fence worker and the poll loop can take as long as they need and no
nfsd thread waits on them.

Give that a try and let me know if that works or needs improvement.


-- 
Chuck Lever (Come to NFS bake-a-thon! https://nfsv4bat.org)

^ permalink raw reply	[flat|nested] 9+ messages in thread

* RE:(2) [PATCH 0/6] nfsd: layout recall and lease break fixes
       [not found] ` <CGME20261006194708epcas2p2df158d7a6f0f9eb57cf1f6c3a0ec0c4b@epcms2p1>
@ 2026-10-07  2:10   ` Daejun Park
  0 siblings, 0 replies; 9+ messages in thread
From: Daejun Park @ 2026-10-07  2:10 UTC (permalink / raw)
  To: Chuck Lever
  Cc: Jeff Layton, NeilBrown, Olga Kornievskaia, Dai Ngo, Tom Talpey,
	Christoph Hellwig, Benjamin Coddington, Trond Myklebust,
	Anna Schumaker, linux-nfs, linux-kernel, Daejun Park

On Tue, Oct 06, 2026 at 03:46:40PM -0400, Chuck Lever wrote:
> I think the fix has to come first. Before the WRITE or SETATTR, call
> break_layout() non-blocking to start the recall, and return
> NFS4ERR_DELAY on -EWOULDBLOCK, as we do for delegations. Then the
> fence worker and the poll loop can take as long as they need and no
> nfsd thread waits on them.

Agreed, I will put that patch first in v2. Besides WRITE and SETATTR
with a size, XFS breaks layouts in fallocate and in remap, where
xfs_iolock_two_inodes_and_break_layout() waits on the source file as
well, so ALLOCATE, DEALLOCATE, CLONE and COPY will be checked too, the
last two on both files.

That also covers a wait that is unbounded today. For the SCSI layout,
when fencing keeps failing, the fence worker retries without limit and
nfsd4_layout_lm_breaker_timedout() returns false while it is in flight,
so the thread in __break_lease() keeps waiting (read from the code, not
run).

Two things I will test and report with v2:

- The check in nfsd and the break_layout() in XFS are not atomic. A
  LAYOUTGET granted between them still puts the thread to sleep in
  xfs_break_leased_layouts(). No new layout lease is granted while a
  recall is in progress, so this needs a layout granted when none was
  held at the check.

- A writer going through the MDS now retries after NFS4ERR_DELAY
  instead of waking up when the layout is returned, so a client that
  keeps getting new layouts could starve it.

Daejun

^ permalink raw reply	[flat|nested] 9+ messages in thread

end of thread, other threads:[~2026-10-07  2:11 UTC | newest]

Thread overview: 9+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2026-10-06  4:29 [PATCH 0/6] nfsd: layout recall and lease break fixes Daejun Park via B4 Relay
2026-10-06  4:29 ` [PATCH 1/6] nfsd: make the time limit of a layout recall work Daejun Park via B4 Relay
2026-10-06  4:29 ` [PATCH 2/6] nfsd: run the fence script when a block layout lease break times out Daejun Park via B4 Relay
2026-10-06  4:29 ` [PATCH 3/6] nfsd: keep polling a layout recall answered with NFS4ERR_OLD_STATEID Daejun Park via B4 Relay
2026-10-06  4:29 ` [PATCH 4/6] nfsd: fail a LAYOUTGET in progress when the layout lease is broken Daejun Park via B4 Relay
2026-10-06  4:29 ` [PATCH 5/6] nfsd: poll a layout recall with a new seqid when the client is past it Daejun Park via B4 Relay
2026-10-06  4:29 ` [PATCH 6/6] nfsd: do not fence a client for a layout recall with no layouts left Daejun Park via B4 Relay
2026-10-06 19:46 ` [PATCH 0/6] nfsd: layout recall and lease break fixes Chuck Lever
     [not found] ` <CGME20261006194708epcas2p2df158d7a6f0f9eb57cf1f6c3a0ec0c4b@epcms2p1>
2026-10-07  2:10   ` Daejun Park

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox

all inboxes | Powered by JetHome®