[PATCH v3 0/7] nfsd: layout recall and lease break fixes

[PATCH v3 0/7] nfsd: layout recall and lease break fixes

[Daejun Park via B4 Relay]

nfsd polls a client that has accepted a CB_LAYOUTRECALL by sending the
callback again every 10 ms, with the stateid of the first callback. A
Linux client from v6.8 on answers the repeat with NFS4ERR_OLD_STATEID,
and nfsd takes that as a failed recall. So a client that needs more than
10 ms to return a layout has the layout revoked and the fence path run.
With the block layout that client loses data it has written and synced,
with the SCSI layout it is cut off from the device. Patch 6 lets nfsd
keep polling instead.

A recall that takes longer makes the operation that started it wait
longer. When that operation is a WRITE, a SETATTR that changes the
size, an ALLOCATE, a DEALLOCATE, a CLONE or a COPY, the nfsd thread
serving it waits in XFS until the layouts are returned or the lease
break times out. As Chuck pointed out on v1, a few such requests can
take every nfsd thread, and the server then cannot serve the
LAYOUTRETURN that would release them. Patch 2 recalls the layouts
before the operation without waiting, gives the layouts of other
clients 30 ms to come back when it starts their recall, and otherwise
answers NFS4ERR_DELAY.

The cost of patch 2: a client that writes to a file through its layout
and, for writes that are not aligned, through the server recalls its
own layout with each such WRITE and has its LAYOUTGETs fail with
NFS4ERR_RECALLCONFLICT meanwhile. In test 2 it wrote 48, 46 and 4 1 MiB
blocks through its layout in 30 seconds instead of 534, 537 and 505;
before, 6 of 7, 4 of 4 and 17 of 20 recalls ended in the failure path
and 7, 4 and 20 blocks were lost. The series on top of this one, with
the NFS client fix it needs, brings that to 193, 192 and 184 blocks.

Patches 3 to 5 are the others that patch 6 needs. Patch 3: the limit of
two lease periods on the poll has never triggered, and without it the
poll would have no end. Patch 4: for the block layout a lease break
gives up after fs.lease-break-time (45 s) without running
/sbin/nfsd-recall-failed. Until now the recall had failed long before
that; with patch 6 it is polled for up to two lease periods (180 s).
Patch 4 fences the client at fs.lease-break-time, as the SCSI layout
does since v7.1. Patch 5 then ends the recall, for both layout types:
polled on, the fenced client would keep every LAYOUTGET for the file
failing with NFS4ERR_RECALLCONFLICT for up to 135 s, and its
LAYOUTCOMMIT would be applied after the operation that broke the lease.
Ending the recall drops its layouts, so what it wrote through them and
had not committed is lost. By default fs.lease-break-time is shorter
than the one lease period RFC 8881 section 12.5.5 has the server wait
after a recall.

Patch 1 fixes a lease break that hits a LAYOUTGET in progress and then
waits for a layout that is never recalled. It comes first because
patch 2 makes that common: the break of patch 2 runs outside the XFS
iolock, so it can land between the allocation of a layout stateid and
the insertion of its first layout.

Patch 6 also handles a client that a LAYOUTRETURN has moved past the
seqid of the recall. Such a client answers every poll
NFS4ERR_OLD_STATEID and is never asked for the layouts it still holds,
so patch 6 polls a client that has not answered NFS4_OK with a new
seqid. Patch 7 deals with a client that answers NFS4ERR_BAD_STATEID
after returning its last layout, which is fenced: it looks for layouts
first.

Patches 2 to 6 have to go together and need patch 1, also in stable
kernels; patch 7 stands on its own. If patch 2 should not go to stable,
patches 3 to 6 should not either; patches 1 and 7 can go alone. Kernels
before v7.1 do not have the lease break timeout that patches 4 and 5
fix, and take neither; there an operation on the server that does not
come from nfsd waits for the recall for up to two lease periods, as it
already does, without a limit, for a client that keeps answering
NFS4ERR_DELAY. Patch 2 uses nfsd_thread_local_info and needs adapting
where that is missing.

Tests. Tests 1 to 6 are from v2; the v3 runs are in test 7. QEMU guests
on one host: an nvmet-tcp target, the server (nfsd, XFS on the nvmet
namespace, pnfs export) and up to three clients, all on the same kernel
with KASAN and lockdep, NFS v4.2, block layout unless noted, O_DIRECT
I/O. "v2" is v2 of this series on nfsd-testing 56589cdb5881, "v1" is v1
on it, and "before" is that commit unless noted. No KASAN or lockdep
report on the server in any run, nor on the clients and the target in
tests 2, 3 and 5, where they were checked.

1. nfsd threads (patch 2), 4 nfsd threads. A client holds a layout of
   a file and its path to the device is cut for 70 s. Another client,
   without pNFS, sends 4 KiB WRITEs to the file through the server, all
   at once, and reads 4 KiB of another file every second:

                                            before     v1        v2
   8 WRITEs, NFSv4.0: the WRITEs take       46.7 s     -         45.2 s
     longest read of the other file         46.7 s     -         0.031 s
   64 WRITEs, NFSv4.0: the WRITEs take      -          49.2 s    45.4 s
     longest read of the other file         -          49.1 s    0.22 s
   8 WRITEs, NFSv3: the WRITEs take         47.0 s     -         51.2 s
     longest read of the other file         46.9 s     -         0.021 s

   The WRITEs wait for the lease break, which ends with
   /sbin/nfsd-recall-failed (patch 4). NFSv3 WRITEs take longer with
   v2 because the client waits 5 s after NFS3ERR_JUKEBOX. In the run
   with 64 WRITEs before this series, the recall failed after 10 ms, as
   described above, and the WRITEs were done in 0.5 s.

2. What patch 2 costs. Two clients write 1 MiB blocks to a new file for
   30 s, one through its layout and one (NFSv4.0) through the server,
   three runs. "+ follow-up" adds the nfsd series that goes on top of
   this one and an NFS client fix that it needs.

                      through the     of those, read      through the
                      layout          back as zeroes      server
   before             416/200/186     28/131/111          120/325/323
   v2                 174/176/170     0/0/0               184/203/220
   v2 + follow-up     170/175/176     0/0/0               284/223/207

   The 30 ms wait is what keeps the writer through the server going:
   two earlier revisions of v2 + follow-up that differ only in it gave
   79/79/80 blocks through the server without the wait and 238/230/137
   with it.

   One client doing both, 1 MiB blocks through its layout and 512 byte
   writes through the server:

                      through the     of those, read      writes through
                      layout          back as zeroes      the server
   before             534/537/505     7/4/20              32/21/87
   v2                 48/46/4         0/0/0               3600/3262/4917
   v2 + follow-up     193/192/184     0/0/0               1082/1488/1669

   With v2 alone, each WRITE of that client through the server recalls
   its layout. About half of its LAYOUTGETs then meet that recall and
   get NFS4ERR_RECALLCONFLICT, as before; the other half are in
   progress when the WRITE breaks the lease, and patch 1 fails them.
   The Linux client waits up to 15 s between LAYOUTGETs that fail with
   NFS4ERR_RECALLCONFLICT. The follow-up hands out the layout of such a
   LAYOUTGET and recalls it at once. That needs the referring call the
   follow-up adds to CB_LAYOUTRECALL: tried in this series, the recall
   overtook the LAYOUTGET reply, and in all three runs 92 to 104 of
   about 260 blocks read back as zeroes.

   On 214e388464bf, before and v2 gave the same picture, except that in
   one of three runs of the one-client case with v2 one block read back
   as zeroes, through the recall race described under "Not changed".

   The series applies to nfsd-testing c32fef6c476d unchanged, and there
   before and v2 gave the same picture again, also for NFSv3, whose
   replies now come from xdrgen. In test 1 the longest read of the
   other file was 47.8 s before and 0.036 s with v2 over NFSv4.0, and
   48.3 s and 0.019 s over NFSv3, where the WRITEs took 48.4 s and
   51.3 s. In this test, with two clients, 48, 94 and 90 of 180, 185
   and 181 blocks through the layout read back as zeroes before, and
   none of 161, 177 and 172 with v2; with one client, 8, 0 and 3
   before and none with v2.

   Writes to a file without leases: fio, NFSv4.0, no KASAN or lockdep,
   10 runs of 30 s, medians without and with an earlier revision of v2
   (the same path for such a file): 1 MiB sequential 65.8 and 67.5
   MiB/s, 4 KiB random 262 and 263 IOPS.

3. Recall polling (patches 3, 4 and 6). With v2 no recall ended in the
   failure path (0 of 9, 9, 63 and 277 in the arms of patch 6) and no
   block written to a new file was lost (0 of 545, 511, 477, 577 and
   36). Patch 6 has the runs before.
   On c32fef6c476d, four of those arms lost 4, 2, 18 and 9 blocks
   before, with 46 recall failures logged, and none with v2, with no
   recall failure.

4. One client writing through a block layout whose path to the device
   is cut while it can still talk to the server, and a 4 KiB write to
   the file on the server ("before" of the 15 s case is 32eb1a60b456).
   The client's write that is in flight stays stuck until the path is
   back.

                                                   before      v2
   path cut for 15 s: the server write takes       0.05 s      15.6 s
     the recall                                    fails       completes
     /sbin/nfsd-recall-failed is run               at once     no
   path cut for 70 s: the server write takes       0.05 s      47.2 s
     /sbin/nfsd-recall-failed is run               at once     at 47.1 s

   The same 70 s case with a script that expires the client it fences
   through /proc/fs/nfsd/clients/<clid>/ctl: with v1's fence worker,
   which runs the script itself, the script never finished and the
   server write went on only after the path was back, after 93 s. With
   v2 the script was done 36 ms after it started, and the server write
   took 46.1 s. With a lease time of 15 s, the recall reaches its limit
   (patch 3) after 30 s, before the lease break times out, and the
   recall failure path runs the script: there it hangs with v2 as well,
   the script in nfsd4_shutdown_callback() and rpciod in
   call_usermodehelper_exec().

5. A lease break during a LAYOUTGET (patch 1). With v2 the longest of
   424 writes on the server took 55 ms, and 108 ms with a delay of
   100 ms added to LAYOUTGET for the test. Patch 1 has the runs without
   it.

6. SCSI layout. The target runs a kernel with "nvmet: fix preempt of
   another registrant by the reservation holder" [1], so that the
   preempt removes the registration. With v2, the path of a client cut
   for 15 s: the server write takes 15.2 s and the client is not
   fenced; cut for 70 s: 47.6 s, and the fence worker fences the client
   (registrants 2 -> 1).

7. v3, on nfsd-testing 78c334bc5ae6, the same guests and kernels with
   KASAN and lockdep. "before" is that commit, and "v2" is v2 of this
   series rebased onto it without changes. No KASAN or lockdep report on
   the server in any v3 run.

   - Test 1, 8 WRITEs: over NFSv4.0 the WRITEs took 48.4 s before and
     45.2 s with v3, and the longest read of the other file 48.4 s and
     0.027 s. Over NFSv3 they took 51.4 s with v3 and the longest read
     0.008 s; before, the recall failed after 10 ms in that run and the
     WRITEs were done at once.
   - Two clients writing 600 1 MiB blocks each to a new file through
     their layouts, three runs: before, 13 and 7 blocks of one client
     read back as zeroes in two runs; with v3 none. One client writing
     through its layout and through the server with unaligned writes:
     before, 12 of 600 blocks read back as zeroes in one of three runs;
     with v3, 52, 13 and 54 blocks through the layout and none zero.
   - Test 3's arms: before, 34 recalls ended in the failure path and 5
     blocks read back as zeroes; with v3, none and none.
   - Test 4 with v3: path cut for 15 s, the server write took 15.4 s and
     the recall completed. Cut for 70 s, the script ran when the lease
     break timed out, the server write went on after 48.4 s, and the
     recall ended right after the script (patch 5); the client's
     LAYOUTCOMMIT when its path was back got NFS4ERR_BADLAYOUT. With v2
     the recall ended when the client returned its layout, and that
     LAYOUTCOMMIT was applied. With the script that expires the client,
     it was done 19 ms after it started.
   - Cut for 210 s: the script ran once with v2 and with v3. With v2 the
     recall was polled until its time limit, 180 s after it started;
     with v3 it ended right after the script, 47.5 s after it started.
     The same with the SCSI layout (test 6): the fence worker fenced the
     client with both, and with v3 the recall ended 10 ms after the
     fence.
   - An 8 MiB asynchronous COPY into a file whose layout holder has its
     path cut for 70 s (patch 2): with v2 the copy kthread waited in XFS
     and the COPY was done after 48 s; with v3 it got NFS4ERR_DELAY 9
     times and was done after 84 s, 31 s after the holder was fenced at
     a retry 53 s in. Without the cut, about 50 ms either way. The data
     was in place in all four runs.
   - Read from the code, not run: the inter-server COPY path of patch 2,
     the retry of a failed allocation in patch 4, and the loss of what a
     fenced client wrote and had not committed (patch 5).

pynfs BLOCK1, BLOCK2, BLOCK4 and BLOCK5 pass before and with v2 (BLOCK3
fails in both with a TypeError in the test), and the same with v3 on
78c334bc5ae6. The commit messages of patches 6 and 7 have pynfs clients
that answer as described there.

Not changed by these patches:

- A layout granted between the check of patch 2 and the break in XFS
  still makes the nfsd thread wait. No new layout lease is granted
  while a break is pending, so that needs a LAYOUTGET for a file that
  had no layout at the check. For an asynchronous COPY the window is
  the whole copy: a layout granted for its destination while it runs
  makes the copy kthread wait in XFS at its next 4 MiB step.

- The recall failure path still runs /sbin/nfsd-recall-failed from
  nfsd4_cb_layout_done(), in rpciod, while the callback is in flight.
  A script that expires the client it fences hangs there, because the
  teardown waits for that callback in nfsd4_shutdown_callback(); test 4
  shows it. Neither path looks at the exit status of the script, and
  the recall failure path does not look at the result of
  ->fence_client(): a fence that fails is taken as done there, and the
  layouts it drops end the retries of a fence worker that a lease break
  has started (patch 3 says so).

- nfsd still polls a client that has answered NFS4_OK with the seqid it
  has used, where RFC 8881 section 12.5.3 has the server increment the
  seqid in each request. Patch 6 only makes nfsd bear the answer and
  gives a new seqid only to a client that has not answered NFS4_OK,
  and while it polls, nfsd sends a callback every 10 ms for up to two
  lease periods.

- CB_LAYOUTRECALL carries no referring call, as CB_RECALL does since
  "NFSD: Send referring calls with CB_RECALL". When a recall overtakes
  the reply to the LAYOUTGET that created the layout stateid, the Linux
  client answers NFS4ERR_NOMATCHING_LAYOUT, nfsd drops the layout and
  the client uses it afterwards. nfsd also trusts
  NFS4ERR_NOMATCHING_LAYOUT while it still has layouts on the layout
  stateid. The series on top of this one adds the referring call and
  resends the recall once before it trusts that answer, and with it
  recalls a layout handed out by a LAYOUTGET that a lease break met,
  instead of failing that LAYOUTGET (see test 2).

- Which client makes progress when several clients use one file: the
  client that holds a layout now keeps it while it does I/O through it,
  where before its layout was revoked after 10 ms. Beyond the limits
  above, nothing here keeps one client from getting its layout back
  right after returning it and keeping others out again.

- A LAYOUTCOMMIT that a fenced client sends before the next poll ends
  its recall is still applied: nfsd4_layoutcommit() does not look at
  ls_fenced. That window lasts until the next answer to the recall: 10
  ms while the client answers.

[1] https://lore.kernel.org/r/20261002065931epcms2p1cb2c16f98cf3b05f8c859ad92b0ad465@epcms2p1

---
Changes in v3:
- Patch 2: recall the layouts of a COPY before it is split into a
  synchronous and an asynchronous copy, so that an asynchronous COPY
  gets NFS4ERR_DELAY as well; an inter-server COPY recalls only its
  destination. Keep the kernel-doc to the contract and the comments
  short, as Chuck asked:
  https://lore.kernel.org/r/deacbd2d-a82f-4691-92e5-0ea98856c3cc@slotpi15m67
- Patch 4, as Chuck suggested: retry a failed allocation of the script
  work item with the backoff of a failed ->fence_client(), skip the
  fence in nfsd4_cb_layout_done() when ls_fenced is set, and build the
  script arguments in one helper. The fence worker clears ls_file when
  it is done, so that path did not fence a second time before either;
  it still can while the worker's script runs. The recall failure path
  still runs the script synchronously, as the layouts are dropped right
  after it:
  https://lore.kernel.org/r/166857d6-bb91-4731-8a94-a6a7fbb722e4@slotpi15m67
- New patch 5 ends the recall once the fence worker of a lease break is
  done, for both layout types, and the layouts are dropped then. Polling
  the fenced client until the time limit kept LAYOUTGETs for the file
  failing with NFS4ERR_RECALLCONFLICT for up to 135 s and applied its
  LAYOUTCOMMIT after the operation that broke the lease.
- Squash v2 patches 5 and 6 into patch 6, as Jeff asked, and shorten
  their comments:
  https://lore.kernel.org/r/574bebd564ccb710cb5217998fdda5588361d12d.camel@kernel.org
- Jeff's Reviewed-by is on patches 1, 3, 6 and 7, and not on patches
  2, 4 and 5, which changed or are new.
- Rebased onto nfsd-testing 78c334bc5ae6, which has "nfsd: prevent a
  double put of revoked layout stateids", without conflicts.
- Link to v2: https://lore.kernel.org/r/20261014-nfsd-layout-recall-v2-0-764462bddfd1@samsung.com

Changes in v2:
- New patch 2, which recalls the layouts without waiting before an
  operation that modifies the file and answers NFS4ERR_DELAY, as Chuck
  asked:
  https://lore.kernel.org/r/63ef8d7c-086f-4cc8-84bb-72c1b23bf596@slotpi15m67
- v1 4/6 is now patch 1, ahead of patch 2, which makes its case common.
  v1 1/6 to 3/6 are patches 3 to 5, v1 5/6 and 6/6 are patches 6 and 7.
- Sashiko on v1:
  - 2/6, the fence script waiting for itself when it expires the client
    through /proc/fs/nfsd/clients/<clid>/ctl: patch 4 runs the script
    in a work item on a new workqueue, nfsd4_fence, and the fence
    worker checks every 100 ms whether it is done. Teardown waits only
    for the fence worker, which no longer waits for the script.
  - 2/6, sleeping fence workers filling system_dfl_wq: the same change;
    a script that runs long occupies nfsd4_fence.
  - 3/6, ktime_get() stops during suspend: patch 3 takes the time with
    ktime_get_boottime().
  - 5/6, si_generation written under sc_lock in ->done and read under
    ls_mutex only: patch 6 reads it under sc_lock in
    nfsd4_preprocess_layout_stateid() as well.
  - 1/6, the result of ->fence_client() not looked at on the recall
    failure path, which also ends the retries of the fence worker: not
    changed, said in patch 3. Patch 3 makes the path reachable after two
    lease periods for a client that keeps answering; before, a client
    from v6.8 on reached it after 10 ms.
  - 2/6, the exit status of the script not looked at: not changed.
    Retrying a failed script would block the file for good on a server
    that has no /sbin/nfsd-recall-failed. Said in patch 4.
  - 6/6, NFS4ERR_NOMATCHING_LAYOUT trusted while layouts are left:
    preexisting, not changed here (see the list above).
- Rebased onto nfsd-testing c32fef6c476d without changes, and tests 1
  to 3 run again there.
- Test numbers in the cover from v2; the commit messages of patches 1,
  4 and 5 keep their runs on nfsd-testing 32eb1a60b456 for the side
  without the patch. Commit messages with one space after a period.
- Link to v1: https://lore.kernel.org/r/20261006-nfsd-layout-recall-v1-0-b59b31caf342@samsung.com

---
Daejun Park (7):
      nfsd: fail a LAYOUTGET in progress when the layout lease is broken
      nfsd: answer NFS4ERR_DELAY instead of waiting for a layout recall
      nfsd: make the time limit of a layout recall work
      nfsd: run the fence script when a block layout lease break times out
      nfsd: end a layout recall once a lease break has fenced the client
      nfsd: keep polling a layout recall answered with NFS4ERR_OLD_STATEID
      nfsd: do not fence a client for a layout recall with no layouts left

 fs/nfsd/nfs4layouts.c | 343 +++++++++++++++++++++++++++++++++++++++++++++-----
 fs/nfsd/nfs4proc.c    |  10 ++
 fs/nfsd/pnfs.h        |  11 ++
 fs/nfsd/state.h       |   9 ++
 fs/nfsd/trace.h       |   2 +
 fs/nfsd/vfs.c         |  78 +++++++++++-
 fs/nfsd/vfs.h         |   1 +
 7 files changed, 423 insertions(+), 31 deletions(-)
---
base-commit: 78c334bc5ae61d23ca1cc2e43079c620f1d40866
change-id: 20261006-nfsd-layout-recall-ba868d33371e

Best regards,
-- 
Daejun Park <daejun7.park@samsung.com>

[PATCH v3 1/7] nfsd: fail a LAYOUTGET in progress when the layout lease is broken

[Daejun Park via B4 Relay]

From: Daejun Park <daejun7.park@samsung.com>

A layout stateid takes its FL_LAYOUT lease when it is allocated, before
the first layout is added to it. When the lease is broken in between,
nfsd4_recall_file_layout() finds no layouts and does nothing. The
LAYOUTGET then hands out a layout that is not recalled: the lease is
already marked as breaking, so ->lm_break() is not called for it again.
The breaker waits until the client gives the layout up by itself or the
lease break time is over; with the SCSI layout the client is then
fenced although it was never sent a recall. While the breaker waits,
kernel_setlease() fails for new layout stateids and the LAYOUTGETs of
other clients get NFS4ERR_DELAY.

Commit c5c707f96fc9 ("nfsd: implement pNFS layout recalls") marked such
a layout stateid as recalled, and since nothing kept the LAYOUTGET from
adding its layout afterwards, that layout was never recalled. Commit
fb610c4dbc99 ("nfsd: fix race to check ls_layouts") therefore made the
recall skip a layout stateid without layouts, which left the case
above.

Let the lease break mark such a layout stateid as recalled again, and
make nfsd4_insert_layout() refuse to add layouts to a recalled layout
stateid, which is what was missing the first time. The LAYOUTGET fails
with NFS4ERR_RECALLCONFLICT, the layout stateid is freed and the lease
goes away. No callback is sent for it; the new
nfsd_layout_recall_empty tracepoint shows where fi_lo_recalls was
raised without one. NFS4ERR_RECALLCONFLICT is what nfsd already answers
to a LAYOUTGET for a file with a recall in progress, also when the
recall went to another client (RFC 8881 section 15.1.14.3: "a
conflicting recall operation that is currently in progress for that
object"). The check in nfsd4_insert_layout() also covers a
layout stateid with layouts that is recalled between the checks in
nfsd4_layoutget() and nfsd4_insert_layout().

Only a lease break does this. A LAYOUTGET that finds a layout stateid
without layouts of another client leaves it alone, as before.

The window is the time a LAYOUTGET needs from allocating the layout
stateid to inserting the layout.

Two clients writing to one file while the server writes 4 KiB to it
every 50 ms, for 30 seconds, block layout (the runs without this patch
on nfsd-testing 32eb1a60b456, the runs with it with this whole series).
Without this patch one write on the server took 9.7 seconds in one run
and 13.9 seconds in another, in both until the clients were done; in a
third run it did not happen. With this patch the longest write on the
server took 55 ms.

To see the new path at work, a delay of 100 ms was added to LAYOUTGET
between the two steps for one pair of runs. Without the patch the
second write on the server took 30 seconds, the whole run. With it the
longest write took 108 ms and nfsd_layout_recall_empty fired 16 times.
(With that delay every LAYOUTGET meets a write on the server, so the
clients got no layout in that run. It shows the path and nothing else.)

Fixes: c5c707f96fc9 ("nfsd: implement pNFS layout recalls")
Signed-off-by: Daejun Park <daejun7.park@samsung.com>
Reviewed-by: Jeff Layton <jlayton@kernel.org>
---
 fs/nfsd/nfs4layouts.c | 49 +++++++++++++++++++++++++++++++++++++++++++------
 fs/nfsd/trace.h       |  1 +
 2 files changed, 44 insertions(+), 6 deletions(-)

diff --git a/fs/nfsd/nfs4layouts.c b/fs/nfsd/nfs4layouts.c
index f2f8620fd5..c33935f486 100644
--- a/fs/nfsd/nfs4layouts.c
+++ b/fs/nfsd/nfs4layouts.c
@@ -346,14 +346,15 @@ nfsd4_preprocess_layout_stateid(struct svc_rqst *rqstp,
 }
 
 static void
-nfsd4_recall_file_layout(struct nfs4_layout_stateid *ls)
+nfsd4_recall_file_layout_locked(struct nfs4_layout_stateid *ls)
 {
-	spin_lock(&ls->ls_lock);
+	lockdep_assert_held(&ls->ls_lock);
+
 	if (ls->ls_recalled)
-		goto out_unlock;
+		return;
 
 	if (list_empty(&ls->ls_layouts))
-		goto out_unlock;
+		return;
 
 	ls->ls_recalled = true;
 	atomic_inc(&ls->ls_stid.sc_file->fi_lo_recalls);
@@ -363,7 +364,34 @@ nfsd4_recall_file_layout(struct nfs4_layout_stateid *ls)
 		refcount_inc(&ls->ls_stid.sc_count);
 		nfsd4_run_cb(&ls->ls_recall);
 	}
-out_unlock:
+}
+
+static void
+nfsd4_recall_file_layout(struct nfs4_layout_stateid *ls)
+{
+	spin_lock(&ls->ls_lock);
+	nfsd4_recall_file_layout_locked(ls);
+	spin_unlock(&ls->ls_lock);
+}
+
+/*
+ * A lease break also has to stop a LAYOUTGET that is in progress. Its
+ * layout stateid has the lease and no layouts yet, so there is nothing to
+ * recall, and the breaker goes on once that layout stateid is gone. Mark it
+ * as recalled without sending a callback: nfsd4_insert_layout() then fails
+ * the LAYOUTGET instead of handing out a layout after the break.
+ */
+static void
+nfsd4_break_file_layout(struct nfs4_layout_stateid *ls)
+{
+	spin_lock(&ls->ls_lock);
+	if (!ls->ls_recalled && list_empty(&ls->ls_layouts)) {
+		ls->ls_recalled = true;
+		atomic_inc(&ls->ls_stid.sc_file->fi_lo_recalls);
+		trace_nfsd_layout_recall_empty(&ls->ls_stid.sc_stateid);
+	} else {
+		nfsd4_recall_file_layout_locked(ls);
+	}
 	spin_unlock(&ls->ls_lock);
 }
 
@@ -442,6 +470,10 @@ nfsd4_insert_layout(struct nfsd4_layoutget *lgp, struct nfs4_layout_stateid *ls)
 	if (nfserr)
 		goto out;
 	spin_lock(&ls->ls_lock);
+	if (ls->ls_recalled) {
+		nfserr = nfserr_recallconflict;
+		goto out_unlock_ls;
+	}
 	list_for_each_entry(lp, &ls->ls_layouts, lo_perstate) {
 		if (layouts_try_merge(&lp->lo_seg, seg))
 			goto done;
@@ -460,6 +492,10 @@ nfsd4_insert_layout(struct nfsd4_layoutget *lgp, struct nfs4_layout_stateid *ls)
 	if (nfserr)
 		goto out;
 	spin_lock(&ls->ls_lock);
+	if (ls->ls_recalled) {
+		nfserr = nfserr_recallconflict;
+		goto out_unlock_ls;
+	}
 	list_for_each_entry(lp, &ls->ls_layouts, lo_perstate) {
 		if (layouts_try_merge(&lp->lo_seg, seg))
 			goto done;
@@ -470,6 +506,7 @@ nfsd4_insert_layout(struct nfsd4_layoutget *lgp, struct nfs4_layout_stateid *ls)
 	new = NULL;
 done:
 	nfs4_inc_and_copy_stateid(&lgp->lg_sid, &ls->ls_stid);
+out_unlock_ls:
 	spin_unlock(&ls->ls_lock);
 out:
 	spin_unlock(&fp->fi_lock);
@@ -806,7 +843,7 @@ nfsd4_layout_lm_break(struct file_lease *fl)
 	 * Enforce break lease timeout to prevent NFSD
 	 * thread from hanging in __break_lease.
 	 */
-	nfsd4_recall_file_layout(fl->c.flc_owner);
+	nfsd4_break_file_layout(fl->c.flc_owner);
 	return false;
 }
 
diff --git a/fs/nfsd/trace.h b/fs/nfsd/trace.h
index e5d19c6f48..abfdbc637c 100644
--- a/fs/nfsd/trace.h
+++ b/fs/nfsd/trace.h
@@ -679,6 +679,7 @@ DEFINE_STATEID_EVENT(layout_get_lookup_fail);
 DEFINE_STATEID_EVENT(layout_commit_lookup_fail);
 DEFINE_STATEID_EVENT(layout_return_lookup_fail);
 DEFINE_STATEID_EVENT(layout_recall);
+DEFINE_STATEID_EVENT(layout_recall_empty);
 DEFINE_STATEID_EVENT(layout_recall_done);
 DEFINE_STATEID_EVENT(layout_recall_fail);
 DEFINE_STATEID_EVENT(layout_recall_release);

-- 
2.43.0

[PATCH v3 2/7] nfsd: answer NFS4ERR_DELAY instead of waiting for a layout recall

[Daejun Park via B4 Relay]

From: Daejun Park <daejun7.park@samsung.com>

A WRITE, a SETATTR that changes the size (also an OPEN that truncates
and an NFSv2 or NFSv3 CREATE of an existing file with a size), an
ALLOCATE, a DEALLOCATE, a CLONE and a COPY make XFS recall the layouts
of the file and wait in __break_lease() until they are returned or the
lease break times out: xfs_break_leased_layouts(), and
xfs_iolock_two_inodes_and_break_layout() for both files of a clone. The
nfsd thread serving the request sleeps there too. The client decides
when it returns its layout, so a few such requests can occupy every nfsd
thread, and the server then cannot process the LAYOUTRETURN that would
release them.

Before calling into the filesystem, recall the layouts without waiting,
as nfsd_open_break_lease() does for delegations, and return
-EWOULDBLOCK while a layout is held, which nfserrno() maps to
NFS4ERR_DELAY (NFS3ERR_JUKEBOX for NFSv3), so that the client retries.
When this starts the recall of a layout of another client, first give
the layouts NFSD_LAYOUTRETURN_TIMEOUT to come back, the 30 ms that
nfsd_wait_for_delegreturn() allows a delegation: a client that is not
doing I/O through its layout returns it within a few milliseconds, and
the retry would cost the client of the request at least 100 ms. Do not
wait for a recall that is already under way, nor when the client of
the request holds a layout of the file itself: it returns that layout
only after its own I/O through it has completed. nfsd4_close_layout()
wakes the waiter when a layout lease is removed. For a COPY or a CLONE,
both recalls are started before the wait. For a SETATTR, the recall
comes before inode_lock, which the LAYOUTCOMMIT that a client sends
before its LAYOUTRETURN takes. A file without leases costs one lockless
check of its lease list; a file with other leases, such as
delegations, a walk of that list under flc_lock.

Answering NFS4ERR_DELAY here was discussed in 2023 (second Link).
Passing a non-blocking flag down to the break in the filesystem would
need VFS and XFS changes and a way through notify_change(),
vfs_fallocate() and the remap path; breaking before the call, as
nfsd_open_break_lease() does for delegations, needs neither.

An nfsd thread can still block in the filesystem when a LAYOUTGET
completes between this check and the recall in the filesystem. No layout
lease can be added while a recall is pending, so this takes a file that
had no layout when nfsd checked. A recall that meets a LAYOUTGET in
progress fails that LAYOUTGET (previous patch), so the layout it would
have handed out is not left unrecalled. A COPY recalls the layouts
before it is split into a synchronous and an asynchronous copy, so an
asynchronous COPY gets NFS4ERR_DELAY as well instead of starting a
kthread that waits for the recall. That kthread copies 4 MiB at a time,
and a layout handed out while it runs still makes it wait in the
filesystem. The source of an inter-server COPY is not on this server, so
only its destination is recalled.

With a client that holds a layout of the destination and has its path to
the device cut for 70 seconds, an 8 MiB asynchronous COPY from another
client got NFS4ERR_DELAY 9 times and was done after 84 seconds: the
client holding the layout was fenced at a retry 53 seconds in, and the
COPY went on at a retry 31 seconds later. With v2 of this series, which
recalled only for a synchronous COPY, the copy kthread waited in XFS and
the COPY was done after 48 seconds. Without the cut, both took about 50
ms, and the data was in place in all four runs.

The cost is on a request that conflicts with a layout. When the layout
does not come back within 30 ms, or the client of the request holds it,
the request completes only when the client sends it again. The Linux
client does that after at least 100 ms for NFSv4 (NFS4_POLL_RETRY_MIN)
and after 5 seconds for NFSv3 (NFS_JUKEBOX_RETRY_TIME), where it used to
wait only until the layout was returned. An NFSv2 WRITE has its reply
dropped, as for any NFSERR_JUKEBOX, and an NFSv2 SETATTR or CREATE is
answered as for a delegation conflict. The recall now also comes before
the checks that would fail the request: a COPY recalls the layouts of
its source as well, because the filesystem may do it as a clone, also
where it cannot, and a CLONE on a filesystem without reflink can get
NFS4ERR_DELAY before NFS4ERR_NOTSUPP. An NFSv2 or NFSv3 CREATE of an
existing file with a size does the SETATTR with the parent directory
locked and holds it during the wait.

With 4 nfsd threads, a client that holds a layout of a file and has
lost its path to the device, and another client (NFSv4.0) that sends 8
4 KiB WRITEs to that file through the metadata server while it reads
4 KiB of another file every second: the WRITEs take about 45 seconds
with and without this patch, as they wait for the lease break, and
without it the reads of the other file wait as long. With it the
longest of 84 reads takes 31 ms, and 0.22 seconds with 64 WRITEs. With
the second client on NFSv3, the longest read takes 21 ms instead of 47
seconds, and the WRITEs take 51 seconds instead of 47, as the client
waits 5 seconds after NFS3ERR_JUKEBOX.

A client that writes to a file both through its layout and through the
server recalls its own layout with each such WRITE. About half of its
LAYOUTGETs then meet that recall and get NFS4ERR_RECALLCONFLICT, as
before; the others are in progress when the WRITE breaks the lease, and
the previous patch fails them. The Linux client waits up to 15 seconds
between LAYOUTGETs that fail with NFS4ERR_RECALLCONFLICT. Writing 1 MiB
blocks through its layout and 512 bytes through the server for 30
seconds, such a client wrote 48, 46 and 4 blocks through the layout in
three runs, against 534, 537 and 505 without this series. There most
recalls of its layout ended in the failure path, and 7, 4 and 20 of the
blocks read back as zeroes after a successful fsync(); none did with
this series.

Writes to a file without leases are not changed: fio with O_DIRECT,
NFSv4.0, 10 runs of 30 seconds, without KASAN and lockdep, gave median
65.8 and 67.5 MiB/s for 1 MiB sequential writes without and with an
earlier revision of this series (the same path for such a file), and 262
and 263 IOPS for 4 KiB random writes.

Fixes: c5c707f96fc9 ("nfsd: implement pNFS layout recalls")
Suggested-by: Chuck Lever <cel@kernel.org>
Link: https://lore.kernel.org/r/63ef8d7c-086f-4cc8-84bb-72c1b23bf596@slotpi15m67
Link: https://lore.kernel.org/r/ZVdj2U+kJ7DlI22g@infradead.org
Signed-off-by: Daejun Park <daejun7.park@samsung.com>
---
 fs/nfsd/nfs4layouts.c | 57 ++++++++++++++++++++++++++++++++++++-
 fs/nfsd/nfs4proc.c    | 10 +++++++
 fs/nfsd/pnfs.h        | 11 ++++++++
 fs/nfsd/state.h       |  1 +
 fs/nfsd/vfs.c         | 78 ++++++++++++++++++++++++++++++++++++++++++++++++++-
 fs/nfsd/vfs.h         |  1 +
 6 files changed, 156 insertions(+), 2 deletions(-)

diff --git a/fs/nfsd/nfs4layouts.c b/fs/nfsd/nfs4layouts.c
index c33935f486..1f06478056 100644
--- a/fs/nfsd/nfs4layouts.c
+++ b/fs/nfsd/nfs4layouts.c
@@ -153,13 +153,68 @@ void nfsd4_close_layout(struct nfs4_layout_stateid *ls)
 	spin_unlock(&ls->ls_stid.sc_file->fi_lock);
 
 	if (fl) {
-		if (!nfsd4_layout_ops[ls->ls_layout_type]->disable_recalls)
+		if (!nfsd4_layout_ops[ls->ls_layout_type]->disable_recalls) {
 			kernel_setlease(fl->nf_file, F_UNLCK, NULL,
 					(void **)&ls);
+			/*
+			 * For nfsd_break_layouts(). No barrier needed: its
+			 * waiter rechecks under flc_lock, which removing the
+			 * lease took.
+			 */
+			wake_up_var(file_inode(fl->nf_file));
+		}
 		nfsd_file_put(fl);
 	}
 }
 
+/**
+ * nfsd4_layout_holders - Report which clients hold layout leases on a file
+ * @inode: file whose layout leases are checked
+ *
+ * Context: Takes the flc_lock of @inode.
+ *
+ * Return: a mask of NFSD_LAYOUTS_OWN if the client of the current request
+ * holds a layout lease on @inode, NFSD_LAYOUTS_OTHER if another client
+ * holds one, and NFSD_LAYOUTS_IDLE if a lease of another client is not
+ * being broken yet.
+ */
+unsigned int nfsd4_layout_holders(struct inode *inode)
+{
+	struct file_lock_context *ctx = locks_inode_context(inode);
+	struct nfs4_client *clp = NULL;
+	unsigned int holders = 0;
+	struct svc_rqst *rqstp;
+	struct file_lease *fl;
+
+	if (!ctx || list_empty_careful(&ctx->flc_lease))
+		return 0;
+	/* Without a compound, as for NFSv3, every layout is another client's */
+	rqstp = nfsd_current_rqst();
+	if (nfsd_v4client(rqstp)) {
+		struct nfsd_thread_local_info *ntli = rqstp->rq_private;
+
+		clp = *ntli->ntli_lease_breaker;
+	}
+
+	spin_lock(&ctx->flc_lock);
+	for_each_file_lock(fl, &ctx->flc_lease) {
+		struct nfs4_layout_stateid *ls = fl->c.flc_owner;
+
+		if (fl->fl_lmops != &nfsd4_layouts_lm_ops)
+			continue;
+		if (clp && ls->ls_stid.sc_client == clp) {
+			holders |= NFSD_LAYOUTS_OWN;
+			continue;
+		}
+		holders |= NFSD_LAYOUTS_OTHER;
+		if (!(fl->c.flc_flags &
+		      (FL_UNLOCK_PENDING | FL_DOWNGRADE_PENDING)))
+			holders |= NFSD_LAYOUTS_IDLE;
+	}
+	spin_unlock(&ctx->flc_lock);
+	return holders;
+}
+
 static void
 nfsd4_free_layout_stateid(struct nfs4_stid *stid)
 {
diff --git a/fs/nfsd/nfs4proc.c b/fs/nfsd/nfs4proc.c
index 0848baad68..a50f9dcb3a 100644
--- a/fs/nfsd/nfs4proc.c
+++ b/fs/nfsd/nfs4proc.c
@@ -2298,6 +2298,7 @@ nfsd4_copy(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate,
 	struct nfsd4_async_copy *async_copy = NULL;
 	struct nfsd4_copy *copy = &u->copy;
 	struct nfsd42_write_res *result;
+	struct inode *src, *dst;
 	__be32 status;
 
 	result = &copy->cp_res;
@@ -2322,6 +2323,15 @@ nfsd4_copy(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate,
 			goto out;
 	}
 
+	/*
+	 * A copy can be done as a clone, which breaks both files. The source
+	 * of an inter-server copy is not local.
+	 */
+	dst = file_inode(copy->nf_dst->nf_file);
+	src = copy->nf_src ? file_inode(copy->nf_src->nf_file) : dst;
+	if (nfsd_break_layouts(dst, src != dst ? src : NULL))
+		goto out_err;
+
 	memcpy(&copy->fh, &cstate->current_fh.fh_handle,
 		sizeof(struct knfsd_fh));
 	if (nfsd4_copy_is_async(copy)) {
diff --git a/fs/nfsd/pnfs.h b/fs/nfsd/pnfs.h
index 9b0f17a4d1..e5f2e11c88 100644
--- a/fs/nfsd/pnfs.h
+++ b/fs/nfsd/pnfs.h
@@ -72,6 +72,11 @@ int nfsd4_set_deviceid(struct nfsd4_deviceid *id, const struct svc_fh *fhp,
 struct nfsd4_deviceid_map *nfsd4_find_devid_map(int idx);
 #endif /* CONFIG_NFSD_V4 */
 
+/* The mask that nfsd4_layout_holders() returns */
+#define NFSD_LAYOUTS_OWN	0x1
+#define NFSD_LAYOUTS_OTHER	0x2
+#define NFSD_LAYOUTS_IDLE	0x4
+
 #ifdef CONFIG_NFSD_PNFS
 void nfsd4_setup_layout_type(struct svc_export *exp);
 void nfsd4_return_all_client_layouts(struct nfs4_client *);
@@ -79,6 +84,7 @@ void nfsd4_return_revoked_layout(struct nfs4_layout_stateid *ls);
 void nfsd4_return_all_file_layouts(struct nfs4_client *clp,
 		struct nfs4_file *fp);
 void nfsd4_close_layout(struct nfs4_layout_stateid *ls);
+unsigned int nfsd4_layout_holders(struct inode *inode);
 void nfsd4_stop_layout_fence(struct nfs4_layout_stateid *ls);
 int nfsd4_init_pnfs(void);
 void nfsd4_exit_pnfs(void);
@@ -107,6 +113,11 @@ static inline void nfsd4_close_layout(struct nfs4_layout_stateid *ls)
 {
 }
 
+static inline unsigned int nfsd4_layout_holders(struct inode *inode)
+{
+	return 0;
+}
+
 static inline void nfsd4_stop_layout_fence(struct nfs4_layout_stateid *ls)
 {
 }
diff --git a/fs/nfsd/state.h b/fs/nfsd/state.h
index 20ab27221c..b1f4c167ea 100644
--- a/fs/nfsd/state.h
+++ b/fs/nfsd/state.h
@@ -62,6 +62,7 @@
 #define	NFS4_CLIENTS_PER_GB		1024
 #define NFSD_DELEGRETURN_TIMEOUT	(HZ / 34)	/* 30ms */
 #define	NFSD_CB_GETATTR_TIMEOUT		NFSD_DELEGRETURN_TIMEOUT
+#define	NFSD_LAYOUTRETURN_TIMEOUT	NFSD_DELEGRETURN_TIMEOUT
 
 typedef struct {
 	u32             cl_boot;
diff --git a/fs/nfsd/vfs.c b/fs/nfsd/vfs.c
index f4069da853..c4b32e41b4 100644
--- a/fs/nfsd/vfs.c
+++ b/fs/nfsd/vfs.c
@@ -46,6 +46,7 @@
 #include "netns.h"
 #include "stats.h"
 #include "vfs.h"
+#include "pnfs.h"
 #include "filecache.h"
 #include "trace.h"
 
@@ -583,6 +584,15 @@ nfsd_setattr(struct svc_rqst *rqstp, struct svc_fh *fhp,
 		err = nfsd_get_write_access(fhp);
 		if (err)
 			return err;
+		/*
+		 * Not under inode_lock: the LAYOUTCOMMIT that a client sends
+		 * before it returns its layout takes it.
+		 */
+		err = nfserrno(nfsd_break_layouts(inode, NULL));
+		if (err) {
+			put_write_access(inode);
+			return err;
+		}
 	}
 
 	inode_lock(inode);
@@ -721,6 +731,14 @@ int nfsd_clone_file_range(struct file *src, u64 src_pos, struct file *dst,
 			  u64 dst_pos, u64 count, errseq_t *since)
 {
 	loff_t cloned;
+	int ret;
+
+	/* The filesystem breaks the layouts of both files */
+	ret = nfsd_break_layouts(file_inode(src),
+				 file_inode(dst) != file_inode(src) ?
+				 file_inode(dst) : NULL);
+	if (ret)
+		return ret;
 
 	*since = READ_ONCE(dst->f_wb_err);
 	cloned = vfs_clone_file_range(src, src_pos, dst, dst_pos, count, 0);
@@ -794,7 +812,9 @@ __be32 nfsd4_vfs_fallocate(struct svc_rqst *rqstp, struct svc_fh *fhp,
 	if (!S_ISREG(file_inode(file)->i_mode))
 		return nfserr_inval;
 
-	error = vfs_fallocate(file, flags, offset, len);
+	error = nfsd_break_layouts(file_inode(file), NULL);
+	if (!error)
+		error = vfs_fallocate(file, flags, offset, len);
 	if (!error)
 		error = commit_metadata(fhp);
 
@@ -881,6 +901,58 @@ int nfsd_open_break_lease(struct inode *inode, int access)
 	return break_lease(inode, mode | O_NONBLOCK);
 }
 
+static bool nfsd_other_layouts(struct inode *inode)
+{
+	return nfsd4_layout_holders(inode) & NFSD_LAYOUTS_OTHER;
+}
+
+/**
+ * nfsd_break_layouts - Recall the layouts of one or two files
+ * @inode: file that is about to be modified
+ * @inode2: the other file of a COPY or CLONE, or NULL
+ *
+ * Context: Process context. Sleeps for up to NFSD_LAYOUTRETURN_TIMEOUT.
+ *
+ * Return: zero if no layout is left, -EWOULDBLOCK while a recall is under
+ * way, or -ENOMEM
+ */
+int nfsd_break_layouts(struct inode *inode, struct inode *inode2)
+{
+	long timeout = NFSD_LAYOUTRETURN_TIMEOUT;
+	unsigned int holders;
+	int ret, ret2 = 0;
+
+	holders = nfsd4_layout_holders(inode);
+	if (inode2)
+		holders |= nfsd4_layout_holders(inode2);
+	if (!holders)
+		return 0;
+
+	/* Start the recalls without waiting, unlike the filesystem's break */
+	ret = break_layout(inode, false);
+	if (inode2)
+		ret2 = break_layout(inode2, false);
+	if (ret != -EWOULDBLOCK && ret2 != -EWOULDBLOCK)
+		return ret ? ret : ret2;
+	/*
+	 * Briefly wait for other clients' layouts if this started their
+	 * recall, as for delegations. The client of this request returns its
+	 * own only after its I/O through them.
+	 */
+	if ((holders & NFSD_LAYOUTS_OWN) || !(holders & NFSD_LAYOUTS_IDLE))
+		return -EWOULDBLOCK;
+
+	timeout = wait_var_event_timeout(inode, !nfsd_other_layouts(inode),
+					 timeout);
+	if (inode2 && timeout)
+		wait_var_event_timeout(inode2, !nfsd_other_layouts(inode2),
+				       timeout);
+	ret = break_layout(inode, false);
+	if (!ret && inode2)
+		ret = break_layout(inode2, false);
+	return ret;
+}
+
 /*
  * Open an existing file or directory.
  * The may_flags argument indicates the type of open (read/write/lock)
@@ -1490,6 +1562,10 @@ nfsd_vfs_write(struct svc_rqst *rqstp, struct svc_fh *fhp,
 		goto out_nfserr;
 	}
 
+	host_err = nfsd_break_layouts(file_inode(file), NULL);
+	if (host_err)
+		goto out_nfserr;
+
 	since = READ_ONCE(file->f_wb_err);
 	if (verf)
 		nfsd_copy_write_verifier(verf, nn);
diff --git a/fs/nfsd/vfs.h b/fs/nfsd/vfs.h
index eb75e4c823..1f05c83a3e 100644
--- a/fs/nfsd/vfs.h
+++ b/fs/nfsd/vfs.h
@@ -171,6 +171,7 @@ __be32		nfsd_setxattr(struct svc_rqst *rqstp, struct svc_fh *fhp,
 			    char *name, void *buf, u32 len, u32 flags);
 #endif
 int 		nfsd_open_break_lease(struct inode *, int);
+int		nfsd_break_layouts(struct inode *inode, struct inode *inode2);
 __be32		nfsd_open(struct svc_rqst *, struct svc_fh *, umode_t,
 				int, struct file **);
 int		nfsd_open_verified(struct svc_fh *fhp, umode_t type, int may_flags,

-- 
2.43.0

[PATCH v3 3/7] nfsd: make the time limit of a layout recall work

[Daejun Park via B4 Relay]

From: Daejun Park <daejun7.park@samsung.com>

nfsd4_cb_layout_done() polls a client that has answered CB_LAYOUTRECALL
and still holds layouts, and is meant to give up after two lease
periods: the recall then fails and the client is fenced. The limit is
computed from task->tk_start. The poll restarts the RPC task from
->rpc_call_done(), and rpc_exit_task() resets the statistics of a task
that was restarted there, tk_start included. So tk_start is the time
of the last poll, and the limit is never reached: a client that keeps
answering is polled every 10 ms for as long as it holds the layout.

Count the two lease periods from the first answer of the client, so that
the time in which a callback could not be delivered is not taken from
it, and take the time with ktime_get_boottime(), which goes on while the
server is suspended, as the lease times of nfsd do. A layout stateid is
recalled once in its life (ls_recalled is not cleared), so the time
needs no reset.

When the limit is reached, the existing failure path is taken. It does
not look at the result of ->fence_client(), so a SCSI layout client
whose fencing fails is treated as fenced there, and the layouts it drops
also end the retries of a fence worker that a lease break has started.
Before this patch a client from v6.8 on reached that path 10 ms after
the recall.

A Linux client before v6.8 answers every poll with NFS4_OK until its
LAYOUTRETURN is prepared (read from the code, not run). If such a
client does not return the layout, it was polled without end and is now
fenced after two lease periods, which is what commit 6b9b21073d3b
("nfsd: give up on CB_LAYOUTRECALLs after two lease periods") and
commit 851238a22f3b ("nfsd: fix error handling for clients that fail to
return the layout") set out to do.

Found by reading the code. The limit could only be tested together with
"nfsd: keep polling a layout recall answered with NFS4ERR_OLD_STATEID",
because a current Linux client does not keep answering a recall with
NFS4_OK or NFS4ERR_DELAY: a client writing through a block layout whose
path to the device is cut for 70 seconds while it can still talk to the
server, a lease time of 15 seconds, and a write to the file on the
server. The recall fails 30 seconds after the first answer of the client
and /sbin/nfsd-recall-failed is run. With the default lease time the
limit is 180 seconds; a recall that comes from a lease break is not
waited for that long: after fs.lease-break-time the breaker goes on,
with the SCSI layout once the client is fenced (and, with a later patch,
the block layout too).

Fixes: 6b9b21073d3b ("nfsd: give up on CB_LAYOUTRECALLs after two lease periods")
Signed-off-by: Daejun Park <daejun7.park@samsung.com>
Reviewed-by: Jeff Layton <jlayton@kernel.org>
---
 fs/nfsd/nfs4layouts.c | 15 +++++++++++----
 fs/nfsd/state.h       |  1 +
 2 files changed, 12 insertions(+), 4 deletions(-)

diff --git a/fs/nfsd/nfs4layouts.c b/fs/nfsd/nfs4layouts.c
index 1f06478056..ab7340343a 100644
--- a/fs/nfsd/nfs4layouts.c
+++ b/fs/nfsd/nfs4layouts.c
@@ -312,6 +312,7 @@ nfsd4_alloc_layout_stateid(struct nfsd4_compound_state *cstate,
 		return NULL;
 	}
 
+	ls->ls_recall_start = 0;
 	ls->ls_fenced = false;
 	ls->ls_fence_inflight = false;
 	ls->ls_fence_stopped = false;
@@ -834,12 +835,18 @@ nfsd4_cb_layout_done(struct nfsd4_callback *cb, struct rpc_task *task)
 			return 1;
 
 		/* Poll the client until it's done with the layout */
-		now = ktime_get();
+		now = ktime_get_boottime();
 		nn = net_generic(ls->ls_stid.sc_client->net, nfsd_net_id);
 
-		/* Client gets 2 lease periods to return it */
-		cutoff = ktime_add_ns(task->tk_start,
-					 (u64)nn->nfsd4_lease * NSEC_PER_SEC * 2);
+		/*
+		 * Client gets 2 lease periods to return it, counted from its
+		 * first answer. task->tk_start cannot be used for this, it
+		 * is set anew each time the callback is restarted.
+		 */
+		if (!ls->ls_recall_start)
+			ls->ls_recall_start = now;
+		cutoff = ktime_add_ns(ls->ls_recall_start,
+				      (u64)nn->nfsd4_lease * NSEC_PER_SEC * 2);
 
 		if (ktime_before(now, cutoff)) {
 			rpc_delay(task, HZ/100); /* 10 mili-seconds */
diff --git a/fs/nfsd/state.h b/fs/nfsd/state.h
index b1f4c167ea..f05e256421 100644
--- a/fs/nfsd/state.h
+++ b/fs/nfsd/state.h
@@ -853,6 +853,7 @@ struct nfs4_layout_stateid {
 	struct nfsd_file		*ls_file;
 	struct nfsd4_callback		ls_recall;
 	stateid_t			ls_recall_sid;
+	ktime_t				ls_recall_start;
 	bool				ls_recalled;
 	struct mutex			ls_mutex;
 

-- 
2.43.0

[PATCH v3 4/7] nfsd: run the fence script when a block layout lease break times out

[Daejun Park via B4 Relay]

From: Daejun Park <daejun7.park@samsung.com>

When a lease break has waited for fs.lease-break-time,
nfsd4_layout_lm_breaker_timedout() has the client that holds the layout
fenced before the breaker goes on. For a layout type without
->fence_client, which is the block layout, it returns true at once: the
lease is removed and the operation on the server goes on while the
client still holds the layout. /sbin/nfsd-recall-failed, which
Documentation/admin-guide/nfs/pnfs-block-server.rst names as the way to
fence such a client, is only run when the CB_LAYOUTRECALL fails.

This hits a client that keeps answering the recall with NFS4_OK or
NFS4ERR_DELAY for longer than fs.lease-break-time, 45 seconds by
default, for example because its path to the device is cut. With the
previous patch the recall is polled for two lease periods, 180 seconds
by default, so the lease break times out first. A Linux client before
v6.8 answers that way (read from the code, not run), and with "nfsd:
keep polling a layout recall answered with NFS4ERR_OLD_STATEID", later
in this series, every Linux client is polled that way; until now a
client from v6.8 on has the recall fail after 10 ms.

Let the fence worker run the script for a layout type without
->fence_client, and give up the lease after that. As on the recall
failure path, the exit status of the script is not looked at. The
same document says that the server retries a failed fence and keeps
the file from other clients meanwhile; that is what the fence worker
does with ->fence_client(). Doing it for the script would block the
file for good on a server that has no /sbin/nfsd-recall-failed, so
this patch leaves it as the recall failure path has it.

As for the SCSI layout since the commit this fixes, the client is then
fenced after fs.lease-break-time. That is shorter than one lease
period by default, while RFC 8881 section 12.5.5 has the server wait
one lease period after a recall before it takes further action.
Setting fs.lease-break-time to at least the lease time gives that wait
for both layout types.

The script runs on its own workqueue, nfsd4_fence, in a work item that
keeps copies of the client address and the device name and nothing of
the layout stateid or the client. The fence worker checks every 100 ms
whether it is done. Tearing down the client waits for the fence worker
since "nfsd: drain pNFS fence work during state teardown", and the
script may be what tears the client down, through
/proc/fs/nfsd/clients/<clid>/ctl: run by the fence worker itself, such a
script would wait for itself. A script that does not exit keeps the
lease, so the operation that broke it waits as long, as it does while a
SCSI fence keeps failing, and unloading nfsd waits for it too. If the
work item cannot be allocated, the fence worker retries with the backoff
of a failed ->fence_client().

The recall failure path still runs the script in rpciod while the
callback is in flight, as the layouts are dropped right after it, and
tearing down the client waits for that callback in
nfsd4_shutdown_callback(). So a script that expires the client it fences
still hangs there, as it does without this patch: in a test with a lease
time of 15 seconds, where the recall reaches its time limit before the
lease break times out, the script stayed in its write to the ctl file,
in nfsd4_shutdown_callback(), and rpciod in call_usermodehelper_exec().
That path now builds the arguments of the script with the same helper,
and skips the fence when ls_fenced is set. The fence worker clears
ls_file when it is done, so nfsd_file_get() returned NULL there and the
failure path did not fence a second time before either; the check makes
that explicit. A recall that fails while the fence worker still waits
for its script runs the script a second time, as before.

This makes a lease break wait longer, and "nfsd: keep polling a layout
recall answered with NFS4ERR_OLD_STATEID" more so. Apply it only
together with "nfsd: answer NFS4ERR_DELAY instead of waiting for a
layout recall", earlier in this series, which keeps nfsd threads out of
that wait.

Tested together with "nfsd: keep polling a layout recall answered with
NFS4ERR_OLD_STATEID": a client writing through a block layout whose path
to the device is cut for 70 seconds while it can still talk to the
server, default lease and lease break times, and a 4 KiB write to the
file on the server. Without this patch (the time limit and that patch on
nfsd-testing 32eb1a60b456) the write on the server goes on after 47
seconds and /sbin/nfsd-recall-failed is not run. With it the script is
run after 47 seconds and the write goes on then. In both runs the client
returns the layout when its path is back, and the recall completes. With
a script that expires the client it fences through its ctl file, the
script was done 36 ms after it started, and the write on the server went
on 46 seconds after it started.

Fixes: f52792f484ba ("NFSD: Enforce timeout on layout recall and integrate lease manager fencing")
Signed-off-by: Daejun Park <daejun7.park@samsung.com>
---
 fs/nfsd/nfs4layouts.c | 171 +++++++++++++++++++++++++++++++++++++++++++++-----
 fs/nfsd/state.h       |   5 ++
 2 files changed, 159 insertions(+), 17 deletions(-)

diff --git a/fs/nfsd/nfs4layouts.c b/fs/nfsd/nfs4layouts.c
index ab7340343a..f3bed0fa98 100644
--- a/fs/nfsd/nfs4layouts.c
+++ b/fs/nfsd/nfs4layouts.c
@@ -24,6 +24,26 @@ struct nfs4_layout {
 
 static struct kmem_cache *nfs4_layout_cache;
 static struct kmem_cache *nfs4_layout_stateid_cache;
+static struct workqueue_struct *nfsd4_fence_wq;
+
+/*
+ * What the fence script is run with: copies, as tearing down the client
+ * frees the layout stateid and the client
+ */
+struct nfsd4_fence_args {
+	char			addr[INET6_ADDRSTRLEN];
+	char			s_id[sizeof_field(struct super_block, s_id)];
+};
+
+/* A fence script that the fence worker runs and waits for */
+struct nfsd4_fence_script {
+	struct work_struct	work;
+	refcount_t		ref;
+	bool			done;
+	struct nfsd4_fence_args	args;
+};
+
+#define NFSD4_FENCE_SCRIPT_POLL	(HZ / 10)
 
 static const struct nfsd4_callback_ops nfsd4_cb_layout_ops;
 static const struct lease_manager_operations nfsd4_layouts_lm_ops;
@@ -767,10 +787,8 @@ nfsd4_return_all_file_layouts(struct nfs4_client *clp, struct nfs4_file *fp)
 }
 
 static void
-nfsd4_cb_layout_fail(struct nfs4_layout_stateid *ls, struct nfsd_file *file)
+nfsd4_run_fence_script(char *addr_str, char *s_id)
 {
-	struct nfs4_client *clp = ls->ls_stid.sc_client;
-	char addr_str[INET6_ADDRSTRLEN];
 	static char const nfsd_recall_failed[] = "/sbin/nfsd-recall-failed";
 	static char *envp[] = {
 		"HOME=/",
@@ -781,15 +799,13 @@ nfsd4_cb_layout_fail(struct nfs4_layout_stateid *ls, struct nfsd_file *file)
 	char *argv[8];
 	int error;
 
-	rpc_ntop((struct sockaddr *)&clp->cl_addr, addr_str, sizeof(addr_str));
-
 	printk(KERN_WARNING
 		"nfsd: client %s failed to respond to layout recall. "
 		"  Fencing..\n", addr_str);
 
 	argv[0] = (char *)nfsd_recall_failed;
 	argv[1] = addr_str;
-	argv[2] = file->nf_file->f_path.mnt->mnt_sb->s_id;
+	argv[2] = s_id;
 	argv[3] = NULL;
 
 	error = call_usermodehelper(nfsd_recall_failed, argv, envp,
@@ -800,6 +816,65 @@ nfsd4_cb_layout_fail(struct nfs4_layout_stateid *ls, struct nfsd_file *file)
 	}
 }
 
+static void
+nfsd4_fence_args_init(struct nfsd4_fence_args *args,
+		      struct nfs4_layout_stateid *ls, struct nfsd_file *nf)
+{
+	struct nfs4_client *clp = ls->ls_stid.sc_client;
+
+	rpc_ntop((struct sockaddr *)&clp->cl_addr, args->addr,
+		 sizeof(args->addr));
+	strscpy(args->s_id, nf->nf_file->f_path.mnt->mnt_sb->s_id);
+}
+
+/* Runs the script synchronously: the layouts are dropped right after it */
+static void
+nfsd4_cb_layout_fail(struct nfs4_layout_stateid *ls, struct nfsd_file *file)
+{
+	struct nfsd4_fence_args args;
+
+	nfsd4_fence_args_init(&args, ls, file);
+	nfsd4_run_fence_script(args.addr, args.s_id);
+}
+
+static void nfsd4_fence_script_put(struct nfsd4_fence_script *fs)
+{
+	if (refcount_dec_and_test(&fs->ref))
+		kfree(fs);
+}
+
+static void nfsd4_fence_script_work(struct work_struct *work)
+{
+	struct nfsd4_fence_script *fs =
+		container_of(work, struct nfsd4_fence_script, work);
+
+	nfsd4_run_fence_script(fs->args.addr, fs->args.s_id);
+	/* Pairs with the smp_load_acquire() in nfsd4_layout_fence_worker() */
+	smp_store_release(&fs->done, true);
+	nfsd4_fence_script_put(fs);
+}
+
+/*
+ * Run the fence script on nfsd4_fence_wq rather than in the fence worker,
+ * which only checks whether it is done. Tearing down the client waits for
+ * the fence worker, and the script may be what tears the client down,
+ * through /proc/fs/nfsd/clients/<clid>/ctl.
+ */
+static struct nfsd4_fence_script *
+nfsd4_start_fence_script(struct nfs4_layout_stateid *ls, struct nfsd_file *nf)
+{
+	struct nfsd4_fence_script *fs;
+
+	fs = kzalloc_obj(*fs);
+	if (!fs)
+		return NULL;
+	refcount_set(&fs->ref, 2);
+	nfsd4_fence_args_init(&fs->args, ls, nf);
+	INIT_WORK(&fs->work, nfsd4_fence_script_work);
+	queue_work(nfsd4_fence_wq, &fs->work);
+	return fs;
+}
+
 static bool
 nfsd4_cb_layout_prepare(struct nfsd4_callback *cb)
 {
@@ -812,6 +887,16 @@ nfsd4_cb_layout_prepare(struct nfsd4_callback *cb)
 	return true;
 }
 
+static bool nfsd4_layout_fenced(struct nfs4_layout_stateid *ls)
+{
+	bool fenced;
+
+	spin_lock(&ls->ls_lock);
+	fenced = ls->ls_fenced;
+	spin_unlock(&ls->ls_lock);
+	return fenced;
+}
+
 static int
 nfsd4_cb_layout_done(struct nfsd4_callback *cb, struct rpc_task *task)
 {
@@ -858,6 +943,9 @@ nfsd4_cb_layout_done(struct nfsd4_callback *cb, struct rpc_task *task)
 		 * Unknown error or non-responding client, we'll need to fence.
 		 */
 		trace_nfsd_layout_recall_fail(&ls->ls_stid.sc_stateid);
+		/* The fence worker of a lease break is done */
+		if (nfsd4_layout_fenced(ls))
+			return 1;
 		rcu_read_lock();
 		fl = nfsd_file_get(ls->ls_file);
 		rcu_read_unlock();
@@ -937,6 +1025,13 @@ nfsd4_layout_lm_open_conflict(struct file *filp, int arg)
 
 static void nfsd4_layout_fence_done(struct nfs4_layout_stateid *ls)
 {
+	struct nfsd4_fence_script *fs = ls->ls_fence_script;
+
+	/* The script runs on, but nothing waits for it any more. */
+	ls->ls_fence_script = NULL;
+	if (fs)
+		nfsd4_fence_script_put(fs);
+
 	/* Unlock the lease so that tasks waiting on it can proceed. */
 	nfsd4_close_layout(ls);
 
@@ -975,11 +1070,23 @@ void nfsd4_stop_layout_fence(struct nfs4_layout_stateid *ls)
 	mutex_unlock(&ls->ls_fence_mutex);
 }
 
+static unsigned int nfsd4_fence_backoff(struct nfs4_layout_stateid *ls)
+{
+	lockdep_assert_held(&ls->ls_lock);
+	if (!ls->ls_fence_delay)
+		ls->ls_fence_delay = HZ;
+	else
+		ls->ls_fence_delay = min(ls->ls_fence_delay << 1,
+					 MAX_FENCE_DELAY);
+	return ls->ls_fence_delay;
+}
+
 static void nfsd4_layout_fence_worker(struct work_struct *work)
 {
 	struct delayed_work *dwork = to_delayed_work(work);
 	struct nfs4_layout_stateid *ls = container_of(dwork,
 			struct nfs4_layout_stateid, ls_fence_work);
+	const struct nfsd4_layout_ops *ops;
 	struct nfsd_file *nf;
 	struct block_device *bdev;
 	struct nfs4_client *clp;
@@ -1003,7 +1110,35 @@ static void nfsd4_layout_fence_worker(struct work_struct *work)
 	clp = ls->ls_stid.sc_client;
 	nn = net_generic(clp->net, nfsd_net_id);
 	bdev = nf->nf_file->f_path.mnt->mnt_sb->s_bdev;
-	if (nfsd4_layout_ops[ls->ls_layout_type]->fence_client(ls, nf)) {
+	ops = nfsd4_layout_ops[ls->ls_layout_type];
+	if (!ops->fence_client) {
+		struct nfsd4_fence_script *fs = ls->ls_fence_script;
+
+		/*
+		 * The layout type cannot fence. Run the fence script, as a
+		 * failed recall does, and give the lease up once it is done.
+		 */
+		if (!fs) {
+			fs = nfsd4_start_fence_script(ls, nf);
+			ls->ls_fence_script = fs;
+		}
+		nfsd_file_put(nf);
+		/* Pairs with nfsd4_fence_script_work() */
+		if (fs && smp_load_acquire(&fs->done))
+			goto dispose;
+		spin_lock(&ls->ls_lock);
+		if (ls->ls_fence_stopped) {
+			spin_unlock(&ls->ls_lock);
+			goto dispose;
+		}
+		/* Retry a failed allocation as a failed fence is retried */
+		mod_delayed_work(system_dfl_wq, &ls->ls_fence_work,
+				 fs ? NFSD4_FENCE_SCRIPT_POLL :
+				      nfsd4_fence_backoff(ls));
+		spin_unlock(&ls->ls_lock);
+		return;
+	}
+	if (ops->fence_client(ls, nf)) {
 		/* fenced ok */
 		nfsd_file_put(nf);
 		pr_warn("%s: FENCED client[%pISpc] clid[%d] to device[%s]\n",
@@ -1041,13 +1176,8 @@ static void nfsd4_layout_fence_worker(struct work_struct *work)
 		spin_unlock(&ls->ls_lock);
 		goto dispose;
 	}
-	if (!ls->ls_fence_delay)
-		ls->ls_fence_delay = HZ;
-	else
-		ls->ls_fence_delay = min(ls->ls_fence_delay << 1,
-					 MAX_FENCE_DELAY);
 	mod_delayed_work(system_dfl_wq, &ls->ls_fence_work,
-			 ls->ls_fence_delay);
+			 nfsd4_fence_backoff(ls));
 	spin_unlock(&ls->ls_lock);
 }
 
@@ -1055,8 +1185,9 @@ static void nfsd4_layout_fence_worker(struct work_struct *work)
  * nfsd4_layout_lm_breaker_timedout - The layout recall has timed out.
  * @fl: file to check
  *
- * If the layout type supports a fence operation, schedule a worker to
- * fence the client from accessing the block device.
+ * Schedule a worker to fence the client from accessing the block device.
+ * If the layout type has no fence operation, the worker runs the fence
+ * script instead.
  *
  * This function runs under the protection of the spin_lock flc_lock.
  * At this time, the file_lease associated with the layout stateid is
@@ -1078,8 +1209,6 @@ nfsd4_layout_lm_breaker_timedout(struct file_lease *fl)
 {
 	struct nfs4_layout_stateid *ls = fl->c.flc_owner;
 
-	if (!nfsd4_layout_ops[ls->ls_layout_type]->fence_client)
-		return true;
 	/*
 	 * Make sure layout has not been returned yet before
 	 * taking a reference count on the layout stateid. The
@@ -1133,6 +1262,13 @@ nfsd4_init_pnfs(void)
 		kmem_cache_destroy(nfs4_layout_cache);
 		return -ENOMEM;
 	}
+
+	nfsd4_fence_wq = alloc_workqueue("nfsd4_fence", WQ_UNBOUND, 0);
+	if (!nfsd4_fence_wq) {
+		kmem_cache_destroy(nfs4_layout_stateid_cache);
+		kmem_cache_destroy(nfs4_layout_cache);
+		return -ENOMEM;
+	}
 	return 0;
 }
 
@@ -1141,6 +1277,7 @@ nfsd4_exit_pnfs(void)
 {
 	int i;
 
+	destroy_workqueue(nfsd4_fence_wq);
 	kmem_cache_destroy(nfs4_layout_cache);
 	kmem_cache_destroy(nfs4_layout_stateid_cache);
 
diff --git a/fs/nfsd/state.h b/fs/nfsd/state.h
index f05e256421..23b2eb3ff1 100644
--- a/fs/nfsd/state.h
+++ b/fs/nfsd/state.h
@@ -843,6 +843,8 @@ static inline struct nfs4_ol_stateid *openlockstateid(struct nfs4_stid *s)
 	return container_of(s, struct nfs4_ol_stateid, st_stid);
 }
 
+struct nfsd4_fence_script;
+
 struct nfs4_layout_stateid {
 	struct nfs4_stid		ls_stid;
 	struct list_head		ls_perclnt;
@@ -860,9 +862,12 @@ struct nfs4_layout_stateid {
 	struct delayed_work		ls_fence_work;
 	struct mutex			ls_fence_mutex; /* serializes fence shutdown */
 	unsigned int			ls_fence_delay;
+	/* The fence worker of a lease break is done */
 	bool				ls_fenced;
 	bool				ls_fence_inflight;
 	bool				ls_fence_stopped;
+	/* The fence script the fence worker waits for */
+	struct nfsd4_fence_script	*ls_fence_script;
 };
 
 static inline struct nfs4_layout_stateid *layoutstateid(struct nfs4_stid *s)

-- 
2.43.0

[PATCH v3 5/7] nfsd: end a layout recall once a lease break has fenced the client

[Daejun Park via B4 Relay]

From: Daejun Park <daejun7.park@samsung.com>

When a lease break on a layout times out,
nfsd4_layout_lm_breaker_timedout() has the fence worker fence the client
that holds the layout, with ->fence_client() for the SCSI layout since
the commit this fixes, and with the fence script for the block layout
since "nfsd: run the fence script when a block layout lease break times
out". The lease is then removed and the operation that broke it goes on.
The CB_LAYOUTRECALL of that layout stateid is not ended: a client whose
path to the device is cut keeps answering it with NFS4_OK or
NFS4ERR_DELAY (a Linux client before v6.8, and every Linux client with
"nfsd: keep polling a layout recall answered with NFS4ERR_OLD_STATEID",
later in this series), and nfsd polls it until the time limit of the
recall, two lease periods. Meanwhile fi_lo_recalls stays raised and
every LAYOUTGET for the file, from any client, gets
NFS4ERR_RECALLCONFLICT, for up to 135 seconds with the default times.
And a LAYOUTCOMMIT that the fenced client sends when its path is back is
applied after the operation that broke the lease went on; XFS then
converts whatever unwritten extents the file has in that range by then.

End the recall at the next poll once the fence worker is done, for both
layout types. nfsd4_cb_layout_release() then drops the layouts, as on
the recall failure path (RFC 8881 section 12.5.6). A LAYOUTCOMMIT that
the client sends after that gets NFS4ERR_BADLAYOUT, so what it wrote
through the layout and had not committed by then is lost; the Linux
client does not report that error. Setting fs.lease-break-time to at
least the lease time gives such a client the one lease period that RFC
8881 section 12.5.5 has the server wait after a recall.

ls_fenced is also set when the fence worker stopped without a fence: the
layout stateid is being torn down or was revoked, its lease is already
gone, or no layouts are left. Ending the recall is right in those cases
too: the caller drops the layouts, or the stateid has been revoked.

Tested together with "nfsd: keep polling a layout recall answered with
NFS4ERR_OLD_STATEID", as a current Linux client otherwise has the recall
fail after 10 ms, on nfsd-testing 78c334bc5ae6. "Without this patch" is
v2 of this series, which fences the same way and does not end the
recall. A client writing through a block layout whose path to the device
is cut for 210 seconds while it can still talk to the server, default
lease and lease break times, and a 4 KiB write to the file on the
server: the fence script ran once either way. Without this patch the
recall was polled until its time limit, 180 seconds after it started;
with it the recall ended 47.5 seconds after it started, right after the
script. With the SCSI layout the fence worker fenced the client either
way, and the recall ended 10 ms after the fence with this patch and at
the time limit without it.

With the path cut for 70 seconds, the recall ended 48 seconds after it
started with this patch, and the LAYOUTCOMMIT that the client sent when
its path was back got NFS4ERR_BADLAYOUT. Without it, the recall ended
when the client returned the layout, 71 seconds after it started, and
the LAYOUTCOMMIT was applied. The loss of what the client wrote and had
not committed is read from the code: the test overwrites a file written
beforehand, so no extent is left unwritten.

Fixes: f52792f484ba ("NFSD: Enforce timeout on layout recall and integrate lease manager fencing")
Signed-off-by: Daejun Park <daejun7.park@samsung.com>
---
 fs/nfsd/nfs4layouts.c | 9 +++++++++
 1 file changed, 9 insertions(+)

diff --git a/fs/nfsd/nfs4layouts.c b/fs/nfsd/nfs4layouts.c
index f3bed0fa98..574d16cbbd 100644
--- a/fs/nfsd/nfs4layouts.c
+++ b/fs/nfsd/nfs4layouts.c
@@ -918,6 +918,15 @@ nfsd4_cb_layout_done(struct nfsd4_callback *cb, struct rpc_task *task)
 		 */
 		if (list_empty(&ls->ls_layouts))
 			return 1;
+		/*
+		 * The fence worker of a lease break is done: revoke the
+		 * layouts (RFC 8881 section 12.5.6), before one lease period
+		 * (section 12.5.5) when fs.lease-break-time is shorter.
+		 */
+		if (nfsd4_layout_fenced(ls)) {
+			trace_nfsd_layout_recall_fail(&ls->ls_stid.sc_stateid);
+			return 1;
+		}
 
 		/* Poll the client until it's done with the layout */
 		now = ktime_get_boottime();

-- 
2.43.0

[PATCH v3 6/7] nfsd: keep polling a layout recall answered with NFS4ERR_OLD_STATEID

[Daejun Park via B4 Relay]

From: Daejun Park <daejun7.park@samsung.com>

When a client answers CB_LAYOUTRECALL with NFS4_OK and still holds
layouts, nfsd4_cb_layout_done() sends the callback again after 10 ms to
see if the client is done. The repeat carries the stateid of the first
callback, although RFC 8881 section 12.5.3 has the server increment the
seqid in each CB_LAYOUTRECALL request.

The same section makes the seqid of a processed CB_LAYOUTRECALL the
current seqid of the client. Since commit dce72920c81b ("NFSv4.1: if
referring calls are complete, trust the stateid argument"), which is in
v6.8, the Linux client answers a recall whose seqid is not newer than
its own with NFS4ERR_OLD_STATEID, unless a LAYOUTRETURN is on the wire
(NFS4ERR_DELAY) or the layouts are gone (NFS4ERR_NOMATCHING_LAYOUT).
Before that commit it answered a repeat of the recall it was working
on with NFS4_OK until its LAYOUTRETURN was prepared, and with
NFS4ERR_OLD_STATEID from then until nfsd had processed the LAYOUTRETURN
(read from the code, not run). So a client from v6.8 on that needs
more than 10 ms to finish its I/O and to send LAYOUTRETURN answers the
second callback with NFS4ERR_OLD_STATEID, and an older client can
answer a later one that way.

nfsd4_cb_layout_done() takes that as a failed recall. With the block
layout it logs "client ... failed to respond to layout recall.
Fencing.." and runs /sbin/nfsd-recall-failed, with the SCSI layout it
fences the client with a persistent reservation preempt, and in both
cases nfsd4_cb_layout_release() drops the layouts. Section 12.5.5
requires the server to wait one lease period after a recall before it
takes further action. Here the client is still using the layout: its
I/O can be in flight when the conflicting operation goes on, and what
it has written to newly allocated blocks stays unwritten on the server,
because the layout is gone when the client wants to commit it.

Clients doing O_DIRECT I/O in 1 MiB requests on one file for 30 seconds,
block layout on XFS. "after" is the polling of this patch, without the
new seqid described below, with "nfsd: make the time limit of a layout
recall work", on nfsd-testing 32eb1a60b456. The same runs with the new
seqid and the rest of this series gave no failed recall and no lost
block either.

Recalls that ended in the failure path, out of the recalls sent:

                                                   before      after
  2 clients read (different halves of the file)    7 / 9       0 / 9
  2 clients write                                  8 / 9       0 / 9
  3 clients write                                  19 / 20     0 / 20
  2 clients write, the server writes 4 KiB to
  the file every 50 ms                             216 / 229   0 / 250

Blocks that read back as zeroes on the server, out of the 1 MiB blocks
that the clients wrote to a new file with O_DIRECT, followed by one
successful fsync():

                                                   before      after
  2 clients write                                  6 / 564     0 / 481
  3 clients write                                  11 / 527    0 / 569
  1 client writes, the server sets the file size
  every second                                     21 / 469    0 / 504
  2 clients write, file preallocated               8 / 601     0 / 602
  1 client writes, 1 client reads                  4 / 493     0 / 508

One client writing through a block layout whose path to the device is
cut for 15 seconds while it can still talk to the server, and a 4 KiB
write to the file on the server:

                                                   before      after
  the write on the server takes                    0.05 s      15.6 s
  the recall                                       fails       completes
  /sbin/nfsd-recall-failed is run                  yes         no

Before, the write of the client that was in flight when the layout was
recalled was stuck for 18.5 seconds and no WRITE was sent to the
server, so it reached the device about 15 seconds after the write on
the server had returned. After, the write on the server waits for it.

Two clients writing one file with the SCSI layout on an nvmet-tcp
namespace: before, both recalls sent ended in nfsd4_scsi_fence_client(),
the number of registrants on the namespace went from 4 to 2, and both
clients logged reservation conflicts and did the rest of the run
through the server. The same with a new file, after a fresh boot: both
clients were fenced, and one of the 630 blocks they had written read
back as zeroes. After, no client is fenced and no block is lost.

Handle NFS4ERR_OLD_STATEID like NFS4ERR_DELAY: the client has seen the
recall, so keep polling until the layouts are returned, the time limit
is reached, or a lease break has had the client fenced. Apply this only
together with the earlier patches in this series: the limit only works
with "nfsd: make the time limit of a layout recall work", with "nfsd:
run the fence script when a block layout lease break times out" a block
layout client is fenced when the lease break gives up, with "nfsd: end a
layout recall once a lease break has fenced the client" its recall ends
then, and "nfsd: answer NFS4ERR_DELAY instead of waiting for a layout
recall" keeps nfsd threads out of the longer wait. Kernels before v7.1
do not have the lease break timeout that the second and third of them
fix. There, an operation on the server that does not come from nfsd, and
an nfsd thread that still ends up in the filesystem's break, waits until
the recall ends, for at most two lease periods. Such a kernel already
waits that way, without a limit, for a client that keeps answering
NFS4ERR_DELAY, and RFC 8881 section 12.5.5 asks the server to wait at
least one lease period before it fences.

The poll sends the callback every 10 ms for as long as the client
holds the layout, as it already does for a client that answers
NFS4ERR_DELAY. With a client that needs long to return the layout
that is about 100 callbacks a second, for up to two lease periods.

A client can also answer NFS4ERR_OLD_STATEID without having taken the
recall up. A LAYOUTRETURN that returns some layouts and leaves others
gets a new seqid; if nfsd processes it after ->prepare, the client takes
in a seqid above the recall's from its reply. Such a client answers
every poll with NFS4ERR_OLD_STATEID. Polling alone would keep LAYOUTGETs
of other clients failing with NFS4ERR_RECALLCONFLICT for up to two lease
periods and then fence the client, although it was never asked for the
layouts it still holds. RFC 8881 section 12.5.3 has a new seqid for each
CB_LAYOUTRECALL request, so when a client that has not answered NFS4_OK
answers NFS4ERR_OLD_STATEID, give the next poll a new seqid, which the
client takes up. ->prepare clears ls_recall_accepted, and ->done sets it
when the client answers NFS4_OK. The new seqid is traced as
nfsd_layout_recall_new_seqid.

->done runs in rpciod and does not take ls_mutex, which could block it;
nfs4_inc_and_copy_stateid() takes sc_lock. So
nfsd4_preprocess_layout_stateid(), which checks under ls_mutex that the
seqid a client presents is not above the current one, now reads the
current seqid under sc_lock as well. While a recall is out, only a
LAYOUTRETURN changes the seqid besides, since "nfsd: fail a LAYOUTGET in
progress when the layout lease is broken" keeps LAYOUTGET from adding
layouts to a recalled layout stateid.

A pynfs client that holds a layout answers a recall whose seqid is not
above the first seqid of the recall with NFS4ERR_OLD_STATEID, and
returns the layout on a later one. With polling alone, nfsd sent the
recall 501 times in six seconds, all with the first seqid, and a
LAYOUTGET of another client still got NFS4ERR_RECALLCONFLICT. With the
new seqid, the second recall carried it, 10 ms later, the client
returned the layout, and the LAYOUTGET succeeded.

A client that has answered NFS4_OK is still polled with the seqid of the
recall. Sending each repeat with a new seqid, or ending the callback on
NFS4_OK and letting the LAYOUTRETURN finish the recall, would follow the
RFC; both are larger changes than a fix for stable kernels should be.

Fixes: c5c707f96fc9 ("nfsd: implement pNFS layout recalls")
Fixes: 6b9b21073d3b ("nfsd: give up on CB_LAYOUTRECALLs after two lease periods")
Signed-off-by: Daejun Park <daejun7.park@samsung.com>
Reviewed-by: Jeff Layton <jlayton@kernel.org>
---
 fs/nfsd/nfs4layouts.c | 33 +++++++++++++++++++++++++++++++--
 fs/nfsd/state.h       |  2 ++
 fs/nfsd/trace.h       |  1 +
 3 files changed, 34 insertions(+), 2 deletions(-)

diff --git a/fs/nfsd/nfs4layouts.c b/fs/nfsd/nfs4layouts.c
index 574d16cbbd..a6934fc21a 100644
--- a/fs/nfsd/nfs4layouts.c
+++ b/fs/nfsd/nfs4layouts.c
@@ -366,6 +366,7 @@ nfsd4_preprocess_layout_stateid(struct svc_rqst *rqstp,
 	struct nfs4_stid *stid;
 	unsigned short typemask = SC_TYPE_LAYOUT;
 	__be32 status;
+	bool after;
 
 	if (create)
 		typemask |= (SC_TYPE_OPEN | SC_TYPE_LOCK | SC_TYPE_DELEG);
@@ -395,7 +396,15 @@ nfsd4_preprocess_layout_stateid(struct svc_rqst *rqstp,
 
 		status = nfserr_bad_stateid;
 		mutex_lock(&ls->ls_mutex);
-		if (nfsd4_stateid_generation_after(stateid, &stid->sc_stateid))
+		/*
+		 * The recall callback can move the seqid on without
+		 * ls_mutex, under sc_lock.
+		 */
+		spin_lock(&stid->sc_lock);
+		after = nfsd4_stateid_generation_after(stateid,
+						       &stid->sc_stateid);
+		spin_unlock(&stid->sc_lock);
+		if (after)
 			goto out_unlock_stid;
 		if (layout_type != ls->ls_layout_type)
 			goto out_unlock_stid;
@@ -884,6 +893,7 @@ nfsd4_cb_layout_prepare(struct nfsd4_callback *cb)
 	mutex_lock(&ls->ls_mutex);
 	nfs4_inc_and_copy_stateid(&ls->ls_recall_sid, &ls->ls_stid);
 	mutex_unlock(&ls->ls_mutex);
+	ls->ls_recall_accepted = false;
 	return true;
 }
 
@@ -909,9 +919,28 @@ nfsd4_cb_layout_done(struct nfsd4_callback *cb, struct rpc_task *task)
 
 	trace_nfsd_cb_layout_done(&ls->ls_stid.sc_stateid, task);
 	switch (task->tk_status) {
-	case 0:
+	case -NFS4ERR_OLD_STATEID:
+		/*
+		 * A client that has not accepted this recall is past its
+		 * seqid, e.g. after a LAYOUTRETURN: poll it with a new seqid
+		 * (RFC 8881 section 12.5.3). No ls_mutex here, it could block
+		 * rpciod; sc_lock orders the seqid.
+		 */
+		if (!ls->ls_recall_accepted) {
+			nfs4_inc_and_copy_stateid(&ls->ls_recall_sid,
+						  &ls->ls_stid);
+			trace_nfsd_layout_recall_new_seqid(&ls->ls_recall_sid);
+		}
+		fallthrough;
 	case -NFS4ERR_DELAY:
+	case 0:
+		if (!task->tk_status)
+			ls->ls_recall_accepted = true;
 		/*
+		 * The poll repeats the last seqid, against RFC 8881 section
+		 * 12.5.3; a client returning its layouts answers it with
+		 * NFS4ERR_OLD_STATEID.
+		 *
 		 * Anything left? If not, then call it done. Note that we don't
 		 * take the spinlock since this is an optimization and nothing
 		 * should get added until the cb counter goes to zero.
diff --git a/fs/nfsd/state.h b/fs/nfsd/state.h
index 23b2eb3ff1..f001edfad9 100644
--- a/fs/nfsd/state.h
+++ b/fs/nfsd/state.h
@@ -856,6 +856,8 @@ struct nfs4_layout_stateid {
 	struct nfsd4_callback		ls_recall;
 	stateid_t			ls_recall_sid;
 	ktime_t				ls_recall_start;
+	/* The client has answered NFS4_OK to the recall */
+	bool				ls_recall_accepted;
 	bool				ls_recalled;
 	struct mutex			ls_mutex;
 
diff --git a/fs/nfsd/trace.h b/fs/nfsd/trace.h
index abfdbc637c..e0908cb21c 100644
--- a/fs/nfsd/trace.h
+++ b/fs/nfsd/trace.h
@@ -681,6 +681,7 @@ DEFINE_STATEID_EVENT(layout_return_lookup_fail);
 DEFINE_STATEID_EVENT(layout_recall);
 DEFINE_STATEID_EVENT(layout_recall_empty);
 DEFINE_STATEID_EVENT(layout_recall_done);
+DEFINE_STATEID_EVENT(layout_recall_new_seqid);
 DEFINE_STATEID_EVENT(layout_recall_fail);
 DEFINE_STATEID_EVENT(layout_recall_release);
 

-- 
2.43.0

[PATCH v3 7/7] nfsd: do not fence a client for a layout recall with no layouts left

[Daejun Park via B4 Relay]

From: Daejun Park <daejun7.park@samsung.com>

nfsd4_cb_layout_done() fences the client for any answer to a
CB_LAYOUTRECALL that it does not handle, without looking at whether the
layout stateid still has layouts. A LAYOUTRETURN can return the last
layout while the recall is still out. The Linux client answers
NFS4ERR_NOMATCHING_LAYOUT then, which RFC 8881 section 20.3.3 allows
for a client that holds no layouts for the file. A client that answers
NFS4ERR_BAD_STATEID is right too, since section 12.5.3 makes a layout
stateid invalid once the client has no layouts for the file, but it is
fenced for nothing.

Check for layouts before fencing, as the polls already do before their
time limit, and end the recall when none are left. This does not depend
on the earlier patches in this series.

A pynfs client that returns its layout on the recall and then answers
it NFS4ERR_BAD_STATEID: without this patch, nfsd logged "failed to
respond to layout recall" and ran the fence for it; with it, the recall
ended without either.

Fixes: c5c707f96fc9 ("nfsd: implement pNFS layout recalls")
Signed-off-by: Daejun Park <daejun7.park@samsung.com>
Reviewed-by: Jeff Layton <jlayton@kernel.org>
---
 fs/nfsd/nfs4layouts.c | 9 +++++++++
 1 file changed, 9 insertions(+)

diff --git a/fs/nfsd/nfs4layouts.c b/fs/nfsd/nfs4layouts.c
index a6934fc21a..e0eb668ca9 100644
--- a/fs/nfsd/nfs4layouts.c
+++ b/fs/nfsd/nfs4layouts.c
@@ -977,6 +977,15 @@ nfsd4_cb_layout_done(struct nfsd4_callback *cb, struct rpc_task *task)
 		}
 		fallthrough;
 	default:
+		/*
+		 * Nothing to fence for once the layouts are gone: a
+		 * LAYOUTRETURN can return the last one while the recall is
+		 * still out.
+		 */
+		if (list_empty(&ls->ls_layouts)) {
+			trace_nfsd_layout_recall_done(&ls->ls_stid.sc_stateid);
+			return 1;
+		}
 		/*
 		 * Unknown error or non-responding client, we'll need to fence.
 		 */

-- 
2.43.0

Re: [PATCH v3 0/7] nfsd: layout recall and lease break fixes

[Benjamin Coddington]

On 15 Oct 2026, at 0:15, Daejun Park via B4 Relay wrote:

> nfsd polls a client that has accepted a CB_LAYOUTRECALL by sending the
> callback again every 10 ms, with the stateid of the first callback. A
> Linux client from v6.8 on answers the repeat with NFS4ERR_OLD_STATEID,
> and nfsd takes that as a failed recall. So a client that needs more than
> 10 ms to return a layout has the layout revoked and the fence path run.
> With the block layout that client loses data it has written and synced,
> with the SCSI layout it is cut off from the device.

[...]

Patch 5's point that the Linux client doesn't report NFS4ERR_BADLAYOUT
on LAYOUTCOMMIT - nfs4_layoutcommit_done() throws it away, so fsync()
returns 0 while the block layout writes into unwritten extents are gone.  I
think thats reachable today through the existing recall failure path, so
it would be a client bug - I'm going to see if I can fix that.

Otherwise, this looks good to me.  For the series:

Reviewed-by: Benjamin Coddington <bcodding@hammerspace.com>

Ben