[REGRESSION] 7.2.4: nfsd sends CB_OFFLOAD with all-zero stateid, async NFSv4.2 COPY never completes

[REGRESSION] 7.2.4: nfsd sends CB_OFFLOAD with all-zero stateid, async NFSv4.2 COPY never completes

[Lubos Rendek]

Since 7.2.4, an NFSv4.2 server-side COPY between two exports of a Linux
NFS server never completes, because nfsd sends CB_OFFLOAD with an
all-zero stateid.

The client sends COPY, the server answers with a copy stateid, does the
copy in the background and then reports completion in CB_OFFLOAD, but
with a zeroed stateid. The client cannot match it to its pending COPY,
so cp(1) waits in nfs42_proc_copy until killed, while the data is
complete on the server. Nothing is logged on the server. This is a
stable-only regression: mainline is fine.

Details:

Tested with kernel.org kernels, all built with the same config, no
out-of-tree modules, untainted:

  7.2.3      works  (3 of 3)
  7.2.4      hangs  (3 of 3)
  7.2.9      hangs  (2 of 2)
  6.18.55    hangs
  7.3-rc6    works  (mainline)
  6.12.112   copies synchronously here (128 x 4 MiB COPY, no
             CB_OFFLOAD), so this reproducer does not reach the
             async path

Stateid in the COPY reply vs. in CB_OFFLOAD, from captures on the
server:

  kernel   COPY reply                  CB_OFFLOAD
  7.2.3    1 3456c76a717bdc2100000000  1 3456c76a717bdc2100000000
  7.2.4    1 ef54c76a2dcf760401000000  0 000000000000000000000000
  7.3-rc6  1 13efc76a9308445405000000  1 13efc76a9308445405000000
  6.18.55  1 1011c86a7674335600000000  0 000000000000000000000000

Cause, from reading the source: in 7.2.4, nfsd4_copy() duplicates the
copy into async_copy before the copy stateid exists:

    dup_copy_fields(copy, async_copy);

    if (!nfs4_init_copy_state(nn, async_copy))
        goto out_dec_async_copy_err;
    memcpy(&result->cb_stateid, &async_copy->cp_stateid.cs_stid,
        sizeof(result->cb_stateid));

So the COPY reply gets the stateid, but async_copy->cp_res.cb_stateid
stays zero, and nfsd4_send_cb_offload() builds the callback from
async_copy->cp_res. 7.2.3 set up the stateid before dup_copy_fields().
Mainline writes cb_stateid before duplicating, with the comment "dup
after writing cb_stateid; duplicating first would leave the callback
stateid zeroed."

7.2.4 took six patches of the "nfsd: copy offload fixes" series, but
not its restructuring patches. The reordering most likely came with:

  14b978e8d05c ("nfsd: fix stale s2s_cp_stateids IDR entry for async COPY")

which is d0beaee498e1 upstream. Mainline gets the right order with
f307b4f7ed17 ("nfsd: split nfsd4_copy into transient and durable async
copy objects"), which is not in any 7.2.y release. Not bisected. The
same patches are in 6.18.50/6.18.51 (6.18.55 tested bad) and
6.12.109/6.12.111.

For confirmation, this minimal change on top of 7.2.4 makes the
reproducer work here: the COPY stays async, and CB_OFFLOAD carries the
same stateid as the COPY reply. It is only a lab test, not a proposed
fix:

--- a/fs/nfsd/nfs4proc.c
+++ b/fs/nfsd/nfs4proc.c
@@ -2238,6 +2238,8 @@
 			goto out_dec_async_copy_err;
 		memcpy(&result->cb_stateid, &async_copy->cp_stateid.cs_stid,
 			sizeof(result->cb_stateid));
+		memcpy(&async_copy->cp_res.cb_stateid, &async_copy->cp_stateid.cs_stid,
+		       sizeof(async_copy->cp_res.cb_stateid));
 		if ((READ_ONCE(copy->nf_dst->nf_file->f_mode) &
 			       FMODE_NOCMTIME) != 0)
 			async_copy->attr_update = true;

Reproducer:

  Server: two exports on two separate ext4 filesystems (also seen with
  two ZFS datasets on a distribution kernel):

    /srv/e1  <client>(rw,sync,no_subtree_check)
    /srv/e2  <client>(rw,sync,no_subtree_check)

  Client:

    mount -t nfs -o vers=4 <server>:/srv/e1 /mnt/share1
    mount -t nfs -o vers=4 <server>:/srv/e2 /mnt/share2
    dd if=/dev/urandom of=/mnt/share1/big.bin bs=1M count=512
    cp /mnt/share1/big.bin /mnt/share2/

  cp blocks in nfs42_proc_copy, /mnt/share2/big.bin stays at 0 bytes
  on the client. With vers=4.1 (no COPY) the copy works.

Environment:

  Server: x86_64 QEMU/KVM guest, Debian 13 (trixie) userspace
    Linux version 7.2.4-stock (gcc (Debian 14.2.0-19) 14.2.0,
    GNU ld (GNU Binutils for Debian) 2.44) #1 SMP PREEMPT_DYNAMIC
  Client: Debian 13, 6.12.107+deb13-amd64

  The 7.2.3 and 7.2.4 configs are identical apart from the version
  line; the same config gives a working kernel on 7.2.3 and 7.3-rc6.

  .config (7.2.4): https://bugzilla.proxmox.com/attachment.cgi?id=2161
  dmesg (7.2.4):   https://bugzilla.proxmox.com/attachment.cgi?id=2162
  captures:        https://bugzilla.proxmox.com/attachment.cgi?id=2160
  full history:    https://bugzilla.proxmox.com/show_bug.cgi?id=8121

First seen by a Proxmox VE user on its 7.0.14-20 kernel, which includes
these stable patches.

#regzbot introduced: v7.2.3..v7.2.4

Thanks,
Lubos Rendek

Re: [REGRESSION] 7.2.4: nfsd sends CB_OFFLOAD with all-zero stateid, async NFSv4.2 COPY never completes

[Greg KH]

On Sun, Oct 11, 2026 at 06:30:40AM +1100, Lubos Rendek wrote:
> Since 7.2.4, an NFSv4.2 server-side COPY between two exports of a Linux
> NFS server never completes, because nfsd sends CB_OFFLOAD with an
> all-zero stateid.

Can you just use git bisect to determine the offending commit?

And is this also the same for 7.3-rc?

thanks,

greg k-h

Re: [REGRESSION] 7.2.4: nfsd sends CB_OFFLOAD with all-zero stateid, async NFSv4.2 COPY never completes

[Jeff Layton]

On Sun, 2026-10-11 at 09:19 +0200, Greg KH wrote:
> On Sun, Oct 11, 2026 at 06:30:40AM +1100, Lubos Rendek wrote:
> > Since 7.2.4, an NFSv4.2 server-side COPY between two exports of a Linux
> > NFS server never completes, because nfsd sends CB_OFFLOAD with an
> > all-zero stateid.
> 
> Can you just use git bisect to determine the offending commit?
> 
> And is this also the same for 7.3-rc?
> 
> 

I took a look: I believe this is a bug in 14b978e8d05c ("nfsd: fix
stale s2s_cp_stateids IDR entry for async COPY"), which is d0beaee498e1
in mainline.

In v7.3, this bug gets fixed by f307b4f7ed17 ("nfsd: split nfsd4_copy
into transient and durable async copy objects"), which was part of the
refactor of the async COPY code.

We should probably just take Lubos' two-line fix as a separate one-off
patch for stable. Backporting the COPY code refactor is too risky IMO.

Clanker analysis follows if you're interested. I think it's largely
correct:

-----------------8<-----------------------

◆ Diagnosis confirmed: the report is correct, and this is a stable-only regression from an
  incomplete backport.

  Root cause is the order of dup_copy_fields() vs nfs4_init_copy_state() in nfsd4_copy():

  • result points at the transient &u->copy.cp_res.
  • dup_copy_fields() copies cp_res (including cb_stateid) from transient to async_copy.
  • nfsd4_send_cb_offload() builds CB_OFFLOAD from async_copy->cp_res.

  Pre-d0beaee498e1, mainline did init → write result->cb_stateid → dup_copy_fields(), so both the
  COPY reply and async_copy->cp_res.cb_stateid got the stateid. But the s2s_cp_stateids IDR stored a
  pointer to &u->copy.cp_stateid, which is transient COMPOUND buffer memory.

  d0beaee498e1 ("nfsd: fix stale s2s_cp_stateids IDR entry for async COPY", stable 14b978e8d05c)
  fixed the IDR lifetime by doing dup_copy_fields() first, then
  nfs4_init_copy_state(nn, async_copy), then writing only result->cb_stateid. That leaves
  async_copy->cp_res.cb_stateid zero, because the dup already happened. Result: COPY reply has a
  valid stateid, CB_OFFLOAD has all-zero, client can't match it in nfs42_proc_copy and hangs. This
  matches your packet table exactly.

  Mainline is fixed by f307b4f7ed17 ("nfsd: split nfsd4_copy into transient and durable async copy
  objects"), which moves the durable stateid out of the duplicated struct so it can safely do init →
  write cb_stateid → dup again. Current code in fs/nfsd/nfs4proc.c:2242 has the explicit comment
  "dup after writing cb_stateid; duplicating first would leave the callback stateid zeroed." I
  verified the callback path in fs/nfsd/nfs4proc.c:2104 copies from cp_copy.cp_res, and
  dup_copy_fields in fs/nfsd/nfs4proc.c:2051 copies cp_res.

  Your minimal stable fix is the right shape for stable. A plain reorder won't work on the old
  layout because dup_copy_fields() also copies cp_stateid, so init-then-dup would clobber
  async_copy->cp_stateid with the transient's zero value. Copying the new stateid into both
  result->cb_stateid and async_copy->cp_res.cb_stateid after init preserves the IDR fix and fixes
  the callback. The alternative is backporting the full restructuring series (f307b4f7ed17,
  80c376183411, etc.), which is heavier for stable.

  #regzbot introduced: v7.2.3..v7.2.4 looks correct. The 6.12 sync-copy observation is expected:
  that branch largely forces COPY synchronous, so it doesn't reach the async/CB_OFFLOAD path.

  I did not run the reproducer or inspect a 7.2.y tree locally; verification was by reading mainline
  and the d0beaee498e1 before/after versions in this checkout.

-- 
Jeff Layton <jlayton@kernel.org>

Re: [REGRESSION] 7.2.4: nfsd sends CB_OFFLOAD with all-zero stateid, async NFSv4.2 COPY never completes

[Sasha Levin]

> Since 7.2.4, an NFSv4.2 server-side COPY between two exports of a Linux
> NFS server never completes, because nfsd sends CB_OFFLOAD with an
> all-zero stateid.

Queued the fix for 7.2 and 6.18, thanks.

-- 
Thanks,
Sasha

Re: [REGRESSION] 7.2.4: nfsd sends CB_OFFLOAD with all-zero stateid, async NFSv4.2 COPY never completes

[Lubos Rendek]

On Sun, Oct 11, 2026 at 09:19 +0200, Greg KH wrote:
> Can you just use git bisect to determine the offending commit?
>
> And is this also the same for 7.3-rc?

7.3-rc6 works fine here, only 7.2.y and 6.18.y are affected.

I didn't bisect, but Jeff confirmed 14b978e8d05c as the culprit and
Sasha has queued the fix, so I think a bisect isn't needed anymore.
Let me know if you still want one.

Thanks everyone for the quick response.

Lubos

Re: [REGRESSION] 7.2.4: nfsd sends CB_OFFLOAD with all-zero stateid, async NFSv4.2 COPY never completes

[Fabian Grünbichler]

On October 11, 2026 6:48 pm, Sasha Levin wrote:
>> Since 7.2.4, an NFSv4.2 server-side COPY between two exports of a Linux
>> NFS server never completes, because nfsd sends CB_OFFLOAD with an
>> all-zero stateid.
> 
> Queued the fix for 7.2 and 6.18, thanks.

AFAICT (for 7.2.x) you queued a full pick of

f307b4f7ed17 nfsd: split nfsd4_copy into transient and durable async copy objects

in

0b2963340727 nfsd: split nfsd4_copy into transient and durable async copy objects

despite Jeff stating the following up-thread:

> We should probably just take Lubos' two-line fix as a separate one-off
> patch for stable. Backporting the COPY code refactor is too risky IMO.

Was this intentional, or an accident?

FWIW, the full refactor would include two more patches:

d6edc2a72513efce59eaf85acb204d2ddac71892 nfsd: drop dead COPY-vs-COPYNOTIFY type handling from s2s stateid IDR
80c376183411300b4c99977c66f3f84f64b38676 nfsd: make the copy offload stateid a first-class nfs4_stid

so this is kind of half-way between the discussed options..

Fabian

Re: [REGRESSION] 7.2.4: nfsd sends CB_OFFLOAD with all-zero stateid, async NFSv4.2 COPY never completes

[Sasha Levin]

> AFAICT (for 7.2.x) you queued a full pick of
>
> f307b4f7ed17 nfsd: split nfsd4_copy into transient and durable async copy objects
>
> in
>
> 0b2963340727 nfsd: split nfsd4_copy into transient and durable async copy objects
>
> despite Jeff stating the following up-thread:
>
>> We should probably just take Lubos' two-line fix as a separate one-off
>> patch for stable. Backporting the COPY code refactor is too risky IMO.
>
> Was this intentional, or an accident?

That was not what Jeff recommended, thanks for catching it. The full pick is
now dropped from the 7.2 and 6.18 queues.

Jeff, Chuck: could one of you send that fix as a proper signed-off patch for
7.2.y and 6.18.y?

-- 
Thanks,
Sasha