[PATCH 0/3] SUNRPC: keep NFS client connections consistent across hibernation

[Andrew Pam <andrew@sericyb.com.au>]

An NFS client that hibernates with a mount active can resume with the
mount hung, until the server's nfsd is restarted. The underlying
problem was described on linux-pm in 2011 ("hibernate hangs TCP") and
not fixed. This series fixes it for SUNRPC clients.

After hibernation takes its snapshot, the kernel thaws devices, the NIC
among them, to write the image. Freezable kernel work stays frozen until
the image is restored, but rpciod, xprtiod and the NFSv4 lease renewal
work are not freezable. So RPCs are still sent and answered while the
image is written, and TCP acknowledges data from the server, none of
which the restored image knows about. On resume the client's TCP
sequence numbers, its acknowledgement point and its NFSv4.1 session slot
sequence ids are all behind the server's. Depending on what happened in
that window, the client:

  - resends data that the server discards as a duplicate, so the
    connection carries nothing;
  - gets NFS4ERR_SEQ_MISORDERED on a slot indefinitely, while the state
    manager waits in nfs4_begin_drain_session() for that slot; or
  - waits for replies it acknowledged before the restore, which the
    server will not send again.

A partial overlap also makes nfsd read the middle of a request as a
record marker ("svc_tcp_read_marker nfsd RPC fragment too large").
Patch 1 includes a capture from a Linux 7.0 client and a 6.12 server
showing the first two cases byte by byte.

Patches 1 and 2 make rpciod, xprtiod and lease renewal freezable, so
nothing is sent after the snapshot. That keeps the session slots
consistent, but not the receive side: TCP still acknowledges replies
that arrive after the snapshot. Patch 3 therefore disconnects every
client transport once the system has resumed. Requests in flight are
retransmitted on the new connection, and the server's reply cache
answers any it has already executed.

Tested in QEMU/KVM on 7.4-rc2: a 7.4-rc2 nfsd with a 10 second lease
and an NFSv4.2 client hibernating in test_resume mode, with the image
write throttled to 81 seconds:

                                         unpatched   1-2 only   1-3
  RPCs received during the image write       12          0        0
  20 hibernations under continuous I/O    hung at 1  hung at 6  20/20

Every freeze of kernel threads completed in 1 ms or less. A capture of
the stress runs confirms patch 3's case: the client's acknowledgement
went backwards on the cycle that hung with patches 1-2 only, and on two
of the twenty cycles with patch 3, where the client reconnected and
carried on. The test harness (two busybox initramfs VMs, no root
needed) is available on request.

Not addressed: the same window affects every active TCP connection, not
only SUNRPC. Keeping network devices quiet while the image is written,
or resetting TCP connections on restore, would fix that generally, but
would need agreement between the PM core and netdev. This series fixes
the case where protocol state outlives the connection and the client
cannot recover by itself. Patches 1 and 2 also freeze rpciod and xprtiod
during suspend to RAM, which is harmless there.

The analysis of the packet captures, the patches, their changelogs, the
QEMU test harness and this letter were drafted with an AI coding
assistant (Claude, from Anthropic), working from the hung client and
server, their logs and captures. It also ran the tests. I have
reviewed the patches and am responsible for them.

Andrew Pam (3):
  SUNRPC: freeze rpciod and xprtiod across system sleep
  NFSv4: run lease renewal on a freezable workqueue
  SUNRPC: reconnect client transports after resuming from hibernation

 fs/nfs/nfs4renewd.c      |  2 +-
 net/sunrpc/clnt.c        | 85 ++++++++++++++++++++++++++++++++++++++++
 net/sunrpc/sched.c       | 11 ++++--
 net/sunrpc/sunrpc.h      |  2 +
 net/sunrpc/sunrpc_syms.c |  7 ++++
 5 files changed, 103 insertions(+), 4 deletions(-)


base-commit: f66469d9905b39d2c260a15d7d6db450f4db67c1
-- 
2.53.0