From: Andrew Pam <andrew@sericyb.com.au>
To: Trond Myklebust <trondmy@kernel.org>, Anna Schumaker <anna@kernel.org>
Cc: Andrew Pam <andrew@sericyb.com.au>, Chuck Lever <cel@kernel.org>,
Jeff Layton <jlayton@kernel.org>, NeilBrown <neil@brown.name>,
Olga Kornievskaia <okorniev@redhat.com>,
Dai Ngo <Dai.Ngo@oracle.com>, Tom Talpey <tom@talpey.com>,
"Rafael J. Wysocki" <rafael@kernel.org>,
Mike Snitzer <snitzer@kernel.org>,
linux-nfs@vger.kernel.org, linux-pm@vger.kernel.org,
netdev@vger.kernel.org, linux-kernel@vger.kernel.org
Subject: [PATCH v3 0/3] SUNRPC: keep NFS client connections consistent across hibernation
Date: Thu, 19 Nov 2026 13:47:15 +1100 [thread overview]
Message-ID: <20261119024719.311474-1-andrew@sericyb.com.au> (raw)
An NFS client that hibernates with a mount active can resume with the
mount hung, until the server's nfsd is restarted. The underlying
problem was described on linux-pm in 2011 ("hibernate hangs TCP") and
not fixed. This series fixes it for SUNRPC clients.
After hibernation takes its snapshot, the kernel thaws devices, the NIC
among them, to write the image. Freezable kernel work stays frozen until
the image is restored, but rpciod, xprtiod and the NFSv4 lease renewal
work are not freezable. So RPCs are still sent and answered while the
image is written, and TCP acknowledges data from the server, none of
which the restored image knows about. On resume the client's TCP
sequence numbers, its acknowledgement point and its NFSv4.1 session slot
sequence ids are all behind the server's.
Captured between a Linux 7.0 client and a 6.12 server:
- 05:06:02 last exchange before the snapshot; client snd_nxt
2711544997.
- 05:06:15 hibernation entry.
- 05:06:44 and 05:07:48, after the snapshot: a lease-renewal
SEQUENCE on slot 0 (seqid 192219), the next (192220), and
DELEGRETURNs on slots 1-5. The server executes all of them and
acknowledges up to TCP sequence 2711546645.
- 14 hours later, on resume: the client retransmits the slot 0
SEQUENCE from TCP sequence 2711544997. The server discards it as a
duplicate and acknowledges 2711546645, which the client ignores as
acknowledging unsent data. Nothing more crosses the connection.
- On reconnecting, slot 0 gets NFS4ERR_SEQ_MISORDERED (server at
192220, client at 192219) every 15 seconds indefinitely; the state
manager waits in nfs4_begin_drain_session() for that task, and
every other RPC waits behind it.
Other resumes produced the same rollback with different symptoms:
requests accepted by the server's TCP but never reaching nfsd, a partial
overlap that left nfsd reading a session ID as a record marker
("svc_tcp_read_marker nfsd RPC fragment too large"), or a client
waiting for replies it had acknowledged before the restore, which the
server will not send again.
Patches 1 and 2 make rpciod, xprtiod and lease renewal freezable, so
nothing is sent after the snapshot. That keeps the session slots
consistent, but not the receive side: TCP still acknowledges replies
that arrive after the snapshot. Patch 3 therefore disconnects every
client transport once the system has resumed. Requests in flight are
retransmitted on the new connection, and the server's reply cache
answers any it has already executed.
Testing, in QEMU/KVM: an NFSv4.2 client hibernating in test_resume mode
(the image is written, then restored at once) against an nfsd with a 10
second lease, with 1.2 GB of incompressible data in memory and the swap
device throttled so that writing the image takes 81 seconds. The stress
test runs 20 hibernations back to back while NFS writes (8 MB, fsync)
and reads run continuously. On 7.4-rc2, for v1:
unpatched 1-2 only 1-3
RPCs received during the image write 12 0 0
20 hibernations under continuous I/O hung at 1 hung at 6 20/20
Unpatched, the restored client also showed a SEQUENCE error, two extra
connections and 10 seconds of queueing after a single hibernation; with
the series it resumed without errors. Every freeze of kernel threads
completed in 1 ms or less. A capture of the stress runs confirms patch
3's case: the client's acknowledgement went backwards on the cycle that
hung with patches 1-2 only, and on two of the twenty cycles with patch
3, where the client reconnected and carried on. The indefinite
NFS4ERR_SEQ_MISORDERED loop seen on real hardware was not reproduced in
the VM: it also needs a request holding a session slot across the
restore.
On 7.4-rc3, unpatched, the client hung on the third of twenty
hibernations; v2 and v3 each completed all twenty. v3 also ran twenty
hibernations under lockdep (PROVE_LOCKING, PROVE_RCU,
DEBUG_ATOMIC_SLEEP) without a report from the new code, and was built
with SUNRPC as a module. The test harness (two busybox
initramfs VMs, no root needed) is available on request.
Not addressed: the same window affects every active TCP connection, not
only SUNRPC. Keeping network devices quiet while the image is written,
or resetting TCP connections on restore, would fix that generally, but
would need agreement between the PM core and netdev. This series fixes
the case where protocol state outlives the connection and the client
cannot recover by itself. Patches 1 and 2 also freeze rpciod and xprtiod
during suspend to RAM, which is harmless there.
This series applies on top of Chuck Lever's "SUNRPC: Set WQ_SYSFS on
rpciod and xprtiod":
https://lore.kernel.org/linux-nfs/20260915-performance-v3-3-ae26d460bfd3@kernel.org/
Mike Snitzer's "SUNRPC: Make rpciod and xprtiod visible in sysfs"
changes the same two alloc_workqueue() calls for the same purpose.
Patch 3 is compatible with commit 9d84719d34cb ("SUNRPC: add
request-scoped disconnects when cancelling tasks"): its conditional
disconnect is skipped once the connection cookie has moved on, so it
cannot close the connection that patch 3's reconnect creates.
Changes in v3:
- Rebased onto Chuck Lever's WQ_SYSFS patch; patch 1 now adds
WQ_FREEZABLE to its shared wq_flags. The original comment in
rpciod_start() is left alone. (Anna Schumaker)
- Patch 3: no array, counting pass or transport references. The
notifier calls xprt_force_disconnect(), which does not sleep, on
each transport while holding its switch's xps_lock, which keeps the
listed transports alive. This also removes the misplaced double
comment. (Anna Schumaker)
- Changelogs trimmed to what each patch does and why; the capture and
test results moved here. (Anna Schumaker)
- v2: https://lore.kernel.org/all/20261116131428.1024230-1-andrew@sericyb.com.au/
The analysis of the packet captures, the patches, the QEMU test harness
and drafts of the changelogs and this letter were produced with an AI
coding assistant (Claude, from Anthropic), working from the hung client
and server, their logs and captures. It also ran the tests. I have
reviewed and edited the patches and am responsible for them.
Andrew Pam (3):
SUNRPC: freeze rpciod and xprtiod across system sleep
NFSv4: run lease renewal on a freezable workqueue
SUNRPC: reconnect client transports after resuming from hibernation
fs/nfs/nfs4renewd.c | 2 +-
net/sunrpc/clnt.c | 61 ++++++++++++++++++++++++++++++++++++++++
net/sunrpc/sched.c | 3 +-
net/sunrpc/sunrpc.h | 2 ++
net/sunrpc/sunrpc_syms.c | 7 +++++
5 files changed, 73 insertions(+), 2 deletions(-)
base-commit: aebb7274476403040d582c61478df8f4a7bf9cd3
prerequisite-patch-id: a8475a72e4e9a2df387e41ca12df8aa67733256b
--
2.53.0
next reply other threads:[~2026-11-19 2:47 UTC|newest]
Thread overview: 4+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-11-19 2:47 Andrew Pam [this message]
2026-11-19 2:47 ` [PATCH v3 1/3] SUNRPC: freeze rpciod and xprtiod across system sleep Andrew Pam
2026-11-19 2:47 ` [PATCH v3 2/3] NFSv4: run lease renewal on a freezable workqueue Andrew Pam
2026-11-19 2:47 ` [PATCH v3 3/3] SUNRPC: reconnect client transports after resuming from hibernation Andrew Pam
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20261119024719.311474-1-andrew@sericyb.com.au \
--to=andrew@sericyb.com.au \
--cc=Dai.Ngo@oracle.com \
--cc=anna@kernel.org \
--cc=cel@kernel.org \
--cc=jlayton@kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-nfs@vger.kernel.org \
--cc=linux-pm@vger.kernel.org \
--cc=neil@brown.name \
--cc=netdev@vger.kernel.org \
--cc=okorniev@redhat.com \
--cc=rafael@kernel.org \
--cc=snitzer@kernel.org \
--cc=tom@talpey.com \
--cc=trondmy@kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®