[PATCH 0/3] SUNRPC: keep NFS client connections consistent across hibernation

[PATCH 0/3] SUNRPC: keep NFS client connections consistent across hibernation

[Andrew Pam]

An NFS client that hibernates with a mount active can resume with the
mount hung, until the server's nfsd is restarted. The underlying
problem was described on linux-pm in 2011 ("hibernate hangs TCP") and
not fixed. This series fixes it for SUNRPC clients.

After hibernation takes its snapshot, the kernel thaws devices, the NIC
among them, to write the image. Freezable kernel work stays frozen until
the image is restored, but rpciod, xprtiod and the NFSv4 lease renewal
work are not freezable. So RPCs are still sent and answered while the
image is written, and TCP acknowledges data from the server, none of
which the restored image knows about. On resume the client's TCP
sequence numbers, its acknowledgement point and its NFSv4.1 session slot
sequence ids are all behind the server's. Depending on what happened in
that window, the client:

  - resends data that the server discards as a duplicate, so the
    connection carries nothing;
  - gets NFS4ERR_SEQ_MISORDERED on a slot indefinitely, while the state
    manager waits in nfs4_begin_drain_session() for that slot; or
  - waits for replies it acknowledged before the restore, which the
    server will not send again.

A partial overlap also makes nfsd read the middle of a request as a
record marker ("svc_tcp_read_marker nfsd RPC fragment too large").
Patch 1 includes a capture from a Linux 7.0 client and a 6.12 server
showing the first two cases byte by byte.

Patches 1 and 2 make rpciod, xprtiod and lease renewal freezable, so
nothing is sent after the snapshot. That keeps the session slots
consistent, but not the receive side: TCP still acknowledges replies
that arrive after the snapshot. Patch 3 therefore disconnects every
client transport once the system has resumed. Requests in flight are
retransmitted on the new connection, and the server's reply cache
answers any it has already executed.

Tested in QEMU/KVM on 7.4-rc2: a 7.4-rc2 nfsd with a 10 second lease
and an NFSv4.2 client hibernating in test_resume mode, with the image
write throttled to 81 seconds:

                                         unpatched   1-2 only   1-3
  RPCs received during the image write       12          0        0
  20 hibernations under continuous I/O    hung at 1  hung at 6  20/20

Every freeze of kernel threads completed in 1 ms or less. A capture of
the stress runs confirms patch 3's case: the client's acknowledgement
went backwards on the cycle that hung with patches 1-2 only, and on two
of the twenty cycles with patch 3, where the client reconnected and
carried on. The test harness (two busybox initramfs VMs, no root
needed) is available on request.

Not addressed: the same window affects every active TCP connection, not
only SUNRPC. Keeping network devices quiet while the image is written,
or resetting TCP connections on restore, would fix that generally, but
would need agreement between the PM core and netdev. This series fixes
the case where protocol state outlives the connection and the client
cannot recover by itself. Patches 1 and 2 also freeze rpciod and xprtiod
during suspend to RAM, which is harmless there.

The analysis of the packet captures, the patches, their changelogs, the
QEMU test harness and this letter were drafted with an AI coding
assistant (Claude, from Anthropic), working from the hung client and
server, their logs and captures. It also ran the tests. I have
reviewed the patches and am responsible for them.

Andrew Pam (3):
  SUNRPC: freeze rpciod and xprtiod across system sleep
  NFSv4: run lease renewal on a freezable workqueue
  SUNRPC: reconnect client transports after resuming from hibernation

 fs/nfs/nfs4renewd.c      |  2 +-
 net/sunrpc/clnt.c        | 85 ++++++++++++++++++++++++++++++++++++++++
 net/sunrpc/sched.c       | 11 ++++--
 net/sunrpc/sunrpc.h      |  2 +
 net/sunrpc/sunrpc_syms.c |  7 ++++
 5 files changed, 103 insertions(+), 4 deletions(-)


base-commit: f66469d9905b39d2c260a15d7d6db450f4db67c1
-- 
2.53.0

[PATCH 1/3] SUNRPC: freeze rpciod and xprtiod across system sleep

[Andrew Pam]

After hibernation takes its snapshot, the kernel thaws devices to write
the image, and the NIC with them. Freezable kernel threads and
workqueues stay frozen until the image is restored, but rpciod and
xprtiod are not freezable, so RPCs continue to be transmitted, and
transports reconnected, until power-off. The server acts on them; the
restored image knows nothing of them.

With NFSv4.1 this wedges the mount after resume. Captured between a
client that hibernated and its server (Linux 7.0 client, 6.12 server):

  - 05:06:02 last exchange before the snapshot; client snd_nxt
    2711544997.
  - 05:06:15 hibernation entry.
  - 05:06:44 and 05:07:48, after the snapshot: a lease-renewal
    SEQUENCE on slot 0 (seqid 192219), the next (192220), and
    DELEGRETURNs on slots 1-5. The server executes all of them and
    acknowledges up to TCP sequence 2711546645.
  - 14 hours later, on resume: the client retransmits the slot 0
    SEQUENCE from TCP sequence 2711544997. The server discards it as a
    duplicate and acknowledges 2711546645, which the client ignores as
    acknowledging unsent data. Nothing more crosses the connection.
  - On reconnecting, slot 0 gets NFS4ERR_SEQ_MISORDERED (server at
    192220, client at 192219) every 15 seconds indefinitely; the state
    manager waits in nfs4_begin_drain_session() for that task, and
    every other RPC waits behind it.

Other resumes produced the same rollback with different symptoms:
requests accepted by the server's TCP but never reaching nfsd, or a
partial overlap that left nfsd reading a session ID as a record marker
("svc_tcp_read_marker nfsd RPC fragment too large"). Restarting the
server is the only recovery.

Make both workqueues freezable, as the SMB client already does for all
of its workqueues (WQ_FREEZABLE | WQ_MEM_RECLAIM). Work queued while
frozen runs after thaw_processes(), in the restored kernel, which is
consistent with the server. Memory preallocation for the image happens
before freeze_kernel_threads(), so writeback through rpciod is still
available then, and pm_restrict_gfp_mask() keeps reclaim from issuing
I/O while the image is written.

WQ_FREEZABLE is consulted only when freezing and thawing, not when work
is queued or run, so normal operation is unaffected. Entering system
sleep now waits for in-flight rpciod and xprtiod items to finish; these
are short state-machine steps with non-blocking sends and connects. If
one ever blocked for longer than freeze_timeout_msecs, suspend or
hibernation would fail, naming the busy workqueue, where today it
proceeds. The SMB client and XFS accept the same trade-off for their
reclaim workqueues.

This does not stop the network stack from acknowledging data the
server sends after the snapshot, such as a callback or a reply to a
request still in flight when the snapshot was taken; that remains
a general hibernation and TCP problem.

No Fixes: tag: rpciod has never been freezable, so no single commit
introduced this.

Tested in QEMU/KVM: an NFSv4.2 client hibernating in test_resume mode
(the image is written, then restored at once) against a 7.4-rc2 nfsd
with a 10 second lease, with 1.2 GB of incompressible data in memory and
the swap device throttled so that writing the image takes 81 seconds.
Without this patch the server received 12 SEQUENCE renewals during the
image write, none of which the restored client knew of; after the
restore the client's mountstats show a SEQUENCE error, two extra
connections and 10 seconds of queueing before it recovered. With this
series the server received nothing from the snapshot until the restore,
and the client resumed without errors. The indefinite hang seen on real
hardware was not reproduced in the VM: it also needs a request holding
a session slot across the restore.

Drafted with an AI coding assistant (Claude), which analysed the packet
capture, wrote the change and this description, and ran the test above.

Signed-off-by: Andrew Pam <andrew@sericyb.com.au>
---
 net/sunrpc/sched.c | 11 ++++++++---
 1 file changed, 8 insertions(+), 3 deletions(-)

diff --git a/net/sunrpc/sched.c b/net/sunrpc/sched.c
index 016f16ca5779..41f25309b8fb 100644
--- a/net/sunrpc/sched.c
+++ b/net/sunrpc/sched.c
@@ -1276,13 +1276,18 @@ static int rpciod_start(void)
 	struct workqueue_struct *wq;
 
 	/*
-	 * Create the rpciod thread and wait for it to start.
+	 * Both workqueues are freezable so that no RPC is transmitted, and
+	 * no connection is made, between taking a hibernation snapshot and
+	 * powering off. Anything sent then changes the server's state but is
+	 * missing from the image, and is forgotten when the image is restored.
 	 */
-	wq = alloc_workqueue("rpciod", WQ_MEM_RECLAIM | WQ_UNBOUND, 0);
+	wq = alloc_workqueue("rpciod",
+			     WQ_MEM_RECLAIM | WQ_UNBOUND | WQ_FREEZABLE, 0);
 	if (!wq)
 		goto out_failed;
 	rpciod_workqueue = wq;
-	wq = alloc_workqueue("xprtiod", WQ_UNBOUND | WQ_MEM_RECLAIM, 0);
+	wq = alloc_workqueue("xprtiod",
+			     WQ_UNBOUND | WQ_MEM_RECLAIM | WQ_FREEZABLE, 0);
 	if (!wq)
 		goto free_rpciod;
 	xprtiod_workqueue = wq;

base-commit: f66469d9905b39d2c260a15d7d6db450f4db67c1
-- 
2.53.0

[PATCH 2/3] NFSv4: run lease renewal on a freezable workqueue

[Andrew Pam]

The renewal work is queued on system_percpu_wq, which is not frozen
during system sleep. During hibernation it can therefore fire after the
snapshot has been taken, while the image is written. It then queues
renewal and DELEGRETURN RPCs and can wake the state manager, all on
behalf of a kernel image whose state is about to be discarded.

With rpciod frozen across system sleep (previous patch) those RPCs are
not transmitted until after resume. Run the renewal itself on
system_freezable_wq too, so it does not run in that window at all; the
delayed work fires after thaw if its time has passed.

Tested together with the previous patch, as described there.

Drafted with an AI coding assistant (Claude).

Signed-off-by: Andrew Pam <andrew@sericyb.com.au>
---
 fs/nfs/nfs4renewd.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/fs/nfs/nfs4renewd.c b/fs/nfs/nfs4renewd.c
index 30065df1482e..a0326f830bca 100644
--- a/fs/nfs/nfs4renewd.c
+++ b/fs/nfs/nfs4renewd.c
@@ -122,7 +122,7 @@ nfs4_schedule_state_renewal(struct nfs_client *clp)
 		timeout = 5 * HZ;
 	dprintk("%s: requeueing work. Lease period = %ld\n",
 			__func__, (timeout + HZ - 1) / HZ);
-	mod_delayed_work(system_percpu_wq, &clp->cl_renewd, timeout);
+	mod_delayed_work(system_freezable_wq, &clp->cl_renewd, timeout);
 	set_bit(NFS_CS_RENEWD, &clp->cl_res_state);
 	spin_unlock(&clp->cl_lock);
 }
-- 
2.53.0

[PATCH 3/3] SUNRPC: reconnect client transports after resuming from hibernation

[Andrew Pam]

With rpciod and xprtiod frozen across hibernation, no RPC is sent after
the snapshot. TCP can still acknowledge data the server sends then,
though: typically a reply that was in flight when the snapshot was
taken. The restored image has not seen that data, and the server, which
saw it acknowledged, will not send it again. The client then waits
indefinitely for replies that never arrive.

Captured in a QEMU test (below): on the one cycle of six that hung, the
client's TCP acknowledgement went backwards by 200 bytes after the
restore, and the requests outstanding (WRITE, COMMIT, GETATTR) stayed
queued on xprt_pending with no further progress.

After resuming, disconnect every client transport. Requests in flight
are retransmitted on the new connection, and the server answers any it
has already executed from its reply cache. Because nothing was sent
after the snapshot, NFSv4.1 session slot sequence ids still agree on
both sides.

The disconnect is done from a PM notifier at PM_POST_HIBERNATION, which
runs in the restored kernel once everything has thawed. It is also sent
when hibernation fails, where the reconnect is unnecessary but harmless.
xprt_put() may sleep, so references to the transports are collected
under the client list lock and RCU, and released afterwards.

Tested in the QEMU setup described in the first patch of this series,
with 20 hibernations in test_resume mode back to back while NFS writes
(8 MB, fsync) and reads ran continuously. Without the series the client
hung on the first cycle; with only the first two patches, on the sixth.
With this patch all 20 cycles completed and I/O progressed on every
one. The capture shows the client's acknowledgement going backwards on
two of the old connections, and the client reconnecting and carrying on
both times. The single-hibernation test is unchanged apart from one
extra connection per resume: nothing reached the server during the
image write, and the client resumed without errors.

Drafted with an AI coding assistant (Claude), which also ran the tests.

Signed-off-by: Andrew Pam <andrew@sericyb.com.au>
---
 net/sunrpc/clnt.c        | 85 ++++++++++++++++++++++++++++++++++++++++
 net/sunrpc/sunrpc.h      |  2 +
 net/sunrpc/sunrpc_syms.c |  7 ++++
 3 files changed, 94 insertions(+)

diff --git a/net/sunrpc/clnt.c b/net/sunrpc/clnt.c
index 6cedc824cf82..a488be2891be 100644
--- a/net/sunrpc/clnt.c
+++ b/net/sunrpc/clnt.c
@@ -32,6 +32,8 @@
 #include <linux/in.h>
 #include <linux/in6.h>
 #include <linux/un.h>
+#include <linux/suspend.h>
+#include <net/net_namespace.h>
 
 #include <linux/sunrpc/clnt.h>
 #include <linux/sunrpc/addr.h>
@@ -878,6 +880,89 @@ int rpc_clnt_iterate_for_each_xprt(struct rpc_clnt *clnt,
 }
 EXPORT_SYMBOL_GPL(rpc_clnt_iterate_for_each_xprt);
 
+struct rpc_pm_xprt {
+	struct list_head	list;
+	struct rpc_xprt		*xprt;
+};
+
+/*
+ * After hibernation takes its snapshot, the kernel keeps running with the
+ * network up while it writes the image. rpciod and xprtiod are frozen by
+ * then, so no RPC is sent, but TCP still acknowledges anything the server
+ * sends, such as a reply that was in flight when the snapshot was taken.
+ * The restored image has not seen that data, and the server will not send
+ * it again, so the connection cannot make progress.
+ *
+ * Once the system has resumed, disconnect every client transport. Requests
+ * in flight are retransmitted on a new connection, and the server answers
+ * any it has already executed from its reply cache. Because nothing was
+ * sent after the snapshot, NFSv4.1 session slot sequence ids still agree.
+ */
+static int rpc_pm_notify(struct notifier_block *nb, unsigned long action,
+			 void *data)
+{
+	struct rpc_pm_xprt *entry, *tmp;
+	LIST_HEAD(xprts);
+	struct net *net;
+
+	if (action != PM_POST_HIBERNATION)
+		return NOTIFY_DONE;
+
+	/* xprt_put() may sleep, so collect references first */
+	down_read(&net_rwsem);
+	for_each_net(net) {
+		struct sunrpc_net *sn = net_generic(net, sunrpc_net_id);
+		struct rpc_clnt *clnt;
+
+		spin_lock(&sn->rpc_client_lock);
+		rcu_read_lock();
+		list_for_each_entry(clnt, &sn->all_clients, cl_clients) {
+			struct rpc_xprt_switch *xps;
+			struct rpc_xprt *xprt;
+
+			xps = rcu_dereference(clnt->cl_xpi.xpi_xpswitch);
+			if (!xps)
+				continue;
+			list_for_each_entry_rcu(xprt, &xps->xps_xprt_list,
+						xprt_switch) {
+				entry = kmalloc_obj(*entry, GFP_ATOMIC);
+				if (!entry)
+					break;
+				entry->xprt = xprt_get(xprt);
+				if (!entry->xprt) {
+					kfree(entry);
+					continue;
+				}
+				list_add(&entry->list, &xprts);
+			}
+		}
+		rcu_read_unlock();
+		spin_unlock(&sn->rpc_client_lock);
+	}
+	up_read(&net_rwsem);
+
+	list_for_each_entry_safe(entry, tmp, &xprts, list) {
+		xprt_force_disconnect(entry->xprt);
+		xprt_put(entry->xprt);
+		kfree(entry);
+	}
+	return NOTIFY_DONE;
+}
+
+static struct notifier_block rpc_pm_nb = {
+	.notifier_call = rpc_pm_notify,
+};
+
+int rpc_pm_notifier_register(void)
+{
+	return register_pm_notifier(&rpc_pm_nb);
+}
+
+void rpc_pm_notifier_unregister(void)
+{
+	unregister_pm_notifier(&rpc_pm_nb);
+}
+
 /*
  * Kill all tasks for the given client.
  * XXX: kill their descendants as well?
diff --git a/net/sunrpc/sunrpc.h b/net/sunrpc/sunrpc.h
index e3c6e3b63f0b..6d7581a115ce 100644
--- a/net/sunrpc/sunrpc.h
+++ b/net/sunrpc/sunrpc.h
@@ -40,6 +40,8 @@ struct svc_serv;
 struct svc_rqst;
 int rpc_clients_notifier_register(void);
 void rpc_clients_notifier_unregister(void);
+int rpc_pm_notifier_register(void);
+void rpc_pm_notifier_unregister(void);
 void auth_domain_cleanup(void);
 void svc_sock_update_bufs(struct svc_serv *serv);
 enum svc_auth_status svc_authenticate(struct svc_rqst *rqstp);
diff --git a/net/sunrpc/sunrpc_syms.c b/net/sunrpc/sunrpc_syms.c
index 1a3884a0376a..d32786f51ae5 100644
--- a/net/sunrpc/sunrpc_syms.c
+++ b/net/sunrpc/sunrpc_syms.c
@@ -116,6 +116,10 @@ init_sunrpc(void)
 	if (err)
 		goto out6;
 
+	err = rpc_pm_notifier_register();
+	if (err)
+		goto out7;
+
 	sunrpc_debugfs_init();
 #if IS_ENABLED(CONFIG_SUNRPC_DEBUG)
 	rpc_register_sysctl();
@@ -124,6 +128,8 @@ init_sunrpc(void)
 	init_socket_xprt();	/* clnt sock transport */
 	return 0;
 
+out7:
+	genl_unregister_family(&sunrpc_nl_family);
 out6:
 	rpc_sysfs_exit();
 out5:
@@ -141,6 +147,7 @@ init_sunrpc(void)
 static void __exit
 cleanup_sunrpc(void)
 {
+	rpc_pm_notifier_unregister();
 	genl_unregister_family(&sunrpc_nl_family);
 	rpc_sysfs_exit();
 	rpc_cleanup_clids();
-- 
2.53.0