From: Andrew Pam <andrew@sericyb.com.au>
To: Trond Myklebust <trondmy@kernel.org>, Anna Schumaker <anna@kernel.org>
Cc: Andrew Pam <andrew@sericyb.com.au>, Chuck Lever <cel@kernel.org>,
Jeff Layton <jlayton@kernel.org>, NeilBrown <neil@brown.name>,
Olga Kornievskaia <okorniev@redhat.com>,
Dai Ngo <Dai.Ngo@oracle.com>, Tom Talpey <tom@talpey.com>,
"Rafael J. Wysocki" <rafael@kernel.org>,
Mike Snitzer <snitzer@kernel.org>,
linux-nfs@vger.kernel.org, linux-pm@vger.kernel.org,
netdev@vger.kernel.org, linux-kernel@vger.kernel.org
Subject: [PATCH v2 3/3] SUNRPC: reconnect client transports after resuming from hibernation
Date: Tue, 17 Nov 2026 00:14:27 +1100 [thread overview]
Message-ID: <20261116131428.1024230-4-andrew@sericyb.com.au> (raw)
In-Reply-To: <20261116131428.1024230-1-andrew@sericyb.com.au>
With rpciod and xprtiod frozen across hibernation, no RPC is sent after
the snapshot. TCP can still acknowledge data the server sends then,
though: typically a reply that was in flight when the snapshot was
taken. The restored image has not seen that data, and the server, which
saw it acknowledged, will not send it again. The client then waits
indefinitely for replies that never arrive.
Captured in a QEMU test (below): on the one cycle of six that hung, the
client's TCP acknowledgement went backwards by 200 bytes after the
restore, and the requests outstanding (WRITE, COMMIT, GETATTR) stayed
queued on xprt_pending with no further progress.
After resuming, disconnect every client transport. Requests in flight
are retransmitted on the new connection, and the server answers any it
has already executed from its reply cache. Because nothing was sent
after the snapshot, NFSv4.1 session slot sequence ids still agree on
both sides.
The disconnect is done from a PM notifier at PM_POST_HIBERNATION, which
runs in the restored kernel once everything has thawed. It is also sent
when hibernation fails, where the reconnect is unnecessary but harmless.
xprt_put() may sleep, so references to the transports are taken under
the client list lock and RCU into an array, and released afterwards. The
array is sized by a counting pass first, so it is allocated with
GFP_KERNEL rather than under the locks, and the walk is repeated if
transports were added in between: one left out would stay stalled.
Tested in the QEMU setup described in the first patch of this series,
with 20 hibernations in test_resume mode back to back while NFS writes
(8 MB, fsync) and reads ran continuously. Without the series the client
hung on the first cycle; with only the first two patches, on the sixth.
With this patch all 20 cycles completed and I/O progressed on every
one. The capture shows the client's acknowledgement going backwards on
two of the old connections, and the client reconnecting and carrying on
both times. The single-hibernation test is unchanged apart from one
extra connection per resume: nothing reached the server during the
image write, and the client resumed without errors.
Drafted with an AI coding assistant (Claude), which also ran the tests.
Signed-off-by: Andrew Pam <andrew@sericyb.com.au>
---
net/sunrpc/clnt.c | 112 +++++++++++++++++++++++++++++++++++++++
net/sunrpc/sunrpc.h | 2 +
net/sunrpc/sunrpc_syms.c | 7 +++
3 files changed, 121 insertions(+)
diff --git a/net/sunrpc/clnt.c b/net/sunrpc/clnt.c
index 2f076b4bfd54..f9c31c49dc0b 100644
--- a/net/sunrpc/clnt.c
+++ b/net/sunrpc/clnt.c
@@ -32,6 +32,8 @@
#include <linux/in.h>
#include <linux/in6.h>
#include <linux/un.h>
+#include <linux/suspend.h>
+#include <net/net_namespace.h>
#include <linux/sunrpc/clnt.h>
#include <linux/sunrpc/addr.h>
@@ -881,6 +883,116 @@ int rpc_clnt_iterate_for_each_xprt(struct rpc_clnt *clnt,
}
EXPORT_SYMBOL_GPL(rpc_clnt_iterate_for_each_xprt);
+/*
+ * After hibernation takes its snapshot, the kernel keeps running with the
+ * network up while it writes the image. rpciod and xprtiod are frozen by
+ * then, so no RPC is sent, but TCP still acknowledges anything the server
+ * sends, such as a reply that was in flight when the snapshot was taken.
+ * The restored image has not seen that data, and the server will not send
+ * it again, so the connection cannot make progress.
+ *
+ * Once the system has resumed, disconnect every client transport. Requests
+ * in flight are retransmitted on a new connection, and the server answers
+ * any it has already executed from its reply cache. Because nothing was
+ * sent after the snapshot, NFSv4.1 session slot sequence ids still agree.
+ */
+/*
+ * Take a reference to each client transport in every network namespace and
+ * store it in @xprts, up to @max of them. Returns the number of transports
+ * found, which exceeds @max if the array was too small. Called with
+ * net_rwsem held for reading.
+ */
+static unsigned int rpc_pm_get_xprts(struct rpc_xprt **xprts,
+ unsigned int max)
+{
+ unsigned int n = 0;
+ struct net *net;
+
+ for_each_net(net) {
+ struct sunrpc_net *sn = net_generic(net, sunrpc_net_id);
+ struct rpc_clnt *clnt;
+
+ spin_lock(&sn->rpc_client_lock);
+ rcu_read_lock();
+ list_for_each_entry(clnt, &sn->all_clients, cl_clients) {
+ struct rpc_xprt_switch *xps;
+ struct rpc_xprt *xprt;
+
+ xps = rcu_dereference(clnt->cl_xpi.xpi_xpswitch);
+ if (!xps)
+ continue;
+ list_for_each_entry_rcu(xprt, &xps->xps_xprt_list,
+ xprt_switch) {
+ if (n < max) {
+ xprts[n] = xprt_get(xprt);
+ if (!xprts[n])
+ continue;
+ }
+ n++;
+ }
+ }
+ rcu_read_unlock();
+ spin_unlock(&sn->rpc_client_lock);
+ }
+ return n;
+}
+
+static int rpc_pm_notify(struct notifier_block *nb, unsigned long action,
+ void *data)
+{
+ struct rpc_xprt **xprts = NULL;
+ unsigned int i, n, max = 0;
+
+ if (action != PM_POST_HIBERNATION)
+ return NOTIFY_DONE;
+
+ /*
+ * The references are taken under spinlocks, but xprt_put() may sleep,
+ * so collect them in an array and disconnect after the locks are
+ * dropped. Count first so that the array can be allocated without
+ * GFP_ATOMIC, and recount if transports were added in the meantime:
+ * a transport left out would keep the stalled connection.
+ */
+ for (;;) {
+ down_read(&net_rwsem);
+ n = rpc_pm_get_xprts(xprts, max);
+ up_read(&net_rwsem);
+ if (n <= max)
+ break;
+
+ for (i = 0; i < max; i++)
+ xprt_put(xprts[i]);
+ kfree(xprts);
+ max = n + 8;
+ xprts = kmalloc_objs(*xprts, max);
+ if (!xprts) {
+ pr_warn("RPC: no memory to reset transports after resume\n");
+ return NOTIFY_DONE;
+ }
+ }
+
+ for (i = 0; i < n; i++) {
+ xprt_force_disconnect(xprts[i]);
+ xprt_put(xprts[i]);
+ }
+ kfree(xprts);
+ return NOTIFY_DONE;
+}
+
+static struct notifier_block rpc_pm_nb = {
+ .notifier_call = rpc_pm_notify,
+};
+
+int rpc_pm_notifier_register(void)
+{
+ return register_pm_notifier(&rpc_pm_nb);
+}
+
+void rpc_pm_notifier_unregister(void)
+{
+ unregister_pm_notifier(&rpc_pm_nb);
+}
+
/*
* Kill all tasks for the given client.
* XXX: kill their descendants as well?
diff --git a/net/sunrpc/sunrpc.h b/net/sunrpc/sunrpc.h
index 11b9e5cead8e..1833b167616c 100644
--- a/net/sunrpc/sunrpc.h
+++ b/net/sunrpc/sunrpc.h
@@ -43,6 +43,8 @@ struct rpc_task;
struct rpc_xprt;
int rpc_clients_notifier_register(void);
void rpc_clients_notifier_unregister(void);
+int rpc_pm_notifier_register(void);
+void rpc_pm_notifier_unregister(void);
void auth_domain_cleanup(void);
void svc_sock_update_bufs(struct svc_serv *serv);
enum svc_auth_status svc_authenticate(struct svc_rqst *rqstp);
diff --git a/net/sunrpc/sunrpc_syms.c b/net/sunrpc/sunrpc_syms.c
index 1a3884a0376a..d32786f51ae5 100644
--- a/net/sunrpc/sunrpc_syms.c
+++ b/net/sunrpc/sunrpc_syms.c
@@ -116,6 +116,10 @@ init_sunrpc(void)
if (err)
goto out6;
+ err = rpc_pm_notifier_register();
+ if (err)
+ goto out7;
+
sunrpc_debugfs_init();
#if IS_ENABLED(CONFIG_SUNRPC_DEBUG)
rpc_register_sysctl();
@@ -124,6 +128,8 @@ init_sunrpc(void)
init_socket_xprt(); /* clnt sock transport */
return 0;
+out7:
+ genl_unregister_family(&sunrpc_nl_family);
out6:
rpc_sysfs_exit();
out5:
@@ -141,6 +147,7 @@ init_sunrpc(void)
static void __exit
cleanup_sunrpc(void)
{
+ rpc_pm_notifier_unregister();
genl_unregister_family(&sunrpc_nl_family);
rpc_sysfs_exit();
rpc_cleanup_clids();
--
2.53.0
next prev parent reply other threads:[~2026-11-16 13:15 UTC|newest]
Thread overview: 6+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-11-16 13:14 [PATCH v2 0/3] SUNRPC: keep NFS client connections consistent across hibernation Andrew Pam
2026-11-16 13:14 ` [PATCH v2 1/3] SUNRPC: freeze rpciod and xprtiod across system sleep Andrew Pam
2026-11-16 19:32 ` Anna Schumaker
2026-11-16 13:14 ` [PATCH v2 2/3] NFSv4: run lease renewal on a freezable workqueue Andrew Pam
2026-11-16 13:14 ` Andrew Pam [this message]
2026-11-16 19:34 ` [PATCH v2 3/3] SUNRPC: reconnect client transports after resuming from hibernation Anna Schumaker
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20261116131428.1024230-4-andrew@sericyb.com.au \
--to=andrew@sericyb.com.au \
--cc=Dai.Ngo@oracle.com \
--cc=anna@kernel.org \
--cc=cel@kernel.org \
--cc=jlayton@kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-nfs@vger.kernel.org \
--cc=linux-pm@vger.kernel.org \
--cc=neil@brown.name \
--cc=netdev@vger.kernel.org \
--cc=okorniev@redhat.com \
--cc=rafael@kernel.org \
--cc=snitzer@kernel.org \
--cc=tom@talpey.com \
--cc=trondmy@kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®