From: "Anna Schumaker" <anna@kernel.org>
To: "Andrew Pam" <andrew@sericyb.com.au>,
"Trond Myklebust" <trondmy@kernel.org>
Cc: "Chuck Lever" <cel@kernel.org>,
"Jeff Layton" <jlayton@kernel.org>, NeilBrown <neil@brown.name>,
"Olga Kornievskaia" <okorniev@redhat.com>,
"Dai Ngo" <Dai.Ngo@oracle.com>, "Tom Talpey" <tom@talpey.com>,
"Rafael J. Wysocki" <rafael@kernel.org>,
"Mike Snitzer" <snitzer@kernel.org>,
linux-nfs@vger.kernel.org, linux-pm@vger.kernel.org,
netdev@vger.kernel.org, linux-kernel@vger.kernel.org
Subject: Re: [PATCH v2 3/3] SUNRPC: reconnect client transports after resuming from hibernation
Date: Mon, 16 Nov 2026 14:34:54 -0500 [thread overview]
Message-ID: <51a2b5d3-4322-42ed-a2ee-e17c125865fa@app.fastmail.com> (raw)
In-Reply-To: <20261116131428.1024230-4-andrew@sericyb.com.au>
Hi Andrew,
On Mon, Nov 16, 2026, at 8:14 AM, Andrew Pam wrote:
> With rpciod and xprtiod frozen across hibernation, no RPC is sent after
> the snapshot. TCP can still acknowledge data the server sends then,
> though: typically a reply that was in flight when the snapshot was
> taken. The restored image has not seen that data, and the server, which
> saw it acknowledged, will not send it again. The client then waits
> indefinitely for replies that never arrive.
>
> Captured in a QEMU test (below): on the one cycle of six that hung, the
> client's TCP acknowledgement went backwards by 200 bytes after the
> restore, and the requests outstanding (WRITE, COMMIT, GETATTR) stayed
> queued on xprt_pending with no further progress.
>
> After resuming, disconnect every client transport. Requests in flight
> are retransmitted on the new connection, and the server answers any it
> has already executed from its reply cache. Because nothing was sent
> after the snapshot, NFSv4.1 session slot sequence ids still agree on
> both sides.
>
> The disconnect is done from a PM notifier at PM_POST_HIBERNATION, which
> runs in the restored kernel once everything has thawed. It is also sent
> when hibernation fails, where the reconnect is unnecessary but harmless.
> xprt_put() may sleep, so references to the transports are taken under
> the client list lock and RCU into an array, and released afterwards. The
> array is sized by a counting pass first, so it is allocated with
> GFP_KERNEL rather than under the locks, and the walk is repeated if
> transports were added in between: one left out would stay stalled.
>
> Tested in the QEMU setup described in the first patch of this series,
> with 20 hibernations in test_resume mode back to back while NFS writes
> (8 MB, fsync) and reads ran continuously. Without the series the client
> hung on the first cycle; with only the first two patches, on the sixth.
> With this patch all 20 cycles completed and I/O progressed on every
> one. The capture shows the client's acknowledgement going backwards on
> two of the old connections, and the client reconnecting and carrying on
> both times. The single-hibernation test is unchanged apart from one
> extra connection per resume: nothing reached the server during the
> image write, and the client resumed without errors.
>
> Drafted with an AI coding assistant (Claude), which also ran the tests.
>
> Signed-off-by: Andrew Pam <andrew@sericyb.com.au>
> ---
> net/sunrpc/clnt.c | 112 +++++++++++++++++++++++++++++++++++++++
> net/sunrpc/sunrpc.h | 2 +
> net/sunrpc/sunrpc_syms.c | 7 +++
> 3 files changed, 121 insertions(+)
>
> diff --git a/net/sunrpc/clnt.c b/net/sunrpc/clnt.c
> index 2f076b4bfd54..f9c31c49dc0b 100644
> --- a/net/sunrpc/clnt.c
> +++ b/net/sunrpc/clnt.c
> @@ -32,6 +32,8 @@
> #include <linux/in.h>
> #include <linux/in6.h>
> #include <linux/un.h>
> +#include <linux/suspend.h>
> +#include <net/net_namespace.h>
>
> #include <linux/sunrpc/clnt.h>
> #include <linux/sunrpc/addr.h>
> @@ -881,6 +883,116 @@ int rpc_clnt_iterate_for_each_xprt(struct rpc_clnt *clnt,
> }
> EXPORT_SYMBOL_GPL(rpc_clnt_iterate_for_each_xprt);
>
> +/*
> + * After hibernation takes its snapshot, the kernel keeps running with the
> + * network up while it writes the image. rpciod and xprtiod are frozen by
> + * then, so no RPC is sent, but TCP still acknowledges anything the server
> + * sends, such as a reply that was in flight when the snapshot was taken.
> + * The restored image has not seen that data, and the server will not send
> + * it again, so the connection cannot make progress.
> + *
> + * Once the system has resumed, disconnect every client transport. Requests
> + * in flight are retransmitted on a new connection, and the server answers
> + * any it has already executed from its reply cache. Because nothing was
> + * sent after the snapshot, NFSv4.1 session slot sequence ids still agree.
> + */
> +/*
> + * Take a reference to each client transport in every network namespace and
> + * store it in @xprts, up to @max of them. Returns the number of transports
> + * found, which exceeds @max if the array was too small. Called with
> + * net_rwsem held for reading.
> + */
What's with the weird double comment here? I like the second one better.
> +static unsigned int rpc_pm_get_xprts(struct rpc_xprt **xprts,
> + unsigned int max)
> +{
> + unsigned int n = 0;
> + struct net *net;
> +
> + for_each_net(net) {
> + struct sunrpc_net *sn = net_generic(net, sunrpc_net_id);
> + struct rpc_clnt *clnt;
> +
> + spin_lock(&sn->rpc_client_lock);
> + rcu_read_lock();
> + list_for_each_entry(clnt, &sn->all_clients, cl_clients) {
> + struct rpc_xprt_switch *xps;
> + struct rpc_xprt *xprt;
> +
> + xps = rcu_dereference(clnt->cl_xpi.xpi_xpswitch);
> + if (!xps)
> + continue;
> + list_for_each_entry_rcu(xprt, &xps->xps_xprt_list,
> + xprt_switch) {
> + if (n < max) {
> + xprts[n] = xprt_get(xprt);
> + if (!xprts[n])
> + continue;
> + }
> + n++;
> + }
> + }
> + rcu_read_unlock();
> + spin_unlock(&sn->rpc_client_lock);
> + }
> + return n;
> +}
> +
> +static int rpc_pm_notify(struct notifier_block *nb, unsigned long action,
> + void *data)
> +{
> + struct rpc_xprt **xprts = NULL;
> + unsigned int i, n, max = 0;
> +
> + if (action != PM_POST_HIBERNATION)
> + return NOTIFY_DONE;
> +
> + /*
> + * The references are taken under spinlocks, but xprt_put() may sleep,
> + * so collect them in an array and disconnect after the locks are
> + * dropped. Count first so that the array can be allocated without
> + * GFP_ATOMIC, and recount if transports were added in the meantime:
> + * a transport left out would keep the stalled connection.
> + */
Is there a way to do this without the double loop? The rpc_xprt_switch has a
"xps_nxprts" field that should hold the number of xprts in the switch. Using
that might be more straightforward than looping twice and counting.
Thanks,
Anna
> + for (;;) {
> + down_read(&net_rwsem);
> + n = rpc_pm_get_xprts(xprts, max);
> + up_read(&net_rwsem);
> + if (n <= max)
> + break;
> +
> + for (i = 0; i < max; i++)
> + xprt_put(xprts[i]);
> + kfree(xprts);
> + max = n + 8;
> + xprts = kmalloc_objs(*xprts, max);
> + if (!xprts) {
> + pr_warn("RPC: no memory to reset transports after resume\n");
> + return NOTIFY_DONE;
> + }
> + }
> +
> + for (i = 0; i < n; i++) {
> + xprt_force_disconnect(xprts[i]);
> + xprt_put(xprts[i]);
> + }
> + kfree(xprts);
> + return NOTIFY_DONE;
> +}
> +
> +static struct notifier_block rpc_pm_nb = {
> + .notifier_call = rpc_pm_notify,
> +};
> +
> +int rpc_pm_notifier_register(void)
> +{
> + return register_pm_notifier(&rpc_pm_nb);
> +}
> +
> +void rpc_pm_notifier_unregister(void)
> +{
> + unregister_pm_notifier(&rpc_pm_nb);
> +}
> +
> /*
> * Kill all tasks for the given client.
> * XXX: kill their descendants as well?
> diff --git a/net/sunrpc/sunrpc.h b/net/sunrpc/sunrpc.h
> index 11b9e5cead8e..1833b167616c 100644
> --- a/net/sunrpc/sunrpc.h
> +++ b/net/sunrpc/sunrpc.h
> @@ -43,6 +43,8 @@ struct rpc_task;
> struct rpc_xprt;
> int rpc_clients_notifier_register(void);
> void rpc_clients_notifier_unregister(void);
> +int rpc_pm_notifier_register(void);
> +void rpc_pm_notifier_unregister(void);
> void auth_domain_cleanup(void);
> void svc_sock_update_bufs(struct svc_serv *serv);
> enum svc_auth_status svc_authenticate(struct svc_rqst *rqstp);
> diff --git a/net/sunrpc/sunrpc_syms.c b/net/sunrpc/sunrpc_syms.c
> index 1a3884a0376a..d32786f51ae5 100644
> --- a/net/sunrpc/sunrpc_syms.c
> +++ b/net/sunrpc/sunrpc_syms.c
> @@ -116,6 +116,10 @@ init_sunrpc(void)
> if (err)
> goto out6;
>
> + err = rpc_pm_notifier_register();
> + if (err)
> + goto out7;
> +
> sunrpc_debugfs_init();
> #if IS_ENABLED(CONFIG_SUNRPC_DEBUG)
> rpc_register_sysctl();
> @@ -124,6 +128,8 @@ init_sunrpc(void)
> init_socket_xprt(); /* clnt sock transport */
> return 0;
>
> +out7:
> + genl_unregister_family(&sunrpc_nl_family);
> out6:
> rpc_sysfs_exit();
> out5:
> @@ -141,6 +147,7 @@ init_sunrpc(void)
> static void __exit
> cleanup_sunrpc(void)
> {
> + rpc_pm_notifier_unregister();
> genl_unregister_family(&sunrpc_nl_family);
> rpc_sysfs_exit();
> rpc_cleanup_clids();
> --
> 2.53.0
prev parent reply other threads:[~2026-11-16 19:35 UTC|newest]
Thread overview: 6+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-11-16 13:14 [PATCH v2 0/3] SUNRPC: keep NFS client connections consistent across hibernation Andrew Pam
2026-11-16 13:14 ` [PATCH v2 1/3] SUNRPC: freeze rpciod and xprtiod across system sleep Andrew Pam
2026-11-16 19:32 ` Anna Schumaker
2026-11-16 13:14 ` [PATCH v2 2/3] NFSv4: run lease renewal on a freezable workqueue Andrew Pam
2026-11-16 13:14 ` [PATCH v2 3/3] SUNRPC: reconnect client transports after resuming from hibernation Andrew Pam
2026-11-16 19:34 ` Anna Schumaker [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=51a2b5d3-4322-42ed-a2ee-e17c125865fa@app.fastmail.com \
--to=anna@kernel.org \
--cc=Dai.Ngo@oracle.com \
--cc=andrew@sericyb.com.au \
--cc=cel@kernel.org \
--cc=jlayton@kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-nfs@vger.kernel.org \
--cc=linux-pm@vger.kernel.org \
--cc=neil@brown.name \
--cc=netdev@vger.kernel.org \
--cc=okorniev@redhat.com \
--cc=rafael@kernel.org \
--cc=snitzer@kernel.org \
--cc=tom@talpey.com \
--cc=trondmy@kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®