[PATCH] NFSv4: avoid an rpciod deadlock when an async RPC frees the nfs_client

[Yue Wang <wangnyue@amazon.com>]

Unmounting many NFSv4 mounts while their server is unreachable can hang
every NFS mount on the machine for good. The rpciod workers of one NUMA
node are stuck in:

  rpc_wait_bit_killable <- rpc_call_sync <- nfs4_proc_destroy_session
  <- nfs41_shutdown_client <- nfs4_free_client <- nfs41_sequence_release
  <- rpc_free_task <- __rpc_execute <- rpc_async_schedule

If an async lease renewal (SEQUENCE, or RENEW for NFSv4.0) or an async
FREE_STATEID holds the last reference to the nfs_client, its release
callback frees the client and sends DESTROY_SESSION and DESTROY_CLIENTID
with rpc_call_sync(). The release runs inline in the rpciod worker. With
the server unreachable, the RPC waits on the delay queue, whose timer has
been rpciod work since commit 7e0a0e38fcfe ("SUNRPC: Replace the queue
timer with a delayed work function"). Once one node's rpciod max_active
share is full of such workers (171 on a 384-CPU, 6-node machine), the
timer can no longer run, so none of them ever wakes up. The delay queue
is global, so retries and reconnects of all other mounts stop too.

Release these tasks from nfsiod, as other NFSv4 async operations with
sleeping release callbacks already do. Only the release callback moves;
the RPC and its wake-ups stay on rpciod.

The cost: nfsiod also completes READ, WRITE and COMMIT tasks. If enough
teardowns pile up to fill a node's nfsiod share, I/O completion on that
node waits until they time out, about 6 minutes with the default
timeouts, instead of hanging forever.

Tested with 1500 NFSv4.1 mounts unmounted while their renewals were
stalled, then made unreachable: without the patch, 1024 rpciod workers
stay blocked and 477 clients are never freed; with it, no rpciod worker
blocks and all clients are freed within 320 s.

Fixes: 7e0a0e38fcfe ("SUNRPC: Replace the queue timer with a delayed work function")
Cc: stable@vger.kernel.org
Signed-off-by: Yue Wang <wangnyue@amazon.com>
---
 fs/nfs/nfs40proc.c | 17 +++++++++++++++--
 fs/nfs/nfs4proc.c  |  4 ++++
 2 files changed, 19 insertions(+), 2 deletions(-)

diff --git a/fs/nfs/nfs40proc.c b/fs/nfs/nfs40proc.c
index b07f2d6ccde4..2c3ecd9289f4 100644
--- a/fs/nfs/nfs40proc.c
+++ b/fs/nfs/nfs40proc.c
@@ -116,7 +116,16 @@ static int nfs4_proc_async_renew(struct nfs_client *clp, const struct cred *cred
 		.rpc_argp	= clp,
 		.rpc_cred	= cred,
 	};
+	/* The release may free @clp synchronously: keep it off rpciod. */
+	struct rpc_task_setup task_setup_data = {
+		.rpc_client = clp->cl_rpcclient,
+		.rpc_message = &msg,
+		.callback_ops = &nfs4_renew_ops,
+		.workqueue = nfsiod_workqueue,
+		.flags = RPC_TASK_ASYNC | RPC_TASK_TIMEOUT,
+	};
 	struct nfs4_renewdata *data;
+	struct rpc_task *task;
 
 	if (renew_flags == 0)
 		return 0;
@@ -129,8 +138,12 @@ static int nfs4_proc_async_renew(struct nfs_client *clp, const struct cred *cred
 	}
 	data->client = clp;
 	data->timestamp = jiffies;
-	return rpc_call_async(clp->cl_rpcclient, &msg, RPC_TASK_TIMEOUT,
-			&nfs4_renew_ops, data);
+	task_setup_data.callback_data = data;
+	task = rpc_run_task(&task_setup_data);
+	if (IS_ERR(task))
+		return PTR_ERR(task);
+	rpc_put_task(task);
+	return 0;
 }
 
 static int nfs4_proc_renew(struct nfs_client *clp, const struct cred *cred)
diff --git a/fs/nfs/nfs4proc.c b/fs/nfs/nfs4proc.c
index e1018b03c1cb..302f891c7a55 100644
--- a/fs/nfs/nfs4proc.c
+++ b/fs/nfs/nfs4proc.c
@@ -9459,6 +9459,8 @@ static struct rpc_task *_nfs41_proc_sequence(struct nfs_client *clp,
 		.rpc_client = clp->cl_rpcclient,
 		.rpc_message = &msg,
 		.callback_ops = &nfs41_sequence_ops,
+		/* The release may free the nfs_client: keep it off rpciod. */
+		.workqueue = nfsiod_workqueue,
 		.flags = RPC_TASK_ASYNC | RPC_TASK_TIMEOUT | RPC_TASK_MOVEABLE,
 	};
 	struct rpc_task *ret;
@@ -10468,6 +10470,8 @@ static int nfs41_free_stateid(struct nfs_server *server,
 		.rpc_client = server->client,
 		.rpc_message = &msg,
 		.callback_ops = &nfs41_free_stateid_ops,
+		/* The release may free the nfs_client: keep it off rpciod. */
+		.workqueue = nfsiod_workqueue,
 		.flags = RPC_TASK_ASYNC | RPC_TASK_MOVEABLE,
 	};
 	struct nfs_free_stateid_data *data;

base-commit: 7e1392a42ea288d70a266f6553fe06f57f8647ea
-- 
2.48.2