From: Jeff Layton <jlayton@kernel.org>
To: Chuck Lever <cel@kernel.org>, NeilBrown <neil@brown.name>,
Olga Kornievskaia <okorniev@redhat.com>,
Dai Ngo <Dai.Ngo@oracle.com>, Tom Talpey <tom@talpey.com>,
Trond Myklebust <trondmy@kernel.org>,
Anna Schumaker <anna@kernel.org>,
"David S. Miller" <davem@davemloft.net>,
Eric Dumazet <edumazet@google.com>,
Jakub Kicinski <kuba@kernel.org>,
Paolo Abeni <pabeni@redhat.com>, Simon Horman <horms@kernel.org>,
"J. Bruce Fields" <bfields@fieldses.org>,
Shuah Khan <shuah@kernel.org>
Cc: linux-nfs@vger.kernel.org, linux-kernel@vger.kernel.org,
netdev@vger.kernel.org, Trond Myklebust <trondmy@gmail.com>,
linux-kselftest@vger.kernel.org,
Jeff Layton <jlayton@kernel.org>
Subject: [PATCH v2 0/8] nfsd/sunrpc: harden the netlink listener interfaces
Date: Tue, 11 Aug 2026 08:02:59 -0400 [thread overview]
Message-ID: <20260811-nfsd-nl-hang-v2-0-c0c92b3953c3@kernel.org> (raw)
This version fixes a number of problems that Sashiko pointed out. The
more serious ones were preexisting issues, but we might as well fix
them while we're in the area. As before, the real way to address the
issues that syzbot keeps reporting is to make the rpcbind calls run
asynchronously.
syzbot keeps landing in nfsd_nl_listener_set_doit(), where a stall under
nfsd_mutex blocks every other NFSD netlink op. This is hardening rather than
a fix for any one report: it narrows what userland can push into that path
and shortens the worst stalls.
1: reject transport names NFSD cannot instantiate, before nfsd_mutex is
taken
2: cap a listener_set request at 1024 entries
3: stop svc_register() losing a registration error to a later program
4: undo the registrations svc_register() made before it failed
5: bound the local rpcbind client to a single 1s attempt
6: report listener creation failures through extack
7: listener_set validation tests
8: a per-netns rpcbind stub, and the listener round-trip tests
Measured against a local rpcbind that accepts the connection and never
replies. The wait is paid per listener, since svc_xprt_create_from_sa()
passes flags of 0 and every listener therefore calls svc_register():
per rpcbind call 10s AF_LOCAL, 60s loopback TCP -> 1s
per listener 20s / 120s -> 2s
entries/request bounded only by message size -> 1024
worst request unbounded -> ~34min
Three things this does not do:
- "rdma" is still accepted, so 1024 entries can still mean 1024
request_module("svcrdma") upcalls under nfsd_mutex where svcrdma is
unavailable. Not counted above.
- write_ports() reaches the same code with the same mutex held. It is
legacy, so it is left alone.
- ~34min is still ~17x the hung-task threshold, so the reproducer should be
expected to keep tripping the watchdog. The durable fix is to make rpcbind
registration asynchronous so those RPCs stop running under nfsd_mutex at
all. That needs behavioural changes we should discuss first, so it is a
separate patchset.
Patch 3 is a flag day for CONFIG_NFS_LOCALIO=y: a registration failure now
aborts listener creation there too, matching CONFIG_NFS_LOCALIO=n. Details
in that patch.
Please consider these for v7.4.
To: Chuck Lever <cel@kernel.org>
To: NeilBrown <neil@brown.name>
To: Olga Kornievskaia <okorniev@redhat.com>
To: Dai Ngo <Dai.Ngo@oracle.com>
To: Tom Talpey <tom@talpey.com>
To: Trond Myklebust <trondmy@kernel.org>
To: Anna Schumaker <anna@kernel.org>
To: David S. Miller <davem@davemloft.net>
To: Eric Dumazet <edumazet@google.com>
To: Jakub Kicinski <kuba@kernel.org>
To: Paolo Abeni <pabeni@redhat.com>
To: Simon Horman <horms@kernel.org>
To: J. Bruce Fields <bfields@fieldses.org>
To: Shuah Khan <shuah@kernel.org>
Cc: linux-nfs@vger.kernel.org
Cc: linux-kernel@vger.kernel.org
Cc: netdev@vger.kernel.org
Cc: Trond Myklebust <trondmy@gmail.com>
Cc: linux-kselftest@vger.kernel.org
Signed-off-by: Jeff Layton <jlayton@kernel.org>
--- Changes in v2:
- New patch 4: svc_register() left the entries it had already set in
rpcbind when a later one failed, pointing at a port the caller then
closed.
- Tests: Behavioral fixes for several tests: several assertions only
checked an errno that both the fixed and the broken kernel return.
val_bad_transport() now requires that the rpcbind stub saw no traffic,
val_second_entry_bad() that no listener came up, func_empty_destroys
that the local rpcbind client was dropped and had to reconnect, and the
two -EBUSY tests that the listener set is unchanged. find_listener()
matches the address too.
- Tests: the stub read the revents of a freshly accepted pollfd that
poll() had not written, so it could enter a blocking read with no
readiness event.
- Tests: the config fragment gained NAMESPACES, SHMEM, TMPFS and UNIX;
without them every test skipped.
- Link to v1: https://lore.kernel.org/r/20260810-nfsd-nl-hang-v1-0-2519fdd5bc1a@kernel.org
---
Jeff Layton (8):
NFSD: validate transport name in listener_set before serv creation
NFSD: cap the number of listeners accepted in listener_set
SUNRPC: keep the first error in svc_register()
SUNRPC: undo partial rpcbind registrations when svc_register() fails
SUNRPC: bound the local rpcbind client timeout to 1s
NFSD: report listener creation failures through extack
selftests/nfsd: exercise listener_set request validation
selftests/nfsd: add a per-netns rpcbind stub and the listener round-trips
fs/nfsd/nfsctl.c | 42 +-
net/sunrpc/rpcb_clnt.c | 12 +
net/sunrpc/svc.c | 40 +-
tools/testing/selftests/Makefile | 1 +
tools/testing/selftests/nfsd/.gitignore | 1 +
tools/testing/selftests/nfsd/Makefile | 6 +
tools/testing/selftests/nfsd/config | 8 +
.../testing/selftests/nfsd/nfsd_netlink_listener.c | 1030 ++++++++++++++++++++
tools/testing/selftests/nfsd/settings | 1 +
9 files changed, 1131 insertions(+), 10 deletions(-)
---
base-commit: 0b6d2c7e3abca8d17fddeecb6e4c32a8438ec2fb
change-id: 20260717-nfsd-nl-hang-10a3b3e93f2a
Best regards,
--
Jeff Layton <jlayton@kernel.org>
next reply other threads:[~2026-08-11 12:03 UTC|newest]
Thread overview: 15+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-08-11 12:02 Jeff Layton [this message]
2026-08-11 12:03 ` [PATCH v2 1/8] NFSD: validate transport name in listener_set before serv creation Jeff Layton
2026-08-11 12:03 ` [PATCH v2 2/8] NFSD: cap the number of listeners accepted in listener_set Jeff Layton
2026-08-11 12:03 ` [PATCH v2 3/8] SUNRPC: keep the first error in svc_register() Jeff Layton
2026-08-11 12:03 ` [PATCH v2 4/8] SUNRPC: undo partial rpcbind registrations when svc_register() fails Jeff Layton
2026-08-11 19:19 ` Chuck Lever
2026-08-12 19:38 ` Jeff Layton
2026-08-12 19:57 ` Chuck Lever
2026-08-14 11:01 ` Jeff Layton
2026-08-14 15:28 ` Chuck Lever
2026-08-11 12:03 ` [PATCH v2 5/8] SUNRPC: bound the local rpcbind client timeout to 1s Jeff Layton
2026-08-11 12:03 ` [PATCH v2 6/8] NFSD: report listener creation failures through extack Jeff Layton
2026-08-11 12:03 ` [PATCH v2 7/8] selftests/nfsd: exercise listener_set request validation Jeff Layton
2026-08-11 12:03 ` [PATCH v2 8/8] selftests/nfsd: add a per-netns rpcbind stub and the listener round-trips Jeff Layton
2026-08-11 19:14 ` Chuck Lever
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20260811-nfsd-nl-hang-v2-0-c0c92b3953c3@kernel.org \
--to=jlayton@kernel.org \
--cc=Dai.Ngo@oracle.com \
--cc=anna@kernel.org \
--cc=bfields@fieldses.org \
--cc=cel@kernel.org \
--cc=davem@davemloft.net \
--cc=edumazet@google.com \
--cc=horms@kernel.org \
--cc=kuba@kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-kselftest@vger.kernel.org \
--cc=linux-nfs@vger.kernel.org \
--cc=neil@brown.name \
--cc=netdev@vger.kernel.org \
--cc=okorniev@redhat.com \
--cc=pabeni@redhat.com \
--cc=shuah@kernel.org \
--cc=tom@talpey.com \
--cc=trondmy@gmail.com \
--cc=trondmy@kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®