[PATCH] nfsd: shut down running servers in nfsd_net_exit()

[PATCH] nfsd: shut down running servers in nfsd_net_exit()

[Shuangpeng Bai]

Since commit 924f4fb003ba ("NFSD: convert write_threads to netlink
command") the server can be started over netlink without mounting the
nfsd filesystem. But that mount was the only thing pinning the module
while threads run (commit f49169c97fce ("NFSD: Remove
svc_serv_ops::svo_module") removed the per-thread reference), and
nfsd_umount() is the only caller of nfsd_shutdown_threads().

So when the server has been started via netlink, a plain "rmmod nfsd"
succeeds while the threads are still running. exit_nfsd() then frees
struct nfsd_net in unregister_pernet_subsys(), destroys the DRC slab
and finally unmaps the module text underneath the running threads:

  BUG: KASAN: slab-use-after-free in nfsd_file_net_dispose (fs/nfsd/filecache.c:483) nfsd
  Read of size 8 at addr ffff888109fe92d0 by task nfsd/300
  Call Trace:
   nfsd_file_net_dispose (fs/nfsd/filecache.c:483) nfsd
   nfsd (fs/nfsd/nfssvc.c:957) nfsd
   kthread (kernel/kthread.c:436)
   ret_from_fork (arch/x86/kernel/process.c:158)
  Freed by task 311:
   kfree (mm/slub.c:6802)
   ops_undo_list (net/core/net_namespace.c:257)
   unregister_pernet_operations (net/core/net_namespace.c:1412)
   unregister_pernet_subsys (net/core/net_namespace.c:1459)
   cleanup_module nfsd
   __se_sys_delete_module (kernel/module/main.c:804)

followed by a general protection fault in nfsd_cache_lookup() as soon
as a thread allocates from the destroyed drc_slab.

Shut down a server that is still running for a net namespace in
nfsd_net_exit(), which is invoked for every net namespace when the
module is unloaded. nfsd_shutdown_threads() is a no-op when no server
exists, so nothing changes for the common case. As the laundromat
work is queued on laundry_wq until the server is shut down, destroy
that workqueue after unregister_pernet_subsys().

To reproduce: start the server with nfsdctl (or any other netlink
client) without mounting the nfsd filesystem, then run "rmmod nfsd".

Fixes: 924f4fb003ba ("NFSD: convert write_threads to netlink command")
Cc: stable@vger.kernel.org # v6.10+
Signed-off-by: Shuangpeng Bai <shuangpeng.kernel@gmail.com>
---
Tested on top of 7.3-rc7 with KASAN and with PROVE_LOCKING: with the
server started over netlink and an NFSv3 request in flight, "rmmod nfsd"
now waits for the request to complete and unloads cleanly; starting and
stopping the server via netlink and via the nfsd filesystem still works.

An alternative would be to take a module reference while a server
exists so that rmmod fails with EBUSY, as it does while the nfsd
filesystem is mounted. Stopping the server in nfsd_net_exit() seemed
simpler and mirrors what nfsd_umount() does, but I am happy to rework
it if the other behaviour is preferred.
 fs/nfsd/nfsctl.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/fs/nfsd/nfsctl.c b/fs/nfsd/nfsctl.c
index f32311f2d7cf..0a9679114b58 100644
--- a/fs/nfsd/nfsctl.c
+++ b/fs/nfsd/nfsctl.c
@@ -3113,6 +3113,7 @@ static __net_exit void nfsd_net_exit(struct net *net)
 {
 	struct nfsd_net *nn = net_generic(net, nfsd_net_id);
 
+	nfsd_shutdown_threads(net);
 	kfree_sensitive(nn->fh_key);
 	nfsd_net_cb_shutdown(nn);
 	nfsd_proc_stat_shutdown(net);
@@ -3200,9 +3201,9 @@ static void __exit exit_nfsd(void)
 	remove_proc_entry("fs/nfs", NULL);
 	genl_unregister_family(&nfsd_nl_family);
 	unregister_filesystem(&nfsd_fs_type);
-	nfsd4_destroy_laundry_wq();
 	unregister_cld_notifier();
 	unregister_pernet_subsys(&nfsd_net_ops);
+	nfsd4_destroy_laundry_wq();
 	nfsd_drc_slab_free();
 	nfsd_lockd_shutdown();
 	nfsd4_exit_pnfs();
-- 
2.43.0

Re: [PATCH] nfsd: shut down running servers in nfsd_net_exit()

[Jeff Layton]

On Mon, 2026-10-26 at 19:56 -0400, Shuangpeng Bai wrote:
> Since commit 924f4fb003ba ("NFSD: convert write_threads to netlink
> command") the server can be started over netlink without mounting the
> nfsd filesystem. But that mount was the only thing pinning the module
> while threads run (commit f49169c97fce ("NFSD: Remove
> svc_serv_ops::svo_module") removed the per-thread reference), and
> nfsd_umount() is the only caller of nfsd_shutdown_threads().
> 
> So when the server has been started via netlink, a plain "rmmod nfsd"
> succeeds while the threads are still running. exit_nfsd() then frees
> struct nfsd_net in unregister_pernet_subsys(), destroys the DRC slab
> and finally unmaps the module text underneath the running threads:
> 
>   BUG: KASAN: slab-use-after-free in nfsd_file_net_dispose (fs/nfsd/filecache.c:483) nfsd
>   Read of size 8 at addr ffff888109fe92d0 by task nfsd/300
>   Call Trace:
>    nfsd_file_net_dispose (fs/nfsd/filecache.c:483) nfsd
>    nfsd (fs/nfsd/nfssvc.c:957) nfsd
>    kthread (kernel/kthread.c:436)
>    ret_from_fork (arch/x86/kernel/process.c:158)
>   Freed by task 311:
>    kfree (mm/slub.c:6802)
>    ops_undo_list (net/core/net_namespace.c:257)
>    unregister_pernet_operations (net/core/net_namespace.c:1412)
>    unregister_pernet_subsys (net/core/net_namespace.c:1459)
>    cleanup_module nfsd
>    __se_sys_delete_module (kernel/module/main.c:804)
> 
> followed by a general protection fault in nfsd_cache_lookup() as soon
> as a thread allocates from the destroyed drc_slab.
> 
> Shut down a server that is still running for a net namespace in
> nfsd_net_exit(), which is invoked for every net namespace when the
> module is unloaded. nfsd_shutdown_threads() is a no-op when no server
> exists, so nothing changes for the common case. As the laundromat
> work is queued on laundry_wq until the server is shut down, destroy
> that workqueue after unregister_pernet_subsys().
> 
> To reproduce: start the server with nfsdctl (or any other netlink
> client) without mounting the nfsd filesystem, then run "rmmod nfsd".
> 
> Fixes: 924f4fb003ba ("NFSD: convert write_threads to netlink command")
> Cc: stable@vger.kernel.org # v6.10+
> Signed-off-by: Shuangpeng Bai <shuangpeng.kernel@gmail.com>
> ---
> Tested on top of 7.3-rc7 with KASAN and with PROVE_LOCKING: with the
> server started over netlink and an NFSv3 request in flight, "rmmod nfsd"
> now waits for the request to complete and unloads cleanly; starting and
> stopping the server via netlink and via the nfsd filesystem still works.
> 
> An alternative would be to take a module reference while a server
> exists so that rmmod fails with EBUSY, as it does while the nfsd
> filesystem is mounted. Stopping the server in nfsd_net_exit() seemed
> simpler and mirrors what nfsd_umount() does, but I am happy to rework
> it if the other behaviour is preferred.
>  fs/nfsd/nfsctl.c | 3 ++-
>  1 file changed, 2 insertions(+), 1 deletion(-)
> 
> diff --git a/fs/nfsd/nfsctl.c b/fs/nfsd/nfsctl.c
> index f32311f2d7cf..0a9679114b58 100644
> --- a/fs/nfsd/nfsctl.c
> +++ b/fs/nfsd/nfsctl.c
> @@ -3113,6 +3113,7 @@ static __net_exit void nfsd_net_exit(struct net *net)
>  {
>  	struct nfsd_net *nn = net_generic(net, nfsd_net_id);
>  
> +	nfsd_shutdown_threads(net);
>  	kfree_sensitive(nn->fh_key);
>  	nfsd_net_cb_shutdown(nn);
>  	nfsd_proc_stat_shutdown(net);
> @@ -3200,9 +3201,9 @@ static void __exit exit_nfsd(void)
>  	remove_proc_entry("fs/nfs", NULL);
>  	genl_unregister_family(&nfsd_nl_family);
>  	unregister_filesystem(&nfsd_fs_type);
> -	nfsd4_destroy_laundry_wq();
>  	unregister_cld_notifier();
>  	unregister_pernet_subsys(&nfsd_net_ops);
> +	nfsd4_destroy_laundry_wq();
>  	nfsd_drc_slab_free();
>  	nfsd_lockd_shutdown();
>  	nfsd4_exit_pnfs();


I think I'm OK with either behavior (shutting down threads on rmmod or
taking a reference), but sashiko has some comments on this patch that
should be addressed:

https://sashiko.dev/#/patchset/20261026235626.649844-1-shuangpeng.kernel%40gmail.com

-- 
Jeff Layton <jlayton@kernel.org>

Re: [PATCH] nfsd: shut down running servers in nfsd_net_exit()

[Shuangpeng]

Thanks for the review.

I agree with the comment about the __module_get() users: stopping the
server from nfsd_net_exit() only covers nfsd's own threads, while
fs/locks.c (via lm_mod_owner) and nfs_common/nfslocalio.c take
__module_get(THIS_MODULE) on the assumption that an active server
keeps the module alive, and nothing stops free_module() once
exit_nfsd() has started.

So v2 takes a module reference while a server exists instead, which
makes "rmmod nfsd" fail with EBUSY as it does while the nfsd
filesystem is mounted. The laundry_wq and cld notifier ordering
concerns go away with that, since exit_nfsd() can no longer run while
a server exists.

The remaining point, an exiting nfsd thread still running a few
instructions of module code after svc_stop_kthreads() has been woken
up, is a pre-existing window on the umount path as well and is the
same with or without this patch, so I left it out of scope.

I'll send v2 shortly.


> On Oct 27, 2026, at 06:48, Jeff Layton <jlayton@kernel.org> wrote:
> 
> On Mon, 2026-10-26 at 19:56 -0400, Shuangpeng Bai wrote:
>> Since commit 924f4fb003ba ("NFSD: convert write_threads to netlink
>> command") the server can be started over netlink without mounting the
>> nfsd filesystem. But that mount was the only thing pinning the module
>> while threads run (commit f49169c97fce ("NFSD: Remove
>> svc_serv_ops::svo_module") removed the per-thread reference), and
>> nfsd_umount() is the only caller of nfsd_shutdown_threads().
>> 
>> So when the server has been started via netlink, a plain "rmmod nfsd"
>> succeeds while the threads are still running. exit_nfsd() then frees
>> struct nfsd_net in unregister_pernet_subsys(), destroys the DRC slab
>> and finally unmaps the module text underneath the running threads:
>> 
>>  BUG: KASAN: slab-use-after-free in nfsd_file_net_dispose (fs/nfsd/filecache.c:483) nfsd
>>  Read of size 8 at addr ffff888109fe92d0 by task nfsd/300
>>  Call Trace:
>>   nfsd_file_net_dispose (fs/nfsd/filecache.c:483) nfsd
>>   nfsd (fs/nfsd/nfssvc.c:957) nfsd
>>   kthread (kernel/kthread.c:436)
>>   ret_from_fork (arch/x86/kernel/process.c:158)
>>  Freed by task 311:
>>   kfree (mm/slub.c:6802)
>>   ops_undo_list (net/core/net_namespace.c:257)
>>   unregister_pernet_operations (net/core/net_namespace.c:1412)
>>   unregister_pernet_subsys (net/core/net_namespace.c:1459)
>>   cleanup_module nfsd
>>   __se_sys_delete_module (kernel/module/main.c:804)
>> 
>> followed by a general protection fault in nfsd_cache_lookup() as soon
>> as a thread allocates from the destroyed drc_slab.
>> 
>> Shut down a server that is still running for a net namespace in
>> nfsd_net_exit(), which is invoked for every net namespace when the
>> module is unloaded. nfsd_shutdown_threads() is a no-op when no server
>> exists, so nothing changes for the common case. As the laundromat
>> work is queued on laundry_wq until the server is shut down, destroy
>> that workqueue after unregister_pernet_subsys().
>> 
>> To reproduce: start the server with nfsdctl (or any other netlink
>> client) without mounting the nfsd filesystem, then run "rmmod nfsd".
>> 
>> Fixes: 924f4fb003ba ("NFSD: convert write_threads to netlink command")
>> Cc: stable@vger.kernel.org # v6.10+
>> Signed-off-by: Shuangpeng Bai <shuangpeng.kernel@gmail.com>
>> ---
>> Tested on top of 7.3-rc7 with KASAN and with PROVE_LOCKING: with the
>> server started over netlink and an NFSv3 request in flight, "rmmod nfsd"
>> now waits for the request to complete and unloads cleanly; starting and
>> stopping the server via netlink and via the nfsd filesystem still works.
>> 
>> An alternative would be to take a module reference while a server
>> exists so that rmmod fails with EBUSY, as it does while the nfsd
>> filesystem is mounted. Stopping the server in nfsd_net_exit() seemed
>> simpler and mirrors what nfsd_umount() does, but I am happy to rework
>> it if the other behaviour is preferred.
>> fs/nfsd/nfsctl.c | 3 ++-
>> 1 file changed, 2 insertions(+), 1 deletion(-)
>> 
>> diff --git a/fs/nfsd/nfsctl.c b/fs/nfsd/nfsctl.c
>> index f32311f2d7cf..0a9679114b58 100644
>> --- a/fs/nfsd/nfsctl.c
>> +++ b/fs/nfsd/nfsctl.c
>> @@ -3113,6 +3113,7 @@ static __net_exit void nfsd_net_exit(struct net *net)
>> {
>> struct nfsd_net *nn = net_generic(net, nfsd_net_id);
>> 
>> + nfsd_shutdown_threads(net);
>> kfree_sensitive(nn->fh_key);
>> nfsd_net_cb_shutdown(nn);
>> nfsd_proc_stat_shutdown(net);
>> @@ -3200,9 +3201,9 @@ static void __exit exit_nfsd(void)
>> remove_proc_entry("fs/nfs", NULL);
>> genl_unregister_family(&nfsd_nl_family);
>> unregister_filesystem(&nfsd_fs_type);
>> - nfsd4_destroy_laundry_wq();
>> unregister_cld_notifier();
>> unregister_pernet_subsys(&nfsd_net_ops);
>> + nfsd4_destroy_laundry_wq();
>> nfsd_drc_slab_free();
>> nfsd_lockd_shutdown();
>> nfsd4_exit_pnfs();
> 
> 
> I think I'm OK with either behavior (shutting down threads on rmmod or
> taking a reference), but sashiko has some comments on this patch that
> should be addressed:
> 
> https://sashiko.dev/#/patchset/20261026235626.649844-1-shuangpeng.kernel%40gmail.com
> 
> -- 
> Jeff Layton <jlayton@kernel.org>

[PATCH v2] nfsd: pin the module while a server exists

[Shuangpeng Bai]

Since commit 924f4fb003ba ("NFSD: convert write_threads to netlink
command") the server can be started over netlink without mounting the
nfsd filesystem. But that mount was the only thing pinning the module
while a server exists (commit f49169c97fce ("NFSD: Remove
svc_serv_ops::svo_module") removed the per-thread reference).

So when the server has been started via netlink, a plain "rmmod nfsd"
succeeds while the threads are still running. exit_nfsd() then frees
struct nfsd_net in unregister_pernet_subsys(), destroys the DRC slab
and finally unmaps the module text underneath the running threads:

  BUG: KASAN: slab-use-after-free in nfsd_file_net_dispose (fs/nfsd/filecache.c:483) nfsd
  Read of size 8 at addr ffff888109fe92d0 by task nfsd/300
  Call Trace:
   nfsd_file_net_dispose (fs/nfsd/filecache.c:483) nfsd
   nfsd (fs/nfsd/nfssvc.c:957) nfsd
   kthread (kernel/kthread.c:436)
   ret_from_fork (arch/x86/kernel/process.c:158)
  Freed by task 311:
   kfree (mm/slub.c:6802)
   ops_undo_list (net/core/net_namespace.c:257)
   unregister_pernet_operations (net/core/net_namespace.c:1412)
   unregister_pernet_subsys (net/core/net_namespace.c:1459)
   cleanup_module nfsd
   __se_sys_delete_module (kernel/module/main.c:804)

followed by a general protection fault in nfsd_cache_lookup() as soon
as a thread allocates from the destroyed drc_slab.

Take a module reference in nfsd_create_serv() and drop it in
nfsd_destroy_serv(), so that "rmmod nfsd" fails with EBUSY while a
server exists, just as it does while the nfsd filesystem is mounted.

Use try_module_get() rather than __module_get(): a netlink command
runs without a reference on the nfsd module, and exit_nfsd() only
waits for it in genl_unregister_family(). A THREADS_SET or
LISTENER_SET request that is already running when "rmmod nfsd" starts
would otherwise still create a server after module unload has begun.
With try_module_get() that request fails with ENODEV instead.

Pinning the module also keeps the assumption behind the
__module_get() calls in fs/locks.c (via lm_mod_owner) and in
nfs_common/nfslocalio.c valid: they rely on an active server keeping
the module alive.

To reproduce: start the server with nfsdctl (or any other netlink
client) without mounting the nfsd filesystem, then run "rmmod nfsd".

Fixes: 924f4fb003ba ("NFSD: convert write_threads to netlink command")
Cc: stable@vger.kernel.org # v6.10+
Signed-off-by: Shuangpeng Bai <shuangpeng.kernel@gmail.com>
---
Changes since v1:
- Pin the module while a server exists instead of shutting the server
  down in nfsd_net_exit(). As the sashiko review pointed out, shutting
  it down from exit_nfsd() still lets every __module_get(THIS_MODULE)
  taken during module exit (fs/locks.c via lm_mod_owner, nfslocalio)
  outlive free_module(). With the reference held, exit_nfsd() never
  runs while a server exists, so the laundry_wq / cld notifier ordering
  changes from v1 are not needed either.
- Take the reference with try_module_get(). A netlink command holds no
  reference on nfsd, so with __module_get() a LISTENER_SET/THREADS_SET
  that was already running when "rmmod nfsd" started could still bring
  up a server, and its threads then ran on the freed module text.
- Not changed: an exiting nfsd thread still executes a few instructions
  of module code after svc_stop_kthreads() has been woken up. That
  window exists on the umount path as well and is independent of this
  fix.

v1: https://lore.kernel.org/linux-nfs/20261026235626.649844-1-shuangpeng.kernel@gmail.com/

Tested on top of 7.3-rc7 with KASAN: with the server started over
netlink, "rmmod nfsd" now fails with EBUSY, also while a request is in
flight and with only a listener configured; after setting the thread
count to 0 the module unloads cleanly; starting and stopping the server
through the nfsd filesystem is unchanged. With a delay injected at the
top of nfsd_create_serv() to hold a netlink start open across "rmmod
nfsd", the start now fails with ENODEV and the module unloads cleanly
(with __module_get() the threads came up and oopsed in freed module
text).
 fs/nfsd/nfssvc.c | 20 +++++++++++++++++---
 1 file changed, 17 insertions(+), 3 deletions(-)

diff --git a/fs/nfsd/nfssvc.c b/fs/nfsd/nfssvc.c
index 067543da3ca0..6ae040c50987 100644
--- a/fs/nfsd/nfssvc.c
+++ b/fs/nfsd/nfssvc.c
@@ -579,6 +579,7 @@ void nfsd_destroy_serv(struct net *net)
 	svc_xprt_destroy_all(serv, net, true);
 	nfsd_shutdown_net(net);
 	svc_destroy(&serv);
+	module_put(THIS_MODULE);
 }
 
 void nfsd_reset_versions(struct nfsd_net *nn)
@@ -662,10 +663,18 @@ int nfsd_create_serv(struct net *net, bool no_rpcbind)
 	if (nn->nfsd_serv)
 		return 0;
 
+	/*
+	 * Keep the module loaded while the server exists, as a mounted
+	 * nfsd filesystem does. A netlink caller holds no module
+	 * reference, so this fails if module unload has already begun.
+	 */
+	if (!try_module_get(THIS_MODULE))
+		return -ENODEV;
+
 	error = percpu_ref_init(&nn->nfsd_net_ref, nfsd_net_free,
 				0, GFP_KERNEL);
 	if (error)
-		return error;
+		goto out_put;
 	init_completion(&nn->nfsd_net_free_done);
 	init_completion(&nn->nfsd_net_confirm_done);
 
@@ -677,7 +686,8 @@ int nfsd_create_serv(struct net *net, bool no_rpcbind)
 				 nn->max_blksize, nfsd);
 	if (serv == NULL) {
 		percpu_ref_exit(&nn->nfsd_net_ref);
-		return -ENOMEM;
+		error = -ENOMEM;
+		goto out_put;
 	}
 	serv->sv_tls = true;
 
@@ -688,7 +698,7 @@ int nfsd_create_serv(struct net *net, bool no_rpcbind)
 	if (error < 0) {
 		svc_destroy(&serv);
 		percpu_ref_exit(&nn->nfsd_net_ref);
-		return error;
+		goto out_put;
 	}
 	/*
 	 * Register before publishing nn->nfsd_serv.  Namespaces are only
@@ -704,6 +714,10 @@ int nfsd_create_serv(struct net *net, bool no_rpcbind)
 
 	nfsd_reset_write_verifier(nn);
 	return 0;
+
+out_put:
+	module_put(THIS_MODULE);
+	return error;
 }
 
 int nfsd_nrpools(struct net *net)
-- 
2.43.0