[PATCH slab/for-next-fixes] mm/slab: do not wake up kswapd in __kfree_rcu_sheaf()

[PATCH slab/for-next-fixes] mm/slab: do not wake up kswapd in __kfree_rcu_sheaf()

[Harry Yoo (Meta)]

Since kfree_rcu() can be called under pi_lock (a raw spinlock in the
scheduler), kfree_rcu() itself should never allocate memory with
__GFP_KSWAPD_RECLAIM as waking up kswapd ends up acquiring pi_lock,
which leads to a deadlock.

Reproducing the issue even intentionally was not straightforward.
The set_cpus_allowed_force() path that is called under pi_lock is
exercised very rarely, and competing tasks that allocate memory will
most likely wake kswapd up.

Therefore the existence of the deadlock was verified with a modified
kernel that has a lockdep map for waking up kswapd:

  ======================================================
  WARNING: possible circular locking dependency detected
  7.2.0-rc1-slab-for-next+ #10 Not tainted
  ------------------------------------------------------
  git/6660 is trying to acquire lock:
  ffff8e524533c550 (&p->pi_lock){-.-.}-{2:2}, at: _raw_spin_lock_irqsave+0x12/0x20

  but task is already holding lock:
  ffff8e57bffff1a0 (&pgdat->kswapd_wait){....}-{3:3}, at: _raw_spin_lock_irqsave+0x12/0x20

  which lock already depends on the new lock.

  the existing dependency chain (in reverse order) is:

  -> #2 (&pgdat->kswapd_wait){....}-{3:3}:
         __lock_acquire+0x5a4/0xc50
         lock_acquire.part.0+0xb7/0x240
         lock_acquire+0x70/0x170
         __raw_spin_lock_irqsave+0x44/0x80
         _raw_spin_lock_irqsave+0x12/0x20
         __wake_up_common_lock+0x31/0xa0
         __wake_up+0x20/0x40
         kswapd_wakeup_wake+0x7e/0x110
         wakeup_kswapd+0x295/0x320
         wake_all_kswapds+0xac/0x1a0
         __alloc_pages_slowpath.constprop.0+0x282/0xf80
         __alloc_frozen_pages_noprof+0x32a/0x360
         alloc_slab_page+0x2e/0x160
         allocate_slab+0x82/0x420
         new_slab+0x52/0xb0
         refill_objects+0x13d/0x190
         refill_sheaf+0x5d/0xd0
         __pcs_replace_empty_main+0x230/0xb10
         [...]

  -> #1 (kswapd_wakeup){-.-.}-{0:0}:
         __lock_acquire+0x5a4/0xc50
         lock_sync.part.0+0x75/0x100
         lock_sync+0x36/0x70
         might_wakeup_kswapd+0x61/0xa0
         __kfree_rcu_sheaf+0x33/0xd90
         kvfree_call_rcu+0x1d4/0x3b0
         set_cpus_allowed_force+0x163/0x220
         cpuset_cpus_allowed_fallback+0x18b/0x240
         select_fallback_rq+0x1e6/0x250
	 [...]

  -> #0 (&p->pi_lock){-.-.}-{2:2}:
         check_prev_add+0xe6/0xe00
         validate_chain+0x51e/0x6e0
         __lock_acquire+0x5a4/0xc50
         lock_acquire.part.0+0xb7/0x240
         lock_acquire+0x70/0x170
         __raw_spin_lock_irqsave+0x44/0x80
         _raw_spin_lock_irqsave+0x12/0x20
         try_to_wake_up+0x77/0xa90
         default_wake_function+0x27/0x60
         autoremove_wake_function+0x23/0xb0
         __wake_up_common+0xb8/0x170
         __wake_up_common_lock+0x51/0xa0
         __wake_up+0x20/0x40
         kswapd_wakeup_wake+0x7e/0x110
         wakeup_kswapd+0x295/0x320
         wake_all_kswapds+0xac/0x1a0
         __alloc_pages_slowpath.constprop.0+0x282/0xf80
         __alloc_frozen_pages_noprof+0x32a/0x360
         alloc_slab_page+0x2e/0x160
         allocate_slab+0x82/0x420
         new_slab+0x52/0xb0
         refill_objects+0x13d/0x190
         refill_sheaf+0x5d/0xd0
         __pcs_replace_empty_main+0x230/0xb10
         [...]

  other info that might help us debug this:

  Chain exists of:
    &p->pi_lock --> kswapd_wakeup --> &pgdat->kswapd_wait

   Possible unsafe locking scenario:

         CPU0                    CPU1
         ----                    ----
    lock(&pgdat->kswapd_wait);
                                 lock(kswapd_wakeup);
                                 lock(&pgdat->kswapd_wait);
    lock(&p->pi_lock);

   *** DEADLOCK ***

  3 locks held by git/6660:
   #0: ffff8e519f0e9e20 (&type->i_mutex_dir_key#6){++++}-{4:4}, at: lookup_slow+0x2d/0x60
   #1: ffffffff8a31c460 (kswapd_wakeup){-.-.}-{0:0}, at: kswapd_wakeup_wake+0x4d/0x110
   #2: ffff8e57bffff1a0 (&pgdat->kswapd_wait){....}-{3:3}, at: _raw_spin_lock_irqsave+0x12/0x20

  [...]

Fix this by always avoiding waking up kswapd in __kfree_rcu_sheaf().
Note that there are two paths that might wake up kswapd:

  1) __kfree_rcu_sheaf()
     // __GFP_KSWAPD_RECLAIM might wake up kswapd
     -> alloc_empty_sheaf(GFP_NOWAIT)

  2) __kfree_rcu_sheaf()
     // Let's say __kfree_rcu_sheaf() doesn't pass GFP_NOWAIT
     -> alloc_empty_sheaf(__GFP_NOWARN)
     -> kmalloc_flags()
     -> slab_alloc_node()
     -> alloc_from_pcs()
     -> __pcs_replace_empty_main()
     // Free a sheaf in an allocation path when the sheaf becomes empty
     // and refilling the sheaf fails
     -> free_empty_sheaf()
     -> slab_free()
     -> free_to_pcs()
     -> __pcs_replace_full_main()
     // However free path always assumes it's safe to wake up kswapd
     -> alloc_empty_sheaf(GFP_NOWAIT)

Drop __GFP_KSWAPD_RECLAIM in both cases. Note that the kfree_rcu() is
not the only user of free_to_pcs() path, but it should be fixed as it
can be invoked under pi_lock.

Reported-by: Sashiko <sashiko-bot@kernel.org>
Closes: https://sashiko.dev/#/message/20260831-b4-kfree_rcu_hotfix-v1-1-4f0fb882638b%40kernel.org
Fixes: ec66e0d59952 ("slab: add sheaf support for batching kfree_rcu() operations")
Link: https://lore.kernel.org/linux-mm/20260831143500.x-saxdAs@linutronix.de
Assisted-by: LLM
Signed-off-by: Harry Yoo (Meta) <harry@kernel.org>
---
The discovery and verification (w/ a modified kernel) of the bug was
assisted by LLMs.

More speicifically, the first path was pointed out by Sashiko, and
the second path was discovered by LLM while reviewing the commit with
review-prompts [1].

Harry Yoo reviewed those findings and manually crafted the patch based
on that.

[1] https://github.com/masoncl/review-prompts

I believe the right direction to address this issue is to make
kfree_nolock() work in any context and replace it with kfree_rcu()
in the scheduler. However for now it won't work under pi_lock,
and resolving that would be a longer journey.

Address this issue by dropping __GFP_KSWAPD_RECLAIM, for now.
---
 mm/slub.c | 7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)

diff --git a/mm/slub.c b/mm/slub.c
index 54ec12503357..544cff39762c 100644
--- a/mm/slub.c
+++ b/mm/slub.c
@@ -5969,7 +5969,8 @@ __pcs_replace_full_main(struct kmem_cache *s, struct slub_percpu_sheaves *pcs,
 	if (!allow_spin)
 		return NULL;
 
-	empty = alloc_empty_sheaf(s, GFP_NOWAIT, SLAB_ALLOC_DEFAULT);
+	/* Don't wake up kswapd, it will cause deadlock under pi_lock */
+	empty = alloc_empty_sheaf(s, __GFP_NOWARN, SLAB_ALLOC_DEFAULT);
 	if (empty)
 		goto got_empty;
 
@@ -6128,7 +6129,6 @@ bool __kfree_rcu_sheaf(struct kmem_cache *s, void *obj, unsigned int free_flags)
 		struct slab_sheaf *empty;
 		struct node_barn *barn;
 		unsigned int alloc_flags = to_alloc_flags(free_flags);
-		gfp_t gfp = allow_spin ? GFP_NOWAIT : __GFP_NOWARN;
 
 		/* Bootstrap or debug cache, fall back */
 		if (unlikely(!cache_has_sheaves(s))) {
@@ -6157,7 +6157,8 @@ bool __kfree_rcu_sheaf(struct kmem_cache *s, void *obj, unsigned int free_flags)
 
 		local_unlock(&s->cpu_sheaves->lock);
 
-		empty = alloc_empty_sheaf(s, gfp, alloc_flags);
+		/* Don't wake up kswapd, it will cause deadlock under pi_lock */
+		empty = alloc_empty_sheaf(s, __GFP_NOWARN, alloc_flags);
 
 		if (!empty)
 			goto fail;

---
base-commit: 4ebdb8a6231bde47d21597d5d0a193dcf85164ac
change-id: 20260922-kfree-rcu-dont-wakeup-kswapd-a53c11fcef25

Best regards,
--  
Cheers,
Harry / Hyeonggon

Re: [PATCH slab/for-next-fixes] mm/slab: do not wake up kswapd in __kfree_rcu_sheaf()

[Harry Yoo]

On Tue, Sep 22, 2026 at 12:56:43PM +0100, Harry Yoo (Meta) wrote:
[ ... ] 
> Reported-by: Sashiko <sashiko-bot@kernel.org>
> Closes: https://sashiko.dev/#/message/20260831-b4-kfree_rcu_hotfix-v1-1-4f0fb882638b%40kernel.org

Oh, the link to the message does not have inline sashiko review.
Incorrect Closes: link.

it should be
Closes: https://sashiko.dev/#/patchset/20260831-b4-kfree_rcu_hotfix-v1-1-4f0fb882638b%40kernel.org

-- 
Cheers,
Harry / Hyeonggon