[PATCH] mm/slub: don't let kmemleak report rcu sheaves

[PATCH] mm/slub: don't let kmemleak report rcu sheaves

[Nguyen Ngoc Thang]

kmemleak reports full rcu sheaves as leaked:

  unreferenced object 0xffff88810005f600 (size 512):
    hex dump (first 32 bytes):
      e0 79 f0 2b 81 88 ff ff 20 00 b4 81 ff ff ff ff
      00 12 04 00 81 88 ff ff 3c 00 00 00 00 00 00 00
    backtrace:
      __alloc_empty_sheaf mm/slub.c:2854 [inline]
      alloc_empty_sheaf mm/slub.c:2874 [inline]
      __pcs_replace_empty_main+0x260/0x2e0 mm/slub.c:4806

The second word is rcu_free_sheaf and the size equals the capacity: the
sheaf was handed to call_rcu() by __kfree_rcu_sheaf() and is waiting for
a grace period. At that point the only reference to it is the ->next
pointer of the previous callback in the rcu callback list.

That previous callback may live in memory kmemleak never scans, e.g. the
rcu_head of a struct slab queued by rcu_free_slab() for a
SLAB_TYPESAFE_BY_RCU cache. Slab pages are frozen (page_count() == 0),
so kmemleak skips their struct page. The chain from rcu_data is cut and
the sheaf, typically allocated long ago and recycled, looks unreferenced.
Since the sheaf is then not scanned either, the objects it is about to
free can be reported as well.

Sheaves are allocator metadata and are always owned by a pcs, a barn or
an rcu callback. Mark them with kmemleak_not_leak(): they are never
reported but still scanned, so objects queued in them stay referenced.
Allocations that can't spin are not tracked by kmemleak, so skip them,
as alloc_slab_obj_exts() does.

Fixes: ec66e0d59952 ("slab: add sheaf support for batching kfree_rcu() operations")
Reported-by: syzbot+527fed49e5b988882c9a@syzkaller.appspotmail.com
Closes: https://syzkaller.appspot.com/bug?extid=527fed49e5b988882c9a
Signed-off-by: Nguyen Ngoc Thang <ngocthang2710.1999@gmail.com>
---
Reproduced in QEMU (syzbot config, 2 vCPUs, numa=fake=2) by stalling a
grace period with a preempted reader on CPU0 while CPU1 interleaves
rcu_free_slab() (filp slabs) and rcu_free_sheaf() (maple nodes freed by
kfree_rcu()) callbacks, then running two kmemleak scans:

  before: 16-18 sheaves reported per run, rcu_head.func == rcu_free_sheaf,
          plus maple nodes held by them
  after:  no sheaf or maple node reports, no new warnings

 mm/slub.c | 8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/mm/slub.c b/mm/slub.c
index cd671b934719..e01416ae0b4a 100644
--- a/mm/slub.c
+++ b/mm/slub.c
@@ -2856,6 +2856,14 @@ static struct slab_sheaf *__alloc_empty_sheaf(struct kmem_cache *s, gfp_t gfp,
 	if (unlikely(!sheaf))
 		return NULL;
 
+	/*
+	 * A queued rcu sheaf may only be referenced by an earlier rcu callback
+	 * in memory kmemleak doesn't scan (e.g. struct slab). Never report
+	 * sheaves, but keep scanning the objects they hold.
+	 */
+	if (alloc_flags_allow_spinning(alloc_flags))
+		kmemleak_not_leak(sheaf);
+
 	sheaf->cache = s;
 
 	stat(s, SHEAF_ALLOC);
-- 
2.43.0