* [PATCH] slub: Make sure cache_from_obj() is inlined
@ 2026-01-15 13:06 Eric Dumazet
2026-01-19 22:07 ` Vlastimil Babka
0 siblings, 1 reply; 3+ messages in thread
From: Eric Dumazet @ 2026-01-15 13:06 UTC (permalink / raw)
To: Vlastimil Babka, Andrew Morton
Cc: linux-kernel, Christoph Lameter, David Rientjes, Roman Gushchin,
Harry Yoo, Eric Dumazet, Eric Dumazet
clang ignores the inline attribute because it thinks cache_from_obj()
is too big.
Moves the slow path in a separate function (__cache_from_obj())
and use __fastpath_inline to please clang and CONFIG_SLUB_TINY configs.
This makes kmem_cache_free() and build_detached_freelist()
slightly faster.
$ size mm/slub.clang.before.o mm/slub.clang.after.o
text data bss dec hex filename
77716 7657 4208 89581 15ded mm/slub.clang.before.o
77766 7673 4208 89647 15e2f mm/slub.clang.after.o
$ scripts/bloat-o-meter -t mm/slub.clang.before.o mm/slub.clang.after.o
Function old new delta
__cache_from_obj - 211 +211
build_detached_freelist 542 569 +27
kmem_cache_free 896 919 +23
cache_from_obj 229 - -229
Signed-off-by: Eric Dumazet <edumazet@google.com>
---
mm/slub.c | 18 +++++++++++-------
1 file changed, 11 insertions(+), 7 deletions(-)
diff --git a/mm/slub.c b/mm/slub.c
index 861592ac54257b9d148ff921e6d8f62aced607b3..88a842411c5c3d770ff0070b592f745832d13d1a 100644
--- a/mm/slub.c
+++ b/mm/slub.c
@@ -6748,15 +6748,10 @@ static inline struct kmem_cache *virt_to_cache(const void *obj)
return slab->slab_cache;
}
-static inline struct kmem_cache *cache_from_obj(struct kmem_cache *s, void *x)
+static struct kmem_cache *__cache_from_obj(struct kmem_cache *s, void *x)
{
- struct kmem_cache *cachep;
+ struct kmem_cache *cachep = virt_to_cache(x);
- if (!IS_ENABLED(CONFIG_SLAB_FREELIST_HARDENED) &&
- !kmem_cache_debug_flags(s, SLAB_CONSISTENCY_CHECKS))
- return s;
-
- cachep = virt_to_cache(x);
if (WARN(cachep && cachep != s,
"%s: Wrong slab cache. %s but object is from %s\n",
__func__, s->name, cachep->name))
@@ -6764,6 +6759,15 @@ static inline struct kmem_cache *cache_from_obj(struct kmem_cache *s, void *x)
return cachep;
}
+static __fastpath_inline struct kmem_cache *cache_from_obj(struct kmem_cache *s, void *x)
+{
+ if (!IS_ENABLED(CONFIG_SLAB_FREELIST_HARDENED) &&
+ !kmem_cache_debug_flags(s, SLAB_CONSISTENCY_CHECKS))
+ return s;
+
+ return __cache_from_obj(s, x);
+}
+
/**
* kmem_cache_free - Deallocate an object
* @s: The cache the allocation was from.
base-commit: 944aacb68baf7624ab8d277d0ebf07f025ca137c
--
2.52.0.457.g6b5491de43-goog
^ permalink raw reply [flat|nested] 3+ messages in thread
* Re: [PATCH] slub: Make sure cache_from_obj() is inlined
2026-01-15 13:06 [PATCH] slub: Make sure cache_from_obj() is inlined Eric Dumazet
@ 2026-01-19 22:07 ` Vlastimil Babka
2026-01-20 8:55 ` Eric Dumazet
0 siblings, 1 reply; 3+ messages in thread
From: Vlastimil Babka @ 2026-01-19 22:07 UTC (permalink / raw)
To: Eric Dumazet, Andrew Morton
Cc: linux-kernel, Christoph Lameter, David Rientjes, Roman Gushchin,
Harry Yoo, Eric Dumazet
On 1/15/26 14:06, Eric Dumazet wrote:
> clang ignores the inline attribute because it thinks cache_from_obj()
> is too big.
>
> Moves the slow path in a separate function (__cache_from_obj())
> and use __fastpath_inline to please clang and CONFIG_SLUB_TINY configs.
>
> This makes kmem_cache_free() and build_detached_freelist()
> slightly faster.
>
> $ size mm/slub.clang.before.o mm/slub.clang.after.o
> text data bss dec hex filename
> 77716 7657 4208 89581 15ded mm/slub.clang.before.o
> 77766 7673 4208 89647 15e2f mm/slub.clang.after.o
>
> $ scripts/bloat-o-meter -t mm/slub.clang.before.o mm/slub.clang.after.o
> Function old new delta
> __cache_from_obj - 211 +211
> build_detached_freelist 542 569 +27
> kmem_cache_free 896 919 +23
> cache_from_obj 229 - -229
>
> Signed-off-by: Eric Dumazet <edumazet@google.com>
I assume this is without CONFIG_SLAB_FREELIST_HARDENED. But almost everyone
uses it today:
https://oracle.github.io/kconfigs/?config=UTS_RELEASE&config=SLAB_FREELIST_HARDENED
And with that enabled it would likely make things slower due to the extra
function call to __cache_from_obj(), which does its own virt_to_slab()
although kmem_cache_free() also does it, etc.
However I'd hope things could be improved differently and for all configs.
cache_from_obj() is mostly a relict from when memcgs had separate kmem_cache
instances. It should have been just removed... but hardening repurposed it.
We can however kick it from build_detached_freelist() completely as we're
not checking every object anyway. And kmem_cache_free() can be rewritten to do
the checks open-coded and calling a warn function if they fail. If anyone
cares to harden build_detached_freelist() properly, it could be done
similarly to this.
How does that look for you wrt performance and bloat-o-meter?
---8<---
From e2a45f6d00e437a77532b9a6168cd5b370a59d4d Mon Sep 17 00:00:00 2001
From: Vlastimil Babka <vbabka@suse.cz>
Date: Mon, 19 Jan 2026 22:42:29 +0100
Subject: [PATCH] slab: replace cache_from_obj() with inline checks
Signed-off-by: Vlastimil Babka <vbabka@suse.cz>
---
mm/slub.c | 52 ++++++++++++++++++++++++++++++----------------------
1 file changed, 30 insertions(+), 22 deletions(-)
diff --git a/mm/slub.c b/mm/slub.c
index 861592ac5425..ddaac6990e0b 100644
--- a/mm/slub.c
+++ b/mm/slub.c
@@ -6738,30 +6738,28 @@ void ___cache_free(struct kmem_cache *cache, void *x, unsigned long addr)
}
#endif
-static inline struct kmem_cache *virt_to_cache(const void *obj)
+static noinline void warn_free_bad_obj(struct kmem_cache *s, void *obj)
{
+ struct kmem_cache *cachep;
struct slab *slab;
slab = virt_to_slab(obj);
- if (WARN_ONCE(!slab, "%s: Object is not a Slab page!\n", __func__))
- return NULL;
- return slab->slab_cache;
-}
+ if (WARN_ONCE(!slab,
+ "%s: Object %p from cache %s, is not in a slab page!\n",
+ __func__, obj, s->name))
+ return;
-static inline struct kmem_cache *cache_from_obj(struct kmem_cache *s, void *x)
-{
- struct kmem_cache *cachep;
+ cachep = slab->slab_cache;
- if (!IS_ENABLED(CONFIG_SLAB_FREELIST_HARDENED) &&
- !kmem_cache_debug_flags(s, SLAB_CONSISTENCY_CHECKS))
- return s;
+ if (WARN_ONCE(!cachep, "%s: Object %p from cache %s has NULL "
+ "slab->slab_cache pointer!\n", __func__,
+ obj, s->name))
+ return;
- cachep = virt_to_cache(x);
- if (WARN(cachep && cachep != s,
- "%s: Wrong slab cache. %s but object is from %s\n",
- __func__, s->name, cachep->name))
- print_tracking(cachep, x);
- return cachep;
+ if (WARN_ONCE(cachep != s,
+ "%s: Object %p freed from cache %s but belongs to cache %s\n",
+ __func__, obj, s->name, cachep->name))
+ print_tracking(cachep, obj);
}
/**
@@ -6774,11 +6772,21 @@ static inline struct kmem_cache *cache_from_obj(struct kmem_cache *s, void *x)
*/
void kmem_cache_free(struct kmem_cache *s, void *x)
{
- s = cache_from_obj(s, x);
- if (!s)
- return;
+ struct slab *slab;
+
+ slab = virt_to_slab(x);
+
+ if (IS_ENABLED(CONFIG_SLAB_FREELIST_HARDENED) ||
+ kmem_cache_debug_flags(s, SLAB_CONSISTENCY_CHECKS)) {
+
+ if (unlikely(!slab || (slab->slab_cache != s))) {
+ warn_free_bad_obj(s, x);
+ return;
+ }
+ }
+
trace_kmem_cache_free(_RET_IP_, x, s);
- slab_free(s, virt_to_slab(x), x, _RET_IP_);
+ slab_free(s, slab, x, _RET_IP_);
}
EXPORT_SYMBOL(kmem_cache_free);
@@ -7305,7 +7313,7 @@ int build_detached_freelist(struct kmem_cache *s, size_t size,
df->s = slab->slab_cache;
} else {
df->slab = slab;
- df->s = cache_from_obj(s, object); /* Support for memcg */
+ df->s = s;
}
/* Start new detached freelist */
--
2.52.0
^ permalink raw reply [flat|nested] 3+ messages in thread
* Re: [PATCH] slub: Make sure cache_from_obj() is inlined
2026-01-19 22:07 ` Vlastimil Babka
@ 2026-01-20 8:55 ` Eric Dumazet
0 siblings, 0 replies; 3+ messages in thread
From: Eric Dumazet @ 2026-01-20 8:55 UTC (permalink / raw)
To: Vlastimil Babka
Cc: Andrew Morton, linux-kernel, Christoph Lameter, David Rientjes,
Roman Gushchin, Harry Yoo, Eric Dumazet
On Mon, Jan 19, 2026 at 11:07 PM Vlastimil Babka <vbabka@suse.cz> wrote:
>
> On 1/15/26 14:06, Eric Dumazet wrote:
> > clang ignores the inline attribute because it thinks cache_from_obj()
> > is too big.
> >
> > Moves the slow path in a separate function (__cache_from_obj())
> > and use __fastpath_inline to please clang and CONFIG_SLUB_TINY configs.
> >
> > This makes kmem_cache_free() and build_detached_freelist()
> > slightly faster.
> >
> > $ size mm/slub.clang.before.o mm/slub.clang.after.o
> > text data bss dec hex filename
> > 77716 7657 4208 89581 15ded mm/slub.clang.before.o
> > 77766 7673 4208 89647 15e2f mm/slub.clang.after.o
> >
> > $ scripts/bloat-o-meter -t mm/slub.clang.before.o mm/slub.clang.after.o
> > Function old new delta
> > __cache_from_obj - 211 +211
> > build_detached_freelist 542 569 +27
> > kmem_cache_free 896 919 +23
> > cache_from_obj 229 - -229
> >
> > Signed-off-by: Eric Dumazet <edumazet@google.com>
>
> I assume this is without CONFIG_SLAB_FREELIST_HARDENED. But almost everyone
> uses it today:
> https://oracle.github.io/kconfigs/?config=UTS_RELEASE&config=SLAB_FREELIST_HARDENED
>
> And with that enabled it would likely make things slower due to the extra
> function call to __cache_from_obj(), which does its own virt_to_slab()
> although kmem_cache_free() also does it, etc.
>
Believe it or not, but when CONFIG_SLAB_FREELIST_HARDENED=y,
cache_from_obj() was/is inlined (before and after my patch) by clang :)
> However I'd hope things could be improved differently and for all configs.
> cache_from_obj() is mostly a relict from when memcgs had separate kmem_cache
> instances. It should have been just removed... but hardening repurposed it.
>
> We can however kick it from build_detached_freelist() completely as we're
> not checking every object anyway. And kmem_cache_free() can be rewritten to do
> the checks open-coded and calling a warn function if they fail. If anyone
> cares to harden build_detached_freelist() properly, it could be done
> similarly to this.
>
> How does that look for you wrt performance and bloat-o-meter?
This looks fine to me, thanks !
scripts/bloat-o-meter -t mm/slub.o.old mm/slub.o | grep -v Ltmp
add/remove: 78/78 grow/shrink: 8/1 up/down: 6862/-6443 (419)
Function old new delta
warn_free_bad_obj - 242 +242
kmem_cache_free 896 929 +33
build_detached_freelist 542 531 -11
cache_from_obj 229 - -229
Total: Before=487832, After=488251, chg +0.09%
^ permalink raw reply [flat|nested] 3+ messages in thread
end of thread, other threads:[~2026-01-20 8:55 UTC | newest]
Thread overview: 3+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2026-01-15 13:06 [PATCH] slub: Make sure cache_from_obj() is inlined Eric Dumazet
2026-01-19 22:07 ` Vlastimil Babka
2026-01-20 8:55 ` Eric Dumazet
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®