[PATCH v3 0/6] mm/slab: Support selecting slab hardening features at boot

[PATCH v3 0/6] mm/slab: Support selecting slab hardening features at boot

[Chris Down]

Slab hardening settings are currently all build time choices. This means
that, unfortunately, hosts with different requirements end up needing
separate kernel images. At Meta, for example, some hosts are built with
these options and the rest of the fleet isn't, and that's one of the
things that keeps those kernels apart.

This series does two things:

- SLAB_FREELIST_HARDENED and SLAB_FREELIST_RANDOM are cheap enough with
  sheaves that it makes more sense to just enable them by default than
  to make them switchable. With both on, system level benchmarks didn't
  change measurably, only slowpath heavy microbenchmarks did [1].

- SLAB_BUCKETS and KMALLOC_PARTITION_CACHES cost memory (Harry measured
  about 1 MiB and 18 MiB respectively after boot on a 64 CPU VM [2]), so
  they get slab_buckets= and kmalloc_partition= boot parameters, which
  default to on when the option is built in.

Whether those two should be folded into a single slab_hardening= is
still open [3].

v3:

- Rebase onto slab/for-next.
- Always build the slab_buckets and kmalloc_partition tests and skip them
  at runtime when the option isn't built in, instead of leaving them out
  with #ifdef (Kees).
- Declare slab_buckets_enabled and kmalloc_partition_enabled in
  mm/slab.h without any #if (Kees).
- Drop the remaining #ifdefs around slab_buckets= and
  kmalloc_partition=. Both are now always registered and ignored when
  the option isn't built in (Kees).
- Keep both flags as plain __ro_after_init bools rather than static
  keys, since they're only read when caches are created and not on the
  allocation path.

v2:

- Make SLAB_FREELIST_HARDENED and SLAB_FREELIST_RANDOM default y instead
  of adding boot parameters for them (Vlastimil).
- Skip the memdup_user() part of the slab_buckets test on !MMU
  (Sashiko).
- Keep slab_buckets= and kmalloc_partition= separate for now. We can
  discuss whether to merge them into slab_hardening= or similar, as
  Harry suggested.

v2: https://lore.kernel.org/r/cover.1793253633.git.chris@chrisdown.name
v1: https://lore.kernel.org/r/cover.1793086056.git.chris@chrisdown.name

[1] https://lore.kernel.org/r/auITixAJMfLtfa7i@chrisdown.name
[2] https://lore.kernel.org/r/179312451613.474906.1475329637924619088@kernel.org
[3] https://lore.kernel.org/r/aud4gPOUzvHtLYnJ@thinkstation

Chris Down (6):
  mm/slab: Enable SLAB_FREELIST_HARDENED by default
  mm/slab: Enable SLAB_FREELIST_RANDOM by default
  mm/slab: Support disabling SLAB_BUCKETS at boot
  slub_kunit: Add a test for the slab_buckets boot parameter
  mm/slab: Support disabling KMALLOC_PARTITION_CACHES at boot
  slub_kunit: Add a test for the kmalloc_partition boot parameter

 .../admin-guide/kernel-parameters.txt         | 15 ++++
 lib/tests/slub_kunit.c                        | 85 +++++++++++++++++++
 mm/Kconfig                                    | 11 +++
 mm/slab.h                                     |  3 +
 mm/slab_common.c                              | 52 ++++++++++--
 5 files changed, 157 insertions(+), 9 deletions(-)


base-commit: ce6518e61e3b4bf4f97f7ac6204049b9eba7eb75
-- 
2.56.1

[PATCH v3 1/6] mm/slab: Enable SLAB_FREELIST_HARDENED by default

[Chris Down]

SLAB_FREELIST_HARDENED obfuscates the free pointers stored in free
objects, and catches naive double frees and frees to the wrong cache.
Most distros already enable it, and with sheaves its cost is small,
since the per-cpu fast paths don't touch freelists. Measurements also
show this quite clearly [1][2].

So let's just enable it by default. SLAB_BUCKETS defaults to
SLAB_FREELIST_HARDENED, so it's now enabled by default too.

Link: https://lore.kernel.org/all/auITixAJMfLtfa7i@chrisdown.name/ [1]
Link: https://lore.kernel.org/all/179312451613.474906.1475329637924619088@kernel.org/ [2]
Suggested-by: Vlastimil Babka <vbabka@kernel.org>
Reviewed-by: Kees Cook <kees@kernel.org>
Signed-off-by: Chris Down <chris@chrisdown.name>
---
 mm/Kconfig | 1 +
 1 file changed, 1 insertion(+)

diff --git a/mm/Kconfig b/mm/Kconfig
index afff1b25b543..f330c2c29fd2 100644
--- a/mm/Kconfig
+++ b/mm/Kconfig
@@ -208,6 +208,7 @@ config SLAB_FREELIST_RANDOM
 config SLAB_FREELIST_HARDENED
 	bool "Harden slab freelist metadata"
 	depends on !SLUB_TINY
+	default y
 	help
 	  Many kernel heap attacks try to target slab cache metadata and
 	  other infrastructure. This options makes minor performance
-- 
2.56.1

[PATCH v3 2/6] mm/slab: Enable SLAB_FREELIST_RANDOM by default

[Chris Down]

SLAB_FREELIST_RANDOM randomises the order in which the objects in a new
slab are handed out. Like SLAB_FREELIST_HARDENED, most distros already
enable it, and its cost is only paid when a new slab is set up.

With it on, the same system level benchmarks didn't change measurably.
Allocating from new slabs does get slower, by up to about 30% per object
in a microbenchmark which does little else.

So let's enable it by default too. SHUFFLE_PAGE_ALLOCATOR defaults to
SLAB_FREELIST_RANDOM && ACPI_NUMA, so it's now built in by default on
those systems, but page shuffling itself stays off unless it's enabled
with page_alloc.shuffle.

Suggested-by: Vlastimil Babka <vbabka@kernel.org>
Reviewed-by: Kees Cook <kees@kernel.org>
Signed-off-by: Chris Down <chris@chrisdown.name>
---
 mm/Kconfig | 1 +
 1 file changed, 1 insertion(+)

diff --git a/mm/Kconfig b/mm/Kconfig
index f330c2c29fd2..d174537d941e 100644
--- a/mm/Kconfig
+++ b/mm/Kconfig
@@ -200,6 +200,7 @@ config SLAB_MERGE_DEFAULT
 config SLAB_FREELIST_RANDOM
 	bool "Randomize slab freelist"
 	depends on !SLUB_TINY
+	default y
 	help
 	  Randomizes the freelist order used on creating new pages. This
 	  security feature reduces the predictability of the kernel slab
-- 
2.56.1

[PATCH v3 3/6] mm/slab: Support disabling SLAB_BUCKETS at boot

[Chris Down]

SLAB_BUCKETS gives allocations with user-controlled size and contents,
like memdup_user() and msg_msg, their own set of kmalloc caches, at the
cost of the memory for those caches. It can only be chosen at build
time, so hosts which want it and hosts which would rather not pay for it
need separate kernels.

Add a slab_buckets= early parameter, enabled by default on kernels built
with CONFIG_SLAB_BUCKETS=y, so that it can be turned off at boot.

Signed-off-by: Chris Down <chris@chrisdown.name>
---
 .../admin-guide/kernel-parameters.txt         |  8 ++++++
 mm/Kconfig                                    |  4 +++
 mm/slab_common.c                              | 26 +++++++++++++++----
 3 files changed, 33 insertions(+), 5 deletions(-)

diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt
index 0b3c140babd9..c27bbe205051 100644
--- a/Documentation/admin-guide/kernel-parameters.txt
+++ b/Documentation/admin-guide/kernel-parameters.txt
@@ -6990,6 +6990,14 @@ Kernel parameters
 			apic=verbose is specified.
 			Example: apic=debug show_lapic=all
 
+	slab_buckets= [MM,EARLY]
+			Format: <bool>
+			Enable or disable separate kmalloc buckets on kernels
+			built with CONFIG_SLAB_BUCKETS=y. When disabled,
+			allocations use regular kmalloc caches. Ignored
+			without that option.
+			Default: on.
+
 	slab_debug[=options[,slabs][;[options[,slabs]]...]]	[MM]
 			Enabling slab_debug allows one to determine the
 			culprit if slab objects become corrupted. Enabling
diff --git a/mm/Kconfig b/mm/Kconfig
index d174537d941e..e602ebabd6e4 100644
--- a/mm/Kconfig
+++ b/mm/Kconfig
@@ -231,6 +231,10 @@ config SLAB_BUCKETS
 	  of extra pages since the bulk of user-controlled allocations
 	  are relatively long-lived.
 
+	  The separate buckets can be disabled at boot with the
+	  "slab_buckets=0" kernel parameter, in which case these
+	  allocations use the regular kmalloc caches.
+
 	  If unsure, say Y.
 
 config SLUB_STATS
diff --git a/mm/slab_common.c b/mm/slab_common.c
index 01c64ef661e7..272d22f411c2 100644
--- a/mm/slab_common.c
+++ b/mm/slab_common.c
@@ -410,6 +410,16 @@ EXPORT_SYMBOL(__kmem_cache_create_args);
 
 static struct kmem_cache *kmem_buckets_cache __ro_after_init;
 
+static bool slab_buckets_enabled __ro_after_init = IS_ENABLED(CONFIG_SLAB_BUCKETS);
+
+static int __init setup_slab_buckets(char *str)
+{
+	if (!IS_ENABLED(CONFIG_SLAB_BUCKETS))
+		return 0;
+	return kstrtobool(str, &slab_buckets_enabled);
+}
+early_param("slab_buckets", setup_slab_buckets);
+
 /**
  * kmem_buckets_create - Create a set of caches that handle dynamic sized
  *			 allocations via kmem_buckets_alloc()
@@ -424,10 +434,12 @@ static struct kmem_cache *kmem_buckets_cache __ro_after_init;
  *
  * Context: Cannot be called within an interrupt, but can be interrupted.
  *
- * Return: a pointer to the cache on success, NULL on failure. When
- * CONFIG_SLAB_BUCKETS is not enabled, ZERO_SIZE_PTR is returned, and
- * subsequent calls to kmem_buckets_alloc() will fall back to kmalloc().
- * (i.e. callers only need to check for NULL on failure.)
+ * Return: a pointer to the cache on success, NULL on failure or when the
+ * separate buckets were disabled with the "slab_buckets=0" boot parameter.
+ * When CONFIG_SLAB_BUCKETS is not enabled, ZERO_SIZE_PTR is returned. In all
+ * of these cases subsequent calls to kmem_buckets_alloc() with the returned
+ * value fall back to kmalloc(), so callers that can live without separate
+ * buckets do not need to check the return value.
  */
 kmem_buckets *kmem_buckets_create(const char *name, slab_flags_t flags,
 				  unsigned int useroffset,
@@ -448,6 +460,10 @@ kmem_buckets *kmem_buckets_create(const char *name, slab_flags_t flags,
 	if (!IS_ENABLED(CONFIG_SLAB_BUCKETS))
 		return ZERO_SIZE_PTR;
 
+	/* NULL makes kmalloc_slab() use the regular kmalloc caches. */
+	if (!slab_buckets_enabled)
+		return NULL;
+
 	if (WARN_ON(!kmem_buckets_cache))
 		return NULL;
 
@@ -1059,7 +1075,7 @@ void __init create_kmalloc_caches(void)
 	/* Kmalloc array is now usable */
 	slab_state = UP;
 
-	if (IS_ENABLED(CONFIG_SLAB_BUCKETS))
+	if (IS_ENABLED(CONFIG_SLAB_BUCKETS) && slab_buckets_enabled)
 		kmem_buckets_cache = kmem_cache_create("kmalloc_buckets",
 						       sizeof(kmem_buckets),
 						       0, SLAB_NO_MERGE, NULL);
-- 
2.56.1

[PATCH v3 4/6] slub_kunit: Add a test for the slab_buckets boot parameter

[Chris Down]

Check that memdup_user() allocates from the memdup_user buckets by
default, and from the regular kmalloc caches when they're turned off
with slab_buckets=0. The test is skipped on kernels built without
CONFIG_SLAB_BUCKETS.

Make slab_buckets_enabled global and export it to KUnit so that the
test can see it.

Signed-off-by: Chris Down <chris@chrisdown.name>
---
 lib/tests/slub_kunit.c | 40 ++++++++++++++++++++++++++++++++++++++++
 mm/slab.h              |  2 ++
 mm/slab_common.c       |  4 +++-
 3 files changed, 45 insertions(+), 1 deletion(-)

diff --git a/lib/tests/slub_kunit.c b/lib/tests/slub_kunit.c
index e3b63f0338d5..1af3f087134f 100644
--- a/lib/tests/slub_kunit.c
+++ b/lib/tests/slub_kunit.c
@@ -9,6 +9,8 @@
 #include <linux/delay.h>
 #include <linux/perf_event.h>
 #include <linux/kprobes.h>
+#include <linux/mman.h>
+#include <linux/string.h>
 #include "../mm/slab.h"
 
 static struct kunit_resource resource;
@@ -296,6 +298,42 @@ static void test_krealloc_redzone_zeroing(struct kunit *test)
 	kmem_cache_destroy(s);
 }
 
+static void test_slab_buckets(struct kunit *test)
+{
+	unsigned long user_addr;
+	const char *name;
+	void *p;
+
+	if (!IS_ENABLED(CONFIG_SLAB_BUCKETS))
+		kunit_skip(test, "requires CONFIG_SLAB_BUCKETS");
+
+	if (!slab_buckets_enabled)
+		KUNIT_EXPECT_NULL(test, kmem_buckets_create("TestSlub_buckets",
+							    0, 0, INT_MAX, NULL));
+
+	if (!IS_ENABLED(CONFIG_MMU))
+		kunit_skip(test, "userspace allocation needs an MMU");
+
+	user_addr = kunit_vm_mmap(test, NULL, 0, PAGE_SIZE,
+				  PROT_READ | PROT_WRITE,
+				  MAP_ANONYMOUS | MAP_PRIVATE, 0);
+	KUNIT_ASSERT_NE_MSG(test, user_addr, 0, "Could not create userspace mm");
+	KUNIT_ASSERT_LT_MSG(test, user_addr, (unsigned long)TASK_SIZE,
+			    "Failed to allocate user memory");
+
+	p = memdup_user((void __user *)user_addr, 64);
+	KUNIT_ASSERT_NOT_ERR_OR_NULL(test, p);
+
+	name = virt_to_slab(p)->slab_cache->name;
+	if (slab_buckets_enabled)
+		KUNIT_EXPECT_TRUE_MSG(test, strstarts(name, "memdup_user-"),
+				      "allocated from %s", name);
+	else
+		KUNIT_EXPECT_TRUE_MSG(test, strstarts(name, "kmalloc-"),
+				      "allocated from %s", name);
+	kfree(p);
+}
+
 #if defined(CONFIG_PERF_EVENTS) || (defined(CONFIG_KPROBES) && defined(CONFIG_SMP))
 #define NR_ITERATIONS 1000
 #define NR_OBJECTS 1000
@@ -489,6 +527,7 @@ static struct kunit_case test_cases[] = {
 	KUNIT_CASE(test_kfree_rcu_wq_destroy),
 	KUNIT_CASE(test_leak_destroy),
 	KUNIT_CASE(test_krealloc_redzone_zeroing),
+	KUNIT_CASE(test_slab_buckets),
 #ifdef CONFIG_PERF_EVENTS
 	KUNIT_CASE_SLOW(test_kmalloc_nolock_and_friends_perf),
 #endif
@@ -505,5 +544,6 @@ static struct kunit_suite test_suite = {
 };
 kunit_test_suite(test_suite);
 
+MODULE_IMPORT_NS("EXPORTED_FOR_KUNIT_TESTING");
 MODULE_DESCRIPTION("Kunit tests for slub allocator");
 MODULE_LICENSE("GPL");
diff --git a/mm/slab.h b/mm/slab.h
index e94ac63d10dc..6a28951f6d5d 100644
--- a/mm/slab.h
+++ b/mm/slab.h
@@ -390,6 +390,8 @@ extern struct list_head slab_caches;
 /* The slab cache that manages slab cache information */
 extern struct kmem_cache *kmem_cache;
 
+extern bool slab_buckets_enabled;
+
 /* A table of kmalloc cache names and sizes */
 extern const struct kmalloc_info_struct {
 	const char *name[NR_KMALLOC_TYPES];
diff --git a/mm/slab_common.c b/mm/slab_common.c
index 272d22f411c2..5575706b8938 100644
--- a/mm/slab_common.c
+++ b/mm/slab_common.c
@@ -29,6 +29,7 @@
 #include <linux/memcontrol.h>
 #include <linux/stackdepot.h>
 #include <trace/events/rcu.h>
+#include <kunit/visibility.h>
 
 #include "../kernel/rcu/rcu.h"
 #include "internal.h"
@@ -410,7 +411,8 @@ EXPORT_SYMBOL(__kmem_cache_create_args);
 
 static struct kmem_cache *kmem_buckets_cache __ro_after_init;
 
-static bool slab_buckets_enabled __ro_after_init = IS_ENABLED(CONFIG_SLAB_BUCKETS);
+bool slab_buckets_enabled __ro_after_init = IS_ENABLED(CONFIG_SLAB_BUCKETS);
+EXPORT_SYMBOL_IF_KUNIT(slab_buckets_enabled);
 
 static int __init setup_slab_buckets(char *str)
 {
-- 
2.56.1

[PATCH v3 5/6] mm/slab: Support disabling KMALLOC_PARTITION_CACHES at boot

[Chris Down]

KMALLOC_PARTITION_CACHES creates several copies of each normal kmalloc
cache and spreads allocations across them. That costs a fair bit of
memory: Harry measured about 18 MiB more slab memory after boot on a 64
CPU, 64 GiB VM, mostly from under-used slabs [1]. Like SLAB_BUCKETS, it
can only be chosen at build time.

Add a kmalloc_partition= early parameter, enabled by default on kernels
built with CONFIG_KMALLOC_PARTITION_CACHES=y. When it's off, the copies
alias the normal caches rather than being created.

Let's leave kmalloc_type() unchanged since each index simply resolves to
the same cache when partitioning is off.

Link: https://lore.kernel.org/r/179312451613.474906.1475329637924619088@kernel.org [1]
Signed-off-by: Chris Down <chris@chrisdown.name>
---
 .../admin-guide/kernel-parameters.txt         |  7 ++++++
 mm/Kconfig                                    |  5 ++++
 mm/slab_common.c                              | 23 +++++++++++++++----
 3 files changed, 31 insertions(+), 4 deletions(-)

diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt
index c27bbe205051..91caeef33c3a 100644
--- a/Documentation/admin-guide/kernel-parameters.txt
+++ b/Documentation/admin-guide/kernel-parameters.txt
@@ -3045,6 +3045,13 @@ Kernel parameters
 			Configure the RouterBoard 532 series on-chip
 			Ethernet adapter MAC address.
 
+	kmalloc_partition= [MM,EARLY]
+			Format: <bool>
+			Enable or disable kmalloc cache partitioning on
+			kernels built with CONFIG_KMALLOC_PARTITION_CACHES=y.
+			Ignored without that option.
+			Default: on.
+
 	kmemleak=	[KNL,EARLY] Boot-time kmemleak enable/disable
 			Valid arguments: on, off
 			Default: on
diff --git a/mm/Kconfig b/mm/Kconfig
index e602ebabd6e4..5bb6e33fab49 100644
--- a/mm/Kconfig
+++ b/mm/Kconfig
@@ -266,6 +266,11 @@ config KMALLOC_PARTITION_CACHES
 	  limited degree of memory and CPU overhead that relates to hardware
 	  and system workload.
 
+	  The partitioning can be disabled at boot with the
+	  "kmalloc_partition=0" kernel parameter, in which case the copies
+	  are not created and all normal kmalloc allocations use the
+	  regular kmalloc caches.
+
 choice
 	prompt "Partitioned slab cache mode"
 	depends on KMALLOC_PARTITION_CACHES
diff --git a/mm/slab_common.c b/mm/slab_common.c
index 5575706b8938..20b066d61703 100644
--- a/mm/slab_common.c
+++ b/mm/slab_common.c
@@ -781,6 +781,17 @@ unsigned long random_kmalloc_seed __ro_after_init;
 EXPORT_SYMBOL(random_kmalloc_seed);
 #endif
 
+static bool kmalloc_partition_enabled __ro_after_init =
+	IS_ENABLED(CONFIG_KMALLOC_PARTITION_CACHES);
+
+static int __init setup_kmalloc_partition(char *str)
+{
+	if (!IS_ENABLED(CONFIG_KMALLOC_PARTITION_CACHES))
+		return 0;
+	return kstrtobool(str, &kmalloc_partition_enabled);
+}
+early_param("kmalloc_partition", setup_kmalloc_partition);
+
 /*
  * Conversion table for small slabs sizes / 8 to the index in the
  * kmalloc array. This is necessary for slabs < 192 since we have non power
@@ -1012,10 +1023,14 @@ new_kmalloc_cache(int idx, enum kmalloc_cache_type type)
 		flags |= SLAB_CACHE_DMA;
 	}
 
-#ifdef CONFIG_KMALLOC_PARTITION_CACHES
-	if (type >= KMALLOC_PARTITION_START && type <= KMALLOC_PARTITION_END)
-		flags |= SLAB_NO_MERGE;
-#endif
+	if (kmalloc_partition_enabled) {
+		if (type >= KMALLOC_PARTITION_START && type <= KMALLOC_PARTITION_END)
+			flags |= SLAB_NO_MERGE;
+	} else if (type > KMALLOC_PARTITION_START && type <= KMALLOC_PARTITION_END) {
+		/* Use the normal caches, which have already been created. */
+		kmalloc_caches[type][idx] = kmalloc_caches[KMALLOC_NORMAL][idx];
+		return;
+	}
 
 	/*
 	 * If memcg_kmem is enabled and this is a KMALLOC_NORMAL cache and not
-- 
2.56.1

[PATCH v3 6/6] slub_kunit: Add a test for the kmalloc_partition boot parameter

[Chris Down]

Check that the kmalloc partitions are separate caches by default, and
alias the normal kmalloc caches when they're turned off with
kmalloc_partition=0. The test is skipped on kernels built without
CONFIG_KMALLOC_PARTITION_CACHES.

Make kmalloc_partition_enabled global and export it to KUnit so that
the test can see it.

Signed-off-by: Chris Down <chris@chrisdown.name>
---
 lib/tests/slub_kunit.c | 45 ++++++++++++++++++++++++++++++++++++++++++
 mm/slab.h              |  1 +
 mm/slab_common.c       |  3 ++-
 3 files changed, 48 insertions(+), 1 deletion(-)

diff --git a/lib/tests/slub_kunit.c b/lib/tests/slub_kunit.c
index 1af3f087134f..61227c242ae2 100644
--- a/lib/tests/slub_kunit.c
+++ b/lib/tests/slub_kunit.c
@@ -334,6 +334,50 @@ static void test_slab_buckets(struct kunit *test)
 	kfree(p);
 }
 
+/*
+ * Each partition copy is a separate cache, unless the copies were disabled
+ * with kmalloc_partition=0: then they alias the KMALLOC_NORMAL caches.
+ */
+static void test_kmalloc_partition(struct kunit *test)
+{
+	enum kmalloc_cache_type type;
+	struct kmem_cache *s;
+	unsigned int idx;
+	bool found;
+	void *p;
+
+	if (!IS_ENABLED(CONFIG_KMALLOC_PARTITION_CACHES))
+		kunit_skip(test, "requires CONFIG_KMALLOC_PARTITION_CACHES");
+
+	for (type = KMALLOC_PARTITION_START + 1; type <= KMALLOC_PARTITION_END; type++) {
+		for (idx = 0; idx < ARRAY_SIZE(kmalloc_caches[type]); idx++) {
+			struct kmem_cache *normal = kmalloc_caches[KMALLOC_NORMAL][idx];
+			struct kmem_cache *part = kmalloc_caches[type][idx];
+
+			if (!kmalloc_partition_enabled) {
+				KUNIT_EXPECT_PTR_EQ(test, part, normal);
+			} else if (normal) {
+				KUNIT_EXPECT_NOT_NULL(test, part);
+				KUNIT_EXPECT_PTR_NE(test, part, normal);
+			}
+		}
+	}
+
+	p = kmalloc(64, GFP_KERNEL);
+	KUNIT_ASSERT_NOT_NULL(test, p);
+	s = virt_to_slab(p)->slab_cache;
+	idx = kmalloc_index(64);
+	if (!kmalloc_partition_enabled) {
+		KUNIT_EXPECT_PTR_EQ(test, s, kmalloc_caches[KMALLOC_NORMAL][idx]);
+	} else {
+		found = false;
+		for (type = KMALLOC_PARTITION_START; type <= KMALLOC_PARTITION_END; type++)
+			found |= s == kmalloc_caches[type][idx];
+		KUNIT_EXPECT_TRUE_MSG(test, found, "allocated from %s", s->name);
+	}
+	kfree(p);
+}
+
 #if defined(CONFIG_PERF_EVENTS) || (defined(CONFIG_KPROBES) && defined(CONFIG_SMP))
 #define NR_ITERATIONS 1000
 #define NR_OBJECTS 1000
@@ -528,6 +572,7 @@ static struct kunit_case test_cases[] = {
 	KUNIT_CASE(test_leak_destroy),
 	KUNIT_CASE(test_krealloc_redzone_zeroing),
 	KUNIT_CASE(test_slab_buckets),
+	KUNIT_CASE(test_kmalloc_partition),
 #ifdef CONFIG_PERF_EVENTS
 	KUNIT_CASE_SLOW(test_kmalloc_nolock_and_friends_perf),
 #endif
diff --git a/mm/slab.h b/mm/slab.h
index 6a28951f6d5d..2a9d6a9a9948 100644
--- a/mm/slab.h
+++ b/mm/slab.h
@@ -391,6 +391,7 @@ extern struct list_head slab_caches;
 extern struct kmem_cache *kmem_cache;
 
 extern bool slab_buckets_enabled;
+extern bool kmalloc_partition_enabled;
 
 /* A table of kmalloc cache names and sizes */
 extern const struct kmalloc_info_struct {
diff --git a/mm/slab_common.c b/mm/slab_common.c
index 20b066d61703..d0695e4d2ab6 100644
--- a/mm/slab_common.c
+++ b/mm/slab_common.c
@@ -781,8 +781,9 @@ unsigned long random_kmalloc_seed __ro_after_init;
 EXPORT_SYMBOL(random_kmalloc_seed);
 #endif
 
-static bool kmalloc_partition_enabled __ro_after_init =
+bool kmalloc_partition_enabled __ro_after_init =
 	IS_ENABLED(CONFIG_KMALLOC_PARTITION_CACHES);
+EXPORT_SYMBOL_IF_KUNIT(kmalloc_partition_enabled);
 
 static int __init setup_kmalloc_partition(char *str)
 {
-- 
2.56.1

Re: [PATCH v3 0/6] mm/slab: Support selecting slab hardening features at boot

[Marco Elver]

On Fri, 13 Nov 2026 at 09:51, Chris Down <chris@chrisdown.name> wrote:
>
> Slab hardening settings are currently all build time choices. This means
> that, unfortunately, hosts with different requirements end up needing
> separate kernel images. At Meta, for example, some hosts are built with
> these options and the rest of the fleet isn't, and that's one of the
> things that keeps those kernels apart.
>
> This series does two things:
>
> - SLAB_FREELIST_HARDENED and SLAB_FREELIST_RANDOM are cheap enough with
>   sheaves that it makes more sense to just enable them by default than
>   to make them switchable. With both on, system level benchmarks didn't
>   change measurably, only slowpath heavy microbenchmarks did [1].
>
> - SLAB_BUCKETS and KMALLOC_PARTITION_CACHES cost memory (Harry measured
>   about 1 MiB and 18 MiB respectively after boot on a 64 CPU VM [2]), so
>   they get slab_buckets= and kmalloc_partition= boot parameters, which
>   default to on when the option is built in.
>
> Whether those two should be folded into a single slab_hardening= is
> still open [3].

A coarse slab_hardening= option can easily be replaced by
documentation. While less knobs is better, an intransparent coarse
variant of slab_hardening= just makes the admin wonder "what does that
actually to, and do I want it?". They still have to go and look up
what it actually does, and at that point it doesn't add much value
(alternative suggestion below).

> > > light: "I'm willying to sacrifice _some_ memory and CPU for security"
> > > full: "I'm willying to sacrifice more overhead for security

Who will decide what is light and full? I think there just won't be
any consensus reached.

One thing that users may want is an "all off" option, similar to
mitigations=off, should they wish to get max performance back with a
single option. So you could consider slab_hardening=off. But at that
point it may be more consistent to drop slab_buckets= and
kmalloc_partition= in favor of arguments to slab_hardening=, such as
slab_hardening=nobuckets,partition. This would mirror mitigations=,
and should cater to the usecase of slab_hardening=off to get max
performance with a single option.

But I would not add options such as 'light', 'full'.

Thanks,
-- Marco

Re: [PATCH v3 0/6] mm/slab: Support selecting slab hardening features at boot

[Harry Yoo]

On Fri, Nov 13, 2026 at 11:56:20AM +0100, Marco Elver wrote:
> On Fri, 13 Nov 2026 at 09:51, Chris Down <chris@chrisdown.name> wrote:
> >
> > Slab hardening settings are currently all build time choices. This means
> > that, unfortunately, hosts with different requirements end up needing
> > separate kernel images. At Meta, for example, some hosts are built with
> > these options and the rest of the fleet isn't, and that's one of the
> > things that keeps those kernels apart.
> >
> > This series does two things:
> >
> > - SLAB_FREELIST_HARDENED and SLAB_FREELIST_RANDOM are cheap enough with
> >   sheaves that it makes more sense to just enable them by default than
> >   to make them switchable. With both on, system level benchmarks didn't
> >   change measurably, only slowpath heavy microbenchmarks did [1].
> >
> > - SLAB_BUCKETS and KMALLOC_PARTITION_CACHES cost memory (Harry measured
> >   about 1 MiB and 18 MiB respectively after boot on a 64 CPU VM [2]), so
> >   they get slab_buckets= and kmalloc_partition= boot parameters, which
> >   default to on when the option is built in.
> >
> > Whether those two should be folded into a single slab_hardening= is
> > still open [3].
> 
> A coarse slab_hardening= option can easily be replaced by
> documentation. While less knobs is better, an intransparent coarse
> variant of slab_hardening= just makes the admin wonder "what does that
> actually to, and do I want it?". They still have to go and look up
> what it actually does, and at that point it doesn't add much value
> (alternative suggestion below).
> 
> > > > light: "I'm willying to sacrifice _some_ memory and CPU for security"
> > > > full: "I'm willying to sacrifice more overhead for security
> 
> Who will decide what is light and full? I think there just won't be
> any consensus reached.

You're probably right.

> One thing that users may want is an "all off" option, similar to
> mitigations=off, should they wish to get max performance back with a
> single option. So you could consider slab_hardening=off.

slab_hardening=off sounds good to me.

However this assumes "all on" is the right thing to do by default,
but not enabling slab hardening is not as nearly critical as not
enabling mitigations to CPU vulnerabilities, no?

I'd rather mirror hardened_usercopy= instead of mitigations=off.

slab_hardening=on (turn on all slab hardening features)
slab_hardening=off (turn off all slab hardening features)
(Otherwise it is determined by CONFIG_*.)

> But at that point it may be more consistent to drop slab_buckets= and
> kmalloc_partition= in favor of arguments to slab_hardening=, such as
> slab_hardening=nobuckets,partition. This would mirror mitigations=,
> and should cater to the usecase of slab_hardening=off to get max
> performance with a single option.

I hope we don't reach this point, but more fine-grained control should
be achivable if it is indeed required.

> But I would not add options such as 'light', 'full'.

Fair enough.

> Thanks,
> -- Marco

-- 
Cheers,
Harry / Hyeonggon

Re: [PATCH v3 0/6] mm/slab: Support selecting slab hardening features at boot

[Harry Yoo]

On Fri, Nov 13, 2026 at 06:36:49PM +0000, Harry Yoo wrote:
> On Fri, Nov 13, 2026 at 11:56:20AM +0100, Marco Elver wrote:
> > On Fri, 13 Nov 2026 at 09:51, Chris Down <chris@chrisdown.name> wrote:
> > >
> > > Slab hardening settings are currently all build time choices. This means
> > > that, unfortunately, hosts with different requirements end up needing
> > > separate kernel images. At Meta, for example, some hosts are built with
> > > these options and the rest of the fleet isn't, and that's one of the
> > > things that keeps those kernels apart.
> > >
> > > This series does two things:
> > >
> > > - SLAB_FREELIST_HARDENED and SLAB_FREELIST_RANDOM are cheap enough with
> > >   sheaves that it makes more sense to just enable them by default than
> > >   to make them switchable. With both on, system level benchmarks didn't
> > >   change measurably, only slowpath heavy microbenchmarks did [1].
> > >
> > > - SLAB_BUCKETS and KMALLOC_PARTITION_CACHES cost memory (Harry measured
> > >   about 1 MiB and 18 MiB respectively after boot on a 64 CPU VM [2]), so
> > >   they get slab_buckets= and kmalloc_partition= boot parameters, which
> > >   default to on when the option is built in.
> > >
> > > Whether those two should be folded into a single slab_hardening= is
> > > still open [3].
> > 
> > A coarse slab_hardening= option can easily be replaced by
> > documentation. While less knobs is better, an intransparent coarse
> > variant of slab_hardening= just makes the admin wonder "what does that
> > actually to, and do I want it?". They still have to go and look up
> > what it actually does, and at that point it doesn't add much value
> > (alternative suggestion below).
> > 
> > > > > light: "I'm willying to sacrifice _some_ memory and CPU for security"
> > > > > full: "I'm willying to sacrifice more overhead for security
> > 
> > Who will decide what is light and full? I think there just won't be
> > any consensus reached.
> 
> You're probably right.
> 
> > One thing that users may want is an "all off" option, similar to
> > mitigations=off, should they wish to get max performance back with a
> > single option. So you could consider slab_hardening=off.
> 
> slab_hardening=off sounds good to me.
> 
> However this assumes "all on" is the right thing to do by default,
> but not enabling slab hardening is not as nearly critical as not
> enabling mitigations to CPU vulnerabilities, no?
> 
> I'd rather mirror hardened_usercopy= instead of mitigations=off.
> 
> slab_hardening=on (turn on all slab hardening features)
> slab_hardening=off (turn off all slab hardening features)
> (Otherwise it is determined by CONFIG_*.)

Should have been CONFIG_*_DEFAULT_ON.

> > But at that point it may be more consistent to drop slab_buckets= and
> > kmalloc_partition= in favor of arguments to slab_hardening=, such as
> > slab_hardening=nobuckets,partition. This would mirror mitigations=,
> > and should cater to the usecase of slab_hardening=off to get max
> > performance with a single option.
> 
> I hope we don't reach this point, but more fine-grained control should
> be achivable if it is indeed required.
> 
> > But I would not add options such as 'light', 'full'.
> 
> Fair enough.
> 
> > Thanks,
> > -- Marco
> 
> -- 
> Cheers,
> Harry / Hyeonggon

-- 
Cheers,
Harry / Hyeonggon

Re: [PATCH v3 5/6] mm/slab: Support disabling KMALLOC_PARTITION_CACHES at boot

[kernel test robot]

Hi Chris,

kernel test robot noticed the following build warnings:

[auto build test WARNING on ce6518e61e3b4bf4f97f7ac6204049b9eba7eb75]

url:    https://github.com/intel-lab-lkp/linux/commits/Chris-Down/mm-slab-Enable-SLAB_FREELIST_HARDENED-by-default/20261113-005125
base:   ce6518e61e3b4bf4f97f7ac6204049b9eba7eb75
patch link:    https://lore.kernel.org/r/9a4caff0336c78a7adc746786c81c9c01f89c40b.1794506262.git.chris%40chrisdown.name
patch subject: [PATCH v3 5/6] mm/slab: Support disabling KMALLOC_PARTITION_CACHES at boot
config: hexagon-allnoconfig (https://download.01.org/0day-ci/archive/20261114/202611140317.3kXloW0R-lkp@intel.com/config)
compiler: clang version 24.0.0git (https://github.com/llvm/llvm-project 1b1fed44d3dcca649a9877915d57c3357a98124e)
reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20261114/202611140317.3kXloW0R-lkp@intel.com/reproduce)

If you fix the issue in a separate patch/commit (i.e. not just a new version of
the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <lkp@intel.com>
| Closes: https://lore.kernel.org/oe-kbuild-all/202611140317.3kXloW0R-lkp@intel.com/

All warnings (new ones prefixed by >>):

>> mm/slab_common.c:1029:44: warning: non-overlapping comparisons always evaluate to false [-Wtautological-overlap-compare]
    1029 |         } else if (type > KMALLOC_PARTITION_START && type <= KMALLOC_PARTITION_END) {
         |                    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
   1 warning generated.


vim +1029 mm/slab_common.c

   999	
  1000	static void __init
  1001	new_kmalloc_cache(int idx, enum kmalloc_cache_type type)
  1002	{
  1003		slab_flags_t flags = 0;
  1004		unsigned int minalign = __kmalloc_minalign();
  1005		unsigned int aligned_size = kmalloc_info[idx].size;
  1006		int aligned_idx = idx;
  1007	
  1008		if ((KMALLOC_RECLAIM != KMALLOC_NORMAL) && (type == KMALLOC_RECLAIM)) {
  1009			flags |= SLAB_RECLAIM_ACCOUNT;
  1010		} else if (IS_ENABLED(CONFIG_MEMCG) && (type == KMALLOC_CGROUP)) {
  1011			if (mem_cgroup_kmem_disabled()) {
  1012				kmalloc_caches[type][idx] = kmalloc_caches[KMALLOC_NORMAL][idx];
  1013				return;
  1014			}
  1015			flags |= SLAB_ACCOUNT;
  1016		} else if (IS_ENABLED(CONFIG_SLAB_OBJ_EXT) && type == KMALLOC_NO_OBJ_EXT) {
  1017			if (!need_kmalloc_no_objext()) {
  1018				kmalloc_caches[type][idx] = kmalloc_caches[KMALLOC_NORMAL][idx];
  1019				return;
  1020			}
  1021			flags |= SLAB_NO_OBJ_EXT | SLAB_NO_MERGE;
  1022		} else if (IS_ENABLED(CONFIG_ZONE_DMA) && (type == KMALLOC_DMA)) {
  1023			flags |= SLAB_CACHE_DMA;
  1024		}
  1025	
  1026		if (kmalloc_partition_enabled) {
  1027			if (type >= KMALLOC_PARTITION_START && type <= KMALLOC_PARTITION_END)
  1028				flags |= SLAB_NO_MERGE;
> 1029		} else if (type > KMALLOC_PARTITION_START && type <= KMALLOC_PARTITION_END) {
  1030			/* Use the normal caches, which have already been created. */
  1031			kmalloc_caches[type][idx] = kmalloc_caches[KMALLOC_NORMAL][idx];
  1032			return;
  1033		}
  1034	
  1035		/*
  1036		 * If memcg_kmem is enabled and this is a KMALLOC_NORMAL cache and not
  1037		 * aliased with any other type, make sure it's never merged with any other
  1038		 * cache.
  1039		 *
  1040		 * In other cases the kmalloc cache may end up being used for a
  1041		 * __GFP_ACCOUNT allocation so mark it as such. The exception is a
  1042		 * KMALLOC_NO_OBJ_EXT cache.
  1043		 */
  1044		if (!mem_cgroup_kmem_disabled()) {
  1045			if (type == KMALLOC_NORMAL && KMALLOC_RECLAIM != KMALLOC_NORMAL)
  1046				flags |= SLAB_NO_MERGE;
  1047			else if (!(flags & SLAB_NO_OBJ_EXT))
  1048				flags |= SLAB_MAY_ACCOUNT;
  1049		}
  1050	
  1051		if (minalign > ARCH_KMALLOC_MINALIGN) {
  1052			aligned_size = ALIGN(aligned_size, minalign);
  1053			aligned_idx = __kmalloc_index(aligned_size, false);
  1054		}
  1055	
  1056		if (!kmalloc_caches[type][aligned_idx])
  1057			kmalloc_caches[type][aligned_idx] = create_kmalloc_cache(
  1058						kmalloc_info[aligned_idx].name[type],
  1059						aligned_size, flags);
  1060		if (idx != aligned_idx)
  1061			kmalloc_caches[type][idx] = kmalloc_caches[type][aligned_idx];
  1062	}
  1063	

--
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki

Re: [PATCH v3 5/6] mm/slab: Support disabling KMALLOC_PARTITION_CACHES at boot

[kernel test robot]

Hi Chris,

kernel test robot noticed the following build warnings:

[auto build test WARNING on ce6518e61e3b4bf4f97f7ac6204049b9eba7eb75]

url:    https://github.com/intel-lab-lkp/linux/commits/Chris-Down/mm-slab-Enable-SLAB_FREELIST_HARDENED-by-default/20261113-005125
base:   ce6518e61e3b4bf4f97f7ac6204049b9eba7eb75
patch link:    https://lore.kernel.org/r/9a4caff0336c78a7adc746786c81c9c01f89c40b.1794506262.git.chris%40chrisdown.name
patch subject: [PATCH v3 5/6] mm/slab: Support disabling KMALLOC_PARTITION_CACHES at boot
config: arm-allnoconfig (https://download.01.org/0day-ci/archive/20261114/202611140551.XAyf9KWi-lkp@intel.com/config)
compiler: clang version 17.0.6 (https://github.com/llvm/llvm-project 6009708b4367171ccdbf4b5905cb6a803753fe18)
reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20261114/202611140551.XAyf9KWi-lkp@intel.com/reproduce)

If you fix the issue in a separate patch/commit (i.e. not just a new version of
the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <lkp@intel.com>
| Closes: https://lore.kernel.org/oe-kbuild-all/202611140551.XAyf9KWi-lkp@intel.com/

All warnings (new ones prefixed by >>):

>> mm/slab_common.c:1029:44: warning: overlapping comparisons always evaluate to false [-Wtautological-overlap-compare]
    1029 |         } else if (type > KMALLOC_PARTITION_START && type <= KMALLOC_PARTITION_END) {
         |                    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
   1 warning generated.


vim +1029 mm/slab_common.c

   999	
  1000	static void __init
  1001	new_kmalloc_cache(int idx, enum kmalloc_cache_type type)
  1002	{
  1003		slab_flags_t flags = 0;
  1004		unsigned int minalign = __kmalloc_minalign();
  1005		unsigned int aligned_size = kmalloc_info[idx].size;
  1006		int aligned_idx = idx;
  1007	
  1008		if ((KMALLOC_RECLAIM != KMALLOC_NORMAL) && (type == KMALLOC_RECLAIM)) {
  1009			flags |= SLAB_RECLAIM_ACCOUNT;
  1010		} else if (IS_ENABLED(CONFIG_MEMCG) && (type == KMALLOC_CGROUP)) {
  1011			if (mem_cgroup_kmem_disabled()) {
  1012				kmalloc_caches[type][idx] = kmalloc_caches[KMALLOC_NORMAL][idx];
  1013				return;
  1014			}
  1015			flags |= SLAB_ACCOUNT;
  1016		} else if (IS_ENABLED(CONFIG_SLAB_OBJ_EXT) && type == KMALLOC_NO_OBJ_EXT) {
  1017			if (!need_kmalloc_no_objext()) {
  1018				kmalloc_caches[type][idx] = kmalloc_caches[KMALLOC_NORMAL][idx];
  1019				return;
  1020			}
  1021			flags |= SLAB_NO_OBJ_EXT | SLAB_NO_MERGE;
  1022		} else if (IS_ENABLED(CONFIG_ZONE_DMA) && (type == KMALLOC_DMA)) {
  1023			flags |= SLAB_CACHE_DMA;
  1024		}
  1025	
  1026		if (kmalloc_partition_enabled) {
  1027			if (type >= KMALLOC_PARTITION_START && type <= KMALLOC_PARTITION_END)
  1028				flags |= SLAB_NO_MERGE;
> 1029		} else if (type > KMALLOC_PARTITION_START && type <= KMALLOC_PARTITION_END) {
  1030			/* Use the normal caches, which have already been created. */
  1031			kmalloc_caches[type][idx] = kmalloc_caches[KMALLOC_NORMAL][idx];
  1032			return;
  1033		}
  1034	
  1035		/*
  1036		 * If memcg_kmem is enabled and this is a KMALLOC_NORMAL cache and not
  1037		 * aliased with any other type, make sure it's never merged with any other
  1038		 * cache.
  1039		 *
  1040		 * In other cases the kmalloc cache may end up being used for a
  1041		 * __GFP_ACCOUNT allocation so mark it as such. The exception is a
  1042		 * KMALLOC_NO_OBJ_EXT cache.
  1043		 */
  1044		if (!mem_cgroup_kmem_disabled()) {
  1045			if (type == KMALLOC_NORMAL && KMALLOC_RECLAIM != KMALLOC_NORMAL)
  1046				flags |= SLAB_NO_MERGE;
  1047			else if (!(flags & SLAB_NO_OBJ_EXT))
  1048				flags |= SLAB_MAY_ACCOUNT;
  1049		}
  1050	
  1051		if (minalign > ARCH_KMALLOC_MINALIGN) {
  1052			aligned_size = ALIGN(aligned_size, minalign);
  1053			aligned_idx = __kmalloc_index(aligned_size, false);
  1054		}
  1055	
  1056		if (!kmalloc_caches[type][aligned_idx])
  1057			kmalloc_caches[type][aligned_idx] = create_kmalloc_cache(
  1058						kmalloc_info[aligned_idx].name[type],
  1059						aligned_size, flags);
  1060		if (idx != aligned_idx)
  1061			kmalloc_caches[type][idx] = kmalloc_caches[type][aligned_idx];
  1062	}
  1063	

--
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki