[PATCH 0/2] tools/nolibc: check for overflow in malloc()

[PATCH 0/2] tools/nolibc: check for overflow in malloc()

[Danish Khateeb]

malloc() does not check its size arithmetic for overflow, so
malloc(SIZE_MAX), calloc(SIZE_MAX, 1) and realloc(ptr, SIZE_MAX) return
a single page instead of NULL, and slightly smaller sizes fail with
EINVAL instead of ENOMEM.

Patch 1 fixes it. Patch 2 adds nolibc-test cases for huge sizes, which
would have caught it.

Tested on top of nolibc/for-next 0e1c44b472e1, on x86_64 (GCC and
clang) and i386, and on arm, arm64 and sparc64 under qemu-user:

- nolibc-test, all tests: no failures with the series. The only
  difference from before is the three new tests, which pass. They also
  pass against glibc (make libc-test), which builds without warnings.

- Without patch 1, all three new tests fail on every build:
  malloc(SIZE_MAX) and calloc(SIZE_MAX, 1) return a pointer, and
  malloc(SIZE_MAX - 4096) fails with EINVAL.

- On x86_64, a separate program shows realloc(ptr, SIZE_MAX) returning
  a pointer before patch 1 and NULL with ENOMEM after it.

Danish Khateeb (2):
  tools/nolibc: check for overflow in malloc()
  selftests/nolibc: test malloc() and calloc() with huge sizes

 tools/include/nolibc/stdlib.h                | 12 +++++++++---
 tools/testing/selftests/nolibc/nolibc-test.c | 11 +++++++++++
 2 files changed, 20 insertions(+), 3 deletions(-)


base-commit: 0e1c44b472e1ec21efdad1df21b10e5c568b65b5
-- 
2.55.0

[PATCH 1/2] tools/nolibc: check for overflow in malloc()

[Danish Khateeb]

malloc() adds the size of its header to the requested size and rounds
the sum up to a multiple of 4096, without checking either step for
overflow. For a size within the header size of SIZE_MAX, the sum wraps
around to a few bytes, and malloc() returns a single page instead of
NULL. For a slightly smaller size, the rounding wraps around to 0, and
malloc() fails with EINVAL from mmap() instead of ENOMEM.

calloc() checks its multiplication for overflow, but then passes the
product to malloc(), so calloc(SIZE_MAX, 1) returns a single page too.
So does realloc(ptr, SIZE_MAX).

Such a size is usually a bug in the caller, for instance a length of -1
used as a size_t, and returning a page instead of NULL can turn it into
a heap overflow.

Fail with ENOMEM when adding the header or rounding up overflows, as
glibc does for sizes this large.

Fixes: 0e0ff638400b ("tools/nolibc/stdlib: Implement `malloc()`, `calloc()`, `realloc()` and `free()`")
Assisted-by: LLM
Signed-off-by: Danish Khateeb <danishkhateeb03@gmail.com>
---
 tools/include/nolibc/stdlib.h | 12 +++++++++---
 1 file changed, 9 insertions(+), 3 deletions(-)

diff --git a/tools/include/nolibc/stdlib.h b/tools/include/nolibc/stdlib.h
index 6c3c620a04cf..ea315e60cdeb 100644
--- a/tools/include/nolibc/stdlib.h
+++ b/tools/include/nolibc/stdlib.h
@@ -130,9 +130,15 @@ void *malloc(size_t len)
 {
 	struct nolibc_heap *heap;
 
-	/* Always allocate memory with size multiple of 4096. */
-	len  = sizeof(*heap) + len;
-	len  = (len + 4095UL) & -4096UL;
+	/*
+	 * Always allocate memory with size multiple of 4096, and reject sizes
+	 * which would wrap around once the header is added and rounded up.
+	 */
+	if (__builtin_expect(__builtin_add_overflow(len, sizeof(*heap) + 4095UL, &len), 0)) {
+		SET_ERRNO(ENOMEM);
+		return NULL;
+	}
+	len &= -4096UL;
 	heap = mmap(NULL, len, PROT_READ|PROT_WRITE, MAP_ANONYMOUS|MAP_PRIVATE,
 		    -1, 0);
 	if (__builtin_expect(heap == MAP_FAILED, 0))
-- 
2.55.0

Re: [PATCH 1/2] tools/nolibc: check for overflow in malloc()

[Thomas Weißschuh]

On 2026-09-26 08:43:31-0500, Danish Khateeb wrote:
> malloc() adds the size of its header to the requested size and rounds
> the sum up to a multiple of 4096, without checking either step for
> overflow. For a size within the header size of SIZE_MAX, the sum wraps
> around to a few bytes, and malloc() returns a single page instead of
> NULL. For a slightly smaller size, the rounding wraps around to 0, and
> malloc() fails with EINVAL from mmap() instead of ENOMEM.
> 
> calloc() checks its multiplication for overflow, but then passes the
> product to malloc(), so calloc(SIZE_MAX, 1) returns a single page too.
> So does realloc(ptr, SIZE_MAX).
> 
> Such a size is usually a bug in the caller, for instance a length of -1
> used as a size_t, and returning a page instead of NULL can turn it into
> a heap overflow.
> 
> Fail with ENOMEM when adding the header or rounding up overflows, as
> glibc does for sizes this large.
> 
> Fixes: 0e0ff638400b ("tools/nolibc/stdlib: Implement `malloc()`, `calloc()`, `realloc()` and `free()`")
> Assisted-by: LLM
> Signed-off-by: Danish Khateeb <danishkhateeb03@gmail.com>
> ---
>  tools/include/nolibc/stdlib.h | 12 +++++++++---
>  1 file changed, 9 insertions(+), 3 deletions(-)
> 
> diff --git a/tools/include/nolibc/stdlib.h b/tools/include/nolibc/stdlib.h
> index 6c3c620a04cf..ea315e60cdeb 100644
> --- a/tools/include/nolibc/stdlib.h
> +++ b/tools/include/nolibc/stdlib.h
> @@ -130,9 +130,15 @@ void *malloc(size_t len)
>  {
>  	struct nolibc_heap *heap;
>  
> -	/* Always allocate memory with size multiple of 4096. */
> -	len  = sizeof(*heap) + len;
> -	len  = (len + 4095UL) & -4096UL;
> +	/*
> +	 * Always allocate memory with size multiple of 4096, and reject sizes
> +	 * which would wrap around once the header is added and rounded up.
> +	 */

The rounding to a multiple to 4096 is probably meant to align the size
to a page boundary. But this is insufficient on systems where the page
size is not 4k. The kernel doesn't actually seem to care.
Let's drop the rounding instead, and then add the overflow check on top.

Also the unittest doesn't seem to be worth it.

> +	if (__builtin_expect(__builtin_add_overflow(len, sizeof(*heap) + 4095UL, &len), 0)) {
> +		SET_ERRNO(ENOMEM);
> +		return NULL;
> +	}
> +	len &= -4096UL;
>  	heap = mmap(NULL, len, PROT_READ|PROT_WRITE, MAP_ANONYMOUS|MAP_PRIVATE,
>  		    -1, 0);
>  	if (__builtin_expect(heap == MAP_FAILED, 0))
> -- 
> 2.55.0
> 

[PATCH 2/2] selftests/nolibc: test malloc() and calloc() with huge sizes

[Danish Khateeb]

test_malloc() only allocates a few pages, so nothing noticed that
malloc() and calloc() went wrong for sizes close to SIZE_MAX.

Check that malloc(SIZE_MAX), malloc(SIZE_MAX - 4096) and
calloc(SIZE_MAX, 1) fail with ENOMEM. The sizes are hidden from the
compiler, as GCC warns about such large constant allocations when
building against glibc.

Assisted-by: LLM
Signed-off-by: Danish Khateeb <danishkhateeb03@gmail.com>
---
 tools/testing/selftests/nolibc/nolibc-test.c | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/tools/testing/selftests/nolibc/nolibc-test.c b/tools/testing/selftests/nolibc/nolibc-test.c
index 1995d67dcca6..0bb5116763bc 100644
--- a/tools/testing/selftests/nolibc/nolibc-test.c
+++ b/tools/testing/selftests/nolibc/nolibc-test.c
@@ -1915,6 +1915,14 @@ int test_malloc(void)
 	return 0;
 }
 
+/* Hide a size from the compiler, which warns about huge constant allocations */
+static size_t hide_size(size_t size)
+{
+	__asm__ ("" : "+r" (size));
+
+	return size;
+}
+
 int run_stdlib(int min, int max)
 {
 	int test;
@@ -2050,6 +2058,9 @@ int run_stdlib(int min, int max)
 		CASE_TEST(major_big);               EXPECT_EQ(1, major(0x1122355667734488), 0x11223344); break;
 		CASE_TEST(minor_big);               EXPECT_EQ(1, minor(0x1122355667734488), 0x55667788); break;
 		CASE_TEST(malloc);                  EXPECT_ZR(1, test_malloc()); break;
+		CASE_TEST(malloc_max);              EXPECT_PTRER(1, malloc(hide_size(SIZE_MAX)), NULL, ENOMEM); break;
+		CASE_TEST(malloc_max_page);         EXPECT_PTRER(1, malloc(hide_size(SIZE_MAX - 4096)), NULL, ENOMEM); break;
+		CASE_TEST(calloc_max);              EXPECT_PTRER(1, calloc(hide_size(SIZE_MAX), 1), NULL, ENOMEM); break;
 		CASE_TEST(bswap_16);                EXPECT_EQ(1, bswap_16(0x0123), 0x2301); break;
 		CASE_TEST(bswap_32);                EXPECT_EQ(1, bswap_32(0x01234567), 0x67452301); break;
 		CASE_TEST(bswap_64);                EXPECT_EQ(1, bswap_64(0x0123456789abcdef), 0xefcdab8967452301); break;
-- 
2.55.0