[PATCH v2 0/2] tools/nolibc: check for overflow in malloc()

[PATCH v2 0/2] tools/nolibc: check for overflow in malloc()

[Danish Khateeb]

malloc() does not check the addition of its header size for overflow,
so malloc(SIZE_MAX), calloc(SIZE_MAX, 1) and realloc(ptr, SIZE_MAX)
return a single page instead of NULL.

As suggested by Thomas, patch 1 drops the rounding to 4096 bytes, and
patch 2 adds the overflow check on top.

Tested on x86_64 (GCC and clang), i386, and on arm, arm64 and sparc64
under qemu-user:

- nolibc-test: no failures, with patch 1 alone and with the series.

- A separate program: malloc(), calloc() and realloc() with sizes close
  to SIZE_MAX fail with ENOMEM, and free() still unmaps every page.

Changes in v2:
- Drop the rounding in a separate patch, instead of checking it for
  overflow (Thomas)
- Drop the test patch (Thomas)
- Link to v1: https://lore.kernel.org/r/20260926134332.58184-1-danishkhateeb03@gmail.com

Danish Khateeb (2):
  tools/nolibc: stop rounding malloc() sizes up to 4096 bytes
  tools/nolibc: check for overflow in malloc()

 tools/include/nolibc/stdlib.h | 8 +++++---
 1 file changed, 5 insertions(+), 3 deletions(-)


base-commit: 0e1c44b472e1ec21efdad1df21b10e5c568b65b5
-- 
2.55.0

[PATCH v2 1/2] tools/nolibc: stop rounding malloc() sizes up to 4096 bytes

[Danish Khateeb]

malloc() rounds the size of each mapping up to a multiple of 4096,
presumably to align it to a page boundary. But the page size is not 4096
on all architectures, and mmap() and munmap() round the length up to the
page size themselves.

Drop the rounding. As heap->len is no longer rounded up, growing a block
with realloc() now always creates a new mapping.

Assisted-by: LLM
Signed-off-by: Danish Khateeb <danishkhateeb03@gmail.com>
---
 tools/include/nolibc/stdlib.h | 2 --
 1 file changed, 2 deletions(-)

diff --git a/tools/include/nolibc/stdlib.h b/tools/include/nolibc/stdlib.h
index 6c3c620a04cf..dc554faff22a 100644
--- a/tools/include/nolibc/stdlib.h
+++ b/tools/include/nolibc/stdlib.h
@@ -130,9 +130,7 @@ void *malloc(size_t len)
 {
 	struct nolibc_heap *heap;
 
-	/* Always allocate memory with size multiple of 4096. */
 	len  = sizeof(*heap) + len;
-	len  = (len + 4095UL) & -4096UL;
 	heap = mmap(NULL, len, PROT_READ|PROT_WRITE, MAP_ANONYMOUS|MAP_PRIVATE,
 		    -1, 0);
 	if (__builtin_expect(heap == MAP_FAILED, 0))
-- 
2.55.0

[PATCH v2 2/2] tools/nolibc: check for overflow in malloc()

[Danish Khateeb]

malloc() adds the size of its header to the requested size without
checking for overflow. For a size within the header size of SIZE_MAX,
the sum wraps around to a few bytes, and malloc() returns a single page
instead of NULL.

calloc() checks its multiplication for overflow, but then passes the
product to malloc(), so calloc(SIZE_MAX, 1) returns a single page too.
So does realloc(ptr, SIZE_MAX).

Such a size is usually a bug in the caller, for instance a length of -1
used as a size_t, and returning a page instead of NULL can turn it into
a heap overflow.

Fail with ENOMEM when adding the header overflows, as glibc does for
sizes this large.

Fixes: 0e0ff638400b ("tools/nolibc/stdlib: Implement `malloc()`, `calloc()`, `realloc()` and `free()`")
Assisted-by: LLM
Signed-off-by: Danish Khateeb <danishkhateeb03@gmail.com>
---
 tools/include/nolibc/stdlib.h | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/tools/include/nolibc/stdlib.h b/tools/include/nolibc/stdlib.h
index dc554faff22a..cea8290ac0b9 100644
--- a/tools/include/nolibc/stdlib.h
+++ b/tools/include/nolibc/stdlib.h
@@ -130,7 +130,11 @@ void *malloc(size_t len)
 {
 	struct nolibc_heap *heap;
 
-	len  = sizeof(*heap) + len;
+	if (__builtin_expect(__builtin_add_overflow(len, sizeof(*heap), &len), 0)) {
+		SET_ERRNO(ENOMEM);
+		return NULL;
+	}
+
 	heap = mmap(NULL, len, PROT_READ|PROT_WRITE, MAP_ANONYMOUS|MAP_PRIVATE,
 		    -1, 0);
 	if (__builtin_expect(heap == MAP_FAILED, 0))
-- 
2.55.0

Re: [PATCH v2 1/2] tools/nolibc: stop rounding malloc() sizes up to 4096 bytes

[Thomas Weißschuh]

On 2026-09-26 10:24:13-0500, Danish Khateeb wrote:
> malloc() rounds the size of each mapping up to a multiple of 4096,
> presumably to align it to a page boundary. But the page size is not 4096
> on all architectures, and mmap() and munmap() round the length up to the
> page size themselves.
> 
> Drop the rounding. As heap->len is no longer rounded up, growing a block
> with realloc() now always creates a new mapping.

The series looks good to me, but I want to hear from Willy before
applying it.

> Assisted-by: LLM
> Signed-off-by: Danish Khateeb <danishkhateeb03@gmail.com>

I'll add a Suggested-by here when applying.

> ---
>  tools/include/nolibc/stdlib.h | 2 --
>  1 file changed, 2 deletions(-)

(...)