Re: [PATCH v3 3/4] mm: hugetlb: Fix subpool usage leak on allocation failure

[Karl Mehltretter <kmehltretter@gmail.com>]

On Wed, 16 Sep 2026 16:39:03 -0700, Ackerley Tng wrote:
>  out_subpool_put:
> +	if (map_chg) {
> +		long gbl_resv_put = hugepage_subpool_put_pages(spool, 1);
> +
> +		hugetlb_acct_memory(h, gbl_resv_get - gbl_resv_put);
>  	}

I reproduced the same fallible-positive-adjustment problem in this
calculation.

The complete subpool put is locally correct, but it can expose capacity
which another operation consumes before this call tries to restore the
corresponding global reservation. A positive hugetlb_acct_memory() can
then fail with -ENOMEM, and its return value is ignored.

I tested the exact patch prefixes on v7.3-rc3 in x86-64 QEMU with 2 MiB
huge pages, at both one and four vCPUs. A default-off test hook enforced
this ordering:

1. A shared MAP_NORESERVE fault acquires one page from a four-page
   minimum subpool while the global pool has no spare capacity.
2. Folio allocation fails.
3. Before rollback, an existing reservation is released and a competing
   reservation consumes that newly available capacity.
4. The subpool put restores the local minimum state, but its required +1
   global correction fails with -ENOMEM.

Both CPU counts produced the same result:

  Source state  Cleanup result  After file removal / after unmount
  ------------  --------------  ---------------------------------
  v7.3-rc3      old cleanup     4 / 0
  patches 1-2   old local leak  3 / 3
  patches 1-3   +1, -ENOMEM     3 / ULONG_MAX
  patches 1-4   +1, -ENOMEM     3 / ULONG_MAX

The expected values are 4 after file removal and 0 after unmount. Patch 3
removes the local usage leak, but the failed positive correction replaces
it with globally unbacked subpool reservations and the unmount underflow.
Patch 4 does not cover this earlier allocation-failure path.

As in the reservation case, the base remains balanced in this controlled
interleaving. Patch 1 makes the path observable on the minimum-only mount,
and patch 3 changes the local leak into globally unbacked reservations.

As for 2/4, I think the subpool get must remain provisional until the
allocation commits or aborts.

A LLM agent helped me with the tests.

Thanks,
Karl