Re: [PATCH v3 2/4] mm: hugetlb: Fix out_put_pages subpool reserve calculation

[Karl Mehltretter <kmehltretter@gmail.com>]

On Wed, 16 Sep 2026 16:39:02 -0700, Ackerley Tng wrote:
> +out_put_pages:
> +	gbl_resv_put = hugepage_subpool_put_pages(spool, chg);
> +	/*
> +	 * There may be a difference between the number of
> +	 * reservations to consume and the number to restore now if
> +	 * there are multiple threads interacting with the subpool -
> +	 * restore the difference.
> +	 */
> +	hugetlb_acct_memory(h, gbl_resv_get - gbl_resv_put);

I was able to turn this positive adjustment into a deterministic runtime
failure.

hugetlb_acct_memory() can fail with -ENOMEM, but the subpool's local
state has already been restored and the return value is ignored.

I tested the exact patch prefixes on v7.3-rc3 in x86-64 QEMU with 2 MiB
huge pages, at both one and four vCPUs. A default-off test hook enforced
this ordering:

1. A four-page reservation consumes the two remaining minimum reservations
   of a min_size=8M mount and needs two more globally.
2. Global accounting fails because another mount has filled the global
   pool.
3. Before rollback, two existing reservations are released and a competing
   reservation consumes that newly available capacity.
4. The complete subpool put restores the local four-page minimum, but the
   required +2 global correction fails with -ENOMEM.

Both CPU counts produced the same result:

  Source state  Cleanup result  After file removal / after unmount
  ------------  --------------  ---------------------------------
  v7.3-rc3      old cleanup    4 / 0
  patch 1       old local leak  2 / 2
  patches 1-2   +2, -ENOMEM     2 / ULONG_MAX-1
  patches 1-4   +2, -ENOMEM     2 / ULONG_MAX-1

The expected values are 4 after file removal and 0 after unmount. The
ULONG_MAX-1 value is the resulting HugePages_Rsvd underflow.

So, although the unchecked positive call is an older pattern, this
controlled interleaving is balanced on the rc3 base. Patch 1 makes it
reachable on the minimum-only mount, and patch 2 changes patch 1's local
leak into globally unbacked subpool reservations.

This confirms that checking the return value only after the put would be
too late. I think the get has to remain provisional until global accounting
either commits or aborts, so rollback cannot expose capacity and then need
to reacquire it.

A LLM agent helped me with the tests.

Thanks,
Karl