From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7278F3101D8 for ; Tue, 7 Jul 2026 20:46:17 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1783457177; cv=none; b=Bpahqp1LLDwz2d8iZFZ4V6JLKHn94K9XOjIvoRXU0HjgI3s4qerD1GgkgI/NFkQZEIDvBNpVOapHzT969JtMIT9ZAsGjIEnnHx+ktF2Y51uOUp4sZYRAynQS41E5MdUO/Wbb/TReudLbBg5OYAoJ3neIHyYUMMBcEZpzKI4c/m4= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1783457177; c=relaxed/simple; bh=3IWJjK6s97lq4LpNV8m481N4JvphBBCt49nzyXyP0oM=; h=From:Subject:Date:Message-Id:MIME-Version:Content-Type:To:Cc; b=kHwfs1ArUST4oppnnFq/ZnXFlLDU8iEkCAOwz8aS+9Cdkvdb+kJPxLdZ1K9bD+bVR4xu54+agIinyfEhKoZtaAin6XzS9rOrjtOqfEq2qV8YWlNXez+LU9zGL+A57j0QVgKjKEX9yHhi53vHH6E3+g2fmK3sOZIWvIlCzcY4Nm0= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=ANUe/9/A; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="ANUe/9/A" Received: by smtp.kernel.org (Postfix) with ESMTPS id 1AE73C2BCB8; Tue, 7 Jul 2026 20:46:17 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1783457177; bh=3IWJjK6s97lq4LpNV8m481N4JvphBBCt49nzyXyP0oM=; h=From:Subject:Date:To:Cc:Reply-To:From; b=ANUe/9/ACRyTK6jcAp2ntPxCJ3/fEDNg6y/ZZ91SPthLZTdoLzlr59hfyowBnW7MV C+jPFd00CNZBs1waeXT/z/JPyqhJ1688zPQM4aRJH1DgY6n9OY+Xy9YV+CX/oYR5MM Udiuw+GPmEkbZavXUaRJo1yChOrQ7yIq5MOjfFCe3o5VoF1TlVnq4FEQ75Gql0SSq/ 2SUd5rJ8ZT3j5xhDAftAVmKsHv9/MYs4AxEJaMjl7P3QMi2SHH3jdoS8WhLZCrZZvp S0a+xkLVqIcHUm4TwSC46SHtb4s0wxjCm14iJedafCyef8O+nlcZICOOiBBhPBrR+7 qDhaqXosBWLSg== Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id ED7A2C43458; Tue, 7 Jul 2026 20:46:16 +0000 (UTC) From: Ackerley Tng via B4 Relay Subject: [PATCH 0/4] Fix bugs on HugeTLB folio allocation failure paths Date: Tue, 07 Jul 2026 13:46:07 -0700 Message-Id: <20260707-hugetlb-alloc-failure-fixes-v1-0-5bbd3a4b836d@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 7bit X-B4-Tracking: v=1; b=H4sIAJBlTWoC/x3M0QpAMBSH8VfRuXZqVixeRS5m/uPUQhtS8u6Wy 19ffQ8lREGirngo4pIk25pRlQW5xa4zWKZs0ko3yqiGl3PGEUa2IWyOvZVwRrCXG4md8cbUcFa 3nvJhj/hDHvTD+35EVLoYbQAAAA== X-Change-ID: 20260706-hugetlb-alloc-failure-fixes-c7f775eca29f To: Muchun Song , Oscar Salvador , David Hildenbrand , Joshua Hahn , Shakeel Butt , Nhat Pham , Andrew Morton , Peter Xu , Wupeng Ma , fvdl@google.com, rientjes@google.com Cc: vannapurve@google.com, erdemaktas@google.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Ackerley Tng X-Mailer: b4 0.14.3 X-Developer-Signature: v=1; a=ed25519-sha256; t=1783457176; l=3948; i=ackerleytng@google.com; s=20260225; h=from:subject:message-id; bh=3IWJjK6s97lq4LpNV8m481N4JvphBBCt49nzyXyP0oM=; b=lch27w8B/JNod6WTxLtbttJlGGBNwOHfR39Xka7cI9ItveylihcL8ihpENvPZdjg53+GflQqu 5GAIHNHwj6cCs8lJZHLGxqOZsb39+g9S7bQNeh5o/Wo1lI9OpaRRu38 X-Developer-Key: i=ackerleytng@google.com; a=ed25519; pk=sAZDYXdm6Iz8FHitpHeFlCMXwabodTm7p8/3/8xUxuU= X-Endpoint-Received: by B4 Relay for ackerleytng@google.com/20260225 with auth_id=649 X-Original-From: Ackerley Tng Reply-To: ackerleytng@google.com When mem_cgroup_charge_hugetlb() fails (e.g. when hitting memcg limits), the HugeTLB allocation path currently handles the errors incorrectly. This series fixes 4 issues related to HugeTLB allocation failure paths. Some of these issues were pointed out by Sashiko while I was working on [1]. 1. Fix subpool usage leak on allocation failure: (Sashiko pointed this out in [2]) When alloc_hugetlb_folio() fails early (e.g. buddy allocation failure or hugetlb cgroup charging failure) and gbl_chg == 1, the error path skips restoring the page to the subpool, leaking the subpool's used_hpages counter. Fix this by calling hugepage_subpool_put_pages() unconditionally if map_chg is true. (1.) above could technically be a separate patch series, but I'm relying on that for fix (4.) below. 2. Fix folio refcount mismatch on memcg charge failure: (Sashiko pointed this out in [3]) The error path in alloc_hugetlb_folio() calls free_huge_folio() directly on a folio with a refcount of 1 (set via folio_ref_unfreeze() earlier). This triggers VM_BUG_ON_FOLIO(folio_ref_count(folio), folio) if CONFIG_DEBUG_VM is enabled, and can corrupt allocator state otherwise. Fix this by using folio_put() instead of free_huge_folio() to correctly drop the refcount before freeing. While fixing the above, the reproducer (I'll send these separately) was causing an infinite loop during allocation, unveiling issue (3.). 3. Return -ENOSPC on memcg charge failure to prevent infinite loop: alloc_hugetlb_folio() propagates -ENOMEM on charge failure, which maps to VM_FAULT_OOM. Because HugeTLB physical allocations are high-order and use __GFP_RETRY_MAYFAIL, the OOM killer is bypassed. Returning VM_FAULT_OOM leaks to the #PF handler, which cannot make progress and retries the faulting instruction indefinitely. Fix this by returning -ENOSPC instead of -ENOMEM on charge failure, which maps to VM_FAULT_SIGBUS, terminating the process cleanly. And while running the reproducer, I checked this routinely head /sys/kernel/mm/hugepages/hugepages-2048kB/* and found that there was a resv_hugepages leakm, 1 per execution of the reproducer, unveiling (4.) below. 4. Move memcg charge earlier to prevent reservation leak: When mem_cgroup_charge_hugetlb() fails, the error path historically bypassed vma_end_reservation(). Since the reservation had already been committed via vma_commit_reservation(), this left the reservation map in an inconsistent state, leaking resv_huge_pages when the process exited. Fix this by moving mem_cgroup_charge_hugetlb() earlier in alloc_hugetlb_folio(), before vma_commit_reservation() is called. Testing: + libhugetlbfs tests pass + ./tools/testing/selftests/mm/ksft_hugetlb.sh passes The series [1] has some changes that are dependent on these fixes, so I'll wait for your reviews before continuing on [1]. Thank you! [1] https://lore.kernel.org/all/20260702-hugetlb-open-up-v4-0-d53cefcccf34@google.com/T/ [2] https://sashiko.dev/#/patchset/20260518-hugetlb-open-up-v3-0-e14b302477f8%40google.com?part=5 [3] https://sashiko.dev/#/patchset/20260702-hugetlb-open-up-v4-0-d53cefcccf34%40google.com?part=6 Signed-off-by: Ackerley Tng --- Ackerley Tng (4): mm: hugetlb: Fix subpool usage leak on allocation failure mm: hugetlb: Fix folio refcount mismatch on memcg charge failure mm: hugetlb: Return -ENOSPC on memcg charge failure mm: hugetlb: Move memcg charge earlier to prevent reservation leak mm/hugetlb.c | 57 ++++++++++++++++++++++++++++++++++++--------------------- 1 file changed, 36 insertions(+), 21 deletions(-) --- base-commit: 0e35b9b6ec0ffcc5e23cbdec09f5c622ad532b53 change-id: 20260706-hugetlb-alloc-failure-fixes-c7f775eca29f Best regards, -- Ackerley Tng