From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-pl1-f202.google.com (mail-pl1-f202.google.com [209.85.214.202]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 82B634A3C for ; Mon, 2 Feb 2026 19:41:30 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.202 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1770061292; cv=none; b=DyejjT3vnuOmIDSXmD/8i2CNkof/hs84Nmq50h/I30j12ULVfCSjbMk0JkwiPGXXuRC3iJ9QxOEYK2pe8zVY/cRaT6iWPSkIiyhGRcYlC0H3bLcuU+x7Ik8t3sWv+gb9eAs+wSSzReI/wGTkmwJ7/ltjxlt8f9NE3TJu0SfzCVQ= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1770061292; c=relaxed/simple; bh=EgkCytkyqZpd4vDypxIzEL1o0N6v6qDA61JtPtm/ePE=; h=Date:Mime-Version:Message-ID:Subject:From:To:Cc:Content-Type; b=u/vpdECGeA9+7s4s5LST0G1ng1qyjaIqfMdTyBrayb0z1OQUnTZ44TJiIHO9DSaTRhSgSY2tV3MbxbxOnr6qrHNQ8vFx+fZkdzjofiWmeIPow8IV0NigZuX7pxSRPpPhTnXjcfnGsCtpjn0KhT70q4XSZqqZQkng5j5pNtS2qJs= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--jiaqiyan.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=fT/JdAid; arc=none smtp.client-ip=209.85.214.202 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--jiaqiyan.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="fT/JdAid" Received: by mail-pl1-f202.google.com with SMTP id d9443c01a7336-2a863be8508so63303515ad.2 for ; Mon, 02 Feb 2026 11:41:30 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1770061290; x=1770666090; darn=vger.kernel.org; h=cc:to:from:subject:message-id:mime-version:date:from:to:cc:subject :date:message-id:reply-to; bh=R651U2F7F++1pgrvVY/Dtwm4BL52ZLzQ9GK04apqwc0=; b=fT/JdAidex+ZUXzjENLuyZBUs5AIZFLtsaaZAshrlus1eJl/+fYIYBZRYjkBbJgtgK 0Oul4oht8MZzxWCDIHodgH8JY8MEBjbFb+KV4NE9TaR7XsVaeVr+paO/CYWwHgT+0O8u DJUQG1ML0KKEWkFdIOfJQTfsU8uMnEGTVBKQtuJk7xaKy/0AM/zUKIuwMQ+NFfuW/MCN oUIDD+hsT//5RkSbHIin0+YAp3xWm7+nslhSO7Zwq89/fYjc2vWgA8WkgQimVSpWomJd Wv0d8IWxD/8IDGDp2wvEDO42orSP3cdZ6NXIjC3DYiFjeOfY+tj77qP0lGD+rvodMGY2 sESQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1770061290; x=1770666090; h=cc:to:from:subject:message-id:mime-version:date:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=R651U2F7F++1pgrvVY/Dtwm4BL52ZLzQ9GK04apqwc0=; b=Nu8iUsRra78CnAc1NWFgbd9sFOMrdkSrot5uyMC4G/w+jHGSL49TsKaoDDJT64gqxm 9iQJA//X575JsYVkT9TL/u6LoEXkb4Rx+5nhe2As7wEG62sS2gmkRf8NSJJdnRdPQ8Dz Pepwfp3o67JH9xk/tYncM26Y0lc9J64h79VszpQdsE9PBh9uLfpcDkjQdXLC1NcOdgri eW9GFgXN9MK88o9Cwap7VD+5pmNWSN7IrNmA1QTpEnU7ijWs+lN01R0jbclRtM6oTuK7 1Wjrjry/2/R+mON1NWyQy0xSIWEIRiah4bNN62Z+2AJeZ13hxhvyyW5UlXVIb0gtkkmn wa1Q== X-Forwarded-Encrypted: i=1; AJvYcCXCShHGViRSpZxS/SdzNHfLHqASIUuw44vjwQxvt9GogFhjrYCNo2czNbJaLwuUtcMx47SjvQMZRx2WNvg=@vger.kernel.org X-Gm-Message-State: AOJu0YygCFFAXEkJf6MIetCS6AjHXBgQfdWfjke75kkySx14k3OZpuDd ajilC5KpQsYuNBvb57dL27coJ5Tpr0Tsw2k4IO6RRYWxLGr0GTVVzKgsFtOT+hdS7oCPva/oyf7 5FXSv7QY6DtTI2g== X-Received: from plqt11.prod.google.com ([2002:a17:902:a5cb:b0:2a7:80f3:ce5a]) (user=jiaqiyan job=prod-delivery.src-stubby-dispatcher) by 2002:a17:903:1590:b0:2a8:fc56:a3e6 with SMTP id d9443c01a7336-2a8fc56aa0fmr66145035ad.24.1770061289697; Mon, 02 Feb 2026 11:41:29 -0800 (PST) Date: Mon, 2 Feb 2026 19:41:22 +0000 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 X-Mailer: git-send-email 2.53.0.rc1.225.gd81095ad13-goog Message-ID: <20260202194125.2191216-1-jiaqiyan@google.com> Subject: [PATCH v4 0/3] Only free healthy pages in high-order has_hwpoisoned folio From: Jiaqi Yan To: jackmanb@google.com, hannes@cmpxchg.org, linmiaohe@huawei.com, ziy@nvidia.com, harry.yoo@oracle.com, willy@infradead.org Cc: nao.horiguchi@gmail.com, david@redhat.com, lorenzo.stoakes@oracle.com, william.roche@oracle.com, tony.luck@intel.com, wangkefeng.wang@huawei.com, jane.chu@oracle.com, akpm@linux-foundation.org, osalvador@suse.de, muchun.song@linux.dev, rientjes@google.com, duenwen@google.com, jthoughton@google.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Liam.Howlett@oracle.com, vbabka@suse.cz, rppt@kernel.org, surenb@google.com, mhocko@suse.com, boudewijn@delta-utec.com, Jiaqi Yan Content-Type: text/plain; charset="UTF-8" At the end of dissolve_free_hugetlb_folio() that a free HugeTLB folio becomes non-HugeTLB, it is released to buddy allocator as a high-order folio, e.g. a folio that contains 262144 pages if the folio was a 1G HugeTLB hugepage. This is problematic if the HugeTLB hugepage contained HWPoison subpages. In that case, since buddy allocator does not check HWPoison for non-zero-order folio, the raw HWPoison page can be given out with its buddy page and be re-used by either kernel or userspace. Memory failure recovery (MFR) in kernel does attempt to take raw HWPoison page off buddy allocator after dissolve_free_hugetlb_folio(). However, there is always a time window between dissolve_free_hugetlb_folio() frees a HWPoison high-order folio to buddy allocator and MFR takes HWPoison raw page off buddy allocator. Another similar situation is when a transparent huge page (THP) is handled by MFR but splitting failed. Such THP will eventually be released to buddy allocator when owning userspace processes are gone, but with certain subpages having HWPoison [9]. One obvious way to avoid both problems is to add page sanity checks in page allocate or free path. However, it is against the past efforts to reduce sanity check overhead [1,2,3]. Introduce free_has_hwpoisoned() to only free the healthy pages and excludes the HWPoison ones in the high-order folio. free_has_hwpoisoned() happens at the end of free_pages_prepare(), which already deals with both decomposing the original compound page, updating page metadata like alloc tag and page owner. It is also only applied when PG_has_hwpoisoned indicates folio contains certain HWPoison page(s) for performance reason. Its idea is to iterate through the sub-pages of the folio to identify contiguous ranges of healthy pages. Instead of freeing pages one by one, decompose healthy ranges into the largest possible blocks. Each block is freed via free_one_page() directly. free_has_hwpoisoned() has linear time complexity wrt the number of pages in the folio. While the power-of-two decomposition ensures that the number of calls to the buddy allocator is logarithmic for each contiguous healthy range, the mandatory linear scan of pages to identify PageHWPoison defines the overall time complexity. I tested with some test-only code [4] and hugetlb-mfr [5], by checking the status of pcplist and freelist immediately after dissolve_free_hugetlb_folio() a free 2M or 1G hugetlb page that contains 1~8 HWPoison raw pages: - HWPoison pages are excluded by free_has_hwpoisoned(). - Some healthy pages can be in zone->per_cpu_pageset (pcplist) because pcp_count is not high enough. Many healthy pages are in some order's zone->free_area[order].free_list (freelist). - In rare cases, some healthy pages are in neither pcplist nor freelist. My best guest is they are allocated before the test checks. To illustrate the latency free_has_hwpoisoned() added to the memory freeing path, I tested its time cost with 8 HWPoison pages with instrument code in [4] for 20 sample runs: - Has HWPoison path: mean=1448us, stdev=174ms - No HWPoison path: mean=66us, stdev=6us free_has_hwpoisoned() is around 22x the baseline. It is far from triggering soft lockup, and the cost is fair for handling exceptional hardware memory errors. With free_has_hwpoisoned() ensuring HWPoison pages never made into buddy allocator, MFR don't need to take_page_off_buddy() anymore after disovling HWPoison hugepages. So replace __page_handle_poison() with new __hugepage_handle_poison() for HugeTLB specific call sites. Based on commit 8dfce8991b95d ("Merge tag 'pinctrl-v6.19-3' of git://git.kernel.org/pub/scm/linux/kernel/git/linusw/linux-pinctrl") Changelog v3 [8] -> v4 - Address comments from Zi Yan, Miaohe Lin, Harry Yoo. - Set has_hwpoisoned flag after introducing free_has_hwpoisoned(). - Unwrap free_pages_prepare_has_hwpoisoned() into free_pages_prepare(). - If folio has HWPoison, its healthy pages will be freed with FPI_NONE right in free_pages_prepare(), who returns false to indicate caller should not proceeding its own freeing action. - Rework the commit on __page_handle_poison(). Only change the handling for HWPoison HugeTLB page, leaving free buddy page and soft offline handling alone. v2 [7] -> v3: - Address comments from Mathew Wilcox, Harry Hoo, Miaohe Lin. - Let free_has_hwpoisoned() happen after free_pages_prepare(), which help to deal with decomposing the original compound page, and with page metadata like alloc tag and page owner. - Tested with "page_owner=on" and CONFIG_MEM_ALLOC_PROFILING*=y. - Wrap checking PG_has_hwpoisoned and free_has_hwpoisoned() into free_pages_prepare_has_hwpoisoned(), which replaces free_pages_prepare() calls in free_frozen_pages(). - Rename free_has_hwpoison_page() to free_has_hwpoisoned(). - Measure latency added by free_has_hwpoisoned(). - Ensure struct page *end is only used for pointer arithmetic, instead of accessed as page. - Refactor page_handl_poison instead of just __page_handle_poison(). v1 [6] -> v2: - Total reimplementation based on discussions with Mathew Wilcox, Harry Hoo, Zi Yan etc - hugetlb_free_hwpoison_folio() => free_has_hwpoison_pages(). - Utilize has_hwpoisoned flag to tell buddy allocator a high-order folio contains HWPoison. - Simplify __page_handle_poison() given that the HWPoison page(s) won't be freed within high-order folio. [1] https://lore.kernel.org/linux-mm/1460711275-1130-15-git-send-email-mgorman@techsingularity.net [2] https://lore.kernel.org/linux-mm/1460711275-1130-16-git-send-email-mgorman@techsingularity.net [3] https://lore.kernel.org/all/20230216095131.17336-1-vbabka@suse.cz [4] https://drive.google.com/file/d/1CzJn1Cc4wCCm183Y77h244fyZIkTLzCt/view?usp=sharing [5] https://lore.kernel.org/linux-mm/20251116013223.1557158-3-jiaqiyan@google.com [6] https://lore.kernel.org/linux-mm/20251116014721.1561456-1-jiaqiyan@google.com [7] https://lore.kernel.org/linux-mm/20251219183346.3627510-1-jiaqiyan@google.com [8] https://lore.kernel.org/linux-mm/20260112004923.888429-1-jiaqiyan@google.com [9] https://lore.kernel.org/linux-mm/20260113205441.506897-1-boudewijn@delta-utec.com Jiaqi Yan (3): mm/page_alloc: only free healthy pages in high-order has_hwpoisoned folio mm/memory-failure: set has_hwpoisoned flags on dissolved HugeTLB folio mm/memory-failure: skip take_page_off_buddy after dissolving HWPoison HugeTLB page include/linux/page-flags.h | 2 +- mm/memory-failure.c | 37 +++++++++-- mm/page_alloc.c | 133 ++++++++++++++++++++++++++++++++++++- 3 files changed, 163 insertions(+), 9 deletions(-) -- 2.53.0.rc2.204.g2597b5adb4-goog