[RFC PATCH 08/18] mm/memory-failure: handle folios with PG_rcu_free

["Kiryl Shutsemau (Meta)" <kas@kernel.org>]

A page whose free waits for an RCU grace period has refcount 0 and is
neither on the LRU nor in the buddy allocator.  memory_failure() and
soft_offline_page() cannot handle such a page: get_any_page() retries a
few times and gives up.  Soft offline of the page fails, and a hard
error is ignored.  If the page is part of a large folio, the folio is
later freed with the poisoned page in it, and that page can be handed
out again.

Free the waiting pages first when the page looks like one of them.
memory_failure() has to do it before it sets PG_hwpoison.  The free of a
poisoned order-0 page drops the page instead of returning it to the
buddy allocator, so a later drain would still leave the error ignored.

A folio with PG_rcu_free that is still in use may be copied by a reader
that holds no reference.  When memory_failure() succeeds, its caller can
unmap the page from the direct map under such a reader.  Wait for a
grace period before returning success.  By then the page has left the
page cache, or is an order-0 folio with PG_hwpoison, which a reader that
starts after the grace period has to check.

Wait at the end, not right after setting PG_hwpoison.  A large folio is
split later, and until the new folios are in the page cache, a reader
can still copy the poisoned page as part of the large one.

Both callers hold mf_mutex and can wait for the grace period.

Assisted-by: LLM
Signed-off-by: Kiryl Shutsemau (Meta) <kas@kernel.org>
---
 mm/memory-failure.c | 24 ++++++++++++++++++++++++
 1 file changed, 24 insertions(+)

diff --git a/mm/memory-failure.c b/mm/memory-failure.c
index 7dde7c3c9a0c..dc66cac05b1f 100644
--- a/mm/memory-failure.c
+++ b/mm/memory-failure.c
@@ -1565,6 +1565,21 @@ static int get_hwpoison_page(struct page *p, unsigned long flags)
 	return ret;
 }
 
+/* A page waiting for a grace period is not in the buddy yet: free it */
+static void mf_free_rcu_pages(struct page *p)
+{
+	if (!IS_ENABLED(CONFIG_PAGE_RCU_FREE))
+		return;
+	if (page_count(p))
+		return;
+	if (!zone_page_state_snapshot(page_zone(p), NR_FREE_RCU_PAGES))
+		return;
+	if (is_free_buddy_page(p))
+		return;
+
+	rcu_free_sync();
+}
+
 /*
  * The caller must guarantee the folio isn't large folio, except hugetlb.
  * try_to_unmap() can't handle it.
@@ -2440,6 +2455,7 @@ int memory_failure(unsigned long pfn, int flags)
 	if (res != -ENOENT)
 		goto unlock_mutex;
 
+	mf_free_rcu_pages(p);
 	if (TestSetPageHWPoison(p)) {
 		res = -EHWPOISON;
 		if (flags & MF_ACTION_REQUIRED)
@@ -2612,6 +2628,13 @@ int memory_failure(unsigned long pfn, int flags)
 
 identify_page_state:
 	res = identify_page_state(pfn, p, page_flags);
+	/*
+	 * The caller may unmap the page once this returns 0.  Lockless
+	 * readers that started before the page got PG_hwpoison, or before a
+	 * split left it in a folio of its own, may still be copying it.
+	 */
+	if (!res && folio_test_rcu_free(folio))
+		synchronize_rcu();
 	mutex_unlock(&mf_mutex);
 	return res;
 unlock_page:
@@ -3003,6 +3026,7 @@ int soft_offline_page(unsigned long pfn, int flags)
 
 retry:
 	get_online_mems();
+	mf_free_rcu_pages(page);
 	ret = get_hwpoison_page(page, flags | MF_SOFT_OFFLINE);
 	put_online_mems();
 
-- 
2.54.0