From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-qk1-f177.google.com (mail-qk1-f177.google.com [209.85.222.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 713BA3B27DB for ; Fri, 26 Jun 2026 18:22:42 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.222.177 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1782498163; cv=none; b=q2LI3pMJzDBHQDabM7scbFIDLJuxGYw+/+sZOAlEd+yyNxjlygqcKTvHtHNwwUz8qinQZbuq7nh2otvid1JbH+SPVxchKu4ez+ycANP93UFFIXNtAFaSRK1wtdzz7sKA989S05tJH8QkanI4W48Gy9U63aUAep0aWBa3XgOlF/c= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1782498163; c=relaxed/simple; bh=qDx7IZepBzB4qaXxabN40+k34yeHoYtxrf742SkSD/g=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=rSWOmLTDmL/GYArTf35S4vUqkNmL3Kmw70wMMGMHGVhiRiGLpoSfPugUH/ffBd7+JHNwPtKIo39LOtQddr+d8rZ7ipQuo3iTdBdZcKRKyBRS5KkTeIZLm5TJwfqZeoWrKchHVk89BzcxXfm0VgMXccbCZ1NrJ4aYKudUD/JheYc= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=cmpxchg.org; spf=pass smtp.mailfrom=cmpxchg.org; dkim=pass (2048-bit key) header.d=cmpxchg.org header.i=@cmpxchg.org header.b=XgfbaZaM; arc=none smtp.client-ip=209.85.222.177 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=cmpxchg.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=cmpxchg.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=cmpxchg.org header.i=@cmpxchg.org header.b="XgfbaZaM" Received: by mail-qk1-f177.google.com with SMTP id af79cd13be357-9157f7c1c0eso156361485a.1 for ; Fri, 26 Jun 2026 11:22:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cmpxchg.org; s=google; t=1782498161; x=1783102961; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=wzgs5H0Grhs1ueQiwh3lj+d4hLeKAITZ3jKxY2kFgsY=; b=XgfbaZaMiyMdyCc0AuecOddnFfg1hOoiRnEpy+vXCajG0owj0ev1xODVBydvVe7v1l DOcxO7FVnvCdPrcGKdVNDd28dOkWxv/Qu0/fdOKHmBUuzeOwh/D2RbG1gaCxNVpLmlnF sT3qeAqrFIVsOa/oQZd3lbOO8+aY61AXxaMjY1PSFMAVEw0d81/DQoZ+ChkxVY/EMKXq 2HaoNO0120B/mR7No3IM3zDZv2XmqICK1Q9wRNdkTe7XIqR+Y4lF/f6IbNtTw/A/Ju1u nYO4a8wjOUe2++oKpSTY477jpNoGaRJqKJiy58bb+eNWky+3QkpOkmDSvw9zzbiKShKV EALg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1782498161; x=1783102961; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=wzgs5H0Grhs1ueQiwh3lj+d4hLeKAITZ3jKxY2kFgsY=; b=aGEf1Sb1ch55ib3wd0OzVq3+XeSgJUUccv/hfo0e1plAoRj7OCv4HRMAlPDa8eEfPN oaOT/swOeSrwSBVaGx9RnVFyVD6JxqTCi06VhD0IzwCP4b2IJdxxlB1KCA21Kzdbv+wl LQHGsqxqpCWi3CEencvpXXGFQ7PG+OrvTuG4lU4t90VOHYLQ1k2b/DbAH+9mnyHASmsu azf7kbW/qcFBP9BCEkl6O0+QiLHUJ1RpbA6uitQgLZDI2TaTego0Z4oiN7IUfQij09M0 36g9IUm6Sn71PppvHTdnxGBvR0LfqflBzu2FGZlBiHQt78pHU5sa/lBkYYojHzycVlK7 rT7w== X-Forwarded-Encrypted: i=1; AFNElJ/gbLxCxTxxhmLUnqieFPgJudrTSjjeaWO7Kf61pax8a9VRA/JcHgzqt+i/2Nf3hMcliijy59OLKIAZKWQ=@vger.kernel.org X-Gm-Message-State: AOJu0YzexRHC5lDgD98zKqMq8kZLND+o5uNxmTyRQBcXZVkWm1y1x06S oB9sosCjHy4BwXzmRyes6BjbeJ2H0C1x0SdzT1oaqxPpx3aP8Reweg0D5EvCJnehKyg= X-Gm-Gg: AfdE7ck7SWcFHzB9wPCG4C8m1rcVxvydRVvgWPv8+AiHJ80Q1WCHd+2qsV6sWw3OGsn X9V1HecaCoG9n81p0vqTMYOxo9TanHEkG7okvUgAU3TlETz6AHf5ReErsGD9xVLR1k3Qm+i69k8 0q0ZWkUWHqADLeV4TO9prM7b3poBfHRG0l2g1AZqPwTIErirNV/pmBUfWGO9STQNmHq/KY6EfO5 8tLF0yBbMauxtXXQhrUfoGtUg4KELBlF1NF7Xiv0qmQarJKteYrNpJfyqc9Vboor3FFvwmfG5OM gXjrBczkWPEYNvhTPQvWUf8jiJeS4s+JpoPG/64gF347LZGqS3TpTIQr45WV2S8NqDjLLqiXzQP OndEcLjPqg+/4dxxlCifstjnE0DT34jCvuK2eXzYp7tlAl3Uu6meUo2ZE8I8xQPrx0fAiYpsK9W UBg7Uh5KaH7wg= X-Received: by 2002:a05:620a:6491:b0:92b:32d4:4af5 with SMTP id af79cd13be357-92b32d46470mr307992685a.5.1782498156661; Fri, 26 Jun 2026 11:22:36 -0700 (PDT) Received: from localhost ([2603:7001:f100:500:365a:60ff:fe62:ff29]) by smtp.gmail.com with ESMTPSA id af79cd13be357-92ac2e0026esm206754585a.10.2026.06.26.11.22.34 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 26 Jun 2026 11:22:35 -0700 (PDT) From: Johannes Weiner To: Andrew Morton , Vlastimil Babka Cc: Suren Baghdasaryan , Michal Hocko , Brendan Jackman , Zi Yan , David Hildenbrand , Lorenzo Stoakes , "Liam R. Howlett" , Mike Rapoport , linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH 3/4] mm: page_alloc: move capture_control to the page allocator Date: Fri, 26 Jun 2026 14:21:19 -0400 Message-ID: <20260626182215.1107966-4-hannes@cmpxchg.org> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20260626182215.1107966-1-hannes@cmpxchg.org> References: <20260626182215.1107966-1-hannes@cmpxchg.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit The compaction capturing code assumes the allocation request order and compaction target order are the same. That won't be true once defrag_mode promotes sub-block allocations to pageblock-order compaction: compaction targets the larger order, capture should remain at the original allocation order. Move the per-task capture_control to the page allocator, so its fields can carry alloc-side information that compaction's compact_control does not. Pass the capture_control through try_to_compact_pages() / compact_zone_order() instead of a bare struct page **; compact_zone_order() sets capc->cc while running. task_capc() now also checks capc->cc to handle the new not-yet-running state. No functional change. Signed-off-by: Johannes Weiner --- include/linux/compaction.h | 3 ++- mm/compaction.c | 33 ++++++++++----------------------- mm/page_alloc.c | 23 +++++++++++++++++++++-- 3 files changed, 33 insertions(+), 26 deletions(-) diff --git a/include/linux/compaction.h b/include/linux/compaction.h index f29ef0653546..66a2f70e9e01 100644 --- a/include/linux/compaction.h +++ b/include/linux/compaction.h @@ -58,6 +58,7 @@ enum compact_result { }; struct alloc_context; /* in mm/internal.h */ +struct capture_control; /* in mm/internal.h */ /* * Number of free order-0 pages that should be available above given watermark @@ -92,7 +93,7 @@ extern int fragmentation_index(struct zone *zone, unsigned int order); extern enum compact_result try_to_compact_pages(gfp_t gfp_mask, unsigned int order, unsigned int alloc_flags, const struct alloc_context *ac, enum compact_priority prio, - struct page **page); + struct capture_control *capc); extern void reset_isolation_suitable(pg_data_t *pgdat); extern bool compaction_suitable(struct zone *zone, int order, unsigned long watermark, int highest_zoneidx); diff --git a/mm/compaction.c b/mm/compaction.c index 7df3a85d43af..c2701bf1d04e 100644 --- a/mm/compaction.c +++ b/mm/compaction.c @@ -2791,7 +2791,7 @@ compact_zone(struct compact_control *cc, struct capture_control *capc) static enum compact_result compact_zone_order(struct zone *zone, int order, gfp_t gfp_mask, enum compact_priority prio, unsigned int alloc_flags, int highest_zoneidx, - struct page **capture) + struct capture_control *capc) { enum compact_result ret; struct compact_control cc = { @@ -2808,35 +2808,22 @@ static enum compact_result compact_zone_order(struct zone *zone, int order, .ignore_skip_hint = (prio == MIN_COMPACT_PRIORITY), .ignore_block_suitable = (prio == MIN_COMPACT_PRIORITY) }; - struct capture_control capc = { - .cc = &cc, - .page = NULL, - }; - /* - * Make sure the structs are really initialized before we expose the - * capture control, in case we are interrupted and the interrupt handler - * frees a page. - */ + /* See the comment in __alloc_pages_direct_compact() */ barrier(); - WRITE_ONCE(current->capture_control, &capc); + WRITE_ONCE(capc->cc, &cc); - ret = compact_zone(&cc, &capc); + ret = compact_zone(&cc, capc); + + WRITE_ONCE(capc->cc, NULL); - /* - * Make sure we hide capture control first before we read the captured - * page pointer, otherwise an interrupt could free and capture a page - * and we would leak it. - */ - WRITE_ONCE(current->capture_control, NULL); - *capture = READ_ONCE(capc.page); /* * Technically, it is also possible that compaction is skipped but * the page is still captured out of luck(IRQ came and freed the page). * Returning COMPACT_SUCCESS in such cases helps in properly accounting * the COMPACT[STALL|FAIL] when compaction is skipped. */ - if (*capture) + if (capc->page) ret = COMPACT_SUCCESS; return ret; @@ -2849,13 +2836,13 @@ static enum compact_result compact_zone_order(struct zone *zone, int order, * @alloc_flags: The allocation flags of the current allocation * @ac: The context of current allocation * @prio: Determines how hard direct compaction should try to succeed - * @capture: Pointer to free page created by compaction will be stored here + * @capc: The context for capturing pages during freeing * * This is the main entry point for direct page compaction. */ enum compact_result try_to_compact_pages(gfp_t gfp_mask, unsigned int order, unsigned int alloc_flags, const struct alloc_context *ac, - enum compact_priority prio, struct page **capture) + enum compact_priority prio, struct capture_control *capc) { struct zoneref *z; struct zone *zone; @@ -2883,7 +2870,7 @@ enum compact_result try_to_compact_pages(gfp_t gfp_mask, unsigned int order, } status = compact_zone_order(zone, order, gfp_mask, prio, - alloc_flags, ac->highest_zoneidx, capture); + alloc_flags, ac->highest_zoneidx, capc); rc = max(status, rc); /* The allocation should succeed, stop compacting */ diff --git a/mm/page_alloc.c b/mm/page_alloc.c index cb422505c6ef..9dee1c47e795 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -718,7 +718,7 @@ static inline struct capture_control *task_capc(struct zone *zone) { struct capture_control *capc = current->capture_control; - return unlikely(capc) && + return unlikely(capc && capc->cc) && !(current->flags & PF_KTHREAD) && !capc->page && capc->cc->zone == zone ? capc : NULL; @@ -4146,23 +4146,42 @@ __alloc_pages_direct_compact(gfp_t gfp_mask, unsigned int order, struct page *page = NULL; unsigned long pflags; unsigned int noreclaim_flag; + struct capture_control capc = { + .page = NULL, + }; if (!order) return NULL; + /* + * Make sure the structs are really initialized before we expose the + * capture control, in case we are interrupted and the interrupt handler + * frees a page. + */ + barrier(); + WRITE_ONCE(current->capture_control, &capc); + psi_memstall_enter(&pflags); delayacct_compact_start(); fs_reclaim_acquire(gfp_mask); noreclaim_flag = memalloc_noreclaim_save(); *compact_result = try_to_compact_pages(gfp_mask, order, alloc_flags, ac, - prio, &page); + prio, &capc); memalloc_noreclaim_restore(noreclaim_flag); fs_reclaim_release(gfp_mask); psi_memstall_leave(&pflags); delayacct_compact_end(); + /* + * Make sure we hide capture control first before we read the captured + * page pointer, otherwise an interrupt could free and capture a page + * and we would leak it. + */ + WRITE_ONCE(current->capture_control, NULL); + page = READ_ONCE(capc.page); + if (*compact_result == COMPACT_SKIPPED || *compact_result == COMPACT_DEFERRED) return NULL; -- 2.54.0