From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-qv1-f46.google.com (mail-qv1-f46.google.com [209.85.219.46]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0990737702E for ; Wed, 22 Jul 2026 15:00:17 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.219.46 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784732420; cv=none; b=J8x9Qr3exnMi23MiOufcQE4NnUm03z0A6lzUI6CobiolrdmVrXm1Pwjon/oZNt5cE0+xityaqsq8rKcBAqceuFH7fzSEwJmUrWW/yvT7VubFyJwfS5LYQHH8EkTv8l08RapnNr12eg/8bnNHc0sGd1mWdy3N6U3B9ytAIudD1Uw= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784732420; c=relaxed/simple; bh=OGdOpi7gb6g7qGXkw7ZMzS314GNbfqv/MOKmMqeaXwc=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=NtC44QTZVHLclcRCf2ygDsnZ4Y4z008skZHP2jt0Uqxk2a8NeMSxUo/blM/TSQLazZj1q2hkrvUJ41a+ZGngbG4uLs+hNGvZtRYPZoAsAu9Q967uhmJa5GtwGhhQQ2txfyF96Xf9AKIXghRSp4ZYYUWkw1DPGN5vu4yvj1nCFC8= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=cmpxchg.org; spf=pass smtp.mailfrom=cmpxchg.org; dkim=pass (2048-bit key) header.d=cmpxchg.org header.i=@cmpxchg.org header.b=BWMdfCoq; arc=none smtp.client-ip=209.85.219.46 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=cmpxchg.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=cmpxchg.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=cmpxchg.org header.i=@cmpxchg.org header.b="BWMdfCoq" Received: by mail-qv1-f46.google.com with SMTP id 6a1803df08f44-8efbafa1bacso104245116d6.1 for ; Wed, 22 Jul 2026 08:00:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cmpxchg.org; s=google; t=1784732417; x=1785337217; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=9w4/w+DgNeKgvmYqz2X0YQytmiACV0MWUeK4q1M7V1c=; b=BWMdfCoq4Z6uUgHmHQoNui7Mnp1xqHnQDeRKWzrSoTjHCQ+UQ5429qCMdj/dWEQLE5 LnK51KbF+VwEzzog8vtW5e/3/ijIZLF0BSEq+qgtt/HEGYmQPTB1e3o9xs3422fFOIwZ stpeC7+hNNl+09XtueSKOUW/czAgVCbQFNtxfzjG2aH8blumbT1uKcwDintRmSd7jI17 j+iDDAXLoz6gqQY+f9jOjeW0R5VbmtTh12Yp68Qm5lf2g4Ob537BEYvDlFdjWWzRA+OP MWBJizEnORj9/RhbjJXLMgy4ckEoa8+bFDNWc+8NMQk84kDad+6mShL6DZF0NHfa8YSD gUhA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1784732417; x=1785337217; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=9w4/w+DgNeKgvmYqz2X0YQytmiACV0MWUeK4q1M7V1c=; b=nanPVfzCguWM/JxIO7QOCLXevEpPux2fCd5pH2RDDOHAf/J/YhMdrK+cR7PGucNfnS OGForsIeOD8LGdM2TIShS6lNz7HGMEqqCAVzk10Hvnzy07bB8rXfPBouR7vH08S8Tjq9 hnAm4D+8fp9dDAaCftf6eqJVrguhzgMCWv97NudlGLQcofN+yE4CFLNtigPYgbA4yjRo rH7Ga+RYV19FM6jJBpAViCbfPNrHdgM0ZqArEG/yWpapNkdz7SK5tBvjJsX2b0ZNfAa9 4dB+VHQMFF8fwN6lZ92Qqj5nADDm1qEBDBKR8lHil8klWZSVc9l6wxmBUUSuHxQ5ZQpB pnMA== X-Forwarded-Encrypted: i=1; AHgh+RpuGaACIn+UwMbUppiiTfWlhCLWz6fVLw3d5fVAanmckYju16m0l/abqYy+rQGLqDCAp9MtFifbuFcHIz4=@vger.kernel.org X-Gm-Message-State: AOJu0YzxDNAyAS8MPb4vt+5VflS9aLRuvk607xVWUwVgIyFUtQM6TdX6 8l9QrOSdudg1nHEKmEUXXWqEbrNQJaNd6d4PGH350f4uCd/o4Hei9vlQ3GMelf/RvlU= X-Gm-Gg: AR+sD12NgyriEQlikEYMzHge/jlqlXO6TJJG2QfFsdVrQc74n3tk7HdH5Q+oJKMn05f aHViTZxKV15s4nJPmt3y4F0QridywDFfDFtvQAwU5DnJrxb8oFhm1JK+IfWNLbY5kdS1V4tms4N IQC2jlfvPI9KpR39lyTkV9EFASMk34mMNK2hkZNcssW9e/lfCG5CTuxbrRap+FSx4zNvOJxI9Er 1Tg+Us2odX8ZZgPTXPJHDXiPKisrIjmQtfRaPyl881M+o4ezDg2//BBZkwaySIVfMkSPObybg7C y/MKx66bLL9FM6J+s1BkrgiRtM541dJqU3eQLQB6H55BpevH2fDvbXMYREFTUbt1jwGO7lzUnpL w7dFNhCHFl1MkdAxUywU4rDLoDq1YUgfR6wHRra/hpRnLr/kgxwdu4KyNcgEULvrQudcAlTjZzK ljrUIx4rYJOho= X-Received: by 2002:a05:6214:3d0d:b0:8ea:184f:c15a with SMTP id 6a1803df08f44-9077836d21amr240711036d6.17.1784732415754; Wed, 22 Jul 2026 08:00:15 -0700 (PDT) Received: from localhost ([2603:7001:f100:500:365a:60ff:fe62:ff29]) by smtp.gmail.com with ESMTPSA id 6a1803df08f44-907baa0ef54sm22285326d6.39.2026.07.22.08.00.14 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 22 Jul 2026 08:00:15 -0700 (PDT) From: Johannes Weiner To: Andrew Morton , Vlastimil Babka Cc: Suren Baghdasaryan , Michal Hocko , Brendan Jackman , Zi Yan , David Hildenbrand , Lorenzo Stoakes , "Liam R . Howlett" , Mike Rapoport , Shakeel Butt , linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v2 3/4] mm: page_alloc: move capture_control to the page allocator Date: Wed, 22 Jul 2026 10:56:46 -0400 Message-ID: <20260722150006.3848560-4-hannes@cmpxchg.org> X-Mailer: git-send-email 2.55.0 In-Reply-To: <20260722150006.3848560-1-hannes@cmpxchg.org> References: <20260722150006.3848560-1-hannes@cmpxchg.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit From: "Vlastimil Babka (SUSE)" The compaction capturing code assumes the allocation request order and compaction target order are the same. That won't be true once defrag_mode promotes sub-block allocations to pageblock-order compaction: compaction targets the larger order, while capture should remain at the original allocation order. Move the capture_control to the page allocator and give it its own copies of what the page freeing path matches against - zone, migratetype and the allocation order - rather than reaching into compaction's live compact_control. __alloc_pages_direct_compact() fills in migratetype and order, and installs and hides current->capture_control around the whole compaction call; try_to_compact_pages() aims capc->zone at each zone while it is being compacted. compact_zone_order() no longer deals with capture at all. Pass the capture_control through try_to_compact_pages() / compact_zone_order() in place of the bare struct page **. No functional change. Signed-off-by: Vlastimil Babka (SUSE) Co-developed-by: Johannes Weiner Signed-off-by: Johannes Weiner --- include/linux/compaction.h | 3 ++- mm/compaction.c | 50 +++++++++++--------------------------- mm/internal.h | 4 ++- mm/page_alloc.c | 45 ++++++++++++++++++++++++++++------ 4 files changed, 57 insertions(+), 45 deletions(-) diff --git a/include/linux/compaction.h b/include/linux/compaction.h index f29ef0653546..66a2f70e9e01 100644 --- a/include/linux/compaction.h +++ b/include/linux/compaction.h @@ -58,6 +58,7 @@ enum compact_result { }; struct alloc_context; /* in mm/internal.h */ +struct capture_control; /* in mm/internal.h */ /* * Number of free order-0 pages that should be available above given watermark @@ -92,7 +93,7 @@ extern int fragmentation_index(struct zone *zone, unsigned int order); extern enum compact_result try_to_compact_pages(gfp_t gfp_mask, unsigned int order, unsigned int alloc_flags, const struct alloc_context *ac, enum compact_priority prio, - struct page **page); + struct capture_control *capc); extern void reset_isolation_suitable(pg_data_t *pgdat); extern bool compaction_suitable(struct zone *zone, int order, unsigned long watermark, int highest_zoneidx); diff --git a/mm/compaction.c b/mm/compaction.c index 225862a00380..977e30b3a2bd 100644 --- a/mm/compaction.c +++ b/mm/compaction.c @@ -2802,9 +2802,8 @@ compact_zone(struct compact_control *cc, struct capture_control *capc) static enum compact_result compact_zone_order(struct zone *zone, int order, gfp_t gfp_mask, enum compact_priority prio, unsigned int alloc_flags, int highest_zoneidx, - struct page **capture) + struct capture_control *capc) { - enum compact_result ret; struct compact_control cc = { .order = order, .search_order = order, @@ -2819,38 +2818,8 @@ static enum compact_result compact_zone_order(struct zone *zone, int order, .ignore_skip_hint = (prio == MIN_COMPACT_PRIORITY), .ignore_block_suitable = (prio == MIN_COMPACT_PRIORITY) }; - struct capture_control capc = { - .cc = &cc, - .page = NULL, - }; - - /* - * Make sure the structs are really initialized before we expose the - * capture control, in case we are interrupted and the interrupt handler - * frees a page. - */ - barrier(); - WRITE_ONCE(current->capture_control, &capc); - ret = compact_zone(&cc, &capc); - - /* - * Make sure we hide capture control first before we read the captured - * page pointer, otherwise an interrupt could free and capture a page - * and we would leak it. - */ - WRITE_ONCE(current->capture_control, NULL); - *capture = READ_ONCE(capc.page); - /* - * Technically, it is also possible that compaction is skipped but - * the page is still captured out of luck(IRQ came and freed the page). - * Returning COMPACT_SUCCESS in such cases helps in properly accounting - * the COMPACT[STALL|FAIL] when compaction is skipped. - */ - if (*capture) - ret = COMPACT_SUCCESS; - - return ret; + return compact_zone(&cc, capc); } /** @@ -2860,13 +2829,13 @@ static enum compact_result compact_zone_order(struct zone *zone, int order, * @alloc_flags: The allocation flags of the current allocation * @ac: The context of current allocation * @prio: Determines how hard direct compaction should try to succeed - * @capture: Pointer to free page created by compaction will be stored here + * @capc: Free page capture bypassing the freelist * * This is the main entry point for direct page compaction. */ enum compact_result try_to_compact_pages(gfp_t gfp_mask, unsigned int order, unsigned int alloc_flags, const struct alloc_context *ac, - enum compact_priority prio, struct page **capture) + enum compact_priority prio, struct capture_control *capc) { struct zoneref *z; struct zone *zone; @@ -2893,8 +2862,17 @@ enum compact_result try_to_compact_pages(gfp_t gfp_mask, unsigned int order, continue; } + WRITE_ONCE(capc->zone, zone); + status = compact_zone_order(zone, order, gfp_mask, prio, - alloc_flags, ac->highest_zoneidx, capture); + alloc_flags, ac->highest_zoneidx, capc); + + WRITE_ONCE(capc->zone, NULL); + + /* Stop if a page has been captured */ + if (READ_ONCE(capc->page)) + status = COMPACT_SUCCESS; + rc = max(status, rc); /* The allocation should succeed, stop compacting */ diff --git a/mm/internal.h b/mm/internal.h index 181e79f1d6a2..6d3402001b93 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -1059,7 +1059,9 @@ struct compact_control { * immediately when one is created during the free path. */ struct capture_control { - struct compact_control *cc; + struct zone *zone; + int migratetype; + int order; struct page *page; }; diff --git a/mm/page_alloc.c b/mm/page_alloc.c index cb422505c6ef..4b9686fedb71 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -721,14 +721,14 @@ static inline struct capture_control *task_capc(struct zone *zone) return unlikely(capc) && !(current->flags & PF_KTHREAD) && !capc->page && - capc->cc->zone == zone ? capc : NULL; + capc->zone == zone ? capc : NULL; } static inline bool compaction_capture(struct capture_control *capc, struct page *page, int order, int migratetype) { - if (!capc || order != capc->cc->order) + if (!capc || order != capc->order) return false; /* Do not accidentally pollute CMA or isolated regions*/ @@ -744,12 +744,12 @@ compaction_capture(struct capture_control *capc, struct page *page, * have trouble finding a high-order free page. */ if (order < pageblock_order && migratetype == MIGRATE_MOVABLE && - capc->cc->migratetype != MIGRATE_MOVABLE) + capc->migratetype != MIGRATE_MOVABLE) return false; - if (migratetype != capc->cc->migratetype) - trace_mm_page_alloc_extfrag(page, capc->cc->order, order, - capc->cc->migratetype, migratetype); + if (migratetype != capc->migratetype) + trace_mm_page_alloc_extfrag(page, capc->order, order, + capc->migratetype, migratetype); capc->page = page; return true; @@ -4146,6 +4146,12 @@ __alloc_pages_direct_compact(gfp_t gfp_mask, unsigned int order, struct page *page = NULL; unsigned long pflags; unsigned int noreclaim_flag; + struct capture_control capc = { + .zone = NULL, + .migratetype = ac->migratetype, + .order = order, + .page = NULL, + }; if (!order) return NULL; @@ -4155,8 +4161,33 @@ __alloc_pages_direct_compact(gfp_t gfp_mask, unsigned int order, fs_reclaim_acquire(gfp_mask); noreclaim_flag = memalloc_noreclaim_save(); + /* + * Make sure the structs are really initialized before we expose the + * capture control, in case we are interrupted and the interrupt handler + * frees a page. + */ + barrier(); + WRITE_ONCE(current->capture_control, &capc); + *compact_result = try_to_compact_pages(gfp_mask, order, alloc_flags, ac, - prio, &page); + prio, &capc); + + /* + * Make sure we hide capture control first before we read the captured + * page pointer, otherwise an interrupt could free and capture a page + * and we would leak it. + */ + WRITE_ONCE(current->capture_control, NULL); + page = READ_ONCE(capc.page); + + /* + * Technically, it is also possible that compaction is skipped but + * the page is still captured out of luck(IRQ came and freed the page). + * Returning COMPACT_SUCCESS in such cases helps in properly accounting + * the COMPACT[STALL|FAIL] when compaction is skipped. + */ + if (page) + *compact_result = COMPACT_SUCCESS; memalloc_noreclaim_restore(noreclaim_flag); fs_reclaim_release(gfp_mask); -- 2.55.0