From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-wr1-f71.google.com (mail-wr1-f71.google.com [209.85.221.71]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 434713E8C74 for ; Sun, 26 Jul 2026 22:24:01 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.221.71 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1785104643; cv=none; b=u9IiSV6esb7lc2ENMvzBoy/R5342UmJTEKuiG7xTGKPH9FXylvq0cyix3FcRDxJk4aX7yCtbzgzc2iVUtk5uHnpi/R/9CAj6eAC0PG/uNc/DBsi5HewtgCdItigoxX2j2fBPCYwYglYsMK58QuVeNaMIicY1ccn0yjExXr876fo= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1785104643; c=relaxed/simple; bh=wmZncbJ/yHNyjHV19tVSPlNUkZcaqiWU2UDgtVD3YaE=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=STFA4xnBjixM6o7hy0Ut4TQWCxmJ+PTT9tvpVhd4o7EETVmwto1CpA0jNF7QTtiURsyBHaP3svxzHdWOqDBSQ+IJSMmQtoiKhPqrE/YP8+PvSqGumLWxn+wfo8WhNuylR0BUquvhfyhWtioDUusuttYta9zmS9Ji6KmwCnqfWwU= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--jackmanb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=l1lYKuOA; arc=none smtp.client-ip=209.85.221.71 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--jackmanb.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="l1lYKuOA" Received: by mail-wr1-f71.google.com with SMTP id ffacd0b85a97d-47f83416551so1911588f8f.3 for ; Sun, 26 Jul 2026 15:24:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1785104639; x=1785709439; darn=vger.kernel.org; h=content-type:cc:to:from:subject:message-id:references:mime-version :in-reply-to:date:from:to:cc:subject:date:message-id:reply-to :content-type; bh=sE7JJfWSnmXwNSqLQXzEAPpj56tITjEMwhehkr9vEgs=; b=l1lYKuOAM28hNGlRpTIrqCff3YQ+32PT9NZC0zYOpsNejRDvZiSDo8biZhaQt9iepM pr7oj1ZdVQS6QvHT/oA0ZDrWkwwLdv1mI0Fwh10Z2cgwZisxeF3g4ZdOubEq55gbJnd4 Bfx9sW23XnhMktasqSUI2iPEiAZMvZ4wIatGA5xse0+yhgxwRoxywFnKYpiAxBEk/lH3 GRdxGAItKyvaXr2cfEpV8mr5xqY+c4AM7vjIFKNAEQ1upLKgLP0lCke57KWeum55fRdb hNc85o6c1FDLnA/2ggDfh8zCFPfbL1ze+vzVoiTOaeV3rwtXo5bk3UvmsmNvoOTUqeID uNdQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1785104639; x=1785709439; h=content-type:cc:to:from:subject:message-id:references:mime-version :in-reply-to:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to:content-type; bh=sE7JJfWSnmXwNSqLQXzEAPpj56tITjEMwhehkr9vEgs=; b=r5Y5v4l/SCsO/9Nrp4IP6AShvqlQuEFnJQv0yQbsZa/zJVx611UDIOWS+fFuKC/g7E BB4k4A+fYBVBDw2nfHnR0mGzOsRM+SqzuMUvGY/gjnQn5/Qn4Ps7DJ6p2DvNWtCUP/gr ii1vfNvE2cI0ZihxVCkrKE/AjpSe5sOzUr2DSI+c9iC5CWxKOXewdiwXxza6d2XF1sVA 5fkKXsDSqPhY1Bnd6SztUKji02/BYnEj0Mn6dps+jkX1Tiz3je7ZrrfLk+HBHk0cihgO oqPQdkidup9auqGGp5qZ+wTm1V18n+irfMgntyuZ0lpo4uvB76BOPBjqSt/FBEf+e0Gm 1OCA== X-Forwarded-Encrypted: i=1; AHgh+Rp2CIZmt73dmtfR85/GSQyZIrChJz/+AYwHyhrJKPrROn7kmGQ+u4vPadMTjJ+MeOGscpaNL0MiPsEU3gU=@vger.kernel.org X-Gm-Message-State: AOJu0YxZi1QoZw3fuzaOE4NN3wBpR+E3fz6TBxsAkgXdth844boPiR1R T3XosyGhaCp1FapVIFp/1e2L6yFDfGEWy85bt2T9fAmP0WdfgnzPliYGmtKUHHk44szJ0ecZcMe ds9oMhFajbeZLKA== X-Received: from wmpg38.prod.google.com ([2002:a05:600c:4ca6:b0:495:49a0:9476]) (user=jackmanb job=prod-delivery.src-stubby-dispatcher) by 2002:a05:600c:1553:b0:493:e97c:216e with SMTP id 5b1f17b1804b1-496b5731987mr83419515e9.39.1785104639198; Sun, 26 Jul 2026 15:23:59 -0700 (PDT) Date: Sun, 26 Jul 2026 22:22:57 +0000 In-Reply-To: <20260726-page_alloc-unmapped-v3-0-6f5729aa9832@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20260726-page_alloc-unmapped-v3-0-6f5729aa9832@google.com> X-Mailer: b4 0.16-dev Message-ID: <20260726-page_alloc-unmapped-v3-24-6f5729aa9832@google.com> Subject: [PATCH v3 24/26] mm/page_alloc: always direct compact for unmapped allocs From: Brendan Jackman To: Borislav Petkov , Dave Hansen , Peter Zijlstra , Andrew Morton , David Hildenbrand , Vlastimil Babka , Mike Rapoport , Wei Xu , Johannes Weiner , Zi Yan , Lorenzo Stoakes Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, x86@kernel.org, rppt@kernel.org, Sumit Garg , Will Deacon , rientjes@google.com, "Kalyazin, Nikita" , patrick.roy@linux.dev, "Itazuri, Takahiro" , Andy Lutomirski , David Kaplan , Thomas Gleixner , Yosry Ahmed , Patrick Bellasi , Reiji Watanabe , Sean Christopherson , Brendan Jackman Content-Type: text/plain; charset="utf-8" This is the minimal solution for ensuring that compaction can service unmapped allocations. Without this, it's possible for compaction to just check watermarks and see plenty of free pages, without being aware of the direct map state, and thereby cause an ALLOC_UNMAPPED allocation to fail unnecessarily. Instead, with this change, promote compact_order to pageblock order for unmapped allocations, much like defrag_mode. Then, check specifically in compaction for the presence of wholly mapped blocks that can be unmapped once direct compact is complete. This all takes advantage of a major simplification: since unmapped blocks are currently always unmovable, this can be asymmetric. There is never a need to promote a !ALLOC_UNMAPPED allocation to compacting at pageblock_order, because compaction would be trying to generate a currently-unmapped block to map; that will always fail because it would require migrating unmapped pages, which is not supported at the moment. Signed-off-by: Brendan Jackman --- mm/compaction.c | 22 ++++++++++++++++++---- mm/page_alloc.c | 9 +++++++++ 2 files changed, 27 insertions(+), 4 deletions(-) diff --git a/mm/compaction.c b/mm/compaction.c index ed12d2fc6fad3..fe1aaf293bbce 100644 --- a/mm/compaction.c +++ b/mm/compaction.c @@ -2531,12 +2531,25 @@ bool compaction_zonelist_suitable(struct alloc_context *ac, int order, static enum compact_result compaction_suit_allocation_order(struct zone *zone, unsigned int order, int highest_zoneidx, unsigned int alloc_flags, - bool async, bool kcompactd) + bool unmapped, bool async, bool kcompactd) { unsigned long free_pages; unsigned long watermark; - if (kcompactd && defrag_mode) + /* + * When trying to generate an unmapped block, check the counter for + * direct-mapped blocks specifically, since we'll need to unmap the + * whole block to service the allocation. + * + * Why doesn't this apply to the other way around too? (Mightn't we need + * to _map_ a whole block, to service a !ALLOC_UNMAPPED allocation?) No, + * because of a likely-temporary simplification: currently, unmapped + * blocks never contain movable pages, so compaction isn't going to free + * up one of those. + */ + if (unmapped) + free_pages = zone_page_state(zone, NR_FREE_PAGES_BLOCKS_MAPPED); + else if (kcompactd && defrag_mode) free_pages = zone_free_pages_blocks(zone); else free_pages = zone_page_state(zone, NR_FREE_PAGES); @@ -2599,6 +2612,7 @@ compact_zone(struct compact_control *cc, struct capture_control *capc) ret = compaction_suit_allocation_order(cc->zone, cc->order, cc->highest_zoneidx, cc->alloc_flags, + freetype_unmapped(cc->freetype), cc->mode == MIGRATE_ASYNC, !cc->direct_compaction); if (ret != COMPACT_CONTINUE) @@ -3084,7 +3098,7 @@ static bool kcompactd_node_suitable(pg_data_t *pgdat) ret = compaction_suit_allocation_order(zone, pgdat->kcompactd_max_order, highest_zoneidx, alloc_flags, - false, true); + false, false, true); if (ret == COMPACT_CONTINUE) return true; } @@ -3127,7 +3141,7 @@ static void kcompactd_do_work(pg_data_t *pgdat) ret = compaction_suit_allocation_order(zone, cc.order, zoneid, cc.alloc_flags, - false, true); + false, false, true); if (ret != COMPACT_CONTINUE) continue; diff --git a/mm/page_alloc.c b/mm/page_alloc.c index d12ce84662ab7..5f1dea7eee15b 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -827,6 +827,9 @@ compaction_capture(struct capture_control *capc, struct page *page, capc_mt != MIGRATE_MOVABLE) return false; + if (freetype_flags(freetype) != freetype_flags(capc->freetype)) + return false; + if (migratetype != capc_mt) trace_mm_page_alloc_extfrag(page, capc->order, order, capc_mt, migratetype); @@ -4523,6 +4526,12 @@ __alloc_pages_direct_compact(gfp_t gfp_mask, unsigned int order, if ((alloc_flags & ALLOC_NOFRAGMENT) && free_to_migratetype(ac->freetype) != MIGRATE_MOVABLE) compact_order = max(order, pageblock_order); + /* + * Unmapped allocations benefit from compaction even at order 0, because the + * allocator will actually grab a whole block. + */ + if (freetype_flags(ac->freetype) & FREETYPE_UNMAPPED) + compact_order = max(order, pageblock_order); if (!compact_order) return NULL; -- 2.54.0