From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-wm1-f73.google.com (mail-wm1-f73.google.com [209.85.128.73]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 2D5E634F247 for ; Mon, 29 Jun 2026 11:21:02 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.73 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1782732063; cv=none; b=MA3mz2pVJ9tQv/KOLKlheZRRuszr9PMxF4Df8JNiyf5pCF2l96cVFU6TLG+qyZoF+eRojy7VLfc5Wt/2ul6DE49SMBXQVLUPj7yEzISTgBCfyGK5CB/3zaDpiEFfxL4Wo/M/5jZmHJlxqhPB1zm3nkgZumVVX3XfRWhT5AuQQnU= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1782732063; c=relaxed/simple; bh=A5F1/Hn/HEx4WQ/VgCDcVrYRf8EdAH5Cykzu2qKSGXs=; h=Date:Mime-Version:Message-ID:Subject:From:To:Cc:Content-Type; b=SO7fXR6QC3F/wZTdKzDxztnx5JEuEgovbYT2ebJVs26SJpnuGYub/RnkUDzu2yutbTQorz1b9ZXdagJNGwcK8R+aA2u+IkFCYZv6wVvCyRBw8yeX4ErWjATkpU1Ljd6Pb2jkuYps+Mmeh7jS0FU8iES9h5yxBos803e3ZfMH1yg= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--jackmanb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=h7W9Edez; arc=none smtp.client-ip=209.85.128.73 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--jackmanb.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="h7W9Edez" Received: by mail-wm1-f73.google.com with SMTP id 5b1f17b1804b1-493b0fe95b6so5158355e9.1 for ; Mon, 29 Jun 2026 04:21:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1782732060; x=1783336860; darn=vger.kernel.org; h=cc:to:from:subject:message-id:mime-version:date:from:to:cc:subject :date:message-id:reply-to; bh=mbDCXnvOe0PS+9eULlKq61NFwxP69D7MKsKlxC6XXmI=; b=h7W9Edez8b3niT39EmXk4Rnd3ufS5jfYnIjbiSQf/8DxETluaz6Tpt55kNalydnblO 6B0Se6iE3T5FSSf6v2pN81jNqo2rXqDufj4oBafhtyrvDDSKcDYN7SzMMw2g4XKyQE2+ /mSuVVVkqDm5M4zkYQf/gkKhvzF+mSFDkLInyK+lD1oHax17D5Evv01DLvx59t46tMFk rcaQ0Q9z9D9+fvZ3vgt74YBIm75DtIVRkyGKyQZGp6mFdWIuIQfEHIzdNl3aHWguGTuA nS+MJs3QLAU9iw48n28j/t8Fj4eN398U7cWlw5XgMSmU+xAiSaVfCEzciAXNXeHADjCc LgqA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1782732060; x=1783336860; h=cc:to:from:subject:message-id:mime-version:date:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=mbDCXnvOe0PS+9eULlKq61NFwxP69D7MKsKlxC6XXmI=; b=LCXYVmeldsq2HTKUS3Kh2jXWbI8PfjpAXjNZ4NO7vElUnnVmdsZJ0vYMKkKyZtKAth CKEE+j7HWuivPLDSO6Rmyru3FptlCfswcCJIV33PoNGUuh2wQsui0ghpfDjbgndwVa7M 5wRiRAgpXoZGn8hEy0NMhxtV6hWZQkqZ9jhuXRM/ZnfBTS5rNYAxTGcwR43j8NGovhKv 4eJfZ628Pojxk+cbZHwkWp/fTNeUOAN/1XVTHw+IU8H5tDDJOBwNPajCQPiYRjIY3yk0 QNgNlyxW+2g6f3d86Mw0/nYGgyoBiW4SCJJbaDv1QWwJ7+Ej1f7QPAQexpvdyO9iEmMR QdbQ== X-Forwarded-Encrypted: i=1; AFNElJ9ljD30gNv808tJzfg5imeosNuRM/q2IItTlSpfNnRzlaBwwuXCHRrXAtNAQL8eCGwSsSirGidC9EFovqE=@vger.kernel.org X-Gm-Message-State: AOJu0Yz/6YhgZyPW849bSUaojmdCUxDG21bJfMQdGlf8IgXxZFW9GXT+ ZLEXZOtzHgiMC+LplDsKKLL+4GoWXv9nRR4jF/deOR9xlYe3ZtRaN0X6R3BMIhmOZcMapWZ4/17 IrNF08d7732uNCg== X-Received: from wmnj20.prod.google.com ([2002:a05:600c:754:b0:493:b691:8628]) (user=jackmanb job=prod-delivery.src-stubby-dispatcher) by 2002:a05:600c:e54a:20b0:492:6093:8837 with SMTP id 5b1f17b1804b1-4926689a4d1mr175771275e9.34.1782732060367; Mon, 29 Jun 2026 04:21:00 -0700 (PDT) Date: Mon, 29 Jun 2026 11:20:49 +0000 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 X-B4-Tracking: v=1; b=H4sIABBVQmoC/33NsQ6CMBAG4FchN1tDC63o5CCrg47GgcIBlwglL Wk0hHe36aSJcfzv/vtuAYeW0MEhWcCiJ0dmDEFsEqj7auyQURMyiFSoVHHBunZiEzpHA7lqDm2 mhSyqXGuJuoFwN1ls6RnNG1zKa3k+wT3Me3Kzsa/4yvO4/aN6zjjbZwp1moca7o6dMd0Dt7UZI ufFJyF/ESIQSrY843tUBS++iHVd30YGlHX+AAAA X-Change-Id: 20260612-gfp-pessimisation-b258a4bb5ebd X-Mailer: b4 0.15.2 Message-ID: <20260629-gfp-pessimisation-v2-1-311ece6a8637@google.com> Subject: [PATCH RESEND v2] mm/page_alloc: drop flag-conversion "optimisation" From: Brendan Jackman To: Andrew Morton , Vlastimil Babka , Suren Baghdasaryan , Michal Hocko , Johannes Weiner , Zi Yan Cc: "Harry Yoo (Oracle)" , Gregory Price , Johannes Weiner , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Brendan Jackman Content-Type: text/plain; charset="utf-8" This code uses flag equivalences to try to optimise conversion from GFP_ to ALLOC_ but there's no clear reason to believe it makes things faster. Even if it gets rid of conditional branches, it just trades them for a data dependency. CPUs are pretty good at conditional branches. But, in my GCC x86 build it doesn't look like there are any branches anyway, the compiler found some conditional instruction tricks. (Caveat: This was extracted & annotated by Gemini AI, I did not actually read the disasm myself) Old code: ae50: 8b 04 24 mov (%rsp),%eax # Load gfp_mask ... ae5d: 41 89 c4 mov %eax,%r12d ae64: 41 81 e4 20 08 00 00 and $0x820,%r12d # Mask both flags at once ... ae6f: 44 89 e1 mov %r12d,%ecx ae77: 83 c9 40 or $0x40,%ecx # OR with ALLOC_CPUSET (0x40) ae7a: 89 4c 24 60 mov %ecx,0x60(%rsp) # Store to alloc_flags New code: For __GFP_HIGH ( 0x20 ): It uses the Carry Flag (via sbb ) to conditionally add 0x20 to the base 0x40 ( ALLOC_CPUSET ) flag: ae63: 83 e0 20 and $0x20,%eax # Test __GFP_HIGH ... ae6a: 83 f8 01 cmp $0x1,%eax # Set carry flag if 0 ae6f: 45 19 e4 sbb %r12d,%r12d # %r12d = (gfp & 0x20) ? 0 : -1 ae80: 41 83 e4 e0 and $0xffffffe0,%r12d # %r12d = (gfp & 0x20) ? 0 : -32 ae87: 41 83 c4 60 add $0x60,%r12d # %r12d = (gfp & 0x20) ? 0x60 : 0x40 For __GFP_KSWAPD_RECLAIM ( 0x800 ): It uses a conditional move ( cmov ) later in the function to set the ALLOC_KSWAPD ( 0x800 ) bit: ae72: 25 00 08 00 00 and $0x800,%eax # Test __GFP_KSWAPD_RECLAIM ae77: 89 44 24 30 mov %eax,0x30(%rsp) # Store result ... af2c: 80 cf 08 or $0x8,%bh # Set ALLOC_KSWAPD (0x800) in temp reg af2f: 45 85 c9 test %r9d,%r9d # Check if __GFP_KSWAPD_RECLAIM was set af32: 0f 44 d8 cmove %eax,%ebx # If not, revert to flags without it Testing with a modified version[0] of lib/free_pages_test.c (adding printks with timing)... [0] https://github.com/bjackman/aethelred/blob/2ccdc84ef087c2a631914f58e106e99e19bd3b98/page-alloc-test/page-alloc-test.c Old results from a Sapphire Rapids consumer CPU: [ 67.157118] page_alloc_test: Testing with GFP_KERNEL [ 67.157122] page_alloc_test: Starting 1,000,000 allocations... [ 70.704446] page_alloc_test: Completed. Time: 3543002 us (Avg: 3543.00 ns per alloc+free loop) [ 70.704456] page_alloc_test: Testing with GFP_KERNEL | __GFP_COMP [ 70.704460] page_alloc_test: Starting 1,000,000 allocations... [ 70.944672] page_alloc_test: Completed. Time: 239980 us (Avg: 239.98 ns per alloc+free loop) [ 70.944675] page_alloc_test: Test completed New results: [ 70.079015] page_alloc_test: Testing with GFP_KERNEL [ 70.079020] page_alloc_test: Starting 1,000,000 allocations... [ 73.669396] page_alloc_test: Completed. Time: 3586954 us (Avg: 3586.95 ns per alloc+free loop) [ 73.669402] page_alloc_test: Testing with GFP_KERNEL | __GFP_COMP [ 73.669405] page_alloc_test: Starting 1,000,000 allocations... [ 73.905084] page_alloc_test: Completed. Time: 235496 us (Avg: 235.49 ns per alloc+free loop) [ 73.905086] page_alloc_test: Test completed Seems like a wash. So, drop the flag value coupling here and let the compiler and CPU do their job. Superscalar CPUs are pretty neat after all. (Used AI for the disasm but the rest is all manual). Reviewed-by: Vlastimil Babka (SUSE) Reviewed-by: Zi Yan Reviewed-by: Gregory Price Acked-by: Johannes Weiner Acked-by: Harry Yoo (Oracle) Signed-off-by: Brendan Jackman --- Resending as my prior v2 was during the "quiet period". Note I have added the Reviewed-by/Acked-by tags without bumping the revision, since nothing else about the patch or commit message has changed. Please let me know if that's annoying. --- Changes in v2: - Updated alloc_flags_nofragment() too. - Link to v1: https://lore.kernel.org/r/20260612-gfp-pessimisation-v1-1-936eb04202e7@google.com To: Andrew Morton To: Vlastimil Babka To: Suren Baghdasaryan To: Michal Hocko To: Brendan Jackman To: Johannes Weiner To: Zi Yan Cc: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org --- mm/page_alloc.c | 23 +++++++---------------- 1 file changed, 7 insertions(+), 16 deletions(-) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index f87e8c0a0c367..d60053a3ffe4d 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -3742,13 +3742,10 @@ static bool zone_allows_reclaim(struct zone *local_zone, struct zone *zone) static inline unsigned int alloc_flags_nofragment(struct zone *zone, gfp_t gfp_mask) { - unsigned int alloc_flags; + unsigned int alloc_flags = 0; - /* - * __GFP_KSWAPD_RECLAIM is assumed to be the same as ALLOC_KSWAPD - * to save a branch. - */ - alloc_flags = (__force int) (gfp_mask & __GFP_KSWAPD_RECLAIM); + if (gfp_mask & __GFP_KSWAPD_RECLAIM) + alloc_flags |= ALLOC_KSWAPD; if (defrag_mode) { alloc_flags |= ALLOC_NOFRAGMENT; @@ -4504,22 +4501,16 @@ gfp_to_alloc_flags(gfp_t gfp_mask, unsigned int order) { unsigned int alloc_flags = ALLOC_WMARK_MIN | ALLOC_CPUSET; - /* - * __GFP_HIGH is assumed to be the same as ALLOC_MIN_RESERVE - * and __GFP_KSWAPD_RECLAIM is assumed to be the same as ALLOC_KSWAPD - * to save two branches. - */ - BUILD_BUG_ON(__GFP_HIGH != (__force gfp_t) ALLOC_MIN_RESERVE); - BUILD_BUG_ON(__GFP_KSWAPD_RECLAIM != (__force gfp_t) ALLOC_KSWAPD); - /* * The caller may dip into page reserves a bit more if the caller * cannot run direct reclaim, or if the caller has realtime scheduling * policy or is asking for __GFP_HIGH memory. GFP_ATOMIC requests will * set both ALLOC_NON_BLOCK and ALLOC_MIN_RESERVE(__GFP_HIGH). */ - alloc_flags |= (__force int) - (gfp_mask & (__GFP_HIGH | __GFP_KSWAPD_RECLAIM)); + if (gfp_mask & __GFP_HIGH) + alloc_flags |= ALLOC_MIN_RESERVE; + if (gfp_mask & __GFP_KSWAPD_RECLAIM) + alloc_flags |= ALLOC_KSWAPD; alloc_flags |= gfp_to_alloc_flags_nonblocking(gfp_mask, order); --- base-commit: fe61612214b618072b4ea3f5065a81296df6dd75 change-id: 20260612-gfp-pessimisation-b258a4bb5ebd Best regards, -- Brendan Jackman