From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-qk1-f180.google.com (mail-qk1-f180.google.com [209.85.222.180]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 350191E2858 for ; Thu, 18 Jun 2026 18:35:30 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.222.180 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1781807733; cv=none; b=PyRsn6Rk/6nZIPd9XTSoRx27KrKclToQirl6GXASHEr/b+KYKiZ81Ub0B4EjK+UopsAn/ieoJJq6EPOnuYeX9NhP/wjA7rIRIT3iqDqUyX7AqJQOd1+fj24cL5scq9p+i3A4rgaT4FV1JquOZk1XQdJrnhHa1nbLCPVK1AA6NgY= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1781807733; c=relaxed/simple; bh=V8m1+R9A9dAIF/G0agS9zupnEltnZBFDDp9zuuYIcMU=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=SJ41e5Hns006vW1drzSRJ/2N8cIKHCf8vP36RAKRoKITM/jjzbGPcHoPb2ffp5RfewHEpdz/fBq+KEhXvueZerRKVDpXWtl0bIBLjcMBzo78QeL5y3K8AUv3Sq0d+tRfeFdhH2k5vVGed15Fa+ktFO5HilOxh7Rpir4GrhpwRD0= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=cmpxchg.org; spf=pass smtp.mailfrom=cmpxchg.org; dkim=pass (2048-bit key) header.d=cmpxchg.org header.i=@cmpxchg.org header.b=f8hZKBC7; arc=none smtp.client-ip=209.85.222.180 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=cmpxchg.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=cmpxchg.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=cmpxchg.org header.i=@cmpxchg.org header.b="f8hZKBC7" Received: by mail-qk1-f180.google.com with SMTP id af79cd13be357-91578122305so179798185a.0 for ; Thu, 18 Jun 2026 11:35:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cmpxchg.org; s=google; t=1781807730; x=1782412530; darn=vger.kernel.org; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=jbQc+XBYiLsPD08oE5SvttVMwckAlDDcKCSnF5heN+s=; b=f8hZKBC7y6QTm3FAyMe6nfQlomXZytvw+P+Thnm9FtzB7RviIifATNYuadPdxJEDvR lsXKHUS219I6eEvxj7SHcuHn6RLGePf7L9A3a0MRAf6ejoHA0/cFzcuDrho7Q5vQsTfl oxFDhkoeE1OGsUzT9w7LdELBwbp5tE4tLk5d19gCSrTocfB71PusHtyDj2RfDNXsyrUL VCKSd7S1n/ihUaRlSRm2x0n8PfSP8lLwiiMoo307DhY16fftVmJoLils4KR5YuAxcGzY nk0iODpkRq4rgwT2UR2D8GJ1AMOhe7McsTnofGiM1oG4Tt0DV//L/uBopQ+qGzcCesJ7 UYIw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1781807730; x=1782412530; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-gg:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=jbQc+XBYiLsPD08oE5SvttVMwckAlDDcKCSnF5heN+s=; b=cuOgsvQc51S28072RxHEtsM5xLp71Burj0TYCG3Wah5swlQMNYPsm2DopdnlYUiq1H 699RVX5CLLrW1mpDzYyJhyHOBrwJN8LiDuzl2LfMf6juh3Qu/mvv25ehT8hzZVkfrfHu 000+Thr3qOQvmA1XgZ8sQJ7eswEeBUw80Ovzx2eZ5tSApukofOxENrJOdbgvCU9KLWP1 coWQr9LXcIobCfDgtS4uUk0NXc2rF5ZvWXCnFT6NY0JtNfGwaFKGmR8X8oHdERkjTazX X6wtSdlpgYSEcb5JMLcHJzFnQ9KMmE14XfBUi/kjcLonX/oB08VPdcf1xK1Jk8oFitgu TGTg== X-Forwarded-Encrypted: i=1; AFNElJ+yHzkEjmJi2TiiCx6yoGt/u/Z0SddzF0LHmNsHshvpIlgO3ilqeDY7HS0+xkFq9z27inzLGsF3rTIxCCA=@vger.kernel.org X-Gm-Message-State: AOJu0Yzasga6Xk1PWaW46xiIIJ3UCA4Zpc1sCXPORQmlq+QO4HYZnbQI zTvu++fM/hORS+UrXE2XfLq42rs4kkEQniuLKzt0AKJgjUaAUqPPR6mXWp4KICBnwsk= X-Gm-Gg: AfdE7claqOSdjYWwV+Y0p8t9FxdyO7jmHpQnBxSUA9CLqUnwx34Z9tweXVJ6jlFG2qx t5PSskX8tMXV/4YuCsXJPHLd9zDc7BxBCsWiQw5+EdTi8/0CpBctA2Z6GLyywt2TM9FYecoKrOQ 8/Fj0UoItehpqWYLDNlZBelnffGUs2EXXbXuXuOaobZ0X+61e4Z7KeicVLf5ETBAhyk+mRyXBZ8 YSXMx4wBeDZsmU4dw63oHuS6RqJjL7HzWyV3upvfgAOy7ym7dmJkrmc5sR8MgpTvdrr5KTUEq6X LrLlQBMnzg5YPg7lkGBK30QYrDzfCPDEegapdMitkjV1w8dPBx9ggtH/zV1bZbSX/6BSUfSaiAi IzvpwbGiK7bRYxy8Q86+n5VD41rL5TtWAXP2N13qQuPsO0NyU7ZjBSkXR++5gSyQJfxAQhM0X01 A2JESX3bR5K5s= X-Received: by 2002:a05:620a:27d5:b0:915:aa7d:4295 with SMTP id af79cd13be357-9208f161133mr27116385a.12.1781807729640; Thu, 18 Jun 2026 11:35:29 -0700 (PDT) Received: from localhost ([2603:7001:f100:500:365a:60ff:fe62:ff29]) by smtp.gmail.com with ESMTPSA id 6a1803df08f44-8d9f22974f5sm102436976d6.22.2026.06.18.11.35.28 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 18 Jun 2026 11:35:28 -0700 (PDT) Date: Thu, 18 Jun 2026 14:35:25 -0400 From: Johannes Weiner To: JP Kobryn Cc: akpm@linux-foundation.org, david@kernel.org, ljs@kernel.org, liam@infradead.org, vbabka@kernel.org, rppt@kernel.org, surenb@google.com, mhocko@suse.com, jackmanb@google.com, ziy@nvidia.com, fvdl@google.com, linux-mm@kvack.org, shakeel.butt@linux.dev, usama.arif@linux.dev, linux-kernel@vger.kernel.org Subject: Re: [PATCH v2] mm/page_alloc: use existing highatomic reserves on the buddy fastpath Message-ID: References: <20260617234958.150339-1-jp.kobryn@linux.dev> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20260617234958.150339-1-jp.kobryn@linux.dev> On Wed, Jun 17, 2026 at 04:49:58PM -0700, JP Kobryn wrote: > ALLOC_HIGHATOMIC currently provides both access to MIGRATE_HIGHATOMIC free > pages and permission to create new highatomic pageblock reserves. This > makes it unsuitable for the fastpath. > > However, the fastpath can reach rmqueue_buddy() while MIGRATE_HIGHATOMIC > reserves have free pages available. In this situation, the allocation can > fall back to other migratetypes without trying those reserves first. > > Allow high-priority non-blocking allocations above order-0 and up to the > costly order to use existing MIGRATE_HIGHATOMIC reserves on the buddy > fastpath. Change the semantics of ALLOC_HIGHATOMIC so that it only allows > access to the reserves without permission to grow them. Add a new flag > ALLOC_HIGHATOMIC_RESERVE that specifically allows growing the reserves. > > A UDP receive workload was run with free MIGRATE_HIGHATOMIC pageblocks > available in the target zone. Before this patch, the workload did not > consume these blocks. With this patch, eligible order-1 allocations > reaching the buddy path consumed existing MIGRATE_HIGHATOMIC pageblocks, > with no highatomic misses observed. The workload did not grow highatomic > reserves and NAPI page-frag allocations remained healthy with no failures > or order-0 fallbacks. Thanks for digging deeper into this! That's a great find. > Signed-off-by: JP Kobryn > Reviewed-by: Vlastimil Babka (SUSE) > --- > v2: > - decouple use semantics from ALLOC_HIGHATOMIC_RESERVE > - update changelog to reflect above change and reword test paragraph > - adjust comment in PCP path > - rebase onto Linus' tree ~v7.2-rc1 > > v1: https://lore.kernel.org/linux-mm/20260616191420.52556-1-jp.kobryn@linux.dev/ > > mm/internal.h | 1 + > mm/page_alloc.c | 30 +++++++++++++++++++++++++----- > 2 files changed, 26 insertions(+), 5 deletions(-) > > diff --git a/mm/internal.h b/mm/internal.h > index 5a2ddcf68e0b..6700659615e8 100644 > --- a/mm/internal.h > +++ b/mm/internal.h > @@ -1478,6 +1478,7 @@ unsigned int reclaim_clean_pages_from_list(struct zone *zone, > #define ALLOC_HIGHATOMIC 0x200 /* Allows access to MIGRATE_HIGHATOMIC */ > #define ALLOC_TRYLOCK 0x400 /* Only use spin_trylock in allocation path */ > #define ALLOC_KSWAPD 0x800 /* allow waking of kswapd, __GFP_KSWAPD_RECLAIM set */ > +#define ALLOC_HIGHATOMIC_RESERVE 0x1000 /* Allows growing MIGRATE_HIGHATOMIC reserves */ > > /* Flags that allow allocations below the min watermark. */ > #define ALLOC_RESERVES (ALLOC_NON_BLOCK|ALLOC_MIN_RESERVE|ALLOC_HIGHATOMIC|ALLOC_OOM) > diff --git a/mm/page_alloc.c b/mm/page_alloc.c > index d49c254174da..ed919e2ac99a 100644 > --- a/mm/page_alloc.c > +++ b/mm/page_alloc.c > @@ -3238,7 +3238,7 @@ struct page *rmqueue_buddy(struct zone *preferred_zone, struct zone *zone, > * If this is a high-order atomic allocation then check > * if the pageblock should be reserved for the future > */ > - if (unlikely(alloc_flags & ALLOC_HIGHATOMIC)) > + if (unlikely(alloc_flags & ALLOC_HIGHATOMIC_RESERVE)) > reserve_highatomic_pageblock(page, order, zone); You could check ALLOC_WMARK_MIN to determine the slowpath. This way you wouldn't need another alloc flag: /* Slowpath (precarious) high-atomic allocation. Maybe reserve block */ if (unlikely((alloc_flags & (ALLOC_HIGHATOMIC|ALLOC_WMARK_MIN)) == (ALLOC_HIGHATOMIC|ALLOC_WMARK_MIN))) reserve_highatomic_pageblock(page, order, zone); [ we really ought to generalize gfp_has_flags() ... ] > __count_zid_vm_events(PGALLOC, page_zonenum(page), 1 << order); > @@ -3320,8 +3320,9 @@ struct page *__rmqueue_pcplist(struct zone *zone, unsigned int order, > * > * Instead, direct it towards the reserves by > * returning NULL, which will make the caller fall > - * back to rmqueue_buddy. This will try to use the > - * reserves first and grow them if needed. > + * back to rmqueue_buddy. There it will try to use > + * the reserves first and grow them if needed and > + * permitted by the ALLOC_HIGHATOMIC_RESERVE flag. > */ > if (alloc_flags & ALLOC_HIGHATOMIC) > return NULL; > @@ -3768,6 +3769,24 @@ alloc_flags_nofragment(struct zone *zone, gfp_t gfp_mask) > return alloc_flags; > } > > +/* > + * Let high-priority non-blocking allocations above order-0 and up > + * to the costly order try to use existing MIGRATE_HIGHATOMIC > + * reserves on the fastpath. > + */ > +static inline unsigned int > +alloc_flags_highatomic_fastpath(gfp_t gfp_mask, unsigned int order) > +{ > + if (!order || order > PAGE_ALLOC_COSTLY_ORDER) > + return 0; There seems to be a mismatch between this and gfp_to_alloc_flags() (slowpath), where slowpath is still allowed to tap highatomic reserves for costly orders. Is that on purpose? > + if (!(gfp_mask & __GFP_HIGH)) > + return 0; > + if (gfp_mask & (__GFP_DIRECT_RECLAIM | __GFP_NOMEMALLOC)) > + return 0; This duplicates gfp_to_alloc_flags() logic which seems fragile. How about the below: > + > + return ALLOC_HIGHATOMIC; > +} > + > /* Must be called after current_gfp_context() which can change gfp_mask */ > static inline unsigned int gfp_to_alloc_flags_cma(gfp_t gfp_mask, > unsigned int alloc_flags) > @@ -4495,7 +4514,7 @@ gfp_to_alloc_flags(gfp_t gfp_mask, unsigned int order) > alloc_flags |= ALLOC_NON_BLOCK; > > if (order > 0 && (alloc_flags & ALLOC_MIN_RESERVE)) > - alloc_flags |= ALLOC_HIGHATOMIC; > + alloc_flags |= (ALLOC_HIGHATOMIC | ALLOC_HIGHATOMIC_RESERVE); > } > > /* > @@ -5215,7 +5234,8 @@ struct page *__alloc_frozen_pages_noprof(gfp_t gfp, unsigned int order, > * Forbid the first pass from falling back to types that fragment > * memory until all local zones are considered. > */ > - alloc_flags |= alloc_flags_nofragment(zonelist_zone(ac.preferred_zoneref), gfp); > + alloc_flags |= alloc_flags_nofragment(zonelist_zone(ac.preferred_zoneref), gfp) | > + alloc_flags_highatomic_fastpath(alloc_gfp, order); alloc_flags |= gfp_to_alloc_flags(gfp, order) & ALLOC_HIGHATOMIC; Or factor gfp_to_alloc_flags_nonblocking() from gfp_to_alloc_flags() and reuse that here, to save a few cycles in the fast path. > /* First allocation attempt */ > page = get_page_from_freelist(alloc_gfp, order, alloc_flags, &ac); > -- > 2.54.0