From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-qt1-f177.google.com (mail-qt1-f177.google.com [209.85.160.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id EC1543AA1A1 for ; Fri, 26 Jun 2026 18:22:39 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.160.177 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1782498161; cv=none; b=CLUUiTFCQx9xcKMiaQUj0vhU5iHY/4hG1RHh8YbN93Cjn8/AgUm0j5/G3qi/yIWj18z8rb03agkxRTFnIdobdcimmp8UfQhypNUmUdN8g/XvzRQAtSynNDGyJx0iD92gEweyVi3puG5cP4QlgKJO7xL9mWUKLgGBFSKNyJhWh0c= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1782498161; c=relaxed/simple; bh=y66o+VUqodm+NfgdRw/6Z/CTaL7zgnp98TDtmBOSeXE=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=MJjDW1+7st89+YY6DeK7e6+RqW/gBVbw6StCq3oxwuI0bvQqmH4T1UXBaXKGfd/LVU/wY9WDlKucHW2I6Dpx6ysIDWnz1itGUCTLG1K9V27UZNk4EE50MQMp39n2w1wgscPbf6bH+P8qzg6mzZxK4+ySgQqhhmOu8pyPM4N3r7s= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=cmpxchg.org; spf=pass smtp.mailfrom=cmpxchg.org; dkim=pass (2048-bit key) header.d=cmpxchg.org header.i=@cmpxchg.org header.b=OsAz7P32; arc=none smtp.client-ip=209.85.160.177 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=cmpxchg.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=cmpxchg.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=cmpxchg.org header.i=@cmpxchg.org header.b="OsAz7P32" Received: by mail-qt1-f177.google.com with SMTP id d75a77b69052e-51a8c0fbe18so3919121cf.0 for ; Fri, 26 Jun 2026 11:22:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cmpxchg.org; s=google; t=1782498159; x=1783102959; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Si58C84HX9Fb8r6EvkY+7noECsiq5uUlXevnn3+D3Fw=; b=OsAz7P32sNS44CnZpMlsc9oqHl5yvvMZfe6X+ImXLu0c+jFPYpRseEP0AF1vQQ/M/w 5yupeZozjyVu9Rjre4/S0UCceCp6hpOAHUtNuDkoEy34iBsBfprRhBZlQ32OpEkNvTza akRlXR6g9RiC53CSZcHRmAndHqYxmR7DBoYDKc59WfOhZRE59uXMsGjbQa8upaQgx9uZ XfRYc7anQw66vf1hQQiLlot4fZ72EDe+7yR/yaI05ZgnS2xkhtI86G+TQ5fpDQDnID4g SiX3+dVHK++ibEjrhmXctbqvP/MEjErBU+LCKAfxal0LTspVzkEiAVhpK2Z3UEadY8KH bfIQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1782498159; x=1783102959; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=Si58C84HX9Fb8r6EvkY+7noECsiq5uUlXevnn3+D3Fw=; b=DudgBtCkAPbzyScZUAKDFJRVlX9cGHb7sDA8oGI4ueLlcegAsVhF/a8jZR03z+qsO+ XKiL+IXfLYmT3hU1To1YV/CA6lnZZODtWfzW9V63rZaTMCFADgEcIpqU+bRKj1VR3wn+ WmpjxAiS2FbU3DkFlJDb7e+p1Yg4kJxubZYaDNwDqX8/cgXopQ9bBG0v6Hh3Hh5swc9n dAIBaChYbUIurJthJwD571qpXRDdJofJONhiGFaqJqwSQjA3hLMuqBWAxJp+28Uczy8d 0IjY1Ujd27y5k/LXnc1Hdj8YBATTFIk4bV/rG7xYa8QVgEJa6mvgaR96KnbigInr7+ri 1gxw== X-Forwarded-Encrypted: i=1; AFNElJ/vu9vjbY2ooWfiIb9ZBryN0O/PBPxiEn2R5Mr+KF3ns1ugnLe3CuKy9je8H/uhYNGxuAxLXZJ1b3TtGmw=@vger.kernel.org X-Gm-Message-State: AOJu0YyeADuCyw+LRAnt9PawSg7KUoXEBr1gKl21jCxkVWFQ6jmkaP4n dlGZUbOWvO2/bz3+Usr2uChlYR+txndzLtES5CZXnN3T1DLR9Ic5niEfQRAG99hcDE0= X-Gm-Gg: AfdE7cmKGl/ZWoFEsv6LuPG3Ul8R+G+GWy5qP1UdhtAd6TmTVcSLrpjIT4CS3HbgKpF mDsJNfUNJbaLQy++a/j8/pfl2bpqr532gsMxZQNHujCHkOWDvRJEQqu+TA21Yle9aV0MkhmXUD7 Sr2s6gcMh5Vnip40HThrRZHSVfTbM4RWqXztlNpFRD9JF0I+lhwjPEfwSI4/9IulnjYqLbIa82H FSRjj/E5rFn53t74j3rv1vAj1HMqL9SJ2+tlDhaEuVqUNj0fo0wo24sTuK+cIlE8hZI+X9js7Tt 9TG4/mZzSNonA3/p/Onkn4iCXYB86MGiuhwW14mEP4kfkdU45zZCCySStxmBMTI5QTHhUDlP8yj WsHETt3v1CEy4PaO0E1ZCiOcbWZBGhGZ1DOEzg5bg/r9GJqe8CZQqBBQjPa3M3QBwj5KnGU8+MW fdc1/Pyhs9JPE= X-Received: by 2002:ac8:578e:0:b0:516:d657:2c65 with SMTP id d75a77b69052e-51a726e1eb7mr102974601cf.6.1782498158863; Fri, 26 Jun 2026 11:22:38 -0700 (PDT) Received: from localhost ([2603:7001:f100:500:365a:60ff:fe62:ff29]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-51a51a9b5b4sm98694961cf.18.2026.06.26.11.22.37 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 26 Jun 2026 11:22:37 -0700 (PDT) From: Johannes Weiner To: Andrew Morton , Vlastimil Babka Cc: Suren Baghdasaryan , Michal Hocko , Brendan Jackman , Zi Yan , David Hildenbrand , Lorenzo Stoakes , "Liam R. Howlett" , Mike Rapoport , linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH 4/4] mm: page_alloc: fix non-movable reclaim storm in defrag_mode Date: Fri, 26 Jun 2026 14:21:20 -0400 Message-ID: <20260626182215.1107966-5-hannes@cmpxchg.org> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20260626182215.1107966-1-hannes@cmpxchg.org> References: <20260626182215.1107966-1-hannes@cmpxchg.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit As we deployed defrag_mode into Meta production, pressure spikes and excessive swapping were observed on some workloads. Tracing confirmed that this is unmovable/reclaimable requests spinning in the allocator and direct reclaim, causing excessive amounts of swap. The initial plan for defrag_mode was to rely on kswapd/kcompactd to produce blocks, and if those are overwhelmed under high pressure, let the allocator fall back (__rmqueue_steal()) after its retry loops. However, that retrying results in more reclaim on some of these workloads than we'd hoped, sometimes excessively so, spurred on by the !costly order conditions in should_reclaim_retry(). The storms are dependent on the request type. Reclaim will inevitably make room in existing movable blocks, since that's where the LRU pages live. So if movable requests retry on reclaim, they make progress. When non-movable requests spin in reclaim that isn't productive. They cannot use the individually freed pages, and the process is unlikely to accidentally free whole blocks to meet the ALLOC_NOFRAGMENT bar. They spin and overreclaim excessively, which tanks performance and triggers userspace guards like swap exhaustion or pressure based OOM. To fix this, send non-movable requests, regardless of order, into pageblock reclaim/compaction. This way, they help move things along to meet the ALLOC_NOFRAGMENT bar. After this patch, the reclaim storms and excess OOM rates are no longer observed in production. The longer-term plan is still to have all requests, including the movable ones, help make blocks to spread the cost of defragmenting more evenly and fairly; combined with proper watermarking to reduce allocation latencies in the common case. However, doing this naively unearths scaling and concurrency limitations in compaction that need to be addressed first. Promoting just non-movables for now is the minimally viable bug fix for the above issue. Fixes: f38356df6474 ("mm: page_alloc: introduce defrag_mode") Signed-off-by: Johannes Weiner --- mm/internal.h | 7 +++++++ mm/page_alloc.c | 36 +++++++++++++++++++++++++++++------- 2 files changed, 36 insertions(+), 7 deletions(-) diff --git a/mm/internal.h b/mm/internal.h index 181e79f1d6a2..1f636cfc859a 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -1060,6 +1060,13 @@ struct compact_control { */ struct capture_control { struct compact_control *cc; + /* + * Allocation request order. May differ from the compaction + * order: defrag_mode promotes sub-block allocations to + * pageblock-order compaction; capture still matches at the + * original allocation order so prep_new_page() is consistent. + */ + int order; struct page *page; }; diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 9dee1c47e795..575a99a4c723 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -728,7 +728,7 @@ static inline bool compaction_capture(struct capture_control *capc, struct page *page, int order, int migratetype) { - if (!capc || order != capc->cc->order) + if (!capc || order != capc->order) return false; /* Do not accidentally pollute CMA or isolated regions*/ @@ -748,7 +748,7 @@ compaction_capture(struct capture_control *capc, struct page *page, return false; if (migratetype != capc->cc->migratetype) - trace_mm_page_alloc_extfrag(page, capc->cc->order, order, + trace_mm_page_alloc_extfrag(page, capc->order, order, capc->cc->migratetype, migratetype); capc->page = page; @@ -4147,10 +4147,27 @@ __alloc_pages_direct_compact(gfp_t gfp_mask, unsigned int order, unsigned long pflags; unsigned int noreclaim_flag; struct capture_control capc = { + .order = order, .page = NULL, }; + int compact_order = order; - if (!order) + /* + * If fallbacks are not permitted (defrag_mode), we either + * need to reclaim space in a block of matching type, or clear + * out an entire block to allow __rmqueue_claim() to convert. + * + * Reclaim by itself is primarily freeing space in movable + * blocks, since that's where the LRU pages live. So this + * works for movable requests, but not for others. + * + * For those, promote the order to help make blocks, instead + * of spinning in reclaim alone unproductively. + */ + if ((alloc_flags & ALLOC_NOFRAGMENT) && ac->migratetype != MIGRATE_MOVABLE) + compact_order = max(order, pageblock_order); + + if (!compact_order) return NULL; /* @@ -4166,8 +4183,8 @@ __alloc_pages_direct_compact(gfp_t gfp_mask, unsigned int order, fs_reclaim_acquire(gfp_mask); noreclaim_flag = memalloc_noreclaim_save(); - *compact_result = try_to_compact_pages(gfp_mask, order, alloc_flags, ac, - prio, &capc); + *compact_result = try_to_compact_pages(gfp_mask, compact_order, + alloc_flags, ac, prio, &capc); memalloc_noreclaim_restore(noreclaim_flag); fs_reclaim_release(gfp_mask); @@ -4203,7 +4220,7 @@ __alloc_pages_direct_compact(gfp_t gfp_mask, unsigned int order, struct zone *zone = page_zone(page); zone->compact_blockskip_flush = false; - compaction_defer_reset(zone, order, true); + compaction_defer_reset(zone, compact_order, true); count_vm_event(COMPACTSUCCESS); return page; } @@ -4443,9 +4460,14 @@ __alloc_pages_direct_reclaim(gfp_t gfp_mask, unsigned int order, struct page *page = NULL; unsigned long pflags; bool drained = false; + int reclaim_order = order; + + /* Match the slowpath compaction promotion in __alloc_pages_direct_compact */ + if ((alloc_flags & ALLOC_NOFRAGMENT) && ac->migratetype != MIGRATE_MOVABLE) + reclaim_order = max(order, pageblock_order); psi_memstall_enter(&pflags); - *did_some_progress = __perform_reclaim(gfp_mask, order, ac); + *did_some_progress = __perform_reclaim(gfp_mask, reclaim_order, ac); if (unlikely(!(*did_some_progress))) goto out; -- 2.54.0