From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-qv1-f48.google.com (mail-qv1-f48.google.com [209.85.219.48]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4C52B37DE84 for ; Wed, 22 Jul 2026 15:00:23 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.219.48 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784732424; cv=none; b=rYBPK7C4ZpAmNr+odKBXMHNoC+IrFQFbv7tbtsS11PHaRIGX65QExbsquHULk+WJ4kiKfcQ+GaCu9is0RP7Q+3Exan868nLDaMfOaEdYWHul+iwp78aafUCP8LPOE7abEsFyOnKOJr0uu5Hsg64aNE45XQnaTKbGpaKhd6zuMSI= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784732424; c=relaxed/simple; bh=QKg5YGY/yCLwZN3vmT5c5UttyqkYjSANMRTryN3kfVI=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=LCbtyeArtw65RoBjxHHBTqy/JoIDFiww0sqUpIn1T7Wolao05aXCsGIwsECM0i5PY2XB+NFJ+ea6bEdAAmCbLou7snb1Ub1HzWJqUQQ1JSqnlsWrCL497V6IsKm4pKLv7kJ3zK4uifIVFkhhk2kQi7vb3bPJLuqa/RTX6vFuUCo= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=cmpxchg.org; spf=pass smtp.mailfrom=cmpxchg.org; dkim=pass (2048-bit key) header.d=cmpxchg.org header.i=@cmpxchg.org header.b=YquYS/A8; arc=none smtp.client-ip=209.85.219.48 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=cmpxchg.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=cmpxchg.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=cmpxchg.org header.i=@cmpxchg.org header.b="YquYS/A8" Received: by mail-qv1-f48.google.com with SMTP id 6a1803df08f44-8ff20870ac7so102871106d6.1 for ; Wed, 22 Jul 2026 08:00:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cmpxchg.org; s=google; t=1784732422; x=1785337222; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=ssUPrZXhmKEtcaomL/NSzb6q5Qkj8exc1H0XDUvt4yE=; b=YquYS/A81i4NStxAz2sixHxEV0vy/5FSbqv9AzkNe9wO41B/Q14lKBPu5zR6toHgVl dcLMhkU/wYun0sQKnBOSM3sSiz7G1NBONgJZDRdueIaC43Qir3PZj6ZEdk6lDW26IETh WltWVga59seLsifIWTqK6+yHhMBVRqaZbGjEhWaQmIjh4Z/sMTjlr3wdCcirq4JOPlkW A+4lye9IOCWnVYAs/4hfhEZcOd6E9QnmF5K4ddc7Unr2wfA8ezMpg6LU+sh1pFVEoFlG oE4/HiAFPIb+InIxP9GKbEr3fKNaJxemnET9GMMJdHDGE8Ht4aflCjPF/OEbL5zcAGCB E0Nw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1784732422; x=1785337222; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=ssUPrZXhmKEtcaomL/NSzb6q5Qkj8exc1H0XDUvt4yE=; b=VLBYw9zu/ckT4tGJVaPtAhDb37R5dZsVsLjnQvYOYrVNXROGBoKsGy1gm5L+pzShG+ nvc9jO+RnxyNaQUsjBkX6MfAx5s5N6iCLE0d6RYvxWeTxK0zcz2s7ZhURs/cvTy/OAA9 OirNk/uHucMBkBzOClBIv+cAJ5G4JkfyypDq0eBIQlIVgAKRmLzbcka4FM0ubrfGcqNC wu7+MfKIlh3Ef4vRoiFwzDRfTam/Liz8BOi2O31t0VERNsqYshuvP7lHOhVF7naDhpbC GMXSLsq7ik+4H8Et9hFyNGoS9ShJkI1VDRlXlkJmAS7IeR1+KdPNMxY4J92RRb/vYe15 tY3g== X-Forwarded-Encrypted: i=1; AHgh+RpczDZ4xdRlTYxUjMQcx32N7FRGZ9z5N/ifua1KttaRDaoZZofs7iTied8o/5LpFdkET1Zgqepg+EkB88c=@vger.kernel.org X-Gm-Message-State: AOJu0Yy4XFSBjGKLKLwlr2R3EqixwXu6iGkxJqBTWUkCXqW4y5rg9upQ 3KrpNJ3bKH6ZPO8MAQ5CgnRpQpFL+pCzmB/xrvd1F5T6jXZTDaOEmC/tJWq2v461zZQ= X-Gm-Gg: AR+sD12V7PD92NM+hmcfH4JR8GNO06uupykqrcHU8gekbMk7O7cWK/PFr66OLQF6c78 3hUP+KpZ/ygE8hQDs7f9pK6UoVy5C4WnB8TwRi7z4TXeaRNX3PdfdWPpDKi348SjjGNsnxFGyEG Kl81WhjK6rzin2pXC5PujLFRS4UhGsyzNRQbmhsRrJxRMRQw857D94TAIQO/j9J1zxY4N7jZJz3 iISuzyQwNkC3mcdI2YnLUhvUPekVvGQ7UCIyfpyEA/GEpYNnE+2Nsy/MXcQClhP2tUI1yzI5fi+ dfd4rtIZ45HAOY5XDD5NjWageGdkGaZhWmpIJQQJUjyaI0KlFSI0ZCU1QIy4ojUdhP+ZcySCbcv IMDGaNj8Y7n5h+dxMKxr2f6kN7oShVhpRd9OydM+Xhhfdk50FLCVcTs5MBGEFOn4YGZJkEtsQ8f Y2 X-Received: by 2002:a05:6214:3912:b0:8f1:323a:eede with SMTP id 6a1803df08f44-9077835ab03mr252766896d6.14.1784732418083; Wed, 22 Jul 2026 08:00:18 -0700 (PDT) Received: from localhost ([2603:7001:f100:500:365a:60ff:fe62:ff29]) by smtp.gmail.com with ESMTPSA id 6a1803df08f44-907baa0a146sm22359506d6.33.2026.07.22.08.00.17 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 22 Jul 2026 08:00:17 -0700 (PDT) From: Johannes Weiner To: Andrew Morton , Vlastimil Babka Cc: Suren Baghdasaryan , Michal Hocko , Brendan Jackman , Zi Yan , David Hildenbrand , Lorenzo Stoakes , "Liam R . Howlett" , Mike Rapoport , Shakeel Butt , linux-mm@kvack.org, linux-kernel@vger.kernel.org, stable@vger.kernel.org Subject: [PATCH v2 4/4] mm: page_alloc: fix non-movable reclaim storm in defrag_mode Date: Wed, 22 Jul 2026 10:56:47 -0400 Message-ID: <20260722150006.3848560-5-hannes@cmpxchg.org> X-Mailer: git-send-email 2.55.0 In-Reply-To: <20260722150006.3848560-1-hannes@cmpxchg.org> References: <20260722150006.3848560-1-hannes@cmpxchg.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit As we deployed defrag_mode into Meta production, pressure spikes and excessive swapping were observed on some workloads. Tracing confirmed that this is unmovable/reclaimable requests spinning in the allocator and direct reclaim, causing excessive amounts of swap. The initial plan for defrag_mode was to rely on kswapd/kcompactd to produce blocks, and if those are overwhelmed under high pressure, let the allocator fall back (__rmqueue_steal()) after its retry loops. However, that retrying results in more reclaim on some of these workloads than we'd hoped, sometimes excessively so, spurred on by the !costly order conditions in should_reclaim_retry(). The storms are dependent on the request type. Reclaim will inevitably make room in existing movable blocks, since that's where the LRU pages live. So if movable requests retry on reclaim, they make progress. When non-movable requests spin in reclaim that isn't productive. They cannot use the individually freed pages, and the process is unlikely to accidentally free whole blocks to meet the ALLOC_NOFRAGMENT bar. They spin and overreclaim excessively, which tanks performance and triggers userspace guards like swap exhaustion or pressure based OOM. To fix this, send non-movable requests, regardless of order, into pageblock reclaim/compaction. This way, they help move things along to meet the ALLOC_NOFRAGMENT bar. After this patch, the reclaim storms and excess OOM rates are no longer observed in production. The longer-term plan is still to have all requests, including the movable ones, help make blocks to spread the cost of defragmenting more evenly and fairly; combined with proper watermarking to reduce allocation latencies in the common case. However, doing this naively unearths scaling and concurrency limitations in compaction that need to be addressed first. Promoting just non-movables for now is the minimally viable bug fix for the above issue. Fixes: e3aa7df331bc ("mm: page_alloc: defrag_mode") Cc: Signed-off-by: Johannes Weiner --- mm/internal.h | 6 ++++++ mm/page_alloc.c | 31 ++++++++++++++++++++++++++----- 2 files changed, 32 insertions(+), 5 deletions(-) diff --git a/mm/internal.h b/mm/internal.h index 6d3402001b93..5acba3470659 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -1061,6 +1061,12 @@ struct compact_control { struct capture_control { struct zone *zone; int migratetype; + /* + * Allocation request order. May differ from the compaction + * order: defrag_mode promotes sub-block allocations to + * pageblock-order compaction; capture still matches at the + * original allocation order so prep_new_page() is consistent. + */ int order; struct page *page; }; diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 4b9686fedb71..f92055827ae9 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -4152,8 +4152,24 @@ __alloc_pages_direct_compact(gfp_t gfp_mask, unsigned int order, .order = order, .page = NULL, }; + int compact_order = order; - if (!order) + /* + * If fallbacks are not permitted (defrag_mode), we either + * need to reclaim space in a block of matching type, or clear + * out an entire block to allow __rmqueue_claim() to convert. + * + * Reclaim by itself is primarily freeing space in movable + * blocks, since that's where the LRU pages live. So this + * works for movable requests, but not for others. + * + * For those, promote the order to help make blocks, instead + * of spinning in reclaim alone unproductively. + */ + if ((alloc_flags & ALLOC_NOFRAGMENT) && ac->migratetype != MIGRATE_MOVABLE) + compact_order = max(order, pageblock_order); + + if (!compact_order) return NULL; psi_memstall_enter(&pflags); @@ -4169,8 +4185,8 @@ __alloc_pages_direct_compact(gfp_t gfp_mask, unsigned int order, barrier(); WRITE_ONCE(current->capture_control, &capc); - *compact_result = try_to_compact_pages(gfp_mask, order, alloc_flags, ac, - prio, &capc); + *compact_result = try_to_compact_pages(gfp_mask, compact_order, + alloc_flags, ac, prio, &capc); /* * Make sure we hide capture control first before we read the captured @@ -4215,7 +4231,7 @@ __alloc_pages_direct_compact(gfp_t gfp_mask, unsigned int order, struct zone *zone = page_zone(page); zone->compact_blockskip_flush = false; - compaction_defer_reset(zone, order, true); + compaction_defer_reset(zone, compact_order, true); count_vm_event(COMPACTSUCCESS); return page; } @@ -4455,9 +4471,14 @@ __alloc_pages_direct_reclaim(gfp_t gfp_mask, unsigned int order, struct page *page = NULL; unsigned long pflags; bool drained = false; + int reclaim_order = order; + + /* Match the slowpath compaction promotion in __alloc_pages_direct_compact */ + if ((alloc_flags & ALLOC_NOFRAGMENT) && ac->migratetype != MIGRATE_MOVABLE) + reclaim_order = max(order, pageblock_order); psi_memstall_enter(&pflags); - *did_some_progress = __perform_reclaim(gfp_mask, order, ac); + *did_some_progress = __perform_reclaim(gfp_mask, reclaim_order, ac); if (unlikely(!(*did_some_progress))) goto out; -- 2.55.0