From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from out-179.mta0.migadu.com (out-179.mta0.migadu.com [91.218.175.179]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id CA49727FD75 for ; Wed, 7 Jan 2026 11:39:40 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=91.218.175.179 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1767785984; cv=none; b=OctBMGvxSz6cuEJYTlvMIqD5MUQgB98H9F8P4LV4GQN9y/QOiM1IOVRAwdRFsPdQ9Chef+Yv84VxHXft8cL/dLobL7Z4LFh4yZ7VhB2Ukb+VsN5akAD5LlgTEE9sJ2X9s7xXbtAUNOoYX+kTpbo5cmQTd2DKRS4sAiwQD6k7n9E= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1767785984; c=relaxed/simple; bh=0NjINbM9+9vM9jVOhtmme2K6VUZDHGN8EjtLlMarDMM=; h=MIME-Version:Date:Content-Type:From:Message-ID:Subject:To:Cc: In-Reply-To:References; b=Z6DZO99PTZ7BC9tVBXNWSmaEE3S0Do0WMbqHUhoWrWxFAd0Ox9+mF4RGLdZGTCNmy7moAAzd2yt6gj0Q4VygfWQK0lIh4WL4RW1qd7J6WbJlKFf8eGQdEbLHE+MUtN8WXe4wsjNcc/P/Uz+RwPC/hF5y5e32vRLfvp6NFz0SyoE= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev; spf=pass smtp.mailfrom=linux.dev; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b=Rl4yebr1; arc=none smtp.client-ip=91.218.175.179 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.dev Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b="Rl4yebr1" Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1767785979; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=5I7EN12qa/zxPyxnS8OTAcJMh+jISKVq561o3m/BAgg=; b=Rl4yebr1thnCWcSva95GejrDoUuH5uZ5JcxY6WPrMyIJeNoIEhH73d1llS1Dy8GtKRSvl1 5eybioGzbG6aRToz09tHR6FPtqrxhUYyx2eDf4Wrgrga1d/ezoGZwSlec6mpO11t61ypKS hjxLAgdovcJmJevjNnD8mqTYzONn1eU= Date: Wed, 07 Jan 2026 11:39:36 +0000 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. From: "Jiayuan Chen" Message-ID: <61b4f3ba49016e68e8d6bfe6543150a7de0bac79@linux.dev> TLS-Required: No Subject: Re: [PATCH v2] mm/vmscan: mitigate spurious kswapd_failures reset from direct reclaim To: "Shakeel Butt" , "Michal Hocko" Cc: linux-mm@kvack.org, "Jiayuan Chen" , "Andrew Morton" , "Johannes Weiner" , "David Hildenbrand" , "Qi Zheng" , "Lorenzo Stoakes" , "Axel Rasmussen" , "Yuanchu Xie" , "Wei Xu" , linux-kernel@vger.kernel.org In-Reply-To: References: <20251226080042.291657-1-jiayuan.chen@linux.dev> X-Migadu-Flow: FLOW_OUT January 7, 2026 at 06:06, "Shakeel Butt" wrote: >=20 >=20On Fri, Dec 26, 2025 at 04:00:42PM +0800, Jiayuan Chen wrote: >=20 >=20>=20 >=20> From: Jiayuan Chen > >=20=20 >=20> This is v2 of this patch series. For v1, see [1]. > >=20=20 >=20> When kswapd fails to reclaim memory, kswapd_failures is incremente= d. > > Once it reaches MAX_RECLAIM_RETRIES, kswapd stops running to avoid > > futile reclaim attempts. However, any successful direct reclaim > > unconditionally resets kswapd_failures to 0, which can cause problem= s. > >=20=20 >=20> We observed an issue in production on a multi-NUMA system where a > > process allocated large amounts of anonymous pages on a single NUMA > > node, causing its watermark to drop below high and evicting most fil= e > > pages: > >=20=20 >=20> $ numastat -m > > Per-node system memory usage (in MBs): > > Node 0 Node 1 Total > > --------------- --------------- --------------- > > MemTotal 128222.19 127983.91 256206.11 > > MemFree 1414.48 1432.80 2847.29 > > MemUsed 126807.71 126551.11 252358.82 > > SwapCached 0.00 0.00 0.00 > > Active 29017.91 25554.57 54572.48 > > Inactive 92749.06 95377.00 188126.06 > > Active(anon) 28998.96 23356.47 52355.43 > > Inactive(anon) 92685.27 87466.11 180151.39 > > Active(file) 18.95 2198.10 2217.05 > > Inactive(file) 63.79 7910.89 7974.68 > >=20=20 >=20> With swap disabled, only file pages can be reclaimed. When kswapd = is > > woken (e.g., via wake_all_kswapds()), it runs continuously but canno= t > > raise free memory above the high watermark since reclaimable file pa= ges > > are insufficient. Normally, kswapd would eventually stop after > > kswapd_failures reaches MAX_RECLAIM_RETRIES. > >=20=20 >=20> However, containers on this machine have memory.high set in their > > cgroup. Business processes continuously trigger the high limit, caus= ing > > frequent direct reclaim that keeps resetting kswapd_failures to 0. T= his > > prevents kswapd from ever stopping. > >=20=20 >=20> The key insight is that direct reclaim triggered by cgroup memory.= high > > performs aggressive scanning to throttle the allocating process. Wit= h > > sufficiently aggressive scanning, even hot pages will eventually be > > reclaimed, making direct reclaim "successful" at freeing some memory= . > > However, this success does not mean the node has reached a balanced > > state - the freed memory may still be insufficient to bring free pag= es > > above the high watermark. Unconditionally resetting kswapd_failures = in > > this case keeps kswapd alive indefinitely. > >=20=20 >=20> The result is that kswapd runs endlessly. Unlike direct reclaim wh= ich > > only reclaims from the allocating cgroup, kswapd scans the entire no= de's > > memory. This causes hot file pages from all workloads on the node to= be > > evicted, not just those from the cgroup triggering memory.high. Thes= e > > pages constantly refault, generating sustained heavy IO READ pressur= e > > across the entire system. > >=20=20 >=20> Fix this by only resetting kswapd_failures when the node is actual= ly > > balanced. This allows both kswapd and direct reclaim to clear > > kswapd_failures upon successful reclaim, but only when the reclaim > > actually resolves the memory pressure (i.e., the node becomes balanc= ed). > >=20=20 >=20> [1] https://lore.kernel.org/all/20251222122022.254268-1-jiayuan.ch= en@linux.dev/ > > Signed-off-by: Jiayuan Chen > >=20 >=20Hi Jiayuan, can you please send v3 of this patch with the following > additional information: >=20 >=201. Impact of the patch on your production jobs i.e. does it really > solves the issue? >=20 >=202. Memory reclaim stats or cpu usage of kswapd with and without patch= . >=20 >=20thanks, > Shakeel > Hi Shakeel, Thanks for the feedback. To be honest, the issue is difficult to reproduce because the boundary co= nditions are quite complex. We also haven't deployed this patch in production yet. I discovered the r= elationship between kswapd_failures and direct reclaim through the following bpftrace script: '''bash bpftrace -e ' #include #include kprobe:balance_pgdat { $pgdat =3D (struct pglist_data *)arg0; if ($pgdat->kswapd_failures > 0) { printf("[node %d] [%lu] kswapd end, kswapd_failures %d\n", $pgdat->node= _id, jiffies, $pgdat->kswapd_failures); } } tracepoint:vmscan:mm_vmscan_direct_reclaim_end { printf("[cpu %d] [%ul] reset kswapd_failures %d \n", cpu, jiffies, args.= nr_reclaimed) } ' ''' The trace results showed that when kswapd_failures reaches 15, continuous= direct reclaim keeps resetting it to 0. This was accompanied by a flood of kswapd_failures log= entries, and shortly after, we observed massive refaults occurring. (Note that I can only observe up to 15 in the trace due to a kprobe limit= ation: the kprobe on balance_pgdat fires at function entry, but kswapd_failures = is incremented to 16 only when balance_pgdat fails to reclaim any pages - at which point kswapd goe= s to sleep and there's no suitable hook point to capture it.) Before I send v3, I'd like to continue the discussion to make sure we're = aligned on the approach: Do you think the bpftrace evidence above is sufficient? If you and Michal are okay with the current approach, I'll prepare v3 wit= h mote detailed comments addressed. By the way, this tracing limitation makes me wonder: would it be appropri= ate to add two tracepoints for kswapd_failures? One for when kswapd_failures reaches MAX_RECLAIM_RETRIES= (16), and another for when it gets reset to 0. Currently, the only way to detect this is by polling nod= e_unreclaimable from /proc/zoneinfo, but the sampling interval is usually too coarse to catch these events. Thanks