From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-qk1-f171.google.com (mail-qk1-f171.google.com [209.85.222.171]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4BC56342173 for ; Fri, 16 Jan 2026 17:00:05 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.222.171 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1768582808; cv=none; b=B/7E3mJcFBphiaM45/EGhDxe05h+kCOwPCVF6KuQh93E76Qh7Dcf/Y+WQW+gYMYDGA3IyNNIrPeE+RYlZDX8+ph8ImCerzbnF2m9RhZXXpyIl4A9ksYp4Z7i3s9esl5WioVK4s48fyoYo0CDwm9AXxXtWfCd4y1bYZ/30QKgKdE= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1768582808; c=relaxed/simple; bh=w12kVIum+5hT80KSmtydVQZ1iiEJ4xy9RJwvCyIwfMY=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=Gg4Tqifo6g4uVhqcUG4CBup1AznOB5teT0jXkI9EvQtYAlScLKtrKURo6DbVzyiYwAcXigOJRYyskAUn/Ouig+CJar2oSE/2Wzi4pQahVroIYGtZ4kgOYr6vOtVriE/fS3y1wqipWRfjRnlisz1OjHka2TnF8xEAgmHC4JTsJnI= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=cmpxchg.org; spf=pass smtp.mailfrom=cmpxchg.org; dkim=pass (2048-bit key) header.d=cmpxchg.org header.i=@cmpxchg.org header.b=lSwLgUxD; arc=none smtp.client-ip=209.85.222.171 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=cmpxchg.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=cmpxchg.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=cmpxchg.org header.i=@cmpxchg.org header.b="lSwLgUxD" Received: by mail-qk1-f171.google.com with SMTP id af79cd13be357-8c0f13e4424so272932185a.1 for ; Fri, 16 Jan 2026 09:00:05 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cmpxchg.org; s=google; t=1768582805; x=1769187605; darn=vger.kernel.org; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=39r+L65CbocMKJ2hSULIQG9yHS2j8i36ygEvkdqqJZE=; b=lSwLgUxD0o35X/iBW9QsKvB7g/cK4P2opSpPbzJT0ownCctwZhZ2zxHHLpQhQ91UQb 2nfNBP/lQYG4aPacOFEK+XHlHDfl3JArjCSowyAvDViqp7NtcWCqIuOhMh2TYdfqhqJB /vwYsGK6TT0/Dx4V+NXYoGE2klgO0AfLixL1I3jV3+T2httzEjA7QuKtNfkXH3eNlrx6 6erzdsuMWgCvnGjisoFqYunp21r5wZ7sbOD4aoiPXmr0NadYotlF4I70n0SWuNyld1F6 2Myb9yAPdJ4kdJ6h4fPDyIpV+zM84yzP7Oc2hf+OVFFfcI2AqJ313Ol7fK+8jq//NXw9 NUZg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1768582805; x=1769187605; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-gg:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=39r+L65CbocMKJ2hSULIQG9yHS2j8i36ygEvkdqqJZE=; b=PO4AvBeYlhnjjaeXC7u7P6cwUVEWR0mtVGaReih5DTq31SmHrbYIG0RTePTrDnSNwZ ryYRAKXMQosZOUj6mHbaNeHK/K2uERtlgfSj53Es5GmAVwsxwM/h0JCa66pkZiUux0Gd ChhmdmWFrNnd2mIO1uVFY4CT5AiIeCfEkXCLLn6tldL0qOx0XzrENcs0f1t178wsV6SO Q9GurukTSQUVaDHRF142oYowhxOPhZLXBKb3770XYmToeapuQ33wS1in0QK7uPS6kD/M qCd1oUv1HovPMPKPBfPrussOGv5Pxoq8w6DhB+bZKe6YFJLPW8NkIKqEEfxZ6LWWvb+9 9ZXA== X-Forwarded-Encrypted: i=1; AJvYcCUxgnlbQN8zViEAH5ECWP9z4S4W4OlNOyPbS6QAXyWrcbZ+L+3gOA5ILUt4Lh3xYMFinuK463H461oayrU=@vger.kernel.org X-Gm-Message-State: AOJu0Yy4+jdzfZz1o5Tb/GSgHEnIkiqMJesGZJEvo7T/0jp6BBGZPfxs MjKiHWI2bbCpEkWJmTivuMCngv5kerRAwvnp6sHMIMAlFP+SECP+e4F3ik/gagoZBEA= X-Gm-Gg: AY/fxX7XLJxcd5feaJhtBwRY1kSpO0e//uraAYJrR1sUFuKXaGrvYUfvBxL2nrmdW4h fl7tZUf/cOJlR/ovNqyoUL/kIQbt2gQda+DmCgpsILQLTaiL2LvgZqKvBrbtnBjlakBJcB5MXtW 5SesguMcaP0z9yd+UpORtOS9pxhkNGm1LOGpyAjmpBrw4/M8WhWnincIgHeABMZsDai5GzikAaH Q4jVC4S0lwAkuSBkiCgTO1Vv7FS7ncaqewPZo7KbgfOJgQi01Gz4cpnUuw+CUNJ482SvNsNvchB lRawvJFybZz9Cy2eTpB8bQIKOUFQtW6yHblej84Q/jm7rkYp96BDY5ufUKNvGJNNVY3p8pON9/T cN4tOor2TGFOIzBMyc3+7euccIUJOXCEcC7vv96IHvuX8D5WGXV+tz/3M+ayibB/imq2xqMKPCn TvSZnrqBaSlA== X-Received: by 2002:a05:620a:710c:b0:8c5:2e1b:7913 with SMTP id af79cd13be357-8c6a66ef7fbmr529344385a.25.1768582804795; Fri, 16 Jan 2026 09:00:04 -0800 (PST) Received: from localhost ([2603:7000:c01:2716:365a:60ff:fe62:ff29]) by smtp.gmail.com with ESMTPSA id af79cd13be357-8c6a71d5b93sm262341185a.23.2026.01.16.09.00.03 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 16 Jan 2026 09:00:03 -0800 (PST) Date: Fri, 16 Jan 2026 12:00:00 -0500 From: Johannes Weiner To: Jiayuan Chen Cc: linux-mm@kvack.org, shakeel.butt@linux.dev, Jiayuan Chen , Andrew Morton , David Hildenbrand , Lorenzo Stoakes , "Liam R. Howlett" , Vlastimil Babka , Mike Rapoport , Suren Baghdasaryan , Michal Hocko , Axel Rasmussen , Yuanchu Xie , Wei Xu , Steven Rostedt , Masami Hiramatsu , Mathieu Desnoyers , Brendan Jackman , Zi Yan , Qi Zheng , linux-kernel@vger.kernel.org, linux-trace-kernel@vger.kernel.org Subject: Re: [PATCH v3 1/2] mm/vmscan: mitigate spurious kswapd_failures reset from direct reclaim Message-ID: References: <20260114074049.229935-1-jiayuan.chen@linux.dev> <20260114074049.229935-2-jiayuan.chen@linux.dev> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20260114074049.229935-2-jiayuan.chen@linux.dev> On Wed, Jan 14, 2026 at 03:40:35PM +0800, Jiayuan Chen wrote: > From: Jiayuan Chen > > When kswapd fails to reclaim memory, kswapd_failures is incremented. > Once it reaches MAX_RECLAIM_RETRIES, kswapd stops running to avoid > futile reclaim attempts. However, any successful direct reclaim > unconditionally resets kswapd_failures to 0, which can cause problems. > > We observed an issue in production on a multi-NUMA system where a > process allocated large amounts of anonymous pages on a single NUMA > node, causing its watermark to drop below high and evicting most file > pages: > > $ numastat -m > Per-node system memory usage (in MBs): > Node 0 Node 1 Total > --------------- --------------- --------------- > MemTotal 128222.19 127983.91 256206.11 > MemFree 1414.48 1432.80 2847.29 > MemUsed 126807.71 126551.11 252358.82 > SwapCached 0.00 0.00 0.00 > Active 29017.91 25554.57 54572.48 > Inactive 92749.06 95377.00 188126.06 > Active(anon) 28998.96 23356.47 52355.43 > Inactive(anon) 92685.27 87466.11 180151.39 > Active(file) 18.95 2198.10 2217.05 > Inactive(file) 63.79 7910.89 7974.68 > > With swap disabled, only file pages can be reclaimed. When kswapd is > woken (e.g., via wake_all_kswapds()), it runs continuously but cannot > raise free memory above the high watermark since reclaimable file pages > are insufficient. Normally, kswapd would eventually stop after > kswapd_failures reaches MAX_RECLAIM_RETRIES. > > However, containers on this machine have memory.high set in their > cgroup. Business processes continuously trigger the high limit, causing > frequent direct reclaim that keeps resetting kswapd_failures to 0. This > prevents kswapd from ever stopping. > > The key insight is that direct reclaim triggered by cgroup memory.high > performs aggressive scanning to throttle the allocating process. With > sufficiently aggressive scanning, even hot pages will eventually be > reclaimed, making direct reclaim "successful" at freeing some memory. > However, this success does not mean the node has reached a balanced > state - the freed memory may still be insufficient to bring free pages > above the high watermark. Unconditionally resetting kswapd_failures in > this case keeps kswapd alive indefinitely. > > The result is that kswapd runs endlessly. Unlike direct reclaim which > only reclaims from the allocating cgroup, kswapd scans the entire node's > memory. This causes hot file pages from all workloads on the node to be > evicted, not just those from the cgroup triggering memory.high. These > pages constantly refault, generating sustained heavy IO READ pressure > across the entire system. > > Fix this by only resetting kswapd_failures when the node is actually > balanced. This allows both kswapd and direct reclaim to clear > kswapd_failures upon successful reclaim, but only when the reclaim > actually resolves the memory pressure (i.e., the node becomes balanced). > > Signed-off-by: Jiayuan Chen > Signed-off-by: Jiayuan Chen Great analysis, and I agree with both the fix and adding tracepoints. Two minor nits: > @@ -2650,6 +2650,25 @@ static bool can_age_anon_pages(struct lruvec *lruvec, > lruvec_memcg(lruvec)); > } > > +static void pgdat_reset_kswapd_failures(pg_data_t *pgdat) > +{ > + atomic_set(&pgdat->kswapd_failures, 0); > +/* > + * Reset kswapd_failures only when the node is balanced. Without this > + * check, successful direct reclaim (e.g., from cgroup memory.high > + * throttling) can keep resetting kswapd_failures even when the node > + * cannot be balanced, causing kswapd to run endlessly. > + */ > +static bool pgdat_balanced(pg_data_t *pgdat, int order, int highest_zoneidx); > +static inline void pgdat_try_reset_kswapd_failures(struct pglist_data *pgdat, Please remove the inline, the compiler will figure it out. > + struct scan_control *sc) > +{ > + if (pgdat_balanced(pgdat, sc->order, sc->reclaim_idx)) > + pgdat_reset_kswapd_failures(pgdat); > +} As this is kswapd API, please move these down to after wakeup_kswapd(). I think we can streamline the names a bit. We already use "hopeless" for that state in the comments; can you please rename the functions kswapd_clear_hopeless() and kswapd_try_clear_hopeless()? We should then also replace the open-coded kswapd_failure checks with kswapd_test_hopeless(). But I can send a follow-up patch if you don't want to, just let me know.