From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from out-181.mta1.migadu.com (out-181.mta1.migadu.com [95.215.58.181]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 21211238166 for ; Tue, 23 Dec 2025 08:22:47 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=95.215.58.181 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1766478171; cv=none; b=h8THJs3uZPEuSSZk2CE8ZfyywusyU9338zGiuTXaX59K2n+KXJH8Vwqq4GuDUgJHlEbhXx+ZjIgfTuVTwJExbxc7jW8/qa8yzEZDDwUW8/vQsgTg4qiCNSP6A2NjSQsgyFKp3fMKQGHC40kYaSyBfz0HrxMqqBKurTm2VpwGBCo= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1766478171; c=relaxed/simple; bh=Pd9GXEDY6e5eI1FhZws3AvZB705Htrbt+cTLPCx7kgE=; h=MIME-Version:Date:Content-Type:From:Message-ID:Subject:To:Cc: In-Reply-To:References; b=BXd/ZrvjDT0+oRQcAWnay8ntYOja2bgjSu8fy6/Lehosqyuhky1BX9t1hnD6DGWC5rxseW36vJuUrbhENkMz+Z7SEuxmFTu5UGaay9uOJlKLpsVN0opbJzlQvMQtdtnjOOoaMATSLbvUgbaCGve9rLiwzn+ht8S1h37CzkwqW8M= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev; spf=pass smtp.mailfrom=linux.dev; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b=LWGKu27t; arc=none smtp.client-ip=95.215.58.181 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.dev Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b="LWGKu27t" Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1766478166; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=g8DrPyrW6XtxAjggK1JgrO3+7taqgHLHtwdI/kQJR+w=; b=LWGKu27tPvA6G/BvHlU1qcMpvMeL6oZcoC6wVaQUt3k4Les05DwUlFnp7Gjahu1gLH5WZB oktQErFDkcBbaLo98juzOcKkUlZauT477Cg0haR9+Kg1nX4zilo1MYpXCXp8yhoogkl1m9 FMaczIARJifLFdHGoGEQC+QIpX7I384= Date: Tue, 23 Dec 2025 08:22:43 +0000 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. From: "Jiayuan Chen" Message-ID: TLS-Required: No Subject: Re: [PATCH v1] mm/vmscan: mitigate spurious kswapd_failures reset from direct reclaim To: "Shakeel Butt" Cc: linux-mm@kvack.org, "Jiayuan Chen" , "Andrew Morton" , "Johannes Weiner" , "David Hildenbrand" , "Michal Hocko" , "Qi Zheng" , "Lorenzo Stoakes" , "Axel Rasmussen" , "Yuanchu Xie" , "Wei Xu" , linux-kernel@vger.kernel.org In-Reply-To: References: <20251222122022.254268-1-jiayuan.chen@linux.dev> <4owaeb7bmkfgfzqd4ztdsi4tefc36cnmpju4yrknsgjm4y32ez@qsgn6lnv3cxb> <2e574085ed3d7775c3b83bb80d302ce45415ac42@linux.dev> X-Migadu-Flow: FLOW_OUT December 23, 2025 at 14:11, "Shakeel Butt" wrote: >=20 >=20On Tue, Dec 23, 2025 at 01:42:37AM +0000, Jiayuan Chen wrote: >=20 >=20>=20 >=20> December 23, 2025 at 05:15, "Shakeel Butt" wrote: > >=20 >=20[...] >=20 >=20>=20 >=20> >=20 >=20> I don't think kswapd is an issue here. The system is out of memory= and > > most of the memory is unreclaimable. Either change the workload to u= se > > less memory or enable swap (or zswap) to have more reclaimable memor= y. > >=20=20 >=20>=20=20 >=20> Hi, > > Thanks for looking into this. > >=20=20 >=20> Sorry, I didn't describe the scenario clearly enough in the origin= al patch. Let me clarify: > >=20=20 >=20> This is a multi-NUMA system where the memory pressure is not globa= l but node-local. The key observation is: > >=20=20 >=20> Node 0: Under memory pressure, most memory is anonymous (unreclaim= able without swap) > > Node 1: Has plenty of reclaimable memory (~60GB file cache out of 12= 5GB total) > >=20 >=20Thanks and now the situation is much more clear. IIUC you are running > multiple workloads (pods) on the system. How is the memcg limits > configured for these workloads. You mentioned memory.high, what about Thanks for the questions. We have pods configured with memory.high and po= ds configured with memory.max. Actually, memory.max itself causes heavy I/O issues for us, because it ke= eps trying to reclaim hot pages within the cgroup aggressively without killing the process.=20 So=20we configured some pods with memory.high instead, since it performs = reclaim in resume_user_mode_work, which somewhat throttles the memory allocation of user processes. > memory.max? Also are you using cpusets to limit the pods to individual > nodes (cpu & memory) or they can run on any node? Yes, we have cpusets(only cpuset.cpus not cpuset.mems) configured for our= cgroups, binding them to specific NUMA nodes. But I don't think this is directly related t= o the issue - the problem can occur with or without cpusets. Even without cpusets.cpus, the= kernel prefers to allocate memory from the node where the process is running, so if a pr= ocess happens to run on a CPU belonging to Node 0, the behavior would be similar. >=20 >=20Overall I still think it is unbalanced numa nodes in terms of memory = and > may for cpu as well. Anyways let's talk about kswapd. > >=20 >=20> Node 0's kswapd runs continuously but cannot reclaim anything > > Direct reclaim succeeds by reclaiming from Node 1 > > Direct reclaim resets kswapd_failures, > >=20 >=20So successful reclaim on one node does not reset kswapd_failures on > other node. The kernel reclaims each node one by one, so if Node 0 > direct reclaim was successfull only then kernel allows to reset the > kswapd_failures of Node 0 to be reset. Let me dig deeper into this. When either memory.max or memory.high is reached, direct reclaim is triggered. The memory being reclaimed depends on the CPU where the process is running. When the problem occurred, we had workloads continuously hitting=20 memory.max=20and workloads continuously hitting memory.high: reclaim_high -> -> try_to_free_mem_cgroup_pages ^ do_try_to_free_pages(zone of current node) | shrink_zones() try_charge_memcg - shrink_node() kswapd_failures =3D 0 Although the pages are hot, if we scan aggressively enough, they will eve= ntually be reclaimed, and then kswapd_failures gets reset to 0 - because even rec= laiming a single page resets kswapd_failures to 0. The end result is that we most workloads, which didn't even hit their hig= h or max limits, experiencing continuous refaults, causing heavy I/O. Thanks. > >=20 >=20> preventing Node 0's kswapd from stopping > > The few file pages on Node 0 are hot and keep refaulting, causing he= avy I/O > >=20 >=20Have you tried numa balancing? Though I think it would be better to > schedule upfront in a way that one node is not overcommitted but numa > balancing provides a dynamic way to adjust the load on each node. Yes, we have tried it. Actually, I submitted a patch about a month ago to= improve its observability: https://lore.kernel.org/all/20251124153331.465306a2@gandalf.local.home/ (though only Steven replied, a bit awkward :( ). We found that the default settings didn't work well for our workloads. Wh= en we tried to increase scan_size to make it more aggressive, we noticed the system l= oad started to increase. So we haven't fully adopted it yet. > Can you dig deeper on who and why Node 0's kswapd_failures is getting > reset? >