From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0E3A5125B2 for ; Sun, 14 Dec 2025 23:35:51 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1765755352; cv=none; b=h+c3eSwY3ZJFAIacIIOOjhBOYs93oMXlOHrcV5q5xGa2QIwNvc0pGygxurapxDE6HI5dxb2YHvEJ3mP/cIyrtvdBkbc7WKAveGJJuTI39OtdSCXWctTM3qunqRXI+0Jn8QSQLpU8I+It3jg+FxQA6UmrRLahF8K8+/YOBOAVwrg= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1765755352; c=relaxed/simple; bh=L3Ab+kcdxPFFSrDyEW8IQM+oer/MoKJhtSPABmB3aDU=; h=Date:From:To:Cc:Subject:Message-Id:In-Reply-To:References: Mime-Version:Content-Type; b=D8s07d59LqWvDbfSNlUp4GS3RShlEa6sCAHM7kwJ6YEiVXBbg/K5eqxNzSMq9ImB4bE1Qfc7LHsF60BYlM7BAULnnY8HT0YoQ+dqUvp6Hmp4Lkoe3raQdxB0MRLQkCrY90DVrjcs1uO18ICKg5OssqHdVB3+mUEKjczAxBEF9BU= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux-foundation.org header.i=@linux-foundation.org header.b=vkf5Xdq4; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux-foundation.org header.i=@linux-foundation.org header.b="vkf5Xdq4" Received: by smtp.kernel.org (Postfix) with ESMTPSA id B5A35C4CEF1; Sun, 14 Dec 2025 23:35:50 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linux-foundation.org; s=korg; t=1765755351; bh=L3Ab+kcdxPFFSrDyEW8IQM+oer/MoKJhtSPABmB3aDU=; h=Date:From:To:Cc:Subject:In-Reply-To:References:From; b=vkf5Xdq4SxZjOlCAuUs4Ju3aZrtYxQgjFs9mf8R1xkzqDTs3SMR9Fw+6qoiC7XI6Q URYnOcujAElZhvFyX8Tyedp5+UnrLBAN5P01m706hM3N48oEhrbqFzbJQtJbSl6KXM EkKNJC0CSvlTrvyaW3JqwrVhoOz4oWZdxGeICajc= Date: Sun, 14 Dec 2025 15:35:50 -0800 From: Andrew Morton To: Mathieu Desnoyers Cc: linux-kernel@vger.kernel.org, "Paul E. McKenney" , Steven Rostedt , Masami Hiramatsu , Dennis Zhou , Tejun Heo , Christoph Lameter , Martin Liu , David Rientjes , christian.koenig@amd.com, Shakeel Butt , SeongJae Park , Michal Hocko , Johannes Weiner , Sweet Tea Dorminy , Lorenzo Stoakes , "Liam R . Howlett" , Mike Rapoport , Suren Baghdasaryan , Vlastimil Babka , Christian Brauner Subject: Re: [PATCH v10 0/3] mm: Fix OOM killer inaccuracy on large many-core systems Message-Id: <20251214153550.10f171f0c98e4ece9a0f1bfe@linux-foundation.org> In-Reply-To: <20251213185608.3418096-1-mathieu.desnoyers@efficios.com> References: <20251213185608.3418096-1-mathieu.desnoyers@efficios.com> X-Mailer: Sylpheed 3.8.0beta1 (GTK+ 2.24.33; x86_64-pc-linux-gnu) Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit On Sat, 13 Dec 2025 13:56:05 -0500 Mathieu Desnoyers wrote: > Introduce hierarchical per-cpu counters and use them for RSS tracking to > fix the per-mm RSS tracking which has become too inaccurate for OOM > killer purposes on large many-core systems. > > The following rss tracking issues were noted by Sweet Tea Dorminy [1], > which lead to picking wrong tasks as OOM kill target: > > Recently, several internal services had an RSS usage regression as part of a > kernel upgrade. Previously, they were on a pre-6.2 kernel and were able to > read RSS statistics in a backup watchdog process to monitor and decide if > they'd overrun their memory budget. Now, however, a representative service > with five threads, expected to use about a hundred MB of memory, on a 250-cpu > machine had memory usage tens of megabytes different from the expected amount > -- this constituted a significant percentage of inaccuracy, causing the > watchdog to act. > > This was a result of commit f1a7941243c1 ("mm: convert mm's rss stats > into percpu_counter") [1]. Previously, the memory error was bounded by > 64*nr_threads pages, a very livable megabyte. Now, however, as a result of > scheduler decisions moving the threads around the CPUs, the memory error could > be as large as a gigabyte. > > This is a really tremendous inaccuracy for any few-threaded program on a > large machine and impedes monitoring significantly. These stat counters are > also used to make OOM killing decisions, so this additional inaccuracy could > make a big difference in OOM situations -- either resulting in the wrong > process being killed, or in less memory being returned from an OOM-kill than > expected. > > The approach proposed here is to replace this by the hierarchical > per-cpu counters, which bounds the inaccuracy based on the system > topology with O(N*logN). > > Notable change for v10: The new patch 3/3 changes the implementation of > the oom killer task selection to a 2-pass algorithm, where the first > pass uses the fast approximation provided by the hierarchical percpu > counters, and the second pass does a precise sum for all tasks which > have badness values within the range of the approximation accuracy. > > I've done moderate testing of this series on a 256-core VM with 128GB > RAM. Figuring out whether this indeed helps solve issues with real-life > workloads will require broader feedback from the community. > > The one request I did not have time to fulfill yet is to port the > tests from the librseq feature branch implementation (userspace) to the > kernel selftests. > > This series is based on v6.18. > > Andrew, are you interested to try this out in mm-new ? Yes. We have to start somewhere. As you kind of mention, it's going to be difficult to determine when this is ready to go upstream. I assume that to really know this will required detailed and lengthy fleet-wide operation and observation. What sort of drawbacks do you think people miht encounter with this change?