From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-qk2-f42.google.com (mail-qk2-f42.google.com [74.125.230.234]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id DAE68509F1A for ; Fri, 25 Sep 2026 23:03:03 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=74.125.230.234 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1790377387; cv=none; b=KN3ZKK1rG9l84vQ51nMQWDVS4c+ExU1rRE5kGxR7ajU9IMG0CgT8vYmGPKt+ZP0JxxtZGfX10AuZDSuyjN1u1mD+TUVcxzZpcvdBJLTyRo5AChtPq9EKicNYTtgFGWQ9zBxpVb8kw3mI6gCH3vM5KCUhxgHlR85Yc5phpkqhD7E= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1790377387; c=relaxed/simple; bh=jandskfkITyjC7rvkXLAEoCYbYIdJkGLy5tzhgU6M4A=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=KrSvbi68JcAi2jArUeptXb+i64640ADXvWAQUo+j9P5lbOrOdGtxbD3DS0OarvpGx4nDh74ifpVN7cSpGXVa8vLbJHO99E2m3PQcG9bxlqpIQLfXoQ2GB+uJE0PONWI45zFEE0fhlO/59zbG4KgCeXnQwyGDewnaxFIJa/M5yMA= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=gourry.net; spf=pass smtp.mailfrom=gourry.net; dkim=pass (2048-bit key) header.d=gourry.net header.i=@gourry.net header.b=A94lZouv; arc=none smtp.client-ip=74.125.230.234 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=gourry.net Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gourry.net Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gourry.net header.i=@gourry.net header.b="A94lZouv" Received: by mail-qk2-f42.google.com with SMTP id af79cd13be357-93bd580489dso134067985a.1 for ; Fri, 25 Sep 2026 16:03:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gourry.net; s=google; t=1790377383; x=1790982183; darn=vger.kernel.org; h=in-reply-to:content-disposition:content-type:mime-version :references:message-id:subject:cc:to:from:date:from:to:cc:subject :date:message-id:reply-to:content-type; bh=PiEVC1BfM3JbJHmz6IrQSpeOS1QFhfgMFnl3j26Htjk=; b=A94lZouvKAzEKsqSL/a6UzMxCA9GzIsQbusgbX18VD/4DwKuWyDb6CLr8AdemoenTW 617YC8IOvzTvhVbOpq/AGAEAVnxLR1trNp2nuOZ/gQBf0n9rM6+fp0sRgPpsvGPcpU14 Us49emjjkXz5t8Ymv5sPiXC+LvBbNpDFTRl0f6Qt5SJepfum1T4JRhrSTH28n/2XHWkL vyz8ZZaHH/XKcZ9yTKCQOZrjW/5wXNTizOddw0dBQfLNqN+LKKjdHZ11eGj3uy6l4U6a i8Ca67ZUXj09625YduPLfKVCsBCTuVPUvbAoNa63euPMbiJmP4dYz6vSluHidDJcvTQL 5XCw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1790377383; x=1790982183; h=in-reply-to:content-disposition:content-type:mime-version :references:message-id:subject:cc:to:from:date:x-gm-gg :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to :content-type; bh=PiEVC1BfM3JbJHmz6IrQSpeOS1QFhfgMFnl3j26Htjk=; b=CMip/TwgJvzzD1bB7HxeVRLSplZF5bGGjRecoPtGUKahPsb9vv3XDnu//eewbV9Hd2 vaNzNu9TiqlCHjWRKAJDl29eWYxTJQ9LBtP7FZQOV2tA54nDLyh01wBu1/Dq4o8FcjcD Qu+qYrhhbWCHYke3K/++TqoYTWNQ3V5/GTq7Cw+/aDZcnH/o/Akx+xmRnVrBquIbIlFt 8UJOZlOAC0SWN5/Kw9pKDRgMpb2oqAQNfmHp8VMtXQ+SJGOMJjYl+9SN9c2658u+bbx9 COeOtyPs89kxaWFvqR0OLdSjMifWPD+/UFsFtQ1oBXQ7irDoMLRBGgnljnTMqnTN1Wm2 /yWQ== X-Forwarded-Encrypted: i=1; AKwUvByzKGwDgFHrx3NT7mVCYyCbj5ysvQ1xTCgCe1dz5WcWfqkeCWuk4ipuHq2xOuyn8YKLD/PpHc3CugtN3C4=@vger.kernel.org X-Gm-Message-State: AFuF++lxp2YAVEb16pgGbgqFFPEieC8A30Y9riTQAlljxLTDERK2jCBA CrF1dEBtFoquFFBd7886sMloNZ6mKexfR49gYkDgyzjKdF0sXXpCL3bd9IZZzfthLkk= X-Gm-Gg: AYBFou3gosCBSLBho3i9QJ0JUtvp9xUQSW8ejRYb5Xz+BgmbG17dLi4XCybOvjCcMVo wYbTOU6cHFro1Dy8Drhs50+1pjMq5Bc1my09MaOhERHYuJV0rM3FLejRtNNFGP3bVuCdTnY053k lAY8BzUuPi3MaFQl8hM66T1YmRHEypSl++TSPRLMvaYUj0o1z+0+N+5dosDOkAnPI6+2LI7aX2u YIaf1wEPho5XuJJPeafnSch2g+FKt0FiMStodVY2XtIiLuRSq24Owcc3syddN5XDA+ONugB0PTW 0cPztF75LGbUekD9otyNpAYteqRAkwgSLbPmU/lZWW9l2XUnJADgUS74xL2eBi8qqzgswuOSi/D zeQT1k12V8D1INQE3ZcAtBERWFaP/8srEZhz9zr0UTRRWOx2gBboYQh8xJmydPrJwIYzWp5cSAe 6LMb56YFRNBxEfNgctGCiq4uBGshmk5bKtT8eUBgvHIELWpbuh8tuyH5eshKz/M4rN1nO54eRVK dmZC6QnQK/XPV1jGEY9a9TcDEYBDx+5KnoZH9x0FW221H4P9LzEQ54= X-Received: by 2002:a05:620a:2614:b0:93a:2bdc:cd57 with SMTP id af79cd13be357-93c43b7f72amr755724385a.2.1790377382584; Fri, 25 Sep 2026 16:03:02 -0700 (PDT) Received: from gourry-fedora-PF4VCD3F (pool-173-79-60-52.washdc.fios.verizon.net. [173.79.60.52]) by smtp.gmail.com with ESMTPSA id af79cd13be357-93c4493af29sm290317085a.28.2026.09.25.16.03.00 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 25 Sep 2026 16:03:02 -0700 (PDT) Date: Fri, 25 Sep 2026 19:02:59 -0400 From: Gregory Price To: linux-mm@kvack.org Cc: Zhigang.Luo@amd.com, arun.george@samsung.com, balbirs@nvidia.com, brendan.jackman@linux.dev, yuzenghui@huawei.com, apopple@nvidia.com, alucerop@amd.com, matthew.brost@intel.com, akpm@linux-foundation.org, david@kernel.org, ljs@kernel.org, liam@infradead.org, vbabka@kernel.org, rppt@kernel.org, surenb@google.com, mhocko@suse.com, corbet@lwn.net, skhan@linuxfoundation.org, gregkh@linuxfoundation.org, rafael@kernel.org, dakr@kernel.org, djbw@kernel.org, vishal.l.verma@intel.com, dave.jiang@intel.com, alison.schofield@intel.com, osandov@osandov.com, jannh@google.com, pfalcato@suse.de, jackmanb@google.com, hannes@cmpxchg.org, ziy@nvidia.com, pbonzini@redhat.com, osalvador@suse.de, joshua.hahnjy@gmail.com, rakie.kim@sk.com, byungchul@sk.com, ying.huang@linux.alibaba.com, kasong@tencent.com, qi.zheng@linux.dev, shakeel.butt@linux.dev, baohua@kernel.org, axelrasmussen@google.com, yuanchu@google.com, weixugc@google.com, yury.norov@gmail.com, linux@rasmusvillemoes.dk, longman@redhat.com, ridong.chen@linux.dev, tj@kernel.org, mkoutny@suse.com, sj@kernel.org, jgg@ziepe.ca, jhubbard@nvidia.com, peterx@redhat.com, baolin.wang@linux.alibaba.com, npache@redhat.com, ryan.roberts@arm.com, dev.jain@arm.com, lance.yang@linux.dev, usama.arif@linux.dev, xu.xin16@zte.com.cn, chengming.zhou@linux.dev, roman.gushchin@linux.dev, muchun.song@linux.dev, linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, driver-core@lists.linux.dev, nvdimm@lists.linux.dev, linux-cxl@vger.kernel.org, linux-debuggers@vger.kernel.org, linux-fsdevel@vger.kernel.org, kvm@vger.kernel.org, cgroups@vger.kernel.org, damon@lists.linux.dev, linux-kselftest@vger.kernel.org, kernel-team@meta.com Subject: Re: [PATCH v5 00/36] Private Memory NUMA Nodes Message-ID: References: <20260720193431.3841992-1-gourry@gourry.net> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20260720193431.3841992-1-gourry@gourry.net> On Mon, Jul 20, 2026 at 03:33:54PM -0400, Gregory Price wrote: > This series introduces the concept of "Private Memory Nodes", which > are opted into two basic functionalities by default: > - page allocation (mm/page_alloc.c) > - OOM killing > All, I wanted to preface v6 with an update, I probably won't be sending it out on-list before LPC (i've been a bit noisey as it is, I'll give it a rest). But I would like to give you the highlights for those interested. The latest working branch can be found here: https://github.com/gourryinverse/linux/tree/scratch/gourry/managed_nodes/rfc6-nonuma You will find the following: - There is no depedency on exported alloc_flags (willy, i found a way), but there is a need for `folio_alloc_private()` to explicitly ask for the ALLOC_ZONELIST_PRIVATE. - feature bits have been converted to node_state[] bits - N_MEMORY_COMMON was added to mean "common memory" - i.e. the existing N_MEMORY set. A private node is now defined as !N_MEMORY_COMMON. - opt-out sites no longer check for !N_MEMORY_PRIVATE, they instead iterate over node_state[N_MEMORY_COMMON]. This is *much* cleaner. - opt-in sites get to define bits in terms of their own service, e.g. /* for each reclaimable node */ for_each_node_mask_state(..., N_MEMORY_RECLAIM) { ... } Or maybe even: for_each_reclaimable_node() :) - There are only 2 node features in the base set: N_MEMORY_RECLAIM - generic reclaim runs on the node N_MEMORY_COMPACTION - compaction will run on the node You'll notice there's no userland numa control support, more on this in a moment. This is all you need to have functional device-private coherent memory that uses the page allocator. Disabling N_MEMORY_RECLAIM does not necessarily mean you're prevented from using vmscan.c and the LRU - it just means we'd need to expose an interface for a !N_MEMORY_RECLAIM node to explicitly ask for reclaim to operate on it. Tiering (Demotion) *to* a node is not supported. We can probably debate whether this should mean tiering *from* a node should be supported or not as well. This is easy to change. - The working branch also includes a minimal compressed-ram service that only supports anonymous memory. Supporting file-backed memory (and shmem) is a whole different can of worms that deserves its own discussion. I will post this as a separate RFC from the base set, but it is available to play with on my working branch. - I ship a basic dax driver for the base series, `anondax` which allows existing devices to online a private node with both compaction and reclaim support for testing. This enables this use case: fd = open(/dev/my_device, ..., O_DIRECT); buf = mmap(fd, ..., MAP_SHARED) my_file = open(myfile, ...); read(buf, myfile); /* fault directly onto device memory */ With swap (and no tiering), this gives you a private over-commitable node for which aspiring drivers can use to implement their own mmu_notifier callback stream to manage device page tables and such. - The reason mempolicy (and user numa in general) is not supported is a critical relationship between the OOM killer and page allocator. In trivial scenarios, if a consumer of a private node overshoots and causes an OOM - it will be the largest consumer and be chosen as the victim. If there are many consumers - what actually happens in practice is the oom killer attempts to select a victim based on *task* policy... which makes every task a candidate. This leads to, among other things, either an OOM storm and/or a full blown panic because a victim cannot be located (depending on the constraints). I've concluded that as-is, this is not tractable. But also, it is not actually needed if the devices provide their own chardev to provide a basic wrapper around folio_alloc_private(). This unfortunately means potential users like guest_memfd are unlikely to be able to use the nodes without hard-coding the allocator interface, as opposed to a mempolicy. I think it's possible instead to *maybe* add N_MEMORY_USER_MIGRATION and allow for explicit movement between nodes, but not full blown mempolicies. The relationship between page_alloc, cpuset, mempolicy, and the oom killer is just too tight to allow allocations without the use of __GFP_THISNODE - which the private allocation interface will enforce. As it stands, the series has been heavily minimized in both line count and patch number. The base series is ~22 commits with the scary diffstat below, but many of the core mm/ patches are similar to the patch below - with nearly all of the real complexity happening in reclaim and memory hotplug. despite the branch name, I intend to drop RFC from here-on, everything else required for the base series is in mm-new. See you at LPC ~Gregory ------- diff --git a/mm/migrate.c b/mm/migrate.c index 7bdcdb57652f8..cd61566b44fa1 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -2266,7 +2266,7 @@ static int __add_folio_for_migration(struct folio *folio, int node, if (is_zero_folio(folio) || is_huge_zero_folio(folio)) return -EFAULT; - if (folio_is_zone_device(folio)) + if (!folio_is_common_memory(folio)) return -ENOENT; if (folio_nid(folio) == node) @@ -2390,7 +2390,7 @@ static int do_pages_move(struct mm_struct *mm, nodemask_t task_nodes, err = -ENODEV; if (node < 0 || node >= MAX_NUMNODES) goto out_flush; - if (!node_state(node, N_MEMORY)) + if (!node_state(node, N_MEMORY_COMMON)) goto out_flush; err = -EACCES; @@ -2475,7 +2475,7 @@ static void do_pages_stat_array(struct mm_struct *mm, unsigned long nr_pages, if (folio) { if (is_zero_folio(folio) || is_huge_zero_folio(folio)) err = -EFAULT; - else if (folio_is_zone_device(folio)) + else if (!folio_is_common_memory(folio)) err = -ENOENT; else err = folio_nid(folio); Documentation/ABI/stable/sysfs-devices-node | 21 +++++++++++++++++ Documentation/admin-guide/cgroup-v2.rst | 7 ++++++ Documentation/mm/physical_memory.rst | 15 ++++++++++++- drivers/base/node.c | 67 +++++++++++++++++++++++++++++++++++++++++++++++++++++-- drivers/dax/kmem.c | 2 +- include/linux/gfp.h | 6 +++++ include/linux/memory_hotplug.h | 2 +- include/linux/mmzone.h | 45 +++++++++++++++++++++++++++++++++++++ include/linux/node.h | 12 ++++++++++ include/linux/nodemask.h | 53 ++++++++++++++++++++++++++++++++++++++++++- kernel/cgroup/cpuset.c | 39 ++++++++++++++++++++++++++------ kernel/sched/fair.c | 6 ++--- mm/compaction.c | 21 +++++++++++------ mm/damon/ops-common.c | 15 ++++++++----- mm/damon/vaddr.c | 20 +++++++++++++---- mm/huge_memory.c | 2 +- mm/internal.h | 22 ++++++++++++++++++ mm/khugepaged.c | 9 ++++++-- mm/ksm.c | 8 ++++--- mm/madvise.c | 6 ++--- mm/memory-tiers.c | 25 +++++++++++---------- mm/memory_hotplug.c | 109 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++------------------------------- mm/mempolicy.c | 51 +++++++++++++++++++++++++----------------- mm/migrate.c | 6 ++--- mm/mm_init.c | 22 +++++++++--------- mm/page_alloc.c | 115 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++--------------- mm/page_alloc.h | 5 +++++ mm/vmscan.c | 31 +++++++++++++++----------- 28 files changed, 585 insertions(+), 157 deletions(-)