Re: [RFC PATCH 0/8] dax/kmem: support for private NUMA nodes

[Richard Cheng <icheng@nvidia.com>]

On Wed, Jul 22, 2026 at 02:35:11PM +0800, Gregory Price wrote:
> This series adds uapi bits for private memory NUMA node [0] control.
> 
> It is not intended for merge - just discussion.
> 
> I have been using this to extensively test mm/ behaviors with private
> nodes [1] and am wondering whether some of this is more generally
> useful for coherent memory devices if private nodes lands.
> 
> For now, this is useful for testing, but locks in UAPI, so this RFC
> is not intended as an intent to upstream - just sharing my testing api.
>   
>   Question 1: Do we want dax control of private node settings?
>               1a) External? (uapi like this?)
> 	      1b) Internal-only? (plumbed through module apis)
> 
>   Question 2: If we do want this, do we want it in kmem or a
>               separate dax driver? (dax/private_mem ?)
> 
>   Question 3: are dax_file and adistance useful regardless?
> 
> This adds 9 interfaces:
> 
>   dax0.0/dax_file
>      this makes /dev/dax0.0 mmap-able when online as kmem
>      it essentially attaches a default mempolicy to the vma
>      that binds it to the dax device's node.
> 
>   dax0.0/private
>      this switches whether hotplug sets N_MEMORY or N_MEMORY_PRIVATE
> 
>   dax0.0/adistance
>      this allows overriding the default adistance to for the node
>      into a specific tier.  That allows for testing tiering in both
>      directions (demotion from/to the node)
> 
>   dax0.0/reclaim
>   dax0.0/user_numa
>   dax0.0/hotunplug
>   dax0.0/demotion
>   dax0.0/numa_balancing
>   dax0.0/ltpin
>      these bits all enable their respective NODE_PRIVATE_CAP_ bit
> 
>      reclaim   - enables reclaim on the node
>      user_numa - enables mbind/mempolicy/move_pages/migrate_pages
>      hotunplug - enables hotunplug to migrate memory
>      demotion  - enables reclaim to demote from or to the node
>      numa_balancing - allows prot_none injection on the node's folios
>      ltpin     - allows longterm pin on node's folios to work
>

Hi Gregory,

I have some thoughts especially for ownership and configs.

I think it would be good to expose dax control of private node, just with
the following guarantees.

Let's make it clear that userspace should never become the owner
of private node, they should be driver-owned and 1-to-1 mapping relationship.

A userspace tool can request to reconfigure some capability of the DAX device, and
maybe how the driver expose it.

A dax range may have several deployment rules, e.g. devdax, system-ram or private, with
the patch series' tool, it looks like a userspace tool can convert a DAX device between
dax and system-ram mode, I think that's unsafe especially when the region is online as
system-ram.

And it will cause coherence issue to be resolved, if no special or strong use cases, I would
prevent having such ability to convert the rules in runtime, especially for fixed-purpose device.

For capability bitmap part, I think it's reasonable to expose it, but with 2 concepts
- supported_caps : immutable, supplied by the provider/vendor
- enabled-caps : selected policy, subset of supported_caps

The invariant should always hold
"""
enabled_caps & ~supported_caps == 0
"""

This way we can prevent userspace from touching things that they shouldn't.

Maybe a demo with a driver would be nice ?

This is my initial thoughts of this, thanks.

Best regards,
Richard Cheng.

> [0] private node series
> https://lore.kernel.org/linux-mm/20260720193431.3841992-1-gourry@gourry.net/
> 
> [1] selftests/dax extensions for all of this that show use cases and
>     testing for the private node series - including all user api's
>     like mbind, set_mempolicy, move_pages, migrate_privates - as well
>     as reclaim behaviors.
> https://github.com/gourryinverse/linux/commits/scratch/gourry/managed_nodes/dax_private-mm-new/
> 
> Gregory Price (8):
>   dax/kmem: add dax_file= to expose an anonymous-fault char device
>   dax/kmem: add private= to online memory as an N_MEMORY_PRIVATE node
>   dax/kmem: add the reclaim opt-in (CAP_RECLAIM) for private nodes
>   dax/kmem: add the user_numa opt-in (CAP_USER_NUMA) for private nodes
>   dax/kmem: add the hotunplug opt-in (CAP_HOTUNPLUG) for private nodes
>   dax/kmem: add the demotion and numa_balancing opt-ins for private
>     nodes
>   dax/kmem: add the ltpin opt-in (CAP_LTPIN) for private nodes
>   dax/kmem: add dax/adistance to make private node adistance
>     configurable
> 
>  drivers/dax/kmem.c | 368 ++++++++++++++++++++++++++++++++++++++++++++-
>  1 file changed, 366 insertions(+), 2 deletions(-)
> 
> -- 
> 2.53.0-Meta
> 
>