[RFC PATCH 0/8] dax/kmem: support for private NUMA nodes

[RFC PATCH 0/8] dax/kmem: support for private NUMA nodes

[Gregory Price]

This series adds uapi bits for private memory NUMA node [0] control.

It is not intended for merge - just discussion.

I have been using this to extensively test mm/ behaviors with private
nodes [1] and am wondering whether some of this is more generally
useful for coherent memory devices if private nodes lands.

For now, this is useful for testing, but locks in UAPI, so this RFC
is not intended as an intent to upstream - just sharing my testing api.
  
  Question 1: Do we want dax control of private node settings?
              1a) External? (uapi like this?)
	      1b) Internal-only? (plumbed through module apis)

  Question 2: If we do want this, do we want it in kmem or a
              separate dax driver? (dax/private_mem ?)

  Question 3: are dax_file and adistance useful regardless?

This adds 9 interfaces:

  dax0.0/dax_file
     this makes /dev/dax0.0 mmap-able when online as kmem
     it essentially attaches a default mempolicy to the vma
     that binds it to the dax device's node.

  dax0.0/private
     this switches whether hotplug sets N_MEMORY or N_MEMORY_PRIVATE

  dax0.0/adistance
     this allows overriding the default adistance to for the node
     into a specific tier.  That allows for testing tiering in both
     directions (demotion from/to the node)

  dax0.0/reclaim
  dax0.0/user_numa
  dax0.0/hotunplug
  dax0.0/demotion
  dax0.0/numa_balancing
  dax0.0/ltpin
     these bits all enable their respective NODE_PRIVATE_CAP_ bit

     reclaim   - enables reclaim on the node
     user_numa - enables mbind/mempolicy/move_pages/migrate_pages
     hotunplug - enables hotunplug to migrate memory
     demotion  - enables reclaim to demote from or to the node
     numa_balancing - allows prot_none injection on the node's folios
     ltpin     - allows longterm pin on node's folios to work

[0] private node series
https://lore.kernel.org/linux-mm/20260720193431.3841992-1-gourry@gourry.net/

[1] selftests/dax extensions for all of this that show use cases and
    testing for the private node series - including all user api's
    like mbind, set_mempolicy, move_pages, migrate_privates - as well
    as reclaim behaviors.
https://github.com/gourryinverse/linux/commits/scratch/gourry/managed_nodes/dax_private-mm-new/

Gregory Price (8):
  dax/kmem: add dax_file= to expose an anonymous-fault char device
  dax/kmem: add private= to online memory as an N_MEMORY_PRIVATE node
  dax/kmem: add the reclaim opt-in (CAP_RECLAIM) for private nodes
  dax/kmem: add the user_numa opt-in (CAP_USER_NUMA) for private nodes
  dax/kmem: add the hotunplug opt-in (CAP_HOTUNPLUG) for private nodes
  dax/kmem: add the demotion and numa_balancing opt-ins for private
    nodes
  dax/kmem: add the ltpin opt-in (CAP_LTPIN) for private nodes
  dax/kmem: add dax/adistance to make private node adistance
    configurable

 drivers/dax/kmem.c | 368 ++++++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 366 insertions(+), 2 deletions(-)

-- 
2.53.0-Meta

[RFC PATCH 1/8] dax/kmem: add dax_file= to expose an anonymous-fault char device

[Gregory Price]

Add a "dax_file" knob to dax/kmem.

When set, installs a char device on the existing /dev/daxN.N whose
mmap() turns the mapping into an ordinary ANONYMOUS mapping bound
to the device's node - so every folio (on fault and swap-in) lands
on that node exactly like an mbind() to it.

Writing 0 removes the char device again.

Because kmem devices hotplug memory to mm/, faulted memory is NOT
limited to just the dax device's memory ranges.  This is not
particularly useful for normal nodes, but will be useful for private
nodes where each node can only have one owner - and therefore the
faults are limited to the device's memory ranges.

Note: The mapping is deliberately not S_DAX - otherwise the vma would
be wrongly excluded from migration, NUMA balancing, and the like.

The node binding is a device-lifetime mempolicy, built once at online
time. A plain fault would place the first folio correctly without it,
but swap-in would likely misplace the folio, so the policy is what
keeps the mapping pinned to the node.

MAP_SHARED is rejected (no shared mappings are possible here).

Signed-off-by: Gregory Price <gourry@gourry.net>
---
 drivers/dax/kmem.c | 171 ++++++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 170 insertions(+), 1 deletion(-)

diff --git a/drivers/dax/kmem.c b/drivers/dax/kmem.c
index c3faf18e7f85d..d9021cf148982 100644
--- a/drivers/dax/kmem.c
+++ b/drivers/dax/kmem.c
@@ -5,11 +5,13 @@
 #include <linux/memory.h>
 #include <linux/module.h>
 #include <linux/device.h>
+#include <linux/cdev.h>
 #include <linux/slab.h>
 #include <linux/dax.h>
 #include <linux/fs.h>
 #include <linux/mm.h>
 #include <linux/mman.h>
+#include <linux/mempolicy.h>
 #include <linux/memory-tiers.h>
 #include <linux/memory_hotplug.h>
 #include <linux/string_helpers.h>
@@ -45,8 +47,11 @@ static int dax_kmem_range(struct dev_dax *dev_dax, int i, struct range *r)
 struct dax_kmem_data {
 	const char *res_name;
 	int mgid;
+	int numa_node;
 	int state;
-	struct mutex lock; /* protects hotplug state transitions */
+	struct mutex lock; /* protects hotplug state transitions and config */
+	bool dax_file; /* when set, cdev allows mmap */
+	struct mempolicy *policy; /* device-lifetime bind for dax-file mmap */
 	struct resource *res[];
 };
 
@@ -65,6 +70,90 @@ static void kmem_put_memory_types(void)
 	mt_put_memory_types(&kmem_memory_types);
 }
 
+#ifdef CONFIG_NUMA
+static void kmem_vma_set_policy(struct vm_area_struct *vma, struct mempolicy *pol)
+{
+	vma->vm_policy = pol;
+}
+#else
+static void kmem_vma_set_policy(struct vm_area_struct *vma, struct mempolicy *pol)
+{
+}
+#endif
+
+/*
+ * On mmap, install MPOL_F_PRIVATE mempolicy to bind to that node's memory,
+ * so every folio (fault and swap-in) lands on the node exactly like an mbind().
+ * Only present when dax_file=true, otherwise returns -ENXIO as before.
+ */
+static int kmem_anon_mmap(struct file *filp, struct vm_area_struct *vma)
+{
+	struct dev_dax *dev_dax = filp->private_data;
+	struct dax_kmem_data *data = dev_get_drvdata(&dev_dax->dev);
+	int rc = 0, id;
+
+	id = dax_read_lock();
+	if (!dax_alive(dev_dax->dax_dev))
+		rc = -ENXIO;
+	dax_read_unlock(id);
+	if (rc)
+		return rc;
+
+	/* Private mappings are not shared by definition, reject MAP_SHARED. */
+	if (vma->vm_flags & VM_SHARED)
+		return -EINVAL;
+
+	/* No policy means the node is not online yet - nothing to map. */
+	if (!data->policy)
+		return -ENXIO;
+
+	vma_set_anonymous(vma);
+	mpol_get(data->policy);
+	kmem_vma_set_policy(vma, data->policy);
+	return 0;
+}
+
+static int kmem_anon_open(struct inode *inode, struct file *filp)
+{
+	struct dax_device *dax_dev = inode_dax(inode);
+	struct dev_dax *dev_dax = dax_get_private(dax_dev);
+
+	filp->private_data = dev_dax;
+	/* Deliberately NOT S_DAX to allow normal mm/ operations on the vma */
+	return 0;
+}
+
+static const struct file_operations kmem_anon_fops = {
+	.llseek = noop_llseek,
+	.owner = THIS_MODULE,
+	.open = kmem_anon_open,
+	.mmap = kmem_anon_mmap,
+};
+
+/* Installs dax-file char device fops on the existing /dev/daxN.N node. */
+static int kmem_dax_file_cdev_add(struct dev_dax *dev_dax)
+{
+	struct dax_device *dax_dev = dev_dax->dax_dev;
+	struct device *dev = &dev_dax->dev;
+	struct cdev *cdev = dax_inode(dax_dev)->i_cdev;
+	int rc;
+
+	cdev_init(cdev, &kmem_anon_fops);
+	cdev->owner = dev->driver->owner;
+	cdev_set_parent(cdev, &dev->kobj);
+	rc = cdev_add(cdev, dev->devt, 1);
+	if (rc)
+		return rc;
+	run_dax(dax_dev);
+	return 0;
+}
+
+static void kmem_dax_file_cdev_del(struct dev_dax *dev_dax)
+{
+	kill_dev_dax(dev_dax);
+	cdev_del(dax_inode(dev_dax->dax_dev)->i_cdev);
+}
+
 /* True for the online states a kmem dax device can hold. */
 static bool dax_kmem_state_is_online(int state)
 {
@@ -339,6 +428,7 @@ static ssize_t state_store(struct device *dev, struct device_attribute *attr,
 {
 	struct dev_dax *dev_dax = to_dev_dax(dev);
 	struct dax_kmem_data *data = dev_get_drvdata(dev);
+	struct mempolicy *pol;
 	int online_type;
 	int rc;
 
@@ -377,8 +467,69 @@ static ssize_t state_store(struct device *dev, struct device_attribute *attr,
 	}
 
 	data->state = online_type;
+
+	/* If in dax-file mode, build the bind policy applied during mmap */
+	if (data->dax_file && !data->policy) {
+		pol = mpol_private_bind(data->numa_node);
+		if (IS_ERR(pol))
+			dev_warn(dev, "dax-file bind failed: %ld\n", PTR_ERR(pol));
+		else
+			data->policy = pol;
+	}
+
+	return len;
+}
+
+static ssize_t dax_file_show(struct device *dev, struct device_attribute *attr,
+			     char *buf)
+{
+	struct dax_kmem_data *data = dev_get_drvdata(dev);
+
+	if (!data)
+		return -ENXIO;
+	return sysfs_emit(buf, "%d\n", data->dax_file);
+}
+
+static ssize_t dax_file_store(struct device *dev, struct device_attribute *attr,
+			      const char *buf, size_t len)
+{
+	struct dev_dax *dev_dax = to_dev_dax(dev);
+	struct dax_kmem_data *data = dev_get_drvdata(dev);
+	bool enable;
+	int rc;
+
+	if (!data)
+		return -ENXIO;
+
+	rc = kstrtobool(buf, &enable);
+	if (rc)
+		return rc;
+
+	guard(mutex)(&data->lock);
+
+	/* dax_file= only reconfigures the device while it holds no memory. */
+	if (data->state != DAX_KMEM_UNPLUGGED)
+		return -EBUSY;
+
+	if (enable == data->dax_file)
+		return len;
+
+	if (enable) {
+		/* Add the anon-fault cdev; mmap returns -ENXIO until onlined. */
+		rc = kmem_dax_file_cdev_add(dev_dax);
+		if (rc)
+			return rc;
+		data->dax_file = true;
+	} else {
+		kmem_dax_file_cdev_del(dev_dax);
+		mpol_put(data->policy);
+		data->policy = NULL;
+		data->dax_file = false;
+	}
+
 	return len;
 }
+static DEVICE_ATTR_RW(dax_file);
 
 static int dev_dax_kmem_probe(struct dev_dax *dev_dax)
 {
@@ -448,6 +599,7 @@ static int dev_dax_kmem_probe(struct dev_dax *dev_dax)
 	if (rc < 0)
 		goto err_reg_mgid;
 	data->mgid = rc;
+	data->numa_node = numa_node;
 	data->state = DAX_KMEM_UNPLUGGED;
 	mutex_init(&data->lock);
 
@@ -516,6 +668,13 @@ static void dev_dax_kmem_remove(struct dev_dax *dev_dax)
 	struct device *dev = &dev_dax->dev;
 	struct dax_kmem_data *data = dev_get_drvdata(dev);
 
+	/* If enabled, clean up the dax-file char device configuration. */
+	if (data->dax_file) {
+		kmem_dax_file_cdev_del(dev_dax);
+		mpol_put(data->policy);
+		data->policy = NULL;
+	}
+
 	/*
 	 * Remove every range that is still added.  dax_kmem_remove_ranges()
 	 * uses remove_memory(), which never offlines: an online block fails
@@ -548,6 +707,15 @@ static void dev_dax_kmem_remove(struct dev_dax *dev_dax)
 #else
 static void dev_dax_kmem_remove(struct dev_dax *dev_dax)
 {
+	struct dax_kmem_data *data = dev_get_drvdata(&dev_dax->dev);
+
+	/* If enabled, clean up the dax-file char device configuration. */
+	if (data && data->dax_file) {
+		kmem_dax_file_cdev_del(dev_dax);
+		mpol_put(data->policy);
+		data->policy = NULL;
+	}
+
 	/*
 	 * Without hotremove purposely leak the request_mem_region() for the
 	 * device-dax range and return '0' to ->remove() attempts. The removal
@@ -563,6 +731,7 @@ static DEVICE_ATTR_RW(state);
 
 static struct attribute *dev_dax_kmem_attrs[] = {
 	&dev_attr_state.attr,
+	&dev_attr_dax_file.attr,
 	NULL,
 };
 ATTRIBUTE_GROUPS(dev_dax_kmem);
-- 
2.53.0-Meta

[RFC PATCH 2/8] dax/kmem: add private= to online memory as an N_MEMORY_PRIVATE node

[Gregory Price]

Add a "private" knob to dax/kmem.  Only writable when state=unplugged.

Switches the device between onlining as N_MEMORY and N_MEMORY_PRIVATE.
This marks the node private and excludes it from every fallback zonelist.

Fails if the node already has online (non-private) memory or a if the
node has a different private owner.

The base plumbing for capability bit flags are added here, but no
capabilities are added until later commits.

A private node after this commit is solely accessible via mmap on
the dax character device (/dev/daxN.M).

Because a fully isolated private node cannot reclaim or compact, its
dax-file mappings default to base-page faults (VM_NOHUGEPAGE). This
makes allocation / failures more deterministic, otherwise a fault may
fail due to a lock of given order despite memory still being free.

When/if reclaim is supported (and the node opted-in), this can be relaxed.

Signed-off-by: Gregory Price <gourry@gourry.net>
---
 drivers/dax/kmem.c | 85 ++++++++++++++++++++++++++++++++++++++++++++--
 1 file changed, 83 insertions(+), 2 deletions(-)

diff --git a/drivers/dax/kmem.c b/drivers/dax/kmem.c
index d9021cf148982..c4c18aeb43c69 100644
--- a/drivers/dax/kmem.c
+++ b/drivers/dax/kmem.c
@@ -14,6 +14,7 @@
 #include <linux/mempolicy.h>
 #include <linux/memory-tiers.h>
 #include <linux/memory_hotplug.h>
+#include <linux/node_private.h>
 #include <linux/string_helpers.h>
 #include "dax-private.h"
 #include "bus.h"
@@ -52,6 +53,9 @@ struct dax_kmem_data {
 	struct mutex lock; /* protects hotplug state transitions and config */
 	bool dax_file; /* when set, cdev allows mmap */
 	struct mempolicy *policy; /* device-lifetime bind for dax-file mmap */
+	bool private; /* when set, memory is onlined as N_MEMORY_PRIVATE */
+	unsigned long caps; /* NODE_PRIVATE_CAP_* flags */
+	struct node_private np;
 	struct resource *res[];
 };
 
@@ -108,6 +112,13 @@ static int kmem_anon_mmap(struct file *filp, struct vm_area_struct *vma)
 		return -ENXIO;
 
 	vma_set_anonymous(vma);
+	/*
+	 * Default a private node's dax-file mappings to base-page faults.
+	 * Without reclaim/compaction, high-order allocations can fail despite
+	 * memory still being available. Can be relaxed with reclaim support.
+	 */
+	if (data->private)
+		vm_flags_set(vma, VM_NOHUGEPAGE);
 	mpol_get(data->policy);
 	kmem_vma_set_policy(vma, data->policy);
 	return 0;
@@ -187,6 +198,10 @@ static int dax_kmem_do_hotplug(struct dev_dax *dev_dax,
 	if (online_type < MMOP_OFFLINE || online_type > MMOP_ONLINE_MOVABLE)
 		return -EINVAL;
 
+	/* Capabilities are stable as long as memory is online */
+	if (data->private)
+		data->np.caps = data->caps;
+
 	for (i = 0; i < dev_dax->nr_range; i++) {
 		struct range range;
 
@@ -215,7 +230,7 @@ static int dax_kmem_do_hotplug(struct dev_dax *dev_dax,
 		 */
 		rc = __add_memory_driver_managed(data->mgid, range.start,
 				range_len(&range), kmem_name, mhp_flags,
-				online_type, NULL);
+				online_type, data->private ? &data->np : NULL);
 
 		if (rc) {
 			dev_warn(dev, "mapping%d: %#llx-%#llx memory add failed\n",
@@ -480,6 +495,55 @@ static ssize_t state_store(struct device *dev, struct device_attribute *attr,
 	return len;
 }
 
+static const struct attribute_group dax_kmem_private_group;
+
+static ssize_t private_show(struct device *dev, struct device_attribute *attr,
+			    char *buf)
+{
+	struct dax_kmem_data *data = dev_get_drvdata(dev);
+
+	if (!data)
+		return -ENXIO;
+	return sysfs_emit(buf, "%d\n", data->private);
+}
+
+static ssize_t private_store(struct device *dev, struct device_attribute *attr,
+			     const char *buf, size_t len)
+{
+	struct dax_kmem_data *data = dev_get_drvdata(dev);
+	bool enable;
+	int rc;
+
+	if (!data)
+		return -ENXIO;
+
+	rc = kstrtobool(buf, &enable);
+	if (rc)
+		return rc;
+
+	guard(mutex)(&data->lock);
+
+	if (data->state != DAX_KMEM_UNPLUGGED)
+		return -EBUSY;
+
+	if (enable == data->private)
+		return len;
+
+	if (enable) {
+		/* Add the per-service opt-in attributes. */
+		rc = sysfs_create_group(&dev->kobj, &dax_kmem_private_group);
+		if (rc)
+			return rc;
+		data->private = true;
+	} else {
+		sysfs_remove_group(&dev->kobj, &dax_kmem_private_group);
+		data->private = false;
+	}
+
+	return len;
+}
+static DEVICE_ATTR_RW(private);
+
 static ssize_t dax_file_show(struct device *dev, struct device_attribute *attr,
 			     char *buf)
 {
@@ -531,6 +595,14 @@ static ssize_t dax_file_store(struct device *dev, struct device_attribute *attr,
 }
 static DEVICE_ATTR_RW(dax_file);
 
+/* Per-service opt-ins. Visibility toggled by 'private' control */
+static struct attribute *dax_kmem_private_attrs[] = {
+	NULL,
+};
+static const struct attribute_group dax_kmem_private_group = {
+	.attrs = dax_kmem_private_attrs,
+};
+
 static int dev_dax_kmem_probe(struct dev_dax *dev_dax)
 {
 	struct device *dev = &dev_dax->dev;
@@ -601,6 +673,7 @@ static int dev_dax_kmem_probe(struct dev_dax *dev_dax)
 	data->mgid = rc;
 	data->numa_node = numa_node;
 	data->state = DAX_KMEM_UNPLUGGED;
+	data->np.owner = data;
 	mutex_init(&data->lock);
 
 	dev_set_drvdata(dev, data);
@@ -668,6 +741,9 @@ static void dev_dax_kmem_remove(struct dev_dax *dev_dax)
 	struct device *dev = &dev_dax->dev;
 	struct dax_kmem_data *data = dev_get_drvdata(dev);
 
+	if (data->private)
+		sysfs_remove_group(&dev->kobj, &dax_kmem_private_group);
+
 	/* If enabled, clean up the dax-file char device configuration. */
 	if (data->dax_file) {
 		kmem_dax_file_cdev_del(dev_dax);
@@ -707,7 +783,11 @@ static void dev_dax_kmem_remove(struct dev_dax *dev_dax)
 #else
 static void dev_dax_kmem_remove(struct dev_dax *dev_dax)
 {
-	struct dax_kmem_data *data = dev_get_drvdata(&dev_dax->dev);
+	struct device *dev = &dev_dax->dev;
+	struct dax_kmem_data *data = dev_get_drvdata(dev);
+
+	if (data && data->private)
+		sysfs_remove_group(&dev->kobj, &dax_kmem_private_group);
 
 	/* If enabled, clean up the dax-file char device configuration. */
 	if (data && data->dax_file) {
@@ -732,6 +812,7 @@ static DEVICE_ATTR_RW(state);
 static struct attribute *dev_dax_kmem_attrs[] = {
 	&dev_attr_state.attr,
 	&dev_attr_dax_file.attr,
+	&dev_attr_private.attr,
 	NULL,
 };
 ATTRIBUTE_GROUPS(dev_dax_kmem);
-- 
2.53.0-Meta

[RFC PATCH 3/8] dax/kmem: add the reclaim opt-in (CAP_RECLAIM) for private nodes

[Gregory Price]

Add the first per-service opt-in for a private node: "reclaim".
Also introduces the KMEM_PRIVATE_CAP_ATTR() helper for future bits.

Sets NODE_PRIVATE_CAP_RECLAIM for the device's private node.

A private kmem node opted into reclaim recovers THP compatibility.

Signed-off-by: Gregory Price <gourry@gourry.net>
---
 drivers/dax/kmem.c | 38 +++++++++++++++++++++++++++++++++++++-
 1 file changed, 37 insertions(+), 1 deletion(-)

diff --git a/drivers/dax/kmem.c b/drivers/dax/kmem.c
index c4c18aeb43c69..cf6cb927e5ed5 100644
--- a/drivers/dax/kmem.c
+++ b/drivers/dax/kmem.c
@@ -117,7 +117,7 @@ static int kmem_anon_mmap(struct file *filp, struct vm_area_struct *vma)
 	 * Without reclaim/compaction, high-order allocations can fail despite
 	 * memory still being available. Can be relaxed with reclaim support.
 	 */
-	if (data->private)
+	if (!node_allows_reclaim(data->numa_node))
 		vm_flags_set(vma, VM_NOHUGEPAGE);
 	mpol_get(data->policy);
 	kmem_vma_set_policy(vma, data->policy);
@@ -595,8 +595,44 @@ static ssize_t dax_file_store(struct device *dev, struct device_attribute *attr,
 }
 static DEVICE_ATTR_RW(dax_file);
 
+/* NODE_PRIVATE_CAP_* opt-in toggles - captured in data->caps */
+#define KMEM_PRIVATE_CAP_ATTR(name, CAP)				\
+static ssize_t name##_show(struct device *dev,				\
+			   struct device_attribute *attr, char *buf)	\
+{									\
+	struct dax_kmem_data *data = dev_get_drvdata(dev);		\
+									\
+	return sysfs_emit(buf, "%d\n", !!(data->caps & (CAP)));		\
+}									\
+static ssize_t name##_store(struct device *dev,				\
+			    struct device_attribute *attr,		\
+			    const char *buf, size_t len)		\
+{									\
+	struct dax_kmem_data *data = dev_get_drvdata(dev);		\
+	bool enable;							\
+	ssize_t rc;							\
+									\
+	rc = kstrtobool(buf, &enable);					\
+	if (rc)								\
+		return rc;						\
+									\
+	guard(mutex)(&data->lock);					\
+									\
+	if (data->state != DAX_KMEM_UNPLUGGED)				\
+		return -EBUSY;						\
+	if (enable)							\
+		data->caps |= (CAP);					\
+	else								\
+		data->caps &= ~(CAP);					\
+	return len;							\
+}									\
+static DEVICE_ATTR_RW(name)
+
+KMEM_PRIVATE_CAP_ATTR(reclaim, NODE_PRIVATE_CAP_RECLAIM);
+
 /* Per-service opt-ins. Visibility toggled by 'private' control */
 static struct attribute *dax_kmem_private_attrs[] = {
+	&dev_attr_reclaim.attr,
 	NULL,
 };
 static const struct attribute_group dax_kmem_private_group = {
-- 
2.53.0-Meta

[RFC PATCH 4/8] dax/kmem: add the user_numa opt-in (CAP_USER_NUMA) for private nodes

[Gregory Price]

Add the "user_numa" setting NODE_PRIVATE_CAP_USER_NUMA so userspace
may control residency on the node via mempolicy and migration.

Signed-off-by: Gregory Price <gourry@gourry.net>
---
 drivers/dax/kmem.c | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/drivers/dax/kmem.c b/drivers/dax/kmem.c
index cf6cb927e5ed5..8023db23e2f99 100644
--- a/drivers/dax/kmem.c
+++ b/drivers/dax/kmem.c
@@ -629,10 +629,12 @@ static ssize_t name##_store(struct device *dev,				\
 static DEVICE_ATTR_RW(name)
 
 KMEM_PRIVATE_CAP_ATTR(reclaim, NODE_PRIVATE_CAP_RECLAIM);
+KMEM_PRIVATE_CAP_ATTR(user_numa, NODE_PRIVATE_CAP_USER_NUMA);
 
 /* Per-service opt-ins. Visibility toggled by 'private' control */
 static struct attribute *dax_kmem_private_attrs[] = {
 	&dev_attr_reclaim.attr,
+	&dev_attr_user_numa.attr,
 	NULL,
 };
 static const struct attribute_group dax_kmem_private_group = {
-- 
2.53.0-Meta

[RFC PATCH 5/8] dax/kmem: add the hotunplug opt-in (CAP_HOTUNPLUG) for private nodes

[Gregory Price]

Add the "hotunplug" setting for NODE_PRIVATE_CAP_HOTUNPLUG, so the
node's memory may be bulk-evacuated for device removal.

Signed-off-by: Gregory Price <gourry@gourry.net>
---
 drivers/dax/kmem.c | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/drivers/dax/kmem.c b/drivers/dax/kmem.c
index 8023db23e2f99..3892ad44d4e46 100644
--- a/drivers/dax/kmem.c
+++ b/drivers/dax/kmem.c
@@ -630,11 +630,13 @@ static DEVICE_ATTR_RW(name)
 
 KMEM_PRIVATE_CAP_ATTR(reclaim, NODE_PRIVATE_CAP_RECLAIM);
 KMEM_PRIVATE_CAP_ATTR(user_numa, NODE_PRIVATE_CAP_USER_NUMA);
+KMEM_PRIVATE_CAP_ATTR(hotunplug, NODE_PRIVATE_CAP_HOTUNPLUG);
 
 /* Per-service opt-ins. Visibility toggled by 'private' control */
 static struct attribute *dax_kmem_private_attrs[] = {
 	&dev_attr_reclaim.attr,
 	&dev_attr_user_numa.attr,
+	&dev_attr_hotunplug.attr,
 	NULL,
 };
 static const struct attribute_group dax_kmem_private_group = {
-- 
2.53.0-Meta

[RFC PATCH 6/8] dax/kmem: add the demotion and numa_balancing opt-ins for private nodes

[Gregory Price]

Expose the two kernel-NUMA capabilities as separate per-device sysfs opt-ins:
'demotion' (NODE_PRIVATE_CAP_DEMOTION) and 'numa_balancing'
(NODE_PRIVATE_CAP_NUMA_BALANCING).  demotion additionally requires reclaim,
enforced at hotplug by node_private_register().

Signed-off-by: Gregory Price <gourry@gourry.net>
---
 drivers/dax/kmem.c | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/drivers/dax/kmem.c b/drivers/dax/kmem.c
index 3892ad44d4e46..1cb3384c003e3 100644
--- a/drivers/dax/kmem.c
+++ b/drivers/dax/kmem.c
@@ -631,12 +631,16 @@ static DEVICE_ATTR_RW(name)
 KMEM_PRIVATE_CAP_ATTR(reclaim, NODE_PRIVATE_CAP_RECLAIM);
 KMEM_PRIVATE_CAP_ATTR(user_numa, NODE_PRIVATE_CAP_USER_NUMA);
 KMEM_PRIVATE_CAP_ATTR(hotunplug, NODE_PRIVATE_CAP_HOTUNPLUG);
+KMEM_PRIVATE_CAP_ATTR(demotion, NODE_PRIVATE_CAP_DEMOTION);
+KMEM_PRIVATE_CAP_ATTR(numa_balancing, NODE_PRIVATE_CAP_NUMA_BALANCING);
 
 /* Per-service opt-ins. Visibility toggled by 'private' control */
 static struct attribute *dax_kmem_private_attrs[] = {
 	&dev_attr_reclaim.attr,
 	&dev_attr_user_numa.attr,
 	&dev_attr_hotunplug.attr,
+	&dev_attr_demotion.attr,
+	&dev_attr_numa_balancing.attr,
 	NULL,
 };
 static const struct attribute_group dax_kmem_private_group = {
-- 
2.53.0-Meta

[RFC PATCH 7/8] dax/kmem: add the ltpin opt-in (CAP_LTPIN) for private nodes

[Gregory Price]

Add "ltpin" opt-in setting for NODE_PRIVATE_CAP_LTPIN so long-term GUP
pins (FOLL_LONGTERM) may target the node's folios.

Long-term pins block migration and hot-unplug indefinitely, so they are
refused on a private node unless it explicitly opts in.

Signed-off-by: Gregory Price <gourry@gourry.net>
---
 drivers/dax/kmem.c | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/drivers/dax/kmem.c b/drivers/dax/kmem.c
index 1cb3384c003e3..c33dac5939edd 100644
--- a/drivers/dax/kmem.c
+++ b/drivers/dax/kmem.c
@@ -633,6 +633,7 @@ KMEM_PRIVATE_CAP_ATTR(user_numa, NODE_PRIVATE_CAP_USER_NUMA);
 KMEM_PRIVATE_CAP_ATTR(hotunplug, NODE_PRIVATE_CAP_HOTUNPLUG);
 KMEM_PRIVATE_CAP_ATTR(demotion, NODE_PRIVATE_CAP_DEMOTION);
 KMEM_PRIVATE_CAP_ATTR(numa_balancing, NODE_PRIVATE_CAP_NUMA_BALANCING);
+KMEM_PRIVATE_CAP_ATTR(ltpin, NODE_PRIVATE_CAP_LTPIN);
 
 /* Per-service opt-ins. Visibility toggled by 'private' control */
 static struct attribute *dax_kmem_private_attrs[] = {
@@ -641,6 +642,7 @@ static struct attribute *dax_kmem_private_attrs[] = {
 	&dev_attr_hotunplug.attr,
 	&dev_attr_demotion.attr,
 	&dev_attr_numa_balancing.attr,
+	&dev_attr_ltpin.attr,
 	NULL,
 };
 static const struct attribute_group dax_kmem_private_group = {
-- 
2.53.0-Meta

[RFC PATCH 8/8] dax/kmem: add dax/adistance to make private node adistance configurable

[Gregory Price]

For testing, it is useful to allow adjustments to node location in the
tiering hierarchy - but this is presently a function of static config
data (SRAT/HMAT).

Private nodes have a contract of one owning service per node, meaning
the registering service can set the adistance for that node without
interfering with other hotplugged memory (no other service can hotplug).

Extend kmem to have dax0.0/adistance when `private=1`, so that the
targeted node's tier membership can be moved around.

Signed-off-by: Gregory Price <gourry@gourry.net>
---
 drivers/dax/kmem.c | 68 ++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 68 insertions(+)

diff --git a/drivers/dax/kmem.c b/drivers/dax/kmem.c
index c33dac5939edd..9fd50ef44d56b 100644
--- a/drivers/dax/kmem.c
+++ b/drivers/dax/kmem.c
@@ -51,6 +51,7 @@ struct dax_kmem_data {
 	int numa_node;
 	int state;
 	struct mutex lock; /* protects hotplug state transitions and config */
+	int adistance; /* node abstract distance */
 	bool dax_file; /* when set, cdev allows mmap */
 	struct mempolicy *policy; /* device-lifetime bind for dax-file mmap */
 	bool private; /* when set, memory is onlined as N_MEMORY_PRIVATE */
@@ -68,6 +69,27 @@ static struct memory_dev_type *kmem_find_alloc_memory_type(int adist)
 	return mt_find_alloc_memory_type(adist, &kmem_memory_types);
 }
 
+/*
+ * Re-register the node's memory type at given abstract distance, refined
+ * by the platform algorithm.  This lets us move nodes to different memory
+ * tiers for testing and if the node does not have static perf data.
+ *
+ * Caller holds data->lock, and device must be in unplugged state.
+ */
+static int kmem_set_adistance(struct dax_kmem_data *data, int adist)
+{
+	struct memory_dev_type *memtype;
+
+	mt_calc_adistance(data->numa_node, &adist);
+	memtype = kmem_find_alloc_memory_type(adist);
+	if (IS_ERR(memtype))
+		return PTR_ERR(memtype);
+	clear_node_memory_type(data->numa_node, NULL);
+	init_node_memory_type(data->numa_node, memtype);
+	data->adistance = adist;
+	return 0;
+}
+
 static void kmem_put_memory_types(void)
 {
 	guard(mutex)(&kmem_memory_type_lock);
@@ -495,6 +517,45 @@ static ssize_t state_store(struct device *dev, struct device_attribute *attr,
 	return len;
 }
 
+static ssize_t adistance_show(struct device *dev, struct device_attribute *attr,
+			      char *buf)
+{
+	struct dax_kmem_data *data = dev_get_drvdata(dev);
+
+	if (!data)
+		return -ENXIO;
+	return sysfs_emit(buf, "%d\n", data->adistance);
+}
+
+static ssize_t adistance_store(struct device *dev, struct device_attribute *attr,
+			       const char *buf, size_t len)
+{
+	struct dax_kmem_data *data = dev_get_drvdata(dev);
+	int adist;
+	ssize_t rc;
+
+	if (!data)
+		return -ENXIO;
+
+	rc = kstrtoint(buf, 0, &adist);
+	if (rc)
+		return rc;
+	if (adist <= 0)
+		return -EINVAL;
+
+	guard(mutex)(&data->lock);
+
+	/* adistance only reconfigures the device while it holds no memory. */
+	if (data->state != DAX_KMEM_UNPLUGGED)
+		return -EBUSY;
+
+	rc = kmem_set_adistance(data, adist);
+	if (rc)
+		return rc;
+	return len;
+}
+static DEVICE_ATTR_RW(adistance);
+
 static const struct attribute_group dax_kmem_private_group;
 
 static ssize_t private_show(struct device *dev, struct device_attribute *attr,
@@ -529,6 +590,11 @@ static ssize_t private_store(struct device *dev, struct device_attribute *attr,
 	if (enable == data->private)
 		return len;
 
+	/* When toggling private, reset custom adistance to default */
+	rc = kmem_set_adistance(data, MEMTIER_DEFAULT_DAX_ADISTANCE);
+	if (rc)
+		return rc;
+
 	if (enable) {
 		/* Add the per-service opt-in attributes. */
 		rc = sysfs_create_group(&dev->kobj, &dax_kmem_private_group);
@@ -637,6 +703,7 @@ KMEM_PRIVATE_CAP_ATTR(ltpin, NODE_PRIVATE_CAP_LTPIN);
 
 /* Per-service opt-ins. Visibility toggled by 'private' control */
 static struct attribute *dax_kmem_private_attrs[] = {
+	&dev_attr_adistance.attr,
 	&dev_attr_reclaim.attr,
 	&dev_attr_user_numa.attr,
 	&dev_attr_hotunplug.attr,
@@ -719,6 +786,7 @@ static int dev_dax_kmem_probe(struct dev_dax *dev_dax)
 	data->mgid = rc;
 	data->numa_node = numa_node;
 	data->state = DAX_KMEM_UNPLUGGED;
+	data->adistance = adist;
 	data->np.owner = data;
 	mutex_init(&data->lock);
 
-- 
2.53.0-Meta

Re: [RFC PATCH 0/8] dax/kmem: support for private NUMA nodes

[Richard Cheng]

On Wed, Jul 22, 2026 at 02:35:11PM +0800, Gregory Price wrote:
> This series adds uapi bits for private memory NUMA node [0] control.
> 
> It is not intended for merge - just discussion.
> 
> I have been using this to extensively test mm/ behaviors with private
> nodes [1] and am wondering whether some of this is more generally
> useful for coherent memory devices if private nodes lands.
> 
> For now, this is useful for testing, but locks in UAPI, so this RFC
> is not intended as an intent to upstream - just sharing my testing api.
>   
>   Question 1: Do we want dax control of private node settings?
>               1a) External? (uapi like this?)
> 	      1b) Internal-only? (plumbed through module apis)
> 
>   Question 2: If we do want this, do we want it in kmem or a
>               separate dax driver? (dax/private_mem ?)
> 
>   Question 3: are dax_file and adistance useful regardless?
> 
> This adds 9 interfaces:
> 
>   dax0.0/dax_file
>      this makes /dev/dax0.0 mmap-able when online as kmem
>      it essentially attaches a default mempolicy to the vma
>      that binds it to the dax device's node.
> 
>   dax0.0/private
>      this switches whether hotplug sets N_MEMORY or N_MEMORY_PRIVATE
> 
>   dax0.0/adistance
>      this allows overriding the default adistance to for the node
>      into a specific tier.  That allows for testing tiering in both
>      directions (demotion from/to the node)
> 
>   dax0.0/reclaim
>   dax0.0/user_numa
>   dax0.0/hotunplug
>   dax0.0/demotion
>   dax0.0/numa_balancing
>   dax0.0/ltpin
>      these bits all enable their respective NODE_PRIVATE_CAP_ bit
> 
>      reclaim   - enables reclaim on the node
>      user_numa - enables mbind/mempolicy/move_pages/migrate_pages
>      hotunplug - enables hotunplug to migrate memory
>      demotion  - enables reclaim to demote from or to the node
>      numa_balancing - allows prot_none injection on the node's folios
>      ltpin     - allows longterm pin on node's folios to work
>

Hi Gregory,

I have some thoughts especially for ownership and configs.

I think it would be good to expose dax control of private node, just with
the following guarantees.

Let's make it clear that userspace should never become the owner
of private node, they should be driver-owned and 1-to-1 mapping relationship.

A userspace tool can request to reconfigure some capability of the DAX device, and
maybe how the driver expose it.

A dax range may have several deployment rules, e.g. devdax, system-ram or private, with
the patch series' tool, it looks like a userspace tool can convert a DAX device between
dax and system-ram mode, I think that's unsafe especially when the region is online as
system-ram.

And it will cause coherence issue to be resolved, if no special or strong use cases, I would
prevent having such ability to convert the rules in runtime, especially for fixed-purpose device.

For capability bitmap part, I think it's reasonable to expose it, but with 2 concepts
- supported_caps : immutable, supplied by the provider/vendor
- enabled-caps : selected policy, subset of supported_caps

The invariant should always hold
"""
enabled_caps & ~supported_caps == 0
"""

This way we can prevent userspace from touching things that they shouldn't.

Maybe a demo with a driver would be nice ?

This is my initial thoughts of this, thanks.

Best regards,
Richard Cheng.

> [0] private node series
> https://lore.kernel.org/linux-mm/20260720193431.3841992-1-gourry@gourry.net/
> 
> [1] selftests/dax extensions for all of this that show use cases and
>     testing for the private node series - including all user api's
>     like mbind, set_mempolicy, move_pages, migrate_privates - as well
>     as reclaim behaviors.
> https://github.com/gourryinverse/linux/commits/scratch/gourry/managed_nodes/dax_private-mm-new/
> 
> Gregory Price (8):
>   dax/kmem: add dax_file= to expose an anonymous-fault char device
>   dax/kmem: add private= to online memory as an N_MEMORY_PRIVATE node
>   dax/kmem: add the reclaim opt-in (CAP_RECLAIM) for private nodes
>   dax/kmem: add the user_numa opt-in (CAP_USER_NUMA) for private nodes
>   dax/kmem: add the hotunplug opt-in (CAP_HOTUNPLUG) for private nodes
>   dax/kmem: add the demotion and numa_balancing opt-ins for private
>     nodes
>   dax/kmem: add the ltpin opt-in (CAP_LTPIN) for private nodes
>   dax/kmem: add dax/adistance to make private node adistance
>     configurable
> 
>  drivers/dax/kmem.c | 368 ++++++++++++++++++++++++++++++++++++++++++++-
>  1 file changed, 366 insertions(+), 2 deletions(-)
> 
> -- 
> 2.53.0-Meta
> 
> 

Re: [RFC PATCH 0/8] dax/kmem: support for private NUMA nodes

[Gregory Price]

On Fri, Jul 24, 2026 at 03:45:23PM +0800, Richard Cheng wrote:
> On Wed, Jul 22, 2026 at 02:35:11PM +0800, Gregory Price wrote:
> 
> Hi Gregory,
> 
> I have some thoughts especially for ownership and configs.
> 
> I think it would be good to expose dax control of private node, just with
> the following guarantees.
> 
> Let's make it clear that userspace should never become the owner
> of private node, they should be driver-owned and 1-to-1 mapping relationship.
> 

The intent here is 100% 1-to-1 driver:node mappings.

Technically there's nothing unreasonable about single-owner multi-node, but
really we just don't want multiple devices on one node.

> A userspace tool can request to reconfigure some capability of the DAX device, and
> maybe how the driver expose it.
> 
> A dax range may have several deployment rules, e.g. devdax, system-ram or private, with
> the patch series' tool, it looks like a userspace tool can convert a DAX device between
> dax and system-ram mode, I think that's unsafe especially when the region is online as
> system-ram.
> 
> And it will cause coherence issue to be resolved, if no special or strong use cases, I would
> prevent having such ability to convert the rules in runtime, especially for fixed-purpose device.
> 

Right now we can't really prevent a user from doing:

echo dax0.0 > dax/drivers/devdax/unbind
echo dax0.0 > dax/drivers/kmem/bind

And in fact we depend on kmem being the default target for CXL to
auto-probe through to hotplug.  For CXL accelerators, the current
intended auto-probe behavior is to terminate at an offline kmem dax
binding, and then expect the user to either rebind the dax device, or
rebind the entire cxl device:

echo $DEVICE > ../drivers/cxl_pci/unbind
echo $DEVICE > ../pci/my_cxl_device_driver/bind

I have some old patches that augment how drivers select the dax mode,
I can see about reviving that and maybe letting the source driver
also dictate which dax modes are valid according to the source.

I'll look into that.

> For capability bitmap part, I think it's reasonable to expose it, but with 2 concepts
> - supported_caps : immutable, supplied by the provider/vendor
> - enabled-caps : selected policy, subset of supported_caps
> 
> The invariant should always hold
> """
> enabled_caps & ~supported_caps == 0
> """
> 
> This way we can prevent userspace from touching things that they shouldn't.
> 

This I like, and i think you've convinced me that this should be a
separate dax device driver so we're not overloading kmem.

That said, for a dax device that was not created by a driver, but
instead created via something like "memmap=" boot parameter, i'm we
can't really dictate supported caps - but for a driver that *does* want
to limit support, I think we can manage that.

> Maybe a demo with a driver would be nice ?
> 

Sure! I think i'm going to include my anon-memory only compressed ram
service back into v6 w/ the dax driver extensions you've suggested.

~Gregory