[PATCH 0/2] Fix and support dax kmem on arm64

[PATCH 0/2] Fix and support dax kmem on arm64

[Jia He]

This patch set is to fix some dax kmem driver issues and then it can 
support to use pmem as normal RAM in arm64 qemu guest.

Jia He (2):
  drivers/dax/kmem: use default numa_mem_id if target_node is invalid
  drivers/dax/kmem: give a warning if CONFIG_DEV_DAX_PMEM_COMPAT is
    enabled

 drivers/dax/kmem.c | 11 ++++++++---
 1 file changed, 8 insertions(+), 3 deletions(-)

-- 
2.17.1

[PATCH 1/2] drivers/dax/kmem: use default numa_mem_id if target_node is invalid

[Jia He]

In some platforms(e.g arm64 guest), the NFIT info might not be ready.
Then target_node might be -1. But if there is a default numa_mem_id(),
we can use it to avoid unnecessary fatal EINVL error.

devm_memremap_pages() also uses this logic if nid is invalid, we can
keep the same page with it.

Signed-off-by: Jia He <justin.he@arm.com>
---
 drivers/dax/kmem.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/drivers/dax/kmem.c b/drivers/dax/kmem.c
index a02318c6d28a..ad62d551d94e 100644
--- a/drivers/dax/kmem.c
+++ b/drivers/dax/kmem.c
@@ -33,9 +33,9 @@ int dev_dax_kmem_probe(struct device *dev)
 	 */
 	numa_node = dev_dax->target_node;
 	if (numa_node < 0) {
-		dev_warn(dev, "rejecting DAX region %pR with invalid node: %d\n",
-			 res, numa_node);
-		return -EINVAL;
+		dev_warn(dev, "DAX %pR with invalid node, assume it as %d\n",
+				res, numa_node, numa_mem_id());
+		numa_node = numa_mem_id();
 	}
 
 	/* Hotplug starting at the beginning of the next block: */
-- 
2.17.1

RE: [PATCH 1/2] drivers/dax/kmem: use default numa_mem_id if target_node is invalid

[Justin He (Arm Technology China)]

Hi
Ping.
The target_node will be -1 if numa disabled. IIUC, it is a generic issue, not only on arm64.


--
Cheers,
Justin (Jia He)



> -----Original Message-----
> From: Jia He <justin.he@arm.com>
> Sent: 2019年8月16日 19:19
> To: Dan Williams <dan.j.williams@intel.com>; Vishal Verma
> <vishal.l.verma@intel.com>
> Cc: Keith Busch <keith.busch@intel.com>; Dave Jiang
> <dave.jiang@intel.com>; linux-nvdimm@lists.01.org; linux-
> kernel@vger.kernel.org; Justin He (Arm Technology China)
> <Justin.He@arm.com>
> Subject: [PATCH 1/2] drivers/dax/kmem: use default numa_mem_id if
> target_node is invalid
>
> In some platforms(e.g arm64 guest), the NFIT info might not be ready.
> Then target_node might be -1. But if there is a default numa_mem_id(),
> we can use it to avoid unnecessary fatal EINVL error.
>
> devm_memremap_pages() also uses this logic if nid is invalid, we can
> keep the same page with it.
>
> Signed-off-by: Jia He <justin.he@arm.com>
> ---
>  drivers/dax/kmem.c | 6 +++---
>  1 file changed, 3 insertions(+), 3 deletions(-)
>
> diff --git a/drivers/dax/kmem.c b/drivers/dax/kmem.c
> index a02318c6d28a..ad62d551d94e 100644
> --- a/drivers/dax/kmem.c
> +++ b/drivers/dax/kmem.c
> @@ -33,9 +33,9 @@ int dev_dax_kmem_probe(struct device *dev)
>        */
>       numa_node = dev_dax->target_node;
>       if (numa_node < 0) {
> -             dev_warn(dev, "rejecting DAX region %pR with invalid
> node: %d\n",
> -                      res, numa_node);
> -             return -EINVAL;
> +             dev_warn(dev, "DAX %pR with invalid node, assume it
> as %d\n",
> +                             res, numa_node, numa_mem_id());
> +             numa_node = numa_mem_id();
>       }
>
>       /* Hotplug starting at the beginning of the next block: */
> --
> 2.17.1

IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.

[PATCH 2/2] drivers/dax/kmem: give a warning if CONFIG_DEV_DAX_PMEM_COMPAT is enabled

[Jia He]

commit c221c0b0308f ("device-dax: "Hotplug" persistent memory for use
like normal RAM") helps to add persistent memory as normal RAM blocks.
But this driver doesn't work if CONFIG_DEV_DAX_PMEM_COMPAT is enabled.

Here is the debugging call trace when CONFIG_DEV_DAX_PMEM_COMPAT is
enabled.
[    4.443730]  devm_memremap_pages+0x4b9/0x540
[    4.443733]  dev_dax_probe+0x112/0x220 [device_dax]
[    4.443735]  dax_pmem_compat_probe+0x58/0x92 [dax_pmem_compat]
[    4.443737]  nvdimm_bus_probe+0x6b/0x150
[    4.443739]  really_probe+0xf5/0x3d0
[    4.443740]  driver_probe_device+0x11b/0x130
[    4.443741]  device_driver_attach+0x58/0x60
[    4.443742]  __driver_attach+0xa3/0x140

Then the dax0.0 device will be registered as "nd" bus instead of
"dax" bus. This causes the error as follows:
root@ubuntu:~# echo dax0.0 > /sys/bus/dax/drivers/device_dax/unbind
-bash: echo: write error: No such device

This gives a warning to notify the user.

Signed-off-by: Jia He <justin.he@arm.com>
---
 drivers/dax/kmem.c | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/drivers/dax/kmem.c b/drivers/dax/kmem.c
index ad62d551d94e..b77f0e880598 100644
--- a/drivers/dax/kmem.c
+++ b/drivers/dax/kmem.c
@@ -93,6 +93,11 @@ static struct dax_device_driver device_dax_kmem_driver = {
 
 static int __init dax_kmem_init(void)
 {
+	if (IS_ENABLED(CONFIG_DEV_DAX_PMEM_COMPAT)) {
+		pr_warn("CONFIG_DEV_DAX_PMEM_COMPAT is not compatible\n");
+		pr_warn("kmem dax driver might not be workable\n");
+	}
+
 	return dax_driver_register(&device_dax_kmem_driver);
 }
 
-- 
2.17.1

Re: [PATCH 2/2] drivers/dax/kmem: give a warning if CONFIG_DEV_DAX_PMEM_COMPAT is enabled

[Jeff Moyer]

Jia He <justin.he@arm.com> writes:

> commit c221c0b0308f ("device-dax: "Hotplug" persistent memory for use
> like normal RAM") helps to add persistent memory as normal RAM blocks.
> But this driver doesn't work if CONFIG_DEV_DAX_PMEM_COMPAT is enabled.
>
> Here is the debugging call trace when CONFIG_DEV_DAX_PMEM_COMPAT is
> enabled.
> [    4.443730]  devm_memremap_pages+0x4b9/0x540
> [    4.443733]  dev_dax_probe+0x112/0x220 [device_dax]
> [    4.443735]  dax_pmem_compat_probe+0x58/0x92 [dax_pmem_compat]
> [    4.443737]  nvdimm_bus_probe+0x6b/0x150
> [    4.443739]  really_probe+0xf5/0x3d0
> [    4.443740]  driver_probe_device+0x11b/0x130
> [    4.443741]  device_driver_attach+0x58/0x60
> [    4.443742]  __driver_attach+0xa3/0x140
>
> Then the dax0.0 device will be registered as "nd" bus instead of
> "dax" bus. This causes the error as follows:
> root@ubuntu:~# echo dax0.0 > /sys/bus/dax/drivers/device_dax/unbind
> -bash: echo: write error: No such device
>
> This gives a warning to notify the user.
>
> Signed-off-by: Jia He <justin.he@arm.com>
> ---
>  drivers/dax/kmem.c | 5 +++++
>  1 file changed, 5 insertions(+)
>
> diff --git a/drivers/dax/kmem.c b/drivers/dax/kmem.c
> index ad62d551d94e..b77f0e880598 100644
> --- a/drivers/dax/kmem.c
> +++ b/drivers/dax/kmem.c
> @@ -93,6 +93,11 @@ static struct dax_device_driver device_dax_kmem_driver = {
>  
>  static int __init dax_kmem_init(void)
>  {
> +	if (IS_ENABLED(CONFIG_DEV_DAX_PMEM_COMPAT)) {
> +		pr_warn("CONFIG_DEV_DAX_PMEM_COMPAT is not compatible\n");
> +		pr_warn("kmem dax driver might not be workable\n");
> +	}
> +
>  	return dax_driver_register(&device_dax_kmem_driver);
>  }

This logic is wrong (and the error message is *very* confusing).  You
can have the driver configured, but not loaded.  In that case, the kmem
driver will load and work properly.

When using daxctl to online memory, you already get the following
message:

libdaxctl: daxctl_dev_disable: dax0.0: error: device model is dax-class

That's still not very helpful.  It would be better if the message
suggested a fix (like using migrate-device-model).  Vishal?

Cheers,
Jeff

Re: [PATCH 2/2] drivers/dax/kmem: give a warning if CONFIG_DEV_DAX_PMEM_COMPAT is enabled

[Verma, Vishal L]

On Thu, 2019-08-22 at 14:55 -0400, Jeff Moyer wrote:
> 
> When using daxctl to online memory, you already get the following
> message:
> 
> libdaxctl: daxctl_dev_disable: dax0.0: error: device model is dax-class
> 
> That's still not very helpful.  It would be better if the message
> suggested a fix (like using migrate-device-model).  Vishal?

Yes, it is on my list to improve this. Currently the man page shows the
above error message, and talks about migrate-device-model, but I
received more feedback to add another bread crumb in the printed message
pointing to migrate-device-model.  I'll send a patch for it soon.

Thanks,
	-Vishal