[PATCH RFC v2 00/14] nvme: Controller Data Queue (CDQ) support

[PATCH RFC v2 00/14] nvme: Controller Data Queue (CDQ) support

[Joel Granados]

This RFC implements Controller Data Queue (CDQ) support in the NVMe
driver, a variation on my two ([2] & [5]) previously shared approaches.
A CDQ is how an NVMe controller reports modified ranges; user space
consumes that stream to aid in data migration.

Motivation
==========
This is about enabling NVMe namespace migration. The timing feels right
as hardware with CDQ capability exists, NVMe fully specifies the feature
and there is growing interest in Live Migration which by extension
includes CDQ.

What
====
Implement CDQ (the backing memory, the controller-side create/delete,
the queue traversal and the asynchronous-event handling) inside the nvme
driver. User space drives everything through a single ioctl. The CDQ is
read through an FD handed back by the nvme driver, which copies new
entries to a user space buffer. A tail pointer trigger can be armed
automatically when there are no new entries. The one shot AEN event
is routed to an eventfd whenever new entries become available.

NVMe LM SW WG
=============
The NVMe Live Migration SoftWare Working Group (mentioned in [6]) was
one of the main feedback points in my previous RFC. My main goal in this
respect is to keep exploring different approaches in the hope that when
it materializes we can "hit the ground running".

vfio-nvme ([1], [4])
====================
This is what I rebased onto [4] and relates to the handling of
controller state in live migration. I was curious to see how it would
interact (if at all) with my CDQ approach; IMO, for now, they are quite
orthogonal. It is **neither** a rejection nor an endorsement of the vfio
approach. From my point of view there is still no consensus on how this
will make its way into mainline.

Feedback
========
1. The CDQ is introduced into core.c. Would putting it in pci.c make
   more sense?
2. I create a new type of submission-less queue for the CDQ. There is no
   way of using the NVMe queue pairs for a submission-less queue, right?
3. Any feedback that can be shared publicly is greatly appreciated.

As always, please tell me if you want to be removed from these moving
forward or if I have missed someone.

Best

Changes in v2:
- CDQ logic (traversal, memory, AEN, ...) is all in the nvme driver.
- CDQ entries are now copied from the CDQ mem to the user space buffer
- Reduced IOCTL command size
- Rebased on top of [4] (wich is [3] with some fixes from my part)
- Don't wait for the completion of the set feature cmd. We handle it
  separately from sending the command to avoid having to wait for the
  round trip to/from the controller.
- Link to v1: [5]

[1] https://lore.kernel.org/20221206055816.292304-1-lei.rao@intel.com
[2] https://lore.kernel.org/20250714-jag-cdq-v1-0-01e027d256d5@kernel.org
[3] https://lore.kernel.org/20250803024705.10256-1-kch@nvidia.com
[4] https://git.kernel.org/pub/scm/linux/kernel/git/joel.granados/linux.git/log/?h=jag/vfio-nvme
[5] https://lore.kernel.org/r/20260424-jag-cdq-lkml-v1-0-d773343a717c@kernel.org
[6] https://lore.kernel.org/de162155-b03a-4626-a416-3425db9235c7@nvidia.com

Signed-off-by: Joel Granados <joel.granados@kernel.org>
---
---
Joel Granados (14):
      nvme: Add NVME_AER_ONE_SHOT callback handler
      nvme: Add NVMe Controller Data Queue command
      nvme: Add track send command
      nvme: Add CDQ to xarray and define delete path
      nvme: Allocate CDQ backing memory from coherent DMA chunks
      nvme: Add file descriptor to read CDQs
      nvme: Add the create CDQ functionality
      nvme: Submit track send command to start/stop the CDQ
      nvme: Coordinate CDQ backing mem and CDQ FD
      nvme: Pin a ctrl ref to every CDQ
      nvme: Implement CDQ traversal on the read path
      nvme: Update the controller with the CDQ set-feature cmd
      nvme: Use eventfd for CDQ Tail pointer triggers
      nvme: Add NVME_IOCTL_CDQ to create/delete migration CDQs

 drivers/nvme/host/Makefile      |   2 +-
 drivers/nvme/host/cdq.c         | 667 ++++++++++++++++++++++++++++++++++++++++
 drivers/nvme/host/cdq.h         | 110 +++++++
 drivers/nvme/host/core.c        |  46 ++-
 drivers/nvme/host/ioctl.c       |  44 ++-
 drivers/nvme/host/nvme.h        |   1 +
 include/linux/nvme.h            |  78 ++++-
 include/uapi/linux/nvme_ioctl.h |  22 ++
 8 files changed, 966 insertions(+), 4 deletions(-)
---
base-commit: 087c08850a40fb37753475f90b0cb706effd752c
change-id: 20260424-jag-cdq-lkml-cd9b7c79983d

Best regards,
-- 
Joel Granados <joel.granados@kernel.org>

[PATCH RFC v2 01/14] nvme: Add NVME_AER_ONE_SHOT callback handler

[Joel Granados]

Handle NVME_AER_ONE_SHOT asynchronous event notifications by extracting
the event parameter from the 64-bit result and passing it to the
handler.

Stub nvme_handle_aen_oneshot() with warnings for now; it will be
extended by subsequent patches to handle specific one-shot event types
like CDQ tail pointer events.

Signed-off-by: Joel Granados <joel.granados@kernel.org>
---
 drivers/nvme/host/core.c | 29 +++++++++++++++++++++++++++++
 include/linux/nvme.h     |  7 +++++++
 2 files changed, 36 insertions(+)

diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c
index fb13f85dd7844331566f05915dda21d91abfea30..6e7533ccf24bfe3df7188f442986ec18a3200660 100644
--- a/drivers/nvme/host/core.c
+++ b/drivers/nvme/host/core.c
@@ -4748,6 +4748,30 @@ static u32 nvme_aer_subtype(u32 result)
 	return (result & 0xff00) >> 8;
 }
 
+static bool nvme_handle_aen_oneshot(struct nvme_ctrl *ctrl, u32 result, u32 event_param)
+{
+	u32 aer_subtype = nvme_aer_subtype(result);
+
+	switch (aer_subtype) {
+	case NVME_AER_ONE_SHOT_CDQ_TAIL_PTR:
+		WARN_ONCE(1, "CDQ Tail Pointer one shot event ignored");
+		break;
+	case NVME_AER_ONE_SHOT_CDQ_FULL:
+		WARN_ONCE(1, "CDQ Full Error one shot event ignored");
+		break;
+	case NVME_AER_ONE_SHOT_PWR_TH:
+		WARN_ONCE(1, "Power Threshold Exceeded one shot event ignored");
+		break;
+	default:
+		WARN_ONCE(1, "Unrecognized One Shot Async Event %d (ignored)",
+			  aer_subtype);
+		break;
+	}
+
+	/* Return true in all cases to reque after handling */
+	return true;
+}
+
 static bool nvme_handle_aen_notice(struct nvme_ctrl *ctrl, u32 result)
 {
 	u32 aer_notice_type = nvme_aer_subtype(result);
@@ -4796,6 +4820,7 @@ void nvme_complete_async_event(struct nvme_ctrl *ctrl, __le16 status,
 		volatile union nvme_result *res)
 {
 	u32 result = le32_to_cpu(res->u32);
+	u32 event_param = 0;
 	u32 aer_type = nvme_aer_type(result);
 	u32 aer_subtype = nvme_aer_subtype(result);
 	bool requeue = true;
@@ -4808,6 +4833,10 @@ void nvme_complete_async_event(struct nvme_ctrl *ctrl, __le16 status,
 	case NVME_AER_NOTICE:
 		requeue = nvme_handle_aen_notice(ctrl, result);
 		break;
+	case NVME_AER_ONE_SHOT:
+		event_param = le64_to_cpu(res->u64) >> 32;
+		requeue = nvme_handle_aen_oneshot(ctrl, result, event_param);
+		break;
 	case NVME_AER_ERROR:
 		/*
 		 * For a persistent internal error, don't run async_event_work
diff --git a/include/linux/nvme.h b/include/linux/nvme.h
index 69a8c48faa6ce19e695286d15be2585a4d4401d4..60fb5b6808615ff7388f30e12ffe6ed5da2ef4f6 100644
--- a/include/linux/nvme.h
+++ b/include/linux/nvme.h
@@ -862,6 +862,7 @@ enum {
 	NVME_AER_ERROR			= 0,
 	NVME_AER_SMART			= 1,
 	NVME_AER_NOTICE			= 2,
+	NVME_AER_ONE_SHOT		= 4,
 	NVME_AER_CSS			= 6,
 	NVME_AER_VS			= 7,
 };
@@ -877,6 +878,12 @@ enum {
 	NVME_AER_NOTICE_DISC_CHANGED	= 0xf0,
 };
 
+enum {
+	NVME_AER_ONE_SHOT_CDQ_TAIL_PTR	= 0x00,
+	NVME_AER_ONE_SHOT_CDQ_FULL	= 0x01,
+	NVME_AER_ONE_SHOT_PWR_TH	= 0x02,
+};
+
 enum {
 	NVME_AEN_BIT_NS_ATTR		= 8,
 	NVME_AEN_BIT_FW_ACT		= 9,

-- 
2.50.1

[PATCH RFC v2 02/14] nvme: Add NVMe Controller Data Queue command

[Joel Granados]

Add the struct (nvme_cdq_cmd) and define the relative values to support
NVMe Controller Data Queue management.
* struct nvme_cdq_cmd: Is the CDQ mgmt command
* nvme_cdq_cmd_mgmt_op: Specifies create or delete
* NVME_CDQ_CMD_MGMT_CREATE_*: Specifies the create operation.

No functional changes are included as this is a preparation commit.

Signed-off-by: Joel Granados <joel.granados@kernel.org>
---
 drivers/nvme/host/core.c |  1 +
 include/linux/nvme.h     | 45 ++++++++++++++++++++++++++++++++++++++++++++-
 2 files changed, 45 insertions(+), 1 deletion(-)

diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c
index 6e7533ccf24bfe3df7188f442986ec18a3200660..d11858300ad43f3e229eab7f2dd77563075b801e 100644
--- a/drivers/nvme/host/core.c
+++ b/drivers/nvme/host/core.c
@@ -5387,6 +5387,7 @@ static inline void _nvme_check_size(void)
 	BUILD_BUG_ON(sizeof(struct nvme_directive_cmd) != 64);
 	BUILD_BUG_ON(sizeof(struct nvme_lm_send_cmd) != 64);
 	BUILD_BUG_ON(sizeof(struct nvme_lm_recv_cmd) != 64);
+	BUILD_BUG_ON(sizeof(struct nvme_cdq_command) != 64);
 	BUILD_BUG_ON(sizeof(struct nvme_feat_host_behavior) != 512);
 }
 
diff --git a/include/linux/nvme.h b/include/linux/nvme.h
index 60fb5b6808615ff7388f30e12ffe6ed5da2ef4f6..1c20ddc06a46066b5ab9acf1dc44b03d941592a9 100644
--- a/include/linux/nvme.h
+++ b/include/linux/nvme.h
@@ -1639,6 +1639,7 @@ enum nvme_admin_opcode {
 	nvme_admin_nvme_mi_recv		= 0x1e,
 	nvme_admin_lm_send		= 0x41,
 	nvme_admin_lm_recv		= 0x42,
+	nvme_admin_cdq			= 0x45,
 	nvme_admin_dbbuf		= 0x7C,
 	nvme_admin_format_nvm		= 0x80,
 	nvme_admin_security_send	= 0x81,
@@ -1679,7 +1680,8 @@ enum nvme_admin_opcode {
 		nvme_admin_opcode_name(nvme_admin_security_send),	\
 		nvme_admin_opcode_name(nvme_admin_security_recv),	\
 		nvme_admin_opcode_name(nvme_admin_sanitize_nvm),	\
-		nvme_admin_opcode_name(nvme_admin_get_lba_status))
+		nvme_admin_opcode_name(nvme_admin_get_lba_status),	\
+		nvme_admin_opcode_name(nvme_admin_cdq))
 
 enum {
 	NVME_QUEUE_PHYS_CONTIG	= (1 << 0),
@@ -1710,6 +1712,7 @@ enum {
 	NVME_FEAT_PLM_WINDOW	= 0x14,
 	NVME_FEAT_HOST_BEHAVIOR	= 0x16,
 	NVME_FEAT_SANITIZE	= 0x17,
+	NVME_FEAT_CDQ		= 0x21,
 	NVME_FEAT_FDP		= 0x1d,
 	NVME_FEAT_SW_PROGRESS	= 0x80,
 	NVME_FEAT_HOST_ID	= 0x81,
@@ -1739,6 +1742,10 @@ enum {
 	NVME_FWACT_ACTV		= (2 << 3),
 };
 
+enum {
+	NVME_FEAT_CDQ_ID_MASK = GENMASK(15, 0),
+};
+
 struct nvme_supported_log {
 	__le32	lids[256];
 };
@@ -1917,6 +1924,41 @@ struct nvme_directive_cmd {
 	__u32			rsvd16[3];
 };
 
+//Select values for dword 10
+enum nvme_cdq_cmd_mgmt_op {
+	NVME_CDQ_CMD_MGMT_CREATE = 0x0,
+	NVME_CDQ_CMD_MGMT_DELETE = 0x1
+};
+#define NVME_CDQ_CMD_MGMT_CREATE_MOS_QT_UDMQ	0x0
+#define NVME_CDQ_CMD_MGMT_CREATE_PC_CONT	0x1
+#define NVME_CDQ_CMD_MGMT_CREATE_PC_DISCONT	0x0
+
+union nvme_cdq_cmd_dw11 {
+	struct {
+		__le16	flags;
+		__le16	cqs;
+	};
+	struct {
+		__le16	cdqid;
+		__le16	rsvd;
+	};
+};
+
+struct nvme_cdq_command {
+	__u8			opcode;
+	__u8			flags;
+	__u16			command_id;
+	__u32			rsvd1[5];
+	__le64			prp1;
+	__le32			rsvd8[2];
+	__u8			sel;
+	__u8			rsvd10;
+	__le16			mos;
+	union nvme_cdq_cmd_dw11	dw11;
+	__le32			cdqsize;
+	__u32			rsvd13[3];
+};
+
 /*
  * Fabrics subcommands.
  */
@@ -2335,6 +2377,7 @@ struct nvme_command {
 		struct nvme_directive_cmd directive;
 		struct nvme_lm_command lm;
 		struct nvme_io_mgmt_recv_cmd imr;
+		struct nvme_cdq_command cdq;
 	};
 };
 

-- 
2.50.1

[PATCH RFC v2 03/14] nvme: Add track send command

[Joel Granados]

The Controller Data Queue is started/stopped using this NVMe command.

Signed-off-by: Joel Granados <joel.granados@kernel.org>
---
 include/linux/nvme.h | 27 ++++++++++++++++++++++++++-
 1 file changed, 26 insertions(+), 1 deletion(-)

diff --git a/include/linux/nvme.h b/include/linux/nvme.h
index 1c20ddc06a46066b5ab9acf1dc44b03d941592a9..7cabedce022eda98fff789596682f257203e3b45 100644
--- a/include/linux/nvme.h
+++ b/include/linux/nvme.h
@@ -1637,6 +1637,7 @@ enum nvme_admin_opcode {
 	nvme_admin_virtual_mgmt		= 0x1c,
 	nvme_admin_nvme_mi_send		= 0x1d,
 	nvme_admin_nvme_mi_recv		= 0x1e,
+	nvme_admin_track_send		= 0x3d,
 	nvme_admin_lm_send		= 0x41,
 	nvme_admin_lm_recv		= 0x42,
 	nvme_admin_cdq			= 0x45,
@@ -1681,7 +1682,8 @@ enum nvme_admin_opcode {
 		nvme_admin_opcode_name(nvme_admin_security_recv),	\
 		nvme_admin_opcode_name(nvme_admin_sanitize_nvm),	\
 		nvme_admin_opcode_name(nvme_admin_get_lba_status),	\
-		nvme_admin_opcode_name(nvme_admin_cdq))
+		nvme_admin_opcode_name(nvme_admin_cdq),			\
+		nvme_admin_opcode_name(nvme_admin_track_send))
 
 enum {
 	NVME_QUEUE_PHYS_CONTIG	= (1 << 0),
@@ -1959,6 +1961,28 @@ struct nvme_cdq_command {
 	__u32			rsvd13[3];
 };
 
+// These are the values for select in dword10
+enum nvme_track_send_cmd_mgmt_op {
+	NVME_TRSND_CMD_MGMT_LOG_USR_DATA = 0x0,
+	NVME_TRSND_CMD_MGMT_TRACK_MEM = 0x1
+};
+
+// These are the values for Logging action LACT
+#define NVME_TRSND_CMD_MGMT_LOG_USR_DATA_START	0x1
+#define NVME_TRSND_CMD_MGMT_LOG_USR_DATA_STOP	0x0
+
+struct track_send_cmd {
+	__u8			opcode;
+	__u8			flags;
+	__u16			command_id;
+	__u32			rsvd1[9];
+	__u8			sel;
+	__u8			rsvd10;
+	__le16			mos;
+	__le32			dw11;
+	__u32			rsvd12[4];
+};
+
 /*
  * Fabrics subcommands.
  */
@@ -2378,6 +2402,7 @@ struct nvme_command {
 		struct nvme_lm_command lm;
 		struct nvme_io_mgmt_recv_cmd imr;
 		struct nvme_cdq_command cdq;
+		struct track_send_cmd track_send;
 	};
 };
 

-- 
2.50.1

[PATCH RFC v2 04/14] nvme: Add CDQ to xarray and define delete path

[Joel Granados]

Add Controller managed CDQs to a new xarray in nvme_ctrl indexed by the
CDQ id returned by the controller. The new cdqs xarray will house
pointers to the struct cdq_nvme_queue which contains individual CDQ
metadata. Start it off with id and ctrl back pointer; it will grow as
features get added.

Commands on the delete path:
1. Individual -vs- "All" CDQ delete:
  - The individual delete path is taken when there is a specific command
    from user space to "delete" one CDQ
  - The "all" delete is taken on contrller reset
2. Separation between host and controller delete:
  - Controller delete takes care of sending and forwarding the result of
    the nvme CDQ delete command
  - Host delete memory on the host side

The separation between host and controller delete is important in
nvme_disable_ctrl as the host should delete the memory but expect the
controller to follow NVMe protocol and remove the CDQ on its side.

Note that there is no way of adding a CDQ just yet. This commit focuses
on the delete path and leaves room for handling the creation in
subsequent commits.

Signed-off-by: Joel Granados <joel.granados@kernel.org>
---
 drivers/nvme/host/Makefile |  2 +-
 drivers/nvme/host/cdq.c    | 62 ++++++++++++++++++++++++++++++++++++++++++++++
 drivers/nvme/host/cdq.h    | 20 +++++++++++++++
 drivers/nvme/host/core.c   | 15 ++++++++++-
 drivers/nvme/host/nvme.h   |  1 +
 5 files changed, 98 insertions(+), 2 deletions(-)

diff --git a/drivers/nvme/host/Makefile b/drivers/nvme/host/Makefile
index 6414ec968f99ae69f991cbd61e7ad79554e5041f..cd895082e259a9b0673c7400f6ba62401f95767f 100644
--- a/drivers/nvme/host/Makefile
+++ b/drivers/nvme/host/Makefile
@@ -10,7 +10,7 @@ obj-$(CONFIG_NVME_FC)			+= nvme-fc.o
 obj-$(CONFIG_NVME_TCP)			+= nvme-tcp.o
 obj-$(CONFIG_NVME_APPLE)		+= nvme-apple.o
 
-nvme-core-y				+= core.o ioctl.o sysfs.o pr.o
+nvme-core-y				+= core.o ioctl.o sysfs.o pr.o cdq.o
 nvme-core-$(CONFIG_NVME_VERBOSE_ERRORS)	+= constants.o
 nvme-core-$(CONFIG_TRACING)		+= trace.o
 nvme-core-$(CONFIG_NVME_MULTIPATH)	+= multipath.o
diff --git a/drivers/nvme/host/cdq.c b/drivers/nvme/host/cdq.c
new file mode 100644
index 0000000000000000000000000000000000000000..0e9ac88a054f6630c1eea73ae4140f124b6bb31f
--- /dev/null
+++ b/drivers/nvme/host/cdq.c
@@ -0,0 +1,62 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * NVMe Controller Data Queue (CDQ) support.
+ */
+
+#include "nvme.h"
+#include "cdq.h"
+
+static int nvme_submit_delete_cdq_cmd(const struct cdq_nvme_queue *cdq)
+{
+	struct nvme_command c = {
+		.cdq.opcode = nvme_admin_cdq,
+		.cdq.sel = NVME_CDQ_CMD_MGMT_DELETE,
+		.cdq.dw11.cdqid = cpu_to_le16(cdq->id)
+	};
+
+	return __nvme_submit_sync_cmd(cdq->ctrl->admin_q, &c, NULL, NULL, 0, NVME_QID_ANY, 0);
+}
+
+/* Sends a CDQ delete NVMe cmd */
+static void nvme_delete_cdq_ctrl(struct cdq_nvme_queue *cdq)
+{
+	if (nvme_submit_delete_cdq_cmd(cdq))
+		WARN_ONCE(1, "Failed delete CDQ (id: %d)", cdq->id);
+}
+
+/* Does NOT send a CDQ delete NVMe cmd */
+static void nvme_delete_cdq_host(struct cdq_nvme_queue *cdq)
+{
+	u16 cdq_id = cdq->id;
+	struct nvme_ctrl *ctrl = cdq->ctrl;
+
+	xa_erase(&ctrl->cdqs, cdq_id);
+}
+
+void nvme_delete_cdq(struct cdq_nvme_queue *cdq)
+{
+	nvme_delete_cdq_ctrl(cdq);
+	nvme_delete_cdq_host(cdq);
+}
+EXPORT_SYMBOL_GPL(nvme_delete_cdq);
+
+/* Will NOT send a CDQ delete NVMe cmd. */
+void nvme_delete_cdqs_host(struct nvme_ctrl *ctrl)
+{
+	struct cdq_nvme_queue *cdq;
+	unsigned long i;
+
+	xa_for_each(&ctrl->cdqs, i, cdq)
+		nvme_delete_cdq_host(cdq);
+}
+
+/* Final teardown at device->release: free all CDQs and destroy the xarray. */
+void nvme_free_cdqs(struct nvme_ctrl *ctrl)
+{
+	/*
+	 * Delete host side CDQ only. NOT sending delete cmd as
+	 * Ctrl should delete on disable.
+	 */
+	nvme_delete_cdqs_host(ctrl);
+	xa_destroy(&ctrl->cdqs);
+}
diff --git a/drivers/nvme/host/cdq.h b/drivers/nvme/host/cdq.h
new file mode 100644
index 0000000000000000000000000000000000000000..4378f97553ce2e649d8998eb168f82039dc762b1
--- /dev/null
+++ b/drivers/nvme/host/cdq.h
@@ -0,0 +1,20 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * NVMe Controller Data Queue (CDQ) support.
+ */
+
+#ifndef _NVME_CDQ_H
+#define _NVME_CDQ_H
+
+#include "nvme.h"
+
+struct cdq_nvme_queue {
+	u16 id;
+	struct nvme_ctrl *ctrl;
+};
+
+void nvme_delete_cdq(struct cdq_nvme_queue *cdq);
+void nvme_delete_cdqs_host(struct nvme_ctrl *ctrl);
+void nvme_free_cdqs(struct nvme_ctrl *ctrl);
+
+#endif /* _NVME_CDQ_H */
diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c
index d11858300ad43f3e229eab7f2dd77563075b801e..fbaa2e065ab846e4700cddd7ae9f30a52e8a2cd9 100644
--- a/drivers/nvme/host/core.c
+++ b/drivers/nvme/host/core.c
@@ -25,6 +25,7 @@
 #include <linux/unaligned.h>
 
 #include "nvme.h"
+#include "cdq.h"
 #include "fabrics.h"
 #include <linux/nvme-auth.h>
 
@@ -2665,9 +2666,19 @@ int nvme_disable_ctrl(struct nvme_ctrl *ctrl, bool shutdown)
 	}
 	if (ctrl->quirks & NVME_QUIRK_DELAY_BEFORE_CHK_RDY)
 		msleep(NVME_QUIRK_DELAY_AMOUNT);
-	return nvme_wait_ready(ctrl, NVME_CSTS_RDY, 0,
+	ret =  nvme_wait_ready(ctrl, NVME_CSTS_RDY, 0,
 			       (NVME_CAP_TIMEOUT(ctrl->cap) + 1) / 2, "reset");
+	if (ret)
+		return ret;
+
+	/*
+	 * Delete host side CDQ only. Purposefully NOT sending delete cmd.
+	 * Ctrl should delete on disable.
+	 */
+	nvme_delete_cdqs_host(ctrl);
+	return ret;
 }
+
 EXPORT_SYMBOL_GPL(nvme_disable_ctrl);
 
 int nvme_enable_ctrl(struct nvme_ctrl *ctrl)
@@ -5074,6 +5085,7 @@ static void nvme_free_ctrl(struct device *dev)
 	if (!subsys || ctrl->instance != subsys->instance)
 		ida_free(&nvme_instance_ida, ctrl->instance);
 	nvme_free_cels(ctrl);
+	nvme_free_cdqs(ctrl);
 	nvme_mpath_uninit(ctrl);
 	cleanup_srcu_struct(&ctrl->srcu);
 	nvme_auth_stop(ctrl);
@@ -5120,6 +5132,7 @@ int nvme_init_ctrl(struct nvme_ctrl *ctrl, struct device *dev,
 	mutex_init(&ctrl->scan_lock);
 	INIT_LIST_HEAD(&ctrl->namespaces);
 	xa_init(&ctrl->cels);
+	xa_init(&ctrl->cdqs);
 	ctrl->dev = dev;
 	ctrl->ops = ops;
 	ctrl->quirks = quirks;
diff --git a/drivers/nvme/host/nvme.h b/drivers/nvme/host/nvme.h
index 654e5c1b64956c877ff845f0b1f7ee2fa0852b64..7b9d1781816c4bad719584c3d4b4f9fdf7b4e3dc 100644
--- a/drivers/nvme/host/nvme.h
+++ b/drivers/nvme/host/nvme.h
@@ -466,6 +466,7 @@ struct nvme_ctrl {
 	enum nvme_dctype dctype;
 
 	u16			awupf; /* 0's based value. */
+	struct xarray cdqs;
 };
 
 static inline enum nvme_ctrl_state nvme_ctrl_state(struct nvme_ctrl *ctrl)

-- 
2.50.1

[PATCH RFC v2 05/14] nvme: Allocate CDQ backing memory from coherent DMA chunks

[Joel Granados]

Back a Controller Data Queue with a set of dma_alloc_coherent() chunks.
This mem is portable to non-coherent archs and stays put for the life of
the CDQ. Splitting it into fixed-size chunks avoids the MAX_PAGE_ORDER
limit on the no-IOMMU path. A single chunk is handed over as PC_CONT;
multiple chunks are stitched together with a chained PRP list
(PC_DISCONT).

Since it is easier to handle 1 chunk (no prp list), try
dma_alloc_coherent with the totality of the CDQ size and fallback on the
chunk allocation only when it was not possible to have a contiguous
allocation. Build the prp list only when there is more than one chunk of
backing memory.

Signed-off-by: Joel Granados <joel.granados@kernel.org>
---
 drivers/nvme/host/cdq.c | 166 +++++++++++++++++++++++++++++++++++++++++++++++-
 drivers/nvme/host/cdq.h |  27 ++++++++
 2 files changed, 191 insertions(+), 2 deletions(-)

diff --git a/drivers/nvme/host/cdq.c b/drivers/nvme/host/cdq.c
index 0e9ac88a054f6630c1eea73ae4140f124b6bb31f..73dd6af42055687bec62ad0350c67736c9c047c2 100644
--- a/drivers/nvme/host/cdq.c
+++ b/drivers/nvme/host/cdq.c
@@ -6,6 +6,166 @@
 #include "nvme.h"
 #include "cdq.h"
 
+static inline void nvme_free_cdqmem_chunks(struct cdq_nvme_queue *cdq)
+{
+	struct device *dev = cdq->ctrl->dev;
+	unsigned int i;
+
+	if (!cdq->chunks)
+		return;
+	for (i = 0; i < cdq->nr_chunks; i++) {
+		if (cdq->chunks[i].vaddr)
+			dma_free_coherent(dev, cdq->chunks[i].size,
+					  cdq->chunks[i].vaddr,
+					  cdq->chunks[i].dma_addr);
+	}
+	kfree(cdq->chunks);
+	cdq->chunks = NULL;
+	cdq->nr_chunks = 0;
+}
+
+static inline int nvme_alloc_cdqmem_chunks(struct cdq_nvme_queue *cdq)
+{
+	struct device *dev = cdq->ctrl->dev;
+	unsigned int i, nr = 1;
+
+	/*
+	 * Try to get a single pointer to the whole cdq, it will take
+	 * the iommu path within the dma_alloc_coherent call
+	 */
+	cdq->chunks = kcalloc(nr, sizeof(*cdq->chunks), GFP_KERNEL);
+	if (!cdq->chunks)
+		return -ENOMEM;
+
+	cdq->chunks[0].vaddr = dma_alloc_coherent(dev, cdq->size_nbyte,
+			&cdq->chunks[0].dma_addr, GFP_KERNEL);
+	if (cdq->chunks[0].vaddr)
+		goto out;
+
+	/* Fall back to allocating several chunks */
+	nr = DIV_ROUND_UP(cdq->size_nbyte, NVME_CDQ_CHUNK_SIZE);
+
+	cdq->chunks = krealloc_array(cdq->chunks, nr, sizeof(*cdq->chunks), GFP_KERNEL);
+	if (!cdq->chunks)
+		return -ENOMEM;
+
+	for (i = 0; i < nr; i++) {
+		cdq->chunks[i].vaddr = dma_alloc_coherent(dev,
+				NVME_CDQ_CHUNK_SIZE, &cdq->chunks[i].dma_addr,
+				GFP_KERNEL);
+		if (!cdq->chunks[i].vaddr) {
+			nvme_free_cdqmem_chunks(cdq);
+			return -ENOMEM;
+		}
+		cdq->chunks[i].size = NVME_CDQ_CHUNK_SIZE;
+	}
+
+out:
+	cdq->nr_chunks = nr;
+	return 0;
+}
+
+static inline void nvme_free_cdqmem_prp_lists(struct cdq_nvme_queue *cdq)
+{
+	struct device *dev = cdq->ctrl->dev;
+	unsigned int prp_idx;
+
+	for (prp_idx = 0; prp_idx < cdq->nr_prp_lists; prp_idx++)
+		dma_free_coherent(dev, PAGE_SIZE, cdq->prp_lists[prp_idx],
+				  cdq->prp_lists_dma[prp_idx]);
+	cdq->nr_prp_lists = 0;
+}
+
+static inline dma_addr_t nvme_get_cdq_pagedma(struct cdq_nvme_queue *cdq,
+					      unsigned int page_idx)
+{
+	return cdq->chunks[page_idx / NVME_CDQ_PAGES_PER_CHUNK].dma_addr +
+		(page_idx % NVME_CDQ_PAGES_PER_CHUNK) * NVME_CTRL_PAGE_SIZE;
+}
+
+static inline int nvme_build_cdqmem_prp_list(struct cdq_nvme_queue *cdq)
+{
+	struct device *dev = cdq->ctrl->dev;
+	const unsigned int prps_per_page = PAGE_SIZE >> 3;
+	unsigned int total_pages =
+		DIV_ROUND_UP(cdq->size_nbyte, NVME_CTRL_PAGE_SIZE);
+	dma_addr_t prp_list_dma;
+	__le64 *prp_list;
+	unsigned int prp_idx, page_idx;
+
+	prp_list = dma_alloc_coherent(dev, PAGE_SIZE, &prp_list_dma, GFP_KERNEL);
+	if (!prp_list)
+		return -ENOMEM;
+	cdq->prp_lists[0] = prp_list;
+	cdq->prp_lists_dma[0] = prp_list_dma;
+	cdq->nr_prp_lists = 1;
+
+	for (page_idx = 0, prp_idx = 0; page_idx < total_pages; page_idx++) {
+		dma_addr_t page_dma = nvme_get_cdq_pagedma(cdq, page_idx);
+
+		/* Current prp_list page full with entries still to place: chain. */
+		if (prp_idx == prps_per_page) {
+			__le64 *old = prp_list;
+
+			if (cdq->nr_prp_lists == MAX_NR_CDQ_PRPS)
+				goto err;
+			prp_list = dma_alloc_coherent(dev, PAGE_SIZE, &prp_list_dma,
+						      GFP_KERNEL);
+			if (!prp_list)
+				goto err;
+			cdq->prp_lists[cdq->nr_prp_lists] = prp_list;
+			cdq->prp_lists_dma[cdq->nr_prp_lists++] = prp_list_dma;
+
+			/* Chain from old to new prp_list */
+			prp_list[0] = old[prps_per_page - 1];
+			old[prps_per_page - 1] = cpu_to_le64(prp_list_dma);
+			prp_idx = 1;
+		}
+		prp_list[prp_idx++] = cpu_to_le64(page_dma);
+	}
+	return 0;
+err:
+	nvme_free_cdqmem_prp_lists(cdq);
+	return -ENOMEM;
+}
+
+/*
+ * Allocate the coherent store and prps for a CDQ.
+ * Expects cdq with size_nbytes and ctrl set.
+ */
+static inline int nvme_create_cdq_backing(struct cdq_nvme_queue *cdq)
+{
+	int ret;
+
+	if (!cdq->size_nbyte)
+		return -EINVAL;
+
+	ret = nvme_alloc_cdqmem_chunks(cdq);
+	if (ret)
+		return ret;
+
+	/* We pass cdq->chunks[0].dma_addr when cdq->nr_chunks == 1 */
+	if (cdq->nr_chunks > 1) {
+		ret = nvme_build_cdqmem_prp_list(cdq);
+		if (ret)
+			goto err_chunks;
+	}
+
+	return 0;
+
+err_chunks:
+	nvme_free_cdqmem_chunks(cdq);
+
+	return ret;
+}
+
+/* Release the coherent DMA backing allocated by nvme_create_cdq_backing(). */
+static inline void nvme_release_cdq_backing(struct cdq_nvme_queue *cdq)
+{
+	nvme_free_cdqmem_prp_lists(cdq);
+	nvme_free_cdqmem_chunks(cdq);
+}
+
 static int nvme_submit_delete_cdq_cmd(const struct cdq_nvme_queue *cdq)
 {
 	struct nvme_command c = {
@@ -27,10 +187,12 @@ static void nvme_delete_cdq_ctrl(struct cdq_nvme_queue *cdq)
 /* Does NOT send a CDQ delete NVMe cmd */
 static void nvme_delete_cdq_host(struct cdq_nvme_queue *cdq)
 {
-	u16 cdq_id = cdq->id;
 	struct nvme_ctrl *ctrl = cdq->ctrl;
 
-	xa_erase(&ctrl->cdqs, cdq_id);
+	if (xa_erase(&ctrl->cdqs, cdq->id) != cdq)
+		return;
+
+	nvme_release_cdq_backing(cdq);
 }
 
 void nvme_delete_cdq(struct cdq_nvme_queue *cdq)
diff --git a/drivers/nvme/host/cdq.h b/drivers/nvme/host/cdq.h
index 4378f97553ce2e649d8998eb168f82039dc762b1..5deaf3705c65250ea9fdc69e2e973987c8f084c1 100644
--- a/drivers/nvme/host/cdq.h
+++ b/drivers/nvme/host/cdq.h
@@ -8,9 +8,36 @@
 
 #include "nvme.h"
 
+/*
+ * The CDQ backing is a set of coherent DMA chunks. Chunk size expressed in
+ * host pages to match dma_alloc_coherency granularity.
+ */
+#define NVME_CDQ_CHUNK_ORDER		2
+#define NVME_CDQ_CHUNK_SIZE		(PAGE_SIZE << NVME_CDQ_CHUNK_ORDER)
+#define NVME_CDQ_PAGES_PER_CHUNK	(NVME_CDQ_CHUNK_SIZE / NVME_CTRL_PAGE_SIZE)
+
+/* Max PRP List pages we are willing to chain to describe a discontiguous CDQ. */
+#define MAX_NR_CDQ_PRPS		20
+
+struct nvme_cdq_chunk {
+	void		*vaddr;
+	dma_addr_t	dma_addr;
+	size_t		size;
+};
+
 struct cdq_nvme_queue {
 	u16 id;
 	struct nvme_ctrl *ctrl;
+	u32 size_nbyte;
+
+	/* Coherent backing store. */
+	struct nvme_cdq_chunk *chunks;
+	unsigned int nr_chunks;
+
+	/* PRP List pages describing the chunks to the controller (PC_DISCONT). */
+	__le64 *prp_lists[MAX_NR_CDQ_PRPS];
+	dma_addr_t prp_lists_dma[MAX_NR_CDQ_PRPS];
+	unsigned int nr_prp_lists;
 };
 
 void nvme_delete_cdq(struct cdq_nvme_queue *cdq);

-- 
2.50.1

[PATCH RFC v2 06/14] nvme: Add file descriptor to read CDQs

[Joel Granados]

CDQs are read using file descriptors provided by nvme_create_cdqfd. Only
add the logic to create and release the file descriptor; return "not
implemented" for now. The CDQ traversal logic will be added in a future
commit.

The CDQ memory is now pointed from two different places (the user space
FD and the nvme controller struct) and should only be freed when there
are no pointers left. Use a kref to ensure that it is in the FD's
private_data even if the controller is goes down.

Signed-off-by: Joel Granados <joel.granados@kernel.org>
---
 drivers/nvme/host/cdq.c | 71 +++++++++++++++++++++++++++++++++++++++++++++++++
 drivers/nvme/host/cdq.h | 21 +++++++++++++++
 2 files changed, 92 insertions(+)

diff --git a/drivers/nvme/host/cdq.c b/drivers/nvme/host/cdq.c
index 73dd6af42055687bec62ad0350c67736c9c047c2..63c0852e0ba17a17227b9a0189be5123358a2262 100644
--- a/drivers/nvme/host/cdq.c
+++ b/drivers/nvme/host/cdq.c
@@ -3,6 +3,9 @@
  * NVMe Controller Data Queue (CDQ) support.
  */
 
+#include <linux/anon_inodes.h>
+#include <linux/file.h>
+
 #include "nvme.h"
 #include "cdq.h"
 
@@ -151,6 +154,9 @@ static inline int nvme_create_cdq_backing(struct cdq_nvme_queue *cdq)
 			goto err_chunks;
 	}
 
+	/* FIXME: put this on the create_cdq function*/
+	kref_init(&cdq->ref);
+
 	return 0;
 
 err_chunks:
@@ -166,6 +172,69 @@ static inline void nvme_release_cdq_backing(struct cdq_nvme_queue *cdq)
 	nvme_free_cdqmem_chunks(cdq);
 }
 
+static ssize_t nvme_cdq_fops_read(struct file *filep, char __user *buf,
+				  size_t size_nbyte, loff_t *ppos)
+{
+	struct cdq_nvme_queue *cdq = filep->private_data;
+	size_t nbytes = round_down(size_nbyte, NVME_CDQ_MQ_ENTRY_NRBYTES);
+
+	if (*ppos)
+		return -ESPIPE;
+
+	if (size_nbyte < NVME_CDQ_MQ_ENTRY_NRBYTES)
+		return -EINVAL;
+
+	if (nbytes > (cdq->size_nbyte))
+		return -EINVAL;
+
+	/* CDQ traversal not implemented yet. */
+	return -EOPNOTSUPP;
+}
+
+/* File reference already dropped by the close path, so don't fput() */
+static int nvme_release_cdqfd(struct cdq_nvme_queue *cdq)
+{
+	nvme_cdq_put(cdq);
+	return 0;
+}
+
+static int nvme_cdq_fops_release(struct inode *inode, struct file *filep)
+{
+	return nvme_release_cdqfd(filep->private_data);
+}
+
+static const struct file_operations cdq_fops = {
+	.owner		= THIS_MODULE,
+	.open		= nonseekable_open,
+	.read		= nvme_cdq_fops_read,
+	.release	= nvme_cdq_fops_release,
+};
+
+__maybe_unused
+static int nvme_create_cdqfd(struct cdq_nvme_queue *cdq, int *cdq_fdno)
+{
+	int fdno;
+	struct file *filep;
+
+	filep = anon_inode_getfile("[cdq-readfd]", &cdq_fops, cdq, O_RDWR);
+	if (IS_ERR(filep))
+		return PTR_ERR(filep);
+
+	/* cdq is being pionted at by ->private_data. increase ref */
+	nvme_cdq_get(cdq);
+
+	fdno = get_unused_fd_flags(O_CLOEXEC | O_RDONLY | O_DIRECT);
+	if (fdno < 0) {
+		fput(filep); /* nvme_cdq_put through release */
+		return fdno;
+	}
+
+	fd_install(fdno, filep);
+	*cdq_fdno = fdno;
+
+	return 0;
+}
+
 static int nvme_submit_delete_cdq_cmd(const struct cdq_nvme_queue *cdq)
 {
 	struct nvme_command c = {
@@ -193,6 +262,8 @@ static void nvme_delete_cdq_host(struct cdq_nvme_queue *cdq)
 		return;
 
 	nvme_release_cdq_backing(cdq);
+
+	nvme_cdq_put(cdq);
 }
 
 void nvme_delete_cdq(struct cdq_nvme_queue *cdq)
diff --git a/drivers/nvme/host/cdq.h b/drivers/nvme/host/cdq.h
index 5deaf3705c65250ea9fdc69e2e973987c8f084c1..8c003aa75bd3e4df9a5f8af7ff21d1957dd43923 100644
--- a/drivers/nvme/host/cdq.h
+++ b/drivers/nvme/host/cdq.h
@@ -8,6 +8,8 @@
 
 #include "nvme.h"
 
+#define NVME_CDQ_MQ_ENTRY_NRBYTES	32
+
 /*
  * The CDQ backing is a set of coherent DMA chunks. Chunk size expressed in
  * host pages to match dma_alloc_coherency granularity.
@@ -38,8 +40,27 @@ struct cdq_nvme_queue {
 	__le64 *prp_lists[MAX_NR_CDQ_PRPS];
 	dma_addr_t prp_lists_dma[MAX_NR_CDQ_PRPS];
 	unsigned int nr_prp_lists;
+
+	/* Manage refs for read FD and controller xarray */
+	struct kref ref;
 };
 
+/* Must not touch cdq->ctrl: Ctrl may have been freed */
+static inline void nvme_free_cdq(struct kref *ref)
+{
+	kfree(container_of(ref, struct cdq_nvme_queue, ref));
+}
+
+static inline void nvme_cdq_get(struct cdq_nvme_queue *cdq)
+{
+	kref_get(&cdq->ref);
+}
+
+static inline void nvme_cdq_put(struct cdq_nvme_queue *cdq)
+{
+	kref_put(&cdq->ref, nvme_free_cdq);
+}
+
 void nvme_delete_cdq(struct cdq_nvme_queue *cdq);
 void nvme_delete_cdqs_host(struct nvme_ctrl *ctrl);
 void nvme_free_cdqs(struct nvme_ctrl *ctrl);

-- 
2.50.1

[PATCH RFC v2 07/14] nvme: Add the create CDQ functionality

[Joel Granados]

The CDQ create allocates the struct cdq_nvme_queue, allocates the memory
for the CDQ entries, sends the nvme command to create the CDQ at the
controller side, creates the file descriptor for user space and inserts
the CDQ struct in the controller cdqs xarray.

Note that:
* All memory that is to be shared with the controller needs to be setup
  before calling nvme_submit_create_cdq_cmd.
* Initializing the refcount before sending the create cmd to the
  controller "assigns" the reference to the controller. In other words,
  that cdq is being used by that controller.
* The creation of the cdq file descriptor is at the end because
  everything needs to be in place for when the user calls read. The FD
  will be "installed" once nvme_create_cdqfd returns.
* nvme_submit_create_cdq_cmd must come before xa_insert as the cdq is
  inserted to the xarray with the id returned by the controller

Signed-off-by: Joel Granados <joel.granados@kernel.org>
---
 drivers/nvme/host/cdq.c | 90 +++++++++++++++++++++++++++++++++++++++++++++++--
 drivers/nvme/host/cdq.h |  4 ++-
 2 files changed, 90 insertions(+), 4 deletions(-)

diff --git a/drivers/nvme/host/cdq.c b/drivers/nvme/host/cdq.c
index 63c0852e0ba17a17227b9a0189be5123358a2262..b1776637191db5747053bdcf4bdf97ef44a66566 100644
--- a/drivers/nvme/host/cdq.c
+++ b/drivers/nvme/host/cdq.c
@@ -154,9 +154,6 @@ static inline int nvme_create_cdq_backing(struct cdq_nvme_queue *cdq)
 			goto err_chunks;
 	}
 
-	/* FIXME: put this on the create_cdq function*/
-	kref_init(&cdq->ref);
-
 	return 0;
 
 err_chunks:
@@ -273,6 +270,93 @@ void nvme_delete_cdq(struct cdq_nvme_queue *cdq)
 }
 EXPORT_SYMBOL_GPL(nvme_delete_cdq);
 
+static int nvme_submit_create_cdq_cmd(struct cdq_nvme_queue *cdq)
+{
+	int ret;
+	union nvme_result result = {};
+	struct nvme_command c = {
+		.cdq.opcode = nvme_admin_cdq,
+		.cdq.sel = NVME_CDQ_CMD_MGMT_CREATE,
+		.cdq.mos = cpu_to_le16(NVME_CDQ_CMD_MGMT_CREATE_MOS_QT_UDMQ),
+		.cdq.dw11.cqs = cpu_to_le16(cdq->mc_id),
+		.cdq.cdqsize = cpu_to_le32(cdq->size_nbyte >> 2) // size is in dwords
+	};
+
+	if (cdq->nr_chunks < 2) {
+		c.cdq.dw11.flags = cpu_to_le16(NVME_CDQ_CMD_MGMT_CREATE_PC_CONT);
+		c.cdq.prp1 = cpu_to_le64(cdq->chunks[0].dma_addr);
+	} else {
+		c.cdq.dw11.flags = cpu_to_le16(NVME_CDQ_CMD_MGMT_CREATE_PC_DISCONT);
+		c.cdq.prp1 = cpu_to_le64(cdq->prp_lists_dma[0]);
+	}
+
+	ret = __nvme_submit_sync_cmd(cdq->ctrl->admin_q, &c, &result, NULL, 0, NVME_QID_ANY, 0);
+	if (ret)
+		return ret;
+
+	cdq->id = le16_to_cpu(result.u16);
+
+	return ret;
+}
+
+int nvme_create_cdq(struct nvme_ctrl *ctrl, const u32 entry_nr, const u16 mc_id)
+{
+	u64 size_nbyte = (u64)entry_nr * NVME_CDQ_MQ_ENTRY_NRBYTES;
+	struct cdq_nvme_queue *cdq = NULL;
+	int ret, cdq_fd;
+
+	/* The backing size and the CDQSIZE field are both u32 (bytes). */
+	if (size_nbyte > U32_MAX)
+		return -EINVAL;
+
+	cdq = kzalloc_obj(*cdq);
+	if (!cdq)
+		return -ENOMEM;
+
+	cdq->mc_id = mc_id;
+	cdq->ctrl = ctrl;
+	cdq->size_nbyte = (u32)size_nbyte;
+
+	ret = nvme_create_cdq_backing(cdq);
+	if (ret) {
+		kfree(cdq);
+		return ret;
+	}
+
+	kref_init(&cdq->ref);
+
+	ret = nvme_submit_create_cdq_cmd(cdq);
+	if (ret)
+		goto del_cdqmem;
+
+	ret = xa_insert(&cdq->ctrl->cdqs, cdq->id, cdq, GFP_KERNEL);
+	if (ret)
+		goto del_cmd;
+
+	ret = nvme_create_cdqfd(cdq, &cdq_fd);
+	if (ret)
+		goto del_xarray;
+
+	return 0;
+
+del_xarray:
+	nvme_delete_cdq(cdq);
+	/*nvme_delete_cdq, has everything */
+	return ret;
+
+del_cmd:
+	if (nvme_submit_delete_cdq_cmd(cdq))
+		WARN_ONCE(1, "Failed delete CDQ (id: %d)", cdq->id);
+
+del_cdqmem:
+	/* puts the ref acquired by kref_init */
+	nvme_release_cdq_backing(cdq);
+	nvme_cdq_put(cdq);
+
+	return ret;
+}
+EXPORT_SYMBOL_GPL(nvme_create_cdq);
+
 /* Will NOT send a CDQ delete NVMe cmd. */
 void nvme_delete_cdqs_host(struct nvme_ctrl *ctrl)
 {
diff --git a/drivers/nvme/host/cdq.h b/drivers/nvme/host/cdq.h
index 8c003aa75bd3e4df9a5f8af7ff21d1957dd43923..773f93817d498784450eb0c8c9ab00a95f0657dc 100644
--- a/drivers/nvme/host/cdq.h
+++ b/drivers/nvme/host/cdq.h
@@ -11,7 +11,7 @@
 #define NVME_CDQ_MQ_ENTRY_NRBYTES	32
 
 /*
- * The CDQ backing is a set of coherent DMA chunks. Chunk size expressed in
+ * The CDQ backing is a set of coherent DMA chunks expressed in
  * host pages to match dma_alloc_coherency granularity.
  */
 #define NVME_CDQ_CHUNK_ORDER		2
@@ -31,6 +31,7 @@ struct cdq_nvme_queue {
 	u16 id;
 	struct nvme_ctrl *ctrl;
 	u32 size_nbyte;
+	u16 mc_id; // migratable controller id
 
 	/* Coherent backing store. */
 	struct nvme_cdq_chunk *chunks;
@@ -62,6 +63,7 @@ static inline void nvme_cdq_put(struct cdq_nvme_queue *cdq)
 }
 
 void nvme_delete_cdq(struct cdq_nvme_queue *cdq);
+int nvme_create_cdq(struct nvme_ctrl *ctrl, const u32 entry_nr, const u16 mc_id);
 void nvme_delete_cdqs_host(struct nvme_ctrl *ctrl);
 void nvme_free_cdqs(struct nvme_ctrl *ctrl);
 

-- 
2.50.1

[PATCH RFC v2 08/14] nvme: Submit track send command to start/stop the CDQ

[Joel Granados]

Send the start command at the end of Controller Data Queue creation.
Send the stop before sending the "delete" message to the controller.

Signed-off-by: Joel Granados <joel.granados@kernel.org>
---
 drivers/nvme/host/cdq.c | 21 +++++++++++++++++++++
 1 file changed, 21 insertions(+)

diff --git a/drivers/nvme/host/cdq.c b/drivers/nvme/host/cdq.c
index b1776637191db5747053bdcf4bdf97ef44a66566..0f01b58a9c3145faae83a81627fa5a93cfc03cf4 100644
--- a/drivers/nvme/host/cdq.c
+++ b/drivers/nvme/host/cdq.c
@@ -243,9 +243,27 @@ static int nvme_submit_delete_cdq_cmd(const struct cdq_nvme_queue *cdq)
 	return __nvme_submit_sync_cmd(cdq->ctrl->admin_q, &c, NULL, NULL, 0, NVME_QID_ANY, 0);
 }
 
+static int nvme_submit_startstop_cdq_cmd(struct cdq_nvme_queue *cdq, uint startstop)
+{
+	struct nvme_command c = {
+		.track_send.opcode = nvme_admin_track_send,
+		.track_send.sel = NVME_TRSND_CMD_MGMT_LOG_USR_DATA,
+		.track_send.mos = cpu_to_le16(startstop),
+		.track_send.dw11 = cpu_to_le32((u32)cdq->id)
+	};
+
+	return  __nvme_submit_sync_cmd(cdq->ctrl->admin_q, &c, NULL, NULL, 0, NVME_QID_ANY, 0);
+}
+#define nvme_start_cdq(cdq) \
+	nvme_submit_startstop_cdq_cmd(cdq, NVME_TRSND_CMD_MGMT_LOG_USR_DATA_START)
+#define nvme_stop_cdq(cdq) \
+	nvme_submit_startstop_cdq_cmd(cdq, NVME_TRSND_CMD_MGMT_LOG_USR_DATA_STOP)
+
 /* Sends a CDQ delete NVMe cmd */
 static void nvme_delete_cdq_ctrl(struct cdq_nvme_queue *cdq)
 {
+	if (nvme_stop_cdq(cdq))
+		WARN_ONCE(1, "Failed to stop CDQ (id: %d)", cdq->id);
 	if (nvme_submit_delete_cdq_cmd(cdq))
 		WARN_ONCE(1, "Failed delete CDQ (id: %d)", cdq->id);
 }
@@ -337,6 +355,9 @@ int nvme_create_cdq(struct nvme_ctrl *ctrl, const u32 entry_nr, const u16 mc_id)
 	if (ret)
 		goto del_xarray;
 
+	ret = nvme_start_cdq(cdq);
+	if (ret)
+		goto del_xarray;
 	return 0;
 
 del_xarray:

-- 
2.50.1

[PATCH RFC v2 09/14] nvme: Coordinate CDQ backing mem and CDQ FD

[Joel Granados]

The user space file descriptor fails if the memory backing the
Controller Data Queue (CDQ) is not "valid". This can happen as part of
the controller tear-down procedure.

Test the valid_mem variable when reading the from the file descriptor.
Set on CDQ creation and unset when the CDQ is being deleted on the host
side.

Signed-off-by: Joel Granados <joel.granados@kernel.org>
---
 drivers/nvme/host/cdq.c | 20 ++++++++++++++++++--
 drivers/nvme/host/cdq.h |  3 +++
 2 files changed, 21 insertions(+), 2 deletions(-)

diff --git a/drivers/nvme/host/cdq.c b/drivers/nvme/host/cdq.c
index 0f01b58a9c3145faae83a81627fa5a93cfc03cf4..7fe08afa0d80db5f3d60993aba05624b5c59b309 100644
--- a/drivers/nvme/host/cdq.c
+++ b/drivers/nvme/host/cdq.c
@@ -184,6 +184,9 @@ static ssize_t nvme_cdq_fops_read(struct file *filep, char __user *buf,
 	if (nbytes > (cdq->size_nbyte))
 		return -EINVAL;
 
+	if (!READ_ONCE(cdq->valid_mem))
+		return -EINVAL;
+
 	/* CDQ traversal not implemented yet. */
 	return -EOPNOTSUPP;
 }
@@ -207,7 +210,16 @@ static const struct file_operations cdq_fops = {
 	.release	= nvme_cdq_fops_release,
 };
 
-__maybe_unused
+/* Should only handle cdq struct and ctrl kref */
+void nvme_free_cdq(struct kref *ref)
+{
+	struct cdq_nvme_queue *cdq = container_of(ref, struct cdq_nvme_queue, ref);
+
+	/* Drop the ctrl kref held since creation */
+	nvme_put_ctrl(cdq->ctrl);
+	kfree(cdq);
+}
+
 static int nvme_create_cdqfd(struct cdq_nvme_queue *cdq, int *cdq_fdno)
 {
 	int fdno;
@@ -276,8 +288,9 @@ static void nvme_delete_cdq_host(struct cdq_nvme_queue *cdq)
 	if (xa_erase(&ctrl->cdqs, cdq->id) != cdq)
 		return;
 
-	nvme_release_cdq_backing(cdq);
+	WRITE_ONCE(cdq->valid_mem, false);
 
+	nvme_release_cdq_backing(cdq);
 	nvme_cdq_put(cdq);
 }
 
@@ -342,11 +355,14 @@ int nvme_create_cdq(struct nvme_ctrl *ctrl, const u32 entry_nr, const u16 mc_id)
 	}
 
 	kref_init(&cdq->ref);
+	nvme_get_ctrl(cdq->ctrl);
 
 	ret = nvme_submit_create_cdq_cmd(cdq);
 	if (ret)
 		goto del_cdqmem;
 
+	WRITE_ONCE(cdq->valid_mem, true);
+
 	ret = xa_insert(&cdq->ctrl->cdqs, cdq->id, cdq, GFP_KERNEL);
 	if (ret)
 		goto del_cmd;
diff --git a/drivers/nvme/host/cdq.h b/drivers/nvme/host/cdq.h
index 773f93817d498784450eb0c8c9ab00a95f0657dc..aa9469af0c7ef08b9f8ebf93e468c7f892aee4b1 100644
--- a/drivers/nvme/host/cdq.h
+++ b/drivers/nvme/host/cdq.h
@@ -42,6 +42,9 @@ struct cdq_nvme_queue {
 	dma_addr_t prp_lists_dma[MAX_NR_CDQ_PRPS];
 	unsigned int nr_prp_lists;
 
+	/* True if mem for chunks and prps is valid */
+	bool valid_mem;
+
 	/* Manage refs for read FD and controller xarray */
 	struct kref ref;
 };

-- 
2.50.1

[PATCH RFC v2 10/14] nvme: Pin a ctrl ref to every CDQ

[Joel Granados]

Make sure that the migration manager controller pointed by the CDQ does
not get freed unless all CDQs are terminated by user space.

Signed-off-by: Joel Granados <joel.granados@kernel.org>
---
 drivers/nvme/host/cdq.h | 23 +++++++++++++++--------
 1 file changed, 15 insertions(+), 8 deletions(-)

diff --git a/drivers/nvme/host/cdq.h b/drivers/nvme/host/cdq.h
index aa9469af0c7ef08b9f8ebf93e468c7f892aee4b1..a8c714ab5fada4325463013c2ffb23a117cd8423 100644
--- a/drivers/nvme/host/cdq.h
+++ b/drivers/nvme/host/cdq.h
@@ -29,7 +29,7 @@ struct nvme_cdq_chunk {
 
 struct cdq_nvme_queue {
 	u16 id;
-	struct nvme_ctrl *ctrl;
+	struct nvme_ctrl *ctrl; // this is kref'ed for the life of the CDQ
 	u32 size_nbyte;
 	u16 mc_id; // migratable controller id
 
@@ -49,11 +49,20 @@ struct cdq_nvme_queue {
 	struct kref ref;
 };
 
-/* Must not touch cdq->ctrl: Ctrl may have been freed */
-static inline void nvme_free_cdq(struct kref *ref)
-{
-	kfree(container_of(ref, struct cdq_nvme_queue, ref));
-}
+/*
+ * nvme_free_cdq:
+ *    - Is called with the cdq struct has no more krefs.
+ *    - Will free (cdq) and decrease ctrl kref.
+ */
+void nvme_free_cdq(struct kref *ref);
+/* nvme_cdq_delete:
+ *    - Sends a nvme delete cmd to the controller
+ *    - removes (cdq) from controller xarray
+ *    - frees the backing CDQ mem
+ *    - Calls nvme_free_cdq if there are no more refs
+ */
+void nvme_delete_cdq(struct cdq_nvme_queue *cdq);
+int nvme_create_cdq(struct nvme_ctrl *ctrl, const u32 entry_nr, const u16 mc_id);
 
 static inline void nvme_cdq_get(struct cdq_nvme_queue *cdq)
 {
@@ -65,8 +74,6 @@ static inline void nvme_cdq_put(struct cdq_nvme_queue *cdq)
 	kref_put(&cdq->ref, nvme_free_cdq);
 }
 
-void nvme_delete_cdq(struct cdq_nvme_queue *cdq);
-int nvme_create_cdq(struct nvme_ctrl *ctrl, const u32 entry_nr, const u16 mc_id);
 void nvme_delete_cdqs_host(struct nvme_ctrl *ctrl);
 void nvme_free_cdqs(struct nvme_ctrl *ctrl);
 

-- 
2.50.1

[PATCH RFC v2 11/14] nvme: Implement CDQ traversal on the read path

[Joel Granados]

Replace the initial -EOPNOTSUPP stub with a call to
nvme_traversecopy_cdq which copies new controller entries to the user
space buffer, forwards and wraps host_head around when it reaches the
end of the queue.

Signed-off-by: Joel Granados <joel.granados@kernel.org>
---
 drivers/nvme/host/cdq.c | 73 +++++++++++++++++++++++++++++++++++++++++++++++--
 drivers/nvme/host/cdq.h |  7 +++++
 2 files changed, 78 insertions(+), 2 deletions(-)

diff --git a/drivers/nvme/host/cdq.c b/drivers/nvme/host/cdq.c
index 7fe08afa0d80db5f3d60993aba05624b5c59b309..36107cf2fe11ff212d5b02dd5513fe02e3ae8f81 100644
--- a/drivers/nvme/host/cdq.c
+++ b/drivers/nvme/host/cdq.c
@@ -5,6 +5,7 @@
 
 #include <linux/anon_inodes.h>
 #include <linux/file.h>
+#include <linux/uaccess.h>
 
 #include "nvme.h"
 #include "cdq.h"
@@ -169,6 +170,75 @@ static inline void nvme_release_cdq_backing(struct cdq_nvme_queue *cdq)
 	nvme_free_cdqmem_chunks(cdq);
 }
 
+
+static inline void *nvme_get_cdq_entryvaddr(struct cdq_nvme_queue *cdq,
+					    unsigned int entry_idx)
+{
+	return cdq->chunks[entry_idx / NVME_CDQ_MQ_ENTRY_PER_CHUNK].vaddr +
+		(entry_idx % NVME_CDQ_MQ_ENTRY_PER_CHUNK) * NVME_CDQ_MQ_ENTRY_NRBYTES;
+}
+
+/* Advance cdq->host_head at most nrbytes and return actual advanced bytes */
+static size_t nvme_advance_cdq(struct cdq_nvme_queue *cdq, size_t max_nrbyte)
+{
+	size_t target_nbyte = 0;
+	void *entry;
+	u8 phase_bit;
+	u32 cdq_nrentry = cdq->size_nbyte / NVME_CDQ_MQ_ENTRY_NRBYTES;
+
+	while (target_nbyte < max_nrbyte) {
+		entry = nvme_get_cdq_entryvaddr(cdq, cdq->host_head);
+		phase_bit = (*(u8 *)(entry + NVME_CDQ_MQ_PHASE_OFFSET) & NVME_CDQ_MQ_PHASE_MASK);
+
+		if (phase_bit == cdq->phase_bit)
+			break;
+
+		cdq->host_head = (cdq->host_head + 1) % cdq_nrentry;
+		target_nbyte += NVME_CDQ_MQ_ENTRY_NRBYTES;
+		if (unlikely(cdq->host_head == 0))
+			cdq->phase_bit = ~cdq->phase_bit & NVME_CDQ_MQ_PHASE_MASK;
+	}
+
+	return target_nbyte;
+}
+
+static ssize_t nvme_traversecopy_cdq(struct cdq_nvme_queue *cdq, size_t max_nrbyte,
+				   void *priv_data)
+{
+	char __user *to_buf = priv_data;
+	void *from_buf;
+	u32 init_host_head = cdq->host_head;
+	u8 init_phase_bit = cdq->phase_bit;
+
+	size_t target_nbyte = nvme_advance_cdq(cdq, max_nrbyte);
+	size_t copied_nbyte = 0, chunks_idx, entry_idx, tx_nbytes;
+
+	if (target_nbyte == 0)
+		goto out;
+
+	for (chunks_idx = init_host_head / NVME_CDQ_MQ_ENTRY_PER_CHUNK,
+	     entry_idx = init_host_head % NVME_CDQ_MQ_ENTRY_PER_CHUNK;
+	     copied_nbyte < target_nbyte;
+	     chunks_idx = (chunks_idx + 1) % cdq->nr_chunks, entry_idx = 0) {
+		from_buf = cdq->chunks[chunks_idx].vaddr +
+			(entry_idx * NVME_CDQ_MQ_ENTRY_NRBYTES);
+		tx_nbytes = min(target_nbyte - copied_nbyte,
+				NVME_CDQ_CHUNK_SIZE - (entry_idx * NVME_CDQ_MQ_ENTRY_NRBYTES));
+		if (copy_to_user(to_buf, from_buf, tx_nbytes))
+			goto err_out;
+		copied_nbyte += tx_nbytes;
+		to_buf += tx_nbytes;
+	}
+
+out:
+	return copied_nbyte;
+
+err_out:
+	cdq->host_head = init_host_head;
+	cdq->phase_bit = init_phase_bit;
+	return -EFAULT;
+}
+
 static ssize_t nvme_cdq_fops_read(struct file *filep, char __user *buf,
 				  size_t size_nbyte, loff_t *ppos)
 {
@@ -187,8 +257,7 @@ static ssize_t nvme_cdq_fops_read(struct file *filep, char __user *buf,
 	if (!READ_ONCE(cdq->valid_mem))
 		return -EINVAL;
 
-	/* CDQ traversal not implemented yet. */
-	return -EOPNOTSUPP;
+	return nvme_traversecopy_cdq(cdq, nbytes, buf);
 }
 
 /* File reference already dropped by the close path, so don't fput() */
diff --git a/drivers/nvme/host/cdq.h b/drivers/nvme/host/cdq.h
index a8c714ab5fada4325463013c2ffb23a117cd8423..96f208264c6144d26aa4ffc34854bb16d3f19af9 100644
--- a/drivers/nvme/host/cdq.h
+++ b/drivers/nvme/host/cdq.h
@@ -9,6 +9,8 @@
 #include "nvme.h"
 
 #define NVME_CDQ_MQ_ENTRY_NRBYTES	32
+#define NVME_CDQ_MQ_PHASE_MASK		0x1
+#define NVME_CDQ_MQ_PHASE_OFFSET	(NVME_CDQ_MQ_ENTRY_NRBYTES - 1)
 
 /*
  * The CDQ backing is a set of coherent DMA chunks expressed in
@@ -17,6 +19,7 @@
 #define NVME_CDQ_CHUNK_ORDER		2
 #define NVME_CDQ_CHUNK_SIZE		(PAGE_SIZE << NVME_CDQ_CHUNK_ORDER)
 #define NVME_CDQ_PAGES_PER_CHUNK	(NVME_CDQ_CHUNK_SIZE / NVME_CTRL_PAGE_SIZE)
+#define NVME_CDQ_MQ_ENTRY_PER_CHUNK	(NVME_CDQ_CHUNK_SIZE / NVME_CDQ_MQ_ENTRY_NRBYTES)
 
 /* Max PRP List pages we are willing to chain to describe a discontiguous CDQ. */
 #define MAX_NR_CDQ_PRPS		20
@@ -45,6 +48,10 @@ struct cdq_nvme_queue {
 	/* True if mem for chunks and prps is valid */
 	bool valid_mem;
 
+	/* How far the CDQ was consumed by the host */
+	u32 host_head;
+	u8 phase_bit;
+
 	/* Manage refs for read FD and controller xarray */
 	struct kref ref;
 };

-- 
2.50.1

[PATCH RFC v2 12/14] nvme: Update the controller with the CDQ set-feature cmd

[Joel Granados]

After advancing the queue head, send the new head position to the
controller using an NVMe set feature command. Avoid unnecessary wait on
the read by not waiting for command completion.

sf_lock serializes reads that finish before the in-flight set feature has
completed; in these cases the cmd will be resent after completion of the
previous one.

Also, re-send the set feature command right after returning and seeing
that the host head has actually moved forward since the last time the
command was sent.

Signed-off-by: Joel Granados <joel.granados@kernel.org>
---
 drivers/nvme/host/cdq.c | 92 +++++++++++++++++++++++++++++++++++++++++++++++++
 drivers/nvme/host/cdq.h | 14 ++++++++
 2 files changed, 106 insertions(+)

diff --git a/drivers/nvme/host/cdq.c b/drivers/nvme/host/cdq.c
index 36107cf2fe11ff212d5b02dd5513fe02e3ae8f81..8e59c16ddf63df4977e1bd869be1e3a7ad19b9c5 100644
--- a/drivers/nvme/host/cdq.c
+++ b/drivers/nvme/host/cdq.c
@@ -202,6 +202,90 @@ static size_t nvme_advance_cdq(struct cdq_nvme_queue *cdq, size_t max_nrbyte)
 	return target_nbyte;
 }
 
+static void nvme_submit_sfcmd_cdq(struct cdq_nvme_queue *cdq);
+
+/* updates cntl_head, exits sf_inflight and re-arms if needed */
+static enum rq_end_io_ret nvme_endio_sfcmd_cdq(struct request *rq,
+					       blk_status_t status,
+					       const struct io_comp_batch *iob)
+{
+	struct cdq_nvme_queue *cdq = rq->end_io_data;
+	unsigned long flags;
+	bool rearm = false;
+
+	WRITE_ONCE(cdq->cntl_head, cdq->sent_head);
+	blk_mq_free_request(rq);
+
+	spin_lock_irqsave(&cdq->sf_lock, flags);
+	cdq->sf_inflight = false;
+	/* cntl_head was just written above, plain read under the lock. */
+	if (READ_ONCE(cdq->host_head) != cdq->cntl_head) {
+		cdq->sf_inflight = true;
+		rearm = true;
+	}
+	spin_unlock_irqrestore(&cdq->sf_lock, flags);
+
+	/* Re-arm (takes its own ref) before dropping ours, so cdq stays alive. */
+	if (rearm)
+		nvme_submit_sfcmd_cdq(cdq);
+
+	nvme_cdq_put(cdq);
+	return RQ_END_IO_NONE;
+}
+
+/* Inform ctrl that head as advance by sending a CDQ set-feature command*/
+static void nvme_submit_sfcmd_cdq(struct cdq_nvme_queue *cdq)
+{
+	struct nvme_command c = { };
+	struct request *rq;
+	unsigned long flags;
+	u32 head = READ_ONCE(cdq->host_head);
+
+	c.features.opcode = nvme_admin_set_features;
+	c.features.fid = cpu_to_le32(NVME_FEAT_CDQ);
+	c.features.dword11 = cpu_to_le32(cdq->id & NVME_FEAT_CDQ_ID_MASK);
+	c.features.dword12 = cpu_to_le32(head);
+
+	rq = blk_mq_alloc_request(cdq->ctrl->admin_q, nvme_req_op(&c),
+				  BLK_MQ_REQ_NOWAIT);
+	if (IS_ERR(rq)) {
+		/*
+		 * No admin tag right now and we cannot sleep. Drop the slot; the
+		 * next read() will re-arm.
+		 */
+		spin_lock_irqsave(&cdq->sf_lock, flags);
+		cdq->sf_inflight = false;
+		spin_unlock_irqrestore(&cdq->sf_lock, flags);
+		return;
+	}
+
+	cdq->sent_head = head;
+	nvme_init_request(rq, &c);
+	rq->end_io = nvme_endio_sfcmd_cdq;
+	rq->end_io_data = cdq;
+
+	/* Pin cdq for the lifetime of the command */
+	nvme_cdq_get(cdq);
+	blk_execute_rq_nowait(rq, false);
+}
+
+/* submit set feature if none are in-flight */
+static void nvme_kick_cdq(struct cdq_nvme_queue *cdq)
+{
+	unsigned long flags;
+	bool submit = false;
+
+	spin_lock_irqsave(&cdq->sf_lock, flags);
+	if (!cdq->sf_inflight) {
+		cdq->sf_inflight = true;
+		submit = true;
+	}
+	spin_unlock_irqrestore(&cdq->sf_lock, flags);
+
+	if (submit)
+		nvme_submit_sfcmd_cdq(cdq);
+}
+
 static ssize_t nvme_traversecopy_cdq(struct cdq_nvme_queue *cdq, size_t max_nrbyte,
 				   void *priv_data)
 {
@@ -231,6 +315,13 @@ static ssize_t nvme_traversecopy_cdq(struct cdq_nvme_queue *cdq, size_t max_nrby
 	}
 
 out:
+	/*
+	 * host_head advanced past consumed entries; tell the controller its head
+	 * can move up. Decoupled from this read: the set-feature admin
+	 * round-trip must not delay the data path.
+	 */
+	if (copied_nbyte)
+		nvme_kick_cdq(cdq);
 	return copied_nbyte;
 
 err_out:
@@ -416,6 +507,7 @@ int nvme_create_cdq(struct nvme_ctrl *ctrl, const u32 entry_nr, const u16 mc_id)
 	cdq->mc_id = mc_id;
 	cdq->ctrl = ctrl;
 	cdq->size_nbyte = (u32)size_nbyte;
+	spin_lock_init(&cdq->sf_lock);
 
 	ret = nvme_create_cdq_backing(cdq);
 	if (ret) {
diff --git a/drivers/nvme/host/cdq.h b/drivers/nvme/host/cdq.h
index 96f208264c6144d26aa4ffc34854bb16d3f19af9..893aeefb2dfc2c25f390e43eb508067b34dce31f 100644
--- a/drivers/nvme/host/cdq.h
+++ b/drivers/nvme/host/cdq.h
@@ -50,8 +50,22 @@ struct cdq_nvme_queue {
 
 	/* How far the CDQ was consumed by the host */
 	u32 host_head;
+
+	/*
+	 * Value sent by the in-flight set-feature cmd
+	 * Differs from cntl_head until set-feature cmd completes
+	 */
+	u32 sent_head;
+
+	/* Last acked CDQ head update. Trails host_head.*/
+	u32 cntl_head;
+
 	u8 phase_bit;
 
+	/* sf_* controlls if the feature set cmd is done or is still inflight */
+	spinlock_t sf_lock;
+	bool sf_inflight;
+
 	/* Manage refs for read FD and controller xarray */
 	struct kref ref;
 };

-- 
2.50.1

[PATCH RFC v2 13/14] nvme: Use eventfd for CDQ Tail pointer triggers

[Joel Granados]

When an eventfd file descriptor is passed to the nvme_create_cdq
function, add the eventfd context to the cdq struct and increase its
reference. Route the AEN one shot event triggered by the controller to
the eventfd signal. Set the tail pointer trigger (TPT) as soon as an
empty CDQ is encountered.

The TPT offset sent to the controller is in pending_tpt which has the
last value assigned to it. For the time being it is set to +1 on a
"zero" read.

Note:
if a set-feature is already in flight on an empty read, the arm lingers
in pending_tpt until the next submitting kick.

Signed-off-by: Joel Granados <joel.granados@kernel.org>
---
 drivers/nvme/host/cdq.c  | 81 +++++++++++++++++++++++++++++++++++++++++++++---
 drivers/nvme/host/cdq.h  |  9 +++++-
 drivers/nvme/host/core.c |  3 +-
 include/linux/nvme.h     |  1 +
 4 files changed, 88 insertions(+), 6 deletions(-)

diff --git a/drivers/nvme/host/cdq.c b/drivers/nvme/host/cdq.c
index 8e59c16ddf63df4977e1bd869be1e3a7ad19b9c5..f4aabf0a8817058a11e6c30dcea9554076861327 100644
--- a/drivers/nvme/host/cdq.c
+++ b/drivers/nvme/host/cdq.c
@@ -4,6 +4,7 @@
  */
 
 #include <linux/anon_inodes.h>
+#include <linux/eventfd.h>
 #include <linux/file.h>
 #include <linux/uaccess.h>
 
@@ -240,10 +241,26 @@ static void nvme_submit_sfcmd_cdq(struct cdq_nvme_queue *cdq)
 	struct request *rq;
 	unsigned long flags;
 	u32 head = READ_ONCE(cdq->host_head);
+	u32 tpt = READ_ONCE(cdq->pending_tpt);
+	u32 dword11 = cdq->id & NVME_FEAT_CDQ_ID_MASK;
+	u32 cdq_nrentry = cdq->size_nbyte / NVME_CDQ_MQ_ENTRY_NRBYTES;
 
 	c.features.opcode = nvme_admin_set_features;
 	c.features.fid = cpu_to_le32(NVME_FEAT_CDQ);
-	c.features.dword11 = cpu_to_le32(cdq->id & NVME_FEAT_CDQ_ID_MASK);
+
+	if (unlikely(tpt != 0)) {
+		/*
+		 * FIXME: There is a small chance that the sent tpt will have
+		 * already been handled by the time this command completes. If
+		 * we find this to be true in the CDQ, we need to send a
+		 * subsequent feature_id to disable the tail pointer trigger.
+		 * section 5.1.25.1.23 nvme base spec.
+		 */
+		dword11 |= NVME_FEAT_CDQ_ETPT_MASK;
+		c.features.dword13 = cpu_to_le32((head + tpt) % cdq_nrentry);
+	}
+
+	c.features.dword11 = cpu_to_le32(dword11);
 	c.features.dword12 = cpu_to_le32(head);
 
 	rq = blk_mq_alloc_request(cdq->ctrl->admin_q, nvme_req_op(&c),
@@ -260,6 +277,7 @@ static void nvme_submit_sfcmd_cdq(struct cdq_nvme_queue *cdq)
 	}
 
 	cdq->sent_head = head;
+	WRITE_ONCE(cdq->pending_tpt, 0);
 	nvme_init_request(rq, &c);
 	rq->end_io = nvme_endio_sfcmd_cdq;
 	rq->end_io_data = cdq;
@@ -320,8 +338,13 @@ static ssize_t nvme_traversecopy_cdq(struct cdq_nvme_queue *cdq, size_t max_nrby
 	 * can move up. Decoupled from this read: the set-feature admin
 	 * round-trip must not delay the data path.
 	 */
-	if (copied_nbyte)
+	if (copied_nbyte) {
 		nvme_kick_cdq(cdq);
+	} else if (cdq->tpt_efd_ctx) {
+		/* Controller will one-shot AEN when more entries are added */
+		WRITE_ONCE(cdq->pending_tpt, 1);
+		nvme_kick_cdq(cdq);
+	}
 	return copied_nbyte;
 
 err_out:
@@ -404,6 +427,31 @@ static int nvme_create_cdqfd(struct cdq_nvme_queue *cdq, int *cdq_fdno)
 	return 0;
 }
 
+static void nvme_put_cdq_tpt(struct cdq_nvme_queue *cdq)
+{
+	if (cdq->tpt_efd_ctx)
+		eventfd_ctx_put(cdq->tpt_efd_ctx);
+	cdq->tpt_efd_ctx = NULL;
+}
+
+static int nvme_get_cdq_tpt(struct cdq_nvme_queue *cdq, const int tpt_fd)
+{
+	struct eventfd_ctx *tmp;
+
+	if (tpt_fd <= 0)
+		return 0;
+
+	/* put the old one */
+	nvme_put_cdq_tpt(cdq);
+
+	tmp = eventfd_ctx_fdget(tpt_fd);
+	if (IS_ERR(tmp))
+		return -EINVAL;
+
+	cdq->tpt_efd_ctx = tmp;
+	return 0;
+}
+
 static int nvme_submit_delete_cdq_cmd(const struct cdq_nvme_queue *cdq)
 {
 	struct nvme_command c = {
@@ -450,6 +498,8 @@ static void nvme_delete_cdq_host(struct cdq_nvme_queue *cdq)
 
 	WRITE_ONCE(cdq->valid_mem, false);
 
+	nvme_put_cdq_tpt(cdq);
+
 	nvme_release_cdq_backing(cdq);
 	nvme_cdq_put(cdq);
 }
@@ -461,6 +511,22 @@ void nvme_delete_cdq(struct cdq_nvme_queue *cdq)
 }
 EXPORT_SYMBOL_GPL(nvme_delete_cdq);
 
+int nvme_handle_cdq_aen_tpevent(struct nvme_ctrl *ctrl, u32 event_param)
+{
+	u16 cdq_id = event_param & NVME_FEAT_CDQ_ID_MASK;
+	struct cdq_nvme_queue *cdq;
+
+	cdq = xa_load(&ctrl->cdqs, cdq_id);
+	if (xa_is_err(cdq))
+		return xa_err(cdq);
+	if (!cdq->tpt_efd_ctx)
+		return -EINVAL;
+
+	eventfd_signal(cdq->tpt_efd_ctx);
+
+	return 0;
+}
+
 static int nvme_submit_create_cdq_cmd(struct cdq_nvme_queue *cdq)
 {
 	int ret;
@@ -490,7 +556,7 @@ static int nvme_submit_create_cdq_cmd(struct cdq_nvme_queue *cdq)
 	return ret;
 }
 
-int nvme_create_cdq(struct nvme_ctrl *ctrl, const u32 entry_nr, const u16 mc_id)
+int nvme_create_cdq(struct nvme_ctrl *ctrl, const u32 entry_nr, const u16 mc_id, const int tpt_fd)
 {
 	u64 size_nbyte = (u64)entry_nr * NVME_CDQ_MQ_ENTRY_NRBYTES;
 	struct cdq_nvme_queue *cdq = NULL;
@@ -518,10 +584,14 @@ int nvme_create_cdq(struct nvme_ctrl *ctrl, const u32 entry_nr, const u16 mc_id)
 	kref_init(&cdq->ref);
 	nvme_get_ctrl(cdq->ctrl);
 
-	ret = nvme_submit_create_cdq_cmd(cdq);
+	ret = nvme_get_cdq_tpt(cdq, tpt_fd);
 	if (ret)
 		goto del_cdqmem;
 
+	ret = nvme_submit_create_cdq_cmd(cdq);
+	if (ret)
+		goto put_tpt;
+
 	WRITE_ONCE(cdq->valid_mem, true);
 
 	ret = xa_insert(&cdq->ctrl->cdqs, cdq->id, cdq, GFP_KERNEL);
@@ -546,6 +616,9 @@ int nvme_create_cdq(struct nvme_ctrl *ctrl, const u32 entry_nr, const u16 mc_id)
 	if (nvme_submit_delete_cdq_cmd(cdq))
 		WARN_ONCE(1, "Failed delete CDQ (id: %d)", cdq->id);
 
+put_tpt:
+	nvme_put_cdq_tpt(cdq);
+
 del_cdqmem:
 	/* puts the ref acquired by kref_init */
 	nvme_release_cdq_backing(cdq);
diff --git a/drivers/nvme/host/cdq.h b/drivers/nvme/host/cdq.h
index 893aeefb2dfc2c25f390e43eb508067b34dce31f..2958c6b92b1312930414b8466690aa7c0eb57754 100644
--- a/drivers/nvme/host/cdq.h
+++ b/drivers/nvme/host/cdq.h
@@ -60,6 +60,9 @@ struct cdq_nvme_queue {
 	/* Last acked CDQ head update. Trails host_head.*/
 	u32 cntl_head;
 
+	/* ETPT offset to arm on next set-feature send, 0 = none */
+	u32 pending_tpt;
+
 	u8 phase_bit;
 
 	/* sf_* controlls if the feature set cmd is done or is still inflight */
@@ -68,6 +71,9 @@ struct cdq_nvme_queue {
 
 	/* Manage refs for read FD and controller xarray */
 	struct kref ref;
+
+	/* Has a value if user setup an event fd for AEN tpt events */
+	struct eventfd_ctx *tpt_efd_ctx;
 };
 
 /*
@@ -83,7 +89,7 @@ void nvme_free_cdq(struct kref *ref);
  *    - Calls nvme_free_cdq if there are no more refs
  */
 void nvme_delete_cdq(struct cdq_nvme_queue *cdq);
-int nvme_create_cdq(struct nvme_ctrl *ctrl, const u32 entry_nr, const u16 mc_id);
+int nvme_create_cdq(struct nvme_ctrl *ctrl, const u32 entry_nr, const u16 mc_id, int tpt_fd);
 
 static inline void nvme_cdq_get(struct cdq_nvme_queue *cdq)
 {
@@ -97,5 +103,6 @@ static inline void nvme_cdq_put(struct cdq_nvme_queue *cdq)
 
 void nvme_delete_cdqs_host(struct nvme_ctrl *ctrl);
 void nvme_free_cdqs(struct nvme_ctrl *ctrl);
+int nvme_handle_cdq_aen_tpevent(struct nvme_ctrl *ctrl, u32 event_param);
 
 #endif /* _NVME_CDQ_H */
diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c
index fbaa2e065ab846e4700cddd7ae9f30a52e8a2cd9..bf6daeb0176fa69f1609280a8c7db89dadf43161 100644
--- a/drivers/nvme/host/core.c
+++ b/drivers/nvme/host/core.c
@@ -4765,7 +4765,8 @@ static bool nvme_handle_aen_oneshot(struct nvme_ctrl *ctrl, u32 result, u32 even
 
 	switch (aer_subtype) {
 	case NVME_AER_ONE_SHOT_CDQ_TAIL_PTR:
-		WARN_ONCE(1, "CDQ Tail Pointer one shot event ignored");
+		if (nvme_handle_cdq_aen_tpevent(ctrl, event_param))
+			WARN_ONCE(1, "Error handling CDQ AEN oneshot");
 		break;
 	case NVME_AER_ONE_SHOT_CDQ_FULL:
 		WARN_ONCE(1, "CDQ Full Error one shot event ignored");
diff --git a/include/linux/nvme.h b/include/linux/nvme.h
index 7cabedce022eda98fff789596682f257203e3b45..3c100fae1a4bd2a1f43fe3264680214194d2d262 100644
--- a/include/linux/nvme.h
+++ b/include/linux/nvme.h
@@ -1746,6 +1746,7 @@ enum {
 
 enum {
 	NVME_FEAT_CDQ_ID_MASK = GENMASK(15, 0),
+	NVME_FEAT_CDQ_ETPT_MASK = GENMASK(31, 31),
 };
 
 struct nvme_supported_log {

-- 
2.50.1

[PATCH RFC v2 14/14] nvme: Add NVME_IOCTL_CDQ to create/delete migration CDQs

[Joel Granados]

Controller Data Queue (CDQ) creation and deletion executed for the
migratable controller id defined in mc_id. When an eventfd file
descriptor is passed in tpt_fd, the AEN tail pointer trigger event is
routed as an eventfd signal. When tpt_fd is <= 0, the routing is turned
off.

Signed-off-by: Joel Granados <joel.granados@kernel.org>
---
 drivers/nvme/host/cdq.c         | 23 ++++++++++++++++++---
 drivers/nvme/host/cdq.h         |  4 +++-
 drivers/nvme/host/ioctl.c       | 44 ++++++++++++++++++++++++++++++++++++++++-
 include/uapi/linux/nvme_ioctl.h | 22 +++++++++++++++++++++
 4 files changed, 88 insertions(+), 5 deletions(-)

diff --git a/drivers/nvme/host/cdq.c b/drivers/nvme/host/cdq.c
index f4aabf0a8817058a11e6c30dcea9554076861327..ac10c8d48864615f53da232a547cd5da72630ed1 100644
--- a/drivers/nvme/host/cdq.c
+++ b/drivers/nvme/host/cdq.c
@@ -511,6 +511,22 @@ void nvme_delete_cdq(struct cdq_nvme_queue *cdq)
 }
 EXPORT_SYMBOL_GPL(nvme_delete_cdq);
 
+int nvme_delete_cdq_mcid(struct nvme_ctrl *ctrl, u16 mc_id)
+{
+	struct cdq_nvme_queue *cdq;
+	unsigned long i;
+
+	xa_for_each(&ctrl->cdqs, i, cdq) {
+		if (cdq->mc_id == mc_id) {
+			nvme_delete_cdq(cdq);
+			return 0;
+		}
+	}
+
+	return -ENOENT;
+}
+EXPORT_SYMBOL_GPL(nvme_delete_cdq_mcid);
+
 int nvme_handle_cdq_aen_tpevent(struct nvme_ctrl *ctrl, u32 event_param)
 {
 	u16 cdq_id = event_param & NVME_FEAT_CDQ_ID_MASK;
@@ -556,11 +572,12 @@ static int nvme_submit_create_cdq_cmd(struct cdq_nvme_queue *cdq)
 	return ret;
 }
 
-int nvme_create_cdq(struct nvme_ctrl *ctrl, const u32 entry_nr, const u16 mc_id, const int tpt_fd)
+int nvme_create_cdq(struct nvme_ctrl *ctrl, const u32 entry_nr, const u16 mc_id,
+		    const int tpt_fd, int *cdq_fd)
 {
 	u64 size_nbyte = (u64)entry_nr * NVME_CDQ_MQ_ENTRY_NRBYTES;
 	struct cdq_nvme_queue *cdq = NULL;
-	int ret, cdq_fd;
+	int ret;
 
 	/* The backing size and the CDQSIZE field are both u32 (bytes). */
 	if (size_nbyte > U32_MAX)
@@ -598,7 +615,7 @@ int nvme_create_cdq(struct nvme_ctrl *ctrl, const u32 entry_nr, const u16 mc_id,
 	if (ret)
 		goto del_cmd;
 
-	ret = nvme_create_cdqfd(cdq, &cdq_fd);
+	ret = nvme_create_cdqfd(cdq, cdq_fd);
 	if (ret)
 		goto del_xarray;
 
diff --git a/drivers/nvme/host/cdq.h b/drivers/nvme/host/cdq.h
index 2958c6b92b1312930414b8466690aa7c0eb57754..06a6224a6583dd2ec7463c62f9381e0edcb6718f 100644
--- a/drivers/nvme/host/cdq.h
+++ b/drivers/nvme/host/cdq.h
@@ -89,7 +89,9 @@ void nvme_free_cdq(struct kref *ref);
  *    - Calls nvme_free_cdq if there are no more refs
  */
 void nvme_delete_cdq(struct cdq_nvme_queue *cdq);
-int nvme_create_cdq(struct nvme_ctrl *ctrl, const u32 entry_nr, const u16 mc_id, int tpt_fd);
+int nvme_delete_cdq_mcid(struct nvme_ctrl *ctrl, u16 mc_id);
+int nvme_create_cdq(struct nvme_ctrl *ctrl, const u32 entry_nr, const u16 mc_id,
+		    int tpt_fd, int *cdq_fd);
 
 static inline void nvme_cdq_get(struct cdq_nvme_queue *cdq)
 {
diff --git a/drivers/nvme/host/ioctl.c b/drivers/nvme/host/ioctl.c
index 8844bbd395159e544218db413e066cae6c24b2f1..c289a220cb18b39bef4edbd0a8daaa4b46f8b4d9 100644
--- a/drivers/nvme/host/ioctl.c
+++ b/drivers/nvme/host/ioctl.c
@@ -8,6 +8,7 @@
 #include <linux/nvme_ioctl.h>
 #include <linux/io_uring/cmd.h>
 #include "nvme.h"
+#include "cdq.h"
 
 enum {
 	NVME_IOCTL_VEC		= (1 << 0),
@@ -538,9 +539,46 @@ static int nvme_uring_cmd_io(struct nvme_ctrl *ctrl, struct nvme_ns *ns,
 	return ret;
 }
 
+static int nvme_user_cdq(struct nvme_ctrl *ctrl, void __user *argp)
+{
+	struct nvme_cdq_cmd cmd;
+	u32 entry_nr;
+	int ret, cdq_fd;
+
+	if (copy_from_user(&cmd, argp, sizeof(cmd)))
+		return -EFAULT;
+
+	/* A zero size requests deletion of the CDQ created for this mc_id. */
+	if (cmd.size_nbyte == 0)
+		return nvme_delete_cdq_mcid(ctrl, cmd.mc_id);
+
+	if (cmd.size_nbyte % NVME_CDQ_MQ_ENTRY_NRBYTES)
+		return -EINVAL;
+
+	entry_nr = cmd.size_nbyte / NVME_CDQ_MQ_ENTRY_NRBYTES;
+
+	ret = nvme_create_cdq(ctrl, entry_nr, cmd.mc_id, cmd.tpt_fd, &cdq_fd);
+	if (ret)
+		goto err_out;
+
+	cmd.cdq_fd = cdq_fd;
+	if (copy_to_user(argp, &cmd, sizeof(cmd))) {
+		ret = -EFAULT;
+		goto err_out;
+	}
+
+	return 0;
+
+err_out:
+	nvme_delete_cdq_mcid(ctrl, cmd.mc_id);
+	return ret;
+
+}
+
 static bool is_ctrl_ioctl(unsigned int cmd)
 {
-	if (cmd == NVME_IOCTL_ADMIN_CMD || cmd == NVME_IOCTL_ADMIN64_CMD)
+	if (cmd == NVME_IOCTL_ADMIN_CMD || cmd == NVME_IOCTL_ADMIN64_CMD ||
+	    cmd == NVME_IOCTL_CDQ)
 		return true;
 	if (is_sed_ioctl(cmd))
 		return true;
@@ -555,6 +593,8 @@ static int nvme_ctrl_ioctl(struct nvme_ctrl *ctrl, unsigned int cmd,
 		return nvme_user_cmd(ctrl, NULL, argp, 0, open_for_write);
 	case NVME_IOCTL_ADMIN64_CMD:
 		return nvme_user_cmd64(ctrl, NULL, argp, 0, open_for_write);
+	case NVME_IOCTL_CDQ:
+		return nvme_user_cdq(ctrl, argp);
 	default:
 		return sed_ioctl(ctrl->opal_dev, cmd, argp);
 	}
@@ -873,6 +913,8 @@ long nvme_dev_ioctl(struct file *file, unsigned int cmd,
 			return -EACCES;
 		nvme_queue_scan(ctrl);
 		return 0;
+	case NVME_IOCTL_CDQ:
+		return nvme_user_cdq(ctrl, argp);
 	default:
 		return -ENOTTY;
 	}
diff --git a/include/uapi/linux/nvme_ioctl.h b/include/uapi/linux/nvme_ioctl.h
index 2f76cba6716637baff53e167a6141b68420d75c3..b3fbab48a903bfb10521ca44d08e61fd2b37bbbd 100644
--- a/include/uapi/linux/nvme_ioctl.h
+++ b/include/uapi/linux/nvme_ioctl.h
@@ -92,6 +92,27 @@ struct nvme_uring_cmd {
 	__u32   rsvd2;
 };
 
+struct nvme_cdq_cmd {
+	/* Migratable controller id (filled by user space). */
+	__u16	mc_id;
+
+	/*
+	 * CDQ size in bytes: (number of entries) * 32. Must be a multiple of
+	 * 32. A size of 0 requests deletion of the CDQ created for mc_id.
+	 */
+	__u32	size_nbyte;
+
+	/*
+	 * Tail Pointer Trigger eventfd file descriptor, passed when creating
+	 * the CDQ. A value <= 0 means there is no eventfd and AERs are not
+	 * forwarded.
+	 */
+	__s32	tpt_fd;
+
+	/* Returned by the kernel: the CDQ file descriptor. */
+	__s32	cdq_fd;
+};
+
 #define nvme_admin_cmd nvme_passthru_cmd
 
 #define NVME_IOCTL_ID		_IO('N', 0x40)
@@ -104,6 +125,7 @@ struct nvme_uring_cmd {
 #define NVME_IOCTL_ADMIN64_CMD	_IOWR('N', 0x47, struct nvme_passthru_cmd64)
 #define NVME_IOCTL_IO64_CMD	_IOWR('N', 0x48, struct nvme_passthru_cmd64)
 #define NVME_IOCTL_IO64_CMD_VEC	_IOWR('N', 0x49, struct nvme_passthru_cmd64)
+#define NVME_IOCTL_CDQ		_IOWR('N', 0x50, struct nvme_cdq_cmd)
 
 /* io_uring async commands: */
 #define NVME_URING_CMD_IO	_IOWR('N', 0x80, struct nvme_uring_cmd)

-- 
2.50.1

Re: [PATCH RFC v2 00/14] nvme: Controller Data Queue (CDQ) support

[Chaitanya Kulkarni]

On 7/24/26 4:10 AM, Joel Granados wrote:
> NVMe LM SW WG
> =============
> The NVMe Live Migration SoftWare Working Group (mentioned in [6]) was
> one of the main feedback points in my previous RFC. My main goal in this
> respect is to keep exploring different approaches in the hope that when
> it materializes we can "hit the ground running".


What does this mean ? -Chaitanya

Re: [PATCH RFC v2 00/14] nvme: Controller Data Queue (CDQ) support

[Joel Granados]

[-- Attachment #1: Type: text/plain, Size: 681 bytes --]

On Wed, Jul 29, 2026 at 11:48:43PM +0000, Chaitanya Kulkarni wrote:

> On 7/24/26 4:10 AM, Joel Granados wrote:
> > NVMe LM SW WG
> > =============
> > The NVMe Live Migration SoftWare Working Group (mentioned in [6]) was
> > one of the main feedback points in my previous RFC. My main goal in this
> > respect is to keep exploring different approaches in the hope that when
> > it materializes we can "hit the ground running".
> 
> 
> What does this mean ? -Chaitanya

With this I meant that the pre-discussions that are possible before the
Working Group materializes might serve as information that feeds the
Working Group itself.

Best

-- 

Joel Granados

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 659 bytes --]