[PATCH 00/11] Subject: [PATCH for-next 00/11] RDMA/cxi: add the HPE Cassini verbs driver

[PATCH 00/11] Subject: [PATCH for-next 00/11] RDMA/cxi: add the HPE Cassini verbs driver

[Doug Ledford]

This series adds cxiverbs, an RDMA driver for the HPE Slingshot Cassini
NIC. It exposes the NIC through the verbs API using the UET reliable
unconnected (RU) queue pair type from the UET RDMA core series.

cxiverbs does not drive the hardware itself. It is an auxiliary bus
client of the cxi-ss1 Cassini core driver and does everything through
that driver's exported CXI interface.

Patches 1-3 extend the RDMA core:

 - read selected counters, with per-item timestamps, through the
   COUNTERS object, so that a monitor can read a few of the NIC's
   thousands of telemetry values
 - allow UET jobs without an address table. Cassini transmit commands
   carry the destination fabric address directly and the hardware has
   no programmable peer-address table equivalent to the UET job table.
   This relaxes a rule in the UET core series and needs agreement from
   its authors; the commit message gives the hardware reasons.
 - add the Slingshot node type, transport, port capability and a
   driver-owned GID type that carries the 20-bit NID of the port

Patches 4-11 add the driver:

 - the user space ABI is built around explicit resource ownership:
   the CXI service, LNI (logical network interface) and PID are
   allocated explicitly through private uverbs methods instead of as a
   side effect of opening the device
 - VNI-isolated RU queue pairs, native memory registration, services,
   jobs, communication profiles, command and event queues, counting
   triggers, and NIC telemetry read through the counters extension

Dependencies
------------

This series depends on:

 - the Cassini NIC driver series for net-next,
   net: hpe: add HPE Slingshot NIC drivers
   https://lore.kernel.org/netdev/20261030-cxi-netdev-submission-v1-0-8c2c391cc283@hpe.com/
 - the UET RDMA core series
   RDMA/core: Ultra Ethernet transport verbs
   https://lore.kernel.org/linux-rdma/20261019-uet-ib-core-v2-0-8bd654f25370@broadcom.com/T/#t

The upcoming cxiverbs rdma-core provider pull request will depend on
this series.

Known open issues
-----------------

Every patch with a known unresolved issue says so in its commit
message. In summary:

 - SR-IOV: patches 9 and 10 carry virtual-function code from an older,
   incomplete version of the Cassini SR-IOV support. V2 will include the
   updated SR-IOV implementation, so please ignore issues in the VF
   paths for now.
 - Patch 9: on a physical function cxi_eq_free() cannot fail. On a
   virtual function it fails if the PF mailbox is unreachable. If that
   happens while the file is closed or the device is removed, the RDMA
   core drops the EQ object without native cleanup, so the VF EQ and its
   references leak. Explicit EQ destroys are retried. This will be fixed
   with the SR-IOV update in V2.

checkpatch
----------

Patches 1-3, 10 and 11 are checkpatch --strict clean. Patches 4-9 still
report CHECK-level items only (157 in total, mostly
PARENTHESIS_ALIGNMENT and OPEN_ENDED_LINE); each patch lists them. This
code is also maintained in internal HPE repositories, so these fixes
need coordination with those repositories and will be made in V2.

Review and testing
------------------

Before posting, the series was reviewed with the Sashiko AI code
review tool and checkpatch. All valid findings are fixed, except the
open items listed above.

Tool use: GitHub Copilot, an AI coding assistant, was used under our
direction to write many of the review fixes, to fold them into these
patches, to write commit messages and to run the build and hardware
tests. Every change was reviewed by the submitter, who takes
responsibility for it as for any other patch.

 - every patch builds incrementally, without compiler warnings, on top
   of the two series it depends on, with an x86_64 configuration that
   enables the driver
 - with this series, the netdev series and the rdma-core provider,
   booted on two Cassini 2 x86_64 nodes, the complete stack
   passed a full test matrix on both nodes: the rdma-core Python suite
   (434 tests; tests that need features this device lacks are
   skipped), resource and VNI isolation tests in default and indexed
   addressing modes, including wrong-VNI rejection, privileged and
   unprivileged CSR access tests, and 36 two-node RU ping-pong and RMA
   traffic cases (polling and event modes) in each direction, 72 in
   total. Native NIC resource accounting was identical before and
   after the traffic, no verbs references remained, and the kernel
   logs had no WARN, BUG or oops reports and no module load errors.
   The only later change is in the netdev series (cxi-eth hardware
   timestamp configuration), which was tested separately on hardware.
 - targeted user space models of the functions changed by the review
   fixes pass under AddressSanitizer and UndefinedBehaviorSanitizer.

Co-developed by Mamila Santosh and Iraimani P, as shown by the trailers
on each patch.

Doug Ledford (11):
  RDMA/core: read selected counters with timestamps
  RDMA/core: allow jobs without an address table
  RDMA/core: add Slingshot identifiers and driver-owned GIDs
  RDMA/cxi: add the native ABI and resource definitions
  RDMA/cxi: add the auxiliary device and driver ops
  RDMA/cxi: add explicit context LNI and PID lifetimes
  RDMA/cxi: add VNI-isolated queue pairs
  RDMA/cxi: add native memory registration
  RDMA/cxi: add services jobs profiles queues and counting triggers
  RDMA/cxi: read the NIC telemetry through counters
  RDMA/cxi: wire the driver into the build

 25 files changed, 5696 insertions(+), 7 deletions(-)

Signed-off-by: Doug Ledford <doug.ledford@hpe.com>
---
Doug Ledford (11):
      RDMA/core: read selected counters with timestamps
      RDMA/core: allow jobs without an address table
      RDMA/core: add Slingshot identifiers and driver-owned GIDs
      RDMA/cxi: add the native ABI and resource definitions
      RDMA/cxi: add the auxiliary device and driver ops
      RDMA/cxi: add explicit context LNI and PID lifetimes
      RDMA/cxi: add VNI-isolated queue pairs
      RDMA/cxi: add native memory registration
      RDMA/cxi: add services jobs profiles queues and counting triggers
      RDMA/cxi: read the NIC telemetry through counters
      RDMA/cxi: wire the driver into the build

 MAINTAINERS                                        |    9 +-
 drivers/infiniband/Kconfig                         |    1 +
 drivers/infiniband/core/cache.c                    |    3 +
 drivers/infiniband/core/device.c                   |    1 +
 drivers/infiniband/core/nldev.c                    |    3 +
 drivers/infiniband/core/roce_gid_mgmt.c            |    3 +
 drivers/infiniband/core/sysfs.c                    |    2 +
 .../infiniband/core/uverbs_std_types_counters.c    |   68 +-
 drivers/infiniband/core/uverbs_std_types_job.c     |    3 +-
 drivers/infiniband/core/verbs.c                    |    2 +
 drivers/infiniband/hw/Makefile                     |    1 +
 drivers/infiniband/hw/cxi/Kconfig                  |   11 +
 drivers/infiniband/hw/cxi/Makefile                 |    5 +
 drivers/infiniband/hw/cxi/cxiverbs.h               |  443 +++++
 drivers/infiniband/hw/cxi/cxiverbs_counters.c      |   38 +
 drivers/infiniband/hw/cxi/cxiverbs_ct.c            | 1906 ++++++++++++++++++++
 drivers/infiniband/hw/cxi/cxiverbs_drv.c           |  270 +++
 drivers/infiniband/hw/cxi/cxiverbs_main.c          | 1105 ++++++++++++
 drivers/infiniband/hw/cxi/cxiverbs_mr.c            |  192 ++
 drivers/infiniband/hw/cxi/cxiverbs_qp.c            |  949 ++++++++++
 include/rdma/ib_verbs.h                            |   25 +
 include/uapi/rdma/cxiverbs-abi.h                   |  658 +++++++
 include/uapi/rdma/ib_user_ioctl_cmds.h             |    2 +
 include/uapi/rdma/ib_user_ioctl_verbs.h            |    2 +
 include/uapi/rdma/rdma_netlink.h                   |    1 +
 25 files changed, 5696 insertions(+), 7 deletions(-)
---
base-commit: 306a2f8432f516069b2e673de7a07043132c034f
change-id: 20261030-cxi-for-next-8f50ba2bd59f

Best regards,
--  
Doug Ledford <doug.ledford@hpe.com>

[PATCH 01/11] RDMA/core: read selected counters with timestamps

[Doug Ledford]

The COUNTERS object reads every counter attached to it into one
array, which suits a fixed set but not a device with thousands of
status values of which a monitor wants a few.  Add two optional
attributes to the read method.  ITEMS is an array of driver defined
item numbers, and the values come back in the same order.  TIMESTAMPS
returns, per item, when the driver sampled it, in nanoseconds since the
epoch.  Either may be used without the other.

A driver advertises the extension by providing read_counters_ex().  A
counters object created for such a driver need not have any counters
attached, because the items are chosen at read time; the check for an
attached counter is skipped for those drivers.  Without the callback
the new attributes are rejected with EOPNOTSUPP, and the original
behaviour is unchanged.

Signed-off-by: Doug Ledford <doug.ledford@hpe.com>
---
 drivers/infiniband/core/device.c                   |  1 +
 .../infiniband/core/uverbs_std_types_counters.c    | 68 ++++++++++++++++++++--
 include/rdma/ib_verbs.h                            | 11 ++++
 include/uapi/rdma/ib_user_ioctl_cmds.h             |  2 +
 4 files changed, 78 insertions(+), 4 deletions(-)

diff --git a/drivers/infiniband/core/device.c b/drivers/infiniband/core/device.c
index 1b838bf3600d..18c0d9243909 100644
--- a/drivers/infiniband/core/device.c
+++ b/drivers/infiniband/core/device.c
@@ -3026,6 +3026,7 @@ void ib_set_device_ops(struct ib_device *dev, const struct ib_device_ops *ops)
 	SET_DEVICE_OP(dev_ops, query_ucontext);
 	SET_DEVICE_OP(dev_ops, rdma_netdev_get_params);
 	SET_DEVICE_OP(dev_ops, read_counters);
+	SET_DEVICE_OP(dev_ops, read_counters_ex);
 	SET_DEVICE_OP(dev_ops, read_comp_cntr);
 	SET_DEVICE_OP(dev_ops, reg_dm_mr);
 	SET_DEVICE_OP(dev_ops, reg_user_mr);
diff --git a/drivers/infiniband/core/uverbs_std_types_counters.c b/drivers/infiniband/core/uverbs_std_types_counters.c
index 381aa5797641..e452854ddaa9 100644
--- a/drivers/infiniband/core/uverbs_std_types_counters.c
+++ b/drivers/infiniband/core/uverbs_std_types_counters.c
@@ -92,12 +92,20 @@ static int UVERBS_HANDLER(UVERBS_METHOD_COUNTERS_READ)(
 	const struct uverbs_attr *uattr;
 	struct ib_counters *counters =
 		uverbs_attr_get_obj(attrs, UVERBS_ATTR_READ_COUNTERS_HANDLE);
+	bool has_items, has_timestamps;
+	size_t len;
 	int ret;
 
-	if (!counters->device->ops.read_counters)
+	/*
+	 * A driver with read_counters_ex defines what its counters are, so
+	 * they need not be attached to anything before they are read.
+	 */
+	if (!counters->device->ops.read_counters &&
+	    !counters->device->ops.read_counters_ex)
 		return -EOPNOTSUPP;
 
-	if (!atomic_read(&counters->usecnt))
+	if (!counters->device->ops.read_counters_ex &&
+	    !atomic_read(&counters->usecnt))
 		return -EINVAL;
 
 	ret = uverbs_get_flags32(&read_attr.flags, attrs,
@@ -115,10 +123,56 @@ static int UVERBS_HANDLER(UVERBS_METHOD_COUNTERS_READ)(
 	if (IS_ERR(read_attr.counters_buff))
 		return PTR_ERR(read_attr.counters_buff);
 
-	ret = counters->device->ops.read_counters(counters, &read_attr, attrs);
+	has_items = uverbs_attr_is_valid(attrs, UVERBS_ATTR_READ_COUNTERS_ITEMS);
+	has_timestamps = uverbs_attr_is_valid(attrs,
+					      UVERBS_ATTR_READ_COUNTERS_TIMESTAMPS);
+	if ((has_items || has_timestamps) &&
+	    !counters->device->ops.read_counters_ex)
+		return -EOPNOTSUPP;
+
+	if (has_items) {
+		u64 *items;
+
+		len = uverbs_attr_get_len(attrs, UVERBS_ATTR_READ_COUNTERS_ITEMS);
+		if (len != array_size(read_attr.ncounters, sizeof(u64)))
+			return -EINVAL;
+		items = uverbs_zalloc(attrs, len);
+		if (IS_ERR(items))
+			return PTR_ERR(items);
+		ret = _uverbs_copy_from(items, attrs,
+					UVERBS_ATTR_READ_COUNTERS_ITEMS, len);
+		if (ret)
+			return ret;
+		read_attr.items = items;
+	}
+
+	if (has_timestamps) {
+		len = uverbs_attr_get_len(attrs,
+					  UVERBS_ATTR_READ_COUNTERS_TIMESTAMPS);
+		if (len != array_size(read_attr.ncounters, sizeof(u64)))
+			return -EINVAL;
+		read_attr.timestamps = uverbs_zalloc(attrs, len);
+		if (IS_ERR(read_attr.timestamps))
+			return PTR_ERR(read_attr.timestamps);
+	}
+
+	if (counters->device->ops.read_counters_ex)
+		ret = counters->device->ops.read_counters_ex(counters,
+							     &read_attr, attrs);
+	else
+		ret = counters->device->ops.read_counters(counters, &read_attr,
+							  attrs);
 	if (ret)
 		return ret;
 
+	if (has_timestamps) {
+		ret = uverbs_copy_to(attrs, UVERBS_ATTR_READ_COUNTERS_TIMESTAMPS,
+				     read_attr.timestamps,
+				     read_attr.ncounters * sizeof(u64));
+		if (ret)
+			return ret;
+	}
+
 	return uverbs_copy_to(attrs, UVERBS_ATTR_READ_COUNTERS_BUFF,
 			      read_attr.counters_buff,
 			      read_attr.ncounters * sizeof(u64));
@@ -148,7 +202,13 @@ DECLARE_UVERBS_NAMED_METHOD(
 			    UVERBS_ATTR_MIN_SIZE(0),
 			    UA_MANDATORY),
 	UVERBS_ATTR_FLAGS_IN(UVERBS_ATTR_READ_COUNTERS_FLAGS,
-			     enum ib_uverbs_read_counters_flags));
+			     enum ib_uverbs_read_counters_flags),
+	UVERBS_ATTR_PTR_IN(UVERBS_ATTR_READ_COUNTERS_ITEMS,
+			   UVERBS_ATTR_MIN_SIZE(sizeof(u64)),
+			   UA_OPTIONAL),
+	UVERBS_ATTR_PTR_OUT(UVERBS_ATTR_READ_COUNTERS_TIMESTAMPS,
+			    UVERBS_ATTR_MIN_SIZE(sizeof(u64)),
+			    UA_OPTIONAL));
 
 DECLARE_UVERBS_NAMED_OBJECT(UVERBS_OBJECT_COUNTERS,
 			    UVERBS_TYPE_ALLOC_IDR(uverbs_free_counters),
diff --git a/include/rdma/ib_verbs.h b/include/rdma/ib_verbs.h
index a93715323ad5..5aa1fa576a44 100644
--- a/include/rdma/ib_verbs.h
+++ b/include/rdma/ib_verbs.h
@@ -2595,6 +2595,10 @@ struct ib_counters_read_attr {
 	u64	*counters_buff;
 	u32	ncounters;
 	u32	flags; /* use enum ib_read_counters_flags */
+	/* Optional: counter numbers to read, ncounters entries */
+	const u64 *items;
+	/* Optional: filled with a timestamp for each counter read */
+	u64	*timestamps;
 };
 
 struct uverbs_attr_bundle;
@@ -2936,6 +2940,13 @@ struct ib_device_ops {
 	int (*create_counters)(struct ib_counters *counters,
 			       struct uverbs_attr_bundle *attrs);
 	int (*destroy_counters)(struct ib_counters *counters);
+	/*
+	 * Used instead of read_counters when the user selects counters with
+	 * items or asks for timestamps.
+	 */
+	int (*read_counters_ex)(struct ib_counters *counters,
+				struct ib_counters_read_attr *counters_read_attr,
+				struct uverbs_attr_bundle *attrs);
 	int (*read_counters)(struct ib_counters *counters,
 			     struct ib_counters_read_attr *counters_read_attr,
 			     struct uverbs_attr_bundle *attrs);
diff --git a/include/uapi/rdma/ib_user_ioctl_cmds.h b/include/uapi/rdma/ib_user_ioctl_cmds.h
index 8de110b8db34..9ee083eed044 100644
--- a/include/uapi/rdma/ib_user_ioctl_cmds.h
+++ b/include/uapi/rdma/ib_user_ioctl_cmds.h
@@ -404,6 +404,8 @@ enum uverbs_attrs_read_counters_cmd_attr_ids {
 	UVERBS_ATTR_READ_COUNTERS_HANDLE,
 	UVERBS_ATTR_READ_COUNTERS_BUFF,
 	UVERBS_ATTR_READ_COUNTERS_FLAGS,
+	UVERBS_ATTR_READ_COUNTERS_ITEMS,
+	UVERBS_ATTR_READ_COUNTERS_TIMESTAMPS,
 };
 
 enum uverbs_methods_actions_counters_ops {

-- 
2.56.0

[PATCH 02/11] RDMA/core: allow jobs without an address table

[Doug Ledford]

max_addr_entries counts slots for remote fabric peers in a job's address
table. An entry describes a destination using address-handle attributes
and a remote QPN; it is not a memory region, mapped page, or ATU address
context.

Cassini transmit commands carry the destination fabric address directly,
including the destination NID and endpoint information. The CXI provider
currently uses address handles and does not implement job address-table
operations or indexed sends. The documented transmit interface does not
expose a programmable peer-address table equivalent to the UET job
table. Cassini's connection/retry state tables are not such an address
vector. Its virtual-memory mappings instead use separate ATU address
contexts and page tables.

Allow an address-handle-only job to request zero peer-table entries
instead of requiring an unused entry. Keep the device upper-bound and
privileged job-allocation checks. JobIDs and job keys remain in use.
Existing index bounds reject every access to a zero-sized job
table. This does not implement indexed addressing for CXI or relax its
memory-mapping resource limits.

This relaxes the shared UET job-allocation contract for providers using
only address handles, not just the CXI driver. It needs core API review.

Signed-off-by: Doug Ledford <doug.ledford@hpe.com>
---
 drivers/infiniband/core/uverbs_std_types_job.c | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)

diff --git a/drivers/infiniband/core/uverbs_std_types_job.c b/drivers/infiniband/core/uverbs_std_types_job.c
index 20acb2aa9871..b05c0619a6b3 100644
--- a/drivers/infiniband/core/uverbs_std_types_job.c
+++ b/drivers/infiniband/core/uverbs_std_types_job.c
@@ -193,8 +193,7 @@ static int UVERBS_HANDLER(UVERBS_METHOD_JOB_ALLOC)(
 		return ret;
 
 	/* the table is sized once, every index is checked against it */
-	if (!attr.max_addr_entries ||
-	    attr.max_addr_entries > ib_dev->attrs.max_addr_entries)
+	if (attr.max_addr_entries > ib_dev->attrs.max_addr_entries)
 		return -EINVAL;
 
 	/* the port carries the job's traffic, so it has to be one we have */

-- 
2.56.0

[PATCH 03/11] RDMA/core: add Slingshot identifiers and driver-owned GIDs

[Doug Ledford]

Add the identifiers the CXI RDMA driver needs from the core: the
RDMA_NODE_SLINGSHOT node type and RDMA_TRANSPORT_SLINGSHOT transport,
the RDMA_CORE_CAP_PROT_SLINGSHOT port capability with the matching
rdma_protocol_slingshot() helper, and the IB_GID_TYPE_SLINGSHOT_UFH
GID type.  Names are added to the sysfs, GID cache and node/transport
lookup tables so the new values are reported like any other.

A Slingshot GID carries only the 20-bit NID of the port, so the port
exposes a single GID type and does not take part in the RoCE style
netdev GID population.  RDMA_CORE_PORT_SLINGSHOT combines the new
capability with RDMA_CORE_CAP_ETH_AH.

Signed-off-by: Doug Ledford <doug.ledford@hpe.com>
---
 drivers/infiniband/core/cache.c         |  3 +++
 drivers/infiniband/core/nldev.c         |  3 +++
 drivers/infiniband/core/roce_gid_mgmt.c |  3 +++
 drivers/infiniband/core/sysfs.c         |  2 ++
 drivers/infiniband/core/verbs.c         |  2 ++
 include/rdma/ib_verbs.h                 | 14 ++++++++++++++
 include/uapi/rdma/ib_user_ioctl_verbs.h |  2 ++
 include/uapi/rdma/rdma_netlink.h        |  1 +
 8 files changed, 30 insertions(+)

diff --git a/drivers/infiniband/core/cache.c b/drivers/infiniband/core/cache.c
index d5e17a36f186..b407e5e25dbc 100644
--- a/drivers/infiniband/core/cache.c
+++ b/drivers/infiniband/core/cache.c
@@ -141,6 +141,7 @@ static const char * const gid_type_str[] = {
 	[IB_GID_TYPE_ROCE_UDP_ENCAP]	= "RoCE v2",
 	[IB_GID_TYPE_UET_UDP]		= "UET UDP",
 	[IB_GID_TYPE_UET_IP]		= "UET IP",
+	[IB_GID_TYPE_SLINGSHOT_UFH] = "SLINGSHOT UFH",
 };
 
 const char *ib_cache_gid_type_str(enum ib_gid_type gid_type)
@@ -1404,6 +1405,8 @@ static int config_non_roce_gid_cache(struct ib_device *device,
 
 	gid_attr.device = device;
 	gid_attr.port_num = port;
+	if (rdma_protocol_slingshot(device, port))
+		gid_attr.gid_type = IB_GID_TYPE_SLINGSHOT_UFH;
 	table = rdma_gid_table(device, port);
 
 	mutex_lock(&table->lock);
diff --git a/drivers/infiniband/core/nldev.c b/drivers/infiniband/core/nldev.c
index 5313eb4849c7..97ee2f5118d6 100644
--- a/drivers/infiniband/core/nldev.c
+++ b/drivers/infiniband/core/nldev.c
@@ -350,6 +350,9 @@ static int fill_dev_info(struct sk_buff *msg, struct ib_device *device)
 		ret = nla_put_string(msg, RDMA_NLDEV_ATTR_DEV_PROTOCOL, "iw");
 	else if (rdma_protocol_roce(device, port))
 		ret = nla_put_string(msg, RDMA_NLDEV_ATTR_DEV_PROTOCOL, "roce");
+	else if (rdma_protocol_slingshot(device, port))
+		ret = nla_put_string(msg, RDMA_NLDEV_ATTR_DEV_PROTOCOL,
+				     "slingshot");
 	else if (rdma_protocol_uet(device, port))
 		ret = nla_put_string(msg, RDMA_NLDEV_ATTR_DEV_PROTOCOL, "uet");
 	else if (rdma_protocol_usnic(device, port))
diff --git a/drivers/infiniband/core/roce_gid_mgmt.c b/drivers/infiniband/core/roce_gid_mgmt.c
index 214af921341a..bcdbb9e45107 100644
--- a/drivers/infiniband/core/roce_gid_mgmt.c
+++ b/drivers/infiniband/core/roce_gid_mgmt.c
@@ -77,6 +77,7 @@ static const struct {
 	{rdma_protocol_roce_udp_encap, IB_GID_TYPE_ROCE_UDP_ENCAP},
 	{rdma_protocol_uet_udp, IB_GID_TYPE_UET_UDP},
 	{rdma_protocol_uet_ip, IB_GID_TYPE_UET_IP},
+	{rdma_protocol_slingshot, IB_GID_TYPE_SLINGSHOT_UFH},
 };
 
 #define CAP_TO_GID_TABLE_SIZE	ARRAY_SIZE(PORT_CAP_TO_GID_TYPE)
@@ -86,6 +87,8 @@ unsigned long roce_gid_type_mask_support(struct ib_device *ib_dev, u32 port)
 	int i;
 	unsigned int ret_flags = 0;
 
+	if (rdma_protocol_slingshot(ib_dev, port))
+		return 1UL << IB_GID_TYPE_SLINGSHOT_UFH;
 	if (!rdma_port_gids_from_netdev(ib_dev, port))
 		return 1UL << IB_GID_TYPE_IB;
 
diff --git a/drivers/infiniband/core/sysfs.c b/drivers/infiniband/core/sysfs.c
index 4cd861b5d953..5121a622f822 100644
--- a/drivers/infiniband/core/sysfs.c
+++ b/drivers/infiniband/core/sysfs.c
@@ -1262,6 +1262,8 @@ static const char *node_type_string(int node_type)
 		return "usNIC UDP";
 	case RDMA_NODE_UNSPECIFIED:
 		return "unspecified";
+	case RDMA_NODE_SLINGSHOT:
+		return "Slingshot";
 	}
 	return "<unknown>";
 }
diff --git a/drivers/infiniband/core/verbs.c b/drivers/infiniband/core/verbs.c
index adcda33b04f2..77d1d71d7167 100644
--- a/drivers/infiniband/core/verbs.c
+++ b/drivers/infiniband/core/verbs.c
@@ -282,6 +282,8 @@ rdma_node_get_transport(unsigned int node_type)
 		return RDMA_TRANSPORT_IWARP;
 	if (node_type == RDMA_NODE_UNSPECIFIED)
 		return RDMA_TRANSPORT_UNSPECIFIED;
+	if (node_type == RDMA_NODE_SLINGSHOT)
+		return RDMA_TRANSPORT_SLINGSHOT;
 
 	return RDMA_TRANSPORT_IB;
 }
diff --git a/include/rdma/ib_verbs.h b/include/rdma/ib_verbs.h
index 5aa1fa576a44..c26d7e562914 100644
--- a/include/rdma/ib_verbs.h
+++ b/include/rdma/ib_verbs.h
@@ -147,6 +147,7 @@ enum ib_gid_type {
 	IB_GID_TYPE_ROCE_UDP_ENCAP = IB_UVERBS_GID_TYPE_ROCE_V2,
 	IB_GID_TYPE_UET_UDP = IB_UVERBS_GID_TYPE_UET_UDP,
 	IB_GID_TYPE_UET_IP = IB_UVERBS_GID_TYPE_UET_IP,
+	IB_GID_TYPE_SLINGSHOT_UFH = IB_UVERBS_GID_TYPE_SLINGSHOT_UFH,
 	IB_GID_TYPE_SIZE
 };
 
@@ -171,6 +172,7 @@ enum rdma_transport_type {
 	RDMA_TRANSPORT_USNIC,
 	RDMA_TRANSPORT_USNIC_UDP,
 	RDMA_TRANSPORT_UNSPECIFIED,
+	RDMA_TRANSPORT_SLINGSHOT,
 };
 
 enum rdma_protocol_type {
@@ -682,6 +684,7 @@ void rdma_free_hw_stats_struct(struct rdma_hw_stats *stats);
 #define RDMA_CORE_CAP_PROT_USNIC        0x02000000
 #define RDMA_CORE_CAP_PROT_UET_UDP      0x04000000
 #define RDMA_CORE_CAP_PROT_UET_IP       0x08000000
+#define RDMA_CORE_CAP_PROT_SLINGSHOT    0x20000000
 
 #define RDMA_CORE_CAP_PROT_UET         (RDMA_CORE_CAP_PROT_UET_UDP \
 					| RDMA_CORE_CAP_PROT_UET_IP)
@@ -722,6 +725,8 @@ void rdma_free_hw_stats_struct(struct rdma_hw_stats *stats);
 
 #define RDMA_CORE_PORT_USNIC		(RDMA_CORE_CAP_PROT_USNIC)
 
+#define RDMA_CORE_PORT_SLINGSHOT        (RDMA_CORE_CAP_PROT_SLINGSHOT | RDMA_CORE_CAP_ETH_AH)
+
 struct ib_port_attr {
 	u64			subnet_prefix;
 	enum ib_port_state	state;
@@ -3596,6 +3601,13 @@ static inline bool rdma_protocol_uet_ip(const struct ib_device *device,
 	       RDMA_CORE_CAP_PROT_UET_IP;
 }
 
+static inline bool rdma_protocol_slingshot(const struct ib_device *device,
+					   u32 port_num)
+{
+	return device->port_data[port_num].immutable.core_cap_flags &
+		RDMA_CORE_CAP_PROT_SLINGSHOT;
+}
+
 static inline bool rdma_port_gids_from_netdev(const struct ib_device *device,
 					      u32 port_num)
 {
@@ -5219,6 +5231,8 @@ void rdma_move_ah_attr(struct rdma_ah_attr *dest, struct rdma_ah_attr *src);
 static inline enum rdma_ah_attr_type rdma_ah_find_type(struct ib_device *dev,
 						       u32 port_num)
 {
+	if (rdma_protocol_slingshot(dev, port_num))
+		return RDMA_AH_ATTR_TYPE_IB;
 	if (rdma_protocol_roce(dev, port_num) ||
 	    rdma_protocol_uet(dev, port_num))
 		return RDMA_AH_ATTR_TYPE_ROCE;
diff --git a/include/uapi/rdma/ib_user_ioctl_verbs.h b/include/uapi/rdma/ib_user_ioctl_verbs.h
index 75cdfdbdb2f7..e8e84aa034ef 100644
--- a/include/uapi/rdma/ib_user_ioctl_verbs.h
+++ b/include/uapi/rdma/ib_user_ioctl_verbs.h
@@ -258,6 +258,7 @@ enum rdma_driver_id {
 	RDMA_DRIVER_ERDMA,
 	RDMA_DRIVER_MANA,
 	RDMA_DRIVER_IONIC,
+	RDMA_DRIVER_CXI,
 };
 
 enum ib_uverbs_gid_type {
@@ -266,6 +267,7 @@ enum ib_uverbs_gid_type {
 	IB_UVERBS_GID_TYPE_ROCE_V2,
 	IB_UVERBS_GID_TYPE_UET_UDP,
 	IB_UVERBS_GID_TYPE_UET_IP,
+	IB_UVERBS_GID_TYPE_SLINGSHOT_UFH,
 };
 
 struct ib_uverbs_gid_entry {
diff --git a/include/uapi/rdma/rdma_netlink.h b/include/uapi/rdma/rdma_netlink.h
index ee11c3bbbae2..69ff1dee2db6 100644
--- a/include/uapi/rdma/rdma_netlink.h
+++ b/include/uapi/rdma/rdma_netlink.h
@@ -158,6 +158,7 @@ enum {
 	RDMA_NODE_USNIC,
 	RDMA_NODE_USNIC_UDP,
 	RDMA_NODE_UNSPECIFIED,
+	RDMA_NODE_SLINGSHOT,
 };
 
 /*

-- 
2.56.0

[PATCH 04/11] RDMA/cxi: add the native ABI and resource definitions

[Doug Ledford]

Start the CXI RDMA (cxiverbs) driver, which exposes a Cassini NIC
through the verbs API using the UET reliable unconnected (RU) queue
pair type, and add its private structures and its user space ABI.

The driver sits on top of the cxi-ss1 core driver as an auxiliary bus
client.  It does not drive the hardware directly: it calls the core
driver's exported CXI interface for everything.  cxiverbs.h holds
the per-device, per-context and per-object state; cxiverbs-abi.h is
the ABI shared with the cxiverbs provider in rdma-core.

The ABI is built around explicit resource ownership instead of a
single implicit context:

 - the ucontext is bare; the CXI service, LNI (logical network
   interface) and PID are allocated explicitly through private uverbs
   methods and are not created as a side effect of opening the device
 - the PID is device-wide, assigned by the driver and bound to a PD
 - the application's service index is carried in the resource index
   bits of the QPN rather than in a core object

QPN layout (UEC-aligned): bit 31 selects absolute (1) or FEP-relative
(0) addressing, bits 23:12 are the resource index and bits 11:0 the
PIDonFEP.  This driver supports only job-relative QPs for now; requests
for an absolute PID or an absolute source QPN are rejected.

Kernel-generated remote keys are limited to 48 bits and always have
bit 31 set.  The ABI version is 1 and has not been published.

The Kconfig and Makefile hookup for this driver is the last patch of
the series.

checkpatch --strict (100 columns) still reports the following in this
patch. They need coordination with the internal repositories this code
is maintained in and will be fixed in V2:
- CHECK: PARENTHESIS_ALIGNMENT (18), UNCOMMENTED_DEFINITION (5)

Co-developed-by: Iraimani P <iraimani.pavadai@hpe.com>
Signed-off-by: Iraimani P <iraimani.pavadai@hpe.com>
Co-developed-by: Mamila Santosh <santosh-kumar.mamila@hpe.com>
Signed-off-by: Mamila Santosh <santosh-kumar.mamila@hpe.com>
Signed-off-by: Doug Ledford <doug.ledford@hpe.com>
---
 drivers/infiniband/hw/cxi/cxiverbs.h | 443 +++++++++++++++++++++++
 include/uapi/rdma/cxiverbs-abi.h     | 658 +++++++++++++++++++++++++++++++++++
 2 files changed, 1101 insertions(+)

diff --git a/drivers/infiniband/hw/cxi/cxiverbs.h b/drivers/infiniband/hw/cxi/cxiverbs.h
new file mode 100644
index 000000000000..11a8d218d879
--- /dev/null
+++ b/drivers/infiniband/hw/cxi/cxiverbs.h
@@ -0,0 +1,443 @@
+/* SPDX-License-Identifier: GPL-2.0 OR BSD-2-Clause */
+/*
+ * Copyright 2018-2025 Hewlett Packard Enterprise Development LP
+ *
+ * Internal header for the CXI InfiniBand RDMA driver.
+ */
+
+#ifndef _CXI_IB_H_
+#define _CXI_IB_H_
+
+#include <linux/bitfield.h>
+#include <linux/bits.h>
+#include <linux/list.h>
+#include <linux/mm_types.h>
+#include <linux/mmu_notifier.h>
+#include <linux/mutex.h>
+#include <linux/pci.h>
+#include <linux/slab.h>
+#include <linux/spinlock.h>
+#include <linux/random.h>
+#include <linux/refcount.h>
+#include <linux/xarray.h>
+
+#include <rdma/ib_verbs.h>
+#include <rdma/ib_umem.h>
+#include <rdma/uverbs_ioctl.h>
+
+#include <linux/auxiliary_bus.h>
+#include <linux/mod_devicetable.h>
+#include <linux/hpe/cxi/cxi.h>
+#include <linux/hpe/cxi/driver.h>
+#include <linux/hpe/cxi/cxi_prov_hw.h>
+#include <linux/idr.h>
+
+#include <uapi/rdma/cxiverbs-abi.h>
+
+/*
+ * Default CXI service ID.  Service 1 is the system default created by
+ * the cxi-driver with an unrestricted resource grant.
+ */
+#define CXI_SVC_DEFAULT 1U
+
+/*
+ * QPN field encoding (UEC-aligned)
+ *   bit  31     : addr_mode  (1=absolute, 0=job-relative)
+ *   bits [23:12]: Resource Index (CXI pid_offset within the PID slice)
+ *   bits [11:0] : PIDonFEP  (process-endpoint address)
+ *
+ * The Resource Index is the application's own receive-bucket selector and is
+ * unrelated to a CXI service ID, which only names a resource-limit pool.
+ */
+#define CXI_QPN_ADDR_MODE_MASK   BIT(31)
+#define CXI_QPN_RI_SHIFT         12U
+#define CXI_QPN_RI_MASK          GENMASK(23, 12)
+#define CXI_QPN_PID_MASK         GENMASK(11, 0)
+
+/* -----------------------------------------------------------------------
+ * Per-device mmap entry: wraps rdma_user_mmap_entry with driver context.
+ * Heap-allocated; freed by cxiverbs_mmap_free().
+ * -----------------------------------------------------------------------
+ */
+struct cxiverbs_mmap_entry {
+	bool admin_only;
+	struct rdma_user_mmap_entry rdma_entry; /* must be first */
+	bool            is_io;      /* true  → io_remap_pfn_range (doorbell) */
+	phys_addr_t     phys;       /* IO mapping physical address */
+	struct page    *first_page; /* first page for buffer mapping */
+	void (*release_owner)(void *owner);
+	void *owner;
+};
+
+struct cxiverbs_job_tc_binding {
+	struct list_head entry;
+	struct cxiverbs_job_entry *job;
+	u32 svc_id;
+};
+
+struct cxiverbs_tc_entry {
+	u32 key;
+	u32 tc;
+	u32 tc_type;
+	bool created;
+	struct list_head job_list;
+};
+
+struct cxiverbs_job_entry {
+	struct ib_job ibjob;
+	struct ib_job_attr attr;
+	struct xarray addr_table;
+	struct mutex addr_lock;
+	u32 job_id;
+	refcount_t refcount;
+};
+
+/* -----------------------------------------------------------------------
+ * Top-level RDMA device container
+ * -----------------------------------------------------------------------
+ */
+struct cxiverbs_dev {
+	struct ib_device  ib_dev;
+	struct cxi_dev   *cdev;      /* CXI public API handle (cxi_adev->cxi_dev) */
+	struct cxi_adev  *cadev;   /* auxiliary device (PCI access)             */
+
+	/* Device geometry – cached once at probe time */
+	unsigned int      nid;
+	unsigned int      pid_bits;
+	unsigned int      pid_count;
+	unsigned int      pid_granule;
+
+	/*
+	 * Full device-info snapshot, same source as the above four fields.
+	 * Surfaced to user space via query_device's driver-specific response
+	 * (cxi_ibv_query_device_resp.devinfo) since the RDMA path has no
+	 * ioctl/sysfs channel to fetch it on demand the way native access does.
+	 */
+	struct cxi_dev_info_use devinfo;
+
+	struct xarray jobs;
+	struct xarray tcs;
+	struct mutex jobs_lock;
+	struct notifier_block nid_notifier;
+	bool		  ib_registered;
+};
+
+/* -----------------------------------------------------------------------
+ * User context (ib_ucontext)  →  JobID-anchored LNI
+ *
+ * The ucontext opens first. A later privileged Job allocation gets or creates
+ * a refcounted device-wide JobID entry and binds this open context to its LNI.
+ * All PDs, QPs, CQs, and other resources share this LNI after binding.
+ *
+ * No per-service approval cache: services are selected per-QP at creation time.
+ * Job/JobID_end define the VNI/authorization range for this process.
+ * -----------------------------------------------------------------------
+ */
+#define CXIVERBS_TC_MAX		8	/* 0–7 traffic classes */
+
+struct cxiverbs_tc_limit {
+	u32 tc;			/* traffic class 0–7 */
+	u32 max_ptes;		/* max portal table entries */
+	u32 max_cps;		/* max communication profiles */
+	u32 max_qps;		/* max queue pairs */
+	u32 reserved;
+};
+
+struct cxiverbs_counters {
+	struct ib_counters ibcounters;
+};
+
+struct cxiverbs_ucontext {
+	struct ib_ucontext  ibucontext;
+	struct xarray lnis;
+	struct mutex resource_lock;
+	struct cxi_lni     *lni;		/* JobID-anchored, per context */
+	struct cxi_adev    *cadev;
+	struct cxiverbs_job_entry *job;
+
+	/* JobID anchor, valid after privileged job allocation */
+	u32                 job_id;
+
+	/* TC-based resource limits (per JobID) */
+	struct cxiverbs_tc_limit tc_limits[CXIVERBS_TC_MAX];
+	u32                 num_tc_limits;
+	struct mutex        tc_lock;
+
+	/* PIDonFEP defaults (optional) */
+	u32                 default_pidonfep;
+	bool                default_pidonfep_valid;
+};
+
+struct cxiverbs_lni {
+	struct cxi_lni *lni;
+	struct cxiverbs_ucontext *ctx;
+	atomic_t usecnt;
+	u32 svc_id;
+	u32 handle;
+};
+
+struct cxiverbs_lni *cxiverbs_lni_get(struct cxiverbs_ucontext *ctx,
+				    u32 handle);
+void cxiverbs_lni_put(struct cxiverbs_lni *lni);
+
+/* -----------------------------------------------------------------------
+ * Protection domain (ib_pd)  →  shared JobID-anchored LNI + RGID (isolation)
+ *
+ * PD no longer owns its own LNI; instead it references the JobID-anchored LNI
+ * from the ucontext. The PD can optionally express a preferred_svc_id for QPs;
+ * if not set, QPs can select any service at creation.
+ * -----------------------------------------------------------------------
+ */
+struct cxiverbs_pd {
+	struct ib_pd    ibpd;
+	struct cxiverbs_lni *lni_ref;
+	struct xarray qpns;
+
+	u32             job_id;		/* cached from ucontext->job_id */
+	u32             preferred_svc_id;	/* optional service preference (0 = any) */
+
+	u32             rgid;			/* RGID for resource isolation */
+	u32             pd_id;			/* RGID-based PD identifier */
+	u32             default_pidonfep;
+	u32             pidonfep;
+	u32             qpn_addr_mode;
+	bool            default_pidonfep_valid;
+	bool            pidonfep_valid;
+	struct mutex     resource_lock;
+};
+
+struct cxiverbs_jkey {
+	struct ib_jkey ibjkey;
+	struct cxi_cp *cp;
+};
+
+/* -----------------------------------------------------------------------
+ * Memory region (ib_mr)  →  CXI ATU memory descriptor
+ *
+ * lkey = low 32 bits of (md->id << 3) | lac
+ * rkey = random 32-bit token (portal match_bits for remote access)
+ * -----------------------------------------------------------------------
+ */
+struct cxiverbs_mr {
+	struct ib_mr                  ibmr;
+	atomic_t native_users;
+	struct cxi_md                *md;
+	u8                            lac;
+	struct cxi_adev  *cadev;
+};
+
+/* -----------------------------------------------------------------------
+ * Completion queue (ib_cq)  →  CXI Event Queue (EQ)
+ * -----------------------------------------------------------------------
+ */
+struct cxiverbs_cq {
+	struct ib_cq   ibcq;
+	struct cxiverbs_cq_resources *resources;
+	struct cxiverbs_lni *lni_ref;
+
+	struct cxi_eq *eq;
+	struct cxi_adev  *cadev;
+
+	/* Kernel-allocated event ring buffer */
+	struct page  **pages;
+	unsigned int   npages;
+	void          *eq_buf;
+	size_t         eq_buf_size;
+
+	/* ATU descriptor covering the EQ ring */
+	struct cxi_md *eq_md;
+
+	/* EQ doorbell CSR physical address */
+	phys_addr_t    db_phys;
+	size_t         db_size;
+
+	/* User-space mmap entries (heap-allocated, freed via mmap_free) */
+	struct cxiverbs_mmap_entry *buf_mmap;
+	struct cxiverbs_mmap_entry *db_mmap;
+};
+
+/* -----------------------------------------------------------------------
+ * Queue pair (ib_qp)  →  CXI Domain + CP + TXQ + TGQ + PTE
+ * -----------------------------------------------------------------------
+ */
+struct cxiverbs_qp {
+	struct ib_qp       ibqp;
+	struct cxiverbs_qp_resources *resources;
+
+	/* CXI hardware resources */
+	struct cxi_lni    *lni;   /* == to_cxiverbs_pd(ibqp.pd)->lni */
+	struct cxi_domain *domain;
+	struct cxi_cp     *cp;
+	struct cxi_cq     *txq;    /* initiator (transmit) command queue */
+	struct cxi_cq     *tgq;    /* target (receive) command queue     */
+	struct cxi_pte    *pte;    /* portal table entry                 */
+	int                pt_map_index; /* RMU portal-list slot, for unmap */
+
+	/* User-space mmap entries for libcxi-style direct submission */
+	struct cxiverbs_mmap_entry *txq_cmds_mmap;
+	struct cxiverbs_mmap_entry *txq_db_mmap;
+	struct cxiverbs_mmap_entry *tgq_cmds_mmap;
+	struct cxiverbs_mmap_entry *tgq_db_mmap;
+
+	/* QP state machine */
+	enum ib_qp_state  state;
+	spinlock_t        lock;        /* publishes state to the data path */
+	struct mutex      modify_lock; /* serializes modify_qp, may sleep */
+	u32               qpn_base;
+	bool identity_reserved;
+
+	/* Fabric addressing configured at modify_qp time */
+	u32               vni;
+	u32               svc_id;
+	u32               ri;    /* Resource Index == PTE pid_offset */
+	u32               tc;
+	u32               lcid;
+	u64               match_bits;
+	u64               match_mask;
+
+	/* Capacity */
+	u32               max_send_wr;
+	u32               max_recv_wr;
+	u32               max_send_sge;
+	u32               max_recv_sge;
+	u32               max_inline_data;
+};
+
+/* -----------------------------------------------------------------------
+ * Address handle (ib_ah)  →  remote NID + DSCP
+ * -----------------------------------------------------------------------
+ */
+struct cxiverbs_ah {
+	struct ib_ah  ibah;
+	u32           nid;   /* 20-bit Cassini NID of the remote endpoint */
+	u16           vni;   /* fabric VNI for the path                   */
+	u8            dscp;  /* DSCP / traffic-class mapped from SL        */
+};
+
+/* -----------------------------------------------------------------------
+ * Accessor macros
+ * -----------------------------------------------------------------------
+ */
+#define to_cxiverbs_dev(ptr)     container_of(ptr, struct cxiverbs_dev, ib_dev)
+#define to_cxiverbs_ucontext(ptr) container_of(ptr, struct cxiverbs_ucontext, ibucontext)
+#define to_cxiverbs_pd(ptr)       container_of(ptr, struct cxiverbs_pd, ibpd)
+#define to_cxiverbs_mr(ptr)       container_of(ptr, struct cxiverbs_mr, ibmr)
+#define to_cxiverbs_cq(ptr)       container_of(ptr, struct cxiverbs_cq, ibcq)
+#define to_cxiverbs_qp(ptr)       container_of(ptr, struct cxiverbs_qp, ibqp)
+#define to_cxiverbs_ah(ptr)       container_of(ptr, struct cxiverbs_ah, ibah)
+
+/* -----------------------------------------------------------------------
+ * Function declarations — cxiverbs_main.c (Phase 0–3, 5)
+ * -----------------------------------------------------------------------
+ */
+
+/* Phase 0 */
+int  cxiverbs_query_device(struct ib_device *ibdev,
+			  struct ib_device_attr *props,
+			  struct ib_udata *udata);
+int  cxiverbs_query_port(struct ib_device *ibdev, u32 port,
+			struct ib_port_attr *props);
+int  cxiverbs_get_port_immutable(struct ib_device *ibdev, u32 port_num,
+				struct ib_port_immutable *immutable);
+struct net_device *cxiverbs_get_netdev(struct ib_device *ibdev, u32 port);
+int cxiverbs_create_counters(struct ib_counters *counters,
+			     struct uverbs_attr_bundle *attrs);
+int cxiverbs_destroy_counters(struct ib_counters *counters);
+int cxiverbs_read_counters_ex(struct ib_counters *counters,
+			      struct ib_counters_read_attr *attr,
+			      struct uverbs_attr_bundle *attrs);
+int  cxiverbs_query_pkey(struct ib_device *device, u32 port_num,
+			u16 index, u16 *pkey);
+int  cxiverbs_query_gid(struct ib_device *device, u32 port_num, int index,
+		       union ib_gid *gid);
+
+/* Phase 1 – user context */
+int  cxiverbs_alloc_ucontext(struct ib_ucontext *uctx, struct ib_udata *udata);
+void cxiverbs_dealloc_ucontext(struct ib_ucontext *uctx);
+
+/* Phase 1 – protection domain */
+int cxiverbs_alloc_pd(struct ib_pd *ibpd, struct ib_udata *udata);
+int cxiverbs_dealloc_pd(struct ib_pd *ibpd, struct ib_udata *udata);
+int cxiverbs_alloc_job(struct ib_job *job, struct ib_job_attr *attr,
+		       struct uverbs_attr_bundle *attrs);
+void cxiverbs_dealloc_job(struct ib_job *job);
+int cxiverbs_query_job(struct ib_job *job, struct ib_job_attr *attr);
+int cxiverbs_job_addr_insert(struct ib_job *job, u32 index,
+			     struct rdma_ah_attr_ex *ah_attr);
+int cxiverbs_job_addr_remove(struct ib_job *job, u32 index);
+int cxiverbs_job_addr_query(struct ib_job *job, u32 index,
+			    struct rdma_ah_attr_ex *ah_attr);
+int cxiverbs_create_jkey(struct ib_jkey *jkey,
+			 struct uverbs_attr_bundle *attrs);
+int cxiverbs_destroy_jkey(struct ib_jkey *jkey,
+			  struct uverbs_attr_bundle *attrs);
+int cxiverbs_query_qp_semantics(struct ib_device *device,
+			       enum ib_qp_type type,
+			       struct ib_qp_semantics *sem);
+
+/* Phase 3 – completion queues */
+int  cxiverbs_create_cq(struct ib_cq *ibcq, const struct ib_cq_init_attr *attr,
+		       struct uverbs_attr_bundle *bundle);
+int  cxiverbs_destroy_cq(struct ib_cq *ibcq, struct ib_udata *udata);
+int  cxiverbs_poll_cq(struct ib_cq *ibcq, int ne, struct ib_wc *wc);
+int  cxiverbs_req_notify_cq(struct ib_cq *ibcq, enum ib_cq_notify_flags flags);
+int  cxiverbs_mmap(struct ib_ucontext *ibcontext, struct vm_area_struct *vma);
+void cxiverbs_mmap_free(struct rdma_user_mmap_entry *rdma_entry);
+
+/* Phase 5 – address handles and GID table */
+int  cxiverbs_create_ah(struct ib_ah *ibah, struct rdma_ah_init_attr *init_attr,
+		       struct ib_udata *udata);
+int  cxiverbs_destroy_ah(struct ib_ah *ibah, u32 flags);
+int  cxiverbs_add_gid(const struct ib_gid_attr *attr, void **context);
+int  cxiverbs_del_gid(const struct ib_gid_attr *attr, void **context);
+
+/* mmap entry helper — used by main and qp */
+struct cxiverbs_mmap_entry *
+cxiverbs_mmap_entry_alloc(struct ib_ucontext *ibctx, bool is_io,
+			 phys_addr_t phys, struct page *first_page,
+			 size_t size, u64 *offset_out);
+
+struct cxiverbs_mmap_entry *
+cxiverbs_mmap_entry_alloc_owned(struct ib_ucontext *ibctx, bool is_io,
+			       phys_addr_t phys, struct page *first_page,
+			       size_t size, u64 *offset_out, void *owner,
+			       void (*release_owner)(void *));
+
+struct cxiverbs_mmap_entry *
+cxiverbs_mmap_entry_alloc_admin(struct ib_ucontext *ibctx, phys_addr_t phys,
+			       size_t size, u64 *offset_out);
+
+/* -----------------------------------------------------------------------
+ * Function declarations — cxiverbs_mr.c (Phase 2)
+ * -----------------------------------------------------------------------
+ */
+struct ib_mr *cxiverbs_reg_user_mr(struct ib_pd *ibpd, u64 start, u64 length,
+				  u64 iova, int access,
+				  struct ib_dmah *dmah,
+				  struct ib_udata *udata);
+int           cxiverbs_dereg_mr(struct ib_mr *ibmr, struct ib_udata *udata);
+
+/* -----------------------------------------------------------------------
+ * Function declarations — cxiverbs_qp.c (Phase 4)
+ * -----------------------------------------------------------------------
+ */
+int cxiverbs_create_qp(struct ib_qp *ibqp, struct ib_qp_init_attr *init_attr,
+		      struct ib_udata *udata);
+int cxiverbs_destroy_qp(struct ib_qp *ibqp, struct ib_udata *udata);
+int cxiverbs_modify_qp(struct ib_qp *ibqp, struct ib_qp_attr *qp_attr,
+		      int qp_attr_mask, struct ib_udata *udata);
+int cxiverbs_query_qp(struct ib_qp *ibqp, struct ib_qp_attr *qp_attr,
+		     int qp_attr_mask, struct ib_qp_init_attr *qp_init_attr);
+int cxiverbs_post_send(struct ib_qp *ibqp, const struct ib_send_wr *wr,
+		      const struct ib_send_wr **bad_wr);
+int cxiverbs_post_recv(struct ib_qp *ibqp, const struct ib_recv_wr *wr,
+		      const struct ib_recv_wr **bad_wr);
+
+/* -----------------------------------------------------------------------
+ * Driver uapi_definition table — Phase 6 (Counting Triggers)
+ * Defined in cxiverbs_ct.c; assigned to ib_dev.driver_def at probe time.
+ * -----------------------------------------------------------------------
+ */
+extern const struct uapi_definition cxiverbs_defs[];
+
+#endif /* _CXI_IB_H_ */
diff --git a/include/uapi/rdma/cxiverbs-abi.h b/include/uapi/rdma/cxiverbs-abi.h
new file mode 100644
index 000000000000..8bf7b11acb22
--- /dev/null
+++ b/include/uapi/rdma/cxiverbs-abi.h
@@ -0,0 +1,658 @@
+/* SPDX-License-Identifier: GPL-2.0 WITH Linux-syscall-note */
+/*
+ * Copyright 2025 Hewlett Packard Enterprise Development LP
+ *
+ * User-space ABI for the CXI InfiniBand RDMA driver.
+ * This header is shared between the kernel driver and the libcxi_verbs
+ * user-space provider in rdma-core.
+ *
+ * QPN encoding (UEC-aligned):
+ *   bit  31     : addr_mode (1 = absolute, 0 = job-relative)
+ *   bits [23:12]: Resource Index (PID-slice receive selector, not service ID)
+ *   bits [11:0] : PIDonFEP (process-endpoint address bits)
+ */
+
+#ifndef __UAPI_RDMA_CXI_ABI_H__
+#define __UAPI_RDMA_CXI_ABI_H__
+
+#include <linux/types.h>
+
+#define CXI_IB_UVERBS_ABI_VERSION	1
+#define CXI_IBV_LNI_ANY		0xffffffffU
+
+/*
+ * alloc_ucontext / GET_CONTEXT
+ *
+ * GET_CONTEXT opens the uverbs device and returns static CXI geometry.
+ * Privileged ibv_alloc_job() creates a device-scoped job; it does not allocate
+ * an LNI or bind the ucontext. LNIs are allocated explicitly and selected by
+ * PDs. QPs require a job key belonging to their PD. PIDonFEP may be supplied
+ * as an optional default for later QP creation.
+ */
+enum cxi_ibv_alloc_ucontext_mask {
+	CXI_IBV_ALLOC_UCONTEXT_PIDONFEP = 1U << 0,
+};
+
+#define CXI_IBV_SERVICE_ID_MAX	0xfffU
+#define CXI_IBV_JOB_ID_MAX	0xffffU
+#define CXI_IBV_PIDONFEP_MAX	0xfffU
+
+/*
+ * A Slingshot GID is just the 20-bit NID: subnet_prefix is zero and the NID
+ * sits in the low bits of interface_id. Userspace never needs to know that -
+ * it exchanges GIDs like any other transport - but a GID handed back to this
+ * device must fit, or it names something we cannot reach.
+ */
+#define CXI_IBV_NID_MASK	0xfffffULL
+
+struct cxi_ibv_alloc_ucontext_cmd {
+	__u32 comp_mask;		/* enum cxi_ibv_alloc_ucontext_mask */
+	__u32 pidonfep;		/* UET PIDonFEP (optional default) */
+};
+
+/*
+ * TC capacity limits (one per traffic class, 0-7).
+ * Used to report per-JobID resource allocation.
+ */
+struct cxi_ibv_tc_limit {
+	__u32 tc;		/* traffic class 0-7 */
+	__u32 max_ptes;		/* max portal table entries for this TC */
+	__u32 max_cps;		/* max communication profiles for this TC */
+	__u32 max_qps;		/* max queue pairs for this TC */
+	__u32 reserved;		/* for future use */
+};
+
+#define CXI_IBV_TC_MAX		8	/* 0-7 traffic classes */
+
+struct cxi_ibv_alloc_ucontext_resp {
+	__u32 nid;		/* 20-bit NIC address (NID) of this port */
+	__u32 pid_bits;		/* width of the PID field (6–9 bits) */
+	__u32 pid_count;	/* number of PIDs per VNI */
+	__u32 pid_granule;	/* PID space granularity */
+	__u32 lni_id;		/* Zero; LNIs are allocated explicitly */
+	__u32 num_tc_limits;	/* number of valid tc_limits[] entries */
+	/* variable-length tc_limits[] follows if num_tc_limits > 0 */
+};
+
+/*
+ * alloc_pd / ALLOC_PD
+ *
+ * The protection domain selects an explicitly allocated LNI. A nonzero
+ * preferred_svc_id must match that LNI's service; zero means no preference.
+ */
+enum cxi_ibv_alloc_pd_mask {
+	CXI_IBV_ALLOC_PD_SVC_ID_PREFERRED = 1U << 0,
+	CXI_IBV_ALLOC_PD_LNI = 1U << 1,
+};
+
+struct cxi_ibv_alloc_pd_cmd {
+	__u32 comp_mask;	/* enum cxi_ibv_alloc_pd_mask */
+	__u32 preferred_svc_id;	/* optional preferred service for QPs (0 = any) */
+	__u32 lni_handle;
+	__u32 reserved;
+};
+
+struct cxi_ibv_alloc_pd_resp {
+	__u32 pd_id;	/* RGID-based PD identifier (informational) */
+	__u32 pad;
+};
+
+/*
+ * create_cq / CREATE_CQ
+ *
+ * The driver allocates a CXI Event Queue (EQ) of the requested size and
+ * maps the ring buffer and doorbell CSR into user space.
+ *
+ * After create_cq succeeds:
+ *   - eq_buf_offset / eq_buf_size : mmap offset+size for the event-ring buffer
+ *     (user polls this for completions)
+ *   - eq_db_offset / eq_db_size   : mmap offset+size for the EQ doorbell/CSR
+ *     (user writes here to arm notifications)
+ *   - eqn                         : hardware EQ index
+ */
+struct cxi_ibv_create_cq_cmd {
+	__u32 n_entries;	/* requested event-ring depth in entries */
+	__u32 cpu_affinity;	/* NUMA / CPU affinity hint for IRQ steering */
+	__u32 lni_handle;
+	__u32 reserved;
+};
+
+struct cxi_ibv_create_cq_resp {
+	__u64 eq_buf_offset;	/* mmap offset for event-ring buffer */
+	__u64 eq_buf_size;	/* event-ring buffer size in bytes */
+	__u64 eq_db_offset;	/* mmap offset for EQ doorbell (CSR) */
+	__u64 eq_db_size;	/* doorbell region size in bytes */
+	__u32 eqn;		/* hardware EQ index */
+	__u32 pad;
+};
+
+/*
+ * create_qp / CREATE_QP
+ *
+ * On creation the driver allocates:
+ *   - a CXI Domain (VNI + PID)
+ *   - a Communication Profile (CP → LCID)
+ *   - a transmit CQ (TXQ)
+ *   - a target CQ (TGQ)
+ *   - a Portal Table Entry (PTE) for receive matching
+ *
+ * The QP uses its required PD job key's JobID. svc_id selects the CXI
+ * service/resource pool for this QP,
+ * independently of its Resource Index, and must match the PD's LNI service.
+ * If omitted, the driver uses that service. TC
+ * defaults to CXI_TC_BEST_EFFORT unless overridden. match_bits / match_mask
+ * control the portal-level receive filter (0 / 0 = accept all).
+ *
+ * After create_qp succeeds:
+ *   - txq_cmds_offset / txq_cmds_size : mmap offset+size for the TXQ command
+ *     ring (user writes commands here)
+ *   - txq_db_offset / txq_db_size     : mmap offset+size for the TXQ doorbell
+ *   - txq_idx                         : TXQ hardware CQ index (0–1023)
+ *   - lcid                            : LCID assigned to this QP (from CP)
+ *   - tgq_cmds_offset / tgq_cmds_size : mmap offset+size for the TGQ command
+ *     ring (user appends receive buffers here)
+ *   - tgq_db_offset / tgq_db_size     : mmap offset+size for the TGQ doorbell
+ *   - tgq_idx                         : TGQ hardware CQ index (1024–1535)
+ *
+ * The PTE hardware index needed to post receives is not known until the QP
+ * reaches RTR (see cxi_ibv_modify_qp_resp below).
+ */
+enum cxi_ibv_create_qp_mask {
+	CXI_IBV_CREATE_QP_SVC_ID = 1U << 0,
+	CXI_IBV_CREATE_QP_TC     = 1U << 1,
+};
+
+struct cxi_ibv_create_qp_cmd {
+	__u32 comp_mask;	/* enum cxi_ibv_create_qp_mask */
+	__u32 svc_id;		/* optional CXI service (resource-limit pool) */
+	__u32 tc;		/* enum cxi_traffic_class */
+	__u32 uec_qp_type;	/* UEC QP type (reserved; use 0 until spec final) */
+	__u32 pad;
+	__u64 match_bits;	/* portal match bits (0 = match all) */
+	__u64 match_mask;	/* matching mask (0 = match all) */
+};
+
+struct cxi_ibv_create_qp_resp {
+	__u64 txq_cmds_offset;	/* mmap offset for TXQ command ring */
+	__u64 txq_cmds_size;	/* TXQ command ring size in bytes */
+	__u64 txq_db_offset;	/* mmap offset for TXQ doorbell */
+	__u64 txq_db_size;	/* TXQ doorbell size in bytes */
+	__u32 txq_idx;		/* TXQ hardware index (0–1023) */
+	__u32 lcid;		/* LCID assigned to this QP */
+	__u64 tgq_cmds_offset;	/* mmap offset for TGQ command ring */
+	__u64 tgq_cmds_size;	/* TGQ command ring size in bytes */
+	__u64 tgq_db_offset;	/* mmap offset for TGQ doorbell */
+	__u64 tgq_db_size;	/* TGQ doorbell size in bytes */
+	__u32 tgq_idx;		/* TGQ hardware index (1024–1535, see cxi_cq_get_cqn()) */
+	__u32 job_id;		/* JobID (VNI) the QP was created under */
+	__u32 pte_index;
+	__u32 reserved;
+};
+
+/*
+ * modify_qp / MODIFY_QP (extended)
+ *
+ * The PTE is mapped into the domain during the INIT->RTR transition, so its
+ * hardware index is not known at create_qp time. Returned here instead.
+ */
+struct cxi_ibv_modify_qp_resp {
+	__u32 pte_index;	/* hardware PTLTE index, valid once QP reaches RTR */
+	__u32 pad;
+};
+
+/*
+ * query_device / QUERY_DEVICE  (driver-specific extended response)
+ *
+ * When the user-space provider calls ibv_query_device_ex(), the kernel
+ * returns this struct as driver-specific data (via udata->out).
+ * The caller must pass at least sizeof(cxi_ibv_query_device_resp) bytes
+ * in the output buffer; unused bytes are zeroed.
+ *
+ * features:
+ *   CXI_DEVICE_CAP_UEC_QPN        - QPN uses UEC bit-31 addr_mode encoding
+ *   CXI_DEVICE_CAP_TRIGGERED_OPS  - counting triggers / triggered operations
+ *   CXI_DEVICE_CAP_ODP            - on-demand paging MRs
+ *
+ * max_ctn: highest counting-trigger number (inclusive) this device supports.
+ */
+#define CXI_DEVICE_CAP_UEC_QPN          (1U << 0)
+#define CXI_DEVICE_CAP_TRIGGERED_OPS    (1U << 1)
+#define CXI_DEVICE_CAP_ODP              (1U << 2)
+
+/*
+ * Device-wide HW properties that the native ABI exposes via the
+ * DEV_INFO_GET ioctl (struct cxi_dev_info_use in uapi/ethernet/cxi-abi.h).
+ * The RDMA path has no ioctl/sysfs control channel, so it must return the
+ * same information here instead. Field-for-field mirror of that struct,
+ * kept separate (rather than #include'd) so this ABI header doesn't pull
+ * in the native ioctl definitions it has no use for; keep the two in sync.
+ */
+struct cxi_ibv_devinfo {
+	__u32 nid;
+	__u32 pid_bits;
+	__u32 pid_count;
+	__u32 pid_granule;
+	__u32 min_free_shift;
+	__u32 rdzv_get_idx;
+	__u32 vendor_id;
+	__u32 device_id;
+	__u32 device_rev;
+	__u32 device_proto;
+	__u32 device_platform;
+	__u32 pct_eq;
+	__s32 uc_nic;
+	__u64 link_mtu;
+	__u64 link_speed;
+	__u8  link_state;
+	__u16 num_ptes;
+	__u16 num_txqs;
+	__u16 num_tgqs;
+	__u16 num_eqs;
+	__u16 num_cts;
+	__u16 num_acs;
+	__u16 num_tles;
+	__u16 num_les;
+	__u32 cassini_version;         /* enum cassini_version */
+	__u32 system_type_identifier;  /* enum system_type_identifier */
+	char  fru_description[16];
+	__u32 vni_matching;
+};
+
+struct cxi_ibv_query_device_resp {
+	__u32 features; /* CXI_DEVICE_CAP_* bitmask */
+	__u32 max_ctn;  /* max counting trigger number */
+	struct cxi_ibv_devinfo devinfo;
+};
+
+/*
+ *
+ * The driver registers user memory with the CXI ATU (Address Translation
+ * Unit).  The lkey encodes the LAC (Local Address Context) slot and the
+ * hardware MD identifier.  The rkey is a random token used as the portal
+ * match-bits for remote RDMA WRITE/READ operations targeting this MR.
+ */
+/*
+ * 64-bit memory keys.
+ *
+ *   lkey64  bits [2:0]   LAC, the hardware address context
+ *           bits [34:3]  ATU memory descriptor index
+ *   rkey64  an opaque token the initiator presents as the portal match bits
+ *
+ * The 32-bit lkey and rkey keep the older narrow encoding, which packs the
+ * LAC into bits [31:24] and truncates the descriptor index to 24 bits, for
+ * callers that cannot carry a wide key.
+ */
+#define CXI_LKEY64_LAC_MASK	0x7ULL
+#define CXI_LKEY64_MD_SHIFT	3
+#define CXI_LKEY64_MD_MASK	0xffffffffULL
+
+/*
+ * A caller may choose the remote key rather than take a generated one. The
+ * value is used exactly as given, so one that fits in 32 bits produces a
+ * region a 32-bit peer can reach without anything having to truncate.
+ */
+#define CXI_REG_MR_USER_RKEY	(1 << 0)
+#define CXI_REG_MR_NATIVE_MAP	(1 << 1)
+
+#define CXI_RKEY_MAX 0xffffffffffffULL
+
+struct cxi_ibv_reg_mr {
+	__aligned_u64 rkey64;
+	__u32 flags;
+	__u32 native_flags;
+};
+
+struct cxi_ibv_mr_info {
+	__u64 va __attribute__((packed));
+	__u64 iova __attribute__((packed));
+	__u64 length __attribute__((packed));
+	__s32 page_shift;
+	__s32 huge_shift;
+	__u32 id;
+	__u32 lac;
+};
+
+struct cxi_ibv_reg_mr_resp {
+	/*
+	 * The narrow keys alias the low word of the wide ones. REG_MR is a
+	 * legacy write command whose response declares __u32 driver_data[], so
+	 * this payload begins at offset 12 and nothing here can be eight-byte
+	 * aligned; packed drops the requirement without moving anything.
+	 */
+	union {
+		__u64 lkey64 __attribute__((packed));
+		struct {
+			__u32 lkey;
+			__u32 lkey_hi;
+		};
+	};
+	union {
+		__u64 rkey64 __attribute__((packed));
+		struct {
+			__u32 rkey;
+			__u32 rkey_hi;
+		};
+	};
+	/* ATU address for @start; commands must use this, not the VA. */
+	__u64 iova __attribute__((packed));
+	struct cxi_ibv_mr_info info;
+};
+
+/*
+ * Custom uverbs for Counting Triggers (CT).
+ *
+ * The driver allocates a CXI Completion Table (CT) of the requested size and
+ * maps the ring buffer (writeback page) and doorbell CSR into user space.
+ *
+ * After create_ct succeeds:
+ *   - ctn               : hardware CT number (u32); used in triggered commands
+ *   - wb_mmap_offset    : mmap() offset for the writeback page
+ *   - wb_mmap_length    : size of the writeback page in bytes
+ *   - db_mmap_offset    : mmap() offset for the CT doorbell CSR
+ *   - db_mmap_length    : size of the doorbell region in bytes
+ *
+ * Object/method/attribute IDs are driver-namespace values
+ * (UVERBS_ID_DRIVER_NS = 1 << UVERBS_ID_NS_SHIFT).
+ */
+#include <rdma/ib_user_ioctl_cmds.h>
+
+enum cxi_ib_object_ids {
+	CXI_IB_OBJECT_CT = (1U << UVERBS_ID_NS_SHIFT),
+	CXI_IB_OBJECT_JOB_TC,
+	CXI_IB_OBJECT_LNI,
+	CXI_IB_OBJECT_SVC,
+	CXI_IB_OBJECT_CP,
+	CXI_IB_OBJECT_EQ,
+	CXI_IB_OBJECT_CSR,
+	CXI_IB_OBJECT_WAIT,
+	CXI_IB_OBJECT_CMDQ,
+};
+
+enum cxi_ib_methods_cmdq {
+	CXI_IB_METHOD_CMDQ_ALLOC = (1U << UVERBS_ID_NS_SHIFT),
+	CXI_IB_METHOD_CMDQ_DESTROY,
+};
+
+enum cxi_ib_attrs_cmdq {
+	CXI_IB_ATTR_CMDQ_HANDLE = (1U << UVERBS_ID_NS_SHIFT),
+	CXI_IB_ATTR_CMDQ_LNI,
+	CXI_IB_ATTR_CMDQ_COUNT,
+	CXI_IB_ATTR_CMDQ_INFO,
+	CXI_IB_ATTR_CMDQ_FLAGS,
+};
+
+enum cxi_ib_cmdq_flags {
+	CXI_IB_CMDQ_TRIGGER = 1U << 0,
+};
+
+struct cxi_ibv_cmdq_info {
+	__aligned_u64 commands_offset;
+	__aligned_u64 commands_length;
+	__aligned_u64 doorbell_offset;
+	__aligned_u64 doorbell_length;
+	__u32 count;
+	__u32 index;
+};
+
+enum cxi_ib_methods_wait {
+	CXI_IB_METHOD_WAIT_ALLOC = (1U << UVERBS_ID_NS_SHIFT),
+};
+
+enum cxi_ib_attrs_wait {
+	CXI_IB_ATTR_WAIT_FD = (1U << UVERBS_ID_NS_SHIFT),
+	CXI_IB_ATTR_WAIT_LNI,
+};
+
+enum cxi_ib_methods_csr {
+	CXI_IB_METHOD_CSR_MAP = (1U << UVERBS_ID_NS_SHIFT),
+	CXI_IB_METHOD_CSR_UNMAP,
+	CXI_IB_METHOD_CSR_INBOUND_WAIT,
+};
+
+enum cxi_ib_attrs_csr {
+	CXI_IB_ATTR_CSR_HANDLE = (1U << UVERBS_ID_NS_SHIFT),
+	CXI_IB_ATTR_CSR_INFO,
+};
+
+struct cxi_ibv_csr_info {
+	__aligned_u64 offset;
+	__aligned_u64 length;
+};
+
+enum cxi_ib_methods_eq {
+	CXI_IB_METHOD_EQ_ALLOC = (1U << UVERBS_ID_NS_SHIFT),
+	CXI_IB_METHOD_EQ_DESTROY,
+};
+
+enum cxi_ib_attrs_eq {
+	CXI_IB_ATTR_EQ_HANDLE = (1U << UVERBS_ID_NS_SHIFT),
+	CXI_IB_ATTR_EQ_LNI,
+	CXI_IB_ATTR_EQ_MR,
+	CXI_IB_ATTR_EQ_CONFIG,
+	CXI_IB_ATTR_EQ_INFO,
+	CXI_IB_ATTR_EQ_FLAGS,
+	CXI_IB_ATTR_EQ_EVENT_WAIT,
+	CXI_IB_ATTR_EQ_STATUS_WAIT,
+};
+
+enum cxi_ib_eq_flags {
+	CXI_IB_EQ_REGISTER_PCT = 1U << 0,
+	CXI_IB_EQ_PASSTHROUGH = 1U << 1,
+};
+
+struct cxi_ibv_eq_config {
+	__aligned_u64 address;
+	__aligned_u64 length;
+	__u32 threshold_base;
+	__u32 threshold_delta;
+	__u32 threshold_count;
+	__u32 ec_delay;
+	__u32 cpu;
+	__u32 reserved_slots;
+};
+
+struct cxi_ibv_eq_info {
+	__aligned_u64 doorbell_offset;
+	__aligned_u64 doorbell_size;
+	__u32 eqn;
+	__u32 reserved;
+};
+
+enum cxi_ib_methods_cp {
+	CXI_IB_METHOD_CP_ALLOC = (1U << UVERBS_ID_NS_SHIFT),
+	CXI_IB_METHOD_CP_DESTROY,
+};
+
+enum cxi_ib_attrs_cp {
+	CXI_IB_ATTR_CP_HANDLE = (1U << UVERBS_ID_NS_SHIFT),
+	CXI_IB_ATTR_CP_LNI,
+	CXI_IB_ATTR_CP_VNI,
+	CXI_IB_ATTR_CP_TC,
+	CXI_IB_ATTR_CP_TYPE,
+	CXI_IB_ATTR_CP_INFO,
+};
+
+struct cxi_ibv_cp_info {
+	__u32 vni;
+	__u32 tc;
+	__u32 type;
+	__u32 lcid;
+};
+
+enum cxi_ib_methods_svc {
+	CXI_IB_METHOD_SVC_ALLOC = (1U << UVERBS_ID_NS_SHIFT),
+	CXI_IB_METHOD_SVC_DESTROY,
+	CXI_IB_METHOD_SVC_QUERY,
+	CXI_IB_METHOD_SVC_ENABLE,
+};
+
+enum cxi_ib_attrs_svc {
+	CXI_IB_ATTR_SVC_ID = (1U << UVERBS_ID_NS_SHIFT),
+	CXI_IB_ATTR_SVC_CONFIG,
+	CXI_IB_ATTR_SVC_FLAGS,
+	CXI_IB_ATTR_SVC_USAGE,
+	CXI_IB_ATTR_SVC_LPR,
+};
+
+enum cxi_ib_svc_flags {
+	CXI_IB_SVC_LIMITS = 1U << 0,
+	CXI_IB_SVC_CREATE_TC = 1U << 1,
+	CXI_IB_SVC_ENABLED = 1U << 2,
+};
+
+enum cxi_ib_svc_resource {
+	CXI_IB_SVC_PTES,
+	CXI_IB_SVC_TXQS,
+	CXI_IB_SVC_TGQS,
+	CXI_IB_SVC_EQS,
+	CXI_IB_SVC_CTS,
+	CXI_IB_SVC_ACS,
+	CXI_IB_SVC_TLES,
+	CXI_IB_SVC_LES,
+	CXI_IB_SVC_RSRC_COUNT,
+};
+
+#define CXI_IB_SVC_MAX_VNIS 4
+
+struct cxi_ibv_svc_config {
+	__u32 svc_id;
+	/* Reserved; must be zero. */
+	__u16 vni_min;
+	__u16 vni_max;
+	/* Exact-list length; zero means system-empty or is_range. */
+	__u32 vni_count;
+	/* System service requires CAP_SYS_ADMIN and svc_id zero on allocation. */
+	__u32 is_system_svc;
+	/* Inclusive range endpoints are carried in vnis[0] and vnis[1]. */
+	__u32 is_range;
+	__u16 vnis[CXI_IB_SVC_MAX_VNIS];
+	__u32 tc_mask;
+	__u32 uid;
+	__u32 flags;
+	__u16 maximum[CXI_IB_SVC_RSRC_COUNT];
+	__u16 reserved[CXI_IB_SVC_RSRC_COUNT];
+};
+
+struct cxi_ibv_svc_rsrc_use {
+	__u32 svc_id;
+	__u32 tle_pool_id;
+	__u16 in_use[CXI_IB_SVC_RSRC_COUNT];
+};
+
+enum cxi_ib_methods_lni {
+	CXI_IB_METHOD_LNI_ALLOC = (1U << UVERBS_ID_NS_SHIFT),
+	CXI_IB_METHOD_LNI_DESTROY,
+};
+
+enum cxi_ib_attrs_lni_alloc {
+	CXI_IB_ATTR_LNI_ALLOC_HANDLE = (1U << UVERBS_ID_NS_SHIFT),
+	CXI_IB_ATTR_LNI_ALLOC_SVC_ID,
+	CXI_IB_ATTR_LNI_ALLOC_ID,
+};
+
+enum cxi_ib_attrs_lni_destroy {
+	CXI_IB_ATTR_LNI_DESTROY_HANDLE = (1U << UVERBS_ID_NS_SHIFT),
+};
+
+enum cxi_ib_methods_pd {
+	CXI_IB_METHOD_PD_PID_ALLOC = (1U << UVERBS_ID_NS_SHIFT),
+	CXI_IB_METHOD_PD_PID_QUERY,
+};
+
+enum cxi_ib_attrs_pd_pid {
+	CXI_IB_ATTR_PD_PID_HANDLE = (1U << UVERBS_ID_NS_SHIFT),
+	CXI_IB_ATTR_PD_PID_FLAGS,
+	CXI_IB_ATTR_PD_PID_VALUE,
+};
+
+#define CXI_IB_PD_PID_ABSOLUTE (1U << 0)
+
+struct cxi_ibv_create_jkey_cmd {
+	__u32 tc;
+	__u32 reserved;
+};
+
+enum cxi_ib_methods_ct {
+	CXI_IB_METHOD_CT_ALLOC   = (1U << UVERBS_ID_NS_SHIFT),
+	CXI_IB_METHOD_CT_DESTROY = (1U << UVERBS_ID_NS_SHIFT) + 1,
+};
+
+enum cxi_ib_attrs_ct_alloc {
+	CXI_IB_ATTR_CT_ALLOC_HANDLE        = (1U << UVERBS_ID_NS_SHIFT),
+	CXI_IB_ATTR_CT_ALLOC_CTN,
+	CXI_IB_ATTR_CT_ALLOC_WB_MMAP_OFFSET,
+	CXI_IB_ATTR_CT_ALLOC_WB_MMAP_LENGTH,
+	CXI_IB_ATTR_CT_ALLOC_DB_MMAP_OFFSET,
+	CXI_IB_ATTR_CT_ALLOC_DB_MMAP_LENGTH,
+	CXI_IB_ATTR_CT_ALLOC_LNI_HANDLE,
+	CXI_IB_ATTR_CT_ALLOC_WB_ADDRESS,
+};
+
+enum cxi_ib_attrs_ct_destroy {
+	CXI_IB_ATTR_CT_DESTROY_HANDLE = (1U << UVERBS_ID_NS_SHIFT),
+};
+
+enum cxi_ib_job_tc_flags {
+	CXI_IB_JOB_TC_F_CREATE = 1U << 0,
+};
+
+enum cxi_ib_methods_job_tc {
+	CXI_IB_METHOD_JOB_TC_BIND = (1U << UVERBS_ID_NS_SHIFT),
+	CXI_IB_METHOD_JOB_TC_CLEAR,
+	CXI_IB_METHOD_JOB_TC_QUERY,
+};
+
+struct cxi_ibv_tc_entry {
+	__u32 tc;
+	__u32 tc_type;
+	__u32 res_req_dscp;
+	__u32 unres_req_dscp;
+	__u32 res_rsp_dscp;
+	__u32 unres_rsp_dscp;
+	__u32 req_pcp;
+	__u32 rsp_pcp;
+	__u32 ocuset;
+	__u32 cq_tc;
+	__u32 req_bc;
+	__u32 flags;
+};
+
+#define CXI_IBV_TC_ENTRY_F_CREATED_BY_RDMA	(1U << 0)
+
+enum cxi_ib_attrs_job_tc_bind {
+	CXI_IB_ATTR_JOB_TC_BIND_FLAGS = (1U << UVERBS_ID_NS_SHIFT),
+	CXI_IB_ATTR_JOB_TC_BIND_JOB_ID,
+	CXI_IB_ATTR_JOB_TC_BIND_TC,
+	CXI_IB_ATTR_JOB_TC_BIND_TC_TYPE,
+	CXI_IB_ATTR_JOB_TC_BIND_SVC_ID,
+	CXI_IB_ATTR_JOB_TC_BIND_PTES_MAX,
+	CXI_IB_ATTR_JOB_TC_BIND_PTES_RES,
+	CXI_IB_ATTR_JOB_TC_BIND_TXQS_MAX,
+	CXI_IB_ATTR_JOB_TC_BIND_TXQS_RES,
+	CXI_IB_ATTR_JOB_TC_BIND_TGQS_MAX,
+	CXI_IB_ATTR_JOB_TC_BIND_TGQS_RES,
+	CXI_IB_ATTR_JOB_TC_BIND_EQS_MAX,
+	CXI_IB_ATTR_JOB_TC_BIND_EQS_RES,
+	CXI_IB_ATTR_JOB_TC_BIND_CTS_MAX,
+	CXI_IB_ATTR_JOB_TC_BIND_CTS_RES,
+	CXI_IB_ATTR_JOB_TC_BIND_LES_MAX,
+	CXI_IB_ATTR_JOB_TC_BIND_LES_RES,
+	CXI_IB_ATTR_JOB_TC_BIND_TLES_MAX,
+	CXI_IB_ATTR_JOB_TC_BIND_TLES_RES,
+	CXI_IB_ATTR_JOB_TC_BIND_ACS_MAX,
+	CXI_IB_ATTR_JOB_TC_BIND_ACS_RES,
+	CXI_IB_ATTR_JOB_TC_BIND_RESP_SVC_ID,
+};
+
+enum cxi_ib_attrs_job_tc_clear {
+	CXI_IB_ATTR_JOB_TC_CLEAR_TC = (1U << UVERBS_ID_NS_SHIFT),
+	CXI_IB_ATTR_JOB_TC_CLEAR_TC_TYPE,
+};
+
+enum cxi_ib_attrs_job_tc_query {
+	CXI_IB_ATTR_JOB_TC_QUERY_MAX_ENTRIES = (1U << UVERBS_ID_NS_SHIFT),
+	CXI_IB_ATTR_JOB_TC_QUERY_RESP_COUNT,
+	CXI_IB_ATTR_JOB_TC_QUERY_RESP_ENTRIES,
+};
+
+#endif /* __UAPI_RDMA_CXI_ABI_H__ */

-- 
2.56.0

[PATCH 05/11] RDMA/cxi: add the auxiliary device and driver ops

[Doug Ledford]

Bind to the cxi-ss1 auxiliary device and register an ib_device for
it: probe and remove, the NID notifier that updates the device NID and
node GUID when the core driver changes the NIC address, and creation
and the device ops table, including the telemetry counters ops added
later in the series.

If ib_register_device() fails after the NID notifier is registered,
the notifier is unregistered before the device is freed, so the
notifier chain is not left holding a pointer to freed memory.

checkpatch --strict (100 columns) still reports the following in this
patch. They need coordination with the internal repositories this code
is maintained in and will be fixed in V2:
- CHECK: PARENTHESIS_ALIGNMENT (3)

Co-developed-by: Iraimani P <iraimani.pavadai@hpe.com>
Signed-off-by: Iraimani P <iraimani.pavadai@hpe.com>
Signed-off-by: Doug Ledford <doug.ledford@hpe.com>
---
 drivers/infiniband/hw/cxi/cxiverbs_drv.c | 270 +++++++++++++++++++++++++++++++
 1 file changed, 270 insertions(+)

diff --git a/drivers/infiniband/hw/cxi/cxiverbs_drv.c b/drivers/infiniband/hw/cxi/cxiverbs_drv.c
new file mode 100644
index 000000000000..9472ff896c80
--- /dev/null
+++ b/drivers/infiniband/hw/cxi/cxiverbs_drv.c
@@ -0,0 +1,270 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * cxi-ib: InfiniBand RDMA driver for the Cassini CXI NIC.
+ * Copyright 2022-2025 Hewlett Packard Enterprise Development LP
+ *
+ * Registers as an auxiliary driver that consumes the "cxi_ss1.rdma"
+ * auxiliary device published by the cxi-driver.
+ */
+
+#include <linux/module.h>
+#include <linux/init.h>
+#include <linux/kernel.h>
+#include <linux/slab.h>
+#include "cxiverbs.h"
+
+MODULE_AUTHOR("Hewlett Packard Enterprise Development LP");
+MODULE_DESCRIPTION("Cassini CXI RDMA verbs driver");
+MODULE_LICENSE("GPL");
+
+/* -----------------------------------------------------------------------
+ * Device operations table
+ * -----------------------------------------------------------------------
+ */
+static const struct ib_device_ops cxiverbs_dev_ops = {
+	.owner           = THIS_MODULE,
+	.driver_id       = RDMA_DRIVER_CXI,
+	.uverbs_abi_ver  = CXI_IB_UVERBS_ABI_VERSION,
+
+	/* Phase 0 – device and port query */
+	.query_device        = cxiverbs_query_device,
+	.query_port          = cxiverbs_query_port,
+	.get_port_immutable  = cxiverbs_get_port_immutable,
+	.get_netdev           = cxiverbs_get_netdev,
+	.query_pkey          = cxiverbs_query_pkey,
+	.query_gid           = cxiverbs_query_gid,
+
+	/* Phase 1 – user context and protection domain */
+	.alloc_ucontext   = cxiverbs_alloc_ucontext,
+	.dealloc_ucontext = cxiverbs_dealloc_ucontext,
+	.alloc_pd         = cxiverbs_alloc_pd,
+	.dealloc_pd       = cxiverbs_dealloc_pd,
+	.alloc_job        = cxiverbs_alloc_job,
+	.dealloc_job      = cxiverbs_dealloc_job,
+	.query_job        = cxiverbs_query_job,
+	.job_addr_insert  = cxiverbs_job_addr_insert,
+	.job_addr_remove  = cxiverbs_job_addr_remove,
+	.job_addr_query   = cxiverbs_job_addr_query,
+	.query_qp_semantics = cxiverbs_query_qp_semantics,
+	.create_jkey      = cxiverbs_create_jkey,
+	.destroy_jkey     = cxiverbs_destroy_jkey,
+
+	/* Telemetry counters */
+	.create_counters   = cxiverbs_create_counters,
+	.destroy_counters  = cxiverbs_destroy_counters,
+	.read_counters_ex  = cxiverbs_read_counters_ex,
+
+	/* Phase 2 – memory registration */
+	.reg_user_mr = cxiverbs_reg_user_mr,
+	.dereg_mr    = cxiverbs_dereg_mr,
+
+	/* Phase 3 – completion queues and mmap */
+	.create_cq      = cxiverbs_create_cq,
+	.destroy_cq     = cxiverbs_destroy_cq,
+	.poll_cq        = cxiverbs_poll_cq,
+	.req_notify_cq  = cxiverbs_req_notify_cq,
+	.mmap           = cxiverbs_mmap,
+	.mmap_free      = cxiverbs_mmap_free,
+
+	/* Phase 4 – queue pairs */
+	.create_qp  = cxiverbs_create_qp,
+	.modify_qp  = cxiverbs_modify_qp,
+	.query_qp   = cxiverbs_query_qp,
+	.destroy_qp = cxiverbs_destroy_qp,
+	.post_send  = cxiverbs_post_send,
+	.post_recv  = cxiverbs_post_recv,
+
+	/* Phase 5 – address handles and GID management */
+	.create_ah  = cxiverbs_create_ah,
+	.create_user_ah = cxiverbs_create_ah,
+	.destroy_ah = cxiverbs_destroy_ah,
+	.add_gid    = cxiverbs_add_gid,
+	.del_gid    = cxiverbs_del_gid,
+
+	/* Pre-allocated slab sizes for RDMA core object allocator */
+	INIT_RDMA_OBJ_SIZE(ib_ah,        cxiverbs_ah,        ibah),
+	INIT_RDMA_OBJ_SIZE(ib_counters,  cxiverbs_counters,  ibcounters),
+	INIT_RDMA_OBJ_SIZE(ib_cq,        cxiverbs_cq,        ibcq),
+	INIT_RDMA_OBJ_SIZE(ib_pd,        cxiverbs_pd,        ibpd),
+	INIT_RDMA_OBJ_SIZE(ib_job,       cxiverbs_job_entry, ibjob),
+	INIT_RDMA_OBJ_SIZE(ib_jkey,      cxiverbs_jkey,      ibjkey),
+	INIT_RDMA_OBJ_SIZE(ib_qp,        cxiverbs_qp,        ibqp),
+	INIT_RDMA_OBJ_SIZE(ib_ucontext,  cxiverbs_ucontext,  ibucontext),
+};
+
+static void cxiverbs_tc_entries_destroy(struct cxiverbs_dev *dev)
+{
+	struct cxiverbs_job_tc_binding *binding, *tmp;
+	struct cxiverbs_tc_entry *tc_entry;
+	unsigned long index;
+
+	mutex_lock(&dev->jobs_lock);
+	xa_for_each(&dev->tcs, index, tc_entry) {
+		list_for_each_entry_safe(binding, tmp, &tc_entry->job_list, entry) {
+			list_del(&binding->entry);
+			kfree(binding);
+		}
+		kfree(tc_entry);
+	}
+	xa_destroy(&dev->tcs);
+	mutex_unlock(&dev->jobs_lock);
+}
+
+static void cxiverbs_gid_change_event(struct cxiverbs_dev *dev)
+{
+	struct ib_event event = {
+		.device = &dev->ib_dev,
+		.element.port_num = 1,
+		.event = IB_EVENT_GID_CHANGE,
+	};
+
+	ib_dispatch_event(&event);
+}
+
+static __be64 cxiverbs_nid_to_node_guid(u32 nid)
+{
+	if (nid == CXI_INVALID_NID)
+		return cpu_to_be64(U64_MAX);
+
+	return cpu_to_be64(nid);
+}
+
+static int cxiverbs_nid_event(struct notifier_block *nb, unsigned long nid,
+			      void *data)
+{
+	struct cxiverbs_dev *dev = container_of(nb, struct cxiverbs_dev,
+					      nid_notifier);
+
+	WRITE_ONCE(dev->nid, nid);
+	WRITE_ONCE(dev->devinfo.nid, nid);
+	WRITE_ONCE(dev->ib_dev.node_guid, cxiverbs_nid_to_node_guid(nid));
+
+	/*
+	 * The NID is only known once the link trains, which is usually after
+	 * probe, so the cached GID still holds the placeholder read at
+	 * registration. Make the core re-read it via query_gid.
+	 */
+	if (READ_ONCE(dev->ib_registered))
+		cxiverbs_gid_change_event(dev);
+
+	return NOTIFY_OK;
+}
+
+/* -----------------------------------------------------------------------
+ * Auxiliary driver probe / remove
+ * -----------------------------------------------------------------------
+ */
+static int cxiverbs_probe(struct auxiliary_device *adev,
+			const struct auxiliary_device_id *id)
+{
+	struct cxi_adev *cxiadev = container_of(adev, struct cxi_adev, adev);
+	struct cxi_dev_info_use info = {};
+	struct cxiverbs_dev *dev;
+	int ret;
+
+	dev = ib_alloc_device(cxiverbs_dev, ib_dev);
+	if (!dev)
+		return -ENOMEM;
+
+	/* Link to the CXI public API and the PCI device.
+	 * cxi_adev->cxi_dev is set by the cxi-driver before publishing
+	 * the auxiliary device.
+	 */
+	dev->cadev = cxiadev;
+	dev->cdev    = (struct cxi_dev *)cxiadev->cxi_dev;
+	if (!dev->cdev) {
+		dev_err(&adev->dev, "cxiverbs: no cxi_dev in cxi_adev\n");
+		ret = -ENODEV;
+		goto err_dealloc;
+	}
+
+	/* Cache device geometry so individual verbs ops can read it
+	 * without calling cxi_dev_info_get() on every request.
+	 */
+	ret = cxiadev->dev_info_get(dev->cdev, &info);
+	if (ret) {
+		dev_err(&adev->dev, "cxiverbs: cxi_dev_info_get failed: %d\n", ret);
+		goto err_dealloc;
+	}
+	dev->nid         = info.nid;
+	dev->pid_bits    = info.pid_bits;
+	dev->pid_count   = info.pid_count;
+	dev->pid_granule = info.pid_granule;
+	dev->devinfo     = info;
+	xa_init(&dev->jobs);
+	xa_init(&dev->tcs);
+	mutex_init(&dev->jobs_lock);
+	dev->nid_notifier.notifier_call = cxiverbs_nid_event;
+	ret = blocking_notifier_chain_register(&cxiadev->nid_notifier,
+					      &dev->nid_notifier);
+	if (ret)
+		goto err_destroy_jobs_lock;
+
+	ib_set_device_ops(&dev->ib_dev, &cxiverbs_dev_ops);
+
+	dev->ib_dev.node_type       = RDMA_NODE_SLINGSHOT;
+	dev->ib_dev.phys_port_cnt   = 1;
+	dev->ib_dev.num_comp_vectors = 1;
+	dev->ib_dev.local_dma_lkey  = 0;
+	dev->ib_dev.dev.parent      = &cxiadev->pdev->dev;
+
+	dev->ib_dev.node_guid = cxiverbs_nid_to_node_guid(dev->nid);
+
+	/* Expose custom uverbs objects (Counting Triggers, Phase 6). */
+	dev->ib_dev.driver_def = cxiverbs_defs;
+
+	ret = ib_register_device(&dev->ib_dev, "cxi%d", cxiadev->dev);
+	if (ret) {
+		dev_err(&adev->dev, "cxiverbs: ib_register_device failed: %d\n", ret);
+		blocking_notifier_chain_unregister(&cxiadev->nid_notifier,
+						   &dev->nid_notifier);
+		goto err_destroy_jobs_lock;
+	}
+
+	dev_set_drvdata(&adev->dev, dev);
+	WRITE_ONCE(dev->ib_registered, true);
+	cxiverbs_gid_change_event(dev);
+	dev_info(&adev->dev, "CXI RDMA device registered as %s\n",
+		 dev_name(&dev->ib_dev.dev));
+	return 0;
+
+err_dealloc:
+	ib_dealloc_device(&dev->ib_dev);
+	return ret;
+
+err_destroy_jobs_lock:
+	cxiverbs_tc_entries_destroy(dev);
+	xa_destroy(&dev->jobs);
+	mutex_destroy(&dev->jobs_lock);
+	goto err_dealloc;
+}
+
+static void cxiverbs_remove(struct auxiliary_device *adev)
+{
+	struct cxiverbs_dev *dev = dev_get_drvdata(&adev->dev);
+
+	blocking_notifier_chain_unregister(&dev->cadev->nid_notifier,
+					  &dev->nid_notifier);
+	ib_unregister_device(&dev->ib_dev);
+	cxiverbs_tc_entries_destroy(dev);
+	xa_destroy(&dev->jobs);
+	mutex_destroy(&dev->jobs_lock);
+	ib_dealloc_device(&dev->ib_dev);
+}
+
+/* -----------------------------------------------------------------------
+ * Auxiliary device ID table and driver registration
+ * -----------------------------------------------------------------------
+ */
+static const struct auxiliary_device_id cxiverbs_auxiliary_ids[] = {
+	{ .name = "cxi_ss1.rdma", },
+	{},
+};
+MODULE_DEVICE_TABLE(auxiliary, cxiverbs_auxiliary_ids);
+
+static struct auxiliary_driver cxiverbs_driver = {
+	.probe    = cxiverbs_probe,
+	.remove   = cxiverbs_remove,
+	.id_table = cxiverbs_auxiliary_ids,
+};
+module_auxiliary_driver(cxiverbs_driver);

-- 
2.56.0

[PATCH 06/11] RDMA/cxi: add explicit context LNI and PID lifetimes

[Doug Ledford]

Add the context and device-level verbs: ucontext allocation, jobs and
job keys, protection domains, completion queues, address handles,
GIDs, device/port queries and the mmap entries for doorbells and
queues.

An LNI and a PID are never created implicitly.  They are bound to the
context or PD by explicit methods added with the object patch, and
are released in an order the mmap entries enforce: a mapping holds
the resource it maps until the last VMA goes away.

Shared RAM buffers are mapped with rdma_user_mmap_io(), as for
device memory, rather than remap_pfn_range() directly.  The helper
registers the VMA with the mmap entry, so the owning CT or queue and
its LNI stay allocated until the last mapping is unmapped; mapping
RAM directly released them when the owning object was destroyed while
a mapping still existed.

Replies to user space use ib_respond_udata() so that a short response
buffer is truncated and a long one is zero filled, as the ib_udata
compatibility rules require.

The CSR aperture mapping is privileged and publishes the mmap entry
only after its permission flags are set, so the entry is never
visible half initialised.  Privileged mappings (CSR, PCT EQ doorbell)
require CAP_SYS_ADMIN when created and when mapped.  These mappings
are specifically for use by the card's Retry Handler (to be introduced
in the rdma-core submission).  The existing Cassini hardware does not
handle retries in hardware, and these CAP_SYS_ADMIN mapping is needed
for the Retry Handler to be able to directly program the card's CSRs
for the necessary packet resends.

Implement optional, sparse per-job address tables with device-capacity
and per-operation index bounds. Explicit-AH callers may allocate jobs
with zero entries; release stored addresses when the job is destroyed.

checkpatch --strict (100 columns) still reports the following in this
patch. They need coordination with the internal repositories this code
is maintained in and will be fixed in V2:
- CHECK: PARENTHESIS_ALIGNMENT (13), OPEN_ENDED_LINE (3), LINE_SPACING
  (1)

Co-developed-by: Iraimani P <iraimani.pavadai@hpe.com>
Signed-off-by: Iraimani P <iraimani.pavadai@hpe.com>
Signed-off-by: Doug Ledford <doug.ledford@hpe.com>
---
 drivers/infiniband/hw/cxi/cxiverbs_main.c | 1105 +++++++++++++++++++++++++++++
 1 file changed, 1105 insertions(+)

diff --git a/drivers/infiniband/hw/cxi/cxiverbs_main.c b/drivers/infiniband/hw/cxi/cxiverbs_main.c
new file mode 100644
index 000000000000..19a796cd5c27
--- /dev/null
+++ b/drivers/infiniband/hw/cxi/cxiverbs_main.c
@@ -0,0 +1,1105 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * CXI InfiniBand RDMA driver — Phase 0-3 and Phase 5 implementations.
+ * Copyright 2022-2025 Hewlett Packard Enterprise Development LP
+ *
+ * Covers:
+ *   Phase 0: query_device, query_port, get_port_immutable, query_pkey
+ *   Phase 1: alloc/dealloc_ucontext, alloc/dealloc_pd
+ *   Phase 3: create/destroy_cq, poll_cq, req_notify_cq, mmap/mmap_free
+ *   Phase 5: create/destroy_ah, add/del_gid
+ */
+
+#include <linux/module.h>
+#include <linux/pci.h>
+#include <linux/slab.h>
+#include <linux/gfp.h>
+#include <linux/random.h>
+#include <linux/mm.h>
+#include <linux/vmalloc.h>
+#include <linux/io.h>
+#include <net/net_namespace.h>
+
+#include <rdma/ib_verbs.h>
+#include <rdma/ib_mad.h>
+#include <rdma/uverbs_ioctl.h>
+
+#include "cxiverbs.h"
+
+/* =======================================================================
+ * Phase 0: Device and port query
+ * =======================================================================
+ */
+
+int cxiverbs_query_device(struct ib_device *ibdev, struct ib_device_attr *props,
+			 struct ib_udata *udata)
+{
+	struct cxiverbs_dev *dev = to_cxiverbs_dev(ibdev);
+	const struct cxi_properties_info *p = &dev->cdev->prop;
+
+	memset(props, 0, sizeof(*props));
+
+	props->vendor_id      = dev->cadev->pdev->vendor;
+	props->vendor_part_id = dev->cadev->pdev->device;
+	props->hw_ver         = dev->cadev->pdev->subsystem_device;
+	props->fw_ver         = 0;
+
+	props->max_mr_size    = ~0ULL;
+	props->page_size_cap  = PAGE_SIZE;
+
+	/* Queue pair limits (bounded by portal table entries) */
+	props->max_qp     = min_t(u32, p->rsrcs.ptes.max, 1024);
+	props->max_qp_wr  = 1024;
+	props->device_cap_flags =
+		IB_DEVICE_MEM_MGT_EXTENSIONS | IB_DEVICE_RU |
+		IB_DEVICE_KEY64 | IB_DEVICE_USER_RKEY | IB_DEVICE_IMM64;
+	/*
+	 * No CXI initiator command carries a gather list and a list entry
+	 * describes one buffer, so a work request maps to exactly one segment.
+	 */
+	props->max_send_sge = 1;
+	props->max_recv_sge = 1;
+
+	/* Completion queue limits (bounded by event queue count) */
+	props->max_cq  = min_t(u32, p->rsrcs.eqs.max, 2048);
+	props->max_cqe = 65536;
+
+	/* Memory registration limits */
+	props->max_mr  = 32768;
+	props->max_pd  = 1024;
+	props->max_job_ids = CXI_IBV_JOB_ID_MAX;
+	props->max_job_keys = props->max_qp;
+	/* The table is sparse and software-managed; the API uses u32 indices. */
+	props->max_addr_entries = U32_MAX;
+
+	/* RDMA atomics */
+	props->max_qp_rd_atom      = 1;
+	props->max_qp_init_rd_atom = 1;
+	props->max_res_rd_atom     = 1024;
+	props->atomic_cap          = IB_ATOMIC_GLOB;
+
+	/* Pkeys / GIDs */
+	props->max_pkeys          = 1;
+	props->local_ca_ack_delay = 0;
+	props->max_sge_rd         = 1;
+
+	/* On-Demand Paging capabilities */
+	props->odp_caps.general_caps = IB_ODP_SUPPORT;
+	props->odp_caps.per_transport_caps.rc_odp_caps =
+		IB_ODP_SUPPORT_WRITE | IB_ODP_SUPPORT_READ | IB_ODP_SUPPORT_SEND;
+
+	/* Driver-specific UEC / vendor capabilities via udata */
+	if (udata) {
+		const struct cxi_dev_info_use *di = &dev->devinfo;
+		struct cxi_ibv_query_device_resp vresp = {
+			.features = CXI_DEVICE_CAP_UEC_QPN |
+				    CXI_DEVICE_CAP_TRIGGERED_OPS |
+				    CXI_DEVICE_CAP_ODP,
+			.max_ctn  = 1024,
+		};
+
+		/*
+		 * Device-wide properties normally read from native's
+		 * DEV_INFO_GET ioctl / sysfs tree — neither exists on the
+		 * RDMA path, so hand them back here instead (see the
+		 * cxi_ibv_devinfo comment in the uapi header).
+		 */
+		vresp.devinfo.nid                 = READ_ONCE(di->nid);
+		vresp.devinfo.pid_bits            = di->pid_bits;
+		vresp.devinfo.pid_count           = di->pid_count;
+		vresp.devinfo.pid_granule         = di->pid_granule;
+		vresp.devinfo.min_free_shift      = di->min_free_shift;
+		vresp.devinfo.rdzv_get_idx        = di->rdzv_get_idx;
+		vresp.devinfo.vendor_id           = di->vendor_id;
+		vresp.devinfo.device_id           = di->device_id;
+		vresp.devinfo.device_rev          = di->device_rev;
+		vresp.devinfo.device_proto        = di->device_proto;
+		vresp.devinfo.device_platform     = di->device_platform;
+		vresp.devinfo.pct_eq              = di->pct_eq;
+		vresp.devinfo.uc_nic              = di->uc_nic;
+		vresp.devinfo.link_mtu            = di->link_mtu;
+		vresp.devinfo.link_speed          = di->link_speed;
+		vresp.devinfo.link_state          = di->link_state;
+		vresp.devinfo.num_ptes            = di->num_ptes;
+		vresp.devinfo.num_txqs            = di->num_txqs;
+		vresp.devinfo.num_tgqs            = di->num_tgqs;
+		vresp.devinfo.num_eqs             = di->num_eqs;
+		vresp.devinfo.num_cts             = di->num_cts;
+		vresp.devinfo.num_acs             = di->num_acs;
+		vresp.devinfo.num_tles            = di->num_tles;
+		vresp.devinfo.num_les             = di->num_les;
+		vresp.devinfo.cassini_version     = di->cassini_version;
+		vresp.devinfo.vni_matching        = p->vni_matching;
+		vresp.devinfo.system_type_identifier = di->system_type_identifier;
+		BUILD_BUG_ON(sizeof(vresp.devinfo.fru_description) !=
+			     sizeof(di->fru_description));
+		memcpy(vresp.devinfo.fru_description, di->fru_description,
+		       sizeof(di->fru_description));
+
+		return ib_respond_udata(udata, vresp);
+	}
+
+	return 0;
+}
+
+int cxiverbs_query_port(struct ib_device *ibdev, u32 port,
+		       struct ib_port_attr *props)
+{
+	struct cxiverbs_dev *dev = to_cxiverbs_dev(ibdev);
+
+	if (port != 1)
+		return -EINVAL;
+
+	memset(props, 0, sizeof(*props));
+
+	props->state  = dev->cadev->is_link_up(dev->cdev) ? IB_PORT_ACTIVE
+						   : IB_PORT_DOWN;
+	props->phys_state = (props->state == IB_PORT_ACTIVE) ?
+		IB_PORT_PHYS_STATE_LINK_UP : IB_PORT_PHYS_STATE_POLLING;
+	props->max_mtu       = IB_MTU_4096;
+	props->active_mtu    = IB_MTU_4096;
+	props->gid_tbl_len   = 1;
+	props->pkey_tbl_len  = 1;
+	props->max_msg_sz    = 0x80000000U;
+	props->active_width  = IB_WIDTH_4X;
+	props->active_speed  = IB_SPEED_HDR; /* nearest IB speed for 200 Gbit */
+	props->port_cap_flags = IB_PORT_CM_SUP;
+
+	return 0;
+}
+
+struct net_device *cxiverbs_get_netdev(struct ib_device *ibdev, u32 port)
+{
+	struct cxiverbs_dev *dev = to_cxiverbs_dev(ibdev);
+
+	if (port != 1 || !dev->cdev->eth_name[0])
+		return NULL;
+
+	return dev_get_by_name(&init_net, dev->cdev->eth_name);
+}
+
+int cxiverbs_get_port_immutable(struct ib_device *ibdev, u32 port_num,
+			       struct ib_port_immutable *immutable)
+{
+	struct ib_port_attr attr;
+	int ret;
+
+	if (port_num != 1)
+		return -EINVAL;
+
+	ret = cxiverbs_query_port(ibdev, port_num, &attr);
+	if (ret)
+		return ret;
+
+	immutable->core_cap_flags = RDMA_CORE_PORT_SLINGSHOT;
+	immutable->gid_tbl_len    = attr.gid_tbl_len;
+	immutable->pkey_tbl_len   = attr.pkey_tbl_len;
+	immutable->max_mad_size   = 0;
+
+	return 0;
+}
+
+int cxiverbs_query_pkey(struct ib_device *device, u32 port_num,
+		       u16 index, u16 *pkey)
+{
+	if (index != 0)
+		return -EINVAL;
+	*pkey = 0xffff;
+	return 0;
+}
+
+int cxiverbs_query_gid(struct ib_device *device, u32 port_num, int index,
+		      union ib_gid *gid)
+{
+	struct cxiverbs_dev *dev = to_cxiverbs_dev(device);
+
+	if (port_num != 1 || index != 0)
+		return -EINVAL;
+
+	memset(gid, 0, sizeof(*gid));
+	gid->global.interface_id = cpu_to_be64(READ_ONCE(dev->nid));
+	return 0;
+}
+
+/* =======================================================================
+ * Phase 1: User context  ->  one JobID-anchored CXI LNI per process open
+ *
+ * GET_CONTEXT opens an unbound context. A later privileged Job allocation gets
+ * or creates the device-wide JobID entry and binds this context to that entry's
+ * LNI. Services are selected later, when QPs are created.
+ * =======================================================================
+ */
+
+static int cxiverbs_parse_ucontext_cmd(struct cxi_ibv_alloc_ucontext_cmd *cmd,
+					     struct ib_udata *udata)
+{
+	static const u32 known_mask = CXI_IBV_ALLOC_UCONTEXT_PIDONFEP;
+	size_t copy_len;
+	int ret;
+
+	if (!udata || !udata->inlen)
+		return 0;
+
+	copy_len = min_t(size_t, udata->inlen, sizeof(*cmd));
+	ret = ib_copy_from_udata(cmd, udata, copy_len);
+	if (ret)
+		return ret;
+
+	if (cmd->comp_mask & ~known_mask)
+		return -EOPNOTSUPP;
+	if ((cmd->comp_mask & CXI_IBV_ALLOC_UCONTEXT_PIDONFEP) &&
+	    cmd->pidonfep > CXI_IBV_PIDONFEP_MAX)
+		return -EINVAL;
+
+	return 0;
+}
+
+int cxiverbs_alloc_ucontext(struct ib_ucontext *uctx, struct ib_udata *udata)
+{
+	struct ib_device *ibdev   = uctx->device;
+	struct cxiverbs_dev *dev    = to_cxiverbs_dev(ibdev);
+	struct cxiverbs_ucontext *ctx = to_cxiverbs_ucontext(uctx);
+	struct cxi_adev *cadev    = dev->cadev;
+	struct cxi_ibv_alloc_ucontext_cmd  cmd  = {};
+	struct cxi_ibv_alloc_ucontext_resp resp = {};
+	int ret;
+	int i;
+
+	ctx->cadev = cadev;
+	xa_init(&ctx->lnis);
+	mutex_init(&ctx->resource_lock);
+	ctx->lni = NULL;
+	ctx->job_id = 0;
+	ctx->default_pidonfep = 0;
+	ctx->default_pidonfep_valid = false;
+
+	/* Parse launch defaults; jobs and LNIs are allocated explicitly. */
+	ret = cxiverbs_parse_ucontext_cmd(&cmd, udata);
+	if (ret) {
+		mutex_destroy(&ctx->resource_lock);
+		xa_destroy(&ctx->lnis);
+		return ret;
+	}
+
+	if (cmd.comp_mask & CXI_IBV_ALLOC_UCONTEXT_PIDONFEP) {
+		ctx->default_pidonfep = cmd.pidonfep;
+		ctx->default_pidonfep_valid = true;
+	}
+
+	/* Initialize TC limits (placeholder; typically populated by service query) */
+	mutex_init(&ctx->tc_lock);
+	ctx->num_tc_limits = 0;
+	/* TODO: Query service for TC limits and populate tc_limits[] */
+	for (i = 0; i < CXIVERBS_TC_MAX; i++) {
+		ctx->tc_limits[i].tc = i;
+		ctx->tc_limits[i].max_ptes = 0;	/* will be filled by service query */
+		ctx->tc_limits[i].max_cps = 0;
+		ctx->tc_limits[i].max_qps = 0;
+	}
+
+	/* Populate response */
+	resp.nid         = READ_ONCE(dev->nid);
+	resp.pid_bits    = dev->pid_bits;
+	resp.pid_count   = dev->pid_count;
+	resp.pid_granule = dev->pid_granule;
+	resp.lni_id      = ctx->lni ? ctx->lni->id : 0;
+	resp.num_tc_limits = ctx->num_tc_limits;
+
+	if (udata) {
+		ret = ib_respond_udata(udata, resp);
+		if (ret)
+			goto err_destroy_context;
+	}
+
+	return 0;
+
+err_destroy_context:
+	xa_destroy(&ctx->lnis);
+	mutex_destroy(&ctx->resource_lock);
+	mutex_destroy(&ctx->tc_lock);
+	return ret;
+}
+
+int cxiverbs_alloc_job(struct ib_job *job, struct ib_job_attr *attr,
+		       struct uverbs_attr_bundle *attrs)
+{
+	struct cxiverbs_dev *dev = to_cxiverbs_dev(job->device);
+	struct cxiverbs_job_entry *entry =
+		container_of(job, struct cxiverbs_job_entry, ibjob);
+	int ret;
+
+	if (!capable(CAP_NET_ADMIN))
+		return -EPERM;
+	if (!attr->job_id || attr->job_id > CXI_IBV_JOB_ID_MAX ||
+	    attr->port_num != 1 || attr->sgid_index)
+		return -EINVAL;
+	if (attr->max_addr_entries > dev->ib_dev.attrs.max_addr_entries)
+		return -EINVAL;
+
+	entry->job_id = attr->job_id;
+	entry->attr = *attr;
+	mutex_init(&entry->addr_lock);
+	xa_init(&entry->addr_table);
+	mutex_lock(&dev->jobs_lock);
+	ret = xa_insert(&dev->jobs, entry->job_id, entry, GFP_KERNEL);
+	mutex_unlock(&dev->jobs_lock);
+	if (ret) {
+		xa_destroy(&entry->addr_table);
+		mutex_destroy(&entry->addr_lock);
+	}
+	return ret;
+}
+
+int cxiverbs_query_job(struct ib_job *job, struct ib_job_attr *attr)
+{
+	struct cxiverbs_job_entry *entry =
+		container_of(job, struct cxiverbs_job_entry, ibjob);
+
+	*attr = entry->attr;
+	return 0;
+}
+
+int cxiverbs_job_addr_insert(struct ib_job *job, u32 index,
+			     struct rdma_ah_attr_ex *ah_attr)
+{
+	struct cxiverbs_job_entry *entry =
+		container_of(job, struct cxiverbs_job_entry, ibjob);
+	struct rdma_ah_attr_ex *copy;
+	int ret;
+
+	if (!ah_attr || index >= job->max_addr_entries)
+		return -EINVAL;
+	copy = kmemdup(ah_attr, sizeof(*copy), GFP_KERNEL);
+	if (!copy)
+		return -ENOMEM;
+
+	mutex_lock(&entry->addr_lock);
+	ret = xa_insert(&entry->addr_table, index, copy, GFP_KERNEL);
+	mutex_unlock(&entry->addr_lock);
+	if (ret)
+		kfree(copy);
+	return ret;
+}
+
+int cxiverbs_job_addr_remove(struct ib_job *job, u32 index)
+{
+	struct cxiverbs_job_entry *entry =
+		container_of(job, struct cxiverbs_job_entry, ibjob);
+	struct rdma_ah_attr_ex *addr;
+
+	if (index >= job->max_addr_entries)
+		return -EINVAL;
+	mutex_lock(&entry->addr_lock);
+	addr = xa_erase(&entry->addr_table, index);
+	mutex_unlock(&entry->addr_lock);
+	if (!addr)
+		return -ENOENT;
+	kfree(addr);
+	return 0;
+}
+
+int cxiverbs_job_addr_query(struct ib_job *job, u32 index,
+			    struct rdma_ah_attr_ex *ah_attr)
+{
+	struct cxiverbs_job_entry *entry =
+		container_of(job, struct cxiverbs_job_entry, ibjob);
+	struct rdma_ah_attr_ex *addr;
+
+	if (!ah_attr || index >= job->max_addr_entries)
+		return -EINVAL;
+	mutex_lock(&entry->addr_lock);
+	addr = xa_load(&entry->addr_table, index);
+	if (addr)
+		*ah_attr = *addr;
+	mutex_unlock(&entry->addr_lock);
+	return addr ? 0 : -ENOENT;
+}
+
+void cxiverbs_dealloc_job(struct ib_job *job)
+{
+	struct cxiverbs_dev *dev = to_cxiverbs_dev(job->device);
+	struct cxiverbs_job_entry *entry =
+		container_of(job, struct cxiverbs_job_entry, ibjob);
+	struct cxiverbs_tc_entry *tc_entry;
+	struct cxiverbs_job_tc_binding *binding, *tmp;
+	struct rdma_ah_attr_ex *addr;
+	unsigned long index;
+
+	mutex_lock(&dev->jobs_lock);
+	xa_erase(&dev->jobs, entry->job_id);
+	xa_for_each(&dev->tcs, index, tc_entry) {
+		list_for_each_entry_safe(binding, tmp, &tc_entry->job_list, entry) {
+			if (binding->job != entry)
+				continue;
+			list_del(&binding->entry);
+			kfree(binding);
+		}
+	}
+	mutex_unlock(&dev->jobs_lock);
+	mutex_lock(&entry->addr_lock);
+	xa_for_each(&entry->addr_table, index, addr)
+		kfree(addr);
+	xa_destroy(&entry->addr_table);
+	mutex_unlock(&entry->addr_lock);
+	mutex_destroy(&entry->addr_lock);
+}
+
+void cxiverbs_dealloc_ucontext(struct ib_ucontext *uctx)
+{
+	struct cxiverbs_ucontext *ctx = to_cxiverbs_ucontext(uctx);
+
+	xa_destroy(&ctx->lnis);
+	mutex_destroy(&ctx->resource_lock);
+	mutex_destroy(&ctx->tc_lock);
+}
+
+/* =======================================================================
+ * Protection domain -> selected explicit LNI + RGID
+ *
+ * PDs select an LNI from the ucontext's handle table and carry PD-local
+ * defaults such as preferred service ID and PIDonFEP.
+ * =======================================================================
+ */
+
+
+int cxiverbs_alloc_pd(struct ib_pd *ibpd, struct ib_udata *udata)
+{
+	struct cxiverbs_ucontext *uctxt;
+	struct cxiverbs_pd *pd          = to_cxiverbs_pd(ibpd);
+	struct cxi_ibv_alloc_pd_cmd  cmd  = {};
+	struct cxi_ibv_alloc_pd_resp resp = {};
+	int ret;
+
+	if (!ibpd->uobject)
+		return -EOPNOTSUPP;
+	uctxt = to_cxiverbs_ucontext(ibpd->uobject->context);
+
+	/* Parse optional preferred_svc_id from user */
+	if (udata && udata->inlen >= sizeof(cmd)) {
+		ret = ib_copy_from_udata(&cmd, udata, sizeof(cmd));
+		if (ret)
+			return ret;
+	}
+
+	if (cmd.comp_mask & ~(CXI_IBV_ALLOC_PD_LNI |
+			      CXI_IBV_ALLOC_PD_SVC_ID_PREFERRED))
+		return -EOPNOTSUPP;
+	pd->lni_ref = cxiverbs_lni_get(uctxt,
+		(cmd.comp_mask & CXI_IBV_ALLOC_PD_LNI) ?
+		cmd.lni_handle : CXI_IBV_LNI_ANY);
+	if (IS_ERR(pd->lni_ref))
+		return PTR_ERR(pd->lni_ref);
+
+	if ((cmd.comp_mask & CXI_IBV_ALLOC_PD_SVC_ID_PREFERRED) &&
+	    cmd.preferred_svc_id &&
+	    cmd.preferred_svc_id != pd->lni_ref->svc_id) {
+		cxiverbs_lni_put(pd->lni_ref);
+		return -EINVAL;
+	}
+
+	pd->job_id = uctxt->job_id;
+	pd->preferred_svc_id = pd->lni_ref->svc_id;
+	pd->rgid = pd->lni_ref->lni->rgid;
+	pd->pd_id = pd->lni_ref->lni->id;
+	pd->qpn_addr_mode = 0;
+
+	/* Inherit PIDonFEP defaults from ucontext if available */
+	pd->default_pidonfep = uctxt->default_pidonfep;
+	pd->default_pidonfep_valid = uctxt->default_pidonfep_valid;
+	pd->pidonfep_valid = false;
+
+	mutex_init(&pd->resource_lock);
+	xa_init_flags(&pd->qpns, XA_FLAGS_ALLOC);
+
+	resp.pd_id = pd->pd_id;
+	if (udata) {
+		ret = ib_respond_udata(udata, resp);
+		if (ret) {
+			mutex_destroy(&pd->resource_lock);
+			xa_destroy(&pd->qpns);
+			cxiverbs_lni_put(pd->lni_ref);
+			return ret;
+		}
+	}
+
+	return 0;
+}
+
+int cxiverbs_dealloc_pd(struct ib_pd *ibpd, struct ib_udata *udata)
+{
+	struct cxiverbs_pd *pd = to_cxiverbs_pd(ibpd);
+	struct cxiverbs_dev *dev = to_cxiverbs_dev(ibpd->device);
+
+	if (pd->pidonfep_valid)
+		dev->cadev->pid_free(pd->lni_ref->lni, pd->pidonfep);
+	xa_destroy(&pd->qpns);
+	mutex_destroy(&pd->resource_lock);
+	cxiverbs_lni_put(pd->lni_ref);
+	return 0;
+}
+
+int cxiverbs_create_jkey(struct ib_jkey *jkey,
+			 struct uverbs_attr_bundle *attrs)
+{
+	struct cxiverbs_jkey *key =
+		container_of(jkey, struct cxiverbs_jkey, ibjkey);
+	struct cxiverbs_pd *pd = to_cxiverbs_pd(jkey->pd);
+	struct cxiverbs_dev *dev = to_cxiverbs_dev(jkey->device);
+	struct cxi_ibv_create_jkey_cmd cmd = { .tc = CXI_TC_BEST_EFFORT };
+	int ret;
+
+	if (!pd->lni_ref)
+		return -EINVAL;
+	if (attrs->driver_udata.inlen >= sizeof(cmd)) {
+		ret = ib_copy_from_udata(&cmd, &attrs->driver_udata, sizeof(cmd));
+		if (ret)
+			return ret;
+	}
+	key->cp = dev->cadev->cp_alloc(pd->lni_ref->lni, jkey->job->job_id,
+				     cmd.tc, CXI_TC_TYPE_DEFAULT);
+	if (IS_ERR(key->cp))
+		return PTR_ERR(key->cp);
+	jkey->jkey = key->cp->lcid;
+	return 0;
+}
+
+int cxiverbs_destroy_jkey(struct ib_jkey *jkey,
+			  struct uverbs_attr_bundle *attrs)
+{
+	struct cxiverbs_jkey *key =
+		container_of(jkey, struct cxiverbs_jkey, ibjkey);
+	struct cxiverbs_dev *dev = to_cxiverbs_dev(jkey->device);
+
+	dev->cadev->cp_free(key->cp);
+	return 0;
+}
+
+/* =======================================================================
+ * Phase 3: Completion queue  →  CXI Event Queue (EQ)
+ *
+ * The EQ ring buffer is allocated from compound pages, pinned in RAM, and
+ * registered with the ATU so hardware can write events directly.  Both the
+ * ring buffer and the doorbell CSR are exposed to user-space via mmap.
+ * =======================================================================
+ */
+
+/*
+ * EQ event callback — called from the CXI interrupt handler.
+ * Notifies the RDMA core that new completions are available.
+ */
+struct cxiverbs_cq_resources {
+	refcount_t refs;
+	spinlock_t callback_lock; /* protects cq during callback delivery */
+	struct cxiverbs_cq *cq;
+	struct cxiverbs_lni *lni;
+	struct cxi_adev *cadev;
+	struct cxi_eq *eq;
+	struct cxi_md *eq_md;
+	struct page **pages;
+	unsigned int order;
+};
+
+static void cxiverbs_eq_event_cb(void *cb_data)
+{
+	struct cxiverbs_cq_resources *resources = cb_data;
+	struct cxiverbs_cq *cq;
+	unsigned long flags;
+
+	spin_lock_irqsave(&resources->callback_lock, flags);
+	cq = resources->cq;
+	if (cq && cq->ibcq.comp_handler)
+		cq->ibcq.comp_handler(&cq->ibcq, cq->ibcq.cq_context);
+	spin_unlock_irqrestore(&resources->callback_lock, flags);
+}
+
+/*
+ * Allocate a heap-allocated cxiverbs_mmap_entry and register it for mmap.
+ * Returns ERR_PTR on failure.
+ */
+static struct cxiverbs_mmap_entry *
+cxiverbs_mmap_entry_create(struct ib_ucontext *ibctx, bool is_io,
+			  bool admin_only, phys_addr_t phys,
+			  struct page *first_page, size_t size, u64 *offset_out,
+			  void *owner, void (*release_owner)(void *))
+{
+	struct cxiverbs_mmap_entry *ent;
+	int ret;
+
+	ent = kzalloc_obj(*ent);
+	if (!ent)
+		return ERR_PTR(-ENOMEM);
+
+	ent->is_io      = is_io;
+	ent->admin_only = admin_only;
+	ent->phys       = phys;
+	ent->first_page = first_page;
+	ent->owner = owner;
+	ent->release_owner = release_owner;
+
+	ret = rdma_user_mmap_entry_insert(ibctx, &ent->rdma_entry, size);
+	if (ret) {
+		kfree(ent);
+		return ERR_PTR(ret);
+	}
+
+	*offset_out = rdma_user_mmap_get_offset(&ent->rdma_entry);
+	return ent;
+}
+
+struct cxiverbs_mmap_entry *
+cxiverbs_mmap_entry_alloc(struct ib_ucontext *ibctx, bool is_io,
+			 phys_addr_t phys, struct page *first_page,
+			 size_t size, u64 *offset_out)
+{
+	return cxiverbs_mmap_entry_create(ibctx, is_io, false, phys, first_page,
+					  size, offset_out, NULL, NULL);
+}
+
+struct cxiverbs_mmap_entry *
+cxiverbs_mmap_entry_alloc_owned(struct ib_ucontext *ibctx, bool is_io,
+			       phys_addr_t phys, struct page *first_page,
+			       size_t size, u64 *offset_out, void *owner,
+			       void (*release_owner)(void *))
+{
+	return cxiverbs_mmap_entry_create(ibctx, is_io, false, phys, first_page,
+					  size, offset_out, owner,
+					  release_owner);
+}
+
+struct cxiverbs_mmap_entry *
+cxiverbs_mmap_entry_alloc_admin(struct ib_ucontext *ibctx, phys_addr_t phys,
+			       size_t size, u64 *offset_out)
+{
+	return cxiverbs_mmap_entry_create(ibctx, true, true, phys, NULL,
+					  size, offset_out, NULL, NULL);
+}
+
+static void cxiverbs_cq_free_ring(struct page *first, unsigned int order)
+{
+	unsigned long i;
+
+	for (i = 0; i < (1UL << order); i++)
+		__free_page(first + i);
+}
+
+static void cxiverbs_cq_resources_put(void *owner)
+{
+	struct cxiverbs_cq_resources *resources = owner;
+
+	if (!refcount_dec_and_test(&resources->refs))
+		return;
+	if (resources->eq && WARN_ON(resources->cadev->eq_free(resources->eq)))
+		return;
+	if (resources->eq_md &&
+	    WARN_ON(resources->cadev->cxi_unmap(resources->eq_md)))
+		return;
+	if (resources->pages) {
+		if (resources->pages[0])
+			cxiverbs_cq_free_ring(resources->pages[0], resources->order);
+		kfree(resources->pages);
+	}
+	cxiverbs_lni_put(resources->lni);
+	kfree(resources);
+}
+
+static void cxiverbs_cq_release(struct cxiverbs_cq *cq)
+{
+	struct cxiverbs_cq_resources *resources = cq->resources;
+	unsigned long flags;
+
+	spin_lock_irqsave(&resources->callback_lock, flags);
+	resources->cq = NULL;
+	spin_unlock_irqrestore(&resources->callback_lock, flags);
+
+	if (cq->db_mmap)
+		rdma_user_mmap_entry_remove(&cq->db_mmap->rdma_entry);
+	if (cq->buf_mmap)
+		rdma_user_mmap_entry_remove(&cq->buf_mmap->rdma_entry);
+	cq->resources = NULL;
+	cxiverbs_cq_resources_put(resources);
+}
+
+int cxiverbs_create_cq(struct ib_cq *ibcq, const struct ib_cq_init_attr *attr,
+		      struct uverbs_attr_bundle *bundle)
+{
+	struct cxiverbs_cq *cq      = to_cxiverbs_cq(ibcq);
+	struct cxiverbs_dev *dev    = to_cxiverbs_dev(ibcq->device);
+	(void)dev; /* reserved for future use */
+	struct ib_udata *udata;
+	struct cxi_ibv_create_cq_cmd cmd = { .lni_handle = CXI_IBV_LNI_ANY };
+	struct cxi_ibv_create_cq_resp resp = {};
+	struct cxiverbs_ucontext *ctx;
+	struct cxi_eq_attr eq_attr = {};
+	unsigned int n_entries     = attr->cqe;
+	unsigned int cpu_aff       = 0;
+	size_t buf_sz;
+	unsigned int npages;
+	int order, i, ret;
+	phys_addr_t db_phys;
+	size_t db_size;
+
+	/* Kernel CQs are not yet supported */
+	if (!bundle || !bundle->context)
+		return -EOPNOTSUPP;
+	udata = &bundle->driver_udata;
+	ctx = to_cxiverbs_ucontext(bundle->context);
+
+	cq->cadev = ctx->cadev;
+
+	/* Optional driver-private fields from user-space */
+	if (udata && udata->inlen >= sizeof(cmd)) {
+		ret = ib_copy_from_udata(&cmd, udata, sizeof(cmd));
+		if (ret)
+			return ret;
+		if (cmd.n_entries)
+			n_entries = cmd.n_entries;
+		cpu_aff = cmd.cpu_affinity;
+	}
+	cq->lni_ref = cxiverbs_lni_get(ctx, cmd.lni_handle);
+	if (IS_ERR(cq->lni_ref))
+		return PTR_ERR(cq->lni_ref);
+
+	cq->resources = kzalloc_obj(*cq->resources);
+	if (!cq->resources) {
+		ret = -ENOMEM;
+		goto err_put_lni;
+	}
+	refcount_set(&cq->resources->refs, 1);
+	spin_lock_init(&cq->resources->callback_lock);
+	cq->resources->cq = cq;
+	cq->resources->lni = cq->lni_ref;
+	cq->resources->cadev = cq->cadev;
+
+	/* Clamp and round up to hardware constraints */
+	n_entries = min_t(unsigned int, n_entries, 65536);
+	n_entries = max_t(unsigned int, n_entries, 64);
+	n_entries = roundup_pow_of_two(n_entries);
+
+	/* Each event slot is C_EE_CFG_ECB_SIZE bytes */
+	buf_sz  = (size_t)n_entries * C_EE_CFG_ECB_SIZE;
+	buf_sz  = PAGE_ALIGN(buf_sz);
+	npages  = buf_sz >> PAGE_SHIFT;
+	order   = get_order(buf_sz);
+	cq->resources->order = order;
+
+	/* Allocate and zero the event ring buffer */
+	cq->pages = kcalloc(npages, sizeof(*cq->pages), GFP_KERNEL);
+	cq->resources->pages = cq->pages;
+	if (!cq->pages) {
+		ret = -ENOMEM;
+		goto err_release;
+	}
+
+	cq->pages[0] = alloc_pages(GFP_KERNEL | __GFP_ZERO, order);
+	if (!cq->pages[0]) {
+		ret = -ENOMEM;
+		goto err_release;
+	}
+	split_page(cq->pages[0], order);
+	for (i = 1; i < (int)npages; i++)
+		cq->pages[i] = cq->pages[0] + i;
+
+	cq->eq_buf      = page_address(cq->pages[0]);
+	cq->eq_buf_size = buf_sz;
+	cq->npages      = npages;
+
+	/* Register with the ATU so hardware can DMA events into the buffer */
+	cq->eq_md = ctx->cadev->cxi_map(cq->lni_ref->lni,
+			    (uintptr_t)cq->eq_buf,
+			    buf_sz, CXI_MAP_PIN | CXI_MAP_WRITE | CXI_MAP_READ,
+			    NULL);
+	if (IS_ERR(cq->eq_md)) {
+		ret = PTR_ERR(cq->eq_md);
+		cq->eq_md = NULL;
+		goto err_release;
+	}
+	cq->resources->eq_md = cq->eq_md;
+
+	/* Configure and allocate the EQ */
+	eq_attr.queue               = cq->eq_buf;
+	eq_attr.queue_len           = buf_sz;
+	eq_attr.cpu_affinity        = cpu_aff;
+	eq_attr.status_thresh_base  = 95;
+	eq_attr.status_thresh_delta = 15;
+	eq_attr.status_thresh_count = 2;
+
+	cq->eq = ctx->cadev->eq_alloc(cq->lni_ref->lni, cq->eq_md, &eq_attr,
+			      cxiverbs_eq_event_cb, cq->resources, NULL, NULL);
+	if (IS_ERR(cq->eq)) {
+		ret = PTR_ERR(cq->eq);
+		cq->eq = NULL;
+		goto err_release;
+	}
+	cq->resources->eq = cq->eq;
+
+	/* Get the doorbell CSR address for user mmap */
+	ret = ctx->cadev->eq_user_info(cq->eq, &db_phys, &db_size);
+	if (ret)
+		goto err_release;
+	cq->db_phys = db_phys;
+	cq->db_size = db_size;
+
+	/* Register mmap entry for the event ring buffer (normal pages) */
+	refcount_inc(&cq->resources->refs);
+	cq->buf_mmap = cxiverbs_mmap_entry_alloc_owned(bundle->context,
+						       false, 0, cq->pages[0], buf_sz,
+						       &resp.eq_buf_offset, cq->resources,
+						       cxiverbs_cq_resources_put);
+	if (IS_ERR(cq->buf_mmap)) {
+		ret = PTR_ERR(cq->buf_mmap);
+		cq->buf_mmap = NULL;
+		refcount_dec(&cq->resources->refs);
+		goto err_release;
+	}
+	resp.eq_buf_size = buf_sz;
+
+	/* Register mmap entry for the doorbell CSR (IO memory) */
+	refcount_inc(&cq->resources->refs);
+	cq->db_mmap = cxiverbs_mmap_entry_alloc_owned(bundle->context,
+						      true, db_phys, NULL, db_size,
+						      &resp.eq_db_offset, cq->resources,
+						      cxiverbs_cq_resources_put);
+	if (IS_ERR(cq->db_mmap)) {
+		ret = PTR_ERR(cq->db_mmap);
+		cq->db_mmap = NULL;
+		refcount_dec(&cq->resources->refs);
+		goto err_release;
+	}
+	resp.eq_db_size = db_size;
+	resp.eqn        = cq->eq->eqn;
+
+	if (udata) {
+		ret = ib_respond_udata(udata, resp);
+		if (ret)
+			goto err_release;
+	}
+
+	return 0;
+
+err_release:
+	cxiverbs_cq_release(cq);
+	return ret;
+err_put_lni:
+	cxiverbs_lni_put(cq->lni_ref);
+	return ret;
+}
+
+int cxiverbs_destroy_cq(struct ib_cq *ibcq, struct ib_udata *udata)
+{
+	struct cxiverbs_cq *cq = to_cxiverbs_cq(ibcq);
+
+	cxiverbs_cq_release(cq);
+	return 0;
+}
+
+/*
+ * Translate a CXI return code to an IB work completion status.
+ */
+static enum ib_wc_status cxi_rc_to_wc_status(int rc)
+{
+	switch (rc) {
+	case C_RC_OK:			return IB_WC_SUCCESS;
+	case C_RC_PERM_VIOLATION:
+	case C_RC_OP_VIOLATION:		return IB_WC_REM_ACCESS_ERR;
+	case C_RC_NO_MATCH:		return IB_WC_REM_OP_ERR;
+	case C_RC_NO_TRANSLATION:
+	case C_RC_PAGE_PERM_ERROR:	return IB_WC_LOC_PROT_ERR;
+	case C_RC_DROPPED:
+	case C_RC_UNDELIVERABLE:	return IB_WC_RETRY_EXC_ERR;
+	case C_RC_AMO_INVAL_OP_ERROR:
+	case C_RC_AMO_ALIGN_ERROR:
+	case C_RC_AMO_FP_INVALID:	return IB_WC_LOC_QP_OP_ERR;
+	default:			return IB_WC_GENERAL_ERR;
+	}
+}
+
+int cxiverbs_poll_cq(struct ib_cq *ibcq, int ne, struct ib_wc *wc)
+{
+	struct cxiverbs_cq *cq = to_cxiverbs_cq(ibcq);
+	const union c_event *ev;
+	bool consumed = false;
+	int npolled = 0;
+
+	while (npolled < ne) {
+		ev = cq->cadev->eq_get_event(cq->eq);
+		if (!ev)
+			break;
+
+		consumed = true;
+		memset(&wc[npolled], 0, sizeof(wc[npolled]));
+
+		switch (ev->hdr.event_type) {
+		case C_EVENT_SEND:
+		case C_EVENT_ACK:
+		case C_EVENT_REPLY:
+			/* Initiator completion (post_send) */
+			wc[npolled].wr_id  = ev->init_short.user_ptr;
+			wc[npolled].status = cxi_rc_to_wc_status(
+				cxi_init_event_rc(ev));
+			if (ev->hdr.event_type == C_EVENT_REPLY)
+				wc[npolled].opcode = IB_WC_RDMA_READ;
+			else
+				wc[npolled].opcode = IB_WC_SEND;
+			if (wc[npolled].status == IB_WC_SUCCESS)
+				wc[npolled].byte_len =
+					(ev->hdr.event_size == C_EVENT_SIZE_16_BYTE) ?
+					ev->init_short.mlength : 0;
+			break;
+
+		case C_EVENT_PUT:
+		case C_EVENT_GET:
+			/* Target completion (post_recv) */
+			wc[npolled].wr_id  = 0;
+			wc[npolled].status = cxi_rc_to_wc_status(
+				cxi_tgt_event_rc(ev));
+			wc[npolled].opcode = IB_WC_RECV;
+			if (wc[npolled].status == IB_WC_SUCCESS &&
+			    ev->hdr.event_size >= C_EVENT_SIZE_32_BYTE)
+				wc[npolled].byte_len = ev->tgt_short.length;
+			break;
+
+		case C_EVENT_ATOMIC:
+		case C_EVENT_FETCH_ATOMIC:
+			wc[npolled].wr_id  = ev->init_short.user_ptr;
+			wc[npolled].status = cxi_rc_to_wc_status(
+				cxi_init_event_rc(ev));
+			wc[npolled].opcode = IB_WC_FETCH_ADD;
+			break;
+
+		case C_EVENT_COMMAND_FAILURE:
+			wc[npolled].status     = IB_WC_LOC_QP_OP_ERR;
+			wc[npolled].vendor_err = ev->cmd_fail.return_code;
+			wc[npolled].opcode     = IB_WC_SEND;
+			break;
+
+		default:
+			/* Administrative / unknown event: consume and skip */
+			continue;
+		}
+
+		npolled++;
+	}
+
+	if (consumed)
+		cxi_eq_ack_events(cq->eq);
+
+	return npolled;
+}
+
+int cxiverbs_req_notify_cq(struct ib_cq *ibcq, enum ib_cq_notify_flags flags)
+{
+	/*
+	 * The EQ interrupt is wired via event_cb in create_cq; the RDMA core
+	 * will re-check the queue after arming.  Always report success.
+	 */
+	return 0;
+}
+
+/* =======================================================================
+ * Phase 3: mmap  —  map EQ ring buffer and doorbell to user space
+ * =======================================================================
+ */
+
+int cxiverbs_mmap(struct ib_ucontext *ibctx, struct vm_area_struct *vma)
+{
+	struct rdma_user_mmap_entry *rdma_entry;
+	struct cxiverbs_mmap_entry *ent;
+	unsigned long size = vma->vm_end - vma->vm_start;
+	int ret;
+
+	rdma_entry = rdma_user_mmap_entry_get(ibctx, vma);
+	if (!rdma_entry)
+		return -EINVAL;
+
+	ent = container_of(rdma_entry, struct cxiverbs_mmap_entry, rdma_entry);
+
+	if (ent->admin_only && !capable(CAP_SYS_ADMIN)) {
+		ret = -EPERM;
+		goto out;
+	}
+
+	if (ent->is_io) {
+		/* Doorbell CSR: uncached IO mapping */
+		ret = rdma_user_mmap_io(ibctx, vma,
+					ent->phys >> PAGE_SHIFT,
+					size,
+					pgprot_noncached(vma->vm_page_prot),
+					rdma_entry);
+	} else {
+		/* Ring buffer: normal page mapping */
+		ret = rdma_user_mmap_io(ibctx, vma,
+					page_to_pfn(ent->first_page), size,
+					vma->vm_page_prot, rdma_entry);
+	}
+
+out:
+	rdma_user_mmap_entry_put(rdma_entry);
+	return ret;
+}
+
+void cxiverbs_mmap_free(struct rdma_user_mmap_entry *rdma_entry)
+{
+	struct cxiverbs_mmap_entry *ent =
+		container_of(rdma_entry, struct cxiverbs_mmap_entry, rdma_entry);
+	if (ent->release_owner)
+		ent->release_owner(ent->owner);
+	kfree(ent);
+}
+
+/* =======================================================================
+ * Phase 5: Address handle  —  NID resolution from GID
+ * =======================================================================
+ */
+
+/*
+ * Extract the Cassini NID from a GID. Anything set outside CXI_IBV_NID_MASK
+ * belongs to some other fabric's addressing, so reject it rather than
+ * truncating it into a NID that names a different node.
+ */
+static int cxi_gid_to_nid(const union ib_gid *gid, u32 *nid)
+{
+	u64 id = be64_to_cpu(gid->global.interface_id);
+
+	if (gid->global.subnet_prefix || (id & ~CXI_IBV_NID_MASK))
+		return -EINVAL;
+
+	*nid = (u32)id;
+	return 0;
+}
+
+int cxiverbs_create_ah(struct ib_ah *ibah, struct rdma_ah_init_attr *init_attr,
+		      struct ib_udata *udata)
+{
+	struct cxiverbs_ah *ah             = to_cxiverbs_ah(ibah);
+	const struct rdma_ah_attr *attr  = init_attr->ah_attr;
+	const struct ib_global_route *grh = rdma_ah_read_grh(attr);
+	int ret;
+
+	if (!grh)
+		return -EINVAL;
+
+	ret = cxi_gid_to_nid(&grh->dgid, &ah->nid);
+	if (ret)
+		return ret;
+
+	ah->dscp = rdma_ah_get_sl(attr); /* SL used as DSCP / TC class index */
+	ah->vni  = 0;                    /* resolved from QP at post_send time */
+	return 0;
+}
+
+int cxiverbs_destroy_ah(struct ib_ah *ibah, u32 flags)
+{
+	/* Nothing to free; struct is embedded in the slab allocation */
+	return 0;
+}
+
+int cxiverbs_add_gid(const struct ib_gid_attr *attr, void **context)
+{
+	/* Accept any GID; no hardware GID table maintenance at this layer */
+	return 0;
+}
+
+int cxiverbs_del_gid(const struct ib_gid_attr *attr, void **context)
+{
+	return 0;
+}

-- 
2.56.0

[PATCH 07/11] RDMA/cxi: add VNI-isolated queue pairs

[Doug Ledford]

Add queue pairs for the RU transport.  A QP is a portal table entry
plus the command and event queues behind it, created in the INIT
state and enabled on the transition to RTR.  Messages are matched on
the packet VNI as well as the key, so two jobs sharing a NIC cannot
match each other's receive buffers or memory regions.  The VNI match
is enabled only on Cassini 2 with the default VNI matching selector;
other configurations are rejected.

modify_qp() follows the UET v2 flow: an RU QP is created ready to send
and the initial modify goes through the v2 path.

If copying the response to user space fails on INIT to RTR after the
PTE has been mapped, the PTE is unmapped and its map index reset.
Without that a retried transition mapped the PTE a second time and
leaked the first slot in the portal list.

Responses use ib_respond_udata().

checkpatch --strict (100 columns) still reports the following in this
patch. They need coordination with the internal repositories this code
is maintained in and will be fixed in V2:
- CHECK: PARENTHESIS_ALIGNMENT (6), PREFER_KERNEL_TYPES (1)

Co-developed-by: Mamila Santosh <santosh-kumar.mamila@hpe.com>
Signed-off-by: Mamila Santosh <santosh-kumar.mamila@hpe.com>
Co-developed-by: Iraimani P <iraimani.pavadai@hpe.com>
Signed-off-by: Iraimani P <iraimani.pavadai@hpe.com>
Signed-off-by: Doug Ledford <doug.ledford@hpe.com>
---
 drivers/infiniband/hw/cxi/cxiverbs_qp.c | 949 ++++++++++++++++++++++++++++++++
 1 file changed, 949 insertions(+)

diff --git a/drivers/infiniband/hw/cxi/cxiverbs_qp.c b/drivers/infiniband/hw/cxi/cxiverbs_qp.c
new file mode 100644
index 000000000000..3875a5de0a98
--- /dev/null
+++ b/drivers/infiniband/hw/cxi/cxiverbs_qp.c
@@ -0,0 +1,949 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * CXI InfiniBand RDMA driver — Queue Pair lifecycle (Phase 4).
+ * Copyright 2025 Hewlett Packard Enterprise Development LP
+ *
+ * Each UEC/CXI Queue Pair allocates the following hardware resources:
+ *
+ *   cxi_domain   — VNI + PID  (fabric endpoint address)
+ *   cxi_cp       — Communication Profile (maps to LCID / traffic class)
+ *   cxi_cq (TXQ) — transmit command queue (initiator path)
+ *   cxi_cq (TGQ) — target command queue   (receive path)
+ *   cxi_pte      — Portal Table Entry     (receive matching)
+ *
+ * QPN encoding:
+ *   bit  31     : addr_mode (1 = absolute, 0 = job-relative)
+ *   bits [23:12]: Resource Index (PTE pid_offset within the PID slice)
+ *   bits [11:0] : PIDonFEP (process-endpoint address)
+ *
+ * QP state machine:
+ *   RESET  -> INIT  : validate parameters; allocate CXI resources
+ *   INIT   -> RTR   : map PTE + insert catch-all receive LE
+ *   RTR    -> RTS   : arm transmit queue
+ *   RTS/RTR -> ERROR: disable PTE and drain TXQ
+ *   any    -> RESET : free all sub-resources
+ */
+
+#include <linux/slab.h>
+#include <linux/gfp.h>
+#include <linux/mm.h>
+#include <linux/random.h>
+#include <linux/io.h>
+#include <linux/spinlock.h>
+
+#include <rdma/ib_verbs.h>
+#include <rdma/uverbs_ioctl.h>
+
+#include "cxiverbs.h"
+
+/* Work request queue depth limits */
+#define CXI_MAX_SEND_WR    1024
+#define CXI_MAX_RECV_WR    1024
+#define CXI_MAX_SEND_SGE   1
+#define CXI_MAX_RECV_SGE   1
+#define CXI_MAX_INLINE     192   /* IDC fast-path maximum bytes */
+
+/* TXQ ring buffer entry count (each entry = C_CQ_CMD_SIZE = 64 bytes) */
+#define CXI_TXQ_DEFAULT_ENTRIES 512
+
+int cxiverbs_query_qp_semantics(struct ib_device *device,
+			       enum ib_qp_type type,
+			       struct ib_qp_semantics *sem)
+{
+	if (type != IB_QPT_RU)
+		return -EOPNOTSUPP;
+	memset(sem, 0, sizeof(*sem));
+	sem->comp_mask = IB_UVERBS_QP_SEMANTICS_MASK_MSG_ORDER |
+		IB_UVERBS_QP_SEMANTICS_MASK_RAW |
+		IB_UVERBS_QP_SEMANTICS_MASK_WAR |
+		IB_UVERBS_QP_SEMANTICS_MASK_WAW |
+		IB_UVERBS_QP_SEMANTICS_MASK_PDU |
+		IB_UVERBS_QP_SEMANTICS_MASK_IMM |
+		IB_UVERBS_QP_SEMANTICS_MASK_USAGE;
+	sem->max_pdu = ib_mtu_enum_to_int(IB_MTU_4096);
+	sem->imm_data_size = 64;
+	sem->usage_flags = IB_UVERBS_QP_USAGE_IMM_DATA_RQ;
+	return 0;
+}
+
+/* =======================================================================
+ * Resource teardown helper
+ * =======================================================================
+ */
+
+struct cxiverbs_qp_resources {
+	refcount_t refs;
+	struct cxi_adev *cadev;
+	struct ib_pd *pd;
+	struct ib_cq *send_cq;
+	struct ib_cq *recv_cq;
+	struct cxi_domain *domain;
+	struct cxi_cp *cp;
+	struct cxi_cq *txq;
+	struct cxi_cq *tgq;
+	struct cxi_pte *pte;
+	int pt_map_index;
+	u32 ri;
+};
+
+static void cxiverbs_qp_resources_put(void *owner)
+{
+	struct cxiverbs_qp_resources *resources = owner;
+	struct cxi_adev *cadev = resources->cadev;
+	struct cxiverbs_pd *pd = to_cxiverbs_pd(resources->pd);
+
+	if (!refcount_dec_and_test(&resources->refs))
+		return;
+
+	cadev->pte_unmap(resources->pte, resources->domain,
+			 resources->pt_map_index);
+	cadev->cxi_pte_free(resources->pte);
+	cadev->cq_free(resources->tgq);
+	cadev->cq_free(resources->txq);
+	cadev->cp_free(resources->cp);
+	cadev->domain_free(resources->domain);
+	xa_erase(&pd->qpns, resources->ri);
+	atomic_dec(&resources->send_cq->usecnt);
+	if (resources->recv_cq)
+		atomic_dec(&resources->recv_cq->usecnt);
+	atomic_dec(&resources->pd->usecnt);
+	kfree(resources);
+}
+
+static struct cxiverbs_mmap_entry *
+cxiverbs_qp_mmap(struct cxiverbs_qp *qp, struct ib_ucontext *ibctx,
+		 bool is_io, phys_addr_t phys, struct page *first_page,
+		 size_t size, u64 *offset)
+{
+	struct cxiverbs_mmap_entry *entry;
+
+	refcount_inc(&qp->resources->refs);
+	entry = cxiverbs_mmap_entry_alloc_owned(ibctx, is_io, phys, first_page,
+						size, offset, qp->resources,
+						cxiverbs_qp_resources_put);
+	if (IS_ERR(entry))
+		cxiverbs_qp_resources_put(qp->resources);
+	return entry;
+}
+
+static void cxiverbs_qp_free_resources(struct cxiverbs_qp *qp)
+{
+	struct cxiverbs_dev *dev = to_cxiverbs_dev(qp->ibqp.device);
+
+	/* Remove user mmap entries (they hold references) */
+	if (qp->tgq_db_mmap) {
+		rdma_user_mmap_entry_remove(&qp->tgq_db_mmap->rdma_entry);
+		qp->tgq_db_mmap = NULL;
+	}
+	if (qp->tgq_cmds_mmap) {
+		rdma_user_mmap_entry_remove(&qp->tgq_cmds_mmap->rdma_entry);
+		qp->tgq_cmds_mmap = NULL;
+	}
+	if (qp->txq_db_mmap) {
+		rdma_user_mmap_entry_remove(&qp->txq_db_mmap->rdma_entry);
+		qp->txq_db_mmap = NULL;
+	}
+	if (qp->txq_cmds_mmap) {
+		rdma_user_mmap_entry_remove(&qp->txq_cmds_mmap->rdma_entry);
+		qp->txq_cmds_mmap = NULL;
+	}
+
+	if (qp->resources) {
+		cxiverbs_qp_resources_put(qp->resources);
+		qp->resources = NULL;
+		qp->pte = NULL;
+		qp->tgq = NULL;
+		qp->txq = NULL;
+		qp->cp = NULL;
+		qp->domain = NULL;
+		qp->lni = NULL;
+		qp->identity_reserved = false;
+		return;
+	}
+
+	/* Unmap PTE from domain before freeing */
+	if (qp->pte) {
+		if (qp->pt_map_index >= 0 && qp->domain)
+			dev->cadev->pte_unmap(qp->pte, qp->domain,
+					      qp->pt_map_index);
+		dev->cadev->cxi_pte_free(qp->pte);
+		qp->pte = NULL;
+	}
+
+	if (qp->tgq) {
+		dev->cadev->cq_free(qp->tgq);
+		qp->tgq = NULL;
+	}
+	if (qp->txq) {
+		dev->cadev->cq_free(qp->txq);
+		qp->txq = NULL;
+	}
+	if (qp->cp) {
+		dev->cadev->cp_free(qp->cp);
+		qp->cp = NULL;
+	}
+	if (qp->domain) {
+		dev->cadev->domain_free(qp->domain);
+		qp->domain = NULL;
+	}
+	qp->lni = NULL;
+	if (qp->identity_reserved) {
+		struct cxiverbs_pd *pd = to_cxiverbs_pd(qp->ibqp.pd);
+
+		xa_erase(&pd->qpns, qp->ri);
+		qp->identity_reserved = false;
+	}
+}
+
+/* =======================================================================
+ * create_qp
+ * =======================================================================
+ */
+
+int cxiverbs_create_qp(struct ib_qp *ibqp, struct ib_qp_init_attr *init_attr,
+		      struct ib_udata *udata)
+{
+	struct ib_pd *ibpd             = ibqp->pd;
+	struct cxiverbs_pd *pd;
+	struct cxiverbs_dev *dev;
+	struct cxiverbs_cq *send_cq;
+	struct cxiverbs_qp *qp           = to_cxiverbs_qp(ibqp);
+	struct cxi_ibv_create_qp_cmd cmd = {};
+	struct cxi_ibv_create_qp_resp resp = {};
+	struct cxi_cq_alloc_opts txq_opts = {};
+	struct cxi_cq_alloc_opts tgq_opts = {};
+	struct cxi_pt_alloc_opts pte_opts = {};
+	size_t txq_cmds_size;
+	struct page *txq_cmds_pages = NULL;
+	phys_addr_t txq_db_phys;
+	size_t txq_db_size;
+	size_t tgq_cmds_size;
+	struct page *tgq_cmds_pages = NULL;
+	phys_addr_t tgq_db_phys;
+	size_t tgq_db_size;
+	u32 requested_pid = init_attr->source_qpn & CXI_QPN_PID_MASK;
+	u32 requested_ri = FIELD_GET(CXI_QPN_RI_MASK, init_attr->source_qpn);
+	u32 requested_addr_mode = init_attr->source_qpn & CXI_QPN_ADDR_MODE_MASK;
+	u32 svc_id;
+	u32 resource_index;
+	u32 ri;
+	u32 tc = CXI_TC_BEST_EFFORT;
+	int pid;
+	int ret;
+	unsigned int pt_index;
+
+	if (!udata || !ibqp->uobject || !ibpd || !ibpd->uobject)
+		return -EOPNOTSUPP;
+
+	pd = to_cxiverbs_pd(ibpd);
+	dev = to_cxiverbs_dev(ibpd->device);
+	send_cq = to_cxiverbs_cq(init_attr->send_cq);
+
+	if (!(dev->cdev->prop.cassini_version & CASSINI_2) ||
+	    dev->cdev->prop.vni_matching != 0)
+		return -EOPNOTSUPP;
+
+	if (!pd->lni_ref || !pd->pidonfep_valid || !init_attr->jkey ||
+	    init_attr->jkey->pd != ibpd)
+		return -EINVAL;
+	if (send_cq->lni_ref != pd->lni_ref ||
+	    (init_attr->recv_cq &&
+	     to_cxiverbs_cq(init_attr->recv_cq)->lni_ref != pd->lni_ref))
+		return -EINVAL;
+
+	if (init_attr->qp_type != IB_QPT_RU)
+		return -EOPNOTSUPP;
+	if (init_attr->qp_semantics) {
+		const struct ib_qp_semantics *sem = init_attr->qp_semantics;
+
+		if (sem->msg_order || sem->max_rdma_raw_size ||
+		    sem->max_rdma_war_size || sem->max_rdma_waw_size ||
+		    (sem->max_pdu &&
+		     sem->max_pdu != ib_mtu_enum_to_int(IB_MTU_4096)) ||
+		    sem->imm_data_size > 64 ||
+		    (sem->usage_flags & ~IB_UVERBS_QP_USAGE_IMM_DATA_RQ))
+			return -EOPNOTSUPP;
+	}
+
+	if (udata->inlen >= sizeof(cmd)) {
+		ret = ib_copy_from_udata(&cmd, udata, sizeof(cmd));
+		if (ret)
+			return ret;
+		if (cmd.comp_mask & ~(CXI_IBV_CREATE_QP_SVC_ID |
+				      CXI_IBV_CREATE_QP_TC))
+			return -EOPNOTSUPP;
+		if ((cmd.comp_mask & CXI_IBV_CREATE_QP_SVC_ID) &&
+		    (!cmd.svc_id || cmd.svc_id > CXI_IBV_SERVICE_ID_MAX))
+			return -EINVAL;
+		if (cmd.comp_mask & CXI_IBV_CREATE_QP_TC)
+			tc = cmd.tc;
+	}
+
+	if (init_attr->create_flags & ~IB_QP_CREATE_SOURCE_QPN)
+		return -EOPNOTSUPP;
+
+	if (init_attr->create_flags & IB_QP_CREATE_SOURCE_QPN) {
+		if (init_attr->source_qpn &
+		    ~(CXI_QPN_ADDR_MODE_MASK | CXI_QPN_RI_MASK | CXI_QPN_PID_MASK))
+			return -EINVAL;
+		if (requested_addr_mode != pd->qpn_addr_mode)
+			return -EINVAL;
+		if (requested_pid != pd->pidonfep)
+			return -EINVAL;
+	}
+
+	/*
+	 * The Resource Index selects which receive bucket of this PD's PID
+	 * slice remote peers address (the PTE pid_offset). It is chosen by the
+	 * application and is independent of the service ID, which only names
+	 * the resource-limit pool the LNI draws from.
+	 */
+	ri = (init_attr->create_flags & IB_QP_CREATE_SOURCE_QPN) ?
+		requested_ri : 0;
+	if (dev->pid_granule && ri >= dev->pid_granule)
+		return -EINVAL;
+
+	svc_id = (cmd.comp_mask & CXI_IBV_CREATE_QP_SVC_ID) ? cmd.svc_id :
+		 pd->preferred_svc_id;
+	if (!svc_id)
+		svc_id = pd->lni_ref->svc_id;
+	if (svc_id != pd->lni_ref->svc_id)
+		return -EINVAL;
+	qp->lni = pd->lni_ref->lni;
+
+	spin_lock_init(&qp->lock);
+	mutex_init(&qp->modify_lock);
+	qp->state           = IB_QPS_RESET;
+	qp->pt_map_index    = -1;
+	qp->max_send_wr     = min_t(u32, init_attr->cap.max_send_wr, CXI_MAX_SEND_WR);
+	qp->max_recv_wr     = min_t(u32, init_attr->cap.max_recv_wr, CXI_MAX_RECV_WR);
+	qp->max_send_sge    = min_t(u32, init_attr->cap.max_send_sge, CXI_MAX_SEND_SGE);
+	qp->max_recv_sge    = min_t(u32, init_attr->cap.max_recv_sge, CXI_MAX_RECV_SGE);
+	qp->max_inline_data = min_t(u32, init_attr->cap.max_inline_data, CXI_MAX_INLINE);
+	qp->vni             = init_attr->jkey->job->job_id;
+	qp->svc_id          = svc_id;
+	qp->ri              = ri;
+	qp->tc              = tc;
+
+	mutex_lock(&pd->resource_lock);
+	if (pd->job_id && pd->job_id != qp->vni) {
+		ret = -EINVAL;
+		goto err_unlock_pd;
+	}
+	if (init_attr->create_flags & IB_QP_CREATE_SOURCE_QPN)
+		ret = xa_insert(&pd->qpns, ri, qp, GFP_KERNEL);
+	else
+		ret = xa_alloc(&pd->qpns, &ri, qp,
+			       XA_LIMIT(0, dev->pid_granule - 1), GFP_KERNEL);
+	if (ret)
+		goto err_unlock_pd;
+	pd->job_id = qp->vni;
+	qp->ri = ri;
+	qp->identity_reserved = true;
+	pid = pd->pidonfep;
+	resource_index = FIELD_PREP(CXI_QPN_RI_MASK, ri);
+	mutex_unlock(&pd->resource_lock);
+
+	qp->domain = dev->cadev->domain_alloc(qp->lni, qp->vni, (unsigned int)pid);
+	if (IS_ERR(qp->domain)) {
+		ret = PTR_ERR(qp->domain);
+		qp->domain = NULL;
+		goto err_identity;
+	}
+
+	/* Step 2: Allocate a Communication Profile (LCID + traffic class) */
+	qp->cp = dev->cadev->cp_alloc(qp->lni, qp->vni, qp->tc, CXI_TC_TYPE_DEFAULT);
+	if (IS_ERR(qp->cp)) {
+		ret = PTR_ERR(qp->cp);
+		qp->cp = NULL;
+		goto err_free_domain;
+	}
+	qp->lcid = qp->cp->lcid;
+
+	/* Step 3: Allocate transmit (initiator) CQ.
+	 * cxi-driver internally allocates the ring buffer for user CQs.
+	 */
+	txq_opts.count = CXI_TXQ_DEFAULT_ENTRIES;
+	txq_opts.flags = CXI_CQ_IS_TX | CXI_CQ_USER;
+	txq_opts.lcid  = qp->lcid;
+
+	qp->txq = dev->cadev->cq_alloc(qp->lni, send_cq->eq, &txq_opts, NUMA_NO_NODE);
+	if (IS_ERR(qp->txq)) {
+		ret = PTR_ERR(qp->txq);
+		qp->txq = NULL;
+		goto err_free_cp;
+	}
+
+	/* Step 4: Allocate target (receive) CQ */
+	tgq_opts.count = CXI_TXQ_DEFAULT_ENTRIES;
+	tgq_opts.flags = CXI_CQ_USER;
+
+	if (init_attr->recv_cq) {
+		struct cxiverbs_cq *recv_cq = to_cxiverbs_cq(init_attr->recv_cq);
+
+		qp->tgq = dev->cadev->cq_alloc(qp->lni, recv_cq->eq, &tgq_opts,
+					NUMA_NO_NODE);
+	} else {
+		qp->tgq = dev->cadev->cq_alloc(qp->lni, send_cq->eq, &tgq_opts,
+					NUMA_NO_NODE);
+	}
+	if (IS_ERR(qp->tgq)) {
+		ret = PTR_ERR(qp->tgq);
+		qp->tgq = NULL;
+		goto err_free_txq;
+	}
+
+	/* Step 5: Allocate the Portal Table Entry */
+	/*
+	 * Matching, so an RMA operation can select a region by its rkey. SEND
+	 * uses match bits of zero, which no rkey takes.
+	 */
+	pte_opts.is_matching    = 1;
+	pte_opts.en_match_on_vni = 1;
+	pte_opts.use_long_event = 0;
+
+	qp->pte = dev->cadev->cxi_pte_alloc(qp->lni, send_cq->eq, &pte_opts);
+	if (IS_ERR(qp->pte)) {
+		ret = PTR_ERR(qp->pte);
+		qp->pte = NULL;
+		goto err_free_tgq;
+	}
+	ret = dev->cadev->pte_map(qp->pte, qp->domain, qp->ri,
+				false, &pt_index);
+	if (ret)
+		goto err_free_pte;
+	qp->pt_map_index = pt_index;
+
+	/* Step 6: Get TXQ mmap info from cxi-driver */
+	ret = dev->cadev->cq_user_info(qp->txq, &txq_cmds_size, &txq_cmds_pages,
+			       &txq_db_phys, &txq_db_size);
+	if (ret)
+		goto err_free_pte;
+
+	/* Step 7: Get TGQ mmap info from cxi-driver */
+	ret = dev->cadev->cq_user_info(qp->tgq, &tgq_cmds_size, &tgq_cmds_pages,
+			       &tgq_db_phys, &tgq_db_size);
+	if (ret)
+		goto err_free_pte;
+
+	/* Register mmap entries for user-space direct submission */
+	qp->resources = kzalloc_obj(*qp->resources);
+	if (!qp->resources) {
+		ret = -ENOMEM;
+		goto err_free_pte;
+	}
+	refcount_set(&qp->resources->refs, 1);
+	qp->resources->cadev = dev->cadev;
+	qp->resources->pd = ibpd;
+	qp->resources->send_cq = init_attr->send_cq;
+	qp->resources->recv_cq = init_attr->recv_cq;
+	qp->resources->domain = qp->domain;
+	qp->resources->cp = qp->cp;
+	qp->resources->txq = qp->txq;
+	qp->resources->tgq = qp->tgq;
+	qp->resources->pte = qp->pte;
+	qp->resources->pt_map_index = qp->pt_map_index;
+	qp->resources->ri = qp->ri;
+	atomic_inc(&ibpd->usecnt);
+	atomic_inc(&init_attr->send_cq->usecnt);
+	if (init_attr->recv_cq)
+		atomic_inc(&init_attr->recv_cq->usecnt);
+
+	{
+		struct ib_ucontext *ibctx = ibqp->pd->uobject->context;
+
+		qp->txq_cmds_mmap = cxiverbs_qp_mmap(qp, ibctx, false, 0,
+						     txq_cmds_pages, txq_cmds_size,
+						     &resp.txq_cmds_offset);
+		if (IS_ERR(qp->txq_cmds_mmap)) {
+			ret = PTR_ERR(qp->txq_cmds_mmap);
+			qp->txq_cmds_mmap = NULL;
+			goto err_free_resources;
+		}
+
+		qp->txq_db_mmap = cxiverbs_qp_mmap(qp, ibctx, true, txq_db_phys,
+						   NULL, txq_db_size, &resp.txq_db_offset);
+		if (IS_ERR(qp->txq_db_mmap)) {
+			ret = PTR_ERR(qp->txq_db_mmap);
+			qp->txq_db_mmap = NULL;
+			goto err_free_resources;
+		}
+
+		qp->tgq_cmds_mmap = cxiverbs_qp_mmap(qp, ibctx, false, 0,
+						     tgq_cmds_pages, tgq_cmds_size,
+						     &resp.tgq_cmds_offset);
+		if (IS_ERR(qp->tgq_cmds_mmap)) {
+			ret = PTR_ERR(qp->tgq_cmds_mmap);
+			qp->tgq_cmds_mmap = NULL;
+			goto err_free_resources;
+		}
+
+		qp->tgq_db_mmap = cxiverbs_qp_mmap(qp, ibctx, true, tgq_db_phys,
+						   NULL, tgq_db_size, &resp.tgq_db_offset);
+		if (IS_ERR(qp->tgq_db_mmap)) {
+			ret = PTR_ERR(qp->tgq_db_mmap);
+			qp->tgq_db_mmap = NULL;
+			goto err_free_resources;
+		}
+	}
+
+	/*
+	 * QPN encoding:
+	 *   bits [11:0]  = PIDonFEP   (filled now)
+	 *   bits [23:12] = Resource Index (PTE pid_offset)
+	 *   bit  31      = addr_mode  (0 = job-relative by default)
+	 */
+	qp->qpn_base = (pd->pidonfep & CXI_QPN_PID_MASK) |
+			resource_index |
+			pd->qpn_addr_mode;
+	ibqp->qp_num = qp->qpn_base;
+
+	resp.txq_cmds_size = txq_cmds_size;
+	resp.txq_db_size   = txq_db_size;
+	resp.txq_idx       = cxi_cq_get_cqn(qp->txq);
+	resp.lcid          = qp->lcid;
+	resp.tgq_cmds_size = tgq_cmds_size;
+	resp.tgq_db_size   = tgq_db_size;
+	resp.tgq_idx       = cxi_cq_get_cqn(qp->tgq);
+	resp.job_id        = qp->vni;
+	resp.pte_index     = qp->pte->id;
+
+	ret = ib_respond_udata(udata, resp);
+	if (ret)
+		goto err_free_resources;
+
+	return 0;
+
+err_free_resources:
+	cxiverbs_qp_free_resources(qp);
+	return ret;
+err_free_pte:
+	if (qp->pt_map_index >= 0) {
+		dev->cadev->pte_unmap(qp->pte, qp->domain, qp->pt_map_index);
+		qp->pt_map_index = -1;
+	}
+	dev->cadev->cxi_pte_free(qp->pte);
+	qp->pte = NULL;
+err_free_tgq:
+	dev->cadev->cq_free(qp->tgq);
+	qp->tgq = NULL;
+err_free_txq:
+	dev->cadev->cq_free(qp->txq);
+	qp->txq = NULL;
+err_free_cp:
+	dev->cadev->cp_free(qp->cp);
+	qp->cp = NULL;
+err_free_domain:
+	dev->cadev->domain_free(qp->domain);
+	qp->domain = NULL;
+err_identity:
+	xa_erase(&pd->qpns, qp->ri);
+	qp->identity_reserved = false;
+	return ret;
+err_unlock_pd:
+	mutex_unlock(&pd->resource_lock);
+	return ret;
+}
+
+/* =======================================================================
+ * destroy_qp
+ * =======================================================================
+ */
+
+int cxiverbs_destroy_qp(struct ib_qp *ibqp, struct ib_udata *udata)
+{
+	struct cxiverbs_qp *qp = to_cxiverbs_qp(ibqp);
+
+	cxiverbs_qp_free_resources(qp);
+	mutex_destroy(&qp->modify_lock);
+	return 0;
+}
+
+/* =======================================================================
+ * modify_qp  —  UEC-aligned state machine
+ * =======================================================================
+ */
+
+int cxiverbs_modify_qp(struct ib_qp *ibqp, struct ib_qp_attr *attr,
+		      int attr_mask, struct ib_udata *udata)
+{
+	struct cxiverbs_qp *qp = to_cxiverbs_qp(ibqp);
+	struct cxiverbs_dev *dev = to_cxiverbs_dev(ibqp->device);
+	enum ib_qp_state cur_state, new_state;
+	int ret = -EINVAL;
+
+	if (attr_mask & ~(IB_QP_STATE | IB_QP_CUR_STATE | IB_QP_PORT |
+			  IB_QP_PKEY_INDEX | IB_QP_ACCESS_FLAGS | IB_QP_PATH_MTU))
+		return -EOPNOTSUPP;
+	if (((attr_mask & IB_QP_PORT) && attr->port_num != 1) ||
+	    ((attr_mask & IB_QP_PKEY_INDEX) && attr->pkey_index))
+		return -EINVAL;
+	if (((attr_mask & IB_QP_PATH_MTU) && attr->path_mtu != IB_MTU_4096) ||
+	    ((attr_mask & IB_QP_ACCESS_FLAGS) && attr->qp_access_flags))
+		return -EOPNOTSUPP;
+
+	/*
+	 * uverbs only takes a read reference on the QP for modify, so several
+	 * modifies can run at once. Hold modify_lock for the whole transition:
+	 * the state read below then stays valid until it is published, and the
+	 * portal table map step, which sleeps and so cannot run under the
+	 * spinlock, is not repeated or raced. qp->lock only publishes the state
+	 * to post_send(), post_recv() and query_qp().
+	 */
+	mutex_lock(&qp->modify_lock);
+
+	spin_lock_irq(&qp->lock);
+	cur_state = qp->state;
+	spin_unlock_irq(&qp->lock);
+	new_state = (attr_mask & IB_QP_STATE) ? attr->qp_state : cur_state;
+	if ((attr_mask & IB_QP_CUR_STATE) && attr->cur_qp_state != cur_state) {
+		ret = -EINVAL;
+		goto out;
+	}
+
+	switch (cur_state) {
+	case IB_QPS_RESET:
+		if (new_state == IB_QPS_RESET || new_state == IB_QPS_INIT ||
+		    new_state == IB_QPS_RTS) {
+			spin_lock_irq(&qp->lock);
+			qp->state = new_state;
+			spin_unlock_irq(&qp->lock);
+			ret = 0;
+		}
+		break;
+
+	case IB_QPS_INIT:
+		if (new_state == IB_QPS_RTR) {
+			struct cxi_ibv_modify_qp_resp resp = {};
+			unsigned int pt_index;
+			bool mapped = false;
+
+			if (qp->pt_map_index < 0) {
+				ret = dev->cadev->pte_map(qp->pte, qp->domain,
+							qp->ri, false, &pt_index);
+				if (ret)
+					goto out;
+				qp->pt_map_index = (int)pt_index;
+				mapped = true;
+			}
+			pt_index = qp->pt_map_index;
+
+			ibqp->qp_num = qp->qpn_base;
+
+			ibdev_info(&dev->ib_dev,
+				   "QP RTR: qpn=0x%06x ri=%u pid=%u vni=%u pte=%u map_slot=%u\n",
+				   qp->qpn_base, qp->ri,
+				   qp->domain ? qp->domain->pid : 0, qp->vni,
+				   qp->pte->id, pt_index);
+
+			/*
+			 * No list entry is appended here: an LE must be bound to
+			 * a registered MR, which only post_recv has. Until one is
+			 * posted the portal matches nothing and inbound traffic is
+			 * dropped, which is what an empty receive queue means.
+			 */
+
+			/*
+			 * pte_map() hands back an RMU portal-list slot, not the
+			 * PtlTE number that target commands address.
+			 */
+			resp.pte_index = qp->pte->id;
+			if (udata && udata->outlen >= sizeof(resp)) {
+				ret = ib_respond_udata(udata, resp);
+				if (ret) {
+					if (mapped) {
+						dev->cadev->pte_unmap(qp->pte,
+							qp->domain, qp->pt_map_index);
+						qp->pt_map_index = -1;
+					}
+					goto out;
+				}
+			}
+
+			spin_lock_irq(&qp->lock);
+			qp->state = IB_QPS_RTR;
+			spin_unlock_irq(&qp->lock);
+			ret = 0;
+		}
+		break;
+
+	case IB_QPS_RTR:
+		if (new_state == IB_QPS_RTS) {
+			spin_lock_irq(&qp->lock);
+			qp->state = IB_QPS_RTS;
+			spin_unlock_irq(&qp->lock);
+			ret = 0;
+		}
+		break;
+
+	case IB_QPS_RTS:
+	case IB_QPS_ERR:
+		if (new_state == cur_state || new_state == IB_QPS_ERR ||
+		    new_state == IB_QPS_RESET) {
+			spin_lock_irq(&qp->lock);
+			qp->state = new_state;
+			spin_unlock_irq(&qp->lock);
+			ret = 0;
+		}
+		break;
+
+	default:
+		break;
+	}
+
+out:
+	mutex_unlock(&qp->modify_lock);
+	return ret;
+}
+
+/* =======================================================================
+ * query_qp
+ * =======================================================================
+ */
+
+int cxiverbs_query_qp(struct ib_qp *ibqp, struct ib_qp_attr *attr,
+		     int attr_mask, struct ib_qp_init_attr *init_attr)
+{
+	struct cxiverbs_qp *qp = to_cxiverbs_qp(ibqp);
+	unsigned long flags;
+
+	memset(attr, 0, sizeof(*attr));
+	memset(init_attr, 0, sizeof(*init_attr));
+
+	spin_lock_irqsave(&qp->lock, flags);
+	attr->qp_state     = qp->state;
+	attr->cur_qp_state = qp->state;
+	spin_unlock_irqrestore(&qp->lock, flags);
+	attr->pkey_index   = 0;
+	attr->port_num     = 1;
+	attr->path_mtu     = IB_MTU_4096;
+
+	attr->cap.max_send_wr     = qp->max_send_wr;
+	attr->cap.max_recv_wr     = qp->max_recv_wr;
+	attr->cap.max_send_sge    = qp->max_send_sge;
+	attr->cap.max_recv_sge    = qp->max_recv_sge;
+	attr->cap.max_inline_data = qp->max_inline_data;
+
+	init_attr->cap = attr->cap;
+
+	return 0;
+}
+
+/* =======================================================================
+ * post_send  —  translate IB work requests to CXI DMA commands
+ * =======================================================================
+ */
+
+int cxiverbs_post_send(struct ib_qp *ibqp, const struct ib_send_wr *wr,
+		      const struct ib_send_wr **bad_wr)
+{
+	struct cxiverbs_qp  *qp  = to_cxiverbs_qp(ibqp);
+	struct cxiverbs_dev *dev = to_cxiverbs_dev(ibqp->device);
+	struct cxiverbs_cq  *scq = to_cxiverbs_cq(ibqp->send_cq);
+	int ret = 0;
+
+	spin_lock(&qp->lock);
+	if (qp->state != IB_QPS_RTS && qp->state != IB_QPS_RTR) {
+		spin_unlock(&qp->lock);
+		*bad_wr = wr;
+		return -EINVAL;
+	}
+
+	for (; wr; wr = wr->next) {
+		union c_fab_addr dfa;
+		uint8_t dfa_ext = 0;
+
+		switch (wr->opcode) {
+		case IB_WR_SEND: {
+			const struct ib_ud_wr *ud_wr =
+				container_of(wr, const struct ib_ud_wr, wr);
+			struct cxiverbs_ah *ah = to_cxiverbs_ah(ud_wr->ah);
+			struct c_full_dma_cmd cmd = {};
+			u32 remote_pid;
+			u32 remote_ri;
+
+			if (!wr->num_sge || !wr->sg_list)
+				goto bad;
+
+			/* No CXI command carries a gather list. */
+			if (wr->num_sge > 1) {
+				ret = -EINVAL;
+				goto bad;
+			}
+
+			remote_pid = ud_wr->remote_qpn & CXI_QPN_PID_MASK;
+			remote_ri = FIELD_GET(CXI_QPN_RI_MASK,
+					      ud_wr->remote_qpn);
+			cxi_build_dfa(ah->nid, remote_pid, dev->pid_bits,
+				      remote_ri, &dfa, &dfa_ext);
+
+			/*
+			 * cxi_cq_emit_nomatch_dma() forces restricted=1, which
+			 * suppresses the target PUT event the receiver needs to
+			 * complete its recv WR, and c_nomatch_dma_cmd has no
+			 * user_ptr to carry wr_id. Use an unrestricted full DMA
+			 * put instead; the destination PtlTE is non-matching, so
+			 * match_bits are ignored.
+			 */
+			cmd.command.opcode = C_CMD_PUT;
+			cmd.lac            = wr->sg_list[0].lkey & CXI_LKEY64_LAC_MASK;
+			cmd.dfa            = dfa;
+			cmd.index_ext      = dfa_ext;
+			cmd.remote_offset  = 0;
+			cmd.local_addr     = wr->sg_list[0].addr;
+			cmd.request_len    = wr->sg_list[0].length;
+			cmd.user_ptr       = wr->wr_id;
+			cmd.eq             = scq->eq->eqn;
+			/* One completion per WR: ACK only, not SEND. */
+			cmd.event_send_disable = 1;
+
+			ret = cxi_cq_emit_dma(qp->txq, &cmd);
+			if (ret)
+				goto bad;
+			break;
+		}
+
+		case IB_WR_RDMA_WRITE:
+		case IB_WR_RDMA_READ: {
+			const struct ib_rdma_wr *rdma_wr =
+				container_of(wr, const struct ib_rdma_wr, wr);
+			struct c_full_dma_cmd cmd = {};
+
+			if (!wr->num_sge || !wr->sg_list)
+				goto bad;
+
+			if (wr->num_sge > 1) {
+				ret = -EINVAL;
+				goto bad;
+			}
+
+			/* For RC, remote NID is encoded in the QP's domain/AH.
+			 * Here we use a simplified direct NID from the rkey.
+			 */
+			cxi_build_dfa(0, 0, dev->pid_bits, 0, &dfa, &dfa_ext);
+
+			cmd.command.opcode = (wr->opcode == IB_WR_RDMA_READ) ?
+					     C_CMD_GET : C_CMD_PUT;
+			cmd.lac            = wr->sg_list[0].lkey & CXI_LKEY64_LAC_MASK;
+			cmd.dfa            = dfa;
+			cmd.index_ext      = dfa_ext;
+			cmd.remote_offset  = rdma_wr->remote_addr;
+			cmd.local_addr     = wr->sg_list[0].addr;
+			cmd.request_len    = wr->sg_list[0].length;
+			cmd.match_bits     = (u64)rdma_wr->rkey;
+			cmd.user_ptr       = wr->wr_id;
+			cmd.eq             = scq->eq->eqn;
+
+			ret = cxi_cq_emit_dma(qp->txq, &cmd);
+			if (ret)
+				goto bad;
+			break;
+		}
+
+		case IB_WR_ATOMIC_FETCH_AND_ADD:
+		case IB_WR_ATOMIC_CMP_AND_SWP: {
+			const struct ib_atomic_wr *atomic_wr =
+				container_of(wr, const struct ib_atomic_wr, wr);
+			const struct ib_rdma_wr *rdma_wr =
+				container_of(wr, const struct ib_rdma_wr, wr);
+			struct c_dma_amo_cmd cmd = {};
+			bool fetching = (wr->opcode == IB_WR_ATOMIC_FETCH_AND_ADD);
+
+			cxi_build_dfa(0, 0, dev->pid_bits, 0, &dfa, &dfa_ext);
+
+			cmd.command.opcode  = C_CMD_ATOMIC;
+			cmd.lac             = 0;
+			cmd.dfa             = dfa;
+			cmd.index_ext       = dfa_ext;
+			cmd.remote_offset   = rdma_wr->remote_addr;
+			cmd.request_len     = 8;
+			cmd.match_bits      = (u64)rdma_wr->rkey;
+			cmd.user_ptr        = wr->wr_id;
+			cmd.eq              = scq->eq->eqn;
+			cmd.atomic_op       = fetching ? C_AMO_OP_SUM : C_AMO_OP_CSWAP;
+			cmd.atomic_type     = C_AMO_TYPE_UINT64_T;
+			cmd.op2_word1       = atomic_wr->compare_add;
+			if (!fetching)
+				cmd.op2_word2 = atomic_wr->swap;
+
+			ret = cxi_cq_emit_dma_amo(qp->txq, &cmd, fetching);
+			if (ret)
+				goto bad;
+			break;
+		}
+
+		default:
+			ret = -EOPNOTSUPP;
+			goto bad;
+		}
+
+		cxi_cq_ring(qp->txq);
+		continue;
+
+bad:
+		*bad_wr = wr;
+		spin_unlock(&qp->lock);
+		return ret ? ret : -EIO;
+	}
+
+	spin_unlock(&qp->lock);
+	return 0;
+}
+
+/* =======================================================================
+ * post_recv  —  append a receive LE to the portal table
+ * =======================================================================
+ */
+
+int cxiverbs_post_recv(struct ib_qp *ibqp, const struct ib_recv_wr *wr,
+		      const struct ib_recv_wr **bad_wr)
+{
+	struct cxiverbs_qp *qp = to_cxiverbs_qp(ibqp);
+	int ret = 0;
+
+	spin_lock(&qp->lock);
+	if (qp->state < IB_QPS_RTR) {
+		spin_unlock(&qp->lock);
+		*bad_wr = wr;
+		return -EINVAL;
+	}
+
+	for (; wr; wr = wr->next) {
+		union c_cmdu cmd = {};
+
+		if (!wr->num_sge || !wr->sg_list) {
+			ret = -EINVAL;
+			*bad_wr = wr;
+			break;
+		}
+
+		/* A list entry describes one buffer. */
+		if (wr->num_sge > 1) {
+			ret = -EINVAL;
+			*bad_wr = wr;
+			break;
+		}
+
+		cmd.command.opcode     = C_CMD_TGT_APPEND;
+		cmd.target.ptlte_index = (u16)qp->pte->id;
+		cmd.target.ptl_list    = C_PTL_LIST_PRIORITY;
+		cmd.target.op_put      = 1;
+		cmd.target.use_once    = 1;
+		cmd.target.lac         = wr->sg_list[0].lkey & CXI_LKEY64_LAC_MASK;
+		cmd.target.start       = wr->sg_list[0].addr;
+		cmd.target.length      = wr->sg_list[0].length;
+		/* SEND carries no key; match exactly the zero an rkey never is. */
+		cmd.target.match_bits  = 0;
+		cmd.target.ignore_bits = 0;
+
+		ret = cxi_cq_emit_target(qp->tgq, &cmd);
+		if (ret) {
+			*bad_wr = wr;
+			break;
+		}
+		cxi_cq_ring(qp->tgq);
+	}
+
+	spin_unlock(&qp->lock);
+	return ret;
+}

-- 
2.56.0

[PATCH 08/11] RDMA/cxi: add native memory registration

[Doug Ledford]

Add memory region registration on top of the core driver's pinned
mapping and its memory descriptors.  A registration returns, with the
usual lkey/rkey, the native mapping description (VA, IOVA, length,
page and huge page shift, descriptor ID and LAC) so the user space
provider can address the region without a second query.

Remote keys have 48 significant bits: a requested key wider than that
is rejected with EOPNOTSUPP rather than truncated.

Responses use ib_respond_udata().

checkpatch --strict (100 columns) still reports the following in this
patch. They need coordination with the internal repositories this code
is maintained in and will be fixed in V2:
- CHECK: PARENTHESIS_ALIGNMENT (1)

Co-developed-by: Iraimani P <iraimani.pavadai@hpe.com>
Signed-off-by: Iraimani P <iraimani.pavadai@hpe.com>
Signed-off-by: Doug Ledford <doug.ledford@hpe.com>
---
 drivers/infiniband/hw/cxi/cxiverbs_mr.c | 192 ++++++++++++++++++++++++++++++++
 1 file changed, 192 insertions(+)

diff --git a/drivers/infiniband/hw/cxi/cxiverbs_mr.c b/drivers/infiniband/hw/cxi/cxiverbs_mr.c
new file mode 100644
index 000000000000..97ca5f9a7bb1
--- /dev/null
+++ b/drivers/infiniband/hw/cxi/cxiverbs_mr.c
@@ -0,0 +1,192 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * CXI InfiniBand RDMA driver — Memory Registration (Phase 2).
+ * Copyright 2025 Hewlett Packard Enterprise Development LP
+ *
+ * Maps user memory into the CXI ATU (Address Translation Unit).
+ *
+ * Key design points
+ * -----------------
+ * lkey = low 32 bits of (md->id << 3) | lac
+ *   The LAC (Local Address Context) is the hardware protection domain
+ *   for local DMA.  md->id is the ATU MD index.
+ *
+ * rkey = random 32-bit token
+ *   Used as match_bits in the portal target LE for RDMA WRITE/READ.
+ *   The remote initiator must present this token for protected access.
+ *
+ * ODP (On-Demand Paging):
+ *   IB_ACCESS_ON_DEMAND selects native ATS/PRS mapping instead of pinning.
+ *   The native ATS/PASID/HMM paths own invalidation and notifier lifetimes.
+ *   The verbs provider must not install a second notifier or clear native
+ *   mappings from an MMU callback.
+ */
+
+#include <linux/slab.h>
+#include <linux/random.h>
+#include <linux/mm.h>
+
+#include <rdma/ib_verbs.h>
+#include <rdma/ib_umem.h>
+
+#include "cxiverbs.h"
+
+/* =======================================================================
+ * reg_user_mr
+ * =======================================================================
+ */
+
+struct ib_mr *cxiverbs_reg_user_mr(struct ib_pd *ibpd, u64 start, u64 length,
+				  u64 iova, int access,
+				  struct ib_dmah *dmah,
+				  struct ib_udata *udata)
+{
+	struct cxiverbs_ucontext *uctxt;
+	struct cxiverbs_pd *pd = to_cxiverbs_pd(ibpd);
+	struct cxi_ibv_reg_mr_resp resp = {};
+	struct cxi_ibv_reg_mr cmd = {};
+	struct cxiverbs_mr *mr;
+	bool is_odp   = !!(access & IB_ACCESS_ON_DEMAND);
+	u32 cxi_flags = CXI_MAP_USER_ADDR;
+	int ret;
+
+	if (!ibpd->uobject)
+		return ERR_PTR(-EOPNOTSUPP);
+	uctxt = to_cxiverbs_ucontext(ibpd->uobject->context);
+
+	if (!pd->lni_ref)
+		return ERR_PTR(-EINVAL);
+	if (udata && udata->inlen >= sizeof(cmd)) {
+		ret = ib_copy_from_udata(&cmd, udata, sizeof(cmd));
+		if (ret)
+			return ERR_PTR(ret);
+		if (cmd.flags & ~(CXI_REG_MR_USER_RKEY | CXI_REG_MR_NATIVE_MAP))
+			return ERR_PTR(-EOPNOTSUPP);
+	}
+
+	mr = kzalloc_obj(*mr);
+	if (!mr)
+		return ERR_PTR(-ENOMEM);
+	mr->cadev = uctxt->cadev;
+
+	if (is_odp) {
+		/* ATS mode: PCIe Address Translation Services / Page Request */
+		cxi_flags |= CXI_MAP_ATS;
+	} else {
+		/* Pin-based: pages locked in RAM for the MR lifetime */
+		cxi_flags |= CXI_MAP_PIN;
+	}
+
+	if (access & (IB_ACCESS_LOCAL_WRITE | IB_ACCESS_REMOTE_WRITE))
+		cxi_flags |= CXI_MAP_WRITE;
+	/* Local read is always needed for SGLs used in post_send */
+	cxi_flags |= CXI_MAP_READ;
+	if (cmd.flags & CXI_REG_MR_NATIVE_MAP) {
+		if (is_odp || !(cmd.native_flags & CXI_MAP_PIN) ||
+		    !(cmd.native_flags & (CXI_MAP_READ | CXI_MAP_WRITE)) ||
+		    (cmd.native_flags & ~(CXI_MAP_PIN | CXI_MAP_READ |
+				  CXI_MAP_WRITE | CXI_MAP_ALLOC_MD))) {
+			ret = -EOPNOTSUPP;
+			goto err_free_mr;
+		}
+		cxi_flags = CXI_MAP_USER_ADDR | cmd.native_flags;
+	}
+
+	/* Map into the ATU under the context's JobID-anchored LNI. */
+	mr->md = uctxt->cadev->cxi_map(pd->lni_ref->lni,
+				     (uintptr_t)start, length,
+					 cxi_flags, NULL);
+	if (IS_ERR(mr->md)) {
+		ret = PTR_ERR(mr->md);
+		mr->md = NULL;
+		goto err_free_mr;
+	}
+
+	mr->lac = mr->md->lac;
+
+	/*
+	 * The narrow keys alias the low word of these, so a caller limited to
+	 * 32 bits gets a usable truncation as long as the descriptor index
+	 * stays under 2^29.
+	 */
+	mr->ibmr.lkey64 = ((u64)mr->lac & CXI_LKEY64_LAC_MASK) |
+			  (((u64)mr->md->id & CXI_LKEY64_MD_MASK) <<
+			   CXI_LKEY64_MD_SHIFT);
+	/*
+	 * Bit 31 keeps the *narrow* key non-zero, so an rkey stays distinct
+	 * from the zero match value SEND uses whether a peer presents the full
+	 * key or only its low word.
+	 */
+	mr->ibmr.rkey64 = (get_random_u64() & CXI_RKEY_MAX) | BIT_ULL(31);
+
+	if (udata && udata->inlen >= sizeof(cmd)) {
+		ret = ib_copy_from_udata(&cmd, udata, sizeof(cmd));
+		if (ret)
+			goto err_unmap;
+
+		/* Taken as given: a narrow value makes a narrow region. */
+		if (cmd.flags & CXI_REG_MR_USER_RKEY) {
+			if (cmd.rkey64 > CXI_RKEY_MAX) {
+				ret = -EOPNOTSUPP;
+				goto err_unmap;
+			}
+			if (!cmd.rkey64) {
+				ret = -EINVAL;
+				goto err_unmap;
+			}
+			mr->ibmr.rkey64 = cmd.rkey64;
+		}
+	}
+	mr->ibmr.lkey = (u32)mr->ibmr.lkey64;
+	mr->ibmr.rkey = (u32)mr->ibmr.rkey64;
+	mr->ibmr.length = length;
+	mr->ibmr.iova   = iova ? iova : start;
+
+	resp.lkey64 = mr->ibmr.lkey64;
+	resp.rkey64 = mr->ibmr.rkey64;
+	resp.iova = CXI_VA_TO_IOVA(mr->md, start);
+	resp.info.va = mr->md->va;
+	resp.info.iova = mr->md->iova;
+	resp.info.length = mr->md->len;
+	resp.info.page_shift = mr->md->page_shift;
+	resp.info.huge_shift = mr->md->huge_shift;
+	resp.info.id = mr->md->id;
+	resp.info.lac = mr->md->lac;
+
+	if (udata) {
+		ret = ib_respond_udata(udata, resp);
+		if (ret)
+			goto err_unmap;
+	}
+
+	return &mr->ibmr;
+
+err_unmap:
+	uctxt->cadev->cxi_unmap(mr->md);
+err_free_mr:
+	kfree(mr);
+	return ERR_PTR(ret);
+}
+
+/* =======================================================================
+ * dereg_mr
+ * =======================================================================
+ */
+
+int cxiverbs_dereg_mr(struct ib_mr *ibmr, struct ib_udata *udata)
+{
+	struct cxiverbs_mr *mr = to_cxiverbs_mr(ibmr);
+	int ret;
+
+	if (atomic_read(&mr->native_users))
+		return -EBUSY;
+	if (mr->md) {
+		ret = mr->cadev->cxi_unmap(mr->md);
+		if (ret)
+			return ret;
+		mr->md = NULL;
+	}
+
+	kfree(mr);
+	return 0;
+}

-- 
2.56.0

[PATCH 09/11] RDMA/cxi: add services jobs profiles queues and counting triggers

[Doug Ledford]

Add the private uverbs object methods through which a privileged
process provisions and drives native Cassini resources.  They exist
so a launcher or workload manager can set policy on the NIC and the
provider can reach native queues without the kernel interpreting every
command:

 - services: allocate, enable, query and destroy a CXI service, with
   VNI, traffic class and resource limits; retry and range VNI modes
   are selectable
 - jobs and job keys, LNI and PD PID allocation and query
 - communication profiles, event queues (with optional pinned PCT
   registration and PCT passthrough), command queues including a target
   command queue and a restricted trigger mode, and counting events
   with a user supplied write-back page
 - priority wait descriptors: a pollable file descriptor that holds the
   LNI and reports event and status wakeups, with positional reads so
   the generation can be consumed without seeking
 - a privileged mapping of the device CSR aperture and an
   administrative inbound-wait method, both requiring CAP_SYS_ADMIN
   and a C1/C2 physical function

Lifetime rules worth knowing when reviewing: a native object stays
allocated until the last VMA that maps it is released, even if the
uverbs object was destroyed first; counting events are accounted
against the LNI and freed together with it; the write-back page for a
counting event must be user memory and is pinned, never a kernel
page, because the core driver treats a kernel address passed this
way as device memory.

Known open issue, deferred to the V2 SR-IOV update (Sashiko H87,
duplicate M88): on a physical function cxi_eq_free() cannot fail. On a
virtual function it fails when the PF mailbox is unreachable; if that
happens while the file is closed or the device is removed, the RDMA core
drops the EQ object without native cleanup and the VF EQ and its
references leak. Explicit EQ destroys are retried.

Known limitation, deferred to V2: this patch carries SR-IOV
virtual-function code (VF-to-PF mailbox requests, mailbox opcodes or VF
resource handling) from an older, incomplete version of the Cassini
SR-IOV support. V2 of this series will include the updated SR-IOV
implementation, so issues in the VF paths should be ignored for now.

checkpatch --strict (100 columns) still reports the following in this
patch. They need coordination with the internal repositories this code
is maintained in and will be fixed in V2:
- CHECK: PARENTHESIS_ALIGNMENT (82), OPEN_ENDED_LINE (23),
  UNCOMMENTED_DEFINITION (1)

Co-developed-by: Mamila Santosh <santosh-kumar.mamila@hpe.com>
Signed-off-by: Mamila Santosh <santosh-kumar.mamila@hpe.com>
Signed-off-by: Doug Ledford <doug.ledford@hpe.com>
---
 drivers/infiniband/hw/cxi/cxiverbs_ct.c | 1906 +++++++++++++++++++++++++++++++
 1 file changed, 1906 insertions(+)

diff --git a/drivers/infiniband/hw/cxi/cxiverbs_ct.c b/drivers/infiniband/hw/cxi/cxiverbs_ct.c
new file mode 100644
index 000000000000..c8f8974c649b
--- /dev/null
+++ b/drivers/infiniband/hw/cxi/cxiverbs_ct.c
@@ -0,0 +1,1906 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * cxiverbs_ct.c: Counting Triggers custom uverbs for the CXI RDMA driver.
+ * Copyright 2025 Hewlett Packard Enterprise Development LP
+ *
+ * Exposes CXI Counting Triggers (CTs) to userspace via the RDMA ioctl()
+ * interface.  A CT is a hardware counter that the NIC decrements when a
+ * triggered operation completes; when the counter hits zero the deferred
+ * command is dispatched.
+ *
+ * ABI:
+ *   CXI_IB_OBJECT_CT / CXI_IB_METHOD_CT_ALLOC
+ *     - allocates a CT against the caller's ucontext LNI
+ *     - returns: ctn, writeback-page mmap params, doorbell CSR mmap params
+ *   CXI_IB_METHOD_CT_DESTROY
+ *     - frees all CT resources; uses default destroy handler
+ *
+ * UVERBS_MODULE_NAME must be defined before uverbs_named_ioctl.h.
+ */
+#define UVERBS_MODULE_NAME cxiverbs
+#include <rdma/uverbs_named_ioctl.h>
+
+#include <linux/slab.h>
+#include <linux/gfp.h>
+#include <linux/file.h>
+#include <linux/poll.h>
+#include <linux/refcount.h>
+#include <rdma/ib_verbs.h>
+#include <rdma/uverbs_ioctl.h>
+#include <uapi/rdma/cxiverbs-abi.h>
+#include "cxiverbs.h"
+
+static struct cxiverbs_ucontext *cxiverbs_get_ucontext(
+		struct uverbs_attr_bundle *attrs)
+{
+	struct ib_ucontext *context = ib_uverbs_get_ucontext(attrs);
+
+	if (IS_ERR(context))
+		return ERR_CAST(context);
+	return to_cxiverbs_ucontext(context);
+}
+
+struct cxiverbs_csr {
+	struct cxiverbs_mmap_entry *mapping;
+};
+
+static int cxiverbs_csr_cleanup(struct ib_uobject *uobj,
+			       enum rdma_remove_reason why,
+			       struct uverbs_attr_bundle *attrs)
+{
+	struct cxiverbs_csr *csr = uobj->object;
+
+	rdma_user_mmap_entry_remove(&csr->mapping->rdma_entry);
+	kfree(csr);
+	return 0;
+}
+
+static int UVERBS_HANDLER(CXI_IB_METHOD_CSR_MAP)(
+		struct uverbs_attr_bundle *attrs)
+{
+	struct ib_uobject *uobj =
+		uverbs_attr_get_uobject(attrs, CXI_IB_ATTR_CSR_HANDLE);
+	struct cxi_ibv_csr_info info = {};
+	struct cxiverbs_ucontext *ctx;
+	struct cxiverbs_csr *csr;
+	struct cxi_dev *cdev;
+	phys_addr_t physical;
+	size_t length;
+	u32 generation;
+	int ret;
+
+	if (!capable(CAP_SYS_ADMIN))
+		return -EPERM;
+	ctx = cxiverbs_get_ucontext(attrs);
+	if (IS_ERR(ctx))
+		return PTR_ERR(ctx);
+	cdev = ctx->cadev->cxi_dev;
+	generation = cdev->prop.cassini_version & 0xf0;
+	if (!cdev->is_physfn ||
+	    (generation != CASSINI_1 && generation != CASSINI_2))
+		return -EOPNOTSUPP;
+	cxi_get_csrs_range(cdev, &physical, &length);
+	if (!length || !IS_ALIGNED(physical, PAGE_SIZE) ||
+	    !IS_ALIGNED(length, PAGE_SIZE))
+		return -EINVAL;
+	csr = kzalloc_obj(*csr, GFP_KERNEL);
+	if (!csr)
+		return -ENOMEM;
+	csr->mapping = cxiverbs_mmap_entry_alloc_admin(&ctx->ibucontext,
+						     physical, length,
+						     &info.offset);
+	if (IS_ERR(csr->mapping)) {
+		ret = PTR_ERR(csr->mapping);
+		kfree(csr);
+		return ret;
+	}
+	info.length = length;
+	uobj->object = csr;
+	uverbs_finalize_uobj_create(attrs, CXI_IB_ATTR_CSR_HANDLE);
+	return uverbs_copy_to(attrs, CXI_IB_ATTR_CSR_INFO, &info, sizeof(info));
+}
+
+DECLARE_UVERBS_NAMED_METHOD(CXI_IB_METHOD_CSR_MAP,
+	UVERBS_ATTR_IDR(CXI_IB_ATTR_CSR_HANDLE, CXI_IB_OBJECT_CSR,
+		       UVERBS_ACCESS_NEW, UA_MANDATORY),
+	UVERBS_ATTR_PTR_OUT(CXI_IB_ATTR_CSR_INFO,
+			    UVERBS_ATTR_TYPE(struct cxi_ibv_csr_info),
+			    UA_MANDATORY));
+
+DECLARE_UVERBS_NAMED_METHOD_DESTROY(CXI_IB_METHOD_CSR_UNMAP,
+	UVERBS_ATTR_IDR(CXI_IB_ATTR_CSR_HANDLE, CXI_IB_OBJECT_CSR,
+		       UVERBS_ACCESS_DESTROY, UA_MANDATORY));
+
+static int
+UVERBS_HANDLER(CXI_IB_METHOD_CSR_INBOUND_WAIT)(struct uverbs_attr_bundle *attrs)
+{
+	struct cxiverbs_ucontext *ctx;
+	struct cxi_dev *cdev;
+	u32 generation;
+
+	if (!capable(CAP_SYS_ADMIN))
+		return -EPERM;
+	ctx = cxiverbs_get_ucontext(attrs);
+	if (IS_ERR(ctx))
+		return PTR_ERR(ctx);
+	cdev = ctx->cadev->cxi_dev;
+	generation = cdev->prop.cassini_version & 0xf0;
+	if (!cdev->is_physfn ||
+	    (generation != CASSINI_1 && generation != CASSINI_2))
+		return -EOPNOTSUPP;
+	return cxi_inbound_wait(cdev);
+}
+
+DECLARE_UVERBS_NAMED_METHOD(CXI_IB_METHOD_CSR_INBOUND_WAIT);
+
+DECLARE_UVERBS_NAMED_OBJECT(CXI_IB_OBJECT_CSR,
+	UVERBS_TYPE_ALLOC_IDR(cxiverbs_csr_cleanup),
+	&UVERBS_METHOD(CXI_IB_METHOD_CSR_MAP),
+	&UVERBS_METHOD(CXI_IB_METHOD_CSR_UNMAP),
+	&UVERBS_METHOD(CXI_IB_METHOD_CSR_INBOUND_WAIT));
+
+struct cxiverbs_lni *cxiverbs_lni_get(struct cxiverbs_ucontext *ctx, u32 handle)
+{
+	struct cxiverbs_lni *lni = NULL;
+	struct cxiverbs_lni *candidate;
+	unsigned long index;
+
+	mutex_lock(&ctx->resource_lock);
+	if (handle != CXI_IBV_LNI_ANY) {
+		lni = xa_load(&ctx->lnis, handle);
+	} else {
+		xa_for_each(&ctx->lnis, index, candidate) {
+			if (lni) {
+				lni = NULL;
+				break;
+			}
+			lni = candidate;
+		}
+	}
+	if (lni)
+		atomic_inc(&lni->usecnt);
+	mutex_unlock(&ctx->resource_lock);
+	return lni ? lni : ERR_PTR(-EINVAL);
+}
+
+void cxiverbs_lni_put(struct cxiverbs_lni *lni)
+{
+	atomic_dec(&lni->usecnt);
+}
+
+static int cxiverbs_lni_cleanup(struct ib_uobject *uobj,
+			       enum rdma_remove_reason why,
+			       struct uverbs_attr_bundle *attrs)
+{
+	struct cxiverbs_lni *lni = uobj->object;
+	struct cxiverbs_ucontext *ctx = lni->ctx;
+
+	mutex_lock(&ctx->resource_lock);
+	if (atomic_read(&lni->usecnt)) {
+		mutex_unlock(&ctx->resource_lock);
+		return -EBUSY;
+	}
+	xa_erase(&ctx->lnis, lni->handle);
+	mutex_unlock(&ctx->resource_lock);
+	ctx->cadev->lni_free(lni->lni);
+	kfree(lni);
+	return 0;
+}
+
+static int UVERBS_HANDLER(CXI_IB_METHOD_LNI_ALLOC)(
+		struct uverbs_attr_bundle *attrs)
+{
+	struct ib_uobject *uobj =
+		uverbs_attr_get_uobject(attrs, CXI_IB_ATTR_LNI_ALLOC_HANDLE);
+	struct cxiverbs_ucontext *ctx = cxiverbs_get_ucontext(attrs);
+	struct cxiverbs_lni *lni;
+	u32 svc_id;
+	int ret;
+
+	if (IS_ERR(ctx))
+		return PTR_ERR(ctx);
+	ret = uverbs_copy_from(&svc_id, attrs, CXI_IB_ATTR_LNI_ALLOC_SVC_ID);
+	if (ret)
+		return ret;
+	if (!svc_id)
+		return -EINVAL;
+	lni = kzalloc_obj(*lni, GFP_KERNEL);
+	if (!lni)
+		return -ENOMEM;
+	lni->lni = ctx->cadev->lni_alloc(ctx->cadev->cxi_dev, svc_id);
+	if (IS_ERR(lni->lni)) {
+		ret = PTR_ERR(lni->lni);
+		goto err_free;
+	}
+	lni->ctx = ctx;
+	lni->svc_id = svc_id;
+	lni->handle = uobj->id;
+	atomic_set(&lni->usecnt, 0);
+	mutex_lock(&ctx->resource_lock);
+	ret = xa_insert(&ctx->lnis, lni->handle, lni, GFP_KERNEL);
+	mutex_unlock(&ctx->resource_lock);
+	if (ret)
+		goto err_lni;
+	uobj->object = lni;
+	uverbs_finalize_uobj_create(attrs, CXI_IB_ATTR_LNI_ALLOC_HANDLE);
+	return uverbs_copy_to(attrs, CXI_IB_ATTR_LNI_ALLOC_ID,
+			     &lni->lni->id, sizeof(lni->lni->id));
+err_lni:
+	ctx->cadev->lni_free(lni->lni);
+err_free:
+	kfree(lni);
+	return ret;
+}
+
+DECLARE_UVERBS_NAMED_METHOD(CXI_IB_METHOD_LNI_ALLOC,
+	UVERBS_ATTR_IDR(CXI_IB_ATTR_LNI_ALLOC_HANDLE, CXI_IB_OBJECT_LNI,
+			UVERBS_ACCESS_NEW, UA_MANDATORY),
+	UVERBS_ATTR_PTR_IN(CXI_IB_ATTR_LNI_ALLOC_SVC_ID,
+			   UVERBS_ATTR_TYPE(u32), UA_MANDATORY),
+	UVERBS_ATTR_PTR_OUT(CXI_IB_ATTR_LNI_ALLOC_ID,
+			    UVERBS_ATTR_TYPE(u32), UA_MANDATORY));
+
+DECLARE_UVERBS_NAMED_METHOD_DESTROY(CXI_IB_METHOD_LNI_DESTROY,
+	UVERBS_ATTR_IDR(CXI_IB_ATTR_LNI_DESTROY_HANDLE, CXI_IB_OBJECT_LNI,
+			UVERBS_ACCESS_DESTROY, UA_MANDATORY));
+
+DECLARE_UVERBS_NAMED_OBJECT(CXI_IB_OBJECT_LNI,
+	UVERBS_TYPE_ALLOC_IDR(cxiverbs_lni_cleanup),
+	&UVERBS_METHOD(CXI_IB_METHOD_LNI_ALLOC),
+	&UVERBS_METHOD(CXI_IB_METHOD_LNI_DESTROY));
+
+static int UVERBS_HANDLER(CXI_IB_METHOD_PD_PID_ALLOC)(
+		struct uverbs_attr_bundle *attrs)
+{
+	struct ib_pd *ibpd = uverbs_attr_get_obj(attrs, CXI_IB_ATTR_PD_PID_HANDLE);
+	struct cxiverbs_pd *pd = to_cxiverbs_pd(ibpd);
+	struct cxiverbs_dev *dev = to_cxiverbs_dev(ibpd->device);
+	u32 flags = 0;
+	u32 value;
+	int pid;
+	int ret;
+
+	ret = uverbs_get_flags32(&flags, attrs, CXI_IB_ATTR_PD_PID_FLAGS,
+				CXI_IB_PD_PID_ABSOLUTE);
+	if (ret)
+		return ret;
+	if (flags || !dev->cadev->pid_alloc || !dev->cadev->pid_free)
+		return -EOPNOTSUPP;
+	mutex_lock(&pd->resource_lock);
+	if (pd->pidonfep_valid) {
+		ret = -EBUSY;
+		goto out_unlock;
+	}
+	pid = dev->cadev->pid_alloc(pd->lni_ref->lni);
+	if (pid < 0) {
+		ret = pid;
+		goto out_unlock;
+	}
+	value = pid;
+	ret = uverbs_copy_to(attrs, CXI_IB_ATTR_PD_PID_VALUE,
+			     &value, sizeof(value));
+	if (ret) {
+		dev->cadev->pid_free(pd->lni_ref->lni, pid);
+		goto out_unlock;
+	}
+	pd->pidonfep = pid;
+	pd->pidonfep_valid = true;
+out_unlock:
+	mutex_unlock(&pd->resource_lock);
+	return ret;
+}
+
+static int UVERBS_HANDLER(CXI_IB_METHOD_PD_PID_QUERY)(
+		struct uverbs_attr_bundle *attrs)
+{
+	struct ib_pd *ibpd = uverbs_attr_get_obj(attrs, CXI_IB_ATTR_PD_PID_HANDLE);
+	struct cxiverbs_pd *pd = to_cxiverbs_pd(ibpd);
+	int ret;
+
+	mutex_lock(&pd->resource_lock);
+	ret = pd->pidonfep_valid ?
+		uverbs_copy_to(attrs, CXI_IB_ATTR_PD_PID_VALUE,
+			       &pd->pidonfep, sizeof(pd->pidonfep)) : -ENOENT;
+	mutex_unlock(&pd->resource_lock);
+	return ret;
+}
+
+DECLARE_UVERBS_NAMED_METHOD(CXI_IB_METHOD_PD_PID_ALLOC,
+	UVERBS_ATTR_IDR(CXI_IB_ATTR_PD_PID_HANDLE, UVERBS_OBJECT_PD,
+			UVERBS_ACCESS_WRITE, UA_MANDATORY),
+	UVERBS_ATTR_FLAGS_IN(CXI_IB_ATTR_PD_PID_FLAGS, u32, UA_OPTIONAL),
+	UVERBS_ATTR_PTR_OUT(CXI_IB_ATTR_PD_PID_VALUE,
+			    UVERBS_ATTR_TYPE(u32), UA_MANDATORY));
+
+DECLARE_UVERBS_NAMED_METHOD(CXI_IB_METHOD_PD_PID_QUERY,
+	UVERBS_ATTR_IDR(CXI_IB_ATTR_PD_PID_HANDLE, UVERBS_OBJECT_PD,
+			UVERBS_ACCESS_READ, UA_MANDATORY),
+	UVERBS_ATTR_PTR_OUT(CXI_IB_ATTR_PD_PID_VALUE,
+			    UVERBS_ATTR_TYPE(u32), UA_MANDATORY));
+
+DECLARE_UVERBS_GLOBAL_METHODS(UVERBS_OBJECT_PD,
+	&UVERBS_METHOD(CXI_IB_METHOD_PD_PID_ALLOC),
+	&UVERBS_METHOD(CXI_IB_METHOD_PD_PID_QUERY));
+
+static int UVERBS_HANDLER(CXI_IB_METHOD_SVC_ALLOC)(
+		struct uverbs_attr_bundle *attrs)
+{
+	struct cxiverbs_ucontext *ctx = cxiverbs_get_ucontext(attrs);
+	struct cxi_ibv_svc_config config;
+	struct cxi_svc_desc desc = {};
+	struct cxi_svc_fail_info fail = {};
+	struct cxi_limits *limits[] = {
+		&desc.limits.ptes, &desc.limits.txqs, &desc.limits.tgqs,
+		&desc.limits.eqs, &desc.limits.cts, &desc.limits.acs,
+		&desc.limits.tles, &desc.limits.les,
+	};
+	u32 svc_id;
+	unsigned int index;
+	unsigned int vni_index;
+	int ret;
+
+	if (IS_ERR(ctx))
+		return PTR_ERR(ctx);
+	if (!capable(CAP_NET_ADMIN))
+		return -EPERM;
+	ret = uverbs_copy_from(&config, attrs, CXI_IB_ATTR_SVC_CONFIG);
+	if (ret)
+		return ret;
+	if (config.is_system_svc && !capable(CAP_SYS_ADMIN))
+		return -EPERM;
+	if (config.is_system_svc > 1 || config.is_range > 1 ||
+	    config.vni_count > CXI_IB_SVC_MAX_VNIS ||
+	    config.vni_min || config.vni_max ||
+	    (config.tc_mask >> CXI_TC_MAX) ||
+	    (config.flags & ~(CXI_IB_SVC_LIMITS | CXI_IB_SVC_CREATE_TC)))
+		return -EINVAL;
+	if (config.is_range) {
+		if (config.is_system_svc || config.vni_count ||
+		    !config.vnis[0] || config.vnis[0] > config.vnis[1] ||
+		    config.vnis[2] || config.vnis[3])
+			return -EINVAL;
+	} else if (config.is_system_svc) {
+		if (config.svc_id || config.vni_count || config.tc_mask ||
+		    (config.flags & CXI_IB_SVC_CREATE_TC) ||
+		    !(config.flags & CXI_IB_SVC_LIMITS) ||
+		    config.vnis[0] || config.vnis[1] || config.vnis[2] ||
+		    config.vnis[3])
+			return -EINVAL;
+		for (index = 0; index < ARRAY_SIZE(limits); index++) {
+			u16 expected = index == CXI_IB_SVC_EQS ||
+				       index == CXI_IB_SVC_ACS ? 1 : 0;
+
+			if (config.maximum[index] != expected ||
+			    config.reserved[index] != expected)
+				return -EINVAL;
+		}
+	} else if (!config.vni_count || !config.tc_mask) {
+		return -EINVAL;
+	}
+	for (index = 0; index < CXI_IB_SVC_MAX_VNIS; index++) {
+		if (index >= config.vni_count) {
+			if (!config.is_range && config.vnis[index] &&
+			    !config.is_system_svc)
+				return -EINVAL;
+			continue;
+		}
+		if (!config.vnis[index])
+			return -EINVAL;
+		for (vni_index = 0; vni_index < index; vni_index++)
+			if (config.vnis[index] == config.vnis[vni_index])
+				return -EINVAL;
+	}
+	desc.svc_id = config.svc_id;
+	desc.enable = true;
+	desc.restricted_members = true;
+	desc.members[0].type = CXI_SVC_MEMBER_UID;
+	desc.members[0].svc_member.uid = config.uid;
+	desc.is_system_svc = config.is_system_svc;
+	desc.restricted_tcs = !config.is_system_svc;
+	desc.resource_limits = !!(config.flags & CXI_IB_SVC_LIMITS);
+	for (index = 0; index < ARRAY_SIZE(limits); index++) {
+		if (config.reserved[index] > config.maximum[index])
+			return -EINVAL;
+		limits[index]->max = config.maximum[index];
+		limits[index]->res = config.reserved[index];
+	}
+	for (index = 0; index < CXI_TC_MAX; index++) {
+		desc.tcs[index] = !!(config.tc_mask & BIT(index));
+		if (!desc.tcs[index] || !(config.flags & CXI_IB_SVC_CREATE_TC))
+			continue;
+		ret = ctx->cadev->tc_cfg(ctx->cadev->cxi_dev, index,
+					CXI_TC_TYPE_DEFAULT);
+		if (ret && ret != -EEXIST)
+			return ret;
+	}
+	if (config.is_system_svc) {
+		desc.restricted_vnis = true;
+	} else if (config.is_range) {
+		desc.restricted_vnis = false;
+	} else {
+		desc.restricted_vnis = true;
+		desc.num_vld_vnis = config.vni_count;
+		memcpy(desc.vnis, config.vnis,
+		       config.vni_count * sizeof(config.vnis[0]));
+	}
+	ret = ctx->cadev->svc_alloc(ctx->cadev->cxi_dev, &desc, &fail,
+				   "cxidv-service");
+	if (ret < 0)
+		return ret;
+	svc_id = ret;
+	if (config.is_range) {
+		ret = ctx->cadev->svc_set_vni_range(ctx->cadev->cxi_dev, svc_id,
+						   config.vnis[0], config.vnis[1]);
+		if (ret)
+			goto err_destroy;
+		ret = ctx->cadev->svc_enable(ctx->cadev->cxi_dev, svc_id, true);
+		if (ret)
+			goto err_destroy;
+	}
+	ret = uverbs_copy_to(attrs, CXI_IB_ATTR_SVC_ID, &svc_id, sizeof(svc_id));
+	if (!ret)
+		return 0;
+err_destroy:
+	ctx->cadev->svc_destroy(ctx->cadev->cxi_dev, svc_id);
+	return ret;
+}
+
+static int UVERBS_HANDLER(CXI_IB_METHOD_SVC_DESTROY)(
+		struct uverbs_attr_bundle *attrs)
+{
+	struct cxiverbs_ucontext *ctx = cxiverbs_get_ucontext(attrs);
+	u32 svc_id;
+	int ret;
+
+	if (IS_ERR(ctx))
+		return PTR_ERR(ctx);
+	if (!capable(CAP_NET_ADMIN))
+		return -EPERM;
+	ret = uverbs_copy_from(&svc_id, attrs, CXI_IB_ATTR_SVC_ID);
+	return ret ? ret : ctx->cadev->svc_destroy(ctx->cadev->cxi_dev, svc_id);
+}
+
+static int UVERBS_HANDLER(CXI_IB_METHOD_SVC_ENABLE)(
+		struct uverbs_attr_bundle *attrs)
+{
+	struct cxiverbs_ucontext *ctx = cxiverbs_get_ucontext(attrs);
+	u32 svc_id;
+	u32 enable;
+	int ret;
+
+	if (IS_ERR(ctx))
+		return PTR_ERR(ctx);
+	if (!capable(CAP_NET_ADMIN))
+		return -EPERM;
+	ret = uverbs_copy_from(&svc_id, attrs, CXI_IB_ATTR_SVC_ID);
+	if (ret)
+		return ret;
+	ret = uverbs_copy_from(&enable, attrs, CXI_IB_ATTR_SVC_FLAGS);
+	if (ret)
+		return ret;
+	if (enable > 1)
+		return -EINVAL;
+	return ctx->cadev->svc_enable(ctx->cadev->cxi_dev, svc_id, enable);
+}
+
+static int UVERBS_HANDLER(CXI_IB_METHOD_SVC_QUERY)(
+		struct uverbs_attr_bundle *attrs)
+{
+	struct cxiverbs_ucontext *ctx = cxiverbs_get_ucontext(attrs);
+	struct cxi_ibv_svc_config config = {};
+	struct cxi_svc_desc desc;
+	const struct cxi_limits *limits[] = {
+		&desc.limits.ptes, &desc.limits.txqs, &desc.limits.tgqs,
+		&desc.limits.eqs, &desc.limits.cts, &desc.limits.acs,
+		&desc.limits.tles, &desc.limits.les,
+	};
+	unsigned int first, last, index;
+	int ret;
+
+	if (IS_ERR(ctx))
+		return PTR_ERR(ctx);
+	ret = uverbs_copy_from(&config.svc_id, attrs, CXI_IB_ATTR_SVC_ID);
+	if (ret)
+		return ret;
+	ret = ctx->cadev->svc_get(ctx->cadev->cxi_dev, config.svc_id, &desc);
+	if (ret)
+		return ret;
+	if (desc.enable)
+		config.flags |= CXI_IB_SVC_ENABLED;
+	if (desc.resource_limits)
+		config.flags |= CXI_IB_SVC_LIMITS;
+	config.is_system_svc = desc.is_system_svc;
+	if (uverbs_attr_is_valid(attrs, CXI_IB_ATTR_SVC_FLAGS)) {
+		ret = uverbs_copy_to(attrs, CXI_IB_ATTR_SVC_FLAGS,
+				     &config.flags, sizeof(config.flags));
+		if (ret)
+			return ret;
+	}
+	if (uverbs_attr_is_valid(attrs, CXI_IB_ATTR_SVC_USAGE)) {
+		struct cxi_rsrc_use native = {};
+		struct cxi_ibv_svc_rsrc_use usage = {};
+
+		BUILD_BUG_ON(sizeof(native.in_use) != sizeof(usage.in_use));
+		ret = ctx->cadev->svc_rsrc_get(ctx->cadev->cxi_dev,
+					      config.svc_id, &native);
+		if (ret)
+			return ret;
+		usage.svc_id = native.svc_id;
+		usage.tle_pool_id = native.tle_pool_id;
+		memcpy(usage.in_use, native.in_use, sizeof(usage.in_use));
+		ret = uverbs_copy_to(attrs, CXI_IB_ATTR_SVC_USAGE,
+				     &usage, sizeof(usage));
+		if (ret)
+			return ret;
+	}
+	if (uverbs_attr_is_valid(attrs, CXI_IB_ATTR_SVC_LPR)) {
+		u32 lpr;
+
+		ret = cxi_svc_get_lpr(ctx->cadev->cxi_dev, config.svc_id);
+		if (ret < 0)
+			return ret;
+		lpr = ret;
+		ret = uverbs_copy_to(attrs, CXI_IB_ATTR_SVC_LPR, &lpr, sizeof(lpr));
+		if (ret)
+			return ret;
+	}
+	if (!uverbs_attr_is_valid(attrs, CXI_IB_ATTR_SVC_CONFIG))
+		return uverbs_attr_is_valid(attrs, CXI_IB_ATTR_SVC_FLAGS) ||
+		       uverbs_attr_is_valid(attrs, CXI_IB_ATTR_SVC_USAGE) ||
+		       uverbs_attr_is_valid(attrs, CXI_IB_ATTR_SVC_LPR) ? 0 : -EINVAL;
+	if (desc.restricted_vnis) {
+		if ((!desc.num_vld_vnis && !desc.is_system_svc) ||
+		    desc.num_vld_vnis > CXI_IB_SVC_MAX_VNIS)
+			return -EOPNOTSUPP;
+		config.vni_count = desc.num_vld_vnis;
+		if (desc.num_vld_vnis)
+			memcpy(config.vnis, desc.vnis,
+			       desc.num_vld_vnis * sizeof(config.vnis[0]));
+	} else {
+		ret = ctx->cadev->svc_get_vni_range(ctx->cadev->cxi_dev,
+						   config.svc_id, &first, &last);
+		if (ret)
+			return ret;
+		config.is_range = 1;
+		config.vnis[0] = first;
+		config.vnis[1] = last;
+	}
+	for (index = 0; index < CXI_TC_MAX; index++)
+		if (!desc.is_system_svc &&
+		    (!desc.restricted_tcs || desc.tcs[index]))
+			config.tc_mask |= BIT(index);
+	config.uid = desc.members[0].svc_member.uid;
+	for (index = 0; index < ARRAY_SIZE(limits); index++) {
+		config.maximum[index] = limits[index]->max;
+		config.reserved[index] = limits[index]->res;
+	}
+	return uverbs_copy_to(attrs, CXI_IB_ATTR_SVC_CONFIG,
+			     &config, sizeof(config));
+}
+
+DECLARE_UVERBS_NAMED_METHOD(CXI_IB_METHOD_SVC_ALLOC,
+	UVERBS_ATTR_PTR_IN(CXI_IB_ATTR_SVC_CONFIG,
+			   UVERBS_ATTR_TYPE(struct cxi_ibv_svc_config), UA_MANDATORY),
+	UVERBS_ATTR_PTR_OUT(CXI_IB_ATTR_SVC_ID,
+			    UVERBS_ATTR_TYPE(u32), UA_MANDATORY));
+
+DECLARE_UVERBS_NAMED_METHOD(CXI_IB_METHOD_SVC_DESTROY,
+	UVERBS_ATTR_PTR_IN(CXI_IB_ATTR_SVC_ID,
+			   UVERBS_ATTR_TYPE(u32), UA_MANDATORY));
+
+DECLARE_UVERBS_NAMED_METHOD(CXI_IB_METHOD_SVC_ENABLE,
+	UVERBS_ATTR_PTR_IN(CXI_IB_ATTR_SVC_ID,
+			   UVERBS_ATTR_TYPE(u32), UA_MANDATORY),
+	UVERBS_ATTR_PTR_IN(CXI_IB_ATTR_SVC_FLAGS,
+			   UVERBS_ATTR_TYPE(u32), UA_MANDATORY));
+
+DECLARE_UVERBS_NAMED_METHOD(CXI_IB_METHOD_SVC_QUERY,
+	UVERBS_ATTR_PTR_IN(CXI_IB_ATTR_SVC_ID,
+			   UVERBS_ATTR_TYPE(u32), UA_MANDATORY),
+	UVERBS_ATTR_PTR_OUT(CXI_IB_ATTR_SVC_CONFIG,
+			    UVERBS_ATTR_TYPE(struct cxi_ibv_svc_config), UA_OPTIONAL),
+	UVERBS_ATTR_PTR_OUT(CXI_IB_ATTR_SVC_FLAGS,
+			    UVERBS_ATTR_TYPE(u32), UA_OPTIONAL),
+	UVERBS_ATTR_PTR_OUT(CXI_IB_ATTR_SVC_USAGE,
+			    UVERBS_ATTR_TYPE(struct cxi_ibv_svc_rsrc_use), UA_OPTIONAL),
+	UVERBS_ATTR_PTR_OUT(CXI_IB_ATTR_SVC_LPR,
+			    UVERBS_ATTR_TYPE(u32), UA_OPTIONAL));
+
+DECLARE_UVERBS_GLOBAL_METHODS(CXI_IB_OBJECT_SVC,
+	&UVERBS_METHOD(CXI_IB_METHOD_SVC_ALLOC),
+	&UVERBS_METHOD(CXI_IB_METHOD_SVC_DESTROY),
+	&UVERBS_METHOD(CXI_IB_METHOD_SVC_QUERY),
+	&UVERBS_METHOD(CXI_IB_METHOD_SVC_ENABLE));
+
+struct cxiverbs_wait {
+	struct ib_uobject uobj;
+	struct cxiverbs_lni *lni;
+	wait_queue_head_t poll_wait;
+	struct mutex read_lock;
+	atomic64_t generation;
+	atomic64_t cleared;
+	bool initialized;
+	bool dead;
+};
+
+static ssize_t cxiverbs_wait_read(struct file *file, char __user *buffer,
+				size_t length, loff_t *offset)
+{
+	struct cxiverbs_wait *wait =
+		container_of(file->private_data, struct cxiverbs_wait, uobj);
+	u64 generation;
+	char value = '1';
+	int ret;
+
+	if (!length)
+		return 0;
+	ret = mutex_lock_interruptible(&wait->read_lock);
+	if (ret)
+		return ret;
+	if (READ_ONCE(wait->dead)) {
+		ret = -EIO;
+		goto unlock;
+	}
+	generation = atomic64_read(&wait->generation);
+	if (copy_to_user(buffer, &value, sizeof(value))) {
+		ret = -EFAULT;
+		goto unlock;
+	}
+	atomic64_set(&wait->cleared, generation);
+	ret = sizeof(value);
+unlock:
+	mutex_unlock(&wait->read_lock);
+	return ret;
+}
+
+static __poll_t cxiverbs_wait_poll(struct file *file, poll_table *table)
+{
+	struct cxiverbs_wait *wait =
+		container_of(file->private_data, struct cxiverbs_wait, uobj);
+
+	poll_wait(file, &wait->poll_wait, table);
+	if (READ_ONCE(wait->dead))
+		return EPOLLHUP | EPOLLERR;
+	if (atomic64_read(&wait->generation) != atomic64_read(&wait->cleared))
+		return EPOLLPRI;
+	return 0;
+}
+
+static const struct file_operations cxiverbs_wait_fops = {
+	.owner = THIS_MODULE,
+	.read = cxiverbs_wait_read,
+	.poll = cxiverbs_wait_poll,
+	.release = uverbs_uobject_fd_release,
+	.llseek = noop_llseek,
+};
+
+static void cxiverbs_wait_cleanup(struct ib_uobject *object,
+				 enum rdma_remove_reason reason)
+{
+	struct cxiverbs_wait *wait =
+		container_of(object, struct cxiverbs_wait, uobj);
+
+	if (!wait->initialized)
+		return;
+	WRITE_ONCE(wait->dead, true);
+	wake_up_interruptible_poll(&wait->poll_wait, EPOLLHUP | EPOLLERR);
+	if (wait->lni) {
+		cxiverbs_lni_put(wait->lni);
+		wait->lni = NULL;
+	}
+}
+
+static int UVERBS_HANDLER(CXI_IB_METHOD_WAIT_ALLOC)(
+		struct uverbs_attr_bundle *attrs)
+{
+	struct ib_uobject *object =
+		uverbs_attr_get_uobject(attrs, CXI_IB_ATTR_WAIT_FD);
+	struct file *file = object->object;
+	struct cxiverbs_wait *wait =
+		container_of(object, struct cxiverbs_wait, uobj);
+	struct cxiverbs_ucontext *ctx;
+	u32 handle;
+	int ret;
+
+	init_waitqueue_head(&wait->poll_wait);
+	mutex_init(&wait->read_lock);
+	atomic64_set(&wait->generation, 1);
+	atomic64_set(&wait->cleared, 0);
+	wait->initialized = true;
+	ctx = cxiverbs_get_ucontext(attrs);
+	if (IS_ERR(ctx))
+		return PTR_ERR(ctx);
+	ret = uverbs_copy_from(&handle, attrs, CXI_IB_ATTR_WAIT_LNI);
+	if (ret)
+		return ret;
+	wait->lni = cxiverbs_lni_get(ctx, handle);
+	if (IS_ERR(wait->lni)) {
+		ret = PTR_ERR(wait->lni);
+		wait->lni = NULL;
+		return ret;
+	}
+	file->f_mode |= FMODE_PREAD;
+	return 0;
+}
+
+DECLARE_UVERBS_NAMED_METHOD(CXI_IB_METHOD_WAIT_ALLOC,
+	UVERBS_ATTR_FD(CXI_IB_ATTR_WAIT_FD, CXI_IB_OBJECT_WAIT,
+		       UVERBS_ACCESS_NEW, UA_MANDATORY),
+	UVERBS_ATTR_PTR_IN(CXI_IB_ATTR_WAIT_LNI,
+			   UVERBS_ATTR_TYPE(u32), UA_MANDATORY));
+
+DECLARE_UVERBS_NAMED_OBJECT(CXI_IB_OBJECT_WAIT,
+	UVERBS_TYPE_ALLOC_FD(sizeof(struct cxiverbs_wait),
+			     cxiverbs_wait_cleanup,
+			     &cxiverbs_wait_fops, "[cxiwait]", O_RDONLY),
+	&UVERBS_METHOD(CXI_IB_METHOD_WAIT_ALLOC));
+
+static void cxiverbs_wait_notify(void *data)
+{
+	struct file *file = data;
+	struct cxiverbs_wait *wait =
+		container_of(file->private_data, struct cxiverbs_wait, uobj);
+
+	if (READ_ONCE(wait->dead))
+		return;
+	atomic64_inc(&wait->generation);
+	wake_up_interruptible_poll(&wait->poll_wait, EPOLLPRI);
+}
+
+static int cxiverbs_eq_get_wait(struct uverbs_attr_bundle *attrs, u16 attr,
+			       struct cxiverbs_lni *lni, struct file **file)
+{
+	struct ib_uobject *object;
+	struct cxiverbs_wait *wait;
+
+	if (!uverbs_attr_is_valid(attrs, attr))
+		return 0;
+	object = uverbs_attr_get_uobject(attrs, attr);
+	wait = container_of(object, struct cxiverbs_wait, uobj);
+	if (READ_ONCE(wait->dead) || wait->lni != lni)
+		return -EINVAL;
+	*file = get_file(object->object);
+	return 0;
+}
+
+struct cxiverbs_native_eq {
+	struct cxiverbs_lni *lni;
+	struct cxiverbs_mr *mr;
+	struct file *event_wait;
+	struct file *status_wait;
+	struct cxi_eq *eq;
+	struct cxiverbs_mmap_entry *doorbell;
+	atomic_t users;
+};
+
+/*
+ * Runs once the hardware EQ is gone and the last doorbell mapping is closed.
+ * Only drops references and frees memory, so it cannot fail.
+ */
+static void cxiverbs_native_eq_release(void *owner)
+{
+	struct cxiverbs_native_eq *queue = owner;
+
+	if (queue->status_wait)
+		fput(queue->status_wait);
+	if (queue->event_wait)
+		fput(queue->event_wait);
+	if (queue->mr)
+		atomic_dec(&queue->mr->native_users);
+	cxiverbs_lni_put(queue->lni);
+	kfree(queue);
+}
+
+static int cxiverbs_native_eq_cleanup(struct ib_uobject *object,
+				     enum rdma_remove_reason reason,
+				     struct uverbs_attr_bundle *attrs)
+{
+	struct cxiverbs_native_eq *queue = object->object;
+	int ret;
+
+	if (atomic_read(&queue->users))
+		return -EBUSY;
+	/*
+	 * eq_free() can fail (a VF may be unable to reach the PF) and leaves
+	 * the EQ untouched when it does. Do it here, where rdma-core can
+	 * report the error and retry, rather than from mmap_free(), which
+	 * has no way to fail.
+	 */
+	ret = queue->lni->ctx->cadev->eq_free(queue->eq);
+	if (ret)
+		return ret;
+	/*
+	 * The queue, and the LNI reference it holds, stay alive until the last
+	 * doorbell mapping is closed, so the EQ number cannot be recycled
+	 * while the page is still mapped.
+	 */
+	if (queue->doorbell)
+		rdma_user_mmap_entry_remove(&queue->doorbell->rdma_entry);
+	else
+		cxiverbs_native_eq_release(queue);
+	return 0;
+}
+
+static int UVERBS_HANDLER(CXI_IB_METHOD_EQ_ALLOC)(struct uverbs_attr_bundle *attrs)
+{
+	struct cxiverbs_ucontext *ctx = cxiverbs_get_ucontext(attrs);
+	struct ib_uobject *object = uverbs_attr_get_uobject(attrs, CXI_IB_ATTR_EQ_HANDLE);
+	struct ib_mr *ibmr = NULL;
+	struct cxiverbs_mr *mr = NULL;
+	struct cxiverbs_native_eq *queue;
+	struct cxi_ibv_eq_config config;
+	struct cxi_ibv_eq_info info = {};
+	struct cxi_eq_attr native = {};
+	phys_addr_t physical;
+	size_t size;
+	u64 offset;
+	u32 lni_handle;
+	u32 flags = 0;
+	bool passthrough;
+	int ret;
+
+	if (IS_ERR(ctx))
+		return PTR_ERR(ctx);
+	ret = uverbs_get_flags32(&flags, attrs, CXI_IB_ATTR_EQ_FLAGS,
+				CXI_IB_EQ_REGISTER_PCT | CXI_IB_EQ_PASSTHROUGH);
+	if (ret)
+		return ret;
+	passthrough = flags & CXI_IB_EQ_PASSTHROUGH;
+	if (passthrough && !(flags & CXI_IB_EQ_REGISTER_PCT))
+		return -EINVAL;
+	if (passthrough == uverbs_attr_is_valid(attrs, CXI_IB_ATTR_EQ_MR))
+		return -EINVAL;
+	if (!passthrough) {
+		ibmr = uverbs_attr_get_obj(attrs, CXI_IB_ATTR_EQ_MR);
+		mr = to_cxiverbs_mr(ibmr);
+	}
+	if (flags & CXI_IB_EQ_REGISTER_PCT) {
+		struct cxi_dev *cdev = ctx->cadev->cxi_dev;
+		u32 generation = cdev->prop.cassini_version & 0xf0;
+
+		if (!capable(CAP_SYS_ADMIN))
+			return -EPERM;
+		if (!cdev->is_physfn ||
+		    (generation != CASSINI_1 && generation != CASSINI_2))
+			return -EOPNOTSUPP;
+	}
+	ret = uverbs_copy_from(&lni_handle, attrs, CXI_IB_ATTR_EQ_LNI);
+	if (ret)
+		return ret;
+	ret = uverbs_copy_from(&config, attrs, CXI_IB_ATTR_EQ_CONFIG);
+	if (ret)
+		return ret;
+	if (!config.length)
+		return -EINVAL;
+	if (mr && (config.address < mr->md->va ||
+		   config.length > mr->md->len ||
+		   config.address - mr->md->va > mr->md->len - config.length))
+		return -EINVAL;
+	queue = kzalloc_obj(*queue, GFP_KERNEL);
+	if (!queue)
+		return -ENOMEM;
+	queue->lni = cxiverbs_lni_get(ctx, lni_handle);
+	if (IS_ERR(queue->lni)) {
+		ret = PTR_ERR(queue->lni);
+		goto free_queue;
+	}
+	if (mr && to_cxiverbs_pd(ibmr->pd)->lni_ref != queue->lni) {
+		ret = -EINVAL;
+		goto put_lni;
+	}
+	queue->mr = mr;
+	if (mr)
+		atomic_inc(&mr->native_users);
+	native.queue = (void *)(uintptr_t)config.address;
+	native.queue_len = config.length;
+	native.flags = CXI_EQ_USER;
+	if (flags & CXI_IB_EQ_REGISTER_PCT)
+		native.flags |= CXI_EQ_REGISTER_PCT;
+	if (passthrough)
+		native.flags |= CXI_EQ_PASSTHROUGH;
+	native.status_thresh_base = config.threshold_base;
+	native.status_thresh_delta = config.threshold_delta;
+	native.status_thresh_count = config.threshold_count;
+	native.ec_delay = config.ec_delay;
+	native.cpu_affinity = config.cpu;
+	native.reserved_slots = config.reserved_slots;
+	ret = cxiverbs_eq_get_wait(attrs, CXI_IB_ATTR_EQ_EVENT_WAIT,
+				   queue->lni, &queue->event_wait);
+	if (ret)
+		goto put_waits;
+	ret = cxiverbs_eq_get_wait(attrs, CXI_IB_ATTR_EQ_STATUS_WAIT,
+				   queue->lni, &queue->status_wait);
+	if (ret)
+		goto put_waits;
+	queue->eq = ctx->cadev->eq_alloc(queue->lni->lni, mr ? mr->md : NULL,
+		&native, queue->event_wait ? cxiverbs_wait_notify : NULL,
+		queue->event_wait,
+		queue->status_wait ? cxiverbs_wait_notify : NULL,
+		queue->status_wait);
+	if (IS_ERR(queue->eq)) {
+		ret = PTR_ERR(queue->eq);
+		goto put_waits;
+	}
+	ret = ctx->cadev->eq_user_info(queue->eq, &physical, &size);
+	if (ret)
+		goto destroy_eq;
+	if (flags & CXI_IB_EQ_REGISTER_PCT)
+		queue->doorbell = cxiverbs_mmap_entry_alloc_admin(
+			&ctx->ibucontext, physical, size, &offset);
+	else
+		queue->doorbell = cxiverbs_mmap_entry_alloc(
+			&ctx->ibucontext, true, physical, NULL, size, &offset);
+	if (IS_ERR(queue->doorbell)) {
+		ret = PTR_ERR(queue->doorbell);
+		queue->doorbell = NULL;
+		goto destroy_eq;
+	}
+	queue->doorbell->owner = queue;
+	queue->doorbell->release_owner = cxiverbs_native_eq_release;
+	info.doorbell_offset = offset;
+	info.doorbell_size = size;
+	info.eqn = queue->eq->eqn;
+	object->object = queue;
+	uverbs_finalize_uobj_create(attrs, CXI_IB_ATTR_EQ_HANDLE);
+	return uverbs_copy_to(attrs, CXI_IB_ATTR_EQ_INFO, &info, sizeof(info));
+destroy_eq:
+	/*
+	 * The hardware EQ exists, so let rdma-core unwind it through
+	 * cxiverbs_native_eq_cleanup(). If eq_free() fails there, the core
+	 * keeps the object and retries instead of leaking it.
+	 */
+	object->object = queue;
+	uverbs_finalize_uobj_create(attrs, CXI_IB_ATTR_EQ_HANDLE);
+	return ret;
+put_waits:
+	if (queue->status_wait)
+		fput(queue->status_wait);
+	if (queue->event_wait)
+		fput(queue->event_wait);
+	if (mr)
+		atomic_dec(&mr->native_users);
+put_lni:
+	cxiverbs_lni_put(queue->lni);
+free_queue:
+	kfree(queue);
+	return ret;
+}
+
+DECLARE_UVERBS_NAMED_METHOD(CXI_IB_METHOD_EQ_ALLOC,
+	UVERBS_ATTR_IDR(CXI_IB_ATTR_EQ_HANDLE, CXI_IB_OBJECT_EQ,
+			UVERBS_ACCESS_NEW, UA_MANDATORY),
+	UVERBS_ATTR_PTR_IN(CXI_IB_ATTR_EQ_LNI, UVERBS_ATTR_TYPE(u32), UA_MANDATORY),
+	UVERBS_ATTR_IDR(CXI_IB_ATTR_EQ_MR, UVERBS_OBJECT_MR,
+			UVERBS_ACCESS_READ, UA_OPTIONAL),
+	UVERBS_ATTR_PTR_IN(CXI_IB_ATTR_EQ_CONFIG,
+			   UVERBS_ATTR_TYPE(struct cxi_ibv_eq_config), UA_MANDATORY),
+	UVERBS_ATTR_PTR_OUT(CXI_IB_ATTR_EQ_INFO,
+			    UVERBS_ATTR_TYPE(struct cxi_ibv_eq_info),
+			    UA_MANDATORY),
+	UVERBS_ATTR_FLAGS_IN(CXI_IB_ATTR_EQ_FLAGS, u32, UA_OPTIONAL),
+	UVERBS_ATTR_FD(CXI_IB_ATTR_EQ_EVENT_WAIT, CXI_IB_OBJECT_WAIT,
+		       UVERBS_ACCESS_READ, UA_OPTIONAL),
+	UVERBS_ATTR_FD(CXI_IB_ATTR_EQ_STATUS_WAIT, CXI_IB_OBJECT_WAIT,
+		       UVERBS_ACCESS_READ, UA_OPTIONAL));
+
+DECLARE_UVERBS_NAMED_METHOD_DESTROY(CXI_IB_METHOD_EQ_DESTROY,
+	UVERBS_ATTR_IDR(CXI_IB_ATTR_EQ_HANDLE, CXI_IB_OBJECT_EQ,
+			UVERBS_ACCESS_DESTROY, UA_MANDATORY));
+
+DECLARE_UVERBS_NAMED_OBJECT(CXI_IB_OBJECT_EQ,
+	UVERBS_TYPE_ALLOC_IDR(cxiverbs_native_eq_cleanup),
+	&UVERBS_METHOD(CXI_IB_METHOD_EQ_ALLOC),
+	&UVERBS_METHOD(CXI_IB_METHOD_EQ_DESTROY));
+
+struct cxiverbs_cp {
+	struct cxiverbs_lni *lni;
+	struct cxi_cp *cp;
+};
+
+static int cxiverbs_cp_cleanup(struct ib_uobject *object,
+			      enum rdma_remove_reason reason,
+			      struct uverbs_attr_bundle *attrs)
+{
+	struct cxiverbs_cp *profile = object->object;
+
+	profile->lni->ctx->cadev->cp_free(profile->cp);
+	cxiverbs_lni_put(profile->lni);
+	kfree(profile);
+	return 0;
+}
+
+static int UVERBS_HANDLER(CXI_IB_METHOD_CP_ALLOC)(struct uverbs_attr_bundle *attrs)
+{
+	struct ib_uobject *object = uverbs_attr_get_uobject(attrs, CXI_IB_ATTR_CP_HANDLE);
+	struct cxiverbs_ucontext *ctx = cxiverbs_get_ucontext(attrs);
+	struct cxiverbs_cp *profile;
+	struct cxi_ibv_cp_info info;
+	u32 lni_handle, vni, tc, type;
+	int ret;
+
+	if (IS_ERR(ctx))
+		return PTR_ERR(ctx);
+	ret = uverbs_copy_from(&lni_handle, attrs, CXI_IB_ATTR_CP_LNI);
+	if (ret)
+		return ret;
+	ret = uverbs_copy_from(&vni, attrs, CXI_IB_ATTR_CP_VNI);
+	if (ret)
+		return ret;
+	ret = uverbs_copy_from(&tc, attrs, CXI_IB_ATTR_CP_TC);
+	if (ret)
+		return ret;
+	ret = uverbs_copy_from(&type, attrs, CXI_IB_ATTR_CP_TYPE);
+	if (ret)
+		return ret;
+	if (!vni || vni > U16_MAX || tc >= CXI_TC_MAX || type >= CXI_TC_TYPE_MAX)
+		return -EINVAL;
+	profile = kzalloc_obj(*profile, GFP_KERNEL);
+	if (!profile)
+		return -ENOMEM;
+	profile->lni = cxiverbs_lni_get(ctx, lni_handle);
+	if (IS_ERR(profile->lni)) {
+		ret = PTR_ERR(profile->lni);
+		goto free_profile;
+	}
+	profile->cp = ctx->cadev->cp_alloc(profile->lni->lni, vni, tc, type);
+	if (IS_ERR(profile->cp)) {
+		ret = PTR_ERR(profile->cp);
+		goto put_lni;
+	}
+	info.vni = profile->cp->vni_pcp;
+	info.tc = profile->cp->tc;
+	info.type = profile->cp->tc_type;
+	info.lcid = profile->cp->lcid;
+	object->object = profile;
+	uverbs_finalize_uobj_create(attrs, CXI_IB_ATTR_CP_HANDLE);
+	return uverbs_copy_to(attrs, CXI_IB_ATTR_CP_INFO, &info, sizeof(info));
+put_lni:
+	cxiverbs_lni_put(profile->lni);
+free_profile:
+	kfree(profile);
+	return ret;
+}
+
+DECLARE_UVERBS_NAMED_METHOD(CXI_IB_METHOD_CP_ALLOC,
+	UVERBS_ATTR_IDR(CXI_IB_ATTR_CP_HANDLE, CXI_IB_OBJECT_CP,
+			UVERBS_ACCESS_NEW, UA_MANDATORY),
+	UVERBS_ATTR_PTR_IN(CXI_IB_ATTR_CP_LNI, UVERBS_ATTR_TYPE(u32), UA_MANDATORY),
+	UVERBS_ATTR_PTR_IN(CXI_IB_ATTR_CP_VNI, UVERBS_ATTR_TYPE(u32), UA_MANDATORY),
+	UVERBS_ATTR_PTR_IN(CXI_IB_ATTR_CP_TC, UVERBS_ATTR_TYPE(u32), UA_MANDATORY),
+	UVERBS_ATTR_PTR_IN(CXI_IB_ATTR_CP_TYPE, UVERBS_ATTR_TYPE(u32), UA_MANDATORY),
+	UVERBS_ATTR_PTR_OUT(CXI_IB_ATTR_CP_INFO,
+			    UVERBS_ATTR_TYPE(struct cxi_ibv_cp_info), UA_MANDATORY));
+
+DECLARE_UVERBS_NAMED_METHOD_DESTROY(CXI_IB_METHOD_CP_DESTROY,
+	UVERBS_ATTR_IDR(CXI_IB_ATTR_CP_HANDLE, CXI_IB_OBJECT_CP,
+			UVERBS_ACCESS_DESTROY, UA_MANDATORY));
+
+DECLARE_UVERBS_NAMED_OBJECT(CXI_IB_OBJECT_CP,
+	UVERBS_TYPE_ALLOC_IDR(cxiverbs_cp_cleanup),
+	&UVERBS_METHOD(CXI_IB_METHOD_CP_ALLOC),
+	&UVERBS_METHOD(CXI_IB_METHOD_CP_DESTROY));
+
+/* -----------------------------------------------------------------------
+ * Per-CT kernel object.  Heap-allocated; freed in cxiverbs_ct_cleanup().
+ * -----------------------------------------------------------------------
+ */
+struct cxiverbs_ct {
+	refcount_t refs;
+	struct cxi_ct            *ct;       /* CXI counting trigger          */
+	struct cxiverbs_lni *lni_ref;
+	struct page              *wb_page;  /* writeback-buffer page         */
+	struct cxiverbs_mmap_entry *wb_mmap;  /* userspace mmap: writeback buf */
+	struct cxiverbs_mmap_entry *db_mmap;  /* userspace mmap: CT doorbell   */
+};
+
+static void cxiverbs_ct_release(void *owner)
+{
+	struct cxiverbs_ct *ct = owner;
+
+	if (!refcount_dec_and_test(&ct->refs))
+		return;
+	if (ct->ct)
+		cxi_ct_free(ct->ct);
+	if (ct->wb_page)
+		unpin_user_page(ct->wb_page);
+	cxiverbs_lni_put(ct->lni_ref);
+	kfree(ct);
+}
+
+/* -----------------------------------------------------------------------
+ * IDR object cleanup — called on CT_DESTROY and on context teardown.
+ * -----------------------------------------------------------------------
+ */
+static int cxiverbs_ct_cleanup(struct ib_uobject *uobj,
+			      enum rdma_remove_reason why,
+			      struct uverbs_attr_bundle *attrs)
+{
+	struct cxiverbs_ct *ct = uobj->object;
+
+	if (ct->db_mmap)
+		rdma_user_mmap_entry_remove(&ct->db_mmap->rdma_entry);
+	if (ct->wb_mmap)
+		rdma_user_mmap_entry_remove(&ct->wb_mmap->rdma_entry);
+	cxiverbs_ct_release(ct);
+	return 0;
+}
+
+/* -----------------------------------------------------------------------
+ * CXI_IB_METHOD_CT_ALLOC handler
+ * -----------------------------------------------------------------------
+ */
+static int UVERBS_HANDLER(CXI_IB_METHOD_CT_ALLOC)(
+		struct uverbs_attr_bundle *attrs)
+{
+	struct ib_uobject  *uobj   = uverbs_attr_get_uobject(attrs,
+					CXI_IB_ATTR_CT_ALLOC_HANDLE);
+	struct ib_ucontext *ibctx  = ib_uverbs_get_ucontext(attrs);
+	struct cxiverbs_ucontext *uctx;
+	struct cxi_adev *cadev;
+	struct cxiverbs_ct   *ct_obj;
+	struct c_ct_writeback *wb;
+	phys_addr_t db_phys;
+	size_t      db_sz;
+	u64         wb_offset = 0, db_offset = 0;
+	u64         wb_len, db_len;
+	u64 wb_address;
+	u32         ctn;
+	u32 lni_handle = CXI_IBV_LNI_ANY;
+	int         ret;
+
+	if (IS_ERR(ibctx))
+		return PTR_ERR(ibctx);
+
+	uctx = to_cxiverbs_ucontext(ibctx);
+	cadev = uctx->cadev;
+	ret = uverbs_copy_from(&wb_address, attrs,
+			       CXI_IB_ATTR_CT_ALLOC_WB_ADDRESS);
+	if (ret)
+		return ret;
+	if (!wb_address || !IS_ALIGNED(wb_address, PAGE_SIZE) ||
+	    wb_address != (unsigned long)wb_address ||
+	    !access_ok(u64_to_user_ptr(wb_address), PAGE_SIZE))
+		return -EINVAL;
+
+	ct_obj = kzalloc_obj(*ct_obj);
+	if (!ct_obj)
+		return -ENOMEM;
+	if (uverbs_attr_is_valid(attrs, CXI_IB_ATTR_CT_ALLOC_LNI_HANDLE)) {
+		ret = uverbs_copy_from(&lni_handle, attrs,
+				       CXI_IB_ATTR_CT_ALLOC_LNI_HANDLE);
+		if (ret)
+			goto err_free_object;
+	}
+	ct_obj->lni_ref = cxiverbs_lni_get(uctx, lni_handle);
+	if (IS_ERR(ct_obj->lni_ref)) {
+		ret = PTR_ERR(ct_obj->lni_ref);
+		goto err_free_object;
+	}
+	refcount_set(&ct_obj->refs, 1);
+
+	ret = pin_user_pages_fast(wb_address, 1, FOLL_WRITE | FOLL_LONGTERM,
+				  &ct_obj->wb_page);
+	if (ret != 1) {
+		ct_obj->wb_page = NULL;
+		if (ret >= 0)
+			ret = -EFAULT;
+		goto err_release;
+	}
+	wb = (struct c_ct_writeback *)(unsigned long)wb_address;
+
+	ct_obj->ct = cadev->ct_alloc(ct_obj->lni_ref->lni, wb, true);
+	if (IS_ERR(ct_obj->ct)) {
+		ret = PTR_ERR(ct_obj->ct);
+		ct_obj->ct = NULL;
+		goto err_release;
+	}
+
+	/* Get the physical address of the CT doorbell CSR for mmap. */
+	ret = cxi_ct_user_info(ct_obj->ct, &db_phys, &db_sz);
+	if (ret)
+		goto err_release;
+
+	wb_len = PAGE_SIZE;
+	db_len = (u64)db_sz;
+
+	/* Create mmap entry for the writeback buffer page. */
+	refcount_inc(&ct_obj->refs);
+	ct_obj->wb_mmap = cxiverbs_mmap_entry_alloc_owned(ibctx, false, 0,
+				ct_obj->wb_page, wb_len, &wb_offset, ct_obj,
+				cxiverbs_ct_release);
+	if (IS_ERR(ct_obj->wb_mmap)) {
+		ret = PTR_ERR(ct_obj->wb_mmap);
+		ct_obj->wb_mmap = NULL;
+		refcount_dec(&ct_obj->refs);
+		goto err_release;
+	}
+
+	/* Create mmap entry for the CT doorbell CSR (IO mapping). */
+	refcount_inc(&ct_obj->refs);
+	ct_obj->db_mmap = cxiverbs_mmap_entry_alloc_owned(ibctx, true, db_phys,
+				NULL, db_len, &db_offset, ct_obj,
+				cxiverbs_ct_release);
+	if (IS_ERR(ct_obj->db_mmap)) {
+		ret = PTR_ERR(ct_obj->db_mmap);
+		ct_obj->db_mmap = NULL;
+		refcount_dec(&ct_obj->refs);
+		goto err_remove_wb;
+	}
+
+	ctn = ct_obj->ct->ctn;
+
+	/* Commit the uobject — after this, returning an error will cause the
+	 * framework to call cxiverbs_ct_cleanup() automatically.
+	 */
+	uobj->object = ct_obj;
+	uverbs_finalize_uobj_create(attrs, CXI_IB_ATTR_CT_ALLOC_HANDLE);
+
+	ret = uverbs_copy_to(attrs, CXI_IB_ATTR_CT_ALLOC_CTN,
+			     &ctn, sizeof(ctn));
+	if (ret)
+		return ret;
+
+	ret = uverbs_copy_to(attrs, CXI_IB_ATTR_CT_ALLOC_WB_MMAP_OFFSET,
+			     &wb_offset, sizeof(wb_offset));
+	if (ret)
+		return ret;
+
+	ret = uverbs_copy_to(attrs, CXI_IB_ATTR_CT_ALLOC_WB_MMAP_LENGTH,
+			     &wb_len, sizeof(wb_len));
+	if (ret)
+		return ret;
+
+	ret = uverbs_copy_to(attrs, CXI_IB_ATTR_CT_ALLOC_DB_MMAP_OFFSET,
+			     &db_offset, sizeof(db_offset));
+	if (ret)
+		return ret;
+
+	ret = uverbs_copy_to(attrs, CXI_IB_ATTR_CT_ALLOC_DB_MMAP_LENGTH,
+			     &db_len, sizeof(db_len));
+	return ret;
+
+err_remove_wb:
+	rdma_user_mmap_entry_remove(&ct_obj->wb_mmap->rdma_entry);
+err_release:
+	cxiverbs_ct_release(ct_obj);
+	return ret;
+err_free_object:
+	kfree(ct_obj);
+	return ret;
+}
+
+/* -----------------------------------------------------------------------
+ * Method definitions
+ * -----------------------------------------------------------------------
+ */
+DECLARE_UVERBS_NAMED_METHOD(
+	CXI_IB_METHOD_CT_ALLOC,
+	UVERBS_ATTR_PTR_IN(CXI_IB_ATTR_CT_ALLOC_WB_ADDRESS,
+			   UVERBS_ATTR_TYPE(u64), UA_MANDATORY),
+	UVERBS_ATTR_PTR_IN(CXI_IB_ATTR_CT_ALLOC_LNI_HANDLE,
+			   UVERBS_ATTR_TYPE(u32), UA_OPTIONAL),
+	UVERBS_ATTR_IDR(CXI_IB_ATTR_CT_ALLOC_HANDLE,
+			CXI_IB_OBJECT_CT,
+			UVERBS_ACCESS_NEW,
+			UA_MANDATORY),
+	UVERBS_ATTR_PTR_OUT(CXI_IB_ATTR_CT_ALLOC_CTN,
+			    UVERBS_ATTR_TYPE(u32),
+			    UA_MANDATORY),
+	UVERBS_ATTR_PTR_OUT(CXI_IB_ATTR_CT_ALLOC_WB_MMAP_OFFSET,
+			    UVERBS_ATTR_TYPE(u64),
+			    UA_MANDATORY),
+	UVERBS_ATTR_PTR_OUT(CXI_IB_ATTR_CT_ALLOC_WB_MMAP_LENGTH,
+			    UVERBS_ATTR_TYPE(u64),
+			    UA_MANDATORY),
+	UVERBS_ATTR_PTR_OUT(CXI_IB_ATTR_CT_ALLOC_DB_MMAP_OFFSET,
+			    UVERBS_ATTR_TYPE(u64),
+			    UA_MANDATORY),
+	UVERBS_ATTR_PTR_OUT(CXI_IB_ATTR_CT_ALLOC_DB_MMAP_LENGTH,
+			    UVERBS_ATTR_TYPE(u64),
+			    UA_MANDATORY));
+
+DECLARE_UVERBS_NAMED_METHOD_DESTROY(
+	CXI_IB_METHOD_CT_DESTROY,
+	UVERBS_ATTR_IDR(CXI_IB_ATTR_CT_DESTROY_HANDLE,
+			CXI_IB_OBJECT_CT,
+			UVERBS_ACCESS_DESTROY,
+			UA_MANDATORY));
+
+/* -----------------------------------------------------------------------
+ * Object definition
+ * -----------------------------------------------------------------------
+ */
+DECLARE_UVERBS_NAMED_OBJECT(
+	CXI_IB_OBJECT_CT,
+	UVERBS_TYPE_ALLOC_IDR(cxiverbs_ct_cleanup),
+	&UVERBS_METHOD(CXI_IB_METHOD_CT_ALLOC),
+	&UVERBS_METHOD(CXI_IB_METHOD_CT_DESTROY));
+
+struct cxiverbs_cmdq {
+	refcount_t refs;
+	struct cxiverbs_lni *lni;
+	struct cxi_cq *cq;
+	struct cxiverbs_mmap_entry *commands;
+	struct cxiverbs_mmap_entry *doorbell;
+};
+
+static void cxiverbs_cmdq_release(void *owner)
+{
+	struct cxiverbs_cmdq *cmdq = owner;
+
+	if (!refcount_dec_and_test(&cmdq->refs))
+		return;
+	if (cmdq->cq)
+		cmdq->lni->ctx->cadev->cq_free(cmdq->cq);
+	cxiverbs_lni_put(cmdq->lni);
+	kfree(cmdq);
+}
+
+static int cxiverbs_cmdq_cleanup(struct ib_uobject *object,
+				enum rdma_remove_reason reason,
+				struct uverbs_attr_bundle *attrs)
+{
+	struct cxiverbs_cmdq *cmdq = object->object;
+
+	if (cmdq->doorbell)
+		rdma_user_mmap_entry_remove(&cmdq->doorbell->rdma_entry);
+	if (cmdq->commands)
+		rdma_user_mmap_entry_remove(&cmdq->commands->rdma_entry);
+	cxiverbs_cmdq_release(cmdq);
+	return 0;
+}
+
+static int UVERBS_HANDLER(CXI_IB_METHOD_CMDQ_ALLOC)(
+		struct uverbs_attr_bundle *attrs)
+{
+	struct ib_uobject *object =
+		uverbs_attr_get_uobject(attrs, CXI_IB_ATTR_CMDQ_HANDLE);
+	struct cxi_cq_alloc_opts opts = { .flags = CXI_CQ_USER };
+	struct cxi_ibv_cmdq_info info = {};
+	struct cxiverbs_ucontext *ctx;
+	struct cxiverbs_cmdq *cmdq;
+	struct page *pages;
+	phys_addr_t doorbell;
+	size_t commands_size;
+	size_t doorbell_size;
+	u64 offset;
+	u32 handle;
+	u32 flags = 0;
+	int ret;
+
+	ctx = cxiverbs_get_ucontext(attrs);
+	if (IS_ERR(ctx))
+		return PTR_ERR(ctx);
+	if (!ctx->cadev->cq_alloc || !ctx->cadev->cq_free ||
+	    !ctx->cadev->cq_user_info)
+		return -EOPNOTSUPP;
+	ret = uverbs_copy_from(&handle, attrs, CXI_IB_ATTR_CMDQ_LNI);
+	if (ret)
+		return ret;
+	ret = uverbs_copy_from(&opts.count, attrs, CXI_IB_ATTR_CMDQ_COUNT);
+	if (ret)
+		return ret;
+	if (!opts.count || opts.count > CXI_MAX_CQ_COUNT)
+		return -EINVAL;
+	if (uverbs_attr_is_valid(attrs, CXI_IB_ATTR_CMDQ_FLAGS)) {
+		ret = uverbs_copy_from(&flags, attrs, CXI_IB_ATTR_CMDQ_FLAGS);
+		if (ret)
+			return ret;
+	}
+	if (flags & ~CXI_IB_CMDQ_TRIGGER)
+		return -EINVAL;
+	if (flags & CXI_IB_CMDQ_TRIGGER)
+		opts.flags |= CXI_CQ_IS_TX | CXI_CQ_TX_WITH_TRIG_CMDS;
+	cmdq = kzalloc_obj(*cmdq, GFP_KERNEL);
+	if (!cmdq)
+		return -ENOMEM;
+	cmdq->lni = cxiverbs_lni_get(ctx, handle);
+	if (IS_ERR(cmdq->lni)) {
+		ret = PTR_ERR(cmdq->lni);
+		kfree(cmdq);
+		return ret;
+	}
+	refcount_set(&cmdq->refs, 1);
+	cmdq->cq = ctx->cadev->cq_alloc(cmdq->lni->lni, NULL, &opts,
+				       NUMA_NO_NODE);
+	if (IS_ERR(cmdq->cq)) {
+		ret = PTR_ERR(cmdq->cq);
+		cmdq->cq = NULL;
+		goto release;
+	}
+	ret = ctx->cadev->cq_user_info(cmdq->cq, &commands_size, &pages,
+				      &doorbell, &doorbell_size);
+	if (ret)
+		goto release;
+	info.count = cmdq->cq->size;
+	info.index = cmdq->cq->idx;
+	info.commands_length = commands_size;
+	info.doorbell_length = doorbell_size;
+	refcount_inc(&cmdq->refs);
+	cmdq->commands = cxiverbs_mmap_entry_alloc_owned(&ctx->ibucontext,
+			false, 0, pages, commands_size, &offset, cmdq,
+			cxiverbs_cmdq_release);
+	if (IS_ERR(cmdq->commands)) {
+		ret = PTR_ERR(cmdq->commands);
+		cmdq->commands = NULL;
+		refcount_dec(&cmdq->refs);
+		goto release;
+	}
+	info.commands_offset = offset;
+	refcount_inc(&cmdq->refs);
+	cmdq->doorbell = cxiverbs_mmap_entry_alloc_owned(&ctx->ibucontext,
+			true, doorbell, NULL, doorbell_size, &offset, cmdq,
+			cxiverbs_cmdq_release);
+	if (IS_ERR(cmdq->doorbell)) {
+		ret = PTR_ERR(cmdq->doorbell);
+		cmdq->doorbell = NULL;
+		refcount_dec(&cmdq->refs);
+		rdma_user_mmap_entry_remove(&cmdq->commands->rdma_entry);
+		goto release;
+	}
+	info.doorbell_offset = offset;
+	object->object = cmdq;
+	uverbs_finalize_uobj_create(attrs, CXI_IB_ATTR_CMDQ_HANDLE);
+	return uverbs_copy_to(attrs, CXI_IB_ATTR_CMDQ_INFO,
+			     &info, sizeof(info));
+release:
+	cxiverbs_cmdq_release(cmdq);
+	return ret;
+}
+
+DECLARE_UVERBS_NAMED_METHOD(CXI_IB_METHOD_CMDQ_ALLOC,
+	UVERBS_ATTR_IDR(CXI_IB_ATTR_CMDQ_HANDLE, CXI_IB_OBJECT_CMDQ,
+		       UVERBS_ACCESS_NEW, UA_MANDATORY),
+	UVERBS_ATTR_PTR_IN(CXI_IB_ATTR_CMDQ_LNI,
+			   UVERBS_ATTR_TYPE(u32), UA_MANDATORY),
+	UVERBS_ATTR_PTR_IN(CXI_IB_ATTR_CMDQ_COUNT,
+			   UVERBS_ATTR_TYPE(u32), UA_MANDATORY),
+	UVERBS_ATTR_PTR_OUT(CXI_IB_ATTR_CMDQ_INFO,
+			    UVERBS_ATTR_TYPE(struct cxi_ibv_cmdq_info),
+			    UA_MANDATORY),
+	UVERBS_ATTR_PTR_IN(CXI_IB_ATTR_CMDQ_FLAGS, UVERBS_ATTR_TYPE(u32)));
+
+DECLARE_UVERBS_NAMED_METHOD_DESTROY(CXI_IB_METHOD_CMDQ_DESTROY,
+	UVERBS_ATTR_IDR(CXI_IB_ATTR_CMDQ_HANDLE, CXI_IB_OBJECT_CMDQ,
+		       UVERBS_ACCESS_DESTROY, UA_MANDATORY));
+
+DECLARE_UVERBS_NAMED_OBJECT(CXI_IB_OBJECT_CMDQ,
+	UVERBS_TYPE_ALLOC_IDR(cxiverbs_cmdq_cleanup),
+	&UVERBS_METHOD(CXI_IB_METHOD_CMDQ_ALLOC),
+	&UVERBS_METHOD(CXI_IB_METHOD_CMDQ_DESTROY));
+
+static int cxiverbs_copy_job_tc_limit(struct uverbs_attr_bundle *attrs,
+				    int max_attr, int res_attr,
+				    struct cxi_limits *limit)
+{
+	int ret;
+
+	if (uverbs_attr_is_valid(attrs, max_attr)) {
+		ret = uverbs_copy_from(&limit->max, attrs, max_attr);
+		if (ret)
+			return ret;
+	}
+	if (uverbs_attr_is_valid(attrs, res_attr)) {
+		ret = uverbs_copy_from(&limit->res, attrs, res_attr);
+		if (ret)
+			return ret;
+	}
+
+	return limit->max < limit->res ? -EINVAL : 0;
+}
+
+static u32 cxiverbs_tc_key(u32 tc, u32 tc_type)
+{
+	return (tc_type << 16) | tc;
+}
+
+static int UVERBS_HANDLER(CXI_IB_METHOD_JOB_TC_BIND)(
+		struct uverbs_attr_bundle *attrs)
+{
+	struct ib_ucontext *ibctx = ib_uverbs_get_ucontext(attrs);
+	struct cxiverbs_ucontext *uctx;
+	struct cxiverbs_dev *dev;
+	struct cxiverbs_job_entry *job;
+	struct cxiverbs_job_tc_binding *binding = NULL;
+	struct cxiverbs_tc_entry *tc_entry = NULL;
+	struct cxi_rsrc_limits limits = {};
+	u32 flags = 0;
+	u32 tc_key;
+	u32 job_id;
+	u32 svc_id;
+	u32 bound_svc_id;
+	u32 tc;
+	u32 tc_type = CXI_TC_TYPE_DEFAULT;
+	bool create_tc;
+	bool created_tc = false;
+	int ret;
+
+	if (IS_ERR(ibctx))
+		return PTR_ERR(ibctx);
+	if (!capable(CAP_NET_ADMIN))
+		return -EPERM;
+
+	uctx = to_cxiverbs_ucontext(ibctx);
+	dev = to_cxiverbs_dev(ibctx->device);
+	if (!uctx->cadev->job_tc_bind)
+		return -EOPNOTSUPP;
+
+	ret = uverbs_get_flags32(&flags, attrs,
+				  CXI_IB_ATTR_JOB_TC_BIND_FLAGS,
+				  CXI_IB_JOB_TC_F_CREATE);
+	if (ret)
+		return ret;
+
+	ret = uverbs_copy_from(&job_id, attrs, CXI_IB_ATTR_JOB_TC_BIND_JOB_ID);
+	if (ret)
+		return ret;
+	ret = uverbs_copy_from(&tc, attrs, CXI_IB_ATTR_JOB_TC_BIND_TC);
+	if (ret)
+		return ret;
+
+	svc_id = 0;
+	if (uverbs_attr_is_valid(attrs, CXI_IB_ATTR_JOB_TC_BIND_SVC_ID)) {
+		ret = uverbs_copy_from(&svc_id, attrs,
+				       CXI_IB_ATTR_JOB_TC_BIND_SVC_ID);
+		if (ret)
+			return ret;
+	}
+	if (uverbs_attr_is_valid(attrs, CXI_IB_ATTR_JOB_TC_BIND_TC_TYPE)) {
+		ret = uverbs_copy_from(&tc_type, attrs,
+				       CXI_IB_ATTR_JOB_TC_BIND_TC_TYPE);
+		if (ret)
+			return ret;
+	}
+
+	ret = cxiverbs_copy_job_tc_limit(attrs,
+			CXI_IB_ATTR_JOB_TC_BIND_PTES_MAX,
+			CXI_IB_ATTR_JOB_TC_BIND_PTES_RES,
+			&limits.ptes);
+	if (ret)
+		return ret;
+	ret = cxiverbs_copy_job_tc_limit(attrs,
+			CXI_IB_ATTR_JOB_TC_BIND_TXQS_MAX,
+			CXI_IB_ATTR_JOB_TC_BIND_TXQS_RES,
+			&limits.txqs);
+	if (ret)
+		return ret;
+	ret = cxiverbs_copy_job_tc_limit(attrs,
+			CXI_IB_ATTR_JOB_TC_BIND_TGQS_MAX,
+			CXI_IB_ATTR_JOB_TC_BIND_TGQS_RES,
+			&limits.tgqs);
+	if (ret)
+		return ret;
+	ret = cxiverbs_copy_job_tc_limit(attrs,
+			CXI_IB_ATTR_JOB_TC_BIND_EQS_MAX,
+			CXI_IB_ATTR_JOB_TC_BIND_EQS_RES,
+			&limits.eqs);
+	if (ret)
+		return ret;
+	ret = cxiverbs_copy_job_tc_limit(attrs,
+			CXI_IB_ATTR_JOB_TC_BIND_CTS_MAX,
+			CXI_IB_ATTR_JOB_TC_BIND_CTS_RES,
+			&limits.cts);
+	if (ret)
+		return ret;
+	ret = cxiverbs_copy_job_tc_limit(attrs,
+			CXI_IB_ATTR_JOB_TC_BIND_LES_MAX,
+			CXI_IB_ATTR_JOB_TC_BIND_LES_RES,
+			&limits.les);
+	if (ret)
+		return ret;
+	ret = cxiverbs_copy_job_tc_limit(attrs,
+			CXI_IB_ATTR_JOB_TC_BIND_TLES_MAX,
+			CXI_IB_ATTR_JOB_TC_BIND_TLES_RES,
+			&limits.tles);
+	if (ret)
+		return ret;
+	ret = cxiverbs_copy_job_tc_limit(attrs,
+			CXI_IB_ATTR_JOB_TC_BIND_ACS_MAX,
+			CXI_IB_ATTR_JOB_TC_BIND_ACS_RES,
+			&limits.acs);
+	if (ret)
+		return ret;
+
+	mutex_lock(&dev->jobs_lock);
+	job = uctx->job;
+	if (!job || job->job_id != job_id) {
+		ret = -EINVAL;
+		goto out_unlock;
+	}
+
+	create_tc = flags & CXI_IB_JOB_TC_F_CREATE;
+	tc_key = cxiverbs_tc_key(tc, tc_type);
+	tc_entry = xa_load(&dev->tcs, tc_key);
+	if (create_tc) {
+		if (tc_entry) {
+			ret = -EEXIST;
+			goto out_unlock;
+		}
+	}
+
+	if (tc_entry) {
+		list_for_each_entry(binding, &tc_entry->job_list, entry) {
+			if (binding->job == job) {
+				ret = -EEXIST;
+				goto out_unlock;
+			}
+		}
+		binding = NULL;
+	}
+
+	binding = kzalloc_obj(*binding);
+	if (!binding) {
+		ret = -ENOMEM;
+		goto out_unlock;
+	}
+
+	ret = uctx->cadev->job_tc_bind(uctx->cadev->cxi_dev, job_id, svc_id,
+					 tc, tc_type, create_tc, &limits,
+					 &bound_svc_id, &created_tc);
+	if (ret)
+		goto err_free_binding;
+
+	if (!tc_entry) {
+		tc_entry = kzalloc_obj(*tc_entry);
+		if (!tc_entry) {
+			ret = -ENOMEM;
+			goto err_clear_created_tc;
+		}
+		tc_entry->key = tc_key;
+		tc_entry->tc = tc;
+		tc_entry->tc_type = tc_type;
+		tc_entry->created = created_tc;
+		INIT_LIST_HEAD(&tc_entry->job_list);
+		ret = xa_insert(&dev->tcs, tc_key, tc_entry, GFP_KERNEL);
+		if (ret) {
+			kfree(tc_entry);
+			tc_entry = NULL;
+			goto err_clear_created_tc;
+		}
+	}
+
+	binding->job = job;
+	binding->svc_id = bound_svc_id;
+	list_add_tail(&binding->entry, &tc_entry->job_list);
+	mutex_unlock(&dev->jobs_lock);
+
+	return uverbs_copy_to(attrs, CXI_IB_ATTR_JOB_TC_BIND_RESP_SVC_ID,
+				    &bound_svc_id, sizeof(bound_svc_id));
+err_clear_created_tc:
+	if (created_tc && uctx->cadev->tc_clear)
+		uctx->cadev->tc_clear(uctx->cadev->cxi_dev, tc, tc_type);
+	if (uctx->cadev->svc_destroy)
+		uctx->cadev->svc_destroy(uctx->cadev->cxi_dev, bound_svc_id);
+err_free_binding:
+	kfree(binding);
+out_unlock:
+	mutex_unlock(&dev->jobs_lock);
+	return ret;
+}
+
+DECLARE_UVERBS_NAMED_METHOD(
+	CXI_IB_METHOD_JOB_TC_BIND,
+	UVERBS_ATTR_FLAGS_IN(CXI_IB_ATTR_JOB_TC_BIND_FLAGS, u32, UA_OPTIONAL),
+	UVERBS_ATTR_PTR_IN(CXI_IB_ATTR_JOB_TC_BIND_JOB_ID,
+			    UVERBS_ATTR_TYPE(u32), UA_MANDATORY),
+	UVERBS_ATTR_PTR_IN(CXI_IB_ATTR_JOB_TC_BIND_TC,
+			    UVERBS_ATTR_TYPE(u32), UA_MANDATORY),
+	UVERBS_ATTR_PTR_IN(CXI_IB_ATTR_JOB_TC_BIND_TC_TYPE,
+			    UVERBS_ATTR_TYPE(u32), UA_OPTIONAL),
+	UVERBS_ATTR_PTR_IN(CXI_IB_ATTR_JOB_TC_BIND_SVC_ID,
+			    UVERBS_ATTR_TYPE(u32), UA_OPTIONAL),
+	UVERBS_ATTR_PTR_IN(CXI_IB_ATTR_JOB_TC_BIND_PTES_MAX,
+			    UVERBS_ATTR_TYPE(u16), UA_OPTIONAL),
+	UVERBS_ATTR_PTR_IN(CXI_IB_ATTR_JOB_TC_BIND_PTES_RES,
+			    UVERBS_ATTR_TYPE(u16), UA_OPTIONAL),
+	UVERBS_ATTR_PTR_IN(CXI_IB_ATTR_JOB_TC_BIND_TXQS_MAX,
+			    UVERBS_ATTR_TYPE(u16), UA_OPTIONAL),
+	UVERBS_ATTR_PTR_IN(CXI_IB_ATTR_JOB_TC_BIND_TXQS_RES,
+			    UVERBS_ATTR_TYPE(u16), UA_OPTIONAL),
+	UVERBS_ATTR_PTR_IN(CXI_IB_ATTR_JOB_TC_BIND_TGQS_MAX,
+			    UVERBS_ATTR_TYPE(u16), UA_OPTIONAL),
+	UVERBS_ATTR_PTR_IN(CXI_IB_ATTR_JOB_TC_BIND_TGQS_RES,
+			    UVERBS_ATTR_TYPE(u16), UA_OPTIONAL),
+	UVERBS_ATTR_PTR_IN(CXI_IB_ATTR_JOB_TC_BIND_EQS_MAX,
+			    UVERBS_ATTR_TYPE(u16), UA_OPTIONAL),
+	UVERBS_ATTR_PTR_IN(CXI_IB_ATTR_JOB_TC_BIND_EQS_RES,
+			    UVERBS_ATTR_TYPE(u16), UA_OPTIONAL),
+	UVERBS_ATTR_PTR_IN(CXI_IB_ATTR_JOB_TC_BIND_CTS_MAX,
+			    UVERBS_ATTR_TYPE(u16), UA_OPTIONAL),
+	UVERBS_ATTR_PTR_IN(CXI_IB_ATTR_JOB_TC_BIND_CTS_RES,
+			    UVERBS_ATTR_TYPE(u16), UA_OPTIONAL),
+	UVERBS_ATTR_PTR_IN(CXI_IB_ATTR_JOB_TC_BIND_LES_MAX,
+			    UVERBS_ATTR_TYPE(u16), UA_OPTIONAL),
+	UVERBS_ATTR_PTR_IN(CXI_IB_ATTR_JOB_TC_BIND_LES_RES,
+			    UVERBS_ATTR_TYPE(u16), UA_OPTIONAL),
+	UVERBS_ATTR_PTR_IN(CXI_IB_ATTR_JOB_TC_BIND_TLES_MAX,
+			    UVERBS_ATTR_TYPE(u16), UA_OPTIONAL),
+	UVERBS_ATTR_PTR_IN(CXI_IB_ATTR_JOB_TC_BIND_TLES_RES,
+			    UVERBS_ATTR_TYPE(u16), UA_OPTIONAL),
+	UVERBS_ATTR_PTR_IN(CXI_IB_ATTR_JOB_TC_BIND_ACS_MAX,
+			    UVERBS_ATTR_TYPE(u16), UA_OPTIONAL),
+	UVERBS_ATTR_PTR_IN(CXI_IB_ATTR_JOB_TC_BIND_ACS_RES,
+			    UVERBS_ATTR_TYPE(u16), UA_OPTIONAL),
+	UVERBS_ATTR_PTR_OUT(CXI_IB_ATTR_JOB_TC_BIND_RESP_SVC_ID,
+			     UVERBS_ATTR_TYPE(u32), UA_MANDATORY));
+
+static int UVERBS_HANDLER(CXI_IB_METHOD_JOB_TC_CLEAR)(
+		struct uverbs_attr_bundle *attrs)
+{
+	struct ib_ucontext *ibctx = ib_uverbs_get_ucontext(attrs);
+	struct cxiverbs_ucontext *uctx;
+	struct cxiverbs_dev *dev;
+	struct cxiverbs_tc_entry *tc_entry;
+	u32 tc;
+	u32 tc_type = CXI_TC_TYPE_DEFAULT;
+	u32 tc_key;
+	int ret;
+
+	if (IS_ERR(ibctx))
+		return PTR_ERR(ibctx);
+	if (!capable(CAP_NET_ADMIN))
+		return -EPERM;
+
+	uctx = to_cxiverbs_ucontext(ibctx);
+	dev = to_cxiverbs_dev(ibctx->device);
+	if (!uctx->cadev->tc_clear)
+		return -EOPNOTSUPP;
+
+	ret = uverbs_copy_from(&tc, attrs, CXI_IB_ATTR_JOB_TC_CLEAR_TC);
+	if (ret)
+		return ret;
+	if (uverbs_attr_is_valid(attrs, CXI_IB_ATTR_JOB_TC_CLEAR_TC_TYPE)) {
+		ret = uverbs_copy_from(&tc_type, attrs,
+				       CXI_IB_ATTR_JOB_TC_CLEAR_TC_TYPE);
+		if (ret)
+			return ret;
+	}
+
+	if (tc >= CXI_TC_MAX || tc_type >= CXI_TC_TYPE_MAX)
+		return -EINVAL;
+
+	tc_key = cxiverbs_tc_key(tc, tc_type);
+	mutex_lock(&dev->jobs_lock);
+	tc_entry = xa_load(&dev->tcs, tc_key);
+	if (!tc_entry) {
+		ret = -ENOENT;
+		goto out_unlock;
+	}
+	if (!list_empty(&tc_entry->job_list)) {
+		ret = -EBUSY;
+		goto out_unlock;
+	}
+
+	ret = tc_entry->created ?
+		uctx->cadev->tc_clear(uctx->cadev->cxi_dev, tc, tc_type) : 0;
+	if (ret)
+		goto out_unlock;
+	xa_erase(&dev->tcs, tc_key);
+	kfree(tc_entry);
+
+out_unlock:
+	mutex_unlock(&dev->jobs_lock);
+	return ret;
+}
+
+DECLARE_UVERBS_NAMED_METHOD(
+	CXI_IB_METHOD_JOB_TC_CLEAR,
+	UVERBS_ATTR_PTR_IN(CXI_IB_ATTR_JOB_TC_CLEAR_TC,
+			    UVERBS_ATTR_TYPE(u32), UA_MANDATORY),
+	UVERBS_ATTR_PTR_IN(CXI_IB_ATTR_JOB_TC_CLEAR_TC_TYPE,
+			    UVERBS_ATTR_TYPE(u32), UA_OPTIONAL));
+
+static int UVERBS_HANDLER(CXI_IB_METHOD_JOB_TC_QUERY)(
+		struct uverbs_attr_bundle *attrs)
+{
+	struct ib_ucontext *ibctx = ib_uverbs_get_ucontext(attrs);
+	struct cxiverbs_ucontext *uctx;
+	struct cxiverbs_dev *dev;
+	struct cxi_tc_info *tc_info;
+	struct cxi_ibv_tc_entry *entries;
+	u32 max_entries = 0;
+	u32 count = 0;
+	u32 copied;
+	int ret;
+	u32 i;
+
+	if (IS_ERR(ibctx))
+		return PTR_ERR(ibctx);
+
+	uctx = to_cxiverbs_ucontext(ibctx);
+	dev = to_cxiverbs_dev(ibctx->device);
+	if (!uctx->cadev->tc_list)
+		return -EOPNOTSUPP;
+
+	if (uverbs_attr_is_valid(attrs, CXI_IB_ATTR_JOB_TC_QUERY_MAX_ENTRIES)) {
+		ret = uverbs_copy_from(&max_entries, attrs,
+				       CXI_IB_ATTR_JOB_TC_QUERY_MAX_ENTRIES);
+		if (ret)
+			return ret;
+	}
+
+	if (!max_entries || max_entries > CXI_TC_MAX * CXI_TC_TYPE_MAX)
+		max_entries = CXI_TC_MAX * CXI_TC_TYPE_MAX;
+
+	tc_info = kcalloc(max_entries, sizeof(*tc_info), GFP_KERNEL);
+	if (!tc_info)
+		return -ENOMEM;
+	entries = kcalloc(max_entries, sizeof(*entries), GFP_KERNEL);
+	if (!entries) {
+		kfree(tc_info);
+		return -ENOMEM;
+	}
+
+	ret = uctx->cadev->tc_list(uctx->cadev->cxi_dev, tc_info, max_entries,
+				    &count);
+	if (ret)
+		goto out_free;
+
+	copied = min(count, max_entries);
+	mutex_lock(&dev->jobs_lock);
+	for (i = 0; i < copied; i++) {
+		struct cxiverbs_tc_entry *tracked;
+
+		entries[i].tc = tc_info[i].tc;
+		entries[i].tc_type = tc_info[i].tc_type;
+		entries[i].res_req_dscp = tc_info[i].res_req_dscp;
+		entries[i].unres_req_dscp = tc_info[i].unres_req_dscp;
+		entries[i].res_rsp_dscp = tc_info[i].res_rsp_dscp;
+		entries[i].unres_rsp_dscp = tc_info[i].unres_rsp_dscp;
+		entries[i].req_pcp = tc_info[i].req_pcp;
+		entries[i].rsp_pcp = tc_info[i].rsp_pcp;
+		entries[i].ocuset = tc_info[i].ocuset;
+		entries[i].cq_tc = tc_info[i].cq_tc;
+		entries[i].req_bc = tc_info[i].req_bc;
+
+		tracked = xa_load(&dev->tcs,
+				  cxiverbs_tc_key(entries[i].tc, entries[i].tc_type));
+		if (tracked && tracked->created)
+			entries[i].flags |= CXI_IBV_TC_ENTRY_F_CREATED_BY_RDMA;
+	}
+	mutex_unlock(&dev->jobs_lock);
+
+	ret = uverbs_copy_to(attrs, CXI_IB_ATTR_JOB_TC_QUERY_RESP_COUNT,
+			     &count, sizeof(count));
+	if (ret)
+		goto out_free;
+	ret = uverbs_copy_to(attrs, CXI_IB_ATTR_JOB_TC_QUERY_RESP_ENTRIES,
+			     entries, copied * sizeof(*entries));
+
+out_free:
+	kfree(entries);
+	kfree(tc_info);
+	return ret;
+}
+
+DECLARE_UVERBS_NAMED_METHOD(
+	CXI_IB_METHOD_JOB_TC_QUERY,
+	UVERBS_ATTR_PTR_IN(CXI_IB_ATTR_JOB_TC_QUERY_MAX_ENTRIES,
+			    UVERBS_ATTR_TYPE(u32), UA_OPTIONAL),
+	UVERBS_ATTR_PTR_OUT(CXI_IB_ATTR_JOB_TC_QUERY_RESP_COUNT,
+			     UVERBS_ATTR_TYPE(u32), UA_MANDATORY),
+	UVERBS_ATTR_PTR_OUT(CXI_IB_ATTR_JOB_TC_QUERY_RESP_ENTRIES,
+			     UVERBS_ATTR_MIN_SIZE(0), UA_MANDATORY));
+
+DECLARE_UVERBS_GLOBAL_METHODS(CXI_IB_OBJECT_JOB_TC,
+	&UVERBS_METHOD(CXI_IB_METHOD_JOB_TC_BIND),
+	&UVERBS_METHOD(CXI_IB_METHOD_JOB_TC_CLEAR),
+	&UVERBS_METHOD(CXI_IB_METHOD_JOB_TC_QUERY));
+
+/* -----------------------------------------------------------------------
+ * Driver uapi_definition table — assigned to ib_dev.driver_def in probe.
+ * -----------------------------------------------------------------------
+ */
+const struct uapi_definition cxiverbs_defs[] = {
+	UAPI_DEF_CHAIN_OBJ_TREE_NAMED(CXI_IB_OBJECT_WAIT),
+	UAPI_DEF_CHAIN_OBJ_TREE_NAMED(CXI_IB_OBJECT_CSR),
+	UAPI_DEF_CHAIN_OBJ_TREE_NAMED(CXI_IB_OBJECT_EQ),
+	UAPI_DEF_CHAIN_OBJ_TREE_NAMED(CXI_IB_OBJECT_CP),
+	UAPI_DEF_CHAIN_OBJ_TREE_NAMED(CXI_IB_OBJECT_SVC),
+	UAPI_DEF_CHAIN_OBJ_TREE_NAMED(UVERBS_OBJECT_PD),
+	UAPI_DEF_CHAIN_OBJ_TREE_NAMED(CXI_IB_OBJECT_LNI),
+	UAPI_DEF_CHAIN_OBJ_TREE_NAMED(CXI_IB_OBJECT_CT),
+	UAPI_DEF_CHAIN_OBJ_TREE_NAMED(CXI_IB_OBJECT_CMDQ),
+	UAPI_DEF_CHAIN_OBJ_TREE_NAMED(CXI_IB_OBJECT_JOB_TC),
+	{},
+};

-- 
2.56.0

[PATCH 10/11] RDMA/cxi: read the NIC telemetry through counters

[Doug Ledford]

Implement read_counters_ex() so that user space can read the NIC
telemetry through the COUNTERS object.  An item number is the index of
the telemetry item in the core driver's table, so the caller names what
it wants without the driver describing thousands of entries in sysfs.
Items the device does not define read as zero with a zero timestamp.
The values come from the core driver's cxi_telem_read(), which reads
the device BAR directly.

Known limitation, deferred to V2: this patch carries SR-IOV
virtual-function code (VF-to-PF mailbox requests, mailbox opcodes or VF
resource handling) from an older, incomplete version of the Cassini
SR-IOV support. V2 of this series will include the updated SR-IOV
implementation, so issues in the VF paths should be ignored for now.

Signed-off-by: Doug Ledford <doug.ledford@hpe.com>
---
 drivers/infiniband/hw/cxi/cxiverbs_counters.c | 38 +++++++++++++++++++++++++++
 1 file changed, 38 insertions(+)

diff --git a/drivers/infiniband/hw/cxi/cxiverbs_counters.c b/drivers/infiniband/hw/cxi/cxiverbs_counters.c
new file mode 100644
index 000000000000..2ece6568ea7d
--- /dev/null
+++ b/drivers/infiniband/hw/cxi/cxiverbs_counters.c
@@ -0,0 +1,38 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * CXI RDMA telemetry counters, read through the uverbs counters object.
+ * Counter numbers are Cassini telemetry item IDs; user space names them.
+ * Copyright 2026 Hewlett Packard Enterprise Development LP
+ */
+
+#include <rdma/ib_verbs.h>
+
+#include "cxiverbs.h"
+
+int cxiverbs_create_counters(struct ib_counters *counters,
+			     struct uverbs_attr_bundle *attrs)
+{
+	return 0;
+}
+
+int cxiverbs_destroy_counters(struct ib_counters *counters)
+{
+	return 0;
+}
+
+/*
+ * Counters are not attached to anything, so the core's attach check does
+ * not apply. With no items, counter N is telemetry item N.
+ */
+int cxiverbs_read_counters_ex(struct ib_counters *counters,
+			      struct ib_counters_read_attr *attr,
+			      struct uverbs_attr_bundle *attrs)
+{
+	struct cxiverbs_dev *dev = to_cxiverbs_dev(counters->device);
+
+	if (!dev->cdev->is_physfn)
+		return -EOPNOTSUPP;
+
+	return cxi_telem_read(dev->cdev, attr->items, attr->counters_buff,
+			      attr->timestamps, attr->ncounters);
+}

-- 
2.56.0

[PATCH 11/11] RDMA/cxi: wire the driver into the build

[Doug Ledford]

Add the Kconfig and Makefile hookup for the CXI RDMA driver and its
MAINTAINERS entry.  INFINIBAND_CXIVERBS depends on HPE_SS1, since it
is an auxiliary client of the core driver.

Signed-off-by: Doug Ledford <doug.ledford@hpe.com>
---
 MAINTAINERS                        |  9 ++++++++-
 drivers/infiniband/Kconfig         |  1 +
 drivers/infiniband/hw/Makefile     |  1 +
 drivers/infiniband/hw/cxi/Kconfig  | 11 +++++++++++
 drivers/infiniband/hw/cxi/Makefile |  5 +++++
 5 files changed, 26 insertions(+), 1 deletion(-)

diff --git a/MAINTAINERS b/MAINTAINERS
index 094263bfe4ad..1f0e3447377c 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -11546,8 +11546,15 @@ HEWLETT PACKARD ENTERPRISE SLINGSHOT NIC DRIVER
 M:	Slingshot Open Source Team <slingshot-os-team@groups.ext.hpe.com>
 L:	netdev@vger.kernel.org
 S:	Supported
-F:	include/linux/hpe
 F:	drivers/net/ethernet/hpe
+F:	include/linux/hpe
+
+HEWLETT PACKARD ENTERPRISE CXIVERBS DRIVER
+M:	Slingshot Open Source Team <slingshot-os-team@groups.ext.hpe.com>
+L:	linux-rdma@vger.kernel.org
+S:	Supported
+F:	drivers/infiniband/hw/cxi
+F:	include/uapi/rdma/cxiverbs-abi.h
 
 HEWLETT-PACKARD SMART ARRAY RAID DRIVER (hpsa)
 M:	Don Brace <don.brace@microchip.com>
diff --git a/drivers/infiniband/Kconfig b/drivers/infiniband/Kconfig
index 086195758a8a..d0c8040f44c3 100644
--- a/drivers/infiniband/Kconfig
+++ b/drivers/infiniband/Kconfig
@@ -86,6 +86,7 @@ if !UML
 source "drivers/infiniband/hw/bnxt_re/Kconfig"
 source "drivers/infiniband/hw/bng_re/Kconfig"
 source "drivers/infiniband/hw/cxgb4/Kconfig"
+source "drivers/infiniband/hw/cxi/Kconfig"
 source "drivers/infiniband/hw/efa/Kconfig"
 source "drivers/infiniband/hw/erdma/Kconfig"
 source "drivers/infiniband/hw/hfi1/Kconfig"
diff --git a/drivers/infiniband/hw/Makefile b/drivers/infiniband/hw/Makefile
index c42b22ac3303..1ccd2c1be330 100644
--- a/drivers/infiniband/hw/Makefile
+++ b/drivers/infiniband/hw/Makefile
@@ -16,3 +16,4 @@ obj-$(CONFIG_INFINIBAND_BNXT_RE)	+= bnxt_re/
 obj-$(CONFIG_INFINIBAND_BNG_RE)		+= bng_re/
 obj-$(CONFIG_INFINIBAND_ERDMA)		+= erdma/
 obj-$(CONFIG_INFINIBAND_IONIC)		+= ionic/
+obj-$(CONFIG_INFINIBAND_CXIVERBS)		+= cxi/
diff --git a/drivers/infiniband/hw/cxi/Kconfig b/drivers/infiniband/hw/cxi/Kconfig
new file mode 100644
index 000000000000..5efa83de744a
--- /dev/null
+++ b/drivers/infiniband/hw/cxi/Kconfig
@@ -0,0 +1,11 @@
+# SPDX-License-Identifier: GPL-2.0-only
+config INFINIBAND_CXIVERBS
+	tristate "Hewlett Packard Network Adapter support"
+	depends on NETDEVICES && ETHERNET && PCI && HPE_SS1
+	depends on INFINIBAND_USER_ACCESS
+	select AUXILIARY_BUS
+	help
+	  This driver provides low-level RDMA support for HPE's Cassini
+	  Network Adapter. CXI supports RDMA features that can be used
+	  for workloads (e.g. DPDK, MPI etc) that uses RDMA verbs to directly
+	  access hardware from user-mode processes.
diff --git a/drivers/infiniband/hw/cxi/Makefile b/drivers/infiniband/hw/cxi/Makefile
new file mode 100644
index 000000000000..b096704947ff
--- /dev/null
+++ b/drivers/infiniband/hw/cxi/Makefile
@@ -0,0 +1,5 @@
+# SPDX-License-Identifier: GPL-2.0-only
+obj-$(CONFIG_INFINIBAND_CXIVERBS) += cxiverbs.o
+
+cxiverbs-y := cxiverbs_drv.o cxiverbs_main.o cxiverbs_mr.o cxiverbs_qp.o \
+	cxiverbs_ct.o cxiverbs_counters.o

-- 
2.56.0

Re: [PATCH 00/11] Subject: [PATCH for-next 00/11] RDMA/cxi: add the HPE Cassini verbs driver

[Leon Romanovsky]

On Fri, Oct 30, 2026 at 09:35:28PM -0700, Doug Ledford wrote:
> This series adds cxiverbs, an RDMA driver for the HPE Slingshot Cassini
> NIC. It exposes the NIC through the verbs API using the UET reliable
> unconnected (RU) queue pair type from the UET RDMA core series.
> 
> cxiverbs does not drive the hardware itself. It is an auxiliary bus
> client of the cxi-ss1 Cassini core driver and does everything through
> that driver's exported CXI interface.

<...>

> Before posting, the series was reviewed with the Sashiko AI code
> review tool and checkpatch. All valid findings are fixed, except the
> open items listed above.

<...>

> ---
> base-commit: 306a2f8432f516069b2e673de7a07043132c034f

Sashiko failed to apply this series as this base-commit doesn't exist.
➜  kernel git:(wip/leon-for-rc) git show 306a2f8432f516069b2e673de7a07043132c034f
fatal: bad object 306a2f8432f516069b2e673de7a07043132c034f

https://sashiko.dev/#/patchset/20261030-cxi-for-next-v1-0-bc6e49e07037@hpe.com

Please make sure that upstream sashiko runs on it too.

Thanks

Re: [PATCH 00/11] Subject: [PATCH for-next 00/11] RDMA/cxi: add the HPE Cassini verbs driver

[Jason Gunthorpe]

On Tue, Nov 03, 2026 at 04:28:26PM +0200, Leon Romanovsky wrote:
> On Fri, Oct 30, 2026 at 09:35:28PM -0700, Doug Ledford wrote:
> > This series adds cxiverbs, an RDMA driver for the HPE Slingshot Cassini
> > NIC. It exposes the NIC through the verbs API using the UET reliable
> > unconnected (RU) queue pair type from the UET RDMA core series.
> > 
> > cxiverbs does not drive the hardware itself. It is an auxiliary bus
> > client of the cxi-ss1 Cassini core driver and does everything through
> > that driver's exported CXI interface.
> 
> <...>
> 
> > Before posting, the series was reviewed with the Sashiko AI code
> > review tool and checkpatch. All valid findings are fixed, except the
> > open items listed above.
> 
> <...>
> 
> > ---
> > base-commit: 306a2f8432f516069b2e673de7a07043132c034f
> 
> Sashiko failed to apply this series as this base-commit doesn't exist.
> ➜  kernel git:(wip/leon-for-rc) git show 306a2f8432f516069b2e673de7a07043132c034f
> fatal: bad object 306a2f8432f516069b2e673de7a07043132c034f
> 
> https://sashiko.dev/#/patchset/20261030-cxi-for-next-v1-0-bc6e49e07037@hpe.com
> 
> Please make sure that upstream sashiko runs on it too.

I don't think he can, it relies on the core code series. I Don't know
how to instruct sashiko how to find another series to stich it
together

Jason