[PATCH 00/40] nitro_enclaves: Add support for Nitro Sandboxes

[PATCH 00/40] nitro_enclaves: Add support for Nitro Sandboxes

[Alexander Graf]

Nitro Sandboxes [1] are a new EC2 feature that build on top of the
technical foundations of Nitro Enclaves - an EC2 feature that allows
you to spawn a virtio-vsock-only VM sibling constructed out of resources
donated by the VM you execute in.

Nitro Sandboxes build on that concept and the same hypervisor interface,
but add the ability to implement virtio devices in the parent, run vCPUs
oversubscribed, add device hotplug support and allow to spawn a sandbox
without attestation, but shared memory. They also boot a full EC2-like VM
that runs with a UEFI stack and executes the EIF-wrapped kernel as UEFI
executable.

In combination, Nitro Sandboxes provide a way to create a confidential
compute VM for almost all workloads, but without the biggest gaps that
other confidential compute offerings have: The parent has no timing
control and the parent also has no control over the base device model,
such as timers, PCIe, UART, etc. All base devices are implemented in
Nitro.

This series adds parent-side support for Nitro Sandboxes to the
nitro_enclaves driver.

This supports adds a few non-obvious technical foundations:

guest_memfd like CMA allocator - With Nitro Enclaves, memory came from
hugetlbfs and user space maintained visibility into it. That led to a
few problems. With this new enablement, we add a guest_memfd like
allocator that seeds from CMA, allows the parent to seed the EIF before
hand-over and then loses any mechanism to access donated memory. Thanks
to CMA, it also provides much larger contiguous chunks than 2M huge pages.

Virtio notifications - To implement virtio in the parent, we need to
handle notifications in both directions. Parent to sandbox goes through
a BAR write; sandbox to parent through multiplexed IRQs. We provide
eventfd support in both directions, which also enables vhost support.

NUMA - With Nitro Sandboxes, NUMA became a first class citizen and all
APIs now support NUMA targeting.

Memory Donations - Every sandbox requires hypervisor memory. To allow
for a larger scale of sandboxes you can run, we add a way to donate
memory from the parent to the hypervisor. The user visible effect of
that is that there is a slot limit sysfs file which you can increase.
On write, we allocate physically contiguous CMA memory and donate it.

This series depends on "nitro_enclaves: Fix enclave release, command
retries and the CPU pool", which I posted earlier today.

I used Kiro, an AI coding assistant running Claude Opus and GPT models,
to draft the initial code and patch descritptions.

I validated all new functionality on x86_64 and arm64 as well as ensured
that Nitro Enclaves are still fully functional after this patch series.

[1] https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/amazon-ec2-nitro-sandboxes.html

Alex

Alexander Graf (40):
  nitro_enclaves: Allow the CPU pool to span NUMA nodes
  nitro_enclaves: Add NE_ADD_ANY_VCPU
  nitro_enclaves: Expose CPU pool state under sysfs
  nitro_enclaves: Document multi-NUMA CPU pools
  dma-contiguous: add dma_get_numa_cma_area() accessor
  nitro_enclaves: Allocate physically contiguous enclave memory
  nitro_enclaves: Add NE_MEMORY_SHARED
  nitro_enclaves: Add NE_ADD_MEMORY_REGION
  nitro_enclaves: Document the memory pool
  nitro_enclaves: Add NE_CREATE_VM2 with a launch flag word
  nitro_enclaves: Add NE_ENCLAVE_PCIE_MODE launch flag
  nitro_enclaves: Name the device's error codes in the UAPI header
  nitro_enclaves: Log a reached device limit at info level
  nitro_enclaves: Add NE_SET_INFO_PAGE
  nitro_enclaves: Document the info page in the memory pool
  nitro_enclaves: Turn the command reply into a per-command union
  nitro_enclaves: Add the slot pool donation commands
  nitro_enclaves: Donate memory for the slot pool
  nitro_enclaves: Add the slot pool growth commands
  nitro_enclaves: Add a helper to grow the slot pool
  nitro_enclaves: Document the slot pool
  nitro_enclaves: Expose slot pool state under sysfs
  nitro_enclaves: Add the device add command
  nitro_enclaves: Add NE_ADD_DEVICE
  nitro_enclaves: Read the interrupt multiplexer geometry at probe
  nitro_enclaves: Donate a bitmap page per multiplexed vector
  nitro_enclaves: Dispatch multiplexed interrupts to per-queue bindings
  nitro_enclaves: Add NE_SET_NOTIFY_EVENTFD
  nitro_enclaves: Report the device's info notification
  nitro_enclaves: Count multiplexed interrupts under debugfs
  nitro_enclaves: Map the doorbell window at probe
  nitro_enclaves: Add NE_SET_VRING_CALL
  nitro_enclaves: Assert the layout of the ioctl argument structs
  nitro_enclaves: Pass on the Nitro Sandbox launch flags
  nitro_enclaves: Add NE_ENCLAVE_DIRECT_MODE
  nitro_enclaves: Add NE_ENCLAVE_PCIE_HOTPLUG_MODE
  nitro_enclaves: Add NE_ENCLAVE_CPU_OVERCOMMIT_MODE
  nitro_enclaves: Add NE_GET_CAPS
  nitro_enclaves: Add NE_UPDATE_DEVICE_CONFIG
  nitro_enclaves: Document the driver side of Nitro Sandboxes

 .../sysfs-devices-virtual-misc-nitro_enclaves |  232 +
 Documentation/virt/ne_overview.rst            |  256 +-
 MAINTAINERS                                   |    1 +
 drivers/virt/nitro_enclaves/Kconfig           |   19 +
 drivers/virt/nitro_enclaves/ne_misc_dev.c     | 4832 +++++++++++++++--
 drivers/virt/nitro_enclaves/ne_misc_dev.h     |  206 +-
 drivers/virt/nitro_enclaves/ne_pci_dev.c      | 1568 +++++-
 drivers/virt/nitro_enclaves/ne_pci_dev.h      |  599 +-
 include/linux/dma-map-ops.h                   |    7 +
 include/uapi/linux/nitro_enclaves.h           | 1182 +++-
 kernel/dma/contiguous.c                       |   36 +
 samples/nitro_enclaves/ne_ioctl_sample.c      |   10 +-
 12 files changed, 8282 insertions(+), 666 deletions(-)
 create mode 100644 Documentation/ABI/testing/sysfs-devices-virtual-misc-nitro_enclaves


base-commit: 6628e7d4c6b25c8d18f1345e3a20d21829aae7e4
prerequisite-patch-id: 542b3d78fda912095b75a49becd1b93109befdbc
prerequisite-patch-id: 318368e966c1925652b7976948f435d42b87c72f
prerequisite-patch-id: eb35b5d713e7f50f6d1d02310c9afe10b1628fc6
prerequisite-patch-id: 223bf6e60620ca8fe72f3ff6f3bd3f5839cdd6d5
prerequisite-patch-id: e5305ce0515a02f7f88e72adf02c6a2bee2fd2e0
prerequisite-patch-id: 759b5920f842d70e675770608d9088639c10559e
prerequisite-patch-id: e1837df37663da3d3ce53afa4594828b2d602152
prerequisite-patch-id: f0d4302bd36f94de95951a0db1ae9eb9747076aa
prerequisite-patch-id: 41159b92e250fef705be52f714b6e2761a2cd1f7
prerequisite-patch-id: a0c3a56907acac2297c800209faa2849a8df0e48
prerequisite-patch-id: a80a95bd682b540be5760e8fdf7044379fe32d61
prerequisite-patch-id: 9c01e0fecd6810f3448582715c87cd3dd7a6cdf1
prerequisite-patch-id: b94dfc8bb1039740d88a30535445bb7b3dd1fe74
prerequisite-patch-id: bd76f15728dc0ee2ca3417cd95a6b23a7d63e91f
prerequisite-patch-id: 0c60f1574b590f68885a25760017e9ff87752f66
prerequisite-patch-id: 63dff43da52934a6e6f5e65c3e235a7114b6b4dc
prerequisite-patch-id: af52cf938525874cadd6d879aa76495e563876d2
prerequisite-patch-id: f53ee22764b5f13708cacd3e48a59a82d71a270c
prerequisite-patch-id: 258b8c18168db49aacbda5156e33978b539666ab
-- 
2.47.1

[PATCH 01/40] nitro_enclaves: Allow the CPU pool to span NUMA nodes

[Alexander Graf]

With Nitro Sandboxes, we allow users to span a sandbox across multiple
NUMA nodes and place their sandbox on a NUMA node different from 0.
Allow the ne_cpus= CPU pool to span NUMA nodes.

Signed-off-by: Alexander Graf <graf@amazon.com>
---
 Documentation/virt/ne_overview.rst        |   3 +-
 drivers/virt/nitro_enclaves/ne_misc_dev.c | 159 +++++++++++++++-------
 drivers/virt/nitro_enclaves/ne_misc_dev.h |   5 +-
 include/uapi/linux/nitro_enclaves.h       |  22 ++-
 samples/nitro_enclaves/ne_ioctl_sample.c  |  10 +-
 5 files changed, 137 insertions(+), 62 deletions(-)

diff --git a/Documentation/virt/ne_overview.rst b/Documentation/virt/ne_overview.rst
index 74c2f5919c886..aecd3704acaf5 100644
--- a/Documentation/virt/ne_overview.rst
+++ b/Documentation/virt/ne_overview.rst
@@ -47,7 +47,8 @@ The memory regions carved out of the primary VM and given to an enclave need to
 be aligned 2 MiB / 1 GiB physically contiguous memory regions (or multiple of
 this size e.g. 8 MiB). The memory can be allocated e.g. by using hugetlbfs from
 user space [2][3][7]. The memory size for an enclave needs to be at least
-64 MiB. The enclave memory and CPUs need to be from the same NUMA node.
+64 MiB. The enclave memory and CPUs need to be from the same NUMA node,
+unless the CPU pool spans several of them.
 
 An enclave runs on dedicated cores. CPU 0 and its CPU siblings need to remain
 available for the primary VM. A CPU pool has to be set for NE purposes by an
diff --git a/drivers/virt/nitro_enclaves/ne_misc_dev.c b/drivers/virt/nitro_enclaves/ne_misc_dev.c
index cb1efc95a7f34..6e36abeff01a2 100644
--- a/drivers/virt/nitro_enclaves/ne_misc_dev.c
+++ b/drivers/virt/nitro_enclaves/ne_misc_dev.c
@@ -24,6 +24,7 @@
 #include <linux/module.h>
 #include <linux/mutex.h>
 #include <linux/nitro_enclaves.h>
+#include <linux/numa.h>
 #include <linux/pci.h>
 #include <linux/poll.h>
 #include <linux/range.h>
@@ -36,8 +37,8 @@
 
 /**
  * NE_CPUS_SIZE - Size for max 128 CPUs, for now, in a cpu-list string, comma
- *		  separated. The NE CPU pool includes CPUs from a single NUMA
- *		  node.
+ *		  separated. The NE CPU pool may include CPUs from more than
+ *		  one NUMA node.
  */
 #define NE_CPUS_SIZE		(512)
 
@@ -126,13 +127,27 @@ static const struct kernel_param *ne_cpus_kp;
  *				from the pool rather than derived from a thread
  *				count, so a machine whose cores do not all have
  *				the same number of threads is sized correctly.
- * @numa_node:			NUMA node of the CPUs in the pool.
+ * @numa_node:			NUMA node of the CPUs in the pool, or
+ *				NUMA_NO_NODE if the pool spans multiple nodes or
+ *				no pool is set up.
+ * @core_nid:			The NUMA node of each core slot, indexed the
+ *				same way as @avail_threads_per_core. Every slot
+ *				holds a core, so every entry names a node. A slot
+ *				keeps its node after its core is handed to an
+ *				enclave, because ne_get_cpu_from_cpu_pool()
+ *				still reads it when it reuses a sibling of that
+ *				core. Read while
+ *				the pool CPUs are still online, before
+ *				ne_setup_cpu_pool() offlines them, because
+ *				cpu_to_node() is not documented to answer for an
+ *				offline CPU.
  */
 struct ne_cpu_pool {
 	cpumask_var_t	*avail_threads_per_core;
 	struct mutex	mutex;
 	unsigned int	nr_parent_vm_cores;
 	int		numa_node;
+	int		*core_nid;
 };
 
 static struct ne_cpu_pool ne_cpu_pool = {
@@ -160,14 +175,16 @@ static void ne_free_core_slots(void)
 {
 	unsigned int i = 0;
 
-	if (!ne_cpu_pool.avail_threads_per_core)
-		return;
+	if (ne_cpu_pool.avail_threads_per_core) {
+		for (i = 0; i < ne_cpu_pool.nr_parent_vm_cores; i++)
+			free_cpumask_var(ne_cpu_pool.avail_threads_per_core[i]);
 
-	for (i = 0; i < ne_cpu_pool.nr_parent_vm_cores; i++)
-		free_cpumask_var(ne_cpu_pool.avail_threads_per_core[i]);
+		kfree(ne_cpu_pool.avail_threads_per_core);
+		ne_cpu_pool.avail_threads_per_core = NULL;
+	}
 
-	kfree(ne_cpu_pool.avail_threads_per_core);
-	ne_cpu_pool.avail_threads_per_core = NULL;
+	kfree(ne_cpu_pool.core_nid);
+	ne_cpu_pool.core_nid = NULL;
 }
 
 /**
@@ -218,6 +235,10 @@ static int ne_build_core_slots(const struct cpumask *cpu_pool)
 
 	ne_cpu_pool.nr_parent_vm_cores = nr_cores;
 
+	ne_cpu_pool.core_nid = kmalloc_objs(*ne_cpu_pool.core_nid, nr_cores);
+	if (!ne_cpu_pool.core_nid)
+		goto free_slots;
+
 	for (i = 0; i < nr_cores; i++)
 		if (!zalloc_cpumask_var(&ne_cpu_pool.avail_threads_per_core[i], GFP_KERNEL))
 			goto free_slots;
@@ -237,6 +258,8 @@ static int ne_build_core_slots(const struct cpumask *cpu_pool)
 			cpumask_set_cpu(cpu_sibling, processed);
 		}
 
+		ne_cpu_pool.core_nid[next_core_idx] = cpu_to_node(cpu);
+
 		next_core_idx++;
 	}
 
@@ -275,7 +298,9 @@ static int ne_setup_cpu_pool(const struct cpumask *cpu_pool)
 {
 	unsigned int cpu = 0;
 	unsigned int cpu_sibling = 0;
-	int numa_node = -1;
+	bool have_node = false;
+	bool multi_node = false;
+	int numa_node = NUMA_NO_NODE;
 	int rc = -EINVAL;
 
 	mutex_lock(&ne_cpu_pool.mutex);
@@ -305,29 +330,34 @@ static int ne_setup_cpu_pool(const struct cpumask *cpu_pool)
 		}
 
 	/*
-	 * Check if the CPUs from the NE CPU pool are from the same NUMA node.
+	 * Determine the NUMA node of the CPUs in the pool. If the pool spans
+	 * multiple NUMA nodes, set numa_node to NUMA_NO_NODE; the pool is then
+	 * treated as node-agnostic for placement and memory-locality checks.
+	 * A CPU that reports no node at all is refused, because otherwise such
+	 * a CPU would join a single-node pool and turn it into a spanning one.
 	 */
-	for_each_cpu(cpu, cpu_pool)
-		if (numa_node < 0) {
-			numa_node = cpu_to_node(cpu);
-			if (numa_node < 0) {
-				pr_err("%s: Invalid NUMA node %d\n",
-				       ne_misc_dev.name, numa_node);
+	for_each_cpu(cpu, cpu_pool) {
+		int nid = cpu_to_node(cpu);
 
-				rc = -EINVAL;
+		if (nid < 0) {
+			pr_err("%s: CPU %d reports no NUMA node\n",
+			       ne_misc_dev.name, cpu);
 
-				goto unlock_hotplug;
-			}
-		} else {
-			if (numa_node != cpu_to_node(cpu)) {
-				pr_err("%s: CPUs with different NUMA nodes\n",
-				       ne_misc_dev.name);
+			rc = -EINVAL;
 
-				rc = -EINVAL;
+			goto unlock_hotplug;
+		}
 
-				goto unlock_hotplug;
-			}
+		if (!have_node) {
+			numa_node = nid;
+			have_node = true;
+		} else if (numa_node != nid) {
+			multi_node = true;
 		}
+	}
+
+	if (multi_node)
+		numa_node = NUMA_NO_NODE;
 
 	/*
 	 * Check if CPU 0 and its siblings are included in the provided CPU pool
@@ -414,7 +444,7 @@ static int ne_setup_cpu_pool(const struct cpumask *cpu_pool)
 	cpus_read_unlock();
 reset_pool:
 	ne_cpu_pool.nr_parent_vm_cores = 0;
-	ne_cpu_pool.numa_node = -1;
+	ne_cpu_pool.numa_node = NUMA_NO_NODE;
 	mutex_unlock(&ne_cpu_pool.mutex);
 
 	return rc;
@@ -452,7 +482,7 @@ static void ne_teardown_cpu_pool(void)
 
 	ne_free_core_slots();
 	ne_cpu_pool.nr_parent_vm_cores = 0;
-	ne_cpu_pool.numa_node = -1;
+	ne_cpu_pool.numa_node = NUMA_NO_NODE;
 
 	mutex_unlock(&ne_cpu_pool.mutex);
 }
@@ -633,7 +663,13 @@ static bool ne_donated_cpu(struct ne_enclave *ne_enclave, unsigned int cpu)
 /**
  * ne_get_unused_core_from_cpu_pool() - Get the id of a full core from the
  *					NE CPU pool.
- * @void:	No parameters provided.
+ * @nid:	Required NUMA node for the core, or NUMA_NO_NODE for no
+ *		node constraint.
+ *
+ * When @nid names a specific node, only a free core whose CPUs are
+ * on that node is returned; if none is available the function returns -1
+ * rather than spilling onto another node. When @nid is NUMA_NO_NODE, any
+ * free core is returned.
  *
  * Context: Process context. This function is called with the ne_enclave and
  *	    ne_cpu_pool mutexes held.
@@ -641,19 +677,19 @@ static bool ne_donated_cpu(struct ne_enclave *ne_enclave, unsigned int cpu)
  * * Core id.
  * * -1 if no CPU core available in the pool.
  */
-static int ne_get_unused_core_from_cpu_pool(void)
+static int ne_get_unused_core_from_cpu_pool(int nid)
 {
-	int core_id = -1;
 	unsigned int i = 0;
 
-	for (i = 0; i < ne_cpu_pool.nr_parent_vm_cores; i++)
-		if (!cpumask_empty(ne_cpu_pool.avail_threads_per_core[i])) {
-			core_id = i;
-
-			break;
-		}
+	for (i = 0; i < ne_cpu_pool.nr_parent_vm_cores; i++) {
+		if (cpumask_empty(ne_cpu_pool.avail_threads_per_core[i]))
+			continue;
+		if (nid != NUMA_NO_NODE && ne_cpu_pool.core_nid[i] != nid)
+			continue;
+		return i;
+	}
 
-	return core_id;
+	return -1;
 }
 
 /**
@@ -710,39 +746,62 @@ static int ne_set_enclave_threads_per_core(struct ne_enclave *ne_enclave,
  *				sibling of a new CPU core.
  * @ne_enclave :	Private data associated with the current enclave.
  * @vcpu_id:		vCPU to get from the NE CPU pool.
+ * @nid:		Required NUMA node for a newly taken core, or
+ *			NUMA_NO_NODE for no node constraint. A core already
+ *			held by the enclave is subject to the same
+ *			constraint, so that a run of requests for one node
+ *			does not hand back a sibling on another.
  *
- * Context: Process context. This function is called with the ne_enclave mutex held.
+ * The sibling walk reads ne_enclave->threads_per_core[] and
+ * ne_cpu_pool.core_nid[] at the same index, and stays below the smaller of
+ * their two lengths.
+ *
+ * Context: Process context. This function is called with the ne_enclave mutex
+ *	    held, and takes the ne_cpu_pool mutex over the sibling walk and the
+ *	    new-core path.
  * Return:
  * * 0 on success.
  * * Negative return value on failure.
  */
-static int ne_get_cpu_from_cpu_pool(struct ne_enclave *ne_enclave, u32 *vcpu_id)
+static int ne_get_cpu_from_cpu_pool(struct ne_enclave *ne_enclave, u32 *vcpu_id,
+				    int nid)
 {
+	unsigned int nr_cores = 0;
 	int core_id = -1;
 	unsigned int cpu = 0;
 	unsigned int i = 0;
 	int rc = -EINVAL;
 
+	mutex_lock(&ne_cpu_pool.mutex);
+
+	nr_cores = min(ne_enclave->nr_parent_vm_cores,
+		       ne_cpu_pool.nr_parent_vm_cores);
+
 	/*
 	 * If previously allocated a thread of a core to this enclave, first
 	 * check remaining sibling(s) for new CPU allocations, so that full
-	 * CPU cores are used for the enclave.
+	 * CPU cores are used for the enclave. Cores on a node other than the
+	 * requested one are skipped, so a run of requests for one node does
+	 * not hand back a sibling on another.
 	 */
-	for (i = 0; i < ne_enclave->nr_parent_vm_cores; i++)
+	for (i = 0; i < nr_cores; i++) {
+		if (nid != NUMA_NO_NODE && ne_cpu_pool.core_nid[i] != nid)
+			continue;
+
 		for_each_cpu(cpu, ne_enclave->threads_per_core[i])
 			if (!ne_donated_cpu(ne_enclave, cpu)) {
 				*vcpu_id = cpu;
+				rc = 0;
 
-				return 0;
+				goto unlock_mutex;
 			}
-
-	mutex_lock(&ne_cpu_pool.mutex);
+	}
 
 	/*
 	 * If no remaining siblings, get a core from the NE CPU pool and keep
 	 * track of all the threads in the enclave threads per core data structure.
 	 */
-	core_id = ne_get_unused_core_from_cpu_pool();
+	core_id = ne_get_unused_core_from_cpu_pool(nid);
 
 	rc = ne_set_enclave_threads_per_core(ne_enclave, core_id, *vcpu_id);
 	if (rc < 0)
@@ -970,7 +1029,8 @@ static int ne_sanity_check_user_mem_region_page(struct ne_enclave *ne_enclave,
 		return -NE_ERR_INVALID_PAGE_SIZE;
 	}
 
-	if (ne_enclave->numa_node != page_to_nid(mem_region_page)) {
+	if (ne_enclave->numa_node != NUMA_NO_NODE &&
+	    ne_enclave->numa_node != page_to_nid(mem_region_page)) {
 		dev_err_ratelimited(ne_misc_dev.this_device,
 				    "Page is not from NUMA node %d\n",
 				    ne_enclave->numa_node);
@@ -1329,7 +1389,8 @@ static long ne_enclave_do_ioctl(struct file *file, unsigned int cmd, unsigned lo
 
 		if (!vcpu_id) {
 			/* Use the CPU pool for choosing a CPU for the enclave. */
-			rc = ne_get_cpu_from_cpu_pool(ne_enclave, &vcpu_id);
+			rc = ne_get_cpu_from_cpu_pool(ne_enclave, &vcpu_id,
+						      NUMA_NO_NODE);
 			if (rc < 0) {
 				dev_err_ratelimited(ne_misc_dev.this_device,
 						    "Error in get CPU from pool [rc=%d]\n",
diff --git a/drivers/virt/nitro_enclaves/ne_misc_dev.h b/drivers/virt/nitro_enclaves/ne_misc_dev.h
index 283c17558f2a3..23a783bab3d2f 100644
--- a/drivers/virt/nitro_enclaves/ne_misc_dev.h
+++ b/drivers/virt/nitro_enclaves/ne_misc_dev.h
@@ -54,7 +54,10 @@ struct ne_mem_region {
  *				per full CPU core in the NE CPU pool the enclave
  *				was created from.
  * @nr_vcpus:			Number of vcpus associated with the enclave.
- * @numa_node:			NUMA node of the enclave memory and CPUs.
+ * @numa_node:			NUMA node of the enclave memory and CPUs, or
+ *				NUMA_NO_NODE if the CPU pool it was created from
+ *				spans nodes, in which case its memory is not held
+ *				to a node.
  * @slot_uid:			Slot unique id mapped to the enclave.
  * @state:			Enclave state, updated during enclave lifetime.
  * @threads_per_core:		Enclave full CPU cores array. Each cpumask in the
diff --git a/include/uapi/linux/nitro_enclaves.h b/include/uapi/linux/nitro_enclaves.h
index e808f5ba124d4..206546f24748e 100644
--- a/include/uapi/linux/nitro_enclaves.h
+++ b/include/uapi/linux/nitro_enclaves.h
@@ -26,8 +26,8 @@
  *		  https://www.kernel.org/doc/html/latest/admin-guide/kernel-parameters.html
  *		  CPU 0 and its siblings have to remain available for the
  *		  primary / parent VM, so they cannot be set for enclaves. Full
- *		  CPU core(s), from the same NUMA node, need(s) to be included
- *		  in the CPU pool.
+ *		  CPU core(s) need(s) to be included in the CPU pool. The
+ *		  pool may span NUMA nodes.
  *
  * Context: Process context.
  * Return:
@@ -50,8 +50,9 @@
  * NE_ADD_VCPU - The command is used to set a vCPU for an enclave. The vCPU can
  *		 be auto-chosen from the NE CPU pool or it can be set by the
  *		 caller, with the note that it needs to be available in the NE
- *		 CPU pool. Full CPU core(s), from the same NUMA node, need(s) to
- *		 be associated with an enclave.
+ *		 CPU pool. Full CPU core(s) need(s) to be associated with an
+ *		 enclave. When the CPU pool spans NUMA nodes, the cores of
+ *		 one enclave can be from different nodes.
  *		 The vCPU id is an input / output parameter. If its value is 0,
  *		 then a CPU is chosen from the enclave CPU pool and returned via
  *		 this parameter.
@@ -108,8 +109,11 @@
 /**
  * NE_SET_USER_MEMORY_REGION - The command is used to set a memory region for an
  *			       enclave, given the allocated memory from the
- *			       userspace. Enclave memory needs to be from the
- *			       same NUMA node as the enclave CPUs.
+ *			       userspace. When the CPU pool sits on one
+ *			       NUMA node, enclave memory needs to be from
+ *			       that node. An enclave built out of a pool
+ *			       that spans nodes has no node of its own, and
+ *			       its memory can come from any node.
  *			       The user memory region is an input parameter. It
  *			       includes info provided by the caller - flags,
  *			       memory size and userspace address.
@@ -138,7 +142,9 @@
  * * NE_ERR_MEM_NOT_HUGE_PAGE		- The memory region is not backed by
  *					  huge pages.
  * * NE_ERR_MEM_DIFFERENT_NUMA_NODE	- The memory region is not from the same
- *					  NUMA node as the CPUs.
+ *					  NUMA node as the CPUs. Not returned
+ *					  for an enclave whose CPU pool spans
+ *					  nodes.
  * * NE_ERR_MEM_MAX_REGIONS		- The number of memory regions set for
  *					  the enclave reached maximum.
  * * NE_ERR_INVALID_PAGE_SIZE		- The memory region is not backed by
@@ -223,6 +229,8 @@
 /**
  * NE_ERR_MEM_DIFFERENT_NUMA_NODE - The user space memory region is backed by
  *				    pages from different NUMA nodes than the CPUs.
+ *				    Not returned for an enclave whose CPU pool
+ *				    spans nodes.
  */
 #define NE_ERR_MEM_DIFFERENT_NUMA_NODE		(264)
 /**
diff --git a/samples/nitro_enclaves/ne_ioctl_sample.c b/samples/nitro_enclaves/ne_ioctl_sample.c
index 765b131c73190..84caf5e66fcbd 100644
--- a/samples/nitro_enclaves/ne_ioctl_sample.c
+++ b/samples/nitro_enclaves/ne_ioctl_sample.c
@@ -11,9 +11,10 @@
  * -----
  *
  * Load the nitro_enclaves module, setting also the enclave CPU pool. The
- * enclave CPUs need to be full cores from the same NUMA node. CPU 0 and its
- * siblings have to remain available for the primary / parent VM, so they
- * cannot be included in the enclave CPU pool.
+ * enclave CPUs need to be full cores. They come from the same NUMA node
+ * unless the CPU pool spans several of them. CPU 0 and its siblings have to
+ * remain available for the primary / parent VM, so they cannot be included
+ * in the enclave CPU pool.
  *
  * See the cpu list section from the kernel documentation.
  * https://www.kernel.org/doc/html/latest/admin-guide/kernel-parameters.html#cpu-lists
@@ -41,7 +42,8 @@
  *	dmesg
  *
  * Setup hugetlbfs huge pages. The memory needs to be from the same NUMA node as
- * the enclave CPUs.
+ * the enclave CPUs, unless the CPU pool spans nodes, in which case the enclave
+ * is not held to a node and its memory can come from any of them.
  *
  * https://www.kernel.org/doc/html/latest/admin-guide/mm/hugetlbpage.html
  *
-- 
2.47.1

[PATCH 02/40] nitro_enclaves: Add NE_ADD_ANY_VCPU

[Alexander Graf]

A Nitro Sandbox can span multiple NUMA nodes. The current NE_ADD_VCPU ioctl
only takes a target CPU id as input or a special ANY id. With NUMA, we want
to be able to take any CPU from a specific NUMA node. Add a new
NE_ADD_ANY_VCPU ioctl with nid parameter to select the target node. The new
ioctl returns the selected CPU id or NE_ERR_NO_CPUS_AVAIL_IN_POOL.

To already prepare for later argument extension, filter out the size field
of the ioctl and handle the fields we know about.

Signed-off-by: Alexander Graf <graf@amazon.com>
---
 drivers/virt/nitro_enclaves/ne_misc_dev.c | 92 ++++++++++++++++++++++-
 include/uapi/linux/nitro_enclaves.h       | 54 +++++++++++++
 2 files changed, 145 insertions(+), 1 deletion(-)

diff --git a/drivers/virt/nitro_enclaves/ne_misc_dev.c b/drivers/virt/nitro_enclaves/ne_misc_dev.c
index 6e36abeff01a2..576f1909c7304 100644
--- a/drivers/virt/nitro_enclaves/ne_misc_dev.c
+++ b/drivers/virt/nitro_enclaves/ne_misc_dev.c
@@ -9,11 +9,13 @@
  */
 
 #include <linux/anon_inodes.h>
+#include <linux/build_bug.h>
 #include <linux/capability.h>
 #include <linux/cpu.h>
 #include <linux/device.h>
 #include <linux/file.h>
 #include <linux/hugetlb.h>
+#include <linux/ioctl.h>
 #include <linux/kernel.h>
 #include <linux/limits.h>
 #include <linux/list.h>
@@ -24,12 +26,15 @@
 #include <linux/module.h>
 #include <linux/mutex.h>
 #include <linux/nitro_enclaves.h>
+#include <linux/nodemask.h>
 #include <linux/numa.h>
 #include <linux/pci.h>
 #include <linux/poll.h>
 #include <linux/range.h>
 #include <linux/slab.h>
+#include <linux/stddef.h>
 #include <linux/types.h>
+#include <linux/uaccess.h>
 #include <uapi/linux/vm_sockets.h>
 
 #include "ne_misc_dev.h"
@@ -42,6 +47,23 @@
  */
 #define NE_CPUS_SIZE		(512)
 
+/**
+ * NE_EA_IOCTL() - Mask the size and direction bits out of an ioctl command, so
+ *		   that an Extensible Argument ioctl matches whatever size of
+ *		   argument the caller passed. The direction bits go with them,
+ *		   so a caller that spells the command _IOW(), _IOR() or
+ *		   _IOWR() reaches the same handler, and the handler decides
+ *		   what it reads and writes.
+ * @cmd:	The ioctl command to mask.
+ */
+#define NE_EA_IOCTL(cmd)	((cmd) & ~(IOC_INOUT | IOCSIZE_MASK))
+
+/*
+ * NE_NUMA_ANY reaches the core allocator as a node id, so the uapi spelling
+ * and the kernel one have to agree.
+ */
+static_assert(NE_NUMA_ANY == NUMA_NO_NODE);
+
 /**
  * NE_EIF_LOAD_OFFSET - The offset where to copy the Enclave Image Format (EIF)
  *			image in enclave memory.
@@ -334,7 +356,8 @@ static int ne_setup_cpu_pool(const struct cpumask *cpu_pool)
 	 * multiple NUMA nodes, set numa_node to NUMA_NO_NODE; the pool is then
 	 * treated as node-agnostic for placement and memory-locality checks.
 	 * A CPU that reports no node at all is refused, because otherwise such
-	 * a CPU would join a single-node pool and turn it into a spanning one.
+	 * a CPU would join a single-node pool and turn it into a spanning one,
+	 * and its core would be unreachable through a node-named pick.
 	 */
 	for_each_cpu(cpu, cpu_pool) {
 		int nid = cpu_to_node(cpu);
@@ -1592,6 +1615,73 @@ static long ne_enclave_do_ioctl(struct file *file, unsigned int cmd, unsigned lo
 
 		return 0;
 	}
+	}
+
+	/* Extensible Argument ioctls */
+	switch (NE_EA_IOCTL(cmd)) {
+	case NE_EA_IOCTL(NE_ADD_ANY_VCPU): {
+		struct ne_add_any_vcpu_args __user *uargs = (void __user *)arg;
+		unsigned int usize = _IOC_SIZE(cmd);
+		struct ne_add_any_vcpu_args args;
+		u32 vcpu_id = 0;
+		int rc;
+
+		if (usize < offsetofend(struct ne_add_any_vcpu_args, vcpu_id))
+			return -EINVAL;
+
+		rc = copy_struct_from_user(&args, sizeof(args), uargs, usize);
+		if (rc)
+			return rc;
+
+		if (args.numa_node != NE_NUMA_ANY &&
+		    (args.numa_node < 0 || args.numa_node >= nr_node_ids ||
+		     !node_state(args.numa_node, N_POSSIBLE))) {
+			dev_err_ratelimited(ne_misc_dev.this_device,
+					    "Invalid NUMA node %d\n", args.numa_node);
+
+			return -EINVAL;
+		}
+
+		rc = mutex_lock_killable(&ne_enclave->enclave_info_mutex);
+		if (rc)
+			return rc;
+
+		if (ne_enclave->state != NE_STATE_INIT) {
+			dev_err_ratelimited(ne_misc_dev.this_device,
+					    "Enclave is not in init state\n");
+
+			mutex_unlock(&ne_enclave->enclave_info_mutex);
+
+			return -NE_ERR_NOT_IN_INIT_STATE;
+		}
+
+		rc = ne_get_cpu_from_cpu_pool(ne_enclave, &vcpu_id, args.numa_node);
+		if (rc < 0) {
+			dev_err_ratelimited(ne_misc_dev.this_device,
+					    "Error in get CPU from pool [rc=%d node=%d]\n",
+					    rc, args.numa_node);
+
+			mutex_unlock(&ne_enclave->enclave_info_mutex);
+
+			return rc;
+		}
+
+		rc = ne_add_vcpu_ioctl(ne_enclave, vcpu_id);
+		if (rc < 0) {
+			mutex_unlock(&ne_enclave->enclave_info_mutex);
+
+			return rc;
+		}
+
+		mutex_unlock(&ne_enclave->enclave_info_mutex);
+
+		args.vcpu_id = vcpu_id;
+
+		if (copy_to_user(uargs, &args, min_t(size_t, usize, sizeof(args))))
+			return -EFAULT;
+
+		return 0;
+	}
 
 	default:
 		return -ENOTTY;
diff --git a/include/uapi/linux/nitro_enclaves.h b/include/uapi/linux/nitro_enclaves.h
index 206546f24748e..e266b8a611c92 100644
--- a/include/uapi/linux/nitro_enclaves.h
+++ b/include/uapi/linux/nitro_enclaves.h
@@ -83,6 +83,60 @@
  */
 #define NE_ADD_VCPU			_IOWR(0xAE, 0x21, __u32)
 
+/**
+ * struct ne_add_any_vcpu_args - Argument for %NE_ADD_ANY_VCPU.
+ * @numa_node:	The NUMA node the vCPU has to come from, or %NE_NUMA_ANY for
+ *		any node.
+ * @vcpu_id:	The vCPU id the kernel chose, set on success. Ignored on
+ *		input.
+ */
+struct ne_add_any_vcpu_args {
+	__s32	numa_node;
+	__u32	vcpu_id;
+};
+
+/**
+ * NE_NUMA_ANY - Value for &struct ne_add_any_vcpu_args.numa_node that asks for
+ *		 a vCPU from any NUMA node.
+ */
+#define NE_NUMA_ANY			(-1)
+
+/**
+ * NE_ADD_ANY_VCPU - The command is used to set a vCPU for an enclave, chosen
+ *		     from the NE CPU pool. @numa_node confines the choice to
+ *		     one NUMA node; %NE_NUMA_ANY leaves it unconstrained. A
+ *		     caller that spreads an enclave over several nodes asks
+ *		     for each node in turn and stops when a node runs out.
+ *		     @vcpu_id carries back the vCPU id the kernel chose, so
+ *		     a caller collects the enclave's vCPU set as it builds
+ *		     it.
+ *		     The ioctl can be invoked on the enclave fd, before an
+ *		     enclave is started.
+ *
+ * Context: Process context.
+ * Return:
+ * * 0				- Logic successfully completed.
+ * * -1				- There was a failure in the ioctl logic.
+ * On failure, errno is set to:
+ * * ENODEV			- The NE PCI device is not available.
+ * * EFAULT			- Argument read or write-back failure.
+ * * EINVAL			- The argument is shorter than @vcpu_id, or
+ *				  @numa_node is neither %NE_NUMA_ANY nor a
+ *				  possible NUMA node.
+ * * EIO			- Current task mm is not the same as the one
+ *				  that created the enclave.
+ * * E2BIG			- The argument is longer than this kernel knows
+ *				  about and the excess is not all zero.
+ * * NE_ERR_NOT_IN_INIT_STATE	- The enclave is not in init state (init =
+ *				  before being started).
+ * * NE_ERR_NO_CPUS_AVAIL_IN_POOL - No CPUs available in the NE CPU pool on
+ *				  @numa_node.
+ * * NE_ERR_VCPU_INVALID_CPU_CORE - The core the pool handed back is outside
+ *				  the enclave's core slots.
+ * * Error codes from the NE PCI device request.
+ */
+#define NE_ADD_ANY_VCPU			_IOWR(0xAE, 0x2D, struct ne_add_any_vcpu_args)
+
 /**
  * NE_GET_IMAGE_LOAD_INFO - The command is used to get information needed for
  *			    in-memory enclave image loading e.g. offset in
-- 
2.47.1

[PATCH 03/40] nitro_enclaves: Expose CPU pool state under sysfs

[Alexander Graf]

When using the dedicated CPU pool, orchestrators need to know whether
there is actually capacity left to spawn a new enclave or sandbox on
a given target NUMA node. The easiest place to expose that is sysfs.

Report the pool under
/sys/devices/virtual/misc/nitro_enclaves/cpu_pool/: total, used and
avail, once as total and once in a nodeN subdirectory per node.
The files are binary attributes, because a CPU list can outgrow the one
page a device attribute gets.

Signed-off-by: Alexander Graf <graf@amazon.com>
---
 .../sysfs-devices-virtual-misc-nitro_enclaves |  75 ++++
 MAINTAINERS                                   |   1 +
 drivers/virt/nitro_enclaves/ne_misc_dev.c     | 388 +++++++++++++++++-
 drivers/virt/nitro_enclaves/ne_misc_dev.h     |   3 +
 drivers/virt/nitro_enclaves/ne_pci_dev.c      |  16 +
 5 files changed, 469 insertions(+), 14 deletions(-)
 create mode 100644 Documentation/ABI/testing/sysfs-devices-virtual-misc-nitro_enclaves

diff --git a/Documentation/ABI/testing/sysfs-devices-virtual-misc-nitro_enclaves b/Documentation/ABI/testing/sysfs-devices-virtual-misc-nitro_enclaves
new file mode 100644
index 0000000000000..596054b75e0cf
--- /dev/null
+++ b/Documentation/ABI/testing/sysfs-devices-virtual-misc-nitro_enclaves
@@ -0,0 +1,75 @@
+What:		/sys/devices/virtual/misc/nitro_enclaves/cpu_pool/total
+What:		/sys/devices/virtual/misc/nitro_enclaves/cpu_pool/used
+What:		/sys/devices/virtual/misc/nitro_enclaves/cpu_pool/avail
+Date:		November 2026
+KernelVersion:	7.5
+Contact:	aws-nitro-enclaves-devel@amazon.com
+Description:
+		Read-only. CPU-list strings describing the Nitro Enclaves CPU
+		pool. The pool is handed out a whole core at a time, so these
+		files name threads but move in units of a core:
+
+		total
+			every CPU thread dedicated to the pool
+
+		used
+			every thread of every core an enclave holds, including
+			the ones it has not made vCPUs of yet
+
+		avail
+			every thread of every core no enclave holds, so the set
+			a new enclave can draw from
+
+		A thread leaves used when the enclave holding it is freed.
+		A release that fails part way through keeps the thread in
+		used. total changes when the pool is reconfigured, which
+		requires that no enclave exists.
+
+		A read longer than one page is served in parts, and each part
+		renders the pool as it stands when that part is read. A thread
+		that changes hands during such a read can be missing from every
+		part, and the parts joined together can fail to parse as a CPU
+		list at all - see the warning in the kerneldoc of
+		bitmap_print_bitmask_to_buf() in lib/bitmap-str.c, which
+		bitmap_print_list_to_buf() refers to.
+
+		NE_ADD_VCPU auto-pick and NE_ADD_ANY_VCPU each look for a free
+		thread of a core the enclave already holds before taking a
+		core out of avail, so the thread either returns can be one
+		avail does not list.
+
+		A read takes the pool lock, so it can wait for a write to
+		ne_cpus to finish.
+
+What:		/sys/devices/virtual/misc/nitro_enclaves/cpu_pool/nodeX/total
+What:		/sys/devices/virtual/misc/nitro_enclaves/cpu_pool/nodeX/used
+What:		/sys/devices/virtual/misc/nitro_enclaves/cpu_pool/nodeX/avail
+Date:		November 2026
+KernelVersion:	7.5
+Contact:	aws-nitro-enclaves-devel@amazon.com
+Description:
+		Read-only. The cpu_pool files of the same name, over the
+		threads of NUMA node X alone. There is a nodeX directory for
+		each node that was online when the device was probed, and its
+		files read empty for a node the pool holds no thread of.
+
+		A thread appears under exactly one node, the one its core was
+		on while that core was still online. The driver offlines the
+		threads it takes into the pool, so
+		/sys/devices/system/node/nodeX/cpulist stops listing them and
+		is not what these files are cut from.
+
+		nodeX/used and nodeX/avail are cut out of nodeX/total the way
+		used and avail are cut out of total, so a node file and its
+		pool-wide counterpart cannot disagree, and the nodeX/total of
+		every node join to total.
+
+		NE_ADD_ANY_VCPU takes a core from the node it names, so
+		nodeX/avail is what a request for that node can draw on. What
+		is written above about a read longer than one page, and about
+		the free thread of a held core that avail does not list,
+		holds for these files as well.
+
+		The directories appear once the device has been probed and the
+		misc device registered, so a reader that opens the misc device
+		the moment it appears can find cpu_pool empty.
diff --git a/MAINTAINERS b/MAINTAINERS
index 20e4a56a197ee..f946a79b34d0d 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -19839,6 +19839,7 @@ R:	The AWS Nitro Enclaves Team <aws-nitro-enclaves-devel@amazon.com>
 L:	linux-kernel@vger.kernel.org
 S:	Supported
 W:	https://aws.amazon.com/ec2/nitro/nitro-enclaves/
+F:	Documentation/ABI/testing/sysfs-devices-virtual-misc-nitro_enclaves
 F:	Documentation/virt/ne_overview.rst
 F:	drivers/virt/nitro_enclaves/
 F:	include/linux/nitro_enclaves.h
diff --git a/drivers/virt/nitro_enclaves/ne_misc_dev.c b/drivers/virt/nitro_enclaves/ne_misc_dev.c
index 576f1909c7304..66bcaec8d3199 100644
--- a/drivers/virt/nitro_enclaves/ne_misc_dev.c
+++ b/drivers/virt/nitro_enclaves/ne_misc_dev.c
@@ -12,11 +12,14 @@
 #include <linux/build_bug.h>
 #include <linux/capability.h>
 #include <linux/cpu.h>
+#include <linux/cpumask.h>
 #include <linux/device.h>
 #include <linux/file.h>
 #include <linux/hugetlb.h>
 #include <linux/ioctl.h>
 #include <linux/kernel.h>
+#include <linux/kobject.h>
+#include <linux/kstrtox.h>
 #include <linux/limits.h>
 #include <linux/list.h>
 #include <linux/miscdevice.h>
@@ -33,6 +36,8 @@
 #include <linux/range.h>
 #include <linux/slab.h>
 #include <linux/stddef.h>
+#include <linux/string.h>
+#include <linux/sysfs.h>
 #include <linux/types.h>
 #include <linux/uaccess.h>
 #include <uapi/linux/vm_sockets.h>
@@ -95,18 +100,6 @@ static const struct file_operations ne_fops = {
 	.compat_ioctl	= compat_ptr_ioctl,
 };
 
-static struct miscdevice ne_misc_dev = {
-	.minor	= MISC_DYNAMIC_MINOR,
-	.name	= "nitro_enclaves",
-	.fops	= &ne_fops,
-	.mode	= 0660,
-};
-
-struct ne_devs ne_devs = {
-	.ne_misc_dev	= &ne_misc_dev,
-	.lock		= __RWSEM_INITIALIZER(ne_devs.lock),
-};
-
 /*
  * TODO: Update logic to create new sysfs entries instead of using
  * a kernel parameter e.g. if multiple sysfs files needed.
@@ -163,6 +156,19 @@ static const struct kernel_param *ne_cpus_kp;
  *				ne_setup_cpu_pool() offlines them, because
  *				cpu_to_node() is not documented to answer for an
  *				offline CPU.
+ * @node_cpus:			The threads of the pool on each NUMA node, indexed
+ *				by node id and sized by nr_node_ids. It partitions
+ *				@total_cpus: a thread sits in the entry of the node
+ *				its core was on while that core was still online,
+ *				which is the node @core_nid records for the same
+ *				core. NULL if no pool is set up.
+ * @total_cpus:			Every CPU thread in the pool, as parsed from
+ *				ne_cpus. Empty if no pool is set up.
+ * @used_cpus:			The subset of @total_cpus that enclaves hold.
+ *				A whole core enters when an enclave takes it
+ *				and leaves when the enclave is freed, in
+ *				the same two places that move the core out of
+ *				and back into @avail_threads_per_core.
  */
 struct ne_cpu_pool {
 	cpumask_var_t	*avail_threads_per_core;
@@ -170,12 +176,330 @@ struct ne_cpu_pool {
 	unsigned int	nr_parent_vm_cores;
 	int		numa_node;
 	int		*core_nid;
+	cpumask_var_t	*node_cpus;
+	struct cpumask	total_cpus;
+	struct cpumask	used_cpus;
 };
 
 static struct ne_cpu_pool ne_cpu_pool = {
 	.mutex = __MUTEX_INITIALIZER(ne_cpu_pool.mutex),
 };
 
+/*
+ * cpu_pool/ sysfs group on the Nitro Enclaves misc device: report the CPU
+ * pool occupancy.
+ *
+ *   /sys/devices/virtual/misc/nitro_enclaves/cpu_pool/{total,used,avail}
+ *
+ * The three files are CPU-list strings rendered from the pool masks under the
+ * pool mutex. avail is what is left of total once used is taken out of it, so
+ * a read has nothing to work out that the allocator has not already decided.
+ * They are binary attributes sized the way the per-node CPU lists under
+ * /sys/devices/system/node are, because a CPU list naming every thread of a
+ * large pool does not fit in the single page a device attribute is handed.
+ */
+
+static ssize_t total_read(struct file *file, struct kobject *kobj,
+			  const struct bin_attribute *attr, char *buf,
+			  loff_t off, size_t count)
+{
+	ssize_t n;
+
+	mutex_lock(&ne_cpu_pool.mutex);
+	n = cpumap_print_list_to_buf(buf, &ne_cpu_pool.total_cpus, off, count);
+	mutex_unlock(&ne_cpu_pool.mutex);
+
+	return n;
+}
+
+static const BIN_ATTR_RO(total, CPULIST_FILE_MAX_BYTES);
+
+static ssize_t used_read(struct file *file, struct kobject *kobj,
+			 const struct bin_attribute *attr, char *buf,
+			 loff_t off, size_t count)
+{
+	ssize_t n;
+
+	mutex_lock(&ne_cpu_pool.mutex);
+	n = cpumap_print_list_to_buf(buf, &ne_cpu_pool.used_cpus, off, count);
+	mutex_unlock(&ne_cpu_pool.mutex);
+
+	return n;
+}
+
+static const BIN_ATTR_RO(used, CPULIST_FILE_MAX_BYTES);
+
+static ssize_t avail_read(struct file *file, struct kobject *kobj,
+			  const struct bin_attribute *attr, char *buf,
+			  loff_t off, size_t count)
+{
+	cpumask_var_t avail;
+	ssize_t n;
+
+	if (!alloc_cpumask_var(&avail, GFP_KERNEL))
+		return -ENOMEM;
+
+	mutex_lock(&ne_cpu_pool.mutex);
+	cpumask_andnot(avail, &ne_cpu_pool.total_cpus, &ne_cpu_pool.used_cpus);
+	mutex_unlock(&ne_cpu_pool.mutex);
+
+	n = cpumap_print_list_to_buf(buf, avail, off, count);
+
+	free_cpumask_var(avail);
+
+	return n;
+}
+
+static const BIN_ATTR_RO(avail, CPULIST_FILE_MAX_BYTES);
+
+static const struct bin_attribute *const ne_cpu_pool_bin_attrs[] = {
+	&bin_attr_total,
+	&bin_attr_used,
+	&bin_attr_avail,
+	NULL,
+};
+
+static const struct attribute_group ne_cpu_pool_group = {
+	.bin_attrs	= ne_cpu_pool_bin_attrs,
+};
+
+/*
+ * cpu_pool/nodeX/ sysfs groups on the Nitro Enclaves misc device: the three
+ * files above, over the threads of one NUMA node.
+ *
+ *   /sys/devices/virtual/misc/nitro_enclaves/cpu_pool/nodeX/{total,used,avail}
+ *
+ * One directory per node that was online when the device was probed, each
+ * holding the same three attributes. The node id is in the directory's name
+ * and is read back out of it, so the three attributes are shared rather than
+ * made per node. A node the pool holds nothing on reads empty in all three.
+ */
+
+/**
+ * enum ne_cpu_pool_view - Which view of a node's pool threads to render.
+ * @NE_CPU_POOL_TOTAL:	Every pool thread of the node.
+ * @NE_CPU_POOL_USED:	The ones enclaves hold.
+ * @NE_CPU_POOL_AVAIL:	The rest.
+ */
+enum ne_cpu_pool_view {
+	NE_CPU_POOL_TOTAL,
+	NE_CPU_POOL_USED,
+	NE_CPU_POOL_AVAIL,
+};
+
+/**
+ * ne_cpu_pool_node_read() - Render one view of one node's pool threads.
+ * @kobj:	The nodeX directory the read came through.
+ * @view:	Which of the three views to render.
+ * @buf:	Buffer to render the CPU list into.
+ * @off:	Offset into the CPU list to render from.
+ * @count:	Bytes of the CPU list to render.
+ *
+ * The node's threads come from the partition of the pool recorded while the
+ * pool's CPUs were online, so this answers for CPUs the driver has offlined.
+ *
+ * Context: Process context.
+ * Return:
+ * * Number of bytes rendered.
+ * * Negative return value on failure.
+ */
+static ssize_t ne_cpu_pool_node_read(struct kobject *kobj,
+				     enum ne_cpu_pool_view view, char *buf,
+				     loff_t off, size_t count)
+{
+	cpumask_var_t threads;
+	ssize_t n;
+	int nid = 0;
+
+	if (kstrtoint(kobject_name(kobj) + strlen("node"), 10, &nid))
+		return -EINVAL;
+
+	if (nid < 0 || (unsigned int)nid >= nr_node_ids)
+		return -EINVAL;
+
+	if (!alloc_cpumask_var(&threads, GFP_KERNEL))
+		return -ENOMEM;
+
+	mutex_lock(&ne_cpu_pool.mutex);
+
+	if (!ne_cpu_pool.node_cpus) {
+		cpumask_clear(threads);
+	} else {
+		cpumask_copy(threads, ne_cpu_pool.node_cpus[nid]);
+
+		if (view == NE_CPU_POOL_USED)
+			cpumask_and(threads, threads, &ne_cpu_pool.used_cpus);
+		else if (view == NE_CPU_POOL_AVAIL)
+			cpumask_andnot(threads, threads,
+				       &ne_cpu_pool.used_cpus);
+	}
+
+	mutex_unlock(&ne_cpu_pool.mutex);
+
+	n = cpumap_print_list_to_buf(buf, threads, off, count);
+
+	free_cpumask_var(threads);
+
+	return n;
+}
+
+static ssize_t node_total_read(struct file *file, struct kobject *kobj,
+			       const struct bin_attribute *attr, char *buf,
+			       loff_t off, size_t count)
+{
+	return ne_cpu_pool_node_read(kobj, NE_CPU_POOL_TOTAL, buf, off, count);
+}
+
+static const struct bin_attribute bin_attr_node_total =
+	__BIN_ATTR(total, 0444, node_total_read, NULL, CPULIST_FILE_MAX_BYTES);
+
+static ssize_t node_used_read(struct file *file, struct kobject *kobj,
+			      const struct bin_attribute *attr, char *buf,
+			      loff_t off, size_t count)
+{
+	return ne_cpu_pool_node_read(kobj, NE_CPU_POOL_USED, buf, off, count);
+}
+
+static const struct bin_attribute bin_attr_node_used =
+	__BIN_ATTR(used, 0444, node_used_read, NULL, CPULIST_FILE_MAX_BYTES);
+
+static ssize_t node_avail_read(struct file *file, struct kobject *kobj,
+			       const struct bin_attribute *attr, char *buf,
+			       loff_t off, size_t count)
+{
+	return ne_cpu_pool_node_read(kobj, NE_CPU_POOL_AVAIL, buf, off, count);
+}
+
+static const struct bin_attribute bin_attr_node_avail =
+	__BIN_ATTR(avail, 0444, node_avail_read, NULL, CPULIST_FILE_MAX_BYTES);
+
+static const struct bin_attribute *const ne_cpu_pool_node_bin_attrs[] = {
+	&bin_attr_node_total,
+	&bin_attr_node_used,
+	&bin_attr_node_avail,
+	NULL,
+};
+
+static const struct attribute_group ne_cpu_pool_node_group = {
+	.bin_attrs	= ne_cpu_pool_node_bin_attrs,
+};
+
+static struct kobject *ne_cpu_pool_kobj;
+static struct kobject **ne_cpu_pool_node_kobj;
+
+static struct miscdevice ne_misc_dev = {
+	.minor	= MISC_DYNAMIC_MINOR,
+	.name	= "nitro_enclaves",
+	.fops	= &ne_fops,
+	.mode	= 0660,
+};
+
+struct ne_devs ne_devs = {
+	.ne_misc_dev	= &ne_misc_dev,
+	.lock		= __RWSEM_INITIALIZER(ne_devs.lock),
+};
+
+/**
+ * ne_cpu_pool_sysfs_exit() - Take down the cpu_pool sysfs tree.
+ * @void:	No parameters provided.
+ *
+ * Idempotent, so the setup path can call it on a tree it built part of.
+ *
+ * Context: Process context.
+ */
+void ne_cpu_pool_sysfs_exit(void)
+{
+	unsigned int nid = 0;
+
+	if (ne_cpu_pool_node_kobj) {
+		for (nid = 0; nid < nr_node_ids; nid++) {
+			if (!ne_cpu_pool_node_kobj[nid])
+				continue;
+
+			sysfs_remove_group(ne_cpu_pool_node_kobj[nid],
+					   &ne_cpu_pool_node_group);
+			kobject_put(ne_cpu_pool_node_kobj[nid]);
+			ne_cpu_pool_node_kobj[nid] = NULL;
+		}
+
+		kfree(ne_cpu_pool_node_kobj);
+		ne_cpu_pool_node_kobj = NULL;
+	}
+
+	if (ne_cpu_pool_kobj) {
+		sysfs_remove_group(ne_cpu_pool_kobj, &ne_cpu_pool_group);
+		kobject_put(ne_cpu_pool_kobj);
+		ne_cpu_pool_kobj = NULL;
+	}
+}
+
+/**
+ * ne_cpu_pool_sysfs_init() - Build the cpu_pool sysfs tree.
+ * @void:	No parameters provided.
+ *
+ * The pool-wide files hang on a kobject of their own rather than on a named
+ * attribute group of the misc device, because the per-node directories are
+ * kobjects and a kobject takes a kobject as its parent. So this runs once the
+ * misc device is registered and its kobject exists.
+ *
+ * A directory is made for each node that is online now. A node that comes
+ * online later has no directory, and the pool cannot take a thread of it
+ * either: ne_setup_cpu_pool() refuses a CPU that is offline.
+ *
+ * Context: Process context.
+ * Return:
+ * * 0 on success.
+ * * Negative return value on failure.
+ */
+int ne_cpu_pool_sysfs_init(void)
+{
+	unsigned int nid = 0;
+	int rc = -ENOMEM;
+
+	ne_cpu_pool_kobj = kobject_create_and_add("cpu_pool",
+						  &ne_misc_dev.this_device->kobj);
+	if (!ne_cpu_pool_kobj)
+		return -ENOMEM;
+
+	rc = sysfs_create_group(ne_cpu_pool_kobj, &ne_cpu_pool_group);
+	if (rc < 0)
+		goto sysfs_exit;
+
+	ne_cpu_pool_node_kobj = kzalloc_objs(*ne_cpu_pool_node_kobj,
+					     nr_node_ids);
+	if (!ne_cpu_pool_node_kobj) {
+		rc = -ENOMEM;
+
+		goto sysfs_exit;
+	}
+
+	for_each_online_node(nid) {
+		char name[16];
+
+		snprintf(name, sizeof(name), "node%u", nid);
+
+		ne_cpu_pool_node_kobj[nid] =
+			kobject_create_and_add(name, ne_cpu_pool_kobj);
+		if (!ne_cpu_pool_node_kobj[nid]) {
+			rc = -ENOMEM;
+
+			goto sysfs_exit;
+		}
+
+		rc = sysfs_create_group(ne_cpu_pool_node_kobj[nid],
+					&ne_cpu_pool_node_group);
+		if (rc < 0)
+			goto sysfs_exit;
+	}
+
+	return 0;
+
+sysfs_exit:
+	ne_cpu_pool_sysfs_exit();
+
+	return rc;
+}
+
 /**
  * struct ne_phys_contig_mem_regions - Contiguous physical memory regions.
  * @num:	The number of regions that currently has.
@@ -196,6 +520,7 @@ struct ne_phys_contig_mem_regions {
 static void ne_free_core_slots(void)
 {
 	unsigned int i = 0;
+	unsigned int nid = 0;
 
 	if (ne_cpu_pool.avail_threads_per_core) {
 		for (i = 0; i < ne_cpu_pool.nr_parent_vm_cores; i++)
@@ -207,6 +532,14 @@ static void ne_free_core_slots(void)
 
 	kfree(ne_cpu_pool.core_nid);
 	ne_cpu_pool.core_nid = NULL;
+
+	if (ne_cpu_pool.node_cpus) {
+		for (nid = 0; nid < nr_node_ids; nid++)
+			free_cpumask_var(ne_cpu_pool.node_cpus[nid]);
+
+		kfree(ne_cpu_pool.node_cpus);
+		ne_cpu_pool.node_cpus = NULL;
+	}
 }
 
 /**
@@ -234,8 +567,10 @@ static int ne_build_core_slots(const struct cpumask *cpu_pool)
 	unsigned int cpu_sibling = 0;
 	unsigned int i = 0;
 	unsigned int next_core_idx = 0;
+	unsigned int nid = 0;
 	unsigned int nr_cores = 0;
 	cpumask_var_t processed;
+	int core_nid = 0;
 	int rc = -ENOMEM;
 
 	if (!zalloc_cpumask_var(&processed, GFP_KERNEL))
@@ -261,6 +596,14 @@ static int ne_build_core_slots(const struct cpumask *cpu_pool)
 	if (!ne_cpu_pool.core_nid)
 		goto free_slots;
 
+	ne_cpu_pool.node_cpus = kzalloc_objs(*ne_cpu_pool.node_cpus, nr_node_ids);
+	if (!ne_cpu_pool.node_cpus)
+		goto free_slots;
+
+	for (nid = 0; nid < nr_node_ids; nid++)
+		if (!zalloc_cpumask_var(&ne_cpu_pool.node_cpus[nid], GFP_KERNEL))
+			goto free_slots;
+
 	for (i = 0; i < nr_cores; i++)
 		if (!zalloc_cpumask_var(&ne_cpu_pool.avail_threads_per_core[i], GFP_KERNEL))
 			goto free_slots;
@@ -280,7 +623,12 @@ static int ne_build_core_slots(const struct cpumask *cpu_pool)
 			cpumask_set_cpu(cpu_sibling, processed);
 		}
 
-		ne_cpu_pool.core_nid[next_core_idx] = cpu_to_node(cpu);
+		core_nid = cpu_to_node(cpu);
+
+		ne_cpu_pool.core_nid[next_core_idx] = core_nid;
+		cpumask_or(ne_cpu_pool.node_cpus[core_nid],
+			   ne_cpu_pool.node_cpus[core_nid],
+			   ne_cpu_pool.avail_threads_per_core[next_core_idx]);
 
 		next_core_idx++;
 	}
@@ -452,6 +800,9 @@ static int ne_setup_cpu_pool(const struct cpumask *cpu_pool)
 		}
 	}
 
+	cpumask_copy(&ne_cpu_pool.total_cpus, cpu_pool);
+	cpumask_clear(&ne_cpu_pool.used_cpus);
+
 	ne_cpu_pool.numa_node = numa_node;
 
 	mutex_unlock(&ne_cpu_pool.mutex);
@@ -466,6 +817,8 @@ static int ne_setup_cpu_pool(const struct cpumask *cpu_pool)
 unlock_hotplug:
 	cpus_read_unlock();
 reset_pool:
+	cpumask_clear(&ne_cpu_pool.total_cpus);
+	cpumask_clear(&ne_cpu_pool.used_cpus);
 	ne_cpu_pool.nr_parent_vm_cores = 0;
 	ne_cpu_pool.numa_node = NUMA_NO_NODE;
 	mutex_unlock(&ne_cpu_pool.mutex);
@@ -504,6 +857,8 @@ static void ne_teardown_cpu_pool(void)
 		}
 
 	ne_free_core_slots();
+	cpumask_clear(&ne_cpu_pool.total_cpus);
+	cpumask_clear(&ne_cpu_pool.used_cpus);
 	ne_cpu_pool.nr_parent_vm_cores = 0;
 	ne_cpu_pool.numa_node = NUMA_NO_NODE;
 
@@ -758,6 +1113,9 @@ static int ne_set_enclave_threads_per_core(struct ne_enclave *ne_enclave,
 	for_each_cpu(cpu, ne_cpu_pool.avail_threads_per_core[core_id])
 		cpumask_set_cpu(cpu, ne_enclave->threads_per_core[core_id]);
 
+	cpumask_or(&ne_cpu_pool.used_cpus, &ne_cpu_pool.used_cpus,
+		   ne_cpu_pool.avail_threads_per_core[core_id]);
+
 	cpumask_clear(ne_cpu_pool.avail_threads_per_core[core_id]);
 
 	return 0;
@@ -1768,9 +2126,11 @@ static void ne_enclave_remove_all_vcpu_id_entries(struct ne_enclave *ne_enclave)
 		       ne_cpu_pool.nr_parent_vm_cores);
 
 	for (i = 0; i < nr_cores; i++)
-		for_each_cpu(cpu, ne_enclave->threads_per_core[i])
+		for_each_cpu(cpu, ne_enclave->threads_per_core[i]) {
 			/* Update the available NE CPU pool. */
 			cpumask_set_cpu(cpu, ne_cpu_pool.avail_threads_per_core[i]);
+			cpumask_clear_cpu(cpu, &ne_cpu_pool.used_cpus);
+		}
 
 	for (i = 0; i < ne_enclave->nr_parent_vm_cores; i++)
 		free_cpumask_var(ne_enclave->threads_per_core[i]);
diff --git a/drivers/virt/nitro_enclaves/ne_misc_dev.h b/drivers/virt/nitro_enclaves/ne_misc_dev.h
index 23a783bab3d2f..483037061eafe 100644
--- a/drivers/virt/nitro_enclaves/ne_misc_dev.h
+++ b/drivers/virt/nitro_enclaves/ne_misc_dev.h
@@ -115,4 +115,7 @@ struct ne_devs {
 /* Nitro Enclaves (NE) data structure for keeping refs to the NE misc and PCI devices. */
 extern struct ne_devs ne_devs;
 
+int ne_cpu_pool_sysfs_init(void);
+void ne_cpu_pool_sysfs_exit(void);
+
 #endif /* _NE_MISC_DEV_H_ */
diff --git a/drivers/virt/nitro_enclaves/ne_pci_dev.c b/drivers/virt/nitro_enclaves/ne_pci_dev.c
index edc72d4f623e5..2d44cae4ac0cf 100644
--- a/drivers/virt/nitro_enclaves/ne_pci_dev.c
+++ b/drivers/virt/nitro_enclaves/ne_pci_dev.c
@@ -695,8 +695,20 @@ static int ne_pci_probe(struct pci_dev *pdev, const struct pci_device_id *id)
 
 	up_write(&ne_devs.lock);
 
+	rc = ne_cpu_pool_sysfs_init();
+	if (rc < 0) {
+		dev_err(&pdev->dev, "Error in cpu pool sysfs init [rc=%d]\n",
+			rc);
+
+		goto deregister_misc_dev;
+	}
+
 	return 0;
 
+deregister_misc_dev:
+	down_write(&ne_devs.lock);
+	ne_pci_dev_detach_enclaves(ne_pci_dev);
+	misc_deregister(ne_devs.ne_misc_dev);
 disable_ne_pci_dev:
 	ne_devs.ne_pci_dev = NULL;
 unlock_ne_devs:
@@ -727,6 +739,8 @@ static void ne_pci_remove(struct pci_dev *pdev)
 {
 	struct ne_pci_dev *ne_pci_dev = pci_get_drvdata(pdev);
 
+	ne_cpu_pool_sysfs_exit();
+
 	down_write(&ne_devs.lock);
 
 	ne_pci_dev_detach_enclaves(ne_pci_dev);
@@ -765,6 +779,8 @@ static void ne_pci_shutdown(struct pci_dev *pdev)
 	if (!ne_pci_dev)
 		return;
 
+	ne_cpu_pool_sysfs_exit();
+
 	down_write(&ne_devs.lock);
 
 	ne_pci_dev_detach_enclaves(ne_pci_dev);
-- 
2.47.1

[PATCH 04/40] nitro_enclaves: Document multi-NUMA CPU pools

[Alexander Graf]

An admin placing enclaves or sandboxes across NUMA nodes needs to know
how the CPU pool, automatic CPU selection and the memory-locality check
interact.

Document those rules in ne_overview.rst, together with the per-node
sysfs directories, and point the cpu_pool ABI file at that section.

Signed-off-by: Alexander Graf <graf@amazon.com>
---
 .../sysfs-devices-virtual-misc-nitro_enclaves |  3 ++-
 Documentation/virt/ne_overview.rst            | 27 ++++++++++++++++++-
 2 files changed, 28 insertions(+), 2 deletions(-)

diff --git a/Documentation/ABI/testing/sysfs-devices-virtual-misc-nitro_enclaves b/Documentation/ABI/testing/sysfs-devices-virtual-misc-nitro_enclaves
index 596054b75e0cf..c549364353186 100644
--- a/Documentation/ABI/testing/sysfs-devices-virtual-misc-nitro_enclaves
+++ b/Documentation/ABI/testing/sysfs-devices-virtual-misc-nitro_enclaves
@@ -39,7 +39,8 @@ Description:
 		avail does not list.
 
 		A read takes the pool lock, so it can wait for a write to
-		ne_cpus to finish.
+		ne_cpus to finish. For how an enclave is given cores from a
+		particular node, see Documentation/virt/ne_overview.rst.
 
 What:		/sys/devices/virtual/misc/nitro_enclaves/cpu_pool/nodeX/total
 What:		/sys/devices/virtual/misc/nitro_enclaves/cpu_pool/nodeX/used
diff --git a/Documentation/virt/ne_overview.rst b/Documentation/virt/ne_overview.rst
index aecd3704acaf5..223ee7fdec51b 100644
--- a/Documentation/virt/ne_overview.rst
+++ b/Documentation/virt/ne_overview.rst
@@ -48,7 +48,7 @@ be aligned 2 MiB / 1 GiB physically contiguous memory regions (or multiple of
 this size e.g. 8 MiB). The memory can be allocated e.g. by using hugetlbfs from
 user space [2][3][7]. The memory size for an enclave needs to be at least
 64 MiB. The enclave memory and CPUs need to be from the same NUMA node,
-unless the CPU pool spans several of them.
+unless the CPU pool spans several of them; see `Multi-NUMA CPU pools`_.
 
 An enclave runs on dedicated cores. CPU 0 and its CPU siblings need to remain
 available for the primary VM. A CPU pool has to be set for NE purposes by an
@@ -99,3 +99,28 @@ enclave process can exit.
 [6] https://www.kernel.org/doc/html/latest/x86/boot.html
 [7] https://www.kernel.org/doc/html/latest/arm64/hugetlbpage.html
 [8] https://www.kernel.org/doc/html/latest/arm64/booting.html
+
+Multi-NUMA CPU pools
+--------------------
+
+The CPU pool may span multiple NUMA nodes. NE_ADD_VCPU with vcpu_id == 0
+takes the first free core in the pool whatever node it sits on, so an
+enclave built that way can end up holding cores on several nodes.
+NE_ADD_ANY_VCPU names the node instead: the kernel takes a free core on
+that node, and fails rather than crossing to another node once that node
+has none left. User space builds an enclave across nodes by asking for
+each node in turn, and sets numa_node to NE_NUMA_ANY when it does not
+care. Both start from a free thread of a core the enclave already holds,
+and only take a new core once there is none.
+
+The node says where cores come from. An enclave built out of a pool that
+spans nodes has no node of its own, so the rule that its memory be on
+that node does not apply to it and its memory can come from any node.
+
+The pool occupancy is readable under
+/sys/devices/virtual/misc/nitro_enclaves/cpu_pool/, and per node under
+its ``node0``, ``node1``, ... subdirectories, one for each node that was
+online when the device was probed. A node the pool holds nothing on
+reads empty rather than being absent. The driver offlines pool threads,
+so ``nodeN/cpulist`` under /sys/devices/system/node stops listing them
+and answers nothing about the pool.
-- 
2.47.1

[PATCH 05/40] dma-contiguous: add dma_get_numa_cma_area() accessor

[Alexander Graf]

With Nitro Sandboxes, we want to allocate per-NUMA memory from the DMA
contiguous allocator, but have no per-NUMA struct device to pass in.
dma_alloc_contiguous() takes the node from dev_to_node(dev), and the
per-node areas sit in dma_contiguous_numa_area[], which is static.

So instead, introduce dma_get_numa_cma_area(), which returns the generic
per-NUMA DMA CMA area that cma_alloc() would use. This way, the
nitro_enclaves driver can build its own generic CMA pool per-node
allocation logic on top of those pools.

Signed-off-by: Alexander Graf <graf@amazon.com>
---
 include/linux/dma-map-ops.h |  7 +++++++
 kernel/dma/contiguous.c     | 36 ++++++++++++++++++++++++++++++++++++
 2 files changed, 43 insertions(+)

diff --git a/include/linux/dma-map-ops.h b/include/linux/dma-map-ops.h
index afaf2c12189b8..5f7138ea6899a 100644
--- a/include/linux/dma-map-ops.h
+++ b/include/linux/dma-map-ops.h
@@ -93,6 +93,7 @@ static inline void set_dma_ops(struct device *dev,
 #ifdef CONFIG_DMA_CMA
 struct cma *dev_get_cma_area(struct device *dev);
 struct cma *dma_contiguous_get_area_by_idx(unsigned int idx);
+struct cma *dma_get_numa_cma_area(int nid);
 
 void dma_contiguous_reserve(phys_addr_t addr_limit);
 int __init dma_contiguous_reserve_area(phys_addr_t size, phys_addr_t base,
@@ -115,6 +116,12 @@ static inline struct cma *dma_contiguous_get_area_by_idx(unsigned int idx)
 {
 	return NULL;
 }
+
+static inline struct cma *dma_get_numa_cma_area(int nid)
+{
+	return NULL;
+}
+
 static inline void dma_contiguous_reserve(phys_addr_t limit)
 {
 }
diff --git a/kernel/dma/contiguous.c b/kernel/dma/contiguous.c
index 90937dc7c627d..363b84284d087 100644
--- a/kernel/dma/contiguous.c
+++ b/kernel/dma/contiguous.c
@@ -443,6 +443,42 @@ struct page *dma_alloc_contiguous(struct device *dev, size_t size, gfp_t gfp)
 	return cma_alloc_aligned(dma_contiguous_default_area, size, gfp);
 }
 
+/**
+ * dma_get_numa_cma_area() - get the CMA area a NUMA node can allocate from
+ * @nid: NUMA node id
+ *
+ * dma_alloc_contiguous() picks a per-node CMA area from dev_to_node(dev),
+ * which a caller cannot override on a per-request basis. This accessor
+ * returns the area a specific node can allocate from, so a caller can
+ * cma_alloc() from the node it wants.
+ *
+ * A per-node area comes from numa_cma= or cma_pernuma=, or from
+ * CONFIG_CMA_SIZE_PERNUMA, which reserves the default area's size on every
+ * online node on top of the default area itself, all three of which are
+ * CONFIG_DMA_NUMA_CMA. Where there is one online node this returns the default
+ * area, so a host that reserved only cma= reaches its reservation through this
+ * accessor as well, and that holds however CONFIG_DMA_NUMA_CMA is set.
+ *
+ * Return: the area @nid can allocate from, or NULL if @nid has none or is not
+ * a valid node (including NUMA_NO_NODE).
+ */
+struct cma *dma_get_numa_cma_area(int nid)
+{
+	if ((unsigned int)nid >= MAX_NUMNODES)
+		return NULL;
+
+#ifdef CONFIG_DMA_NUMA_CMA
+	if (dma_contiguous_numa_area[nid])
+		return dma_contiguous_numa_area[nid];
+#endif
+
+	if (nr_online_nodes == 1 && node_online(nid))
+		return dma_contiguous_default_area;
+
+	return NULL;
+}
+EXPORT_SYMBOL_GPL(dma_get_numa_cma_area);
+
 /**
  * dma_free_contiguous() - release allocated pages
  * @dev:   Pointer to device for which the pages were allocated.
-- 
2.47.1

[PATCH 06/40] nitro_enclaves: Add NE_ALLOC_MEMORY ioctl

[Alexander Graf]

When donating memory to the hypervisor for Enclave or Sandbox creation,
the hypervisor only provides a limited number of slots of physically
contiguous memory regions that make up the full slot memory donation.

So far, we used hugetlbfs and mmap based memory donation. But that
scheme has a few issues. It means that user space retains access to
memory that is actually donated. Access to that memory turns into an
exception. It also means that any donating allocation ends up with
page tables that back them, even when nobody really should use them.

So we copy the guest_memfd flow of allocating memory, but not exposing
it as mapping to user space by default. EIF writes are a simple fd write.
Each allocation gets its own fd. And for direct mode, where memory really
is supposed to be accessible to the parent, we still allow mmap.

To reduce the number of separate memory regions these fds span, we use
CMA to allocate very large physically contiguous chunks.

Select CMA and DMA_CMA to always have the foundations available.
DMA_NUMA_CMA stays optional.

Signed-off-by: Alexander Graf <graf@amazon.com>
---
 drivers/virt/nitro_enclaves/Kconfig       |  18 +
 drivers/virt/nitro_enclaves/ne_misc_dev.c | 730 +++++++++++++++++++++-
 drivers/virt/nitro_enclaves/ne_misc_dev.h |  61 ++
 include/uapi/linux/nitro_enclaves.h       |  90 ++-
 4 files changed, 895 insertions(+), 4 deletions(-)

diff --git a/drivers/virt/nitro_enclaves/Kconfig b/drivers/virt/nitro_enclaves/Kconfig
index dc4d25c26256e..7b2456cdbb9fb 100644
--- a/drivers/virt/nitro_enclaves/Kconfig
+++ b/drivers/virt/nitro_enclaves/Kconfig
@@ -8,7 +8,9 @@
 config NITRO_ENCLAVES
 	tristate "Nitro Enclaves Support"
 	depends on (ARM64 || X86) && HOTPLUG_CPU && PCI && SMP
+	select CMA
+	select DMA_CMA
 	help
 	  This driver consists of support for enclave lifetime management
 	  for Nitro Enclaves (NE).
 
diff --git a/drivers/virt/nitro_enclaves/ne_misc_dev.c b/drivers/virt/nitro_enclaves/ne_misc_dev.c
index 66bcaec8d3199..56ca1a9d99d41 100644
--- a/drivers/virt/nitro_enclaves/ne_misc_dev.c
+++ b/drivers/virt/nitro_enclaves/ne_misc_dev.c
@@ -9,12 +9,17 @@
  */
 
 #include <linux/anon_inodes.h>
+#include <linux/bsearch.h>
 #include <linux/build_bug.h>
 #include <linux/capability.h>
+#include <linux/cma.h>
 #include <linux/cpu.h>
 #include <linux/cpumask.h>
 #include <linux/device.h>
+#include <linux/dma-map-ops.h>
 #include <linux/file.h>
+#include <linux/fs.h>
+#include <linux/highmem.h>
 #include <linux/hugetlb.h>
 #include <linux/ioctl.h>
 #include <linux/kernel.h>
@@ -33,13 +38,16 @@
 #include <linux/numa.h>
 #include <linux/pci.h>
 #include <linux/poll.h>
+#include <linux/preempt.h>
 #include <linux/range.h>
+#include <linux/sched/mm.h>
 #include <linux/slab.h>
 #include <linux/stddef.h>
 #include <linux/string.h>
 #include <linux/sysfs.h>
 #include <linux/types.h>
 #include <linux/uaccess.h>
+#include <linux/uio.h>
 #include <uapi/linux/vm_sockets.h>
 
 #include "ne_misc_dev.h"
@@ -86,6 +94,16 @@ static_assert(NE_NUMA_ANY == NUMA_NO_NODE);
  */
 #define NE_MIN_MEM_REGION_SIZE	(2 * 1024UL * 1024UL)
 
+/**
+ * NE_ZERO_SPAN_PAGES - The most one span of the zeroing walk covers.
+ */
+#define NE_ZERO_SPAN_PAGES	(1024UL)
+
+/**
+ * NE_WRITE_SPAN_PAGES - The most one span of the write walk copies.
+ */
+#define NE_WRITE_SPAN_PAGES	(1024UL)
+
 /**
  * NE_PARENT_VM_CID - The CID for the vsock device of the primary / parent VM.
  */
@@ -93,6 +111,16 @@ static_assert(NE_NUMA_ANY == NUMA_NO_NODE);
 
 static long ne_ioctl(struct file *file, unsigned int cmd, unsigned long arg);
 
+/*
+ * Handed to an enclave as ne_enclave.id once it has a slot, so that an
+ * NE_ALLOC_MEMORY allocation names the enclave it was made for without holding
+ * a reference on it. A 64-bit counter does not come round again, so an id names
+ * one enclave for the lifetime of the driver.
+ */
+static atomic64_t ne_enclave_seq = ATOMIC64_INIT(0);
+
+static int ne_alloc_memory_ioctl(struct ne_enclave *ne_enclave,
+				 struct ne_alloc_memory_args *args);
 static const struct file_operations ne_fops = {
 	.owner		= THIS_MODULE,
 	.llseek		= noop_llseek,
@@ -1731,7 +1759,7 @@ static int ne_start_enclave_ioctl(struct ne_enclave *ne_enclave,
  * Context: Process context. This function is called with ne_devs.lock held
  *	    for reading, on an enclave that is not detached.
  * Return:
- * * 0 on success.
+ * * Ioctl result (e.g. memory allocation file descriptor) on success.
  * * Negative return value on failure.
  */
 static long ne_enclave_do_ioctl(struct file *file, unsigned int cmd, unsigned long arg)
@@ -2041,6 +2069,39 @@ static long ne_enclave_do_ioctl(struct file *file, unsigned int cmd, unsigned lo
 		return 0;
 	}
 
+	case NE_EA_IOCTL(NE_ALLOC_MEMORY): {
+		struct ne_alloc_memory_args __user *uargs = (void __user *)arg;
+		unsigned int usize = _IOC_SIZE(cmd);
+		struct ne_alloc_memory_args args;
+		int rc;
+
+		if (usize < offsetofend(struct ne_alloc_memory_args, flags))
+			return -EINVAL;
+
+		rc = copy_struct_from_user(&args, sizeof(args), uargs, usize);
+		if (rc)
+			return rc;
+
+		rc = mutex_lock_killable(&ne_enclave->enclave_info_mutex);
+		if (rc)
+			return rc;
+
+		if (ne_enclave->state != NE_STATE_INIT) {
+			dev_err_ratelimited(ne_misc_dev.this_device,
+					    "Enclave is not in init state\n");
+
+			mutex_unlock(&ne_enclave->enclave_info_mutex);
+
+			return -NE_ERR_NOT_IN_INIT_STATE;
+		}
+
+		rc = ne_alloc_memory_ioctl(ne_enclave, &args);
+
+		mutex_unlock(&ne_enclave->enclave_info_mutex);
+
+		return rc;
+	}
+
 	default:
 		return -ENOTTY;
 	}
@@ -2427,6 +2488,8 @@ static int ne_create_vm_ioctl(struct ne_pci_dev *ne_pci_dev, u64 __user *slot_ui
 		goto put_file;
 	}
 
+	ne_enclave->id = atomic64_inc_return(&ne_enclave_seq);
+
 	init_waitqueue_head(&ne_enclave->eventq);
 	ne_enclave->has_event = false;
 	mutex_init(&ne_enclave->enclave_info_mutex);
@@ -2516,6 +2579,671 @@ static long ne_ioctl(struct file *file, unsigned int cmd, unsigned long arg)
 	return 0;
 }
 
+/*
+ * Physically-contiguous memory for enclaves.
+ *
+ * An enclave's memory is handed to the hypervisor in physically contiguous
+ * 2 MiB (or larger) chunks. NE_ALLOC_MEMORY allocates contiguous memory from a
+ * host NUMA node's generic CMA area and returns a file descriptor for it. A
+ * per-node area comes from numa_cma= or cma_pernuma=, or from
+ * CONFIG_CMA_SIZE_PERNUMA, which reserves the cma= default area's size on
+ * every online node on top of that area itself, none of which a kernel has
+ * without CONFIG_DMA_NUMA_CMA. On a host with one online node the accessor
+ * returns the default area, so cma= on its own is enough there, and that much
+ * needs CONFIG_DMA_CMA alone.
+ *
+ * A request the node's area cannot serve fails. Anyone who can open
+ * /dev/nitro_enclaves can create an enclave and ask it for memory, so
+ * serving such a request out of movable memory would take that memory from
+ * the rest of the host and leave the reservation bounding nothing.
+ *
+ * A request larger than one contiguous run is satisfied by several runs, which
+ * the driver keeps on a list of its own so that an enclave can be given every
+ * one of them.
+ *
+ * An allocation is never mapped, and the enclave image reaches it through
+ * write().
+ *
+ * The runs belong to the file the allocation is handed out on. Zeroing a fresh
+ * run and a write into an allocation go through ne_walk_run(), so that neither
+ * can run for as long as the allocation is large; the release path walks the
+ * runs with a preemption point of its own.
+ */
+
+/**
+ * ne_contig_free_run() - Give a contiguous run of pages back.
+ * @cma:	CMA area the run came from.
+ * @page:	First page of the run.
+ * @nr_pages:	Number of pages in the run.
+ */
+void ne_contig_free_run(struct cma *cma, struct page *page,
+			unsigned long nr_pages)
+{
+	cma_release(cma, page, nr_pages);
+}
+
+static void ne_contig_free_alloc(struct ne_contig_alloc *alloc)
+{
+	ne_contig_free_run(alloc->cma, alloc->page, alloc->nr_pages);
+}
+
+/*
+ * Free every run of an allocation. A 512 GiB allocation made of 2 MiB runs is a
+ * quarter of a million entries and cma_release() does not reschedule, so this
+ * walk needs a preemption point of its own. It runs from ->release, which can
+ * land on a workqueue.
+ */
+static void ne_contig_free_allocs(struct ne_contig_mem *mem)
+{
+	unsigned int i;
+
+	for (i = 0; i < mem->nr_allocs; i++) {
+		cond_resched();
+
+		ne_contig_free_alloc(&mem->allocs[i]);
+	}
+
+	mem->nr_allocs = 0;
+	kvfree(mem->allocs);
+	mem->allocs = NULL;
+}
+
+/**
+ * typedef ne_span_fn - What to do with one span of a contiguous run.
+ * @arg:	Context the caller of ne_walk_run() passed.
+ * @page:	Page the span starts in.
+ * @off:	Byte offset of the span inside @page, less than PAGE_SIZE.
+ * @len:	Length of the span in bytes.
+ *
+ * Return: 0 on success, negative return value on failure.
+ */
+typedef int (*ne_span_fn)(void *arg, struct page *page, u64 off, u64 len);
+
+/**
+ * ne_walk_run() - Hand @fn one span of a run at a time.
+ * @page:	First page of the run.
+ * @off:	Byte offset into the run to start at.
+ * @len:	Number of bytes to cover.
+ * @span_pages:	The most one call to @fn covers.
+ * @fn:		What to do with a span.
+ * @arg:	Context for @fn.
+ *
+ * A run can be as large as the whole allocation and none of the things done to
+ * one reschedules on its own, so this is where the driver gets its
+ * preemption points and its bail-out on a fatal signal. @fn gets at most
+ * @span_pages worth of bytes at a time and nothing else bounds it, which is
+ * what keeps the work between two cond_resched() calls from growing with the
+ * allocation. An unaligned @off spreads those bytes over one page more.
+ *
+ * Return: 0 on success, -EINTR when a fatal signal arrived, or whatever @fn
+ *	   returned.
+ */
+static int ne_walk_run(struct page *page, u64 off, u64 len,
+		       unsigned long span_pages, ne_span_fn fn, void *arg)
+{
+	u64 span = (u64)span_pages << PAGE_SHIFT;
+	u64 done = 0;
+
+	while (done < len) {
+		u64 pos = off + done;
+		u64 this = min(len - done, span);
+		int rc;
+
+		rc = fn(arg, page + (pos >> PAGE_SHIFT), pos & ~PAGE_MASK,
+			this);
+		if (rc)
+			return rc;
+
+		done += this;
+
+		if (fatal_signal_pending(current))
+			return -EINTR;
+
+		cond_resched();
+	}
+
+	return 0;
+}
+
+/**
+ * ne_contig_alloc_cmp() - Order a byte offset against one run.
+ * @key:	Byte offset into the allocation, as a u64.
+ * @elem:	The run.
+ *
+ * Return: 0 when @key falls inside @elem, negative when it lies before it and
+ *	   positive when it lies after it.
+ */
+static int ne_contig_alloc_cmp(const void *key, const void *elem)
+{
+	const struct ne_contig_alloc *alloc = elem;
+	u64 off = *(const u64 *)key;
+
+	if (off < alloc->off)
+		return -1;
+
+	if (off >= alloc->off + ((u64)alloc->nr_pages << PAGE_SHIFT))
+		return 1;
+
+	return 0;
+}
+
+/**
+ * ne_find_alloc() - The run a byte offset falls in.
+ * @mem:	The allocation.
+ * @off:	Byte offset into the allocation.
+ *
+ * The runs are ordered by offset and cover the allocation back to back, so a
+ * binary search finds one. A 512 GiB allocation made of 2 MiB runs is a quarter
+ * of a million of them and a fault resolves an offset every time it is taken,
+ * which is why this is not a walk. The array is not mutated after the file is
+ * published, so this needs no lock.
+ *
+ * Return: the run, or NULL when @off is past the end of the allocation.
+ */
+static struct ne_contig_alloc *ne_find_alloc(struct ne_contig_mem *mem, u64 off)
+{
+	return bsearch(&off, mem->allocs, mem->nr_allocs, sizeof(*mem->allocs),
+		       ne_contig_alloc_cmp);
+}
+
+/**
+ * ne_walk_allocs() - Hand @fn the part of an allocation at @off, run by run.
+ * @mem:	The allocation.
+ * @off:	Byte offset into the allocation to start at.
+ * @len:	Number of bytes to cover, which has to fit inside the allocation.
+ * @span_pages:	The most one call to @fn covers.
+ * @fn:		What to do with a span.
+ * @arg:	Context for @fn.
+ *
+ * The runs are handed over in order and back to back, so a caller that only
+ * advances an output cursor sees one flat region.
+ *
+ * Return: 0 on success, negative return value on failure.
+ */
+static int ne_walk_allocs(struct ne_contig_mem *mem, u64 off, u64 len,
+			  unsigned long span_pages, ne_span_fn fn, void *arg)
+{
+	struct ne_contig_alloc *end = mem->allocs + mem->nr_allocs;
+	struct ne_contig_alloc *alloc = ne_find_alloc(mem, off);
+	u64 done = 0;
+
+	if (!alloc)
+		return 0;
+
+	for (; done < len && alloc < end; alloc++) {
+		u64 run = (u64)alloc->nr_pages << PAGE_SHIFT;
+		u64 skip = off > alloc->off ? off - alloc->off : 0;
+		u64 this = min(run - skip, len - done);
+		int rc;
+
+		cond_resched();
+
+		rc = ne_walk_run(alloc->page, skip, this, span_pages, fn, arg);
+		if (rc)
+			return rc;
+
+		done += this;
+	}
+
+	return 0;
+}
+
+static struct cma *ne_numa_cma_area(int nid)
+{
+	struct cma *cma = dma_get_numa_cma_area(nid);
+
+	/*
+	 * A node with no area of its own can serve nothing, however much free
+	 * memory it has, and that is a reservation the host was booted without
+	 * rather than one it has used up. Say so once, since the areas are
+	 * reserved at boot and no node acquires one later.
+	 */
+	if (!cma)
+		pr_warn_once("%s: node %d has no CMA area to serve enclave memory from, reserve one with numa_cma=, cma_pernuma= or cma=\n",
+			     ne_misc_dev.name, nid);
+
+	return cma;
+}
+
+static struct page *ne_alloc_contig_node(unsigned long nr_pages, int nid,
+					 struct cma **out_cma)
+{
+	unsigned int align = ilog2(NE_MIN_MEM_REGION_SIZE >> PAGE_SHIFT);
+	struct cma *cma = ne_numa_cma_area(nid);
+	struct page *page;
+
+	if (!cma)
+		return NULL;
+
+	/*
+	 * The chunk loop in ne_alloc_memory_ioctl() walks sizes downwards and
+	 * expects most of these to fail, so keep a failure cheap and quiet.
+	 */
+	page = cma_alloc(cma, nr_pages, align, true);
+	if (page)
+		*out_cma = cma;
+
+	return page;
+}
+
+/*
+ * Zero one span of a freshly allocated run. CMA pages can carry residual data
+ * from earlier movable use, so nothing reaches an enclave until this has run
+ * over all of it.
+ *
+ * The run is on its way to an enclave and may be mapped into the caller, so this
+ * is clear_user_highpages()' job rather than clear_pages()'. It maps a highmem
+ * page before clearing it and falls back to a batched clear_pages() where there
+ * is no highmem, which is every configuration this driver builds for bar i386
+ * with highmem.
+ * The user address it is given is zero. Neither x86 nor arm64 supplies its own
+ * clear_user_page() or clear_user_highpage(), so the generic clear_user_page()
+ * runs and does not read the address.
+ *
+ * @off is zero and @len a multiple of PAGE_SIZE here, because a run is walked
+ * whole and the span size is a page count.
+ */
+static int ne_zero_span(void *arg, struct page *page, u64 off, u64 len)
+{
+	clear_user_highpages(page, 0, len >> PAGE_SHIFT);
+
+	return 0;
+}
+
+/*
+ * How much of a run one span of the zeroing walk covers. clear_user_highpages()
+ * does not reschedule, so under a cooperatively scheduled model a caller has to
+ * keep a batch within PROCESS_PAGES_NON_PREEMPT_BATCH, which is a single page on
+ * every architecture that does not supply its own clear_pages(). Under
+ * PREEMPT_DYNAMIC the model comes from the command line, so this asks at run
+ * time rather than testing a config symbol. Either way NE_ZERO_SPAN_PAGES stays
+ * the upper bound, so a fatal signal is still noticed within a few megabytes of
+ * clearing.
+ */
+static unsigned long ne_zero_span_pages(void)
+{
+	if (preempt_model_preemptible())
+		return NE_ZERO_SPAN_PAGES;
+
+	return min_t(unsigned long, NE_ZERO_SPAN_PAGES,
+		     PROCESS_PAGES_NON_PREEMPT_BATCH);
+}
+
+/**
+ * ne_alloc_contig() - Allocate a physically-contiguous, zeroed run of pages.
+ * @nr_pages:	Number of pages to allocate.
+ * @nid:	NUMA node to allocate from.
+ * @out_cma:	Set to the CMA area the run came from.
+ *
+ * The run is drawn from @nid's generic CMA area and from nowhere else: an
+ * enclave's memory has to sit on the node the caller asked for, so a run from
+ * another node is no more use to it than no run at all, and a run from outside
+ * the reservation is memory the host budgeted for something else.
+ *
+ * Zeroing a large run takes a while, so a fatal signal abandons it rather than
+ * making the task wait to be killed.
+ *
+ * Return: the first page of the run, NULL on allocation failure, or
+ *	   ERR_PTR(-EINTR) when a fatal signal arrived while zeroing.
+ */
+struct page *ne_alloc_contig(unsigned long nr_pages, int nid,
+			     struct cma **out_cma)
+{
+	struct page *page;
+	int rc;
+
+	*out_cma = NULL;
+
+	page = ne_alloc_contig_node(nr_pages, nid, out_cma);
+	if (!page)
+		return NULL;
+
+	rc = ne_walk_run(page, 0, (u64)nr_pages << PAGE_SHIFT,
+			 ne_zero_span_pages(), ne_zero_span, NULL);
+	if (rc) {
+		struct ne_contig_alloc discard = {
+			.page = page,
+			.nr_pages = nr_pages,
+			.cma = *out_cma,
+		};
+
+		ne_contig_free_alloc(&discard);
+
+		return ERR_PTR(rc);
+	}
+
+	return page;
+}
+
+/**
+ * struct ne_write_ctx - Where the write walk has got to.
+ * @iter:	Where the bytes come from.
+ * @copied:	Bytes copied so far.
+ */
+struct ne_write_ctx {
+	struct iov_iter	*iter;
+	size_t		copied;
+};
+
+/*
+ * Copy one span of a run out of the iterator. Where the kernel has no highmem a
+ * run is contiguous in the direct map, so a span goes in one call. Where it may
+ * have some, page_address() is not enough on its own, so every page goes through
+ * copy_page_from_iter(), which maps it for the copy.
+ */
+static int ne_write_span(void *arg, struct page *page, u64 off, u64 len)
+{
+	struct ne_write_ctx *ctx = arg;
+	size_t n = 0;
+
+	if (!IS_ENABLED(CONFIG_HIGHMEM)) {
+		n = copy_from_iter(page_address(page) + off, len, ctx->iter);
+	} else {
+		while (n < len) {
+			u64 pos = off + n;
+			size_t this;
+
+			this = min_t(size_t, len - n,
+				     PAGE_SIZE - (pos & ~PAGE_MASK));
+			this = copy_page_from_iter(page + (pos >> PAGE_SHIFT),
+						   pos & ~PAGE_MASK, this,
+						   ctx->iter);
+			if (!this)
+				break;
+
+			n += this;
+		}
+	}
+
+	ctx->copied += n;
+
+	return n < len ? -EFAULT : 0;
+}
+
+/**
+ * ne_contig_write_iter() - Place bytes in an allocation.
+ * @iocb:	Holds the allocation's file and the offset to write at.
+ * @from:	Where the bytes come from.
+ *
+ * An enclave image has to be in enclave memory before the enclave starts, and a
+ * private allocation is never mapped, so this is how it gets there. The write
+ * has to fit inside the allocation: a partly written image is of no use to
+ * anyone, so a write running past the end is refused rather than shortened. The
+ * allocation has a fixed maximum offset and no notion of free space, so that
+ * refusal is -EFBIG, the same as generic_write_check_limits() gives for a write
+ * beyond s_maxbytes. There is no RLIMIT_FSIZE in it, so no SIGXFSZ either.
+ *
+ * Return: the number of bytes written, or a negative return value on failure.
+ */
+static ssize_t ne_contig_write_iter(struct kiocb *iocb, struct iov_iter *from)
+{
+	struct ne_contig_mem *mem = iocb->ki_filp->private_data;
+	struct ne_write_ctx ctx = { .iter = from };
+	size_t count = iov_iter_count(from);
+	u64 pos = iocb->ki_pos;
+	int rc;
+
+	if (!count)
+		return 0;
+
+	if (pos >= mem->size || count > mem->size - pos)
+		return -EFBIG;
+
+	rc = ne_walk_allocs(mem, pos, count, NE_WRITE_SPAN_PAGES, ne_write_span,
+			    &ctx);
+
+	if (ctx.copied) {
+		iocb->ki_pos = pos + ctx.copied;
+
+		return ctx.copied;
+	}
+
+	return rc;
+}
+
+/**
+ * ne_contig_release() - Free an allocation once nothing holds it any more.
+ * @inode:	The allocation's inode, unused.
+ * @file:	The allocation's file.
+ *
+ * The last reference on the file is what gets here, and a mapping holds one, so
+ * by this point no page table entry points at the runs.
+ *
+ * Return: 0.
+ */
+static int ne_contig_release(struct inode *inode, struct file *file)
+{
+	struct ne_contig_mem *mem = file->private_data;
+
+	ne_contig_free_allocs(mem);
+
+	if (mem->nr_pinned)
+		atomic64_sub(mem->nr_pinned, &mem->mm->pinned_vm);
+
+	mmdrop(mem->mm);
+
+	kfree(mem);
+
+	return 0;
+}
+
+static const struct file_operations ne_contig_fops = {
+	.owner			= THIS_MODULE,
+	.llseek			= generic_file_llseek,
+	.write_iter		= ne_contig_write_iter,
+	.release		= ne_contig_release,
+};
+
+/**
+ * ne_contig_create_file() - Build a file for an allocation.
+ * @mem:	The allocation, whose runs the file takes over on success.
+ *
+ * On success the file owns @mem, and the last fput() runs ne_contig_release()
+ * to free the runs. On failure @mem is left to the caller.
+ *
+ * Return: the new file, or an ERR_PTR.
+ */
+static struct file *ne_contig_create_file(struct ne_contig_mem *mem)
+{
+	struct file *file;
+
+	/*
+	 * An inode of its own is what carries the size, so that fstat() reports
+	 * it and it bounds the offsets a mapping or a write may use. It also
+	 * gets a security label rather than staying S_PRIVATE, which is what
+	 * user space reaching an inode through a descriptor wants.
+	 */
+	file = anon_inode_create_getfile("ne-mem", &ne_contig_fops, mem, O_RDWR,
+					 NULL);
+	if (IS_ERR(file))
+		return file;
+
+	i_size_write(file_inode(file), mem->size);
+	inode_set_bytes(file_inode(file), mem->size);
+
+	/*
+	 * A file that came through open() would have these from
+	 * do_dentry_open(); one built here does not, and FMODE_PWRITE is what
+	 * lets a positional write reach an allocation that is never mapped.
+	 * FMODE_PREAD gates pread() rather than read(), and there is no read
+	 * operation here for either to reach, so all it buys is pread()
+	 * failing with EINVAL like read() does instead of with ESPIPE.
+	 * Nothing else has this file yet, so there is nobody to see the
+	 * intermediate state.
+	 */
+	file->f_mode |= FMODE_PREAD | FMODE_PWRITE;
+	file->f_flags |= O_LARGEFILE;
+
+	return file;
+}
+
+/**
+ * ne_alloc_memory_ioctl() - Allocate contiguous memory for an enclave.
+ * @ne_enclave:	Enclave the allocation is made for.
+ * @args:	Requested size, node and flags.
+ *
+ * Context: Process context. This function is called with the enclave
+ *	    information mutex held.
+ * Return: a file descriptor on success, negative return value on failure.
+ */
+static int ne_alloc_memory_ioctl(struct ne_enclave *ne_enclave,
+				 struct ne_alloc_memory_args *args)
+{
+	struct ne_contig_mem *mem;
+	unsigned long remaining;
+	u64 pinned, lock_limit;
+	unsigned long nr_pages;
+	struct file *file;
+	int fd, rc;
+
+	if (ne_enclave->mm != current->mm)
+		return -EIO;
+
+	if (args->flags >= NE_ALLOC_MEMORY_MAX_FLAG_VAL)
+		return -NE_ERR_INVALID_FLAG_VALUE;
+
+	if (!args->size || (args->size & (NE_MIN_MEM_REGION_SIZE - 1)))
+		return -EINVAL;
+
+	if (args->size > ULONG_MAX)
+		return -EINVAL;
+
+	if (args->nid >= MAX_NUMNODES || !node_state(args->nid, N_MEMORY))
+		return -EINVAL;
+
+	/*
+	 * An enclave whose cores all sit on one node takes its memory from that
+	 * node, which is what ne_sanity_check_user_mem_region() asks of a
+	 * hugetlbfs region. An enclave built from a CPU pool that spans nodes
+	 * has no node of its own and may take memory from any of them. Refuse
+	 * here, so that a request naming the wrong node does not spend minutes
+	 * zeroing first.
+	 */
+	if (ne_enclave->numa_node != NUMA_NO_NODE &&
+	    ne_enclave->numa_node != (int)args->nid)
+		return -NE_ERR_MEM_DIFFERENT_NUMA_NODE;
+
+	/*
+	 * A node without an area serves nothing, and no area serves more than
+	 * the host has, so refuse those requests before they are charged and
+	 * before the run array is sized from them.
+	 */
+	if (!ne_numa_cma_area(args->nid) ||
+	    args->size > (u64)totalram_pages() << PAGE_SHIFT)
+		return -ENOMEM;
+
+	nr_pages = args->size >> PAGE_SHIFT;
+
+	/*
+	 * Account the pages against RLIMIT_MEMLOCK, the way a driver that pins
+	 * pages for a device accounts them. pinned_vm is an atomic64, so it can
+	 * be uncharged from ne_contig_release() under whatever locks that path
+	 * happens to hold.
+	 */
+	lock_limit = rlimit(RLIMIT_MEMLOCK) >> PAGE_SHIFT;
+	pinned = atomic64_add_return(nr_pages, &current->mm->pinned_vm);
+	if (pinned > lock_limit && !capable(CAP_IPC_LOCK)) {
+		atomic64_sub(nr_pages, &current->mm->pinned_vm);
+
+		return -ENOMEM;
+	}
+
+	mem = kzalloc_obj(*mem);
+	if (!mem) {
+		atomic64_sub(nr_pages, &current->mm->pinned_vm);
+
+		return -ENOMEM;
+	}
+
+	/*
+	 * A run is at least NE_MIN_MEM_REGION_SIZE, so that bounds how many of
+	 * them an allocation can be made of and the array can be sized for the
+	 * worst case here. At 512 GiB of 2 MiB runs it is 8 MiB, next to nothing
+	 * beside the memory about to be allocated, and it keeps the allocation
+	 * loop from having to grow anything.
+	 */
+	mem->allocs = kvmalloc_array(args->size / NE_MIN_MEM_REGION_SIZE,
+				     sizeof(*mem->allocs),
+				     GFP_KERNEL | __GFP_NOWARN);
+	if (!mem->allocs) {
+		atomic64_sub(nr_pages, &current->mm->pinned_vm);
+		kfree(mem);
+
+		return -ENOMEM;
+	}
+
+	mmgrab(current->mm);
+	mem->mm = current->mm;
+	mem->nr_pinned = nr_pages;
+	mem->size = args->size;
+	mem->enclave_id = ne_enclave->id;
+
+	/*
+	 * Take the runs largest power-of-two chunk first, down to the 2 MiB
+	 * minimum. cma_alloc() is given the 2 MiB alignment the hypervisor asks
+	 * for whatever the size, so every run meets it.
+	 */
+	remaining = args->size;
+	while (remaining) {
+		struct ne_contig_alloc *alloc;
+		struct cma *cma = NULL;
+		struct page *page = NULL;
+		unsigned long chunk;
+
+		for (chunk = remaining; chunk >= NE_MIN_MEM_REGION_SIZE;
+		     chunk >>= 1) {
+			chunk = rounddown_pow_of_two(chunk);
+			page = ne_alloc_contig(chunk >> PAGE_SHIFT, args->nid,
+					       &cma);
+			if (IS_ERR(page)) {
+				rc = PTR_ERR(page);
+				goto err_free;
+			}
+			if (page)
+				break;
+		}
+		if (!page) {
+			rc = -ENOMEM;
+			goto err_free;
+		}
+
+		alloc = &mem->allocs[mem->nr_allocs++];
+		alloc->page = page;
+		alloc->nr_pages = chunk >> PAGE_SHIFT;
+		alloc->cma = cma;
+		alloc->off = args->size - remaining;
+
+		remaining -= chunk;
+		cond_resched();
+	}
+
+	fd = get_unused_fd_flags(O_CLOEXEC);
+	if (fd < 0) {
+		rc = fd;
+		goto err_free;
+	}
+
+	file = ne_contig_create_file(mem);
+	if (IS_ERR(file)) {
+		rc = PTR_ERR(file);
+		goto err_put_fd;
+	}
+
+	fd_install(fd, file);
+
+	return fd;
+
+err_put_fd:
+	put_unused_fd(fd);
+err_free:
+	ne_contig_free_allocs(mem);
+	atomic64_sub(nr_pages, &mem->mm->pinned_vm);
+	mmdrop(mem->mm);
+	kfree(mem);
+
+	return rc;
+}
+
 #if defined(CONFIG_NITRO_ENCLAVES_MISC_DEV_TEST)
 #include "ne_misc_dev_test.c"
 #endif
diff --git a/drivers/virt/nitro_enclaves/ne_misc_dev.h b/drivers/virt/nitro_enclaves/ne_misc_dev.h
index 483037061eafe..1a76c12815298 100644
--- a/drivers/virt/nitro_enclaves/ne_misc_dev.h
+++ b/drivers/virt/nitro_enclaves/ne_misc_dev.h
@@ -17,6 +17,48 @@
 
 #include "ne_pci_dev.h"
 
+struct cma;
+
+/**
+ * struct ne_contig_alloc - One physically-contiguous run backing a mapping.
+ * @page:	First page of the run.
+ * @nr_pages:	Number of pages in the run.
+ * @off:	Byte offset of the run inside the allocation.
+ * @cma:	CMA area the run came from.
+ */
+struct ne_contig_alloc {
+	struct page		*page;
+	unsigned long		nr_pages;
+	u64			off;
+	struct cma		*cma;
+};
+
+/**
+ * struct ne_contig_mem - Backing state for one contiguous allocation.
+ * @allocs:	Contiguous runs backing the allocation, ordered by offset.
+ * @nr_allocs:	Number of runs in @allocs.
+ * @size:	Size of the allocation in bytes, mirrored in i_size.
+ * @enclave_id:	Id of the enclave the allocation was made for.
+ * @mm:		The mm charged for the pinned pages, held via mmgrab().
+ * @nr_pinned:	Pages charged to @mm->pinned_vm.
+ *
+ * The runs hang off the file, and a mapping's page table entries cannot outlive
+ * them because a VMA holds vm_file for as long as it exists.
+ *
+ * The array is built before the file is published and not mutated afterwards, so
+ * any holder of a reference can read it without a lock of its own, which is what
+ * lets a page fault resolve an offset to a run. @size is set in the same place
+ * and is read the same way.
+ */
+struct ne_contig_mem {
+	struct ne_contig_alloc	*allocs;
+	unsigned int		nr_allocs;
+	u64			size;
+	u64			enclave_id;
+	struct mm_struct	*mm;
+	unsigned long		nr_pinned;
+};
+
 /**
  * struct ne_mem_region - Entry in the enclave user space memory regions list.
  * @mem_region_list_entry:	Entry in the list of enclave memory regions.
@@ -44,6 +86,9 @@ struct ne_mem_region {
  *				the enclave process via the poll function.
  * @has_event:			Variable used to determine if the out-of-band event
  *				was triggered.
+ * @id:				Id of this enclave, unique for the lifetime of the
+ *				driver, recorded in an %NE_ALLOC_MEMORY allocation
+ *				as the enclave it was made for.
  * @max_mem_regions:		The maximum number of memory regions that can be
  *				handled by the hypervisor.
  * @mem_regions_list:		Enclave user space memory regions list.
@@ -72,6 +117,7 @@ struct ne_enclave {
 	struct list_head	enclave_list_entry;
 	wait_queue_head_t	eventq;
 	bool			has_event;
+	u64			id;
 	u64			max_mem_regions;
 	struct list_head	mem_regions_list;
 	u64			mem_size;
@@ -112,6 +158,21 @@ struct ne_devs {
 	struct rw_semaphore	lock;
 };
 
+/**
+ * ne_alloc_contig() - Allocate a physically-contiguous, zeroed run of pages.
+ * @nr_pages:	Number of pages to allocate.
+ * @nid:	NUMA node to allocate from.
+ * @out_cma:	Set to the CMA area the run came from.
+ *
+ * Return: the first page of the run, NULL on allocation failure, or
+ *	   ERR_PTR(-EINTR) when a fatal signal arrived while zeroing.
+ */
+struct page *ne_alloc_contig(unsigned long nr_pages, int nid,
+			     struct cma **out_cma);
+
+void ne_contig_free_run(struct cma *cma, struct page *page,
+			unsigned long nr_pages);
+
 /* Nitro Enclaves (NE) data structure for keeping refs to the NE misc and PCI devices. */
 extern struct ne_devs ne_devs;
 
diff --git a/include/uapi/linux/nitro_enclaves.h b/include/uapi/linux/nitro_enclaves.h
index e266b8a611c92..a2919565bc883 100644
--- a/include/uapi/linux/nitro_enclaves.h
+++ b/include/uapi/linux/nitro_enclaves.h
@@ -209,6 +209,76 @@ struct ne_add_any_vcpu_args {
  */
 #define NE_SET_USER_MEMORY_REGION	_IOW(0xAE, 0x23, struct ne_user_memory_region)
 
+/**
+ * struct ne_alloc_memory_args - Request for physically contiguous enclave
+ *				 memory (in).
+ * @size:	Number of bytes to allocate, a multiple of 2 MiB.
+ * @nid:	Host NUMA node to allocate from. One node, named; unlike
+ *		%NE_ADD_ANY_VCPU it does not take %NE_NUMA_ANY.
+ * @flags:	%NE_MEMORY_PRIVATE.
+ */
+struct ne_alloc_memory_args {
+	__u64	size;
+	__u32	nid;
+	__u32	flags;
+};
+
+/**
+ * NE_ALLOC_MEMORY - The command is used to allocate physically contiguous
+ *		     memory to give to an enclave, drawn from one host NUMA
+ *		     node. The argument is input only; on success the return
+ *		     value is a file descriptor for the allocation.
+ *		     Writing to the descriptor places bytes in the allocation,
+ *		     which is how an enclave image gets there.
+ *		     O_APPEND has no effect. There is no read operation, so
+ *		     read() and pread() both fail with EINVAL and a caller that
+ *		     wants to check what it wrote keeps its own copy.
+ *		     A private allocation is never mapped, so an enclave image
+ *		     goes into one through write() or pwrite() and not through
+ *		     a copy into a mapping, and every write is a call the
+ *		     kernel handles and can act on or refuse.
+ *		     The allocation lives until the descriptor is gone.
+ *		     The ioctl can be invoked on the enclave fd, before an
+ *		     enclave is started.
+ *
+ * Context: Process context.
+ * Return:
+ * * >= 0			- File descriptor for the allocation.
+ * * -1				- There was a failure in the ioctl logic.
+ * On failure, errno is set to:
+ * * EFAULT			- Argument read failure.
+ * * EINVAL			- The argument is shorter than @flags, or @size
+ *				  is zero, not a multiple of 2 MiB or larger than
+ *				  this kernel can represent, or @nid is not a node
+ *				  with memory.
+ * * EIO			- Current task mm is not the same as the one
+ *				  that created the enclave.
+ * * E2BIG			- The argument is longer than this kernel knows
+ *				  about and the excess is not all zero.
+ * * EINTR			- A fatal signal arrived while zeroing.
+ * * ENODEV			- The kernel has no contiguous allocator.
+ * * ENOMEM			- No contiguous run of the requested size is
+ *				  available on that node, or %RLIMIT_MEMLOCK
+ *				  would be exceeded.
+ * * NE_ERR_NOT_IN_INIT_STATE	- The enclave is not in init state (init =
+ *				  before being started).
+ * * NE_ERR_INVALID_FLAG_VALUE	- The value of the provided flag is invalid.
+ * * NE_ERR_MEM_DIFFERENT_NUMA_NODE
+ *				- @nid is not the node the enclave's CPUs are on.
+ *				  Not returned for an enclave whose CPU pool spans
+ *				  nodes.
+ * * Error codes from get_unused_fd_flags() and anon_inode_create_getfile().
+ *
+ * Writing to the descriptor sets errno to:
+ * * EFBIG			- The write does not fit inside the allocation.
+ * * EFAULT			- Part of the buffer is not readable. Any bytes
+ *				  copied before that are reported as a short
+ *				  write, so a caller that has to know how far it
+ *				  got looks at the return value. A fatal signal
+ *				  part way through is reported the same way.
+ */
+#define NE_ALLOC_MEMORY			_IOW(0xAE, 0x2E, struct ne_alloc_memory_args)
+
 /**
  * NE_START_ENCLAVE - The command is used to trigger enclave start after the
  *		      enclave resources, such as memory and CPU, have been set.
@@ -282,9 +352,10 @@ struct ne_add_any_vcpu_args {
 #define NE_ERR_MEM_NOT_HUGE_PAGE		(263)
 /**
  * NE_ERR_MEM_DIFFERENT_NUMA_NODE - The user space memory region is backed by
- *				    pages from different NUMA nodes than the CPUs.
- *				    Not returned for an enclave whose CPU pool
- *				    spans nodes.
+ *				    pages from different NUMA nodes than the CPUs,
+ *				    or a requested node is not the one the CPUs
+ *				    are on. Not returned for an enclave whose CPU
+ *				    pool spans nodes.
  */
 #define NE_ERR_MEM_DIFFERENT_NUMA_NODE		(264)
 /**
@@ -391,6 +462,19 @@ struct ne_user_memory_region {
 	__u64	userspace_addr;
 };
 
+/**
+ * DOC: Enclave memory allocation flags
+ */
+
+/**
+ * NE_MEMORY_PRIVATE - An allocation the parent has no address for, because it
+ *		       cannot be mapped. Writing to the descriptor is the only
+ *		       way to put an enclave image in one.
+ */
+#define NE_MEMORY_PRIVATE		(0x00)
+
+#define NE_ALLOC_MEMORY_MAX_FLAG_VAL	(0x01)
+
 /**
  * DOC: Enclave start info flags
  */
-- 
2.47.1

[PATCH 07/40] nitro_enclaves: Add NE_MEMORY_SHARED

[Alexander Graf]

In preparation to support Nitro Sandboxes, whose parent keeps using some
of the memory it hands over, add NE_MEMORY_SHARED, a flag for
NE_ALLOC_MEMORY that makes an allocation mappable. The parent needs that
to read state the device writes for it, and for virtqueues and buffers
that both the parent and the sandbox access.

Signed-off-by: Alexander Graf <graf@amazon.com>
---
 drivers/virt/nitro_enclaves/ne_misc_dev.c | 160 +++++++++++++++++++++-
 drivers/virt/nitro_enclaves/ne_misc_dev.h |   6 +-
 include/uapi/linux/nitro_enclaves.h       |  36 ++++-
 3 files changed, 190 insertions(+), 12 deletions(-)

diff --git a/drivers/virt/nitro_enclaves/ne_misc_dev.c b/drivers/virt/nitro_enclaves/ne_misc_dev.c
index 56ca1a9d99d41..55dbeb5b14d29 100644
--- a/drivers/virt/nitro_enclaves/ne_misc_dev.c
+++ b/drivers/virt/nitro_enclaves/ne_misc_dev.c
@@ -2597,17 +2597,21 @@ static long ne_ioctl(struct file *file, unsigned int cmd, unsigned long arg)
  * serving such a request out of movable memory would take that memory from
  * the rest of the host and leave the reservation bounding nothing.
  *
- * A request larger than one contiguous run is satisfied by several runs, which
- * the driver keeps on a list of its own so that an enclave can be given every
- * one of them.
+ * A request larger than one contiguous run is satisfied by several runs, and
+ * mmap() maps them back to back into one VM_PFNMAP VMA, so user space sees a
+ * single region. The driver keeps the runs on a list of its own, so that an
+ * enclave can be given them without anyone reading them back out of the page
+ * tables.
  *
- * An allocation is never mapped, and the enclave image reaches it through
- * write().
+ * An allocation is either private or shared. A private one is never mapped, and
+ * the enclave image reaches it through write() instead. A shared one can also be
+ * mapped, at an offset and a length its own size bounds.
  *
  * The runs belong to the file the allocation is handed out on. Zeroing a fresh
  * run and a write into an allocation go through ne_walk_run(), so that neither
- * can run for as long as the allocation is large; the release path walks the
- * runs with a preemption point of its own.
+ * can run for as long as the allocation is large. A fault resolves the run its
+ * offset falls in with a binary search instead of walking, and the release path
+ * walks the runs with a preemption point of its own.
  */
 
 /**
@@ -2788,6 +2792,82 @@ static int ne_walk_allocs(struct ne_contig_mem *mem, u64 off, u64 len,
 	return 0;
 }
 
+/**
+ * ne_contig_fault_order() - Install one page table entry for a fault.
+ * @vmf:	The fault.
+ * @order:	Order of the entry to install, zero for a PTE.
+ *
+ * A mapping is populated as it is touched, so a caller that maps a whole
+ * allocation and builds an enclave image at the front of it pays page tables
+ * for the part it touched.
+ *
+ * An entry of @order needs that much of one run behind it, at a matching
+ * physical alignment. A run is a power-of-two multiple of 2 MiB aligned to at
+ * least 2 MiB, so a PMD sized entry fits inside one whenever the address it
+ * lands at is aligned too, and anything that does not fit falls back to the
+ * next order down.
+ *
+ * Return: a VM_FAULT_* value.
+ */
+static vm_fault_t ne_contig_fault_order(struct vm_fault *vmf, unsigned int order)
+{
+	struct vm_area_struct *vma = vmf->vma;
+	struct ne_contig_mem *mem = vma->vm_file->private_data;
+	unsigned long len = PAGE_SIZE << order;
+	unsigned long addr = ALIGN_DOWN(vmf->address, len);
+	struct ne_contig_alloc *alloc;
+	unsigned long pfn;
+	u64 off;
+
+	if (addr < vma->vm_start || addr + len > vma->vm_end)
+		return VM_FAULT_FALLBACK;
+
+	off = ((u64)vma->vm_pgoff << PAGE_SHIFT) + (addr - vma->vm_start);
+
+	alloc = ne_find_alloc(mem, off);
+	if (!alloc)
+		return VM_FAULT_SIGBUS;
+
+	if (off + len > alloc->off + ((u64)alloc->nr_pages << PAGE_SHIFT))
+		return VM_FAULT_FALLBACK;
+
+	pfn = page_to_pfn(alloc->page) + ((off - alloc->off) >> PAGE_SHIFT);
+	if (!IS_ALIGNED(pfn, len >> PAGE_SHIFT))
+		return VM_FAULT_FALLBACK;
+
+	if (!order)
+		return vmf_insert_pfn(vma, addr, pfn);
+
+	if (IS_ENABLED(CONFIG_ARCH_SUPPORTS_PMD_PFNMAP) && order == PMD_ORDER)
+		return vmf_insert_pfn_pmd(vmf, pfn,
+					  vmf->flags & FAULT_FLAG_WRITE);
+
+	if (IS_ENABLED(CONFIG_ARCH_SUPPORTS_PUD_PFNMAP) && order == PUD_ORDER)
+		return vmf_insert_pfn_pud(vmf, pfn,
+					  vmf->flags & FAULT_FLAG_WRITE);
+
+	return VM_FAULT_FALLBACK;
+}
+
+static vm_fault_t ne_contig_vma_fault(struct vm_fault *vmf)
+{
+	return ne_contig_fault_order(vmf, 0);
+}
+
+#ifdef CONFIG_ARCH_SUPPORTS_HUGE_PFNMAP
+static vm_fault_t ne_contig_huge_fault(struct vm_fault *vmf, unsigned int order)
+{
+	return ne_contig_fault_order(vmf, order);
+}
+#endif
+
+static const struct vm_operations_struct ne_contig_vm_ops = {
+	.fault		= ne_contig_vma_fault,
+#ifdef CONFIG_ARCH_SUPPORTS_HUGE_PFNMAP
+	.huge_fault	= ne_contig_huge_fault,
+#endif
+};
+
 static struct cma *ne_numa_cma_area(int nid)
 {
 	struct cma *cma = dma_get_numa_cma_area(nid);
@@ -2915,6 +2995,69 @@ struct page *ne_alloc_contig(unsigned long nr_pages, int nid,
 	return page;
 }
 
+/**
+ * ne_contig_supports_mmap() - Whether an allocation may be mapped at all.
+ * @file:	The allocation's file.
+ *
+ * A private allocation is the one an enclave gets to itself, and it is never
+ * mapped. The attribute is fixed when the allocation is made and the file is
+ * not published until it is set, so this reads it without a lock.
+ *
+ * Return: true when the caller may map the allocation.
+ */
+static bool ne_contig_supports_mmap(struct file *file)
+{
+	struct ne_contig_mem *mem = file->private_data;
+
+	return mem->shared;
+}
+
+/**
+ * ne_contig_mmap_prepare() - Set a mapping of part or all of an allocation up.
+ * @desc:	The mapping to set up.
+ *
+ * The offset and the length both have to be multiples of 2 MiB, which is the
+ * unit the allocation itself is made in, and the mapping has to stay inside
+ * the allocation. A caller may map part of an allocation, and may map one
+ * allocation more than once; the runs are not mutated after the file is
+ * published and the file holds them for as long as any mapping exists.
+ *
+ * ne_contig_fault_order() populates the mapping as it is touched. The mapping
+ * asks for VM_HUGEPAGE because a fault on a VM_PFNMAP mapping without it only
+ * reaches ->huge_fault where the transparent_hugepage policy is "always".
+ *
+ * Return: 0 on success, negative return value on failure.
+ */
+static int ne_contig_mmap_prepare(struct vm_area_desc *desc)
+{
+	struct ne_contig_mem *mem = desc->file->private_data;
+	unsigned long size = vma_desc_size(desc);
+	u64 offset = (u64)desc->pgoff << PAGE_SHIFT;
+
+	if (!ne_contig_supports_mmap(desc->file))
+		return -ENODEV;
+
+	/*
+	 * MAP_PRIVATE of an allocation has no meaning: the memory exists to be
+	 * given to an enclave, and a copy that only this process can see never
+	 * reaches one.
+	 */
+	if (!vma_desc_test(desc, VMA_SHARED_BIT))
+		return -EINVAL;
+
+	if ((size | offset) & (NE_MIN_MEM_REGION_SIZE - 1))
+		return -EINVAL;
+
+	if (offset > mem->size || size > mem->size - offset)
+		return -EINVAL;
+
+	desc->vm_ops = &ne_contig_vm_ops;
+	vma_desc_set_flags(desc, VMA_IO_BIT, VMA_PFNMAP_BIT, VMA_DONTEXPAND_BIT,
+			   VMA_DONTDUMP_BIT, VMA_DONTCOPY_BIT, VMA_HUGEPAGE_BIT);
+
+	return 0;
+}
+
 /**
  * struct ne_write_ctx - Where the write walk has got to.
  * @iter:	Where the bytes come from.
@@ -3030,6 +3173,8 @@ static int ne_contig_release(struct inode *inode, struct file *file)
 static const struct file_operations ne_contig_fops = {
 	.owner			= THIS_MODULE,
 	.llseek			= generic_file_llseek,
+	.mmap_prepare		= ne_contig_mmap_prepare,
+	.get_unmapped_area	= thp_get_unmapped_area,
 	.write_iter		= ne_contig_write_iter,
 	.release		= ne_contig_release,
 };
@@ -3176,6 +3321,7 @@ static int ne_alloc_memory_ioctl(struct ne_enclave *ne_enclave,
 	mem->mm = current->mm;
 	mem->nr_pinned = nr_pages;
 	mem->size = args->size;
+	mem->shared = args->flags == NE_MEMORY_SHARED;
 	mem->enclave_id = ne_enclave->id;
 
 	/*
diff --git a/drivers/virt/nitro_enclaves/ne_misc_dev.h b/drivers/virt/nitro_enclaves/ne_misc_dev.h
index 1a76c12815298..fcdf457af5800 100644
--- a/drivers/virt/nitro_enclaves/ne_misc_dev.h
+++ b/drivers/virt/nitro_enclaves/ne_misc_dev.h
@@ -38,6 +38,7 @@ struct ne_contig_alloc {
  * @allocs:	Contiguous runs backing the allocation, ordered by offset.
  * @nr_allocs:	Number of runs in @allocs.
  * @size:	Size of the allocation in bytes, mirrored in i_size.
+ * @shared:	Whether the allocation may be mapped by the caller.
  * @enclave_id:	Id of the enclave the allocation was made for.
  * @mm:		The mm charged for the pinned pages, held via mmgrab().
  * @nr_pinned:	Pages charged to @mm->pinned_vm.
@@ -47,13 +48,14 @@ struct ne_contig_alloc {
  *
  * The array is built before the file is published and not mutated afterwards, so
  * any holder of a reference can read it without a lock of its own, which is what
- * lets a page fault resolve an offset to a run. @size is set in the same place
- * and is read the same way.
+ * lets a page fault resolve an offset to a run. @shared and @size are set in the
+ * same place and are read the same way.
  */
 struct ne_contig_mem {
 	struct ne_contig_alloc	*allocs;
 	unsigned int		nr_allocs;
 	u64			size;
+	bool			shared;
 	u64			enclave_id;
 	struct mm_struct	*mm;
 	unsigned long		nr_pinned;
diff --git a/include/uapi/linux/nitro_enclaves.h b/include/uapi/linux/nitro_enclaves.h
index a2919565bc883..f1315c9c876de 100644
--- a/include/uapi/linux/nitro_enclaves.h
+++ b/include/uapi/linux/nitro_enclaves.h
@@ -215,7 +215,7 @@ struct ne_add_any_vcpu_args {
  * @size:	Number of bytes to allocate, a multiple of 2 MiB.
  * @nid:	Host NUMA node to allocate from. One node, named; unlike
  *		%NE_ADD_ANY_VCPU it does not take %NE_NUMA_ANY.
- * @flags:	%NE_MEMORY_PRIVATE.
+ * @flags:	%NE_MEMORY_PRIVATE or %NE_MEMORY_SHARED.
  */
 struct ne_alloc_memory_args {
 	__u64	size;
@@ -237,7 +237,16 @@ struct ne_alloc_memory_args {
  *		     goes into one through write() or pwrite() and not through
  *		     a copy into a mapping, and every write is a call the
  *		     kernel handles and can act on or refuse.
- *		     The allocation lives until the descriptor is gone.
+ *		     A %NE_MEMORY_SHARED allocation can also be mapped, at
+ *		     offset 0 or at any other multiple of 2 MiB, for a multiple
+ *		     of 2 MiB, with %MAP_SHARED. A caller maps one to keep an
+ *		     address for memory it goes on using after it has handed the
+ *		     allocation over. Such a mapping is readable too, which is
+ *		     the only way to read allocated memory back. The mapping is
+ *		     not inherited across fork(); a child that needs one maps
+ *		     the descriptor itself.
+ *		     The allocation lives until the last of the descriptor and
+ *		     its mappings is gone.
  *		     The ioctl can be invoked on the enclave fd, before an
  *		     enclave is started.
  *
@@ -276,6 +285,17 @@ struct ne_alloc_memory_args {
  *				  write, so a caller that has to know how far it
  *				  got looks at the return value. A fatal signal
  *				  part way through is reported the same way.
+ *
+ * An mmap() of the descriptor sets errno to:
+ * * ENODEV			- The allocation is %NE_MEMORY_PRIVATE.
+ * * EINVAL			- The mapping is %MAP_PRIVATE, or its offset or
+ *				  length is not a multiple of 2 MiB, or it does
+ *				  not fit inside the allocation.
+ * A mapping is populated as it is touched, so the page tables for a page are
+ * allocated at first touch and not by mmap(). A touch by the process itself
+ * that fails to get them is retried until it succeeds or the process is
+ * killed. A touch the kernel makes for the process, as read() into the
+ * mapping does, reports EFAULT or a short transfer instead.
  */
 #define NE_ALLOC_MEMORY			_IOW(0xAE, 0x2E, struct ne_alloc_memory_args)
 
@@ -472,8 +492,18 @@ struct ne_user_memory_region {
  *		       way to put an enclave image in one.
  */
 #define NE_MEMORY_PRIVATE		(0x00)
+/**
+ * NE_MEMORY_SHARED - An allocation the parent may also map. A mapping is what a
+ *		      caller wants for memory it goes on reading or writing
+ *		      after it has handed the allocation over. It costs page
+ *		      tables for the part of it a caller touches, and it leaves
+ *		      the memory reachable from the parent for as long as it
+ *		      lives, so an allocation that is only being handed over
+ *		      does not ask for one.
+ */
+#define NE_MEMORY_SHARED		(0x01)
 
-#define NE_ALLOC_MEMORY_MAX_FLAG_VAL	(0x01)
+#define NE_ALLOC_MEMORY_MAX_FLAG_VAL	(0x02)
 
 /**
  * DOC: Enclave start info flags
-- 
2.47.1

[PATCH 08/40] nitro_enclaves: Add NE_ADD_MEMORY_REGION

[Alexander Graf]

The new allocation scheme also requires a new memory add ioctl. Introduce
NE_ADD_MEMORY_REGION which uses the fd that NE_ALLOC_MEMORY returns to
back a slot's memory.

Once memory is donated using this mechanism, disable the ability for user
space to write() into it. It is fully considered donated at this point.

For now, consider slot memory donation as mandatorily donated. You can
not use a SHARED mapping to back it until we introduce DIRECT mode later.

Signed-off-by: Alexander Graf <graf@amazon.com>
---
 drivers/virt/nitro_enclaves/ne_misc_dev.c | 262 ++++++++++++++++++++++
 drivers/virt/nitro_enclaves/ne_misc_dev.h |  32 ++-
 include/uapi/linux/nitro_enclaves.h       | 104 +++++++--
 3 files changed, 379 insertions(+), 19 deletions(-)

diff --git a/drivers/virt/nitro_enclaves/ne_misc_dev.c b/drivers/virt/nitro_enclaves/ne_misc_dev.c
index 55dbeb5b14d29..38345ad7c7985 100644
--- a/drivers/virt/nitro_enclaves/ne_misc_dev.c
+++ b/drivers/virt/nitro_enclaves/ne_misc_dev.c
@@ -119,6 +119,8 @@ static long ne_ioctl(struct file *file, unsigned int cmd, unsigned long arg);
  */
 static atomic64_t ne_enclave_seq = ATOMIC64_INIT(0);
 
+static int ne_add_memory_region_ioctl(struct ne_enclave *ne_enclave,
+				      struct ne_add_memory_region_args *args);
 static int ne_alloc_memory_ioctl(struct ne_enclave *ne_enclave,
 				 struct ne_alloc_memory_args *args);
 static const struct file_operations ne_fops = {
@@ -2069,6 +2071,39 @@ static long ne_enclave_do_ioctl(struct file *file, unsigned int cmd, unsigned lo
 		return 0;
 	}
 
+	case NE_EA_IOCTL(NE_ADD_MEMORY_REGION): {
+		struct ne_add_memory_region_args __user *uargs = (void __user *)arg;
+		unsigned int usize = _IOC_SIZE(cmd);
+		struct ne_add_memory_region_args args;
+		int rc;
+
+		if (usize < offsetofend(struct ne_add_memory_region_args, flags))
+			return -EINVAL;
+
+		rc = copy_struct_from_user(&args, sizeof(args), uargs, usize);
+		if (rc)
+			return rc;
+
+		rc = mutex_lock_killable(&ne_enclave->enclave_info_mutex);
+		if (rc)
+			return rc;
+
+		if (ne_enclave->state != NE_STATE_INIT) {
+			dev_err_ratelimited(ne_misc_dev.this_device,
+					    "Enclave is not in init state\n");
+
+			mutex_unlock(&ne_enclave->enclave_info_mutex);
+
+			return -NE_ERR_NOT_IN_INIT_STATE;
+		}
+
+		rc = ne_add_memory_region_ioctl(ne_enclave, &args);
+
+		mutex_unlock(&ne_enclave->enclave_info_mutex);
+
+		return rc;
+	}
+
 	case NE_EA_IOCTL(NE_ALLOC_MEMORY): {
 		struct ne_alloc_memory_args __user *uargs = (void __user *)arg;
 		unsigned int usize = _IOC_SIZE(cmd);
@@ -2139,6 +2174,33 @@ static long ne_enclave_ioctl(struct file *file, unsigned int cmd, unsigned long
 	return rc;
 }
 
+/**
+ * ne_enclave_remove_all_contig_regions() - Drop the enclave's references to the
+ *					    allocations it was given.
+ * @ne_enclave :	Private data associated with the current enclave.
+ *
+ * The last reference on an allocation's file frees its runs and gives back its
+ * %RLIMIT_MEMLOCK charge, so an allocation whose descriptor and mappings are
+ * already gone is freed here.
+ *
+ * Context: Process context. This function is called on an enclave that is on
+ *	    no list and has no file left.
+ */
+static void ne_enclave_remove_all_contig_regions(struct ne_enclave *ne_enclave)
+{
+	struct ne_contig_region *ne_contig_region = NULL;
+	struct ne_contig_region *tmp = NULL;
+
+	list_for_each_entry_safe(ne_contig_region, tmp,
+				 &ne_enclave->contig_regions_list, list) {
+		list_del(&ne_contig_region->list);
+
+		fput(ne_contig_region->file);
+
+		kfree(ne_contig_region);
+	}
+}
+
 /**
  * ne_enclave_remove_all_mem_region_entries() - Remove all memory region entries
  *						from the enclave data structure.
@@ -2278,6 +2340,7 @@ static int ne_enclave_free_slot(struct ne_pci_dev *ne_pci_dev,
  */
 static void ne_enclave_free(struct ne_enclave *ne_enclave)
 {
+	ne_enclave_remove_all_contig_regions(ne_enclave);
 	ne_enclave_remove_all_mem_region_entries(ne_enclave);
 	ne_enclave_remove_all_vcpu_id_entries(ne_enclave);
 
@@ -2494,6 +2557,7 @@ static int ne_create_vm_ioctl(struct ne_pci_dev *ne_pci_dev, u64 __user *slot_ui
 	ne_enclave->has_event = false;
 	mutex_init(&ne_enclave->enclave_info_mutex);
 	ne_enclave->max_mem_regions = cmd_reply.mem_regions;
+	INIT_LIST_HEAD(&ne_enclave->contig_regions_list);
 	INIT_LIST_HEAD(&ne_enclave->mem_regions_list);
 	ne_enclave->mm = current->mm;
 	ne_enclave->slot_uid = cmd_reply.slot_uid;
@@ -3116,6 +3180,12 @@ static int ne_write_span(void *arg, struct page *page, u64 off, u64 len)
  * refusal is -EFBIG, the same as generic_write_check_limits() gives for a write
  * beyond s_maxbytes. There is no RLIMIT_FSIZE in it, so no SIGXFSZ either.
  *
+ * Donating an allocation closes this path, and @lock is what makes the two
+ * orderly. It is taken here and in ne_add_memory_region_ioctl() and nowhere
+ * else; setting a mapping up and faulting one in both read only fields that
+ * never change, so nothing takes it under mmap_write_lock or in a fault, and
+ * copying from user space under it cannot invert against the fault path.
+ *
  * Return: the number of bytes written, or a negative return value on failure.
  */
 static ssize_t ne_contig_write_iter(struct kiocb *iocb, struct iov_iter *from)
@@ -3132,6 +3202,11 @@ static ssize_t ne_contig_write_iter(struct kiocb *iocb, struct iov_iter *from)
 	if (pos >= mem->size || count > mem->size - pos)
 		return -EFBIG;
 
+	guard(mutex)(&mem->lock);
+
+	if (mem->donated)
+		return -EPERM;
+
 	rc = ne_walk_allocs(mem, pos, count, NE_WRITE_SPAN_PAGES, ne_write_span,
 			    &ctx);
 
@@ -3165,6 +3240,8 @@ static int ne_contig_release(struct inode *inode, struct file *file)
 
 	mmdrop(mem->mm);
 
+	mutex_destroy(&mem->lock);
+
 	kfree(mem);
 
 	return 0;
@@ -3317,6 +3394,7 @@ static int ne_alloc_memory_ioctl(struct ne_enclave *ne_enclave,
 		return -ENOMEM;
 	}
 
+	mutex_init(&mem->lock);
 	mmgrab(current->mm);
 	mem->mm = current->mm;
 	mem->nr_pinned = nr_pages;
@@ -3385,11 +3463,195 @@ static int ne_alloc_memory_ioctl(struct ne_enclave *ne_enclave,
 	ne_contig_free_allocs(mem);
 	atomic64_sub(nr_pages, &mem->mm->pinned_vm);
 	mmdrop(mem->mm);
+	mutex_destroy(&mem->lock);
 	kfree(mem);
 
 	return rc;
 }
 
+/**
+ * ne_add_memory_region_ioctl() - Give the slot associated with the current
+ *				  enclave a whole allocation.
+ * @ne_enclave:	Private data associated with the current enclave.
+ * @args:	Descriptor for the allocation, and flags.
+ *
+ * Every run the allocation is made of becomes a memory region, except that runs
+ * which turn out to be physically adjacent are merged into one, so a request
+ * that took several runs takes up several of the enclave's regions or fewer.
+ * The runs are 2 MiB aligned and a multiple of 2 MiB long as they come out of
+ * NE_ALLOC_MEMORY, which is what the hypervisor asks of a region, and no
+ * virtual address is involved on this path at all.
+ *
+ * The reference on the file is what keeps the runs and their %RLIMIT_MEMLOCK
+ * charge alive for as long as the enclave has them, whatever user space does
+ * with the descriptor and its mappings afterwards.
+ *
+ * Context: Process context. This function is called with the ne_enclave mutex held.
+ * Return:
+ * * 0 on success.
+ * * Negative return value on failure.
+ */
+static int ne_add_memory_region_ioctl(struct ne_enclave *ne_enclave,
+				      struct ne_add_memory_region_args *args)
+{
+	struct ne_phys_contig_mem_regions phys_contig_mem_regions = {};
+	struct pci_dev *pdev = ne_devs.ne_pci_dev->pdev;
+	struct ne_contig_region *ne_contig_region = NULL;
+	struct ne_contig_region *held = NULL;
+	struct ne_contig_mem *mem = NULL;
+	unsigned long i = 0;
+	int rc = -EINVAL;
+
+	if (ne_enclave->mm != current->mm)
+		return -EIO;
+
+	if (args->flags) {
+		dev_err_ratelimited(ne_misc_dev.this_device,
+				    "Incorrect flag for memory region\n");
+
+		return -NE_ERR_INVALID_FLAG_VALUE;
+	}
+
+	CLASS(fd, f)(args->fd);
+
+	if (fd_empty(f))
+		return -EBADF;
+
+	if (fd_file(f)->f_op != &ne_contig_fops) {
+		dev_err_ratelimited(ne_misc_dev.this_device,
+				    "Not an enclave memory allocation\n");
+
+		return -EINVAL;
+	}
+
+	/*
+	 * Every file with these operations came from ne_contig_create_file(),
+	 * which hands the allocation to anon_inode_create_getfile() before there
+	 * is a descriptor to pass in here, so it is there to be read.
+	 */
+	mem = fd_file(f)->private_data;
+
+	if (mem->enclave_id != ne_enclave->id) {
+		dev_err_ratelimited(ne_misc_dev.this_device,
+				    "Allocation was made for another enclave\n");
+
+		return -EINVAL;
+	}
+
+	/*
+	 * The parent may map a shared allocation and keeps any mapping it has
+	 * for as long as the allocation lives. Memory a region is made of is
+	 * the enclave's alone.
+	 */
+	if (mem->shared) {
+		dev_err_ratelimited(ne_misc_dev.this_device,
+				    "The memory region allocation is shared\n");
+
+		return -EINVAL;
+	}
+
+	list_for_each_entry(held, &ne_enclave->contig_regions_list, list) {
+		if (held->file == fd_file(f)) {
+			dev_err_ratelimited(ne_misc_dev.this_device,
+					    "The memory region is already used\n");
+
+			return -NE_ERR_MEM_REGION_ALREADY_USED;
+		}
+	}
+
+	ne_contig_region = kzalloc_obj(*ne_contig_region);
+	if (!ne_contig_region)
+		return -ENOMEM;
+
+	/*
+	 * One entry per run is the worst case, since merging only ever reduces
+	 * the count. That is 4 MiB of entries for a 512 GiB allocation the
+	 * allocator had to satisfy in 2 MiB runs, so this uses kvmalloc like
+	 * the run array.
+	 */
+	phys_contig_mem_regions.regions = kvzalloc_objs(*phys_contig_mem_regions.regions,
+							mem->nr_allocs);
+	if (!phys_contig_mem_regions.regions) {
+		rc = -ENOMEM;
+
+		goto free_contig_region;
+	}
+
+	for (i = 0; i < mem->nr_allocs; i++) {
+		struct ne_contig_alloc *alloc = &mem->allocs[i];
+
+		rc = ne_merge_phys_contig_memory_regions(&phys_contig_mem_regions,
+							 page_to_phys(alloc->page),
+							 (u64)alloc->nr_pages << PAGE_SHIFT);
+		if (rc < 0)
+			goto free_regions;
+	}
+
+	if ((ne_enclave->nr_mem_regions + phys_contig_mem_regions.num) >
+	    ne_enclave->max_mem_regions) {
+		dev_err_ratelimited(ne_misc_dev.this_device,
+				    "Reached max memory regions %lld\n",
+				    ne_enclave->max_mem_regions);
+
+		rc = -NE_ERR_MEM_MAX_REGIONS;
+
+		goto free_regions;
+	}
+
+	/*
+	 * From here the allocation is the enclave's. Closing the write path under
+	 * the same lock a write holds means a write either finished before this
+	 * point or is refused after it, so a donation cannot land in the middle of
+	 * one. Whether the whole image got there is the caller's to check: a write
+	 * cut short by an unreadable buffer reports how far it got.
+	 */
+	scoped_guard(mutex, &mem->lock)
+		mem->donated = true;
+
+	ne_contig_region->file = get_file(fd_file(f));
+
+	list_add(&ne_contig_region->list, &ne_enclave->contig_regions_list);
+
+	for (i = 0; i < phys_contig_mem_regions.num; i++) {
+		struct ne_pci_dev_cmd_reply cmd_reply = {};
+		struct slot_add_mem_req slot_add_mem_req = {};
+
+		slot_add_mem_req.slot_uid = ne_enclave->slot_uid;
+		slot_add_mem_req.paddr = phys_contig_mem_regions.regions[i].start;
+		slot_add_mem_req.size = range_len(&phys_contig_mem_regions.regions[i]);
+
+		rc = ne_do_request(pdev, SLOT_ADD_MEM,
+				   &slot_add_mem_req, sizeof(slot_add_mem_req),
+				   &cmd_reply, sizeof(cmd_reply));
+		if (rc < 0) {
+			dev_err_ratelimited(ne_misc_dev.this_device,
+					    "Error in slot add mem [rc=%d]\n", rc);
+
+			kvfree(phys_contig_mem_regions.regions);
+
+			/*
+			 * Exit here without dropping the reference on the file
+			 * as memory regions may already been added.
+			 */
+			return rc;
+		}
+
+		ne_enclave->mem_size += slot_add_mem_req.size;
+		ne_enclave->nr_mem_regions++;
+	}
+
+	kvfree(phys_contig_mem_regions.regions);
+
+	return 0;
+
+free_regions:
+	kvfree(phys_contig_mem_regions.regions);
+free_contig_region:
+	kfree(ne_contig_region);
+
+	return rc;
+}
+
 #if defined(CONFIG_NITRO_ENCLAVES_MISC_DEV_TEST)
 #include "ne_misc_dev_test.c"
 #endif
diff --git a/drivers/virt/nitro_enclaves/ne_misc_dev.h b/drivers/virt/nitro_enclaves/ne_misc_dev.h
index fcdf457af5800..2502cca3f3f64 100644
--- a/drivers/virt/nitro_enclaves/ne_misc_dev.h
+++ b/drivers/virt/nitro_enclaves/ne_misc_dev.h
@@ -42,14 +42,23 @@ struct ne_contig_alloc {
  * @enclave_id:	Id of the enclave the allocation was made for.
  * @mm:		The mm charged for the pinned pages, held via mmgrab().
  * @nr_pinned:	Pages charged to @mm->pinned_vm.
+ * @lock:	Serialises a write against the donation that ends writing.
+ * @donated:	Whether the allocation has been given to its enclave.
  *
- * The runs hang off the file, and a mapping's page table entries cannot outlive
- * them because a VMA holds vm_file for as long as it exists.
+ * The runs hang off the file, so that closing the last descriptor does not free
+ * memory an enclave is still using: a donated region holds the file with
+ * get_file(). That is also what keeps a mapping's page table entries from
+ * outliving the runs, since a VMA holds vm_file for as long as it exists.
  *
  * The array is built before the file is published and not mutated afterwards, so
  * any holder of a reference can read it without a lock of its own, which is what
  * lets a page fault resolve an offset to a run. @shared and @size are set in the
- * same place and are read the same way.
+ * same place and are read the same way. @donated is the one field that changes
+ * after publication, and @lock covers it.
+ *
+ * @enclave_id is compared, never dereferenced, and the counter it comes from is
+ * 64 bits wide, so an allocation cannot be donated to a later enclave that
+ * happens to have been placed at the address of the one it was made for.
  */
 struct ne_contig_mem {
 	struct ne_contig_alloc	*allocs;
@@ -59,6 +68,20 @@ struct ne_contig_mem {
 	u64			enclave_id;
 	struct mm_struct	*mm;
 	unsigned long		nr_pinned;
+	/* Serialises a write against the donation that ends writing. */
+	struct mutex		lock;
+	bool			donated;
+};
+
+/**
+ * struct ne_contig_region - Enclave memory given from one allocation.
+ * @list:	Entry in the enclave's list of allocations given to it.
+ * @file:	File holding the runs, held with get_file() so that they outlive the
+ *		descriptor the allocation was handed out on and its mappings.
+ */
+struct ne_contig_region {
+	struct list_head	list;
+	struct file		*file;
 };
 
 /**
@@ -82,6 +105,8 @@ struct ne_mem_region {
  * @enclave_info_mutex :	Mutex for accessing this internal state.
  * @detached:			The NE PCI device the enclave was created on is
  *				gone. Written with ne_devs.lock held for writing.
+ * @contig_regions_list:	List of %NE_ALLOC_MEMORY allocations given to the
+ *				enclave with %NE_ADD_MEMORY_REGION.
  * @enclave_list_entry :	Entry in the list of created enclaves.
  * @eventq:			Wait queue used for out-of-band event notifications
  *				triggered from the PCI device event handler to
@@ -116,6 +141,7 @@ struct ne_mem_region {
 struct ne_enclave {
 	struct mutex		enclave_info_mutex;
 	bool			detached;
+	struct list_head	contig_regions_list;
 	struct list_head	enclave_list_entry;
 	wait_queue_head_t	eventq;
 	bool			has_event;
diff --git a/include/uapi/linux/nitro_enclaves.h b/include/uapi/linux/nitro_enclaves.h
index f1315c9c876de..16515fe27b692 100644
--- a/include/uapi/linux/nitro_enclaves.h
+++ b/include/uapi/linux/nitro_enclaves.h
@@ -225,11 +225,13 @@ struct ne_alloc_memory_args {
 
 /**
  * NE_ALLOC_MEMORY - The command is used to allocate physically contiguous
- *		     memory to give to an enclave, drawn from one host NUMA
- *		     node. The argument is input only; on success the return
- *		     value is a file descriptor for the allocation.
+ *		     memory to give to an enclave with %NE_ADD_MEMORY_REGION,
+ *		     drawn from one host NUMA node. The argument is input only;
+ *		     on success the return value is a file descriptor for the
+ *		     allocation.
  *		     Writing to the descriptor places bytes in the allocation,
- *		     which is how an enclave image gets there.
+ *		     which is how an enclave image gets there, and stops
+ *		     working once the allocation has been donated.
  *		     O_APPEND has no effect. There is no read operation, so
  *		     read() and pread() both fail with EINVAL and a caller that
  *		     wants to check what it wrote keeps its own copy.
@@ -241,12 +243,19 @@ struct ne_alloc_memory_args {
  *		     offset 0 or at any other multiple of 2 MiB, for a multiple
  *		     of 2 MiB, with %MAP_SHARED. A caller maps one to keep an
  *		     address for memory it goes on using after it has handed the
- *		     allocation over. Such a mapping is readable too, which is
- *		     the only way to read allocated memory back. The mapping is
- *		     not inherited across fork(); a child that needs one maps
- *		     the descriptor itself.
- *		     The allocation lives until the last of the descriptor and
- *		     its mappings is gone.
+ *		     allocation over, including once the allocation has been
+ *		     donated and writing to the descriptor has stopped working.
+ *		     Such a mapping is readable too, which is the only way to
+ *		     read allocated memory back. The mapping is not inherited
+ *		     across fork(); a child that needs one maps the descriptor
+ *		     itself.
+ *		     Donating does not take a mapping away, so a parent that
+ *		     mapped a shared allocation can still write to it through
+ *		     that mapping afterwards.
+ *		     The allocation lives until the last of the descriptor, its
+ *		     mappings and the enclave regions donated from it is gone,
+ *		     so a caller can close the descriptor as soon as it has
+ *		     donated.
  *		     The ioctl can be invoked on the enclave fd, before an
  *		     enclave is started.
  *
@@ -279,6 +288,8 @@ struct ne_alloc_memory_args {
  * * Error codes from get_unused_fd_flags() and anon_inode_create_getfile().
  *
  * Writing to the descriptor sets errno to:
+ * * EPERM			- The allocation has been donated with
+ *				  %NE_ADD_MEMORY_REGION.
  * * EFBIG			- The write does not fit inside the allocation.
  * * EFAULT			- Part of the buffer is not readable. Any bytes
  *				  copied before that are reported as a short
@@ -299,6 +310,66 @@ struct ne_alloc_memory_args {
  */
 #define NE_ALLOC_MEMORY			_IOW(0xAE, 0x2E, struct ne_alloc_memory_args)
 
+/**
+ * struct ne_add_memory_region_args - Allocation to give to an enclave (in).
+ * @fd:		Descriptor from %NE_ALLOC_MEMORY.
+ * @flags:	Must be zero.
+ */
+struct ne_add_memory_region_args {
+	__s32	fd;
+	__u32	flags;
+};
+
+/**
+ * NE_ADD_MEMORY_REGION - The command is used to give an enclave the whole of an
+ *			  %NE_ALLOC_MEMORY allocation, as one memory region per
+ *			  contiguous run it is made of, or fewer where runs turn
+ *			  out to be physically adjacent. The argument is input
+ *			  only. The allocation has to have been made for this
+ *			  enclave, and can be given away once.
+ *			  Writing to the descriptor stops working here, so an
+ *			  enclave image goes in before this call. The allocation
+ *			  has to be %NE_MEMORY_PRIVATE, since memory the parent
+ *			  can still reach is not memory the enclave has to
+ *			  itself.
+ *			  An enclave's memory is laid out in the order its
+ *			  allocations are given to it, and the offset from
+ *			  %NE_GET_IMAGE_LOAD_INFO is an offset into that layout,
+ *			  so a caller building an enclave out of several
+ *			  allocations picks the order first, writes at the
+ *			  offsets that order implies, and gives them away in it.
+ *			  The ioctl can be invoked on the enclave fd, before an
+ *			  enclave is started.
+ *
+ * Context: Process context.
+ * Return:
+ * * 0					- Logic successfully completed.
+ * * -1					- There was a failure in the ioctl logic.
+ * On failure, errno is set to:
+ * * ENODEV				- The NE PCI device is not available.
+ * * EFAULT				- Argument read failure.
+ * * EBADF				- @fd is not an open file descriptor.
+ * * EINVAL				- The argument is shorter than @flags, or @fd
+ *					  is not a descriptor from %NE_ALLOC_MEMORY, or
+ *					  the allocation was made for another enclave,
+ *					  or the allocation is %NE_MEMORY_SHARED.
+ * * EIO				- Current task mm is not the same as the one
+ *					  that created the enclave.
+ * * E2BIG				- The argument is longer than this kernel knows
+ *					  about and the excess is not all zero.
+ * * ENOMEM				- Memory allocation failure for internal
+ *					  bookkeeping variables.
+ * * NE_ERR_NOT_IN_INIT_STATE		- The enclave is not in init state (init =
+ *					  before being started).
+ * * NE_ERR_INVALID_FLAG_VALUE		- The value of the provided flag is invalid.
+ * * NE_ERR_MEM_REGION_ALREADY_USED	- The allocation was already given to the
+ *					  enclave.
+ * * NE_ERR_MEM_MAX_REGIONS		- The number of memory regions set for the
+ *					  enclave reached maximum.
+ * * Error codes from the NE PCI device request.
+ */
+#define NE_ADD_MEMORY_REGION		_IOW(0xAE, 0x2F, struct ne_add_memory_region_args)
+
 /**
  * NE_START_ENCLAVE - The command is used to trigger enclave start after the
  *		      enclave resources, such as memory and CPU, have been set.
@@ -362,7 +433,8 @@ struct ne_alloc_memory_args {
  */
 #define NE_ERR_UNALIGNED_MEM_REGION_ADDR	(261)
 /**
- * NE_ERR_MEM_REGION_ALREADY_USED - The user space memory region is already used.
+ * NE_ERR_MEM_REGION_ALREADY_USED - The user space memory region, or the
+ *				      %NE_ALLOC_MEMORY allocation, is already used.
  */
 #define NE_ERR_MEM_REGION_ALREADY_USED		(262)
 /**
@@ -495,11 +567,11 @@ struct ne_user_memory_region {
 /**
  * NE_MEMORY_SHARED - An allocation the parent may also map. A mapping is what a
  *		      caller wants for memory it goes on reading or writing
- *		      after it has handed the allocation over. It costs page
- *		      tables for the part of it a caller touches, and it leaves
- *		      the memory reachable from the parent for as long as it
- *		      lives, so an allocation that is only being handed over
- *		      does not ask for one.
+ *		      after it has handed the allocation over, including after
+ *		      it has been donated. It costs page tables for the part of
+ *		      it a caller touches, and it leaves the memory reachable
+ *		      from the parent for as long as it lives, so an allocation
+ *		      that is only being handed over does not ask for one.
  */
 #define NE_MEMORY_SHARED		(0x01)
 
-- 
2.47.1

[PATCH 09/40] nitro_enclaves: Document the memory pool

[Alexander Graf]

An admin reserving memory for enclaves needs to pick the CMA area and
the kernel parameters that fill it. Document the pool: allocating with
NE_ALLOC_MEMORY, writing the image through the returned descriptor and
adding it with NE_ADD_MEMORY_REGION, the config options a reservation
needs to be reachable, and the difference between private and shared
allocations and how long an allocation lives.

Signed-off-by: Alexander Graf <graf@amazon.com>
---
 Documentation/virt/ne_overview.rst | 83 ++++++++++++++++++++++++++++--
 1 file changed, 80 insertions(+), 3 deletions(-)

diff --git a/Documentation/virt/ne_overview.rst b/Documentation/virt/ne_overview.rst
index 223ee7fdec51b..4c4b7febaff0a 100644
--- a/Documentation/virt/ne_overview.rst
+++ b/Documentation/virt/ne_overview.rst
@@ -113,9 +113,10 @@ each node in turn, and sets numa_node to NE_NUMA_ANY when it does not
 care. Both start from a free thread of a core the enclave already holds,
 and only take a new core once there is none.
 
-The node says where cores come from. An enclave built out of a pool that
-spans nodes has no node of its own, so the rule that its memory be on
-that node does not apply to it and its memory can come from any node.
+The node says where cores come from. NE_ALLOC_MEMORY takes its own node;
+see `Memory pool`_. An enclave built out of a pool that spans
+nodes has no node of its own, so the rule that its memory be on that node
+does not apply to it and its memory can come from any node.
 
 The pool occupancy is readable under
 /sys/devices/virtual/misc/nitro_enclaves/cpu_pool/, and per node under
@@ -124,3 +125,79 @@ online when the device was probed. A node the pool holds nothing on
 reads empty rather than being absent. The driver offlines pool threads,
 so ``nodeN/cpulist`` under /sys/devices/system/node stops listing them
 and answers nothing about the pool.
+
+Memory pool
+-----------
+
+Enclave memory must be physically contiguous, in chunks of at least 2 MiB,
+and, for an enclave whose cores all sit on one node, on that node. It has
+traditionally come from hugetlbfs [2], which provides the hugepage sizes the
+architecture has -- 2 MiB and 1 GiB on x86, and 32 MiB, 512 MiB or 16 GiB on
+arm64 depending on the page size -- and needs the hugepage count managed out
+of band.
+
+The driver can hand that memory out itself, backed per NUMA node by the
+kernel's Contiguous Memory Allocator (CMA). Reserve an area on every node
+you intend to run enclaves on, with the numa_cma= or cma_pernuma= kernel
+parameters documented alongside the rest of them [4]. On a host with one
+online node, cma= reserves the one area there is to reserve and the driver
+uses it.
+
+Reaching a reserved area takes CONFIG_DMA_CMA, which NITRO_ENCLAVES selects,
+so the driver always has it. numa_cma= and cma_pernuma= need
+CONFIG_DMA_NUMA_CMA on top of that, which a host with more than one online
+node enables to name the node a reservation lands on. cma= needs neither, so
+on a host with one online node the driver's own select is the whole
+requirement.
+
+NE_ALLOC_MEMORY on the enclave file descriptor allocates 2 MiB aligned,
+physically contiguous memory from one host NUMA node's CMA area. A request
+that area cannot serve fails with -ENOMEM, so the reservation bounds what the
+enclaves on a node can take between them.
+It returns a file descriptor for the allocation. For an enclave whose cores
+all sit on one node the ioctl refuses any other node, so name the same node
+here and in NE_ADD_ANY_VCPU. An enclave built out of a pool that spans nodes
+has no node of its own and takes memory from any of them.
+
+An allocation is NE_MEMORY_PRIVATE unless the flags say NE_MEMORY_SHARED.
+A private one cannot be mapped, and mmap() on it returns ENODEV. Writing to
+the descriptor places bytes in either kind, which is how an enclave image
+gets into one. There is no read operation on the descriptor.
+
+A shared allocation can also be mapped, at any 2 MiB aligned offset for a
+multiple of 2 MiB, with MAP_SHARED. A caller maps one to keep an address for
+memory it goes on using after handing it over, including after the allocation
+has been donated, which is where writing to the descriptor stops working. Such
+a mapping is readable too, and is the only way to read allocated memory back.
+
+The mapping is VM_PFNMAP, so it holds no struct page references and any DMA pin
+over it returns EFAULT, long-term or not; io_uring buffer registration and RDMA
+memory registration are two that a caller might expect to work and that do not.
+Reaching enclave memory from a device in the primary VM is not something this
+interface offers.
+
+The allocation is accounted against RLIMIT_MEMLOCK. NE_ADD_MEMORY_REGION
+takes the descriptor and gives the enclave the whole allocation, as one
+memory region per contiguous run it is made of, or fewer where runs turn out
+to be physically adjacent. It also ends writing: the memory is the enclave's
+from that point, so an image goes in first. It takes private allocations
+only, since memory the parent can still reach is not memory the enclave has
+to itself.
+
+An enclave's memory is laid out in the order its allocations are given to it,
+and the offset NE_GET_IMAGE_LOAD_INFO returns is an offset into that layout.
+A caller building an enclave out of several allocations therefore picks the
+order first, writes at the offsets that order implies, and gives them away in
+it.
+
+A mapping is populated as it is touched, so it costs page tables for the part
+a caller reaches and not for all of it. It keeps the memory reachable from the
+parent for as long as it lives, so an allocation that is only being handed to
+an enclave stays private.
+
+NE_SET_USER_MEMORY_REGION is unchanged and still takes the address of a
+hugetlbfs region.
+
+The allocation lives until the last of its descriptor, its mappings and the
+regions given from it is gone, so neither closing the descriptor nor
+unmapping takes memory away from a running enclave.
-- 
2.47.1

[PATCH 10/40] nitro_enclaves: Add NE_CREATE_VM2 with a launch flag word

[Alexander Graf]

In preparation to support Nitro Sandboxes, whose properties the
hypervisor needs before it allocates a slot, add NE_CREATE_VM2. It takes
a word of launch flags for the slot. The slot id comes back in the
same struct ne_create_vm_args.

No flag is defined in this patch, so any set bit fails with
-NE_ERR_INVALID_FLAG_VALUE.

Signed-off-by: Alexander Graf <graf@amazon.com>
---
 drivers/virt/nitro_enclaves/ne_misc_dev.c | 53 +++++++++++++++++++++--
 include/uapi/linux/nitro_enclaves.h       | 48 ++++++++++++++++++++
 2 files changed, 97 insertions(+), 4 deletions(-)

diff --git a/drivers/virt/nitro_enclaves/ne_misc_dev.c b/drivers/virt/nitro_enclaves/ne_misc_dev.c
index 38345ad7c7985..68c497df4f14b 100644
--- a/drivers/virt/nitro_enclaves/ne_misc_dev.c
+++ b/drivers/virt/nitro_enclaves/ne_misc_dev.c
@@ -2452,6 +2452,8 @@ static const struct file_operations ne_enclave_fops = {
  *			  enclave file descriptor to be further used for enclave
  *			  resources handling e.g. memory regions and CPUs.
  * @ne_pci_dev :	Private data associated with the PCI device.
+ * @flags:		Launch-time flag word from NE_CREATE_VM2, or zero when
+ *			the caller used the flag-less NE_CREATE_VM.
  * @slot_uid:		User pointer to store the generated unique slot id
  *			associated with an enclave to.
  *
@@ -2461,7 +2463,8 @@ static const struct file_operations ne_enclave_fops = {
  * * Enclave fd on success.
  * * Negative return value on failure.
  */
-static int ne_create_vm_ioctl(struct ne_pci_dev *ne_pci_dev, u64 __user *slot_uid)
+static int ne_create_vm_ioctl(struct ne_pci_dev *ne_pci_dev, u64 flags,
+			      u64 __user *slot_uid)
 {
 	struct ne_pci_dev_cmd_reply cmd_reply = {};
 	int enclave_fd = -1;
@@ -2472,6 +2475,13 @@ static int ne_create_vm_ioctl(struct ne_pci_dev *ne_pci_dev, u64 __user *slot_ui
 	int rc = -EINVAL;
 	struct slot_alloc_req slot_alloc_req = {};
 
+	if (flags) {
+		dev_err_ratelimited(ne_misc_dev.this_device,
+				    "Unknown launch flag bit set 0x%llx\n", flags);
+
+		return -NE_ERR_INVALID_FLAG_VALUE;
+	}
+
 	mutex_lock(&ne_cpu_pool.mutex);
 
 	for (i = 0; i < ne_cpu_pool.nr_parent_vm_cores; i++)
@@ -2626,7 +2636,44 @@ static long ne_ioctl(struct file *file, unsigned int cmd, unsigned long arg)
 		if (ne_pci_dev) {
 			rc = mutex_lock_killable(&ne_pci_dev->enclaves_list_mutex);
 			if (!rc) {
-				rc = ne_create_vm_ioctl(ne_pci_dev, slot_uid);
+				rc = ne_create_vm_ioctl(ne_pci_dev, 0, slot_uid);
+				mutex_unlock(&ne_pci_dev->enclaves_list_mutex);
+			}
+		}
+
+		up_read(&ne_devs.lock);
+
+		return rc;
+	}
+	}
+
+	/* Extensible Argument ioctls */
+	switch (NE_EA_IOCTL(cmd)) {
+	case NE_EA_IOCTL(NE_CREATE_VM2): {
+		struct ne_create_vm_args args;
+		struct ne_create_vm_args __user *uargs = (void __user *)arg;
+		unsigned int usize = _IOC_SIZE(cmd);
+		struct ne_pci_dev *ne_pci_dev;
+		int rc;
+
+		if (usize < offsetofend(struct ne_create_vm_args, slot_uid))
+			return -EINVAL;
+
+		rc = copy_struct_from_user(&args, sizeof(args), uargs, usize);
+		if (rc)
+			return rc;
+
+		rc = down_read_killable(&ne_devs.lock);
+		if (rc)
+			return rc;
+
+		rc = -ENODEV;
+		ne_pci_dev = ne_devs.ne_pci_dev;
+		if (ne_pci_dev) {
+			rc = mutex_lock_killable(&ne_pci_dev->enclaves_list_mutex);
+			if (!rc) {
+				rc = ne_create_vm_ioctl(ne_pci_dev, args.flags,
+							&uargs->slot_uid);
 				mutex_unlock(&ne_pci_dev->enclaves_list_mutex);
 			}
 		}
@@ -2639,8 +2686,6 @@ static long ne_ioctl(struct file *file, unsigned int cmd, unsigned long arg)
 	default:
 		return -ENOTTY;
 	}
-
-	return 0;
 }
 
 /*
diff --git a/include/uapi/linux/nitro_enclaves.h b/include/uapi/linux/nitro_enclaves.h
index 16515fe27b692..8e43be9a0a1fa 100644
--- a/include/uapi/linux/nitro_enclaves.h
+++ b/include/uapi/linux/nitro_enclaves.h
@@ -46,6 +46,54 @@
  */
 #define NE_CREATE_VM			_IOR(0xAE, 0x20, __u64)
 
+/**
+ * struct ne_create_vm_args - NE_CREATE_VM2 argument (in / out).
+ * @flags:	Launch-time flag word (in). Flags select properties of the
+ *		enclave slot that the hypervisor has to know at allocation
+ *		time. NE_ENCLAVE_DEBUG_MODE stays a start-time property and
+ *		is rejected here.
+ * @slot_uid:	Generated unique slot id associated with an enclave (out).
+ */
+struct ne_create_vm_args {
+	__u64	flags;
+	__u64	slot_uid;
+};
+
+/**
+ * NE_CREATE_VM2 - As NE_CREATE_VM, additionally taking a launch-time flag
+ *		   word. The flags carry properties of the slot that have to
+ *		   be decided when the slot is allocated rather than when the
+ *		   enclave starts.
+ *
+ *		   NE_CREATE_VM continues to work; calling it is equivalent
+ *		   to calling NE_CREATE_VM2 with a zero flag word.
+ *
+ *		   The argument is read with copy_struct_from_user(), so a
+ *		   caller built against a header that predates a field this
+ *		   kernel knows leaves that field zero, and a caller that
+ *		   sets a field this kernel does not know is refused.
+ *
+ * Context: Process context.
+ * Return:
+ * * Enclave file descriptor		- As for NE_CREATE_VM.
+ * *  -1				- There was a failure in the ioctl logic.
+ * On failure, errno is set to:
+ * * ENODEV				- The NE PCI device is not available.
+ * * EFAULT				- Argument read or write failure.
+ * * NE_ERR_INVALID_FLAG_VALUE		- Unknown flag bit set, or
+ *					  NE_ENCLAVE_DEBUG_MODE set.
+ * * EINVAL				- The argument stops before @slot_uid.
+ * * E2BIG				- The argument carries a field this kernel
+ *					  does not know, set to a non-zero value.
+ * * ENOMEM				- Memory allocation failure for internal
+ *					  bookkeeping variables.
+ * * NE_ERR_NO_CPUS_AVAIL_IN_POOL	- No NE CPU pool set / no CPUs available
+ *					  in the pool.
+ * * Error codes from get_unused_fd_flags() and anon_inode_getfile().
+ * * Error codes from the NE PCI device request.
+ */
+#define NE_CREATE_VM2			_IOWR(0xAE, 0x30, struct ne_create_vm_args)
+
 /**
  * NE_ADD_VCPU - The command is used to set a vCPU for an enclave. The vCPU can
  *		 be auto-chosen from the NE CPU pool or it can be set by the
-- 
2.47.1

[PATCH 11/40] nitro_enclaves: Add NE_ENCLAVE_PCIE_MODE launch flag

[Alexander Graf]

Nitro Sandboxes give their guest virtio devices that the parent
provides, on a PCIe bus. The hypervisor sets up that bus when it
allocates the slot, so user space asks for it with a launch flag.

Define NE_ENCLAVE_PCIE_MODE, the first launch flag, and forward it in
SLOT_ALLOC. struct slot_alloc_req gains a flag word, and at offset 0,
over the unused field, a payload_version byte that tells the device to
read it. We send version 1 when a flag is set, and the all-zero request
of today otherwise.

A device that takes launch flags publishes a vendor-specific PCI
capability. A device that predates them ignores the flag word and
allocates a plain slot, so refuse a launch flag with -EOPNOTSUPP on a
device without the capability.

Signed-off-by: Alexander Graf <graf@amazon.com>
---
 drivers/virt/nitro_enclaves/ne_misc_dev.c | 19 ++++++++++++++++++-
 drivers/virt/nitro_enclaves/ne_pci_dev.h  | 23 +++++++++++++++++++++--
 include/uapi/linux/nitro_enclaves.h       | 14 ++++++++++++++
 3 files changed, 53 insertions(+), 3 deletions(-)

diff --git a/drivers/virt/nitro_enclaves/ne_misc_dev.c b/drivers/virt/nitro_enclaves/ne_misc_dev.c
index 68c497df4f14b..0bf62f703806e 100644
--- a/drivers/virt/nitro_enclaves/ne_misc_dev.c
+++ b/drivers/virt/nitro_enclaves/ne_misc_dev.c
@@ -2475,13 +2475,30 @@ static int ne_create_vm_ioctl(struct ne_pci_dev *ne_pci_dev, u64 flags,
 	int rc = -EINVAL;
 	struct slot_alloc_req slot_alloc_req = {};
 
-	if (flags) {
+	if (flags & ~(u64)NE_ENCLAVE_PCIE_MODE) {
 		dev_err_ratelimited(ne_misc_dev.this_device,
 				    "Unknown launch flag bit set 0x%llx\n", flags);
 
 		return -NE_ERR_INVALID_FLAG_VALUE;
 	}
 
+	/*
+	 * A device that takes launch flags publishes a vendor-specific
+	 * capability. One without it takes no launch flags and would allocate a
+	 * slot without the properties they ask for.
+	 */
+	if (flags && !pci_find_capability(ne_pci_dev->pdev, PCI_CAP_ID_VNDR)) {
+		dev_err_ratelimited(ne_misc_dev.this_device,
+				    "Device takes no launch flags\n");
+
+		return -EOPNOTSUPP;
+	}
+
+	if (flags) {
+		slot_alloc_req.payload_version = 1;
+		slot_alloc_req.flags = flags;
+	}
+
 	mutex_lock(&ne_cpu_pool.mutex);
 
 	for (i = 0; i < ne_cpu_pool.nr_parent_vm_cores; i++)
diff --git a/drivers/virt/nitro_enclaves/ne_pci_dev.h b/drivers/virt/nitro_enclaves/ne_pci_dev.h
index 19e129d14a3a0..b0957e2606ec3 100644
--- a/drivers/virt/nitro_enclaves/ne_pci_dev.h
+++ b/drivers/virt/nitro_enclaves/ne_pci_dev.h
@@ -7,10 +7,12 @@
 #define _NE_PCI_DEV_H_
 
 #include <linux/atomic.h>
+#include <linux/build_bug.h>
 #include <linux/list.h>
 #include <linux/mutex.h>
 #include <linux/pci.h>
 #include <linux/pci_ids.h>
+#include <linux/stddef.h>
 #include <linux/wait.h>
 
 /**
@@ -174,12 +176,29 @@ struct enclave_stop_req {
 
 /**
  * struct slot_alloc_req - SLOT_ALLOC request.
- * @unused:	In order to avoid weird sizeof edge cases.
+ * @payload_version:	Wire version at offset 0. Zero is the value a driver
+ *			without launch flags leaves here, and tells the device
+ *			to disregard @reserved and @flags. 1 means @flags is
+ *			valid.
+ * @reserved:		Alignment padding between @payload_version and
+ *			@flags. Zero.
+ * @flags:		Launch flag word from NE_CREATE_VM2, valid when
+ *			@payload_version >= 1.
  */
 struct slot_alloc_req {
-	u8	unused;
+	u8	payload_version;
+	u8	reserved[3];
+	u32	flags;
 };
 
+/*
+ * The device picks these fields out of the payload by offset, so a change
+ * to the layout here changes the SLOT_ALLOC wire format.
+ */
+static_assert(sizeof(struct slot_alloc_req) == 8);
+static_assert(offsetof(struct slot_alloc_req, reserved) == 1);
+static_assert(offsetof(struct slot_alloc_req, flags) == 4);
+
 /**
  * struct slot_free_req - SLOT_FREE request.
  * @slot_uid:	Slot unique id mapped to the slot to free.
diff --git a/include/uapi/linux/nitro_enclaves.h b/include/uapi/linux/nitro_enclaves.h
index 8e43be9a0a1fa..8ccf16e9cf5fb 100644
--- a/include/uapi/linux/nitro_enclaves.h
+++ b/include/uapi/linux/nitro_enclaves.h
@@ -46,6 +46,18 @@
  */
 #define NE_CREATE_VM			_IOR(0xAE, 0x20, __u64)
 
+/**
+ * DOC: NE_CREATE_VM2 launch flags
+ */
+
+/**
+ * NE_ENCLAVE_PCIE_MODE - Allocate a slot with a PCIe bus. The
+ *			  hypervisor selects PCIe backing for the slot at
+ *			  SLOT_ALLOC time, so a caller that wants it asks
+ *			  for it here and not at enclave start.
+ */
+#define NE_ENCLAVE_PCIE_MODE		(1u << 6)
+
 /**
  * struct ne_create_vm_args - NE_CREATE_VM2 argument (in / out).
  * @flags:	Launch-time flag word (in). Flags select properties of the
@@ -82,6 +94,8 @@ struct ne_create_vm_args {
  * * EFAULT				- Argument read or write failure.
  * * NE_ERR_INVALID_FLAG_VALUE		- Unknown flag bit set, or
  *					  NE_ENCLAVE_DEBUG_MODE set.
+ * * EOPNOTSUPP			- A launch flag is set and the NE PCI
+ *					  device takes no launch flags.
  * * EINVAL				- The argument stops before @slot_uid.
  * * E2BIG				- The argument carries a field this kernel
  *					  does not know, set to a non-zero value.
-- 
2.47.1

[PATCH 12/40] nitro_enclaves: Name the device's error codes in the UAPI header

[Alexander Graf]

The nitro_enclaves device refuses a command with an errno or with an
NE_ERR_ code, which ne_do_request() passes on to user space as it is.
The codes of its own that the device returns start at 276, after the
driver's, and we want the device to be able to add more of them later.

Name the device's codes in the header, so that a caller can tell for
example a refused flag from an invalid slot, and document three ranges:
the driver's original codes from 256, the device's codes from 276 to
511, and codes only the driver returns from 1024. That way new codes on
either side never collide. 511 stays below the kernel-internal error
codes, and 1024 well below MAX_ERRNO, so user space reads every code as
an error.

Signed-off-by: Alexander Graf <graf@amazon.com>
---
 include/uapi/linux/nitro_enclaves.h | 142 ++++++++++++++++++++++++++++
 1 file changed, 142 insertions(+)

diff --git a/include/uapi/linux/nitro_enclaves.h b/include/uapi/linux/nitro_enclaves.h
index 8ccf16e9cf5fb..6dd00c41c07d5 100644
--- a/include/uapi/linux/nitro_enclaves.h
+++ b/include/uapi/linux/nitro_enclaves.h
@@ -567,6 +567,148 @@ struct ne_add_memory_region_args {
  */
 #define NE_ERR_INVALID_ENCLAVE_CID		(275)
 
+/**
+ * DOC: NE error code ranges
+ *
+ * The NE_ERR_ codes come from three ranges:
+ *
+ * * 256 to 275 - The driver's original codes. The NE PCI device also
+ *   returns NE_ERR_MEM_MAX_REGIONS, with the same meaning.
+ * * 276 to 511 - Codes the NE PCI device returns. The driver passes them on
+ *   to the caller unchanged, and the device adds new codes upward from the
+ *   last one defined here. The range ends below the kernel-internal error
+ *   codes, which start at 512.
+ * * 1024 and above - Codes only the driver returns, added upward from 1024.
+ */
+
+/**
+ * NE_ERR_INVALID_FLAGS - The command's reserved flags field was not zero.
+ */
+#define NE_ERR_INVALID_FLAGS			(276)
+/**
+ * NE_ERR_INVALID_ALIGNMENT - An address or a size the command carries is
+ *			      misaligned or too small.
+ */
+#define NE_ERR_INVALID_ALIGNMENT		(277)
+/**
+ * NE_ERR_INVALID_SIZE - A size or count the command carries is zero, too large
+ *			 or too short.
+ */
+#define NE_ERR_INVALID_SIZE			(278)
+/**
+ * NE_ERR_INVALID_SLOT - The slot the command names does not exist.
+ */
+#define NE_ERR_INVALID_SLOT			(279)
+/**
+ * NE_ERR_INVALID_DEVICE - The command names a device, an address or a memory
+ *			   range the device does not accept.
+ */
+#define NE_ERR_INVALID_DEVICE			(280)
+/**
+ * NE_ERR_TOO_MANY_DEVICES - The enclave already has as many devices as it may.
+ */
+#define NE_ERR_TOO_MANY_DEVICES			(281)
+/**
+ * NE_ERR_TOO_MANY_VQS - The device already has as many queues as it may.
+ */
+#define NE_ERR_TOO_MANY_VQS			(282)
+/**
+ * NE_ERR_NO_RESOURCES - The device could not put together something of its own
+ *			 that the command needs.
+ */
+#define NE_ERR_NO_RESOURCES			(283)
+/**
+ * NE_ERR_ALREADY_EXISTS - The slot already has what the command gives it.
+ */
+#define NE_ERR_ALREADY_EXISTS			(284)
+/**
+ * NE_ERR_MISSING_INFO_PAGE - The slot has no info page, or the one it has is too
+ *			      small for what the command writes there.
+ */
+#define NE_ERR_MISSING_INFO_PAGE		(285)
+/**
+ * NE_ERR_INVALID_FLAG - A flag bit the device does not know, or does not allow
+ *			 here, was set.
+ */
+#define NE_ERR_INVALID_FLAG			(286)
+/**
+ * NE_ERR_RESERVED_FLAG - A reserved flag bit was set.
+ */
+#define NE_ERR_RESERVED_FLAG			(287)
+/**
+ * NE_ERR_MISSING_PCIE - A flag that depends on the PCIe flag was set without it.
+ */
+#define NE_ERR_MISSING_PCIE			(288)
+/**
+ * NE_ERR_INCOMPATIBLE_FLAGS - The flags asked for cannot be had together.
+ */
+#define NE_ERR_INCOMPATIBLE_FLAGS		(289)
+/**
+ * NE_ERR_EIF_FLAG_MISMATCH - The image's matched flags are not the ones the
+ *			      enclave was started with.
+ */
+#define NE_ERR_EIF_FLAG_MISMATCH		(290)
+/**
+ * NE_ERR_DEVICE_DISALLOWED - The device type asked for is not allowed.
+ */
+#define NE_ERR_DEVICE_DISALLOWED		(291)
+/**
+ * NE_ERR_HOTPLUG_DISABLED - The enclave does not take devices once it is
+ *			     running.
+ */
+#define NE_ERR_HOTPLUG_DISABLED			(292)
+/**
+ * NE_ERR_BAD_EIF - The enclave image is missing, or did not pass validation when
+ *		    the enclave was started.
+ */
+#define NE_ERR_BAD_EIF				(293)
+/**
+ * NE_ERR_MISSING_PCIE_VIRTIO - A flag that depends on the virtio-PCIe flag was
+ *				set without it.
+ */
+#define NE_ERR_MISSING_PCIE_VIRTIO		(294)
+/**
+ * NE_ERR_BAD_PAYLOAD_VERSION - The request's payload version is one the device
+ *				does not take.
+ */
+#define NE_ERR_BAD_PAYLOAD_VERSION		(295)
+/**
+ * NE_ERR_INFO_VECTORS_EXHAUSTED - No notification vector is left.
+ */
+#define NE_ERR_INFO_VECTORS_EXHAUSTED		(296)
+/**
+ * NE_ERR_TOO_MANY_ENCLAVES - The instance already has as many enclaves as its
+ *			      quota allows. A terminal answer: neither
+ *			      resending the command nor donating more memory
+ *			      changes it.
+ */
+#define NE_ERR_TOO_MANY_ENCLAVES		(297)
+/**
+ * NE_ERR_NO_MEM_SLOTS_ON_NODE - The memory budget of one NUMA node is used up,
+ *				 which -ENOSPC does not distinguish from the
+ *				 budget as a whole.
+ */
+#define NE_ERR_NO_MEM_SLOTS_ON_NODE		(298)
+/**
+ * NE_ERR_NUMA_SPAN - The enclave's CPUs and memory span more than one host NUMA
+ *		      node, and the enclave was not set up to span nodes.
+ */
+#define NE_ERR_NUMA_SPAN			(299)
+/**
+ * NE_ERR_NUMA_ALIGNMENT - The memory of a guest NUMA node other than the last
+ *			   is not a multiple of the guest's memory section size.
+ */
+#define NE_ERR_NUMA_ALIGNMENT			(300)
+/**
+ * NE_ERR_LOW_RAM_TOO_SMALL - The lowest contiguous run of enclave memory is too
+ *			      small for the boot layout the enclave needs.
+ */
+#define NE_ERR_LOW_RAM_TOO_SMALL		(301)
+/**
+ * NE_ERR_ZERO_REGION - The enclave's memory layout has a region of size zero.
+ */
+#define NE_ERR_ZERO_REGION			(302)
+
 /**
  * DOC: Image load info flags
  */
-- 
2.47.1

[PATCH 13/40] nitro_enclaves: Log a reached device limit at info level

[Alexander Graf]

When a parent asks for more enclave slots than the device has room for,
or for more enclaves than its quota allows, the device answers -ENOSPC
or NE_ERR_TOO_MANY_ENCLAVES and the driver logs an error for each
refused command, although user space gets the code back. Log the two at
info level.

Signed-off-by: Alexander Graf <graf@amazon.com>
---
 drivers/virt/nitro_enclaves/ne_pci_dev.c | 14 ++++++++++++--
 1 file changed, 12 insertions(+), 2 deletions(-)

diff --git a/drivers/virt/nitro_enclaves/ne_pci_dev.c b/drivers/virt/nitro_enclaves/ne_pci_dev.c
index 2d44cae4ac0cf..f258e04ee9e47 100644
--- a/drivers/virt/nitro_enclaves/ne_pci_dev.c
+++ b/drivers/virt/nitro_enclaves/ne_pci_dev.c
@@ -261,8 +261,18 @@ int ne_do_request(struct pci_dev *pdev, enum ne_pci_dev_cmd_type cmd_type,
 	if (cmd_reply->rc < 0) {
 		rc = cmd_reply->rc;
 
-		dev_err_ratelimited(&pdev->dev, "Error in cmd process logic, cmd type=%u [rc=%d]\n",
-				    cmd_type, rc);
+		if (rc == -ENOSPC) {
+			dev_info_ratelimited(&pdev->dev,
+					     "No room for cmd type=%u [rc=%d]\n",
+					     cmd_type, rc);
+		} else if (rc == -NE_ERR_TOO_MANY_ENCLAVES) {
+			dev_info_ratelimited(&pdev->dev,
+					     "Enclave quota reached for cmd type=%u [rc=%d]\n",
+					     cmd_type, rc);
+		} else {
+			dev_err_ratelimited(&pdev->dev, "Error in cmd process logic, cmd type=%u [rc=%d]\n",
+					    cmd_type, rc);
+		}
 
 		goto unlock_mutex;
 	}
-- 
2.47.1

[PATCH 14/40] nitro_enclaves: Add NE_SET_INFO_PAGE

[Alexander Graf]

Nitro Sandboxes introduce the info page, a page of parent memory that
the device writes and the parent reads, to share various information about
the state of devices inside the slot.

Add the NE_SET_INFO_PAGE ioctl to register a NE_MEMORY_SHARED allocation
as info page. This page is always read-only to user space.

Signed-off-by: Alexander Graf <graf@amazon.com>
---
 drivers/virt/nitro_enclaves/ne_misc_dev.c | 276 +++++++++++++++++++++-
 drivers/virt/nitro_enclaves/ne_misc_dev.h |  35 ++-
 drivers/virt/nitro_enclaves/ne_pci_dev.c  |  25 ++
 drivers/virt/nitro_enclaves/ne_pci_dev.h  |  21 ++
 include/uapi/linux/nitro_enclaves.h       | 106 ++++++++-
 5 files changed, 435 insertions(+), 28 deletions(-)

diff --git a/drivers/virt/nitro_enclaves/ne_misc_dev.c b/drivers/virt/nitro_enclaves/ne_misc_dev.c
index 0bf62f703806e..610d98264e5a1 100644
--- a/drivers/virt/nitro_enclaves/ne_misc_dev.c
+++ b/drivers/virt/nitro_enclaves/ne_misc_dev.c
@@ -123,6 +123,8 @@ static int ne_add_memory_region_ioctl(struct ne_enclave *ne_enclave,
 				      struct ne_add_memory_region_args *args);
 static int ne_alloc_memory_ioctl(struct ne_enclave *ne_enclave,
 				 struct ne_alloc_memory_args *args);
+static int ne_set_info_page_ioctl(struct ne_enclave *ne_enclave,
+				  struct ne_set_info_page_args *args);
 static const struct file_operations ne_fops = {
 	.owner		= THIS_MODULE,
 	.llseek		= noop_llseek,
@@ -2137,6 +2139,39 @@ static long ne_enclave_do_ioctl(struct file *file, unsigned int cmd, unsigned lo
 		return rc;
 	}
 
+	case NE_EA_IOCTL(NE_SET_INFO_PAGE): {
+		struct ne_set_info_page_args __user *uargs = (void __user *)arg;
+		unsigned int usize = _IOC_SIZE(cmd);
+		struct ne_set_info_page_args args;
+		int rc;
+
+		if (usize < offsetofend(struct ne_set_info_page_args, flags))
+			return -EINVAL;
+
+		rc = copy_struct_from_user(&args, sizeof(args), uargs, usize);
+		if (rc)
+			return rc;
+
+		rc = mutex_lock_killable(&ne_enclave->enclave_info_mutex);
+		if (rc)
+			return rc;
+
+		if (ne_enclave->state != NE_STATE_INIT) {
+			dev_err_ratelimited(ne_misc_dev.this_device,
+					    "Enclave is not in init state\n");
+
+			mutex_unlock(&ne_enclave->enclave_info_mutex);
+
+			return -NE_ERR_NOT_IN_INIT_STATE;
+		}
+
+		rc = ne_set_info_page_ioctl(ne_enclave, &args);
+
+		mutex_unlock(&ne_enclave->enclave_info_mutex);
+
+		return rc;
+	}
+
 	default:
 		return -ENOTTY;
 	}
@@ -2344,6 +2379,9 @@ static void ne_enclave_free(struct ne_enclave *ne_enclave)
 	ne_enclave_remove_all_mem_region_entries(ne_enclave);
 	ne_enclave_remove_all_vcpu_id_entries(ne_enclave);
 
+	if (ne_enclave->info_file)
+		fput(ne_enclave->info_file);
+
 	kfree(ne_enclave);
 }
 
@@ -3152,6 +3190,9 @@ static bool ne_contig_supports_mmap(struct file *file)
  * asks for VM_HUGEPAGE because a fault on a VM_PFNMAP mapping without it only
  * reaches ->huge_fault where the transparent_hugepage policy is "always".
  *
+ * An allocation that is a slot's info page is mapped without write access, and
+ * without the right to gain it with mprotect().
+ *
  * Return: 0 on success, negative return value on failure.
  */
 static int ne_contig_mmap_prepare(struct vm_area_desc *desc)
@@ -3177,6 +3218,13 @@ static int ne_contig_mmap_prepare(struct vm_area_desc *desc)
 	if (offset > mem->size || size > mem->size - offset)
 		return -EINVAL;
 
+	if (READ_ONCE(mem->read_only)) {
+		if (vma_desc_test(desc, VMA_WRITE_BIT))
+			return -EACCES;
+
+		vma_desc_clear_flags(desc, VMA_MAYWRITE_BIT);
+	}
+
 	desc->vm_ops = &ne_contig_vm_ops;
 	vma_desc_set_flags(desc, VMA_IO_BIT, VMA_PFNMAP_BIT, VMA_DONTEXPAND_BIT,
 			   VMA_DONTDUMP_BIT, VMA_DONTCOPY_BIT, VMA_HUGEPAGE_BIT);
@@ -3242,11 +3290,12 @@ static int ne_write_span(void *arg, struct page *page, u64 off, u64 len)
  * refusal is -EFBIG, the same as generic_write_check_limits() gives for a write
  * beyond s_maxbytes. There is no RLIMIT_FSIZE in it, so no SIGXFSZ either.
  *
- * Donating an allocation closes this path, and @lock is what makes the two
- * orderly. It is taken here and in ne_add_memory_region_ioctl() and nowhere
- * else; setting a mapping up and faulting one in both read only fields that
- * never change, so nothing takes it under mmap_write_lock or in a fault, and
- * copying from user space under it cannot invert against the fault path.
+ * Donating an allocation closes this path, and so does making it an info page.
+ * This function, ne_add_memory_region_ioctl(), ne_contig_make_read_only() and
+ * ne_contig_allow_writes() take @lock, which orders those state changes against
+ * a write. Setting a mapping up and faulting one in do not take it, so nothing
+ * takes it under mmap_write_lock or in a fault, and copying from user space
+ * under it cannot invert against the fault path.
  *
  * Return: the number of bytes written, or a negative return value on failure.
  */
@@ -3266,7 +3315,7 @@ static ssize_t ne_contig_write_iter(struct kiocb *iocb, struct iov_iter *from)
 
 	guard(mutex)(&mem->lock);
 
-	if (mem->donated)
+	if (mem->donated || mem->read_only)
 		return -EPERM;
 
 	rc = ne_walk_allocs(mem, pos, count, NE_WRITE_SPAN_PAGES, ne_write_span,
@@ -3281,6 +3330,60 @@ static ssize_t ne_contig_write_iter(struct kiocb *iocb, struct iov_iter *from)
 	return rc;
 }
 
+/**
+ * ne_contig_make_read_only() - Take write access to an allocation away from
+ *				user space.
+ * @file:	The allocation's file.
+ *
+ * From here on a write to the descriptor fails and a new mapping has no write
+ * access, so only the device and the kernel change the memory. The descriptor
+ * is open for writing, so a mapping made before could be made writable with
+ * mprotect(), and one that exists makes this fail.
+ *
+ * mmap_region() raises i_mmap_writable with full ordering before it calls
+ * ne_contig_mmap_prepare() for a mapping that may write, and the mapping keeps
+ * it raised for as long as it exists. @read_only is set before the count is
+ * read, so a mapping that ne_contig_mmap_prepare() set up without seeing it is
+ * counted here. A mapping that is being set up as this runs counts too, even
+ * one that ends up without write access, so a caller sees -EBUSY for it and
+ * can try again.
+ *
+ * Return: 0 on success, -EBUSY if the allocation is mapped.
+ */
+static int ne_contig_make_read_only(struct file *file)
+{
+	struct ne_contig_mem *mem = file->private_data;
+
+	guard(mutex)(&mem->lock);
+
+	WRITE_ONCE(mem->read_only, true);
+
+	/* Pairs with the full ordering of mapping_map_writable(). */
+	smp_mb();
+
+	if (mapping_writably_mapped(file->f_mapping)) {
+		WRITE_ONCE(mem->read_only, false);
+
+		return -EBUSY;
+	}
+
+	return 0;
+}
+
+/**
+ * ne_contig_allow_writes() - Give write access to an allocation back to user
+ *			      space.
+ * @file:	The allocation's file.
+ */
+static void ne_contig_allow_writes(struct file *file)
+{
+	struct ne_contig_mem *mem = file->private_data;
+
+	guard(mutex)(&mem->lock);
+
+	WRITE_ONCE(mem->read_only, false);
+}
+
 /**
  * ne_contig_release() - Free an allocation once nothing holds it any more.
  * @inode:	The allocation's inode, unused.
@@ -3612,6 +3715,13 @@ static int ne_add_memory_region_ioctl(struct ne_enclave *ne_enclave,
 		return -EINVAL;
 	}
 
+	if (fd_file(f) == ne_enclave->info_file) {
+		dev_err_ratelimited(ne_misc_dev.this_device,
+				    "The allocation is the enclave's info page\n");
+
+		return -NE_ERR_MEM_REGION_ALREADY_USED;
+	}
+
 	list_for_each_entry(held, &ne_enclave->contig_regions_list, list) {
 		if (held->file == fd_file(f)) {
 			dev_err_ratelimited(ne_misc_dev.this_device,
@@ -3714,6 +3824,160 @@ static int ne_add_memory_region_ioctl(struct ne_enclave *ne_enclave,
 	return rc;
 }
 
+/**
+ * ne_set_info_page_ioctl() - Give the slot associated with the current enclave
+ *			      the info page.
+ * @ne_enclave:	Private data associated with the current enclave.
+ * @args:	Descriptor for the allocation, and flags.
+ *
+ * The device writes the info page, so the run behind it has to stay where the
+ * device was told it is for as long as the enclave lives. The runs belong to the
+ * file, so a reference on it keeps them whatever user space does with the
+ * descriptor and its mappings, and ne_enclave_free() drops that reference
+ * after SLOT_FREE and after the enclave's memory regions. The reference is taken
+ * before the request, since a request that timed out may still be completed by
+ * the device. A rejected request drops it again.
+ *
+ * Only the device writes the page, and the parent reads back from it what the
+ * device reports about the slot, so user space loses write access to the
+ * allocation before the device is told about it and gets it back only if the
+ * device rejects the request.
+ *
+ * Context: Process context. This function is called with the ne_enclave mutex held.
+ * Return:
+ * * 0 on success.
+ * * Negative return value on failure.
+ * * -ETIMEDOUT if the device did not answer, in which case the slot keeps the
+ *   allocation until the enclave goes away.
+ */
+static int ne_set_info_page_ioctl(struct ne_enclave *ne_enclave,
+				  struct ne_set_info_page_args *args)
+{
+	struct ne_pci_dev_cmd_reply cmd_reply = {};
+	struct slot_add_info_page_req req = {};
+	struct pci_dev *pdev = ne_devs.ne_pci_dev->pdev;
+	struct ne_contig_mem *mem = NULL;
+	int rc = -EINVAL;
+
+	if (ne_enclave->mm != current->mm)
+		return -EIO;
+
+	if (args->flags) {
+		dev_err_ratelimited(ne_misc_dev.this_device,
+				    "Incorrect flag for info page\n");
+
+		return -NE_ERR_INVALID_FLAG_VALUE;
+	}
+
+	if (ne_enclave->info_file) {
+		dev_err_ratelimited(ne_misc_dev.this_device,
+				    "The slot already has an info page\n");
+
+		return -EEXIST;
+	}
+
+	CLASS(fd, f)(args->fd);
+
+	if (fd_empty(f))
+		return -EBADF;
+
+	if (fd_file(f)->f_op != &ne_contig_fops) {
+		dev_err_ratelimited(ne_misc_dev.this_device,
+				    "Not an enclave memory allocation\n");
+
+		return -EINVAL;
+	}
+
+	mem = fd_file(f)->private_data;
+
+	if (mem->enclave_id != ne_enclave->id) {
+		dev_err_ratelimited(ne_misc_dev.this_device,
+				    "Allocation was made for another enclave\n");
+
+		return -EINVAL;
+	}
+
+	if (!mem->shared) {
+		dev_err_ratelimited(ne_misc_dev.this_device,
+				    "The info page allocation is not shared\n");
+
+		return -EINVAL;
+	}
+
+	/*
+	 * The device takes one address and one length, so an allocation the
+	 * allocator had to satisfy in several runs has no single address to
+	 * name. A request of NE_MIN_MEM_REGION_SIZE cannot be split, since the
+	 * allocator has nothing smaller to fall back to.
+	 */
+	if (mem->nr_allocs != 1) {
+		dev_err_ratelimited(ne_misc_dev.this_device,
+				    "The info page allocation is not one run\n");
+
+		return -EINVAL;
+	}
+
+	/*
+	 * An allocation the enclave already has as memory is not available to
+	 * be its info page as well, since the device would be handed the same
+	 * run twice in two roles. @donated is only ever set from
+	 * ne_add_memory_region_ioctl(), which runs under the same enclave mutex
+	 * as this function for the one enclave the allocation was made for, so
+	 * the read needs no lock of its own.
+	 */
+	if (mem->donated) {
+		dev_err_ratelimited(ne_misc_dev.this_device,
+				    "The info page allocation was donated\n");
+
+		return -NE_ERR_MEM_REGION_ALREADY_USED;
+	}
+
+	rc = ne_contig_make_read_only(fd_file(f));
+	if (rc < 0) {
+		dev_err_ratelimited(ne_misc_dev.this_device,
+				    "The info page allocation is mapped\n");
+
+		return rc;
+	}
+
+	req.slot_uid = ne_enclave->slot_uid;
+	req.info_phys = page_to_phys(mem->allocs[0].page);
+	req.info_size = mem->size;
+
+	/*
+	 * Take the reference before the request, so that the run stays where the
+	 * device is told it is from the moment the device is told, as the memory
+	 * region path does.
+	 */
+	ne_enclave->info_file = get_file(fd_file(f));
+
+	rc = ne_do_request(pdev, SLOT_ADD_INFO_PAGE,
+			   &req, sizeof(req), &cmd_reply, sizeof(cmd_reply));
+	if (rc < 0) {
+		dev_err_ratelimited(ne_misc_dev.this_device,
+				    "Error in slot add info page [rc=%d]\n", rc);
+
+		/*
+		 * A request the device never answered was submitted with the
+		 * run's physical address in it, so the device may write the page
+		 * yet. Keep the reference and let ne_enclave_free() drop it
+		 * after SLOT_FREE. Every other error means the device either
+		 * never saw the request or answered to reject it, and the run is
+		 * the parent's again. ne_request_unanswered() tells the two
+		 * apart.
+		 */
+		if (!ne_request_unanswered(rc)) {
+			ne_contig_allow_writes(ne_enclave->info_file);
+			fput(ne_enclave->info_file);
+			ne_enclave->info_file = NULL;
+		}
+
+		return rc;
+	}
+
+	return 0;
+}
+
 #if defined(CONFIG_NITRO_ENCLAVES_MISC_DEV_TEST)
 #include "ne_misc_dev_test.c"
 #endif
diff --git a/drivers/virt/nitro_enclaves/ne_misc_dev.h b/drivers/virt/nitro_enclaves/ne_misc_dev.h
index 2502cca3f3f64..d0d1973bfbe9d 100644
--- a/drivers/virt/nitro_enclaves/ne_misc_dev.h
+++ b/drivers/virt/nitro_enclaves/ne_misc_dev.h
@@ -38,27 +38,37 @@ struct ne_contig_alloc {
  * @allocs:	Contiguous runs backing the allocation, ordered by offset.
  * @nr_allocs:	Number of runs in @allocs.
  * @size:	Size of the allocation in bytes, mirrored in i_size.
- * @shared:	Whether the allocation may be mapped by the caller.
+ * @shared:	Whether the caller asked for a shared allocation. A shared one may
+ *		be mapped by the caller and may be a slot's info page; a private
+ *		one is neither.
  * @enclave_id:	Id of the enclave the allocation was made for.
  * @mm:		The mm charged for the pinned pages, held via mmgrab().
  * @nr_pinned:	Pages charged to @mm->pinned_vm.
- * @lock:	Serialises a write against the donation that ends writing.
+ * @lock:	Serialises a write against the donation or the info page that
+ *		ends writing.
  * @donated:	Whether the allocation has been given to its enclave.
+ * @read_only:	Whether user space has lost write access to the allocation,
+ *		which it does when the allocation becomes a slot's info page.
  *
  * The runs hang off the file, so that closing the last descriptor does not free
  * memory an enclave is still using: a donated region holds the file with
- * get_file(). That is also what keeps a mapping's page table entries from
- * outliving the runs, since a VMA holds vm_file for as long as it exists.
+ * get_file(), and so does the enclave the allocation is the info page of. That
+ * is also what keeps a mapping's page table entries from outliving the runs,
+ * since a VMA holds vm_file for as long as it exists.
  *
  * The array is built before the file is published and not mutated afterwards, so
  * any holder of a reference can read it without a lock of its own, which is what
  * lets a page fault resolve an offset to a run. @shared and @size are set in the
- * same place and are read the same way. @donated is the one field that changes
- * after publication, and @lock covers it.
+ * same place and are read the same way. @donated and @read_only are the fields
+ * that change after publication. @lock covers both on the write path;
+ * ne_set_info_page_ioctl() reads @donated under the enclave mutex its writer also
+ * holds, and ne_contig_mmap_prepare() reads @read_only without a lock, which
+ * ne_contig_make_read_only() orders against.
  *
  * @enclave_id is compared, never dereferenced, and the counter it comes from is
- * 64 bits wide, so an allocation cannot be donated to a later enclave that
- * happens to have been placed at the address of the one it was made for.
+ * 64 bits wide, so an allocation cannot be donated to, or become the info page
+ * of, a later enclave that happens to have been placed at the address of the one
+ * it was made for.
  */
 struct ne_contig_mem {
 	struct ne_contig_alloc	*allocs;
@@ -68,9 +78,10 @@ struct ne_contig_mem {
 	u64			enclave_id;
 	struct mm_struct	*mm;
 	unsigned long		nr_pinned;
-	/* Serialises a write against the donation that ends writing. */
+	/* Serialises a write against what ends writing. */
 	struct mutex		lock;
 	bool			donated;
+	bool			read_only;
 };
 
 /**
@@ -116,6 +127,11 @@ struct ne_mem_region {
  * @id:				Id of this enclave, unique for the lifetime of the
  *				driver, recorded in an %NE_ALLOC_MEMORY allocation
  *				as the enclave it was made for.
+ * @info_file:			File of the %NE_ALLOC_MEMORY allocation the slot
+ *				was given as its info page, held with get_file()
+ *				so that the device keeps the page after user space
+ *				has closed the descriptor. NULL until
+ *				%NE_SET_INFO_PAGE is called.
  * @max_mem_regions:		The maximum number of memory regions that can be
  *				handled by the hypervisor.
  * @mem_regions_list:		Enclave user space memory regions list.
@@ -146,6 +162,7 @@ struct ne_enclave {
 	wait_queue_head_t	eventq;
 	bool			has_event;
 	u64			id;
+	struct file		*info_file;
 	u64			max_mem_regions;
 	struct list_head	mem_regions_list;
 	u64			mem_size;
diff --git a/drivers/virt/nitro_enclaves/ne_pci_dev.c b/drivers/virt/nitro_enclaves/ne_pci_dev.c
index f258e04ee9e47..a07dbc6d8a9b7 100644
--- a/drivers/virt/nitro_enclaves/ne_pci_dev.c
+++ b/drivers/virt/nitro_enclaves/ne_pci_dev.c
@@ -128,6 +128,31 @@ static int ne_wait_for_reply(struct pci_dev *pdev, unsigned int timeout_msecs)
 	return 0;
 }
 
+/**
+ * ne_request_unanswered() - Whether a failure left a submitted request in
+ *			     flight.
+ * @rc:		Failure a command request came back with.
+ *
+ * A request the device never answered was submitted with whatever the caller
+ * put in it, and there is nothing to cancel it with, so a resource whose
+ * address is in it may still be written by the device. A caller that has to
+ * decide who owns such a resource asks here instead of naming an errno of its
+ * own.
+ *
+ * Every code the wait for a reply can fail a submitted request with belongs
+ * here. ne_wait_for_reply() times out, so -ETIMEDOUT is named. A code raised
+ * before ne_submit_request(), and a code the device answered with, do not:
+ * nothing is in flight in either case. A
+ * device that answers one of the codes named here of its own accord lands here
+ * too and costs one held resource, which is the side to be wrong on.
+ *
+ * Return: true if the request may still be in flight.
+ */
+bool ne_request_unanswered(int rc)
+{
+	return rc == -ETIMEDOUT;
+}
+
 /**
  * ne_cmd_frees_slot() - Whether a command is one of the two that free a slot.
  * @cmd_type:	Command type of the request.
diff --git a/drivers/virt/nitro_enclaves/ne_pci_dev.h b/drivers/virt/nitro_enclaves/ne_pci_dev.h
index b0957e2606ec3..4f670c53dd3c0 100644
--- a/drivers/virt/nitro_enclaves/ne_pci_dev.h
+++ b/drivers/virt/nitro_enclaves/ne_pci_dev.h
@@ -123,6 +123,8 @@
  * @NEXT_SLOT:			Get the next slot in the list of allocated slots.
  * @SLOT_INFO:			Get the info for a slot e.g. slot uid, vCPUs count.
  * @SLOT_ADD_BULK_VCPUS:	Add a number of vCPUs, not providing CPU ids.
+ * @SLOT_ADD_INFO_PAGE:		Give a slot the info page, a memory region the
+ *				device writes.
  * @MAX_CMD:			A gatekeeper for max possible command type.
  */
 enum ne_pci_dev_cmd_type {
@@ -138,6 +140,8 @@ enum ne_pci_dev_cmd_type {
 	NEXT_SLOT		= 9,
 	SLOT_INFO		= 10,
 	SLOT_ADD_BULK_VCPUS	= 11,
+	/* 12 is a command this driver does not implement. */
+	SLOT_ADD_INFO_PAGE	= 13,
 	MAX_CMD,
 };
 
@@ -266,6 +270,20 @@ struct slot_add_bulk_vcpus_req {
 	u64	nr_vcpus;
 };
 
+/**
+ * struct slot_add_info_page_req - SLOT_ADD_INFO_PAGE request.
+ * @slot_uid:	Slot unique id mapped to the slot to give the info page to.
+ * @flags:	Flags for the info page. Zero is the only value the driver sends.
+ * @info_phys:	Physical address of the info page.
+ * @info_size:	Size, in bytes, of the info page.
+ */
+struct slot_add_info_page_req {
+	u64	slot_uid;
+	u64	flags;
+	u64	info_phys;
+	u64	info_size;
+};
+
 /**
  * struct ne_pci_dev_cmd_reply - NE PCI device command reply.
  * @rc :		Return code of the logic that processed the request.
@@ -327,6 +345,9 @@ struct ne_pci_dev {
 	struct pci_dev		*pdev;
 };
 
+/* Whether a failure left a submitted request in flight. */
+bool ne_request_unanswered(int rc);
+
 /**
  * ne_do_request() - Submit command request to the PCI device based on the command
  *		     type and retrieve the associated reply.
diff --git a/include/uapi/linux/nitro_enclaves.h b/include/uapi/linux/nitro_enclaves.h
index 6dd00c41c07d5..4f604712983f1 100644
--- a/include/uapi/linux/nitro_enclaves.h
+++ b/include/uapi/linux/nitro_enclaves.h
@@ -287,13 +287,15 @@ struct ne_alloc_memory_args {
 
 /**
  * NE_ALLOC_MEMORY - The command is used to allocate physically contiguous
- *		     memory to give to an enclave with %NE_ADD_MEMORY_REGION,
- *		     drawn from one host NUMA node. The argument is input only;
+ *		     memory to give to an enclave with %NE_ADD_MEMORY_REGION or
+ *		     to provide to the device with %NE_SET_INFO_PAGE, drawn from
+ *		     one host NUMA node. The argument is input only;
  *		     on success the return value is a file descriptor for the
  *		     allocation.
  *		     Writing to the descriptor places bytes in the allocation,
  *		     which is how an enclave image gets there, and stops
- *		     working once the allocation has been donated.
+ *		     working once the allocation has been donated or provided
+ *		     with %NE_SET_INFO_PAGE.
  *		     O_APPEND has no effect. There is no read operation, so
  *		     read() and pread() both fail with EINVAL and a caller that
  *		     wants to check what it wrote keeps its own copy.
@@ -308,16 +310,18 @@ struct ne_alloc_memory_args {
  *		     allocation over, including once the allocation has been
  *		     donated and writing to the descriptor has stopped working.
  *		     Such a mapping is readable too, which is the only way to
- *		     read allocated memory back. The mapping is not inherited
- *		     across fork(); a child that needs one maps the descriptor
- *		     itself.
+ *		     read allocated memory back, and is how a parent reads an
+ *		     info page provided with %NE_SET_INFO_PAGE, which it maps
+ *		     without %PROT_WRITE. The mapping is not inherited across
+ *		     fork(); a child that needs one maps the descriptor itself.
  *		     Donating does not take a mapping away, so a parent that
  *		     mapped a shared allocation can still write to it through
  *		     that mapping afterwards.
  *		     The allocation lives until the last of the descriptor, its
- *		     mappings and the enclave regions donated from it is gone,
- *		     so a caller can close the descriptor as soon as it has
- *		     donated.
+ *		     mappings, the enclave regions donated from it and the
+ *		     enclave it is the info page of is gone, so a caller can
+ *		     close the descriptor as soon as it has donated it or
+ *		     provided it to the device.
  *		     The ioctl can be invoked on the enclave fd, before an
  *		     enclave is started.
  *
@@ -351,7 +355,8 @@ struct ne_alloc_memory_args {
  *
  * Writing to the descriptor sets errno to:
  * * EPERM			- The allocation has been donated with
- *				  %NE_ADD_MEMORY_REGION.
+ *				  %NE_ADD_MEMORY_REGION or provided with
+ *				  %NE_SET_INFO_PAGE.
  * * EFBIG			- The write does not fit inside the allocation.
  * * EFAULT			- Part of the buffer is not readable. Any bytes
  *				  copied before that are reported as a short
@@ -361,6 +366,9 @@ struct ne_alloc_memory_args {
  *
  * An mmap() of the descriptor sets errno to:
  * * ENODEV			- The allocation is %NE_MEMORY_PRIVATE.
+ * * EACCES			- The mapping asks for %PROT_WRITE and the
+ *				  allocation has been provided with
+ *				  %NE_SET_INFO_PAGE.
  * * EINVAL			- The mapping is %MAP_PRIVATE, or its offset or
  *				  length is not a multiple of 2 MiB, or it does
  *				  not fit inside the allocation.
@@ -388,7 +396,8 @@ struct ne_add_memory_region_args {
  *			  contiguous run it is made of, or fewer where runs turn
  *			  out to be physically adjacent. The argument is input
  *			  only. The allocation has to have been made for this
- *			  enclave, and can be given away once.
+ *			  enclave, cannot be its info page, and can be given away
+ *			  once.
  *			  Writing to the descriptor stops working here, so an
  *			  enclave image goes in before this call. The allocation
  *			  has to be %NE_MEMORY_PRIVATE, since memory the parent
@@ -425,13 +434,82 @@ struct ne_add_memory_region_args {
  *					  before being started).
  * * NE_ERR_INVALID_FLAG_VALUE		- The value of the provided flag is invalid.
  * * NE_ERR_MEM_REGION_ALREADY_USED	- The allocation was already given to the
- *					  enclave.
+ *					  enclave, or is its info page.
  * * NE_ERR_MEM_MAX_REGIONS		- The number of memory regions set for the
  *					  enclave reached maximum.
  * * Error codes from the NE PCI device request.
  */
 #define NE_ADD_MEMORY_REGION		_IOW(0xAE, 0x2F, struct ne_add_memory_region_args)
 
+/**
+ * struct ne_set_info_page_args - Allocation to provide to the device (in).
+ * @fd:		Descriptor from %NE_ALLOC_MEMORY, made for this enclave, asked
+ *		for as %NE_MEMORY_SHARED, and made of one contiguous run.
+ * @flags:	Must be zero.
+ */
+struct ne_set_info_page_args {
+	__s32	fd;
+	__u32	flags;
+};
+
+/**
+ * NE_SET_INFO_PAGE - The command is used to give an enclave's slot the info
+ *		      page, a %NE_ALLOC_MEMORY allocation the device writes. The
+ *		      argument is input only. The allocation has to be
+ *		      %NE_MEMORY_SHARED, since a mapping is the only way for
+ *		      the parent to read an allocation back. It also has to be
+ *		      one contiguous run, since the device is given one address
+ *		      and one length for it, and a request of 2 MiB is the one
+ *		      size the allocator cannot come back from in several runs.
+ *		      The info page is not enclave memory: it does not count
+ *		      towards the memory regions %NE_ADD_MEMORY_REGION fills.
+ *		      Only the device writes the page, so from this call on the
+ *		      allocation is read-only to user space: writing to the
+ *		      descriptor fails, and the descriptor is mapped without
+ *		      %PROT_WRITE. A mapping made before the call could still be
+ *		      made writable, so the call fails while one exists, and a
+ *		      caller maps the page after the call. A call the device
+ *		      rejects leaves the allocation writable again.
+ *		      A slot takes one info page, and an allocation is either an
+ *		      enclave's memory or its info page. The driver holds the
+ *		      allocation until the enclave is gone, so closing the
+ *		      descriptor does not free the run the device was given.
+ *		      The ioctl can be invoked on the enclave fd, before an
+ *		      enclave is started.
+ *
+ * Context: Process context.
+ * Return:
+ * * 0					- Logic successfully completed.
+ * * -1					- There was a failure in the ioctl logic.
+ * On failure, errno is set to:
+ * * ENODEV				- The NE PCI device is not available.
+ * * EFAULT				- Argument read failure.
+ * * EBADF				- @fd is not an open file descriptor.
+ * * EINVAL				- The argument is shorter than @flags, or @fd
+ *					  is not a descriptor from %NE_ALLOC_MEMORY,
+ *					  or the allocation was made for another
+ *					  enclave, or it is %NE_MEMORY_PRIVATE, or it
+ *					  is made of more than one contiguous run.
+ * * EIO				- Current task mm is not the same as the one
+ *					  that created the enclave.
+ * * EEXIST				- The slot already has an info page.
+ * * EBUSY				- The allocation has a mapping made before the
+ *					  call, or one is being set up.
+ * * E2BIG				- The argument is longer than this kernel knows
+ *					  about and the excess is not all zero.
+ * * NE_ERR_NOT_IN_INIT_STATE		- The enclave is not in init state (init =
+ *					  before being started).
+ * * NE_ERR_INVALID_FLAG_VALUE		- The value of the provided flag is invalid.
+ * * NE_ERR_MEM_REGION_ALREADY_USED	- The allocation was already given to the
+ *					  enclave with %NE_ADD_MEMORY_REGION.
+ * * ETIMEDOUT				- The device did not answer. The slot keeps
+ *					  the allocation, read-only, until the
+ *					  enclave is gone, since the device may
+ *					  still write the page.
+ * * Error codes from the NE PCI device request.
+ */
+#define NE_SET_INFO_PAGE		_IOW(0xAE, 0x25, struct ne_set_info_page_args)
+
 /**
  * NE_START_ENCLAVE - The command is used to trigger enclave start after the
  *		      enclave resources, such as memory and CPU, have been set.
@@ -775,7 +853,9 @@ struct ne_user_memory_region {
  *		      it has been donated. It costs page tables for the part of
  *		      it a caller touches, and it leaves the memory reachable
  *		      from the parent for as long as it lives, so an allocation
- *		      that is only being handed over does not ask for one.
+ *		      that is only being handed over does not ask for one. It is
+ *		      also what an allocation given to %NE_SET_INFO_PAGE has to
+ *		      be.
  */
 #define NE_MEMORY_SHARED		(0x01)
 
-- 
2.47.1

[PATCH 15/40] nitro_enclaves: Document the info page in the memory pool

[Alexander Graf]

An info page takes one contiguous allocation from the same CMA
reservation as enclave memory, so an admin reserving the pool needs to
leave room for it. Document its size, that it is registered before it is
mapped and is read-only to user space from then on, and that it keeps
its allocation alive like a memory region does. Also say that a larger
NE_ALLOC_MEMORY request comes back in several runs where it has to.

Signed-off-by: Alexander Graf <graf@amazon.com>
---
 Documentation/virt/ne_overview.rst | 28 +++++++++++++++++++++-------
 1 file changed, 21 insertions(+), 7 deletions(-)

diff --git a/Documentation/virt/ne_overview.rst b/Documentation/virt/ne_overview.rst
index 4c4b7febaff0a..3cfc50f788433 100644
--- a/Documentation/virt/ne_overview.rst
+++ b/Documentation/virt/ne_overview.rst
@@ -150,10 +150,11 @@ node enables to name the node a reservation lands on. cma= needs neither, so
 on a host with one online node the driver's own select is the whole
 requirement.
 
-NE_ALLOC_MEMORY on the enclave file descriptor allocates 2 MiB aligned,
-physically contiguous memory from one host NUMA node's CMA area. A request
-that area cannot serve fails with -ENOMEM, so the reservation bounds what the
-enclaves on a node can take between them.
+NE_ALLOC_MEMORY on the enclave file descriptor allocates 2 MiB aligned memory
+from one host NUMA node's CMA area, in one physically contiguous run where it
+can and several where it cannot. A request that area cannot serve fails with
+-ENOMEM, so the reservation bounds what the enclaves on a node can take
+between them.
 It returns a file descriptor for the allocation. For an enclave whose cores
 all sit on one node the ioctl refuses any other node, so name the same node
 here and in NE_ADD_ANY_VCPU. An enclave built out of a pool that spans nodes
@@ -195,9 +196,22 @@ a caller reaches and not for all of it. It keeps the memory reachable from the
 parent for as long as it lives, so an allocation that is only being handed to
 an enclave stays private.
 
+An enclave can also have an info page: an allocation the NE PCI device writes,
+provided to the device by NE_SET_INFO_PAGE. It does not become the enclave's
+memory. Only the device writes it, so from NE_SET_INFO_PAGE on it is read-only
+to user space: write() on the descriptor fails and a mapping has no write
+access. The call fails while a mapping made before it exists, so a caller
+maps the page after it. It is asked for as NE_MEMORY_SHARED, since a mapping is
+the only way for the parent to read back what the device wrote. The device is
+given one address and one length for it, so it has to be one contiguous run,
+and 2 MiB is the only size that cannot come back split. Ask for 2 MiB per
+enclave that will have one, on the node its cores sit on, or on every node it
+may draw from where the pool spans nodes.
+
 NE_SET_USER_MEMORY_REGION is unchanged and still takes the address of a
 hugetlbfs region.
 
-The allocation lives until the last of its descriptor, its mappings and the
-regions given from it is gone, so neither closing the descriptor nor
-unmapping takes memory away from a running enclave.
+An allocation lives until the last of the things holding it is gone: its
+descriptor, its mappings, the regions given from it and the enclave it is the
+info page of. So neither closing the descriptor nor unmapping takes memory away
+from a running enclave.
-- 
2.47.1

[PATCH 16/40] nitro_enclaves: Turn the command reply into a per-command union

[Alexander Graf]

In preparation to support Nitro Sandboxes, whose commands carry
unrelated payloads in their replies, convert struct ne_pci_dev_cmd_reply
into a per-command union. A command with fields of its own in the reply
gets an arm holding them, and a caller reads the reply through the arm
for the command it issued.

The device writes every reply into the same 72 byte window of the PCI
BAR and picks the offset of each field itself, so we place an arm's
fields at the offsets the flat struct gave them and name the holes where
another command owns the bytes. static_assert() on every offset and on
the total size pins each arm to the layout the device uses.

Signed-off-by: Alexander Graf <graf@amazon.com>
---
 drivers/virt/nitro_enclaves/ne_misc_dev.c |  6 +-
 drivers/virt/nitro_enclaves/ne_pci_dev.c  |  4 +-
 drivers/virt/nitro_enclaves/ne_pci_dev.h  | 86 +++++++++++++++++------
 3 files changed, 71 insertions(+), 25 deletions(-)

diff --git a/drivers/virt/nitro_enclaves/ne_misc_dev.c b/drivers/virt/nitro_enclaves/ne_misc_dev.c
index 610d98264e5a1..77b15eda8d120 100644
--- a/drivers/virt/nitro_enclaves/ne_misc_dev.c
+++ b/drivers/virt/nitro_enclaves/ne_misc_dev.c
@@ -1749,7 +1749,7 @@ static int ne_start_enclave_ioctl(struct ne_enclave *ne_enclave,
 
 	ne_enclave->state = NE_STATE_RUNNING;
 
-	enclave_start_info->enclave_cid = cmd_reply.enclave_cid;
+	enclave_start_info->enclave_cid = cmd_reply.start.enclave_cid;
 
 	return 0;
 }
@@ -2621,11 +2621,11 @@ static int ne_create_vm_ioctl(struct ne_pci_dev *ne_pci_dev, u64 flags,
 	init_waitqueue_head(&ne_enclave->eventq);
 	ne_enclave->has_event = false;
 	mutex_init(&ne_enclave->enclave_info_mutex);
-	ne_enclave->max_mem_regions = cmd_reply.mem_regions;
+	ne_enclave->max_mem_regions = cmd_reply.alloc.mem_regions;
 	INIT_LIST_HEAD(&ne_enclave->contig_regions_list);
 	INIT_LIST_HEAD(&ne_enclave->mem_regions_list);
 	ne_enclave->mm = current->mm;
-	ne_enclave->slot_uid = cmd_reply.slot_uid;
+	ne_enclave->slot_uid = cmd_reply.alloc.slot_uid;
 	ne_enclave->state = NE_STATE_INIT;
 
 	list_add(&ne_enclave->enclave_list_entry, &ne_pci_dev->enclaves_list);
diff --git a/drivers/virt/nitro_enclaves/ne_pci_dev.c b/drivers/virt/nitro_enclaves/ne_pci_dev.c
index a07dbc6d8a9b7..8ed2349dea878 100644
--- a/drivers/virt/nitro_enclaves/ne_pci_dev.c
+++ b/drivers/virt/nitro_enclaves/ne_pci_dev.c
@@ -404,8 +404,8 @@ static void ne_event_work_handler(struct work_struct *work)
 		}
 
 		/* Notify enclave process that the enclave state changed. */
-		if (ne_enclave->state != cmd_reply.state) {
-			ne_enclave->state = cmd_reply.state;
+		if (ne_enclave->state != cmd_reply.info.state) {
+			ne_enclave->state = cmd_reply.info.state;
 
 			ne_enclave->has_event = true;
 
diff --git a/drivers/virt/nitro_enclaves/ne_pci_dev.h b/drivers/virt/nitro_enclaves/ne_pci_dev.h
index 4f670c53dd3c0..835413fca8040 100644
--- a/drivers/virt/nitro_enclaves/ne_pci_dev.h
+++ b/drivers/virt/nitro_enclaves/ne_pci_dev.h
@@ -286,32 +286,78 @@ struct slot_add_info_page_req {
 
 /**
  * struct ne_pci_dev_cmd_reply - NE PCI device command reply.
- * @rc :		Return code of the logic that processed the request.
- * @padding0:		Padding for the overall data structure.
- * @slot_uid:		Valid for all commands except SLOT_COUNT.
- * @enclave_cid:	Valid for ENCLAVE_START command.
- * @slot_count :	Valid for SLOT_COUNT command.
- * @mem_regions:	Valid for SLOT_ALLOC and SLOT_INFO commands.
- * @mem_size:		Valid for SLOT_INFO command.
- * @nr_vcpus:		Valid for SLOT_INFO command.
- * @flags:		Valid for SLOT_INFO command.
- * @state:		Valid for SLOT_INFO command.
- * @padding1:		Padding for the overall data structure.
+ *
+ * The device writes the reply of every command into the same 72-byte
+ * window. A command whose reply carries fields of its own has an arm of
+ * the union below holding them, at the offsets the device writes them
+ * to, and a caller reads through the arm for the command it issued.
+ *
+ * @rc :			Return code of the logic that processed the
+ *				request.
+ * @padding0:			Padding for the overall data structure.
+ * @start:			Fields ENCLAVE_START fills.
+ * @start.slot_uid:		ENCLAVE_START. Slot of the started enclave.
+ * @start.enclave_cid:		ENCLAVE_START. Context ID of the enclave
+ *				vsock device.
+ * @alloc:			Fields SLOT_ALLOC fills.
+ * @alloc.slot_uid:		SLOT_ALLOC. Slot unique id of the new slot.
+ * @alloc.reserved8:		SLOT_ALLOC. Bytes owned by other commands.
+ * @alloc.mem_regions:		SLOT_ALLOC. Maximum number of memory
+ *				regions the slot accepts.
+ * @info:			Fields SLOT_INFO fills.
+ * @info.slot_uid:		SLOT_INFO. Slot the reply describes.
+ * @info.enclave_cid:		SLOT_INFO. Context ID of the enclave vsock
+ *				device.
+ * @info.reserved24:		SLOT_INFO. Bytes owned by other commands.
+ * @info.mem_regions:		SLOT_INFO. Maximum number of memory regions
+ *				the slot accepts.
+ * @info.mem_size:		SLOT_INFO. Memory size added to the slot.
+ * @info.nr_vcpus:		SLOT_INFO. Number of vCPUs added to the slot.
+ * @info.flags:			SLOT_INFO. Slot flags.
+ * @info.state:			SLOT_INFO. Slot state.
+ * @info.padding1:		SLOT_INFO. Padding for the overall data
+ *				structure.
  */
 struct ne_pci_dev_cmd_reply {
 	s32	rc;
 	u8	padding0[4];
-	u64	slot_uid;
-	u64	enclave_cid;
-	u64	slot_count;
-	u64	mem_regions;
-	u64	mem_size;
-	u64	nr_vcpus;
-	u64	flags;
-	u16	state;
-	u8	padding1[6];
+	union {
+		struct {
+			u64	slot_uid;
+			u64	enclave_cid;
+		} start;
+		struct {
+			u64	slot_uid;
+			u64	reserved8[2];
+			u64	mem_regions;
+		} alloc;
+		struct {
+			u64	slot_uid;
+			u64	enclave_cid;
+			u64	reserved24;
+			u64	mem_regions;
+			u64	mem_size;
+			u64	nr_vcpus;
+			u64	flags;
+			u16	state;
+			u8	padding1[6];
+		} info;
+	};
 };
 
+static_assert(sizeof(struct ne_pci_dev_cmd_reply) == 72);
+static_assert(offsetof(struct ne_pci_dev_cmd_reply, start.slot_uid) == 8);
+static_assert(offsetof(struct ne_pci_dev_cmd_reply, start.enclave_cid) == 16);
+static_assert(offsetof(struct ne_pci_dev_cmd_reply, alloc.slot_uid) == 8);
+static_assert(offsetof(struct ne_pci_dev_cmd_reply, alloc.mem_regions) == 32);
+static_assert(offsetof(struct ne_pci_dev_cmd_reply, info.slot_uid) == 8);
+static_assert(offsetof(struct ne_pci_dev_cmd_reply, info.enclave_cid) == 16);
+static_assert(offsetof(struct ne_pci_dev_cmd_reply, info.mem_regions) == 32);
+static_assert(offsetof(struct ne_pci_dev_cmd_reply, info.mem_size) == 40);
+static_assert(offsetof(struct ne_pci_dev_cmd_reply, info.nr_vcpus) == 48);
+static_assert(offsetof(struct ne_pci_dev_cmd_reply, info.flags) == 56);
+static_assert(offsetof(struct ne_pci_dev_cmd_reply, info.state) == 64);
+
 /**
  * struct ne_pci_dev - Nitro Enclaves (NE) PCI device.
  * @cmd_reply_avail:		Variable set if a reply has been sent by the
-- 
2.47.1

[PATCH 17/40] nitro_enclaves: Add the slot pool donation commands

[Alexander Graf]

Nitro Sandboxes introduce the concept of memory donations. In order to
launch more sandboxes, the parent instance needs to donate some of its
memory. A device that supports Nitro Sandboxes keeps its slot pool in
that memory: the descriptors of its slots and the memory the slots run
on. In preparation to donate that memory, add the commands that donate
memory for the slot descriptors and the registers that say how much of
it the device wants.

Signed-off-by: Alexander Graf <graf@amazon.com>
---
 drivers/virt/nitro_enclaves/ne_pci_dev.h | 60 ++++++++++++++++++++++++
 1 file changed, 60 insertions(+)

diff --git a/drivers/virt/nitro_enclaves/ne_pci_dev.h b/drivers/virt/nitro_enclaves/ne_pci_dev.h
index 835413fca8040..e651e74ab656e 100644
--- a/drivers/virt/nitro_enclaves/ne_pci_dev.h
+++ b/drivers/virt/nitro_enclaves/ne_pci_dev.h
@@ -95,6 +95,42 @@
  */
 #define NE_RECV_DATA_SIZE	(240)
 
+/**
+ * DOC: Slot pool registers in the NE PCI device MMIO BAR
+ */
+
+/**
+ * NE_SLOT_POOL_MAX - (2 bytes) Register reporting the highest number of
+ *		      slots the device lets the pool grow to, or zero on a
+ *		      device that needs no donation (Read-Only).
+ */
+#define NE_SLOT_POOL_MAX	(0x01f2)
+
+/**
+ * NE_SLOT_POOL_LIMIT - (2 bytes) Register reporting how many slots the
+ *			pool has room for with the pages donated so far
+ *			(Read-Only).
+ */
+#define NE_SLOT_POOL_LIMIT	(0x01f6)
+
+/**
+ * NE_SLOT_POOL_GRAN_SIZE - (4 bytes) Register reporting the size, in bytes,
+ *			    of one donation to the pool (Read-Only).
+ */
+#define NE_SLOT_POOL_GRAN_SIZE	(0x0200)
+
+/**
+ * NE_SLOT_POOL_GRAN_ALIGN - (4 bytes) Register reporting the alignment a
+ *			     donation to the pool has to meet (Read-Only).
+ */
+#define NE_SLOT_POOL_GRAN_ALIGN	(0x0204)
+
+/**
+ * NE_SLOT_POOL_BASE_SIZE - (4 bytes) Register reporting the size, in bytes,
+ *			    of the base donation (Read-Only).
+ */
+#define NE_SLOT_POOL_BASE_SIZE	(0x0208)
+
 /**
  * DOC: MSI-X interrupt vectors
  */
@@ -125,6 +161,10 @@
  * @SLOT_ADD_BULK_VCPUS:	Add a number of vCPUs, not providing CPU ids.
  * @SLOT_ADD_INFO_PAGE:		Give a slot the info page, a memory region the
  *				device writes.
+ * @SLOT_POOL_DONATE:		Donate one page of parent memory to the slot
+ *				pool.
+ * @SLOT_POOL_SHUTDOWN:		Give the whole slot pool back.
+ * @SLOT_POOL_BASE_DONATE:	Donate the base run of the slot pool.
  * @MAX_CMD:			A gatekeeper for max possible command type.
  */
 enum ne_pci_dev_cmd_type {
@@ -142,6 +182,9 @@ enum ne_pci_dev_cmd_type {
 	SLOT_ADD_BULK_VCPUS	= 11,
 	/* 12 is a command this driver does not implement. */
 	SLOT_ADD_INFO_PAGE	= 13,
+	SLOT_POOL_DONATE	= 16,
+	SLOT_POOL_SHUTDOWN	= 17,
+	SLOT_POOL_BASE_DONATE	= 25,
 	MAX_CMD,
 };
 
@@ -284,6 +327,23 @@ struct slot_add_info_page_req {
 	u64	info_size;
 };
 
+/**
+ * struct slot_pool_donate_req - SLOT_POOL_DONATE and SLOT_POOL_BASE_DONATE
+ *				 request.
+ * @paddr:	Physical address of the donated run of pages.
+ */
+struct slot_pool_donate_req {
+	u64	paddr;
+};
+
+/**
+ * struct slot_pool_shutdown_req - SLOT_POOL_SHUTDOWN request.
+ * @unused:	In order to avoid weird sizeof edge cases.
+ */
+struct slot_pool_shutdown_req {
+	u8	unused;
+};
+
 /**
  * struct ne_pci_dev_cmd_reply - NE PCI device command reply.
  *
-- 
2.47.1

[PATCH 18/40] nitro_enclaves: Donate memory for the slot pool

[Alexander Graf]

To launch the first Nitro Sandbox, the parent must donate memory to the
device. So at probe, on a device whose vendor-specific PCI capability
reports a slot pool, donate the base run and one donation of
NE_SLOT_POOL_GRAN_SIZE bytes from ne_alloc_contig().

The device holds a donated page until SLOT_POOL_SHUTDOWN, which it
refuses while a slot is in use. So remove, shutdown and a probe that
fails after donating first run ne_enclave_free_slot() on every enclave
and then take the pool back, before any page goes to the allocator. That
way the device stays usable after an unbind or a kexec with enclaves
running. Like the two free commands, SLOT_POOL_SHUTDOWN waits for
pci_dev_mutex even for a process that is being killed. This teardown
runs on every device, and a slot the device confirms as free lets the
driver release the memory and CPUs of a classic enclave too, once its
file is closed. ENCLAVE_STOP now goes out in any state, because an
enclave that exited may still be scrubbed, and the device answers a stop
for a slot that never started as well.

An enclave whose slot was freed that way gives its CPUs back on close,
so count the enclaves created from the CPU pool and refuse to replace
the pool while the count is not zero. A donation that got no answer
stays recorded as donated, because the device may take it.

Signed-off-by: Alexander Graf <graf@amazon.com>
---
 drivers/virt/nitro_enclaves/ne_misc_dev.c |  53 +++-
 drivers/virt/nitro_enclaves/ne_misc_dev.h |   7 +
 drivers/virt/nitro_enclaves/ne_pci_dev.c  | 334 +++++++++++++++++++++-
 drivers/virt/nitro_enclaves/ne_pci_dev.h  |  25 ++
 4 files changed, 401 insertions(+), 18 deletions(-)

diff --git a/drivers/virt/nitro_enclaves/ne_misc_dev.c b/drivers/virt/nitro_enclaves/ne_misc_dev.c
index 77b15eda8d120..2fae731acb977 100644
--- a/drivers/virt/nitro_enclaves/ne_misc_dev.c
+++ b/drivers/virt/nitro_enclaves/ne_misc_dev.c
@@ -201,6 +201,11 @@ static const struct kernel_param *ne_cpus_kp;
  *				and leaves when the enclave is freed, in
  *				the same two places that move the core out of
  *				and back into @avail_threads_per_core.
+ * @nr_enclaves:			How many enclaves were created from the pool and
+ *				have not been freed yet, whether or not they are
+ *				on the list of a device. The pool is not replaced
+ *				while any is, so the cores an enclave gives back
+ *				go back to the pool they came from.
  */
 struct ne_cpu_pool {
 	cpumask_var_t	*avail_threads_per_core;
@@ -211,6 +216,7 @@ struct ne_cpu_pool {
 	cpumask_var_t	*node_cpus;
 	struct cpumask	total_cpus;
 	struct cpumask	used_cpus;
+	unsigned int	nr_enclaves;
 };
 
 static struct ne_cpu_pool ne_cpu_pool = {
@@ -931,6 +937,7 @@ static int ne_replace_cpu_pool(const char *val, const struct kernel_param *kp)
 	cpumask_var_t cpu_pool;
 	cpumask_var_t old_pool;
 	struct ne_pci_dev *ne_pci_dev = NULL;
+	unsigned int nr_enclaves = 0;
 	int rc = -EINVAL;
 
 	if (!zalloc_cpumask_var(&cpu_pool, GFP_KERNEL))
@@ -961,7 +968,11 @@ static int ne_replace_cpu_pool(const char *val, const struct kernel_param *kp)
 			goto unlock_devs;
 	}
 
-	if (ne_pci_dev && !list_empty(&ne_pci_dev->enclaves_list)) {
+	mutex_lock(&ne_cpu_pool.mutex);
+	nr_enclaves = ne_cpu_pool.nr_enclaves;
+	mutex_unlock(&ne_cpu_pool.mutex);
+
+	if (nr_enclaves) {
 		pr_err("%s: The CPU pool is used by enclave(s)\n", ne_misc_dev.name);
 
 		rc = -EPERM;
@@ -2293,6 +2304,8 @@ static void ne_enclave_remove_all_vcpu_id_entries(struct ne_enclave *ne_enclave)
 	for (i = 0; i < ne_enclave->nr_parent_vm_cores; i++)
 		free_cpumask_var(ne_enclave->threads_per_core[i]);
 
+	ne_cpu_pool.nr_enclaves--;
+
 	mutex_unlock(&ne_cpu_pool.mutex);
 
 	kfree(ne_enclave->threads_per_core);
@@ -2337,8 +2350,8 @@ static void ne_pci_dev_remove_enclave_entry(struct ne_enclave *ne_enclave,
  * * 0 once the device has freed the slot.
  * * Negative return value on failure.
  */
-static int ne_enclave_free_slot(struct ne_pci_dev *ne_pci_dev,
-				struct ne_enclave *ne_enclave)
+int ne_enclave_free_slot(struct ne_pci_dev *ne_pci_dev,
+			 struct ne_enclave *ne_enclave)
 {
 	struct ne_pci_dev_cmd_reply cmd_reply = {};
 	struct enclave_stop_req enclave_stop_request = {};
@@ -2346,17 +2359,20 @@ static int ne_enclave_free_slot(struct ne_pci_dev *ne_pci_dev,
 	struct pci_dev *pdev = ne_pci_dev->pdev;
 	int rc;
 
-	if (ne_enclave->state != NE_STATE_INIT && ne_enclave->state != NE_STATE_STOPPED) {
-		enclave_stop_request.slot_uid = ne_enclave->slot_uid;
+	/*
+	 * An enclave that exited may still be scrubbed, so the stop goes out
+	 * whatever the state, and the device answers it once the slot is
+	 * neither running nor being scrubbed.
+	 */
+	enclave_stop_request.slot_uid = ne_enclave->slot_uid;
 
-		rc = ne_do_request(pdev, ENCLAVE_STOP,
-				   &enclave_stop_request, sizeof(enclave_stop_request),
-				   &cmd_reply, sizeof(cmd_reply));
-		if (rc < 0)
-			return rc;
+	rc = ne_do_request(pdev, ENCLAVE_STOP,
+			   &enclave_stop_request, sizeof(enclave_stop_request),
+			   &cmd_reply, sizeof(cmd_reply));
+	if (rc < 0)
+		return rc;
 
-		memset(&cmd_reply, 0, sizeof(cmd_reply));
-	}
+	memset(&cmd_reply, 0, sizeof(cmd_reply));
 
 	slot_free_req.slot_uid = ne_enclave->slot_uid;
 
@@ -2416,10 +2432,13 @@ static int ne_enclave_release(struct inode *inode, struct file *file)
 	/*
 	 * The device the enclave was created on is gone, and with it the only
 	 * party that can confirm that the slot no longer uses its memory and
-	 * its CPUs, so the enclave keeps both.
+	 * its CPUs, so the enclave keeps both, unless the device freed the
+	 * slot before it went away.
 	 */
 	if (ne_enclave->detached) {
-		rc = -ENODEV;
+		rc = ne_enclave->slot_freed ? 0 : -ENODEV;
+		if (!rc)
+			ne_enclave_free(ne_enclave);
 
 		goto unlock_devs;
 	}
@@ -2525,7 +2544,7 @@ static int ne_create_vm_ioctl(struct ne_pci_dev *ne_pci_dev, u64 flags,
 	 * capability. One without it takes no launch flags and would allocate a
 	 * slot without the properties they ask for.
 	 */
-	if (flags && !pci_find_capability(ne_pci_dev->pdev, PCI_CAP_ID_VNDR)) {
+	if (flags && !ne_pci_dev->has_vendor_cap) {
 		dev_err_ratelimited(ne_misc_dev.this_device,
 				    "Device takes no launch flags\n");
 
@@ -2630,6 +2649,10 @@ static int ne_create_vm_ioctl(struct ne_pci_dev *ne_pci_dev, u64 flags,
 
 	list_add(&ne_enclave->enclave_list_entry, &ne_pci_dev->enclaves_list);
 
+	mutex_lock(&ne_cpu_pool.mutex);
+	ne_cpu_pool.nr_enclaves++;
+	mutex_unlock(&ne_cpu_pool.mutex);
+
 	if (copy_to_user(slot_uid, &ne_enclave->slot_uid, sizeof(ne_enclave->slot_uid))) {
 		/*
 		 * As we're holding the only reference to 'enclave_file', fput()
diff --git a/drivers/virt/nitro_enclaves/ne_misc_dev.h b/drivers/virt/nitro_enclaves/ne_misc_dev.h
index d0d1973bfbe9d..59374a35c3c7d 100644
--- a/drivers/virt/nitro_enclaves/ne_misc_dev.h
+++ b/drivers/virt/nitro_enclaves/ne_misc_dev.h
@@ -116,6 +116,9 @@ struct ne_mem_region {
  * @enclave_info_mutex :	Mutex for accessing this internal state.
  * @detached:			The NE PCI device the enclave was created on is
  *				gone. Written with ne_devs.lock held for writing.
+ * @slot_freed:			The device freed the slot of a detached enclave
+ *				before it went away. Written with ne_devs.lock
+ *				held for writing.
  * @contig_regions_list:	List of %NE_ALLOC_MEMORY allocations given to the
  *				enclave with %NE_ADD_MEMORY_REGION.
  * @enclave_list_entry :	Entry in the list of created enclaves.
@@ -157,6 +160,7 @@ struct ne_mem_region {
 struct ne_enclave {
 	struct mutex		enclave_info_mutex;
 	bool			detached;
+	bool			slot_freed;
 	struct list_head	contig_regions_list;
 	struct list_head	enclave_list_entry;
 	wait_queue_head_t	eventq;
@@ -221,6 +225,9 @@ void ne_contig_free_run(struct cma *cma, struct page *page,
 /* Nitro Enclaves (NE) data structure for keeping refs to the NE misc and PCI devices. */
 extern struct ne_devs ne_devs;
 
+int ne_enclave_free_slot(struct ne_pci_dev *ne_pci_dev,
+			 struct ne_enclave *ne_enclave);
+
 int ne_cpu_pool_sysfs_init(void);
 void ne_cpu_pool_sysfs_exit(void);
 
diff --git a/drivers/virt/nitro_enclaves/ne_pci_dev.c b/drivers/virt/nitro_enclaves/ne_pci_dev.c
index 8ed2349dea878..e42be1742a39c 100644
--- a/drivers/virt/nitro_enclaves/ne_pci_dev.c
+++ b/drivers/virt/nitro_enclaves/ne_pci_dev.c
@@ -7,14 +7,20 @@
  * DOC: Nitro Enclaves (NE) PCI device driver.
  */
 
+#include <linux/cma.h>
 #include <linux/delay.h>
 #include <linux/device.h>
+#include <linux/err.h>
 #include <linux/iopoll.h>
+#include <linux/jiffies.h>
 #include <linux/list.h>
+#include <linux/minmax.h>
 #include <linux/module.h>
 #include <linux/mutex.h>
 #include <linux/nitro_enclaves.h>
 #include <linux/pci.h>
+#include <linux/sizes.h>
+#include <linux/sysfs.h>
 #include <linux/types.h>
 #include <linux/wait.h>
 
@@ -211,10 +217,11 @@ int ne_do_request(struct pci_dev *pdev, enum ne_pci_dev_cmd_type cmd_type,
 	/*
 	 * Use this mutex so that the PCI device handles one command request at
 	 * a time. A slot being freed can hold it for minutes, so let a task
-	 * that waits for it be killed, unless the task frees a slot itself:
-	 * release has to do that for a process that is being killed too.
+	 * that waits for it be killed, unless the task frees a slot or takes
+	 * the slot pool back: release and unbind have to finish those for a
+	 * process that is being killed too.
 	 */
-	if (ne_cmd_frees_slot(cmd_type)) {
+	if (ne_cmd_frees_slot(cmd_type) || cmd_type == SLOT_POOL_SHUTDOWN) {
 		mutex_lock(&ne_pci_dev->pci_dev_mutex);
 	} else {
 		rc = mutex_lock_killable(&ne_pci_dev->pci_dev_mutex);
@@ -609,6 +616,14 @@ static void ne_pci_dev_disable(struct pci_dev *pdev)
  *				  goes away.
  * @ne_pci_dev:	Private data associated with the PCI device.
  *
+ * A device with a slot pool refuses SLOT_POOL_SHUTDOWN while one of its slots
+ * is in use, and the pages it was donated would stay with it. Each enclave is
+ * therefore stopped and its slot freed here, before ne_slot_pool_release()
+ * runs, the same way as on release. An enclave gives its memory and CPUs back
+ * when its file is released, if the device freed its slot here. An enclave
+ * whose file is already gone is one the device did not free on release, and
+ * it keeps both.
+ *
  * Context: Process context. This function is called with ne_devs.lock held
  *	    for writing.
  */
@@ -625,6 +640,7 @@ static void ne_pci_dev_detach_enclaves(struct ne_pci_dev *ne_pci_dev)
 
 		mutex_lock(&ne_enclave->enclave_info_mutex);
 
+		ne_enclave->slot_freed = !ne_enclave_free_slot(ne_pci_dev, ne_enclave);
 		ne_enclave->detached = true;
 		ne_enclave->has_event = true;
 
@@ -636,6 +652,281 @@ static void ne_pci_dev_detach_enclaves(struct ne_pci_dev *ne_pci_dev)
 	mutex_unlock(&ne_pci_dev->enclaves_list_mutex);
 }
 
+/**
+ * ne_slot_pool_read_geometry() - Read how much memory the slot pool wants.
+ * @pdev:	PCI device to read the slot pool registers of.
+ *
+ * The slot pool registers only exist on a device that publishes a
+ * vendor-specific capability, so a device without one keeps a zero ceiling and
+ * reads no slot pool register at all. A device that stores its slot
+ * descriptors itself reports a zero ceiling and the driver donates nothing to
+ * it. A device that asks for parent memory is only usable if it asks in units
+ * the driver can allocate, so a granularity other than 2 MiB leaves the
+ * ceiling at zero as well.
+ *
+ * Context: Process context.
+ */
+static void ne_slot_pool_read_geometry(struct pci_dev *pdev)
+{
+	struct ne_pci_dev *ne_pci_dev = pci_get_drvdata(pdev);
+	u32 size, align;
+
+	if (!ne_pci_dev->has_vendor_cap)
+		return;
+
+	ne_pci_dev->slot_pool_max = ioread16(ne_pci_dev->iomem_base + NE_SLOT_POOL_MAX);
+	if (!ne_pci_dev->slot_pool_max)
+		return;
+
+	size = ioread32(ne_pci_dev->iomem_base + NE_SLOT_POOL_GRAN_SIZE);
+	align = ioread32(ne_pci_dev->iomem_base + NE_SLOT_POOL_GRAN_ALIGN);
+	if (size != SZ_2M || align != SZ_2M) {
+		dev_dbg(&pdev->dev, "Slot pool granularity size=%u align=%u unusable\n",
+			size, align);
+
+		ne_pci_dev->slot_pool_max = 0;
+
+		return;
+	}
+
+	ne_pci_dev->slot_pool_gran = size;
+}
+
+/**
+ * ne_slot_pool_donate_base() - Donate the base run of the slot pool.
+ * @pdev:	PCI device to donate to.
+ *
+ * The device turns down a donation to the slot pool until it has the base
+ * run, so the base run is donated before any other donation. Donating a
+ * second time is a no-op.
+ *
+ * A donation the device never answered was submitted with the run's physical
+ * address in it, so the run is recorded as donated and left to
+ * ne_slot_pool_release() rather than given back.
+ *
+ * Context: Process context. This function is called with the enclaves list
+ *	    mutex held.
+ * Return:
+ * * 0 on success or if the base run has already been donated.
+ * * Negative return value on failure.
+ * * -ETIMEDOUT, or any other code ne_request_unanswered() names, if the
+ *   request went unanswered, in which case the run stays recorded as donated
+ *   because the driver cannot establish that the device refused it. A device
+ *   that answers such a code itself lands here too and costs one held run.
+ */
+static int ne_slot_pool_donate_base(struct pci_dev *pdev)
+{
+	struct ne_pci_dev *ne_pci_dev = pci_get_drvdata(pdev);
+	struct ne_pci_dev_cmd_reply cmd_reply = {};
+	struct slot_pool_donate_req donate_req = {};
+	struct cma *cma = NULL;
+	unsigned long nr_pages;
+	struct page *page;
+	u32 size;
+	int rc;
+
+	if (ne_pci_dev->slot_pool_base_page)
+		return 0;
+
+	size = ioread32(ne_pci_dev->iomem_base + NE_SLOT_POOL_BASE_SIZE);
+	if (!size || !IS_ALIGNED(size, PAGE_SIZE)) {
+		dev_warn(&pdev->dev, "Slot pool base size=%u unusable\n", size);
+
+		return -EOPNOTSUPP;
+	}
+
+	nr_pages = size >> PAGE_SHIFT;
+
+	page = ne_alloc_contig(nr_pages, numa_node_id(), &cma);
+	if (IS_ERR_OR_NULL(page)) {
+		rc = page ? PTR_ERR(page) : -ENOMEM;
+
+		dev_warn(&pdev->dev,
+			 "Slot pool base size=%u not allocated on node %d [rc=%d]\n",
+			 size, numa_node_id(), rc);
+
+		return rc;
+	}
+
+	donate_req.paddr = page_to_phys(page);
+
+	rc = ne_do_request(pdev, SLOT_POOL_BASE_DONATE,
+			   &donate_req, sizeof(donate_req),
+			   &cmd_reply, sizeof(cmd_reply));
+	if (rc < 0) {
+		dev_err_ratelimited(&pdev->dev, "Error in slot pool base donate [rc=%d]\n",
+				    rc);
+
+		/*
+		 * A request the device never answered was submitted with the
+		 * run's physical address in it, and there is nothing to cancel
+		 * it with, so the device may take the run yet.
+		 * ne_request_unanswered() picks out that case. Record the run as
+		 * donated, so that ne_slot_pool_release() owns it from here the
+		 * way it owns a donation the device accepted. The errors left
+		 * over say the device answered to refuse the donation, or never
+		 * saw it, and the run is the parent's again.
+		 */
+		if (ne_request_unanswered(rc)) {
+			ne_pci_dev->slot_pool_base_page = page;
+			ne_pci_dev->slot_pool_base_cma = cma;
+			ne_pci_dev->slot_pool_base_nr_pages = nr_pages;
+		} else {
+			ne_contig_free_run(cma, page, nr_pages);
+		}
+
+		return rc;
+	}
+
+	ne_pci_dev->slot_pool_base_page = page;
+	ne_pci_dev->slot_pool_base_cma = cma;
+	ne_pci_dev->slot_pool_base_nr_pages = nr_pages;
+
+	return 0;
+}
+
+/**
+ * ne_slot_pool_donate() - Donate one granularity of pages to the slot pool.
+ * @pdev:	PCI device to donate to.
+ *
+ * A donation raises the number of slots the pool has room for, up to the
+ * ceiling the device reports, and a pool that has reached that ceiling takes no
+ * further donation.
+ *
+ * Context: Process context. This function is called with the enclaves list
+ *	    mutex held.
+ * Return:
+ * * 0 on success.
+ * * -ENOSPC if the pool has reached the ceiling the device reports.
+ * * Negative return value on failure.
+ * * -ETIMEDOUT, or any other code ne_request_unanswered() names, if the
+ *   request went unanswered, in which case the run stays recorded as donated
+ *   because the driver cannot establish that the device refused it. A device
+ *   that answers such a code itself lands here too and costs one held run.
+ */
+static int ne_slot_pool_donate(struct pci_dev *pdev)
+{
+	struct ne_pci_dev *ne_pci_dev = pci_get_drvdata(pdev);
+	struct ne_pci_dev_cmd_reply cmd_reply = {};
+	struct slot_pool_donate_req donate_req = {};
+	struct cma *cma = NULL;
+	unsigned long nr_pages;
+	struct page *page;
+	int rc;
+
+	if (ne_pci_dev->slot_pool_page)
+		return 0;
+
+	if (!ne_pci_dev->slot_pool_gran)
+		return -EOPNOTSUPP;
+
+	if (ioread16(ne_pci_dev->iomem_base + NE_SLOT_POOL_LIMIT) >=
+	    ne_pci_dev->slot_pool_max)
+		return -ENOSPC;
+
+	nr_pages = ne_pci_dev->slot_pool_gran >> PAGE_SHIFT;
+
+	page = ne_alloc_contig(nr_pages, numa_node_id(), &cma);
+	if (IS_ERR_OR_NULL(page)) {
+		rc = page ? PTR_ERR(page) : -ENOMEM;
+
+		dev_warn(&pdev->dev,
+			 "Slot pool granularity size=%u not allocated on node %d [rc=%d]\n",
+			 ne_pci_dev->slot_pool_gran, numa_node_id(), rc);
+
+		return rc;
+	}
+
+	donate_req.paddr = page_to_phys(page);
+
+	rc = ne_do_request(pdev, SLOT_POOL_DONATE,
+			   &donate_req, sizeof(donate_req),
+			   &cmd_reply, sizeof(cmd_reply));
+	if (rc < 0) {
+		dev_err_ratelimited(&pdev->dev, "Error in slot pool donate [rc=%d]\n", rc);
+
+		/*
+		 * Record a run whose donation went unanswered as donated
+		 * instead of giving it back, as ne_slot_pool_donate_base()
+		 * does and for the same reason.
+		 */
+		if (ne_request_unanswered(rc)) {
+			ne_pci_dev->slot_pool_page = page;
+			ne_pci_dev->slot_pool_cma = cma;
+		} else {
+			ne_contig_free_run(cma, page, nr_pages);
+		}
+
+		return rc;
+	}
+
+	ne_pci_dev->slot_pool_page = page;
+	ne_pci_dev->slot_pool_cma = cma;
+
+	return 0;
+}
+
+/**
+ * ne_slot_pool_release() - Take the whole slot pool back from the device.
+ * @pdev:	PCI device to take the pool back from.
+ *
+ * The device keeps a slot's descriptor in a page the parent donated, and
+ * SLOT_POOL_SHUTDOWN names no slot, so a pool taken back while the device still
+ * has a slot is parent memory the device goes on writing after the allocator has
+ * handed it to somebody else. ne_pci_dev_detach_enclaves() frees the slot of
+ * every enclave before this runs, and the device refuses SLOT_POOL_SHUTDOWN
+ * while a slot it would not free is still in use. A SLOT_POOL_SHUTDOWN that
+ * fails leaves the pages with the device whatever the failure was: a refusal
+ * means the device still has the pool, and an answer that never came means it
+ * may still have it.
+ *
+ * A donation that was never answered is recorded as donated, and the pages stay
+ * with the device: ne_do_request() sends no command after one that went
+ * unanswered, so SLOT_POOL_SHUTDOWN fails as well.
+ *
+ * Context: Process context.
+ */
+static void ne_slot_pool_release(struct pci_dev *pdev)
+{
+	struct ne_pci_dev *ne_pci_dev = pci_get_drvdata(pdev);
+	struct ne_pci_dev_cmd_reply cmd_reply = {};
+	struct slot_pool_shutdown_req shutdown_req = {};
+	int rc;
+
+	if (!ne_pci_dev->slot_pool_base_page)
+		return;
+
+	mutex_lock(&ne_pci_dev->enclaves_list_mutex);
+
+	rc = ne_do_request(pdev, SLOT_POOL_SHUTDOWN,
+			   &shutdown_req, sizeof(shutdown_req),
+			   &cmd_reply, sizeof(cmd_reply));
+	if (rc < 0) {
+		dev_err(&pdev->dev, "Error in slot pool shutdown [rc=%d]\n", rc);
+
+		goto unlock_mutex;
+	}
+
+	if (ne_pci_dev->slot_pool_page) {
+		ne_contig_free_run(ne_pci_dev->slot_pool_cma,
+				   ne_pci_dev->slot_pool_page,
+				   ne_pci_dev->slot_pool_gran >> PAGE_SHIFT);
+
+		ne_pci_dev->slot_pool_page = NULL;
+		ne_pci_dev->slot_pool_cma = NULL;
+	}
+
+	ne_contig_free_run(ne_pci_dev->slot_pool_base_cma,
+			   ne_pci_dev->slot_pool_base_page,
+			   ne_pci_dev->slot_pool_base_nr_pages);
+
+	ne_pci_dev->slot_pool_base_page = NULL;
+	ne_pci_dev->slot_pool_base_cma = NULL;
+
+unlock_mutex:
+	mutex_unlock(&ne_pci_dev->enclaves_list_mutex);
+}
+
 /**
  * ne_pci_probe() - Probe function for the NE PCI device.
  * @pdev:	PCI device to match with the NE PCI driver.
@@ -689,6 +980,15 @@ static int ne_pci_probe(struct pci_dev *pdev, const struct pci_device_id *id)
 	mutex_init(&ne_pci_dev->pci_dev_mutex);
 	ne_pci_dev->pdev = pdev;
 
+	/*
+	 * The registers past the command interface only exist on a device that
+	 * publishes a vendor-specific capability, and a read of a register the
+	 * device does not implement has no defined value. The answer holds for
+	 * the life of the device, so look it up once here.
+	 */
+	ne_pci_dev->has_vendor_cap =
+		pci_find_capability(pdev, PCI_CAP_ID_VNDR) != 0;
+
 	rc = ne_setup_msix(pdev);
 	if (rc < 0) {
 		dev_err(&pdev->dev, "Error in pci dev msix setup [rc=%d]\n", rc);
@@ -721,6 +1021,29 @@ static int ne_pci_probe(struct pci_dev *pdev, const struct pci_device_id *id)
 
 	ne_devs.ne_pci_dev = ne_pci_dev;
 
+	ne_slot_pool_read_geometry(pdev);
+
+	if (ne_pci_dev->slot_pool_max) {
+		const char *stage;
+
+		mutex_lock(&ne_pci_dev->enclaves_list_mutex);
+
+		rc = ne_slot_pool_donate_base(pdev);
+		if (rc < 0) {
+			stage = "base";
+		} else {
+			rc = ne_slot_pool_donate(pdev);
+			stage = "granularity";
+		}
+
+		mutex_unlock(&ne_pci_dev->enclaves_list_mutex);
+
+		if (rc < 0)
+			dev_warn(&pdev->dev,
+				 "No slot donated to the slot pool [stage=%s rc=%d]\n",
+				 stage, rc);
+	}
+
 	rc = misc_register(ne_devs.ne_misc_dev);
 	if (rc < 0) {
 		dev_err(&pdev->dev, "Error in misc dev register [rc=%d]\n", rc);
@@ -745,6 +1068,7 @@ static int ne_pci_probe(struct pci_dev *pdev, const struct pci_device_id *id)
 	ne_pci_dev_detach_enclaves(ne_pci_dev);
 	misc_deregister(ne_devs.ne_misc_dev);
 disable_ne_pci_dev:
+	ne_slot_pool_release(pdev);
 	ne_devs.ne_pci_dev = NULL;
 unlock_ne_devs:
 	up_write(&ne_devs.lock);
@@ -782,6 +1106,8 @@ static void ne_pci_remove(struct pci_dev *pdev)
 
 	misc_deregister(ne_devs.ne_misc_dev);
 
+	ne_slot_pool_release(pdev);
+
 	ne_devs.ne_pci_dev = NULL;
 
 	ne_pci_dev_disable(pdev);
@@ -822,6 +1148,8 @@ static void ne_pci_shutdown(struct pci_dev *pdev)
 
 	misc_deregister(ne_devs.ne_misc_dev);
 
+	ne_slot_pool_release(pdev);
+
 	ne_devs.ne_pci_dev = NULL;
 
 	ne_pci_dev_disable(pdev);
diff --git a/drivers/virt/nitro_enclaves/ne_pci_dev.h b/drivers/virt/nitro_enclaves/ne_pci_dev.h
index e651e74ab656e..4e991c4b0e70f 100644
--- a/drivers/virt/nitro_enclaves/ne_pci_dev.h
+++ b/drivers/virt/nitro_enclaves/ne_pci_dev.h
@@ -433,10 +433,27 @@ static_assert(offsetof(struct ne_pci_dev_cmd_reply, info.state) == 64);
  *				triggered by the Nitro Hypervisor which require
  *				enclave state scanning and propagation to the
  *				enclave process.
+ * @has_vendor_cap:		Whether the PCI device publishes a vendor-specific
+ *				capability, and with it the registers past the
+ *				command interface.
  * @iomem_base :		MMIO region of the PCI device.
  * @notify_work:		Work item for every received out-of-band event.
  * @pci_dev_mutex:		Mutex for accessing the PCI device MMIO space.
  * @pdev:			PCI device data structure.
+ * @slot_pool_max:		Highest number of slots the device lets the slot
+ *				pool grow to, zero on a device that takes no
+ *				donation.
+ * @slot_pool_gran:		Size, in bytes, of one donation to the slot pool.
+ *				Written once at probe, before any donation is
+ *				made, so a later reader of it sees the size the
+ *				donation was allocated with.
+ * @slot_pool_base_page:	First page of the base donation, NULL until it
+ *				has been made.
+ * @slot_pool_base_cma:		CMA area the base donation came from.
+ * @slot_pool_base_nr_pages:	Number of pages in the base donation.
+ * @slot_pool_page:		First page of the donation to the slot pool, NULL
+ *				until it has been made.
+ * @slot_pool_cma:		CMA area the donation came from.
  */
 struct ne_pci_dev {
 	atomic_t		cmd_reply_avail;
@@ -445,10 +462,18 @@ struct ne_pci_dev {
 	struct list_head	enclaves_list;
 	struct mutex		enclaves_list_mutex;
 	struct workqueue_struct	*event_wq;
+	bool			has_vendor_cap;
 	void __iomem		*iomem_base;
 	struct work_struct	notify_work;
 	struct mutex		pci_dev_mutex;
 	struct pci_dev		*pdev;
+	u16			slot_pool_max;
+	u32			slot_pool_gran;
+	struct page		*slot_pool_base_page;
+	struct cma		*slot_pool_base_cma;
+	unsigned long		slot_pool_base_nr_pages;
+	struct page		*slot_pool_page;
+	struct cma		*slot_pool_cma;
 };
 
 /* Whether a failure left a submitted request in flight. */
-- 
2.47.1

[PATCH 19/40] nitro_enclaves: Add the slot pool growth commands

[Alexander Graf]

In preparation to grow the slot pool at runtime, add
SLOT_POOL_MEM_DONATE and SLOT_POOL_GET_LIMITS.

The slot pool has two halves: the slot descriptors, which
SLOT_POOL_DONATE grows, and the memory the device runs slots out of,
which SLOT_POOL_MEM_DONATE grows. Its reply names the part of the
offered run that the device took. A slot needs room in both halves.
SLOT_POOL_GET_LIMITS reports both counts, along with how many slots one
memory donation adds, so that the parent knows which half to grow next.
Both replies get an arm of the reply union.

Signed-off-by: Alexander Graf <graf@amazon.com>
---
 drivers/virt/nitro_enclaves/ne_pci_dev.h | 50 ++++++++++++++++++++++++
 1 file changed, 50 insertions(+)

diff --git a/drivers/virt/nitro_enclaves/ne_pci_dev.h b/drivers/virt/nitro_enclaves/ne_pci_dev.h
index 4e991c4b0e70f..3718f893e4f65 100644
--- a/drivers/virt/nitro_enclaves/ne_pci_dev.h
+++ b/drivers/virt/nitro_enclaves/ne_pci_dev.h
@@ -164,6 +164,10 @@
  * @SLOT_POOL_DONATE:		Donate one page of parent memory to the slot
  *				pool.
  * @SLOT_POOL_SHUTDOWN:		Give the whole slot pool back.
+ * @SLOT_POOL_MEM_DONATE:	Donate a run of parent memory to the memory the
+ *				device runs a slot out of.
+ * @SLOT_POOL_GET_LIMITS:	Report how many slots each half of the pool has
+ *				room for.
  * @SLOT_POOL_BASE_DONATE:	Donate the base run of the slot pool.
  * @MAX_CMD:			A gatekeeper for max possible command type.
  */
@@ -185,6 +189,8 @@ enum ne_pci_dev_cmd_type {
 	SLOT_POOL_DONATE	= 16,
 	SLOT_POOL_SHUTDOWN	= 17,
 	SLOT_POOL_BASE_DONATE	= 25,
+	SLOT_POOL_MEM_DONATE	= 26,
+	SLOT_POOL_GET_LIMITS	= 27,
 	MAX_CMD,
 };
 
@@ -344,6 +350,16 @@ struct slot_pool_shutdown_req {
 	u8	unused;
 };
 
+/**
+ * struct slot_pool_mem_donate_req - SLOT_POOL_MEM_DONATE request.
+ * @paddr:	Physical address of the donated run of memory.
+ * @size:	Size, in bytes, of the donated run.
+ */
+struct slot_pool_mem_donate_req {
+	u64	paddr;
+	u64	size;
+};
+
 /**
  * struct ne_pci_dev_cmd_reply - NE PCI device command reply.
  *
@@ -377,6 +393,21 @@ struct slot_pool_shutdown_req {
  * @info.state:			SLOT_INFO. Slot state.
  * @info.padding1:		SLOT_INFO. Padding for the overall data
  *				structure.
+ * @mem_donate:			Fields SLOT_POOL_MEM_DONATE fills.
+ * @mem_donate.consumed_paddr:	SLOT_POOL_MEM_DONATE. Physical address of the
+ *				run the device took out of the donation.
+ * @mem_donate.consumed_size:	SLOT_POOL_MEM_DONATE. Size, in bytes, of the
+ *				run the device took, zero where it took none.
+ * @limits:			Fields SLOT_POOL_GET_LIMITS fills.
+ * @limits.meta_limit:		SLOT_POOL_GET_LIMITS. Slots the descriptors
+ *				donated so far have room for.
+ * @limits.mem_limit:		SLOT_POOL_GET_LIMITS. Slots the memory donated
+ *				so far has room to run.
+ * @limits.slots_per_donation:	SLOT_POOL_GET_LIMITS. Slots one donation of
+ *				memory adds to the memory the device runs a
+ *				slot out of.
+ * @limits.reserved14:		SLOT_POOL_GET_LIMITS. Bytes the device writes as
+ *				zero.
  */
 struct ne_pci_dev_cmd_reply {
 	s32	rc;
@@ -402,6 +433,16 @@ struct ne_pci_dev_cmd_reply {
 			u16	state;
 			u8	padding1[6];
 		} info;
+		struct {
+			u64	consumed_paddr;
+			u64	consumed_size;
+		} mem_donate;
+		struct {
+			u16	meta_limit;
+			u16	mem_limit;
+			u16	slots_per_donation;
+			u16	reserved14;
+		} limits;
 	};
 };
 
@@ -417,6 +458,15 @@ static_assert(offsetof(struct ne_pci_dev_cmd_reply, info.mem_size) == 40);
 static_assert(offsetof(struct ne_pci_dev_cmd_reply, info.nr_vcpus) == 48);
 static_assert(offsetof(struct ne_pci_dev_cmd_reply, info.flags) == 56);
 static_assert(offsetof(struct ne_pci_dev_cmd_reply, info.state) == 64);
+static_assert(offsetof(struct ne_pci_dev_cmd_reply,
+		       mem_donate.consumed_paddr) == 8);
+static_assert(offsetof(struct ne_pci_dev_cmd_reply,
+		       mem_donate.consumed_size) == 16);
+static_assert(offsetof(struct ne_pci_dev_cmd_reply, limits.meta_limit) == 8);
+static_assert(offsetof(struct ne_pci_dev_cmd_reply, limits.mem_limit) == 10);
+static_assert(offsetof(struct ne_pci_dev_cmd_reply,
+		       limits.slots_per_donation) == 12);
+static_assert(offsetof(struct ne_pci_dev_cmd_reply, limits.reserved14) == 14);
 
 /**
  * struct ne_pci_dev - Nitro Enclaves (NE) PCI device.
-- 
2.47.1

[PATCH 20/40] nitro_enclaves: Add a helper to grow the slot pool

[Alexander Graf]

In addition to the early boot pool allocations, users want to be able to
grow their pool at runtime. To the user, we only present a single maximum
slot count. Under the hood, there are 2 slot counters and the MIN() of
defines how many slots there really are. When growing, grow both of the
limits to satisfy the desired customer set maximum.

Signed-off-by: Alexander Graf <graf@amazon.com>
---
 drivers/virt/nitro_enclaves/ne_pci_dev.c | 466 +++++++++++++++++++++--
 drivers/virt/nitro_enclaves/ne_pci_dev.h |  35 +-
 2 files changed, 474 insertions(+), 27 deletions(-)

diff --git a/drivers/virt/nitro_enclaves/ne_pci_dev.c b/drivers/virt/nitro_enclaves/ne_pci_dev.c
index e42be1742a39c..9e304ffc2bd13 100644
--- a/drivers/virt/nitro_enclaves/ne_pci_dev.c
+++ b/drivers/virt/nitro_enclaves/ne_pci_dev.c
@@ -18,8 +18,10 @@
 #include <linux/module.h>
 #include <linux/mutex.h>
 #include <linux/nitro_enclaves.h>
+#include <linux/nodemask.h>
 #include <linux/pci.h>
 #include <linux/sizes.h>
+#include <linux/slab.h>
 #include <linux/sysfs.h>
 #include <linux/types.h>
 #include <linux/wait.h>
@@ -692,6 +694,61 @@ static void ne_slot_pool_read_geometry(struct pci_dev *pdev)
 	ne_pci_dev->slot_pool_gran = size;
 }
 
+/**
+ * ne_slot_pool_alloc_run() - Allocate one run for the slot pool.
+ * @nr_pages:	Number of pages in the run.
+ * @pref_nid:	NUMA node to try first.
+ * @out_cma:	Set to the CMA area the run came from.
+ *
+ * ne_alloc_contig() confines a run to the node it is given, which is what an
+ * enclave needs. A run in the slot pool is read and written by the device
+ * alone, so it can sit on any node that has memory, and a pool that stops
+ * growing because one node ran out is worth less than a pool spread over
+ * several. So walk the nodes that have memory, starting at @pref_nid, and take
+ * the first run any of them serves.
+ *
+ * A caller runs on a node that need not have memory of its own, and there is no
+ * run to be had from such a node, so a @pref_nid outside the set of nodes that
+ * have memory starts the walk at the first node that does. The walk then covers
+ * every node that can serve it.
+ *
+ * Context: Process context.
+ * Return: the first page of the run, NULL when no node served one, or the
+ *	   ERR_PTR() ne_alloc_contig() returned.
+ */
+static struct page *ne_slot_pool_alloc_run(unsigned long nr_pages,
+					   int pref_nid, struct cma **out_cma)
+{
+	int nid = pref_nid;
+	struct page *page;
+	int nr_nodes;
+	int i;
+
+	nr_nodes = num_node_state(N_MEMORY);
+	if (nr_nodes < 1)
+		return NULL;
+
+	if (pref_nid < 0 || pref_nid >= MAX_NUMNODES ||
+	    !node_state(pref_nid, N_MEMORY))
+		nid = first_node(node_states[N_MEMORY]);
+
+	for (i = 0; i < nr_nodes; i++) {
+		page = ne_alloc_contig(nr_pages, nid, out_cma);
+
+		/*
+		 * An error ends the walk along with a run: only a node that had
+		 * no run to give leaves NULL behind, and only then does the next
+		 * node get asked.
+		 */
+		if (page)
+			return page;
+
+		nid = next_node_in(nid, node_states[N_MEMORY]);
+	}
+
+	return NULL;
+}
+
 /**
  * ne_slot_pool_donate_base() - Donate the base run of the slot pool.
  * @pdev:	PCI device to donate to.
@@ -708,6 +765,7 @@ static void ne_slot_pool_read_geometry(struct pci_dev *pdev)
  *	    mutex held.
  * Return:
  * * 0 on success or if the base run has already been donated.
+ * * -EINTR if a fatal signal arrived while a run was being zeroed.
  * * Negative return value on failure.
  * * -ETIMEDOUT, or any other code ne_request_unanswered() names, if the
  *   request went unanswered, in which case the run stays recorded as donated
@@ -737,13 +795,13 @@ static int ne_slot_pool_donate_base(struct pci_dev *pdev)
 
 	nr_pages = size >> PAGE_SHIFT;
 
-	page = ne_alloc_contig(nr_pages, numa_node_id(), &cma);
+	page = ne_slot_pool_alloc_run(nr_pages, numa_node_id(), &cma);
 	if (IS_ERR_OR_NULL(page)) {
 		rc = page ? PTR_ERR(page) : -ENOMEM;
 
 		dev_warn(&pdev->dev,
-			 "Slot pool base size=%u not allocated on node %d [rc=%d]\n",
-			 size, numa_node_id(), rc);
+			 "Slot pool base size=%u not allocated [rc=%d]\n",
+			 size, rc);
 
 		return rc;
 	}
@@ -798,6 +856,7 @@ static int ne_slot_pool_donate_base(struct pci_dev *pdev)
  * Return:
  * * 0 on success.
  * * -ENOSPC if the pool has reached the ceiling the device reports.
+ * * -EINTR if a fatal signal arrived while a run was being zeroed.
  * * Negative return value on failure.
  * * -ETIMEDOUT, or any other code ne_request_unanswered() names, if the
  *   request went unanswered, in which case the run stays recorded as donated
@@ -809,14 +868,12 @@ static int ne_slot_pool_donate(struct pci_dev *pdev)
 	struct ne_pci_dev *ne_pci_dev = pci_get_drvdata(pdev);
 	struct ne_pci_dev_cmd_reply cmd_reply = {};
 	struct slot_pool_donate_req donate_req = {};
+	struct ne_slot_pool_run *run;
 	struct cma *cma = NULL;
 	unsigned long nr_pages;
 	struct page *page;
 	int rc;
 
-	if (ne_pci_dev->slot_pool_page)
-		return 0;
-
 	if (!ne_pci_dev->slot_pool_gran)
 		return -EOPNOTSUPP;
 
@@ -826,17 +883,27 @@ static int ne_slot_pool_donate(struct pci_dev *pdev)
 
 	nr_pages = ne_pci_dev->slot_pool_gran >> PAGE_SHIFT;
 
-	page = ne_alloc_contig(nr_pages, numa_node_id(), &cma);
+	run = kzalloc_obj(*run);
+	if (!run)
+		return -ENOMEM;
+
+	page = ne_slot_pool_alloc_run(nr_pages, numa_node_id(), &cma);
 	if (IS_ERR_OR_NULL(page)) {
 		rc = page ? PTR_ERR(page) : -ENOMEM;
 
 		dev_warn(&pdev->dev,
-			 "Slot pool granularity size=%u not allocated on node %d [rc=%d]\n",
-			 ne_pci_dev->slot_pool_gran, numa_node_id(), rc);
+			 "Slot pool granularity size=%u not allocated [rc=%d]\n",
+			 ne_pci_dev->slot_pool_gran, rc);
+
+		kfree(run);
 
 		return rc;
 	}
 
+	run->page = page;
+	run->cma = cma;
+	run->nr_pages = nr_pages;
+
 	donate_req.paddr = page_to_phys(page);
 
 	rc = ne_do_request(pdev, SLOT_POOL_DONATE,
@@ -851,21 +918,362 @@ static int ne_slot_pool_donate(struct pci_dev *pdev)
 		 * does and for the same reason.
 		 */
 		if (ne_request_unanswered(rc)) {
-			ne_pci_dev->slot_pool_page = page;
-			ne_pci_dev->slot_pool_cma = cma;
+			list_add(&run->node, &ne_pci_dev->slot_pool_meta_runs);
+			ne_pci_dev->slot_pool_meta_nr++;
 		} else {
 			ne_contig_free_run(cma, page, nr_pages);
+			kfree(run);
 		}
 
 		return rc;
 	}
 
-	ne_pci_dev->slot_pool_page = page;
-	ne_pci_dev->slot_pool_cma = cma;
+	list_add(&run->node, &ne_pci_dev->slot_pool_meta_runs);
+	ne_pci_dev->slot_pool_meta_nr++;
 
 	return 0;
 }
 
+/*
+ * Size of one run of memory the slot pool takes, and of the span the driver
+ * offers so that the device can place such a run inside it. A run begins on a
+ * boundary of its own size in the device's address space, and an address the
+ * parent aligns does not stay aligned once the device has mapped it, so the
+ * parent cannot place the run itself. A span of twice the size holds a run the
+ * device can place wherever the span begins.
+ */
+#define NE_SLOT_POOL_MEM_SIZE	SZ_128M
+#define NE_SLOT_POOL_MEM_SPAN	(2 * NE_SLOT_POOL_MEM_SIZE)
+
+/**
+ * ne_slot_pool_donate_nid() - Pick the node the next memory donation comes from.
+ * @nr_donated:	Number of runs of memory the slot pool already holds.
+ *
+ * The device turns a run of donated memory into memory it gives to enclaves, and
+ * the host grows the zone it takes that memory from on the node the run sits on.
+ * Every donation coming from one node grows that node's zone alone, so the
+ * enclaves the host places on any other node run on memory out of a zone that
+ * never grew, and the host runs that zone out and takes every enclave on the
+ * machine down with it. So spread the donations instead: step through the nodes
+ * that have memory in time with the runs already held, and successive donations
+ * come from successive nodes.
+ *
+ * A node that has no run to give costs a donation nothing, because
+ * ne_slot_pool_alloc_run() carries on from the node named here to the rest.
+ *
+ * Context: Process context.
+ * Return: the node the donation is to come from first.
+ */
+static int ne_slot_pool_donate_nid(unsigned int nr_donated)
+{
+	int nr_nodes = num_node_state(N_MEMORY);
+	unsigned int steps;
+	unsigned int i;
+	int nid;
+
+	if (nr_nodes < 1)
+		return numa_node_id();
+
+	steps = nr_donated % (unsigned int)nr_nodes;
+	nid = first_node(node_states[N_MEMORY]);
+
+	for (i = 0; i < steps; i++)
+		nid = next_node_in(nid, node_states[N_MEMORY]);
+
+	return nid;
+}
+
+/**
+ * ne_slot_pool_mem_donate() - Donate a run of memory to the slot pool.
+ * @pdev:	PCI device to donate to.
+ *
+ * The device counts the slots the pool's records support and the slots its
+ * memory supports separately, and hands out the smaller of the two counts.
+ * This donation raises the second count.
+ *
+ * The driver offers a span and the device answers with the run it took out of
+ * it, so the pages before and after that run go back to the allocator and the
+ * run alone stays recorded. A device that answers without naming a run inside
+ * the span leaves the whole span recorded, because which of the span's pages
+ * the device is using is then not established.
+ *
+ * Context: Process context. This function is called with the enclaves list
+ *	    mutex held.
+ * Return:
+ * * 0 on success.
+ * * -EINTR if a fatal signal arrived while the span was being zeroed.
+ * * Negative return value on failure.
+ * * -ETIMEDOUT, or any other code ne_request_unanswered() names, if the
+ *   request went unanswered, in which case the span stays recorded as donated
+ *   because the driver cannot establish that the device refused it. A device
+ *   that answers such a code itself lands here too and costs one held span.
+ */
+static int ne_slot_pool_mem_donate(struct pci_dev *pdev)
+{
+	const unsigned long span_pages = NE_SLOT_POOL_MEM_SPAN >> PAGE_SHIFT;
+	const unsigned long run_pages = NE_SLOT_POOL_MEM_SIZE >> PAGE_SHIFT;
+	struct ne_pci_dev *ne_pci_dev = pci_get_drvdata(pdev);
+	struct slot_pool_mem_donate_req donate_req = {};
+	struct ne_pci_dev_cmd_reply cmd_reply = {};
+	u64 span_paddr, taken_paddr, taken_size;
+	unsigned long head_pages, tail_pages;
+	struct ne_slot_pool_run *run;
+	struct cma *cma = NULL;
+	struct page *page;
+	int rc;
+
+	run = kzalloc_obj(*run);
+	if (!run)
+		return -ENOMEM;
+
+	page = ne_slot_pool_alloc_run(span_pages,
+				      ne_slot_pool_donate_nid(ne_pci_dev->slot_pool_mem_nr),
+				      &cma);
+	if (IS_ERR_OR_NULL(page)) {
+		rc = page ? PTR_ERR(page) : -ENOMEM;
+
+		dev_warn(&pdev->dev,
+			 "Slot pool memory span size=%u not allocated [rc=%d]\n",
+			 NE_SLOT_POOL_MEM_SPAN, rc);
+
+		kfree(run);
+
+		return rc;
+	}
+
+	span_paddr = page_to_phys(page);
+
+	run->page = page;
+	run->cma = cma;
+	run->nr_pages = span_pages;
+
+	donate_req.paddr = span_paddr;
+	donate_req.size = NE_SLOT_POOL_MEM_SPAN;
+
+	rc = ne_do_request(pdev, SLOT_POOL_MEM_DONATE,
+			   &donate_req, sizeof(donate_req),
+			   &cmd_reply, sizeof(cmd_reply));
+	if (rc < 0) {
+		dev_err_ratelimited(&pdev->dev,
+				    "Error in slot pool memory donate [rc=%d]\n",
+				    rc);
+
+		/*
+		 * Record a span whose donation went unanswered as donated
+		 * instead of giving it back, as ne_slot_pool_donate_base() does
+		 * and for the same reason. The whole span is recorded, because
+		 * the run the device took out of it is what the answer that
+		 * never came would have named.
+		 */
+		if (ne_request_unanswered(rc)) {
+			list_add(&run->node, &ne_pci_dev->slot_pool_mem_runs);
+			ne_pci_dev->slot_pool_mem_nr++;
+		} else {
+			ne_contig_free_run(cma, page, span_pages);
+			kfree(run);
+		}
+
+		return rc;
+	}
+
+	taken_paddr = cmd_reply.mem_donate.consumed_paddr;
+	taken_size = cmd_reply.mem_donate.consumed_size;
+
+	/*
+	 * The run the device names has to be one run's worth, inside the span
+	 * it was offered, and to begin on a page boundary of the span, since
+	 * the pages given back are counted from where it begins. The device
+	 * places the run on a boundary of its own address space, which the
+	 * span's physical address says nothing about, so where in the span it
+	 * begins is only bounded, not fixed.
+	 */
+	if (taken_size == NE_SLOT_POOL_MEM_SIZE &&
+	    taken_paddr >= span_paddr &&
+	    taken_paddr - span_paddr <=
+		    NE_SLOT_POOL_MEM_SPAN - NE_SLOT_POOL_MEM_SIZE &&
+	    IS_ALIGNED(taken_paddr - span_paddr, PAGE_SIZE)) {
+		head_pages = (taken_paddr - span_paddr) >> PAGE_SHIFT;
+		tail_pages = span_pages - head_pages - run_pages;
+
+		run->page = page + head_pages;
+		run->nr_pages = run_pages;
+
+		if (head_pages)
+			ne_contig_free_run(cma, page, head_pages);
+
+		if (tail_pages)
+			ne_contig_free_run(cma, run->page + run_pages,
+					   tail_pages);
+	} else {
+		dev_warn(&pdev->dev,
+			 "Slot pool memory span kept, the device took paddr=0x%llx size=0x%llx\n",
+			 taken_paddr, taken_size);
+	}
+
+	list_add(&run->node, &ne_pci_dev->slot_pool_mem_runs);
+	ne_pci_dev->slot_pool_mem_nr++;
+
+	return 0;
+}
+
+/**
+ * ne_slot_pool_limits() - Read how many slots each half of the slot pool has
+ *			   room for.
+ * @pdev:	PCI device to read the slot pool limits of.
+ * @cmd_reply:	Reply the limits are read out of, valid on success.
+ *
+ * The reply is a snapshot. A caller that donates on the strength of it and then
+ * reads it again holds the enclaves list mutex across both.
+ *
+ * Context: Process context.
+ * Return:
+ * * 0 on success.
+ * * Negative return value on failure.
+ */
+static int ne_slot_pool_limits(struct pci_dev *pdev,
+			       struct ne_pci_dev_cmd_reply *cmd_reply)
+{
+	u8 unused = 0;
+
+	return ne_do_request(pdev, SLOT_POOL_GET_LIMITS,
+			     &unused, sizeof(unused),
+			     cmd_reply, sizeof(*cmd_reply));
+}
+
+/**
+ * ne_slot_pool_effective_limit() - Read how many slots the parent can use out
+ *				    of a reply the device filled.
+ * @cmd_reply:	Reply a SLOT_POOL_GET_LIMITS filled.
+ *
+ * A slot needs room in both halves of the pool, so the number of slots the
+ * parent can use is the smaller of the two the device reports.
+ *
+ * Context: Process context.
+ * Return: the number of slots the parent can use.
+ */
+static u16 ne_slot_pool_effective_limit(const struct ne_pci_dev_cmd_reply *cmd_reply)
+{
+	return min(cmd_reply->limits.meta_limit, cmd_reply->limits.mem_limit);
+}
+
+/**
+ * ne_slot_pool_grow() - Grow the slot pool until it holds @target slots.
+ * @pdev:	PCI device to donate to.
+ * @target:	Number of slots the pool is to hold.
+ *
+ * The device hands out the smaller of the slots its records support and the
+ * slots its memory supports, so each round donates to whichever of the two is
+ * behind and then asks the device for both counts again. Neither is behind
+ * when the two are level, and a round donates to both then, because a
+ * donation to one half of a level pair leaves the smaller count where it was.
+ *
+ * A round that leaves both counts where they were ends the growth: the device
+ * took every donation the round offered and got no further with any of them,
+ * and another round would allocate again for the same answer.
+ *
+ * The pool only grows. A run the device accepted cannot be taken back on its
+ * own, so growth that stops short leaves the pool at the size it reached.
+ *
+ * Context: Process context. This function is called with the enclaves list
+ *	    mutex held.
+ * Return:
+ * * 0 once the pool holds @target slots or more.
+ * * -EOPNOTSUPP on a device that takes no donation.
+ * * -EINVAL where @target is above the number of slots the device allows.
+ * * -ENOSPC where a round raised neither count and the pool is below @target.
+ * * -EINTR if a fatal signal arrived while a run was being zeroed.
+ * * Negative return value on failure.
+ */
+int ne_slot_pool_grow(struct pci_dev *pdev, u16 target)
+{
+	struct ne_pci_dev *ne_pci_dev = pci_get_drvdata(pdev);
+	struct ne_pci_dev_cmd_reply cmd_reply = {};
+	u16 effective;
+	int rc;
+
+	if (!ne_pci_dev->slot_pool_max)
+		return -EOPNOTSUPP;
+
+	if (target > ne_pci_dev->slot_pool_max)
+		return -EINVAL;
+
+	rc = ne_slot_pool_donate_base(pdev);
+	if (rc < 0)
+		return rc;
+
+	rc = ne_slot_pool_limits(pdev, &cmd_reply);
+	if (rc < 0)
+		return rc;
+
+	effective = ne_slot_pool_effective_limit(&cmd_reply);
+
+	while (effective < target) {
+		u16 meta_before = cmd_reply.limits.meta_limit;
+		u16 mem_before = cmd_reply.limits.mem_limit;
+
+		if (cmd_reply.limits.mem_limit <= cmd_reply.limits.meta_limit) {
+			rc = ne_slot_pool_mem_donate(pdev);
+			if (rc < 0)
+				return rc;
+		}
+
+		if (cmd_reply.limits.meta_limit <= cmd_reply.limits.mem_limit) {
+			rc = ne_slot_pool_donate(pdev);
+			if (rc < 0)
+				return rc;
+		}
+
+		rc = ne_slot_pool_limits(pdev, &cmd_reply);
+		if (rc < 0)
+			return rc;
+
+		if (cmd_reply.limits.meta_limit <= meta_before &&
+		    cmd_reply.limits.mem_limit <= mem_before)
+			return -ENOSPC;
+
+		effective = ne_slot_pool_effective_limit(&cmd_reply);
+	}
+
+	return 0;
+}
+
+/**
+ * enum ne_slot_pool_runs_owner - Who keeps the runs of a drained list.
+ * @NE_RUNS_TO_ALLOCATOR:	The device handed the runs back, so they go back
+ *				to the allocator.
+ * @NE_RUNS_STAY_WITH_DEVICE:	The device may still use the runs, so only the
+ *				records of them go away.
+ */
+enum ne_slot_pool_runs_owner {
+	NE_RUNS_TO_ALLOCATOR,
+	NE_RUNS_STAY_WITH_DEVICE,
+};
+
+/**
+ * ne_slot_pool_runs_drain() - Empty a list of donated runs.
+ * @runs:	List of runs to empty.
+ * @nr:		Count of runs on @runs, set to zero.
+ * @owner:	Who keeps the runs.
+ *
+ * Context: Process context. This function is called with the enclaves list
+ *	    mutex held.
+ */
+static void ne_slot_pool_runs_drain(struct list_head *runs, unsigned int *nr,
+				    enum ne_slot_pool_runs_owner owner)
+{
+	struct ne_slot_pool_run *run, *next;
+
+	list_for_each_entry_safe(run, next, runs, node) {
+		if (owner == NE_RUNS_TO_ALLOCATOR)
+			ne_contig_free_run(run->cma, run->page, run->nr_pages);
+
+		list_del(&run->node);
+		kfree(run);
+	}
+
+	*nr = 0;
+}
+
 /**
  * ne_slot_pool_release() - Take the whole slot pool back from the device.
  * @pdev:	PCI device to take the pool back from.
@@ -904,17 +1312,15 @@ static void ne_slot_pool_release(struct pci_dev *pdev)
 	if (rc < 0) {
 		dev_err(&pdev->dev, "Error in slot pool shutdown [rc=%d]\n", rc);
 
-		goto unlock_mutex;
+		goto forget_runs;
 	}
 
-	if (ne_pci_dev->slot_pool_page) {
-		ne_contig_free_run(ne_pci_dev->slot_pool_cma,
-				   ne_pci_dev->slot_pool_page,
-				   ne_pci_dev->slot_pool_gran >> PAGE_SHIFT);
-
-		ne_pci_dev->slot_pool_page = NULL;
-		ne_pci_dev->slot_pool_cma = NULL;
-	}
+	ne_slot_pool_runs_drain(&ne_pci_dev->slot_pool_meta_runs,
+				&ne_pci_dev->slot_pool_meta_nr,
+				NE_RUNS_TO_ALLOCATOR);
+	ne_slot_pool_runs_drain(&ne_pci_dev->slot_pool_mem_runs,
+				&ne_pci_dev->slot_pool_mem_nr,
+				NE_RUNS_TO_ALLOCATOR);
 
 	ne_contig_free_run(ne_pci_dev->slot_pool_base_cma,
 			   ne_pci_dev->slot_pool_base_page,
@@ -923,6 +1329,20 @@ static void ne_slot_pool_release(struct pci_dev *pdev)
 	ne_pci_dev->slot_pool_base_page = NULL;
 	ne_pci_dev->slot_pool_base_cma = NULL;
 
+	goto unlock_mutex;
+
+forget_runs:
+	/*
+	 * The device keeps these runs, so only the records of them go, and the
+	 * pages they name stay the device's for as long as it is bound.
+	 */
+	ne_slot_pool_runs_drain(&ne_pci_dev->slot_pool_meta_runs,
+				&ne_pci_dev->slot_pool_meta_nr,
+				NE_RUNS_STAY_WITH_DEVICE);
+	ne_slot_pool_runs_drain(&ne_pci_dev->slot_pool_mem_runs,
+				&ne_pci_dev->slot_pool_mem_nr,
+				NE_RUNS_STAY_WITH_DEVICE);
+
 unlock_mutex:
 	mutex_unlock(&ne_pci_dev->enclaves_list_mutex);
 }
@@ -976,6 +1396,8 @@ static int ne_pci_probe(struct pci_dev *pdev, const struct pci_device_id *id)
 	atomic_set(&ne_pci_dev->cmd_reply_avail, 0);
 	init_waitqueue_head(&ne_pci_dev->cmd_reply_wait_q);
 	INIT_LIST_HEAD(&ne_pci_dev->enclaves_list);
+	INIT_LIST_HEAD(&ne_pci_dev->slot_pool_meta_runs);
+	INIT_LIST_HEAD(&ne_pci_dev->slot_pool_mem_runs);
 	mutex_init(&ne_pci_dev->enclaves_list_mutex);
 	mutex_init(&ne_pci_dev->pci_dev_mutex);
 	ne_pci_dev->pdev = pdev;
diff --git a/drivers/virt/nitro_enclaves/ne_pci_dev.h b/drivers/virt/nitro_enclaves/ne_pci_dev.h
index 3718f893e4f65..cab5434fc3e0d 100644
--- a/drivers/virt/nitro_enclaves/ne_pci_dev.h
+++ b/drivers/virt/nitro_enclaves/ne_pci_dev.h
@@ -468,6 +468,24 @@ static_assert(offsetof(struct ne_pci_dev_cmd_reply,
 		       limits.slots_per_donation) == 12);
 static_assert(offsetof(struct ne_pci_dev_cmd_reply, limits.reserved14) == 14);
 
+/**
+ * struct ne_slot_pool_run - One run of pages the driver gave the slot pool.
+ * @node:	Entry on the list of runs donated to one of the pools.
+ * @page:	First page of the run.
+ * @cma:	CMA area the run came from.
+ * @nr_pages:	Number of pages in the run.
+ *
+ * A record is allocated before the run it describes, and the run is recorded
+ * before the request carrying its address goes out, so a run the device
+ * accepted is a run ne_slot_pool_release() finds.
+ */
+struct ne_slot_pool_run {
+	struct list_head	node;
+	struct page		*page;
+	struct cma		*cma;
+	unsigned long		nr_pages;
+};
+
 /**
  * struct ne_pci_dev - Nitro Enclaves (NE) PCI device.
  * @cmd_reply_avail:		Variable set if a reply has been sent by the
@@ -501,9 +519,11 @@ static_assert(offsetof(struct ne_pci_dev_cmd_reply, limits.reserved14) == 14);
  *				has been made.
  * @slot_pool_base_cma:		CMA area the base donation came from.
  * @slot_pool_base_nr_pages:	Number of pages in the base donation.
- * @slot_pool_page:		First page of the donation to the slot pool, NULL
- *				until it has been made.
- * @slot_pool_cma:		CMA area the donation came from.
+ * @slot_pool_meta_runs:	Runs donated to hold the pool's slot descriptors.
+ * @slot_pool_meta_nr:		Number of runs on @slot_pool_meta_runs.
+ * @slot_pool_mem_runs:		Runs donated for the enclaves in the pool's
+ *				slots to run in.
+ * @slot_pool_mem_nr:		Number of runs on @slot_pool_mem_runs.
  */
 struct ne_pci_dev {
 	atomic_t		cmd_reply_avail;
@@ -522,13 +542,18 @@ struct ne_pci_dev {
 	struct page		*slot_pool_base_page;
 	struct cma		*slot_pool_base_cma;
 	unsigned long		slot_pool_base_nr_pages;
-	struct page		*slot_pool_page;
-	struct cma		*slot_pool_cma;
+	struct list_head	slot_pool_meta_runs;
+	unsigned int		slot_pool_meta_nr;
+	struct list_head	slot_pool_mem_runs;
+	unsigned int		slot_pool_mem_nr;
 };
 
 /* Whether a failure left a submitted request in flight. */
 bool ne_request_unanswered(int rc);
 
+/* Grow the slot pool until it holds a given number of slots. */
+int ne_slot_pool_grow(struct pci_dev *pdev, u16 target);
+
 /**
  * ne_do_request() - Submit command request to the PCI device based on the command
  *		     type and retrieve the associated reply.
-- 
2.47.1

[PATCH 21/40] nitro_enclaves: Document the slot pool

[Alexander Graf]

An admin sizing a CMA reservation for Nitro Sandboxes needs to allow for
the memory the driver donates to the slot pool as well as for the
enclaves' own allocations. Document both halves of the pool: which
device asks for them, what probe donates, how they grow, where the
memory comes from and when it goes back.

Signed-off-by: Alexander Graf <graf@amazon.com>
---
 Documentation/virt/ne_overview.rst | 31 ++++++++++++++++++++++++++++++
 1 file changed, 31 insertions(+)

diff --git a/Documentation/virt/ne_overview.rst b/Documentation/virt/ne_overview.rst
index 3cfc50f788433..16d6b4a2771c7 100644
--- a/Documentation/virt/ne_overview.rst
+++ b/Documentation/virt/ne_overview.rst
@@ -215,3 +215,34 @@ An allocation lives until the last of the things holding it is gone: its
 descriptor, its mappings, the regions given from it and the enclave it is the
 info page of. So neither closing the descriptor nor unmapping takes memory away
 from a running enclave.
+
+Slot pool
+---------
+
+The device keeps a record for every enclave slot it hands out. On a device that
+holds those records in parent memory the driver donates the memory for them
+before the device can allocate a slot at all, and that device draws the memory
+an enclave runs on from parent memory too. It counts both donations in slots: a
+record pool that holds so many records, and a memory pool that backs so many
+enclaves. Neither number on its own says how many enclaves can exist, because
+each one needs a record and the memory behind it, so the answer is whichever
+pool carries fewer. Such a device reports the highest number of slots it will
+grow to, the size and alignment of one donation to the record pool, the size of
+the base donation that pool hangs off, and how many slots one donation to the
+memory pool is worth. A device that holds the records itself reports no slots
+and takes no donation.
+
+The driver reads those numbers when it probes and donates the base run the
+device asked for and one run of the granularity it reported, which is the
+record pool an enclave needs to be created at all. Both runs are asked of the
+node the probe runs on first, through the allocator an enclave allocation uses,
+so a per-node CMA reservation there covers the pool as well as the enclaves. A
+run that node cannot serve comes from any node that has memory, because a run
+in the pool is read by the device alone. Probing carries on when a donation
+fails, with a warning naming the error. The memory pool starts at the number of
+slots the device backs without a donation.
+
+The device holds every donated page until it is told to let go, so the driver
+takes the whole pool back when a probe fails after donating, when it is unbound
+and when the machine shuts down, and the pages reach the allocator only after
+that has returned. Pages the device will not give back stay with it.
-- 
2.47.1

[PATCH 22/40] nitro_enclaves: Expose slot pool state under sysfs

[Alexander Graf]

An operator who wants a host to run more sandboxes needs to see how many
slots the slot pool has and what was donated to it, and to grow it.

Report the pool size, donations and limits in sysfs under slot_pool/ on
the misc device.

Writes to the limit allow an admin to increase the pool size. Pools are
increase only. To reduce the pool donation, you need to unload the driver.

Signed-off-by: Alexander Graf <graf@amazon.com>
---
 .../sysfs-devices-virtual-misc-nitro_enclaves | 156 ++++++++++++++++
 Documentation/virt/ne_overview.rst            |   9 +
 drivers/virt/nitro_enclaves/ne_misc_dev.c     |   6 +
 drivers/virt/nitro_enclaves/ne_pci_dev.c      | 173 +++++++++++++++++-
 drivers/virt/nitro_enclaves/ne_pci_dev.h      |   3 +
 5 files changed, 346 insertions(+), 1 deletion(-)

diff --git a/Documentation/ABI/testing/sysfs-devices-virtual-misc-nitro_enclaves b/Documentation/ABI/testing/sysfs-devices-virtual-misc-nitro_enclaves
index c549364353186..a8741ae97f122 100644
--- a/Documentation/ABI/testing/sysfs-devices-virtual-misc-nitro_enclaves
+++ b/Documentation/ABI/testing/sysfs-devices-virtual-misc-nitro_enclaves
@@ -74,3 +74,159 @@ Description:
 		The directories appear once the device has been probed and the
 		misc device registered, so a reader that opens the misc device
 		the moment it appears can find cpu_pool empty.
+
+What:		/sys/devices/virtual/misc/nitro_enclaves/slot_pool/max
+What:		/sys/devices/virtual/misc/nitro_enclaves/slot_pool/granularity
+What:		/sys/devices/virtual/misc/nitro_enclaves/slot_pool/base_size
+What:		/sys/devices/virtual/misc/nitro_enclaves/slot_pool/base_donated
+What:		/sys/devices/virtual/misc/nitro_enclaves/slot_pool/gran_donated
+Date:		November 2026
+KernelVersion:	7.5
+Contact:	aws-nitro-enclaves-devel@amazon.com
+Description:
+		Read-only. One value each, describing the memory the driver
+		donated to the device's slot pool. A device that keeps its slot
+		descriptors itself asks for no donation and reports max as 0,
+		and then the remaining files describe nothing:
+
+		max
+			highest number of slots the device lets the pool hold,
+			as the device reports it. It reads 0 in three cases
+			that it does not tell apart: the device asks for no
+			donation, the device publishes no vendor capability so
+			the register was never read, or the driver turned down
+			the donation geometry the device reported. Only the
+			last of those names itself, in a dev_dbg() carrying
+			the size and alignment that were refused
+
+		granularity
+			size in bytes of one donation to the record pool, or 0
+			where the driver turned the geometry the device reported
+			down
+
+		base_size
+			size in bytes the device asks for as its base donation
+
+		base_donated
+			1 where the base donation was made, 0 where it was not.
+			The pool takes one base donation and no more
+
+		gran_donated
+			how many donations of granularity bytes were made to the
+			record pool. Probe makes one and a write to limit makes
+			the rest
+
+		A donation is drawn from the CMA area of the node the driver
+		asks first and, where that area cannot serve it, from the CMA
+		area of each other node that has memory in turn. It fails when
+		none of them can serve it. Comparing base_size and granularity
+		against base_donated and gran_donated says whether a larger
+		reservation would have changed the outcome.
+
+		These files change while the device is bound, because a write
+		to limit donates. A read takes the enclaves list mutex, which
+		is the mutex a donation and the release of the whole pool take,
+		so a read sees the pool between donations rather than part way
+		through one, and it can wait for a donation, an enclave or a
+		write to limit to finish. -ENODEV says the driver has no device
+		to describe.
+
+What:		/sys/devices/virtual/misc/nitro_enclaves/slot_pool/limit
+Date:		November 2026
+KernelVersion:	7.5
+Contact:	aws-nitro-enclaves-devel@amazon.com
+Description:
+		How many enclaves the device can hold on the memory it has
+		now, and the file to write to give it more.
+
+		The device counts each of its two pools in slots and an
+		enclave needs one in both, one for the record the device keeps
+		of it and one for the memory it runs on, so a read reports
+		whichever pool carries fewer. That is the number of enclaves
+		that can exist at once. It does not count the enclaves that
+		already do, so no difference between this file and max or
+		in_use is a count of the enclaves still to be had; whether one
+		can be had is what asking for it answers.
+
+		Writing a number that is no larger than max and no smaller
+		than the number read donates memory to whichever pool carries
+		fewer, one donation at a time, until both carry that many.
+		Writing the number already read donates nothing and succeeds.
+
+		A write that fails part-way neither reverts nor completes: the
+		driver cannot take a slot pool donation the device accepted
+		back one run at a time, so the pool stays at whatever size it
+		reached, and reading the file back reports that size. Each
+		individual donation is all-or-nothing, so a failed write never
+		leaves a run half given: what it leaves is fewer slots than
+		were asked for. The write returns the error the donation
+		failed with, and retrying carries on from where it stopped
+		rather than starting again.
+
+		A write can be refused with:
+
+		-EINVAL
+			the number is larger than max, or is not a number
+
+		-ERANGE
+			the number does not fit
+
+		-EOPNOTSUPP
+			the number is smaller than the number read, or the
+			device takes no donation. The device is never told to
+			give a slot's worth of memory back, so the pools do not
+			shrink
+
+		-ENOSPC
+			the device took a donation into each pool that was
+			behind and raised neither, or it reports a pool is at
+			its ceiling, so no further donation can raise the
+			number
+
+		-ENOMEM
+			no node's CMA area could serve a run to donate
+
+		-ENODEV
+			the driver has no device to describe
+
+		-EAGAIN
+			the device kept asking for a donation to be sent again
+
+		-ETIMEDOUT
+			the device did not answer a donation. It may hold the
+			memory that donation offered
+
+		-EINTR
+			a fatal signal arrived while donated memory was being
+			cleared
+
+		The device may answer a donation with any of its own errors,
+		and a write passes that through as well.
+
+		A read asks the device rather than reading a register, so it
+		can fail for the same reasons a write can. Both a read and a
+		write take the enclaves list mutex, and a write holds it for
+		every donation it makes, so no enclave can be created or
+		released while one is growing the pool, and a read waits for
+		it to finish.
+
+What:		/sys/devices/virtual/misc/nitro_enclaves/slot_pool/in_use
+Date:		November 2026
+KernelVersion:	7.5
+Contact:	aws-nitro-enclaves-devel@amazon.com
+Description:
+		Read-only. How many enclaves hold a slot now. An enclave holds
+		one from the moment the device allocates it until the device
+		frees it. That is normally when the enclave's file descriptor
+		is closed. Where the device turns the free down, the enclave
+		keeps its slot, its memory and its CPU threads for good, and
+		the driver sends the device no further command.
+
+		in_use is not a capacity. Nothing here or in max or limit can
+		be subtracted to give the slots a new enclave can have; the
+		device answers that when it is asked for one, and it can refuse
+		while in_use reads 0.
+
+		A read takes the enclaves list mutex, so it can wait for an
+		enclave to be created or released and for a write to limit to
+		finish.
diff --git a/Documentation/virt/ne_overview.rst b/Documentation/virt/ne_overview.rst
index 16d6b4a2771c7..ba030cde59989 100644
--- a/Documentation/virt/ne_overview.rst
+++ b/Documentation/virt/ne_overview.rst
@@ -242,6 +242,15 @@ in the pool is read by the device alone. Probing carries on when a donation
 fails, with a warning naming the error. The memory pool starts at the number of
 slots the device backs without a donation.
 
+slot_pool/limit is what a parent grows the pools with. It reads the number of
+enclaves the two carry now, and writing a larger number donates to whichever
+one carries fewer, a donation at a time, until both carry that many. A slot
+pool donation the device has taken is the device's until the whole pool goes
+back, so a write that fails part way leaves what it donated before the failure
+in place and the number the file reads afterwards is where it stopped. Writing
+a smaller number is refused: the device is told about memory it may put records
+or enclaves in, and it is never told to give a slot's worth of that back.
+
 The device holds every donated page until it is told to let go, so the driver
 takes the whole pool back when a probe fails after donating, when it is unbound
 and when the machine shuts down, and the pages reach the allocator only after
diff --git a/drivers/virt/nitro_enclaves/ne_misc_dev.c b/drivers/virt/nitro_enclaves/ne_misc_dev.c
index 2fae731acb977..208c1997609d3 100644
--- a/drivers/virt/nitro_enclaves/ne_misc_dev.c
+++ b/drivers/virt/nitro_enclaves/ne_misc_dev.c
@@ -425,11 +425,17 @@ static const struct attribute_group ne_cpu_pool_node_group = {
 static struct kobject *ne_cpu_pool_kobj;
 static struct kobject **ne_cpu_pool_node_kobj;
 
+static const struct attribute_group *ne_groups[] = {
+	&ne_slot_pool_group,
+	NULL,
+};
+
 static struct miscdevice ne_misc_dev = {
 	.minor	= MISC_DYNAMIC_MINOR,
 	.name	= "nitro_enclaves",
 	.fops	= &ne_fops,
 	.mode	= 0660,
+	.groups	= ne_groups,
 };
 
 struct ne_devs ne_devs = {
diff --git a/drivers/virt/nitro_enclaves/ne_pci_dev.c b/drivers/virt/nitro_enclaves/ne_pci_dev.c
index 9e304ffc2bd13..1aa4d5508dabf 100644
--- a/drivers/virt/nitro_enclaves/ne_pci_dev.c
+++ b/drivers/virt/nitro_enclaves/ne_pci_dev.c
@@ -13,6 +13,7 @@
 #include <linux/err.h>
 #include <linux/iopoll.h>
 #include <linux/jiffies.h>
+#include <linux/kstrtox.h>
 #include <linux/list.h>
 #include <linux/minmax.h>
 #include <linux/module.h>
@@ -849,7 +850,8 @@ static int ne_slot_pool_donate_base(struct pci_dev *pdev)
  *
  * A donation raises the number of slots the pool has room for, up to the
  * ceiling the device reports, and a pool that has reached that ceiling takes no
- * further donation.
+ * further donation. The pool takes as many of these as the device will
+ * hold. Probe makes one and a write to slot_pool/limit makes the rest.
  *
  * Context: Process context. This function is called with the enclaves list
  *	    mutex held.
@@ -1347,6 +1349,175 @@ static void ne_slot_pool_release(struct pci_dev *pdev)
 	mutex_unlock(&ne_pci_dev->enclaves_list_mutex);
 }
 
+/**
+ * ne_slot_pool_in_use() - Count the enclaves holding a slot.
+ * @ne_pci_dev:	Device whose enclaves to count.
+ *
+ * An enclave holds one slot from the moment the device allocates it until the
+ * device frees it, so the enclaves the driver has a record of are the slots
+ * in use, including those whose slot the device did not free when their file
+ * was closed.
+ *
+ * Context: Process context. This function is called with the enclaves list
+ *	    mutex held.
+ * Return: Number of enclaves holding a slot.
+ */
+static unsigned int ne_slot_pool_in_use(struct ne_pci_dev *ne_pci_dev)
+{
+	struct ne_enclave *ne_enclave;
+	unsigned int n = 0;
+
+	list_for_each_entry(ne_enclave, &ne_pci_dev->enclaves_list,
+			    enclave_list_entry)
+		n++;
+
+	return n;
+}
+
+/*
+ * slot_pool/ sysfs group on the Nitro Enclaves misc device: report what the
+ * driver donated to the device's slot pool, what the device asked for, and how
+ * many enclaves the pools carry.
+ *
+ *   /sys/devices/virtual/misc/nitro_enclaves/slot_pool/
+ *
+ * The device counts each of its two pools in slots and an enclave needs one in
+ * both, so the number of enclaves that can exist is whichever pool carries
+ * fewer, and limit reports that number rather than either pool's own. Writing
+ * limit donates to the pool that is behind until both carry what was asked
+ * for. The other files say what was donated, how large the runs the device
+ * asked for were and which allocator served them, which is what an operator
+ * needs to decide whether a larger CMA reservation would have changed the
+ * outcome. A device that keeps its slot descriptors itself asks for nothing
+ * and reports max as 0, and then none of the other files describe anything.
+ *
+ * Every file here takes the enclaves list mutex, and so does every donation
+ * and ne_slot_pool_release(), so a read sees the pool between donations and
+ * never part way through one, and a read during teardown sees it whole or
+ * gone. Reads and writes land only between misc_register() and
+ * misc_deregister(), and neither sits inside a region that mutex covers, so a
+ * read waits on a donation, a write or an enclave and never on a teardown that
+ * is waiting on the read. base_size is a device register and the mutex says
+ * nothing about the device, so it is what the device reports at the moment of
+ * the read. A read answers -ENODEV where no device is bound.
+ */
+
+#define NE_SLOT_POOL_ATTR(_name, _fmt, _expr)					\
+static ssize_t _name##_show(struct device *dev,					\
+			    struct device_attribute *attr, char *buf)		\
+{										\
+	struct ne_pci_dev *ne_pci_dev = ne_devs.ne_pci_dev;			\
+	ssize_t n;								\
+										\
+	if (!ne_pci_dev)							\
+		return -ENODEV;							\
+										\
+	n = mutex_lock_killable(&ne_pci_dev->enclaves_list_mutex);			\
+	if (n)									\
+		return n;							\
+										\
+	n = sysfs_emit(buf, _fmt "\n", _expr);					\
+	mutex_unlock(&ne_pci_dev->enclaves_list_mutex);				\
+										\
+	return n;								\
+}										\
+static DEVICE_ATTR_RO(_name)
+
+NE_SLOT_POOL_ATTR(max, "%u", ne_pci_dev->slot_pool_max);
+NE_SLOT_POOL_ATTR(granularity, "%u", ne_pci_dev->slot_pool_gran);
+NE_SLOT_POOL_ATTR(base_size, "%u",
+		  ioread32(ne_pci_dev->iomem_base + NE_SLOT_POOL_BASE_SIZE));
+NE_SLOT_POOL_ATTR(base_donated, "%d", !!ne_pci_dev->slot_pool_base_page);
+NE_SLOT_POOL_ATTR(gran_donated, "%u", ne_pci_dev->slot_pool_meta_nr);
+NE_SLOT_POOL_ATTR(in_use, "%u", ne_slot_pool_in_use(ne_pci_dev));
+
+#undef NE_SLOT_POOL_ATTR
+
+/*
+ * limit is hand-written rather than generated: its read issues a command that
+ * can fail, which the macro's single expression has no way to return, and it
+ * is the only file in the group that is written.
+ */
+static ssize_t limit_show(struct device *dev, struct device_attribute *attr,
+			  char *buf)
+{
+	struct ne_pci_dev *ne_pci_dev = ne_devs.ne_pci_dev;
+	struct ne_pci_dev_cmd_reply cmd_reply = {};
+	ssize_t rc;
+
+	if (!ne_pci_dev)
+		return -ENODEV;
+
+	rc = mutex_lock_killable(&ne_pci_dev->enclaves_list_mutex);
+	if (rc)
+		return rc;
+
+	rc = ne_slot_pool_limits(ne_pci_dev->pdev, &cmd_reply);
+	if (rc >= 0)
+		rc = sysfs_emit(buf, "%u\n",
+				ne_slot_pool_effective_limit(&cmd_reply));
+
+	mutex_unlock(&ne_pci_dev->enclaves_list_mutex);
+
+	return rc;
+}
+
+static ssize_t limit_store(struct device *dev, struct device_attribute *attr,
+			   const char *buf, size_t count)
+{
+	struct ne_pci_dev *ne_pci_dev = ne_devs.ne_pci_dev;
+	struct ne_pci_dev_cmd_reply cmd_reply = {};
+	u16 target;
+	ssize_t rc;
+
+	if (!ne_pci_dev)
+		return -ENODEV;
+
+	rc = kstrtou16(buf, 0, &target);
+	if (rc < 0)
+		return rc;
+
+	rc = mutex_lock_killable(&ne_pci_dev->enclaves_list_mutex);
+	if (rc)
+		return rc;
+
+	rc = ne_slot_pool_limits(ne_pci_dev->pdev, &cmd_reply);
+	if (rc < 0)
+		goto unlock;
+
+	if (target < ne_slot_pool_effective_limit(&cmd_reply)) {
+		rc = -EOPNOTSUPP;
+		goto unlock;
+	}
+
+	rc = ne_slot_pool_grow(ne_pci_dev->pdev, target);
+	if (rc < 0)
+		goto unlock;
+
+	rc = count;
+unlock:
+	mutex_unlock(&ne_pci_dev->enclaves_list_mutex);
+
+	return rc;
+}
+static DEVICE_ATTR_RW(limit);
+
+static struct attribute *ne_slot_pool_attrs[] = {
+	&dev_attr_max.attr,
+	&dev_attr_limit.attr,
+	&dev_attr_granularity.attr,
+	&dev_attr_base_size.attr,
+	&dev_attr_base_donated.attr,
+	&dev_attr_gran_donated.attr,
+	&dev_attr_in_use.attr,
+	NULL,
+};
+
+const struct attribute_group ne_slot_pool_group = {
+	.name	= "slot_pool",
+	.attrs	= ne_slot_pool_attrs,
+};
+
 /**
  * ne_pci_probe() - Probe function for the NE PCI device.
  * @pdev:	PCI device to match with the NE PCI driver.
diff --git a/drivers/virt/nitro_enclaves/ne_pci_dev.h b/drivers/virt/nitro_enclaves/ne_pci_dev.h
index cab5434fc3e0d..f5cb24312d442 100644
--- a/drivers/virt/nitro_enclaves/ne_pci_dev.h
+++ b/drivers/virt/nitro_enclaves/ne_pci_dev.h
@@ -578,4 +578,7 @@ int ne_do_request(struct pci_dev *pdev, enum ne_pci_dev_cmd_type cmd_type,
 /* Nitro Enclaves (NE) PCI device driver */
 extern struct pci_driver ne_pci_driver;
 
+/* slot_pool/ sysfs group on the NE misc device */
+extern const struct attribute_group ne_slot_pool_group;
+
 #endif /* _NE_PCI_DEV_H_ */
-- 
2.47.1

[PATCH 23/40] nitro_enclaves: Add the device add command

[Alexander Graf]

In preparation to support Nitro Sandboxes, which allow the parent to
back arbitrary virtio devices, add the command definition for adding
devices to a slot.

The device add command specifies the virtio device shape (type, nr VQs,
features, config space), but also allows to specify backing ranges for
DMB or SHM regions.

The full response with the device information is part of the info page
at the returned offset.

Signed-off-by: Alexander Graf <graf@amazon.com>
---
 drivers/virt/nitro_enclaves/ne_pci_dev.h | 88 ++++++++++++++++++++++++
 1 file changed, 88 insertions(+)

diff --git a/drivers/virt/nitro_enclaves/ne_pci_dev.h b/drivers/virt/nitro_enclaves/ne_pci_dev.h
index f5cb24312d442..3445318b84672 100644
--- a/drivers/virt/nitro_enclaves/ne_pci_dev.h
+++ b/drivers/virt/nitro_enclaves/ne_pci_dev.h
@@ -161,6 +161,7 @@
  * @SLOT_ADD_BULK_VCPUS:	Add a number of vCPUs, not providing CPU ids.
  * @SLOT_ADD_INFO_PAGE:		Give a slot the info page, a memory region the
  *				device writes.
+ * @SLOT_ADD_DEVICE:		Give a slot a device the enclave will drive.
  * @SLOT_POOL_DONATE:		Donate one page of parent memory to the slot
  *				pool.
  * @SLOT_POOL_SHUTDOWN:		Give the whole slot pool back.
@@ -186,6 +187,7 @@ enum ne_pci_dev_cmd_type {
 	SLOT_ADD_BULK_VCPUS	= 11,
 	/* 12 is a command this driver does not implement. */
 	SLOT_ADD_INFO_PAGE	= 13,
+	SLOT_ADD_DEVICE		= 14,
 	SLOT_POOL_DONATE	= 16,
 	SLOT_POOL_SHUTDOWN	= 17,
 	SLOT_POOL_BASE_DONATE	= 25,
@@ -333,6 +335,73 @@ struct slot_add_info_page_req {
 	u64	info_size;
 };
 
+/**
+ * NE_MAX_VQS_PER_DEVICE - Highest number of virtqueues one device may have.
+ */
+#define NE_MAX_VQS_PER_DEVICE		(64)
+
+/**
+ * NE_MAX_BACKING_RANGES - Highest number of memory ranges one device may be
+ *			   backed by.
+ */
+#define NE_MAX_BACKING_RANGES		(4)
+
+/**
+ * struct ne_backing_range - One run of enclave memory a device works out of.
+ * @phys_addr:	Physical address of the run, in the parent's address space.
+ * @size:	Size, in bytes, of the run.
+ */
+struct ne_backing_range {
+	u64	phys_addr;
+	u64	size;
+};
+
+/**
+ * struct ne_vq_wiring - How the enclave reaches one virtqueue of a device.
+ * @msix_vector:		Interrupt vector the device raises for the queue.
+ * @notify_index:		Index the enclave writes to notify the device.
+ * @msix_entry:			Entry in the enclave's interrupt table holding
+ *				@msix_vector.
+ * @doorbell_offset:		Offset of the queue's doorbell in the device's
+ *				doorbell window.
+ */
+struct ne_vq_wiring {
+	u16	msix_vector;
+	u16	notify_index;
+	u16	msix_entry;
+	u16	doorbell_offset;
+};
+
+static_assert(sizeof(struct ne_vq_wiring) == 8);
+
+/**
+ * struct slot_add_device_req - SLOT_ADD_DEVICE request.
+ * @slot_uid:			Slot unique id mapped to the slot to give the
+ *				device to.
+ * @device_type:		Type of device to give the slot.
+ * @num_vqs:			Number of virtqueues the device has, at most
+ *				NE_MAX_VQS_PER_DEVICE.
+ * @device_features:		Feature bits the parent offers the enclave for
+ *				the device.
+ * @config_size:		Number of bytes of @device_config the device
+ *				reads.
+ * @num_backing_ranges:		Number of entries of @backing_ranges the
+ *				device reads, at most NE_MAX_BACKING_RANGES.
+ * @device_config:		Type-specific configuration of the device.
+ * @backing_ranges:		Runs of enclave memory the device works out of.
+ */
+struct slot_add_device_req {
+	u64	slot_uid;
+	u32	device_type;
+	u32	num_vqs;
+	u8	device_features[64];
+	u32	config_size;
+	u32	num_backing_ranges;
+	u8	device_config[64];
+	struct ne_backing_range backing_ranges[NE_MAX_BACKING_RANGES];
+} __packed;
+static_assert(sizeof(struct slot_add_device_req) <= NE_SEND_DATA_SIZE);
+
 /**
  * struct slot_pool_donate_req - SLOT_POOL_DONATE and SLOT_POOL_BASE_DONATE
  *				 request.
@@ -408,6 +477,15 @@ struct slot_pool_mem_donate_req {
  *				slot out of.
  * @limits.reserved14:		SLOT_POOL_GET_LIMITS. Bytes the device writes as
  *				zero.
+ * @add_device:			Fields SLOT_ADD_DEVICE fills.
+ * @add_device.device_uid:	SLOT_ADD_DEVICE. Unique id of the new device.
+ * @add_device.num_vqs:		SLOT_ADD_DEVICE. Number of entries of wiring
+ *				the device wrote to the info page.
+ * @add_device.reserved12:	SLOT_ADD_DEVICE. Bytes owned by other
+ *				commands.
+ * @add_device.vq_wiring_offset:	SLOT_ADD_DEVICE. Offset into the info
+ *				page at which @num_vqs entries of struct
+ *				ne_vq_wiring start.
  */
 struct ne_pci_dev_cmd_reply {
 	s32	rc;
@@ -443,6 +521,12 @@ struct ne_pci_dev_cmd_reply {
 			u16	slots_per_donation;
 			u16	reserved14;
 		} limits;
+		struct {
+			u64	device_uid;
+			u32	num_vqs;
+			u32	reserved12;
+			u64	vq_wiring_offset;
+		} add_device;
 	};
 };
 
@@ -452,6 +536,10 @@ static_assert(offsetof(struct ne_pci_dev_cmd_reply, start.enclave_cid) == 16);
 static_assert(offsetof(struct ne_pci_dev_cmd_reply, alloc.slot_uid) == 8);
 static_assert(offsetof(struct ne_pci_dev_cmd_reply, alloc.mem_regions) == 32);
 static_assert(offsetof(struct ne_pci_dev_cmd_reply, info.slot_uid) == 8);
+static_assert(offsetof(struct ne_pci_dev_cmd_reply, add_device.device_uid) == 8);
+static_assert(offsetof(struct ne_pci_dev_cmd_reply, add_device.num_vqs) == 16);
+static_assert(offsetof(struct ne_pci_dev_cmd_reply,
+		       add_device.vq_wiring_offset) == 24);
 static_assert(offsetof(struct ne_pci_dev_cmd_reply, info.enclave_cid) == 16);
 static_assert(offsetof(struct ne_pci_dev_cmd_reply, info.mem_regions) == 32);
 static_assert(offsetof(struct ne_pci_dev_cmd_reply, info.mem_size) == 40);
-- 
2.47.1

[PATCH 24/40] nitro_enclaves: Add NE_ADD_DEVICE

[Alexander Graf]

Nitro Sandboxes allow the parent to back arbitrary virtio devices:
virtio-blk, virtio-net, virtio-console, etc. Their backend runs in the
user space process that drives the nitro_enclaves device.

Add the NE_ADD_DEVICE ioctl to enable user space to drive device addition.
It takes a device type, the offered feature bits, a configuration blob,
and struct ne_shm_region entries that name the NE_ALLOC_MEMORY allocations
the device works out of and the id the sandbox knows each one by.

When the hypervisor successfully added the device to the slot, it adds
the full device info, including VQ mapping, into the info page.

Signed-off-by: Alexander Graf <graf@amazon.com>
---
 Documentation/virt/ne_overview.rst        |   6 +-
 drivers/virt/nitro_enclaves/ne_misc_dev.c | 463 +++++++++++++++++++++-
 drivers/virt/nitro_enclaves/ne_misc_dev.h |  19 +-
 drivers/virt/nitro_enclaves/ne_pci_dev.h  |  16 +-
 include/uapi/linux/nitro_enclaves.h       | 195 ++++++++-
 5 files changed, 671 insertions(+), 28 deletions(-)

diff --git a/Documentation/virt/ne_overview.rst b/Documentation/virt/ne_overview.rst
index ba030cde59989..012fcb394a2ef 100644
--- a/Documentation/virt/ne_overview.rst
+++ b/Documentation/virt/ne_overview.rst
@@ -212,9 +212,9 @@ NE_SET_USER_MEMORY_REGION is unchanged and still takes the address of a
 hugetlbfs region.
 
 An allocation lives until the last of the things holding it is gone: its
-descriptor, its mappings, the regions given from it and the enclave it is the
-info page of. So neither closing the descriptor nor unmapping takes memory away
-from a running enclave.
+descriptor, its mappings, the regions given from it, the enclave it is the info
+page of and the enclave it is a device's memory of. So neither closing the
+descriptor nor unmapping takes memory away from a running enclave.
 
 Slot pool
 ---------
diff --git a/drivers/virt/nitro_enclaves/ne_misc_dev.c b/drivers/virt/nitro_enclaves/ne_misc_dev.c
index 208c1997609d3..7698c7210eb86 100644
--- a/drivers/virt/nitro_enclaves/ne_misc_dev.c
+++ b/drivers/virt/nitro_enclaves/ne_misc_dev.c
@@ -125,6 +125,8 @@ static int ne_alloc_memory_ioctl(struct ne_enclave *ne_enclave,
 				 struct ne_alloc_memory_args *args);
 static int ne_set_info_page_ioctl(struct ne_enclave *ne_enclave,
 				  struct ne_set_info_page_args *args);
+static int ne_add_device_ioctl(struct ne_enclave *ne_enclave,
+			       struct ne_add_device_args *args);
 static const struct file_operations ne_fops = {
 	.owner		= THIS_MODULE,
 	.llseek		= noop_llseek,
@@ -2189,6 +2191,45 @@ static long ne_enclave_do_ioctl(struct file *file, unsigned int cmd, unsigned lo
 		return rc;
 	}
 
+	case NE_EA_IOCTL(NE_ADD_DEVICE): {
+		struct ne_add_device_args __user *uargs = (void __user *)arg;
+		unsigned int usize = _IOC_SIZE(cmd);
+		struct ne_add_device_args args;
+		int rc;
+
+		if (usize < offsetofend(struct ne_add_device_args, vq_info))
+			return -EINVAL;
+
+		rc = copy_struct_from_user(&args, sizeof(args), uargs, usize);
+		if (rc)
+			return rc;
+
+		rc = mutex_lock_killable(&ne_enclave->enclave_info_mutex);
+		if (rc)
+			return rc;
+
+		if (ne_enclave->state != NE_STATE_INIT) {
+			dev_err_ratelimited(ne_misc_dev.this_device,
+					    "Enclave is not in init state\n");
+
+			mutex_unlock(&ne_enclave->enclave_info_mutex);
+
+			return -NE_ERR_NOT_IN_INIT_STATE;
+		}
+
+		rc = ne_add_device_ioctl(ne_enclave, &args);
+
+		mutex_unlock(&ne_enclave->enclave_info_mutex);
+
+		if (rc < 0)
+			return rc;
+
+		if (copy_to_user(uargs, &args, min_t(size_t, usize, sizeof(args))))
+			return -EFAULT;
+
+		return 0;
+	}
+
 	default:
 		return -ENOTTY;
 	}
@@ -2228,8 +2269,8 @@ static long ne_enclave_ioctl(struct file *file, unsigned int cmd, unsigned long
 
 /**
  * ne_enclave_remove_all_contig_regions() - Drop the enclave's references to the
- *					    allocations it was given.
- * @ne_enclave :	Private data associated with the current enclave.
+ *					    allocations on one of its lists.
+ * @regions :	One of the enclave's lists of held allocations.
  *
  * The last reference on an allocation's file frees its runs and gives back its
  * %RLIMIT_MEMLOCK charge, so an allocation whose descriptor and mappings are
@@ -2238,13 +2279,12 @@ static long ne_enclave_ioctl(struct file *file, unsigned int cmd, unsigned long
  * Context: Process context. This function is called on an enclave that is on
  *	    no list and has no file left.
  */
-static void ne_enclave_remove_all_contig_regions(struct ne_enclave *ne_enclave)
+static void ne_enclave_remove_all_contig_regions(struct list_head *regions)
 {
 	struct ne_contig_region *ne_contig_region = NULL;
 	struct ne_contig_region *tmp = NULL;
 
-	list_for_each_entry_safe(ne_contig_region, tmp,
-				 &ne_enclave->contig_regions_list, list) {
+	list_for_each_entry_safe(ne_contig_region, tmp, regions, list) {
 		list_del(&ne_contig_region->list);
 
 		fput(ne_contig_region->file);
@@ -2397,7 +2437,8 @@ int ne_enclave_free_slot(struct ne_pci_dev *ne_pci_dev,
  */
 static void ne_enclave_free(struct ne_enclave *ne_enclave)
 {
-	ne_enclave_remove_all_contig_regions(ne_enclave);
+	ne_enclave_remove_all_contig_regions(&ne_enclave->contig_regions_list);
+	ne_enclave_remove_all_contig_regions(&ne_enclave->device_regions_list);
 	ne_enclave_remove_all_mem_region_entries(ne_enclave);
 	ne_enclave_remove_all_vcpu_id_entries(ne_enclave);
 
@@ -2648,6 +2689,7 @@ static int ne_create_vm_ioctl(struct ne_pci_dev *ne_pci_dev, u64 flags,
 	mutex_init(&ne_enclave->enclave_info_mutex);
 	ne_enclave->max_mem_regions = cmd_reply.alloc.mem_regions;
 	INIT_LIST_HEAD(&ne_enclave->contig_regions_list);
+	INIT_LIST_HEAD(&ne_enclave->device_regions_list);
 	INIT_LIST_HEAD(&ne_enclave->mem_regions_list);
 	ne_enclave->mm = current->mm;
 	ne_enclave->slot_uid = cmd_reply.alloc.slot_uid;
@@ -3751,6 +3793,15 @@ static int ne_add_memory_region_ioctl(struct ne_enclave *ne_enclave,
 		return -NE_ERR_MEM_REGION_ALREADY_USED;
 	}
 
+	list_for_each_entry(held, &ne_enclave->device_regions_list, list) {
+		if (held->file == fd_file(f)) {
+			dev_err_ratelimited(ne_misc_dev.this_device,
+					    "The allocation is a device's\n");
+
+			return -NE_ERR_MEM_REGION_ALREADY_USED;
+		}
+	}
+
 	list_for_each_entry(held, &ne_enclave->contig_regions_list, list) {
 		if (held->file == fd_file(f)) {
 			dev_err_ratelimited(ne_misc_dev.this_device,
@@ -3885,6 +3936,7 @@ static int ne_set_info_page_ioctl(struct ne_enclave *ne_enclave,
 	struct ne_pci_dev_cmd_reply cmd_reply = {};
 	struct slot_add_info_page_req req = {};
 	struct pci_dev *pdev = ne_devs.ne_pci_dev->pdev;
+	struct ne_contig_region *held = NULL;
 	struct ne_contig_mem *mem = NULL;
 	int rc = -EINVAL;
 
@@ -3961,6 +4013,15 @@ static int ne_set_info_page_ioctl(struct ne_enclave *ne_enclave,
 		return -NE_ERR_MEM_REGION_ALREADY_USED;
 	}
 
+	list_for_each_entry(held, &ne_enclave->device_regions_list, list) {
+		if (held->file == fd_file(f)) {
+			dev_err_ratelimited(ne_misc_dev.this_device,
+					    "The info page allocation is a device's\n");
+
+			return -NE_ERR_MEM_REGION_ALREADY_USED;
+		}
+	}
+
 	rc = ne_contig_make_read_only(fd_file(f));
 	if (rc < 0) {
 		dev_err_ratelimited(ne_misc_dev.this_device,
@@ -4007,6 +4068,396 @@ static int ne_set_info_page_ioctl(struct ne_enclave *ne_enclave,
 	return 0;
 }
 
+/**
+ * ne_add_device_region() - Take one of a device's shared memory regions.
+ * @ne_enclave:	Private data associated with the current enclave.
+ * @region:	Descriptor for the allocation the region is made of, and the id
+ *		the enclave sees the region as.
+ * @req:	Request being built. The runs of the allocation are appended to
+ *		its backing ranges, each tagged with the region's id.
+ * @regions:	List the reference taken on the allocation is recorded on, so
+ *		that a later region failing gives every earlier one back.
+ *
+ * Context: Process context. This function is called with the ne_enclave mutex held.
+ * Return:
+ * * 0 on success.
+ * * Negative return value on failure.
+ */
+static int ne_add_device_region(struct ne_enclave *ne_enclave,
+				const struct ne_shm_region *region,
+				struct slot_add_device_req *req,
+				struct list_head *regions)
+{
+	struct ne_contig_region *ne_contig_region = NULL;
+	struct ne_contig_region *held = NULL;
+	struct ne_contig_mem *mem = NULL;
+	struct file *file = NULL;
+	unsigned int i = 0;
+	int rc = -EINVAL;
+
+	if (region->reserved[0] || region->reserved[1] || region->reserved[2])
+		return -EINVAL;
+
+	file = fget(region->fd);
+	if (!file)
+		return -EBADF;
+
+	if (file->f_op != &ne_contig_fops) {
+		dev_err_ratelimited(ne_misc_dev.this_device,
+				    "Not an enclave memory allocation\n");
+
+		goto put_file;
+	}
+
+	mem = file->private_data;
+
+	if (mem->enclave_id != ne_enclave->id) {
+		dev_err_ratelimited(ne_misc_dev.this_device,
+				    "Allocation was made for another enclave\n");
+
+		goto put_file;
+	}
+
+	/*
+	 * The parent drives its end of the device's queues through a mapping of
+	 * the allocation, and a private allocation cannot be mapped.
+	 */
+	if (!mem->shared) {
+		dev_err_ratelimited(ne_misc_dev.this_device,
+				    "The device allocation is not shared\n");
+
+		goto put_file;
+	}
+
+	if (mem->donated) {
+		dev_err_ratelimited(ne_misc_dev.this_device,
+				    "The device allocation was donated\n");
+
+		rc = -NE_ERR_MEM_REGION_ALREADY_USED;
+
+		goto put_file;
+	}
+
+	if (file == ne_enclave->info_file) {
+		dev_err_ratelimited(ne_misc_dev.this_device,
+				    "The allocation is the enclave's info page\n");
+
+		rc = -NE_ERR_MEM_REGION_ALREADY_USED;
+
+		goto put_file;
+	}
+
+	list_for_each_entry(held, &ne_enclave->device_regions_list, list) {
+		if (held->file == file) {
+			dev_err_ratelimited(ne_misc_dev.this_device,
+					    "The allocation is already a device's\n");
+
+			rc = -NE_ERR_MEM_REGION_ALREADY_USED;
+
+			goto put_file;
+		}
+	}
+
+	list_for_each_entry(held, regions, list) {
+		if (held->file == file) {
+			dev_err_ratelimited(ne_misc_dev.this_device,
+					    "Two regions of the device name one allocation\n");
+
+			rc = -NE_ERR_MEM_REGION_ALREADY_USED;
+
+			goto put_file;
+		}
+	}
+
+	/*
+	 * The device takes one address and one length per run of every region,
+	 * and the request has room for NE_MAX_BACKING_RANGES of them.
+	 */
+	if (mem->nr_allocs > NE_MAX_BACKING_RANGES - req->num_backing_ranges) {
+		dev_err_ratelimited(ne_misc_dev.this_device,
+				    "The device regions are more than %d runs\n",
+				    NE_MAX_BACKING_RANGES);
+
+		goto put_file;
+	}
+
+	/*
+	 * A run's length shares its word with the id of the region the run
+	 * belongs to, so a length that reached into the id would arrive at the
+	 * device as another region's memory rather than as a wrong length.
+	 */
+	for (i = 0; i < mem->nr_allocs; i++) {
+		u64 run = (u64)mem->allocs[i].nr_pages << PAGE_SHIFT;
+
+		if (run > NE_RANGE_SIZE_MASK) {
+			dev_err_ratelimited(ne_misc_dev.this_device,
+					    "A device region run is longer than %llu bytes\n",
+					    NE_RANGE_SIZE_MASK);
+
+			goto put_file;
+		}
+	}
+
+	ne_contig_region = kzalloc_obj(*ne_contig_region);
+	if (!ne_contig_region) {
+		rc = -ENOMEM;
+
+		goto put_file;
+	}
+
+	for (i = 0; i < mem->nr_allocs; i++) {
+		struct ne_backing_range *range =
+			&req->backing_ranges[req->num_backing_ranges++];
+
+		range->phys_addr = page_to_phys(mem->allocs[i].page);
+		range->size = ((u64)mem->allocs[i].nr_pages << PAGE_SHIFT) |
+			      ((u64)region->shm_id << NE_RANGE_SHMID_SHIFT);
+	}
+
+	ne_contig_region->file = file;
+	list_add(&ne_contig_region->list, regions);
+
+	return 0;
+
+put_file:
+	fput(file);
+
+	return rc;
+}
+
+/**
+ * ne_add_device_ioctl() - Give the enclave a device it drives itself.
+ * @ne_enclave:	Private data associated with the current enclave.
+ * @args:	Descriptors for the allocations the device works from, what the
+ *		device is, and where the wiring of its queues is returned.
+ *
+ * The device works out of the runs of the allocations, so they have to stay where
+ * the device was told they are for as long as the enclave lives, and the
+ * references this takes on the files keep them whatever user space does with the
+ * descriptors and their mappings. ne_enclave_free() drops those references
+ * after SLOT_FREE. The allocations are not donated, so the parent keeps writing
+ * to them and reading them back to drive its end of the device's queues.
+ *
+ * The device writes the wiring of the queues to the info page and the reply says
+ * where, so this reads it from there rather than out of the reply, and returns
+ * it to the caller.
+ *
+ * Context: Process context. This function is called with the ne_enclave mutex held.
+ * Return:
+ * * 0 on success.
+ * * Negative return value on failure.
+ */
+static int ne_add_device_ioctl(struct ne_enclave *ne_enclave,
+			       struct ne_add_device_args *args)
+{
+	struct ne_pci_dev_cmd_reply cmd_reply = {};
+	struct pci_dev *pdev = ne_devs.ne_pci_dev->pdev;
+	struct ne_shm_region *shm_regions = NULL;
+	const struct ne_vq_wiring *wiring = NULL;
+	struct slot_add_device_req *req = NULL;
+	struct ne_contig_mem *info = NULL;
+	LIST_HEAD(regions);
+	u8 buf[NE_MAX_VQS * sizeof(struct ne_vq_wiring)];
+	u64 wiring_off = 0;
+	u64 wiring_len = 0;
+	u64 copied = 0;
+	u64 n = 0;
+	unsigned int i = 0;
+	int rc = -EINVAL;
+
+	if (ne_enclave->mm != current->mm)
+		return -EIO;
+
+	if (args->flags >= NE_ADD_DEVICE_MAX_FLAG_VAL) {
+		dev_err_ratelimited(ne_misc_dev.this_device,
+				    "Incorrect flag for adding a device\n");
+
+		return -NE_ERR_INVALID_FLAG_VALUE;
+	}
+
+	if (args->padding)
+		return -EINVAL;
+
+	if (args->num_vqs > NE_MAX_VQS) {
+		dev_err_ratelimited(ne_misc_dev.this_device,
+				    "The device has more than %d queues\n",
+				    NE_MAX_VQS);
+
+		return -EINVAL;
+	}
+
+	if (args->config_size > sizeof(args->device_config)) {
+		dev_err_ratelimited(ne_misc_dev.this_device,
+				    "The device configuration is longer than %zu bytes\n",
+				    sizeof(args->device_config));
+
+		return -EINVAL;
+	}
+
+	/*
+	 * The device reports the wiring of the queues by writing the info page,
+	 * so a slot without one has nowhere for it to land.
+	 */
+	if (!ne_enclave->info_file) {
+		dev_err_ratelimited(ne_misc_dev.this_device,
+				    "The slot has no info page\n");
+
+		return -EINVAL;
+	}
+
+	if (args->flags & NE_DEVICE_NO_SHM_REGIONS) {
+		if (args->num_shm_regions || args->shm_regions_ptr) {
+			dev_err_ratelimited(ne_misc_dev.this_device,
+					    "The device names memory it works out of anyway\n");
+
+			return -EINVAL;
+		}
+	} else if (!args->num_shm_regions) {
+		dev_err_ratelimited(ne_misc_dev.this_device,
+				    "The device names no memory to work out of\n");
+
+		return -EINVAL;
+	}
+
+	/*
+	 * Every region is made of at least one run, so the number of runs the
+	 * device takes bounds how many regions it can have.
+	 */
+	if (args->num_shm_regions > NE_MAX_BACKING_RANGES) {
+		dev_err_ratelimited(ne_misc_dev.this_device,
+				    "The device has more than %d regions\n",
+				    NE_MAX_BACKING_RANGES);
+
+		return -EINVAL;
+	}
+
+	if (args->num_shm_regions) {
+		shm_regions = memdup_array_user(u64_to_user_ptr(args->shm_regions_ptr),
+						args->num_shm_regions,
+						sizeof(*shm_regions));
+		if (IS_ERR(shm_regions))
+			return PTR_ERR(shm_regions);
+	}
+
+	req = kzalloc_obj(*req);
+	if (!req) {
+		rc = -ENOMEM;
+
+		goto free;
+	}
+
+	for (i = 0; i < args->num_shm_regions; i++) {
+		rc = ne_add_device_region(ne_enclave, &shm_regions[i], req,
+					  &regions);
+		if (rc < 0)
+			goto free;
+	}
+
+	req->slot_uid = ne_enclave->slot_uid;
+	req->device_type = args->device_type;
+	req->num_vqs = args->num_vqs;
+	req->config_size = args->config_size;
+
+	memcpy(req->device_features, args->device_features,
+	       sizeof(req->device_features));
+	memcpy(req->device_config, args->device_config, args->config_size);
+
+	rc = ne_do_request(pdev, SLOT_ADD_DEVICE, req, sizeof(*req),
+			   &cmd_reply, sizeof(cmd_reply));
+	if (rc < 0) {
+		dev_err_ratelimited(ne_misc_dev.this_device,
+				    "Error in slot add device [rc=%d]\n", rc);
+
+		/*
+		 * A negative return does not say whether the device took the
+		 * runs. The command may have been turned down, or it may never
+		 * have been answered and still be in flight, and one request
+		 * carries the runs of every region, so a device that turned it
+		 * down may have taken some of them first. The enclave holds
+		 * them either way and ne_enclave_free() gives them back after
+		 * SLOT_FREE, as the two SLOT_ADD_MEM sites do.
+		 */
+		list_splice_init(&regions, &ne_enclave->device_regions_list);
+
+		goto free;
+	}
+
+	/*
+	 * The device works out of the runs from here on, so the enclave holds
+	 * the allocations even if this call goes on to fail, and
+	 * ne_enclave_free() gives them back after SLOT_FREE.
+	 */
+	list_splice_init(&regions, &ne_enclave->device_regions_list);
+
+	if (cmd_reply.add_device.num_vqs != args->num_vqs) {
+		dev_err_ratelimited(ne_misc_dev.this_device,
+				    "Device wired %u of the %u queues asked for\n",
+				    cmd_reply.add_device.num_vqs,
+				    args->num_vqs);
+
+		rc = -EPROTO;
+
+		goto free;
+	}
+
+	info = ne_enclave->info_file->private_data;
+	wiring_off = cmd_reply.add_device.vq_wiring_offset;
+	wiring_len = (u64)args->num_vqs * sizeof(struct ne_vq_wiring);
+
+	if (args->num_vqs && (wiring_off > info->size ||
+			      wiring_len > info->size - wiring_off)) {
+		dev_err_ratelimited(ne_misc_dev.this_device,
+				    "Device put the wiring of %u queues at %llu of the info page\n",
+				    args->num_vqs, wiring_off);
+
+		rc = -EPROTO;
+
+		goto free;
+	}
+
+	/*
+	 * Where the kernel may have highmem a run is not contiguous in the
+	 * direct map, so the entries are copied a page at a time rather than
+	 * read through one address, and the copy makes no assumption about how
+	 * the device aligned them.
+	 */
+	for (copied = 0; copied < wiring_len; copied += n) {
+		u64 pos = wiring_off + copied;
+
+		n = min_t(u64, wiring_len - copied,
+			  PAGE_SIZE - (pos & ~PAGE_MASK));
+		memcpy_from_page(buf + copied,
+				 info->allocs[0].page + (pos >> PAGE_SHIFT),
+				 pos & ~PAGE_MASK, n);
+	}
+
+	wiring = (const struct ne_vq_wiring *)buf;
+
+	for (i = 0; i < args->num_vqs; i++) {
+		args->vq_info[i].msix_vector = wiring[i].msix_vector;
+		args->vq_info[i].notify_index = wiring[i].notify_index;
+		args->vq_info[i].msix_entry = wiring[i].msix_entry;
+		args->vq_info[i].doorbell_offset = wiring[i].doorbell_offset;
+	}
+
+	args->device_uid = cmd_reply.add_device.device_uid;
+
+	rc = 0;
+
+free:
+	/*
+	 * Whatever is still on the local list belongs to a call that was refused
+	 * before the device was told about any of it, so those references go
+	 * back. A call the device was told about has already put its regions on
+	 * the enclave, leaving nothing here.
+	 */
+	ne_enclave_remove_all_contig_regions(&regions);
+	kfree(req);
+	kfree(shm_regions);
+
+	return rc;
+}
+
 #if defined(CONFIG_NITRO_ENCLAVES_MISC_DEV_TEST)
 #include "ne_misc_dev_test.c"
 #endif
diff --git a/drivers/virt/nitro_enclaves/ne_misc_dev.h b/drivers/virt/nitro_enclaves/ne_misc_dev.h
index 59374a35c3c7d..4f695f1410aec 100644
--- a/drivers/virt/nitro_enclaves/ne_misc_dev.h
+++ b/drivers/virt/nitro_enclaves/ne_misc_dev.h
@@ -39,8 +39,8 @@ struct ne_contig_alloc {
  * @nr_allocs:	Number of runs in @allocs.
  * @size:	Size of the allocation in bytes, mirrored in i_size.
  * @shared:	Whether the caller asked for a shared allocation. A shared one may
- *		be mapped by the caller and may be a slot's info page; a private
- *		one is neither.
+ *		be mapped by the caller and may be a slot's info page or the memory
+ *		of one of its devices; a private one is none of those.
  * @enclave_id:	Id of the enclave the allocation was made for.
  * @mm:		The mm charged for the pinned pages, held via mmgrab().
  * @nr_pinned:	Pages charged to @mm->pinned_vm.
@@ -52,7 +52,8 @@ struct ne_contig_alloc {
  *
  * The runs hang off the file, so that closing the last descriptor does not free
  * memory an enclave is still using: a donated region holds the file with
- * get_file(), and so does the enclave the allocation is the info page of. That
+ * get_file(), and so does the enclave the allocation is the info page or a
+ * device's memory of. That
  * is also what keeps a mapping's page table entries from outliving the runs,
  * since a VMA holds vm_file for as long as it exists.
  *
@@ -67,8 +68,8 @@ struct ne_contig_alloc {
  *
  * @enclave_id is compared, never dereferenced, and the counter it comes from is
  * 64 bits wide, so an allocation cannot be donated to, or become the info page
- * of, a later enclave that happens to have been placed at the address of the one
- * it was made for.
+ * or a device's memory of, a later enclave that happens to have been placed at
+ * the address of the one it was made for.
  */
 struct ne_contig_mem {
 	struct ne_contig_alloc	*allocs;
@@ -85,8 +86,8 @@ struct ne_contig_mem {
 };
 
 /**
- * struct ne_contig_region - Enclave memory given from one allocation.
- * @list:	Entry in the enclave's list of allocations given to it.
+ * struct ne_contig_region - One %NE_ALLOC_MEMORY allocation an enclave holds.
+ * @list:	Entry in one of the enclave's lists of held allocations.
  * @file:	File holding the runs, held with get_file() so that they outlive the
  *		descriptor the allocation was handed out on and its mappings.
  */
@@ -121,6 +122,9 @@ struct ne_mem_region {
  *				held for writing.
  * @contig_regions_list:	List of %NE_ALLOC_MEMORY allocations given to the
  *				enclave with %NE_ADD_MEMORY_REGION.
+ * @device_regions_list:	List of %NE_ALLOC_MEMORY allocations a device of
+ *				the enclave works from, given with
+ *				%NE_ADD_DEVICE.
  * @enclave_list_entry :	Entry in the list of created enclaves.
  * @eventq:			Wait queue used for out-of-band event notifications
  *				triggered from the PCI device event handler to
@@ -162,6 +166,7 @@ struct ne_enclave {
 	bool			detached;
 	bool			slot_freed;
 	struct list_head	contig_regions_list;
+	struct list_head	device_regions_list;
 	struct list_head	enclave_list_entry;
 	wait_queue_head_t	eventq;
 	bool			has_event;
diff --git a/drivers/virt/nitro_enclaves/ne_pci_dev.h b/drivers/virt/nitro_enclaves/ne_pci_dev.h
index 3445318b84672..a0bca5863e2a1 100644
--- a/drivers/virt/nitro_enclaves/ne_pci_dev.h
+++ b/drivers/virt/nitro_enclaves/ne_pci_dev.h
@@ -7,6 +7,7 @@
 #define _NE_PCI_DEV_H_
 
 #include <linux/atomic.h>
+#include <linux/bits.h>
 #include <linux/build_bug.h>
 #include <linux/list.h>
 #include <linux/mutex.h>
@@ -346,10 +347,23 @@ struct slot_add_info_page_req {
  */
 #define NE_MAX_BACKING_RANGES		(4)
 
+/**
+ * NE_RANGE_SIZE_MASK - The part of a backing range's size that is a byte count.
+ */
+#define NE_RANGE_SIZE_MASK		GENMASK_ULL(55, 0)
+
+/**
+ * NE_RANGE_SHMID_SHIFT - Where a backing range's size carries the id of the
+ *			  shared memory region the run belongs to.
+ */
+#define NE_RANGE_SHMID_SHIFT		(56)
+
 /**
  * struct ne_backing_range - One run of enclave memory a device works out of.
  * @phys_addr:	Physical address of the run, in the parent's address space.
- * @size:	Size, in bytes, of the run.
+ * @size:	Size, in bytes, of the run in the low %NE_RANGE_SIZE_MASK bits,
+ *		and the id of the shared memory region the run belongs to in the
+ *		8 bits above them. A device tells its regions apart by that id.
  */
 struct ne_backing_range {
 	u64	phys_addr;
diff --git a/include/uapi/linux/nitro_enclaves.h b/include/uapi/linux/nitro_enclaves.h
index 4f604712983f1..6f83a8b0762b3 100644
--- a/include/uapi/linux/nitro_enclaves.h
+++ b/include/uapi/linux/nitro_enclaves.h
@@ -317,11 +317,11 @@ struct ne_alloc_memory_args {
  *		     Donating does not take a mapping away, so a parent that
  *		     mapped a shared allocation can still write to it through
  *		     that mapping afterwards.
- *		     The allocation lives until the last of the descriptor, its
- *		     mappings, the enclave regions donated from it and the
- *		     enclave it is the info page of is gone, so a caller can
- *		     close the descriptor as soon as it has donated it or
- *		     provided it to the device.
+ *		     The allocation lives until the descriptor, its mappings,
+ *		     the enclave regions donated from it and the enclave that
+ *		     uses it as an info page or as device memory are all gone,
+ *		     so a caller can close the descriptor as soon as it has
+ *		     handed it over.
  *		     The ioctl can be invoked on the enclave fd, before an
  *		     enclave is started.
  *
@@ -396,8 +396,8 @@ struct ne_add_memory_region_args {
  *			  contiguous run it is made of, or fewer where runs turn
  *			  out to be physically adjacent. The argument is input
  *			  only. The allocation has to have been made for this
- *			  enclave, cannot be its info page, and can be given away
- *			  once.
+ *			  enclave, cannot be its info page or the memory of one
+ *			  of its devices, and can be given away once.
  *			  Writing to the descriptor stops working here, so an
  *			  enclave image goes in before this call. The allocation
  *			  has to be %NE_MEMORY_PRIVATE, since memory the parent
@@ -434,7 +434,8 @@ struct ne_add_memory_region_args {
  *					  before being started).
  * * NE_ERR_INVALID_FLAG_VALUE		- The value of the provided flag is invalid.
  * * NE_ERR_MEM_REGION_ALREADY_USED	- The allocation was already given to the
- *					  enclave, or is its info page.
+ *					  enclave, or is its info page, or is the
+ *					  memory of one of its devices.
  * * NE_ERR_MEM_MAX_REGIONS		- The number of memory regions set for the
  *					  enclave reached maximum.
  * * Error codes from the NE PCI device request.
@@ -470,8 +471,9 @@ struct ne_set_info_page_args {
  *		      made writable, so the call fails while one exists, and a
  *		      caller maps the page after the call. A call the device
  *		      rejects leaves the allocation writable again.
- *		      A slot takes one info page, and an allocation is either an
- *		      enclave's memory or its info page. The driver holds the
+ *		      A slot takes one info page, and an allocation is an
+ *		      enclave's memory, its info page or the memory of one of its
+ *		      devices, never two of those. The driver holds the
  *		      allocation until the enclave is gone, so closing the
  *		      descriptor does not free the run the device was given.
  *		      The ioctl can be invoked on the enclave fd, before an
@@ -501,7 +503,8 @@ struct ne_set_info_page_args {
  *					  before being started).
  * * NE_ERR_INVALID_FLAG_VALUE		- The value of the provided flag is invalid.
  * * NE_ERR_MEM_REGION_ALREADY_USED	- The allocation was already given to the
- *					  enclave with %NE_ADD_MEMORY_REGION.
+ *					  enclave with %NE_ADD_MEMORY_REGION, or is
+ *					  the memory of one of its devices.
  * * ETIMEDOUT				- The device did not answer. The slot keeps
  *					  the allocation, read-only, until the
  *					  enclave is gone, since the device may
@@ -510,6 +513,176 @@ struct ne_set_info_page_args {
  */
 #define NE_SET_INFO_PAGE		_IOW(0xAE, 0x25, struct ne_set_info_page_args)
 
+/**
+ * NE_MAX_VQS - Highest number of virtqueues one device may have.
+ */
+#define NE_MAX_VQS			(64)
+
+/**
+ * struct ne_vq_info - How the enclave reaches one virtqueue of a device.
+ * @msix_vector:	Interrupt vector the device raises for the queue.
+ * @notify_index:	Index the enclave writes to notify the device.
+ * @msix_entry:		Entry in the enclave's interrupt table holding
+ *			@msix_vector.
+ * @doorbell_offset:	Offset of the queue's doorbell in the device's doorbell
+ *			window.
+ */
+struct ne_vq_info {
+	__u16	msix_vector;
+	__u16	notify_index;
+	__u16	msix_entry;
+	__u16	doorbell_offset;
+};
+
+/**
+ * NE_DEVICE_NO_SHM_REGIONS - The call names no shared memory region, so
+ *			      @num_shm_regions and @shm_regions_ptr both have
+ *			      to be zero. A device that works out of the
+ *			      enclave's own %NE_ADD_MEMORY_REGION memory is
+ *			      what it is for.
+ */
+#define NE_DEVICE_NO_SHM_REGIONS	(0x01)
+
+#define NE_ADD_DEVICE_MAX_FLAG_VAL	(0x02)
+
+/**
+ * NE_DEVICE_TYPE_VIRTIO - A virtio device. The low 16 bits of @device_type
+ *			   carry the virtio device id the enclave sees.
+ */
+#define NE_DEVICE_TYPE_VIRTIO		(0u << 16)
+
+/**
+ * struct ne_shm_region - One shared memory region of a device (in).
+ * @fd:		Descriptor from %NE_ALLOC_MEMORY, made for this enclave, asked
+ *		for as %NE_MEMORY_SHARED, and named by one region only.
+ * @shm_id:	Shared memory region id the enclave sees the region as. The
+ *		driver puts it in the top 8 bits of the size it gives the device
+ *		for every run of the allocation, so that the enclave and the
+ *		device can tell one of a device's regions from another. The
+ *		parent and the enclave agree on the numbering; the driver does
+ *		not read it.
+ * @reserved:	Must be zero.
+ */
+struct ne_shm_region {
+	__s32	fd;
+	__u8	shm_id;
+	__u8	reserved[3];
+};
+
+/**
+ * struct ne_add_device_args - Device to give an enclave (in / out).
+ * @flags:		%NE_DEVICE_NO_SHM_REGIONS or zero.
+ * @device_type:	Type of device to give the enclave. The top 16 bits name
+ *			the class of device, of which %NE_DEVICE_TYPE_VIRTIO is
+ *			the only one, and the class says what the low 16 bits
+ *			carry.
+ * @num_vqs:		Number of virtqueues the device has, at most
+ *			%NE_MAX_VQS.
+ * @config_size:	Number of bytes of @device_config the device reads.
+ * @num_shm_regions:	Number of entries of @shm_regions_ptr, at least one
+ *			unless %NE_DEVICE_NO_SHM_REGIONS is set, and made
+ *			of no more than four runs in total.
+ * @padding:		Must be zero.
+ * @device_features:	Feature bits the parent offers the enclave for the
+ *			device.
+ * @device_config:	Type-specific configuration of the device.
+ * @shm_regions_ptr:	Address of @num_shm_regions struct ne_shm_region.
+ * @device_uid:		Id the device gave the new device (out).
+ * @vq_info:		How the enclave reaches the device's first @num_vqs
+ *			virtqueues (out).
+ */
+struct ne_add_device_args {
+	__u32	flags;
+	__u32	device_type;
+	__u32	num_vqs;
+	__u32	config_size;
+	__u32	num_shm_regions;
+	__u32	padding;
+	__u8	device_features[64];
+	__u8	device_config[64];
+	__u64	shm_regions_ptr;
+	__u64	device_uid;
+	struct ne_vq_info vq_info[NE_MAX_VQS];
+};
+
+/**
+ * NE_ADD_DEVICE - The command is used to give an enclave a device it drives
+ *		   itself, out of the %NE_ALLOC_MEMORY allocations the device
+ *		   works from. One allocation is one shared memory region of the
+ *		   device, and the region's id goes to the enclave with it, so
+ *		   that a device made of several of them tells them apart. Every
+ *		   allocation has to be %NE_MEMORY_SHARED, since the parent
+ *		   drives the other end of the device's virtqueues through a
+ *		   mapping of one, and none of them is donated, so writing to
+ *		   the descriptors keeps working. They are not enclave memory
+ *		   either: they do not count towards the memory regions
+ *		   %NE_ADD_MEMORY_REGION fills, and an allocation is a device's
+ *		   memory, an enclave's memory or its info page, never two of
+ *		   those. The driver holds the allocations until the enclave is
+ *		   gone, so closing the descriptors does not free the runs the
+ *		   device was given.
+ *		   The device is given one address and one length per
+ *		   contiguous run of every region, and it takes four of
+ *		   them in total, so regions that a fragmented node splits
+ *		   into more runs than that are refused. On a system with
+ *		   that much fragmentation, you wouldn't get enough
+ *		   contiguous memory for the enclave itself either.
+ *		   A device that works out of the enclave's own memory
+ *		   regions says so with %NE_DEVICE_NO_SHM_REGIONS and names
+ *		   no region.
+ *		   The slot needs its info page first: the device reports how
+ *		   the enclave reaches each virtqueue by writing the info page,
+ *		   and the reply says where in it those entries are. The driver
+ *		   reads them from there and returns them in @vq_info, so a
+ *		   caller learns the interrupt vector and the doorbell offset
+ *		   of every queue from this call.
+ *		   The ioctl can be invoked on the enclave fd, before an
+ *		   enclave is started.
+ *
+ * Context: Process context.
+ * Return:
+ * * 0					- Logic successfully completed.
+ * * -1					- There was a failure in the ioctl logic.
+ * On failure, errno is set to:
+ * * ENODEV				- The NE PCI device is not available.
+ * * EFAULT				- Argument read or write failure.
+ * * EBADF				- A region's @fd is not an open file
+ *					  descriptor.
+ * * EINVAL				- The argument is shorter than @vq_info, or
+ *					  @padding is not zero, or @num_vqs is
+ *					  more than %NE_MAX_VQS, or @config_size
+ *					  is more than @device_config holds, or
+ *					  the slot has no info page, or the
+ *					  argument names no region and does not
+ *					  set %NE_DEVICE_NO_SHM_REGIONS, or it
+ *					  sets that flag and names one anyway,
+ *					  or a region's @reserved is not zero,
+ *					  or its @fd is not a descriptor from
+ *					  %NE_ALLOC_MEMORY, or the allocation was
+ *					  made for another enclave, or it is
+ *					  %NE_MEMORY_PRIVATE, or the regions are
+ *					  made of more runs in total than the
+ *					  device takes.
+ * * EIO				- Current task mm is not the same as the one
+ *					  that created the enclave.
+ * * E2BIG				- The argument is longer than this kernel knows
+ *					  about and the excess is not all zero.
+ * * ENOMEM				- Memory allocation failure for internal
+ *					  bookkeeping variables.
+ * * EPROTO				- The device replied for a different number of
+ *					  virtqueues than it was asked for, or placed
+ *					  their entries outside the info page.
+ * * NE_ERR_NOT_IN_INIT_STATE		- The enclave is not in init state (init =
+ *					  before being started).
+ * * NE_ERR_INVALID_FLAG_VALUE		- The value of the provided flag is invalid.
+ * * NE_ERR_MEM_REGION_ALREADY_USED	- An allocation was already given to the
+ *					  enclave, or is its info page, or is the
+ *					  memory of one of its devices, or was
+ *					  named by two regions of this device.
+ * * Error codes from the NE PCI device request.
+ */
+#define NE_ADD_DEVICE			_IOWR(0xAE, 0x26, struct ne_add_device_args)
+
 /**
  * NE_START_ENCLAVE - The command is used to trigger enclave start after the
  *		      enclave resources, such as memory and CPU, have been set.
-- 
2.47.1

[PATCH 25/40] nitro_enclaves: Read the interrupt multiplexer geometry at probe

[Alexander Graf]

In preparation to support per-virtqueue events for Nitro Sandboxes, read
the geometry of the interrupt multiplexer at probe. The interrupt
multiplexer is built into the nitro_enclaves device. It is a pool of
MSI-X vectors that the device shares between the virtqueues of every
sandbox: it raises a pooled vector and records in a bitmap which queues
signalled. NE_MUX_NUM_VECTORS holds the size of the pool and
NE_MUX_VECTOR_BASE its first vector.

Read them only on a device with the PCI vendor-specific capability that
defines them. Leave the pool unused when it is larger than
NE_MUX_MAX_VECTORS or than the vectors of the device, or when it
overlaps the reply and event vectors.

Signed-off-by: Alexander Graf <graf@amazon.com>
---
 drivers/virt/nitro_enclaves/ne_pci_dev.c | 60 ++++++++++++++++++++++--
 drivers/virt/nitro_enclaves/ne_pci_dev.h | 27 ++++++++++-
 2 files changed, 81 insertions(+), 6 deletions(-)

diff --git a/drivers/virt/nitro_enclaves/ne_pci_dev.c b/drivers/virt/nitro_enclaves/ne_pci_dev.c
index 1aa4d5508dabf..05dd3040afbc3 100644
--- a/drivers/virt/nitro_enclaves/ne_pci_dev.c
+++ b/drivers/virt/nitro_enclaves/ne_pci_dev.c
@@ -550,6 +550,55 @@ static void ne_teardown_msix(struct pci_dev *pdev)
 	pci_free_irq_vectors(pdev);
 }
 
+/**
+ * ne_mux_setup() - Read the interrupt multiplexer geometry from the device.
+ * @pdev:	PCI device to read the multiplexer geometry of.
+ *
+ * The multiplexer registers only exist on a device that publishes a
+ * vendor-specific capability, so a device without one leaves the pool unused
+ * and reads no multiplexer register at all.
+ *
+ * Context: Process context.
+ * Return:
+ * * 0 on success.
+ */
+static int ne_mux_setup(struct pci_dev *pdev)
+{
+	struct ne_pci_dev *ne_pci_dev = pci_get_drvdata(pdev);
+	int msix_nr_vecs = 0;
+	u32 nr_vecs = 0;
+	u32 vec_base = 0;
+
+	if (!ne_pci_dev->has_vendor_cap)
+		return 0;
+
+	nr_vecs = ioread16(ne_pci_dev->iomem_base + NE_MUX_NUM_VECTORS);
+	if (!nr_vecs)
+		return 0;
+
+	vec_base = ioread16(ne_pci_dev->iomem_base + NE_MUX_VECTOR_BASE);
+	msix_nr_vecs = pci_msix_vec_count(pdev);
+
+	/*
+	 * An enclave whose queues cannot reach the pool binds them to the
+	 * reply and event vectors instead, which costs interrupt throughput
+	 * and no functionality. Refusing to probe would take Nitro Enclaves
+	 * away from the parent altogether.
+	 */
+	if (nr_vecs > NE_MUX_MAX_VECTORS || vec_base <= NE_VEC_EVENT ||
+	    msix_nr_vecs < 0 || vec_base + nr_vecs > (u32)msix_nr_vecs) {
+		dev_dbg(&pdev->dev, "Unusable multiplexer pool of %u vectors at vector %u\n",
+			nr_vecs, vec_base);
+
+		return 0;
+	}
+
+	ne_pci_dev->mux_nr_vecs = nr_vecs;
+	ne_pci_dev->mux_vec_base = vec_base;
+
+	return 0;
+}
+
 /**
  * ne_pci_dev_enable() - Select the PCI device version and enable it.
  * @pdev:	PCI device to select version for and then enable.
@@ -1574,10 +1623,11 @@ static int ne_pci_probe(struct pci_dev *pdev, const struct pci_device_id *id)
 	ne_pci_dev->pdev = pdev;
 
 	/*
-	 * The registers past the command interface only exist on a device that
-	 * publishes a vendor-specific capability, and a read of a register the
-	 * device does not implement has no defined value. The answer holds for
-	 * the life of the device, so look it up once here.
+	 * Launch flags, the multiplexer and the slot pool registers are
+	 * implemented by a device that publishes a vendor-specific capability,
+	 * and a read of a register the device does not implement has no defined
+	 * value. The answer holds
+	 * for the life of the device, so look it up once here.
 	 */
 	ne_pci_dev->has_vendor_cap =
 		pci_find_capability(pdev, PCI_CAP_ID_VNDR) != 0;
@@ -1598,6 +1648,8 @@ static int ne_pci_probe(struct pci_dev *pdev, const struct pci_device_id *id)
 		goto teardown_msix;
 	}
 
+	ne_mux_setup(pdev);
+
 	down_write(&ne_devs.lock);
 
 	/*
diff --git a/drivers/virt/nitro_enclaves/ne_pci_dev.h b/drivers/virt/nitro_enclaves/ne_pci_dev.h
index a0bca5863e2a1..aee22b4ed8b6e 100644
--- a/drivers/virt/nitro_enclaves/ne_pci_dev.h
+++ b/drivers/virt/nitro_enclaves/ne_pci_dev.h
@@ -54,6 +54,12 @@
  */
 #define NE_COMMAND		(0x0004)
 
+/**
+ * NE_MUX_VECTOR_BASE - (2 bytes) Register holding the first MSI-X vector of
+ *			the interrupt multiplexer pool (Read-Only).
+ */
+#define NE_MUX_VECTOR_BASE	(0x0008)
+
 /**
  * NE_EVTCNT - (4 bytes) Register to notify the driver that a reply or a device
  *	       event is available (Read-Only):
@@ -82,6 +88,12 @@
  */
 #define NE_RECV_DATA		(0x0100)
 
+/**
+ * NE_MUX_NUM_VECTORS - (2 bytes) Register holding the number of MSI-X vectors
+ *			in the interrupt multiplexer pool (Read-Only).
+ */
+#define NE_MUX_NUM_VECTORS	(0x01f0)
+
 /**
  * DOC: Device MMIO buffer sizes
  */
@@ -146,6 +158,11 @@
  */
 #define NE_VEC_EVENT		(1)
 
+/**
+ * NE_MUX_MAX_VECTORS - Largest interrupt multiplexer pool the driver accepts.
+ */
+#define NE_MUX_MAX_VECTORS	(256)
+
 /**
  * enum ne_pci_dev_cmd_type - Device command types.
  * @INVALID_CMD:		Invalid command.
@@ -604,9 +621,13 @@ struct ne_slot_pool_run {
  *				enclave state scanning and propagation to the
  *				enclave process.
  * @has_vendor_cap:		Whether the PCI device publishes a vendor-specific
- *				capability, and with it the registers past the
- *				command interface.
+ *				capability, and with it the multiplexer and slot
+ *				pool registers.
  * @iomem_base :		MMIO region of the PCI device.
+ * @mux_nr_vecs:		Number of MSI-X vectors in the interrupt
+ *				multiplexer pool.
+ * @mux_vec_base:		First MSI-X vector of the interrupt multiplexer
+ *				pool.
  * @notify_work:		Work item for every received out-of-band event.
  * @pci_dev_mutex:		Mutex for accessing the PCI device MMIO space.
  * @pdev:			PCI device data structure.
@@ -636,6 +657,8 @@ struct ne_pci_dev {
 	struct workqueue_struct	*event_wq;
 	bool			has_vendor_cap;
 	void __iomem		*iomem_base;
+	u32			mux_nr_vecs;
+	u32			mux_vec_base;
 	struct work_struct	notify_work;
 	struct mutex		pci_dev_mutex;
 	struct pci_dev		*pdev;
-- 
2.47.1

[PATCH 26/40] nitro_enclaves: Donate a bitmap page per multiplexed vector

[Alexander Graf]

In preparation to support the interrupt multiplexer, give the device the
bitmaps it records events in. At probe, donate one zeroed page per
pooled vector by writing its address to the matching NE_MUX_DONATE_BASE
slot, and read the outcome from NE_MUX_DONATE_ERR. A page holds
NE_MUX_MAX_BITS queue bits. The device sets the bit of a queue and then
raises the vector even when other bits are set already, which the
dispatch code relies on.

On teardown, write NE_MUX_SHUTDOWN before the pages go back to the
allocator.

Signed-off-by: Alexander Graf <graf@amazon.com>
---
 drivers/virt/nitro_enclaves/ne_pci_dev.c | 87 +++++++++++++++++++++++-
 drivers/virt/nitro_enclaves/ne_pci_dev.h | 50 ++++++++++++++
 2 files changed, 136 insertions(+), 1 deletion(-)

diff --git a/drivers/virt/nitro_enclaves/ne_pci_dev.c b/drivers/virt/nitro_enclaves/ne_pci_dev.c
index 05dd3040afbc3..0fafd42162f3a 100644
--- a/drivers/virt/nitro_enclaves/ne_pci_dev.c
+++ b/drivers/virt/nitro_enclaves/ne_pci_dev.c
@@ -11,6 +11,7 @@
 #include <linux/delay.h>
 #include <linux/device.h>
 #include <linux/err.h>
+#include <linux/io-64-nonatomic-lo-hi.h>
 #include <linux/iopoll.h>
 #include <linux/jiffies.h>
 #include <linux/kstrtox.h>
@@ -550,6 +551,32 @@ static void ne_teardown_msix(struct pci_dev *pdev)
 	pci_free_irq_vectors(pdev);
 }
 
+/**
+ * ne_mux_teardown() - Take the interrupt multiplexer bitmaps back from the
+ *		       device.
+ * @pdev:	PCI device to take the multiplexer bitmaps back from.
+ *
+ * Context: Process context.
+ */
+static void ne_mux_teardown(struct pci_dev *pdev)
+{
+	struct ne_pci_dev *ne_pci_dev = pci_get_drvdata(pdev);
+	u32 i = 0;
+
+	if (!ne_pci_dev->mux)
+		return;
+
+	iowrite32(NE_MUX_SHUTDOWN_ON, ne_pci_dev->iomem_base + NE_MUX_SHUTDOWN);
+
+	for (i = 0; i < ne_pci_dev->mux_nr_vecs; i++)
+		free_page((unsigned long)ne_pci_dev->mux[i].bitmap);
+
+	kfree(ne_pci_dev->mux);
+
+	ne_pci_dev->mux = NULL;
+	ne_pci_dev->mux_nr_vecs = 0;
+}
+
 /**
  * ne_mux_setup() - Read the interrupt multiplexer geometry from the device.
  * @pdev:	PCI device to read the multiplexer geometry of.
@@ -561,6 +588,7 @@ static void ne_teardown_msix(struct pci_dev *pdev)
  * Context: Process context.
  * Return:
  * * 0 on success.
+ * * Negative return value on failure.
  */
 static int ne_mux_setup(struct pci_dev *pdev)
 {
@@ -568,6 +596,8 @@ static int ne_mux_setup(struct pci_dev *pdev)
 	int msix_nr_vecs = 0;
 	u32 nr_vecs = 0;
 	u32 vec_base = 0;
+	int rc = -EINVAL;
+	u32 i = 0;
 
 	if (!ne_pci_dev->has_vendor_cap)
 		return 0;
@@ -593,10 +623,54 @@ static int ne_mux_setup(struct pci_dev *pdev)
 		return 0;
 	}
 
+	ne_pci_dev->mux = kcalloc(nr_vecs, sizeof(*ne_pci_dev->mux),
+				  GFP_KERNEL);
+	if (!ne_pci_dev->mux)
+		return -ENOMEM;
+
 	ne_pci_dev->mux_nr_vecs = nr_vecs;
 	ne_pci_dev->mux_vec_base = vec_base;
 
+	for (i = 0; i < nr_vecs; i++) {
+		u16 donate_err = 0;
+
+		ne_pci_dev->mux[i].bitmap = (u64 *)get_zeroed_page(GFP_KERNEL);
+		if (!ne_pci_dev->mux[i].bitmap) {
+			rc = -ENOMEM;
+
+			goto teardown_mux;
+		}
+
+		/*
+		 * On a kernel without a 64-bit MMIO accessor this write is two
+		 * 32-bit writes, low half first, so the device sees a complete
+		 * address only once the high half has arrived. The device is
+		 * expected to take the donation from the high half and not from
+		 * the low one, which is also what lets the outcome be read back
+		 * right after.
+		 */
+		writeq(virt_to_phys(ne_pci_dev->mux[i].bitmap),
+		       ne_pci_dev->iomem_base + NE_MUX_DONATE_BASE +
+		       i * sizeof(u64));
+
+		donate_err = ioread16(ne_pci_dev->iomem_base +
+				      NE_MUX_DONATE_ERR);
+		if (donate_err) {
+			dev_err(&pdev->dev, "Error in bitmap donate for vector %u [err=%u]\n",
+				i, donate_err);
+
+			rc = -EIO;
+
+			goto teardown_mux;
+		}
+	}
+
 	return 0;
+
+teardown_mux:
+	ne_mux_teardown(pdev);
+
+	return rc;
 }
 
 /**
@@ -1648,7 +1722,12 @@ static int ne_pci_probe(struct pci_dev *pdev, const struct pci_device_id *id)
 		goto teardown_msix;
 	}
 
-	ne_mux_setup(pdev);
+	rc = ne_mux_setup(pdev);
+	if (rc < 0) {
+		dev_err(&pdev->dev, "Error in mux setup [rc=%d]\n", rc);
+
+		goto pci_dev_disable;
+	}
 
 	down_write(&ne_devs.lock);
 
@@ -1717,6 +1796,8 @@ static int ne_pci_probe(struct pci_dev *pdev, const struct pci_device_id *id)
 	ne_devs.ne_pci_dev = NULL;
 unlock_ne_devs:
 	up_write(&ne_devs.lock);
+pci_dev_disable:
+	ne_mux_teardown(pdev);
 	ne_pci_dev_disable(pdev);
 teardown_msix:
 	ne_teardown_msix(pdev);
@@ -1755,6 +1836,8 @@ static void ne_pci_remove(struct pci_dev *pdev)
 
 	ne_devs.ne_pci_dev = NULL;
 
+	ne_mux_teardown(pdev);
+
 	ne_pci_dev_disable(pdev);
 
 	ne_teardown_msix(pdev);
@@ -1797,6 +1880,8 @@ static void ne_pci_shutdown(struct pci_dev *pdev)
 
 	ne_devs.ne_pci_dev = NULL;
 
+	ne_mux_teardown(pdev);
+
 	ne_pci_dev_disable(pdev);
 
 	ne_teardown_msix(pdev);
diff --git a/drivers/virt/nitro_enclaves/ne_pci_dev.h b/drivers/virt/nitro_enclaves/ne_pci_dev.h
index aee22b4ed8b6e..2cef40c627f9f 100644
--- a/drivers/virt/nitro_enclaves/ne_pci_dev.h
+++ b/drivers/virt/nitro_enclaves/ne_pci_dev.h
@@ -94,6 +94,30 @@
  */
 #define NE_MUX_NUM_VECTORS	(0x01f0)
 
+/**
+ * NE_MUX_DONATE_ERR - (2 bytes) Register holding the outcome of the last
+ *		       interrupt multiplexer bitmap donation (Read-Only).
+ */
+#define NE_MUX_DONATE_ERR	(0x01f4)
+
+/**
+ * NE_MUX_SHUTDOWN - (4 bytes) Register to tell the device to stop using every
+ *		     donated interrupt multiplexer bitmap (Write-Only).
+ */
+#define NE_MUX_SHUTDOWN		(0x01f8)
+#define NE_MUX_SHUTDOWN_ON	(0x01)
+
+/**
+ * NE_MUX_DONATE_BASE - (8 bytes per pooled vector) Array of registers to hand
+ *			the device the bitmap of one pooled vector (Read/Write).
+ *
+ * The device sets the bit of an enclave queue in the bitmap of the vector that
+ * queue shares, and then raises that vector's interrupt. It raises for every
+ * bit it sets, whatever the bitmap holds at the time, so bits a driver leaves
+ * in a bitmap never keep a later kick from arriving.
+ */
+#define NE_MUX_DONATE_BASE	(0x0800)
+
 /**
  * DOC: Device MMIO buffer sizes
  */
@@ -163,6 +187,20 @@
  */
 #define NE_MUX_MAX_VECTORS	(256)
 
+/**
+ * NE_MUX_MAX_BITS - Largest number of enclave queues one pooled vector
+ *		     multiplexes, which is how many bits the page donated as its
+ *		     bitmap holds.
+ */
+#define NE_MUX_MAX_BITS		(PAGE_SIZE * BITS_PER_BYTE)
+
+/*
+ * A bitmap is read in u64 words, so the words that span NE_MUX_MAX_BITS bits
+ * have to stay inside the one page donated for it.
+ */
+static_assert(NE_MUX_MAX_BITS / BITS_PER_TYPE(u64) * sizeof(u64) <= PAGE_SIZE,
+	      "the bitmap of a pooled vector has to fit its donated page");
+
 /**
  * enum ne_pci_dev_cmd_type - Device command types.
  * @INVALID_CMD:		Invalid command.
@@ -605,6 +643,16 @@ struct ne_slot_pool_run {
 	unsigned long		nr_pages;
 };
 
+/**
+ * struct ne_mux_vec - Interrupt multiplexer state of one pooled MSI-X vector.
+ * @bitmap:			Page-sized bitmap in which the device sets a bit
+ *				for every enclave queue that signalled through
+ *				this vector.
+ */
+struct ne_mux_vec {
+	u64			*bitmap;
+};
+
 /**
  * struct ne_pci_dev - Nitro Enclaves (NE) PCI device.
  * @cmd_reply_avail:		Variable set if a reply has been sent by the
@@ -624,6 +672,7 @@ struct ne_slot_pool_run {
  *				capability, and with it the multiplexer and slot
  *				pool registers.
  * @iomem_base :		MMIO region of the PCI device.
+ * @mux:			Per-pooled-vector interrupt multiplexer state.
  * @mux_nr_vecs:		Number of MSI-X vectors in the interrupt
  *				multiplexer pool.
  * @mux_vec_base:		First MSI-X vector of the interrupt multiplexer
@@ -657,6 +706,7 @@ struct ne_pci_dev {
 	struct workqueue_struct	*event_wq;
 	bool			has_vendor_cap;
 	void __iomem		*iomem_base;
+	struct ne_mux_vec	*mux;
 	u32			mux_nr_vecs;
 	u32			mux_vec_base;
 	struct work_struct	notify_work;
-- 
2.47.1

[PATCH 27/40] nitro_enclaves: Dispatch multiplexed interrupts to per-queue bindings

[Alexander Graf]

Nitro Sandboxes provide a way for user space to receive
per-device-virtqueue events. These are multiplexed through a
nitro_enclaves device built-in interrupt multiplexer. Implement logic to
support this multiplexer and distribute the respective interrupt events.

Signed-off-by: Alexander Graf <graf@amazon.com>
---
 drivers/virt/nitro_enclaves/ne_pci_dev.c | 179 ++++++++++++++++++++++-
 drivers/virt/nitro_enclaves/ne_pci_dev.h |  47 ++++++
 2 files changed, 225 insertions(+), 1 deletion(-)

diff --git a/drivers/virt/nitro_enclaves/ne_pci_dev.c b/drivers/virt/nitro_enclaves/ne_pci_dev.c
index 0fafd42162f3a..de96d68906bbf 100644
--- a/drivers/virt/nitro_enclaves/ne_pci_dev.c
+++ b/drivers/virt/nitro_enclaves/ne_pci_dev.c
@@ -7,6 +7,7 @@
  * DOC: Nitro Enclaves (NE) PCI device driver.
  */
 
+#include <linux/bitops.h>
 #include <linux/cma.h>
 #include <linux/delay.h>
 #include <linux/device.h>
@@ -27,6 +28,7 @@
 #include <linux/sysfs.h>
 #include <linux/types.h>
 #include <linux/wait.h>
+#include <linux/xarray.h>
 
 #include "ne_misc_dev.h"
 #include "ne_pci_dev.h"
@@ -551,6 +553,129 @@ static void ne_teardown_msix(struct pci_dev *pdev)
 	pci_free_irq_vectors(pdev);
 }
 
+/**
+ * ne_mux_binding_rcu_free() - RCU callback freeing one enclave queue binding.
+ * @head:	RCU head of the binding to free.
+ *
+ * Context: Softirq or process context.
+ */
+static void ne_mux_binding_rcu_free(struct rcu_head *head)
+{
+	struct ne_mux_binding *binding = container_of(head,
+						      struct ne_mux_binding,
+						      rcu);
+
+	eventfd_ctx_put(binding->ctx);
+
+	kfree(binding);
+}
+
+void ne_mux_binding_free_rcu(struct ne_mux_binding *binding)
+{
+	call_rcu(&binding->rcu, ne_mux_binding_rcu_free);
+}
+
+/**
+ * ne_mux_scan() - Deliver the bits set in the bitmap of a pooled vector that it
+ *		   has a binding for.
+ * @mux_vec:	Multiplexer state of the pooled vector.
+ *
+ * A bit leaves the bitmap by the same atomic read-modify-write that decides to
+ * deliver it, so a bit this vector has no binding for stays set. It is left set
+ * on purpose: a queue can be kicked before user space has bound an eventfd to
+ * it, and a bit taken out of the bitmap on the strength of a binding that is not
+ * there yet is a kick nobody can ever be told about. The bit waits instead, and
+ * the bind of that queue takes it and delivers it.
+ *
+ * So a set bit is consumed at most once, and exactly once if a scan finds a
+ * binding for it or a bind for its index runs. A bit whose index is never bound
+ * is consumed never, and stays set until the page is reclaimed when the device
+ * unbinds. A bit whose bind is still coming looks the same to the driver, so it
+ * holds both.
+ *
+ * The bound only ever rises, so a scan reads every word a binding has reached
+ * into and not only the words the bindings of the moment need.
+ *
+ * Context: Interrupt context. Called with the RCU read-side held.
+ * Return:
+ * * true if the scan found no bit it could deliver, false if it found at least
+ *   one.
+ */
+static bool ne_mux_scan(struct ne_mux_vec *mux_vec)
+{
+	struct ne_pci_dev *ne_pci_dev = mux_vec->ne_pci_dev;
+	u32 bits_per_word = BITS_PER_TYPE(*mux_vec->bitmap);
+	u32 nr_words = READ_ONCE(mux_vec->max_bit) / bits_per_word + 1;
+	bool empty = true;
+	u32 i = 0;
+
+	for (i = 0; i < nr_words; i++) {
+		u64 word = READ_ONCE(mux_vec->bitmap[i]);
+
+		while (word) {
+			u32 bit = i * bits_per_word + __ffs64(word);
+			unsigned long notify_index = bit * ne_pci_dev->mux_nr_vecs +
+						     mux_vec->vec_rel;
+			struct ne_mux_binding *binding = NULL;
+
+			word &= word - 1;
+
+			binding = xa_load(&ne_pci_dev->notify_xa, notify_index);
+			if (!binding)
+				continue;
+
+			empty = false;
+
+			if (test_and_clear_bit(bit, (unsigned long *)mux_vec->bitmap))
+				eventfd_signal(binding->ctx);
+		}
+	}
+
+	return empty;
+}
+
+/**
+ * ne_mux_handler() - Interrupt handler for one pooled MSI-X vector.
+ * @irq:	Interrupt line the pooled vector is on.
+ * @data:	Multiplexer state of the pooled vector.
+ *
+ * Context: Interrupt context.
+ * Return:
+ * * IRQ_HANDLED on interrupt handled.
+ */
+static irqreturn_t ne_mux_handler(int irq, void *data)
+{
+	struct ne_mux_vec *mux_vec = data;
+	u32 pass = 0;
+
+	/*
+	 * xa_load() drops its own RCU read-side before it returns, so take one
+	 * here to keep a binding alive for as long as it is dereferenced.
+	 */
+	rcu_read_lock();
+
+	/*
+	 * The device sets a bit and then raises the interrupt, and one interrupt
+	 * can stand for several bits, so a scan can miss a bit set while it runs
+	 * or one past the bound it read. The device raises for every bit it sets
+	 * whatever the bitmap holds, so such a bit has an interrupt of its own
+	 * still to come and reaches its eventfd on that one. Scanning again,
+	 * reading the bound again, delivers it on this interrupt instead, so stop
+	 * on the first scan that delivers nothing. Stopping on an empty bitmap
+	 * instead would wait for a read a bit no bind is coming for never gives.
+	 * NE_MUX_MAX_PASSES caps the scans either way, so a device that goes on
+	 * setting bits cannot hold the handler.
+	 */
+	do {
+		if (ne_mux_scan(mux_vec))
+			break;
+	} while (++pass < NE_MUX_MAX_PASSES);
+
+	rcu_read_unlock();
+
+	return IRQ_HANDLED;
+}
+
 /**
  * ne_mux_teardown() - Take the interrupt multiplexer bitmaps back from the
  *		       device.
@@ -561,6 +686,8 @@ static void ne_teardown_msix(struct pci_dev *pdev)
 static void ne_mux_teardown(struct pci_dev *pdev)
 {
 	struct ne_pci_dev *ne_pci_dev = pci_get_drvdata(pdev);
+	struct ne_mux_binding *binding = NULL;
+	unsigned long notify_index = 0;
 	u32 i = 0;
 
 	if (!ne_pci_dev->mux)
@@ -568,13 +695,40 @@ static void ne_mux_teardown(struct pci_dev *pdev)
 
 	iowrite32(NE_MUX_SHUTDOWN_ON, ne_pci_dev->iomem_base + NE_MUX_SHUTDOWN);
 
-	for (i = 0; i < ne_pci_dev->mux_nr_vecs; i++)
+	for (i = 0; i < ne_pci_dev->mux_nr_vecs; i++) {
+		if (!ne_pci_dev->mux[i].bitmap)
+			continue;
+
+		if (ne_pci_dev->mux[i].irq_requested)
+			free_irq(pci_irq_vector(pdev,
+						ne_pci_dev->mux_vec_base + i),
+				 &ne_pci_dev->mux[i]);
+
 		free_page((unsigned long)ne_pci_dev->mux[i].bitmap);
+	}
 
 	kfree(ne_pci_dev->mux);
 
 	ne_pci_dev->mux = NULL;
 	ne_pci_dev->mux_nr_vecs = 0;
+
+	xa_for_each(&ne_pci_dev->notify_xa, notify_index, binding) {
+		xa_erase(&ne_pci_dev->notify_xa, notify_index);
+
+		eventfd_ctx_put(binding->ctx);
+
+		kfree(binding);
+	}
+
+	/*
+	 * Freeing the handlers above quiesced the readers that hold no lock,
+	 * and the loop above freed every binding that outlived them. Wait for
+	 * the callbacks a replaced binding already queued, so that none of
+	 * them runs once the module text is gone.
+	 */
+	rcu_barrier();
+
+	xa_destroy(&ne_pci_dev->notify_xa);
 }
 
 /**
@@ -631,8 +785,11 @@ static int ne_mux_setup(struct pci_dev *pdev)
 	ne_pci_dev->mux_nr_vecs = nr_vecs;
 	ne_pci_dev->mux_vec_base = vec_base;
 
+	xa_init(&ne_pci_dev->notify_xa);
+
 	for (i = 0; i < nr_vecs; i++) {
 		u16 donate_err = 0;
+		int irq = 0;
 
 		ne_pci_dev->mux[i].bitmap = (u64 *)get_zeroed_page(GFP_KERNEL);
 		if (!ne_pci_dev->mux[i].bitmap) {
@@ -663,6 +820,26 @@ static int ne_mux_setup(struct pci_dev *pdev)
 
 			goto teardown_mux;
 		}
+
+		ne_pci_dev->mux[i].ne_pci_dev = ne_pci_dev;
+		ne_pci_dev->mux[i].vec_rel = i;
+
+		irq = pci_irq_vector(pdev, vec_base + i);
+		if (irq < 0) {
+			rc = irq;
+
+			goto teardown_mux;
+		}
+
+		rc = request_irq(irq, ne_mux_handler, 0, "enclave_mux",
+				 &ne_pci_dev->mux[i]);
+		if (rc < 0) {
+			dev_err(&pdev->dev, "Error in request irq mux [rc=%d]\n", rc);
+
+			goto teardown_mux;
+		}
+
+		ne_pci_dev->mux[i].irq_requested = true;
 	}
 
 	return 0;
diff --git a/drivers/virt/nitro_enclaves/ne_pci_dev.h b/drivers/virt/nitro_enclaves/ne_pci_dev.h
index 2cef40c627f9f..0dbce83cb06bd 100644
--- a/drivers/virt/nitro_enclaves/ne_pci_dev.h
+++ b/drivers/virt/nitro_enclaves/ne_pci_dev.h
@@ -9,12 +9,15 @@
 #include <linux/atomic.h>
 #include <linux/bits.h>
 #include <linux/build_bug.h>
+#include <linux/eventfd.h>
 #include <linux/list.h>
 #include <linux/mutex.h>
 #include <linux/pci.h>
 #include <linux/pci_ids.h>
+#include <linux/rcupdate.h>
 #include <linux/stddef.h>
 #include <linux/wait.h>
+#include <linux/xarray.h>
 
 /**
  * DOC: Nitro Enclaves (NE) PCI device
@@ -201,6 +204,12 @@
 static_assert(NE_MUX_MAX_BITS / BITS_PER_TYPE(u64) * sizeof(u64) <= PAGE_SIZE,
 	      "the bitmap of a pooled vector has to fit its donated page");
 
+/**
+ * NE_MUX_MAX_PASSES - Largest number of times the handler of a pooled vector
+ *		       scans its bitmap in one interrupt.
+ */
+#define NE_MUX_MAX_PASSES	(4)
+
 /**
  * enum ne_pci_dev_cmd_type - Device command types.
  * @INVALID_CMD:		Invalid command.
@@ -648,9 +657,35 @@ struct ne_slot_pool_run {
  * @bitmap:			Page-sized bitmap in which the device sets a bit
  *				for every enclave queue that signalled through
  *				this vector.
+ * @max_bit:			Highest bitmap bit an enclave queue has been
+ *				bound to, bounding the scan of @bitmap. It only
+ *				rises, so a scan reads the words of bindings that
+ *				are gone as well.
+ * @ne_pci_dev:			NE PCI device this vector belongs to.
+ * @vec_rel:			Index of this vector within the pool.
+ * @irq_requested:		Whether the interrupt handler of this vector is
+ *				installed, which is not implied by @bitmap: a
+ *				vector whose bitmap the device already owns may
+ *				have no handler yet.
  */
 struct ne_mux_vec {
 	u64			*bitmap;
+	u32			max_bit;
+	struct ne_pci_dev	*ne_pci_dev;
+	u32			vec_rel;
+	bool			irq_requested;
+};
+
+/**
+ * struct ne_mux_binding - Eventfd bound to one enclave queue.
+ * @ctx:			Eventfd signalled when the queue is kicked.
+ * @rcu:			Head used to free the binding after a grace
+ *				period, so that an interrupt handler which
+ *				already loaded it can still dereference it.
+ */
+struct ne_mux_binding {
+	struct eventfd_ctx	*ctx;
+	struct rcu_head		rcu;
 };
 
 /**
@@ -672,6 +707,8 @@ struct ne_mux_vec {
  *				capability, and with it the multiplexer and slot
  *				pool registers.
  * @iomem_base :		MMIO region of the PCI device.
+ * @notify_xa:			Enclave queue bindings, keyed by the notify index of
+ *				the queue.
  * @mux:			Per-pooled-vector interrupt multiplexer state.
  * @mux_nr_vecs:		Number of MSI-X vectors in the interrupt
  *				multiplexer pool.
@@ -706,6 +743,7 @@ struct ne_pci_dev {
 	struct workqueue_struct	*event_wq;
 	bool			has_vendor_cap;
 	void __iomem		*iomem_base;
+	struct xarray		notify_xa;
 	struct ne_mux_vec	*mux;
 	u32			mux_nr_vecs;
 	u32			mux_vec_base;
@@ -750,6 +788,15 @@ int ne_do_request(struct pci_dev *pdev, enum ne_pci_dev_cmd_type cmd_type,
 		  struct ne_pci_dev_cmd_reply *cmd_reply,
 		  size_t cmd_reply_size);
 
+/**
+ * ne_mux_binding_free_rcu() - Free an enclave queue binding after a grace
+ *			       period.
+ * @binding:	Binding that has been removed from the queue bindings xarray.
+ *
+ * Context: Process context.
+ */
+void ne_mux_binding_free_rcu(struct ne_mux_binding *binding);
+
 /* Nitro Enclaves (NE) PCI device driver */
 extern struct pci_driver ne_pci_driver;
 
-- 
2.47.1

[PATCH 28/40] nitro_enclaves: Add NE_SET_NOTIFY_EVENTFD

[Alexander Graf]

Nitro Sandboxes notify the parent through the interrupt multiplexer
when a sandbox kicks a virtqueue, so that the device backend in the
parent can wait for work on its queues on an eventfd.

Add NE_SET_NOTIFY_EVENTFD on the enclave fd. It binds an eventfd to the
notify index of a queue, and the multiplexer handler signals it on a
kick. Binding again replaces the eventfd and NE_EVENTFD_UNBIND removes
it. A bind delivers a kick that was already pending, and an unbind drops
one, so that the next owner of the index does not get it. A notify index
is device-wide, so a bind only takes the indices NE_ADD_DEVICE reported
to the same enclave.

Signed-off-by: Alexander Graf <graf@amazon.com>
---
 drivers/virt/nitro_enclaves/Kconfig       |   1 +
 drivers/virt/nitro_enclaves/ne_misc_dev.c | 418 ++++++++++++++++++++++
 drivers/virt/nitro_enclaves/ne_misc_dev.h |  31 ++
 drivers/virt/nitro_enclaves/ne_pci_dev.c  |  41 ++-
 include/uapi/linux/nitro_enclaves.h       |  66 ++++
 5 files changed, 547 insertions(+), 10 deletions(-)

diff --git a/drivers/virt/nitro_enclaves/Kconfig b/drivers/virt/nitro_enclaves/Kconfig
index 7b2456cdbb9fb..c78a5c654a241 100644
--- a/drivers/virt/nitro_enclaves/Kconfig
+++ b/drivers/virt/nitro_enclaves/Kconfig
@@ -10,6 +10,7 @@ config NITRO_ENCLAVES
 	depends on (ARM64 || X86) && HOTPLUG_CPU && PCI && SMP
 	select CMA
 	select DMA_CMA
+	select EVENTFD
 	help
 	  This driver consists of support for enclave lifetime management
 	  for Nitro Enclaves (NE).
diff --git a/drivers/virt/nitro_enclaves/ne_misc_dev.c b/drivers/virt/nitro_enclaves/ne_misc_dev.c
index 7698c7210eb86..bc78580bad475 100644
--- a/drivers/virt/nitro_enclaves/ne_misc_dev.c
+++ b/drivers/virt/nitro_enclaves/ne_misc_dev.c
@@ -9,6 +9,7 @@
  */
 
 #include <linux/anon_inodes.h>
+#include <linux/bitops.h>
 #include <linux/bsearch.h>
 #include <linux/build_bug.h>
 #include <linux/capability.h>
@@ -17,6 +18,7 @@
 #include <linux/cpumask.h>
 #include <linux/device.h>
 #include <linux/dma-map-ops.h>
+#include <linux/eventfd.h>
 #include <linux/file.h>
 #include <linux/fs.h>
 #include <linux/highmem.h>
@@ -48,6 +50,7 @@
 #include <linux/types.h>
 #include <linux/uaccess.h>
 #include <linux/uio.h>
+#include <linux/xarray.h>
 #include <uapi/linux/vm_sockets.h>
 
 #include "ne_misc_dev.h"
@@ -1773,6 +1776,357 @@ static int ne_start_enclave_ioctl(struct ne_enclave *ne_enclave,
 	return 0;
 }
 
+/**
+ * ne_mux_drop_bit() - Take the bit of an enclave queue out of the bitmap of its
+ *		       pooled vector.
+ * @ne_pci_dev:		Private data associated with the PCI device.
+ * @notify_index:	Notify index of the queue whose bit to take out.
+ *
+ * The handler leaves a bit it finds no binding for set, so that a bind can take
+ * it and deliver the kick it stands for. Once the binding is gone, the bind
+ * that bit was waiting for has already happened, and the next bind of the index
+ * belongs to whoever holds the queue by then, so leaving the bit would hand one
+ * caller's kick to another. It also costs every later interrupt on the vector a
+ * lookup for a bit nothing will claim. Take it out.
+ *
+ * The caller takes the binding out of the xarray first, so at this point no
+ * binding for the index is left to look up, and the kick this destroys is one
+ * no later scan could have delivered. In the other order a kick the device
+ * recorded in between would be left with a live binding and no interrupt of
+ * its own still to come.
+ *
+ * A kick the enclave raises between the two is dropped. It is a kick that
+ * arrived as its caller stopped listening, so the ioctl documentation says a
+ * release leaves a later bind of the index no kick to collect.
+ *
+ * Context: Process context.
+ */
+static void ne_mux_drop_bit(struct ne_pci_dev *ne_pci_dev, u32 notify_index)
+{
+	u32 nr_vecs = 0;
+	u32 vec_rel = 0;
+	u32 bit = 0;
+
+	/*
+	 * ne_mux_teardown() sets the pool size back to zero, so take the size
+	 * once and do the arithmetic on that copy, as a bind does. Reading the
+	 * field again below would divide by whatever it holds by then.
+	 */
+	nr_vecs = READ_ONCE(ne_pci_dev->mux_nr_vecs);
+	if (!nr_vecs)
+		return;
+
+	vec_rel = notify_index % nr_vecs;
+	bit = notify_index / nr_vecs;
+	if (bit >= NE_MUX_MAX_BITS)
+		return;
+
+	if (!ne_pci_dev->mux || !ne_pci_dev->mux[vec_rel].bitmap)
+		return;
+
+	clear_bit(bit, (unsigned long *)ne_pci_dev->mux[vec_rel].bitmap);
+}
+
+/**
+ * ne_del_notify_eventfd() - Release the eventfd bound to the interrupt of an
+ *			     enclave queue.
+ * @ne_enclave :	Private data associated with the current enclave.
+ * @ne_pci_dev:		Private data associated with the PCI device, taken by
+ *			the caller so that the release runs on the device the
+ *			caller looked the enclave up on.
+ * @notify_index:	Notify index of the queue to release.
+ *
+ * Context: Process context.
+ * Return:
+ * * 0 on success.
+ * * Negative return value on failure.
+ */
+static int ne_del_notify_eventfd(struct ne_enclave *ne_enclave,
+				 struct ne_pci_dev *ne_pci_dev, u32 notify_index)
+{
+	struct ne_notify_binding *notify_binding = NULL;
+	struct ne_mux_binding *binding = NULL;
+	struct ne_notify_binding *entry = NULL;
+	int rc;
+
+	rc = mutex_lock_killable(&ne_enclave->enclave_info_mutex);
+	if (rc)
+		return rc;
+
+	list_for_each_entry(entry, &ne_enclave->notify_bindings,
+			    notify_binding_list_entry) {
+		if (entry->notify_index == notify_index) {
+			notify_binding = entry;
+
+			break;
+		}
+	}
+
+	if (!notify_binding) {
+		mutex_unlock(&ne_enclave->enclave_info_mutex);
+
+		return -ENOENT;
+	}
+
+	binding = xa_erase(&ne_pci_dev->notify_xa, notify_index);
+	if (binding) {
+		/*
+		 * A handler that looked the binding up before the erase can
+		 * take the bit after it and then look the index up again. A
+		 * bind of the index can run as soon as the unbind returns, so
+		 * wait for that handler first. Otherwise its second lookup
+		 * finds the new binding and signals a kick the unbind drops.
+		 */
+		synchronize_rcu();
+
+		ne_mux_binding_free_rcu(binding);
+	}
+
+	ne_mux_drop_bit(ne_pci_dev, notify_index);
+
+	list_del(&notify_binding->notify_binding_list_entry);
+
+	mutex_unlock(&ne_enclave->enclave_info_mutex);
+
+	kfree(notify_binding);
+
+	return 0;
+}
+
+/**
+ * ne_enclave_owns_notify() - Check that NE_ADD_DEVICE reported a notify index
+ *			      to an enclave.
+ * @ne_enclave :	Private data associated with the current enclave.
+ * @notify_index:	Notify index to look for.
+ *
+ * Context: Process context. This function is called with the ne_enclave mutex
+ *	    held.
+ * Return: true if one of the enclave's devices has the notify index.
+ */
+static bool ne_enclave_owns_notify(struct ne_enclave *ne_enclave,
+				   u32 notify_index)
+{
+	struct ne_device_wiring *dev = NULL;
+	unsigned int i = 0;
+
+	list_for_each_entry(dev, &ne_enclave->devices, list_entry)
+		for (i = 0; i < dev->num_vqs; i++)
+			if (dev->notify_index[i] == notify_index)
+				return true;
+
+	return false;
+}
+
+/**
+ * ne_set_notify_eventfd() - Bind an eventfd to the interrupt of an enclave
+ *			     queue.
+ * @ne_enclave :	Private data associated with the current enclave.
+ * @notify:		The eventfd and the notify index.
+ *
+ * Context: Process context.
+ * Return:
+ * * 0 on success.
+ * * Negative return value on failure.
+ */
+static int ne_set_notify_eventfd(struct ne_enclave *ne_enclave,
+				 struct ne_notify_eventfd *notify)
+{
+	struct ne_pci_dev *ne_pci_dev = ne_devs.ne_pci_dev;
+	struct ne_notify_binding *notify_binding = NULL;
+	struct ne_mux_binding *binding = NULL;
+	struct ne_notify_binding *entry = NULL;
+	struct ne_mux_binding *old = NULL;
+	struct eventfd_ctx *ctx = NULL;
+	bool rebind = false;
+	u32 nr_vecs = 0;
+	u32 max_bit = 0;
+	int rc = -EINVAL;
+	u32 vec_rel = 0;
+	u32 bit = 0;
+
+	/*
+	 * ne_mux_teardown() sets the pool size back to zero, so take the size
+	 * once and do the arithmetic on that copy. Reading the field again
+	 * below would divide by whatever it holds by then.
+	 */
+	nr_vecs = READ_ONCE(ne_pci_dev->mux_nr_vecs);
+	if (!nr_vecs)
+		return -EINVAL;
+
+	vec_rel = notify->notify_index % nr_vecs;
+	bit = notify->notify_index / nr_vecs;
+	if (bit >= NE_MUX_MAX_BITS)
+		return -EINVAL;
+
+	if (notify->fd == NE_EVENTFD_UNBIND)
+		return ne_del_notify_eventfd(ne_enclave, ne_pci_dev,
+					     notify->notify_index);
+
+	ctx = eventfd_ctx_fdget(notify->fd);
+	if (IS_ERR(ctx))
+		return PTR_ERR(ctx);
+
+	binding = kzalloc_obj(*binding);
+	if (!binding) {
+		rc = -ENOMEM;
+
+		goto put_ctx;
+	}
+
+	binding->ctx = ctx;
+
+	notify_binding = kzalloc_obj(*notify_binding);
+	if (!notify_binding) {
+		rc = -ENOMEM;
+
+		goto free_binding;
+	}
+
+	notify_binding->notify_index = notify->notify_index;
+
+	rc = mutex_lock_killable(&ne_enclave->enclave_info_mutex);
+	if (rc)
+		goto free_notify_binding;
+
+	/*
+	 * A notify index names a notification across the whole NE PCI device,
+	 * so an enclave only binds the ones NE_ADD_DEVICE reported to it.
+	 */
+	if (!ne_enclave_owns_notify(ne_enclave, notify->notify_index)) {
+		rc = -EINVAL;
+
+		goto unlock_mutex;
+	}
+
+	list_for_each_entry(entry, &ne_enclave->notify_bindings,
+			    notify_binding_list_entry) {
+		if (entry->notify_index == notify->notify_index) {
+			rebind = true;
+
+			break;
+		}
+	}
+
+	if (rebind) {
+		/*
+		 * This enclave owns the notify index and, by the invariant the
+		 * fresh bind below upholds, is the only owner of its xarray
+		 * entry, so the entry that comes back is the binding it set
+		 * last time.
+		 */
+		old = xa_store(&ne_pci_dev->notify_xa, notify->notify_index, binding,
+			       GFP_KERNEL);
+		if (xa_is_err(old)) {
+			rc = xa_err(old);
+
+			goto unlock_mutex;
+		}
+
+		if (old)
+			ne_mux_binding_free_rcu(old);
+
+		kfree(notify_binding);
+	} else {
+		/*
+		 * Bind against an empty entry so that an enclave which asks
+		 * for a notify index another one still owns leaves that owner's
+		 * eventfd in place.
+		 */
+		old = xa_cmpxchg(&ne_pci_dev->notify_xa, notify->notify_index, NULL,
+				 binding, GFP_KERNEL);
+		if (xa_is_err(old)) {
+			rc = xa_err(old);
+
+			goto unlock_mutex;
+		}
+
+		if (old) {
+			rc = -EEXIST;
+
+			goto unlock_mutex;
+		}
+
+		list_add(&notify_binding->notify_binding_list_entry,
+			 &ne_enclave->notify_bindings);
+	}
+
+	/*
+	 * Raise the scan bound of the vector the queue shares. Enclaves bind
+	 * under their own mutex, so two of them can reach the same vector at
+	 * once and a plain read-modify-write would lose a raise and with it a
+	 * queue kick.
+	 */
+	max_bit = READ_ONCE(ne_pci_dev->mux[vec_rel].max_bit);
+	while (bit > max_bit)
+		if (try_cmpxchg(&ne_pci_dev->mux[vec_rel].max_bit, &max_bit,
+				bit))
+			break;
+
+	/*
+	 * The device may have set the bit of this queue before there was a
+	 * binding to deliver it on, and the handler leaves a bit it has no
+	 * binding for set, so that bit is still there to be taken. Take it
+	 * here, after the binding is in the xarray so that a handler taking it
+	 * instead of this path finds the binding, and after the raise above so
+	 * that such a handler reaches the word the bit is in. The handler takes
+	 * a bit with an atomic read-modify-write on the same word, so exactly
+	 * one of the two consumes it. The bitmap words are the width the device
+	 * writes them in, which is the width the bit numbering of the handler
+	 * is in as well.
+	 */
+	if (test_and_clear_bit(bit, (unsigned long *)ne_pci_dev->mux[vec_rel].bitmap))
+		eventfd_signal(binding->ctx);
+
+	mutex_unlock(&ne_enclave->enclave_info_mutex);
+
+	return 0;
+
+unlock_mutex:
+	mutex_unlock(&ne_enclave->enclave_info_mutex);
+free_notify_binding:
+	kfree(notify_binding);
+free_binding:
+	kfree(binding);
+put_ctx:
+	eventfd_ctx_put(ctx);
+
+	return rc;
+}
+
+/**
+ * ne_enclave_remove_all_notify_binding_entries() - Remove all the enclave queue
+ *						    bindings from the device
+ *						    wide bindings xarray.
+ * @ne_enclave :	Private data associated with the current enclave.
+ * @ne_pci_dev:		Private data associated with the PCI device, taken by
+ *			the caller so that the removal runs on the device the
+ *			caller looked the enclave up on.
+ *
+ * Context: Process context. This function is called with the enclave
+ *	    information mutex held.
+ */
+static void ne_enclave_remove_all_notify_binding_entries(struct ne_enclave *ne_enclave,
+							 struct ne_pci_dev *ne_pci_dev)
+{
+	struct ne_notify_binding *notify_binding = NULL;
+	struct ne_notify_binding *notify_binding_tmp = NULL;
+	struct ne_mux_binding *binding = NULL;
+
+	list_for_each_entry_safe(notify_binding, notify_binding_tmp,
+				 &ne_enclave->notify_bindings,
+				 notify_binding_list_entry) {
+		binding = xa_erase(&ne_pci_dev->notify_xa, notify_binding->notify_index);
+		if (binding)
+			ne_mux_binding_free_rcu(binding);
+
+		ne_mux_drop_bit(ne_pci_dev, notify_binding->notify_index);
+
+		list_del(&notify_binding->notify_binding_list_entry);
+
+		kfree(notify_binding);
+	}
+}
+
 /**
  * ne_enclave_do_ioctl() - Ioctl function provided by the enclave file.
  * @file:	File associated with this ioctl function.
@@ -2230,6 +2584,22 @@ static long ne_enclave_do_ioctl(struct file *file, unsigned int cmd, unsigned lo
 		return 0;
 	}
 
+	case NE_EA_IOCTL(NE_SET_NOTIFY_EVENTFD): {
+		struct ne_notify_eventfd __user *uargs = (void __user *)arg;
+		unsigned int usize = _IOC_SIZE(cmd);
+		struct ne_notify_eventfd args;
+		int rc;
+
+		if (usize < offsetofend(struct ne_notify_eventfd, notify_index))
+			return -EINVAL;
+
+		rc = copy_struct_from_user(&args, sizeof(args), uargs, usize);
+		if (rc)
+			return rc;
+
+		return ne_set_notify_eventfd(ne_enclave, &args);
+	}
+
 	default:
 		return -ENOTTY;
 	}
@@ -2415,6 +2785,27 @@ int ne_enclave_free_slot(struct ne_pci_dev *ne_pci_dev,
 	rc = ne_do_request(pdev, ENCLAVE_STOP,
 			   &enclave_stop_request, sizeof(enclave_stop_request),
 			   &cmd_reply, sizeof(cmd_reply));
+
+	/*
+	 * The notify index of a queue belongs to the slot, so the device is free
+	 * to give it to another enclave once the slot is freed. Take the
+	 * bindings of this enclave out of the xarray the handler looks them up
+	 * in before that, or a bind for the reissued index finds the entry
+	 * still taken. This happens whether or not the stop went through,
+	 * because an unbind of the device tears the multiplexer down next.
+	 *
+	 * A handler that looked up a binding of this enclave before it was
+	 * erased here can still be running. Were the index reissued under it,
+	 * it would take the bit of a notification the device raises for the
+	 * next owner and signal this enclave's eventfd instead, so wait for it
+	 * before the slot goes.
+	 */
+	if (!list_empty(&ne_enclave->notify_bindings)) {
+		ne_enclave_remove_all_notify_binding_entries(ne_enclave, ne_pci_dev);
+
+		synchronize_rcu();
+	}
+
 	if (rc < 0)
 		return rc;
 
@@ -2437,6 +2828,9 @@ int ne_enclave_free_slot(struct ne_pci_dev *ne_pci_dev,
  */
 static void ne_enclave_free(struct ne_enclave *ne_enclave)
 {
+	struct ne_device_wiring *dev_tmp = NULL;
+	struct ne_device_wiring *dev = NULL;
+
 	ne_enclave_remove_all_contig_regions(&ne_enclave->contig_regions_list);
 	ne_enclave_remove_all_contig_regions(&ne_enclave->device_regions_list);
 	ne_enclave_remove_all_mem_region_entries(ne_enclave);
@@ -2445,6 +2839,9 @@ static void ne_enclave_free(struct ne_enclave *ne_enclave)
 	if (ne_enclave->info_file)
 		fput(ne_enclave->info_file);
 
+	list_for_each_entry_safe(dev, dev_tmp, &ne_enclave->devices, list_entry)
+		kfree(dev);
+
 	kfree(ne_enclave);
 }
 
@@ -2686,6 +3083,8 @@ static int ne_create_vm_ioctl(struct ne_pci_dev *ne_pci_dev, u64 flags,
 
 	init_waitqueue_head(&ne_enclave->eventq);
 	ne_enclave->has_event = false;
+	INIT_LIST_HEAD(&ne_enclave->notify_bindings);
+	INIT_LIST_HEAD(&ne_enclave->devices);
 	mutex_init(&ne_enclave->enclave_info_mutex);
 	ne_enclave->max_mem_regions = cmd_reply.alloc.mem_regions;
 	INIT_LIST_HEAD(&ne_enclave->contig_regions_list);
@@ -4255,6 +4654,7 @@ static int ne_add_device_ioctl(struct ne_enclave *ne_enclave,
 	struct ne_shm_region *shm_regions = NULL;
 	const struct ne_vq_wiring *wiring = NULL;
 	struct slot_add_device_req *req = NULL;
+	struct ne_device_wiring *dev = NULL;
 	struct ne_contig_mem *info = NULL;
 	LIST_HEAD(regions);
 	u8 buf[NE_MAX_VQS * sizeof(struct ne_vq_wiring)];
@@ -4346,6 +4746,17 @@ static int ne_add_device_ioctl(struct ne_enclave *ne_enclave,
 		goto free;
 	}
 
+	/*
+	 * Allocate the record of what the device reports before it is told
+	 * about the device, so that nothing can fail once it has been.
+	 */
+	dev = kzalloc_obj(*dev);
+	if (!dev) {
+		rc = -ENOMEM;
+
+		goto free;
+	}
+
 	for (i = 0; i < args->num_shm_regions; i++) {
 		rc = ne_add_device_region(ne_enclave, &shm_regions[i], req,
 					  &regions);
@@ -4438,10 +4849,16 @@ static int ne_add_device_ioctl(struct ne_enclave *ne_enclave,
 		args->vq_info[i].notify_index = wiring[i].notify_index;
 		args->vq_info[i].msix_entry = wiring[i].msix_entry;
 		args->vq_info[i].doorbell_offset = wiring[i].doorbell_offset;
+
+		dev->notify_index[i] = wiring[i].notify_index;
 	}
 
 	args->device_uid = cmd_reply.add_device.device_uid;
 
+	dev->num_vqs = args->num_vqs;
+	list_add(&dev->list_entry, &ne_enclave->devices);
+	dev = NULL;
+
 	rc = 0;
 
 free:
@@ -4452,6 +4869,7 @@ static int ne_add_device_ioctl(struct ne_enclave *ne_enclave,
 	 * the enclave, leaving nothing here.
 	 */
 	ne_enclave_remove_all_contig_regions(&regions);
+	kfree(dev);
 	kfree(req);
 	kfree(shm_regions);
 
diff --git a/drivers/virt/nitro_enclaves/ne_misc_dev.h b/drivers/virt/nitro_enclaves/ne_misc_dev.h
index 4f695f1410aec..57e20fe3ee1da 100644
--- a/drivers/virt/nitro_enclaves/ne_misc_dev.h
+++ b/drivers/virt/nitro_enclaves/ne_misc_dev.h
@@ -112,6 +112,29 @@ struct ne_mem_region {
 	u64			userspace_addr;
 };
 
+/**
+ * struct ne_device_wiring - What %NE_ADD_DEVICE reported for one device of an
+ *			     enclave.
+ * @list_entry:		Entry in the list of the enclave's devices.
+ * @num_vqs:		Number of entries of @notify_index.
+ * @notify_index:	Notify index of each virtqueue of the device.
+ */
+struct ne_device_wiring {
+	struct list_head	list_entry;
+	u32			num_vqs;
+	u32			notify_index[NE_MAX_VQS_PER_DEVICE];
+};
+
+/**
+ * struct ne_notify_binding - Entry in the list of enclave virtqueue bindings.
+ * @notify_binding_list_entry:	Entry in the list of enclave queue bindings.
+ * @notify_index:		Notify index of the bound queue.
+ */
+struct ne_notify_binding {
+	struct list_head	notify_binding_list_entry;
+	unsigned long		notify_index;
+};
+
 /**
  * struct ne_enclave - Per-enclave data used for enclave lifetime management.
  * @enclave_info_mutex :	Mutex for accessing this internal state.
@@ -139,6 +162,12 @@ struct ne_mem_region {
  *				so that the device keeps the page after user space
  *				has closed the descriptor. NULL until
  *				%NE_SET_INFO_PAGE is called.
+ * @notify_bindings:		List of the virtqueues this enclave has bound an
+ *				eventfd to.
+ * @devices:			List of struct ne_device_wiring, one for each device
+ *				%NE_ADD_DEVICE gave the enclave. The notify indices
+ *				in it are the only ones the enclave may bind an
+ *				eventfd to.
  * @max_mem_regions:		The maximum number of memory regions that can be
  *				handled by the hypervisor.
  * @mem_regions_list:		Enclave user space memory regions list.
@@ -172,6 +201,8 @@ struct ne_enclave {
 	bool			has_event;
 	u64			id;
 	struct file		*info_file;
+	struct list_head	notify_bindings;
+	struct list_head	devices;
 	u64			max_mem_regions;
 	struct list_head	mem_regions_list;
 	u64			mem_size;
diff --git a/drivers/virt/nitro_enclaves/ne_pci_dev.c b/drivers/virt/nitro_enclaves/ne_pci_dev.c
index de96d68906bbf..a079abb2b4618 100644
--- a/drivers/virt/nitro_enclaves/ne_pci_dev.c
+++ b/drivers/virt/nitro_enclaves/ne_pci_dev.c
@@ -580,12 +580,15 @@ void ne_mux_binding_free_rcu(struct ne_mux_binding *binding)
  *		   has a binding for.
  * @mux_vec:	Multiplexer state of the pooled vector.
  *
- * A bit leaves the bitmap by the same atomic read-modify-write that decides to
- * deliver it, so a bit this vector has no binding for stays set. It is left set
- * on purpose: a queue can be kicked before user space has bound an eventfd to
- * it, and a bit taken out of the bitmap on the strength of a binding that is not
+ * A bit is delivered by an atomic read-modify-write that clears it as it reads
+ * it, so a bit this vector has no binding for stays set. It is left set on
+ * purpose: a queue can be kicked before user space has bound an eventfd to it,
+ * and a bit taken out of the bitmap on the strength of a binding that is not
  * there yet is a kick nobody can ever be told about. The bit waits instead, and
- * the bind of that queue takes it and delivers it.
+ * the bind of that queue takes it and delivers it. A bit is cleared without
+ * being delivered only for an index whose binding has just been erased: by
+ * ne_mux_drop_bit(), or by a scan that took the bit and then found the binding
+ * gone.
  *
  * So a set bit is consumed at most once, and exactly once if a scan finds a
  * binding for it or a bind for its index runs. A bit whose index is never bound
@@ -598,8 +601,8 @@ void ne_mux_binding_free_rcu(struct ne_mux_binding *binding)
  *
  * Context: Interrupt context. Called with the RCU read-side held.
  * Return:
- * * true if the scan found no bit it could deliver, false if it found at least
- *   one.
+ * * true if the scan found no set bit with a binding, false if it found at
+ *   least one.
  */
 static bool ne_mux_scan(struct ne_mux_vec *mux_vec)
 {
@@ -626,8 +629,26 @@ static bool ne_mux_scan(struct ne_mux_vec *mux_vec)
 
 			empty = false;
 
-			if (test_and_clear_bit(bit, (unsigned long *)mux_vec->bitmap))
-				eventfd_signal(binding->ctx);
+			if (!test_and_clear_bit(bit, (unsigned long *)mux_vec->bitmap))
+				continue;
+
+			/*
+			 * A rebind can replace the binding between the load above
+			 * and the bit taken here. The rebind stores its binding
+			 * before it tries to take the bit, and test_and_clear_bit()
+			 * is fully ordered on both sides, also when it finds the bit
+			 * clear, so a bit taken here after that attempt sees the new
+			 * binding. Load it again and signal the binding that load
+			 * returns, or drop the kick if the entry is gone. An unbind
+			 * waits for this handler before the index can be bound
+			 * again, so this load never returns a binding made after
+			 * the unbind.
+			 */
+			binding = xa_load(&ne_pci_dev->notify_xa, notify_index);
+			if (!binding)
+				continue;
+
+			eventfd_signal(binding->ctx);
 		}
 	}
 
@@ -661,7 +682,7 @@ static irqreturn_t ne_mux_handler(int irq, void *data)
 	 * whatever the bitmap holds, so such a bit has an interrupt of its own
 	 * still to come and reaches its eventfd on that one. Scanning again,
 	 * reading the bound again, delivers it on this interrupt instead, so stop
-	 * on the first scan that delivers nothing. Stopping on an empty bitmap
+	 * on the first scan that finds no bound bit. Stopping on an empty bitmap
 	 * instead would wait for a read a bit no bind is coming for never gives.
 	 * NE_MUX_MAX_PASSES caps the scans either way, so a device that goes on
 	 * setting bits cannot hold the handler.
diff --git a/include/uapi/linux/nitro_enclaves.h b/include/uapi/linux/nitro_enclaves.h
index 6f83a8b0762b3..ff1b32e380e7b 100644
--- a/include/uapi/linux/nitro_enclaves.h
+++ b/include/uapi/linux/nitro_enclaves.h
@@ -1061,4 +1061,70 @@ struct ne_enclave_start_info {
 	__u64	enclave_cid;
 };
 
+/**
+ * NE_EVENTFD_UNBIND - Value of an eventfd field that releases the binding the
+ *		       caller holds, instead of naming an eventfd to bind. A kick
+ *		       the enclave raises as the release lands is dropped, so a
+ *		       caller that binds the queue again gets no signal for it.
+ *		       Any other negative value names no open file and is an
+ *		       error.
+ */
+#define NE_EVENTFD_UNBIND		(-1)
+
+/**
+ * NE_SET_NOTIFY_EVENTFD - The command is used to bind an eventfd to the
+ *			   interrupt of one enclave virtqueue. When the enclave
+ *			   kicks the queue, the NE PCI device raises the
+ *			   interrupt the queue shares and the driver signals the
+ *			   eventfd. A queue the caller has already bound is
+ *			   rebound to the new eventfd, and the driver drops its
+ *			   reference on the one it replaces. A kick that is
+ *			   already being delivered when the rebind lands is
+ *			   signalled on the outgoing eventfd, so a caller that
+ *			   rebinds a queue an enclave is still kicking can see
+ *			   one last kick arrive on the eventfd it replaced.
+ *			   %NE_EVENTFD_UNBIND
+ *			   releases the binding the caller holds on the queue.
+ *			   The ioctl can be invoked on the enclave fd.
+ *
+ * Context: Process context.
+ * Return:
+ * * 0					- Logic successfully completed.
+ * * -1					- There was a failure in the ioctl logic.
+ * On failure, errno is set to:
+ * * ENODEV				- The NE PCI device is not available.
+ * * EFAULT				- Argument read failure.
+ * * EINVAL				- The argument is shorter than @notify_index,
+ *					  the NE PCI device shares no interrupts,
+ *					  the notify index addresses a bit outside
+ *					  the bitmap page of the vector it shares,
+ *					  %NE_ADD_DEVICE did not report the notify
+ *					  index to this enclave, or the given fd is
+ *					  not an eventfd.
+ * * EBADF				- There is no open file for the given fd, which
+ *					  is the case for every negative fd other
+ *					  than %NE_EVENTFD_UNBIND.
+ * * E2BIG				- The argument is longer than this kernel knows
+ *					  about and the excess is not all zero.
+ * * ENOMEM				- Memory allocation failure.
+ * * EEXIST				- Another enclave holds a binding on the queue.
+ * * ENOENT				- The caller holds no binding on the queue to
+ *					  release.
+ */
+#define NE_SET_NOTIFY_EVENTFD		_IOW(0xAE, 0x28, struct ne_notify_eventfd)
+
+/**
+ * struct ne_notify_eventfd - Eventfd to signal on an enclave virtqueue kick
+ *			      (in).
+ * @fd:			Eventfd to signal when the queue is kicked, or
+ *			%NE_EVENTFD_UNBIND to release the binding on the
+ *			queue (in).
+ * @notify_index:	Index the enclave writes to notify the NE PCI device of
+ *			the virtqueue (in).
+ */
+struct ne_notify_eventfd {
+	__s32	fd;
+	__u32	notify_index;
+};
+
 #endif /* _UAPI_LINUX_NITRO_ENCLAVES_H_ */
-- 
2.47.1

[PATCH 29/40] nitro_enclaves: Report the device's info notification

[Alexander Graf]

When the device updates the entry of a device in the info page, it
raises an event through the interrupt multiplexer. User space needs to
learn about these events, so we tell it the index as part of its
SLOT_ADD_DEVICE ioctl response in the new info_notify_index field.
It can then use it to request device update notifications the same
way it listens to VQ events.

Signed-off-by: Alexander Graf <graf@amazon.com>
---
 drivers/virt/nitro_enclaves/ne_misc_dev.c | 64 ++++++++++---------
 drivers/virt/nitro_enclaves/ne_misc_dev.h | 15 +++--
 drivers/virt/nitro_enclaves/ne_pci_dev.c  |  3 +-
 drivers/virt/nitro_enclaves/ne_pci_dev.h  | 30 ++++++---
 include/uapi/linux/nitro_enclaves.h       | 77 ++++++++++++++---------
 5 files changed, 115 insertions(+), 74 deletions(-)

diff --git a/drivers/virt/nitro_enclaves/ne_misc_dev.c b/drivers/virt/nitro_enclaves/ne_misc_dev.c
index bc78580bad475..d47fc8b514bcc 100644
--- a/drivers/virt/nitro_enclaves/ne_misc_dev.c
+++ b/drivers/virt/nitro_enclaves/ne_misc_dev.c
@@ -1777,27 +1777,27 @@ static int ne_start_enclave_ioctl(struct ne_enclave *ne_enclave,
 }
 
 /**
- * ne_mux_drop_bit() - Take the bit of an enclave queue out of the bitmap of its
- *		       pooled vector.
+ * ne_mux_drop_bit() - Take the bit of an enclave notification out of the bitmap
+ *		       of its pooled vector.
  * @ne_pci_dev:		Private data associated with the PCI device.
- * @notify_index:	Notify index of the queue whose bit to take out.
+ * @notify_index:	Notify index of the notification whose bit to take out.
  *
  * The handler leaves a bit it finds no binding for set, so that a bind can take
- * it and deliver the kick it stands for. Once the binding is gone, the bind
- * that bit was waiting for has already happened, and the next bind of the index
- * belongs to whoever holds the queue by then, so leaving the bit would hand one
- * caller's kick to another. It also costs every later interrupt on the vector a
- * lookup for a bit nothing will claim. Take it out.
+ * it and deliver the notification it stands for. Once the binding is gone, the
+ * bind that bit was waiting for has already happened, and the next bind of the
+ * index belongs to whoever holds the notification by then, so leaving the bit
+ * would hand one caller's notification to another. It also costs every later
+ * interrupt on the vector a lookup for a bit nothing will claim. Take it out.
  *
  * The caller takes the binding out of the xarray first, so at this point no
- * binding for the index is left to look up, and the kick this destroys is one
- * no later scan could have delivered. In the other order a kick the device
- * recorded in between would be left with a live binding and no interrupt of
- * its own still to come.
+ * binding for the index is left to look up, and the notification this destroys
+ * is one no later scan could have delivered. In the other order a notification
+ * the device recorded in between would be left with a live binding and no
+ * interrupt of its own still to come.
  *
- * A kick the enclave raises between the two is dropped. It is a kick that
- * arrived as its caller stopped listening, so the ioctl documentation says a
- * release leaves a later bind of the index no kick to collect.
+ * A notification the enclave raises between the two is dropped. It arrived as
+ * its caller stopped listening, so the ioctl documentation says a release
+ * leaves a later bind of the index no notification to collect.
  *
  * Context: Process context.
  */
@@ -1828,13 +1828,13 @@ static void ne_mux_drop_bit(struct ne_pci_dev *ne_pci_dev, u32 notify_index)
 }
 
 /**
- * ne_del_notify_eventfd() - Release the eventfd bound to the interrupt of an
- *			     enclave queue.
+ * ne_del_notify_eventfd() - Release the eventfd bound to an enclave
+ *			     notification.
  * @ne_enclave :	Private data associated with the current enclave.
  * @ne_pci_dev:		Private data associated with the PCI device, taken by
  *			the caller so that the release runs on the device the
  *			caller looked the enclave up on.
- * @notify_index:	Notify index of the queue to release.
+ * @notify_index:	Notify index of the notification to release.
  *
  * Context: Process context.
  * Return:
@@ -1909,17 +1909,20 @@ static bool ne_enclave_owns_notify(struct ne_enclave *ne_enclave,
 	struct ne_device_wiring *dev = NULL;
 	unsigned int i = 0;
 
-	list_for_each_entry(dev, &ne_enclave->devices, list_entry)
+	list_for_each_entry(dev, &ne_enclave->devices, list_entry) {
+		if (dev->info_notify_index == notify_index)
+			return true;
+
 		for (i = 0; i < dev->num_vqs; i++)
 			if (dev->notify_index[i] == notify_index)
 				return true;
+	}
 
 	return false;
 }
 
 /**
- * ne_set_notify_eventfd() - Bind an eventfd to the interrupt of an enclave
- *			     queue.
+ * ne_set_notify_eventfd() - Bind an eventfd to an enclave notification.
  * @ne_enclave :	Private data associated with the current enclave.
  * @notify:		The eventfd and the notify index.
  *
@@ -2051,10 +2054,10 @@ static int ne_set_notify_eventfd(struct ne_enclave *ne_enclave,
 	}
 
 	/*
-	 * Raise the scan bound of the vector the queue shares. Enclaves bind
-	 * under their own mutex, so two of them can reach the same vector at
-	 * once and a plain read-modify-write would lose a raise and with it a
-	 * queue kick.
+	 * Raise the scan bound of the vector the notification shares. Enclaves
+	 * bind under their own mutex, so two of them can reach the same vector
+	 * at once and a plain read-modify-write would lose a raise and with it
+	 * a notification.
 	 */
 	max_bit = READ_ONCE(ne_pci_dev->mux[vec_rel].max_bit);
 	while (bit > max_bit)
@@ -2094,9 +2097,10 @@ static int ne_set_notify_eventfd(struct ne_enclave *ne_enclave,
 }
 
 /**
- * ne_enclave_remove_all_notify_binding_entries() - Remove all the enclave queue
- *						    bindings from the device
- *						    wide bindings xarray.
+ * ne_enclave_remove_all_notify_binding_entries() - Remove all the enclave
+ *						    notification bindings from
+ *						    the device wide bindings
+ *						    xarray.
  * @ne_enclave :	Private data associated with the current enclave.
  * @ne_pci_dev:		Private data associated with the PCI device, taken by
  *			the caller so that the removal runs on the device the
@@ -4675,7 +4679,7 @@ static int ne_add_device_ioctl(struct ne_enclave *ne_enclave,
 		return -NE_ERR_INVALID_FLAG_VALUE;
 	}
 
-	if (args->padding)
+	if (args->padding || args->padding2)
 		return -EINVAL;
 
 	if (args->num_vqs > NE_MAX_VQS) {
@@ -4854,8 +4858,10 @@ static int ne_add_device_ioctl(struct ne_enclave *ne_enclave,
 	}
 
 	args->device_uid = cmd_reply.add_device.device_uid;
+	args->info_notify_index = cmd_reply.add_device.info_notify_index;
 
 	dev->num_vqs = args->num_vqs;
+	dev->info_notify_index = args->info_notify_index;
 	list_add(&dev->list_entry, &ne_enclave->devices);
 	dev = NULL;
 
diff --git a/drivers/virt/nitro_enclaves/ne_misc_dev.h b/drivers/virt/nitro_enclaves/ne_misc_dev.h
index 57e20fe3ee1da..0df403f0080e5 100644
--- a/drivers/virt/nitro_enclaves/ne_misc_dev.h
+++ b/drivers/virt/nitro_enclaves/ne_misc_dev.h
@@ -117,18 +117,23 @@ struct ne_mem_region {
  *			     enclave.
  * @list_entry:		Entry in the list of the enclave's devices.
  * @num_vqs:		Number of entries of @notify_index.
+ * @info_notify_index:	Notification the enclave raises when it has changed
+ *			the info page of the device.
  * @notify_index:	Notify index of each virtqueue of the device.
  */
 struct ne_device_wiring {
 	struct list_head	list_entry;
 	u32			num_vqs;
+	u32			info_notify_index;
 	u32			notify_index[NE_MAX_VQS_PER_DEVICE];
 };
 
 /**
- * struct ne_notify_binding - Entry in the list of enclave virtqueue bindings.
- * @notify_binding_list_entry:	Entry in the list of enclave queue bindings.
- * @notify_index:		Notify index of the bound queue.
+ * struct ne_notify_binding - Entry in the list of enclave notification
+ *			      bindings.
+ * @notify_binding_list_entry:	Entry in the list of enclave notification
+ *				bindings.
+ * @notify_index:		Notify index of the bound notification.
  */
 struct ne_notify_binding {
 	struct list_head	notify_binding_list_entry;
@@ -162,8 +167,8 @@ struct ne_notify_binding {
  *				so that the device keeps the page after user space
  *				has closed the descriptor. NULL until
  *				%NE_SET_INFO_PAGE is called.
- * @notify_bindings:		List of the virtqueues this enclave has bound an
- *				eventfd to.
+ * @notify_bindings:		List of the notifications this enclave has bound
+ *				an eventfd to.
  * @devices:			List of struct ne_device_wiring, one for each device
  *				%NE_ADD_DEVICE gave the enclave. The notify indices
  *				in it are the only ones the enclave may bind an
diff --git a/drivers/virt/nitro_enclaves/ne_pci_dev.c b/drivers/virt/nitro_enclaves/ne_pci_dev.c
index a079abb2b4618..cedf828240214 100644
--- a/drivers/virt/nitro_enclaves/ne_pci_dev.c
+++ b/drivers/virt/nitro_enclaves/ne_pci_dev.c
@@ -554,7 +554,8 @@ static void ne_teardown_msix(struct pci_dev *pdev)
 }
 
 /**
- * ne_mux_binding_rcu_free() - RCU callback freeing one enclave queue binding.
+ * ne_mux_binding_rcu_free() - RCU callback freeing one enclave notification
+ *			       binding.
  * @head:	RCU head of the binding to free.
  *
  * Context: Softirq or process context.
diff --git a/drivers/virt/nitro_enclaves/ne_pci_dev.h b/drivers/virt/nitro_enclaves/ne_pci_dev.h
index 0dbce83cb06bd..18aafe8111a67 100644
--- a/drivers/virt/nitro_enclaves/ne_pci_dev.h
+++ b/drivers/virt/nitro_enclaves/ne_pci_dev.h
@@ -559,8 +559,11 @@ struct slot_pool_mem_donate_req {
  * @add_device.device_uid:	SLOT_ADD_DEVICE. Unique id of the new device.
  * @add_device.num_vqs:		SLOT_ADD_DEVICE. Number of entries of wiring
  *				the device wrote to the info page.
- * @add_device.reserved12:	SLOT_ADD_DEVICE. Bytes owned by other
- *				commands.
+ * @add_device.info_notify_index:	SLOT_ADD_DEVICE. Notification the
+ *				enclave raises when it has changed the info page
+ *				of the device.
+ * @add_device.reserved14:	SLOT_ADD_DEVICE. Bytes the device writes as
+ *				zero.
  * @add_device.vq_wiring_offset:	SLOT_ADD_DEVICE. Offset into the info
  *				page at which @num_vqs entries of struct
  *				ne_vq_wiring start.
@@ -602,7 +605,8 @@ struct ne_pci_dev_cmd_reply {
 		struct {
 			u64	device_uid;
 			u32	num_vqs;
-			u32	reserved12;
+			u16	info_notify_index;
+			u16	reserved14;
 			u64	vq_wiring_offset;
 		} add_device;
 	};
@@ -616,6 +620,10 @@ static_assert(offsetof(struct ne_pci_dev_cmd_reply, alloc.mem_regions) == 32);
 static_assert(offsetof(struct ne_pci_dev_cmd_reply, info.slot_uid) == 8);
 static_assert(offsetof(struct ne_pci_dev_cmd_reply, add_device.device_uid) == 8);
 static_assert(offsetof(struct ne_pci_dev_cmd_reply, add_device.num_vqs) == 16);
+static_assert(offsetof(struct ne_pci_dev_cmd_reply,
+		       add_device.info_notify_index) == 20);
+static_assert(offsetof(struct ne_pci_dev_cmd_reply,
+		       add_device.reserved14) == 22);
 static_assert(offsetof(struct ne_pci_dev_cmd_reply,
 		       add_device.vq_wiring_offset) == 24);
 static_assert(offsetof(struct ne_pci_dev_cmd_reply, info.enclave_cid) == 16);
@@ -677,8 +685,9 @@ struct ne_mux_vec {
 };
 
 /**
- * struct ne_mux_binding - Eventfd bound to one enclave queue.
- * @ctx:			Eventfd signalled when the queue is kicked.
+ * struct ne_mux_binding - Eventfd bound to one enclave notification.
+ * @ctx:			Eventfd signalled when the notification is
+ *				raised.
  * @rcu:			Head used to free the binding after a grace
  *				period, so that an interrupt handler which
  *				already loaded it can still dereference it.
@@ -707,8 +716,8 @@ struct ne_mux_binding {
  *				capability, and with it the multiplexer and slot
  *				pool registers.
  * @iomem_base :		MMIO region of the PCI device.
- * @notify_xa:			Enclave queue bindings, keyed by the notify index of
- *				the queue.
+ * @notify_xa:			Enclave notification bindings, keyed by notify
+ *				index.
  * @mux:			Per-pooled-vector interrupt multiplexer state.
  * @mux_nr_vecs:		Number of MSI-X vectors in the interrupt
  *				multiplexer pool.
@@ -789,9 +798,10 @@ int ne_do_request(struct pci_dev *pdev, enum ne_pci_dev_cmd_type cmd_type,
 		  size_t cmd_reply_size);
 
 /**
- * ne_mux_binding_free_rcu() - Free an enclave queue binding after a grace
- *			       period.
- * @binding:	Binding that has been removed from the queue bindings xarray.
+ * ne_mux_binding_free_rcu() - Free an enclave notification binding after a
+ *			       grace period.
+ * @binding:	Binding that has been removed from the notification bindings
+ *		xarray.
  *
  * Context: Process context.
  */
diff --git a/include/uapi/linux/nitro_enclaves.h b/include/uapi/linux/nitro_enclaves.h
index ff1b32e380e7b..4f2ca77f1b827 100644
--- a/include/uapi/linux/nitro_enclaves.h
+++ b/include/uapi/linux/nitro_enclaves.h
@@ -590,6 +590,9 @@ struct ne_shm_region {
  * @device_uid:		Id the device gave the new device (out).
  * @vq_info:		How the enclave reaches the device's first @num_vqs
  *			virtqueues (out).
+ * @info_notify_index:	Notification raised when the device has updated its
+ *			entry in the info page (out).
+ * @padding2:		Must be zero.
  */
 struct ne_add_device_args {
 	__u32	flags;
@@ -603,6 +606,8 @@ struct ne_add_device_args {
 	__u64	shm_regions_ptr;
 	__u64	device_uid;
 	struct ne_vq_info vq_info[NE_MAX_VQS];
+	__u32	info_notify_index;
+	__u32	padding2;
 };
 
 /**
@@ -636,6 +641,12 @@ struct ne_add_device_args {
  *		   reads them from there and returns them in @vq_info, so a
  *		   caller learns the interrupt vector and the doorbell offset
  *		   of every queue from this call.
+ *		   One more notification is raised when the device has updated
+ *		   its entry in the info page, and @info_notify_index names it,
+ *		   so %NE_SET_NOTIFY_EVENTFD binds an eventfd to it the same way
+ *		   it binds one to a queue.
+ *		   An argument that ends at @vq_info is taken and gets no
+ *		   @info_notify_index back.
  *		   The ioctl can be invoked on the enclave fd, before an
  *		   enclave is started.
  *
@@ -648,7 +659,8 @@ struct ne_add_device_args {
  * * EFAULT				- Argument read or write failure.
  * * EBADF				- A region's @fd is not an open file
  *					  descriptor.
- * * EINVAL				- The argument is shorter than @vq_info, or
+ * * EINVAL				- The argument does not reach the end of
+ *					  @vq_info, or
  *					  @padding is not zero, or @num_vqs is
  *					  more than %NE_MAX_VQS, or @config_size
  *					  is more than @device_config holds, or
@@ -662,7 +674,8 @@ struct ne_add_device_args {
  *					  made for another enclave, or it is
  *					  %NE_MEMORY_PRIVATE, or the regions are
  *					  made of more runs in total than the
- *					  device takes.
+ *					  device takes, or @padding2 is not
+ *					  zero.
  * * EIO				- Current task mm is not the same as the one
  *					  that created the enclave.
  * * E2BIG				- The argument is longer than this kernel knows
@@ -1063,29 +1076,35 @@ struct ne_enclave_start_info {
 
 /**
  * NE_EVENTFD_UNBIND - Value of an eventfd field that releases the binding the
- *		       caller holds, instead of naming an eventfd to bind. A kick
- *		       the enclave raises as the release lands is dropped, so a
- *		       caller that binds the queue again gets no signal for it.
- *		       Any other negative value names no open file and is an
- *		       error.
+ *		       caller holds, instead of naming an eventfd to bind. A
+ *		       notification the enclave raises as the release lands is
+ *		       dropped, so a caller that binds it again gets no signal
+ *		       for it. Any other negative value names no open file and
+ *		       is an error.
  */
 #define NE_EVENTFD_UNBIND		(-1)
 
 /**
- * NE_SET_NOTIFY_EVENTFD - The command is used to bind an eventfd to the
- *			   interrupt of one enclave virtqueue. When the enclave
- *			   kicks the queue, the NE PCI device raises the
- *			   interrupt the queue shares and the driver signals the
- *			   eventfd. A queue the caller has already bound is
- *			   rebound to the new eventfd, and the driver drops its
- *			   reference on the one it replaces. A kick that is
+ * NE_SET_NOTIFY_EVENTFD - The command is used to bind an eventfd to one of the
+ *			   notifications an enclave raises, which is the kick of
+ *			   one of its virtqueues or an update to a device's
+ *			   entry in the info page. When the enclave raises the
+ *			   notification, the NE PCI device raises the interrupt
+ *			   the notification shares and the driver signals the
+ *			   eventfd. A notification the caller has already bound
+ *			   is rebound to the new eventfd, and the driver drops
+ *			   its reference on the one it replaces. A notification
  *			   already being delivered when the rebind lands is
  *			   signalled on the outgoing eventfd, so a caller that
- *			   rebinds a queue an enclave is still kicking can see
- *			   one last kick arrive on the eventfd it replaced.
- *			   %NE_EVENTFD_UNBIND
- *			   releases the binding the caller holds on the queue.
- *			   The ioctl can be invoked on the enclave fd.
+ *			   rebinds a notification an enclave is still raising can
+ *			   see one last signal arrive on the eventfd it replaced.
+ *			   %NE_EVENTFD_UNBIND releases the binding the caller
+ *			   holds on the notification. The ioctl can be invoked
+ *			   on the enclave fd.
+ *			   @notify_index names a notification across the
+ *			   whole NE PCI device, and an enclave fd binds only
+ *			   the indices %NE_ADD_DEVICE reported to its enclave,
+ *			   in @vq_info or in @info_notify_index.
  *
  * Context: Process context.
  * Return:
@@ -1107,20 +1126,20 @@ struct ne_enclave_start_info {
  * * E2BIG				- The argument is longer than this kernel knows
  *					  about and the excess is not all zero.
  * * ENOMEM				- Memory allocation failure.
- * * EEXIST				- Another enclave holds a binding on the queue.
- * * ENOENT				- The caller holds no binding on the queue to
- *					  release.
+ * * EEXIST				- Another enclave holds a binding on the
+ *					  notification.
+ * * ENOENT				- The caller holds no binding on the notification
+ *					  to release.
  */
 #define NE_SET_NOTIFY_EVENTFD		_IOW(0xAE, 0x28, struct ne_notify_eventfd)
 
 /**
- * struct ne_notify_eventfd - Eventfd to signal on an enclave virtqueue kick
- *			      (in).
- * @fd:			Eventfd to signal when the queue is kicked, or
- *			%NE_EVENTFD_UNBIND to release the binding on the
- *			queue (in).
- * @notify_index:	Index the enclave writes to notify the NE PCI device of
- *			the virtqueue (in).
+ * struct ne_notify_eventfd - Eventfd to signal on an enclave notification (in).
+ * @fd:			Eventfd to signal when the notification is raised, or
+ *			%NE_EVENTFD_UNBIND to release the binding on it
+ *			(in).
+ * @notify_index:	Index the enclave writes to raise the notification with
+ *			the NE PCI device (in).
  */
 struct ne_notify_eventfd {
 	__s32	fd;
-- 
2.47.1

[PATCH 30/40] nitro_enclaves: Count multiplexed interrupts under debugfs

[Alexander Graf]

To debug event delivery of Nitro Sandboxes, count what the multiplexer
handler and a bind do, and report it under debugfs:

  /sys/kernel/debug/nitro_enclaves/mux_vectors
  /sys/kernel/debug/nitro_enclaves/notifications

Signed-off-by: Alexander Graf <graf@amazon.com>
---
 drivers/virt/nitro_enclaves/ne_misc_dev.c |  9 ++-
 drivers/virt/nitro_enclaves/ne_pci_dev.c  | 84 +++++++++++++++++++++++
 drivers/virt/nitro_enclaves/ne_pci_dev.h  | 19 +++++
 3 files changed, 110 insertions(+), 2 deletions(-)

diff --git a/drivers/virt/nitro_enclaves/ne_misc_dev.c b/drivers/virt/nitro_enclaves/ne_misc_dev.c
index d47fc8b514bcc..80ca23cd18251 100644
--- a/drivers/virt/nitro_enclaves/ne_misc_dev.c
+++ b/drivers/virt/nitro_enclaves/ne_misc_dev.c
@@ -2075,10 +2075,15 @@ static int ne_set_notify_eventfd(struct ne_enclave *ne_enclave,
 	 * a bit with an atomic read-modify-write on the same word, so exactly
 	 * one of the two consumes it. The bitmap words are the width the device
 	 * writes them in, which is the width the bit numbering of the handler
-	 * is in as well.
+	 * is in as well. Count the bit as delivered here, so that a bit this path
+	 * takes is counted where it is taken the way one the handler takes is.
 	 */
-	if (test_and_clear_bit(bit, (unsigned long *)ne_pci_dev->mux[vec_rel].bitmap))
+	if (test_and_clear_bit(bit, (unsigned long *)ne_pci_dev->mux[vec_rel].bitmap)) {
+		atomic64_inc(&ne_pci_dev->mux[vec_rel].nr_delivered);
+		atomic64_inc(&binding->nr_signals);
+
 		eventfd_signal(binding->ctx);
+	}
 
 	mutex_unlock(&ne_enclave->enclave_info_mutex);
 
diff --git a/drivers/virt/nitro_enclaves/ne_pci_dev.c b/drivers/virt/nitro_enclaves/ne_pci_dev.c
index cedf828240214..4678799998f4a 100644
--- a/drivers/virt/nitro_enclaves/ne_pci_dev.c
+++ b/drivers/virt/nitro_enclaves/ne_pci_dev.c
@@ -9,6 +9,7 @@
 
 #include <linux/bitops.h>
 #include <linux/cma.h>
+#include <linux/debugfs.h>
 #include <linux/delay.h>
 #include <linux/device.h>
 #include <linux/err.h>
@@ -23,6 +24,7 @@
 #include <linux/nitro_enclaves.h>
 #include <linux/nodemask.h>
 #include <linux/pci.h>
+#include <linux/seq_file.h>
 #include <linux/sizes.h>
 #include <linux/slab.h>
 #include <linux/sysfs.h>
@@ -649,6 +651,9 @@ static bool ne_mux_scan(struct ne_mux_vec *mux_vec)
 			if (!binding)
 				continue;
 
+			atomic64_inc(&mux_vec->nr_delivered);
+			atomic64_inc(&binding->nr_signals);
+
 			eventfd_signal(binding->ctx);
 		}
 	}
@@ -670,6 +675,8 @@ static irqreturn_t ne_mux_handler(int irq, void *data)
 	struct ne_mux_vec *mux_vec = data;
 	u32 pass = 0;
 
+	atomic64_inc(&mux_vec->nr_interrupts);
+
 	/*
 	 * xa_load() drops its own RCU read-side before it returns, so take one
 	 * here to keep a binding alive for as long as it is dereferenced.
@@ -698,6 +705,66 @@ static irqreturn_t ne_mux_handler(int irq, void *data)
 	return IRQ_HANDLED;
 }
 
+/*
+ * The multiplexer's debugfs files, under /sys/kernel/debug/nitro_enclaves/.
+ *
+ *   mux_vectors		one row per pooled MSI-X vector
+ *   notifications		one row per bound enclave notification
+ *
+ * A row is rendered without serialising against the handler or against a bind,
+ * so it holds a snapshot of each counter and not one of the pool.
+ */
+
+static int ne_mux_vectors_show(struct seq_file *s, void *unused)
+{
+	struct ne_pci_dev *ne_pci_dev = s->private;
+	u32 i = 0;
+
+	seq_puts(s, "vector msix interrupts delivered max_bit\n");
+
+	for (i = 0; i < ne_pci_dev->mux_nr_vecs; i++) {
+		struct ne_mux_vec *mux_vec = &ne_pci_dev->mux[i];
+
+		seq_printf(s, "%6u %4u %10lld %9lld %7u\n",
+			   i, ne_pci_dev->mux_vec_base + i,
+			   atomic64_read(&mux_vec->nr_interrupts),
+			   atomic64_read(&mux_vec->nr_delivered),
+			   READ_ONCE(mux_vec->max_bit));
+	}
+
+	return 0;
+}
+DEFINE_SHOW_ATTRIBUTE(ne_mux_vectors);
+
+static int ne_notifications_show(struct seq_file *s, void *unused)
+{
+	struct ne_pci_dev *ne_pci_dev = s->private;
+	struct ne_mux_binding *binding = NULL;
+	unsigned long notify_index = 0;
+
+	seq_puts(s, "index vector bit signals\n");
+
+	/*
+	 * A replaced or erased binding is freed after a grace period by
+	 * ne_mux_binding_free_rcu(), so hold the RCU read side across the walk
+	 * the way ne_mux_handler() does. seq_printf() renders into the
+	 * seq_file buffer and does not sleep.
+	 */
+	rcu_read_lock();
+
+	xa_for_each(&ne_pci_dev->notify_xa, notify_index, binding)
+		seq_printf(s, "%5lu %6lu %3lu %7lld\n",
+			   notify_index,
+			   notify_index % ne_pci_dev->mux_nr_vecs,
+			   notify_index / ne_pci_dev->mux_nr_vecs,
+			   atomic64_read(&binding->nr_signals));
+
+	rcu_read_unlock();
+
+	return 0;
+}
+DEFINE_SHOW_ATTRIBUTE(ne_notifications);
+
 /**
  * ne_mux_teardown() - Take the interrupt multiplexer bitmaps back from the
  *		       device.
@@ -712,6 +779,16 @@ static void ne_mux_teardown(struct pci_dev *pdev)
 	unsigned long notify_index = 0;
 	u32 i = 0;
 
+	/*
+	 * The walk in ne_notifications_show() is kept safe by the RCU read
+	 * side against the deferred free, and not against the loop below,
+	 * which frees the surviving bindings outright. Take the files away
+	 * first: debugfs_remove_recursive() waits for a reader that is already
+	 * in one.
+	 */
+	debugfs_remove_recursive(ne_pci_dev->debugfs_dir);
+	ne_pci_dev->debugfs_dir = NULL;
+
 	if (!ne_pci_dev->mux)
 		return;
 
@@ -864,6 +941,13 @@ static int ne_mux_setup(struct pci_dev *pdev)
 		ne_pci_dev->mux[i].irq_requested = true;
 	}
 
+	ne_pci_dev->debugfs_dir = debugfs_create_dir("nitro_enclaves", NULL);
+
+	debugfs_create_file("mux_vectors", 0400, ne_pci_dev->debugfs_dir,
+			    ne_pci_dev, &ne_mux_vectors_fops);
+	debugfs_create_file("notifications", 0400, ne_pci_dev->debugfs_dir,
+			    ne_pci_dev, &ne_notifications_fops);
+
 	return 0;
 
 teardown_mux:
diff --git a/drivers/virt/nitro_enclaves/ne_pci_dev.h b/drivers/virt/nitro_enclaves/ne_pci_dev.h
index 18aafe8111a67..71aaa8f557679 100644
--- a/drivers/virt/nitro_enclaves/ne_pci_dev.h
+++ b/drivers/virt/nitro_enclaves/ne_pci_dev.h
@@ -669,6 +669,11 @@ struct ne_slot_pool_run {
  *				bound to, bounding the scan of @bitmap. It only
  *				rises, so a scan reads the words of bindings that
  *				are gone as well.
+ * @nr_interrupts:		Interrupts this vector's handler has taken.
+ * @nr_delivered:		Bits it has taken out of @bitmap and delivered
+ *				to an eventfd. An interrupt can deliver several
+ *				bits, and one that finds no bit it can deliver
+ *				delivers none.
  * @ne_pci_dev:			NE PCI device this vector belongs to.
  * @vec_rel:			Index of this vector within the pool.
  * @irq_requested:		Whether the interrupt handler of this vector is
@@ -679,6 +684,8 @@ struct ne_slot_pool_run {
 struct ne_mux_vec {
 	u64			*bitmap;
 	u32			max_bit;
+	atomic64_t		nr_interrupts;
+	atomic64_t		nr_delivered;
 	struct ne_pci_dev	*ne_pci_dev;
 	u32			vec_rel;
 	bool			irq_requested;
@@ -691,10 +698,13 @@ struct ne_mux_vec {
  * @rcu:			Head used to free the binding after a grace
  *				period, so that an interrupt handler which
  *				already loaded it can still dereference it.
+ * @nr_signals:			Times @ctx has been signalled for this
+ *				notification.
  */
 struct ne_mux_binding {
 	struct eventfd_ctx	*ctx;
 	struct rcu_head		rcu;
+	atomic64_t		nr_signals;
 };
 
 /**
@@ -706,6 +716,14 @@ struct ne_mux_binding {
  *				the device after that.
  * @cmd_reply_wait_q:		Wait queue for handling command reply from the
  *				PCI device.
+ * @debugfs_dir:		Directory holding the multiplexer's debugfs
+ *				files. NULL until the multiplexer pool is set up
+ *				and once it is torn down. debugfs_create_dir()
+ *				answers with an error pointer rather than NULL
+ *				where debugfs is not built in or the directory
+ *				could not be made, and the calls that take it
+ *				turn such a pointer down themselves, so it is
+ *				kept as it came.
  * @enclaves_list:		List of the enclaves managed by the PCI device.
  * @enclaves_list_mutex:	Mutex for accessing the list of enclaves.
  * @event_wq:			Work queue for handling out-of-band events
@@ -747,6 +765,7 @@ struct ne_pci_dev {
 	atomic_t		cmd_reply_avail;
 	bool			failed;
 	wait_queue_head_t	cmd_reply_wait_q;
+	struct dentry		*debugfs_dir;
 	struct list_head	enclaves_list;
 	struct mutex		enclaves_list_mutex;
 	struct workqueue_struct	*event_wq;
-- 
2.47.1

[PATCH 31/40] nitro_enclaves: Map the doorbell window at probe

[Alexander Graf]

In preparation to support notifying a sandbox virtqueue from the parent,
map the doorbell window of the NE PCI device. The doorbell window is a
second MMIO BAR of the device holding an array of two-byte doorbells; a
write to the doorbell of a virtqueue tells the device that the queue has
data waiting for the sandbox.

A device that does not offer the window serves every enclave operation
that does not involve a virtqueue, so keep the mapping optional and let
probe carry on without it. Record the length of the window so that a
doorbell offset can be bounded against it.

Signed-off-by: Alexander Graf <graf@amazon.com>
---
 drivers/virt/nitro_enclaves/ne_pci_dev.c | 17 +++++++++++++++++
 drivers/virt/nitro_enclaves/ne_pci_dev.h |  9 +++++++++
 2 files changed, 26 insertions(+)

diff --git a/drivers/virt/nitro_enclaves/ne_pci_dev.c b/drivers/virt/nitro_enclaves/ne_pci_dev.c
index 4678799998f4a..b5a7ecfffb4b6 100644
--- a/drivers/virt/nitro_enclaves/ne_pci_dev.c
+++ b/drivers/virt/nitro_enclaves/ne_pci_dev.c
@@ -1968,6 +1968,15 @@ static int ne_pci_probe(struct pci_dev *pdev, const struct pci_device_id *id)
 		goto release_pci_regions;
 	}
 
+	/*
+	 * A device without a doorbell window serves every enclave operation
+	 * that does not involve a virtqueue, so carry on without one.
+	 */
+	ne_pci_dev->doorbell_base = pci_iomap(pdev, PCI_BAR_NE_DOORBELL, 0);
+	if (ne_pci_dev->doorbell_base)
+		ne_pci_dev->doorbell_len =
+			pci_resource_len(pdev, PCI_BAR_NE_DOORBELL);
+
 	pci_set_drvdata(pdev, ne_pci_dev);
 
 	atomic_set(&ne_pci_dev->cmd_reply_avail, 0);
@@ -2086,6 +2095,8 @@ static int ne_pci_probe(struct pci_dev *pdev, const struct pci_device_id *id)
 	ne_teardown_msix(pdev);
 iounmap_pci_bar:
 	pci_set_drvdata(pdev, NULL);
+	if (ne_pci_dev->doorbell_base)
+		pci_iounmap(pdev, ne_pci_dev->doorbell_base);
 	pci_iounmap(pdev, ne_pci_dev->iomem_base);
 release_pci_regions:
 	pci_release_regions(pdev);
@@ -2127,6 +2138,9 @@ static void ne_pci_remove(struct pci_dev *pdev)
 
 	pci_set_drvdata(pdev, NULL);
 
+	if (ne_pci_dev->doorbell_base)
+		pci_iounmap(pdev, ne_pci_dev->doorbell_base);
+
 	pci_iounmap(pdev, ne_pci_dev->iomem_base);
 
 	pci_release_regions(pdev);
@@ -2171,6 +2185,9 @@ static void ne_pci_shutdown(struct pci_dev *pdev)
 
 	pci_set_drvdata(pdev, NULL);
 
+	if (ne_pci_dev->doorbell_base)
+		pci_iounmap(pdev, ne_pci_dev->doorbell_base);
+
 	pci_iounmap(pdev, ne_pci_dev->iomem_base);
 
 	pci_release_regions(pdev);
diff --git a/drivers/virt/nitro_enclaves/ne_pci_dev.h b/drivers/virt/nitro_enclaves/ne_pci_dev.h
index 71aaa8f557679..5a92e0c565ba0 100644
--- a/drivers/virt/nitro_enclaves/ne_pci_dev.h
+++ b/drivers/virt/nitro_enclaves/ne_pci_dev.h
@@ -31,6 +31,10 @@
  * PCI_BAR_NE - Nitro Enclaves PCI device MMIO BAR.
  */
 #define PCI_BAR_NE		(0x03)
+/**
+ * PCI_BAR_NE_DOORBELL - Nitro Enclaves PCI device virtqueue doorbell BAR.
+ */
+#define PCI_BAR_NE_DOORBELL	(0x04)
 
 /**
  * DOC: Device registers in the NE PCI device MMIO BAR
@@ -724,6 +728,9 @@ struct ne_mux_binding {
  *				could not be made, and the calls that take it
  *				turn such a pointer down themselves, so it is
  *				kept as it came.
+ * @doorbell_base:		Virtqueue doorbell window of the PCI device, NULL
+ *				on a device that does not offer one.
+ * @doorbell_len:		Length of the virtqueue doorbell window.
  * @enclaves_list:		List of the enclaves managed by the PCI device.
  * @enclaves_list_mutex:	Mutex for accessing the list of enclaves.
  * @event_wq:			Work queue for handling out-of-band events
@@ -766,6 +773,8 @@ struct ne_pci_dev {
 	bool			failed;
 	wait_queue_head_t	cmd_reply_wait_q;
 	struct dentry		*debugfs_dir;
+	void __iomem		*doorbell_base;
+	resource_size_t		doorbell_len;
 	struct list_head	enclaves_list;
 	struct mutex		enclaves_list_mutex;
 	struct workqueue_struct	*event_wq;
-- 
2.47.1

[PATCH 32/40] nitro_enclaves: Add NE_SET_VRING_CALL

[Alexander Graf]

Nitro Sandboxes let a device backend in the parent tell the sandbox that
a virtqueue has data waiting. Give user space a path to trigger this by
signalling an eventfd. The driver then turns that signal into a write
to the doorbell of the queue.

Add NE_SET_VRING_CALL, which binds an eventfd to a doorbell offset that
NE_ADD_DEVICE reported to the same enclave.

Signed-off-by: Alexander Graf <graf@amazon.com>
---
 drivers/virt/nitro_enclaves/ne_misc_dev.c | 357 +++++++++++++++++++++-
 drivers/virt/nitro_enclaves/ne_misc_dev.h |  33 +-
 include/uapi/linux/nitro_enclaves.h       |  52 ++++
 3 files changed, 434 insertions(+), 8 deletions(-)

diff --git a/drivers/virt/nitro_enclaves/ne_misc_dev.c b/drivers/virt/nitro_enclaves/ne_misc_dev.c
index 80ca23cd18251..e2ff807ec673e 100644
--- a/drivers/virt/nitro_enclaves/ne_misc_dev.c
+++ b/drivers/virt/nitro_enclaves/ne_misc_dev.c
@@ -2136,6 +2136,328 @@ static void ne_enclave_remove_all_notify_binding_entries(struct ne_enclave *ne_e
 	}
 }
 
+/**
+ * ne_doorbell_write() - Ring the doorbell of the virtqueue of a binding.
+ * @binding:	Doorbell binding with its doorbell address set.
+ *
+ * Context: Process context or interrupt context.
+ */
+static void ne_doorbell_write(struct ne_doorbell_binding *binding)
+{
+	iowrite16(binding->offset / sizeof(u16), binding->doorbell);
+}
+
+/**
+ * ne_doorbell_ring() - Ring the doorbell of the virtqueue an eventfd is bound
+ *			to.
+ * @wait:	Wait queue entry of the binding, sitting on the eventfd.
+ * @mode:	Wake up mode, unused.
+ * @sync:	Wake up sync flag, unused.
+ * @key:	Poll events the eventfd has woken up for.
+ *
+ * Context: Process context or interrupt context.
+ * Return: 0, so that the entry keeps sitting on the eventfd.
+ */
+static int ne_doorbell_ring(wait_queue_entry_t *wait, unsigned int mode,
+			    int sync, void *key)
+{
+	struct ne_doorbell_binding *binding = NULL;
+	u64 cnt = 0;
+
+	if (!(key_to_poll(key) & EPOLLIN))
+		return 0;
+
+	binding = container_of(wait, struct ne_doorbell_binding, wait);
+
+	eventfd_ctx_do_read(binding->ctx, &cnt);
+
+	ne_doorbell_write(binding);
+
+	return 0;
+}
+
+/**
+ * ne_doorbell_queue_proc() - Put the wait queue entry of a doorbell binding on
+ *			      the wait queue of its eventfd.
+ * @file:	Eventfd the binding is being armed on.
+ * @wqh:	Wait queue head of the eventfd.
+ * @pt:		Poll table of the binding.
+ *
+ * Context: Process context.
+ */
+static void ne_doorbell_queue_proc(struct file *file, wait_queue_head_t *wqh,
+				   poll_table *pt)
+{
+	struct ne_doorbell_binding *binding = NULL;
+
+	binding = container_of(pt, struct ne_doorbell_binding, pt);
+
+	binding->wqh = wqh;
+
+	add_wait_queue(wqh, &binding->wait);
+}
+
+/**
+ * ne_doorbell_arm() - Have a signal of an eventfd ring the doorbell of a
+ *		       binding.
+ * @binding:	Doorbell binding with its eventfd and doorbell address set.
+ * @file:	Eventfd of the binding.
+ *
+ * Context: Process context. This function is called with the ne_enclave mutex
+ *	    held.
+ */
+static void ne_doorbell_arm(struct ne_doorbell_binding *binding,
+			    struct file *file)
+{
+	__poll_t mask = 0;
+
+	init_waitqueue_func_entry(&binding->wait, ne_doorbell_ring);
+	init_poll_funcptr(&binding->pt, ne_doorbell_queue_proc);
+
+	mask = vfs_poll(file, &binding->pt);
+
+	/*
+	 * Ring for a signal the eventfd was already carrying. The count stays
+	 * on the eventfd for the wake up path to take, which runs with the
+	 * lock eventfd_ctx_do_read() needs held.
+	 */
+	if (mask & EPOLLIN)
+		ne_doorbell_write(binding);
+}
+
+/**
+ * ne_doorbell_disarm() - Stop a signal of an eventfd from ringing the doorbell
+ *			  of a binding and give up the eventfd.
+ * @binding:	Armed doorbell binding.
+ *
+ * Context: Process context. This function is called with the ne_enclave mutex
+ *	    held.
+ */
+static void ne_doorbell_disarm(struct ne_doorbell_binding *binding)
+{
+	remove_wait_queue(binding->wqh, &binding->wait);
+
+	eventfd_ctx_put(binding->ctx);
+}
+
+/**
+ * ne_doorbell_binding_remove() - Remove one doorbell binding from the enclave
+ *				  that owns it.
+ * @binding:	Doorbell binding of the enclave.
+ *
+ * Context: Process context. This function is called with the ne_enclave mutex
+ *	    held.
+ */
+static void ne_doorbell_binding_remove(struct ne_doorbell_binding *binding)
+{
+	ne_doorbell_disarm(binding);
+
+	list_del(&binding->doorbell_binding_list_entry);
+
+	kfree(binding);
+}
+
+/**
+ * ne_enclave_remove_all_doorbell_binding_entries() - Remove all the doorbell
+ *						     binding entries from the
+ *						     enclave data structure.
+ * @ne_enclave:	Private data associated with the current enclave.
+ *
+ * Context: Process context. This function is called with the ne_enclave mutex
+ *	    held.
+ */
+static void ne_enclave_remove_all_doorbell_binding_entries(struct ne_enclave *ne_enclave)
+{
+	struct ne_doorbell_binding *binding = NULL;
+	struct ne_doorbell_binding *binding_tmp = NULL;
+
+	list_for_each_entry_safe(binding, binding_tmp,
+				 &ne_enclave->doorbell_bindings,
+				 doorbell_binding_list_entry)
+		ne_doorbell_binding_remove(binding);
+}
+
+/**
+ * ne_del_vring_call() - Release the eventfd bound to the doorbell of an enclave
+ *			 virtqueue.
+ * @ne_enclave:		Private data associated with the current enclave.
+ * @doorbell_offset:	Offset the NE PCI device gave the doorbell of the
+ *			virtqueue in the doorbell window.
+ *
+ * Context: Process context.
+ * Return:
+ * * 0 on success.
+ * * Negative return value on failure.
+ */
+static int ne_del_vring_call(struct ne_enclave *ne_enclave, u16 doorbell_offset)
+{
+	struct ne_doorbell_binding *binding = NULL;
+	struct ne_doorbell_binding *entry = NULL;
+	int rc;
+
+	rc = mutex_lock_killable(&ne_enclave->enclave_info_mutex);
+	if (rc)
+		return rc;
+
+	list_for_each_entry(entry, &ne_enclave->doorbell_bindings,
+			    doorbell_binding_list_entry) {
+		if (entry->offset == doorbell_offset) {
+			binding = entry;
+
+			break;
+		}
+	}
+
+	if (!binding) {
+		mutex_unlock(&ne_enclave->enclave_info_mutex);
+
+		return -ENOENT;
+	}
+
+	ne_doorbell_binding_remove(binding);
+
+	mutex_unlock(&ne_enclave->enclave_info_mutex);
+
+	return 0;
+}
+
+/**
+ * ne_enclave_owns_doorbell() - Check that NE_ADD_DEVICE reported a doorbell
+ *				offset to an enclave.
+ * @ne_enclave :	Private data associated with the current enclave.
+ * @offset:		Doorbell offset to look for.
+ *
+ * Context: Process context. This function is called with the ne_enclave mutex
+ *	    held.
+ * Return: true if one of the enclave's devices has the doorbell.
+ */
+static bool ne_enclave_owns_doorbell(struct ne_enclave *ne_enclave, u16 offset)
+{
+	struct ne_device_wiring *dev = NULL;
+	unsigned int i = 0;
+
+	list_for_each_entry(dev, &ne_enclave->devices, list_entry)
+		for (i = 0; i < dev->num_vqs; i++)
+			if (dev->doorbell_offset[i] == offset)
+				return true;
+
+	return false;
+}
+
+/**
+ * ne_set_vring_call() - Bind an eventfd to the doorbell of an enclave
+ *			 virtqueue, or release the binding on that doorbell.
+ * @ne_enclave:		Private data associated with the current enclave.
+ * @vring_call:	The doorbell and the eventfd, or the doorbell alone when the
+ *		eventfd is NE_EVENTFD_UNBIND.
+ *
+ * Context: Process context.
+ * Return:
+ * * 0 on success.
+ * * Negative return value on failure.
+ */
+static int ne_set_vring_call(struct ne_enclave *ne_enclave,
+			     struct ne_vring_call *vring_call)
+{
+	struct ne_pci_dev *ne_pci_dev = ne_devs.ne_pci_dev;
+	struct ne_doorbell_binding *binding = NULL;
+	struct ne_doorbell_binding *entry = NULL;
+	struct eventfd_ctx *ctx = NULL;
+	bool rebind = false;
+	int rc = -EINVAL;
+
+	if (vring_call->reserved)
+		return -EINVAL;
+
+	if (!ne_pci_dev->doorbell_base)
+		return -EOPNOTSUPP;
+
+	if (ne_pci_dev->doorbell_len < sizeof(u16) ||
+	    vring_call->doorbell_offset > ne_pci_dev->doorbell_len - sizeof(u16))
+		return -EINVAL;
+
+	/*
+	 * An unaligned MMIO access is undefined on some of the architectures
+	 * the driver builds for.
+	 */
+	if (!IS_ALIGNED(vring_call->doorbell_offset, sizeof(u16)))
+		return -EINVAL;
+
+	if (vring_call->fd == NE_EVENTFD_UNBIND)
+		return ne_del_vring_call(ne_enclave, vring_call->doorbell_offset);
+
+	/*
+	 * Take the eventfd from the file the fd resolves to and arm the
+	 * binding on that same file, so that the eventfd a signal arrives on
+	 * is the eventfd the binding rings for.
+	 */
+	CLASS(fd, f)(vring_call->fd);
+	if (fd_empty(f))
+		return -EBADF;
+
+	ctx = eventfd_ctx_fileget(fd_file(f));
+	if (IS_ERR(ctx))
+		return PTR_ERR(ctx);
+
+	rc = mutex_lock_killable(&ne_enclave->enclave_info_mutex);
+	if (rc)
+		goto put_ctx;
+
+	/*
+	 * The doorbell window spans the whole NE PCI device, so an enclave only
+	 * binds the doorbells NE_ADD_DEVICE reported to it.
+	 */
+	if (!ne_enclave_owns_doorbell(ne_enclave, vring_call->doorbell_offset)) {
+		rc = -EINVAL;
+
+		goto unlock_mutex;
+	}
+
+	list_for_each_entry(entry, &ne_enclave->doorbell_bindings,
+			    doorbell_binding_list_entry) {
+		if (entry->offset == vring_call->doorbell_offset) {
+			rebind = true;
+
+			break;
+		}
+	}
+
+	if (rebind) {
+		binding = entry;
+
+		ne_doorbell_disarm(binding);
+	} else {
+		binding = kzalloc_obj(*binding);
+		if (!binding) {
+			rc = -ENOMEM;
+
+			goto unlock_mutex;
+		}
+
+		binding->doorbell = ne_pci_dev->doorbell_base +
+				    vring_call->doorbell_offset;
+		binding->offset = vring_call->doorbell_offset;
+
+		list_add(&binding->doorbell_binding_list_entry,
+			 &ne_enclave->doorbell_bindings);
+	}
+
+	binding->ctx = ctx;
+
+	ne_doorbell_arm(binding, fd_file(f));
+
+	mutex_unlock(&ne_enclave->enclave_info_mutex);
+
+	return 0;
+
+unlock_mutex:
+	mutex_unlock(&ne_enclave->enclave_info_mutex);
+put_ctx:
+	eventfd_ctx_put(ctx);
+
+	return rc;
+}
+
 /**
  * ne_enclave_do_ioctl() - Ioctl function provided by the enclave file.
  * @file:	File associated with this ioctl function.
@@ -2609,6 +2931,22 @@ static long ne_enclave_do_ioctl(struct file *file, unsigned int cmd, unsigned lo
 		return ne_set_notify_eventfd(ne_enclave, &args);
 	}
 
+	case NE_EA_IOCTL(NE_SET_VRING_CALL): {
+		struct ne_vring_call __user *uargs = (void __user *)arg;
+		unsigned int usize = _IOC_SIZE(cmd);
+		struct ne_vring_call args;
+		int rc;
+
+		if (usize < offsetofend(struct ne_vring_call, doorbell_offset))
+			return -EINVAL;
+
+		rc = copy_struct_from_user(&args, sizeof(args), uargs, usize);
+		if (rc)
+			return rc;
+
+		return ne_set_vring_call(ne_enclave, &args);
+	}
+
 	default:
 		return -ENOTTY;
 	}
@@ -2796,12 +3134,15 @@ int ne_enclave_free_slot(struct ne_pci_dev *ne_pci_dev,
 			   &cmd_reply, sizeof(cmd_reply));
 
 	/*
-	 * The notify index of a queue belongs to the slot, so the device is free
-	 * to give it to another enclave once the slot is freed. Take the
-	 * bindings of this enclave out of the xarray the handler looks them up
-	 * in before that, or a bind for the reissued index finds the entry
-	 * still taken. This happens whether or not the stop went through,
-	 * because an unbind of the device tears the multiplexer down next.
+	 * The notify index of a queue and the offset of its doorbell belong to
+	 * the slot, so the device is free to give either to another enclave once
+	 * the slot is freed. Take the bindings of this enclave out of the xarray
+	 * the handler looks them up in, and off the eventfds that ring their
+	 * doorbells, before that. Otherwise a bind for the reissued index finds
+	 * the entry still taken, and a signal on an eventfd left armed rings a
+	 * doorbell the device has given away. This happens whether or not the
+	 * stop went through, because an unbind of the device tears the
+	 * multiplexer and the doorbell window down next.
 	 *
 	 * A handler that looked up a binding of this enclave before it was
 	 * erased here can still be running. Were the index reissued under it,
@@ -2815,6 +3156,8 @@ int ne_enclave_free_slot(struct ne_pci_dev *ne_pci_dev,
 		synchronize_rcu();
 	}
 
+	ne_enclave_remove_all_doorbell_binding_entries(ne_enclave);
+
 	if (rc < 0)
 		return rc;
 
@@ -3098,6 +3441,7 @@ static int ne_create_vm_ioctl(struct ne_pci_dev *ne_pci_dev, u64 flags,
 	ne_enclave->max_mem_regions = cmd_reply.alloc.mem_regions;
 	INIT_LIST_HEAD(&ne_enclave->contig_regions_list);
 	INIT_LIST_HEAD(&ne_enclave->device_regions_list);
+	INIT_LIST_HEAD(&ne_enclave->doorbell_bindings);
 	INIT_LIST_HEAD(&ne_enclave->mem_regions_list);
 	ne_enclave->mm = current->mm;
 	ne_enclave->slot_uid = cmd_reply.alloc.slot_uid;
@@ -4860,6 +5204,7 @@ static int ne_add_device_ioctl(struct ne_enclave *ne_enclave,
 		args->vq_info[i].doorbell_offset = wiring[i].doorbell_offset;
 
 		dev->notify_index[i] = wiring[i].notify_index;
+		dev->doorbell_offset[i] = wiring[i].doorbell_offset;
 	}
 
 	args->device_uid = cmd_reply.add_device.device_uid;
diff --git a/drivers/virt/nitro_enclaves/ne_misc_dev.h b/drivers/virt/nitro_enclaves/ne_misc_dev.h
index 0df403f0080e5..f9ef79fafa2be 100644
--- a/drivers/virt/nitro_enclaves/ne_misc_dev.h
+++ b/drivers/virt/nitro_enclaves/ne_misc_dev.h
@@ -7,12 +7,14 @@
 #define _NE_MISC_DEV_H_
 
 #include <linux/cpumask.h>
+#include <linux/eventfd.h>
 #include <linux/list.h>
 #include <linux/miscdevice.h>
 #include <linux/mm.h>
 #include <linux/mutex.h>
 #include <linux/pci.h>
 #include <linux/rwsem.h>
+#include <linux/poll.h>
 #include <linux/wait.h>
 
 #include "ne_pci_dev.h"
@@ -120,12 +122,14 @@ struct ne_mem_region {
  * @info_notify_index:	Notification the enclave raises when it has changed
  *			the info page of the device.
  * @notify_index:	Notify index of each virtqueue of the device.
+ * @doorbell_offset:	Doorbell offset of each virtqueue of the device.
  */
 struct ne_device_wiring {
 	struct list_head	list_entry;
 	u32			num_vqs;
 	u32			info_notify_index;
 	u32			notify_index[NE_MAX_VQS_PER_DEVICE];
+	u16			doorbell_offset[NE_MAX_VQS_PER_DEVICE];
 };
 
 /**
@@ -140,8 +144,32 @@ struct ne_notify_binding {
 	unsigned long		notify_index;
 };
 
+/**
+ * struct ne_doorbell_binding - Entry in the list of enclave doorbell bindings.
+ * @ctx:			Eventfd whose signal rings the doorbell.
+ * @doorbell:			Address of the doorbell in the doorbell window.
+ * @doorbell_binding_list_entry: Entry in the list of enclave doorbell
+ *				bindings.
+ * @offset:			Offset of the doorbell in the doorbell window.
+ * @pt:				Poll table that registers @wait on the eventfd.
+ * @wait:			Wait queue entry woken up when the eventfd is
+ *				signalled.
+ * @wqh:			Wait queue head of the eventfd @wait sits on.
+ */
+struct ne_doorbell_binding {
+	struct eventfd_ctx	*ctx;
+	void __iomem		*doorbell;
+	struct list_head	doorbell_binding_list_entry;
+	u16			offset;
+	poll_table		pt;
+	wait_queue_entry_t	wait;
+	wait_queue_head_t	*wqh;
+};
+
 /**
  * struct ne_enclave - Per-enclave data used for enclave lifetime management.
+ * @doorbell_bindings:		List of the virtqueue doorbells this enclave has
+ *				bound an eventfd to.
  * @enclave_info_mutex :	Mutex for accessing this internal state.
  * @detached:			The NE PCI device the enclave was created on is
  *				gone. Written with ne_devs.lock held for writing.
@@ -171,8 +199,8 @@ struct ne_notify_binding {
  *				an eventfd to.
  * @devices:			List of struct ne_device_wiring, one for each device
  *				%NE_ADD_DEVICE gave the enclave. The notify indices
- *				in it are the only ones the enclave may bind an
- *				eventfd to.
+ *				and doorbell offsets in it are the only ones the
+ *				enclave may bind an eventfd to.
  * @max_mem_regions:		The maximum number of memory regions that can be
  *				handled by the hypervisor.
  * @mem_regions_list:		Enclave user space memory regions list.
@@ -196,6 +224,7 @@ struct ne_notify_binding {
  * @vcpu_ids:			Cpumask of the vCPUs that are set for the enclave.
  */
 struct ne_enclave {
+	struct list_head	doorbell_bindings;
 	struct mutex		enclave_info_mutex;
 	bool			detached;
 	bool			slot_freed;
diff --git a/include/uapi/linux/nitro_enclaves.h b/include/uapi/linux/nitro_enclaves.h
index 4f2ca77f1b827..01bec4d072d6a 100644
--- a/include/uapi/linux/nitro_enclaves.h
+++ b/include/uapi/linux/nitro_enclaves.h
@@ -1146,4 +1146,56 @@ struct ne_notify_eventfd {
 	__u32	notify_index;
 };
 
+/**
+ * NE_SET_VRING_CALL - The command is used to bind an eventfd to the doorbell
+ *		       of one enclave virtqueue. When the eventfd is signalled,
+ *		       the driver rings the doorbell and the enclave learns that
+ *		       the queue has data waiting for it. The caller names the
+ *		       doorbell by the offset %NE_ADD_DEVICE reported for it in
+ *		       @vq_info, and the driver takes no other offset. A
+ *		       doorbell the caller has already bound is rebound to the
+ *		       new eventfd, and %NE_EVENTFD_UNBIND releases the binding
+ *		       the caller holds on the doorbell. The ioctl can be
+ *		       invoked on the enclave fd.
+ *
+ * Context: Process context.
+ * Return:
+ * * 0					- Logic successfully completed.
+ * * -1					- There was a failure in the ioctl logic.
+ * On failure, errno is set to:
+ * * ENODEV				- The NE PCI device is not available.
+ * * EFAULT				- Argument read failure.
+ * * EOPNOTSUPP				- The NE PCI device offers no doorbells.
+ * * EINVAL				- The argument is shorter than
+ *					  @doorbell_offset, the offset is not that
+ *					  of a doorbell in the window the device
+ *					  offers, %NE_ADD_DEVICE did not report
+ *					  it to this enclave, @reserved is not
+ *					  zero, or the given fd is not an eventfd.
+ * * EBADF				- There is no open file for the given fd, which
+ *					  is the case for every negative fd other
+ *					  than %NE_EVENTFD_UNBIND.
+ * * E2BIG				- The argument is longer than this kernel knows
+ *					  about and the excess is not all zero.
+ * * ENOMEM				- Memory allocation failure.
+ * * ENOENT				- The caller holds no binding on the doorbell to
+ *					  release.
+ */
+#define NE_SET_VRING_CALL		_IOW(0xAE, 0x27, struct ne_vring_call)
+
+/**
+ * struct ne_vring_call - Eventfd that rings an enclave virtqueue doorbell (in).
+ * @fd:			Eventfd whose signal rings the doorbell, or
+ *			%NE_EVENTFD_UNBIND to release the binding on the
+ *			doorbell (in).
+ * @doorbell_offset:	Offset the NE PCI device gave the doorbell of the
+ *			virtqueue in the doorbell window (in).
+ * @reserved:		Must be zero (in).
+ */
+struct ne_vring_call {
+	__s32	fd;
+	__u16	doorbell_offset;
+	__u16	reserved;
+};
+
 #endif /* _UAPI_LINUX_NITRO_ENCLAVES_H_ */
-- 
2.47.1

[PATCH 33/40] nitro_enclaves: Assert the layout of the ioctl argument structs

[Alexander Graf]

The structs the ioctls carry are the interface user space compiles
against, and a member that changes width or place breaks every caller
built before the change. Two members of the same width swapped for each
other leave sizeof alone, so a size check does not see it.

Assert the size of every struct in the uapi header along with the offset
of every one of its members. A transposition now stops the build at the
line that encodes the old layout.

Signed-off-by: Alexander Graf <graf@amazon.com>
---
 drivers/virt/nitro_enclaves/ne_misc_dev.c | 74 +++++++++++++++++++++++
 1 file changed, 74 insertions(+)

diff --git a/drivers/virt/nitro_enclaves/ne_misc_dev.c b/drivers/virt/nitro_enclaves/ne_misc_dev.c
index e2ff807ec673e..f3ad8ddc31e5b 100644
--- a/drivers/virt/nitro_enclaves/ne_misc_dev.c
+++ b/drivers/virt/nitro_enclaves/ne_misc_dev.c
@@ -80,6 +80,80 @@
  */
 static_assert(NE_NUMA_ANY == NUMA_NO_NODE);
 
+/*
+ * User space and the driver have to lay the ioctl interface structs out the
+ * same way. A size assert does not catch two members of the same width swapped
+ * for each other, so assert the offsets as well.
+ */
+static_assert(sizeof(struct ne_vq_info) == 8);
+static_assert(offsetof(struct ne_vq_info, msix_vector) == 0);
+static_assert(offsetof(struct ne_vq_info, notify_index) == 2);
+static_assert(offsetof(struct ne_vq_info, msix_entry) == 4);
+static_assert(offsetof(struct ne_vq_info, doorbell_offset) == 6);
+
+static_assert(sizeof(struct ne_shm_region) == 8);
+static_assert(offsetof(struct ne_shm_region, fd) == 0);
+static_assert(offsetof(struct ne_shm_region, shm_id) == 4);
+static_assert(offsetof(struct ne_shm_region, reserved) == 5);
+
+static_assert(sizeof(struct ne_add_device_args) == 688);
+static_assert(offsetof(struct ne_add_device_args, flags) == 0);
+static_assert(offsetof(struct ne_add_device_args, device_type) == 4);
+static_assert(offsetof(struct ne_add_device_args, num_vqs) == 8);
+static_assert(offsetof(struct ne_add_device_args, config_size) == 12);
+static_assert(offsetof(struct ne_add_device_args, num_shm_regions) == 16);
+static_assert(offsetof(struct ne_add_device_args, padding) == 20);
+static_assert(offsetof(struct ne_add_device_args, device_features) == 24);
+static_assert(offsetof(struct ne_add_device_args, device_config) == 88);
+static_assert(offsetof(struct ne_add_device_args, shm_regions_ptr) == 152);
+static_assert(offsetof(struct ne_add_device_args, device_uid) == 160);
+static_assert(offsetof(struct ne_add_device_args, vq_info) == 168);
+static_assert(offsetof(struct ne_add_device_args, info_notify_index) == 680);
+static_assert(offsetof(struct ne_add_device_args, padding2) == 684);
+
+static_assert(sizeof(struct ne_notify_eventfd) == 8);
+static_assert(offsetof(struct ne_notify_eventfd, fd) == 0);
+static_assert(offsetof(struct ne_notify_eventfd, notify_index) == 4);
+
+static_assert(sizeof(struct ne_vring_call) == 8);
+static_assert(offsetof(struct ne_vring_call, fd) == 0);
+static_assert(offsetof(struct ne_vring_call, doorbell_offset) == 4);
+static_assert(offsetof(struct ne_vring_call, reserved) == 6);
+
+static_assert(sizeof(struct ne_create_vm_args) == 16);
+static_assert(offsetof(struct ne_create_vm_args, flags) == 0);
+static_assert(offsetof(struct ne_create_vm_args, slot_uid) == 8);
+
+static_assert(sizeof(struct ne_add_any_vcpu_args) == 8);
+static_assert(offsetof(struct ne_add_any_vcpu_args, numa_node) == 0);
+static_assert(offsetof(struct ne_add_any_vcpu_args, vcpu_id) == 4);
+
+static_assert(sizeof(struct ne_alloc_memory_args) == 16);
+static_assert(offsetof(struct ne_alloc_memory_args, size) == 0);
+static_assert(offsetof(struct ne_alloc_memory_args, nid) == 8);
+static_assert(offsetof(struct ne_alloc_memory_args, flags) == 12);
+
+static_assert(sizeof(struct ne_add_memory_region_args) == 8);
+static_assert(offsetof(struct ne_add_memory_region_args, fd) == 0);
+static_assert(offsetof(struct ne_add_memory_region_args, flags) == 4);
+
+static_assert(sizeof(struct ne_set_info_page_args) == 8);
+static_assert(offsetof(struct ne_set_info_page_args, fd) == 0);
+static_assert(offsetof(struct ne_set_info_page_args, flags) == 4);
+
+static_assert(sizeof(struct ne_image_load_info) == 16);
+static_assert(offsetof(struct ne_image_load_info, flags) == 0);
+static_assert(offsetof(struct ne_image_load_info, memory_offset) == 8);
+
+static_assert(sizeof(struct ne_user_memory_region) == 24);
+static_assert(offsetof(struct ne_user_memory_region, flags) == 0);
+static_assert(offsetof(struct ne_user_memory_region, memory_size) == 8);
+static_assert(offsetof(struct ne_user_memory_region, userspace_addr) == 16);
+
+static_assert(sizeof(struct ne_enclave_start_info) == 16);
+static_assert(offsetof(struct ne_enclave_start_info, flags) == 0);
+static_assert(offsetof(struct ne_enclave_start_info, enclave_cid) == 8);
+
 /**
  * NE_EIF_LOAD_OFFSET - The offset where to copy the Enclave Image Format (EIF)
  *			image in enclave memory.
-- 
2.47.1

[PATCH 34/40] nitro_enclaves: Pass on the Nitro Sandbox launch flags

[Alexander Graf]

Nitro Sandboxes support multiple flags to enable or disable particular
bits of functionality. These bits are typically in sync with the EIF file
and need to get set already on slot creation, not only when we start the
sandbox.

Add definitions for the first batch of launch flags and make sure user
space can only set known flags:

NE_ENCLAVE_PCIE_VSOCK_MODE - Parent may add one single vsock device
NE_ENCLAVE_PCIE_VIRTIO_MODE - Parent may add (almost) any virtio device
NE_ENCLAVE_PCIE_NUMA_MODE - Slot may span multiple NUMA nodes

Signed-off-by: Alexander Graf <graf@amazon.com>
---
 drivers/virt/nitro_enclaves/ne_misc_dev.c |  2 +-
 drivers/virt/nitro_enclaves/ne_misc_dev.h |  8 +++++
 include/uapi/linux/nitro_enclaves.h       | 38 +++++++++++++++++++++++
 3 files changed, 47 insertions(+), 1 deletion(-)

diff --git a/drivers/virt/nitro_enclaves/ne_misc_dev.c b/drivers/virt/nitro_enclaves/ne_misc_dev.c
index f3ad8ddc31e5b..961b821d6038b 100644
--- a/drivers/virt/nitro_enclaves/ne_misc_dev.c
+++ b/drivers/virt/nitro_enclaves/ne_misc_dev.c
@@ -3402,7 +3402,7 @@ static int ne_create_vm_ioctl(struct ne_pci_dev *ne_pci_dev, u64 flags,
 	int rc = -EINVAL;
 	struct slot_alloc_req slot_alloc_req = {};
 
-	if (flags & ~(u64)NE_ENCLAVE_PCIE_MODE) {
+	if (flags & ~(u64)NE_ENCLAVE_FLAG_VALID_MASK) {
 		dev_err_ratelimited(ne_misc_dev.this_device,
 				    "Unknown launch flag bit set 0x%llx\n", flags);
 
diff --git a/drivers/virt/nitro_enclaves/ne_misc_dev.h b/drivers/virt/nitro_enclaves/ne_misc_dev.h
index f9ef79fafa2be..b0d0438157260 100644
--- a/drivers/virt/nitro_enclaves/ne_misc_dev.h
+++ b/drivers/virt/nitro_enclaves/ne_misc_dev.h
@@ -19,6 +19,14 @@
 
 #include "ne_pci_dev.h"
 
+/*
+ * NE_ENCLAVE_FLAG_VALID_MASK - Every launch flag this kernel passes on.
+ */
+#define NE_ENCLAVE_FLAG_VALID_MASK	(NE_ENCLAVE_PCIE_MODE | \
+					 NE_ENCLAVE_PCIE_VSOCK_MODE | \
+					 NE_ENCLAVE_PCIE_VIRTIO_MODE | \
+					 NE_ENCLAVE_PCIE_NUMA_MODE)
+
 struct cma;
 
 /**
diff --git a/include/uapi/linux/nitro_enclaves.h b/include/uapi/linux/nitro_enclaves.h
index 01bec4d072d6a..67be5c383c54b 100644
--- a/include/uapi/linux/nitro_enclaves.h
+++ b/include/uapi/linux/nitro_enclaves.h
@@ -58,6 +58,44 @@
  */
 #define NE_ENCLAVE_PCIE_MODE		(1u << 6)
 
+/**
+ * NE_ENCLAVE_PCIE_VSOCK_MODE - Allocate a slot whose PCIe bus holds one
+ *				virtio-vsock device, attached before the enclave
+ *				starts. Requires NE_ENCLAVE_PCIE_MODE and
+ *				excludes NE_ENCLAVE_PCIE_VIRTIO_MODE.
+ */
+#define NE_ENCLAVE_PCIE_VSOCK_MODE	(1u << 7)
+
+/**
+ * NE_ENCLAVE_PCIE_VIRTIO_MODE - Allocate a slot whose PCIe bus takes any
+ *				 virtio device type %NE_ADD_DEVICE offers it.
+ *				 Requires NE_ENCLAVE_PCIE_MODE and excludes
+ *				 NE_ENCLAVE_PCIE_VSOCK_MODE.
+ */
+#define NE_ENCLAVE_PCIE_VIRTIO_MODE	(1u << 9)
+
+/**
+ * NE_ENCLAVE_PCIE_NUMA_MODE - Allocate a slot whose memory and vCPUs may come
+ *			       from more than one host NUMA node. Without it the
+ *			       hypervisor refuses a slot that would straddle a
+ *			       node boundary. Requires NE_ENCLAVE_PCIE_MODE.
+ */
+#define NE_ENCLAVE_PCIE_NUMA_MODE	(1u << 13)
+
+/**
+ * NE_ENCLAVE_FLAG_UNMATCHED_MASK - The launch flags the hypervisor does not
+ *				    enumerate. It answers for them when a slot is
+ *				    allocated with them and not before.
+ */
+#define NE_ENCLAVE_FLAG_UNMATCHED_MASK	(0x003fu)
+
+/**
+ * NE_ENCLAVE_FLAG_MATCHED_MASK - The launch flags the hypervisor enumerates.
+ *				  It answers for them before a slot is
+ *				  allocated with them.
+ */
+#define NE_ENCLAVE_FLAG_MATCHED_MASK	(0xffc0u)
+
 /**
  * struct ne_create_vm_args - NE_CREATE_VM2 argument (in / out).
  * @flags:	Launch-time flag word (in). Flags select properties of the
-- 
2.47.1

[PATCH 35/40] nitro_enclaves: Add NE_ENCLAVE_DIRECT_MODE

[Alexander Graf]

Nitro Sandboxes introduce a mode called direct mode, in which the parent
keeps access to most of the memory it donates to the sandbox. This mode
disables any attestation, but enables guest kernels that are not aware
of the Device Memory Buffer (DMB) to work. A DMB-aware guest kernel
places its virtqueues and buffers in memory the device provides, so the
parent needs access to that memory only.

Add NE_ENCLAVE_DIRECT_MODE, which asks the hypervisor for a direct-mode
slot. Remember the flags a slot was allocated with in
ne_enclave->launch_flags. Allow NE_ADD_MEMORY_REGION to use an
NE_MEMORY_SHARED allocation for a slot that carries this flag.

Signed-off-by: Alexander Graf <graf@amazon.com>
---
 Documentation/virt/ne_overview.rst        |  4 +++-
 drivers/virt/nitro_enclaves/ne_misc_dev.c |  9 +++++++--
 drivers/virt/nitro_enclaves/ne_misc_dev.h |  7 ++++++-
 include/uapi/linux/nitro_enclaves.h       | 19 +++++++++++++++++--
 4 files changed, 33 insertions(+), 6 deletions(-)

diff --git a/Documentation/virt/ne_overview.rst b/Documentation/virt/ne_overview.rst
index 012fcb394a2ef..5984fd32ec07a 100644
--- a/Documentation/virt/ne_overview.rst
+++ b/Documentation/virt/ne_overview.rst
@@ -183,7 +183,9 @@ memory region per contiguous run it is made of, or fewer where runs turn out
 to be physically adjacent. It also ends writing: the memory is the enclave's
 from that point, so an image goes in first. It takes private allocations
 only, since memory the parent can still reach is not memory the enclave has
-to itself.
+to itself. A slot allocated with NE_ENCLAVE_DIRECT_MODE asks for an enclave
+whose memory the parent keeps access to, and takes a shared allocation as a
+memory region.
 
 An enclave's memory is laid out in the order its allocations are given to it,
 and the offset NE_GET_IMAGE_LOAD_INFO returns is an offset into that layout.
diff --git a/drivers/virt/nitro_enclaves/ne_misc_dev.c b/drivers/virt/nitro_enclaves/ne_misc_dev.c
index 961b821d6038b..cccfceeb575cc 100644
--- a/drivers/virt/nitro_enclaves/ne_misc_dev.c
+++ b/drivers/virt/nitro_enclaves/ne_misc_dev.c
@@ -3447,6 +3447,8 @@ static int ne_create_vm_ioctl(struct ne_pci_dev *ne_pci_dev, u64 flags,
 	if (!ne_enclave)
 		return -ENOMEM;
 
+	ne_enclave->launch_flags = flags;
+
 	mutex_lock(&ne_cpu_pool.mutex);
 
 	ne_enclave->nr_parent_vm_cores = ne_cpu_pool.nr_parent_vm_cores;
@@ -4603,9 +4605,12 @@ static int ne_add_memory_region_ioctl(struct ne_enclave *ne_enclave,
 	/*
 	 * The parent may map a shared allocation and keeps any mapping it has
 	 * for as long as the allocation lives. Memory a region is made of is
-	 * the enclave's alone.
+	 * the enclave's alone, unless the slot was allocated for an enclave
+	 * whose memory the parent keeps access to, which is what
+	 * NE_ENCLAVE_DIRECT_MODE asks for.
 	 */
-	if (mem->shared) {
+	if (mem->shared &&
+	    !(ne_enclave->launch_flags & NE_ENCLAVE_DIRECT_MODE)) {
 		dev_err_ratelimited(ne_misc_dev.this_device,
 				    "The memory region allocation is shared\n");
 
diff --git a/drivers/virt/nitro_enclaves/ne_misc_dev.h b/drivers/virt/nitro_enclaves/ne_misc_dev.h
index b0d0438157260..1fd93bddf7766 100644
--- a/drivers/virt/nitro_enclaves/ne_misc_dev.h
+++ b/drivers/virt/nitro_enclaves/ne_misc_dev.h
@@ -22,7 +22,8 @@
 /*
  * NE_ENCLAVE_FLAG_VALID_MASK - Every launch flag this kernel passes on.
  */
-#define NE_ENCLAVE_FLAG_VALID_MASK	(NE_ENCLAVE_PCIE_MODE | \
+#define NE_ENCLAVE_FLAG_VALID_MASK	(NE_ENCLAVE_DIRECT_MODE | \
+					 NE_ENCLAVE_PCIE_MODE | \
 					 NE_ENCLAVE_PCIE_VSOCK_MODE | \
 					 NE_ENCLAVE_PCIE_VIRTIO_MODE | \
 					 NE_ENCLAVE_PCIE_NUMA_MODE)
@@ -203,6 +204,9 @@ struct ne_doorbell_binding {
  *				so that the device keeps the page after user space
  *				has closed the descriptor. NULL until
  *				%NE_SET_INFO_PAGE is called.
+ * @launch_flags:		The launch flag word the slot was allocated with,
+ *				as %NE_CREATE_VM2 received it. Read to answer a
+ *				later call the way the slot was asked for.
  * @notify_bindings:		List of the notifications this enclave has bound
  *				an eventfd to.
  * @devices:			List of struct ne_device_wiring, one for each device
@@ -243,6 +247,7 @@ struct ne_enclave {
 	bool			has_event;
 	u64			id;
 	struct file		*info_file;
+	u64			launch_flags;
 	struct list_head	notify_bindings;
 	struct list_head	devices;
 	u64			max_mem_regions;
diff --git a/include/uapi/linux/nitro_enclaves.h b/include/uapi/linux/nitro_enclaves.h
index 67be5c383c54b..b346e0582c5fa 100644
--- a/include/uapi/linux/nitro_enclaves.h
+++ b/include/uapi/linux/nitro_enclaves.h
@@ -58,6 +58,17 @@
  */
 #define NE_ENCLAVE_PCIE_MODE		(1u << 6)
 
+/**
+ * NE_ENCLAVE_DIRECT_MODE - Allocate a slot whose memory the parent keeps
+ *			    access to. The enclave produces no attestation,
+ *			    since nothing separates its memory from the VM that
+ *			    launched it. An %NE_MEMORY_SHARED allocation can be
+ *			    given to such a slot as a memory region, which a
+ *			    slot without this flag refuses. Requires
+ *			    NE_ENCLAVE_PCIE_MODE.
+ */
+#define NE_ENCLAVE_DIRECT_MODE		(1u << 1)
+
 /**
  * NE_ENCLAVE_PCIE_VSOCK_MODE - Allocate a slot whose PCIe bus holds one
  *				virtio-vsock device, attached before the enclave
@@ -440,7 +451,9 @@ struct ne_add_memory_region_args {
  *			  enclave image goes in before this call. The allocation
  *			  has to be %NE_MEMORY_PRIVATE, since memory the parent
  *			  can still reach is not memory the enclave has to
- *			  itself.
+ *			  itself, unless the slot was allocated with
+ *			  %NE_ENCLAVE_DIRECT_MODE, which asks for an enclave
+ *			  whose memory the parent keeps access to.
  *			  An enclave's memory is laid out in the order its
  *			  allocations are given to it, and the offset from
  *			  %NE_GET_IMAGE_LOAD_INFO is an offset into that layout,
@@ -461,7 +474,9 @@ struct ne_add_memory_region_args {
  * * EINVAL				- The argument is shorter than @flags, or @fd
  *					  is not a descriptor from %NE_ALLOC_MEMORY, or
  *					  the allocation was made for another enclave,
- *					  or the allocation is %NE_MEMORY_SHARED.
+ *					  or the allocation is %NE_MEMORY_SHARED
+ *					  and the slot was not allocated with
+ *					  %NE_ENCLAVE_DIRECT_MODE.
  * * EIO				- Current task mm is not the same as the one
  *					  that created the enclave.
  * * E2BIG				- The argument is longer than this kernel knows
-- 
2.47.1

[PATCH 36/40] nitro_enclaves: Add NE_ENCLAVE_PCIE_HOTPLUG_MODE

[Alexander Graf]

Users want to add devices to a Nitro Sandbox while it runs. The
hypervisor supports that for a slot it sets up for it when it allocates
the slot.

Add NE_ENCLAVE_PCIE_HOTPLUG_MODE, which asks for such a slot, and let
NE_ADD_DEVICE take a device in the running state for a slot that carries
the flag.

Signed-off-by: Alexander Graf <graf@amazon.com>
---
 drivers/virt/nitro_enclaves/ne_misc_dev.c |  9 ++++++++-
 drivers/virt/nitro_enclaves/ne_misc_dev.h |  1 +
 include/uapi/linux/nitro_enclaves.h       | 24 ++++++++++++++++++++---
 3 files changed, 30 insertions(+), 4 deletions(-)

diff --git a/drivers/virt/nitro_enclaves/ne_misc_dev.c b/drivers/virt/nitro_enclaves/ne_misc_dev.c
index cccfceeb575cc..870b3a6368cf9 100644
--- a/drivers/virt/nitro_enclaves/ne_misc_dev.c
+++ b/drivers/virt/nitro_enclaves/ne_misc_dev.c
@@ -2967,7 +2967,14 @@ static long ne_enclave_do_ioctl(struct file *file, unsigned int cmd, unsigned lo
 		if (rc)
 			return rc;
 
-		if (ne_enclave->state != NE_STATE_INIT) {
+		/*
+		 * A slot allocated to take a device once its enclave runs
+		 * takes one in the running state as well. An enclave that has
+		 * stopped takes nothing either way.
+		 */
+		if (ne_enclave->state != NE_STATE_INIT &&
+		    !(ne_enclave->state == NE_STATE_RUNNING &&
+		      (ne_enclave->launch_flags & NE_ENCLAVE_PCIE_HOTPLUG_MODE))) {
 			dev_err_ratelimited(ne_misc_dev.this_device,
 					    "Enclave is not in init state\n");
 
diff --git a/drivers/virt/nitro_enclaves/ne_misc_dev.h b/drivers/virt/nitro_enclaves/ne_misc_dev.h
index 1fd93bddf7766..ed1b29997e1bb 100644
--- a/drivers/virt/nitro_enclaves/ne_misc_dev.h
+++ b/drivers/virt/nitro_enclaves/ne_misc_dev.h
@@ -26,6 +26,7 @@
 					 NE_ENCLAVE_PCIE_MODE | \
 					 NE_ENCLAVE_PCIE_VSOCK_MODE | \
 					 NE_ENCLAVE_PCIE_VIRTIO_MODE | \
+					 NE_ENCLAVE_PCIE_HOTPLUG_MODE | \
 					 NE_ENCLAVE_PCIE_NUMA_MODE)
 
 struct cma;
diff --git a/include/uapi/linux/nitro_enclaves.h b/include/uapi/linux/nitro_enclaves.h
index b346e0582c5fa..f204d0f6e7efd 100644
--- a/include/uapi/linux/nitro_enclaves.h
+++ b/include/uapi/linux/nitro_enclaves.h
@@ -85,6 +85,17 @@
  */
 #define NE_ENCLAVE_PCIE_VIRTIO_MODE	(1u << 9)
 
+/**
+ * NE_ENCLAVE_PCIE_HOTPLUG_MODE - Allocate a slot whose PCIe bus can take
+ *				  a device once the enclave is running. The
+ *				  hypervisor decides it at SLOT_ALLOC time, and
+ *				  %NE_ADD_DEVICE takes a device in the running
+ *				  state for a slot that carries it. Requires
+ *				  NE_ENCLAVE_PCIE_MODE and excludes
+ *				  NE_ENCLAVE_PCIE_VSOCK_MODE.
+ */
+#define NE_ENCLAVE_PCIE_HOTPLUG_MODE	(1u << 8)
+
 /**
  * NE_ENCLAVE_PCIE_NUMA_MODE - Allocate a slot whose memory and vCPUs may come
  *			       from more than one host NUMA node. Without it the
@@ -701,7 +712,11 @@ struct ne_add_device_args {
  *		   An argument that ends at @vq_info is taken and gets no
  *		   @info_notify_index back.
  *		   The ioctl can be invoked on the enclave fd, before an
- *		   enclave is started.
+ *		   enclave is started, and while it is running if the slot
+ *		   was allocated with %NE_ENCLAVE_PCIE_HOTPLUG_MODE. A
+ *		   started enclave refuses %NE_ALLOC_MEMORY, so a device
+ *		   given to one works from allocations made before it
+ *		   started.
  *
  * Context: Process context.
  * Return:
@@ -738,8 +753,11 @@ struct ne_add_device_args {
  * * EPROTO				- The device replied for a different number of
  *					  virtqueues than it was asked for, or placed
  *					  their entries outside the info page.
- * * NE_ERR_NOT_IN_INIT_STATE		- The enclave is not in init state (init =
- *					  before being started).
+ * * NE_ERR_NOT_IN_INIT_STATE		- The enclave left its init state (init =
+ *					  before being started) and the slot was
+ *					  not allocated with
+ *					  %NE_ENCLAVE_PCIE_HOTPLUG_MODE, or the
+ *					  enclave has stopped.
  * * NE_ERR_INVALID_FLAG_VALUE		- The value of the provided flag is invalid.
  * * NE_ERR_MEM_REGION_ALREADY_USED	- An allocation was already given to the
  *					  enclave, or is its info page, or is the
-- 
2.47.1

[PATCH 37/40] nitro_enclaves: Add NE_ENCLAVE_CPU_OVERCOMMIT_MODE

[Alexander Graf]

Nitro Sandboxes relax the dedicated-vCPU requirement and allow a sandbox
to opt-in to CPU overcommitment. When active, the hypervisor performs
vCPU scheduling of sandbox vCPUs (vCores really) on top of the parent's
vCPUs.

Add the NE_ENCLAVE_CPU_OVERCOMMIT_MODE flag and relax the dedicated CPU
validation and pooling logic when the mode is active.

Signed-off-by: Alexander Graf <graf@amazon.com>
---
 Documentation/virt/ne_overview.rst        |  15 +-
 drivers/virt/nitro_enclaves/ne_misc_dev.c | 249 ++++++++++++++++++++--
 drivers/virt/nitro_enclaves/ne_misc_dev.h |   1 +
 include/uapi/linux/nitro_enclaves.h       |  45 +++-
 4 files changed, 283 insertions(+), 27 deletions(-)

diff --git a/Documentation/virt/ne_overview.rst b/Documentation/virt/ne_overview.rst
index 5984fd32ec07a..3aa33d3f06cf2 100644
--- a/Documentation/virt/ne_overview.rst
+++ b/Documentation/virt/ne_overview.rst
@@ -113,10 +113,23 @@ each node in turn, and sets numa_node to NE_NUMA_ANY when it does not
 care. Both start from a free thread of a core the enclave already holds,
 and only take a new core once there is none.
 
+A slot allocated with NE_ENCLAVE_CPU_OVERCOMMIT_MODE has its vCPUs
+scheduled by the host alongside the parent's own and takes no core out of
+the pool, so the pool is not consulted for it. Both NE_ADD_VCPU and
+NE_ADD_ANY_VCPU choose an id the enclave does not hold yet, and the host
+schedules the vCPU alongside the parent CPU that id names, so the node
+NE_ADD_ANY_VCPU takes still places it, and a node with no free id left
+fails the request. Both prefer a free id whose SMT sibling the enclave
+already holds, so its vCPUs share cores as they do with a pool, and
+otherwise take the lowest free id. A parent that never set a pool up runs
+such an enclave.
+
 The node says where cores come from. NE_ALLOC_MEMORY takes its own node;
 see `Memory pool`_. An enclave built out of a pool that spans
 nodes has no node of its own, so the rule that its memory be on that node
-does not apply to it and its memory can come from any node.
+does not apply to it and its memory can come from any node. A slot
+allocated with NE_ENCLAVE_CPU_OVERCOMMIT_MODE holds no core at all, so it
+has no node of its own either.
 
 The pool occupancy is readable under
 /sys/devices/virtual/misc/nitro_enclaves/cpu_pool/, and per node under
diff --git a/drivers/virt/nitro_enclaves/ne_misc_dev.c b/drivers/virt/nitro_enclaves/ne_misc_dev.c
index 870b3a6368cf9..e79740eac6f29 100644
--- a/drivers/virt/nitro_enclaves/ne_misc_dev.c
+++ b/drivers/virt/nitro_enclaves/ne_misc_dev.c
@@ -267,6 +267,12 @@ static const struct kernel_param *ne_cpus_kp;
  *				ne_setup_cpu_pool() offlines them, because
  *				cpu_to_node() is not documented to answer for an
  *				offline CPU.
+ * @core_cpus:			The threads of each core slot, indexed the same
+ *				way as @avail_threads_per_core. Unlike that
+ *				array it does not change while the pool is set
+ *				up, so it names the SMT siblings of a pool CPU
+ *				while the CPU is offline and
+ *				topology_sibling_cpumask() no longer names them.
  * @node_cpus:			The threads of the pool on each NUMA node, indexed
  *				by node id and sized by nr_node_ids. It partitions
  *				@total_cpus: a thread sits in the entry of the node
@@ -292,6 +298,7 @@ struct ne_cpu_pool {
 	unsigned int	nr_parent_vm_cores;
 	int		numa_node;
 	int		*core_nid;
+	cpumask_var_t	*core_cpus;
 	cpumask_var_t	*node_cpus;
 	struct cpumask	total_cpus;
 	struct cpumask	used_cpus;
@@ -656,6 +663,14 @@ static void ne_free_core_slots(void)
 	kfree(ne_cpu_pool.core_nid);
 	ne_cpu_pool.core_nid = NULL;
 
+	if (ne_cpu_pool.core_cpus) {
+		for (i = 0; i < ne_cpu_pool.nr_parent_vm_cores; i++)
+			free_cpumask_var(ne_cpu_pool.core_cpus[i]);
+
+		kfree(ne_cpu_pool.core_cpus);
+		ne_cpu_pool.core_cpus = NULL;
+	}
+
 	if (ne_cpu_pool.node_cpus) {
 		for (nid = 0; nid < nr_node_ids; nid++)
 			free_cpumask_var(ne_cpu_pool.node_cpus[nid]);
@@ -719,6 +734,10 @@ static int ne_build_core_slots(const struct cpumask *cpu_pool)
 	if (!ne_cpu_pool.core_nid)
 		goto free_slots;
 
+	ne_cpu_pool.core_cpus = kzalloc_objs(*ne_cpu_pool.core_cpus, nr_cores);
+	if (!ne_cpu_pool.core_cpus)
+		goto free_slots;
+
 	ne_cpu_pool.node_cpus = kzalloc_objs(*ne_cpu_pool.node_cpus, nr_node_ids);
 	if (!ne_cpu_pool.node_cpus)
 		goto free_slots;
@@ -728,7 +747,8 @@ static int ne_build_core_slots(const struct cpumask *cpu_pool)
 			goto free_slots;
 
 	for (i = 0; i < nr_cores; i++)
-		if (!zalloc_cpumask_var(&ne_cpu_pool.avail_threads_per_core[i], GFP_KERNEL))
+		if (!zalloc_cpumask_var(&ne_cpu_pool.avail_threads_per_core[i], GFP_KERNEL) ||
+		    !zalloc_cpumask_var(&ne_cpu_pool.core_cpus[i], GFP_KERNEL))
 			goto free_slots;
 
 	cpumask_clear(processed);
@@ -746,6 +766,9 @@ static int ne_build_core_slots(const struct cpumask *cpu_pool)
 			cpumask_set_cpu(cpu_sibling, processed);
 		}
 
+		cpumask_copy(ne_cpu_pool.core_cpus[next_core_idx],
+			     ne_cpu_pool.avail_threads_per_core[next_core_idx]);
+
 		core_nid = cpu_to_node(cpu);
 
 		ne_cpu_pool.core_nid[next_core_idx] = core_nid;
@@ -1166,6 +1189,118 @@ static bool ne_donated_cpu(struct ne_enclave *ne_enclave, unsigned int cpu)
 	return false;
 }
 
+/**
+ * ne_cpu_pool_core() - Find the core slot of a CPU in the NE CPU pool.
+ * @cpu:	CPU to look up.
+ *
+ * Context: Process context. This function is called with the ne_cpu_pool mutex
+ *	    held.
+ * Return:
+ * * Index of the core slot that holds @cpu.
+ * * -1 if @cpu is not in the pool.
+ */
+static int ne_cpu_pool_core(unsigned int cpu)
+{
+	unsigned int i = 0;
+
+	if (!cpumask_test_cpu(cpu, &ne_cpu_pool.total_cpus))
+		return -1;
+
+	for (i = 0; i < ne_cpu_pool.nr_parent_vm_cores; i++)
+		if (cpumask_test_cpu(cpu, ne_cpu_pool.core_cpus[i]))
+			return i;
+
+	return -1;
+}
+
+/**
+ * ne_get_floating_vcpu_id() - Choose a vCPU id for a slot whose vCPUs the host
+ *				schedules.
+ * @ne_enclave :	Private data associated with the current enclave.
+ * @vcpu_id:		The chosen vCPU id, set on success.
+ * @lowest_id:		The lowest id the caller allows to be chosen.
+ * @nid:		The NUMA node the vCPU should land on, or NUMA_NO_NODE
+ *			for no preference.
+ *
+ * The host schedules such a vCPU wherever it schedules the parent's own
+ * threads, so no core is taken and there is no pool to take one out of. It
+ * reads the id as the parent CPU to schedule the vCPU alongside, so
+ * cpu_to_node() on the id says which node the vCPU lands on, and only a free
+ * id on @nid is chosen. A node with no free id fails instead of yielding an
+ * id on another node. Without NE_ENCLAVE_PCIE_NUMA_MODE the hypervisor
+ * refuses a slot whose vCPUs straddle nodes, and the caller would only learn
+ * that when it starts the enclave.
+ *
+ * A free id whose SMT sibling already holds one of the enclave's vCPUs comes
+ * first, so the vCPUs fill whole cores the way they do for a slot that takes
+ * its cores from the pool. Otherwise the lowest free id is chosen. The siblings
+ * and the node of a CPU in the NE CPU pool come from what the pool recorded
+ * while the CPU was online, because the pool may have taken it offline since.
+ * An id that is not a possible CPU names no host CPU and is only chosen when
+ * @nid is NUMA_NO_NODE.
+ *
+ * @lowest_id is 1 where the caller hands the id back in the same word it
+ * read the request from, since zero is the request there.
+ *
+ * Context: Process context. This function is called with the ne_enclave mutex held.
+ * Return:
+ * * 0 on success.
+ * * Negative return value on failure.
+ */
+static int ne_get_floating_vcpu_id(struct ne_enclave *ne_enclave, u32 *vcpu_id,
+				   unsigned int lowest_id, int nid)
+{
+	const struct cpumask *siblings;
+	unsigned int first = nr_cpu_ids;
+	unsigned int cpu;
+	int core;
+
+	mutex_lock(&ne_cpu_pool.mutex);
+
+	for (cpu = lowest_id; cpu < nr_cpu_ids; cpu++) {
+		if (cpumask_test_cpu(cpu, ne_enclave->vcpu_ids))
+			continue;
+
+		if (!cpu_possible(cpu)) {
+			if (nid != NUMA_NO_NODE)
+				continue;
+
+			if (first == nr_cpu_ids)
+				first = cpu;
+
+			continue;
+		}
+
+		core = ne_cpu_pool_core(cpu);
+		if (core >= 0)
+			siblings = ne_cpu_pool.core_cpus[core];
+		else
+			siblings = topology_sibling_cpumask(cpu);
+
+		if (nid != NUMA_NO_NODE &&
+		    (core >= 0 ? ne_cpu_pool.core_nid[core] : cpu_to_node(cpu)) != nid)
+			continue;
+
+		if (first == nr_cpu_ids)
+			first = cpu;
+
+		if (cpumask_intersects(ne_enclave->vcpu_ids, siblings))
+			break;
+	}
+
+	mutex_unlock(&ne_cpu_pool.mutex);
+
+	if (cpu >= nr_cpu_ids)
+		cpu = first;
+
+	if (cpu >= nr_cpu_ids)
+		return -NE_ERR_INVALID_VCPU;
+
+	*vcpu_id = cpu;
+
+	return 0;
+}
+
 /**
  * ne_get_unused_core_from_cpu_pool() - Get the id of a full core from the
  *					NE CPU pool.
@@ -2578,7 +2713,37 @@ static long ne_enclave_do_ioctl(struct file *file, unsigned int cmd, unsigned lo
 			return -NE_ERR_INVALID_VCPU;
 		}
 
-		if (!vcpu_id) {
+		if (ne_enclave->launch_flags & NE_ENCLAVE_CPU_OVERCOMMIT_MODE) {
+			/*
+			 * The pool hands out the parent VM cores a slot runs
+			 * its vCPUs on, and a slot the host schedules the vCPUs
+			 * of takes none, so the pool is not consulted for it.
+			 * An id of zero asks this kernel to choose, as it does
+			 * for any other slot; an enclave still cannot be given
+			 * the same id twice.
+			 */
+			if (!vcpu_id) {
+				rc = ne_get_floating_vcpu_id(ne_enclave, &vcpu_id,
+							     1, NUMA_NO_NODE);
+				if (rc < 0) {
+					dev_err_ratelimited(ne_misc_dev.this_device,
+							    "Error in choosing a vCPU id [rc=%d]\n",
+							    rc);
+
+					mutex_unlock(&ne_enclave->enclave_info_mutex);
+
+					return rc;
+				}
+			} else if (ne_donated_cpu(ne_enclave, vcpu_id)) {
+				dev_err_ratelimited(ne_misc_dev.this_device,
+						    "CPU %d already used\n",
+						    vcpu_id);
+
+				mutex_unlock(&ne_enclave->enclave_info_mutex);
+
+				return -NE_ERR_VCPU_ALREADY_USED;
+			}
+		} else if (!vcpu_id) {
 			/* Use the CPU pool for choosing a CPU for the enclave. */
 			rc = ne_get_cpu_from_cpu_pool(ne_enclave, &vcpu_id,
 						      NUMA_NO_NODE);
@@ -2823,15 +2988,38 @@ static long ne_enclave_do_ioctl(struct file *file, unsigned int cmd, unsigned lo
 			return -NE_ERR_NOT_IN_INIT_STATE;
 		}
 
-		rc = ne_get_cpu_from_cpu_pool(ne_enclave, &vcpu_id, args.numa_node);
-		if (rc < 0) {
-			dev_err_ratelimited(ne_misc_dev.this_device,
-					    "Error in get CPU from pool [rc=%d node=%d]\n",
-					    rc, args.numa_node);
+		if (ne_enclave->launch_flags & NE_ENCLAVE_CPU_OVERCOMMIT_MODE) {
+			/*
+			 * The host schedules the vCPUs of such a slot wherever
+			 * it schedules the parent's own threads, so the slot
+			 * takes no core out of the pool and the pool is not
+			 * consulted for it. It reads the id as the parent CPU
+			 * to schedule the vCPU alongside, so @numa_node still
+			 * says which node the vCPU lands on and the id is
+			 * chosen to match it.
+			 */
+			rc = ne_get_floating_vcpu_id(ne_enclave, &vcpu_id, 0,
+						     args.numa_node);
+			if (rc < 0) {
+				dev_err_ratelimited(ne_misc_dev.this_device,
+						    "Error in choosing a vCPU id [rc=%d]\n",
+						    rc);
 
-			mutex_unlock(&ne_enclave->enclave_info_mutex);
+				mutex_unlock(&ne_enclave->enclave_info_mutex);
 
-			return rc;
+				return rc;
+			}
+		} else {
+			rc = ne_get_cpu_from_cpu_pool(ne_enclave, &vcpu_id, args.numa_node);
+			if (rc < 0) {
+				dev_err_ratelimited(ne_misc_dev.this_device,
+						    "Error in get CPU from pool [rc=%d node=%d]\n",
+						    rc, args.numa_node);
+
+				mutex_unlock(&ne_enclave->enclave_info_mutex);
+
+				return rc;
+			}
 		}
 
 		rc = ne_add_vcpu_ioctl(ne_enclave, vcpu_id);
@@ -3433,22 +3621,31 @@ static int ne_create_vm_ioctl(struct ne_pci_dev *ne_pci_dev, u64 flags,
 		slot_alloc_req.flags = flags;
 	}
 
-	mutex_lock(&ne_cpu_pool.mutex);
+	/*
+	 * The pool hands out the cores a slot runs its vCPUs on, and it has
+	 * none left once every core in it has had its threads taken, which is
+	 * also where a parent that never set one up starts. A slot the host
+	 * schedules the vCPUs of takes no core out of the pool, so what the
+	 * pool has left says nothing about it.
+	 */
+	if (!(flags & NE_ENCLAVE_CPU_OVERCOMMIT_MODE)) {
+		mutex_lock(&ne_cpu_pool.mutex);
 
-	for (i = 0; i < ne_cpu_pool.nr_parent_vm_cores; i++)
-		if (!cpumask_empty(ne_cpu_pool.avail_threads_per_core[i]))
-			break;
+		for (i = 0; i < ne_cpu_pool.nr_parent_vm_cores; i++)
+			if (!cpumask_empty(ne_cpu_pool.avail_threads_per_core[i]))
+				break;
 
-	if (i == ne_cpu_pool.nr_parent_vm_cores) {
-		dev_err_ratelimited(ne_misc_dev.this_device,
-				    "No CPUs available in CPU pool\n");
+		if (i == ne_cpu_pool.nr_parent_vm_cores) {
+			dev_err_ratelimited(ne_misc_dev.this_device,
+					    "No CPUs available in CPU pool\n");
 
-		mutex_unlock(&ne_cpu_pool.mutex);
+			mutex_unlock(&ne_cpu_pool.mutex);
 
-		return -NE_ERR_NO_CPUS_AVAIL_IN_POOL;
-	}
+			return -NE_ERR_NO_CPUS_AVAIL_IN_POOL;
+		}
 
-	mutex_unlock(&ne_cpu_pool.mutex);
+		mutex_unlock(&ne_cpu_pool.mutex);
+	}
 
 	ne_enclave = kzalloc_obj(*ne_enclave);
 	if (!ne_enclave)
@@ -3459,7 +3656,17 @@ static int ne_create_vm_ioctl(struct ne_pci_dev *ne_pci_dev, u64 flags,
 	mutex_lock(&ne_cpu_pool.mutex);
 
 	ne_enclave->nr_parent_vm_cores = ne_cpu_pool.nr_parent_vm_cores;
-	ne_enclave->numa_node = ne_cpu_pool.numa_node;
+
+	/*
+	 * The node an enclave takes its memory from is the node its cores
+	 * are on. A slot the host schedules the vCPUs of holds no core, so
+	 * it has no node of its own and its memory can come from any of
+	 * them.
+	 */
+	if (flags & NE_ENCLAVE_CPU_OVERCOMMIT_MODE)
+		ne_enclave->numa_node = NUMA_NO_NODE;
+	else
+		ne_enclave->numa_node = ne_cpu_pool.numa_node;
 
 	mutex_unlock(&ne_cpu_pool.mutex);
 
diff --git a/drivers/virt/nitro_enclaves/ne_misc_dev.h b/drivers/virt/nitro_enclaves/ne_misc_dev.h
index ed1b29997e1bb..f18b57a225325 100644
--- a/drivers/virt/nitro_enclaves/ne_misc_dev.h
+++ b/drivers/virt/nitro_enclaves/ne_misc_dev.h
@@ -27,6 +27,7 @@
 					 NE_ENCLAVE_PCIE_VSOCK_MODE | \
 					 NE_ENCLAVE_PCIE_VIRTIO_MODE | \
 					 NE_ENCLAVE_PCIE_HOTPLUG_MODE | \
+					 NE_ENCLAVE_CPU_OVERCOMMIT_MODE | \
 					 NE_ENCLAVE_PCIE_NUMA_MODE)
 
 struct cma;
diff --git a/include/uapi/linux/nitro_enclaves.h b/include/uapi/linux/nitro_enclaves.h
index f204d0f6e7efd..bfe1ca5ef7f7f 100644
--- a/include/uapi/linux/nitro_enclaves.h
+++ b/include/uapi/linux/nitro_enclaves.h
@@ -96,6 +96,21 @@
  */
 #define NE_ENCLAVE_PCIE_HOTPLUG_MODE	(1u << 8)
 
+/**
+ * NE_ENCLAVE_CPU_OVERCOMMIT_MODE - Allocate a slot whose vCPUs the host
+ *				    schedules alongside the parent's own,
+ *				    instead of running them on cores taken out
+ *				    of the NE CPU pool. The pool is not consulted
+ *				    for such a slot: %NE_ADD_VCPU and
+ *				    %NE_ADD_ANY_VCPU both choose an id the enclave
+ *				    does not hold yet, and the host schedules the
+ *				    vCPU alongside the parent CPU that id names.
+ *				    Requires
+ *				    NE_ENCLAVE_PCIE_MODE and
+ *				    NE_ENCLAVE_PCIE_VIRTIO_MODE.
+ */
+#define NE_ENCLAVE_CPU_OVERCOMMIT_MODE	(1u << 11)
+
 /**
  * NE_ENCLAVE_PCIE_NUMA_MODE - Allocate a slot whose memory and vCPUs may come
  *			       from more than one host NUMA node. Without it the
@@ -162,7 +177,9 @@ struct ne_create_vm_args {
  * * ENOMEM				- Memory allocation failure for internal
  *					  bookkeeping variables.
  * * NE_ERR_NO_CPUS_AVAIL_IN_POOL	- No NE CPU pool set / no CPUs available
- *					  in the pool.
+ *					  in the pool, for a slot allocated
+ *					  without
+ *					  %NE_ENCLAVE_CPU_OVERCOMMIT_MODE.
  * * Error codes from get_unused_fd_flags() and anon_inode_getfile().
  * * Error codes from the NE PCI device request.
  */
@@ -177,7 +194,12 @@ struct ne_create_vm_args {
  *		 one enclave can be from different nodes.
  *		 The vCPU id is an input / output parameter. If its value is 0,
  *		 then a CPU is chosen from the enclave CPU pool and returned via
- *		 this parameter.
+ *		 this parameter. A slot allocated with
+ *		 %NE_ENCLAVE_CPU_OVERCOMMIT_MODE takes no CPU out of the pool,
+ *		 so the pool is not consulted for it: an id of 0 asks this
+ *		 kernel for one the enclave does not hold yet, preferring an
+ *		 id whose SMT sibling the enclave already holds, and the id
+ *		 it chooses is never 0.
  *		 The ioctl can be invoked on the enclave fd, before an enclave
  *		 is started.
  *
@@ -200,7 +222,9 @@ struct ne_create_vm_args {
  * * NE_ERR_NOT_IN_INIT_STATE		- The enclave is not in init state
  *					  (init = before being started).
  * * NE_ERR_INVALID_VCPU		- The provided vCPU is not in the available
- *					  CPUs range.
+ *					  CPUs range, or the slot was allocated
+ *					  with %NE_ENCLAVE_CPU_OVERCOMMIT_MODE and
+ *					  every id is taken.
  * * Error codes from the NE PCI device request.
  */
 #define NE_ADD_VCPU			_IOWR(0xAE, 0x21, __u32)
@@ -208,7 +232,9 @@ struct ne_create_vm_args {
 /**
  * struct ne_add_any_vcpu_args - Argument for %NE_ADD_ANY_VCPU.
  * @numa_node:	The NUMA node the vCPU has to come from, or %NE_NUMA_ANY for
- *		any node.
+ *		any node. For a slot allocated with
+ *		%NE_ENCLAVE_CPU_OVERCOMMIT_MODE it is the node the host
+ *		should schedule the vCPU on.
  * @vcpu_id:	The vCPU id the kernel chose, set on success. Ignored on
  *		input.
  */
@@ -231,7 +257,13 @@ struct ne_add_any_vcpu_args {
  *		     for each node in turn and stops when a node runs out.
  *		     @vcpu_id carries back the vCPU id the kernel chose, so
  *		     a caller collects the enclave's vCPU set as it builds
- *		     it.
+ *		     it. For a slot allocated with
+ *		     %NE_ENCLAVE_CPU_OVERCOMMIT_MODE the host schedules the
+ *		     vCPU rather than a core out of the pool running it, so
+ *		     the pool is not consulted and the chosen id is the
+ *		     parent CPU the host schedules it alongside. The kernel
+ *		     prefers an id whose SMT sibling the enclave already
+ *		     holds.
  *		     The ioctl can be invoked on the enclave fd, before an
  *		     enclave is started.
  *
@@ -251,6 +283,9 @@ struct ne_add_any_vcpu_args {
  *				  about and the excess is not all zero.
  * * NE_ERR_NOT_IN_INIT_STATE	- The enclave is not in init state (init =
  *				  before being started).
+ * * NE_ERR_INVALID_VCPU	- The slot was allocated with
+ *				  %NE_ENCLAVE_CPU_OVERCOMMIT_MODE and every id
+ *				  on @numa_node is taken.
  * * NE_ERR_NO_CPUS_AVAIL_IN_POOL - No CPUs available in the NE CPU pool on
  *				  @numa_node.
  * * NE_ERR_VCPU_INVALID_CPU_CORE - The core the pool handed back is outside
-- 
2.47.1

[PATCH 38/40] nitro_enclaves: Add NE_GET_CAPS

[Alexander Graf]

User space that launches Nitro Sandboxes needs to know which of the new
calls and launch flags the kernel and the host implement, how many
devices a slot takes, the size of the info page, the queue limit per
device and the configuration space NE_ADD_DEVICE takes, before it
allocates or fills them.

Add the NE_GET_CAPS ioctl on the /dev/nitro_enclaves fd. struct ne_caps
reports those sizes, the NE_CAP_* bits of the optional calls this kernel
implements, NE_ABI_VERSION, and how many bytes the kernel filled in.

supported_matched_flags reports the matched launch flags this kernel
passes on, and hypervisor_matched_flags those the device reports.

Signed-off-by: Alexander Graf <graf@amazon.com>
---
 drivers/virt/nitro_enclaves/ne_misc_dev.c |  73 +++++++++++++
 drivers/virt/nitro_enclaves/ne_pci_dev.c  |  33 ++++++
 drivers/virt/nitro_enclaves/ne_pci_dev.h  |  22 ++++
 include/uapi/linux/nitro_enclaves.h       | 124 +++++++++++++++++++++-
 4 files changed, 251 insertions(+), 1 deletion(-)

diff --git a/drivers/virt/nitro_enclaves/ne_misc_dev.c b/drivers/virt/nitro_enclaves/ne_misc_dev.c
index e79740eac6f29..abbba71bbef06 100644
--- a/drivers/virt/nitro_enclaves/ne_misc_dev.c
+++ b/drivers/virt/nitro_enclaves/ne_misc_dev.c
@@ -154,6 +154,19 @@ static_assert(sizeof(struct ne_enclave_start_info) == 16);
 static_assert(offsetof(struct ne_enclave_start_info, flags) == 0);
 static_assert(offsetof(struct ne_enclave_start_info, enclave_cid) == 8);
 
+static_assert(sizeof(struct ne_caps) == 64);
+static_assert(offsetof(struct ne_caps, struct_size) == 0);
+static_assert(offsetof(struct ne_caps, abi_version) == 4);
+static_assert(offsetof(struct ne_caps, caps) == 8);
+static_assert(offsetof(struct ne_caps, supported_matched_flags) == 16);
+static_assert(offsetof(struct ne_caps, max_devices) == 24);
+static_assert(offsetof(struct ne_caps, max_vqs_per_device) == 28);
+static_assert(offsetof(struct ne_caps, max_config_size) == 32);
+static_assert(offsetof(struct ne_caps, reserved) == 36);
+static_assert(offsetof(struct ne_caps, info_page_size) == 40);
+static_assert(offsetof(struct ne_caps, info_page_align) == 48);
+static_assert(offsetof(struct ne_caps, hypervisor_matched_flags) == 56);
+
 /**
  * NE_EIF_LOAD_OFFSET - The offset where to copy the Enclave Image Format (EIF)
  *			image in enclave memory.
@@ -3817,6 +3830,66 @@ static long ne_ioctl(struct file *file, unsigned int cmd, unsigned long arg)
 
 	/* Extensible Argument ioctls */
 	switch (NE_EA_IOCTL(cmd)) {
+	case NE_EA_IOCTL(NE_GET_CAPS): {
+		struct ne_caps caps = {};
+		struct ne_caps __user *uargs = (void __user *)arg;
+		struct ne_pci_dev *ne_pci_dev;
+		unsigned int usize = _IOC_SIZE(cmd);
+		size_t wsize;
+		u32 asize;
+		int rc;
+
+		if (usize < offsetofend(struct ne_caps, supported_matched_flags))
+			return -EINVAL;
+
+		if (get_user(asize, &uargs->struct_size))
+			return -EFAULT;
+
+		if (asize < offsetofend(struct ne_caps, supported_matched_flags))
+			return -EINVAL;
+
+		wsize = min3((size_t)asize, (size_t)usize, sizeof(caps));
+
+		caps.struct_size = wsize;
+		caps.abi_version = NE_ABI_VERSION;
+		caps.caps = NE_CAP_CREATE_VM2 | NE_CAP_DEVICES;
+		caps.supported_matched_flags = NE_ENCLAVE_FLAG_VALID_MASK &
+					       NE_ENCLAVE_FLAG_MATCHED_MASK;
+		caps.max_vqs_per_device = NE_MAX_VQS;
+		caps.max_config_size = sizeof_field(struct ne_add_device_args,
+						    device_config);
+		caps.info_page_size = NE_INFO_PAGE_SIZE;
+		caps.info_page_align = NE_INFO_PAGE_ALIGN;
+
+		/*
+		 * Hold ne_devs.lock over the register reads alone, so that a
+		 * fault on the argument does not hold up an unbind of the
+		 * device.
+		 */
+		rc = down_read_killable(&ne_devs.lock);
+		if (rc)
+			return rc;
+
+		ne_pci_dev = ne_devs.ne_pci_dev;
+		if (ne_pci_dev) {
+			u32 matched_flags;
+			u16 max_devices;
+
+			ne_read_device_caps(ne_pci_dev, &matched_flags,
+					    &max_devices);
+
+			caps.max_devices = max_devices;
+			caps.hypervisor_matched_flags = matched_flags;
+		}
+
+		up_read(&ne_devs.lock);
+
+		if (copy_to_user(uargs, &caps, wsize))
+			return -EFAULT;
+
+		return 0;
+	}
+
 	case NE_EA_IOCTL(NE_CREATE_VM2): {
 		struct ne_create_vm_args args;
 		struct ne_create_vm_args __user *uargs = (void __user *)arg;
diff --git a/drivers/virt/nitro_enclaves/ne_pci_dev.c b/drivers/virt/nitro_enclaves/ne_pci_dev.c
index b5a7ecfffb4b6..591c43590a845 100644
--- a/drivers/virt/nitro_enclaves/ne_pci_dev.c
+++ b/drivers/virt/nitro_enclaves/ne_pci_dev.c
@@ -1101,6 +1101,39 @@ static void ne_slot_pool_read_geometry(struct pci_dev *pdev)
 	ne_pci_dev->slot_pool_gran = size;
 }
 
+/**
+ * ne_read_device_caps() - Read what the device reports about itself.
+ * @ne_pci_dev:	PCI device state to read the registers of.
+ * @matched_flags:	Where the launch flags the device reports go.
+ * @max_devices:	Where the number of devices it reports goes.
+ *
+ * The registers are part of the block a vendor-specific capability publishes,
+ * so a device without one is not read at all and both values come back zero.
+ * A device that has the block and reports zero in one of them names no value
+ * for it, which is also what a device predating that register reads as. A
+ * caller cannot tell those apart, so a zero is not an answer it can refuse a
+ * launch on.
+ *
+ * The registers are read on every call. A hypervisor update changes
+ * which flags the host implements and how many devices it lets one enclave
+ * have, so a caller asking after one gets what the host implements now.
+ *
+ * Context: Process context.
+ */
+void ne_read_device_caps(struct ne_pci_dev *ne_pci_dev, u32 *matched_flags,
+			 u16 *max_devices)
+{
+	*matched_flags = 0;
+	*max_devices = 0;
+
+	if (!ne_pci_dev->has_vendor_cap)
+		return;
+
+	*matched_flags = ioread32(ne_pci_dev->iomem_base + NE_MATCHED_FLAGS) &
+			 NE_ENCLAVE_FLAG_MATCHED_MASK;
+	*max_devices = ioread16(ne_pci_dev->iomem_base + NE_MAX_DEVICES);
+}
+
 /**
  * ne_slot_pool_alloc_run() - Allocate one run for the slot pool.
  * @nr_pages:	Number of pages in the run.
diff --git a/drivers/virt/nitro_enclaves/ne_pci_dev.h b/drivers/virt/nitro_enclaves/ne_pci_dev.h
index 5a92e0c565ba0..44491e451082a 100644
--- a/drivers/virt/nitro_enclaves/ne_pci_dev.h
+++ b/drivers/virt/nitro_enclaves/ne_pci_dev.h
@@ -175,6 +175,24 @@
  */
 #define NE_SLOT_POOL_BASE_SIZE	(0x0208)
 
+/**
+ * DOC: Device capability registers in the NE PCI device MMIO BAR
+ */
+
+/**
+ * NE_MATCHED_FLAGS - (4 bytes) Register reporting which of the launch flags in
+ *		      NE_ENCLAVE_FLAG_MATCHED_MASK the device implements, zero
+ *		      on a device that reports none (Read-Only).
+ */
+#define NE_MATCHED_FLAGS	(0x01fc)
+
+/**
+ * NE_MAX_DEVICES - (2 bytes) Register reporting how many devices the device
+ *		    lets one enclave have, zero on a device that reports no
+ *		    number (Read-Only).
+ */
+#define NE_MAX_DEVICES		(0x020e)
+
 /**
  * DOC: MSI-X interrupt vectors
  */
@@ -835,6 +853,10 @@ int ne_do_request(struct pci_dev *pdev, enum ne_pci_dev_cmd_type cmd_type,
  */
 void ne_mux_binding_free_rcu(struct ne_mux_binding *binding);
 
+/* Read what the device reports about itself, zero where it reports nothing. */
+void ne_read_device_caps(struct ne_pci_dev *ne_pci_dev, u32 *matched_flags,
+			 u16 *max_devices);
+
 /* Nitro Enclaves (NE) PCI device driver */
 extern struct pci_driver ne_pci_driver;
 
diff --git a/include/uapi/linux/nitro_enclaves.h b/include/uapi/linux/nitro_enclaves.h
index bfe1ca5ef7f7f..f8eeacd2e7062 100644
--- a/include/uapi/linux/nitro_enclaves.h
+++ b/include/uapi/linux/nitro_enclaves.h
@@ -129,7 +129,9 @@
 /**
  * NE_ENCLAVE_FLAG_MATCHED_MASK - The launch flags the hypervisor enumerates.
  *				  It answers for them before a slot is
- *				  allocated with them.
+ *				  allocated with them, and
+ *				  &ne_caps.supported_matched_flags reports the
+ *				  ones this kernel passes on.
  */
 #define NE_ENCLAVE_FLAG_MATCHED_MASK	(0xffc0u)
 
@@ -185,6 +187,114 @@ struct ne_create_vm_args {
  */
 #define NE_CREATE_VM2			_IOWR(0xAE, 0x30, struct ne_create_vm_args)
 
+/**
+ * NE_ABI_VERSION - Revision of this interface the kernel implements, reported
+ *		    in &ne_caps.abi_version. It counts additions to the
+ *		    interface as a whole. Some additions carry a %NE_CAP_* bit,
+ *		    and a caller after one of those reads its bit. An addition
+ *		    with no bit is probed by issuing the call, which a kernel
+ *		    that does not implement it turns down with ENOTTY.
+ */
+#define NE_ABI_VERSION			(1)
+
+/**
+ * NE_CAP_CREATE_VM2 - This kernel implements %NE_CREATE_VM2.
+ */
+#define NE_CAP_CREATE_VM2		(1ULL << 0)
+
+/**
+ * NE_CAP_DEVICES - This kernel implements %NE_SET_INFO_PAGE,
+ *			 %NE_ADD_DEVICE, %NE_SET_NOTIFY_EVENTFD and
+ *			 %NE_SET_VRING_CALL.
+ */
+#define NE_CAP_DEVICES			(1ULL << 1)
+
+/**
+ * struct ne_caps - What this kernel implements, and what the device it reaches
+ *		    reports about itself (in / out).
+ * @struct_size:	Bytes of this structure the caller has (in), and the bytes
+ *			the kernel filled in (out). A caller that finds a shorter
+ *			value than it asked for has the fields past that point
+ *			unwritten, which is a different answer from a zero.
+ * @abi_version:	%NE_ABI_VERSION this kernel implements (out).
+ * @caps:		%NE_CAP_* bits (out). A set bit says this kernel
+ *			implements the calls that bit's own documentation
+ *			lists, and not that the device the caller reaches will
+ *			serve them, which is what that device's answer to the
+ *			call says. Support for a launch flag outside
+ *			%NE_ENCLAVE_FLAG_MATCHED_MASK is reported here or, in
+ *			the end, by what %NE_CREATE_VM2 does with it.
+ * @supported_matched_flags: The launch flags inside
+ *			%NE_ENCLAVE_FLAG_MATCHED_MASK this kernel passes to the
+ *			hypervisor (out). A set bit says the bit is accepted on
+ *			its own, not that every combination is. Bits outside that
+ *			mask read zero here whatever the hypervisor does with
+ *			them, so a caller masks its request before comparing.
+ * @max_devices:	Devices one enclave may have, as the hypervisor reports
+ *			the number (out). It answers a request past it itself,
+ *			so this is what a caller plans against rather than what
+ *			it is held to. Zero names no number, which is what a
+ *			device reporting no number, a device publishing no
+ *			vendor capability, a device predating the register and
+ *			no device bound at all leave here, so a zero bounds no
+ *			request and %NE_ADD_DEVICE is what answers.
+ * @max_vqs_per_device:	Largest @num_vqs %NE_ADD_DEVICE takes (out).
+ * @max_config_size:	Largest @config_size %NE_ADD_DEVICE takes (out).
+ * @reserved:		Zero (out).
+ * @info_page_size:	Size the %NE_SET_INFO_PAGE allocation has to have (out).
+ * @info_page_align:	Alignment it has to have (out).
+ * @hypervisor_matched_flags: The launch flags inside
+ *			%NE_ENCLAVE_FLAG_MATCHED_MASK the hypervisor reports it
+ *			implements (out). @supported_matched_flags is what this
+ *			kernel passes on and is the same on every host; this is
+ *			what the host behind the device answers for. A caller
+ *			requires a flag in @supported_matched_flags always, and
+ *			in this field as well when this field is non-zero.
+ *			Zero names no flag, which is what a device reporting
+ *			no flag, a device publishing no vendor capability, a
+ *			device predating the register and no device bound at
+ *			all leave here, so a zero refuses no flag and
+ *			%NE_CREATE_VM2 is what says so.
+ */
+struct ne_caps {
+	__u32	struct_size;
+	__u32	abi_version;
+	__u64	caps;
+	__u64	supported_matched_flags;
+	__u32	max_devices;
+	__u32	max_vqs_per_device;
+	__u32	max_config_size;
+	__u32	reserved;
+	__u64	info_page_size;
+	__u64	info_page_align;
+	__u64	hypervisor_matched_flags;
+};
+
+/**
+ * NE_GET_CAPS - The command is used to read what this kernel implements and
+ *		 what the device it reaches reports about itself, so that a
+ *		 caller sizes its allocations and reads which launch flags will
+ *		 be passed on and answered for. The ioctl can be invoked on the
+ *		 /dev/nitro_enclaves fd at any time, including before an
+ *		 enclave exists.
+ *
+ *		 The argument carries its own length in @struct_size, and the
+ *		 kernel writes back how many bytes it filled, so a caller built
+ *		 against a header this kernel predates can tell a field this
+ *		 kernel never wrote from one it wrote a zero into.
+ *
+ * Context: Process context.
+ * Return:
+ * * 0					- Logic successfully completed.
+ * * -1					- There was a failure in the ioctl logic.
+ * On failure, errno is set to:
+ * * EFAULT				- Argument read or write failure.
+ * * EINVAL				- The argument stops before
+ *					  @supported_matched_flags, or @struct_size
+ *					  names a length that stops before it.
+ */
+#define NE_GET_CAPS			_IOWR(0xAE, 0x29, struct ne_caps)
+
 /**
  * NE_ADD_VCPU - The command is used to set a vCPU for an enclave. The vCPU can
  *		 be auto-chosen from the NE CPU pool or it can be set by the
@@ -612,6 +722,18 @@ struct ne_set_info_page_args {
  */
 #define NE_SET_INFO_PAGE		_IOW(0xAE, 0x25, struct ne_set_info_page_args)
 
+/**
+ * NE_INFO_PAGE_SIZE - Size the info page allocation has to have, reported in
+ *		       &ne_caps.info_page_size.
+ */
+#define NE_INFO_PAGE_SIZE		(0x200000)
+
+/**
+ * NE_INFO_PAGE_ALIGN - Alignment the info page allocation has to have,
+ *			reported in &ne_caps.info_page_align.
+ */
+#define NE_INFO_PAGE_ALIGN		(0x200000)
+
 /**
  * NE_MAX_VQS - Highest number of virtqueues one device may have.
  */
-- 
2.47.1

[PATCH 39/40] nitro_enclaves: Add NE_UPDATE_DEVICE_CONFIG

[Alexander Graf]

A virtio device reports state that changes while the guest runs, such as
a network link going down, through its configuration space. Nitro
Sandboxes let the device backend in the parent write that space while
the sandbox runs through an explicit full-config-update call, which also
raises the virtio configuration-change notification inside the sandbox.

Add the NE_UPDATE_DEVICE_CONFIG ioctl on the enclave fd to give user
space an API to call with the updated config space.

Signed-off-by: Alexander Graf <graf@amazon.com>
---
 drivers/virt/nitro_enclaves/ne_misc_dev.c | 113 +++++++++++++++++++++-
 drivers/virt/nitro_enclaves/ne_pci_dev.h  |  24 +++++
 include/uapi/linux/nitro_enclaves.h       | 100 ++++++++++++++++++-
 3 files changed, 231 insertions(+), 6 deletions(-)

diff --git a/drivers/virt/nitro_enclaves/ne_misc_dev.c b/drivers/virt/nitro_enclaves/ne_misc_dev.c
index abbba71bbef06..a5550a6be86ea 100644
--- a/drivers/virt/nitro_enclaves/ne_misc_dev.c
+++ b/drivers/virt/nitro_enclaves/ne_misc_dev.c
@@ -120,6 +120,12 @@ static_assert(offsetof(struct ne_vring_call, fd) == 0);
 static_assert(offsetof(struct ne_vring_call, doorbell_offset) == 4);
 static_assert(offsetof(struct ne_vring_call, reserved) == 6);
 
+static_assert(sizeof(struct ne_update_device_config) == 24);
+static_assert(offsetof(struct ne_update_device_config, device_uid) == 0);
+static_assert(offsetof(struct ne_update_device_config, config_addr) == 8);
+static_assert(offsetof(struct ne_update_device_config, config_size) == 16);
+static_assert(offsetof(struct ne_update_device_config, config_offset) == 20);
+
 static_assert(sizeof(struct ne_create_vm_args) == 16);
 static_assert(offsetof(struct ne_create_vm_args, flags) == 0);
 static_assert(offsetof(struct ne_create_vm_args, slot_uid) == 8);
@@ -162,7 +168,7 @@ static_assert(offsetof(struct ne_caps, supported_matched_flags) == 16);
 static_assert(offsetof(struct ne_caps, max_devices) == 24);
 static_assert(offsetof(struct ne_caps, max_vqs_per_device) == 28);
 static_assert(offsetof(struct ne_caps, max_config_size) == 32);
-static_assert(offsetof(struct ne_caps, reserved) == 36);
+static_assert(offsetof(struct ne_caps, max_config_update_size) == 36);
 static_assert(offsetof(struct ne_caps, info_page_size) == 40);
 static_assert(offsetof(struct ne_caps, info_page_align) == 48);
 static_assert(offsetof(struct ne_caps, hypervisor_matched_flags) == 56);
@@ -217,6 +223,8 @@ static int ne_set_info_page_ioctl(struct ne_enclave *ne_enclave,
 				  struct ne_set_info_page_args *args);
 static int ne_add_device_ioctl(struct ne_enclave *ne_enclave,
 			       struct ne_add_device_args *args);
+static int ne_update_device_config_ioctl(struct ne_enclave *ne_enclave,
+					 struct ne_update_device_config *args);
 static const struct file_operations ne_fops = {
 	.owner		= THIS_MODULE,
 	.llseek		= noop_llseek,
@@ -3229,6 +3237,39 @@ static long ne_enclave_do_ioctl(struct file *file, unsigned int cmd, unsigned lo
 		return ne_set_vring_call(ne_enclave, &args);
 	}
 
+	case NE_EA_IOCTL(NE_UPDATE_DEVICE_CONFIG): {
+		struct ne_update_device_config __user *uargs = (void __user *)arg;
+		unsigned int usize = _IOC_SIZE(cmd);
+		struct ne_update_device_config args;
+		int rc;
+
+		if (usize < offsetofend(struct ne_update_device_config, config_offset))
+			return -EINVAL;
+
+		rc = copy_struct_from_user(&args, sizeof(args), uargs, usize);
+		if (rc)
+			return rc;
+
+		rc = mutex_lock_killable(&ne_enclave->enclave_info_mutex);
+		if (rc)
+			return rc;
+
+		if (ne_enclave->state != NE_STATE_RUNNING) {
+			dev_err_ratelimited(ne_misc_dev.this_device,
+					    "Enclave is not in running state\n");
+
+			mutex_unlock(&ne_enclave->enclave_info_mutex);
+
+			return -NE_ERR_NOT_IN_RUNNING_STATE;
+		}
+
+		rc = ne_update_device_config_ioctl(ne_enclave, &args);
+
+		mutex_unlock(&ne_enclave->enclave_info_mutex);
+
+		return rc;
+	}
+
 	default:
 		return -ENOTTY;
 	}
@@ -3858,6 +3899,9 @@ static long ne_ioctl(struct file *file, unsigned int cmd, unsigned long arg)
 		caps.max_vqs_per_device = NE_MAX_VQS;
 		caps.max_config_size = sizeof_field(struct ne_add_device_args,
 						    device_config);
+		caps.max_config_update_size =
+			sizeof_field(struct slot_update_device_config_req,
+				     config_data);
 		caps.info_page_size = NE_INFO_PAGE_SIZE;
 		caps.info_page_align = NE_INFO_PAGE_ALIGN;
 
@@ -5598,6 +5642,73 @@ static int ne_add_device_ioctl(struct ne_enclave *ne_enclave,
 	return rc;
 }
 
+/**
+ * ne_update_device_config_ioctl() - Write part of the configuration space of a
+ *				     device the enclave is driving.
+ * @ne_enclave:	Private data associated with the current enclave.
+ * @args:	Where the bytes to write are, how many of them there are and
+ *		where in the device's configuration space they go.
+ *
+ * The bytes are copied out of the caller before the request is built, so what
+ * the caller does with its buffer afterwards does not reach the device, and a
+ * call that is refused here sends the device nothing.
+ *
+ * Context: Process context. This function is called with the ne_enclave mutex held.
+ * Return:
+ * * 0 on success.
+ * * Negative return value on failure.
+ */
+static int ne_update_device_config_ioctl(struct ne_enclave *ne_enclave,
+					 struct ne_update_device_config *args)
+{
+	struct slot_update_device_config_req req = {};
+	struct ne_pci_dev_cmd_reply cmd_reply = {};
+	int rc;
+
+	if (!args->config_size) {
+		dev_err_ratelimited(ne_misc_dev.this_device,
+				    "The configuration write is empty\n");
+
+		return -EINVAL;
+	}
+
+	/*
+	 * The request carries the bytes in a fixed array and tells the device
+	 * separately where in its configuration space to put them, so what the
+	 * request bounds is how many bytes one call moves. How far into the
+	 * space a write may reach is the device's to bound, out of the
+	 * configuration space it was given for that device.
+	 */
+	if (args->config_size > sizeof(req.config_data)) {
+		dev_err_ratelimited(ne_misc_dev.this_device,
+				    "The configuration write is longer than %zu bytes\n",
+				    sizeof(req.config_data));
+
+		return -EINVAL;
+	}
+
+	if (copy_from_user(req.config_data, u64_to_user_ptr(args->config_addr),
+			   args->config_size))
+		return -EFAULT;
+
+	req.slot_uid = ne_enclave->slot_uid;
+	req.device_uid = args->device_uid;
+	req.config_size = args->config_size;
+	req.config_offset = args->config_offset;
+
+	rc = ne_do_request(ne_devs.ne_pci_dev->pdev, SLOT_UPDATE_DEVICE_CONFIG,
+			   &req, sizeof(req), &cmd_reply, sizeof(cmd_reply));
+	if (rc < 0) {
+		dev_err_ratelimited(ne_misc_dev.this_device,
+				    "Error in slot update device config [rc=%d]\n",
+				    rc);
+
+		return rc;
+	}
+
+	return 0;
+}
+
 #if defined(CONFIG_NITRO_ENCLAVES_MISC_DEV_TEST)
 #include "ne_misc_dev_test.c"
 #endif
diff --git a/drivers/virt/nitro_enclaves/ne_pci_dev.h b/drivers/virt/nitro_enclaves/ne_pci_dev.h
index 44491e451082a..74728d2a15472 100644
--- a/drivers/virt/nitro_enclaves/ne_pci_dev.h
+++ b/drivers/virt/nitro_enclaves/ne_pci_dev.h
@@ -256,6 +256,8 @@ static_assert(NE_MUX_MAX_BITS / BITS_PER_TYPE(u64) * sizeof(u64) <= PAGE_SIZE,
  *				device runs a slot out of.
  * @SLOT_POOL_GET_LIMITS:	Report how many slots each half of the pool has
  *				room for.
+ * @SLOT_UPDATE_DEVICE_CONFIG:	Write part of the configuration space of a
+ *				device a slot is driving.
  * @SLOT_POOL_BASE_DONATE:	Donate the base run of the slot pool.
  * @MAX_CMD:			A gatekeeper for max possible command type.
  */
@@ -277,6 +279,7 @@ enum ne_pci_dev_cmd_type {
 	SLOT_ADD_DEVICE		= 14,
 	SLOT_POOL_DONATE	= 16,
 	SLOT_POOL_SHUTDOWN	= 17,
+	SLOT_UPDATE_DEVICE_CONFIG = 24,
 	SLOT_POOL_BASE_DONATE	= 25,
 	SLOT_POOL_MEM_DONATE	= 26,
 	SLOT_POOL_GET_LIMITS	= 27,
@@ -502,6 +505,27 @@ struct slot_add_device_req {
 } __packed;
 static_assert(sizeof(struct slot_add_device_req) <= NE_SEND_DATA_SIZE);
 
+/**
+ * struct slot_update_device_config_req - SLOT_UPDATE_DEVICE_CONFIG request.
+ * @slot_uid:			Slot unique id mapped to the slot driving the
+ *				device.
+ * @device_uid:			Device to write the configuration of.
+ * @config_size:		Number of bytes of @config_data the device
+ *				reads.
+ * @config_offset:		Byte offset within the device's configuration
+ *				space the device is to place @config_data at.
+ * @config_data:		Configuration to write into the device's
+ *				configuration space.
+ */
+struct slot_update_device_config_req {
+	u64	slot_uid;
+	u64	device_uid;
+	u32	config_size;
+	u32	config_offset;
+	u8	config_data[64];
+} __packed;
+static_assert(sizeof(struct slot_update_device_config_req) <= NE_SEND_DATA_SIZE);
+
 /**
  * struct slot_pool_donate_req - SLOT_POOL_DONATE and SLOT_POOL_BASE_DONATE
  *				 request.
diff --git a/include/uapi/linux/nitro_enclaves.h b/include/uapi/linux/nitro_enclaves.h
index f8eeacd2e7062..4a0a8eb30caa4 100644
--- a/include/uapi/linux/nitro_enclaves.h
+++ b/include/uapi/linux/nitro_enclaves.h
@@ -204,8 +204,8 @@ struct ne_create_vm_args {
 
 /**
  * NE_CAP_DEVICES - This kernel implements %NE_SET_INFO_PAGE,
- *			 %NE_ADD_DEVICE, %NE_SET_NOTIFY_EVENTFD and
- *			 %NE_SET_VRING_CALL.
+ *			 %NE_ADD_DEVICE, %NE_SET_NOTIFY_EVENTFD,
+ *			 %NE_SET_VRING_CALL and %NE_UPDATE_DEVICE_CONFIG.
  */
 #define NE_CAP_DEVICES			(1ULL << 1)
 
@@ -240,7 +240,15 @@ struct ne_create_vm_args {
  *			request and %NE_ADD_DEVICE is what answers.
  * @max_vqs_per_device:	Largest @num_vqs %NE_ADD_DEVICE takes (out).
  * @max_config_size:	Largest @config_size %NE_ADD_DEVICE takes (out).
- * @reserved:		Zero (out).
+ * @max_config_update_size:	Largest @config_size one %NE_UPDATE_DEVICE_CONFIG
+ *			call takes (out). It bounds how many bytes a call moves,
+ *			not how far into the configuration space they go: a write
+ *			carries its own offset, and how far a device lets one
+ *			reach is bounded by the configuration space that device
+ *			was given in @device_config. It is reported apart from
+ *			@max_config_size because the size of @device_config is
+ *			part of the ABI while this one is not, so this value can
+ *			grow without a new argument structure.
  * @info_page_size:	Size the %NE_SET_INFO_PAGE allocation has to have (out).
  * @info_page_align:	Alignment it has to have (out).
  * @hypervisor_matched_flags: The launch flags inside
@@ -264,7 +272,7 @@ struct ne_caps {
 	__u32	max_devices;
 	__u32	max_vqs_per_device;
 	__u32	max_config_size;
-	__u32	reserved;
+	__u32	max_config_update_size;
 	__u64	info_page_size;
 	__u64	info_page_align;
 	__u64	hypervisor_matched_flags;
@@ -806,7 +814,11 @@ struct ne_shm_region {
  * @padding:		Must be zero.
  * @device_features:	Feature bits the parent offers the enclave for the
  *			device.
- * @device_config:	Type-specific configuration of the device.
+ * @device_config:	The opening bytes of the device's configuration space, as
+ *			the enclave first sees it. It is the device-specific
+ *			configuration space a virtio driver reads, not PCI
+ *			configuration space, and @max_config_size bounds how much
+ *			of it may be handed over here.
  * @shm_regions_ptr:	Address of @num_shm_regions struct ne_shm_region.
  * @device_uid:		Id the device gave the new device (out).
  * @vq_info:		How the enclave reaches the device's first @num_vqs
@@ -1200,6 +1212,13 @@ struct ne_add_device_args {
  * NE_ERR_ZERO_REGION - The enclave's memory layout has a region of size zero.
  */
 #define NE_ERR_ZERO_REGION			(302)
+/**
+ * NE_ERR_NOT_IN_RUNNING_STATE - The enclave has not been started yet, or has
+ *				 already exited, when writing the configuration
+ *				 of one of its devices. The first code of the
+ *				 driver-only range.
+ */
+#define NE_ERR_NOT_IN_RUNNING_STATE		(1024)
 
 /**
  * DOC: Image load info flags
@@ -1426,4 +1445,75 @@ struct ne_vring_call {
 	__u16	reserved;
 };
 
+/**
+ * NE_UPDATE_DEVICE_CONFIG - The command is used to write part of the
+ *			     configuration space of a device an enclave is
+ *			     already driving, so that the device backend in the
+ *			     parent tells the driver in the enclave about state
+ *			     that moved after the device was attached. The
+ *			     hypervisor raises a configuration-change
+ *			     notification inside the enclave for the write.
+ *			     The parent holds the only copy of the space: no
+ *			     command reads a device's configuration back, so a
+ *			     caller that changes fields which do not sit next to
+ *			     each other either writes each of them on its own or
+ *			     writes the whole range out of the copy it already
+ *			     has.
+ *			     The ioctl waits for the reply of the device. When
+ *			     none comes, the outcome of the write is unknown and
+ *			     the driver sends that device no further command.
+ *			     The driver reads the
+ *			     bytes at @config_addr before it builds the request,
+ *			     so writing to the buffer afterwards changes nothing,
+ *			     and it serialises the calls of one enclave against
+ *			     each other and
+ *			     every call against the other commands it sends the
+ *			     NE PCI device. A call the driver refuses sends nothing, so
+ *			     the device keeps the configuration the last accepted
+ *			     write left it.
+ *			     What the enclave sees while a write is applied is
+ *			     not specified: whether the driver inside it can read
+ *			     part of a range, whether every accepted write raises
+ *			     a notification of its own, and what a write leaves
+ *			     the bytes outside it holding, are the hypervisor's
+ *			     to answer.
+ *			     The ioctl can be invoked on the enclave fd, after an
+ *			     enclave is started.
+ *
+ * Context: Process context.
+ * Return:
+ * * 0					- Logic successfully completed.
+ * * -1					- There was a failure in the ioctl logic.
+ * On failure, errno is set to:
+ * * ENODEV				- The NE PCI device is not available.
+ * * EFAULT				- Argument or @config_addr read failure.
+ * * EINVAL				- The argument does not reach through
+ *					  @config_offset, @config_size is zero, or
+ *					  @config_size is larger than
+ *					  @max_config_update_size.
+ * * E2BIG				- The argument is longer than this kernel knows
+ *					  about and the excess is not all zero.
+ * * NE_ERR_NOT_IN_RUNNING_STATE	- The enclave is not running.
+ * * Error codes from the NE PCI device request.
+ */
+#define NE_UPDATE_DEVICE_CONFIG		_IOW(0xAE, 0x2C, struct ne_update_device_config)
+
+/**
+ * struct ne_update_device_config - Configuration to write to a device an
+ *				    enclave is already driving (in).
+ * @device_uid:		The device to write the configuration of, as
+ *			%NE_ADD_DEVICE reported it.
+ * @config_addr:	User space address of the @config_size bytes to write.
+ * @config_size:	Number of bytes to write, at least one and at most the
+ *			@max_config_update_size %NE_GET_CAPS reports.
+ * @config_offset:	Byte offset within the device's configuration space to
+ *			write them at.
+ */
+struct ne_update_device_config {
+	__u64	device_uid;
+	__u64	config_addr;
+	__u32	config_size;
+	__u32	config_offset;
+};
+
 #endif /* _UAPI_LINUX_NITRO_ENCLAVES_H_ */
-- 
2.47.1

[PATCH 40/40] nitro_enclaves: Document the driver side of Nitro Sandboxes

[Alexander Graf]

Users of Nitro Sandboxes and authors of their parent tooling need one
place that explains how the calls that Nitro Sandboxes add fit together.

Document the driver functionality that Nitro Sandboxes add in
ne_overview.rst: what NE_GET_CAPS reports, the launch flags
NE_CREATE_VM2 takes, direct mode, why a slot needs its info page first,
what NE_ADD_DEVICE names and returns, and the two directions an event
travels. Link the public AWS documentation of Nitro Sandboxes for the
user side.

Signed-off-by: Alexander Graf <graf@amazon.com>
---
 Documentation/virt/ne_overview.rst | 82 ++++++++++++++++++++++++++++++
 1 file changed, 82 insertions(+)

diff --git a/Documentation/virt/ne_overview.rst b/Documentation/virt/ne_overview.rst
index 3aa33d3f06cf2..dbf9261d0391e 100644
--- a/Documentation/virt/ne_overview.rst
+++ b/Documentation/virt/ne_overview.rst
@@ -270,3 +270,85 @@ The device holds every donated page until it is told to let go, so the driver
 takes the whole pool back when a probe fails after donating, when it is unbound
 and when the machine shuts down, and the pages reach the allocator only after
 that has returned. Pages the device will not give back stay with it.
+
+Nitro Sandboxes
+---------------
+
+Nitro Sandboxes [9] build on top of Nitro Enclaves technology. They provide
+a sandbox VM with parent backed virtio devices on a PCIe bus, so it can have
+block devices, network interfaces and a console like any other virtio guest,
+and the parent provides them from an ordinary user space process. The driver
+functionality that Nitro Sandboxes add is described here.
+
+NE_GET_CAPS reports what this kernel implements: NE_CAP_DEVICES covers
+NE_SET_INFO_PAGE, NE_ADD_DEVICE, NE_SET_NOTIFY_EVENTFD, NE_SET_VRING_CALL and
+NE_UPDATE_DEVICE_CONFIG. The argument carries its own length in struct_size and
+the kernel writes back how many bytes it filled, so a caller built against a
+header this kernel predates can tell a field this kernel never wrote from one it
+wrote a zero into.
+
+NE_CREATE_VM2 takes a word of launch flags, and NE_CREATE_VM is the same call
+with a zero one. NE_ENCLAVE_PCIE_MODE asks for a slot with a PCIe bus, and the
+other launch flags shape that bus or the slot. The flags in
+NE_ENCLAVE_FLAG_MATCHED_MASK are the ones the hypervisor enumerates, and two
+fields answer for them. ne_caps.supported_matched_flags reports which of them
+this kernel passes on, and is the same on every host.
+ne_caps.hypervisor_matched_flags reports which of them the host behind the
+device says it implements. A caller requires a flag in supported_matched_flags
+always, and in hypervisor_matched_flags as well when that field is non-zero.
+A zero hypervisor_matched_flags means that the host gave no answer, because it
+reports no flag or because there is no device to ask, and NE_CREATE_VM2 is then
+the only test. On a device that takes no launch flags at all, it fails with
+EOPNOTSUPP. For the flags in NE_ENCLAVE_FLAG_UNMATCHED_MASK the answer comes
+from allocating a slot with them. Bits outside the matched mask read zero in
+both fields whatever the hypervisor does with them, so a caller masks its
+request before comparing.
+
+Nitro Sandboxes introduce a mode called direct mode, selected with
+NE_ENCLAVE_DIRECT_MODE, in which the parent keeps access to most of the memory
+it donates to the sandbox. Direct mode disables attestation, but lets guest
+kernels that do not place their virtqueues in device-provided memory run in a
+sandbox. A slot in direct mode takes NE_MEMORY_SHARED allocations as memory
+regions.
+
+A slot needs its info page before it can be given a device. The device reports
+how the sandbox reaches each of its virtqueues by writing the info page, and
+NE_ADD_DEVICE reads those entries back out of it, so NE_SET_INFO_PAGE comes
+first; see `Memory pool`_.
+
+NE_ADD_DEVICE names the device with a type whose top 16 bits are the class, of
+which NE_DEVICE_TYPE_VIRTIO is the only one, and whose low 16 bits carry the
+virtio device id the sandbox sees. It carries the feature bits the parent offers
+for the device and the opening bytes of the device's configuration space, and
+names the NE_ALLOC_MEMORY allocations the device works out of. Each of those has
+to be NE_MEMORY_SHARED, because the parent drives its end of the queues through
+a mapping of one, and each is named with the id the sandbox knows that region
+by. The driver puts that id in the top 8 bits of the size it reports for every
+run of the allocation, which is how a device made of several regions tells them
+apart; the parent and the sandbox agree on the numbering and the driver does not
+read it. A device that works out of the sandbox's own memory regions says so
+with NE_DEVICE_NO_SHM_REGIONS and names no region. An allocation is a device's
+memory, an enclave's memory or an info page, never two of those, and the driver
+holds it until the enclave is gone, so closing the descriptor does not take the
+runs away from the device. The call returns the id the device was given and, for
+each of its virtqueues, the interrupt vector, the notify index and the doorbell
+offset. It takes a device on a running sandbox when the slot was allocated with
+NE_ENCLAVE_PCIE_HOTPLUG_MODE.
+
+Notifications travel both ways. Towards the parent, the device signals through
+an interrupt multiplexer built into the nitro_enclaves device: it raises one of
+a pool of MSI-X vectors, and records in a bitmap page per vector, which the
+driver donates, which notifications were raised. NE_SET_NOTIFY_EVENTFD binds an
+eventfd to one of them, which is the kick of one of the sandbox's virtqueues or
+the change of a device's entry in the info page. Towards the sandbox,
+NE_SET_VRING_CALL binds an eventfd whose signal rings a virtqueue doorbell, so
+the sandbox learns that the queue has data waiting for it. Either takes
+NE_EVENTFD_UNBIND in place of a descriptor to release the binding, and binding
+one the caller already holds replaces the eventfd on it.
+
+NE_UPDATE_DEVICE_CONFIG writes the configuration space of a device a sandbox
+already has, naming it by the id NE_ADD_DEVICE returned.
+ne_caps.max_config_update_size bounds how many bytes one call moves, not how far
+into that space they reach.
+
+[9] https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/amazon-ec2-nitro-sandboxes.html
-- 
2.47.1