[PATCH 09/40] nitro_enclaves: Document the memory pool

[Alexander Graf <graf@amazon.com>]

An admin reserving memory for enclaves needs to pick the CMA area and
the kernel parameters that fill it. Document the pool: allocating with
NE_ALLOC_MEMORY, writing the image through the returned descriptor and
adding it with NE_ADD_MEMORY_REGION, the config options a reservation
needs to be reachable, and the difference between private and shared
allocations and how long an allocation lives.

Signed-off-by: Alexander Graf <graf@amazon.com>
---
 Documentation/virt/ne_overview.rst | 83 ++++++++++++++++++++++++++++--
 1 file changed, 80 insertions(+), 3 deletions(-)

diff --git a/Documentation/virt/ne_overview.rst b/Documentation/virt/ne_overview.rst
index 223ee7fdec51b..4c4b7febaff0a 100644
--- a/Documentation/virt/ne_overview.rst
+++ b/Documentation/virt/ne_overview.rst
@@ -113,9 +113,10 @@ each node in turn, and sets numa_node to NE_NUMA_ANY when it does not
 care. Both start from a free thread of a core the enclave already holds,
 and only take a new core once there is none.
 
-The node says where cores come from. An enclave built out of a pool that
-spans nodes has no node of its own, so the rule that its memory be on
-that node does not apply to it and its memory can come from any node.
+The node says where cores come from. NE_ALLOC_MEMORY takes its own node;
+see `Memory pool`_. An enclave built out of a pool that spans
+nodes has no node of its own, so the rule that its memory be on that node
+does not apply to it and its memory can come from any node.
 
 The pool occupancy is readable under
 /sys/devices/virtual/misc/nitro_enclaves/cpu_pool/, and per node under
@@ -124,3 +125,79 @@ online when the device was probed. A node the pool holds nothing on
 reads empty rather than being absent. The driver offlines pool threads,
 so ``nodeN/cpulist`` under /sys/devices/system/node stops listing them
 and answers nothing about the pool.
+
+Memory pool
+-----------
+
+Enclave memory must be physically contiguous, in chunks of at least 2 MiB,
+and, for an enclave whose cores all sit on one node, on that node. It has
+traditionally come from hugetlbfs [2], which provides the hugepage sizes the
+architecture has -- 2 MiB and 1 GiB on x86, and 32 MiB, 512 MiB or 16 GiB on
+arm64 depending on the page size -- and needs the hugepage count managed out
+of band.
+
+The driver can hand that memory out itself, backed per NUMA node by the
+kernel's Contiguous Memory Allocator (CMA). Reserve an area on every node
+you intend to run enclaves on, with the numa_cma= or cma_pernuma= kernel
+parameters documented alongside the rest of them [4]. On a host with one
+online node, cma= reserves the one area there is to reserve and the driver
+uses it.
+
+Reaching a reserved area takes CONFIG_DMA_CMA, which NITRO_ENCLAVES selects,
+so the driver always has it. numa_cma= and cma_pernuma= need
+CONFIG_DMA_NUMA_CMA on top of that, which a host with more than one online
+node enables to name the node a reservation lands on. cma= needs neither, so
+on a host with one online node the driver's own select is the whole
+requirement.
+
+NE_ALLOC_MEMORY on the enclave file descriptor allocates 2 MiB aligned,
+physically contiguous memory from one host NUMA node's CMA area. A request
+that area cannot serve fails with -ENOMEM, so the reservation bounds what the
+enclaves on a node can take between them.
+It returns a file descriptor for the allocation. For an enclave whose cores
+all sit on one node the ioctl refuses any other node, so name the same node
+here and in NE_ADD_ANY_VCPU. An enclave built out of a pool that spans nodes
+has no node of its own and takes memory from any of them.
+
+An allocation is NE_MEMORY_PRIVATE unless the flags say NE_MEMORY_SHARED.
+A private one cannot be mapped, and mmap() on it returns ENODEV. Writing to
+the descriptor places bytes in either kind, which is how an enclave image
+gets into one. There is no read operation on the descriptor.
+
+A shared allocation can also be mapped, at any 2 MiB aligned offset for a
+multiple of 2 MiB, with MAP_SHARED. A caller maps one to keep an address for
+memory it goes on using after handing it over, including after the allocation
+has been donated, which is where writing to the descriptor stops working. Such
+a mapping is readable too, and is the only way to read allocated memory back.
+
+The mapping is VM_PFNMAP, so it holds no struct page references and any DMA pin
+over it returns EFAULT, long-term or not; io_uring buffer registration and RDMA
+memory registration are two that a caller might expect to work and that do not.
+Reaching enclave memory from a device in the primary VM is not something this
+interface offers.
+
+The allocation is accounted against RLIMIT_MEMLOCK. NE_ADD_MEMORY_REGION
+takes the descriptor and gives the enclave the whole allocation, as one
+memory region per contiguous run it is made of, or fewer where runs turn out
+to be physically adjacent. It also ends writing: the memory is the enclave's
+from that point, so an image goes in first. It takes private allocations
+only, since memory the parent can still reach is not memory the enclave has
+to itself.
+
+An enclave's memory is laid out in the order its allocations are given to it,
+and the offset NE_GET_IMAGE_LOAD_INFO returns is an offset into that layout.
+A caller building an enclave out of several allocations therefore picks the
+order first, writes at the offsets that order implies, and gives them away in
+it.
+
+A mapping is populated as it is touched, so it costs page tables for the part
+a caller reaches and not for all of it. It keeps the memory reachable from the
+parent for as long as it lives, so an allocation that is only being handed to
+an enclave stays private.
+
+NE_SET_USER_MEMORY_REGION is unchanged and still takes the address of a
+hugetlbfs region.
+
+The allocation lives until the last of its descriptor, its mappings and the
+regions given from it is gone, so neither closing the descriptor nor
+unmapping takes memory away from a running enclave.
-- 
2.47.1