[PATCH 40/40] nitro_enclaves: Document the driver side of Nitro Sandboxes

[Alexander Graf <graf@amazon.com>]

Users of Nitro Sandboxes and authors of their parent tooling need one
place that explains how the calls that Nitro Sandboxes add fit together.

Document the driver functionality that Nitro Sandboxes add in
ne_overview.rst: what NE_GET_CAPS reports, the launch flags
NE_CREATE_VM2 takes, direct mode, why a slot needs its info page first,
what NE_ADD_DEVICE names and returns, and the two directions an event
travels. Link the public AWS documentation of Nitro Sandboxes for the
user side.

Signed-off-by: Alexander Graf <graf@amazon.com>
---
 Documentation/virt/ne_overview.rst | 82 ++++++++++++++++++++++++++++++
 1 file changed, 82 insertions(+)

diff --git a/Documentation/virt/ne_overview.rst b/Documentation/virt/ne_overview.rst
index 3aa33d3f06cf2..dbf9261d0391e 100644
--- a/Documentation/virt/ne_overview.rst
+++ b/Documentation/virt/ne_overview.rst
@@ -270,3 +270,85 @@ The device holds every donated page until it is told to let go, so the driver
 takes the whole pool back when a probe fails after donating, when it is unbound
 and when the machine shuts down, and the pages reach the allocator only after
 that has returned. Pages the device will not give back stay with it.
+
+Nitro Sandboxes
+---------------
+
+Nitro Sandboxes [9] build on top of Nitro Enclaves technology. They provide
+a sandbox VM with parent backed virtio devices on a PCIe bus, so it can have
+block devices, network interfaces and a console like any other virtio guest,
+and the parent provides them from an ordinary user space process. The driver
+functionality that Nitro Sandboxes add is described here.
+
+NE_GET_CAPS reports what this kernel implements: NE_CAP_DEVICES covers
+NE_SET_INFO_PAGE, NE_ADD_DEVICE, NE_SET_NOTIFY_EVENTFD, NE_SET_VRING_CALL and
+NE_UPDATE_DEVICE_CONFIG. The argument carries its own length in struct_size and
+the kernel writes back how many bytes it filled, so a caller built against a
+header this kernel predates can tell a field this kernel never wrote from one it
+wrote a zero into.
+
+NE_CREATE_VM2 takes a word of launch flags, and NE_CREATE_VM is the same call
+with a zero one. NE_ENCLAVE_PCIE_MODE asks for a slot with a PCIe bus, and the
+other launch flags shape that bus or the slot. The flags in
+NE_ENCLAVE_FLAG_MATCHED_MASK are the ones the hypervisor enumerates, and two
+fields answer for them. ne_caps.supported_matched_flags reports which of them
+this kernel passes on, and is the same on every host.
+ne_caps.hypervisor_matched_flags reports which of them the host behind the
+device says it implements. A caller requires a flag in supported_matched_flags
+always, and in hypervisor_matched_flags as well when that field is non-zero.
+A zero hypervisor_matched_flags means that the host gave no answer, because it
+reports no flag or because there is no device to ask, and NE_CREATE_VM2 is then
+the only test. On a device that takes no launch flags at all, it fails with
+EOPNOTSUPP. For the flags in NE_ENCLAVE_FLAG_UNMATCHED_MASK the answer comes
+from allocating a slot with them. Bits outside the matched mask read zero in
+both fields whatever the hypervisor does with them, so a caller masks its
+request before comparing.
+
+Nitro Sandboxes introduce a mode called direct mode, selected with
+NE_ENCLAVE_DIRECT_MODE, in which the parent keeps access to most of the memory
+it donates to the sandbox. Direct mode disables attestation, but lets guest
+kernels that do not place their virtqueues in device-provided memory run in a
+sandbox. A slot in direct mode takes NE_MEMORY_SHARED allocations as memory
+regions.
+
+A slot needs its info page before it can be given a device. The device reports
+how the sandbox reaches each of its virtqueues by writing the info page, and
+NE_ADD_DEVICE reads those entries back out of it, so NE_SET_INFO_PAGE comes
+first; see `Memory pool`_.
+
+NE_ADD_DEVICE names the device with a type whose top 16 bits are the class, of
+which NE_DEVICE_TYPE_VIRTIO is the only one, and whose low 16 bits carry the
+virtio device id the sandbox sees. It carries the feature bits the parent offers
+for the device and the opening bytes of the device's configuration space, and
+names the NE_ALLOC_MEMORY allocations the device works out of. Each of those has
+to be NE_MEMORY_SHARED, because the parent drives its end of the queues through
+a mapping of one, and each is named with the id the sandbox knows that region
+by. The driver puts that id in the top 8 bits of the size it reports for every
+run of the allocation, which is how a device made of several regions tells them
+apart; the parent and the sandbox agree on the numbering and the driver does not
+read it. A device that works out of the sandbox's own memory regions says so
+with NE_DEVICE_NO_SHM_REGIONS and names no region. An allocation is a device's
+memory, an enclave's memory or an info page, never two of those, and the driver
+holds it until the enclave is gone, so closing the descriptor does not take the
+runs away from the device. The call returns the id the device was given and, for
+each of its virtqueues, the interrupt vector, the notify index and the doorbell
+offset. It takes a device on a running sandbox when the slot was allocated with
+NE_ENCLAVE_PCIE_HOTPLUG_MODE.
+
+Notifications travel both ways. Towards the parent, the device signals through
+an interrupt multiplexer built into the nitro_enclaves device: it raises one of
+a pool of MSI-X vectors, and records in a bitmap page per vector, which the
+driver donates, which notifications were raised. NE_SET_NOTIFY_EVENTFD binds an
+eventfd to one of them, which is the kick of one of the sandbox's virtqueues or
+the change of a device's entry in the info page. Towards the sandbox,
+NE_SET_VRING_CALL binds an eventfd whose signal rings a virtqueue doorbell, so
+the sandbox learns that the queue has data waiting for it. Either takes
+NE_EVENTFD_UNBIND in place of a descriptor to release the binding, and binding
+one the caller already holds replaces the eventfd on it.
+
+NE_UPDATE_DEVICE_CONFIG writes the configuration space of a device a sandbox
+already has, naming it by the id NE_ADD_DEVICE returned.
+ne_caps.max_config_update_size bounds how many bytes one call moves, not how far
+into that space they reach.
+
+[9] https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/amazon-ec2-nitro-sandboxes.html
-- 
2.47.1