[PATCH v4 3/3] Documentation: add documentation for MFD_MF_KEEP_UE_MAPPED

[Jiaqi Yan <jiaqiyan@google.com>]

Document its motivation, userspace API, behaviors, and limitations.

Signed-off-by: Jiaqi Yan <jiaqiyan@google.com>
---
 Documentation/userspace-api/index.rst         |  1 +
 .../userspace-api/mfd_mfr_policy.rst          | 81 +++++++++++++++++++
 2 files changed, 82 insertions(+)
 create mode 100644 Documentation/userspace-api/mfd_mfr_policy.rst

diff --git a/Documentation/userspace-api/index.rst b/Documentation/userspace-api/index.rst
index a68b1bea57a8..9c40445ea354 100644
--- a/Documentation/userspace-api/index.rst
+++ b/Documentation/userspace-api/index.rst
@@ -69,3 +69,4 @@ Everything else
    futex2
    perf_ring_buffer
    ntsync
+   mfd_mfr_policy
diff --git a/Documentation/userspace-api/mfd_mfr_policy.rst b/Documentation/userspace-api/mfd_mfr_policy.rst
new file mode 100644
index 000000000000..a5441bfbe78b
--- /dev/null
+++ b/Documentation/userspace-api/mfd_mfr_policy.rst
@@ -0,0 +1,81 @@
+.. SPDX-License-Identifier: GPL-2.0
+
+==================================================
+Userspace Memory Failure Recovery Policy via memfd
+==================================================
+
+:Author:
+    Jiaqi Yan <jiaqiyan@google.com>
+
+
+Motivation
+==========
+
+When a userspace process is able to recover from memory failures (MF)
+caused by uncorrected memory error (UE) in the DIMM, especially when it is
+able to avoid consuming known UEs, keeping the memory page mapped and
+accessible is beneficial to the owning process for a couple of reasons:
+
+- The memory pages affected by UE have a large smallest granularity, for
+  example 1G hugepage, but the actual corrupted amount of the page is only
+  several cachelines. Losing the entire hugepage of data is unacceptable to
+  the application.
+
+- In addition to keeping the data accessible, the application still wants
+  to access with a large page size for the fastest virtual-to-physical
+  translations.
+
+Memory failure recovery for 1G or larger HugeTLB is a good example. With
+memfd userspace process can control whether the kernel hard offlines its
+hugepages that backs the in-RAM file created by memfd.
+
+
+User API
+========
+
+``int memfd_create(const char *name, unsigned int flags)``
+
+``MFD_MF_KEEP_UE_MAPPED``
+
+	When ``MFD_MF_KEEP_UE_MAPPED`` bit is set in ``flags``, MF recovery
+	in the kernel does not hard offline memory due to UE until the
+	returned ``memfd`` is released. IOW, the HWPoison-ed memory remains
+	accessible via the returned ``memfd`` or the memory mapping created
+	with the returned ``memfd``. Note the affected memory will be
+	immediately isolated and prevented from future use once the memfd
+	is closed. By default ``MFD_MF_KEEP_UE_MAPPED`` is not set, and
+	kernel hard offlines memory having UEs.
+
+Notes about the behavior and limitations
+
+- Even if the page affected by UE is kept, a portion of the (huge)page is
+  already lost due to hardware corruption, and the size of the portion
+  is the smallest page size that kernel uses to manage memory on the
+  architecture, i.e. PAGE_SIZE. Accessing a virtual address within any of
+  these parts results in a SIGBUS (BUS_MCEERR_AR or BUS_MCEERR_AO) with
+  si_addr_lsb = PAGE_SHIFT. Accessing virtual address outside these
+  parts is good until it is corrupted by new UEs.
+
+- ``MFD_MF_KEEP_UE_MAPPED`` currently only works for HugeTLB, so
+  ``MFD_HUGETLB`` must also be set when setting ``MFD_MF_KEEP_UE_MAPPED``.
+  Otherwise ``memfd_create`` returns EINVAL.
+
+- For ``MAP_PRIVATE`` mappings, even with ``MFD_MF_KEEP_UE_MAPPED``,
+  the first write anywhere in a hugepage that has memory error fails
+  for the whole hugepage, because the copy-on-write must copy the
+  entire hugepage and inevitably consumes the UEs. The result is
+  likely a SIGBUS BUS_MCEERR_AR with si_addr_lsb = 30 for 1G hugepage,
+  as modern x86 hardware usually supports CONFIG_ARCH_HAS_COPY_MC to
+  recover various in-kernel memcpy routines from memory error.
+
+- ``MFD_MF_KEEP_UE_MAPPED`` works with ``MFD_ALLOW_SEALING``, and file
+  seals work as usual. Seals, including ``F_SEAL_WRITE``, can still be
+  added after hugepage having UEs, since memory failure recovery doesn't
+  hold extra references on the kept hugepages.
+
+- Truncate, hole punch, and close operations from userspace all dissolve
+  the hugepage. If dissolving fails, the HWPoison-ed HugeTLB page stays
+  out of use. Note that ``F_SEAL_SHRINK``, ``F_SEAL_WRITE`` and
+  ``F_SEAL_FUTURE_WRITE`` prevent truncating or punching holes in the
+  memfd, so a HWPoison-ed hugepage in a sealed memfd is only dissolved
+  when the memfd is finally released.
-- 
2.56.0.611.g1030f0b940-goog