mirror of https://lore.kernel.org/lkml/
 help / color / mirror / Atom feed
From: Randy Dunlap <rdunlap@infradead.org>
To: Gregory Price <gourry@gourry.net>, linux-cxl@vger.kernel.org
Cc: linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org,
	kernel-team@meta.com, dave@stgolabs.net,
	jonathan.cameron@huawei.com, dave.jiang@intel.com,
	alison.schofield@intel.com, vishal.l.verma@intel.com,
	ira.weiny@intel.com, dan.j.williams@intel.com, corbet@lwn.net
Subject: Re: [RFC PATCH v2 02/18] cxl: docs/devices - device reference and uefi placeholder
Date: Sat, 10 May 2025 18:57:05 -0700	[thread overview]
Message-ID: <d8eadf5e-1b9f-4020-95de-cde60e4bee48@infradead.org> (raw)
In-Reply-To: <20250430181048.1197475-3-gourry@gourry.net>



On 4/30/25 11:10 AM, Gregory Price wrote:
> Add a simple device primer sufficient to understand the theory
> of operation documentation.
> 
> Add carve-out for CDAT with a TODO.
> 
> Signed-off-by: Gregory Price <gourry@gourry.net>
> ---
>  .../driver-api/cxl/devices/device-types.rst   | 170 ++++++++++++++++++
>  Documentation/driver-api/cxl/devices/uefi.rst |  10 ++
>  Documentation/driver-api/cxl/index.rst        |   7 +
>  3 files changed, 187 insertions(+)
>  create mode 100644 Documentation/driver-api/cxl/devices/device-types.rst
>  create mode 100644 Documentation/driver-api/cxl/devices/uefi.rst
> 
> diff --git a/Documentation/driver-api/cxl/devices/device-types.rst b/Documentation/driver-api/cxl/devices/device-types.rst
> new file mode 100644
> index 000000000000..dfe8d4711987
> --- /dev/null
> +++ b/Documentation/driver-api/cxl/devices/device-types.rst
> @@ -0,0 +1,170 @@
> +.. SPDX-License-Identifier: GPL-2.0
> +
> +=====================
> +Devices and Protocols
> +=====================
> +
> +The type of CXL device (Memory, Accelerator, etc) dictates many configuration steps. This section
> +covers some basic background on device types and on-device resources used by the platform and OS
> +which impact configuration.
> +
> +Protocols
> +=========
> +
> +There are three core protocols to CXL.  For the purpose of this documentation,
> +we will only discuss very high level definitions as the specific hardware
> +details are largely abstracted away from Linux.  See the CXL specification
> +for more details.
> +
> +CXL.io
> +------
> +The basic interaction protocol, similar to PCIe configuration mechanisms.
> +Typically used for initialization, configuration, and I/O access for anything
> +other than memory (CXL.mem) or cache (CXL.cache) operations.
> +
> +The Linux CXL driver exposes access to .io functionalty via the various sysfs
> +interfaces and /dev/cxl/ devices (which exposes direct access to device
> +mailboxes).
> +
> +CXL.cache
> +---------
> +The mechanism by which a device may coherently access and cache host memory.
> +
> +Largely transparent to Linux once configured.
> +
> +CXL.mem
> +---------
> +The mechanism by which the CPU may coherently access and cache device memory.
> +
> +Largely transparent to Linux once configured.
> +
> +
> +Device Types
> +============
> +
> +Type-1
> +------
> +
> +A Type-1 CXL device:
> +
> +* Supports cxl.io and cxl.cache protocols
> +* Implements a fully coherent cache
> +* Allow Device-to-Host coherence and Host-to-Device snoops.

     Allows

> +* Does NOT have host-managed device memory (HDM)
> +
> +Typical examples of type-1 devices is a Smart NIC - which may want to
> +directly operate on host-memory (DMA) to store incoming packets. These
> +devices largely rely on CPU-attached memory.
> +
> +Type-2
> +------
> +
> +A Type-2 CXL Device:
> +
> +* Supports cxl.io, cxl.cache, and cxl.mem protocols
> +* Optionally implements coherent cache and Host-Managed Device Memory
> +* Is typically an accelerator device w/ high bandwidth memory.
> +
> +The primary difference between a type-1 and type-2 device is the presence
> +of host-managed device memory, which allows the device to operate on a
> +local memory bank - while the CPU sill has coherent DMA to the same memory.
> +
> +The allows things like GPUs to expose their memory via DAX devices or file
> +descriptors, allows drivers and programs direct access to device memory
> +rather than use block-transfer semantics.
> +
> +Type-3
> +------
> +
> +A Type-3 CXL Device
> +
> +* Supports cxl.io and cxl.mem
> +* Implements Host-Managed Device Memory
> +* May provide either Volatile or Persistent memory capacity (or both).
> +
> +A basic example of a type-3 device is a simple memory expanded, whose

                                                         expander  ?

> +local memory capacity is exposed to the CPU for access directly via
> +basic coherent DMA.
> +
> +Switch
> +------
> +
> +A CXL switch is a device capacity of routing any CXL (and by extension, PCIe)
> +protocol between an upstream, downstream, or peer devices.  Many devices, such
> +as Multi-Logical Devices, imply the presence of switching in some manner.
> +
> +Logical Devices and Heads
> +-------------------------
> +
> +A CXL device may present one or more "Logical Devices" to one or more hosts
> +(via physical "Heads").
> +
> +A Single-Logical Device (SLD) is a device which presents a single device to
> +one or more heads.
> +
> +A Multi-Logical Device (MLD) is a device which may present multiple devices
> +to one or more devices.
> +
> +A Single-Headed Device exposes only a single physical connection.
> +
> +A Multi-Headed Device exposes multiple physical connections.
> +
> +MHSLD
> +~~~~~
> +A Multi-Headed Single-Logical Device (MHSLD) exposes a single logical
> +device to multiple heads which may be connected to one or more discrete
> +hosts.  An example of this would be a simple memory-pool which may be
> +statically configured (prior to boot) to expose portions of its memory
> +to Linux via the CEDT ACPI table.
> +
> +MHMLD
> +~~~~~
> +A Multi-Headed Multi-Logical Device (MHMLD) exposes multiple logical
> +devices to multiple heads which may be connected to one or more discrete
> +hosts.  An example of this would be a Dynamic Capacity Device or which
> +may be configured at runtime to expose portions of its memory to Linux.
> +
> +Example Devices
> +===============
> +
> +Memory Expander
> +---------------
> +The simplest form of Type-3 device is a memory expander.  A memory expander
> +exposes Host-Managed Device Memory (HDM) to Linux.  This memory may be
> +Volatile or Non-Volatile (Persistent).
> +
> +Memory Expanders will typically be considered a form of Single-Headed,
> +Single-Logical Device - as its form factor will typically be an add-in-card
> +(AIC) or some other similar form-factor.
> +
> +The Linux CXL driver provides support for static or dynamic configuration of
> +basic memory expanders.  The platform may program decoders prior to OS init
> +(e.g. auto-decoders), or the user may program the fabric if the platform
> +defers these operations to the OS.
> +
> +Multiple Memory Expanders may be added to an external chassis and exposed to
> +a host via a head attached to a CXL switch.  This is a "memory pool", and
> +would be considered an MHSLD or MHMLD depending on the management capabilities
> +provided by the switch platform.
> +
> +As of v6.14, Linux does not provide a formalized interface to manage non-DCD
> +MHSLD or MHMLD devices.
> +
> +Dynamic Capacity Device (DCD)
> +-----------------------------
> +
> +A Dynamic Capacity Device is a Type-3 device which provides dynamic management
> +of memory capacity. The basic premise of a DCD to provide an allocator-like
> +interface for physical memory capacity to a "Fabric Manager" (an external,
> +privileged host with privileges to change configurations for other hosts).
> +
> +A DCD manages "Memory Extents", which may be volatile or persistent. Extents
> +may also be exclusive to a single host or shared across multiple.

                                                           multiple hosts.

> +
> +As of v6.14, Linux does not provide a formalized interface to manage DCD
> +devices, however there is active work on LKML targeting future release.
> +
> +Type-2 Device
> +-------------
> +
> +Todo


-- 
~Randy


  reply	other threads:[~2025-05-11  1:57 UTC|newest]

Thread overview: 32+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2025-04-30 18:10 [RFC PATCH v2 00/18] CXL Boot to Bash Documentation Gregory Price
2025-04-30 18:10 ` [RFC PATCH v2 01/18] cxl: update documentation structure in prep for new docs Gregory Price
2025-05-11  7:34   ` Bagas Sanjaya
2025-04-30 18:10 ` [RFC PATCH v2 02/18] cxl: docs/devices - device reference and uefi placeholder Gregory Price
2025-05-11  1:57   ` Randy Dunlap [this message]
2025-04-30 18:10 ` [RFC PATCH v2 03/18] cxl: docs/platform/bios-and-efi documentation Gregory Price
2025-04-30 18:10 ` [RFC PATCH v2 04/18] cxl: docs/platform/acpi reference documentation Gregory Price
2025-05-11  2:02   ` Randy Dunlap
2025-04-30 18:10 ` [RFC PATCH v2 05/18] cxl: docs/platform/example-configs documentation Gregory Price
2025-04-30 18:10 ` [RFC PATCH v2 06/18] cxl: docs/linux - overview Gregory Price
2025-05-11  2:06   ` Randy Dunlap
2025-04-30 18:10 ` [RFC PATCH v2 07/18] cxl: docs/linux - early boot configuration Gregory Price
2025-05-11  2:11   ` Randy Dunlap
2025-04-30 18:10 ` [RFC PATCH v2 08/18] cxl: docs/linux - add cxl-driver theory of operation Gregory Price
2025-04-30 18:10 ` [RFC PATCH v2 09/18] cxl: docs/linux/cxl-driver - add example configurations Gregory Price
2025-04-30 18:10 ` [RFC PATCH v2 10/18] cxl: docs/linux/dax-driver documentation Gregory Price
2025-05-11  2:18   ` Randy Dunlap
2025-05-12 14:31     ` Gregory Price
2025-04-30 18:10 ` [RFC PATCH v2 11/18] cxl: docs/linux/memory-hotplug Gregory Price
2025-05-11  2:22   ` Randy Dunlap
2025-04-30 18:10 ` [RFC PATCH v2 12/18] cxl: docs/allocation/dax Gregory Price
2025-04-30 18:10 ` [RFC PATCH v2 13/18] cxl: docs/allocation/page-allocator Gregory Price
2025-05-11  2:28   ` Randy Dunlap
2025-04-30 18:10 ` [RFC PATCH v2 14/18] cxl: docs/allocation/reclaim Gregory Price
2025-05-11  2:31   ` Randy Dunlap
2025-04-30 18:10 ` [RFC PATCH v2 15/18] cxl: docs/allocation/hugepages Gregory Price
2025-04-30 18:10 ` [RFC PATCH v2 16/18] cxl: docs/allocation/tiering Gregory Price
2025-04-30 18:10 ` [RFC PATCH v2 17/18] cxl: docs/use-cases Gregory Price
2025-04-30 18:10 ` [RFC PATCH v2 18/18] cxl: docs - add self-referencing cross-links Gregory Price
2025-05-11  2:54   ` Randy Dunlap
2025-05-11  9:14   ` Bagas Sanjaya
2025-04-30 23:15 ` [PATCH FIXUP] cxl: docs - fixup cedt.rst reference in access-coordinates Gregory Price

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=d8eadf5e-1b9f-4020-95de-cde60e4bee48@infradead.org \
    --to=rdunlap@infradead.org \
    --cc=alison.schofield@intel.com \
    --cc=corbet@lwn.net \
    --cc=dan.j.williams@intel.com \
    --cc=dave.jiang@intel.com \
    --cc=dave@stgolabs.net \
    --cc=gourry@gourry.net \
    --cc=ira.weiny@intel.com \
    --cc=jonathan.cameron@huawei.com \
    --cc=kernel-team@meta.com \
    --cc=linux-cxl@vger.kernel.org \
    --cc=linux-doc@vger.kernel.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=vishal.l.verma@intel.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox

all inboxes | Powered by JetHome®