From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-qk1-f193.google.com (mail-qk1-f193.google.com [209.85.222.193]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 49EFC36828D for ; Fri, 19 Dec 2025 15:56:06 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.222.193 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1766159768; cv=none; b=edk0s1KCKv1XhAAsU3zVB7Y/bIjk8ND0dEJOLVeec4NdqovTAPKPeYukRJRbm/52wU302vkn3xS9MwwNoIdnPGm/9Nosd5lIARezZIpQcZq7h/V1bJhzxNyaSOdCAP06NmYICvX1x3veZ6vI4HJMgxs1/81EpafEUodtYBTyfH0= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1766159768; c=relaxed/simple; bh=udrDqH7m9tAEUdcwb9UBMMMgYYt3mEDbwsWLQknUbFY=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=bLdAnAcQIN4G6zTufPn/aU2rVGduUgxfTd9/+5uwZlKTIJaU6XEOmPE4rut6cSxlE9llKsnOnKFfP4m+l4+CGrP6bwQsxWb8r1qJygXfJkZbt6aR81VMefp5xo2ahJt8vcPf4MXilURpLnfSX6wpC4+61jIofP9NgMPmb+d66LY= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=gourry.net; spf=pass smtp.mailfrom=gourry.net; dkim=pass (2048-bit key) header.d=gourry.net header.i=@gourry.net header.b=G+O9A6kM; arc=none smtp.client-ip=209.85.222.193 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=gourry.net Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gourry.net Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gourry.net header.i=@gourry.net header.b="G+O9A6kM" Received: by mail-qk1-f193.google.com with SMTP id af79cd13be357-8b5c81bd953so215285485a.1 for ; Fri, 19 Dec 2025 07:56:06 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gourry.net; s=google; t=1766159765; x=1766764565; darn=vger.kernel.org; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=G252Wm2bHFmVmyOD5yAM+H+Ep/B9uA42o2RNtCi8kzo=; b=G+O9A6kMb02+kiS3XxkbgMctxscgr+GmleYISJKMgQpMsxDUWGzwtkP6ONRSoVScTO eVbJzTLIvJvAGz38bqngLJuAupAwS2YWbjU1rVC3JtszCKAwmTVtQpl0cIObS/LL4zpP f2phRvUmHh0EB/OCK0/swvH89I3/KAUlYrwFKYOJhYOB+6V1Pgptt+RAriyRTTP0dNa3 crb3tvEB+doy9lYrPydOdkrI3HJBWvJvhhCWRLaVpKRkwtgKatC1yg3NuLKnIlSd4QP/ ESib+8bUas8kyZ8gAVkjEY70wp4euBPBYvlH3zt+8nwnAz0KFCZ+Cty5rAfF6wIXyxyq DfSw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1766159765; x=1766764565; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-gg:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=G252Wm2bHFmVmyOD5yAM+H+Ep/B9uA42o2RNtCi8kzo=; b=gtzUYIIlKx6UqzFSw+5KsfQSFeXdtJi3fizD0SpdMZXI211yWL8KKB6/djhexX53C6 3geYnvUd6S+kPGJQ4vQo/JQtYpUEVTJ6ndUCNF1cknsSkRUne4iPDTM8CrxJTcDhIaeQ uw+r2LQwDK+7+8gVN9PujvuyUnqtae97nDzqeQ+F1thtgX6x8+AiMiMLyKjVlqzirEs0 QCfZd6rYnBvjcTmmxhMGOclXAeaRwbPFydQEMFLQFsSlItXCHT6WD9oQgZEHZH5MhBJ7 mF7oHnybUyOed8WnppIzc3d6wqBCVkR7RSGDY5PFcNOqHNVUn/i7iDTMxYAu/0YU7KXZ 6kHw== X-Forwarded-Encrypted: i=1; AJvYcCUheuugzjkkS3JmVTwTdOtUvWX649KkArSLiHq0Vu/JEgY2e9uIJt071FCm6XmSk+Rug6s+8Ad2KDH2178=@vger.kernel.org X-Gm-Message-State: AOJu0YzDs+RCJZbFZFTo2SsFkDWJqEv8sqX3X+vU5i3bCDu9u06ky7LB ZG/Rcw/8ntUQrD6gnmlBSBQSVbANzWaAF/5B/uPG9HFJzfiL7fqUldZFQQvOX+ziSqs= X-Gm-Gg: AY/fxX7ZGPLkVWXowzo5ePTMuXDa+X6BsRFMr/pMWp8wNIOVF6R4Po9soytPRU2JIDg EU8/SDqzuXnNppt12KYn5ckb5iFUkGqcioxZSK8+PhOkrXVej7qgkCI1t4kX+TlUCyZuORKcRW9 g6BeX4SHktcpFHPYEeXQr/ATInQsvvD5UTQw3+oRqoJP+xTT/vt/TaX+TxfwLxhg8TUr1/KrtC5 72EsvqzYH07lLxtqc3iTYV/PVCgt46W7MD8sXaAET/uFs/cgBLen1odk9ycgtwoheCvrdg3vbif cSKz+WNBDoVNkzgCxTFin1e15IviHa18ugoUbIMqLVX5jyNF6tfsNNDJZTPukNu0UJ/RBcAg0pY kYV3nOG9Ciur1T9G6/11r15e6WEaaDLjb41ugkIz8QNMpwCeF97rSAXX6Jw2AHx8J6FolCJi7j+ jZN/ldopw1LDb0JQgt0A+W8j8jzPFwMzL6KQRawp5CETwR7bbBTb2bXTzAOqjEQGTIeBuDp0Veh 4/UtS0m X-Google-Smtp-Source: AGHT+IGmULJofkH7FmYqyF/vaE/KL6pX3RmjYcu7YInfWOqUFvJXXmjUlxYtaAqFnuQKcGB8qosJEA== X-Received: by 2002:a05:620a:c53:b0:8b2:f182:6941 with SMTP id af79cd13be357-8c08fd222bfmr492220685a.57.1766159765204; Fri, 19 Dec 2025 07:56:05 -0800 (PST) Received: from gourry-fedora-PF4VCD3F (pool-96-255-20-138.washdc.ftas.verizon.net. [96.255.20.138]) by smtp.gmail.com with ESMTPSA id af79cd13be357-8c0973f0807sm205888485a.41.2025.12.19.07.56.04 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 19 Dec 2025 07:56:04 -0800 (PST) Date: Fri, 19 Dec 2025 10:55:27 -0500 From: Gregory Price To: Alejandro Lucero Palau Cc: linux-cxl@vger.kernel.org, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, kernel-team@meta.com, dave@stgolabs.net, jonathan.cameron@huawei.com, dave.jiang@intel.com, alison.schofield@intel.com, vishal.l.verma@intel.com, ira.weiny@intel.com, dan.j.williams@intel.com, corbet@lwn.net Subject: Re: [PATCH v2] Documentation/driver-api/cxl: device hotplug section Message-ID: References: <20251218170747.1278327-1-gourry@gourry.net> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: On Fri, Dec 19, 2025 at 03:06:21PM +0000, Alejandro Lucero Palau wrote: > > +To support memory hotplug directly on the host bridge, or on a switch > > +downstream of the host bridge (but not contained within a CXL memory device), > > +a platform must construct a CEDT CFMWS at boot with sufficient resources to > > +support the max possible (or expected) hotplug memory capacity. :: > > + > > + HB0 HB1 > > + RP0 RP1 RP2 > > + | | | > > + Empty Empty USP > > + ________|________ > > + | | | | > > + DSP DSP DSP DSP > > + | | | | > > + All Empty > > + > > +For example, a BIOS/EFI may expose an option to configure a CEDT CFMWS with > > +a pre-configured amount of memory capacity (per host bridge, or host bridge > > +interleave set), even if no device is attached to Root Ports or Downstream > > +Ports at boot (as depicted in the figure above). > > All this is fine, but my concern is what the BIOS will do when a device ends > up being hotplugged. It should do nothing more than what is required to bring the device up and make it visible on the PCIe/CXL link. BIOS should not be programmign the device at runtime. I realize this is likely infeasible for certain setups, like the current setup of Zen5 and it's address translation setup, but then I think most platforms right now don't really handle any of this. > Assuming that programmability from the Host/user space > for creating regions on demand based on requirements like bandwidth, and > therefore relying on interleaving and granularity (I know this is not > trivial but I think this is possible in the near future, or at least > possible to be demanded) then the BIOS should not do any HDM programming at > all ... > Correct. That said, I can only speak from a BIOS -> Linux transition perspective. I know enough about the ACPI spec, CXL spec, Linux MM, and the Linux driver to say "what linux expects" - but I have absolutely no idea what BIOS *actually does* at boot or hotplug time. If you have some of that perspective, please propose changes to the documentation based on what is feasible/possible. All I can do is write from the base of what I know. > I think it is worth to document this somehow and maybe to discuss this with > BIOSes vendors if we consider this convenient. > The entire point of this document is really to spell out that the BIOS is only responsible for setting up the environment *at boot*, and then at hotplug time the BIOS should do nothing wrt to programming stuff :] I can add this explicitly at the top of this document in v3 (and probably should be spelled out in the BIOS/EFI config doc) https://docs.kernel.org/driver-api/cxl/platform/bios-and-efi.html Would something like this suffice? """ Linux expects BIOS/EFI software to construct sufficient ACPI tables (such as CEDT, SRAT, HMAT, etc) and platform-specific configurations (such as HPA spaces and host-bridge interleave configurations) to allow the Linux driver to subsequently configure the devices in the CXL fabric at runtime. Programming of HDM decoders and switch ports is not required, and may be deferred to the CXL driver based on admin policy (e.g. udev rules). Some platforms may require pre-programming HDM decoders and locking them due to quirks (see: Zen5 address translation), but this is not the normal, "expected" configuration path. This should be avoided if possible. For hotplugged devices, BIOS/EFI software is expected to configure sufficient resources **at boot time** to allow hotplugged devices to be configured by software (such as proximity domains, HPA regions, and host-bridge configurations). BIOS/EFI is not expected (**nor suggested**) to configure hotplugged devices at hotplug time (i.e. HDM decoders should be leftunprogrammed). """ ~Gregory