mirror of https://lore.kernel.org/lkml/
 help / color / mirror / Atom feed
From: Srirangan Madhavan <smadhavan@nvidia.com>
To: "Lucero Palau, Alejandro" <alejandro.lucero-palau@amd.com>,
	Alison Schofield <alison.schofield@intel.com>,
	Bjorn Helgaas <bhelgaas@google.com>,
	Dave Jiang <dave.jiang@intel.com>,
	Davidlohr Bueso <dave@stgolabs.net>,
	Ira Weiny <iweiny@kernel.org>,
	Jonathan Cameron <jic23@kernel.org>,
	Vishal Verma <vishal.l.verma@intel.com>,
	linux-cxl@vger.kernel.org, linux-pci@vger.kernel.org,
	linux-kernel@vger.kernel.org
Cc: Li Ming <ming.li@zohomail.com>,
	Joshua Machauer <machaujo@amazon.de>,
	Alex Williamson <alex.williamson@redhat.com>,
	vsethi@nvidia.com, alwilliamson@nvidia.com,
	Sai Yashwanth Reddy Kancherla <skancherla@nvidia.com>,
	Vishal Aslot <vaslot@nvidia.com>,
	Manish Honap <mhonap@nvidia.com>, Jiandi An <jan@nvidia.com>,
	Richard Cheng <icheng@nvidia.com>,
	linux-tegra@vger.kernel.org
Subject: Re: [PATCH v17 12/19] PCI/CXL: Add CXL Device Reset sequencing
Date: Fri, 20 Nov 2026 15:14:30 -0800	[thread overview]
Message-ID: <8a63033e-8f9a-4da0-95db-a3310cc0e585@nvidia.com> (raw)
In-Reply-To: <f5fdc266-aed3-47e6-a10f-e8188b58c93f@amd.com>

On 11/11/26 2:12 AM, Lucero Palau, Alejandro wrote:
> External email: Use caution opening links or attachments
> 
> 
> On 05/11/2026 18:59, Srirangan Madhavan wrote:
>> Implement CXL Device Reset for eligible Type 2 devices: disable CXL.cache,
>> perform supported writeback, drain transactions, prepare the IOMMU,
>> initiate reset with Memory Clear enabled, and wait for completion.
>>
>> Observe the 100 ms post-reset quiet period before polling Status2, and
>> treat an all-ones read as not ready. Wait for the timeout the device
>> advertises, up to 100 seconds, and access only Status2 until reset
>> completion, error or expiry, as CXL 4.0 section 9.7.3 requires. Control2
>> cleanup runs only afterwards. Treat Control2 command bits as
>> transient state, restore cache policy on exit, preserve the primary error,
>> and abort when IOMMU preparation fails.
>>
>> Reuse Device DVSEC discovery from cache initialization and keep the
>> reset entry point declaration private to PCI core.
>>
>> Signed-off-by: Srirangan Madhavan <smadhavan@nvidia.com>
> 
> 
> Hi Srirangan,
> 
> 
> A concern regarding IOMMU handling below. If it turns to be unfounded,
> you can add my review tag.
> 
> 
> Thank you,
> 
> Alejandro
> 
> 
> <snip>
> 
>> +static int __cxl_reset_execute(struct pci_dev *pdev, int dvsec, u16 cap)
>> +{
>> +     int rc, rc2;
>> +
>> +     if (!pci_wait_for_pending_transaction(pdev)) {
>> +             pci_err(pdev, "timed out waiting for pending transactions\n");
>> +             return -ETIMEDOUT;
>> +     }
>> +
>> +     rc = pci_dev_reset_iommu_prepare(pdev);
> 
> 
> I'm not sure if this is safe enough. I can see a situation where a Type2
> device could not be using CXL.mem and CXL.cache but CXL.io. This would
> imply no CXL initialization from the Type2 driver but:
> 
> 
>      1) Such a driver could be giving functionality without CXL
> 
>      2) The device belonging to an IOMMU domain which such functionality
> is using
> 
>      3) The HDM decoder range as a system resource is free.
> 
> 
> If I am right, a CXL reset could happen, and although it should not be
> triggered by user space under these circumstances, the door is open ...
> 
> 
> The solution would be, currently, to check for a  driver not bound to
> the device in an IOMMU enabled system. I say currently because CXL.cache
> will need extra work with IOMMU mapping which are not yet there.

Hi Alejandro,

Thanks for the review.

I don't think I can add a bound-driver check in this series. For VFIO
passthrough, vfio-pci is always the bound driver at every reset VFIO
starts, and VFIO needs an IOMMU outside no-IOMMU mode, so the check would
always fire. VFIO resets through the PCI reset methods, so a refusal would
make every reset of a Type 2 device fail or fall back to FLR, which isn't
what passthrough needs. Manish describes the vfio-cxl side in his reply
to patch 15 of the v6 series:
https://lore.kernel.org/linux-cxl/20261016161839.2765592-1-mhonap@nvidia.com/

I guess the same exposure also exists for FLR and the other methods, which go 
through the same pci_dev_reset_iommu_prepare()?
For now, the reset file is root-only, and a CXL.io-only device doesn't reach 
cxl_reset: cxl_reset_function() needs CXL.cache and CXL.mem enabled, otherwise 
it returns -ENOTTY and the normal methods are used.
If this is okay, can we revisit the point once vfio-cxl's reset model is settled 
and the CXL.cache IOMMU side lands as you mentioned?

-- 
Regards,
Srirangan

  reply	other threads:[~2026-11-20 23:14 UTC|newest]

Thread overview: 44+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-11-05 18:58 [PATCH v17 00/19] PCI/CXL: Add CXL reset support for Type 2 devices Srirangan Madhavan
2026-11-05 18:58 ` [PATCH v17 01/19] cxl: Clarify decoder interleave limit Srirangan Madhavan
2026-11-05 18:59 ` [PATCH v17 02/19] cxl: Share CXL port upstream PCI device lookup Srirangan Madhavan
2026-11-10 13:36   ` Li Ming
2026-11-13 15:50     ` Dave Jiang
2026-11-15  3:25       ` Li Ming
2026-11-05 18:59 ` [PATCH v17 03/19] cxl: Move decoder declarations to shared header Srirangan Madhavan
2026-11-05 18:59 ` [PATCH v17 04/19] cxl: Embed decoder configuration in a standalone structure Srirangan Madhavan
2026-11-05 18:59 ` [PATCH v17 05/19] cxl: Introduce endpoint HDM decoder settings Srirangan Madhavan
2026-11-05 18:59 ` [PATCH v17 06/19] cxl: Move HDM decoder helpers to built-in code Srirangan Madhavan
2026-11-05 18:59 ` [PATCH v17 07/19] cxl: Share HDM decoder register unpacking Srirangan Madhavan
2026-11-12 21:35   ` Dave Jiang
2026-11-05 18:59 ` [PATCH v17 08/19] cxl: Move cxl_rwsem into built-in code Srirangan Madhavan
2026-11-12 23:49   ` Dave Jiang
2026-11-05 18:59 ` [PATCH v17 09/19] cxl: Refresh cached PCI HDM decoder settings Srirangan Madhavan
2026-11-13  0:38   ` Dave Jiang
2026-11-05 18:59 ` [PATCH v17 10/19] cxl: Build register block helpers into the kernel Srirangan Madhavan
2026-11-13 18:02   ` Dave Jiang
2026-11-05 18:59 ` [PATCH v17 11/19] PCI/CXL: Cache endpoint HDM state during PCI enumeration Srirangan Madhavan
2026-11-10 14:12   ` Li Ming
2026-11-11  3:54     ` Li Ming
2026-11-17 17:29   ` Alison Schofield
2026-11-20  8:17     ` Srirangan Madhavan
2026-11-05 18:59 ` [PATCH v17 12/19] PCI/CXL: Add CXL Device Reset sequencing Srirangan Madhavan
2026-11-10 14:29   ` Li Ming
2026-11-11 10:12   ` Lucero Palau, Alejandro
2026-11-20 23:14     ` Srirangan Madhavan [this message]
2026-11-05 18:59 ` [PATCH v17 13/19] cxl: Validate and synchronize HDM ranges around reset Srirangan Madhavan
2026-11-17 17:32   ` Alison Schofield
2026-11-20  8:20     ` Srirangan Madhavan
2026-11-05 18:59 ` [PATCH v17 14/19] PCI/CXL: Reject reset with unsafe function scope Srirangan Madhavan
2026-11-05 18:59 ` [PATCH v17 15/19] PCI/CXL: Restore CXL state after PCI reset Srirangan Madhavan
2026-11-05 18:59 ` [PATCH v17 16/19] PCI/CXL: Expose CXL Reset as a PCI reset method Srirangan Madhavan
2026-11-17 17:37   ` Alison Schofield
2026-11-20  8:21     ` Srirangan Madhavan
2026-11-05 18:59 ` [PATCH v17 17/19] PCI/CXL: Restore CXL state after CXL bus reset Srirangan Madhavan
2026-11-13 17:37   ` Dave Jiang
2026-11-05 18:59 ` [PATCH v17 18/19] PCI/CXL: Flush host and device caches for " Srirangan Madhavan
2026-11-05 18:59 ` [PATCH v17 19/19] cxl: Add KUnit checks for HDM decoder unpacking Srirangan Madhavan
2026-11-05 19:31 ` [PATCH v17 00/19] PCI/CXL: Add CXL reset support for Type 2 devices Alison Schofield
2026-11-16 19:11 ` Dave Jiang
2026-11-20  7:32   ` Srirangan Madhavan
2026-11-17 17:28 ` Alison Schofield
2026-11-20  8:16   ` Srirangan Madhavan

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=8a63033e-8f9a-4da0-95db-a3310cc0e585@nvidia.com \
    --to=smadhavan@nvidia.com \
    --cc=alejandro.lucero-palau@amd.com \
    --cc=alex.williamson@redhat.com \
    --cc=alison.schofield@intel.com \
    --cc=alwilliamson@nvidia.com \
    --cc=bhelgaas@google.com \
    --cc=dave.jiang@intel.com \
    --cc=dave@stgolabs.net \
    --cc=icheng@nvidia.com \
    --cc=iweiny@kernel.org \
    --cc=jan@nvidia.com \
    --cc=jic23@kernel.org \
    --cc=linux-cxl@vger.kernel.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=linux-pci@vger.kernel.org \
    --cc=linux-tegra@vger.kernel.org \
    --cc=machaujo@amazon.de \
    --cc=mhonap@nvidia.com \
    --cc=ming.li@zohomail.com \
    --cc=skancherla@nvidia.com \
    --cc=vaslot@nvidia.com \
    --cc=vishal.l.verma@intel.com \
    --cc=vsethi@nvidia.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox

all inboxes | Powered by JetHome®