From: Bjorn Helgaas <helgaas@kernel.org>
To: Richard Cheng <icheng@nvidia.com>
Cc: dave@stgolabs.net, jic23@kernel.org, dave.jiang@intel.com,
alison.schofield@intel.com, vishal.l.verma@intel.com,
helgaas@google.com, iweiny@kernel.org, ming.li@zohomail.com,
linux-cxl@vger.kernel.org, linux-kernel@vger.kernel.org,
linux-pci@vger.kernel.org, tglx@kernel.org,
wangruikang@iscas.ac.cn, alifm@linux.ibm.com,
shawn.lin@rock-chips.com, lihaoxiang@isrc.iscas.ac.cn,
18255117159@163.com, xiangzao@linux.alibaba.com,
vidyas@nvidia.com, kaihengf@nvidia.com, kobak@nvidia.com,
newtonl@nvidia.com, kristinc@nvidia.com,
Benjamin Cheatham <benjamin.cheatham@amd.com>,
Alex Williamson <alex@shazbot.org>
Subject: Re: [PATCH v3 1/2] PCI/MSI: Skip MSI/MSI-X restoration when config space is inaccessible
Date: Fri, 20 Nov 2026 10:39:09 -0600 [thread overview]
Message-ID: <20261120163909.GA878361@bhelgaas> (raw)
In-Reply-To: <av_8-NLyjXUQC0lK@MWDK4CY14F>
[+cc Ben, Alex]
On Fri, Nov 20, 2026 at 04:09:13PM +0800, Richard Cheng wrote:
> On Mon, Nov 16, 2026 at 01:09:39PM +0800, Bjorn Helgaas wrote:
> > On Mon, Nov 16, 2026 at 02:52:28PM +0800, Richard Cheng wrote:
> > > PCI state restoration runs even when a reset fails. If the device's
> > > config space remains inaccessible, MSI-X restoration still writes the
> > > MSI-X table. The table lives in device memory space, and these MMIO
> > > accesses can trigger additional errors, including DPC.
> > >
> > > MSI is programmed entirely through config space, so it doesn't issue
> > > such MMIO accesses. Restoring it is still pointless when config space is
> > > inaccessible: reads return all ones and the restore path writes back
> > > values derived from them.
> > >
> > > Check whether config space of pci device is accessible or not at the
> > > start of both MSI and MSI-X path, after msi_enabled/msix_enabled test
> > > and before any config access, and skip the restoration if the device is
> > > inaccessible.
> > > Make the helper non-static and declare it in pci.h so the MSI code can
> > > use it, too.
> > >
> > > Signed-off-by: Richard Cheng <icheng@nvidia.com>
> > > ---
> > > Changelog:
> > >
> > > v2 -> v3:
> > > - Cover MSI restoration path as well (Farhan)
> > > - Use pci_dev_config_accessible() instead of open-coding the check, and
> > > make it non-static (Farhan)
> > > - Move the check before any config space access in the restore
> > > functions (Farhan)
> > > - The return value of the PCI Command read is no longer checked, since
> > > the helper only tests for an all-ones value.
> > > - Update the subject and commit message accordingly
> > > ---
> > > drivers/pci/msi/msi.c | 6 ++++++
> > > drivers/pci/pci.c | 2 +-
> > > drivers/pci/pci.h | 1 +
> > > 3 files changed, 8 insertions(+), 1 deletion(-)
> > >
> > > diff --git a/drivers/pci/msi/msi.c b/drivers/pci/msi/msi.c
> > > index 8a0133a6a286..2865f73f5ead 100644
> > > --- a/drivers/pci/msi/msi.c
> > > +++ b/drivers/pci/msi/msi.c
> > > @@ -524,6 +524,9 @@ void __pci_restore_msi_state(struct pci_dev *dev)
> > > if (!dev->msi_enabled)
> > > return;
> > >
> > > + if (!pci_dev_config_accessible(dev, "restore MSI state"))
> > > + return;
> >
> > Sigh. I hate the fact that pci_dev_config_accessible() is
> > proliferating even though it's obviously racy. Maybe it's the best we
> > can do, but I just feel like we're digging this hole deeper.
> >
> Hmm I agree this is racy even though we're holding the device lock
> throughout the reset/restore process.
>
> I assume you mean something like a separate sources DPC happened and
> make the device inaccessible , some scenario like that ?
Yes. A PCI device can stop responding at any time, e.g., it got
physically removed from a slot, a cable was removed, its internal
firmware crashed, power failed, etc.
> > Also, this is called in the pci_restore_state() path, which is full of
> > other things that also write to config space and will also fail in
> > this situation.
>
> Actually I was trying to cover the whole pci_restore_state() in v1 [1],
> and Benjamin raise the concern about the drivers may depen on PCI state
> having been restored by the time reset_done() runs.
I think you're referring specifically to Ben's comment [2] that driver
.reset_done() handlers expect that pci_restore_state() has already
been completed, and your "[PATCH 3/4] PCI: Skip state restoration when
configuration is inaccessible" patch calls .reset_done() without any
restoration if the device was inaccessible:
static void pci_dev_restore(struct pci_dev *dev)
{
const struct pci_error_handlers *err_handler =
dev->driver ? dev->driver->err_handler : NULL;
- pci_restore_state(dev);
+ /* MSI-X restoration may access device MMIO. */
+ if (pci_dev_config_accessible(dev, "restore state"))
+ pci_restore_state(dev);
/*
* dev->driver->err_handler->reset_done() is protected against
* races with ->remove() by the device lock, which must be held by
* the caller.
*/
if (err_handler && err_handler->reset_done)
err_handler->reset_done(dev);
I think Ben is absolutely right that .reset_done() handlers expect the
device to be accessible and config state to have been restored, and I
don't think it's reasonable to expect every driver's .reset_done() to
handle that generic problem.
I think pci_restore_state() needs to return failure and callers need
to figure out how to deal with that. There are a lot of callers, so
it's messy, but we should recognize that it can fail.
> What my bug actually hit was MSI-X table write triggered DPC, I can't
> justify for all kinds of driver, so I limited the scope.
>
> If you think a broader check in pci_dev_restore() is a better place to
> handle this, I'm happy to give a deeper research and revisit it.
>
> Best regards,
> Richard Cheng.
>
> [1]: https://lore.kernel.org/all/20260929020405.9724-1-icheng@nvidia.com/
[2] https://lore.kernel.org/all/8a19d63c-db9a-40ab-a811-5d761252638f@amd.com/
> > > entry = irq_get_msi_desc(dev->irq);
> > >
> > > pci_intx_for_msi(dev, 0);
> > > @@ -877,6 +880,9 @@ void __pci_restore_msix_state(struct pci_dev *dev)
> > > if (!dev->msix_enabled)
> > > return;
> > >
> > > + if (!pci_dev_config_accessible(dev, "restore MSI-X state"))
> > > + return;
> > > +
> > > /* route the table */
> > > pci_intx_for_msi(dev, 0);
> > > pci_msix_clear_and_set_ctrl(dev, 0,
> > > diff --git a/drivers/pci/pci.c b/drivers/pci/pci.c
> > > index 7c6f2aba8144..671ffe4da1c8 100644
> > > --- a/drivers/pci/pci.c
> > > +++ b/drivers/pci/pci.c
> > > @@ -714,7 +714,7 @@ u16 pci_find_dvsec_capability(struct pci_dev *dev, u16 vendor, u16 dvsec)
> > > }
> > > EXPORT_SYMBOL_GPL(pci_find_dvsec_capability);
> > >
> > > -static bool pci_dev_config_accessible(struct pci_dev *dev, char *msg)
> > > +bool pci_dev_config_accessible(struct pci_dev *dev, char *msg)
> > > {
> > > u32 val;
> > >
> > > diff --git a/drivers/pci/pci.h b/drivers/pci/pci.h
> > > index 77efa4c5c960..98dd3dbd5edb 100644
> > > --- a/drivers/pci/pci.h
> > > +++ b/drivers/pci/pci.h
> > > @@ -240,6 +240,7 @@ enum pci_mmap_api {
> > > int pci_mmap_fits(struct pci_dev *pdev, int resno, struct vm_area_struct *vmai,
> > > enum pci_mmap_api mmap_api);
> > >
> > > +bool pci_dev_config_accessible(struct pci_dev *dev, char *msg);
> > > bool pci_reset_supported(struct pci_dev *dev);
> > > void pci_init_reset_methods(struct pci_dev *dev);
> > > int pci_bridge_secondary_bus_reset(struct pci_dev *dev);
> > > --
> > > 2.43.0
> > >
> >
next prev parent reply other threads:[~2026-11-20 16:39 UTC|newest]
Thread overview: 6+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-11-16 6:52 [PATCH v3 0/2] PCI/CXL: Guard device accesses during reset recovery Richard Cheng
2026-11-16 6:52 ` [PATCH v3 1/2] PCI/MSI: Skip MSI/MSI-X restoration when config space is inaccessible Richard Cheng
2026-11-16 19:09 ` Bjorn Helgaas
2026-11-20 8:09 ` Richard Cheng
2026-11-20 16:39 ` Bjorn Helgaas [this message]
2026-11-16 6:52 ` [PATCH v3 2/2] cxl/pci: Guard decoder accesses after reset Richard Cheng
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20261120163909.GA878361@bhelgaas \
--to=helgaas@kernel.org \
--cc=18255117159@163.com \
--cc=alex@shazbot.org \
--cc=alifm@linux.ibm.com \
--cc=alison.schofield@intel.com \
--cc=benjamin.cheatham@amd.com \
--cc=dave.jiang@intel.com \
--cc=dave@stgolabs.net \
--cc=helgaas@google.com \
--cc=icheng@nvidia.com \
--cc=iweiny@kernel.org \
--cc=jic23@kernel.org \
--cc=kaihengf@nvidia.com \
--cc=kobak@nvidia.com \
--cc=kristinc@nvidia.com \
--cc=lihaoxiang@isrc.iscas.ac.cn \
--cc=linux-cxl@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-pci@vger.kernel.org \
--cc=ming.li@zohomail.com \
--cc=newtonl@nvidia.com \
--cc=shawn.lin@rock-chips.com \
--cc=tglx@kernel.org \
--cc=vidyas@nvidia.com \
--cc=vishal.l.verma@intel.com \
--cc=wangruikang@iscas.ac.cn \
--cc=xiangzao@linux.alibaba.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®