From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-alma10-1.taild15c8.ts.net [100.103.45.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id EBE6C25B094; Fri, 25 Sep 2026 20:02:13 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=100.103.45.18 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1790366535; cv=none; b=VYueY9mF/P9ZdGRXpKkX3YlFOChoKL78QwYx1xemk9UW1vi4ce4OZu6b72HTkihwSi/pXgb06IpR0OAJ3kTu3vnt1DORFRpferdycPWKuNfZ7HTpKEnW44t8BPiCbxj6JHu6y853Eoqedm/wTkwFOiYrbW7UsmOKyFQyzjldpUM= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1790366535; c=relaxed/simple; bh=1bSPSvzh4LBB2E9HtT06HEe1/VLBUwD9ql+PHLICK9g=; h=Date:From:To:Cc:Subject:Message-ID:MIME-Version:Content-Type: Content-Disposition:In-Reply-To; b=iq2UCOnReCKOb6goM4wNOiuqYk2PqzVlk+6V+jAAlrDpp3kyiNi5CdBkRrnzT4ftuJiX0HemGkvv++cv6ROO2Unu+GElUUdfeXtTbsp2/3mG+Ax1g7LKFricJUdwjffn4tXVLk9GmZj4kNE1hdLQZAvF7nTmlqkIvlxazXIRRy4= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=on3Al0dK; arc=none smtp.client-ip=100.103.45.18 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="on3Al0dK" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 824AC1F000FF; Fri, 25 Sep 2026 20:02:13 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1790366533; bh=ZtcT91k1BC068tpcYnPNOSbfWg5Hqn83Pwqha6rsoNc=; h=Date:From:To:Cc:Subject:In-Reply-To; b=on3Al0dKls36XkavMiRKhbx1pcKKa2K91EoZiaTZ0prTK8rD+RHOaik47qljzr34R +GXaeDdTW2vGYZl6qFBRXOHgjGxRgAAgXuw3OUgfk33Nz2H86ebgFuZChQfCvTNX8p mBL7/HF+Y2UAHzVXXHxI+9ybi5qxb2JWKPiI8Pk/8qvo1amDCrGa3vVahXKYdX6Ml/ DvoQpVXmDdHo//gb5ORSFA5XyTVA9b8NqMUCFipE3oLVpmhtyXZ9WuEUtxbykuKryD v/oj8bTLN/eyJk6sAVRvEIeqTDZED8fpRR4VDfMtcBdDTVt3Rxm+sPL/Tqv7cdB8Kr kdjnm4M+NMU9w== Date: Fri, 25 Sep 2026 15:02:12 -0500 From: Bjorn Helgaas To: David Matlack Cc: kexec@lists.infradead.org, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-pci@vger.kernel.org, Adithya Jayachandran , Alexander Graf , Alex Williamson , Bjorn Helgaas , Chris Li , David Rientjes , Jacob Pan , Jason Gunthorpe , Jonathan Corbet , Josh Hilke , Leon Romanovsky , Lukas Wunner , Mike Rapoport , Parav Pandit , Pasha Tatashin , Pranjal Shrivastava , Pratyush Yadav , Randy Dunlap , Saeed Mahameed , Samiullah Khawaja , Shuah Khan , Vipin Sharma , William Tu , Yi Liu Subject: Re: [PATCH v9 05/13] PCI: liveupdate: Auto-preserve upstream bridges across Live Update Message-ID: <20260925200212.GA2082495@bhelgaas> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20260918200640.887030-6-dmatlack@google.com> On Fri, Sep 18, 2026 at 08:06:31PM +0000, David Matlack wrote: > When a PCI device is preserved across a Live Update, all of its upstream > bridges up to the root port must also be preserved. This enables the PCI > core and any drivers bound to the bridges to manage bridges correctly > across a Live Update. > > Notably, this will be used in subsequent commits to ensure that > preserved devices can continue performing memory transactions without a > disruption or change in routing. > > To preserve bridges, the PCI core tracks the number of downstream > devices preserved under each bridge using a reference count in struct > pci_dev_ser. This allows a bridge to remain preserved until all its > downstream preserved devices are unpreserved or finish their > participation in the Live Update. > > Reviewed-by: Pasha Tatashin > Reviewed-by: Pranjal Shrivastava > Signed-off-by: David Matlack Reviewed-by: Bjorn Helgaas > --- > drivers/pci/liveupdate.c | 125 +++++++++++++++++++++++++++--------- > include/linux/kho/abi/pci.h | 5 +- > include/linux/pci.h | 3 + > 3 files changed, 99 insertions(+), 34 deletions(-) > > diff --git a/drivers/pci/liveupdate.c b/drivers/pci/liveupdate.c > index ec8db86ed66d..825df024eec4 100644 > --- a/drivers/pci/liveupdate.c > +++ b/drivers/pci/liveupdate.c > @@ -122,7 +122,6 @@ > * preserved. These may be relaxed in the future: > * > * * The device cannot be a Virtual Function (VF). > - * * The device cannot be behind a PCI-to-PCI bridge. > * > * Driver Binding > * ============== > @@ -137,6 +136,18 @@ > * bound to the correct driver. The PCI core does not protect against a device > * getting preserved by driver A in the outgoing kernel and then getting bound > * to driver B in the incoming kernel. > + * > + * PCI-to-PCI Bridges > + * ================== > + * > + * Any PCI-to-PCI bridges upstream of a preserved device are automatically > + * preserved when the device is preserved. The PCI core keeps track of the > + * number of downstream devices that are preserved under a bridge so that the > + * bridge is only unpreserved once all downstream devices are unpreserved. > + * > + * This enables the PCI core and any drivers bound to the bridge to participate > + * in the Live Update so that preserved endpoints can continue issuing memory > + * transactions during the Live Update. > */ > > #define pr_fmt(fmt) "PCI: liveupdate: " fmt > @@ -407,55 +418,84 @@ static struct pci_dev_ser *pci_flb_alloc_dev_ser(struct pci_flb_outgoing *outgoi > return dev_ser; > } > > -static void pci_liveupdate_unpreserve_device(struct pci_flb_outgoing *outgoing, > - struct pci_dev *dev) > +static int pci_liveupdate_unpreserve_device(struct pci_flb_outgoing *outgoing, > + struct pci_dev *dev) > { > struct pci_dev_ser *dev_ser = dev->liveupdate.outgoing; > > if (!dev_ser) { > pci_warn(dev, "Cannot unpreserve device that is not preserved\n"); > - return; > + return -EINVAL; > } > > + if (!dev_ser->refcount) { > + pci_WARN(dev, 1, "Preserved device has a 0 refcount!\n"); > + return -EINVAL; > + } > + > + if (--dev_ser->refcount) > + return 0; > + > pci_info(dev, "Device will no longer be preserved across next Live Update\n"); > outgoing->ser->nr_devices--; > memset(dev_ser, 0, sizeof(*dev_ser)); > dev->liveupdate.outgoing = NULL; > + return 0; > +} > + > +static void pci_liveupdate_unpreserve_path(struct pci_flb_outgoing *outgoing, > + struct pci_dev *dev, > + struct pci_dev *end) > +{ > + for_each_pci_dev_in_path(dev) { > + if (dev == end) > + break; > + > + if (pci_liveupdate_unpreserve_device(outgoing, dev)) > + return; > + } > } > > static int pci_liveupdate_preserve_device(struct pci_flb_outgoing *outgoing, > struct pci_dev *dev) > { > - struct pci_dev_ser *dev_ser; > - > if (dev->is_virtfn) { > pci_warn(dev, "Cannot preserve Virtual Functions\n"); > return -EINVAL; > } > > - if (dev->liveupdate.outgoing) { > + /* > + * Endpoint devices should not be preserved more than once. > + * Bridges are preserved once for every downstream device that > + * is preserved. > + */ > + if (dev->liveupdate.outgoing && !dev->subordinate) { > pci_warn(dev, "Device is already preserved\n"); > return -EBUSY; > } > > - if (!pci_is_root_bus(dev->bus)) { > - pci_warn(dev, "Cannot preserve devices behind bridges\n"); > + if (dev->liveupdate.outgoing && !dev->liveupdate.outgoing->refcount) { > + pci_WARN(dev, 1, "Preserved device with 0 refcount!\n"); > return -EINVAL; > } > > - dev_ser = pci_flb_alloc_dev_ser(outgoing); > - if (IS_ERR(dev_ser)) > - return PTR_ERR(dev_ser); > + if (!dev->liveupdate.outgoing) { > + struct pci_dev_ser *dev_ser; > > - pci_info(dev, "Device will be preserved across next Live Update\n"); > - outgoing->ser->nr_devices++; > - outgoing->ser->devices = kho_block_set_head_pa(&outgoing->block_set); > + dev_ser = pci_flb_alloc_dev_ser(outgoing); > + if (IS_ERR(dev_ser)) > + return PTR_ERR(dev_ser); > > - dev_ser->domain = pci_domain_nr(dev->bus); > - dev_ser->bdf = pci_dev_id(dev); > - dev_ser->refcount++; > + pci_info(dev, "Device will be preserved across next Live Update\n"); > + outgoing->ser->nr_devices++; > + outgoing->ser->devices = kho_block_set_head_pa(&outgoing->block_set); > + > + dev_ser->domain = pci_domain_nr(dev->bus); > + dev_ser->bdf = pci_dev_id(dev); > + dev->liveupdate.outgoing = dev_ser; > + } > > - dev->liveupdate.outgoing = dev_ser; > + dev->liveupdate.outgoing->refcount++; Nice, thanks for this, I think it reads much better! > return 0; > } > > @@ -468,12 +508,16 @@ static int pci_liveupdate_preserve_device(struct pci_flb_outgoing *outgoing, > * pci_liveupdate_preserve() from their struct liveupdate_file_handler > * preserve() callback to ensure the outgoing struct pci_ser is already set up. > * > + * pci_liveupdate_preserve() automatically preserves all bridges upstream of > + * @dev. > + * > * Returns: 0 on success, <0 on failure. > */ > int pci_liveupdate_preserve(struct pci_dev *dev) > { > struct pci_flb_outgoing *outgoing = NULL; > - int ret; > + struct pci_dev *start = dev; > + int ret = -ENODEV; > > guard(rwsem_write)(&pci_liveupdate.rwsem); > > @@ -481,7 +525,13 @@ int pci_liveupdate_preserve(struct pci_dev *dev) > if (IS_ERR(outgoing)) > return PTR_ERR(outgoing); > > - ret = pci_liveupdate_preserve_device(outgoing, dev); > + for_each_pci_dev_in_path(dev) { > + ret = pci_liveupdate_preserve_device(outgoing, dev); > + if (ret) { > + pci_liveupdate_unpreserve_path(outgoing, start, dev); > + break; > + } > + } > > pci_liveupdate_flb_put_outgoing(); > return ret; > @@ -497,6 +547,9 @@ EXPORT_SYMBOL_GPL(pci_liveupdate_preserve); > * pci_liveupdate_unpreserve() from their struct liveupdate_file_handler > * unpreserve() callback to ensure the outgoing struct pci_ser is already set > * up. > + * > + * pci_liveupdate_unpreserve() automatically unpreserves all bridges upstream of > + * @dev. > */ > void pci_liveupdate_unpreserve(struct pci_dev *dev) > { > @@ -510,7 +563,7 @@ void pci_liveupdate_unpreserve(struct pci_dev *dev) > return; > } > > - pci_liveupdate_unpreserve_device(outgoing, dev); > + pci_liveupdate_unpreserve_path(outgoing, dev, /*end=*/NULL); > pci_liveupdate_flb_put_outgoing(); > } > EXPORT_SYMBOL_GPL(pci_liveupdate_unpreserve); > @@ -600,28 +653,30 @@ void pci_liveupdate_cleanup_device(struct pci_dev *dev) > pci_WARN(dev, 1, "Destroying incoming-preserved device!\n"); > } > > -static void pci_liveupdate_finish_device(struct pci_ser *ser, struct pci_dev *dev) > +static int pci_liveupdate_finish_device(struct pci_ser *ser, struct pci_dev *dev) > { > if (!dev->liveupdate.incoming) { > pci_warn(dev, "Cannot finish preserving an unpreserved device\n"); > - return; > + return -EINVAL; > } > > - if (dev->liveupdate.incoming->refcount != 1) { > - pci_WARN(dev, 1, "Preserved device has a corrupted refcount!\n"); > - return; > + if (!dev->liveupdate.incoming->refcount) { > + pci_WARN(dev, 1, "Preserved device has a 0 refcount!\n"); > + return -EINVAL; > } > > /* > - * Drop the refcount so this device does not get treated as an incoming > - * device again, e.g. in case pci_liveupdate_setup_device() gets called > - * again because the device is hot-plugged. > + * Decrement the refcount so this device does not get treated as an > + * incoming device again, e.g. in case pci_liveupdate_setup_device() > + * gets called again because the device is hot-plugged. > */ > - dev->liveupdate.incoming->refcount = 0; > + if (--dev->liveupdate.incoming->refcount) > + return 0; > > pci_info(dev, "Device is finished participating in Live Update\n"); > dev->liveupdate.incoming = NULL; > ser->nr_devices--; > + return 0; > } > > /** > @@ -633,6 +688,8 @@ static void pci_liveupdate_finish_device(struct pci_ser *ser, struct pci_dev *de > * Update. Drivers must call pci_liveupdate_finish() from their struct > * liveupdate_file_handler finish() callback to ensure the incoming struct > * pci_ser is allocated. > + * > + * pci_liveupdate_finish() automatically finishes all bridges upstream of @dev. > */ > void pci_liveupdate_finish(struct pci_dev *dev) > { > @@ -646,7 +703,11 @@ void pci_liveupdate_finish(struct pci_dev *dev) > return; > } > > - pci_liveupdate_finish_device(incoming->ser, dev); > + for_each_pci_dev_in_path(dev) { > + if (pci_liveupdate_finish_device(incoming->ser, dev)) > + break; > + } > + > pci_liveupdate_flb_put_incoming(); > } > EXPORT_SYMBOL_GPL(pci_liveupdate_finish); > diff --git a/include/linux/kho/abi/pci.h b/include/linux/kho/abi/pci.h > index 4096e3cd3324..9485ed73c351 100644 > --- a/include/linux/kho/abi/pci.h > +++ b/include/linux/kho/abi/pci.h > @@ -24,7 +24,7 @@ > */ > > #define PCI_LUO_FLB_COMPATIBLE "pci" > -#define PCI_LUO_FLB_VERSION 1 > +#define PCI_LUO_FLB_VERSION 2 > > /** > * struct pci_dev_ser - Serialized state about a single PCI device. > @@ -33,7 +33,8 @@ > * @bdf: The device's PCI bus, device, and function number. > * @refcount: Reference count used by the PCI core to keep track of whether it > * is done using a device's struct pci_dev_ser. The value of the > - * refcount is equal to 1 when the struct pci_dev_ser is in use, and > + * refcount is equal to the number of preserved devices at or below > + * it in the PCI hierarchy when the struct pci_dev_ser is in use, and > * 0 otherwise. > */ > struct pci_dev_ser { > diff --git a/include/linux/pci.h b/include/linux/pci.h > index 76abe884e3dc..b35ac263c451 100644 > --- a/include/linux/pci.h > +++ b/include/linux/pci.h > @@ -836,6 +836,9 @@ static inline struct pci_dev *pci_upstream_bridge(struct pci_dev *dev) > return dev->bus->self; > } > > +#define for_each_pci_dev_in_path(dev) \ > + for (; dev; dev = pci_upstream_bridge(dev)) > + > #ifdef CONFIG_PCI_MSI > static inline bool pci_dev_msi_enabled(struct pci_dev *pci_dev) > { > -- > 2.55.0.1082.g2b9226bbc0-goog >