From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-dl1-f53.google.com (mail-dl1-f53.google.com [74.125.82.53]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1211221D596 for ; Tue, 16 Jun 2026 22:21:01 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=pass smtp.client-ip=74.125.82.53 ARC-Seal:i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1781648463; cv=pass; b=by85g7TU3pRmEq704H3Osu/rJk92e5FYFkZpQqLrYp/zgvcgg0J2mFJC8qFc+pJcI2f9tTxtchBdI+25JZ4gmqIEvWWUYDoFlkVhNcS3VnLdG0vAyDLsFS6YgVNQ8AADJ6uWutkELiOuxphQ9Ma49xfv68ZOkK88TW93Pe9SEQM= ARC-Message-Signature:i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1781648463; c=relaxed/simple; bh=qme9hMeTybbvHWlc65qwT/wsA705F7QW3wsJxaveMyU=; h=MIME-Version:References:In-Reply-To:From:Date:Message-ID:Subject: To:Cc:Content-Type; b=NhsUHCZtP+EnBbh0by+1YgIivb851UTeg1hvKpJCpvODNxOVeeFAb5LZVQJ+OtXalhz4OXo7jxy6XBtrWjcJJhyFSF8+55Px/K4xUh1HBIrvTc6PjycWQsifOpXa2eVGPS9gfJZuNWvqo2nACYNIESR9HHOb+uqhBgXRYVZ44zo= ARC-Authentication-Results:i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=WaSnrYEs; arc=pass smtp.client-ip=74.125.82.53 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="WaSnrYEs" Received: by mail-dl1-f53.google.com with SMTP id a92af1059eb24-1384ebe7a10so207573c88.1 for ; Tue, 16 Jun 2026 15:21:01 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1781648461; cv=none; d=google.com; s=arc-20240605; b=J65k1pywun60h4YCD5iACccoiZ9t0O6zdy8skm15EH9aQzbOUAouwHVUGO6wGzXl00 QANAd5CSCP9w83VZ2wL2mpacxMgYJwi9nYENJtL23Pai6QhwIf0WYUWWPm3nNXmgSCH8 i63xFyI1+2tlpzIcOP3jnNVA+ufA9OGSeGfnAwfNwEdoTmMDZiG4Cacbb2kAwFOoDJVE wxDiTvxYbW56zaPp3RuSW/NN0hfUjonTtss0UTeClHLiwNMye4gkyGWoPrnrDfapbYKG xE/quV7Mv8EJkgHreCdqfhi28xltv70slwjyPPqluxa/NvrrU1IdDHbX7iXTYojUS0YQ YKog== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20240605; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:dkim-signature; bh=5hm1HVaCy9nZT8d+x5IZzYgyaULXvQNG2Dw2s8INhX4=; fh=vCJtN7qEKgdHqJA4i0OrPByE2zvRY+bfwEP0acMOLiE=; b=LsjssMeuJXaTJkzCzH9tI+PJaoj40ukm9RLCbPRzviG/wuIIAiaMZkY4j7q1JvQ29Q b8tjTsHU2JUYDuamt42Bep5utLRs8xtvXAlgd0qMAoP+Z6/qgfdgYkoIUpFknAHbxLJs h8IYeIUuwxvJQRkUNX0ZdTRJvCBygXFs2Y+Pc/1fNfGEW5Dgm2yj4ZEgndaWskr0IZ73 IlfHP5im2jEqd6gTAqeWe3E7qk/6Hxjflf/aFltu3eOIaqc12FlZ/+hFenVukeshuBoV GBqzECqByytTt1LROX/rI9eXBgz1nZtgusK6n7czdomQs0Qsz4fpTYaOtMjbbTFWWMes pc6g==; darn=vger.kernel.org ARC-Authentication-Results: i=1; mx.google.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1781648461; x=1782253261; darn=vger.kernel.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=5hm1HVaCy9nZT8d+x5IZzYgyaULXvQNG2Dw2s8INhX4=; b=WaSnrYEslwMKTZ1E4L/bXRbsZUJBd3u1ey5KCjQyJcXHmlFiKCXDWkQSdY7WuihHc/ PPql6fvqNUNyZ6m/xVA4RG3pRHqmCL99OmscMJkledKET3Q6WA5hq8Q94y6K/FJDucbF x2AKkA2y0oYjnsGSbCKwNe6s1QoGRPJ2IjWfU80HLl1rosF5wlWMWSgz3uCfvS5fbLB7 o6YEpilC2wGv7uqWKV0WQqSWSEaAsXrKXkN1HSyIOfextoTptgjxbetLhC0pDXfPDUGx 3YbAd7r+MdVQLSyJTXp1ksxDcvsTFMBoeeUEG7JwTgxa8RNbqtPOYP3HrSxAcY8NTC9Z AFjA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1781648461; x=1782253261; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=5hm1HVaCy9nZT8d+x5IZzYgyaULXvQNG2Dw2s8INhX4=; b=gxJk51HtH0Wmv10+nZbr96sitaDw8Cmddg5EXHqXCkU0T30tdt9WN39w2VwAiYBCyn NitMdxudFXUX682JLgI3F/lsuHW/8Hwdnm/FvyKRqNNL1KvOVVwtdDSOC30tYY9bj/Gb HIDNLg25H6NzsR5KSOou5DhtKpCVYxxwWRaiGTP9XsL9SW6fxCpvM/ro18G9MEwB3Np+ ng5TW11Smjs5QGNxuTI3VAKIHwilMrrjNCoI1A0th4Y4owYfZ8kmOud+uzVTRXhvDT35 muSiVmX6d77Iu9TSl1N6lOB7dMPbFOdN8YefBVOSY4QxexFIPHa8aBEq0DJRBAxwQ2Tf jtXQ== X-Forwarded-Encrypted: i=1; AFNElJ8NHwvDl/r+2zXaA7RgKABQ4lHMYwrYmPtrVui4zDWcQZy/HsZTsaZhw8nuqyaO9qoM8ieoakOxquxK9jE=@vger.kernel.org X-Gm-Message-State: AOJu0YwSAFMgnUxPp2c+wfOy1ol6U3aOXGhrHKWgKZxZXb8JiNFvRjsM 5O2bJAnpSWtrC8E0EebitC+xvKbjXYnyu5m8nAEoj9hjBUF+opyRS/iP5wTf1jVKz9ZNPwnukVX z3Uz/2AauwMUW9YCNEneJktakPgZrzTrCjsBg/EE7 X-Gm-Gg: Acq92OEiId1LBwH+afHy4tEkFEuNTkK5lzMFQ6dB25CW2cRTjAPQzh45e9aFDqTrdpp jWEvR1QVWoU8A0zTSniP9nWtluRwVlXBFUYdeXzbFYfpr3Ks3OW3hPN84hs/5lMQqHnbKQ/hzL8 THzrHj9TFIcOc2rF8fyOwbKLkl0PHTITxfmvNPqJD7e1LMLcZ3loKSKBdTU22dJn+0+/JPN1hIs oYXEoOKpTN3jlEK8VdjPdsokpKb6edto3wpp+bhLnVdgePKQwFaGOUa6TlyJsNqWFHKE6YlvXmD VdMgAXU= X-Received: by 2002:a05:7022:6ba8:b0:136:ac69:b0f7 with SMTP id a92af1059eb24-1398f680985mr453889c88.16.1781648460211; Tue, 16 Jun 2026 15:21:00 -0700 (PDT) Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 References: <20260522202410.3104264-1-dmatlack@google.com> <20260522202410.3104264-4-dmatlack@google.com> In-Reply-To: From: David Matlack Date: Tue, 16 Jun 2026 15:20:33 -0700 X-Gm-Features: AVVi8CfxK4mMKiv0sputj5dz4geA42BjbfG1FN5Obs96hOQRjIUk8-dblh_dgDE Message-ID: Subject: Re: [PATCH v6 03/12] PCI: liveupdate: Track incoming preserved PCI devices To: Samiullah Khawaja Cc: kexec@lists.infradead.org, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-pci@vger.kernel.org, Adithya Jayachandran , Alexander Graf , Alex Williamson , Bjorn Helgaas , Chris Li , David Rientjes , Jacob Pan , Jason Gunthorpe , Jonathan Corbet , Josh Hilke , Leon Romanovsky , Lukas Wunner , Mike Rapoport , Parav Pandit , Pasha Tatashin , Pranjal Shrivastava , Pratyush Yadav , Saeed Mahameed , Shuah Khan , Vipin Sharma , William Tu , Yi Liu Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Tue, Jun 16, 2026 at 1:09=E2=80=AFPM Samiullah Khawaja wrote: > > On Fri, May 22, 2026 at 08:24:01PM +0000, David Matlack wrote: > >During PCI enumeration, the previous kernel might have passed state abou= t > >devices that were preserved across kexec. The PCI core needs to fetch > >this state to identify which devices are "incoming" and require special > >handling. > > > >Add pci_liveupdate_setup_device() which is called during device setup > >to fetch the serialized state (struct pci_ser) from the Live Update > >Orchestrator. The first time this happens, pci_flb_retrieve() will run > >and convert the array of pci_dev_ser structs into an xarray so that it > >can be looked up efficiently. > > > >If a device is found in the xarray, the PCI core stores a pointer to its > >state in dev->liveupdate_incoming and holds a reference to the incoming > >FLB until pci_liveupdate_finish() is called by the driver. > > > >This ensures proper lifecycle management for incoming preserved devices > >and allows the PCI core and drivers to apply specific Live Update > >logic to them in subsequent commits. > > > >Drivers can check if a device is an incoming preserved device (e.g. > >during probe) by calling pci_liveupdate_is_incoming(). > > > >CONFIG_64BIT is now required to enable CONFIG_PCI_LIVEUPDATE so that the > >domain and bdf can be guaranteed to fit in an unsigned long and be used > >as the xarray key. > > > >Signed-off-by: David Matlack > >--- > > MAINTAINERS | 1 + > > drivers/pci/Kconfig | 2 +- > > drivers/pci/liveupdate.c | 230 ++++++++++++++++++++++++++++++++- > > drivers/pci/liveupdate.h | 5 + > > drivers/pci/probe.c | 3 + > > include/linux/pci_liveupdate.h | 13 ++ > > 6 files changed, 251 insertions(+), 3 deletions(-) > > > > [snip] > > > > static int pci_flb_retrieve(struct liveupdate_flb_op_args *args) > > { > >- args->obj =3D phys_to_virt(args->data); > >+ struct pci_ser *ser =3D phys_to_virt(args->data); > >+ struct pci_flb_incoming *incoming; > >+ int ret =3D -ENOMEM; > >+ u32 i; > >+ > >+ incoming =3D kmalloc_obj(*incoming); > >+ if (!incoming) > >+ goto err_restore_free; > >+ > >+ incoming->ser =3D ser; > >+ xa_init(&incoming->xa); > >+ > >+ for (i =3D 0; i < incoming->ser->max_nr_devices; i++) { > >+ struct pci_dev_ser *dev_ser =3D &incoming->ser->devices[i= ]; > >+ unsigned long key; > >+ > >+ if (!dev_ser->refcount) > >+ continue; > >+ > >+ key =3D pci_ser_xa_key(dev_ser->domain, dev_ser->bdf); > >+ ret =3D xa_insert(&incoming->xa, key, dev_ser, GFP_KERNEL= ); > >+ if (ret) > >+ goto err_xa_destroy; > >+ } > >+ > >+ args->obj =3D incoming; > > return 0; > >+ > >+err_xa_destroy: > >+ xa_destroy(&incoming->xa); > >+ kfree(incoming); > >+err_restore_free: > >+ kho_restore_free(ser); > >+ return ret; > > Hmm.. This is interesting, so the KHO state is freed and it cannot be > reused. I see you already pointed out that we are putting an LUO policy > to say that the retry is not allowed. > > But what should be the behaviour of liveupdate in this regard? Let the > system boot in a normal way? This might break other subsystems as they > might depend on PCIe restoring state properly. Also I think some of the > PCIe state, like device-id, BAR addresses, ACLs etc, might be used as > source of truth by other components. > > For example, lets say FLB retrieve() of PCIe fails, but succeeds for > VFIO/IOMMU, now VFIO/IOMMU are restoring state of a device that is not > restored/preserved? > > Should this be considered fatal? If PCI FLB retrieve fails then there are certain things that cannot be guaranteed, such as BDF (B specifically) remaining constant. This could lead to memory corruption as the IOMMU may have live translations in place for those specific RequesterIDs. And, in the future, preserved devices may be doing P2P which depends on BARs not moving. If the PCI core cannot retrieve the FLB saved by the previous kernel, it cannot make these guarantees. So yeah I think you're right that PCI core should treat FLB retrieve as fatal and just panic. > > } > > > > static void pci_flb_finish(struct liveupdate_flb_op_args *args) > > { > >- kho_restore_free(args->obj); > >+ struct pci_flb_incoming *incoming =3D args->obj; > >+ > >+ xa_destroy(&incoming->xa); > >+ kho_restore_free(incoming->ser); > >+ kfree(incoming); > > } > > > > static struct liveupdate_flb_ops pci_liveupdate_flb_ops =3D { > >@@ -270,6 +335,91 @@ void pci_liveupdate_unpreserve(struct pci_dev *dev) > > } > > EXPORT_SYMBOL_GPL(pci_liveupdate_unpreserve); > > > >+static struct pci_flb_incoming *pci_liveupdate_flb_get_incoming(void) > >+{ > >+ struct pci_flb_incoming *incoming =3D NULL; > >+ int ret; > >+ > >+ ret =3D liveupdate_flb_get_incoming(&pci_liveupdate_flb, (void **= )&incoming); > >+ > >+ /* Live Update is not enabled. */ > >+ if (ret =3D=3D -EOPNOTSUPP) > >+ return NULL; > >+ > >+ /* Live Update is enabled, but there is no incoming FLB data. */ > >+ if (ret =3D=3D -ENODATA) > >+ return NULL; > >+ > >+ /* > >+ * Live Update is enabled and there is incoming FLB data, but non= e of it > >+ * matches pci_liveupdate_flb.compatible. > >+ * > >+ * This could mean that no PCI FLB data was passed by the previou= s > >+ * kernel, but it could also mean the previous kernel used a diff= erent > >+ * compatibility string (i.e. a different ABI). > >+ */ > >+ if (ret =3D=3D -ENOENT) { > >+ pr_info_once("No incoming FLB matched %s\n", pci_liveupda= te_flb.compatible); > >+ return NULL; > >+ } > >+ > >+ /* > >+ * There is incoming FLB data that matches pci_liveupdate_flb.com= patible > >+ * but it cannot be retrieved. > >+ */ > >+ if (ret) { > >+ WARN_ONCE(ret, "Failed to retrieve incoming FLB data\n"); > > I think this should probably be considered fatal as mentioned above or > the caller of this function should get an error so it can fail. I think > retrievel of preserved state should generally not fail unless there is > memory corruption or ABI is incompatible. Yeah. I think I will just call panic() here to cover all cases. > >+ return NULL; > >+ } > >+ > >+ return incoming; > >+} > >+ > > [snip] > >+ > >+static inline bool pci_liveupdate_is_incoming(struct pci_dev *dev) > >+{ > >+ return false; > >+} > > #endif > > > > #endif /* LINUX_PCI_LIVEUPDATE_H */ > >-- > >2.54.0.746.g67dd491aae-goog > > > > Sami