From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-dl1-f47.google.com (mail-dl1-f47.google.com [74.125.82.47]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5CD7E33120C for ; Tue, 30 Jun 2026 22:35:24 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=pass smtp.client-ip=74.125.82.47 ARC-Seal:i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1782858927; cv=pass; b=hL1tSZ28u4XyLouQbxRyvH4tLKgj87z688m0VhTpu8v8vrP2jdYutO3SeEmNeG100i4OMQFY71HOONnc/TEcT3Iv9ywu7sFVSj0Q/QSo/s/lbCgkU0CS+Acec1Cauh9bQHwNTFVx1MXTCJ4z8DZOZmzvvBQTkcL5y2q14q4AU6o= ARC-Message-Signature:i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1782858927; c=relaxed/simple; bh=IIH6gb9Aspc978S77GNhCcY8LVqXRyGmxXPBoO9kTVQ=; h=MIME-Version:References:In-Reply-To:From:Date:Message-ID:Subject: To:Cc:Content-Type; b=WxthELNlmbkDLM9IuLj/D4vIJJ9mbCjVm9S2ysKNvCIlQlsLhmoM7yFpQQcTY3saswpx1mmUQDJqe93Micbx3aGJnqM+eWyk42bQSmAZXfxdep9n8upUr78d4fBcMV3Da7XXK1vYAtZnynW686bckGO7rDPeU9dKyhBWhs1HMqQ= ARC-Authentication-Results:i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=tIU74OsF; arc=pass smtp.client-ip=74.125.82.47 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="tIU74OsF" Received: by mail-dl1-f47.google.com with SMTP id a92af1059eb24-137335bc3caso10776495c88.0 for ; Tue, 30 Jun 2026 15:35:24 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1782858923; cv=none; d=google.com; s=arc-20260327; b=adJkqEMP3UWly/zJfWy4iwhCVFK6buDp59PDeAu9+JdgZ1jbORxqYhMZlF9NY1vfE4 bbM0Y4URpelri9MeK8Gt1w5cZZz1OK8wA+8xrNw7v+t4B4kXLrrVqtT4eMtGwfmnqqRX YKA7ucZoZ+ZAqtWpdhlWBGBHRTmSPIfKfWqk+Na64DaX96Gz/6eoHdX8ZpsIAWNY2CBB QO2sP0WHjeWObwe5235zwZ/h6s/Tl78tFhEOSdnYTeOOSEfadEc6mmm0GFFjKSYM0uGO 07yY9h7JF3IYIsU5w1pgiF7rYnov/9IXPHB1SXkWH7YyBGTvpE3yi0Vbdjp8F25v4IeR YWcg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20260327; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:dkim-signature; bh=OLXeb8c7iyYBBm1u3dFIHz9SDW0xbbNuKTG6qGPD7go=; fh=h5eUGsdYvf8FfBNEBIS5SlHNb8DR7vevt4MSAJx/pDg=; b=Skr4OBUpcTUp2WjBsf5zOdr0eFBLU0A0zGdXY4THjx79kJk8vtB6X2tydsLtoQJc93 UsdN3zryC6ZHK91lMxUUtUDrVpoG1r9oL+r3g7CpualgXuA+WaWh4mUuMkmny6Afx7EV kQGkftYEz4HUSgs/7wqRi9ifOGGuOZRn7siMgxC5raTi8uDLzLG7abBvrIqVVmtt5O8g alFWjBIdrRI3a6oNVjh/oeV5fdnrz5ZDJ6CsnYnSchJYhnQ/JpAL3xJ/M5pKnaodWKDS EmcmCAy0IYMvsD/VAqnYxWFPqgeUMxjvHtiDATXYygZf0sMz4yzuV3rsnLm/1RNaHmJa DslQ==; darn=vger.kernel.org ARC-Authentication-Results: i=1; mx.google.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1782858923; x=1783463723; darn=vger.kernel.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=OLXeb8c7iyYBBm1u3dFIHz9SDW0xbbNuKTG6qGPD7go=; b=tIU74OsFUlV3w0RTbXhocTIMXQyGIfdxV4w2f0/ZyNa30O5QCKoAji2YPY6tUioxzw buympSlkGYL6Bpn4SaMaEvcx4JTonwICxKPb1ktbWYiyCbn9r8dd7bGW/jX4QJdqsLLe gqRKdDNMnEQHh5gJ6pV/TDeTpHyjMYiMdEXAQWByHY1FKAR0p74n3fJ1iv6etIpHbDLc HsPkgXZDMaVb9vXoBwpdsXd5hV8ojGNHFpXzQEz2Q4sP/8xrEY4SxKtyEBwxWvDp05+T 0yMU3daHBKYyhevq2Hut6rfWQvHVZiKaB5i47V++4rRCr/c7hTUxE1iOUPqiAWx4OY2g 08Fw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1782858923; x=1783463723; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=OLXeb8c7iyYBBm1u3dFIHz9SDW0xbbNuKTG6qGPD7go=; b=SSbyJXnRis3DSZb++LXthx5zJSgamNYRqESK2ElkAZZt/HTAhpnF9QlRRDkPkrXzdL FaVz90doBRX/TvPgCIFHIpQcYzADD7llktboqF+hvNYyh3zyDtcYkV+adzuGJX0c88GF wHWNRhmBUjO0YlpV4VkMDyNsc/qOdRasvlBtQeiUHD+ytuRX+Y11SXg63FQc0QJSbaqV Ctuurfl4415rCehbpwUuufbuzx2GMIIHJ3jRQRf+uClHXd23lTQJBXuWfM6t/8I+bhUw 14iR/5t+qIv0rjGmyCJRlc3tY/q5Nk56ZNcxrwYruQ5Xo2SuDQqHWhqSyhSKTxxMfOID we9A== X-Forwarded-Encrypted: i=1; AFNElJ+y52ZcfApMZYbjHZ8FjpdQm/gs/vmQ+795/PgbI6bIPKn+Sqj1pO1Aa8F0MptDB6bPTWBBuE/ATAWQ8To=@vger.kernel.org X-Gm-Message-State: AOJu0YxbH1Z7uPIx0hMEJ5V5JFjGvw/UxkuTy+Xm7HLadNY9QvWKiVHm gbmyk2hQGqHLI1WUGeZOpLHxXOkx39LTvx6X++plg3p+Uo6/OPOlWB2iUZFSvL1KMldRlf2pR31 qqv5vCdRmJKK8bCQYY7qd/5hatARw13c4dXS06ipU X-Gm-Gg: AfdE7cnooeJwRdbjQ374bho4eh5zP76yb8o6SunLHfqkiiCA6BZR9swfn1ZxR6tA83N A39eFFMwTPjz+LKmF5WQGs0dUCuNwpl3Jl1ZdDy4PSNfCWRFBNZaQAgBQgOG23kbM0vdAGUKAYJ rdMutj35lP+X1detWEpFd/JgR835UHE2uoNffr8phU26XaRA935C/JBOxPUNehSxqQ+ywNf0P9j 95MsEcFV+zXQnKr7vzp7kG7F/Yn1qhbuk35x2oqnRXuDbqeAW8zPo/5JzqlC29HlzScWZVY X-Received: by 2002:a05:7022:f690:b0:139:ed59:4b8e with SMTP id a92af1059eb24-13b2a1c7f1dmr3156303c88.50.1782858922721; Tue, 30 Jun 2026 15:35:22 -0700 (PDT) Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 References: <20260522202410.3104264-1-dmatlack@google.com> <20260522202410.3104264-10-dmatlack@google.com> <20260608181906.GP1962447@nvidia.com> In-Reply-To: <20260608181906.GP1962447@nvidia.com> From: David Matlack Date: Tue, 30 Jun 2026 15:34:56 -0700 X-Gm-Features: AVVi8Cd-vdSCkwSqwAz2qbX9b8clCFCtpt-XfTl_1BZFKIa6zUmlOyUobbTWOp8 Message-ID: Subject: Re: [PATCH v6 09/12] PCI: liveupdate: Inherit ARI Forwarding Enable on preserved bridges To: Jason Gunthorpe Cc: Pranjal Shrivastava , kexec@lists.infradead.org, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-pci@vger.kernel.org, Adithya Jayachandran , Alexander Graf , Alex Williamson , Bjorn Helgaas , Chris Li , David Rientjes , Jacob Pan , Jonathan Corbet , Josh Hilke , Leon Romanovsky , Lukas Wunner , Mike Rapoport , Parav Pandit , Pasha Tatashin , Pratyush Yadav , Saeed Mahameed , Samiullah Khawaja , Shuah Khan , Vipin Sharma , William Tu , Yi Liu Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Mon, Jun 8, 2026 at 11:19=E2=80=AFAM Jason Gunthorpe wr= ote: > > On Mon, Jun 08, 2026 at 11:33:07AM +0000, Pranjal Shrivastava wrote: > > On Fri, May 22, 2026 at 08:24:07PM +0000, David Matlack wrote: > > > Inherit the ARI Forwarding Enable on preserved bridges and update > > > pci_dev->ari_enabled accordingly during a Live Update. This ensures t= hat > > > the preserved devices on the bridge's secondary bus can be identified > > > with the same expanded 8-bit function number after a Live Update. > > > > > > Signed-off-by: David Matlack > > > --- > > > drivers/pci/liveupdate.c | 18 ++++++++++++++++++ > > > drivers/pci/liveupdate.h | 6 ++++++ > > > drivers/pci/pci.c | 8 +++++++- > > > 3 files changed, 31 insertions(+), 1 deletion(-) > > > > > > diff --git a/drivers/pci/liveupdate.c b/drivers/pci/liveupdate.c > > > index a93b7ef065f2..701276ef6cfb 100644 > > > --- a/drivers/pci/liveupdate.c > > > +++ b/drivers/pci/liveupdate.c > > > @@ -128,6 +128,10 @@ > > > * way after Live Update and ensures that IOMMU groups do not cha= nge. Note > > > * that a device will use its inherited ACS flags for the lifetim= e of its > > > * struct pci_dev (i.e. even after pci_liveupdate_finish()). > > > + * > > > + * * The PCI core inherits ARI Forwarding Enable on all bridges wit= h downstream > > > + * preserved devices to ensure that all preserved devices on the = bridge's > > > + * secondary bus are addressable after the Live Update. > > > */ > > > > > > #define pr_fmt(fmt) "PCI: liveupdate: " fmt > > > @@ -756,6 +760,20 @@ int pci_liveupdate_enable_acs(struct pci_dev *de= v) > > > return 0; > > > } > > > > > > +int pci_liveupdate_configure_ari(struct pci_dev *dev) > > > +{ > > > + u16 val; > > > + > > > + guard(rwsem_read)(&pci_liveupdate.rwsem); > > > + > > > + if (!dev->liveupdate.incoming) > > > + return -EINVAL; > > > + > > > + pcie_capability_read_word(dev, PCI_EXP_DEVCTL2, &val); > > > > Again, I might be thinking out loud here, but since these are > > hot-pluggable devices, with some FW / SW running on them, I'm a little > > worried while assuming the HW registers can be trusted across a kexec. This is, in-part [*], why I explicitly read the capability rather than caching what the previous kernel thinks it was in struct pci_dev_ser. In other words, the kerenl does not assume that hardware registers remain unchanged across kexec. It reads them after kexec to determine what state the device is in and inherits that. [*] The other reason is just to keep the ABI as minimal as possible. > > Say, if the bridge experiences a reset (e.g. link drop etc) during the > > kexec blackout, the PCI_EXP_DEVCTL2 register could revert to its defaul= t > > state, meaning the ARI bit will be 0. > > This does seem like something to be concerned about, but realistically > I think if you get a PCIe error I'm not sure the incoming kernel is > equipped to handle it at all :\ > > Just resuming the driver is going to fail too, I don't know how VFIO > can learn and forward the event, and so on.. > > But maybe it is worth being a little more defensive here If this were to happen, the kernel would see that ARI is disabled on the bridge after kexec and proceed with that state (ARI disabled). Devices (devfn) 8+ on the bus would not be available to use or restore (the kernel would not discover them during scanning). It should be fairly obvious to the user that something went wrong (devices they are trying to restore do not even exist) and they can reboot at that point to recover. This should be extremely rare though as it requires a PCIe error that affects the bridge to race with kexec. This seems like a reasonable way to handle this scenario, at least in the initial version. We can always look at more gracefully handling PCIe errors during kexec for Live Update if-and-when there is value.