From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-pl1-f169.google.com (mail-pl1-f169.google.com [209.85.214.169]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7F6E4369222 for ; Mon, 16 Mar 2026 21:49:40 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.169 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1773697782; cv=none; b=WJ06F4G1Mv4OVCZz0FKSdcBI1U6Mn8xnnivzILhA4NRFvg4Iqi+WXtheppJvczkb10crxR+frZOyqPQghwIpXTnexjIqMwHOvqj1Do+PsMBc0X1dY/UbYLqKI3vT3wfTzH8jifhxFIMgoPlas1vhAhVCTjtAuiBvTcKfQv/eUe0= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1773697782; c=relaxed/simple; bh=wQ/TGkC56TEVU9edxRoDW8klAaLsoZRqlEjOzTkcHtQ=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=Ht304k2C5/2od0dGUr+UuSXUCU5ooCB+LDgSJddB7nj/UsAk4MoGy1dFXKEhYVJcfRAhynqP086bTw05dFdMmA5CWGFHe/9yUZHcfZibEbI8ZzZ/sOboeV8c9baKnCyOeX+YMs+GbaQ42vSrn2vNyAbjE48lCJLNoZlAr/V20vQ= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=Sz9aLFQ5; arc=none smtp.client-ip=209.85.214.169 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="Sz9aLFQ5" Received: by mail-pl1-f169.google.com with SMTP id d9443c01a7336-2aeab6ff148so13415ad.1 for ; Mon, 16 Mar 2026 14:49:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1773697780; x=1774302580; darn=vger.kernel.org; h=in-reply-to:content-transfer-encoding:content-disposition :mime-version:references:message-id:subject:cc:to:from:date:from:to :cc:subject:date:message-id:reply-to; bh=/8z1/dIhYqxHzj3/umwHNkEcO1PtVcvAJP18LcAQq1Y=; b=Sz9aLFQ5ZpzjCMQbhP0GaJlUqctaX3L9daYdMHHAWK69ofV+JdrQQNZqqHNfyhJHWT enhHiksoJUu1HFOS7H5sV/6mmQvX287gMHDXX5CQnSr3xv5I/ftgY3RMZpoqGhum6QYn z4R+1BiLxyabC86ANdO18xTJk7zbt2Z3SrDV8yGup3msBjRNI2SYLUdBjwZUnwAGSVn6 LMhCIwfYfZWZXxrJCyIpIuAKBOekqIsZhuhPIywVxn1rrlOMsD6YZI8HwEDuT/lydBNT 3Zpgfy+IOr6GTSSADS1sYM5OjTSeSAkISsKbU/j27ic2WwLzAmamilpN7CSDCTcFAxux 9+Vg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1773697780; x=1774302580; h=in-reply-to:content-transfer-encoding:content-disposition :mime-version:references:message-id:subject:cc:to:from:date:x-gm-gg :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=/8z1/dIhYqxHzj3/umwHNkEcO1PtVcvAJP18LcAQq1Y=; b=esqN9yKRWkJ/Ox/0Fj8UheZlfLZrKlAikhllMjPeWHVcoeljjuO0ZsagZCsKPXrxuE 8Y7kQwqg2w+U6DIIXiwplhkmmif4M3gMckE/ZtzqDQaO5GzRVGI00kT1XCm9pi1Qky+X e8oo8O9z8MiGPjbwD+XI1ArILLsd4EUpQQaS3GdLtPNLuOut/dkkmI76pFSrMauxzHPL h5/PX8/ionlm2C8oDSO+DgI9xKdw5GvWMGKRBb71Ic94A2nO0dTJyUHH3norkX+hmY+M JLUzo8vZEe8VM3aKrrkIcmBqHZrl2qdA+ECUFMOUfDxTA0Ff00O/RpNKX4EYG102cK7M j97A== X-Forwarded-Encrypted: i=1; AJvYcCUtORLP+TeB+NpkCBbjEXbLFeFVIVq05S6y7UuKyB4mehHSt7ve0JwN1LykMUAgHf/qQb0UetlhLCFrMbQ=@vger.kernel.org X-Gm-Message-State: AOJu0Yw14Kd6+0Mmss5abQZvuBLIaHig5kTl5BfceL+ZOImcXJlLBc4U t53VqKrJUFVXSmkK7pWbRj77exdSevpCK5DJ1e5yj8Hvj3zn4NcAfjka116gL9oz1g== X-Gm-Gg: ATEYQzxoaMKNwTmxIIEd9wuN76f3F5/mqNOaGSt+32Tt82DWAM0BQ76yhtnQ7FRZi2n y9thegg6+S4y2wyHp4ziCvp5LQK8OBVZri7jPILKbAxqoNa1fRGMgaDeIewEiK4OTuKFBva0sKZ gBn5qwD+605yO+10wyehqTEnozUGjes64cwMI+djecJhACj8rCXIz5z0nqHCzSrn2sLCpql7k1g sgTs7NT+wWKskCd5CCeIqh3sDUNQK4Bl4FmdclVlazWTc3VNac6ymNeOCLhYmBHS8It2+ezHO5Z OB6FIRNoDYdKRImhPFKXw8bq+xTvZwp1DGeQ3ABred9A+0VCJ2h6FY89UZYuJOX2a+4ntsQq6GL NtCBuxDOCTN8QIKF+2TF78KOaktE3IDpQocFf/FI4ZcJ91ttYyz6bmUAmEtU6QSvsj8dM1i4M0J 4/VL573Kz5iEheM4oUuTLMG6uwi7pLYXOwNZeWudR9YC+0nsDzRcNhjcr03c6y X-Received: by 2002:a17:903:98b:b0:2a8:ffed:4663 with SMTP id d9443c01a7336-2b06402f49emr1238925ad.12.1773697779367; Mon, 16 Mar 2026 14:49:39 -0700 (PDT) Received: from google.com (176.13.105.34.bc.googleusercontent.com. [34.105.13.176]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-82a07365b22sm14359627b3a.45.2026.03.16.14.49.38 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 16 Mar 2026 14:49:38 -0700 (PDT) Date: Mon, 16 Mar 2026 14:49:34 -0700 From: Vipin Sharma To: David Matlack Cc: Alex Williamson , Adithya Jayachandran , Alexander Graf , Alex Mastro , Alistair Popple , Andrew Morton , Ankit Agrawal , Bjorn Helgaas , Chris Li , David Rientjes , Jacob Pan , Jason Gunthorpe , Jason Gunthorpe , Jonathan Corbet , Josh Hilke , Kevin Tian , kexec@lists.infradead.org, kvm@vger.kernel.org, Leon Romanovsky , Leon Romanovsky , linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-kselftest@vger.kernel.org, linux-mm@kvack.org, linux-pci@vger.kernel.org, Lukas Wunner , =?utf-8?Q?Micha=C5=82?= Winiarski , Mike Rapoport , Parav Pandit , Pasha Tatashin , Pranjal Shrivastava , Pratyush Yadav , Raghavendra Rao Ananta , Rodrigo Vivi , Saeed Mahameed , Samiullah Khawaja , Shuah Khan , Thomas =?utf-8?Q?Hellstr=C3=B6m?= , Tomita Moeko , Vivek Kasireddy , William Tu , Yi Liu , Zhu Yanjun Subject: Re: [PATCH v2 10/22] vfio/pci: Skip reset of preserved device after Live Update Message-ID: <20260316214055.GB1846904.vipinsh@google.com> References: <20260129212510.967611-11-dmatlack@google.com> <20260226170030.5a938c74@shazbot.org> <20260227084658.3767d801@shazbot.org> <20260227105720.522ca97f@shazbot.org> <20260316160759.GA1767448.vipinsh@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: On Mon, Mar 16, 2026 at 10:18:22AM -0700, David Matlack wrote: > On Mon, Mar 16, 2026 at 9:22 AM Vipin Sharma wrote: > > > > On Thu, Mar 12, 2026 at 11:39:45PM +0000, David Matlack wrote: > > > On 2026-03-09 10:32 AM, David Matlack wrote: > > > > On Fri, Feb 27, 2026 at 9:57 AM Alex Williamson wrote: > > > > > > > > Sorry if I don't have the whole model in my head yet, but is exposing > > > > > the restriction to the vfio user of the device sufficient to manage the > > > > > liveupdate orchestration? For example, a VFIO_DEVICE_INFO_CAP pushes > > > > > the knowledge to QEMU... what does QEMU do with that knowledge? Who > > > > > imposes the policy decision to decide what support is sufficient? > > > > > > > > Hm.. good questions. I don't think we want userspace inspecting bits > > > > exposed by the kernel and trying to infer exactly what's being > > > > preserved and whether it's "good enough" to use. And such a UAPI would > > > > become tech debt once we finish development, I suspect. > > > > > > > > A better approach would be to hide this support from userspace until > > > > we decide it is ready for production use-cases. > > > > > > > > To enable development and testing, we can add an opt-in mechanism > > > > > > Here is what I am trending towards sending in v3 as the opt-in mechanism: > > > > > > diff --git a/drivers/vfio/pci/Kconfig b/drivers/vfio/pci/Kconfig > > > index 1e82b44bda1a..770231554221 100644 > > > --- a/drivers/vfio/pci/Kconfig > > > +++ b/drivers/vfio/pci/Kconfig > > > @@ -58,6 +58,27 @@ config VFIO_PCI_ZDEV_KVM > > > config VFIO_PCI_DMABUF > > > def_bool y if VFIO_PCI_CORE && PCI_P2PDMA && DMA_SHARED_BUFFER > > > > > > +config VFIO_PCI_LIVEUPDATE > > > + bool "VFIO PCI support for Live Update (EXPERIMENTAL)" > > > + depends on LIVEUPDATE && VFIO_PCI > > > + help > > > + Support for preserving devices bound to vfio-pci across a Live > > > + Update. The eventual goal is that preserved devices can run > > > + uninterrupted during a Live Update, including DMA to preserved > > > + memory buffers and P2P. However there are many steps still needed to > > > + achieve this, including: > > > + > > > + - Preservation of iommufd files > > > + - Preservation of IOMMU driver state > > > + - Preservation of PCI state (BAR resources, device state, ...) > > > + - Preservation of vfio-pci driver state > > > + > > > + This option should only be enabled by developers working on > > > + implementing this support. Once enough support has landed in the > > > + kernel, this option will no longer be marked EXPERIMENTAL. > > > + > > > + If you don't know what to do here, say N. > > > + > > > > To use VFIO liveupdate, user has to do at least two things: > > 1. Enable CONFIG_LIVEUPDATE > > 2. Pass VFIO FD to a live update session. > > > > This means someone using it has to know what live update is and > > intentionally pass the VFIO FDs. Isn't act of doing this itself an > > opt-in mechanism? > > If it is, then I can leave this out. Alex? > > My thinking was: Distros are free to enable LIVEUPDATE and use it. The > support it enables today is all fully functional (albeit new). > vfio-cdev, OTOH, is not. A separate Kconfig can help express that > difference. > > Consider that LIVEUPDATE could be enabled by default in a future > release, but vfio-cdev support might not be ready yet at that point. But that also requires point 2 above i.e. userspace explicitly passing VFIO FD to liveupdate. Unless there is a capability mechanism like KVM then userspace cannot know what is exactly supported. Also, users who are using these APIs will already be advanced users and have to know many details about what liveupdate supports or not. > > > I am not sure providing VFIO_PCI_LIVEUPDATE alleviate Alex's concern > > about how userspace will know that sufficient VFIO support exists. > > I was thinking we can flip VFIO_PCI_LIVEUPDATE to be enabled by > default (if LIVEUPDATE and VFIO_PCI are enabled), and drop > "(EXPERIMENTAL)" from the option title. That would be how distros and > downstream users of the kernel know that sufficient support exists to > enable VFIO_PCI_LIVEUPDATE. > > > May be write in liveupdate documentation (PATCH 11 of this series) that > > support is experimental? > > The documentation in patch 11 includes largely the same text that I > put under VFIO_PCI_LIVEU"PDATE. But I can explicitly mention > "experimental" as well if that's what you're asking. Yeah, even though documentation do get stale but I think there is no better way in this scenario.