From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-alma10-1.taild15c8.ts.net [100.103.45.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0B30A3BE646; Thu, 8 Oct 2026 06:02:39 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=100.103.45.18 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1791439361; cv=none; b=YAXxD3aBeUhyBE3TDxuJXggrMjpp0nlTVgyaQlnHbiSF5q0pb3VxVXoUZh9rG3BTmxzGuK4Wt6AfX5N67RLE+RTXlR9B4GQEMsFdcmS+NHWm6WZRLIa5X5LRwHzs/AmLxxZZRcuZKA5oChP9fXcgFwj+8taCwQJZHawbmB//ZAE= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1791439361; c=relaxed/simple; bh=ySD69JaYnYXUWe8qqHxCFspzzyiheAdkp9kpF9AQoDA=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=Wcyzz3n66hSc4rHXbRtiyv3XpBbx6PiG9IUM38JuABd+GdGjZOYtWqECI9YwFx4W49SJbAExGyRHu01msSIJtB/02nzN2JBm23dRrP/Zap4M9szatWE0KhOjQxUYGB8On3EPSwIV5qrw7leOmMjerB3zcjoA5GwFXS5y4CQo1fE= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=HHiD2C3O; arc=none smtp.client-ip=100.103.45.18 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="HHiD2C3O" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 594AF1F00893; Thu, 8 Oct 2026 06:02:31 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1791439359; bh=PH2hucRXCaT23rZGCU5tNpw03+Q8jQ2iFx4qgOvu23A=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=HHiD2C3OAvKRh3kfhypdVp/Hmpje5WwTlaZeXzO2kFz7AheFFLwnxh+P797KsiWXD DJ12mWjsi3lVNLhe6HuwWbZUsS2Na/ICC6xurIhSKwbt2qZSnWBqdl6stAnHURLfjI 7LrY4rBOlh4Uyyw2CVrX6NJNV5HAW6nBgEd4PAtzRIFMED7W3X8t1+vT3u/KR/5xs2 Av41Y60Nc4qH1C2ecuSeXcx8Heu71xCPXradqxfhBZYvFH6hCHz4EC3Fp95glufOj0 +N79s+fvnMcbbhyKUvnSOkwCjR4xpcqOLeVfNUGEq1DB4f/M3BRcFHU0a5tF6SDNQY c+qRuIFuR8mng== From: "Aneesh Kumar K.V (Arm)" To: iommu@lists.linux.dev Cc: "Aneesh Kumar K.V (Arm)" , Alex Williamson , Alexey Kardashevskiy , Bjorn Helgaas , Catalin Marinas , Jacob Pan , Jason Gunthorpe , Joerg Roedel , Jonathan Cameron , Jonathan Hunter , Kevin Tian , Krishna Reddy , Lukas Wunner , Nicolin Chen , Robin Murphy , Samuel Ortiz , Shameer Kolothum , Steven Price , Suravee Suthikulpanit , Suzuki K Poulose , Thierry Reding , Vasant Hegde , Will Deacon , Xu Yilun , kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-coco@lists.linux.dev, linux-kernel@vger.kernel.org, linux-pci@vger.kernel.org, linux-tegra@vger.kernel.org, Jonathan Cameron Subject: [PATCH v7 16/16] PCI/TSM: wait for vdevice contexts before removing a DSM Date: Thu, 8 Oct 2026 11:29:55 +0530 Message-ID: <20261008055955.4014342-17-aneesh.kumar@kernel.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20261008055955.4014342-1-aneesh.kumar@kernel.org> References: <20261008055955.4014342-1-aneesh.kumar@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit A PF0 DSM can be removed while a sibling function still has a vdevice. The context pins both the pci dev, but those references do not keep the PF0 DOE mailbox alive. Ignoring -EBUSY from link disconnect lets PCI continue to pci_doe_destroy(), leaving the vdevice with a stale mailbox pointer. This follows the VFIO PCI removal model: vfio_unregister_group_dev() prevents new userspace opens and waits for existing users to release the device before teardown proceeds. Likewise, DSM removal rejects new contexts and waits for existing vdevice contexts to drain before destroying the DOE mailbox. Mark the DSM as removing so no new contexts or subfunctions can attach. Wait for the last context to be released before disconnecting the link, VFIO PCI also uses an eventfd to notify userspace that the device should be released. This patch does not add an equivalent notification for vdevice contexts; DSM removal waits for userspace to release them. Such a notification can be added later if required. Cc: Bjorn Helgaas Cc: Jonathan Cameron Cc: Alexey Kardashevskiy Cc: Xu Yilun Cc: Lukas Wunner Cc: Samuel Ortiz Cc: Suzuki K Poulose Cc: Jason Gunthorpe Cc: Kevin Tian Signed-off-by: Aneesh Kumar K.V (Arm) --- drivers/pci/tsm.c | 63 +++++++++++++++++++++++++++++++++++++++-- include/linux/pci-tsm.h | 5 ++++ 2 files changed, 65 insertions(+), 3 deletions(-) diff --git a/drivers/pci/tsm.c b/drivers/pci/tsm.c index c8603867e09d..8aa74ba31932 100644 --- a/drivers/pci/tsm.c +++ b/drivers/pci/tsm.c @@ -10,10 +10,13 @@ #include #include +#include #include #include #include #include +#include +#include #include #include #include @@ -354,6 +357,12 @@ struct pci_tsm_context *pci_tsm_context_get(struct pci_dev *pdev, return ERR_PTR(-ENOMEM); guard(mutex)(&pf0->lock); + if (pf0->removing) { + kfree(context); + return ERR_PTR(-ENODEV); + } + if (!pf0->context_users) + reinit_completion(&pf0->contexts_drained); pf0->context_users++; context->pf0 = pf0; context->pdev = pci_dev_get(pdev); @@ -368,8 +377,11 @@ void pci_tsm_context_put(struct pci_tsm_context *context) down_read(&pci_tsm_rwsem); mutex_lock(&pf0->lock); - if (!WARN_ON(!pf0->context_users)) - pf0->context_users--; + if (WARN_ON(!pf0->context_users)) + goto out_unlock; + if (!--pf0->context_users) + complete_all(&pf0->contexts_drained); +out_unlock: mutex_unlock(&pf0->lock); up_read(&pci_tsm_rwsem); @@ -452,6 +464,9 @@ static ssize_t disconnect_store(struct device *dev, tsm_dev = pdev->tsm->tsm_dev; if (!sysfs_streq(buf, dev_name(&tsm_dev->dev))) return -EINVAL; + if (is_link_tsm(tsm_dev) && is_pci_tsm_pf0(pdev) && + to_pci_tsm_pf0(pdev->tsm)->removing) + return -ENODEV; rc = pci_tsm_disconnect(pdev); if (rc) @@ -663,6 +678,9 @@ int pci_tsm_pf0_constructor(struct pci_dev *pdev, struct pci_tsm_pf0 *tsm, struct tsm_dev *tsm_dev) { mutex_init(&tsm->lock); + init_completion(&tsm->contexts_drained); + tsm->context_users = 0; + tsm->removing = false; tsm->doe_mb = pci_find_doe_mailbox(pdev, PCI_VENDOR_ID_PCI_SIG, PCI_DOE_FEATURE_CMA); if (!tsm->doe_mb) { @@ -751,8 +769,44 @@ static void __pci_tsm_destroy(struct pci_dev *pdev, struct tsm_dev *tsm_dev) void pci_tsm_destroy(struct pci_dev *pdev) { - guard(rwsem_write)(&pci_tsm_rwsem); + struct pci_tsm_pf0 *pf0 = NULL; + struct completion *drained; + bool interrupted = false; + long rc; + + down_write(&pci_tsm_rwsem); + if (pdev->tsm && is_link_tsm(pdev->tsm->tsm_dev) && + is_pci_tsm_pf0(pdev)) { + pf0 = to_pci_tsm_pf0(pdev->tsm); + drained = &pf0->contexts_drained; + mutex_lock(&pf0->lock); + pf0->removing = true; + mutex_unlock(&pf0->lock); + + /* An unused DSM may never have completed contexts_drained. */ + rc = pf0->context_users ? + try_wait_for_completion(drained) : 1; + /* Context release needs the read side of pci_tsm_rwsem. */ + up_write(&pci_tsm_rwsem); + while (rc <= 0) { + if (interrupted) { + rc = wait_for_completion_timeout(drained, HZ * 10); + } else { + rc = wait_for_completion_interruptible_timeout(drained, + HZ * 10); + if (rc < 0) { + interrupted = true; + pci_warn(pdev, "Task \"%s\" (%d) blocked until vdevices are released\n", + current->comm, task_pid_nr(current)); + } + } + if (!rc) + pci_warn(pdev, "TSM connection is in use, waiting for vdevices\n"); + } + down_write(&pci_tsm_rwsem); + } __pci_tsm_destroy(pdev, NULL); + up_write(&pci_tsm_rwsem); } void pci_tsm_init(struct pci_dev *pdev) @@ -779,6 +833,9 @@ void pci_tsm_init(struct pci_dev *pdev) */ if (!dsm->tsm) return; + if (is_link_tsm(dsm->tsm->tsm_dev) && + to_pci_tsm_pf0(dsm->tsm)->removing) + return; probe_fn(pdev, dsm); } diff --git a/include/linux/pci-tsm.h b/include/linux/pci-tsm.h index b27f7cf99f22..3adc317d0f9b 100644 --- a/include/linux/pci-tsm.h +++ b/include/linux/pci-tsm.h @@ -1,6 +1,7 @@ /* SPDX-License-Identifier: GPL-2.0 */ #ifndef __PCI_TSM_H #define __PCI_TSM_H +#include #include #include @@ -107,12 +108,16 @@ struct pci_tsm { * @lock: mutual exclustion for pci_tsm_ops invocation * @context_users: live per-function contexts on this PF0; a nonzero count * blocks link disconnect + * @contexts_drained: completed when the last context is released + * @removing: reject new contexts while the DSM is being removed * @doe_mb: PCIe Data Object Exchange mailbox */ struct pci_tsm_pf0 { struct pci_tsm base_tsm; struct mutex lock; unsigned int context_users; + struct completion contexts_drained; + bool removing; struct pci_doe_mb *doe_mb; }; -- 2.43.0