[PATCH 11/17] dmaengine: idxd: disable the device before its work queues are unregistered

[Daniel Byrne <daniel1.byrne@intel.com>]

idxd_remove() unregisters the work queues, and with them their
software state, before idxd_shutdown() disables the device. The
sub-driver's remove leaves the device enabled when a device command has
timed out, and a late completion can clear that fail-fast state at any
time, so the disable in idxd_shutdown() may be the one that runs. It
then clears the software state of queues that no longer exist.

Disable the device right after the sub-driver is unbound, while the
queues still exist. Mark the device disabled once bus mastering is off,
so that nothing reached through the device reads it as enabled, and do
so before the reference taken for the cleanup is dropped: that
reference is the last one, and its release frees the device.

Assisted-by: Claude:claude-opus-5-5
Assisted-by: Claude:claude-fable-5-1
Signed-off-by: Daniel Byrne <daniel1.byrne@intel.com>
---
 drivers/dma/idxd/init.c | 16 ++++++++++++++--
 1 file changed, 14 insertions(+), 2 deletions(-)

diff --git a/drivers/dma/idxd/init.c b/drivers/dma/idxd/init.c
index c4319ac27bfb..a037a6e76362 100644
--- a/drivers/dma/idxd/init.c
+++ b/drivers/dma/idxd/init.c
@@ -1296,6 +1296,13 @@ static void idxd_remove(struct pci_dev *pdev)
 	 * that driver while the queues still exist.
 	 */
 	device_release_driver(idxd_confdev(idxd));
+	/*
+	 * Disable the device while the queues still exist too. The
+	 * sub-driver's remove leaves the device enabled when a command has
+	 * timed out, a late completion can clear that state at any time,
+	 * and the disable below then clears the queues' software state.
+	 */
+	idxd_shutdown(pdev);
 	idxd_unregister_devices(idxd);
 	/*
 	 * When ->release() is called for the idxd->conf_dev, it frees all
@@ -1305,15 +1312,20 @@ static void idxd_remove(struct pci_dev *pdev)
 	 */
 	get_device(idxd_confdev(idxd));
 	device_unregister(idxd_confdev(idxd));
-	idxd_shutdown(pdev);
 	idxd_device_remove_debugfs(idxd);
 	perfmon_pmu_remove(idxd);
 	idxd_cleanup_interrupts(idxd);
 	if (device_pasid_enabled(idxd))
 		idxd_disable_system_pasid(idxd);
 	pci_iounmap(pdev, idxd->reg_base);
-	put_device(idxd_confdev(idxd));
 	pci_disable_device(pdev);
+	/*
+	 * Bus mastering is off, so nothing the device holds can complete.
+	 * The store comes before the reference taken above is dropped:
+	 * that reference is the last one, and its release frees the device.
+	 */
+	idxd->state = IDXD_DEV_DISABLED;
+	put_device(idxd_confdev(idxd));
 }
 
 static struct pci_driver idxd_pci_driver = {
-- 
2.43.0