[PATCH 14/17] dmaengine: idxd: complete an iaa_crypto descriptor through its own driver on abort and flush

[Daniel Byrne <daniel1.byrne@intel.com>]

The teardown flush and the submit abort complete every descriptor as a
dmaengine transaction, whichever driver owns the queue. An iaa_crypto
queue is a kernel queue as well, and its interrupt-armed descriptors
carry the crypto request where a dmaengine descriptor has its txd, so
completing one of them as a transaction takes the request pointer for
a cookie and a DMA address for the channel, and either hits the cookie
BUG_ON() or stores through that address. A single iaa_crypto client
reaches this without a halt or a shared channel: in the teardown flush
after a drain that failed or after free_irq() lost a completion
interrupt, and in a guest on every failed interrupt-armed submission to
a shared queue that other guests keep full.

Complete such a descriptor through the completion of the driver that
owns the queue, as the interrupt thread does, and mark one the device
never finished as aborted first so that the driver fails its request.
The txd handling, the callback strip and the quarantine stay with
dmaengine queues. A failed submission of another driver's descriptor is
left entirely to its submitter, which learns of it from the return
value.

The teardown flush frees each descriptor it completes only when that
completion returns, and iaa_crypto's verify allocates from inside one.
A blocking allocation therefore fails at once on a queue whose
wq_active is dead with no revive to come, rather than wait for a slot
only the flush could free. Every submission there fails anyway.

Fixes: 09646c98d0bf ("crypto: iaa - Add irq support for the crypto async interface")
Cc: stable@vger.kernel.org # 6.8+
Assisted-by: Claude:claude-opus-5
Assisted-by: Claude:claude-opus-5-5
Assisted-by: Claude:claude-fable-5-1
Signed-off-by: Daniel Byrne <daniel1.byrne@intel.com>
---
 drivers/dma/idxd/device.c | 23 +++++++++++++++++++----
 drivers/dma/idxd/idxd.h   | 10 ++++++++++
 drivers/dma/idxd/submit.c | 22 +++++++++++++++++++++-
 3 files changed, 50 insertions(+), 5 deletions(-)

diff --git a/drivers/dma/idxd/device.c b/drivers/dma/idxd/device.c
index 222c201669bd..a33c4309f5d4 100644
--- a/drivers/dma/idxd/device.c
+++ b/drivers/dma/idxd/device.c
@@ -1534,10 +1534,25 @@ static void idxd_flush_pending_descs(struct idxd_irq_entry *ie,
 		list_del(&desc->list);
 		ctype = desc->completion->status ? IDXD_COMPLETE_NORMAL : IDXD_COMPLETE_ABORT;
 		/*
-		 * wq is being disabled. Any remaining descriptors are
-		 * likely to be stuck and can be dropped. callback could
-		 * point to code that is no longer accessible, for example
-		 * if dmatest module has been unloaded.
+		 * Another driver's descriptor carries that driver's request
+		 * in place of a txd, and a client may still wait for it, so it
+		 * goes back through that driver's completion. The completion
+		 * reads the outcome from the record, which the device never
+		 * wrote for an unfinished descriptor. Only a dmaengine channel
+		 * terminates, so nothing of such a queue is quarantined.
+		 */
+		if (!idxd_wq_has_txd(wq)) {
+			if (ctype == IDXD_COMPLETE_ABORT)
+				desc->completion->status = IDXD_COMP_DESC_ABORT;
+			idxd_desc_complete(desc, ctype, true);
+			continue;
+		}
+		/*
+		 * wq is being disabled or terminated. Any remaining
+		 * descriptors are likely to be stuck and can be dropped.
+		 * With no client left the callback could point to code that
+		 * is no longer accessible (dmatest unloaded), and a client's
+		 * own terminate must not get it, so it is stripped.
 		 */
 		tx = &desc->txd;
 		tx->callback = NULL;
diff --git a/drivers/dma/idxd/idxd.h b/drivers/dma/idxd/idxd.h
index 9d88b0d82fee..3caf9fc00f83 100644
--- a/drivers/dma/idxd/idxd.h
+++ b/drivers/dma/idxd/idxd.h
@@ -755,6 +755,16 @@ void idxd_dma_complete_txd(struct idxd_desc *desc,
 			   enum idxd_complete_type comp_type,
 			   bool free_desc, void *ctx, u32 *status);
 
+/*
+ * Whether the queue's descriptors are dmaengine transactions. Another
+ * driver, such as iaa_crypto, keeps its own context where they keep
+ * their txd.
+ */
+static inline bool idxd_wq_has_txd(struct idxd_wq *wq)
+{
+	return wq_to_idxd_drv(wq)->desc_complete == idxd_dma_complete_txd;
+}
+
 static inline void idxd_desc_complete(struct idxd_desc *desc,
 				      enum idxd_complete_type comp_type,
 				      bool free_desc)
diff --git a/drivers/dma/idxd/submit.c b/drivers/dma/idxd/submit.c
index 5806b1b74a2f..fab3e0176cba 100644
--- a/drivers/dma/idxd/submit.c
+++ b/drivers/dma/idxd/submit.c
@@ -52,6 +52,21 @@ struct idxd_desc *idxd_alloc_desc(struct idxd_wq *wq, enum idxd_op_type optype)
 		return __get_desc(wq, idx, cpu);
 	}
 
+	/*
+	 * A dying reference whose resurrect has completed belongs to a
+	 * queue quiesced for good: no kill in progress will revive it, and
+	 * every submission fails. Waiting for a slot there gains nothing
+	 * and may never end. The teardown flush and the interrupt thread
+	 * free the slot of each descriptor they complete only when its
+	 * completion returns, and iaa_crypto's verify of a compression
+	 * allocates from inside one. The two reads are not atomic, so an
+	 * allocation that races a revive can fail like a request the kill
+	 * caught.
+	 */
+	if (data_race(percpu_ref_is_dying(&wq->wq_active)) &&
+	    completion_done(&wq->wq_resurrect))
+		return ERR_PTR(-ENXIO);
+
 	ws = &sbq->ws[0];
 	for (;;) {
 		sbitmap_prepare_to_wait(sbq, ws, &wait, TASK_INTERRUPTIBLE);
@@ -149,7 +164,12 @@ static void llist_abort_desc(struct idxd_wq *wq, struct idxd_irq_entry *ie,
 	 * complete and free it, and the caller frees it as well.
 	 */
 	WARN_ON_ONCE(!found);
-	if (found)
+	/*
+	 * Another driver's submitter cleans up all of its own: it learns
+	 * of the failure from the return value and its descriptor carries
+	 * its request where a dmaengine descriptor has its txd.
+	 */
+	if (found && idxd_wq_has_txd(wq))
 		idxd_dma_complete_txd(desc, IDXD_COMPLETE_ABORT, false,
 				      NULL, NULL);
 }
-- 
2.43.0