[PATCHv3] dmaengine: mv_xor: order descriptor writes before engine access

[PATCHv3] dmaengine: mv_xor: order descriptor writes before engine access

[Rosen Penev]

The descriptor pool is allocated with dma_alloc_wc(), so descriptor
writes sit in the CPU write buffers and only reach the engine when the
buffers are flushed.  The engine can fetch a descriptor as soon as it
is pointed at one, so those writes must be ordered against the MMIO
registers and the descriptor links that hand ownership over.

mv_chan_set_next_descriptor() programs the next-descriptor register
with writel() instead of writel_relaxed(); the barrier inside writel()
drains prior descriptor writes before the MMIO store that points the
engine at them.

The chain-append path in mv_xor_tx_submit() needs two more barriers.
A dma_wmb() before mv_desc_set_next_desc() orders the stores that
initialize sw_desc->hw_desc in the prep functions before the store
that links the descriptor into the chain; otherwise a running engine
could follow the new link and read a half-written descriptor.  The
existing mb() after it orders the link store before the busy-status
and current-descriptor register reads, which the relaxed readl path
would otherwise bypass while the link write is still in the write
buffer, letting the CPU restart a channel the engine has already raced
past the tail of.

Fixes: ff7b04796d98 ("dmaengine: DMA engine driver for Marvell XOR engine")
Assisted-by: LLM
Signed-off-by: Rosen Penev <rosenp@gmail.com>
---
 v3: add back dma_wmb
 v2: split off from main patch
 drivers/dma/mv_xor.c | 15 ++++++++++++++-
 1 file changed, 14 insertions(+), 1 deletion(-)

diff --git a/drivers/dma/mv_xor.c b/drivers/dma/mv_xor.c
index da8eea8789ae..c46ee95fe01a 100644
--- a/drivers/dma/mv_xor.c
+++ b/drivers/dma/mv_xor.c
@@ -103,7 +103,11 @@ static u32 mv_chan_get_current_desc(struct mv_xor_chan *chan)
 static void mv_chan_set_next_descriptor(struct mv_xor_chan *chan,
 					u32 next_desc_addr)
 {
-	writel_relaxed(next_desc_addr, XOR_NEXT_DESC(chan));
+	/*
+	 * writel drains descriptor writes to DRAM before the engine
+	 * is pointed at them
+	 */
+	writel(next_desc_addr, XOR_NEXT_DESC(chan));
 }
 
 static void mv_chan_unmask_interrupts(struct mv_xor_chan *chan)
@@ -407,9 +411,18 @@ mv_xor_tx_submit(struct dma_async_tx_descriptor *tx)
 		dev_dbg(mv_chan_to_devp(mv_chan), "Append to last desc %pa\n",
 			&old_chain_tail->async_tx.phys);
 
+		/* commit the new descriptor before chaining it in */
+		dma_wmb();
+
 		/* fix up the hardware chain */
 		mv_desc_set_next_desc(old_chain_tail, sw_desc->async_tx.phys);
 
+		/*
+		 * make the new link visible to the engine before we read
+		 * the channel state, the device may fetch it at any point
+		 */
+		mb();
+
 		/* if the channel is not busy */
 		if (!mv_chan_is_busy(mv_chan)) {
 			u32 current_desc = mv_chan_get_current_desc(mv_chan);
-- 
2.55.0