[PATCH net] net: xilinx: axienet: Free outstanding DMA buffers on dmaengine stop

[PATCH net] net: xilinx: axienet: Free outstanding DMA buffers on dmaengine stop

[Suraj Gupta]

In the dmaengine path the driver pre-submits RX buffers (and holds
in-flight TX buffers) whose SKBs are allocated and DMA-mapped by the
driver and reclaimed only in the completion callbacks. On ndo_stop()
the driver calls dmaengine_terminate_sync(), which aborts these
descriptors without running their callbacks, and then frees only the
skbuf_dma_descriptor ring shells with kfree().

As a result every SKB still owned by the DMA engine, together with its
DMA mapping, is leaked on each interface down. With RX_BUF_NUM_DEFAULT
(128) RX buffers pre-posted per channel, an ifup/ifdown cycle leaks up
to 128 SKBs and their DMA mappings; on systems with a limited IOMMU
aperture the mapping leak can eventually exhaust the address space.

Clear the ring slot's skb pointer in the TX and RX completion callbacks
so that a non-NULL skb reliably marks a slot that still owns a live,
DMA-mapped buffer. On stop, after terminating the channels, walk the
whole ring and unmap and free every such buffer before releasing the
rings, mirroring what the completion callbacks do.

Fixes: 6a91b846af85 ("net: axienet: Introduce dmaengine support")
Cc: stable@vger.kernel.org
Signed-off-by: Suraj Gupta <suraj.gupta2@amd.com>
---
 .../net/ethernet/xilinx/xilinx_axienet_main.c | 34 ++++++++++++++++---
 1 file changed, 30 insertions(+), 4 deletions(-)

diff --git a/drivers/net/ethernet/xilinx/xilinx_axienet_main.c b/drivers/net/ethernet/xilinx/xilinx_axienet_main.c
index 1722b7038f34..7828fbb09dc8 100644
--- a/drivers/net/ethernet/xilinx/xilinx_axienet_main.c
+++ b/drivers/net/ethernet/xilinx/xilinx_axienet_main.c
@@ -881,6 +881,7 @@ static void axienet_dma_tx_cb(void *data, const struct dmaengine_result *result)
 	u64_stats_update_end(&lp->tx_stat_sync);
 	dma_unmap_sg(lp->dev, skbuf_dma->sgl, skbuf_dma->sg_len, DMA_TO_DEVICE);
 	dev_consume_skb_any(skbuf_dma->skb);
+	skbuf_dma->skb = NULL;
 	netif_txq_completed_wake(txq, 1, len,
 				 CIRC_SPACE(lp->tx_ring_head, lp->tx_ring_tail, TX_BD_NUM_MAX),
 				 2);
@@ -1171,6 +1172,7 @@ static void axienet_dma_rx_cb(void *data, const struct dmaengine_result *result)
 						       &meta_max_len);
 	dma_unmap_single(lp->dev, skbuf_dma->dma_address, lp->max_frm_size,
 			 DMA_FROM_DEVICE);
+	skbuf_dma->skb = NULL;
 
 	if (IS_ERR(app_metadata)) {
 		if (net_ratelimit())
@@ -1752,16 +1754,40 @@ static int axienet_stop(struct net_device *ndev)
 		free_irq(lp->rx_irq, ndev);
 		axienet_dma_bd_release(ndev);
 	} else {
+		struct skbuf_dma_descriptor *skbuf_dma;
+
 		dmaengine_terminate_sync(lp->tx_chan);
 		dmaengine_synchronize(lp->tx_chan);
 		dmaengine_terminate_sync(lp->rx_chan);
 		dmaengine_synchronize(lp->rx_chan);
 
-		for (i = 0; i < TX_BD_NUM_MAX; i++)
-			kfree(lp->tx_skb_ring[i]);
+		/* dmaengine_terminate_sync() aborts the descriptors still owned
+		 * by the DMA engine without running their completion callbacks.
+		 * A ring slot owns a live, DMA-mapped SKB iff its skb pointer is
+		 * non-NULL (the callbacks clear it on completion), so unmap and
+		 * free those here. Otherwise every outstanding TX/RX SKB and its
+		 * DMA mapping is leaked on ifdown.
+		 */
+		for (i = 0; i < TX_BD_NUM_MAX; i++) {
+			skbuf_dma = lp->tx_skb_ring[i];
+			if (skbuf_dma && skbuf_dma->skb) {
+				dma_unmap_sg(lp->dev, skbuf_dma->sgl,
+					     skbuf_dma->sg_len, DMA_TO_DEVICE);
+				dev_kfree_skb_any(skbuf_dma->skb);
+			}
+			kfree(skbuf_dma);
+		}
 		kfree(lp->tx_skb_ring);
-		for (i = 0; i < RX_BUF_NUM_DEFAULT; i++)
-			kfree(lp->rx_skb_ring[i]);
+
+		for (i = 0; i < RX_BUF_NUM_DEFAULT; i++) {
+			skbuf_dma = lp->rx_skb_ring[i];
+			if (skbuf_dma && skbuf_dma->skb) {
+				dma_unmap_single(lp->dev, skbuf_dma->dma_address,
+						 lp->max_frm_size, DMA_FROM_DEVICE);
+				dev_kfree_skb_any(skbuf_dma->skb);
+			}
+			kfree(skbuf_dma);
+		}
 		kfree(lp->rx_skb_ring);
 
 		dma_release_channel(lp->rx_chan);
-- 
2.25.1

Re: [PATCH net] net: xilinx: axienet: Free outstanding DMA buffers on dmaengine stop

[Jakub Kicinski]

Let's fix what AI is pointing out in another patch in the same series?
Or please explain in the commit msg why the issue is not real,
I'm not very familiar with the DMA engine API, TBH.

AI review:
---
net: xilinx: axienet: Free outstanding DMA buffers on dmaengine stop

This adds clearing of the ring slot skb pointer in the TX and RX dmaengine
completion callbacks, and makes axienet_stop() walk the TX and RX rings
after terminating the channels to unmap and free any slot that still has a
non-NULL skb, instead of only kfree()ing the ring shells.

> diff --git a/drivers/net/ethernet/xilinx/xilinx_axienet_main.c b/drivers/net/ethernet/xilinx/xilinx_axienet_main.c
> index 1722b7038f340..7828fbb09dc87 100644
> --- a/drivers/net/ethernet/xilinx/xilinx_axienet_main.c
> +++ b/drivers/net/ethernet/xilinx/xilinx_axienet_main.c

[ ... ]

> @@ -1171,6 +1172,7 @@ static void axienet_dma_rx_cb(void *data, const struct dmaengine_result *result)
>  						       &meta_max_len);
>  	dma_unmap_single(lp->dev, skbuf_dma->dma_address, lp->max_frm_size,
>  			 DMA_FROM_DEVICE);
> +	skbuf_dma->skb = NULL;
>  
>  	if (IS_ERR(app_metadata)) {
>  		if (net_ratelimit())

> @@ -1752,16 +1754,40 @@ static int axienet_stop(struct net_device *ndev)
>  		free_irq(lp->rx_irq, ndev);
>  		axienet_dma_bd_release(ndev);
>  	} else {
> +		struct skbuf_dma_descriptor *skbuf_dma;
> +
>  		dmaengine_terminate_sync(lp->tx_chan);
>  		dmaengine_synchronize(lp->tx_chan);
>  		dmaengine_terminate_sync(lp->rx_chan);
>  		dmaengine_synchronize(lp->rx_chan);
>  
> -		for (i = 0; i < TX_BD_NUM_MAX; i++)
> -			kfree(lp->tx_skb_ring[i]);
> +		/* dmaengine_terminate_sync() aborts the descriptors still owned
> +		 * by the DMA engine without running their completion callbacks.
> +		 * A ring slot owns a live, DMA-mapped SKB iff its skb pointer is
> +		 * non-NULL (the callbacks clear it on completion), so unmap and
> +		 * free those here. Otherwise every outstanding TX/RX SKB and its
> +		 * DMA mapping is leaked on ifdown.
> +		 */

[ ... TX ring loop snipped ... ]

> +
> +		for (i = 0; i < RX_BUF_NUM_DEFAULT; i++) {
> +			skbuf_dma = lp->rx_skb_ring[i];
> +			if (skbuf_dma && skbuf_dma->skb) {
> +				dma_unmap_single(lp->dev, skbuf_dma->dma_address,
> +						 lp->max_frm_size, DMA_FROM_DEVICE);
> +				dev_kfree_skb_any(skbuf_dma->skb);
> +			}
> +			kfree(skbuf_dma);
> +		}
>  		kfree(lp->rx_skb_ring);
>  
>  		dma_release_channel(lp->rx_chan);

[Severity: High]

Is the invariant described in the comment, "A ring slot owns a live,
DMA-mapped SKB iff its skb pointer is non-NULL", guaranteed to hold once
the terminate calls have returned?

axienet_dma_rx_cb() re-arms the RX channel unconditionally at the end of
every completion, with no check against a stopping flag or
netif_running():

drivers/net/ethernet/xilinx/xilinx_axienet_main.c:axienet_dma_rx_cb() {
	...
rx_submit:
	for (i = 0; i < CIRC_SPACE(lp->rx_ring_head, lp->rx_ring_tail,
				   RX_BUF_NUM_DEFAULT); i++)
		axienet_rx_submit_desc(lp->ndev);
	dma_async_issue_pending(lp->rx_chan);
}

Note lp->stopping is only set in the !lp->use_dmaengine branch of
axienet_stop(), so it does not fence this path.

The callback runs from the provider tasklet with chan->lock dropped, so it
can run on another CPU while axienet_stop() is between the terminate and
the free loop:

drivers/dma/xilinx/xilinx_dma.c:xilinx_dma_chan_desc_cleanup() {
	...
	spin_unlock_irqrestore(&chan->lock, flags);
	dmaengine_desc_get_callback_invoke(&desc->async_tx, &result);
	spin_lock_irqsave(&chan->lock, flags);
}

If the callback wins that race, axienet_rx_submit_desc() maps a fresh
buffer, installs it in the slot and submits it:

	skbuf_dma->skb = skb;
	skbuf_dma->dma_address = sg_dma_address(skbuf_dma->sgl);
	skbuf_dma->desc = dma_rx_desc;
	...
	lp->rx_ring_head++;
	dmaengine_submit(dma_rx_desc);

and xilinx_dma_tx_submit() queues it and clears the provider's guard:

	append_desc_queue(chan, desc);
	...
	chan->terminating = false;

The following dma_async_issue_pending() then restarts the channel.
dmaengine_synchronize() is only tasklet_kill() here:

drivers/dma/xilinx/xilinx_dma.c:
static void xilinx_dma_synchronize(struct dma_chan *dchan)
{
	struct xilinx_dma_chan *chan = to_xilinx_chan(dchan);

	tasklet_kill(&chan->tasklet);
}

so it waits for the callback to return but does not abort the descriptor the
callback just submitted.

In that case the new RX loop sees skbuf_dma->skb != NULL for that slot and
performs dma_unmap_single() plus dev_kfree_skb_any() on a buffer the engine
still owns.  Can this let the hardware write into a freed, unmapped skb, or
trigger an IOMMU fault?

There is also the ordering of kfree(lp->rx_skb_ring) before
dma_release_channel(lp->rx_chan).  If the late descriptor completes in that
window, axienet_dma_rx_cb() does:

	skbuf_dma = axienet_get_rx_desc(lp, lp->rx_ring_tail++);
	skb = skbuf_dma->skb;

Is that a use-after-free of lp->rx_skb_ring?

Before this change the same window only leaked the buffer.  Would it make
sense to fence resubmission first, for example by setting the stopping flag
for the dmaengine path before the terminate calls and checking it in
axienet_dma_rx_cb() before the rx_submit loop, and to release the channels
before freeing the rings?

Note also that dmaengine_terminate_sync() already ends with
dmaengine_synchronize(), so the explicit dmaengine_synchronize() calls that
follow it here are redundant, though harmless.
-- 
pw-bot: cr