From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-pf1-f180.google.com (mail-pf1-f180.google.com [209.85.210.180]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 2710F4334A8 for ; Mon, 27 Jul 2026 20:41:27 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.180 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1785184889; cv=none; b=Ev5s2/439zTFCnvG3UyCC5w9FJHFeUNYDfteYG4jTWBemkOgaZwKbDxJVvCZlI3QHc4PEOAV4PXtt2y/a6MesGLu/I4Qzf01ze9Dz8BLe27K+fsrmiDN6N5CaHb3qjF/Dk8iWWuoEXMHCJuaZPv0yiAJvKZF2Jw/vICZ6eNMqlg= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1785184889; c=relaxed/simple; bh=csZmG/QQd9RZsCilaK/kTmTTDK/yeFglGk5V6PZWL5M=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=AyD6xwzEliwhTH6lB0NRVR/8Ikdozmpt2JJLQ1hJ3khk9+IDbi3MXi0w3mydUWj6ddlOhqiQ3oBevgb0528LrTdaRCzzCuCTcT0gPESLnDLoDacoJPSL2UoCPQEB1k6cBsoZkmFpKqK/Xj2eHa0Kla8t7yXtTwdEpFB/DFZJ4rM= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=VXxBieTx; arc=none smtp.client-ip=209.85.210.180 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="VXxBieTx" Received: by mail-pf1-f180.google.com with SMTP id d2e1a72fcca58-8487b7b3fc8so3031106b3a.3 for ; Mon, 27 Jul 2026 13:41:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1785184887; x=1785789687; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=mlYzshcUuqeXbSVd1G0xYE71JjWKlgpwKVLPDuR4LcQ=; b=VXxBieTxMhzgADMZBOaVRgJzFWR58I+NJQRoyk2TGCSFkMuvwJTgoTQxTk13LYZ96l 5Uya5N9S/B1mRy2xsToG8wBWaxiYberC5k2nXPbFxwY4b/d30mq7OIlTWofyL562OX6P PizmJ+67azAUX7RfOD1Y8MdEmEWOZYmMGo/1qKDAO7mTBCqiCIJyac8CGUVSdB68CZJV GxrtOExtscNjRaAcFOojl/37w30+BYp5vPj+ksyZz/ruVGgi0NSZ8Kq49JjH9Pu0hEtR dt49mp5yVJmCEdep9TsTiMp80PNA4oRDMDNIe7TuF2mGfoPS2zVPAgUIFAD+Ykwod3fI MQgA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1785184887; x=1785789687; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=mlYzshcUuqeXbSVd1G0xYE71JjWKlgpwKVLPDuR4LcQ=; b=lZmxDoQxpKUsI887RhMKJtpE26Cp6v9LbY9DE6SvSA/QKEosXSLqKOAk36zU2VgfT1 nvc0sw2NF6AxJ4/0LBJ9pIDDJ344GZPGdYZxHRKRI75UToco8OhLp8umR2Odm/xsCloA CVOP3p9Gl5KHIhv75YlRCxI2Qx/Q5bVLTxhb5d++glmJuoTIZaf+F74ZYbL5MXxfVXJD Y8oLh82hb8JctJAQqD6KKpKnJHDy4VRIWoV9Oj692FMLCvO5ZqRSrLTc9CDWTlq7KxjY wb1FRjDPludhMYuYWqc7k8+okbPT9Vx+MLSKI+i12EOtp/72gSheC/XxyE0z4JwUtkr4 qP2g== X-Forwarded-Encrypted: i=1; AHgh+RrbDhnWs6m5aGfNhMnC0mIoIXYL3hlhiph5mL6Km6r5QKc3lzL8fNDORNz14z9cpPAtxJDGH+afAZ751x4=@vger.kernel.org X-Gm-Message-State: AOJu0YxGbD/iM1dHki0iuQJAv6VJSgaxueJKBcOQXLL5geAfTKKP6ww0 7Pe5ymiWkvkpQXyIY0nbc8TxmIjSwnhlFy0BClsCRHZtlXzKtkJI9R7Z2H2yYQ== X-Gm-Gg: AR+sD128qeTtjYT6WRbFlrbAz/0Fu3ligeojL7K1YMuSMiNVaT+NOyeH/i5rlO82INQ 3g3y2r9G7x+gXnbNKorzt36dx4zrJs1Og4E54/ceuzIYUg8LZNNs0SH/+acgQpnLpm4C321Wx4/ lJ34dXIwFz8bHS/+5MYkZ4HE20Lo8h3IVIIPjPyt6xhU3wa/e9mJMmBQxNVcNrXPtfDSfmi/JK2 WkXBnQH8hzEKQ3DgwmckCh1oOA4DKl0rXA0vuzMi1GYgnL8r0XXvvkiDJU6fgdZMR7z8ZAKQ+vP Tw7i1qkpIIKXH4HgSgkD05pydE9KNDzMgYzBBrwZ0vY0gYyfGvxnBaJdqmRMP/dAd4zFIicy4Wg nzGw02rWkSHbfuzJ3+jHzpNzgfMag4yjADRjIWDA4O/Kix8e7rjJuju55WxBHzEvnVoHQEOJfVe fxL+xH X-Received: by 2002:a05:6a00:1bc6:b0:84a:6726:62f9 with SMTP id d2e1a72fcca58-84e8d0ec187mr868586b3a.9.1785184887318; Mon, 27 Jul 2026 13:41:27 -0700 (PDT) Received: from cryptic.lan ([2001:569:7fab:e200:3789:64f9:8855:c0af]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-84e533d1351sm3583338b3a.39.2026.07.27.13.41.26 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 27 Jul 2026 13:41:26 -0700 (PDT) From: Allen Pais To: Vinod Koul , Frank Li Cc: Allen Pais , dmaengine@vger.kernel.org, linux-kernel@vger.kernel.org, Arnd Bergmann , Kees Cook , Viresh Kumar , Andy Shevchenko Subject: [PATCH v2 61/64] dmaengine: dw: defer callbacks via channel BH Date: Mon, 27 Jul 2026 13:39:23 -0700 Message-ID: <4399a85a5ffe0dcbace462217f15c682c5f8d28a.1785183549.git.allen.lkml@gmail.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: References: <20260108080332.2341725-1-allen.lkml@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Queue completed descriptors on the channel and invoke their callbacks from per-channel BH work instead of the controller tasklet. Add a synchronize hook so termination can wait for deferred callbacks before their descriptor storage is released. Signed-off-by: Allen Pais --- drivers/dma/dw/core.c | 42 ++++++++++++++++++++++++++++++++++-------- drivers/dma/dw/regs.h | 1 + 2 files changed, 35 insertions(+), 8 deletions(-) diff --git a/drivers/dma/dw/core.c b/drivers/dma/dw/core.c index dd75f97a33b3..e780116a944b 100644 --- a/drivers/dma/dw/core.c +++ b/drivers/dma/dw/core.c @@ -236,25 +236,24 @@ dwc_descriptor_complete(struct dw_dma_chan *dwc, struct dw_desc *desc, struct dma_async_tx_descriptor *txd = &desc->txd; struct dw_desc *child; unsigned long flags; - struct dmaengine_desc_callback cb; dev_vdbg(chan2dev(&dwc->chan), "descriptor %u complete\n", txd->cookie); spin_lock_irqsave(&dwc->lock, flags); dma_cookie_complete(txd); - if (callback_required) - dmaengine_desc_get_callback(txd, &cb); - else - memset(&cb, 0, sizeof(cb)); + list_del_init(&desc->desc_node); /* async_tx_ack */ list_for_each_entry(child, &desc->tx_list, desc_node) async_tx_ack(&child->txd); async_tx_ack(&desc->txd); - dwc_desc_put(dwc, desc); + if (callback_required) { + list_add_tail(&desc->desc_node, &dwc->completed_list); + dma_chan_schedule_bh(&dwc->chan); + } else { + dwc_desc_put(dwc, desc); + } spin_unlock_irqrestore(&dwc->lock, flags); - - dmaengine_desc_callback_invoke(&cb, NULL); } static void dwc_complete_all(struct dw_dma *dw, struct dw_dma_chan *dwc) @@ -285,6 +284,24 @@ static void dwc_complete_all(struct dw_dma *dw, struct dw_dma_chan *dwc) dwc_descriptor_complete(dwc, desc, true); } +static void dwc_chan_bh(struct dma_chan *chan) +{ + struct dw_dma_chan *dwc = to_dw_dma_chan(chan); + struct dw_desc *desc, *_desc; + struct dmaengine_desc_callback cb; + LIST_HEAD(list); + + spin_lock_irq(&dwc->lock); + list_splice_init(&dwc->completed_list, &list); + spin_unlock_irq(&dwc->lock); + + list_for_each_entry_safe(desc, _desc, &list, desc_node) { + dmaengine_desc_get_callback(&desc->txd, &cb); + dmaengine_desc_callback_invoke(&cb, NULL); + dwc_desc_put(dwc, desc); + } +} + /* Returns how many bytes were already received from source */ static inline u32 dwc_get_sent(struct dw_dma_chan *dwc) { @@ -965,6 +982,11 @@ static int dwc_terminate_all(struct dma_chan *chan) return 0; } +static void dwc_synchronize(struct dma_chan *chan) +{ + dma_chan_kill_bh(chan); +} + static struct dw_desc *dwc_find_desc(struct dw_dma_chan *dwc, dma_cookie_t c) { struct dw_desc *desc; @@ -1256,6 +1278,8 @@ int do_dma_probe(struct dw_dma_chip *chip) INIT_LIST_HEAD(&dwc->active_list); INIT_LIST_HEAD(&dwc->queue); + INIT_LIST_HEAD(&dwc->completed_list); + dma_chan_init_bh(&dwc->chan, dwc_chan_bh); channel_clear_bit(dw, CH_EN, dwc->mask); @@ -1321,6 +1345,7 @@ int do_dma_probe(struct dw_dma_chip *chip) dw->dma.device_pause = dwc_pause; dw->dma.device_resume = dwc_resume; dw->dma.device_terminate_all = dwc_terminate_all; + dw->dma.device_synchronize = dwc_synchronize; dw->dma.device_tx_status = dwc_tx_status; dw->dma.device_issue_pending = dwc_issue_pending; @@ -1374,6 +1399,7 @@ int do_dma_remove(struct dw_dma_chip *chip) list_for_each_entry_safe(dwc, _dwc, &dw->dma.channels, chan.device_node) { + dma_chan_kill_bh(&dwc->chan); list_del(&dwc->chan.device_node); channel_clear_bit(dw, CH_EN, dwc->mask); } diff --git a/drivers/dma/dw/regs.h b/drivers/dma/dw/regs.h index 5969d9cc8d7a..c9e32bd6110a 100644 --- a/drivers/dma/dw/regs.h +++ b/drivers/dma/dw/regs.h @@ -279,6 +279,7 @@ struct dw_dma_chan { unsigned long flags; struct list_head active_list; struct list_head queue; + struct list_head completed_list; unsigned int descs_allocated; -- 2.43.0