* [PATCH v8 2/2] dmaengine: atcdmac300: Add driver for Andes ATCDMAC300 DMA controller
2026-10-01 1:45 [PATCH v8 0/2] dmaengine: atcdmac300: Add Andes ATCDMAC300 DMA driver CL Wang
2026-10-01 1:45 ` [PATCH v8 1/2] dt-bindings: dmaengine: Add support for ATCDMAC300 DMA engine CL Wang
@ 2026-10-01 1:45 ` CL Wang
2026-10-01 20:18 ` Frank Li
1 sibling, 1 reply; 6+ messages in thread
From: CL Wang @ 2026-10-01 1:45 UTC (permalink / raw)
To: Vinod Koul, Rob Herring, Krzysztof Kozlowski, Conor Dooley
Cc: Frank Li, Kees Cook, Gustavo A . R . Silva, tim609, cl634,
dmaengine, devicetree, linux-hardening, linux-kernel
This patch adds support for the Andes ATCDMAC300 DMA controller.
The ATCDMAC300 is a memory-to-memory and peripheral DMA controller
that provides scatter-gather, cyclic, and slave transfer capabilities.
Signed-off-by: CL Wang <cl634@andestech.com>
---
Changes for v8:
- Fold in fourteen further correctness fixes; no interface or binding
change. Five of them answer findings reported on v7; the other
three findings from that posting are explained in the reply to it
rather than changed. The remaining nine come from a review of the
whole driver.
Descriptor lifetime:
- Track cyclic mode on the descriptor rather than on the channel.
A cyclic and a non-cyclic transfer on one channel could leave
the flag set for the wrong descriptor and dereference a NULL
callback.
- Drop the num_sg field, which was written but never read.
Suspend and resume:
- Restore stop_mask when suspend times out, instead of leaving
every channel stopped for good.
- Start transfers that were queued while suspended.
Driver model:
- Do not offer the controller for unbinding. A client holding a
channel across an unbind would run against devres memory that
remove() has already released.
Register encoding:
- Reject a DMA request line the request-select field cannot hold,
rather than truncating it silently.
- Keep the burst field out of its reserved range, which the
controller reports as an error exception.
- Refuse a transfer width the width field cannot encode.
Slave configuration:
- Clear the stored configuration when a channel is allocated, so a
client cannot inherit the previous one's device address.
- Accept a configuration that describes only the direction it
uses. Requiring all four of the width and burst fields to be
powers of two rejected any client that fills in one direction,
and the transfer then divided by a zero device width.
- Reject a length that is not a whole number of device-width
units, which either transfers fewer bytes than asked for or
programs a zero transfer size.
Probe:
- Use struct_size() for the channel array, which is already
declared with __counted_by().
- Check the configuration register read, and refuse a channel
count of zero rather than registering a controller with no
channels.
Cleanups:
- Return a bool from the channel-enabled helper rather than a
value that is negative on a failed register read; pass a
register value rather than a bool to regmap_update_bits(); use
dma_cookie_status() directly.
Changes for v7:
- Fold in twelve correctness fixes found by review and confirmed on an
FPGA; no interface or binding change. Interrupt path:
- Acknowledge only the status bits that were actually read, and key
the acknowledge on channel ownership rather than on in-flight
state the handler itself clears. The old mask could drop a
completion that arrived while it was being cleared, leaving the
source asserted until the core disabled the line ("nobody cared").
- Use raw spinlocks for the state shared with the hard IRQ handler.
Termination and descriptor lifetime:
- Do not run the callback twice, or recycle a descriptor, when a
transfer is terminated while its completion is in flight.
- Initialise desc_node when a descriptor is allocated.
- Wait for the channel to stop without holding the channel lock;
the poll could spin 100 ms with interrupts disabled.
Suspend and resume:
- Move the PM callbacks to the late phase so clients terminate
their transfers first, and name the channels that fail to idle.
- Drain in-flight completions across suspend and resume.
- Do not restart an idle cyclic channel on resume.
Probe:
- Register the DMA device only after the interrupt and cache
coherency setup, closing a window in which a client could obtain
a channel through dma_request_channel() on an unready controller.
- Do not read-modify-write write-only registers.
- Name the device in the probe-time reset error.
Changes for v6:
- Rebase onto dmaengine/next; no functional changes from v5
- Fold the MAINTAINERS entry into this patch instead of carrying it
as a separate patch
Changes for v5:
- Update copyright year to 2026
- Remove redundant headers (init.h, iopoll.h, mod_devicetable.h)
- Move atcdmac_init_iocp() before of_dma_controller_register() in probe
- Change builtin_platform_driver() to module_platform_driver()
- Implement .remove callback to support safe module unloading
- Update Kconfig entry from bool to tristate
- Add MODULE_AUTHOR, MODULE_DESCRIPTION, MODULE_LICENSE macros
Changes for v4:
- No changes from v3
Changes for v3:
- Remove "andestech,atcdmac300" from of_device_id
- Replace deprecated tasklet with threaded IRQ using
devm_request_threaded_irq() and IRQF_ONESHOT to handle bottom-half
processing.
- Update locking mechanism from spin_lock_bh() to spin_lock_irqsave()
- Minor cleanups and correctness fixes
- Initialize descriptor pointers (first = NULL) explicitly
- Add missing headers (err.h, iopoll.h, log2.h, sprintf.h)
- Remove unused code paths related to tasklets
- Use builtin_platform_driver() instead of module_platform_driver()
- Remove "select DMATEST" from Kconfig
---
MAINTAINERS | 6 +
drivers/dma/Kconfig | 11 +
drivers/dma/Makefile | 1 +
drivers/dma/atcdmac300.c | 1759 ++++++++++++++++++++++++++++++++++++++
drivers/dma/atcdmac300.h | 298 +++++++
5 files changed, 2075 insertions(+)
create mode 100644 drivers/dma/atcdmac300.c
create mode 100644 drivers/dma/atcdmac300.h
diff --git a/MAINTAINERS b/MAINTAINERS
index c31b81eb42d7..2c48bb3053b3 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -1974,6 +1974,12 @@ S: Supported
F: drivers/clk/analogbits/*
F: include/linux/clk/analogbits*
+ANDES ATCDMAC300 DMA DRIVER
+M: CL Wang <cl634@andestech.com>
+S: Supported
+F: Documentation/devicetree/bindings/dma/andestech,ae350-dma.yaml
+F: drivers/dma/atcdmac300*
+
ANDES ATCSPI200 SPI DRIVER
M: CL Wang <cl634@andestech.com>
S: Supported
diff --git a/drivers/dma/Kconfig b/drivers/dma/Kconfig
index ae6a682c9f76..dabd2c8cc94f 100644
--- a/drivers/dma/Kconfig
+++ b/drivers/dma/Kconfig
@@ -100,6 +100,17 @@ config ARM_DMA350
help
Enable support for the Arm DMA-350 controller.
+config ATCDMAC300
+ tristate "Andes ATCDMAC300 DMA support"
+ depends on ARCH_ANDES
+ depends on OF
+ select DMA_ENGINE
+ help
+ Enable support for the Andes ATCDMAC300 DMA controller.
+ Select Y or M if your platform includes an ATCDMAC300 device that
+ requires DMA engine support. This driver supports DMA_SLAVE,
+ DMA_MEMCPY, and DMA_CYCLIC transfer modes.
+
config AT_HDMAC
tristate "Atmel AHB DMA support"
depends on ARCH_AT91 || COMPILE_TEST
diff --git a/drivers/dma/Makefile b/drivers/dma/Makefile
index 14aa086629d5..c8fffb31efc4 100644
--- a/drivers/dma/Makefile
+++ b/drivers/dma/Makefile
@@ -18,6 +18,7 @@ obj-$(CONFIG_AMBA_PL08X) += amba-pl08x.o
obj-$(CONFIG_AMCC_PPC440SPE_ADMA) += ppc4xx/
obj-$(CONFIG_APPLE_ADMAC) += apple-admac.o
obj-$(CONFIG_ARM_DMA350) += arm-dma350.o
+obj-$(CONFIG_ATCDMAC300) += atcdmac300.o
obj-$(CONFIG_AT_HDMAC) += at_hdmac.o
obj-$(CONFIG_AT_XDMAC) += at_xdmac.o
obj-$(CONFIG_AXI_DMAC) += dma-axi-dmac.o
diff --git a/drivers/dma/atcdmac300.c b/drivers/dma/atcdmac300.c
new file mode 100644
index 000000000000..b67675647b36
--- /dev/null
+++ b/drivers/dma/atcdmac300.c
@@ -0,0 +1,1759 @@
+// SPDX-License-Identifier: GPL-2.0-or-later
+/*
+ * Andes ATCDMAC300 controller driver
+ *
+ * Copyright (C) 2026 Andes Technology Corporation
+ */
+#include <linux/align.h>
+#include <linux/bitfield.h>
+#include <linux/bits.h>
+#include <linux/dmaengine.h>
+#include <linux/dmapool.h>
+#include <linux/dma-mapping.h>
+#include <linux/delay.h>
+#include <linux/err.h>
+#include <linux/interrupt.h>
+#include <linux/log2.h>
+#include <linux/module.h>
+#include <linux/of.h>
+#include <linux/of_dma.h>
+#include <linux/of_platform.h>
+#include <linux/platform_device.h>
+#include <linux/pm.h>
+#include <linux/slab.h>
+#include <linux/sprintf.h>
+#include <linux/regmap.h>
+#include "dmaengine.h"
+#include "atcdmac300.h"
+
+static bool atcdmac_is_chan_enabled(struct atcdmac_chan *dmac_chan)
+{
+ struct atcdmac_dmac *dmac =
+ atcdmac_dev_to_dmac(dmac_chan->dma_chan.device);
+
+ /*
+ * regmap_test_bits() reports a failed read as a negative errno,
+ * which every caller here would otherwise read as "enabled".
+ */
+ return regmap_test_bits(dmac->regmap,
+ REG_CH_EN,
+ BIT(dmac_chan->chan_id)) > 0;
+}
+
+static void atcdmac_enable_chan(struct atcdmac_chan *dmac_chan, bool enable)
+{
+ regmap_update_bits(dmac_chan->regmap, REG_CH_CTL_OFF, CHEN,
+ enable ? CHEN : 0);
+}
+
+static void atcdmac_abort_chan(struct atcdmac_chan *dmac_chan)
+{
+ /*
+ * ChAbort is write-only: write 1 to the channel's bit to abort it.
+ * Do not read-modify-write, the read value is undefined and writing
+ * it back could abort a channel owned by someone else.
+ */
+ regmap_write(dmac_chan->dma_dev->regmap,
+ REG_CH_ABT,
+ BIT(dmac_chan->chan_id));
+}
+
+static dma_cookie_t atcdmac_tx_submit(struct dma_async_tx_descriptor *tx)
+{
+ struct atcdmac_chan *dmac_chan = atcdmac_chan_to_dmac_chan(tx->chan);
+ struct atcdmac_desc *desc = atcdmac_txd_to_dma_desc(tx);
+ dma_cookie_t cookie;
+ unsigned long flags;
+
+ spin_lock_irqsave(&dmac_chan->lock, flags);
+ cookie = dma_cookie_assign(tx);
+ list_add_tail(&desc->desc_node, &dmac_chan->queue_list);
+ spin_unlock_irqrestore(&dmac_chan->lock, flags);
+
+ return cookie;
+}
+
+static struct atcdmac_desc *
+atcdmac_get_active_head(struct atcdmac_chan *dmac_chan)
+{
+ return list_first_entry(&dmac_chan->active_list,
+ struct atcdmac_desc,
+ desc_node);
+}
+
+static struct atcdmac_desc *atcdmac_alloc_desc(struct dma_chan *chan,
+ gfp_t gfp_flags)
+{
+ struct atcdmac_dmac *dmac = atcdmac_dev_to_dmac(chan->device);
+ struct atcdmac_desc *desc;
+ dma_addr_t phys;
+
+ desc = dma_pool_zalloc(dmac->dma_desc_pool, gfp_flags, &phys);
+ if (desc) {
+ INIT_LIST_HEAD(&desc->desc_node);
+ INIT_LIST_HEAD(&desc->tx_list);
+ dma_async_tx_descriptor_init(&desc->txd, chan);
+ desc->txd.flags = DMA_CTRL_ACK;
+ desc->txd.tx_submit = atcdmac_tx_submit;
+ desc->txd.phys = phys;
+ }
+
+ return desc;
+}
+
+static struct atcdmac_desc *atcdmac_get_desc(struct atcdmac_chan *dmac_chan)
+{
+ struct atcdmac_desc *ret = NULL;
+ struct atcdmac_desc *desc_next;
+ struct atcdmac_desc *desc;
+ unsigned long flags;
+
+ spin_lock_irqsave(&dmac_chan->lock, flags);
+ list_for_each_entry_safe(desc, desc_next,
+ &dmac_chan->free_list,
+ desc_node) {
+ if (async_tx_test_ack(&desc->txd)) {
+ list_del_init(&desc->desc_node);
+ ret = desc;
+ break;
+ }
+ }
+ spin_unlock_irqrestore(&dmac_chan->lock, flags);
+
+ if (!ret) {
+ ret = atcdmac_alloc_desc(&dmac_chan->dma_chan, GFP_ATOMIC);
+ if (ret) {
+ spin_lock_irqsave(&dmac_chan->lock, flags);
+ dmac_chan->descs_allocated++;
+ spin_unlock_irqrestore(&dmac_chan->lock, flags);
+ } else {
+ dev_warn(atcdmac_chan_to_dev(&dmac_chan->dma_chan),
+ "not enough descriptors available\n");
+ }
+ }
+
+ return ret;
+}
+
+/**
+ * atcdmac_put_desc_nolock - move a descriptor to the free list
+ * @dmac_chan: DMA channel we work on
+ * @desc: Head of the descriptor chain to be added to the free list
+ *
+ * This function does not use a lock to protect any linked lists in
+ * 'struct atcdmac_chan', so please remember to add a proper lock when
+ * calling the function.
+ */
+static void atcdmac_put_desc_nolock(struct atcdmac_chan *dmac_chan,
+ struct atcdmac_desc *desc)
+{
+ struct atcdmac_desc *child, *tmp;
+
+ if (desc) {
+ list_for_each_entry_safe(child,
+ tmp,
+ &desc->tx_list,
+ desc_node) {
+ list_del_init(&child->desc_node);
+ child->at = NULL;
+ child->cyclic = false;
+ INIT_LIST_HEAD(&child->tx_list);
+ list_add_tail(&child->desc_node,
+ &dmac_chan->free_list);
+ }
+
+ list_del_init(&desc->desc_node);
+ desc->at = NULL;
+ desc->cyclic = false;
+ INIT_LIST_HEAD(&desc->tx_list);
+ list_add_tail(&desc->desc_node, &dmac_chan->free_list);
+ }
+}
+
+static void atcdmac_put_desc(struct atcdmac_chan *dmac_chan,
+ struct atcdmac_desc *desc)
+{
+ unsigned long flags;
+
+ if (!desc) {
+ dev_err(atcdmac_chan_to_dev(&dmac_chan->dma_chan),
+ "A NULL descriptor was found.\n");
+ return;
+ }
+
+ spin_lock_irqsave(&dmac_chan->lock, flags);
+ atcdmac_put_desc_nolock(dmac_chan, desc);
+ spin_unlock_irqrestore(&dmac_chan->lock, flags);
+}
+
+static void atcdmac_show_desc(struct atcdmac_chan *dmac_chan,
+ struct atcdmac_desc *desc)
+{
+ struct device *dev = atcdmac_chan_to_dev(&dmac_chan->dma_chan);
+
+ dev_dbg(dev, "Dump desc info of chan: %u\n", dmac_chan->chan_id);
+ dev_dbg(dev, "chan ctrl: 0x%08x\n", desc->regs.ctrl);
+ dev_dbg(dev, "trans size: 0x%08x\n", desc->regs.trans_size);
+ dev_dbg(dev, "src addr: hi:0x%08x lo:0x%08x\n",
+ desc->regs.src_addr_hi,
+ desc->regs.src_addr_lo);
+ dev_dbg(dev, "dst addr: hi:0x%08x lo:0x%08x\n",
+ desc->regs.dst_addr_hi,
+ desc->regs.dst_addr_lo);
+ dev_dbg(dev, "link addr: hi:0x%08x lo:0x%08x\n",
+ desc->regs.ll_ptr_hi,
+ desc->regs.ll_ptr_lo);
+}
+
+/**
+ * atcdmac_chain_desc - Chain a DMA descriptor into a linked-list
+ * @first: Pointer to the first descriptor in the chain
+ * @prev: Pointer to the previous descriptor in the chain
+ * @desc: The descriptor to be added to the chain
+ * @cyclic: Indicates if the transfer operates in cyclic mode
+ *
+ * This function appends a DMA descriptor (desc) to a linked-list of
+ * descriptors. If this is the first descriptor being added, it initializes
+ * the list and sets *first to point to desc. Otherwise, it links the
+ * new descriptor to the end of the list managed by *first.
+ *
+ * For non-cyclic descriptors, it updates the hardware linked list pointers
+ * (ll_ptr_lo and ll_ptr_hi) of the previous descriptor (*prev) to point to
+ * the physical address of the new descriptor.
+ *
+ * Finally, it adds the new descriptor to the list and updates *prev to
+ * point to the current descriptor (desc).
+ */
+static void atcdmac_chain_desc(struct atcdmac_desc **first,
+ struct atcdmac_desc **prev,
+ struct atcdmac_desc *desc,
+ bool cyclic)
+{
+ if (!(*first)) {
+ *first = desc;
+ desc->at = &desc->tx_list;
+ } else {
+ if (!cyclic) {
+ (*prev)->regs.ll_ptr_lo =
+ lower_32_bits(desc->txd.phys);
+ (*prev)->regs.ll_ptr_hi =
+ upper_32_bits(desc->txd.phys);
+ }
+ list_add_tail(&desc->desc_node, &(*first)->tx_list);
+ }
+ *prev = desc;
+
+ desc->regs.ll_ptr_hi = 0;
+ desc->regs.ll_ptr_lo = 0;
+}
+
+/**
+ * atcdmac_start_transfer - Start the DMA engine with the provided descriptor
+ * @dmac_chan: The DMA channel to be started
+ * @first_desc: The first descriptor in the list to begin the transfer
+ *
+ * This function configures the DMA engine by programming the hardware
+ * registers with the information from the provided descriptor (first_desc).
+ * It then starts the DMA transfer for the specified channel (dmac_chan).
+ *
+ * The first_desc contains the initial configuration for the transfer,
+ * including source and destination addresses, transfer size, and any linked
+ * list pointers for subsequent descriptors in the chain.
+ */
+static void atcdmac_start_transfer(struct atcdmac_chan *dmac_chan,
+ struct atcdmac_desc *first_desc)
+{
+ struct atcdmac_dmac *dmac = dmac_chan->dma_dev;
+ struct regmap *reg = dmac_chan->regmap;
+ unsigned long flags;
+
+ raw_spin_lock_irqsave(&dmac->lock, flags);
+ dmac->owned_chan |= BIT(dmac_chan->chan_id);
+ raw_spin_unlock_irqrestore(&dmac->lock, flags);
+
+ regmap_write(reg, REG_CH_CTL_OFF, first_desc->regs.ctrl);
+ regmap_write(reg, REG_CH_SIZE_OFF, first_desc->regs.trans_size);
+ regmap_write(reg, REG_CH_SRC_LOW_OFF, first_desc->regs.src_addr_lo);
+ regmap_write(reg, REG_CH_DST_LOW_OFF, first_desc->regs.dst_addr_lo);
+ regmap_write(reg, REG_CH_LLP_LOW_OFF, first_desc->regs.ll_ptr_lo);
+ regmap_write(reg, REG_CH_SRC_HIGH_OFF, first_desc->regs.src_addr_hi);
+ regmap_write(reg, REG_CH_DST_HIGH_OFF, first_desc->regs.dst_addr_hi);
+ regmap_write(reg, REG_CH_LLP_HIGH_OFF, first_desc->regs.ll_ptr_hi);
+ atcdmac_enable_chan(dmac_chan, 1);
+}
+
+/**
+ * atcdmac_start_next_trans - Retrieve and initiate the next DMA transfer
+ * @dmac_chan: Pointer to the DMA channel structure
+ *
+ * Whether a transfer is cyclic is a property of the descriptor, not of the
+ * channel, so the head of active_list decides what happens here.
+ *
+ * If it is cyclic, the function advances the tx_list cursor and restarts the
+ * next period, which keeps the transfer running. If it is not, that
+ * descriptor is simply started.
+ *
+ * If active_list is empty, nothing is running: descriptors from queue_list
+ * (if any) are moved over and started. If both lists are empty, the channel
+ * is marked as unused.
+ */
+static void atcdmac_start_next_trans(struct atcdmac_chan *dmac_chan)
+{
+ struct atcdmac_desc *next_tx = NULL;
+ struct atcdmac_desc *dma_desc;
+
+ dma_desc = list_first_entry_or_null(&dmac_chan->active_list,
+ struct atcdmac_desc, desc_node);
+
+ if (dma_desc && dma_desc->cyclic) {
+ /* Get the next DMA descriptor from tx_list. */
+ dma_desc->at = dma_desc->at->next;
+ if ((uintptr_t)dma_desc->at == (uintptr_t)&dma_desc->tx_list)
+ next_tx = list_entry(dma_desc->at,
+ struct atcdmac_desc,
+ tx_list);
+ else
+ next_tx = list_entry(dma_desc->at,
+ struct atcdmac_desc,
+ desc_node);
+ } else if (dma_desc) {
+ next_tx = dma_desc;
+ } else if (!list_empty(&dmac_chan->queue_list)) {
+ list_splice_init(&dmac_chan->queue_list,
+ &dmac_chan->active_list);
+ next_tx = atcdmac_get_active_head(dmac_chan);
+ }
+
+ if (next_tx) {
+ dmac_chan->chan_used = 1;
+ atcdmac_start_transfer(dmac_chan, next_tx);
+ } else {
+ dmac_chan->chan_used = 0;
+ }
+}
+
+/*
+ * Retire a descriptor without telling the client about it. Used on the
+ * termination path, where the dmaengine API forbids running the completion
+ * callback.
+ */
+static void atcdmac_retire_desc(struct atcdmac_desc *desc)
+{
+ struct dma_async_tx_descriptor *txd = &desc->txd;
+
+ dma_cookie_complete(txd);
+ dma_descriptor_unmap(txd);
+}
+
+static void atcdmac_run_tx_complete_actions(struct atcdmac_desc *desc,
+ enum dmaengine_tx_result result)
+{
+ struct dma_async_tx_descriptor *txd = &desc->txd;
+ struct dmaengine_result res;
+
+ res.result = result;
+ atcdmac_retire_desc(desc);
+ dmaengine_desc_get_callback_invoke(txd, &res);
+ dma_run_dependencies(txd);
+}
+
+/**
+ * atcdmac_advance_work - Process the completed transaction and move to the
+ * next descriptor
+ * @dmac_chan: DMA channel where the transaction has completed
+ *
+ * This function is responsible for performing necessary operations after
+ * a DMA transaction completes successfully. It retrieves the next descriptor
+ * from the active list, initiates its transfer, and communicates the result
+ * of the completed transaction to the DMA engine framework.
+ */
+static void atcdmac_advance_work(struct atcdmac_chan *dmac_chan)
+{
+ struct atcdmac_dmac *dmac =
+ atcdmac_dev_to_dmac(dmac_chan->dma_chan.device);
+ struct atcdmac_desc *desc_next, *dma_desc, *desc;
+ struct dmaengine_desc_callback cb;
+ struct dmaengine_result res;
+ LIST_HEAD(completed);
+ unsigned long flags;
+ unsigned short stop;
+
+ spin_lock_irqsave(&dmac_chan->lock, flags);
+ if (list_empty(&dmac_chan->active_list)) {
+ spin_unlock_irqrestore(&dmac_chan->lock, flags);
+ return;
+ }
+
+ dma_desc = atcdmac_get_active_head(dmac_chan);
+ stop = READ_ONCE(dmac->stop_mask) & BIT(dmac_chan->chan_id);
+ if (dma_desc->cyclic) {
+ if (!stop)
+ atcdmac_start_next_trans(dmac_chan);
+
+ /*
+ * The descriptor stays on active_list for the whole cyclic
+ * transfer, so atcdmac_terminate_all() may recycle it as soon
+ * as the lock is dropped. Take a copy of the callback while
+ * it is still safe to look at.
+ */
+ dmaengine_desc_get_callback(&dma_desc->txd, &cb);
+ spin_unlock_irqrestore(&dmac_chan->lock, flags);
+ res.result = DMA_TRANS_NOERROR;
+ dmaengine_desc_callback_invoke(&cb, &res);
+ } else {
+ if (list_is_singular(&dmac_chan->active_list)) {
+ list_splice_init(&dmac_chan->active_list, &completed);
+ list_splice_init(&dmac_chan->queue_list,
+ &dmac_chan->active_list);
+ } else {
+ list_move_tail(&dma_desc->desc_node, &completed);
+ }
+
+ if (!stop)
+ atcdmac_start_next_trans(dmac_chan);
+
+ spin_unlock_irqrestore(&dmac_chan->lock, flags);
+
+ list_for_each_entry_safe(desc,
+ desc_next,
+ &completed,
+ desc_node) {
+ atcdmac_run_tx_complete_actions(desc,
+ DMA_TRANS_NOERROR);
+ atcdmac_put_desc(dmac_chan, desc);
+ }
+ }
+}
+
+/**
+ * atcdmac_handle_error - Handle errors reported by the DMA controller
+ * @dmac_chan: DMA channel where the error occurred
+ *
+ * This function is invoked when the DMA controller detects an error during a
+ * transaction. This function ensures that the DMA channel can recover from
+ * errors while reporting issues to aid in debugging. It prevents the DMA
+ * controller from operating on potentially invalid memory regions and
+ * attempts to maintain system stability.
+ */
+static void atcdmac_handle_error(struct atcdmac_chan *dmac_chan)
+{
+ struct device *dev = atcdmac_chan_to_dev(&dmac_chan->dma_chan);
+ struct atcdmac_dmac *dmac =
+ atcdmac_dev_to_dmac(dmac_chan->dma_chan.device);
+ struct atcdmac_desc *bad_desc;
+ unsigned long flags;
+ unsigned short stop;
+
+ spin_lock_irqsave(&dmac_chan->lock, flags);
+
+ /*
+ * If the active list is empty, the descriptor has already been
+ * handled by another function (e.g., atcdmac_terminate_all()).
+ * Therefore, no further action is needed.
+ */
+ if (!list_empty(&dmac_chan->active_list)) {
+ /*
+ * Identify the problematic descriptor at the head of the
+ * active list and remove it from the list for further
+ * processing.
+ */
+ bad_desc = atcdmac_get_active_head(dmac_chan);
+ list_del_init(&bad_desc->desc_node);
+
+ /*
+ * Transfer any pending descriptors from the queue list to the
+ * active list, allowing them to be processed in subsequent
+ * operations.
+ */
+ list_splice_init(&dmac_chan->queue_list,
+ dmac_chan->active_list.prev);
+
+ stop = READ_ONCE(dmac->stop_mask) & BIT(dmac_chan->chan_id);
+ if (!list_empty(&dmac_chan->active_list) && stop == 0)
+ atcdmac_start_transfer(dmac_chan, atcdmac_get_active_head(dmac_chan));
+ else
+ dmac_chan->chan_used = 0;
+
+ spin_unlock_irqrestore(&dmac_chan->lock, flags);
+
+ /*
+ * Show the detailed information of the bad descriptor and
+ * return "DMA_TRANS_ABORTED" to the DMA engine framework.
+ */
+ dev_err(dev, "DMA transaction failed, possible DMA desc error\n");
+ atcdmac_show_desc(dmac_chan, bad_desc);
+ atcdmac_run_tx_complete_actions(bad_desc, DMA_TRANS_ABORTED);
+
+ atcdmac_put_desc(dmac_chan, bad_desc);
+
+ return;
+ }
+
+ spin_unlock_irqrestore(&dmac_chan->lock, flags);
+}
+
+static irqreturn_t atcdmac_irq_thread(int irq, void *dev_id)
+{
+ struct atcdmac_dmac *dmac = dev_id;
+ struct atcdmac_chan *dmac_chan;
+ int i;
+ bool handled = false;
+
+ for (i = 0; i < dmac->num_ch; i++) {
+ dmac_chan = &dmac->chan[i];
+
+ if (test_and_clear_bit(ATCDMAC_STA_TC, &dmac_chan->status)) {
+ atcdmac_advance_work(dmac_chan);
+ handled = true;
+ }
+
+ if (test_and_clear_bit(ATCDMAC_STA_ERR, &dmac_chan->status)) {
+ atcdmac_handle_error(dmac_chan);
+ handled = true;
+ }
+
+ /*
+ * ATCDMAC_STA_ABORT only occurs when the DMA channel is
+ * terminated or freed, and all descriptors and callbacks
+ * have already been processed. Therefore, no additional
+ * handling is required.
+ */
+ if (test_and_clear_bit(ATCDMAC_STA_ABORT, &dmac_chan->status))
+ handled = true;
+ }
+
+ return handled ? IRQ_HANDLED : IRQ_NONE;
+}
+
+static irqreturn_t atcdmac_interrupt(int irq, void *dev_id)
+{
+ struct atcdmac_dmac *dmac = dev_id;
+ struct atcdmac_chan *dmac_chan;
+ unsigned int status;
+ unsigned int int_ch;
+ int ret = IRQ_NONE;
+ int i;
+
+ regmap_read(dmac->regmap, REG_INT_STA, &status);
+ int_ch = READ_ONCE(dmac->owned_chan) & DMA_INT_ALL(status);
+
+ while (int_ch) {
+ /*
+ * Acknowledge exactly the events this pass is about to
+ * dispatch. Anything that asserts after the status was read
+ * stays pending and is picked up by the re-read below.
+ */
+ regmap_write(dmac->regmap, REG_INT_STA,
+ status & DMA_INT_CLR(int_ch));
+
+ for (i = 0; i < dmac->num_ch; i++) {
+ if (int_ch & BIT(i)) {
+ int_ch &= ~BIT(i);
+ dmac_chan = &dmac->chan[i];
+
+ if (status & DMA_TC(i))
+ set_bit(ATCDMAC_STA_TC,
+ &dmac_chan->status);
+
+ if (status & DMA_ERR(i))
+ set_bit(ATCDMAC_STA_ERR,
+ &dmac_chan->status);
+
+ if (status & DMA_ABT(i))
+ set_bit(ATCDMAC_STA_ABORT,
+ &dmac_chan->status);
+
+ ret = IRQ_WAKE_THREAD;
+ }
+ if (!int_ch)
+ break;
+ }
+
+ regmap_read(dmac->regmap, REG_INT_STA, &status);
+ int_ch = READ_ONCE(dmac->owned_chan) & DMA_INT_ALL(status);
+ }
+
+ return ret;
+}
+
+/**
+ * atcdmac_issue_pending - Trigger the execution of queued DMA transactions
+ * @chan: DMA channel on which to issue pending transactions
+ *
+ * This function checks if the DMA channel is currently idle and if there are
+ * descriptors waiting in the queue_list. If the channel is idle and the
+ * queue_list is not empty, it moves the first queued descriptor to the active
+ * list and starts the DMA transfer by calling atcdmac_start_transfer().
+ */
+static void atcdmac_issue_pending(struct dma_chan *chan)
+{
+ struct atcdmac_chan *dmac_chan = atcdmac_chan_to_dmac_chan(chan);
+ struct atcdmac_dmac *dmac =
+ atcdmac_dev_to_dmac(dmac_chan->dma_chan.device);
+ unsigned long flags;
+ unsigned short stop;
+
+ spin_lock_irqsave(&dmac_chan->lock, flags);
+ stop = READ_ONCE(dmac->stop_mask) & BIT(dmac_chan->chan_id);
+ if (dmac_chan->chan_used == 0 && stop == 0 &&
+ !list_empty(&dmac_chan->queue_list)) {
+ dmac_chan->chan_used = 1;
+ list_move(dmac_chan->queue_list.next, &dmac_chan->active_list);
+ atcdmac_start_transfer(dmac_chan,
+ atcdmac_get_active_head(dmac_chan));
+ }
+
+ spin_unlock_irqrestore(&dmac_chan->lock, flags);
+}
+
+static unsigned char atcdmac_map_buswidth(enum dma_slave_buswidth addr_width)
+{
+ switch (addr_width) {
+ case DMA_SLAVE_BUSWIDTH_1_BYTE:
+ return 0;
+ case DMA_SLAVE_BUSWIDTH_2_BYTES:
+ return 1;
+ case DMA_SLAVE_BUSWIDTH_4_BYTES:
+ return 2;
+ case DMA_SLAVE_BUSWIDTH_8_BYTES:
+ return 3;
+ case DMA_SLAVE_BUSWIDTH_16_BYTES:
+ return 4;
+ case DMA_SLAVE_BUSWIDTH_32_BYTES:
+ return 5;
+ default:
+ return 0;
+ }
+}
+
+static unsigned int atcdmac_map_tran_width(dma_addr_t src,
+ dma_addr_t dst,
+ size_t len,
+ unsigned int max_align_bytes)
+{
+ unsigned int align = src | dst | len | max_align_bytes;
+ unsigned int width;
+
+ if (!(align & 0x1F))
+ width = WIDTH_32_BYTES;
+ else if (!(align & 0xF))
+ width = WIDTH_16_BYTES;
+ else if (!(align & 0x7))
+ width = WIDTH_8_BYTES;
+ else if (!(align & 0x3))
+ width = WIDTH_4_BYTES;
+ else if (!(align & 0x1))
+ width = WIDTH_2_BYTES;
+ else
+ width = WIDTH_1_BYTE;
+
+ return width;
+}
+
+/**
+ * atcdmac_convert_burst - Convert burst size to a power of two index
+ * @burst_size: Actual burst size in bytes
+ *
+ * This function converts a burst size (e.g., 1, 2, 4, 8...) into the
+ * corresponding hardware-encoded value, which represents the index of
+ * the power of two.
+ *
+ * Return: The zero-based power-of-two index corresponding to @burst_size.
+ *
+ * Example:
+ * If burst_size is 8 (binary 1000), the most significant bit is at position
+ * 4, so the function returns 3 (i.e., 4 - 1).
+ */
+static unsigned char atcdmac_convert_burst(unsigned int burst_size)
+{
+ return fls(burst_size) - 1;
+}
+
+/*
+ * The burst field says how many transfers the channel makes before it is
+ * re-arbitrated, and the controller raises its error exception for an
+ * encoding it does not define. The transfer itself is correct whatever the
+ * burst is, so cap the encoding rather than refuse the transfer.
+ */
+static unsigned int atcdmac_burst_enc(unsigned int burst_size)
+{
+ return min_t(unsigned int, ilog2(burst_size), SRC_BURST_SIZE_MAX_ENC);
+}
+
+static struct atcdmac_desc *
+atcdmac_build_desc(struct atcdmac_chan *dmac_chan,
+ dma_addr_t src,
+ dma_addr_t dst,
+ unsigned int ctrl,
+ unsigned int trans_size)
+{
+ struct atcdmac_desc *desc;
+
+ desc = atcdmac_get_desc(dmac_chan);
+ if (!desc)
+ return NULL;
+
+ desc->regs.src_addr_lo = lower_32_bits(src);
+ desc->regs.src_addr_hi = upper_32_bits(src);
+ desc->regs.dst_addr_lo = lower_32_bits(dst);
+ desc->regs.dst_addr_hi = upper_32_bits(dst);
+ desc->regs.ctrl = ctrl;
+ desc->regs.trans_size = trans_size;
+
+ return desc;
+}
+
+/**
+ * atcdmac_prep_dma_memcpy - Prepare a DMA memcpy operation for the specified
+ * channel
+ * @chan: DMA channel to configure for the operation
+ * @dst: Physical destination address for the transfer
+ * @src: Physical source address for the transfer
+ * @len: Size of the data to transfer, in bytes
+ * @flags: Status flags for the transfer descriptor
+ *
+ * This function sets up a DMA memcpy operation to transfer data from the
+ * specified source address to the destination address. It returns a DMA
+ * descriptor that represents the configured transaction.
+ *
+ * Return: The prepared descriptor, or NULL on failure.
+ */
+static struct dma_async_tx_descriptor *
+atcdmac_prep_dma_memcpy(struct dma_chan *chan,
+ dma_addr_t dst,
+ dma_addr_t src,
+ size_t len,
+ unsigned long flags)
+{
+ struct atcdmac_chan *dmac_chan = atcdmac_chan_to_dmac_chan(chan);
+ struct atcdmac_dmac *dmac = atcdmac_dev_to_dmac(chan->device);
+ struct atcdmac_desc *desc;
+ unsigned int src_width;
+ unsigned int dst_width;
+ unsigned int ctrl;
+ unsigned char src_max_burst;
+
+ if (unlikely(!len)) {
+ dev_warn(atcdmac_chan_to_dev(chan),
+ "Failed to prepare DMA operation: len is zero\n");
+ return NULL;
+ }
+
+ src_max_burst =
+ atcdmac_convert_burst((unsigned int)SRC_BURST_SIZE_1024);
+ src_width = atcdmac_map_tran_width(src,
+ dst,
+ len,
+ 1 << dmac->data_width);
+ dst_width = src_width;
+ ctrl = SRC_BURST_SIZE(src_max_burst) |
+ SRC_ADDR_MODE_INCR |
+ DST_ADDR_MODE_INCR |
+ DST_WIDTH(dst_width) |
+ SRC_WIDTH(src_width);
+
+ desc = atcdmac_build_desc(dmac_chan, src, dst, ctrl,
+ len >> src_width);
+ if (!desc)
+ goto err_desc_get;
+
+ return &desc->txd;
+
+err_desc_get:
+ dev_warn(atcdmac_chan_to_dev(chan), "Failed to allocate descriptor\n");
+ return NULL;
+}
+
+static int atcdmac_get_slave_cfg(struct dma_slave_config *sconfig,
+ enum dma_transfer_direction dir,
+ struct atcdmac_slave_cfg *cfg)
+{
+ if (dir == DMA_MEM_TO_DEV) {
+ cfg->reg = sconfig->dst_addr;
+ cfg->burst_bytes = sconfig->dst_addr_width * sconfig->dst_maxburst;
+ cfg->dev_width = sconfig->dst_addr_width;
+ cfg->width_src = atcdmac_map_buswidth(sconfig->src_addr_width);
+ cfg->width_dst = atcdmac_map_buswidth(sconfig->dst_addr_width);
+ } else if (dir == DMA_DEV_TO_MEM) {
+ cfg->reg = sconfig->src_addr;
+ cfg->burst_bytes = sconfig->src_addr_width * sconfig->src_maxburst;
+ cfg->dev_width = sconfig->src_addr_width;
+ cfg->width_src = atcdmac_map_buswidth(sconfig->src_addr_width);
+ cfg->width_dst = atcdmac_map_buswidth(sconfig->dst_addr_width);
+ } else {
+ return -EINVAL;
+ }
+
+ /*
+ * A zero width means the client never ran dmaengine_slave_config(),
+ * or described the other direction only. Its device address has not
+ * been supplied either, so the transfer would run against address
+ * zero, and atcdmac_build_slave_desc() divides by the width.
+ */
+ if (!cfg->dev_width)
+ return -EINVAL;
+
+ /*
+ * A zero burst means the client expressed no preference rather than
+ * asking for nothing, so transfer one word at a time.
+ */
+ if (!cfg->burst_bytes)
+ cfg->burst_bytes = cfg->dev_width;
+
+ /*
+ * The device address is the source for DEV_TO_MEM and the destination
+ * for MEM_TO_DEV, and either way the controller accesses it at the
+ * device width. DS118 Tables 14 and 16 raise an error event if it is
+ * not aligned to the width it is accessed at.
+ */
+ if (!IS_ALIGNED(cfg->reg, cfg->dev_width))
+ return -EINVAL;
+
+ return 0;
+}
+
+static struct atcdmac_desc *
+atcdmac_build_slave_desc(struct atcdmac_chan *dmac_chan,
+ const struct atcdmac_slave_cfg *cfg,
+ dma_addr_t mem, unsigned int len,
+ enum dma_transfer_direction dir,
+ unsigned int data_width)
+{
+ unsigned int width_cal;
+ unsigned short burst_size;
+ unsigned int ctrl;
+ dma_addr_t src, dst;
+
+ width_cal = atcdmac_map_tran_width(mem, cfg->reg, len,
+ (1 << data_width) | cfg->burst_bytes);
+ if (dir == DMA_MEM_TO_DEV) {
+ if (cfg->burst_bytes < (1 << width_cal)) {
+ burst_size = cfg->burst_bytes;
+ width_cal = WIDTH_1_BYTE;
+ } else {
+ burst_size = cfg->burst_bytes / (1 << width_cal);
+ }
+ ctrl = SRC_ADDR_MODE_INCR | DST_ADDR_MODE_FIXED |
+ DST_HS | DST_REQ(dmac_chan->req_num) |
+ SRC_WIDTH(width_cal) | DST_WIDTH(cfg->width_dst) |
+ SRC_BURST_SIZE(atcdmac_burst_enc(burst_size));
+ src = mem;
+ dst = cfg->reg;
+ } else {
+ burst_size = cfg->burst_bytes / cfg->dev_width;
+ ctrl = SRC_ADDR_MODE_FIXED | DST_ADDR_MODE_INCR |
+ SRC_HS | SRC_REQ(dmac_chan->req_num) |
+ SRC_WIDTH(cfg->width_src) | DST_WIDTH(width_cal) |
+ SRC_BURST_SIZE(atcdmac_burst_enc(burst_size));
+ src = cfg->reg;
+ dst = mem;
+ width_cal = cfg->width_src;
+ }
+ return atcdmac_build_desc(dmac_chan, src, dst, ctrl,
+ len >> width_cal);
+}
+
+/**
+ * atcdmac_prep_device_sg - Prepare descriptors for memory/device DMA
+ * transactions
+ * @chan: DMA channel to configure for the operation
+ * @sgl: Scatter-gather list representing the memory regions to transfer
+ * @sg_len: Number of entries in the scatter-gather list
+ * @direction: Direction of the DMA transfer
+ * @flags: Status flags for the transfer descriptor
+ * @context: transaction context (ignored)
+ *
+ * This function prepares a DMA transaction by setting up the required
+ * descriptors based on the provided scatter-gather list and parameters.
+ * It supports memory-to-device and device-to-memory DMA transfers.
+ *
+ * Return: The first descriptor of the prepared chain, or NULL on failure.
+ */
+static struct dma_async_tx_descriptor *
+atcdmac_prep_device_sg(struct dma_chan *chan,
+ struct scatterlist *sgl,
+ unsigned int sg_len,
+ enum dma_transfer_direction direction,
+ unsigned long flags,
+ void *context)
+{
+ struct atcdmac_chan *dmac_chan = atcdmac_chan_to_dmac_chan(chan);
+ struct atcdmac_dmac *dmac = atcdmac_dev_to_dmac(chan->device);
+ struct atcdmac_slave_cfg cfg;
+ struct atcdmac_desc *first = NULL;
+ struct atcdmac_desc *prev = NULL;
+ struct scatterlist *sg;
+ unsigned int i;
+
+ if (unlikely(!sg_len)) {
+ dev_warn(atcdmac_chan_to_dev(chan), "sg_len is zero\n");
+ return NULL;
+ }
+
+ if (atcdmac_get_slave_cfg(&dmac_chan->dma_sconfig, direction, &cfg)) {
+ dev_err(atcdmac_chan_to_dev(chan),
+ "Invalid transfer direction %d\n", direction);
+ return NULL;
+ }
+
+ for_each_sg(sgl, sg, sg_len, i) {
+ struct atcdmac_desc *desc;
+ dma_addr_t mem = sg_dma_address(sg);
+ unsigned int len = sg_dma_len(sg);
+
+ if (unlikely(!len)) {
+ dev_err(atcdmac_chan_to_dev(chan),
+ "sg(%u) data len is zero\n", i);
+ goto err;
+ }
+
+ /*
+ * TranSize counts device-width units, so a length that is not
+ * a whole number of them either transfers fewer bytes than
+ * asked for without reporting it, or programs a zero
+ * TranSize, which DS118 Table 13 makes an error event. The
+ * width the device is accessed at cannot be lowered to suit a
+ * short segment, so leave the division to the client.
+ */
+ if (!IS_ALIGNED(len, cfg.dev_width)) {
+ dev_err(atcdmac_chan_to_dev(chan),
+ "sg(%u) length %u is not a multiple of the %u-byte device width\n",
+ i, len, cfg.dev_width);
+ goto err;
+ }
+
+ desc = atcdmac_build_slave_desc(dmac_chan, &cfg, mem, len,
+ direction, dmac->data_width);
+ if (!desc)
+ goto err_desc_get;
+
+ atcdmac_chain_desc(&first, &prev, desc, false);
+ }
+
+ first->txd.cookie = -EBUSY;
+ first->txd.flags = flags;
+
+ return &first->txd;
+
+err_desc_get:
+ dev_warn(atcdmac_chan_to_dev(chan), "Failed to allocate descriptor\n");
+
+err:
+ if (first)
+ atcdmac_put_desc(dmac_chan, first);
+ return NULL;
+}
+
+static struct dma_async_tx_descriptor *
+atcdmac_prep_dma_cyclic(struct dma_chan *chan,
+ dma_addr_t buf_addr,
+ size_t buf_len,
+ size_t period_len,
+ enum dma_transfer_direction direction,
+ unsigned long flags)
+{
+ struct atcdmac_chan *dmac_chan = atcdmac_chan_to_dmac_chan(chan);
+ struct atcdmac_dmac *dmac = atcdmac_dev_to_dmac(chan->device);
+ struct atcdmac_slave_cfg cfg;
+ struct atcdmac_desc *first = NULL;
+ struct atcdmac_desc *prev = NULL;
+ unsigned int period;
+
+ if (period_len == 0 || buf_len == 0) {
+ dev_warn(atcdmac_chan_to_dev(chan),
+ "invalid cyclic params buf_len=%zu period_len=%zu\n",
+ buf_len, period_len);
+ return NULL;
+ }
+
+ if (atcdmac_get_slave_cfg(&dmac_chan->dma_sconfig, direction, &cfg)) {
+ dev_err(atcdmac_chan_to_dev(chan),
+ "Invalid transfer direction %d\n", direction);
+ return NULL;
+ }
+
+ /*
+ * Every period has to be a whole number of device-width units, for
+ * the reason given in atcdmac_prep_device_sg(). Each one is
+ * period_len bytes except a trailing short one, whose length is
+ * buf_len % period_len, so checking both covers all of them and
+ * there is nothing left to check inside the loop.
+ */
+ if (!IS_ALIGNED(period_len, cfg.dev_width) ||
+ !IS_ALIGNED(buf_len, cfg.dev_width)) {
+ dev_err(atcdmac_chan_to_dev(chan),
+ "Invalid cyclic length for a %u-byte device width: buf_len=%zu period_len=%zu\n",
+ cfg.dev_width, buf_len, period_len);
+ return NULL;
+ }
+
+ for (period = 0; period < buf_len; period += period_len) {
+ struct atcdmac_desc *desc;
+ dma_addr_t mem = buf_addr + period;
+ unsigned int len = min_t(unsigned int, period_len,
+ buf_len - period);
+
+ desc = atcdmac_build_slave_desc(dmac_chan, &cfg, mem, len,
+ direction, dmac->data_width);
+ if (!desc)
+ goto err_desc_get;
+ atcdmac_chain_desc(&first, &prev, desc, true);
+ }
+
+ first->txd.flags = flags;
+ first->cyclic = true;
+
+ return &first->txd;
+
+err_desc_get:
+ dev_warn(atcdmac_chan_to_dev(chan), "Failed to allocate descriptor\n");
+ if (first)
+ atcdmac_put_desc(dmac_chan, first);
+
+ return NULL;
+}
+
+static int atcdmac_set_device_config(struct dma_chan *chan,
+ struct dma_slave_config *sconfig)
+{
+ struct atcdmac_chan *dmac_chan = atcdmac_chan_to_dmac_chan(chan);
+
+ /* Check if this chan is configured for device transfers */
+ if (!dmac_chan->dev_chan)
+ return -EINVAL;
+
+ /*
+ * A zero field means the client did not describe that side. The
+ * direction is not known here - a channel may be used for both - so
+ * a field is only checked when it is present, and
+ * atcdmac_get_slave_cfg() rejects a direction whose width is still
+ * missing once the direction is known.
+ *
+ * Must be powers of two according to ATCDMAC300 spec.
+ */
+ if ((sconfig->src_maxburst && !is_power_of_2(sconfig->src_maxburst)) ||
+ (sconfig->dst_maxburst && !is_power_of_2(sconfig->dst_maxburst)) ||
+ (sconfig->src_addr_width && !is_power_of_2(sconfig->src_addr_width)) ||
+ (sconfig->dst_addr_width && !is_power_of_2(sconfig->dst_addr_width)))
+ return -EINVAL;
+
+ /*
+ * The transfer width field has no encoding beyond 32 bytes, and
+ * atcdmac_map_buswidth() would report a wider request as one byte
+ * rather than say so.
+ */
+ if (sconfig->src_addr_width > DMA_SLAVE_BUSWIDTH_32_BYTES ||
+ sconfig->dst_addr_width > DMA_SLAVE_BUSWIDTH_32_BYTES)
+ return -EINVAL;
+
+ memcpy(&dmac_chan->dma_sconfig, sconfig, sizeof(*sconfig));
+
+ return 0;
+}
+
+static int atcdmac_terminate_all(struct dma_chan *chan)
+{
+ struct atcdmac_chan *dmac_chan = atcdmac_chan_to_dmac_chan(chan);
+ struct atcdmac_desc *desc_cur, *desc_next;
+ LIST_HEAD(list);
+ unsigned long flags;
+ unsigned int val;
+ int ret;
+
+ spin_lock_irqsave(&dmac_chan->lock, flags);
+ atcdmac_abort_chan(dmac_chan);
+ atcdmac_enable_chan(dmac_chan, 0);
+ list_splice_init(&dmac_chan->queue_list, &list);
+ list_splice_init(&dmac_chan->active_list, &list);
+ dmac_chan->chan_used = 0;
+ spin_unlock_irqrestore(&dmac_chan->lock, flags);
+
+ /*
+ * The abort only cancels the pending transfers, the one in flight is
+ * still finished by the hardware. Wait for the channel to actually go
+ * down before the descriptors, which double as the hardware linked
+ * list nodes, are handed out again. The lists are already detached,
+ * so nothing can restart the channel while this runs.
+ */
+ ret = regmap_read_poll_timeout_atomic(dmac_chan->dma_dev->regmap,
+ REG_CH_EN,
+ val,
+ !(val & BIT(dmac_chan->chan_id)),
+ 10,
+ ATCDMAC_CHAN_TIMEOUT_US);
+ if (ret)
+ dev_err(atcdmac_chan_to_dev(chan),
+ "Timed out waiting for channel to disable\n");
+
+ list_for_each_entry_safe(desc_cur, desc_next, &list, desc_node) {
+ atcdmac_retire_desc(desc_cur);
+ atcdmac_put_desc(dmac_chan, desc_cur);
+ }
+
+ return ret;
+}
+
+/*
+ * All completion callbacks run from the threaded IRQ handler, so waiting for
+ * it to go idle is enough to guarantee that none is running or pending.
+ */
+static void atcdmac_synchronize(struct dma_chan *chan)
+{
+ struct atcdmac_dmac *dmac = atcdmac_dev_to_dmac(chan->device);
+
+ synchronize_irq(dmac->irq);
+}
+
+static int atcdmac_wait_chan_idle(struct atcdmac_dmac *dmac,
+ unsigned short chan_mask,
+ unsigned int timeout_us)
+{
+ unsigned int val;
+ int ret;
+
+ ret = regmap_read_poll_timeout(dmac->regmap,
+ REG_CH_EN,
+ val,
+ !(val & chan_mask),
+ 50,
+ timeout_us);
+ if (ret)
+ dev_err(dmac->dma_device.dev,
+ "Timeout waiting for channels to go idle (%d), still enabled: %#x\n",
+ ret, val & chan_mask);
+
+ return ret;
+}
+
+/**
+ * atcdmac_alloc_chan_resources - Allocate resources for a DMA channel
+ * @chan: The DMA channel for which resources are being allocated
+ *
+ * This function sets up and allocates the necessary resources for the
+ * specified DMA channel (chan). It ensures the channel is prepared to
+ * handle DMA requests from clients by allocating descriptors and any other
+ * required resources.
+ *
+ * Return: The number of descriptors successfully allocated, or a negative
+ * error code on failure.
+ */
+static int atcdmac_alloc_chan_resources(struct dma_chan *chan)
+{
+ struct atcdmac_chan *dmac_chan = atcdmac_chan_to_dmac_chan(chan);
+ struct atcdmac_dmac *dmac = atcdmac_dev_to_dmac(chan->device);
+ struct atcdmac_desc *desc;
+ int i;
+
+ if (atcdmac_is_chan_enabled(dmac_chan)) {
+ dev_err(atcdmac_chan_to_dev(chan),
+ "DMA channel is not in an idle state\n");
+ return -EBUSY;
+ }
+
+ /*
+ * Hand the channel over with no configuration of its own: the client
+ * has to run dmaengine_slave_config() for itself, and clients do
+ * exist that call it without looking at whether it succeeded. The
+ * core only gets here on the first reference, so this cannot discard
+ * a configuration that is still in use.
+ */
+ memset(&dmac_chan->dma_sconfig, 0, sizeof(dmac_chan->dma_sconfig));
+
+ if (!list_empty(&dmac_chan->free_list))
+ return dmac_chan->descs_allocated;
+
+ /*
+ * Spin-lock protection is not necessary during DMA channel
+ * initialization, as the channel is not yet in use at this stage,
+ * and the shared resources are not accessed by other threads.
+ */
+ for (i = 0; i < ATCDMAC_DESC_PER_CHAN; i++) {
+ desc = atcdmac_alloc_desc(chan, GFP_KERNEL);
+ if (!desc) {
+ dev_warn(dmac->dma_device.dev,
+ "Insufficient descriptors: only %d descriptors available\n",
+ i);
+ break;
+ }
+ list_add_tail(&desc->desc_node, &dmac_chan->free_list);
+ }
+ dmac_chan->descs_allocated = i;
+ dma_cookie_init(chan);
+ raw_spin_lock_irq(&dmac->lock);
+ dmac->stop_mask &= ~BIT(dmac_chan->chan_id);
+ raw_spin_unlock_irq(&dmac->lock);
+
+ return dmac_chan->descs_allocated;
+}
+
+/**
+ * atcdmac_free_chan_resources - Release a DMA channel's resources
+ * @chan: The DMA channel to release
+ *
+ * This function ensures that any remaining DMA descriptors in the
+ * active_list and queue_list are properly reclaimed. It releases all
+ * resources associated with the specified DMA channel and resets the
+ * channel's management structures to their initial states.
+ */
+static void atcdmac_free_chan_resources(struct dma_chan *chan)
+{
+ struct atcdmac_chan *dmac_chan = atcdmac_chan_to_dmac_chan(chan);
+ struct atcdmac_dmac *dmac = atcdmac_dev_to_dmac(chan->device);
+ struct atcdmac_desc *desc_next, *desc;
+ unsigned long flags;
+
+ WARN_ON_ONCE(atcdmac_is_chan_enabled(dmac_chan));
+
+ raw_spin_lock_irq(&dmac->lock);
+ dmac->stop_mask |= BIT(dmac_chan->chan_id);
+ raw_spin_unlock_irq(&dmac->lock);
+
+ atcdmac_terminate_all(chan);
+
+ /*
+ * Give up ownership only after the abort raised by the termination
+ * above has been serviced. Dropping the bit first would mask that
+ * status out of the acknowledge and leave the interrupt asserted.
+ */
+ atcdmac_synchronize(chan);
+
+ raw_spin_lock_irq(&dmac->lock);
+ dmac->owned_chan &= ~BIT(dmac_chan->chan_id);
+ raw_spin_unlock_irq(&dmac->lock);
+
+ spin_lock_irqsave(&dmac_chan->lock, flags);
+ list_for_each_entry_safe(desc,
+ desc_next,
+ &dmac_chan->free_list,
+ desc_node) {
+ list_del(&desc->desc_node);
+ dma_pool_free(dmac->dma_desc_pool, desc, desc->txd.phys);
+ }
+
+ INIT_LIST_HEAD(&dmac_chan->free_list);
+ dmac_chan->descs_allocated = 0;
+ dmac_chan->status = 0;
+ dmac_chan->chan_used = 0;
+ dmac_chan->dev_chan = false;
+ spin_unlock_irqrestore(&dmac_chan->lock, flags);
+}
+
+static bool atcdmac_filter_chan(struct dma_chan *chan, void *dma_dev)
+{
+ return dma_dev == chan->device->dev;
+}
+
+static struct dma_chan *atcdmac_dma_xlate_handler(struct of_phandle_args *dmac,
+ struct of_dma *of_dma)
+{
+ struct platform_device *dmac_pdev;
+ struct atcdmac_chan *dmac_chan;
+ struct dma_chan *chan;
+ dma_cap_mask_t mask;
+
+ if (dmac->args[0] > FIELD_MAX(SRC_REQ_SEL_MASK)) {
+ pr_err("%pOF: DMA request %u out of range, maximum is %lu\n",
+ dmac->np, dmac->args[0], FIELD_MAX(SRC_REQ_SEL_MASK));
+ return NULL;
+ }
+
+ dmac_pdev = of_find_device_by_node(dmac->np);
+ if (!dmac_pdev)
+ return NULL;
+
+ dma_cap_zero(mask);
+ dma_cap_set(DMA_SLAVE, mask);
+ chan = dma_request_channel(mask, atcdmac_filter_chan, &dmac_pdev->dev);
+ put_device(&dmac_pdev->dev);
+
+ if (!chan)
+ return NULL;
+
+ dmac_chan = atcdmac_chan_to_dmac_chan(chan);
+ dmac_chan->dev_chan = true;
+ dmac_chan->req_num = dmac->args[0];
+
+ return chan;
+}
+
+/**
+ * atcdmac_reset_and_wait_chan_idle - Reset the DMA controller and wait for
+ * all channels to become idle
+ * @dmac: Pointer to the DMA controller structure
+ *
+ * This function performs a reset of the DMA controller and ensures that all
+ * DMA channels are disabled.
+ *
+ * Return: 0 on success, or a negative error code if a channel did not
+ * become idle in time.
+ */
+static int atcdmac_reset_and_wait_chan_idle(struct atcdmac_dmac *dmac)
+{
+ /* Reset is write-only, so write the register rather than updating it. */
+ regmap_write(dmac->regmap, REG_CTL, DMAC_RESET);
+ msleep(20);
+ regmap_write(dmac->regmap, REG_CTL, 0);
+
+ return atcdmac_wait_chan_idle(dmac,
+ BIT(dmac->num_ch) - 1,
+ ATCDMAC_CHAN_TIMEOUT_US);
+}
+
+static int atcdmac_restore_iocp(struct atcdmac_dmac *dmac)
+{
+ if (!dmac->regmap_iocp)
+ return 0;
+
+ return regmap_write(dmac->regmap_iocp,
+ 0,
+ IOCP_CACHE_DMAC0_AW |
+ IOCP_CACHE_DMAC0_AR |
+ IOCP_CACHE_DMAC1_AW |
+ IOCP_CACHE_DMAC1_AR);
+}
+
+static int atcdmac_init_iocp(struct platform_device *pdev,
+ struct atcdmac_dmac *dmac)
+{
+ const struct regmap_config iocp_regmap_config = {
+ .name = "iocp",
+ .reg_bits = 32,
+ .val_bits = 32,
+ .reg_stride = 4,
+ .pad_bits = 0,
+ .max_register = 0,
+ .cache_type = REGCACHE_NONE,
+ };
+ struct resource *res;
+ void __iomem *regs;
+
+ res = platform_get_resource_byname(pdev, IORESOURCE_MEM, "iocp");
+ if (!res) {
+ dmac->regmap_iocp = NULL;
+ return 0;
+ }
+
+ regs = devm_ioremap_resource(&pdev->dev, res);
+ if (IS_ERR(regs)) {
+ dmac->regmap_iocp = NULL;
+ return dev_err_probe(&pdev->dev, PTR_ERR(regs),
+ "Failed to create ioremap for IOCP\n");
+ }
+
+ dmac->regmap_iocp = devm_regmap_init_mmio(&pdev->dev,
+ regs,
+ &iocp_regmap_config);
+ if (IS_ERR(dmac->regmap_iocp)) {
+ int ret = PTR_ERR(dmac->regmap_iocp);
+
+ dmac->regmap_iocp = NULL;
+ return dev_err_probe(&pdev->dev, ret,
+ "Failed to create regmap for IOCP\n");
+ }
+
+ return atcdmac_restore_iocp(dmac);
+}
+
+static void atcdmac_init_dma_device(struct atcdmac_dmac *dmac)
+{
+ struct dma_device *device = &dmac->dma_device;
+
+ device->device_alloc_chan_resources = atcdmac_alloc_chan_resources;
+ device->device_free_chan_resources = atcdmac_free_chan_resources;
+ device->device_tx_status = dma_cookie_status;
+ device->device_issue_pending = atcdmac_issue_pending;
+ device->device_prep_dma_memcpy = atcdmac_prep_dma_memcpy;
+ device->device_prep_slave_sg = atcdmac_prep_device_sg;
+ device->device_config = atcdmac_set_device_config;
+ device->device_terminate_all = atcdmac_terminate_all;
+ device->device_synchronize = atcdmac_synchronize;
+ device->device_prep_dma_cyclic = atcdmac_prep_dma_cyclic;
+
+ device->src_addr_widths = BIT(DMA_SLAVE_BUSWIDTH_1_BYTE) |
+ BIT(DMA_SLAVE_BUSWIDTH_2_BYTES) |
+ BIT(DMA_SLAVE_BUSWIDTH_4_BYTES);
+ device->dst_addr_widths = device->src_addr_widths;
+ device->directions = BIT(DMA_DEV_TO_MEM) | BIT(DMA_MEM_TO_DEV);
+ device->residue_granularity = DMA_RESIDUE_GRANULARITY_DESCRIPTOR;
+
+ dma_cap_set(DMA_SLAVE, device->cap_mask);
+ dma_cap_set(DMA_MEMCPY, device->cap_mask);
+ dma_cap_set(DMA_CYCLIC, device->cap_mask);
+}
+
+static int atcdmac_init_channels(struct platform_device *pdev,
+ struct atcdmac_dmac *dmac)
+{
+ struct regmap_config chan_regmap_config = {
+ .reg_bits = 32,
+ .val_bits = 32,
+ .reg_stride = 4,
+ .pad_bits = 0,
+ .cache_type = REGCACHE_NONE,
+ .max_register = REG_CH_LLP_HIGH_OFF,
+ };
+ struct atcdmac_chan *dmac_chan;
+ int ret;
+ int i;
+
+ INIT_LIST_HEAD(&dmac->dma_device.channels);
+
+ for (i = 0; i < dmac->num_ch; i++) {
+ char *regmap_name = kasprintf(GFP_KERNEL, "chan%d", i);
+
+ if (!regmap_name)
+ return -ENOMEM;
+
+ dmac_chan = &dmac->chan[i];
+ chan_regmap_config.name = regmap_name;
+ dmac_chan->regmap =
+ devm_regmap_init_mmio(&pdev->dev,
+ dmac->regs + REG_CH_OFF(i),
+ &chan_regmap_config);
+ kfree(regmap_name);
+
+ if (IS_ERR(dmac_chan->regmap)) {
+ ret = PTR_ERR(dmac_chan->regmap);
+ dev_err_probe(&pdev->dev, ret,
+ "Failed to create regmap for DMA chan%d\n",
+ i);
+ return ret;
+ }
+
+ spin_lock_init(&dmac_chan->lock);
+ dmac_chan->dma_chan.device = &dmac->dma_device;
+ dmac_chan->dma_dev = dmac;
+ dmac_chan->chan_id = i;
+ dmac_chan->chan_used = 0;
+
+ INIT_LIST_HEAD(&dmac_chan->active_list);
+ INIT_LIST_HEAD(&dmac_chan->queue_list);
+ INIT_LIST_HEAD(&dmac_chan->free_list);
+
+ list_add_tail(&dmac_chan->dma_chan.device_node,
+ &dmac->dma_device.channels);
+ dma_cookie_init(&dmac_chan->dma_chan);
+ }
+
+ return 0;
+}
+
+static int atcdmac_init_desc_pool(struct platform_device *pdev,
+ struct atcdmac_dmac *dmac)
+{
+ dmac->dma_desc_pool = dmam_pool_create(dev_name(&pdev->dev),
+ &pdev->dev,
+ sizeof(struct atcdmac_desc),
+ 64,
+ 4096);
+ if (!dmac->dma_desc_pool)
+ return dev_err_probe(&pdev->dev, -ENOMEM,
+ "Failed to create memory pool for DMA descriptors\n");
+ return 0;
+}
+
+static int atcdmac_init_irq(struct platform_device *pdev,
+ struct atcdmac_dmac *dmac)
+{
+ int irq = platform_get_irq(pdev, 0);
+
+ if (irq < 0)
+ return irq;
+
+ dmac->irq = irq;
+
+ return devm_request_threaded_irq(&pdev->dev,
+ irq,
+ atcdmac_interrupt,
+ atcdmac_irq_thread,
+ IRQF_SHARED | IRQF_ONESHOT,
+ dev_name(&pdev->dev),
+ dmac);
+}
+
+static int atcdmac_init_ioremap_and_regmap(struct platform_device *pdev,
+ struct atcdmac_dmac **out_dmac)
+{
+ const struct regmap_config dmac_regmap_config = {
+ .name = dev_name(&pdev->dev),
+ .reg_bits = 32,
+ .val_bits = 32,
+ .reg_stride = 4,
+ .pad_bits = 0,
+ .cache_type = REGCACHE_NONE,
+ .max_register = REG_CH_EN,
+ /* Accessed from the hard IRQ handler, see atcdmac_interrupt() */
+ .use_raw_spinlock = true,
+ };
+ struct atcdmac_dmac *dmac;
+ struct regmap *regmap;
+ void __iomem *regs;
+ unsigned int val;
+ int ret = 0;
+ unsigned char num_ch;
+
+ regs = devm_platform_ioremap_resource(pdev, 0);
+ if (IS_ERR(regs))
+ return dev_err_probe(&pdev->dev, PTR_ERR(regs),
+ "Failed to ioremap I/O resource\n");
+
+ regmap = devm_regmap_init_mmio(&pdev->dev, regs, &dmac_regmap_config);
+ if (IS_ERR(regmap))
+ return dev_err_probe(&pdev->dev, PTR_ERR(regmap),
+ "Failed to create regmap for I/O\n");
+
+ ret = regmap_read(regmap, REG_CFG, &val);
+ if (ret)
+ return dev_err_probe(&pdev->dev, ret,
+ "Failed to read the DMAC configuration\n");
+
+ num_ch = val & CH_NUM;
+ if (!num_ch)
+ return dev_err_probe(&pdev->dev, -ENODEV,
+ "Controller reports no DMA channel\n");
+
+ dmac = devm_kzalloc(&pdev->dev, struct_size(dmac, chan, num_ch),
+ GFP_KERNEL);
+ if (!dmac)
+ return -ENOMEM;
+
+ dmac->regmap = regmap;
+ dmac->num_ch = num_ch;
+ dmac->regs = regs;
+ /*
+ * Set early: atcdmac_wait_chan_idle() reports through this device and
+ * runs before atcdmac_init_dma_device() during probe.
+ */
+ dmac->dma_device.dev = &pdev->dev;
+
+ /*
+ * Adjust the AXI bus data width (from the DMAC Configuration
+ * Register) to align with the transfer width encoding (in the
+ * Channel n Control Register). For example, an AXI width of 0
+ * (32-bit) corresponds to a transfer width of 2 (word transfer).
+ */
+ dmac->data_width = FIELD_GET(DATA_WIDTH, val) + 2;
+ raw_spin_lock_init(&dmac->lock);
+
+ platform_set_drvdata(pdev, dmac);
+ if (dma_set_mask_and_coherent(&pdev->dev, DMA_BIT_MASK(64)))
+ ret = dma_set_mask_and_coherent(&pdev->dev, DMA_BIT_MASK(32));
+ if (ret)
+ return dev_err_probe(&pdev->dev, ret,
+ "Failed to set DMA mask\n");
+
+ *out_dmac = dmac;
+
+ return ret;
+}
+
+static int atcdmac_probe(struct platform_device *pdev)
+{
+ struct atcdmac_dmac *dmac;
+ int ret;
+
+ ret = atcdmac_init_ioremap_and_regmap(pdev, &dmac);
+ if (ret)
+ return ret;
+
+ ret = atcdmac_reset_and_wait_chan_idle(dmac);
+ if (ret)
+ return ret;
+
+ ret = atcdmac_init_desc_pool(pdev, dmac);
+ if (ret)
+ return ret;
+
+ ret = atcdmac_init_channels(pdev, dmac);
+ if (ret)
+ return ret;
+
+ ret = atcdmac_init_iocp(pdev, dmac);
+ if (ret)
+ return ret;
+
+ ret = atcdmac_init_irq(pdev, dmac);
+ if (ret)
+ return ret;
+
+ atcdmac_init_dma_device(dmac);
+
+ /*
+ * Only now that cache coherency is configured and the interrupt is
+ * hooked up may clients be allowed to grab a channel.
+ */
+ ret = dma_async_device_register(&dmac->dma_device);
+ if (ret)
+ return ret;
+
+ ret = of_dma_controller_register(pdev->dev.of_node,
+ atcdmac_dma_xlate_handler,
+ dmac);
+ if (ret)
+ goto err_dma_async_register;
+
+ return 0;
+
+err_dma_async_register:
+ dma_async_device_unregister(&dmac->dma_device);
+
+ return ret;
+}
+
+static int atcdmac_resume(struct device *dev)
+{
+ struct dma_chan *chan;
+ struct dma_chan *chan_next;
+ struct atcdmac_dmac *dmac = dev_get_drvdata(dev);
+ struct atcdmac_chan *dmac_chan;
+ unsigned long flags;
+ int ret;
+
+ ret = atcdmac_reset_and_wait_chan_idle(dmac);
+ if (ret)
+ return ret;
+
+ ret = atcdmac_restore_iocp(dmac);
+ if (ret)
+ return ret;
+
+ /*
+ * Interrupts are re-enabled before this callback runs, so a completion
+ * left over from before the suspend may still be queued. Drain it
+ * while stop_mask is still set, so it retires its descriptor without
+ * starting anything, rather than racing with the restart below and
+ * recycling a descriptor the hardware has just been pointed at.
+ */
+ synchronize_irq(dmac->irq);
+
+ raw_spin_lock_irqsave(&dmac->lock, flags);
+ dmac->stop_mask = 0;
+ raw_spin_unlock_irqrestore(&dmac->lock, flags);
+ list_for_each_entry_safe(chan,
+ chan_next,
+ &dmac->dma_device.channels,
+ device_node) {
+ dmac_chan = atcdmac_chan_to_dmac_chan(chan);
+ spin_lock_irqsave(&dmac_chan->lock, flags);
+ if (!list_empty(&dmac_chan->active_list) ||
+ !list_empty(&dmac_chan->queue_list))
+ atcdmac_start_next_trans(dmac_chan);
+ spin_unlock_irqrestore(&dmac_chan->lock, flags);
+ }
+
+ return 0;
+}
+
+static int atcdmac_suspend(struct device *dev)
+{
+ struct atcdmac_dmac *dmac = dev_get_drvdata(dev);
+ unsigned short chan_mask = BIT(dmac->num_ch) - 1;
+ struct atcdmac_chan *dmac_chan;
+ struct dma_chan *chan;
+ int ret;
+ unsigned long flags;
+
+ raw_spin_lock_irqsave(&dmac->lock, flags);
+ dmac->stop_mask = chan_mask;
+ raw_spin_unlock_irqrestore(&dmac->lock, flags);
+
+ /*
+ * stop_mask is written under dmac->lock but read under the channel
+ * lock, so a completion that is already in flight may still chain the
+ * next descriptor. Wait for the handler to finish before looking at
+ * the channels, otherwise a channel can be started again just after
+ * atcdmac_wait_chan_idle() has seen it idle and the system suspends
+ * with a transfer running.
+ */
+ synchronize_irq(dmac->irq);
+
+ ret = atcdmac_wait_chan_idle(dmac,
+ chan_mask,
+ ATCDMAC_CHAN_TIMEOUT_US * dmac->num_ch);
+ if (ret) {
+ /*
+ * The PM core does not run the resume callback for a device
+ * whose suspend failed, so nothing would ever clear stop_mask
+ * again and the controller would never start another
+ * transfer. Undo it here.
+ *
+ * Clearing the mask is not enough on its own. A channel that
+ * completed while it was set retired its descriptor without
+ * chaining the next one, so it can sit idle with work still on
+ * active_list and nothing left to start it: no completion is
+ * coming and atcdmac_issue_pending() refuses while chan_used
+ * is set. Give those a push. A channel that is still enabled
+ * is mid transfer - on this path, the one that failed to go
+ * idle - and chains its own next descriptor, so leave it be.
+ */
+ raw_spin_lock_irqsave(&dmac->lock, flags);
+ dmac->stop_mask = 0;
+ raw_spin_unlock_irqrestore(&dmac->lock, flags);
+
+ list_for_each_entry(chan, &dmac->dma_device.channels,
+ device_node) {
+ dmac_chan = atcdmac_chan_to_dmac_chan(chan);
+ spin_lock_irqsave(&dmac_chan->lock, flags);
+ if (!atcdmac_is_chan_enabled(dmac_chan))
+ atcdmac_start_next_trans(dmac_chan);
+ spin_unlock_irqrestore(&dmac_chan->lock, flags);
+ }
+ }
+
+ return ret;
+}
+
+/*
+ * Run in the late phase. dmaengine declares no dependency between a channel
+ * and its consumer, so this is the only ordering that puts every client's
+ * suspend callback - and the transfers it terminates there - ahead of the
+ * idle check in atcdmac_suspend(). Interrupts are still enabled this early,
+ * suspend_device_irqs() only runs in the noirq phase.
+ */
+static const struct dev_pm_ops atcdmac_pm_ops = {
+ LATE_SYSTEM_SLEEP_PM_OPS(atcdmac_suspend, atcdmac_resume)
+};
+
+static void atcdmac_remove(struct platform_device *pdev)
+{
+ struct atcdmac_dmac *dmac = platform_get_drvdata(pdev);
+
+ /*
+ * Quiesce the controller first so the devm-managed IRQ, which is
+ * released only after remove() returns, has nothing left to service.
+ */
+ atcdmac_reset_and_wait_chan_idle(dmac);
+ of_dma_controller_free(pdev->dev.of_node);
+ dma_async_device_unregister(&dmac->dma_device);
+}
+
+static const struct of_device_id atcdmac_dt_ids[] = {
+ { .compatible = "andestech,ae350-dma", },
+ { /* sentinel */ },
+};
+MODULE_DEVICE_TABLE(of, atcdmac_dt_ids);
+
+static struct platform_driver atcdmac_driver = {
+ .probe = atcdmac_probe,
+ .remove = atcdmac_remove,
+ .driver = {
+ .name = "atcdmac300",
+ .of_match_table = atcdmac_dt_ids,
+ /*
+ * A channel cannot be taken back from a client, and every
+ * resource atcdmac_free_chan_resources() needs - the
+ * controller structure, the register mapping, the interrupt
+ * and the descriptor pool - is devres managed and therefore
+ * gone once remove() returns. Unloading the module is
+ * already refused while a client holds a channel, so drop
+ * the sysfs knob that bypasses that.
+ */
+ .suppress_bind_attrs = true,
+ .pm = pm_sleep_ptr(&atcdmac_pm_ops),
+ },
+};
+module_platform_driver(atcdmac_driver);
+
+MODULE_AUTHOR("CL Wang <cl634@andestech.com>");
+MODULE_DESCRIPTION("Andes ATCDMAC300 controller driver");
+MODULE_LICENSE("GPL");
diff --git a/drivers/dma/atcdmac300.h b/drivers/dma/atcdmac300.h
new file mode 100644
index 000000000000..828b1a7b03d1
--- /dev/null
+++ b/drivers/dma/atcdmac300.h
@@ -0,0 +1,298 @@
+/* SPDX-License-Identifier: GPL-2.0-or-later */
+/*
+ * Header file for Andes ATCDMAC300 DMA controller driver
+ *
+ * Copyright (C) 2025 Andes Technology Corporation
+ */
+#ifndef ATCDMAC300_H
+#define ATCDMAC300_H
+
+#include <linux/bitfield.h>
+#include <linux/dmaengine.h>
+#include <linux/dmapool.h>
+#include <linux/interrupt.h>
+#include <linux/regmap.h>
+#include <linux/spinlock.h>
+
+/*
+ * Register Map Definitions for ATCDMAC300 DMA Controller
+ *
+ * These macros define the offsets and bit masks for various registers
+ * within the ATCDMAC300 DMA controller.
+ */
+
+/* Global DMAC Registers */
+#define REG_CFG 0x10
+#define CH_NUM GENMASK(3, 0)
+#define DATA_WIDTH GENMASK(25, 24)
+
+#define REG_CTL 0x20
+#define DMAC_RESET BIT(0)
+
+#define REG_CH_ABT 0x24
+
+#define REG_INT_STA 0x30
+#define TC_OFFSET 16
+#define ABT_OFFSET 8
+#define ERR_OFFSET 0
+#define DMA_TC(val) BIT(TC_OFFSET + (val))
+#define DMA_ABT(val) BIT(ABT_OFFSET + (val))
+#define DMA_ERR(val) BIT(ERR_OFFSET + (val))
+#define DMA_TC_FIELD GENMASK(TC_OFFSET + 7, TC_OFFSET)
+#define DMA_ABT_FIELD GENMASK(ABT_OFFSET + 7, ABT_OFFSET)
+#define DMA_ERR_FIELD GENMASK(ERR_OFFSET + 7, ERR_OFFSET)
+#define DMA_INT_ALL(val) (FIELD_GET(DMA_TC_FIELD, (val)) | \
+ FIELD_GET(DMA_ABT_FIELD, (val)) | \
+ FIELD_GET(DMA_ERR_FIELD, (val)))
+#define DMA_INT_CLR(val) (FIELD_PREP(DMA_TC_FIELD, (val)) | \
+ FIELD_PREP(DMA_ABT_FIELD, (val)) | \
+ FIELD_PREP(DMA_ERR_FIELD, (val)))
+
+#define REG_CH_EN 0x34
+
+#define REG_CH_OFF(ch) ((ch) * 0x20 + 0x40)
+#define REG_CH_CTL_OFF 0x0
+#define REG_CH_SIZE_OFF 0x4
+#define REG_CH_SRC_LOW_OFF 0x8
+#define REG_CH_SRC_HIGH_OFF 0xC
+#define REG_CH_DST_LOW_OFF 0x10
+#define REG_CH_DST_HIGH_OFF 0x14
+#define REG_CH_LLP_LOW_OFF 0x18
+#define REG_CH_LLP_HIGH_OFF 0x1C
+
+#define SRC_BURST_SIZE_1 BIT(0)
+#define SRC_BURST_SIZE_2 BIT(1)
+#define SRC_BURST_SIZE_4 BIT(2)
+#define SRC_BURST_SIZE_8 BIT(3)
+#define SRC_BURST_SIZE_16 BIT(4)
+#define SRC_BURST_SIZE_32 BIT(5)
+#define SRC_BURST_SIZE_64 BIT(6)
+#define SRC_BURST_SIZE_128 BIT(7)
+#define SRC_BURST_SIZE_256 BIT(8)
+#define SRC_BURST_SIZE_512 BIT(9)
+#define SRC_BURST_SIZE_1024 BIT(10)
+#define SRC_BURST_SIZE_MASK GENMASK(27, 24)
+/* 0xa is 1024 transfers; 0xb and above are reserved by the controller */
+#define SRC_BURST_SIZE_MAX_ENC 0xa
+#define SRC_BURST_SIZE(size) FIELD_PREP(SRC_BURST_SIZE_MASK, size)
+
+#define WIDTH_1_BYTE 0x0
+#define WIDTH_2_BYTES 0x1
+#define WIDTH_4_BYTES 0x2
+#define WIDTH_8_BYTES 0x3
+#define WIDTH_16_BYTES 0x4
+#define WIDTH_32_BYTES 0x5
+#define SRC_WIDTH_MASK GENMASK(23, 21)
+#define SRC_WIDTH(width) FIELD_PREP(SRC_WIDTH_MASK, width)
+#define SRC_WIDTH_GET(val) FIELD_GET(SRC_WIDTH_MASK, val)
+#define DST_WIDTH_MASK GENMASK(20, 18)
+#define DST_WIDTH(width) FIELD_PREP(DST_WIDTH_MASK, width)
+#define DST_WIDTH_GET(val) FIELD_GET(DST_WIDTH_MASK, val)
+
+/* DMA handshake mode */
+#define SRC_HS BIT(17)
+#define DST_HS BIT(16)
+
+/* Address control */
+#define SRC_ADDR_CTRL_MASK GENMASK(15, 14)
+#define SRC_ADDR_MODE_INCR FIELD_PREP(SRC_ADDR_CTRL_MASK, 0x0)
+#define SRC_ADDR_MODE_DECR FIELD_PREP(SRC_ADDR_CTRL_MASK, 0x1)
+#define SRC_ADDR_MODE_FIXED FIELD_PREP(SRC_ADDR_CTRL_MASK, 0x2)
+#define DST_ADDR_CTRL_MASK GENMASK(13, 12)
+#define DST_ADDR_MODE_INCR FIELD_PREP(DST_ADDR_CTRL_MASK, 0x0)
+#define DST_ADDR_MODE_DECR FIELD_PREP(DST_ADDR_CTRL_MASK, 0x1)
+#define DST_ADDR_MODE_FIXED FIELD_PREP(DST_ADDR_CTRL_MASK, 0x2)
+
+/* DMA request select */
+#define SRC_REQ_SEL_MASK GENMASK(11, 8)
+#define SRC_REQ(req_num) FIELD_PREP(SRC_REQ_SEL_MASK, (req_num))
+#define DST_REQ_SEL_MASK GENMASK(7, 4)
+#define DST_REQ(req_num) FIELD_PREP(DST_REQ_SEL_MASK, (req_num))
+
+/* Channel abort interrupt mask */
+#define INT_ABT_MASK BIT(3)
+/* Channel error interrupt mask */
+#define INT_ERR_MASK BIT(2)
+/* Channel terminal count interrupt mask */
+#define INT_TC_MASK BIT(1)
+
+/* Channel Enable */
+#define CHEN BIT(0)
+
+#define IOCP_CACHE_DMAC0_AW GENMASK(3, 0)
+#define IOCP_CACHE_DMAC0_AR GENMASK(7, 4)
+#define IOCP_CACHE_DMAC1_AW GENMASK(11, 8)
+#define IOCP_CACHE_DMAC1_AR GENMASK(15, 12)
+
+#define ATCDMAC_DESC_PER_CHAN 64
+#define ATCDMAC_CHAN_TIMEOUT_US 100000
+/*
+ * Status for bottom-half processing.
+ */
+enum dma_sta {
+ ATCDMAC_STA_TC = 0,
+ ATCDMAC_STA_ERR,
+ ATCDMAC_STA_ABORT
+};
+
+/**
+ * struct atcdmac_regs - Hardware DMA descriptor registers.
+ *
+ * @ctrl: Channel Control Register.
+ * @trans_size: Transfer Size Register.
+ * @src_addr_lo: Source Address Register (low 32-bit).
+ * @src_addr_hi: Source Address Register (high 32-bit).
+ * @dst_addr_lo: Destination Address Register (low 32-bit).
+ * @dst_addr_hi: Destination Address Register (high 32-bit).
+ * @ll_ptr_lo: Linked List Pointer Register (low 32-bit).
+ * @ll_ptr_hi: Linked List Pointer Register (high 32-bit).
+ */
+struct atcdmac_regs {
+ unsigned int ctrl;
+ unsigned int trans_size;
+ unsigned int src_addr_lo;
+ unsigned int src_addr_hi;
+ unsigned int dst_addr_lo;
+ unsigned int dst_addr_hi;
+ unsigned int ll_ptr_lo;
+ unsigned int ll_ptr_hi;
+};
+
+/**
+ * struct atcdmac_desc - Internal representation of a DMA descriptor.
+ *
+ * @regs: Hardware registers for this descriptor.
+ * @txd: DMA transaction descriptor for dmaengine framework.
+ * @desc_node: Node for linking descriptors in software lists.
+ * @tx_list: List head for chaining multiple descriptors in a single transfer.
+ * @at: Next descriptor in a cyclic transfer (for internal use).
+ * @cyclic: Set when this descriptor belongs to a cyclic transfer.
+ */
+struct atcdmac_desc {
+ struct atcdmac_regs regs;
+ struct dma_async_tx_descriptor txd;
+ struct list_head desc_node;
+ struct list_head tx_list;
+ struct list_head *at;
+ bool cyclic;
+};
+
+/**
+ * struct atcdmac_chan - Private data for each DMA channel.
+ *
+ * @dma_chan: Common DMA engine channel object members
+ * @dma_dev: Pointer to the struct atcdmac_dmac
+ * @dma_sconfig: DMA transfer config for device-to-memory or memory-to-device
+ * @regmap: Regmap for the DMA channel register
+ * @active_list: List of descriptors being processed by the DMA engine
+ * @queue_list: List of descriptors ready to be submitted to the DMA engine
+ * @free_list: List of descriptors available for reuse by the channel
+ * @lock: Protects data access in atomic operations
+ * @status: Transmit status info, shared between top-half and threaded IRQ
+ * @descs_allocated: Number of descriptors currently allocated in the pool
+ * @chan_id: Channel ID number
+ * @req_num: Request number assigned to the channel
+ * @chan_used: Indicates whether the DMA channel is currently in use
+ * @dev_chan: Indicates if the DMA channel transfers data between device
+ * and memory
+ */
+struct atcdmac_chan {
+ struct dma_chan dma_chan;
+ struct atcdmac_dmac *dma_dev;
+ struct dma_slave_config dma_sconfig;
+ struct regmap *regmap;
+
+ struct list_head active_list;
+ struct list_head queue_list;
+ struct list_head free_list;
+
+ spinlock_t lock; /* protects active_list, queue_list, status */
+ unsigned long status;
+ unsigned short descs_allocated;
+ unsigned char chan_id;
+ unsigned char req_num;
+ bool chan_used;
+ bool dev_chan;
+};
+
+/**
+ * struct atcdmac_dmac - Representation of the ATCDMAC300 DMA controller
+ * @dma_device: DMA device object for integration with DMA engine framework
+ * @regmap: Regmap for main DMA controller registers
+ * @regmap_iocp: Regmap for IOCP registers
+ * @dma_desc_pool: DMA descriptor pool for allocating and managing descriptors
+ * @regs: Memory-mapped base address of the main DMA controller registers
+ * @irq: Interrupt line shared by all channels, used by device_synchronize
+ * @lock: Protects data access in atomic operations
+ * @owned_chan: Bitmask of DMA channels this driver has started a transfer on
+ * @stop_mask: Stops the DMA channel after the current transaction completes
+ * @data_width: Max data bus width supported by the DMA controller
+ * @num_ch: Total number of DMA channels available in the controller
+ * @chan: Array of DMA channel structures, sized by the controller's channel
+ * count
+ */
+struct atcdmac_dmac {
+ struct dma_device dma_device;
+ struct regmap *regmap;
+ struct regmap *regmap_iocp;
+ struct dma_pool *dma_desc_pool;
+ void __iomem *regs;
+ int irq;
+ raw_spinlock_t lock; /* protects owned_chan, stop_mask */
+ unsigned short owned_chan;
+ unsigned short stop_mask;
+ unsigned char data_width;
+ unsigned char num_ch;
+ struct atcdmac_chan chan[] __counted_by(num_ch);
+};
+
+/**
+ * struct atcdmac_slave_cfg - Pre-computed per-transfer slave DMA configuration
+ *
+ * @reg: Device-side register address (source for DEV_TO_MEM, destination
+ * for MEM_TO_DEV).
+ * @burst_bytes: Burst size in bytes (addr_width * maxburst from
+ * dma_slave_config). Used to derive the hardware burst-count
+ * field and to select the optimal transfer width.
+ * @dev_width: Device-side bus width in bytes (src_addr_width for DEV_TO_MEM,
+ * dst_addr_width for MEM_TO_DEV). Used to convert @burst_bytes
+ * into a hardware burst-count for the device side.
+ * @width_src: Hardware-encoded source transfer width (output of
+ * atcdmac_map_buswidth() on the source bus width).
+ * @width_dst: Hardware-encoded destination transfer width (output of
+ * atcdmac_map_buswidth() on the destination bus width).
+ */
+struct atcdmac_slave_cfg {
+ dma_addr_t reg;
+ unsigned int burst_bytes;
+ unsigned int dev_width;
+ unsigned int width_src;
+ unsigned int width_dst;
+};
+
+/*
+ * Helper functions to convert between dmaengine and internal structures.
+ */
+static inline struct atcdmac_desc *
+atcdmac_txd_to_dma_desc(struct dma_async_tx_descriptor *txd)
+{
+ return container_of(txd, struct atcdmac_desc, txd);
+}
+
+static inline struct atcdmac_chan *
+atcdmac_chan_to_dmac_chan(struct dma_chan *chan)
+{
+ return container_of(chan, struct atcdmac_chan, dma_chan);
+}
+
+static inline struct atcdmac_dmac *atcdmac_dev_to_dmac(struct dma_device *dev)
+{
+ return container_of(dev, struct atcdmac_dmac, dma_device);
+}
+
+static inline struct device *atcdmac_chan_to_dev(struct dma_chan *chan)
+{
+ return &chan->dev->device;
+}
+
+#endif
--
2.34.1
^ permalink raw reply [flat|nested] 6+ messages in thread