Re: [RFC PATCH 0/4] spi: cadence-xspi: add ACMD PIO support for NAND and NOR

["Nuno Sá" <nuno.sa@analog.com>]

Hi Mark, Fei,

So funny enough I have been also working on ACMD for this controller.
Even though it's running on a platform which is not upstream (but we
plan to do that - at some point -) I was also thinking in submitting
this. But Fei was faster. My approach is fairly simpler but it comes
with some assumptions from what I saw from the memory chips I tested.

The patch is in [1]: More comments below...

On Mon, Sep 21, 2026 at 05:36:57PM +0800, Fei Xie wrote:
> From: "fei.xie" <fei.xie@horizon.auto>
> 
> The Cadence XSPI controller provides an automatic command (ACMD) engine.
> It can execute complete flash transactions, including command sequences,
> device-ready polling and master-DMA data transfers.  Software submits a PIO
> command containing the flash address, DMA address and transfer length.
> 
> For SPI NAND, for example, a read transaction combines PAGE READ, status
> polling and READ CACHE in hardware.  Program and erase transactions also
> combine write enable, data or erase commands, and status polling.  This
> avoids submitting each command separately through STIG mode.
> 
> This RFC contains a working controller-local implementation for both SPI
> NAND and SPI NOR.  Unsupported operations continue to use STIG mode.
> 
> There is an important layering issue in this implementation.  In order to
> program the hardware sequences, the controller driver obtains upper-layer
> driver data and directly uses struct spinand_device or struct spi_nor.  It
> also keeps state across individual spi_mem_exec_op() calls in order to
> recognize a complete NAND transaction.  We do not consider either property
> desirable for a final implementation.
> 

> The hardware nevertheless needs information which is not available in one
> spi_mem_op.  A NAND read is represented by separate PAGE READ, status-poll
> and READ CACHE operations, while the ACMD engine needs all three before the
> transaction is started.  NAND geometry is also needed to translate row and
> column addresses for the ACMD address space.
> 
> We would appreciate guidance on the preferred interface boundary:
> 
>   1. Is a controller-local implementation acceptable because these command
>      sequences are specific to the Cadence ACMD engine?

Don't think so.

> 
>   2. Should SPI mem instead provide an interface for submitting an ordered
>      group of operations, including status-poll semantics, with SPI NAND
>      and SPI NOR constructing the group and the controller translating it
>      into hardware sequences?
> 
>   3. If the latter is preferred, should NAND geometry be supplied through
>      the sequence interface, or should it be exposed to controllers
>      separately?

I think a spi_mem callback with nand geometry/info could be something
admissible.

But now comes the experience I had... I also tried to use the ACMD for
page read + status poll + read cache and I actually think we could get
away with it by caching some info (like the row in the 0x13 command
which we always get AFAIR).

But more importantly I really did not saw such a performance improvement to
justify the added complexity and so I kind of dropped it.

And on top of that and now it starts to get more interesting, the real game
changer (for speed) is when the nand chip supports continuous reads and walks pages
by itself. Continuous reads are the only way (AFAIK) for the spi_mem core to
give the controllers reads bigger than the page size (like a full erase block).
And even if we somehow add a new controller capability to let the core know
that it can give bigger blocks to the controller (as the controller can also
walk pages), I suspect things would go south for nand chips with
continuous mode + ACMD also trying to walk the pages. Hence, we would
also need to take care of that.

Another thing worth mentioning is that you take an "all or nothing"
approach for ACMD. What I saw on the NOR flash I tested (which is an
octal one) is that STIG mode was actually more performant for smaller
chunks of data (my threshold is around 2k).

On ERASE and PROGRAM I also did not saw any meaningful gain at all.
Pretty much because the device programming time is the big bottleneck.

So, to sum things up I ended only using ACMD for reads and just using
profile 1 on the controller with no geometry knowledge which worked for
both the nand [2] and nor [3] chips I'm using (and all other commands
still use STIG). Again using the sequencer for nand might still be worth
it if the chip does not support continuous mode and has a big enough
page size or we somehow support letting the spi_mem core request bigger
than page size chunks of data for controllers that can handle it. But we
still need to take care to not break chips where cont mode is supported
where I think it makes more sense to let the memory chip walk the pages
rather than the controller.

Some speed tests for comparison on my side:

cat /proc/mtd
dev:    size   erasesize  name
mtd0: 00040000 00010000 "u-boot spl"
mtd1: 000c0000 00010000 "u-boot proper"
mtd2: 02000000 00010000 "kernel"
mtd3: 0df00000 00010000 "rootfs"
mtd4: 08000000 00020000 "xspi0-nor"
mtd5: 40000000 00040000 "xspi1-nand"


SPI-NOR
-----------------------------------------------

ACMD (only reads are > 2k are done in ACMD )
----

flash_speed -c 5 -d /dev/mtd4
not NAND flash, assume page size is 512 bytes.
scanning for bad eraseblocks
scanned 5 eraseblocks, 0 are bad
testing eraseblock write speed
eraseblock write speed is 1054 KiB/s
testing eraseblock read speed
eraseblock read speed is 106666 KiB/s
testing page write speed
page write speed is 1035 KiB/s
testing page read speed
page read speed is 15238 KiB/s
testing 2 page write speed
2 page write speed is 1044 KiB/s
testing 2 page read speed
2 page read speed is 26666 KiB/s
Testing erase speed
erase speed is 433 KiB/s
Testing 2x multi-block erase speed
2x multi-block erase speed is 428 KiB/s
Testing 4x multi-block erase speed
4x multi-block erase speed is 432 KiB/s
Testing 8x multi-block erase speed
8x multi-block erase speed is 431 KiB/s
Testing 16x multi-block erase speed
16x multi-block erase speed is 430 KiB/s
Testing 32x multi-block erase speed
32x multi-block erase speed is 428 KiB/s
Testing 64x multi-block erase speed
64x multi-block erase speed is 430 KiB/s
finished

STIG
----

flash_speed -c 5 -d /dev/mtd4
not NAND flash, assume page size is 512 bytes.
scanning for bad eraseblocks
scanned 5 eraseblocks, 0 are bad
testing eraseblock write speed
eraseblock write speed is 1054 KiB/s
testing eraseblock read speed
eraseblock read speed is 64000 KiB/s
testing page write speed
page write speed is 1042 KiB/s
testing page read speed
page read speed is 14883 KiB/s
testing 2 page write speed
2 page write speed is 1049 KiB/s
testing 2 page read speed
2 page read speed is 26666 KiB/s
Testing erase speed
erase speed is 435 KiB/s
Testing 2x multi-block erase speed
2x multi-block erase speed is 435 KiB/s
Testing 4x multi-block erase speed
4x multi-block erase speed is 436 KiB/s
Testing 8x multi-block erase speed
8x multi-block erase speed is 433 KiB/s
Testing 16x multi-block erase speed
16x multi-block erase speed is 434 KiB/s
Testing 32x multi-block erase speed
32x multi-block erase speed is 429 KiB/s
Testing 64x multi-block erase speed
64x multi-block erase speed is 431 KiB/s
finished


SPI-NAND
--------

ACMD (all reads are ACMD given that page size is 4k)
----

flash_speed -c 5 -d /dev/mtd5
scanning for bad eraseblocks
scanned 5 eraseblocks, 0 are bad
testing eraseblock write speed
eraseblock write speed is 6497 KiB/s
testing eraseblock read speed
eraseblock read speed is 44137 KiB/s
testing page write speed
page write speed is 6336 KiB/s
testing page read speed
page read speed is 16842 KiB/s
testing 2 page write speed
2 page write speed is 6400 KiB/s
testing 2 page read speed
2 page read speed is 19104 KiB/s
Testing erase speed
erase speed is 60952 KiB/s
Testing 2x multi-block erase speed
2x multi-block erase speed is 60952 KiB/s
Testing 4x multi-block erase speed
4x multi-block erase speed is 60952 KiB/s
Testing 8x multi-block erase speed
8x multi-block erase speed is 60952 KiB/s
Testing 16x multi-block erase speed
16x multi-block erase speed is 60952 KiB/s
Testing 32x multi-block erase speed
32x multi-block erase speed is 60952 KiB/s
Testing 64x multi-block erase speed
64x multi-block erase speed is 60952 KiB/s
finished

STIG
----

flash_speed -c 5 -d /dev/mtd5
scanning for bad eraseblocks
scanned 5 eraseblocks, 0 are bad
testing eraseblock write speed
eraseblock write speed is 6564 KiB/s
testing eraseblock read speed
eraseblock read speed is 19104 KiB/s
testing page write speed
page write speed is 6368 KiB/s
testing page read speed
page read speed is 18028 KiB/s
testing 2 page write speed
2 page write speed is 6432 KiB/s
testing 2 page read speed
2 page read speed is 18550 KiB/s
Testing erase speed
erase speed is 60952 KiB/s
Testing 2x multi-block erase speed
2x multi-block erase speed is 60952 KiB/s
Testing 4x multi-block erase speed
4x multi-block erase speed is 60952 KiB/s
Testing 8x multi-block erase speed
8x multi-block erase speed is 60952 KiB/s
Testing 16x multi-block erase speed
16x multi-block erase speed is 60952 KiB/s
Testing 32x multi-block erase speed
32x multi-block erase speed is 60952 KiB/s
Testing 64x multi-block erase speed
64x multi-block erase speed is 60952 KiB/s
finished

As you can see for the NAND, for single pages I do not see any major
difference (things become interesting when chunks of data are big
enough). The way I see it:

1. Do we really want to support erase + program? How much performance
   do we gain for those? My experience was pretty much none but may you
   saw something else.
2. For NORs, things are relatively simple and there's nothing we need
   from the core. But we should have a way to decide when ACMD pays off and
   I'm not sure an hardcoded threshold like mine is good for everybody. It
   might depend on number of lanes, clock speed, etc...
3. For NANDs, things are more complex. I would say that if the chip supports
   continuous reads, we should treat reads > page size pretty much the same way
   as we do for NOR (what I'm doing now). The question is what do we
   want to do when the memory chip does not have cont mode and what we
   want to do for single page read? Do we wanna go with the more
   complex approach of "exporting" nand geometry to the controller? For
   single page read and IIRC, I think I did not see that big of an
   improvement.

Would be nice to see some real performance numbers like the above.

Also, If I understood correctly, you tested this on an older kernel than
upstream right? Have you backported [4] for STIG? It kind of matters...

[1]: https://github.com/analogdevicesinc/linux/pull/3478/changes/81b30eea886fc9decf45d22f0a185d5f76e3bb86 
[2]: https://lore.kernel.org/linux-mtd/20260911-mtd-nand-new-chip-support-v2-1-e2925bf78c26@analog.com/
[3]: https://lore.kernel.org/linux-mtd/20260914-mtd-spi-nor-new-issi-chip-v2-2-3cd4d7e434b2@analog.com/
[4]: https://lore.kernel.org/linux-spi/178056886874.53724.4850286391745939707.b4-ty@b4/

Thx!
- Nuno Sá

> 
> We have prototyped the second approach locally by adding sequence callbacks
> to drivers/spi/spi-mem.c and constructing the NAND transactions in
> drivers/mtd/nand/spi/core.c.  It removes the controller dependency on MTD
> private structures, but the initial version duplicated page/OOB preparation
> and made the generic sequence interface carry NAND-specific geometry.  That
> prototype is therefore not included here; we would first like agreement on
> the interface.
> 
> Performance
> ===========
> 
> The SPI NAND path was tested on a Horizon Robotics J6B platform with a
> GigaDevice GD5F4GM8RE device.  A vendor Linux 6.1 backport of the same ACMD
> implementation was used for the hardware tests.
> 
> Under the same 80 MHz transfer configuration, the sequential-read results
> are:
> 
>   STIG using readq:  8.24 MB/s (7.86 MiB/s)
>   ACMD PIO + MDMA:  10.08 MB/s (9.61 MiB/s), about 22% higher
>   STIG read CPU:    59.74% / 40.87%
>   ACMD read CPU:    approximately 16%
> 
> Using the lower STIG CPU figure for a conservative comparison, ACMD reduced
> read CPU usage by 24.87 percentage points, or approximately 61% relative.
> 
> The CPU reduction comes from moving device-ready polling and command
> sequencing into the controller.  Software no longer needs to submit and
> wait for each STIG command while a page is being transferred.
> 
> The ACMD program path averaged 2.94 MiB/s over ten 128 KiB writes,
> excluding erase time.  A 128 KiB random-data erase/program/read test
> produced identical SHA-256 hashes and byte-for-byte comparison.  Read,
> program and erase were also traced to confirm that the ACMD path was used.
> 
> The mainline series has been compile-tested for arm64.  The SPI NAND
> hardware tests were performed with the Linux 6.1 backport because the test
> platform currently runs the vendor 6.1 kernel.  SPI NOR has been
> compile-tested but has not yet been tested on hardware.
> 
> This series is intended to start the API and layering discussion rather
> than to propose the current upper-layer coupling as the final design.
> 
> fei.xie (4):
>   dt-bindings: spi: cdns,xspi: add SPI NAND compatible
>   spi: cadence-xspi: add ACMD support for SPI NAND
>   spi: cadence-xspi: factor out reusable ACMD helpers
>   spi: cadence-xspi: add ACMD support for SPI NOR
> 
>  .../devicetree/bindings/spi/cdns,xspi.yaml    |    3 +-
>  drivers/spi/spi-cadence-xspi.c                | 1259 ++++++++++++++++-
>  2 files changed, 1260 insertions(+), 2 deletions(-)
> 
> -- 
> 2.34.1