Re: [PATCH V5 00/19] accel/amdxdna: Kernel submission and PM for AIE4

[Lizhi Hou <lizhi.hou@amd.com>]

Applied patches to drm-misc-next

On 10/19/26 09:47, David Zhang wrote:
> This patch series extends the amdxdna accelerator driver to support
> AMD AIE4 (NPU3) platforms, including classic and PF/VF for SR-IOV with
> kernel-mode command submission (KMQ), system power management, and
> runtime power management for the classic device.
>
> Changes in v5:
> - Rebase on the uninterruptible HMM invalidate wait fix, and move
>    aie2_hmm_invalidate() into common GEM code for AIE2 and AIE4.
> - Rebase on the umap range fault serialization fix, and move
>    aie2_populate_range() into common GEM code for AIE2 and AIE4.
> - Move runtime PM reference calls to the last patch, so each patch is
>    bisectable on its own.
>
> Changes in v4:
> - Return -ENODEV instead of -EAGAIN from aie4_cmd_wait() when the
>    context is in error, so waiters fail after a failed resume instead
>    of retrying forever.
> - Take a runtime PM reference in aie4_hwctx_init() so context creation
>    does not send mailbox messages to a suspended device.
> - Make the HMM invalidate wait uninterruptible, so a pending signal
>    does not let pages be freed while the NPU may still access them.
> - Record whether a command wait ended on completion or disconnect,
>    so a context unlinked and relinked during the wait cannot report
>    a false completion, and a torn read_index cannot stop the worker.
> - Resume the device in the wait ioctl when the context is suspended
>    by power management instead of returning -EAGAIN, and return 0
>    for commands completed before suspend.
> - Add a job timeout that stops the hardware context when a job does not
>    complete, so job fences published to dma_resv are always signaled in
>    finite time.
>
> Changes in v3 (addresses v2 review comments):
> - Lock, check and fence chained sub-command BOs with the job BOs,
>    since the NPU writes completion to command BO headers.
> - Use the validated payload pointer in submit_one_cmd() to fix a
>    TOCTOU on the user-writable command header.
> - Drop the last_read_index cache; get_read_index() returns 0 when
>    read_index is still invalid after a re-sample, so callers treat the
>    command as incomplete and keep waiting.
> - Align the runtime config message buffer to 4 bytes.
> - Move the HMM invalidate wait and populate_range into common GEM
>    code, and repopulate invalidated user mappings in AIE4 command
>    submission instead of failing the submit.
> - Drop the patch that finalized runtime PM before taking dev_lock on
>    removal; the PCI core keeps the device active during remove, so
>    there is no deadlock.
> - Use a stateless PF/VF device link, folded into the suspend and
>    resume patch, so it survives VF driver rebind such as vfio-pci.
> - Return -EBUSY from runtime suspend when a device has no
>    runtime_suspend callback, so the PM core keeps it active.
>
> Changes in v2:
> - Drop the redundant cert_feature_table configuration.
> - Call aie4_partition_fini() when aie4_restore_power_mode() fails, so
>    the partition is not leaked on hardware start failure.
> - Return -ETIME from best-effort firmware configuration steps, so a
>    mailbox timeout fails the probe instead of letting it succeed with
>    unresponsive firmware.
> - Fix a synchronize_srcu() deadlock in context teardown by stopping
>    the hardware context and waking all waiters before destroying it.
> - Use READ_ONCE() for read_index and for fields read from user-shared
>    command buffers, so checked values cannot change after validation.
> - Check that cert_comp is still linked in all command wait paths, so
>    -EAGAIN is only returned on a real disconnect.
> - Check that trace buffer addresses fit within 48 bits, as required by
>    the current CERT design, instead of silently truncating them.
> - Hold references to chained sub-command BOs until job release, so
>    userspace cannot free a BO while hardware may still access it.
> - Document that job fences are signaled when submission fails or the
>    hardware context is stopped, destroyed or suspended, and that job
>    timeout detection and recovery (TDR) will follow in a separate
>    change to guarantee the fence signaling in finite time.
> - Address use-after-free comments on fence dev_name().
> - Abort hardware contexts when resume fails to start the hardware,
>    so submitters waiting for a context to reconnect fail instead of
>    blocking forever.
> - Document that VF driver rebind requires re-enabling SR-IOV via
>    sriov_numvfs, which recreates the PF/VF PM device links.
> - Add pm_ptr() wrapper to fix the build with CONFIG_PM=n.
> - Remove hwctx_config for AIE4; use-after-free cases in the debug
>    buffer handling need more work.
> - Detect VF passthrough by comparing against the PF driver instead of
>    the module owner, which is NULL for built-in drivers.
>
> Changes in v1:
> - Firmware version 6.0: return IOMMU_PASID_INVALID from aie4_msg_pasid().
> - Power mode: consolidate cached power mode override restoration on
>    hardware start directly into the power mode introduction patch, and
>    respect user buffer size.
> - Transport hooks: fix header include to use <drm/amdxdna_accel.h>.
> - Command submission & fencing: fold fence timeline naming and unique
>    timeline context allocation into the command submission patch.
>
> Series structure:
> The patch layout is the same as v0 except for the v3 and v4 changes above;
> see the v0 cover letter for a per-patch overview.
>
> Testing:
> - Tested on AMD AIE4/NPU3 hardware in both classic and SR-IOV (PF/VF)
>    modes.
> - Verified kernel-mode command submission with direct and indirect
>    execution packets under concurrent workloads.
> - Verified system suspend/resume (S2idle/S3) during idle and active
>    command submission, and runtime autosuspend on the classic device.
> - Verified SR-IOV VF binding and execution.
> - Confirmed no regression on existing AIE2 devices (NPU1/NPU4).
>
>
> David Zhang (19):
>    accel/amdxdna: Rename NPU3 firmware files
>    accel/amdxdna: Remove mmap for doorbell
>    accel/amdxdna: Add CERT firmware version support
>    accel/amdxdna: Support firmware 6.0
>    accel/amdxdna: Add NPU3 classic device support
>    accel/amdxdna: Add AIE version query to aie4_get_info
>    accel/amdxdna: Add get and set power_mode for AIE4
>    accel/amdxdna: Add clock, DPM frequency, and resource info queries for
>      AIE4
>    accel/amdxdna: Add context switch hysteresis with debugfs control
>    accel/amdxdna: Refactor AIE4 hardware initialization sequence
>    accel/amdxdna: Decouple AIE4 doorbell and MSI-X notify transport hooks
>    accel/amdxdna: Implement AIE4 kernel queue lifecycle and memory layout
>    accel/amdxdna: Prepare for AIE4 command submission
>    accel/amdxdna: Move HMM invalidate wait into common GEM code
>    accel/amdxdna: Make populate_range common for AIE2 and AIE4
>    accel/amdxdna: Implement AIE4 command packet building and submission
>    accel/amdxdna: Enable AIE4 firmware logging to DRAM
>    accel/amdxdna: Implement AIE4 suspend and resume
>    accel/amdxdna: Implement runtime suspend and resume support
>
>   drivers/accel/amdxdna/aie.c             |   33 +-
>   drivers/accel/amdxdna/aie.h             |   47 +-
>   drivers/accel/amdxdna/aie2_ctx.c        |   91 +-
>   drivers/accel/amdxdna/aie2_message.c    |    4 +-
>   drivers/accel/amdxdna/aie2_pci.c        |   67 +-
>   drivers/accel/amdxdna/aie2_pci.h        |   40 +-
>   drivers/accel/amdxdna/aie2_pm.c         |   10 +-
>   drivers/accel/amdxdna/aie4_ctx.c        | 1129 +++++++++++++++++++++--
>   drivers/accel/amdxdna/aie4_host_queue.h |   72 +-
>   drivers/accel/amdxdna/aie4_message.c    |  245 +++++
>   drivers/accel/amdxdna/aie4_msg_priv.h   |  149 ++-
>   drivers/accel/amdxdna/aie4_pci.c        |  818 +++++++++++++++-
>   drivers/accel/amdxdna/aie4_pci.h        |  117 ++-
>   drivers/accel/amdxdna/aie4_sriov.c      |   66 +-
>   drivers/accel/amdxdna/amdxdna_ctx.c     |   72 +-
>   drivers/accel/amdxdna/amdxdna_ctx.h     |   28 +-
>   drivers/accel/amdxdna/amdxdna_debugfs.c |    3 +
>   drivers/accel/amdxdna/amdxdna_gem.c     |   87 +-
>   drivers/accel/amdxdna/amdxdna_gem.h     |    1 +
>   drivers/accel/amdxdna/amdxdna_pci_drv.c |   46 +-
>   drivers/accel/amdxdna/amdxdna_pci_drv.h |   20 +-
>   drivers/accel/amdxdna/amdxdna_pm.c      |   26 +
>   drivers/accel/amdxdna/amdxdna_pm.h      |    2 +
>   drivers/accel/amdxdna/amdxdna_sysfs.c   |    2 +-
>   drivers/accel/amdxdna/npu1_regs.c       |   19 +-
>   drivers/accel/amdxdna/npu3_regs.c       |  105 ++-
>   drivers/accel/amdxdna/npu4_regs.c       |   30 +-
>   27 files changed, 2922 insertions(+), 407 deletions(-)
>