Re: [PATCH 0/3] remoteproc: xlnx: remote crash recovery

[Peng Fan <peng.fan@oss.nxp.com>]

Hi Tanmay,

On Mon, Oct 27, 2025 at 09:57:28PM -0700, Tanmay Shah wrote:
>Remote processor can crash or hang during normal execution. Linux
>remoteproc framework supports different mechanisms to recover the
>remote processor and re-establish the RPMsg communication in such case.
>
>Crash reporting:
>
>1) Using debugfs node
>
>User can report the crash to the core framework via debugfs node using
>following command:
>
>echo 1 > /sys/kernel/debug/remoteproc/remoteproc0/crash
>
>2) Remoteproc notify to the host about crash state and crash reason
>via the resource table
>
>This is a platform specific method where the remote firmware contains
>vendor specific resource to update the crash state and the crash
>reason. Then the remote notifies the crash to the host via mailbox
>notification. The host then will check this resource on every mbox
>notification and reports the crash to the core framework if needed.
>
>Crash recovery mechanism:
>
>There are two mechanisms available to recover the remote processor from
>the crash. 1) boot recovery, 2) attach on recovery
>
>Remoteproc core framework will choose proper mechanism based on the
>rproc features set by the platform driver.
>
>1) Boot recovery
>
>This is the default mechanism to recover the remote processor.
>In this method core framework will first stop the remote processor,
>load the firmware again and then starts the remote processor. On
>AMD-Xilinx platforms this method is supported. The coredump callback in
>the platform driver isn't implemented so far, but that shouldn't cause
>the recovery failure.
>
>2) Attach on recovery
>
>If RPROC_ATTACH_ON_RECOVERY feature is enabled by the platform driver,
>then the core framework will choose this method for recovery.
>
>On zynqmp platform following is the sequence of events expected during
>remoteproc crash and attach on recovery:
>
>a) rproc attach/detach flow is working, and RPMsg comm is established
>b) Remote processor (RPU) crashed (crash not reported yet)
>c) Platform management controller stops and reloads elf on inactive
>   remote processor before reboot
>d) platform management controller reboots the remote processor
>e) Remote processor boots again, and detects previous crash (platform
>   specific mechanism to detect the crash)
>f) Remote processor Reports crash to the Linux (Host) and wait for
>   the recovery.
>g) Linux performs full detach and reattach to remote processor.
>h) Normal RPMsg communication is established.
>
>It is required to destroy all RPMsg related resource and re-create them
>during recovery to establish successful RPMsg communication. To achieve
>this complete rproc_detach followed by rproc_attach calls are needed.
>
>
>Tanmay Shah (3):
>  remoteproc: xlnx: enable boot recovery
>  remoteproc: core: full attach detach during recovery
>  remoteproc: xlnx: add crash detection mechanism
>

I gave a test on i.MX8QM-MEK, there are failures, 1st test pass, 2nd test fail.
Without this patch, I not see failures.
root@imx8qmmek:~#
remoteproc remoteproc0: crash detected in imx-rproc: type watchdog
Partition3 reset!
remoteproc remoteproc0: handling crash #1 in imx-rproc
remoteproc remoteproc0: detached remote processor imx-rproc
rproc-virtio rproc-virtio.1.auto: assigned reserved memory node vdevbuffer@90400000
virtio_rpmsg_bus virtio0: rpmsg host is online
rproc-virtio rproc-virtio.1.auto: registered virtio0 (type 7)
rproc-virtio rproc-virtio.2.auto: assigned reserved memory node vdevbuffer@90400000
virtio_rpmsg_bus virtio1: rpmsg host is online
rproc-virtio rproc-virtio.2.auto: registered virtio1 (type 7)
remoteproc remoteproc0: remote processor imx-rproc is now attached
virtio_rpmsg_bus virtio1: creating channel rpmsg-openamp-demo-channel addr 0x1e

remoteproc remoteproc0: crash detected in imx-rproc: type watchdog
Partition3 reset!
remoteproc remoteproc0: handling crash #2 in imx-rproc
rproc-virtio rproc-virtio.1.auto: assigned reserved memory node vdevbuffer@90400000
virtio_rpmsg_bus virtio4: probe with driver virtio_rpmsg_bus failed with error -12
rproc-virtio rproc-virtio.1.auto: registered virtio4 (type 7)
rproc-virtio rproc-virtio.2.auto: assigned reserved memory node vdevbuffer@90400000
virtio_rpmsg_bus virtio5: probe with driver virtio_rpmsg_bus failed with error -12
rproc-virtio rproc-virtio.2.auto: registered virtio5 (type 7)
rproc-virtio rproc-virtio.5.auto: assigned reserved memory node vdevbuffer@90400000
virtio_rpmsg_bus virtio6: probe with driver virtio_rpmsg_bus failed with error -12
rproc-virtio rproc-virtio.5.auto: registered virtio6 (type 7)
rproc-virtio rproc-virtio.6.auto: assigned reserved memory node vdevbuffer@90400000
virtio_rpmsg_bus virtio7: probe with driver virtio_rpmsg_bus failed with error -12
rproc-virtio rproc-virtio.6.auto: registered virtio7 (type 7)
remoteproc remoteproc0: remote processor imx-rproc is now attached

Thanks,
Peng