[PATCH 0/2] remoteproc: k3: Recover a crashed remote core

[PATCH 0/2] remoteproc: k3: Recover a crashed remote core

[Chrispine Tinega]

A K3 remote core can tell Linux it has crashed by sending RP_MBOX_CRASH,
but the K3 drivers only log it. Nothing reaches the remoteproc core, so
the crash is not counted and recovery never runs. The core stays halted
until the next boot, with its rpmsg channels dead.

Patch 1 reports the crash with rproc_report_crash(). This applies to
every K3 remote that uses the common mailbox callback.

Patch 2 leaves recovery enabled on the M4F. Its stop is a plain reset,
so recovery does not need the crashed core to answer. The R5F and DSP
drivers keep recovery disabled. A crashed core there moves to
RPROC_CRASHED, and can be stopped and started again through sysfs.

Tested on an AM625 board (BLIIoT BL350) running TI's 6.12 kernel with
the equivalent change. The M4F runs Zephyr, whose fatal-error handler
sends RP_MBOX_CRASH. After a forced fault, recovery reset and rebooted
the M4F in 71 ms, and Linux kept running. Mainline's M4F stop is the
same plain reset that this test used for a crashed core.

Chrispine Tinega (2):
  remoteproc: k3: Report a remote crash to the remoteproc core
  remoteproc: k3-m4: Leave crash recovery enabled

 drivers/remoteproc/ti_k3_common.c        | 5 +----
 drivers/remoteproc/ti_k3_m4_remoteproc.c | 1 -
 2 files changed, 1 insertion(+), 5 deletions(-)


base-commit: ec1f9ba57b02fe12a6fa5f141206d439ba3d044e
-- 
2.53.0

[PATCH 1/2] remoteproc: k3: Report a remote crash to the remoteproc core

[Chrispine Tinega]

A K3 remote core that faults can send RP_MBOX_CRASH over the mailbox,
but k3_rproc_mbox_callback() only logs it. The remoteproc core never
learns of the crash, so its recovery never runs, the crash is not
counted, and the core stays halted with every rpmsg channel to it dead.

Call rproc_report_crash(), which may be called from atomic context. The
core then moves to RPROC_CRASHED and the crash is counted. Where the
driver leaves recovery enabled, recovery stops the core and boots it
again. Where recovery is disabled, the core can still be stopped and
started again through sysfs.

Signed-off-by: Chrispine Tinega <dev@chrispinetinega.com>
---
 drivers/remoteproc/ti_k3_common.c | 5 +----
 1 file changed, 1 insertion(+), 4 deletions(-)

diff --git a/drivers/remoteproc/ti_k3_common.c b/drivers/remoteproc/ti_k3_common.c
index 4b6da3363..d799f377b 100644
--- a/drivers/remoteproc/ti_k3_common.c
+++ b/drivers/remoteproc/ti_k3_common.c
@@ -60,11 +60,8 @@ void k3_rproc_mbox_callback(struct mbox_client *client, void *data)
 
 	switch (msg) {
 	case RP_MBOX_CRASH:
-		/*
-		 * remoteproc detected an exception, but error recovery is not
-		 * supported. So, just log this for now
-		 */
 		dev_err(dev, "K3 rproc %s crashed\n", rproc->name);
+		rproc_report_crash(rproc, RPROC_FATAL_ERROR);
 		break;
 	case RP_MBOX_ECHO_REPLY:
 		dev_info(dev, "received echo reply from %s\n", rproc->name);
-- 
2.53.0

[PATCH 2/2] remoteproc: k3-m4: Leave crash recovery enabled

[Chrispine Tinega]

The M4 driver disables recovery, but nothing reported a crash to the
remoteproc core, so there was nothing to recover from. Now that
RP_MBOX_CRASH reaches rproc_report_crash(), recovery resets the M4F and
boots its firmware again, while Linux keeps running. The stop is a
plain reset, so it does not need the crashed core's cooperation.

Leave recovery enabled by default. Userspace can still disable it
through /sys/class/remoteproc/remoteprocN/recovery.

Signed-off-by: Chrispine Tinega <dev@chrispinetinega.com>
---
 drivers/remoteproc/ti_k3_m4_remoteproc.c | 1 -
 1 file changed, 1 deletion(-)

diff --git a/drivers/remoteproc/ti_k3_m4_remoteproc.c b/drivers/remoteproc/ti_k3_m4_remoteproc.c
index 4c46f4a25..fbd2b7c04 100644
--- a/drivers/remoteproc/ti_k3_m4_remoteproc.c
+++ b/drivers/remoteproc/ti_k3_m4_remoteproc.c
@@ -56,7 +56,6 @@ static int k3_m4_rproc_probe(struct platform_device *pdev)
 		return -ENOMEM;
 
 	rproc->has_iommu = false;
-	rproc->recovery_disabled = true;
 	kproc = rproc->priv;
 	kproc->dev = dev;
 	kproc->rproc = rproc;
-- 
2.53.0

Re: [PATCH 0/2] remoteproc: k3: Recover a crashed remote core

[Francesco Valla]

Hello Chrispine,

9 Nov 2026 06:49:26 Chrispine Tinega <dev@chrispinetinega.com>:

> A K3 remote core can tell Linux it has crashed by sending RP_MBOX_CRASH,
> but the K3 drivers only log it. Nothing reaches the remoteproc core, so
> the crash is not counted and recovery never runs. The core stays halted
> until the next boot, with its rpmsg channels dead.
>
> Patch 1 reports the crash with rproc_report_crash(). This applies to
> every K3 remote that uses the common mailbox callback.
>
> Patch 2 leaves recovery enabled on the M4F. Its stop is a plain reset,
> so recovery does not need the crashed core to answer. The R5F and DSP
> drivers keep recovery disabled. A crashed core there moves to
> RPROC_CRASHED, and can be stopped and started again through sysfs.
>
> Tested on an AM625 board (BLIIoT BL350) running TI's 6.12 kernel with
> the equivalent change. The M4F runs Zephyr, whose fatal-error handler
> sends RP_MBOX_CRASH. After a forced fault, recovery reset and rebooted
> the M4F in 71 ms, and Linux kept running. Mainline's M4F stop is the
> same plain reset that this test used for a crashed core.
>

If you share details on the Zephyr side of things, I can test this against rproc-next on a AM67A (BeagleY-AI).

Regards,
Francesco

> Chrispine Tinega (2):
>   remoteproc: k3: Report a remote crash to the remoteproc core
>   remoteproc: k3-m4: Leave crash recovery enabled
>
> drivers/remoteproc/ti_k3_common.c        | 5 +----
> drivers/remoteproc/ti_k3_m4_remoteproc.c | 1 -
> 2 files changed, 1 insertion(+), 5 deletions(-)
>
>
> base-commit: ec1f9ba57b02fe12a6fa5f141206d439ba3d044e
> --
> 2.53.0

Re: [PATCH 0/2] remoteproc: k3: Recover a crashed remote core

[Chrispine Tinega]

Hi Francesco,

On Mon, 9 Nov 2026, Francesco Valla wrote:
> If you share details on the Zephyr side of things, I can test this
> against rproc-next on a AM67A (BeagleY-AI).

Thank you, that would help. The AM67A has no M4F, so patch 2 does not
apply there. Patch 1 does: the R5F driver uses the same mailbox
callback, and it keeps recovery disabled. So on the main R5F I would
expect this:

- Without the series, the crash is only logged, and the core stays in
  "running" until the next boot.
- With the series, the core moves to "crashed", the crash is counted,
  and it can be stopped and started again through sysfs.

The Zephyr side is upstream's openamp_rsc_table sample with one file
added. The handler sends RP_MBOX_CRASH over the mailbox before it
halts, and a thread forces a fault 20 s after boot. This builds on
Zephyr v4.5.0 for beagley_ai/j722s/main_r5f0_0, but I have not run it
on that board.

Copy samples/subsys/ipc/openamp_rsc_table, add src/crash.c below, and
list it in CMakeLists.txt:

  target_sources(app PRIVATE src/main_remote.c src/crash.c)

Build it:

  west build -b beagley_ai/j722s/main_r5f0_0 path/to/the/copy

Load build/zephyr/zephyr_openamp_rsc_table.elf as the main R5F's
firmware. After 20 s, check the state:

  cat /sys/class/remoteproc/remoteprocN/state
  dmesg | grep -i crash

To bring the core back:

  echo stop > /sys/class/remoteproc/remoteprocN/state
  echo start > /sys/class/remoteproc/remoteprocN/state

It will crash again 20 s after each start. That is a quick way to check
that a second crash is handled as well.

--- src/crash.c ---
/*
 * Report a fatal error to Linux as RP_MBOX_CRASH, then force one.
 */
#include <zephyr/kernel.h>
#include <zephyr/drivers/ipm.h>
#include <zephyr/fatal.h>
#include <zephyr/logging/log_ctrl.h>

#define RP_MBOX_CRASH 0xFFFFFF02U
#define CRASH_AFTER_S 20

static const struct device *const crash_ipm =
	DEVICE_DT_GET(DT_CHOSEN(zephyr_ipc));

void k_sys_fatal_error_handler(unsigned int reason,
			       const struct arch_esf *esf)
{
	uint32_t crash = RP_MBOX_CRASH;

	ARG_UNUSED(reason);
	ARG_UNUSED(esf);

	LOG_PANIC();
	/* The FIFO can be briefly full of vring kicks. */
	for (int i = 0; i < 100; i++) {
		if (ipm_send(crash_ipm, 0, 0, &crash, sizeof(crash)) != -EBUSY)
			break;
		k_busy_wait(100);
	}
	(void)arch_irq_lock();
	for (;;)
		arch_cpu_idle();
}

static void crash_thread(void *a, void *b, void *c)
{
	k_sleep(K_SECONDS(CRASH_AFTER_S));
	k_oops();
}

K_THREAD_DEFINE(crash_tid, 1024, crash_thread, NULL, NULL, NULL,
		K_PRIO_PREEMPT(10), 0, 0);
--- end ---

Regards,
Chrispine

Re: [PATCH 0/2] remoteproc: k3: Recover a crashed remote core

[Francesco Valla]

Hi Chrispine,

On Mon, Nov 09, 2026 at 07:04:30PM +0300, Chrispine Tinega wrote:
> Hi Francesco,
> 
> On Mon, 9 Nov 2026, Francesco Valla wrote:
> > If you share details on the Zephyr side of things, I can test this
> > against rproc-next on a AM67A (BeagleY-AI).
> 
> Thank you, that would help. The AM67A has no M4F, so patch 2 does not
> apply there. Patch 1 does: the R5F driver uses the same mailbox
> callback, and it keeps recovery disabled. So on the main R5F I would
> expect this:
> 
> - Without the series, the crash is only logged, and the core stays in
>   "running" until the next boot.
> - With the series, the core moves to "crashed", the crash is counted,
>   and it can be stopped and started again through sysfs.
> 
> The Zephyr side is upstream's openamp_rsc_table sample with one file
> added. The handler sends RP_MBOX_CRASH over the mailbox before it
> halts, and a thread forces a fault 20 s after boot. This builds on
> Zephyr v4.5.0 for beagley_ai/j722s/main_r5f0_0, but I have not run it
> on that board.

I did run it on the "MCU" R5F, and I think I found a bug... in the Zephyr
ti_dmtimer code! With that fixed, the functionality seems to be working
as expected, but I'll stress it a bit before declaring a Tested-by.

> 
> Copy samples/subsys/ipc/openamp_rsc_table, add src/crash.c below, and
> list it in CMakeLists.txt:
> 
>   target_sources(app PRIVATE src/main_remote.c src/crash.c)
> 
> Build it:
> 
>   west build -b beagley_ai/j722s/main_r5f0_0 path/to/the/copy
> 
> Load build/zephyr/zephyr_openamp_rsc_table.elf as the main R5F's
> firmware. After 20 s, check the state:
> 
>   cat /sys/class/remoteproc/remoteprocN/state
>   dmesg | grep -i crash
> 
> To bring the core back:
> 
>   echo stop > /sys/class/remoteproc/remoteprocN/state
>   echo start > /sys/class/remoteproc/remoteprocN/state
> 
> It will crash again 20 s after each start. That is a quick way to check
> that a second crash is handled as well.
> 
> --- src/crash.c ---
> /*
>  * Report a fatal error to Linux as RP_MBOX_CRASH, then force one.
>  */
> #include <zephyr/kernel.h>
> #include <zephyr/drivers/ipm.h>
> #include <zephyr/fatal.h>
> #include <zephyr/logging/log_ctrl.h>
> 
> #define RP_MBOX_CRASH 0xFFFFFF02U
> #define CRASH_AFTER_S 20
> 
> static const struct device *const crash_ipm =
> 	DEVICE_DT_GET(DT_CHOSEN(zephyr_ipc));
> 
> void k_sys_fatal_error_handler(unsigned int reason,
> 			       const struct arch_esf *esf)
> {
> 	uint32_t crash = RP_MBOX_CRASH;
> 
> 	ARG_UNUSED(reason);
> 	ARG_UNUSED(esf);
> 
> 	LOG_PANIC();
> 	/* The FIFO can be briefly full of vring kicks. */
> 	for (int i = 0; i < 100; i++) {
> 		if (ipm_send(crash_ipm, 0, 0, &crash, sizeof(crash)) != -EBUSY)
> 			break;
> 		k_busy_wait(100);
> 	}
> 	(void)arch_irq_lock();
> 	for (;;)
> 		arch_cpu_idle();
> }
> 
> static void crash_thread(void *a, void *b, void *c)
> {
> 	k_sleep(K_SECONDS(CRASH_AFTER_S));
> 	k_oops();
> }
> 
> K_THREAD_DEFINE(crash_tid, 1024, crash_thread, NULL, NULL, NULL,
> 		K_PRIO_PREEMPT(10), 0, 0);
> --- end ---
> 
> Regards,
> Chrispine
> 

Regards,
Francesco