[PATCH] arm64: smp: Signal EOI after handling IPI_CPU_STOP*

[PATCH] arm64: smp: Signal EOI after handling IPI_CPU_STOP*

[Andrea della Porta]

On kdump/kexec, the boot CPU triggers an IPI_CPU_STOP (and subsequently
an IPI_CPU_STOP_NMI if the first one does not respond) to the secondary
CPUs, and the IPI handler eventually calls the firmware to shut down
each CPU. Since the IPI is never acknowledged via EOI, the interrupt
remains in an active state when the CPU goes idle.

In a virtualized environment, if the hypervisor does not reset the
interrupt state, this causes the crashkernel (with cmdline option
maxcpus > 1) to be unable to synchronize between the boot CPU and
secondary CPUs via IPI_CALL_FUNC, leading the kernel to wait indefinitely
for the CPUs to respond. This has been observed with the Hyper-V
implementation.

Both the Linux kernel and the Hyper-V firmware appear to violate the
PSCI specification:

- The PSCI spec states that the OS kernel must migrate any interrupt away
from the CPU that is about to be shut down via CPU_OFF, which by extension
implies that no active interrupts are allowed. The kernel does not
currently do this in the kdump crash path.

- The PSCI spec also states that the PSCI firmware must reset the CPU
registers to their default values when turning on a CPU via the CPU_ON
command.

Fixing this on the kernel side has the advantage of being
hypervisor-agnostic.

Signal EOI at the end of the crash handler to prevent the subsequent
crashkernel from hanging.

Signed-off-by: Andrea della Porta <andrea.porta@suse.com>
---
 arch/arm64/kernel/smp.c | 22 ++++++++++++++++++++++
 1 file changed, 22 insertions(+)

diff --git a/arch/arm64/kernel/smp.c b/arch/arm64/kernel/smp.c
index a61dc3016a117..cc7f3261fe537 100644
--- a/arch/arm64/kernel/smp.c
+++ b/arch/arm64/kernel/smp.c
@@ -988,6 +988,27 @@ void kgdb_roundup_cpus(void)
 }
 #endif
 
+static void ipi_eoi(int ipinr)
+{
+	unsigned int cpu = smp_processor_id();
+	struct irq_desc *desc;
+	struct irq_chip *chip;
+	struct irq_data *d;
+
+	if (ipinr >= MAX_IPI)
+		return;
+
+	desc = get_ipi_desc(cpu, ipinr);
+
+	if (desc) {
+		chip = irq_desc_get_chip(desc);
+		d = irq_desc_get_irq_data(desc);
+
+		if (chip && chip->irq_eoi)
+			chip->irq_eoi(d);
+	}
+}
+
 /*
  * Main handler for inter-processor interrupts
  */
@@ -1009,6 +1030,7 @@ static void do_handle_IPI(int ipinr)
 
 	case IPI_CPU_STOP:
 	case IPI_CPU_STOP_NMI:
+		ipi_eoi(ipinr);
 		arm64_nmi_cpu_stop(get_irq_regs(), true);
 		break;
 
-- 
2.35.3

Re: [PATCH] arm64: smp: Signal EOI after handling IPI_CPU_STOP*

[Will Deacon]

[+Marc]

On Tue, Sep 22, 2026 at 04:01:09PM +0200, Andrea della Porta wrote:
> On kdump/kexec, the boot CPU triggers an IPI_CPU_STOP (and subsequently
> an IPI_CPU_STOP_NMI if the first one does not respond) to the secondary
> CPUs, and the IPI handler eventually calls the firmware to shut down
> each CPU. Since the IPI is never acknowledged via EOI, the interrupt
> remains in an active state when the CPU goes idle.
> 
> In a virtualized environment, if the hypervisor does not reset the
> interrupt state, this causes the crashkernel (with cmdline option
> maxcpus > 1) to be unable to synchronize between the boot CPU and
> secondary CPUs via IPI_CALL_FUNC, leading the kernel to wait indefinitely
> for the CPUs to respond. This has been observed with the Hyper-V
> implementation.

Hmm, how is this different from panic()ing inside an interrupt handler?

> Both the Linux kernel and the Hyper-V firmware appear to violate the
> PSCI specification:
> 
> - The PSCI spec states that the OS kernel must migrate any interrupt away
> from the CPU that is about to be shut down via CPU_OFF, which by extension
> implies that no active interrupts are allowed. The kernel does not
> currently do this in the kdump crash path.
> 
> - The PSCI spec also states that the PSCI firmware must reset the CPU
> registers to their default values when turning on a CPU via the CPU_ON
> command.
> 
> Fixing this on the kernel side has the advantage of being
> hypervisor-agnostic.
> 
> Signal EOI at the end of the crash handler to prevent the subsequent
> crashkernel from hanging.
> 
> Signed-off-by: Andrea della Porta <andrea.porta@suse.com>
> ---
>  arch/arm64/kernel/smp.c | 22 ++++++++++++++++++++++
>  1 file changed, 22 insertions(+)
> 
> diff --git a/arch/arm64/kernel/smp.c b/arch/arm64/kernel/smp.c
> index a61dc3016a117..cc7f3261fe537 100644
> --- a/arch/arm64/kernel/smp.c
> +++ b/arch/arm64/kernel/smp.c
> @@ -988,6 +988,27 @@ void kgdb_roundup_cpus(void)
>  }
>  #endif
>  
> +static void ipi_eoi(int ipinr)
> +{
> +	unsigned int cpu = smp_processor_id();
> +	struct irq_desc *desc;
> +	struct irq_chip *chip;
> +	struct irq_data *d;
> +
> +	if (ipinr >= MAX_IPI)
> +		return;
> +
> +	desc = get_ipi_desc(cpu, ipinr);
> +
> +	if (desc) {
> +		chip = irq_desc_get_chip(desc);
> +		d = irq_desc_get_irq_data(desc);
> +
> +		if (chip && chip->irq_eoi)
> +			chip->irq_eoi(d);
> +	}
> +}

Doesn't this hard-code the flow handler for the irqchip? It feels like it
would be better for the GIC driver to get a callback during the kexec
sequence (if it doesn't already) to prepare itself.

Will

>  /*
>   * Main handler for inter-processor interrupts
>   */
> @@ -1009,6 +1030,7 @@ static void do_handle_IPI(int ipinr)
>  
>  	case IPI_CPU_STOP:
>  	case IPI_CPU_STOP_NMI:
> +		ipi_eoi(ipinr);
>  		arm64_nmi_cpu_stop(get_irq_regs(), true);
>  		break;
>  
> -- 
> 2.35.3
> 

Re: [PATCH] arm64: smp: Signal EOI after handling IPI_CPU_STOP*

[Marc Zyngier]

On Wed, 23 Sep 2026 16:41:42 +0100,
Will Deacon <will@kernel.org> wrote:
> 
> [+Marc]
> 
> On Tue, Sep 22, 2026 at 04:01:09PM +0200, Andrea della Porta wrote:
> > On kdump/kexec, the boot CPU triggers an IPI_CPU_STOP (and subsequently
> > an IPI_CPU_STOP_NMI if the first one does not respond) to the secondary
> > CPUs, and the IPI handler eventually calls the firmware to shut down
> > each CPU. Since the IPI is never acknowledged via EOI, the interrupt
> > remains in an active state when the CPU goes idle.
> > 
> > In a virtualized environment, if the hypervisor does not reset the
> > interrupt state, this causes the crashkernel (with cmdline option
> > maxcpus > 1) to be unable to synchronize between the boot CPU and
> > secondary CPUs via IPI_CALL_FUNC, leading the kernel to wait indefinitely
> > for the CPUs to respond. This has been observed with the Hyper-V
> > implementation.
> 
> Hmm, how is this different from panic()ing inside an interrupt
> handler?

Also, what is the architectural requirement for the hypervisor to do
*anything* on the state of the interrupts? This doesn't happen on HW,
and there is no reason to impose this on a hypervisor either.

The expectations are that the guest kernel should reset the interrupt
state on boot, and it feels that this is missing somehow, see below.

> 
> > Both the Linux kernel and the Hyper-V firmware appear to violate the
> > PSCI specification:
> > 
> > - The PSCI spec states that the OS kernel must migrate any interrupt away
> > from the CPU that is about to be shut down via CPU_OFF, which by extension
> > implies that no active interrupts are allowed. The kernel does not
> > currently do this in the kdump crash path.

No. This is strictly about affinity, nothing else. It is there so that
global devices can continue to have their interrupts handled. This
evidently can't affect CPU-local interrupts.

> > 
> > - The PSCI spec also states that the PSCI firmware must reset the CPU
> > registers to their default values when turning on a CPU via the CPU_ON
> > command.

CPU registers. Not external peripherals such as the GIC.

> > 
> > Fixing this on the kernel side has the advantage of being
> > hypervisor-agnostic.
> > 
> > Signal EOI at the end of the crash handler to prevent the subsequent
> > crashkernel from hanging.
> > 
> > Signed-off-by: Andrea della Porta <andrea.porta@suse.com>
> > ---
> >  arch/arm64/kernel/smp.c | 22 ++++++++++++++++++++++
> >  1 file changed, 22 insertions(+)
> > 
> > diff --git a/arch/arm64/kernel/smp.c b/arch/arm64/kernel/smp.c
> > index a61dc3016a117..cc7f3261fe537 100644
> > --- a/arch/arm64/kernel/smp.c
> > +++ b/arch/arm64/kernel/smp.c
> > @@ -988,6 +988,27 @@ void kgdb_roundup_cpus(void)
> >  }
> >  #endif
> >  
> > +static void ipi_eoi(int ipinr)
> > +{
> > +	unsigned int cpu = smp_processor_id();
> > +	struct irq_desc *desc;
> > +	struct irq_chip *chip;
> > +	struct irq_data *d;
> > +
> > +	if (ipinr >= MAX_IPI)
> > +		return;
> > +
> > +	desc = get_ipi_desc(cpu, ipinr);
> > +
> > +	if (desc) {
> > +		chip = irq_desc_get_chip(desc);
> > +		d = irq_desc_get_irq_data(desc);
> > +
> > +		if (chip && chip->irq_eoi)
> > +			chip->irq_eoi(d);
> > +	}
> > +}
> 
> Doesn't this hard-code the flow handler for the irqchip? It feels like it
> would be better for the GIC driver to get a callback during the kexec
> sequence (if it doesn't already) to prepare itself.

Yeah, that's not an acceptable approach.

The other observation is that the GIC drivers already clear the active
state at boot time (gic_cpu_init()). So what isn't that working?

Could it be that the hypervisor doesn't correctly handle the writes to
GICR_ICACTIVER0 to nuke the active state? Because this works correctly
on KVM as is.

Thanks,

	M.


-- 
Without deviation from the norm, progress is not possible.

Re: [PATCH] arm64: smp: Signal EOI after handling IPI_CPU_STOP*

[Andrea della Porta]

Hi Will and Marc,

On 19:23 Wed 23 Sep     , Marc Zyngier wrote:
> On Wed, 23 Sep 2026 16:41:42 +0100,
> Will Deacon <will@kernel.org> wrote:
> > 
> > [+Marc]
> > 
> > On Tue, Sep 22, 2026 at 04:01:09PM +0200, Andrea della Porta wrote:
> > > On kdump/kexec, the boot CPU triggers an IPI_CPU_STOP (and subsequently
> > > an IPI_CPU_STOP_NMI if the first one does not respond) to the secondary
> > > CPUs, and the IPI handler eventually calls the firmware to shut down
> > > each CPU. Since the IPI is never acknowledged via EOI, the interrupt
> > > remains in an active state when the CPU goes idle.
> > > 
> > > In a virtualized environment, if the hypervisor does not reset the
> > > interrupt state, this causes the crashkernel (with cmdline option
> > > maxcpus > 1) to be unable to synchronize between the boot CPU and
> > > secondary CPUs via IPI_CALL_FUNC, leading the kernel to wait indefinitely
> > > for the CPUs to respond. This has been observed with the Hyper-V
> > > implementation.
> > 
> > Hmm, how is this different from panic()ing inside an interrupt
> > handler?

It is not, in fact I should have put the EOI in all cases, but I just
focused on the crash driver by IPI only. The patch should be extended
but from the following discussion it does not seem relevant anymore.

> 
> Also, what is the architectural requirement for the hypervisor to do
> *anything* on the state of the interrupts? This doesn't happen on HW,
> and there is no reason to impose this on a hypervisor either.
> 
> The expectations are that the guest kernel should reset the interrupt
> state on boot, and it feels that this is missing somehow, see below.

Ack.

> 
> > 
> > > Both the Linux kernel and the Hyper-V firmware appear to violate the
> > > PSCI specification:
> > > 
> > > - The PSCI spec states that the OS kernel must migrate any interrupt away
> > > from the CPU that is about to be shut down via CPU_OFF, which by extension
> > > implies that no active interrupts are allowed. The kernel does not
> > > currently do this in the kdump crash path.
> 
> No. This is strictly about affinity, nothing else. It is there so that
> global devices can continue to have their interrupts handled. This
> evidently can't affect CPU-local interrupts.

Noted.

> 
> > > 
> > > - The PSCI spec also states that the PSCI firmware must reset the CPU
> > > registers to their default values when turning on a CPU via the CPU_ON
> > > command.
> 
> CPU registers. Not external peripherals such as the GIC.

True. Those are distributor related.

> 
> > > 
> > > Fixing this on the kernel side has the advantage of being
> > > hypervisor-agnostic.
> > > 
> > > Signal EOI at the end of the crash handler to prevent the subsequent
> > > crashkernel from hanging.
> > > 
> > > Signed-off-by: Andrea della Porta <andrea.porta@suse.com>
> > > ---
> > >  arch/arm64/kernel/smp.c | 22 ++++++++++++++++++++++
> > >  1 file changed, 22 insertions(+)
> > > 
> > > diff --git a/arch/arm64/kernel/smp.c b/arch/arm64/kernel/smp.c
> > > index a61dc3016a117..cc7f3261fe537 100644
> > > --- a/arch/arm64/kernel/smp.c
> > > +++ b/arch/arm64/kernel/smp.c
> > > @@ -988,6 +988,27 @@ void kgdb_roundup_cpus(void)
> > >  }
> > >  #endif
> > >  
> > > +static void ipi_eoi(int ipinr)
> > > +{
> > > +	unsigned int cpu = smp_processor_id();
> > > +	struct irq_desc *desc;
> > > +	struct irq_chip *chip;
> > > +	struct irq_data *d;
> > > +
> > > +	if (ipinr >= MAX_IPI)
> > > +		return;
> > > +
> > > +	desc = get_ipi_desc(cpu, ipinr);
> > > +
> > > +	if (desc) {
> > > +		chip = irq_desc_get_chip(desc);
> > > +		d = irq_desc_get_irq_data(desc);
> > > +
> > > +		if (chip && chip->irq_eoi)
> > > +			chip->irq_eoi(d);
> > > +	}
> > > +}
> > 
> > Doesn't this hard-code the flow handler for the irqchip? It feels like it
> > would be better for the GIC driver to get a callback during the kexec
> > sequence (if it doesn't already) to prepare itself.
> 
> Yeah, that's not an acceptable approach.

We're in a non returning handler which is about to shutdown the CPU in a few
instructions, so I'm not sure how to callback into the GIC driver.

> 
> The other observation is that the GIC drivers already clear the active
> state at boot time (gic_cpu_init()). So what isn't that working?
> 
> Could it be that the hypervisor doesn't correctly handle the writes to
> GICR_ICACTIVER0 to nuke the active state? Because this works correctly
> on KVM as is.

That's what I suppose, but obviously I have no access to the implementation
so I canot add more. This is currently under investigation by the folks who
can, though...

So the bottom line is that we need to wait for a patch from the hypervisor 
vendor (and from any vendor that *could* suffer from the same issue), I guess?

Many thanks,
Andrea

> 
> Thanks,
> 
> 	M.
> 
> 
> -- 
> Without deviation from the norm, progress is not possible.

Re: [PATCH] arm64: smp: Signal EOI after handling IPI_CPU_STOP*

[Marc Zyngier]

On Thu, 24 Sep 2026 10:03:56 +0100,
Andrea della Porta <andrea.porta@suse.com> wrote:
> 
> Hi Will and Marc,
> 
> On 19:23 Wed 23 Sep     , Marc Zyngier wrote:
> > On Wed, 23 Sep 2026 16:41:42 +0100,
> > Will Deacon <will@kernel.org> wrote:
> > > 
> > > [+Marc]
> > > 
> > > On Tue, Sep 22, 2026 at 04:01:09PM +0200, Andrea della Porta wrote:
> > > > On kdump/kexec, the boot CPU triggers an IPI_CPU_STOP (and subsequently
> > > > an IPI_CPU_STOP_NMI if the first one does not respond) to the secondary
> > > > CPUs, and the IPI handler eventually calls the firmware to shut down
> > > > each CPU. Since the IPI is never acknowledged via EOI, the interrupt
> > > > remains in an active state when the CPU goes idle.
> > > > 
> > > > In a virtualized environment, if the hypervisor does not reset the
> > > > interrupt state, this causes the crashkernel (with cmdline option
> > > > maxcpus > 1) to be unable to synchronize between the boot CPU and
> > > > secondary CPUs via IPI_CALL_FUNC, leading the kernel to wait indefinitely
> > > > for the CPUs to respond. This has been observed with the Hyper-V
> > > > implementation.
> > > 
> > > Hmm, how is this different from panic()ing inside an interrupt
> > > handler?
> 
> It is not, in fact I should have put the EOI in all cases, but I just
> focused on the crash driver by IPI only. The patch should be extended
> but from the following discussion it does not seem relevant anymore.

The other problem here is that you're looking at the IPI that got you
there. But this IPI could have a higher priority than other interrupts
(NMI), and therefore have preempted them whilst they were active.

What happens with them? You'd need to iterate over all the interrupts
that are in progress and EOI/DIR them *in the correct order*. Which
would be replicating the GIC state machine. You can't do that in an
arbitrary order as that would violate the interrupt life cycle which
on some implementations results in a terminal SError.

[...]

> > >
> > > Doesn't this hard-code the flow handler for the irqchip? It feels like it
> > > would be better for the GIC driver to get a callback during the kexec
> > > sequence (if it doesn't already) to prepare itself.
> > 
> > Yeah, that's not an acceptable approach.
> 
> We're in a non returning handler which is about to shutdown the CPU in a few
> instructions, so I'm not sure how to callback into the GIC driver.

I reckon crash_kexec_post_notifiers and co could be of help, and could
be to some extent tucked away in the HV-specific code (but see below
for my full take on this).

> > The other observation is that the GIC drivers already clear the active
> > state at boot time (gic_cpu_init()). So what isn't that working?
> > 
> > Could it be that the hypervisor doesn't correctly handle the writes to
> > GICR_ICACTIVER0 to nuke the active state? Because this works correctly
> > on KVM as is.
> 
> That's what I suppose, but obviously I have no access to the implementation
> so I canot add more. This is currently under investigation by the folks who
> can, though...

OK. It'd be interesting to understand why the existing code fails in
your context, and getting feedback from the HV people would help.

> 
> So the bottom line is that we need to wait for a patch from the hypervisor 
> vendor (and from any vendor that *could* suffer from the same issue), I guess?

My take is that there is no point adding anything to the kernel until
we understand exactly what is at stake. The kernel's expectation is
that there is no difference between bare metal and virtualised, and
that fundamental parts of the architecture (such as interrupts) should
work as expected, no ifs, no buts.

If this isn't fixable, or that we don't know when this will be fixed,
we can always add an erratum workaround. But it needs to be captured
as such, which implies that we have the full understanding of the
issue.

Thanks,

	M.

-- 
Without deviation from the norm, progress is not possible.