[REGRESSION] ACPI: threaded SCI handler causes soft lockup in kdump capture kernel

[REGRESSION] ACPI: threaded SCI handler causes soft lockup in kdump capture kernel

[Mohan Yelugoti]

Hi,

We ship SONiC[0] on top of Debian stable with a small set of patches
on top (see [1]). After moving SONiC from Debian bookworm
to trixie, we started seeing a regression where the kdump capture
kernel hits a soft lockup while collecting the crash dump, so no
vmcore is produced.

We trigger the original panic in the kernel with:

      echo c > /proc/sysrq-trigger

The capture kernel then boots and hangs. To capture a usable trace
rather than waiting indefinitely, I added the following to the
capture-kernel command line so that it panics on soft lockup (and
dumps backtraces from all CPUs) when the hang is detected:

      debug=1 loglevel=7 softlockup_all_cpu_backtrace=1
softlockup_panic=1

With those in place, the capture kernel prints:

      watchdog: BUG: soft lockup - CPU#0 stuck for 26s! [irq/9-acpi:39]
      CPU: 0 UID: 0 PID: 39 Comm: irq/9-acpi Not tainted
        6.12.41+deb13-sonic-amd64 #1 Debian 6.12.41-1
      RIP: 0010:acpi_os_read_port+0x30/0xa0
      Call Trace:
       <TASK>
       acpi_hw_gpe_read+0x61/0x80
       acpi_ev_detect_gpe+0x74/0x180
       acpi_ev_gpe_detect+0xe1/0x130
       acpi_ev_sci_xrupt_handler+0x1d/0x40
       acpi_irq+0x1c/0x40
       irq_thread_fn+0x23/0x60
       irq_thread+0x1b3/0x2f0
       kthread+0xd2/0x100
       ret_from_fork+0x34/0x50
       ret_from_fork_asm+0x1a/0x30
       </TASK>
      Kernel panic - not syncing: softlockup: hung tasks

Between the bookworm and trixie kernels, the ACPI SCI handler was
moved from a hardirq handler to a threaded handler by:

    7a36b901a6eb ("ACPI: OSL: Use a threaded interrupt handler for SCI")

To confirm this is the trigger, I reverted that commit on top of
6.12.41 and re-ran the same sysrq-trigger reproducer. The capture
kernel then completes the crash dump without tripping the soft
lockup watchdog.

Happy to test patches or gather more data as this is reproducible
consistently on our systems.

This issue has been raised in the SONiC project at [2].

  Context
  -------
- Kernel:        6.12.41+deb13-sonic-amd64 (Debian 6.12.41-1);
- Last known good: Debian bookworm kernel (6.1.123), predates
                     7a36b901a6eb.

- Default capture-kernel cmdline (from /etc/default/kdump-tools,
  KDUMP_CMDLINE_APPEND):

      irqpoll nr_cpus=1 nousb systemd.unit=kdump-tools.service
      ata_piix.prefer_ms_hyperv=0 panic=10 debug loglevel=4
      hpet=disable pcie_port=compat pci=nommconf

For the trace above, loglevel was raised from 4 to 7 and the
following were added:

     debug=1 softlockup_all_cpu_backtrace=1 softlockup_panic=1

[0]: https://github.com/sonic-net
[1]: https://github.com/sonic-net/sonic-linux-kernel/tree/master/patches-sonic
[2]: https://github.com/sonic-net/sonic-linux-kernel/pull/580

Re: [REGRESSION] ACPI: threaded SCI handler causes soft lockup in kdump capture kernel

[Rafael J. Wysocki]

On Monday, June 1, 2026 10:29:08 PM CEST Mohan Yelugoti wrote:
> Hi,
> 
> We ship SONiC[0] on top of Debian stable with a small set of patches
> on top (see [1]). After moving SONiC from Debian bookworm
> to trixie, we started seeing a regression where the kdump capture
> kernel hits a soft lockup while collecting the crash dump, so no
> vmcore is produced.
> 
> We trigger the original panic in the kernel with:
> 
>       echo c > /proc/sysrq-trigger
> 
> The capture kernel then boots and hangs. To capture a usable trace
> rather than waiting indefinitely, I added the following to the
> capture-kernel command line so that it panics on soft lockup (and
> dumps backtraces from all CPUs) when the hang is detected:
> 
>       debug=1 loglevel=7 softlockup_all_cpu_backtrace=1 softlockup_panic=1
> 
> With those in place, the capture kernel prints:
> 
>       watchdog: BUG: soft lockup - CPU#0 stuck for 26s! [irq/9-acpi:39]
>       CPU: 0 UID: 0 PID: 39 Comm: irq/9-acpi Not tainted
>         6.12.41+deb13-sonic-amd64 #1 Debian 6.12.41-1
>       RIP: 0010:acpi_os_read_port+0x30/0xa0

I have no idea what possibly may lock up in that function, sorry.

In this particular case it is reading a byte from the I/O space via inb(),
that's all.

I'm wondering if it gets interrupted for some reason and then something
odd happens that prevents it from continuing.

>       Call Trace:
>        <TASK>
>        acpi_hw_gpe_read+0x61/0x80
>        acpi_ev_detect_gpe+0x74/0x180
>        acpi_ev_gpe_detect+0xe1/0x130
>        acpi_ev_sci_xrupt_handler+0x1d/0x40
>        acpi_irq+0x1c/0x40
>        irq_thread_fn+0x23/0x60
>        irq_thread+0x1b3/0x2f0
>        kthread+0xd2/0x100
>        ret_from_fork+0x34/0x50
>        ret_from_fork_asm+0x1a/0x30
>        </TASK>
>       Kernel panic - not syncing: softlockup: hung tasks
> 
> Between the bookworm and trixie kernels, the ACPI SCI handler was
> moved from a hardirq handler to a threaded handler by:
> 
>     7a36b901a6eb ("ACPI: OSL: Use a threaded interrupt handler for SCI")

Maybe it exposed some latent issue.

It's too late to revert it from the mainline anyway.

> To confirm this is the trigger, I reverted that commit on top of
> 6.12.41 and re-ran the same sysrq-trigger reproducer. The capture
> kernel then completes the crash dump without tripping the soft
> lockup watchdog.
> 
> Happy to test patches or gather more data as this is reproducible
> consistently on our systems.

Does it always occur at the same spot?  If so, you could try to
instrument the code around it and look for clues.

> This issue has been raised in the SONiC project at [2].
> 
>   Context
>   -------
> - Kernel:        6.12.41+deb13-sonic-amd64 (Debian 6.12.41-1);
> - Last known good: Debian bookworm kernel (6.1.123), predates
>                      7a36b901a6eb.
> 
> - Default capture-kernel cmdline (from /etc/default/kdump-tools,
>   KDUMP_CMDLINE_APPEND):
> 
>       irqpoll nr_cpus=1 nousb systemd.unit=kdump-tools.service
>       ata_piix.prefer_ms_hyperv=0 panic=10 debug loglevel=4
>       hpet=disable pcie_port=compat pci=nommconf
> 
> For the trace above, loglevel was raised from 4 to 7 and the
> following were added:
> 
>      debug=1 softlockup_all_cpu_backtrace=1 softlockup_panic=1
> 
> [0]: https://github.com/sonic-net
> [1]: https://github.com/sonic-net/sonic-linux-kernel/tree/master/patches-sonic
> [2]: https://github.com/sonic-net/sonic-linux-kernel/pull/580
> 
> 

Re: [REGRESSION] ACPI: threaded SCI handler causes soft lockup in kdump capture kernel

[Mario Limonciello]

On 6/2/26 12:17, Rafael J. Wysocki wrote:
> On Monday, June 1, 2026 10:29:08 PM CEST Mohan Yelugoti wrote:
>> Hi,
>>
>> We ship SONiC[0] on top of Debian stable with a small set of patches
>> on top (see [1]). After moving SONiC from Debian bookworm
>> to trixie, we started seeing a regression where the kdump capture
>> kernel hits a soft lockup while collecting the crash dump, so no
>> vmcore is produced.
>>
>> We trigger the original panic in the kernel with:
>>
>>        echo c > /proc/sysrq-trigger
>>
>> The capture kernel then boots and hangs. To capture a usable trace
>> rather than waiting indefinitely, I added the following to the
>> capture-kernel command line so that it panics on soft lockup (and
>> dumps backtraces from all CPUs) when the hang is detected:
>>
>>        debug=1 loglevel=7 softlockup_all_cpu_backtrace=1 softlockup_panic=1
>>
>> With those in place, the capture kernel prints:
>>
>>        watchdog: BUG: soft lockup - CPU#0 stuck for 26s! [irq/9-acpi:39]
>>        CPU: 0 UID: 0 PID: 39 Comm: irq/9-acpi Not tainted
>>          6.12.41+deb13-sonic-amd64 #1 Debian 6.12.41-1
>>        RIP: 0010:acpi_os_read_port+0x30/0xa0
> 
> I have no idea what possibly may lock up in that function, sorry.
> 
> In this particular case it is reading a byte from the I/O space via inb(),
> that's all.
> 
> I'm wondering if it gets interrupted for some reason and then something
> odd happens that prevents it from continuing.
> 
>>        Call Trace:
>>         <TASK>
>>         acpi_hw_gpe_read+0x61/0x80
>>         acpi_ev_detect_gpe+0x74/0x180
>>         acpi_ev_gpe_detect+0xe1/0x130
>>         acpi_ev_sci_xrupt_handler+0x1d/0x40
>>         acpi_irq+0x1c/0x40
>>         irq_thread_fn+0x23/0x60
>>         irq_thread+0x1b3/0x2f0
>>         kthread+0xd2/0x100
>>         ret_from_fork+0x34/0x50
>>         ret_from_fork_asm+0x1a/0x30
>>         </TASK>
>>        Kernel panic - not syncing: softlockup: hung tasks
>>
>> Between the bookworm and trixie kernels, the ACPI SCI handler was
>> moved from a hardirq handler to a threaded handler by:
>>
>>      7a36b901a6eb ("ACPI: OSL: Use a threaded interrupt handler for SCI")
> 
> Maybe it exposed some latent issue.
> 
> It's too late to revert it from the mainline anyway.
> 
>> To confirm this is the trigger, I reverted that commit on top of
>> 6.12.41 and re-ran the same sysrq-trigger reproducer. The capture
>> kernel then completes the crash dump without tripping the soft
>> lockup watchdog.
>>
>> Happy to test patches or gather more data as this is reproducible
>> consistently on our systems.
> 
> Does it always occur at the same spot?  If so, you could try to
> instrument the code around it and look for clues.

Especially if it's just a latent exposed issue before spending too much 
time intrumenting it's probably worth cross referencing 7.1-rc6 as well 
to see if the same issue can occur.

> 
>> This issue has been raised in the SONiC project at [2].
>>
>>    Context
>>    -------
>> - Kernel:        6.12.41+deb13-sonic-amd64 (Debian 6.12.41-1);
>> - Last known good: Debian bookworm kernel (6.1.123), predates
>>                       7a36b901a6eb.
>>
>> - Default capture-kernel cmdline (from /etc/default/kdump-tools,
>>    KDUMP_CMDLINE_APPEND):
>>
>>        irqpoll nr_cpus=1 nousb systemd.unit=kdump-tools.service
>>        ata_piix.prefer_ms_hyperv=0 panic=10 debug loglevel=4
>>        hpet=disable pcie_port=compat pci=nommconf
>>
>> For the trace above, loglevel was raised from 4 to 7 and the
>> following were added:
>>
>>       debug=1 softlockup_all_cpu_backtrace=1 softlockup_panic=1
>>
>> [0]: https://github.com/sonic-net
>> [1]: https://github.com/sonic-net/sonic-linux-kernel/tree/master/patches-sonic
>> [2]: https://github.com/sonic-net/sonic-linux-kernel/pull/580
>>
>>
> 
> 
> 
> 

Re: [REGRESSION] ACPI: threaded SCI handler causes soft lockup in kdump capture kernel

[Rafael J. Wysocki]

On Tue, Jun 2, 2026 at 7:20 PM Mario Limonciello
<mario.limonciello@amd.com> wrote:
> On 6/2/26 12:17, Rafael J. Wysocki wrote:
> > On Monday, June 1, 2026 10:29:08 PM CEST Mohan Yelugoti wrote:
> >> Hi,
> >>
> >> We ship SONiC[0] on top of Debian stable with a small set of patches
> >> on top (see [1]). After moving SONiC from Debian bookworm
> >> to trixie, we started seeing a regression where the kdump capture
> >> kernel hits a soft lockup while collecting the crash dump, so no
> >> vmcore is produced.
> >>
> >> We trigger the original panic in the kernel with:
> >>
> >>        echo c > /proc/sysrq-trigger
> >>
> >> The capture kernel then boots and hangs. To capture a usable trace
> >> rather than waiting indefinitely, I added the following to the
> >> capture-kernel command line so that it panics on soft lockup (and
> >> dumps backtraces from all CPUs) when the hang is detected:
> >>
> >>        debug=1 loglevel=7 softlockup_all_cpu_backtrace=1 softlockup_panic=1
> >>
> >> With those in place, the capture kernel prints:
> >>
> >>        watchdog: BUG: soft lockup - CPU#0 stuck for 26s! [irq/9-acpi:39]
> >>        CPU: 0 UID: 0 PID: 39 Comm: irq/9-acpi Not tainted
> >>          6.12.41+deb13-sonic-amd64 #1 Debian 6.12.41-1
> >>        RIP: 0010:acpi_os_read_port+0x30/0xa0
> >
> > I have no idea what possibly may lock up in that function, sorry.
> >
> > In this particular case it is reading a byte from the I/O space via inb(),
> > that's all.
> >
> > I'm wondering if it gets interrupted for some reason and then something
> > odd happens that prevents it from continuing.
> >
> >>        Call Trace:
> >>         <TASK>
> >>         acpi_hw_gpe_read+0x61/0x80
> >>         acpi_ev_detect_gpe+0x74/0x180
> >>         acpi_ev_gpe_detect+0xe1/0x130
> >>         acpi_ev_sci_xrupt_handler+0x1d/0x40
> >>         acpi_irq+0x1c/0x40
> >>         irq_thread_fn+0x23/0x60
> >>         irq_thread+0x1b3/0x2f0
> >>         kthread+0xd2/0x100
> >>         ret_from_fork+0x34/0x50
> >>         ret_from_fork_asm+0x1a/0x30
> >>         </TASK>
> >>        Kernel panic - not syncing: softlockup: hung tasks
> >>
> >> Between the bookworm and trixie kernels, the ACPI SCI handler was
> >> moved from a hardirq handler to a threaded handler by:
> >>
> >>      7a36b901a6eb ("ACPI: OSL: Use a threaded interrupt handler for SCI")
> >
> > Maybe it exposed some latent issue.
> >
> > It's too late to revert it from the mainline anyway.
> >
> >> To confirm this is the trigger, I reverted that commit on top of
> >> 6.12.41 and re-ran the same sysrq-trigger reproducer. The capture
> >> kernel then completes the crash dump without tripping the soft
> >> lockup watchdog.
> >>
> >> Happy to test patches or gather more data as this is reproducible
> >> consistently on our systems.
> >
> > Does it always occur at the same spot?  If so, you could try to
> > instrument the code around it and look for clues.
>
> Especially if it's just a latent exposed issue before spending too much
> time intrumenting it's probably worth cross referencing 7.1-rc6 as well
> to see if the same issue can occur.

Good idea.

> >
> >> This issue has been raised in the SONiC project at [2].
> >>
> >>    Context
> >>    -------
> >> - Kernel:        6.12.41+deb13-sonic-amd64 (Debian 6.12.41-1);
> >> - Last known good: Debian bookworm kernel (6.1.123), predates
> >>                       7a36b901a6eb.
> >>
> >> - Default capture-kernel cmdline (from /etc/default/kdump-tools,
> >>    KDUMP_CMDLINE_APPEND):
> >>
> >>        irqpoll nr_cpus=1 nousb systemd.unit=kdump-tools.service
> >>        ata_piix.prefer_ms_hyperv=0 panic=10 debug loglevel=4
> >>        hpet=disable pcie_port=compat pci=nommconf

Also, I would try to remove irqpoll from this command line and retest.

> >> For the trace above, loglevel was raised from 4 to 7 and the
> >> following were added:
> >>
> >>       debug=1 softlockup_all_cpu_backtrace=1 softlockup_panic=1
> >>
> >> [0]: https://github.com/sonic-net
> >> [1]: https://github.com/sonic-net/sonic-linux-kernel/tree/master/patches-sonic
> >> [2]: https://github.com/sonic-net/sonic-linux-kernel/pull/580