[REGRESSION] ACPI: threaded SCI handler causes soft lockup in kdump capture kernel

[Mohan Yelugoti <ymd@arista.com>]

Hi,

We ship SONiC[0] on top of Debian stable with a small set of patches
on top (see [1]). After moving SONiC from Debian bookworm
to trixie, we started seeing a regression where the kdump capture
kernel hits a soft lockup while collecting the crash dump, so no
vmcore is produced.

We trigger the original panic in the kernel with:

      echo c > /proc/sysrq-trigger

The capture kernel then boots and hangs. To capture a usable trace
rather than waiting indefinitely, I added the following to the
capture-kernel command line so that it panics on soft lockup (and
dumps backtraces from all CPUs) when the hang is detected:

      debug=1 loglevel=7 softlockup_all_cpu_backtrace=1
softlockup_panic=1

With those in place, the capture kernel prints:

      watchdog: BUG: soft lockup - CPU#0 stuck for 26s! [irq/9-acpi:39]
      CPU: 0 UID: 0 PID: 39 Comm: irq/9-acpi Not tainted
        6.12.41+deb13-sonic-amd64 #1 Debian 6.12.41-1
      RIP: 0010:acpi_os_read_port+0x30/0xa0
      Call Trace:
       <TASK>
       acpi_hw_gpe_read+0x61/0x80
       acpi_ev_detect_gpe+0x74/0x180
       acpi_ev_gpe_detect+0xe1/0x130
       acpi_ev_sci_xrupt_handler+0x1d/0x40
       acpi_irq+0x1c/0x40
       irq_thread_fn+0x23/0x60
       irq_thread+0x1b3/0x2f0
       kthread+0xd2/0x100
       ret_from_fork+0x34/0x50
       ret_from_fork_asm+0x1a/0x30
       </TASK>
      Kernel panic - not syncing: softlockup: hung tasks

Between the bookworm and trixie kernels, the ACPI SCI handler was
moved from a hardirq handler to a threaded handler by:

    7a36b901a6eb ("ACPI: OSL: Use a threaded interrupt handler for SCI")

To confirm this is the trigger, I reverted that commit on top of
6.12.41 and re-ran the same sysrq-trigger reproducer. The capture
kernel then completes the crash dump without tripping the soft
lockup watchdog.

Happy to test patches or gather more data as this is reproducible
consistently on our systems.

This issue has been raised in the SONiC project at [2].

  Context
  -------
- Kernel:        6.12.41+deb13-sonic-amd64 (Debian 6.12.41-1);
- Last known good: Debian bookworm kernel (6.1.123), predates
                     7a36b901a6eb.

- Default capture-kernel cmdline (from /etc/default/kdump-tools,
  KDUMP_CMDLINE_APPEND):

      irqpoll nr_cpus=1 nousb systemd.unit=kdump-tools.service
      ata_piix.prefer_ms_hyperv=0 panic=10 debug loglevel=4
      hpet=disable pcie_port=compat pci=nommconf

For the trace above, loglevel was raised from 4 to 7 and the
following were added:

     debug=1 softlockup_all_cpu_backtrace=1 softlockup_panic=1

[0]: https://github.com/sonic-net
[1]: https://github.com/sonic-net/sonic-linux-kernel/tree/master/patches-sonic
[2]: https://github.com/sonic-net/sonic-linux-kernel/pull/580