Need inputs on hard lockups with "cat large_file.txt" use-case on 6.6 kernel

[Naman Jain <namjain@linux.microsoft.com>]

Hi,
I am debugging a hard-lockup issue which is seen when I do "cat" of a 
large text file (~3 MB or more) on my arm64 machine, which is based on 
6.6.116 kernel version and is using pl011 driver for UART.

I came across NBCON framework which tries to deal with such problems, 
but it is available on later kernels, and it would not be easy for me to 
do kernel upgrade at this moment.

I wanted to reach out here for any inputs that would help me debug this 
issue further - whether such lockups are expected, any kernel parameter, 
or patch that could fix this, etc.

Thanks in advance.

More details below:


I tried below experiments, but nothing worked:
* changing interrupt affinity of UART irq to other CPUs or group of 
CPUs, since all the UART interrupts were landing on CPU0.
* finding kernel cmdline parameters that can be tweaked - didn't seem to 
be related to "cat largefile" usecase, but for controlling general 
logging from drivers.
* Adding touch_nmi_watchdog(), cpu_relax() in different paths in the driver.
* adding handle_sysrq('l') in hardlockup handler to get CPU call stacks, 
or enable hardlockup_all_cpu_backtrace, but did not get call stacks of 
stuck CPUs.


Kernel cmdline:

console=ttyAMA0 earlycon=pl011,mmio32,0xFF000000,115200n8 reboot=w 
ima_hash=sha384 systemd.show_status=false ovlboot.bootdevice=E-MMC 
root=/dev/ram rw sysctl.kernel.printk_ratelimit=30 
sysctl.kernel.printk_ratelimit_burst=75 ima_policy=critical_data 
rcupdate.rcu_expedited=1 systemd.gpt_auto=no systemd.ssh_auto=no fips=1 
SYSTEMD_DEFAULT_MOUNT_RATE_LIMIT_BURST=100 systemd.log_ratelimit_kmsg=0 
crashkernel=512M


root@localhost:~# cat /proc/sys/kernel/printk
2       1       1       2


DT:
        serial0: serial@f1920000 {
                 bootph-all;
                 compatible = "arm,pl011", "arm,primecell";
                 status = "disabled";   -> enabled later in board
                 reg = <0 0xf1920000 0 0x1000>;
                 interrupts = <0 25 4>;
                 reg-io-width = <4>;
                 clock-names = "uartclk", "apb_pclk";
                 current-speed = <115200>;
         };

Regards,
Naman