[BUG] NOHZ tick-stop error: local softirq work is pending

[BUG] NOHZ tick-stop error: local softirq work is pending

[Michal Pecio]

Hi,

I'm getting these warnings on current mainline. Quite reliable trigger
is I/O load (like md5sum /dev/sda) on an otherwise idle system:

[   86.082138] NOHZ tick-stop error: local softirq work is pending, handler #40!!!
[  117.959975] NOHZ tick-stop error: local softirq work is pending, handler #40!!!
[  121.054544] NOHZ tick-stop error: local softirq work is pending, handler #40!!!
[  121.988208] NOHZ tick-stop error: local softirq work is pending, handler #40!!!
[  143.932445] NOHZ tick-stop error: local softirq work is pending, handler #40!!!
[  156.083120] NOHZ tick-stop error: local softirq work is pending, handler #40!!!
[  156.685222] NOHZ tick-stop error: local softirq work is pending, handler #40!!!
[  156.789911] NOHZ tick-stop error: local softirq work is pending, handler #40!!!
[  158.986070] NOHZ tick-stop error: local softirq work is pending, handler #40!!!

I see it consistently across several reboots. I also did two v7.2
boots with the same workload and these ran cleanly 10 minutes each,
so it looks like a merge window regression.

The system is ordinary x86-64 with CONFIG_PREEMPT.

grep HZ .config: 

CONFIG_NO_HZ_COMMON=y
# CONFIG_HZ_PERIODIC is not set
CONFIG_NO_HZ_IDLE=y
# CONFIG_NO_HZ_FULL is not set
# CONFIG_NO_HZ is not set
# CONFIG_HZ_100 is not set
# CONFIG_HZ_250 is not set
# CONFIG_HZ_300 is not set
CONFIG_HZ_1000=y
CONFIG_HZ=1000

It seems bisectable, but maybe somebody has better ideas?

Regards,
Michal

Re: [BUG] NOHZ tick-stop error: local softirq work is pending

[Thomas Gleixner]

On Sat, Aug 29 2026 at 20:14, Michal Pecio wrote:
> I'm getting these warnings on current mainline. Quite reliable trigger
> is I/O load (like md5sum /dev/sda) on an otherwise idle system:
>
> [   86.082138] NOHZ tick-stop error: local softirq work is pending, handler #40!!!
> [  117.959975] NOHZ tick-stop error: local softirq work is pending, handler #40!!!
> [  121.054544] NOHZ tick-stop error: local softirq work is pending, handler #40!!!
> [  121.988208] NOHZ tick-stop error: local softirq work is pending, handler #40!!!
> [  143.932445] NOHZ tick-stop error: local softirq work is pending, handler #40!!!
> [  156.083120] NOHZ tick-stop error: local softirq work is pending, handler #40!!!
> [  156.685222] NOHZ tick-stop error: local softirq work is pending, handler #40!!!
> [  156.789911] NOHZ tick-stop error: local softirq work is pending, handler #40!!!
> [  158.986070] NOHZ tick-stop error: local softirq work is pending, handler #40!!!
>
> I see it consistently across several reboots. I also did two v7.2
> boots with the same workload and these ran cleanly 10 minutes each,
> so it looks like a merge window regression.
>
>
> It seems bisectable, but maybe somebody has better ideas?

It's likely the issue being discussed here:

   https://lore.kernel.org/lkml/20260824105523.GA4121620@noisy.programming.kicks-ass.net/

There is a revert queued which probably will cure it for you unless you
are using Rust drivers. Plus a fix for the remaining Rust specific issue
is on the way.

Thanks,

        tglx

Re: [BUG] NOHZ tick-stop error: local softirq work is pending

[Michal Pecio]

On Sun, 30 Aug 2026 07:56:49 +0200, Thomas Gleixner wrote:
> On Sat, Aug 29 2026 at 20:14, Michal Pecio wrote:
> > I'm getting these warnings on current mainline. Quite reliable trigger
> > is I/O load (like md5sum /dev/sda) on an otherwise idle system:
> >
> > [   86.082138] NOHZ tick-stop error: local softirq work is pending, handler #40!!!
> > [  117.959975] NOHZ tick-stop error: local softirq work is pending, handler #40!!!
> > [  121.054544] NOHZ tick-stop error: local softirq work is pending, handler #40!!!
> > [  121.988208] NOHZ tick-stop error: local softirq work is pending, handler #40!!!
> > [  143.932445] NOHZ tick-stop error: local softirq work is pending, handler #40!!!
> > [  156.083120] NOHZ tick-stop error: local softirq work is pending, handler #40!!!
> > [  156.685222] NOHZ tick-stop error: local softirq work is pending, handler #40!!!
> > [  156.789911] NOHZ tick-stop error: local softirq work is pending, handler #40!!!
> > [  158.986070] NOHZ tick-stop error: local softirq work is pending, handler #40!!!
> >
> > I see it consistently across several reboots. I also did two v7.2
> > boots with the same workload and these ran cleanly 10 minutes each,
> > so it looks like a merge window regression.
> >
> >
> > It seems bisectable, but maybe somebody has better ideas?  
> 
> It's likely the issue being discussed here:
> 
>    https://lore.kernel.org/lkml/20260824105523.GA4121620@noisy.programming.kicks-ass.net/
> 
> There is a revert queued which probably will cure it for you unless
> you are using Rust drivers. Plus a fix for the remaining Rust
> specific issue is on the way.

Thank you, cherry picking 46094a7708b7 from tip solved this.

And no, I absolutely loathe this cargo cult and it saddens me to see
people's time and CPU cycles being wasted on converting the kernel to
idiomatic C++ with its arbitrary limitations.

Surely all those irq_disable counters in Linux data centers around the
world will have a carbon footprint, even if (hypothetically) they are
"free" on superscalar CPUs with branch prediction?

And all of that for the sake of potential bugs which may or may not
exist, and which look like they could be found (or proven absence of)
by static analysis if required, and because of an experiment 15 years
which showed that static analysis beats reference counting?

Could only be more ironic if Moore's law were approaching a wall and
energy costs were rising everywhere with no end in sight...

Have a nice Sunday everyone,

Michal