Re: [PATCH printk v2] printk: Postpone kthread startup

[Derek Barbosa <debarbos@redhat.com>]

On Fri, Oct 16, 2026 at 05:06:27PM +0200, Petr Mladek wrote:
> On Fri 2026-10-16 09:36:41, Derek Barbosa wrote:
> > On Fri, Oct 16, 2026 at 11:00:34AM +0206, John Ogness wrote:
> > > There have been various bug reports related to SMP systems hanging
> > > on boot. The reports vary but seem to have some overlap with regard
> > > to missed IPI's during SMP setup. The reports also show that
> > > reverting NBCON console usage allows the bugs to disappear.
> > > 
> > > The NBCON kthreads for console printing are started very early
> > > during boot using early_initcall(). Since these kthreads are
> > > asynchronously enabling/disabling interrupts during early boot, they
> > > may be interfering with early SMP setup on some hardware platforms.
> > > 
> > > Starting the console printing kthreads later has shown to avoid the
> > > hanging on systems that could previously reproduce the issue.
> > > 
> > > Postpone kthread starting until after all arch initcalls have
> > > completed.
> > 
> > In the v1 thread, I mentioned that I would leave the reproducer spinning in the
> > test environment for a few more hours. It has been well over 9 hours. No signs
> > of the existing issue popping up.
> > 
> > Tested-by: Derek Barbosa <debarbos@redhat.com>
> 
> Thanks a lot for testing this. It is good to know that it helped.
> 
> I am a bit nervous that we do not know what is the culprit.
> For example, I wonder if the same problem might happen during
> CPU hot-plug at runtime.

FWIW, this fixes a corner-case that seems to affect older platforms that had
been noted to have "issues" with some workloads on older firmware/microcode.

For the cpu hot-plug example, I am unsure of how to properly stress/test that
condition to quell these concerns. I could run a modified version of John's old
console_blast script and take some CPUs on/off-line in a loop with:

    $ echo 1 > /sys/devices/system/cpu/cpu$N/online
    # do some delay here
    $ echo 0 > /sys/devices/system/cpu/cpu$N/online

but I don't think that does a good job of emulating the original bug, where
kicking on the console-printing-kthreads very early in the init process (prior
to smp's secondary cpu bringup) could interfere with SMP setup. I could always
give it a shot.

> 
> Anyway, I had a brain storming session with John and we were not able
> to find any hole in the nbcon API design. We did not find any location
> where it could cause deadlock. All busy loops are under
> con->device_lock(), aka port->lock. And the port lock is taken with
> IRQs disabled. And neither the con->device_lock() nor the busy loop
> is done in NMI.
> 
> So, this rather looks like an existing problem. And the parallel
> handling of the serial console just made it more likely to happen.

I also hold this to be true. Considering I haven't seen anything quite like this
(up until this point) even with older versions of the code, I am chalking it up
to some sort perfect-storm of bad interactions.

> 
> The converted uart 8250 is being used on many production systems
> which are running the RT kernel and got this change from the external
> RT patchset. And I have heard about this problem for the first time just
> a few weeks ago. This makes me believe that it affects only small
> amount of HW.
> 
> This change clearly helped. Which makes me even more confident
> that we could keep the uart 8250 driver converted in 7.3.
> I am going to commit this patch and create a pull request...
> 
> Best Regards,
> Petr
> 

Cheers,

-- 
Derek <debarbos@redhat.com>