[PATCH tty 0/1] Make 8250 deferred MSR handling LAZY

[PATCH tty 0/1] Make 8250 deferred MSR handling LAZY

[John Ogness]

There was a report and follow-up discussion [0] about a problem
using regular irq_work queuing on some platforms when entering
cpuidle. To circumvent this problem, all irq_work for printk will
become IRQ_WORK_LAZY [1].

The 8250 console driver uses irq_work to defer MSR handling during
atomic printing. This means it is vulnerable to the same problem if,
for example, a WARN() is hit while entering cpuidle on one of the
affected platforms.

Avoid the problem for deferred MSR handling by changing
console_msr_work to IRQ_WORK_LAZY.

[0] https://lore.kernel.org/lkml/f3757a75-0ba1-4558-bf57-f19ab7e59a4c@nvidia.com
[1] https://lore.kernel.org/lkml/20260828140218.232439-1-john.ogness@linutronix.de

John Ogness (1):
  serial: 8250: Change console_msr_work to IRQ_WORK_LAZY

 drivers/tty/serial/8250/8250_port.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)


base-commit: 45c13f3f9e3bb15fd89ff2864c6f627a3b4b4229
-- 
2.47.3

[PATCH tty 1/1] serial: 8250: Change console_msr_work to IRQ_WORK_LAZY

[John Ogness]

For some platforms it is a problem to queue irq_work when entering
cpuidle states. Since irq_work is used for deferred MSR handling,
any atomic console printing when entering cpuidle states can lead
to the affected hardware hanging. Tegra20 and Tegra30 are examples
of such platforms. Avoiding raising the irq_work IRQ has shown to
circumvent the problem.

Change the console_msr_work to be IRQ_WORK_LAZY, thus not raising
an IRQ upon irq_work queuing. The irq_work is then handled on the
next interrupt (worst case, kernel tick).

Link: https://lore.kernel.org/lkml/f3757a75-0ba1-4558-bf57-f19ab7e59a4c@nvidia.com
Fixes: d3539347022a ("serial: 8250: Switch to nbcon console, take 2")
Signed-off-by: John Ogness <john.ogness@linutronix.de>
---
 drivers/tty/serial/8250/8250_port.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/drivers/tty/serial/8250/8250_port.c b/drivers/tty/serial/8250/8250_port.c
index 38fa45e74a37a..1acb73e45f8a0 100644
--- a/drivers/tty/serial/8250/8250_port.c
+++ b/drivers/tty/serial/8250/8250_port.c
@@ -3597,7 +3597,7 @@ int serial8250_console_setup(struct uart_port *port, char *options, bool probe)
 
 	up->console_line_ended = true;
 	up->console_msr_work_allow = true;
-	init_irq_work(&up->console_msr_work, console_msr_handler);
+	up->console_msr_work = IRQ_WORK_INIT_LAZY(console_msr_handler);
 
 	if (options)
 		uart_parse_options(options, &baud, &parity, &bits, &flow);
-- 
2.47.3

Re: [PATCH tty 1/1] serial: 8250: Change console_msr_work to IRQ_WORK_LAZY

[Sebastian Andrzej Siewior]

On 2026-08-28 18:04:08 [+0206], John Ogness wrote:
> For some platforms it is a problem to queue irq_work when entering
> cpuidle states. Since irq_work is used for deferred MSR handling,
> any atomic console printing when entering cpuidle states can lead
> to the affected hardware hanging. Tegra20 and Tegra30 are examples
> of such platforms. Avoiding raising the irq_work IRQ has shown to
> circumvent the problem.

I would argue that this is done for the benefit of less interrupts since
there no need to handle console_msr_work immediately. With the side
effect that it might fix a bug that was not yet reported on Tegra[23]0. 

Ideally the Tegra folks should figure out what exactly is broken instead
of adding duct tape everywhere else.

> Change the console_msr_work to be IRQ_WORK_LAZY, thus not raising
> an IRQ upon irq_work queuing. The irq_work is then handled on the
> next interrupt (worst case, kernel tick).

Not on the next interrupt, on the next jiffies tick. Should you run
tickless and should the CPU have the tick switch off then it will raise
the irq-work interrupt anyway.

side-note: Wasn't there another tegra workaround in v5/take1?

Sebastian

Re: [PATCH tty 1/1] serial: 8250: Change console_msr_work to IRQ_WORK_LAZY

[John Ogness]

On 2026-08-28, Sebastian Andrzej Siewior <bigeasy@linutronix.de> wrote:
> On 2026-08-28 18:04:08 [+0206], John Ogness wrote:
>> For some platforms it is a problem to queue irq_work when entering
>> cpuidle states. Since irq_work is used for deferred MSR handling,
>> any atomic console printing when entering cpuidle states can lead
>> to the affected hardware hanging. Tegra20 and Tegra30 are examples
>> of such platforms. Avoiding raising the irq_work IRQ has shown to
>> circumvent the problem.
>
> I would argue that this is done for the benefit of less interrupts since
> there no need to handle console_msr_work immediately.

Indeed that is a better argument, but strictly speaking, it isn't the
motivation. I think it is important to clarify that _currently_ there is
a problem that the explicit lazy is working around.

I expect most irq_work users could be changed to lazy with the argument
of reducing interrupts.

> side-note: Wasn't there another tegra workaround in v5/take1?

Yes, related to suspend/resume, implemented at the printk level [0]. Now
that we better understand the problem, that could be reverted. However,
it also provided a side-effect benefit: switching to atomic mode on
suspend when no_console_suspend is specified. I suppose at some point
(once the nbcon churn settles down) we will clean those changes so that
the naming and semantics really focus on "switching to atomic" rather
than "block irqwork".

John

[0] https://lore.kernel.org/lkml/20251113160351.113031-3-john.ogness@linutronix.de

Re: [PATCH tty 1/1] serial: 8250: Change console_msr_work to IRQ_WORK_LAZY

[Jon Hunter]

On 28/08/2026 17:06, Sebastian Andrzej Siewior wrote:
> On 2026-08-28 18:04:08 [+0206], John Ogness wrote:
>> For some platforms it is a problem to queue irq_work when entering
>> cpuidle states. Since irq_work is used for deferred MSR handling,
>> any atomic console printing when entering cpuidle states can lead
>> to the affected hardware hanging. Tegra20 and Tegra30 are examples
>> of such platforms. Avoiding raising the irq_work IRQ has shown to
>> circumvent the problem.
> 
> I would argue that this is done for the benefit of less interrupts since
> there no need to handle console_msr_work immediately. With the side
> effect that it might fix a bug that was not yet reported on Tegra[23]0.
> 
> Ideally the Tegra folks should figure out what exactly is broken instead
> of adding duct tape everywhere else.

Yes that would be ideal. Unfortunately these are old devices now and very
much in maintenance mode so getting to the bottom of this now is unlikely.
You may say then why don't we completely deprecate these, but then you
could argue the other way and say they have been working fine up until
now.

Jon

-- 
nvpublic

Re: [PATCH tty 1/1] serial: 8250: Change console_msr_work to IRQ_WORK_LAZY

[Andy Shevchenko]

On Fri, Aug 28, 2026 at 06:06:50PM +0100, Jon Hunter wrote:
> On 28/08/2026 17:06, Sebastian Andrzej Siewior wrote:
> > On 2026-08-28 18:04:08 [+0206], John Ogness wrote:
> > > For some platforms it is a problem to queue irq_work when entering
> > > cpuidle states. Since irq_work is used for deferred MSR handling,
> > > any atomic console printing when entering cpuidle states can lead
> > > to the affected hardware hanging. Tegra20 and Tegra30 are examples
> > > of such platforms. Avoiding raising the irq_work IRQ has shown to
> > > circumvent the problem.
> > 
> > I would argue that this is done for the benefit of less interrupts since
> > there no need to handle console_msr_work immediately. With the side
> > effect that it might fix a bug that was not yet reported on Tegra[23]0.
> > 
> > Ideally the Tegra folks should figure out what exactly is broken instead
> > of adding duct tape everywhere else.
> 
> Yes that would be ideal. Unfortunately these are old devices now and very
> much in maintenance mode so getting to the bottom of this now is unlikely.
> You may say then why don't we completely deprecate these, but then you
> could argue the other way and say they have been working fine up until
> now.

This is a trade-off between burden of supporting them vs. developing and move
on on more important things. Personally I consider non-blocking console feature
is *much* more important than a few "working fine up until now" outdated
platforms (side example, recently the whole i486 support was ripped off, while
working fine...).

-- 
With Best Regards,
Andy Shevchenko

Re: [PATCH tty 1/1] serial: 8250: Change console_msr_work to IRQ_WORK_LAZY

[Sebastian Andrzej Siewior]

On 2026-08-28 18:06:50 [+0100], Jon Hunter wrote:
> 
> On 28/08/2026 17:06, Sebastian Andrzej Siewior wrote:
> > On 2026-08-28 18:04:08 [+0206], John Ogness wrote:
> > > For some platforms it is a problem to queue irq_work when entering
> > > cpuidle states. Since irq_work is used for deferred MSR handling,
> > > any atomic console printing when entering cpuidle states can lead
> > > to the affected hardware hanging. Tegra20 and Tegra30 are examples
> > > of such platforms. Avoiding raising the irq_work IRQ has shown to
> > > circumvent the problem.
> > 
> > I would argue that this is done for the benefit of less interrupts since
> > there no need to handle console_msr_work immediately. With the side
> > effect that it might fix a bug that was not yet reported on Tegra[23]0.
> > 
> > Ideally the Tegra folks should figure out what exactly is broken instead
> > of adding duct tape everywhere else.
> 
> Yes that would be ideal. Unfortunately these are old devices now and very
> much in maintenance mode so getting to the bottom of this now is unlikely.

So this is maintenance mode.

> You may say then why don't we completely deprecate these, but then you
> could argue the other way and say they have been working fine up until
> now.

So a couple of years ago this platform stalled printk development due
its irq_work usage. Now again. Imagine we would have a ext4 feature
which freezes just this platform on boot. What would happen here?

The current workaround is to use LAZY which in general defers the IRQ
work to the next tick and it works for you. If you enable NOHZ and the
CPU goes tickless even a scheduled LAZY irq-work will raise the irq-work
interrupt to ensure the CPU does not go idle and we do have a wake up.

This might bring the problem back at some point.

> Jon

Sebastian

Re: [PATCH tty 1/1] serial: 8250: Change console_msr_work to IRQ_WORK_LAZY

[Petr Mladek]

On Fri 2026-08-28 18:06:52, Sebastian Andrzej Siewior wrote:
> On 2026-08-28 18:04:08 [+0206], John Ogness wrote:
> > For some platforms it is a problem to queue irq_work when entering
> > cpuidle states. Since irq_work is used for deferred MSR handling,
> > any atomic console printing when entering cpuidle states can lead
> > to the affected hardware hanging. Tegra20 and Tegra30 are examples
> > of such platforms. Avoiding raising the irq_work IRQ has shown to
> > circumvent the problem.
> 
> > Change the console_msr_work to be IRQ_WORK_LAZY, thus not raising
> > an IRQ upon irq_work queuing. The irq_work is then handled on the
> > next interrupt (worst case, kernel tick).
>
> Not on the next interrupt, on the next jiffies tick. Should you run
> tickless and should the CPU have the tick switch off then it will raise
> the irq-work interrupt anyway.

This seems to be more precise. There is one more scenario. It would
run in a dedicated kthread with PREEMPT_RT.

I hope that nobody would run tickless system on the old
Tegra hardware. So, this change should be good enough in reality.

Anyway, I would prefer to get this "workaround" in for 7.3.
In each case, I would not want to revert the uart 8250 conversion
to nbcon because it did break the Tegra hardware.

Best Regards,
Petr

Re: [PATCH tty 1/1] serial: 8250: Change console_msr_work to IRQ_WORK_LAZY

[Sebastian Andrzej Siewior]

On 2026-08-31 17:23:54 [+0200], Petr Mladek wrote:
> On Fri 2026-08-28 18:06:52, Sebastian Andrzej Siewior wrote:
> > On 2026-08-28 18:04:08 [+0206], John Ogness wrote:
> > > For some platforms it is a problem to queue irq_work when entering
> > > cpuidle states. Since irq_work is used for deferred MSR handling,
> > > any atomic console printing when entering cpuidle states can lead
> > > to the affected hardware hanging. Tegra20 and Tegra30 are examples
> > > of such platforms. Avoiding raising the irq_work IRQ has shown to
> > > circumvent the problem.
> > 
> > > Change the console_msr_work to be IRQ_WORK_LAZY, thus not raising
> > > an IRQ upon irq_work queuing. The irq_work is then handled on the
> > > next interrupt (worst case, kernel tick).
> >
> > Not on the next interrupt, on the next jiffies tick. Should you run
> > tickless and should the CPU have the tick switch off then it will raise
> > the irq-work interrupt anyway.
> 
> This seems to be more precise. There is one more scenario. It would
> run in a dedicated kthread with PREEMPT_RT.

That is an implementation detail same as the tickless thingy.

> I hope that nobody would run tickless system on the old
> Tegra hardware. So, this change should be good enough in reality.
> 
> Anyway, I would prefer to get this "workaround" in for 7.3.
> In each case, I would not want to revert the uart 8250 conversion
> to nbcon because it did break the Tegra hardware.

I would see this as an improvement and not a workaround. It has the
side effect that it works now on Tegra which has apparently a broken low
idle implementation.

*If* this LAZY mode couldn't be used for some reason, I would suggest to
disable the offending low idle mode
	https://lore.kernel.org/all/791131d7-cf8d-4572-8eec-df5fff1783a9@nvidia.com/

rather than reverting 8250-nbcon for everyone.


> Best Regards,
> Petr

Sebastian

Re: [PATCH tty 1/1] serial: 8250: Change console_msr_work to IRQ_WORK_LAZY

[Jon Hunter]

On 31/08/2026 09:20, Sebastian Andrzej Siewior wrote:
> On 2026-08-28 18:06:50 [+0100], Jon Hunter wrote:
>>
>> On 28/08/2026 17:06, Sebastian Andrzej Siewior wrote:
>>> On 2026-08-28 18:04:08 [+0206], John Ogness wrote:
>>>> For some platforms it is a problem to queue irq_work when entering
>>>> cpuidle states. Since irq_work is used for deferred MSR handling,
>>>> any atomic console printing when entering cpuidle states can lead
>>>> to the affected hardware hanging. Tegra20 and Tegra30 are examples
>>>> of such platforms. Avoiding raising the irq_work IRQ has shown to
>>>> circumvent the problem.
>>>
>>> I would argue that this is done for the benefit of less interrupts since
>>> there no need to handle console_msr_work immediately. With the side
>>> effect that it might fix a bug that was not yet reported on Tegra[23]0.
>>>
>>> Ideally the Tegra folks should figure out what exactly is broken instead
>>> of adding duct tape everywhere else.
>>
>> Yes that would be ideal. Unfortunately these are old devices now and very
>> much in maintenance mode so getting to the bottom of this now is unlikely.
> 
> So this is maintenance mode.
> 
>> You may say then why don't we completely deprecate these, but then you
>> could argue the other way and say they have been working fine up until
>> now.
> 
> So a couple of years ago this platform stalled printk development due
> its irq_work usage. Now again. Imagine we would have a ext4 feature
> which freezes just this platform on boot. What would happen here?

I simply reported a regression. I did not set out to stall any work.
If the work stalled because of this issue alone, why was this not
discussed back when this happened? I am sure we could have come to
a reasonable comprise.

Please note that these legacy boards have caught some other issues in
the past and so I do see some value in keeping them working. This one
is a bit of an odd ball because clearly this platform has some issues
related to CPU idle. Again we are happy to discuss that and come to a
reasonable comprise.
> The current workaround is to use LAZY which in general defers the IRQ
> work to the next tick and it works for you. If you enable NOHZ and the
> CPU goes tickless even a scheduled LAZY irq-work will raise the irq-work
> interrupt to ensure the CPU does not go idle and we do have a wake up.
> 
> This might bring the problem back at some point.

That may be, and we will cross that bridge if and when we come to it.

Jon

-- 
nvpublic

Re: [PATCH tty 1/1] serial: 8250: Change console_msr_work to IRQ_WORK_LAZY

[Jon Hunter]

On 28/08/2026 16:58, John Ogness wrote:
> For some platforms it is a problem to queue irq_work when entering
> cpuidle states. Since irq_work is used for deferred MSR handling,
> any atomic console printing when entering cpuidle states can lead
> to the affected hardware hanging. Tegra20 and Tegra30 are examples
> of such platforms. Avoiding raising the irq_work IRQ has shown to
> circumvent the problem.
> 
> Change the console_msr_work to be IRQ_WORK_LAZY, thus not raising
> an IRQ upon irq_work queuing. The irq_work is then handled on the
> next interrupt (worst case, kernel tick).
> 
> Link: https://lore.kernel.org/lkml/f3757a75-0ba1-4558-bf57-f19ab7e59a4c@nvidia.com
> Fixes: d3539347022a ("serial: 8250: Switch to nbcon console, take 2")
> Signed-off-by: John Ogness <john.ogness@linutronix.de>
> ---
>   drivers/tty/serial/8250/8250_port.c | 2 +-
>   1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/drivers/tty/serial/8250/8250_port.c b/drivers/tty/serial/8250/8250_port.c
> index 38fa45e74a37a..1acb73e45f8a0 100644
> --- a/drivers/tty/serial/8250/8250_port.c
> +++ b/drivers/tty/serial/8250/8250_port.c
> @@ -3597,7 +3597,7 @@ int serial8250_console_setup(struct uart_port *port, char *options, bool probe)
>   
>   	up->console_line_ended = true;
>   	up->console_msr_work_allow = true;
> -	init_irq_work(&up->console_msr_work, console_msr_handler);
> +	up->console_msr_work = IRQ_WORK_INIT_LAZY(console_msr_handler);
>   
>   	if (options)
>   		uart_parse_options(options, &baud, &parity, &bits, &flow);


FWIW ...

Tested-by: Jon Hunter <jonathanh@nvidia.com>

Thanks!
Jon

-- 
nvpublic