Re: [PATCH 0/2] Re-raise irq_work lost during Xen suspend

Re: [PATCH 0/2] Re-raise irq_work lost during Xen suspend

[Sebastian Andrzej Siewior]

On 2026-11-06 15:57:40 [+0100], Julian Vetter wrote:
> We hit this issue with an Ubuntu 26.04 cloudimage guests (7.0 based
> kernel with the 8250 as an nbcon console) on XCP-ng 8.3 / Xen 4.17. With
> a kernel commandline adding two consoles, i.e., 'console=ttyS1
> console=ttyS0'. During a live migration on that setup the printk()s from
> gnttab_resume() and xen_irq_resume() queue nbcon_irq_work while the APIC
> is disabled. Thus the list transitions from empty to non-empty, but
> doesn't receive the IPI to empty the list again. Afterwards the list
> stays non-empty, so the IPI never fires and all irq_work piles up.
> 
> The Commit 560f4deda327 ("printk/nbcon: Change nbcon_irq_work to
> IRQ_WORK_LAZY") removed this particular issue because declaring the IRQ
> work to be lazy, causes the timer interrupt to trigger the irq work
> instead. But any other non-lazy irq_work queued in that window is still
> lost the same way and will cause the list to fill up without being
> handled.

Adrian has this:
   https://lore.kernel.org/all/20261102074039.26983-2-adrian.hunter@intel.com/

I sort of asked for flush on suspend but Adrian pointed out the NMI
problem.

Looking at #1, you could probably limit it to a nop on
!arch_irq_work_has_interrupt(). The tick should not be disabled if
irq_work_needs_cpu() returns true (you don't add anything).

And lazy items get handled via irq_work_tick() anyway. This make me
think if it gets any easier if we handle both in irq_work_tick(). On x86
raised_list is evaluated anyway so by dropping
arch_irq_work_has_interrupt() we would avoid the APIC check and handle
list in this rare case. If the IRQ-work is working then the list remains
empty.

> Signed-off-by: Julian Vetter <julian.vetter@vates.tech>

Sebastian

Re: [PATCH 0/2] Re-raise irq_work lost during Xen suspend

[Julian Vetter]

On 11/6/26 4:39 PM, Sebastian Andrzej Siewior wrote:
> On 2026-11-06 15:57:40 [+0100], Julian Vetter wrote:
>> We hit this issue with an Ubuntu 26.04 cloudimage guests (7.0 based
>> kernel with the 8250 as an nbcon console) on XCP-ng 8.3 / Xen 4.17. With
>> a kernel commandline adding two consoles, i.e., 'console=ttyS1
>> console=ttyS0'. During a live migration on that setup the printk()s from
>> gnttab_resume() and xen_irq_resume() queue nbcon_irq_work while the APIC
>> is disabled. Thus the list transitions from empty to non-empty, but
>> doesn't receive the IPI to empty the list again. Afterwards the list
>> stays non-empty, so the IPI never fires and all irq_work piles up.
>>
>> The Commit 560f4deda327 ("printk/nbcon: Change nbcon_irq_work to
>> IRQ_WORK_LAZY") removed this particular issue because declaring the IRQ
>> work to be lazy, causes the timer interrupt to trigger the irq work
>> instead. But any other non-lazy irq_work queued in that window is still
>> lost the same way and will cause the list to fill up without being
>> handled.
> 
> Adrian has this:
>     https://lore.kernel.org/all/20261102074039.26983-2-adrian.hunter@intel.com/
> 
> I sort of asked for flush on suspend but Adrian pointed out the NMI
> problem.

Thanks, I missed that. It is the same issue, just a different trigger. 
Adrian's example even ends in the same hung task we see 
(fsnotify_mark_destroy_workfn waiting on SRCU). Also, Adrian's 
syscore_resume() placement would cover the Xen case as well, as 
xen_suspend() calls syscore_resume() after those printk()s.

> 
> Looking at #1, you could probably limit it to a nop on
> !arch_irq_work_has_interrupt(). The tick should not be disabled if
> irq_work_needs_cpu() returns true (you don't add anything).
> 

Yes, right, the lazy part is pointless.

> And lazy items get handled via irq_work_tick() anyway. This make me
> think if it gets any easier if we handle both in irq_work_tick(). On x86
> raised_list is evaluated anyway so by dropping
> arch_irq_work_has_interrupt() we would avoid the APIC check and handle
> list in this rare case. If the IRQ-work is working then the list remains
> empty.
> 

This is maybe a cleaner solution, it needs no APIC access and no call 
site in the suspend paths. But irq_work_needs_cpu() would need the same 
change, otherwise in a CONFIG_NOHZ=y setting an idle CPU would still sit 
on the work until it leaves idle and re-enables the timer tick, no? I 
will test this.

>> Signed-off-by: Julian Vetter <julian.vetter@vates.tech>
> 
> Sebastian
> 

(Sorry, sending from my personal address, my work mail server currently 
mangles mails to the list.)

Julian

Re: [PATCH 0/2] Re-raise irq_work lost during Xen suspend

[Adrian Hunter]

On 09/11/2026 12:16, Julian Vetter wrote:
> On 11/6/26 4:39 PM, Sebastian Andrzej Siewior wrote:
>>
>> Looking at #1, you could probably limit it to a nop on
>> !arch_irq_work_has_interrupt(). The tick should not be disabled if
>> irq_work_needs_cpu() returns true (you don't add anything).
>>
> 
> Yes, right, the lazy part is pointless.
> 
>> And lazy items get handled via irq_work_tick() anyway. This make me
>> think if it gets any easier if we handle both in irq_work_tick(). On x86
>> raised_list is evaluated anyway so by dropping
>> arch_irq_work_has_interrupt() we would avoid the APIC check and handle
>> list in this rare case. If the IRQ-work is working then the list remains
>> empty.
>>
> 
> This is maybe a cleaner solution, it needs no APIC access and no call site in the suspend paths. But irq_work_needs_cpu() would need the same change, otherwise in a CONFIG_NOHZ=y setting an idle CPU would still sit on the work until it leaves idle and re-enables the timer tick, no? I will test this.

A fix for stable is needed, so need to check that there is no
irq_work that would have a problem running in the tick context.