[PATCH] virtio_net: fix infinite loop in virtnet_poll_cleantx when device is broken

[PATCH] virtio_net: fix infinite loop in virtnet_poll_cleantx when device is broken

[Jinqian Yang]

virtnet_poll_cleantx() contains a do-while loop that cleans up
transmitted TX buffers and calls virtqueue_enable_cb_delayed() to check
whether more buffers need processing. When the virtio backend stops
responding during guest reboot, used->idx is never updated, so
virtqueue_enable_cb_delayed() always returns false and the loop never
terminates. Then it will block reboot process, and the guest will hang.

The problem occurs during guest reboot under network traffic:

  1. kernel_restart() -> device_shutdown() traverses the device list
  2. virtio_dev_shutdown() calls virtio_break_device() which sets
     vq->broken = true
  3. virtio_dev_shutdown() then calls virtio_synchronize_cbs() to wait
     for in-flight callbacks to complete
  4. A virtio interrupt fires, softirq is deferred to ksoftirqd which
     calls net_rx_action() -> virtnet_poll() -> virtnet_poll_cleantx()
  5. virtnet_poll_cleantx() enters the do-while loop and never exits
     because the QEMU backend has stopped updating used->idx, despite
     vq->broken having been set to true in step 2.

Since the loop runs inside ksoftirqd (a SCHED_OTHER kthread), it is
visible to the scheduler and does not trigger a hard lockup. However,
the kthread never leaves the loop, so RCU detects it as a CPU stall
and reports it periodically. Meanwhile, the reboot process remains
blocked in device_shutdown() because virtio_dev_shutdown() cannot
complete its synchronization step, and the guest hangs permanently.

This can be reproduced on a guest with a virtio-net device: run iperf3
traffic in the guest, then trigger reboot. The reboot occasionally hangs
permanently with RCU stall on ksoftirqd.

Observed on ARM64 KVM guest:

  CPU#1 RCU stall (ksoftirqd/1), repeated periodically:
    virtqueue_enable_cb_delayed_split <- virtnet_poll <- __napi_poll <-
    net_rx_action <- handle_softirqs <- run_ksoftirqd <-
    smpboot_thread_fn <- kthread

Fix by adding a virtqueue_is_broken() check to the loop condition, so
that the loop exits immediately when the device is broken, allowing
the device shutdown to proceed.

Signed-off-by: Jinqian Yang <yangjinqian1@huawei.com>
---
 drivers/net/virtio_net.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/drivers/net/virtio_net.c b/drivers/net/virtio_net.c
index 7d2eeb9b1226..c8d2d420c31d 100644
--- a/drivers/net/virtio_net.c
+++ b/drivers/net/virtio_net.c
@@ -2970,7 +2970,8 @@ static void virtnet_poll_cleantx(struct receive_queue *rq, int budget)
 		do {
 			virtqueue_disable_cb(sq->vq);
 			free_old_xmit(sq, txq, !!budget);
-		} while (unlikely(!virtqueue_enable_cb_delayed(sq->vq)));
+		} while (!virtqueue_is_broken(sq->vq) &&
+			 unlikely(!virtqueue_enable_cb_delayed(sq->vq)));
 
 		if (sq->vq->num_free >= MAX_SKB_FRAGS + 2)
 			virtnet_tx_wake_queue(vi, sq);
-- 
2.33.0

Re: [PATCH] virtio_net: fix infinite loop in virtnet_poll_cleantx when device is broken

[Michael S. Tsirkin]

On Mon, Jul 13, 2026 at 09:20:25PM +0800, Jinqian Yang wrote:
> virtnet_poll_cleantx() contains a do-while loop that cleans up
> transmitted TX buffers and calls virtqueue_enable_cb_delayed() to check
> whether more buffers need processing. When the virtio backend stops
> responding during guest reboot, used->idx is never updated, so
> virtqueue_enable_cb_delayed() always returns false and the loop never
> terminates. Then it will block reboot process, and the guest will hang.
> 
> The problem occurs during guest reboot under network traffic:
> 
>   1. kernel_restart() -> device_shutdown() traverses the device list
>   2. virtio_dev_shutdown() calls virtio_break_device() which sets
>      vq->broken = true
>   3. virtio_dev_shutdown() then calls virtio_synchronize_cbs() to wait
>      for in-flight callbacks to complete
>   4. A virtio interrupt fires, softirq is deferred to ksoftirqd which
>      calls net_rx_action() -> virtnet_poll() -> virtnet_poll_cleantx()
>   5. virtnet_poll_cleantx() enters the do-while loop and never exits
>      because the QEMU backend has stopped updating used->idx, despite
>      vq->broken having been set to true in step 2.
> 
> Since the loop runs inside ksoftirqd (a SCHED_OTHER kthread), it is
> visible to the scheduler and does not trigger a hard lockup. However,
> the kthread never leaves the loop, so RCU detects it as a CPU stall
> and reports it periodically. Meanwhile, the reboot process remains
> blocked in device_shutdown() because virtio_dev_shutdown() cannot
> complete its synchronization step, and the guest hangs permanently.
> 
> This can be reproduced on a guest with a virtio-net device: run iperf3
> traffic in the guest, then trigger reboot. The reboot occasionally hangs
> permanently with RCU stall on ksoftirqd.
> 
> Observed on ARM64 KVM guest:
> 
>   CPU#1 RCU stall (ksoftirqd/1), repeated periodically:
>     virtqueue_enable_cb_delayed_split <- virtnet_poll <- __napi_poll <-
>     net_rx_action <- handle_softirqs <- run_ksoftirqd <-
>     smpboot_thread_fn <- kthread
> 
> Fix by adding a virtqueue_is_broken() check to the loop condition, so
> that the loop exits immediately when the device is broken, allowing
> the device shutdown to proceed.
> 
> Signed-off-by: Jinqian Yang <yangjinqian1@huawei.com>

I'd expect lots of drivers have this issue?  Wouldn't it make more sense
to check virtqueue_is_broken in
virtqueue_enable_cb_delayed/virtqueue_enable_cb? This way it works for
all drivers.



> ---
>  drivers/net/virtio_net.c | 3 ++-
>  1 file changed, 2 insertions(+), 1 deletion(-)
> 
> diff --git a/drivers/net/virtio_net.c b/drivers/net/virtio_net.c
> index 7d2eeb9b1226..c8d2d420c31d 100644
> --- a/drivers/net/virtio_net.c
> +++ b/drivers/net/virtio_net.c
> @@ -2970,7 +2970,8 @@ static void virtnet_poll_cleantx(struct receive_queue *rq, int budget)
>  		do {
>  			virtqueue_disable_cb(sq->vq);
>  			free_old_xmit(sq, txq, !!budget);
> -		} while (unlikely(!virtqueue_enable_cb_delayed(sq->vq)));
> +		} while (!virtqueue_is_broken(sq->vq) &&
> +			 unlikely(!virtqueue_enable_cb_delayed(sq->vq)));
>  
>  		if (sq->vq->num_free >= MAX_SKB_FRAGS + 2)
>  			virtnet_tx_wake_queue(vi, sq);
> -- 
> 2.33.0

Re: [PATCH] virtio_net: fix infinite loop in virtnet_poll_cleantx when device is broken

[Jinqian Yang]

Hi,

On 2026/7/14 21:17, Michael S. Tsirkin wrote:
> On Mon, Jul 13, 2026 at 09:20:25PM +0800, Jinqian Yang wrote:
>> virtnet_poll_cleantx() contains a do-while loop that cleans up
>> transmitted TX buffers and calls virtqueue_enable_cb_delayed() to check
>> whether more buffers need processing. When the virtio backend stops
>> responding during guest reboot, used->idx is never updated, so
>> virtqueue_enable_cb_delayed() always returns false and the loop never
>> terminates. Then it will block reboot process, and the guest will hang.
>>
>> The problem occurs during guest reboot under network traffic:
>>
>>    1. kernel_restart() -> device_shutdown() traverses the device list
>>    2. virtio_dev_shutdown() calls virtio_break_device() which sets
>>       vq->broken = true
>>    3. virtio_dev_shutdown() then calls virtio_synchronize_cbs() to wait
>>       for in-flight callbacks to complete
>>    4. A virtio interrupt fires, softirq is deferred to ksoftirqd which
>>       calls net_rx_action() -> virtnet_poll() -> virtnet_poll_cleantx()
>>    5. virtnet_poll_cleantx() enters the do-while loop and never exits
>>       because the QEMU backend has stopped updating used->idx, despite
>>       vq->broken having been set to true in step 2.
>>
>> Since the loop runs inside ksoftirqd (a SCHED_OTHER kthread), it is
>> visible to the scheduler and does not trigger a hard lockup. However,
>> the kthread never leaves the loop, so RCU detects it as a CPU stall
>> and reports it periodically. Meanwhile, the reboot process remains
>> blocked in device_shutdown() because virtio_dev_shutdown() cannot
>> complete its synchronization step, and the guest hangs permanently.
>>
>> This can be reproduced on a guest with a virtio-net device: run iperf3
>> traffic in the guest, then trigger reboot. The reboot occasionally hangs
>> permanently with RCU stall on ksoftirqd.
>>
>> Observed on ARM64 KVM guest:
>>
>>    CPU#1 RCU stall (ksoftirqd/1), repeated periodically:
>>      virtqueue_enable_cb_delayed_split <- virtnet_poll <- __napi_poll <-
>>      net_rx_action <- handle_softirqs <- run_ksoftirqd <-
>>      smpboot_thread_fn <- kthread
>>
>> Fix by adding a virtqueue_is_broken() check to the loop condition, so
>> that the loop exits immediately when the device is broken, allowing
>> the device shutdown to proceed.
>>
>> Signed-off-by: Jinqian Yang <yangjinqian1@huawei.com>
> 
> I'd expect lots of drivers have this issue?  Wouldn't it make more sense
> to check virtqueue_is_broken in
> virtqueue_enable_cb_delayed/virtqueue_enable_cb? This way it works for
> all drivers.
> 

In virtqueue_enable_cb->virtqueue_poll, a check for vq->broken is
performed, so other devices do not have this issue.

Indeed, it is more reasonable to check vq->broken inside
virtqueue_enable_cb_delayed. I will make the changes in v2.

Thanks,
Jinqian

> 
> 
>> ---
>>   drivers/net/virtio_net.c | 3 ++-
>>   1 file changed, 2 insertions(+), 1 deletion(-)
>>
>> diff --git a/drivers/net/virtio_net.c b/drivers/net/virtio_net.c
>> index 7d2eeb9b1226..c8d2d420c31d 100644
>> --- a/drivers/net/virtio_net.c
>> +++ b/drivers/net/virtio_net.c
>> @@ -2970,7 +2970,8 @@ static void virtnet_poll_cleantx(struct receive_queue *rq, int budget)
>>   		do {
>>   			virtqueue_disable_cb(sq->vq);
>>   			free_old_xmit(sq, txq, !!budget);
>> -		} while (unlikely(!virtqueue_enable_cb_delayed(sq->vq)));
>> +		} while (!virtqueue_is_broken(sq->vq) &&
>> +			 unlikely(!virtqueue_enable_cb_delayed(sq->vq)));
>>   
>>   		if (sq->vq->num_free >= MAX_SKB_FRAGS + 2)
>>   			virtnet_tx_wake_queue(vi, sq);
>> -- 
>> 2.33.0
> 
>