[PATCH] nbd: fail requests stuck on a dead socket when timeout is 0

[PATCH] nbd: fail requests stuck on a dead socket when timeout is 0

[Nguyen Ngoc Thang]

With a zero command timeout, nbd_xmit_timeout() only logs a "possible
stuck request" message and resets the timer. A request already sent on
a socket that has since died is never completed this way, because
nothing will ever answer it.

That blocks nbd_set_size(), which freezes the queue while holding
nbd->config_lock and genl_mutex. The freeze waits for the stuck request
forever, and reconnecting a socket (which would change the cookie and
get the request requeued) can't happen as it is handled later by the
same reconfigure call. Everything touching the device then hangs,
including partition scanning in udev:

  INFO: task udevd blocked for more than 143 seconds.
   __bio_queue_enter+0x369/0x7f0 block/blk-core.c:396
   blk_mq_submit_bio+0x776/0x2a70 block/blk-mq.c:3127
   ...
  INFO: task syz.0.17 blocked for more than 143 seconds.
   blk_mq_freeze_queue_wait+0x101/0x180 block/blk-mq.c:191
   queue_limits_commit_update_frozen+0x55/0xd0 block/blk-settings.c:590
   nbd_set_size+0x4c5/0x7d0 drivers/block/nbd.c:383
   nbd_genl_reconfigure+0x7e1/0x1040 drivers/block/nbd.c:2444

Requeue the request when its socket is dead, as is already done when
the socket was replaced. The submit path then waits for a reconnect up
to dead_conn_timeout and otherwise fails the request with an I/O error.

Reported-by: syzbot+90895f5bfaa7243c2d20@syzkaller.appspotmail.com
Closes: https://syzkaller.appspot.com/bug?extid=90895f5bfaa7243c2d20
Fixes: 2da22da57348 ("nbd: fix zero cmd timeout handling v2")
Signed-off-by: Nguyen Ngoc Thang <ngocthang2710.1999@gmail.com>
---
 drivers/block/nbd.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/drivers/block/nbd.c b/drivers/block/nbd.c
index ffce519bf008..218b99f9e1ef 100644
--- a/drivers/block/nbd.c
+++ b/drivers/block/nbd.c
@@ -532,7 +532,8 @@ static enum blk_eh_timer_return nbd_xmit_timeout(struct request *req)
 			blk_rq_bytes(req), (req->timeout / HZ) * cmd->retries);
 
 		mutex_lock(&nsock->tx_lock);
-		if (cmd->cookie != nsock->cookie) {
+		/* A dead socket will never complete this request. */
+		if (cmd->cookie != nsock->cookie || nsock->dead) {
 			nbd_requeue_cmd(cmd);
 			mutex_unlock(&nsock->tx_lock);
 			mutex_unlock(&cmd->lock);
-- 
2.43.0

Re: [PATCH] nbd: fail requests stuck on a dead socket when timeout is 0

[Bart Van Assche]

On 10/10/26 8:37 PM, Nguyen Ngoc Thang wrote:
> With a zero command timeout, nbd_xmit_timeout() only logs a "possible
> stuck request" message and resets the timer. A request already sent on
> a socket that has since died is never completed this way, because
> nothing will ever answer it.
> 
> That blocks nbd_set_size(), which freezes the queue while holding
> nbd->config_lock and genl_mutex. The freeze waits for the stuck request
> forever, and reconnecting a socket (which would change the cookie and
> get the request requeued) can't happen as it is handled later by the
> same reconfigure call. Everything touching the device then hangs,
> including partition scanning in udev:
> 
>    INFO: task udevd blocked for more than 143 seconds.
>     __bio_queue_enter+0x369/0x7f0 block/blk-core.c:396
>     blk_mq_submit_bio+0x776/0x2a70 block/blk-mq.c:3127
>     ...
>    INFO: task syz.0.17 blocked for more than 143 seconds.
>     blk_mq_freeze_queue_wait+0x101/0x180 block/blk-mq.c:191
>     queue_limits_commit_update_frozen+0x55/0xd0 block/blk-settings.c:590
>     nbd_set_size+0x4c5/0x7d0 drivers/block/nbd.c:383
>     nbd_genl_reconfigure+0x7e1/0x1040 drivers/block/nbd.c:2444
> 
> Requeue the request when its socket is dead, as is already done when
> the socket was replaced. The submit path then waits for a reconnect up
> to dead_conn_timeout and otherwise fails the request with an I/O error.
> 
> Reported-by: syzbot+90895f5bfaa7243c2d20@syzkaller.appspotmail.com
> Closes: https://syzkaller.appspot.com/bug?extid=90895f5bfaa7243c2d20
> Fixes: 2da22da57348 ("nbd: fix zero cmd timeout handling v2")
> Signed-off-by: Nguyen Ngoc Thang <ngocthang2710.1999@gmail.com>
> ---
>   drivers/block/nbd.c | 3 ++-
>   1 file changed, 2 insertions(+), 1 deletion(-)
> 
> diff --git a/drivers/block/nbd.c b/drivers/block/nbd.c
> index ffce519bf008..218b99f9e1ef 100644
> --- a/drivers/block/nbd.c
> +++ b/drivers/block/nbd.c
> @@ -532,7 +532,8 @@ static enum blk_eh_timer_return nbd_xmit_timeout(struct request *req)
>   			blk_rq_bytes(req), (req->timeout / HZ) * cmd->retries);
>   
>   		mutex_lock(&nsock->tx_lock);
> -		if (cmd->cookie != nsock->cookie) {
> +		/* A dead socket will never complete this request. */
> +		if (cmd->cookie != nsock->cookie || nsock->dead) {
>   			nbd_requeue_cmd(cmd);
>   			mutex_unlock(&nsock->tx_lock);
>   			mutex_unlock(&cmd->lock);

If AI was used to create this patch an Assisted-by tag should be added.
See also Documentation/process/coding-assistants.rst. Anyway, since this
patch looks good to me:

Reviewed-by: Bart Van Assche <bvanassche@acm.org>