[BUG] ublk: hung tasks in bdev_open during partition scan (v7.4-rc2)

[BUG] ublk: hung tasks in bdev_open during partition scan (v7.4-rc2)

[铮]

Hi Ming and Jens,

I encountered an `INFO: task hung in bdev_open()` while running
syzkaller on Linux v7.4-rc2.

Two tasks, a udev worker and a syzkaller executor, were blocked for
more than 143 seconds in `bdev_open()` (block/bdev.c:990), waiting on
the same gendisk `open_mutex` while opening block device nodes via
`openat()`.

The lock dump also identifies a ublk partition-scan workqueue
associated with the same mutex. This suggests that the prolonged wait
may be related to an unfinished ublk partition scan.

The issue was observed twice during concurrent fuzzing. A subsequent
state-primed verification run also reproduced the hung-task condition
on a freshly booted VM.

All raw logs, crash reports, reproduction programs, and kernel/fuzzing
configurations are available at:

https://github.com/Zheng-zheng-codes/bdev-open-hang-7.4-rc2

All relative paths below refer to files in this repository.

**Kernel information:**

- Kernel version: 7.4.0-rc2-kernsight-kasan #1
- Base: v7.4-rc2 (`7475ea8b5`)
- Architecture: x86_64
- Environment: QEMU Standard PC (i440FX + PIIX), KVM, systemd-based guest
- Configuration: KASAN enabled, `hung_task_panic=0`
- Fuzzing framework: syzkaller
- The full kernel configuration and QEMU/fuzzing settings are included
in the repository.

**Hung-task report (verification run):**

```text
INFO: task (udev-worker):9713 blocked for more than 143 seconds.
      Not tainted 7.4.0-rc2-kernsight-kasan #1
task:(udev-worker) state:D stack:25208 pid:9713 tgid:9713 ppid:4787

Call Trace:
 <TASK>
 __schedule+0x1425/0x6840 kernel/sched/core.c:7389
 schedule+0xec/0x2d0 kernel/sched/core.c:7481
 __mutex_lock_common kernel/locking/mutex.c:726 [inline]
 __mutex_lock+0xe0c/0x1dd0 kernel/locking/mutex.c:821
 bdev_open+0x3fa/0xe50 block/bdev.c:990
 blkdev_open+0x355/0x500 block/fops.c:686
 do_dentry_open+0x693/0x14f0 fs/open.c:1000
 vfs_open_consume+0xd1/0x460 fs/open.c:1131
 do_open fs/namei.c:4851 [inline]
 path_openat+0x1c1b/0x2240 fs/namei.c:5016
 do_file_open+0x21b/0x470 fs/namei.c:5045
 do_sys_openat2+0x13e/0x1d0 fs/open.c:1443
 __x64_sys_openat+0x144/0x200 fs/open.c:1460
 entry_SYSCALL_64_after_hwframe+0x77/0x7f
 </TASK>

INFO: task syz.2.29:12841 blocked for more than 143 seconds.
task:syz.2.29 state:D stack:27784 pid:12841 tgid:12818 ppid:10317

Call Trace:
 <TASK>
 __mutex_lock
 bdev_open        block/bdev.c:990
 blkdev_open      block/fops.c:686
 do_dentry_open
 path_openat
 __x64_sys_openat
 </TASK>
```

**Lock observations:**

In the verification report, the lock dump identifies the ublk
partition-scan worker:

```text
locks held by kworker/2:2/1815:

#1: ((work_completion)(&ub->partition_scan_work))
    at process_one_work

#2: ff1100017b50c350 (&disk->open_mutex)
    at ublk_partition_scan_work+0xdb/0x170
    drivers/block/ublk_drv.c:2478
```

The same mutex address appears in the lock records for the two blocked tasks:

```text
locks held by (udev-worker)/9713:
    ff1100017b50c350 (&disk->open_mutex)
    at bdev_open+0x3fa/0xe50

locks held by syz.2.29/12841:
    ff1100017b50c350 (&disk->open_mutex)
    at bdev_open+0x3fa/0xe50
```

A separate fuzzing occurrence shows a similar relationship:

```text
kworker/3:2/1260:
    ublk_partition_scan_work
    (&disk->open_mutex)

(udev-worker)/9505:
    bdev_open
    (&disk->open_mutex)

syz.4.2301/42546:
    bdev_open
    (&disk->open_mutex)
```

The repeated association between the blocked openers and
`ublk_partition_scan_work()` suggests that the partition scan may be
involved in the prolonged wait.

However, the current logs do not establish why the partition-scan
worker fails to complete. In particular, I have not determined whether
it is waiting for outstanding I/O from the userspace ublk backend or
encountering another condition.

**Reproduction:**

The repository contains a minimized target program together with a
recorded state-reconstruction sequence.

The minimized `repro/repro.prog` is not sufficient by itself to
reliably reproduce the hang. The state-primed reproduction uses the
recorded sequence and the target program in the same VM.

The relevant files are:

- `repro/repro.prog`: minimized target program
- `repro/repro.priming.prog`: recorded state-reconstruction sequence
- `repro/repro.kernsight.prog`: combined state-primed reproduction input
- `repro/repro.README.txt`: detailed reproduction instructions

The state-primed reproduction can be attempted using:

```bash
./syz-execprog \
    -executor=./syz-executor \
    -procs=1 \
    -repeat=1 \
    ./repro.kernsight.prog
```

The guest should be booted using the kernel configuration and QEMU
settings documented in the repository. The same syzkaller environment
should be used for the reproduction.

The verification run recorded a hung-task report, and the repository
contains the corresponding execution log and report.

One caveat is that the udev worker appears to have entered the blocked
state before the final minimized target program was executed.
Therefore, I do not claim that the minimized target program alone
causes the hang. The preceding state-reconstruction sequence may be
responsible for establishing the condition.

**Observed results:**

- Two occurrences during concurrent syzkaller fuzzing
- One recorded successful state-primed verification run
- Both the udev worker and a syzkaller executor were observed waiting
in `bdev_open()`
- The lock dumps associate the same `disk->open_mutex` with the ublk
partition-scan workqueue
- The VM remained running with `hung_task_panic=0`

**Repository contents:**

Reproduction artifacts:

- `repro/repro.prog`
- `repro/repro.kernsight.prog`
- `repro/repro.priming.prog`
- `repro/repro.README.txt`
- `repro/repro.report`
- `repro/repro.log`
- `repro/repro.state.json`
- `repro/repro.context.json`
- `repro/repro.attempts.json`
- `repro/repro.outcomes.jsonl`

Original fuzzing evidence:

- `logs/log0`, `logs/log1`
- `logs/report0`, `logs/report1`
- `logs/machineInfo0`, `logs/machineInfo1`
- `logs/pre_state0`, `logs/pre_state1`
- `logs/crash-excerpt.txt`

Environment:

- `config-7.4.0-rc2-kernsight-kasan`: full kernel configuration
- `syzkaller.cfg`: syzkaller and QEMU configuration

**Related upstream work:**

I noticed the earlier ublk fixes concerning canceled I/O commands and
partition-scan hangs, including the change titled:

"ublk: refuse to go live after an io command was canceled"

This issue was observed on v7.4-rc2, but I have not established
whether it involves the same underlying condition or a different
problem.

Could you please advise whether this corresponds to a known issue, or
whether additional diagnostics from the partition-scan worker or the
ublk I/O queues would be useful?

Thank you for your time.

Best regards,
Zheng Zheng

Re: [BUG] ublk: hung tasks in bdev_open during partition scan (v7.4-rc2)

[Ming Lei]

On Tue, Nov 10, 2026 at 01:15:43AM +0800, 铮 wrote:
> Hi Ming and Jens,
> 
> I encountered an `INFO: task hung in bdev_open()` while running
> syzkaller on Linux v7.4-rc2.
> 
> Two tasks, a udev worker and a syzkaller executor, were blocked for
> more than 143 seconds in `bdev_open()` (block/bdev.c:990), waiting on
> the same gendisk `open_mutex` while opening block device nodes via
> `openat()`.
> 
> The lock dump also identifies a ublk partition-scan workqueue
> associated with the same mutex. This suggests that the prolonged wait
> may be related to an unfinished ublk partition scan.
> 
> The issue was observed twice during concurrent fuzzing. A subsequent
> state-primed verification run also reproduced the hung-task condition
> on a freshly booted VM.
> 
> All raw logs, crash reports, reproduction programs, and kernel/fuzzing
> configurations are available at:
> 
> https://github.com/Zheng-zheng-codes/bdev-open-hang-7.4-rc2
> 
> All relative paths below refer to files in this repository.
> 
> **Kernel information:**
> 
> - Kernel version: 7.4.0-rc2-kernsight-kasan #1
> - Base: v7.4-rc2 (`7475ea8b5`)
> - Architecture: x86_64
> - Environment: QEMU Standard PC (i440FX + PIIX), KVM, systemd-based guest
> - Configuration: KASAN enabled, `hung_task_panic=0`
> - Fuzzing framework: syzkaller
> - The full kernel configuration and QEMU/fuzzing settings are included
> in the repository.
> 
> **Hung-task report (verification run):**
> 
> ```text
> INFO: task (udev-worker):9713 blocked for more than 143 seconds.
>       Not tainted 7.4.0-rc2-kernsight-kasan #1
> task:(udev-worker) state:D stack:25208 pid:9713 tgid:9713 ppid:4787
> 
> Call Trace:
>  <TASK>
>  __schedule+0x1425/0x6840 kernel/sched/core.c:7389
>  schedule+0xec/0x2d0 kernel/sched/core.c:7481
>  __mutex_lock_common kernel/locking/mutex.c:726 [inline]
>  __mutex_lock+0xe0c/0x1dd0 kernel/locking/mutex.c:821
>  bdev_open+0x3fa/0xe50 block/bdev.c:990
>  blkdev_open+0x355/0x500 block/fops.c:686
>  do_dentry_open+0x693/0x14f0 fs/open.c:1000
>  vfs_open_consume+0xd1/0x460 fs/open.c:1131
>  do_open fs/namei.c:4851 [inline]
>  path_openat+0x1c1b/0x2240 fs/namei.c:5016
>  do_file_open+0x21b/0x470 fs/namei.c:5045
>  do_sys_openat2+0x13e/0x1d0 fs/open.c:1443
>  __x64_sys_openat+0x144/0x200 fs/open.c:1460
>  entry_SYSCALL_64_after_hwframe+0x77/0x7f
>  </TASK>
> 
> INFO: task syz.2.29:12841 blocked for more than 143 seconds.
> task:syz.2.29 state:D stack:27784 pid:12841 tgid:12818 ppid:10317
> 
> Call Trace:
>  <TASK>
>  __mutex_lock
>  bdev_open        block/bdev.c:990
>  blkdev_open      block/fops.c:686
>  do_dentry_open
>  path_openat
>  __x64_sys_openat
>  </TASK>
> ```
> 
> **Lock observations:**
> 
> In the verification report, the lock dump identifies the ublk
> partition-scan worker:
> 
> ```text
> locks held by kworker/2:2/1815:
> 
> #1: ((work_completion)(&ub->partition_scan_work))
>     at process_one_work
> 
> #2: ff1100017b50c350 (&disk->open_mutex)
>     at ublk_partition_scan_work+0xdb/0x170
>     drivers/block/ublk_drv.c:2478
> ```
> 
> The same mutex address appears in the lock records for the two blocked tasks:
> 
> ```text
> locks held by (udev-worker)/9713:
>     ff1100017b50c350 (&disk->open_mutex)
>     at bdev_open+0x3fa/0xe50
> 
> locks held by syz.2.29/12841:
>     ff1100017b50c350 (&disk->open_mutex)
>     at bdev_open+0x3fa/0xe50
> ```
> 
> A separate fuzzing occurrence shows a similar relationship:
> 
> ```text
> kworker/3:2/1260:
>     ublk_partition_scan_work
>     (&disk->open_mutex)
> 
> (udev-worker)/9505:
>     bdev_open
>     (&disk->open_mutex)
> 
> syz.4.2301/42546:
>     bdev_open
>     (&disk->open_mutex)
> ```
> 
> The repeated association between the blocked openers and
> `ublk_partition_scan_work()` suggests that the partition scan may be
> involved in the prolonged wait.
> 
> However, the current logs do not establish why the partition-scan
> worker fails to complete. In particular, I have not determined whether
> it is waiting for outstanding I/O from the userspace ublk backend or
> encountering another condition.
> 
> **Reproduction:**
> 
> The repository contains a minimized target program together with a
> recorded state-reconstruction sequence.
> 
> The minimized `repro/repro.prog` is not sufficient by itself to
> reliably reproduce the hang. The state-primed reproduction uses the
> recorded sequence and the target program in the same VM.
> 
> The relevant files are:
> 
> - `repro/repro.prog`: minimized target program
> - `repro/repro.priming.prog`: recorded state-reconstruction sequence
> - `repro/repro.kernsight.prog`: combined state-primed reproduction input
> - `repro/repro.README.txt`: detailed reproduction instructions
> 
> The state-primed reproduction can be attempted using:
> 
> ```bash
> ./syz-execprog \
>     -executor=./syz-executor \
>     -procs=1 \
>     -repeat=1 \
>     ./repro.kernsight.prog
> ```
> 
> The guest should be booted using the kernel configuration and QEMU
> settings documented in the repository. The same syzkaller environment
> should be used for the reproduction.
> 
> The verification run recorded a hung-task report, and the repository
> contains the corresponding execution log and report.
> 
> One caveat is that the udev worker appears to have entered the blocked
> state before the final minimized target program was executed.
> Therefore, I do not claim that the minimized target program alone
> causes the hang. The preceding state-reconstruction sequence may be
> responsible for establishing the condition.
> 
> **Observed results:**
> 
> - Two occurrences during concurrent syzkaller fuzzing
> - One recorded successful state-primed verification run
> - Both the udev worker and a syzkaller executor were observed waiting
> in `bdev_open()`
> - The lock dumps associate the same `disk->open_mutex` with the ublk
> partition-scan workqueue
> - The VM remained running with `hung_task_panic=0`
> 
> **Repository contents:**
> 
> Reproduction artifacts:
> 
> - `repro/repro.prog`
> - `repro/repro.kernsight.prog`
> - `repro/repro.priming.prog`
> - `repro/repro.README.txt`
> - `repro/repro.report`
> - `repro/repro.log`
> - `repro/repro.state.json`
> - `repro/repro.context.json`
> - `repro/repro.attempts.json`
> - `repro/repro.outcomes.jsonl`
> 
> Original fuzzing evidence:
> 
> - `logs/log0`, `logs/log1`
> - `logs/report0`, `logs/report1`
> - `logs/machineInfo0`, `logs/machineInfo1`
> - `logs/pre_state0`, `logs/pre_state1`
> - `logs/crash-excerpt.txt`
> 
> Environment:
> 
> - `config-7.4.0-rc2-kernsight-kasan`: full kernel configuration
> - `syzkaller.cfg`: syzkaller and QEMU configuration
> 
> **Related upstream work:**
> 
> I noticed the earlier ublk fixes concerning canceled I/O commands and
> partition-scan hangs, including the change titled:
> 
> "ublk: refuse to go live after an io command was canceled"
> 
> This issue was observed on v7.4-rc2, but I have not established
> whether it involves the same underlying condition or a different
> problem.
> 
> Could you please advise whether this corresponds to a known issue, or
> whether additional diagnostics from the partition-scan worker or the
> ublk I/O queues would be useful?


Thanks for the report!

From the ublk program in your log, the same process does the following:

	syz_ublk_setup_queues()			/* FETCH */
	START_DEV
	syz_open_dev$ublk_bdev()		/* open /dev/ublkbN */
	syz_ublk_process_io()			/* handle io */

After START_DEV, ublk scans partitions from one work function, which
grabs disk->open_mutex and reads partition table from the ublk server.
But the server is opening its own disk before handling any io, so it
waits on the same open_mutex in bdev_open(), and the read can't be
served. That is one deadlock in the test program itself.

That is definitely one *wrong* ublk server implementation, ublk server
can't issue IO from its own frontend(`/dev/ublkbN`).


Thanks,
Ming