[BUG] tpm: prolonged stall in vtpm_proxy_fops_release() on Linux 7.3

[BUG] tpm: prolonged stall in vtpm_proxy_fops_release() on Linux 7.3

[铮]

Hi Peter, Jarkko and Jason,

I observed a task hang in vtpm_proxy_fops_release() while fuzzing the
vTPM proxy with syzkaller. I reproduced it on Linux 7.3.0: the campaign
observed the hang 4 times; replaying the stored sequence reproduced it
once in an automated replay and three times in manual replays. In every
case a process is blocked for 143+ seconds in
vtpm_proxy_fops_release() -> vtpm_proxy_work_stop() -> flush_work(),
while at the same time the sysrq-t dumps show a chip-register work
(vtpm_proxy_work -> tpm_chip_register() -> tpm_chip_start() ->
vtpm_proxy_request_locality()) running on the tpm-vtpm workqueue plus a
backlog of further queued chip-register works. Which queued work the
blocked release is waiting for is unconfirmed. All logs end while the
task is still blocked, so I cannot say how long the stall ultimately
lasts; I am not claiming a permanent deadlock.

Environment
-----------
- Kernel: 7.3.0-kernsight-kasan #1 SMP PREEMPT_DYNAMIC, built 2026-10-20,
  x86_64. Vanilla 7.3.0 source tarball (sha256
  87363c24f10e06321eb362765521094e94471f6bff8bce2d6b3e40f39f004953),
  built with KASAN/KCOV. No TPM changes were made for this testing
  (drivers/char/tpm/ matches vanilla apart from build artifacts).
- Guest: QEMU/KVM, 4 vCPU, 6 GiB, 8 scratch virtio disks, 2 NVDIMM/pmem
  devices, 2 NUMA nodes (the syzkaller standard environment). A fresh VM
  was used for every reproduction run.
- Syzkaller: revision 853b104bb454141a5b0fc0e4218ec78e9f34ffb1+
  (2026-09-18). The "+" denotes local KernSight integration changes
  (seed ordering/state feedback: pkg/kernsight, prog/kernsight_*,
  syz-manager/kernsight_*, pkg/execbackend, snapshot support in vm/qemu).
  They do not change the TPM syscall descriptions or the executor
  semantics.
- Reproduction: the campaign observed the hang 4 times; replaying the
  stored programs reproduced it once in an automated replay, three
  times as a single 50-program sequence, and once with the two-step
  form above (one of the manual replays is captured only as a sysrq-t
  dump). Four runs on a bare fresh VM with a *different kernel command
  line* and without the scratch/pmem/NUMA layout recorded no hang --
  since the command line differs as well, these runs cannot attribute
  the effect to the device layout.

Kernel command line (full text in the repository, cmdline/cmdline-7.3.txt):
  root=/dev/sda console=ttyS0 fstab=no rw earlyprintk=serial net.ifnames=0
  oops=panic panic_on_warn=1 panic=86400 hung_task_panic=1 softlockup_panic=1
  nmi_watchdog=panic ... page_owner=on secretmem.enable=1
  dummy_hcd.num=16 max_loop=32 nbd.nbds_max=32 kunit.autorun=0
  (for the live-inspection runs, hung_task_panic=0 was used instead; see
  cmdline/cmdline-7.3-retest.txt)

Reproduction
------------
Two equivalent ways are verified; the two-step form is the simplest.

From the host:

  scp syz-execprog syz-executor repro.priming.prog repro.prog root@guest:/tmp/
  ssh root@guest
  cd /tmp && chmod +x syz-execprog syz-executor

Step 1 -- run the state-construction sequence (49 programs) once:

  ./syz-execprog -executor=/tmp/syz-executor -procs=1 -repeat=1
/tmp/repro.priming.prog

It runs for ~20 s and exits normally. When its executor process exits,
the vtpm fds it holds are released and the release path hangs: the
hung-task monitor reports at t≈267 s (logs/serial-twostep.log):

  INFO: task syz.0.33:10843 blocked for more than 143 seconds.
        Blocked by coredump.

while `ps` shows `kworker/X:Y+tpm-vtpm` and the executor task both in
state D (sysrq-t dump: logs/dmesg-sysrq-twostep.txt), and the serial log
shows `tpm tpm1/tpm2: Operation Timed out` while the release is blocked.
Step 2 -- optionally run the crashing testcase afterwards
(./syz-execprog ... /tmp/repro.prog); on the already-blocked VM it
simply runs and exits.

The single-file form replays the state construction and the crashing
testcase as one 50-program sequence (logs/execprog-verify4.out:
"parsed 50 programs"):

  ./syz-execprog -executor=/tmp/syz-executor -procs=1 -repeat=1
/tmp/repro.kernsight.prog

In both forms programs run sequentially (procs=1, repeat=1,
sandbox=none), but a program can leave asynchronous kernel-side work
behind. The sequence creates several vTPM proxy devices and
closes/exits over them. The hang appears when a closing task's release
path races a still-running or queued chip-register work. The
hung-task monitor then reports (t≈480 s in logs/serial-verify4.log):

  INFO: task syz.0.33:10857 blocked for more than 143 seconds.
        Not tainted 7.3.0-kernsight-kasan #1
        Blocked by coredump.
  task:syz.0.33        state:D stack:26264 pid:10857 tgid:10857 ppid:10246
  Call Trace:
   <TASK>
   __schedule+0x134d/0x6880
   schedule_timeout+0x24a/0x290
   __wait_for_common+0x1d3/0x4e0
   __flush_work+0x6ab/0xd90
   vtpm_proxy_work_stop     drivers/char/tpm/tpm_vtpm_proxy.c:470 [inline]
   vtpm_proxy_delete_device drivers/char/tpm/tpm_vtpm_proxy.c:590 [inline]
   vtpm_proxy_fops_release+0xa8/0x1d0 drivers/char/tpm/tpm_vtpm_proxy.c:239
   __fput+0x408/0xb60
   task_work_run+0x170/0x260
   do_exit+0x90c/0x2b60
   ...

  Showing all locks held in the system:
   locks held by kworker/3:2/1235: 2, on CPU#3:
    #0: ff110000267a8540 ((wq_completion)tpm-vtpm){+.+.}-{0:0}
    #1: ffa0000006cafd08 ((work_completion)(&proxy_dev->work)){+.+.}-{0:0}

While the task was blocked I captured full task dumps with
`echo t > /proc/sysrq-trigger` (hung_task_panic=0 variant; three dumps
included). In the first dump the kworker running a vtpm_proxy_work is
itself blocked in the transmit loop:

  task:kworker/2:3     state:D stack:24648 pid:9703
  Workqueue: tpm-vtpm vtpm_proxy_work
  Call Trace:
   <TASK>
   schedule+0xec/0x2d0
   schedule_hrtimeout_range_clock+0x11d/0x310
   usleep_range_state+0x178/0x220
   ? vtpm_proxy_tpm_req_canceled
   tpm_transmit+0x4bf/0xb90
   tpm_transmit_cmd+0x62/0x1f0
   vtpm_proxy_request_locality+0x140/0x280
   tpm_chip_start+0x18f/0x340
   tpm_chip_bootstrap+0x80/0x1c0
   tpm_chip_register+0x27/0x4d0
   vtpm_proxy_work+0x41/0x100
   process_one_work+0xa9c/0x1dc0
   ...

All three dumps also show that the tpm-vtpm workqueue carries a backlog
of further chip-register works behind the running one
(logs/dmesg-sysrq-verify3.txt):

  workqueue tpm-vtpm: flags=0x40108
    pwq 6: cpus=1 node=0 active=1 refcnt=33
      in-flight: 24:vtpm_proxy_work for 75s
      inactive: 4*vtpm_proxy_work BAR(10837), 26*vtpm_proxy_work

(logs/dmesg-sysrq-verify4.txt: in-flight for 109 s,
inactive: 29*vtpm_proxy_work BAR(10857).) With this backlog I cannot
tell which specific queued work the blocked release is waiting for; the
running work shown in the stacks may or may not be that one.

While the release was blocked, the serial log shows
"tpm tpm3: Operation Timed out" / "tpm tpm31: Operation Timed out"
lines. All included logs end while the task is still blocked, so I have
no log evidence of the release completing; I have also not established
an upper bound for the stall.

Two entry paths were observed for the blocked release:
- process exit: do_exit() -> task_work_run() -> __fput()
  (campaign reports, the automated reproduction, dmesg-sysrq-verify4.txt)
- close_range() returning to user space: task_work_run() -> __fput()
  (dmesg-sysrq-verify3.txt)

The minimized single-program testcase does not reproduce the hang; the
full 50-program sequence above is the reproducer. I have not isolated
which preceding operations or timing conditions are required.

Question
--------
vtpm_proxy_tpm_req_canceled() already exists and the driver marks the
device as being torn down before flush_work(); from the captured dumps
I cannot tell why the wait nevertheless persists for minutes. Should
the release path be bounded, or should the in-flight chip-register work
be ended more promptly once the proxy is being torn down? I am happy to
test proposed patches.

Materials (the reproducer, the sysrq-t task dumps, full serial logs,
the syz-execprog output, kernel config, command lines, a sha256 manifest,
and a verify.sh that re-checks the numbers quoted here):
  https://github.com/Zheng-zheng-codes/vtpm-proxy-release-hang-repro

Repository contents
-------------------
  repro/repro.priming.prog           state-construction sequence (49 programs)
  repro/repro.prog                   crashing testcase (use after the
priming run)
  repro/repro.kernsight.prog         single-file form (50 programs)
  logs/vtpm-sysrq-t-all-tasks.txt    sysrq-t dump: kworker stack +
backlog (run 1)
  logs/dmesg-sysrq-verify3.txt       sysrq-t dump: backlog,
close_range entry path
  logs/dmesg-sysrq-verify4.txt       sysrq-t dump: backlog, do_exit entry path
  logs/serial-7.3-auto-repro.log     automated state-primed reproduction
  logs/serial-7.3-manual-repro.log   manual replay, run 1
  logs/serial-verify4.log            manual replay, run 2 (serial)
  logs/serial-twostep.log            two-step reproduction (serial)
  logs/execprog-twostep-step1.out    two-step step 1 output ("parsed
49 programs")
  logs/dmesg-sysrq-twostep.txt       two-step sysrq-t dump
  logs/execprog-verify4.out          run 2 syz-execprog stdout
("parsed 50 programs")
  logs/serial-7.3-bare-no-trigger-*.log  4 bare-VM runs (no hang recorded)
  logs/report-7.3-*.txt, log-7.3-*.txt   the 4 campaign instances
  logs/crash-title.txt               crash title
  kernel/kernel-7.3.config           kernel configuration
  cmdline/cmdline-7.3.txt            kernel command line (syzkaller
standard set)
  cmdline/cmdline-7.3-retest.txt     verification variant (hung_task_panic=0)
  verify.sh                          re-checks the numbers quoted in this mail
  MANIFEST.sha256                    sha256 of every file (except itself)

I am happy to provide additional information and test proposed patches.

Best regards,
Zheng Zheng

Re: [BUG] tpm: prolonged stall in vtpm_proxy_fops_release() on Linux 7.3

[Jarkko Sakkinen]

On Sun, Nov 01, 2026 at 09:22:30AM +0800, 铮 wrote:
> Hi Peter, Jarkko and Jason,

Adding Stefan. 

> 
> I observed a task hang in vtpm_proxy_fops_release() while fuzzing the
> vTPM proxy with syzkaller. I reproduced it on Linux 7.3.0: the campaign
> observed the hang 4 times; replaying the stored sequence reproduced it
> once in an automated replay and three times in manual replays. In every
> case a process is blocked for 143+ seconds in
> vtpm_proxy_fops_release() -> vtpm_proxy_work_stop() -> flush_work(),
> while at the same time the sysrq-t dumps show a chip-register work
> (vtpm_proxy_work -> tpm_chip_register() -> tpm_chip_start() ->
> vtpm_proxy_request_locality()) running on the tpm-vtpm workqueue plus a
> backlog of further queued chip-register works. Which queued work the
> blocked release is waiting for is unconfirmed. All logs end while the
> task is still blocked, so I cannot say how long the stall ultimately
> lasts; I am not claiming a permanent deadlock.
> 
> Environment
> -----------
> - Kernel: 7.3.0-kernsight-kasan #1 SMP PREEMPT_DYNAMIC, built 2026-10-20,
>   x86_64. Vanilla 7.3.0 source tarball (sha256
>   87363c24f10e06321eb362765521094e94471f6bff8bce2d6b3e40f39f004953),
>   built with KASAN/KCOV. No TPM changes were made for this testing
>   (drivers/char/tpm/ matches vanilla apart from build artifacts).
> - Guest: QEMU/KVM, 4 vCPU, 6 GiB, 8 scratch virtio disks, 2 NVDIMM/pmem
>   devices, 2 NUMA nodes (the syzkaller standard environment). A fresh VM
>   was used for every reproduction run.
> - Syzkaller: revision 853b104bb454141a5b0fc0e4218ec78e9f34ffb1+
>   (2026-09-18). The "+" denotes local KernSight integration changes
>   (seed ordering/state feedback: pkg/kernsight, prog/kernsight_*,
>   syz-manager/kernsight_*, pkg/execbackend, snapshot support in vm/qemu).
>   They do not change the TPM syscall descriptions or the executor
>   semantics.
> - Reproduction: the campaign observed the hang 4 times; replaying the
>   stored programs reproduced it once in an automated replay, three
>   times as a single 50-program sequence, and once with the two-step
>   form above (one of the manual replays is captured only as a sysrq-t
>   dump). Four runs on a bare fresh VM with a *different kernel command
>   line* and without the scratch/pmem/NUMA layout recorded no hang --
>   since the command line differs as well, these runs cannot attribute
>   the effect to the device layout.
> 
> Kernel command line (full text in the repository, cmdline/cmdline-7.3.txt):
>   root=/dev/sda console=ttyS0 fstab=no rw earlyprintk=serial net.ifnames=0
>   oops=panic panic_on_warn=1 panic=86400 hung_task_panic=1 softlockup_panic=1
>   nmi_watchdog=panic ... page_owner=on secretmem.enable=1
>   dummy_hcd.num=16 max_loop=32 nbd.nbds_max=32 kunit.autorun=0
>   (for the live-inspection runs, hung_task_panic=0 was used instead; see
>   cmdline/cmdline-7.3-retest.txt)
> 
> Reproduction
> ------------
> Two equivalent ways are verified; the two-step form is the simplest.
> 
> From the host:
> 
>   scp syz-execprog syz-executor repro.priming.prog repro.prog root@guest:/tmp/
>   ssh root@guest
>   cd /tmp && chmod +x syz-execprog syz-executor
> 
> Step 1 -- run the state-construction sequence (49 programs) once:
> 
>   ./syz-execprog -executor=/tmp/syz-executor -procs=1 -repeat=1
> /tmp/repro.priming.prog
> 
> It runs for ~20 s and exits normally. When its executor process exits,
> the vtpm fds it holds are released and the release path hangs: the
> hung-task monitor reports at t≈267 s (logs/serial-twostep.log):
> 
>   INFO: task syz.0.33:10843 blocked for more than 143 seconds.
>         Blocked by coredump.
> 
> while `ps` shows `kworker/X:Y+tpm-vtpm` and the executor task both in
> state D (sysrq-t dump: logs/dmesg-sysrq-twostep.txt), and the serial log
> shows `tpm tpm1/tpm2: Operation Timed out` while the release is blocked.
> Step 2 -- optionally run the crashing testcase afterwards
> (./syz-execprog ... /tmp/repro.prog); on the already-blocked VM it
> simply runs and exits.
> 
> The single-file form replays the state construction and the crashing
> testcase as one 50-program sequence (logs/execprog-verify4.out:
> "parsed 50 programs"):
> 
>   ./syz-execprog -executor=/tmp/syz-executor -procs=1 -repeat=1
> /tmp/repro.kernsight.prog
> 
> In both forms programs run sequentially (procs=1, repeat=1,
> sandbox=none), but a program can leave asynchronous kernel-side work
> behind. The sequence creates several vTPM proxy devices and
> closes/exits over them. The hang appears when a closing task's release
> path races a still-running or queued chip-register work. The
> hung-task monitor then reports (t≈480 s in logs/serial-verify4.log):
> 
>   INFO: task syz.0.33:10857 blocked for more than 143 seconds.
>         Not tainted 7.3.0-kernsight-kasan #1
>         Blocked by coredump.
>   task:syz.0.33        state:D stack:26264 pid:10857 tgid:10857 ppid:10246
>   Call Trace:
>    <TASK>
>    __schedule+0x134d/0x6880
>    schedule_timeout+0x24a/0x290
>    __wait_for_common+0x1d3/0x4e0
>    __flush_work+0x6ab/0xd90
>    vtpm_proxy_work_stop     drivers/char/tpm/tpm_vtpm_proxy.c:470 [inline]
>    vtpm_proxy_delete_device drivers/char/tpm/tpm_vtpm_proxy.c:590 [inline]
>    vtpm_proxy_fops_release+0xa8/0x1d0 drivers/char/tpm/tpm_vtpm_proxy.c:239
>    __fput+0x408/0xb60
>    task_work_run+0x170/0x260
>    do_exit+0x90c/0x2b60
>    ...
> 
>   Showing all locks held in the system:
>    locks held by kworker/3:2/1235: 2, on CPU#3:
>     #0: ff110000267a8540 ((wq_completion)tpm-vtpm){+.+.}-{0:0}
>     #1: ffa0000006cafd08 ((work_completion)(&proxy_dev->work)){+.+.}-{0:0}
> 
> While the task was blocked I captured full task dumps with
> `echo t > /proc/sysrq-trigger` (hung_task_panic=0 variant; three dumps
> included). In the first dump the kworker running a vtpm_proxy_work is
> itself blocked in the transmit loop:
> 
>   task:kworker/2:3     state:D stack:24648 pid:9703
>   Workqueue: tpm-vtpm vtpm_proxy_work
>   Call Trace:
>    <TASK>
>    schedule+0xec/0x2d0
>    schedule_hrtimeout_range_clock+0x11d/0x310
>    usleep_range_state+0x178/0x220
>    ? vtpm_proxy_tpm_req_canceled
>    tpm_transmit+0x4bf/0xb90
>    tpm_transmit_cmd+0x62/0x1f0
>    vtpm_proxy_request_locality+0x140/0x280
>    tpm_chip_start+0x18f/0x340
>    tpm_chip_bootstrap+0x80/0x1c0
>    tpm_chip_register+0x27/0x4d0
>    vtpm_proxy_work+0x41/0x100
>    process_one_work+0xa9c/0x1dc0
>    ...
> 
> All three dumps also show that the tpm-vtpm workqueue carries a backlog
> of further chip-register works behind the running one
> (logs/dmesg-sysrq-verify3.txt):
> 
>   workqueue tpm-vtpm: flags=0x40108
>     pwq 6: cpus=1 node=0 active=1 refcnt=33
>       in-flight: 24:vtpm_proxy_work for 75s
>       inactive: 4*vtpm_proxy_work BAR(10837), 26*vtpm_proxy_work
> 
> (logs/dmesg-sysrq-verify4.txt: in-flight for 109 s,
> inactive: 29*vtpm_proxy_work BAR(10857).) With this backlog I cannot
> tell which specific queued work the blocked release is waiting for; the
> running work shown in the stacks may or may not be that one.
> 
> While the release was blocked, the serial log shows
> "tpm tpm3: Operation Timed out" / "tpm tpm31: Operation Timed out"
> lines. All included logs end while the task is still blocked, so I have
> no log evidence of the release completing; I have also not established
> an upper bound for the stall.
> 
> Two entry paths were observed for the blocked release:
> - process exit: do_exit() -> task_work_run() -> __fput()
>   (campaign reports, the automated reproduction, dmesg-sysrq-verify4.txt)
> - close_range() returning to user space: task_work_run() -> __fput()
>   (dmesg-sysrq-verify3.txt)
> 
> The minimized single-program testcase does not reproduce the hang; the
> full 50-program sequence above is the reproducer. I have not isolated
> which preceding operations or timing conditions are required.
> 
> Question
> --------
> vtpm_proxy_tpm_req_canceled() already exists and the driver marks the
> device as being torn down before flush_work(); from the captured dumps
> I cannot tell why the wait nevertheless persists for minutes. Should
> the release path be bounded, or should the in-flight chip-register work
> be ended more promptly once the proxy is being torn down? I am happy to
> test proposed patches.
> 
> Materials (the reproducer, the sysrq-t task dumps, full serial logs,
> the syz-execprog output, kernel config, command lines, a sha256 manifest,
> and a verify.sh that re-checks the numbers quoted here):
>   https://github.com/Zheng-zheng-codes/vtpm-proxy-release-hang-repro

Thanks for making it public on one go. Here it is the best pick.

> 
> Repository contents
> -------------------
>   repro/repro.priming.prog           state-construction sequence (49 programs)
>   repro/repro.prog                   crashing testcase (use after the
> priming run)
>   repro/repro.kernsight.prog         single-file form (50 programs)
>   logs/vtpm-sysrq-t-all-tasks.txt    sysrq-t dump: kworker stack +
> backlog (run 1)
>   logs/dmesg-sysrq-verify3.txt       sysrq-t dump: backlog,
> close_range entry path
>   logs/dmesg-sysrq-verify4.txt       sysrq-t dump: backlog, do_exit entry path
>   logs/serial-7.3-auto-repro.log     automated state-primed reproduction
>   logs/serial-7.3-manual-repro.log   manual replay, run 1
>   logs/serial-verify4.log            manual replay, run 2 (serial)
>   logs/serial-twostep.log            two-step reproduction (serial)
>   logs/execprog-twostep-step1.out    two-step step 1 output ("parsed
> 49 programs")
>   logs/dmesg-sysrq-twostep.txt       two-step sysrq-t dump
>   logs/execprog-verify4.out          run 2 syz-execprog stdout
> ("parsed 50 programs")
>   logs/serial-7.3-bare-no-trigger-*.log  4 bare-VM runs (no hang recorded)
>   logs/report-7.3-*.txt, log-7.3-*.txt   the 4 campaign instances
>   logs/crash-title.txt               crash title
>   kernel/kernel-7.3.config           kernel configuration
>   cmdline/cmdline-7.3.txt            kernel command line (syzkaller
> standard set)
>   cmdline/cmdline-7.3-retest.txt     verification variant (hung_task_panic=0)
>   verify.sh                          re-checks the numbers quoted in this mail
>   MANIFEST.sha256                    sha256 of every file (except itself)
> 
> I am happy to provide additional information and test proposed patches.
> 
> Best regards,
> Zheng Zheng

Thank you for reporting this.

Br, Jarkko

Re: [BUG] tpm: prolonged stall in vtpm_proxy_fops_release() on Linux 7.3

[Stefan Berger]

On 11/1/26 12:36 PM, Jarkko Sakkinen wrote:
> On Sun, Nov 01, 2026 at 09:22:30AM +0800, 铮 wrote:
>> Hi Peter, Jarkko and Jason,
> 
> Adding Stefan.
> 
>>
>> I observed a task hang in vtpm_proxy_fops_release() while fuzzing the
>> vTPM proxy with syzkaller. I reproduced it on Linux 7.3.0: the campaign
>> observed the hang 4 times; replaying the stored sequence reproduced it
>> once in an automated replay and three times in manual replays. In every
>> case a process is blocked for 143+ seconds in
>> vtpm_proxy_fops_release() -> vtpm_proxy_work_stop() -> flush_work(),
>> while at the same time the sysrq-t dumps show a chip-register work
>> (vtpm_proxy_work -> tpm_chip_register() -> tpm_chip_start() ->
>> vtpm_proxy_request_locality()) running on the tpm-vtpm workqueue plus a
>> backlog of further queued chip-register works. Which queued work the
>> blocked release is waiting for is unconfirmed. All logs end while the
>> task is still blocked, so I cannot say how long the stall ultimately
>> lasts; I am not claiming a permanent deadlock.
>>
>> Environment
>> -----------
>> - Kernel: 7.3.0-kernsight-kasan #1 SMP PREEMPT_DYNAMIC, built 2026-10-20,
>>    x86_64. Vanilla 7.3.0 source tarball (sha256
>>    87363c24f10e06321eb362765521094e94471f6bff8bce2d6b3e40f39f004953),
>>    built with KASAN/KCOV. No TPM changes were made for this testing
>>    (drivers/char/tpm/ matches vanilla apart from build artifacts).
>> - Guest: QEMU/KVM, 4 vCPU, 6 GiB, 8 scratch virtio disks, 2 NVDIMM/pmem
>>    devices, 2 NUMA nodes (the syzkaller standard environment). A fresh VM
>>    was used for every reproduction run.
>> - Syzkaller: revision 853b104bb454141a5b0fc0e4218ec78e9f34ffb1+
>>    (2026-09-18). The "+" denotes local KernSight integration changes
>>    (seed ordering/state feedback: pkg/kernsight, prog/kernsight_*,
>>    syz-manager/kernsight_*, pkg/execbackend, snapshot support in vm/qemu).
>>    They do not change the TPM syscall descriptions or the executor
>>    semantics.
>> - Reproduction: the campaign observed the hang 4 times; replaying the
>>    stored programs reproduced it once in an automated replay, three
>>    times as a single 50-program sequence, and once with the two-step
>>    form above (one of the manual replays is captured only as a sysrq-t
>>    dump). Four runs on a bare fresh VM with a *different kernel command

Four runs out of four?

>>    line* and without the scratch/pmem/NUMA layout recorded no hang --

Does this mean that the 'scratch/pemem/NUMA layout' is required to 
reproduce the hang?

>>    since the command line differs as well, these runs cannot attribute
>>    the effect to the device layout.

What is 'device layout'?

Re: [BUG] tpm: prolonged stall in vtpm_proxy_fops_release() on Linux 7.3

[铮]

Hi Stefan,

To make the reproduction evidence easier to inspect, the repository
(https://github.com/Zheng-zheng-codes/vtpm-proxy-release-hang-repro)
contains five positive reproductions in total:

- Four replays of the 50-program single-file reproducer (one automated
  and three manual).
- One separate two-step reproduction (a 49-program state-construction
  run, with the hang occurring when its executor exits).

The serial logs show the hung-task reports, with the release path
blocked in flush_work(). The accompanying sysrq-t dumps provide
snapshots of the tpm-vtpm workqueue. The first captures the full
running chip-register work stack, the blocked release, and the
workqueue backlog. The other dumps also show the blocked release and
the in-flight work/backlog state, although the full worker stack is
not available in those captures.

The files are:

Serial logs (5 positive reproductions):

  logs/serial-7.3-auto-repro.log    -- automated replay
  logs/serial-7.3-manual-repro.log  -- manual replay 1
  logs/serial-verify3.log           -- manual replay 2
  logs/serial-verify4.log           -- manual replay 3 (with
                                       syz-execprog stdout in
                                       logs/execprog-verify4.out)
  logs/serial-twostep.log           -- two-step replay (with
                                       syz-execprog stdout in
                                       logs/execprog-twostep-step1.out)

Sysrq-t task dumps (4):

  logs/vtpm-sysrq-t-all-tasks.txt   -- replay 1: full kworker stack,
                                       blocked release, and backlog
  logs/dmesg-sysrq-verify3.txt      -- replay 2: backlog and blocked
                                       release
  logs/dmesg-sysrq-verify4.txt      -- replay 3: backlog and blocked
                                       release
  logs/dmesg-sysrq-twostep.txt      -- two-step replay: in-flight work,
                                       backlog, and blocked release

Campaign instances:

  logs/report-7.3-*.txt
  logs/log-7.3-*.txt

Answering your questions in order:

> Four runs out of four?

Yes. Those four were the negative controls, not the positive
reproductions. The sequence was run four times on four freshly booted
bare VMs, and no hang was recorded in any of them:

  logs/serial-7.3-bare-no-trigger-1..4.log

> Does this mean that the 'scratch/pmem/NUMA layout' is required to
> reproduce the hang?

That was not established by the original comparison. The bare-VM runs
differed from the positive runs in two variables at once: the kernel
command line and the QEMU device layout. Therefore, the original
comparison could not isolate the effect of either factor.

To investigate this, I ran a controlled 2x2 test matrix using the same
50-program sequence, the same kernel, hung_task_panic=0, and a fresh VM
for each run.

The results were:

  Full command line + full layout    4/4 hangs (one automated and
                                     three manual replays)
  Full command line + bare layout    0/2
  Minimal command line + full layout 0/2
  Minimal command line + bare layout 0/4

In this test matrix, the hang was reproduced only when both the full
kernel command line and the full device layout were used together.
Neither one alone was sufficient to reproduce it in the tested runs.

The 4/4 result counts the four 50-program single-file reproductions
and does not include the separate two-step reproduction.

My current hypothesis is that the combination affects guest execution
timing. Possible contributors include page_owner overhead,
initialization of the additional devices, and NUMA-related I/O. This
may increase the likelihood that chip-register work is still running
or queued when the closing task reaches flush_work(). However, the
exact mechanism has not yet been established.

The logs and per-run kernel command lines are available in the
repository under logs/controlled/ and cmdline/.

> What is 'device layout'?

By device layout, I mean the virtual hardware configuration and
topology presented to the guest by QEMU.

The full layout includes, in addition to the root IDE disk:

  - 8 scratch virtio disks (2 x 16 GiB and 6 x 12 GiB, created with
    `qemu-img create -f qcow2 ...` by the QEMU wrapper for each VM;
    the wrapper excerpt, the sizes and a `qemu-img info` output are in
    cmdline/qemu-wrapper-scratch-create.txt);
  - 1 additional virtio disk used as a write log;
  - 2 NVDIMM/pmem devices (1 GiB memory-backend files each);
  - 2 NUMA nodes (2 vCPUs and 3 GiB of memory per node).

The bare layout has only the root IDE disk as a block device, with no
additional virtio disks or NVDIMM/pmem devices, and a single NUMA node.

The repository contains a captured QEMU invocation for the standard
run (cmdline/cmdline-7.3.txt, with hung_task_panic=1), the kernel
command line used for live retesting
(cmdline/cmdline-7.3-retest.txt, with hung_task_panic=0), and
sanitized QEMU argument templates for the full and bare layouts:

  cmdline/qemu-args-full-layout.txt
  cmdline/qemu-args-bare-layout.txt

The controlled retest uses hung_task_panic=0; the per-configuration
command lines are also available in cmdline/.

I hope this clarifies the reproduction conditions and the distinction
between the kernel command line and the QEMU device layout.

Best regards,
Zheng Zheng