From: "Vineeth Pillai (Google)" <vineeth@bitbyteword.org>
To: sonam.sanju@intel.corp-partner.google.com
Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org,
pbonzini@redhat.com, seanjc@google.com, sonam.sanju@intel.com,
dmaluka@chromium.org,
"Vineeth Pillai (Google)" <vineeth@bitbyteword.org>
Subject: Re: [PATCH] KVM: irqfd: fix shutdown deadlock by moving SRCU sync outside resampler_lock
Date: Fri, 20 Mar 2026 08:56:33 -0400 [thread overview]
Message-ID: <20260320125633.2290675-1-vineeth@bitbyteword.org> (raw)
In-Reply-To: <20260317162754.3231300-1-sonam.sanju@intel.corp-partner.google.com>
Hi Sonam,
> irqfd_resampler_shutdown() calls synchronize_srcu_expedited() while
> holding kvm->irqfds.resampler_lock. This can deadlock when multiple
> irqfd_shutdown workers run concurrently on the kvm-irqfd-cleanup
> workqueue during VM teardown (e.g. crosvm shutdown on Android):
>
> CPU A (mutex holder) CPU B/C/D (mutex waiters)
> irqfd_shutdown() irqfd_shutdown()
> irqfd_resampler_shutdown() irqfd_resampler_shutdown()
> mutex_lock(resampler_lock) <---- mutex_lock(resampler_lock) // BLOCKED
> list_del_rcu(...) ...blocked...
> synchronize_srcu_expedited() // Waiters block workqueue,
> // waits for SRCU grace preventing SRCU grace
> // period which requires period from completing
> // workqueue progress --- DEADLOCK ---
I think we might have this issue in the kvm_irqfd_assign path as well
where synchronize_srcu_expedited is called with the resampler_lock
held. I saw similar lockup during a stress test where VMs were created
and destroyed continously. I could see one task waiting on SRCU GP:
[ T93] task:crosvm_security state:D stack:0 pid:8215 tgid:8215 ppid:1 task_flags:0x400000 flags:0x00080002.
[ T93] Call Trace:
[ T93] <TASK>
[ T93] __schedule+0x87a/0xd60
[ T93] schedule+0x5e/0xe0
[ T93] schedule_timeout+0x2e/0x130
[ T93] ? queue_delayed_work_on+0x7f/0xd0
[ T93] wait_for_common+0xf7/0x1f0
[ T93] synchronize_srcu_expedited+0x109/0x140
[ T93] ? __cfi_wakeme_after_rcu+0x10/0x10
[ T93] kvm_irqfd+0x362/0x5e0
[ T93] kvm_vm_ioctl+0x706/0x780
[ T93] ? fd_install+0x2c/0xf0
[ T93] __se_sys_ioctl+0x7a/0xd0
[ T93] do_syscall_64+0x61/0xf10
[ T93] ? arch_exit_to_user_mode_prepare+0x9/0xb0
[ T93] entry_SYSCALL_64_after_hwframe+0x76/0x7e
[ T93] RIP: 0033:0x79048f9bdd67
[ T93] RSP: 002b:00007ffc3aa82028 EFLAGS: 00000206.
And another task waiting on the mutex:
[ C0] task:kworker/11:2 state:R running task stack:0 pid:25180 tgid:25180 ppid:2 task_flags:0x4208060 flags:0x00080000
[ C0] Workqueue: kvm-irqfd-cleanup irqfd_shutdown
[ C0] Call Trace:
[ C0] <TASK>
[ C0] __schedule+0x87a/0xd60
[ C0] schedule+0x5e/0xe0
[ C0] schedule_preempt_disabled+0x10/0x20
[ C0] __mutex_lock+0x413/0xe40
[ C0] irqfd_resampler_shutdown+0x23/0x150
[ C0] irqfd_shutdown+0x66/0xc0
[ C0] process_scheduled_works+0x219/0x450
[ C0] worker_thread+0x30b/0x450
[ C0] ? __cfi_worker_thread+0x10/0x10
[ C0] kthread+0x230/0x270
[ C0] ? __cfi_kthread+0x10/0x10
[ C0] ret_from_fork+0xf2/0x150
[ C0] ? __cfi_kthread+0x10/0x10
[ C0] ret_from_fork_asm+0x1a/0x30
[ C0] </TASK>
The work queue was full as well I think:
[ C0] pwq 46: cpus=11 node=0 flags=0x0 nice=0 active=1024 refcnt=2062
There were other tasks waiting for SRCU GP completion in the resampler
shutdown path. Also, there were other traces showing lockups (mostly in
mm), but I think thats a secondary effect of this lockup and might not
be relevant. I can provide the full logs if needed.
Please have a look and see if this path needs to be handled to fully fix
this issue.
Thanks,
Vineeth
next prev parent reply other threads:[~2026-03-20 12:56 UTC|newest]
Thread overview: 21+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-03-16 7:20 Sonam Sanju
2026-03-17 16:27 ` Sonam Sanju
2026-03-20 12:56 ` Vineeth Pillai (Google) [this message]
2026-03-23 5:33 ` Sonam Sanju
2026-03-23 6:42 ` [PATCH v2] KVM: irqfd: fix deadlock by moving synchronize_srcu out of resampler_lock Sonam Sanju
2026-03-31 18:17 ` Sean Christopherson
2026-03-31 20:51 ` Paul E. McKenney
2026-04-01 9:47 ` Sonam Sanju
2026-04-06 23:09 ` Paul E. McKenney
2026-04-01 9:34 ` Kunwu Chan
2026-04-01 14:24 ` Sonam Sanju
2026-04-06 14:20 ` Kunwu Chan
2026-04-17 1:18 ` Vineeth Pillai
2026-04-19 3:03 ` Vineeth Remanan Pillai
2026-04-21 16:54 ` [PATCH v2] KVM: eventfd: Use WQ_UNBOUND workqueue for irqfd cleanup - New logs confirm preemption race Sonam Sanju
2026-04-21 18:22 ` Tejun Heo
2026-04-23 9:01 ` [PATCH v2] KVM: irqfd: fix deadlock by moving synchronize_srcu out of resampler_lock Sonam Sanju
2026-04-23 13:25 ` Vineeth Remanan Pillai
2026-04-21 5:12 ` Sonam Sanju
-- strict thread matches above, loose matches on Subject: below --
2026-03-16 7:08 [PATCH] KVM: irqfd: fix shutdown deadlock by moving SRCU sync outside resampler_lock Sonam Sanju
2026-03-16 9:42 ` Sonam Sanju
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20260320125633.2290675-1-vineeth@bitbyteword.org \
--to=vineeth@bitbyteword.org \
--cc=dmaluka@chromium.org \
--cc=kvm@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=pbonzini@redhat.com \
--cc=seanjc@google.com \
--cc=sonam.sanju@intel.com \
--cc=sonam.sanju@intel.corp-partner.google.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®