* [PATCH v2 1/6] s390/smp: Export arch_smp_send_reschedule() for KVM
2026-11-19 23:56 [PATCH v2 0/6] KVM: Send reschedule IPIs for non-waiting requests Sean Christopherson
@ 2026-11-19 23:56 ` Sean Christopherson
2026-11-19 23:56 ` [PATCH v2 2/6] KVM: Add struct and helpers to handle kicking multiple (v)CPUs Sean Christopherson
` (4 subsequent siblings)
5 siblings, 0 replies; 7+ messages in thread
From: Sean Christopherson @ 2026-11-19 23:56 UTC (permalink / raw)
To: Paolo Bonzini
Cc: Sean Christopherson, kvm, linux-kernel, Satish Kharat,
Sashiko Bot, Oliver Upton, Marc Zyngier
Export arch_smp_send_reschedule() so that KVM can use smp_send_reschedule()
in common "kick vCPU" code without running afoul of build issues on s390.
KVM currently uses smp_call_function_many() when kicking multiple vCPUs,
but several KVM architectures (at least arm64 and x86) kick vCPUs with IRQs
disabled. Using SMP function calls with IRQs disabled is unsafe as it can
lead to two CPUs waiting for each other to respond to the function call,
i.e. KVM needs to use smp_send_reschedule() to avoid potential deadlock on
other architectures.
Alternatively, KVM could go out of its way to avoid smp_send_reschedule()
on s390, as KVM does when kicking a single vCPU, but given that all modules
can reach smp_send_reschedule() via kick_process(), forcing KVM to dance
around s390 adds no practical value and would add non-trivial complexity to
KVM.
Note, thanks to s390 now building kvm.ko and kvm-arm64.ko, using
EXPORT_SYMBOL_FOR_KVM doesn't Just Work. Exporting the symbol just for KVM
would also complicate backporting the upcoming fixes to older kernels. For
now, keep it simple.
Cc: stable@vger.kernel.org
Signed-off-by: Sean Christopherson <seanjc@google.com>
---
arch/s390/kernel/smp.c | 1 +
1 file changed, 1 insertion(+)
diff --git a/arch/s390/kernel/smp.c b/arch/s390/kernel/smp.c
index 32499cad86f0..39318aa7ab7f 100644
--- a/arch/s390/kernel/smp.c
+++ b/arch/s390/kernel/smp.c
@@ -531,6 +531,7 @@ void arch_smp_send_reschedule(int cpu)
{
pcpu_ec_call(per_cpu_ptr(&pcpu_devices, cpu), ec_schedule);
}
+EXPORT_SYMBOL_GPL(arch_smp_send_reschedule);
#ifdef CONFIG_IRQ_WORK
void arch_irq_work_raise(void)
--
2.56.1.708.g77570a6a1d-goog
^ permalink raw reply [flat|nested] 7+ messages in thread* [PATCH v2 2/6] KVM: Add struct and helpers to handle kicking multiple (v)CPUs
2026-11-19 23:56 [PATCH v2 0/6] KVM: Send reschedule IPIs for non-waiting requests Sean Christopherson
2026-11-19 23:56 ` [PATCH v2 1/6] s390/smp: Export arch_smp_send_reschedule() for KVM Sean Christopherson
@ 2026-11-19 23:56 ` Sean Christopherson
2026-11-19 23:56 ` [PATCH v2 3/6] KVM: Add dedicated flag to track if multi-vCPU request needs at least one kick Sean Christopherson
` (3 subsequent siblings)
5 siblings, 0 replies; 7+ messages in thread
From: Sean Christopherson @ 2026-11-19 23:56 UTC (permalink / raw)
To: Paolo Bonzini
Cc: Sean Christopherson, kvm, linux-kernel, Satish Kharat,
Sashiko Bot, Oliver Upton, Marc Zyngier
Add a "kvm_cpu_kick" structure along with init()+finish() helpers to dedup
the core logic for kicking multiple vCPUs, and to make it easier to handle
non-waiting kicks via smp_send_reschedule() instead of
smp_call_function_many().
No functional change intended.
Signed-off-by: Sean Christopherson <seanjc@google.com>
---
virt/kvm/kvm_main.c | 77 +++++++++++++++++++++++++--------------------
1 file changed, 43 insertions(+), 34 deletions(-)
diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c
index cf4cb586f140..c2231a130c4c 100644
--- a/virt/kvm/kvm_main.c
+++ b/virt/kvm/kvm_main.c
@@ -222,15 +222,43 @@ static inline bool kvm_kick_many_cpus(struct cpumask *cpus, bool wait)
return true;
}
-static void kvm_make_vcpu_request(struct kvm_vcpu *vcpu, unsigned int req,
- struct cpumask *tmp, int current_cpu)
+struct kvm_cpu_kick {
+ int me;
+ unsigned int req;
+ struct cpumask *cpus;
+};
+
+static __always_inline struct kvm_cpu_kick kvm_init_cpu_kick(unsigned int req)
+{
+ struct kvm_cpu_kick kick = { .req = req };
+
+ kick.me = get_cpu();
+
+ kick.cpus = this_cpu_cpumask_var_ptr(cpu_kick_mask);
+ cpumask_clear(kick.cpus);
+
+ return kick;
+}
+
+static bool kvm_finish_cpu_kick(struct kvm_cpu_kick *kick)
+{
+ bool kicked;
+
+ kicked = kvm_kick_many_cpus(kick->cpus, !!(kick->req & KVM_REQUEST_WAIT));
+
+ put_cpu();
+ return kicked;
+}
+
+static void kvm_make_vcpu_request(struct kvm_vcpu *vcpu,
+ struct kvm_cpu_kick *kick)
{
int cpu;
- if (likely(!(req & KVM_REQUEST_NO_ACTION)))
- __kvm_make_request(req, vcpu);
+ if (likely(!(kick->req & KVM_REQUEST_NO_ACTION)))
+ __kvm_make_request(kick->req, vcpu);
- if (!(req & KVM_REQUEST_NO_WAKEUP) && kvm_vcpu_wake_up(vcpu))
+ if (!(kick->req & KVM_REQUEST_NO_WAKEUP) && kvm_vcpu_wake_up(vcpu))
return;
/*
@@ -243,59 +271,40 @@ static void kvm_make_vcpu_request(struct kvm_vcpu *vcpu, unsigned int req,
* for vCPUs that were reading SPTEs _before_ any changes were
* finalized. See kvm_vcpu_kick() for more details on handling requests.
*/
- if (kvm_request_needs_ipi(vcpu, req)) {
+ if (kvm_request_needs_ipi(vcpu, kick->req)) {
cpu = READ_ONCE(vcpu->cpu);
- if (cpu != -1 && cpu != current_cpu)
- __cpumask_set_cpu(cpu, tmp);
+ if (cpu != -1 && cpu != kick->me)
+ __cpumask_set_cpu(cpu, kick->cpus);
}
}
bool kvm_make_vcpus_request_mask(struct kvm *kvm, unsigned int req,
unsigned long *vcpu_bitmap)
{
+ struct kvm_cpu_kick kick = kvm_init_cpu_kick(req);
struct kvm_vcpu *vcpu;
- struct cpumask *cpus;
- int i, me;
- bool called;
-
- me = get_cpu();
-
- cpus = this_cpu_cpumask_var_ptr(cpu_kick_mask);
- cpumask_clear(cpus);
+ int i;
for_each_set_bit(i, vcpu_bitmap, KVM_MAX_VCPUS) {
vcpu = kvm_get_vcpu(kvm, i);
if (!vcpu)
continue;
- kvm_make_vcpu_request(vcpu, req, cpus, me);
+ kvm_make_vcpu_request(vcpu, &kick);
}
- called = kvm_kick_many_cpus(cpus, !!(req & KVM_REQUEST_WAIT));
- put_cpu();
-
- return called;
+ return kvm_finish_cpu_kick(&kick);
}
bool kvm_make_all_cpus_request(struct kvm *kvm, unsigned int req)
{
+ struct kvm_cpu_kick kick = kvm_init_cpu_kick(req);
struct kvm_vcpu *vcpu;
- struct cpumask *cpus;
unsigned long i;
- bool called;
- int me;
-
- me = get_cpu();
-
- cpus = this_cpu_cpumask_var_ptr(cpu_kick_mask);
- cpumask_clear(cpus);
kvm_for_each_vcpu(i, vcpu, kvm)
- kvm_make_vcpu_request(vcpu, req, cpus, me);
+ kvm_make_vcpu_request(vcpu, &kick);
- called = kvm_kick_many_cpus(cpus, !!(req & KVM_REQUEST_WAIT));
- put_cpu();
-
- return called;
+ return kvm_finish_cpu_kick(&kick);
}
EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_make_all_cpus_request);
--
2.56.1.708.g77570a6a1d-goog
^ permalink raw reply [flat|nested] 7+ messages in thread* [PATCH v2 3/6] KVM: Add dedicated flag to track if multi-vCPU request needs at least one kick
2026-11-19 23:56 [PATCH v2 0/6] KVM: Send reschedule IPIs for non-waiting requests Sean Christopherson
2026-11-19 23:56 ` [PATCH v2 1/6] s390/smp: Export arch_smp_send_reschedule() for KVM Sean Christopherson
2026-11-19 23:56 ` [PATCH v2 2/6] KVM: Add struct and helpers to handle kicking multiple (v)CPUs Sean Christopherson
@ 2026-11-19 23:56 ` Sean Christopherson
2026-11-19 23:56 ` [PATCH v2 4/6] KVM: Don't kick offline CPUs when making a multi-vCPU request Sean Christopherson
` (2 subsequent siblings)
5 siblings, 0 replies; 7+ messages in thread
From: Sean Christopherson @ 2026-11-19 23:56 UTC (permalink / raw)
To: Paolo Bonzini
Cc: Sean Christopherson, kvm, linux-kernel, Satish Kharat,
Sashiko Bot, Oliver Upton, Marc Zyngier
Add a dedicated flag to track if a multi-vCPU request needs to kick at
least one CPU, so that non-waiting requests can avoid using the per-CPU
cpu_kick_mask by immediately sending a reschedule IPI instead of populating
and processing a cpumask set.
No functional change intended.
Signed-off-by: Sean Christopherson <seanjc@google.com>
---
virt/kvm/kvm_main.c | 22 ++++++++--------------
1 file changed, 8 insertions(+), 14 deletions(-)
diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c
index c2231a130c4c..d2a858d4b8e9 100644
--- a/virt/kvm/kvm_main.c
+++ b/virt/kvm/kvm_main.c
@@ -213,19 +213,11 @@ static void ack_kick(void *_completed)
{
}
-static inline bool kvm_kick_many_cpus(struct cpumask *cpus, bool wait)
-{
- if (cpumask_empty(cpus))
- return false;
-
- smp_call_function_many(cpus, ack_kick, NULL, wait);
- return true;
-}
-
struct kvm_cpu_kick {
int me;
unsigned int req;
struct cpumask *cpus;
+ bool need_kick;
};
static __always_inline struct kvm_cpu_kick kvm_init_cpu_kick(unsigned int req)
@@ -242,12 +234,12 @@ static __always_inline struct kvm_cpu_kick kvm_init_cpu_kick(unsigned int req)
static bool kvm_finish_cpu_kick(struct kvm_cpu_kick *kick)
{
- bool kicked;
-
- kicked = kvm_kick_many_cpus(kick->cpus, !!(kick->req & KVM_REQUEST_WAIT));
+ if (kick->need_kick)
+ smp_call_function_many(kick->cpus, ack_kick, NULL,
+ !!(kick->req & KVM_REQUEST_WAIT));
put_cpu();
- return kicked;
+ return kick->need_kick;
}
static void kvm_make_vcpu_request(struct kvm_vcpu *vcpu,
@@ -273,8 +265,10 @@ static void kvm_make_vcpu_request(struct kvm_vcpu *vcpu,
*/
if (kvm_request_needs_ipi(vcpu, kick->req)) {
cpu = READ_ONCE(vcpu->cpu);
- if (cpu != -1 && cpu != kick->me)
+ if (cpu != -1 && cpu != kick->me) {
+ kick->need_kick = true;
__cpumask_set_cpu(cpu, kick->cpus);
+ }
}
}
--
2.56.1.708.g77570a6a1d-goog
^ permalink raw reply [flat|nested] 7+ messages in thread* [PATCH v2 4/6] KVM: Don't kick offline CPUs when making a multi-vCPU request
2026-11-19 23:56 [PATCH v2 0/6] KVM: Send reschedule IPIs for non-waiting requests Sean Christopherson
` (2 preceding siblings ...)
2026-11-19 23:56 ` [PATCH v2 3/6] KVM: Add dedicated flag to track if multi-vCPU request needs at least one kick Sean Christopherson
@ 2026-11-19 23:56 ` Sean Christopherson
2026-11-19 23:56 ` [PATCH v2 5/6] KVM: Send reschedule IPIs instead of SMP function calls for non-waiting requests Sean Christopherson
2026-11-19 23:56 ` [PATCH v2 6/6] KVM: Extract and dedup the kick code to check if a CPU needs an IPI Sean Christopherson
5 siblings, 0 replies; 7+ messages in thread
From: Sean Christopherson @ 2026-11-19 23:56 UTC (permalink / raw)
To: Paolo Bonzini
Cc: Sean Christopherson, kvm, linux-kernel, Satish Kharat,
Sashiko Bot, Oliver Upton, Marc Zyngier
Don't send IPIs to offline CPUs when sending a request to multiple vCPUs,
as it's obviously unnecessary and sending a reschedule IPI (spoiler alert)
to an offline CPU will trigger a WARN in x86's native_smp_send_reschedule().
Note, as explained in commit 85b640450ddc ("KVM: Clean up benign vcpu->cpu
data races when kicking vCPUs"), checking for an online CPU is inherently
racy, but losing the race is functionally a-ok. I.e. the goal is purely to
avoid tripping x86's WARN.
Suggested-by: Satish Kharat <satishkh@cisco.com>
Signed-off-by: Sean Christopherson <seanjc@google.com>
---
virt/kvm/kvm_main.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c
index d2a858d4b8e9..07cc0dd10b71 100644
--- a/virt/kvm/kvm_main.c
+++ b/virt/kvm/kvm_main.c
@@ -265,7 +265,7 @@ static void kvm_make_vcpu_request(struct kvm_vcpu *vcpu,
*/
if (kvm_request_needs_ipi(vcpu, kick->req)) {
cpu = READ_ONCE(vcpu->cpu);
- if (cpu != -1 && cpu != kick->me) {
+ if (cpu != -1 && cpu != kick->me && cpu_online(cpu)) {
kick->need_kick = true;
__cpumask_set_cpu(cpu, kick->cpus);
}
--
2.56.1.708.g77570a6a1d-goog
^ permalink raw reply [flat|nested] 7+ messages in thread* [PATCH v2 5/6] KVM: Send reschedule IPIs instead of SMP function calls for non-waiting requests
2026-11-19 23:56 [PATCH v2 0/6] KVM: Send reschedule IPIs for non-waiting requests Sean Christopherson
` (3 preceding siblings ...)
2026-11-19 23:56 ` [PATCH v2 4/6] KVM: Don't kick offline CPUs when making a multi-vCPU request Sean Christopherson
@ 2026-11-19 23:56 ` Sean Christopherson
2026-11-19 23:56 ` [PATCH v2 6/6] KVM: Extract and dedup the kick code to check if a CPU needs an IPI Sean Christopherson
5 siblings, 0 replies; 7+ messages in thread
From: Sean Christopherson @ 2026-11-19 23:56 UTC (permalink / raw)
To: Paolo Bonzini
Cc: Sean Christopherson, kvm, linux-kernel, Satish Kharat,
Sashiko Bot, Oliver Upton, Marc Zyngier
When kicking multiple vCPUs for a non-waiting request, send a reschedule
IPI to each vCPU instead of batching IPIs via a single SMP function call so
that KVM can make non-waiting requests with IRQs disabled. I.e. handle
multi-vCPU kicks the same way KVM handles single-vCPU kicks, for the exact
same reason (and more).
Avoiding SMP function calls for non-waiting requests fixes a potential
deadlock if KVM VMX triggers KVM_REQ_UNBLOCK to all vCPUs when starting
device assignmennt:
WARNING: kernel/smp.c:943 at smp_call_function_many_cond
kvm_make_all_cpus_request [kvm]
kvm_arch_irq_bypass_add_producer [kvm]
__connect [irqbypass]
irq_bypass_register_producer [irqbypass]
vfio_msi_set_vector_signal [vfio_pci_core]
Deliberately avoid using the per-CPU cpu_kick_mask for non-waiting to fix
an even more urgent bug on arm64, where KVM could fail to wait for a vCPU,
e.g. for a remote TLB flush, if an IRQ triggers a multi-vCPU kick in the
middle of another waiting kick.
kvm_flush_remote_tlbs()
kvm_make_all_cpus_request(kvm, KVM_REQ_TLB_FLUSH)
cpumask_clear(cpus)
kvm_make_vcpu_request(vcpu0, ...)
kvm_vcpu_exiting_guest_mode(vcpu0)
cmpxchg(&vcpu0->mode, IN_GUEST_MODE, EXITING_GUEST_MODE)
__cpumask_set_cpu(vcpu0->cpu, cpus)
<hardirq>
irqfd_wakeup()
...
kvm_make_all_cpus_request(kvm, KVM_REQ_IRQ_PENDING)
cpumask_clear(cpus) <-- wipes vcpu0->cpu from cpu_kick_mask
kvm_make_vcpu_request(vcpu0, ...)
kvm_request_needs_ipi(vcpu0, ...) <-- false (EXITING_GUEST_MODE)
kvm_kick_many_cpus(cpus, true) <-- misses vcpu0->cpu
Send reschedule IPIs immediately to avoid having to re-walk the set of CPUs,
and obviously because using the per-CPU mask is unsafe. As a bonus, it's
generally advantageous to kick vCPUs as soon as possible, e.g. so that the
target vCPU(s) can immediately process pending IRQs.
Fixes: 77e1b8332d1d ("KVM: x86: Decouple device assignment from IRQ bypass")
Cc: stable@vger.kernel.org
Reported-by: Satish Kharat <satishkh@cisco.com>
Closes: https://lore.kernel.org/all/20261116-b4-kvm-irq-bypass-irqs-20261116-v1-1-6cc721a062fd@cisco.com
Reported-by: Sashiko Bot <sashiko-bot@kernel.org>
Closes: https://lore.kernel.org/all/sashiko-outbox-219682@kernel.org
Cc: Oliver Upton <oupton@kernel.org>
Cc: Marc Zyngier <maz@kernel.org>
Signed-off-by: Sean Christopherson <seanjc@google.com>
---
virt/kvm/kvm_main.c | 21 ++++++++++++++++-----
1 file changed, 16 insertions(+), 5 deletions(-)
diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c
index 07cc0dd10b71..48fef91b8530 100644
--- a/virt/kvm/kvm_main.c
+++ b/virt/kvm/kvm_main.c
@@ -226,15 +226,17 @@ static __always_inline struct kvm_cpu_kick kvm_init_cpu_kick(unsigned int req)
kick.me = get_cpu();
- kick.cpus = this_cpu_cpumask_var_ptr(cpu_kick_mask);
- cpumask_clear(kick.cpus);
+ if (req & KVM_REQUEST_WAIT) {
+ kick.cpus = this_cpu_cpumask_var_ptr(cpu_kick_mask);
+ cpumask_clear(kick.cpus);
+ }
return kick;
}
static bool kvm_finish_cpu_kick(struct kvm_cpu_kick *kick)
{
- if (kick->need_kick)
+ if (kick->need_kick && kick->cpus)
smp_call_function_many(kick->cpus, ack_kick, NULL,
!!(kick->req & KVM_REQUEST_WAIT));
@@ -261,13 +263,22 @@ static void kvm_make_vcpu_request(struct kvm_vcpu *vcpu,
* satisfied if the vCPU migrates. Entering READING_SHADOW_PAGE_TABLES
* after this point is also OK, as the requirement is only that KVM wait
* for vCPUs that were reading SPTEs _before_ any changes were
- * finalized. See kvm_vcpu_kick() for more details on handling requests.
+ * finalized.
+ *
+ * See kvm_vcpu_kick() for more details on handling requests, including
+ * the use of a reschedule IPI for non-waiting requests. If waiting is
+ * required (kick->cpus is non-NULL), batch the SMP function call until
+ * after all to-be-kicked CPUs are gathered, to avoid waiting multiple
+ * times.
*/
if (kvm_request_needs_ipi(vcpu, kick->req)) {
cpu = READ_ONCE(vcpu->cpu);
if (cpu != -1 && cpu != kick->me && cpu_online(cpu)) {
kick->need_kick = true;
- __cpumask_set_cpu(cpu, kick->cpus);
+ if (kick->cpus)
+ __cpumask_set_cpu(cpu, kick->cpus);
+ else
+ smp_send_reschedule(cpu);
}
}
}
--
2.56.1.708.g77570a6a1d-goog
^ permalink raw reply [flat|nested] 7+ messages in thread* [PATCH v2 6/6] KVM: Extract and dedup the kick code to check if a CPU needs an IPI
2026-11-19 23:56 [PATCH v2 0/6] KVM: Send reschedule IPIs for non-waiting requests Sean Christopherson
` (4 preceding siblings ...)
2026-11-19 23:56 ` [PATCH v2 5/6] KVM: Send reschedule IPIs instead of SMP function calls for non-waiting requests Sean Christopherson
@ 2026-11-19 23:56 ` Sean Christopherson
5 siblings, 0 replies; 7+ messages in thread
From: Sean Christopherson @ 2026-11-19 23:56 UTC (permalink / raw)
To: Paolo Bonzini
Cc: Sean Christopherson, kvm, linux-kernel, Satish Kharat,
Sashiko Bot, Oliver Upton, Marc Zyngier
Extract the kick logic's code to determine if a target CPU actually needs
an IPI into a helper function. In addition to deduplicating the existing
checks in common KVM, the helper can "likely" (read: will) be eventually
used in arch specific code as well.
No functional change intended: KVM always uses '-1' to indicate an invalid
vcpu->cpu, so "!= -1", "(unsigned int)cpu < nr_cpu_ids", and ">= 0" are all
functionally equivalent.
Signed-off-by: Sean Christopherson <seanjc@google.com>
---
virt/kvm/kvm_main.c | 9 +++++++--
1 file changed, 7 insertions(+), 2 deletions(-)
diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c
index 48fef91b8530..d8027ea0c447 100644
--- a/virt/kvm/kvm_main.c
+++ b/virt/kvm/kvm_main.c
@@ -209,6 +209,11 @@ static bool kvm_request_needs_ipi(struct kvm_vcpu *vcpu, unsigned req)
return mode == IN_GUEST_MODE;
}
+static bool kvm_cpu_needs_ipi(int cpu, int me)
+{
+ return cpu >= 0 && cpu != me && cpu_online(cpu);
+}
+
static void ack_kick(void *_completed)
{
}
@@ -273,7 +278,7 @@ static void kvm_make_vcpu_request(struct kvm_vcpu *vcpu,
*/
if (kvm_request_needs_ipi(vcpu, kick->req)) {
cpu = READ_ONCE(vcpu->cpu);
- if (cpu != -1 && cpu != kick->me && cpu_online(cpu)) {
+ if (kvm_cpu_needs_ipi(cpu, kick->me)) {
kick->need_kick = true;
if (kick->cpus)
__cpumask_set_cpu(cpu, kick->cpus);
@@ -3951,7 +3956,7 @@ void __kvm_vcpu_kick(struct kvm_vcpu *vcpu, bool wait)
*/
if (kvm_arch_vcpu_should_kick(vcpu)) {
cpu = READ_ONCE(vcpu->cpu);
- if (cpu != me && (unsigned int)cpu < nr_cpu_ids && cpu_online(cpu)) {
+ if (kvm_cpu_needs_ipi(cpu, me)) {
/*
* Use a reschedule IPI to kick the vCPU if the caller
* doesn't need to wait for a response, as KVM allows
--
2.56.1.708.g77570a6a1d-goog
^ permalink raw reply [flat|nested] 7+ messages in thread