[PATCH 0/2] genirq, KVM: arm64: Avoid the irq descriptor lock on vCPU load

[Amir Noohi <amirmnoohi@gmail.com>]

KVM/arm64 vCPUs slow each other down even when every vCPU has a physical
CPU to itself. On a 32 CPU Ampere Altra (GICv3, VHE) running mainline,
with an in-kernel GICv3 and each vCPU thread pinned to its own otherwise
idle CPU, a vCPU which exits to userspace in a loop manages 320k exits/s
on its own, 120k when 16 vCPUs do the same and 59k with 31. The aggregate
rate stops growing at 8 vCPUs. No two vCPUs share a CPU and nothing is
over-subscribed.

The cause is the interrupt descriptor lock of the host timer interrupts.
Every vCPU load sets their active state with irq_set_irqchip_state()
(twice on VHE: virtual and physical timer). The timer interrupts are per
CPU interrupts, so all CPUs share one descriptor and its lock, while the
state being written is per CPU (a redistributor register on GICv3, a
system register on GICv5). The lock serializes nothing but CPUs updating
their own state. At 8 vCPUs, 99.9% of all lock contention events during
the test have __irq_get_desc_lock() or irq_set_irqchip_state() on the
stack.

Patch 1 adds irq_set_percpu_irqchip_state(), which sets the state of the
calling CPU's instance of a per CPU devid interrupt without taking the
descriptor lock. Patch 2 makes the KVM timer use it. With the series the
same test gets 326k exits/s per vCPU at 31 vCPUs (5.6x), the per vCPU
rate no longer depends on the number of vCPUs, the aggregate rate goes
from 1.8M to 10.1M exits/s, and at 8 vCPUs no lock contention event has
either function on the stack.

The series is based on tip/irq/urgent, as it builds on
__irq_set_irqchip_state() from Igor Velkov's patch 717862eac6ef
("genirq/kexec: Do not take the bus lock of an interrupt chip on the
crash path") [1]. It also applies cleanly to v7.3, current mainline and
kvmarm/next on top of that patch.

I'm aware this arrives during the merge window. There is no urgency on
my side; I'm posting it now so it can be looked at after -rc1, and I'm
happy to rebase and resend then.

Results
-------

Reproducer: a /dev/kvm program (no guest OS) which makes N vCPUs exit to
userspace in a loop, each KVM_RUN being one vCPU load/put, each vCPU
thread pinned to its own otherwise idle CPU and running from its own
guest page, with an in-kernel GICv3 or a userspace irqchip.
The test program is at
https://gist.github.com/amirmnoohi/91560854a63cd3e5fc7a359eb5e17302

32 CPU Ampere Altra (Neoverse-N1, GICv3, VHE), mainline 864c07de0c7d
(v7.4 merge window) with and without the series, same kernel config,
exits/s per vCPU, mean of three 5 second runs:

                in-kernel GICv3           userspace irqchip
  vCPUs       before      after          before      after
      1       319694     324584          319670     345088
      2       298808     326063          289274     343884
      4       294895     328420          284269     343528
      8       278579     328736          112423     343984
     16       119943     328887           56120     282905
     24        77267     327478           36948     253529
     31        58722     326056           28114     250586

Aggregate with the in-kernel GICv3: 2.23M exits/s at 8 vCPUs and 1.82M
at 31 before the series, 2.63M and 10.1M with it. With a userspace
irqchip, which also clears the active state after every exit, the rate
with the series still drops to 251k per vCPU at 31 vCPUs (runs varied by
about 8% there); I have not investigated that.

Lock contention events within 5 seconds at 8 vCPUs with
__irq_get_desc_lock() or irq_set_irqchip_state() on the stack: 122800 of
122847 (in-kernel GICv3) and 7865871 of 7865907 (userspace irqchip)
before, none of 49 and 24 with the series.

If all vCPUs instead run from one shared guest page, the aggregate rate
with the series stops at about 3.6M exits/s from 16 vCPUs on this
machine (also with the timer's GIC writes removed altogether, as an
experiment), and the extra time goes to guest context (front-end
stalls) rather than to the host. Measured that way, the series gains
2.0x at 31 vCPUs instead of 5.6x.

A second machine shows the same: on a 384 CPU HiSilicon Kunpeng 950
(GICv3, VHE, 6.6 based distribution kernel, userspace irqchip), 8 vCPUs
on 8 idle CPUs ran 2.4x slower each than a single vCPU, and 69% of the
time spent in irq_set_irqchip_state() was spent waiting for the lock.

Under QEMU TCG (virt, -cpu max), 8 vCPUs, the series removes every
acquisition of the descriptor lock on this path (LOCK_STAT, 10 seconds):

  host             irqchip     irq_desc_lock_class before      after
                               acquisitions  contentions
  GICv3, VHE       in-kernel        393878        36682        none
  GICv3, nVHE      in-kernel        199654        32080        none
  GICv3, nVHE      userspace        429928       122152        none
  GICv2, VHE       userspace        321430        59542        none

Throughput under TCG does not change, so the real hardware numbers
above are the ones that matter.

Testing
-------

- Built for arm64 (defconfig, debug config), x86_64 (defconfig,
  allnoconfig) and i386 (defconfig) without new warnings, no new W=1 or
  sparse warnings (arch_timer.c has two pre-existing sparse warnings),
  checkpatch clean, bisectable.
- On the Ampere Altra, with and without the series: KVM selftests
  arch_timer (8 and 31 vCPUs migrating every 2ms), arch_timer_edge_cases,
  vgic_irq and vgic_init pass.
- Under QEMU with PROVE_LOCKING, DEBUG_PREEMPT and DEBUG_ATOMIC_SLEEP:
  arch_timer, vgic_irq, vgic_init, vgic_lpi_stress, kvm_create_max_vcpus
  and steal_time pass on GICv3 VHE and GICv3 nVHE hosts (on a GICv2 host
  the vGICv3 based tests skip, the others pass), without warnings or
  lockdep splats.
- Not tested: GICv5 (no hardware), hosts with no_hw_deactivation
  (Apple), protected mode.
- Not changed: on hosts without a vGIC (e.g. Raspberry Pi 3), KVM
  toggles the timer interrupt with enable/disable_percpu_irq(), which
  take the same descriptor lock. Those hosts have few CPUs and their
  interrupt controllers do not implement irq_set_irqchip_state(), so
  the series leaves that path alone.

[1] https://lore.kernel.org/r/20261015-kexec-irq-bus-lock-v2-1-2af15f0757b6@iav.lv

Amir Noohi (2):
  genirq/manage: Provide irq_set_percpu_irqchip_state()
  KVM: arm64: timer: Avoid the irq descriptor lock on vCPU load

 arch/arm64/kvm/arch_timer.c |  2 +-
 include/linux/interrupt.h   |  2 ++
 kernel/irq/manage.c         | 36 ++++++++++++++++++++++++++++++++++++
 3 files changed, 39 insertions(+), 1 deletion(-)


base-commit: 717862eac6eff0eea8ac30a0c0de0bed84922aab
-- 
2.53.0