Re: [RESEND][PATCH 0/2] Batch register access for live migration optimization

[Yize Wang <wangyize7@huawei.com>]

在 2026/9/18 20:08, Marc Zyngier 写道:
> On Fri, 18 Sep 2026 09:18:13 +0100,
> Yize Wang <wangyize7@huawei.com> wrote:
>> This series adds batch register access support to KVM/arm64 to reduce
>> syscall overhead during VM live migration.
>>
>> Currently, QEMU issues one ioctl per register when saving/restoring VGIC
>> state. On large VM configurations this means tens of thousands of syscalls,
>> where lock acquisition and context switch overhead dominates migration
>> downtime. Thus, we provide a batch register method to allow userspace
>> read/write multiple distributor and redistributor registers in a single call.
>> In this way, we can significantly reduce syscalls and migration downtime.
>>
>> Test the VM migration time under pressure conditions.
>> The VM specifications for migration are as follows:
>> - VM use 4-K page;
>> - the number of VCPU is 160;
>> - the total memory is 320Gigabit;
>> - use 'Redis SET-benchmark' to pressurize VM;
>>
>> Performance results (3-run average, ms):
>>      | Metric              | Without patch | With patch | Improvement |
>>      |---------------------|---------------|------------|-------------|
>>      | Migration downtime  |        536    |     321    |     40%     |
>>      | Source (total)      |        344    |     230    |     33%     |
>>      |   - VGIC put        |        158    |      40    |     75%     |
>>      |   - VGIC get        |        120    |      19    |     84%     |
>>      | Destination (total) |        192    |      91    |     53%     |
>>      |   - VGIC put        |        132    |      27    |     80%     |
>>
>> Yize Wang (2):
>>    KVM: arm64: Add batch group constant and data structure to UAPI header
>>    KVM: arm64: Add VGIC v3 batch register access implementation
> Questions:
>
> - Why only the MMIO registers?
>
> - Why not the sysregs?
>
> - Why only the GIC?
>
> - Why not all of the state?
>
> - Where is the corresponding userspace code?
>
> More importantly, since this is about batching system calls:
>
> - Why can't this be done with io_uring instead?
>
> 	M.


Hi, Marc! Thank you for the review.


These patches focus on optimizing GICv3 register access during live 
migration. We found that there are a large number of locks (kvm->lock, 
vcpus, config_lock) in the GIC, these lock operations wil cost large 
time waste. The batches of sysreg for vcpu optimization will come in 
follow as a separate series. And let me address these questions one by one.


1. Why only the MMIO registers?

In vgic_v3_batch_access(), we use 'entries' structure to implement batch 
read/write of register status. The structure is 'struct 
kvm_dev_arm_vgic_batch_entry', where the group information can be freely 
specified by userspace. Thus, vgic_v3_batch_access() supports all VGIC 
device attr groups, not only MMIO registers.


2. Why not the sysregs?

The newly added vgic_v3_batch_access() just forwards the groups received 
fromQEMU in batches to 'vgic_v3_attr_regs_access()'. And this function 
already has 'KVM_DEV_ARM_VGIC_GRP_CPU_SYSREGS' to handle sysregs. Our 
patch does not modify the existing sysreg handling logic.


3. Why only the GIC?

During live migration, the GIC is the device with the largest number of 
registers, and we found that there are a large number of locks in it. If 
each register is locked and unlocked individually, it would cause large 
time consumption. Thus, we want to optimize the GIC register time 
consumption during live migrate. The experimental results also show that 
batch processing significantly reduces VGIC handling time during migration.

Similarly, vCPU register save/restore also costs significantly time. The 
patch of vCPU sysreg batch access will be submitted separately in the 
future.


4. Why not all of the state?

As different devices have different lock hierarchies and access 
paths(e.g., GIC goes through the device fd, CPU regs go through the vcpu 
fd), it's difficult for us to realize in a single batch handler. 
Additionally, introducing too many changes at once would make review 
harder. These patches focus on GIC-related optimization, and a separate 
series for vCPU batch processing will follow.


5. Where is the corresponding userspace code?

The QEMU-side implementation has been posted to qemu-devel. The link is 
below:

https://lore.kernel.org/qemu-devel/20260918092120.370805-1-wangyize7@huawei.com/T/#t


6. Why can't this be done with io_uring instead?

The core idea of io_uring is to reduce the number of context switches 
between user space and kernel space by utilizing two ring buffers. 
However, each SQE is still processed independently in kernel space. For 
the VGIC registers, each SQE still require lock -> read/write -> unlock, 
so the per-register lock overhead remains unchanged. We hope to 
read/write a set of register states with a single lock operation to 
reduce the time cost. Therefore, io_uring does not meet our needs.