Re: [PATCH v10 00/43] arm64: Support for Arm CCA in KVM

[Steven Price <steven.price@arm.com>]

On 04/09/2025 01:46, Gavin Shan wrote:
> 
> On 8/21/25 12:55 AM, Steven Price wrote:
>> This series adds support for running protected VMs using KVM under the
>> Arm Confidential Compute Architecture (CCA).
>>
>> The related guest support was merged for v6.14-rc1 so you no longer need
>> that separately.
>>
>> There are a few changes since v9, many thanks for the review
>> comments. The highlights are below, and individual patches have a
>> changelog.
>>
>>   * Fix a potential issue where the host was walking the stage 2 page
>> tables on
>>     realm destruction. If the RMM didn't zero when undelegated (which
>> it isn't
>>     required to) then the kernel would attempt to work the junk values
>> and crash.
>>
>>   * Avoid RCU stall warnings by correctly settign may_block in
>>     kvm_free_stage2_pgd().
>>
>>   * Rebased onto v6.17-rc1.
>>
>> Things to note:
>>
>>   * The magic numbers for capabilities and ioctls have been updated. So
>>     you'll need to update your VMM. See below for the updated kvmtool
>> branch.
>>
>>   * This series doesn't attempt to integrate with the guest-memfd
>> changes that
>>     are being discussed (see below).
>>
>>   * Vishal raised an important question about what to do in the case of
>>     undelegate failures (also see below).
>>
> 
> [...]
> 
> I tried to boot a guest using the following combinations, nothing
> obvious went to
> wrong except several long existing issues (described below). So feel
> free to add:
> 
> Tested-by: Gavin Shan <gshan@redhat.com>

Thanks for testing!

Regards,
Steve

> 
> Combination
> ===========
> host.tf-a        https://git.trustedfirmware.org/TF-A/trusted-firmware-
> a.git      (v2.13-rc0)
> host.tf-rmm      https://git.codelinaro.org/linaro/dcap/
> rmm                       (cca/v8)
> host.edk2        git@github.com:tianocore/
> edk2.git                                (edk2-stable202411)
> host.kernel      git@github.com:gwshan/
> linux.git                                  (cca/host-v10) (this series)
> host.qemu        https://git.qemu.org/git/
> qemu.git                                (stable-9.2)
> host.buildroot   https://github.com/buildroot/
> buildroot                           (master)
> guest.qemu       https://git.codelinaro.org/linaro/dcap/
> qemu.git                  (cca/latest) (with linux-headers sync'ed)
> guest.kvmtool    https://gitlab.arm.com/linux-arm/kvmtool-
> cca                     (cca/latest)
> guest.edk2       https://git.codelinaro.org/linaro/dcap/
> edk2                      (cca/latest)
> guest.kernel     git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/
> linux.git (v6.17.rc3)
> guest.buildroot  https://github.com/buildroot/
> buildroot                           (master)
> 
> Script to start the host
> ========================
> gshan@nvidia-grace-hopper-01:~/sandbox/qemu/host$ cat start.sh
> #!/bin/sh
> HOST_PATH=/home/gshan/sandbox/qemu/host
> GUEST_PATH=/home/gshan/sandbox/qemu/guest
> IF_UP_SCRIPT=/etc/qemu-ifup-gshan
> IF_DOWN_SCRIPT=/etc/qemu-ifdown-gshan
> 
> sudo ${HOST_PATH}/qemu/build/qemu-system-aarch64                        \
> -M virt,virtualization=on,secure=on,gic-version=3,acpi=off              \
> -cpu max,x-rme=on -m 3G -smp 8                                          \
> -serial mon:stdio -monitor none -nographic -nodefaults                  \
> -bios ${HOST_PATH}/tf-a/flash.bin                                       \
> -kernel ${HOST_PATH}/linux/arch/arm64/boot/Image                        \
> -initrd ${HOST_PATH}/buildroot/output/images/rootfs.cpio.xz             \
> -device pcie-root-port,bus=pcie.0,chassis=1,id=pcie.1                   \
> -device pcie-root-port,bus=pcie.0,chassis=2,id=pcie.2                   \
> -device pcie-root-port,bus=pcie.0,chassis=3,id=pcie.3                   \
> -device pcie-root-port,bus=pcie.0,chassis=4,id=pcie.4                   \
> -device virtio-9p-device,fsdev=shr0,mount_tag=shr0                      \
> -fsdev local,security_model=none,path=${GUEST_PATH},id=shr0             \
> -netdev tap,id=tap1,script=${IF_UP_SCRIPT},downscript=${IF_DOWN_SCRIPT} \
> -device virtio-net-pci,bus=pcie.2,netdev=tap1,mac=b8:3f:d2:1d:3e:f1
> 
> Script to start the guest
> =========================
> gshan@nvidia-grace-hopper-01:~/sandbox/qemu/guest$ cat start_full.sh
> #!/bin/sh
> key="VGhlIHJlYWxtIGd1ZXN0IHBlcnNvbmFsaXphdGlvbiBrZXkgaW4gZm9ybWF0IG9mIGJhc2U2NCAgICAgICAgIA=="
> IF_UP_SCRIPT=/etc/qemu-ifup
> IF_DOWN_SCRIPT=/etc/qemu-ifdown
> 
> qemu-system-aarch64 -enable-kvm \
> -object rme-guest,id=rme0,measurement-algorithm=sha512,personalization-
> value=${key} \
> -M virt,gic-version=3,confidential-guest-
> support=rme0                               \
> -cpu host -smp 4 -m 2G -boot
> c                                                      \
> -serial mon:stdio -monitor none -nographic -
> nodefaults                              \
> -bios /mnt/edk2/Build/ArmVirtQemu-AARCH64/RELEASE_GCC5/FV/
> QEMU_EFI.fd               \
> -device pcie-root-
> port,bus=pcie.0,chassis=1,id=pcie.1                               \
> -device pcie-root-
> port,bus=pcie.0,chassis=2,id=pcie.2                               \
> -drive file=/mnt/
> rhel10.qcow2,if=none,id=drive0                                     \
> -device virtio-blk-pci,id=virtblk0,bus=pcie.1,drive=drive0,num-
> queues=4             \
> -netdev
> tap,id=tap0,script=${IF_UP_SCRIPT},downscript=${IF_DOWN_SCRIPT}             \
> -device virtio-net-pci,bus=pcie.2,netdev=tap0,mac=b8:3f:d2:1d:3e:f9
> 
> Issues
> ======
> 1. virtio-iommu isn't supported by QEMU. The guest kernel becomes stuck
> at IOMMU
> probing time where the endpoint's capabilities is queried by sending
> request over
> virtio device's vring and the response is expected to be fed by QEMU.
> The request
> can't be seen by QEMU due to the wrong IOMMU address translation used in
> QEMU as
> virtio-iommu provides a different IOMMU address translation operations
> to override
> the platform one, leading the DMA address (in the shared space) can't be
> properly
> recognized. The information has been shared to Jean.
> 
> 2. 'reboot' command doesn't work in the guest. QEMU complains some
> registers aren't
> accessible from QEMU. I didn't sorted out a workaround for this.
> 
> 3. HMP command 'dump-guest-memory' causes QEMU to exit abnormally. The
> cause is the
> realm is reconfigured when the VM is resumed after the guest memory is
> dumped. The
> reconfiguration is rejected by the host, leading QEMU's abnormal exit.
> The fix would
> be to avoid the reconfiguration on the realm. The issue was originally
> reported by
> Fujitsu and all the information has been shared to Fujitsu.
> 
> 4. In QEMU, the CPU property 'kvm-no-adjvtime' can't be set to off.
> Otherwise, QEMU
> tries to access the timer registers, which have been hidden by the host.
> So we need
> to take the parameter (for QEMU) to by pass it: "-cpu host,kvm-no-
> adjvtime=on".
> 
> 5. I didn't try virtio-mem and memory balloon, which isn't expected to
> work, especially
> when the guest memory is hot added or hot removed.
> 
> Thanks,
> Gavin
> 
>