Re: [RFC PATCH 00/46] Orphaned Virtual Machines

["Graf (AWS), Alexander" <graf@amazon.de>]

On 20.09.26 21:36, Pasha Tatashin wrote:
> This RFC series is a PoC intended to showcase e2e working OrphanVMs
> uninterrupted continuing execution on preserved physical CPUs across a
> host kernel live update, prepared for the LPC'26 presentation at the
> KVM Microconf [1].
>
> The original design proposal discussion can be found at [2].
>
> This series is applied on top of v7.3-rc1 + the "[PATCH v4 00/11]
> liveupdate: kvm: Guest_memfd preservation" series [3]. The whole branch
> can be accessed at [4].
>
> It was tested on Intel Xeon Granite Rapids-AP, AMD EPYC Turin, and ARM
> Neoverse V2, as well as in emulation on QEMU and Intel Simics. However,
> it is still very early WIP, and is not anywhere near being production
> ready.
>
> Overview & Submission Strategy
> ==============================
> While this RFC bundles the e2e stack, it is NOT intended to be merged as
> a single series, the goal of this series is to have a discussion about
> the technical layers, and the challenges that need to be solved in order
> to have full Caretaker support within the kernel.
>
> Some of the arch/* changes duplicate small helpers (such as GICv3 list
> register accessors or low-level VMCS/VMCB/sysreg helpers) that could be
> shared with existing KVM/arch code if those routines and their callees
> are moved to static inlines or annotated with __cpu_preserved_text (and
> built without ftrace/KASAN/per-CPU references, as enforced by objtool
> and modpost). To avoid sprinkling __cpu_preserved_text annotations
> across core arch/* files in this PoC, those helpers are kept local to
> caretaker.c for now and can be consolidated when individual platform
> support is upstreamed.
>
> The series is structured into at least 8 workstreams that will be
> discussed and worked on separately:
>
> 1. Workstream 1: Preparation Patches (Patches 01-06)
>     Architecture-neutral and arch-specific preparatory cleanups (in
>     future these could be bundled with the workstreams that require these
>     changes):
>       01. KVM: luo: Delegate VM creation type to kvm_arch_vm_luo_preserve
>       02. KVM: arm64: Split demux_c15_{get,set}_val from userspace
>           accessors
>       03. KVM: arm64: Split kvm_sys_reg_{get,set}_user from kernel
>           accessors
>       04. x86/mm/ident_map: Add force_pte to support 4K PTE identity
>           mappings
>       05. arm64: mm: Add trans_pgd_map_range() support
>       06. x86/smp: Skip offline CPUs for REBOOT_VECTOR in
>           native_stop_other_cpus()
>
> 2. Workstream 2: In-Kernel In-RAM vCPU State Preservation via LUO
>     (Patches 07-10)
>     Allows performing suspend/resume-type preservation across kexec
>     without carrying the vCPU internal state in the VMM in userspace,
>     keeping all vCPU state in-kernel:
>       07. KVM: luo: Support vCPU file preservation across live updates
>       08. KVM: x86: Add x86 vCPU LUO preservation ABI and register
>           helpers
>       09. KVM: x86: Implement architectural vCPU state preservation via
>           LUO
>       10. KVM: arm64: Implement architectural vCPU state preservation via
>           LUO
>
> 3. Workstream 3: Physical CPU Preservation Core & x86 Support
>     (Patches 11-24)
>     Introduces the physical CPU preservation subsystem (cpu_preserve),
>     linker sections (.text.cpu_preserved, .data.cpu_preserved) with
>     build-time section isolation checks in modpost and objtool, isolated
>     transition page table management (struct cpu_preserved_as), and sysfs
>     interfaces. Allows booting the machine with some CPUs removed from
>     the OS scheduler, and provides infrastructure for executing orphaned
>     jobs on such CPUs:
>       11. liveupdate: Define CPU preservation linker sections
>       12. liveupdate: Add liveupdate_session_name() helper
>       13. cpu_preserve: Add physical CPU preservation ABI and core API
>           headers
>       14. cpu_preserve: Add core physical CPU preservation state and park
>           loop
>       15. cpu_preserve: Add physical CPU preservation lifecycle and build
>           rules
>       16. liveupdate: cpu_preserve: Add sysfs interface
>       17. liveupdate: cpu_preserve: Add isolated address space management
>           API
>       18. liveupdate: cpu_preserve: Add LUO file handler for preserved
>           physical CPUs
>       19. x86: liveupdate: Add low-level physical CPU preservation
>           assembly
>       20. x86: liveupdate: Add physical CPU preservation context and page
>           table support
>       21. selftests: liveupdate: Add physical CPU preservation unit tests
>       22. selftests: liveupdate: Add physical CPU preservation live
>           update tests
>       23. Documentation: liveupdate: Add physical CPU preservation
>           documentation
>       24. MAINTAINERS: Add entry for KVM Caretaker
>
> 4. Workstream 4: ARM64 Support for Physical CPU Preservation (Patch 25)
>     Enables physical CPU preservation on ARM64 using isolated transition
>     page tables (trans_pgd_map_range()), cache maintenance to PoC, and
>     GICv3 redistributor SGI wake helpers:
>       25. arm64: liveupdate: Add support for physical CPU preservation
>
> 5. Workstream 5: On-Core Scheduling Framework (oncore) (Patches 26-27)
>     (Note: This layer sits between cpu_preserve and the KVM Caretaker and
>     Kernel Caretaker (Future Work for Orphaned Processes))
>     Provides a runqueue and time-slice scheduling framework
>     (kernel/liveupdate/oncore.c) that multiplexes preserved workloads
>     onto preserved physical CPUs during the kexec blackout window:
>       26. oncore: Add on-core KHO ABI and public framework headers
>       27. oncore: Implement on-core session lifecycle and scheduling loop
>
> 6. Workstream 6: KVM Caretaker Core & Intel VMX Support
>     (Patches 28-39)
>     Introduces the KVM Caretaker execution engine (virt/kvm/caretaker.c),
>     vCPU detach/attach/cancel lifecycle, optional cross-kexec execution
>     telemetry and debugfs reporting, x86 TDP MMU KHO page table
>     preservation, x86 common Caretaker runtime (LAPIC timer injection,
>     VM-exit dispatch, instruction decoding), and the Intel VMX Caretaker
>     backend:
>       28. KVM: caretaker: Add Caretaker control block and architecture
>           ops headers
>       29. KVM: caretaker: Implement Caretaker session memory mapping
>           helpers
>       30. KVM: caretaker: Integrate Caretaker vCPU detach, attach, and
>           cancel with KVM
>       31. KVM: caretaker: Add generic KHO ABI telemetry and debugfs
>           reporting
>       32. KVM: x86: Add TDP MMU KHO preservation helpers
>       33. KVM: x86: Add Caretaker x86 KHO ABI and runtime context headers
>       34. KVM: x86: Implement Caretaker LAPIC timer and interrupt
>           injection
>       35. KVM: x86: Implement Caretaker VM-exit dispatch and instruction
>           decoders
>       36. KVM: x86: Implement Caretaker run loop and LUO detach/attach
>           lifecycle
>       37. KVM: VMX: Add Caretaker VMX assembly guest entry/exit routine
>           and helpers
>       38. KVM: VMX: Implement Caretaker VMX VMCS lifecycle and exit
>           dispatch
>       39. KVM: VMX: Integrate Caretaker VMX detach serialization and KVM
>           registration
>
> 7. Workstream 7: KVM Caretaker for AMD SVM (Patches 40-41)
>     Implements the AMD SVM Caretaker backend (caretaker_vmenter.S, VMCB
>     lifecycle, exit decoding, and next-RIP sync):
>       40. KVM: SVM: Add Caretaker SVM assembly guest entry/exit routine
>       41. KVM: SVM: Implement Caretaker SVM VMCB lifecycle and exit
>           dispatch
>
> 8. Workstream 8: KVM Caretaker for ARM64 (Patches 42-46)
>     Implements the ARM64 VHE Caretaker backend (caretaker_vmenter.S,
>     EL2 exception vectors, GICv3 CPU interface & arch timer emulation,
>     system register trap handling, and Stage-2 fault handling):
>       42. KVM: arm64: Add Caretaker arm64 KHO ABI and runtime context
>           headers
>       43. KVM: arm64: Add Caretaker EL2 exception vectors and guest
>           entry/exit assembly
>       44. KVM: arm64: Implement Caretaker GICv3 CPU interface and arch
>           timer emulation
>       45. KVM: arm64: Implement Caretaker system register trap and
>           exception handlers
>       46. KVM: arm64: Implement Caretaker vCPU run loop and LUO
>           detach/attach lifecycle
>
> Architectural Design Layers
> ===========================
> The OrphanVM architecture is layered so that each subsystem has zero
> upward symbol dependencies:
>
>    +--------------------------------------------------------------------+
>    | Layer 4: KVM Caretaker Engine (virt/kvm/caretaker.c + arch/kvm/*)  |
>    |   - VMENTER/VMRUN/ERET guest execution loop                        |
>    |   - Handling of timer preemption, HLT/WFI, MSR/sysreg traps        |
>    |   - Yield on unhandled I/O or Stage-2 MMIO faults                  |
>    +--------------------------------------------------------------------+


I had a superficial look at the code only, but it looked like we're not 
reusing any of the existing KVM code for entry or exit handling. I was 
hoping we could refactor some bits of KVM code into functionality that 
is safe stand-alone and functionality that does require the larger 
kernel ecosystem, then annotate the stand-alone ones as caretaker (btw, 
abbreviating as kvmct would shorten your function names significantly 
:)) and then consume them from both contexts.

Do you have any timing measurements for the approach as is?

Also, your patch set is cut off at 39/46. I was very curious to see how 
you deal with vGIC and context switching in the caretaker environment :).


Alex