[RFC PATCH 00/46] Orphaned Virtual Machines

[RFC PATCH 00/46] Orphaned Virtual Machines

[Pasha Tatashin]

This RFC series is a PoC intended to showcase e2e working OrphanVMs
uninterrupted continuing execution on preserved physical CPUs across a
host kernel live update, prepared for the LPC'26 presentation at the
KVM Microconf [1].

The original design proposal discussion can be found at [2].

This series is applied on top of v7.3-rc1 + the "[PATCH v4 00/11]
liveupdate: kvm: Guest_memfd preservation" series [3]. The whole branch
can be accessed at [4].

It was tested on Intel Xeon Granite Rapids-AP, AMD EPYC Turin, and ARM
Neoverse V2, as well as in emulation on QEMU and Intel Simics. However,
it is still very early WIP, and is not anywhere near being production
ready.

Overview & Submission Strategy
==============================
While this RFC bundles the e2e stack, it is NOT intended to be merged as
a single series, the goal of this series is to have a discussion about
the technical layers, and the challenges that need to be solved in order
to have full Caretaker support within the kernel.

Some of the arch/* changes duplicate small helpers (such as GICv3 list
register accessors or low-level VMCS/VMCB/sysreg helpers) that could be
shared with existing KVM/arch code if those routines and their callees
are moved to static inlines or annotated with __cpu_preserved_text (and
built without ftrace/KASAN/per-CPU references, as enforced by objtool
and modpost). To avoid sprinkling __cpu_preserved_text annotations
across core arch/* files in this PoC, those helpers are kept local to
caretaker.c for now and can be consolidated when individual platform
support is upstreamed.

The series is structured into at least 8 workstreams that will be
discussed and worked on separately:

1. Workstream 1: Preparation Patches (Patches 01-06)
   Architecture-neutral and arch-specific preparatory cleanups (in
   future these could be bundled with the workstreams that require these
   changes):
     01. KVM: luo: Delegate VM creation type to kvm_arch_vm_luo_preserve
     02. KVM: arm64: Split demux_c15_{get,set}_val from userspace
         accessors
     03. KVM: arm64: Split kvm_sys_reg_{get,set}_user from kernel
         accessors
     04. x86/mm/ident_map: Add force_pte to support 4K PTE identity
         mappings
     05. arm64: mm: Add trans_pgd_map_range() support
     06. x86/smp: Skip offline CPUs for REBOOT_VECTOR in
         native_stop_other_cpus()

2. Workstream 2: In-Kernel In-RAM vCPU State Preservation via LUO
   (Patches 07-10)
   Allows performing suspend/resume-type preservation across kexec
   without carrying the vCPU internal state in the VMM in userspace,
   keeping all vCPU state in-kernel:
     07. KVM: luo: Support vCPU file preservation across live updates
     08. KVM: x86: Add x86 vCPU LUO preservation ABI and register
         helpers
     09. KVM: x86: Implement architectural vCPU state preservation via
         LUO
     10. KVM: arm64: Implement architectural vCPU state preservation via
         LUO

3. Workstream 3: Physical CPU Preservation Core & x86 Support
   (Patches 11-24)
   Introduces the physical CPU preservation subsystem (cpu_preserve),
   linker sections (.text.cpu_preserved, .data.cpu_preserved) with
   build-time section isolation checks in modpost and objtool, isolated
   transition page table management (struct cpu_preserved_as), and sysfs
   interfaces. Allows booting the machine with some CPUs removed from
   the OS scheduler, and provides infrastructure for executing orphaned
   jobs on such CPUs:
     11. liveupdate: Define CPU preservation linker sections
     12. liveupdate: Add liveupdate_session_name() helper
     13. cpu_preserve: Add physical CPU preservation ABI and core API
         headers
     14. cpu_preserve: Add core physical CPU preservation state and park
         loop
     15. cpu_preserve: Add physical CPU preservation lifecycle and build
         rules
     16. liveupdate: cpu_preserve: Add sysfs interface
     17. liveupdate: cpu_preserve: Add isolated address space management
         API
     18. liveupdate: cpu_preserve: Add LUO file handler for preserved
         physical CPUs
     19. x86: liveupdate: Add low-level physical CPU preservation
         assembly
     20. x86: liveupdate: Add physical CPU preservation context and page
         table support
     21. selftests: liveupdate: Add physical CPU preservation unit tests
     22. selftests: liveupdate: Add physical CPU preservation live
         update tests
     23. Documentation: liveupdate: Add physical CPU preservation
         documentation
     24. MAINTAINERS: Add entry for KVM Caretaker

4. Workstream 4: ARM64 Support for Physical CPU Preservation (Patch 25)
   Enables physical CPU preservation on ARM64 using isolated transition
   page tables (trans_pgd_map_range()), cache maintenance to PoC, and
   GICv3 redistributor SGI wake helpers:
     25. arm64: liveupdate: Add support for physical CPU preservation

5. Workstream 5: On-Core Scheduling Framework (oncore) (Patches 26-27)
   (Note: This layer sits between cpu_preserve and the KVM Caretaker and
   Kernel Caretaker (Future Work for Orphaned Processes))
   Provides a runqueue and time-slice scheduling framework
   (kernel/liveupdate/oncore.c) that multiplexes preserved workloads
   onto preserved physical CPUs during the kexec blackout window:
     26. oncore: Add on-core KHO ABI and public framework headers
     27. oncore: Implement on-core session lifecycle and scheduling loop

6. Workstream 6: KVM Caretaker Core & Intel VMX Support
   (Patches 28-39)
   Introduces the KVM Caretaker execution engine (virt/kvm/caretaker.c),
   vCPU detach/attach/cancel lifecycle, optional cross-kexec execution
   telemetry and debugfs reporting, x86 TDP MMU KHO page table
   preservation, x86 common Caretaker runtime (LAPIC timer injection,
   VM-exit dispatch, instruction decoding), and the Intel VMX Caretaker
   backend:
     28. KVM: caretaker: Add Caretaker control block and architecture
         ops headers
     29. KVM: caretaker: Implement Caretaker session memory mapping
         helpers
     30. KVM: caretaker: Integrate Caretaker vCPU detach, attach, and
         cancel with KVM
     31. KVM: caretaker: Add generic KHO ABI telemetry and debugfs
         reporting
     32. KVM: x86: Add TDP MMU KHO preservation helpers
     33. KVM: x86: Add Caretaker x86 KHO ABI and runtime context headers
     34. KVM: x86: Implement Caretaker LAPIC timer and interrupt
         injection
     35. KVM: x86: Implement Caretaker VM-exit dispatch and instruction
         decoders
     36. KVM: x86: Implement Caretaker run loop and LUO detach/attach
         lifecycle
     37. KVM: VMX: Add Caretaker VMX assembly guest entry/exit routine
         and helpers
     38. KVM: VMX: Implement Caretaker VMX VMCS lifecycle and exit
         dispatch
     39. KVM: VMX: Integrate Caretaker VMX detach serialization and KVM
         registration

7. Workstream 7: KVM Caretaker for AMD SVM (Patches 40-41)
   Implements the AMD SVM Caretaker backend (caretaker_vmenter.S, VMCB
   lifecycle, exit decoding, and next-RIP sync):
     40. KVM: SVM: Add Caretaker SVM assembly guest entry/exit routine
     41. KVM: SVM: Implement Caretaker SVM VMCB lifecycle and exit
         dispatch

8. Workstream 8: KVM Caretaker for ARM64 (Patches 42-46)
   Implements the ARM64 VHE Caretaker backend (caretaker_vmenter.S,
   EL2 exception vectors, GICv3 CPU interface & arch timer emulation,
   system register trap handling, and Stage-2 fault handling):
     42. KVM: arm64: Add Caretaker arm64 KHO ABI and runtime context
         headers
     43. KVM: arm64: Add Caretaker EL2 exception vectors and guest
         entry/exit assembly
     44. KVM: arm64: Implement Caretaker GICv3 CPU interface and arch
         timer emulation
     45. KVM: arm64: Implement Caretaker system register trap and
         exception handlers
     46. KVM: arm64: Implement Caretaker vCPU run loop and LUO
         detach/attach lifecycle

Architectural Design Layers
===========================
The OrphanVM architecture is layered so that each subsystem has zero
upward symbol dependencies:

  +--------------------------------------------------------------------+
  | Layer 4: KVM Caretaker Engine (virt/kvm/caretaker.c + arch/kvm/*)  |
  |   - VMENTER/VMRUN/ERET guest execution loop                        |
  |   - Handling of timer preemption, HLT/WFI, MSR/sysreg traps        |
  |   - Yield on unhandled I/O or Stage-2 MMIO faults                  |
  +--------------------------------------------------------------------+
                                    |
                                    v (registers oncore jobs)
  +--------------------------------------------------------------------+
  | Layer 3: On-Core Scheduler Framework (kernel/liveupdate/oncore.c)  |
  |   - Runqueue and time-slice scheduler on preserved cores           |
  |   - Architecture-neutral counter frequency deadline calculation    |
  |   - Session memory mapping into isolated address spaces            |
  +--------------------------------------------------------------------+
                                    |
                                    v (attaches cpu_preserve workload)
  +--------------------------------------------------------------------+
  | Layer 2: Physical CPU Preservation (kernel/liveupdate/cpu_preserve)|
  |   - Offlines CPUs into preserved park loop (.text.cpu_preserved)   |
  |   - Switches CR3/TTBR to isolated transition page tables           |
  |   - Manages cpu_present state across kexec and hotplug reclaim     |
  +--------------------------------------------------------------------+
                                    |
                                    v (persists state via KHO / LUO)
  +--------------------------------------------------------------------+
  | Layer 1: KHO & LUO                                                 |
  |   - Preserves guest_memfd RAM, TDP/Stage-2 MMU, and vCPU state     |
  |   - Cross-kexec ABI (include/linux/kho/abi/*)                      |
  +--------------------------------------------------------------------+

1. Layer 1: KHO & LUO In-RAM Preservation (kvm_luo)
   Preserves VM and vCPU file descriptors across kexec using KHO memory
   preservation. Guest memory (guest_memfd) and secondary MMU page
   tables remain in place in physical RAM. Architectural vCPU state is
   serialized into struct kvm_vcpu_arch_luo_state buffers.

2. Layer 2: Physical CPU Preservation (cpu_preserve)
   Before kexec, selected physical CPUs are taken offline and
   transitioned into a park loop compiled into .text.cpu_preserved and
   copied into KHO-preserved pages outside KHO scratch memory. Each
   preserved CPU switches to an isolated address space (struct
   cpu_preserved_as) that maps ONLY the preserved text/data, per-CPU
   stack, and mapped workload buffers. Build-time section checks in
   modpost and objtool enforce that .text.cpu_preserved and
   .data.cpu_preserved do not reference unmapped host kernel symbols.
   Because preserved CPUs are marked !cpu_present(cpu) while preserved,
   the host kernel's SMP stop routines and incoming kernel's SMP boot
   bringup skip them without hotplug workarounds.

3. Layer 3: On-Core Scheduling Framework (oncore)
   Provides a runqueue and session scheduler (kernel/liveupdate/oncore.c)
   that runs inside the cpu_preserve park loop. Multiple preserved
   workloads (such as vCPUs) can be scheduled across one or more
   preserved physical CPUs using hardware counter deadlines (TSC on x86,
   CNTPCT_EL0 on ARM64).

4. Layer 4: KVM Caretaker Execution Engine (caretaker)
   Attaches preserved KVM vCPUs to oncore jobs. During the kexec
   blackout window, the Caretaker enters the guest directly from
   preserved text with baseline speculative execution mitigations
   (VERW CPU buffer clearing on x86 VM-entry, RSB stuffing on x86
   VM-exit, and Spectre-BHB loop clearing on ARM64 EL2 guest exit):
   - On Intel VMX (vmx/caretaker.c), executes VMLAUNCH/VMRESUME against
     a preserved VMCS page and handles VMX preemption timer / physical
     LAPIC timer, HLT, PAUSE, CPUID, RDTSC, COM1 UART, and basic MSR
     exits. Software emulation of x2APIC ICR IPIs is intentionally
     omitted (trapped ICR writes park the vCPU until the incoming kernel
     reclaims it); instead, the Caretaker KHO-preserves the virtual APIC
     page and the VM's Posted-Interrupt Descriptor pointer table
     (pid_table), allowing hardware with APICv and Intel IPIv to deliver
     inter-vCPU IPIs without VM-exits.
   - On AMD SVM (svm/caretaker.c), executes VMRUN with a preserved VMCB,
     preserved MSR/IO permission bitmaps (msrpm/iopm), and host save
     area (MSR_VM_HSAVE_PA), sharing the common x86 Caretaker exit and
     timer handlers.
   - On ARM64 VHE (arm64/kvm/caretaker.c), switches EL2 exception
     vectors (VBAR_EL2) to Caretaker vectors, emulates the GICv3 CPU
     interface (including inter-vCPU SGI delivery via ICC_SGI1R_EL1
     trapping and GICv3 List Register injection) and arch timer, and
     handles WFI/WFE and system register traps.
   - Note that the set of supported VM-exits can be adjusted as needed;
     initial upstreaming can start by not supporting any VM-exits at
     all, waiting in the Caretaker loop if a vCPU exits until the
     incoming host kernel finishes booting and reclaims the vCPU.
   - Cross-Kexec Re-Attachment: When the incoming kernel boots and
     retrieves the preserved VM, it signals detach via the Caretaker
     control block. The Caretaker serializes live hardware state on the
     preserved core into struct kvm_vcpu_arch_luo_state before yielding,
     exits the park loop, and allows the incoming kernel to reclaim the
     physical CPU and resume KVM execution. Optional execution telemetry
     (runs, exits, stalls, exit reasons, and last RIP) is preserved via
     the KHO ABI and exposed in dmesg and per-vCPU debugfs
     (caretaker_telemetry), both live during detached execution and upon
     re-attachment.

Links
=====
[1] https://lpc.events/event/20/contributions/2553/
[2] https://lore.kernel.org/all/afEwWZksU0Fw61oT@plex/
[3] https://lore.kernel.org/all/20260728121138.1103610-1-tarunsahu@google.com/#r
[4] https://github.com/soleen/linux/tree/ovm/rfc/v1

Pasha Tatashin (46):
  KVM: luo: Delegate VM creation type to kvm_arch_vm_luo_preserve
  KVM: arm64: Split demux_c15_{get,set}_val from userspace accessors
  KVM: arm64: Split kvm_sys_reg_{get,set}_user from kernel accessors
  x86/mm/ident_map: Add force_pte to support 4K PTE identity mappings
  arm64: mm: Add trans_pgd_map_range() support
  x86/smp: Skip offline CPUs for REBOOT_VECTOR in
    native_stop_other_cpus()
  KVM: luo: Support vCPU file preservation across live updates
  KVM: x86: Add x86 vCPU LUO preservation ABI and register helpers
  KVM: x86: Implement architectural vCPU state preservation via LUO
  KVM: arm64: Implement architectural vCPU state preservation via LUO
  liveupdate: Define CPU preservation linker sections
  liveupdate: Add liveupdate_session_name() helper
  cpu_preserve: Add physical CPU preservation ABI and core API headers
  cpu_preserve: Add core physical CPU preservation state and park loop
  cpu_preserve: Add physical CPU preservation lifecycle and build rules
  liveupdate: cpu_preserve: Add sysfs interface
  liveupdate: cpu_preserve: Add isolated address space management API
  liveupdate: cpu_preserve: Add LUO file handler for preserved physical
    CPUs
  x86: liveupdate: Add low-level physical CPU preservation assembly
  x86: liveupdate: Add physical CPU preservation context and page table
    support
  selftests: liveupdate: Add physical CPU preservation unit tests
  selftests: liveupdate: Add physical CPU preservation live update tests
  Documentation: liveupdate: Add physical CPU preservation documentation
  MAINTAINERS: Add entry for KVM Caretaker
  arm64: liveupdate: Add support for physical CPU preservation
  oncore: Add on-core KHO ABI and public framework headers
  oncore: Implement on-core session lifecycle and scheduling loop
  KVM: caretaker: Add Caretaker control block and architecture ops
    headers
  KVM: caretaker: Implement Caretaker session memory mapping helpers
  KVM: caretaker: Integrate Caretaker vCPU detach, attach, and cancel
    with KVM
  KVM: caretaker: Add generic KHO ABI telemetry and debugfs reporting
  KVM: x86: Add TDP MMU KHO preservation helpers
  KVM: x86: Add Caretaker x86 KHO ABI and runtime context headers
  KVM: x86: Implement Caretaker LAPIC timer and interrupt injection
  KVM: x86: Implement Caretaker VM-exit dispatch and instruction
    decoders
  KVM: x86: Implement Caretaker run loop and LUO detach/attach lifecycle
  KVM: VMX: Add Caretaker VMX assembly guest entry/exit routine and
    helpers
  KVM: VMX: Implement Caretaker VMX VMCS lifecycle and exit dispatch
  KVM: VMX: Integrate Caretaker VMX detach serialization and KVM
    registration
  KVM: SVM: Add Caretaker SVM assembly guest entry/exit routine
  KVM: SVM: Implement Caretaker SVM VMCB lifecycle and exit dispatch
  KVM: arm64: Add Caretaker arm64 KHO ABI and runtime context headers
  KVM: arm64: Add Caretaker EL2 exception vectors and guest entry/exit
    assembly
  KVM: arm64: Implement Caretaker GICv3 CPU interface and arch timer
    emulation
  KVM: arm64: Implement Caretaker system register trap and exception
    handlers
  KVM: arm64: Implement Caretaker vCPU run loop and LUO detach/attach
    lifecycle

 .../ABI/testing/sysfs-devices-system-cpu      |   20 +
 Documentation/liveupdate/cpu_preservation.rst |   48 +
 Documentation/liveupdate/index.rst            |   11 +
 Documentation/liveupdate/vmm.rst              |   15 +
 Documentation/virt/kvm/api.rst                |   16 +
 MAINTAINERS                                   |   11 +
 arch/arm64/Kconfig                            |    1 +
 arch/arm64/include/asm/caretaker.h            |   56 +
 arch/arm64/include/asm/cpu_preserve.h         |   44 +
 arch/arm64/include/asm/oncore.h               |   24 +
 arch/arm64/include/asm/trans_pgd.h            |    4 +
 arch/arm64/kernel/Makefile                    |    7 +
 arch/arm64/kernel/cpu_preserve.c              |  674 ++++++
 arch/arm64/kernel/preserve_cpu.S              |   44 +
 arch/arm64/kernel/smp.c                       |    8 +-
 arch/arm64/kernel/vmlinux.lds.S               |    1 +
 arch/arm64/kvm/Kconfig                        |    2 +
 arch/arm64/kvm/Makefile                       |   14 +
 arch/arm64/kvm/caretaker.c                    | 1162 +++++++++++
 arch/arm64/kvm/caretaker.h                    |  130 ++
 arch/arm64/kvm/caretaker_vmenter.S            |  248 +++
 arch/arm64/kvm/kvm_luo.c                      |  304 +++
 arch/arm64/kvm/sys_regs.c                     |  247 ++-
 arch/arm64/kvm/sys_regs.h                     |    4 +
 arch/arm64/mm/trans_pgd.c                     |   70 +
 arch/x86/Kconfig                              |    1 +
 arch/x86/include/asm/cpu_preserve.h           |   34 +
 arch/x86/include/asm/init.h                   |    3 +-
 arch/x86/include/asm/oncore.h                 |   25 +
 arch/x86/kernel/Makefile                      |    7 +
 arch/x86/kernel/cpu_preserve.c                |  432 ++++
 arch/x86/kernel/preserve_cpu.S                |   62 +
 arch/x86/kernel/smp.c                         |    2 +-
 arch/x86/kernel/vmlinux.lds.S                 |    2 +
 arch/x86/kvm/Kconfig                          |    2 +
 arch/x86/kvm/Makefile                         |   32 +
 arch/x86/kvm/caretaker.c                      | 1129 ++++++++++
 arch/x86/kvm/caretaker.h                      |  238 +++
 arch/x86/kvm/caretaker_asm.h                  |  117 ++
 arch/x86/kvm/cpuid.c                          |    4 +-
 arch/x86/kvm/cpuid.h                          |    1 +
 arch/x86/kvm/kvm-asm-offsets.c                |   32 +
 arch/x86/kvm/kvm_luo.c                        |  368 ++++
 arch/x86/kvm/mmu.h                            |    7 +
 arch/x86/kvm/mmu/kho.c                        |  193 ++
 arch/x86/kvm/msrs.c                           |   17 +
 arch/x86/kvm/msrs.h                           |    2 +
 arch/x86/kvm/regs.c                           |    8 +-
 arch/x86/kvm/regs.h                           |    4 +
 arch/x86/kvm/svm/caretaker.c                  |  467 +++++
 arch/x86/kvm/svm/caretaker.h                  |   63 +
 arch/x86/kvm/svm/caretaker_vmenter.S          |   91 +
 arch/x86/kvm/svm/svm.c                        |   11 +-
 arch/x86/kvm/vmx/caretaker.c                  |  658 ++++++
 arch/x86/kvm/vmx/caretaker.h                  |   78 +
 arch/x86/kvm/vmx/caretaker_vmenter.S          |  110 +
 arch/x86/kvm/vmx/vmx.c                        |    6 +-
 arch/x86/kvm/vmx/vmx.h                        |    1 +
 arch/x86/kvm/x86.c                            |    8 +-
 arch/x86/kvm/x86.h                            |    4 +
 arch/x86/mm/ident_map.c                       |   72 +-
 include/asm-generic/vmlinux.lds.h             |   29 +
 include/linux/cpu_preserve.h                  |  464 +++++
 include/linux/kho/abi/cpu.h                   |  151 ++
 include/linux/kho/abi/kvm.h                   |  137 +-
 include/linux/kho/abi/kvm_arm64.h             |   93 +
 include/linux/kho/abi/kvm_x86.h               |  106 +
 include/linux/kvm_caretaker.h                 |  439 ++++
 include/linux/kvm_host.h                      |   63 +
 include/linux/liveupdate.h                    |    6 +
 include/linux/oncore.h                        |  129 ++
 include/uapi/linux/kvm.h                      |    2 +
 kernel/cpu.c                                  |    2 +
 kernel/liveupdate/Kconfig                     |   45 +
 kernel/liveupdate/Makefile                    |   15 +
 kernel/liveupdate/cpu_preserve.c              | 1811 +++++++++++++++++
 kernel/liveupdate/luo_session.c               |   14 +
 kernel/liveupdate/oncore.c                    |  947 +++++++++
 scripts/mod/modpost.c                         |   32 +-
 tools/objtool/check.c                         |   35 +-
 tools/objtool/include/objtool/check.h         |    1 +
 tools/objtool/include/objtool/elf.h           |    2 +-
 tools/testing/selftests/liveupdate/Makefile   |    1 +
 tools/testing/selftests/liveupdate/config     |    6 +
 tools/testing/selftests/liveupdate/init.c     |   27 +-
 .../testing/selftests/liveupdate/liveupdate.c |  251 +++
 .../selftests/liveupdate/luo_cpu_preserve.c   |  145 ++
 .../selftests/liveupdate/run-vmtests.sh       |    1 +
 tools/testing/selftests/liveupdate/vmtest.sh  |    4 +-
 virt/kvm/Kconfig                              |   41 +
 virt/kvm/Makefile.kvm                         |    8 +
 virt/kvm/caretaker.c                          |  639 ++++++
 virt/kvm/caretaker_debug.c                    |  120 ++
 virt/kvm/kvm_luo.c                            |  249 ++-
 virt/kvm/kvm_main.c                           |   66 +-
 95 files changed, 13415 insertions(+), 122 deletions(-)
 create mode 100644 Documentation/liveupdate/cpu_preservation.rst
 create mode 100644 Documentation/liveupdate/index.rst
 create mode 100644 arch/arm64/include/asm/caretaker.h
 create mode 100644 arch/arm64/include/asm/cpu_preserve.h
 create mode 100644 arch/arm64/include/asm/oncore.h
 create mode 100644 arch/arm64/kernel/cpu_preserve.c
 create mode 100644 arch/arm64/kernel/preserve_cpu.S
 create mode 100644 arch/arm64/kvm/caretaker.c
 create mode 100644 arch/arm64/kvm/caretaker.h
 create mode 100644 arch/arm64/kvm/caretaker_vmenter.S
 create mode 100644 arch/arm64/kvm/kvm_luo.c
 create mode 100644 arch/x86/include/asm/cpu_preserve.h
 create mode 100644 arch/x86/include/asm/oncore.h
 create mode 100644 arch/x86/kernel/cpu_preserve.c
 create mode 100644 arch/x86/kernel/preserve_cpu.S
 create mode 100644 arch/x86/kvm/caretaker.c
 create mode 100644 arch/x86/kvm/caretaker.h
 create mode 100644 arch/x86/kvm/caretaker_asm.h
 create mode 100644 arch/x86/kvm/kvm_luo.c
 create mode 100644 arch/x86/kvm/mmu/kho.c
 create mode 100644 arch/x86/kvm/svm/caretaker.c
 create mode 100644 arch/x86/kvm/svm/caretaker.h
 create mode 100644 arch/x86/kvm/svm/caretaker_vmenter.S
 create mode 100644 arch/x86/kvm/vmx/caretaker.c
 create mode 100644 arch/x86/kvm/vmx/caretaker.h
 create mode 100644 arch/x86/kvm/vmx/caretaker_vmenter.S
 create mode 100644 include/linux/cpu_preserve.h
 create mode 100644 include/linux/kho/abi/cpu.h
 create mode 100644 include/linux/kho/abi/kvm_arm64.h
 create mode 100644 include/linux/kho/abi/kvm_x86.h
 create mode 100644 include/linux/kvm_caretaker.h
 create mode 100644 include/linux/oncore.h
 create mode 100644 kernel/liveupdate/cpu_preserve.c
 create mode 100644 kernel/liveupdate/oncore.c
 create mode 100644 tools/testing/selftests/liveupdate/luo_cpu_preserve.c
 create mode 100644 virt/kvm/caretaker.c
 create mode 100644 virt/kvm/caretaker_debug.c

-- 
2.55.0.1082.g2b9226bbc0-goog

[RFC PATCH 01/46] KVM: luo: Delegate VM creation type to kvm_arch_vm_luo_preserve

[Pasha Tatashin]

Remove the architecture-specific #ifdef ladder setting ser->type in
kvm_luo_preserve(). Leaving ser->type zero by default allows
kvm_arch_vm_luo_preserve() to populate the architecture-defined VM
creation flags without referencing unexported architecture symbols in
generic KVM code.

Signed-off-by: Pasha Tatashin <pasha.tatashin@soleen.com>
---
 virt/kvm/kvm_luo.c | 22 +++++++---------------
 1 file changed, 7 insertions(+), 15 deletions(-)

diff --git a/virt/kvm/kvm_luo.c b/virt/kvm/kvm_luo.c
index 025b53151b6a..b3bd9bebe17a 100644
--- a/virt/kvm/kvm_luo.c
+++ b/virt/kvm/kvm_luo.c
@@ -73,16 +73,13 @@ static int kvm_luo_preserve(struct liveupdate_file_op_args *args)
 	if (IS_ERR(ser))
 		return PTR_ERR(ser);
 
-#if defined(CONFIG_X86)
-	ser->type = kvm->arch.vm_type;
-#elif defined(CONFIG_ARM64)
-	ser->type = kvm_phys_shift(&kvm->arch.mmu);
-	if (kvm_vm_is_protected(kvm))
-		ser->type |= KVM_VM_TYPE_ARM_PROTECTED;
-
-#else
+	/*
+	 * @type is the argument the new kernel will pass to KVM_CREATE_VM, and
+	 * only the architecture knows how to spell it.  kvm_arch_vm_luo_preserve()
+	 * fills it in; leaving it zero here is the right answer for an
+	 * architecture that does not implement the hook.
+	 */
 	ser->type = 0;
-#endif
 
 	args->serialized_data = virt_to_phys(ser);
 	return 0;
@@ -115,7 +112,6 @@ static int kvm_luo_retrieve(struct liveupdate_file_op_args *args)
 	kvm = file->private_data;
 
 	args->file = file;
-	kho_restore_free(ser);
 
 	kvm_uevent_notify_vm_create(kvm);
 	return 0;
@@ -147,11 +143,7 @@ static void kvm_luo_finish(struct liveupdate_file_op_args *args)
 {
 	struct kvm_luo_ser *ser;
 
-	/*
-	 * If retrieve_status is true or set to error, nothing to do here.
-	 * Already cleaned up in kvm_luo_retrieve().
-	 */
-	if (args->retrieve_status)
+	if (args->retrieve_status < 0)
 		return;
 
 	if (!args->serialized_data)
-- 
2.55.0.1082.g2b9226bbc0-goog

[RFC PATCH 02/46] KVM: arm64: Split demux_c15_{get,set}_val from userspace accessors

[Pasha Tatashin]

Factor demux_c15_get_val() and demux_c15_set_val() out of demux_c15_get()
and demux_c15_set() so kernel callers can read and write demultiplexed
coprocessor registers without going through userspace pointers.

Signed-off-by: Pasha Tatashin <pasha.tatashin@soleen.com>
---
 arch/arm64/kvm/sys_regs.c | 48 +++++++++++++++++++++++++++------------
 1 file changed, 34 insertions(+), 14 deletions(-)

diff --git a/arch/arm64/kvm/sys_regs.c b/arch/arm64/kvm/sys_regs.c
index 44aae52c473d..b873ccf1ab14 100644
--- a/arch/arm64/kvm/sys_regs.c
+++ b/arch/arm64/kvm/sys_regs.c
@@ -5496,10 +5496,9 @@ id_to_sys_reg_desc(struct kvm_vcpu *vcpu, u64 id,
 	return r;
 }
 
-static int demux_c15_get(struct kvm_vcpu *vcpu, u64 id, void __user *uaddr)
+static int demux_c15_get_val(struct kvm_vcpu *vcpu, u64 id, u32 *val)
 {
-	u32 val;
-	u32 __user *uval = uaddr;
+	u32 idx;
 
 	/* Fail if we have unknown bits set. */
 	if (id & ~(KVM_REG_ARCH_MASK|KVM_REG_SIZE_MASK|KVM_REG_ARM_COPROC_MASK
@@ -5510,21 +5509,34 @@ static int demux_c15_get(struct kvm_vcpu *vcpu, u64 id, void __user *uaddr)
 	case KVM_REG_ARM_DEMUX_ID_CCSIDR:
 		if (KVM_REG_SIZE(id) != 4)
 			return -ENOENT;
-		val = (id & KVM_REG_ARM_DEMUX_VAL_MASK)
+		idx = (id & KVM_REG_ARM_DEMUX_VAL_MASK)
 			>> KVM_REG_ARM_DEMUX_VAL_SHIFT;
-		if (val >= CSSELR_MAX)
+		if (idx >= CSSELR_MAX)
 			return -ENOENT;
 
-		return put_user(get_ccsidr(vcpu, val), uval);
+		*val = get_ccsidr(vcpu, idx);
+		return 0;
 	default:
 		return -ENOENT;
 	}
 }
 
-static int demux_c15_set(struct kvm_vcpu *vcpu, u64 id, void __user *uaddr)
+static int demux_c15_get(struct kvm_vcpu *vcpu, u64 id, void __user *uaddr)
 {
-	u32 val, newval;
 	u32 __user *uval = uaddr;
+	u32 val;
+	int ret;
+
+	ret = demux_c15_get_val(vcpu, id, &val);
+	if (!ret)
+		ret = put_user(val, uval);
+
+	return ret;
+}
+
+static int demux_c15_set_val(struct kvm_vcpu *vcpu, u64 id, u32 newval)
+{
+	u32 idx;
 
 	/* Fail if we have unknown bits set. */
 	if (id & ~(KVM_REG_ARCH_MASK|KVM_REG_SIZE_MASK|KVM_REG_ARM_COPROC_MASK
@@ -5535,20 +5547,28 @@ static int demux_c15_set(struct kvm_vcpu *vcpu, u64 id, void __user *uaddr)
 	case KVM_REG_ARM_DEMUX_ID_CCSIDR:
 		if (KVM_REG_SIZE(id) != 4)
 			return -ENOENT;
-		val = (id & KVM_REG_ARM_DEMUX_VAL_MASK)
+		idx = (id & KVM_REG_ARM_DEMUX_VAL_MASK)
 			>> KVM_REG_ARM_DEMUX_VAL_SHIFT;
-		if (val >= CSSELR_MAX)
+		if (idx >= CSSELR_MAX)
 			return -ENOENT;
 
-		if (get_user(newval, uval))
-			return -EFAULT;
-
-		return set_ccsidr(vcpu, val, newval);
+		return set_ccsidr(vcpu, idx, newval);
 	default:
 		return -ENOENT;
 	}
 }
 
+static int demux_c15_set(struct kvm_vcpu *vcpu, u64 id, void __user *uaddr)
+{
+	u32 __user *uval = uaddr;
+	u32 newval;
+
+	if (get_user(newval, uval))
+		return -EFAULT;
+
+	return demux_c15_set_val(vcpu, id, newval);
+}
+
 static u64 kvm_one_reg_to_id(const struct kvm_one_reg *reg)
 {
 	switch(reg->id) {
-- 
2.55.0.1082.g2b9226bbc0-goog

[RFC PATCH 03/46] KVM: arm64: Split kvm_sys_reg_{get,set}_user from kernel accessors

[Pasha Tatashin]

Factor __kvm_sys_reg_get() and __kvm_sys_reg_set() out of the userspace
ioctl handlers and introduce struct sys_reg_index_sink so in-kernel
callers can enumerate, read, and write architectural system registers
via kvm_arm_get_sys_reg_indices(), kvm_arm_sys_reg_read(), and
kvm_arm_sys_reg_write().

Signed-off-by: Pasha Tatashin <pasha.tatashin@soleen.com>
---
 arch/arm64/kvm/sys_regs.c | 199 ++++++++++++++++++++++++++++++--------
 arch/arm64/kvm/sys_regs.h |   4 +
 2 files changed, 160 insertions(+), 43 deletions(-)

diff --git a/arch/arm64/kvm/sys_regs.c b/arch/arm64/kvm/sys_regs.c
index b873ccf1ab14..0049e460e5ad 100644
--- a/arch/arm64/kvm/sys_regs.c
+++ b/arch/arm64/kvm/sys_regs.c
@@ -5569,38 +5569,43 @@ static int demux_c15_set(struct kvm_vcpu *vcpu, u64 id, void __user *uaddr)
 	return demux_c15_set_val(vcpu, id, newval);
 }
 
-static u64 kvm_one_reg_to_id(const struct kvm_one_reg *reg)
+static u64 kvm_one_reg_to_id(u64 reg_id)
 {
-	switch(reg->id) {
+	switch (reg_id) {
 	case KVM_REG_ARM_TIMER_CVAL:
 		return TO_ARM64_SYS_REG(CNTV_CVAL_EL0);
 	case KVM_REG_ARM_TIMER_CNT:
 		return TO_ARM64_SYS_REG(CNTVCT_EL0);
 	default:
-		return reg->id;
+		return reg_id;
 	}
 }
 
-int kvm_sys_reg_get_user(struct kvm_vcpu *vcpu, const struct kvm_one_reg *reg,
-			 const struct sys_reg_desc table[], unsigned int num)
+static int __kvm_sys_reg_get(struct kvm_vcpu *vcpu, u64 reg_id, u64 *val,
+			     const struct sys_reg_desc table[], unsigned int num)
 {
-	u64 __user *uaddr = (u64 __user *)(unsigned long)reg->addr;
 	const struct sys_reg_desc *r;
-	u64 id = kvm_one_reg_to_id(reg);
-	u64 val;
-	int ret;
+	u64 id = kvm_one_reg_to_id(reg_id);
 
 	r = id_to_sys_reg_desc(vcpu, id, table, num);
 	if (!r || sysreg_hidden(vcpu, r))
 		return -ENOENT;
 
-	if (r->get_user) {
-		ret = (r->get_user)(vcpu, r, &val);
-	} else {
-		val = __vcpu_sys_reg(vcpu, r->reg);
-		ret = 0;
-	}
+	if (r->get_user)
+		return (r->get_user)(vcpu, r, val);
+
+	*val = __vcpu_sys_reg(vcpu, r->reg);
+	return 0;
+}
+
+int kvm_sys_reg_get_user(struct kvm_vcpu *vcpu, const struct kvm_one_reg *reg,
+			 const struct sys_reg_desc table[], unsigned int num)
+{
+	u64 __user *uaddr = (u64 __user *)(unsigned long)reg->addr;
+	u64 val;
+	int ret;
 
+	ret = __kvm_sys_reg_get(vcpu, reg->id, &val, table, num);
 	if (!ret)
 		ret = put_user(val, uaddr);
 
@@ -5618,17 +5623,11 @@ int kvm_arm_sys_reg_get_reg(struct kvm_vcpu *vcpu, const struct kvm_one_reg *reg
 				    sys_reg_descs, ARRAY_SIZE(sys_reg_descs));
 }
 
-int kvm_sys_reg_set_user(struct kvm_vcpu *vcpu, const struct kvm_one_reg *reg,
-			 const struct sys_reg_desc table[], unsigned int num)
+static int __kvm_sys_reg_set(struct kvm_vcpu *vcpu, u64 reg_id, u64 val,
+			     const struct sys_reg_desc table[], unsigned int num)
 {
-	u64 __user *uaddr = (u64 __user *)(unsigned long)reg->addr;
 	const struct sys_reg_desc *r;
-	u64 id = kvm_one_reg_to_id(reg);
-	u64 val;
-	int ret;
-
-	if (get_user(val, uaddr))
-		return -EFAULT;
+	u64 id = kvm_one_reg_to_id(reg_id);
 
 	r = id_to_sys_reg_desc(vcpu, id, table, num);
 	if (!r || sysreg_hidden(vcpu, r))
@@ -5637,14 +5636,23 @@ int kvm_sys_reg_set_user(struct kvm_vcpu *vcpu, const struct kvm_one_reg *reg,
 	if (sysreg_user_write_ignore(vcpu, r))
 		return 0;
 
-	if (r->set_user) {
-		ret = (r->set_user)(vcpu, r, val);
-	} else {
-		__vcpu_assign_sys_reg(vcpu, r->reg, val);
-		ret = 0;
-	}
+	if (r->set_user)
+		return (r->set_user)(vcpu, r, val);
 
-	return ret;
+	__vcpu_assign_sys_reg(vcpu, r->reg, val);
+	return 0;
+}
+
+int kvm_sys_reg_set_user(struct kvm_vcpu *vcpu, const struct kvm_one_reg *reg,
+			 const struct sys_reg_desc table[], unsigned int num)
+{
+	u64 __user *uaddr = (u64 __user *)(unsigned long)reg->addr;
+	u64 val;
+
+	if (get_user(val, uaddr))
+		return -EFAULT;
+
+	return __kvm_sys_reg_set(vcpu, reg->id, val, table, num);
 }
 
 int kvm_arm_sys_reg_set_reg(struct kvm_vcpu *vcpu, const struct kvm_one_reg *reg)
@@ -5688,25 +5696,24 @@ static u64 sys_reg_to_index(const struct sys_reg_desc *reg)
 		(reg->Op2 << KVM_REG_ARM64_SYSREG_OP2_SHIFT));
 }
 
-static bool copy_reg_to_user(const struct sys_reg_desc *reg, u64 __user **uind)
+static u64 reg_to_user_idx(const struct sys_reg_desc *reg)
 {
-	u64 idx;
-
-	if (!*uind)
-		return true;
-
 	switch (reg_to_encoding(reg)) {
 	case SYS_CNTV_CVAL_EL0:
-		idx = KVM_REG_ARM_TIMER_CVAL;
-		break;
+		return KVM_REG_ARM_TIMER_CVAL;
 	case SYS_CNTVCT_EL0:
-		idx = KVM_REG_ARM_TIMER_CNT;
-		break;
+		return KVM_REG_ARM_TIMER_CNT;
 	default:
-		idx = sys_reg_to_index(reg);
+		return sys_reg_to_index(reg);
 	}
+}
 
-	if (put_user(idx, *uind))
+static bool copy_reg_to_user(const struct sys_reg_desc *reg, u64 __user **uind)
+{
+	if (!*uind)
+		return true;
+
+	if (put_user(reg_to_user_idx(reg), *uind))
 		return false;
 
 	(*uind)++;
@@ -5986,3 +5993,109 @@ int __init kvm_sys_reg_table_init(void)
 
 	return ret;
 }
+
+/* The GICv3 CPU interface sysregs, or NULL if the in-kernel GICv3 is unused. */
+static const struct sys_reg_desc *vcpu_gic_sysreg_table(struct kvm_vcpu *vcpu,
+							unsigned int *num)
+{
+	if (!irqchip_in_kernel(vcpu->kvm) ||
+	    vcpu->kvm->arch.vgic.vgic_model != KVM_DEV_TYPE_ARM_VGIC_V3)
+		return NULL;
+
+	return vgic_v3_get_sysreg_table(num);
+}
+
+static void collect_table_indices(struct kvm_vcpu *vcpu,
+				  const struct sys_reg_desc *table,
+				  unsigned int num, u64 **out, int *count)
+{
+	unsigned int i;
+
+	for (i = 0; i < num; i++) {
+		const struct sys_reg_desc *rd = &table[i];
+
+		if (!(rd->reg || rd->get_user) || sysreg_hidden(vcpu, rd))
+			continue;
+		if (*out)
+			*(*out)++ = reg_to_user_idx(rd);
+		(*count)++;
+	}
+}
+
+/*
+ * Enumerate every system register that kvm_arm_sys_reg_read() can return for
+ * @vcpu.  This is a superset of KVM_GET_REG_LIST: it also covers the GICv3 CPU
+ * interface table, which has no uAPI representation.  Pass a NULL @indices to
+ * query the number of entries the array needs to hold.
+ */
+int kvm_arm_get_sys_reg_indices(struct kvm_vcpu *vcpu, u64 *indices)
+{
+	const struct sys_reg_desc *gic_regs;
+	unsigned int sz = 0, i;
+	int count = 0;
+
+	collect_table_indices(vcpu, sys_reg_descs, ARRAY_SIZE(sys_reg_descs),
+			      &indices, &count);
+
+	gic_regs = vcpu_gic_sysreg_table(vcpu, &sz);
+	if (gic_regs)
+		collect_table_indices(vcpu, gic_regs, sz, &indices, &count);
+
+	for (i = 0; i < CSSELR_MAX; i++) {
+		if (indices)
+			*indices++ = KVM_REG_ARM64 | KVM_REG_SIZE_U32 |
+				     KVM_REG_ARM_DEMUX |
+				     KVM_REG_ARM_DEMUX_ID_CCSIDR | i;
+		count++;
+	}
+
+	return count;
+}
+
+int kvm_arm_sys_reg_read(struct kvm_vcpu *vcpu, u64 reg_id, u64 *val)
+{
+	const struct sys_reg_desc *gic_regs;
+	unsigned int sz = 0;
+	int ret;
+
+	if ((reg_id & KVM_REG_ARM_COPROC_MASK) == KVM_REG_ARM_DEMUX) {
+		u32 uval;
+
+		ret = demux_c15_get_val(vcpu, reg_id, &uval);
+		if (!ret)
+			*val = uval;
+		return ret;
+	}
+
+	ret = __kvm_sys_reg_get(vcpu, reg_id, val, sys_reg_descs,
+				ARRAY_SIZE(sys_reg_descs));
+	if (ret != -ENOENT)
+		return ret;
+
+	gic_regs = vcpu_gic_sysreg_table(vcpu, &sz);
+	if (!gic_regs)
+		return ret;
+
+	return __kvm_sys_reg_get(vcpu, reg_id, val, gic_regs, sz);
+}
+
+int kvm_arm_sys_reg_write(struct kvm_vcpu *vcpu, u64 reg_id, u64 val)
+{
+	const struct sys_reg_desc *gic_regs;
+	unsigned int sz = 0;
+	int ret;
+
+	if ((reg_id & KVM_REG_ARM_COPROC_MASK) == KVM_REG_ARM_DEMUX)
+		return demux_c15_set_val(vcpu, reg_id, (u32)val);
+
+	ret = __kvm_sys_reg_set(vcpu, reg_id, val, sys_reg_descs,
+				ARRAY_SIZE(sys_reg_descs));
+	if (ret != -ENOENT)
+		return ret;
+
+	gic_regs = vcpu_gic_sysreg_table(vcpu, &sz);
+	if (!gic_regs)
+		return ret;
+
+	return __kvm_sys_reg_set(vcpu, reg_id, val, gic_regs, sz);
+}
diff --git a/arch/arm64/kvm/sys_regs.h b/arch/arm64/kvm/sys_regs.h
index bd56a45abbf9..9837db99909b 100644
--- a/arch/arm64/kvm/sys_regs.h
+++ b/arch/arm64/kvm/sys_regs.h
@@ -292,4 +292,8 @@ int kvm_finalize_sys_regs(struct kvm_vcpu *vcpu);
 					      sys_reg_CRm(SYS_ ## r),	\
 					      sys_reg_Op2(SYS_ ## r))
 
+int kvm_arm_get_sys_reg_indices(struct kvm_vcpu *vcpu, u64 *indices);
+int kvm_arm_sys_reg_read(struct kvm_vcpu *vcpu, u64 reg_id, u64 *val);
+int kvm_arm_sys_reg_write(struct kvm_vcpu *vcpu, u64 reg_id, u64 val);
+
 #endif /* __ARM64_KVM_SYS_REGS_LOCAL_H__ */
-- 
2.55.0.1082.g2b9226bbc0-goog

[RFC PATCH 04/46] x86/mm/ident_map: Add force_pte to support 4K PTE identity mappings

[Pasha Tatashin]

Extend kernel_ident_mapping_init() with ident_pte_init() and a force_pte
flag in struct x86_mapping_info so callers can request 4K PTE leaf
mappings instead of 2M PMD or 1G PUD leaves.

Signed-off-by: Pasha Tatashin <pasha.tatashin@soleen.com>
---
 arch/x86/include/asm/init.h |  3 +-
 arch/x86/mm/ident_map.c     | 72 ++++++++++++++++++++++++++++++++-----
 2 files changed, 65 insertions(+), 10 deletions(-)

diff --git a/arch/x86/include/asm/init.h b/arch/x86/include/asm/init.h
index 01ccdd168df0..d9caed494e7d 100644
--- a/arch/x86/include/asm/init.h
+++ b/arch/x86/include/asm/init.h
@@ -6,10 +6,11 @@ struct x86_mapping_info {
 	void *(*alloc_pgt_page)(void *); /* allocate buf for page table */
 	void (*free_pgt_page)(void *, void *); /* free buf for page table */
 	void *context;			 /* context for alloc_pgt_page */
-	unsigned long page_flag;	 /* page flag for PMD or PUD entry */
+	unsigned long page_flag;	 /* page flag for PTE, PMD or PUD entry */
 	unsigned long offset;		 /* ident mapping offset */
 	bool direct_gbpages;		 /* PUD level 1GB page support */
 	unsigned long kernpg_flag;	 /* kernel pagetable flag override */
+	bool force_pte;			 /* force 4K PTE mappings */
 };
 
 int kernel_ident_mapping_init(struct x86_mapping_info *info, pgd_t *pgd_page,
diff --git a/arch/x86/mm/ident_map.c b/arch/x86/mm/ident_map.c
index 5a15bffe6574..83013513ba94 100644
--- a/arch/x86/mm/ident_map.c
+++ b/arch/x86/mm/ident_map.c
@@ -77,24 +77,73 @@ void kernel_ident_mapping_free(struct x86_mapping_info *info, pgd_t *pgd)
 	info->free_pgt_page(pgd, info->context);
 }
 
-static void ident_pmd_init(struct x86_mapping_info *info, pmd_t *pmd_page,
-			   unsigned long addr, unsigned long end)
+static int ident_pte_init(struct x86_mapping_info *info, pte_t *pte_page,
+			  unsigned long addr, unsigned long end)
 {
-	addr &= PMD_MASK;
-	for (; addr < end; addr += PMD_SIZE) {
+	addr &= PAGE_MASK;
+	for (; addr < end; addr += PAGE_SIZE) {
+		pte_t *pte = pte_page + pte_index(addr);
+
+		if (pte_present(*pte))
+			continue;
+
+		set_pte(pte, __pte(((addr - info->offset) | info->page_flag) & ~_PAGE_PSE));
+	}
+
+	return 0;
+}
+
+static int ident_pmd_init(struct x86_mapping_info *info, pmd_t *pmd_page,
+			  unsigned long addr, unsigned long end)
+{
+	unsigned long next;
+	int result;
+
+	for (; addr < end; addr = next) {
 		pmd_t *pmd = pmd_page + pmd_index(addr);
+		pte_t *pte;
 
-		if (pmd_present(*pmd))
+		next = pmd_addr_end(addr, end);
+
+		if (!info->force_pte) {
+			if (pmd_present(*pmd))
+				continue;
+
+			set_pmd(pmd, __pmd(((addr & PMD_MASK) - info->offset) | info->page_flag));
 			continue;
+		}
 
-		set_pmd(pmd, __pmd((addr - info->offset) | info->page_flag));
+		/* if this is already a 2MB page, this portion is already mapped */
+		if (pmd_leaf(*pmd))
+			continue;
+
+		if (pmd_present(*pmd)) {
+			pte = pte_offset_kernel(pmd, 0);
+			result = ident_pte_init(info, pte, addr, next);
+			if (result)
+				return result;
+			continue;
+		}
+
+		pte = (pte_t *)info->alloc_pgt_page(info->context);
+		if (!pte)
+			return -ENOMEM;
+
+		result = ident_pte_init(info, pte, addr, next);
+		if (result)
+			return result;
+
+		set_pmd(pmd, __pmd(__pa(pte) | info->kernpg_flag));
 	}
+
+	return 0;
 }
 
 static int ident_pud_init(struct x86_mapping_info *info, pud_t *pud_page,
 			  unsigned long addr, unsigned long end)
 {
 	unsigned long next;
+	int result;
 
 	for (; addr < end; addr = next) {
 		pud_t *pud = pud_page + pud_index(addr);
@@ -108,7 +157,7 @@ static int ident_pud_init(struct x86_mapping_info *info, pud_t *pud_page,
 			continue;
 
 		/* Is using a gbpage allowed? */
-		use_gbpage = info->direct_gbpages;
+		use_gbpage = info->direct_gbpages && !info->force_pte;
 
 		/* Don't use gbpage if it maps more than the requested region. */
 		/* at the beginning: */
@@ -129,13 +178,17 @@ static int ident_pud_init(struct x86_mapping_info *info, pud_t *pud_page,
 
 		if (pud_present(*pud)) {
 			pmd = pmd_offset(pud, 0);
-			ident_pmd_init(info, pmd, addr, next);
+			result = ident_pmd_init(info, pmd, addr, next);
+			if (result)
+				return result;
 			continue;
 		}
 		pmd = (pmd_t *)info->alloc_pgt_page(info->context);
 		if (!pmd)
 			return -ENOMEM;
-		ident_pmd_init(info, pmd, addr, next);
+		result = ident_pmd_init(info, pmd, addr, next);
+		if (result)
+			return result;
 		set_pud(pud, __pud(__pa(pmd) | info->kernpg_flag));
 	}
 
@@ -189,6 +242,7 @@ int kernel_ident_mapping_init(struct x86_mapping_info *info, pgd_t *pgd_page,
 
 	/* Filter out unsupported __PAGE_KERNEL_* bits: */
 	info->kernpg_flag &= __default_kernel_pte_mask;
+	info->page_flag &= __default_kernel_pte_mask;
 
 	for (; addr < end; addr = next) {
 		pgd_t *pgd = pgd_page + pgd_index(addr);
-- 
2.55.0.1082.g2b9226bbc0-goog

[RFC PATCH 05/46] arm64: mm: Add trans_pgd_map_range() support

[Pasha Tatashin]

Introduce trans_pgd_map_range() to allow mapping arbitrary physical
memory ranges into transitional page tables.

This helper allocates intermediate translation levels (P4D, PUD, PMD,
and PTE) on demand using the trans_pgd allocator, enabling precise
virtual-to-physical address mappings in transitional contexts such as
isolated execution environments.

Add comprehensive kernel documentation detailing the isolated page
table mappings: Caretaker text/rodata (ROX), writable data (RW),
stacks (RW), and preserved buffers without cloning the direct map.

Signed-off-by: Pasha Tatashin <pasha.tatashin@soleen.com>
---
 arch/arm64/include/asm/trans_pgd.h |  4 ++
 arch/arm64/mm/trans_pgd.c          | 70 ++++++++++++++++++++++++++++++
 2 files changed, 74 insertions(+)

diff --git a/arch/arm64/include/asm/trans_pgd.h b/arch/arm64/include/asm/trans_pgd.h
index 033d400a4ea4..1e17ae145b0c 100644
--- a/arch/arm64/include/asm/trans_pgd.h
+++ b/arch/arm64/include/asm/trans_pgd.h
@@ -30,6 +30,10 @@ struct trans_pgd_info {
 int trans_pgd_create_copy(struct trans_pgd_info *info, pgd_t **trans_pgd,
 			  unsigned long start, unsigned long end);
 
+int trans_pgd_map_range(struct trans_pgd_info *info, pgd_t *trans_pgd,
+			phys_addr_t pa, unsigned long va, size_t size,
+			pgprot_t prot);
+
 int trans_pgd_idmap_page(struct trans_pgd_info *info, phys_addr_t *trans_ttbr0,
 			 unsigned long *t0sz, void *page);
 
diff --git a/arch/arm64/mm/trans_pgd.c b/arch/arm64/mm/trans_pgd.c
index cca9706a875c..f0a55d4270cc 100644
--- a/arch/arm64/mm/trans_pgd.c
+++ b/arch/arm64/mm/trans_pgd.c
@@ -200,6 +200,76 @@ int trans_pgd_create_copy(struct trans_pgd_info *info, pgd_t **dst_pgdp,
 	return rc;
 }
 
+/**
+ * trans_pgd_map_range - Map a physical address range into a transition page table
+ * @info: Transition page table allocation info containing the page allocator
+ * @trans_pgd: Root transition page table pointer
+ * @pa: Physical address to map
+ * @va: Virtual address to map to
+ * @size: Size of the range to map in bytes (page-aligned)
+ * @prot: Page protection attributes (e.g. PAGE_KERNEL, PAGE_KERNEL_ROX)
+ *
+ * Populates the page table hierarchy (P4D, PUD, PMD, PTE) allocating new
+ * intermediate tables using info->trans_alloc_page() as needed. Used by
+ * liveupdate Caretaker to construct isolated page tables.
+ *
+ * Return: 0 on success, -ENOMEM on allocation failure.
+ */
+int trans_pgd_map_range(struct trans_pgd_info *info, pgd_t *trans_pgd,
+			phys_addr_t pa, unsigned long va, size_t size,
+			pgprot_t prot)
+{
+	unsigned long end = va + size;
+	unsigned long addr = va;
+
+	while (addr < end) {
+		pgd_t *pgdp = pgd_offset_pgd(trans_pgd, addr);
+		p4d_t *p4dp;
+		pud_t *pudp;
+		pmd_t *pmdp;
+		pte_t *ptep;
+
+		if (pgd_none(READ_ONCE(*pgdp))) {
+			p4dp = trans_alloc(info);
+			if (!p4dp)
+				return -ENOMEM;
+			pgd_populate(&init_mm, pgdp, p4dp);
+		}
+		p4dp = p4d_offset(pgdp, addr);
+
+		if (p4d_none(READ_ONCE(*p4dp))) {
+			pudp = trans_alloc(info);
+			if (!pudp)
+				return -ENOMEM;
+			p4d_populate(&init_mm, p4dp, pudp);
+		}
+		pudp = pud_offset(p4dp, addr);
+
+		if (pud_none(READ_ONCE(*pudp))) {
+			pmdp = trans_alloc(info);
+			if (!pmdp)
+				return -ENOMEM;
+			pud_populate(&init_mm, pudp, pmdp);
+		}
+		pmdp = pmd_offset(pudp, addr);
+
+		if (pmd_none(READ_ONCE(*pmdp))) {
+			ptep = trans_alloc(info);
+			if (!ptep)
+				return -ENOMEM;
+			pmd_populate_kernel(&init_mm, pmdp, ptep);
+		}
+		ptep = pte_offset_kernel(pmdp, addr);
+
+		set_pte(ptep, pfn_pte(PHYS_PFN(pa), prot));
+
+		addr += PAGE_SIZE;
+		pa += PAGE_SIZE;
+	}
+
+	return 0;
+}
+
 /*
  * The page we want to idmap may be outside the range covered by VA_BITS that
  * can be built using the kernel's p?d_populate() helpers. As a one off, for a
-- 
2.55.0.1082.g2b9226bbc0-goog

[RFC PATCH 06/46] x86/smp: Skip offline CPUs for REBOOT_VECTOR in native_stop_other_cpus()

[Pasha Tatashin]

In native_stop_other_cpus(), cpus_stop_mask is initialized to
cpu_online_mask minus the calling CPU. However,
apic_send_IPI_allbutself(REBOOT_VECTOR) uses the APIC broadcast
shorthand when enabled, which delivers the IPI to all physical APICs
including offline CPUs.

Use __apic_send_IPI_mask(&cpus_stop_mask, REBOOT_VECTOR) so that
REBOOT_VECTOR is sent only to the target online CPUs in
cpus_stop_mask.

Signed-off-by: Pasha Tatashin <pasha.tatashin@soleen.com>
---
 arch/x86/kernel/smp.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/arch/x86/kernel/smp.c b/arch/x86/kernel/smp.c
index 985103cab16c..2449d15c6667 100644
--- a/arch/x86/kernel/smp.c
+++ b/arch/x86/kernel/smp.c
@@ -192,7 +192,7 @@ static void native_stop_other_cpus(int wait)
 	cpumask_clear_cpu(this_cpu, &cpus_stop_mask);
 
 	if (!cpumask_empty(&cpus_stop_mask)) {
-		apic_send_IPI_allbutself(REBOOT_VECTOR);
+		__apic_send_IPI_mask(&cpus_stop_mask, REBOOT_VECTOR);
 
 		/*
 		 * Don't wait longer than a second for IPI completion. The
-- 
2.55.0.1082.g2b9226bbc0-goog

[RFC PATCH 07/46] KVM: luo: Support vCPU file preservation across live updates

[Pasha Tatashin]

Register LUO file handler for KVM vCPU file descriptors to allow
preserving vCPU state across live updates.

Signed-off-by: Pasha Tatashin <pasha.tatashin@soleen.com>
---
 include/linux/kho/abi/kvm.h |  25 ++++++
 include/linux/kvm_host.h    |  53 ++++++++++++
 include/uapi/linux/kvm.h    |   1 +
 virt/kvm/Kconfig            |   3 +
 virt/kvm/kvm_luo.c          | 161 +++++++++++++++++++++++++++++++++++-
 virt/kvm/kvm_main.c         |  58 ++++++++++---
 6 files changed, 286 insertions(+), 15 deletions(-)

diff --git a/include/linux/kho/abi/kvm.h b/include/linux/kho/abi/kvm.h
index 42074d76e04a..166a0a2f13c8 100644
--- a/include/linux/kho/abi/kvm.h
+++ b/include/linux/kho/abi/kvm.h
@@ -39,6 +39,31 @@ struct kvm_luo_ser {
 /* The compatibility string for KVM VM file handler */
 #define KVM_LUO_FH_COMPATIBLE	"kvm_vm_luo_v1"
 
+/**
+ * enum kvm_vcpu_luo_flags - Flags for KVM vCPU LUO preservation
+ * @KVM_VCPU_LUO_FLAG_CARETAKER: vCPU is preserved with on-core Caretaker execution.
+ */
+enum kvm_vcpu_luo_flags {
+	KVM_VCPU_LUO_FLAG_CARETAKER = BIT(0),
+};
+
+/**
+ * struct kvm_vcpu_ser - Main serialization structure for a KVM vCPU.
+ * @vcpu_id:    The ID of the virtual CPU.
+ * @flags:      Flags for vCPU preservation.
+ * @vm_token:   Token of the associated KVM VM instance.
+ * @arch_state: Preservation pointer to vCPU architectural state.
+ */
+struct kvm_vcpu_ser {
+	u32 vcpu_id;
+	u32 flags;
+	u64 vm_token;
+	DECLARE_KHOSER_PTR(arch_state, struct kvm_vcpu_arch_ser *);
+} __packed;
+
+/* The compatibility string for KVM vCPU file handler */
+#define KVM_VCPU_LUO_FH_COMPATIBLE	"kvm_vcpu_luo_v1"
+
 /**
  * struct guest_memfd_luo_folio_ser - Serialization layout for a single folio in guest_memfd.
  * @pfn:   Page Frame Number of the folio.
diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h
index 1217cd6ec6ab..643b941286c2 100644
--- a/include/linux/kvm_host.h
+++ b/include/linux/kvm_host.h
@@ -1094,7 +1094,9 @@ void kvm_get_kvm(struct kvm *kvm);
 bool kvm_get_kvm_safe(struct kvm *kvm);
 void kvm_put_kvm(struct kvm *kvm);
 bool file_is_kvm(struct file *file);
+bool file_is_kvm_vcpu(struct file *file);
 struct file *kvm_create_vm_file(unsigned long type, const char *fdname);
+struct file *kvm_create_vcpu_file(struct kvm *kvm, unsigned long id);
 void kvm_put_kvm_no_destroy(struct kvm *kvm);
 void kvm_uevent_notify_vm_create(struct kvm *kvm);
 
@@ -2661,4 +2663,55 @@ long kvm_arch_vcpu_pre_fault_memory(struct kvm_vcpu *vcpu,
 				    struct kvm_pre_fault_memory *range);
 #endif
 
+struct kvm_luo_ser;
+struct kvm_vcpu_ser;
+
+#ifdef CONFIG_HAVE_KVM_ARCH_VCPU_PRESERVE
+int kvm_arch_vm_luo_preserve(struct kvm *kvm, struct kvm_luo_ser *ser);
+int kvm_arch_vm_luo_retrieve(struct kvm *kvm, struct kvm_luo_ser *ser);
+void kvm_arch_vm_luo_unpreserve(struct kvm *kvm, struct kvm_luo_ser *ser);
+/*
+ * Unlike unpreserve(), finish() runs after the kexec on a VM that was never
+ * reclaimed, so there is no struct kvm to pass: it may only free what @ser
+ * points at.
+ */
+void kvm_arch_vm_luo_finish(struct kvm_luo_ser *ser);
+
+int kvm_arch_vcpu_luo_preserve(struct kvm_vcpu *vcpu, struct kvm_vcpu_ser *ser);
+int kvm_arch_vcpu_luo_retrieve(struct kvm_vcpu *vcpu, struct kvm_vcpu_ser *ser);
+void kvm_arch_vcpu_luo_unpreserve(struct kvm_vcpu_ser *ser);
+void kvm_arch_vcpu_luo_finish(struct kvm_vcpu_ser *ser);
+#else
+static inline int kvm_arch_vm_luo_preserve(struct kvm *kvm,
+					   struct kvm_luo_ser *ser)
+{
+	return 0;
+}
+
+static inline int kvm_arch_vm_luo_retrieve(struct kvm *kvm,
+					   struct kvm_luo_ser *ser)
+{
+	return 0;
+}
+
+static inline void kvm_arch_vm_luo_unpreserve(struct kvm *kvm,
+					      struct kvm_luo_ser *ser) {}
+static inline void kvm_arch_vm_luo_finish(struct kvm_luo_ser *ser) {}
+
+static inline int kvm_arch_vcpu_luo_preserve(struct kvm_vcpu *vcpu,
+					     struct kvm_vcpu_ser *ser)
+{
+	return 0;
+}
+
+static inline int kvm_arch_vcpu_luo_retrieve(struct kvm_vcpu *vcpu,
+					     struct kvm_vcpu_ser *ser)
+{
+	return 0;
+}
+
+static inline void kvm_arch_vcpu_luo_unpreserve(struct kvm_vcpu_ser *ser) {}
+static inline void kvm_arch_vcpu_luo_finish(struct kvm_vcpu_ser *ser) {}
+#endif
+
 #endif
diff --git a/include/uapi/linux/kvm.h b/include/uapi/linux/kvm.h
index ac2d77d14963..4b1754050681 100644
--- a/include/uapi/linux/kvm.h
+++ b/include/uapi/linux/kvm.h
@@ -999,6 +999,7 @@ struct kvm_enable_cap {
 #define KVM_CAP_S390_HPAGE_2G 249
 #define KVM_CAP_PPC_COMPAT_CAPS 250
 #define KVM_CAP_ARM_PMU_V3_STRICT 251
+#define KVM_CAP_VCPU_PRESERVE 252
 
 struct kvm_irq_routing_irqchip {
 	__u32 irqchip;
diff --git a/virt/kvm/Kconfig b/virt/kvm/Kconfig
index c3c0ee253fc7..fcaf57377e73 100644
--- a/virt/kvm/Kconfig
+++ b/virt/kvm/Kconfig
@@ -71,6 +71,9 @@ config KVM_GENERIC_DIRTYLOG_READ_PROTECT
 config KVM_GENERIC_PRE_FAULT_MEMORY
        bool
 
+config HAVE_KVM_ARCH_VCPU_PRESERVE
+       bool
+
 config KVM_COMPAT
        def_bool y
        depends on KVM && COMPAT && !(S390 || ARM64 || RISCV)
diff --git a/virt/kvm/kvm_luo.c b/virt/kvm/kvm_luo.c
index b3bd9bebe17a..f7e259bef4b7 100644
--- a/virt/kvm/kvm_luo.c
+++ b/virt/kvm/kvm_luo.c
@@ -35,7 +35,6 @@
  *
  * The preservation does not cover:
  *
- * - vCPUs and vCPU states
  * - Memspots / Memory slot layout (memslots)
  * - Interrupt controllers and IRQ routings
  * - Coalesced MMIO zones
@@ -46,6 +45,7 @@
 #include <linux/liveupdate.h>
 #include <linux/kvm_host.h>
 #include <linux/pagemap.h>
+#include <linux/fdtable.h>
 #include <linux/file.h>
 #include <linux/err.h>
 #include <linux/anon_inodes.h>
@@ -65,6 +65,7 @@ static int kvm_luo_preserve(struct liveupdate_file_op_args *args)
 {
 	struct kvm *kvm = args->file->private_data;
 	struct kvm_luo_ser *ser;
+	int err;
 
 	if (kvm->vm_dead || kvm->vm_bugged)
 		return -EINVAL;
@@ -80,6 +81,11 @@ static int kvm_luo_preserve(struct liveupdate_file_op_args *args)
 	 * architecture that does not implement the hook.
 	 */
 	ser->type = 0;
+	err = kvm_arch_vm_luo_preserve(kvm, ser);
+	if (err) {
+		kho_unpreserve_free(ser);
+		return err;
+	}
 
 	args->serialized_data = virt_to_phys(ser);
 	return 0;
@@ -111,18 +117,26 @@ static int kvm_luo_retrieve(struct liveupdate_file_op_args *args)
 
 	kvm = file->private_data;
 
+	err = kvm_arch_vm_luo_retrieve(kvm, ser);
+	if (err) {
+		fput(file);
+		goto err_free_ser;
+	}
+
 	args->file = file;
 
 	kvm_uevent_notify_vm_create(kvm);
 	return 0;
 
 err_free_ser:
+	kvm_arch_vm_luo_finish(ser);
 	kho_restore_free(ser);
 	return err;
 }
 
 static void kvm_luo_unpreserve(struct liveupdate_file_op_args *args)
 {
+	struct kvm *kvm = args->file ? args->file->private_data : NULL;
 	struct kvm_luo_ser *ser;
 
 	/*
@@ -135,7 +149,7 @@ static void kvm_luo_unpreserve(struct liveupdate_file_op_args *args)
 		return;
 
 	ser = phys_to_virt(args->serialized_data);
-
+	kvm_arch_vm_luo_unpreserve(kvm, ser);
 	kho_unpreserve_free(ser);
 }
 
@@ -150,7 +164,7 @@ static void kvm_luo_finish(struct liveupdate_file_op_args *args)
 		return;
 
 	ser = phys_to_virt(args->serialized_data);
-
+	kvm_arch_vm_luo_finish(ser);
 	kho_restore_free(ser);
 }
 
@@ -168,6 +182,139 @@ static struct liveupdate_file_handler kvm_luo_handler = {
 	.compatible = KVM_LUO_FH_COMPATIBLE,
 };
 
+static bool kvm_vcpu_luo_can_preserve(struct liveupdate_file_handler *handler,
+				      struct file *file)
+{
+	return file_is_kvm_vcpu(file);
+}
+
+static int kvm_vcpu_luo_preserve(struct liveupdate_file_op_args *args)
+{
+	struct kvm_vcpu *vcpu = args->file->private_data;
+	struct kvm_vcpu_ser *ser;
+	struct file *kvm_file;
+	u64 vm_token;
+	int err;
+
+	kvm_file = get_file_active(&vcpu->kvm->vm_file);
+	if (!kvm_file)
+		return -ENOENT;
+
+	err = liveupdate_get_token_outgoing(args->session, kvm_file, &vm_token);
+	fput(kvm_file);
+	if (err)
+		return err;
+
+	if (mutex_lock_killable(&vcpu->mutex))
+		return -EINTR;
+
+	ser = kho_alloc_preserve(sizeof(*ser));
+	if (IS_ERR(ser)) {
+		mutex_unlock(&vcpu->mutex);
+		return PTR_ERR(ser);
+	}
+
+	ser->vcpu_id = vcpu->vcpu_id;
+	ser->flags = 0;
+	ser->vm_token = vm_token;
+
+	err = kvm_arch_vcpu_luo_preserve(vcpu, ser);
+	mutex_unlock(&vcpu->mutex);
+	if (err) {
+		kho_unpreserve_free(ser);
+		return err;
+	}
+
+	args->serialized_data = virt_to_phys(ser);
+	return 0;
+}
+
+static int kvm_vcpu_luo_retrieve(struct liveupdate_file_op_args *args)
+{
+	struct kvm_vcpu_ser *ser;
+	struct file *vm_file, *file;
+	struct kvm_vcpu *vcpu;
+	int err;
+
+	if (!args->serialized_data)
+		return -EINVAL;
+
+	ser = phys_to_virt(args->serialized_data);
+
+	err = liveupdate_get_file_incoming(args->session, ser->vm_token, &vm_file);
+	if (err)
+		goto err_free_ser;
+
+	if (!file_is_kvm(vm_file)) {
+		fput(vm_file);
+		err = -EINVAL;
+		goto err_free_ser;
+	}
+
+	file = kvm_create_vcpu_file(vm_file->private_data, ser->vcpu_id);
+	fput(vm_file);
+	if (IS_ERR(file)) {
+		err = PTR_ERR(file);
+		goto err_free_ser;
+	}
+
+	vcpu = file->private_data;
+	err = kvm_arch_vcpu_luo_retrieve(vcpu, ser);
+	if (err) {
+		fput(file);
+		goto err_free_ser;
+	}
+
+	args->file = file;
+	return 0;
+
+err_free_ser:
+	kvm_arch_vcpu_luo_finish(ser);
+	kho_restore_free(ser);
+	return err;
+}
+
+static void kvm_vcpu_luo_unpreserve(struct liveupdate_file_op_args *args)
+{
+	struct kvm_vcpu_ser *ser;
+
+	if (WARN_ON_ONCE(!args->serialized_data))
+		return;
+
+	ser = phys_to_virt(args->serialized_data);
+	kvm_arch_vcpu_luo_unpreserve(ser);
+	kho_unpreserve_free(ser);
+}
+
+static void kvm_vcpu_luo_finish(struct liveupdate_file_op_args *args)
+{
+	struct kvm_vcpu_ser *ser;
+
+	if (args->retrieve_status < 0)
+		return;
+
+	if (!args->serialized_data)
+		return;
+
+	ser = phys_to_virt(args->serialized_data);
+	kvm_arch_vcpu_luo_finish(ser);
+	kho_restore_free(ser);
+}
+
+static const struct liveupdate_file_ops kvm_vcpu_luo_file_ops = {
+	.can_preserve = kvm_vcpu_luo_can_preserve,
+	.preserve = kvm_vcpu_luo_preserve,
+	.retrieve = kvm_vcpu_luo_retrieve,
+	.unpreserve = kvm_vcpu_luo_unpreserve,
+	.finish = kvm_vcpu_luo_finish,
+	.owner = THIS_MODULE,
+};
+
+static struct liveupdate_file_handler kvm_vcpu_luo_handler = {
+	.ops = &kvm_vcpu_luo_file_ops,
+	.compatible = KVM_VCPU_LUO_FH_COMPATIBLE,
+};
+
 int kvm_luo_init(void)
 {
 	int err = liveupdate_register_file_handler(&kvm_luo_handler);
@@ -177,11 +324,19 @@ int kvm_luo_init(void)
 		return err;
 	}
 
+	err = liveupdate_register_file_handler(&kvm_vcpu_luo_handler);
+	if (err && err != -EOPNOTSUPP) {
+		pr_err("Could not register kvm_vcpu_luo handler: %pe\n", ERR_PTR(err));
+		liveupdate_unregister_file_handler(&kvm_luo_handler);
+		return err;
+	}
+
 	return 0;
 }
 
 void kvm_luo_exit(void)
 {
+	liveupdate_unregister_file_handler(&kvm_vcpu_luo_handler);
 	liveupdate_unregister_file_handler(&kvm_luo_handler);
 }
 
diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c
index 4b03a65a8428..ae91123e1822 100644
--- a/virt/kvm/kvm_main.c
+++ b/virt/kvm/kvm_main.c
@@ -4121,12 +4121,12 @@ static struct file_operations kvm_vcpu_fops = {
 /*
  * Allocates an inode for the vcpu.
  */
-static int create_vcpu_fd(struct kvm_vcpu *vcpu)
+static struct file *create_vcpu_file(struct kvm_vcpu *vcpu)
 {
 	char name[8 + 1 + ITOA_MAX_LEN + 1];
 
 	snprintf(name, sizeof(name), "kvm-vcpu:%d", vcpu->vcpu_id);
-	return anon_inode_getfd(name, &kvm_vcpu_fops, vcpu, O_RDWR | O_CLOEXEC);
+	return anon_inode_getfile(name, &kvm_vcpu_fops, vcpu, O_RDWR);
 }
 
 #ifdef __KVM_HAVE_ARCH_VCPU_DEBUGFS
@@ -4163,11 +4163,12 @@ static void kvm_create_vcpu_debugfs(struct kvm_vcpu *vcpu)
 /*
  * Creates some virtual cpus.  Good luck creating more than one.
  */
-static int kvm_vm_ioctl_create_vcpu(struct kvm *kvm, unsigned long id)
+struct file *kvm_create_vcpu_file(struct kvm *kvm, unsigned long id)
 {
-	int r;
 	struct kvm_vcpu *vcpu;
+	struct file *file;
 	struct page *page;
+	int r;
 
 	/*
 	 * KVM tracks vCPU IDs as 'int', be kind to userspace and reject
@@ -4179,23 +4180,23 @@ static int kvm_vm_ioctl_create_vcpu(struct kvm *kvm, unsigned long id)
 	 */
 	BUILD_BUG_ON(KVM_MAX_VCPU_IDS > INT_MAX);
 	if (id >= KVM_MAX_VCPU_IDS)
-		return -EINVAL;
+		return ERR_PTR(-EINVAL);
 
 	mutex_lock(&kvm->lock);
 	if (kvm->created_vcpus >= kvm->max_vcpus) {
 		mutex_unlock(&kvm->lock);
-		return -EINVAL;
+		return ERR_PTR(-EINVAL);
 	}
 
 	if (test_bit(id, kvm->vcpu_ids)) {
 		mutex_unlock(&kvm->lock);
-		return -EEXIST;
+		return ERR_PTR(-EEXIST);
 	}
 
 	r = kvm_arch_vcpu_precreate(kvm, id);
 	if (r) {
 		mutex_unlock(&kvm->lock);
-		return r;
+		return ERR_PTR(r);
 	}
 
 	kvm->created_vcpus++;
@@ -4259,9 +4260,11 @@ static int kvm_vm_ioctl_create_vcpu(struct kvm *kvm, unsigned long id)
 	 */
 	mutex_lock(&vcpu->mutex);
 	kvm_get_kvm(kvm);
-	r = create_vcpu_fd(vcpu);
-	if (r < 0)
+	file = create_vcpu_file(vcpu);
+	if (IS_ERR(file)) {
+		r = PTR_ERR(file);
 		goto kvm_put_xa_erase;
+	}
 
 	/*
 	 * Pairs with smp_rmb() in kvm_get_vcpu.  Store the vcpu
@@ -4274,7 +4277,7 @@ static int kvm_vm_ioctl_create_vcpu(struct kvm *kvm, unsigned long id)
 	mutex_unlock(&kvm->lock);
 	kvm_arch_vcpu_postcreate(vcpu);
 	kvm_create_vcpu_debugfs(vcpu);
-	return r;
+	return file;
 
 kvm_put_xa_erase:
 	mutex_unlock(&vcpu->mutex);
@@ -4295,7 +4298,30 @@ static int kvm_vm_ioctl_create_vcpu(struct kvm *kvm, unsigned long id)
 	kvm->created_vcpus--;
 	__clear_bit(id, kvm->vcpu_ids);
 	mutex_unlock(&kvm->lock);
-	return r;
+	return ERR_PTR(r);
+}
+
+static int kvm_vm_ioctl_create_vcpu(struct kvm *kvm, unsigned long id)
+{
+	struct file *file;
+	int fd;
+
+	/*
+	 * Reserve the fd up front: kvm_create_vcpu_file() publishes the vCPU
+	 * in kvm->vcpu_array, and there is no clean way to unwind that.
+	 */
+	fd = get_unused_fd_flags(O_RDWR | O_CLOEXEC);
+	if (fd < 0)
+		return fd;
+
+	file = kvm_create_vcpu_file(kvm, id);
+	if (IS_ERR(file)) {
+		put_unused_fd(fd);
+		return PTR_ERR(file);
+	}
+
+	fd_install(fd, file);
+	return fd;
 }
 
 static int kvm_vcpu_ioctl_set_sigmask(struct kvm_vcpu *vcpu, sigset_t *sigset)
@@ -4961,6 +4987,8 @@ static int kvm_vm_ioctl_check_extension_generic(struct kvm *kvm, long arg)
 	case KVM_CAP_GUEST_MEMFD_FLAGS:
 		return kvm_gmem_get_supported_flags(kvm);
 #endif
+	case KVM_CAP_VCPU_PRESERVE:
+		return IS_ENABLED(CONFIG_HAVE_KVM_ARCH_VCPU_PRESERVE);
 	default:
 		break;
 	}
@@ -5506,6 +5534,12 @@ bool file_is_kvm(struct file *file)
 }
 EXPORT_SYMBOL_FOR_KVM_INTERNAL(file_is_kvm);
 
+bool file_is_kvm_vcpu(struct file *file)
+{
+	return file && file->f_op == &kvm_vcpu_fops;
+}
+EXPORT_SYMBOL_FOR_KVM_INTERNAL(file_is_kvm_vcpu);
+
 struct file *kvm_create_vm_file(unsigned long type, const char *fdname)
 {
 	struct kvm *kvm = kvm_create_vm(type, fdname);
-- 
2.55.0.1082.g2b9226bbc0-goog

[RFC PATCH 08/46] KVM: x86: Add x86 vCPU LUO preservation ABI and register helpers

[Pasha Tatashin]

Add struct kvm_vcpu_arch_ser in include/linux/kho/abi/kvm_x86.h
and export x86 register and MSR access helpers for LUO state save/restore.

Signed-off-by: Pasha Tatashin <pasha.tatashin@soleen.com>
---
 arch/x86/kvm/cpuid.c            |  4 +-
 arch/x86/kvm/cpuid.h            |  1 +
 arch/x86/kvm/msrs.c             | 15 ++++++
 arch/x86/kvm/msrs.h             |  2 +
 arch/x86/kvm/regs.c             |  8 ++--
 arch/x86/kvm/regs.h             |  4 ++
 arch/x86/kvm/x86.c              |  8 ++--
 arch/x86/kvm/x86.h              |  4 ++
 include/linux/kho/abi/kvm_x86.h | 84 +++++++++++++++++++++++++++++++++
 9 files changed, 120 insertions(+), 10 deletions(-)
 create mode 100644 include/linux/kho/abi/kvm_x86.h

diff --git a/arch/x86/kvm/cpuid.c b/arch/x86/kvm/cpuid.c
index 851f151efb35..5aba16e0610b 100644
--- a/arch/x86/kvm/cpuid.c
+++ b/arch/x86/kvm/cpuid.c
@@ -507,8 +507,8 @@ u64 kvm_vcpu_reserved_gpa_bits_raw(struct kvm_vcpu *vcpu)
 	return rsvd_bits(cpuid_maxphyaddr(vcpu), 63);
 }
 
-static int kvm_set_cpuid(struct kvm_vcpu *vcpu, struct kvm_cpuid_entry2 *e2,
-                        int nent)
+int kvm_set_cpuid(struct kvm_vcpu *vcpu, struct kvm_cpuid_entry2 *e2,
+		  int nent)
 {
 	u32 vcpu_caps[NR_KVM_CPU_CAPS];
 	int r;
diff --git a/arch/x86/kvm/cpuid.h b/arch/x86/kvm/cpuid.h
index 8d863f45585d..e5ec112d39cc 100644
--- a/arch/x86/kvm/cpuid.h
+++ b/arch/x86/kvm/cpuid.h
@@ -21,6 +21,7 @@ static inline void kvm_finalize_cpu_caps(void)
 }
 
 void kvm_vcpu_after_set_cpuid(struct kvm_vcpu *vcpu);
+int kvm_set_cpuid(struct kvm_vcpu *vcpu, struct kvm_cpuid_entry2 *e2, int nent);
 struct kvm_cpuid_entry2 *kvm_find_cpuid_entry2(struct kvm_cpuid_entry2 *entries,
 					       int nent, u32 function, u64 index);
 /*
diff --git a/arch/x86/kvm/msrs.c b/arch/x86/kvm/msrs.c
index dd3bb04878ca..ae849e0b7d14 100644
--- a/arch/x86/kvm/msrs.c
+++ b/arch/x86/kvm/msrs.c
@@ -339,6 +339,21 @@ static u32 msr_based_features[ARRAY_SIZE(msr_based_features_all_except_vmx) +
 			      (KVM_LAST_EMULATED_VMX_MSR - KVM_FIRST_EMULATED_VMX_MSR + 1)];
 static unsigned int num_msr_based_features;
 
+unsigned int kvm_num_msrs_to_save(void)
+{
+	return num_msrs_to_save + num_emulated_msrs;
+}
+
+u32 kvm_get_msr_to_save_index(unsigned int i)
+{
+	if (i < num_msrs_to_save)
+		return msrs_to_save[i];
+	i -= num_msrs_to_save;
+	if (i < num_emulated_msrs)
+		return emulated_msrs[i];
+	return 0;
+}
+
 int kvm_get_msr_index_list(struct kvm_msr_list __user *user_msr_list)
 {
 	struct kvm_msr_list msr_list;
diff --git a/arch/x86/kvm/msrs.h b/arch/x86/kvm/msrs.h
index 7cc182a15b3b..3a5e39041d99 100644
--- a/arch/x86/kvm/msrs.h
+++ b/arch/x86/kvm/msrs.h
@@ -65,6 +65,8 @@ int __kvm_emulate_msr_read(struct kvm_vcpu *vcpu, u32 index, u64 *data);
 int __kvm_emulate_msr_write(struct kvm_vcpu *vcpu, u32 index, u64 data);
 int kvm_msr_read(struct kvm_vcpu *vcpu, u32 index, u64 *data);
 int kvm_msr_write(struct kvm_vcpu *vcpu, u32 index, u64 data);
+unsigned int kvm_num_msrs_to_save(void);
+u32 kvm_get_msr_to_save_index(unsigned int i);
 int kvm_emulate_rdmsr(struct kvm_vcpu *vcpu);
 int kvm_emulate_rdmsr_imm(struct kvm_vcpu *vcpu, u32 msr, int reg);
 int kvm_emulate_wrmsr(struct kvm_vcpu *vcpu);
diff --git a/arch/x86/kvm/regs.c b/arch/x86/kvm/regs.c
index 8f66438989e4..02f041c355cc 100644
--- a/arch/x86/kvm/regs.c
+++ b/arch/x86/kvm/regs.c
@@ -51,7 +51,7 @@ void kvm_set_rflags(struct kvm_vcpu *vcpu, unsigned long rflags)
 }
 EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_set_rflags);
 
-static void __get_regs(struct kvm_vcpu *vcpu, struct kvm_regs *regs)
+void __get_regs(struct kvm_vcpu *vcpu, struct kvm_regs *regs)
 {
 	if (vcpu->arch.emulate_regs_need_sync_to_vcpu) {
 		/*
@@ -99,7 +99,7 @@ int kvm_arch_vcpu_ioctl_get_regs(struct kvm_vcpu *vcpu, struct kvm_regs *regs)
 	return 0;
 }
 
-static void __set_regs(struct kvm_vcpu *vcpu, struct kvm_regs *regs)
+void __set_regs(struct kvm_vcpu *vcpu, struct kvm_regs *regs)
 {
 	vcpu->arch.emulate_regs_need_sync_from_vcpu = true;
 	vcpu->arch.emulate_regs_need_sync_to_vcpu = false;
@@ -498,7 +498,7 @@ static void __get_sregs_common(struct kvm_vcpu *vcpu, struct kvm_sregs *sregs)
 	sregs->apic_base = vcpu->arch.apic_base;
 }
 
-static void __get_sregs(struct kvm_vcpu *vcpu, struct kvm_sregs *sregs)
+void __get_sregs(struct kvm_vcpu *vcpu, struct kvm_sregs *sregs)
 {
 	__get_sregs_common(vcpu, sregs);
 
@@ -635,7 +635,7 @@ static int __set_sregs_common(struct kvm_vcpu *vcpu, struct kvm_sregs *sregs,
 	return 0;
 }
 
-static int __set_sregs(struct kvm_vcpu *vcpu, struct kvm_sregs *sregs)
+int __set_sregs(struct kvm_vcpu *vcpu, struct kvm_sregs *sregs)
 {
 	int pending_vec, max_bits;
 	int mmu_reset_needed = 0;
diff --git a/arch/x86/kvm/regs.h b/arch/x86/kvm/regs.h
index 447f0ec3e63e..40947b1be404 100644
--- a/arch/x86/kvm/regs.h
+++ b/arch/x86/kvm/regs.h
@@ -517,5 +517,9 @@ int kvm_vcpu_ioctl_x86_get_debugregs(struct kvm_vcpu *vcpu,
 int kvm_vcpu_ioctl_x86_set_debugregs(struct kvm_vcpu *vcpu,
 				     struct kvm_debugregs *dbgregs);
 
+void __get_regs(struct kvm_vcpu *vcpu, struct kvm_regs *regs);
+void __set_regs(struct kvm_vcpu *vcpu, struct kvm_regs *regs);
+void __get_sregs(struct kvm_vcpu *vcpu, struct kvm_sregs *sregs);
+int __set_sregs(struct kvm_vcpu *vcpu, struct kvm_sregs *sregs);
 
 #endif
diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
index 79468ddfe473..69e0d67a309b 100644
--- a/arch/x86/kvm/x86.c
+++ b/arch/x86/kvm/x86.c
@@ -2959,8 +2959,8 @@ void kvm_handle_exception_payload_quirk(struct kvm_vcpu *vcpu)
 		kvm_deliver_exception_payload(vcpu, ex);
 }
 
-static void kvm_vcpu_ioctl_x86_get_vcpu_events(struct kvm_vcpu *vcpu,
-					       struct kvm_vcpu_events *events)
+void kvm_vcpu_ioctl_x86_get_vcpu_events(struct kvm_vcpu *vcpu,
+					struct kvm_vcpu_events *events)
 {
 	struct kvm_queued_exception *ex = kvm_get_exception_to_save(vcpu);
 
@@ -3028,8 +3028,8 @@ static void kvm_vcpu_ioctl_x86_get_vcpu_events(struct kvm_vcpu *vcpu,
 	}
 }
 
-static int kvm_vcpu_ioctl_x86_set_vcpu_events(struct kvm_vcpu *vcpu,
-					      struct kvm_vcpu_events *events)
+int kvm_vcpu_ioctl_x86_set_vcpu_events(struct kvm_vcpu *vcpu,
+				       struct kvm_vcpu_events *events)
 {
 	if (events->flags & ~(KVM_VCPUEVENT_VALID_NMI_PENDING
 			      | KVM_VCPUEVENT_VALID_SIPI_VECTOR
diff --git a/arch/x86/kvm/x86.h b/arch/x86/kvm/x86.h
index 0f5919b092e4..936ee660ccec 100644
--- a/arch/x86/kvm/x86.h
+++ b/arch/x86/kvm/x86.h
@@ -492,6 +492,10 @@ int kvm_task_switch(struct kvm_vcpu *vcpu, u16 tss_selector, int idt_index,
 		    int reason, bool has_error_code, u32 error_code);
 
 int __kvm_set_xcr(struct kvm_vcpu *vcpu, u32 index, u64 xcr);
+void kvm_vcpu_ioctl_x86_get_vcpu_events(struct kvm_vcpu *vcpu,
+					struct kvm_vcpu_events *events);
+int kvm_vcpu_ioctl_x86_set_vcpu_events(struct kvm_vcpu *vcpu,
+				       struct kvm_vcpu_events *events);
 int kvm_emulate_xsetbv(struct kvm_vcpu *vcpu);
 int kvm_emulate_rdpmc(struct kvm_vcpu *vcpu);
 
diff --git a/include/linux/kho/abi/kvm_x86.h b/include/linux/kho/abi/kvm_x86.h
new file mode 100644
index 000000000000..316250482db5
--- /dev/null
+++ b/include/linux/kho/abi/kvm_x86.h
@@ -0,0 +1,84 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright (c) 2026, Google LLC.
+ * Pasha Tatashin <pasha.tatashin@soleen.com>
+ */
+#ifndef _LINUX_KHO_ABI_KVM_X86_H
+#define _LINUX_KHO_ABI_KVM_X86_H
+
+#ifdef CONFIG_X86_64
+
+#include <linux/build_bug.h>
+#include <linux/stddef.h>
+#include <linux/types.h>
+#include <linux/kho/abi/kvm.h>
+#include <uapi/linux/kvm.h>
+#include <uapi/asm/kvm.h>
+
+/**
+ * DOC: x86 KVM Live Update ABI
+ *
+ * x86 KVM uses the ABI defined below for preserving architectural VM and vCPU
+ * state across a kexec reboot using LUO.
+ *
+ * The vCPU-level architectural state contains the compact register sets and
+ * CPUID table for each vCPU.
+ *
+ * All sub-structures (struct kvm_regs, struct kvm_sregs, struct kvm_mp_state,
+ * struct kvm_cpuid_entry2) are uAPI contracts.
+ */
+
+/**
+ * struct kvm_vcpu_arch_ser - Preserved x86 architectural vCPU state in RAM.
+ * @regs:          General-purpose registers (uAPI struct kvm_regs).
+ * @sregs:         Segment and control registers (uAPI struct kvm_sregs).
+ * @mp_state:      Multiprocessor state (uAPI struct kvm_mp_state).
+ * @pad:           Padding to maintain 64-bit alignment after mp_state.
+ * @xcrs:          Extended control registers including XCR0 (uAPI struct kvm_xcrs).
+ * @lapic:         In-kernel local APIC register state (uAPI struct kvm_lapic_state).
+ * @pad_xsave:     Padding that aligns @xsave to 64 bytes; see the static_assert
+ *                 below.  Must be zero.
+ * @xsave:         Extended processor state (uAPI struct kvm_xsave).  This is the
+ *                 only copy of the guest FPU state; there is no kvm_fpu twin.
+ * @events:        vCPU exception, interrupt, NMI and SMI events (uAPI struct kvm_vcpu_events).
+ * @debugregs:     Hardware debug registers DR0-DR7 (uAPI struct kvm_debugregs).
+ * @num_msrs:      Number of valid MSR entries in msrs.
+ * @cpuid_nent:    Number of valid CPUID entries immediately following msrs[num_msrs].
+ * @msrs:          Array of preserved architectural and paravirtual MSR entries,
+ *                 followed by @cpuid_nent struct kvm_cpuid_entry2 entries.
+ */
+struct kvm_vcpu_arch_ser {
+	struct kvm_regs regs;
+	struct kvm_sregs sregs;
+	struct kvm_mp_state mp_state;
+	u32 pad;
+	struct kvm_xcrs xcrs;
+	struct kvm_lapic_state lapic;
+	u8 pad_xsave[40];
+	struct kvm_xsave xsave;
+	struct kvm_vcpu_events events;
+	struct kvm_debugregs debugregs;
+	u32 num_msrs;
+	u32 cpuid_nent;
+	struct kvm_msr_entry msrs[];
+} __packed;
+
+static_assert(offsetof(struct kvm_vcpu_arch_ser, msrs) % sizeof(u64) == 0,
+	      "msrs must be 64-bit aligned");
+
+/*
+ * @xsave must be a legal XSAVE destination.  The structure is __packed, so
+ * without @pad_xsave the member lands at offset 1880, which is not even
+ * 16-byte aligned, and the XSAVE/XRSTOR family faults on anything less than
+ * 64.  The allocation itself is fine: kho_alloc_preserve() returns a folio
+ * address, so the base is page aligned.
+ *
+ * If this assert fires, a member above @xsave changed size; adjust
+ * @pad_xsave rather than deleting the assert.
+ */
+static_assert(offsetof(struct kvm_vcpu_arch_ser, xsave) % 64 == 0,
+	      "xsave must be 64-byte aligned to be XSAVE-able in place");
+
+#endif /* CONFIG_X86_64 */
+
+#endif /* _LINUX_KHO_ABI_KVM_X86_H */
-- 
2.55.0.1082.g2b9226bbc0-goog

[RFC PATCH 09/46] KVM: x86: Implement architectural vCPU state preservation via LUO

[Pasha Tatashin]

Implement x86 KVM vCPU state preservation and restoration handlers for
LUO in arch/x86/kvm/kvm_luo.c.

Signed-off-by: Pasha Tatashin <pasha.tatashin@soleen.com>
---
 arch/x86/kvm/Kconfig   |   1 +
 arch/x86/kvm/Makefile  |   1 +
 arch/x86/kvm/kvm_luo.c | 338 +++++++++++++++++++++++++++++++++++++++++
 3 files changed, 340 insertions(+)
 create mode 100644 arch/x86/kvm/kvm_luo.c

diff --git a/arch/x86/kvm/Kconfig b/arch/x86/kvm/Kconfig
index 538ed1e80332..bae79fded6ff 100644
--- a/arch/x86/kvm/Kconfig
+++ b/arch/x86/kvm/Kconfig
@@ -48,6 +48,7 @@ config KVM_X86
 	select KVM_GENERIC_PRE_FAULT_MEMORY
 	select KVM_WERROR if WERROR
 	select KVM_GUEST_MEMFD if X86_64
+	select HAVE_KVM_ARCH_VCPU_PRESERVE
 
 config KVM
 	tristate "Kernel-based Virtual Machine (KVM) support"
diff --git a/arch/x86/kvm/Makefile b/arch/x86/kvm/Makefile
index 0474604ab8a1..2cf0f1f2a59b 100644
--- a/arch/x86/kvm/Makefile
+++ b/arch/x86/kvm/Makefile
@@ -7,6 +7,7 @@ include $(srctree)/virt/kvm/Makefile.kvm
 
 kvm-y			+= x86.o emulate.o irq.o lapic.o cpuid.o msrs.o pmu.o regs.o \
 			   mtrr.o debugfs.o mmu/mmu.o mmu/page_track.o mmu/spte.o
+kvm-$(CONFIG_LIVEUPDATE) += kvm_luo.o
 
 kvm-$(CONFIG_X86_64) += mmu/tdp_iter.o mmu/tdp_mmu.o
 kvm-$(CONFIG_KVM_IOAPIC) += i8259.o i8254.o ioapic.o
diff --git a/arch/x86/kvm/kvm_luo.c b/arch/x86/kvm/kvm_luo.c
new file mode 100644
index 000000000000..899b193812bb
--- /dev/null
+++ b/arch/x86/kvm/kvm_luo.c
@@ -0,0 +1,338 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Copyright (c) 2026, Google LLC.
+ * Pasha Tatashin <pasha.tatashin@soleen.com>
+ *
+ * x86 KVM LUO architectural preservation and retrieval handlers.
+ */
+
+#include <linux/cpu.h>
+#include <linux/kexec_handover.h>
+#include <linux/kho/abi/kvm_x86.h>
+#include <linux/kvm_host.h>
+#include <linux/mm.h>
+#include <linux/slab.h>
+#include <linux/string.h>
+
+#include <asm/fpu/api.h>
+#include <asm/fpu/xcr.h>
+#include <linux/kvm_host.h>
+#include <linux/mem_encrypt.h>
+#include <asm/virt.h>
+
+#include "cpuid.h"
+#include "fpu.h"
+#include "lapic.h"
+#include "msrs.h"
+#include "pmu.h"
+#include "regs.h"
+#include "x86.h"
+
+int kvm_arch_vm_luo_preserve(struct kvm *kvm, struct kvm_luo_ser *ser)
+{
+	ser->type = kvm->arch.vm_type;
+	return 0;
+}
+EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_arch_vm_luo_preserve);
+
+int kvm_arch_vm_luo_retrieve(struct kvm *kvm, struct kvm_luo_ser *ser)
+{
+	return 0;
+}
+EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_arch_vm_luo_retrieve);
+
+void kvm_arch_vm_luo_unpreserve(struct kvm *kvm, struct kvm_luo_ser *ser)
+{
+}
+EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_arch_vm_luo_unpreserve);
+
+void kvm_arch_vm_luo_finish(struct kvm_luo_ser *ser)
+{
+}
+EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_arch_vm_luo_finish);
+
+int kvm_arch_vcpu_luo_preserve(struct kvm_vcpu *vcpu, struct kvm_vcpu_ser *ser)
+{
+	struct kvm_vcpu_arch_ser *state;
+	bool fpu_loaded = false;
+	unsigned int max_msrs;
+	u32 nent;
+	size_t size;
+	int i;
+
+	/*
+	 * The guest FPU state travels in @xsave, which can only be filled in if
+	 * the host has XSAVE.  There is no second copy to fall back on, so fail
+	 * the preserve rather than silently dropping the guest's FPU registers.
+	 * A confidential guest keeps its FPU state in the VMSA, where KVM can
+	 * neither read nor restore it, so it needs nothing from us.
+	 */
+	if (!boot_cpu_has(X86_FEATURE_XSAVE) &&
+	    !fpstate_is_confidential(&vcpu->arch.guest_fpu))
+		return -EOPNOTSUPP;
+
+	/*
+	 * Nested virtualisation state does not survive the handover.
+	 * arm64 already refuses vcpu_has_nv() outright; do the same here.
+	 */
+	if (is_guest_mode(vcpu))
+		return -EOPNOTSUPP;
+
+	if (kvm_nested_ops.enabled && kvm_nested_ops.get_state) {
+		int nested_size = kvm_nested_ops.get_state(vcpu, NULL, 0);
+
+		if (nested_size < 0)
+			return nested_size;
+		if (nested_size > sizeof(struct kvm_nested_state))
+			return -EOPNOTSUPP;
+	}
+
+	/*
+	 * @xsave is a fixed-size struct kvm_xsave.  A guest whose XSAVE area
+	 * outgrows it -- AMX today -- needs KVM_GET_XSAVE2 and would otherwise
+	 * be truncated silently by fpu_copy_guest_fpstate_to_uabi() below.
+	 * This mirrors the check kvm_arch_vcpu_ioctl() makes for KVM_GET_XSAVE.
+	 */
+	if (vcpu->arch.guest_fpu.uabi_size > sizeof(struct kvm_xsave))
+		return -EOPNOTSUPP;
+
+	max_msrs = kvm_num_msrs_to_save();
+	nent = vcpu->arch.cpuid_entries ? vcpu->arch.cpuid_nent : 0;
+	size = sizeof(*state) + max_msrs * sizeof(struct kvm_msr_entry) +
+	       nent * sizeof(struct kvm_cpuid_entry2);
+	state = kho_alloc_preserve(size);
+	if (IS_ERR(state))
+		return PTR_ERR(state);
+
+	/*
+	 * kvm_arch_vcpu_ioctl_{get,set}_mpstate() take the vCPU themselves, so
+	 * they have to be called outside the vcpu_load() region below:
+	 * vcpu_load() registers a preempt notifier and is not reentrant.
+	 */
+	kvm_arch_vcpu_ioctl_get_mpstate(vcpu, &state->mp_state);
+	state->pad = 0;
+
+	vcpu_load(vcpu);
+
+	__get_regs(vcpu, &state->regs);
+	__get_sregs(vcpu, &state->sregs);
+	state->xcrs.nr_xcrs = 1;
+	state->xcrs.flags = 0;
+	state->xcrs.xcrs[0].xcr = XCR_XFEATURE_ENABLED_MASK;
+	state->xcrs.xcrs[0].value = vcpu->arch.xcr0;
+
+	if (boot_cpu_has(X86_FEATURE_XSAVE) &&
+	    !fpstate_is_confidential(&vcpu->arch.guest_fpu)) {
+		u64 supported_xcr0 = vcpu->arch.guest_supported_xcr0 |
+				     XFEATURE_MASK_FPSSE;
+
+		fpu_copy_guest_fpstate_to_uabi(&vcpu->arch.guest_fpu,
+					       state->xsave.region,
+					       sizeof(state->xsave.region),
+					       supported_xcr0,
+					       vcpu->arch.pkru);
+	}
+
+	kvm_vcpu_ioctl_x86_get_debugregs(vcpu, &state->debugregs);
+
+	/*
+	 * A hardware maskable interrupt can only be queued for injection when
+	 * RFLAGS.IF is set.  If RFLAGS.IF is already clear, the interrupt gate
+	 * delivery has already completed (e.g., under QEMU TCG where
+	 * exit_int_info can remain set across the first instruction of the
+	 * handler), so clear any stale injected flag to avoid delivering the
+	 * same interrupt a second time with RFLAGS.IF == 0 before SWAPGS.
+	 */
+	if (vcpu->arch.interrupt.injected && !vcpu->arch.interrupt.soft &&
+	    !(state->regs.rflags & X86_EFLAGS_IF))
+		vcpu->arch.interrupt.injected = false;
+
+	kvm_vcpu_ioctl_x86_get_vcpu_events(vcpu, &state->events);
+
+	if (lapic_in_kernel(vcpu))
+		kvm_apic_get_state(vcpu, &state->lapic);
+
+	if (!vcpu->arch.guest_fpu.fpstate->in_use) {
+		kvm_load_guest_fpu(vcpu);
+		fpu_loaded = true;
+	}
+
+	state->num_msrs = 0;
+	for (i = 0; i < max_msrs; i++) {
+		u32 msr = kvm_get_msr_to_save_index(i);
+		u64 val = 0;
+
+		if (kvm_msr_read(vcpu, msr, &val) == 0) {
+			state->msrs[state->num_msrs].index = msr;
+			state->msrs[state->num_msrs].reserved = 0;
+			state->msrs[state->num_msrs].data = val;
+			state->num_msrs++;
+		}
+	}
+
+	state->cpuid_nent = nent;
+	if (nent) {
+		struct kvm_cpuid_entry2 *cpuid_dst =
+			(void *)&state->msrs[state->num_msrs];
+
+		memcpy(cpuid_dst, vcpu->arch.cpuid_entries,
+		       nent * sizeof(*cpuid_dst));
+	}
+
+	if (fpu_loaded)
+		kvm_put_guest_fpu(vcpu);
+
+	vcpu_put(vcpu);
+
+	KHOSER_STORE_PTR(ser->arch_state, state);
+	return 0;
+}
+EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_arch_vcpu_luo_preserve);
+
+int kvm_arch_vcpu_luo_retrieve(struct kvm_vcpu *vcpu, struct kvm_vcpu_ser *ser)
+{
+	struct kvm_vcpu_arch_ser *state;
+	bool fpu_loaded = false;
+	int ret, i;
+
+	if (!ser->arch_state.phys)
+		return 0;
+
+	state = KHOSER_LOAD_PTR(ser->arch_state);
+
+	vcpu_load(vcpu);
+
+	if (state->cpuid_nent > 0) {
+		struct kvm_cpuid_entry2 *cpuid_src =
+			(void *)&state->msrs[state->num_msrs];
+		struct kvm_cpuid_entry2 *entries;
+
+		entries = kmemdup(cpuid_src,
+				  state->cpuid_nent * sizeof(*entries),
+				  GFP_KERNEL);
+		if (!entries) {
+			ret = -ENOMEM;
+			goto out;
+		}
+
+		ret = kvm_set_cpuid(vcpu, entries, state->cpuid_nent);
+		if (ret) {
+			kvfree(entries);
+			goto out;
+		}
+	}
+
+	ret = __set_sregs(vcpu, &state->sregs);
+	if (ret)
+		goto out;
+
+	if (boot_cpu_has(X86_FEATURE_XSAVE)) {
+		__kvm_set_xcr(vcpu, state->xcrs.xcrs[0].xcr,
+			      state->xcrs.xcrs[0].value);
+	}
+
+	for (i = 0; i < state->num_msrs; i++) {
+		if (state->msrs[i].index == MSR_IA32_XFD ||
+		    state->msrs[i].index == MSR_IA32_XFD_ERR ||
+		    state->msrs[i].index == MSR_IA32_XSS) {
+			kvm_msr_write(vcpu, state->msrs[i].index, state->msrs[i].data);
+		}
+	}
+
+	if (boot_cpu_has(X86_FEATURE_XSAVE) &&
+	    !fpstate_is_confidential(&vcpu->arch.guest_fpu)) {
+		union fpregs_state *xstate = (union fpregs_state *)state->xsave.region;
+
+		xstate->xsave.header.xfeatures &= ~vcpu->arch.guest_fpu.fpstate->xfd;
+		ret = fpu_copy_uabi_to_guest_fpstate(&vcpu->arch.guest_fpu,
+						     state->xsave.region,
+						     kvm_caps.supported_xcr0,
+						     &vcpu->arch.pkru);
+		if (ret)
+			goto out;
+	}
+
+	ret = kvm_vcpu_ioctl_x86_set_debugregs(vcpu, &state->debugregs);
+	if (ret)
+		goto out;
+
+	if (kvm_nested_ops.enabled)
+		kvm_leave_nested(vcpu);
+
+	if (!(ser->flags & KVM_VCPU_LUO_FLAG_CARETAKER)) {
+		ret = kvm_vcpu_ioctl_x86_set_vcpu_events(vcpu, &state->events);
+		if (ret)
+			goto out;
+	}
+
+	if (lapic_in_kernel(vcpu)) {
+		if (ser->flags & KVM_VCPU_LUO_FLAG_CARETAKER) {
+			struct kvm_lapic_state lapic;
+			int k;
+
+			memcpy(&lapic, &state->lapic, sizeof(lapic));
+			for (k = 0; k < 8; k++) {
+				*(u32 *)(lapic.regs + APIC_ISR + 0x10 * k) = 0;
+				*(u32 *)(lapic.regs + APIC_IRR + 0x10 * k) = 0;
+			}
+			ret = kvm_apic_set_state(vcpu, &lapic);
+		} else {
+			ret = kvm_apic_set_state(vcpu, &state->lapic);
+		}
+		if (ret)
+			goto out;
+	}
+
+	if (!vcpu->arch.guest_fpu.fpstate->in_use) {
+		kvm_load_guest_fpu(vcpu);
+		fpu_loaded = true;
+	}
+
+	for (i = 0; i < state->num_msrs; i++) {
+		if ((ser->flags & KVM_VCPU_LUO_FLAG_CARETAKER) &&
+		    (state->msrs[i].index == MSR_IA32_TSC ||
+		     state->msrs[i].index == MSR_IA32_TSC_ADJUST)) {
+			continue;
+		}
+		kvm_msr_write(vcpu, state->msrs[i].index, state->msrs[i].data);
+	}
+
+	__set_regs(vcpu, &state->regs);
+
+	ret = 0;
+out:
+	if (fpu_loaded)
+		kvm_put_guest_fpu(vcpu);
+	vcpu_put(vcpu);
+
+	/* Takes the vCPU itself; see the comment in the preserve path. */
+	if (!ret)
+		ret = kvm_arch_vcpu_ioctl_set_mpstate(vcpu, &state->mp_state);
+
+	return ret;
+}
+EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_arch_vcpu_luo_retrieve);
+
+void kvm_arch_vcpu_luo_unpreserve(struct kvm_vcpu_ser *ser)
+{
+	if (ser->arch_state.phys) {
+		struct kvm_vcpu_arch_ser *state =
+			phys_to_virt(__sme_clr(ser->arch_state.phys));
+
+		kho_unpreserve_free(state);
+		ser->arch_state.phys = 0;
+	}
+}
+EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_arch_vcpu_luo_unpreserve);
+
+void kvm_arch_vcpu_luo_finish(struct kvm_vcpu_ser *ser)
+{
+	if (ser->arch_state.phys) {
+		struct kvm_vcpu_arch_ser *state =
+			phys_to_virt(__sme_clr(ser->arch_state.phys));
+
+		kho_restore_free(state);
+		ser->arch_state.phys = 0;
+	}
+}
+EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_arch_vcpu_luo_finish);
-- 
2.55.0.1082.g2b9226bbc0-goog

[RFC PATCH 10/46] KVM: arm64: Implement architectural vCPU state preservation via LUO

[Pasha Tatashin]

Implement ARM64 architectural vCPU state preservation and retrieval
handlers over LUO using struct kvm_vcpu_arch_ser in
include/linux/kho/abi/kvm_arm64.h.

Signed-off-by: Pasha Tatashin <pasha.tatashin@soleen.com>
---
 arch/arm64/kvm/Kconfig            |   1 +
 arch/arm64/kvm/Makefile           |   2 +
 arch/arm64/kvm/kvm_luo.c          | 201 ++++++++++++++++++++++++++++++
 include/linux/kho/abi/kvm_arm64.h |  61 +++++++++
 4 files changed, 265 insertions(+)
 create mode 100644 arch/arm64/kvm/kvm_luo.c
 create mode 100644 include/linux/kho/abi/kvm_arm64.h

diff --git a/arch/arm64/kvm/Kconfig b/arch/arm64/kvm/Kconfig
index 449154f9a485..fd25b3c0d8ca 100644
--- a/arch/arm64/kvm/Kconfig
+++ b/arch/arm64/kvm/Kconfig
@@ -37,6 +37,7 @@ menuconfig KVM
 	select SCHED_INFO
 	select GUEST_PERF_EVENTS if PERF_EVENTS
 	select KVM_GUEST_MEMFD
+	select HAVE_KVM_ARCH_VCPU_PRESERVE
 	help
 	  Support hosting virtualized guest machines.
 
diff --git a/arch/arm64/kvm/Makefile b/arch/arm64/kvm/Makefile
index 59612d2f277c..d5e888d1bd50 100644
--- a/arch/arm64/kvm/Makefile
+++ b/arch/arm64/kvm/Makefile
@@ -47,3 +47,5 @@ $(obj)/hyp_constants.h: $(obj)/hyp-constants.s FORCE
 
 obj-kvm := $(addprefix $(obj)/, $(kvm-y))
 $(obj-kvm): $(obj)/hyp_constants.h
+
+kvm-y += kvm_luo.o
diff --git a/arch/arm64/kvm/kvm_luo.c b/arch/arm64/kvm/kvm_luo.c
new file mode 100644
index 000000000000..59c56836cda8
--- /dev/null
+++ b/arch/arm64/kvm/kvm_luo.c
@@ -0,0 +1,201 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * Copyright (c) 2026, Google LLC.
+ * Pasha Tatashin <pasha.tatashin@soleen.com>
+ *
+ * ARM64 KVM LUO preservation and retrieval handlers.
+ */
+
+#include <linux/kexec_handover.h>
+#include <linux/kho/abi/kvm_arm64.h>
+#include <linux/kvm_host.h>
+#include <linux/sched.h>
+
+#include <asm/kvm_emulate.h>
+#include <asm/kvm_mmu.h>
+
+#include <kvm/arm_arch_timer.h>
+#include <kvm/arm_vgic.h>
+
+#include "sys_regs.h"
+#include "vgic/vgic.h"
+
+int kvm_arch_vm_luo_preserve(struct kvm *kvm, struct kvm_luo_ser *ser)
+{
+	ser->type = kvm_phys_shift(&kvm->arch.mmu);
+	if (kvm_vm_is_protected(kvm))
+		ser->type |= KVM_VM_TYPE_ARM_PROTECTED;
+
+	return 0;
+}
+
+int kvm_arch_vm_luo_retrieve(struct kvm *kvm, struct kvm_luo_ser *ser)
+{
+	return 0;
+}
+
+void kvm_arch_vm_luo_unpreserve(struct kvm *kvm, struct kvm_luo_ser *ser)
+{
+}
+
+void kvm_arch_vm_luo_finish(struct kvm_luo_ser *ser)
+{
+}
+
+static void kvm_arm_luo_get_regs(struct kvm_vcpu *vcpu, struct kvm_regs *regs)
+{
+	regs->regs = vcpu->arch.ctxt.regs;
+	regs->sp_el1 = ctxt_sys_reg(&vcpu->arch.ctxt, SP_EL1);
+	regs->elr_el1 = ctxt_sys_reg(&vcpu->arch.ctxt, ELR_EL1);
+	regs->spsr[KVM_SPSR_EL1] = ctxt_sys_reg(&vcpu->arch.ctxt, SPSR_EL1);
+	regs->spsr[KVM_SPSR_ABT] = vcpu->arch.ctxt.spsr_abt;
+	regs->spsr[KVM_SPSR_UND] = vcpu->arch.ctxt.spsr_und;
+	regs->spsr[KVM_SPSR_IRQ] = vcpu->arch.ctxt.spsr_irq;
+	regs->spsr[KVM_SPSR_FIQ] = vcpu->arch.ctxt.spsr_fiq;
+	regs->fp_regs = vcpu->arch.ctxt.fp_regs;
+}
+
+static void kvm_arm_luo_set_regs(struct kvm_vcpu *vcpu, const struct kvm_regs *regs)
+{
+	vcpu->arch.ctxt.regs = regs->regs;
+	ctxt_sys_reg(&vcpu->arch.ctxt, SP_EL1) = regs->sp_el1;
+	ctxt_sys_reg(&vcpu->arch.ctxt, ELR_EL1) = regs->elr_el1;
+	ctxt_sys_reg(&vcpu->arch.ctxt, SPSR_EL1) = regs->spsr[KVM_SPSR_EL1];
+	vcpu->arch.ctxt.spsr_abt = regs->spsr[KVM_SPSR_ABT];
+	vcpu->arch.ctxt.spsr_und = regs->spsr[KVM_SPSR_UND];
+	vcpu->arch.ctxt.spsr_irq = regs->spsr[KVM_SPSR_IRQ];
+	vcpu->arch.ctxt.spsr_fiq = regs->spsr[KVM_SPSR_FIQ];
+	vcpu->arch.ctxt.fp_regs = regs->fp_regs;
+}
+
+int kvm_arch_vcpu_luo_preserve(struct kvm_vcpu *vcpu, struct kvm_vcpu_ser *ser)
+{
+	struct kvm_vcpu_arch_ser *state;
+	int num_sysregs;
+	u64 *indices;
+	size_t size;
+	int i;
+
+	num_sysregs = kvm_arm_get_sys_reg_indices(vcpu, NULL);
+	indices = kmalloc_array(num_sysregs, sizeof(u64), GFP_KERNEL);
+	if (!indices)
+		return -ENOMEM;
+	num_sysregs = kvm_arm_get_sys_reg_indices(vcpu, indices);
+
+	size = struct_size(state, sysregs, num_sysregs);
+	state = kho_alloc_preserve(size);
+	if (IS_ERR(state)) {
+		kfree(indices);
+		return PTR_ERR(state);
+	}
+
+	/* Core register state (uAPI struct kvm_regs) */
+	kvm_arm_luo_get_regs(vcpu, &state->regs);
+
+	/* Multiprocessor execution state (uAPI struct kvm_mp_state) */
+	kvm_arch_vcpu_ioctl_get_mpstate(vcpu, &state->mp_state);
+	state->pad = 0;
+
+	/* Exception / SError injection events (uAPI struct kvm_vcpu_events) */
+	__kvm_arm_vcpu_get_events(vcpu, &state->events);
+
+	/* CPU target and feature configuration (uAPI struct kvm_vcpu_init) */
+	state->init.target = KVM_ARM_TARGET_GENERIC_V8;
+	bitmap_to_arr32(state->init.features, vcpu->kvm->arch.vcpu_features,
+			KVM_VCPU_MAX_FEATURES);
+
+	/* System registers (uAPI struct kvm_one_reg array) */
+	state->num_sysregs = 0;
+	state->reserved = 0;
+	for (i = 0; i < num_sysregs; i++) {
+		u64 val;
+
+		if (kvm_arm_sys_reg_read(vcpu, indices[i], &val) == 0) {
+			state->sysregs[state->num_sysregs].id = indices[i];
+			state->sysregs[state->num_sysregs].addr = val;
+			state->num_sysregs++;
+		}
+	}
+	kfree(indices);
+
+	KHOSER_STORE_PTR(ser->arch_state, state);
+	return 0;
+}
+
+int kvm_arch_vcpu_luo_retrieve(struct kvm_vcpu *vcpu, struct kvm_vcpu_ser *ser)
+{
+	struct kvm_vcpu_arch_ser *state;
+	int i;
+
+	if (!ser->arch_state.phys)
+		return 0;
+
+	state = KHOSER_LOAD_PTR(ser->arch_state);
+
+	/* Restore vCPU features */
+	bitmap_from_arr32(vcpu->kvm->arch.vcpu_features, state->init.features,
+			  KVM_VCPU_MAX_FEATURES);
+	set_bit(KVM_ARCH_FLAG_VCPU_FEATURES_CONFIGURED, &vcpu->kvm->arch.flags);
+	kvm_reset_vcpu(vcpu);
+	vcpu_set_flag(vcpu, VCPU_INITIALIZED);
+
+	vcpu_reset_hcr(vcpu);
+
+	/* Restore core registers */
+	kvm_arm_luo_set_regs(vcpu, &state->regs);
+
+	if (irqchip_in_kernel(vcpu->kvm) &&
+	    vcpu->kvm->arch.vgic.vgic_model == KVM_DEV_TYPE_ARM_VGIC_V3) {
+		vgic_v3_reset(vcpu);
+	}
+
+	/* Restore system registers */
+	for (i = 0; i < state->num_sysregs; i++)
+		kvm_arm_sys_reg_write(vcpu, state->sysregs[i].id, state->sysregs[i].addr);
+
+	/* Restore multiprocessor execution state */
+	kvm_arch_vcpu_ioctl_set_mpstate(vcpu, &state->mp_state);
+
+	/* Restore exception / SError injection events */
+	__kvm_arm_vcpu_set_events(vcpu, &state->events);
+
+	if (irqchip_in_kernel(vcpu->kvm)) {
+		struct vgic_irq *irq;
+		unsigned long flags;
+
+		vcpu->kvm->arch.vgic.enabled = true;
+
+		irq = vgic_get_vcpu_irq(vcpu, timer_irq(vcpu_vtimer(vcpu)));
+		if (irq) {
+			raw_spin_lock_irqsave(&irq->irq_lock, flags);
+			irq->enabled = true;
+			raw_spin_unlock_irqrestore(&irq->irq_lock, flags);
+			vgic_put_irq(vcpu->kvm, irq);
+		}
+		irq = vgic_get_vcpu_irq(vcpu, timer_irq(vcpu_ptimer(vcpu)));
+		if (irq) {
+			raw_spin_lock_irqsave(&irq->irq_lock, flags);
+			irq->enabled = true;
+			raw_spin_unlock_irqrestore(&irq->irq_lock, flags);
+			vgic_put_irq(vcpu->kvm, irq);
+		}
+	}
+
+	return 0;
+}
+
+void kvm_arch_vcpu_luo_unpreserve(struct kvm_vcpu_ser *ser)
+{
+	if (ser->arch_state.phys) {
+		kho_unpreserve_free(phys_to_virt(ser->arch_state.phys));
+		ser->arch_state.phys = 0;
+	}
+}
+
+void kvm_arch_vcpu_luo_finish(struct kvm_vcpu_ser *ser)
+{
+	if (ser->arch_state.phys) {
+		kho_restore_free(phys_to_virt(ser->arch_state.phys));
+		ser->arch_state.phys = 0;
+	}
+}
diff --git a/include/linux/kho/abi/kvm_arm64.h b/include/linux/kho/abi/kvm_arm64.h
new file mode 100644
index 000000000000..dc121224a177
--- /dev/null
+++ b/include/linux/kho/abi/kvm_arm64.h
@@ -0,0 +1,61 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright (c) 2026, Google LLC.
+ * Pasha Tatashin <pasha.tatashin@soleen.com>
+ */
+#ifndef _LINUX_KHO_ABI_KVM_ARM64_H
+#define _LINUX_KHO_ABI_KVM_ARM64_H
+
+#ifdef CONFIG_ARM64
+
+#include <linux/build_bug.h>
+#include <linux/stddef.h>
+#include <linux/types.h>
+#include <linux/kho/abi/kvm.h>
+#include <uapi/linux/kvm.h>
+#include <uapi/asm/kvm.h>
+
+/**
+ * DOC: arm64 KVM vCPU Live Update ABI
+ *
+ * arm64 KVM uses the ABI defined below for preserving architectural vCPU state
+ * across a kexec reboot using LUO.
+ *
+ * The state is serialized into a packed structure `struct kvm_vcpu_arch_ser`
+ * which is handed over to the next kernel via KHO.
+ *
+ * The core register structure (struct kvm_regs) is a uAPI contract.
+ *
+ * This interface is a contract. Any modification to the structure layout
+ * constitutes a breaking change. Such changes require incrementing the version
+ * number in the KVM_VCPU_LUO_FH_COMPATIBLE string.
+ */
+
+/**
+ * struct kvm_vcpu_arch_ser - Preserved arm64 architectural vCPU state in RAM.
+ * @regs:        Core general-purpose and floating-point registers (uAPI struct kvm_regs).
+ * @mp_state:    Multiprocessor execution state (uAPI struct kvm_mp_state).
+ * @pad:         Padding to maintain 64-bit alignment after mp_state.
+ * @events:      Exception and SError injection state (uAPI struct kvm_vcpu_events).
+ * @init:        Target CPU and vCPU feature bitmap (uAPI struct kvm_vcpu_init).
+ * @num_sysregs: Number of serialized system registers in sysregs array.
+ * @reserved:    Reserved padding for 64-bit alignment.
+ * @sysregs:     Guest architectural system registers (uAPI struct kvm_one_reg array).
+ */
+struct kvm_vcpu_arch_ser {
+	struct kvm_regs regs;
+	struct kvm_mp_state mp_state;
+	u32 pad;
+	struct kvm_vcpu_events events;
+	struct kvm_vcpu_init init;
+	u32 num_sysregs;
+	u32 reserved;
+	struct kvm_one_reg sysregs[];
+} __packed;
+
+static_assert(offsetof(struct kvm_vcpu_arch_ser, sysregs) % sizeof(u64) == 0,
+	      "sysregs must be 64-bit aligned");
+
+#endif /* CONFIG_ARM64 */
+
+#endif /* _LINUX_KHO_ABI_KVM_ARM64_H */
-- 
2.55.0.1082.g2b9226bbc0-goog

[RFC PATCH 11/46] liveupdate: Define CPU preservation linker sections

[Pasha Tatashin]

Define CPU_PRESERVED_TEXT and CPU_PRESERVED_DATA linker macros to place
code and data required for CPU preservation into dedicated sections
(.text.cpu_preserved and .data.cpu_preserved).

These sections are preserved across live update transitions and mapped
executable/read-write as needed.

Signed-off-by: Pasha Tatashin <pasha.tatashin@soleen.com>
---
 include/asm-generic/vmlinux.lds.h     | 29 +++++++++++++++++
 include/linux/cpu_preserve.h          | 47 +++++++++++++++++++++++++++
 scripts/mod/modpost.c                 | 32 +++++++++++++++---
 tools/objtool/check.c                 | 35 +++++++++++++++++++-
 tools/objtool/include/objtool/check.h |  1 +
 tools/objtool/include/objtool/elf.h   |  2 +-
 6 files changed, 140 insertions(+), 6 deletions(-)
 create mode 100644 include/linux/cpu_preserve.h

diff --git a/include/asm-generic/vmlinux.lds.h b/include/asm-generic/vmlinux.lds.h
index b2988aa12f66..0dbaac582bcc 100644
--- a/include/asm-generic/vmlinux.lds.h
+++ b/include/asm-generic/vmlinux.lds.h
@@ -655,6 +655,34 @@
 		*(.static_call.text)					\
 		__static_call_text_end = .;
 
+/*
+ * Page-aligned text and data sections for preserved CPUs.
+ * This code and data are KHO preserved when CPUs are preserved across
+ * live update.
+ */
+#ifdef CONFIG_LIVEUPDATE_CPU
+#define CPU_PRESERVED_TEXT						\
+		. = ALIGN(PAGE_SIZE);					\
+		__cpu_preserved_text_start = .;				\
+		*(.text.cpu_preserved .text.cpu_preserved.*)		\
+		*(.cpu_preserved.text .cpu_preserved.text.*)		\
+		. = ALIGN(PAGE_SIZE);					\
+		__cpu_preserved_text_end = .;
+
+#define CPU_PRESERVED_DATA						\
+		. = ALIGN(PAGE_SIZE);					\
+		__cpu_preserved_data_start = .;				\
+		*(.data.cpu_preserved .data.cpu_preserved.*)		\
+		*(.cpu_preserved.data .cpu_preserved.data.*)		\
+		*(.rodata.cpu_preserved .rodata.cpu_preserved.*)	\
+		*(.bss..data.cpu_preserved .bss..data.cpu_preserved.*)	\
+		. = ALIGN(PAGE_SIZE);					\
+		__cpu_preserved_data_end = .;
+#else
+#define CPU_PRESERVED_TEXT
+#define CPU_PRESERVED_DATA
+#endif
+
 /* Section used for early init (in .S files) */
 #define HEAD_TEXT  KEEP(*(.head.text))
 
@@ -1155,6 +1183,7 @@
 		INIT_TASK_DATA(inittask)				\
 		NOSAVE_DATA						\
 		PAGE_ALIGNED_DATA(pagealigned)				\
+		CPU_PRESERVED_DATA					\
 		CACHE_HOT_DATA(cacheline)				\
 		CACHELINE_ALIGNED_DATA(cacheline)			\
 		READ_MOSTLY_DATA(cacheline)				\
diff --git a/include/linux/cpu_preserve.h b/include/linux/cpu_preserve.h
new file mode 100644
index 000000000000..f653838c383d
--- /dev/null
+++ b/include/linux/cpu_preserve.h
@@ -0,0 +1,47 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright (c) 2026, Google LLC.
+ * Pasha Tatashin <pasha.tatashin@soleen.com>
+ *
+ * Preserved CPU across Live Update
+ */
+#ifndef _LINUX_CPU_PRESERVE_H
+#define _LINUX_CPU_PRESERVE_H
+
+#include <linux/compiler.h>
+
+#ifdef CONFIG_LIVEUPDATE_CPU
+
+/*
+ * __cpu_preserved_text: Code executed by preserved physical CPUs during live
+ * update kexec handover in orphan mode.
+ *
+ * All code in this section must run without stack protector checks because
+ * per-CPU canary state may be invalid during handover and __stack_chk_fail()
+ * resides in regular .text, which gets overwritten during kexec before the
+ * incoming kernel boots.
+ *
+ * Architecture-specific requirements (such as disabling external retpolines
+ * and return thunks on x86) are supplied via ARCH_CPU_PRESERVED_TEXT.
+ */
+#ifndef ARCH_CPU_PRESERVED_TEXT
+#define ARCH_CPU_PRESERVED_TEXT
+#endif
+
+#define __cpu_preserved_text					\
+	__section(".text.cpu_preserved")			\
+	__no_stack_protector					\
+	ARCH_CPU_PRESERVED_TEXT
+#define __cpu_preserved_data	__section(".data.cpu_preserved")
+
+extern char __cpu_preserved_text_start[], __cpu_preserved_text_end[];
+extern char __cpu_preserved_data_start[], __cpu_preserved_data_end[];
+
+#else /* !CONFIG_LIVEUPDATE_CPU */
+
+#define __cpu_preserved_text
+#define __cpu_preserved_data
+
+#endif /* CONFIG_LIVEUPDATE_CPU */
+
+#endif /* _LINUX_CPU_PRESERVE_H */
diff --git a/scripts/mod/modpost.c b/scripts/mod/modpost.c
index 75374c64b8cc..51631cd2f6c4 100644
--- a/scripts/mod/modpost.c
+++ b/scripts/mod/modpost.c
@@ -810,13 +810,27 @@ static void check_section(struct module *mod, struct elf_info *elf,
 #define ALL_INIT_SECTIONS ".init.*"
 #define ALL_EXIT_SECTIONS ".exit.*"
 
+#define ALL_CPU_PRESERVED_TEXT_SECTIONS \
+	".text.cpu_preserved", ".text.cpu_preserved.*", \
+	".cpu_preserved.text", ".cpu_preserved.text.*"
+
+#define ALL_CPU_PRESERVED_DATA_SECTIONS \
+	".data.cpu_preserved", ".data.cpu_preserved.*", \
+	".cpu_preserved.data", ".cpu_preserved.data.*", \
+	".rodata.cpu_preserved", ".rodata.cpu_preserved.*", \
+	".bss..data.cpu_preserved", ".bss..data.cpu_preserved.*"
+
+#define ALL_CPU_PRESERVED_SECTIONS \
+	ALL_CPU_PRESERVED_TEXT_SECTIONS, ALL_CPU_PRESERVED_DATA_SECTIONS
+
 #define DATA_SECTIONS ".data", ".data.rel"
 #define TEXT_SECTIONS ".text", ".text.*", ".sched.text", \
 		".kprobes.text", ".cpuidle.text", ".noinstr.text", \
 		".ltext", ".ltext.*"
 #define OTHER_TEXT_SECTIONS ".ref.text", ".head.text", ".spinlock.text", \
 		".fixup", ".entry.text", ".exception.text", \
-		".coldtext", ".softirqentry.text", ".irqentry.text"
+		".coldtext", ".softirqentry.text", ".irqentry.text", \
+		ALL_CPU_PRESERVED_TEXT_SECTIONS
 
 #define ALL_TEXT_SECTIONS  ".init.text", ".exit.text", \
 		TEXT_SECTIONS, OTHER_TEXT_SECTIONS
@@ -827,6 +841,7 @@ enum mismatch {
 	ANY_INIT_TO_ANY_EXIT,
 	ANY_EXIT_TO_ANY_INIT,
 	EXTABLE_TO_NON_TEXT,
+	CPU_PRESERVED_TO_NON_PRESERVED,
 };
 
 /**
@@ -843,13 +858,19 @@ enum mismatch {
  * @mismatch: Type of mismatch.
  */
 struct sectioncheck {
-	const char *fromsec[20];
-	const char *bad_tosec[20];
-	const char *good_tosec[20];
+	const char *fromsec[32];
+	const char *bad_tosec[32];
+	const char *good_tosec[32];
 	enum mismatch mismatch;
 };
 
 static const struct sectioncheck sectioncheck[] = {
+/* Do not reference non-preserved code/data from cpu_preserved sections */
+{
+	.fromsec = { ALL_CPU_PRESERVED_SECTIONS, NULL },
+	.good_tosec = { ALL_CPU_PRESERVED_SECTIONS, NULL },
+	.mismatch = CPU_PRESERVED_TO_NON_PRESERVED,
+},
 /* Do not reference init/exit code/data from
  * normal code and data
  */
@@ -960,6 +981,9 @@ static const struct sectioncheck *section_mismatch(
 static int secref_whitelist(const char *fromsec, const char *fromsym,
 			    const char *tosec, const char *tosym)
 {
+	if (match(fromsec, PATTERNS(ALL_CPU_PRESERVED_SECTIONS)))
+		return 1;
+
 	/* Check for pattern 1 */
 	if (match(tosec, PATTERNS(ALL_INIT_DATA_SECTIONS)) &&
 	    match(fromsec, PATTERNS(DATA_SECTIONS)) &&
diff --git a/tools/objtool/check.c b/tools/objtool/check.c
index 464f6c9d9ff0..77389fbfd9ea 100644
--- a/tools/objtool/check.c
+++ b/tools/objtool/check.c
@@ -329,6 +329,8 @@ static void init_insn_state(struct objtool_file *file, struct insn_state *state,
 
 	if (opts.noinstr && sec)
 		state->noinstr = sec->noinstr;
+	if (sec)
+		state->cpu_preserved = sec->cpu_preserved;
 }
 
 static struct cfi_state *cfi_alloc(void)
@@ -419,6 +421,14 @@ static int decode_instructions(struct objtool_file *file)
 		u8 prev_len = 0;
 		u8 idx = 0;
 
+		if (!strncmp(sec->name, ".text.cpu_preserved", 19) ||
+		    !strncmp(sec->name, ".cpu_preserved.text", 19) ||
+		    !strncmp(sec->name, ".data.cpu_preserved", 19) ||
+		    !strncmp(sec->name, ".cpu_preserved.data", 19) ||
+		    !strncmp(sec->name, ".rodata.cpu_preserved", 21) ||
+		    !strncmp(sec->name, ".bss..data.cpu_preserved", 24))
+			sec->cpu_preserved = true;
+
 		if (!is_text_sec(sec))
 			continue;
 
@@ -3511,6 +3521,17 @@ static int validate_call(struct objtool_file *file,
 			 struct instruction *insn,
 			 struct insn_state *state)
 {
+	if (state->cpu_preserved) {
+		struct symbol *dest = insn_call_dest(insn);
+
+		if (dest && (dest->sec->idx != SHN_UNDEF || opts.link) &&
+		    !dest->sec->cpu_preserved) {
+			WARN_INSN(insn, "call to %s() leaves .text.cpu_preserved section",
+				  call_dest_name(insn));
+			return 1;
+		}
+	}
+
 	if (state->noinstr && state->instr <= 0 &&
 	    !noinstr_call_dest(file, insn, insn_call_dest(insn))) {
 		WARN_INSN(insn, "call to %s() leaves .noinstr.text section", call_dest_name(insn));
@@ -4164,7 +4185,13 @@ static int validate_retpoline(struct objtool_file *file)
 		if (insn->retpoline_safe)
 			continue;
 
-		if (insn->sec->init)
+		/*
+		 * Preserved CPU text (.text.cpu_preserved) executes across
+		 * kexec when the outgoing kernel's retpoline/rethunk targets
+		 * are no longer mapped.
+		 */
+		if (insn->sec->init ||
+		    !strcmp(insn->sec->name, ".text.cpu_preserved"))
 			continue;
 
 		if (insn->type == INSN_RETURN) {
@@ -4440,6 +4467,12 @@ static int validate_noinstr_sections(struct objtool_file *file)
 		warnings += validate_unwind_hints(file, sec);
 	}
 
+	sec = find_section_by_name(file->elf, ".text.cpu_preserved");
+	if (sec) {
+		warnings += validate_section(file, sec);
+		warnings += validate_unwind_hints(file, sec);
+	}
+
 	return warnings;
 }
 
diff --git a/tools/objtool/include/objtool/check.h b/tools/objtool/include/objtool/check.h
index 063f5985fecd..18318f2faf11 100644
--- a/tools/objtool/include/objtool/check.h
+++ b/tools/objtool/include/objtool/check.h
@@ -16,6 +16,7 @@ struct insn_state {
 	bool uaccess;
 	bool df;
 	bool noinstr;
+	bool cpu_preserved;
 	s8 instr;
 };
 
diff --git a/tools/objtool/include/objtool/elf.h b/tools/objtool/include/objtool/elf.h
index a82517a76a0f..bac9e5f4921f 100644
--- a/tools/objtool/include/objtool/elf.h
+++ b/tools/objtool/include/objtool/elf.h
@@ -58,7 +58,7 @@ struct section {
 	Elf_Data *data;
 	const char *name;
 	int idx;
-	bool _changed, text, rodata, noinstr, init, truncate;
+	bool _changed, text, rodata, noinstr, init, cpu_preserved, truncate;
 	struct reloc *relocs;
 	unsigned long nr_alloc_relocs;
 	struct section *twin;
-- 
2.55.0.1082.g2b9226bbc0-goog

[RFC PATCH 12/46] liveupdate: Add liveupdate_session_name() helper

[Pasha Tatashin]

Provide an accessor function liveupdate_session_name() to retrieve the
name of a live update session from its session handle.

This allows kernel subsystems like CPU preservation to identify the
session managing preserved resources.

Signed-off-by: Pasha Tatashin <pasha.tatashin@soleen.com>
---
 include/linux/liveupdate.h      |  6 ++++++
 kernel/liveupdate/luo_session.c | 14 ++++++++++++++
 2 files changed, 20 insertions(+)

diff --git a/include/linux/liveupdate.h b/include/linux/liveupdate.h
index 6051abc0612c..119ad0f82705 100644
--- a/include/linux/liveupdate.h
+++ b/include/linux/liveupdate.h
@@ -256,6 +256,7 @@ int liveupdate_get_file_incoming(struct liveupdate_session *s, u64 token,
 /* Get a token for an outgoing file, or -ENOENT if file is not preserved */
 int liveupdate_get_token_outgoing(struct liveupdate_session *s,
 				  struct file *file, u64 *tokenp);
+const char *liveupdate_session_name(struct liveupdate_session *s);
 
 #else /* CONFIG_LIVEUPDATE */
 
@@ -321,5 +322,10 @@ static inline int liveupdate_get_token_outgoing(struct liveupdate_session *s,
 	return -EOPNOTSUPP;
 }
 
+static inline const char *liveupdate_session_name(struct liveupdate_session *s)
+{
+	return NULL;
+}
+
 #endif /* CONFIG_LIVEUPDATE */
 #endif /* _LINUX_LIVEUPDATE_H */
diff --git a/kernel/liveupdate/luo_session.c b/kernel/liveupdate/luo_session.c
index f48e9a4185f9..3c4f195d375d 100644
--- a/kernel/liveupdate/luo_session.c
+++ b/kernel/liveupdate/luo_session.c
@@ -683,3 +683,17 @@ int luo_session_serialize(void)
 
 	return err;
 }
+
+/**
+ * liveupdate_session_name - Get the name of a liveupdate session
+ * @s: Live update session pointer.
+ *
+ * Return: Pointer to session name string.
+ */
+const char *liveupdate_session_name(struct liveupdate_session *s)
+{
+	struct luo_session *session = (struct luo_session *)s;
+
+	return session->name;
+}
+EXPORT_SYMBOL_GPL(liveupdate_session_name);
-- 
2.55.0.1082.g2b9226bbc0-goog

[RFC PATCH 13/46] cpu_preserve: Add physical CPU preservation ABI and core API headers

[Pasha Tatashin]

Define the KHO ABI structures for physical CPU preservation in
include/linux/kho/abi/cpu.h and the core kernel interfaces in
include/linux/cpu_preserve.h.

Signed-off-by: Pasha Tatashin <pasha.tatashin@soleen.com>
---
 include/linux/cpu_preserve.h | 352 +++++++++++++++++++++++++++++++++++
 include/linux/kho/abi/cpu.h  | 112 +++++++++++
 2 files changed, 464 insertions(+)
 create mode 100644 include/linux/kho/abi/cpu.h

diff --git a/include/linux/cpu_preserve.h b/include/linux/cpu_preserve.h
index f653838c383d..94181952df1a 100644
--- a/include/linux/cpu_preserve.h
+++ b/include/linux/cpu_preserve.h
@@ -9,9 +9,75 @@
 #define _LINUX_CPU_PRESERVE_H
 
 #include <linux/compiler.h>
+#include <linux/cpumask.h>
+#include <linux/errno.h>
+#include <linux/kho/abi/cpu.h>
+#include <linux/list.h>
+#include <linux/smp.h>
+#include <linux/types.h>
 
 #ifdef CONFIG_LIVEUPDATE_CPU
 
+#include <asm/cpu_preserve.h>
+#include <asm/page.h>
+
+#define CPU_PRESERVED_STACK_ORDER	ARCH_CPU_PRESERVED_STACK_ORDER
+#define CPU_PRESERVED_STACK_SIZE	((size_t)PAGE_SIZE << CPU_PRESERVED_STACK_ORDER)
+#define CPU_PRESERVED_STACK_HEADROOM	256
+#define CPU_PRESERVED_STACK_MAGIC	0x435055505354414bULL	/* "CPUPSTAK" */
+
+/**
+ * struct cpu_preserved_stack_context - Context header at base of preserved CPU stack
+ * @magic:            Validation signature (%CPU_PRESERVED_STACK_MAGIC).
+ * @cpu:              Logical CPU identifier of the preserved physical core.
+ * @reserved:         Must be zero.
+ * @workload_context: Opaque owning workload or session context.
+ * @session_pgd_pa:   Session root page table physical address, or 0.
+ *
+ * This structure lives at the base of a preserved CPU's dedicated stack and is
+ * accessed by the preserved CPU during parking and workload execution. It is
+ * private to the preserved CPU execution context of the kernel that allocated it.
+ */
+struct cpu_preserved_stack_context {
+	u64 magic;
+	u32 cpu;
+	u32 reserved;
+	u64 workload_context;
+	u64 session_pgd_pa;
+};
+
+static_assert(sizeof(struct cpu_preserved_stack_context) == 32);
+static_assert(offsetof(struct cpu_preserved_stack_context, magic) == 0,
+	      "magic must lead: the struct is found by masking the stack pointer");
+
+/**
+ * cpu_preserved_get_stack_context - Return the context block at the base of the preserved stack
+ *
+ * Masks the current stack pointer to %CPU_PRESERVED_STACK_SIZE alignment and
+ * validates %CPU_PRESERVED_STACK_MAGIC.
+ *
+ * Return: Pointer to &struct cpu_preserved_stack_context if executing on a
+ *         preserved CPU stack, or %NULL otherwise.
+ */
+static inline struct cpu_preserved_stack_context *
+cpu_preserved_get_stack_context(void)
+{
+	struct cpu_preserved_stack_context *sctx;
+	unsigned long sp;
+
+#if defined(CONFIG_X86_64)
+	asm volatile("mov %%rsp, %0" : "=r"(sp));
+#elif defined(CONFIG_ARM64)
+	asm volatile("mov %0, sp" : "=r"(sp));
+#else
+	return NULL;
+#endif
+	sctx = (struct cpu_preserved_stack_context *)(sp & ~(CPU_PRESERVED_STACK_SIZE - 1));
+	if (sctx && sctx->magic == CPU_PRESERVED_STACK_MAGIC)
+		return sctx;
+	return NULL;
+}
+
 /*
  * __cpu_preserved_text: Code executed by preserved physical CPUs during live
  * update kexec handover in orphan mode.
@@ -36,12 +102,298 @@
 
 extern char __cpu_preserved_text_start[], __cpu_preserved_text_end[];
 extern char __cpu_preserved_data_start[], __cpu_preserved_data_end[];
+bool cpu_is_preserved(int cpu);
+bool cpu_preserved_should_exit(int cpu);
+void cpu_preserved_set_dead(int cpu);
+void cpu_preserved_park(int cpu);
+void cpu_preserved_park_loop(int cpu);
+const struct cpumask *cpu_get_preserved_mask(void);
+phys_addr_t cpu_preserved_get_pgd(int cpu);
+int cpu_preserved_attach_workload(int cpu,
+				  void (*entry_fn)(void *data), void *data);
+int cpu_preserved_detach_workload(int cpu);
+void cpu_preserved_set_workload_context(int cpu, void *ctx, phys_addr_t pgd_pa);
+
+/**
+ * cpu_preserved_report_dead - Park preserved CPU when reporting dead in hotplug
+ *
+ * Invoked by cpuhp_ap_report_dead() after CPU hotplug offline synchronization
+ * is complete. If the calling CPU is marked for preservation across live update,
+ * transition it into the preserved parking loop instead of powering down.
+ */
+static inline void cpu_preserved_report_dead(void)
+{
+	if (cpu_is_preserved(raw_smp_processor_id()))
+		cpu_preserved_park(raw_smp_processor_id());
+}
+
+/*
+ * Architecture-specific hooks for CPU preservation.
+ */
+
+/**
+ * arch_cpu_preserved_kick - Signal or wake up a preserved physical CPU
+ * @cpu: Logical CPU identifier.
+ *
+ * Architecture backend hook to wake up the specified preserved CPU from its
+ * low-power parking state (e.g. via IPI, NMI, or SGI).
+ *
+ * This function must be placed in the __cpu_preserved_text section.
+ */
+void arch_cpu_preserved_kick(int cpu);
+
+/**
+ * arch_cpu_preserved_park_wait - Architecture low-power wait in parking loop
+ *
+ * Architecture backend hook to execute a low-power wait instruction
+ * (e.g., cpu_relax/pause, wfe) while parked.
+ *
+ * This function must be placed in the __cpu_preserved_text section.
+ */
+void arch_cpu_preserved_park_wait(void);
+
+/**
+ * arch_cpu_preserved_park_init - Architecture setup upon entering park loop
+ * @cpu: Logical CPU identifier.
+ *
+ * Architecture backend hook to configure the physical core (e.g., disable
+ * or mask local interrupts) upon entering the park loop.
+ *
+ * This function must be placed in the __cpu_preserved_text section.
+ */
+void arch_cpu_preserved_park_init(int cpu);
+
+/**
+ * arch_cpu_preserved_early_init - Arch early-boot init for incoming preserved CPUs
+ *
+ * Called during early boot in the incoming kernel when preserved physical CPUs
+ * are adopted from KHO metadata.
+ */
+void arch_cpu_preserved_early_init(void);
+
+/**
+ * arch_cpu_preserved_park_finish - Architecture cleanup on park loop exit
+ * @cpu: Logical CPU identifier.
+ *
+ * Architecture backend hook to execute cleanup or CPU powerdown sequence
+ * when the park loop exits.
+ *
+ * This function must be placed in the __cpu_preserved_text section.
+ */
+void arch_cpu_preserved_park_finish(int cpu);
+
+/**
+ * arch_cpu_preserved_park_on_stack - Switch stack and enter park loop
+ * @cpu: Logical CPU identifier.
+ * @stack_top: Top address of the preserved stack.
+ *
+ * Architecture backend hook to switch to the preserved execution stack
+ * and invoke cpu_preserved_park_loop().
+ *
+ * This function must be placed in the __cpu_preserved_text section.
+ */
+void arch_cpu_preserved_park_on_stack(int cpu, unsigned long stack_top);
+
+/**
+ * arch_cpu_preserved_dcache_clean - Clean data cache for address range
+ * @start: Starting virtual address.
+ * @end: Ending virtual address.
+ *
+ * Architecture backend hook to flush/clean data caches to PoC for memory
+ * preservation across live update.
+ *
+ * This function must be placed in the __cpu_preserved_text section.
+ */
+void arch_cpu_preserved_dcache_clean(unsigned long start, unsigned long end);
+
+/**
+ * arch_cpu_preserved_dcache_inval - Invalidate/clean data cache for range
+ * @start: Starting virtual address.
+ * @end: Ending virtual address.
+ *
+ * Architecture backend hook to clean/invalidate data caches across live
+ * update transitions.
+ *
+ * This function must be placed in the __cpu_preserved_text section.
+ */
+void arch_cpu_preserved_dcache_inval(unsigned long start, unsigned long end);
+
+/**
+ * arch_cpu_preserved_wait_dead - Wait for CPU to reach dead state
+ * @cpu: Logical CPU identifier.
+ *
+ * Architecture backend hook to wait for a CPU to be fully stopped.
+ *
+ * Executed in normal text context during CPU teardown.
+ */
+void arch_cpu_preserved_wait_dead(int cpu);
+
+struct page;
+
+/**
+ * arch_cpu_preserved_setup_buffer - Map preserved execution buffer outside Scratch
+ * @text_page: Head page of allocated preserved text memory.
+ * @text_nr_pages: Number of pages in the preserved text buffer.
+ * @data_page: Head page of allocated preserved data memory.
+ * @data_nr_pages: Number of pages in the preserved data buffer.
+ *
+ * Architecture backend hook to remap kernel page table entries for
+ * __cpu_preserved_text and __cpu_preserved_data to the newly allocated
+ * pages outside Scratch memory.
+ *
+ * Return: 0 on success, or a negative errno on failure.
+ */
+int arch_cpu_preserved_setup_buffer(struct page *text_page,
+				    unsigned int text_nr_pages,
+				    struct page *data_page,
+				    unsigned int data_nr_pages);
+
+/**
+ * struct cpu_preserved_as - In-memory address space handle for a preserved CPU
+ * @node:        Entry on the global list of preserved address spaces.
+ * @pgd:         Root page table virtual address in host direct map.
+ * @pgd_pa:      Physical address of @pgd, as loaded into CR3 / TTBR1.
+ * @is_incoming: This address space was built by the previous kernel.
+ * @ser:         Preserved serialization descriptor holding page table physical addresses.
+ */
+struct cpu_preserved_as {
+	struct list_head		node;
+	void				*pgd;
+	phys_addr_t			pgd_pa;
+	bool				is_incoming;
+	struct cpu_preserved_as_ser	*ser;
+};
+
+struct cpu_preserved_as *cpu_preserved_as_create(void);
+void cpu_preserved_as_destroy(struct cpu_preserved_as *as);
+struct cpu_preserved_as *cpu_preserved_as_adopt(struct cpu_preserved_as_ser *ser);
+int cpu_preserved_as_map(struct cpu_preserved_as *as, phys_addr_t pa,
+			 unsigned long va, size_t size, pgprot_t prot);
 
 #else /* !CONFIG_LIVEUPDATE_CPU */
 
 #define __cpu_preserved_text
 #define __cpu_preserved_data
 
+static inline bool cpu_is_preserved(int cpu) { return false; }
+static inline bool cpu_preserved_should_exit(int cpu) { return true; }
+static inline void cpu_preserved_park(int cpu) {}
+static inline void cpu_preserved_set_dead(int cpu) {}
+static inline void cpu_preserved_report_dead(void) {}
+static inline const struct cpumask *cpu_get_preserved_mask(void)
+{
+	return cpu_none_mask;
+}
+
+static inline int cpu_preserved_attach_workload(int cpu,
+						void (*entry_fn)(void *data),
+						void *data)
+{
+	return -EOPNOTSUPP;
+}
+
+static inline int cpu_preserved_detach_workload(int cpu)
+{
+	return -EOPNOTSUPP;
+}
+
+static inline void cpu_preserved_set_workload_context(int cpu, void *ctx,
+						      phys_addr_t pgd_pa) {}
+static inline void arch_cpu_preserved_kick(int cpu) {}
+static inline void arch_cpu_preserved_park_wait(void) {}
+static inline void arch_cpu_preserved_park_init(int cpu) {}
+static inline void arch_cpu_preserved_early_init(void) {}
+static inline void arch_cpu_preserved_park_finish(int cpu) {}
+static inline void arch_cpu_preserved_dcache_clean(unsigned long start,
+						   unsigned long end) {}
+static inline void arch_cpu_preserved_dcache_inval(unsigned long start,
+						   unsigned long end) {}
+static inline void arch_cpu_preserved_wait_dead(int cpu) {}
+static inline phys_addr_t cpu_preserved_get_pgd(int cpu) { return 0; }
+static inline int arch_cpu_preserved_setup_buffer(struct page *text_page,
+						  unsigned int text_nr_pages,
+						  struct page *data_page,
+						  unsigned int data_nr_pages)
+{
+	return 0;
+}
+
+static inline int cpu_preserved_map_range(phys_addr_t pa, unsigned long va,
+					  size_t size, pgprot_t prot) { return 0; }
+static inline int cpu_preserved_map_buffer(void *va, size_t size) { return 0; }
+static inline int arch_cpu_preserved_mpidr_to_cpu(u64 mpidr) { return -EINVAL; }
+static inline bool arch_cpu_preserved_is_active(void) { return false; }
+static inline void arch_cpu_preserved_switch_pgd(phys_addr_t pgd_pa) {}
+static inline struct cpu_preserved_stack_context *
+cpu_preserved_get_stack_context(void)
+{
+	return NULL;
+}
+
 #endif /* CONFIG_LIVEUPDATE_CPU */
 
+/*
+ * Object-granular wrappers around the arch dcache hooks.
+ *
+ * Every preserved-memory handshake flushes or invalidates a whole object, so
+ * spell that out once instead of open-coding (addr, addr + size) at each call
+ * site: the size can then never drift from the object it is supposed to cover.
+ *
+ * @p is a pointer to the object.  For a statically sized array, pass &array so
+ * that sizeof(*(p)) is the size of the whole array rather than of one element.
+ * Use the _sz() forms for flexible-array structures and for raw page buffers,
+ * where the length is not derivable from the type.
+ */
+#define cpu_preserved_clean_sz(p, sz)					\
+	arch_cpu_preserved_dcache_clean((unsigned long)(p),		\
+					(unsigned long)(p) + (sz))
+#define cpu_preserved_inval_sz(p, sz)					\
+	arch_cpu_preserved_dcache_inval((unsigned long)(p),		\
+					(unsigned long)(p) + (sz))
+#define cpu_preserved_clean(p)		cpu_preserved_clean_sz(p, sizeof(*(p)))
+#define cpu_preserved_inval(p)		cpu_preserved_inval_sz(p, sizeof(*(p)))
+
+/**
+ * cpu_preserved_memset - Compiler-barrier-safe byte fill for preserved runtime code
+ * @s: Destination buffer.
+ * @c: Byte value to fill.
+ * @n: Number of bytes to fill.
+ *
+ * Uses WRITE_ONCE() per byte so the compiler's loop-idiom pass cannot replace
+ * the loop with a call to the unmapped host kernel memset() inside
+ * __cpu_preserved_text.
+ *
+ * Return: @s.
+ */
+static inline void *cpu_preserved_memset(void *s, int c, size_t n)
+{
+	unsigned char *p = s;
+
+	while (n--)
+		WRITE_ONCE(*p++, (unsigned char)c);
+	return s;
+}
+
+/**
+ * cpu_preserved_memcpy - Compiler-barrier-safe byte copy for preserved runtime code
+ * @dest: Destination buffer.
+ * @src:  Source buffer.
+ * @n:    Number of bytes to copy.
+ *
+ * Uses READ_ONCE()/WRITE_ONCE() per byte so the compiler's loop-idiom pass
+ * cannot replace the loop with a call to the unmapped host kernel memcpy()
+ * inside __cpu_preserved_text.
+ *
+ * Return: @dest.
+ */
+static inline void *cpu_preserved_memcpy(void *dest, const void *src, size_t n)
+{
+	const unsigned char *s = src;
+	unsigned char *d = dest;
+
+	while (n--)
+		WRITE_ONCE(*d++, READ_ONCE(*s++));
+	return dest;
+}
+
 #endif /* _LINUX_CPU_PRESERVE_H */
diff --git a/include/linux/kho/abi/cpu.h b/include/linux/kho/abi/cpu.h
new file mode 100644
index 000000000000..5926be37f7a0
--- /dev/null
+++ b/include/linux/kho/abi/cpu.h
@@ -0,0 +1,112 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright (c) 2026, Google LLC.
+ * Pasha Tatashin <pasha.tatashin@soleen.com>
+ */
+
+#ifndef _LINUX_KHO_ABI_CPU_H
+#define _LINUX_KHO_ABI_CPU_H
+
+#include <linux/build_bug.h>
+#include <linux/kho/abi/kexec_handover.h>
+#include <linux/stddef.h>
+#include <linux/types.h>
+#include <uapi/linux/liveupdate.h>
+
+/**
+ * DOC: CPU Preservation Live Update ABI
+ *
+ * Physical CPU preservation uses the ABI defined below to serialize
+ * and restore the state of preserved CPUs across a live update kexec
+ * reboot using the LUO.
+ *
+ * Preserved CPUs are isolated from host scheduling and remain active
+ * in a parking loop or running workload across the live update reboot.
+ * This ABI provides the contract for communicating preserved core metadata
+ * and per-file descriptor state to the incoming kernel.
+ *
+ * The state is serialized into packed structures
+ * (struct cpu_preserved_global_ser and struct cpu_preserved_file_ser) which
+ * are handed over to the next kernel via the KHO mechanism.
+ *
+ * This interface is a contract. Any modification to the structure
+ * fields, compatible strings, or the layout of the `__packed`
+ * serialization structures defined here constitutes a breaking change.
+ * Such changes require incrementing the version number in the
+ * CPU_PRESERVED_LUO_FLB_COMPATIBLE or CPU_PRESERVED_LUO_FH_COMPATIBLE
+ * compatibility strings to prevent a new kernel from misinterpreting
+ * data from an old kernel.
+ *
+ * Changes are allowed provided the compatibility version is
+ * incremented; however, backward/forward compatibility is only
+ * guaranteed for kernels supporting the same ABI version.
+ */
+
+/* The compatibility string for preserved CPU FLB */
+#define CPU_PRESERVED_LUO_FLB_COMPATIBLE	"cpu_flb_v1"
+
+/* The compatibility string for preserved CPU file handler */
+#define CPU_PRESERVED_LUO_FH_COMPATIBLE		"cpu_fh_v1"
+
+enum cpu_preserved_workload {
+	CPU_PRESERVED_PARKED = 1,
+	CPU_PRESERVED_WORKLOAD = 2,
+	CPU_PRESERVED_EXITING = 3,
+	CPU_PRESERVED_DEAD = 4,
+};
+
+/**
+ * struct cpu_preserved_pcpu_ser - Per-CPU mailbox in preserved memory
+ * @workload: Preserved workload state (enum cpu_preserved_workload).
+ */
+struct cpu_preserved_pcpu_ser {
+	u32 workload;
+} __packed;
+
+/**
+ * struct cpu_preserved_global_ser - Global FLB serialization header
+ * @nr_cpu_words:       Number of u64 words in @cpu_preserved_bitmap.
+ * @reserved:           Must be zero.
+ * @text_runtime_pa:    Physical address of preserved CPU runtime text.
+ * @text_runtime_size:  Size of preserved CPU runtime text in bytes.
+ * @data_runtime_pa:    Physical address of preserved CPU runtime data.
+ * @data_runtime_size:  Size of preserved CPU runtime data in bytes.
+ * @pcpus_runtime:      Preservation pointer to struct cpu_preserved_pcpu_ser array.
+ * @transition_as:      Preservation pointer to transition struct cpu_preserved_as_ser.
+ * @cpu_preserved_bitmap: Bitmap of physical CPUs preserved across live update.
+ *
+ * The preserved-CPU bitmap is serialized as an explicitly sized array of u64
+ * rather than as a cpumask_t: sizeof(cpumask_t) depends on NR_CPUS, so
+ * embedding one would make the offset of every subsequent field depend on the
+ * .config of the kernel that wrote it.
+ */
+struct cpu_preserved_global_ser {
+	u32 nr_cpu_words;
+	u32 reserved;
+	u64 text_runtime_pa;
+	u64 text_runtime_size;
+	u64 data_runtime_pa;
+	u64 data_runtime_size;
+	DECLARE_KHOSER_PTR(pcpus_runtime, struct cpu_preserved_pcpu_ser *);
+	DECLARE_KHOSER_PTR(transition_as, struct cpu_preserved_as_ser *);
+	u64 cpu_preserved_bitmap[];
+} __packed;
+
+static_assert(offsetof(struct cpu_preserved_global_ser,
+		       cpu_preserved_bitmap) % sizeof(u64) == 0,
+	      "cpu_preserved_bitmap must be 64-bit aligned");
+
+/**
+ * struct cpu_preserved_file_ser - Per-file serialized state for preserved CPU fd
+ * @cpu:      Logical CPU identifier.
+ * @reserved: Must be zero.
+ * @stack_pa: Physical address of this CPU's preserved stack.
+ * @oncore:   Preservation pointer to on-core session serialized metadata.
+ */
+struct cpu_preserved_file_ser {
+	u32 cpu;
+	u32 reserved;
+	u64 stack_pa;
+} __packed;
+
+#endif /* _LINUX_KHO_ABI_CPU_H */
-- 
2.55.0.1082.g2b9226bbc0-goog

[RFC PATCH 14/46] cpu_preserve: Add core physical CPU preservation state and park loop

[Pasha Tatashin]

Introduce the physical CPU preservation state tracking, cache maintenance
wrappers, and standalone park execution loop in
kernel/liveupdate/cpu_preserve.c.

Signed-off-by: Pasha Tatashin <pasha.tatashin@soleen.com>
---
 kernel/liveupdate/cpu_preserve.c | 249 +++++++++++++++++++++++++++++++
 1 file changed, 249 insertions(+)
 create mode 100644 kernel/liveupdate/cpu_preserve.c

diff --git a/kernel/liveupdate/cpu_preserve.c b/kernel/liveupdate/cpu_preserve.c
new file mode 100644
index 000000000000..e19208670c3e
--- /dev/null
+++ b/kernel/liveupdate/cpu_preserve.c
@@ -0,0 +1,249 @@
+// SPDX-License-Identifier: GPL-2.0
+
+/*
+ * Copyright (c) 2026, Google LLC.
+ * Pasha Tatashin <pasha.tatashin@soleen.com>
+ */
+
+/**
+ * DOC: Preserved CPU Subsystem
+ *
+ * Live Update allows updating the host kernel while preserving the state of
+ * hardware resources across the transition. While memfd-based memory
+ * preservation is supported via LUO and PCI device preservation is handled
+ * by VFIO and IOMMU, physical CPU cores represent another fundamental class
+ * of hardware resource that requires preservation.
+ *
+ * A primary motivation is preserving virtual machine (VM) workloads across
+ * host kernel updates without pausing the guest. By separating a physical
+ * core from standard host scheduling and keeping it active across the kexec
+ * reboot, guest vCPUs or dedicated bare-metal tasks can continue
+ * uninterrupted execution on-core.
+ *
+ * This subsystem provides the generic, hypervisor-agnostic foundation for
+ * physical CPU preservation.
+ *
+ * Lifecycle
+ * =========
+ *
+ * CPU lifecycle state progression::
+ *
+ *     +-------------------------------------------------------------+
+ *     |                          ONLINE                             |
+ *     |               (Normal host task scheduling)                 |
+ *     +-------------------------------------------------------------+
+ *                                    |
+ *                                    | preserve (via LUO fd)
+ *                                    v
+ *     +-------------------------------------------------------------+
+ *     |                     PRESERVED_PARKED                        |
+ *     |          (Removed from scheduler, loops in park)            |
+ *     +-------------------------------------------------------------+
+ *                                    |
+ *                                    | [Live Update: kexec]
+ *                                    v
+ *     +-------------------------------------------------------------+
+ *     |                     INCOMING PRESERVED                      |
+ *     |         (Parked on-core, skipped in secondary boot)         |
+ *     |       (State restored upon session retrieve; stays running) |
+ *     +-------------------------------------------------------------+
+ *                                    |
+ *                                    | unpreserve / finish (via LUO session)
+ *                                    v
+ *     +-------------------------------------------------------------+
+ *     |                          OFFLINE                            |
+ *     |            (Park loop exited, architecturally idle)         |
+ *     +-------------------------------------------------------------+
+ *                                    |
+ *                                    | automatic add_cpu()
+ *                                    v
+ *     +-------------------------------------------------------------+
+ *     |                          ONLINE                             |
+ *     |                (Rejoined host scheduling)                   |
+ *     +-------------------------------------------------------------+
+ *
+ * File Descriptor Binding
+ * =======================
+ *
+ * 1. **Sysfs control file:** Each hotpluggable CPU exports a read-only sysfs
+ *    attribute at ``/sys/devices/system/cpu/cpu<N>/preserve``. The file
+ *    descriptor of this file handles the lifecycle of the preserved CPU.
+ *
+ * 2. **Preservation via LUO:** Userspace opens this file and registers the fd
+ *    with LUO. Preserving the file offlines the core from host scheduling,
+ *    migrates its interrupts and tasks, and transitions the CPU from online
+ *    into the parked state (cpu_preserved_park()). Preservation integrates with
+ *    the On-Core framework (oncore_session_add_cpu()).
+ *
+ * 3. **KHO and memory preservation:** The parking loop, dedicated preserved
+ *    CPU stacks, runtime execution buffers outside Scratch memory, and
+ *    preserved CPU state reside in memory preserved across kexec via KHO.
+ *
+ * 4. **Incoming boot:** During early boot, the incoming kernel restores the
+ *    preserved CPU mask from the KHO FLB before secondary SMP bringup and
+ *    skips bringing preserved cores online, maintaining isolation.
+ *
+ * 5. **Retrieval and unpreservation:** When userspace retrieves the session in
+ *    the incoming kernel, it receives the open ``preserve`` file descriptor.
+ *    Retrieving the session reconnects the descriptors and restores on-core session
+ *    state while keeping the core running. Finalizing the session (``finish``)
+ *    or closing the fd unpreserves the CPU, signaling the core to exit the
+ *    parking loop and automatically restoring it online via add_cpu().
+ *
+ * Architecture Requirements
+ * =========================
+ *
+ * In addition to CPU hotplug (``CONFIG_HOTPLUG_CPU``), an architecture
+ * selecting ``ARCH_SUPPORTS_LIVEUPDATE_CPU`` must provide:
+ *
+ * - **Linker script:** Include ``CPU_PRESERVED_TEXT`` in
+ *   ``arch/<arch>/kernel/vmlinux.lds.S`` within the executable text section.
+ *
+ * - **Preserved text section:** Functions executed by a parked core or during
+ *   live update transitions must be annotated with ``__cpu_preserved_text`` so
+ *   their instructions reside in the KHO-preserved ``.text.cpu_preserved``
+ *   section. These are the ``arch_cpu_preserved_*()`` hooks documented in
+ *   ``include/linux/cpu_preserve.h``.
+ *
+ * - **Address-space mapping hooks:** arch_cpu_preserved_as_map(),
+ *   arch_cpu_preserved_as_flush_tlb(), and
+ *   arch_cpu_preserved_set_transition_as() populate and manage isolated page
+ *   tables built by the core layer using cpu_preserved_as_alloc_page().
+ *
+ * - **Buffer relocation hook:** arch_cpu_preserved_setup_buffer() relocates
+ *   preserved text and data sections outside KHO Scratch memory so the
+ *   incoming kernel can unpack safely.
+ *
+ * - **CPU hotplug and stop-IPI isolation:** Exclude preserved CPUs from stop
+ *   signals (NMI or stop IPIs in the machine reboot and crash paths), and
+ *   avoid tearing down local interrupt controllers (LAPIC, GIC CPU interface)
+ *   during CPU disable when the core is being preserved.
+ *
+ * Isolated Address Space
+ * ======================
+ *
+ * A preserved core does not run on the kernel's own page tables. Before it is
+ * handed over, the core layer builds a transition page table
+ * (cpu_preserved_as_create()) containing only what on-core execution needs,
+ * so that a core still running a workload cannot touch memory the new kernel
+ * has taken ownership of:
+ *
+ * - Preserved text and read-only data, ``PAGE_KERNEL_ROX``
+ *   (``__cpu_preserved_text``) -- park loops, world-switch routines, ops
+ *   vector tables, and exception stubs;
+ * - Preserved writable globals, ``PAGE_KERNEL`` NX
+ *   (``__cpu_preserved_data``) -- state machines, session descriptors,
+ *   per-CPU control blocks, and the preserved-CPU masks;
+ * - The per-CPU dedicated preserved stack, ``PAGE_KERNEL`` NX;
+ * - The KHO-preserved workload state pages, ``PAGE_KERNEL`` NX;
+ * - Hardware control MMIO, ``PAGE_KERNEL_IO``, only where the interrupt
+ *   controller still requires it (e.g., GICv3 in system-register mode needs
+ *   none).
+ *
+ * Deliberately absent: the linear direct map, all user address ranges, the
+ * kernel heap, vmalloc, modules, and BPF JIT. Guest memory is not mapped
+ * either -- it is reached through stage-2 translation.
+ *
+ * On arm64 these mappings are built with trans_pgd_map_range(), on x86 with
+ * the identity-map helpers in ``arch/x86/mm/ident_map.c``. Custom workload
+ * address spaces can also be created and adopted across kexec via
+ * cpu_preserved_as_adopt().
+ *
+ * Workload Integration
+ * ====================
+ *
+ * Physical cores preserved across live update execute payloads managed by the
+ * On-Core execution framework. CPU preservation integrates directly with On-Core
+ * session lifecycle:
+ *
+ * - On-Core assigns jobs to preserved cores via cpu_preserved_attach_workload().
+ * - When a CPU file is preserved or unpreserved, oncore_session_add_cpu() and
+ *   oncore_session_remove_cpu() update the session CPU bitmap.
+ * - At kexec handover, oncore_session_get_ser() serializes the session state into
+ *   the preserved CPU file descriptor, and oncore_session_restore() reconstructs
+ *   the session in the incoming kernel.
+ */
+
+#define pr_fmt(fmt) "cpu_preserve: " fmt
+
+#include <linux/cpu.h>
+#include <linux/cpu_preserve.h>
+#include <linux/delay.h>
+#include <linux/device.h>
+#include <linux/device/bus.h>
+#include <linux/kexec.h>
+#include <linux/kexec_handover.h>
+#include <linux/kho/abi/cpu.h>
+#include <linux/kho_block.h>
+#include <linux/liveupdate.h>
+#include <linux/mm.h>
+#include <linux/objtool.h>
+#include <linux/reboot.h>
+
+#include <asm/sections.h>
+
+/**
+ * struct cpu_preserved_pcpu - Per-CPU host runtime state for CPU preservation
+ * @stack_pa: Physical address of the preserved stack for this CPU.
+ * @pgd_pa: Page table root PA for the preserved CPU context.
+ * @entry_fn: Workload callback function executed repeatedly on the physical
+ *            core while parked in cpu_preserved_park().
+ * @entry_data: Opaque argument passed to @entry_fn.
+ *
+ * Tracks host runtime state for a preserved physical core. Allocated locally
+ * in host memory; not preserved across kexec handover.
+ */
+struct cpu_preserved_pcpu {
+	phys_addr_t stack_pa;
+	phys_addr_t pgd_pa;
+	void (*entry_fn)(void *data) ____cacheline_aligned;
+	void *entry_data;
+};
+
+/*
+ * struct cpu_preserved_state - Host-side preserved CPU state (incoming or outgoing)
+ * @mask: Mask of preserved CPUs.
+ * @pcpus: Host runtime state array.
+ * @pcpus_ser: Per-CPU mailbox array in preserved memory.
+ */
+struct cpu_preserved_state {
+	cpumask_t mask;
+	struct cpu_preserved_pcpu *pcpus;
+	struct cpu_preserved_pcpu_ser *pcpus_ser;
+};
+
+static DEFINE_MUTEX(cpu_preserved_lock);
+static struct cpu_preserved_state cpu_preserved_incoming;
+static struct cpu_preserved_state cpu_preserved_outgoing;
+static cpumask_t cpu_preserved_mask __cpu_preserved_data;
+static struct cpu_preserved_pcpu_ser *cpu_preserved_pcpus_va __cpu_preserved_data;
+static struct cpu_preserved_pcpu *cpu_preserved_host_pcpus_va __cpu_preserved_data;
+static struct cpu_preserved_global_ser *cpu_preserved_global_ser;
+
+static struct page *cpu_preserved_text_pages;
+static unsigned int cpu_preserved_text_order;
+static struct page *cpu_preserved_data_pages;
+static unsigned int cpu_preserved_data_order;
+static bool cpu_preserved_runtime_preserved;
+
+/*
+ * Address spaces are mapped into under @cpu_preserved_as_map_lock and
+ * enumerated under @cpu_preserved_as_list_lock.  cpu_preserved_map_range()
+ * holds the list lock across the map lock; nothing takes them the other way
+ * round.
+ */
+static DEFINE_MUTEX(cpu_preserved_as_list_lock);
+static DEFINE_MUTEX(cpu_preserved_as_map_lock);
+static LIST_HEAD(cpu_preserved_as_list);
+static struct cpu_preserved_as *cpu_preserved_transition_as;
+
+static phys_addr_t cpu_preserved_get_text_pa(void)
+{
+	return cpu_preserved_text_pages ? page_to_phys(cpu_preserved_text_pages) : 0;
+}
+
+static phys_addr_t cpu_preserved_get_data_pa(void)
+{
+	return cpu_preserved_data_pages ? page_to_phys(cpu_preserved_data_pages) : 0;
+}
+
-- 
2.55.0.1082.g2b9226bbc0-goog

[RFC PATCH 15/46] cpu_preserve: Add physical CPU preservation lifecycle and build rules

[Pasha Tatashin]

Implement physical CPU preservation lifecycle management, stack allocation,
and build rules with -mbranch-protection=none and -fno-stack-protector.

Signed-off-by: Pasha Tatashin <pasha.tatashin@soleen.com>
---
 kernel/cpu.c                     |   2 +
 kernel/liveupdate/Kconfig        |  35 +++
 kernel/liveupdate/Makefile       |   3 +
 kernel/liveupdate/cpu_preserve.c | 359 +++++++++++++++++++++++++++++++
 4 files changed, 399 insertions(+)

diff --git a/kernel/cpu.c b/kernel/cpu.c
index b3c8553d7bd6..8637b5e7f2de 100644
--- a/kernel/cpu.c
+++ b/kernel/cpu.c
@@ -14,6 +14,7 @@
 #include <linux/sched/smt.h>
 #include <linux/unistd.h>
 #include <linux/cpu.h>
+#include <linux/cpu_preserve.h>
 #include <linux/oom.h>
 #include <linux/rcupdate.h>
 #include <linux/delay.h>
@@ -349,6 +350,7 @@ static inline void cpuhp_ap_update_sync_state(enum cpuhp_sync_state state) { }
 void cpuhp_ap_report_dead(void)
 {
 	cpuhp_ap_update_sync_state(SYNC_STATE_DEAD);
+	cpu_preserved_report_dead();
 }
 
 void __weak arch_cpuhp_cleanup_dead_cpu(unsigned int cpu) { }
diff --git a/kernel/liveupdate/Kconfig b/kernel/liveupdate/Kconfig
index 2490f9a9b838..52c5fd7a620b 100644
--- a/kernel/liveupdate/Kconfig
+++ b/kernel/liveupdate/Kconfig
@@ -101,4 +101,39 @@ config LIVEUPDATE_GUEST_MEMFD
 
 	  If unsure, say N.
 
+config ARCH_SUPPORTS_LIVEUPDATE_CPU
+	bool
+
+config LIVEUPDATE_CPU
+	bool "Live update support for preserved CPUs"
+	depends on LIVEUPDATE
+	depends on ARCH_SUPPORTS_LIVEUPDATE_CPU
+	depends on HOTPLUG_CPU
+	default LIVEUPDATE
+	help
+	  Enable live update support for physical CPU preservation.
+
+	  This feature allows physical CPU cores to be preserved across
+	  kexec-based live updates without undergoing a hardware reset or
+	  full kernel offline/online cycle.
+
+	  When a core is preserved, it is isolated from host scheduling and
+	  parked in a dedicated execution loop within preserved memory pages.
+	  Its execution context, state, and necessary page tables are handed
+	  over across the kexec transition using the KHO
+	  framework. The incoming kernel detects preserved cores during early
+	  boot and avoids bringing them up during smp_init(), keeping them
+	  isolated and running their assigned workloads.
+
+	  This capability is a core building block for live-updating virtual
+	  machine hosts (e.g. Caretaker/OrphanVM) without pausing or
+	  disrupting running guest workloads, and can also speed up kernel
+	  live updates by skipping CPU re-initialization.
+
+	  Preservation lifecycle is managed from userspace via LUO sessions
+	  associated with /sys/devices/system/cpu/cpu<N>/preserve file
+	  descriptors, and monitored via sysfs.
+
+	  If unsure, say N.
+
 endmenu
diff --git a/kernel/liveupdate/Makefile b/kernel/liveupdate/Makefile
index d9f469462556..486f7854bb75 100644
--- a/kernel/liveupdate/Makefile
+++ b/kernel/liveupdate/Makefile
@@ -1,5 +1,7 @@
 # SPDX-License-Identifier: GPL-2.0
 
+CFLAGS_cpu_preserve.o += $(call cc-option,-mbranch-protection=none) -fno-stack-protector $(call cc-option,-ftrivial-auto-var-init=uninitialized) $(call cc-option,-fno-jump-tables)
+
 luo-y :=								\
 		kho_block.o						\
 		luo_core.o						\
@@ -11,3 +13,4 @@ obj-$(CONFIG_KEXEC_HANDOVER)		+= kexec_handover.o
 obj-$(CONFIG_KEXEC_HANDOVER_DEBUGFS)	+= kexec_handover_debugfs.o
 
 obj-$(CONFIG_LIVEUPDATE)		+= luo.o
+obj-$(CONFIG_LIVEUPDATE_CPU)		+= cpu_preserve.o
diff --git a/kernel/liveupdate/cpu_preserve.c b/kernel/liveupdate/cpu_preserve.c
index e19208670c3e..c5fd3a07ccb5 100644
--- a/kernel/liveupdate/cpu_preserve.c
+++ b/kernel/liveupdate/cpu_preserve.c
@@ -247,3 +247,362 @@ static phys_addr_t cpu_preserved_get_data_pa(void)
 	return cpu_preserved_data_pages ? page_to_phys(cpu_preserved_data_pages) : 0;
 }
 
+static void cpu_preserved_sync_global_ser(void)
+{
+	struct cpu_preserved_global_ser *ser = cpu_preserved_global_ser;
+
+	if (!ser)
+		return;
+
+	bitmap_to_arr64(ser->cpu_preserved_bitmap,
+			cpumask_bits(&cpu_preserved_mask), nr_cpu_ids);
+	if (cpu_preserved_text_pages) {
+		ser->text_runtime_pa = page_to_phys(cpu_preserved_text_pages);
+		ser->text_runtime_size =
+			(1UL << cpu_preserved_text_order) * PAGE_SIZE;
+	}
+	if (cpu_preserved_data_pages) {
+		ser->data_runtime_pa = page_to_phys(cpu_preserved_data_pages);
+		ser->data_runtime_size =
+			(1UL << cpu_preserved_data_order) * PAGE_SIZE;
+	}
+	KHOSER_STORE_PTR(ser->pcpus_runtime, cpu_preserved_outgoing.pcpus_ser);
+	cpu_preserved_clean_sz(ser,
+			       struct_size(ser, cpu_preserved_bitmap, ser->nr_cpu_words));
+}
+
+static void cpu_preserved_free_kho(void *va, bool is_incoming)
+{
+	if (!va)
+		return;
+
+	if (is_incoming)
+		kho_restore_free(va);
+	else
+		kho_unpreserve_free(va);
+}
+
+static void cpu_preserved_free_stack(phys_addr_t stack_pa, bool is_incoming)
+{
+	if (stack_pa)
+		cpu_preserved_free_kho(phys_to_virt(stack_pa), is_incoming);
+}
+
+static void cpu_preserved_state_cleanup(struct cpu_preserved_state *st,
+					bool is_incoming)
+{
+	if (!cpumask_empty(&st->mask))
+		return;
+
+	cpu_preserved_free_kho(st->pcpus_ser, is_incoming);
+	st->pcpus_ser = NULL;
+	kfree(st->pcpus);
+	st->pcpus = NULL;
+}
+
+/*
+ * Drop @cpu out of the preserved state, free its preserved stack, and
+ * republish the globals a parked core may still be reading.  The caller holds
+ * cpu_preserved_lock and has already made the core leave the park loop.
+ */
+static void __cpu_unpreserve_locked(unsigned int cpu)
+{
+	struct cpu_preserved_state *incoming = &cpu_preserved_incoming;
+	struct cpu_preserved_state *outgoing = &cpu_preserved_outgoing;
+	struct cpu_preserved_pcpu_ser *ser = cpu_preserved_get_pcpu_ser(cpu);
+	struct cpu_preserved_pcpu *pcpu = cpu_preserved_get_pcpu(cpu);
+	bool is_incoming = cpu_preserved_is_incoming(cpu);
+	phys_addr_t stack_pa = 0;
+
+	lockdep_assert_held(&cpu_preserved_lock);
+
+	cpumask_clear_cpu(cpu, &outgoing->mask);
+	cpumask_clear_cpu(cpu, &incoming->mask);
+	cpumask_clear_cpu(cpu, &cpu_preserved_mask);
+	cpu_preserved_clean(&cpu_preserved_mask);
+	set_cpu_present(cpu, true);
+
+	if (ser)
+		WRITE_ONCE(ser->workload, 0);
+
+	if (pcpu) {
+		stack_pa = pcpu->stack_pa;
+		memset(pcpu, 0, sizeof(*pcpu));
+	}
+
+	cpu_preserved_free_stack(stack_pa, is_incoming);
+
+	/* @pcpu and @ser point into these arrays: do not touch past this point. */
+	cpu_preserved_state_cleanup(outgoing, false);
+	cpu_preserved_state_cleanup(incoming, true);
+
+	if (cpumask_empty(&cpu_preserved_mask)) {
+		WRITE_ONCE(cpu_preserved_pcpus_va, NULL);
+		WRITE_ONCE(cpu_preserved_host_pcpus_va, NULL);
+		cpu_preserved_clean(&cpu_preserved_pcpus_va);
+		cpu_preserved_clean(&cpu_preserved_host_pcpus_va);
+	}
+
+	cpu_preserved_sync_global_ser();
+}
+
+/**
+ * cpu_unpreserve - Unpreserve a physical CPU and restore it to online state
+ * @cpu: Logical CPU identifier.
+ *
+ * Signals the CPU to exit the parking loop, cleans up preserved stack memory,
+ * and restores the core to host scheduling via standard add_cpu().
+ */
+static void cpu_unpreserve(unsigned int cpu)
+{
+	int ret;
+
+	scoped_guard(mutex, &cpu_preserved_lock) {
+		if (!cpu_is_preserved(cpu))
+			return;
+
+		cpu_signal_exit(cpu);
+		arch_cpu_preserved_kick(cpu);
+	}
+
+	/*
+	 * cpu_wait_dead() busy-polls for up to 20 seconds.  Do not hold
+	 * cpu_preserved_lock across it: the poll only reads pcpu->state, which
+	 * stays valid for as long as the CPU is preserved, and holding the lock
+	 * here would stall every other preservation operation and every sysfs
+	 * reader for the entire window.
+	 */
+	if (cpu_wait_dead(cpu))
+		return;
+
+	scoped_guard(mutex, &cpu_preserved_lock) {
+		if (!cpu_is_preserved(cpu))
+			return;
+
+		__cpu_unpreserve_locked(cpu);
+	}
+
+	ret = add_cpu(cpu);
+	if (ret < 0)
+		pr_err("Failed to bring unpreserved cpu %u back online: %d\n",
+		       cpu, ret);
+}
+
+/*
+ * FLB Ops for Preserved CPUs
+ */
+static int cpu_preserved_flb_preserve(struct liveupdate_flb_op_args *argp)
+{
+	unsigned int nr_words = BITS_TO_U64(nr_cpu_ids);
+	struct cpu_preserved_global_ser *ser;
+	size_t ser_sz;
+	int ret;
+
+	ret = cpu_preserved_init_runtime_buffer();
+	if (ret)
+		return ret;
+
+	ser_sz = struct_size(ser, cpu_preserved_bitmap, nr_words);
+
+	mutex_lock(&cpu_preserved_lock);
+	ser = kho_alloc_preserve(ser_sz);
+	if (IS_ERR(ser)) {
+		mutex_unlock(&cpu_preserved_lock);
+		return PTR_ERR(ser);
+	}
+
+	memset(ser, 0, ser_sz);
+	ser->nr_cpu_words = nr_words;
+	cpu_preserved_global_ser = ser;
+	cpu_preserved_sync_global_ser();
+	mutex_unlock(&cpu_preserved_lock);
+
+	argp->data = virt_to_phys(ser);
+	argp->obj = ser;
+	return 0;
+}
+
+static void cpu_preserved_flb_unpreserve(struct liveupdate_flb_op_args *argp)
+{
+	struct cpu_preserved_global_ser *ser;
+
+	if (!argp->data)
+		return;
+
+	ser = phys_to_virt(argp->data);
+	mutex_lock(&cpu_preserved_lock);
+	cpu_preserved_global_ser = NULL;
+	mutex_unlock(&cpu_preserved_lock);
+
+	cpu_preserved_unpreserve_runtime_buffer();
+	kho_unpreserve_free(ser);
+}
+
+static int cpu_preserved_flb_retrieve(struct liveupdate_flb_op_args *argp)
+{
+	struct cpu_preserved_global_ser *ser;
+	struct cpu_preserved_pcpu_ser *pcpus;
+	u64 nr_bits;
+	int cpu;
+
+	if (!argp->data)
+		return -EINVAL;
+
+	ser = phys_to_virt(argp->data);
+	arch_cpu_preserved_early_init();
+
+	/*
+	 * The outgoing kernel may have been built with a larger NR_CPUS.  Any
+	 * preserved CPU we cannot represent would be silently forgotten and
+	 * left spinning in its park loop forever, so refuse the handover
+	 * instead.
+	 */
+	nr_bits = (u64)ser->nr_cpu_words * BITS_PER_TYPE(u64);
+	if (nr_bits > nr_cpu_ids &&
+	    find_next_bit((const unsigned long *)ser->cpu_preserved_bitmap,
+			  nr_bits, nr_cpu_ids) < nr_bits) {
+		pr_err("preserved CPU above nr_cpu_ids=%u in handover data\n",
+		       nr_cpu_ids);
+		return -ERANGE;
+	}
+
+	mutex_lock(&cpu_preserved_lock);
+	bitmap_from_arr64(cpumask_bits(&cpu_preserved_mask),
+			  ser->cpu_preserved_bitmap, min_t(u64, nr_bits, nr_cpu_ids));
+	cpumask_copy(&cpu_preserved_incoming.mask, &cpu_preserved_mask);
+	pcpus = KHOSER_LOAD_PTR(ser->pcpus_runtime);
+
+	if (pcpus) {
+		cpu_preserved_incoming.pcpus_ser = pcpus;
+		WRITE_ONCE(cpu_preserved_pcpus_va, pcpus);
+		cpu_preserved_clean(&cpu_preserved_pcpus_va);
+	}
+	cpu_preserved_clean(&cpu_preserved_mask);
+	for_each_cpu(cpu, &cpu_preserved_mask)
+		set_cpu_present(cpu, false);
+	mutex_unlock(&cpu_preserved_lock);
+
+	argp->obj = ser;
+	return 0;
+}
+
+static void cpu_preserved_flb_finish(struct liveupdate_flb_op_args *argp)
+{
+	struct cpu_preserved_global_ser *ser;
+
+	if (!argp->obj)
+		return;
+
+	ser = argp->obj;
+
+	scoped_guard(mutex, &cpu_preserved_lock) {
+		if (cpu_preserved_incoming.pcpus_ser) {
+			kho_restore_free(cpu_preserved_incoming.pcpus_ser);
+			cpu_preserved_incoming.pcpus_ser = NULL;
+		}
+		kfree(cpu_preserved_incoming.pcpus);
+		cpu_preserved_incoming.pcpus = NULL;
+	}
+
+	if (ser->text_runtime_pa && ser->text_runtime_size) {
+		unsigned long nr_pages = ser->text_runtime_size >> PAGE_SHIFT;
+		struct page *page = kho_restore_pages(ser->text_runtime_pa, nr_pages);
+
+		if (page) {
+			for (unsigned long i = 0; i < nr_pages; i++)
+				__free_page(page + i);
+		}
+	}
+
+	if (ser->data_runtime_pa && ser->data_runtime_size) {
+		unsigned long nr_pages = ser->data_runtime_size >> PAGE_SHIFT;
+		struct page *page = kho_restore_pages(ser->data_runtime_pa, nr_pages);
+
+		if (page) {
+			for (unsigned long i = 0; i < nr_pages; i++)
+				__free_page(page + i);
+		}
+	}
+
+	kho_restore_free(ser);
+}
+
+static const struct liveupdate_flb_ops cpu_preserved_flb_ops = {
+	.preserve   = cpu_preserved_flb_preserve,
+	.unpreserve = cpu_preserved_flb_unpreserve,
+	.retrieve   = cpu_preserved_flb_retrieve,
+	.finish     = cpu_preserved_flb_finish,
+	.owner      = THIS_MODULE,
+};
+
+static struct liveupdate_flb cpu_preserved_flb = {
+	.ops        = &cpu_preserved_flb_ops,
+	.compatible = CPU_PRESERVED_LUO_FLB_COMPATIBLE,
+};
+
+static int cpu_preserve_reboot_notify(struct notifier_block *nb,
+				      unsigned long action, void *data)
+{
+	int cpu;
+
+	scoped_guard(mutex, &cpu_preserved_lock) {
+		for_each_cpu(cpu, &cpu_preserved_mask) {
+			/*
+			 * If this CPU is not being preserved across an outgoing
+			 * live update, signal it to exit the park loop and
+			 * offline it.
+			 */
+			if (kexec_in_progress && liveupdate_enabled() &&
+			    !cpu_preserved_is_incoming(cpu))
+				continue;
+
+			cpu_signal_exit(cpu);
+			arch_cpu_preserved_kick(cpu);
+			if (cpu_wait_dead(cpu))
+				continue;
+
+			__cpu_unpreserve_locked(cpu);
+		}
+	}
+
+	return NOTIFY_OK;
+}
+
+static struct notifier_block cpu_preserve_reboot_nb = {
+	.notifier_call = cpu_preserve_reboot_notify,
+	.priority = 0,
+};
+
+/**
+ * cpu_preserve_early_init - Early boot registration & retrieval of CPUs
+ *
+ * Registers the preserved CPU file handler and FLB with LUO, retrieves incoming
+ * preserved CPU state prior to secondary SMP bringup, and registers the reboot
+ * notifier.
+ *
+ * Return: 0 on success, or negative error code on failure.
+ */
+static int __init cpu_preserve_early_init(void)
+{
+	void *obj;
+	int err;
+
+	if (!liveupdate_enabled())
+		cpumask_clear(&cpu_preserved_mask);
+	cpumask_clear(&cpu_preserved_outgoing.mask);
+	cpumask_clear(&cpu_preserved_incoming.mask);
+	cpu_preserved_outgoing.pcpus = NULL;
+	cpu_preserved_outgoing.pcpus_ser = NULL;
+	cpu_preserved_incoming.pcpus = NULL;
+	cpu_preserved_incoming.pcpus_ser = NULL;
+	cpu_preserved_global_ser = NULL;
+
+	/* Retrieve incoming preserved CPUs before secondary CPU bringup */
+	if (liveupdate_enabled())
+		liveupdate_flb_get_incoming(&cpu_preserved_flb, &obj);
+
+	register_reboot_notifier(&cpu_preserve_reboot_nb);
+
+	return 0;
+}
+early_initcall(cpu_preserve_early_init);
-- 
2.55.0.1082.g2b9226bbc0-goog

[RFC PATCH 16/46] liveupdate: cpu_preserve: Add sysfs interface

[Pasha Tatashin]

Expose /sys/devices/system/cpu/preserved (global cpumask of preserved
cores) and /sys/devices/system/cpu/cpuX/preserve (per-CPU preservation
attribute and file descriptor target for LUO sessions), and document
both attributes in Documentation/ABI/testing/sysfs-devices-system-cpu.

Signed-off-by: Pasha Tatashin <pasha.tatashin@soleen.com>
---
 .../ABI/testing/sysfs-devices-system-cpu      | 20 ++++++++
 kernel/liveupdate/cpu_preserve.c              | 49 +++++++++++++++++++
 2 files changed, 69 insertions(+)

diff --git a/Documentation/ABI/testing/sysfs-devices-system-cpu b/Documentation/ABI/testing/sysfs-devices-system-cpu
index 82d10d556cc8..88ab03efd433 100644
--- a/Documentation/ABI/testing/sysfs-devices-system-cpu
+++ b/Documentation/ABI/testing/sysfs-devices-system-cpu
@@ -806,3 +806,23 @@ Date:		Nov 2022
 Contact:	Linux kernel mailing list <linux-kernel@vger.kernel.org>
 Description:
 		(RO) the list of CPUs that can be brought online.
+
+What:		/sys/devices/system/cpu/preserved
+Date:		March 2026
+Contact:	Pasha Tatashin <pasha.tatashin@soleen.com>
+Description:
+		(RO) cpumask list of physical CPUs currently isolated and
+		preserved across a host kernel live update. Note the naming
+		distinction from the per-CPU /sys/devices/system/cpu/cpuX/preserve
+		attribute.
+
+What:		/sys/devices/system/cpu/cpuX/preserve
+Date:		March 2026
+Contact:	Pasha Tatashin <pasha.tatashin@soleen.com>
+Description:
+		(RO) returns 1 if cpuX is currently preserved across a host
+		kernel live update, 0 otherwise. Although read-only, this sysfs
+		node exists to be opened and passed by file descriptor to a Live
+		Update Orchestrator (LUO) session via LIVEUPDATE_SESSION_PRESERVE_FD
+		to request preservation of cpuX across kexec.
+
diff --git a/kernel/liveupdate/cpu_preserve.c b/kernel/liveupdate/cpu_preserve.c
index c5fd3a07ccb5..cc27d1624d29 100644
--- a/kernel/liveupdate/cpu_preserve.c
+++ b/kernel/liveupdate/cpu_preserve.c
@@ -606,3 +606,52 @@ static int __init cpu_preserve_early_init(void)
 	return 0;
 }
 early_initcall(cpu_preserve_early_init);
+static ssize_t preserved_show(struct device *dev,
+			      struct device_attribute *attr, char *buf)
+{
+	return sysfs_emit(buf, "%*pbl\n",
+			  cpumask_pr_args(cpu_get_preserved_mask()));
+}
+static DEVICE_ATTR_RO(preserved);
+
+static ssize_t preserve_show(struct device *dev,
+			     struct device_attribute *attr, char *buf)
+{
+	return sysfs_emit(buf, "%d\n", cpu_is_preserved(dev->id));
+}
+static DEVICE_ATTR_RO(preserve);
+
+static int __init cpu_preserve_sysfs_init(void)
+{
+	struct device *dev_root = bus_get_dev_root(&cpu_subsys);
+	int cpu, ret;
+
+	if (dev_root) {
+		ret = sysfs_create_file(&dev_root->kobj, &dev_attr_preserved.attr);
+		put_device(dev_root);
+		if (ret)
+			pr_warn("Failed to create cpu preserved sysfs attribute: %d\n", ret);
+	}
+
+	for_each_possible_cpu(cpu) {
+		struct device *dev = get_cpu_device(cpu);
+
+		if (!dev && cpu_is_preserved(cpu)) {
+			set_cpu_present(cpu, true);
+			arch_register_cpu(cpu);
+			dev = get_cpu_device(cpu);
+		}
+
+		if (dev) {
+			ret = sysfs_create_file(&dev->kobj, &dev_attr_preserve.attr);
+			if (ret)
+				pr_warn("Failed to create cpu%d preserve sysfs attribute: %d\n",
+					cpu, ret);
+		}
+
+		if (cpu_is_preserved(cpu))
+			set_cpu_present(cpu, false);
+	}
+	return 0;
+}
+late_initcall(cpu_preserve_sysfs_init);
-- 
2.55.0.1082.g2b9226bbc0-goog

[RFC PATCH 17/46] liveupdate: cpu_preserve: Add isolated address space management API

[Pasha Tatashin]

Add struct cpu_preserved_as and cpu_preserved_as_create/map/adopt/destroy
interfaces to manage isolated page tables for preserved physical CPUs.

Signed-off-by: Pasha Tatashin <pasha.tatashin@soleen.com>
---
 include/linux/cpu_preserve.h     |  65 +++
 include/linux/kho/abi/cpu.h      |  38 ++
 kernel/liveupdate/cpu_preserve.c | 958 +++++++++++++++++++++++++++++++
 3 files changed, 1061 insertions(+)

diff --git a/include/linux/cpu_preserve.h b/include/linux/cpu_preserve.h
index 94181952df1a..472fcf3933a2 100644
--- a/include/linux/cpu_preserve.h
+++ b/include/linux/cpu_preserve.h
@@ -269,6 +269,71 @@ void cpu_preserved_as_destroy(struct cpu_preserved_as *as);
 struct cpu_preserved_as *cpu_preserved_as_adopt(struct cpu_preserved_as_ser *ser);
 int cpu_preserved_as_map(struct cpu_preserved_as *as, phys_addr_t pa,
 			 unsigned long va, size_t size, pgprot_t prot);
+void *cpu_preserved_as_alloc_page(void *arg);
+
+/**
+ * arch_cpu_preserved_as_map - Add one range to a preserved address space
+ * @as: Address space to map into; @as->pgd is the root to populate.
+ * @pa: Physical address of the range.
+ * @va: Virtual address the range must appear at.
+ * @size: Size of the range in bytes.
+ * @prot: Protection to apply.
+ *
+ * Architecture backend for cpu_preserved_as_map().  Page table pages must be
+ * obtained from cpu_preserved_as_alloc_page() with @as as its argument, so
+ * that the core layer can preserve and later free them; the caller holds the
+ * mapping lock and takes care of cache maintenance and of the TLB.
+ *
+ * Return: 0 on success, or a negative errno on failure.
+ */
+int arch_cpu_preserved_as_map(struct cpu_preserved_as *as, phys_addr_t pa,
+			      unsigned long va, size_t size, pgprot_t prot);
+
+/**
+ * arch_cpu_preserved_as_flush_tlb - Publish preserved page table updates
+ *
+ * Called after every successful arch_cpu_preserved_as_map().  Architectures
+ * whose preserved CPUs can hold stale translations for these address spaces
+ * must invalidate them here; the others need do nothing.
+ */
+void arch_cpu_preserved_as_flush_tlb(void);
+
+/**
+ * arch_cpu_preserved_set_transition_as - Publish the default address space
+ * @as: Address space a preserved CPU parks in when its workload has none.
+ *
+ * The value has to be readable from preserved text after the kexec, which is
+ * architecture specific storage, so the core layer hands it over rather than
+ * exporting a variable.
+ */
+void arch_cpu_preserved_set_transition_as(struct cpu_preserved_as *as);
+
+int cpu_preserved_map_range(phys_addr_t pa, unsigned long va,
+			    size_t size, pgprot_t prot);
+int cpu_preserved_map_buffer(void *va, size_t size);
+
+/**
+ * arch_cpu_preserved_mpidr_to_cpu - Map an ARM64 MPIDR affinity value to a logical CPU ID
+ * @mpidr: Hardware MPIDR affinity value.
+ *
+ * Return: Logical CPU identifier, or -EINVAL if not found.
+ */
+int arch_cpu_preserved_mpidr_to_cpu(u64 mpidr);
+
+/**
+ * arch_cpu_preserved_is_active - Check whether any preserved CPU runtime mapping is active
+ *
+ * Return: %true if preserved runtime mappings are active, %false otherwise.
+ */
+bool arch_cpu_preserved_is_active(void);
+
+/**
+ * arch_cpu_preserved_switch_pgd - Switch the current preserved CPU to an isolated PGD
+ * @pgd_pa: Physical address of the root page table to install.
+ *
+ * This function must be placed in the __cpu_preserved_text section.
+ */
+void arch_cpu_preserved_switch_pgd(phys_addr_t pgd_pa);
 
 #else /* !CONFIG_LIVEUPDATE_CPU */
 
diff --git a/include/linux/kho/abi/cpu.h b/include/linux/kho/abi/cpu.h
index 5926be37f7a0..f75bfb4c6161 100644
--- a/include/linux/kho/abi/cpu.h
+++ b/include/linux/kho/abi/cpu.h
@@ -96,6 +96,44 @@ static_assert(offsetof(struct cpu_preserved_global_ser,
 		       cpu_preserved_bitmap) % sizeof(u64) == 0,
 	      "cpu_preserved_bitmap must be 64-bit aligned");
 
+#define CPU_PRESERVED_AS_MAX_PGTABLE_PAGES	1024
+
+/**
+ * struct cpu_preserved_as_ser - Serialized preserved address space metadata
+ * @nr_pgtable_pages: Number of valid entries in @pgtable_pages.
+ * @reserved:         Must be zero.
+ * @pgtable_pages:    Physical addresses of all page table pages (root PGD is at index 0).
+ */
+struct cpu_preserved_as_ser {
+	u32 nr_pgtable_pages;
+	u32 reserved;
+	u64 pgtable_pages[CPU_PRESERVED_AS_MAX_PGTABLE_PAGES];
+} __packed;
+
+static_assert(offsetof(struct cpu_preserved_as_ser, pgtable_pages) == 8);
+
+/**
+ * struct oncore_session_ser - Serialized on-core session metadata
+ * @session_name:  LUO session name.
+ * @sess_pa:       Opaque physical address of preserved struct oncore_session
+ *                 (retained for freeing across kexec, never dereferenced).
+ * @as:            Preservation pointer to struct cpu_preserved_as_ser.
+ * @nr_cpu_words:  Number of 64-bit words in @cpus_bitmap.
+ * @reserved:      Must be zero.
+ * @cpus_bitmap:   Bitmap of physical CPUs assigned to this on-core session.
+ */
+struct oncore_session_ser {
+	char session_name[LIVEUPDATE_SESSION_NAME_LENGTH];
+	u64 sess_pa;
+	DECLARE_KHOSER_PTR(as, struct cpu_preserved_as_ser *);
+	u32 nr_cpu_words;
+	u32 reserved;
+	u64 cpus_bitmap[];
+} __packed;
+
+static_assert(offsetof(struct oncore_session_ser, cpus_bitmap) % sizeof(u64) == 0,
+	      "cpus_bitmap must be 64-bit aligned");
+
 /**
  * struct cpu_preserved_file_ser - Per-file serialized state for preserved CPU fd
  * @cpu:      Logical CPU identifier.
diff --git a/kernel/liveupdate/cpu_preserve.c b/kernel/liveupdate/cpu_preserve.c
index cc27d1624d29..2f92dfee82b1 100644
--- a/kernel/liveupdate/cpu_preserve.c
+++ b/kernel/liveupdate/cpu_preserve.c
@@ -267,6 +267,8 @@ static void cpu_preserved_sync_global_ser(void)
 			(1UL << cpu_preserved_data_order) * PAGE_SIZE;
 	}
 	KHOSER_STORE_PTR(ser->pcpus_runtime, cpu_preserved_outgoing.pcpus_ser);
+	KHOSER_STORE_PTR(ser->transition_as,
+			 cpu_preserved_transition_as ? cpu_preserved_transition_as->ser : NULL);
 	cpu_preserved_clean_sz(ser,
 			       struct_size(ser, cpu_preserved_bitmap, ser->nr_cpu_words));
 }
@@ -282,6 +284,857 @@ static void cpu_preserved_free_kho(void *va, bool is_incoming)
 		kho_unpreserve_free(va);
 }
 
+/**
+ * cpu_preserved_as_alloc_page - Allocate a page table page for @arg
+ * @arg: The struct cpu_preserved_as being populated.
+ *
+ * Page table allocator handed to the architecture page table builders.
+ *
+ * There is deliberately no alloc_page() fallback.  It would be
+ * kho_alloc_preserve() open-coded, and the only way it could differ is by
+ * ignoring the preservation error -- which would hand back an unpreserved
+ * page table page.  The orphaned core has no fault handler, so that failure
+ * is unrecoverable and must not be silent.
+ *
+ * Return: A zeroed, preserved page, or NULL.
+ */
+void *cpu_preserved_as_alloc_page(void *arg)
+{
+	struct cpu_preserved_as *as = arg;
+	void *ptr;
+
+	if (WARN_ON_ONCE(as->ser->nr_pgtable_pages >= ARRAY_SIZE(as->ser->pgtable_pages)))
+		return NULL;
+
+	ptr = kho_alloc_preserve(PAGE_SIZE);
+	if (IS_ERR_OR_NULL(ptr))
+		return NULL;
+
+	cpu_preserved_clean_sz(ptr, PAGE_SIZE);
+	as->ser->pgtable_pages[as->ser->nr_pgtable_pages++] = virt_to_phys(ptr);
+
+	return ptr;
+}
+EXPORT_SYMBOL_GPL(cpu_preserved_as_alloc_page);
+
+/*
+ * Page table pages are preserved as they are allocated, but a cancelled live
+ * update unpreserves everything, so state the preservation again after every
+ * change.  Pages inherited from the previous kernel already belong to KHO.
+ */
+static int cpu_preserved_as_preserve_pgtables(struct cpu_preserved_as *as)
+{
+	unsigned int i;
+
+	if (as->is_incoming)
+		return 0;
+
+	for (i = 0; i < as->ser->nr_pgtable_pages; i++) {
+		void *p = phys_to_virt(as->ser->pgtable_pages[i]);
+		int ret;
+
+		cpu_preserved_clean_sz(p, PAGE_SIZE);
+		ret = kho_preserve_pages(virt_to_page(p), 1);
+		if (ret)
+			return ret;
+	}
+
+	return 0;
+}
+
+static void cpu_preserved_as_unpreserve_pgtables(struct cpu_preserved_as *as)
+{
+	for (unsigned int i = 0; i < as->ser->nr_pgtable_pages; i++)
+		kho_unpreserve_pages(virt_to_page(phys_to_virt(as->ser->pgtable_pages[i])), 1);
+}
+
+/**
+ * cpu_preserved_as_map - Map one range into one preserved address space
+ * @as: Address space to map into.
+ * @pa: Physical address of the range.
+ * @va: Virtual address the range must appear at.
+ * @size: Size of the range in bytes.
+ * @prot: Protection to apply.
+ *
+ * Return: 0 on success, negative errno on failure.
+ */
+int cpu_preserved_as_map(struct cpu_preserved_as *as, phys_addr_t pa,
+			 unsigned long va, size_t size, pgprot_t prot)
+{
+	int ret;
+
+	guard(mutex)(&cpu_preserved_as_map_lock);
+
+	ret = arch_cpu_preserved_as_map(as, pa, va, size, prot);
+	if (ret)
+		return ret;
+
+	ret = cpu_preserved_as_preserve_pgtables(as);
+	if (ret)
+		return ret;
+
+	arch_cpu_preserved_as_flush_tlb();
+
+	return 0;
+}
+EXPORT_SYMBOL_GPL(cpu_preserved_as_map);
+
+static int cpu_preserved_init_runtime_buffer(void);
+
+static int cpu_preserved_as_map_buf(struct cpu_preserved_as *as,
+				    void *va, size_t size)
+{
+	if (!va || !size)
+		return 0;
+
+	return cpu_preserved_as_map(as, virt_to_phys(va), (unsigned long)va,
+				    size, PAGE_KERNEL);
+}
+
+static int cpu_preserved_as_map_runtime(struct cpu_preserved_as *as)
+{
+	unsigned long text_start = (unsigned long)__cpu_preserved_text_start;
+	unsigned long data_start = (unsigned long)__cpu_preserved_data_start;
+	size_t text_sz = (unsigned long)__cpu_preserved_text_end - text_start;
+	size_t data_sz = (unsigned long)__cpu_preserved_data_end - data_start;
+	struct cpu_preserved_state *outgoing = &cpu_preserved_outgoing;
+	int cpu, ret;
+
+	ret = cpu_preserved_as_map(as, cpu_preserved_get_text_pa(),
+				   text_start, text_sz, PAGE_KERNEL_ROX);
+	if (ret)
+		return ret;
+
+	ret = cpu_preserved_as_map(as, cpu_preserved_get_data_pa(),
+				   data_start, data_sz, PAGE_KERNEL);
+	if (ret)
+		return ret;
+
+	ret = cpu_preserved_as_map_buf(as, outgoing->pcpus_ser,
+				       sizeof(*outgoing->pcpus_ser) * nr_cpu_ids);
+	if (ret)
+		return ret;
+
+	ret = cpu_preserved_as_map_buf(as, outgoing->pcpus,
+				       sizeof(*outgoing->pcpus) * nr_cpu_ids);
+	if (ret)
+		return ret;
+
+	for_each_cpu(cpu, &outgoing->mask) {
+		phys_addr_t spa = outgoing->pcpus[cpu].stack_pa;
+
+		if (!spa)
+			continue;
+		ret = cpu_preserved_as_map_buf(as, phys_to_virt(spa),
+					       CPU_PRESERVED_STACK_SIZE);
+		if (ret)
+			return ret;
+	}
+
+	return 0;
+}
+
+/**
+ * cpu_preserved_as_create - Build a new preserved address space
+ *
+ * Allocates a root page table, maps the preserved text and data into it, and
+ * publishes it so that subsequent cpu_preserved_map_range() calls reach it.
+ *
+ * Return: The new address space, or an ERR_PTR() on failure.
+ */
+struct cpu_preserved_as *cpu_preserved_as_create(void)
+{
+	struct cpu_preserved_as *as;
+	int ret;
+
+	ret = cpu_preserved_init_runtime_buffer();
+	if (ret)
+		return ERR_PTR(ret);
+
+	as = kzalloc_obj(*as, GFP_KERNEL);
+	if (!as)
+		return ERR_PTR(-ENOMEM);
+	INIT_LIST_HEAD(&as->node);
+
+	as->ser = kho_alloc_preserve(sizeof(*as->ser));
+	if (IS_ERR(as->ser)) {
+		ret = PTR_ERR(as->ser);
+		kfree(as);
+		return ERR_PTR(ret);
+	}
+	memset(as->ser, 0, sizeof(*as->ser));
+
+	as->pgd = cpu_preserved_as_alloc_page(as);
+	if (!as->pgd) {
+		ret = -ENOMEM;
+		goto err;
+	}
+	as->pgd_pa = virt_to_phys(as->pgd);
+
+	ret = cpu_preserved_as_map_runtime(as);
+	if (ret)
+		goto err;
+
+	scoped_guard(mutex, &cpu_preserved_as_list_lock)
+		list_add_tail(&as->node, &cpu_preserved_as_list);
+
+	return as;
+
+err:
+	cpu_preserved_as_destroy(as);
+	return ERR_PTR(ret);
+}
+EXPORT_SYMBOL_GPL(cpu_preserved_as_create);
+
+/**
+ * cpu_preserved_as_destroy - Tear down a preserved address space
+ * @as: Address space to release.  NULL is accepted and does nothing.
+ */
+void cpu_preserved_as_destroy(struct cpu_preserved_as *as)
+{
+	if (!as)
+		return;
+
+	scoped_guard(mutex, &cpu_preserved_as_list_lock)
+		list_del_init(&as->node);
+
+	if (as->ser) {
+		scoped_guard(mutex, &cpu_preserved_as_map_lock) {
+			for (unsigned int i = 0; i < as->ser->nr_pgtable_pages; i++) {
+				void *va = phys_to_virt(as->ser->pgtable_pages[i]);
+
+				cpu_preserved_free_kho(va, as->is_incoming);
+			}
+		}
+		cpu_preserved_free_kho(as->ser, as->is_incoming);
+	}
+
+	kfree(as);
+}
+EXPORT_SYMBOL_GPL(cpu_preserved_as_destroy);
+
+/**
+ * cpu_preserved_as_adopt - Take over an address space from the previous kernel
+ * @ser: Address space serialization descriptor recovered from preserved memory.
+ *
+ * The page tables are left exactly as the outgoing kernel built them --
+ * preserved CPUs are running out of them right now -- but the list linkage is
+ * stale and has to be rebuilt, and the pages now belong to KHO rather than to
+ * this kernel's allocator.
+ */
+struct cpu_preserved_as *cpu_preserved_as_adopt(struct cpu_preserved_as_ser *ser)
+{
+	struct cpu_preserved_as *as;
+
+	if (!ser)
+		return NULL;
+
+	as = kzalloc_obj(*as, GFP_KERNEL);
+	if (!as)
+		return NULL;
+
+	as->ser = ser;
+	as->pgd_pa = ser->nr_pgtable_pages ? ser->pgtable_pages[0] : 0;
+	as->pgd = phys_to_virt(as->pgd_pa);
+	as->is_incoming = true;
+	INIT_LIST_HEAD(&as->node);
+
+	guard(mutex)(&cpu_preserved_as_list_lock);
+	list_add_tail(&as->node, &cpu_preserved_as_list);
+
+	return as;
+}
+EXPORT_SYMBOL_GPL(cpu_preserved_as_adopt);
+
+static void cpu_preserved_preserve_runtime_buffer(void)
+{
+	if (cpu_preserved_runtime_preserved)
+		return;
+
+	/*
+	 * This is the text the orphaned core executes and the data it reads
+	 * after the kexec.  If either cannot be preserved there is nothing to
+	 * hand over, so do not claim the runtime is preserved.
+	 */
+	if (WARN_ON_ONCE(kho_preserve_pages(cpu_preserved_text_pages,
+					    1 << cpu_preserved_text_order)))
+		return;
+	if (WARN_ON_ONCE(kho_preserve_pages(cpu_preserved_data_pages,
+					    1 << cpu_preserved_data_order)))
+		return;
+
+	WARN_ON_ONCE(kho_preserve_pages(virt_to_page(cpu_preserved_transition_as->ser),
+					1 << get_order(sizeof(*cpu_preserved_transition_as->ser))));
+
+	scoped_guard(mutex, &cpu_preserved_as_map_lock)
+		WARN_ON_ONCE(cpu_preserved_as_preserve_pgtables(cpu_preserved_transition_as));
+
+	cpu_preserved_runtime_preserved = true;
+}
+
+static void cpu_preserved_unpreserve_runtime_buffer(void)
+{
+	if (!cpu_preserved_runtime_preserved)
+		return;
+
+	kho_unpreserve_pages(cpu_preserved_text_pages,
+			     1 << cpu_preserved_text_order);
+	kho_unpreserve_pages(cpu_preserved_data_pages,
+			     1 << cpu_preserved_data_order);
+	kho_unpreserve_pages(virt_to_page(cpu_preserved_transition_as->ser),
+			     1 << get_order(sizeof(*cpu_preserved_transition_as->ser)));
+
+	scoped_guard(mutex, &cpu_preserved_as_map_lock)
+		cpu_preserved_as_unpreserve_pgtables(cpu_preserved_transition_as);
+
+	cpu_preserved_runtime_preserved = false;
+}
+
+/**
+ * cpu_preserved_init_runtime_buffer - Allocate execution buffer outside Scratch
+ *
+ * The compiled __cpu_preserved_text and __cpu_preserved_data sections are
+ * part of the host kernel binary image. During a host kexec live update, the
+ * memory range occupied by the current kernel is designated as KHO Scratch
+ * memory to allow the incoming kernel to be placed and unpacked. By definition,
+ * Scratch memory must not contain preserved memory, as the incoming kernel
+ * will overwrite Scratch during boot.
+ *
+ * Preserving the compiled text and data sections in-place would create a
+ * conflict where preserved memory overlaps Scratch, triggering handover
+ * failures or memory corruption when the incoming kernel overwrites the old
+ * kernel text while preserved physical CPUs are still executing Caretaker loops
+ * on their cores.
+ *
+ * To avoid this, we dynamically allocate dedicated text and data buffer pages
+ * from free memory (outside Scratch) via alloc_pages(GFP_KERNEL), copy the
+ * compiled text and data into them, remap the virtual addresses in the page
+ * tables to point to these newly allocated pages, and preserve only these
+ * external pages with KHO. Preserved CPUs execute out of these external pages,
+ * allowing the incoming kernel to freely overwrite Scratch.
+ *
+ * Return: 0 on success, or negative error code on allocation/setup failure.
+ */
+static int cpu_preserved_init_runtime_buffer(void)
+{
+	size_t text_size = (unsigned long)__cpu_preserved_text_end -
+			   (unsigned long)__cpu_preserved_text_start;
+	size_t data_size = (unsigned long)__cpu_preserved_data_end -
+			   (unsigned long)__cpu_preserved_data_start;
+	unsigned int text_nr_pages = DIV_ROUND_UP(text_size, PAGE_SIZE);
+	unsigned int data_nr_pages = DIV_ROUND_UP(data_size, PAGE_SIZE);
+	int ret;
+
+	if (cpu_preserved_text_pages) {
+		if (cpu_preserved_transition_as)
+			cpu_preserved_preserve_runtime_buffer();
+		return 0;
+	}
+
+	cpu_preserved_text_order = get_order(text_size);
+	cpu_preserved_text_pages = alloc_pages(GFP_KERNEL, cpu_preserved_text_order);
+	if (!cpu_preserved_text_pages)
+		return -ENOMEM;
+
+	cpu_preserved_data_order = get_order(data_size);
+	cpu_preserved_data_pages = alloc_pages(GFP_KERNEL, cpu_preserved_data_order);
+	if (!cpu_preserved_data_pages) {
+		__free_pages(cpu_preserved_text_pages, cpu_preserved_text_order);
+		cpu_preserved_text_pages = NULL;
+		return -ENOMEM;
+	}
+
+	memcpy(page_address(cpu_preserved_text_pages),
+	       __cpu_preserved_text_start, text_size);
+	memcpy(page_address(cpu_preserved_data_pages),
+	       __cpu_preserved_data_start, data_size);
+
+	ret = arch_cpu_preserved_setup_buffer(cpu_preserved_text_pages,
+					      text_nr_pages,
+					      cpu_preserved_data_pages,
+					      data_nr_pages);
+	if (ret)
+		goto err_free;
+
+	/*
+	 * The address space a preserved CPU parks in when its workload has not
+	 * given it one of its own.  It has to exist before anything can be
+	 * mapped for preserved CPUs, so build it here and let the architecture
+	 * record it where preserved text can reach it after the kexec.
+	 */
+	cpu_preserved_transition_as = cpu_preserved_as_create();
+	if (IS_ERR(cpu_preserved_transition_as)) {
+		ret = PTR_ERR(cpu_preserved_transition_as);
+		cpu_preserved_transition_as = NULL;
+		goto err_free;
+	}
+	arch_cpu_preserved_set_transition_as(cpu_preserved_transition_as);
+
+	cpu_preserved_preserve_runtime_buffer();
+	return 0;
+
+err_free:
+	__free_pages(cpu_preserved_data_pages, cpu_preserved_data_order);
+	__free_pages(cpu_preserved_text_pages, cpu_preserved_text_order);
+	cpu_preserved_data_pages = NULL;
+	cpu_preserved_text_pages = NULL;
+	return ret;
+}
+
+/**
+ * cpu_preserved_map_range - Map a physical range into every preserved address space
+ * @pa: Physical address
+ * @va: Virtual address
+ * @size: Size in bytes
+ * @prot: Page protection flags
+ *
+ * Anything a preserved CPU may touch has to be reachable from whichever
+ * address space it ends up running in, and which one that is depends on the
+ * workload, so map it into all of them.
+ *
+ * Return: 0 on success, negative errno on failure.
+ */
+int cpu_preserved_map_range(phys_addr_t pa, unsigned long va,
+			    size_t size, pgprot_t prot)
+{
+	struct cpu_preserved_as *as;
+	int ret;
+
+	guard(mutex)(&cpu_preserved_as_list_lock);
+
+	list_for_each_entry(as, &cpu_preserved_as_list, node) {
+		ret = cpu_preserved_as_map(as, pa, va, size, prot);
+		if (ret)
+			return ret;
+	}
+
+	return 0;
+}
+EXPORT_SYMBOL_GPL(cpu_preserved_map_range);
+
+/**
+ * cpu_preserved_map_buffer - Map a virtual buffer into transition page tables
+ * @va: Virtual address in kernel direct map
+ * @size: Size in bytes
+ *
+ * Return: 0 on success, negative errno on failure.
+ */
+int cpu_preserved_map_buffer(void *va, size_t size)
+{
+	if (!va || !size)
+		return 0;
+	return cpu_preserved_map_range(virt_to_phys(va),
+				       (unsigned long)va,
+				       size, PAGE_KERNEL);
+}
+EXPORT_SYMBOL_GPL(cpu_preserved_map_buffer);
+
+/**
+ * cpu_is_preserved - Check whether a CPU is currently preserved
+ * @cpu: Logical CPU identifier.
+ *
+ * Return: True if @cpu is currently preserved, false otherwise.
+ */
+bool __cpu_preserved_text cpu_is_preserved(int cpu)
+{
+	if ((unsigned int)cpu >= CONFIG_NR_CPUS)
+		return false;
+	cpu_preserved_inval(&cpu_preserved_mask);
+	return cpumask_test_cpu(cpu, &cpu_preserved_mask);
+}
+EXPORT_SYMBOL_GPL(cpu_is_preserved);
+
+static bool cpu_preserved_is_incoming(int cpu)
+{
+	if ((unsigned int)cpu >= CONFIG_NR_CPUS)
+		return false;
+	return cpumask_test_cpu(cpu, &cpu_preserved_incoming.mask);
+}
+
+static struct cpu_preserved_pcpu_ser * __cpu_preserved_text cpu_preserved_get_pcpu_ser(int cpu)
+{
+	struct cpu_preserved_pcpu_ser *pcpus;
+
+	if ((unsigned int)cpu >= CONFIG_NR_CPUS)
+		return NULL;
+
+	cpu_preserved_inval(&cpu_preserved_pcpus_va);
+	pcpus = READ_ONCE(cpu_preserved_pcpus_va);
+	return pcpus ? &pcpus[cpu] : NULL;
+}
+
+static struct cpu_preserved_pcpu * __cpu_preserved_text cpu_preserved_get_pcpu(int cpu)
+{
+	struct cpu_preserved_pcpu *pcpus;
+
+	if ((unsigned int)cpu >= CONFIG_NR_CPUS)
+		return NULL;
+
+	cpu_preserved_inval(&cpu_preserved_host_pcpus_va);
+	pcpus = READ_ONCE(cpu_preserved_host_pcpus_va);
+	return pcpus ? &pcpus[cpu] : NULL;
+}
+
+/*
+ * The preserved stack is handed over by physical address: the same page need
+ * not be mapped at the same virtual address by two different kernels, so each
+ * side derives its own VA rather than sharing one.
+ */
+static void * __cpu_preserved_text
+cpu_preserved_stack_va(int cpu)
+{
+	struct cpu_preserved_pcpu *pcpu = cpu_preserved_get_pcpu(cpu);
+	phys_addr_t pa;
+
+	if (!pcpu)
+		return NULL;
+
+	cpu_preserved_inval(&pcpu->stack_pa);
+	pa = READ_ONCE(pcpu->stack_pa);
+	if (!pa)
+		return NULL;
+
+	return phys_to_virt(pa);
+}
+
+/**
+ * cpu_preserved_get_pgd - Get root page table physical address for a preserved CPU
+ * @cpu: Logical CPU identifier.
+ *
+ * Return: Root PGD physical address assigned to @cpu, or 0 if not set.
+ */
+phys_addr_t __cpu_preserved_text cpu_preserved_get_pgd(int cpu)
+{
+	struct cpu_preserved_pcpu *pcpu = cpu_preserved_get_pcpu(cpu);
+
+	if (!pcpu)
+		return 0;
+
+	cpu_preserved_inval(&pcpu->pgd_pa);
+	return READ_ONCE(pcpu->pgd_pa);
+}
+EXPORT_SYMBOL_GPL(cpu_preserved_get_pgd);
+
+/**
+ * cpu_get_preserved_mask - Get the mask of all currently preserved CPUs
+ *
+ * Return: Read-only pointer to the cpumask of preserved CPUs.
+ */
+const struct cpumask *cpu_get_preserved_mask(void)
+{
+	return &cpu_preserved_mask;
+}
+EXPORT_SYMBOL_GPL(cpu_get_preserved_mask);
+
+/**
+ * cpu_preserved_set_dead - Mark a preserved CPU as fully dead/stopped
+ * @cpu: Logical CPU identifier.
+ *
+ * Publishes %CPU_PRESERVED_DEAD in the KHO-preserved per-CPU state block when
+ * @cpu finishes exiting the preserved parking loop.
+ */
+void __cpu_preserved_text cpu_preserved_set_dead(int cpu)
+{
+	struct cpu_preserved_pcpu_ser *ser = cpu_preserved_get_pcpu_ser(cpu);
+
+	if (ser)
+		WRITE_ONCE(ser->workload, CPU_PRESERVED_DEAD);
+}
+EXPORT_SYMBOL_GPL(cpu_preserved_set_dead);
+
+static void cpu_signal_exit(int cpu)
+{
+	struct cpu_preserved_pcpu_ser *ser = cpu_preserved_get_pcpu_ser(cpu);
+	struct cpu_preserved_pcpu *pcpu = cpu_preserved_get_pcpu(cpu);
+
+	if (ser) {
+		WRITE_ONCE(ser->workload, CPU_PRESERVED_EXITING);
+		cpu_preserved_clean(ser);
+	}
+	if (pcpu) {
+		WRITE_ONCE(pcpu->entry_fn, NULL);
+		WRITE_ONCE(pcpu->entry_data, NULL);
+		cpu_preserved_clean(pcpu);
+	}
+}
+
+/**
+ * cpu_preserved_should_exit - Check if a running preserved workload should exit
+ * @cpu: Logical CPU identifier.
+ *
+ * Polled by workloads executing on preserved physical CPUs to detect when the
+ * host kernel has requested workload detachment or CPU reclamation.
+ *
+ * Return: %true if the workload on @cpu must exit back to the park loop,
+ *         %false otherwise.
+ */
+bool __cpu_preserved_text cpu_preserved_should_exit(int cpu)
+{
+	struct cpu_preserved_pcpu_ser *ser = cpu_preserved_get_pcpu_ser(cpu);
+
+	if (!ser)
+		return false;
+
+	cpu_preserved_inval(ser);
+	return READ_ONCE(ser->workload) != CPU_PRESERVED_WORKLOAD;
+}
+EXPORT_SYMBOL_GPL(cpu_preserved_should_exit);
+
+/**
+ * cpu_preserved_attach_workload - Attach & start workload execution on core
+ * @cpu: Logical CPU identifier.
+ * @entry_fn: Workload callback to execute repeatedly on the physical core.
+ * @data: Opaque argument passed to @entry_fn.
+ *
+ * Transitions @cpu from idle parking to executing @entry_fn(@data) on the
+ * physical core, and kicks the CPU to begin execution immediately.
+ *
+ * Return: 0 on success, -EINVAL if @cpu is invalid, -ENODEV if not preserved,
+ * or -EBUSY if a workload is already attached.
+ */
+int cpu_preserved_attach_workload(int cpu,
+				  void (*entry_fn)(void *data), void *data)
+{
+	struct cpu_preserved_pcpu_ser *ser;
+	struct cpu_preserved_pcpu *pcpu;
+
+	if ((unsigned int)cpu >= nr_cpu_ids)
+		return -EINVAL;
+
+	mutex_lock(&cpu_preserved_lock);
+	if (!cpumask_test_cpu(cpu, &cpu_preserved_outgoing.mask)) {
+		mutex_unlock(&cpu_preserved_lock);
+		return -ENODEV;
+	}
+
+	ser = &cpu_preserved_outgoing.pcpus_ser[cpu];
+	pcpu = &cpu_preserved_outgoing.pcpus[cpu];
+	if (ser->workload != CPU_PRESERVED_PARKED || pcpu->entry_fn) {
+		mutex_unlock(&cpu_preserved_lock);
+		return -EBUSY;
+	}
+
+	WRITE_ONCE(pcpu->entry_data, data);
+	WRITE_ONCE(pcpu->entry_fn, entry_fn);
+	WRITE_ONCE(ser->workload, CPU_PRESERVED_WORKLOAD);
+
+	cpu_preserved_clean(pcpu);
+	cpu_preserved_clean(ser);
+
+	arch_cpu_preserved_kick(cpu);
+	mutex_unlock(&cpu_preserved_lock);
+	return 0;
+}
+EXPORT_SYMBOL_GPL(cpu_preserved_attach_workload);
+
+/**
+ * cpu_preserved_detach_workload - Detach workload and return core to idle park
+ * @cpu: Logical CPU identifier.
+ *
+ * Clears any attached workload on @cpu, returning the core to the default
+ * idle parking loop.
+ *
+ * Return: 0 on success, -EINVAL if @cpu is invalid, or -ENODEV if
+ * not preserved.
+ */
+int cpu_preserved_detach_workload(int cpu)
+{
+	struct cpu_preserved_pcpu_ser *ser;
+	struct cpu_preserved_pcpu *pcpu;
+
+	if ((unsigned int)cpu >= nr_cpu_ids)
+		return -EINVAL;
+
+	mutex_lock(&cpu_preserved_lock);
+	if (!cpumask_test_cpu(cpu, &cpu_preserved_mask)) {
+		mutex_unlock(&cpu_preserved_lock);
+		return -ENODEV;
+	}
+
+	ser = cpu_preserved_get_pcpu_ser(cpu);
+	pcpu = cpu_preserved_get_pcpu(cpu);
+	if (!ser) {
+		mutex_unlock(&cpu_preserved_lock);
+		return -ENODEV;
+	}
+
+	if (READ_ONCE(ser->workload) == CPU_PRESERVED_WORKLOAD) {
+		WRITE_ONCE(ser->workload, CPU_PRESERVED_PARKED);
+		cpu_preserved_clean(ser);
+	}
+	if (pcpu) {
+		WRITE_ONCE(pcpu->entry_fn, NULL);
+		WRITE_ONCE(pcpu->entry_data, NULL);
+		cpu_preserved_clean(pcpu);
+	}
+
+	arch_cpu_preserved_kick(cpu);
+	mutex_unlock(&cpu_preserved_lock);
+	return 0;
+}
+EXPORT_SYMBOL_GPL(cpu_preserved_detach_workload);
+
+/**
+ * cpu_preserved_set_workload_context - Set workload context and root page table
+ * @cpu: Logical CPU identifier.
+ * @ctx: Opaque owning workload context pointer.
+ * @pgd_pa: Physical address of workload root page table (or 0 for default).
+ */
+void cpu_preserved_set_workload_context(int cpu, void *ctx, phys_addr_t pgd_pa)
+{
+	struct cpu_preserved_stack_context *sctx;
+	struct cpu_preserved_pcpu *pcpu;
+
+	if (cpu < 0 || cpu >= nr_cpu_ids)
+		return;
+
+	mutex_lock(&cpu_preserved_lock);
+	pcpu = cpu_preserved_get_pcpu(cpu);
+	sctx = cpu_preserved_stack_va(cpu);
+	/*
+	 * Validate the signature before writing through it.  The read side
+	 * (cpu_preserved_get_stack_context()) has always done this; this path
+	 * did not, so a stale or not-yet-initialised stack_pa would have been
+	 * scribbled over.
+	 */
+	if (sctx && sctx->magic == CPU_PRESERVED_STACK_MAGIC) {
+		sctx->workload_context = (u64)(uintptr_t)ctx;
+		sctx->session_pgd_pa = pgd_pa;
+		pcpu->pgd_pa = pgd_pa;
+	}
+	mutex_unlock(&cpu_preserved_lock);
+}
+EXPORT_SYMBOL_GPL(cpu_preserved_set_workload_context);
+
+#define CPU_WAIT_DEAD_TIMEOUT_US	20000000
+#define CPU_WAIT_DEAD_STEP_US		100
+#define CPU_WAIT_DEAD_KICK_STEPS	50
+
+/**
+ * cpu_wait_dead - Wait for a preserved CPU to exit the park loop and power down
+ * @cpu: Logical CPU identifier.
+ *
+ * Polls the KHO-preserved per-CPU state block until @cpu observes
+ * %CPU_PRESERVED_EXITING, leaves cpu_preserved_park_loop(), and publishes
+ * %CPU_PRESERVED_DEAD, periodically sending an IPI kick to wake it from any
+ * low-power wait state.  Once %CPU_PRESERVED_DEAD is observed, invokes
+ * arch_cpu_preserved_wait_dead() to wait for final hardware teardown.
+ *
+ * Return: 0 on success, -ENODEV if @cpu has no preserved state block, or
+ *         -ETIMEDOUT if @cpu did not reach %CPU_PRESERVED_DEAD within
+ *         %CPU_WAIT_DEAD_TIMEOUT_US microseconds.
+ */
+static int cpu_wait_dead(int cpu)
+{
+	struct cpu_preserved_pcpu_ser *ser = cpu_preserved_get_pcpu_ser(cpu);
+	int i;
+
+	if (!ser)
+		return -ENODEV;
+
+	for (i = 0; i < CPU_WAIT_DEAD_TIMEOUT_US / CPU_WAIT_DEAD_STEP_US; i++) {
+		cpu_preserved_inval(ser);
+		if (READ_ONCE(ser->workload) == CPU_PRESERVED_DEAD) {
+			arch_cpu_preserved_wait_dead(cpu);
+			return 0;
+		}
+		if (i % CPU_WAIT_DEAD_KICK_STEPS == 0)
+			arch_cpu_preserved_kick(cpu);
+		udelay(CPU_WAIT_DEAD_STEP_US);
+	}
+
+	pr_err("Timed out waiting for preserved cpu %d to stop (workload=%u)\n",
+	       cpu, READ_ONCE(ser->workload));
+	return -ETIMEDOUT;
+}
+
+static void __cpu_preserved_text
+cpu_preserved_run_workload(struct cpu_preserved_pcpu_ser *ser,
+			   struct cpu_preserved_pcpu *pcpu)
+{
+	void (*fn)(void *data);
+	void *arg;
+
+	if (!pcpu)
+		return;
+
+	cpu_preserved_inval(pcpu);
+	fn = READ_ONCE(pcpu->entry_fn);
+	arg = READ_ONCE(pcpu->entry_data);
+	if (fn)
+		fn(arg);
+
+	cpu_preserved_inval(ser);
+	if (cmpxchg(&ser->workload, CPU_PRESERVED_WORKLOAD,
+		    CPU_PRESERVED_PARKED) == CPU_PRESERVED_WORKLOAD)
+		cpu_preserved_clean(ser);
+}
+STACK_FRAME_NON_STANDARD(cpu_preserved_run_workload);
+
+/**
+ * cpu_preserved_park_loop - Generic execution loop for a parked preserved CPU
+ * @cpu: Logical CPU identifier.
+ *
+ * Core execution loop executed on the dedicated preserved stack in
+ * __cpu_preserved_text.  Waits in low-power park state, dispatches attached
+ * workload callbacks, and exits when the CPU is unpreserved and reclaimed.
+ */
+void __cpu_preserved_text cpu_preserved_park_loop(int cpu)
+{
+	struct cpu_preserved_pcpu_ser *ser = cpu_preserved_get_pcpu_ser(cpu);
+	struct cpu_preserved_pcpu *pcpu = cpu_preserved_get_pcpu(cpu);
+
+	if (!ser)
+		return;
+
+	WRITE_ONCE(ser->workload, CPU_PRESERVED_PARKED);
+	cpu_preserved_clean(ser);
+
+	arch_cpu_preserved_park_init(cpu);
+
+	for (;;) {
+		cpu_preserved_inval(ser);
+		switch (READ_ONCE(ser->workload)) {
+		case CPU_PRESERVED_EXITING:
+		case CPU_PRESERVED_DEAD:
+			WRITE_ONCE(ser->workload, CPU_PRESERVED_DEAD);
+			cpu_preserved_clean(ser);
+			return;
+		case CPU_PRESERVED_WORKLOAD:
+			cpu_preserved_run_workload(ser, pcpu);
+			break;
+		default:
+			arch_cpu_preserved_park_wait();
+			break;
+		}
+	}
+}
+EXPORT_SYMBOL_GPL(cpu_preserved_park_loop);
+STACK_FRAME_NON_STANDARD(cpu_preserved_park_loop);
+
+/**
+ * cpu_preserved_park - Main execution and parking loop for a preserved CPU
+ * @cpu: Logical CPU identifier of the calling core.
+ *
+ * Called on the physical CPU being offlined/preserved. Enters a dedicated
+ * low-power parking loop in preserved memory, repeatedly executing any
+ * attached workload callback, until signaled to exit upon unpreservation.
+ */
+void cpu_preserved_park(int cpu)
+{
+	void *stack = cpu_preserved_stack_va(cpu);
+
+	if (stack) {
+		unsigned long top_of_stack = (unsigned long)stack +
+			CPU_PRESERVED_STACK_SIZE - CPU_PRESERVED_STACK_HEADROOM;
+		arch_cpu_preserved_park_on_stack(cpu, top_of_stack);
+	} else {
+		cpu_preserved_park_loop(cpu);
+		arch_cpu_preserved_park_finish(cpu);
+	}
+}
+EXPORT_SYMBOL_GPL(cpu_preserved_park);
+
 static void cpu_preserved_free_stack(phys_addr_t stack_pa, bool is_incoming)
 {
 	if (stack_pa)
@@ -346,6 +1199,106 @@ static void __cpu_unpreserve_locked(unsigned int cpu)
 	cpu_preserved_sync_global_ser();
 }
 
+static int cpu_preserved_init_outgoing(void)
+{
+	struct cpu_preserved_state *outgoing = &cpu_preserved_outgoing;
+	size_t ser_sz = sizeof(*outgoing->pcpus_ser) * nr_cpu_ids;
+	int ret;
+
+	if (outgoing->pcpus_ser)
+		return 0;
+
+	ret = cpu_preserved_init_runtime_buffer();
+	if (ret)
+		return ret;
+
+	outgoing->pcpus = kcalloc(nr_cpu_ids, sizeof(*outgoing->pcpus),
+				  GFP_KERNEL);
+	if (!outgoing->pcpus)
+		return -ENOMEM;
+
+	outgoing->pcpus_ser = kho_alloc_preserve(ser_sz);
+	if (IS_ERR(outgoing->pcpus_ser)) {
+		ret = PTR_ERR(outgoing->pcpus_ser);
+		kfree(outgoing->pcpus);
+		outgoing->pcpus = NULL;
+		outgoing->pcpus_ser = NULL;
+		return ret;
+	}
+	memset(outgoing->pcpus_ser, 0, ser_sz);
+
+	WRITE_ONCE(cpu_preserved_pcpus_va, outgoing->pcpus_ser);
+	WRITE_ONCE(cpu_preserved_host_pcpus_va, outgoing->pcpus);
+
+	cpu_preserved_map_buffer(outgoing->pcpus_ser, ser_sz);
+	cpu_preserved_map_buffer(outgoing->pcpus,
+				 sizeof(*outgoing->pcpus) * nr_cpu_ids);
+
+	cpu_preserved_clean(&cpu_preserved_pcpus_va);
+	cpu_preserved_clean(&cpu_preserved_host_pcpus_va);
+
+	return 0;
+}
+
+static int cpu_preserve(unsigned int cpu)
+{
+	struct cpu_preserved_state *outgoing = &cpu_preserved_outgoing;
+	struct cpu_preserved_stack_context *sctx;
+	struct cpu_preserved_pcpu_ser *ser;
+	struct cpu_preserved_pcpu *pcpu;
+	void *stack;
+	int ret;
+
+	stack = kho_alloc_preserve(CPU_PRESERVED_STACK_SIZE);
+	if (IS_ERR(stack))
+		return PTR_ERR(stack);
+
+	sctx = stack;
+	sctx->magic = CPU_PRESERVED_STACK_MAGIC;
+	sctx->cpu = cpu;
+
+	scoped_guard(mutex, &cpu_preserved_lock) {
+		if (cpu_is_preserved(cpu)) {
+			kho_unpreserve_free(stack);
+			return -EBUSY;
+		}
+
+		ret = cpu_preserved_init_outgoing();
+		if (ret) {
+			kho_unpreserve_free(stack);
+			return ret;
+		}
+
+		cpumask_set_cpu(cpu, &outgoing->mask);
+		cpumask_set_cpu(cpu, &cpu_preserved_mask);
+		cpu_preserved_clean(&cpu_preserved_mask);
+
+		ser = &outgoing->pcpus_ser[cpu];
+		pcpu = &outgoing->pcpus[cpu];
+		WRITE_ONCE(ser->workload, CPU_PRESERVED_PARKED);
+		pcpu->stack_pa = virt_to_phys(stack);
+		cpu_preserved_map_buffer(stack, CPU_PRESERVED_STACK_SIZE);
+		pcpu->pgd_pa = cpu_preserved_transition_as->pgd_pa;
+		WRITE_ONCE(pcpu->entry_fn, NULL);
+		WRITE_ONCE(pcpu->entry_data, NULL);
+		cpu_preserved_sync_global_ser();
+	}
+
+	if (cpu_online(cpu)) {
+		ret = remove_cpu(cpu);
+		if (ret < 0) {
+			pr_err("Failed to offline preserved cpu %u: %d\n",
+			       cpu, ret);
+			scoped_guard(mutex, &cpu_preserved_lock)
+				__cpu_unpreserve_locked(cpu);
+			return ret;
+		}
+	}
+
+	set_cpu_present(cpu, false);
+	return 0;
+}
+
 /**
  * cpu_unpreserve - Unpreserve a physical CPU and restore it to online state
  * @cpu: Logical CPU identifier.
@@ -488,6 +1441,7 @@ static int cpu_preserved_flb_retrieve(struct liveupdate_flb_op_args *argp)
 
 static void cpu_preserved_flb_finish(struct liveupdate_flb_op_args *argp)
 {
+	struct cpu_preserved_as_ser *trans_as;
 	struct cpu_preserved_global_ser *ser;
 
 	if (!argp->obj)
@@ -495,6 +1449,10 @@ static void cpu_preserved_flb_finish(struct liveupdate_flb_op_args *argp)
 
 	ser = argp->obj;
 
+	trans_as = KHOSER_LOAD_PTR(ser->transition_as);
+	if (trans_as)
+		cpu_preserved_as_destroy(cpu_preserved_as_adopt(trans_as));
+
 	scoped_guard(mutex, &cpu_preserved_lock) {
 		if (cpu_preserved_incoming.pcpus_ser) {
 			kho_restore_free(cpu_preserved_incoming.pcpus_ser);
-- 
2.55.0.1082.g2b9226bbc0-goog

[RFC PATCH 18/46] liveupdate: cpu_preserve: Add LUO file handler for preserved physical CPUs

[Pasha Tatashin]

Implement the LUO file handler (cpu_preserve_handler) and early boot
incoming CPU adoption for /sys/devices/system/cpu/cpuX/preserve fds.

Signed-off-by: Pasha Tatashin <pasha.tatashin@soleen.com>
---
 kernel/liveupdate/cpu_preserve.c | 189 ++++++++++++++++++++++++++++++-
 1 file changed, 185 insertions(+), 4 deletions(-)

diff --git a/kernel/liveupdate/cpu_preserve.c b/kernel/liveupdate/cpu_preserve.c
index 2f92dfee82b1..9a039ba9e912 100644
--- a/kernel/liveupdate/cpu_preserve.c
+++ b/kernel/liveupdate/cpu_preserve.c
@@ -837,8 +837,10 @@ void __cpu_preserved_text cpu_preserved_set_dead(int cpu)
 {
 	struct cpu_preserved_pcpu_ser *ser = cpu_preserved_get_pcpu_ser(cpu);
 
-	if (ser)
-		WRITE_ONCE(ser->workload, CPU_PRESERVED_DEAD);
+	if (ser) {
+		smp_store_release(&ser->workload, CPU_PRESERVED_DEAD);
+		cpu_preserved_clean(ser);
+	}
 }
 EXPORT_SYMBOL_GPL(cpu_preserved_set_dead);
 
@@ -1097,8 +1099,6 @@ void __cpu_preserved_text cpu_preserved_park_loop(int cpu)
 		switch (READ_ONCE(ser->workload)) {
 		case CPU_PRESERVED_EXITING:
 		case CPU_PRESERVED_DEAD:
-			WRITE_ONCE(ser->workload, CPU_PRESERVED_DEAD);
-			cpu_preserved_clean(ser);
 			return;
 		case CPU_PRESERVED_WORKLOAD:
 			cpu_preserved_run_workload(ser, pcpu);
@@ -1131,6 +1131,7 @@ void cpu_preserved_park(int cpu)
 	} else {
 		cpu_preserved_park_loop(cpu);
 		arch_cpu_preserved_park_finish(cpu);
+		cpu_preserved_set_dead(cpu);
 	}
 }
 EXPORT_SYMBOL_GPL(cpu_preserved_park);
@@ -1498,6 +1499,171 @@ static struct liveupdate_flb cpu_preserved_flb = {
 	.compatible = CPU_PRESERVED_LUO_FLB_COMPATIBLE,
 };
 
+/*
+ * LUO File Handler Callbacks for /sys/devices/system/cpu/cpu<N>/preserve
+ */
+static int file_to_cpu(struct file *file, unsigned int *cpup)
+{
+	struct dentry *dentry, *parent;
+	unsigned int cpu;
+
+	if (!file || !file->f_path.dentry)
+		return -EINVAL;
+
+	if (file_inode(file)->i_sb->s_magic != SYSFS_MAGIC)
+		return -EINVAL;
+
+	dentry = file->f_path.dentry;
+	if (strcmp(dentry->d_name.name, "preserve"))
+		return -EINVAL;
+
+	parent = dentry->d_parent;
+	if (!parent || sscanf(parent->d_name.name, "cpu%u", &cpu) != 1)
+		return -EINVAL;
+
+	if (cpu >= nr_cpu_ids || !cpu_possible(cpu) ||
+	    !cpu_is_hotpluggable(cpu)) {
+		return -EINVAL;
+	}
+
+	*cpup = cpu;
+	return 0;
+}
+
+static bool cpu_preserve_can_preserve(struct liveupdate_file_handler *handler,
+				      struct file *file)
+{
+	unsigned int cpu;
+
+	return IS_ENABLED(CONFIG_LIVEUPDATE_ONCORE) && file_to_cpu(file, &cpu) == 0;
+}
+
+static int cpu_preserve_preserve(struct liveupdate_file_op_args *args)
+{
+	struct cpu_preserved_file_ser *fser;
+	unsigned int cpu;
+	int ret;
+
+	ret = file_to_cpu(args->file, &cpu);
+	if (ret)
+		return ret;
+
+	ret = cpu_preserve(cpu);
+	if (ret)
+		return ret;
+
+	fser = kho_alloc_preserve(sizeof(*fser));
+	if (IS_ERR(fser)) {
+		cpu_unpreserve(cpu);
+		return PTR_ERR(fser);
+	}
+
+	memset(fser, 0, sizeof(*fser));
+	fser->cpu = cpu;
+
+	scoped_guard(mutex, &cpu_preserved_lock)
+		fser->stack_pa = cpu_preserved_outgoing.pcpus[cpu].stack_pa;
+
+	args->serialized_data = virt_to_phys(fser);
+	return 0;
+}
+
+static void cpu_preserve_unpreserve(struct liveupdate_file_op_args *args)
+{
+	struct cpu_preserved_file_ser *fser;
+	unsigned int cpu;
+
+	if (!args->serialized_data)
+		return;
+
+	fser = phys_to_virt(args->serialized_data);
+	cpu = fser->cpu;
+
+	cpu_unpreserve(cpu);
+
+	kho_unpreserve_free(fser);
+}
+
+static void cpu_preserve_restore_incoming_cpu(struct liveupdate_session *session,
+					      struct cpu_preserved_file_ser *fser)
+{
+	unsigned int cpu = fser->cpu;
+
+	scoped_guard(mutex, &cpu_preserved_lock) {
+		cpumask_set_cpu(cpu, &cpu_preserved_incoming.mask);
+		cpumask_set_cpu(cpu, &cpu_preserved_mask);
+
+		if (!cpu_preserved_incoming.pcpus) {
+			cpu_preserved_incoming.pcpus =
+				kcalloc(nr_cpu_ids,
+					sizeof(*cpu_preserved_incoming.pcpus),
+					GFP_KERNEL);
+			WRITE_ONCE(cpu_preserved_host_pcpus_va,
+				   cpu_preserved_incoming.pcpus);
+			cpu_preserved_clean(&cpu_preserved_host_pcpus_va);
+		}
+
+		if (cpu_preserved_incoming.pcpus)
+			cpu_preserved_incoming.pcpus[cpu].stack_pa = fser->stack_pa;
+
+		cpu_preserved_clean(&cpu_preserved_mask);
+	}
+}
+
+static int cpu_preserve_retrieve(struct liveupdate_file_op_args *args)
+{
+	struct cpu_preserved_file_ser *fser;
+	struct file *file;
+	char path[64];
+
+	if (!args->serialized_data)
+		return -EINVAL;
+
+	fser = phys_to_virt(args->serialized_data);
+
+	snprintf(path, sizeof(path),
+		 "/sys/devices/system/cpu/cpu%u/preserve", fser->cpu);
+	file = filp_open(path, O_RDONLY, 0);
+	if (IS_ERR(file))
+		return PTR_ERR(file);
+
+	args->file = file;
+	cpu_preserve_restore_incoming_cpu(args->session, fser);
+	cpu_preserved_detach_workload(fser->cpu);
+
+	return 0;
+}
+
+static void cpu_preserve_finish(struct liveupdate_file_op_args *args)
+{
+	struct cpu_preserved_file_ser *fser;
+
+	if (!args->serialized_data)
+		return;
+
+	fser = phys_to_virt(args->serialized_data);
+	if (args->retrieve_status <= 0)
+		cpu_preserve_restore_incoming_cpu(args->session, fser);
+
+	cpu_unpreserve(fser->cpu);
+
+	kho_restore_free(fser);
+}
+
+static const struct liveupdate_file_ops cpu_preserve_file_ops = {
+	.can_preserve = cpu_preserve_can_preserve,
+	.preserve     = cpu_preserve_preserve,
+	.retrieve     = cpu_preserve_retrieve,
+	.unpreserve   = cpu_preserve_unpreserve,
+	.finish       = cpu_preserve_finish,
+	.owner        = THIS_MODULE,
+};
+
+static struct liveupdate_file_handler cpu_preserve_handler = {
+	.ops        = &cpu_preserve_file_ops,
+	.compatible = CPU_PRESERVED_LUO_FH_COMPATIBLE,
+};
+
 static int cpu_preserve_reboot_notify(struct notifier_block *nb,
 				      unsigned long action, void *data)
 {
@@ -1555,6 +1721,21 @@ static int __init cpu_preserve_early_init(void)
 	cpu_preserved_incoming.pcpus_ser = NULL;
 	cpu_preserved_global_ser = NULL;
 
+	err = liveupdate_register_file_handler(&cpu_preserve_handler);
+	if (err && err != -EOPNOTSUPP) {
+		pr_err("Could not register cpu_preserve file handler: %pe\n",
+		       ERR_PTR(err));
+		return err;
+	}
+
+	err = liveupdate_register_flb(&cpu_preserve_handler,
+				      &cpu_preserved_flb);
+	if (err && err != -EOPNOTSUPP) {
+		pr_err("Could not register cpu_preserved FLB: %pe\n",
+		       ERR_PTR(err));
+		return err;
+	}
+
 	/* Retrieve incoming preserved CPUs before secondary CPU bringup */
 	if (liveupdate_enabled())
 		liveupdate_flb_get_incoming(&cpu_preserved_flb, &obj);
-- 
2.55.0.1082.g2b9226bbc0-goog

[RFC PATCH 19/46] x86: liveupdate: Add low-level physical CPU preservation assembly

[Pasha Tatashin]

Add arch/x86/kernel/preserve_cpu.S stack switch routine, preserved linker
sections in vmlinux.lds.S, and SMP stop exclusion for preserved cores.

Signed-off-by: Pasha Tatashin <pasha.tatashin@soleen.com>
---
 arch/x86/include/asm/cpu_preserve.h | 31 +++++++++++++++
 arch/x86/kernel/preserve_cpu.S      | 62 +++++++++++++++++++++++++++++
 arch/x86/kernel/vmlinux.lds.S       |  2 +
 3 files changed, 95 insertions(+)
 create mode 100644 arch/x86/include/asm/cpu_preserve.h
 create mode 100644 arch/x86/kernel/preserve_cpu.S

diff --git a/arch/x86/include/asm/cpu_preserve.h b/arch/x86/include/asm/cpu_preserve.h
new file mode 100644
index 000000000000..969903eb70cb
--- /dev/null
+++ b/arch/x86/include/asm/cpu_preserve.h
@@ -0,0 +1,31 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright (c) 2026, Google LLC.
+ * Pasha Tatashin <pasha.tatashin@soleen.com>
+ */
+#ifndef __ASM_X86_CPU_PRESERVE_H
+#define __ASM_X86_CPU_PRESERVE_H
+
+#include <asm/page_types.h>
+
+#define ARCH_CPU_PRESERVED_STACK_ORDER	THREAD_SIZE_ORDER
+
+#ifdef CONFIG_CC_IS_GCC
+#define ARCH_CPU_PRESERVED_TEXT \
+	__attribute__((indirect_branch("keep"), function_return("keep")))
+#else
+#define ARCH_CPU_PRESERVED_TEXT
+#endif
+
+#ifdef CONFIG_LIVEUPDATE_CPU
+void arch_cpu_preserved_load_desc(void);
+bool arch_cpu_preserved_is_active(void);
+void x86_preserved_iret_stub(void);
+void x86_preserved_iret_err_stub(void);
+void x86_preserved_apic_eoi_stub(void);
+#else
+static inline void arch_cpu_preserved_load_desc(void) {}
+static inline bool arch_cpu_preserved_is_active(void) { return false; }
+#endif
+
+#endif /* __ASM_X86_CPU_PRESERVE_H */
diff --git a/arch/x86/kernel/preserve_cpu.S b/arch/x86/kernel/preserve_cpu.S
new file mode 100644
index 000000000000..8acb98d46f90
--- /dev/null
+++ b/arch/x86/kernel/preserve_cpu.S
@@ -0,0 +1,62 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright (c) 2026, Google LLC.
+ * Pasha Tatashin <pasha.tatashin@soleen.com>
+ */
+#include <linux/linkage.h>
+#include <asm/nospec-branch.h>
+#include <asm/unwind_hints.h>
+
+	.section .text.cpu_preserved, "ax"
+
+SYM_CODE_START(x86_preserved_iret_err_stub)
+	UNWIND_HINT_IRET_REGS
+	ENDBR
+	addq	$8, %rsp
+	iretq
+SYM_CODE_END(x86_preserved_iret_err_stub)
+
+SYM_CODE_START(x86_preserved_iret_stub)
+	UNWIND_HINT_IRET_REGS
+	ENDBR
+	iretq
+SYM_CODE_END(x86_preserved_iret_stub)
+
+SYM_CODE_START(x86_preserved_apic_eoi_stub)
+	UNWIND_HINT_IRET_REGS
+	ENDBR
+	pushq	%rax
+	pushq	%rcx
+	pushq	%rdx
+	/* x2APIC mode: MSR 0x80b (APIC_EOI) = 0 */
+	movl	$0x80b, %ecx
+	xorl	%eax, %eax
+	xorl	%edx, %edx
+	wrmsr
+	popq	%rdx
+	popq	%rcx
+	popq	%rax
+	iretq
+SYM_CODE_END(x86_preserved_apic_eoi_stub)
+
+/*
+ * arch_cpu_preserved_call_on_stack - Switch stack and call function
+ * @cpu: CPU ID (%rdi)
+ * @stack: Target stack top (%rsi)
+ * @fn: Function pointer to invoke (%rdx)
+ */
+SYM_CODE_START(arch_cpu_preserved_call_on_stack)
+	pushq	%rbp
+	movq	%rsp, %rbp
+	pushq	%r12
+
+	movq	%rsp, %r12	/* save old stack pointer */
+	movq	%rsi, %rsp	/* switch to target stack */
+
+	call	*%rdx		/* call fn(cpu) */
+
+	movq	%r12, %rsp	/* restore old stack pointer */
+	popq	%r12
+	leave
+	ret
+SYM_CODE_END(arch_cpu_preserved_call_on_stack)
diff --git a/arch/x86/kernel/vmlinux.lds.S b/arch/x86/kernel/vmlinux.lds.S
index 2438b89a4620..9b8b70057bcf 100644
--- a/arch/x86/kernel/vmlinux.lds.S
+++ b/arch/x86/kernel/vmlinux.lds.S
@@ -155,6 +155,7 @@ SECTIONS
 #endif
 		ALIGN_ENTRY_TEXT_END
 
+		CPU_PRESERVED_TEXT
 		TEXT_TEXT
 		SCHED_TEXT
 		LOCK_TEXT
@@ -194,6 +195,7 @@ SECTIONS
 #endif
 
 		PAGE_ALIGNED_DATA(PAGE_SIZE)
+		CPU_PRESERVED_DATA
 
 		CACHE_HOT_DATA(L1_CACHE_BYTES)
 
-- 
2.55.0.1082.g2b9226bbc0-goog

[RFC PATCH 20/46] x86: liveupdate: Add physical CPU preservation context and page table support

[Pasha Tatashin]

Implement x86 physical CPU preservation context management, APIC wakeup,
and isolated transition page tables in arch/x86/kernel/cpu_preserve.c.

Signed-off-by: Pasha Tatashin <pasha.tatashin@soleen.com>
---
 arch/x86/Kconfig                    |   1 +
 arch/x86/include/asm/cpu_preserve.h |   5 +-
 arch/x86/kernel/Makefile            |   7 +
 arch/x86/kernel/cpu_preserve.c      | 432 ++++++++++++++++++++++++++++
 4 files changed, 444 insertions(+), 1 deletion(-)
 create mode 100644 arch/x86/kernel/cpu_preserve.c

diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig
index 15fd9ec5ecac..c57b54b2b2b0 100644
--- a/arch/x86/Kconfig
+++ b/arch/x86/Kconfig
@@ -71,6 +71,7 @@ config X86
 	select ARCH_ENABLE_MEMORY_HOTPLUG if X86_64
 	select ARCH_ENABLE_SPLIT_PMD_PTLOCK if (PGTABLE_LEVELS > 2) && (X86_64 || X86_PAE)
 	select ARCH_HAS_PMD_SOFTLEAVES if X86_64 && TRANSPARENT_HUGEPAGE
+	select ARCH_SUPPORTS_LIVEUPDATE_CPU if LIVEUPDATE
 	select ARCH_HAS_ACPI_TABLE_UPGRADE	if ACPI
 	select ARCH_HAS_CPU_ATTACK_VECTORS	if CPU_MITIGATIONS
 	select ARCH_HAS_CACHE_LINE_SIZE
diff --git a/arch/x86/include/asm/cpu_preserve.h b/arch/x86/include/asm/cpu_preserve.h
index 969903eb70cb..83f72d39a44b 100644
--- a/arch/x86/include/asm/cpu_preserve.h
+++ b/arch/x86/include/asm/cpu_preserve.h
@@ -10,9 +10,12 @@
 
 #define ARCH_CPU_PRESERVED_STACK_ORDER	THREAD_SIZE_ORDER
 
-#ifdef CONFIG_CC_IS_GCC
+#if __has_attribute(indirect_branch) && __has_attribute(function_return)
 #define ARCH_CPU_PRESERVED_TEXT \
 	__attribute__((indirect_branch("keep"), function_return("keep")))
+#elif __has_attribute(indirect_branch)
+#define ARCH_CPU_PRESERVED_TEXT \
+	__attribute__((indirect_branch("keep")))
 #else
 #define ARCH_CPU_PRESERVED_TEXT
 #endif
diff --git a/arch/x86/kernel/Makefile b/arch/x86/kernel/Makefile
index 31f46fd00527..6e238444039d 100644
--- a/arch/x86/kernel/Makefile
+++ b/arch/x86/kernel/Makefile
@@ -78,6 +78,13 @@ obj-$(CONFIG_IRQ_WORK)  += irq_work.o
 obj-y			+= probe_roms.o
 obj-$(CONFIG_X86_32)	+= sys_ia32.o
 obj-$(CONFIG_IA32_EMULATION)	+= sys_ia32.o signal_32.o
+KASAN_SANITIZE_cpu_preserve.o := n
+KCSAN_SANITIZE_cpu_preserve.o := n
+UBSAN_SANITIZE_cpu_preserve.o := n
+KCOV_INSTRUMENT_cpu_preserve.o := n
+CFLAGS_REMOVE_cpu_preserve.o = $(CC_FLAGS_FTRACE)
+CFLAGS_cpu_preserve.o += -fno-stack-protector $(call cc-option,-ftrivial-auto-var-init=uninitialized) $(call cc-option,-fno-jump-tables)
+obj-$(CONFIG_LIVEUPDATE_CPU)	+= cpu_preserve.o preserve_cpu.o
 obj-$(CONFIG_X86_64)	+= sys_x86_64.o
 obj-$(CONFIG_X86_ESPFIX64)	+= espfix_64.o
 obj-$(CONFIG_SYSFS)	+= ksysfs.o
diff --git a/arch/x86/kernel/cpu_preserve.c b/arch/x86/kernel/cpu_preserve.c
new file mode 100644
index 000000000000..15327bf469db
--- /dev/null
+++ b/arch/x86/kernel/cpu_preserve.c
@@ -0,0 +1,432 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Copyright (c) 2026, Google LLC.
+ * Pasha Tatashin <pasha.tatashin@soleen.com>
+ *
+ * Architecture specific CPU preservation support for x86.
+ */
+#include <linux/cpu_preserve.h>
+#include <linux/kexec_handover.h>
+#include <linux/mm.h>
+#include <linux/nospec.h>
+#include <linux/objtool.h>
+#include <linux/sched/mm.h>
+
+#include <asm/apic.h>
+#include <linux/cacheflush.h>
+#include <linux/cpufeature.h>
+#include <asm/desc.h>
+#include <asm/fixmap.h>
+#include <asm/msr.h>
+#include <linux/pgtable.h>
+#include <asm/set_memory.h>
+#include <linux/smp.h>
+#include <asm/tlbflush.h>
+#include <asm/trapnr.h>
+#include <asm/init.h>
+
+static u32 x86_preserved_apicid[NR_CPUS] __cpu_preserved_data = {
+	[0 ... NR_CPUS - 1] = BAD_APICID,
+};
+
+static bool x86_preserved_x2apic __cpu_preserved_data;
+
+/*
+ * Signal or wake up a preserved physical CPU via APIC ICR NMI.
+ */
+void __cpu_preserved_text arch_cpu_preserved_kick(int cpu)
+{
+	u32 apicid;
+	u64 val;
+
+	if ((unsigned int)cpu >= NR_CPUS || !cpu_is_preserved(cpu))
+		return;
+
+	cpu = array_index_nospec(cpu, NR_CPUS);
+	apicid = x86_preserved_apicid[cpu];
+	if (apicid == BAD_APICID)
+		return;
+
+	val = ((u64)apicid << 32) | APIC_DM_NMI;
+	native_wrmsrq(APIC_BASE_MSR + (APIC_ICR >> 4), val);
+}
+
+/*
+ * Low-power wait in parking loop.
+ */
+void __cpu_preserved_text arch_cpu_preserved_park_wait(void)
+{
+	cpu_relax();
+}
+
+static gate_desc x86_preserved_idt[IDT_ENTRIES] __cpu_preserved_data __aligned(PAGE_SIZE);
+static struct desc_ptr x86_preserved_idt_desc __cpu_preserved_data;
+
+static struct desc_struct x86_preserved_gdt[GDT_ENTRIES] __cpu_preserved_data __aligned(PAGE_SIZE);
+static struct desc_ptr x86_preserved_gdt_desc __cpu_preserved_data;
+static bool x86_preserved_has_svm __cpu_preserved_data;
+
+/*
+ * Initialize the preserved IDT with stub handlers.
+ *
+ * Vectors 0..31 are x86 architecture exceptions/traps:
+ * - Exceptions with error codes (#DF, #TS, #NP, #SS, #GP, #PF, #AC, #CP, #VC,
+ *   and #SX / vector 30) push an 8-byte error code onto the stack before the
+ *   IRET frame. They must use iret_err_handler (x86_preserved_iret_err_stub)
+ *   to pop the error code before iretq.
+ * - Other exceptions push no error code and return directly via iret_handler
+ *   (x86_preserved_iret_stub).
+ *
+ * Vectors >= FIRST_EXTERNAL_VECTOR (32) are device and IPI interrupts:
+ * - External interrupts push no error code but require an APIC EOI before
+ *   returning, handled by eoi_handler (x86_preserved_apic_eoi_stub), so the
+ *   Local APIC does not block subsequent interrupts.
+ */
+static void init_preserved_idt(void)
+{
+	unsigned long iret_err_handler = (unsigned long)&x86_preserved_iret_err_stub;
+	unsigned long eoi_handler = (unsigned long)&x86_preserved_apic_eoi_stub;
+	unsigned long iret_handler = (unsigned long)&x86_preserved_iret_stub;
+	int v;
+
+	for (v = 0; v < IDT_ENTRIES; v++) {
+		bool has_err = (v == X86_TRAP_DF ||
+				(v >= X86_TRAP_TS && v <= X86_TRAP_PF) ||
+				v == X86_TRAP_AC || v == X86_TRAP_CP ||
+				v == X86_TRAP_VC || v == 30); /* 30: #SX */
+		unsigned long handler = (v >= FIRST_EXTERNAL_VECTOR) ? eoi_handler :
+			(has_err ? iret_err_handler : iret_handler);
+
+		pack_gate(&x86_preserved_idt[v], GATE_INTERRUPT, handler, 0,
+			  0, __KERNEL_CS);
+	}
+	x86_preserved_idt_desc.size = sizeof(x86_preserved_idt) - 1;
+	x86_preserved_idt_desc.address = (unsigned long)&x86_preserved_idt[0];
+	cpu_preserved_clean(&x86_preserved_idt);
+	cpu_preserved_clean(&x86_preserved_idt_desc);
+}
+
+static void init_preserved_gdt(void)
+{
+	struct desc_struct *gdt;
+	int i;
+
+	gdt = get_current_gdt_rw();
+	for (i = 0; i < GDT_ENTRIES; i++)
+		x86_preserved_gdt[i] = gdt[i];
+	x86_preserved_gdt_desc.size = GDT_SIZE - 1;
+	x86_preserved_gdt_desc.address = (unsigned long)&x86_preserved_gdt[0];
+	cpu_preserved_clean(&x86_preserved_gdt);
+	cpu_preserved_clean(&x86_preserved_gdt_desc);
+}
+
+void __cpu_preserved_text arch_cpu_preserved_load_desc(void)
+{
+	native_load_gdt(&x86_preserved_gdt_desc);
+	native_load_idt(&x86_preserved_idt_desc);
+}
+EXPORT_SYMBOL_GPL(arch_cpu_preserved_load_desc);
+
+/*
+ * Disables local interrupts on the physical core and loads preserved IDT and GDT.
+ */
+void __cpu_preserved_text arch_cpu_preserved_park_init(int cpu)
+{
+	u32 spiv;
+
+	local_irq_disable();
+	arch_cpu_preserved_load_desc();
+
+	if (cpu >= 0 && cpu < CONFIG_NR_CPUS && x86_preserved_x2apic) {
+		x86_preserved_apicid[cpu] =
+			(u32)native_rdmsrq(APIC_BASE_MSR + (APIC_ID >> 4));
+		cpu_preserved_clean(&x86_preserved_apicid[cpu]);
+	}
+
+	spiv = (u32)native_rdmsrq(APIC_BASE_MSR + (APIC_SPIV >> 4));
+	if (!(spiv & APIC_SPIV_APIC_ENABLED)) {
+		spiv |= APIC_SPIV_APIC_ENABLED;
+		native_wrmsrq(APIC_BASE_MSR + (APIC_SPIV >> 4), spiv);
+	}
+}
+
+void arch_cpu_preserved_early_init(void)
+{
+	int i;
+
+	x86_preserved_has_svm = boot_cpu_has(X86_FEATURE_SVM);
+	x86_preserved_x2apic = x2apic_mode != 0;
+	cpu_preserved_clean(&x86_preserved_x2apic);
+
+	for (i = 0; i < nr_cpu_ids; i++) {
+		u32 apicid = cpu_physical_id(i);
+
+		if (apicid == BAD_APICID)
+			apicid = cpuid_to_apicid[i];
+		x86_preserved_apicid[i] = apicid;
+	}
+	cpu_preserved_clean(&x86_preserved_apicid);
+
+	init_preserved_idt();
+	init_preserved_gdt();
+}
+EXPORT_SYMBOL_GPL(arch_cpu_preserved_early_init);
+
+/*
+ * Disable hardware virtualization on physical core so INIT is recognized.
+ */
+static void __cpu_preserved_text arch_cpu_preserved_virt_teardown(void)
+{
+	if (__read_cr4() & X86_CR4_VMXE) {
+		asm volatile("1: vmxoff\n\t"
+			     "2:\n\t"
+			     _ASM_EXTABLE(1b, 2b)
+			     : : : "memory", "cc");
+		asm volatile("mov %0, %%cr4" : : "r" (__read_cr4() & ~X86_CR4_VMXE) : "memory");
+	}
+
+	if (x86_preserved_has_svm) {
+		u64 efer = native_rdmsrq(MSR_EFER);
+
+		if (efer & EFER_SVME) {
+			asm volatile("stgi" : : : "memory");
+			native_wrmsrq(MSR_EFER, efer & ~EFER_SVME);
+		}
+	}
+}
+
+/*
+ * Architecture cleanup on park loop exit.
+ */
+void __cpu_preserved_text arch_cpu_preserved_park_finish(int cpu __maybe_unused)
+{
+	arch_cpu_preserved_load_desc();
+	arch_cpu_preserved_virt_teardown();
+}
+
+phys_addr_t __cpu_preserved_data x86_caretaker_pgd_pa;
+EXPORT_SYMBOL_GPL(x86_caretaker_pgd_pa);
+
+bool __cpu_preserved_text arch_cpu_preserved_is_active(void)
+{
+	struct cpu_preserved_stack_context *sctx = cpu_preserved_get_stack_context();
+	unsigned long cr3 = __read_cr3();
+
+	if (sctx && sctx->session_pgd_pa && cr3 == sctx->session_pgd_pa)
+		return true;
+
+	if (x86_caretaker_pgd_pa)
+		return cr3 == x86_caretaker_pgd_pa;
+
+	return false;
+}
+EXPORT_SYMBOL_GPL(arch_cpu_preserved_is_active);
+
+void __cpu_preserved_text arch_cpu_preserved_switch_pgd(phys_addr_t pgd_pa)
+{
+	if (pgd_pa && __read_cr3() != pgd_pa)
+		write_cr3(pgd_pa);
+}
+EXPORT_SYMBOL_GPL(arch_cpu_preserved_switch_pgd);
+
+asmlinkage void arch_cpu_preserved_call_on_stack(int cpu, unsigned long stack,
+						 void (*fn)(int cpu));
+
+static void __cpu_preserved_text arch_cpu_preserved_park_worker(int cpu)
+{
+	struct cpu_preserved_stack_context *sctx = cpu_preserved_get_stack_context();
+	phys_addr_t pgd_pa = 0;
+
+	arch_cpu_preserved_park_init(cpu);
+
+	if (sctx && sctx->session_pgd_pa)
+		pgd_pa = sctx->session_pgd_pa;
+	else
+		pgd_pa = cpu_preserved_get_pgd(cpu);
+
+	if (!pgd_pa)
+		pgd_pa = x86_caretaker_pgd_pa;
+
+	if (pgd_pa)
+		write_cr3(pgd_pa);
+
+	cpu_preserved_park_loop(cpu);
+
+	arch_cpu_preserved_park_finish(cpu);
+	native_irq_disable();
+	cpu_preserved_set_dead(cpu);
+	while (1) {
+		native_irq_disable();
+		asm volatile("hlt");
+	}
+}
+STACK_FRAME_NON_STANDARD(arch_cpu_preserved_park_worker);
+
+static void arch_cpu_preserved_set_max_perf(void)
+{
+	u64 cap;
+
+	/* Intel HWP (Speed Shift): autonomously request maximum performance */
+	if (boot_cpu_has(X86_FEATURE_HWP) &&
+	    !rdmsrq_safe(MSR_HWP_CAPABILITIES, &cap)) {
+		u8 highest = HWP_HIGHEST_PERF(cap);
+
+		if (highest) {
+			wrmsrq_safe(MSR_HWP_REQUEST, HWP_MIN_PERF(highest) |
+				    HWP_MAX_PERF(highest) |
+				    HWP_DESIRED_PERF(highest));
+		}
+	}
+
+	/* Intel Energy Performance Bias: hint for maximum performance */
+	if (boot_cpu_has(X86_FEATURE_EPB))
+		wrmsrq_safe(MSR_IA32_ENERGY_PERF_BIAS, ENERGY_PERF_BIAS_PERFORMANCE);
+
+	/* AMD CPPC: request maximum performance ratio and zero energy preference */
+	if (boot_cpu_has(X86_FEATURE_CPPC)) {
+		wrmsrq_safe(MSR_AMD_CPPC_REQ, AMD_CPPC_MAX_PERF_MASK |
+			    AMD_CPPC_MIN_PERF_MASK | AMD_CPPC_DES_PERF_MASK);
+	}
+}
+
+/*
+ * Switch stack and enter park loop.
+ */
+void arch_cpu_preserved_park_on_stack(int cpu, unsigned long stack_top)
+{
+	if (cpu >= 0 && cpu < CONFIG_NR_CPUS) {
+		if (x86_preserved_x2apic)
+			x86_preserved_apicid[cpu] =
+				(u32)native_rdmsrq(APIC_BASE_MSR + (APIC_ID >> 4));
+		else
+			x86_preserved_apicid[cpu] = cpu_physical_id(cpu);
+		cpu_preserved_clean(&x86_preserved_apicid[cpu]);
+	}
+	arch_cpu_preserved_set_max_perf();
+	arch_cpu_preserved_call_on_stack(cpu, stack_top, arch_cpu_preserved_park_worker);
+}
+EXPORT_SYMBOL_GPL(arch_cpu_preserved_park_on_stack);
+
+/*
+ * Clean data cache for address range. x86 has hardware coherent caches,
+ * so a memory barrier suffices without calling unpreserved external routines.
+ */
+void __cpu_preserved_text arch_cpu_preserved_dcache_clean(unsigned long start, unsigned long end)
+{
+	/* Memory barrier to serialize cache operations on x86 */
+	mb();
+}
+
+/*
+ * Invalidate data cache for address range.
+ */
+void __cpu_preserved_text arch_cpu_preserved_dcache_inval(unsigned long start, unsigned long end)
+{
+	/* Memory barrier to serialize cache operations on x86 */
+	mb();
+}
+
+/**
+ * arch_cpu_preserved_as_map - Populate an isolated page table on x86
+ * @as: Address space to map into.
+ * @pa: Physical address of the range.
+ * @va: Virtual address the range must appear at.
+ * @size: Size of the range in bytes.
+ * @prot: Protection to apply.
+ *
+ * Return: 0 on success, or a negative errno on failure.
+ */
+int arch_cpu_preserved_as_map(struct cpu_preserved_as *as, phys_addr_t pa,
+			      unsigned long va, size_t size, pgprot_t prot)
+{
+	unsigned long offset = va & ~PAGE_MASK;
+	size_t page_size = PAGE_ALIGN(offset + size);
+	unsigned long page_va = va & PAGE_MASK;
+	phys_addr_t page_pa = (pa & PAGE_MASK);
+	struct x86_mapping_info info = {
+		.alloc_pgt_page = cpu_preserved_as_alloc_page,
+		.context = as,
+		.page_flag = pgprot_val(prot),
+		.offset = page_va - page_pa,
+		.force_pte = true,
+	};
+
+	return kernel_ident_mapping_init(&info, as->pgd, page_pa,
+					 page_pa + page_size);
+}
+
+/*
+ * A preserved CPU only ever enters one of these address spaces by loading
+ * CR3, which flushes everything that is not global, and nothing in them is
+ * mapped global.
+ */
+void arch_cpu_preserved_as_flush_tlb(void)
+{
+}
+
+void arch_cpu_preserved_set_transition_as(struct cpu_preserved_as *as)
+{
+	x86_caretaker_pgd_pa = as ? as->pgd_pa : 0;
+}
+
+int arch_cpu_preserved_setup_buffer(struct page *text_page,
+				    unsigned int text_nr_pages,
+				    struct page *data_page,
+				    unsigned int data_nr_pages)
+{
+	unsigned long text_start = (unsigned long)__cpu_preserved_text_start;
+	unsigned long data_start = (unsigned long)__cpu_preserved_data_start;
+	unsigned int i;
+	int ret;
+
+	if (!x2apic_enabled()) {
+		pr_warn("cpu_preserve: x2APIC is required\n");
+		return -EOPNOTSUPP;
+	}
+
+	/* Split kernel large pages into 4K PTEs */
+	ret = set_memory_4k(text_start, text_nr_pages);
+	if (ret)
+		return ret;
+
+	ret = set_memory_4k(data_start, data_nr_pages);
+	if (ret)
+		return ret;
+
+	/* Remap init_mm kernel mappings to point to allocated buffer pages */
+	for (i = 0; i < text_nr_pages; i++) {
+		unsigned int level;
+		pte_t *pte = lookup_address(text_start + i * PAGE_SIZE, &level);
+
+		if (pte && level == PG_LEVEL_4K) {
+			phys_addr_t pa = page_to_phys(text_page) + i * PAGE_SIZE;
+
+			set_pte(pte, pfn_pte(PHYS_PFN(pa), pte_pgprot(*pte)));
+		}
+	}
+
+	for (i = 0; i < data_nr_pages; i++) {
+		unsigned int level;
+		pte_t *pte = lookup_address(data_start + i * PAGE_SIZE, &level);
+
+		if (pte && level == PG_LEVEL_4K) {
+			phys_addr_t pa = page_to_phys(data_page) + i * PAGE_SIZE;
+
+			set_pte(pte, pfn_pte(PHYS_PFN(pa), pte_pgprot(*pte)));
+		}
+	}
+
+	flush_tlb_all();
+
+	/* Ensure preserved GDT, IDT, and arch flags are initialized */
+	arch_cpu_preserved_early_init();
+	init_preserved_idt();
+	init_preserved_gdt();
+	cpu_preserved_clean(&x86_preserved_has_svm);
+
+	return 0;
+}
+
+void arch_cpu_preserved_wait_dead(int cpu)
+{
+}
-- 
2.55.0.1082.g2b9226bbc0-goog

[RFC PATCH 21/46] selftests: liveupdate: Add physical CPU preservation unit tests

[Pasha Tatashin]

Add unit tests exercising CPU preservation sysfs interface:
- Verify /sys/devices/system/cpu/cpu<N>/preserve operations.
- Test CPU offline and preserve sequencing.
- Verify status reporting and error handling.

Signed-off-by: Pasha Tatashin <pasha.tatashin@soleen.com>
---
 tools/testing/selftests/liveupdate/config     |   6 +
 .../testing/selftests/liveupdate/liveupdate.c | 251 ++++++++++++++++++
 2 files changed, 257 insertions(+)

diff --git a/tools/testing/selftests/liveupdate/config b/tools/testing/selftests/liveupdate/config
index 016d009dba13..9458b9c70eeb 100644
--- a/tools/testing/selftests/liveupdate/config
+++ b/tools/testing/selftests/liveupdate/config
@@ -5,8 +5,14 @@ CONFIG_KEXEC_HANDOVER=y
 CONFIG_KEXEC_HANDOVER_ENABLE_DEFAULT=y
 CONFIG_KEXEC_HANDOVER_DEBUGFS=y
 CONFIG_KEXEC_HANDOVER_DEBUG=y
+CONFIG_SMP=y
+CONFIG_HOTPLUG_CPU=y
 CONFIG_LIVEUPDATE=y
 CONFIG_LIVEUPDATE_TEST=y
+CONFIG_LIVEUPDATE_CPU=y
+CONFIG_ACPI=y
+CONFIG_ACPI_PROCESSOR=y
+CONFIG_ACPI_HOTPLUG_CPU=y
 CONFIG_MEMFD_CREATE=y
 CONFIG_TMPFS=y
 CONFIG_SHMEM=y
diff --git a/tools/testing/selftests/liveupdate/liveupdate.c b/tools/testing/selftests/liveupdate/liveupdate.c
index 2dedd5fc2534..b3c9cd7ea653 100644
--- a/tools/testing/selftests/liveupdate/liveupdate.c
+++ b/tools/testing/selftests/liveupdate/liveupdate.c
@@ -530,4 +530,255 @@ TEST_F(liveupdate_device, preserve_many_files)
 	ASSERT_EQ(close(session_fd), 0);
 }
 
+static int find_hotplug_cpu(void)
+{
+	char path[128];
+	int cpu;
+
+	for (cpu = 1; cpu < 256; cpu++) {
+		snprintf(path, sizeof(path), "/sys/devices/system/cpu/cpu%d/preserve", cpu);
+		if (access(path, R_OK) == 0) {
+			snprintf(path, sizeof(path), "/sys/devices/system/cpu/cpu%d/online", cpu);
+			if (access(path, R_OK) == 0)
+				return cpu;
+		}
+	}
+	return -1;
+}
+
+static int read_cpu_online(int cpu)
+{
+	char path[128], buf[16];
+	int fd, val;
+
+	snprintf(path, sizeof(path), "/sys/devices/system/cpu/cpu%d/online", cpu);
+	fd = open(path, O_RDONLY);
+	if (fd < 0)
+		return -1;
+	memset(buf, 0, sizeof(buf));
+	if (read(fd, buf, sizeof(buf) - 1) <= 0) {
+		close(fd);
+		return -1;
+	}
+	close(fd);
+	val = atoi(buf);
+	return val;
+}
+
+static int write_cpu_online(int cpu, int val)
+{
+	char path[128], buf[16];
+	int fd, ret;
+
+	snprintf(path, sizeof(path), "/sys/devices/system/cpu/cpu%d/online", cpu);
+	fd = open(path, O_WRONLY);
+	if (fd < 0)
+		return -errno;
+	snprintf(buf, sizeof(buf), "%d\n", val);
+	ret = write(fd, buf, strlen(buf));
+	if (ret < 0) {
+		ret = -errno;
+		close(fd);
+		return ret;
+	}
+	close(fd);
+	return 0;
+}
+
+/*
+ * Test Case: CPU Preservation Basic
+ *
+ * Verifies that a CPU preserve sysfs file descriptor can be preserved in a
+ * LUO session, offlining the CPU from scheduler and returning it online upon
+ * session teardown.
+ */
+TEST_F(liveupdate_device, cpu_preserve_basic)
+{
+	char path[128];
+	int target_cpu;
+	int session_fd, cpu_fd;
+
+	self->fd1 = open(LIVEUPDATE_DEV, O_RDWR);
+	if (self->fd1 < 0 && errno == ENOENT)
+		SKIP(return, "%s does not exist", LIVEUPDATE_DEV);
+	ASSERT_GE(self->fd1, 0);
+
+	target_cpu = find_hotplug_cpu();
+	if (target_cpu < 0)
+		SKIP(return, "No hotpluggable CPU with preserve attribute found");
+
+	snprintf(path, sizeof(path), "/sys/devices/system/cpu/cpu%d/preserve", target_cpu);
+	cpu_fd = open(path, O_RDONLY);
+	ASSERT_GE(cpu_fd, 0);
+
+	session_fd = luo_create_session(self->fd1, "cpu-preserve-test");
+	ASSERT_GE(session_fd, 0);
+
+	ASSERT_EQ(luo_session_preserve_fd(session_fd, cpu_fd, 0x42), 0);
+	close(cpu_fd);
+
+	/* Verify target CPU is now offline */
+	ASSERT_EQ(read_cpu_online(target_cpu), 0);
+
+	/* Closing session unpreserves and brings CPU back online */
+	ASSERT_EQ(close(session_fd), 0);
+	ASSERT_EQ(read_cpu_online(target_cpu), 1);
+}
+
+/*
+ * Test Case: CPU Preservation Sysfs Online/Offline Rejection
+ *
+ * Verifies that while a CPU is preserved, attempting to bring it online
+ * via sysfs fails with -EBUSY. After session closure, normal online/offline
+ * hotplug operations succeed.
+ */
+TEST_F(liveupdate_device, cpu_preserve_sysfs_online_offline)
+{
+	char path[128];
+	int target_cpu;
+	int session_fd, cpu_fd;
+
+	self->fd1 = open(LIVEUPDATE_DEV, O_RDWR);
+	if (self->fd1 < 0 && errno == ENOENT)
+		SKIP(return, "%s does not exist", LIVEUPDATE_DEV);
+	ASSERT_GE(self->fd1, 0);
+
+	target_cpu = find_hotplug_cpu();
+	if (target_cpu < 0)
+		SKIP(return, "No hotpluggable CPU with preserve attribute found");
+
+	snprintf(path, sizeof(path), "/sys/devices/system/cpu/cpu%d/preserve", target_cpu);
+	cpu_fd = open(path, O_RDONLY);
+	ASSERT_GE(cpu_fd, 0);
+
+	session_fd = luo_create_session(self->fd1, "cpu-preserve-sysfs");
+	ASSERT_GE(session_fd, 0);
+
+	ASSERT_EQ(luo_session_preserve_fd(session_fd, cpu_fd, 0x100), 0);
+	close(cpu_fd);
+
+	/* Target CPU is offline while preserved */
+	ASSERT_EQ(read_cpu_online(target_cpu), 0);
+
+	/* Attempting to bring the preserved CPU online via sysfs must fail with EBUSY */
+	EXPECT_EQ(write_cpu_online(target_cpu, 1), -EBUSY);
+	ASSERT_EQ(read_cpu_online(target_cpu), 0);
+
+	/* Writing 0 to online (offline) should succeed since it is already offline */
+	EXPECT_EQ(write_cpu_online(target_cpu, 0), 0);
+
+	/* Close session -> unpreserve and restore online */
+	ASSERT_EQ(close(session_fd), 0);
+	ASSERT_EQ(read_cpu_online(target_cpu), 1);
+
+	/* Verify normal offline and online now succeed */
+	ASSERT_EQ(write_cpu_online(target_cpu, 0), 0);
+	ASSERT_EQ(read_cpu_online(target_cpu), 0);
+	ASSERT_EQ(write_cpu_online(target_cpu, 1), 0);
+	ASSERT_EQ(read_cpu_online(target_cpu), 1);
+}
+
+/*
+ * Test Case: Prevent Double CPU Preservation
+ *
+ * Verifies that a CPU cannot be preserved twice across the same or different
+ * sessions simultaneously.
+ */
+TEST_F(liveupdate_device, cpu_preserve_prevent_double)
+{
+	char path[128];
+	int target_cpu;
+	int session_fd1, session_fd2, cpu_fd1, cpu_fd2;
+
+	self->fd1 = open(LIVEUPDATE_DEV, O_RDWR);
+	if (self->fd1 < 0 && errno == ENOENT)
+		SKIP(return, "%s does not exist", LIVEUPDATE_DEV);
+	ASSERT_GE(self->fd1, 0);
+
+	target_cpu = find_hotplug_cpu();
+	if (target_cpu < 0)
+		SKIP(return, "No hotpluggable CPU with preserve attribute found");
+
+	session_fd1 = luo_create_session(self->fd1, "cpu-double-1");
+	ASSERT_GE(session_fd1, 0);
+	session_fd2 = luo_create_session(self->fd1, "cpu-double-2");
+	ASSERT_GE(session_fd2, 0);
+
+	snprintf(path, sizeof(path), "/sys/devices/system/cpu/cpu%d/preserve", target_cpu);
+	cpu_fd1 = open(path, O_RDONLY);
+	ASSERT_GE(cpu_fd1, 0);
+
+	ASSERT_EQ(luo_session_preserve_fd(session_fd1, cpu_fd1, 0x10), 0);
+	close(cpu_fd1);
+
+	/* Attempting to preserve the same CPU in session2 must fail with -EBUSY */
+	cpu_fd2 = open(path, O_RDONLY);
+	ASSERT_GE(cpu_fd2, 0);
+	EXPECT_EQ(luo_session_preserve_fd(session_fd2, cpu_fd2, 0x20), -EBUSY);
+	close(cpu_fd2);
+
+	/* Attempting to preserve again in session1 must also fail with -EBUSY */
+	cpu_fd2 = open(path, O_RDONLY);
+	ASSERT_GE(cpu_fd2, 0);
+	EXPECT_EQ(luo_session_preserve_fd(session_fd1, cpu_fd2, 0x30), -EBUSY);
+	close(cpu_fd2);
+
+	ASSERT_EQ(close(session_fd1), 0);
+	ASSERT_EQ(close(session_fd2), 0);
+	ASSERT_EQ(read_cpu_online(target_cpu), 1);
+}
+
+/*
+ * Test Case: Multiple Preserved CPUs
+ *
+ * Verifies preserving multiple distinct CPUs simultaneously in the same session.
+ */
+TEST_F(liveupdate_device, cpu_preserve_multiple_cpus)
+{
+	char path[128];
+	int target_cpus[8];
+	int num_targets = 0;
+	int session_fd, cpu_fd, cpu, i;
+
+	self->fd1 = open(LIVEUPDATE_DEV, O_RDWR);
+	if (self->fd1 < 0 && errno == ENOENT)
+		SKIP(return, "%s does not exist", LIVEUPDATE_DEV);
+	ASSERT_GE(self->fd1, 0);
+
+	for (cpu = 1; cpu < 256 && num_targets < 8; cpu++) {
+		snprintf(path, sizeof(path), "/sys/devices/system/cpu/cpu%d/preserve", cpu);
+		if (access(path, R_OK) == 0) {
+			snprintf(path, sizeof(path), "/sys/devices/system/cpu/cpu%d/online", cpu);
+			if (access(path, R_OK) == 0)
+				target_cpus[num_targets++] = cpu;
+		}
+	}
+
+	if (num_targets < 2)
+		SKIP(return, "Need at least 2 hotpluggable CPUs for multiple CPU preserve test");
+
+	session_fd = luo_create_session(self->fd1, "multi-cpu-preserve");
+	ASSERT_GE(session_fd, 0);
+
+	for (i = 0; i < num_targets; i++) {
+		snprintf(path, sizeof(path),
+			 "/sys/devices/system/cpu/cpu%d/preserve", target_cpus[i]);
+		cpu_fd = open(path, O_RDONLY);
+		ASSERT_GE(cpu_fd, 0);
+		ASSERT_EQ(luo_session_preserve_fd(session_fd, cpu_fd, 0x1000 + i), 0);
+		close(cpu_fd);
+	}
+
+	/* Verify all targets are offline and reject sysfs online */
+	for (i = 0; i < num_targets; i++) {
+		ASSERT_EQ(read_cpu_online(target_cpus[i]), 0);
+		EXPECT_EQ(write_cpu_online(target_cpus[i], 1), -EBUSY);
+	}
+
+	/* Close session and verify all targets return online */
+	ASSERT_EQ(close(session_fd), 0);
+	for (i = 0; i < num_targets; i++)
+		ASSERT_EQ(read_cpu_online(target_cpus[i]), 1);
+}
+
 TEST_HARNESS_MAIN
-- 
2.55.0.1082.g2b9226bbc0-goog

[RFC PATCH 22/46] selftests: liveupdate: Add physical CPU preservation live update tests

[Pasha Tatashin]

Add integration tests performing kexec live updates with preserved
CPUs:
- Preserve physical cores across kexec reboot.
- Verify cores remain preserved and isolated in incoming kernel.
- Verify unpreserve and online restoration.

Signed-off-by: Pasha Tatashin <pasha.tatashin@soleen.com>
---
 tools/testing/selftests/liveupdate/Makefile   |   1 +
 tools/testing/selftests/liveupdate/init.c     |  27 +++-
 .../selftests/liveupdate/luo_cpu_preserve.c   | 145 ++++++++++++++++++
 .../selftests/liveupdate/run-vmtests.sh       |   1 +
 tools/testing/selftests/liveupdate/vmtest.sh  |   4 +-
 5 files changed, 175 insertions(+), 3 deletions(-)
 create mode 100644 tools/testing/selftests/liveupdate/luo_cpu_preserve.c

diff --git a/tools/testing/selftests/liveupdate/Makefile b/tools/testing/selftests/liveupdate/Makefile
index 634211c66652..0fa80f26a191 100644
--- a/tools/testing/selftests/liveupdate/Makefile
+++ b/tools/testing/selftests/liveupdate/Makefile
@@ -3,6 +3,7 @@
 TEST_GEN_PROGS += liveupdate
 
 TEST_GEN_PROGS_EXTENDED += luo_kexec_simple
+TEST_GEN_PROGS_EXTENDED += luo_cpu_preserve
 TEST_GEN_PROGS_EXTENDED += luo_multi_session
 TEST_GEN_PROGS_EXTENDED += luo_stress_sessions
 TEST_GEN_PROGS_EXTENDED += luo_stress_files
diff --git a/tools/testing/selftests/liveupdate/init.c b/tools/testing/selftests/liveupdate/init.c
index fb08bd58b9b9..a3850ff95c00 100644
--- a/tools/testing/selftests/liveupdate/init.c
+++ b/tools/testing/selftests/liveupdate/init.c
@@ -11,6 +11,7 @@
 #include <string.h>
 #include <sys/mount.h>
 #include <sys/reboot.h>
+#include <sys/stat.h>
 #include <sys/syscall.h>
 #include <sys/wait.h>
 #include <unistd.h>
@@ -27,11 +28,15 @@ static int mount_filesystems(void)
 		return -1;
 	}
 
-	if (mount("debugfs", "/debugfs", "debugfs", 0, NULL) < 0) {
-		fprintf(stderr, "INIT: Failed to mount debugfs\n");
+	if (mount("sysfs", "/sys", "sysfs", 0, NULL) < 0) {
+		fprintf(stderr, "INIT: Failed to mount sysfs\n");
 		return -1;
 	}
 
+	mkdir("/sys/kernel/debug", 0755);
+	mount("debugfs", "/sys/kernel/debug", "debugfs", 0, NULL);
+	mount("debugfs", "/debugfs", "debugfs", 0, NULL);
+
 	if (mount("proc", "/proc", "proc", 0, NULL) < 0) {
 		fprintf(stderr, "INIT: Failed to mount proc\n");
 		return -1;
@@ -116,7 +121,23 @@ static int run_test(int stage)
 	}
 
 	waitpid(pid, &status, 0);
+	if (WIFEXITED(status) && WEXITSTATUS(status) == 0)
+		return 0;
+
+	/* If test binary does not accept -s (e.g. kselftest harness), run without -s */
+	pid = fork();
+	if (pid < 0)
+		return -1;
+
+	if (!pid) {
+		char *const argv[] = {TEST_BINARY, NULL};
 
+		execve(TEST_BINARY, argv, NULL);
+		fprintf(stderr, "INIT: execve failed\n");
+		_exit(1);
+	}
+
+	waitpid(pid, &status, 0);
 	return (WIFEXITED(status) && WEXITSTATUS(status) == 0) ? 0 : -1;
 }
 
@@ -168,6 +189,8 @@ int main(int argc, char *argv[])
 	}
 
 	printf("INIT: Stage %d completed successfully.\n", current_stage);
+	if (current_stage == 2)
+		printf("\n--- TEST PASSED ---\n");
 	reboot(current_stage == 1 ? RB_KEXEC : RB_AUTOBOOT);
 
 	return 0;
diff --git a/tools/testing/selftests/liveupdate/luo_cpu_preserve.c b/tools/testing/selftests/liveupdate/luo_cpu_preserve.c
new file mode 100644
index 000000000000..85d17ccf3d90
--- /dev/null
+++ b/tools/testing/selftests/liveupdate/luo_cpu_preserve.c
@@ -0,0 +1,145 @@
+// SPDX-License-Identifier: GPL-2.0-only
+
+/*
+ * Copyright (c) 2026, Google LLC.
+ * Pasha Tatashin <pasha.tatashin@soleen.com>
+ *
+ * A selftest to validate the lifecycle of preserved CPUs across Live Update
+ * and within a local session.
+ */
+
+#include <fcntl.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <sys/ioctl.h>
+#include <unistd.h>
+
+#include <libliveupdate.h>
+
+#define TEST_SESSION_NAME "cpu-preserve-session"
+#define STATE_SESSION_NAME "cpu_preserve_state"
+#define STATE_MEMFD_TOKEN 999
+#define TEST_CPU_TOKEN 0x100
+
+static int find_target_cpu(void)
+{
+	char path[128];
+	int cpu;
+
+	for (cpu = 1; cpu < 256; cpu++) {
+		snprintf(path, sizeof(path), "/sys/devices/system/cpu/cpu%d/preserve", cpu);
+		if (access(path, R_OK) == 0) {
+			snprintf(path, sizeof(path), "/sys/devices/system/cpu/cpu%d/online", cpu);
+			if (access(path, R_OK) == 0)
+				return cpu;
+		}
+	}
+	return -1;
+}
+
+static int read_cpu_online(int cpu)
+{
+	char path[128], buf[16];
+	int fd, val = -1;
+
+	snprintf(path, sizeof(path), "/sys/devices/system/cpu/cpu%d/online", cpu);
+	fd = open(path, O_RDONLY);
+	if (fd < 0)
+		return -1;
+	if (read(fd, buf, sizeof(buf) - 1) > 0)
+		val = atoi(buf);
+	close(fd);
+	return val;
+}
+
+static int preserve_cpu_fd(int session_fd, int cpu, int token)
+{
+	struct liveupdate_session_preserve_fd pfd = {};
+	char path[128];
+	int cpu_fd, ret;
+
+	snprintf(path, sizeof(path), "/sys/devices/system/cpu/cpu%d/preserve", cpu);
+	cpu_fd = open(path, O_RDONLY);
+	if (cpu_fd < 0)
+		return -1;
+
+	pfd.size = sizeof(pfd);
+	pfd.fd = cpu_fd;
+	pfd.token = token;
+
+	ret = ioctl(session_fd, LIVEUPDATE_SESSION_PRESERVE_FD, &pfd);
+	close(cpu_fd);
+	return ret;
+}
+
+/* Stage 1: Executed before kexec */
+static void run_stage_1(int luo_fd)
+{
+	int target_cpu, session_fd;
+
+	target_cpu = find_target_cpu();
+	if (target_cpu < 0)
+		fail_exit("No hotpluggable CPU with preserve attribute found");
+
+	ksft_print_msg("[STAGE 1] Target CPU for preservation: %d\n", target_cpu);
+
+	create_state_file(luo_fd, STATE_SESSION_NAME, STATE_MEMFD_TOKEN, target_cpu);
+
+	session_fd = luo_create_session(luo_fd, TEST_SESSION_NAME);
+	if (session_fd < 0)
+		fail_exit("luo_create_session for '%s'", TEST_SESSION_NAME);
+
+	if (preserve_cpu_fd(session_fd, target_cpu, TEST_CPU_TOKEN) < 0)
+		fail_exit("preserve_cpu_fd for cpu %d", target_cpu);
+
+	if (read_cpu_online(target_cpu) != 0)
+		fail_exit("CPU %d was expected to be offline after preservation", target_cpu);
+
+	ksft_print_msg("[STAGE 1] CPU %d successfully preserved and offlined\n", target_cpu);
+
+	close(luo_fd);
+	daemonize_and_wait();
+}
+
+/* Stage 2: Executed after kexec */
+static void run_stage_2(int luo_fd, int state_session_fd)
+{
+	struct liveupdate_session_retrieve_fd rfd = {};
+	int target_cpu, session_fd;
+
+	restore_and_read_stage(state_session_fd, STATE_MEMFD_TOKEN, &target_cpu);
+	ksft_print_msg("[STAGE 2] Restored target CPU: %d\n", target_cpu);
+
+	session_fd = luo_retrieve_session(luo_fd, TEST_SESSION_NAME);
+	if (session_fd < 0)
+		fail_exit("luo_retrieve_session for '%s'", TEST_SESSION_NAME);
+
+	rfd.size = sizeof(rfd);
+	rfd.token = TEST_CPU_TOKEN;
+	if (ioctl(session_fd, LIVEUPDATE_SESSION_RETRIEVE_FD, &rfd) < 0)
+		fail_exit("LIVEUPDATE_SESSION_RETRIEVE_FD for cpu token %#x", TEST_CPU_TOKEN);
+
+	close(rfd.fd);
+
+	if (read_cpu_online(target_cpu) != 1)
+		fail_exit("CPU %d was expected to be online after retrieval", target_cpu);
+
+	ksft_print_msg("[STAGE 2] CPU %d successfully retrieved and online\n", target_cpu);
+
+	if (luo_session_finish(session_fd) < 0)
+		fail_exit("luo_session_finish for test session");
+	close(session_fd);
+
+	if (luo_session_finish(state_session_fd) < 0)
+		fail_exit("luo_session_finish for state session");
+	close(state_session_fd);
+
+	ksft_print_msg("\n--- CPU PRESERVATION KEXEC TEST PASSED ---\n");
+}
+
+int main(int argc, char *argv[])
+{
+	return luo_test(argc, argv, STATE_SESSION_NAME,
+			run_stage_1, run_stage_2);
+}
diff --git a/tools/testing/selftests/liveupdate/run-vmtests.sh b/tools/testing/selftests/liveupdate/run-vmtests.sh
index d656ce58c5a7..1b90d56ec4a2 100755
--- a/tools/testing/selftests/liveupdate/run-vmtests.sh
+++ b/tools/testing/selftests/liveupdate/run-vmtests.sh
@@ -20,6 +20,7 @@ TEST_NAMES=(
 	"luo_multi_session"
 	"luo_stress_files"
 	"luo_stress_sessions"
+	"luo_cpu_preserve"
 )
 
 function usage() {
diff --git a/tools/testing/selftests/liveupdate/vmtest.sh b/tools/testing/selftests/liveupdate/vmtest.sh
index 64fd2ab55b76..c934c76c00a0 100755
--- a/tools/testing/selftests/liveupdate/vmtest.sh
+++ b/tools/testing/selftests/liveupdate/vmtest.sh
@@ -145,6 +145,7 @@ function mkinitrd() {
 	cat > "$workspace_dir/cpio_list_inner" <<EOF
 dir /dev 0755 0 0
 dir /proc 0755 0 0
+dir /sys 0755 0 0
 dir /debugfs 0755 0 0
 nod /dev/console 0600 0 0 c 5 1
 file /init $workspace_dir/init 0755 0 0
@@ -157,6 +158,7 @@ EOF
 	cat > "$workspace_dir/cpio_list" <<EOF
 dir /dev 0755 0 0
 dir /proc 0755 0 0
+dir /sys 0755 0 0
 dir /debugfs 0755 0 0
 nod /dev/console 0600 0 0 c 5 1
 file /init $workspace_dir/init 0755 0 0
@@ -179,7 +181,7 @@ function run_qemu() {
 
 	echo "# Serial Log: $serial"
 	timeout 30s \
-	$qemu_cmd -m 1G -smp 2 -no-reboot -nographic -nodefaults \
+	$qemu_cmd -m 1G -smp 4 -no-reboot -nographic -nodefaults \
 		  -accel tcg -accel hvf -accel kvm \
 		  -serial file:"$serial" \
 		  -append "$cmdline" \
-- 
2.55.0.1082.g2b9226bbc0-goog

[RFC PATCH 23/46] Documentation: liveupdate: Add physical CPU preservation documentation

[Pasha Tatashin]

Add documentation for physical CPU preservation subsystem, workload
attachment API, architecture backend hooks, and KHO serialization ABI.

Signed-off-by: Pasha Tatashin <pasha.tatashin@soleen.com>
---
 Documentation/liveupdate/cpu_preservation.rst | 34 +++++++++++++++++++
 Documentation/liveupdate/index.rst            | 11 ++++++
 2 files changed, 45 insertions(+)
 create mode 100644 Documentation/liveupdate/cpu_preservation.rst
 create mode 100644 Documentation/liveupdate/index.rst

diff --git a/Documentation/liveupdate/cpu_preservation.rst b/Documentation/liveupdate/cpu_preservation.rst
new file mode 100644
index 000000000000..a7a534188685
--- /dev/null
+++ b/Documentation/liveupdate/cpu_preservation.rst
@@ -0,0 +1,34 @@
+.. SPDX-License-Identifier: GPL-2.0-or-later
+
+=========================
+Physical CPU Preservation
+=========================
+
+.. kernel-doc:: kernel/liveupdate/cpu_preserve.c
+   :doc: Preserved CPU Subsystem
+
+CPU Preservation Workload API
+=============================
+
+.. kernel-doc:: kernel/liveupdate/cpu_preserve.c
+   :export:
+
+Architecture Backend Interface
+==============================
+
+.. kernel-doc:: include/linux/cpu_preserve.h
+
+CPU Preservation ABI
+====================
+
+.. kernel-doc:: include/linux/kho/abi/cpu.h
+   :doc: CPU Preservation Live Update ABI
+
+.. kernel-doc:: include/linux/kho/abi/cpu.h
+
+See Also
+========
+
+- :doc:`/core-api/liveupdate`
+- :doc:`/liveupdate/vmm`
+- :doc:`/mm/memfd_preservation`
diff --git a/Documentation/liveupdate/index.rst b/Documentation/liveupdate/index.rst
new file mode 100644
index 000000000000..415ac748c9f7
--- /dev/null
+++ b/Documentation/liveupdate/index.rst
@@ -0,0 +1,11 @@
+.. SPDX-License-Identifier: GPL-2.0
+
+===========
+Live Update
+===========
+
+.. toctree::
+   :maxdepth: 1
+
+   vmm
+   cpu_preservation
-- 
2.55.0.1082.g2b9226bbc0-goog

[RFC PATCH 24/46] MAINTAINERS: Add entry for KVM Caretaker

[Pasha Tatashin]

Add KERNEL VIRTUAL MACHINE CARETAKER section in MAINTAINERS covering
Caretaker headers, architecture backends, and core execution files.

Signed-off-by: Pasha Tatashin <pasha.tatashin@soleen.com>
---
 MAINTAINERS | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/MAINTAINERS b/MAINTAINERS
index 23f332cfbd16..86806245041e 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -14296,6 +14296,17 @@ F:	tools/kvm/
 F:	tools/testing/selftests/kvm/
 F:	virt/kvm/*
 
+KERNEL VIRTUAL MACHINE CARETAKER
+M:	Pasha Tatashin <pasha.tatashin@soleen.com>
+L:	kvm@vger.kernel.org
+L:	kexec@lists.infradead.org
+S:	Maintained
+F:	arch/*/include/asm/kvm_caretaker*
+F:	arch/*/kvm/*caretaker*
+F:	arch/x86/kvm/mmu/kho.c
+F:	include/linux/kvm_caretaker.h
+F:	virt/kvm/caretaker*.c
+
 KERNEL VIRTUAL MACHINE FOR ARM64 (KVM/arm64)
 M:	Marc Zyngier <maz@kernel.org>
 M:	Oliver Upton <oupton@kernel.org>
-- 
2.55.0.1082.g2b9226bbc0-goog

[RFC PATCH 25/46] arm64: liveupdate: Add support for physical CPU preservation

[Pasha Tatashin]

Enable physical CPU preservation on arm64 by selecting
ARCH_SUPPORTS_LIVEUPDATE_CPU.

Integrate preserved CPU handling into the arm64 SMP and CPU hotplug
paths with isolated transition page tables:
- Build isolated transition page tables mapping exclusively preserved
  text/rodata (ROX), data (RW), stacks (RW), and preserved buffers.
- Completely eliminate bulk cloning of the kernel linear direct map.
- Limit KHO page table memory footprint to < 10 pages total.
- Include CPU_PRESERVED_TEXT in arm64 vmlinux.lds.S.
- Compile cpu_preserve.o with -mbranch-protection=none and
  -fno-stack-protector to prevent PAC key mismatch aborts.
- In __cpu_disable(), avoid tearing down IPIs for preserved CPUs.
- In op_cpu_kill(), skip waiting for the CPU to die if preserved.
- In smp_send_stop(), exclude preserved CPUs from stop IPIs.
- Add cache maintenance to PoC for secondary boot data and page tables.

Signed-off-by: Pasha Tatashin <pasha.tatashin@soleen.com>
---
 arch/arm64/Kconfig                    |   1 +
 arch/arm64/include/asm/cpu_preserve.h |  44 ++++
 arch/arm64/kernel/Makefile            |   7 +
 arch/arm64/kernel/cpu_preserve.c      | 365 ++++++++++++++++++++++++++
 arch/arm64/kernel/preserve_cpu.S      |  44 ++++
 arch/arm64/kernel/smp.c               |   8 +-
 arch/arm64/kernel/vmlinux.lds.S       |   1 +
 7 files changed, 469 insertions(+), 1 deletion(-)
 create mode 100644 arch/arm64/include/asm/cpu_preserve.h
 create mode 100644 arch/arm64/kernel/cpu_preserve.c
 create mode 100644 arch/arm64/kernel/preserve_cpu.S

diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig
index b5a51b0ef944..957ec38a9680 100644
--- a/arch/arm64/Kconfig
+++ b/arch/arm64/Kconfig
@@ -38,6 +38,7 @@ config ARM64
 	select ARCH_HAS_MEMBARRIER_SYNC_CORE
 	select ARCH_HAS_MEM_ENCRYPT
 	select ARCH_SUPPORTS_MSEAL_SYSTEM_MAPPINGS
+	select ARCH_SUPPORTS_LIVEUPDATE_CPU if LIVEUPDATE
 	select ARCH_HAS_NMI_SAFE_THIS_CPU_OPS
 	select ARCH_HAS_NON_OVERLAPPING_ADDRESS_SPACE
 	select ARCH_HAS_NONLEAF_PMD_YOUNG if ARM64_HAFT
diff --git a/arch/arm64/include/asm/cpu_preserve.h b/arch/arm64/include/asm/cpu_preserve.h
new file mode 100644
index 000000000000..7c7c94f21ccc
--- /dev/null
+++ b/arch/arm64/include/asm/cpu_preserve.h
@@ -0,0 +1,44 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright (c) 2026, Google LLC.
+ * Pasha Tatashin <pasha.tatashin@soleen.com>
+ */
+#ifndef __ASM_ARM64_CPU_PRESERVE_H
+#define __ASM_ARM64_CPU_PRESERVE_H
+
+#include <asm/memory.h>
+#include <asm/tlbflush.h>
+#include <asm/virt.h>
+
+#define ARCH_CPU_PRESERVED_STACK_ORDER	(THREAD_SIZE_ORDER + 1)
+
+bool arch_cpu_preserved_is_active(void);
+asmlinkage void __arch_cpu_preserved_dcache_clean(unsigned long start, unsigned long end);
+asmlinkage void arch_cpu_preserved_dcache_clean(unsigned long start, unsigned long end);
+asmlinkage void arch_cpu_preserved_dcache_inval(unsigned long start, unsigned long end);
+
+static inline void arm64_flush_host_tlb_local(void)
+{
+	dsb(nshst);
+	if (is_kernel_in_hyp_mode()) {
+		asm volatile("tlbi alle2\n"
+			     "dsb nsh\n"
+			     "isb\n" ::: "memory");
+	} else {
+		local_flush_tlb_all();
+	}
+}
+
+static inline void arm64_flush_host_tlb_all(void)
+{
+	dsb(ishst);
+	if (is_kernel_in_hyp_mode()) {
+		asm volatile("tlbi alle2is\n"
+			     "dsb ish\n"
+			     "isb\n" ::: "memory");
+	} else {
+		flush_tlb_all();
+	}
+}
+
+#endif /* __ASM_ARM64_CPU_PRESERVE_H */
diff --git a/arch/arm64/kernel/Makefile b/arch/arm64/kernel/Makefile
index d2690c3ec528..151513a5350b 100644
--- a/arch/arm64/kernel/Makefile
+++ b/arch/arm64/kernel/Makefile
@@ -70,6 +70,13 @@ obj-$(CONFIG_ARM_SDE_INTERFACE)		+= sdei.o
 obj-$(CONFIG_ARM64_PTR_AUTH)		+= pointer_auth.o
 obj-$(CONFIG_ARM64_MPAM)		+= mpam.o
 obj-$(CONFIG_ARM64_MTE)			+= mte.o
+obj-$(CONFIG_LIVEUPDATE_CPU)		+= cpu_preserve.o preserve_cpu.o
+KASAN_SANITIZE_cpu_preserve.o := n
+KCSAN_SANITIZE_cpu_preserve.o := n
+UBSAN_SANITIZE_cpu_preserve.o := n
+KCOV_INSTRUMENT_cpu_preserve.o := n
+CFLAGS_REMOVE_cpu_preserve.o = $(CC_FLAGS_FTRACE)
+CFLAGS_cpu_preserve.o += $(call cc-option,-mbranch-protection=none) -fno-stack-protector $(call cc-option,-ftrivial-auto-var-init=uninitialized) $(call cc-option,-fno-jump-tables)
 obj-y					+= vdso-wrap.o
 obj-$(CONFIG_COMPAT_VDSO)		+= vdso32-wrap.o
 
diff --git a/arch/arm64/kernel/cpu_preserve.c b/arch/arm64/kernel/cpu_preserve.c
new file mode 100644
index 000000000000..15de1ca09062
--- /dev/null
+++ b/arch/arm64/kernel/cpu_preserve.c
@@ -0,0 +1,365 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Copyright (c) 2026, Google LLC.
+ * Pasha Tatashin <pasha.tatashin@soleen.com>
+ *
+ * Architecture specific CPU preservation support for ARM64.
+ */
+#include <linux/arm-smccc.h>
+#include <linux/cpu_preserve.h>
+#include <linux/irqchip/arm-gic-v3.h>
+#include <linux/kexec_handover.h>
+#include <linux/kho/abi/cpu.h>
+#include <linux/mm.h>
+#include <linux/psci.h>
+#include <linux/sched/mm.h>
+#include <uapi/linux/psci.h>
+
+#include <asm/barrier.h>
+#include <linux/cacheflush.h>
+#include <asm/cpu_ops.h>
+#include <asm/daifflags.h>
+#include <asm/kernel-pgtable.h>
+#include <asm/kvm_asm.h>
+#include <linux/pgtable.h>
+#include <asm/sysreg.h>
+#include <asm/tlbflush.h>
+#include <asm/trans_pgd.h>
+#include <asm/virt.h>
+
+static enum arm_smccc_conduit arm64_psci_conduit __cpu_preserved_data;
+phys_addr_t arm64_caretaker_pgd_pa __cpu_preserved_data;
+static u64 arm64_cpu_mpidr[NR_CPUS] __cpu_preserved_data;
+
+/*
+ * Signal or wake up a preserved physical CPU via SEV.
+ */
+void __cpu_preserved_text arch_cpu_preserved_kick(int cpu)
+{
+	dsb(ishst);
+	sev();
+	isb();
+}
+
+/*
+ * Low-power wait in parking loop.
+ */
+void __cpu_preserved_text arch_cpu_preserved_park_wait(void)
+{
+	wfe();
+}
+
+int __cpu_preserved_text arch_cpu_preserved_mpidr_to_cpu(u64 mpidr)
+{
+	int c;
+
+	for (c = 0; c < ARRAY_SIZE(arm64_cpu_mpidr); c++) {
+		if ((arm64_cpu_mpidr[c] & MPIDR_HWID_BITMASK) == (mpidr & MPIDR_HWID_BITMASK))
+			return c;
+	}
+	return -EINVAL;
+}
+EXPORT_SYMBOL_GPL(arch_cpu_preserved_mpidr_to_cpu);
+
+bool __cpu_preserved_text arch_cpu_preserved_is_active(void)
+{
+	struct cpu_preserved_stack_context *sctx = cpu_preserved_get_stack_context();
+	u64 ttbr1 = read_sysreg(ttbr1_el1);
+
+	if (sctx && sctx->session_pgd_pa && ttbr1 == sctx->session_pgd_pa)
+		return true;
+
+	if (arm64_caretaker_pgd_pa)
+		return ttbr1 == arm64_caretaker_pgd_pa;
+
+	return false;
+}
+EXPORT_SYMBOL_GPL(arch_cpu_preserved_is_active);
+
+void __cpu_preserved_text arch_cpu_preserved_switch_pgd(phys_addr_t pgd_pa)
+{
+	if (pgd_pa && read_sysreg(ttbr1_el1) != pgd_pa) {
+		write_sysreg(pgd_pa, ttbr1_el1);
+		isb();
+		arm64_flush_host_tlb_local();
+	}
+}
+EXPORT_SYMBOL_GPL(arch_cpu_preserved_switch_pgd);
+
+static pte_t *arm64_get_kernel_pte(unsigned long addr)
+{
+	pgd_t *pgdp = pgd_offset_k(addr);
+	p4d_t *p4dp;
+	pud_t *pudp;
+	pmd_t *pmdp;
+
+	if (pgd_none(READ_ONCE(*pgdp)))
+		return NULL;
+
+	p4dp = p4d_offset(pgdp, addr);
+	if (p4d_none(READ_ONCE(*p4dp)))
+		return NULL;
+
+	pudp = pud_offset(p4dp, addr);
+	if (pud_none(READ_ONCE(*pudp)) || pud_leaf(READ_ONCE(*pudp)))
+		return NULL;
+
+	pmdp = pmd_offset(pudp, addr);
+	if (pmd_none(READ_ONCE(*pmdp)) || pmd_leaf(READ_ONCE(*pmdp)))
+		return NULL;
+
+	return pte_offset_kernel(pmdp, addr);
+}
+
+/**
+ * arch_cpu_preserved_as_map - Populate an isolated page table on arm64
+ * @as: Address space to map into.
+ * @pa: Physical address of the range.
+ * @va: Virtual address the range must appear at.
+ * @size: Size of the range in bytes.
+ * @prot: Protection to apply.
+ *
+ * Return: 0 on success, or a negative errno on failure.
+ */
+int arch_cpu_preserved_as_map(struct cpu_preserved_as *as, phys_addr_t pa,
+			      unsigned long va, size_t size, pgprot_t prot)
+{
+	struct trans_pgd_info info = {
+		.trans_alloc_page = cpu_preserved_as_alloc_page,
+		.trans_alloc_arg = as,
+	};
+	unsigned long offset = va & ~PAGE_MASK;
+	size_t page_size = PAGE_ALIGN(offset + size);
+	unsigned long page_va = va & PAGE_MASK;
+	phys_addr_t page_pa = (pa & PAGE_MASK);
+
+	return trans_pgd_map_range(&info, as->pgd, page_pa,
+				   page_va, page_size, prot);
+}
+
+void arch_cpu_preserved_as_flush_tlb(void)
+{
+	arm64_flush_host_tlb_all();
+}
+
+void arch_cpu_preserved_set_transition_as(struct cpu_preserved_as *as)
+{
+	arm64_caretaker_pgd_pa = as ? as->pgd_pa : 0;
+	cpu_preserved_clean(&arm64_caretaker_pgd_pa);
+}
+
+/**
+ * arch_cpu_preserved_setup_buffer - Set up runtime buffer and page tables
+ * @text_page: Runtime-allocated physical page backing preserved text
+ * @text_nr_pages: Number of pages in text buffer
+ * @data_page: Runtime-allocated physical page backing preserved data
+ * @data_nr_pages: Number of pages in data buffer
+ *
+ * Remap init_mm kernel mappings for __cpu_preserved_text and
+ * __cpu_preserved_data to point to the runtime-allocated pages outside
+ * Scratch.
+ *
+ * Return: 0 on success, or -ENOMEM on failure.
+ */
+static void arm64_split_contpte_range(unsigned long start, unsigned long end)
+{
+	unsigned long addr;
+
+	if (start >= end)
+		return;
+
+	for (addr = ALIGN_DOWN(start, CONT_PTE_SIZE); addr < end; addr += CONT_PTE_SIZE) {
+		pte_t *ptep = arm64_get_kernel_pte(addr);
+		int i;
+
+		if (!ptep)
+			continue;
+
+		ptep = PTR_ALIGN_DOWN(ptep, sizeof(*ptep) * CONT_PTES);
+
+		for (i = 0; i < CONT_PTES; i++) {
+			pte_t pte = __ptep_get(&ptep[i]);
+
+			if (pte_valid_cont(pte))
+				__set_pte(&ptep[i], pte_mknoncont(pte));
+		}
+	}
+
+	flush_tlb_kernel_range(ALIGN_DOWN(start, CONT_PTE_SIZE),
+			       ALIGN(end, CONT_PTE_SIZE));
+	arm64_flush_host_tlb_all();
+}
+
+int arch_cpu_preserved_setup_buffer(struct page *text_page,
+				    unsigned int text_nr_pages,
+				    struct page *data_page,
+				    unsigned int data_nr_pages)
+{
+	unsigned long text_start = (unsigned long)__cpu_preserved_text_start;
+	unsigned long data_start = (unsigned long)__cpu_preserved_data_start;
+	unsigned int i;
+
+	/* Split any contiguous 64KB mappings before replacing individual PTEs */
+	arm64_split_contpte_range(text_start, text_start + text_nr_pages * PAGE_SIZE);
+	arm64_split_contpte_range(data_start, data_start + data_nr_pages * PAGE_SIZE);
+
+	/* Clean old mappings before switching PTEs */
+	__arch_cpu_preserved_dcache_clean(text_start, text_start + text_nr_pages * PAGE_SIZE);
+	__arch_cpu_preserved_dcache_clean(data_start, data_start + data_nr_pages * PAGE_SIZE);
+
+	/* Remap init_mm kernel mappings to point to allocated buffer pages */
+	for (i = 0; i < text_nr_pages; i++) {
+		unsigned long va = text_start + i * PAGE_SIZE;
+		pte_t *ptep = arm64_get_kernel_pte(va);
+
+		if (!ptep)
+			return -EINVAL;
+
+		pgprot_t prot = __pgprot(pgprot_val(pte_pgprot(*ptep)) & ~PTE_CONT);
+		phys_addr_t pa = page_to_phys(text_page) + i * PAGE_SIZE;
+
+		set_pte_at(&init_mm, va, ptep, pfn_pte(PHYS_PFN(pa), prot));
+	}
+
+	for (i = 0; i < data_nr_pages; i++) {
+		unsigned long va = data_start + i * PAGE_SIZE;
+		pte_t *ptep = arm64_get_kernel_pte(va);
+
+		if (!ptep)
+			return -EINVAL;
+
+		pgprot_t prot = __pgprot(pgprot_val(pte_pgprot(*ptep)) & ~PTE_CONT);
+		phys_addr_t pa = page_to_phys(data_page) + i * PAGE_SIZE;
+
+		set_pte_at(&init_mm, va, ptep, pfn_pte(PHYS_PFN(pa), prot));
+	}
+
+	arm64_flush_host_tlb_all();
+	flush_icache_range(text_start, text_start + (text_nr_pages * PAGE_SIZE));
+
+	for (i = 0; i < nr_cpu_ids; i++)
+		arm64_cpu_mpidr[i] = cpu_logical_map(i);
+	cpu_preserved_clean(&arm64_cpu_mpidr);
+
+	return 0;
+}
+
+/*
+ * Masks DAIF interrupts and enables GIC CPU interface for WFx wakeups.
+ */
+void __cpu_preserved_text arch_cpu_preserved_park_init(int cpu)
+{
+	struct cpu_preserved_stack_context *sctx = cpu_preserved_get_stack_context();
+	phys_addr_t pgd_pa = 0;
+
+	if (sctx && sctx->session_pgd_pa)
+		pgd_pa = sctx->session_pgd_pa;
+	else
+		pgd_pa = cpu_preserved_get_pgd(cpu);
+
+	local_daif_mask();
+	cpu_preserved_inval(&arm64_psci_conduit);
+	cpu_preserved_inval(&arm64_cpu_mpidr);
+	if (!pgd_pa) {
+		cpu_preserved_inval(&arm64_caretaker_pgd_pa);
+		pgd_pa = READ_ONCE(arm64_caretaker_pgd_pa);
+	}
+
+	write_sysreg(0, ttbr0_el1);
+	if (pgd_pa)
+		write_sysreg(pgd_pa, ttbr1_el1);
+	isb();
+	arm64_flush_host_tlb_local();
+
+	write_sysreg_s(0xff, SYS_ICC_PMR_EL1);
+	write_sysreg_s(1, SYS_ICC_IGRPEN1_EL1);
+	isb();
+}
+
+void arch_cpu_preserved_early_init(void)
+{
+	int c;
+
+	for (c = 0; c < ARRAY_SIZE(arm64_cpu_mpidr); c++)
+		arm64_cpu_mpidr[c] = cpu_logical_map(c);
+	cpu_preserved_clean(&arm64_cpu_mpidr);
+
+	cpu_preserved_inval(&arm64_psci_conduit);
+	if (arm64_psci_conduit == SMCCC_CONDUIT_NONE) {
+		arm64_psci_conduit = arm_smccc_1_1_get_conduit();
+		cpu_preserved_clean(&arm64_psci_conduit);
+	}
+}
+EXPORT_SYMBOL_GPL(arch_cpu_preserved_early_init);
+
+void __cpu_preserved_text arch_cpu_preserved_park_finish(int cpu)
+{
+	u32 el = (read_sysreg(CurrentEL) >> 2) & 3;
+	enum arm_smccc_conduit conduit;
+
+	cpu_preserved_inval(&arm64_psci_conduit);
+	conduit = READ_ONCE(arm64_psci_conduit);
+
+	local_daif_mask();
+	write_sysreg_s(0, SYS_ICC_PMR_EL1);
+	write_sysreg_s(0, SYS_ICC_IGRPEN1_EL1);
+	isb();
+
+	if (el == 2 || conduit == SMCCC_CONDUIT_NONE)
+		conduit = (el == 2) ? SMCCC_CONDUIT_SMC : SMCCC_CONDUIT_HVC;
+
+	cpu_preserved_set_dead(cpu);
+
+	/*
+	 * Direct PSCI CPU_OFF call in preserved text without relying on
+	 * unpreserved kernel data structures or function pointers.
+	 *
+	 * x0: PSCI_0_2_FN_CPU_OFF (0x84000002)
+	 * x1: Power down state (0x00010000)
+	 */
+	if (conduit == SMCCC_CONDUIT_HVC) {
+		asm volatile("mov	x0, #0x0002\n"
+			"movk	x0, #0x8400, lsl #16\n"
+			"mov	x1, #0\n"
+			"mov	x2, #0\n"
+			"mov	x3, #0\n"
+			"mov	x4, #0\n"
+			"mov	x5, #0\n"
+			"mov	x6, #0\n"
+			"mov	x7, #0\n"
+			"hvc	#0\n"
+			:
+			:
+			: "x0", "x1", "x2", "x3", "x4", "x5", "x6", "x7", "memory"
+		);
+	} else {
+		asm volatile("mov	x0, #0x0002\n"
+			"movk	x0, #0x8400, lsl #16\n"
+			"mov	x1, #0\n"
+			"mov	x2, #0\n"
+			"mov	x3, #0\n"
+			"mov	x4, #0\n"
+			"mov	x5, #0\n"
+			"mov	x6, #0\n"
+			"mov	x7, #0\n"
+			"smc	#0\n"
+			:
+			:
+			: "x0", "x1", "x2", "x3", "x4", "x5", "x6", "x7", "memory"
+		);
+	}
+
+	while (1) {
+		wfi();
+		wfe();
+	}
+}
+
+void arch_cpu_preserved_wait_dead(int cpu)
+{
+	const struct cpu_operations *ops = get_cpu_ops(cpu);
+
+	if (ops && ops->cpu_kill)
+		ops->cpu_kill(cpu);
+}
+
diff --git a/arch/arm64/kernel/preserve_cpu.S b/arch/arm64/kernel/preserve_cpu.S
new file mode 100644
index 000000000000..c7e3cd8133f0
--- /dev/null
+++ b/arch/arm64/kernel/preserve_cpu.S
@@ -0,0 +1,44 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/*
+ * Low-level assembly routines for ARM64 physical CPU preservation.
+ *
+ * Copyright (c) 2026, Google LLC.
+ * Pasha Tatashin <pasha.tatashin@soleen.com>
+ */
+#include <linux/linkage.h>
+#include <asm/assembler.h>
+
+	.section .text.cpu_preserved, "ax"
+
+/*
+ * arch_cpu_preserved_park_on_stack - Switch stack and enter park loop
+ * @cpu: Logical CPU identifier (x0)
+ * @stack_top: Top of runtime allocated stack (x1)
+ */
+SYM_FUNC_START(arch_cpu_preserved_park_on_stack)
+	mov	sp, x1
+	mov	x19, x0
+	bl	cpu_preserved_park_loop
+	mov	x0, x19
+	bl	arch_cpu_preserved_park_finish
+	b	.
+SYM_FUNC_END(arch_cpu_preserved_park_on_stack)
+EXPORT_SYMBOL(arch_cpu_preserved_park_on_stack)
+
+/*
+ * __arch_cpu_preserved_dcache_clean - Clean and invalidate data cache to PoC
+ * @start: Virtual start address (x0)
+ * @end: Virtual end address (x1)
+ */
+SYM_FUNC_START(__arch_cpu_preserved_dcache_clean)
+	dsb	sy
+	raw_dcache_line_size x2, x3
+	dcache_by_myline_op_nosync civac, x0, x1, x2, x3
+	dsb	sy
+	ret
+SYM_FUNC_END(__arch_cpu_preserved_dcache_clean)
+SYM_FUNC_ALIAS(arch_cpu_preserved_dcache_clean, __arch_cpu_preserved_dcache_clean)
+SYM_FUNC_ALIAS(arch_cpu_preserved_dcache_inval, __arch_cpu_preserved_dcache_clean)
+EXPORT_SYMBOL(__arch_cpu_preserved_dcache_clean)
+EXPORT_SYMBOL(arch_cpu_preserved_dcache_clean)
+EXPORT_SYMBOL(arch_cpu_preserved_dcache_inval)
diff --git a/arch/arm64/kernel/smp.c b/arch/arm64/kernel/smp.c
index a61dc3016a11..43651eceb09c 100644
--- a/arch/arm64/kernel/smp.c
+++ b/arch/arm64/kernel/smp.c
@@ -21,6 +21,7 @@
 #include <linux/mm.h>
 #include <linux/err.h>
 #include <linux/cpu.h>
+#include <linux/cpu_preserve.h>
 #include <linux/smp.h>
 #include <linux/seq_file.h>
 #include <linux/irq.h>
@@ -327,7 +328,12 @@ int __cpu_disable(void)
 
 static int op_cpu_kill(unsigned int cpu)
 {
-	const struct cpu_operations *ops = get_cpu_ops(cpu);
+	const struct cpu_operations *ops;
+
+	if (cpu_is_preserved(cpu))
+		return 0;
+
+	ops = get_cpu_ops(cpu);
 
 	/*
 	 * If we have no means of synchronising with the dying CPU, then assume
diff --git a/arch/arm64/kernel/vmlinux.lds.S b/arch/arm64/kernel/vmlinux.lds.S
index af1d72020976..9e809b6908a1 100644
--- a/arch/arm64/kernel/vmlinux.lds.S
+++ b/arch/arm64/kernel/vmlinux.lds.S
@@ -197,6 +197,7 @@ SECTIONS
 			IRQENTRY_TEXT
 			SOFTIRQENTRY_TEXT
 			ENTRY_TEXT
+			CPU_PRESERVED_TEXT
 			TEXT_TEXT
 			SCHED_TEXT
 			LOCK_TEXT
-- 
2.55.0.1082.g2b9226bbc0-goog

[RFC PATCH 26/46] oncore: Add on-core KHO ABI and public framework headers

[Pasha Tatashin]

Define the On-Core KHO serialization ABI (include/linux/kho/abi/cpu.h)
and public session and job management APIs (include/linux/oncore.h).

Signed-off-by: Pasha Tatashin <pasha.tatashin@soleen.com>
---
 arch/arm64/include/asm/oncore.h |  24 ++++++
 arch/x86/include/asm/oncore.h   |  25 +++++++
 include/linux/kho/abi/cpu.h     |   1 +
 include/linux/oncore.h          | 129 ++++++++++++++++++++++++++++++++
 kernel/liveupdate/Kconfig       |  10 +++
 5 files changed, 189 insertions(+)
 create mode 100644 arch/arm64/include/asm/oncore.h
 create mode 100644 arch/x86/include/asm/oncore.h
 create mode 100644 include/linux/oncore.h

diff --git a/arch/arm64/include/asm/oncore.h b/arch/arm64/include/asm/oncore.h
new file mode 100644
index 000000000000..fbd0f218d867
--- /dev/null
+++ b/arch/arm64/include/asm/oncore.h
@@ -0,0 +1,24 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright (c) 2026, Google LLC.
+ * Pasha Tatashin <pasha.tatashin@soleen.com>
+ *
+ * ARM64 architecture hooks for On-Core execution.
+ */
+#ifndef _ASM_ARM64_ONCORE_H
+#define _ASM_ARM64_ONCORE_H
+
+#include <linux/types.h>
+#include <asm/arch_timer.h>
+
+static inline u64 __cpu_preserved_text arch_oncore_read_counter(void)
+{
+	return __arch_counter_get_cntpct();
+}
+
+static inline u64 arch_oncore_counter_freq_hz(void)
+{
+	return arch_timer_get_cntfrq();
+}
+
+#endif /* _ASM_ARM64_ONCORE_H */
diff --git a/arch/x86/include/asm/oncore.h b/arch/x86/include/asm/oncore.h
new file mode 100644
index 000000000000..f64697a95d6a
--- /dev/null
+++ b/arch/x86/include/asm/oncore.h
@@ -0,0 +1,25 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright (c) 2026, Google LLC.
+ * Pasha Tatashin <pasha.tatashin@soleen.com>
+ *
+ * x86 architecture hooks for On-Core execution.
+ */
+#ifndef _ASM_X86_ONCORE_H
+#define _ASM_X86_ONCORE_H
+
+#include <linux/types.h>
+#include <asm/msr.h>
+#include <asm/tsc.h>
+
+static inline u64 __cpu_preserved_text arch_oncore_read_counter(void)
+{
+	return rdtsc();
+}
+
+static inline u64 arch_oncore_counter_freq_hz(void)
+{
+	return (u64)tsc_khz * 1000ULL;
+}
+
+#endif /* _ASM_X86_ONCORE_H */
diff --git a/include/linux/kho/abi/cpu.h b/include/linux/kho/abi/cpu.h
index f75bfb4c6161..4d63e623f162 100644
--- a/include/linux/kho/abi/cpu.h
+++ b/include/linux/kho/abi/cpu.h
@@ -145,6 +145,7 @@ struct cpu_preserved_file_ser {
 	u32 cpu;
 	u32 reserved;
 	u64 stack_pa;
+	DECLARE_KHOSER_PTR(oncore, struct oncore_session_ser *);
 } __packed;
 
 #endif /* _LINUX_KHO_ABI_CPU_H */
diff --git a/include/linux/oncore.h b/include/linux/oncore.h
new file mode 100644
index 000000000000..2817b0282a32
--- /dev/null
+++ b/include/linux/oncore.h
@@ -0,0 +1,129 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright (c) 2026, Google LLC.
+ * Pasha Tatashin <pasha.tatashin@soleen.com>
+ *
+ * On-Core Session and Scheduling Framework for Live Update
+ */
+#ifndef __LINUX_ONCORE_H
+#define __LINUX_ONCORE_H
+
+#include <linux/cpu_preserve.h>
+#include <linux/types.h>
+
+/*
+ * Only ever used as an opaque handle here, so do not include
+ * <linux/liveupdate.h>: this header is reached from <linux/kvm_host.h> via
+ * <linux/kvm_caretaker.h>, and including it would drag the entire LUO header
+ * stack into every KVM translation unit on every architecture.
+ */
+struct liveupdate_session;
+struct oncore_session;
+struct oncore_session_ser;
+struct oncore_job;
+
+#include <asm/oncore.h>
+
+/**
+ * enum oncore_exit_reason - Why an on-core job returned to the scheduler
+ * @ONCORE_EXIT_QUANTUM_EXPIRED: Time slice elapsed; the job is still runnable.
+ * @ONCORE_EXIT_ATTACH_SIGNALED: The incoming kernel asked for the CPU back.
+ * @ONCORE_EXIT_YIELD_IDLE: The job has no work right now (guest HLT/WFI).
+ * @ONCORE_EXIT_ERROR: The job hit an unrecoverable error and must be dropped.
+ * @ONCORE_EXIT_STALL: The job hit something it cannot handle on-core and
+ *                     cannot make forward progress until the incoming kernel
+ *                     reclaims it.  Unlike %ONCORE_EXIT_QUANTUM_EXPIRED, an
+ *                     immediate re-run is guaranteed to hit the same wall, so
+ *                     the scheduler backs off instead of spinning.
+ */
+enum oncore_exit_reason {
+	ONCORE_EXIT_QUANTUM_EXPIRED = 0,
+	ONCORE_EXIT_ATTACH_SIGNALED,
+	ONCORE_EXIT_YIELD_IDLE,
+	ONCORE_EXIT_ERROR,
+	ONCORE_EXIT_STALL,
+};
+
+typedef enum oncore_exit_reason (*oncore_job_fn)(void *data,
+						 u64 deadline_ticks);
+
+#ifdef CONFIG_LIVEUPDATE_ONCORE
+int oncore_session_add_cpu(struct liveupdate_session *s, int cpu);
+void oncore_session_remove_cpu(struct liveupdate_session *s, int cpu);
+struct oncore_session_ser *oncore_session_get_ser(struct liveupdate_session *s);
+void oncore_session_restore(struct liveupdate_session *s,
+			    struct oncore_session_ser *ser);
+phys_addr_t oncore_session_get_pgd_pa(struct oncore_session *sess);
+struct oncore_job *oncore_session_submit_job(struct liveupdate_session *s,
+					     oncore_job_fn run_fn,
+					     void *data);
+int oncore_session_activate_job(struct liveupdate_session *s,
+				struct oncore_job *job);
+void oncore_job_set_data(struct oncore_job *job, void *data);
+int oncore_job_cpu(const struct oncore_job *job);
+struct oncore_session *oncore_job_session(const struct oncore_job *job);
+int oncore_session_cancel_job(struct liveupdate_session *s,
+			      struct oncore_job *job);
+int oncore_session_map_range(struct oncore_session *sess, phys_addr_t pa,
+			     unsigned long va, size_t size, pgprot_t prot);
+int oncore_session_map_buffer(struct oncore_session *sess, void *va,
+			      size_t size);
+#else
+static inline int oncore_session_add_cpu(struct liveupdate_session *s, int cpu) { return 0; }
+
+static inline void oncore_session_remove_cpu(struct liveupdate_session *s, int cpu) {}
+
+static inline struct oncore_session_ser *
+oncore_session_get_ser(struct liveupdate_session *s)
+{
+	return NULL;
+}
+
+static inline void oncore_session_restore(struct liveupdate_session *s,
+					  struct oncore_session_ser *ser) {}
+
+static inline phys_addr_t oncore_session_get_pgd_pa(struct oncore_session *sess) { return 0; }
+
+static inline struct oncore_job *oncore_session_submit_job(struct liveupdate_session *s,
+							   oncore_job_fn run_fn,
+							   void *data)
+{
+	return NULL;
+}
+
+static inline int oncore_session_activate_job(struct liveupdate_session *s,
+					      struct oncore_job *job)
+{
+	return -EOPNOTSUPP;
+}
+
+static inline void oncore_job_set_data(struct oncore_job *job, void *data) {}
+
+static inline int oncore_job_cpu(const struct oncore_job *job) { return -1; }
+
+static inline struct oncore_session *
+oncore_job_session(const struct oncore_job *job)
+{
+	return NULL;
+}
+
+static inline int oncore_session_cancel_job(struct liveupdate_session *s,
+					    struct oncore_job *job)
+{
+	return 0;
+}
+
+static inline int oncore_session_map_range(struct oncore_session *sess, phys_addr_t pa,
+					   unsigned long va, size_t size, pgprot_t prot)
+{
+	return 0;
+}
+
+static inline int oncore_session_map_buffer(struct oncore_session *sess, void *va,
+					    size_t size)
+{
+	return 0;
+}
+#endif /* CONFIG_LIVEUPDATE_ONCORE */
+
+#endif /* __LINUX_ONCORE_H */
diff --git a/kernel/liveupdate/Kconfig b/kernel/liveupdate/Kconfig
index 52c5fd7a620b..0e1f72de2a48 100644
--- a/kernel/liveupdate/Kconfig
+++ b/kernel/liveupdate/Kconfig
@@ -136,4 +136,14 @@ config LIVEUPDATE_CPU
 
 	  If unsure, say N.
 
+config LIVEUPDATE_ONCORE
+	bool "On-core execution framework"
+	depends on LIVEUPDATE_CPU
+	default LIVEUPDATE_CPU
+	help
+	  Provide the on-core execution session and job scheduling framework
+	  for workloads running on preserved physical CPUs across a host
+	  live update (kexec handover). This allows subsystems such as KVM
+	  Caretaker to bind jobs to isolated physical cores during kexec.
+
 endmenu
-- 
2.55.0.1082.g2b9226bbc0-goog

[RFC PATCH 27/46] oncore: Implement on-core session lifecycle and scheduling loop

[Pasha Tatashin]

Implement On-Core session management, job submission and cancellation,
isolated address space mapping helpers, and the round-robin time-sliced
scheduling loop for preserved physical CPUs in kernel/liveupdate/oncore.c,
and include the On-Core kernel-doc documentation in
Documentation/liveupdate/cpu_preservation.rst.

Signed-off-by: Pasha Tatashin <pasha.tatashin@soleen.com>
---
 Documentation/liveupdate/cpu_preservation.rst |  14 +
 kernel/liveupdate/Makefile                    |  12 +
 kernel/liveupdate/cpu_preserve.c              |  15 +
 kernel/liveupdate/oncore.c                    | 947 ++++++++++++++++++
 4 files changed, 988 insertions(+)
 create mode 100644 kernel/liveupdate/oncore.c

diff --git a/Documentation/liveupdate/cpu_preservation.rst b/Documentation/liveupdate/cpu_preservation.rst
index a7a534188685..605d808e61c8 100644
--- a/Documentation/liveupdate/cpu_preservation.rst
+++ b/Documentation/liveupdate/cpu_preservation.rst
@@ -18,6 +18,20 @@ Architecture Backend Interface
 
 .. kernel-doc:: include/linux/cpu_preserve.h
 
+On-Core Execution and Scheduling Framework
+==========================================
+
+.. kernel-doc:: kernel/liveupdate/oncore.c
+   :doc: On-Core Execution and Scheduling Framework
+
+On-Core Session & Job API
+=========================
+
+.. kernel-doc:: include/linux/oncore.h
+
+.. kernel-doc:: kernel/liveupdate/oncore.c
+   :identifiers:
+
 CPU Preservation ABI
 ====================
 
diff --git a/kernel/liveupdate/Makefile b/kernel/liveupdate/Makefile
index 486f7854bb75..ab0d44079fb2 100644
--- a/kernel/liveupdate/Makefile
+++ b/kernel/liveupdate/Makefile
@@ -1,6 +1,17 @@
 # SPDX-License-Identifier: GPL-2.0
 
+KASAN_SANITIZE_cpu_preserve.o := n
+KCSAN_SANITIZE_cpu_preserve.o := n
+UBSAN_SANITIZE_cpu_preserve.o := n
+KCOV_INSTRUMENT_cpu_preserve.o := n
+KASAN_SANITIZE_oncore.o := n
+KCSAN_SANITIZE_oncore.o := n
+UBSAN_SANITIZE_oncore.o := n
+KCOV_INSTRUMENT_oncore.o := n
+CFLAGS_REMOVE_cpu_preserve.o = $(CC_FLAGS_FTRACE)
+CFLAGS_REMOVE_oncore.o = $(CC_FLAGS_FTRACE)
 CFLAGS_cpu_preserve.o += $(call cc-option,-mbranch-protection=none) -fno-stack-protector $(call cc-option,-ftrivial-auto-var-init=uninitialized) $(call cc-option,-fno-jump-tables)
+CFLAGS_oncore.o       += $(call cc-option,-mbranch-protection=none) -fno-stack-protector $(call cc-option,-ftrivial-auto-var-init=uninitialized) $(call cc-option,-fno-jump-tables)
 
 luo-y :=								\
 		kho_block.o						\
@@ -14,3 +25,4 @@ obj-$(CONFIG_KEXEC_HANDOVER_DEBUGFS)	+= kexec_handover_debugfs.o
 
 obj-$(CONFIG_LIVEUPDATE)		+= luo.o
 obj-$(CONFIG_LIVEUPDATE_CPU)		+= cpu_preserve.o
+obj-$(CONFIG_LIVEUPDATE_ONCORE)		+= oncore.o
diff --git a/kernel/liveupdate/cpu_preserve.c b/kernel/liveupdate/cpu_preserve.c
index 9a039ba9e912..430afbca860d 100644
--- a/kernel/liveupdate/cpu_preserve.c
+++ b/kernel/liveupdate/cpu_preserve.c
@@ -178,6 +178,7 @@
 #include <linux/liveupdate.h>
 #include <linux/mm.h>
 #include <linux/objtool.h>
+#include <linux/oncore.h>
 #include <linux/reboot.h>
 
 #include <asm/sections.h>
@@ -1552,14 +1553,22 @@ static int cpu_preserve_preserve(struct liveupdate_file_op_args *args)
 	if (ret)
 		return ret;
 
+	ret = oncore_session_add_cpu(args->session, cpu);
+	if (ret) {
+		cpu_unpreserve(cpu);
+		return ret;
+	}
+
 	fser = kho_alloc_preserve(sizeof(*fser));
 	if (IS_ERR(fser)) {
 		cpu_unpreserve(cpu);
+		oncore_session_remove_cpu(args->session, cpu);
 		return PTR_ERR(fser);
 	}
 
 	memset(fser, 0, sizeof(*fser));
 	fser->cpu = cpu;
+	KHOSER_STORE_PTR(fser->oncore, oncore_session_get_ser(args->session));
 
 	scoped_guard(mutex, &cpu_preserved_lock)
 		fser->stack_pa = cpu_preserved_outgoing.pcpus[cpu].stack_pa;
@@ -1580,6 +1589,7 @@ static void cpu_preserve_unpreserve(struct liveupdate_file_op_args *args)
 	cpu = fser->cpu;
 
 	cpu_unpreserve(cpu);
+	oncore_session_remove_cpu(args->session, cpu);
 
 	kho_unpreserve_free(fser);
 }
@@ -1587,8 +1597,12 @@ static void cpu_preserve_unpreserve(struct liveupdate_file_op_args *args)
 static void cpu_preserve_restore_incoming_cpu(struct liveupdate_session *session,
 					      struct cpu_preserved_file_ser *fser)
 {
+	struct oncore_session_ser *oncore = KHOSER_LOAD_PTR(fser->oncore);
 	unsigned int cpu = fser->cpu;
 
+	if (oncore)
+		oncore_session_restore(session, oncore);
+
 	scoped_guard(mutex, &cpu_preserved_lock) {
 		cpumask_set_cpu(cpu, &cpu_preserved_incoming.mask);
 		cpumask_set_cpu(cpu, &cpu_preserved_mask);
@@ -1646,6 +1660,7 @@ static void cpu_preserve_finish(struct liveupdate_file_op_args *args)
 		cpu_preserve_restore_incoming_cpu(args->session, fser);
 
 	cpu_unpreserve(fser->cpu);
+	oncore_session_remove_cpu(args->session, fser->cpu);
 
 	kho_restore_free(fser);
 }
diff --git a/kernel/liveupdate/oncore.c b/kernel/liveupdate/oncore.c
new file mode 100644
index 000000000000..196ed0a67563
--- /dev/null
+++ b/kernel/liveupdate/oncore.c
@@ -0,0 +1,947 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * Copyright (c) 2026, Google LLC.
+ * Pasha Tatashin <pasha.tatashin@soleen.com>
+ *
+ * On-Core Session and Scheduler Framework for Live Update
+ */
+
+/**
+ * DOC: On-Core Execution and Scheduling Framework
+ *
+ * The On-Core framework enables latency-sensitive workloads to continue
+ * executing directly on preserved physical CPUs (__cpu_preserved_text /
+ * __cpu_preserved_data) across a kexec Live Update (CONFIG_LIVEUPDATE_CPU)
+ * while the outgoing host kernel shuts down and the incoming kernel boots.
+ *
+ * Each Live Update session (struct liveupdate_session) can create an
+ * oncore_session that manages:
+ *
+ * 1. Isolated Address Space (struct cpu_preserved_as):
+ *    Page tables containing only the preserved runtime text/data sections,
+ *    the session and runqueue metadata, and workload-specific buffers
+ *    explicitly mapped via oncore_session_map_range() or
+ *    oncore_session_map_buffer().
+ *
+ * 2. Preserved Physical CPU Pool:
+ *    One or more physical CPUs isolated from Linux scheduling via
+ *    cpu_preserve and attached to the session via oncore_session_add_cpu().
+ *    Each preserved CPU switches to the session's isolated page tables and
+ *    executes oncore_cpu_schedule_loop().
+ *
+ * 3. Cooperative Time-Sliced Runqueue (struct oncore_runqueue):
+ *    A lockless/atomic round-robin FIFO scheduler supporting M jobs across N
+ *    preserved physical CPUs (including M > N oversubscription).  It provides
+ *    initial-run starvation avoidance, least-loaded CPU assignment, work
+ *    stealing, single-job fast-path continuation, and low-power backoff (WFE on
+ *    arm64, PAUSE on x86) when the queue is idle or a job returns
+ *    ONCORE_EXIT_STALL.
+ *
+ * Integration with KVM Caretaker and Future Kernel Caretaker Workloads
+ * --------------------------------------------------------------------
+ * On-Core is workload- and hypervisor-agnostic:
+ *
+ * - KVM Caretaker: During LUO prepare/freeze, KVM detaches each vCPU into a
+ *   self-contained KHO-preserved Caretaker page (caretaker_x86_page on x86,
+ *   caretaker_arm64_page on arm64), maps the page and guest/virtualization
+ *   structures into the session's address space, and submits an oncore_job
+ *   whose callback enters the guest (VMLAUNCH/VMRESUME, VMRUN, or EL2 world
+ *   switch) until the time-slice deadline (deadline_ticks) expires, the vCPU
+ *   yields on HLT/WFI (ONCORE_EXIT_YIELD_IDLE), hits an exit requiring the
+ *   incoming kernel (ONCORE_EXIT_STALL), or observes the incoming kernel's
+ *   reclaim signal (ONCORE_EXIT_ATTACH_SIGNALED).
+ *
+ * - Future Kernel Caretaker Workloads: Any self-contained kernel subsystem
+ *   compiled into the preserved runtime sections (such as hardware watchdog
+ *   feeders, health/heartbeat responders, or zero-loss network/storage
+ *   polling drivers) can submit an oncore_job_fn(void *data, u64 deadline) to
+ *   share preserved physical cores alongside or independently of KVM vCPUs.
+ *
+ * Architecture Requirements
+ * -------------------------
+ * To support CONFIG_LIVEUPDATE_ONCORE, an architecture must implement:
+ *
+ * - <asm/oncore.h>:
+ *   - arch_oncore_read_counter(): Read a monotonic hardware counter directly
+ *     from __cpu_preserved_text without relying on kernel timekeeping
+ *     (e.g., rdtsc() on x86, __arch_counter_get_cntpct() on arm64).
+ *   - arch_oncore_counter_freq_hz(): Return the hardware counter frequency in
+ *     Hz used to convert the time quantum (oncore.quantum_ms) into ticks.
+ *
+ * - <asm/cpu_preserve.h> (from CONFIG_LIVEUPDATE_CPU):
+ *   - arch_cpu_preserved_switch_pgd(): Switch to the session's isolated PGD.
+ *   - arch_cpu_preserved_park_wait() / arch_cpu_preserved_kick(): Low-power
+ *     wait instruction and cross-CPU wakeup mechanism.
+ *
+ * Public Interfaces (<linux/oncore.h>)
+ * ------------------------------------
+ * - Session & CPU Lifecycle:
+ *   oncore_session_add_cpu(), oncore_session_remove_cpu(),
+ *   oncore_session_get_ser(), oncore_session_restore().
+ * - Session Address Space Mapping:
+ *   oncore_session_map_range(), oncore_session_map_buffer(),
+ *   oncore_session_get_pgd_pa().
+ * - Job Submission & Control:
+ *   oncore_session_submit_job(), oncore_job_set_data(),
+ *   oncore_job_cpu(), oncore_job_session(),
+ *   oncore_session_activate_job(), oncore_session_cancel_job().
+ */
+
+#define pr_fmt(fmt) "oncore: " fmt
+
+#include <linux/cpu_preserve.h>
+#include <linux/cpumask.h>
+#include <linux/delay.h>
+#include <linux/init.h>
+#include <linux/io.h>
+#include <linux/kexec.h>
+#include <linux/kexec_handover.h>
+#include <linux/list.h>
+#include <linux/liveupdate.h>
+#include <linux/mm.h>
+#include <linux/module.h>
+#include <linux/mutex.h>
+#include <linux/oncore.h>
+#include <linux/overflow.h>
+#include <linux/slab.h>
+#include <linux/string.h>
+#include <linux/types.h>
+
+#define ONCORE_DEFAULT_QUANTUM_MS	10
+#define ONCORE_CANCEL_TIMEOUT_US	1000000
+#define ONCORE_CANCEL_STEP_US		100
+
+enum oncore_job_state {
+	ONCORE_JOB_NEW = 0,
+	ONCORE_JOB_RUNNABLE,
+	ONCORE_JOB_RUNNING,
+	ONCORE_JOB_CANCELING,
+	ONCORE_JOB_DEAD,
+};
+
+struct oncore_job {
+	struct list_head		node;
+	struct list_head		sess_node;
+	phys_addr_t			next_job_pa;
+	struct oncore_session		*session;
+	enum oncore_job_state		state;
+	oncore_job_fn			run_fn;
+	void				*data;
+	int				assigned_cpu;
+	int				last_cpu;
+	u64				total_runs;
+};
+
+struct oncore_sched_config {
+	u32				quantum_ms;
+	u64				quantum_ticks;
+	u64				counter_freq_hz;
+};
+
+struct oncore_runqueue {
+	atomic_t			lock;
+	struct list_head		runnable;
+	unsigned int			nr_runnable;
+};
+
+struct oncore_session {
+	struct list_head		node;
+	/* Protects session state and job list */
+	struct mutex			lock;
+	struct list_head		jobs;
+	phys_addr_t			first_job_pa;
+	struct oncore_runqueue		rq;
+	struct oncore_sched_config	sched_config;
+	struct cpu_preserved_as		*as;
+	struct oncore_session_ser	*ser;
+	bool				is_incoming;
+};
+
+static struct oncore_sched_config global_oncore_sched_config __cpu_preserved_data;
+static DEFINE_MUTEX(oncore_sessions_lock);
+static LIST_HEAD(oncore_sessions);
+
+static inline struct cpumask *oncore_session_cpus(struct oncore_session *sess)
+{
+	return to_cpumask((unsigned long *)sess->ser->cpus_bitmap);
+}
+
+static inline void oncore_rq_lock(struct oncore_runqueue *rq)
+{
+	while (atomic_cmpxchg_acquire(&rq->lock, 0, 1) != 0) {
+		while (atomic_read(&rq->lock) != 0)
+			cpu_relax();
+	}
+}
+
+static inline void oncore_rq_unlock(struct oncore_runqueue *rq)
+{
+	atomic_set_release(&rq->lock, 0);
+}
+
+static int __init parse_oncore_quantum(char *arg)
+{
+	u32 val;
+
+	if (kstrtou32(arg, 0, &val) == 0 && val >= 1 && val <= 1000)
+		global_oncore_sched_config.quantum_ms = val;
+	return 0;
+}
+early_param("oncore.quantum_ms", parse_oncore_quantum);
+
+static void oncore_sched_update_ticks(void)
+{
+	u32 ms = global_oncore_sched_config.quantum_ms ? : ONCORE_DEFAULT_QUANTUM_MS;
+	u64 freq = arch_oncore_counter_freq_hz();
+
+	global_oncore_sched_config.counter_freq_hz = freq;
+	global_oncore_sched_config.quantum_ticks = (freq * ms) / 1000ULL;
+}
+
+static void oncore_runqueue_init(struct oncore_runqueue *rq)
+{
+	atomic_set(&rq->lock, 0);
+	INIT_LIST_HEAD(&rq->runnable);
+	rq->nr_runnable = 0;
+}
+
+static int oncore_sched_enqueue(struct oncore_runqueue *rq,
+				struct oncore_job *job)
+{
+	int cpu;
+
+	oncore_rq_lock(rq);
+	job->state = ONCORE_JOB_RUNNABLE;
+	list_add_tail(&job->node, &rq->runnable);
+	rq->nr_runnable++;
+	oncore_rq_unlock(rq);
+
+	for_each_cpu(cpu, oncore_session_cpus(job->session)) {
+		if (cpu_is_preserved(cpu))
+			arch_cpu_preserved_kick(cpu);
+	}
+
+	return 0;
+}
+
+static int oncore_sched_dequeue(struct oncore_runqueue *rq,
+				struct oncore_job *job)
+{
+	oncore_rq_lock(rq);
+	if (!list_empty(&job->node)) {
+		list_del_init(&job->node);
+		rq->nr_runnable--;
+	}
+	if (job->state == ONCORE_JOB_RUNNING)
+		job->state = ONCORE_JOB_CANCELING;
+	else
+		job->state = ONCORE_JOB_DEAD;
+	oncore_rq_unlock(rq);
+	return 0;
+}
+
+static struct oncore_job *__cpu_preserved_text
+oncore_sched_pick_next(struct oncore_runqueue *rq, int cpu)
+{
+	struct oncore_job *job = NULL, *iter;
+
+	if (!rq || READ_ONCE(rq->nr_runnable) == 0)
+		return NULL;
+
+	oncore_rq_lock(rq);
+
+	/* 1. Starvation avoidance: if head job has NEVER run, take it immediately */
+	if (!list_empty(&rq->runnable)) {
+		iter = list_first_entry(&rq->runnable, struct oncore_job, node);
+		if (iter->total_runs == 0) {
+			job = iter;
+			goto found;
+		}
+	}
+
+	/* 2. Prefer job affine to this core */
+	list_for_each_entry(iter, &rq->runnable, node) {
+		if (iter->assigned_cpu == cpu) {
+			job = iter;
+			goto found;
+		}
+	}
+
+	/* 3. Work-stealing fallback: take oldest job from head of queue */
+	if (!list_empty(&rq->runnable))
+		job = list_first_entry(&rq->runnable, struct oncore_job, node);
+
+found:
+	if (job) {
+		list_del_init(&job->node);
+		rq->nr_runnable--;
+	}
+
+	oncore_rq_unlock(rq);
+	return job;
+}
+
+static void __cpu_preserved_text
+oncore_sched_put_prev(struct oncore_runqueue *rq,
+		      struct oncore_job *job)
+{
+	oncore_rq_lock(rq);
+	job->state = ONCORE_JOB_RUNNABLE;
+	list_add_tail(&job->node, &rq->runnable);
+	rq->nr_runnable++;
+	oncore_rq_unlock(rq);
+}
+
+static void __cpu_preserved_text
+oncore_cpu_schedule_loop(int cpu, struct oncore_runqueue *rq,
+			 struct oncore_sched_config *cfg)
+{
+	enum oncore_exit_reason reason;
+	struct oncore_job *curr = NULL;
+	u64 deadline;
+
+	if (!rq || !cfg)
+		return;
+
+	while (!cpu_preserved_should_exit(cpu)) {
+		/* 1. Pick the next runnable job from the FIFO queue */
+		if (!curr) {
+			curr = oncore_sched_pick_next(rq, cpu);
+			if (!curr) {
+				/* No runnable jobs; execute low-power park wait */
+				arch_cpu_preserved_park_wait();
+				continue;
+			}
+		}
+
+		/* 2. Compute quantum deadline */
+		deadline = arch_oncore_read_counter() + cfg->quantum_ticks;
+
+		/* 3. Execute workload on physical silicon */
+		curr->last_cpu = cpu;
+		curr->state = ONCORE_JOB_RUNNING;
+		reason = curr->run_fn(curr->data, deadline);
+
+		/* 4. Update telemetry and accounting */
+		curr->total_runs++;
+
+		/*
+		 * A stalled job cannot make progress until the incoming kernel
+		 * reclaims it, so re-running it immediately would just repeat
+		 * the same unhandled exit as fast as the hardware allows.  Back
+		 * off in the architecture's low-power wait (WFE on arm64, PAUSE
+		 * on x86) first.  arch_cpu_preserved_kick() wakes it, so the
+		 * attach signal is still observed promptly by the checks below
+		 * and at the top of the loop.
+		 */
+		if (reason == ONCORE_EXIT_STALL)
+			arch_cpu_preserved_park_wait();
+
+		/* Fast-path: single runnable job continues uninterrupted */
+		if (READ_ONCE(rq->nr_runnable) == 0 &&
+		    READ_ONCE(curr->state) == ONCORE_JOB_RUNNING &&
+		    reason != ONCORE_EXIT_ATTACH_SIGNALED &&
+		    reason != ONCORE_EXIT_ERROR &&
+		    !cpu_preserved_should_exit(cpu)) {
+			continue;
+		}
+
+		/* 5. Handle exit and return job to queue */
+		oncore_rq_lock(rq);
+		if (curr->state == ONCORE_JOB_CANCELING ||
+		    curr->state == ONCORE_JOB_DEAD ||
+		    reason == ONCORE_EXIT_ATTACH_SIGNALED ||
+		    reason == ONCORE_EXIT_ERROR) {
+			WRITE_ONCE(curr->state, ONCORE_JOB_DEAD);
+			oncore_rq_unlock(rq);
+			curr = NULL;
+			continue;
+		}
+		oncore_rq_unlock(rq);
+
+		oncore_sched_put_prev(rq, curr);
+		curr = NULL;
+	}
+}
+
+static void __cpu_preserved_text oncore_sched_cpu_worker(void *data)
+{
+	struct cpu_preserved_stack_context *sctx = cpu_preserved_get_stack_context();
+	struct oncore_session *sess;
+	int cpu;
+
+	if (!sctx)
+		return;
+
+	sess = sctx->workload_context ?
+		(struct oncore_session *)(uintptr_t)sctx->workload_context :
+		data;
+	if (!sess)
+		return;
+	cpu = sctx->cpu;
+
+	if (sctx->session_pgd_pa)
+		arch_cpu_preserved_switch_pgd(sctx->session_pgd_pa);
+
+	oncore_cpu_schedule_loop(cpu, &sess->rq, &sess->sched_config);
+}
+
+static int __init oncore_sched_init(void)
+{
+	if (!arch_oncore_counter_freq_hz()) {
+		pr_err("Counter frequency is unknown; on-core scheduler disabled\n");
+		return -ENODEV;
+	}
+	if (!global_oncore_sched_config.quantum_ms)
+		global_oncore_sched_config.quantum_ms = ONCORE_DEFAULT_QUANTUM_MS;
+	oncore_sched_update_ticks();
+	pr_info("Round-Robin scheduler initialized (quantum=%u ms, ticks=%llu)\n",
+		global_oncore_sched_config.quantum_ms,
+		global_oncore_sched_config.quantum_ticks);
+	return 0;
+}
+early_initcall(oncore_sched_init);
+
+/**
+ * oncore_session_map_range - Map a physical memory range into the session's address space
+ * @sess: On-Core session (may be %NULL to map into the global preserved address space).
+ * @pa:   Start physical address of the range.
+ * @va:   Target virtual address in the isolated page tables.
+ * @size: Size of the mapping in bytes.
+ * @prot: Page protection flags.
+ *
+ * Maps `[pa, pa + size)` at `@va` in the isolated page tables used by
+ * preserved CPUs executing workloads in @sess across kexec.
+ *
+ * Return: 0 on success, or a negative errno on failure.
+ */
+int oncore_session_map_range(struct oncore_session *sess, phys_addr_t pa,
+			     unsigned long va, size_t size, pgprot_t prot)
+{
+	return cpu_preserved_map_range(pa, va, size, prot);
+}
+EXPORT_SYMBOL_GPL(oncore_session_map_range);
+
+/**
+ * oncore_session_map_buffer - Map a direct-map kernel buffer into the session's address space
+ * @sess: On-Core session.
+ * @va:   Kernel direct-map virtual address of the buffer (no-op if %NULL).
+ * @size: Size of the buffer in bytes (no-op if 0).
+ *
+ * Convenience helper that resolves `virt_to_phys(@va)` and maps the buffer at
+ * its existing kernel virtual address `@va` with %PAGE_KERNEL permissions in
+ * the session's isolated page tables.
+ *
+ * Return: 0 on success, or a negative errno on failure.
+ */
+int oncore_session_map_buffer(struct oncore_session *sess, void *va,
+			      size_t size)
+{
+	if (!va || !size)
+		return 0;
+	return oncore_session_map_range(sess, virt_to_phys(va),
+					(unsigned long)va, size, PAGE_KERNEL);
+}
+EXPORT_SYMBOL_GPL(oncore_session_map_buffer);
+
+/**
+ * oncore_session_get_pgd_pa - Return the root page table physical address of a session
+ * @sess: On-Core session.
+ *
+ * Return: Physical address of the session's isolated PGD, or 0 if @sess or its
+ *         address space is %NULL.
+ */
+phys_addr_t oncore_session_get_pgd_pa(struct oncore_session *sess)
+{
+	return (sess && sess->as) ? sess->as->pgd_pa : 0;
+}
+EXPORT_SYMBOL_GPL(oncore_session_get_pgd_pa);
+
+static struct oncore_session *oncore_find_session_locked(const char *sname)
+{
+	struct oncore_session *sess;
+
+	if (!sname || !sname[0])
+		return NULL;
+
+	list_for_each_entry(sess, &oncore_sessions, node) {
+		if (strcmp(sess->ser->session_name, sname) == 0)
+			return sess;
+	}
+
+	return NULL;
+}
+
+static struct oncore_session *oncore_find_session(struct liveupdate_session *s)
+{
+	guard(mutex)(&oncore_sessions_lock);
+
+	return oncore_find_session_locked(liveupdate_session_name(s));
+}
+
+static struct oncore_session *oncore_create_session_locked(const char *sname)
+{
+	unsigned int nr_ser_words = BITS_TO_U64(nr_cpu_ids);
+	struct oncore_session *sess;
+	size_t ser_sz;
+
+	ser_sz = struct_size(sess->ser, cpus_bitmap, nr_ser_words);
+
+	sess = kho_alloc_preserve(sizeof(*sess));
+	if (IS_ERR(sess))
+		return NULL;
+
+	memset(sess, 0, sizeof(*sess));
+	mutex_init(&sess->lock);
+	INIT_LIST_HEAD(&sess->jobs);
+	oncore_runqueue_init(&sess->rq);
+	oncore_sched_update_ticks();
+	sess->sched_config = global_oncore_sched_config;
+
+	sess->as = cpu_preserved_as_create();
+	if (IS_ERR(sess->as))
+		sess->as = NULL;
+
+	oncore_session_map_range(sess, virt_to_phys(sess),
+				 (unsigned long)sess, sizeof(*sess),
+				 PAGE_KERNEL);
+
+	sess->ser = kho_alloc_preserve(ser_sz);
+	if (IS_ERR(sess->ser)) {
+		cpu_preserved_as_destroy(sess->as);
+		kho_unpreserve_free(sess);
+		return NULL;
+	}
+
+	memset(sess->ser, 0, ser_sz);
+	sess->ser->nr_cpu_words = nr_ser_words;
+	strscpy(sess->ser->session_name, sname, sizeof(sess->ser->session_name));
+	sess->ser->sess_pa = virt_to_phys(sess);
+	KHOSER_STORE_PTR(sess->ser->as, sess->as ? sess->as->ser : NULL);
+
+	list_add_tail(&sess->node, &oncore_sessions);
+	return sess;
+}
+
+static struct oncore_session *oncore_get_or_create_session(struct liveupdate_session *s)
+{
+	const char *sname = liveupdate_session_name(s);
+	struct oncore_session *sess;
+
+	if (!sname || !sname[0])
+		return NULL;
+
+	guard(mutex)(&oncore_sessions_lock);
+
+	sess = oncore_find_session_locked(sname);
+	if (sess)
+		return sess;
+
+	return oncore_create_session_locked(sname);
+}
+
+/**
+ * oncore_session_add_cpu - Attach a preserved physical CPU to an On-Core session
+ * @s:   Live Update session handle.
+ * @cpu: Logical ID of the preserved physical CPU to add.
+ *
+ * Creates the On-Core session for @s if it does not yet exist, adds @cpu to
+ * the session's CPU mask, configures @cpu's preserved stack context with the
+ * session pointer and isolated PGD physical address, and attaches
+ * oncore_sched_cpu_worker() so @cpu begins servicing the session's runqueue.
+ *
+ * Return: 0 on success, or a negative errno on failure.
+ */
+int oncore_session_add_cpu(struct liveupdate_session *s, int cpu)
+{
+	struct oncore_session *sess;
+	int ret = 0;
+
+	if (!arch_oncore_counter_freq_hz()) {
+		pr_err("On-core counter frequency is unknown; refusing to add CPU\n");
+		return -ENODEV;
+	}
+
+	sess = oncore_get_or_create_session(s);
+	if (!sess || cpu < 0 || cpu >= nr_cpu_ids)
+		return -EINVAL;
+
+	guard(mutex)(&sess->lock);
+	cpumask_set_cpu(cpu, oncore_session_cpus(sess));
+
+	cpu_preserved_set_workload_context(cpu, sess,
+					   oncore_session_get_pgd_pa(sess));
+
+	ret = cpu_preserved_attach_workload(cpu,
+					    oncore_sched_cpu_worker,
+					    sess);
+	if (ret) {
+		cpu_preserved_set_workload_context(cpu, NULL, 0);
+		cpumask_clear_cpu(cpu, oncore_session_cpus(sess));
+		return ret;
+	}
+
+	return 0;
+}
+
+static void oncore_sync_jobs_pa(struct oncore_session *sess)
+{
+	phys_addr_t *tail = &sess->first_job_pa;
+	struct oncore_job *j;
+
+	list_for_each_entry(j, &sess->jobs, sess_node) {
+		j->next_job_pa = 0;
+		*tail = virt_to_phys(j);
+		tail = &j->next_job_pa;
+	}
+	*tail = 0;
+}
+
+static void oncore_session_destroy_incoming(struct oncore_session *sess)
+{
+	struct oncore_session *old_sess;
+	struct oncore_job *job;
+	phys_addr_t job_pa;
+
+	if (!sess->ser) {
+		kfree(sess);
+		return;
+	}
+
+	if (sess->ser->sess_pa) {
+		old_sess = phys_to_virt(sess->ser->sess_pa);
+		job_pa = old_sess->first_job_pa;
+		while (job_pa) {
+			job = phys_to_virt(job_pa);
+			job_pa = job->next_job_pa;
+			kho_restore_free(job);
+		}
+		kho_restore_free(old_sess);
+	}
+
+	kho_restore_free(sess->ser);
+	kfree(sess);
+}
+
+static void oncore_session_destroy_outgoing(struct oncore_session *sess)
+{
+	struct oncore_job *job, *tmp;
+
+	list_for_each_entry_safe(job, tmp, &sess->jobs, sess_node) {
+		list_del_init(&job->sess_node);
+		kho_unpreserve_free(job);
+	}
+
+	if (sess->ser)
+		kho_unpreserve_free(sess->ser);
+	kho_unpreserve_free(sess);
+}
+
+static void oncore_session_destroy(struct oncore_session *sess)
+{
+	scoped_guard(mutex, &oncore_sessions_lock)
+		list_del_init(&sess->node);
+
+	cpu_preserved_as_destroy(sess->as);
+
+	if (sess->is_incoming)
+		oncore_session_destroy_incoming(sess);
+	else
+		oncore_session_destroy_outgoing(sess);
+}
+
+/**
+ * oncore_session_remove_cpu - Detach a preserved physical CPU from an On-Core session
+ * @s:   Live Update session handle.
+ * @cpu: Logical ID of the preserved physical CPU to remove.
+ *
+ * Detaches the On-Core scheduler worker from @cpu and clears @cpu from the
+ * session's CPU mask.  When the last CPU in the session is removed, destroys
+ * the session's isolated address space and reclaims all KHO-preserved job and
+ * session structures (via kho_restore_free() in the incoming kernel after
+ * kexec, or kho_unpreserve_free() in the outgoing kernel on cancellation).
+ */
+void oncore_session_remove_cpu(struct liveupdate_session *s, int cpu)
+{
+	struct oncore_session *sess = oncore_find_session(s);
+
+	if (!sess || cpu < 0 || cpu >= nr_cpu_ids)
+		return;
+
+	scoped_guard(mutex, &sess->lock)
+		cpumask_clear_cpu(cpu, oncore_session_cpus(sess));
+
+	cpu_preserved_detach_workload(cpu);
+	cpu_preserved_set_workload_context(cpu, NULL, 0);
+
+	if (cpumask_empty(oncore_session_cpus(sess)))
+		oncore_session_destroy(sess);
+}
+
+/**
+ * oncore_session_get_ser - Retrieve the KHO serialization block for an On-Core session
+ * @s: Live Update session handle.
+ *
+ * Called by the preserved-CPU LUO file handler (cpu_preserve) when serializing
+ * a preserved CPU file descriptor so it can store a KHO pointer
+ * (&cpu_preserved_file_ser.oncore) to the session's metadata across kexec.
+ *
+ * Return: Pointer to the KHO-preserved &struct oncore_session_ser for @s, or
+ *         %NULL if @s has no On-Core session.
+ */
+struct oncore_session_ser *oncore_session_get_ser(struct liveupdate_session *s)
+{
+	struct oncore_session *sess = oncore_find_session(s);
+
+	return sess ? sess->ser : NULL;
+}
+
+/**
+ * oncore_session_restore - Reconstruct an On-Core session in the incoming kernel
+ * @s:   Incoming Live Update session handle.
+ * @ser: KHO-preserved session metadata handed over from the outgoing kernel.
+ *
+ * Adopts the KHO-preserved session metadata and isolated page tables (@ser->as)
+ * in the incoming kernel so that subsequent calls to oncore_session_remove_cpu()
+ * as preserved CPUs reattach to Linux can cleanly release the preserved page
+ * tables, jobs, and session structures.  Idempotent if @s has already been
+ * restored.
+ */
+void oncore_session_restore(struct liveupdate_session *s,
+			    struct oncore_session_ser *ser)
+{
+	struct cpu_preserved_as_ser *as_ser;
+	struct oncore_session *sess;
+	const char *sname;
+
+	if (!ser)
+		return;
+
+	sname = liveupdate_session_name(s);
+	if (!sname || !sname[0])
+		sname = ser->session_name;
+
+	guard(mutex)(&oncore_sessions_lock);
+
+	if (oncore_find_session_locked(sname))
+		return;
+
+	sess = kzalloc_obj(*sess, GFP_KERNEL);
+	if (!sess)
+		return;
+
+	mutex_init(&sess->lock);
+	INIT_LIST_HEAD(&sess->jobs);
+	oncore_runqueue_init(&sess->rq);
+	sess->is_incoming = true;
+	sess->ser = ser;
+
+	as_ser = KHOSER_LOAD_PTR(ser->as);
+	if (as_ser)
+		sess->as = cpu_preserved_as_adopt(as_ser);
+
+	list_add_tail(&sess->node, &oncore_sessions);
+}
+
+static unsigned int oncore_session_cpu_job_count(struct oncore_session *sess,
+						 int cpu)
+{
+	struct oncore_job *j;
+	unsigned int count = 0;
+
+	list_for_each_entry(j, &sess->jobs, sess_node) {
+		if (j->assigned_cpu == cpu)
+			count++;
+	}
+
+	return count;
+}
+
+static int oncore_select_job_cpu(struct oncore_session *sess)
+{
+	struct cpumask *cpus = oncore_session_cpus(sess);
+	unsigned int min_count = UINT_MAX, count;
+	int cpu, min_cpu = -1;
+
+	for_each_cpu(cpu, cpus) {
+		count = oncore_session_cpu_job_count(sess, cpu);
+		if (count < min_count) {
+			min_count = count;
+			min_cpu = cpu;
+		}
+	}
+
+	return min_cpu;
+}
+
+/**
+ * oncore_session_submit_job - Allocate and register a workload job in an On-Core session
+ * @s:      Live Update session handle.
+ * @run_fn: Workload callback executed on a preserved physical CPU.
+ * @data:   Opaque context pointer passed to @run_fn (may be %NULL and set
+ *          later via oncore_job_set_data() before activation).
+ *
+ * Allocates a KHO-preserved &struct oncore_job, maps it into the session's
+ * isolated address space, assigns it to the least-loaded preserved CPU in the
+ * session, and links it into @s.  The job is not placed on the runqueue until
+ * oncore_session_activate_job() is called.
+ *
+ * Return: Pointer to the allocated &struct oncore_job on success, %NULL if @s
+ *         has no preserved CPUs, or an ERR_PTR() on failure.
+ */
+struct oncore_job *oncore_session_submit_job(struct liveupdate_session *s,
+					     oncore_job_fn run_fn,
+					     void *data)
+{
+	struct oncore_session *sess = oncore_find_session(s);
+	struct oncore_job *job;
+
+	if (!run_fn)
+		return ERR_PTR(-EINVAL);
+	if (!sess)
+		return NULL;
+
+	guard(mutex)(&sess->lock);
+	if (cpumask_empty(oncore_session_cpus(sess)))
+		return NULL;
+
+	job = kho_alloc_preserve(sizeof(*job));
+	if (IS_ERR(job))
+		return job;
+
+	memset(job, 0, sizeof(*job));
+	oncore_session_map_buffer(sess, job, sizeof(*job));
+	INIT_LIST_HEAD(&job->node);
+	INIT_LIST_HEAD(&job->sess_node);
+	job->session = sess;
+	job->state = ONCORE_JOB_NEW;
+	job->run_fn = run_fn;
+	job->data = data;
+	job->last_cpu = -1;
+	job->assigned_cpu = oncore_select_job_cpu(sess);
+
+	list_add_tail(&job->sess_node, &sess->jobs);
+	oncore_sync_jobs_pa(sess);
+
+	return job;
+}
+EXPORT_SYMBOL_GPL(oncore_session_submit_job);
+
+/**
+ * oncore_session_activate_job - Map a submitted job's data and enqueue it for execution
+ * @s:   Live Update session handle.
+ * @job: Job previously returned by oncore_session_submit_job().
+ *
+ * Maps the first page of @job->data (if non-%NULL) into the session's isolated
+ * address space, places @job onto the session's round-robin FIFO runqueue, and
+ * kicks the assigned preserved physical CPU so it wakes from low-power park
+ * wait and begins executing @job.
+ *
+ * Return: 0 on success, or -EINVAL if @s or @job is invalid.
+ */
+int oncore_session_activate_job(struct liveupdate_session *s,
+				struct oncore_job *job)
+{
+	struct oncore_session *sess = oncore_find_session(s);
+
+	if (!sess || !job)
+		return -EINVAL;
+
+	guard(mutex)(&sess->lock);
+	if (job->data)
+		oncore_session_map_buffer(sess, job->data, PAGE_SIZE);
+	oncore_sched_enqueue(&sess->rq, job);
+	if (cpu_is_preserved(job->assigned_cpu))
+		arch_cpu_preserved_kick(job->assigned_cpu);
+
+	return 0;
+}
+EXPORT_SYMBOL_GPL(oncore_session_activate_job);
+
+/**
+ * oncore_job_set_data - Set the opaque argument passed to a job's run callback
+ * @job:  Job to update.
+ * @data: Pointer handed to @job's run_fn.  May be %NULL.
+ *
+ * Callers that cannot determine @data at submission time submit with %NULL and
+ * call this once the object exists.  It must be called before
+ * oncore_session_activate_job(), which is what maps @data into the session's
+ * address space.
+ */
+void oncore_job_set_data(struct oncore_job *job, void *data)
+{
+	if (job)
+		job->data = data;
+}
+EXPORT_SYMBOL_GPL(oncore_job_set_data);
+
+/**
+ * oncore_job_cpu - Return the preserved physical CPU assigned to a job
+ * @job: Job to query.
+ *
+ * Return: Logical CPU ID assigned to @job, or -1 if @job is %NULL.
+ */
+int oncore_job_cpu(const struct oncore_job *job)
+{
+	return job ? job->assigned_cpu : -1;
+}
+EXPORT_SYMBOL_GPL(oncore_job_cpu);
+
+/**
+ * oncore_job_session - Return the On-Core session that owns a job
+ * @job: Job to query.
+ *
+ * Return: Pointer to the owning &struct oncore_session, or %NULL if @job is %NULL.
+ */
+struct oncore_session *oncore_job_session(const struct oncore_job *job)
+{
+	return job ? job->session : NULL;
+}
+EXPORT_SYMBOL_GPL(oncore_job_session);
+
+/**
+ * oncore_session_cancel_job - Stop and free a submitted or running On-Core job
+ * @s:   Live Update session handle.
+ * @job: Job to cancel.
+ *
+ * Removes @job from the session and runqueue.  If @job is currently executing
+ * on a preserved physical CPU (%ONCORE_JOB_CANCELING), kicks that CPU and
+ * waits for the current scheduling quantum to finish before unpreserving and
+ * freeing @job.
+ *
+ * Return: 0 on success, or -EINVAL if @s or @job is invalid.
+ */
+int oncore_session_cancel_job(struct liveupdate_session *s,
+			      struct oncore_job *job)
+{
+	struct oncore_session *sess = oncore_find_session(s);
+	int cpu, retries;
+
+	if (!sess || !job)
+		return -EINVAL;
+
+	guard(mutex)(&sess->lock);
+	list_del_init(&job->sess_node);
+	oncore_sync_jobs_pa(sess);
+	if (job->assigned_cpu >= 0)
+		job->assigned_cpu = -1;
+	oncore_sched_dequeue(&sess->rq, job);
+
+	if (READ_ONCE(job->state) == ONCORE_JOB_CANCELING) {
+		cpu = READ_ONCE(job->last_cpu);
+		retries = 0;
+
+		while (READ_ONCE(job->state) == ONCORE_JOB_CANCELING &&
+		       retries < (ONCORE_CANCEL_TIMEOUT_US / ONCORE_CANCEL_STEP_US)) {
+			if ((retries % 50) == 0 && cpu >= 0)
+				arch_cpu_preserved_kick(cpu);
+			udelay(ONCORE_CANCEL_STEP_US);
+			retries++;
+		}
+	}
+
+	kho_unpreserve_free(job);
+
+	return 0;
+}
+EXPORT_SYMBOL_GPL(oncore_session_cancel_job);
-- 
2.55.0.1082.g2b9226bbc0-goog

[RFC PATCH 28/46] KVM: caretaker: Add Caretaker control block and architecture ops headers

[Pasha Tatashin]

Define struct kvm_caretaker_cb, struct kvm_caretaker_ops, and
KVM_CAP_CARETAKER documentation for OrphanVM on-core execution.

Signed-off-by: Pasha Tatashin <pasha.tatashin@soleen.com>
---
 Documentation/virt/kvm/api.rst |  16 ++
 include/linux/kho/abi/kvm.h    | 112 ++++++++++-
 include/linux/kvm_caretaker.h  | 341 +++++++++++++++++++++++++++++++++
 include/linux/kvm_host.h       |  10 +
 include/uapi/linux/kvm.h       |   1 +
 5 files changed, 479 insertions(+), 1 deletion(-)
 create mode 100644 include/linux/kvm_caretaker.h

diff --git a/Documentation/virt/kvm/api.rst b/Documentation/virt/kvm/api.rst
index e0430cc750c9..a5c7a48617c3 100644
--- a/Documentation/virt/kvm/api.rst
+++ b/Documentation/virt/kvm/api.rst
@@ -9600,6 +9600,22 @@ take care to differentiate between these cases.
 The presence of this capability indicates that the nested KVM guest can
 start in ESA mode.
 
+8.48 KVM_CAP_CARETAKER
+----------------------
+
+:Architectures: x86, arm64
+
+This capability, if ``KVM_CHECK_EXTENSION`` returns a non-zero value,
+indicates that the host kernel supports Caretaker on-core guest execution
+across kexec-based host live updates.
+
+When both a vCPU file descriptor and its backing physical CPU's preservation
+file descriptor (``/sys/devices/system/cpu/cpuX/preserve``) are preserved in a
+Live Update Orchestrator (LUO) session, KVM schedules the vCPU to continue
+executing on-core within the preserved Caretaker environment throughout the
+host kexec transition, rather than pausing vCPU execution until the incoming
+kernel re-attaches.
+
 9. Known KVM API problems
 =========================
 
diff --git a/include/linux/kho/abi/kvm.h b/include/linux/kho/abi/kvm.h
index 166a0a2f13c8..85fafb786747 100644
--- a/include/linux/kho/abi/kvm.h
+++ b/include/linux/kho/abi/kvm.h
@@ -30,10 +30,12 @@
 
 /**
  * struct kvm_luo_ser - Main serialization structure for a KVM VM.
- * @type:         The type of VM.
+ * @type:       The type of VM.
+ * @kho_folios: Preservation pointer to VM-wide KHO-preserved folios.
  */
 struct kvm_luo_ser {
 	u64 type;
+	DECLARE_KHOSER_PTR(kho_folios, struct kvm_kho_folios_ser *);
 } __packed;
 
 /* The compatibility string for KVM VM file handler */
@@ -47,18 +49,126 @@ enum kvm_vcpu_luo_flags {
 	KVM_VCPU_LUO_FLAG_CARETAKER = BIT(0),
 };
 
+/**
+ * enum kvm_caretaker_pcpu - Special Caretaker physical CPU identifiers
+ * @KVM_CARETAKER_INVALID_PCPU: Unassigned physical CPU identifier.
+ */
+enum kvm_caretaker_pcpu {
+	KVM_CARETAKER_INVALID_PCPU = U32_MAX,
+};
+
+/**
+ * enum kvm_caretaker_state - Caretaker vCPU execution state machine
+ * @KVM_CARETAKER_PAUSED:   Initial state upon preservation and between
+ *                          oncore_sched time-sharing quantums (or when parked
+ *                          after an unhandled VM exit).  Live architectural
+ *                          state is fully serialized in @arch_state.
+ * @KVM_CARETAKER_RUNNING:  Actively executing a time-sharing quantum on the
+ *                          preserved physical CPU.  Hardware registers and
+ *                          VMCS/VMCB/EL2 state are live on silicon; @arch_state
+ *                          in memory is stale until the quantum exits.
+ * @KVM_CARETAKER_STOPPING: Host requested reclaim while in
+ *                          %KVM_CARETAKER_RUNNING and sent a physical IPI kick.
+ *                          Caretaker will exit guest mode, serialize live
+ *                          hardware state into @arch_state, and transition to
+ *                          %KVM_CARETAKER_STOPPED.
+ * @KVM_CARETAKER_STOPPED:  Terminal state.  Caretaker execution has permanently
+ *                          ceased and @arch_state is valid in memory.  Reached
+ *                          either directly via host cmpxchg from
+ *                          %KVM_CARETAKER_PAUSED, or by the preserved CPU from
+ *                          %KVM_CARETAKER_STOPPING after serialization completes.
+ *
+ * State transitions are coordinated locklessly via atomic cmpxchg(&cb->state):
+ *   - Each scheduler quantum on the preserved CPU transitions
+ *     %KVM_CARETAKER_PAUSED -> %KVM_CARETAKER_RUNNING on entry and
+ *     %KVM_CARETAKER_RUNNING -> %KVM_CARETAKER_PAUSED after serializing guest
+ *     state on quantum exit.
+ *   - When host KVM reclaims the vCPU (kvm_caretaker_wait_for_attach()):
+ *     1. If @cb->state is %KVM_CARETAKER_PAUSED, host atomically transitions it
+ *        to %KVM_CARETAKER_STOPPED in 0 ns; if the preserved CPU later attempts
+ *        to start a quantum, its cmpxchg(%KVM_CARETAKER_PAUSED ->
+ *        %KVM_CARETAKER_RUNNING) fails and it immediately exits.
+ *     2. If @cb->state is %KVM_CARETAKER_RUNNING, host atomically transitions
+ *        it to %KVM_CARETAKER_STOPPING, sends an IPI to preempt guest mode, and
+ *        spins until the preserved CPU finishes detach_serialize() and stores
+ *        %KVM_CARETAKER_STOPPED.
+ */
+enum kvm_caretaker_state {
+	KVM_CARETAKER_PAUSED = 0,
+	KVM_CARETAKER_RUNNING = 1,
+	KVM_CARETAKER_STOPPING = 2,
+	KVM_CARETAKER_STOPPED = 3,
+};
+
+/**
+ * struct kvm_caretaker_telemetry_ser - Serialized Caretaker execution telemetry
+ * @total_runs: Total guest entry attempts across Caretaker quantums.
+ * @total_exits: Total VM exits decoded during Caretaker execution.
+ * @stall_count: Number of times Caretaker exited due to stall or error.
+ * @last_exit_reason: Raw architecture exit reason of most recent VM exit.
+ * @last_exit_rip: Guest instruction pointer at most recent VM exit.
+ * @stall_exit_reason: Raw architecture exit reason or error at stall.
+ * @stall_exit_rip: Guest instruction pointer at stall.
+ */
+struct kvm_caretaker_telemetry_ser {
+	u64 total_runs;
+	u64 total_exits;
+	u64 stall_count;
+	u64 last_exit_reason;
+	u64 last_exit_rip;
+	u64 stall_exit_reason;
+	u64 stall_exit_rip;
+} __packed;
+
+/**
+ * struct kvm_kho_folios_ser - Serialized list of KHO-preserved folios for a VM
+ * @nr_folios: Number of physical folio addresses in @folios_pa.
+ * @folios_pa: Physical addresses of folios preserved via kho_preserve_folio().
+ */
+struct kvm_kho_folios_ser {
+	u64 nr_folios;
+	u64 folios_pa[];
+} __packed;
+
+/**
+ * struct kvm_caretaker_cb_ser - KVM Caretaker Control Block
+ * @state:     Current Caretaker execution state (enum kvm_caretaker_state).
+ * @pcpu_id:   Physical CPU ID where this vCPU runs while in Caretaker.
+ * @vcpu_id:   Guest vCPU identifier.
+ * @reserved:  Must be zero.
+ * @telemetry: Preservation pointer to Caretaker execution telemetry.
+ *
+ * Coordinates vCPU execution state across hypervisor detachment,
+ * live update, and Caretaker CPU preservation.
+ */
+struct kvm_caretaker_cb_ser {
+	u32 state;
+	u32 pcpu_id;
+	u32 vcpu_id;
+	u32 reserved;
+	DECLARE_KHOSER_PTR(telemetry, struct kvm_caretaker_telemetry_ser *);
+} __packed;
+
 /**
  * struct kvm_vcpu_ser - Main serialization structure for a KVM vCPU.
  * @vcpu_id:    The ID of the virtual CPU.
  * @flags:      Flags for vCPU preservation.
  * @vm_token:   Token of the associated KVM VM instance.
  * @arch_state: Preservation pointer to vCPU architectural state.
+ * @cb:         Preservation pointer to Caretaker Control Block.
+ *
+ * Cross-kexec invariant: the incoming kernel may only dereference structures
+ * declared in include/linux/kho/abi/ headers.  When @flags includes
+ * %KVM_VCPU_LUO_FLAG_CARETAKER, the preserved Caretaker text writes live guest
+ * state into @arch_state at detach time before transitioning @cb to
+ * %KVM_CARETAKER_STOPPED; the incoming kernel reads only @cb and @arch_state.
  */
 struct kvm_vcpu_ser {
 	u32 vcpu_id;
 	u32 flags;
 	u64 vm_token;
 	DECLARE_KHOSER_PTR(arch_state, struct kvm_vcpu_arch_ser *);
+	DECLARE_KHOSER_PTR(cb, struct kvm_caretaker_cb_ser *);
 } __packed;
 
 /* The compatibility string for KVM vCPU file handler */
diff --git a/include/linux/kvm_caretaker.h b/include/linux/kvm_caretaker.h
new file mode 100644
index 000000000000..3e89738d7fdf
--- /dev/null
+++ b/include/linux/kvm_caretaker.h
@@ -0,0 +1,341 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/*
+ * Copyright (c) 2026, Google LLC.
+ * Pasha Tatashin <pasha.tatashin@soleen.com>
+ *
+ * Header for common KVM Caretaker framework across architectures.
+ */
+#ifndef __LINUX_KVM_CARETAKER_H
+#define __LINUX_KVM_CARETAKER_H
+
+#include <linux/types.h>
+#include <linux/kho/abi/kvm.h>
+
+struct kvm_vcpu;
+struct kvm_vcpu_ser;
+struct liveupdate_session;
+
+/**
+ * enum kvm_caretaker_exit_type - Normalized cross-architecture VM exit classification
+ * @KVM_CARETAKER_EXIT_UNKNOWN:       Unclassified exit; treated as unhandled stall.
+ * @KVM_CARETAKER_EXIT_IDLE:          Guest idle instruction (HLT, PAUSE, WFI, WFE).
+ * @KVM_CARETAKER_EXIT_CONSOLE:       Early console port I/O or MMIO access.
+ * @KVM_CARETAKER_EXIT_PREEMPT_TIMER: Hardware quantum preemption timer expired.
+ * @KVM_CARETAKER_EXIT_CROSS_VCPU:    Cross-vCPU notification or IPI (e.g., SGI).
+ * @KVM_CARETAKER_EXIT_INSN_STEP:     Instruction emulated by decode; advance RIP/PC.
+ * @KVM_CARETAKER_EXIT_ARCH:          Architecture-specific exit routed to @handle_arch_exit.
+ * @KVM_CARETAKER_EXIT_UNHANDLED:     Exit requiring full KVM/VMM; stalls the vCPU.
+ */
+enum kvm_caretaker_exit_type {
+	KVM_CARETAKER_EXIT_UNKNOWN = 0,
+	KVM_CARETAKER_EXIT_IDLE,
+	KVM_CARETAKER_EXIT_CONSOLE,
+	KVM_CARETAKER_EXIT_PREEMPT_TIMER,
+	KVM_CARETAKER_EXIT_CROSS_VCPU,
+	KVM_CARETAKER_EXIT_INSN_STEP,
+	KVM_CARETAKER_EXIT_ARCH,
+	KVM_CARETAKER_EXIT_UNHANDLED,
+};
+
+/**
+ * struct kvm_caretaker_exit - Normalized cross-architecture VM exit representation
+ * @type:       Normalized exit classification (&enum kvm_caretaker_exit_type).
+ * @rip:        Guest instruction pointer (RIP on x86, PC on arm64) at exit.
+ * @insn_len:   Length in bytes of the trapping instruction.
+ * @raw_reason: Raw hardware exit code (VMX exit reason, SVM exit code, or ESR_EL2).
+ * @mmio_io:    Decoded port I/O or MMIO access parameters.
+ * @msr:        Decoded x86 MSR read/write parameters.
+ * @sgi:        Decoded arm64 GICv3 Software Generated Interrupt parameters.
+ */
+struct kvm_caretaker_exit {
+	enum kvm_caretaker_exit_type type;
+	u64 rip;
+	u32 insn_len;
+	u64 raw_reason;
+	union {
+		struct {
+			u64 addr;
+			u64 val;
+			u64 *val_ptr;
+			u32 size;
+			bool is_write;
+			bool is_mmio;
+		} mmio_io;
+		struct {
+			u32 msr;
+			u64 val;
+			bool is_write;
+		} msr;
+		struct {
+			u32 sgi_id;
+			u64 target_mask;
+		} sgi;
+	};
+};
+
+struct kvm_caretaker_vcpu;
+
+/**
+ * struct kvm_caretaker_ops - Architecture operations vector for Caretaker vCPU execution
+ * @enter_guest:      Perform low-level hardware guest entry (VMLAUNCH/VMRESUME,
+ *                    VMRUN, or EL2 ERET). Returns 0 on guest exit, or non-zero
+ *                    on entry failure.
+ * @decode_exit:      Read hardware exit registers and populate @exit.
+ * @handle_arch_exit: Emulate an architecture-specific exit (@exit). Returns
+ *                    %true if handled (and updates @exit->rip if needed), or
+ *                    %false to stall the vCPU until the incoming kernel attaches.
+ * @advance_rip:      Write updated @next_rip back into hardware guest state.
+ * @arm_timer:        Program hardware preemption timer to fire at @deadline_ticks
+ *                    (or disarm if @deadline_ticks is 0).
+ * @disarm_timer:     Disarm the hardware preemption timer after leaving the loop.
+ * @pre_run:          Optional per-quantum setup hook invoked before guest entry.
+ * @post_run:         Optional per-quantum teardown hook invoked after leaving
+ *                    the guest loop.
+ */
+struct kvm_caretaker_ops {
+	int (*enter_guest)(void *vcpu_data);
+	void (*decode_exit)(void *vcpu_data, struct kvm_caretaker_exit *exit);
+	bool (*handle_arch_exit)(void *vcpu_data, struct kvm_caretaker_exit *exit);
+	void (*advance_rip)(void *vcpu_data, u64 next_rip);
+	void (*arm_timer)(void *vcpu_data, u64 deadline_ticks);
+	void (*disarm_timer)(void *vcpu_data);
+	void (*pre_run)(void *vcpu_data);
+	void (*post_run)(void *vcpu_data);
+};
+
+/**
+ * struct kvm_caretaker_vcpu - Common per-vCPU Caretaker runtime descriptor
+ * @cb:        Pointer to KHO-preserved Caretaker control block (&struct kvm_caretaker_cb_ser).
+ * @ops:       Architecture operations vector (&struct kvm_caretaker_ops).
+ * @arch_data: Architecture-specific runtime context passed to @ops callbacks.
+ */
+struct kvm_caretaker_vcpu {
+	struct kvm_caretaker_cb_ser *cb;
+	const struct kvm_caretaker_ops *ops;
+	void *arch_data;
+};
+
+struct kvm_kho_folios_ser *kvm_kho_folios_alloc(unsigned int max_folios);
+void kvm_kho_folios_unpreserve(struct kvm_kho_folios_ser *folios);
+void kvm_kho_folios_finish(struct kvm_kho_folios_ser *folios);
+
+#ifdef CONFIG_KVM_CARETAKER
+
+#include <linux/cpu_preserve.h>
+#include <linux/oncore.h>
+
+#define __caretaker_text __cpu_preserved_text
+#define __caretaker_data __cpu_preserved_data
+
+struct dentry;
+struct oncore_session;
+
+/**
+ * struct kvm_vcpu_caretaker - Host-side Caretaker state embedded in struct kvm_vcpu
+ * @cb:             Pointer to the KHO-preserved Caretaker control block while
+ *                  preserved, or %NULL when executing normally under KVM.
+ * @job:            On-Core scheduler job handle for this vCPU, or %NULL.
+ * @last_telemetry: Snapshot of @cb->telemetry captured upon re-attachment.
+ */
+struct kvm_vcpu_caretaker {
+	struct kvm_caretaker_cb_ser		*cb;
+	struct oncore_job			*job;
+	struct kvm_caretaker_telemetry_ser	last_telemetry;
+};
+
+/**
+ * kvm_caretaker_is_stopped - Check whether a Caretaker control block is stopped
+ * @cb: Caretaker control block (may be %NULL).
+ *
+ * Return: %true if @cb is %NULL or in state %KVM_CARETAKER_STOPPED, %false otherwise.
+ */
+static inline bool kvm_caretaker_is_stopped(const struct kvm_caretaker_cb_ser *cb)
+{
+	return !cb || smp_load_acquire(&cb->state) == KVM_CARETAKER_STOPPED;
+}
+
+/**
+ * kvm_caretaker_pause - Transition a Caretaker control block to PAUSED state
+ * @cb: Caretaker control block.
+ *
+ * Publishes %KVM_CARETAKER_PAUSED with release semantics so a preserved CPU
+ * can claim the vCPU for on-core execution.
+ */
+static inline void kvm_caretaker_pause(struct kvm_caretaker_cb_ser *cb)
+{
+	/* Pairs with smp_load_acquire() in caretaker loop */
+	smp_store_release(&cb->state, KVM_CARETAKER_PAUSED);
+}
+
+/**
+ * kvm_caretaker_stop - Transition a Caretaker control block to STOPPED state
+ * @cb: Caretaker control block.
+ *
+ * Publishes %KVM_CARETAKER_STOPPED with release semantics once the vCPU has
+ * detached from Caretaker execution.
+ */
+static inline void kvm_caretaker_stop(struct kvm_caretaker_cb_ser *cb)
+{
+	/* Pairs with smp_load_acquire() in caretaker loop */
+	smp_store_release(&cb->state, KVM_CARETAKER_STOPPED);
+}
+
+bool kvm_caretaker_vcpu_is_attached(struct kvm_vcpu *vcpu);
+void kvm_caretaker_init_common_vcpu(struct kvm_caretaker_vcpu *cvcpu,
+				    struct kvm_caretaker_cb_ser *cb,
+				    struct kvm_vcpu *vcpu,
+				    void *runtime_va,
+				    size_t runtime_size,
+				    const struct kvm_caretaker_ops *ops,
+				    void *arch_data);
+bool kvm_caretaker_should_exit(struct kvm_caretaker_vcpu *cvcpu);
+enum oncore_exit_reason
+kvm_caretaker_vcpu_run(struct kvm_caretaker_vcpu *cvcpu, u64 deadline_ticks);
+int kvm_caretaker_wait_for_attach(struct kvm_caretaker_cb_ser *cb, int pcpu);
+void kvm_caretaker_post_attach_vcpu(struct kvm_vcpu *vcpu);
+
+/**
+ * kvm_arch_vcpu_caretaker_run - Architecture entry point for Caretaker vCPU run
+ * @data:           Pointer to KHO-preserved Caretaker control block
+ *                  (&struct kvm_caretaker_cb_ser) embedded in the architecture
+ *                  runtime page.
+ * @deadline_ticks: Hardware counter deadline (TSC on x86, CNTPCT on arm64) at
+ *                  which the current scheduling quantum expires, or 0 if
+ *                  unbounded.
+ *
+ * Invoked by the On-Core scheduler loop on a preserved physical CPU during the
+ * kexec handover window.  Atomically transitions the vCPU from
+ * %KVM_CARETAKER_PAUSED to %KVM_CARETAKER_RUNNING, switches to Caretaker host
+ * state and isolated page tables, invokes kvm_caretaker_vcpu_run(), and
+ * serializes updated guest registers back to KHO memory upon exit.
+ *
+ * Context: Runs in __cpu_preserved_text with local interrupts disabled.
+ * Return: &enum oncore_exit_reason indicating why the vCPU yielded the core.
+ */
+enum oncore_exit_reason
+kvm_arch_vcpu_caretaker_run(void *data, u64 deadline_ticks);
+
+/**
+ * kvm_arch_vcpu_luo_pre_retrieve_caretaker - Signal Caretaker vCPU to stop and serialize
+ * @vcpu: Incoming or cancelled-handover KVM vCPU structure.
+ * @ser:  Serialized KHO vCPU metadata.
+ *
+ * Signals the preserved physical CPU executing this vCPU in Caretaker to exit
+ * guest mode, serialize live guest hardware state into @ser->arch_state, and
+ * transition to %KVM_CARETAKER_STOPPED.  Must run before
+ * kvm_arch_vcpu_luo_retrieve() reads @ser->arch_state.
+ */
+void kvm_arch_vcpu_luo_pre_retrieve_caretaker(struct kvm_vcpu *vcpu,
+					      struct kvm_vcpu_ser *ser);
+
+/**
+ * kvm_arch_vcpu_luo_attach_caretaker - Complete architecture vCPU attachment after retrieve
+ * @vcpu: Incoming kernel vCPU structure being restored.
+ * @ser:  Serialized KHO vCPU metadata containing the Caretaker control block PA.
+ *
+ * Runs after kvm_arch_vcpu_luo_retrieve() has restored @ser->arch_state into
+ * @vcpu; synchronizes architecture-specific hardware state (VMCS/VMCB/VGIC and
+ * emulated UART/timer state) from the preserved Caretaker page and completes
+ * attachment via kvm_caretaker_post_attach_vcpu().
+ */
+void kvm_arch_vcpu_luo_attach_caretaker(struct kvm_vcpu *vcpu,
+					struct kvm_vcpu_ser *ser);
+
+int kvm_caretaker_vcpu_pre_preserve(struct kvm_vcpu *vcpu,
+				    struct liveupdate_session *session,
+				    struct kvm_vcpu_ser *ser);
+int kvm_caretaker_vcpu_post_preserve(struct kvm_vcpu *vcpu,
+				     struct liveupdate_session *session,
+				     struct kvm_vcpu_ser *ser,
+				     int arch_err);
+void kvm_caretaker_vcpu_pre_retrieve(struct kvm_vcpu *vcpu,
+				     struct kvm_vcpu_ser *ser);
+void kvm_caretaker_vcpu_retrieve(struct kvm_vcpu *vcpu,
+				 struct kvm_vcpu_ser *ser);
+void kvm_caretaker_vcpu_unpreserve(struct kvm_vcpu *vcpu,
+				   struct liveupdate_session *session,
+				   struct kvm_vcpu_ser *ser);
+void kvm_caretaker_vcpu_finish(struct kvm_vcpu *vcpu,
+			       struct liveupdate_session *session,
+			       struct kvm_vcpu_ser *ser);
+
+#else /* !CONFIG_KVM_CARETAKER */
+
+#define __caretaker_text
+#define __caretaker_data
+
+static inline bool kvm_caretaker_is_stopped(const struct kvm_caretaker_cb_ser *cb)
+{
+	return true;
+}
+
+static inline void kvm_caretaker_pause(struct kvm_caretaker_cb_ser *cb) {}
+
+static inline void kvm_caretaker_stop(struct kvm_caretaker_cb_ser *cb) {}
+
+static inline bool kvm_caretaker_vcpu_is_attached(struct kvm_vcpu *vcpu)
+{
+	return true;
+}
+
+static inline void kvm_caretaker_vm_pre_retrieve(void) {}
+
+static inline int kvm_caretaker_vcpu_pre_preserve(struct kvm_vcpu *vcpu,
+						  struct liveupdate_session *session,
+						  struct kvm_vcpu_ser *ser)
+{
+	return 0;
+}
+
+static inline int kvm_caretaker_vcpu_post_preserve(struct kvm_vcpu *vcpu,
+						   struct liveupdate_session *session,
+						   struct kvm_vcpu_ser *ser,
+						   int arch_err)
+{
+	return arch_err;
+}
+
+static inline void kvm_caretaker_init_common_vcpu(struct kvm_caretaker_vcpu *cvcpu,
+						  struct kvm_caretaker_cb_ser *cb,
+						 struct kvm_vcpu *vcpu,
+						 void *runtime_va,
+						 size_t runtime_size,
+						 const struct kvm_caretaker_ops *ops,
+						 void *arch_data) {}
+
+static inline bool kvm_caretaker_should_exit(struct kvm_caretaker_vcpu *cvcpu)
+{
+	return true;
+}
+
+static inline int kvm_caretaker_wait_for_attach(struct kvm_caretaker_cb_ser *cb, int pcpu,
+						void (*arch_kick)(int pcpu))
+{
+	return 0;
+}
+
+static inline void kvm_caretaker_post_attach_vcpu(struct kvm_vcpu *vcpu) {}
+
+static inline void kvm_arch_vcpu_luo_pre_retrieve_caretaker(struct kvm_vcpu *vcpu,
+							    struct kvm_vcpu_ser *ser) {}
+
+static inline void kvm_arch_vcpu_luo_attach_caretaker(struct kvm_vcpu *vcpu,
+						      struct kvm_vcpu_ser *ser) {}
+
+static inline void kvm_caretaker_vcpu_pre_retrieve(struct kvm_vcpu *vcpu,
+						   struct kvm_vcpu_ser *ser) {}
+
+static inline void kvm_caretaker_vcpu_retrieve(struct kvm_vcpu *vcpu,
+					       struct kvm_vcpu_ser *ser) {}
+
+static inline void kvm_caretaker_vcpu_unpreserve(struct kvm_vcpu *vcpu,
+						 struct liveupdate_session *session,
+						 struct kvm_vcpu_ser *ser) {}
+
+static inline void kvm_caretaker_vcpu_finish(struct kvm_vcpu *vcpu,
+					     struct liveupdate_session *session,
+					     struct kvm_vcpu_ser *ser) {}
+
+#endif /* CONFIG_KVM_CARETAKER */
+
+#endif /* __LINUX_KVM_CARETAKER_H */
diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h
index 643b941286c2..aca3c41346fb 100644
--- a/include/linux/kvm_host.h
+++ b/include/linux/kvm_host.h
@@ -44,6 +44,7 @@
 
 #include <asm/kvm_host.h>
 #include <linux/kvm_dirty_ring.h>
+#include <linux/kvm_caretaker.h>
 
 #ifndef KVM_MAX_VCPU_IDS
 #define KVM_MAX_VCPU_IDS KVM_MAX_VCPUS
@@ -398,6 +399,9 @@ struct kvm_vcpu {
 	 */
 	struct kvm_memory_slot *last_used_slot;
 	u64 last_used_slot_gen;
+#ifdef CONFIG_KVM_CARETAKER
+	struct kvm_vcpu_caretaker caretaker;
+#endif
 };
 
 /*
@@ -887,6 +891,12 @@ struct kvm {
 	 * RCU (e.g. via get_file_active() to prevent ABA races).
 	 */
 	struct file __rcu *vm_file;
+#endif
+#ifdef CONFIG_LIVEUPDATE
+	struct kvm_kho_folios_ser *kho_folios;
+#endif
+#ifdef CONFIG_KVM_CARETAKER
+	void *caretaker_vm;
 #endif
 	char stats_id[KVM_STATS_NAME_SIZE];
 };
diff --git a/include/uapi/linux/kvm.h b/include/uapi/linux/kvm.h
index 4b1754050681..1042cff30fef 100644
--- a/include/uapi/linux/kvm.h
+++ b/include/uapi/linux/kvm.h
@@ -1000,6 +1000,7 @@ struct kvm_enable_cap {
 #define KVM_CAP_PPC_COMPAT_CAPS 250
 #define KVM_CAP_ARM_PMU_V3_STRICT 251
 #define KVM_CAP_VCPU_PRESERVE 252
+#define KVM_CAP_CARETAKER 253
 
 struct kvm_irq_routing_irqchip {
 	__u32 irqchip;
-- 
2.55.0.1082.g2b9226bbc0-goog

[RFC PATCH 29/46] KVM: caretaker: Implement Caretaker session memory mapping helpers

[Pasha Tatashin]

Implement Caretaker guest memory mapping, control block transitions,
and architecture ops registration in virt/kvm/caretaker.c.

Signed-off-by: Pasha Tatashin <pasha.tatashin@soleen.com>
---
 virt/kvm/caretaker.c | 452 +++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 452 insertions(+)
 create mode 100644 virt/kvm/caretaker.c

diff --git a/virt/kvm/caretaker.c b/virt/kvm/caretaker.c
new file mode 100644
index 000000000000..b4a203562727
--- /dev/null
+++ b/virt/kvm/caretaker.c
@@ -0,0 +1,452 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * Copyright (c) 2026, Google LLC.
+ * Pasha Tatashin <pasha.tatashin@soleen.com>
+ *
+ * Core KVM Caretaker common execution engine and lifecycle management.
+ */
+
+/**
+ * DOC: KVM Caretaker Architecture and Lifecycle
+ *
+ * Overview: Orphaned Virtual Machines During Live Update
+ * ------------------------------------------------------
+ * During a host kernel Live Update (kexec), the userspace VMM and the outgoing
+ * Linux kernel tear down and vanish while the incoming kernel boots and a new
+ * userspace VMM process adopts the preserved state.  Throughout this handover
+ * window, preserved virtual machines are temporarily "orphaned" from any full
+ * host operating system or userspace VMM.
+ *
+ * Instead of freezing all guest vCPUs in RAM for the duration of kexec, the KVM
+ * Caretaker framework keeps preserved guest vCPUs actively executing on
+ * preserved physical CPUs (CONFIG_LIVEUPDATE_CPU) scheduled by the On-Core
+ * runtime (CONFIG_LIVEUPDATE_ONCORE).  When a Live Update session has preserved
+ * physical CPUs attached, Caretaker detaches each vCPU into a self-contained
+ * KHO-preserved runtime page and schedules it on the preserved cores
+ * (%KVM_VCPU_LUO_FLAG_CARETAKER).  If a session has zero preserved physical
+ * CPUs, Caretaker transparently falls back to RAM-only vCPU preservation.
+ *
+ * System Layering
+ * ---------------
+ * Orphaned VM execution is structured across four layers:
+ *
+ * 1. Physical CPU Preservation (kernel/liveupdate/cpu_preserve.c):
+ *    Isolates physical CPU cores from Linux hotplug teardown on dedicated
+ *    KHO-preserved stacks (&struct cpu_preserved_stack_context), switches them
+ *    to isolated page tables (&struct cpu_preserved_as) mapping only
+ *    .text.cpu_preserved and .data.cpu_preserved outside KHO Scratch memory,
+ *    and provides cache-coherency and cross-CPU wake primitives.
+ *
+ * 2. On-Core Session & Scheduler (kernel/liveupdate/oncore.c):
+ *    Groups preserved physical CPUs and an isolated address space per
+ *    &struct liveupdate_session, and runs a lockless round-robin FIFO
+ *    scheduler (oncore_cpu_schedule_loop()) that time-slices M workload jobs
+ *    across N preserved physical CPUs (supporting both 1:1 dedicated pinning
+ *    and M > N oversubscription).
+ *
+ * 3. Common KVM Caretaker Engine:
+ *    Implements the architecture-neutral vCPU quantum loop
+ *    (kvm_caretaker_vcpu_run()), normalized VM-exit dispatch
+ *    (kvm_caretaker_dispatch_exit()), guest idle handling (HLT/PAUSE/WFI),
+ *    unhandled-exit stall parking (%ONCORE_EXIT_STALL), lockless control-block
+ *    state transitions (&struct kvm_caretaker_cb_ser), and KVM LUO lifecycle
+ *    hooks.
+ *
+ * 4. Architecture & Vendor Backends (arch/x86/kvm/ and arch/arm64/kvm/):
+ *    Implement &struct kvm_caretaker_ops (guest entry/exit assembly, hardware
+ *    timer programming, and VMCS/VMCB/EL2 context management) and self-contained
+ *    on-core emulation for performance-critical exits (early UART console,
+ *    CPUID, MSRs, RDTSC, GICv3 CPU interface / SGI delivery, and architectural
+ *    timers) while keeping guest EPT/NPT/Stage-2 page tables live in hardware.
+ *
+ * Orphaned vCPU Lifecycle and State Machine
+ * -----------------------------------------
+ * Each vCPU's KHO-preserved control block (&struct kvm_caretaker_cb_ser)
+ * transitions through four phases across a live update:
+ *
+ * 1. Pre-Preserve & Activation (Outgoing Kernel -- LUO Prepare/Freeze):
+ *    - kvm_caretaker_vcpu_pre_preserve() submits an &struct oncore_job for
+ *      kvm_arch_vcpu_caretaker_run() to the session's least-loaded preserved
+ *      physical CPU and sets %KVM_VCPU_LUO_FLAG_CARETAKER.
+ *    - kvm_arch_vcpu_luo_preserve() allocates the architecture Caretaker page,
+ *      preserves stage-2/TDP MMU page tables, captures guest register and
+ *      virtualization hardware state, and calls kvm_caretaker_init_common_vcpu()
+ *      to map the runtime page into the session's isolated PGD and initialize
+ *      @cb->state to %KVM_CARETAKER_PAUSED.
+ *    - kvm_caretaker_vcpu_post_preserve() binds @cb to the job, cleans @cb to
+ *      PoC, and calls oncore_session_activate_job() to enqueue the job and kick
+ *      the assigned preserved physical CPU.
+ *
+ * 2. Orphaned On-Core Execution (Across Kexec Handover):
+ *    - The preserved CPU invokes kvm_arch_vcpu_caretaker_run(), which
+ *      atomically transitions @cb->state from %KVM_CARETAKER_PAUSED to
+ *      %KVM_CARETAKER_RUNNING and calls kvm_caretaker_vcpu_run().
+ *    - kvm_caretaker_vcpu_run() arms the preemption timer for @deadline_ticks
+ *      and repeatedly enters the guest (ops->enter_guest()), decodes exits
+ *      (ops->decode_exit()), and dispatches on-core handlers.
+ *    - When the time quantum expires (%ONCORE_EXIT_QUANTUM_EXPIRED), the guest
+ *      executes HLT/WFI (%ONCORE_EXIT_YIELD_IDLE), or an exit cannot be
+ *      emulated on-core (%ONCORE_EXIT_STALL, leaving RIP/PC on the faulting
+ *      instruction), the backend serializes live hardware state into
+ *      @ser->arch_state, transitions @cb->state back to %KVM_CARETAKER_PAUSED,
+ *      and returns to the On-Core scheduler.
+ *
+ * 3. Re-Attachment & Adoption (Incoming Kernel -- LUO Retrieve/Finish):
+ *    - During LUO retrieve, kvm_caretaker_vcpu_pre_retrieve() invokes
+ *      kvm_arch_vcpu_luo_pre_retrieve_caretaker(), which calls
+ *      kvm_caretaker_wait_for_attach().
+ *    - Fast path: if @cb->state is %KVM_CARETAKER_PAUSED, a single cmpxchg()
+ *      transitions it to %KVM_CARETAKER_STOPPED immediately.
+ *    - Slow path: if @cb->state is %KVM_CARETAKER_RUNNING, it transitions to
+ *      %KVM_CARETAKER_STOPPING and sends an IPI kick to force a VM exit.  The
+ *      preserved CPU observes kvm_caretaker_should_exit(), serializes final
+ *      guest state into @ser->arch_state, publishes %KVM_CARETAKER_STOPPED, and
+ *      exits with %ONCORE_EXIT_ATTACH_SIGNALED.
+ *    - kvm_arch_vcpu_luo_retrieve() and kvm_caretaker_vcpu_retrieve() then
+ *      load the updated state into the incoming &struct kvm_vcpu and resume
+ *      normal KVM execution.
+ *
+ * 4. Cancellation / Rollback (Outgoing Kernel -- LUO Unpreserve):
+ *    - If live update is cancelled before kexec, kvm_caretaker_vcpu_unpreserve()
+ *      executes the same attach handshake to stop the vCPU on the preserved
+ *      core, synchronizes any guest state changes back into the outgoing
+ *      &struct kvm_vcpu, and cancels the job via oncore_session_cancel_job().
+ */
+
+#include <linux/cpu_preserve.h>
+#include <linux/delay.h>
+#include <linux/io.h>
+#include <linux/kernel.h>
+#include <linux/kho/abi/kvm.h>
+#include <linux/kvm_caretaker.h>
+#include <linux/kvm_host.h>
+#include <linux/liveupdate.h>
+#include <linux/objtool.h>
+#include <linux/oncore.h>
+
+/**
+ * kvm_caretaker_vcpu_is_attached - Check whether a vCPU has attached back to host KVM
+ * @vcpu: Target KVM vCPU.
+ *
+ * Return: %true if @vcpu is not running under Caretaker (control block is %NULL
+ *         or in %KVM_CARETAKER_STOPPED), %false if still owned by Caretaker.
+ */
+bool kvm_caretaker_vcpu_is_attached(struct kvm_vcpu *vcpu)
+{
+	return kvm_caretaker_is_stopped(vcpu->caretaker.cb);
+}
+
+/**
+ * kvm_caretaker_init_common_vcpu - Initialize common Caretaker vCPU runtime state
+ * @cvcpu:        Common Caretaker vCPU descriptor to initialize.
+ * @cb:           KHO-preserved Caretaker control block embedded in the arch page.
+ * @vcpu:         Host KVM vCPU being preserved.
+ * @runtime_va:   Virtual address of the architecture Caretaker runtime page.
+ * @runtime_size: Size in bytes of @runtime_va.
+ * @ops:          Architecture operations table (&struct kvm_caretaker_ops).
+ * @arch_data:    Architecture context pointer passed to @ops callbacks.
+ *
+ * Initializes @cb in %KVM_CARETAKER_PAUSED state with the preserved physical
+ * CPU ID assigned to @vcpu->caretaker.job, maps the runtime buffer into the
+ * On-Core session's isolated page tables, and allocates KHO telemetry state.
+ */
+void kvm_caretaker_init_common_vcpu(struct kvm_caretaker_vcpu *cvcpu,
+				    struct kvm_caretaker_cb_ser *cb,
+				    struct kvm_vcpu *vcpu,
+				    void *runtime_va,
+				    size_t runtime_size,
+				    const struct kvm_caretaker_ops *ops,
+				    void *arch_data)
+{
+	struct oncore_session *sess = oncore_job_session(vcpu->caretaker.job);
+
+	cb->state = KVM_CARETAKER_PAUSED;
+	cb->pcpu_id = oncore_job_cpu(vcpu->caretaker.job);
+	cb->vcpu_id = vcpu->vcpu_id;
+	cb->reserved = 0;
+	cb->telemetry.phys = 0;
+
+	cvcpu->cb = cb;
+	cvcpu->ops = ops;
+	cvcpu->arch_data = arch_data ? arch_data : cvcpu;
+	cvcpu->telemetry = NULL;
+
+	vcpu->caretaker.cb = cb;
+
+	oncore_session_map_buffer(sess, runtime_va, runtime_size);
+	kvm_caretaker_telemetry_init(cvcpu, sess);
+}
+
+/**
+ * kvm_caretaker_should_exit - Check whether the Caretaker vCPU loop must exit for attachment
+ * @cvcpu: Common Caretaker vCPU descriptor.
+ *
+ * Invalidates cache lines for @cvcpu->cb and checks whether the host kernel has
+ * requested attachment (%KVM_CARETAKER_STOPPING / %KVM_CARETAKER_STOPPED) or
+ * whether the underlying preserved physical CPU is exiting its workload loop.
+ *
+ * Return: %true if the vCPU must immediately exit guest execution and serialize
+ *         its state for host attachment, %false otherwise.
+ */
+bool __cpu_preserved_text
+kvm_caretaker_should_exit(struct kvm_caretaker_vcpu *cvcpu)
+{
+	struct cpu_preserved_stack_context *sctx;
+	u32 st;
+
+	cpu_preserved_inval(cvcpu->cb);
+
+	st = READ_ONCE(cvcpu->cb->state);
+	if (st != KVM_CARETAKER_PAUSED && st != KVM_CARETAKER_RUNNING)
+		return true;
+
+	sctx = cpu_preserved_get_stack_context();
+	return cpu_preserved_should_exit(sctx ? sctx->cpu : cvcpu->cb->pcpu_id);
+}
+
+static bool __cpu_preserved_text
+kvm_caretaker_dispatch_exit(struct kvm_caretaker_vcpu *cvcpu,
+			    struct kvm_caretaker_exit *exit)
+{
+	switch (exit->type) {
+	case KVM_CARETAKER_EXIT_IDLE:
+		cpu_relax();
+		exit->rip += exit->insn_len;
+		return false;
+
+	case KVM_CARETAKER_EXIT_PREEMPT_TIMER:
+	case KVM_CARETAKER_EXIT_UNHANDLED:
+	case KVM_CARETAKER_EXIT_UNKNOWN:
+		return false;
+
+	case KVM_CARETAKER_EXIT_CONSOLE:
+	case KVM_CARETAKER_EXIT_CROSS_VCPU:
+	case KVM_CARETAKER_EXIT_INSN_STEP:
+	case KVM_CARETAKER_EXIT_ARCH:
+	default:
+		if (cvcpu->ops->handle_arch_exit)
+			return cvcpu->ops->handle_arch_exit(cvcpu->arch_data,
+							    exit);
+		return false;
+	}
+}
+STACK_FRAME_NON_STANDARD(kvm_caretaker_dispatch_exit);
+
+static int __cpu_preserved_text
+kvm_caretaker_enter_guest(struct kvm_caretaker_vcpu *cvcpu)
+{
+	void *arch_data = cvcpu->arch_data;
+	int ret;
+
+	ret = cvcpu->ops->enter_guest(arch_data);
+	kvm_caretaker_telemetry_run(cvcpu);
+	if (ret && !kvm_caretaker_should_exit(cvcpu)) {
+		cpu_relax();
+		ret = cvcpu->ops->enter_guest(arch_data);
+		kvm_caretaker_telemetry_run(cvcpu);
+	}
+	if (ret)
+		kvm_caretaker_telemetry_stall(cvcpu, (u32)ret, 0);
+
+	return ret;
+}
+STACK_FRAME_NON_STANDARD(kvm_caretaker_enter_guest);
+
+static bool __cpu_preserved_text
+kvm_caretaker_handle_exit(struct kvm_caretaker_vcpu *cvcpu,
+			  enum oncore_exit_reason *reason)
+{
+	const struct kvm_caretaker_ops *ops = cvcpu->ops;
+	struct kvm_caretaker_exit exit __uninitialized;
+	void *arch_data = cvcpu->arch_data;
+	bool handled;
+
+	cpu_preserved_memset(&exit, 0, sizeof(exit));
+	if (ops->decode_exit)
+		ops->decode_exit(arch_data, &exit);
+
+	kvm_caretaker_telemetry_exit(cvcpu, &exit);
+	if (kvm_caretaker_should_exit(cvcpu))
+		return false;
+
+	handled = kvm_caretaker_dispatch_exit(cvcpu, &exit);
+	if (ops->advance_rip)
+		ops->advance_rip(arch_data, exit.rip);
+	if (handled)
+		return true;
+
+	/*
+	 * A preemption-timer exit is the normal end of a time slice, so leave
+	 * @reason alone for it.  For anything else nothing emulated the exit
+	 * and RIP was left on the faulting instruction, so re-entering the
+	 * guest would take the exact same exit again.  Tell the scheduler the
+	 * job is stuck rather than letting it look like an expired time slice,
+	 * so that it backs off instead of spinning on VM entry/exit until the
+	 * incoming kernel reclaims the vCPU.
+	 */
+	if (exit.type == KVM_CARETAKER_EXIT_IDLE) {
+		*reason = ONCORE_EXIT_YIELD_IDLE;
+	} else if (exit.type != KVM_CARETAKER_EXIT_PREEMPT_TIMER) {
+		*reason = ONCORE_EXIT_STALL;
+		kvm_caretaker_telemetry_stall(cvcpu, exit.raw_reason, exit.rip);
+	}
+
+	return false;
+}
+STACK_FRAME_NON_STANDARD(kvm_caretaker_handle_exit);
+
+/**
+ * kvm_caretaker_vcpu_run - Common hardware vCPU execution loop for Caretaker
+ * @cvcpu:          Common Caretaker vCPU descriptor.
+ * @deadline_ticks: Hardware counter deadline for the current scheduling quantum.
+ *
+ * Arms the hardware preemption timer for @deadline_ticks and repeatedly enters
+ * the guest via @cvcpu->ops->enter_guest(), decodes VM exits, and dispatches
+ * on-core exit handlers until the time slice expires, the guest yields on
+ * HLT/WFI, the incoming kernel signals attachment, or an unhandled exit stalls
+ * the vCPU.
+ *
+ * Context: Preserved physical CPU (__cpu_preserved_text) with IRQs disabled.
+ * Return: &enum oncore_exit_reason indicating why the vCPU left the loop.
+ */
+enum oncore_exit_reason __cpu_preserved_text
+kvm_caretaker_vcpu_run(struct kvm_caretaker_vcpu *cvcpu, u64 deadline_ticks)
+{
+	struct cpu_preserved_stack_context *sctx = cpu_preserved_get_stack_context();
+	enum oncore_exit_reason reason = ONCORE_EXIT_QUANTUM_EXPIRED;
+	const struct kvm_caretaker_ops *ops = cvcpu->ops;
+	void *arch_data = cvcpu->arch_data;
+
+	if (sctx)
+		cvcpu->cb->pcpu_id = sctx->cpu;
+
+	if (kvm_caretaker_should_exit(cvcpu))
+		return ONCORE_EXIT_ATTACH_SIGNALED;
+
+	if (ops->pre_run)
+		ops->pre_run(arch_data);
+
+	if (ops->arm_timer)
+		ops->arm_timer(arch_data, deadline_ticks);
+
+	while (!kvm_caretaker_should_exit(cvcpu)) {
+		if (arch_oncore_read_counter() >= deadline_ticks)
+			break;
+
+		if (kvm_caretaker_enter_guest(cvcpu)) {
+			reason = ONCORE_EXIT_ERROR;
+			break;
+		}
+
+		if (!kvm_caretaker_handle_exit(cvcpu, &reason))
+			break;
+	}
+
+	if (ops->disarm_timer)
+		ops->disarm_timer(arch_data);
+
+	if (ops->post_run)
+		ops->post_run(arch_data);
+
+	kvm_caretaker_telemetry_flush(cvcpu);
+
+	if (kvm_caretaker_should_exit(cvcpu))
+		return ONCORE_EXIT_ATTACH_SIGNALED;
+
+	return reason;
+}
+STACK_FRAME_NON_STANDARD(kvm_caretaker_vcpu_run);
+
+#define KVM_CARETAKER_ATTACH_TIMEOUT_US		2000000
+#define KVM_CARETAKER_ATTACH_STEP_US		10
+#define KVM_CARETAKER_ATTACH_KICK_STEPS		100
+
+static bool kvm_caretaker_try_stop(struct kvm_caretaker_cb_ser *cb)
+{
+	cpu_preserved_inval(cb);
+
+	if (READ_ONCE(cb->state) == KVM_CARETAKER_STOPPED)
+		return true;
+
+	if (cmpxchg(&cb->state, KVM_CARETAKER_PAUSED,
+		    KVM_CARETAKER_STOPPED) == KVM_CARETAKER_PAUSED) {
+		cpu_preserved_clean(cb);
+		return true;
+	}
+
+	return false;
+}
+
+/**
+ * kvm_caretaker_wait_for_attach - Stop a Caretaker vCPU and wait for state serialization
+ * @cb:   KHO-preserved Caretaker control block.
+ * @pcpu: Logical ID of the preserved physical CPU running the vCPU.
+ *
+ * Synchronizes with the preserved physical CPU executing @cb so that the vCPU
+ * exits guest mode, serializes its final architectural state into KHO memory,
+ * and reaches %KVM_CARETAKER_STOPPED before the host kernel reads back the
+ * serialized state.
+ *
+ * Return: 0 on success, or -ETIMEDOUT if the preserved CPU failed to stop.
+ */
+int kvm_caretaker_wait_for_attach(struct kvm_caretaker_cb_ser *cb, int pcpu)
+{
+	int i;
+
+	if (!cb || kvm_caretaker_try_stop(cb))
+		return 0;
+
+	if (!cpu_is_preserved(pcpu)) {
+		WRITE_ONCE(cb->state, KVM_CARETAKER_STOPPED);
+		cpu_preserved_clean(cb);
+		/* Ensure state update is visible before returning to caller */
+		smp_wmb();
+		return 0;
+	}
+
+	/*
+	 * Slow path: the vCPU is actively executing a quantum on the preserved
+	 * physical CPU (%KVM_CARETAKER_RUNNING).  Request a stop by moving it
+	 * to %KVM_CARETAKER_STOPPING, send an IPI kick to force a VM exit, and
+	 * spin until the preserved CPU finishes detach_serialize() and publishes
+	 * %KVM_CARETAKER_STOPPED.
+	 */
+	cmpxchg(&cb->state, KVM_CARETAKER_RUNNING, KVM_CARETAKER_STOPPING);
+	cpu_preserved_clean(cb);
+	/* Order state update before kicking and polling the preserved CPU */
+	smp_mb();
+
+	for (i = 0; i < KVM_CARETAKER_ATTACH_TIMEOUT_US / KVM_CARETAKER_ATTACH_STEP_US; i++) {
+		if (i % KVM_CARETAKER_ATTACH_KICK_STEPS == 0)
+			arch_cpu_preserved_kick(pcpu);
+		if (kvm_caretaker_try_stop(cb))
+			return 0;
+		udelay(KVM_CARETAKER_ATTACH_STEP_US);
+	}
+
+	pr_warn("kvm: caretaker attach handshake timed out for pCPU %d\n", pcpu);
+	return -ETIMEDOUT;
+}
+STACK_FRAME_NON_STANDARD(kvm_caretaker_wait_for_attach);
+
+/**
+ * kvm_caretaker_post_attach_vcpu - Finalize host vCPU state after Caretaker attachment
+ * @vcpu: KVM vCPU that has just re-attached from Caretaker.
+ *
+ * Resets @vcpu->mode and @vcpu->cpu, reports Caretaker execution telemetry to
+ * the kernel log and vCPU debugfs snapshot, marks the control block stopped,
+ * and clears @vcpu->caretaker.cb.
+ */
+void kvm_caretaker_post_attach_vcpu(struct kvm_vcpu *vcpu)
+{
+	/* Ensure vCPU mode update is globally visible before clearing cpu */
+	smp_store_mb(vcpu->mode, EXITING_GUEST_MODE);
+	vcpu->cpu = -1;
+
+	if (vcpu->caretaker.cb) {
+		kvm_caretaker_telemetry_report(vcpu, vcpu->caretaker.cb);
+		kvm_caretaker_stop(vcpu->caretaker.cb);
+		vcpu->caretaker.cb = NULL;
+	}
+}
+
-- 
2.55.0.1082.g2b9226bbc0-goog

[RFC PATCH 30/46] KVM: caretaker: Integrate Caretaker vCPU detach, attach, and cancel with KVM

[Pasha Tatashin]

Integrate Caretaker vCPU preservation, re-attachment, and cancellation
with KVM LUO handlers and enable CONFIG_KVM_CARETAKER build rules.

Signed-off-by: Pasha Tatashin <pasha.tatashin@soleen.com>
---
 Documentation/liveupdate/vmm.rst |  15 +++
 include/linux/kvm_caretaker.h    |   1 +
 virt/kvm/Kconfig                 |  28 +++++
 virt/kvm/Makefile.kvm            |   7 ++
 virt/kvm/caretaker.c             | 197 ++++++++++++++++++++++++++++++-
 virt/kvm/kvm_luo.c               |  70 ++++++++++-
 virt/kvm/kvm_main.c              |   7 ++
 7 files changed, 318 insertions(+), 7 deletions(-)

diff --git a/Documentation/liveupdate/vmm.rst b/Documentation/liveupdate/vmm.rst
index b8e8fb2b63c2..a0b54d6f1220 100644
--- a/Documentation/liveupdate/vmm.rst
+++ b/Documentation/liveupdate/vmm.rst
@@ -100,8 +100,23 @@ VM & Guest_Memfd Preservation ABI
 .. kernel-doc:: include/linux/kho/abi/kvm.h
    :internal:
 
+KVM Caretaker (Orphaned VM Execution)
+=====================================
+
+.. kernel-doc:: virt/kvm/caretaker.c
+   :doc: KVM Caretaker Architecture and Lifecycle
+
+KVM Caretaker Core & Architecture API
+=====================================
+
+.. kernel-doc:: include/linux/kvm_caretaker.h
+
+.. kernel-doc:: virt/kvm/caretaker.c
+   :identifiers:
+
 See Also
 ========
 
 - :doc:`/core-api/liveupdate`
+- :doc:`/liveupdate/cpu_preservation`
 - :doc:`/userspace-api/liveupdate`
diff --git a/include/linux/kvm_caretaker.h b/include/linux/kvm_caretaker.h
index 3e89738d7fdf..0d4bdde7b1c3 100644
--- a/include/linux/kvm_caretaker.h
+++ b/include/linux/kvm_caretaker.h
@@ -241,6 +241,7 @@ void kvm_arch_vcpu_luo_pre_retrieve_caretaker(struct kvm_vcpu *vcpu,
 void kvm_arch_vcpu_luo_attach_caretaker(struct kvm_vcpu *vcpu,
 					struct kvm_vcpu_ser *ser);
 
+void kvm_caretaker_vm_pre_retrieve(void);
 int kvm_caretaker_vcpu_pre_preserve(struct kvm_vcpu *vcpu,
 				    struct liveupdate_session *session,
 				    struct kvm_vcpu_ser *ser);
diff --git a/virt/kvm/Kconfig b/virt/kvm/Kconfig
index fcaf57377e73..165e0c85b8aa 100644
--- a/virt/kvm/Kconfig
+++ b/virt/kvm/Kconfig
@@ -125,3 +125,31 @@ config HAVE_KVM_ARCH_GMEM_INVALIDATE
 config HAVE_KVM_ARCH_GMEM_POPULATE
        bool
        depends on KVM_GUEST_MEMFD
+
+config HAVE_KVM_ARCH_CARETAKER
+	bool
+
+config KVM_CARETAKER
+	bool "Orphaned VM Caretaker execution and lifecycle management"
+	depends on KVM
+	depends on LIVEUPDATE_CPU
+	depends on SMP
+	depends on HOTPLUG_CPU
+	depends on HAVE_KVM_ARCH_CARETAKER
+	select LIVEUPDATE_ONCORE
+	default LIVEUPDATE_CPU
+	help
+	  Enable Caretaker CPU preservation and standalone execution for
+	  orphaned virtual machines across host kernel live updates.
+
+	  During a host kernel live update, user space hypervisors
+	  and host services terminate, leaving running guest virtual
+	  machines in an orphaned state. Caretaker keeps designated physical
+	  CPUs running their respective guest vCPUs on-core in a standalone
+	  execution loop throughout the entire live update transition.
+
+	  Caretaker attaches to preserved CPUs via the cpu_preserve subsystem
+	  and continues guest vCPU execution across the kexec live update
+	  without pausing the guest.
+
+	  If unsure, say N.
diff --git a/virt/kvm/Makefile.kvm b/virt/kvm/Makefile.kvm
index d30fca094c42..ae1fcf9861b7 100644
--- a/virt/kvm/Makefile.kvm
+++ b/virt/kvm/Makefile.kvm
@@ -14,3 +14,10 @@ kvm-$(CONFIG_HAVE_KVM_DIRTY_RING) += $(KVM)/dirty_ring.o
 kvm-$(CONFIG_HAVE_KVM_PFNCACHE) += $(KVM)/pfncache.o
 kvm-$(CONFIG_KVM_GUEST_MEMFD) += $(KVM)/guest_memfd.o
 kvm-$(CONFIG_LIVEUPDATE_GUEST_MEMFD) += $(KVM)/guest_memfd_luo.o $(KVM)/kvm_luo.o
+kvm-$(CONFIG_KVM_CARETAKER) += $(KVM)/caretaker.o
+KASAN_SANITIZE_$(KVM)/caretaker.o := n
+KCSAN_SANITIZE_$(KVM)/caretaker.o := n
+UBSAN_SANITIZE_$(KVM)/caretaker.o := n
+KCOV_INSTRUMENT_$(KVM)/caretaker.o := n
+CFLAGS_REMOVE_$(KVM)/caretaker.o = $(CC_FLAGS_FTRACE)
+CFLAGS_$(KVM)/caretaker.o += -fno-stack-protector $(call cc-option,-ftrivial-auto-var-init=uninitialized) -fno-builtin-memset -fno-builtin-memcpy $(call cc-option,-fno-tree-loop-distribute-patterns) $(call cc-option,-fno-jump-tables)
diff --git a/virt/kvm/caretaker.c b/virt/kvm/caretaker.c
index b4a203562727..dac64c466d76 100644
--- a/virt/kvm/caretaker.c
+++ b/virt/kvm/caretaker.c
@@ -196,7 +196,7 @@ kvm_caretaker_should_exit(struct kvm_caretaker_vcpu *cvcpu)
 
 	cpu_preserved_inval(cvcpu->cb);
 
-	st = READ_ONCE(cvcpu->cb->state);
+	st = smp_load_acquire(&cvcpu->cb->state);
 	if (st != KVM_CARETAKER_PAUSED && st != KVM_CARETAKER_RUNNING)
 		return true;
 
@@ -365,7 +365,7 @@ static bool kvm_caretaker_try_stop(struct kvm_caretaker_cb_ser *cb)
 {
 	cpu_preserved_inval(cb);
 
-	if (READ_ONCE(cb->state) == KVM_CARETAKER_STOPPED)
+	if (smp_load_acquire(&cb->state) == KVM_CARETAKER_STOPPED)
 		return true;
 
 	if (cmpxchg(&cb->state, KVM_CARETAKER_PAUSED,
@@ -397,10 +397,8 @@ int kvm_caretaker_wait_for_attach(struct kvm_caretaker_cb_ser *cb, int pcpu)
 		return 0;
 
 	if (!cpu_is_preserved(pcpu)) {
-		WRITE_ONCE(cb->state, KVM_CARETAKER_STOPPED);
+		smp_store_release(&cb->state, KVM_CARETAKER_STOPPED);
 		cpu_preserved_clean(cb);
-		/* Ensure state update is visible before returning to caller */
-		smp_wmb();
 		return 0;
 	}
 
@@ -450,3 +448,192 @@ void kvm_caretaker_post_attach_vcpu(struct kvm_vcpu *vcpu)
 	}
 }
 
+/**
+ * kvm_caretaker_vcpu_pre_preserve - Submit an On-Core job for a vCPU prior to arch preserve
+ * @vcpu:    KVM vCPU being preserved.
+ * @session: Active Live Update session.
+ * @ser:     Serialized KHO vCPU descriptor to populate.
+ *
+ * Submits a Caretaker job to @session before kvm_arch_vcpu_luo_preserve() runs.
+ * If @session has preserved physical CPUs, assigns the job to the least-loaded
+ * preserved CPU and sets %KVM_VCPU_LUO_FLAG_CARETAKER in @ser->flags so the
+ * architecture hook allocates and populates a Caretaker runtime page.  If
+ * @session has no preserved physical CPUs, returns 0 without setting the flag
+ * so the vCPU is preserved in RAM only.
+ *
+ * Return: 0 on success, or a negative errno on job allocation failure.
+ */
+int kvm_caretaker_vcpu_pre_preserve(struct kvm_vcpu *vcpu,
+				    struct liveupdate_session *session,
+				    struct kvm_vcpu_ser *ser)
+{
+	struct oncore_job *job;
+
+	/*
+	 * Submit with no data: the run callback's argument is the caretaker
+	 * control block, which does not exist until the architecture's
+	 * kvm_arch_vcpu_luo_preserve() has allocated it.  It is installed with
+	 * oncore_job_set_data() from _post_preserve(), before activation.
+	 */
+	job = oncore_session_submit_job(session, kvm_arch_vcpu_caretaker_run,
+					NULL);
+	if (IS_ERR(job))
+		return PTR_ERR(job);
+	if (!job)
+		return 0;
+
+	vcpu->caretaker.job = job;
+	ser->flags |= KVM_VCPU_LUO_FLAG_CARETAKER;
+
+	return 0;
+}
+
+/**
+ * kvm_caretaker_vcpu_post_preserve - Activate the Caretaker On-Core job after arch preserve
+ * @vcpu:     KVM vCPU being preserved.
+ * @session:  Active Live Update session.
+ * @ser:      Serialized KHO vCPU descriptor.
+ * @arch_err: Result of kvm_arch_vcpu_luo_preserve() (non-zero on failure).
+ *
+ * If @arch_err is non-zero, cancels and frees any job created in
+ * kvm_caretaker_vcpu_pre_preserve().  Otherwise, installs @vcpu->caretaker.cb
+ * as the job's run argument, flushes the control block to PoC, and activates
+ * the job on the session's runqueue so the preserved physical CPU begins
+ * executing the vCPU.
+ *
+ * Return: 0 on success, or @arch_err / negative errno on failure.
+ */
+int kvm_caretaker_vcpu_post_preserve(struct kvm_vcpu *vcpu,
+				     struct liveupdate_session *session,
+				     struct kvm_vcpu_ser *ser,
+				     int arch_err)
+{
+	struct kvm_caretaker_cb_ser *cb = vcpu->caretaker.cb;
+	struct oncore_job *job = vcpu->caretaker.job;
+	int err;
+
+	if (arch_err) {
+		if (job)
+			oncore_session_cancel_job(session, job);
+		vcpu->caretaker.job = NULL;
+		vcpu->caretaker.cb = NULL;
+		return arch_err;
+	}
+
+	if (!(ser->flags & KVM_VCPU_LUO_FLAG_CARETAKER) || !cb)
+		return 0;
+
+	oncore_job_set_data(job, cb);
+
+	kvm_caretaker_pause(cb);
+	cpu_preserved_clean(cb);
+
+	err = oncore_session_activate_job(session, job);
+	if (err) {
+		oncore_session_cancel_job(session, job);
+		vcpu->caretaker.job = NULL;
+		kvm_caretaker_stop(cb);
+		vcpu->caretaker.cb = NULL;
+		return err;
+	}
+
+	return 0;
+}
+
+/**
+ * kvm_caretaker_vm_pre_retrieve - Stop Caretaker execution before retrieving VM state
+ *
+ * Detaches preserved physical CPU workloads before the incoming kernel creates
+ * the restored KVM VM instance so Caretaker execution stops immediately when
+ * userspace begins reclaiming the VM session.
+ */
+void kvm_caretaker_vm_pre_retrieve(void)
+{
+	int cpu;
+
+	for_each_cpu(cpu, cpu_get_preserved_mask())
+		cpu_preserved_detach_workload(cpu);
+}
+
+/**
+ * kvm_caretaker_vcpu_pre_retrieve - Stop Caretaker execution before retrieving vCPU state
+ * @vcpu: Incoming KVM vCPU being restored.
+ * @ser:  Serialized KHO vCPU descriptor.
+ *
+ * Resolves @ser->cb and invokes kvm_arch_vcpu_luo_pre_retrieve_caretaker() so
+ * the preserved physical CPU stops guest execution and serializes its latest
+ * state into @ser->arch_state before kvm_arch_vcpu_luo_retrieve() reads it.
+ */
+void kvm_caretaker_vcpu_pre_retrieve(struct kvm_vcpu *vcpu,
+				     struct kvm_vcpu_ser *ser)
+{
+	if (ser->flags & KVM_VCPU_LUO_FLAG_CARETAKER)
+		vcpu->caretaker.cb = KHOSER_LOAD_PTR(ser->cb);
+
+	kvm_arch_vcpu_luo_pre_retrieve_caretaker(vcpu, ser);
+}
+
+/**
+ * kvm_caretaker_vcpu_retrieve - Complete Caretaker hardware attachment during vCPU retrieve
+ * @vcpu: Incoming KVM vCPU being restored.
+ * @ser:  Serialized KHO vCPU descriptor.
+ *
+ * Invokes kvm_arch_vcpu_luo_attach_caretaker() after architectural register
+ * state has been restored into @vcpu.
+ */
+void kvm_caretaker_vcpu_retrieve(struct kvm_vcpu *vcpu,
+				 struct kvm_vcpu_ser *ser)
+{
+	kvm_arch_vcpu_luo_attach_caretaker(vcpu, ser);
+}
+
+/**
+ * kvm_caretaker_vcpu_unpreserve - Roll back Caretaker execution on live update cancellation
+ * @vcpu:    Outgoing KVM vCPU being unpreserved.
+ * @session: Live Update session being cancelled.
+ * @ser:     Serialized KHO vCPU descriptor.
+ *
+ * Stops the vCPU on the preserved physical CPU, synchronizes any guest state
+ * updates back into the outgoing @vcpu, frees KHO telemetry buffers, and
+ * cancels the On-Core job.
+ */
+void kvm_caretaker_vcpu_unpreserve(struct kvm_vcpu *vcpu,
+				   struct liveupdate_session *session,
+				   struct kvm_vcpu_ser *ser)
+{
+	if (ser->flags & KVM_VCPU_LUO_FLAG_CARETAKER) {
+		kvm_arch_vcpu_luo_pre_retrieve_caretaker(vcpu, ser);
+		kvm_arch_vcpu_luo_retrieve(vcpu, ser);
+		kvm_arch_vcpu_luo_attach_caretaker(vcpu, ser);
+		kvm_caretaker_telemetry_free(ser, false);
+	}
+
+	if (vcpu->caretaker.job) {
+		oncore_session_cancel_job(session, vcpu->caretaker.job);
+		vcpu->caretaker.job = NULL;
+	}
+	vcpu->caretaker.cb = NULL;
+}
+
+/**
+ * kvm_caretaker_vcpu_finish - Release Caretaker KHO resources after live update completion
+ * @vcpu:    Incoming KVM vCPU (or %NULL on retrieve failure cleanup).
+ * @session: Completed Live Update session.
+ * @ser:     Serialized KHO vCPU descriptor.
+ *
+ * Ensures the Caretaker vCPU has detached and frees KHO-preserved telemetry
+ * buffers in the incoming kernel.
+ */
+void kvm_caretaker_vcpu_finish(struct kvm_vcpu *vcpu,
+			       struct liveupdate_session *session,
+			       struct kvm_vcpu_ser *ser)
+{
+	if (ser->flags & KVM_VCPU_LUO_FLAG_CARETAKER) {
+		kvm_arch_vcpu_luo_pre_retrieve_caretaker(vcpu, ser);
+		kvm_caretaker_telemetry_free(ser, true);
+	}
+
+	if (vcpu)
+		vcpu->caretaker.cb = NULL;
+}
+
diff --git a/virt/kvm/kvm_luo.c b/virt/kvm/kvm_luo.c
index f7e259bef4b7..fe30f2d41eb4 100644
--- a/virt/kvm/kvm_luo.c
+++ b/virt/kvm/kvm_luo.c
@@ -44,6 +44,7 @@
  */
 #include <linux/liveupdate.h>
 #include <linux/kvm_host.h>
+#include <linux/kvm_caretaker.h>
 #include <linux/pagemap.h>
 #include <linux/fdtable.h>
 #include <linux/file.h>
@@ -55,6 +56,45 @@
 #include <linux/kho/abi/kvm.h>
 #include "kvm_mm.h"
 
+struct kvm_kho_folios_ser *kvm_kho_folios_alloc(unsigned int max_folios)
+{
+	struct kvm_kho_folios_ser *kp;
+	size_t sz = struct_size(kp, folios_pa, max_folios);
+
+	kp = kho_alloc_preserve(sz);
+	if (IS_ERR(kp))
+		return kp;
+
+	kp->nr_folios = 0;
+	return kp;
+}
+
+void kvm_kho_folios_unpreserve(struct kvm_kho_folios_ser *kp)
+{
+	unsigned int i;
+
+	if (!kp)
+		return;
+
+	for (i = 0; i < kp->nr_folios; i++)
+		kho_unpreserve_folio(page_folio(phys_to_page(kp->folios_pa[i])));
+
+	kho_unpreserve_free(kp);
+}
+
+void kvm_kho_folios_finish(struct kvm_kho_folios_ser *kp)
+{
+	unsigned int i;
+
+	if (!kp)
+		return;
+
+	for (i = 0; i < kp->nr_folios; i++)
+		kho_restore_free(phys_to_virt(kp->folios_pa[i]));
+
+	kho_restore_free(kp);
+}
+
 static bool kvm_luo_can_preserve(struct liveupdate_file_handler *handler,
 				 struct file *file)
 {
@@ -81,6 +121,7 @@ static int kvm_luo_preserve(struct liveupdate_file_op_args *args)
 	 * architecture that does not implement the hook.
 	 */
 	ser->type = 0;
+	ser->kho_folios.phys = 0;
 	err = kvm_arch_vm_luo_preserve(kvm, ser);
 	if (err) {
 		kho_unpreserve_free(ser);
@@ -104,6 +145,8 @@ static int kvm_luo_retrieve(struct liveupdate_file_op_args *args)
 	if (!args->serialized_data)
 		return -EINVAL;
 
+	kvm_caretaker_vm_pre_retrieve();
+
 	ser = phys_to_virt(args->serialized_data);
 
 	snprintf(fdname, sizeof(fdname), "%d",
@@ -130,6 +173,7 @@ static int kvm_luo_retrieve(struct liveupdate_file_op_args *args)
 
 err_free_ser:
 	kvm_arch_vm_luo_finish(ser);
+	kvm_kho_folios_finish(KHOSER_LOAD_PTR(ser->kho_folios));
 	kho_restore_free(ser);
 	return err;
 }
@@ -150,6 +194,9 @@ static void kvm_luo_unpreserve(struct liveupdate_file_op_args *args)
 
 	ser = phys_to_virt(args->serialized_data);
 	kvm_arch_vm_luo_unpreserve(kvm, ser);
+	if (kvm)
+		kvm->kho_folios = NULL;
+	kvm_kho_folios_unpreserve(KHOSER_LOAD_PTR(ser->kho_folios));
 	kho_unpreserve_free(ser);
 }
 
@@ -165,6 +212,7 @@ static void kvm_luo_finish(struct liveupdate_file_op_args *args)
 
 	ser = phys_to_virt(args->serialized_data);
 	kvm_arch_vm_luo_finish(ser);
+	kvm_kho_folios_finish(KHOSER_LOAD_PTR(ser->kho_folios));
 	kho_restore_free(ser);
 }
 
@@ -217,8 +265,15 @@ static int kvm_vcpu_luo_preserve(struct liveupdate_file_op_args *args)
 	ser->vcpu_id = vcpu->vcpu_id;
 	ser->flags = 0;
 	ser->vm_token = vm_token;
-
-	err = kvm_arch_vcpu_luo_preserve(vcpu, ser);
+	ser->arch_state.phys = 0;
+	ser->cb.phys = 0;
+
+	err = kvm_caretaker_vcpu_pre_preserve(vcpu, args->session, ser);
+	if (!err) {
+		err = kvm_arch_vcpu_luo_preserve(vcpu, ser);
+		err = kvm_caretaker_vcpu_post_preserve(vcpu, args->session,
+						       ser, err);
+	}
 	mutex_unlock(&vcpu->mutex);
 	if (err) {
 		kho_unpreserve_free(ser);
@@ -259,16 +314,19 @@ static int kvm_vcpu_luo_retrieve(struct liveupdate_file_op_args *args)
 	}
 
 	vcpu = file->private_data;
+	kvm_caretaker_vcpu_pre_retrieve(vcpu, ser);
 	err = kvm_arch_vcpu_luo_retrieve(vcpu, ser);
 	if (err) {
 		fput(file);
 		goto err_free_ser;
 	}
+	kvm_caretaker_vcpu_retrieve(vcpu, ser);
 
 	args->file = file;
 	return 0;
 
 err_free_ser:
+	kvm_caretaker_vcpu_finish(NULL, args->session, ser);
 	kvm_arch_vcpu_luo_finish(ser);
 	kho_restore_free(ser);
 	return err;
@@ -276,18 +334,24 @@ static int kvm_vcpu_luo_retrieve(struct liveupdate_file_op_args *args)
 
 static void kvm_vcpu_luo_unpreserve(struct liveupdate_file_op_args *args)
 {
+	struct kvm_vcpu *vcpu = args->file ? args->file->private_data : NULL;
 	struct kvm_vcpu_ser *ser;
 
 	if (WARN_ON_ONCE(!args->serialized_data))
 		return;
 
 	ser = phys_to_virt(args->serialized_data);
+
+	if (vcpu)
+		kvm_caretaker_vcpu_unpreserve(vcpu, args->session, ser);
+
 	kvm_arch_vcpu_luo_unpreserve(ser);
 	kho_unpreserve_free(ser);
 }
 
 static void kvm_vcpu_luo_finish(struct liveupdate_file_op_args *args)
 {
+	struct kvm_vcpu *vcpu = args->file ? args->file->private_data : NULL;
 	struct kvm_vcpu_ser *ser;
 
 	if (args->retrieve_status < 0)
@@ -297,6 +361,8 @@ static void kvm_vcpu_luo_finish(struct liveupdate_file_op_args *args)
 		return;
 
 	ser = phys_to_virt(args->serialized_data);
+
+	kvm_caretaker_vcpu_finish(vcpu, args->session, ser);
 	kvm_arch_vcpu_luo_finish(ser);
 	kho_restore_free(ser);
 }
diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c
index ae91123e1822..4f661c5d2ee2 100644
--- a/virt/kvm/kvm_main.c
+++ b/virt/kvm/kvm_main.c
@@ -4521,6 +4521,11 @@ static long kvm_vcpu_ioctl(struct file *filp,
 
 			put_pid(oldpid);
 		}
+
+		if (!kvm_caretaker_vcpu_is_attached(vcpu)) {
+			r = -EBUSY;
+			break;
+		}
 		vcpu->wants_to_run = !READ_ONCE(vcpu->run->immediate_exit__unsafe);
 		r = kvm_arch_vcpu_ioctl_run(vcpu);
 		vcpu->wants_to_run = false;
@@ -4989,6 +4994,8 @@ static int kvm_vm_ioctl_check_extension_generic(struct kvm *kvm, long arg)
 #endif
 	case KVM_CAP_VCPU_PRESERVE:
 		return IS_ENABLED(CONFIG_HAVE_KVM_ARCH_VCPU_PRESERVE);
+	case KVM_CAP_CARETAKER:
+		return IS_ENABLED(CONFIG_KVM_CARETAKER);
 	default:
 		break;
 	}
-- 
2.55.0.1082.g2b9226bbc0-goog

[RFC PATCH 31/46] KVM: caretaker: Add generic KHO ABI telemetry and debugfs reporting

[Pasha Tatashin]

When a preserved vCPU executes inside the Caretaker during a host live
update blackout window, standard kernel tracing and KVM exit counters
are unavailable because the host kernel is being replaced via kexec. If
a guest vCPU hits an unhandled exit and parks, or experiences unexpected
exit patterns during the transition, the incoming kernel has no
visibility into what occurred while the vCPU was detached.

Add optional cross-kexec Caretaker execution telemetry behind
CONFIG_KVM_CARETAKER_DEBUG. During detached execution, the Caretaker
records run counts, exit counts, stall counts, the last exit reason and
guest PC, and stall diagnostics into a KHO-preserved telemetry buffer
referenced from struct kvm_caretaker_cb_ser. When the incoming kernel
reclaims the vCPU, it logs a summary to dmesg and exposes the preserved
telemetry under /sys/kernel/debug/kvm/<vm>/vcpu<N>/caretaker_telemetry.

Signed-off-by: Pasha Tatashin <pasha.tatashin@soleen.com>
---
 include/linux/kvm_caretaker.h |  97 +++++++++++++++++++++++++++
 virt/kvm/Kconfig              |  10 +++
 virt/kvm/Makefile.kvm         |   1 +
 virt/kvm/caretaker_debug.c    | 120 ++++++++++++++++++++++++++++++++++
 virt/kvm/kvm_main.c           |   1 +
 5 files changed, 229 insertions(+)
 create mode 100644 virt/kvm/caretaker_debug.c

diff --git a/include/linux/kvm_caretaker.h b/include/linux/kvm_caretaker.h
index 0d4bdde7b1c3..f889d17212b0 100644
--- a/include/linux/kvm_caretaker.h
+++ b/include/linux/kvm_caretaker.h
@@ -108,11 +108,13 @@ struct kvm_caretaker_ops {
  * @cb:        Pointer to KHO-preserved Caretaker control block (&struct kvm_caretaker_cb_ser).
  * @ops:       Architecture operations vector (&struct kvm_caretaker_ops).
  * @arch_data: Architecture-specific runtime context passed to @ops callbacks.
+ * @telemetry: Pointer to KHO-preserved telemetry counters, or %NULL.
  */
 struct kvm_caretaker_vcpu {
 	struct kvm_caretaker_cb_ser *cb;
 	const struct kvm_caretaker_ops *ops;
 	void *arch_data;
+	struct kvm_caretaker_telemetry_ser *telemetry;
 };
 
 struct kvm_kho_folios_ser *kvm_kho_folios_alloc(unsigned int max_folios);
@@ -260,11 +262,106 @@ void kvm_caretaker_vcpu_finish(struct kvm_vcpu *vcpu,
 			       struct liveupdate_session *session,
 			       struct kvm_vcpu_ser *ser);
 
+#ifdef CONFIG_KVM_CARETAKER_DEBUG
+void kvm_caretaker_telemetry_init(struct kvm_caretaker_vcpu *cvcpu,
+				  struct oncore_session *sess);
+void kvm_caretaker_telemetry_report(struct kvm_vcpu *vcpu,
+				    struct kvm_caretaker_cb_ser *cb);
+void kvm_caretaker_telemetry_free(struct kvm_vcpu_ser *ser, bool is_incoming);
+void kvm_caretaker_create_vcpu_debugfs(struct kvm_vcpu *vcpu,
+				       struct dentry *debugfs_dentry);
+
+/**
+ * kvm_caretaker_telemetry_run - Record a guest entry in Caretaker telemetry
+ * @cvcpu: Common Caretaker vCPU descriptor.
+ */
+static __always_inline __caretaker_text void
+kvm_caretaker_telemetry_run(struct kvm_caretaker_vcpu *cvcpu)
+{
+	if (cvcpu->telemetry)
+		cvcpu->telemetry->total_runs++;
+}
+
+/**
+ * kvm_caretaker_telemetry_exit - Record a decoded VM exit in Caretaker telemetry
+ * @cvcpu: Common Caretaker vCPU descriptor.
+ * @exit:  Decoded VM exit descriptor.
+ */
+static __always_inline __caretaker_text void
+kvm_caretaker_telemetry_exit(struct kvm_caretaker_vcpu *cvcpu,
+			     const struct kvm_caretaker_exit *exit)
+{
+	if (cvcpu->telemetry) {
+		cvcpu->telemetry->total_exits++;
+		cvcpu->telemetry->last_exit_reason = exit->raw_reason;
+		cvcpu->telemetry->last_exit_rip = exit->rip;
+	}
+}
+
+/**
+ * kvm_caretaker_telemetry_stall - Record an unhandled stall exit in Caretaker telemetry
+ * @cvcpu:  Common Caretaker vCPU descriptor.
+ * @reason: Raw hardware exit reason or entry failure code.
+ * @rip:    Guest instruction pointer at stall (0 if entry failure).
+ */
+static __always_inline __caretaker_text void
+kvm_caretaker_telemetry_stall(struct kvm_caretaker_vcpu *cvcpu,
+			      u64 reason, u64 rip)
+{
+	if (cvcpu->telemetry) {
+		cvcpu->telemetry->stall_count++;
+		cvcpu->telemetry->last_exit_reason = reason;
+		cvcpu->telemetry->stall_exit_reason = reason;
+		if (rip)
+			cvcpu->telemetry->stall_exit_rip = rip;
+	}
+}
+
+/**
+ * kvm_caretaker_telemetry_flush - Clean Caretaker telemetry counters to PoC
+ * @cvcpu: Common Caretaker vCPU descriptor.
+ */
+static __always_inline __caretaker_text void
+kvm_caretaker_telemetry_flush(struct kvm_caretaker_vcpu *cvcpu)
+{
+	if (cvcpu->telemetry)
+		cpu_preserved_clean_sz(cvcpu->telemetry,
+				       sizeof(*cvcpu->telemetry));
+}
+#else
+static inline void kvm_caretaker_telemetry_init(struct kvm_caretaker_vcpu *cvcpu,
+						struct oncore_session *sess) {}
+
+static inline void kvm_caretaker_telemetry_report(struct kvm_vcpu *vcpu,
+						  struct kvm_caretaker_cb_ser *cb) {}
+
+static inline void kvm_caretaker_telemetry_free(struct kvm_vcpu_ser *ser,
+						bool is_incoming) {}
+
+static inline void kvm_caretaker_create_vcpu_debugfs(struct kvm_vcpu *vcpu,
+						     struct dentry *debugfs_dentry) {}
+static __always_inline void
+kvm_caretaker_telemetry_run(struct kvm_caretaker_vcpu *cvcpu) {}
+static __always_inline void
+kvm_caretaker_telemetry_exit(struct kvm_caretaker_vcpu *cvcpu,
+			     const struct kvm_caretaker_exit *exit) {}
+static __always_inline void
+kvm_caretaker_telemetry_stall(struct kvm_caretaker_vcpu *cvcpu,
+			      u64 reason, u64 rip) {}
+static __always_inline void
+kvm_caretaker_telemetry_flush(struct kvm_caretaker_vcpu *cvcpu) {}
+#endif
+
 #else /* !CONFIG_KVM_CARETAKER */
 
 #define __caretaker_text
 #define __caretaker_data
 
+struct dentry;
+
+static inline void kvm_caretaker_create_vcpu_debugfs(struct kvm_vcpu *vcpu,
+						     struct dentry *debugfs_dentry) {}
+
 static inline bool kvm_caretaker_is_stopped(const struct kvm_caretaker_cb_ser *cb)
 {
 	return true;
diff --git a/virt/kvm/Kconfig b/virt/kvm/Kconfig
index 165e0c85b8aa..bee73dd079c6 100644
--- a/virt/kvm/Kconfig
+++ b/virt/kvm/Kconfig
@@ -153,3 +153,13 @@ config KVM_CARETAKER
 	  without pausing the guest.
 
 	  If unsure, say N.
+
+config KVM_CARETAKER_DEBUG
+	bool "Caretaker execution telemetry and debug reporting"
+	depends on KVM_CARETAKER
+	default n
+	help
+	  Enable Caretaker telemetry collection and reporting of VM exits,
+	  runs, and stall diagnostics via debugfs and dmesg upon vCPU resume
+	  following a host kexec live update or cancelled live update session.
+	  If unsure, say N.
diff --git a/virt/kvm/Makefile.kvm b/virt/kvm/Makefile.kvm
index ae1fcf9861b7..157d0bf6132d 100644
--- a/virt/kvm/Makefile.kvm
+++ b/virt/kvm/Makefile.kvm
@@ -15,6 +15,7 @@ kvm-$(CONFIG_HAVE_KVM_PFNCACHE) += $(KVM)/pfncache.o
 kvm-$(CONFIG_KVM_GUEST_MEMFD) += $(KVM)/guest_memfd.o
 kvm-$(CONFIG_LIVEUPDATE_GUEST_MEMFD) += $(KVM)/guest_memfd_luo.o $(KVM)/kvm_luo.o
 kvm-$(CONFIG_KVM_CARETAKER) += $(KVM)/caretaker.o
+kvm-$(CONFIG_KVM_CARETAKER_DEBUG) += $(KVM)/caretaker_debug.o
 KASAN_SANITIZE_$(KVM)/caretaker.o := n
 KCSAN_SANITIZE_$(KVM)/caretaker.o := n
 UBSAN_SANITIZE_$(KVM)/caretaker.o := n
diff --git a/virt/kvm/caretaker_debug.c b/virt/kvm/caretaker_debug.c
new file mode 100644
index 000000000000..8905938bfec3
--- /dev/null
+++ b/virt/kvm/caretaker_debug.c
@@ -0,0 +1,120 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * Copyright (c) 2026, Google LLC.
+ * Pasha Tatashin <pasha.tatashin@soleen.com>
+ *
+ * KVM Caretaker execution telemetry and debugfs reporting.
+ */
+
+#include <linux/cpu_preserve.h>
+#include <linux/debugfs.h>
+#include <linux/kernel.h>
+#include <linux/kexec_handover.h>
+#include <linux/kho/abi/kvm.h>
+#include <linux/kvm_caretaker.h>
+#include <linux/kvm_host.h>
+#include <linux/oncore.h>
+#include <linux/seq_file.h>
+
+void kvm_caretaker_telemetry_init(struct kvm_caretaker_vcpu *cvcpu,
+				  struct oncore_session *sess)
+{
+	struct kvm_caretaker_telemetry_ser *tel;
+
+	tel = kho_alloc_preserve(sizeof(*tel));
+	if (IS_ERR_OR_NULL(tel))
+		return;
+
+	memset(tel, 0, sizeof(*tel));
+	oncore_session_map_buffer(sess, tel, sizeof(*tel));
+	cpu_preserved_clean_sz(tel, sizeof(*tel));
+	cvcpu->telemetry = tel;
+	KHOSER_STORE_PTR(cvcpu->cb->telemetry, tel);
+}
+
+void kvm_caretaker_telemetry_report(struct kvm_vcpu *vcpu,
+				    struct kvm_caretaker_cb_ser *cb)
+{
+	struct kvm_caretaker_telemetry_ser *tel;
+
+	if (!vcpu || !cb)
+		return;
+
+	cpu_preserved_inval(cb);
+	tel = KHOSER_LOAD_PTR(cb->telemetry);
+	if (!tel)
+		return;
+
+	cpu_preserved_inval(tel);
+	vcpu->caretaker.last_telemetry = *tel;
+
+	pr_info("caretaker: vcpu %d resume: runs=%llu exits=%llu stalls=%llu last_exit=0x%llx (%llu) rip=0x%llx\n",
+		vcpu->vcpu_id, tel->total_runs, tel->total_exits,
+		tel->stall_count, tel->last_exit_reason,
+		tel->last_exit_reason, tel->last_exit_rip);
+	if (tel->stall_count) {
+		pr_info("caretaker: vcpu %d stall info: reason=0x%llx (%llu) rip=0x%llx\n",
+			vcpu->vcpu_id, tel->stall_exit_reason,
+			tel->stall_exit_reason, tel->stall_exit_rip);
+	}
+}
+
+void kvm_caretaker_telemetry_free(struct kvm_vcpu_ser *ser, bool is_incoming)
+{
+	struct kvm_caretaker_telemetry_ser *tel;
+	struct kvm_caretaker_cb_ser *cb;
+
+	if (!ser || !(ser->flags & KVM_VCPU_LUO_FLAG_CARETAKER))
+		return;
+
+	cb = KHOSER_LOAD_PTR(ser->cb);
+	tel = cb ? KHOSER_LOAD_PTR(cb->telemetry) : NULL;
+	if (tel) {
+		cb->telemetry.phys = 0;
+		if (is_incoming)
+			kho_restore_free(tel);
+		else
+			kho_unpreserve_free(tel);
+	}
+}
+
+static int caretaker_telemetry_show(struct seq_file *m, void *v)
+{
+	struct kvm_vcpu *vcpu = m->private;
+	const struct kvm_caretaker_telemetry_ser *t =
+		&vcpu->caretaker.last_telemetry;
+	struct kvm_caretaker_cb_ser *cb = READ_ONCE(vcpu->caretaker.cb);
+
+	if (cb) {
+		struct kvm_caretaker_telemetry_ser *live;
+
+		cpu_preserved_inval(cb);
+		live = KHOSER_LOAD_PTR(cb->telemetry);
+		if (live) {
+			cpu_preserved_inval(live);
+			t = live;
+		}
+	}
+
+	seq_printf(m, "runs: %llu\n", t->total_runs);
+	seq_printf(m, "exits: %llu\n", t->total_exits);
+	seq_printf(m, "stalls: %llu\n", t->stall_count);
+	seq_printf(m, "last_exit_reason: 0x%llx (%llu)\n",
+		   t->last_exit_reason, t->last_exit_reason);
+	seq_printf(m, "last_exit_rip: 0x%llx\n", t->last_exit_rip);
+	seq_printf(m, "stall_exit_reason: 0x%llx (%llu)\n",
+		   t->stall_exit_reason, t->stall_exit_reason);
+	seq_printf(m, "stall_exit_rip: 0x%llx\n", t->stall_exit_rip);
+	return 0;
+}
+DEFINE_SHOW_ATTRIBUTE(caretaker_telemetry);
+
+void kvm_caretaker_create_vcpu_debugfs(struct kvm_vcpu *vcpu,
+				       struct dentry *debugfs_dentry)
+{
+	if (!debugfs_dentry)
+		return;
+
+	debugfs_create_file("caretaker_telemetry", 0444, debugfs_dentry, vcpu,
+			    &caretaker_telemetry_fops);
+}
diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c
index 4f661c5d2ee2..84b4b96d7916 100644
--- a/virt/kvm/kvm_main.c
+++ b/virt/kvm/kvm_main.c
@@ -4156,6 +4156,7 @@ static void kvm_create_vcpu_debugfs(struct kvm_vcpu *vcpu)
 	debugfs_create_file("pid", 0444, debugfs_dentry, vcpu,
 			    &vcpu_get_pid_fops);
 
+	kvm_caretaker_create_vcpu_debugfs(vcpu, debugfs_dentry);
 	kvm_arch_create_vcpu_debugfs(vcpu, debugfs_dentry);
 }
 #endif
-- 
2.55.0.1082.g2b9226bbc0-goog

[RFC PATCH 32/46] KVM: x86: Add TDP MMU KHO preservation helpers

[Pasha Tatashin]

Add arch/x86/kvm/mmu/kho.c to preserve and adopt TDP MMU EPT/NPT
root page tables across Kexec Handover (KHO) live updates.

Signed-off-by: Pasha Tatashin <pasha.tatashin@soleen.com>
---
 arch/x86/kvm/mmu.h     |   7 ++
 arch/x86/kvm/mmu/kho.c | 193 +++++++++++++++++++++++++++++++++++++++++
 2 files changed, 200 insertions(+)
 create mode 100644 arch/x86/kvm/mmu/kho.c

diff --git a/arch/x86/kvm/mmu.h b/arch/x86/kvm/mmu.h
index 2ae7f9ed4cf8..f35948f0906c 100644
--- a/arch/x86/kvm/mmu.h
+++ b/arch/x86/kvm/mmu.h
@@ -410,4 +410,11 @@ static inline bool kvm_is_gfn_alias(struct kvm *kvm, gfn_t gfn)
 {
 	return gfn & kvm_gfn_direct_bits(kvm);
 }
+
+/*
+ * Declared here rather than in asm/kvm_host.h: it is internal to
+ * arch/x86/kvm and has no callers outside it.  Defined in mmu/kho.c.
+ */
+int kvm_mmu_preserve_kho(struct kvm *kvm);
+
 #endif
diff --git a/arch/x86/kvm/mmu/kho.c b/arch/x86/kvm/mmu/kho.c
new file mode 100644
index 000000000000..a04600f0c3e0
--- /dev/null
+++ b/arch/x86/kvm/mmu/kho.c
@@ -0,0 +1,193 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Copyright (c) 2026, Google LLC.
+ * Pasha Tatashin <pasha.tatashin@soleen.com>
+ *
+ * KHO preservation of the x86 KVM MMU page tables.
+ *
+ * An orphaned vCPU keeps running its guest out of the shadow/TDP page tables
+ * while the VM is detached, so every page those tables are built from has to
+ * survive the kexec.
+ */
+
+#include <linux/kexec_handover.h>
+#include <linux/kvm_host.h>
+
+#include "mmu.h"
+#include "mmu_internal.h"
+#include "spte.h"
+#include "tdp_iter.h"
+#include "tdp_mmu.h"
+
+/*
+ * Page-pointer accumulator.
+ *
+ * kho_preserve_pages() cannot be called while holding kvm->mmu_lock: it is a
+ * rwlock_t, so the section is atomic, whereas kho_radix_add_key() below it
+ * calls might_sleep(), takes a mutex and allocates with GFP_KERNEL.  So the
+ * walk runs in two phases -- collect the pages under the lock, preserve them
+ * after dropping it.
+ *
+ * A NULL @pages simply counts, which is how the caller sizes the array.
+ */
+struct kvm_mmu_kho_pages {
+	struct page **pages;
+	unsigned long nr;
+	unsigned long capacity;
+	bool overflow;
+};
+
+static void kvm_mmu_kho_add(struct kvm_mmu_kho_pages *acc, struct page *page)
+{
+	if (!acc->pages) {
+		acc->nr++;
+		return;
+	}
+
+	if (acc->nr >= acc->capacity) {
+		acc->overflow = true;
+		return;
+	}
+
+	acc->pages[acc->nr++] = page;
+}
+
+static void kvm_tdp_mmu_collect(struct kvm *kvm,
+				struct kvm_mmu_kho_pages *acc)
+{
+	gfn_t end = kvm_mmu_max_gfn() + 1;
+	struct kvm_mmu_page *root;
+	struct tdp_iter iter;
+
+	lockdep_assert_held_write(&kvm->mmu_lock);
+
+	rcu_read_lock();
+	list_for_each_entry_rcu(root, &kvm->arch.tdp_mmu_roots, link) {
+		if (root->spt)
+			kvm_mmu_kho_add(acc, virt_to_page(root->spt));
+
+		for_each_tdp_pte(iter, kvm, root, 0, end) {
+			struct page *page;
+
+			if (!is_shadow_present_pte(iter.old_spte) ||
+			    is_last_spte(iter.old_spte, iter.level))
+				continue;
+
+			page = pfn_to_page(spte_to_pfn(iter.old_spte));
+			kvm_mmu_kho_add(acc, page);
+		}
+	}
+	rcu_read_unlock();
+}
+
+/*
+ * The per-vCPU root page tables are not linked into active_mmu_pages, so they
+ * have to be walked separately.  pae_root, pml4_root and pml5_root are each
+ * NULL unless the corresponding paging mode is in use.
+ */
+static void kvm_mmu_collect_roots(struct kvm_mmu *mmu,
+				  struct kvm_mmu_kho_pages *acc)
+{
+	void *const roots[] = { mmu->pae_root, mmu->pml4_root, mmu->pml5_root };
+	int i;
+
+	for (i = 0; i < ARRAY_SIZE(roots); i++) {
+		if (roots[i])
+			kvm_mmu_kho_add(acc, virt_to_page(roots[i]));
+	}
+}
+
+/* Collect every page backing this VM's MMU.  Must be called under mmu_lock. */
+static void kvm_mmu_collect_all(struct kvm *kvm,
+				struct kvm_mmu_kho_pages *acc)
+{
+	struct kvm_mmu_page *sp;
+	struct kvm_vcpu *vcpu;
+	unsigned long i;
+
+	lockdep_assert_held_write(&kvm->mmu_lock);
+
+	acc->nr = 0;
+	acc->overflow = false;
+
+	if (tdp_mmu_enabled)
+		kvm_tdp_mmu_collect(kvm, acc);
+
+	list_for_each_entry(sp, &kvm->arch.active_mmu_pages, link) {
+		if (sp->spt)
+			kvm_mmu_kho_add(acc, virt_to_page(sp->spt));
+	}
+
+	kvm_for_each_vcpu(i, vcpu, kvm) {
+		if (vcpu->arch.mmu)
+			kvm_mmu_collect_roots(vcpu->arch.mmu, acc);
+		kvm_mmu_collect_roots(&vcpu->arch.guest_mmu, acc);
+	}
+}
+
+int kvm_mmu_preserve_kho(struct kvm *kvm)
+{
+	struct kvm_mmu_kho_pages acc = {};
+	struct kvm_kho_folios_ser *kp;
+	unsigned long i;
+	int ret = 0;
+	int attempt;
+
+	/*
+	 * Size the array, then fill it.  The guest can fault in new page
+	 * tables between the two passes, so re-check for overflow and retry
+	 * with a larger array; the slack makes repeated growth unlikely.
+	 */
+	for (attempt = 0; attempt < 5; attempt++) {
+		write_lock(&kvm->mmu_lock);
+		kvm_mmu_collect_all(kvm, &acc);
+		write_unlock(&kvm->mmu_lock);
+
+		if (acc.pages && !acc.overflow)
+			break;
+
+		acc.capacity = acc.nr + (acc.nr >> 2) + 16;
+		kvfree(acc.pages);
+		acc.pages = kvmalloc_array(acc.capacity, sizeof(*acc.pages),
+					   GFP_KERNEL);
+		if (!acc.pages)
+			return -ENOMEM;
+	}
+
+	if (acc.overflow) {
+		ret = -EAGAIN;
+		goto out;
+	}
+
+	if (!acc.nr)
+		goto out;
+
+	kp = kvm_kho_folios_alloc(acc.nr);
+	if (IS_ERR(kp)) {
+		ret = PTR_ERR(kp);
+		goto out;
+	}
+
+	for (i = 0; i < acc.nr; i++) {
+		ret = kho_preserve_folio(page_folio(acc.pages[i]));
+		if (ret) {
+			/*
+			 * Undo the partial preservation: leaving pages marked
+			 * would pin them in the incoming kernel forever with
+			 * nothing owning them.
+			 */
+			while (i--)
+				kho_unpreserve_folio(page_folio(acc.pages[i]));
+			kho_unpreserve_free(kp);
+			goto out;
+		}
+		kp->folios_pa[i] = page_to_phys(acc.pages[i]);
+	}
+
+	kp->nr_folios = acc.nr;
+	kvm->kho_folios = kp;
+
+out:
+	kvfree(acc.pages);
+	return ret;
+}
-- 
2.55.0.1082.g2b9226bbc0-goog

[RFC PATCH 33/46] KVM: x86: Add Caretaker x86 KHO ABI and runtime context headers

[Pasha Tatashin]

Add struct kvm_caretaker_arch_ser, x86 Caretaker runtime context headers,
and generated assembly offsets for Caretaker guest entry/exit.

Signed-off-by: Pasha Tatashin <pasha.tatashin@soleen.com>
---
 arch/x86/kvm/caretaker.h        | 230 ++++++++++++++++++++++++++++++++
 arch/x86/kvm/caretaker_asm.h    | 117 ++++++++++++++++
 arch/x86/kvm/kvm-asm-offsets.c  |  32 +++++
 include/linux/kho/abi/kvm_x86.h |  22 +++
 4 files changed, 401 insertions(+)
 create mode 100644 arch/x86/kvm/caretaker.h
 create mode 100644 arch/x86/kvm/caretaker_asm.h

diff --git a/arch/x86/kvm/caretaker.h b/arch/x86/kvm/caretaker.h
new file mode 100644
index 000000000000..9dd8815d414b
--- /dev/null
+++ b/arch/x86/kvm/caretaker.h
@@ -0,0 +1,230 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/*
+ * Copyright (c) 2026, Google LLC.
+ * Pasha Tatashin <pasha.tatashin@soleen.com>
+ *
+ * Header for x86 KVM Caretaker common C execution engine and helpers.
+ */
+#ifndef __ARCH_X86_KVM_CARETAKER_H
+#define __ARCH_X86_KVM_CARETAKER_H
+
+/*
+ * Size of the caretaker's standalone VMX exit and IST stack in the upper half
+ * of struct caretaker_x86_page ([2048..4096)).
+ */
+#define CXP_STACK_SIZE		2048
+
+/* COM1 serial port range intercepted by Caretaker */
+#define COM1_PORT_BASE		0x3f8
+#define COM1_PORT_END		0x3ff
+
+/* Number of x2APIC MSRs (0x800 - 0x83f) */
+#define X2APIC_MSR_COUNT	0x40
+
+/* Caretaker APIC version: 6 LVT entries (max index 5), version 0x14 */
+#define CARETAKER_APIC_LVR	((5 << 16) | 0x14)
+
+#ifndef __ASSEMBLY__
+
+#include <linux/types.h>
+#include <linux/oncore.h>
+#include <linux/kho/abi/kvm_x86.h>
+#include <linux/kvm_caretaker.h>
+
+/* Architecture-specific VM exit types for x86 */
+#define KVM_CARETAKER_EXIT_CPUID \
+	((enum kvm_caretaker_exit_type)(KVM_CARETAKER_EXIT_ARCH + 1))
+#define KVM_CARETAKER_EXIT_MSR \
+	((enum kvm_caretaker_exit_type)(KVM_CARETAKER_EXIT_ARCH + 2))
+#define KVM_CARETAKER_EXIT_RDTSC \
+	((enum kvm_caretaker_exit_type)(KVM_CARETAKER_EXIT_ARCH + 3))
+
+/* 8250 UART register state for guest early printk emulation */
+struct caretaker_uart {
+	u8 lcr;
+	u8 ier;
+	u8 mcr;
+	u8 scr;
+	u8 dll;
+	u8 dlm;
+};
+
+#include <linux/processor.h>
+#include <asm/desc.h>
+#include <asm/page.h>
+
+static inline void caretaker_set_tss_desc(struct desc_struct *gdt,
+					  unsigned long addr, unsigned int size)
+{
+	struct ldttss_desc *desc = (struct ldttss_desc *)&gdt[GDT_ENTRY_TSS];
+
+	memset(desc, 0, sizeof(*desc));
+	desc->limit0 = (u16)size;
+	desc->base0 = (u16)addr;
+	desc->base1 = (addr >> 16) & 0xFF;
+	desc->type = DESC_TSS;
+	desc->p = 1;
+	desc->limit1 = (size >> 16) & 0xF;
+	desc->base2 = (addr >> 24) & 0xFF;
+	desc->base3 = (u32)(addr >> 32);
+}
+
+struct kvm_vcpu;
+struct kvm_caretaker_exit;
+
+/**
+ * struct caretaker_x86_page - Vendor-common x86 Caretaker runtime page (4 KB)
+ * @abi:              Cross-kexec KHO ABI header (must remain at offset 0).
+ * @vcpu:             Common Caretaker vCPU execution engine descriptor.
+ * @arch_state:       Pointer to KHO-preserved vCPU architectural state buffer.
+ * @stack_orig:       Saved caller %rsp across vmx_caretaker_enter() guest entry.
+ * @rax:              Guest RAX switched in caretaker_vmenter.S.
+ * @rbx:              Guest RBX switched in caretaker_vmenter.S.
+ * @rcx:              Guest RCX switched in caretaker_vmenter.S.
+ * @rdx:              Guest RDX switched in caretaker_vmenter.S.
+ * @rsi:              Guest RSI switched in caretaker_vmenter.S.
+ * @rdi:              Guest RDI switched in caretaker_vmenter.S.
+ * @rbp:              Guest RBP switched in caretaker_vmenter.S.
+ * @r8:               Guest R8 switched in caretaker_vmenter.S.
+ * @r9:               Guest R9 switched in caretaker_vmenter.S.
+ * @r10:              Guest R10 switched in caretaker_vmenter.S.
+ * @r11:              Guest R11 switched in caretaker_vmenter.S.
+ * @r12:              Guest R12 switched in caretaker_vmenter.S.
+ * @r13:              Guest R13 switched in caretaker_vmenter.S.
+ * @r14:              Guest R14 switched in caretaker_vmenter.S.
+ * @r15:              Guest R15 switched in caretaker_vmenter.S.
+ * @last_exit_rip:    Guest RIP captured on VM exit and updated on instruction skip.
+ * @last_exit_rsp:    Guest RSP captured on VM exit.
+ * @last_exit_rflags: Guest RFLAGS captured on VM exit.
+ * @host_cr3:         Isolated Caretaker page table root PA loaded during run.
+ * @cr3:              Guest CR3 shadow value captured on VM exit.
+ * @cr0:              Guest CR0 shadow value captured on VM exit.
+ * @cr4:              Guest CR4 shadow value captured on VM exit.
+ * @efer:             Guest EFER shadow value captured on VM exit.
+ * @kernel_gs_base:   Guest MSR_KERNEL_GS_BASE switched across VMX entry/exit.
+ * @uart:             Emulated 8250 UART register state for early guest console.
+ * @save_guest_fpu:   True if guest FPU state can be captured via XSAVE at detach.
+ * @gdt:              Per-vCPU preserved GDT containing the active Caretaker TSS.
+ * @tss:              Per-vCPU hardware TSS whose SP0/IST point to @stack.
+ * @stack:            2 KB standalone host stack occupying the upper half of the
+ *                    page ([2048..4096)).  VMX sets HOST_RSP to the top of this
+ *                    page so vmx_caretaker_exit_handler can recover the base
+ *                    address of struct caretaker_x86_page via (%rsp & PAGE_MASK).
+ *
+ * Cross-kexec invariant: Only @abi (and @arch_state) may be dereferenced by the
+ * incoming kernel.  All remaining fields are private to the preserved Caretaker
+ * text executing on the isolated physical CPU during the kexec handover window.
+ */
+struct caretaker_x86_page {
+	/* KHO ABI prefix (offset 0) and common scheduler descriptor */
+	struct kvm_caretaker_arch_ser abi;
+	struct kvm_caretaker_vcpu vcpu;
+
+	struct kvm_vcpu_arch_ser *arch_state;
+	u64 stack_orig;
+
+	/* Guest GPRs switched in caretaker_vmenter.S */
+	u64 rax, rbx, rcx, rdx, rsi, rdi, rbp;
+	u64 r8, r9, r10, r11, r12, r13, r14, r15;
+
+	/* Guest instruction/stack pointers and control registers at VM exit */
+	u64 last_exit_rip;
+	u64 last_exit_rsp;
+	u64 last_exit_rflags;
+	u64 host_cr3;
+	u64 cr3;
+	u64 cr0;
+	u64 cr4;
+	u64 efer;
+	u64 kernel_gs_base;
+
+	/* Emulated UART and FPU capability state */
+	struct caretaker_uart uart;
+	bool save_guest_fpu;
+
+	/* Isolated host descriptors loaded while Caretaker owns the pCPU */
+	struct desc_struct gdt[GDT_ENTRIES] __aligned(16);
+	struct x86_hw_tss tss __aligned(16);
+
+	/* Upper half of Page 0 (2 KB): standalone VMX exit and IST stack */
+	u8 stack[CXP_STACK_SIZE] __aligned(CXP_STACK_SIZE);
+} __aligned(PAGE_SIZE);
+
+static_assert(offsetof(struct caretaker_x86_page, abi) == 0);
+static_assert(offsetof(struct caretaker_x86_page, abi.cb) == 0);
+static_assert(offsetof(struct caretaker_x86_page, stack) == CXP_STACK_SIZE);
+static_assert(sizeof(struct caretaker_x86_page) == PAGE_SIZE);
+
+int kvm_x86_caretaker_preserve_page(struct kvm_caretaker_arch_ser *abi,
+				    struct page *page);
+void kvm_x86_caretaker_unpreserve_pages(struct kvm_caretaker_arch_ser *abi);
+
+/* Shared page table, IDT, and GPR helpers */
+extern gate_desc caretaker_x86_idt[IDT_ENTRIES];
+#ifdef CONFIG_LIVEUPDATE_CPU
+extern phys_addr_t x86_caretaker_pgd_pa;
+#else
+#define x86_caretaker_pgd_pa 0ULL
+#endif
+
+int kvm_x86_caretaker_init_common_page(struct caretaker_x86_page *cxp,
+				       struct kvm_vcpu *vcpu,
+				       size_t full_page_size);
+
+void kvm_x86_caretaker_sync_vcpu_common(struct kvm_vcpu *vcpu);
+__caretaker_text void
+kvm_x86_caretaker_detach_serialize_common(struct caretaker_x86_page *cxp,
+					  struct kvm_vcpu_arch_ser *state);
+__caretaker_text void
+kvm_x86_caretaker_update_msr(struct kvm_vcpu_arch_ser *state,
+			     u32 msr, u64 val);
+__caretaker_text bool
+kvm_x86_caretaker_handle_exit(void *data, struct kvm_caretaker_exit *exit);
+
+void x86_preserved_iret_stub(void);
+void x86_preserved_iret_err_stub(void);
+void x86_preserved_apic_eoi_stub(void);
+__caretaker_text void kvm_x86_caretaker_arm_timer(u64 deadline_ticks);
+__caretaker_text void kvm_x86_caretaker_disarm_timer(void);
+
+/**
+ * struct kvm_x86_caretaker_runtime_ops - Preserved runtime vectors for x86 Caretaker
+ * @detach_serialize: Serialize live vendor guest state into struct kvm_vcpu_arch_ser.
+ * @common:           Common Caretaker operations table (enter_guest, decode_exit, etc.).
+ */
+struct kvm_x86_caretaker_runtime_ops {
+	void (*detach_serialize)(void *page, struct kvm_vcpu_arch_ser *state);
+	struct kvm_caretaker_ops common;
+};
+
+/**
+ * struct kvm_x86_caretaker_ops - Vendor virtualization vectors for Caretaker
+ * @name:      Vendor name identifier ("vmx" or "svm").
+ * @init:      Initialize vendor-specific Caretaker page and hardware state for vCPU.
+ * @sync_vcpu: Synchronize preserved hardware state back into @vcpu during attach.
+ * @runtime:   Preserved runtime operations table placed in __cpu_preserved_data.
+ */
+struct kvm_x86_caretaker_ops {
+	const char *name;
+	void (*init)(struct kvm_vcpu *vcpu);
+	void (*sync_vcpu)(struct kvm_vcpu *vcpu, void *vcpu_data);
+	const struct kvm_x86_caretaker_runtime_ops *runtime;
+};
+
+void kvm_x86_caretaker_register_ops(const struct kvm_x86_caretaker_ops *ops);
+void kvm_x86_caretaker_unregister_ops(const struct kvm_x86_caretaker_ops *ops);
+
+struct kvm_vcpu_ser;
+
+#ifdef CONFIG_KVM_CARETAKER
+void kvm_arch_vcpu_caretaker_init(struct kvm_vcpu *vcpu);
+void kvm_arch_vcpu_caretaker_unpreserve(struct kvm_vcpu_ser *ser);
+void kvm_arch_vcpu_caretaker_finish(struct kvm_vcpu_ser *ser);
+#else
+static inline void kvm_arch_vcpu_caretaker_init(struct kvm_vcpu *vcpu) {}
+static inline void kvm_arch_vcpu_caretaker_unpreserve(struct kvm_vcpu_ser *ser) {}
+static inline void kvm_arch_vcpu_caretaker_finish(struct kvm_vcpu_ser *ser) {}
+#endif
+#endif /* !__ASSEMBLY__ */
+
+#endif /* __ARCH_X86_KVM_CARETAKER_H */
diff --git a/arch/x86/kvm/caretaker_asm.h b/arch/x86/kvm/caretaker_asm.h
new file mode 100644
index 000000000000..8bae9bf07e16
--- /dev/null
+++ b/arch/x86/kvm/caretaker_asm.h
@@ -0,0 +1,117 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/*
+ * Copyright (c) 2026, Google LLC.
+ * Pasha Tatashin <pasha.tatashin@soleen.com>
+ *
+ * Assembly macros for x86 Caretaker host context switch and guest GPRs.
+ */
+#ifndef __ARCH_X86_KVM_CARETAKER_ASM_H
+#define __ARCH_X86_KVM_CARETAKER_ASM_H
+
+#include <linux/objtool.h>
+#include <asm/segment.h>
+#include "caretaker.h"
+#include "kvm-asm-offsets.h"
+
+/* Save host callee-saved registers to caller stack */
+.macro CARETAKER_PUSH_HOST_REGS
+	pushq %rbp
+	movq %rsp, %rbp
+	pushq %rbx
+	pushq %r12
+	pushq %r13
+	pushq %r14
+	pushq %r15
+.endm
+
+/* Restore host callee-saved registers from caller stack */
+.macro CARETAKER_POP_HOST_REGS
+	popq %r15
+	popq %r14
+	popq %r13
+	popq %r12
+	popq %rbx
+	popq %rbp
+.endm
+
+/* Restore guest GPRs (RBX..R15) from preserved page structure */
+.macro CARETAKER_RESTORE_GPRS_NO_RAX base:req
+	movq CXP_REG_RBX(\base), %rbx
+	movq CXP_REG_RCX(\base), %rcx
+	movq CXP_REG_RDX(\base), %rdx
+	movq CXP_REG_RSI(\base), %rsi
+	movq CXP_REG_RBP(\base), %rbp
+	movq CXP_REG_R8(\base),  %r8
+	movq CXP_REG_R9(\base),  %r9
+	movq CXP_REG_R10(\base), %r10
+	movq CXP_REG_R11(\base), %r11
+	movq CXP_REG_R12(\base), %r12
+	movq CXP_REG_R13(\base), %r13
+	movq CXP_REG_R14(\base), %r14
+	movq CXP_REG_R15(\base), %r15
+.endm
+
+/* Restore guest GPRs including RAX from preserved page structure */
+.macro CARETAKER_RESTORE_GPRS base:req
+	movq CXP_REG_RAX(\base), %rax
+	CARETAKER_RESTORE_GPRS_NO_RAX \base
+.endm
+
+/* Save guest GPRs (RBX..R15) to preserved page structure */
+.macro CARETAKER_SAVE_GPRS_NO_RAX base:req
+	movq %rbx, CXP_REG_RBX(\base)
+	movq %rcx, CXP_REG_RCX(\base)
+	movq %rdx, CXP_REG_RDX(\base)
+	movq %rsi, CXP_REG_RSI(\base)
+	movq %rbp, CXP_REG_RBP(\base)
+	movq %r8,  CXP_REG_R8(\base)
+	movq %r9,  CXP_REG_R9(\base)
+	movq %r10, CXP_REG_R10(\base)
+	movq %r11, CXP_REG_R11(\base)
+	movq %r12, CXP_REG_R12(\base)
+	movq %r13, CXP_REG_R13(\base)
+	movq %r14, CXP_REG_R14(\base)
+	movq %r15, CXP_REG_R15(\base)
+.endm
+
+/* Save guest GPRs including RAX to preserved page structure */
+.macro CARETAKER_SAVE_GPRS base:req
+	movq %rax, CXP_REG_RAX(\base)
+	CARETAKER_SAVE_GPRS_NO_RAX \base
+.endm
+
+/*
+ * Clear CPU microarchitectural buffers (MDS/TAA/MMIO Stale Data/RFDS) via VERW
+ * before entering guest mode, using a selector stored in .data.cpu_preserved.
+ */
+.macro CARETAKER_CLEAR_CPU_BUFFERS
+	.pushsection .data.cpu_preserved, "aw"
+	.align 2
+.Lcaretaker_verw_sel\@:
+	.word __KERNEL_DS
+	.popsection
+	verw .Lcaretaker_verw_sel\@(%rip)
+.endm
+
+/*
+ * Stuff the 32-entry hardware Return Stack Buffer (RSB) on VM-exit without
+ * referencing external thunks or unmapped per-CPU call-depth variables.
+ */
+.macro CARETAKER_FILL_RETURN_BUFFER reg:req
+	mov	$(RSB_CLEAR_LOOPS / 2), \reg
+.Lrsb_loop\@:
+	ANNOTATE_INTRA_FUNCTION_CALL
+	call	.Lrsb_1\@
+	int3
+.Lrsb_1\@:
+	ANNOTATE_INTRA_FUNCTION_CALL
+	call	.Lrsb_2\@
+	int3
+.Lrsb_2\@:
+	add	$16, %rsp
+	dec	\reg
+	jnz	.Lrsb_loop\@
+	lfence
+.endm
+
+#endif /* __ARCH_X86_KVM_CARETAKER_ASM_H */
diff --git a/arch/x86/kvm/kvm-asm-offsets.c b/arch/x86/kvm/kvm-asm-offsets.c
index 36ac61724dd7..e743ddae255b 100644
--- a/arch/x86/kvm/kvm-asm-offsets.c
+++ b/arch/x86/kvm/kvm-asm-offsets.c
@@ -9,6 +9,9 @@
 #include <linux/kbuild.h>
 #include "vmx/vmx.h"
 #include "svm/svm.h"
+#ifdef CONFIG_KVM_CARETAKER
+#include "caretaker.h"
+#endif
 
 static void __used common(void)
 {
@@ -27,4 +30,33 @@ static void __used common(void)
 		OFFSET(VMX_vcpu_arch_regs, vcpu_vmx, vcpu.arch.regs);
 		OFFSET(VMX_spec_ctrl, vcpu_vmx, spec_ctrl);
 	}
+
+#ifdef CONFIG_KVM_CARETAKER
+	/*
+	 * Offsets into the caretaker page used by the on-core guest entry and
+	 * exit paths in {vmx,svm}/caretaker_vmenter.S.  These used to be a
+	 * hand-maintained table of literals in caretaker.h.
+	 */
+	BLANK();
+	OFFSET(CXP_VMCS_PA, caretaker_x86_page, abi.vmcs_pa);
+	OFFSET(CXP_STACK_ORIG, caretaker_x86_page, stack_orig);
+	OFFSET(CXP_KERNEL_GS_BASE, caretaker_x86_page, kernel_gs_base);
+
+	BLANK();
+	OFFSET(CXP_REG_RAX, caretaker_x86_page, rax);
+	OFFSET(CXP_REG_RBX, caretaker_x86_page, rbx);
+	OFFSET(CXP_REG_RCX, caretaker_x86_page, rcx);
+	OFFSET(CXP_REG_RDX, caretaker_x86_page, rdx);
+	OFFSET(CXP_REG_RSI, caretaker_x86_page, rsi);
+	OFFSET(CXP_REG_RDI, caretaker_x86_page, rdi);
+	OFFSET(CXP_REG_RBP, caretaker_x86_page, rbp);
+	OFFSET(CXP_REG_R8, caretaker_x86_page, r8);
+	OFFSET(CXP_REG_R9, caretaker_x86_page, r9);
+	OFFSET(CXP_REG_R10, caretaker_x86_page, r10);
+	OFFSET(CXP_REG_R11, caretaker_x86_page, r11);
+	OFFSET(CXP_REG_R12, caretaker_x86_page, r12);
+	OFFSET(CXP_REG_R13, caretaker_x86_page, r13);
+	OFFSET(CXP_REG_R14, caretaker_x86_page, r14);
+	OFFSET(CXP_REG_R15, caretaker_x86_page, r15);
+#endif
 }
diff --git a/include/linux/kho/abi/kvm_x86.h b/include/linux/kho/abi/kvm_x86.h
index 316250482db5..72038ca65b0f 100644
--- a/include/linux/kho/abi/kvm_x86.h
+++ b/include/linux/kho/abi/kvm_x86.h
@@ -79,6 +79,28 @@ static_assert(offsetof(struct kvm_vcpu_arch_ser, msrs) % sizeof(u64) == 0,
 static_assert(offsetof(struct kvm_vcpu_arch_ser, xsave) % 64 == 0,
 	      "xsave must be 64-byte aligned to be XSAVE-able in place");
 
+#define KVM_X86_CARETAKER_MAX_PAGES	5
+
+/**
+ * struct kvm_caretaker_arch_ser - x86-specific Caretaker control block ABI
+ * @cb:                 Common Caretaker control block header (must be at offset 0).
+ * @apic_id:            Hardware APIC ID of the preserved core, used by the incoming
+ *                      kernel to deliver the attach-signal IPI.
+ * @nr_preserved_pages: Number of valid physical addresses in @preserved_pages_pa.
+ * @vmcs_pa:            Physical address of the VMCS the Caretaker ran the vCPU on
+ *                      (Intel VMX), used by the incoming kernel to VMCLEAR and
+ *                      flush the VMCS after the vCPU stops.
+ * @preserved_pages_pa: Physical addresses of per-vCPU hardware control pages
+ *                      preserved via kho_preserve_pages().
+ */
+struct kvm_caretaker_arch_ser {
+	struct kvm_caretaker_cb_ser cb;
+	u32 apic_id;
+	u32 nr_preserved_pages;
+	u64 vmcs_pa;
+	u64 preserved_pages_pa[KVM_X86_CARETAKER_MAX_PAGES];
+} __packed;
+
 #endif /* CONFIG_X86_64 */
 
 #endif /* _LINUX_KHO_ABI_KVM_X86_H */
-- 
2.55.0.1082.g2b9226bbc0-goog

[RFC PATCH 34/46] KVM: x86: Implement Caretaker LAPIC timer and interrupt injection

[Pasha Tatashin]

Implement standalone Caretaker LAPIC timer deadline programming and
interrupt injection helpers in arch/x86/kvm/caretaker.c.

Signed-off-by: Pasha Tatashin <pasha.tatashin@soleen.com>
---
 arch/x86/kvm/caretaker.c | 312 +++++++++++++++++++++++++++++++++++++++
 1 file changed, 312 insertions(+)
 create mode 100644 arch/x86/kvm/caretaker.c

diff --git a/arch/x86/kvm/caretaker.c b/arch/x86/kvm/caretaker.c
new file mode 100644
index 000000000000..9e0f20cbb137
--- /dev/null
+++ b/arch/x86/kvm/caretaker.c
@@ -0,0 +1,312 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Copyright (c) 2026, Google LLC.
+ * Pasha Tatashin <pasha.tatashin@soleen.com>
+ *
+ * x86 KVM Caretaker execution loop and hardware virtualization attachment.
+ */
+
+#include <linux/cpu.h>
+#include <linux/cpu_preserve.h>
+#include <linux/delay.h>
+#include <linux/kexec_handover.h>
+#include <linux/kho/abi/kvm.h>
+#include <linux/kvm_host.h>
+#include <linux/oncore.h>
+#include <linux/smp.h>
+#include <uapi/linux/serial_reg.h>
+
+#include <asm/apic.h>
+#include <asm/cpu_entry_area.h>
+#include <linux/cpu_preserve.h>
+#include <asm/desc.h>
+#include <asm/fpu/api.h>
+#include <asm/fixmap.h>
+#include <asm/irq_vectors.h>
+#include <linux/kvm_host.h>
+#include <asm/msr.h>
+#include <linux/sync_core.h>
+#include <asm/trapnr.h>
+#include <asm/virt.h>
+
+#include "caretaker.h"
+#include "cpuid.h"
+#include "lapic.h"
+#include "regs.h"
+#include "x86.h"
+
+/* Host register state saved around an on-core caretaker run. */
+struct caretaker_x86_host_state {
+	struct desc_ptr orig_idt;
+	unsigned long orig_cr2;
+	unsigned long orig_fs_base;
+	unsigned long orig_gs_base;
+	unsigned long orig_kernel_gs_base;
+	u64 orig_star;
+	u64 orig_lstar;
+	u64 orig_fmask;
+};
+
+/* Defined below their first use. */
+static void kvm_x86_caretaker_save_gprs(struct kvm_vcpu *vcpu, u64 *gprs);
+static void kvm_x86_caretaker_init_idt(gate_desc *idt);
+static void kvm_x86_caretaker_init_gdt_tss(struct desc_struct *gdt,
+					   struct x86_hw_tss *tss,
+					   unsigned long stack_top);
+
+/*
+ * A preserved page is handed over by physical address.  The SME/SEV C-bit is
+ * an encryption attribute, not part of the address, so strip it before
+ * forming a kernel virtual address.
+ *
+ * Both helpers are __always_inline because callers live in
+ * __cpu_preserved_text: an out-of-line copy would sit outside the section
+ * that survives the kexec.
+ */
+static __always_inline void *caretaker_pa_to_va(u64 pa)
+{
+	return phys_to_virt(__sme_clr(pa));
+}
+
+/* The control block is embedded in the vendor-agnostic caretaker page. */
+static __always_inline struct caretaker_x86_page *
+cxp_from_cb(struct kvm_caretaker_cb_ser *cb)
+{
+	return container_of(cb, struct caretaker_x86_page, abi.cb);
+}
+
+static const struct kvm_x86_caretaker_ops *kvm_x86_caretaker_ops __cpu_preserved_data;
+
+void kvm_x86_caretaker_register_ops(const struct kvm_x86_caretaker_ops *ops)
+{
+	WRITE_ONCE(kvm_x86_caretaker_ops, ops);
+}
+EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_x86_caretaker_register_ops);
+
+void kvm_x86_caretaker_unregister_ops(const struct kvm_x86_caretaker_ops *ops)
+{
+	if (kvm_x86_caretaker_ops == ops)
+		WRITE_ONCE(kvm_x86_caretaker_ops, NULL);
+}
+EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_x86_caretaker_unregister_ops);
+
+static bool caretaker_x86_has_tsc_deadline __cpu_preserved_data;
+static u32 caretaker_x86_lapic_timer_period __cpu_preserved_data;
+static u32 caretaker_x86_tsc_khz __cpu_preserved_data;
+gate_desc caretaker_x86_idt[IDT_ENTRIES] __caretaker_data __aligned(16);
+EXPORT_SYMBOL_FOR_KVM_INTERNAL(caretaker_x86_idt);
+static bool caretaker_x86_idt_initialized;
+
+static void kvm_x86_caretaker_init_uart(struct caretaker_uart *uart)
+{
+	if (!uart)
+		return;
+
+	uart->lcr = UART_LCR_WLEN8;
+	uart->ier = 0x00;
+	uart->mcr = UART_MCR_DTR | UART_MCR_RTS;
+	uart->scr = 0x00;
+	uart->dll = 0x01;
+	uart->dlm = 0x00;
+}
+
+int kvm_x86_caretaker_preserve_page(struct kvm_caretaker_arch_ser *abi,
+				    struct page *page)
+{
+	int ret;
+
+	if (WARN_ON_ONCE(abi->nr_preserved_pages >= KVM_X86_CARETAKER_MAX_PAGES))
+		return -ENOSPC;
+
+	ret = kho_preserve_pages(page, 1);
+	if (ret)
+		return ret;
+
+	abi->preserved_pages_pa[abi->nr_preserved_pages++] = page_to_phys(page);
+	return 0;
+}
+EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_x86_caretaker_preserve_page);
+
+void kvm_x86_caretaker_unpreserve_pages(struct kvm_caretaker_arch_ser *abi)
+{
+	u32 i;
+
+	for (i = 0; i < abi->nr_preserved_pages; i++)
+		kho_unpreserve_pages(phys_to_page(__sme_clr(abi->preserved_pages_pa[i])), 1);
+	abi->nr_preserved_pages = 0;
+}
+EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_x86_caretaker_unpreserve_pages);
+
+int kvm_x86_caretaker_init_common_page(struct caretaker_x86_page *cxp,
+				       struct kvm_vcpu *vcpu,
+				       size_t full_page_size)
+{
+	struct oncore_session *sess;
+	phys_addr_t pgd_pa;
+	u32 apic_id;
+	int pcpu;
+	int ret;
+
+	if (!cxp || !vcpu)
+		return -EINVAL;
+
+	caretaker_x86_has_tsc_deadline = boot_cpu_has(X86_FEATURE_TSC_DEADLINE_TIMER);
+	caretaker_x86_lapic_timer_period = lapic_timer_period;
+	caretaker_x86_tsc_khz = tsc_khz;
+	cpu_preserved_clean(&caretaker_x86_has_tsc_deadline);
+	cpu_preserved_clean(&caretaker_x86_lapic_timer_period);
+	cpu_preserved_clean(&caretaker_x86_tsc_khz);
+	if (!caretaker_x86_idt_initialized) {
+		kvm_x86_caretaker_init_idt(caretaker_x86_idt);
+		cpu_preserved_clean_sz(caretaker_x86_idt, sizeof(caretaker_x86_idt));
+		caretaker_x86_idt_initialized = true;
+	}
+
+	memset(cxp, 0, full_page_size);
+
+	kvm_caretaker_init_common_vcpu(&cxp->vcpu, &cxp->abi.cb, vcpu, cxp,
+				       full_page_size, NULL, cxp);
+
+	pcpu = cxp->abi.cb.pcpu_id;
+
+	apic_id = apic->cpu_present_to_apicid(pcpu);
+	if (apic_id == BAD_APICID)
+		apic_id = pcpu;
+	cxp->abi.apic_id = apic_id;
+
+	kvm_x86_caretaker_init_uart(&cxp->uart);
+
+	cxp->save_guest_fpu = boot_cpu_has(X86_FEATURE_XSAVE) &&
+			      !fpstate_is_confidential(&vcpu->arch.guest_fpu);
+
+	/* Capture guest GPRs */
+	kvm_x86_caretaker_save_gprs(vcpu, &cxp->rax);
+
+	/* Preserved CR3 from session or fallback to global cpu_preserve page table */
+	sess = oncore_job_session(vcpu->caretaker.job);
+	pgd_pa = oncore_session_get_pgd_pa(sess);
+	cxp->host_cr3 = pgd_pa ? pgd_pa : x86_caretaker_pgd_pa;
+	cxp->cr3 = kvm_read_cr3(vcpu);
+	cxp->cr0 = kvm_read_cr0(vcpu);
+	cxp->cr4 = kvm_read_cr4(vcpu);
+	cxp->efer = vcpu->arch.efer;
+
+	/* Build KHO-preserved Host GDT and TSS */
+	kvm_x86_caretaker_init_gdt_tss(cxp->gdt, &cxp->tss,
+				       (unsigned long)&cxp->stack[CXP_STACK_SIZE]);
+
+	/* Preserve in-kernel local APIC register page across kexec */
+	if (vcpu->arch.apic && vcpu->arch.apic->regs) {
+		ret = kvm_x86_caretaker_preserve_page(&cxp->abi,
+						      virt_to_page(vcpu->arch.apic->regs));
+		if (ret)
+			return ret;
+	}
+
+	return 0;
+}
+EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_x86_caretaker_init_common_page);
+
+static void kvm_x86_caretaker_save_gprs(struct kvm_vcpu *vcpu, u64 *gprs)
+{
+	gprs[0]  = kvm_register_read_raw(vcpu, VCPU_REGS_RAX);
+	gprs[1]  = kvm_register_read_raw(vcpu, VCPU_REGS_RBX);
+	gprs[2]  = kvm_register_read_raw(vcpu, VCPU_REGS_RCX);
+	gprs[3]  = kvm_register_read_raw(vcpu, VCPU_REGS_RDX);
+	gprs[4]  = kvm_register_read_raw(vcpu, VCPU_REGS_RSI);
+	gprs[5]  = kvm_register_read_raw(vcpu, VCPU_REGS_RDI);
+	gprs[6]  = kvm_register_read_raw(vcpu, VCPU_REGS_RBP);
+	gprs[7]  = kvm_register_read_raw(vcpu, VCPU_REGS_R8);
+	gprs[8]  = kvm_register_read_raw(vcpu, VCPU_REGS_R9);
+	gprs[9]  = kvm_register_read_raw(vcpu, VCPU_REGS_R10);
+	gprs[10] = kvm_register_read_raw(vcpu, VCPU_REGS_R11);
+	gprs[11] = kvm_register_read_raw(vcpu, VCPU_REGS_R12);
+	gprs[12] = kvm_register_read_raw(vcpu, VCPU_REGS_R13);
+	gprs[13] = kvm_register_read_raw(vcpu, VCPU_REGS_R14);
+	gprs[14] = kvm_register_read_raw(vcpu, VCPU_REGS_R15);
+}
+
+static void kvm_x86_caretaker_init_idt(gate_desc *idt)
+{
+	int v;
+
+	for (v = 0; v < IDT_ENTRIES; v++) {
+		bool has_err = (v == X86_TRAP_DF ||
+				(v >= X86_TRAP_TS && v <= X86_TRAP_PF) ||
+				v == X86_TRAP_AC || v == X86_TRAP_CP ||
+				v == X86_TRAP_VC || v == 30);
+		unsigned long handler = (v >= FIRST_EXTERNAL_VECTOR) ?
+			(unsigned long)&x86_preserved_apic_eoi_stub :
+			(has_err ? (unsigned long)&x86_preserved_iret_err_stub :
+				   (unsigned long)&x86_preserved_iret_stub);
+
+		pack_gate(&idt[v], GATE_INTERRUPT, handler, 0, 0, __KERNEL_CS);
+	}
+}
+
+static void kvm_x86_caretaker_init_gdt_tss(struct desc_struct *gdt,
+					   struct x86_hw_tss *tss,
+					   unsigned long stack_top)
+{
+	int k;
+
+	memcpy(gdt, get_current_gdt_ro(), sizeof(struct desc_struct) * GDT_ENTRIES);
+	memset(tss, 0, sizeof(*tss));
+	tss->sp0 = stack_top;
+	tss->io_bitmap_base = sizeof(*tss);
+	for (k = 0; k < ARRAY_SIZE(tss->ist); k++)
+		tss->ist[k] = stack_top;
+
+	caretaker_set_tss_desc(gdt, (unsigned long)tss, sizeof(struct x86_hw_tss) - 1);
+}
+
+__caretaker_text void kvm_x86_caretaker_arm_timer(u64 deadline_ticks)
+{
+	if (!deadline_ticks)
+		return;
+
+	if (caretaker_x86_has_tsc_deadline) {
+		u32 lvtt = LOCAL_TIMER_VECTOR | APIC_LVT_TIMER_TSCDEADLINE;
+
+		native_wrmsrq(APIC_BASE_MSR + (APIC_LVTT >> 4), lvtt);
+		native_wrmsrq(MSR_IA32_TSC_DEADLINE, deadline_ticks);
+	} else {
+		u64 now = rdtsc();
+		u64 delta_tsc = (deadline_ticks > now) ? (deadline_ticks - now) : 1;
+		u32 lvtt = LOCAL_TIMER_VECTOR;
+		u64 count;
+
+		if (caretaker_x86_tsc_khz != 0 && caretaker_x86_lapic_timer_period != 0) {
+			u64 period = caretaker_x86_lapic_timer_period;
+			u64 apic_khz = (period * HZ) / 1000ULL;
+
+			count = (delta_tsc * apic_khz) /
+				((u64)caretaker_x86_tsc_khz * 16ULL);
+		} else {
+			count = delta_tsc >> 4;
+		}
+		if (count == 0)
+			count = 1;
+		if (count > U32_MAX)
+			count = U32_MAX;
+
+		native_wrmsrq(APIC_BASE_MSR + (APIC_TDCR >> 4),
+			      APIC_TDR_DIV_16);
+		native_wrmsrq(APIC_BASE_MSR + (APIC_LVTT >> 4), lvtt);
+		native_wrmsrq(APIC_BASE_MSR + (APIC_TMICT >> 4), (u32)count);
+	}
+}
+EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_x86_caretaker_arm_timer);
+
+__caretaker_text void kvm_x86_caretaker_disarm_timer(void)
+{
+	if (caretaker_x86_has_tsc_deadline)
+		native_wrmsrq(MSR_IA32_TSC_DEADLINE, 0);
+	else
+		native_wrmsrq(APIC_BASE_MSR + (APIC_TMICT >> 4), 0);
+
+	native_wrmsrq(APIC_BASE_MSR + (APIC_LVTT >> 4),
+		      APIC_LVT_MASKED | LOCAL_TIMER_VECTOR);
+}
+EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_x86_caretaker_disarm_timer);
+
-- 
2.55.0.1082.g2b9226bbc0-goog

[RFC PATCH 35/46] KVM: x86: Implement Caretaker VM-exit dispatch and instruction decoders

[Pasha Tatashin]

Implement standalone Caretaker VM-exit decoding, MSR/CPUID handling,
and I/O port emulation in arch/x86/kvm/caretaker.c.

Signed-off-by: Pasha Tatashin <pasha.tatashin@soleen.com>
---
 arch/x86/kvm/caretaker.c | 593 +++++++++++++++++++++++++++++++++++++++
 1 file changed, 593 insertions(+)

diff --git a/arch/x86/kvm/caretaker.c b/arch/x86/kvm/caretaker.c
index 9e0f20cbb137..dd2b2d582b69 100644
--- a/arch/x86/kvm/caretaker.c
+++ b/arch/x86/kvm/caretaker.c
@@ -260,6 +260,599 @@ static void kvm_x86_caretaker_init_gdt_tss(struct desc_struct *gdt,
 	caretaker_set_tss_desc(gdt, (unsigned long)tss, sizeof(struct x86_hw_tss) - 1);
 }
 
+static void __cpu_preserved_text
+kvm_x86_caretaker_load_desc(struct desc_struct *gdt, size_t gdt_size,
+			    gate_desc *idt, size_t idt_size,
+			    void *tss)
+{
+	struct desc_ptr gdt_desc = {
+		.size = gdt_size - 1,
+		.address = (unsigned long)gdt,
+	};
+	struct desc_ptr idt_desc = {
+		.size = idt_size - 1,
+		.address = (unsigned long)idt,
+	};
+
+	caretaker_set_tss_desc(gdt, (unsigned long)tss, sizeof(struct x86_hw_tss) - 1);
+	load_gdt(&gdt_desc);
+	native_load_idt(&idt_desc);
+	asm volatile("ltr %w0" : : "q" ((u16)(GDT_ENTRY_TSS * 8)));
+}
+
+static __caretaker_text void
+kvm_x86_caretaker_save_host_state(struct caretaker_x86_host_state *host,
+				  struct caretaker_x86_page *cxp)
+{
+	struct cpu_preserved_stack_context *sctx;
+	u64 apic_base;
+
+	store_idt(&host->orig_idt);
+	host->orig_cr2 = native_read_cr2();
+	/*
+	 * MSR_FS_BASE is in the guest-writable passthrough set below, so it
+	 * has to be saved here or a guest WRMSR to it survives the run and
+	 * corrupts the host's FS base.
+	 */
+	host->orig_fs_base = native_rdmsrq(MSR_FS_BASE);
+	host->orig_gs_base = native_rdmsrq(MSR_GS_BASE);
+	host->orig_kernel_gs_base = native_rdmsrq(MSR_KERNEL_GS_BASE);
+	host->orig_star = native_rdmsrq(MSR_STAR);
+	host->orig_lstar = native_rdmsrq(MSR_LSTAR);
+	host->orig_fmask = native_rdmsrq(MSR_SYSCALL_MASK);
+
+	/* Ensure Local APIC is software enabled */
+	apic_base = native_rdmsrq(MSR_IA32_APICBASE);
+	if (!(apic_base & MSR_IA32_APICBASE_ENABLE))
+		native_wrmsrq(MSR_IA32_APICBASE,
+			      apic_base | MSR_IA32_APICBASE_ENABLE);
+
+	/* Switch to self-contained Caretaker GDT, IDT, and TSS before CR3 switch */
+	kvm_x86_caretaker_load_desc(cxp->gdt, sizeof(cxp->gdt),
+				    caretaker_x86_idt, sizeof(caretaker_x86_idt),
+				    &cxp->tss);
+
+	/* Switch to preserved CR3 if specified */
+	sctx = cpu_preserved_get_stack_context();
+	if (sctx && sctx->session_pgd_pa)
+		cxp->host_cr3 = sctx->session_pgd_pa;
+	else if (!cxp->host_cr3 && x86_caretaker_pgd_pa)
+		cxp->host_cr3 = x86_caretaker_pgd_pa;
+	if (cxp->host_cr3 && __read_cr3() != cxp->host_cr3)
+		write_cr3(cxp->host_cr3);
+
+	raw_local_irq_disable();
+}
+
+static __caretaker_text void
+kvm_x86_caretaker_restore_host_state(const struct caretaker_x86_host_state *host,
+				     int pcpu)
+{
+	/*
+	 * Restore unconditionally.  These are all in the guest-writable
+	 * passthrough set, so skipping the write when the saved value happens
+	 * to be zero leaves the *guest's* value live in the host MSR.
+	 */
+	native_write_cr2(host->orig_cr2);
+	native_wrmsrq(MSR_FS_BASE, host->orig_fs_base);
+	native_wrmsrq(MSR_GS_BASE, host->orig_gs_base);
+	native_wrmsrq(MSR_KERNEL_GS_BASE, host->orig_kernel_gs_base);
+	native_wrmsrq(MSR_LSTAR, host->orig_lstar);
+	native_wrmsrq(MSR_STAR, host->orig_star);
+	native_wrmsrq(MSR_SYSCALL_MASK, host->orig_fmask);
+
+	if (cpu_is_preserved(pcpu))
+		arch_cpu_preserved_load_desc();
+	else if (host->orig_idt.size)
+		native_load_idt(&host->orig_idt);
+}
+
+__caretaker_text void
+kvm_x86_caretaker_update_msr(struct kvm_vcpu_arch_ser *state,
+			     u32 msr, u64 val)
+{
+	u32 i;
+
+	if (!state)
+		return;
+
+	for (i = 0; i < state->num_msrs; i++) {
+		if (state->msrs[i].index == msr) {
+			state->msrs[i].data = val;
+			return;
+		}
+	}
+}
+EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_x86_caretaker_update_msr);
+
+static __caretaker_text bool
+kvm_x86_caretaker_read_msr(const struct kvm_vcpu_arch_ser *state,
+			   u32 msr, u64 *val)
+{
+	u32 i;
+
+	if (!state)
+		return false;
+
+	for (i = 0; i < state->num_msrs; i++) {
+		if (state->msrs[i].index == msr) {
+			*val = state->msrs[i].data;
+			return true;
+		}
+	}
+	return false;
+}
+
+/*
+ * Capture the guest FPU registers into the LUO ABI buffer.
+ *
+ * The Caretaker runs the guest with the guest's FPU state live in hardware,
+ * restoring it via XRSTOR64 at the start of each quantum and saving it via
+ * XSAVE64 at the end of each quantum and upon detach.
+ *
+ * XSAVE -- as opposed to XSAVES -- writes the standard, non-compacted layout,
+ * which is bit-for-bit the uAPI struct kvm_xsave layout that the incoming
+ * kernel feeds to fpu_copy_uabi_to_guest_fpstate().  No format conversion is
+ * needed and the ABI stays uAPI.
+ *
+ * The requested-feature bitmap comes from the XCR0 recorded at preserve time
+ * rather than from XGETBV, because XGETBV requires CR4.OSXSAVE and the guest
+ * is free to clear it.  The recorded value cannot have gone stale: the
+ * Caretaker never emulates XSETBV, so the guest cannot change XCR0 while it
+ * runs here.
+ *
+ * The destination cannot overflow: kvm_arch_vcpu_luo_preserve() refuses the
+ * preserve when guest_fpu.uabi_size exceeds sizeof(struct kvm_xsave), and
+ * RFBM is a subset of guest_supported_xcr0, which is what uabi_size sizes.
+ */
+__caretaker_text static void
+caretaker_save_guest_fpu(struct caretaker_x86_page *cxp,
+			 struct kvm_vcpu_arch_ser *state)
+{
+	union fpregs_state *xstate = (union fpregs_state *)state->xsave.region;
+	u64 rfbm = state->xcrs.xcrs[0].value | XFEATURE_MASK_FPSSE;
+
+	if (!cxp->save_guest_fpu)
+		return;
+
+	if (native_read_cr0() & X86_CR0_TS)
+		asm volatile("clts" : : : "memory");
+
+	/*
+	 * XSAVE leaves XSTATE_BV bits for components outside RFBM untouched,
+	 * so the preserve-time header would survive and advertise stale
+	 * component data.  Clear it and let XSAVE set only what it writes.
+	 */
+	cpu_preserved_memset(&xstate->xsave.header, 0,
+			     sizeof(xstate->xsave.header));
+
+	asm volatile("1: xsave64 %[buf]\n\t"
+		     "2:\n\t"
+		     _ASM_EXTABLE(1b, 2b)
+		     : [buf] "+m" (*xstate)
+		     : "a" ((u32)rfbm), "d" ((u32)(rfbm >> 32))
+		     : "memory");
+}
+
+__caretaker_text void
+kvm_x86_caretaker_detach_serialize_common(struct caretaker_x86_page *cxp,
+					  struct kvm_vcpu_arch_ser *state)
+{
+	if (!cxp || !state)
+		return;
+
+	state->regs.rax = cxp->rax;
+	state->regs.rbx = cxp->rbx;
+	state->regs.rcx = cxp->rcx;
+	state->regs.rdx = cxp->rdx;
+	state->regs.rsi = cxp->rsi;
+	state->regs.rdi = cxp->rdi;
+	state->regs.rbp = cxp->rbp;
+	state->regs.r8  = cxp->r8;
+	state->regs.r9  = cxp->r9;
+	state->regs.r10 = cxp->r10;
+	state->regs.r11 = cxp->r11;
+	state->regs.r12 = cxp->r12;
+	state->regs.r13 = cxp->r13;
+	state->regs.r14 = cxp->r14;
+	state->regs.r15 = cxp->r15;
+
+	if (cxp->last_exit_rip)
+		state->regs.rip = cxp->last_exit_rip;
+	if (cxp->last_exit_rsp)
+		state->regs.rsp = cxp->last_exit_rsp;
+	if (cxp->last_exit_rflags)
+		state->regs.rflags = cxp->last_exit_rflags;
+
+	if (cxp->cr0)
+		state->sregs.cr0 = cxp->cr0;
+	if (cxp->cr3)
+		state->sregs.cr3 = cxp->cr3;
+	if (cxp->cr4)
+		state->sregs.cr4 = cxp->cr4;
+	if (cxp->efer)
+		state->sregs.efer = cxp->efer;
+
+	state->events.exception.injected = 0;
+	state->events.interrupt.injected = 0;
+
+	caretaker_save_guest_fpu(cxp, state);
+}
+EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_x86_caretaker_detach_serialize_common);
+
+void kvm_x86_caretaker_sync_vcpu_common(struct kvm_vcpu *vcpu)
+{
+	kvm_register_mark_dirty(vcpu, VCPU_REG_CR3);
+	kvm_clear_interrupt_queue(vcpu);
+	kvm_clear_exception_queue(vcpu);
+
+	vcpu->cpu = -1;
+	kvm_make_request(KVM_REQ_LOAD_MMU_PGD, vcpu);
+	kvm_make_request(KVM_REQ_TLB_FLUSH_CURRENT, vcpu);
+	kvm_make_request(KVM_REQ_RECALC_INTERCEPTS, vcpu);
+}
+EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_x86_caretaker_sync_vcpu_common);
+
+static __caretaker_text void kvm_caretaker_emulate_cpuid(u64 *rax,
+							 u64 *rbx,
+							 u64 *rcx,
+							 u64 *rdx)
+{
+	unsigned int a = (unsigned int)*rax;
+	unsigned int b = (unsigned int)*rbx;
+	unsigned int c = (unsigned int)*rcx;
+	unsigned int d = (unsigned int)*rdx;
+
+	asm volatile("cpuid"
+		     : "=a" (a), "=b" (b), "=c" (c), "=d" (d)
+		     : "0" (a), "2" (c));
+
+	*rax = a;
+	*rbx = b;
+	*rcx = c;
+	*rdx = d;
+}
+
+static __caretaker_text bool kvm_caretaker_emulate_msr(struct caretaker_x86_page *cxp,
+						       u32 msr, bool write,
+						       u64 *rax,
+						       u64 *rdx)
+{
+	bool x2apic = msr >= APIC_BASE_MSR &&
+		      msr < APIC_BASE_MSR + X2APIC_MSR_COUNT;
+	u32 apic_id = cxp ? cxp->abi.cb.vcpu_id : 0;
+	u64 val;
+
+	if (write) {
+		val = (u32)(*rax) | ((*rdx) << 32);
+
+		/*
+		 * Guest x2APIC writes are not emulated.  ICR would send an
+		 * IPI, TMICT would arm the APIC timer, and the LVT and TPR
+		 * registers reprogram delivery.  The caretaker implements
+		 * none of that, so absorbing the write promises the guest an
+		 * interrupt that will never arrive -- it wedges rather than
+		 * stalls, and it cannot tell the difference.
+		 *
+		 * Park instead, and let the incoming kernel's full KVM apply
+		 * the write to the emulated LAPIC when it reclaims the vCPU.
+		 *
+		 * Exception: APIC_EOI (0x80b).  If a vCPU was caught inside an
+		 * interrupt handler when detached, acknowledging EOI lets it
+		 * finish the ISR and IRETQ back to user space; kvm_luo clears
+		 * APIC_ISR on retrieve anyway.
+		 */
+		if (x2apic) {
+			if (msr == APIC_BASE_MSR + (APIC_EOI >> 4))
+				return true;
+			return false;
+		}
+
+		switch (msr) {
+		case MSR_IA32_SPEC_CTRL:
+		case MSR_IA32_PRED_CMD:
+			/*
+			 * The guest is arming a speculation mitigation
+			 * (IBRS/STIBP/SSBD, or an IBPB barrier).  The
+			 * caretaker does not apply these, so acknowledging
+			 * the write would leave the guest believing it is
+			 * protected when it is not -- a security downgrade
+			 * the guest cannot observe.
+			 *
+			 * Refuse the exit instead: the vCPU parks here and
+			 * the incoming kernel's KVM applies the write for
+			 * real when it reclaims the vCPU.
+			 */
+			return false;
+		case MSR_IA32_TSC_DEADLINE:
+			/*
+			 * Record the guest's next timer deadline in preserved
+			 * arch_state so full KVM restores and arms it upon
+			 * reclaiming the vCPU, while allowing a guest caught
+			 * in its timer ISR to return to user space.
+			 */
+			if (cxp && cxp->arch_state)
+				kvm_x86_caretaker_update_msr(cxp->arch_state,
+							     MSR_IA32_TSC_DEADLINE,
+							     val);
+			return true;
+		case MSR_IA32_TSC:
+		case MSR_IA32_TSC_ADJUST:
+			/*
+			 * Discarding these silently rewrites the guest's view of
+			 * time.
+			 */
+			return false;
+		case MSR_KERNEL_GS_BASE:
+			/* Also cached, so the read side can answer without an rdmsr. */
+			if (cxp)
+				cxp->kernel_gs_base = val;
+			fallthrough;
+		case MSR_FS_BASE:
+		case MSR_GS_BASE:
+		case MSR_LSTAR:
+		case MSR_STAR:
+		case MSR_SYSCALL_MASK:
+			native_wrmsrq(msr, val);
+			return true;
+		case MSR_IA32_APICBASE:
+			/*
+			 * This used to be passed through to native_wrmsrq(),
+			 * which let the guest relocate or disable the *physical*
+			 * APIC of the CPU the caretaker is running on.  Nothing
+			 * saved or restored it around the run, so the damage
+			 * outlived the guest: on the "staying in this kernel"
+			 * path there is no INIT-SIPI-SIPI to clean up after.
+			 *
+			 * APIC base is host state here.  Refuse the write.
+			 */
+			return false;
+		}
+		return false;
+	}
+
+	if (x2apic) {
+		switch ((msr - APIC_BASE_MSR) << 4) {
+		case APIC_ID:
+			val = apic_id;
+			break;
+		case APIC_LVR:
+			val = CARETAKER_APIC_LVR;
+			break;
+		case APIC_SPIV:
+			val = APIC_SPIV_APIC_ENABLED | APIC_VECTOR_MASK;
+			break;
+		case APIC_LDR:
+			val = ((apic_id >> 4) << 16) | (1U << (apic_id & 0xf));
+			break;
+		default:
+			/*
+			 * ICR, IRR, ISR, TMCCT and friends.  Zero reads as
+			 * "nothing pending" or "timer already expired", which
+			 * the guest cannot distinguish from the truth.  The four
+			 * cases above are answered because they are static
+			 * identity registers whose values really are known.
+			 */
+			return false;
+		}
+		goto out;
+	}
+
+	switch (msr) {
+	case MSR_IA32_SPEC_CTRL:
+		/*
+		 * Returning 0 here would tell the guest its speculation
+		 * mitigations are disabled, which is both wrong and
+		 * unobservable.  Park instead; see the write path above.
+		 */
+		return false;
+	case MSR_IA32_TSC:
+		val = rdtsc();
+		break;
+	case MSR_IA32_TSC_DEADLINE:
+		if (cxp && kvm_x86_caretaker_read_msr(cxp->arch_state,
+						      MSR_IA32_TSC_DEADLINE,
+						      &val))
+			break;
+		return false;
+	case MSR_IA32_TSC_ADJUST:
+		return false;
+	case MSR_KERNEL_GS_BASE:
+		if (cxp && cxp->kernel_gs_base)
+			val = cxp->kernel_gs_base;
+		else
+			val = native_rdmsrq(MSR_KERNEL_GS_BASE);
+		break;
+	case MSR_IA32_APICBASE:
+		val = native_rdmsrq(MSR_IA32_APICBASE);
+		if (!val)
+			val = APIC_DEFAULT_PHYS_BASE | MSR_IA32_APICBASE_ENABLE;
+		if (cxp && apic_id == 0)
+			val |= MSR_IA32_APICBASE_BSP;
+		else
+			val &= ~MSR_IA32_APICBASE_BSP;
+		break;
+	case MSR_FS_BASE:
+	case MSR_GS_BASE:
+	case MSR_LSTAR:
+	case MSR_STAR:
+	case MSR_SYSCALL_MASK:
+		val = native_rdmsrq(msr);
+		break;
+	default:
+		return false;
+	}
+
+out:
+	*rax = (u32)val;
+	*rdx = (u32)(val >> 32);
+	return true;
+}
+
+static bool __cpu_preserved_text
+kvm_x86_caretaker_emulate_uart8250(struct caretaker_uart *uart,
+				   u16 port, int in, int size,
+				   unsigned long *rax)
+{
+	u8 offset;
+
+	if (port < COM1_PORT_BASE || port > COM1_PORT_END)
+		return false;
+
+	offset = port - COM1_PORT_BASE;
+
+	if (in) {
+		unsigned long val = 0;
+
+		switch (offset) {
+		case UART_RX:
+			val = (uart && (uart->lcr & UART_LCR_DLAB)) ? uart->dll : 0;
+			break;
+		case UART_IER:
+			val = (uart && (uart->lcr & UART_LCR_DLAB)) ? uart->dlm :
+				(uart ? uart->ier : 0);
+			break;
+		case UART_IIR:
+			val = UART_IIR_NO_INT;
+			break;
+		case UART_LCR:
+			val = uart ? uart->lcr : UART_LCR_WLEN8;
+			break;
+		case UART_MCR:
+			val = uart ? uart->mcr : (UART_MCR_DTR | UART_MCR_RTS);
+			break;
+		case UART_LSR:
+			val = UART_LSR_TEMT | UART_LSR_THRE;
+			break;
+		case UART_MSR:
+			val = UART_MSR_DCD | UART_MSR_DSR | UART_MSR_CTS;
+			break;
+		case UART_SCR:
+			val = uart ? uart->scr : 0;
+			break;
+		}
+
+		if (size < (int)sizeof(unsigned long)) {
+			unsigned long mask = (1UL << (size * 8)) - 1;
+			*rax = (*rax & ~mask) | (val & mask);
+		} else {
+			*rax = val;
+		}
+	} else {
+		u8 out_val = (u8)*rax;
+
+		if (uart) {
+			switch (offset) {
+			case UART_TX:
+				if (uart->lcr & UART_LCR_DLAB)
+					uart->dll = out_val;
+				break;
+			case UART_IER:
+				if (uart->lcr & UART_LCR_DLAB)
+					uart->dlm = out_val;
+				else
+					uart->ier = out_val;
+				break;
+			case UART_LCR:
+				uart->lcr = out_val;
+				break;
+			case UART_MCR:
+				uart->mcr = out_val;
+				break;
+			case UART_SCR:
+				uart->scr = out_val;
+				break;
+			}
+		}
+	}
+
+	return true;
+}
+STACK_FRAME_NON_STANDARD(kvm_x86_caretaker_emulate_uart8250);
+
+__caretaker_text bool
+kvm_x86_caretaker_handle_exit(void *data, struct kvm_caretaker_exit *exit)
+{
+	struct caretaker_x86_page *cxp = data;
+	bool handled = false;
+
+	if (exit->type == KVM_CARETAKER_EXIT_CROSS_VCPU) {
+		/*
+		 * x86 has no cross-vCPU emulation.  The decoders route
+		 * VMCALL, APIC_ACCESS, APIC_WRITE, EOI_INDUCED and
+		 * INTERRUPT_WINDOW here, and every one of them has a
+		 * guest-visible effect the Caretaker cannot produce: a
+		 * hypercall it cannot service, an APIC register write it
+		 * cannot apply, an EOI it cannot retire, an IPI it cannot
+		 * deliver to a vCPU parked on another core.
+		 *
+		 * Returning true absorbed all of it.  Worse, nothing
+		 * advanced RIP afterwards, so VMCALL re-executed forever.
+		 *
+		 * Stall instead.  The vCPU parks on the instruction and the
+		 * incoming kernel's full KVM emulates it properly.  arm64
+		 * does handle its CROSS_VCPU case (SGI delivery) and keeps
+		 * returning true.
+		 */
+		return false;
+	}
+
+	switch ((int)exit->type) {
+	case KVM_CARETAKER_EXIT_CONSOLE: {
+		unsigned long *target = exit->mmio_io.val_ptr ?
+					(unsigned long *)exit->mmio_io.val_ptr :
+					(unsigned long *)&exit->mmio_io.val;
+
+		handled = kvm_x86_caretaker_emulate_uart8250(&cxp->uart,
+							     (u16)exit->mmio_io.addr,
+							     !exit->mmio_io.is_write,
+							     exit->mmio_io.size,
+							     target);
+		break;
+	}
+	case KVM_CARETAKER_EXIT_CPUID:
+		kvm_caretaker_emulate_cpuid(&cxp->rax, &cxp->rbx, &cxp->rcx, &cxp->rdx);
+		handled = true;
+		break;
+	case KVM_CARETAKER_EXIT_MSR:
+		handled = kvm_caretaker_emulate_msr(cxp, exit->msr.msr, exit->msr.is_write,
+						    &cxp->rax, &cxp->rdx);
+		break;
+	case KVM_CARETAKER_EXIT_RDTSC: {
+		u64 tsc = rdtsc();
+
+		cxp->rax = (u32)tsc;
+		cxp->rdx = (u32)(tsc >> 32);
+		handled = true;
+		break;
+	}
+	case KVM_CARETAKER_EXIT_INSN_STEP:
+		handled = true;
+		break;
+	case KVM_CARETAKER_EXIT_ARCH:
+	default:
+		/*
+		 * Nothing above recognised this exit, so nothing emulated it.
+		 * Advancing RIP here would step over an instruction whose
+		 * architectural effect never happened (MOV to CRn, XSETBV,
+		 * INVLPG, WBINVD, RDPMC, ...), leaving the guest running on
+		 * silently wrong state with no way to detect it.
+		 *
+		 * Report the exit as unhandled instead.  The caretaker run
+		 * loop stops re-entering the guest and the vCPU stays parked
+		 * on this instruction until the incoming kernel reclaims it
+		 * and full KVM emulates the exit properly.
+		 */
+		return false;
+	}
+
+	if (handled)
+		exit->rip += exit->insn_len;
+
+	return handled;
+}
+EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_x86_caretaker_handle_exit);
+
 __caretaker_text void kvm_x86_caretaker_arm_timer(u64 deadline_ticks)
 {
 	if (!deadline_ticks)
-- 
2.55.0.1082.g2b9226bbc0-goog

[RFC PATCH 36/46] KVM: x86: Implement Caretaker run loop and LUO detach/attach lifecycle

[Pasha Tatashin]

Complete the x86 Caretaker vCPU execution loop, detach-time state
serialization, and LUO integration in arch/x86/kvm/caretaker.c.

Signed-off-by: Pasha Tatashin <pasha.tatashin@soleen.com>
---
 arch/x86/kvm/Kconfig     |   1 +
 arch/x86/kvm/Makefile    |   3 +-
 arch/x86/kvm/caretaker.c | 230 ++++++++++++++++++++++++++++++++++++++-
 arch/x86/kvm/caretaker.h |  12 +-
 arch/x86/kvm/kvm_luo.c   |  30 +++++
 5 files changed, 270 insertions(+), 6 deletions(-)

diff --git a/arch/x86/kvm/Kconfig b/arch/x86/kvm/Kconfig
index bae79fded6ff..2d1cb82ac3f1 100644
--- a/arch/x86/kvm/Kconfig
+++ b/arch/x86/kvm/Kconfig
@@ -22,6 +22,7 @@ config KVM_X86
 	select KVM_COMMON
 	select KVM_ELIDE_TLB_FLUSH_IF_YOUNG
 	select KVM_MMU_LOCKLESS_AGING
+	select HAVE_KVM_ARCH_CARETAKER
 	select HAVE_KVM_IRQCHIP
 	select HAVE_KVM_PFNCACHE
 	select HAVE_KVM_DIRTY_RING_TSO
diff --git a/arch/x86/kvm/Makefile b/arch/x86/kvm/Makefile
index 2cf0f1f2a59b..11e67c072258 100644
--- a/arch/x86/kvm/Makefile
+++ b/arch/x86/kvm/Makefile
@@ -7,7 +7,8 @@ include $(srctree)/virt/kvm/Makefile.kvm
 
 kvm-y			+= x86.o emulate.o irq.o lapic.o cpuid.o msrs.o pmu.o regs.o \
 			   mtrr.o debugfs.o mmu/mmu.o mmu/page_track.o mmu/spte.o
-kvm-$(CONFIG_LIVEUPDATE) += kvm_luo.o
+kvm-$(CONFIG_LIVEUPDATE) += kvm_luo.o mmu/kho.o
+kvm-$(CONFIG_KVM_CARETAKER) += caretaker.o
 
 kvm-$(CONFIG_X86_64) += mmu/tdp_iter.o mmu/tdp_mmu.o
 kvm-$(CONFIG_KVM_IOAPIC) += i8259.o i8254.o ioapic.o
diff --git a/arch/x86/kvm/caretaker.c b/arch/x86/kvm/caretaker.c
index dd2b2d582b69..c43913e94a9d 100644
--- a/arch/x86/kvm/caretaker.c
+++ b/arch/x86/kvm/caretaker.c
@@ -39,6 +39,7 @@
 struct caretaker_x86_host_state {
 	struct desc_ptr orig_idt;
 	unsigned long orig_cr2;
+	unsigned long orig_cr8;
 	unsigned long orig_fs_base;
 	unsigned long orig_gs_base;
 	unsigned long orig_kernel_gs_base;
@@ -53,6 +54,8 @@ static void kvm_x86_caretaker_init_idt(gate_desc *idt);
 static void kvm_x86_caretaker_init_gdt_tss(struct desc_struct *gdt,
 					   struct x86_hw_tss *tss,
 					   unsigned long stack_top);
+static enum oncore_exit_reason __cpu_preserved_text
+kvm_x86_caretaker_run_page(struct caretaker_x86_page *cxp, u64 deadline_ticks);
 
 /*
  * A preserved page is handed over by physical address.  The SME/SEV C-bit is
@@ -75,21 +78,129 @@ cxp_from_cb(struct kvm_caretaker_cb_ser *cb)
 	return container_of(cb, struct caretaker_x86_page, abi.cb);
 }
 
-static const struct kvm_x86_caretaker_ops *kvm_x86_caretaker_ops __cpu_preserved_data;
+static const struct kvm_x86_caretaker_ops *kvm_x86_caretaker_host_ops;
+static const struct kvm_x86_caretaker_runtime_ops *kvm_x86_caretaker_ops __cpu_preserved_data;
 
 void kvm_x86_caretaker_register_ops(const struct kvm_x86_caretaker_ops *ops)
 {
-	WRITE_ONCE(kvm_x86_caretaker_ops, ops);
+	WRITE_ONCE(kvm_x86_caretaker_host_ops, ops);
+	WRITE_ONCE(kvm_x86_caretaker_ops, ops ? ops->runtime : NULL);
+	cpu_preserved_clean(&kvm_x86_caretaker_ops);
 }
 EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_x86_caretaker_register_ops);
 
 void kvm_x86_caretaker_unregister_ops(const struct kvm_x86_caretaker_ops *ops)
 {
-	if (kvm_x86_caretaker_ops == ops)
+	if (kvm_x86_caretaker_host_ops == ops) {
+		WRITE_ONCE(kvm_x86_caretaker_host_ops, NULL);
 		WRITE_ONCE(kvm_x86_caretaker_ops, NULL);
+		cpu_preserved_clean(&kvm_x86_caretaker_ops);
+	}
 }
 EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_x86_caretaker_unregister_ops);
 
+enum oncore_exit_reason __cpu_preserved_text
+kvm_arch_vcpu_caretaker_run(void *data, u64 deadline_ticks)
+{
+	struct kvm_caretaker_cb_ser *cb = data;
+
+	/*
+	 * @data is always a struct kvm_caretaker_cb_ser:
+	 * kvm_caretaker_vcpu_post_preserve() installs it with
+	 * oncore_job_set_data() before activating the job.
+	 */
+	if (!cb)
+		return ONCORE_EXIT_ERROR;
+
+	return kvm_x86_caretaker_run_page(cxp_from_cb(cb), deadline_ticks);
+}
+
+static void kvm_arch_vcpu_caretaker_init(struct kvm_vcpu *vcpu)
+{
+	if (kvm_x86_caretaker_host_ops && kvm_x86_caretaker_host_ops->init)
+		kvm_x86_caretaker_host_ops->init(vcpu);
+}
+
+int kvm_arch_vcpu_caretaker_preserve(struct kvm_vcpu *vcpu,
+				     struct kvm_vcpu_ser *ser,
+				     struct kvm_vcpu_arch_ser *state, size_t size)
+{
+	struct kvm_caretaker_arch_ser *abi;
+
+	kvm_arch_vcpu_caretaker_init(vcpu);
+	if (!vcpu->caretaker.cb)
+		return -ENOMEM;
+
+	ser->cb.phys = virt_to_phys(vcpu->caretaker.cb);
+	abi = phys_to_virt(ser->cb.phys);
+	container_of(abi, struct caretaker_x86_page, abi)->arch_state = state;
+	cpu_preserved_map_buffer(state, size);
+
+	return 0;
+}
+
+static void kvm_x86_caretaker_signal_attach(struct kvm_vcpu *vcpu, u64 cb_pa)
+{
+	struct kvm_caretaker_arch_ser *abi;
+	struct kvm_caretaker_cb_ser *cb;
+	int target_pcpu;
+	u32 apic_id;
+
+	if (!cb_pa)
+		return;
+
+	abi = caretaker_pa_to_va(cb_pa);
+	cb = &abi->cb;
+	target_pcpu = cb->pcpu_id;
+
+	if (cpu_is_preserved(target_pcpu)) {
+		apic_id = apic->cpu_present_to_apicid(target_pcpu);
+		if (apic_id == BAD_APICID)
+			apic_id = cpuid_to_apicid[target_pcpu];
+		if (apic_id == BAD_APICID)
+			apic_id = abi->apic_id ? abi->apic_id : target_pcpu;
+		if (apic_id != BAD_APICID && apic_id != (u32)-1 && apic_id != 0)
+			per_cpu(x86_cpu_to_apicid, target_pcpu) = apic_id;
+	}
+
+	kvm_caretaker_wait_for_attach(cb, target_pcpu);
+	if (vcpu)
+		vcpu->cpu = -1;
+}
+
+static void kvm_x86_caretaker_attach(struct kvm_vcpu *vcpu, u64 cb_pa)
+{
+	const struct kvm_x86_caretaker_ops *ops = kvm_x86_caretaker_host_ops;
+
+	if (cb_pa) {
+		struct kvm_caretaker_arch_ser *abi = caretaker_pa_to_va(cb_pa);
+
+		vcpu_load(vcpu);
+		if (ops && ops->sync_vcpu)
+			ops->sync_vcpu(vcpu, abi);
+		vcpu_put(vcpu);
+	}
+}
+
+void kvm_arch_vcpu_luo_pre_retrieve_caretaker(struct kvm_vcpu *vcpu,
+					      struct kvm_vcpu_ser *ser)
+{
+	if (!ser || !ser->cb.phys || !(ser->flags & KVM_VCPU_LUO_FLAG_CARETAKER))
+		return;
+
+	kvm_x86_caretaker_signal_attach(vcpu, ser->cb.phys);
+}
+
+void kvm_arch_vcpu_luo_attach_caretaker(struct kvm_vcpu *vcpu,
+					struct kvm_vcpu_ser *ser)
+{
+	if (!ser || !ser->cb.phys || !(ser->flags & KVM_VCPU_LUO_FLAG_CARETAKER))
+		return;
+
+	kvm_x86_caretaker_attach(vcpu, ser->cb.phys);
+	kvm_caretaker_post_attach_vcpu(vcpu);
+}
+
 static bool caretaker_x86_has_tsc_deadline __cpu_preserved_data;
 static u32 caretaker_x86_lapic_timer_period __cpu_preserved_data;
 static u32 caretaker_x86_tsc_khz __cpu_preserved_data;
@@ -289,6 +400,8 @@ kvm_x86_caretaker_save_host_state(struct caretaker_x86_host_state *host,
 
 	store_idt(&host->orig_idt);
 	host->orig_cr2 = native_read_cr2();
+	asm volatile("mov %%cr8, %0" : "=r" (host->orig_cr8));
+	asm volatile("mov %0, %%cr8" : : "r" (0UL) : "memory");
 	/*
 	 * MSR_FS_BASE is in the guest-writable passthrough set below, so it
 	 * has to be saved here or a guest WRMSR to it survives the run and
@@ -334,6 +447,7 @@ kvm_x86_caretaker_restore_host_state(const struct caretaker_x86_host_state *host
 	 * to be zero leaves the *guest's* value live in the host MSR.
 	 */
 	native_write_cr2(host->orig_cr2);
+	asm volatile("mov %0, %%cr8" : : "r" (host->orig_cr8) : "memory");
 	native_wrmsrq(MSR_FS_BASE, host->orig_fs_base);
 	native_wrmsrq(MSR_GS_BASE, host->orig_gs_base);
 	native_wrmsrq(MSR_KERNEL_GS_BASE, host->orig_kernel_gs_base);
@@ -853,6 +967,87 @@ kvm_x86_caretaker_handle_exit(void *data, struct kvm_caretaker_exit *exit)
 }
 EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_x86_caretaker_handle_exit);
 
+__caretaker_text static void
+caretaker_restore_guest_fpu(struct caretaker_x86_page *cxp,
+			    struct kvm_vcpu_arch_ser *state)
+{
+	union fpregs_state *xstate;
+	u64 rfbm;
+
+	if (!cxp || !state || !cxp->save_guest_fpu)
+		return;
+
+	xstate = (union fpregs_state *)state->xsave.region;
+	rfbm = state->xcrs.xcrs[0].value | XFEATURE_MASK_FPSSE;
+
+	if (native_read_cr0() & X86_CR0_TS)
+		asm volatile("clts" : : : "memory");
+
+	asm volatile("1: xrstor64 %[buf]\n\t"
+		     "2:\n\t"
+		     _ASM_EXTABLE(1b, 2b)
+		     :
+		     : [buf] "m" (*xstate),
+		       "a" ((u32)rfbm), "d" ((u32)(rfbm >> 32))
+		     : "memory");
+}
+
+STACK_FRAME_NON_STANDARD(kvm_x86_caretaker_run_page);
+
+static enum oncore_exit_reason __cpu_preserved_text
+kvm_x86_caretaker_run_page(struct caretaker_x86_page *cxp, u64 deadline_ticks)
+{
+	const struct kvm_x86_caretaker_runtime_ops *ops = kvm_x86_caretaker_ops;
+	enum oncore_exit_reason reason = ONCORE_EXIT_QUANTUM_EXPIRED;
+	struct cpu_preserved_stack_context *sctx;
+	struct caretaker_x86_host_state host_state;
+	int pcpu;
+
+	if (!cxp || !ops)
+		return ONCORE_EXIT_ERROR;
+
+	sctx = cpu_preserved_get_stack_context();
+	if (sctx && sctx->cpu >= 0 && sctx->cpu < CONFIG_NR_CPUS)
+		pcpu = sctx->cpu;
+	else
+		pcpu = cxp->abi.cb.pcpu_id;
+	cxp->abi.cb.pcpu_id = pcpu;
+
+	if (cmpxchg(&cxp->abi.cb.state, KVM_CARETAKER_PAUSED,
+		    KVM_CARETAKER_RUNNING) != KVM_CARETAKER_PAUSED ||
+	    kvm_caretaker_should_exit(&cxp->vcpu)) {
+		smp_store_release(&cxp->abi.cb.state, KVM_CARETAKER_STOPPED);
+		return ONCORE_EXIT_ATTACH_SIGNALED;
+	}
+
+	/* Save host context, switch to Caretaker descriptors and CR3 */
+	kvm_x86_caretaker_save_host_state(&host_state, cxp);
+
+	if (cxp->arch_state)
+		caretaker_restore_guest_fpu(cxp, cxp->arch_state);
+
+	cxp->vcpu.ops = &ops->common;
+
+	reason = kvm_caretaker_vcpu_run(&cxp->vcpu, deadline_ticks);
+
+	iret_to_self();
+
+	if (ops->detach_serialize && cxp->arch_state)
+		ops->detach_serialize(cxp, cxp->arch_state);
+
+	kvm_x86_caretaker_restore_host_state(&host_state, pcpu);
+
+	if (reason == ONCORE_EXIT_ATTACH_SIGNALED ||
+	    kvm_caretaker_should_exit(&cxp->vcpu) ||
+	    cmpxchg(&cxp->abi.cb.state, KVM_CARETAKER_RUNNING,
+		    KVM_CARETAKER_PAUSED) != KVM_CARETAKER_RUNNING) {
+		reason = ONCORE_EXIT_ATTACH_SIGNALED;
+		smp_store_release(&cxp->abi.cb.state, KVM_CARETAKER_STOPPED);
+	}
+
+	return reason;
+}
+
 __caretaker_text void kvm_x86_caretaker_arm_timer(u64 deadline_ticks)
 {
 	if (!deadline_ticks)
@@ -903,3 +1098,32 @@ __caretaker_text void kvm_x86_caretaker_disarm_timer(void)
 }
 EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_x86_caretaker_disarm_timer);
 
+void kvm_arch_vcpu_caretaker_unpreserve(struct kvm_vcpu_ser *ser)
+{
+	if (ser->cb.phys) {
+		struct kvm_caretaker_arch_ser *abi = caretaker_pa_to_va(ser->cb.phys);
+
+		kvm_x86_caretaker_unpreserve_pages(abi);
+		kho_unpreserve_free(abi);
+		ser->cb.phys = 0;
+	}
+}
+
+void kvm_arch_vcpu_caretaker_finish(struct kvm_vcpu_ser *ser)
+{
+	if (ser->cb.phys) {
+		struct kvm_caretaker_arch_ser *abi = caretaker_pa_to_va(ser->cb.phys);
+		u32 i;
+
+		for (i = 0; i < abi->nr_preserved_pages; i++) {
+			phys_addr_t pa = __sme_clr(abi->preserved_pages_pa[i]);
+			struct page *page = kho_restore_pages(pa, 1);
+
+			if (page)
+				__free_pages(page, 0);
+		}
+		abi->nr_preserved_pages = 0;
+		kho_restore_free(abi);
+		ser->cb.phys = 0;
+	}
+}
diff --git a/arch/x86/kvm/caretaker.h b/arch/x86/kvm/caretaker.h
index 9dd8815d414b..8c0fc336be6b 100644
--- a/arch/x86/kvm/caretaker.h
+++ b/arch/x86/kvm/caretaker.h
@@ -217,11 +217,19 @@ void kvm_x86_caretaker_unregister_ops(const struct kvm_x86_caretaker_ops *ops);
 struct kvm_vcpu_ser;
 
 #ifdef CONFIG_KVM_CARETAKER
-void kvm_arch_vcpu_caretaker_init(struct kvm_vcpu *vcpu);
+int kvm_arch_vcpu_caretaker_preserve(struct kvm_vcpu *vcpu,
+				     struct kvm_vcpu_ser *ser,
+				     struct kvm_vcpu_arch_ser *state, size_t size);
 void kvm_arch_vcpu_caretaker_unpreserve(struct kvm_vcpu_ser *ser);
 void kvm_arch_vcpu_caretaker_finish(struct kvm_vcpu_ser *ser);
 #else
-static inline void kvm_arch_vcpu_caretaker_init(struct kvm_vcpu *vcpu) {}
+static inline int kvm_arch_vcpu_caretaker_preserve(struct kvm_vcpu *vcpu,
+						   struct kvm_vcpu_ser *ser,
+						   struct kvm_vcpu_arch_ser *state,
+						   size_t size)
+{
+	return 0;
+}
 static inline void kvm_arch_vcpu_caretaker_unpreserve(struct kvm_vcpu_ser *ser) {}
 static inline void kvm_arch_vcpu_caretaker_finish(struct kvm_vcpu_ser *ser) {}
 #endif
diff --git a/arch/x86/kvm/kvm_luo.c b/arch/x86/kvm/kvm_luo.c
index 899b193812bb..ff6acf3acb52 100644
--- a/arch/x86/kvm/kvm_luo.c
+++ b/arch/x86/kvm/kvm_luo.c
@@ -20,9 +20,11 @@
 #include <linux/mem_encrypt.h>
 #include <asm/virt.h>
 
+#include "caretaker.h"
 #include "cpuid.h"
 #include "fpu.h"
 #include "lapic.h"
+#include "mmu.h"
 #include "msrs.h"
 #include "pmu.h"
 #include "regs.h"
@@ -30,7 +32,23 @@
 
 int kvm_arch_vm_luo_preserve(struct kvm *kvm, struct kvm_luo_ser *ser)
 {
+	int ret;
+
 	ser->type = kvm->arch.vm_type;
+
+	/*
+	 * Shadow/TDP page tables are a VM-wide resource: an orphaned vCPU keeps
+	 * running the guest out of them while the VM is detached, so they must
+	 * survive the kexec.  Preserve them once here rather than once per vCPU
+	 * from the caretaker init hook -- the walk is O(size of the guest's page
+	 * tables) and holds mmu_lock for write, so repeating it per vCPU is both
+	 * redundant and a scalability problem on large guests.
+	 */
+	ret = kvm_mmu_preserve_kho(kvm);
+	if (ret)
+		return ret;
+
+	KHOSER_STORE_PTR(ser->kho_folios, kvm->kho_folios);
 	return 0;
 }
 EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_arch_vm_luo_preserve);
@@ -185,6 +203,16 @@ int kvm_arch_vcpu_luo_preserve(struct kvm_vcpu *vcpu, struct kvm_vcpu_ser *ser)
 	vcpu_put(vcpu);
 
 	KHOSER_STORE_PTR(ser->arch_state, state);
+
+	if (ser->flags & KVM_VCPU_LUO_FLAG_CARETAKER) {
+		int err = kvm_arch_vcpu_caretaker_preserve(vcpu, ser, state, size);
+
+		if (err) {
+			kho_unpreserve_free(state);
+			return err;
+		}
+	}
+
 	return 0;
 }
 EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_arch_vcpu_luo_preserve);
@@ -315,6 +343,7 @@ EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_arch_vcpu_luo_retrieve);
 
 void kvm_arch_vcpu_luo_unpreserve(struct kvm_vcpu_ser *ser)
 {
+	kvm_arch_vcpu_caretaker_unpreserve(ser);
 	if (ser->arch_state.phys) {
 		struct kvm_vcpu_arch_ser *state =
 			phys_to_virt(__sme_clr(ser->arch_state.phys));
@@ -327,6 +356,7 @@ EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_arch_vcpu_luo_unpreserve);
 
 void kvm_arch_vcpu_luo_finish(struct kvm_vcpu_ser *ser)
 {
+	kvm_arch_vcpu_caretaker_finish(ser);
 	if (ser->arch_state.phys) {
 		struct kvm_vcpu_arch_ser *state =
 			phys_to_virt(__sme_clr(ser->arch_state.phys));
-- 
2.55.0.1082.g2b9226bbc0-goog

[RFC PATCH 37/46] KVM: VMX: Add Caretaker VMX assembly guest entry/exit routine and helpers

[Pasha Tatashin]

Add arch/x86/kvm/vmx/caretaker_vmenter.S and caretaker.h for standalone
Intel VMX VMLAUNCH/VMRESUME world switch on preserved physical cores.

Signed-off-by: Pasha Tatashin <pasha.tatashin@soleen.com>
---
 arch/x86/kvm/msrs.c                  |   2 +
 arch/x86/kvm/vmx/caretaker.h         |  78 +++++++++++++++++++
 arch/x86/kvm/vmx/caretaker_vmenter.S | 112 +++++++++++++++++++++++++++
 3 files changed, 192 insertions(+)
 create mode 100644 arch/x86/kvm/vmx/caretaker.h
 create mode 100644 arch/x86/kvm/vmx/caretaker_vmenter.S

diff --git a/arch/x86/kvm/msrs.c b/arch/x86/kvm/msrs.c
index ae849e0b7d14..536c8d1959dc 100644
--- a/arch/x86/kvm/msrs.c
+++ b/arch/x86/kvm/msrs.c
@@ -882,11 +882,13 @@ int kvm_msr_write(struct kvm_vcpu *vcpu, u32 index, u64 data)
 {
 	return __kvm_set_msr(vcpu, index, data, true);
 }
+EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_msr_write);
 
 int kvm_msr_read(struct kvm_vcpu *vcpu, u32 index, u64 *data)
 {
 	return __kvm_get_msr(vcpu, index, data, true);
 }
+EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_msr_read);
 
 int __kvm_emulate_msr_read(struct kvm_vcpu *vcpu, u32 index, u64 *data)
 {
diff --git a/arch/x86/kvm/vmx/caretaker.h b/arch/x86/kvm/vmx/caretaker.h
new file mode 100644
index 000000000000..d4272e202e73
--- /dev/null
+++ b/arch/x86/kvm/vmx/caretaker.h
@@ -0,0 +1,78 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/*
+ * Copyright (c) 2026, Google LLC.
+ * Pasha Tatashin <pasha.tatashin@soleen.com>
+ */
+#ifndef __VMX_CARETAKER_H
+#define __VMX_CARETAKER_H
+
+#define VMCS_VM_INSTRUCTION_ERROR	0x4400
+
+#ifndef __ASSEMBLY__
+#include <linux/types.h>
+#include <linux/kvm_host.h>
+#include <linux/oncore.h>
+#include <asm/vmx.h>
+#endif
+
+#include "../caretaker.h"
+
+#ifndef __ASSEMBLY__
+#include <asm/desc.h>
+#include <linux/processor.h>
+
+/* Length of instructions */
+#define PAUSE_INSN_LEN			2
+#define VMCALL_INSN_LEN			3
+
+/* Exit qualification bitfields for I/O instructions (Intel SDM Vol 3C) */
+#define VMX_IO_SIZE_MASK		0x7
+#define VMX_IO_DIRECTION_BIT		BIT(3)
+#define VMX_IO_PORT_SHIFT		16
+
+/* Default VMX preemption timer shift (counts down every 2^5 TSC ticks) */
+#define VMX_PREEMPTION_TIMER_SHIFT	5
+
+struct caretaker_vmx_page {
+	struct caretaker_x86_page common;
+	/* Guest syscall state not automatically switched by VMCS */
+	u64 star;
+	u64 lstar;
+	u64 fmask;
+	u64 vmxon_pa;
+	u32 timer_shift;
+	u32 ple_supported;
+	u8 vmxon_area[PAGE_SIZE] __aligned(PAGE_SIZE);
+} __aligned(PAGE_SIZE);
+
+static inline unsigned long vmx_vmread(unsigned long field)
+{
+	unsigned long val;
+
+	asm volatile("vmread %1, %0" : "=r" (val) : "r" (field) : "cc");
+	return val;
+}
+
+static inline void vmx_vmwrite(unsigned long field, unsigned long val)
+{
+	asm volatile("vmwrite %1, %0" : : "r" (field), "r" (val) : "cc");
+}
+
+#ifdef CONFIG_KVM_CARETAKER
+int vmx_caretaker_enter(void *page);
+void vmx_caretaker_exit_handler(void);
+void vmx_caretaker_register(void);
+void vmx_caretaker_unregister(void);
+void vmx_caretaker_decode_exit(void *page,
+			       struct kvm_caretaker_exit *exit);
+void vmx_caretaker_init_host_vmcs(struct caretaker_vmx_page *cvp);
+void vmx_caretaker_init(struct kvm_vcpu *vcpu);
+#else
+static inline void vmx_caretaker_register(void) {}
+static inline void vmx_caretaker_unregister(void) {}
+static inline void vmx_caretaker_init(struct kvm_vcpu *vcpu) {}
+#endif
+
+#endif /* !__ASSEMBLY__ */
+
+#endif /* __VMX_CARETAKER_H */
diff --git a/arch/x86/kvm/vmx/caretaker_vmenter.S b/arch/x86/kvm/vmx/caretaker_vmenter.S
new file mode 100644
index 000000000000..ff135cba854d
--- /dev/null
+++ b/arch/x86/kvm/vmx/caretaker_vmenter.S
@@ -0,0 +1,112 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright (c) 2026, Google LLC.
+ * Pasha Tatashin <pasha.tatashin@soleen.com>
+ */
+#include <linux/linkage.h>
+#include <linux/cfi_types.h>
+#include <linux/objtool.h>
+#include <asm/asm.h>
+#include <asm/bitsperlong.h>
+#include <asm/kvm_vcpu_regs.h>
+#include <asm/nospec-branch.h>
+#include <asm/msr-index.h>
+#include <asm/page_types.h>
+#include "caretaker.h"
+#include "../caretaker_asm.h"
+
+.section .text.cpu_preserved, "ax"
+
+/**
+ * vmx_caretaker_enter - Low-level transition to VMX guest mode and back
+ * @cvp: Pointer to struct caretaker_vmx_page (%rdi)
+ */
+SYM_TYPED_FUNC_START(vmx_caretaker_enter)
+	ENDBR
+	CARETAKER_PUSH_HOST_REGS
+	movq %rsp, CXP_STACK_ORIG(%rdi)
+
+	/* Switch to private preserved stack */
+	movq CXP_STACK_TOP(%rdi), %rsp
+
+	/* Activate VMCS on this pCPU */
+	vmptrld CXP_VMCS_PA(%rdi)
+
+	/* Restore guest MSR_KERNEL_GS_BASE if valid/canonical */
+	movq CXP_KERNEL_GS_BASE(%rdi), %rax
+	movq %rax, %rdx
+	sarq $56, %rdx
+	incq %rdx
+	cmpq $1, %rdx
+	ja 2f
+	movl $MSR_KERNEL_GS_BASE, %ecx
+	movq CXP_KERNEL_GS_BASE(%rdi), %rax
+	movq %rax, %rdx
+	shrq $32, %rdx
+1:	wrmsr
+2:
+	_ASM_EXTABLE(1b, 2b)
+
+	/* Restore guest GPRs */
+	CARETAKER_RESTORE_GPRS %rdi
+
+	pushq %rdi
+	movq CXP_REG_RDI(%rdi), %rdi
+
+	/* Clear CPU microarchitectural buffers before guest entry */
+	CARETAKER_CLEAR_CPU_BUFFERS
+
+	/* Launch or resume */
+	vmresume
+	vmlaunch
+
+	/* If both fail, record error and return */
+	popq %rdi
+	mov $VMCS_VM_INSTRUCTION_ERROR, %eax
+	vmread %rax, %rax
+	movq %rax, CXP_LAST_EXIT_CODE(%rdi)
+	btsq $31, %rax
+	jmp .Lvmx_ret
+SYM_FUNC_END(vmx_caretaker_enter)
+
+SYM_FUNC_START(vmx_caretaker_exit_handler)
+	ENDBR
+	/*
+	 * Hardware jumps here on VM-Exit with:
+	 * RSP = HOST_RSP (CXP_STACK_TOP)
+	 * CR3 = HOST_CR3
+	 */
+	pushq %rdi
+
+	/* Recompute CVP pointer from RSP: CVP = (RSP & PAGE_MASK) - CXP_STACK_OFFSET */
+	movq %rsp, %rdi
+	andq $PAGE_MASK, %rdi
+	subq $CXP_STACK_OFFSET, %rdi
+
+	/* Save guest GPRs into CVP */
+	CARETAKER_SAVE_GPRS %rdi
+	popq CXP_REG_RDI(%rdi)
+
+	/* Stuff Return Stack Buffer to prevent guest RSB poisoning */
+	CARETAKER_FILL_RETURN_BUFFER %rax
+
+	/* Read guest MSR_KERNEL_GS_BASE */
+	movl $MSR_KERNEL_GS_BASE, %ecx
+	rdmsr
+	shlq $32, %rdx
+	orq %rax, %rdx
+	movq %rdx, CXP_KERNEL_GS_BASE(%rdi)
+
+	xorq %rax, %rax
+	jmp .Lvmx_ret
+
+.Lvmx_ret:
+	/* Restore caller stack */
+	movq CXP_STACK_ORIG(%rdi), %rsp
+
+	/* Restore host callee-saved registers */
+	CARETAKER_POP_HOST_REGS
+	ret
+SYM_FUNC_END(vmx_caretaker_exit_handler)
+STACK_FRAME_NON_STANDARD(vmx_caretaker_enter)
+STACK_FRAME_NON_STANDARD(vmx_caretaker_exit_handler)
-- 
2.55.0.1082.g2b9226bbc0-goog

[RFC PATCH 38/46] KVM: VMX: Implement Caretaker VMX VMCS lifecycle and exit dispatch

[Pasha Tatashin]

Implement Intel VMX Caretaker VMCS setup, exit decoding, and guest
entry/exit loop in arch/x86/kvm/vmx/caretaker.c.

Signed-off-by: Pasha Tatashin <pasha.tatashin@soleen.com>
---
 arch/x86/kvm/vmx/caretaker.c         | 340 +++++++++++++++++++++++++++
 arch/x86/kvm/vmx/caretaker_vmenter.S |  12 +-
 arch/x86/kvm/vmx/vmx.c               |   2 +-
 arch/x86/kvm/vmx/vmx.h               |   1 +
 4 files changed, 347 insertions(+), 8 deletions(-)
 create mode 100644 arch/x86/kvm/vmx/caretaker.c

diff --git a/arch/x86/kvm/vmx/caretaker.c b/arch/x86/kvm/vmx/caretaker.c
new file mode 100644
index 000000000000..47b4743650e5
--- /dev/null
+++ b/arch/x86/kvm/vmx/caretaker.c
@@ -0,0 +1,340 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * Intel VMX Caretaker Standalone Execution Engine
+ *
+ * Copyright (c) 2026, Google LLC.
+ * Pasha Tatashin <pasha.tatashin@soleen.com>
+ *
+ * Runs the Intel VMX guest execution loop in an isolated, KHO-preserved memory
+ * page that remains alive and executing across kexec relocation and
+ * kernel handover.
+ */
+
+#include <linux/cleanup.h>
+#include <linux/cpu_preserve.h>
+#include <linux/delay.h>
+#include <linux/kernel.h>
+#include <linux/kexec.h>
+#include <linux/kexec_handover.h>
+#include <linux/kvm_host.h>
+#include <linux/objtool.h>
+#include <linux/oncore.h>
+
+#include <asm/apic.h>
+#include <asm/cpu_entry_area.h>
+#include <asm/desc.h>
+#include <asm/fixmap.h>
+#include <linux/pgtable.h>
+#include <linux/processor.h>
+#include <asm/segment.h>
+#include <asm/set_memory.h>
+#include <asm/virt.h>
+#include <asm/vmx.h>
+
+#include "../caretaker.h"
+#include "caretaker.h"
+#include "vmx.h"
+#include "vmx_ops.h"
+#include "x86.h"
+#include "x86_ops.h"
+
+static int vmx_caretaker_init_page(struct caretaker_vmx_page *cvp,
+				   struct kvm_vcpu *vcpu);
+STACK_FRAME_NON_STANDARD(vmx_caretaker_init_page);
+
+static int vmx_caretaker_init_page(struct caretaker_vmx_page *cvp,
+				   struct kvm_vcpu *vcpu)
+{
+	struct vcpu_vmx *vmx = to_vmx(vcpu);
+	u64 basic_msr, misc_msr;
+	int ret;
+
+	if (!vmx->vmcs01.vmcs)
+		return -EINVAL;
+
+	ret = kvm_x86_caretaker_init_common_page(&cvp->common, vcpu, sizeof(*cvp));
+	if (ret)
+		return ret;
+	cvp->common.abi.vmcs_pa = virt_to_phys(vmx->vmcs01.vmcs);
+	cvp->vmxon_pa = virt_to_phys(cvp->vmxon_area);
+
+	memset(cvp->vmxon_area, 0, PAGE_SIZE);
+	basic_msr = native_rdmsrq(MSR_IA32_VMX_BASIC);
+	*(u32 *)cvp->vmxon_area = vmx_basic_vmcs_revision_id(basic_msr);
+
+	if (!rdmsrq_safe(MSR_IA32_VMX_MISC, &misc_msr))
+		cvp->timer_shift = vmx_misc_preemption_timer_rate(misc_msr);
+	else
+		cvp->timer_shift = VMX_PREEMPTION_TIMER_SHIFT;
+
+	cvp->ple_supported = cpu_has_vmx_ple();
+
+	vcpu_load(vcpu);
+
+	cvp->common.kernel_gs_base = vmx->msr_guest_kernel_gs_base;
+	kvm_msr_read(vcpu, MSR_STAR, &cvp->star);
+	kvm_msr_read(vcpu, MSR_LSTAR, &cvp->lstar);
+	kvm_msr_read(vcpu, MSR_SYSCALL_MASK, &cvp->fmask);
+
+	/* Save current guest control registers from KVM */
+	cvp->common.cr0 = kvm_read_cr0(vcpu);
+	cvp->common.cr3 = kvm_read_cr3(vcpu);
+	cvp->common.cr4 = kvm_read_cr4(vcpu);
+	cvp->common.efer = vcpu->arch.efer;
+
+	cvp->common.last_exit_rip = kvm_rip_read(vcpu);
+	cvp->common.last_exit_rsp = kvm_rsp_read(vcpu);
+	cvp->common.last_exit_rflags = kvm_get_rflags(vcpu);
+
+	if (kvm_host.efer & EFER_NX)
+		cvp->common.efer |= EFER_NX;
+
+	vcpu_put(vcpu);
+
+	if (vmx->loaded_vmcs)
+		loaded_vmcs_clear(vmx->loaded_vmcs);
+
+	return 0;
+}
+
+void vmx_caretaker_init(struct kvm_vcpu *vcpu)
+{
+	struct vcpu_vmx *vmx = to_vmx(vcpu);
+	struct caretaker_vmx_page *cvp;
+	struct kvm_caretaker_arch_ser *abi;
+
+	cvp = kho_alloc_preserve(sizeof(*cvp));
+	if (IS_ERR(cvp)) {
+		pr_err("caretaker vmx: failed to allocate preserved page\n");
+		return;
+	}
+
+	if (vmx_caretaker_init_page(cvp, vcpu))
+		goto err_free;
+
+	abi = &cvp->common.abi;
+	if (vmx->vmcs01.vmcs &&
+	    kvm_x86_caretaker_preserve_page(abi, virt_to_page(vmx->vmcs01.vmcs)))
+		goto err_free;
+	if (vmx->vmcs01.msr_bitmap &&
+	    kvm_x86_caretaker_preserve_page(abi, virt_to_page(vmx->vmcs01.msr_bitmap)))
+		goto err_free;
+	if (vmx->pml_pg &&
+	    kvm_x86_caretaker_preserve_page(abi, vmx->pml_pg))
+		goto err_free;
+	if (vmx->ve_info &&
+	    kvm_x86_caretaker_preserve_page(abi, virt_to_page(vmx->ve_info)))
+		goto err_free;
+
+	return;
+
+err_free:
+	kvm_x86_caretaker_unpreserve_pages(&cvp->common.abi);
+	vcpu->caretaker.cb = NULL;
+	kho_unpreserve_free(cvp);
+}
+
+static __cpu_preserved_text void vmx_caretaker_disarm_timer(void *page)
+{
+	u32 pin = (u32)vmx_vmread(PIN_BASED_VM_EXEC_CONTROL);
+
+	if (pin & PIN_BASED_VMX_PREEMPTION_TIMER) {
+		vmx_vmwrite(PIN_BASED_VM_EXEC_CONTROL,
+			    pin & ~PIN_BASED_VMX_PREEMPTION_TIMER);
+	}
+}
+
+static inline unsigned long vmx_caretaker_read_cr0(void)
+{
+	unsigned long mask = vmx_vmread(CR0_GUEST_HOST_MASK);
+
+	return (vmx_vmread(CR0_READ_SHADOW) & mask) |
+	       (vmx_vmread(GUEST_CR0) & ~mask);
+}
+
+static inline unsigned long vmx_caretaker_read_cr4(void)
+{
+	unsigned long mask = vmx_vmread(CR4_GUEST_HOST_MASK);
+
+	return (vmx_vmread(CR4_READ_SHADOW) & mask) |
+	       (vmx_vmread(GUEST_CR4) & ~mask);
+}
+
+static inline u64 vmx_caretaker_read_efer(void)
+{
+	u64 efer = vmx_vmread(GUEST_IA32_EFER);
+
+	if (!efer)
+		efer = native_rdmsrq(MSR_EFER);
+	if (vmx_vmread(VM_ENTRY_CONTROLS) & VM_ENTRY_IA32E_MODE)
+		efer |= EFER_LMA | EFER_LME;
+	return efer;
+}
+
+void __cpu_preserved_text
+vmx_caretaker_decode_exit(void *page,
+			  struct kvm_caretaker_exit *exit)
+{
+	struct caretaker_vmx_page *cvp = page;
+	u32 insn_len = (u32)vmx_vmread(VM_EXIT_INSTRUCTION_LEN);
+	u16 exit_reason = (u16)vmx_vmread(VM_EXIT_REASON);
+	u64 qual = vmx_vmread(EXIT_QUALIFICATION);
+	u64 rip = vmx_vmread(GUEST_RIP);
+
+	cvp->common.last_exit_rip = rip;
+	cvp->common.last_exit_rsp = vmx_vmread(GUEST_RSP);
+	cvp->common.last_exit_rflags = vmx_vmread(GUEST_RFLAGS);
+	cvp->common.cr3 = vmx_vmread(GUEST_CR3);
+	cvp->common.cr0 = vmx_caretaker_read_cr0();
+	cvp->common.cr4 = vmx_caretaker_read_cr4();
+
+	exit->rip = rip;
+	exit->insn_len = insn_len;
+	exit->raw_reason = exit_reason;
+	exit->type = KVM_CARETAKER_EXIT_ARCH;
+
+	switch (exit_reason) {
+	case EXIT_REASON_IO_INSTRUCTION: {
+		u16 port = (u16)(qual >> VMX_IO_PORT_SHIFT);
+
+		if (port >= COM1_PORT_BASE && port <= COM1_PORT_END) {
+			exit->type = KVM_CARETAKER_EXIT_CONSOLE;
+			exit->mmio_io.addr = port;
+			exit->mmio_io.is_write = !(qual & VMX_IO_DIRECTION_BIT);
+			exit->mmio_io.size = (u8)((qual & VMX_IO_SIZE_MASK) + 1);
+			exit->mmio_io.is_mmio = false;
+			exit->mmio_io.val_ptr = &cvp->common.rax;
+		}
+		break;
+	}
+	case EXIT_REASON_HLT:
+		exit->type = KVM_CARETAKER_EXIT_IDLE;
+		break;
+	case EXIT_REASON_PAUSE_INSTRUCTION:
+		exit->type = KVM_CARETAKER_EXIT_IDLE;
+		exit->insn_len = insn_len ? insn_len : PAUSE_INSN_LEN;
+		break;
+	case EXIT_REASON_CPUID:
+		exit->type = KVM_CARETAKER_EXIT_CPUID;
+		break;
+	case EXIT_REASON_VMCALL:
+		/*
+		 * Do not write a return value.  The Caretaker cannot run the
+		 * hypercall, and reporting success for something like
+		 * KVM_HC_SEND_IPI or a PV TLB flush is worse than not
+		 * answering at all.  CROSS_VCPU now stalls, so the guest
+		 * stays parked on the VMCALL with RAX untouched until the
+		 * incoming kernel services it.
+		 */
+		exit->type = KVM_CARETAKER_EXIT_CROSS_VCPU;
+		exit->insn_len = insn_len ? insn_len : VMCALL_INSN_LEN;
+		break;
+	case EXIT_REASON_MSR_READ:
+		exit->type = KVM_CARETAKER_EXIT_MSR;
+		exit->msr.msr = (u32)cvp->common.rcx;
+		exit->msr.is_write = false;
+		break;
+	case EXIT_REASON_MSR_WRITE:
+		exit->type = KVM_CARETAKER_EXIT_MSR;
+		exit->msr.msr = (u32)cvp->common.rcx;
+		exit->msr.is_write = true;
+		break;
+	case EXIT_REASON_RDTSC:
+		exit->type = KVM_CARETAKER_EXIT_RDTSC;
+		break;
+	case EXIT_REASON_EPT_VIOLATION:
+		exit->type = KVM_CARETAKER_EXIT_UNHANDLED;
+		break;
+	case EXIT_REASON_PREEMPTION_TIMER:
+		exit->type = KVM_CARETAKER_EXIT_PREEMPT_TIMER;
+		exit->insn_len = 0;
+		vmx_caretaker_disarm_timer(cvp);
+		break;
+	case EXIT_REASON_EOI_INDUCED:
+	case EXIT_REASON_APIC_WRITE:
+	case EXIT_REASON_APIC_ACCESS:
+	case EXIT_REASON_INTERRUPT_WINDOW:
+		exit->type = KVM_CARETAKER_EXIT_CROSS_VCPU;
+		exit->insn_len = 0;
+		break;
+	case EXIT_REASON_EXTERNAL_INTERRUPT:
+	case EXIT_REASON_EXCEPTION_NMI:
+	case EXIT_REASON_INIT_SIGNAL:
+	case EXIT_REASON_SIPI_SIGNAL:
+		exit->type = KVM_CARETAKER_EXIT_PREEMPT_TIMER;
+		exit->insn_len = 0;
+		break;
+	default:
+		break;
+	}
+}
+
+void __cpu_preserved_text
+vmx_caretaker_init_host_vmcs(struct caretaker_vmx_page *cvp)
+{
+	u64 fs_base = 0, gs_base = 0;
+	unsigned long pin, cpu_ctl;
+
+	phys_addr_t host_cr3 = cvp->common.host_cr3;
+
+	/* Configure Host Controls */
+	vmx_vmwrite(HOST_CR0, read_cr0());
+	vmx_vmwrite(HOST_CR4, __read_cr4());
+	if (!host_cr3) {
+		struct cpu_preserved_stack_context *sctx = cpu_preserved_get_stack_context();
+
+		if (sctx && sctx->session_pgd_pa)
+			host_cr3 = sctx->session_pgd_pa;
+		else
+			host_cr3 = x86_caretaker_pgd_pa;
+	}
+	if (host_cr3)
+		vmx_vmwrite(HOST_CR3, host_cr3);
+	vmx_vmwrite(HOST_RSP, (unsigned long)&cvp->common.stack[CXP_STACK_SIZE]);
+	vmx_vmwrite(HOST_RIP, (unsigned long)&vmx_caretaker_exit_handler);
+
+	/* Configure Host Selectors */
+	vmx_vmwrite(HOST_CS_SELECTOR, __KERNEL_CS);
+	vmx_vmwrite(HOST_SS_SELECTOR, __KERNEL_DS);
+	vmx_vmwrite(HOST_DS_SELECTOR, __KERNEL_DS);
+	vmx_vmwrite(HOST_ES_SELECTOR, __KERNEL_DS);
+	vmx_vmwrite(HOST_FS_SELECTOR, 0);
+	vmx_vmwrite(HOST_GS_SELECTOR, 0);
+	vmx_vmwrite(HOST_TR_SELECTOR, GDT_ENTRY_TSS * 8);
+
+	/* Configure Host Bases */
+	fs_base = native_rdmsrq(MSR_FS_BASE);
+	gs_base = native_rdmsrq(MSR_GS_BASE);
+	vmx_vmwrite(HOST_FS_BASE, fs_base);
+	vmx_vmwrite(HOST_GS_BASE, gs_base);
+	vmx_vmwrite(HOST_TR_BASE, (unsigned long)&cvp->common.tss);
+	vmx_vmwrite(HOST_GDTR_BASE, (unsigned long)&cvp->common.gdt[0]);
+	vmx_vmwrite(HOST_IDTR_BASE, (unsigned long)&caretaker_x86_idt[0]);
+
+	/* Configure PIN and CPU execution controls */
+	pin = vmx_vmread(PIN_BASED_VM_EXEC_CONTROL);
+	pin |= (PIN_BASED_EXT_INTR_MASK | PIN_BASED_NMI_EXITING);
+	pin &= ~(PIN_BASED_VMX_PREEMPTION_TIMER | PIN_BASED_POSTED_INTR);
+	vmx_vmwrite(PIN_BASED_VM_EXEC_CONTROL, pin);
+
+	cpu_ctl = vmx_vmread(CPU_BASED_VM_EXEC_CONTROL);
+	cpu_ctl &= ~(CPU_BASED_INTR_WINDOW_EXITING |
+		     CPU_BASED_NMI_WINDOW_EXITING);
+	cpu_ctl |= (CPU_BASED_HLT_EXITING |
+		    CPU_BASED_PAUSE_EXITING |
+		    CPU_BASED_MWAIT_EXITING |
+		    CPU_BASED_MONITOR_EXITING |
+		    CPU_BASED_UNCOND_IO_EXITING);
+
+	if (cvp && cvp->ple_supported &&
+	    (cpu_ctl & CPU_BASED_ACTIVATE_SECONDARY_CONTROLS)) {
+		unsigned long sec_ctl = vmx_vmread(SECONDARY_VM_EXEC_CONTROL);
+
+		sec_ctl |= SECONDARY_EXEC_PAUSE_LOOP_EXITING;
+		vmx_vmwrite(SECONDARY_VM_EXEC_CONTROL, sec_ctl);
+		vmx_vmwrite(PLE_GAP, 4096);
+		vmx_vmwrite(PLE_WINDOW, 4096);
+	}
+	vmx_vmwrite(CPU_BASED_VM_EXEC_CONTROL, cpu_ctl);
+}
+
diff --git a/arch/x86/kvm/vmx/caretaker_vmenter.S b/arch/x86/kvm/vmx/caretaker_vmenter.S
index ff135cba854d..51781e3e61e2 100644
--- a/arch/x86/kvm/vmx/caretaker_vmenter.S
+++ b/arch/x86/kvm/vmx/caretaker_vmenter.S
@@ -26,8 +26,8 @@ SYM_TYPED_FUNC_START(vmx_caretaker_enter)
 	CARETAKER_PUSH_HOST_REGS
 	movq %rsp, CXP_STACK_ORIG(%rdi)
 
-	/* Switch to private preserved stack */
-	movq CXP_STACK_TOP(%rdi), %rsp
+	/* Switch to private preserved stack at top of caretaker_x86_page */
+	leaq PAGE_SIZE(%rdi), %rsp
 
 	/* Activate VMCS on this pCPU */
 	vmptrld CXP_VMCS_PA(%rdi)
@@ -60,11 +60,10 @@ SYM_TYPED_FUNC_START(vmx_caretaker_enter)
 	vmresume
 	vmlaunch
 
-	/* If both fail, record error and return */
+	/* If both fail, return error in %rax */
 	popq %rdi
 	mov $VMCS_VM_INSTRUCTION_ERROR, %eax
 	vmread %rax, %rax
-	movq %rax, CXP_LAST_EXIT_CODE(%rdi)
 	btsq $31, %rax
 	jmp .Lvmx_ret
 SYM_FUNC_END(vmx_caretaker_enter)
@@ -73,15 +72,14 @@ SYM_FUNC_START(vmx_caretaker_exit_handler)
 	ENDBR
 	/*
 	 * Hardware jumps here on VM-Exit with:
-	 * RSP = HOST_RSP (CXP_STACK_TOP)
+	 * RSP = HOST_RSP (top of 4 KB struct caretaker_x86_page)
 	 * CR3 = HOST_CR3
 	 */
 	pushq %rdi
 
-	/* Recompute CVP pointer from RSP: CVP = (RSP & PAGE_MASK) - CXP_STACK_OFFSET */
+	/* Recover CVP base from RSP: stack lives in the upper half of page 0 */
 	movq %rsp, %rdi
 	andq $PAGE_MASK, %rdi
-	subq $CXP_STACK_OFFSET, %rdi
 
 	/* Save guest GPRs into CVP */
 	CARETAKER_SAVE_GPRS %rdi
diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c
index 612ab07d4100..48b6e69f99bf 100644
--- a/arch/x86/kvm/vmx/vmx.c
+++ b/arch/x86/kvm/vmx/vmx.c
@@ -839,7 +839,7 @@ static void __loaded_vmcs_clear(void *arg)
 	loaded_vmcs->launched = 0;
 }
 
-static void loaded_vmcs_clear(struct loaded_vmcs *loaded_vmcs)
+void loaded_vmcs_clear(struct loaded_vmcs *loaded_vmcs)
 {
 	int cpu = loaded_vmcs->cpu;
 
diff --git a/arch/x86/kvm/vmx/vmx.h b/arch/x86/kvm/vmx/vmx.h
index dc8517f15bc4..5cb64ae4c8ee 100644
--- a/arch/x86/kvm/vmx/vmx.h
+++ b/arch/x86/kvm/vmx/vmx.h
@@ -342,6 +342,7 @@ static __always_inline u32 vmx_get_intr_info(struct kvm_vcpu *vcpu)
 }
 
 void vmx_vcpu_load_vmcs(struct kvm_vcpu *vcpu, int cpu);
+void loaded_vmcs_clear(struct loaded_vmcs *loaded_vmcs);
 int allocate_vpid(void);
 void free_vpid(int vpid);
 void vmx_set_constant_host_state(struct vcpu_vmx *vmx);
-- 
2.55.0.1082.g2b9226bbc0-goog

[RFC PATCH 39/46] KVM: VMX: Integrate Caretaker VMX detach serialization and KVM registration

[Pasha Tatashin]

Complete Intel VMX Caretaker detach-time serialization, vCPU sync,
and ops registration in arch/x86/kvm/vmx/caretaker.c.

Signed-off-by: Pasha Tatashin <pasha.tatashin@soleen.com>
---
 arch/x86/kvm/Makefile        |   9 +
 arch/x86/kvm/vmx/caretaker.c | 322 ++++++++++++++++++++++++++++++++++-
 arch/x86/kvm/vmx/vmx.c       |   4 +
 3 files changed, 333 insertions(+), 2 deletions(-)

diff --git a/arch/x86/kvm/Makefile b/arch/x86/kvm/Makefile
index 11e67c072258..b550a3fd0708 100644
--- a/arch/x86/kvm/Makefile
+++ b/arch/x86/kvm/Makefile
@@ -18,6 +18,8 @@ kvm-$(CONFIG_KVM_SMM)	+= smm.o
 
 kvm-intel-y		+= vmx/vmx.o vmx/vmenter.o vmx/pmu_intel.o vmx/vmcs12.o \
 			   vmx/nested.o vmx/posted_intr.o vmx/main.o
+kvm-intel-$(CONFIG_KVM_CARETAKER) += vmx/caretaker.o vmx/caretaker_vmenter.o
+
 
 kvm-intel-$(CONFIG_X86_SGX_KVM)	+= vmx/sgx.o
 kvm-intel-$(CONFIG_KVM_HYPERV)	+= vmx/hyperv.o vmx/hyperv_evmcs.o
@@ -44,6 +46,13 @@ $(obj)/svm/vmenter.o: $(obj)/kvm-asm-offsets.h
 AFLAGS_vmx/vmenter.o    := -iquote $(obj)
 $(obj)/vmx/vmenter.o: $(obj)/kvm-asm-offsets.h
 
+AFLAGS_vmx/caretaker_vmenter.o := -iquote $(obj)
+$(obj)/vmx/caretaker_vmenter.o: $(obj)/kvm-asm-offsets.h
+
+CFLAGS_REMOVE_vmx/caretaker.o = $(CC_FLAGS_FTRACE)
+CFLAGS_vmx/caretaker.o := -fno-stack-protector
+
+
 $(obj)/kvm-asm-offsets.h: $(obj)/kvm-asm-offsets.s FORCE
 	$(call filechk,offsets,__KVM_ASM_OFFSETS_H__)
 
diff --git a/arch/x86/kvm/vmx/caretaker.c b/arch/x86/kvm/vmx/caretaker.c
index 47b4743650e5..efef1c30c0d1 100644
--- a/arch/x86/kvm/vmx/caretaker.c
+++ b/arch/x86/kvm/vmx/caretaker.c
@@ -49,8 +49,8 @@ static int vmx_caretaker_init_page(struct caretaker_vmx_page *cvp,
 	u64 basic_msr, misc_msr;
 	int ret;
 
-	if (!vmx->vmcs01.vmcs)
-		return -EINVAL;
+	if (!vmx->vmcs01.vmcs || is_guest_mode(vcpu))
+		return -EOPNOTSUPP;
 
 	ret = kvm_x86_caretaker_init_common_page(&cvp->common, vcpu, sizeof(*cvp));
 	if (ret)
@@ -336,5 +336,323 @@ vmx_caretaker_init_host_vmcs(struct caretaker_vmx_page *cvp)
 		vmx_vmwrite(PLE_WINDOW, 4096);
 	}
 	vmx_vmwrite(CPU_BASED_VM_EXEC_CONTROL, cpu_ctl);
+
+	/*
+	 * Disable hardware MSR autoload lists while running in Caretaker so
+	 * VM-entry/VM-exit do not dereference unpreserved host vmx->msr_autoload
+	 * memory after kexec.
+	 */
+	vmx_vmwrite(VM_EXIT_MSR_LOAD_COUNT, 0);
+	vmx_vmwrite(VM_EXIT_MSR_STORE_COUNT, 0);
+	vmx_vmwrite(VM_ENTRY_MSR_LOAD_COUNT, 0);
+}
+
+/*
+ * Guest-visible VMCS fields carried from the VMCS the caretaker ran the vCPU
+ * on to the VMCS the new kernel allocated for it.
+ *
+ * The new kernel does not adopt the old VMCS: it belongs to the previous
+ * kernel's struct loaded_vmcs, whose layout is not part of any handover ABI,
+ * and the VMCS region itself is opaque and implementation defined.  So the
+ * architecturally defined guest state is copied field by field instead.
+ *
+ * GUEST_IA32_EFER is handled separately because it is only written back when
+ * the caretaker actually recorded a value for it.
+ */
+static const u16 vmx_caretaker_guest_fields[] = {
+	GUEST_CS_SELECTOR,	GUEST_CS_LIMIT,
+	GUEST_CS_AR_BYTES,	GUEST_CS_BASE,
+	GUEST_SS_SELECTOR,	GUEST_SS_LIMIT,
+	GUEST_SS_AR_BYTES,	GUEST_SS_BASE,
+	GUEST_DS_SELECTOR,	GUEST_DS_LIMIT,
+	GUEST_DS_AR_BYTES,	GUEST_DS_BASE,
+	GUEST_ES_SELECTOR,	GUEST_ES_LIMIT,
+	GUEST_ES_AR_BYTES,	GUEST_ES_BASE,
+	GUEST_FS_SELECTOR,	GUEST_FS_LIMIT,
+	GUEST_FS_AR_BYTES,	GUEST_FS_BASE,
+	GUEST_GS_SELECTOR,	GUEST_GS_LIMIT,
+	GUEST_GS_AR_BYTES,	GUEST_GS_BASE,
+	GUEST_TR_SELECTOR,	GUEST_TR_LIMIT,
+	GUEST_TR_AR_BYTES,	GUEST_TR_BASE,
+	GUEST_LDTR_SELECTOR,	GUEST_LDTR_LIMIT,
+	GUEST_LDTR_AR_BYTES,	GUEST_LDTR_BASE,
+	GUEST_GDTR_LIMIT,	GUEST_GDTR_BASE,
+	GUEST_IDTR_LIMIT,	GUEST_IDTR_BASE,
+	GUEST_INTERRUPTIBILITY_INFO,
+	GUEST_ACTIVITY_STATE,
+	GUEST_IA32_DEBUGCTL,
+	GUEST_SYSENTER_CS,
+	GUEST_SYSENTER_ESP,
+	GUEST_SYSENTER_EIP,
+};
+
+static void
+vmx_caretaker_sync_vcpu(struct kvm_vcpu *vcpu, void *vcpu_data)
+{
+	struct kvm_caretaker_arch_ser *abi = vcpu_data;
+	struct vcpu_vmx *vmx = to_vmx(vcpu);
+	phys_addr_t cur_vmcs_pa = vmx->loaded_vmcs ? virt_to_phys(vmx->loaded_vmcs->vmcs) : 0;
+	struct vmcs *prev_vmcs;
+
+	guard(preempt)();
+	prev_vmcs = this_cpu_read(current_vmcs);
+
+	if (abi->vmcs_pa && cur_vmcs_pa && abi->vmcs_pa != cur_vmcs_pa) {
+		/* 42 * 8 bytes; this runs on the host stack, not a preserved one. */
+		unsigned long val[ARRAY_SIZE(vmx_caretaker_guest_fields)];
+		unsigned long guest_efer;
+		int i;
+
+		asm volatile("vmptrld %0" : : "m" (abi->vmcs_pa) : "memory", "cc");
+
+		for (i = 0; i < ARRAY_SIZE(vmx_caretaker_guest_fields); i++)
+			val[i] = vmx_vmread(vmx_caretaker_guest_fields[i]);
+		guest_efer = vmx_caretaker_read_efer();
+
+		asm volatile("vmclear %0" : : "m" (abi->vmcs_pa) : "memory", "cc");
+		asm volatile("vmptrld %0" : : "m" (cur_vmcs_pa) : "memory", "cc");
+
+		for (i = 0; i < ARRAY_SIZE(vmx_caretaker_guest_fields); i++)
+			vmx_vmwrite(vmx_caretaker_guest_fields[i], val[i]);
+		if (guest_efer)
+			vmx_vmwrite(GUEST_IA32_EFER, guest_efer);
+
+		abi->vmcs_pa = cur_vmcs_pa;
+	} else if (cur_vmcs_pa) {
+		asm volatile("vmptrld %0" : : "m" (cur_vmcs_pa) : "memory", "cc");
+	}
+
+	kvm_x86_caretaker_sync_vcpu_common(vcpu);
+
+	if (vmx->loaded_vmcs) {
+		pin_controls_clearbit(vmx, PIN_BASED_VMX_PREEMPTION_TIMER);
+		vmcs_write32(PIN_BASED_VM_EXEC_CONTROL, pin_controls_get(vmx));
+		vmcs_write32(CPU_BASED_VM_EXEC_CONTROL, exec_controls_get(vmx));
+		vmcs_write32(VMX_PREEMPTION_TIMER_VALUE, 0);
+		vmcs_write32(VM_EXIT_MSR_LOAD_COUNT, vmx->msr_autoload.host.nr);
+		vmcs_write32(VM_ENTRY_MSR_LOAD_COUNT, vmx->msr_autoload.guest.nr);
+		memset(&vmx->loaded_vmcs->host_state, 0,
+		       sizeof(struct vmcs_host_state));
+		list_del_init(&vmx->loaded_vmcs->loaded_vmcss_on_cpu_link);
+		vmx->loaded_vmcs->cpu = -1;
+		vmx->loaded_vmcs->launched = 0;
+	}
+	vmx_segment_cache_clear(vmx);
+	vmx->vt.guest_state_loaded = false;
+	vmx->guest_uret_msrs_loaded = false;
+
+	vmcs_write32(VM_ENTRY_INTR_INFO_FIELD, 0);
+	vmcs_write32(GUEST_INTERRUPTIBILITY_INFO, 0);
+	vmcs_write32(GUEST_ACTIVITY_STATE, GUEST_ACTIVITY_ACTIVE);
+	vmcs_writel(GUEST_PENDING_DBG_EXCEPTIONS, 0);
+
+	vmcs_writel(GUEST_RIP, kvm_rip_read(vcpu));
+	vmcs_writel(GUEST_RSP, kvm_rsp_read(vcpu));
+	vmcs_writel(GUEST_RFLAGS, kvm_get_rflags(vcpu));
+	vmx_set_cr0(vcpu, vcpu->arch.cr0);
+	vmcs_writel(GUEST_CR3, vcpu->arch.cr3);
+	vmx_set_cr4(vcpu, vcpu->arch.cr4);
+	vmx_set_efer(vcpu, vcpu->arch.efer);
+
+	if (vmx->loaded_vmcs)
+		vmx_set_constant_host_state(vmx);
+
+	if (cur_vmcs_pa)
+		asm volatile("vmclear %0" : : "m" (cur_vmcs_pa) : "memory", "cc");
+
+	if (prev_vmcs && (!vmx->loaded_vmcs || prev_vmcs != vmx->loaded_vmcs->vmcs)) {
+		vmcs_load(prev_vmcs);
+		this_cpu_write(current_vmcs, prev_vmcs);
+	} else {
+		this_cpu_write(current_vmcs, NULL);
+	}
+}
+
+static __cpu_preserved_text void
+vmx_caretaker_detach_serialize(void *page, struct kvm_vcpu_arch_ser *state)
+{
+	struct caretaker_vmx_page *cvp = page;
+
+	kvm_x86_caretaker_detach_serialize_common(&cvp->common, state);
+	kvm_x86_caretaker_update_msr(state, MSR_STAR, cvp->star);
+	kvm_x86_caretaker_update_msr(state, MSR_LSTAR, cvp->lstar);
+	kvm_x86_caretaker_update_msr(state, MSR_SYSCALL_MASK, cvp->fmask);
+	kvm_x86_caretaker_update_msr(state, MSR_KERNEL_GS_BASE,
+				     cvp->common.kernel_gs_base);
+}
+
+static __cpu_preserved_text void vmx_caretaker_arm_timer(void *page, u64 deadline_ticks)
+{
+	struct caretaker_vmx_page *cvp = page;
+	u32 shift = (cvp && cvp->timer_shift) ? cvp->timer_shift : VMX_PREEMPTION_TIMER_SHIFT;
+	u32 timer_value = 0;
+	u32 pin;
+
+	if (deadline_ticks) {
+		u64 now = arch_oncore_read_counter();
+
+		if (deadline_ticks > now) {
+			u64 remaining = deadline_ticks - now;
+
+			timer_value = (u32)(remaining >> shift);
+			if (timer_value == 0)
+				timer_value = 1;
+		} else {
+			timer_value = 1;
+		}
+	}
+
+	if (timer_value > 0) {
+		vmx_vmwrite(VMX_PREEMPTION_TIMER_VALUE, timer_value);
+		pin = (u32)vmx_vmread(PIN_BASED_VM_EXEC_CONTROL);
+		pin |= PIN_BASED_VMX_PREEMPTION_TIMER;
+		vmx_vmwrite(PIN_BASED_VM_EXEC_CONTROL, pin);
+	} else {
+		vmx_caretaker_disarm_timer(page);
+	}
 }
 
+static __cpu_preserved_text void
+vmx_caretaker_advance_rip(void *page, u64 rip)
+{
+	struct caretaker_vmx_page *cvp = page;
+
+	cvp->common.last_exit_rip = rip;
+	vmx_vmwrite(GUEST_RIP, rip);
+}
+
+static __cpu_preserved_text void vmx_caretaker_pre_enter(void *page)
+{
+	struct caretaker_vmx_page *cvp = page;
+
+	/* Ensure VMX is active on this core */
+	if (!(__read_cr4() & X86_CR4_VMXE)) {
+		asm volatile("mov %0, %%cr4" : : "r" (__read_cr4() | X86_CR4_VMXE) : "memory");
+		if (cvp->vmxon_pa) {
+			asm volatile("1: vmxon %[vmxon_pa]\n\t"
+				     "2:\n\t"
+				     _ASM_EXTABLE(1b, 2b)
+				     : : [vmxon_pa] "m" (cvp->vmxon_pa)
+				     : "memory", "cc");
+		}
+	}
+
+	/* Activate VMCS on this pCPU */
+	asm volatile("vmptrld %0" : : "m" (cvp->common.abi.vmcs_pa) : "memory", "cc");
+
+	/* Configure Caretaker host VMCS */
+	vmx_caretaker_init_host_vmcs(cvp);
+
+	if (cvp->common.arch_state)
+		native_write_cr2(cvp->common.arch_state->sregs.cr2);
+
+	native_wrmsrq(MSR_STAR, cvp->star);
+	native_wrmsrq(MSR_LSTAR, cvp->lstar);
+	native_wrmsrq(MSR_SYSCALL_MASK, cvp->fmask);
+}
+
+static __cpu_preserved_text void
+vmx_caretaker_read_seg(struct kvm_segment *var, u16 sel_field,
+		       u16 base_field, u16 limit_field, u16 ar_field)
+{
+	u32 ar = (u32)vmx_vmread(ar_field);
+
+	var->base = vmx_vmread(base_field);
+	var->limit = (u32)vmx_vmread(limit_field);
+	var->selector = (u16)vmx_vmread(sel_field);
+	var->unusable = (ar >> 16) & 1;
+	var->type = ar & 15;
+	var->s = (ar >> 4) & 1;
+	var->dpl = (ar >> 5) & 3;
+	var->present = !var->unusable;
+	var->avl = (ar >> 12) & 1;
+	var->l = (ar >> 13) & 1;
+	var->db = (ar >> 14) & 1;
+	var->g = (ar >> 15) & 1;
+}
+
+static __cpu_preserved_text void vmx_caretaker_post_exit(void *page)
+{
+	struct caretaker_vmx_page *cvp = page;
+	struct kvm_vcpu_arch_ser *state = cvp->common.arch_state;
+	u64 efer;
+
+	cvp->star = native_rdmsrq(MSR_STAR);
+	cvp->lstar = native_rdmsrq(MSR_LSTAR);
+	cvp->fmask = native_rdmsrq(MSR_SYSCALL_MASK);
+
+	cvp->common.cr0 = vmx_caretaker_read_cr0();
+	cvp->common.cr3 = vmx_vmread(GUEST_CR3);
+	cvp->common.cr4 = vmx_caretaker_read_cr4();
+	efer = vmx_caretaker_read_efer();
+	if (efer)
+		cvp->common.efer = efer;
+	cvp->common.last_exit_rip = vmx_vmread(GUEST_RIP);
+	cvp->common.last_exit_rsp = vmx_vmread(GUEST_RSP);
+	cvp->common.last_exit_rflags = vmx_vmread(GUEST_RFLAGS);
+
+	if (state) {
+		state->sregs.cr2 = native_read_cr2();
+		vmx_caretaker_read_seg(&state->sregs.cs, GUEST_CS_SELECTOR,
+				       GUEST_CS_BASE, GUEST_CS_LIMIT, GUEST_CS_AR_BYTES);
+		vmx_caretaker_read_seg(&state->sregs.ds, GUEST_DS_SELECTOR,
+				       GUEST_DS_BASE, GUEST_DS_LIMIT, GUEST_DS_AR_BYTES);
+		vmx_caretaker_read_seg(&state->sregs.es, GUEST_ES_SELECTOR,
+				       GUEST_ES_BASE, GUEST_ES_LIMIT, GUEST_ES_AR_BYTES);
+		vmx_caretaker_read_seg(&state->sregs.fs, GUEST_FS_SELECTOR,
+				       GUEST_FS_BASE, GUEST_FS_LIMIT, GUEST_FS_AR_BYTES);
+		vmx_caretaker_read_seg(&state->sregs.gs, GUEST_GS_SELECTOR,
+				       GUEST_GS_BASE, GUEST_GS_LIMIT, GUEST_GS_AR_BYTES);
+		vmx_caretaker_read_seg(&state->sregs.ss, GUEST_SS_SELECTOR,
+				       GUEST_SS_BASE, GUEST_SS_LIMIT, GUEST_SS_AR_BYTES);
+		vmx_caretaker_read_seg(&state->sregs.tr, GUEST_TR_SELECTOR,
+				       GUEST_TR_BASE, GUEST_TR_LIMIT, GUEST_TR_AR_BYTES);
+		vmx_caretaker_read_seg(&state->sregs.ldt, GUEST_LDTR_SELECTOR,
+				       GUEST_LDTR_BASE, GUEST_LDTR_LIMIT, GUEST_LDTR_AR_BYTES);
+		state->sregs.gdt.base = vmx_vmread(GUEST_GDTR_BASE);
+		state->sregs.gdt.limit = (u16)vmx_vmread(GUEST_GDTR_LIMIT);
+		state->sregs.idt.base = vmx_vmread(GUEST_IDTR_BASE);
+		state->sregs.idt.limit = (u16)vmx_vmread(GUEST_IDTR_LIMIT);
+
+		kvm_x86_caretaker_update_msr(state, MSR_IA32_SYSENTER_CS,
+					     vmx_vmread(GUEST_SYSENTER_CS));
+		kvm_x86_caretaker_update_msr(state, MSR_IA32_SYSENTER_ESP,
+					     vmx_vmread(GUEST_SYSENTER_ESP));
+		kvm_x86_caretaker_update_msr(state, MSR_IA32_SYSENTER_EIP,
+					     vmx_vmread(GUEST_SYSENTER_EIP));
+	}
+
+	/* Flush VMCS cache so host and incoming kernel see latest guest state */
+	asm volatile("vmclear %0" : : "m" (cvp->common.abi.vmcs_pa) : "memory", "cc");
+}
+
+static const struct kvm_x86_caretaker_runtime_ops vmx_caretaker_runtime_ops __cpu_preserved_data = {
+	.detach_serialize = vmx_caretaker_detach_serialize,
+	.common = {
+		.enter_guest = vmx_caretaker_enter,
+		.decode_exit = vmx_caretaker_decode_exit,
+		.handle_arch_exit = kvm_x86_caretaker_handle_exit,
+		.advance_rip = vmx_caretaker_advance_rip,
+		.arm_timer = vmx_caretaker_arm_timer,
+		.disarm_timer = vmx_caretaker_disarm_timer,
+		.pre_run = vmx_caretaker_pre_enter,
+		.post_run = vmx_caretaker_post_exit,
+	},
+};
+
+static const struct kvm_x86_caretaker_ops vmx_caretaker_ops = {
+	.name = "vmx",
+	.init = vmx_caretaker_init,
+	.sync_vcpu = vmx_caretaker_sync_vcpu,
+	.runtime = &vmx_caretaker_runtime_ops,
+};
+
+void vmx_caretaker_register(void)
+{
+	kvm_x86_caretaker_register_ops(&vmx_caretaker_ops);
+}
+
+void vmx_caretaker_unregister(void)
+{
+	kvm_x86_caretaker_unregister_ops(&vmx_caretaker_ops);
+}
diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c
index 48b6e69f99bf..817623f7429a 100644
--- a/arch/x86/kvm/vmx/vmx.c
+++ b/arch/x86/kvm/vmx/vmx.c
@@ -54,6 +54,7 @@
 #include <trace/events/ipi.h>
 
 #include "capabilities.h"
+#include "caretaker.h"
 #include "common.h"
 #include "cpuid.h"
 #include "hyperv.h"
@@ -8548,6 +8549,7 @@ void vmx_migrate_timers(struct kvm_vcpu *vcpu)
 
 void vmx_hardware_unsetup(void)
 {
+	vmx_caretaker_unregister();
 	kvm_set_posted_intr_wakeup_handler(NULL);
 
 	if (nested)
@@ -8855,6 +8857,8 @@ __init int vmx_hardware_setup(void)
 
 	kvm_caps.inapplicable_quirks &= ~KVM_X86_QUIRK_IGNORE_GUEST_PAT;
 
+	vmx_caretaker_register();
+
 	return 0;
 }
 
-- 
2.55.0.1082.g2b9226bbc0-goog

Re: [RFC PATCH 00/46] Orphaned Virtual Machines

[Graf (AWS), Alexander]

On 20.09.26 21:36, Pasha Tatashin wrote:
> This RFC series is a PoC intended to showcase e2e working OrphanVMs
> uninterrupted continuing execution on preserved physical CPUs across a
> host kernel live update, prepared for the LPC'26 presentation at the
> KVM Microconf [1].
>
> The original design proposal discussion can be found at [2].
>
> This series is applied on top of v7.3-rc1 + the "[PATCH v4 00/11]
> liveupdate: kvm: Guest_memfd preservation" series [3]. The whole branch
> can be accessed at [4].
>
> It was tested on Intel Xeon Granite Rapids-AP, AMD EPYC Turin, and ARM
> Neoverse V2, as well as in emulation on QEMU and Intel Simics. However,
> it is still very early WIP, and is not anywhere near being production
> ready.
>
> Overview & Submission Strategy
> ==============================
> While this RFC bundles the e2e stack, it is NOT intended to be merged as
> a single series, the goal of this series is to have a discussion about
> the technical layers, and the challenges that need to be solved in order
> to have full Caretaker support within the kernel.
>
> Some of the arch/* changes duplicate small helpers (such as GICv3 list
> register accessors or low-level VMCS/VMCB/sysreg helpers) that could be
> shared with existing KVM/arch code if those routines and their callees
> are moved to static inlines or annotated with __cpu_preserved_text (and
> built without ftrace/KASAN/per-CPU references, as enforced by objtool
> and modpost). To avoid sprinkling __cpu_preserved_text annotations
> across core arch/* files in this PoC, those helpers are kept local to
> caretaker.c for now and can be consolidated when individual platform
> support is upstreamed.
>
> The series is structured into at least 8 workstreams that will be
> discussed and worked on separately:
>
> 1. Workstream 1: Preparation Patches (Patches 01-06)
>     Architecture-neutral and arch-specific preparatory cleanups (in
>     future these could be bundled with the workstreams that require these
>     changes):
>       01. KVM: luo: Delegate VM creation type to kvm_arch_vm_luo_preserve
>       02. KVM: arm64: Split demux_c15_{get,set}_val from userspace
>           accessors
>       03. KVM: arm64: Split kvm_sys_reg_{get,set}_user from kernel
>           accessors
>       04. x86/mm/ident_map: Add force_pte to support 4K PTE identity
>           mappings
>       05. arm64: mm: Add trans_pgd_map_range() support
>       06. x86/smp: Skip offline CPUs for REBOOT_VECTOR in
>           native_stop_other_cpus()
>
> 2. Workstream 2: In-Kernel In-RAM vCPU State Preservation via LUO
>     (Patches 07-10)
>     Allows performing suspend/resume-type preservation across kexec
>     without carrying the vCPU internal state in the VMM in userspace,
>     keeping all vCPU state in-kernel:
>       07. KVM: luo: Support vCPU file preservation across live updates
>       08. KVM: x86: Add x86 vCPU LUO preservation ABI and register
>           helpers
>       09. KVM: x86: Implement architectural vCPU state preservation via
>           LUO
>       10. KVM: arm64: Implement architectural vCPU state preservation via
>           LUO
>
> 3. Workstream 3: Physical CPU Preservation Core & x86 Support
>     (Patches 11-24)
>     Introduces the physical CPU preservation subsystem (cpu_preserve),
>     linker sections (.text.cpu_preserved, .data.cpu_preserved) with
>     build-time section isolation checks in modpost and objtool, isolated
>     transition page table management (struct cpu_preserved_as), and sysfs
>     interfaces. Allows booting the machine with some CPUs removed from
>     the OS scheduler, and provides infrastructure for executing orphaned
>     jobs on such CPUs:
>       11. liveupdate: Define CPU preservation linker sections
>       12. liveupdate: Add liveupdate_session_name() helper
>       13. cpu_preserve: Add physical CPU preservation ABI and core API
>           headers
>       14. cpu_preserve: Add core physical CPU preservation state and park
>           loop
>       15. cpu_preserve: Add physical CPU preservation lifecycle and build
>           rules
>       16. liveupdate: cpu_preserve: Add sysfs interface
>       17. liveupdate: cpu_preserve: Add isolated address space management
>           API
>       18. liveupdate: cpu_preserve: Add LUO file handler for preserved
>           physical CPUs
>       19. x86: liveupdate: Add low-level physical CPU preservation
>           assembly
>       20. x86: liveupdate: Add physical CPU preservation context and page
>           table support
>       21. selftests: liveupdate: Add physical CPU preservation unit tests
>       22. selftests: liveupdate: Add physical CPU preservation live
>           update tests
>       23. Documentation: liveupdate: Add physical CPU preservation
>           documentation
>       24. MAINTAINERS: Add entry for KVM Caretaker
>
> 4. Workstream 4: ARM64 Support for Physical CPU Preservation (Patch 25)
>     Enables physical CPU preservation on ARM64 using isolated transition
>     page tables (trans_pgd_map_range()), cache maintenance to PoC, and
>     GICv3 redistributor SGI wake helpers:
>       25. arm64: liveupdate: Add support for physical CPU preservation
>
> 5. Workstream 5: On-Core Scheduling Framework (oncore) (Patches 26-27)
>     (Note: This layer sits between cpu_preserve and the KVM Caretaker and
>     Kernel Caretaker (Future Work for Orphaned Processes))
>     Provides a runqueue and time-slice scheduling framework
>     (kernel/liveupdate/oncore.c) that multiplexes preserved workloads
>     onto preserved physical CPUs during the kexec blackout window:
>       26. oncore: Add on-core KHO ABI and public framework headers
>       27. oncore: Implement on-core session lifecycle and scheduling loop
>
> 6. Workstream 6: KVM Caretaker Core & Intel VMX Support
>     (Patches 28-39)
>     Introduces the KVM Caretaker execution engine (virt/kvm/caretaker.c),
>     vCPU detach/attach/cancel lifecycle, optional cross-kexec execution
>     telemetry and debugfs reporting, x86 TDP MMU KHO page table
>     preservation, x86 common Caretaker runtime (LAPIC timer injection,
>     VM-exit dispatch, instruction decoding), and the Intel VMX Caretaker
>     backend:
>       28. KVM: caretaker: Add Caretaker control block and architecture
>           ops headers
>       29. KVM: caretaker: Implement Caretaker session memory mapping
>           helpers
>       30. KVM: caretaker: Integrate Caretaker vCPU detach, attach, and
>           cancel with KVM
>       31. KVM: caretaker: Add generic KHO ABI telemetry and debugfs
>           reporting
>       32. KVM: x86: Add TDP MMU KHO preservation helpers
>       33. KVM: x86: Add Caretaker x86 KHO ABI and runtime context headers
>       34. KVM: x86: Implement Caretaker LAPIC timer and interrupt
>           injection
>       35. KVM: x86: Implement Caretaker VM-exit dispatch and instruction
>           decoders
>       36. KVM: x86: Implement Caretaker run loop and LUO detach/attach
>           lifecycle
>       37. KVM: VMX: Add Caretaker VMX assembly guest entry/exit routine
>           and helpers
>       38. KVM: VMX: Implement Caretaker VMX VMCS lifecycle and exit
>           dispatch
>       39. KVM: VMX: Integrate Caretaker VMX detach serialization and KVM
>           registration
>
> 7. Workstream 7: KVM Caretaker for AMD SVM (Patches 40-41)
>     Implements the AMD SVM Caretaker backend (caretaker_vmenter.S, VMCB
>     lifecycle, exit decoding, and next-RIP sync):
>       40. KVM: SVM: Add Caretaker SVM assembly guest entry/exit routine
>       41. KVM: SVM: Implement Caretaker SVM VMCB lifecycle and exit
>           dispatch
>
> 8. Workstream 8: KVM Caretaker for ARM64 (Patches 42-46)
>     Implements the ARM64 VHE Caretaker backend (caretaker_vmenter.S,
>     EL2 exception vectors, GICv3 CPU interface & arch timer emulation,
>     system register trap handling, and Stage-2 fault handling):
>       42. KVM: arm64: Add Caretaker arm64 KHO ABI and runtime context
>           headers
>       43. KVM: arm64: Add Caretaker EL2 exception vectors and guest
>           entry/exit assembly
>       44. KVM: arm64: Implement Caretaker GICv3 CPU interface and arch
>           timer emulation
>       45. KVM: arm64: Implement Caretaker system register trap and
>           exception handlers
>       46. KVM: arm64: Implement Caretaker vCPU run loop and LUO
>           detach/attach lifecycle
>
> Architectural Design Layers
> ===========================
> The OrphanVM architecture is layered so that each subsystem has zero
> upward symbol dependencies:
>
>    +--------------------------------------------------------------------+
>    | Layer 4: KVM Caretaker Engine (virt/kvm/caretaker.c + arch/kvm/*)  |
>    |   - VMENTER/VMRUN/ERET guest execution loop                        |
>    |   - Handling of timer preemption, HLT/WFI, MSR/sysreg traps        |
>    |   - Yield on unhandled I/O or Stage-2 MMIO faults                  |
>    +--------------------------------------------------------------------+


I had a superficial look at the code only, but it looked like we're not 
reusing any of the existing KVM code for entry or exit handling. I was 
hoping we could refactor some bits of KVM code into functionality that 
is safe stand-alone and functionality that does require the larger 
kernel ecosystem, then annotate the stand-alone ones as caretaker (btw, 
abbreviating as kvmct would shorten your function names significantly 
:)) and then consume them from both contexts.

Do you have any timing measurements for the approach as is?

Also, your patch set is cut off at 39/46. I was very curious to see how 
you deal with vGIC and context switching in the caretaker environment :).


Alex

[RFC PATCH 39/46] KVM: VMX: Integrate Caretaker VMX detach serialization and KVM registration

[Pasha Tatashin]

Complete Intel VMX Caretaker detach-time serialization, vCPU sync,
and ops registration in arch/x86/kvm/vmx/caretaker.c.

Signed-off-by: Pasha Tatashin <pasha.tatashin@soleen.com>
---
 arch/x86/kvm/Makefile        |   9 +
 arch/x86/kvm/vmx/caretaker.c | 322 ++++++++++++++++++++++++++++++++++-
 arch/x86/kvm/vmx/vmx.c       |   4 +
 3 files changed, 333 insertions(+), 2 deletions(-)

diff --git a/arch/x86/kvm/Makefile b/arch/x86/kvm/Makefile
index 11e67c072258..b550a3fd0708 100644
--- a/arch/x86/kvm/Makefile
+++ b/arch/x86/kvm/Makefile
@@ -18,6 +18,8 @@ kvm-$(CONFIG_KVM_SMM)	+= smm.o
 
 kvm-intel-y		+= vmx/vmx.o vmx/vmenter.o vmx/pmu_intel.o vmx/vmcs12.o \
 			   vmx/nested.o vmx/posted_intr.o vmx/main.o
+kvm-intel-$(CONFIG_KVM_CARETAKER) += vmx/caretaker.o vmx/caretaker_vmenter.o
+
 
 kvm-intel-$(CONFIG_X86_SGX_KVM)	+= vmx/sgx.o
 kvm-intel-$(CONFIG_KVM_HYPERV)	+= vmx/hyperv.o vmx/hyperv_evmcs.o
@@ -44,6 +46,13 @@ $(obj)/svm/vmenter.o: $(obj)/kvm-asm-offsets.h
 AFLAGS_vmx/vmenter.o    := -iquote $(obj)
 $(obj)/vmx/vmenter.o: $(obj)/kvm-asm-offsets.h
 
+AFLAGS_vmx/caretaker_vmenter.o := -iquote $(obj)
+$(obj)/vmx/caretaker_vmenter.o: $(obj)/kvm-asm-offsets.h
+
+CFLAGS_REMOVE_vmx/caretaker.o = $(CC_FLAGS_FTRACE)
+CFLAGS_vmx/caretaker.o := -fno-stack-protector
+
+
 $(obj)/kvm-asm-offsets.h: $(obj)/kvm-asm-offsets.s FORCE
 	$(call filechk,offsets,__KVM_ASM_OFFSETS_H__)
 
diff --git a/arch/x86/kvm/vmx/caretaker.c b/arch/x86/kvm/vmx/caretaker.c
index 47b4743650e5..efef1c30c0d1 100644
--- a/arch/x86/kvm/vmx/caretaker.c
+++ b/arch/x86/kvm/vmx/caretaker.c
@@ -49,8 +49,8 @@ static int vmx_caretaker_init_page(struct caretaker_vmx_page *cvp,
 	u64 basic_msr, misc_msr;
 	int ret;
 
-	if (!vmx->vmcs01.vmcs)
-		return -EINVAL;
+	if (!vmx->vmcs01.vmcs || is_guest_mode(vcpu))
+		return -EOPNOTSUPP;
 
 	ret = kvm_x86_caretaker_init_common_page(&cvp->common, vcpu, sizeof(*cvp));
 	if (ret)
@@ -336,5 +336,323 @@ vmx_caretaker_init_host_vmcs(struct caretaker_vmx_page *cvp)
 		vmx_vmwrite(PLE_WINDOW, 4096);
 	}
 	vmx_vmwrite(CPU_BASED_VM_EXEC_CONTROL, cpu_ctl);
+
+	/*
+	 * Disable hardware MSR autoload lists while running in Caretaker so
+	 * VM-entry/VM-exit do not dereference unpreserved host vmx->msr_autoload
+	 * memory after kexec.
+	 */
+	vmx_vmwrite(VM_EXIT_MSR_LOAD_COUNT, 0);
+	vmx_vmwrite(VM_EXIT_MSR_STORE_COUNT, 0);
+	vmx_vmwrite(VM_ENTRY_MSR_LOAD_COUNT, 0);
+}
+
+/*
+ * Guest-visible VMCS fields carried from the VMCS the caretaker ran the vCPU
+ * on to the VMCS the new kernel allocated for it.
+ *
+ * The new kernel does not adopt the old VMCS: it belongs to the previous
+ * kernel's struct loaded_vmcs, whose layout is not part of any handover ABI,
+ * and the VMCS region itself is opaque and implementation defined.  So the
+ * architecturally defined guest state is copied field by field instead.
+ *
+ * GUEST_IA32_EFER is handled separately because it is only written back when
+ * the caretaker actually recorded a value for it.
+ */
+static const u16 vmx_caretaker_guest_fields[] = {
+	GUEST_CS_SELECTOR,	GUEST_CS_LIMIT,
+	GUEST_CS_AR_BYTES,	GUEST_CS_BASE,
+	GUEST_SS_SELECTOR,	GUEST_SS_LIMIT,
+	GUEST_SS_AR_BYTES,	GUEST_SS_BASE,
+	GUEST_DS_SELECTOR,	GUEST_DS_LIMIT,
+	GUEST_DS_AR_BYTES,	GUEST_DS_BASE,
+	GUEST_ES_SELECTOR,	GUEST_ES_LIMIT,
+	GUEST_ES_AR_BYTES,	GUEST_ES_BASE,
+	GUEST_FS_SELECTOR,	GUEST_FS_LIMIT,
+	GUEST_FS_AR_BYTES,	GUEST_FS_BASE,
+	GUEST_GS_SELECTOR,	GUEST_GS_LIMIT,
+	GUEST_GS_AR_BYTES,	GUEST_GS_BASE,
+	GUEST_TR_SELECTOR,	GUEST_TR_LIMIT,
+	GUEST_TR_AR_BYTES,	GUEST_TR_BASE,
+	GUEST_LDTR_SELECTOR,	GUEST_LDTR_LIMIT,
+	GUEST_LDTR_AR_BYTES,	GUEST_LDTR_BASE,
+	GUEST_GDTR_LIMIT,	GUEST_GDTR_BASE,
+	GUEST_IDTR_LIMIT,	GUEST_IDTR_BASE,
+	GUEST_INTERRUPTIBILITY_INFO,
+	GUEST_ACTIVITY_STATE,
+	GUEST_IA32_DEBUGCTL,
+	GUEST_SYSENTER_CS,
+	GUEST_SYSENTER_ESP,
+	GUEST_SYSENTER_EIP,
+};
+
+static void
+vmx_caretaker_sync_vcpu(struct kvm_vcpu *vcpu, void *vcpu_data)
+{
+	struct kvm_caretaker_arch_ser *abi = vcpu_data;
+	struct vcpu_vmx *vmx = to_vmx(vcpu);
+	phys_addr_t cur_vmcs_pa = vmx->loaded_vmcs ? virt_to_phys(vmx->loaded_vmcs->vmcs) : 0;
+	struct vmcs *prev_vmcs;
+
+	guard(preempt)();
+	prev_vmcs = this_cpu_read(current_vmcs);
+
+	if (abi->vmcs_pa && cur_vmcs_pa && abi->vmcs_pa != cur_vmcs_pa) {
+		/* 42 * 8 bytes; this runs on the host stack, not a preserved one. */
+		unsigned long val[ARRAY_SIZE(vmx_caretaker_guest_fields)];
+		unsigned long guest_efer;
+		int i;
+
+		asm volatile("vmptrld %0" : : "m" (abi->vmcs_pa) : "memory", "cc");
+
+		for (i = 0; i < ARRAY_SIZE(vmx_caretaker_guest_fields); i++)
+			val[i] = vmx_vmread(vmx_caretaker_guest_fields[i]);
+		guest_efer = vmx_caretaker_read_efer();
+
+		asm volatile("vmclear %0" : : "m" (abi->vmcs_pa) : "memory", "cc");
+		asm volatile("vmptrld %0" : : "m" (cur_vmcs_pa) : "memory", "cc");
+
+		for (i = 0; i < ARRAY_SIZE(vmx_caretaker_guest_fields); i++)
+			vmx_vmwrite(vmx_caretaker_guest_fields[i], val[i]);
+		if (guest_efer)
+			vmx_vmwrite(GUEST_IA32_EFER, guest_efer);
+
+		abi->vmcs_pa = cur_vmcs_pa;
+	} else if (cur_vmcs_pa) {
+		asm volatile("vmptrld %0" : : "m" (cur_vmcs_pa) : "memory", "cc");
+	}
+
+	kvm_x86_caretaker_sync_vcpu_common(vcpu);
+
+	if (vmx->loaded_vmcs) {
+		pin_controls_clearbit(vmx, PIN_BASED_VMX_PREEMPTION_TIMER);
+		vmcs_write32(PIN_BASED_VM_EXEC_CONTROL, pin_controls_get(vmx));
+		vmcs_write32(CPU_BASED_VM_EXEC_CONTROL, exec_controls_get(vmx));
+		vmcs_write32(VMX_PREEMPTION_TIMER_VALUE, 0);
+		vmcs_write32(VM_EXIT_MSR_LOAD_COUNT, vmx->msr_autoload.host.nr);
+		vmcs_write32(VM_ENTRY_MSR_LOAD_COUNT, vmx->msr_autoload.guest.nr);
+		memset(&vmx->loaded_vmcs->host_state, 0,
+		       sizeof(struct vmcs_host_state));
+		list_del_init(&vmx->loaded_vmcs->loaded_vmcss_on_cpu_link);
+		vmx->loaded_vmcs->cpu = -1;
+		vmx->loaded_vmcs->launched = 0;
+	}
+	vmx_segment_cache_clear(vmx);
+	vmx->vt.guest_state_loaded = false;
+	vmx->guest_uret_msrs_loaded = false;
+
+	vmcs_write32(VM_ENTRY_INTR_INFO_FIELD, 0);
+	vmcs_write32(GUEST_INTERRUPTIBILITY_INFO, 0);
+	vmcs_write32(GUEST_ACTIVITY_STATE, GUEST_ACTIVITY_ACTIVE);
+	vmcs_writel(GUEST_PENDING_DBG_EXCEPTIONS, 0);
+
+	vmcs_writel(GUEST_RIP, kvm_rip_read(vcpu));
+	vmcs_writel(GUEST_RSP, kvm_rsp_read(vcpu));
+	vmcs_writel(GUEST_RFLAGS, kvm_get_rflags(vcpu));
+	vmx_set_cr0(vcpu, vcpu->arch.cr0);
+	vmcs_writel(GUEST_CR3, vcpu->arch.cr3);
+	vmx_set_cr4(vcpu, vcpu->arch.cr4);
+	vmx_set_efer(vcpu, vcpu->arch.efer);
+
+	if (vmx->loaded_vmcs)
+		vmx_set_constant_host_state(vmx);
+
+	if (cur_vmcs_pa)
+		asm volatile("vmclear %0" : : "m" (cur_vmcs_pa) : "memory", "cc");
+
+	if (prev_vmcs && (!vmx->loaded_vmcs || prev_vmcs != vmx->loaded_vmcs->vmcs)) {
+		vmcs_load(prev_vmcs);
+		this_cpu_write(current_vmcs, prev_vmcs);
+	} else {
+		this_cpu_write(current_vmcs, NULL);
+	}
+}
+
+static __cpu_preserved_text void
+vmx_caretaker_detach_serialize(void *page, struct kvm_vcpu_arch_ser *state)
+{
+	struct caretaker_vmx_page *cvp = page;
+
+	kvm_x86_caretaker_detach_serialize_common(&cvp->common, state);
+	kvm_x86_caretaker_update_msr(state, MSR_STAR, cvp->star);
+	kvm_x86_caretaker_update_msr(state, MSR_LSTAR, cvp->lstar);
+	kvm_x86_caretaker_update_msr(state, MSR_SYSCALL_MASK, cvp->fmask);
+	kvm_x86_caretaker_update_msr(state, MSR_KERNEL_GS_BASE,
+				     cvp->common.kernel_gs_base);
+}
+
+static __cpu_preserved_text void vmx_caretaker_arm_timer(void *page, u64 deadline_ticks)
+{
+	struct caretaker_vmx_page *cvp = page;
+	u32 shift = (cvp && cvp->timer_shift) ? cvp->timer_shift : VMX_PREEMPTION_TIMER_SHIFT;
+	u32 timer_value = 0;
+	u32 pin;
+
+	if (deadline_ticks) {
+		u64 now = arch_oncore_read_counter();
+
+		if (deadline_ticks > now) {
+			u64 remaining = deadline_ticks - now;
+
+			timer_value = (u32)(remaining >> shift);
+			if (timer_value == 0)
+				timer_value = 1;
+		} else {
+			timer_value = 1;
+		}
+	}
+
+	if (timer_value > 0) {
+		vmx_vmwrite(VMX_PREEMPTION_TIMER_VALUE, timer_value);
+		pin = (u32)vmx_vmread(PIN_BASED_VM_EXEC_CONTROL);
+		pin |= PIN_BASED_VMX_PREEMPTION_TIMER;
+		vmx_vmwrite(PIN_BASED_VM_EXEC_CONTROL, pin);
+	} else {
+		vmx_caretaker_disarm_timer(page);
+	}
 }
 
+static __cpu_preserved_text void
+vmx_caretaker_advance_rip(void *page, u64 rip)
+{
+	struct caretaker_vmx_page *cvp = page;
+
+	cvp->common.last_exit_rip = rip;
+	vmx_vmwrite(GUEST_RIP, rip);
+}
+
+static __cpu_preserved_text void vmx_caretaker_pre_enter(void *page)
+{
+	struct caretaker_vmx_page *cvp = page;
+
+	/* Ensure VMX is active on this core */
+	if (!(__read_cr4() & X86_CR4_VMXE)) {
+		asm volatile("mov %0, %%cr4" : : "r" (__read_cr4() | X86_CR4_VMXE) : "memory");
+		if (cvp->vmxon_pa) {
+			asm volatile("1: vmxon %[vmxon_pa]\n\t"
+				     "2:\n\t"
+				     _ASM_EXTABLE(1b, 2b)
+				     : : [vmxon_pa] "m" (cvp->vmxon_pa)
+				     : "memory", "cc");
+		}
+	}
+
+	/* Activate VMCS on this pCPU */
+	asm volatile("vmptrld %0" : : "m" (cvp->common.abi.vmcs_pa) : "memory", "cc");
+
+	/* Configure Caretaker host VMCS */
+	vmx_caretaker_init_host_vmcs(cvp);
+
+	if (cvp->common.arch_state)
+		native_write_cr2(cvp->common.arch_state->sregs.cr2);
+
+	native_wrmsrq(MSR_STAR, cvp->star);
+	native_wrmsrq(MSR_LSTAR, cvp->lstar);
+	native_wrmsrq(MSR_SYSCALL_MASK, cvp->fmask);
+}
+
+static __cpu_preserved_text void
+vmx_caretaker_read_seg(struct kvm_segment *var, u16 sel_field,
+		       u16 base_field, u16 limit_field, u16 ar_field)
+{
+	u32 ar = (u32)vmx_vmread(ar_field);
+
+	var->base = vmx_vmread(base_field);
+	var->limit = (u32)vmx_vmread(limit_field);
+	var->selector = (u16)vmx_vmread(sel_field);
+	var->unusable = (ar >> 16) & 1;
+	var->type = ar & 15;
+	var->s = (ar >> 4) & 1;
+	var->dpl = (ar >> 5) & 3;
+	var->present = !var->unusable;
+	var->avl = (ar >> 12) & 1;
+	var->l = (ar >> 13) & 1;
+	var->db = (ar >> 14) & 1;
+	var->g = (ar >> 15) & 1;
+}
+
+static __cpu_preserved_text void vmx_caretaker_post_exit(void *page)
+{
+	struct caretaker_vmx_page *cvp = page;
+	struct kvm_vcpu_arch_ser *state = cvp->common.arch_state;
+	u64 efer;
+
+	cvp->star = native_rdmsrq(MSR_STAR);
+	cvp->lstar = native_rdmsrq(MSR_LSTAR);
+	cvp->fmask = native_rdmsrq(MSR_SYSCALL_MASK);
+
+	cvp->common.cr0 = vmx_caretaker_read_cr0();
+	cvp->common.cr3 = vmx_vmread(GUEST_CR3);
+	cvp->common.cr4 = vmx_caretaker_read_cr4();
+	efer = vmx_caretaker_read_efer();
+	if (efer)
+		cvp->common.efer = efer;
+	cvp->common.last_exit_rip = vmx_vmread(GUEST_RIP);
+	cvp->common.last_exit_rsp = vmx_vmread(GUEST_RSP);
+	cvp->common.last_exit_rflags = vmx_vmread(GUEST_RFLAGS);
+
+	if (state) {
+		state->sregs.cr2 = native_read_cr2();
+		vmx_caretaker_read_seg(&state->sregs.cs, GUEST_CS_SELECTOR,
+				       GUEST_CS_BASE, GUEST_CS_LIMIT, GUEST_CS_AR_BYTES);
+		vmx_caretaker_read_seg(&state->sregs.ds, GUEST_DS_SELECTOR,
+				       GUEST_DS_BASE, GUEST_DS_LIMIT, GUEST_DS_AR_BYTES);
+		vmx_caretaker_read_seg(&state->sregs.es, GUEST_ES_SELECTOR,
+				       GUEST_ES_BASE, GUEST_ES_LIMIT, GUEST_ES_AR_BYTES);
+		vmx_caretaker_read_seg(&state->sregs.fs, GUEST_FS_SELECTOR,
+				       GUEST_FS_BASE, GUEST_FS_LIMIT, GUEST_FS_AR_BYTES);
+		vmx_caretaker_read_seg(&state->sregs.gs, GUEST_GS_SELECTOR,
+				       GUEST_GS_BASE, GUEST_GS_LIMIT, GUEST_GS_AR_BYTES);
+		vmx_caretaker_read_seg(&state->sregs.ss, GUEST_SS_SELECTOR,
+				       GUEST_SS_BASE, GUEST_SS_LIMIT, GUEST_SS_AR_BYTES);
+		vmx_caretaker_read_seg(&state->sregs.tr, GUEST_TR_SELECTOR,
+				       GUEST_TR_BASE, GUEST_TR_LIMIT, GUEST_TR_AR_BYTES);
+		vmx_caretaker_read_seg(&state->sregs.ldt, GUEST_LDTR_SELECTOR,
+				       GUEST_LDTR_BASE, GUEST_LDTR_LIMIT, GUEST_LDTR_AR_BYTES);
+		state->sregs.gdt.base = vmx_vmread(GUEST_GDTR_BASE);
+		state->sregs.gdt.limit = (u16)vmx_vmread(GUEST_GDTR_LIMIT);
+		state->sregs.idt.base = vmx_vmread(GUEST_IDTR_BASE);
+		state->sregs.idt.limit = (u16)vmx_vmread(GUEST_IDTR_LIMIT);
+
+		kvm_x86_caretaker_update_msr(state, MSR_IA32_SYSENTER_CS,
+					     vmx_vmread(GUEST_SYSENTER_CS));
+		kvm_x86_caretaker_update_msr(state, MSR_IA32_SYSENTER_ESP,
+					     vmx_vmread(GUEST_SYSENTER_ESP));
+		kvm_x86_caretaker_update_msr(state, MSR_IA32_SYSENTER_EIP,
+					     vmx_vmread(GUEST_SYSENTER_EIP));
+	}
+
+	/* Flush VMCS cache so host and incoming kernel see latest guest state */
+	asm volatile("vmclear %0" : : "m" (cvp->common.abi.vmcs_pa) : "memory", "cc");
+}
+
+static const struct kvm_x86_caretaker_runtime_ops vmx_caretaker_runtime_ops __cpu_preserved_data = {
+	.detach_serialize = vmx_caretaker_detach_serialize,
+	.common = {
+		.enter_guest = vmx_caretaker_enter,
+		.decode_exit = vmx_caretaker_decode_exit,
+		.handle_arch_exit = kvm_x86_caretaker_handle_exit,
+		.advance_rip = vmx_caretaker_advance_rip,
+		.arm_timer = vmx_caretaker_arm_timer,
+		.disarm_timer = vmx_caretaker_disarm_timer,
+		.pre_run = vmx_caretaker_pre_enter,
+		.post_run = vmx_caretaker_post_exit,
+	},
+};
+
+static const struct kvm_x86_caretaker_ops vmx_caretaker_ops = {
+	.name = "vmx",
+	.init = vmx_caretaker_init,
+	.sync_vcpu = vmx_caretaker_sync_vcpu,
+	.runtime = &vmx_caretaker_runtime_ops,
+};
+
+void vmx_caretaker_register(void)
+{
+	kvm_x86_caretaker_register_ops(&vmx_caretaker_ops);
+}
+
+void vmx_caretaker_unregister(void)
+{
+	kvm_x86_caretaker_unregister_ops(&vmx_caretaker_ops);
+}
diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c
index 48b6e69f99bf..817623f7429a 100644
--- a/arch/x86/kvm/vmx/vmx.c
+++ b/arch/x86/kvm/vmx/vmx.c
@@ -54,6 +54,7 @@
 #include <trace/events/ipi.h>
 
 #include "capabilities.h"
+#include "caretaker.h"
 #include "common.h"
 #include "cpuid.h"
 #include "hyperv.h"
@@ -8548,6 +8549,7 @@ void vmx_migrate_timers(struct kvm_vcpu *vcpu)
 
 void vmx_hardware_unsetup(void)
 {
+	vmx_caretaker_unregister();
 	kvm_set_posted_intr_wakeup_handler(NULL);
 
 	if (nested)
@@ -8855,6 +8857,8 @@ __init int vmx_hardware_setup(void)
 
 	kvm_caps.inapplicable_quirks &= ~KVM_X86_QUIRK_IGNORE_GUEST_PAT;
 
+	vmx_caretaker_register();
+
 	return 0;
 }
 
-- 
2.55.0.1082.g2b9226bbc0-goog

[RFC PATCH 40/46] KVM: SVM: Add Caretaker SVM assembly guest entry/exit routine

[Pasha Tatashin]

Add arch/x86/kvm/svm/caretaker_vmenter.S and caretaker.h for standalone
AMD SVM VMRUN world switch on preserved physical cores.

Signed-off-by: Pasha Tatashin <pasha.tatashin@soleen.com>
---
 arch/x86/kvm/svm/caretaker.h         | 63 +++++++++++++++++++
 arch/x86/kvm/svm/caretaker_vmenter.S | 91 ++++++++++++++++++++++++++++
 2 files changed, 154 insertions(+)
 create mode 100644 arch/x86/kvm/svm/caretaker.h
 create mode 100644 arch/x86/kvm/svm/caretaker_vmenter.S

diff --git a/arch/x86/kvm/svm/caretaker.h b/arch/x86/kvm/svm/caretaker.h
new file mode 100644
index 000000000000..54a0673cb1ea
--- /dev/null
+++ b/arch/x86/kvm/svm/caretaker.h
@@ -0,0 +1,63 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/*
+ * Copyright (c) 2026, Google LLC.
+ * Pasha Tatashin <pasha.tatashin@soleen.com>
+ */
+#ifndef __SVM_CARETAKER_H
+#define __SVM_CARETAKER_H
+
+#ifndef __ASSEMBLY__
+#include <linux/types.h>
+#include <linux/kvm_host.h>
+#include <linux/oncore.h>
+#include "svm.h"
+#endif
+
+#include "../caretaker.h"
+
+#define CSP_VMCB_OFFSET		0x1000
+#define CSP_HSAVE_PA_OFFSET	0x3000
+#define CSP_VMCB_PA_OFFSET	0x3010
+#define VMCB_RAX_OFFSET		0x5f8
+
+#ifndef __ASSEMBLY__
+
+struct caretaker_svm_page {
+	struct caretaker_x86_page common;
+
+	/* Page 1 (4KB): Preserved VMCB */
+	struct vmcb vmcb __aligned(PAGE_SIZE);
+
+	/* Page 2 (4KB): Preserved HSAVE area */
+	u8 hsave_area[PAGE_SIZE] __aligned(PAGE_SIZE);
+	u64 hsave_pa;
+	u64 orig_hsave_pa;
+	u64 vmcb_pa;
+	u64 orig_efer;
+
+	/* Preserved MSR and I/O permission bitmaps referenced by VMCB */
+	u8 msrpm[MSRPM_SIZE] __aligned(PAGE_SIZE);
+	u8 iopm[IOPM_SIZE] __aligned(PAGE_SIZE);
+} __aligned(PAGE_SIZE);
+
+static_assert(offsetof(struct caretaker_svm_page, vmcb) == CSP_VMCB_OFFSET);
+static_assert(offsetof(struct caretaker_svm_page, hsave_pa) == CSP_HSAVE_PA_OFFSET);
+static_assert(offsetof(struct caretaker_svm_page, vmcb_pa) == CSP_VMCB_PA_OFFSET);
+
+void svm_recalc_intercepts(struct kvm_vcpu *vcpu);
+
+#ifdef CONFIG_KVM_CARETAKER
+int svm_caretaker_enter(void *page);
+void svm_caretaker_decode_exit(void *page, struct kvm_caretaker_exit *exit);
+void svm_caretaker_register(void);
+void svm_caretaker_unregister(void);
+void svm_caretaker_init(struct kvm_vcpu *vcpu);
+#else
+static inline void svm_caretaker_register(void) {}
+static inline void svm_caretaker_unregister(void) {}
+static inline void svm_caretaker_init(struct kvm_vcpu *vcpu) {}
+#endif
+
+#endif /* !__ASSEMBLY__ */
+
+#endif /* __SVM_CARETAKER_H */
diff --git a/arch/x86/kvm/svm/caretaker_vmenter.S b/arch/x86/kvm/svm/caretaker_vmenter.S
new file mode 100644
index 000000000000..e0276a34403e
--- /dev/null
+++ b/arch/x86/kvm/svm/caretaker_vmenter.S
@@ -0,0 +1,91 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright (c) 2026, Google LLC.
+ * Pasha Tatashin <pasha.tatashin@soleen.com>
+ */
+#include <linux/linkage.h>
+#include <linux/cfi_types.h>
+#include <asm/asm.h>
+#include <asm/bitsperlong.h>
+#include <asm/nospec-branch.h>
+#include "caretaker.h"
+#include "../caretaker_asm.h"
+
+.section .text.cpu_preserved, "ax"
+
+/**
+ * svm_caretaker_enter - Low-level transition to SVM guest mode and back
+ * @csp: Pointer to struct caretaker_svm_page (%rdi)
+ */
+SYM_TYPED_FUNC_START(svm_caretaker_enter)
+	ENDBR
+	CARETAKER_PUSH_HOST_REGS
+	movq %cr3, %rax
+	pushq %rax                  /* saved_cr3 */
+
+	clgi
+	sti
+
+	/* 0. Save host VMCB state (FS, GS, TR, LDTR) to host save area */
+	movq CXP_HSAVE_PA(%rdi), %rax
+	testq %rax, %rax
+	jz 10f
+	vmsave %rax
+10:
+
+	/* 1. Sync guest RAX to VMCB and load guest state (FS, GS, TR, LDTR) */
+	leaq CSP_VMCB_OFFSET(%rdi), %rax
+	movq CXP_REG_RAX(%rdi), %rbx
+	movq %rbx, VMCB_RAX_OFFSET(%rax)
+	movq CSP_VMCB_PA_OFFSET(%rdi), %rax
+	vmload %rax
+
+	/* 2. Restore guest GPRs */
+	CARETAKER_RESTORE_GPRS_NO_RAX %rdi
+
+	/* 3. Run guest */
+	pushq %rdi
+	movq CXP_REG_RDI(%rdi), %rdi
+	movq (%rsp), %rax
+	movq CSP_VMCB_PA_OFFSET(%rax), %rax
+	CARETAKER_CLEAR_CPU_BUFFERS
+	vmrun %rax
+	cli
+
+	/* 4. Guest exited: Save guest GPRs */
+	xchgq %rdi, (%rsp)          /* rdi = csp, stack = guest rdi */
+	popq CXP_REG_RDI(%rdi)
+
+	CARETAKER_SAVE_GPRS_NO_RAX %rdi
+
+	/* Stuff Return Stack Buffer to prevent guest RSB poisoning */
+	CARETAKER_FILL_RETURN_BUFFER %rax
+
+	/* 5. Save guest state to VMCB */
+	movq CSP_VMCB_PA_OFFSET(%rdi), %rax
+	vmsave %rax
+	leaq CSP_VMCB_OFFSET(%rdi), %rax
+	movq VMCB_RAX_OFFSET(%rax), %rbx      /* save.rax */
+	movq %rbx, CXP_REG_RAX(%rdi)
+
+	/* 5a. Restore host VMCB state (FS, GS, TR, LDTR) from HSAVE */
+	movq CXP_HSAVE_PA(%rdi), %rax
+	testq %rax, %rax
+	jz 40f
+	vmload %rax
+40:
+
+	/* 5b. Restore host CR3 if clobbered */
+	popq %rbx                   /* saved_cr3 */
+	movq %cr3, %rcx
+	cmpq %rbx, %rcx
+	je 50f
+	movq %rbx, %cr3
+50:
+	stgi
+
+	/* 6. Restore host callee-saved registers & return to C */
+	CARETAKER_POP_HOST_REGS
+	xorq %rax, %rax
+	ret
+SYM_FUNC_END(svm_caretaker_enter)
-- 
2.55.0.1082.g2b9226bbc0-goog

[RFC PATCH 41/46] KVM: SVM: Implement Caretaker SVM VMCB lifecycle and exit dispatch

[Pasha Tatashin]

Implement AMD SVM Caretaker VMCB preservation, exit decoding, and
detach-time serialization in arch/x86/kvm/svm/caretaker.c.

Signed-off-by: Pasha Tatashin <pasha.tatashin@soleen.com>
---
 arch/x86/kvm/Makefile                |  23 +-
 arch/x86/kvm/svm/caretaker.c         | 467 +++++++++++++++++++++++++++
 arch/x86/kvm/svm/caretaker_vmenter.S |   4 +-
 arch/x86/kvm/svm/svm.c               |  11 +-
 4 files changed, 500 insertions(+), 5 deletions(-)
 create mode 100644 arch/x86/kvm/svm/caretaker.c

diff --git a/arch/x86/kvm/Makefile b/arch/x86/kvm/Makefile
index b550a3fd0708..9213d274817e 100644
--- a/arch/x86/kvm/Makefile
+++ b/arch/x86/kvm/Makefile
@@ -26,6 +26,7 @@ kvm-intel-$(CONFIG_KVM_HYPERV)	+= vmx/hyperv.o vmx/hyperv_evmcs.o
 kvm-intel-$(CONFIG_KVM_INTEL_TDX)	+= vmx/tdx.o
 
 kvm-amd-y		+= svm/svm.o svm/vmenter.o svm/pmu.o svm/nested.o svm/avic.o
+kvm-amd-$(CONFIG_KVM_CARETAKER) += svm/caretaker.o svm/caretaker_vmenter.o
 
 kvm-amd-$(CONFIG_KVM_AMD_SEV)	+= svm/sev.o
 kvm-amd-$(CONFIG_KVM_HYPERV)	+= svm/hyperv.o
@@ -43,6 +44,26 @@ obj-$(CONFIG_KVM_AMD)	+= kvm-amd.o
 AFLAGS_svm/vmenter.o    := -iquote $(obj)
 $(obj)/svm/vmenter.o: $(obj)/kvm-asm-offsets.h
 
+AFLAGS_svm/caretaker_vmenter.o := -iquote $(obj)
+$(obj)/svm/caretaker_vmenter.o: $(obj)/kvm-asm-offsets.h
+
+KASAN_SANITIZE_caretaker.o := n
+KCSAN_SANITIZE_caretaker.o := n
+UBSAN_SANITIZE_caretaker.o := n
+KCOV_INSTRUMENT_caretaker.o := n
+KASAN_SANITIZE_svm/caretaker.o := n
+KCSAN_SANITIZE_svm/caretaker.o := n
+UBSAN_SANITIZE_svm/caretaker.o := n
+KCOV_INSTRUMENT_svm/caretaker.o := n
+KASAN_SANITIZE_vmx/caretaker.o := n
+KCSAN_SANITIZE_vmx/caretaker.o := n
+UBSAN_SANITIZE_vmx/caretaker.o := n
+KCOV_INSTRUMENT_vmx/caretaker.o := n
+CFLAGS_REMOVE_caretaker.o = $(CC_FLAGS_FTRACE)
+CFLAGS_caretaker.o += -fno-stack-protector $(call cc-option,-ftrivial-auto-var-init=uninitialized) $(call cc-option,-fno-jump-tables)
+CFLAGS_REMOVE_svm/caretaker.o = $(CC_FLAGS_FTRACE)
+CFLAGS_svm/caretaker.o := -fno-stack-protector $(call cc-option,-ftrivial-auto-var-init=uninitialized) $(call cc-option,-fno-jump-tables)
+
 AFLAGS_vmx/vmenter.o    := -iquote $(obj)
 $(obj)/vmx/vmenter.o: $(obj)/kvm-asm-offsets.h
 
@@ -50,7 +71,7 @@ AFLAGS_vmx/caretaker_vmenter.o := -iquote $(obj)
 $(obj)/vmx/caretaker_vmenter.o: $(obj)/kvm-asm-offsets.h
 
 CFLAGS_REMOVE_vmx/caretaker.o = $(CC_FLAGS_FTRACE)
-CFLAGS_vmx/caretaker.o := -fno-stack-protector
+CFLAGS_vmx/caretaker.o := -fno-stack-protector $(call cc-option,-ftrivial-auto-var-init=uninitialized) $(call cc-option,-fno-jump-tables)
 
 
 $(obj)/kvm-asm-offsets.h: $(obj)/kvm-asm-offsets.s FORCE
diff --git a/arch/x86/kvm/svm/caretaker.c b/arch/x86/kvm/svm/caretaker.c
new file mode 100644
index 000000000000..cb30e336f014
--- /dev/null
+++ b/arch/x86/kvm/svm/caretaker.c
@@ -0,0 +1,467 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * AMD SVM Caretaker Standalone Execution Engine
+ *
+ * Copyright (c) 2026, Google LLC.
+ * Pasha Tatashin <pasha.tatashin@soleen.com>
+ *
+ * Runs the AMD SVM guest VMRUN loop in an isolated, KHO-preserved memory
+ * page that remains alive and executing across kexec relocation and
+ * kernel handover.
+ */
+
+#include <linux/cpu_preserve.h>
+#include <linux/delay.h>
+#include <linux/kernel.h>
+#include <linux/kexec_handover.h>
+#include <linux/kvm_host.h>
+#include <linux/objtool.h>
+#include <linux/oncore.h>
+
+#include <asm/apic.h>
+#include <asm/cpu_entry_area.h>
+#include <asm/desc.h>
+#include <linux/pgtable.h>
+#include <linux/processor.h>
+#include <asm/set_memory.h>
+#include <asm/svm.h>
+#include <asm/tlbflush.h>
+
+#include "../caretaker.h"
+#include "caretaker.h"
+#include "mmu.h"
+#include "svm.h"
+#include "svm_ops.h"
+
+static int svm_caretaker_init_page(struct caretaker_svm_page *csp, struct kvm_vcpu *vcpu);
+STACK_FRAME_NON_STANDARD(svm_caretaker_init_page);
+
+static int svm_caretaker_init_page(struct caretaker_svm_page *csp, struct kvm_vcpu *vcpu)
+{
+	struct vcpu_svm *svm = to_svm(vcpu);
+	int ret;
+
+	if (is_guest_mode(vcpu) || is_sev_guest(vcpu) ||
+	    kvm_vcpu_apicv_activated(vcpu))
+		return -EOPNOTSUPP;
+
+	ret = kvm_x86_caretaker_init_common_page(&csp->common, vcpu, sizeof(*csp));
+	if (ret)
+		return ret;
+
+	if (svm->vmcb01.ptr)
+		csp->vmcb = *svm->vmcb01.ptr;
+	csp->vmcb_pa = virt_to_phys(&csp->vmcb);
+	csp->hsave_pa = virt_to_phys(csp->hsave_area);
+
+	if (svm->msrpm)
+		memcpy(csp->msrpm, svm->msrpm, MSRPM_SIZE);
+	else
+		memset(csp->msrpm, 0xff, MSRPM_SIZE);
+	memset(csp->iopm, 0xff, IOPM_SIZE);
+	csp->vmcb.control.msrpm_base_pa = virt_to_phys(csp->msrpm);
+	csp->vmcb.control.iopm_base_pa = virt_to_phys(csp->iopm);
+	vmcb_set_intercept(&csp->vmcb.control, INTERCEPT_MSR_PROT);
+	vmcb_set_intercept(&csp->vmcb.control, INTERCEPT_IOIO_PROT);
+
+	/* Enable HLT/CPUID/VMMCALL intercepts handled by standalone loop */
+	vmcb_set_intercept(&csp->vmcb.control, INTERCEPT_HLT);
+	vmcb_set_intercept(&csp->vmcb.control, INTERCEPT_PAUSE);
+	csp->vmcb.control.pause_filter_count = 4096;
+	csp->vmcb.control.pause_filter_thresh = 128;
+	vmcb_clr_intercept(&csp->vmcb.control, INTERCEPT_RDTSC);
+	vmcb_set_intercept(&csp->vmcb.control, INTERCEPT_VMMCALL);
+	vmcb_set_intercept(&csp->vmcb.control, INTERCEPT_CPUID);
+	vmcb_clr_intercept(&csp->vmcb.control, INTERCEPT_MONITOR);
+	vmcb_clr_intercept(&csp->vmcb.control, INTERCEPT_MWAIT);
+	vmcb_set_intercept(&csp->vmcb.control, INTERCEPT_NMI);
+	vmcb_clr_intercept(&csp->vmcb.control, INTERCEPT_INTR);
+	vmcb_set_intercept(&csp->vmcb.control, INTERCEPT_INIT);
+
+	/*
+	 * Clear interrupt-window intercepts and virtual IRQ injection state
+	 * left behind in vmcb01 if vcpu_enter_guest() was aborted while
+	 * KVM had an active interrupt window open (matching VMX clearing
+	 * CPU_BASED_INTR_WINDOW_EXITING | CPU_BASED_NMI_WINDOW_EXITING).
+	 */
+	vmcb_clr_intercept(&csp->vmcb.control, INTERCEPT_VINTR);
+	vmcb_clr_intercept(&csp->vmcb.control, INTERCEPT_IRET);
+	vmcb_clr_intercept(&csp->vmcb.control, INTERCEPT_STGI);
+	vmcb_clr_intercept(&csp->vmcb.control, INTERCEPT_CR8_WRITE);
+
+	/* Clear VMCB clean bits and flush TLB for execution on Caretaker CPU */
+	csp->vmcb.control.clean = 0;
+	csp->vmcb.control.tlb_ctl = TLB_CONTROL_FLUSH_ALL_ASID;
+	csp->vmcb.control.int_ctl &= ~V_IRQ_INJECTION_BITS_MASK;
+	csp->vmcb.control.int_ctl |= V_INTR_MASKING_MASK;
+	if (csp->vmcb.control.int_ctl & V_GIF_ENABLE_MASK)
+		csp->vmcb.control.int_ctl |= V_GIF_MASK;
+	csp->vmcb.control.int_vector = 0;
+	csp->vmcb.control.int_state = 0;
+	csp->vmcb.control.event_inj = 0;
+	csp->vmcb.control.exit_int_info = 0;
+	if (csp->vmcb.control.asid == 0)
+		csp->vmcb.control.asid = 1;
+	csp->vmcb.save.rflags = kvm_get_rflags(vcpu);
+	csp->vmcb.save.rax = csp->common.rax;
+	csp->vmcb.save.rsp = kvm_rsp_read(vcpu);
+	csp->vmcb.save.rip = kvm_rip_read(vcpu);
+	csp->common.cr3 = csp->vmcb.save.cr3;
+	csp->common.cr0 = csp->vmcb.save.cr0;
+	csp->common.cr4 = csp->vmcb.save.cr4;
+	csp->common.efer = csp->vmcb.save.efer;
+	csp->common.last_exit_rsp = csp->vmcb.save.rsp;
+	csp->common.last_exit_rip = csp->vmcb.save.rip;
+	csp->common.last_exit_rflags = csp->vmcb.save.rflags;
+
+	return 0;
+}
+
+void svm_caretaker_init(struct kvm_vcpu *vcpu)
+{
+	struct caretaker_svm_page *csp;
+
+	csp = kho_alloc_preserve(sizeof(*csp));
+	if (IS_ERR(csp)) {
+		pr_err("caretaker svm: failed to allocate preserved page\n");
+		return;
+	}
+
+	if (svm_caretaker_init_page(csp, vcpu)) {
+		kvm_x86_caretaker_unpreserve_pages(&csp->common.abi);
+		vcpu->caretaker.cb = NULL;
+		kho_unpreserve_free(csp);
+	}
+}
+
+static __cpu_preserved_text void
+svm_caretaker_seg_to_kvm(struct kvm_segment *var, const struct vmcb_seg *s, int seg)
+{
+	var->base = s->base;
+	var->limit = s->limit;
+	var->selector = s->selector;
+	var->type = s->attrib & SVM_SELECTOR_TYPE_MASK;
+	var->s = (s->attrib >> SVM_SELECTOR_S_SHIFT) & 1;
+	var->dpl = (s->attrib >> SVM_SELECTOR_DPL_SHIFT) & 3;
+	var->present = (s->attrib >> SVM_SELECTOR_P_SHIFT) & 1;
+	var->avl = (s->attrib >> SVM_SELECTOR_AVL_SHIFT) & 1;
+	var->l = (s->attrib >> SVM_SELECTOR_L_SHIFT) & 1;
+	var->db = (s->attrib >> SVM_SELECTOR_DB_SHIFT) & 1;
+	var->g = s->limit > 0xfffff;
+	var->unusable = !var->present;
+	if (seg == VCPU_SREG_TR)
+		var->type |= 0x2;
+}
+
+static __cpu_preserved_text void
+svm_caretaker_detach_serialize(void *page, struct kvm_vcpu_arch_ser *state)
+{
+	struct caretaker_svm_page *csp = page;
+	struct vmcb_save_area *save = &csp->vmcb.save;
+
+	csp->common.cr0 = save->cr0;
+	csp->common.cr3 = save->cr3;
+	csp->common.cr4 = save->cr4;
+	csp->common.efer = save->efer;
+	csp->common.last_exit_rip = save->rip;
+	csp->common.last_exit_rsp = save->rsp;
+	csp->common.last_exit_rflags = save->rflags;
+
+	kvm_x86_caretaker_detach_serialize_common(&csp->common, state);
+
+	state->sregs.cr2 = save->cr2;
+	svm_caretaker_seg_to_kvm(&state->sregs.cs, &save->cs, VCPU_SREG_CS);
+	svm_caretaker_seg_to_kvm(&state->sregs.ds, &save->ds, VCPU_SREG_DS);
+	svm_caretaker_seg_to_kvm(&state->sregs.es, &save->es, VCPU_SREG_ES);
+	svm_caretaker_seg_to_kvm(&state->sregs.fs, &save->fs, VCPU_SREG_FS);
+	svm_caretaker_seg_to_kvm(&state->sregs.gs, &save->gs, VCPU_SREG_GS);
+	svm_caretaker_seg_to_kvm(&state->sregs.ss, &save->ss, VCPU_SREG_SS);
+	svm_caretaker_seg_to_kvm(&state->sregs.tr, &save->tr, VCPU_SREG_TR);
+	svm_caretaker_seg_to_kvm(&state->sregs.ldt, &save->ldtr, VCPU_SREG_LDTR);
+	state->sregs.gdt.base = save->gdtr.base;
+	state->sregs.gdt.limit = save->gdtr.limit;
+	state->sregs.idt.base = save->idtr.base;
+	state->sregs.idt.limit = save->idtr.limit;
+
+	kvm_x86_caretaker_update_msr(state, MSR_STAR, save->star);
+	kvm_x86_caretaker_update_msr(state, MSR_LSTAR, save->lstar);
+	kvm_x86_caretaker_update_msr(state, MSR_CSTAR, save->cstar);
+	kvm_x86_caretaker_update_msr(state, MSR_SYSCALL_MASK, save->sfmask);
+	kvm_x86_caretaker_update_msr(state, MSR_KERNEL_GS_BASE, save->kernel_gs_base);
+	kvm_x86_caretaker_update_msr(state, MSR_IA32_SYSENTER_CS, save->sysenter_cs);
+	kvm_x86_caretaker_update_msr(state, MSR_IA32_SYSENTER_ESP, save->sysenter_esp);
+	kvm_x86_caretaker_update_msr(state, MSR_IA32_SYSENTER_EIP, save->sysenter_eip);
+}
+
+static void svm_caretaker_sync_vcpu(struct kvm_vcpu *vcpu, void *vcpu_data)
+{
+	struct vcpu_svm *svm = to_svm(vcpu);
+	struct svm_cpu_data *sd = per_cpu_ptr(&svm_data, raw_smp_processor_id());
+
+	if (sd && sd->save_area_pa)
+		wrmsrq(MSR_VM_HSAVE_PA, sd->save_area_pa);
+
+	kvm_x86_caretaker_sync_vcpu_common(vcpu);
+
+	if (svm->vmcb) {
+		if (!svm->vmcb->control.next_rip && svm->vmcb->control.insn_len)
+			svm->vmcb->control.next_rip = kvm_rip_read(vcpu) +
+						      svm->vmcb->control.insn_len;
+		if (!svm->vmcb->control.next_rip) {
+			switch (svm->vmcb->control.exit_code) {
+			case SVM_EXIT_CPUID:
+			case SVM_EXIT_MSR:
+			case SVM_EXIT_PAUSE:
+				svm->vmcb->control.next_rip = kvm_rip_read(vcpu) + 2;
+				break;
+			case SVM_EXIT_HLT:
+				svm->vmcb->control.next_rip = kvm_rip_read(vcpu) + 1;
+				break;
+			case SVM_EXIT_VMMCALL:
+			case SVM_EXIT_XSETBV:
+			case SVM_EXIT_INVLPGA:
+				svm->vmcb->control.next_rip = kvm_rip_read(vcpu) + 3;
+				break;
+			default:
+				break;
+			}
+		}
+
+		svm->vmcb->control.clean = 0;
+		svm->vmcb->control.tlb_ctl = TLB_CONTROL_FLUSH_ALL_ASID;
+		svm->vmcb->control.int_ctl &= ~V_IRQ_INJECTION_BITS_MASK;
+		svm->vmcb->control.int_ctl |= V_INTR_MASKING_MASK;
+		if (vgif)
+			svm_set_gif(svm, true);
+		svm_clr_intercept(svm, INTERCEPT_VINTR);
+		svm->vmcb->control.int_state = 0;
+		svm->nmi_masked = false;
+		svm->awaiting_iret_completion = false;
+		svm_set_intercept(svm, INTERCEPT_INTR);
+		svm_set_intercept(svm, INTERCEPT_NMI);
+		svm_set_intercept(svm, INTERCEPT_INIT);
+		svm_clr_intercept(svm, INTERCEPT_RDTSC);
+		svm_clr_intercept(svm, INTERCEPT_PAUSE);
+		svm_recalc_intercepts(vcpu);
+		svm->vmcb->control.event_inj = 0;
+		svm->vmcb->control.exit_int_info = 0;
+	}
+}
+
+void __cpu_preserved_text
+svm_caretaker_decode_exit(void *page,
+			  struct kvm_caretaker_exit *exit)
+{
+	struct caretaker_svm_page *csp = page;
+	u64 exit_code = csp->vmcb.control.exit_code;
+	u64 info1 = csp->vmcb.control.exit_info_1;
+	u64 info2 = csp->vmcb.control.exit_info_2;
+	struct vmcb *vmcb = &csp->vmcb;
+	u64 rip = vmcb->save.rip;
+	u32 insn_len;
+
+	if (vmcb->control.next_rip && vmcb->control.next_rip > rip)
+		insn_len = vmcb->control.next_rip - rip;
+	else
+		insn_len = vmcb->control.insn_len;
+
+	csp->common.last_exit_rip = rip;
+	csp->common.last_exit_rsp = vmcb->save.rsp;
+	csp->common.last_exit_rflags = vmcb->save.rflags;
+	csp->common.cr3 = vmcb->save.cr3;
+	csp->common.cr0 = vmcb->save.cr0;
+	csp->common.cr4 = vmcb->save.cr4;
+	csp->common.efer = vmcb->save.efer;
+	vmcb->control.exit_int_info = 0;
+
+	exit->rip = rip;
+	exit->insn_len = insn_len;
+	exit->raw_reason = exit_code;
+	exit->type = KVM_CARETAKER_EXIT_ARCH;
+
+	switch (exit_code) {
+	case SVM_EXIT_IOIO: {
+		u16 port = (u16)(info1 >> 16);
+
+		if (!exit->insn_len && info2 > rip)
+			exit->insn_len = (u32)(info2 - rip);
+
+		if (port >= COM1_PORT_BASE && port <= COM1_PORT_END) {
+			exit->type = KVM_CARETAKER_EXIT_CONSOLE;
+			exit->mmio_io.addr = port;
+			exit->mmio_io.is_write = !(info1 & SVM_IOIO_TYPE_MASK);
+			exit->mmio_io.size = (u8)((info1 & SVM_IOIO_SIZE_MASK) >>
+						  SVM_IOIO_SIZE_SHIFT);
+			exit->mmio_io.is_mmio = false;
+			exit->mmio_io.val_ptr = &csp->common.rax;
+		}
+		break;
+	}
+	case SVM_EXIT_HLT:
+		exit->type = KVM_CARETAKER_EXIT_IDLE;
+		if (!exit->insn_len)
+			exit->insn_len = 1;
+		break;
+	case SVM_EXIT_PAUSE:
+		exit->type = KVM_CARETAKER_EXIT_IDLE;
+		if (!exit->insn_len)
+			exit->insn_len = 2;
+		break;
+	case SVM_EXIT_CPUID:
+		exit->type = KVM_CARETAKER_EXIT_CPUID;
+		if (!exit->insn_len)
+			exit->insn_len = 2;
+		break;
+	case SVM_EXIT_MSR:
+		exit->type = KVM_CARETAKER_EXIT_MSR;
+		exit->msr.msr = (u32)csp->common.rcx;
+		exit->msr.is_write = (info1 != 0);
+		if (!exit->insn_len)
+			exit->insn_len = 2;
+		break;
+	case SVM_EXIT_INVD:
+	case SVM_EXIT_WBINVD:
+		exit->type = KVM_CARETAKER_EXIT_INSN_STEP;
+		if (!exit->insn_len)
+			exit->insn_len = 2;
+		break;
+	case SVM_EXIT_NPF:
+		exit->type = KVM_CARETAKER_EXIT_UNHANDLED;
+		break;
+	case SVM_EXIT_INTR:
+		kvm_x86_caretaker_disarm_timer();
+		asm volatile("sti\n\tnop\n\tpause\n\tcli" : : : "memory");
+		exit->type = KVM_CARETAKER_EXIT_PREEMPT_TIMER;
+		exit->insn_len = 0;
+		break;
+	case SVM_EXIT_NMI:
+	case SVM_EXIT_INIT:
+		exit->type = KVM_CARETAKER_EXIT_PREEMPT_TIMER;
+		exit->insn_len = 0;
+		break;
+	case SVM_EXIT_VINTR:
+	case SVM_EXIT_WRITE_CR8:
+		exit->type = KVM_CARETAKER_EXIT_CROSS_VCPU;
+		exit->insn_len = 0;
+		break;
+	case SVM_EXIT_VMMCALL:
+		exit->type = KVM_CARETAKER_EXIT_CROSS_VCPU;
+		if (!exit->insn_len)
+			exit->insn_len = 3;
+		break;
+	case SVM_EXIT_RDTSC:
+		exit->type = KVM_CARETAKER_EXIT_RDTSC;
+		if (!exit->insn_len)
+			exit->insn_len = 2;
+		break;
+	default:
+		/*
+		 * Everything not decoded above keeps KVM_CARETAKER_EXIT_ARCH
+		 * and stalls in kvm_x86_caretaker_handle_exit().
+		 *
+		 * XSETBV and RDTSCP used to be decoded here and are
+		 * deliberately not any more.  INSN_STEP means "the instruction
+		 * had no architectural effect worth emulating", which holds
+		 * for INVD and WBINVD on a coherent, guest_memfd-backed VM but
+		 * not for these: stepping XSETBV leaves XCR0 unchanged while
+		 * the guest believes it changed.  RDTSCP is the same story via
+		 * KVM_CARETAKER_EXIT_RDTSC, which writes only RAX and RDX and
+		 * would leave a stale RCX instead of IA32_TSC_AUX.
+		 */
+		break;
+	}
+}
+
+static __cpu_preserved_text void
+svm_caretaker_advance_rip(void *page, u64 rip)
+{
+	struct caretaker_svm_page *csp = page;
+
+	csp->vmcb.save.rax = csp->common.rax;
+	csp->vmcb.save.rip = rip;
+	csp->common.last_exit_rip = rip;
+	csp->common.last_exit_rsp = csp->vmcb.save.rsp;
+}
+
+static __cpu_preserved_text void
+svm_caretaker_arm_timer(void *page, u64 deadline_ticks)
+{
+	struct caretaker_svm_page *csp = page;
+
+	if (deadline_ticks) {
+		vmcb_set_intercept(&csp->vmcb.control, INTERCEPT_INTR);
+		kvm_x86_caretaker_arm_timer(deadline_ticks);
+	} else {
+		vmcb_clr_intercept(&csp->vmcb.control, INTERCEPT_INTR);
+	}
+}
+
+static __cpu_preserved_text void
+svm_caretaker_disarm_timer(void *page)
+{
+	struct caretaker_svm_page *csp = page;
+
+	kvm_x86_caretaker_disarm_timer();
+	vmcb_clr_intercept(&csp->vmcb.control, INTERCEPT_INTR);
+	asm volatile("sti\n\tnop\n\tpause\n\tcli" : : : "memory");
+}
+
+static __cpu_preserved_text void
+svm_caretaker_pre_enter(void *page)
+{
+	struct caretaker_svm_page *csp = page;
+	u64 efer;
+
+	csp->orig_hsave_pa = native_rdmsrq(MSR_VM_HSAVE_PA);
+
+	/* Ensure EFER_SVME is enabled and HSAVE is configured */
+	efer = native_rdmsrq(MSR_EFER);
+	csp->orig_efer = efer;
+	native_wrmsrq(MSR_EFER, efer | EFER_SVME);
+	native_wrmsrq(MSR_VM_HSAVE_PA, csp->hsave_pa);
+}
+
+static __cpu_preserved_text void
+svm_caretaker_post_exit(void *page)
+{
+	struct caretaker_svm_page *csp = page;
+
+	native_wrmsrq(MSR_VM_HSAVE_PA, csp->orig_hsave_pa);
+	asm volatile("1: stgi\n\t"
+		     "2:\n\t"
+		     _ASM_EXTABLE(1b, 2b)
+		     : : : "memory");
+	if (!(csp->orig_efer & EFER_SVME))
+		native_wrmsrq(MSR_EFER, native_rdmsrq(MSR_EFER) & ~EFER_SVME);
+}
+
+static const struct kvm_x86_caretaker_runtime_ops svm_caretaker_runtime_ops __cpu_preserved_data = {
+	.detach_serialize = svm_caretaker_detach_serialize,
+	.common = {
+		.enter_guest = svm_caretaker_enter,
+		.decode_exit = svm_caretaker_decode_exit,
+		.handle_arch_exit = kvm_x86_caretaker_handle_exit,
+		.advance_rip = svm_caretaker_advance_rip,
+		.arm_timer = svm_caretaker_arm_timer,
+		.disarm_timer = svm_caretaker_disarm_timer,
+		.pre_run = svm_caretaker_pre_enter,
+		.post_run = svm_caretaker_post_exit,
+	},
+};
+
+static const struct kvm_x86_caretaker_ops svm_caretaker_ops = {
+	.name = "svm",
+	.init = svm_caretaker_init,
+	.sync_vcpu = svm_caretaker_sync_vcpu,
+	.runtime = &svm_caretaker_runtime_ops,
+};
+
+void svm_caretaker_register(void)
+{
+	kvm_x86_caretaker_register_ops(&svm_caretaker_ops);
+}
+
+void svm_caretaker_unregister(void)
+{
+	kvm_x86_caretaker_unregister_ops(&svm_caretaker_ops);
+}
diff --git a/arch/x86/kvm/svm/caretaker_vmenter.S b/arch/x86/kvm/svm/caretaker_vmenter.S
index e0276a34403e..10303268844e 100644
--- a/arch/x86/kvm/svm/caretaker_vmenter.S
+++ b/arch/x86/kvm/svm/caretaker_vmenter.S
@@ -27,7 +27,7 @@ SYM_TYPED_FUNC_START(svm_caretaker_enter)
 	sti
 
 	/* 0. Save host VMCB state (FS, GS, TR, LDTR) to host save area */
-	movq CXP_HSAVE_PA(%rdi), %rax
+	movq CSP_HSAVE_PA_OFFSET(%rdi), %rax
 	testq %rax, %rax
 	jz 10f
 	vmsave %rax
@@ -69,7 +69,7 @@ SYM_TYPED_FUNC_START(svm_caretaker_enter)
 	movq %rbx, CXP_REG_RAX(%rdi)
 
 	/* 5a. Restore host VMCB state (FS, GS, TR, LDTR) from HSAVE */
-	movq CXP_HSAVE_PA(%rdi), %rax
+	movq CSP_HSAVE_PA_OFFSET(%rdi), %rax
 	testq %rax, %rax
 	jz 40f
 	vmload %rax
diff --git a/arch/x86/kvm/svm/svm.c b/arch/x86/kvm/svm/svm.c
index 7d59d301e1e5..ca6b0d1e80fe 100644
--- a/arch/x86/kvm/svm/svm.c
+++ b/arch/x86/kvm/svm/svm.c
@@ -54,6 +54,7 @@
 #include "svm.h"
 #include "svm_ops.h"
 
+#include "caretaker.h"
 #include "hyperv.h"
 #include "kvm_onhyperv.h"
 #include "svm_onhyperv.h"
@@ -973,6 +974,8 @@ static void svm_hardware_unsetup(void)
 {
 	int cpu;
 
+	svm_caretaker_unregister();
+
 	avic_hardware_unsetup();
 
 	sev_hardware_unsetup();
@@ -1117,7 +1120,7 @@ static void svm_recalc_instruction_intercepts(struct kvm_vcpu *vcpu)
 		svm_clr_intercept(svm, INTERCEPT_RDPMC);
 }
 
-static void svm_recalc_intercepts(struct kvm_vcpu *vcpu)
+void svm_recalc_intercepts(struct kvm_vcpu *vcpu)
 {
 	svm_recalc_instruction_intercepts(vcpu);
 	svm_recalc_msr_intercepts(vcpu);
@@ -4344,6 +4347,7 @@ static void svm_complete_interrupts(struct kvm_vcpu *vcpu)
 	bool nmi_l1_to_l2 = svm->nmi_l1_to_l2;
 	bool soft_int_injected = svm->soft_int_injected;
 
+	svm->vmcb->control.exit_int_info = 0;
 	svm->nmi_l1_to_l2 = false;
 	svm->soft_int_injected = false;
 
@@ -4396,7 +4400,8 @@ static void svm_complete_interrupts(struct kvm_vcpu *vcpu)
 		break;
 	}
 	case SVM_EXITINTINFO_TYPE_INTR:
-		kvm_queue_interrupt(vcpu, vector, false);
+		if (kvm_get_rflags(vcpu) & X86_EFLAGS_IF)
+			kvm_queue_interrupt(vcpu, vector, false);
 		break;
 	case SVM_EXITINTINFO_TYPE_SOFT:
 		kvm_queue_interrupt(vcpu, vector, true);
@@ -5762,6 +5767,8 @@ static __init int svm_hardware_setup(void)
 			goto err;
 	}
 
+	svm_caretaker_register();
+
 	return 0;
 
 err:
-- 
2.55.0.1082.g2b9226bbc0-goog

[RFC PATCH 42/46] KVM: arm64: Add Caretaker arm64 KHO ABI and runtime context headers

[Pasha Tatashin]

Define struct kvm_caretaker_arch_ser (256B), ARM64 Caretaker runtime
context structures, and EL2 exception vector interfaces.

Signed-off-by: Pasha Tatashin <pasha.tatashin@soleen.com>
---
 arch/arm64/include/asm/caretaker.h |  56 ++++++
 arch/arm64/kernel/cpu_preserve.c   | 309 +++++++++++++++++++++++++++++
 arch/arm64/kvm/caretaker.h         | 130 ++++++++++++
 include/linux/kho/abi/kvm_arm64.h  |  32 +++
 4 files changed, 527 insertions(+)
 create mode 100644 arch/arm64/include/asm/caretaker.h
 create mode 100644 arch/arm64/kvm/caretaker.h

diff --git a/arch/arm64/include/asm/caretaker.h b/arch/arm64/include/asm/caretaker.h
new file mode 100644
index 000000000000..7683534a28f8
--- /dev/null
+++ b/arch/arm64/include/asm/caretaker.h
@@ -0,0 +1,56 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright (c) 2026, Google LLC.
+ * Pasha Tatashin <pasha.tatashin@soleen.com>
+ */
+#ifndef __ASM_ARM64_CARETAKER_H
+#define __ASM_ARM64_CARETAKER_H
+
+#include <linux/types.h>
+#include <linux/cpu_preserve.h>
+#include <asm/arch_timer.h>
+#include <asm/cputype.h>
+#include <asm/pgtable-types.h>
+#include <asm/sysreg.h>
+
+#ifdef CONFIG_LIVEUPDATE_CPU
+extern phys_addr_t arm64_caretaker_pgd_pa;
+#else
+#define arm64_caretaker_pgd_pa 0ULL
+#endif
+
+extern char caretaker_hyp_vector[];
+
+extern bool arm64_caretaker_has_ptrauth;
+
+static inline int arm64_caretaker_get_pcpu(void)
+{
+	u64 mpidr = read_cpuid_mpidr() & MPIDR_HWID_BITMASK;
+	int cpu = arch_cpu_preserved_mpidr_to_cpu(mpidr);
+
+	if (cpu >= 0)
+		return cpu;
+	return (int)MPIDR_AFFINITY_LEVEL(mpidr, 0);
+}
+
+#ifdef CONFIG_LIVEUPDATE_CPU
+void gicv3_caretaker_clear_active_priorities(void);
+void gicv3_caretaker_enable_sgi(void);
+void gicv3_caretaker_clear_sgi(void);
+void gicv3_caretaker_kick_cpu(int cpu);
+int gicv3_caretaker_get_redist_region(int idx, phys_addr_t *pa,
+				      unsigned long *va, size_t *size);
+#else
+static inline void gicv3_caretaker_clear_active_priorities(void) {}
+static inline void gicv3_caretaker_enable_sgi(void) {}
+static inline void gicv3_caretaker_clear_sgi(void) {}
+static inline void gicv3_caretaker_kick_cpu(int cpu) {}
+static inline int gicv3_caretaker_get_redist_region(int idx, phys_addr_t *pa,
+						    unsigned long *va,
+						    size_t *size)
+{
+	return -ENOENT;
+}
+#endif
+
+#endif /* __ASM_ARM64_CARETAKER_H */
diff --git a/arch/arm64/kernel/cpu_preserve.c b/arch/arm64/kernel/cpu_preserve.c
index 15de1ca09062..21e12ba6c788 100644
--- a/arch/arm64/kernel/cpu_preserve.c
+++ b/arch/arm64/kernel/cpu_preserve.c
@@ -7,16 +7,23 @@
  */
 #include <linux/arm-smccc.h>
 #include <linux/cpu_preserve.h>
+#include <linux/io.h>
+#include <linux/ioport.h>
 #include <linux/irqchip/arm-gic-v3.h>
 #include <linux/kexec_handover.h>
 #include <linux/kho/abi/cpu.h>
+#include <linux/kvm_host.h>
 #include <linux/mm.h>
+#include <linux/nospec.h>
+#include <linux/of.h>
+#include <linux/of_address.h>
 #include <linux/psci.h>
 #include <linux/sched/mm.h>
 #include <uapi/linux/psci.h>
 
 #include <asm/barrier.h>
 #include <linux/cacheflush.h>
+#include <asm/caretaker.h>
 #include <asm/cpu_ops.h>
 #include <asm/daifflags.h>
 #include <asm/kernel-pgtable.h>
@@ -27,9 +34,40 @@
 #include <asm/trans_pgd.h>
 #include <asm/virt.h>
 
+#define CARETAKER_RWP_TIMEOUT_COUNT	1000000
+#define CARETAKER_SGI_MASK		GENMASK(15, 0)
+#define CARETAKER_HYP_TIMER_PPI		26
+#define CARETAKER_HYP_VIRT_TIMER_PPI	30
+#define GICR_INT_PRIORITY(intid)	(GICR_IPRIORITYR0 + (intid))
+#define CARETAKER_MAX_RDIST_REGIONS	8
+
+#define MPIDR_TO_SGI_AFFINITY(cluster_id, level) \
+	(MPIDR_AFFINITY_LEVEL(cluster_id, level) \
+		<< ICC_SGI1R_AFFINITY_## level ##_SHIFT)
+#define MPIDR_TO_SGI_CLUSTER_ID(mpidr)	((mpidr) & ~0xFUL)
+#define MPIDR_RS(mpidr)			(((mpidr) & 0xf0ULL) >> 4)
+#define MPIDR_TO_SGI_RS(mpidr)		(MPIDR_RS(mpidr) << ICC_SGI1R_RS_SHIFT)
+
+struct caretaker_rdist_region {
+	phys_addr_t	pa;
+	void __iomem	*va;
+	size_t		size;
+};
+
+struct caretaker_gic_state {
+	void __iomem			*cpu_rdist[NR_CPUS];
+	struct caretaker_rdist_region	regions[CARETAKER_MAX_RDIST_REGIONS];
+	int				nr_regions;
+};
+
 static enum arm_smccc_conduit arm64_psci_conduit __cpu_preserved_data;
 phys_addr_t arm64_caretaker_pgd_pa __cpu_preserved_data;
 static u64 arm64_cpu_mpidr[NR_CPUS] __cpu_preserved_data;
+static struct caretaker_gic_state caretaker_gic __cpu_preserved_data;
+__cpu_preserved_data bool arm64_caretaker_has_ptrauth;
+EXPORT_SYMBOL_GPL(arm64_caretaker_has_ptrauth);
+
+static void arm64_caretaker_gic_init(void);
 
 /*
  * Signal or wake up a preserved physical CPU via SEV.
@@ -38,6 +76,7 @@ void __cpu_preserved_text arch_cpu_preserved_kick(int cpu)
 {
 	dsb(ishst);
 	sev();
+	gicv3_caretaker_kick_cpu(cpu);
 	isb();
 }
 
@@ -61,6 +100,261 @@ int __cpu_preserved_text arch_cpu_preserved_mpidr_to_cpu(u64 mpidr)
 }
 EXPORT_SYMBOL_GPL(arch_cpu_preserved_mpidr_to_cpu);
 
+__cpu_preserved_text static void __iomem *gicv3_get_rdist_for_cpu(int cpu)
+{
+	if (cpu < 0) {
+		u64 mpidr = read_sysreg(mpidr_el1) & MPIDR_HWID_BITMASK;
+
+		cpu = arch_cpu_preserved_mpidr_to_cpu(mpidr);
+	}
+	if (cpu >= 0 && cpu < ARRAY_SIZE(caretaker_gic.cpu_rdist))
+		return caretaker_gic.cpu_rdist[cpu];
+	return NULL;
+}
+
+__cpu_preserved_text static inline void
+gicv3_caretaker_wait_for_rwp(void __iomem *base, u32 bit)
+{
+	int count = CARETAKER_RWP_TIMEOUT_COUNT;
+
+	while (count-- > 0) {
+		if (!(readl_relaxed(base + GICR_CTLR) & bit))
+			return;
+		cpu_relax();
+	}
+}
+
+__cpu_preserved_text void gicv3_caretaker_clear_active_priorities(void)
+{
+	u32 ctlr = read_sysreg_s(SYS_ICC_CTLR_EL1);
+	u32 pribits = ((ctlr & ICC_CTLR_EL1_PRI_BITS_MASK) >>
+		       ICC_CTLR_EL1_PRI_BITS_SHIFT) + 1;
+
+	switch (pribits) {
+	case 8:
+	case 7:
+		write_sysreg_s(0, SYS_ICC_AP1R3_EL1);
+		write_sysreg_s(0, SYS_ICC_AP1R2_EL1);
+		fallthrough;
+	case 6:
+		write_sysreg_s(0, SYS_ICC_AP1R1_EL1);
+		fallthrough;
+	case 5:
+	case 4:
+	default:
+		write_sysreg_s(0, SYS_ICC_AP1R0_EL1);
+		break;
+	}
+	isb();
+}
+EXPORT_SYMBOL_GPL(gicv3_caretaker_clear_active_priorities);
+
+__cpu_preserved_text void gicv3_caretaker_enable_sgi(void)
+{
+	void __iomem *ptr = gicv3_get_rdist_for_cpu(-1);
+
+	if (ptr) {
+		void __iomem *rbase = ptr + SZ_64K;
+
+		writel_relaxed(~0U, rbase + GICR_IGROUPR0);
+		writel_relaxed(0, rbase + GICR_IGRPMODR0);
+		writeb_relaxed(0x00, rbase + GICR_INT_PRIORITY(0));
+		writeb_relaxed(0x00, rbase + GICR_INT_PRIORITY(CARETAKER_HYP_TIMER_PPI));
+		writeb_relaxed(0x00, rbase + GICR_INT_PRIORITY(CARETAKER_HYP_VIRT_TIMER_PPI));
+		writel_relaxed(CARETAKER_SGI_MASK |
+			       BIT(CARETAKER_HYP_TIMER_PPI) |
+			       BIT(CARETAKER_HYP_VIRT_TIMER_PPI),
+			       rbase + GICR_ISENABLER0);
+		writel_relaxed(~0U, rbase + GICR_ICACTIVER0);
+		gicv3_caretaker_wait_for_rwp(ptr, GICR_CTLR_RWP);
+	}
+
+	write_sysreg_s(0, SYS_ICC_BPR1_EL1);
+	gicv3_caretaker_clear_active_priorities();
+}
+EXPORT_SYMBOL_GPL(gicv3_caretaker_enable_sgi);
+
+__cpu_preserved_text void gicv3_caretaker_clear_sgi(void)
+{
+	void __iomem *ptr = gicv3_get_rdist_for_cpu(-1);
+
+	if (ptr) {
+		void __iomem *rbase = ptr + SZ_64K;
+
+		writel_relaxed(~0U, rbase + GICR_ICPENDR0);
+		writel_relaxed(~0U, rbase + GICR_ICACTIVER0);
+		gicv3_caretaker_wait_for_rwp(ptr, GICR_CTLR_RWP);
+	}
+}
+EXPORT_SYMBOL_GPL(gicv3_caretaker_clear_sgi);
+
+__cpu_preserved_text void gicv3_caretaker_kick_cpu(int cpu)
+{
+	u64 mpidr, cluster_id, sgi1r;
+	void __iomem *ptr;
+	u16 tlist;
+
+	if (cpu < 0 || cpu >= ARRAY_SIZE(arm64_cpu_mpidr))
+		return;
+
+	cpu = array_index_nospec(cpu, ARRAY_SIZE(arm64_cpu_mpidr));
+	mpidr = arm64_cpu_mpidr[cpu];
+	if ((mpidr & MPIDR_HWID_BITMASK) == (read_sysreg(mpidr_el1) & MPIDR_HWID_BITMASK))
+		return;
+
+	ptr = gicv3_get_rdist_for_cpu(cpu);
+	if (ptr) {
+		void __iomem *sgi_base = ptr + SZ_64K;
+		u32 val = readl_relaxed(ptr + GICR_WAKER);
+
+		if (val & GICR_WAKER_ProcessorSleep) {
+			int count = CARETAKER_RWP_TIMEOUT_COUNT;
+
+			val &= ~GICR_WAKER_ProcessorSleep;
+			writel_relaxed(val, ptr + GICR_WAKER);
+			while (count-- > 0) {
+				val = readl_relaxed(ptr + GICR_WAKER);
+				if (!(val & GICR_WAKER_ChildrenAsleep))
+					break;
+				cpu_relax();
+			}
+		}
+
+		writel_relaxed(~0U, sgi_base + GICR_IGROUPR0);
+		writel_relaxed(0, sgi_base + GICR_IGRPMODR0);
+		writel_relaxed(0, sgi_base + GICR_INT_PRIORITY(0));
+		writel_relaxed(0, sgi_base + GICR_INT_PRIORITY(4));
+		writel_relaxed(0, sgi_base + GICR_INT_PRIORITY(8));
+		writel_relaxed(0, sgi_base + GICR_INT_PRIORITY(12));
+		writel_relaxed(CARETAKER_SGI_MASK | BIT(CARETAKER_HYP_TIMER_PPI),
+			       sgi_base + GICR_ISENABLER0);
+		gicv3_caretaker_wait_for_rwp(ptr, GICR_CTLR_RWP);
+	}
+
+	cluster_id = MPIDR_TO_SGI_CLUSTER_ID(mpidr);
+	tlist = 1 << (mpidr & 0xf);
+
+	dsb(ishst);
+	sgi1r = (MPIDR_TO_SGI_AFFINITY(cluster_id, 3) |
+		 MPIDR_TO_SGI_AFFINITY(cluster_id, 2) |
+		 (0ULL << ICC_SGI1R_SGI_ID_SHIFT) |
+		 MPIDR_TO_SGI_AFFINITY(cluster_id, 1) |
+		 MPIDR_TO_SGI_RS(cluster_id) |
+		 ((u64)tlist << ICC_SGI1R_TARGET_LIST_SHIFT));
+	write_sysreg_s(sgi1r, SYS_ICC_SGI1R_EL1);
+	isb();
+}
+EXPORT_SYMBOL_GPL(gicv3_caretaker_kick_cpu);
+
+static void arm64_add_gicr_region(phys_addr_t pa, size_t size, u64 stride)
+{
+	void __iomem *va, *ptr;
+	size_t map_size;
+	int i;
+
+	for (i = 0; i < caretaker_gic.nr_regions; i++)
+		if (caretaker_gic.regions[i].pa == pa)
+			return;
+
+	if (caretaker_gic.nr_regions >= ARRAY_SIZE(caretaker_gic.regions))
+		return;
+
+	map_size = max_t(size_t, size, nr_cpu_ids * (stride ? : SZ_128K));
+	va = ioremap(pa, map_size);
+	if (!va)
+		return;
+
+	i = caretaker_gic.nr_regions++;
+	caretaker_gic.regions[i].pa = pa;
+	caretaker_gic.regions[i].va = va;
+	caretaker_gic.regions[i].size = map_size;
+
+	ptr = va;
+	do {
+		u64 typer = readq_relaxed(ptr + GICR_TYPER);
+		u32 aff = typer >> 32;
+		int cpu;
+		bool last = !!(typer & GICR_TYPER_LAST);
+
+		for_each_possible_cpu(cpu) {
+			u64 mpidr = cpu_logical_map(cpu);
+			u32 cpu_aff = (MPIDR_AFFINITY_LEVEL(mpidr, 3) << 24) |
+				      (MPIDR_AFFINITY_LEVEL(mpidr, 2) << 16) |
+				      (MPIDR_AFFINITY_LEVEL(mpidr, 1) << 8) |
+				      MPIDR_AFFINITY_LEVEL(mpidr, 0);
+			if (aff == cpu_aff)
+				caretaker_gic.cpu_rdist[cpu] = ptr;
+		}
+
+		if (stride) {
+			ptr += stride;
+		} else {
+			ptr += SZ_64K * 2;
+			if (typer & GICR_TYPER_VLPIS)
+				ptr += SZ_64K * 2;
+		}
+		if (last)
+			break;
+	} while ((ptr - va) < map_size);
+
+	cpu_preserved_clean(&caretaker_gic);
+}
+
+static void arm64_discover_gicr_res(struct resource *res)
+{
+	for (; res; res = res->sibling) {
+		if (res->name && !strcmp(res->name, "GICR"))
+			arm64_add_gicr_region(res->start, resource_size(res), 0);
+		if (res->child)
+			arm64_discover_gicr_res(res->child);
+	}
+}
+
+static void arm64_caretaker_gic_init(void)
+{
+	struct device_node *node;
+
+	if (caretaker_gic.nr_regions > 0 || arch_cpu_preserved_is_active())
+		return;
+
+	node = of_find_compatible_node(NULL, NULL, "arm,gic-v3");
+	if (node) {
+		u32 nr_redist_regions = 1;
+		u64 stride = 0;
+		int i;
+
+		of_property_read_u32(node, "#redistributor-regions",
+				     &nr_redist_regions);
+		of_property_read_u64(node, "redistributor-stride", &stride);
+		for (i = 0; i < nr_redist_regions; i++) {
+			struct resource res;
+
+			if (of_address_to_resource(node, 1 + i, &res) == 0)
+				arm64_add_gicr_region(res.start,
+						      resource_size(&res),
+						      stride);
+		}
+		of_node_put(node);
+	}
+
+	if (caretaker_gic.nr_regions == 0)
+		arm64_discover_gicr_res(&iomem_resource);
+}
+
+int gicv3_caretaker_get_redist_region(int idx, phys_addr_t *pa,
+				      unsigned long *va, size_t *size)
+{
+	arm64_caretaker_gic_init();
+	if (idx < 0 || idx >= caretaker_gic.nr_regions)
+		return -ENOENT;
+
+	*pa = caretaker_gic.regions[idx].pa;
+	*va = (unsigned long)caretaker_gic.regions[idx].va;
+	*size = caretaker_gic.regions[idx].size;
+	return 0;
+}
+EXPORT_SYMBOL_GPL(gicv3_caretaker_get_redist_region);
+
 bool __cpu_preserved_text arch_cpu_preserved_is_active(void)
 {
 	struct cpu_preserved_stack_context *sctx = cpu_preserved_get_stack_context();
@@ -240,6 +534,7 @@ int arch_cpu_preserved_setup_buffer(struct page *text_page,
 	for (i = 0; i < nr_cpu_ids; i++)
 		arm64_cpu_mpidr[i] = cpu_logical_map(i);
 	cpu_preserved_clean(&arm64_cpu_mpidr);
+	arm64_caretaker_gic_init();
 
 	return 0;
 }
@@ -260,11 +555,18 @@ void __cpu_preserved_text arch_cpu_preserved_park_init(int cpu)
 	local_daif_mask();
 	cpu_preserved_inval(&arm64_psci_conduit);
 	cpu_preserved_inval(&arm64_cpu_mpidr);
+	cpu_preserved_inval(&caretaker_gic);
 	if (!pgd_pa) {
 		cpu_preserved_inval(&arm64_caretaker_pgd_pa);
 		pgd_pa = READ_ONCE(arm64_caretaker_pgd_pa);
 	}
 
+#if IS_ENABLED(CONFIG_KVM_CARETAKER)
+	write_sysreg((unsigned long)caretaker_hyp_vector, vbar_el1);
+	write_sysreg_s((unsigned long)caretaker_hyp_vector, SYS_VBAR_EL2);
+	isb();
+#endif
+
 	write_sysreg(0, ttbr0_el1);
 	if (pgd_pa)
 		write_sysreg(pgd_pa, ttbr1_el1);
@@ -272,6 +574,8 @@ void __cpu_preserved_text arch_cpu_preserved_park_init(int cpu)
 	arm64_flush_host_tlb_local();
 
 	write_sysreg_s(0xff, SYS_ICC_PMR_EL1);
+	isb();
+
 	write_sysreg_s(1, SYS_ICC_IGRPEN1_EL1);
 	isb();
 }
@@ -283,12 +587,17 @@ void arch_cpu_preserved_early_init(void)
 	for (c = 0; c < ARRAY_SIZE(arm64_cpu_mpidr); c++)
 		arm64_cpu_mpidr[c] = cpu_logical_map(c);
 	cpu_preserved_clean(&arm64_cpu_mpidr);
+	arm64_caretaker_gic_init();
 
 	cpu_preserved_inval(&arm64_psci_conduit);
 	if (arm64_psci_conduit == SMCCC_CONDUIT_NONE) {
 		arm64_psci_conduit = arm_smccc_1_1_get_conduit();
 		cpu_preserved_clean(&arm64_psci_conduit);
 	}
+
+	arm64_caretaker_has_ptrauth = IS_ENABLED(CONFIG_ARM64_PTR_AUTH) &&
+				      system_has_full_ptr_auth();
+	cpu_preserved_clean(&arm64_caretaker_has_ptrauth);
 }
 EXPORT_SYMBOL_GPL(arch_cpu_preserved_early_init);
 
diff --git a/arch/arm64/kvm/caretaker.h b/arch/arm64/kvm/caretaker.h
new file mode 100644
index 000000000000..97271a49f5a8
--- /dev/null
+++ b/arch/arm64/kvm/caretaker.h
@@ -0,0 +1,130 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/*
+ * Copyright (c) 2026, Google LLC.
+ * Pasha Tatashin <pasha.tatashin@soleen.com>
+ *
+ * Header for ARM64 KVM Caretaker execution engine and helpers.
+ */
+#ifndef __ARCH_ARM64_KVM_CARETAKER_H
+#define __ARCH_ARM64_KVM_CARETAKER_H
+
+#define CAP_FAULT_ESR		0x00
+#define CAP_FAULT_FAR		0x08
+#define CAP_FAULT_HPFAR		0x10
+#define CAP_CTXT_OFFSET		0x20
+
+/* Special INTID range (1020-1023) reserved by GIC architecture */
+#define GIC_SPECIAL_INTID_START	1020
+
+#ifndef __ASSEMBLY__
+
+#include <linux/types.h>
+#include <linux/refcount.h>
+#include <linux/sizes.h>
+#include <linux/oncore.h>
+#include <linux/kvm_host.h>
+#include <linux/kho/abi/kvm_arm64.h>
+#include <linux/kvm_caretaker.h>
+#include <linux/irqchip/arm-gic-v3.h>
+#include <asm/esr.h>
+
+/* Default priority for software-injected SGIs */
+#define GIC_DEFAULT_SGI_PRIO		0xa0ULL
+
+/* System register opcodes for SGI generation */
+#define ESR_ELx_SYS64_ISS_SYS_ICC_SGI1R_EL1 \
+	(ESR_ELx_SYS64_ISS_SYS_VAL(3, 0, 5, 12, 11) | ESR_ELx_SYS64_ISS_DIR_WRITE)
+#define ESR_ELx_SYS64_ISS_SYS_ICC_ASGI1R_EL1 \
+	(ESR_ELx_SYS64_ISS_SYS_VAL(3, 0, 6, 12, 11) | ESR_ELx_SYS64_ISS_DIR_WRITE)
+#define ESR_ELx_SYS64_ISS_SYS_ICC_SGI0R_EL1 \
+	(ESR_ELx_SYS64_ISS_SYS_VAL(3, 0, 7, 12, 11) | ESR_ELx_SYS64_ISS_DIR_WRITE)
+
+/**
+ * struct caretaker_arm64_context - Preserved ARM64 vCPU architectural context
+ * @fault:            Guest fault syndrome registers from VM exits.
+ * @ctxt:             CPU registers, FP/SIMD, and system registers.
+ * @hcr_el2:          Hypervisor Configuration Register.
+ * @mdcr_el2:         Monitor Debug Configuration Register.
+ * @cflags:           vCPU execution flags.
+ * @vtcr_el2:         Stage-2 translation control register.
+ * @vttbr_el2:        Stage-2 translation table base register.
+ * @vgic_initialized: Whether VGICv3 virtual CPU interface is active.
+ * @vgic_v3:          VGICv3 virtual CPU interface registers.
+ */
+struct caretaker_arm64_context {
+	struct kvm_vcpu_fault_info fault;
+	struct kvm_cpu_context ctxt;
+
+	u64 hcr_el2;
+	u64 mdcr_el2;
+	u64 cflags;
+
+	u64 vtcr_el2;
+	u64 vttbr_el2;
+
+	bool vgic_initialized;
+	struct vgic_v3_cpu_if vgic_v3;
+
+	u64 cntv_cval_el0;
+	u64 cntv_ctl_el0;
+	u64 cntvoff_el2;
+	u16 pending_sgis;
+};
+
+struct caretaker_arm64_page;
+
+struct arm64_caretaker_ptrauth_keys {
+	u64 apia_lo, apia_hi;
+	u64 apib_lo, apib_hi;
+	u64 apda_lo, apda_hi;
+	u64 apdb_lo, apdb_hi;
+	u64 apga_lo, apga_hi;
+};
+
+/**
+ * struct caretaker_arm64_page - Old-text-private ARM64 Caretaker execution page
+ *
+ * Invariant: Only the embedded @abi prefix (struct kvm_caretaker_arch_ser,
+ * offset 0) is part of the KHO handover ABI and may be dereferenced
+ * by the incoming kernel.  All remaining fields are private to preserved
+ * Caretaker text executing on the isolated core.
+ */
+struct caretaker_arm64_page {
+	struct kvm_caretaker_arch_ser abi;
+	struct kvm_caretaker_vcpu vcpu;
+	struct caretaker_arm64_context ctx;
+	struct caretaker_arm64_page *next_vcpu;
+	struct kvm_vcpu_arch_ser *arch_state;
+	u64 last_ret;
+	struct arm64_caretaker_ptrauth_keys ptrauth_keys;
+};
+
+static_assert(offsetof(struct caretaker_arm64_page, abi) == 0);
+static_assert(offsetof(struct caretaker_arm64_page, abi.cb) == 0);
+
+u64 caretaker_guest_enter(struct caretaker_arm64_context *ctx);
+void kvm_arch_vcpu_luo_pre_retrieve_caretaker(struct kvm_vcpu *vcpu,
+					      struct kvm_vcpu_ser *ser);
+void kvm_arch_vcpu_luo_attach_caretaker(struct kvm_vcpu *vcpu,
+					struct kvm_vcpu_ser *ser);
+void arm64_caretaker_handle_invalid(u64 elr, u64 esr, u64 far);
+
+#ifdef CONFIG_KVM_CARETAKER
+int arm64_kvm_caretaker_preserve(struct kvm_vcpu *vcpu,
+				 struct kvm_vcpu_ser *ser);
+void arm64_kvm_caretaker_unpreserve(struct kvm_vcpu_ser *ser);
+void arm64_kvm_caretaker_finish(struct kvm_vcpu_ser *ser);
+#else
+static inline int arm64_kvm_caretaker_preserve(struct kvm_vcpu *vcpu,
+					       struct kvm_vcpu_ser *ser)
+{
+	return -EOPNOTSUPP;
+}
+
+static inline void arm64_kvm_caretaker_unpreserve(struct kvm_vcpu_ser *ser) {}
+static inline void arm64_kvm_caretaker_finish(struct kvm_vcpu_ser *ser) {}
+#endif
+
+#endif /* !__ASSEMBLY__ */
+
+#endif /* __ARCH_ARM64_KVM_CARETAKER_H */
diff --git a/include/linux/kho/abi/kvm_arm64.h b/include/linux/kho/abi/kvm_arm64.h
index dc121224a177..4aee91dd41b3 100644
--- a/include/linux/kho/abi/kvm_arm64.h
+++ b/include/linux/kho/abi/kvm_arm64.h
@@ -56,6 +56,38 @@ struct kvm_vcpu_arch_ser {
 static_assert(offsetof(struct kvm_vcpu_arch_ser, sysregs) % sizeof(u64) == 0,
 	      "sysregs must be 64-bit aligned");
 
+/**
+ * struct kvm_caretaker_arch_ser - ARM64-specific Caretaker control block ABI
+ * @cb:               Common Caretaker control block header (must be at offset 0).
+ * @vgic_initialized: Non-zero if @vgic_* fields hold live VGICv3 CPU interface state.
+ * @cflags:           KVM vCPU architectural flags.
+ * @cntvoff_el2:      Guest virtual counter offset active during Caretaker execution.
+ * @hcr_el2:          Hypervisor Configuration Register active during Caretaker execution.
+ * @mdcr_el2:         Monitor Debug Configuration Register active during Caretaker execution.
+ * @used_lrs:         Number of active VGICv3 List Registers.
+ * @vgic_hcr:         VGICv3 Hypervisor Control Register.
+ * @vgic_vmcr:        VGICv3 Virtual Machine Control Register.
+ * @reserved:         Must be zero.
+ * @vgic_ap0r:        VGICv3 Active Priorities Group 0 Registers.
+ * @vgic_ap1r:        VGICv3 Active Priorities Group 1 Registers.
+ * @vgic_lr:          VGICv3 List Registers.
+ */
+struct kvm_caretaker_arch_ser {
+	struct kvm_caretaker_cb_ser cb;
+	u32 vgic_initialized;
+	u32 cflags;
+	u64 cntvoff_el2;
+	u64 hcr_el2;
+	u64 mdcr_el2;
+	u32 used_lrs;
+	u32 vgic_hcr;
+	u32 vgic_vmcr;
+	u32 reserved;
+	u32 vgic_ap0r[4];
+	u32 vgic_ap1r[4];
+	u64 vgic_lr[16];
+} __packed;
+
 #endif /* CONFIG_ARM64 */
 
 #endif /* _LINUX_KHO_ABI_KVM_ARM64_H */
-- 
2.55.0.1082.g2b9226bbc0-goog

[RFC PATCH 43/46] KVM: arm64: Add Caretaker EL2 exception vectors and guest entry/exit assembly

[Pasha Tatashin]

Add arch/arm64/kvm/caretaker_vmenter.S for standalone VHE EL2 guest
entry/exit world switch and exception vector dispatch on preserved cores.

Signed-off-by: Pasha Tatashin <pasha.tatashin@soleen.com>
---
 arch/arm64/kvm/caretaker_vmenter.S | 248 +++++++++++++++++++++++++++++
 1 file changed, 248 insertions(+)
 create mode 100644 arch/arm64/kvm/caretaker_vmenter.S

diff --git a/arch/arm64/kvm/caretaker_vmenter.S b/arch/arm64/kvm/caretaker_vmenter.S
new file mode 100644
index 000000000000..6dff9fbe931f
--- /dev/null
+++ b/arch/arm64/kvm/caretaker_vmenter.S
@@ -0,0 +1,248 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/*
+ * Caretaker low-level guest entry/exit and vector table for ARM64 VHE.
+ *
+ * Copyright (c) 2026, Google LLC.
+ * Pasha Tatashin <pasha.tatashin@soleen.com>
+ */
+
+#include <linux/linkage.h>
+#include <asm/assembler.h>
+#include <asm/kvm_arm.h>
+#include <asm/kvm_asm.h>
+#include <asm/sysreg.h>
+#include "caretaker.h"
+
+	.pushsection ".text.cpu_preserved", "ax"
+
+.macro cventry target
+	.align 7
+	stp	x0, x1, [sp, #-16]!
+	b	\target
+.endm
+
+	.align 11
+SYM_CODE_START(caretaker_hyp_vector)
+	/* Current EL with SP0 */
+	cventry	caretaker_invalid
+	cventry	caretaker_el2h_irq
+	cventry	caretaker_el2h_irq
+	cventry	caretaker_invalid
+
+	/* Current EL with SPx (EL2h) */
+	cventry	caretaker_invalid
+	cventry	caretaker_el2h_irq
+	cventry	caretaker_el2h_irq
+	cventry	caretaker_invalid
+
+	/* Lower EL using AArch64 (Guest EL1/EL0) */
+	cventry	caretaker_el1_sync
+	cventry	caretaker_el1_irq
+	cventry	caretaker_el1_fiq
+	cventry	caretaker_el1_error
+
+	/* Lower EL using AArch32 */
+	cventry	caretaker_el1_sync
+	cventry	caretaker_el1_irq
+	cventry	caretaker_el1_fiq
+	cventry	caretaker_el1_error
+SYM_CODE_END(caretaker_hyp_vector)
+
+SYM_CODE_START_LOCAL(caretaker_invalid)
+	mrs	x0, elr_el2
+	mrs	x1, esr_el2
+	mrs	x2, far_el2
+	b	arm64_caretaker_handle_invalid
+SYM_CODE_END(caretaker_invalid)
+
+SYM_CODE_START_LOCAL(caretaker_el2h_irq)
+	stp	x2, x3,   [sp, #-16]!
+	stp	x4, x5,   [sp, #-16]!
+	stp	x6, x7,   [sp, #-16]!
+	stp	x8, x9,   [sp, #-16]!
+	stp	x10, x11, [sp, #-16]!
+	stp	x12, x13, [sp, #-16]!
+	stp	x14, x15, [sp, #-16]!
+	stp	x16, x17, [sp, #-16]!
+	stp	x29, x30, [sp, #-16]!
+
+	mrs_s	x0, SYS_ICC_IAR1_EL1
+	cmp	x0, #GIC_SPECIAL_INTID_START
+	b.ge	1f
+	msr_s	SYS_ICC_EOIR1_EL1, x0
+	msr_s	SYS_ICC_DIR_EL1, x0
+1:
+	/*
+	 * Silence the EL2 physical timer before returning.
+	 *
+	 * It is the only EL2 interrupt source the Caretaker arms
+	 * (arm64_caretaker_op_arm_timer() writes CNTHP_CTL_EL2.ENABLE to
+	 * drive the on-core quantum), and it is level-triggered off
+	 * CNTHP_CTL_EL2.ISTATUS.  Neither EOIR1 nor DIR clears ISTATUS, so
+	 * without disabling ENABLE here the line re-asserts the instant we
+	 * eret and the core live-locks taking the same interrupt forever --
+	 * it never reaches the run loop, which means it can never observe a
+	 * detach request and can never be reclaimed.
+	 *
+	 * This is done on the special-INTID path too: a spurious read of
+	 * IAR1 does not mean the timer has stopped asserting.  The run loop
+	 * re-arms the timer for each new quantum, so dropping it here costs
+	 * at most the remainder of the current one.
+	 */
+	msr_s	SYS_CNTHP_CTL_EL2, xzr
+	isb
+
+	ldp	x29, x30, [sp], #16
+	ldp	x16, x17, [sp], #16
+	ldp	x14, x15, [sp], #16
+	ldp	x12, x13, [sp], #16
+	ldp	x10, x11, [sp], #16
+	ldp	x8, x9,   [sp], #16
+	ldp	x6, x7,   [sp], #16
+	ldp	x4, x5,   [sp], #16
+	ldp	x2, x3,   [sp], #16
+	ldp	x0, x1,   [sp], #16
+	eret
+	sb
+SYM_CODE_END(caretaker_el2h_irq)
+
+SYM_CODE_START_LOCAL(caretaker_el1_sync)
+	mov	x0, #ARM_EXCEPTION_TRAP
+	b	caretaker_guest_exit
+SYM_CODE_END(caretaker_el1_sync)
+
+SYM_CODE_START_LOCAL(caretaker_el1_irq)
+SYM_CODE_START_LOCAL(caretaker_el1_fiq)
+	mov	x0, #ARM_EXCEPTION_IRQ
+	b	caretaker_guest_exit
+SYM_CODE_END(caretaker_el1_fiq)
+SYM_CODE_END(caretaker_el1_irq)
+
+SYM_CODE_START_LOCAL(caretaker_el1_error)
+	mov	x0, #ARM_EXCEPTION_EL1_SERROR
+	b	caretaker_guest_exit
+SYM_CODE_END(caretaker_el1_error)
+
+/*
+ * u64 caretaker_guest_enter(struct kvm_vcpu *vcpu);
+ *   x0: vcpu pointer
+ * Returns exit code in x0.
+ */
+SYM_FUNC_START(caretaker_guest_enter)
+	// Save host callee-saved registers on stack
+	stp	x19, x20, [sp, #-112]!
+	stp	x21, x22, [sp, #16]
+	stp	x23, x24, [sp, #32]
+	stp	x25, x26, [sp, #48]
+	stp	x27, x28, [sp, #64]
+	stp	x29, x30, [sp, #80]
+	str	x0, [sp, #96]			// Save vcpu pointer on host stack
+	mrs	x1, sp_el0
+	str	x1, [sp, #104]			// Save host sp_el0 (current task)
+
+	// Install caretaker vectors
+	adr_l	x1, caretaker_hyp_vector
+	msr	vbar_el1, x1
+	msr	vbar_el2, x1
+	isb
+
+	// Restore guest context from ctx->ctxt
+	add	x29, x0, #CAP_CTXT_OFFSET
+
+	ldr	x1, [x29, #CPU_XREG_OFFSET(32)]	// regs.pc
+	msr	elr_el2, x1
+	ldr	x1, [x29, #CPU_XREG_OFFSET(33)]	// regs.pstate
+	msr	spsr_el2, x1
+	ldr	x1, [x29, #CPU_XREG_OFFSET(31)]	// regs.sp
+	msr	sp_el0, x1
+
+	// Restore guest GPRs x0-x28, lr (x30)
+	ldp	x0, x1, [x29, #CPU_XREG_OFFSET(0)]
+	ldp	x2, x3, [x29, #CPU_XREG_OFFSET(2)]
+	ldp	x4, x5, [x29, #CPU_XREG_OFFSET(4)]
+	ldp	x6, x7, [x29, #CPU_XREG_OFFSET(6)]
+	ldp	x8, x9, [x29, #CPU_XREG_OFFSET(8)]
+	ldp	x10, x11, [x29, #CPU_XREG_OFFSET(10)]
+	ldp	x12, x13, [x29, #CPU_XREG_OFFSET(12)]
+	ldp	x14, x15, [x29, #CPU_XREG_OFFSET(14)]
+	ldp	x16, x17, [x29, #CPU_XREG_OFFSET(16)]
+	ldp	x18, x19, [x29, #CPU_XREG_OFFSET(18)]
+	ldp	x20, x21, [x29, #CPU_XREG_OFFSET(20)]
+	ldp	x22, x23, [x29, #CPU_XREG_OFFSET(22)]
+	ldp	x24, x25, [x29, #CPU_XREG_OFFSET(24)]
+	ldp	x26, x27, [x29, #CPU_XREG_OFFSET(26)]
+	ldr	x30,      [x29, #CPU_XREG_OFFSET(30)]
+	ldp	x28, x29, [x29, #CPU_XREG_OFFSET(28)]
+
+	eret
+	sb
+SYM_FUNC_END(caretaker_guest_enter)
+
+SYM_CODE_START_LOCAL(caretaker_guest_exit)
+	// Save exception code in x0 temporarily
+	str	x0, [sp, #-16]!
+
+	// Retrieve ctx pointer from host stack
+	ldr	x0, [sp, #128]
+	add	x1, x0, #CAP_CTXT_OFFSET
+
+	// Save guest GPRs x2-x29, lr (x30)
+	stp	x2, x3,   [x1, #CPU_XREG_OFFSET(2)]
+	stp	x4, x5,   [x1, #CPU_XREG_OFFSET(4)]
+	stp	x6, x7,   [x1, #CPU_XREG_OFFSET(6)]
+	stp	x8, x9,   [x1, #CPU_XREG_OFFSET(8)]
+	stp	x10, x11, [x1, #CPU_XREG_OFFSET(10)]
+	stp	x12, x13, [x1, #CPU_XREG_OFFSET(12)]
+	stp	x14, x15, [x1, #CPU_XREG_OFFSET(14)]
+	stp	x16, x17, [x1, #CPU_XREG_OFFSET(16)]
+	stp	x18, x19, [x1, #CPU_XREG_OFFSET(18)]
+	stp	x20, x21, [x1, #CPU_XREG_OFFSET(20)]
+	stp	x22, x23, [x1, #CPU_XREG_OFFSET(22)]
+	stp	x24, x25, [x1, #CPU_XREG_OFFSET(24)]
+	stp	x26, x27, [x1, #CPU_XREG_OFFSET(26)]
+	stp	x28, x29, [x1, #CPU_XREG_OFFSET(28)]
+	str	x30,      [x1, #CPU_XREG_OFFSET(30)]
+
+	// Clear Branch History Buffer (Spectre-BHB mitigation) across EL1->EL2 exit
+	mov	x2, #32
+1:	b	2f
+2:	subs	x2, x2, #1
+	b.ne	1b
+	isb
+
+	// Retrieve and save guest x0, x1 (from vector entry on stack)
+	ldp	x2, x3, [sp, #16]
+	stp	x2, x3, [x1, #CPU_XREG_OFFSET(0)]
+
+	// Save guest system registers
+	mrs	x2, elr_el2
+	str	x2, [x1, #CPU_XREG_OFFSET(32)]	// regs.pc
+	mrs	x2, spsr_el2
+	str	x2, [x1, #CPU_XREG_OFFSET(33)]	// regs.pstate
+	mrs	x2, sp_el0
+	str	x2, [x1, #CPU_XREG_OFFSET(31)]	// regs.sp
+	mrs	x2, esr_el2
+	str	x2, [x0, #CAP_FAULT_ESR]
+	mrs	x2, far_el2
+	str	x2, [x0, #CAP_FAULT_FAR]
+	mrs	x2, hpfar_el2
+	str	x2, [x0, #CAP_FAULT_HPFAR]
+
+	// Restore return value (exception code) in x0
+	ldr	x0, [sp], #32			// pops exception code + saved x0,x1
+
+	// Restore host callee-saved registers
+	ldp	x19, x20, [sp, #0]
+	ldp	x21, x22, [sp, #16]
+	ldp	x23, x24, [sp, #32]
+	ldp	x25, x26, [sp, #48]
+	ldp	x27, x28, [sp, #64]
+	ldp	x29, x30, [sp, #80]
+	ldr	x1, [sp, #104]
+	msr	sp_el0, x1
+	add	sp, sp, #112
+
+	ret
+SYM_CODE_END(caretaker_guest_exit)
+
+	.popsection
-- 
2.55.0.1082.g2b9226bbc0-goog

[RFC PATCH 44/46] KVM: arm64: Implement Caretaker GICv3 CPU interface and arch timer emulation

[Pasha Tatashin]

Implement standalone Caretaker GICv3 ICH_* register save/restore and
virtual timer deadline emulation in arch/arm64/kvm/caretaker.c.

Signed-off-by: Pasha Tatashin <pasha.tatashin@soleen.com>
---
 arch/arm64/kvm/caretaker.c | 521 +++++++++++++++++++++++++++++++++++++
 1 file changed, 521 insertions(+)
 create mode 100644 arch/arm64/kvm/caretaker.c

diff --git a/arch/arm64/kvm/caretaker.c b/arch/arm64/kvm/caretaker.c
new file mode 100644
index 000000000000..65c11fe8c2fa
--- /dev/null
+++ b/arch/arm64/kvm/caretaker.c
@@ -0,0 +1,521 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * Copyright (c) 2026, Google LLC.
+ * Pasha Tatashin <pasha.tatashin@soleen.com>
+ */
+#include <linux/cpu_preserve.h>
+#include <linux/delay.h>
+#include <linux/irqchip/arm-gic-v3.h>
+#include <linux/kexec_handover.h>
+#include <linux/kho/abi/kvm.h>
+#include <linux/kvm_host.h>
+#include <linux/objtool.h>
+#include <linux/oncore.h>
+#include <linux/sched.h>
+
+#include <asm/barrier.h>
+#include <asm/caretaker.h>
+#include <asm/cpu_ops.h>
+#include <linux/cpufeature.h>
+#include <asm/cputype.h>
+#include <asm/fpsimd.h>
+#include <asm/kernel-pgtable.h>
+#include <linux/kexec.h>
+#include <asm/kvm_emulate.h>
+#include <asm/kvm_hyp.h>
+#include <asm/kvm_mmu.h>
+#include <asm/kvm_pgtable.h>
+#include <asm/kvm_ptrauth.h>
+#include <linux/mmu_context.h>
+#include <linux/pgtable.h>
+#include <asm/smp_plat.h>
+#include <asm/sysreg.h>
+#include <asm/tlbflush.h>
+#include <asm/vectors.h>
+
+#include <kvm/arm_arch_timer.h>
+#include <kvm/arm_vgic.h>
+
+#include "caretaker.h"
+
+struct caretaker_fault_info {
+	u64 elr;
+	u64 esr;
+	u64 far;
+	u64 count;
+};
+
+static struct caretaker_fault_info arm64_caretaker_faults[NR_CPUS] __cpu_preserved_data;
+
+void __caretaker_text
+arm64_caretaker_handle_invalid(u64 elr, u64 esr, u64 far)
+{
+	int cpu = arm64_caretaker_get_pcpu();
+
+	if (cpu >= 0 && cpu < ARRAY_SIZE(arm64_caretaker_faults)) {
+		arm64_caretaker_faults[cpu].elr = elr;
+		arm64_caretaker_faults[cpu].esr = esr;
+		arm64_caretaker_faults[cpu].far = far;
+		arm64_caretaker_faults[cpu].count++;
+	}
+
+	while (1) {
+		if (cpu_preserved_should_exit(cpu)) {
+			cpu_preserved_set_dead(cpu);
+			arch_cpu_preserved_park_finish(cpu);
+		}
+		arch_cpu_preserved_park_wait();
+	}
+}
+
+__caretaker_text static inline void
+arm64_caretaker_load_sysregs(struct kvm_cpu_context *ctxt)
+{
+	u64 mpidr = ctxt_sys_reg(ctxt, MPIDR_EL1);
+	u64 midr = read_cpuid_id();
+
+	write_sysreg(midr, vpidr_el2);
+	write_sysreg(mpidr, vmpidr_el2);
+
+	write_sysreg_el1(ctxt_sys_reg(ctxt, SCTLR_EL1), SYS_SCTLR);
+	write_sysreg_el1(ctxt_sys_reg(ctxt, CPACR_EL1), SYS_CPACR);
+	write_sysreg_el1(ctxt_sys_reg(ctxt, TTBR0_EL1), SYS_TTBR0);
+	write_sysreg_el1(ctxt_sys_reg(ctxt, TTBR1_EL1), SYS_TTBR1);
+	write_sysreg_el1(ctxt_sys_reg(ctxt, TCR_EL1), SYS_TCR);
+	if (cpus_have_final_cap(ARM64_HAS_TCR2)) {
+		write_sysreg_el1(ctxt_sys_reg(ctxt, TCR2_EL1), SYS_TCR2);
+		if (cpus_have_final_cap(ARM64_HAS_S1PIE)) {
+			write_sysreg_el1(ctxt_sys_reg(ctxt, PIR_EL1), SYS_PIR);
+			write_sysreg_el1(ctxt_sys_reg(ctxt, PIRE0_EL1), SYS_PIRE0);
+		}
+	}
+	if (cpus_have_final_cap(ARM64_HAS_SCTLR2))
+		write_sysreg_el1(ctxt_sys_reg(ctxt, SCTLR2_EL1), SYS_SCTLR2);
+	write_sysreg_el1(ctxt_sys_reg(ctxt, ESR_EL1), SYS_ESR);
+	write_sysreg_el1(ctxt_sys_reg(ctxt, AFSR0_EL1), SYS_AFSR0);
+	write_sysreg_el1(ctxt_sys_reg(ctxt, AFSR1_EL1), SYS_AFSR1);
+	write_sysreg_el1(ctxt_sys_reg(ctxt, FAR_EL1), SYS_FAR);
+	write_sysreg_el1(ctxt_sys_reg(ctxt, MAIR_EL1), SYS_MAIR);
+	write_sysreg_el1(ctxt_sys_reg(ctxt, VBAR_EL1), SYS_VBAR);
+	write_sysreg_el1(ctxt_sys_reg(ctxt, CONTEXTIDR_EL1), SYS_CONTEXTIDR);
+	write_sysreg_el1(ctxt_sys_reg(ctxt, AMAIR_EL1), SYS_AMAIR);
+	write_sysreg_el1(ctxt_sys_reg(ctxt, CNTKCTL_EL1), SYS_CNTKCTL);
+	write_sysreg(ctxt_sys_reg(ctxt, PAR_EL1), par_el1);
+	write_sysreg(ctxt_sys_reg(ctxt, TPIDR_EL1), tpidr_el1);
+	write_sysreg(ctxt_sys_reg(ctxt, TPIDR_EL0), tpidr_el0);
+	write_sysreg(ctxt_sys_reg(ctxt, TPIDRRO_EL0), tpidrro_el0);
+	write_sysreg(ctxt_sys_reg(ctxt, SP_EL1), sp_el1);
+	write_sysreg_el1(ctxt_sys_reg(ctxt, ELR_EL1), SYS_ELR);
+	write_sysreg_el1(ctxt_sys_reg(ctxt, SPSR_EL1), SYS_SPSR);
+	write_sysreg(ctxt_sys_reg(ctxt, MDSCR_EL1), mdscr_el1);
+}
+
+__caretaker_text static inline void
+arm64_caretaker_save_sysregs(struct kvm_cpu_context *ctxt)
+{
+	ctxt_sys_reg(ctxt, SCTLR_EL1) = read_sysreg_el1(SYS_SCTLR);
+	ctxt_sys_reg(ctxt, CPACR_EL1) = read_sysreg_el1(SYS_CPACR);
+	ctxt_sys_reg(ctxt, TTBR0_EL1) = read_sysreg_el1(SYS_TTBR0);
+	ctxt_sys_reg(ctxt, TTBR1_EL1) = read_sysreg_el1(SYS_TTBR1);
+	ctxt_sys_reg(ctxt, TCR_EL1) = read_sysreg_el1(SYS_TCR);
+	if (cpus_have_final_cap(ARM64_HAS_TCR2)) {
+		ctxt_sys_reg(ctxt, TCR2_EL1) = read_sysreg_el1(SYS_TCR2);
+		if (cpus_have_final_cap(ARM64_HAS_S1PIE)) {
+			ctxt_sys_reg(ctxt, PIR_EL1) = read_sysreg_el1(SYS_PIR);
+			ctxt_sys_reg(ctxt, PIRE0_EL1) = read_sysreg_el1(SYS_PIRE0);
+		}
+	}
+	if (cpus_have_final_cap(ARM64_HAS_SCTLR2))
+		ctxt_sys_reg(ctxt, SCTLR2_EL1) = read_sysreg_el1(SYS_SCTLR2);
+	ctxt_sys_reg(ctxt, ESR_EL1) = read_sysreg_el1(SYS_ESR);
+	ctxt_sys_reg(ctxt, AFSR0_EL1) = read_sysreg_el1(SYS_AFSR0);
+	ctxt_sys_reg(ctxt, AFSR1_EL1) = read_sysreg_el1(SYS_AFSR1);
+	ctxt_sys_reg(ctxt, FAR_EL1) = read_sysreg_el1(SYS_FAR);
+	ctxt_sys_reg(ctxt, MAIR_EL1) = read_sysreg_el1(SYS_MAIR);
+	ctxt_sys_reg(ctxt, VBAR_EL1) = read_sysreg_el1(SYS_VBAR);
+	ctxt_sys_reg(ctxt, CONTEXTIDR_EL1) = read_sysreg_el1(SYS_CONTEXTIDR);
+	ctxt_sys_reg(ctxt, AMAIR_EL1) = read_sysreg_el1(SYS_AMAIR);
+	ctxt_sys_reg(ctxt, CNTKCTL_EL1) = read_sysreg_el1(SYS_CNTKCTL);
+	ctxt_sys_reg(ctxt, PAR_EL1) = read_sysreg_par();
+	ctxt_sys_reg(ctxt, TPIDR_EL1) = read_sysreg(tpidr_el1);
+	ctxt_sys_reg(ctxt, TPIDR_EL0) = read_sysreg(tpidr_el0);
+	ctxt_sys_reg(ctxt, TPIDRRO_EL0) = read_sysreg(tpidrro_el0);
+	ctxt_sys_reg(ctxt, SP_EL1) = read_sysreg(sp_el1);
+	ctxt_sys_reg(ctxt, ELR_EL1) = read_sysreg_el1(SYS_ELR);
+	ctxt_sys_reg(ctxt, SPSR_EL1) = read_sysreg_el1(SYS_SPSR);
+	ctxt_sys_reg(ctxt, MDSCR_EL1) = read_sysreg(mdscr_el1);
+}
+
+int arm64_kvm_caretaker_preserve(struct kvm_vcpu *vcpu,
+				 struct kvm_vcpu_ser *ser)
+{
+	struct kvm_s2_mmu *mmu = vcpu->arch.hw_mmu ?
+				 vcpu->arch.hw_mmu : &vcpu->kvm->arch.mmu;
+	struct oncore_session *sess = oncore_job_session(vcpu->caretaker.job);
+	struct arch_timer_context *vtimer;
+	struct caretaker_arm64_page *head;
+	struct caretaker_arm64_page *cap;
+
+	if (!has_vhe() || is_protected_kvm_enabled() || vcpu_has_nv(vcpu))
+		return -EOPNOTSUPP;
+	/*
+	 * Same predicate as kvm_arch_vcpu_luo_preserve() (kvm_luo.c) and
+	 * kvm_arm_copy_sys_reg_indices() (sys_regs.c): what matters is whether
+	 * an in-kernel irqchip exists at all, not whether it has been
+	 * initialised yet.  vgic_initialized() would let a created-but-not-ready
+	 * in-kernel VGICv2 through, and the caretaker cannot emulate one.
+	 */
+	if (irqchip_in_kernel(vcpu->kvm) &&
+	    vcpu->kvm->arch.vgic.vgic_model != KVM_DEV_TYPE_ARM_VGIC_V3)
+		return -EOPNOTSUPP;
+	if (kvm_has_mte(vcpu->kvm) || kvm_has_s1poe(vcpu->kvm) ||
+	    vcpu_has_sve(vcpu) || vcpu_has_ptrauth(vcpu) ||
+	    test_bit(KVM_ARCH_FLAG_WRITABLE_IMP_ID_REGS, &vcpu->kvm->arch.flags))
+		return -EOPNOTSUPP;
+
+	cap = kho_alloc_preserve(sizeof(*cap));
+	if (IS_ERR(cap)) {
+		pr_err("caretaker arm64: failed to allocate preserved page\n");
+		return -ENOMEM;
+	}
+	oncore_session_map_buffer(sess, cap, sizeof(*cap));
+
+	memset(cap, 0, sizeof(*cap));
+
+	kvm_caretaker_init_common_vcpu(&cap->vcpu, &cap->abi.cb, vcpu, cap,
+				       sizeof(*cap), NULL, cap);
+
+	/* Copy architectural execution state */
+	cap->ctx.ctxt = vcpu->arch.ctxt;
+	cap->ctx.fault = vcpu->arch.fault;
+	cap->ctx.hcr_el2 = vcpu->arch.hcr_el2;
+	cap->ctx.mdcr_el2 = vcpu->arch.mdcr_el2;
+	cap->ctx.cflags = vcpu->arch.cflags;
+
+	cap->ctx.vtcr_el2 = mmu->vtcr;
+	cap->ctx.vttbr_el2 = kvm_get_vttbr(mmu);
+
+	if (vgic_initialized(vcpu->kvm)) {
+		int i;
+
+		cap->ctx.vgic_initialized = true;
+		cap->ctx.vgic_v3 = vcpu->arch.vgic_cpu.vgic_v3;
+
+		for (i = 0; ; i++) {
+			phys_addr_t rpa;
+			unsigned long rva;
+			size_t rsize;
+
+			if (gicv3_caretaker_get_redist_region(i, &rpa, &rva, &rsize))
+				break;
+			oncore_session_map_range(sess, rpa, rva, rsize,
+						 pgprot_device(PAGE_KERNEL));
+		}
+	}
+
+	vtimer = vcpu_vtimer(vcpu);
+	cap->ctx.cntvoff_el2 = timer_get_offset(vtimer);
+	cap->ctx.cntv_cval_el0 = timer_get_cval(vtimer);
+	cap->ctx.cntv_ctl_el0 = timer_get_ctl(vtimer);
+
+	head = vcpu->kvm->caretaker_vm;
+	if (!head) {
+		cap->next_vcpu = cap;
+		vcpu->kvm->caretaker_vm = cap;
+	} else {
+		struct caretaker_arm64_page *peer;
+
+		cap->next_vcpu = head->next_vcpu;
+		head->next_vcpu = cap;
+		cpu_preserved_clean(head);
+
+		for (peer = cap->next_vcpu; peer != cap; peer = peer->next_vcpu)
+			oncore_session_map_buffer(sess, peer, sizeof(*peer));
+	}
+
+	cap->abi.vgic_initialized = cap->ctx.vgic_initialized ? 1 : 0;
+	cap->abi.cntvoff_el2 = cap->ctx.cntvoff_el2;
+	cap->abi.hcr_el2 = cap->ctx.hcr_el2;
+	cap->abi.mdcr_el2 = cap->ctx.mdcr_el2;
+	cap->abi.cflags = (u32)cap->ctx.cflags;
+	if (cap->ctx.vgic_initialized) {
+		int i;
+
+		cap->abi.used_lrs = cap->ctx.vgic_v3.used_lrs;
+		cap->abi.vgic_hcr = cap->ctx.vgic_v3.vgic_hcr;
+		cap->abi.vgic_vmcr = cap->ctx.vgic_v3.vgic_vmcr;
+		for (i = 0; i < 4; i++) {
+			cap->abi.vgic_ap0r[i] = cap->ctx.vgic_v3.vgic_ap0r[i];
+			cap->abi.vgic_ap1r[i] = cap->ctx.vgic_v3.vgic_ap1r[i];
+		}
+		for (i = 0; i < 16; i++)
+			cap->abi.vgic_lr[i] = cap->ctx.vgic_v3.vgic_lr[i];
+	}
+
+	if (ser->arch_state.phys) {
+		struct kvm_vcpu_arch_ser *state =
+			phys_to_virt(ser->arch_state.phys);
+		size_t sz = struct_size(state, sysregs, state->num_sysregs);
+
+		cap->arch_state = state;
+		oncore_session_map_buffer(sess, state, sz);
+	}
+
+	ser->cb.phys = virt_to_phys(cap);
+
+	cpu_preserved_clean(cap);
+
+	return 0;
+}
+
+static __always_inline void arm64_caretaker_save_ptrauth(struct arm64_caretaker_ptrauth_keys *k)
+{
+	if (!arm64_caretaker_has_ptrauth)
+		return;
+
+	k->apia_lo = read_sysreg_s(SYS_APIAKEYLO_EL1);
+	k->apia_hi = read_sysreg_s(SYS_APIAKEYHI_EL1);
+	k->apib_lo = read_sysreg_s(SYS_APIBKEYLO_EL1);
+	k->apib_hi = read_sysreg_s(SYS_APIBKEYHI_EL1);
+	k->apda_lo = read_sysreg_s(SYS_APDAKEYLO_EL1);
+	k->apda_hi = read_sysreg_s(SYS_APDAKEYHI_EL1);
+	k->apdb_lo = read_sysreg_s(SYS_APDBKEYLO_EL1);
+	k->apdb_hi = read_sysreg_s(SYS_APDBKEYHI_EL1);
+	k->apga_lo = read_sysreg_s(SYS_APGAKEYLO_EL1);
+	k->apga_hi = read_sysreg_s(SYS_APGAKEYHI_EL1);
+}
+
+static __always_inline void
+arm64_caretaker_restore_ptrauth(const struct arm64_caretaker_ptrauth_keys *k)
+{
+	if (!arm64_caretaker_has_ptrauth)
+		return;
+
+	write_sysreg_s(k->apia_lo, SYS_APIAKEYLO_EL1);
+	write_sysreg_s(k->apia_hi, SYS_APIAKEYHI_EL1);
+	write_sysreg_s(k->apib_lo, SYS_APIBKEYLO_EL1);
+	write_sysreg_s(k->apib_hi, SYS_APIBKEYHI_EL1);
+	write_sysreg_s(k->apda_lo, SYS_APDAKEYLO_EL1);
+	write_sysreg_s(k->apda_hi, SYS_APDAKEYHI_EL1);
+	write_sysreg_s(k->apdb_lo, SYS_APDBKEYLO_EL1);
+	write_sysreg_s(k->apdb_hi, SYS_APDBKEYHI_EL1);
+	write_sysreg_s(k->apga_lo, SYS_APGAKEYLO_EL1);
+	write_sysreg_s(k->apga_hi, SYS_APGAKEYHI_EL1);
+	isb();
+}
+
+#define __caretaker_apr_save(_v, _reg)		((_v) = read_sysreg_s(_reg))
+#define __caretaker_apr_restore(_v, _reg)	write_sysreg_s((_v), _reg)
+
+/*
+ * ICH_AP0R<n>_EL2 and ICH_AP1R<n>_EL2 encode <n> in the instruction, so the
+ * accesses cannot be put in a loop.  Generate the ladder for either group and
+ * either direction from one pattern instead of writing it out four times.
+ *
+ * @_dir: save or restore
+ * @_grp: 0 or 1, selecting ICH_AP0R<n>_EL2 or ICH_AP1R<n>_EL2
+ */
+#define caretaker_vgic_v3_apr(_dir, _grp, _regs, _nr_pre_bits)			\
+do {										\
+	switch (_nr_pre_bits) {							\
+	case 7:									\
+		__caretaker_apr_##_dir((_regs)[3], SYS_ICH_AP##_grp##R3_EL2);	\
+		__caretaker_apr_##_dir((_regs)[2], SYS_ICH_AP##_grp##R2_EL2);	\
+		fallthrough;							\
+	case 6:									\
+		__caretaker_apr_##_dir((_regs)[1], SYS_ICH_AP##_grp##R1_EL2);	\
+		fallthrough;							\
+	default:								\
+		__caretaker_apr_##_dir((_regs)[0], SYS_ICH_AP##_grp##R0_EL2);	\
+	}									\
+} while (0)
+
+/*
+ * Acknowledge and retire whichever Group 1 interrupt is pending on this CPU.
+ * EOImode is 1 (priority drop and deactivation are separate), so a real INTID
+ * needs both an EOIR1 and a DIR write.  The special INTIDs (1020-1023) mean
+ * "nothing pending" and must not be written back.
+ */
+__caretaker_text static void caretaker_gic_drain_iar(void)
+{
+	u32 iar = read_sysreg_s(SYS_ICC_IAR1_EL1);
+
+	if (iar < GIC_SPECIAL_INTID_START) {
+		write_sysreg_s(iar, SYS_ICC_EOIR1_EL1);
+		write_sysreg_s(iar, SYS_ICC_DIR_EL1);
+	}
+}
+
+/*
+ * Put EL2 back into host (VHE) configuration.  Both vector bases are pointed
+ * at the caretaker's own hyp vectors: while a preserved core is running a
+ * guest the caretaker owns EL2, and the kernel that installed kvm_hyp_vector
+ * may no longer exist.
+ */
+__caretaker_text static void caretaker_restore_host_el2(void)
+{
+	write_sysreg_hcr(HCR_HOST_VHE_FLAGS);
+	write_sysreg((unsigned long)caretaker_hyp_vector, vbar_el1);
+	write_sysreg_s((unsigned long)caretaker_hyp_vector, SYS_VBAR_EL2);
+	dsb(sy);
+	isb();
+}
+
+__caretaker_text static u64 caretaker_gic_v3_get_lr(unsigned int lr)
+{
+	switch (lr & 0xf) {
+	case 0:
+		return read_gicreg(ICH_LR0_EL2);
+	case 1:
+		return read_gicreg(ICH_LR1_EL2);
+	case 2:
+		return read_gicreg(ICH_LR2_EL2);
+	case 3:
+		return read_gicreg(ICH_LR3_EL2);
+	case 4:
+		return read_gicreg(ICH_LR4_EL2);
+	case 5:
+		return read_gicreg(ICH_LR5_EL2);
+	case 6:
+		return read_gicreg(ICH_LR6_EL2);
+	case 7:
+		return read_gicreg(ICH_LR7_EL2);
+	case 8:
+		return read_gicreg(ICH_LR8_EL2);
+	case 9:
+		return read_gicreg(ICH_LR9_EL2);
+	case 10:
+		return read_gicreg(ICH_LR10_EL2);
+	case 11:
+		return read_gicreg(ICH_LR11_EL2);
+	case 12:
+		return read_gicreg(ICH_LR12_EL2);
+	case 13:
+		return read_gicreg(ICH_LR13_EL2);
+	case 14:
+		return read_gicreg(ICH_LR14_EL2);
+	default:
+		return read_gicreg(ICH_LR15_EL2);
+	}
+}
+
+__caretaker_text static void caretaker_gic_v3_set_lr(u64 val, unsigned int lr)
+{
+	switch (lr & 0xf) {
+	case 0:
+		write_gicreg(val, ICH_LR0_EL2);
+		break;
+	case 1:
+		write_gicreg(val, ICH_LR1_EL2);
+		break;
+	case 2:
+		write_gicreg(val, ICH_LR2_EL2);
+		break;
+	case 3:
+		write_gicreg(val, ICH_LR3_EL2);
+		break;
+	case 4:
+		write_gicreg(val, ICH_LR4_EL2);
+		break;
+	case 5:
+		write_gicreg(val, ICH_LR5_EL2);
+		break;
+	case 6:
+		write_gicreg(val, ICH_LR6_EL2);
+		break;
+	case 7:
+		write_gicreg(val, ICH_LR7_EL2);
+		break;
+	case 8:
+		write_gicreg(val, ICH_LR8_EL2);
+		break;
+	case 9:
+		write_gicreg(val, ICH_LR9_EL2);
+		break;
+	case 10:
+		write_gicreg(val, ICH_LR10_EL2);
+		break;
+	case 11:
+		write_gicreg(val, ICH_LR11_EL2);
+		break;
+	case 12:
+		write_gicreg(val, ICH_LR12_EL2);
+		break;
+	case 13:
+		write_gicreg(val, ICH_LR13_EL2);
+		break;
+	case 14:
+		write_gicreg(val, ICH_LR14_EL2);
+		break;
+	default:
+		write_gicreg(val, ICH_LR15_EL2);
+		break;
+	}
+}
+
+__caretaker_text static void caretaker_vgic_v3_restore(struct caretaker_arm64_page *cap)
+{
+	struct vgic_v3_cpu_if *cpu_if = &cap->ctx.vgic_v3;
+	unsigned int used_lrs = cpu_if->used_lrs;
+	u64 vtr = read_sysreg_s(SYS_ICH_VTR_EL2);
+	unsigned int max_lrs = FIELD_GET(ICH_VTR_EL2_ListRegs, vtr) + 1;
+	u32 nr_pre_bits = FIELD_GET(ICH_VTR_EL2_PREbits, vtr) + 1;
+	unsigned int i;
+
+	/* Drain any pending software-generated SGIs into empty LRs */
+	if (cap->ctx.pending_sgis) {
+		for (i = 0; i < max_lrs && cap->ctx.pending_sgis; i++) {
+			if (i >= used_lrs || (cpu_if->vgic_lr[i] & ICH_LR_STATE) == 0) {
+				int sgi = __ffs(cap->ctx.pending_sgis);
+
+				cap->ctx.pending_sgis &= ~BIT(sgi);
+				cpu_if->vgic_lr[i] = ((u64)sgi & 0xf) |
+						     ICH_LR_PENDING_BIT |
+						     ICH_LR_GROUP |
+						     ((u64)GIC_DEFAULT_SGI_PRIO <<
+						      ICH_LR_PRIORITY_SHIFT);
+				if (i >= used_lrs)
+					used_lrs = i + 1;
+			}
+		}
+		cpu_if->used_lrs = used_lrs;
+	}
+
+	write_sysreg_s(cpu_if->vgic_vmcr, SYS_ICH_VMCR_EL2);
+
+	caretaker_vgic_v3_apr(restore, 0, cpu_if->vgic_ap0r, nr_pre_bits);
+	caretaker_vgic_v3_apr(restore, 1, cpu_if->vgic_ap1r, nr_pre_bits);
+
+	write_sysreg_s(cpu_if->vgic_hcr | ICH_HCR_EL2_En, SYS_ICH_HCR_EL2);
+
+	used_lrs = min3(used_lrs, max_lrs, (unsigned int)VGIC_V3_MAX_LRS);
+
+	for (i = 0; i < used_lrs; i++)
+		caretaker_gic_v3_set_lr(cpu_if->vgic_lr[i], i);
+	isb();
+}
+
+__caretaker_text static void caretaker_vgic_v3_save(struct vgic_v3_cpu_if *cpu_if)
+{
+	u64 vtr = read_sysreg_s(SYS_ICH_VTR_EL2);
+	unsigned int max_lrs = FIELD_GET(ICH_VTR_EL2_ListRegs, vtr) + 1;
+	u32 nr_pre_bits = FIELD_GET(ICH_VTR_EL2_PREbits, vtr) + 1;
+	unsigned int used_lrs = cpu_if->used_lrs;
+	unsigned int i;
+
+	used_lrs = min3(used_lrs, max_lrs, (unsigned int)VGIC_V3_MAX_LRS);
+
+	for (i = 0; i < used_lrs; i++) {
+		cpu_if->vgic_lr[i] = __gic_v3_get_lr(i);
+		__gic_v3_set_lr(0, i);
+	}
+
+	cpu_if->vgic_vmcr = read_sysreg_s(SYS_ICH_VMCR_EL2);
+
+	caretaker_vgic_v3_apr(save, 0, cpu_if->vgic_ap0r, nr_pre_bits);
+	caretaker_vgic_v3_apr(save, 1, cpu_if->vgic_ap1r, nr_pre_bits);
+
+	write_sysreg_s(0, SYS_ICH_HCR_EL2);
+	isb();
+}
+
-- 
2.55.0.1082.g2b9226bbc0-goog

[RFC PATCH 45/46] KVM: arm64: Implement Caretaker system register trap and exception handlers

[Pasha Tatashin]

Implement standalone Caretaker EL2 exception decoding, system register
trap handling, and stage-2 non-RAM IPA yield in arch/arm64/kvm/caretaker.c.

Signed-off-by: Pasha Tatashin <pasha.tatashin@soleen.com>
---
 arch/arm64/kvm/caretaker.c | 408 ++++++++++++++++++++++++++++++++++++-
 1 file changed, 405 insertions(+), 3 deletions(-)

diff --git a/arch/arm64/kvm/caretaker.c b/arch/arm64/kvm/caretaker.c
index 65c11fe8c2fa..6980c1f6b998 100644
--- a/arch/arm64/kvm/caretaker.c
+++ b/arch/arm64/kvm/caretaker.c
@@ -60,10 +60,8 @@ arm64_caretaker_handle_invalid(u64 elr, u64 esr, u64 far)
 	}
 
 	while (1) {
-		if (cpu_preserved_should_exit(cpu)) {
-			cpu_preserved_set_dead(cpu);
+		if (cpu_preserved_should_exit(cpu))
 			arch_cpu_preserved_park_finish(cpu);
-		}
 		arch_cpu_preserved_park_wait();
 	}
 }
@@ -519,3 +517,407 @@ __caretaker_text static void caretaker_vgic_v3_save(struct vgic_v3_cpu_if *cpu_i
 	isb();
 }
 
+__caretaker_text static void
+caretaker_arm64_inject_sgi(struct caretaker_arm64_page *target_cap, u32 sgi)
+{
+	int slot = -1;
+	int i;
+
+	if (!target_cap)
+		return;
+
+	cpu_preserved_inval(target_cap);
+
+	/* Check if the SGI is already pending or active */
+	for (i = 0; i < target_cap->ctx.vgic_v3.used_lrs; i++) {
+		u64 lr = target_cap->ctx.vgic_v3.vgic_lr[i];
+
+		if ((lr & ICH_LR_VIRTUAL_ID_MASK) == (sgi & 0xf) && (lr & ICH_LR_STATE))
+			return;
+		if ((lr & ICH_LR_STATE) == 0 && slot < 0)
+			slot = i;
+	}
+
+	if (slot < 0 && target_cap->ctx.vgic_v3.used_lrs < VGIC_V3_MAX_LRS) {
+		slot = target_cap->ctx.vgic_v3.used_lrs;
+		target_cap->ctx.vgic_v3.used_lrs++;
+	}
+
+	if (slot >= 0) {
+		target_cap->ctx.vgic_v3.vgic_lr[slot] =
+			((u64)sgi & 0xf) |
+			ICH_LR_PENDING_BIT |
+			ICH_LR_GROUP |
+			(GIC_DEFAULT_SGI_PRIO << ICH_LR_PRIORITY_SHIFT);
+	} else {
+		target_cap->ctx.pending_sgis |= BIT(sgi & 0xf);
+	}
+
+	cpu_preserved_clean(target_cap);
+
+	/* If target vCPU is running on a remote physical CPU, kick it */
+	if (target_cap->abi.cb.pcpu_id >= 0 &&
+	    target_cap->abi.cb.pcpu_id != arm64_caretaker_get_pcpu()) {
+		arch_cpu_preserved_kick(target_cap->abi.cb.pcpu_id);
+	}
+}
+
+__caretaker_text static void
+caretaker_arm64_handle_sgi(struct caretaker_arm64_page *src_cap, u64 reg)
+{
+	struct caretaker_arm64_page *target;
+	u32 sgi = FIELD_GET(ICC_SGI1R_SGI_ID_MASK, reg);
+	unsigned int i;
+
+	if (!src_cap || !src_cap->next_vcpu)
+		return;
+
+	if (reg & BIT_ULL(ICC_SGI1R_IRQ_ROUTING_MODE_BIT)) {
+		/* Broadcast to all other vCPUs */
+		for (target = src_cap->next_vcpu;
+		     target && target != src_cap;
+		     target = target->next_vcpu) {
+			cpu_preserved_inval(target);
+			caretaker_arm64_inject_sgi(target, sgi);
+		}
+	} else {
+		u64 aff3 = FIELD_GET(ICC_SGI1R_AFFINITY_3_MASK, reg);
+		u64 aff2 = FIELD_GET(ICC_SGI1R_AFFINITY_2_MASK, reg);
+		u64 aff1 = FIELD_GET(ICC_SGI1R_AFFINITY_1_MASK, reg);
+		u64 rs = FIELD_GET(ICC_SGI1R_RS_MASK, reg);
+		u64 cluster_mpidr = (aff3 << MPIDR_LEVEL_SHIFT(3)) |
+				    (aff2 << MPIDR_LEVEL_SHIFT(2)) |
+				    (aff1 << MPIDR_LEVEL_SHIFT(1));
+		u64 target_list = FIELD_GET(ICC_SGI1R_TARGET_LIST_MASK, reg);
+
+		for (i = 0; i < 16; i++) {
+			u64 target_mpidr;
+
+			if (!(target_list & BIT(i)))
+				continue;
+
+			target_mpidr = cluster_mpidr |
+				       ((rs * 16 + i) << MPIDR_LEVEL_SHIFT(0));
+
+			target = src_cap;
+			do {
+				cpu_preserved_inval(target);
+				if ((ctxt_sys_reg(&target->ctx.ctxt, MPIDR_EL1) &
+				     MPIDR_HWID_BITMASK) == target_mpidr) {
+					caretaker_arm64_inject_sgi(target, sgi);
+					break;
+				}
+				target = target->next_vcpu;
+			} while (target && target != src_cap);
+		}
+	}
+}
+
+static __caretaker_text int arm64_caretaker_op_enter(void *data)
+{
+	struct caretaker_arm64_page *cap = data;
+	u64 guest_hcr;
+
+	gicv3_caretaker_clear_active_priorities();
+	write_sysreg_s(ICC_PMR_EL1_MASK, SYS_ICC_PMR_EL1);
+	write_sysreg_s(ICC_CTLR_EL1_EOImode_drop, SYS_ICC_CTLR_EL1);
+	write_sysreg_s(ICC_IGRPEN1_EL1_MASK, SYS_ICC_IGRPEN1_EL1);
+	pmr_sync();
+
+	guest_hcr = (cap->ctx.hcr_el2 | HCR_AMO | HCR_IMO | HCR_FMO | HCR_E2H) & ~HCR_TGE;
+	write_sysreg_hcr(guest_hcr);
+	isb();
+
+	cap->last_ret = caretaker_guest_enter(&cap->ctx);
+
+	write_sysreg_hcr(HCR_HOST_VHE_FLAGS);
+	isb();
+
+	return 0;
+}
+
+static __caretaker_text void
+arm64_caretaker_op_arm_timer(void *data, u64 deadline_ticks)
+{
+	if (deadline_ticks) {
+		write_sysreg_s(deadline_ticks, SYS_CNTHP_CVAL_EL2);
+		isb();
+		write_sysreg_s(1, SYS_CNTHP_CTL_EL2);
+	} else {
+		write_sysreg_s(0, SYS_CNTHP_CTL_EL2);
+	}
+	isb();
+}
+
+static __caretaker_text void
+arm64_caretaker_op_disarm_timer(void *data)
+{
+	write_sysreg_s(0, SYS_CNTHP_CTL_EL2);
+	isb();
+}
+
+static __caretaker_text void
+arm64_caretaker_op_decode_exit(void *data, struct kvm_caretaker_exit *exit)
+{
+	struct caretaker_arm64_page *cap = data;
+	u64 ret = cap->last_ret;
+
+	exit->rip = cap->ctx.ctxt.regs.pc;
+	exit->insn_len = 0;
+	exit->type = KVM_CARETAKER_EXIT_UNKNOWN;
+
+	if (ARM_EXCEPTION_CODE(ret) == ARM_EXCEPTION_IRQ) {
+		caretaker_gic_drain_iar();
+		gicv3_caretaker_clear_active_priorities();
+		dsb(sy);
+		isb();
+
+		exit->type = KVM_CARETAKER_EXIT_PREEMPT_TIMER;
+		return;
+	}
+
+	if (ARM_EXCEPTION_IS_TRAP(ret)) {
+		u64 esr = cap->ctx.fault.esr_el2;
+		u8 ec = ESR_ELx_EC(esr);
+
+		exit->insn_len = 4;
+
+		if (ec == ESR_ELx_EC_WFx) {
+			exit->type = KVM_CARETAKER_EXIT_IDLE;
+			return;
+		}
+
+		if (ec == ESR_ELx_EC_SYS64) {
+			u32 iss = ESR_ELx_ISS(esr);
+			u32 sys_op = iss & ESR_ELx_SYS64_ISS_SYS_OP_MASK;
+
+			if (sys_op == ESR_ELx_SYS64_ISS_SYS_ICC_SGI1R_EL1 ||
+			    sys_op == ESR_ELx_SYS64_ISS_SYS_ICC_ASGI1R_EL1 ||
+			    sys_op == ESR_ELx_SYS64_ISS_SYS_ICC_SGI0R_EL1) {
+				u32 rt = ESR_ELx_SYS64_ISS_RT(esr);
+				u64 val = (rt < 31) ? cap->ctx.ctxt.regs.regs[rt] : 0;
+
+				exit->type = KVM_CARETAKER_EXIT_CROSS_VCPU;
+				exit->sgi.sgi_id = FIELD_GET(ICC_SGI1R_SGI_ID_MASK, val);
+				exit->sgi.target_mask = val;
+				return;
+			}
+		}
+
+		if (ec == ESR_ELx_EC_DABT_LOW || ec == ESR_ELx_EC_IABT_LOW) {
+			/*
+			 * Stage-2 abort on a non-RAM IPA (e.g. MMIO device).
+			 * Do not advance PC: yield this vCPU out of guest mode
+			 * so the incoming kernel's real KVM + VMM handles the
+			 * fault on re-attachment.
+			 */
+			exit->type = KVM_CARETAKER_EXIT_IDLE;
+			exit->insn_len = 0;
+			return;
+		}
+
+		exit->type = KVM_CARETAKER_EXIT_ARCH;
+		exit->raw_reason = esr;
+		return;
+	}
+
+	exit->type = KVM_CARETAKER_EXIT_ARCH;
+}
+
+static __caretaker_text void
+arm64_caretaker_op_advance_rip(void *data, u64 next_rip)
+{
+	struct caretaker_arm64_page *cap = data;
+
+	cap->ctx.ctxt.regs.pc = next_rip;
+}
+
+static __caretaker_text bool
+arm64_caretaker_op_handle_exit(void *data, struct kvm_caretaker_exit *exit)
+{
+	struct caretaker_arm64_page *cap = data;
+
+	if (exit->type == KVM_CARETAKER_EXIT_CROSS_VCPU) {
+		caretaker_arm64_handle_sgi(cap, exit->sgi.target_mask);
+		exit->rip += exit->insn_len;
+		return true;
+	}
+
+	/*
+	 * KVM_CARETAKER_EXIT_ARCH is the fallback type assigned by
+	 * arm64_caretaker_op_decode_exit() to every trap it did not decode,
+	 * including system register accesses.  Skipping the instruction here
+	 * would leave the destination register holding whatever it held
+	 * before the trap and bypass KVM's ID register sanitisation, with the
+	 * guest none the wiser.
+	 *
+	 * Leave PC pointing at the trapping instruction and report the exit
+	 * as unhandled so the vCPU parks until the incoming kernel reclaims
+	 * it and full KVM handles the trap.
+	 */
+	return false;
+}
+
+static __caretaker_text inline u64 arm64_sysreg_to_uapi_id(u32 reg)
+{
+	if (reg == SYS_CNTV_CVAL_EL0)
+		return KVM_REG_ARM_TIMER_CVAL;
+	return (KVM_REG_ARM64 | KVM_REG_SIZE_U64 |
+		KVM_REG_ARM64_SYSREG |
+		((u64)sys_reg_Op0(reg) << KVM_REG_ARM64_SYSREG_OP0_SHIFT) |
+		((u64)sys_reg_Op1(reg) << KVM_REG_ARM64_SYSREG_OP1_SHIFT) |
+		((u64)sys_reg_CRn(reg) << KVM_REG_ARM64_SYSREG_CRN_SHIFT) |
+		((u64)sys_reg_CRm(reg) << KVM_REG_ARM64_SYSREG_CRM_SHIFT) |
+		((u64)sys_reg_Op2(reg) << KVM_REG_ARM64_SYSREG_OP2_SHIFT));
+}
+
+static __caretaker_text void
+arm64_caretaker_update_sysreg(struct kvm_vcpu_arch_ser *state,
+			      u32 reg, u64 val)
+{
+	u64 id = arm64_sysreg_to_uapi_id(reg);
+	u32 i;
+
+	for (i = 0; i < state->num_sysregs; i++) {
+		if (state->sysregs[i].id == id) {
+			state->sysregs[i].addr = val;
+			return;
+		}
+	}
+}
+
+static __caretaker_text void
+arm64_caretaker_detach_serialize(struct caretaker_arm64_page *cap)
+{
+	cap->abi.vgic_initialized = cap->ctx.vgic_initialized ? 1 : 0;
+	cap->abi.cntvoff_el2 = cap->ctx.cntvoff_el2;
+	cap->abi.hcr_el2 = cap->ctx.hcr_el2;
+	cap->abi.mdcr_el2 = cap->ctx.mdcr_el2;
+	cap->abi.cflags = (u32)cap->ctx.cflags;
+	if (cap->ctx.vgic_initialized) {
+		int i;
+
+		cap->abi.used_lrs = cap->ctx.vgic_v3.used_lrs;
+		cap->abi.vgic_hcr = cap->ctx.vgic_v3.vgic_hcr;
+		cap->abi.vgic_vmcr = cap->ctx.vgic_v3.vgic_vmcr;
+		for (i = 0; i < 4; i++) {
+			cap->abi.vgic_ap0r[i] = cap->ctx.vgic_v3.vgic_ap0r[i];
+			cap->abi.vgic_ap1r[i] = cap->ctx.vgic_v3.vgic_ap1r[i];
+		}
+		for (i = 0; i < 16; i++)
+			cap->abi.vgic_lr[i] = cap->ctx.vgic_v3.vgic_lr[i];
+	}
+
+	if (cap->arch_state) {
+		struct kvm_vcpu_arch_ser *state = cap->arch_state;
+
+		cpu_preserved_memcpy(&state->regs.regs, &cap->ctx.ctxt.regs,
+				     sizeof(state->regs.regs));
+		state->regs.sp_el1 = ctxt_sys_reg(&cap->ctx.ctxt, SP_EL1);
+		state->regs.elr_el1 = ctxt_sys_reg(&cap->ctx.ctxt, ELR_EL1);
+		state->regs.spsr[KVM_SPSR_EL1] = ctxt_sys_reg(&cap->ctx.ctxt, SPSR_EL1);
+		state->regs.spsr[KVM_SPSR_ABT] = cap->ctx.ctxt.spsr_abt;
+		state->regs.spsr[KVM_SPSR_UND] = cap->ctx.ctxt.spsr_und;
+		state->regs.spsr[KVM_SPSR_IRQ] = cap->ctx.ctxt.spsr_irq;
+		state->regs.spsr[KVM_SPSR_FIQ] = cap->ctx.ctxt.spsr_fiq;
+		cpu_preserved_memcpy(&state->regs.fp_regs, &cap->ctx.ctxt.fp_regs,
+				     sizeof(state->regs.fp_regs));
+
+		arm64_caretaker_update_sysreg(state, SYS_SCTLR_EL1,
+					      ctxt_sys_reg(&cap->ctx.ctxt, SCTLR_EL1));
+		arm64_caretaker_update_sysreg(state, SYS_CPACR_EL1,
+					      ctxt_sys_reg(&cap->ctx.ctxt, CPACR_EL1));
+		arm64_caretaker_update_sysreg(state, SYS_TTBR0_EL1,
+					      ctxt_sys_reg(&cap->ctx.ctxt, TTBR0_EL1));
+		arm64_caretaker_update_sysreg(state, SYS_TTBR1_EL1,
+					      ctxt_sys_reg(&cap->ctx.ctxt, TTBR1_EL1));
+		arm64_caretaker_update_sysreg(state, SYS_TCR_EL1,
+					      ctxt_sys_reg(&cap->ctx.ctxt, TCR_EL1));
+		arm64_caretaker_update_sysreg(state, SYS_ESR_EL1,
+					      ctxt_sys_reg(&cap->ctx.ctxt, ESR_EL1));
+		arm64_caretaker_update_sysreg(state, SYS_AFSR0_EL1,
+					      ctxt_sys_reg(&cap->ctx.ctxt, AFSR0_EL1));
+		arm64_caretaker_update_sysreg(state, SYS_AFSR1_EL1,
+					      ctxt_sys_reg(&cap->ctx.ctxt, AFSR1_EL1));
+		arm64_caretaker_update_sysreg(state, SYS_FAR_EL1,
+					      ctxt_sys_reg(&cap->ctx.ctxt, FAR_EL1));
+		arm64_caretaker_update_sysreg(state, SYS_MAIR_EL1,
+					      ctxt_sys_reg(&cap->ctx.ctxt, MAIR_EL1));
+		arm64_caretaker_update_sysreg(state, SYS_VBAR_EL1,
+					      ctxt_sys_reg(&cap->ctx.ctxt, VBAR_EL1));
+		arm64_caretaker_update_sysreg(state, SYS_CONTEXTIDR_EL1,
+					      ctxt_sys_reg(&cap->ctx.ctxt, CONTEXTIDR_EL1));
+		arm64_caretaker_update_sysreg(state, SYS_AMAIR_EL1,
+					      ctxt_sys_reg(&cap->ctx.ctxt, AMAIR_EL1));
+		arm64_caretaker_update_sysreg(state, SYS_CNTKCTL_EL1,
+					      ctxt_sys_reg(&cap->ctx.ctxt, CNTKCTL_EL1));
+		arm64_caretaker_update_sysreg(state, SYS_PAR_EL1,
+					      ctxt_sys_reg(&cap->ctx.ctxt, PAR_EL1));
+		arm64_caretaker_update_sysreg(state, SYS_TPIDR_EL1,
+					      ctxt_sys_reg(&cap->ctx.ctxt, TPIDR_EL1));
+		arm64_caretaker_update_sysreg(state, SYS_TPIDR_EL0,
+					      ctxt_sys_reg(&cap->ctx.ctxt, TPIDR_EL0));
+		arm64_caretaker_update_sysreg(state, SYS_TPIDRRO_EL0,
+					      ctxt_sys_reg(&cap->ctx.ctxt, TPIDRRO_EL0));
+		arm64_caretaker_update_sysreg(state, SYS_SP_EL1,
+					      ctxt_sys_reg(&cap->ctx.ctxt, SP_EL1));
+		arm64_caretaker_update_sysreg(state, SYS_ELR_EL1,
+					      ctxt_sys_reg(&cap->ctx.ctxt, ELR_EL1));
+		arm64_caretaker_update_sysreg(state, SYS_SPSR_EL1,
+					      ctxt_sys_reg(&cap->ctx.ctxt, SPSR_EL1));
+		arm64_caretaker_update_sysreg(state, SYS_MDSCR_EL1,
+					      ctxt_sys_reg(&cap->ctx.ctxt, MDSCR_EL1));
+		arm64_caretaker_update_sysreg(state, SYS_CNTV_CVAL_EL0,
+					      cap->ctx.cntv_cval_el0);
+		arm64_caretaker_update_sysreg(state, SYS_CNTV_CTL_EL0,
+					      cap->ctx.cntv_ctl_el0);
+		cpu_preserved_clean_sz(state,
+				       struct_size(state, sysregs, state->num_sysregs));
+	}
+
+	cpu_preserved_clean(&cap->abi);
+}
+
+static void arm64_caretaker_sync_vcpu(struct kvm_vcpu *vcpu,
+				      void *data)
+{
+	struct arch_timer_context *vtimer = vcpu_vtimer(vcpu);
+	struct kvm_caretaker_arch_ser *abi = data;
+	u64 cval, ctl;
+	int t;
+
+	/* Sync handover ABI prefix back into incoming vcpu */
+	vcpu->arch.hcr_el2 = abi->hcr_el2;
+	vcpu->arch.mdcr_el2 = abi->mdcr_el2;
+	vcpu->arch.cflags = abi->cflags;
+
+	if (abi->vgic_initialized) {
+		int i;
+
+		vcpu->arch.vgic_cpu.vgic_v3.used_lrs = abi->used_lrs;
+		vcpu->arch.vgic_cpu.vgic_v3.vgic_hcr = abi->vgic_hcr;
+		vcpu->arch.vgic_cpu.vgic_v3.vgic_vmcr = abi->vgic_vmcr;
+		for (i = 0; i < 4; i++) {
+			vcpu->arch.vgic_cpu.vgic_v3.vgic_ap0r[i] = abi->vgic_ap0r[i];
+			vcpu->arch.vgic_cpu.vgic_v3.vgic_ap1r[i] = abi->vgic_ap1r[i];
+		}
+		for (i = 0; i < 16; i++)
+			vcpu->arch.vgic_cpu.vgic_v3.vgic_lr[i] = abi->vgic_lr[i];
+	}
+
+	cval = __vcpu_sys_reg(vcpu, CNTV_CVAL_EL0);
+	ctl = __vcpu_sys_reg(vcpu, CNTV_CTL_EL0);
+	timer_set_offset(vtimer, abi->cntvoff_el2);
+	write_sysreg(abi->cntvoff_el2, cntvoff_el2);
+	write_sysreg_el0(cval, SYS_CNTV_CVAL);
+	write_sysreg_el0(ctl, SYS_CNTV_CTL);
+	isb();
+
+	vcpu_set_flag(vcpu, VCPU_INITIALIZED);
+
+	for (t = 0; t < NR_KVM_TIMERS; t++)
+		vcpu->arch.timer_cpu.timers[t].loaded = false;
+
+	kvm_make_request(KVM_REQ_IRQ_PENDING, vcpu);
+}
+
-- 
2.55.0.1082.g2b9226bbc0-goog

[RFC PATCH 46/46] KVM: arm64: Implement Caretaker vCPU run loop and LUO detach/attach lifecycle

[Pasha Tatashin]

Complete ARM64 Caretaker vCPU execution loop, detach-time state
serialization, and LUO integration in arch/arm64/kvm/caretaker.c.

Signed-off-by: Pasha Tatashin <pasha.tatashin@soleen.com>
---
 arch/arm64/kvm/Kconfig     |   1 +
 arch/arm64/kvm/Makefile    |  12 ++
 arch/arm64/kvm/caretaker.c | 243 ++++++++++++++++++++++++++++++++++++-
 arch/arm64/kvm/kvm_luo.c   | 103 ++++++++++++++++
 4 files changed, 357 insertions(+), 2 deletions(-)

diff --git a/arch/arm64/kvm/Kconfig b/arch/arm64/kvm/Kconfig
index fd25b3c0d8ca..9c43010873a3 100644
--- a/arch/arm64/kvm/Kconfig
+++ b/arch/arm64/kvm/Kconfig
@@ -38,6 +38,7 @@ menuconfig KVM
 	select GUEST_PERF_EVENTS if PERF_EVENTS
 	select KVM_GUEST_MEMFD
 	select HAVE_KVM_ARCH_VCPU_PRESERVE
+	select HAVE_KVM_ARCH_CARETAKER
 	help
 	  Support hosting virtualized guest machines.
 
diff --git a/arch/arm64/kvm/Makefile b/arch/arm64/kvm/Makefile
index d5e888d1bd50..58550a0a8160 100644
--- a/arch/arm64/kvm/Makefile
+++ b/arch/arm64/kvm/Makefile
@@ -12,6 +12,18 @@ obj-$(CONFIG_KVM) += hyp/
 
 CFLAGS_sys_regs.o += -Wno-override-init
 CFLAGS_handle_exit.o += -Wno-override-init
+KASAN_SANITIZE_caretaker.o := n
+KCSAN_SANITIZE_caretaker.o := n
+UBSAN_SANITIZE_caretaker.o := n
+KCOV_INSTRUMENT_caretaker.o := n
+CFLAGS_REMOVE_caretaker.o = $(CC_FLAGS_FTRACE)
+CFLAGS_caretaker.o += $(call cc-option,-mbranch-protection=none) \
+		      -fno-stack-protector \
+		      $(call cc-option,-ftrivial-auto-var-init=uninitialized) \
+		      $(call cc-option,-fno-jump-tables)
+AFLAGS_caretaker_vmenter.o += -D__KVM_VHE_HYPERVISOR__
+
+kvm-$(CONFIG_KVM_CARETAKER) += caretaker.o caretaker_vmenter.o
 
 kvm-y += arm.o mmu.o mmio.o psci.o hypercalls.o pvtime.o \
 	 inject_fault.o va_layout.o handle_exit.o config.o \
diff --git a/arch/arm64/kvm/caretaker.c b/arch/arm64/kvm/caretaker.c
index 6980c1f6b998..8cd73099a60a 100644
--- a/arch/arm64/kvm/caretaker.c
+++ b/arch/arm64/kvm/caretaker.c
@@ -193,6 +193,9 @@ int arm64_kvm_caretaker_preserve(struct kvm_vcpu *vcpu,
 	cap->ctx.vtcr_el2 = mmu->vtcr;
 	cap->ctx.vttbr_el2 = kvm_get_vttbr(mmu);
 
+	if (irqchip_in_kernel(vcpu->kvm) && !vgic_initialized(vcpu->kvm))
+		kvm_vgic_map_resources(vcpu->kvm);
+
 	if (vgic_initialized(vcpu->kvm)) {
 		int i;
 
@@ -504,8 +507,8 @@ __caretaker_text static void caretaker_vgic_v3_save(struct vgic_v3_cpu_if *cpu_i
 	used_lrs = min3(used_lrs, max_lrs, (unsigned int)VGIC_V3_MAX_LRS);
 
 	for (i = 0; i < used_lrs; i++) {
-		cpu_if->vgic_lr[i] = __gic_v3_get_lr(i);
-		__gic_v3_set_lr(0, i);
+		cpu_if->vgic_lr[i] = caretaker_gic_v3_get_lr(i);
+		caretaker_gic_v3_set_lr(0, i);
 	}
 
 	cpu_if->vgic_vmcr = read_sysreg_s(SYS_ICH_VMCR_EL2);
@@ -921,3 +924,239 @@ static void arm64_caretaker_sync_vcpu(struct kvm_vcpu *vcpu,
 	kvm_make_request(KVM_REQ_IRQ_PENDING, vcpu);
 }
 
+static __caretaker_text void
+arm64_caretaker_op_pre_run(void *data)
+{
+	struct caretaker_arm64_page *cap = data;
+
+	local_daif_mask();
+	arm64_caretaker_save_ptrauth(&cap->ptrauth_keys);
+
+	/* Pre-job: load guest context */
+	cpu_preserved_inval(cap);
+
+	arm64_caretaker_load_sysregs(&cap->ctx.ctxt);
+
+	if (cap->ctx.vgic_initialized)
+		caretaker_vgic_v3_restore(cap);
+
+	write_sysreg(cap->ctx.cntvoff_el2, cntvoff_el2);
+	write_sysreg_el0(cap->ctx.cntv_cval_el0, SYS_CNTV_CVAL);
+	write_sysreg_el0(cap->ctx.cntv_ctl_el0, SYS_CNTV_CTL);
+	isb();
+
+	if (cap->ctx.vtcr_el2 && cap->ctx.vttbr_el2) {
+		write_sysreg(cap->ctx.vtcr_el2, vtcr_el2);
+		write_sysreg(cap->ctx.vttbr_el2, vttbr_el2);
+		asm(ALTERNATIVE("nop", "isb", ARM64_WORKAROUND_SPECULATIVE_AT));
+		__tlbi(vmalle1);
+		asm volatile("ic iallu");
+		dsb(nsh);
+		isb();
+	}
+
+	write_sysreg(CPACR_EL1_FPEN_EL0EN | CPACR_EL1_FPEN_EL1EN |
+		     CPACR_EL1_ZEN_EL0EN | CPACR_EL1_ZEN_EL1EN,
+		     cpacr_el1);
+	isb();
+
+	fpsimd_load_state(&cap->ctx.ctxt.fp_regs);
+
+	gicv3_caretaker_enable_sgi();
+	write_sysreg_s(ICC_CTLR_EL1_EOImode_drop, SYS_ICC_CTLR_EL1);
+	write_sysreg_s(ICC_SRE_EL1_SRE, SYS_ICC_SRE_EL1);
+	write_sysreg_s(0, SYS_ICC_BPR1_EL1);
+	gicv3_caretaker_clear_active_priorities();
+	write_sysreg_s(ICC_PMR_EL1_MASK, SYS_ICC_PMR_EL1);
+	write_sysreg_s(ICC_IGRPEN1_EL1_MASK, SYS_ICC_IGRPEN1_EL1);
+	caretaker_gic_drain_iar();
+	dsb(sy);
+	isb();
+}
+
+static __caretaker_text void
+arm64_caretaker_op_post_run(void *data)
+{
+	struct cpu_preserved_stack_context *sctx;
+	struct caretaker_arm64_page *cap = data;
+	int cpu = cap->abi.cb.pcpu_id;
+	phys_addr_t pgd_pa;
+
+	/* Post-run: restore host hypervisor mode then save guest context */
+	write_sysreg_s(0, SYS_CNTHP_CTL_EL2);
+	caretaker_restore_host_el2();
+
+	fpsimd_save_state(&cap->ctx.ctxt.fp_regs);
+
+	cap->ctx.cntv_cval_el0 = read_sysreg_el0(SYS_CNTV_CVAL);
+	cap->ctx.cntv_ctl_el0 = read_sysreg_el0(SYS_CNTV_CTL);
+
+	if (cap->ctx.vgic_initialized)
+		caretaker_vgic_v3_save(&cap->ctx.vgic_v3);
+
+	arm64_caretaker_save_sysregs(&cap->ctx.ctxt);
+
+	sctx = cpu_preserved_get_stack_context();
+	if (sctx && sctx->session_pgd_pa)
+		pgd_pa = sctx->session_pgd_pa;
+	else
+		pgd_pa = cpu_preserved_get_pgd(cpu);
+
+	if (!pgd_pa)
+		pgd_pa = READ_ONCE(arm64_caretaker_pgd_pa);
+
+	write_sysreg(0, ttbr0_el1);
+	if (pgd_pa && read_sysreg(ttbr1_el1) != pgd_pa) {
+		write_sysreg(pgd_pa, ttbr1_el1);
+		isb();
+		arm64_flush_host_tlb_local();
+	}
+
+	cpu_preserved_clean(cap);
+
+	if (smp_load_acquire(&cap->abi.cb.state) >= KVM_CARETAKER_STOPPING ||
+	    kvm_caretaker_should_exit(&cap->vcpu)) {
+		local_daif_mask();
+		isb();
+
+		caretaker_gic_drain_iar();
+		gicv3_caretaker_clear_active_priorities();
+		write_sysreg_s(0, SYS_ICC_IGRPEN1_EL1);
+		write_sysreg_s(0, SYS_ICC_PMR_EL1);
+		write_sysreg_s(0, SYS_ICC_BPR1_EL1);
+		gicv3_caretaker_clear_sgi();
+		dsb(sy);
+		isb();
+	}
+
+	arm64_caretaker_detach_serialize(cap);
+
+	arm64_caretaker_restore_ptrauth(&cap->ptrauth_keys);
+	caretaker_restore_host_el2();
+}
+
+static struct kvm_caretaker_ops arm64_caretaker_ops __cpu_preserved_data = {
+	.enter_guest = arm64_caretaker_op_enter,
+	.decode_exit = arm64_caretaker_op_decode_exit,
+	.handle_arch_exit = arm64_caretaker_op_handle_exit,
+	.advance_rip = arm64_caretaker_op_advance_rip,
+	.arm_timer = arm64_caretaker_op_arm_timer,
+	.disarm_timer = arm64_caretaker_op_disarm_timer,
+	.pre_run = arm64_caretaker_op_pre_run,
+	.post_run = arm64_caretaker_op_post_run,
+};
+
+static __caretaker_text enum oncore_exit_reason
+caretaker_arch_run_page(struct caretaker_arm64_page *cap, u64 deadline_ticks)
+{
+	enum oncore_exit_reason reason;
+	int cpu;
+
+	if (!cap)
+		return ONCORE_EXIT_ERROR;
+
+	cpu_preserved_inval(&cap->abi.cb);
+	cpu = cap->abi.cb.pcpu_id;
+	if (cpu < 0 || cpu >= ARRAY_SIZE(arm64_caretaker_faults))
+		cpu = arm64_caretaker_get_pcpu();
+
+	cap->abi.cb.pcpu_id = cpu;
+	if (cmpxchg(&cap->abi.cb.state, KVM_CARETAKER_PAUSED,
+		    KVM_CARETAKER_RUNNING) != KVM_CARETAKER_PAUSED ||
+	    cpu_preserved_should_exit(cpu)) {
+		arm64_caretaker_detach_serialize(cap);
+		smp_store_release(&cap->abi.cb.state, KVM_CARETAKER_STOPPED);
+		cpu_preserved_clean(&cap->abi.cb);
+		return ONCORE_EXIT_ATTACH_SIGNALED;
+	}
+	cpu_preserved_clean(&cap->abi.cb);
+
+	cap->vcpu.ops = &arm64_caretaker_ops;
+
+	reason = kvm_caretaker_vcpu_run(&cap->vcpu, deadline_ticks);
+
+	cpu_preserved_inval(&cap->abi.cb);
+	if (reason == ONCORE_EXIT_ATTACH_SIGNALED ||
+	    reason == ONCORE_EXIT_ERROR ||
+	    cpu_preserved_should_exit(cpu) ||
+	    cmpxchg(&cap->abi.cb.state, KVM_CARETAKER_RUNNING,
+		    KVM_CARETAKER_PAUSED) != KVM_CARETAKER_RUNNING) {
+		reason = ONCORE_EXIT_ATTACH_SIGNALED;
+		arm64_caretaker_detach_serialize(cap);
+		smp_store_release(&cap->abi.cb.state, KVM_CARETAKER_STOPPED);
+		cpu_preserved_clean(&cap->abi.cb);
+	} else {
+		cpu_preserved_clean(&cap->abi.cb);
+	}
+
+	return reason;
+}
+
+__caretaker_text enum oncore_exit_reason
+kvm_arch_vcpu_caretaker_run(void *data, u64 deadline_ticks)
+{
+	struct kvm_caretaker_cb_ser *cb = data;
+
+	/*
+	 * @data is always a struct kvm_caretaker_cb_ser: kvm_caretaker_vcpu_preserve()
+	 * installs it with oncore_job_set_data() before activating the job.
+	 */
+	if (!cb)
+		return ONCORE_EXIT_ERROR;
+
+	return caretaker_arch_run_page(container_of(cb,
+						    struct caretaker_arm64_page,
+						    abi.cb),
+				       deadline_ticks);
+}
+
+void kvm_arch_vcpu_luo_pre_retrieve_caretaker(struct kvm_vcpu *vcpu,
+					      struct kvm_vcpu_ser *ser)
+{
+	if ((ser->flags & KVM_VCPU_LUO_FLAG_CARETAKER) && ser->cb.phys) {
+		struct kvm_caretaker_arch_ser *abi = phys_to_virt(ser->cb.phys);
+		int pcpu = abi->cb.pcpu_id;
+
+		if (pcpu >= 0) {
+			kvm_caretaker_wait_for_attach(&abi->cb, pcpu);
+			cpu_preserved_inval(abi);
+			if (ser->arch_state.phys) {
+				struct kvm_vcpu_arch_ser *state =
+					phys_to_virt(ser->arch_state.phys);
+
+				cpu_preserved_inval(state);
+				cpu_preserved_inval_sz(state,
+						       struct_size(state, sysregs,
+								   state->num_sysregs));
+			}
+		}
+	}
+}
+
+void kvm_arch_vcpu_luo_attach_caretaker(struct kvm_vcpu *vcpu,
+					struct kvm_vcpu_ser *ser)
+{
+	if ((ser->flags & KVM_VCPU_LUO_FLAG_CARETAKER) && ser->cb.phys) {
+		struct kvm_caretaker_arch_ser *abi = phys_to_virt(ser->cb.phys);
+
+		arm64_caretaker_sync_vcpu(vcpu, abi);
+	}
+
+	kvm_caretaker_post_attach_vcpu(vcpu);
+}
+
+void arm64_kvm_caretaker_unpreserve(struct kvm_vcpu_ser *ser)
+{
+	if (ser->cb.phys) {
+		kho_unpreserve_free(phys_to_virt(ser->cb.phys));
+		ser->cb.phys = 0;
+	}
+}
+
+void arm64_kvm_caretaker_finish(struct kvm_vcpu_ser *ser)
+{
+	if (ser->cb.phys) {
+		kho_restore_free(phys_to_virt(ser->cb.phys));
+		ser->cb.phys = 0;
+	}
+}
diff --git a/arch/arm64/kvm/kvm_luo.c b/arch/arm64/kvm/kvm_luo.c
index 59c56836cda8..ae8baef49f97 100644
--- a/arch/arm64/kvm/kvm_luo.c
+++ b/arch/arm64/kvm/kvm_luo.c
@@ -6,6 +6,7 @@
  * ARM64 KVM LUO preservation and retrieval handlers.
  */
 
+#include <linux/cpu_preserve.h>
 #include <linux/kexec_handover.h>
 #include <linux/kho/abi/kvm_arm64.h>
 #include <linux/kvm_host.h>
@@ -17,15 +18,100 @@
 #include <kvm/arm_arch_timer.h>
 #include <kvm/arm_vgic.h>
 
+#include "caretaker.h"
 #include "sys_regs.h"
 #include "vgic/vgic.h"
 
+#ifdef CONFIG_KVM_CARETAKER
+struct arm64_stage2_kho_walk {
+	struct kvm_kho_folios_ser *kp;
+	unsigned int count;
+};
+
+static int stage2_kho_visitor(const struct kvm_pgtable_visit_ctx *ctx,
+			      enum kvm_pgtable_walk_flags visit)
+{
+	struct arm64_stage2_kho_walk *w = ctx->arg;
+
+	if (kvm_pte_valid(ctx->old) && ctx->level != KVM_PGTABLE_LAST_LEVEL &&
+	    FIELD_GET(KVM_PTE_TYPE, ctx->old) == KVM_PTE_TYPE_TABLE) {
+		u64 phys = kvm_pte_to_phys(ctx->old);
+		struct page *p = phys_to_page(phys);
+
+		if (!p)
+			return 0;
+
+		if (!w->kp) {
+			w->count++;
+			return 0;
+		}
+
+		if (w->kp->nr_folios >= w->count)
+			return -ENOSPC;
+
+		if (kho_preserve_folio(page_folio(p)))
+			return -ENOMEM;
+
+		w->kp->folios_pa[w->kp->nr_folios++] = phys;
+	}
+	return 0;
+}
+#endif
+
 int kvm_arch_vm_luo_preserve(struct kvm *kvm, struct kvm_luo_ser *ser)
 {
+#ifdef CONFIG_KVM_CARETAKER
+	struct kvm_s2_mmu *mmu = &kvm->arch.mmu;
+	struct arm64_stage2_kho_walk walk = {};
+	struct kvm_pgtable_walker walker = {
+		.cb = stage2_kho_visitor,
+		.flags = KVM_PGTABLE_WALK_TABLE_PRE,
+		.arg = &walk,
+	};
+	struct kvm_kho_folios_ser *kp;
+	int ret;
+#endif
+
 	ser->type = kvm_phys_shift(&kvm->arch.mmu);
 	if (kvm_vm_is_protected(kvm))
 		ser->type |= KVM_VM_TYPE_ARM_PROTECTED;
 
+#ifdef CONFIG_KVM_CARETAKER
+	kvm->caretaker_vm = NULL;
+
+	if (mmu->pgd_phys)
+		walk.count++;
+	if (mmu->pgt) {
+		ret = kvm_pgtable_walk(mmu->pgt, 0, BIT(mmu->pgt->ia_bits), &walker);
+		if (ret)
+			return ret;
+	}
+
+	kp = kvm_kho_folios_alloc(walk.count);
+	if (IS_ERR(kp))
+		return PTR_ERR(kp);
+	walk.kp = kp;
+
+	if (mmu->pgd_phys) {
+		ret = kho_preserve_folio(page_folio(phys_to_page(mmu->pgd_phys)));
+		if (ret) {
+			kvm_kho_folios_unpreserve(kp);
+			return ret;
+		}
+		kp->folios_pa[kp->nr_folios++] = mmu->pgd_phys;
+	}
+
+	if (mmu->pgt) {
+		ret = kvm_pgtable_walk(mmu->pgt, 0, BIT(mmu->pgt->ia_bits), &walker);
+		if (ret) {
+			kvm_kho_folios_unpreserve(kp);
+			return ret;
+		}
+	}
+
+	kvm->kho_folios = kp;
+	KHOSER_STORE_PTR(ser->kho_folios, kp);
+#endif
 	return 0;
 }
 
@@ -36,6 +122,10 @@ int kvm_arch_vm_luo_retrieve(struct kvm *kvm, struct kvm_luo_ser *ser)
 
 void kvm_arch_vm_luo_unpreserve(struct kvm *kvm, struct kvm_luo_ser *ser)
 {
+#ifdef CONFIG_KVM_CARETAKER
+	if (kvm)
+		kvm->caretaker_vm = NULL;
+#endif
 }
 
 void kvm_arch_vm_luo_finish(struct kvm_luo_ser *ser)
@@ -119,6 +209,17 @@ int kvm_arch_vcpu_luo_preserve(struct kvm_vcpu *vcpu, struct kvm_vcpu_ser *ser)
 	kfree(indices);
 
 	KHOSER_STORE_PTR(ser->arch_state, state);
+
+	if (ser->flags & KVM_VCPU_LUO_FLAG_CARETAKER) {
+		int ret = arm64_kvm_caretaker_preserve(vcpu, ser);
+
+		if (ret) {
+			kho_unpreserve_free(state);
+			ser->arch_state.phys = 0;
+			return ret;
+		}
+	}
+
 	return 0;
 }
 
@@ -186,6 +287,7 @@ int kvm_arch_vcpu_luo_retrieve(struct kvm_vcpu *vcpu, struct kvm_vcpu_ser *ser)
 
 void kvm_arch_vcpu_luo_unpreserve(struct kvm_vcpu_ser *ser)
 {
+	arm64_kvm_caretaker_unpreserve(ser);
 	if (ser->arch_state.phys) {
 		kho_unpreserve_free(phys_to_virt(ser->arch_state.phys));
 		ser->arch_state.phys = 0;
@@ -194,6 +296,7 @@ void kvm_arch_vcpu_luo_unpreserve(struct kvm_vcpu_ser *ser)
 
 void kvm_arch_vcpu_luo_finish(struct kvm_vcpu_ser *ser)
 {
+	arm64_kvm_caretaker_finish(ser);
 	if (ser->arch_state.phys) {
 		kho_restore_free(phys_to_virt(ser->arch_state.phys));
 		ser->arch_state.phys = 0;
-- 
2.55.0.1082.g2b9226bbc0-goog