[PATCH v3 00/23] KVM: arm64 on s390 System Register Handling

[PATCH v3 00/23] KVM: arm64 on s390 System Register Handling

[Steffen Eiden]

Add system register handling for KVM/arm64 on s390. Share KVM/arm64
code, introduce ARM guest management functions for s390 hosts, and
implement host system register & exception handling.

This is the second series on the way to arm64 guests on s390. The first
series laid the KVM foundation (basic module, vCPU run, simple page
faults) and is included in v7.4. This series builds the system register
layer on top and closes the biggest of the remaining gaps towards a
functional guest support.

This series requires 5 fixup patches not yet merged in order to build
clean.

[PATCH 0/3] KVM: s390: arm64: Fixes for arm64 KVM support [0]
[PATCH v2] s390: Guard emitting assembler macros for insn-common-asm.h [1]
[PATCH] KVM: s390: arm64: Tolerate unused functions in extracted source [2]

The series applies without those patches on 7.4-rc1 but the build will
fail or trigger warnings w/o those patches.

Changes in detail:

arm64:

Refactor the id register fields in arch_vcpu into a dedicated structure
such that it can be reused by s390. Refactor detection of the number of
sysregs. Mark sysreg related code as shared.

A few code pieces are moved inside arm headers/code to reduce the amount
of shared sections. This adds a bit of churn in the diff but reduces the
shared marker __churn__ at the end.

The shared regions deliberately contain more code and more headers than
s390 currently needs. Larger contiguous regions mean less churn whenever
the surrounding arm64 code changes. Some of this code will be used by
later series anyways. Some of the shared code will probably never be
reused by s390 at all; I trust the compiler to remove it. The reduced
churn outweighs this drawback. Since unused function checking is
disabled for shared code [3], sharing more than required does not
introduce warnings once everything is merged.

s390:

Add s390 instruction support for ARM guest management: easr/sasr for
system register access, QAAF for feature queries, and ptff extensions
for guest time handling.

Implement complete sysreg handling for s390 including feature
sanitisation, register enumeration and access, exception injection,
and finalized page fault handling.

Emulation:

Guest accesses to ID registers are trapped and served from a sanitized
copy of the QAAF query results, using the ID register sanitisation and
finalisation infrastructure shared with arm64. Userspace sees and
controls guest ID registers with the usual semantics; after the first
vCPU run the values are final. All other system registers live in the
SAE save area and are accessed by the host via EASR/SASR, either on
guest traps or through the one-reg/two-reg UAPI. The save area itself is
initialized from the QAAF-provided initial save area.

Exception handling:

Unsupported or invalid guest actions now raise proper arm64 exceptions
instead of failing exits. Page faults beyond the guest's address limits
are injected as address size faults or synchronous external aborts,
mirroring arm64's kvm_handle_guest_abort(); injected faults re-enter the
guest with the pending exception instead of exiting to userspace.

Features and userspace:

SVE is offered to guests at 128-bit vector length, backed by s390's
Vector Extension, and pointer authentication is passed through where
hardware support exists. The AEF capabilities are reported under
/sys/firmware/aef/, including the raw QAAF query results; the interface
is documented in Documentation/ABI.

For merging we will need a feature branch.

A branch  with patches [0,1,2] + this series is available at
kernel.org [3].

	Steffen

[0] https://lore.kernel.org/all/20261103-s390_arm64_kvm_fixups-v1-0-21c36fedd878@linux.ibm.com/
[1] https://lore.kernel.org/all/20261105-insn-v2-1-ff27987aa212@linux.ibm.com
[2] https://lore.kernel.org/all/20261105-series1-fixups2-v1-1-55f59e6c4c7b@linux.ibm.com
[3] https://git.kernel.org/pub/scm/linux/kernel/git/kvms390/linux.git/log/?h=sae

---
Changes in v3:
- rebase into 7.4-rc2 (including the first batch of arm on s390 patches)
- guard style changed from `#ifdef ARM64_S390_COMMON` to `#if ARM64_S390_COMMON == 1`
- Improve and clarify commit messages
 arm64:
- generate OSLSR definitions
- consolidate more shared sections by:
  - moving some code in beforehand
  - deliberately adding more code/header than required.
    due to [3] there will not be any unused warnings when merged.
- reduce sharing churn

s390
- streamline sasr/easr instruction generation and use
- aef_info() aggregate removed; new minimal aef_usable()/aef_gisrsa() API
- qaaf enum numbering reorganized, on reserved field was incorrectly
  identified as a register.
- aef.c no longer aggregates or gates on CONFIG_KVM_S390_ARM64.
- spectre.c stubs replaced with real queries of sanitized ID regs.
- Many new sys_reg_descs entries
- QAAF init save area: save area now memcpy'd from cached aef_gisrsa()
  instead of kvm_vcpu_init_save_area()
- improved pagefault handling
- reworked SVE helpers
- fixed various smaller bugs
- Link to v2: https://lore.kernel.org/all/20260831145536.913567-1-seiden@linux.ibm.com/

Changes in v2:
- rework the entire sharing code concept to use marker instead of moving
  code.

---
Steffen Eiden (23):
      KVM: arm64: Refactor idreg caching into dedicated structure
      KVM: arm64: Extract number of sys_reg_desc into a constant
      arm64: sysreg: Convert OSLSR to automatic register generation
      arm64: Prepare headers for sysreg sharing with s390
      arm64: Share system register related headers and functions with s390
      KVM: s390: arm64: Prepare for sharing arm64 SVE and ftr-reg code
      KVM: arm64: Prepare sys_regs.c for sharing with s390
      KVM: arm64: Share system register code with s390
      s390: tools: Share arm64/kvm headers
      s390: Add range check asm macro
      s390: Read/write ARM system registers
      s390: Query arm guest time
      KVM: s390: arm64: Query Available Arm features
      KVM: s390: arm64: Verify machine feature set before loading
      KVM: s390: arm64: Implement feature sanitisation
      KVM: s390: arm64: Implement arm sysreg managing infrastructure
      KVM: s390: arm64: Integrate sysreg into the host
      KVM: s390: arm64: Use QAAF init save area
      KVM: s390: arm64: Implement exception injection
      KVM: s390: arm64: Finalize page fault handling
      KVM: s390: arm64: Implement SVE for arm guests
      KVM: s390: arm64: Promote PTRAUTH capability
      s390: Report AEF features to sysfs

 Documentation/ABI/testing/sysfs-firmware-aef |  70 +++
 arch/arm64/include/asm/cache.h               |   3 +
 arch/arm64/include/asm/cputype.h             |   3 +
 arch/arm64/include/asm/kvm_emulate.h         |   5 +-
 arch/arm64/include/asm/kvm_host.h            |  64 ++-
 arch/arm64/include/asm/spectre.h             |  13 +-
 arch/arm64/include/asm/sysreg.h              |  12 +-
 arch/arm64/kvm/arm.c                         |   3 +
 arch/arm64/kvm/config.c                      |  10 +-
 arch/arm64/kvm/guest.c                       |  11 +-
 arch/arm64/kvm/hyp/nvhe/pkvm.c               |   7 +-
 arch/arm64/kvm/hyp/nvhe/sys_regs.c           |   4 +-
 arch/arm64/kvm/sys_regs.c                    | 109 ++--
 arch/arm64/kvm/sys_regs.h                    |   1 +
 arch/arm64/tools/sysreg                      |  14 +
 arch/s390/include/arm64/kvm_hyp.h            |  18 +
 arch/s390/include/arm64/kvm_nested.h         |   5 +
 arch/s390/include/arm64/sys_regs.h           |   9 +
 arch/s390/include/asm/aef.h                  |  12 +
 arch/s390/include/asm/insn-common-asm.h      |  28 +
 arch/s390/include/asm/kvm_host_arm64.h       | 237 +++++++-
 arch/s390/include/asm/kvm_host_arm64_types.h | 115 ++++
 arch/s390/include/asm/sae-asm.h              |  45 ++
 arch/s390/include/asm/sae.h                  |  93 +++
 arch/s390/include/asm/timex.h                |  53 ++
 arch/s390/kernel/Makefile                    |   2 +-
 arch/s390/kernel/aef.c                       | 207 +++++++
 arch/s390/kernel/dis.c                       |   1 +
 arch/s390/kvm/arm64/Makefile                 |   7 +-
 arch/s390/kvm/arm64/Makefile.gen             |   3 +-
 arch/s390/kvm/arm64/arm.c                    | 101 +++-
 arch/s390/kvm/arm64/exception.c              | 109 ++++
 arch/s390/kvm/arm64/feature.c                | 188 ++++++
 arch/s390/kvm/arm64/feature.h                |  97 ++++
 arch/s390/kvm/arm64/guest.c                  | 268 ++++++++-
 arch/s390/kvm/arm64/handle_exit.c            |   1 +
 arch/s390/kvm/arm64/inject_fault.c           |  65 ++-
 arch/s390/kvm/arm64/mmu.c                    |  74 ++-
 arch/s390/kvm/arm64/qaaf.c                   |  85 +++
 arch/s390/kvm/arm64/qaaf.h                   |  12 +
 arch/s390/kvm/arm64/reset.c                  |  71 +++
 arch/s390/kvm/arm64/spectre.c                |  51 ++
 arch/s390/kvm/arm64/sys_regs.c               | 826 +++++++++++++++++++++++++++
 arch/s390/kvm/arm64/trace.h                  |  50 ++
 arch/s390/tools/Makefile.arm64h              |  11 +-
 arch/s390/tools/opcodes.txt                  |   3 +
 include/kvm/arm_arch_timer.h                 |   3 +
 47 files changed, 3030 insertions(+), 149 deletions(-)

base-commit: bc71431221f5ba8bb33918399552636e059c2c45
change-id: 20261106-sysreg-d6adf0cd34d4
prerequisite-message-id: 20261103-s390_arm64_kvm_fixups-v1-0-21c36fedd878@linux.ibm.com
prerequisite-patch-id: 93d5fe93939c707a0d0c57a847270c706a9bae29
prerequisite-patch-id: af10f106ab88b2632aa77e739bf8f64e8f3b5499
prerequisite-patch-id: cf8d158bdbb43de4ae2d524efe9005f9e42ead1f
prerequisite-message-id: 20261105-insn-v2-1-ff27987aa212@linux.ibm.com
prerequisite-patch-id: ed28ef94c9056cb81bb738f1b54745d012cb613f
prerequisite-message-id: 20261105-series1-fixups2-v1-1-55f59e6c4c7b@linux.ibm.com
prerequisite-patch-id: a1648acb9dedea7550a5bfcce937a21016d569a8

[PATCH v3 01/23] KVM: arm64: Refactor idreg caching into dedicated structure

[Steffen Eiden]

Move VM-wide ID register emulation fields from struct kvm_arch into a
new struct kvm_vm_id_regs. The new struct is placed in a shared region.
Update all users to use the new structure.

No functional changes.

Co-developed-by: Nina Schoetterl-Glausch <nsg@linux.ibm.com>
Signed-off-by: Nina Schoetterl-Glausch <nsg@linux.ibm.com>
Co-developed-by: Andreas Grapentin <gra@linux.ibm.com>
Signed-off-by: Andreas Grapentin <gra@linux.ibm.com>
Signed-off-by: Steffen Eiden <seiden@linux.ibm.com>
---
 arch/arm64/include/asm/kvm_host.h  | 50 ++++++++++++++++++++------------------
 arch/arm64/kvm/config.c            | 10 ++++----
 arch/arm64/kvm/hyp/nvhe/pkvm.c     |  7 +++---
 arch/arm64/kvm/hyp/nvhe/sys_regs.c |  4 +--
 arch/arm64/kvm/sys_regs.c          |  2 +-
 5 files changed, 39 insertions(+), 34 deletions(-)

diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h
index 97332ffcb130..7f051dad79c7 100644
--- a/arch/arm64/include/asm/kvm_host.h
+++ b/arch/arm64/include/asm/kvm_host.h
@@ -105,6 +105,25 @@ struct vcpu_reset_state {
 	bool		reset;
 };
 
+struct kvm_vm_id_regs {
+	/*
+	 * Emulated CPU ID registers per VM
+	 * (Op0, Op1, CRn, CRm, Op2) of the ID registers to be saved in it
+	 * is (3, 0, 0, crm, op2), where 1<=crm<8, 0<=op2<8.
+	 *
+	 * These emulated idregs are VM-wide, but accessed from the context of a vCPU.
+	 * Atomic access to multiple idregs are guarded by kvm_arch.config_lock.
+	 */
+#define IDREG_IDX(id)		(((sys_reg_CRm(id) - 1) << 3) | sys_reg_Op2(id))
+#define KVM_ARM_ID_REG_NUM	(IDREG_IDX(sys_reg(3, 0, 0, 7, 7)) + 1)
+	u64 ftr_reg[KVM_ARM_ID_REG_NUM];
+
+	u64 midr_el1;
+	u64 revidr_el1;
+	u64 aidr_el1;
+	u64 ctr_el0;
+};
+
 #endif /* ARM64_S390_COMMON == 1 */
 
 #define KVM_DIRTY_LOG_MANUAL_CAPS   (KVM_DIRTY_LOG_MANUAL_PROTECT_ENABLE | \
@@ -478,22 +497,7 @@ struct kvm_arch {
 	struct kvm_smccc_features smccc_feat;
 	struct maple_tree smccc_filter;
 
-	/*
-	 * Emulated CPU ID registers per VM
-	 * (Op0, Op1, CRn, CRm, Op2) of the ID registers to be saved in it
-	 * is (3, 0, 0, crm, op2), where 1<=crm<8, 0<=op2<8.
-	 *
-	 * These emulated idregs are VM-wide, but accessed from the context of a vCPU.
-	 * Atomic access to multiple idregs are guarded by kvm_arch.config_lock.
-	 */
-#define IDREG_IDX(id)		(((sys_reg_CRm(id) - 1) << 3) | sys_reg_Op2(id))
-#define KVM_ARM_ID_REG_NUM	(IDREG_IDX(sys_reg(3, 0, 0, 7, 7)) + 1)
-	u64 id_regs[KVM_ARM_ID_REG_NUM];
-
-	u64 midr_el1;
-	u64 revidr_el1;
-	u64 aidr_el1;
-	u64 ctr_el0;
+	struct kvm_vm_id_regs id_regs;
 
 	/* Masks for VNCR-backed and general EL2 sysregs */
 	struct kvm_sysreg_masks	*sysreg_masks;
@@ -1634,19 +1638,19 @@ void kvm_arm_vcpu_power_off(struct kvm_vcpu *vcpu);
 bool kvm_arm_vcpu_stopped(struct kvm_vcpu *vcpu);
 bool kvm_pkvm_vcpu_is_powered_off(struct kvm_vcpu *vcpu);
 
-static inline u64 *__vm_id_reg(struct kvm_arch *ka, u32 reg)
+static inline u64 *__vm_id_reg(struct kvm_vm_id_regs *id_regs, u32 reg)
 {
 	switch (reg) {
 	case sys_reg(3, 0, 0, 1, 0) ... sys_reg(3, 0, 0, 7, 7):
-		return &ka->id_regs[IDREG_IDX(reg)];
+		return &id_regs->ftr_reg[IDREG_IDX(reg)];
 	case SYS_CTR_EL0:
-		return &ka->ctr_el0;
+		return &id_regs->ctr_el0;
 	case SYS_MIDR_EL1:
-		return &ka->midr_el1;
+		return &id_regs->midr_el1;
 	case SYS_REVIDR_EL1:
-		return &ka->revidr_el1;
+		return &id_regs->revidr_el1;
 	case SYS_AIDR_EL1:
-		return &ka->aidr_el1;
+		return &id_regs->aidr_el1;
 	default:
 		WARN_ON_ONCE(1);
 		return NULL;
@@ -1654,7 +1658,7 @@ static inline u64 *__vm_id_reg(struct kvm_arch *ka, u32 reg)
 }
 
 #define kvm_read_vm_id_reg(kvm, reg)					\
-	({ u64 __val = *__vm_id_reg(&(kvm)->arch, reg); __val; })
+	({ u64 __val = *__vm_id_reg(&(kvm)->arch.id_regs, reg); __val; })
 
 void kvm_set_vm_id_reg(struct kvm *kvm, u32 reg, u64 val);
 
diff --git a/arch/arm64/kvm/config.c b/arch/arm64/kvm/config.c
index adaf26ab93b8..49bbe68c0b9e 100644
--- a/arch/arm64/kvm/config.c
+++ b/arch/arm64/kvm/config.c
@@ -1467,11 +1467,11 @@ static bool idreg_apply_deps(struct kvm *kvm, const struct reg_feat_map_desc *r)
 			resx.res0 |= reg_feat_map_bits(&r->bit_feat_map[i]);
 	}
 
-	val = kvm->arch.id_regs[r->idreg_idx];
-	kvm->arch.id_regs[r->idreg_idx] &= ~resx.res0;
-	kvm->arch.id_regs[r->idreg_idx] |= resx.res1;
+	val = kvm->arch.id_regs.ftr_reg[r->idreg_idx];
+	kvm->arch.id_regs.ftr_reg[r->idreg_idx] &= ~resx.res0;
+	kvm->arch.id_regs.ftr_reg[r->idreg_idx] |= resx.res1;
 
-	return val != kvm->arch.id_regs[r->idreg_idx];
+	return val != kvm->arch.id_regs.ftr_reg[r->idreg_idx];
 }
 
 void kvm_idregs_enforce_dependencies(struct kvm *kvm)
@@ -1488,7 +1488,7 @@ void kvm_idregs_enforce_dependencies(struct kvm *kvm)
 
 static bool idreg_feat_match(struct kvm *kvm, const struct reg_bits_to_feat_map *map)
 {
-	u64 regval = kvm->arch.id_regs[map->regidx];
+	u64 regval = kvm->arch.id_regs.ftr_reg[map->regidx];
 	u64 regfld = (regval >> map->shift) & GENMASK(map->width - 1, 0);
 
 	if (map->sign) {
diff --git a/arch/arm64/kvm/hyp/nvhe/pkvm.c b/arch/arm64/kvm/hyp/nvhe/pkvm.c
index fc0475f0877d..41ee83290697 100644
--- a/arch/arm64/kvm/hyp/nvhe/pkvm.c
+++ b/arch/arm64/kvm/hyp/nvhe/pkvm.c
@@ -399,7 +399,7 @@ static void pkvm_init_features_from_host(struct pkvm_hyp_vm *hyp_vm, const struc
 	DECLARE_BITMAP(allowed_features, KVM_VCPU_MAX_FEATURES);
 
 	/* CTR_EL0 is always under host control, even for protected VMs. */
-	hyp_vm->kvm.arch.ctr_el0 = host_kvm->arch.ctr_el0;
+	hyp_vm->kvm.arch.id_regs.ctr_el0 = host_kvm->arch.id_regs.ctr_el0;
 
 	/*
 	 * pKVM runs only GICv3. Anything else becomes 0, the model of a VM
@@ -419,7 +419,7 @@ static void pkvm_init_features_from_host(struct pkvm_hyp_vm *hyp_vm, const struc
 			    KVM_VCPU_MAX_FEATURES);
 
 		if (test_bit(KVM_ARCH_FLAG_WRITABLE_IMP_ID_REGS, &host_arch_flags))
-			hyp_vm->kvm.arch.midr_el1 = host_kvm->arch.midr_el1;
+			hyp_vm->kvm.arch.id_regs.midr_el1 = host_kvm->arch.id_regs.midr_el1;
 
 		goto out;
 	}
@@ -574,7 +574,8 @@ static int vm_copy_id_regs(struct pkvm_hyp_vcpu *hyp_vcpu)
 	if (test_and_set_bit(KVM_ARCH_FLAG_ID_REGS_INITIALIZED, &kvm->arch.flags))
 		return 0;
 
-	memcpy(kvm->arch.id_regs, host_kvm->arch.id_regs, sizeof(kvm->arch.id_regs));
+	memcpy(kvm->arch.id_regs.ftr_reg, host_kvm->arch.id_regs.ftr_reg,
+	       sizeof(kvm->arch.id_regs.ftr_reg));
 
 	return 0;
 }
diff --git a/arch/arm64/kvm/hyp/nvhe/sys_regs.c b/arch/arm64/kvm/hyp/nvhe/sys_regs.c
index 32e82653ebf7..10b144e50feb 100644
--- a/arch/arm64/kvm/hyp/nvhe/sys_regs.c
+++ b/arch/arm64/kvm/hyp/nvhe/sys_regs.c
@@ -296,7 +296,7 @@ static u64 read_id_reg(const struct kvm_vcpu *vcpu,
 		return 0;
 
 	if (reg >= sys_reg(3, 0, 0, 1, 0) && reg <= sys_reg(3, 0, 0, 7, 7))
-		return kvm->arch.id_regs[IDREG_IDX(reg)];
+		return kvm->arch.id_regs.ftr_reg[IDREG_IDX(reg)];
 
 	return 0;
 }
@@ -607,7 +607,7 @@ void kvm_init_pvm_id_regs(struct kvm_vcpu *vcpu)
 	 * for protected VMs.
 	 */
 	for (r = sys_reg(3, 0, 0, 4, 0); r <= sys_reg(3, 0, 0, 7, 7); r += sys_reg(0, 0, 0, 0, 1))
-		ka->id_regs[IDREG_IDX(r)] = pvm_calc_id_reg(vcpu, r);
+		ka->id_regs.ftr_reg[IDREG_IDX(r)] = pvm_calc_id_reg(vcpu, r);
 
 	set_bit(KVM_ARCH_FLAG_ID_REGS_INITIALIZED, &kvm->arch.flags);
 }
diff --git a/arch/arm64/kvm/sys_regs.c b/arch/arm64/kvm/sys_regs.c
index 850eba167aad..0f0b0495b710 100644
--- a/arch/arm64/kvm/sys_regs.c
+++ b/arch/arm64/kvm/sys_regs.c
@@ -2543,7 +2543,7 @@ static int set_id_reg(struct kvm_vcpu *vcpu, const struct sys_reg_desc *rd,
 
 void kvm_set_vm_id_reg(struct kvm *kvm, u32 reg, u64 val)
 {
-	u64 *p = __vm_id_reg(&kvm->arch, reg);
+	u64 *p = __vm_id_reg(&kvm->arch.id_regs, reg);
 
 	lockdep_assert_held(&kvm->arch.config_lock);
 

-- 
2.56.0

[PATCH v3 02/23] KVM: arm64: Extract number of sys_reg_desc into a constant

[Steffen Eiden]

Define the constant `num_sys_reg_descs` which stores the number of
elements in the sys_reg_descs array. Replace all occurrences of
ARRAY_SIZE(sys_reg_descs) with the new constant.

This refactoring eases the integration of handling system registers for
arm on s390. s390 needs code from the shared code to define the
sys_reg_descs. But also the shared code needs the definition of the
sys_reg_array and we cannot forward declare arrays and use their size
before it is properly defined. This patch solves this chicken or egg
problem. For the time being s390 cannot process the full arm64 defined
sys_reg_descs array.

No functional changes.

Co-developed-by: Andreas Grapentin <gra@linux.ibm.com>
Signed-off-by: Andreas Grapentin <gra@linux.ibm.com>
Signed-off-by: Steffen Eiden <seiden@linux.ibm.com>
---
 arch/arm64/kvm/sys_regs.c | 26 ++++++++++++++------------
 1 file changed, 14 insertions(+), 12 deletions(-)

diff --git a/arch/arm64/kvm/sys_regs.c b/arch/arm64/kvm/sys_regs.c
index 0f0b0495b710..2b28b929f9f3 100644
--- a/arch/arm64/kvm/sys_regs.c
+++ b/arch/arm64/kvm/sys_regs.c
@@ -3976,6 +3976,8 @@ static const struct sys_reg_desc sys_reg_descs[] = {
 	EL2_REG(SP_EL2, NULL, reset_unknown, 0),
 };
 
+static const size_t num_sys_reg_descs = ARRAY_SIZE(sys_reg_descs);
+
 static bool handle_at_s1e01(struct kvm_vcpu *vcpu, struct sys_reg_params *p,
 			    const struct sys_reg_desc *r)
 {
@@ -5187,7 +5189,7 @@ static bool emulate_sys_reg(struct kvm_vcpu *vcpu,
 {
 	const struct sys_reg_desc *r;
 
-	r = find_reg(params, sys_reg_descs, ARRAY_SIZE(sys_reg_descs));
+	r = find_reg(params, sys_reg_descs, num_sys_reg_descs);
 	if (likely(r)) {
 		perform_access(vcpu, params, r);
 		return true;
@@ -5205,7 +5207,7 @@ static const struct sys_reg_desc *idregs_debug_find(struct kvm *kvm, loff_t pos)
 {
 	unsigned long i, idreg_idx = 0;
 
-	for (i = 0; i < ARRAY_SIZE(sys_reg_descs); i++) {
+	for (i = 0; i < num_sys_reg_descs; i++) {
 		const struct sys_reg_desc *r = &sys_reg_descs[i];
 
 		if (!is_vm_ftr_id_reg(reg_to_encoding(r)))
@@ -5268,7 +5270,7 @@ static const struct sys_reg_desc *sr_resx_find(struct kvm *kvm, loff_t pos)
 {
 	unsigned long i, sr_idx = 0;
 
-	for (i = 0; i < ARRAY_SIZE(sys_reg_descs); i++) {
+	for (i = 0; i < num_sys_reg_descs; i++) {
 		const struct sys_reg_desc *r = &sys_reg_descs[i];
 
 		if (r->reg < __SANITISED_REG_START__)
@@ -5370,7 +5372,7 @@ void kvm_reset_sys_regs(struct kvm_vcpu *vcpu)
 	struct kvm *kvm = vcpu->kvm;
 	unsigned long i;
 
-	for (i = 0; i < ARRAY_SIZE(sys_reg_descs); i++) {
+	for (i = 0; i < num_sys_reg_descs; i++) {
 		const struct sys_reg_desc *r = &sys_reg_descs[i];
 
 		if (!r->reset)
@@ -5594,8 +5596,8 @@ int kvm_arm_sys_reg_get_reg(struct kvm_vcpu *vcpu, const struct kvm_one_reg *reg
 	if ((reg->id & KVM_REG_ARM_COPROC_MASK) == KVM_REG_ARM_DEMUX)
 		return demux_c15_get(vcpu, reg->id, uaddr);
 
-	return kvm_sys_reg_get_user(vcpu, reg,
-				    sys_reg_descs, ARRAY_SIZE(sys_reg_descs));
+	return kvm_sys_reg_get_user(vcpu, reg, sys_reg_descs,
+				    num_sys_reg_descs);
 }
 
 int kvm_sys_reg_set_user(struct kvm_vcpu *vcpu, const struct kvm_one_reg *reg,
@@ -5634,8 +5636,8 @@ int kvm_arm_sys_reg_set_reg(struct kvm_vcpu *vcpu, const struct kvm_one_reg *reg
 	if ((reg->id & KVM_REG_ARM_COPROC_MASK) == KVM_REG_ARM_DEMUX)
 		return demux_c15_set(vcpu, reg->id, uaddr);
 
-	return kvm_sys_reg_set_user(vcpu, reg,
-				    sys_reg_descs, ARRAY_SIZE(sys_reg_descs));
+	return kvm_sys_reg_set_user(vcpu, reg, sys_reg_descs,
+				    num_sys_reg_descs);
 }
 
 static unsigned int num_demux_regs(void)
@@ -5723,7 +5725,7 @@ static int walk_sys_regs(struct kvm_vcpu *vcpu, u64 __user *uind)
 	int err;
 
 	i2 = sys_reg_descs;
-	end2 = sys_reg_descs + ARRAY_SIZE(sys_reg_descs);
+	end2 = sys_reg_descs + num_sys_reg_descs;
 
 	while (i2 != end2) {
 		err = walk_one_sys_reg(vcpu, i2++, &uind, &total);
@@ -5772,7 +5774,7 @@ int kvm_vm_ioctl_get_reg_writable_masks(struct kvm *kvm, struct reg_mask_range *
 	if (clear_user(masks, KVM_ARM_FEATURE_ID_RANGE_SIZE * sizeof(__u64)))
 		return -EFAULT;
 
-	for (int i = 0; i < ARRAY_SIZE(sys_reg_descs); i++) {
+	for (int i = 0; i < num_sys_reg_descs; i++) {
 		const struct sys_reg_desc *reg = &sys_reg_descs[i];
 		u32 encoding = reg_to_encoding(reg);
 		u64 val;
@@ -5957,7 +5959,7 @@ int __init kvm_sys_reg_table_init(void)
 	int ret = 0;
 
 	/* Make sure tables are unique and in order. */
-	valid &= check_sysreg_table(sys_reg_descs, ARRAY_SIZE(sys_reg_descs), true);
+	valid &= check_sysreg_table(sys_reg_descs, num_sys_reg_descs, true);
 	valid &= check_sysreg_table(cp14_regs, ARRAY_SIZE(cp14_regs), false);
 	valid &= check_sysreg_table(cp14_64_regs, ARRAY_SIZE(cp14_64_regs), false);
 	valid &= check_sysreg_table(cp15_regs, ARRAY_SIZE(cp15_regs), false);
@@ -5980,7 +5982,7 @@ int __init kvm_sys_reg_table_init(void)
 
 	check_feature_map();
 
-	for (i = 0; !ret && i < ARRAY_SIZE(sys_reg_descs); i++)
+	for (i = 0; !ret && i < num_sys_reg_descs; i++)
 		ret = populate_sysreg_config(sys_reg_descs + i, i);
 
 	for (i = 0; !ret && i < ARRAY_SIZE(sys_insn_descs); i++)

-- 
2.56.0

[PATCH v3 03/23] arm64: sysreg: Convert OSLSR to automatic register generation

[Steffen Eiden]

Convert OSLSR to automatic register generation as per DDI0601. As
OSLSR_OLSM is a composite field using bit 0 and bit 3 the combined
values are still defined manually. While at it rename
OSLSR_EL1_OSLM_IMPLEMENTED to OSLSR_EL1_OSLM_IMP to be consistent with
the rest of the system register values.

No functional changes.

Co-developed-by: Andreas Grapentin <gra@linux.ibm.com>
Signed-off-by: Andreas Grapentin <gra@linux.ibm.com>
Signed-off-by: Steffen Eiden <seiden@linux.ibm.com>
---
 arch/arm64/include/asm/sysreg.h | 12 +++++++-----
 arch/arm64/kvm/sys_regs.c       |  2 +-
 arch/arm64/tools/sysreg         | 14 ++++++++++++++
 3 files changed, 22 insertions(+), 6 deletions(-)

diff --git a/arch/arm64/include/asm/sysreg.h b/arch/arm64/include/asm/sysreg.h
index e3b60ee03023..2ec9ba9cd896 100644
--- a/arch/arm64/include/asm/sysreg.h
+++ b/arch/arm64/include/asm/sysreg.h
@@ -186,11 +186,13 @@
 #define SYS_DBGWCRn_EL1(n)		sys_reg(2, 0, 0, n, 7)
 #define SYS_MDRAR_EL1			sys_reg(2, 0, 1, 0, 0)
 
-#define SYS_OSLSR_EL1			sys_reg(2, 0, 1, 1, 4)
-#define OSLSR_EL1_OSLM_MASK		(BIT(3) | BIT(0))
-#define OSLSR_EL1_OSLM_NI		0
-#define OSLSR_EL1_OSLM_IMPLEMENTED	BIT(3)
-#define OSLSR_EL1_OSLK			BIT(1)
+#define OSLSR_EL1_OSLM_MASK (OSLSR_EL1_OSLM0_MASK | OSLSR_EL1_OSLM1_MASK)
+#define OSLSR_EL1_OSLM_NI						\
+	((OSLSR_EL1_OSLM1_NI << OSLSR_EL1_OSLM1_SHIFT) |		\
+	(OSLSR_EL1_OSLM0_VAL_0 << OSLSR_EL1_OSLM0_SHIFT))
+#define OSLSR_EL1_OSLM_IMP						\
+	((OSLSR_EL1_OSLM1_IMP << OSLSR_EL1_OSLM1_SHIFT) |		\
+	(OSLSR_EL1_OSLM0_VAL_0 << OSLSR_EL1_OSLM0_SHIFT))
 
 #define SYS_OSDLR_EL1			sys_reg(2, 0, 1, 3, 4)
 #define SYS_DBGPRCR_EL1			sys_reg(2, 0, 1, 4, 4)
diff --git a/arch/arm64/kvm/sys_regs.c b/arch/arm64/kvm/sys_regs.c
index 2b28b929f9f3..8c91de6a4097 100644
--- a/arch/arm64/kvm/sys_regs.c
+++ b/arch/arm64/kvm/sys_regs.c
@@ -3309,7 +3309,7 @@ static const struct sys_reg_desc sys_reg_descs[] = {
 	{ SYS_DESC(SYS_MDRAR_EL1), trap_raz_wi },
 	{ SYS_DESC(SYS_OSLAR_EL1), trap_oslar_el1 },
 	{ SYS_DESC(SYS_OSLSR_EL1), trap_oslsr_el1, reset_val, OSLSR_EL1,
-		OSLSR_EL1_OSLM_IMPLEMENTED, .set_user = set_oslsr_el1, },
+		OSLSR_EL1_OSLM_IMP, .set_user = set_oslsr_el1, },
 	{ SYS_DESC(SYS_OSDLR_EL1), trap_raz_wi },
 	{ SYS_DESC(SYS_DBGPRCR_EL1), trap_raz_wi },
 	{ SYS_DESC(SYS_DBGCLAIMSET_EL1), trap_raz_wi },
diff --git a/arch/arm64/tools/sysreg b/arch/arm64/tools/sysreg
index 997c62ab72e6..5d38f115bac1 100644
--- a/arch/arm64/tools/sysreg
+++ b/arch/arm64/tools/sysreg
@@ -122,6 +122,20 @@ Res0	63:1
 Field	0	OSLK
 EndSysreg
 
+Sysreg	OSLSR_EL1	2	0	1	1	4
+Res0	63:4
+UnsignedEnum	3	OSLM1
+	 0b0	NI
+	 0b1	IMP
+EndEnum
+UnsignedEnum	2	nTT
+EndEnum
+Field	1	OSLK
+UnsignedEnum	0	OSLM0
+	 0b0	VAL_0
+EndEnum
+EndSysreg
+
 Sysreg	SPMACCESSR_EL1	2	0	9	13	3
 UnsignedEnum	63:62	P31
 	0b00	TRAP_RW

-- 
2.56.0

[PATCH v3 04/23] arm64: Prepare headers for sysreg sharing with s390

[Steffen Eiden]

Move the get-spectre-state functions to a convenient location. Guard
GICv5 headers away from s390 for arm_arch_timer.h.

No functional changes.

Co-developed-by: Andreas Grapentin <gra@linux.ibm.com>
Signed-off-by: Andreas Grapentin <gra@linux.ibm.com>
Signed-off-by: Steffen Eiden <seiden@linux.ibm.com>
---
 arch/arm64/include/asm/spectre.h | 10 +++++-----
 include/kvm/arm_arch_timer.h     |  3 +++
 2 files changed, 8 insertions(+), 5 deletions(-)

diff --git a/arch/arm64/include/asm/spectre.h b/arch/arm64/include/asm/spectre.h
index 296ae3420bfd..2ecc9eddbebb 100644
--- a/arch/arm64/include/asm/spectre.h
+++ b/arch/arm64/include/asm/spectre.h
@@ -26,6 +26,11 @@ enum mitigation_state {
 	SPECTRE_VULNERABLE,
 };
 
+enum mitigation_state arm64_get_spectre_v2_state(void);
+enum mitigation_state arm64_get_spectre_v4_state(void);
+enum mitigation_state arm64_get_meltdown_state(void);
+enum mitigation_state arm64_get_spectre_bhb_state(void);
+
 struct pt_regs;
 struct task_struct;
 
@@ -81,21 +86,16 @@ static __always_inline void arm64_apply_bp_hardening(void)
 		d->fn();
 }
 
-enum mitigation_state arm64_get_spectre_v2_state(void);
 bool has_spectre_v2(const struct arm64_cpu_capabilities *cap, int scope);
 void spectre_v2_enable_mitigation(const struct arm64_cpu_capabilities *__unused);
 
 bool has_spectre_v3a(const struct arm64_cpu_capabilities *cap, int scope);
 void spectre_v3a_enable_mitigation(const struct arm64_cpu_capabilities *__unused);
 
-enum mitigation_state arm64_get_spectre_v4_state(void);
 bool has_spectre_v4(const struct arm64_cpu_capabilities *cap, int scope);
 void spectre_v4_enable_mitigation(const struct arm64_cpu_capabilities *__unused);
 void spectre_v4_enable_task_mitigation(struct task_struct *tsk);
 
-enum mitigation_state arm64_get_meltdown_state(void);
-
-enum mitigation_state arm64_get_spectre_bhb_state(void);
 bool is_spectre_bhb_affected(const struct arm64_cpu_capabilities *entry, int scope);
 extern bool __nospectre_bhb;
 u8 get_spectre_bhb_loop_value(void);
diff --git a/include/kvm/arm_arch_timer.h b/include/kvm/arm_arch_timer.h
index 80f96ea59f1a..dcffad87b88f 100644
--- a/include/kvm/arm_arch_timer.h
+++ b/include/kvm/arm_arch_timer.h
@@ -10,7 +10,10 @@
 #include <linux/clocksource.h>
 #include <linux/hrtimer.h>
 
+/* no GICv5 for s390 */
+#ifndef __s390__
 #include <linux/irqchip/arm-gic-v5.h>
+#endif /* !__s390__ */
 
 enum kvm_arch_timers {
 	TIMER_PTIMER,

-- 
2.56.0

[PATCH v3 05/23] arm64: Share system register related headers and functions with s390

[Steffen Eiden]

Mark more headers as shared or move definitions to an already shared
location if the context is similar. The new shared headers cover cover
CPU identification, Exception emulation, ID register state, Mitigation
state, and the kvm system register management infrastructure. Also add
definitions useful for later feature additions to reduce further churn.

With these headers s390 is able to implement system register handling
for its arm64 guests.

Co-developed-by: Andreas Grapentin <gra@linux.ibm.com>
Signed-off-by: Andreas Grapentin <gra@linux.ibm.com>
Signed-off-by: Steffen Eiden <seiden@linux.ibm.com>
---
 arch/arm64/include/asm/cache.h       |  3 +++
 arch/arm64/include/asm/cputype.h     |  3 +++
 arch/arm64/include/asm/kvm_emulate.h |  5 ++++-
 arch/arm64/include/asm/kvm_host.h    | 14 +++++++-------
 arch/arm64/include/asm/spectre.h     |  3 +++
 arch/arm64/kvm/sys_regs.h            |  1 +
 6 files changed, 21 insertions(+), 8 deletions(-)

diff --git a/arch/arm64/include/asm/cache.h b/arch/arm64/include/asm/cache.h
index 10a7ffadee3d..ee3078c68768 100644
--- a/arch/arm64/include/asm/cache.h
+++ b/arch/arm64/include/asm/cache.h
@@ -8,6 +8,7 @@
 #define L1_CACHE_SHIFT		(6)
 #define L1_CACHE_BYTES		(1 << L1_CACHE_SHIFT)
 
+#if ARM64_S390_COMMON == 1
 #define CLIDR_LOUU_SHIFT	27
 #define CLIDR_LOC_SHIFT		24
 #define CLIDR_LOUIS_SHIFT	21
@@ -25,6 +26,8 @@
 /* Ttypen, bits [2(n - 1) + 34 : 2(n - 1) + 33], for n = 1 to 7 */
 #define CLIDR_TTYPE_SHIFT(level)	(2 * ((level) - 1) + CLIDR_EL1_Ttypen_SHIFT)
 
+#endif /* ARM64_S390_COMMON == 1 */
+
 /*
  * Memory returned by kmalloc() may be used for DMA, so we must make
  * sure that all such allocations are cache aligned. Otherwise,
diff --git a/arch/arm64/include/asm/cputype.h b/arch/arm64/include/asm/cputype.h
index 6002d1d5f55f..a8225467cb09 100644
--- a/arch/arm64/include/asm/cputype.h
+++ b/arch/arm64/include/asm/cputype.h
@@ -5,6 +5,7 @@
 #ifndef __ASM_CPUTYPE_H
 #define __ASM_CPUTYPE_H
 
+#if ARM64_S390_COMMON == 1
 #define INVALID_HWID		ULONG_MAX
 
 #define MPIDR_UP_BITMASK	(0x1 << 30)
@@ -261,6 +262,8 @@
 #define MIDR_FUJITSU_ERRATUM_010001_MASK	(~MIDR_CPU_VAR_REV(1, 0))
 #define TCR_CLEAR_FUJITSU_ERRATUM_010001	(TCR_EL1_NFD1 | TCR_EL1_NFD0)
 
+#endif /* ARM64_S390_COMMON == 1 */
+
 #ifndef __ASSEMBLER__
 
 #include <asm/sysreg.h>
diff --git a/arch/arm64/include/asm/kvm_emulate.h b/arch/arm64/include/asm/kvm_emulate.h
index 334211b8b2ee..4ad190d4559b 100644
--- a/arch/arm64/include/asm/kvm_emulate.h
+++ b/arch/arm64/include/asm/kvm_emulate.h
@@ -23,6 +23,7 @@
 #include <asm/cputype.h>
 #include <asm/virt.h>
 
+#if ARM64_S390_COMMON == 1
 #define CURRENT_EL_SP_EL0_VECTOR	0x0
 #define CURRENT_EL_SP_ELx_VECTOR	0x200
 #define LOWER_EL_AArch64_VECTOR		0x400
@@ -35,6 +36,8 @@ enum exception_type {
 	except_type_serror	= 0x180,
 };
 
+#endif /* ARM64_S390_COMMON == 1 */
+
 #define kvm_exception_type_names		\
 	{ except_type_sync,	"SYNC"   },	\
 	{ except_type_irq,	"IRQ"    },	\
@@ -507,13 +510,13 @@ static __always_inline bool kvm_vcpu_abt_issea(const struct kvm_vcpu *vcpu)
 	return esr_abt_is_sea(kvm_vcpu_get_esr(vcpu));
 }
 
+#if ARM64_S390_COMMON == 1
 static __always_inline int kvm_vcpu_sys_get_rt(struct kvm_vcpu *vcpu)
 {
 	u64 esr = kvm_vcpu_get_esr(vcpu);
 	return ESR_ELx_SYS64_ISS_RT(esr);
 }
 
-#if ARM64_S390_COMMON == 1
 static inline bool esr_abt_is_write_fault(unsigned long esr)
 {
 	if (esr_abt_is_s1ptw(esr)) {
diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h
index 7f051dad79c7..01de095cf0fb 100644
--- a/arch/arm64/include/asm/kvm_host.h
+++ b/arch/arm64/include/asm/kvm_host.h
@@ -1332,9 +1332,6 @@ u64 kvm_vcpu_apply_reg_masks(const struct kvm_vcpu *, enum vcpu_sysreg, u64);
 		__v;							\
 	})
 
-u64 vcpu_read_sys_reg(const struct kvm_vcpu *, enum vcpu_sysreg);
-void vcpu_write_sys_reg(struct kvm_vcpu *, u64, enum vcpu_sysreg);
-
 struct kvm_vm_stat {
 	struct kvm_vm_stat_generic generic;
 };
@@ -1371,11 +1368,9 @@ int kvm_handle_cp14_32(struct kvm_vcpu *vcpu);
 int kvm_handle_cp14_64(struct kvm_vcpu *vcpu);
 int kvm_handle_cp15_32(struct kvm_vcpu *vcpu);
 int kvm_handle_cp15_64(struct kvm_vcpu *vcpu);
-int kvm_handle_sys_reg(struct kvm_vcpu *vcpu);
 int kvm_handle_cp10_id(struct kvm_vcpu *vcpu);
 
 void kvm_sys_regs_create_debugfs(struct kvm *kvm);
-void kvm_reset_sys_regs(struct kvm_vcpu *vcpu);
 
 int __init kvm_sys_reg_table_init(void);
 struct sys_reg_desc;
@@ -1386,6 +1381,11 @@ int __init populate_nv_trap_config(void);
 void kvm_calculate_traps(struct kvm_vcpu *vcpu);
 
 #if ARM64_S390_COMMON == 1
+u64 vcpu_read_sys_reg(const struct kvm_vcpu *, enum vcpu_sysreg);
+void vcpu_write_sys_reg(struct kvm_vcpu *, u64, enum vcpu_sysreg);
+int kvm_handle_sys_reg(struct kvm_vcpu *vcpu);
+void kvm_reset_sys_regs(struct kvm_vcpu *vcpu);
+
 unsigned long kvm_arm_num_regs(struct kvm_vcpu *vcpu);
 int kvm_arm_copy_reg_indices(struct kvm_vcpu *vcpu, u64 __user *indices);
 int kvm_arm_get_reg(struct kvm_vcpu *vcpu, const struct kvm_one_reg *reg);
@@ -1623,8 +1623,6 @@ static inline bool __vcpu_has_feature(const struct kvm_arch *ka, int feature)
 
 #define kvm_vcpu_initialized(v) vcpu_get_flag(v, VCPU_INITIALIZED)
 
-#endif /* ARM64_S390_COMMON == 1 */
-
 int kvm_trng_call(struct kvm_vcpu *vcpu);
 #ifdef CONFIG_KVM
 extern phys_addr_t hyp_mem_base;
@@ -1711,6 +1709,8 @@ void kvm_set_vm_id_reg(struct kvm *kvm, u32 reg, u64 val);
 	(kvm_cmp_feat(kvm, id, fld, >=, min) &&				\
 	kvm_cmp_feat(kvm, id, fld, <=, max))
 
+#endif /* ARM64_S390_COMMON == 1 */
+
 /* Check for a given level of PAuth support */
 #define kvm_has_pauth(k, l)						\
 	({								\
diff --git a/arch/arm64/include/asm/spectre.h b/arch/arm64/include/asm/spectre.h
index 2ecc9eddbebb..4c6c07c9b515 100644
--- a/arch/arm64/include/asm/spectre.h
+++ b/arch/arm64/include/asm/spectre.h
@@ -19,6 +19,7 @@
 #include <asm/cpufeature.h>
 #include <asm/virt.h>
 
+#if ARM64_S390_COMMON == 1
 /* Watch out, ordering is important here. */
 enum mitigation_state {
 	SPECTRE_UNAFFECTED,
@@ -31,6 +32,8 @@ enum mitigation_state arm64_get_spectre_v4_state(void);
 enum mitigation_state arm64_get_meltdown_state(void);
 enum mitigation_state arm64_get_spectre_bhb_state(void);
 
+#endif /* ARM64_S390_COMMON == 1 */
+
 struct pt_regs;
 struct task_struct;
 
diff --git a/arch/arm64/kvm/sys_regs.h b/arch/arm64/kvm/sys_regs.h
index a3cccad2766f..7cb7ad208e05 100644
--- a/arch/arm64/kvm/sys_regs.h
+++ b/arch/arm64/kvm/sys_regs.h
@@ -7,6 +7,7 @@
  * Copyright (C) 2012 - Virtual Open Systems and Columbia University
  * Authors: Christoffer Dall <c.dall@virtualopensystems.com>
  */
+/* Whole file is shared with s390 */
 
 #ifndef __ARM64_KVM_SYS_REGS_LOCAL_H__
 #define __ARM64_KVM_SYS_REGS_LOCAL_H__

-- 
2.56.0

[PATCH v3 06/23] KVM: s390: arm64: Prepare for sharing arm64 SVE and ftr-reg code

[Steffen Eiden]

Add required SVE and feature-register definitions to the arm64 on s390
host code. The definitions are required to be able to build shared SVE
related code.

No functional changes.

Co-developed-by: Andreas Grapentin <gra@linux.ibm.com>
Signed-off-by: Andreas Grapentin <gra@linux.ibm.com>
Signed-off-by: Steffen Eiden <seiden@linux.ibm.com>
---
 arch/s390/kvm/arm64/arm.c   | 2 ++
 arch/s390/kvm/arm64/guest.c | 6 ++++++
 2 files changed, 8 insertions(+)

diff --git a/arch/s390/kvm/arm64/arm.c b/arch/s390/kvm/arm64/arm.c
index bd9d85235f43..29d3add0bbda 100644
--- a/arch/s390/kvm/arm64/arm.c
+++ b/arch/s390/kvm/arm64/arm.c
@@ -8,6 +8,7 @@
 #include <linux/kvm_host.h>
 #include <linux/cleanup.h>
 #include <linux/fpu.h>
+#include <linux/bitfield.h>
 
 #include <asm/access-regs.h>
 
@@ -24,6 +25,7 @@
 #include "trace.h"
 
 static unsigned long system_supported_vcpu_features(struct kvm_vcpu *vcpu);
+#define read_sanitised_ftr_reg(_id) 0
 
 #define __INCL_GEN_ARM_FILE
 #include "generated/arm.inc"
diff --git a/arch/s390/kvm/arm64/guest.c b/arch/s390/kvm/arm64/guest.c
index 4ea6fe2c270d..8b5ee9e79a4f 100644
--- a/arch/s390/kvm/arm64/guest.c
+++ b/arch/s390/kvm/arm64/guest.c
@@ -5,6 +5,12 @@
 #include <arm64/kvm_emulate.h>
 #include <arm64/kvm_nested.h>
 
+#define SVE_VQ_MIN	__SVE_VQ_MIN
+#define SVE_NUM_ZREGS	KVM_ARM64_SVE_NUM_ZREGS
+#define SVE_NUM_PREGS	KVM_ARM64_SVE_NUM_PREGS
+
+#define vcpu_sve_slices(_vcpu) 1
+
 #define __INCL_GEN_ARM_FILE
 #include "generated/guest.inc"
 #undef __INCL_GEN_ARM_FILE

-- 
2.56.0

[PATCH v3 07/23] KVM: arm64: Prepare sys_regs.c for sharing with s390

[Steffen Eiden]

Prepare sharing of sys_regs.c with s390 by moving
reset_{vm,vcpu}_ftr_id_regs just before the UAPI definitions. Also, move
kvm_handle_cp14_load_store to other cp functions. This small refactoring
allows to have less shared sections.

No functional changes.

Co-developed-by: Andreas Grapentin <gra@linux.ibm.com>
Signed-off-by: Andreas Grapentin <gra@linux.ibm.com>
Signed-off-by: Steffen Eiden <seiden@linux.ibm.com>
---
 arch/arm64/kvm/sys_regs.c | 52 +++++++++++++++++++++++++----------------------
 1 file changed, 28 insertions(+), 24 deletions(-)

diff --git a/arch/arm64/kvm/sys_regs.c b/arch/arm64/kvm/sys_regs.c
index 8c91de6a4097..e71c1b169b49 100644
--- a/arch/arm64/kvm/sys_regs.c
+++ b/arch/arm64/kvm/sys_regs.c
@@ -4855,12 +4855,6 @@ static bool check_sysreg_table(const struct sys_reg_desc *table, unsigned int n,
 	return true;
 }
 
-int kvm_handle_cp14_load_store(struct kvm_vcpu *vcpu)
-{
-	kvm_inject_undefined(vcpu);
-	return 1;
-}
-
 static void perform_access(struct kvm_vcpu *vcpu,
 			   struct sys_reg_params *params,
 			   const struct sys_reg_desc *r)
@@ -4888,6 +4882,12 @@ static void perform_access(struct kvm_vcpu *vcpu,
 		kvm_incr_pc(vcpu);
 }
 
+int kvm_handle_cp14_load_store(struct kvm_vcpu *vcpu)
+{
+	kvm_inject_undefined(vcpu);
+	return 1;
+}
+
 /*
  * emulate_cp --  tries to match a sys_reg access in a handling table, and
  *                call the corresponding trap handler.
@@ -5340,25 +5340,9 @@ void kvm_sys_regs_create_debugfs(struct kvm *kvm)
 			    &sr_resx_fops);
 }
 
-static void reset_vm_ftr_id_reg(struct kvm_vcpu *vcpu, const struct sys_reg_desc *reg)
-{
-	u32 id = reg_to_encoding(reg);
-	struct kvm *kvm = vcpu->kvm;
-
-	if (test_bit(KVM_ARCH_FLAG_ID_REGS_INITIALIZED, &kvm->arch.flags))
-		return;
-
-	kvm_set_vm_id_reg(kvm, id, reg->reset(vcpu, reg));
-}
-
+static void reset_vm_ftr_id_reg(struct kvm_vcpu *vcpu, const struct sys_reg_desc *reg);
 static void reset_vcpu_ftr_id_reg(struct kvm_vcpu *vcpu,
-				  const struct sys_reg_desc *reg)
-{
-	if (kvm_vcpu_initialized(vcpu))
-		return;
-
-	reg->reset(vcpu, reg);
-}
+				  const struct sys_reg_desc *reg);
 
 /**
  * kvm_reset_sys_regs - sets system registers to reset value
@@ -5432,6 +5416,26 @@ int kvm_handle_sys_reg(struct kvm_vcpu *vcpu)
 	return 1;
 }
 
+static void reset_vm_ftr_id_reg(struct kvm_vcpu *vcpu, const struct sys_reg_desc *reg)
+{
+	u32 id = reg_to_encoding(reg);
+	struct kvm *kvm = vcpu->kvm;
+
+	if (test_bit(KVM_ARCH_FLAG_ID_REGS_INITIALIZED, &kvm->arch.flags))
+		return;
+
+	kvm_set_vm_id_reg(kvm, id, reg->reset(vcpu, reg));
+}
+
+static void reset_vcpu_ftr_id_reg(struct kvm_vcpu *vcpu,
+				  const struct sys_reg_desc *reg)
+{
+	if (kvm_vcpu_initialized(vcpu))
+		return;
+
+	reg->reset(vcpu, reg);
+}
+
 /******************************************************************************
  * Userspace API
  *****************************************************************************/

-- 
2.56.0

[PATCH v3 08/23] KVM: arm64: Share system register code with s390

[Steffen Eiden]

Mark functions that s390 can use to implement system registers for arm
on s390 guests as shared functions. The new shared functionality covers
system register infrastructure for KVM, Pointer auth checks and
shareable parts of the SVE register API. In order to reduce the number
of shared sections more code than s390 requires is shared. This is no
problem as for the shared code unused function checking is disabled.

Co-developed-by: Andreas Grapentin <gra@linux.ibm.com>
Signed-off-by: Andreas Grapentin <gra@linux.ibm.com>
Signed-off-by: Steffen Eiden <seiden@linux.ibm.com>
---
 arch/arm64/kvm/arm.c      |  3 +++
 arch/arm64/kvm/guest.c    | 11 +++++++----
 arch/arm64/kvm/sys_regs.c | 27 +++++++++++++++++++++++++++
 3 files changed, 37 insertions(+), 4 deletions(-)

diff --git a/arch/arm64/kvm/arm.c b/arch/arm64/kvm/arm.c
index 157bf4b81202..d83df5ff68f4 100644
--- a/arch/arm64/kvm/arm.c
+++ b/arch/arm64/kvm/arm.c
@@ -366,6 +366,7 @@ void kvm_arch_destroy_vm(struct kvm *kvm)
 	kvm_arm_teardown_hypercalls(kvm);
 }
 
+#if ARM64_S390_COMMON == 1
 static bool kvm_has_full_ptr_auth(void)
 {
 	bool apa, gpa, api, gpi, apa3, gpa3;
@@ -400,6 +401,8 @@ static bool kvm_has_full_ptr_auth(void)
 		(apa + api + apa3) == 1);
 }
 
+#endif /* ARM64_S390_COMMON == 1 */
+
 int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext)
 {
 	int r;
diff --git a/arch/arm64/kvm/guest.c b/arch/arm64/kvm/guest.c
index 878ea4500c5c..26ec4751e17e 100644
--- a/arch/arm64/kvm/guest.c
+++ b/arch/arm64/kvm/guest.c
@@ -307,12 +307,12 @@ static int set_core_reg(struct kvm_vcpu *vcpu, const struct kvm_one_reg *reg)
 	return err;
 }
 
-#endif /* ARM64_S390_COMMON == 1 */
-
 #define vq_word(vq) (((vq) - SVE_VQ_MIN) / 64)
 #define vq_mask(vq) ((u64)1 << ((vq) - SVE_VQ_MIN) % 64)
 #define vq_present(vqs, vq) (!!((vqs)[vq_word(vq)] & vq_mask(vq)))
 
+#endif /* ARM64_S390_COMMON == 1 */
+
 static int get_sve_vls(struct kvm_vcpu *vcpu, const struct kvm_one_reg *reg)
 {
 	unsigned int max_vq, vq;
@@ -383,6 +383,7 @@ static int set_sve_vls(struct kvm_vcpu *vcpu, const struct kvm_one_reg *reg)
 	return 0;
 }
 
+#if ARM64_S390_COMMON == 1
 #define SVE_REG_SLICE_SHIFT	0
 #define SVE_REG_SLICE_BITS	5
 #define SVE_REG_ID_SHIFT	(SVE_REG_SLICE_SHIFT + SVE_REG_SLICE_BITS)
@@ -399,6 +400,8 @@ static int set_sve_vls(struct kvm_vcpu *vcpu, const struct kvm_one_reg *reg)
 #define KVM_SVE_ZREG_SIZE KVM_REG_SIZE(KVM_REG_ARM64_SVE_ZREG(0, 0))
 #define KVM_SVE_PREG_SIZE KVM_REG_SIZE(KVM_REG_ARM64_SVE_PREG(0, 0))
 
+#endif /* ARM64_S390_COMMON == 1 */
+
 /*
  * Number of register slices required to cover each whole SVE register.
  * NOTE: Only the first slice every exists, for now.
@@ -595,8 +598,6 @@ static unsigned long num_core_regs(const struct kvm_vcpu *vcpu)
 	return copy_core_reg_indices(vcpu, NULL);
 }
 
-#endif /* ARM64_S390_COMMON == 1 */
-
 static unsigned long num_sve_regs(const struct kvm_vcpu *vcpu)
 {
 	const unsigned int slices = vcpu_sve_slices(vcpu);
@@ -658,6 +659,8 @@ static int copy_sve_reg_indices(const struct kvm_vcpu *vcpu,
 	return num_regs;
 }
 
+#endif /* ARM64_S390_COMMON == 1 */
+
 /**
  * kvm_arm_num_regs - how many registers do we present via KVM_GET_ONE_REG
  * @vcpu: the vCPU pointer
diff --git a/arch/arm64/kvm/sys_regs.c b/arch/arm64/kvm/sys_regs.c
index e71c1b169b49..fa1ed13daeb9 100644
--- a/arch/arm64/kvm/sys_regs.c
+++ b/arch/arm64/kvm/sys_regs.c
@@ -45,6 +45,7 @@
  * 64bit interface.
  */
 
+#if ARM64_S390_COMMON == 1
 static u64 sys_reg_to_index(const struct sys_reg_desc *reg);
 static int set_id_reg(struct kvm_vcpu *vcpu, const struct sys_reg_desc *rd,
 		      u64 val);
@@ -82,6 +83,8 @@ static bool write_to_read_only(struct kvm_vcpu *vcpu,
 			"sys_reg write to read-only register");
 }
 
+#endif /* ARM64_S390_COMMON == 1 */
+
 enum sr_loc_attr {
 	SR_LOC_MEMORY	= 0,	  /* Register definitely in memory */
 	SR_LOC_LOADED	= BIT(0), /* Register on CPU, unless it cannot */
@@ -428,6 +431,7 @@ void vcpu_write_sys_reg(struct kvm_vcpu *vcpu, u64 val, enum vcpu_sysreg reg)
 	__vcpu_assign_sys_reg(vcpu, reg, val);
 }
 
+#if ARM64_S390_COMMON == 1
 /* CSSELR values; used to index KVM_REG_ARM_DEMUX_ID_CCSIDR */
 #define CSSELR_MAX 14
 
@@ -533,6 +537,8 @@ static bool access_rw(struct kvm_vcpu *vcpu,
 	return true;
 }
 
+#endif /* ARM64_S390_COMMON == 1 */
+
 /*
  * See note at ARMv7 ARM B1.14.4 (TL;DR: S/W ops are not easily virtualized).
  */
@@ -800,6 +806,7 @@ static bool access_gicv5_ppi_enabler(struct kvm_vcpu *vcpu,
 	return true;
 }
 
+#if ARM64_S390_COMMON == 1
 static bool trap_raz_wi(struct kvm_vcpu *vcpu,
 			struct sys_reg_params *p,
 			const struct sys_reg_desc *r)
@@ -853,6 +860,8 @@ static bool trap_oslsr_el1(struct kvm_vcpu *vcpu,
 	return true;
 }
 
+#endif /* ARM64_S390_COMMON == 1 */
+
 static int set_oslsr_el1(struct kvm_vcpu *vcpu, const struct sys_reg_desc *rd,
 			 u64 val)
 {
@@ -1012,6 +1021,7 @@ static u64 reset_actlr(struct kvm_vcpu *vcpu, const struct sys_reg_desc *r)
 	return actlr;
 }
 
+#if ARM64_S390_COMMON == 1
 static u64 reset_mpidr(struct kvm_vcpu *vcpu, const struct sys_reg_desc *r)
 {
 	u64 mpidr = kvm_calculate_mpidr(vcpu);
@@ -1020,6 +1030,8 @@ static u64 reset_mpidr(struct kvm_vcpu *vcpu, const struct sys_reg_desc *r)
 	return mpidr;
 }
 
+#endif /* ARM64_S390_COMMON == 1 */
+
 static unsigned int hidden_visibility(const struct kvm_vcpu *vcpu,
 				      const struct sys_reg_desc *r)
 {
@@ -1589,6 +1601,7 @@ static int set_pmcr(struct kvm_vcpu *vcpu, const struct sys_reg_desc *r,
 #define AMU_AMEVCNTR1_EL0(n) { SYS_DESC(SYS_AMEVCNTR1_EL0(n)), undef_access }
 #define AMU_AMEVTYPER1_EL0(n) { SYS_DESC(SYS_AMEVTYPER1_EL0(n)), undef_access }
 
+#if ARM64_S390_COMMON == 1
 static unsigned int ptrauth_visibility(const struct kvm_vcpu *vcpu,
 			const struct sys_reg_desc *rd)
 {
@@ -1609,6 +1622,8 @@ static unsigned int ptrauth_visibility(const struct kvm_vcpu *vcpu,
 	__PTRAUTH_KEY(k ## KEYLO_EL1),					\
 	__PTRAUTH_KEY(k ## KEYHI_EL1)
 
+#endif /* ARM64_S390_COMMON == 1 */
+
 static bool access_arch_timer(struct kvm_vcpu *vcpu,
 			      struct sys_reg_params *p,
 			      const struct sys_reg_desc *r)
@@ -1906,6 +1921,7 @@ static int arm64_check_features(struct kvm_vcpu *vcpu,
 	return 0;
 }
 
+#if ARM64_S390_COMMON == 1
 static u8 pmuver_to_perfmon(u8 pmuver)
 {
 	switch (pmuver) {
@@ -2840,6 +2856,8 @@ static bool bad_redir_trap(struct kvm_vcpu *vcpu,
 	.val = 0,				\
 }
 
+#endif /* ARM64_S390_COMMON == 1 */
+
 static bool access_sp_el1(struct kvm_vcpu *vcpu,
 			  struct sys_reg_params *p,
 			  const struct sys_reg_desc *r)
@@ -3212,6 +3230,7 @@ static void init_imp_id_regs(void)
 	boot_cpu_aidr_val = read_sysreg(aidr_el1);
 }
 
+#if ARM64_S390_COMMON == 1
 static u64 reset_imp_id_reg(struct kvm_vcpu *vcpu, const struct sys_reg_desc *r)
 {
 	switch (reg_to_encoding(r)) {
@@ -3269,6 +3288,8 @@ static int set_imp_id_reg(struct kvm_vcpu *vcpu, const struct sys_reg_desc *r,
 	.val = mask,					\
 	}
 
+#endif /* ARM64_S390_COMMON == 1 */
+
 static u64 reset_mdcr(struct kvm_vcpu *vcpu, const struct sys_reg_desc *r)
 {
 	__vcpu_assign_sys_reg(vcpu, r->reg, vcpu->kvm->arch.nr_pmu_counters);
@@ -4833,6 +4854,7 @@ static const struct sys_reg_desc cp15_64_regs[] = {
 	{ SYS_DESC(SYS_AARCH32_CNTVCTSS),     access_arch_timer },
 };
 
+#if ARM64_S390_COMMON == 1
 static bool check_sysreg_table(const struct sys_reg_desc *table, unsigned int n,
 			       bool reset_check)
 {
@@ -4882,6 +4904,8 @@ static void perform_access(struct kvm_vcpu *vcpu,
 		kvm_incr_pc(vcpu);
 }
 
+#endif /* ARM64_S390_COMMON == 1 */
+
 int kvm_handle_cp14_load_store(struct kvm_vcpu *vcpu)
 {
 	kvm_inject_undefined(vcpu);
@@ -5416,6 +5440,7 @@ int kvm_handle_sys_reg(struct kvm_vcpu *vcpu)
 	return 1;
 }
 
+#if ARM64_S390_COMMON == 1
 static void reset_vm_ftr_id_reg(struct kvm_vcpu *vcpu, const struct sys_reg_desc *reg)
 {
 	u32 id = reg_to_encoding(reg);
@@ -5757,6 +5782,8 @@ int kvm_arm_copy_sys_reg_indices(struct kvm_vcpu *vcpu, u64 __user *uindices)
 	return write_demux_regids(uindices);
 }
 
+#endif /* ARM64_S390_COMMON == 1 */
+
 #define KVM_ARM_FEATURE_ID_RANGE_INDEX(r)			\
 	KVM_ARM_FEATURE_ID_RANGE_IDX(sys_reg_Op0(r),		\
 		sys_reg_Op1(r),					\

-- 
2.56.0

[PATCH v3 09/23] s390: tools: Share arm64/kvm headers

[Steffen Eiden]

In addition to sharing headers in arch/arm64/include/{asm, uapi/asm}
also allow to share headers in arch/arm64/kvm which provide local KVM
definitions.

Co-developed-by: Andreas Grapentin <gra@linux.ibm.com>
Signed-off-by: Andreas Grapentin <gra@linux.ibm.com>
Signed-off-by: Steffen Eiden <seiden@linux.ibm.com>
---
 arch/s390/tools/Makefile.arm64h | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)

diff --git a/arch/s390/tools/Makefile.arm64h b/arch/s390/tools/Makefile.arm64h
index f3f8e9d9bdd8..88dcce26179c 100644
--- a/arch/s390/tools/Makefile.arm64h
+++ b/arch/s390/tools/Makefile.arm64h
@@ -7,10 +7,11 @@ arm64api := $(objtree)/arch/$(ARCH)/include/generated/arm64
 arm64uapi := $(objtree)/arch/$(ARCH)/include/generated/uapi/arm64
 
 # List of ARM64 headers to share with s390 KVM
-# Format: [uapi:]header.h or header-gen.h
+# Format: [uapi:|kvm:]header.h or header-gen.h
 #
 # * header.h - from arch/arm64/include/asm/ to arm64/
 # * uapi:header.h - from arch/arm64/include/uapi/asm/ to uapi/arm64/
+# * kvm:header.h - from arch/arm64/kvm/ to arm64/
 # * header-gen.h - generated as header-gen.h from ARM64 header header.h
 # If a header contains partial markers (#if ARM64_S390_COMMON == 1) only this part will be copied.
 # If the file does not contain any markers the whole file will be copied.
@@ -36,7 +37,9 @@ arm64-type = $(firstword $(subst :, ,$(1)))
 arm64-base = $(or $(word 2,$(subst :, ,$(1))),$(1))
 arm64-src-base = $(patsubst %-gen.h,%.h,$(call arm64-base,$(1)))
 
-arm64-src = $(srctree)/arch/arm64/include/$(if $(filter uapi,$(call arm64-type,$(1))),uapi/)asm/$(call arm64-src-base,$(1))
+arm64-src = $(if $(filter kvm,$(call arm64-type,$(1))),\
+		$(srctree)/arch/arm64/kvm/$(call arm64-src-base,$(1)),\
+		$(srctree)/arch/arm64/include/$(if $(filter uapi,$(call arm64-type,$(1))),uapi/)asm/$(call arm64-src-base,$(1)))
 arm64-dst = $(if $(filter uapi,$(call arm64-type,$(1))),$(arm64uapi),$(arm64api))/$(call arm64-base,$(1))
 
 $(foreach hdr,$(ARM64_SHARED_HEADERS),\

-- 
2.56.0

[PATCH v3 10/23] s390: Add range check asm macro

[Steffen Eiden]

The range check macro turns cryptic assembler errors into an easy to
understand error if e.g. someone mistypes an input value in an asm
macro.

Add an asm macro that verifies that a given input value is in a specific
range. Add macros which check for ranges often used when creating
instructions. While at it, add a range check for the register numbers in
the GR_NUM and VX_NUM macro.

Co-developed-by: Jens Remus <jremus@linux.ibm.com>
Signed-off-by: Jens Remus <jremus@linux.ibm.com>
Signed-off-by: Steffen Eiden <seiden@linux.ibm.com>
---
 arch/s390/include/asm/insn-common-asm.h | 28 ++++++++++++++++++++++++++++
 1 file changed, 28 insertions(+)

diff --git a/arch/s390/include/asm/insn-common-asm.h b/arch/s390/include/asm/insn-common-asm.h
index fd9b3cacb7c5..f3f10f2c92bb 100644
--- a/arch/s390/include/asm/insn-common-asm.h
+++ b/arch/s390/include/asm/insn-common-asm.h
@@ -9,6 +9,32 @@
 
 #ifdef __ASSEMBLER__
 
+.macro	RANGECHECK	min max val
+	.if (\val < \min) || (\val > \max)
+	.error "operand out of range (\val is not between \min and \max)"
+	.endif
+.endm
+
+.macro	ASSERT_U16	val
+	RANGECHECK 0, 65535, \val
+.endm
+
+.macro	ASSERT_S16	val
+	RANGECHECK -32768, 32767, \val
+.endm
+
+.macro	ASSERT_U8	val
+	RANGECHECK 0, 255, \val
+.endm
+
+.macro	ASSERT_U4	val
+	RANGECHECK 0, 15, \val
+.endm
+
+.macro	ASSERT_U5	val
+	RANGECHECK 0, 31, \val
+.endm
+
 /*
  * GR_NUM - Retrieve general-purpose register number
  *
@@ -24,6 +50,7 @@
 	.endr
 	.if \opd == 255
 		\opd = \gr
+		ASSERT_U4	\opd
 	.endif
 .endm
 
@@ -46,6 +73,7 @@
 	.endr
 	.if \opd == 255
 		\opd = \vxr
+		ASSERT_U5	\opd
 	.endif
 .endm
 

-- 
2.56.0

[PATCH v3 11/23] s390: Read/write ARM system registers

[Steffen Eiden]

Introduce Extract Arm System Register and Set Arm System Register to
enable s390 hosts to read and write system registers for arm64 guests.
The new instructions use the new RIE_H instruction format. Add assembler
macros to create instructions in RIE_H format manually. Add Support for
disassembling the new instructions.

Co-developed-by: Andreas Grapentin <gra@linux.ibm.com>
Signed-off-by: Andreas Grapentin <gra@linux.ibm.com>
Signed-off-by: Steffen Eiden <seiden@linux.ibm.com>
---
 arch/s390/include/asm/sae-asm.h | 45 +++++++++++++++++++++++++++
 arch/s390/include/asm/sae.h     | 67 +++++++++++++++++++++++++++++++++++++++++
 arch/s390/kernel/dis.c          |  1 +
 arch/s390/tools/opcodes.txt     |  2 ++
 4 files changed, 115 insertions(+)

diff --git a/arch/s390/include/asm/sae-asm.h b/arch/s390/include/asm/sae-asm.h
new file mode 100644
index 000000000000..a1abb07b5a77
--- /dev/null
+++ b/arch/s390/include/asm/sae-asm.h
@@ -0,0 +1,45 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef __ASM_S390_SAE_ASM_H
+#define __ASM_S390_SAE_ASM_H
+
+#ifndef __ASM_S390_SAE_H
+#error only <asm/sae.h> can be included directly
+#endif
+
+#include <asm/insn-common-asm.h>
+
+#ifdef __ASSEMBLER__
+
+/*
+ * INSN_RIE_H - RIE-h instruction format
+ *
+ * RIE-h format: <insn> R1, R3, I2, M4
+ *    +--------+----+----+----+-----------------+----+--------+
+ *    | OpCode | R1 |////| R3 |        I2       | M4 | Opcode |
+ *    +--------+----+----+----+-----------------+----+--------+
+ *    0        8    12   16   20                36   40      47
+ */
+.macro	INSN_RIE_H	opc, gr1, gr3, imm2, m4
+	GR_NUM	r1, \gr1
+	GR_NUM	r3, \gr3
+	ASSERT_U16	\opc
+	ASSERT_U16	\imm2
+	ASSERT_U4	\m4
+	.byte	(\opc & 0xff00) >> 8
+	.byte	r1 << 4
+	.byte	(r3 << 4) | ((\imm2 & 0xf000) >> 12)
+	.byte	((\imm2 & 0x0ff0) >> 4)
+	.byte	((\imm2 & 0x000f) << 4) | (\m4 & 0xf)
+	.byte	\opc & 0xff
+.endm
+
+.macro	SASR	r1, r3, i2, m4
+	INSN_RIE_H 0xed99, \r1, \r3, \i2, \m4
+.endm
+
+.macro	EASR	r1, r3, i2, m4
+	INSN_RIE_H 0xed9b, \r1, \r3, \i2, \m4
+.endm
+
+#endif /* __ASSEMBLER__ */
+#endif /* __ASM_S390_SAE_ASM_H */
diff --git a/arch/s390/include/asm/sae.h b/arch/s390/include/asm/sae.h
index 26d89fae0836..cac4fbf58e88 100644
--- a/arch/s390/include/asm/sae.h
+++ b/arch/s390/include/asm/sae.h
@@ -4,6 +4,7 @@
 
 #include <linux/linkage.h>
 #include <linux/types.h>
+#include <asm/sae-asm.h>
 
 #ifndef __ASSEMBLER__
 
@@ -12,6 +13,72 @@ asmlinkage void sae(phys_addr_t sae_block_phys);
 
 #include <linux/io.h>
 #include <asm/kvm_host_arm64_types.h>
+#include <asm/insn-common-asm-emit.h>
+
+asm(".include \"asm/sae-asm.h\"\n");
+
+#define _SAE_ASR_REG_SHIFT	5
+#define SASR_FLAG_INITIALIZED  0x8
+#define EASR_FLAG_SA           0x8
+
+/**
+ * sasr() - Set Arm System Register
+ * @arm_reg: ARM system register identifier; compile-time constant
+ * @val: Value to set
+ * @save_area: Pointer to SAE save area
+ * @flags: Operation flags; compile-time constant
+ *
+ * Sets an ARM system register value.
+ */
+static __always_inline void sasr(unsigned int arm_reg, u64 val,
+				 struct kvm_sae_save_area *save_area,
+				 u64 flags)
+{
+	struct kvm_sae_block *sdo = phys_to_virt(save_area->sdo);
+	u16 reg = arm_reg >> _SAE_ASR_REG_SHIFT;
+
+	/*
+	 * SASR expects an U16 for imm2 but s390 inline asm can only provide S16 values
+	 * via the K modifier. Convert i2 to unsigned by and-ing with 0xffff
+	 */
+	asm_inline volatile(
+		"	SASR	%[r1],%[r3],%[i2] & 0xffff,%[m4]\n"
+		: "+m" (*save_area), "+m" (*sdo)
+		: [r1] "d" (val),
+		  [r3] "a" (save_area), [i2] "K" (reg), [m4] "I" (flags)
+	);
+}
+
+/**
+ * easr() - Extract Arm System Register
+ * @arm_reg: ARM system register identifier; compile-time constant
+ * @save_area: Pointer to SAE save area
+ * @flags: Operation flags; compile-time constant
+ *
+ * Reads an ARM system register value.
+ *
+ * Return: Register value
+ */
+static __always_inline u64 easr(unsigned int arm_reg,
+				const struct kvm_sae_save_area *save_area,
+				u64 flags)
+{
+	struct kvm_sae_block *sdo = phys_to_virt(save_area->sdo);
+	u16 reg = arm_reg >> _SAE_ASR_REG_SHIFT;
+	u64 val;
+
+	/*
+	 * EASR expects an U16 for imm2 but s390 inline asm can only provide S16 values
+	 * via the K modifier. Convert i2 to unsigned by and-ing with 0xffff
+	 */
+	asm_inline volatile(
+		"	EASR	%[r1],%[r3],%[i2] & 0xffff,%[m4]\n"
+		: [r1] "=d" (val)
+		: "m" (*save_area),
+		  "m" (*sdo), [r3] "a" (save_area), [i2] "K" (reg), [m4] "I" (flags)
+	);
+	return val;
+}
 
 /**
  * stiasrm() - STore and Invalidate Arm System Register Multiple
diff --git a/arch/s390/kernel/dis.c b/arch/s390/kernel/dis.c
index d2d299ba8b86..065264667db3 100644
--- a/arch/s390/kernel/dis.c
+++ b/arch/s390/kernel/dis.c
@@ -208,6 +208,7 @@ static const unsigned char formats[][6] = {
 	[INSTR_RIE_RUI0]     = { R_8, I16_16, U4_12, 0, 0, 0 },
 	[INSTR_RIE_RUPI]     = { R_8, I8_32, U4_12, J16_16, 0, 0 },
 	[INSTR_RIE_RUPU]     = { R_8, U8_32, U4_12, J16_16, 0, 0 },
+	[INSTR_RIE_R0RUU]    = { R_8, R_16, U16_20, U4_36, 0, 0 },
 	[INSTR_RIL_RI]	     = { R_8, I32_16, 0, 0, 0, 0 },
 	[INSTR_RIL_RP]	     = { R_8, J32_16, 0, 0, 0, 0 },
 	[INSTR_RIL_RU]	     = { R_8, U32_16, 0, 0, 0, 0 },
diff --git a/arch/s390/tools/opcodes.txt b/arch/s390/tools/opcodes.txt
index 0e4773c94af0..22fb9c500a56 100644
--- a/arch/s390/tools/opcodes.txt
+++ b/arch/s390/tools/opcodes.txt
@@ -1255,6 +1255,8 @@ ed64	ley	RXY_FRRD
 ed65	ldy	RXY_FRRD
 ed66	stey	RXY_FRRD
 ed67	stdy	RXY_FRRD
+ed99	sasr	RIE_R0RUU
+ed9b	easr	RIE_R0RUU
 eda8	czdt	RSL_LRDFU
 eda9	czxt	RSL_LRDFU
 edaa	cdzt	RSL_LRDFU

-- 
2.56.0

[PATCH v3 12/23] s390: Query arm guest time

[Steffen Eiden]

Add functions to convert between ARM guest time (LSB0) and s390 host
time (MSB0) using new ptff function codes.

Co-developed-by: Nico Boehr <nrb@linux.ibm.com>
Signed-off-by: Nico Boehr <nrb@linux.ibm.com>
Co-developed-by: Andreas Grapentin <gra@linux.ibm.com>
Signed-off-by: Andreas Grapentin <gra@linux.ibm.com>
Signed-off-by: Steffen Eiden <seiden@linux.ibm.com>
---
 arch/s390/include/asm/timex.h | 53 +++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 53 insertions(+)

diff --git a/arch/s390/include/asm/timex.h b/arch/s390/include/asm/timex.h
index 23421b6eb225..8ee69fa3c3a1 100644
--- a/arch/s390/include/asm/timex.h
+++ b/arch/s390/include/asm/timex.h
@@ -81,6 +81,8 @@ extern unsigned char ptff_function_mask[16];
 #define PTFF_QSI	0x02	/* query steering information */
 #define PTFF_QPT	0x03	/* query physical clock */
 #define PTFF_QUI	0x04	/* query UTC information */
+#define PTFF_QAGTO	0x10	/* query arm guest time offset */
+#define PTFF_QAGPT	0x11	/* query arm guest physical time */
 #define PTFF_ATO	0x40	/* adjust tod offset */
 #define PTFF_STO	0x41	/* set tod offset */
 #define PTFF_SFS	0x42	/* set fine steering rate */
@@ -118,6 +120,17 @@ struct ptff_qui {
 	unsigned int pad_0x5c[41];
 } __packed;
 
+/*
+ * Query Arm Guest Time
+ * used for:
+ *	- Query Arm Guest Time Offset
+ *	- Query Arm Guest Physical Time
+ */
+struct ptff_qagt {
+	u64 in;
+	u64 out;
+};
+
 /*
  * ptff - Perform timing facility function
  * @ptff_block: Pointer to ptff parameter block
@@ -266,4 +279,44 @@ static inline int tod_after_eq(unsigned long a, unsigned long b)
 	return a >= b;
 }
 
+/*
+ * ptff_qagto() - Query Arm Guest Time Offset
+ *
+ * @physical_time: Arm guest physical time
+ *
+ * Converts the Arm guest physical time into the Arm guest time offset.
+ *
+ * The caller must ensure that this PTFF function is available,
+ * before calling this function. The condition code is discarded.
+ *
+ * Return: Arm guest time offset
+ */
+static inline u64 ptff_qagto(u64 physical_time)
+{
+	struct ptff_qagt qagto = { .in = physical_time };
+
+	ptff(&qagto, sizeof(qagto), PTFF_QAGTO);
+	return qagto.out;
+}
+
+/*
+ * ptff_qagpt() - Query Arm Guest Physical Time
+ *
+ * @guest_time_offset: Arm guest time offset
+ *
+ * Converts the Arm guest time offset into the Arm guest physical time.
+ *
+ * The caller must ensure that this PTFF function is available,
+ * before calling this function. The condition code is discarded.
+ *
+ * Return: Arm guest physical time
+ */
+static inline u64 ptff_qagpt(u64 guest_time_offset)
+{
+	struct ptff_qagt qagpt = { .in = guest_time_offset };
+
+	ptff(&qagpt, sizeof(qagpt), PTFF_QAGPT);
+	return qagpt.out;
+}
+
 #endif

-- 
2.56.0

[PATCH v3 13/23] KVM: s390: arm64: Query Available Arm features

[Steffen Eiden]

QAAF enables s390 hosts to gain information about the support
and handling of various arm features supported by the machine.

Function code 1 provides general information about available formats,
machine defined content of system/id register, and other various
information for running arm guests.

Function code 2 provides a initialised save area.

Co-developed-by: Andreas Grapentin <gra@linux.ibm.com>
Signed-off-by: Andreas Grapentin <gra@linux.ibm.com>
Signed-off-by: Steffen Eiden <seiden@linux.ibm.com>
---
 arch/s390/include/asm/aef.h                  |  12 +++
 arch/s390/include/asm/kvm_host_arm64_types.h | 115 +++++++++++++++++++++++++++
 arch/s390/include/asm/sae.h                  |  26 ++++++
 arch/s390/kernel/Makefile                    |   2 +-
 arch/s390/kernel/aef.c                       |  94 ++++++++++++++++++++++
 arch/s390/kvm/arm64/Makefile                 |   1 +
 arch/s390/kvm/arm64/arm.c                    |   6 +-
 arch/s390/kvm/arm64/qaaf.c                   |  83 +++++++++++++++++++
 arch/s390/kvm/arm64/qaaf.h                   |  12 +++
 arch/s390/tools/opcodes.txt                  |   1 +
 10 files changed, 350 insertions(+), 2 deletions(-)

diff --git a/arch/s390/include/asm/aef.h b/arch/s390/include/asm/aef.h
new file mode 100644
index 000000000000..06519cc575be
--- /dev/null
+++ b/arch/s390/include/asm/aef.h
@@ -0,0 +1,12 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+
+#ifndef _ASM_S390_AEF_H
+#define _ASM_S390_AEF_H
+
+#include <asm/kvm_host_arm64_types.h>
+
+const struct kvm_sae_save_area *aef_gisrsa(void);
+const struct qaaf_qmc_block *aef_qmc(void);
+bool aef_usable(void);
+
+#endif /* _ASM_S390_AEF_H */
diff --git a/arch/s390/include/asm/kvm_host_arm64_types.h b/arch/s390/include/asm/kvm_host_arm64_types.h
index 8783a66d3029..d9471f3b0d46 100644
--- a/arch/s390/include/asm/kvm_host_arm64_types.h
+++ b/arch/s390/include/asm/kvm_host_arm64_types.h
@@ -121,4 +121,119 @@ struct kvm_sae_save_area {
 } __packed __aligned(PAGE_SIZE);
 static_assert(sizeof(struct kvm_sae_save_area) == PAGE_SIZE);
 
+#define QAAF_FC_QMC	1
+#define QAAF_FC_GISRSA	2
+
+union qaaf_gr0_gisrsa {
+	struct {
+		u8 _0000[6];
+		u8 saf;
+		u8 : 2;
+		u8 fc : 6;
+	};
+	u64 val;
+};
+
+static_assert(sizeof(union qaaf_gr0_gisrsa) == sizeof(u64));
+
+/* QAAF Query Model Capabilities */
+struct qaaf_qmc_block {
+	u64	_0000;			/* 0x0000 */
+	u8	ssdf;			/* 0x0008 */
+	u8	_0009;			/* 0x0009 */
+	u8	ssaf;			/* 0x000a */
+	u8	_000b[3];		/* 0x000b */
+	u16	maxncpu;		/* 0x000e */
+	u64	_0010[2];		/* 0x0010 */
+	u64	regs[0x1fc];		/* 0x0020 */
+} __aligned(PAGE_SIZE);
+static_assert(sizeof(struct qaaf_qmc_block) == PAGE_SIZE);
+
+union qaaf_block {
+	struct qaaf_qmc_block qmc;
+	struct kvm_sae_save_area save_area;
+} __aligned(PAGE_SIZE);
+static_assert(sizeof(union qaaf_block) == PAGE_SIZE);
+
+/*
+ * Keep in sync with mapping from SYS_* to QAAF_* in qaaf.c!
+ */
+enum {
+	QAAF_REG_MIDR_EL1		= 0x00,
+	/* 0x01-0x04 reserved */
+	QAAF_REG_MPIDR_EL1		= 0x05,
+	QAAF_REG_REVIDR_EL1		= 0x06,
+	/* 0x07 reserved */
+	QAAF_REG_ID_PFR0_EL1		= 0x08,
+	QAAF_REG_ID_PFR1_EL1		= 0x09,
+	QAAF_REG_ID_DFR0_EL1		= 0x0a,
+	QAAF_REG_ID_AFR0_EL1		= 0x0b,
+	QAAF_REG_ID_MMFR0_EL1		= 0x0c,
+	QAAF_REG_ID_MMFR1_EL1		= 0x0d,
+	QAAF_REG_ID_MMFR2_EL1		= 0x0e,
+	QAAF_REG_ID_MMFR3_EL1		= 0x0f,
+	QAAF_REG_ID_ISAR0_EL1		= 0x10,
+	QAAF_REG_ID_ISAR1_EL1		= 0x11,
+	QAAF_REG_ID_ISAR2_EL1		= 0x12,
+	QAAF_REG_ID_ISAR3_EL1		= 0x13,
+	QAAF_REG_ID_ISAR4_EL1		= 0x14,
+	QAAF_REG_ID_ISAR5_EL1		= 0x15,
+	QAAF_REG_ID_MMFR4_EL1		= 0x16,
+	QAAF_REG_ID_ISAR6_EL1		= 0x17,
+	QAAF_REG_MVFR0_EL1		= 0x18,
+	QAAF_REG_MVFR1_EL1		= 0x19,
+	QAAF_REG_MVFR2_EL1		= 0x1a,
+	/* 0x1b reserved */
+	QAAF_REG_ID_PFR2_EL1		= 0x1c,
+	QAAF_REG_ID_DFR1_EL1		= 0x1d,
+	QAAF_REG_ID_MMFR5_EL1		= 0x1e,
+	/* 0x1f reserved */
+	QAAF_REG_ID_AA64PFR0_EL1	= 0x20,
+	QAAF_REG_ID_AA64PFR1_EL1	= 0x21,
+	QAAF_REG_ID_AA64PFR2_EL1	= 0x22,
+	/* 0x23 reserved */
+	QAAF_REG_ID_AA64ZFR0_EL1	= 0x24,
+	QAAF_REG_ID_AA64SMFR0_EL1	= 0x25,
+	/* 0x26 reserved */
+	QAAF_REG_ID_AA64FPFR0_EL1	= 0x27,
+	QAAF_REG_ID_AA64DFR0_EL1	= 0x28,
+	QAAF_REG_ID_AA64DFR1_EL1	= 0x29,
+	QAAF_REG_ID_AA64DFR2_EL1	= 0x2a,
+	/* 0x2b reserved */
+	QAAF_REG_ID_AA64AFR0_EL1	= 0x2c,
+	QAAF_REG_ID_AA64AFR1_EL1	= 0x2d,
+	/* 0x2e,0x2f reserved */
+	QAAF_REG_ID_AA64ISAR0_EL1	= 0x30,
+	QAAF_REG_ID_AA64ISAR1_EL1	= 0x31,
+	QAAF_REG_ID_AA64ISAR2_EL1	= 0x32,
+	QAAF_REG_ID_AA64ISAR3_EL1	= 0x33,
+	/* 0x34-0x37 reserved */
+	QAAF_REG_ID_AA64MMFR0_EL1	= 0x38,
+	QAAF_REG_ID_AA64MMFR1_EL1	= 0x39,
+	QAAF_REG_ID_AA64MMFR2_EL1	= 0x3a,
+	QAAF_REG_ID_AA64MMFR3_EL1	= 0x3b,
+	QAAF_REG_ID_AA64MMFR4_EL1	= 0x3c,
+	/* 0x3d-0x3f reserved */
+	QAAF_REG_CNTFRQ_EL0		= 0x40,
+	QAAF_REG_CTR_EL0		= 0x41,
+	QAAF_REG_AIDR_EL1		= 0x42,
+	/* 0x43-0x47 reserved */
+	QAAF_IRPTC			= 0x48,
+	/* 0x49 reserved */
+	QAAF_REG_ICH_VTR_EL2		= 0x4a,
+	QAAF_GIC_ATTR			= 0x4b,
+	/* 0x4c-0x4f reserved */
+	QAAF_REG_PMMIR_EL1		= 0x50,
+	QAAF_REG_PMCR_EL0		= 0x51,
+	QAAF_REG_PMCEID0_EL0		= 0x52,
+	QAAF_REG_PMCEID1_EL0		= 0x53,
+	/* 0x54-0x1fb reserved */
+	_QAAF_MAX
+};
+
+static_assert((sizeof(struct qaaf_qmc_block) -
+	       offsetof(struct qaaf_qmc_block, regs)) /
+		      8 >=
+	      _QAAF_MAX);
+
 #endif /* ASM_KVM_HOST_ARM64_TYPES_H */
diff --git a/arch/s390/include/asm/sae.h b/arch/s390/include/asm/sae.h
index cac4fbf58e88..d1b9af770108 100644
--- a/arch/s390/include/asm/sae.h
+++ b/arch/s390/include/asm/sae.h
@@ -12,6 +12,7 @@
 asmlinkage void sae(phys_addr_t sae_block_phys);
 
 #include <linux/io.h>
+#include <asm/extable.h>
 #include <asm/kvm_host_arm64_types.h>
 #include <asm/insn-common-asm-emit.h>
 
@@ -112,5 +113,30 @@ static __always_inline void lasrm(struct kvm_sae_save_area *save_area)
 	);
 }
 
+/**
+ * qaaf() - Query Available Arm Features
+ * @gr0: QAAF function code
+ * @qaaf_block: Pointer to the page for the output
+ *
+ * Perform QAAF. The result is written to qaaf_block.
+ */
+static __always_inline int qaaf(u64 gr0, union qaaf_block *qaaf_block)
+{
+	int exception = 1;
+
+	asm_inline volatile(
+		"	lgr	0,%[r0]\n"
+		"0:	.insn	rre,0xb9ad0000,%[r1],0\n"
+		"	lhi	%[exc],0\n"
+		"1:\n"
+		EX_TABLE_INSN(0b, 1b)
+		: "=m" (*qaaf_block), [exc] "+d" (exception)
+		: [r1] "a" (qaaf_block), [r0] "d" (gr0)
+		: "0"
+	);
+
+	return exception;
+}
+
 #endif /* !__ASSEMBLER__ */
 #endif /* __ASM_S390_SAE_H */
diff --git a/arch/s390/kernel/Makefile b/arch/s390/kernel/Makefile
index f84b8fb41331..0f3b9fcd1885 100644
--- a/arch/s390/kernel/Makefile
+++ b/arch/s390/kernel/Makefile
@@ -46,7 +46,7 @@ obj-y	+= runtime_instr.o cache.o fpu.o dumpstack.o guarded_storage.o sthyi.o
 obj-y	+= entry.o reipl.o kdebugfs.o alternative.o skey.o
 obj-y	+= nospec-branch.o ipl_vmparm.o machine_kexec_reloc.o unwind_bc.o
 obj-y	+= smp.o text_amode31.o stacktrace.o abs_lowcore.o facility.o uv.o wti.o
-obj-y	+= diag/
+obj-y	+= diag/ aef.o
 
 always-$(KBUILD_BUILTIN)	+= vmlinux.lds
 
diff --git a/arch/s390/kernel/aef.c b/arch/s390/kernel/aef.c
new file mode 100644
index 000000000000..31e18953d7b3
--- /dev/null
+++ b/arch/s390/kernel/aef.c
@@ -0,0 +1,94 @@
+// SPDX-License-Identifier: GPL-2.0
+
+#include <linux/export.h>
+#include <linux/sysfs.h>
+#include <linux/kobject.h>
+#include <linux/slab.h>
+
+#include <asm/aef.h>
+#include <asm/sae.h>
+#include <asm/sclp.h>
+
+static struct qaaf_qmc_block qmc = {};
+static struct kvm_sae_save_area save_area = {};
+
+const struct kvm_sae_save_area *aef_gisrsa(void)
+{
+	return &save_area;
+}
+EXPORT_SYMBOL_GPL(aef_gisrsa);
+
+const struct qaaf_qmc_block *aef_qmc(void)
+{
+	return &qmc;
+}
+EXPORT_SYMBOL_GPL(aef_qmc);
+
+/* set once the QMC block and GISRSA save area were fetched successfully */
+static bool aef_valid;
+
+/**
+ * aef_usable() - Report whether the QAAF query data is available
+ *
+ * false if sclp.has_aef is not set or the QAAF instruction failed.
+ */
+bool aef_usable(void)
+{
+	return aef_valid;
+}
+EXPORT_SYMBOL_GPL(aef_usable);
+
+static int __init aef_init_save_area(void)
+{
+	int ret;
+
+	union qaaf_gr0_gisrsa gr0 = {
+		.fc = QAAF_FC_GISRSA,
+		.saf = SAE_SAVE_AREA_FORMAT_0,
+	};
+
+	if (!sclp.has_aef)
+		return 0;
+
+	ret = qaaf(gr0.val, (union qaaf_block *)&save_area);
+	if (ret)
+		return -EIO;
+
+	return 0;
+}
+
+static int __init aef_query_info(void)
+{
+	int ret;
+
+	if (!sclp.has_aef)
+		return 0;
+
+	ret = qaaf(QAAF_FC_QMC, (union qaaf_block *)&qmc);
+	if (ret)
+		return -EIO;
+
+	return 0;
+}
+
+static int __init aef_sysfs_init(void)
+{
+	int rc;
+
+	if (!sclp.has_aef)
+		return 0;
+
+	rc = aef_query_info();
+	if (rc)
+		return rc;
+
+	rc = aef_init_save_area();
+	if (rc)
+		return rc;
+
+	aef_valid = true;
+
+	return 0;
+}
+
+arch_initcall(aef_sysfs_init);
diff --git a/arch/s390/kvm/arm64/Makefile b/arch/s390/kvm/arm64/Makefile
index b48dc65b7aaa..e9587c6d6de9 100644
--- a/arch/s390/kvm/arm64/Makefile
+++ b/arch/s390/kvm/arm64/Makefile
@@ -16,6 +16,7 @@ kvm-arm64-local := \
 	inject_fault.o \
 	mmio.o \
 	mmu.o \
+	qaaf.o \
 	reset.o
 
 kvm-arm64-obj := $(kvm-arm64-local)
diff --git a/arch/s390/kvm/arm64/arm.c b/arch/s390/kvm/arm64/arm.c
index 29d3add0bbda..62c2b45599b9 100644
--- a/arch/s390/kvm/arm64/arm.c
+++ b/arch/s390/kvm/arm64/arm.c
@@ -11,6 +11,7 @@
 #include <linux/bitfield.h>
 
 #include <asm/access-regs.h>
+#include <asm/aef.h>
 
 #include <arm64/kvm_emulate.h>
 #include <arm64/sysreg.h>
@@ -20,6 +21,7 @@
 
 #include "arm.h"
 #include "handle_exit.h"
+#include "qaaf.h"
 
 #define CREATE_TRACE_POINTS
 #include "trace.h"
@@ -741,9 +743,11 @@ long kvm_arch_vcpu_unlocked_ioctl(struct file *filp, unsigned int ioctl,
 
 static int __init kvm_s390_arm64_init(void)
 {
-	if (!sclp.has_aef)
+	if (!sclp.has_aef || !aef_usable())
 		return -ENODEV;
 
+	kvm_init_qaaf();
+
 	return kvm_init(sizeof(struct kvm_vcpu), 0, THIS_MODULE);
 }
 
diff --git a/arch/s390/kvm/arm64/qaaf.c b/arch/s390/kvm/arm64/qaaf.c
new file mode 100644
index 000000000000..f6b1750aed68
--- /dev/null
+++ b/arch/s390/kvm/arm64/qaaf.c
@@ -0,0 +1,83 @@
+// SPDX-License-Identifier: GPL-2.0-only
+
+#include <linux/kvm_host.h>
+#include <asm/aef.h>
+#include "qaaf.h"
+
+static struct qaaf_qmc_block qmc_sanitized;
+
+/**
+ * kvm_qaaf_qmc() - Get the sanitised QAAF QMC block
+ */
+struct qaaf_qmc_block *kvm_qaaf_qmc(void)
+{
+	return &qmc_sanitized;
+}
+
+void __init kvm_init_qaaf(void)
+{
+	qmc_sanitized = *aef_qmc();
+}
+
+#define _qaaf_reg_case(id) case SYS_##id: return qmc_sanitized.regs[QAAF_REG_##id]
+u64 kvm_qaaf_read_ftr_reg(u32 id)
+{
+	switch (id) {
+	_qaaf_reg_case(MIDR_EL1);
+	_qaaf_reg_case(MPIDR_EL1);
+	_qaaf_reg_case(REVIDR_EL1);
+	_qaaf_reg_case(ID_PFR0_EL1);
+	_qaaf_reg_case(ID_PFR1_EL1);
+	_qaaf_reg_case(ID_DFR0_EL1);
+	_qaaf_reg_case(ID_AFR0_EL1);
+	_qaaf_reg_case(ID_MMFR0_EL1);
+	_qaaf_reg_case(ID_MMFR1_EL1);
+	_qaaf_reg_case(ID_MMFR2_EL1);
+	_qaaf_reg_case(ID_MMFR3_EL1);
+	_qaaf_reg_case(ID_ISAR0_EL1);
+	_qaaf_reg_case(ID_ISAR1_EL1);
+	_qaaf_reg_case(ID_ISAR2_EL1);
+	_qaaf_reg_case(ID_ISAR3_EL1);
+	_qaaf_reg_case(ID_ISAR4_EL1);
+	_qaaf_reg_case(ID_ISAR5_EL1);
+	_qaaf_reg_case(ID_MMFR4_EL1);
+	_qaaf_reg_case(ID_ISAR6_EL1);
+	_qaaf_reg_case(MVFR0_EL1);
+	_qaaf_reg_case(MVFR1_EL1);
+	_qaaf_reg_case(MVFR2_EL1);
+	_qaaf_reg_case(ID_PFR2_EL1);
+	_qaaf_reg_case(ID_DFR1_EL1);
+	_qaaf_reg_case(ID_MMFR5_EL1);
+	_qaaf_reg_case(ID_AA64PFR0_EL1);
+	_qaaf_reg_case(ID_AA64PFR1_EL1);
+	_qaaf_reg_case(ID_AA64PFR2_EL1);
+	_qaaf_reg_case(ID_AA64ZFR0_EL1);
+	_qaaf_reg_case(ID_AA64SMFR0_EL1);
+	_qaaf_reg_case(ID_AA64FPFR0_EL1);
+	_qaaf_reg_case(ID_AA64DFR0_EL1);
+	_qaaf_reg_case(ID_AA64DFR1_EL1);
+	_qaaf_reg_case(ID_AA64DFR2_EL1);
+	_qaaf_reg_case(ID_AA64AFR0_EL1);
+	_qaaf_reg_case(ID_AA64AFR1_EL1);
+	_qaaf_reg_case(ID_AA64ISAR0_EL1);
+	_qaaf_reg_case(ID_AA64ISAR1_EL1);
+	_qaaf_reg_case(ID_AA64ISAR2_EL1);
+	_qaaf_reg_case(ID_AA64ISAR3_EL1);
+	_qaaf_reg_case(ID_AA64MMFR0_EL1);
+	_qaaf_reg_case(ID_AA64MMFR1_EL1);
+	_qaaf_reg_case(ID_AA64MMFR2_EL1);
+	_qaaf_reg_case(ID_AA64MMFR3_EL1);
+	_qaaf_reg_case(ID_AA64MMFR4_EL1);
+	_qaaf_reg_case(CNTFRQ_EL0);
+	_qaaf_reg_case(CTR_EL0);
+	_qaaf_reg_case(AIDR_EL1);
+	_qaaf_reg_case(ICH_VTR_EL2);
+	_qaaf_reg_case(PMMIR_EL1);
+	_qaaf_reg_case(PMCR_EL0);
+	_qaaf_reg_case(PMCEID0_EL0);
+	_qaaf_reg_case(PMCEID1_EL0);
+	default:
+		WARN(true, "Unknown feature register 0x%x\n", id);
+		return 0xbad1234bad;
+	}
+}
diff --git a/arch/s390/kvm/arm64/qaaf.h b/arch/s390/kvm/arm64/qaaf.h
new file mode 100644
index 000000000000..f1d50a9654b1
--- /dev/null
+++ b/arch/s390/kvm/arm64/qaaf.h
@@ -0,0 +1,12 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+
+#ifndef ARCH_S390_KVM_ARM64_QAAF_H
+#define ARCH_S390_KVM_ARM64_QAAF_H
+
+#include <asm/aef.h>
+
+__init void kvm_init_qaaf(void);
+struct qaaf_qmc_block *kvm_qaaf_qmc(void);
+u64 kvm_qaaf_read_ftr_reg(u32 id);
+
+#endif
diff --git a/arch/s390/tools/opcodes.txt b/arch/s390/tools/opcodes.txt
index 22fb9c500a56..bb4ba4038cb0 100644
--- a/arch/s390/tools/opcodes.txt
+++ b/arch/s390/tools/opcodes.txt
@@ -600,6 +600,7 @@ b9a7	stiasrm	RRE_R0
 b9aa	lptea	RRF_RURR2
 b9ab	essa	RRF_U0RR
 b9ac	irbm	RRE_RR
+b9ad	qaaf	RRE_R0
 b9ae	rrbm	RRE_RR
 b9af	pfmf	RRE_RR
 b9b0	cu14	RRF_U0RR

-- 
2.56.0

[PATCH v3 14/23] KVM: s390: arm64: Verify machine feature set before loading

[Steffen Eiden]

Arm guests on s390 need more than the Arm Execution Facility alone. In
addition, the PTFF time conversion functions are required. In addition
the guest mapping relies on EDAT2 for large pages.

Verify that these features are present before loading the kvm_arm64
module. Every machine having the aef facility present should also
support those. Print an error message if this is not the case.

Co-developed-by: Andreas Grapentin <gra@linux.ibm.com>
Signed-off-by: Andreas Grapentin <gra@linux.ibm.com>
Signed-off-by: Steffen Eiden <seiden@linux.ibm.com>
---
 arch/s390/kvm/arm64/arm.c | 21 ++++++++++++++++++++-
 1 file changed, 20 insertions(+), 1 deletion(-)

diff --git a/arch/s390/kvm/arm64/arm.c b/arch/s390/kvm/arm64/arm.c
index 62c2b45599b9..3af0b47b8377 100644
--- a/arch/s390/kvm/arm64/arm.c
+++ b/arch/s390/kvm/arm64/arm.c
@@ -741,9 +741,28 @@ long kvm_arch_vcpu_unlocked_ioctl(struct file *filp, unsigned int ioctl,
 	return -ENOIOCTLCMD;
 }
 
+static bool __init machine_supports_arm64_kvm(void)
+{
+	if (!sclp.has_aef || !aef_usable()) {
+		pr_info("ARM Execution Facility not available\n");
+		return false;
+	}
+
+	if (!ptff_query(PTFF_QAGTO) || !ptff_query(PTFF_QAGPT)) {
+		pr_err("No ARM time conversion functions available (unexpected machine feature set)\n");
+		return false;
+	}
+	if (!cpu_has_edat2()) {
+		pr_err("No EDAT2 facility available (unexpected machine feature set)\n");
+		return false;
+	}
+
+	return true;
+}
+
 static int __init kvm_s390_arm64_init(void)
 {
-	if (!sclp.has_aef || !aef_usable())
+	if (!machine_supports_arm64_kvm())
 		return -ENODEV;
 
 	kvm_init_qaaf();

-- 
2.56.0

[PATCH v3 15/23] KVM: s390: arm64: Implement feature sanitisation

[Steffen Eiden]

Implement feature detection and sanitization for arm64 KVM guests on
s390.

Query hardware capabilities using QAAF instruction and sanitize arm64 ID
registers. Provide accessor functions for queried capabilities and
integrate feature detection into module initialization sequence.

Co-developed-by: Andreas Grapentin <gra@linux.ibm.com>
Signed-off-by: Andreas Grapentin <gra@linux.ibm.com>
Signed-off-by: Steffen Eiden <seiden@linux.ibm.com>
---
 arch/s390/include/asm/kvm_host_arm64.h |  13 +++
 arch/s390/kvm/arm64/Makefile           |   1 +
 arch/s390/kvm/arm64/arm.c              |   2 +-
 arch/s390/kvm/arm64/feature.c          | 163 +++++++++++++++++++++++++++++++++
 arch/s390/kvm/arm64/feature.h          |  86 +++++++++++++++++
 arch/s390/kvm/arm64/qaaf.c             |   2 +
 6 files changed, 266 insertions(+), 1 deletion(-)

diff --git a/arch/s390/include/asm/kvm_host_arm64.h b/arch/s390/include/asm/kvm_host_arm64.h
index 3e69724f17c9..9afe37f6696c 100644
--- a/arch/s390/include/asm/kvm_host_arm64.h
+++ b/arch/s390/include/asm/kvm_host_arm64.h
@@ -15,6 +15,19 @@ enum vcpu_sysreg;
 
 #include <asm/sae.h>
 
+enum {
+	ARM64_HAS_WFXT,
+	ARM64_HAS_RASV1P1_EXTN,
+	ARM64_HAS_STAGE2_FWB,
+	ARM64_HAS_RAS_EXTN,
+	ARM64_HAS_EVT,
+	ARM64_HAS_HCR_NV1,
+	ARM64_MISMATCHED_CACHE_TYPE,
+	ARM64_HAS_ECV_CNTPOFF,
+};
+
+bool cpus_have_final_cap(unsigned int num);
+
 #define KVM_HAVE_MMU_RWLOCK
 #define KVM_MAX_VCPUS 1
 #define __KVM_HAVE_VCPU_RUN_PID_CHANGE
diff --git a/arch/s390/kvm/arm64/Makefile b/arch/s390/kvm/arm64/Makefile
index e9587c6d6de9..e75b38071f8a 100644
--- a/arch/s390/kvm/arm64/Makefile
+++ b/arch/s390/kvm/arm64/Makefile
@@ -11,6 +11,7 @@ ccflags-y += -I $(src) -I $(objtree)/$(obj) -I$(srctree)/arch/$(SRCARCH)/kvm/gma
 
 kvm-arm64-local := \
 	arm.o \
+	feature.o \
 	guest.o \
 	handle_exit.o \
 	inject_fault.o \
diff --git a/arch/s390/kvm/arm64/arm.c b/arch/s390/kvm/arm64/arm.c
index 3af0b47b8377..ba6235629c7d 100644
--- a/arch/s390/kvm/arm64/arm.c
+++ b/arch/s390/kvm/arm64/arm.c
@@ -20,6 +20,7 @@
 #include <kvm_mmu.h>
 
 #include "arm.h"
+#include "feature.h"
 #include "handle_exit.h"
 #include "qaaf.h"
 
@@ -27,7 +28,6 @@
 #include "trace.h"
 
 static unsigned long system_supported_vcpu_features(struct kvm_vcpu *vcpu);
-#define read_sanitised_ftr_reg(_id) 0
 
 #define __INCL_GEN_ARM_FILE
 #include "generated/arm.inc"
diff --git a/arch/s390/kvm/arm64/feature.c b/arch/s390/kvm/arm64/feature.c
new file mode 100644
index 000000000000..f8b23ba85f9e
--- /dev/null
+++ b/arch/s390/kvm/arm64/feature.c
@@ -0,0 +1,163 @@
+// SPDX-License-Identifier: GPL-2.0-only
+
+#include <linux/kvm_host.h>
+#include <linux/printk.h>
+#include <linux/bitfield.h>
+#include <asm/sae.h>
+#include <asm/kvm_host_types.h>
+
+#include <arm64/sysreg.h>
+
+#include "feature.h"
+
+#define MASK_RESERVED(_qaafp, _id)				\
+	({							\
+		u64 *_reg = &((_qaafp)->regs[QAAF_REG_##_id]);	\
+		*_reg = (*_reg & ~_id##_RES0) | _id##_RES1;	\
+	})
+
+#define QAAF_FIELD_MODIFY(_idp, _id_field, _val) \
+	FIELD_MODIFY(_id_field##_MASK, _idp, _id_field##_##_val)
+
+#define MODIFY(_qaafp, _id, _field, _val) \
+	QAAF_FIELD_MODIFY(&((_qaafp)->regs[QAAF_REG_##_id]), _id##_##_field, _val)
+
+void __init kvm_arm_host_sanitize_features(struct qaaf_qmc_block *qaaf_qmc)
+{
+	MASK_RESERVED(qaaf_qmc, ID_AA64ISAR0_EL1);
+
+	MODIFY(qaaf_qmc, ID_AA64ISAR1_EL1, LS64, NI);
+	MASK_RESERVED(qaaf_qmc, ID_AA64ISAR1_EL1);
+
+	MODIFY(qaaf_qmc, ID_AA64ISAR2_EL1, SYSINSTR_128, NI);
+	MODIFY(qaaf_qmc, ID_AA64ISAR2_EL1, SYSREG_128, NI);
+	MODIFY(qaaf_qmc, ID_AA64ISAR2_EL1, PAC_frac, NI);
+	MASK_RESERVED(qaaf_qmc, ID_AA64ISAR2_EL1);
+
+	MASK_RESERVED(qaaf_qmc, ID_AA64ISAR3_EL1);
+
+	MASK_RESERVED(qaaf_qmc, ID_AA64MMFR0_EL1);
+	MASK_RESERVED(qaaf_qmc, ID_AA64MMFR1_EL1);
+	MASK_RESERVED(qaaf_qmc, ID_AA64MMFR2_EL1);
+
+	MODIFY(qaaf_qmc, ID_AA64MMFR3_EL1, Spec_FPACC, NI);
+	MODIFY(qaaf_qmc, ID_AA64MMFR3_EL1, ADERR, NI);
+	MODIFY(qaaf_qmc, ID_AA64MMFR3_EL1, SDERR, NI);
+	MODIFY(qaaf_qmc, ID_AA64MMFR3_EL1, ANERR, NI);
+	MODIFY(qaaf_qmc, ID_AA64MMFR3_EL1, SNERR, NI);
+	MODIFY(qaaf_qmc, ID_AA64MMFR3_EL1, MEC, NI);
+	MASK_RESERVED(qaaf_qmc, ID_AA64MMFR3_EL1);
+
+	MODIFY(qaaf_qmc, ID_AA64MMFR4_EL1, SRMASK, NI);
+	MODIFY(qaaf_qmc, ID_AA64MMFR4_EL1, E3DSE, NI);
+	MODIFY(qaaf_qmc, ID_AA64MMFR4_EL1, RMEGDI, NI);
+	MODIFY(qaaf_qmc, ID_AA64MMFR4_EL1, FGWTE3, NI);
+	MODIFY(qaaf_qmc, ID_AA64MMFR4_EL1, ASID2, NI);
+	MASK_RESERVED(qaaf_qmc, ID_AA64MMFR4_EL1);
+
+	MODIFY(qaaf_qmc, ID_AA64PFR0_EL1, SEL2, NI);
+	MODIFY(qaaf_qmc, ID_AA64PFR0_EL1, SVE, NI);
+	MASK_RESERVED(qaaf_qmc, ID_AA64PFR0_EL1);
+	MASK_RESERVED(qaaf_qmc, ID_AA64PFR1_EL1);
+	MASK_RESERVED(qaaf_qmc, ID_AA64PFR2_EL1);
+
+	MODIFY(qaaf_qmc, ID_AA64DFR0_EL1, PMUVer, NI);
+	MASK_RESERVED(qaaf_qmc, ID_AA64DFR0_EL1);
+	MASK_RESERVED(qaaf_qmc, ID_AA64DFR1_EL1);
+	MASK_RESERVED(qaaf_qmc, ID_AA64DFR2_EL1);
+
+	MASK_RESERVED(qaaf_qmc, ID_AA64AFR0_EL1);
+	MASK_RESERVED(qaaf_qmc, ID_AA64AFR1_EL1);
+
+	MASK_RESERVED(qaaf_qmc, ID_AA64ZFR0_EL1);
+
+	MASK_RESERVED(qaaf_qmc, ID_AA64SMFR0_EL1);
+
+	MASK_RESERVED(qaaf_qmc, ID_AA64FPFR0_EL1);
+
+	MASK_RESERVED(qaaf_qmc, ID_PFR0_EL1);
+	MASK_RESERVED(qaaf_qmc, ID_PFR1_EL1);
+	MASK_RESERVED(qaaf_qmc, ID_PFR2_EL1);
+
+	MASK_RESERVED(qaaf_qmc, ID_DFR0_EL1);
+	MASK_RESERVED(qaaf_qmc, ID_DFR1_EL1);
+
+	MASK_RESERVED(qaaf_qmc, ID_MMFR0_EL1);
+	MASK_RESERVED(qaaf_qmc, ID_MMFR1_EL1);
+	MASK_RESERVED(qaaf_qmc, ID_MMFR2_EL1);
+	MASK_RESERVED(qaaf_qmc, ID_MMFR3_EL1);
+	MASK_RESERVED(qaaf_qmc, ID_MMFR4_EL1);
+	MASK_RESERVED(qaaf_qmc, ID_MMFR5_EL1);
+
+	MASK_RESERVED(qaaf_qmc, ID_ISAR0_EL1);
+	MASK_RESERVED(qaaf_qmc, ID_ISAR1_EL1);
+	MASK_RESERVED(qaaf_qmc, ID_ISAR2_EL1);
+	MASK_RESERVED(qaaf_qmc, ID_ISAR3_EL1);
+	MASK_RESERVED(qaaf_qmc, ID_ISAR4_EL1);
+	MASK_RESERVED(qaaf_qmc, ID_ISAR5_EL1);
+	MASK_RESERVED(qaaf_qmc, ID_ISAR6_EL1);
+
+	MASK_RESERVED(qaaf_qmc, MVFR0_EL1);
+	MASK_RESERVED(qaaf_qmc, MVFR1_EL1);
+	MASK_RESERVED(qaaf_qmc, MVFR2_EL1);
+
+	MASK_RESERVED(qaaf_qmc, CTR_EL0);
+
+	MASK_RESERVED(qaaf_qmc, ICH_VTR_EL2);
+}
+
+/**
+ * cpus_have_final_cap - Check if an ARM64 CPU capability is supported
+ * @num: Capability number to check
+ *
+ * Maps ARM64 CPU capabilities to their corresponding ID register fields
+ * and checks if the feature is supported by the s390 hardware via QAAF.
+ *
+ * Return: true if capability is supported, false otherwise
+ */
+bool cpus_have_final_cap(unsigned int num)
+{
+	u64 reg_val;
+
+	switch (num) {
+	case ARM64_HAS_STAGE2_FWB:
+		reg_val = read_sanitised_ftr_reg(SYS_ID_AA64MMFR2_EL1);
+		return SYS_FIELD_GET(ID_AA64MMFR2_EL1, FWB, reg_val) >=
+		       ID_AA64MMFR2_EL1_FWB_IMP;
+
+	case ARM64_HAS_WFXT:
+		reg_val = read_sanitised_ftr_reg(SYS_ID_AA64ISAR2_EL1);
+		return SYS_FIELD_GET(ID_AA64ISAR2_EL1, WFxT, reg_val) >=
+		       ID_AA64ISAR2_EL1_WFxT_IMP;
+
+	case ARM64_HAS_RASV1P1_EXTN:
+		reg_val = read_sanitised_ftr_reg(SYS_ID_AA64PFR1_EL1);
+		return SYS_FIELD_GET(ID_AA64PFR1_EL1, RAS_frac, reg_val) ==
+		       ID_AA64PFR1_EL1_RAS_frac_RASv1p1;
+
+	case ARM64_HAS_HCR_NV1:
+		reg_val = read_sanitised_ftr_reg(SYS_ID_AA64MMFR4_EL1);
+		return SYS_FIELD_GET(ID_AA64MMFR4_EL1, E2H0, reg_val) ==
+		       ID_AA64MMFR4_EL1_E2H0_IMP;
+
+	case ARM64_HAS_RAS_EXTN:
+		reg_val = read_sanitised_ftr_reg(SYS_ID_AA64PFR0_EL1);
+		return SYS_FIELD_GET(ID_AA64PFR0_EL1, RAS, reg_val) >=
+		       ID_AA64PFR0_EL1_RAS_IMP;
+
+	case ARM64_HAS_EVT:
+		reg_val = read_sanitised_ftr_reg(SYS_ID_AA64MMFR2_EL1);
+		return SYS_FIELD_GET(ID_AA64MMFR2_EL1, EVT, reg_val) >=
+		       ID_AA64MMFR2_EL1_EVT_IMP;
+
+	case ARM64_HAS_ECV_CNTPOFF:
+		reg_val = read_sanitised_ftr_reg(SYS_ID_AA64MMFR0_EL1);
+		return SYS_FIELD_GET(ID_AA64MMFR0_EL1, ECV, reg_val) >=
+		       ID_AA64MMFR0_EL1_ECV_CNTPOFF;
+
+	case ARM64_MISMATCHED_CACHE_TYPE:
+		return false;
+	default:
+		return false;
+	}
+}
diff --git a/arch/s390/kvm/arm64/feature.h b/arch/s390/kvm/arm64/feature.h
new file mode 100644
index 000000000000..0a59047fda6e
--- /dev/null
+++ b/arch/s390/kvm/arm64/feature.h
@@ -0,0 +1,86 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+
+#ifndef ARCH_S390_KVM_FEATURE_H
+#define ARCH_S390_KVM_FEATURE_H
+
+#include <linux/types.h>
+#include <linux/bitfield.h>
+
+#include <asm/sae.h>
+
+#include "qaaf.h"
+
+void __init kvm_arm_host_sanitize_features(struct qaaf_qmc_block *qaaf_qmc);
+
+/**
+ * read_sanitised_ftr_reg() - Get the value of a sys register
+ *
+ * @id - ARM sysreg id
+ *
+ * Get the value of a sys register describing the host configuration.
+ * The returned value indicates support by the machine (HW/FW)
+ * after some sanitisation.
+ *
+ * Example:
+ *	read_sanitised_ftr_reg(SYS_ID_AA64PFR0_EL1)
+ *
+ * Return: Sanitised HW value of the specified register
+ */
+static inline u64 read_sanitised_ftr_reg(u32 id)
+{
+	return kvm_qaaf_read_ftr_reg(id);
+}
+
+/**
+ * kvm_sae_supported_sd_formats() - Retrieve supported SAE SD formats
+ *
+ * Return: Bitmap of supported SAE state description formats.
+ */
+static inline u32 kvm_sae_supported_sd_formats(void)
+{
+	struct qaaf_qmc_block *qmc = kvm_qaaf_qmc();
+
+	return qmc->ssdf;
+}
+
+/**
+ * kvm_sae_supported_sa_formats() - Retrieve supported SAE SA formats
+ *
+ * Return: Bitmap of supported SAE save area formats.
+ */
+static inline u32 kvm_sae_supported_sa_formats(void)
+{
+	struct qaaf_qmc_block *qmc = kvm_qaaf_qmc();
+
+	return qmc->ssaf;
+}
+
+/**
+ * kvm_sae_max_vcpus() - Retrieve maximum supported vcpus
+ *
+ * Return: Max number of vCPUs supported by the SAE instruction/the machine
+ *	as indicated by QAAF, not necessarily that supported by the host software.
+ */
+static inline u32 kvm_sae_max_vcpus(void)
+{
+	struct qaaf_qmc_block *qmc = kvm_qaaf_qmc();
+
+	/* QAAF QMC reports the max id not the max num */
+	return qmc->maxncpu + 1;
+}
+
+/**
+ * kvm_sae_irptc() - Retrieve supported SAE IRPTCs
+ *
+ * Return: Bitmap of supported SAE IRPTCs.
+ */
+static inline u64 kvm_sae_irptc(void)
+{
+	struct qaaf_qmc_block *qmc = kvm_qaaf_qmc();
+
+	return qmc->regs[QAAF_IRPTC];
+}
+
+#define kvm_vcpu_has_pmu(_v) false
+
+#endif /* ARCH_S390_KVM_FEATURE_H */
diff --git a/arch/s390/kvm/arm64/qaaf.c b/arch/s390/kvm/arm64/qaaf.c
index f6b1750aed68..36487025013c 100644
--- a/arch/s390/kvm/arm64/qaaf.c
+++ b/arch/s390/kvm/arm64/qaaf.c
@@ -3,6 +3,7 @@
 #include <linux/kvm_host.h>
 #include <asm/aef.h>
 #include "qaaf.h"
+#include "feature.h"
 
 static struct qaaf_qmc_block qmc_sanitized;
 
@@ -17,6 +18,7 @@ struct qaaf_qmc_block *kvm_qaaf_qmc(void)
 void __init kvm_init_qaaf(void)
 {
 	qmc_sanitized = *aef_qmc();
+	kvm_arm_host_sanitize_features(&qmc_sanitized);
 }
 
 #define _qaaf_reg_case(id) case SYS_##id: return qmc_sanitized.regs[QAAF_REG_##id]

-- 
2.56.0

[PATCH v3 16/23] KVM: s390: arm64: Implement arm sysreg managing infrastructure

[Steffen Eiden]

Implement the infrastructures for the s390 host to read and write the
arm64 guest system register. The host can access and modify (non-id)
system registers through the new instructions EASR/SASR or their
multiple variant. Guest accesses to ID registers are trapped and handled
by KVM. KVM gets the data form the result of QAAF and sanitizes it using
the shared infrastructure with arm64.

Co-developed-by: Andreas Grapentin <gra@linux.ibm.com>
Signed-off-by: Andreas Grapentin <gra@linux.ibm.com>
Signed-off-by: Steffen Eiden <seiden@linux.ibm.com>
---
 arch/s390/include/arm64/kvm_hyp.h      |  18 +
 arch/s390/include/arm64/kvm_nested.h   |   5 +
 arch/s390/include/arm64/sys_regs.h     |   9 +
 arch/s390/include/asm/kvm_host_arm64.h | 205 ++++++++-
 arch/s390/kvm/arm64/Makefile           |   4 +-
 arch/s390/kvm/arm64/Makefile.gen       |   3 +-
 arch/s390/kvm/arm64/spectre.c          |  51 +++
 arch/s390/kvm/arm64/sys_regs.c         | 815 +++++++++++++++++++++++++++++++++
 arch/s390/kvm/arm64/trace.h            |  50 ++
 arch/s390/tools/Makefile.arm64h        |   4 +
 10 files changed, 1161 insertions(+), 3 deletions(-)

diff --git a/arch/s390/include/arm64/kvm_hyp.h b/arch/s390/include/arm64/kvm_hyp.h
new file mode 100644
index 000000000000..e425b7c3c80d
--- /dev/null
+++ b/arch/s390/include/arm64/kvm_hyp.h
@@ -0,0 +1,18 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+
+#ifndef __S390_ARM64_KVM_HYP_H__
+#define __S390_ARM64_KVM_HYP_H__
+
+#include <linux/compiler.h>
+
+static __always_inline bool has_vhe(void)
+{
+	return true;
+}
+
+static __always_inline bool is_protected_kvm_enabled(void)
+{
+	return false;
+}
+
+#endif /* __S390_ARM64_KVM_HYP_H__ */
diff --git a/arch/s390/include/arm64/kvm_nested.h b/arch/s390/include/arm64/kvm_nested.h
index e950b1a10c41..9c02ac5c62cc 100644
--- a/arch/s390/include/arm64/kvm_nested.h
+++ b/arch/s390/include/arm64/kvm_nested.h
@@ -8,4 +8,9 @@ static inline bool vcpu_has_nv(const struct kvm_vcpu *vcpu)
 	return false;
 }
 
+static inline u64 limit_nv_id_reg(struct kvm *kvm, u32 id, u64 val)
+{
+	return val;
+}
+
 #endif /* ASM_KVM_NESTED_H */
diff --git a/arch/s390/include/arm64/sys_regs.h b/arch/s390/include/arm64/sys_regs.h
new file mode 100644
index 000000000000..dafb090ee1af
--- /dev/null
+++ b/arch/s390/include/arm64/sys_regs.h
@@ -0,0 +1,9 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+
+#ifndef __S390_ARM64_SYS_REGS_H__
+#define __S390_ARM64_SYS_REGS_H__
+
+#include <arm64/cputype.h>
+#include <arm64/sys_regs-gen.h>
+
+#endif /* __S390_ARM64_SYS_REGS_H__ */
diff --git a/arch/s390/include/asm/kvm_host_arm64.h b/arch/s390/include/asm/kvm_host_arm64.h
index 9afe37f6696c..35b6d8972359 100644
--- a/arch/s390/include/asm/kvm_host_arm64.h
+++ b/arch/s390/include/asm/kvm_host_arm64.h
@@ -3,6 +3,7 @@
 #define ASM_KVM_HOST_ARM64_H
 
 #include <linux/bug.h>
+#include <linux/bitfield.h>
 
 #include <asm/kvm_host_types.h>
 #include <asm/debug.h>
@@ -10,6 +11,7 @@
 #define vcpu_gp_regs(v)		((v)->arch.sae_block.gpr)
 enum vcpu_sysreg;
 
+#include <arm64/cputype.h>
 #include <arm64/kvm_host.h>
 #include <arm64/ptrace.h>
 
@@ -41,7 +43,12 @@ bool cpus_have_final_cap(unsigned int num);
 #define KVM_IRQCHIP_NUM_PINS 1
 #define irqchip_in_kernel(_k) false
 
-#define __ctxt_sys_reg(ctx, reg) NULL
+#define __ctxt_sys_reg(ctx, reg)					\
+	({								\
+		BUILD_BUG_ON(!__builtin_constant_p(reg));		\
+		___ctxt_sys_reg(ctx, reg);				\
+	})
+
 struct kvm_cpu_context {
 	/*
 	 * These are just for 32 bit, which we don't have, making them RES0.
@@ -53,6 +60,10 @@ struct kvm_cpu_context {
 	u64	spsr_fiq;
 
 	__vector128 __aligned(16) vregs[32];
+
+	/* shadowed sysregs to comply with arm64 infrastructure */
+	u64 elr_el1;
+	u64 spsr_el1;
 };
 
 struct kvm_vcpu_arch {
@@ -60,10 +71,19 @@ struct kvm_vcpu_arch {
 	struct kvm_sae_save_area save_area;
 	struct kvm_cpu_context ctxt;
 
+	/* Guest system registers not part of save area or ID registers */
+	u64 sys_reg_clidr_el1;
+	u64 sys_reg_csselr_el1;
+	/* Per-vcpu CCSIDR override or NULL */
+	u32 *ccsidr;
+
 	u32 host_acrs[NUM_ACRS];
 
 	/* Hypervisor Configuration Register */
 	u64 hcr_el2;
+	u64 hcrx_el2;
+
+	u64 mpidr;
 
 	/* Configuration flags, set once and for all before the vcpu can run */
 	u8 cflags;
@@ -113,6 +133,9 @@ struct kvm_vcpu_stat {
 #define vcpu_has_sve(_vcpu) false
 #define vcpu_has_ptrauth(_vcpu) false
 
+#define kvm_has_fpmr(kvm) \
+	 kvm_has_feat((kvm), ID_AA64PFR2_EL1, FPMR, IMP)
+
 struct kvm_arch_memory_slot {
 };
 
@@ -131,6 +154,9 @@ struct kvm_arch {
 	DECLARE_BITMAP(vcpu_features, KVM_VCPU_MAX_FEATURES);
 
 	unsigned long mem_limit;
+
+	/* Per-VM ID register storage */
+	struct kvm_vm_id_regs id_regs;
 };
 
 static inline bool __vcpu_has_feature(const struct kvm_arch *ka, int feature)
@@ -219,4 +245,181 @@ static inline void kvm_arch_async_page_present_queued(struct kvm_vcpu *vcpu)
 
 #define kvm_supports_32bit_el0() false
 
+#define __vcpu_sys_reg(__vcpu, __reg) \
+	vcpu_read_sys_reg(__vcpu, __reg)
+
+#define __vcpu_assign_sys_reg(__vcpu, __reg, __val) \
+	vcpu_write_sys_reg(__vcpu, __val, __reg)
+
+#define __vcpu_rmw_sys_reg(v, r, op, val)				\
+	do {								\
+		u64 __v = vcpu_read_sys_reg(v, r);			\
+		__v op (val);						\
+		vcpu_write_sys_reg(v, __v, r);				\
+	} while (0)
+
+/**
+ * _vcpu_read_sys_reg() - read a guest sysreg with easr
+ * @C: vcpu to read from; if loaded, may only be called from its own thread
+ * @R: sysreg id; must be readable by easr; must be compile time constant
+ *
+ *   if SYSREGS_ON_CPU: proceed with flags = 0
+ *   otherwise:         proceed with flags = EASR_FLAG_SA
+ *
+ */
+#define _vcpu_read_sys_reg(C, R) \
+	({	BUILD_BUG_ON(!__builtin_constant_p((R))); \
+		KVM_BUG_ON(vcpu_is_loaded(C) && smp_processor_id() != (C)->cpu, (C)->kvm); \
+		(vcpu_is_loaded(C)) \
+			? __vcpu_read_sr((C), (R), 0) \
+			: __vcpu_read_sr((C), (R), EASR_FLAG_SA); })
+
+/**
+ * _vcpu_write_sys_reg() - write a guest sysreg with sasr
+ *
+ * @C: vcpu to write to; if loaded, may only be called from its own thread
+ * @V: value to write
+ * @R: sysreg id; must be writable by sasr; must be compile time constant
+
+ *   if SYSREGS_ON_CPU: proceed with flags = 0
+ *   otherwise:         proceed with flags = SASR_FLAG_INITIALIZED
+ */
+#define _vcpu_write_sys_reg(C, V, R) \
+	({	BUILD_BUG_ON(!__builtin_constant_p((R))); \
+		KVM_BUG_ON(vcpu_is_loaded(C) && smp_processor_id() != (C)->cpu, (C)->kvm); \
+		(vcpu_is_loaded(C)) \
+			? __vcpu_write_sr((C), (V), (R), 0) \
+			: __vcpu_write_sr((C), (V), (R), SASR_FLAG_INITIALIZED); })
+
+#define __vcpu_read_sr(C, R, F) \
+	({	BUILD_BUG_ON(!__builtin_constant_p((R))); \
+		BUILD_BUG_ON(!__builtin_constant_p((F))); \
+		easr((R), &(C)->arch.save_area, (F)); })
+
+#define __vcpu_write_sr(C, V, R, F) \
+	({	BUILD_BUG_ON(!__builtin_constant_p((R))); \
+		BUILD_BUG_ON(!__builtin_constant_p((F))); \
+		sasr((R), (V), &(C)->arch.save_area, (F)); })
+
+#define SR_GROUP(NAME, ...)	\
+	__##NAME##_BEGIN__, __after_##NAME##_BEGIN__ = __##NAME##_BEGIN__ - 1,	\
+	__VA_ARGS__								\
+	__##NAME##_END__, __after_##NAME##_END__ = __##NAME##_END__ - 1
+
+/** enum vcpu_sysreg - available guest sysregs
+ *
+ * Contains all arm64 guest-system register supported by s390.
+ */
+enum vcpu_sysreg {
+	__INVALID_SYSREG__, /* 0 is reserved as an invalid value */
+
+	/* EL 0,1 Register from state description in order of appearance */
+	SR_GROUP(STATE_DESC,
+	CNTP_CTL_EL0,
+	CNTV_CTL_EL0,
+	CONTEXTIDR_EL1,
+	SP_EL1,
+	),
+
+	/* EL 0,1 Register requiring special handling. */
+	SR_GROUP(SPECIAL,
+	CSSELR_EL1,
+	CLIDR_EL1,
+	MPIDR_EL1,
+	ELR_EL1,
+	SPSR_EL1,
+	),
+
+	/* EL 0,1 register from save area in order of appearance */
+	SR_GROUP(SAVE_AREA,
+	ACTLR_EL1,
+	AFSR0_EL1,
+	AFSR1_EL1,
+	CNTFRQ_EL0,
+	CNTP_CVAL_EL0,
+	CNTV_CVAL_EL0,
+	DISR_EL1,
+	MIDR_EL1,
+	OSLSR_EL1,
+	PAR_EL1,
+	OSLAR_EL1,
+	SCTLR_EL1,
+	CPACR_EL1,
+	VBAR_EL1,
+	ESR_EL1,
+	TCR_EL1,
+	MAIR_EL1,
+	TTBR0_EL1,
+	TTBR1_EL1,
+	FAR_EL1,
+	TPIDR_EL0,
+	TPIDR_EL1,
+	TPIDRRO_EL0,
+	CNTKCTL_EL1,
+	ZCR_EL1,
+	SCXTNUM_EL0,
+	SCXTNUM_EL1,
+	APIBKEYLO_EL1,
+	APIBKEYHI_EL1,
+	APIAKEYLO_EL1,
+	APIAKEYHI_EL1,
+	APGAKEYLO_EL1,
+	APGAKEYHI_EL1,
+	APDBKEYLO_EL1,
+	APDBKEYHI_EL1,
+	APDAKEYLO_EL1,
+	APDAKEYHI_EL1,
+	MDSCR_EL1,
+	),
+
+	NR_SYS_REGS /* Nothing after this line! */
+};
+
+static __always_inline u64 *___ctxt_sys_reg(struct kvm_cpu_context *ctxt,
+					    const enum vcpu_sysreg reg)
+{
+	switch (reg) {
+	case ELR_EL1:
+		return &ctxt->elr_el1;
+	case SPSR_EL1:
+		return &ctxt->spsr_el1;
+	default:
+		BUG();
+		break;
+	}
+}
+
+void vcpu_write_host_sys_reg(struct kvm_vcpu *vcpu, u64 val, int reg);
+u64 vcpu_read_host_sys_reg(const struct kvm_vcpu *vcpu, int reg);
+
+#define kvm_debug_handle_oslar(_v, _val) /* debug not implemented yet*/
+
+#define kvm_vm_has_ran_once(kvm)					\
+	(test_bit(KVM_ARCH_FLAG_HAS_RAN_ONCE, &(kvm)->arch.flags))
+
+#define kvm_id_regs_final(kvm)						\
+	test_bit(KVM_ARCH_FLAG_ID_REGS_FINAL, &(kvm)->arch.flags)
+
+static inline u8 kvm_arm_pmu_get_pmuver_limit(void)
+{
+	return 0;
+}
+
+int __init kvm_sys_reg_table_init(void);
+
+static inline bool system_supports_poe(void)
+{
+	return false;
+}
+
+static inline bool vgic_host_has_gicv3(void)
+{
+	return false;
+}
+
+static inline bool vgic_host_has_gicv5(void)
+{
+	return false;
+}
+
 #endif /* ASM_KVM_HOST_ARM64_H */
diff --git a/arch/s390/kvm/arm64/Makefile b/arch/s390/kvm/arm64/Makefile
index e75b38071f8a..24ac6f6f6278 100644
--- a/arch/s390/kvm/arm64/Makefile
+++ b/arch/s390/kvm/arm64/Makefile
@@ -18,7 +18,9 @@ kvm-arm64-local := \
 	mmio.o \
 	mmu.o \
 	qaaf.o \
-	reset.o
+	reset.o \
+	spectre.o \
+	sys_regs.o
 
 kvm-arm64-obj := $(kvm-arm64-local)
 kvm-arm64-obj += $(patsubst %.o,%-arm64.o,$(kvm-y))
diff --git a/arch/s390/kvm/arm64/Makefile.gen b/arch/s390/kvm/arm64/Makefile.gen
index 9c33b068fab5..6c62ac412d57 100644
--- a/arch/s390/kvm/arm64/Makefile.gen
+++ b/arch/s390/kvm/arm64/Makefile.gen
@@ -7,7 +7,8 @@
 ARM64_CFILES := \
 	arm.c \
 	guest.c \
-	mmio.c
+	mmio.c \
+	sys_regs.c
 
 quiet_cmd_extract_inc = GEN     $@
       cmd_extract_inc = mkdir -p $(dir $@); \
diff --git a/arch/s390/kvm/arm64/spectre.c b/arch/s390/kvm/arm64/spectre.c
new file mode 100644
index 000000000000..d32b40b9f1bd
--- /dev/null
+++ b/arch/s390/kvm/arm64/spectre.c
@@ -0,0 +1,51 @@
+// SPDX-License-Identifier: GPL-2.0
+
+#include <arm64/spectre.h>
+#include <arm64/sysreg.h>
+
+#include "feature.h"
+
+enum mitigation_state arm64_get_spectre_v2_state(void)
+{
+	u64 pfr0 = read_sanitised_ftr_reg(SYS_ID_AA64PFR0_EL1);
+	enum mitigation_state ms = SPECTRE_VULNERABLE;
+
+	if (SYS_FIELD_GET(ID_AA64PFR0_EL1, CSV2, pfr0) >= ID_AA64PFR0_EL1_CSV2_IMP)
+		ms = SPECTRE_UNAFFECTED;
+
+	return ms;
+}
+
+enum mitigation_state arm64_get_meltdown_state(void)
+{
+	u64 pfr0 = read_sanitised_ftr_reg(SYS_ID_AA64PFR0_EL1);
+	enum mitigation_state ms = SPECTRE_VULNERABLE;
+
+	if (SYS_FIELD_GET(ID_AA64PFR0_EL1, CSV3, pfr0) >= ID_AA64PFR0_EL1_CSV3_IMP)
+		ms = SPECTRE_UNAFFECTED;
+
+	return ms;
+}
+
+enum mitigation_state arm64_get_spectre_v4_state(void)
+{
+	u64 pfr1 = read_sanitised_ftr_reg(SYS_ID_AA64PFR1_EL1);
+	/* TODO: assuming the core is affected by spectre v4 */
+	enum mitigation_state ms = SPECTRE_VULNERABLE;
+
+	if (SYS_FIELD_GET(ID_AA64PFR1_EL1, SSBS, pfr1) >= ID_AA64PFR1_EL1_SSBS_IMP)
+		ms = SPECTRE_MITIGATED;
+
+	return ms;
+}
+
+enum mitigation_state arm64_get_spectre_bhb_state(void)
+{
+	u64 pfr0 = read_sanitised_ftr_reg(SYS_ID_AA64PFR0_EL1);
+	enum mitigation_state ms = SPECTRE_VULNERABLE;
+
+	if (SYS_FIELD_GET(ID_AA64PFR0_EL1, CSV2, pfr0) >= ID_AA64PFR0_EL1_CSV2_CSV2_3)
+		ms = SPECTRE_UNAFFECTED;
+
+	return ms;
+}
diff --git a/arch/s390/kvm/arm64/sys_regs.c b/arch/s390/kvm/arm64/sys_regs.c
new file mode 100644
index 000000000000..b1d4335779c7
--- /dev/null
+++ b/arch/s390/kvm/arm64/sys_regs.c
@@ -0,0 +1,815 @@
+// SPDX-License-Identifier: GPL-2.0-only
+
+#include <linux/kvm_host.h>
+#include <linux/printk.h>
+#include <linux/cpumask.h>
+
+#include <arm64/kvm_emulate.h>
+#include <arm64/sys_regs.h>
+#include <arm64/sysreg.h>
+#include <arm64/cputype.h>
+#include <arm64/cache.h>
+#include <arm64/kvm_nested.h>
+#include <arm64/spectre.h>
+#include <arm64/kvm_hyp.h>
+
+#include "feature.h"
+
+#include <kvm/arm_arch_timer.h>
+
+#include "trace.h"
+
+static int arm64_check_features(struct kvm_vcpu *vcpu,
+				const struct sys_reg_desc *rd, u64 val);
+static const struct sys_reg_desc sys_reg_descs[];
+static const size_t num_sys_reg_descs;
+static u64 __ro_after_init boot_cpu_midr_val;
+static u64 __ro_after_init boot_cpu_revidr_val;
+static u64 __ro_after_init boot_cpu_aidr_val;
+
+#define __INCL_GEN_ARM_FILE
+#include "generated/sys_regs.inc"
+#undef __INCL_GEN_ARM_FILE
+
+DEFINE_STATIC_KEY_FALSE(broken_cntvoff_key);
+
+#define SR_INVALID 0x8badf00d8badf00d
+
+enum sr_loc_attr {
+	SR_LOC_SAVE_AREA,	/* Register on SA, loaded (on cpu) */
+	SR_LOC_SPECIAL,		/* Register in state description or Special care register*/
+	SR_LOC_INVALID,		/* Register is unknown */
+};
+
+struct sr_loc {
+	enum sr_loc_attr loc;
+};
+
+#define SREG_RANGE(NAME) __##NAME##_BEGIN__ ... __##NAME##_END__ - 1
+static __always_inline void locate_register(const struct kvm_vcpu *vcpu,
+					    enum vcpu_sysreg reg,
+					    struct sr_loc *loc)
+{
+	switch (reg) {
+	case SREG_RANGE(STATE_DESC):
+	case SREG_RANGE(SPECIAL):
+		loc->loc = SR_LOC_SPECIAL;
+		break;
+	case SREG_RANGE(SAVE_AREA):
+		loc->loc = SR_LOC_SAVE_AREA;
+		break;
+	default:
+		WARN(true, "%s wants to read invalid register %x", __func__, reg);
+		loc->loc = SR_LOC_INVALID;
+	}
+}
+
+static __always_inline u64 read_special_sr(const struct kvm_vcpu *vcpu,
+					   enum vcpu_sysreg reg)
+{
+	switch (reg) {
+	case CLIDR_EL1:		return vcpu->arch.sys_reg_clidr_el1;
+	case CSSELR_EL1:	return vcpu->arch.sys_reg_csselr_el1;
+	case MPIDR_EL1:		return vcpu->arch.mpidr;
+	case ELR_EL1:		return vcpu->arch.ctxt.elr_el1;
+	case SPSR_EL1:		return vcpu->arch.ctxt.spsr_el1;
+	case CNTP_CTL_EL0:	return vcpu->arch.sae_block.cntp_ctl;
+	case CNTV_CTL_EL0:	return vcpu->arch.sae_block.cntv_ctl;
+	case CONTEXTIDR_EL1:	return vcpu->arch.sae_block.contextidr_el1;
+	case SP_EL1:		return vcpu->arch.sae_block.sp_el1;
+	default:
+		WARN(true, "%s wants to read non-special register %x", __func__, reg);
+		return SR_INVALID;
+	}
+}
+
+static __always_inline u64 read_sr_from_vcpu(const struct kvm_vcpu *vcpu,
+					     enum vcpu_sysreg reg)
+{
+	switch (reg) {
+	case ACTLR_EL1:		return _vcpu_read_sys_reg(vcpu, SYS_ACTLR_EL1);
+	case AFSR0_EL1:		return _vcpu_read_sys_reg(vcpu, SYS_AFSR0_EL1);
+	case AFSR1_EL1:		return _vcpu_read_sys_reg(vcpu, SYS_AFSR1_EL1);
+	case CNTFRQ_EL0:	return _vcpu_read_sys_reg(vcpu, SYS_CNTFRQ_EL0);
+	case CNTP_CVAL_EL0:	return _vcpu_read_sys_reg(vcpu, SYS_CNTP_CVAL_EL0);
+	case CNTV_CVAL_EL0:	return _vcpu_read_sys_reg(vcpu, SYS_CNTV_CVAL_EL0);
+	case DISR_EL1:		return _vcpu_read_sys_reg(vcpu, SYS_DISR_EL1);
+	case MIDR_EL1:		return _vcpu_read_sys_reg(vcpu, SYS_MIDR_EL1);
+	case OSLSR_EL1:		return _vcpu_read_sys_reg(vcpu, SYS_OSLSR_EL1);
+	case PAR_EL1:		return _vcpu_read_sys_reg(vcpu, SYS_PAR_EL1);
+	case SCTLR_EL1:		return _vcpu_read_sys_reg(vcpu, SYS_SCTLR_EL1);
+	case CPACR_EL1:		return _vcpu_read_sys_reg(vcpu, SYS_CPACR_EL1);
+	case VBAR_EL1:		return _vcpu_read_sys_reg(vcpu, SYS_VBAR_EL1);
+	case ESR_EL1:		return _vcpu_read_sys_reg(vcpu, SYS_ESR_EL1);
+	case TCR_EL1:		return _vcpu_read_sys_reg(vcpu, SYS_TCR_EL1);
+	case MAIR_EL1:		return _vcpu_read_sys_reg(vcpu, SYS_MAIR_EL1);
+	case TTBR0_EL1:		return _vcpu_read_sys_reg(vcpu, SYS_TTBR0_EL1);
+	case TTBR1_EL1:		return _vcpu_read_sys_reg(vcpu, SYS_TTBR1_EL1);
+	case FAR_EL1:		return _vcpu_read_sys_reg(vcpu, SYS_FAR_EL1);
+	case TPIDR_EL0:		return _vcpu_read_sys_reg(vcpu, SYS_TPIDR_EL0);
+	case TPIDR_EL1:		return _vcpu_read_sys_reg(vcpu, SYS_TPIDR_EL1);
+	case TPIDRRO_EL0:	return _vcpu_read_sys_reg(vcpu, SYS_TPIDRRO_EL0);
+	case CNTKCTL_EL1:	return _vcpu_read_sys_reg(vcpu, SYS_CNTKCTL_EL1);
+	case ZCR_EL1:		return _vcpu_read_sys_reg(vcpu, SYS_ZCR_EL1);
+	case SCXTNUM_EL0:	return _vcpu_read_sys_reg(vcpu, SYS_SCXTNUM_EL0);
+	case SCXTNUM_EL1:	return _vcpu_read_sys_reg(vcpu, SYS_SCXTNUM_EL1);
+	case APIBKEYLO_EL1:	return _vcpu_read_sys_reg(vcpu, SYS_APIBKEYLO_EL1);
+	case APIBKEYHI_EL1:	return _vcpu_read_sys_reg(vcpu, SYS_APIBKEYHI_EL1);
+	case APIAKEYLO_EL1:	return _vcpu_read_sys_reg(vcpu, SYS_APIAKEYLO_EL1);
+	case APIAKEYHI_EL1:	return _vcpu_read_sys_reg(vcpu, SYS_APIAKEYHI_EL1);
+	case APGAKEYLO_EL1:	return _vcpu_read_sys_reg(vcpu, SYS_APGAKEYLO_EL1);
+	case APGAKEYHI_EL1:	return _vcpu_read_sys_reg(vcpu, SYS_APGAKEYHI_EL1);
+	case APDBKEYLO_EL1:	return _vcpu_read_sys_reg(vcpu, SYS_APDBKEYLO_EL1);
+	case APDBKEYHI_EL1:	return _vcpu_read_sys_reg(vcpu, SYS_APDBKEYHI_EL1);
+	case APDAKEYLO_EL1:	return _vcpu_read_sys_reg(vcpu, SYS_APDAKEYLO_EL1);
+	case APDAKEYHI_EL1:	return _vcpu_read_sys_reg(vcpu, SYS_APDAKEYHI_EL1);
+	case MDSCR_EL1:		return _vcpu_read_sys_reg(vcpu, SYS_MDSCR_EL1);
+	default:
+		WARN(true, "%s: failed to resolve %x", __func__,  reg);
+		return SR_INVALID;
+	}
+}
+
+u64 vcpu_read_sys_reg(const struct kvm_vcpu *vcpu, enum vcpu_sysreg reg)
+{
+	struct sr_loc loc = {};
+
+	locate_register(vcpu, reg, &loc);
+
+	if (unlikely(loc.loc == SR_LOC_INVALID)) {
+		WARN(true, "%s: failed to locate %u", __func__,  reg);
+		return SR_INVALID;
+	}
+
+	if (loc.loc == SR_LOC_SPECIAL)
+		return read_special_sr(vcpu, reg);
+
+	return read_sr_from_vcpu(vcpu, reg);
+}
+
+static __always_inline void write_special_sr(struct kvm_vcpu *vcpu, u64 val,
+					     enum vcpu_sysreg reg)
+{
+	switch (reg) {
+	case CLIDR_EL1:		vcpu->arch.sys_reg_clidr_el1 = val;		break;
+	case CSSELR_EL1:	vcpu->arch.sys_reg_csselr_el1 = val;		break;
+	case MPIDR_EL1:		vcpu->arch.mpidr = val;				break;
+	case CNTP_CTL_EL0:	vcpu->arch.sae_block.cntp_ctl = val;		break;
+	case CNTV_CTL_EL0:	vcpu->arch.sae_block.cntv_ctl = val;		break;
+	case CONTEXTIDR_EL1:	vcpu->arch.sae_block.contextidr_el1 = val;	break;
+	case SP_EL1:		vcpu->arch.sae_block.sp_el1 = val;		break;
+	case ELR_EL1:		vcpu->arch.ctxt.elr_el1 = val;			break;
+	case SPSR_EL1:		vcpu->arch.ctxt.spsr_el1 = val;			break;
+	default:
+		WARN(true, "%s wants to write non-special register %x", __func__, reg);
+	}
+}
+
+static __always_inline void write_sr_to_vcpu(struct kvm_vcpu *vcpu, u64 val,
+					     enum vcpu_sysreg reg)
+{
+	switch (reg) {
+	case ACTLR_EL1:		_vcpu_write_sys_reg(vcpu, val, SYS_ACTLR_EL1);		break;
+	case AFSR0_EL1:		_vcpu_write_sys_reg(vcpu, val, SYS_AFSR0_EL1);		break;
+	case AFSR1_EL1:		_vcpu_write_sys_reg(vcpu, val, SYS_AFSR1_EL1);		break;
+	case CNTFRQ_EL0:	_vcpu_write_sys_reg(vcpu, val, SYS_CNTFRQ_EL0);		break;
+	case CNTP_CVAL_EL0:	_vcpu_write_sys_reg(vcpu, val, SYS_CNTP_CVAL_EL0);	break;
+	case CNTV_CVAL_EL0:	_vcpu_write_sys_reg(vcpu, val, SYS_CNTV_CVAL_EL0);	break;
+	case DISR_EL1:		_vcpu_write_sys_reg(vcpu, val, SYS_DISR_EL1);		break;
+	case MIDR_EL1:		_vcpu_write_sys_reg(vcpu, val, SYS_MIDR_EL1);		break;
+	case PAR_EL1:		_vcpu_write_sys_reg(vcpu, val, SYS_PAR_EL1);		break;
+	case OSLAR_EL1:		_vcpu_write_sys_reg(vcpu, val, SYS_OSLAR_EL1);		break;
+	case SCTLR_EL1:		_vcpu_write_sys_reg(vcpu, val, SYS_SCTLR_EL1);		break;
+	case CPACR_EL1:		_vcpu_write_sys_reg(vcpu, val, SYS_CPACR_EL1);		break;
+	case VBAR_EL1:		_vcpu_write_sys_reg(vcpu, val, SYS_VBAR_EL1);		break;
+	case ESR_EL1:		_vcpu_write_sys_reg(vcpu, val, SYS_ESR_EL1);		break;
+	case TCR_EL1:		_vcpu_write_sys_reg(vcpu, val, SYS_TCR_EL1);		break;
+	case MAIR_EL1:		_vcpu_write_sys_reg(vcpu, val, SYS_MAIR_EL1);		break;
+	case TTBR0_EL1:		_vcpu_write_sys_reg(vcpu, val, SYS_TTBR0_EL1);		break;
+	case TTBR1_EL1:		_vcpu_write_sys_reg(vcpu, val, SYS_TTBR1_EL1);		break;
+	case FAR_EL1:		_vcpu_write_sys_reg(vcpu, val, SYS_FAR_EL1);		break;
+	case TPIDR_EL0:		_vcpu_write_sys_reg(vcpu, val, SYS_TPIDR_EL0);		break;
+	case TPIDR_EL1:		_vcpu_write_sys_reg(vcpu, val, SYS_TPIDR_EL1);		break;
+	case TPIDRRO_EL0:	_vcpu_write_sys_reg(vcpu, val, SYS_TPIDRRO_EL0);	break;
+	case CNTKCTL_EL1:	_vcpu_write_sys_reg(vcpu, val, SYS_CNTKCTL_EL1);	break;
+	case ZCR_EL1:		_vcpu_write_sys_reg(vcpu, val, SYS_ZCR_EL1);		break;
+	case SCXTNUM_EL0:	_vcpu_write_sys_reg(vcpu, val, SYS_SCXTNUM_EL0);	break;
+	case SCXTNUM_EL1:	_vcpu_write_sys_reg(vcpu, val, SYS_SCXTNUM_EL1);	break;
+	case APIBKEYLO_EL1:	_vcpu_write_sys_reg(vcpu, val, SYS_APIBKEYLO_EL1);	break;
+	case APIBKEYHI_EL1:	_vcpu_write_sys_reg(vcpu, val, SYS_APIBKEYHI_EL1);	break;
+	case APIAKEYLO_EL1:	_vcpu_write_sys_reg(vcpu, val, SYS_APIAKEYLO_EL1);	break;
+	case APIAKEYHI_EL1:	_vcpu_write_sys_reg(vcpu, val, SYS_APIAKEYHI_EL1);	break;
+	case APGAKEYLO_EL1:	_vcpu_write_sys_reg(vcpu, val, SYS_APGAKEYLO_EL1);	break;
+	case APGAKEYHI_EL1:	_vcpu_write_sys_reg(vcpu, val, SYS_APGAKEYHI_EL1);	break;
+	case APDBKEYLO_EL1:	_vcpu_write_sys_reg(vcpu, val, SYS_APDBKEYLO_EL1);	break;
+	case APDBKEYHI_EL1:	_vcpu_write_sys_reg(vcpu, val, SYS_APDBKEYHI_EL1);	break;
+	case APDAKEYLO_EL1:	_vcpu_write_sys_reg(vcpu, val, SYS_APDAKEYLO_EL1);	break;
+	case APDAKEYHI_EL1:	_vcpu_write_sys_reg(vcpu, val, SYS_APDAKEYHI_EL1);	break;
+	case MDSCR_EL1:		_vcpu_write_sys_reg(vcpu, val, SYS_MDSCR_EL1);		break;
+	default:
+		WARN(true, "%s: failed to resolve %x", __func__,  reg);
+	}
+}
+
+void vcpu_write_sys_reg(struct kvm_vcpu *vcpu, u64 val, enum vcpu_sysreg reg)
+{
+	struct sr_loc loc = {};
+
+	locate_register(vcpu, reg, &loc);
+
+	if (unlikely(loc.loc == SR_LOC_INVALID)) {
+		WARN(true, "%s: failed to locate %u", __func__,  reg);
+		return;
+	}
+
+	if (loc.loc == SR_LOC_SPECIAL) {
+		write_special_sr(vcpu, val, reg);
+		return;
+	}
+
+	write_sr_to_vcpu(vcpu, val, reg);
+}
+
+u64 vcpu_read_host_sys_reg(const struct kvm_vcpu *vcpu, int reg)
+{
+	switch (reg) {
+	case SYS_ICH_LR0_EL2:	return vcpu->arch.sae_block.ic_regs.ich_lrn_el2[0];
+	case SYS_ICH_LR1_EL2:	return vcpu->arch.sae_block.ic_regs.ich_lrn_el2[1];
+	case SYS_ICH_LR2_EL2:	return vcpu->arch.sae_block.ic_regs.ich_lrn_el2[2];
+	case SYS_ICH_LR3_EL2:	return vcpu->arch.sae_block.ic_regs.ich_lrn_el2[3];
+	case SYS_ICH_HCR_EL2:	return vcpu->arch.sae_block.ic_regs.ich_hcr_el2;
+	case SYS_ICH_AP0R0_EL2:	return vcpu->arch.sae_block.ic_regs.ich_ap0r0_el2;
+	case SYS_ICH_AP1R0_EL2:	return vcpu->arch.sae_block.ic_regs.ich_ap1r0_el2;
+	case SYS_ICH_VMCR_EL2:	return vcpu->arch.sae_block.ic_regs.ich_vmcr_el2;
+	case SYS_VSESR_EL2:	return _vcpu_read_sys_reg(vcpu, SYS_VSESR_EL2);
+	case SYS_HCR_EL2:	return _vcpu_read_sys_reg(vcpu, SYS_HCR_EL2);
+	case SYS_CPTR_EL2:	return _vcpu_read_sys_reg(vcpu, SYS_CPTR_EL2);
+	case SYS_MDCR_EL2:	return _vcpu_read_sys_reg(vcpu, SYS_MDCR_EL2);
+	case SYS_HCRX_EL2:	return _vcpu_read_sys_reg(vcpu, SYS_HCRX_EL2);
+	case SYS_HFGRTR_EL2:	return _vcpu_read_sys_reg(vcpu, SYS_HFGRTR_EL2);
+	case SYS_HFGWTR_EL2:	return _vcpu_read_sys_reg(vcpu, SYS_HFGWTR_EL2);
+	case SYS_HDFGWTR_EL2:	return _vcpu_read_sys_reg(vcpu, SYS_HDFGWTR_EL2);
+	case SYS_HDFGRTR_EL2:	return _vcpu_read_sys_reg(vcpu, SYS_HDFGRTR_EL2);
+	case SYS_HFGITR_EL2:	return _vcpu_read_sys_reg(vcpu, SYS_HFGITR_EL2);
+	case SYS_CNTHCTL_EL2:	return _vcpu_read_sys_reg(vcpu, SYS_CNTHCTL_EL2);
+	case SYS_ICC_SRE_EL2:	return _vcpu_read_sys_reg(vcpu, SYS_ICC_SRE_EL2);
+	case SYS_VMPIDR_EL2:	return _vcpu_read_sys_reg(vcpu, SYS_VMPIDR_EL2);
+	default:
+		WARN(true, "%s wants to read non-host register %x", __func__, reg);
+		return SR_INVALID;
+	}
+}
+
+void vcpu_write_host_sys_reg(struct kvm_vcpu *vcpu, u64 val, int reg)
+{
+	switch (reg) {
+	case SYS_ICH_LR0_EL2:	vcpu->arch.sae_block.ic_regs.ich_lrn_el2[0] = val;	break;
+	case SYS_ICH_LR1_EL2:	vcpu->arch.sae_block.ic_regs.ich_lrn_el2[1] = val;	break;
+	case SYS_ICH_LR2_EL2:	vcpu->arch.sae_block.ic_regs.ich_lrn_el2[2] = val;	break;
+	case SYS_ICH_LR3_EL2:	vcpu->arch.sae_block.ic_regs.ich_lrn_el2[3] = val;	break;
+	case SYS_ICH_HCR_EL2:	vcpu->arch.sae_block.ic_regs.ich_hcr_el2 = val;		break;
+	case SYS_ICH_AP0R0_EL2:	vcpu->arch.sae_block.ic_regs.ich_ap0r0_el2 = val;	break;
+	case SYS_ICH_AP1R0_EL2:	vcpu->arch.sae_block.ic_regs.ich_ap1r0_el2 = val;	break;
+	case SYS_ICH_VMCR_EL2:	vcpu->arch.sae_block.ic_regs.ich_vmcr_el2 = val;	break;
+	case SYS_VSESR_EL2:	_vcpu_write_sys_reg(vcpu, val, SYS_VSESR_EL2);		break;
+	case SYS_HCR_EL2:	_vcpu_write_sys_reg(vcpu, val, SYS_HCR_EL2);		break;
+	case SYS_CPTR_EL2:	_vcpu_write_sys_reg(vcpu, val, SYS_CPTR_EL2);		break;
+	case SYS_MDCR_EL2:	_vcpu_write_sys_reg(vcpu, val, SYS_MDCR_EL2);		break;
+	case SYS_HCRX_EL2:	_vcpu_write_sys_reg(vcpu, val, SYS_HCRX_EL2);		break;
+	case SYS_HFGRTR_EL2:	_vcpu_write_sys_reg(vcpu, val, SYS_HFGRTR_EL2);		break;
+	case SYS_HFGWTR_EL2:	_vcpu_write_sys_reg(vcpu, val, SYS_HFGWTR_EL2);		break;
+	case SYS_HDFGWTR_EL2:	_vcpu_write_sys_reg(vcpu, val, SYS_HDFGWTR_EL2);	break;
+	case SYS_HDFGRTR_EL2:	_vcpu_write_sys_reg(vcpu, val, SYS_HDFGRTR_EL2);	break;
+	case SYS_HFGITR_EL2:	_vcpu_write_sys_reg(vcpu, val, SYS_HFGITR_EL2);		break;
+	case SYS_CNTHCTL_EL2:	_vcpu_write_sys_reg(vcpu, val, SYS_CNTHCTL_EL2);	break;
+	case SYS_ICC_SRE_EL2:	_vcpu_write_sys_reg(vcpu, val, SYS_ICC_SRE_EL2);	break;
+	case SYS_VMPIDR_EL2:	_vcpu_write_sys_reg(vcpu, val, SYS_VMPIDR_EL2);		break;
+	default:
+		WARN(true, "%s wants to write non-host register %x", __func__, reg);
+	}
+}
+
+static int arm64_check_features(struct kvm_vcpu *vcpu,
+				const struct sys_reg_desc *rd, u64 val)
+{
+	u32 id = reg_to_encoding(rd);
+	u64 host_val;
+	u64 old_val;
+
+	/*
+	 * This is a very minimal implementation to sanitise only the craziest
+	 * shenanigans we could get from userspace. The final goal is to share
+	 * the sanitisation with arm as well but this needs access to arm
+	 * (core) kernel code for which there is no suitable solution yet. TODO
+	 */
+
+	/*
+	 * If the register is RAZ we know the only safe value is 0.
+	 */
+	if (sysreg_visible_as_raz(vcpu, rd))
+		return val ? -E2BIG : 0;
+
+	host_val = read_sanitised_ftr_reg(id);
+
+	switch (id) {
+	case SYS_ID_AA64MMFR0_EL1:
+		/* Forbid PARANGE values larger than the host supports */
+		if (SYS_FIELD_GET(ID_AA64MMFR0_EL1, PARANGE, val) >
+		    SYS_FIELD_GET(ID_AA64MMFR0_EL1, PARANGE, host_val))
+			return -E2BIG;
+		break;
+	case SYS_CTR_EL0:
+		/* forbid upgrading from VIPT to PIPT */
+		old_val = read_id_reg(vcpu, rd);
+		if (SYS_FIELD_GET(CTR_EL0, L1Ip, old_val) == CTR_EL0_L1Ip_VIPT &&
+		    SYS_FIELD_GET(CTR_EL0, L1Ip, val) == CTR_EL0_L1Ip_PIPT)
+			return -E2BIG;
+		break;
+	}
+	return 0;
+}
+
+static void init_imp_id_regs(void)
+{
+	boot_cpu_midr_val = read_sanitised_ftr_reg(SYS_MIDR_EL1);
+	boot_cpu_revidr_val = read_sanitised_ftr_reg(SYS_REVIDR_EL1);
+	boot_cpu_aidr_val = read_sanitised_ftr_reg(SYS_AIDR_EL1);
+}
+
+static bool access_imp_id_reg(struct kvm_vcpu *vcpu, struct sys_reg_params *p,
+			      const struct sys_reg_desc *r)
+{
+	if (p->is_write)
+		return write_to_read_only(vcpu, p, r);
+
+	/*
+	 * Return the VM-scoped implementation ID register values if userspace
+	 * has made them writable.
+	 */
+	if (test_bit(KVM_ARCH_FLAG_WRITABLE_IMP_ID_REGS, &vcpu->kvm->arch.flags))
+		return access_id_reg(vcpu, p, r);
+
+	switch (reg_to_encoding(r)) {
+	case SYS_REVIDR_EL1:
+		/* No historical ABI to preserve here */
+		p->regval = boot_cpu_revidr_val;
+		break;
+	case SYS_AIDR_EL1:
+		p->regval = boot_cpu_aidr_val;
+		break;
+	default:
+		WARN_ON_ONCE(1);
+	}
+
+	return true;
+}
+
+static int get_mpidr(struct kvm_vcpu *vcpu, const struct sys_reg_desc *rd, u64 *val)
+{
+	*val = vcpu->arch.mpidr;
+	return 0;
+}
+
+static int set_mpidr(struct kvm_vcpu *vcpu, const struct sys_reg_desc *rd, u64 val)
+{
+	vcpu->arch.mpidr = val;
+	return 0;
+}
+
+static u64 reset_midr(struct kvm_vcpu *vcpu, const struct sys_reg_desc *r)
+{
+	_vcpu_write_sys_reg(vcpu, boot_cpu_midr_val, SYS_MIDR_EL1);
+	return boot_cpu_midr_val;
+}
+
+static int set_oslsr_el1(struct kvm_vcpu *vcpu, const struct sys_reg_desc *rd,
+			 u64 val)
+{
+	u64 oslk;
+
+	/*
+	 * The only modifiable bit is the OSLK bit. Refuse the write if
+	 * userspace attempts to change any other bit in the register.
+	 */
+	if ((val ^ rd->val) & ~OSLSR_EL1_OSLK)
+		return -EINVAL;
+
+	/*
+	 * Redirect the write to the proper control register.
+	 * OSLSR is read-only
+	 */
+	oslk = SYS_FIELD_GET(OSLSR_EL1, OSLK, val);
+	__vcpu_assign_sys_reg(vcpu, OSLAR_EL1, SYS_FIELD_PREP(OSLAR_EL1, OSLK, oslk));
+	return 0;
+}
+
+static u64 reset_oslsr_el1(struct kvm_vcpu *vcpu, const struct sys_reg_desc *r)
+{
+	u64 oslsr = r->val;
+
+	set_oslsr_el1(vcpu, r, oslsr);
+	return oslsr;
+}
+
+static int set_cntfrq(struct kvm_vcpu *vcpu, const struct sys_reg_desc *rd, u64 val)
+{
+	/*
+	 * reject the write if it disagrees with our view of CNTFRQ_EL0
+	 */
+	if (val != read_sanitised_ftr_reg(SYS_CNTFRQ_EL0))
+		return -EINVAL;
+	return 0;
+}
+
+static u64 reset_cntfrq(struct kvm_vcpu *vcpu, const struct sys_reg_desc *r)
+{
+	u64 cntfrq = read_sanitised_ftr_reg(SYS_CNTFRQ_EL0);
+
+	_vcpu_write_sys_reg(vcpu, cntfrq, SYS_CNTFRQ_EL0);
+	return cntfrq;
+}
+
+static int arch_timer_set_user(struct kvm_vcpu *vcpu,
+			       const struct sys_reg_desc *rd,
+			       u64 val)
+{
+	u64 gpto;
+
+	switch (reg_to_encoding(rd)) {
+	/* PTFF knows only one SAE guest counter offset */
+	case SYS_CNTVCT_EL0:
+	case SYS_CNTPCT_EL0:
+		gpto = ptff_qagto(val);
+		vcpu->arch.sae_block.gpto = gpto;
+		return 0;
+	}
+
+	__vcpu_assign_sys_reg(vcpu, rd->reg, val);
+	return 0;
+}
+
+static int arch_timer_get_user(struct kvm_vcpu *vcpu,
+			       const struct sys_reg_desc *rd,
+			       u64 *val)
+{
+	switch (reg_to_encoding(rd)) {
+	/* PTFF knows only one SAE guest counter offset */
+	case SYS_CNTVCT_EL0:
+	case SYS_CNTPCT_EL0:
+		*val = ptff_qagpt(vcpu->arch.sae_block.gpto);
+		break;
+	default:
+		*val = __vcpu_sys_reg(vcpu, rd->reg);
+	}
+
+	return 0;
+}
+
+#define GUEST_TIMER_REG(name) { \
+	SYS_DESC(SYS_##name),			\
+	.reset = reset_val,			\
+	.reg = name,				\
+	.get_user = arch_timer_get_user,	\
+	.set_user = arch_timer_set_user,	\
+}
+
+/*
+ * Architected system registers.
+ * Important: Must be sorted ascending by Op0, Op1, CRn, CRm, Op2
+ */
+static const struct sys_reg_desc sys_reg_descs[] = {
+	/* Op0 = 2 */
+	{ SYS_DESC(SYS_MDSCR_EL1), access_rw, reset_val, MDSCR_EL1, 0 },
+	{ SYS_DESC(SYS_OSLAR_EL1), trap_oslar_el1 },
+	{ SYS_DESC(SYS_OSLSR_EL1), trap_oslsr_el1, reset_oslsr_el1, OSLSR_EL1,
+	  OSLSR_EL1_OSLM_IMP, NULL, set_oslsr_el1 },
+
+	/* Op0 = 3 */
+	/* Op1 = 0 */
+	/* CRn = 0 */
+	/* CRm = 0 */
+
+	{ SYS_DESC(SYS_MIDR_EL1), NULL, reset_midr, 0, GENMASK_ULL(31, 0),
+	  get_id_reg, set_imp_id_reg },
+	{ SYS_DESC(SYS_MPIDR_EL1), NULL, reset_mpidr, MPIDR_EL1, 0, get_mpidr,
+	  set_mpidr },
+	IMPLEMENTATION_ID(REVIDR_EL1, GENMASK_ULL(63, 0)),
+
+	/*
+	 * ID regs: all ID_SANITISED() entries here must have corresponding
+	 * entries in arm64_ftr_regs[].
+	 */
+
+	/* AArch64 mappings of the AArch32 ID registers */
+	/* CRm=1 */
+	ID_HIDDEN(ID_PFR0_EL1),
+	ID_HIDDEN(ID_PFR1_EL1),
+	ID_HIDDEN(ID_DFR0_EL1),
+	ID_HIDDEN(ID_AFR0_EL1),
+	ID_HIDDEN(ID_MMFR0_EL1),
+	ID_HIDDEN(ID_MMFR1_EL1),
+	ID_HIDDEN(ID_MMFR2_EL1),
+	ID_HIDDEN(ID_MMFR3_EL1),
+
+	/* CRm=2 */
+	ID_HIDDEN(ID_ISAR0_EL1),
+	ID_HIDDEN(ID_ISAR1_EL1),
+	ID_HIDDEN(ID_ISAR2_EL1),
+	ID_HIDDEN(ID_ISAR3_EL1),
+	ID_HIDDEN(ID_ISAR4_EL1),
+	ID_HIDDEN(ID_ISAR5_EL1),
+	ID_HIDDEN(ID_MMFR4_EL1),
+	ID_HIDDEN(ID_ISAR6_EL1),
+
+	/* CRm=3 */
+	ID_HIDDEN(MVFR0_EL1),
+	ID_HIDDEN(MVFR1_EL1),
+	ID_HIDDEN(MVFR2_EL1),
+	ID_UNALLOCATED(3, 3),
+	ID_HIDDEN(ID_PFR2_EL1),
+	ID_HIDDEN(ID_DFR1_EL1),
+	ID_HIDDEN(ID_MMFR5_EL1),
+	ID_UNALLOCATED(3, 7),
+
+	/* AArch64 ID registers */
+	/* CRm=4 */
+	ID_FILTERED(ID_AA64PFR0_EL1, id_aa64pfr0_el1,
+		    ~(ID_AA64PFR0_EL1_AMU | ID_AA64PFR0_EL1_MPAM |
+		      ID_AA64PFR0_EL1_SVE | ID_AA64PFR0_EL1_RAS |
+		      ID_AA64PFR0_EL1_GIC | ID_AA64PFR0_EL1_AdvSIMD |
+		      ID_AA64PFR0_EL1_FP)),
+	ID_FILTERED(ID_AA64PFR1_EL1, id_aa64pfr1_el1,
+		    ~(ID_AA64PFR1_EL1_PFAR | ID_AA64PFR1_EL1_DF2 |
+		      ID_AA64PFR1_EL1_MTEX | ID_AA64PFR1_EL1_THE |
+		      ID_AA64PFR1_EL1_GCS | ID_AA64PFR1_EL1_MTE_frac |
+		      ID_AA64PFR1_EL1_NMI | ID_AA64PFR1_EL1_RNDR_trap |
+		      ID_AA64PFR1_EL1_SME | ID_AA64PFR1_EL1_RES0 |
+		      ID_AA64PFR1_EL1_MPAM_frac | ID_AA64PFR1_EL1_RAS_frac |
+		      ID_AA64PFR1_EL1_MTE)),
+	ID_FILTERED(ID_AA64PFR2_EL1, id_aa64pfr2_el1,
+		    (ID_AA64PFR2_EL1_FPMR		|
+		     ID_AA64PFR2_EL1_MTEFAR		|
+		     ID_AA64PFR2_EL1_MTESTOREONLY	|
+		     ID_AA64PFR2_EL1_GCIE)),
+	ID_UNALLOCATED(4, 3),
+	ID_WRITABLE(ID_AA64ZFR0_EL1, ~ID_AA64ZFR0_EL1_RES0),
+	ID_HIDDEN(ID_AA64SMFR0_EL1),
+	ID_UNALLOCATED(4, 6),
+	ID_WRITABLE(ID_AA64FPFR0_EL1, ~ID_AA64FPFR0_EL1_RES0),
+
+	/* CRm=5 */
+	/*
+	 * Prior to FEAT_Debugv8.9, the architecture defines context-aware
+	 * breakpoints (CTX_CMPs) as the highest numbered breakpoints (BRPs).
+	 * KVM does not trap + emulate the breakpoint registers, and as such
+	 * cannot support a layout that misaligns with the underlying hardware.
+	 * While it may be possible to describe a subset that aligns with
+	 * hardware, just prevent changes to BRPs and CTX_CMPs altogether for
+	 * simplicity.
+	 *
+	 * See DDI0487K.a, section D2.8.3 Breakpoint types and linking
+	 * of breakpoints for more details.
+	 */
+	ID_FILTERED(ID_AA64DFR0_EL1, id_aa64dfr0_el1,
+		    ID_AA64DFR0_EL1_DoubleLock_MASK |
+			    ID_AA64DFR0_EL1_WRPs_MASK |
+			    ID_AA64DFR0_EL1_PMUVer_MASK |
+			    ID_AA64DFR0_EL1_DebugVer_MASK),
+	ID_SANITISED(ID_AA64DFR1_EL1),
+	ID_UNALLOCATED(5, 2),
+	ID_UNALLOCATED(5, 3),
+	ID_HIDDEN(ID_AA64AFR0_EL1),
+	ID_HIDDEN(ID_AA64AFR1_EL1),
+	ID_UNALLOCATED(5, 6),
+	ID_UNALLOCATED(5, 7),
+
+	/* CRm=6 */
+	ID_WRITABLE(ID_AA64ISAR0_EL1, ~ID_AA64ISAR0_EL1_RES0),
+	ID_WRITABLE(ID_AA64ISAR1_EL1,
+		    ~(ID_AA64ISAR1_EL1_GPI | ID_AA64ISAR1_EL1_GPA |
+		      ID_AA64ISAR1_EL1_API | ID_AA64ISAR1_EL1_APA)),
+	ID_WRITABLE(ID_AA64ISAR2_EL1,
+		    ~(ID_AA64ISAR2_EL1_RES0 | ID_AA64ISAR2_EL1_APA3 |
+		      ID_AA64ISAR2_EL1_GPA3)),
+	ID_WRITABLE(ID_AA64ISAR3_EL1,
+		    (ID_AA64ISAR3_EL1_FPRCVT | ID_AA64ISAR3_EL1_FAMINMAX)),
+	ID_UNALLOCATED(6, 4),
+	ID_UNALLOCATED(6, 5),
+	ID_UNALLOCATED(6, 6),
+	ID_UNALLOCATED(6, 7),
+
+	/* CRm=7 */
+	ID_FILTERED(ID_AA64MMFR0_EL1, id_aa64mmfr0_el1,
+		    ~(ID_AA64MMFR0_EL1_RES0 | ID_AA64MMFR0_EL1_ASIDBITS)),
+	ID_WRITABLE(ID_AA64MMFR1_EL1,
+		    ~(ID_AA64MMFR1_EL1_RES0 | ID_AA64MMFR1_EL1_HCX |
+		      ID_AA64MMFR1_EL1_TWED | ID_AA64MMFR1_EL1_XNX |
+		      ID_AA64MMFR1_EL1_VH | ID_AA64MMFR1_EL1_VMIDBits)),
+	ID_FILTERED(ID_AA64MMFR2_EL1, id_aa64mmfr2_el1,
+		    ~(ID_AA64MMFR2_EL1_RES0 | ID_AA64MMFR2_EL1_EVT |
+		      ID_AA64MMFR2_EL1_FWB | ID_AA64MMFR2_EL1_IDS |
+		      ID_AA64MMFR2_EL1_NV | ID_AA64MMFR2_EL1_CCIDX)),
+	ID_WRITABLE(ID_AA64MMFR3_EL1,
+		    (ID_AA64MMFR3_EL1_TCRX | ID_AA64MMFR3_EL1_S1PIE |
+		     ID_AA64MMFR3_EL1_S1POE)),
+	ID_WRITABLE(ID_AA64MMFR4_EL1, ID_AA64MMFR4_EL1_NV_frac),
+	ID_UNALLOCATED(7, 5),
+	ID_UNALLOCATED(7, 6),
+	ID_UNALLOCATED(7, 7),
+
+	/* CRn = 1 */
+	{ SYS_DESC(SYS_SCTLR_EL1), NULL, reset_val, SCTLR_EL1, 0x00C50078 },
+	{ SYS_DESC(SYS_ACTLR_EL1), access_rw, reset_val, ACTLR_EL1, 0 },
+	{ SYS_DESC(SYS_CPACR_EL1), NULL, reset_val, CPACR_EL1, 0 },
+	{ SYS_DESC(SYS_ZCR_EL1), NULL, reset_val, ZCR_EL1, 0,
+	  .visibility = sve_visibility },
+
+	/* CRn = 2 */
+	{ SYS_DESC(SYS_TTBR0_EL1), access_rw, reset_val, TTBR0_EL1, 0 },
+	{ SYS_DESC(SYS_TTBR1_EL1), access_rw, reset_val, TTBR1_EL1, 0 },
+	{ SYS_DESC(SYS_TCR_EL1), access_rw, reset_val, TCR_EL1, 0 },
+
+	/* CRn = 5 */
+	{ SYS_DESC(SYS_AFSR0_EL1), access_rw, reset_unknown, AFSR0_EL1 },
+	{ SYS_DESC(SYS_AFSR1_EL1), access_rw, reset_unknown, AFSR1_EL1 },
+	{ SYS_DESC(SYS_ESR_EL1), access_rw, reset_unknown, ESR_EL1 },
+	/* CRn = 6 */
+	{ SYS_DESC(SYS_FAR_EL1), access_rw, reset_unknown, FAR_EL1 },
+	/* CRn = 7 */
+	{ SYS_DESC(SYS_PAR_EL1), access_rw, reset_unknown, PAR_EL1 },
+
+	{ SYS_DESC(SYS_PMMIR_EL1), trap_raz_wi },
+	/* CRn = 10 */
+	{ SYS_DESC(SYS_MAIR_EL1), NULL, reset_unknown, MAIR_EL1 },
+	{ SYS_DESC(SYS_LORSA_EL1), trap_loregion },
+	{ SYS_DESC(SYS_LOREA_EL1), trap_loregion },
+	{ SYS_DESC(SYS_LORN_EL1), trap_loregion },
+	{ SYS_DESC(SYS_LORC_EL1), trap_loregion },
+	{ SYS_DESC(SYS_LORID_EL1), trap_loregion },
+
+	/* CRn = 12 */
+	{ SYS_DESC(SYS_VBAR_EL1), access_rw, reset_val, VBAR_EL1, 0 },
+
+	/* CRn = 13 */
+	{ SYS_DESC(SYS_CONTEXTIDR_EL1), access_rw, reset_val, CONTEXTIDR_EL1, 0 },
+	{ SYS_DESC(SYS_TPIDR_EL1), NULL, reset_unknown, TPIDR_EL1 },
+	{ SYS_DESC(SYS_SCXTNUM_EL1), access_rw, reset_unknown, SCXTNUM_EL1 },
+
+	/* CRn = 14 */
+	{ SYS_DESC(SYS_CNTKCTL_EL1), access_rw, reset_val, CNTKCTL_EL1, 0 },
+
+	/* Op1 = 1 */
+	/* CRn = 0 */
+	/* CRm = 0 */
+	{ SYS_DESC(SYS_CCSIDR_EL1), access_ccsidr },
+	{ SYS_DESC(SYS_CLIDR_EL1), access_clidr, reset_clidr, CLIDR_EL1,
+	  ~CLIDR_EL1_RES0, .set_user = set_clidr },
+	IMPLEMENTATION_ID(AIDR_EL1, GENMASK_ULL(63, 0)),
+	{ SYS_DESC(SYS_CSSELR_EL1), access_csselr, reset_unknown, CSSELR_EL1 },
+	ID_FILTERED(CTR_EL0, ctr_el0,
+		    CTR_EL0_DIC_MASK | CTR_EL0_IDC_MASK |
+			    CTR_EL0_DminLine_MASK | CTR_EL0_L1Ip_MASK |
+			    CTR_EL0_IminLine_MASK),
+
+	/* Op1 = 3 */
+	/* CRn = 13 */
+	{ SYS_DESC(SYS_TPIDR_EL0), access_rw, reset_unknown, TPIDR_EL0 },
+	{ SYS_DESC(SYS_TPIDRRO_EL0), access_rw, reset_unknown, TPIDRRO_EL0 },
+	{ SYS_DESC(SYS_SCXTNUM_EL0), access_rw, reset_unknown, SCXTNUM_EL0 },
+
+	{ SYS_DESC(SYS_CNTFRQ_EL0), NULL, reset_cntfrq, CNTFRQ_EL0, .set_user = set_cntfrq },
+	{ SYS_DESC(SYS_CNTPCT_EL0), .get_user = arch_timer_get_user,
+	  .set_user = arch_timer_set_user },
+	{ SYS_DESC(SYS_CNTVCT_EL0), .get_user = arch_timer_get_user,
+	  .set_user = arch_timer_set_user },
+	GUEST_TIMER_REG(CNTP_CTL_EL0),
+	GUEST_TIMER_REG(CNTP_CVAL_EL0),
+	GUEST_TIMER_REG(CNTV_CTL_EL0),
+	GUEST_TIMER_REG(CNTV_CVAL_EL0),
+};
+
+static const size_t num_sys_reg_descs = ARRAY_SIZE(sys_reg_descs);
+
+/**
+ * kvm_reset_sys_regs - sets system registers to reset value
+ * @vcpu: The VCPU pointer
+ *
+ * This function finds the right table above and sets the registers on the
+ * virtual CPU struct to their architecturally defined reset values.
+ */
+void kvm_reset_sys_regs(struct kvm_vcpu *vcpu)
+{
+	struct kvm *kvm = vcpu->kvm;
+	unsigned long i;
+
+	for (i = 0; i < num_sys_reg_descs; i++) {
+		const struct sys_reg_desc *r = &sys_reg_descs[i];
+
+		if (!r->reset)
+			continue;
+
+		if (is_vm_ftr_id_reg(reg_to_encoding(r)))
+			reset_vm_ftr_id_reg(vcpu, r);
+		else if (is_vcpu_ftr_id_reg(reg_to_encoding(r)))
+			reset_vcpu_ftr_id_reg(vcpu, r);
+		else
+			r->reset(vcpu, r);
+	}
+
+	/*
+	 * kvm_reset_vcpu will override the save area with the QAAF GISRSA response
+	 * unconditionally, which clobbers the value of MIDR_EL1, so write it here.
+	 */
+	_vcpu_write_sys_reg(vcpu, boot_cpu_midr_val, SYS_MIDR_EL1);
+
+	set_bit(KVM_ARCH_FLAG_ID_REGS_INITIALIZED, &kvm->arch.flags);
+
+	if (kvm_vcpu_has_pmu(vcpu))
+		kvm_make_request(KVM_REQ_RELOAD_PMU, vcpu);
+}
+
+/*
+ * kvm_handle_sys_reg -- handles a system instruction or mrs/msr instruction
+ *			 trap on a guest execution
+ * @vcpu: The VCPU pointer
+ */
+int kvm_handle_sys_reg(struct kvm_vcpu *vcpu)
+{
+	const struct sys_reg_desc *desc = NULL;
+	struct sys_reg_params params;
+	unsigned long esr = kvm_vcpu_get_esr(vcpu);
+	int Rt = kvm_vcpu_sys_get_rt(vcpu);
+
+	trace_kvm_handle_sys_reg(esr);
+
+	params = esr_sys64_to_params(esr);
+	params.regval = vcpu_get_reg(vcpu, Rt);
+
+	/* System registers have Op0=={2,3}, as per DDI487 J.a C5.1.2 */
+	if (params.Op0 == 2 || params.Op0 == 3)
+		desc = find_reg(&params, sys_reg_descs, num_sys_reg_descs);
+	else
+		kvm_pr_unimpl("system instruction handling not supported");
+
+	if (!desc) {
+		if (!(params.Op0 == 3 && (params.CRn & 0b1011) == 0b1011))
+			print_sys_reg_msg(&params,
+					  "Unsupported guest access at: %lx\n",
+					  *vcpu_pc(vcpu));
+		kvm_inject_undefined(vcpu);
+		return 1;
+	} else {
+		perform_access(vcpu, &params, desc);
+	}
+
+	/* Read from system register? */
+	if (!params.is_write &&
+	    (params.Op0 == 2 || params.Op0 == 3))
+		vcpu_set_reg(vcpu, Rt, params.regval);
+
+	return 1;
+}
+
+int __init kvm_sys_reg_table_init(void)
+{
+	init_imp_id_regs();
+
+	bool valid =
+		check_sysreg_table(sys_reg_descs, num_sys_reg_descs, false);
+
+	if (!valid)
+		return -EINVAL;
+
+	return 0;
+}
+
+static int finalize_sys_regs(struct kvm *kvm)
+{
+	lockdep_assert_held(&kvm->arch.config_lock);
+
+	if (kvm_id_regs_final(kvm))
+		return 0;
+
+	if (!irqchip_in_kernel(kvm)) {
+		u64 val;
+
+		val = kvm_read_vm_id_reg(kvm, SYS_ID_AA64PFR0_EL1) & ~ID_AA64PFR0_EL1_GIC;
+		kvm_set_vm_id_reg(kvm, SYS_ID_AA64PFR0_EL1, val);
+		val = kvm_read_vm_id_reg(kvm, SYS_ID_AA64PFR2_EL1) & ~ID_AA64PFR2_EL1_GCIE;
+		kvm_set_vm_id_reg(kvm, SYS_ID_AA64PFR2_EL1, val);
+		val = kvm_read_vm_id_reg(kvm, SYS_ID_PFR1_EL1) & ~ID_PFR1_EL1_GIC;
+		kvm_set_vm_id_reg(kvm, SYS_ID_PFR1_EL1, val);
+	}
+
+	set_bit(KVM_ARCH_FLAG_ID_REGS_FINAL, &kvm->arch.flags);
+
+	return 0;
+}
+
+int kvm_vcpu_finalize_sys_regs(struct kvm_vcpu *vcpu)
+{
+	struct kvm *kvm = vcpu->kvm;
+	
+	guard(mutex)(&kvm->arch.config_lock);
+	return finalize_sys_regs(kvm);
+}
diff --git a/arch/s390/kvm/arm64/trace.h b/arch/s390/kvm/arm64/trace.h
index ee232c4f5a6d..2f0ac8aa7a06 100644
--- a/arch/s390/kvm/arm64/trace.h
+++ b/arch/s390/kvm/arm64/trace.h
@@ -4,6 +4,8 @@
 
 #include <linux/tracepoint.h>
 
+#include <arm64/sys_regs.h>
+
 #undef TRACE_SYSTEM
 #define TRACE_SYSTEM kvm_arm64
 #undef TRACE_SYSTEM_VAR
@@ -90,6 +92,54 @@ TRACE_EVENT(kvm_guest_fault,
 		  __entry->hxfar, __entry->vcpu_pc)
 );
 
+TRACE_EVENT(kvm_handle_sys_reg,
+	TP_PROTO(unsigned long hsr),
+	TP_ARGS(hsr),
+
+	TP_STRUCT__entry(
+		__field(unsigned long,	hsr)
+	),
+
+	TP_fast_assign(
+		__entry->hsr = hsr;
+	),
+
+	TP_printk("HSR 0x%08lx", __entry->hsr)
+);
+
+TRACE_EVENT(kvm_sys_access,
+	TP_PROTO(unsigned long vcpu_pc, struct sys_reg_params *params, const struct sys_reg_desc *reg),
+	TP_ARGS(vcpu_pc, params, reg),
+
+	TP_STRUCT__entry(
+		__field(unsigned long,			vcpu_pc)
+		__field(bool,				is_write)
+		__field(const char *,			name)
+		__field(u8,				Op0)
+		__field(u8,				Op1)
+		__field(u8,				CRn)
+		__field(u8,				CRm)
+		__field(u8,				Op2)
+	),
+
+	TP_fast_assign(
+		__entry->vcpu_pc = vcpu_pc;
+		__entry->is_write = params->is_write;
+		__entry->name = reg->name;
+		__entry->Op0 = reg->Op0;
+		__entry->Op1 = reg->Op1;
+		__entry->CRn = reg->CRn;
+		__entry->CRm = reg->CRm;
+		__entry->Op2 = reg->Op2;
+	),
+
+	TP_printk("PC: %lx %s (%d,%d,%d,%d,%d) %s",
+		  __entry->vcpu_pc, __entry->name ?: "UNKN",
+		  __entry->Op0, __entry->Op1, __entry->CRn,
+		  __entry->CRm, __entry->Op2,
+		  str_write_read(__entry->is_write))
+);
+
 #endif /* KVM_ARM64_TRACE_KVM_H */
 
 #undef TRACE_INCLUDE_PATH
diff --git a/arch/s390/tools/Makefile.arm64h b/arch/s390/tools/Makefile.arm64h
index 88dcce26179c..9761b0bfcfbd 100644
--- a/arch/s390/tools/Makefile.arm64h
+++ b/arch/s390/tools/Makefile.arm64h
@@ -17,12 +17,16 @@ arm64uapi := $(objtree)/arch/$(ARCH)/include/generated/uapi/arm64
 # If the file does not contain any markers the whole file will be copied.
 ARM64_SHARED_HEADERS := \
 	brk-imm.h \
+	cache.h \
+	cputype.h \
 	esr.h \
 	kvm_arm.h \
 	kvm_emulate-gen.h \
 	kvm_host.h \
 	ptrace-gen.h \
 	sysreg-gen.h \
+	spectre.h \
+	kvm:sys_regs-gen.h \
 	uapi:hwcap.h \
 	uapi:kvm.h \
 	uapi:ptrace.h \

-- 
2.56.0

[PATCH v3 17/23] KVM: s390: arm64: Integrate sysreg into the host

[Steffen Eiden]

Integrate the previously established system register managing
infrastructure into the host. Initialize sysregs during vcpu init.
Finalize sysregs on PID change, Initialize sysreg table on kvm init,
report availability to userspace and let userspace read&modify guest system
register.

Co-developed-by: Andreas Grapentin <gra@linux.ibm.com>
Signed-off-by: Andreas Grapentin <gra@linux.ibm.com>
Signed-off-by: Steffen Eiden <seiden@linux.ibm.com>
---
 arch/s390/include/asm/kvm_host_arm64.h |  3 +++
 arch/s390/kvm/arm64/arm.c              | 29 +++++++++++++++++++++++
 arch/s390/kvm/arm64/guest.c            | 12 ++++++----
 arch/s390/kvm/arm64/handle_exit.c      |  1 +
 arch/s390/kvm/arm64/reset.c            | 42 ++++++++++++++++++++++++++++++++++
 arch/s390/kvm/arm64/sys_regs.c         |  1 +
 6 files changed, 84 insertions(+), 4 deletions(-)

diff --git a/arch/s390/include/asm/kvm_host_arm64.h b/arch/s390/include/asm/kvm_host_arm64.h
index 35b6d8972359..44deae423ace 100644
--- a/arch/s390/include/asm/kvm_host_arm64.h
+++ b/arch/s390/include/asm/kvm_host_arm64.h
@@ -157,6 +157,9 @@ struct kvm_arch {
 
 	/* Per-VM ID register storage */
 	struct kvm_vm_id_regs id_regs;
+
+	/* The EPD as it should be on all VCPUs. */
+	u64 epd;
 };
 
 static inline bool __vcpu_has_feature(const struct kvm_arch *ka, int feature)
diff --git a/arch/s390/kvm/arm64/arm.c b/arch/s390/kvm/arm64/arm.c
index ba6235629c7d..62fd71af8968 100644
--- a/arch/s390/kvm/arm64/arm.c
+++ b/arch/s390/kvm/arm64/arm.c
@@ -15,6 +15,7 @@
 
 #include <arm64/kvm_emulate.h>
 #include <arm64/sysreg.h>
+#include <arm64/sys_regs.h>
 
 #include <gmap.h>
 #include <kvm_mmu.h>
@@ -50,6 +51,7 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext)
 		break;
 	case KVM_CAP_IOEVENTFD:
 	case KVM_CAP_PRE_FAULT_MEMORY:
+	case KVM_CAP_ONE_REG:
 		ret = 1;
 		break;
 	default:
@@ -136,6 +138,8 @@ int kvm_arch_init_vm(struct kvm *kvm, unsigned long type)
 		goto out_err;
 	kvm->arch.mem_limit = kvm->arch.guest_phys_size;
 
+	kvm->arch.epd = ptff_qagto(0);
+
 	VM_EVENT(kvm, 3, "vm created with type %lu", type);
 	return 0;
 
@@ -193,6 +197,7 @@ int kvm_arch_vcpu_create(struct kvm_vcpu *vcpu)
 	sae_block->mso = 0L;
 	sae_block->msl = vcpu->kvm->arch.mem_limit - 1;
 	sae_block->save_area = virt_to_phys(save_area);
+	sae_block->gpto = vcpu->kvm->arch.epd;
 
 	VM_EVENT(vcpu->kvm, 3,
 		 "create cpu %d at 0x%p, sae block at 0x%p, satellite at 0x%p",
@@ -207,6 +212,7 @@ void kvm_arch_vcpu_postcreate(struct kvm_vcpu *vcpu)
 
 void kvm_arch_vcpu_destroy(struct kvm_vcpu *vcpu)
 {
+	kfree(vcpu->arch.ccsidr);
 	kvm_s390_free_mmu_cache(vcpu->arch.mc);
 
 	VCPU_EVENT(vcpu, 3, "%s", "free cpu");
@@ -218,6 +224,8 @@ void kvm_arch_vcpu_load(struct kvm_vcpu *vcpu, int cpu)
 	vcpu->cpu = cpu;
 
 	lasrm(&vcpu->arch.save_area);
+
+	vcpu->arch.sae_block.gpto = vcpu->kvm->arch.epd;
 }
 
 void kvm_arch_vcpu_put(struct kvm_vcpu *vcpu)
@@ -261,6 +269,7 @@ bool kvm_arch_vcpu_in_kernel(struct kvm_vcpu *vcpu)
 int kvm_arch_vcpu_run_pid_change(struct kvm_vcpu *vcpu)
 {
 	struct kvm *kvm = vcpu->kvm;
+	int ret;
 
 	if (!kvm_vcpu_initialized(vcpu))
 		return -ENOEXEC;
@@ -271,6 +280,10 @@ int kvm_arch_vcpu_run_pid_change(struct kvm_vcpu *vcpu)
 	if (likely(READ_ONCE(vcpu->pid)))
 		return 0;
 
+	ret = kvm_vcpu_finalize_sys_regs(vcpu);
+	if (ret)
+		return ret;
+
 	scoped_guard(mutex, &kvm->arch.config_lock)
 		set_bit(KVM_ARCH_FLAG_HAS_RAN_ONCE, &kvm->arch.flags);
 
@@ -477,9 +490,17 @@ int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu *vcpu)
 		trace_kvm_entry(*vcpu_pc(vcpu));
 
 		adjust_pc(vcpu);
+		vcpu_write_host_sys_reg(vcpu, vcpu->arch.hcr_el2, SYS_HCR_EL2);
+		vcpu_write_host_sys_reg(vcpu, vcpu->arch.hcrx_el2, SYS_HCRX_EL2);
+		vcpu_write_host_sys_reg(vcpu, vcpu->arch.mpidr, SYS_VMPIDR_EL2);
+		_vcpu_write_sys_reg(vcpu, vcpu->arch.ctxt.elr_el1, SYS_ELR_EL1);
+		_vcpu_write_sys_reg(vcpu, vcpu->arch.ctxt.spsr_el1, SYS_SPSR_EL1);
 
 		arm_vcpu_enter_exit(virt_to_phys(&vcpu->arch.sae_block));
 
+		vcpu->arch.ctxt.elr_el1 = _vcpu_read_sys_reg(vcpu, SYS_ELR_EL1);
+		vcpu->arch.ctxt.spsr_el1 = _vcpu_read_sys_reg(vcpu, SYS_SPSR_EL1);
+
 		vcpu->mode = OUTSIDE_GUEST_MODE;
 		vcpu->stat.exits++;
 
@@ -762,11 +783,19 @@ static bool __init machine_supports_arm64_kvm(void)
 
 static int __init kvm_s390_arm64_init(void)
 {
+	int ret;
+
 	if (!machine_supports_arm64_kvm())
 		return -ENODEV;
 
 	kvm_init_qaaf();
 
+	ret = kvm_sys_reg_table_init();
+	if (ret) {
+		kvm_info("Error initializing system register tables\n");
+		return ret;
+	}
+
 	return kvm_init(sizeof(struct kvm_vcpu), 0, THIS_MODULE);
 }
 
diff --git a/arch/s390/kvm/arm64/guest.c b/arch/s390/kvm/arm64/guest.c
index 8b5ee9e79a4f..047460975936 100644
--- a/arch/s390/kvm/arm64/guest.c
+++ b/arch/s390/kvm/arm64/guest.c
@@ -4,6 +4,7 @@
 
 #include <arm64/kvm_emulate.h>
 #include <arm64/kvm_nested.h>
+#include <arm64/sys_regs.h>
 
 #define SVE_VQ_MIN	__SVE_VQ_MIN
 #define SVE_NUM_ZREGS	KVM_ARM64_SVE_NUM_ZREGS
@@ -60,12 +61,15 @@ int kvm_arm_copy_reg_indices(struct kvm_vcpu *vcpu, u64 __user *uindices)
 		return ret;
 	uindices += ret;
 
-	return 0;
+	return kvm_arm_copy_sys_reg_indices(vcpu, uindices);
 }
 
 unsigned long kvm_arm_num_regs(struct kvm_vcpu *vcpu)
 {
-	return num_core_regs(vcpu);
+	unsigned long num = num_core_regs(vcpu);
+
+	num += kvm_arm_num_sys_reg_descs(vcpu);
+	return num;
 }
 
 int kvm_arm_get_reg(struct kvm_vcpu *vcpu, const struct kvm_one_reg *reg)
@@ -78,7 +82,7 @@ int kvm_arm_get_reg(struct kvm_vcpu *vcpu, const struct kvm_one_reg *reg)
 	case KVM_REG_ARM_CORE:
 		return get_core_reg(vcpu, reg);
 	default:
-		return -EINVAL;
+		return kvm_arm_sys_reg_get_reg(vcpu, reg);
 	}
 }
 
@@ -92,7 +96,7 @@ int kvm_arm_set_reg(struct kvm_vcpu *vcpu, const struct kvm_one_reg *reg)
 	case KVM_REG_ARM_CORE:
 		return set_core_reg(vcpu, reg);
 	default:
-		return -EINVAL;
+		return kvm_arm_sys_reg_set_reg(vcpu, reg);
 	}
 }
 
diff --git a/arch/s390/kvm/arm64/handle_exit.c b/arch/s390/kvm/arm64/handle_exit.c
index e0c7773c2165..1653163ed6d6 100644
--- a/arch/s390/kvm/arm64/handle_exit.c
+++ b/arch/s390/kvm/arm64/handle_exit.c
@@ -62,6 +62,7 @@ static int kvm_handle_unknown_ec(struct kvm_vcpu *vcpu)
 static exit_handle_fn arm_exit_handlers[] = {
 	[0 ... ESR_ELx_EC_MAX]	= kvm_handle_unknown_ec,
 	[ESR_ELx_EC_HVC64]	= handle_hvc,
+	[ESR_ELx_EC_SYS64]	= kvm_handle_sys_reg,
 	[ESR_ELx_EC_IABT_LOW]	= kvm_handle_guest_abort,
 	[ESR_ELx_EC_DABT_LOW]	= kvm_handle_guest_abort,
 };
diff --git a/arch/s390/kvm/arm64/reset.c b/arch/s390/kvm/arm64/reset.c
index e4cdce1dbf9d..0339901f5fd0 100644
--- a/arch/s390/kvm/arm64/reset.c
+++ b/arch/s390/kvm/arm64/reset.c
@@ -5,6 +5,10 @@
 
 #include <arm64/kvm_emulate.h>
 #include <arm64/kvm_nested.h>
+#include <arm64/sys_regs.h>
+#include <arm64/sysreg.h>
+
+#include <clocksource/arm_arch_timer.h>
 
 bool kvm_arm_vcpu_is_finalized(struct kvm_vcpu *vcpu)
 {
@@ -21,6 +25,39 @@ static inline void vcpu_reset_hcr(struct kvm_vcpu *vcpu)
 			      HCR_EL2_TIDCP | HCR_EL2_TLOR;
 }
 
+static inline void vcpu_reset_hcrx(struct kvm_vcpu *vcpu)
+{
+	vcpu->arch.hcrx_el2 = 0;
+	if (kvm_has_feat(vcpu->kvm, ID_AA64ISAR2_EL1, MOPS, IMP))
+		vcpu->arch.hcrx_el2 |= HCRX_EL2_MSCEn;
+}
+
+static inline void vcpu_reset_cptr(struct kvm_vcpu *vcpu)
+{
+	u64 cptr;
+
+	/* we unconditionally have E2H, so disable traps to EL2 for FP and SVE, if enabled */
+	cptr = CPACR_EL1_FPEN;
+	if (vcpu_has_sve(vcpu))
+		cptr |= CPACR_EL1_ZEN;
+	vcpu_write_host_sys_reg(vcpu, cptr, SYS_CPTR_EL2);
+}
+
+static inline void vcpu_reset_cnthctl(struct kvm_vcpu *vcpu)
+{
+	u64 cnthctl;
+
+	/* we unconditionally have E2H, so disable traps to EL2 for physical timer registers */
+	cnthctl = (CNTHCTL_EL1PCEN | CNTHCTL_EL1PCTEN) << 10;
+	vcpu_write_host_sys_reg(vcpu, cnthctl, SYS_CNTHCTL_EL2);
+}
+
+static inline void vcpu_reset_icc(struct kvm_vcpu *vcpu)
+{
+	/* ensure ICC_SRE_EL2.Enable = 1 so ICC_SRE_EL1 is handled in hardware */
+	vcpu_write_host_sys_reg(vcpu, 0xf, SYS_ICC_SRE_EL2);
+}
+
 void kvm_reset_vcpu(struct kvm_vcpu *vcpu)
 {
 	struct vcpu_reset_state reset_state;
@@ -43,9 +80,14 @@ void kvm_reset_vcpu(struct kvm_vcpu *vcpu)
 		vcpu_put(vcpu);
 
 	kvm_reset_vcpu_core(vcpu);
+	kvm_reset_sys_regs(vcpu);
 
 	/* Reset special registers */
 	vcpu_reset_hcr(vcpu);
+	vcpu_reset_hcrx(vcpu);
+	vcpu_reset_cptr(vcpu);
+	vcpu_reset_cnthctl(vcpu);
+	vcpu_reset_icc(vcpu);
 
 	if (reset_state.reset) {
 		*vcpu_pc(vcpu) = reset_state.pc;
diff --git a/arch/s390/kvm/arm64/sys_regs.c b/arch/s390/kvm/arm64/sys_regs.c
index b1d4335779c7..417a05ce9015 100644
--- a/arch/s390/kvm/arm64/sys_regs.c
+++ b/arch/s390/kvm/arm64/sys_regs.c
@@ -441,6 +441,7 @@ static int arch_timer_set_user(struct kvm_vcpu *vcpu,
 	case SYS_CNTPCT_EL0:
 		gpto = ptff_qagto(val);
 		vcpu->arch.sae_block.gpto = gpto;
+		vcpu->kvm->arch.epd = gpto;
 		return 0;
 	}
 

-- 
2.56.0

[PATCH v3 18/23] KVM: s390: arm64: Use QAAF init save area

[Steffen Eiden]

Use the (cached) result of QAAF fc2 (Generate Initial
System-Registers-Save Area) to initialize the Save Area.

Co-developed-by: Andreas Grapentin <gra@linux.ibm.com>
Signed-off-by: Andreas Grapentin <gra@linux.ibm.com>
Signed-off-by: Steffen Eiden <seiden@linux.ibm.com>
---
 arch/s390/kvm/arm64/arm.c   | 1 -
 arch/s390/kvm/arm64/reset.c | 5 +++++
 2 files changed, 5 insertions(+), 1 deletion(-)

diff --git a/arch/s390/kvm/arm64/arm.c b/arch/s390/kvm/arm64/arm.c
index 62fd71af8968..4de0f17052cb 100644
--- a/arch/s390/kvm/arm64/arm.c
+++ b/arch/s390/kvm/arm64/arm.c
@@ -371,7 +371,6 @@ static int kvm_arch_vcpu_ioctl_vcpu_init(struct kvm_vcpu *vcpu,
 	int ret;
 
 	sae_block->save_area = virt_to_phys(save_area);
-	save_area->sdo = virt_to_phys(sae_block);
 
 	ret = kvm_vcpu_set_target(vcpu, init);
 	if (ret)
diff --git a/arch/s390/kvm/arm64/reset.c b/arch/s390/kvm/arm64/reset.c
index 0339901f5fd0..dc00bf398b15 100644
--- a/arch/s390/kvm/arm64/reset.c
+++ b/arch/s390/kvm/arm64/reset.c
@@ -3,6 +3,8 @@
 #include <linux/kvm_host.h>
 #include <linux/fpu.h>
 
+#include <asm/aef.h>
+
 #include <arm64/kvm_emulate.h>
 #include <arm64/kvm_nested.h>
 #include <arm64/sys_regs.h>
@@ -79,6 +81,9 @@ void kvm_reset_vcpu(struct kvm_vcpu *vcpu)
 	if (loaded)
 		vcpu_put(vcpu);
 
+	memcpy(&vcpu->arch.save_area, aef_gisrsa(), sizeof(vcpu->arch.save_area));
+	vcpu->arch.save_area.sdo = virt_to_phys(&vcpu->arch.sae_block);
+
 	kvm_reset_vcpu_core(vcpu);
 	kvm_reset_sys_regs(vcpu);
 

-- 
2.56.0

[PATCH v3 19/23] KVM: s390: arm64: Implement exception injection

[Steffen Eiden]

Add exception injection support for s390 KVM arm64 architecture. This
implements the core exception entry mechanism and fault injection
functions needed for proper guest exception handling.

Co-developed-by: Andreas Grapentin <gra@linux.ibm.com>
Signed-off-by: Andreas Grapentin <gra@linux.ibm.com>
Signed-off-by: Steffen Eiden <seiden@linux.ibm.com>
---
 arch/s390/include/asm/kvm_host_arm64.h |   2 +
 arch/s390/kvm/arm64/Makefile           |   1 +
 arch/s390/kvm/arm64/arm.c              |  15 ++---
 arch/s390/kvm/arm64/exception.c        | 109 +++++++++++++++++++++++++++++++++
 arch/s390/kvm/arm64/inject_fault.c     |  65 +++++++++++++++++++-
 5 files changed, 179 insertions(+), 13 deletions(-)

diff --git a/arch/s390/include/asm/kvm_host_arm64.h b/arch/s390/include/asm/kvm_host_arm64.h
index 44deae423ace..79549434a1d6 100644
--- a/arch/s390/include/asm/kvm_host_arm64.h
+++ b/arch/s390/include/asm/kvm_host_arm64.h
@@ -425,4 +425,6 @@ static inline bool vgic_host_has_gicv5(void)
 	return false;
 }
 
+void kvm_adjust_pc(struct kvm_vcpu *vcpu);
+
 #endif /* ASM_KVM_HOST_ARM64_H */
diff --git a/arch/s390/kvm/arm64/Makefile b/arch/s390/kvm/arm64/Makefile
index 24ac6f6f6278..b0f2da6ac482 100644
--- a/arch/s390/kvm/arm64/Makefile
+++ b/arch/s390/kvm/arm64/Makefile
@@ -11,6 +11,7 @@ ccflags-y += -I $(src) -I $(objtree)/$(obj) -I$(srctree)/arch/$(SRCARCH)/kvm/gma
 
 kvm-arm64-local := \
 	arm.o \
+	exception.o \
 	feature.o \
 	guest.o \
 	handle_exit.o \
diff --git a/arch/s390/kvm/arm64/arm.c b/arch/s390/kvm/arm64/arm.c
index 4de0f17052cb..91e9f4a75ce7 100644
--- a/arch/s390/kvm/arm64/arm.c
+++ b/arch/s390/kvm/arm64/arm.c
@@ -390,14 +390,6 @@ int kvm_vm_ioctl_irq_line(struct kvm *kvm, struct kvm_irq_level *irq_level,
 	return -EINVAL;
 }
 
-static void adjust_pc(struct kvm_vcpu *vcpu)
-{
-	if (vcpu_get_flag(vcpu, INCREMENT_PC)) {
-		kvm_skip_instr(vcpu);
-		vcpu_clear_flag(vcpu, INCREMENT_PC);
-	}
-}
-
 static noinstr void arm_vcpu_enter_exit(phys_addr_t sae_block)
 {
 	guest_timing_enter_irqoff();
@@ -488,7 +480,7 @@ int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu *vcpu)
 
 		trace_kvm_entry(*vcpu_pc(vcpu));
 
-		adjust_pc(vcpu);
+		kvm_adjust_pc(vcpu);
 		vcpu_write_host_sys_reg(vcpu, vcpu->arch.hcr_el2, SYS_HCR_EL2);
 		vcpu_write_host_sys_reg(vcpu, vcpu->arch.hcrx_el2, SYS_HCRX_EL2);
 		vcpu_write_host_sys_reg(vcpu, vcpu->arch.mpidr, SYS_VMPIDR_EL2);
@@ -511,8 +503,9 @@ int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu *vcpu)
 
 	kvm_sigset_deactivate(vcpu);
 out:
-	if (unlikely(vcpu_get_flag(vcpu, INCREMENT_PC)))
-		adjust_pc(vcpu);
+	if (unlikely(vcpu_get_flag(vcpu, PENDING_EXCEPTION) ||
+		     vcpu_get_flag(vcpu, INCREMENT_PC)))
+		kvm_adjust_pc(vcpu);
 
 	save_vx_regs(vcpu->arch.ctxt.vregs);
 	kernel_fpu_end(&fpu_save, KERNEL_FPC | KERNEL_VXR);
diff --git a/arch/s390/kvm/arm64/exception.c b/arch/s390/kvm/arm64/exception.c
new file mode 100644
index 000000000000..93a2a2a8bcc8
--- /dev/null
+++ b/arch/s390/kvm/arm64/exception.c
@@ -0,0 +1,109 @@
+// SPDX-License-Identifier: GPL-2.0-only
+
+#include <linux/kvm_host.h>
+#include <arm64/kvm_emulate.h>
+#include <arm64/sysreg-defs.h>
+
+#include <arm64/ptrace.h>
+
+/*
+ * This performs the exception entry at a given EL (@target_mode), stashing PC
+ * and PSTATE into ELR and SPSR respectively, and compute the new PC/PSTATE.
+ * The EL passed to this function *must* be a non-secure, privileged mode with
+ * bit 0 being set (PSTATE.SP == 1).
+ *
+ * When an exception is taken, most PSTATE fields are left unchanged in the
+ * handler. However, some are explicitly overridden (e.g. M[4:0]). Luckily all
+ * of the inherited bits have the same position in the AArch64/AArch32 SPSR_ELx
+ * layouts, so we don't need to shuffle these for exceptions from AArch32 EL0.
+ *
+ * For the SPSR_ELx layout for AArch64, see ARM DDI 0487E.a page C5-429.
+ * For the SPSR_ELx layout for AArch32, see ARM DDI 0487E.a page C5-426.
+ *
+ * Here we manipulate the fields in order of the AArch64 SPSR_ELx layout, from
+ * MSB to LSB.
+ */
+static void enter_exception64(struct kvm_vcpu *vcpu, unsigned long target_mode,
+			      enum exception_type type)
+{
+	unsigned long sctlr, vbar, old_pstate, new, mode;
+	u64 exc_offset;
+
+	old_pstate = *vcpu_cpsr(vcpu);
+	mode = old_pstate & (PSR_MODE_MASK | PSR_MODE32_BIT);
+
+	if      (mode == target_mode)
+		exc_offset = CURRENT_EL_SP_ELx_VECTOR;
+	else if ((mode | PSR_MODE_THREAD_BIT) == target_mode)
+		exc_offset = CURRENT_EL_SP_EL0_VECTOR;
+	else if (!(mode & PSR_MODE32_BIT))
+		exc_offset = LOWER_EL_AArch64_VECTOR;
+	else
+		exc_offset = LOWER_EL_AArch32_VECTOR;
+
+	switch (target_mode) {
+	case PSR_MODE_EL1h:
+		vbar = vcpu_read_sys_reg(vcpu, VBAR_EL1);
+		sctlr = vcpu_read_sys_reg(vcpu, SCTLR_EL1);
+		vcpu_write_sys_reg(vcpu, *vcpu_pc(vcpu), ELR_EL1);
+		vcpu_write_sys_reg(vcpu, old_pstate, SPSR_EL1);
+		break;
+	default:
+		panic("Unknown PSR Mode: 0x%lx.", target_mode);
+	}
+
+	*vcpu_pc(vcpu) = vbar + exc_offset + type;
+
+	new = 0;
+
+	new |= (old_pstate & PSR_N_BIT);
+	new |= (old_pstate & PSR_Z_BIT);
+	new |= (old_pstate & PSR_C_BIT);
+	new |= (old_pstate & PSR_V_BIT);
+	new |= (old_pstate & PSR_DIT_BIT);
+	new |= (old_pstate & PSR_PAN_BIT);
+
+	if (!(sctlr & SCTLR_EL1_SPAN))
+		new |= PSR_PAN_BIT;
+
+	if (sctlr & SCTLR_ELx_DSSBS)
+		new |= PSR_SSBS_BIT;
+
+	new |= PSR_D_BIT;
+	new |= PSR_A_BIT;
+	new |= PSR_I_BIT;
+	new |= PSR_F_BIT;
+
+	new |= target_mode;
+
+	*vcpu_cpsr(vcpu) = new;
+}
+
+static void kvm_inject_exception(struct kvm_vcpu *vcpu)
+{
+	switch (vcpu_get_flag(vcpu, EXCEPT_MASK)) {
+	case unpack_vcpu_flag(EXCEPT_AA64_EL1_SYNC):
+		enter_exception64(vcpu, PSR_MODE_EL1h, except_type_sync);
+		break;
+	default:
+		WARN_ONCE(1, "Unsupported pending exception type: 0x%lx\n",
+			  vcpu_get_flag(vcpu, EXCEPT_MASK));
+		break;
+	}
+}
+
+/*
+ * Adjust the guest PC (and potentially exception state) depending on
+ * flags provided by the emulation code.
+ */
+void kvm_adjust_pc(struct kvm_vcpu *vcpu)
+{
+	if (vcpu_get_flag(vcpu, PENDING_EXCEPTION)) {
+		kvm_inject_exception(vcpu);
+		vcpu_clear_flag(vcpu, PENDING_EXCEPTION);
+		vcpu_clear_flag(vcpu, EXCEPT_MASK);
+	} else if (vcpu_get_flag(vcpu, INCREMENT_PC)) {
+		kvm_skip_instr(vcpu);
+		vcpu_clear_flag(vcpu, INCREMENT_PC);
+	}
+}
diff --git a/arch/s390/kvm/arm64/inject_fault.c b/arch/s390/kvm/arm64/inject_fault.c
index 425dbeaa421c..33f40c5188c9 100644
--- a/arch/s390/kvm/arm64/inject_fault.c
+++ b/arch/s390/kvm/arm64/inject_fault.c
@@ -2,6 +2,38 @@
 
 #include <arm64/kvm_emulate.h>
 
+static void pend_sync_exception(struct kvm_vcpu *vcpu)
+{
+	kvm_pend_exception(vcpu, EXCEPT_AA64_EL1_SYNC);
+}
+
+static void inject_abt64(struct kvm_vcpu *vcpu, bool is_iabt, unsigned long addr)
+{
+	unsigned long cpsr = *vcpu_cpsr(vcpu);
+	bool is_aarch32 = vcpu_mode_is_32bit(vcpu);
+	u64 esr = 0;
+
+	if (kvm_vcpu_abt_iss1tw(vcpu))
+		esr |= ESR_ELx_FSC_SEA_TTW(kvm_vcpu_abt_gltl(vcpu));
+	else
+		esr |= ESR_ELx_FSC_EXTABT;
+
+	pend_sync_exception(vcpu);
+
+	esr |= ESR_ELx_IL;
+
+	if (is_aarch32 || (cpsr & PSR_MODE_MASK) == PSR_MODE_EL0t)
+		esr |= (ESR_ELx_EC_IABT_LOW << ESR_ELx_EC_SHIFT);
+	else
+		esr |= (ESR_ELx_EC_IABT_CUR << ESR_ELx_EC_SHIFT);
+
+	if (!is_iabt)
+		esr |= ESR_ELx_EC_DABT_LOW << ESR_ELx_EC_SHIFT;
+
+	vcpu_write_sys_reg(vcpu, addr, FAR_EL1);
+	vcpu_write_sys_reg(vcpu, esr, ESR_EL1);
+}
+
 /**
  * kvm_inject_undefined - inject an undefined instruction into the guest
  * @vcpu: The vCPU in which to inject the exception
@@ -11,11 +43,40 @@
  */
 void kvm_inject_undefined(struct kvm_vcpu *vcpu)
 {
-	/* Stub until s390 supports arm64 sysregs TODO sysregs*/
+	u64 esr = (ESR_ELx_EC_UNKNOWN << ESR_ELx_EC_SHIFT) | ESR_ELx_IL;
+
+	kvm_inject_sync(vcpu, esr);
+}
+
+static void __kvm_inject_sea(struct kvm_vcpu *vcpu, bool iabt, u64 addr)
+{
+	inject_abt64(vcpu, iabt, addr);
 }
 
 int kvm_inject_sea(struct kvm_vcpu *vcpu, bool iabt, u64 addr)
 {
-	/* Stub until s390 supports arm64 sysregs TODO sysregs*/
+	lockdep_assert_held(&vcpu->mutex);
+
+	__kvm_inject_sea(vcpu, iabt, addr);
 	return 1;
 }
+
+void kvm_inject_size_fault(struct kvm_vcpu *vcpu)
+{
+	unsigned long addr, esr;
+
+	addr  = kvm_vcpu_get_fault_ipa(vcpu);
+	addr |= FAR_TO_FIPA_OFFSET(kvm_vcpu_get_hfar(vcpu));
+
+	__kvm_inject_sea(vcpu, kvm_vcpu_trap_is_iabt(vcpu), addr);
+
+	esr = vcpu_read_sys_reg(vcpu, ESR_EL1);
+	esr &= ~GENMASK_ULL(5, 0);
+	vcpu_write_sys_reg(vcpu, esr, ESR_EL1);
+}
+
+void kvm_inject_sync(struct kvm_vcpu *vcpu, u64 esr)
+{
+	pend_sync_exception(vcpu);
+	vcpu_write_sys_reg(vcpu, esr, ESR_EL1);
+}

-- 
2.56.0

[PATCH v3 20/23] KVM: s390: arm64: Finalize page fault handling

[Steffen Eiden]

Replace the temporary -ENOEXEC returns of the arm64-on-s390 page fault
handling with exception injection into the guest, mirroring arm64's
kvm_handle_guest_abort().

Translation faults beyond the physical address range  are injected as an
address size fault, while faults falling between the VM's IPA size and
that range are injected as a synchronous external abort. Instruction
aborts and stage-1 walk faults on host addresses that back no guest page
are injected as synchronous external aborts as well, and a
PGM-addressing from the fault-in path is injected as an address size
fault, the closest arm64 equivalent. Injected faults return 1 so the
vCPU re-enters the guest with the pending exception instead of exiting
to userspace.

Co-developed-by: Andreas Grapentin <gra@linux.ibm.com>
Signed-off-by: Andreas Grapentin <gra@linux.ibm.com>
Signed-off-by: Steffen Eiden <seiden@linux.ibm.com>
---
 arch/s390/include/asm/kvm_host_arm64.h |  1 +
 arch/s390/kvm/arm64/arm.c              |  6 +++
 arch/s390/kvm/arm64/mmu.c              | 74 +++++++++++++++++++++++-----------
 arch/s390/kvm/arm64/sys_regs.c         |  4 ++
 4 files changed, 62 insertions(+), 23 deletions(-)

diff --git a/arch/s390/include/asm/kvm_host_arm64.h b/arch/s390/include/asm/kvm_host_arm64.h
index 79549434a1d6..20f51cf9ce37 100644
--- a/arch/s390/include/asm/kvm_host_arm64.h
+++ b/arch/s390/include/asm/kvm_host_arm64.h
@@ -426,5 +426,6 @@ static inline bool vgic_host_has_gicv5(void)
 }
 
 void kvm_adjust_pc(struct kvm_vcpu *vcpu);
+size_t kvm_parange_to_address_sanitized(u32 id_parange);
 
 #endif /* ASM_KVM_HOST_ARM64_H */
diff --git a/arch/s390/kvm/arm64/arm.c b/arch/s390/kvm/arm64/arm.c
index 91e9f4a75ce7..b5a0f8c1e9a4 100644
--- a/arch/s390/kvm/arm64/arm.c
+++ b/arch/s390/kvm/arm64/arm.c
@@ -63,9 +63,15 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext)
 
 static u64 kvm_max_guest_address(void)
 {
+	u64 mmfr0 = read_sanitised_ftr_reg(SYS_ID_AA64MMFR0_EL1);
+	u64 id_pa_addr;
 	u64 max_addr;
 
+	id_pa_addr = kvm_parange_to_address_sanitized(
+		SYS_FIELD_GET(ID_AA64MMFR0_EL1, PARANGE, mmfr0));
+
 	max_addr = min_t(u64, TASK_SIZE_MAX, sclp.hamax);
+	max_addr = min_t(u64, max_addr, id_pa_addr);
 	max_addr = max_t(u64, max_addr, SZ_1G - 1);
 	return ALIGN_DOWN(max_addr + 1, SZ_1G) - 1;
 }
diff --git a/arch/s390/kvm/arm64/mmu.c b/arch/s390/kvm/arm64/mmu.c
index 31e0d036896f..7a6f6ba326c1 100644
--- a/arch/s390/kvm/arm64/mmu.c
+++ b/arch/s390/kvm/arm64/mmu.c
@@ -16,7 +16,8 @@ static inline bool kvm_s390_cur_gmap_fault_is_write(struct kvm_vcpu *vcpu)
 /*
  * user_mem_abort() - handle a dat fault for the gmap of a vcpu
  *
- * Return: 0 on success, < 0 in case of error.
+ * Return: 0 if the fault was resolved, 1 if an exception was injected into
+ * the guest; the caller resumes the vCPU in both cases. < 0 in case of error.
  * Context: The mm lock must not be held before calling. May sleep.
  */
 static int user_mem_abort(struct kvm_vcpu *vcpu, gpa_t fault_ipa,
@@ -32,12 +33,14 @@ static int user_mem_abort(struct kvm_vcpu *vcpu, gpa_t fault_ipa,
 	ret = kvm_s390_faultin_gfn(vcpu, NULL, &f);
 	if (ret <= 0)
 		return ret;
-	if (ret == PGM_ADDRESSING)
+	if (ret == PGM_ADDRESSING) {
 		/*
-		 * Without the relevant sysregs we cannot do anything for now.
-		 * Go back to userspace with an error. TODO sysreg handling
+		 * There is no page with the requested address. Inject size fault
+		 * which is the closest arm match to PGM-addressing
 		 */
-		return -ENOEXEC;
+		kvm_inject_size_fault(vcpu);
+		return 1;
+	}
 	KVM_BUG_ON(ret, vcpu->kvm);
 	return -EINVAL;
 }
@@ -70,6 +73,35 @@ static int kvm_handle_pic(struct kvm_vcpu *vcpu, bool *translation)
 	return 0;
 }
 
+size_t kvm_parange_to_address_sanitized(u32 id_parange)
+{
+	static u32 ranges_map[8] = {
+		[ID_AA64MMFR0_EL1_PARANGE_32] = 32,
+		[ID_AA64MMFR0_EL1_PARANGE_36] = 36,
+		[ID_AA64MMFR0_EL1_PARANGE_40] = 40,
+		[ID_AA64MMFR0_EL1_PARANGE_42] = 42,
+		[ID_AA64MMFR0_EL1_PARANGE_44] = 44,
+		[ID_AA64MMFR0_EL1_PARANGE_48] = 48,
+		[ID_AA64MMFR0_EL1_PARANGE_52] = 52,
+		[ID_AA64MMFR0_EL1_PARANGE_56] = 56,
+	};
+	u32 parange;
+
+	/*
+	 * Future values must be higher than we know already.
+	 * See ARM DDI 0487C.a. Return a safe limit.
+	 */
+	parange = ranges_map[min(id_parange, ID_AA64MMFR0_EL1_PARANGE_56)];
+	return BIT_ULL(parange);
+}
+
+static size_t kvm_get_pa_address(struct kvm *kvm)
+{
+	u32 id_parange = get_idreg_field_enum(kvm, ID_AA64MMFR0_EL1, PARANGE);
+
+	return kvm_parange_to_address_sanitized(id_parange);
+}
+
 int kvm_handle_guest_abort(struct kvm_vcpu *vcpu)
 {
 	struct kvm_memory_slot *memslot;
@@ -98,16 +130,17 @@ int kvm_handle_guest_abort(struct kvm_vcpu *vcpu)
 	}
 
 	if (translation) {
-		/*
-		 * For both cases:
-		 * Without the relevant sysregs we cannot do anything for now.
-		 * Go back to userspace with an error. TODO sysreg handling
-		 */
-		if (fault_ipa >= BIT_ULL(get_kvm_ipa_limit()))
-			return -ENOEXEC;
+		/* Beyond the PA range programmed for the guest */
+		if (fault_ipa >= kvm_get_pa_address(vcpu->kvm)) {
+			kvm_inject_size_fault(vcpu);
+			return 1;
+		}
 
-		if (fault_ipa >= vcpu->kvm->arch.guest_phys_size)
-			return -ENOEXEC;
+		/* Falls between the IPA range and the PA range? */
+		if (fault_ipa >= vcpu->kvm->arch.guest_phys_size) {
+			fault_ipa |= FAR_TO_FIPA_OFFSET(kvm_vcpu_get_hfar(vcpu));
+			return kvm_inject_sea(vcpu, is_iabt, fault_ipa);
+		}
 	}
 
 	trace_kvm_guest_fault(*vcpu_pc(vcpu), kvm_vcpu_get_esr(vcpu),
@@ -121,23 +154,18 @@ int kvm_handle_guest_abort(struct kvm_vcpu *vcpu)
 	hva = gfn_to_hva_memslot_prot(memslot, gfn, &writable);
 	write_fault = kvm_s390_cur_gmap_fault_is_write(vcpu);
 	if (kvm_is_error_hva(hva) || (write_fault && !writable)) {
-		ret = -ENOEXEC;
 		/*
 		 * The guest has put either its instructions or its page-tables
 		 * somewhere it shouldn't have. Userspace won't be able to do
 		 * anything about this (there's no syndrome for a start).
-		 *
-		 * Without the relevant sysregs we cannot do anything for now.
-		 * Go back to userspace with an error. TODO sysreg handling
 		 */
-		if (is_iabt)
+		if (is_iabt) {
+			ret = kvm_inject_sea_iabt(vcpu, kvm_vcpu_get_hfar(vcpu));
 			goto out_unlock;
+		}
 
 		if (kvm_vcpu_abt_iss1tw(vcpu)) {
-			/*
-			 * Without the relevant sysregs we cannot do anything for now.
-			 * Go back to userspace with an error. TODO sysreg handling
-			 */
+			ret = kvm_inject_sea_dabt(vcpu, kvm_vcpu_get_hfar(vcpu));
 			goto out_unlock;
 		}
 
diff --git a/arch/s390/kvm/arm64/sys_regs.c b/arch/s390/kvm/arm64/sys_regs.c
index 417a05ce9015..951a02d89fc5 100644
--- a/arch/s390/kvm/arm64/sys_regs.c
+++ b/arch/s390/kvm/arm64/sys_regs.c
@@ -317,6 +317,10 @@ static int arm64_check_features(struct kvm_vcpu *vcpu,
 		if (SYS_FIELD_GET(ID_AA64MMFR0_EL1, PARANGE, val) >
 		    SYS_FIELD_GET(ID_AA64MMFR0_EL1, PARANGE, host_val))
 			return -E2BIG;
+		if (kvm_parange_to_address_sanitized(
+			    SYS_FIELD_GET(ID_AA64MMFR0_EL1, PARANGE, val)) <
+		    vcpu->kvm->arch.guest_phys_size)
+			return -E2BIG;
 		break;
 	case SYS_CTR_EL0:
 		/* forbid upgrading from VIPT to PIPT */

-- 
2.56.0

[PATCH v3 21/23] KVM: s390: arm64: Implement SVE for arm guests

[Steffen Eiden]

Add support for Scalable Vector Extension (SVE) in ARM64 guests running
on s390 hardware, providing 128-bit vector operations backed by s390's
Vector Extension (VX).

The implementation is limited to 128-bit vectors (VQ=1) matching s390 VX
register width. Z registers reuse existing FP/SIMD register space (vregs),
while P registers and FFR are stored in the SAE block.

Co-developed-by: Friedrich Welter <fritz@linux.ibm.com>
Signed-off-by: Friedrich Welter <fritz@linux.ibm.com>
Co-developed-by: Andreas Grapentin <gra@linux.ibm.com>
Signed-off-by: Andreas Grapentin <gra@linux.ibm.com>
Signed-off-by: Steffen Eiden <seiden@linux.ibm.com>
---
 arch/s390/include/asm/kvm_host_arm64.h |   5 +-
 arch/s390/kvm/arm64/arm.c              |   6 +
 arch/s390/kvm/arm64/feature.c          |   9 +-
 arch/s390/kvm/arm64/feature.h          |   6 +
 arch/s390/kvm/arm64/guest.c            | 250 +++++++++++++++++++++++++++++++++
 arch/s390/kvm/arm64/reset.c            |  24 ++++
 6 files changed, 297 insertions(+), 3 deletions(-)

diff --git a/arch/s390/include/asm/kvm_host_arm64.h b/arch/s390/include/asm/kvm_host_arm64.h
index 20f51cf9ce37..f096faceb19d 100644
--- a/arch/s390/include/asm/kvm_host_arm64.h
+++ b/arch/s390/include/asm/kvm_host_arm64.h
@@ -34,7 +34,9 @@ bool cpus_have_final_cap(unsigned int num);
 #define KVM_MAX_VCPUS 1
 #define __KVM_HAVE_VCPU_RUN_PID_CHANGE
 
-#define KVM_S390_ARM64_IMPL_FEATURES 0
+#define KVM_S390_ARM64_IMPL_FEATURES (		\
+	 BIT(KVM_ARM_VCPU_SVE)			\
+)
 #define KVM_HALT_POLL_NS_DEFAULT 50000
 
 #define KVM_S390_MANAGES_S390_GUEST 0
@@ -130,7 +132,6 @@ struct kvm_vcpu_stat {
 };
 
 #define kvm_has_mte(_kvm) false
-#define vcpu_has_sve(_vcpu) false
 #define vcpu_has_ptrauth(_vcpu) false
 
 #define kvm_has_fpmr(kvm) \
diff --git a/arch/s390/kvm/arm64/arm.c b/arch/s390/kvm/arm64/arm.c
index b5a0f8c1e9a4..8c2a07a96ee3 100644
--- a/arch/s390/kvm/arm64/arm.c
+++ b/arch/s390/kvm/arm64/arm.c
@@ -54,6 +54,9 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext)
 	case KVM_CAP_ONE_REG:
 		ret = 1;
 		break;
+	case KVM_CAP_ARM_SVE:
+		ret = system_supports_sve();
+		break;
 	default:
 		ret = 0;
 	}
@@ -264,6 +267,9 @@ static unsigned long system_supported_vcpu_features(struct kvm_vcpu *vcpu)
 {
 	unsigned long features = KVM_S390_ARM64_IMPL_FEATURES;
 
+	if (!system_supports_sve())
+		clear_bit(KVM_ARM_VCPU_SVE, &features);
+
 	return features;
 }
 
diff --git a/arch/s390/kvm/arm64/feature.c b/arch/s390/kvm/arm64/feature.c
index f8b23ba85f9e..0979e0cadc66 100644
--- a/arch/s390/kvm/arm64/feature.c
+++ b/arch/s390/kvm/arm64/feature.c
@@ -56,7 +56,6 @@ void __init kvm_arm_host_sanitize_features(struct qaaf_qmc_block *qaaf_qmc)
 	MASK_RESERVED(qaaf_qmc, ID_AA64MMFR4_EL1);
 
 	MODIFY(qaaf_qmc, ID_AA64PFR0_EL1, SEL2, NI);
-	MODIFY(qaaf_qmc, ID_AA64PFR0_EL1, SVE, NI);
 	MASK_RESERVED(qaaf_qmc, ID_AA64PFR0_EL1);
 	MASK_RESERVED(qaaf_qmc, ID_AA64PFR1_EL1);
 	MASK_RESERVED(qaaf_qmc, ID_AA64PFR2_EL1);
@@ -161,3 +160,11 @@ bool cpus_have_final_cap(unsigned int num)
 		return false;
 	}
 }
+
+bool system_supports_sve(void)
+{
+	return cpu_has_vx() &&
+	       SYS_FIELD_GET(ID_AA64PFR0_EL1, SVE,
+			     read_sanitised_ftr_reg(SYS_ID_AA64PFR0_EL1)) ==
+		       ID_AA64PFR0_EL1_SVE_IMP;
+}
diff --git a/arch/s390/kvm/arm64/feature.h b/arch/s390/kvm/arm64/feature.h
index 0a59047fda6e..9af6173f6b11 100644
--- a/arch/s390/kvm/arm64/feature.h
+++ b/arch/s390/kvm/arm64/feature.h
@@ -83,4 +83,10 @@ static inline u64 kvm_sae_irptc(void)
 
 #define kvm_vcpu_has_pmu(_v) false
 
+#define vcpu_has_sve(_vcpu)							\
+	(system_supports_sve() &&						\
+	 test_bit(KVM_ARCH_FLAG_GUEST_HAS_SVE, &((_vcpu)->kvm)->arch.flags))
+
+bool system_supports_sve(void);
+
 #endif /* ARCH_S390_KVM_FEATURE_H */
diff --git a/arch/s390/kvm/arm64/guest.c b/arch/s390/kvm/arm64/guest.c
index 047460975936..9e9d0034ddfc 100644
--- a/arch/s390/kvm/arm64/guest.c
+++ b/arch/s390/kvm/arm64/guest.c
@@ -5,6 +5,9 @@
 #include <arm64/kvm_emulate.h>
 #include <arm64/kvm_nested.h>
 #include <arm64/sys_regs.h>
+#include <arm64/sve_context.h>
+
+#include "feature.h"
 
 #define SVE_VQ_MIN	__SVE_VQ_MIN
 #define SVE_NUM_ZREGS	KVM_ARM64_SVE_NUM_ZREGS
@@ -52,6 +55,243 @@ const struct kvm_stats_header kvm_vcpu_stats_header = {
 		       sizeof(kvm_vcpu_stats_desc),
 };
 
+static int sve_zreg_index(__u64 id, unsigned int *regnum)
+{
+	/* Currently only one slice is supported on arm, and our zreg is only 128 bit */
+	const u64 zreg_id_max = KVM_REG_ARM64_SVE_ZREG(KVM_ARM64_SVE_NUM_ZREGS - 1,
+						       KVM_ARM64_SVE_MAX_SLICES - 1);
+	const u64 zreg_id_min = KVM_REG_ARM64_SVE_ZREG(0, 0);
+
+	if (id < zreg_id_min || id > zreg_id_max)
+		return -EINVAL;
+	if ((id & SVE_REG_SLICE_MASK) > 0)
+		return -ENOENT;
+
+	*regnum = (id & SVE_REG_ID_MASK) >> SVE_REG_ID_SHIFT;
+
+	return 0;
+}
+
+static int sve_preg_index(u64 id, unsigned int *regnum)
+{
+	const u64 preg_id_max = KVM_REG_ARM64_SVE_FFR(KVM_ARM64_SVE_MAX_SLICES - 1);
+	const u64 preg_id_min = KVM_REG_ARM64_SVE_PREG(0, 0);
+
+	if (id < preg_id_min || id > preg_id_max)
+		return -EINVAL;
+	if ((id & SVE_REG_SLICE_MASK) > 0)
+		return -ENOENT;
+
+	*regnum = (id & SVE_REG_ID_MASK) >> SVE_REG_ID_SHIFT;
+
+	return 0;
+}
+
+static int sve_ffr_index(u64 id, unsigned int *regnum)
+{
+	if (id != KVM_REG_ARM64_SVE_FFR(0))
+		return -EINVAL;
+	if ((id & SVE_REG_SLICE_MASK) > 0)
+		return -ENOENT;
+
+	*regnum = (id & SVE_REG_ID_MASK) >> SVE_REG_ID_SHIFT;
+
+	return 0;
+}
+
+static inline int get_sve_ffr_reg(struct kvm_vcpu *vcpu, unsigned int regnum,
+				  u16 __user *uptr)
+{
+	/* ffr is pregmax + 1 */
+	if (regnum != KVM_ARM64_SVE_NUM_PREGS)
+		return -EINVAL;
+
+	if (put_user(vcpu->arch.sae_block.sve_ffr, uptr))
+		return -EFAULT;
+	return sizeof(vcpu->arch.sae_block.sve_ffr);
+}
+
+static inline int get_sve_preg(struct kvm_vcpu *vcpu, unsigned int regnum,
+			       u16 __user *uptr)
+{
+	if (regnum >= KVM_ARM64_SVE_NUM_PREGS)
+		return -EINVAL;
+
+	if (put_user(vcpu->arch.sae_block.sve_pregs[regnum], uptr))
+		return -EFAULT;
+	return sizeof(vcpu->arch.sae_block.sve_pregs[regnum]);
+}
+
+static inline int get_sve_zreg(struct kvm_vcpu *vcpu, unsigned int regnum,
+			       __vector128 __user *uptr)
+{
+	if (regnum >= KVM_ARM64_SVE_NUM_ZREGS)
+		return -EINVAL;
+
+	/* vreg and svreg overlap and zreg is also just 128 bit so we reuse the vreg space */
+	if (copy_to_user(uptr, &vcpu->arch.ctxt.vregs[regnum],
+			 sizeof(vcpu->arch.ctxt.vregs[regnum])))
+		return -EFAULT;
+	return sizeof(vcpu->arch.ctxt.vregs[regnum]);
+}
+
+static inline int set_sve_ffr_reg(struct kvm_vcpu *vcpu, unsigned int regnum,
+				  const u16 __user *uptr)
+{
+	/* ffr is pregmax + 1*/
+	if (regnum != KVM_ARM64_SVE_NUM_PREGS)
+		return -EINVAL;
+
+	if (get_user(vcpu->arch.sae_block.sve_ffr, uptr))
+		return -EFAULT;
+	return 0;
+}
+
+static inline int set_sve_preg(struct kvm_vcpu *vcpu, unsigned int regnum,
+			       const u16 __user *uptr)
+{
+	if (regnum >= KVM_ARM64_SVE_NUM_PREGS)
+		return -EINVAL;
+
+	if (get_user(vcpu->arch.sae_block.sve_pregs[regnum], uptr))
+		return -EFAULT;
+	return 0;
+}
+
+static inline int set_sve_zreg(struct kvm_vcpu *vcpu, unsigned int regnum,
+			       const __vector128 __user *uptr)
+{
+	if (regnum >= KVM_ARM64_SVE_NUM_ZREGS)
+		return -EINVAL;
+
+	/*vreg and svreg overlap and zreg is also just 128 bit so we reuse the vreg space*/
+	if (copy_from_user(&vcpu->arch.ctxt.vregs[regnum], uptr,
+			   sizeof(vcpu->arch.ctxt.vregs[regnum])))
+		return -EFAULT;
+	return 0;
+}
+
+static int set_sve_vls(struct kvm_vcpu *vcpu, const void __user *uptr)
+{
+	u64 vqs[KVM_ARM64_SVE_VLS_WORDS] = { 0 };
+	unsigned int vq;
+
+	if (!vcpu_has_sve(vcpu))
+		return -ENOENT;
+
+	if (kvm_arm_vcpu_sve_finalized(vcpu))
+		return -EPERM;
+
+	if (copy_from_user(vqs, uptr, sizeof(vqs)))
+		return -EFAULT;
+
+	/* only 128 bit and 1 VQ are supported , nothing saved just check validity */
+	for (vq = KVM_ARM64_SVE_VQ_MIN + 1; vq <= KVM_ARM64_SVE_VQ_MAX; ++vq)
+		if (vq_present(vqs, vq))
+			return -EINVAL;
+
+	/* run with a vl of 0 not valid */
+	if (!vq_present(vqs, KVM_ARM64_SVE_VQ_MIN))
+		return -EINVAL;
+
+	return 0;
+}
+
+static int get_sve_vls(struct kvm_vcpu *vcpu, void __user *uptr)
+{
+	u64 vqs[KVM_ARM64_SVE_VLS_WORDS] = { 0 };
+
+	if (!vcpu_has_sve(vcpu))
+		return -ENOENT;
+
+	/* currently only 128 bit are supported so we only set bit 0  hardcoded */
+	vqs[0] |= vq_mask(KVM_ARM64_SVE_VQ_MIN);
+
+	if (copy_to_user(uptr, vqs, sizeof(vqs)))
+		return -EFAULT;
+
+	return 0;
+}
+
+static int sve_reg_index(u64 id, unsigned int *regnum)
+{
+	int ret;
+
+	ret = sve_ffr_index(id, regnum);
+	if (ret == -EINVAL)
+		ret = sve_preg_index(id, regnum);
+	if (ret == -EINVAL)
+		ret = sve_zreg_index(id, regnum);
+
+	return ret;
+}
+
+static int get_sve_reg(struct kvm_vcpu *vcpu, const struct kvm_one_reg *reg)
+{
+	void __user *uptr = (void __user *)reg->addr;
+	unsigned int regnum;
+	int size;
+	int ret;
+
+	if (reg->id == KVM_REG_ARM64_SVE_VLS)
+		return get_sve_vls(vcpu, uptr);
+
+	ret = sve_reg_index(reg->id, &regnum);
+	if (ret)
+		return ret;
+
+	if (!vcpu_has_sve(vcpu))
+		return -ENOENT;
+
+	if (!kvm_arm_vcpu_sve_finalized(vcpu))
+		return -EPERM;
+
+	if (sve_ffr_index(reg->id, &regnum) == 0)
+		size = get_sve_ffr_reg(vcpu, regnum, uptr);
+	else if (sve_preg_index(reg->id, &regnum) == 0)
+		size = get_sve_preg(vcpu, regnum, uptr);
+	else
+		size = get_sve_zreg(vcpu, regnum, uptr);
+
+	if (size < 0)
+		return size;
+
+	/*
+	 * Only the low 128 bits of a Z register and the low 16 bits of a
+	 * P register or the FFR are live: zero-pad the rest.
+	 */
+	if (clear_user((u8 __user *)uptr + size, KVM_REG_SIZE(reg->id) - size))
+		return -EFAULT;
+
+	return 0;
+}
+
+static int set_sve_reg(struct kvm_vcpu *vcpu, const struct kvm_one_reg *reg)
+{
+	const void __user *uptr = (const void __user *)reg->addr;
+	unsigned int regnum;
+	int ret;
+
+	if (reg->id == KVM_REG_ARM64_SVE_VLS)
+		return set_sve_vls(vcpu, uptr);
+
+	ret = sve_reg_index(reg->id, &regnum);
+	if (ret)
+		return ret;
+
+	if (!vcpu_has_sve(vcpu))
+		return -ENOENT;
+
+	if (!kvm_arm_vcpu_sve_finalized(vcpu))
+		return -EPERM;
+
+	if (sve_ffr_index(reg->id, &regnum) == 0)
+		return set_sve_ffr_reg(vcpu, regnum, uptr);
+	if (sve_preg_index(reg->id, &regnum) == 0)
+		return set_sve_preg(vcpu, regnum, uptr);
+	return set_sve_zreg(vcpu, regnum, uptr);
+}
+
 int kvm_arm_copy_reg_indices(struct kvm_vcpu *vcpu, u64 __user *uindices)
 {
 	int ret;
@@ -61,6 +301,11 @@ int kvm_arm_copy_reg_indices(struct kvm_vcpu *vcpu, u64 __user *uindices)
 		return ret;
 	uindices += ret;
 
+	ret = copy_sve_reg_indices(vcpu, uindices);
+	if (ret < 0)
+		return ret;
+	uindices += ret;
+
 	return kvm_arm_copy_sys_reg_indices(vcpu, uindices);
 }
 
@@ -68,6 +313,7 @@ unsigned long kvm_arm_num_regs(struct kvm_vcpu *vcpu)
 {
 	unsigned long num = num_core_regs(vcpu);
 
+	num += num_sve_regs(vcpu);
 	num += kvm_arm_num_sys_reg_descs(vcpu);
 	return num;
 }
@@ -81,6 +327,8 @@ int kvm_arm_get_reg(struct kvm_vcpu *vcpu, const struct kvm_one_reg *reg)
 	switch (reg->id & KVM_REG_ARM_COPROC_MASK) {
 	case KVM_REG_ARM_CORE:
 		return get_core_reg(vcpu, reg);
+	case KVM_REG_ARM64_SVE:
+		return get_sve_reg(vcpu, reg);
 	default:
 		return kvm_arm_sys_reg_get_reg(vcpu, reg);
 	}
@@ -95,6 +343,8 @@ int kvm_arm_set_reg(struct kvm_vcpu *vcpu, const struct kvm_one_reg *reg)
 	switch (reg->id & KVM_REG_ARM_COPROC_MASK) {
 	case KVM_REG_ARM_CORE:
 		return set_core_reg(vcpu, reg);
+	case KVM_REG_ARM64_SVE:
+		return set_sve_reg(vcpu, reg);
 	default:
 		return kvm_arm_sys_reg_set_reg(vcpu, reg);
 	}
diff --git a/arch/s390/kvm/arm64/reset.c b/arch/s390/kvm/arm64/reset.c
index dc00bf398b15..1ce05a95689f 100644
--- a/arch/s390/kvm/arm64/reset.c
+++ b/arch/s390/kvm/arm64/reset.c
@@ -2,6 +2,7 @@
 
 #include <linux/kvm_host.h>
 #include <linux/fpu.h>
+#include <linux/cpufeature.h>
 
 #include <asm/aef.h>
 
@@ -12,8 +13,13 @@
 
 #include <clocksource/arm_arch_timer.h>
 
+#include "feature.h"
+
 bool kvm_arm_vcpu_is_finalized(struct kvm_vcpu *vcpu)
 {
+	if (vcpu_has_sve(vcpu) && !kvm_arm_vcpu_sve_finalized(vcpu))
+		return false;
+
 	return true;
 }
 
@@ -76,6 +82,14 @@ void kvm_reset_vcpu(struct kvm_vcpu *vcpu)
 	 */
 	preempt_disable();
 
+	if (!vcpu_get_flag(vcpu, VCPU_SVE_FINALIZED)) {
+		if (vcpu_has_feature(vcpu, KVM_ARM_VCPU_SVE))
+			set_bit(KVM_ARCH_FLAG_GUEST_HAS_SVE, &vcpu->kvm->arch.flags);
+	} else if (vcpu_has_sve(vcpu)) {
+		memset(vcpu->arch.sae_block.sve_pregs, 0, sizeof(vcpu->arch.sae_block.sve_pregs));
+		vcpu->arch.sae_block.sve_ffr = 0;
+	}
+
 	/* The reset must run with an unloaded save area */
 	loaded = vcpu_is_loaded(vcpu);
 	if (loaded)
@@ -114,5 +128,15 @@ void kvm_reset_vcpu(struct kvm_vcpu *vcpu)
 
 int kvm_arm_vcpu_finalize(struct kvm_vcpu *vcpu, int feature)
 {
+	switch (feature) {
+	case KVM_ARM_VCPU_SVE:
+		if (!vcpu_has_sve(vcpu))
+			return -EINVAL;
+		if (vcpu_get_flag(vcpu, VCPU_SVE_FINALIZED))
+			return -EPERM;
+		vcpu_set_flag(vcpu, VCPU_SVE_FINALIZED);
+		return 0;
+	}
+
 	return -EINVAL;
 }

-- 
2.56.0

[PATCH v3 22/23] KVM: s390: arm64: Promote PTRAUTH capability

[Steffen Eiden]

Promote the pointer authentication capability if supported by hardware
and allow userspace to access those.

Co-developed-by: Andreas Grapentin <gra@linux.ibm.com>
Signed-off-by: Andreas Grapentin <gra@linux.ibm.com>
Signed-off-by: Steffen Eiden <seiden@linux.ibm.com>
---
 arch/s390/include/asm/kvm_host_arm64.h | 12 ++++--------
 arch/s390/kvm/arm64/arm.c              | 17 +++++++++++++++++
 arch/s390/kvm/arm64/feature.c          | 18 ++++++++++++++++++
 arch/s390/kvm/arm64/feature.h          |  5 +++++
 arch/s390/kvm/arm64/sys_regs.c         | 10 ++++++++--
 5 files changed, 52 insertions(+), 10 deletions(-)

diff --git a/arch/s390/include/asm/kvm_host_arm64.h b/arch/s390/include/asm/kvm_host_arm64.h
index f096faceb19d..5a443bba0018 100644
--- a/arch/s390/include/asm/kvm_host_arm64.h
+++ b/arch/s390/include/asm/kvm_host_arm64.h
@@ -34,8 +34,10 @@ bool cpus_have_final_cap(unsigned int num);
 #define KVM_MAX_VCPUS 1
 #define __KVM_HAVE_VCPU_RUN_PID_CHANGE
 
-#define KVM_S390_ARM64_IMPL_FEATURES (		\
-	 BIT(KVM_ARM_VCPU_SVE)			\
+#define KVM_S390_ARM64_IMPL_FEATURES (			\
+	 BIT(KVM_ARM_VCPU_PTRAUTH_ADDRESS)	|	\
+	 BIT(KVM_ARM_VCPU_PTRAUTH_GENERIC)	|	\
+	 BIT(KVM_ARM_VCPU_SVE)				\
 )
 #define KVM_HALT_POLL_NS_DEFAULT 50000
 
@@ -132,7 +134,6 @@ struct kvm_vcpu_stat {
 };
 
 #define kvm_has_mte(_kvm) false
-#define vcpu_has_ptrauth(_vcpu) false
 
 #define kvm_has_fpmr(kvm) \
 	 kvm_has_feat((kvm), ID_AA64PFR2_EL1, FPMR, IMP)
@@ -172,11 +173,6 @@ struct kvm_vm_stat {
 	struct kvm_vm_stat_generic generic;
 };
 
-static inline bool system_has_full_ptr_auth(void)
-{
-	return true;
-}
-
 #define kvm_vm_is_protected(_kvm) false
 #define vcpu_is_protected(_vcpu) false
 
diff --git a/arch/s390/kvm/arm64/arm.c b/arch/s390/kvm/arm64/arm.c
index 8c2a07a96ee3..ee05d7d9f331 100644
--- a/arch/s390/kvm/arm64/arm.c
+++ b/arch/s390/kvm/arm64/arm.c
@@ -57,6 +57,10 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext)
 	case KVM_CAP_ARM_SVE:
 		ret = system_supports_sve();
 		break;
+	case KVM_CAP_ARM_PTRAUTH_ADDRESS:
+	case KVM_CAP_ARM_PTRAUTH_GENERIC:
+		ret = kvm_has_full_ptr_auth();
+		break;
 	default:
 		ret = 0;
 	}
@@ -227,6 +231,12 @@ void kvm_arch_vcpu_destroy(struct kvm_vcpu *vcpu)
 	VCPU_EVENT(vcpu, 3, "%s", "free cpu");
 }
 
+static void vcpu_set_pauth_traps(struct kvm_vcpu *vcpu)
+{
+	if (vcpu_has_ptrauth(vcpu))
+		vcpu->arch.hcr_el2 |= (HCR_EL2_API | HCR_EL2_APK);
+}
+
 void kvm_arch_vcpu_load(struct kvm_vcpu *vcpu, int cpu)
 {
 	save_access_regs(&vcpu->arch.host_acrs[0]);
@@ -235,6 +245,8 @@ void kvm_arch_vcpu_load(struct kvm_vcpu *vcpu, int cpu)
 	lasrm(&vcpu->arch.save_area);
 
 	vcpu->arch.sae_block.gpto = vcpu->kvm->arch.epd;
+
+	vcpu_set_pauth_traps(vcpu);
 }
 
 void kvm_arch_vcpu_put(struct kvm_vcpu *vcpu)
@@ -270,6 +282,11 @@ static unsigned long system_supported_vcpu_features(struct kvm_vcpu *vcpu)
 	if (!system_supports_sve())
 		clear_bit(KVM_ARM_VCPU_SVE, &features);
 
+	if (!kvm_has_full_ptr_auth()) {
+		clear_bit(KVM_ARM_VCPU_PTRAUTH_ADDRESS, &features);
+		clear_bit(KVM_ARM_VCPU_PTRAUTH_GENERIC, &features);
+	}
+
 	return features;
 }
 
diff --git a/arch/s390/kvm/arm64/feature.c b/arch/s390/kvm/arm64/feature.c
index 0979e0cadc66..13a56647d78c 100644
--- a/arch/s390/kvm/arm64/feature.c
+++ b/arch/s390/kvm/arm64/feature.c
@@ -168,3 +168,21 @@ bool system_supports_sve(void)
 			     read_sanitised_ftr_reg(SYS_ID_AA64PFR0_EL1)) ==
 		       ID_AA64PFR0_EL1_SVE_IMP;
 }
+
+bool system_has_full_ptr_auth(void)
+{
+	u64 isar1 = read_sanitised_ftr_reg(SYS_ID_AA64ISAR1_EL1);
+	u64 isar2 = read_sanitised_ftr_reg(SYS_ID_AA64ISAR2_EL1);
+	bool address_auth;
+	bool generic_auth;
+
+	address_auth = SYS_FIELD_GET(ID_AA64ISAR1_EL1, APA, isar1) >= ID_AA64ISAR1_EL1_APA_PAuth ||
+		       SYS_FIELD_GET(ID_AA64ISAR1_EL1, API, isar1) >= ID_AA64ISAR1_EL1_API_PAuth ||
+		       SYS_FIELD_GET(ID_AA64ISAR2_EL1, APA3, isar2) >= ID_AA64ISAR2_EL1_APA3_PAuth;
+
+	generic_auth = SYS_FIELD_GET(ID_AA64ISAR1_EL1, GPA, isar1) >= ID_AA64ISAR1_EL1_GPA_IMP ||
+		       SYS_FIELD_GET(ID_AA64ISAR1_EL1, GPI, isar1) >= ID_AA64ISAR1_EL1_GPI_IMP ||
+		       SYS_FIELD_GET(ID_AA64ISAR2_EL1, GPA3, isar2) >= ID_AA64ISAR2_EL1_GPA3_IMP;
+
+	return address_auth && generic_auth;
+}
diff --git a/arch/s390/kvm/arm64/feature.h b/arch/s390/kvm/arm64/feature.h
index 9af6173f6b11..3ca1ce0139b0 100644
--- a/arch/s390/kvm/arm64/feature.h
+++ b/arch/s390/kvm/arm64/feature.h
@@ -89,4 +89,9 @@ static inline u64 kvm_sae_irptc(void)
 
 bool system_supports_sve(void);
 
+#define vcpu_has_ptrauth(vcpu)						\
+	 (vcpu_has_feature(vcpu, KVM_ARM_VCPU_PTRAUTH_ADDRESS) ||       \
+	  vcpu_has_feature(vcpu, KVM_ARM_VCPU_PTRAUTH_GENERIC))
+bool system_has_full_ptr_auth(void);
+
 #endif /* ARCH_S390_KVM_FEATURE_H */
diff --git a/arch/s390/kvm/arm64/sys_regs.c b/arch/s390/kvm/arm64/sys_regs.c
index 951a02d89fc5..8ee17e7fd388 100644
--- a/arch/s390/kvm/arm64/sys_regs.c
+++ b/arch/s390/kvm/arm64/sys_regs.c
@@ -32,7 +32,6 @@ static u64 __ro_after_init boot_cpu_aidr_val;
 #undef __INCL_GEN_ARM_FILE
 
 DEFINE_STATIC_KEY_FALSE(broken_cntvoff_key);
-
 #define SR_INVALID 0x8badf00d8badf00d
 
 enum sr_loc_attr {
@@ -634,6 +633,14 @@ static const struct sys_reg_desc sys_reg_descs[] = {
 	{ SYS_DESC(SYS_TTBR1_EL1), access_rw, reset_val, TTBR1_EL1, 0 },
 	{ SYS_DESC(SYS_TCR_EL1), access_rw, reset_val, TCR_EL1, 0 },
 
+	/* CRm = 1 */
+	PTRAUTH_KEY(APIA),
+	PTRAUTH_KEY(APIB),
+	/* CRm = 2 */
+	PTRAUTH_KEY(APDA),
+	PTRAUTH_KEY(APDB),
+	/* CRm = 3 */
+	PTRAUTH_KEY(APGA),
 	/* CRn = 5 */
 	{ SYS_DESC(SYS_AFSR0_EL1), access_rw, reset_unknown, AFSR0_EL1 },
 	{ SYS_DESC(SYS_AFSR1_EL1), access_rw, reset_unknown, AFSR1_EL1 },
@@ -642,7 +649,6 @@ static const struct sys_reg_desc sys_reg_descs[] = {
 	{ SYS_DESC(SYS_FAR_EL1), access_rw, reset_unknown, FAR_EL1 },
 	/* CRn = 7 */
 	{ SYS_DESC(SYS_PAR_EL1), access_rw, reset_unknown, PAR_EL1 },
-
 	{ SYS_DESC(SYS_PMMIR_EL1), trap_raz_wi },
 	/* CRn = 10 */
 	{ SYS_DESC(SYS_MAIR_EL1), NULL, reset_unknown, MAIR_EL1 },

-- 
2.56.0

[PATCH v3 23/23] s390: Report AEF features to sysfs

[Steffen Eiden]

Display various information about the Arm execution Facility (AEF) to
userspace. The files are located in /sys/firmware/aef/. Show if the
feature is available and usable for the host, which formats are
available, and how many arm vCPUS per guest are supported by the
machine. Additionally, provide the raw binary output of QAAF fc1 (Query
Model Capabilities) in the file qmc and fc2 (Initial
System-Registers-Save Area) in the file save_area.

Co-developed-by: Janosch Frank <frankja@linux.ibm.com>
Signed-off-by: Janosch Frank <frankja@linux.ibm.com>
Co-developed-by: Andreas Grapentin <gra@linux.ibm.com>
Signed-off-by: Andreas Grapentin <gra@linux.ibm.com>
Signed-off-by: Steffen Eiden <seiden@linux.ibm.com>
---
 Documentation/ABI/testing/sysfs-firmware-aef |  70 +++++++++++++++++
 arch/s390/kernel/aef.c                       | 113 +++++++++++++++++++++++++++
 2 files changed, 183 insertions(+)

diff --git a/Documentation/ABI/testing/sysfs-firmware-aef b/Documentation/ABI/testing/sysfs-firmware-aef
new file mode 100644
index 000000000000..6ccd8491a017
--- /dev/null
+++ b/Documentation/ABI/testing/sysfs-firmware-aef
@@ -0,0 +1,70 @@
+What:		/sys/firmware/aef/
+KernelVersion:	7.5
+Contact:	linux-s390@vger.kernel.org
+Description:	Various interfaces to query information about the ARM
+		Execution Facility (AEF).
+
+		The directory is present only on machines that provide
+		the AEF (SCLP facility bit).
+
+What:		/sys/firmware/aef/arm_guest_supp
+KernelVersion:	7.5
+Contact:	linux-s390@vger.kernel.org
+Description:	Read-only, decimal boolean. Reports 1 if all requirements
+		for running arm64 KVM guests (KVM module kvm_arm64) are
+		satisfied by the machine, 0 otherwise. This is equivalent to the
+		logical AND of sae_avail, ptff_avail and edat2_avail below.
+
+What:		/sys/firmware/aef/query/
+KernelVersion:	7.5
+Contact:	linux-s390@vger.kernel.org
+Description:	Raw machine characteristics as reported by the QAAF
+		instruction.
+
+What:		/sys/firmware/aef/query/sae_avail
+KernelVersion:	7.5
+Contact:	linux-s390@vger.kernel.org
+Description:	Read-only, decimal boolean. Reports 1 if the machine
+		provides the SAE instruction.
+
+What:		/sys/firmware/aef/query/ptff_avail
+KernelVersion:	7.5
+Contact:	linux-s390@vger.kernel.org
+Description:	Read-only, decimal boolean. Reports 1 if the PTFF
+		instruction supports the QAGTO and QAGPT subfunctions.
+
+What:		/sys/firmware/aef/query/edat2_avail
+KernelVersion:	7.5
+Contact:	linux-s390@vger.kernel.org
+Description:	Read-only, decimal boolean. Reports 1 if the EDAT-2
+		facility is available.
+
+What:		/sys/firmware/aef/query/supp_state_desc_formats
+KernelVersion:	7.5
+Contact:	linux-s390@vger.kernel.org
+Description:	Read-only, 0x-prefixed hexadecimal bitmask. Bitmask of
+		the SAE state description formats supported by the machine.
+
+What:		/sys/firmware/aef/query/supp_save_area_formats
+KernelVersion:	7.5
+Contact:	linux-s390@vger.kernel.org
+Description:	Read-only, 0x-prefixed hexadecimal bitmask. Bitmask of
+		the SAE save area formats supported by the machine.
+
+What:		/sys/firmware/aef/query/max_num_vcpu
+KernelVersion:	7.5
+Contact:	linux-s390@vger.kernel.org
+Description:	Read-only, decimal number. Maximum number of guest CPUs
+		per VM.
+
+What:		/sys/firmware/aef/query/qmc
+KernelVersion:	7.5
+Contact:	linux-s390@vger.kernel.org
+Description:	Read-only binary attribute, 4096 bytes. Contains the
+		unmodified, machine specific QMC response block.
+
+What:		/sys/firmware/aef/query/save_area
+KernelVersion:	7.5
+Contact:	linux-s390@vger.kernel.org
+Description:	Read-only binary attribute, 4096 bytes. Contains the
+		unmodified, machine specific GISRSA block.
diff --git a/arch/s390/kernel/aef.c b/arch/s390/kernel/aef.c
index 31e18953d7b3..96a6d63b2695 100644
--- a/arch/s390/kernel/aef.c
+++ b/arch/s390/kernel/aef.c
@@ -11,6 +11,15 @@
 
 static struct qaaf_qmc_block qmc = {};
 static struct kvm_sae_save_area save_area = {};
+static struct aef_info {
+	unsigned long arm_guest_supp;
+	unsigned long sae_avail;
+	unsigned long ptff_avail;
+	unsigned long edat2_avail;
+	unsigned long supp_state_desc_formats;
+	unsigned long supp_save_area_formats;
+	unsigned long max_num_vcpu;
+} info = {};
 
 const struct kvm_sae_save_area *aef_gisrsa(void)
 {
@@ -38,6 +47,68 @@ bool aef_usable(void)
 }
 EXPORT_SYMBOL_GPL(aef_usable);
 
+static struct kset *aef_query_kset;
+static struct kobject *aef_kobj;
+
+#define aef_sysfs_qval_def(_name, _fmt)				\
+	static ssize_t _name##_show(struct kobject *kobj,		\
+				    struct kobj_attribute *attr,	\
+				    char *buf)				\
+	{								\
+		return sysfs_emit(buf, _fmt "\n", info._name);		\
+	}								\
+	static struct kobj_attribute _name##_attr = __ATTR_RO(_name)
+
+#define aef_sysfs_qval_attr(_name) &_name##_attr.attr
+
+aef_sysfs_qval_def(arm_guest_supp, "%lu");
+aef_sysfs_qval_def(sae_avail, "%lu");
+aef_sysfs_qval_def(ptff_avail, "%lu");
+aef_sysfs_qval_def(edat2_avail, "%lu");
+aef_sysfs_qval_def(supp_state_desc_formats, "0x%lx");
+aef_sysfs_qval_def(supp_save_area_formats, "0x%lx");
+aef_sysfs_qval_def(max_num_vcpu, "%lu");
+
+static ssize_t qmc_read(struct file *filp, struct kobject *kobj,
+			const struct bin_attribute *attr, char *buf, loff_t off,
+			size_t count)
+{
+	return memory_read_from_buffer(buf, count, &off, &qmc, sizeof(qmc));
+}
+
+BIN_ATTR_RO(qmc, sizeof(qmc));
+
+static ssize_t save_area_read(struct file *filp, struct kobject *kobj,
+			      const struct bin_attribute *attr, char *buf,
+			      loff_t off, size_t count)
+{
+	return memory_read_from_buffer(buf, count, &off, &save_area,
+				       sizeof(save_area));
+}
+
+BIN_ATTR_RO(save_area, sizeof(save_area));
+
+static struct attribute *aef_query_attrs[] = {
+	aef_sysfs_qval_attr(sae_avail),
+	aef_sysfs_qval_attr(ptff_avail),
+	aef_sysfs_qval_attr(edat2_avail),
+	aef_sysfs_qval_attr(supp_state_desc_formats),
+	aef_sysfs_qval_attr(supp_save_area_formats),
+	aef_sysfs_qval_attr(max_num_vcpu),
+	NULL,
+};
+
+static const struct bin_attribute *aef_query_bin_attrs[] = {
+	&bin_attr_qmc,
+	&bin_attr_save_area,
+	NULL,
+};
+
+static struct attribute_group aef_query_attr_group = {
+	.attrs = aef_query_attrs,
+	.bin_attrs = aef_query_bin_attrs,
+};
+
 static int __init aef_init_save_area(void)
 {
 	int ret;
@@ -71,6 +142,17 @@ static int __init aef_query_info(void)
 	return 0;
 }
 
+static void __init aef_info_init(void)
+{
+	info.sae_avail = sclp.has_aef;
+	info.ptff_avail = ptff_query(PTFF_QAGTO) && ptff_query(PTFF_QAGPT);
+	info.edat2_avail = cpu_has_edat2();
+	info.arm_guest_supp = info.sae_avail && info.ptff_avail && info.edat2_avail;
+	info.supp_state_desc_formats = qmc.ssdf;
+	info.supp_save_area_formats = qmc.ssaf;
+	info.max_num_vcpu = qmc.maxncpu + 1;
+}
+
 static int __init aef_sysfs_init(void)
 {
 	int rc;
@@ -86,9 +168,40 @@ static int __init aef_sysfs_init(void)
 	if (rc)
 		return rc;
 
+	/* sysfs failure below does not invalidate the AEF data */
 	aef_valid = true;
 
+	aef_info_init();
+
+	aef_kobj = kobject_create_and_add("aef", firmware_kobj);
+	if (!aef_kobj)
+		return -ENOMEM;
+
+	rc = sysfs_create_file(aef_kobj, aef_sysfs_qval_attr(arm_guest_supp));
+	if (rc)
+		goto out_kobj;
+
+	aef_query_kset = kset_create_and_add("query", NULL, aef_kobj);
+	if (!aef_query_kset) {
+		rc = -ENOMEM;
+		goto out_arm_guest_supp;
+	}
+
+	rc = sysfs_create_group(&aef_query_kset->kobj, &aef_query_attr_group);
+	if (rc)
+		goto out_kset;
+
 	return 0;
+
+out_kset:
+	kset_unregister(aef_query_kset);
+out_arm_guest_supp:
+	sysfs_remove_file(aef_kobj, &arm_guest_supp_attr.attr);
+out_kobj:
+	kobject_del(aef_kobj);
+	kobject_put(aef_kobj);
+
+	return rc;
 }
 
 arch_initcall(aef_sysfs_init);

-- 
2.56.0