From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-alma10-1.taild15c8.ts.net [100.103.45.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1D1152874E6; Sat, 5 Sep 2026 13:46:15 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=100.103.45.18 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788615978; cv=none; b=nXWpR+Ttt4pTtBicfyuC3ueUA0o6PSrtvem7Q7343wlAjsxUCN6x3Y4P8rIEimWq8uGClwF0yILzBzeyHe6cPl6YIkeKt1ltn6cLeLw7eED7PvbPV8W80kyzUWK2H7+vj7Ohy3b7tmZKHXPLCRlGm74zSLKjxd8Ul3RvCLs0nRc= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788615978; c=relaxed/simple; bh=nw6fhZTnssNJ9AFFTMsK1RX9MQOx/jIgjT81SgsWk00=; h=Date:Message-ID:From:To:Cc:Subject:In-Reply-To:References: MIME-Version:Content-Type; b=Fo8Fx6OPkJkN2YY/slG9eTfXT3kF6jcWL0ln5zulheFjzUdvERUWnYhlC0Qlov/xORwa/pTZnmhArLwGJEaOsOQFPHvxOmG1hQQdRKpPu6Xe1R2IhNbpriLmmeCKk1oSIWk/0qbNyXVbhKVAYvF9xi/XsVI+KaZ3gcFdm3Of36Y= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=KKYfrYUw; arc=none smtp.client-ip=100.103.45.18 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="KKYfrYUw" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 41BBE1F00A3D; Sat, 5 Sep 2026 13:46:14 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1788615974; bh=RqcAI7pAD1odDIdi2AWSF2YS3bFn9A/ym55+hvQNRWc=; h=Date:From:To:Cc:Subject:In-Reply-To:References; b=KKYfrYUws700otlYQUHyLcpOJNhQo01jl4qSFhjAYAEg0mtiyIbMeU6W/VHtKikB/ Y83twcaz62+2oPyG0hn8yLdncehFHY2Y0rH/rjcUXLEDxwTCcWpDYsNr9w6g/1/tuM HEK0uIISqdM2ObqPBnCTrduionIOZnyuHl677fkE+Gluy6BTyIk9B5L7yFT40XI5gV JaPCzUij6A2+MGbbGPrMbv5s0m+5Z7WaxkPgzfm5+XYYWdJr1m+VU8P8B9uf1lKZSw 9T3aSjGkx0Ic8KqclXZ40gBj+Fzu20sO0oSHBcPAVDn6JFErGBNvNlVxDVJrnjwDUV or0reerjfpCAQ== Received: from sofa.misterjones.org ([185.219.108.64] helo=lobster-girl.misterjones.org) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.98.2) (envelope-from ) id 1x2qih-00000005CjH-2Ap3; Sat, 05 Sep 2026 13:46:11 +0000 Date: Sat, 05 Sep 2026 14:48:41 +0100 Message-ID: <87pkyr4xmu.wl-maz@kernel.org> From: Marc Zyngier To: Wei-Lin Chang Cc: Wang Han , linux-arm-kernel@lists.infradead.org, kvmarm@lists.linux.dev, linux-kernel@vger.kernel.org, oupton@kernel.org, tabba@google.com, joey.gouly@arm.com, seiden@linux.ibm.com, suzuki.poulose@arm.com, catalin.marinas@arm.com, will@kernel.org, ljs@kernel.org, itaru.kitayama@fujitsu.com Subject: Re: [PATCH v5 0/6] KVM: arm64: nv: Implement nested stage-2 reverse map (new data structure) In-Reply-To: References: <20260810205038.118843-1-weilin.chang@arm.com> <20260902163500.1841671-1-wanghan@linux.alibaba.com> <877bl26aqg.wl-maz@kernel.org> <875x0l5udh.wl-maz@kernel.org> User-Agent: Wanderlust/2.15.9 (Almost Unreal) SEMI-EPG/1.14.7 (Harue) FLIM-LB/1.14.9 (=?UTF-8?B?R29qxY0=?=) APEL-LB/10.8 EasyPG/1.0.0 Emacs/30.1 (aarch64-unknown-linux-gnu) MULE/6.0 (HANACHIRUSATO) Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 (generated by SEMI-EPG 1.14.7 - "Harue") Content-Type: text/plain; charset=US-ASCII X-SA-Exim-Connect-IP: 185.219.108.64 X-SA-Exim-Rcpt-To: weilin.chang@arm.com, wanghan@linux.alibaba.com, linux-arm-kernel@lists.infradead.org, kvmarm@lists.linux.dev, linux-kernel@vger.kernel.org, oupton@kernel.org, tabba@google.com, joey.gouly@arm.com, seiden@linux.ibm.com, suzuki.poulose@arm.com, catalin.marinas@arm.com, will@kernel.org, ljs@kernel.org, itaru.kitayama@fujitsu.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false On Fri, 04 Sep 2026 23:42:15 +0100, Wei-Lin Chang wrote: > > On Fri, Sep 04, 2026 at 08:49:14AM +0100, Marc Zyngier wrote: > > [...] > > > > > > > > > > > The L1 was started with 8 vCPUs and 32 GiB of RAM using: > > > > > > > > > > qemu-system-aarch64 -smp 8 -m 32G \ > > > > > -machine virt,accel=kvm,gic-version=3,virtualization=on \ > > > > > -cpu host -nographic -enable-kvm \ > > > > > -drive if=pflash,format=raw,readonly=on,file=pflash0_bak.img \ > > > > > -drive if=pflash,format=raw,file=pflash1_bak.img \ > > > > > -drive file=./ubuntu-vm.qcow2,format=qcow2,if=virtio,cache=none,aio=native \ > > > > > -nic user,model=virtio-net-pci,hostfwd=tcp::11234-:22 \ > > > > > -serial mon:stdio > > > > > > > > > > > > > Puzzling. If you are only running an L1 in VHE mode, there is no > > > > shadow S2, and therefore nothing to unmap. For shadow S2s to be built > > > > and affect the MMU notifiers, you need to run an L2. > > > > > > I was thinking the same at first, but realized even with L1 in VHE mode > > > there is a small period of time where L1 runs in its EL1 during boot, so > > > one nested MMU will become valid for each vCPU. That causes > > > kvm_nested_s2_unmap() to iterate through the entire IPA space 8 times > > > (-smp 8). > > > > It should be one nested MMU for the whole VM, not one per vcpu. > > That's assuming they share the same VMID+VTCR. > > > > > What I am curious about is whether one single notifier unmap is enough > > > to hang L1, or were there multiple notifier unmaps. > > > > > > QEMU with -machine virt uses 40 IPA bits only, unmapping that takes: > > > 1024 (4KB pages, unmapping 1GB per iteration) > > > 32768 (16KB pages, unmapping 32MB per iteration) > > > 2048 (64KB pages, unmapping 512MB per iteration) > > > iterations for each page size. There aren't many mappings in each > > > iteration too. Does this really take that long on real hardware (even if > > > this must be done 8 times)? > > > > This should be close to being at zero cost, so something else is > > amiss. > > > > Guess what? I am completely wrong once again. > > The complete range of the nested mmus isn't about the value > supplied by KVM_VM_TYPE_ARM_IPA_SIZE. Instead it is the PARange exposed > to the guest. That's because the guest hypervisor is allowed to expose > whatever memory map it likes to its nested guests, as long as it is > within the guest's PARange. This is explained clearly in > init_nested_s2_mmu(). > > Now the PARange exposed to guests should be much larger than 40bits. On > the neoverse-n2 it should be 48bits (256TB), therefore the iterations > required for each unmapping would be 256x: > > 262144 (4KB pages, unmapping 1GB per iteration) > 8388608 (16KB pages, unmapping 32MB per iteration) > 524288 (64KB pages, unmapping 512MB per iteration) > > I think these numbers match the hang time Shuai have shown. > > Just to demonstrate the difference I have tried manually limiting > kvm_ipa_limit to 40, and tested with software emulation. Before the > manual change a kvm_nested_s2_unmap() finished after 42 seconds, after > limiting kvm_ipa_limit kvm_nested_s2_unmap() can finish in less than a > second. > > With this, I think there aren't other underlying issues, it just is that > slow unfortunately.. I don't buy this. There should be very little mapped in the shadow S2, a few dozen pages at most. As you noticed, there is very little the kernel executes at EL1 before upgrading to EL2. Invalidating this should be cheap. Furthermore, once invalidated after the last vcpu has booted, the cost should be exactly zero. It seems that these two assumptions are not holding up, and I wonder why. M. -- Jazz isn't dead. It just smells funny.