From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail.loongson.cn (mail.loongson.cn [114.242.206.163]) by smtp.subspace.kernel.org (Postfix) with ESMTP id E43D822A7F1; Mon, 29 Dec 2025 08:52:59 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=114.242.206.163 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1766998384; cv=none; b=o8gT9R428APM/eoIjl2PuIXwVhYbrfWtt7h8De3SbN0BpEMIeB5UvzBx4E4x4gt7KokouwaHELvpgd0sz9DZqO4HkYhpCLERWdKfelUOmSHKhqVjreAUc8PUTKuDLyGr988XTjFpxHJB1k2TEvf9PiYVAG1JQpVHbKq4BAvgRmc= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1766998384; c=relaxed/simple; bh=1d9RO0KDVFMKwVGy0z3bG4f8MOgC1CsTpOkIxDtoEA0=; h=Subject:To:Cc:References:From:Message-ID:Date:MIME-Version: In-Reply-To:Content-Type; b=nappWRhYT6rsUyGp623avALK792gNqR32GJqpYhaNrEJrMSy1QEDm3qN8yAoBp0QQpmMZM1yRXLEiECkdxhsFdPojfRzMq4Uqzb4vEdjItZFvOMdo1F+DFnRcsNxtXs+QayFRtB5VEPEz5lX4gmrPq1EIiWY9XS5pntkwSRoDLs= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=loongson.cn; spf=pass smtp.mailfrom=loongson.cn; arc=none smtp.client-ip=114.242.206.163 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=loongson.cn Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=loongson.cn Received: from loongson.cn (unknown [10.20.42.62]) by gateway (Coremail) with SMTP id _____8DxAfFlQVJpjAoEAA--.13295S3; Mon, 29 Dec 2025 16:52:53 +0800 (CST) Received: from [10.20.42.62] (unknown [10.20.42.62]) by front1 (Coremail) with SMTP id qMiowJBx78JfQVJpPSIGAA--.16867S3; Mon, 29 Dec 2025 16:52:49 +0800 (CST) Subject: Re: [PATCH V3 2/2] LoongArch: KVM: fix "unreliable stack" issue To: Jinyang He , lixianglai Cc: loongarch@lists.linux.dev, linux-kernel@vger.kernel.org, kvm@vger.kernel.org, stable@vger.kernel.org, Huacai Chen , WANG Xuerui , Tianrui Zhao , Charlie Jenkins , Thomas Gleixner , Tiezhu Yang References: <20251227012712.2921408-1-lixianglai@loongson.cn> <20251227012712.2921408-3-lixianglai@loongson.cn> <08143343-cb10-9376-e7df-68ad854b9275@loongson.cn> <9e1a8d4f-251f-f78e-01a3-5c483249fac8@loongson.cn> From: Bibo Mao Message-ID: Date: Mon, 29 Dec 2025 16:50:15 +0800 User-Agent: Mozilla/5.0 (X11; Linux loongarch64; rv:68.0) Gecko/20100101 Thunderbird/68.7.0 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 8bit X-CM-TRANSID:qMiowJBx78JfQVJpPSIGAA--.16867S3 X-CM-SenderInfo: xpdruxter6z05rqj20fqof0/ X-Coremail-Antispam: 1Uk129KBj93XoW3JFy5Aw47Xr4ruw4DtrW3XFc_yoWfCrWkpw nayFsIyrWDA34kAw4UKa4DAryxtrW8G3WUWr1xXFy8Jr4qvr1Yqr48Xr1v9ryDC3y0gFyq qFWUt3ZrZan8ArcCm3ZEXasCq-sJn29KB7ZKAUJUUUU7529EdanIXcx71UUUUU7KY7ZEXa sCq-sGcSsGvfJ3Ic02F40EFcxC0VAKzVAqx4xG6I80ebIjqfuFe4nvWSU5nxnvy29KBjDU 0xBIdaVrnRJUUUB2b4IE77IF4wAFF20E14v26r1j6r4UM7CY07I20VC2zVCF04k26cxKx2 IYs7xG6rWj6s0DM7CIcVAFz4kK6r1Y6r17M28lY4IEw2IIxxk0rwA2F7IY1VAKz4vEj48v e4kI8wA2z4x0Y4vE2Ix0cI8IcVAFwI0_Gr0_Xr1l84ACjcxK6xIIjxv20xvEc7CjxVAFwI 0_Gr0_Cr1l84ACjcxK6I8E87Iv67AKxVW8Jr0_Cr1UM28EF7xvwVC2z280aVCY1x0267AK xVW8Jr0_Cr1UM2kKe7AKxVWUXVWUAwAS0I0E0xvYzxvE52x082IY62kv0487Mc804VCY07 AIYIkI8VC2zVCFFI0UMc02F40EFcxC0VAKzVAqx4xG6I80ewAv7VC0I7IYx2IY67AKxVWU AVWUtwAv7VC2z280aVAFwI0_Gr0_Cr1lOx8S6xCaFVCjc4AY6r1j6r4UM4x0Y48IcVAKI4 8JMxk0xIA0c2IEe2xFo4CEbIxvr21l42xK82IYc2Ij64vIr41l4I8I3I0E4IkC6x0Yz7v_ Jr0_Gr1l4IxYO2xFxVAFwI0_Jrv_JF1lx2IqxVAqx4xG67AKxVWUJVWUGwC20s026x8Gjc xK67AKxVWUGVWUWwC2zVAF1VAY17CE14v26r1q6r43MIIYrxkI7VAKI48JMIIF0xvE2Ix0 cI8IcVAFwI0_Jr0_JF4lIxAIcVC0I7IYx2IY6xkF7I0E14v26r1j6r4UMIIF0xvE42xK8V AvwI8IcIk0rVWUJVWUCwCI42IY6I8E87Iv67AKxVW8JVWxJwCI42IY6I8E87Iv6xkF7I0E 14v26r4j6r4UJbIYCTnIWIevJa73UjIFyTuYvjxU466zUUUUU On 2025/12/29 下午3:10, Jinyang He wrote: > > On 2025-12-29 11:53, lixianglai wrote: >> Hi Jinyang: >>> On 2025-12-27 09:27, Xianglai Li wrote: >>> >>>> Insert the appropriate UNWIND macro definition into the >>>> kvm_exc_entry in >>>> the assembly function to guide the generation of correct ORC table >>>> entries, >>>> thereby solving the timeout problem of loading the livepatch-sample >>>> module >>>> on a physical machine running multiple vcpus virtual machines. >>>> >>>> While solving the above problems, we have gained an additional benefit, >>>> that is, we can obtain more call stack information >>>> >>>> Stack information that can be obtained before the problem is fixed: >>>> [<0>] kvm_vcpu_block+0x88/0x120 [kvm] >>>> [<0>] kvm_vcpu_halt+0x68/0x580 [kvm] >>>> [<0>] kvm_emu_idle+0xd4/0xf0 [kvm] >>>> [<0>] kvm_handle_gspr+0x7c/0x700 [kvm] >>>> [<0>] kvm_handle_exit+0x160/0x270 [kvm] >>>> [<0>] kvm_exc_entry+0x100/0x1e0 >>>> >>>> Stack information that can be obtained after the problem is fixed: >>>> [<0>] kvm_vcpu_block+0x88/0x120 [kvm] >>>> [<0>] kvm_vcpu_halt+0x68/0x580 [kvm] >>>> [<0>] kvm_emu_idle+0xd4/0xf0 [kvm] >>>> [<0>] kvm_handle_gspr+0x7c/0x700 [kvm] >>>> [<0>] kvm_handle_exit+0x160/0x270 [kvm] >>>> [<0>] kvm_exc_entry+0x104/0x1e4 >>>> [<0>] kvm_enter_guest+0x38/0x11c >>>> [<0>] kvm_arch_vcpu_ioctl_run+0x26c/0x498 [kvm] >>>> [<0>] kvm_vcpu_ioctl+0x200/0xcf8 [kvm] >>>> [<0>] sys_ioctl+0x498/0xf00 >>>> [<0>] do_syscall+0x98/0x1d0 >>>> [<0>] handle_syscall+0xb8/0x158 >>>> >>>> Cc: stable@vger.kernel.org >>>> Signed-off-by: Xianglai Li >>>> --- >>>> Cc: Huacai Chen >>>> Cc: WANG Xuerui >>>> Cc: Tianrui Zhao >>>> Cc: Bibo Mao >>>> Cc: Charlie Jenkins >>>> Cc: Xianglai Li >>>> Cc: Thomas Gleixner >>>> Cc: Tiezhu Yang >>>> >>>>   arch/loongarch/kvm/switch.S | 28 +++++++++++++++++++--------- >>>>   1 file changed, 19 insertions(+), 9 deletions(-) >>>> >>>> diff --git a/arch/loongarch/kvm/switch.S b/arch/loongarch/kvm/switch.S >>>> index 93845ce53651..a3ea9567dbe5 100644 >>>> --- a/arch/loongarch/kvm/switch.S >>>> +++ b/arch/loongarch/kvm/switch.S >>>> @@ -10,6 +10,7 @@ >>>>   #include >>>>   #include >>>>   #include >>>> +#include >>>>     #define HGPR_OFFSET(x)        (PT_R0 + 8*x) >>>>   #define GGPR_OFFSET(x)        (KVM_ARCH_GGPR + 8*x) >>>> @@ -110,9 +111,9 @@ >>>>        * need to copy world switch code to DMW area. >>>>        */ >>>>       .text >>>> +    .p2align PAGE_SHIFT >>>>       .cfi_sections    .debug_frame >>>>   SYM_CODE_START(kvm_exc_entry) >>>> -    .p2align PAGE_SHIFT >>>>       UNWIND_HINT_UNDEFINED >>>>       csrwr    a2,   KVM_TEMP_KS >>>>       csrrd    a2,   KVM_VCPU_KS >>>> @@ -170,6 +171,7 @@ SYM_CODE_START(kvm_exc_entry) >>>>       /* restore per cpu register */ >>>>       ld.d    u0, a2, KVM_ARCH_HPERCPU >>>>       addi.d    sp, sp, -PT_SIZE >>>> +    UNWIND_HINT_REGS >>>>         /* Prepare handle exception */ >>>>       or    a0, s0, zero >>>> @@ -200,7 +202,7 @@ ret_to_host: >>>>       jr      ra >>>>     SYM_CODE_END(kvm_exc_entry) >>>> -EXPORT_SYMBOL(kvm_exc_entry) >>>> +EXPORT_SYMBOL_FOR_KVM(kvm_exc_entry) >>>>     /* >>>>    * int kvm_enter_guest(struct kvm_run *run, struct kvm_vcpu *vcpu) >>>> @@ -215,6 +217,14 @@ SYM_FUNC_START(kvm_enter_guest) >>>>       /* Save host GPRs */ >>>>       kvm_save_host_gpr a2 >>>>   +    /* >>>> +     * The csr_era member variable of the pt_regs structure is >>>> required >>>> +     * for unwinding orc to perform stack traceback, so we need to put >>>> +     * pc into csr_era member variable here. >>>> +     */ >>>> +    pcaddi    t0, 0 >>>> +    st.d    t0, a2, PT_ERA >>> Hi, Xianglai, >>> >>> It should use `SYM_CODE_START` to mark the `kvm_enter_guest` rather than >>> `SYM_FUNC_START`, since the `SYM_FUNC_START` is used to mark "C-likely" >>> asm functionw. >> >> Ok, I will use SYM_CODE_START to mark kvm_enter_guest in the next >> version. >> >>> I guess the kvm_enter_guest is something like exception >>> handler becuase the last instruction is "ertn". So usually it should >>> mark UNWIND_HINT_REGS where can find last frame info by "$sp". >>> However, all info is store to "$a2", this mark should be >>>   `UNWIND_HINT sp_reg=ORC_REG_A2(???) type=UNWIND_HINT_TYPE_REGS`. >>> I don't konw why save this function internal PC here by `pcaddi t0, 0`, >>> and I think it is no meaning(, for exception handler, they save last PC >>> by read CSR.ERA). The `kvm_enter_guest` saves registers by >>> "$a2"("$sp" - PT_REGS) beyond stack ("$sp"), it is dangerous if IE >>> is enable. So I wonder if there is really a stacktrace through this >>> function? >>> >> The stack backtracking issue in switch.S is rather complex because it >> involves the switching between cpu root-mode and guest-mode: >> Real stack backtracking should be divided into two parts: >> part 1: >>     [<0>] kvm_enter_guest+0x38/0x11c >>     [<0>] kvm_arch_vcpu_ioctl_run+0x26c/0x498 [kvm] >>     [<0>] kvm_vcpu_ioctl+0x200/0xcf8 [kvm] >>     [<0>] sys_ioctl+0x498/0xf00 >>     [<0>] do_syscall+0x98/0x1d0 >>     [<0>] handle_syscall+0xb8/0x158 >> >> part 2: >>     [<0>] kvm_vcpu_block+0x88/0x120 [kvm] >>     [<0>] kvm_vcpu_halt+0x68/0x580 [kvm] >>     [<0>] kvm_emu_idle+0xd4/0xf0 [kvm] >>     [<0>] kvm_handle_gspr+0x7c/0x700 [kvm] >>     [<0>] kvm_handle_exit+0x160/0x270 [kvm] >>     [<0>] kvm_exc_entry+0x104/0x1e4 >> >> >> In "part 1", after executing kvm_enter_guest, the cpu switches from >> root-mode to guest-mode. >> In this case, stack backtracking is indeed very rare. >> >> In "part 2", the cpu switches from the guest-mode to the root-mode, >> and most of the stack backtracking occurs during this phase. >> >> To obtain the longest call chain, we save pc in kvm_enter_guest to >> pt_regs.csr_era, >> and after restoring the sp of the root-mode cpu in kvm_exc_entry, >> The ORC entry was re-established using "UNWIND_HINT_REGS", >>  and then we obtained the following stack backtrace as we wanted: >> >>     [<0>] kvm_vcpu_block+0x88/0x120 [kvm] >>     [<0>] kvm_vcpu_halt+0x68/0x580 [kvm] >>     [<0>] kvm_emu_idle+0xd4/0xf0 [kvm] >>     [<0>] kvm_handle_gspr+0x7c/0x700 [kvm] >>     [<0>] kvm_handle_exit+0x160/0x270 [kvm] >>     [<0>] kvm_exc_entry+0x104/0x1e4 > I found this might be a coincidence—correct behavior due to the incorrect > UNWIND_HINT_REGS mark and unusual stack adjustment. > > First, the kvm_enter_guest contains only a single branch instruction, ertn. > It hardware-jump to the CSR.ERA address directly, jump into kvm_exc_entry. > > At this point, the stack layout looks like this: > ------------------------------- >   frame from call to `kvm_enter_guest` > -------------------------------  <- $sp >   PT_REGS > -------------------------------  <- $a2 > > Then kvm_exc_entry adjust stack without save any register (e.g. $ra, $sp) > but still marked UNWIND_HINT_REGS. > After the adjustment: > ------------------------------- >   frame from call to `kvm_enter_guest` > ------------------------------- >   PT_REGS > -------------------------------  <- $a2, new $sp > > During unwinding, when the unwinder reaches kvm_exc_entry, > it meets the mark of PT_REGS and correctly recovers >  pc = regs.csr_era, sp = regs.sp, ra = regs.ra > > a) Can we avoid "ertn" rather than `jr reg (or jirl ra, reg, 0)` > instead, like call? The kvm_exc_entry cannot back to kvm_enter_guest > if we use "ertn", so should the kvm_enter_guest appear on the stacktrace? > > b) Can we adjust $sp before entering kvm_exc_entry? Then we can mark > UNWIND_HINT_REGS at the beginning of kvm_exc_entry, which something > like ret_from_kernel_thread_asm. Talk is cheap. Could you show your code? Regards Bibo Mao > >> [<0>] kvm_enter_guest+0x38/0x11c >>     [<0>] kvm_arch_vcpu_ioctl_run+0x26c/0x498 [kvm] >>     [<0>] kvm_vcpu_ioctl+0x200/0xcf8 [kvm] >>     [<0>] sys_ioctl+0x498/0xf00 >>     [<0>] do_syscall+0x98/0x1d0 >>     [<0>] handle_syscall+0xb8/0x158 >> >> Doing so is equivalent to ignoring the details of the cpu root-mode >> and guest-mode switching. >> About what you said in the IE enable phase is dangerous, >> interrupts are always off during the cpu root-mode and guest-mode >> switching in kvm_enter_guest and kvm_exc_entry. >> >