* Re: Hang on reboot in multi-core FreeBSD guest on Linux KVM host with Intel Sierra Forest CPU [not found] ` <CAMGffE=P5HJkJxh2mj3c_oh6busFKYb0TGuhAc36toc5_uD72w@mail.gmail.com> @ 2025-10-08 15:55 ` Sean Christopherson 2025-10-09 3:44 ` Jinpu Wang 0 siblings, 1 reply; 6+ messages in thread From: Sean Christopherson @ 2025-10-08 15:55 UTC (permalink / raw) To: Jinpu Wang; +Cc: fanwenyi0529, kvm, Paolo Bonzini, Xiaoyao Li, linux-kernel Trimmed Cc: to drop people from the original thread. In the future, just start a new bug report. Piggybacking a 10 year old bug just because the symptoms are similar does more harm than good. Whatever the old thread was chasing was already fixed, _10 years_ ago; they were just trying to identy exactly what commit fixed the problem. I.e. whatever they were chasing _can't_ be the same root cause, because even if it's literally the same code bug, it would require a code change and thus a regression between v4.0 and v6.1. On Wed, Oct 08, 2025, Jinpu Wang wrote: > On Wed, Oct 8, 2025 at 2:44 PM Jack Wang <jinpu.wang@ionos.com> wrote: > > Sorry for bump this old thread, we hit same issue on Intel Sierra Forest > > machines with LTS kernel 6.1/6.12, maybe KVM comunity could help fix it. Are there any host kernels that _do_ work? E.g. have you tried a bleeding edge host kernel? > > ### **[BUG] Hang on FreeBSD Guest Reboot under KVM on Intel SierraForest (Xeon 6710E)** > > > > **Summary:** > > Multi-cores FreeBSD guests hang during reboot under KVM on systems with > > Intel(R) Xeon(R) 6710E (SierraForest). The issue is fully reproducible with > > APICv enabled and disappears when disabling APICv (`enable_apicv=N`). The > > same configuration works correctly on Ice Lake (Xeon Gold 6338). Does Sierra Forest have IPI virtualization? If so, you could try running with APICv enabled, but enable_ipiv=false to specifically disable IPI virtualization. ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Hang on reboot in multi-core FreeBSD guest on Linux KVM host with Intel Sierra Forest CPU 2025-10-08 15:55 ` Hang on reboot in multi-core FreeBSD guest on Linux KVM host with Intel Sierra Forest CPU Sean Christopherson @ 2025-10-09 3:44 ` Jinpu Wang 2025-10-09 11:21 ` Jinpu Wang 0 siblings, 1 reply; 6+ messages in thread From: Jinpu Wang @ 2025-10-09 3:44 UTC (permalink / raw) To: Sean Christopherson Cc: fanwenyi0529, kvm, Paolo Bonzini, Xiaoyao Li, linux-kernel Hi Sean, On Wed, Oct 8, 2025 at 5:55 PM Sean Christopherson <seanjc@google.com> wrote: > > Trimmed Cc: to drop people from the original thread. In the future, just start > a new bug report. Piggybacking a 10 year old bug just because the symptoms are > similar does more harm than good. Whatever the old thread was chasing was already > fixed, _10 years_ ago; they were just trying to identy exactly what commit fixed > the problem. I.e. whatever they were chasing _can't_ be the same root cause, > because even if it's literally the same code bug, it would require a code change > and thus a regression between v4.0 and v6.1. Thx for the reply, it makes sense. I will remember this next time. > > On Wed, Oct 08, 2025, Jinpu Wang wrote: > > On Wed, Oct 8, 2025 at 2:44 PM Jack Wang <jinpu.wang@ionos.com> wrote: > > > Sorry for bump this old thread, we hit same issue on Intel Sierra Forest > > > machines with LTS kernel 6.1/6.12, maybe KVM comunity could help fix it. > > Are there any host kernels that _do_ work? E.g. have you tried a bleeding edge > host kernel? I will try linus/master today. > > > > ### **[BUG] Hang on FreeBSD Guest Reboot under KVM on Intel SierraForest (Xeon 6710E)** > > > > > > **Summary:** > > > Multi-cores FreeBSD guests hang during reboot under KVM on systems with > > > Intel(R) Xeon(R) 6710E (SierraForest). The issue is fully reproducible with > > > APICv enabled and disappears when disabling APICv (`enable_apicv=N`). The > > > same configuration works correctly on Ice Lake (Xeon Gold 6338). > > Does Sierra Forest have IPI virtualization? If so, you could try running with > APICv enabled, but enable_ipiv=false to specifically disable IPI virtualization. Yes, it does: $ grep . /sys/module/kvm_intel/parameters/* /sys/module/kvm_intel/parameters/allow_smaller_maxphyaddr:N /sys/module/kvm_intel/parameters/dump_invalid_vmcs:N /sys/module/kvm_intel/parameters/emulate_invalid_guest_state:Y /sys/module/kvm_intel/parameters/enable_apicv:Y /sys/module/kvm_intel/parameters/enable_ipiv:Y /sys/module/kvm_intel/parameters/enable_shadow_vmcs:Y /sys/module/kvm_intel/parameters/ept:Y /sys/module/kvm_intel/parameters/eptad:Y /sys/module/kvm_intel/parameters/error_on_inconsistent_vmcs_config:Y /sys/module/kvm_intel/parameters/fasteoi:Y /sys/module/kvm_intel/parameters/flexpriority:Y /sys/module/kvm_intel/parameters/nested:Y /sys/module/kvm_intel/parameters/nested_early_check:N /sys/module/kvm_intel/parameters/ple_gap:128 /sys/module/kvm_intel/parameters/ple_window:4096 /sys/module/kvm_intel/parameters/ple_window_grow:2 /sys/module/kvm_intel/parameters/ple_window_max:4294967295 /sys/module/kvm_intel/parameters/ple_window_shrink:0 /sys/module/kvm_intel/parameters/pml:Y /sys/module/kvm_intel/parameters/preemption_timer:Y /sys/module/kvm_intel/parameters/sgx:N /sys/module/kvm_intel/parameters/unrestricted_guest:Y /sys/module/kvm_intel/parameters/vmentry_l1d_flush:not required /sys/module/kvm_intel/parameters/vnmi:Y /sys/module/kvm_intel/parameters/vpid:Y I tried to disable ipiv, but it doesn't help. freebsd hang on reboot. sudo modprobe -r kvm_intel sudo modprobe kvm_intel enable_ipiv=N /sys/module/kvm_intel/parameters/enable_ipiv:N Thx! ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Hang on reboot in multi-core FreeBSD guest on Linux KVM host with Intel Sierra Forest CPU 2025-10-09 3:44 ` Jinpu Wang @ 2025-10-09 11:21 ` Jinpu Wang 2025-10-09 12:36 ` Jinpu Wang 0 siblings, 1 reply; 6+ messages in thread From: Jinpu Wang @ 2025-10-09 11:21 UTC (permalink / raw) To: Sean Christopherson Cc: fanwenyi0529, kvm, Paolo Bonzini, Xiaoyao Li, linux-kernel, vkuznets On Thu, Oct 9, 2025 at 5:44 AM Jinpu Wang <jinpu.wang@ionos.com> wrote: > > Hi Sean, > > On Wed, Oct 8, 2025 at 5:55 PM Sean Christopherson <seanjc@google.com> wrote: > > > > Trimmed Cc: to drop people from the original thread. In the future, just start > > a new bug report. Piggybacking a 10 year old bug just because the symptoms are > > similar does more harm than good. Whatever the old thread was chasing was already > > fixed, _10 years_ ago; they were just trying to identy exactly what commit fixed > > the problem. I.e. whatever they were chasing _can't_ be the same root cause, > > because even if it's literally the same code bug, it would require a code change > > and thus a regression between v4.0 and v6.1. > Thx for the reply, it makes sense. I will remember this next time. > > > > On Wed, Oct 08, 2025, Jinpu Wang wrote: > > > On Wed, Oct 8, 2025 at 2:44 PM Jack Wang <jinpu.wang@ionos.com> wrote: > > > > Sorry for bump this old thread, we hit same issue on Intel Sierra Forest > > > > machines with LTS kernel 6.1/6.12, maybe KVM comunity could help fix it. > > > > Are there any host kernels that _do_ work? E.g. have you tried a bleeding edge > > host kernel? > I will try linus/master today. > > > > > > ### **[BUG] Hang on FreeBSD Guest Reboot under KVM on Intel SierraForest (Xeon 6710E)** > > > > > > > > **Summary:** > > > > Multi-cores FreeBSD guests hang during reboot under KVM on systems with > > > > Intel(R) Xeon(R) 6710E (SierraForest). The issue is fully reproducible with > > > > APICv enabled and disappears when disabling APICv (`enable_apicv=N`). The > > > > same configuration works correctly on Ice Lake (Xeon Gold 6338). > > > > Does Sierra Forest have IPI virtualization? If so, you could try running with > > APICv enabled, but enable_ipiv=false to specifically disable IPI virtualization. > Yes, it does: > $ grep . /sys/module/kvm_intel/parameters/* > /sys/module/kvm_intel/parameters/allow_smaller_maxphyaddr:N > /sys/module/kvm_intel/parameters/dump_invalid_vmcs:N > /sys/module/kvm_intel/parameters/emulate_invalid_guest_state:Y > /sys/module/kvm_intel/parameters/enable_apicv:Y > /sys/module/kvm_intel/parameters/enable_ipiv:Y > /sys/module/kvm_intel/parameters/enable_shadow_vmcs:Y > /sys/module/kvm_intel/parameters/ept:Y > /sys/module/kvm_intel/parameters/eptad:Y > /sys/module/kvm_intel/parameters/error_on_inconsistent_vmcs_config:Y > /sys/module/kvm_intel/parameters/fasteoi:Y > /sys/module/kvm_intel/parameters/flexpriority:Y > /sys/module/kvm_intel/parameters/nested:Y > /sys/module/kvm_intel/parameters/nested_early_check:N > /sys/module/kvm_intel/parameters/ple_gap:128 > /sys/module/kvm_intel/parameters/ple_window:4096 > /sys/module/kvm_intel/parameters/ple_window_grow:2 > /sys/module/kvm_intel/parameters/ple_window_max:4294967295 > /sys/module/kvm_intel/parameters/ple_window_shrink:0 > /sys/module/kvm_intel/parameters/pml:Y > /sys/module/kvm_intel/parameters/preemption_timer:Y > /sys/module/kvm_intel/parameters/sgx:N > /sys/module/kvm_intel/parameters/unrestricted_guest:Y > /sys/module/kvm_intel/parameters/vmentry_l1d_flush:not required > /sys/module/kvm_intel/parameters/vnmi:Y > /sys/module/kvm_intel/parameters/vpid:Y > > I tried to disable ipiv, but it doesn't help. freebsd hang on reboot. > sudo modprobe -r kvm_intel > sudo modprobe kvm_intel enable_ipiv=N > /sys/module/kvm_intel/parameters/enable_ipiv:N > > Thx! +cc Vitaly Sorry, I missed one detail, we are use hyper-V enlightment features: "+hv-relaxed,+hv-vapic,+hv-time,+hv-runtime,hv-spinlocks=0x1fff,+hv-vpindex,+hv-synic,+hv-stimer,+hv-tlbflush,hv-ipi." did a lot tests with different features, and looks the hang is related to +hv-synic,+hv-stimer. hv-synic seems the key which causes boot hang of Freebsd 14. But the problem seems fixed with FreeBSD 15? I guess it's this fix: https://github.com/systemd/systemd/issues/28001 Seems it's a bug from freebsd side, rather than on kvm side to me, but I'm puzzled by disable apicv helps? Thx! ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Hang on reboot in multi-core FreeBSD guest on Linux KVM host with Intel Sierra Forest CPU 2025-10-09 11:21 ` Jinpu Wang @ 2025-10-09 12:36 ` Jinpu Wang 2025-10-09 12:52 ` Vitaly Kuznetsov 0 siblings, 1 reply; 6+ messages in thread From: Jinpu Wang @ 2025-10-09 12:36 UTC (permalink / raw) To: Sean Christopherson Cc: fanwenyi0529, kvm, Paolo Bonzini, Xiaoyao Li, linux-kernel, vkuznets On Thu, Oct 9, 2025 at 1:21 PM Jinpu Wang <jinpu.wang@ionos.com> wrote: > > On Thu, Oct 9, 2025 at 5:44 AM Jinpu Wang <jinpu.wang@ionos.com> wrote: > > > > Hi Sean, > > > > On Wed, Oct 8, 2025 at 5:55 PM Sean Christopherson <seanjc@google.com> wrote: > > > > > > Trimmed Cc: to drop people from the original thread. In the future, just start > > > a new bug report. Piggybacking a 10 year old bug just because the symptoms are > > > similar does more harm than good. Whatever the old thread was chasing was already > > > fixed, _10 years_ ago; they were just trying to identy exactly what commit fixed > > > the problem. I.e. whatever they were chasing _can't_ be the same root cause, > > > because even if it's literally the same code bug, it would require a code change > > > and thus a regression between v4.0 and v6.1. > > Thx for the reply, it makes sense. I will remember this next time. > > > > > > On Wed, Oct 08, 2025, Jinpu Wang wrote: > > > > On Wed, Oct 8, 2025 at 2:44 PM Jack Wang <jinpu.wang@ionos.com> wrote: > > > > > Sorry for bump this old thread, we hit same issue on Intel Sierra Forest > > > > > machines with LTS kernel 6.1/6.12, maybe KVM comunity could help fix it. > > > > > > Are there any host kernels that _do_ work? E.g. have you tried a bleeding edge > > > host kernel? > > I will try linus/master today. > > > > > > > > ### **[BUG] Hang on FreeBSD Guest Reboot under KVM on Intel SierraForest (Xeon 6710E)** > > > > > > > > > > **Summary:** > > > > > Multi-cores FreeBSD guests hang during reboot under KVM on systems with > > > > > Intel(R) Xeon(R) 6710E (SierraForest). The issue is fully reproducible with > > > > > APICv enabled and disappears when disabling APICv (`enable_apicv=N`). The > > > > > same configuration works correctly on Ice Lake (Xeon Gold 6338). > > > > > > Does Sierra Forest have IPI virtualization? If so, you could try running with > > > APICv enabled, but enable_ipiv=false to specifically disable IPI virtualization. > > Yes, it does: > > $ grep . /sys/module/kvm_intel/parameters/* > > /sys/module/kvm_intel/parameters/allow_smaller_maxphyaddr:N > > /sys/module/kvm_intel/parameters/dump_invalid_vmcs:N > > /sys/module/kvm_intel/parameters/emulate_invalid_guest_state:Y > > /sys/module/kvm_intel/parameters/enable_apicv:Y > > /sys/module/kvm_intel/parameters/enable_ipiv:Y > > /sys/module/kvm_intel/parameters/enable_shadow_vmcs:Y > > /sys/module/kvm_intel/parameters/ept:Y > > /sys/module/kvm_intel/parameters/eptad:Y > > /sys/module/kvm_intel/parameters/error_on_inconsistent_vmcs_config:Y > > /sys/module/kvm_intel/parameters/fasteoi:Y > > /sys/module/kvm_intel/parameters/flexpriority:Y > > /sys/module/kvm_intel/parameters/nested:Y > > /sys/module/kvm_intel/parameters/nested_early_check:N > > /sys/module/kvm_intel/parameters/ple_gap:128 > > /sys/module/kvm_intel/parameters/ple_window:4096 > > /sys/module/kvm_intel/parameters/ple_window_grow:2 > > /sys/module/kvm_intel/parameters/ple_window_max:4294967295 > > /sys/module/kvm_intel/parameters/ple_window_shrink:0 > > /sys/module/kvm_intel/parameters/pml:Y > > /sys/module/kvm_intel/parameters/preemption_timer:Y > > /sys/module/kvm_intel/parameters/sgx:N > > /sys/module/kvm_intel/parameters/unrestricted_guest:Y > > /sys/module/kvm_intel/parameters/vmentry_l1d_flush:not required > > /sys/module/kvm_intel/parameters/vnmi:Y > > /sys/module/kvm_intel/parameters/vpid:Y > > > > I tried to disable ipiv, but it doesn't help. freebsd hang on reboot. > > sudo modprobe -r kvm_intel > > sudo modprobe kvm_intel enable_ipiv=N > > /sys/module/kvm_intel/parameters/enable_ipiv:N > > > > Thx! > +cc Vitaly > Sorry, I missed one detail, we are use hyper-V enlightment features: > "+hv-relaxed,+hv-vapic,+hv-time,+hv-runtime,hv-spinlocks=0x1fff,+hv-vpindex,+hv-synic,+hv-stimer,+hv-tlbflush,hv-ipi." > > did a lot tests with different features, and looks the hang is related > to +hv-synic,+hv-stimer. hv-synic seems the key which causes boot > hang of Freebsd 14. > > But the problem seems fixed with FreeBSD 15? I guess it's this fix: https://reviews.freebsd.org/D43508 > > Seems it's a bug from freebsd side, rather than on kvm side to me, but > I'm puzzled by disable apicv helps? > > Thx! ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Hang on reboot in multi-core FreeBSD guest on Linux KVM host with Intel Sierra Forest CPU 2025-10-09 12:36 ` Jinpu Wang @ 2025-10-09 12:52 ` Vitaly Kuznetsov 2025-10-10 5:04 ` Jinpu Wang 0 siblings, 1 reply; 6+ messages in thread From: Vitaly Kuznetsov @ 2025-10-09 12:52 UTC (permalink / raw) To: Jinpu Wang, Sean Christopherson Cc: fanwenyi0529, kvm, Paolo Bonzini, Xiaoyao Li, linux-kernel Jinpu Wang <jinpu.wang@ionos.com> writes: > On Thu, Oct 9, 2025 at 1:21 PM Jinpu Wang <jinpu.wang@ionos.com> wrote: >> >> On Thu, Oct 9, 2025 at 5:44 AM Jinpu Wang <jinpu.wang@ionos.com> wrote: >> > >> > Hi Sean, >> > >> > On Wed, Oct 8, 2025 at 5:55 PM Sean Christopherson <seanjc@google.com> wrote: >> > > >> > > Trimmed Cc: to drop people from the original thread. In the future, just start >> > > a new bug report. Piggybacking a 10 year old bug just because the symptoms are >> > > similar does more harm than good. Whatever the old thread was chasing was already >> > > fixed, _10 years_ ago; they were just trying to identy exactly what commit fixed >> > > the problem. I.e. whatever they were chasing _can't_ be the same root cause, >> > > because even if it's literally the same code bug, it would require a code change >> > > and thus a regression between v4.0 and v6.1. >> > Thx for the reply, it makes sense. I will remember this next time. >> > > >> > > On Wed, Oct 08, 2025, Jinpu Wang wrote: >> > > > On Wed, Oct 8, 2025 at 2:44 PM Jack Wang <jinpu.wang@ionos.com> wrote: >> > > > > Sorry for bump this old thread, we hit same issue on Intel Sierra Forest >> > > > > machines with LTS kernel 6.1/6.12, maybe KVM comunity could help fix it. >> > > >> > > Are there any host kernels that _do_ work? E.g. have you tried a bleeding edge >> > > host kernel? >> > I will try linus/master today. >> > > >> > > > > ### **[BUG] Hang on FreeBSD Guest Reboot under KVM on Intel SierraForest (Xeon 6710E)** >> > > > > >> > > > > **Summary:** >> > > > > Multi-cores FreeBSD guests hang during reboot under KVM on systems with >> > > > > Intel(R) Xeon(R) 6710E (SierraForest). The issue is fully reproducible with >> > > > > APICv enabled and disappears when disabling APICv (`enable_apicv=N`). The >> > > > > same configuration works correctly on Ice Lake (Xeon Gold 6338). >> > > >> > > Does Sierra Forest have IPI virtualization? If so, you could try running with >> > > APICv enabled, but enable_ipiv=false to specifically disable IPI virtualization. >> > Yes, it does: >> > $ grep . /sys/module/kvm_intel/parameters/* >> > /sys/module/kvm_intel/parameters/allow_smaller_maxphyaddr:N >> > /sys/module/kvm_intel/parameters/dump_invalid_vmcs:N >> > /sys/module/kvm_intel/parameters/emulate_invalid_guest_state:Y >> > /sys/module/kvm_intel/parameters/enable_apicv:Y >> > /sys/module/kvm_intel/parameters/enable_ipiv:Y >> > /sys/module/kvm_intel/parameters/enable_shadow_vmcs:Y >> > /sys/module/kvm_intel/parameters/ept:Y >> > /sys/module/kvm_intel/parameters/eptad:Y >> > /sys/module/kvm_intel/parameters/error_on_inconsistent_vmcs_config:Y >> > /sys/module/kvm_intel/parameters/fasteoi:Y >> > /sys/module/kvm_intel/parameters/flexpriority:Y >> > /sys/module/kvm_intel/parameters/nested:Y >> > /sys/module/kvm_intel/parameters/nested_early_check:N >> > /sys/module/kvm_intel/parameters/ple_gap:128 >> > /sys/module/kvm_intel/parameters/ple_window:4096 >> > /sys/module/kvm_intel/parameters/ple_window_grow:2 >> > /sys/module/kvm_intel/parameters/ple_window_max:4294967295 >> > /sys/module/kvm_intel/parameters/ple_window_shrink:0 >> > /sys/module/kvm_intel/parameters/pml:Y >> > /sys/module/kvm_intel/parameters/preemption_timer:Y >> > /sys/module/kvm_intel/parameters/sgx:N >> > /sys/module/kvm_intel/parameters/unrestricted_guest:Y >> > /sys/module/kvm_intel/parameters/vmentry_l1d_flush:not required >> > /sys/module/kvm_intel/parameters/vnmi:Y >> > /sys/module/kvm_intel/parameters/vpid:Y >> > >> > I tried to disable ipiv, but it doesn't help. freebsd hang on reboot. >> > sudo modprobe -r kvm_intel >> > sudo modprobe kvm_intel enable_ipiv=N >> > /sys/module/kvm_intel/parameters/enable_ipiv:N >> > >> > Thx! >> +cc Vitaly >> Sorry, I missed one detail, we are use hyper-V enlightment features: >> "+hv-relaxed,+hv-vapic,+hv-time,+hv-runtime,hv-spinlocks=0x1fff,+hv-vpindex,+hv-synic,+hv-stimer,+hv-tlbflush,hv-ipi." >> >> did a lot tests with different features, and looks the hang is related >> to +hv-synic,+hv-stimer. hv-synic seems the key which causes boot >> hang of Freebsd 14. >> >> But the problem seems fixed with FreeBSD 15? I guess it's this fix: > https://reviews.freebsd.org/D43508 > >> >> Seems it's a bug from freebsd side, rather than on kvm side to me, but >> I'm puzzled by disable apicv helps? In theory, FreeBSD should work well even if KVM is misdetected as genuine Hyper-V. Apparently, our emulation is not 1:1 and there are subtle differences which cause the hang. I did not look at FreeBSD code at all but my wild guess is that SynIC/stimer are not disabled properly upon reboot and this causes the problem. If we somehow manage to find how genuine Hyper-V's behavior is different, it would make sense to update KVM/QEMU to match. -- Vitaly ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Hang on reboot in multi-core FreeBSD guest on Linux KVM host with Intel Sierra Forest CPU 2025-10-09 12:52 ` Vitaly Kuznetsov @ 2025-10-10 5:04 ` Jinpu Wang 0 siblings, 0 replies; 6+ messages in thread From: Jinpu Wang @ 2025-10-10 5:04 UTC (permalink / raw) To: Vitaly Kuznetsov Cc: Sean Christopherson, fanwenyi0529, kvm, Paolo Bonzini, Xiaoyao Li, linux-kernel On Thu, Oct 9, 2025 at 2:53 PM Vitaly Kuznetsov <vkuznets@redhat.com> wrote: > > Jinpu Wang <jinpu.wang@ionos.com> writes: > > > On Thu, Oct 9, 2025 at 1:21 PM Jinpu Wang <jinpu.wang@ionos.com> wrote: > >> > >> On Thu, Oct 9, 2025 at 5:44 AM Jinpu Wang <jinpu.wang@ionos.com> wrote: > >> > > >> > Hi Sean, > >> > > >> > On Wed, Oct 8, 2025 at 5:55 PM Sean Christopherson <seanjc@google.com> wrote: > >> > > > >> > > Trimmed Cc: to drop people from the original thread. In the future, just start > >> > > a new bug report. Piggybacking a 10 year old bug just because the symptoms are > >> > > similar does more harm than good. Whatever the old thread was chasing was already > >> > > fixed, _10 years_ ago; they were just trying to identy exactly what commit fixed > >> > > the problem. I.e. whatever they were chasing _can't_ be the same root cause, > >> > > because even if it's literally the same code bug, it would require a code change > >> > > and thus a regression between v4.0 and v6.1. > >> > Thx for the reply, it makes sense. I will remember this next time. > >> > > > >> > > On Wed, Oct 08, 2025, Jinpu Wang wrote: > >> > > > On Wed, Oct 8, 2025 at 2:44 PM Jack Wang <jinpu.wang@ionos.com> wrote: > >> > > > > Sorry for bump this old thread, we hit same issue on Intel Sierra Forest > >> > > > > machines with LTS kernel 6.1/6.12, maybe KVM comunity could help fix it. > >> > > > >> > > Are there any host kernels that _do_ work? E.g. have you tried a bleeding edge > >> > > host kernel? > >> > I will try linus/master today. > >> > > > >> > > > > ### **[BUG] Hang on FreeBSD Guest Reboot under KVM on Intel SierraForest (Xeon 6710E)** > >> > > > > > >> > > > > **Summary:** > >> > > > > Multi-cores FreeBSD guests hang during reboot under KVM on systems with > >> > > > > Intel(R) Xeon(R) 6710E (SierraForest). The issue is fully reproducible with > >> > > > > APICv enabled and disappears when disabling APICv (`enable_apicv=N`). The > >> > > > > same configuration works correctly on Ice Lake (Xeon Gold 6338). > >> > > > >> > > Does Sierra Forest have IPI virtualization? If so, you could try running with > >> > > APICv enabled, but enable_ipiv=false to specifically disable IPI virtualization. > >> > Yes, it does: > >> > $ grep . /sys/module/kvm_intel/parameters/* > >> > /sys/module/kvm_intel/parameters/allow_smaller_maxphyaddr:N > >> > /sys/module/kvm_intel/parameters/dump_invalid_vmcs:N > >> > /sys/module/kvm_intel/parameters/emulate_invalid_guest_state:Y > >> > /sys/module/kvm_intel/parameters/enable_apicv:Y > >> > /sys/module/kvm_intel/parameters/enable_ipiv:Y > >> > /sys/module/kvm_intel/parameters/enable_shadow_vmcs:Y > >> > /sys/module/kvm_intel/parameters/ept:Y > >> > /sys/module/kvm_intel/parameters/eptad:Y > >> > /sys/module/kvm_intel/parameters/error_on_inconsistent_vmcs_config:Y > >> > /sys/module/kvm_intel/parameters/fasteoi:Y > >> > /sys/module/kvm_intel/parameters/flexpriority:Y > >> > /sys/module/kvm_intel/parameters/nested:Y > >> > /sys/module/kvm_intel/parameters/nested_early_check:N > >> > /sys/module/kvm_intel/parameters/ple_gap:128 > >> > /sys/module/kvm_intel/parameters/ple_window:4096 > >> > /sys/module/kvm_intel/parameters/ple_window_grow:2 > >> > /sys/module/kvm_intel/parameters/ple_window_max:4294967295 > >> > /sys/module/kvm_intel/parameters/ple_window_shrink:0 > >> > /sys/module/kvm_intel/parameters/pml:Y > >> > /sys/module/kvm_intel/parameters/preemption_timer:Y > >> > /sys/module/kvm_intel/parameters/sgx:N > >> > /sys/module/kvm_intel/parameters/unrestricted_guest:Y > >> > /sys/module/kvm_intel/parameters/vmentry_l1d_flush:not required > >> > /sys/module/kvm_intel/parameters/vnmi:Y > >> > /sys/module/kvm_intel/parameters/vpid:Y > >> > > >> > I tried to disable ipiv, but it doesn't help. freebsd hang on reboot. > >> > sudo modprobe -r kvm_intel > >> > sudo modprobe kvm_intel enable_ipiv=N > >> > /sys/module/kvm_intel/parameters/enable_ipiv:N > >> > > >> > Thx! > >> +cc Vitaly > >> Sorry, I missed one detail, we are use hyper-V enlightment features: > >> "+hv-relaxed,+hv-vapic,+hv-time,+hv-runtime,hv-spinlocks=0x1fff,+hv-vpindex,+hv-synic,+hv-stimer,+hv-tlbflush,hv-ipi." > >> > >> did a lot tests with different features, and looks the hang is related > >> to +hv-synic,+hv-stimer. hv-synic seems the key which causes boot > >> hang of Freebsd 14. > >> > >> But the problem seems fixed with FreeBSD 15? I guess it's this fix: > > https://reviews.freebsd.org/D43508 > > > >> > >> Seems it's a bug from freebsd side, rather than on kvm side to me, but > >> I'm puzzled by disable apicv helps? > > In theory, FreeBSD should work well even if KVM is misdetected as > genuine Hyper-V. Apparently, our emulation is not 1:1 and there are > subtle differences which cause the hang. I did not look at FreeBSD code > at all but my wild guess is that SynIC/stimer are not disabled properly > upon reboot and this causes the problem. If we somehow manage to find > how genuine Hyper-V's behavior is different, it would make sense to > update KVM/QEMU to match. > > -- > Vitaly Hi Vitaly, Thx for the reply, that's another view. I spent a few hours to build freebsd stable/14 with commit 6744fd8e7503 ("x86/cpu: improve hypervisor detection") applied, the bug is fixed, no boot hung, and boot is much faster ~ 40 seconds, as Freebsd 15, without the fix Freebsd takes 150 seconds if sucessfully boots, and often hang forever during reboot. Jinpu > ^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2025-10-10 5:04 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
[not found] <539FC243.2070906@redhat.com>
[not found] ` <20140617060500.GA20764@minantech.com>
[not found] ` <FFEF5F78-D9E6-4333-BC1A-78076C132CBF@jnielsen.net>
[not found] ` <6850B127-F16B-465F-BDDB-BA3F99B9E446@jnielsen.net>
[not found] ` <jpgioafjtxb.fsf@redhat.com>
[not found] ` <74412BDB-EF6F-4C20-84C8-C6EF3A25885C@jnielsen.net>
[not found] ` <558AD1B0.5060200@redhat.com>
[not found] ` <FAFB2BA9-E924-4E70-A84A-E5F2D97BC2F0@jnielsen.net>
[not found] ` <CACzj_yVTyescyWBRuA3MMCC0Ymg7TKF-+sCW1N+Xwfffvw_Wsg@mail.gmail.com>
[not found] ` <CAMGffE=P5HJkJxh2mj3c_oh6busFKYb0TGuhAc36toc5_uD72w@mail.gmail.com>
2025-10-08 15:55 ` Hang on reboot in multi-core FreeBSD guest on Linux KVM host with Intel Sierra Forest CPU Sean Christopherson
2025-10-09 3:44 ` Jinpu Wang
2025-10-09 11:21 ` Jinpu Wang
2025-10-09 12:36 ` Jinpu Wang
2025-10-09 12:52 ` Vitaly Kuznetsov
2025-10-10 5:04 ` Jinpu Wang
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox
Powered by JetHome