* Re: Hang on reboot in multi-core FreeBSD guest on Linux KVM host with Intel Sierra Forest CPU
[not found] ` <CAMGffE=P5HJkJxh2mj3c_oh6busFKYb0TGuhAc36toc5_uD72w@mail.gmail.com>
@ 2025-10-08 15:55 ` Sean Christopherson
2025-10-09 3:44 ` Jinpu Wang
0 siblings, 1 reply; 6+ messages in thread
From: Sean Christopherson @ 2025-10-08 15:55 UTC (permalink / raw)
To: Jinpu Wang; +Cc: fanwenyi0529, kvm, Paolo Bonzini, Xiaoyao Li, linux-kernel
Trimmed Cc: to drop people from the original thread. In the future, just start
a new bug report. Piggybacking a 10 year old bug just because the symptoms are
similar does more harm than good. Whatever the old thread was chasing was already
fixed, _10 years_ ago; they were just trying to identy exactly what commit fixed
the problem. I.e. whatever they were chasing _can't_ be the same root cause,
because even if it's literally the same code bug, it would require a code change
and thus a regression between v4.0 and v6.1.
On Wed, Oct 08, 2025, Jinpu Wang wrote:
> On Wed, Oct 8, 2025 at 2:44 PM Jack Wang <jinpu.wang@ionos.com> wrote:
> > Sorry for bump this old thread, we hit same issue on Intel Sierra Forest
> > machines with LTS kernel 6.1/6.12, maybe KVM comunity could help fix it.
Are there any host kernels that _do_ work? E.g. have you tried a bleeding edge
host kernel?
> > ### **[BUG] Hang on FreeBSD Guest Reboot under KVM on Intel SierraForest (Xeon 6710E)**
> >
> > **Summary:**
> > Multi-cores FreeBSD guests hang during reboot under KVM on systems with
> > Intel(R) Xeon(R) 6710E (SierraForest). The issue is fully reproducible with
> > APICv enabled and disappears when disabling APICv (`enable_apicv=N`). The
> > same configuration works correctly on Ice Lake (Xeon Gold 6338).
Does Sierra Forest have IPI virtualization? If so, you could try running with
APICv enabled, but enable_ipiv=false to specifically disable IPI virtualization.
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Hang on reboot in multi-core FreeBSD guest on Linux KVM host with Intel Sierra Forest CPU
2025-10-08 15:55 ` Hang on reboot in multi-core FreeBSD guest on Linux KVM host with Intel Sierra Forest CPU Sean Christopherson
@ 2025-10-09 3:44 ` Jinpu Wang
2025-10-09 11:21 ` Jinpu Wang
0 siblings, 1 reply; 6+ messages in thread
From: Jinpu Wang @ 2025-10-09 3:44 UTC (permalink / raw)
To: Sean Christopherson
Cc: fanwenyi0529, kvm, Paolo Bonzini, Xiaoyao Li, linux-kernel
Hi Sean,
On Wed, Oct 8, 2025 at 5:55 PM Sean Christopherson <seanjc@google.com> wrote:
>
> Trimmed Cc: to drop people from the original thread. In the future, just start
> a new bug report. Piggybacking a 10 year old bug just because the symptoms are
> similar does more harm than good. Whatever the old thread was chasing was already
> fixed, _10 years_ ago; they were just trying to identy exactly what commit fixed
> the problem. I.e. whatever they were chasing _can't_ be the same root cause,
> because even if it's literally the same code bug, it would require a code change
> and thus a regression between v4.0 and v6.1.
Thx for the reply, it makes sense. I will remember this next time.
>
> On Wed, Oct 08, 2025, Jinpu Wang wrote:
> > On Wed, Oct 8, 2025 at 2:44 PM Jack Wang <jinpu.wang@ionos.com> wrote:
> > > Sorry for bump this old thread, we hit same issue on Intel Sierra Forest
> > > machines with LTS kernel 6.1/6.12, maybe KVM comunity could help fix it.
>
> Are there any host kernels that _do_ work? E.g. have you tried a bleeding edge
> host kernel?
I will try linus/master today.
>
> > > ### **[BUG] Hang on FreeBSD Guest Reboot under KVM on Intel SierraForest (Xeon 6710E)**
> > >
> > > **Summary:**
> > > Multi-cores FreeBSD guests hang during reboot under KVM on systems with
> > > Intel(R) Xeon(R) 6710E (SierraForest). The issue is fully reproducible with
> > > APICv enabled and disappears when disabling APICv (`enable_apicv=N`). The
> > > same configuration works correctly on Ice Lake (Xeon Gold 6338).
>
> Does Sierra Forest have IPI virtualization? If so, you could try running with
> APICv enabled, but enable_ipiv=false to specifically disable IPI virtualization.
Yes, it does:
$ grep . /sys/module/kvm_intel/parameters/*
/sys/module/kvm_intel/parameters/allow_smaller_maxphyaddr:N
/sys/module/kvm_intel/parameters/dump_invalid_vmcs:N
/sys/module/kvm_intel/parameters/emulate_invalid_guest_state:Y
/sys/module/kvm_intel/parameters/enable_apicv:Y
/sys/module/kvm_intel/parameters/enable_ipiv:Y
/sys/module/kvm_intel/parameters/enable_shadow_vmcs:Y
/sys/module/kvm_intel/parameters/ept:Y
/sys/module/kvm_intel/parameters/eptad:Y
/sys/module/kvm_intel/parameters/error_on_inconsistent_vmcs_config:Y
/sys/module/kvm_intel/parameters/fasteoi:Y
/sys/module/kvm_intel/parameters/flexpriority:Y
/sys/module/kvm_intel/parameters/nested:Y
/sys/module/kvm_intel/parameters/nested_early_check:N
/sys/module/kvm_intel/parameters/ple_gap:128
/sys/module/kvm_intel/parameters/ple_window:4096
/sys/module/kvm_intel/parameters/ple_window_grow:2
/sys/module/kvm_intel/parameters/ple_window_max:4294967295
/sys/module/kvm_intel/parameters/ple_window_shrink:0
/sys/module/kvm_intel/parameters/pml:Y
/sys/module/kvm_intel/parameters/preemption_timer:Y
/sys/module/kvm_intel/parameters/sgx:N
/sys/module/kvm_intel/parameters/unrestricted_guest:Y
/sys/module/kvm_intel/parameters/vmentry_l1d_flush:not required
/sys/module/kvm_intel/parameters/vnmi:Y
/sys/module/kvm_intel/parameters/vpid:Y
I tried to disable ipiv, but it doesn't help. freebsd hang on reboot.
sudo modprobe -r kvm_intel
sudo modprobe kvm_intel enable_ipiv=N
/sys/module/kvm_intel/parameters/enable_ipiv:N
Thx!
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Hang on reboot in multi-core FreeBSD guest on Linux KVM host with Intel Sierra Forest CPU
2025-10-09 3:44 ` Jinpu Wang
@ 2025-10-09 11:21 ` Jinpu Wang
2025-10-09 12:36 ` Jinpu Wang
0 siblings, 1 reply; 6+ messages in thread
From: Jinpu Wang @ 2025-10-09 11:21 UTC (permalink / raw)
To: Sean Christopherson
Cc: fanwenyi0529, kvm, Paolo Bonzini, Xiaoyao Li, linux-kernel, vkuznets
On Thu, Oct 9, 2025 at 5:44 AM Jinpu Wang <jinpu.wang@ionos.com> wrote:
>
> Hi Sean,
>
> On Wed, Oct 8, 2025 at 5:55 PM Sean Christopherson <seanjc@google.com> wrote:
> >
> > Trimmed Cc: to drop people from the original thread. In the future, just start
> > a new bug report. Piggybacking a 10 year old bug just because the symptoms are
> > similar does more harm than good. Whatever the old thread was chasing was already
> > fixed, _10 years_ ago; they were just trying to identy exactly what commit fixed
> > the problem. I.e. whatever they were chasing _can't_ be the same root cause,
> > because even if it's literally the same code bug, it would require a code change
> > and thus a regression between v4.0 and v6.1.
> Thx for the reply, it makes sense. I will remember this next time.
> >
> > On Wed, Oct 08, 2025, Jinpu Wang wrote:
> > > On Wed, Oct 8, 2025 at 2:44 PM Jack Wang <jinpu.wang@ionos.com> wrote:
> > > > Sorry for bump this old thread, we hit same issue on Intel Sierra Forest
> > > > machines with LTS kernel 6.1/6.12, maybe KVM comunity could help fix it.
> >
> > Are there any host kernels that _do_ work? E.g. have you tried a bleeding edge
> > host kernel?
> I will try linus/master today.
> >
> > > > ### **[BUG] Hang on FreeBSD Guest Reboot under KVM on Intel SierraForest (Xeon 6710E)**
> > > >
> > > > **Summary:**
> > > > Multi-cores FreeBSD guests hang during reboot under KVM on systems with
> > > > Intel(R) Xeon(R) 6710E (SierraForest). The issue is fully reproducible with
> > > > APICv enabled and disappears when disabling APICv (`enable_apicv=N`). The
> > > > same configuration works correctly on Ice Lake (Xeon Gold 6338).
> >
> > Does Sierra Forest have IPI virtualization? If so, you could try running with
> > APICv enabled, but enable_ipiv=false to specifically disable IPI virtualization.
> Yes, it does:
> $ grep . /sys/module/kvm_intel/parameters/*
> /sys/module/kvm_intel/parameters/allow_smaller_maxphyaddr:N
> /sys/module/kvm_intel/parameters/dump_invalid_vmcs:N
> /sys/module/kvm_intel/parameters/emulate_invalid_guest_state:Y
> /sys/module/kvm_intel/parameters/enable_apicv:Y
> /sys/module/kvm_intel/parameters/enable_ipiv:Y
> /sys/module/kvm_intel/parameters/enable_shadow_vmcs:Y
> /sys/module/kvm_intel/parameters/ept:Y
> /sys/module/kvm_intel/parameters/eptad:Y
> /sys/module/kvm_intel/parameters/error_on_inconsistent_vmcs_config:Y
> /sys/module/kvm_intel/parameters/fasteoi:Y
> /sys/module/kvm_intel/parameters/flexpriority:Y
> /sys/module/kvm_intel/parameters/nested:Y
> /sys/module/kvm_intel/parameters/nested_early_check:N
> /sys/module/kvm_intel/parameters/ple_gap:128
> /sys/module/kvm_intel/parameters/ple_window:4096
> /sys/module/kvm_intel/parameters/ple_window_grow:2
> /sys/module/kvm_intel/parameters/ple_window_max:4294967295
> /sys/module/kvm_intel/parameters/ple_window_shrink:0
> /sys/module/kvm_intel/parameters/pml:Y
> /sys/module/kvm_intel/parameters/preemption_timer:Y
> /sys/module/kvm_intel/parameters/sgx:N
> /sys/module/kvm_intel/parameters/unrestricted_guest:Y
> /sys/module/kvm_intel/parameters/vmentry_l1d_flush:not required
> /sys/module/kvm_intel/parameters/vnmi:Y
> /sys/module/kvm_intel/parameters/vpid:Y
>
> I tried to disable ipiv, but it doesn't help. freebsd hang on reboot.
> sudo modprobe -r kvm_intel
> sudo modprobe kvm_intel enable_ipiv=N
> /sys/module/kvm_intel/parameters/enable_ipiv:N
>
> Thx!
+cc Vitaly
Sorry, I missed one detail, we are use hyper-V enlightment features:
"+hv-relaxed,+hv-vapic,+hv-time,+hv-runtime,hv-spinlocks=0x1fff,+hv-vpindex,+hv-synic,+hv-stimer,+hv-tlbflush,hv-ipi."
did a lot tests with different features, and looks the hang is related
to +hv-synic,+hv-stimer. hv-synic seems the key which causes boot
hang of Freebsd 14.
But the problem seems fixed with FreeBSD 15? I guess it's this fix:
https://github.com/systemd/systemd/issues/28001
Seems it's a bug from freebsd side, rather than on kvm side to me, but
I'm puzzled by disable apicv helps?
Thx!
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Hang on reboot in multi-core FreeBSD guest on Linux KVM host with Intel Sierra Forest CPU
2025-10-09 11:21 ` Jinpu Wang
@ 2025-10-09 12:36 ` Jinpu Wang
2025-10-09 12:52 ` Vitaly Kuznetsov
0 siblings, 1 reply; 6+ messages in thread
From: Jinpu Wang @ 2025-10-09 12:36 UTC (permalink / raw)
To: Sean Christopherson
Cc: fanwenyi0529, kvm, Paolo Bonzini, Xiaoyao Li, linux-kernel, vkuznets
On Thu, Oct 9, 2025 at 1:21 PM Jinpu Wang <jinpu.wang@ionos.com> wrote:
>
> On Thu, Oct 9, 2025 at 5:44 AM Jinpu Wang <jinpu.wang@ionos.com> wrote:
> >
> > Hi Sean,
> >
> > On Wed, Oct 8, 2025 at 5:55 PM Sean Christopherson <seanjc@google.com> wrote:
> > >
> > > Trimmed Cc: to drop people from the original thread. In the future, just start
> > > a new bug report. Piggybacking a 10 year old bug just because the symptoms are
> > > similar does more harm than good. Whatever the old thread was chasing was already
> > > fixed, _10 years_ ago; they were just trying to identy exactly what commit fixed
> > > the problem. I.e. whatever they were chasing _can't_ be the same root cause,
> > > because even if it's literally the same code bug, it would require a code change
> > > and thus a regression between v4.0 and v6.1.
> > Thx for the reply, it makes sense. I will remember this next time.
> > >
> > > On Wed, Oct 08, 2025, Jinpu Wang wrote:
> > > > On Wed, Oct 8, 2025 at 2:44 PM Jack Wang <jinpu.wang@ionos.com> wrote:
> > > > > Sorry for bump this old thread, we hit same issue on Intel Sierra Forest
> > > > > machines with LTS kernel 6.1/6.12, maybe KVM comunity could help fix it.
> > >
> > > Are there any host kernels that _do_ work? E.g. have you tried a bleeding edge
> > > host kernel?
> > I will try linus/master today.
> > >
> > > > > ### **[BUG] Hang on FreeBSD Guest Reboot under KVM on Intel SierraForest (Xeon 6710E)**
> > > > >
> > > > > **Summary:**
> > > > > Multi-cores FreeBSD guests hang during reboot under KVM on systems with
> > > > > Intel(R) Xeon(R) 6710E (SierraForest). The issue is fully reproducible with
> > > > > APICv enabled and disappears when disabling APICv (`enable_apicv=N`). The
> > > > > same configuration works correctly on Ice Lake (Xeon Gold 6338).
> > >
> > > Does Sierra Forest have IPI virtualization? If so, you could try running with
> > > APICv enabled, but enable_ipiv=false to specifically disable IPI virtualization.
> > Yes, it does:
> > $ grep . /sys/module/kvm_intel/parameters/*
> > /sys/module/kvm_intel/parameters/allow_smaller_maxphyaddr:N
> > /sys/module/kvm_intel/parameters/dump_invalid_vmcs:N
> > /sys/module/kvm_intel/parameters/emulate_invalid_guest_state:Y
> > /sys/module/kvm_intel/parameters/enable_apicv:Y
> > /sys/module/kvm_intel/parameters/enable_ipiv:Y
> > /sys/module/kvm_intel/parameters/enable_shadow_vmcs:Y
> > /sys/module/kvm_intel/parameters/ept:Y
> > /sys/module/kvm_intel/parameters/eptad:Y
> > /sys/module/kvm_intel/parameters/error_on_inconsistent_vmcs_config:Y
> > /sys/module/kvm_intel/parameters/fasteoi:Y
> > /sys/module/kvm_intel/parameters/flexpriority:Y
> > /sys/module/kvm_intel/parameters/nested:Y
> > /sys/module/kvm_intel/parameters/nested_early_check:N
> > /sys/module/kvm_intel/parameters/ple_gap:128
> > /sys/module/kvm_intel/parameters/ple_window:4096
> > /sys/module/kvm_intel/parameters/ple_window_grow:2
> > /sys/module/kvm_intel/parameters/ple_window_max:4294967295
> > /sys/module/kvm_intel/parameters/ple_window_shrink:0
> > /sys/module/kvm_intel/parameters/pml:Y
> > /sys/module/kvm_intel/parameters/preemption_timer:Y
> > /sys/module/kvm_intel/parameters/sgx:N
> > /sys/module/kvm_intel/parameters/unrestricted_guest:Y
> > /sys/module/kvm_intel/parameters/vmentry_l1d_flush:not required
> > /sys/module/kvm_intel/parameters/vnmi:Y
> > /sys/module/kvm_intel/parameters/vpid:Y
> >
> > I tried to disable ipiv, but it doesn't help. freebsd hang on reboot.
> > sudo modprobe -r kvm_intel
> > sudo modprobe kvm_intel enable_ipiv=N
> > /sys/module/kvm_intel/parameters/enable_ipiv:N
> >
> > Thx!
> +cc Vitaly
> Sorry, I missed one detail, we are use hyper-V enlightment features:
> "+hv-relaxed,+hv-vapic,+hv-time,+hv-runtime,hv-spinlocks=0x1fff,+hv-vpindex,+hv-synic,+hv-stimer,+hv-tlbflush,hv-ipi."
>
> did a lot tests with different features, and looks the hang is related
> to +hv-synic,+hv-stimer. hv-synic seems the key which causes boot
> hang of Freebsd 14.
>
> But the problem seems fixed with FreeBSD 15? I guess it's this fix:
https://reviews.freebsd.org/D43508
>
> Seems it's a bug from freebsd side, rather than on kvm side to me, but
> I'm puzzled by disable apicv helps?
>
> Thx!
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Hang on reboot in multi-core FreeBSD guest on Linux KVM host with Intel Sierra Forest CPU
2025-10-09 12:36 ` Jinpu Wang
@ 2025-10-09 12:52 ` Vitaly Kuznetsov
2025-10-10 5:04 ` Jinpu Wang
0 siblings, 1 reply; 6+ messages in thread
From: Vitaly Kuznetsov @ 2025-10-09 12:52 UTC (permalink / raw)
To: Jinpu Wang, Sean Christopherson
Cc: fanwenyi0529, kvm, Paolo Bonzini, Xiaoyao Li, linux-kernel
Jinpu Wang <jinpu.wang@ionos.com> writes:
> On Thu, Oct 9, 2025 at 1:21 PM Jinpu Wang <jinpu.wang@ionos.com> wrote:
>>
>> On Thu, Oct 9, 2025 at 5:44 AM Jinpu Wang <jinpu.wang@ionos.com> wrote:
>> >
>> > Hi Sean,
>> >
>> > On Wed, Oct 8, 2025 at 5:55 PM Sean Christopherson <seanjc@google.com> wrote:
>> > >
>> > > Trimmed Cc: to drop people from the original thread. In the future, just start
>> > > a new bug report. Piggybacking a 10 year old bug just because the symptoms are
>> > > similar does more harm than good. Whatever the old thread was chasing was already
>> > > fixed, _10 years_ ago; they were just trying to identy exactly what commit fixed
>> > > the problem. I.e. whatever they were chasing _can't_ be the same root cause,
>> > > because even if it's literally the same code bug, it would require a code change
>> > > and thus a regression between v4.0 and v6.1.
>> > Thx for the reply, it makes sense. I will remember this next time.
>> > >
>> > > On Wed, Oct 08, 2025, Jinpu Wang wrote:
>> > > > On Wed, Oct 8, 2025 at 2:44 PM Jack Wang <jinpu.wang@ionos.com> wrote:
>> > > > > Sorry for bump this old thread, we hit same issue on Intel Sierra Forest
>> > > > > machines with LTS kernel 6.1/6.12, maybe KVM comunity could help fix it.
>> > >
>> > > Are there any host kernels that _do_ work? E.g. have you tried a bleeding edge
>> > > host kernel?
>> > I will try linus/master today.
>> > >
>> > > > > ### **[BUG] Hang on FreeBSD Guest Reboot under KVM on Intel SierraForest (Xeon 6710E)**
>> > > > >
>> > > > > **Summary:**
>> > > > > Multi-cores FreeBSD guests hang during reboot under KVM on systems with
>> > > > > Intel(R) Xeon(R) 6710E (SierraForest). The issue is fully reproducible with
>> > > > > APICv enabled and disappears when disabling APICv (`enable_apicv=N`). The
>> > > > > same configuration works correctly on Ice Lake (Xeon Gold 6338).
>> > >
>> > > Does Sierra Forest have IPI virtualization? If so, you could try running with
>> > > APICv enabled, but enable_ipiv=false to specifically disable IPI virtualization.
>> > Yes, it does:
>> > $ grep . /sys/module/kvm_intel/parameters/*
>> > /sys/module/kvm_intel/parameters/allow_smaller_maxphyaddr:N
>> > /sys/module/kvm_intel/parameters/dump_invalid_vmcs:N
>> > /sys/module/kvm_intel/parameters/emulate_invalid_guest_state:Y
>> > /sys/module/kvm_intel/parameters/enable_apicv:Y
>> > /sys/module/kvm_intel/parameters/enable_ipiv:Y
>> > /sys/module/kvm_intel/parameters/enable_shadow_vmcs:Y
>> > /sys/module/kvm_intel/parameters/ept:Y
>> > /sys/module/kvm_intel/parameters/eptad:Y
>> > /sys/module/kvm_intel/parameters/error_on_inconsistent_vmcs_config:Y
>> > /sys/module/kvm_intel/parameters/fasteoi:Y
>> > /sys/module/kvm_intel/parameters/flexpriority:Y
>> > /sys/module/kvm_intel/parameters/nested:Y
>> > /sys/module/kvm_intel/parameters/nested_early_check:N
>> > /sys/module/kvm_intel/parameters/ple_gap:128
>> > /sys/module/kvm_intel/parameters/ple_window:4096
>> > /sys/module/kvm_intel/parameters/ple_window_grow:2
>> > /sys/module/kvm_intel/parameters/ple_window_max:4294967295
>> > /sys/module/kvm_intel/parameters/ple_window_shrink:0
>> > /sys/module/kvm_intel/parameters/pml:Y
>> > /sys/module/kvm_intel/parameters/preemption_timer:Y
>> > /sys/module/kvm_intel/parameters/sgx:N
>> > /sys/module/kvm_intel/parameters/unrestricted_guest:Y
>> > /sys/module/kvm_intel/parameters/vmentry_l1d_flush:not required
>> > /sys/module/kvm_intel/parameters/vnmi:Y
>> > /sys/module/kvm_intel/parameters/vpid:Y
>> >
>> > I tried to disable ipiv, but it doesn't help. freebsd hang on reboot.
>> > sudo modprobe -r kvm_intel
>> > sudo modprobe kvm_intel enable_ipiv=N
>> > /sys/module/kvm_intel/parameters/enable_ipiv:N
>> >
>> > Thx!
>> +cc Vitaly
>> Sorry, I missed one detail, we are use hyper-V enlightment features:
>> "+hv-relaxed,+hv-vapic,+hv-time,+hv-runtime,hv-spinlocks=0x1fff,+hv-vpindex,+hv-synic,+hv-stimer,+hv-tlbflush,hv-ipi."
>>
>> did a lot tests with different features, and looks the hang is related
>> to +hv-synic,+hv-stimer. hv-synic seems the key which causes boot
>> hang of Freebsd 14.
>>
>> But the problem seems fixed with FreeBSD 15? I guess it's this fix:
> https://reviews.freebsd.org/D43508
>
>>
>> Seems it's a bug from freebsd side, rather than on kvm side to me, but
>> I'm puzzled by disable apicv helps?
In theory, FreeBSD should work well even if KVM is misdetected as
genuine Hyper-V. Apparently, our emulation is not 1:1 and there are
subtle differences which cause the hang. I did not look at FreeBSD code
at all but my wild guess is that SynIC/stimer are not disabled properly
upon reboot and this causes the problem. If we somehow manage to find
how genuine Hyper-V's behavior is different, it would make sense to
update KVM/QEMU to match.
--
Vitaly
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Hang on reboot in multi-core FreeBSD guest on Linux KVM host with Intel Sierra Forest CPU
2025-10-09 12:52 ` Vitaly Kuznetsov
@ 2025-10-10 5:04 ` Jinpu Wang
0 siblings, 0 replies; 6+ messages in thread
From: Jinpu Wang @ 2025-10-10 5:04 UTC (permalink / raw)
To: Vitaly Kuznetsov
Cc: Sean Christopherson, fanwenyi0529, kvm, Paolo Bonzini,
Xiaoyao Li, linux-kernel
On Thu, Oct 9, 2025 at 2:53 PM Vitaly Kuznetsov <vkuznets@redhat.com> wrote:
>
> Jinpu Wang <jinpu.wang@ionos.com> writes:
>
> > On Thu, Oct 9, 2025 at 1:21 PM Jinpu Wang <jinpu.wang@ionos.com> wrote:
> >>
> >> On Thu, Oct 9, 2025 at 5:44 AM Jinpu Wang <jinpu.wang@ionos.com> wrote:
> >> >
> >> > Hi Sean,
> >> >
> >> > On Wed, Oct 8, 2025 at 5:55 PM Sean Christopherson <seanjc@google.com> wrote:
> >> > >
> >> > > Trimmed Cc: to drop people from the original thread. In the future, just start
> >> > > a new bug report. Piggybacking a 10 year old bug just because the symptoms are
> >> > > similar does more harm than good. Whatever the old thread was chasing was already
> >> > > fixed, _10 years_ ago; they were just trying to identy exactly what commit fixed
> >> > > the problem. I.e. whatever they were chasing _can't_ be the same root cause,
> >> > > because even if it's literally the same code bug, it would require a code change
> >> > > and thus a regression between v4.0 and v6.1.
> >> > Thx for the reply, it makes sense. I will remember this next time.
> >> > >
> >> > > On Wed, Oct 08, 2025, Jinpu Wang wrote:
> >> > > > On Wed, Oct 8, 2025 at 2:44 PM Jack Wang <jinpu.wang@ionos.com> wrote:
> >> > > > > Sorry for bump this old thread, we hit same issue on Intel Sierra Forest
> >> > > > > machines with LTS kernel 6.1/6.12, maybe KVM comunity could help fix it.
> >> > >
> >> > > Are there any host kernels that _do_ work? E.g. have you tried a bleeding edge
> >> > > host kernel?
> >> > I will try linus/master today.
> >> > >
> >> > > > > ### **[BUG] Hang on FreeBSD Guest Reboot under KVM on Intel SierraForest (Xeon 6710E)**
> >> > > > >
> >> > > > > **Summary:**
> >> > > > > Multi-cores FreeBSD guests hang during reboot under KVM on systems with
> >> > > > > Intel(R) Xeon(R) 6710E (SierraForest). The issue is fully reproducible with
> >> > > > > APICv enabled and disappears when disabling APICv (`enable_apicv=N`). The
> >> > > > > same configuration works correctly on Ice Lake (Xeon Gold 6338).
> >> > >
> >> > > Does Sierra Forest have IPI virtualization? If so, you could try running with
> >> > > APICv enabled, but enable_ipiv=false to specifically disable IPI virtualization.
> >> > Yes, it does:
> >> > $ grep . /sys/module/kvm_intel/parameters/*
> >> > /sys/module/kvm_intel/parameters/allow_smaller_maxphyaddr:N
> >> > /sys/module/kvm_intel/parameters/dump_invalid_vmcs:N
> >> > /sys/module/kvm_intel/parameters/emulate_invalid_guest_state:Y
> >> > /sys/module/kvm_intel/parameters/enable_apicv:Y
> >> > /sys/module/kvm_intel/parameters/enable_ipiv:Y
> >> > /sys/module/kvm_intel/parameters/enable_shadow_vmcs:Y
> >> > /sys/module/kvm_intel/parameters/ept:Y
> >> > /sys/module/kvm_intel/parameters/eptad:Y
> >> > /sys/module/kvm_intel/parameters/error_on_inconsistent_vmcs_config:Y
> >> > /sys/module/kvm_intel/parameters/fasteoi:Y
> >> > /sys/module/kvm_intel/parameters/flexpriority:Y
> >> > /sys/module/kvm_intel/parameters/nested:Y
> >> > /sys/module/kvm_intel/parameters/nested_early_check:N
> >> > /sys/module/kvm_intel/parameters/ple_gap:128
> >> > /sys/module/kvm_intel/parameters/ple_window:4096
> >> > /sys/module/kvm_intel/parameters/ple_window_grow:2
> >> > /sys/module/kvm_intel/parameters/ple_window_max:4294967295
> >> > /sys/module/kvm_intel/parameters/ple_window_shrink:0
> >> > /sys/module/kvm_intel/parameters/pml:Y
> >> > /sys/module/kvm_intel/parameters/preemption_timer:Y
> >> > /sys/module/kvm_intel/parameters/sgx:N
> >> > /sys/module/kvm_intel/parameters/unrestricted_guest:Y
> >> > /sys/module/kvm_intel/parameters/vmentry_l1d_flush:not required
> >> > /sys/module/kvm_intel/parameters/vnmi:Y
> >> > /sys/module/kvm_intel/parameters/vpid:Y
> >> >
> >> > I tried to disable ipiv, but it doesn't help. freebsd hang on reboot.
> >> > sudo modprobe -r kvm_intel
> >> > sudo modprobe kvm_intel enable_ipiv=N
> >> > /sys/module/kvm_intel/parameters/enable_ipiv:N
> >> >
> >> > Thx!
> >> +cc Vitaly
> >> Sorry, I missed one detail, we are use hyper-V enlightment features:
> >> "+hv-relaxed,+hv-vapic,+hv-time,+hv-runtime,hv-spinlocks=0x1fff,+hv-vpindex,+hv-synic,+hv-stimer,+hv-tlbflush,hv-ipi."
> >>
> >> did a lot tests with different features, and looks the hang is related
> >> to +hv-synic,+hv-stimer. hv-synic seems the key which causes boot
> >> hang of Freebsd 14.
> >>
> >> But the problem seems fixed with FreeBSD 15? I guess it's this fix:
> > https://reviews.freebsd.org/D43508
> >
> >>
> >> Seems it's a bug from freebsd side, rather than on kvm side to me, but
> >> I'm puzzled by disable apicv helps?
>
> In theory, FreeBSD should work well even if KVM is misdetected as
> genuine Hyper-V. Apparently, our emulation is not 1:1 and there are
> subtle differences which cause the hang. I did not look at FreeBSD code
> at all but my wild guess is that SynIC/stimer are not disabled properly
> upon reboot and this causes the problem. If we somehow manage to find
> how genuine Hyper-V's behavior is different, it would make sense to
> update KVM/QEMU to match.
>
> --
> Vitaly
Hi Vitaly,
Thx for the reply, that's another view.
I spent a few hours to build freebsd stable/14 with commit
6744fd8e7503 ("x86/cpu: improve hypervisor detection") applied, the
bug is fixed, no boot hung, and boot is much faster ~ 40 seconds, as
Freebsd 15, without the fix Freebsd takes 150 seconds if sucessfully
boots, and often hang forever during reboot.
Jinpu
>
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2025-10-10 5:04 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
[not found] <539FC243.2070906@redhat.com>
[not found] ` <20140617060500.GA20764@minantech.com>
[not found] ` <FFEF5F78-D9E6-4333-BC1A-78076C132CBF@jnielsen.net>
[not found] ` <6850B127-F16B-465F-BDDB-BA3F99B9E446@jnielsen.net>
[not found] ` <jpgioafjtxb.fsf@redhat.com>
[not found] ` <74412BDB-EF6F-4C20-84C8-C6EF3A25885C@jnielsen.net>
[not found] ` <558AD1B0.5060200@redhat.com>
[not found] ` <FAFB2BA9-E924-4E70-A84A-E5F2D97BC2F0@jnielsen.net>
[not found] ` <CACzj_yVTyescyWBRuA3MMCC0Ymg7TKF-+sCW1N+Xwfffvw_Wsg@mail.gmail.com>
[not found] ` <CAMGffE=P5HJkJxh2mj3c_oh6busFKYb0TGuhAc36toc5_uD72w@mail.gmail.com>
2025-10-08 15:55 ` Hang on reboot in multi-core FreeBSD guest on Linux KVM host with Intel Sierra Forest CPU Sean Christopherson
2025-10-09 3:44 ` Jinpu Wang
2025-10-09 11:21 ` Jinpu Wang
2025-10-09 12:36 ` Jinpu Wang
2025-10-09 12:52 ` Vitaly Kuznetsov
2025-10-10 5:04 ` Jinpu Wang
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
Powered by JetHome