[PATCH 0/1] KVM: powerpc/book3s_hv: Handle deferred CFS bandwidth throttle on guest re-entry

[PATCH 0/1] KVM: powerpc/book3s_hv: Handle deferred CFS bandwidth throttle on guest re-entry

[Vishal Chourasia]

This series fixes a KVM scheduling bug on Book3S HV (POWER8/POWER9/POWER10)
where a guest VM under a cpu.max bandwidth limit can run arbitrarily past its
quota and then appear completely frozen for minutes afterwards.

== Background ==

Commit 2cd571245b43 ("sched/fair: Add related data structure for task based
throttle"), merged in v6.18, changed how CFS bandwidth throttling enforces
its limit. Previously, throttle_cfs_rq() dequeued tasks directly. Under the
new scheme it queues a task_work item via task_work_add(..., TWA_RESUME),
sets TIF_NOTIFY_RESUME, and relies on that work running on the kernel return
path to actually dequeue the task.

For KVM guests this means the work must be drained before each guest entry,
not just on the normal syscall return path. commit 935ace2fb5cc ("entry:
Provide infrastructure for work before transitioning to guest mode")
introduced kvm_xfer_to_guest_mode_handle_work() for exactly this purpose.
x86 (commit 72c3c0fe54a3), arm64 (commit 6caa5812e2d1), riscv, s390, and
loongarch all adopted it. Book3S HV did not. [1]

== Root Cause ==

Book3S HV's vCPU run loops — kvmhv_run_single_vcpu() for POWER9+ and
kvmppc_run_vcpu() for pre-POWER9 — only test TIF_SIGPENDING and
TIF_NEED_RESCHED before re-entering the guest. TIF_NOTIFY_RESUME is never
checked, and the deferred throttle task_work therefore never runs while a
vCPU is inside the run loop.

For a CPU-bound guest that generates few KVM exits back to QEMU user space
(e.g. a compute-heavy or busy-looping workload), the vCPU thread never
returns to user mode. throttle_cfs_rq() sets cfs_rq->throttled = 1 and
queues the task_work, but the guest continues to run unchecked.
cfs_rq->runtime_remaining goes increasingly negative with every scheduling
period while the throttle flag sits ignored.

The only mechanism recovering that debt is the periodic bandwidth timer
replenishment: 30 ms of quota is added per 100 ms period. When
runtime_remaining has drifted hundreds of seconds negative, recovering to
zero at 300 ms/s takes minutes — during which the cgroup is legitimately
throttled and the VM is completely frozen once it finally exits to user
space.

== Debugging ==

vCPU was placed in a cgroup where CPU bandwidth limits were set.
quota = 30ms
period = 100ms

The bug was diagnosed using a bpftrace script probing throttle_cfs_rq()
and unthrottle_cfs_rq() and sampling cfs_rq->runtime_remaining every
second. The trace shows the debt accumulation phase, the slow recovery
phase, and the immediate re-throttle on resumption:

  Debt accumulation (vCPU in guest, no exits):
    +1471 s  runtime_remaining=-209702865115 ns  throttled=1
    +1472 s  runtime_remaining=-210402866357 ns  throttled=1
    ...                                    # ~-700 ms/s (growing debt)
    +1477 s  runtime_remaining=-213902833931 ns  throttled=1

  Recovery (vCPU exits to QEMU user space; bandwidth timer replenishes):
    +1478 s  runtime_remaining=-213617443453 ns  throttled=1
    +1479 s  runtime_remaining=-213317443453 ns  throttled=1
    ...                                    # ~+300 ms/s (30ms quota/100ms)

  After ~710 seconds of recovery, debt reaches zero:
    ──── unthrottle_cfs_rq @ cpu=768  +2190.029568131 s ────
      runtime_remaining = 1 ns            # just crossed zero

  The vCPU immediately re-enters the guest and over-runs its quota again:
    ──── throttle_cfs_rq @ cpu=768  +2190.055327252 s ────
      runtime_remaining = -5667293 ns     # 26 ms of debt already

  The cycle then repeats identically from a fresh -700 ms/s accumulation.

cpu.stat confirms the pathology — 100% throttle rate and virtually all
CPU time accumulated in kernel (KVM) mode:

  nr_periods   = 117457
  nr_throttled = 117457     # every single period
  system_usec  = 4334782636 # >99.99% kernel time (QEMU in KVM_RUN)

strace of the QEMU vCPU thread confirms long stretches where
ioctl(KVM_RUN) does not return — the vCPU is running in guest mode
with no VM-exits reaching user space.

== Fix Summary ==

Opt Book3S HV into VIRT_XFER_TO_GUEST_WORK and drain pending guest-mode
work (including the deferred CFS throttle task_work) on every guest
re-entry in both run loops. The changes are supersets of the existing
need_resched() checks and do not alter the signal or exit accounting.

[1] https://lore.kernel.org/all/20250421102837.78515-2-sshegde@linux.ibm.com/

Vishal Chourasia (1):
  KVM: powerpc/book3s_hv: Use generic xfer to guest work function

 arch/powerpc/kvm/Kconfig     |  1 +
 arch/powerpc/kvm/book3s_hv.c | 58 +++++++++++++++++++++++++++++++-----
 2 files changed, 52 insertions(+), 7 deletions(-)

-- 
2.54.0

[PATCH 1/1] KVM: powerpc/book3s_hv: Use generic xfer to guest work function

[Vishal Chourasia]

Use the generic infrastructure to check for and handle pending work
before transitioning into guest mode, replacing the open-coded
need_resched() and cond_resched() checks.

This picks up handling for TIF_NOTIFY_RESUME, which was previously
ignored, meaning task work will now be correctly handled on every
guest re-entry.

Signed-off-by: Vishal Chourasia <vishalc@linux.ibm.com>
---
 arch/powerpc/kvm/Kconfig     |  1 +
 arch/powerpc/kvm/book3s_hv.c | 58 +++++++++++++++++++++++++++++++-----
 2 files changed, 52 insertions(+), 7 deletions(-)

diff --git a/arch/powerpc/kvm/Kconfig b/arch/powerpc/kvm/Kconfig
index 9a0d1c1aca6c..36aec58c5f22 100644
--- a/arch/powerpc/kvm/Kconfig
+++ b/arch/powerpc/kvm/Kconfig
@@ -81,6 +81,7 @@ config KVM_BOOK3S_64_HV
 	depends on KVM_BOOK3S_64 && PPC_POWERNV
 	select KVM_BOOK3S_HV_POSSIBLE
 	select KVM_BOOK3S_HV_PMU
+	select VIRT_XFER_TO_GUEST_WORK
 	select CMA
 	help
 	  Support running unmodified book3s_64 guest kernels in
diff --git a/arch/powerpc/kvm/book3s_hv.c b/arch/powerpc/kvm/book3s_hv.c
index 61dbeea317f3..b012512342e6 100644
--- a/arch/powerpc/kvm/book3s_hv.c
+++ b/arch/powerpc/kvm/book3s_hv.c
@@ -3850,10 +3850,20 @@ static noinline void kvmppc_run_core(struct kvmppc_vcore *vc)
 	 * and return without going into the guest(s).
 	 * If the mmu_ready flag has been cleared, don't go into the
 	 * guest because that means a HPT resize operation is in progress.
+	 *
+	 * xfer_to_guest_mode_work_pending() is the IRQs-disabled recheck for
+	 * pending guest-mode work (reschedule, signals, and TIF_NOTIFY_RESUME
+	 * task_work such as the deferred CFS throttle). It is the pre-POWER9
+	 * analog of the final gate in kvmhv_run_single_vcpu(), and a superset
+	 * of the old need_resched() check: it catches work that raced in after
+	 * the drain in kvmppc_run_vcpu(), so a CPU-bound vCPU is throttled here
+	 * instead of running one more guest dispatch past its quota. IRQs are
+	 * hard-disabled just above, so the non-__ variant (which asserts that)
+	 * is the correct one.
 	 */
 	local_irq_disable();
 	hard_irq_disable();
-	if (lazy_irq_pending() || need_resched() ||
+	if (lazy_irq_pending() || xfer_to_guest_mode_work_pending() ||
 	    recheck_signals_and_mmu(&core_info)) {
 		local_irq_enable();
 		vc->vcore_state = VCORE_INACTIVE;
@@ -4824,10 +4834,24 @@ static int kvmppc_run_vcpu(struct kvm_vcpu *vcpu)
 		vc->runner = vcpu;
 		if (n_ceded == vc->n_runnable) {
 			kvmppc_vcore_blocked(vc);
-		} else if (need_resched()) {
+		} else if (__xfer_to_guest_mode_work_pending()) {
 			kvmppc_vcore_preempt(vc);
-			/* Let something else run */
-			cond_resched_lock(&vc->lock);
+			/*
+			 * Let something else run, and run pending guest-mode
+			 * work (reschedule, and TIF_NOTIFY_RESUME task_work such
+			 * as the deferred CFS throttle) before we would re-enter
+			 * the guest, so a CPU-bound vCPU is actually throttled
+			 * here instead of running past its quota. This is a
+			 * superset of the old need_resched() check. Use the raw
+			 * helper, not the kvm_ wrapper: signals (KVM_EXIT_INTR
+			 * and the signal_exits stat) are accounted by this path's
+			 * existing handling below, so going through the wrapper
+			 * here would double-count them. The helper may schedule(),
+			 * so the vcore lock is dropped around it.
+			 */
+			spin_unlock(&vc->lock);
+			xfer_to_guest_mode_handle_work();
+			spin_lock(&vc->lock);
 			if (vc->vcore_state == VCORE_PREEMPT)
 				kvmppc_vcore_end_preempt(vc);
 		} else {
@@ -4899,8 +4923,21 @@ int kvmhv_run_single_vcpu(struct kvm_vcpu *vcpu, u64 time_limit,
 		}
 	}
 
-	if (need_resched())
-		cond_resched();
+	/*
+	 * Run pending work before (re-)entering the guest, most importantly
+	 * task_work queued via TWA_RESUME (e.g. the deferred CFS bandwidth
+	 * throttle, which only sets TIF_NOTIFY_RESUME). Without this a CPU-bound
+	 * vCPU that keeps returning RESUME_GUEST never reaches an exit-to-user
+	 * point, so the throttle is never enforced and the task runs far beyond
+	 * its quota. The helper also handles reschedule and signals, replacing
+	 * the cond_resched() that was here. It may schedule(), so it runs before
+	 * preemption and IRQs are disabled, with no vcore/KVM locks held. This
+	 * is the per-reentry site shared by the bare-metal and pseries (nested)
+	 * paths, so both are covered.
+	 */
+	r = kvm_xfer_to_guest_mode_handle_work(vcpu);
+	if (r)	/* -EINTR: signal pending, exit to userspace (KVM_EXIT_INTR) */
+		return r;
 
 	kvmppc_update_vpas(vcpu);
 
@@ -4916,7 +4953,14 @@ int kvmhv_run_single_vcpu(struct kvm_vcpu *vcpu, u64 time_limit,
 
 	if (signal_pending(current))
 		goto sigpend;
-	if (need_resched() || !kvm->arch.mmu_ready)
+	/*
+	 * Re-check for pending guest-mode work with IRQs disabled, to catch
+	 * anything (e.g. a TIF_NOTIFY_RESUME task_work such as the deferred CFS
+	 * throttle) that raced in after the check above. Bail back to the outer
+	 * loop, which re-enters here and runs the work. This is a superset of
+	 * the previous need_resched() check.
+	 */
+	if (xfer_to_guest_mode_work_pending() || !kvm->arch.mmu_ready)
 		goto out;
 
 	vcpu->cpu = pcpu;
-- 
2.54.0

Re: [PATCH 1/1] KVM: powerpc/book3s_hv: Use generic xfer to guest work function

[Shrikanth Hegde]

Hi Vishal,

On 6/26/26 4:23 PM, Vishal Chourasia wrote:
> Use the generic infrastructure to check for and handle pending work
> before transitioning into guest mode, replacing the open-coded
> need_resched() and cond_resched() checks.
> 
> This picks up handling for TIF_NOTIFY_RESUME, which was previously
> ignored, meaning task work will now be correctly handled on every
> guest re-entry.

It would indeed be good if powerpc moves go generic VIRT_XFER_TO_GUEST_WORK.
It does take care of RESUME.

In addition today, I doubt powerpc kvm works well for LAZY preemption.
generic infra will take care of it too.

> 
> Signed-off-by: Vishal Chourasia <vishalc@linux.ibm.com>
> ---
>   arch/powerpc/kvm/Kconfig     |  1 +
>   arch/powerpc/kvm/book3s_hv.c | 58 +++++++++++++++++++++++++++++++-----
>   2 files changed, 52 insertions(+), 7 deletions(-)
> 
> diff --git a/arch/powerpc/kvm/Kconfig b/arch/powerpc/kvm/Kconfig
> index 9a0d1c1aca6c..36aec58c5f22 100644
> --- a/arch/powerpc/kvm/Kconfig
> +++ b/arch/powerpc/kvm/Kconfig
> @@ -81,6 +81,7 @@ config KVM_BOOK3S_64_HV
>   	depends on KVM_BOOK3S_64 && PPC_POWERNV
>   	select KVM_BOOK3S_HV_POSSIBLE
>   	select KVM_BOOK3S_HV_PMU
> +	select VIRT_XFER_TO_GUEST_WORK

This takes care of HV only.
Does PR/booke run into the same problem?
Does anyone out there who runs them still and care about cgroup bandwidth control mechanism on it?

It may be worth adding comment that PR still runs into the same issue.

>   	select CMA
>   	help
>   	  Support running unmodified book3s_64 guest kernels in
> diff --git a/arch/powerpc/kvm/book3s_hv.c b/arch/powerpc/kvm/book3s_hv.c
> index 61dbeea317f3..b012512342e6 100644
> --- a/arch/powerpc/kvm/book3s_hv.c
> +++ b/arch/powerpc/kvm/book3s_hv.c
> @@ -3850,10 +3850,20 @@ static noinline void kvmppc_run_core(struct kvmppc_vcore *vc)
>   	 * and return without going into the guest(s).
>   	 * If the mmu_ready flag has been cleared, don't go into the
>   	 * guest because that means a HPT resize operation is in progress.
> +	 *
> +	 * xfer_to_guest_mode_work_pending() is the IRQs-disabled recheck for
> +	 * pending guest-mode work (reschedule, signals, and TIF_NOTIFY_RESUME
> +	 * task_work such as the deferred CFS throttle). It is the pre-POWER9
> +	 * analog of the final gate in kvmhv_run_single_vcpu(), and a superset
> +	 * of the old need_resched() check: it catches work that raced in after
> +	 * the drain in kvmppc_run_vcpu(), so a CPU-bound vCPU is throttled here
> +	 * instead of running one more guest dispatch past its quota. IRQs are
> +	 * hard-disabled just above, so the non-__ variant (which asserts that)
> +	 * is the correct one.
>   	 */
>   	local_irq_disable();
>   	hard_irq_disable();
> -	if (lazy_irq_pending() || need_resched() ||
> +	if (lazy_irq_pending() || xfer_to_guest_mode_work_pending() ||
>   	    recheck_signals_and_mmu(&core_info)) {
>   		local_irq_enable();
>   		vc->vcore_state = VCORE_INACTIVE;
> @@ -4824,10 +4834,24 @@ static int kvmppc_run_vcpu(struct kvm_vcpu *vcpu)
>   		vc->runner = vcpu;
>   		if (n_ceded == vc->n_runnable) {
>   			kvmppc_vcore_blocked(vc);
> -		} else if (need_resched()) {
> +		} else if (__xfer_to_guest_mode_work_pending()) {
>   			kvmppc_vcore_preempt(vc);
> -			/* Let something else run */
> -			cond_resched_lock(&vc->lock);
> +			/*
> +			 * Let something else run, and run pending guest-mode
> +			 * work (reschedule, and TIF_NOTIFY_RESUME task_work such
> +			 * as the deferred CFS throttle) before we would re-enter
> +			 * the guest, so a CPU-bound vCPU is actually throttled
> +			 * here instead of running past its quota. This is a
> +			 * superset of the old need_resched() check. Use the raw
> +			 * helper, not the kvm_ wrapper: signals (KVM_EXIT_INTR
> +			 * and the signal_exits stat) are accounted by this path's
> +			 * existing handling below, so going through the wrapper
> +			 * here would double-count them. The helper may schedule(),
> +			 * so the vcore lock is dropped around it.
> +			 */
> +			spin_unlock(&vc->lock);
> +			xfer_to_guest_mode_handle_work();
> +			spin_lock(&vc->lock);
>   			if (vc->vcore_state == VCORE_PREEMPT)
>   				kvmppc_vcore_end_preempt(vc);
>   		} else {
> @@ -4899,8 +4923,21 @@ int kvmhv_run_single_vcpu(struct kvm_vcpu *vcpu, u64 time_limit,
>   		}
>   	}
>   
> -	if (need_resched())
> -		cond_resched();
> +	/*
> +	 * Run pending work before (re-)entering the guest, most importantly
> +	 * task_work queued via TWA_RESUME (e.g. the deferred CFS bandwidth
> +	 * throttle, which only sets TIF_NOTIFY_RESUME). Without this a CPU-bound
> +	 * vCPU that keeps returning RESUME_GUEST never reaches an exit-to-user
> +	 * point, so the throttle is never enforced and the task runs far beyond
> +	 * its quota. The helper also handles reschedule and signals, replacing
> +	 * the cond_resched() that was here. It may schedule(), so it runs before
> +	 * preemption and IRQs are disabled, with no vcore/KVM locks held. This
> +	 * is the per-reentry site shared by the bare-metal and pseries (nested)
> +	 * paths, so both are covered.
> +	 */
> +	r = kvm_xfer_to_guest_mode_handle_work(vcpu);
> +	if (r)	/* -EINTR: signal pending, exit to userspace (KVM_EXIT_INTR) */
> +		return r;
>   
>   	kvmppc_update_vpas(vcpu);
>   
> @@ -4916,7 +4953,14 @@ int kvmhv_run_single_vcpu(struct kvm_vcpu *vcpu, u64 time_limit,
>   
>   	if (signal_pending(current))
>   		goto sigpend;

xfer_to_guest_mode_work_pending checks for signals too right?

#define XFER_TO_GUEST_MODE_WORK                                         \
         (_TIF_NEED_RESCHED | _TIF_NEED_RESCHED_LAZY | _TIF_SIGPENDING | \
          _TIF_NOTIFY_SIGNAL | _TIF_NOTIFY_RESUME |                      \
          ARCH_XFER_TO_GUEST_MODE_WORK)

> -	if (need_resched() || !kvm->arch.mmu_ready)
> +	/*
> +	 * Re-check for pending guest-mode work with IRQs disabled, to catch
> +	 * anything (e.g. a TIF_NOTIFY_RESUME task_work such as the deferred CFS
> +	 * throttle) that raced in after the check above. Bail back to the outer
> +	 * loop, which re-enters here and runs the work. This is a superset of
> +	 * the previous need_resched() check.
> +	 */
> +	if (xfer_to_guest_mode_work_pending() || !kvm->arch.mmu_ready)
>   		goto out;
>   
>   	vcpu->cpu = pcpu;

Copying from the discussion thread at that time,

"""
on x86:
kvm_arch_vcpu_ioctl_run
	vcpu_run
		for () {
			.. run guest..
			xfer_to_guest_mode_handle_work
				schedule
		}


on Powerpc:  ( taking book3s_hv flavour):
kvm_arch_vcpu_ioctl_run
kvmppc_vcpu_run_hv  *1
	do while() {
		kvmhv_run_single_vcpu or kvmppc_run_vcpu
			-- checking for need_resched and signals and bails out *2
	}


*1 - checks for need resched and signals before entering guest
*2 - checks for need resched and signals while running the guest


This patch is addressing only *1 but it needs to address *2 as well using generic framework.
I think it is doable for books3s_hv atleast. (though might need rewrite)
"""


A few questions that comes to my mind.

1. I think you are doing for both *1 and *2 right? Is *1 necessary still for powerpc?
2. There is a for loop. What is it doing? Does this still need similar checks?

         if (is_kvmppc_resume_guest(r) && !kvmppc_vcpu_check_block(vcpu)) {
                 kvmppc_set_timer(vcpu);

                 prepare_to_rcuwait(wait);
                 for (;;) {
                         set_current_state(TASK_INTERRUPTIBLE);
                         if (signal_pending(current)) {
                                 vcpu->stat.signal_exits++;
                                 run->exit_reason = KVM_EXIT_INTR;
                                 vcpu->arch.ret = -EINTR;
                                 break;
                         }

                         if (kvmppc_vcpu_check_block(vcpu))
                                 break;

                         trace_kvmppc_vcore_blocked(vcpu, 0);
                         schedule();
                         trace_kvmppc_vcore_blocked(vcpu, 1);
                 }
                 finish_rcuwait(wait);
         }
         vcpu->arch.ceded = 0;


PS: cc'ing me would have helped me to see the mail earlier since you had referred
to the patch I had sent.

Re: [PATCH 1/1] KVM: powerpc/book3s_hv: Use generic xfer to guest work function

[Vishal Chourasia]

On 01/07/26 11:43, Shrikanth Hegde wrote:
> Hi Vishal,
Hi Shrikanth, Thanks for looking into it.
>
> On 6/26/26 4:23 PM, Vishal Chourasia wrote:
>> Use the generic infrastructure to check for and handle pending work
>> before transitioning into guest mode, replacing the open-coded
>> need_resched() and cond_resched() checks.
>>
>> This picks up handling for TIF_NOTIFY_RESUME, which was previously
>> ignored, meaning task work will now be correctly handled on every
>> guest re-entry.
Yes
>
> It would indeed be good if powerpc moves go generic 
> VIRT_XFER_TO_GUEST_WORK.
> It does take care of RESUME.
>
> In addition today, I doubt powerpc kvm works well for LAZY preemption.
> generic infra will take care of it too.
>
nice
>>
>> Signed-off-by: Vishal Chourasia <vishalc@linux.ibm.com>
>> ---
>>   arch/powerpc/kvm/Kconfig     |  1 +
>>   arch/powerpc/kvm/book3s_hv.c | 58 +++++++++++++++++++++++++++++++-----
>>   2 files changed, 52 insertions(+), 7 deletions(-)
>>
>> diff --git a/arch/powerpc/kvm/Kconfig b/arch/powerpc/kvm/Kconfig
>> index 9a0d1c1aca6c..36aec58c5f22 100644
>> --- a/arch/powerpc/kvm/Kconfig
>> +++ b/arch/powerpc/kvm/Kconfig
>> @@ -81,6 +81,7 @@ config KVM_BOOK3S_64_HV
>>       depends on KVM_BOOK3S_64 && PPC_POWERNV
>>       select KVM_BOOK3S_HV_POSSIBLE
>>       select KVM_BOOK3S_HV_PMU
>> +    select VIRT_XFER_TO_GUEST_WORK
>
> This takes care of HV only.
> Does PR/booke run into the same problem?
Yes, I would think so, if the KVM vCPU runs inside system.
> Does anyone out there who runs them still and care about cgroup 
> bandwidth control mechanism on it?
>
> It may be worth adding comment that PR still runs into the same issue.
>
>>       select CMA
>>       help
>>         Support running unmodified book3s_64 guest kernels in
>> diff --git a/arch/powerpc/kvm/book3s_hv.c b/arch/powerpc/kvm/book3s_hv.c
>> index 61dbeea317f3..b012512342e6 100644
>> --- a/arch/powerpc/kvm/book3s_hv.c
>> +++ b/arch/powerpc/kvm/book3s_hv.c
>> @@ -3850,10 +3850,20 @@ static noinline void kvmppc_run_core(struct 
>> kvmppc_vcore *vc)
>>        * and return without going into the guest(s).
>>        * If the mmu_ready flag has been cleared, don't go into the
>>        * guest because that means a HPT resize operation is in progress.
>> +     *
>> +     * xfer_to_guest_mode_work_pending() is the IRQs-disabled 
>> recheck for
>> +     * pending guest-mode work (reschedule, signals, and 
>> TIF_NOTIFY_RESUME
>> +     * task_work such as the deferred CFS throttle). It is the 
>> pre-POWER9
>> +     * analog of the final gate in kvmhv_run_single_vcpu(), and a 
>> superset
>> +     * of the old need_resched() check: it catches work that raced 
>> in after
>> +     * the drain in kvmppc_run_vcpu(), so a CPU-bound vCPU is 
>> throttled here
>> +     * instead of running one more guest dispatch past its quota. 
>> IRQs are
>> +     * hard-disabled just above, so the non-__ variant (which 
>> asserts that)
>> +     * is the correct one.
>>        */
>>       local_irq_disable();
>>       hard_irq_disable();
>> -    if (lazy_irq_pending() || need_resched() ||
>> +    if (lazy_irq_pending() || xfer_to_guest_mode_work_pending() ||
>>           recheck_signals_and_mmu(&core_info)) {
>>           local_irq_enable();
>>           vc->vcore_state = VCORE_INACTIVE;
>> @@ -4824,10 +4834,24 @@ static int kvmppc_run_vcpu(struct kvm_vcpu 
>> *vcpu)
>>           vc->runner = vcpu;
>>           if (n_ceded == vc->n_runnable) {
>>               kvmppc_vcore_blocked(vc);
>> -        } else if (need_resched()) {
>> +        } else if (__xfer_to_guest_mode_work_pending()) {
>>               kvmppc_vcore_preempt(vc);
>> -            /* Let something else run */
>> -            cond_resched_lock(&vc->lock);
>> +            /*
>> +             * Let something else run, and run pending guest-mode
>> +             * work (reschedule, and TIF_NOTIFY_RESUME task_work such
>> +             * as the deferred CFS throttle) before we would re-enter
>> +             * the guest, so a CPU-bound vCPU is actually throttled
>> +             * here instead of running past its quota. This is a
>> +             * superset of the old need_resched() check. Use the raw
>> +             * helper, not the kvm_ wrapper: signals (KVM_EXIT_INTR
>> +             * and the signal_exits stat) are accounted by this path's
>> +             * existing handling below, so going through the wrapper
>> +             * here would double-count them. The helper may schedule(),
>> +             * so the vcore lock is dropped around it.
>> +             */
>> +            spin_unlock(&vc->lock);
>> +            xfer_to_guest_mode_handle_work();
>> +            spin_lock(&vc->lock);
>>               if (vc->vcore_state == VCORE_PREEMPT)
>>                   kvmppc_vcore_end_preempt(vc);
>>           } else {
>> @@ -4899,8 +4923,21 @@ int kvmhv_run_single_vcpu(struct kvm_vcpu 
>> *vcpu, u64 time_limit,
>>           }
>>       }
>>   -    if (need_resched())
>> -        cond_resched();
>> +    /*
>> +     * Run pending work before (re-)entering the guest, most 
>> importantly
>> +     * task_work queued via TWA_RESUME (e.g. the deferred CFS bandwidth
>> +     * throttle, which only sets TIF_NOTIFY_RESUME). Without this a 
>> CPU-bound
>> +     * vCPU that keeps returning RESUME_GUEST never reaches an 
>> exit-to-user
>> +     * point, so the throttle is never enforced and the task runs 
>> far beyond
>> +     * its quota. The helper also handles reschedule and signals, 
>> replacing
>> +     * the cond_resched() that was here. It may schedule(), so it 
>> runs before
>> +     * preemption and IRQs are disabled, with no vcore/KVM locks 
>> held. This
>> +     * is the per-reentry site shared by the bare-metal and pseries 
>> (nested)
>> +     * paths, so both are covered.
>> +     */
>> +    r = kvm_xfer_to_guest_mode_handle_work(vcpu);
>> +    if (r)    /* -EINTR: signal pending, exit to userspace 
>> (KVM_EXIT_INTR) */
>> +        return r;
>>         kvmppc_update_vpas(vcpu);
>>   @@ -4916,7 +4953,14 @@ int kvmhv_run_single_vcpu(struct kvm_vcpu 
>> *vcpu, u64 time_limit,
>>         if (signal_pending(current))
>>           goto sigpend;
>
> xfer_to_guest_mode_work_pending checks for signals too right?
>
> #define XFER_TO_GUEST_MODE_WORK                                         \
>         (_TIF_NEED_RESCHED | _TIF_NEED_RESCHED_LAZY | _TIF_SIGPENDING | \
>          _TIF_NOTIFY_SIGNAL | _TIF_NOTIFY_RESUME |                      \
>          ARCH_XFER_TO_GUEST_MODE_WORK)
Yes, I see originally we only had need_resched() check. will remove in v2
>
>> -    if (need_resched() || !kvm->arch.mmu_ready)
>> +    /*
>> +     * Re-check for pending guest-mode work with IRQs disabled, to 
>> catch
>> +     * anything (e.g. a TIF_NOTIFY_RESUME task_work such as the 
>> deferred CFS
>> +     * throttle) that raced in after the check above. Bail back to 
>> the outer
>> +     * loop, which re-enters here and runs the work. This is a 
>> superset of
>> +     * the previous need_resched() check.
>> +     */
>> +    if (xfer_to_guest_mode_work_pending() || !kvm->arch.mmu_ready)
>>           goto out;
>>         vcpu->cpu = pcpu;
>
> Copying from the discussion thread at that time,
>
> """
> on x86:
> kvm_arch_vcpu_ioctl_run
>     vcpu_run
>         for () {
>             .. run guest..
>             xfer_to_guest_mode_handle_work
>                 schedule
>         }
>
>
> on Powerpc:  ( taking book3s_hv flavour):
> kvm_arch_vcpu_ioctl_run
> kvmppc_vcpu_run_hv  *1
>     do while() {
>         kvmhv_run_single_vcpu or kvmppc_run_vcpu
>             -- checking for need_resched and signals and bails out *2
>     }
>
>
> *1 - checks for need resched and signals before entering guest
> *2 - checks for need resched and signals while running the guest
>
>
> This patch is addressing only *1 but it needs to address *2 as well 
> using generic framework.

Not sure about *2 (while running the guest) part. Here the patch checks 
twice before entering the guest.
Once when IRQs are not disabled and once after disabling IRQs.

> I think it is doable for books3s_hv atleast. (though might need rewrite)
do what exactly?
> """
>
>
> A few questions that comes to my mind.
>
> 1. I think you are doing for both *1 and *2 right? Is *1 necessary 
> still for powerpc?
yes, before entering the guest we check twice. Once when IRQs are not 
disabled and once after disabling IRQs.
> 2. There is a for loop. What is it doing? Does this still need similar 
> checks?
>
>         if (is_kvmppc_resume_guest(r) && 
> !kvmppc_vcpu_check_block(vcpu)) {
>                 kvmppc_set_timer(vcpu);
>
>                 prepare_to_rcuwait(wait);
>                 for (;;) {
>                         set_current_state(TASK_INTERRUPTIBLE);
>                         if (signal_pending(current)) {
>                                 vcpu->stat.signal_exits++;
>                                 run->exit_reason = KVM_EXIT_INTR;
>                                 vcpu->arch.ret = -EINTR;
>                                 break;
>                         }
>
>                         if (kvmppc_vcpu_check_block(vcpu))
>                                 break;
>
>                         trace_kvmppc_vcore_blocked(vcpu, 0);
>                         schedule();
>                         trace_kvmppc_vcore_blocked(vcpu, 1);
>                 }
>                 finish_rcuwait(wait);
>         }
>         vcpu->arch.ceded = 0;

IIUC, the vCPU has voluntarily went idle in the guest and it's
about to re-enter, but there's genuinely nothing to deliver yet.
So, it parks itself here until, something actually wake this vCPU.

I am not quite sure, if signal_pending() should be replaced with
xfer_*_pending() check, because vCPU is not re-entering the guest.

Adding the xfer_*_pending() check here will break the loop even
when other flags causing busy spins (cede/re-enter).

>
> PS: cc'ing me would have helped me to see the mail earlier since you 
> had referred
> to the patch I had sent.

Yes. I missed it.