[PATCH] x86/aperfmperf: Refresh stale sample via IPI for busy NOHZ_FULL CPUs

[PATCH] x86/aperfmperf: Refresh stale sample via IPI for busy NOHZ_FULL CPUs

[Jing Wu]

An isolated CPU covered by nohz_full stops its periodic tick once it
has only one runnable task, since sched_can_stop_tick() only checks
scheduling-class fairness and has no notion of cpufreq reporting
needs. arch_scale_freq_tick() runs only from scheduler_tick(), so
cpu_samples for that CPU is never refreshed again.

arch_freq_get_on_cpu() then permanently hits its staleness check and
falls back to cpufreq_quick_get(), which returns whatever policy->cur
was left at (typically the P-state floor). This happens even though
HWP hardware keeps running the CPU at full turbo autonomously, as
confirmed by turbostat and by directly reading APERF/MPERF.

Reproduce on an isolated, nohz_full CPU with intel_pstate/HWP by
loading it and watching scaling_cur_freq stay pinned at the floor:

    taskset -c $CPU stress --cpu 1 &
    for i in $(seq 10); do
        cat /sys/devices/system/cpu/cpu$CPU/cpufreq/scaling_cur_freq
        sleep 0.5
    done
    turbostat --cpu $CPU --interval 1 --num_iterations 5

scaling_cur_freq stays at the floor for the whole run, while
turbostat's Bzy_MHz confirms the CPU is actually at full turbo.

Refresh the stale sample with one on-demand arch_scale_freq_tick()
via IPI before falling back, but only when the target CPU is online
and not idle. APERF/MPERF both stop advancing during idle (C1+), so
a delta computed over an arbitrarily long stale window still yields
a correct busy-time frequency average.

Fixes: 7d84c1ebf9dd ("x86/aperfmperf: Replace aperfmperf_get_khz()")
Co-developed-by: Qiliang Yuan <yuanql9@chinatelecom.cn>
Signed-off-by: Qiliang Yuan <yuanql9@chinatelecom.cn>
Co-developed-by: Jian Zhang <zhangj332@chinatelecom.cn>
Signed-off-by: Jian Zhang <zhangj332@chinatelecom.cn>
Signed-off-by: Jing Wu <realwujing@gmail.com>
---
 arch/x86/kernel/cpu/aperfmperf.c | 34 +++++++++++++++++++++++++++++++++-
 1 file changed, 33 insertions(+), 1 deletion(-)

diff --git a/arch/x86/kernel/cpu/aperfmperf.c b/arch/x86/kernel/cpu/aperfmperf.c
index 7ffc78d5ebf21..e51544db9f9bb 100644
--- a/arch/x86/kernel/cpu/aperfmperf.c
+++ b/arch/x86/kernel/cpu/aperfmperf.c
@@ -12,6 +12,7 @@
 #include <linux/math64.h>
 #include <linux/percpu.h>
 #include <linux/rcupdate.h>
+#include <linux/sched.h>
 #include <linux/sched/isolation.h>
 #include <linux/sched/topology.h>
 #include <linux/smp.h>
@@ -503,16 +504,41 @@ void arch_scale_freq_tick(void)
  */
 #define MAX_SAMPLE_AGE	((unsigned long)HZ / 50)
 
+static void aperfmperf_snapshot_cpu_ipi(void *info)
+{
+	arch_scale_freq_tick();
+}
+
+/*
+ * A NOHZ_FULL CPU with a single runnable task (e.g. an isolated CPU running
+ * a pinned PMD/busy-poll workload) stops its periodic tick, so nothing ever
+ * calls arch_scale_freq_tick() for it again and cpu_samples goes stale
+ * forever, not just for one MAX_SAMPLE_AGE window. Force one on-demand
+ * sample via IPI so a deliberate frequency read doesn't report the P-state
+ * floor from before isolation took effect. Skip idle CPUs: their frequency
+ * genuinely doesn't matter and there is no point poking them with an IPI.
+ */
+static bool aperfmperf_refresh_stale_sample(int cpu)
+{
+	if (!cpu_online(cpu) || idle_cpu(cpu))
+		return false;
+
+	smp_call_function_single(cpu, aperfmperf_snapshot_cpu_ipi, NULL, 1);
+	return true;
+}
+
 int arch_freq_get_on_cpu(int cpu)
 {
 	struct aperfmperf *s = per_cpu_ptr(&cpu_samples, cpu);
 	unsigned int seq, freq;
 	unsigned long last;
+	bool refreshed = false;
 	u64 acnt, mcnt;
 
 	if (!cpu_feature_enabled(X86_FEATURE_APERFMPERF))
 		goto fallback;
 
+again:
 	do {
 		seq = raw_read_seqcount_begin(&s->seq);
 		last = s->last_update;
@@ -524,8 +550,14 @@ int arch_freq_get_on_cpu(int cpu)
 	 * Bail on invalid count and when the last update was too long ago,
 	 * which covers idle and NOHZ full CPUs.
 	 */
-	if (!mcnt || (jiffies - last) > MAX_SAMPLE_AGE)
+	if (!mcnt || (jiffies - last) > MAX_SAMPLE_AGE) {
+		if (!refreshed) {
+			refreshed = true;
+			if (aperfmperf_refresh_stale_sample(cpu))
+				goto again;
+		}
 		goto fallback;
+	}
 
 	return div64_u64((cpu_khz * acnt), mcnt);
 

---
base-commit: 502d801f0ab03e4f32f9a33d203154ce84887921
change-id: 20260728-bug-isolatecpu-cpufreq-864a5fba85b7

Best regards,
-- 
Jing Wu <realwujing@gmail.com>

Re: [PATCH] x86/aperfmperf: Refresh stale sample via IPI for busy NOHZ_FULL CPUs

[Peter Zijlstra]

On Tue, Jul 28, 2026 at 07:27:54PM +0800, Jing Wu wrote:
> An isolated CPU covered by nohz_full stops its periodic tick once it
> has only one runnable task, since sched_can_stop_tick() only checks
> scheduling-class fairness and has no notion of cpufreq reporting
> needs. arch_scale_freq_tick() runs only from scheduler_tick(), so
> cpu_samples for that CPU is never refreshed again.

Sending IPIs to NOHZ_FULL Cpus should be a no-no.

Re: [PATCH] x86/aperfmperf: Refresh stale sample via IPI for busy NOHZ_FULL CPUs

[Peter Zijlstra]

On Tue, Jul 28, 2026 at 07:27:54PM +0800, Jing Wu wrote:
> An isolated CPU covered by nohz_full stops its periodic tick once it
> has only one runnable task, since sched_can_stop_tick() only checks
> scheduling-class fairness and has no notion of cpufreq reporting
> needs. arch_scale_freq_tick() runs only from scheduler_tick(), so
> cpu_samples for that CPU is never refreshed again.
> 
> arch_freq_get_on_cpu() then permanently hits its staleness check and
> falls back to cpufreq_quick_get(), which returns whatever policy->cur
> was left at (typically the P-state floor). This happens even though
> HWP hardware keeps running the CPU at full turbo autonomously, as
> confirmed by turbostat and by directly reading APERF/MPERF.
> 
> Reproduce on an isolated, nohz_full CPU with intel_pstate/HWP by
> loading it and watching scaling_cur_freq stay pinned at the floor:
> 
>     taskset -c $CPU stress --cpu 1 &
>     for i in $(seq 10); do
>         cat /sys/devices/system/cpu/cpu$CPU/cpufreq/scaling_cur_freq
>         sleep 0.5
>     done
>     turbostat --cpu $CPU --interval 1 --num_iterations 5
> 
> scaling_cur_freq stays at the floor for the whole run, while
> turbostat's Bzy_MHz confirms the CPU is actually at full turbo.
> 
> Refresh the stale sample with one on-demand arch_scale_freq_tick()
> via IPI before falling back, but only when the target CPU is online
> and not idle. APERF/MPERF both stop advancing during idle (C1+), so
> a delta computed over an arbitrarily long stale window still yields
> a correct busy-time frequency average.

Aside from the fact that sending IPIs to NOHZ_FULL is just plain wrong,
this whole thing makes no sense.

When the CPU is isolated, nothing should care about the ratio anyway.
Just set the thing to '1' (1024) when the CPU enters NOHZ_FULL mode and
ensure it isn't ever modified.

Re: [PATCH] x86/aperfmperf: Refresh stale sample via IPI for busy NOHZ_FULL CPUs

[Jing Wu]

On Tue, Jul 28, 2026 at 04:42:24PM +0200, Peter Zijlstra wrote:
> Aside from the fact that sending IPIs to NOHZ_FULL is just plain wrong,
> this whole thing makes no sense.
>
> When the CPU is isolated, nothing should care about the ratio anyway.
> Just set the thing to '1' (1024) when the CPU enters NOHZ_FULL mode and
> ensure it isn't ever modified.

Fair, understood.

For context on why I went looking in the first place: stressing an
isolated, nohz_full CPU shows both /proc/cpuinfo's "cpu MHz" and
/sys/devices/system/cpu/cpuN/cpufreq/scaling_cur_freq stuck at the
P-state floor (e.g. 800MHz) for as long as the CPU stays busy and
isolated, while turbostat confirms the hardware is actually running
at full turbo (e.g. 3.2GHz) the whole time. Both interfaces go
through arch_freq_get_on_cpu(), so whatever affects one affects both.

Getting the exact value would need an on-demand rdmsr on the target
CPU - which is what turbostat itself does via /dev/cpu/N/msr's
rdmsr_safe_regs_on_cpu(), i.e. the same smp_call_function_single()
IPI, just triggered manually by a human running a diagnostic tool
instead of sitting behind a commonly-polled sysfs file.

I looked for a way around that: PCU mailbox telemetry can expose a
per-core P-state on some Xeon uncores without touching the target
CPU, and HFI publishes a shared table too, but that's a per-core
performance/efficiency class, not an achieved clock, and PCU access
is uncore/generation-specific, not a general mechanism. So as far as
I can tell there's no way to get the exact value for an isolated CPU
without an IPI of some form.

Thanks,
Jing Wu

Re: [PATCH] x86/aperfmperf: Refresh stale sample via IPI for busy NOHZ_FULL CPUs

[Peter Zijlstra]

On Wed, Jul 29, 2026 at 04:22:24PM +0800, Jing Wu wrote:
> On Tue, Jul 28, 2026 at 04:42:24PM +0200, Peter Zijlstra wrote:
> > Aside from the fact that sending IPIs to NOHZ_FULL is just plain wrong,
> > this whole thing makes no sense.
> >
> > When the CPU is isolated, nothing should care about the ratio anyway.
> > Just set the thing to '1' (1024) when the CPU enters NOHZ_FULL mode and
> > ensure it isn't ever modified.
> 
> Fair, understood.
> 
> For context on why I went looking in the first place: stressing an
> isolated, nohz_full CPU shows both /proc/cpuinfo's "cpu MHz" and
> /sys/devices/system/cpu/cpuN/cpufreq/scaling_cur_freq stuck at the
> P-state floor (e.g. 800MHz) for as long as the CPU stays busy and
> isolated, while turbostat confirms the hardware is actually running
> at full turbo (e.g. 3.2GHz) the whole time. Both interfaces go
> through arch_freq_get_on_cpu(), so whatever affects one affects both.
> 
> Getting the exact value would need an on-demand rdmsr on the target
> CPU - which is what turbostat itself does via /dev/cpu/N/msr's
> rdmsr_safe_regs_on_cpu(), i.e. the same smp_call_function_single()
> IPI, just triggered manually by a human running a diagnostic tool
> instead of sitting behind a commonly-polled sysfs file.
> 
> I looked for a way around that: PCU mailbox telemetry can expose a
> per-core P-state on some Xeon uncores without touching the target
> CPU, and HFI publishes a shared table too, but that's a per-core
> performance/efficiency class, not an achieved clock, and PCU access
> is uncore/generation-specific, not a general mechanism. So as far as
> I can tell there's no way to get the exact value for an isolated CPU
> without an IPI of some form.

Oh, you care about the silly sysfs files? I though this was about the
scheduler use of aperf/mperf ratio.

Both are driven from the same source, but the scheduler use makes no
sense when isolated/NOHZ_FULL. And I would argue that keeping the CPU
isolated is more important than having the silly number 'accurate'.

Something like so perhaps?

diff --git a/arch/x86/kernel/cpu/aperfmperf.c b/arch/x86/kernel/cpu/aperfmperf.c
index 7ffc78d5ebf2..cedb40e6b5e6 100644
--- a/arch/x86/kernel/cpu/aperfmperf.c
+++ b/arch/x86/kernel/cpu/aperfmperf.c
@@ -438,7 +438,8 @@ static void scale_freq_tick(u64 acnt, u64 mcnt)
 {
 	u64 freq_scale, freq_ratio;
 
-	if (!arch_scale_freq_invariant())
+	if (!arch_scale_freq_invariant() ||
+	    !housekeeping_cpu(smp_processor_id(), HK_TYPE_TICK))
 		return;
 
 	if (check_shl_overflow(acnt, 2*SCHED_CAPACITY_SHIFT, &acnt))
@@ -510,6 +511,9 @@ int arch_freq_get_on_cpu(int cpu)
 	unsigned long last;
 	u64 acnt, mcnt;
 
+	if (!housekeeping_cpu(cpu, HK_TYPE_TICK))
+		return -EOPNOTSUPP;
+
 	if (!cpu_feature_enabled(X86_FEATURE_APERFMPERF))
 		goto fallback;
 

Re: [PATCH] x86/aperfmperf: Refresh stale sample via IPI for busy NOHZ_FULL CPUs

[Jing Wu]

On Wed, Jul 29, 2026 at 08:39:12PM +0800, Peter Zijlstra wrote:
> Oh, you care about the silly sysfs files? I though this was about the
> scheduler use of aperf/mperf ratio.
>
> Both are driven from the same source, but the scheduler use makes no
> sense when isolated/NOHZ_FULL. And I would argue that keeping the CPU
> isolated is more important than having the silly number 'accurate'.

Yes, sorry for the confusion - it's the sysfs files. To be concrete
about why: when someone is chasing a real incident (packet latency in
an OVS/DPDK PMD pipeline pinned to one of these cores, say), reading
scaling_cur_freq/cpuinfo is one of the first things they check. Seeing
the P-state floor while the core is genuinely at full turbo sends that
investigation down the wrong path and burns real engineering time.

-EOPNOTSUPP doesn't fix that particular problem: the operator still
gets nothing useful from the standard interface and has to fall back
to turbostat regardless, so the diagnostic workflow ends up no better
off than it is today.

Given that, I'd like to make the case for going back to something
closer to the original approach: it's a small, self-contained change,
and it reports the actual value instead of an estimate or nothing.
The concern was IPIs turning periodic if something scrapes this file
on a schedule - but for the isolated cores we run PMD workloads on,
nothing polls per-core scaling_cur_freq/cpuinfo on any kind of
schedule; it only gets read by a person during an investigation, which
is a rare, deliberate action, not unlike running turbostat by hand.

Would you reconsider on that basis, or is there a middle ground you'd
be open to?