[PATCH v4 0/2] arm64/cpufreq: report and track frequencies above 4.19 GHz

[PATCH v4 0/2] arm64/cpufreq: report and track frequencies above 4.19 GHz

[Oleg Keri]

The Snapdragon X2 Elite (Glymur) is the first arm64 laptop part I have
seen whose boost OPP, 4723200 kHz, sits above 4194304 kHz.  Two
independent problems become visible there, both of which make the
kernel believe a boosted CPU is running slower than it is.

Patch 1 fixes an overflow in arch_freq_get_on_cpu(): the u64 product of
the frequency scale and the reference frequency is truncated to
unsigned int before being shifted back down, which wraps for any
reference frequency above 2^32 / SCHED_CAPACITY_SCALE = 4194304 kHz.

Patch 2 makes capacity_freq_ref include the boost frequencies.  For
frequency table drivers it is latched from policy->cpuinfo.max_freq on
CPUFREQ_CREATE_POLICY, and that value excludes the boost entries while
boost is off, so a machine that boots with boost disabled keeps the
sustained maximum as its reference forever.  On arm64 that saturates the
AMU frequency scale at SCHED_CAPACITY_SCALE once boost is enabled, so
the scheduler cannot distinguish a boosted CPU from one at the sustained
maximum, and arch_freq_get_on_cpu() cannot report above the reference.
With the boost maximum as the reference, a disabled boost is expressed
as cpufreq pressure instead, which is what CPPC based systems already
get from highest_perf.

The order matters: patch 2 puts the reference above 4194304 kHz on this
machine from boot, so patch 1 has to land with or before it.

Measured on a Lenovo Yoga Slim 7x Gen 11 (Glymur, scmi-cpufreq,
4032000 kHz sustained, 4723200 kHz boost, boost off at boot) with both
patches, pinning a big-core policy to a single OPP and sampling
cpuinfo_avg_freq under a single-threaded load:

  requested OPP    boost    cpuinfo_avg_freq
  --------------------------------------------
  4032000 kHz      off      4031325   (0.02% low)
  4723200 kHz      on       4709362   (0.29% low)

The reference of the big cores moves from 4032000 to 4723200 kHz, so
the little cores' cpu_capacity goes from 647 to 553 (647 * 4032000 /
4723200) while the big cores stay at 1024.  Without patch 1 the second
row wraps; v1 of this series measured 524283 kHz in that state.

Note that cpuinfo_cur_freq still reports 4032000 kHz at the boost OPP
on this machine.  That is a separate path -- scmi_dvfs_freq_get()
asking firmware for the current performance level -- with no clamp in
the kernel, and it is not addressed here.

Changes in v4:
- Patch 1: use cap_scale() and move its definition from
  kernel/sched/sched.h to include/linux/topology.h, as Dietmar
  proposed on v3.  Same semantics, one open-coded shift less.  Beata's
  Ack on v3 is not carried over since the diff changed.  The scheduler
  maintainers are on Cc for the header move.
- Patch 2: reworked after Dietmar's review of v3.  Instead of updating
  capacity_freq_ref from cpufreq when the boost state changes, take the
  highest frequency table entry, boost included, as the reference when
  the policy is created, and let cpufreq pressure express a disabled
  boost.  The cpufreq core hook and the arch_update_freq_ref() plumbing
  in the arm, arm64 and riscv headers are gone; the patch now touches
  drivers/base/arch_topology.c only.
- Link to v3: https://lore.kernel.org/all/20260910063440.4677-1-okerixx@gmail.com/

Changes in v3:
- Patch 1: drop the Suggested-by trailer.  It was not warranted for a
  review comment on the shape of an existing patch; my mistake.
- No code changes.
- Link to v2: https://lore.kernel.org/all/20260909192351.33910-1-okerixx@gmail.com/

Changes in v2:
- Patch 1: fold the multiply and the shift into a single expression, as
  suggested on v1, instead of reusing the u64 scale variable as scratch.
- Patch 2: unchanged.
- Link to v1: https://lore.kernel.org/all/cover.1788712186.git.okerixx@gmail.com/

Oleg Keri (2):
  arm64: topology: fix arch_freq_get_on_cpu() overflow above 4.19 GHz
  arch_topology: use the boost frequencies for capacity_freq_ref

 arch/arm64/kernel/topology.c |  5 +----
 drivers/base/arch_topology.c | 16 +++++++++++++++-
 include/linux/topology.h     |  2 ++
 kernel/sched/sched.h         |  2 --
 4 files changed, 18 insertions(+), 7 deletions(-)

-- 
2.55.0

base-commit: a7728f5e1fc3d472a314acdabca6039f71ec3a9d

[PATCH v4 1/2] arm64: topology: fix arch_freq_get_on_cpu() overflow above 4.19 GHz

[Oleg Keri]

arch_freq_get_on_cpu() computes the product of the frequency scale and
the reference frequency as a u64, but assigns it to an unsigned int
before shifting it back down:

	freq = scale * arch_scale_freq_ref(cpu);
	freq >>= SCHED_CAPACITY_SHIFT;

The product is truncated to 32 bits before the shift, so the result
wraps once arch_scale_freq_ref() exceeds 2^32 / SCHED_CAPACITY_SCALE,
i.e. 4194304 kHz.

On a Snapdragon X2 Elite (Glymur) laptop, whose boost OPP is 4723200
kHz, cpuinfo_avg_freq reports 524283 kHz instead of ~4723200 kHz while
the CPU demonstrably runs at the boost frequency: a fixed workload
completes in 1.72 s at the 4723200 kHz OPP versus 2.01 s at 4032000
kHz, matching the 1.171 frequency ratio.

Use cap_scale(), which the scheduler already has for exactly this
capacity scaling, so the multiply and the shift stay in 64 bits and
only the final value is narrowed by the return type. Move the macro
from the scheduler's private header to <linux/topology.h> so it can be
used outside kernel/sched.

Fixes: 16d1e27475f6 ("arm64: Provide an AMU-based version of arch_freq_get_on_cpu")
Signed-off-by: Oleg Keri <okerixx@gmail.com>
---
 arch/arm64/kernel/topology.c | 5 +----
 include/linux/topology.h     | 2 ++
 kernel/sched/sched.h         | 2 --
 3 files changed, 3 insertions(+), 6 deletions(-)

diff --git a/arch/arm64/kernel/topology.c b/arch/arm64/kernel/topology.c
index d28438f8b83f..07b8c497c9e6 100644
--- a/arch/arm64/kernel/topology.c
+++ b/arch/arm64/kernel/topology.c
@@ -186,7 +186,6 @@ int arch_freq_get_on_cpu(int cpu)
 	struct amu_cntr_sample *amu_sample;
 	unsigned int start_cpu = cpu;
 	unsigned long last_update;
-	unsigned int freq = 0;
 	u64 scale;
 
 	if (!amu_fie_cpu_supported(cpu) || !arch_scale_freq_ref(cpu))
@@ -245,9 +244,7 @@ int arch_freq_get_on_cpu(int cpu)
 	 * (see amu_scale_freq_tick for details)
 	 */
 	scale = arch_scale_freq_capacity(cpu);
-	freq = scale * arch_scale_freq_ref(cpu);
-	freq >>= SCHED_CAPACITY_SHIFT;
-	return freq;
+	return cap_scale(arch_scale_freq_ref(cpu), scale);
 }
 
 static void amu_fie_setup(const struct cpumask *cpus)
diff --git a/include/linux/topology.h b/include/linux/topology.h
index 709a2dcf4c73..0a4ee12a98d5 100644
--- a/include/linux/topology.h
+++ b/include/linux/topology.h
@@ -351,4 +351,6 @@ static inline unsigned long topology_get_cpu_scale(int cpu)
 
 void topology_set_cpu_scale(unsigned int cpu, unsigned long capacity);
 
+#define cap_scale(v, s)		((v)*(s) >> SCHED_CAPACITY_SHIFT)
+
 #endif /* _LINUX_TOPOLOGY_H */
diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
index 6c3ad70e58b8..45796fccdc84 100644
--- a/kernel/sched/sched.h
+++ b/kernel/sched/sched.h
@@ -236,8 +236,6 @@ static inline int task_has_dl_policy(struct task_struct *p)
 	return dl_policy(p->policy);
 }
 
-#define cap_scale(v, s)		((v)*(s) >> SCHED_CAPACITY_SHIFT)
-
 static inline void update_avg(u64 *avg, u64 sample)
 {
 	s64 diff = sample - *avg;
-- 
2.55.0

Re: [PATCH v4 1/2] arm64: topology: fix arch_freq_get_on_cpu() overflow above 4.19 GHz

[Peter Zijlstra]

On Thu, Sep 17, 2026 at 02:51:11PM +0200, Oleg Keri wrote:
> arch_freq_get_on_cpu() computes the product of the frequency scale and
> the reference frequency as a u64, but assigns it to an unsigned int
> before shifting it back down:
> 
> 	freq = scale * arch_scale_freq_ref(cpu);
> 	freq >>= SCHED_CAPACITY_SHIFT;
> 
> The product is truncated to 32 bits before the shift, so the result
> wraps once arch_scale_freq_ref() exceeds 2^32 / SCHED_CAPACITY_SCALE,
> i.e. 4194304 kHz.
> 
> On a Snapdragon X2 Elite (Glymur) laptop, whose boost OPP is 4723200
> kHz, cpuinfo_avg_freq reports 524283 kHz instead of ~4723200 kHz while
> the CPU demonstrably runs at the boost frequency: a fixed workload
> completes in 1.72 s at the 4723200 kHz OPP versus 2.01 s at 4032000
> kHz, matching the 1.171 frequency ratio.
> 
> Use cap_scale(), which the scheduler already has for exactly this
> capacity scaling, so the multiply and the shift stay in 64 bits and
> only the final value is narrowed by the return type. Move the macro
> from the scheduler's private header to <linux/topology.h> so it can be
> used outside kernel/sched.
> 
> Fixes: 16d1e27475f6 ("arm64: Provide an AMU-based version of arch_freq_get_on_cpu")
> Signed-off-by: Oleg Keri <okerixx@gmail.com>
> ---
>  arch/arm64/kernel/topology.c | 5 +----
>  include/linux/topology.h     | 2 ++
>  kernel/sched/sched.h         | 2 --
>  3 files changed, 3 insertions(+), 6 deletions(-)
> 
> diff --git a/arch/arm64/kernel/topology.c b/arch/arm64/kernel/topology.c
> index d28438f8b83f..07b8c497c9e6 100644
> --- a/arch/arm64/kernel/topology.c
> +++ b/arch/arm64/kernel/topology.c
> @@ -186,7 +186,6 @@ int arch_freq_get_on_cpu(int cpu)
>  	struct amu_cntr_sample *amu_sample;
>  	unsigned int start_cpu = cpu;
>  	unsigned long last_update;
> -	unsigned int freq = 0;
>  	u64 scale;
>  
>  	if (!amu_fie_cpu_supported(cpu) || !arch_scale_freq_ref(cpu))
> @@ -245,9 +244,7 @@ int arch_freq_get_on_cpu(int cpu)
>  	 * (see amu_scale_freq_tick for details)
>  	 */
>  	scale = arch_scale_freq_capacity(cpu);
> -	freq = scale * arch_scale_freq_ref(cpu);
> -	freq >>= SCHED_CAPACITY_SHIFT;
> -	return freq;
> +	return cap_scale(arch_scale_freq_ref(cpu), scale);
>  }
>  
>  static void amu_fie_setup(const struct cpumask *cpus)
> diff --git a/include/linux/topology.h b/include/linux/topology.h
> index 709a2dcf4c73..0a4ee12a98d5 100644
> --- a/include/linux/topology.h
> +++ b/include/linux/topology.h
> @@ -351,4 +351,6 @@ static inline unsigned long topology_get_cpu_scale(int cpu)
>  
>  void topology_set_cpu_scale(unsigned int cpu, unsigned long capacity);
>  
> +#define cap_scale(v, s)		((v)*(s) >> SCHED_CAPACITY_SHIFT)

This might not be a very good generic helper, since it relies on either
of the variables to be u64 for correctness.

Re: [PATCH v4 1/2] arm64: topology: fix arch_freq_get_on_cpu() overflow above 4.19 GHz

[Dietmar Eggemann]

On 17.09.26 17:32, Peter Zijlstra wrote:
> On Thu, Sep 17, 2026 at 02:51:11PM +0200, Oleg Keri wrote:

[...]
>> diff --git a/include/linux/topology.h b/include/linux/topology.h
>> index 709a2dcf4c73..0a4ee12a98d5 100644
>> --- a/include/linux/topology.h
>> +++ b/include/linux/topology.h
>> @@ -351,4 +351,6 @@ static inline unsigned long topology_get_cpu_scale(int cpu)
>>  
>>  void topology_set_cpu_scale(unsigned int cpu, unsigned long capacity);
>>  
>> +#define cap_scale(v, s)		((v)*(s) >> SCHED_CAPACITY_SHIFT)
> 
> This might not be a very good generic helper, since it relies on either
> of the variables to be u64 for correctness.

So moving it to a static inline function instead?

static inline u64 cap_scale(u64 value, u64 scale)
{
       return value * scale >> SCHED_CAPACITY_SHIFT;
}

Re: [PATCH v4 1/2] arm64: topology: fix arch_freq_get_on_cpu() overflow above 4.19 GHz

[Peter Zijlstra]

On Thu, Sep 17, 2026 at 05:55:44PM +0200, Dietmar Eggemann wrote:
> On 17.09.26 17:32, Peter Zijlstra wrote:
> > On Thu, Sep 17, 2026 at 02:51:11PM +0200, Oleg Keri wrote:
> 
> [...]
> >> diff --git a/include/linux/topology.h b/include/linux/topology.h
> >> index 709a2dcf4c73..0a4ee12a98d5 100644
> >> --- a/include/linux/topology.h
> >> +++ b/include/linux/topology.h
> >> @@ -351,4 +351,6 @@ static inline unsigned long topology_get_cpu_scale(int cpu)
> >>  
> >>  void topology_set_cpu_scale(unsigned int cpu, unsigned long capacity);
> >>  
> >> +#define cap_scale(v, s)		((v)*(s) >> SCHED_CAPACITY_SHIFT)
> > 
> > This might not be a very good generic helper, since it relies on either
> > of the variables to be u64 for correctness.
> 
> So moving it to a static inline function instead?
> 
> static inline u64 cap_scale(u64 value, u64 scale)
> {
>        return value * scale >> SCHED_CAPACITY_SHIFT;
> }

Yes, that would be much better defined.

[PATCH v4 2/2] arch_topology: use the boost frequencies for capacity_freq_ref

[Oleg Keri]

capacity_freq_ref is latched from policy->cpuinfo.max_freq by
init_cpu_capacity_callback() on CPUFREQ_CREATE_POLICY.
cpufreq_frequency_table_cpuinfo() excludes CPUFREQ_BOOST_FREQ entries
while boost is disabled, so on a system that boots with boost off the
reference is the non-boost maximum and stays there.

Once boost is enabled the CPU runs above its reference.  On arm64, where
the AMU drives frequency invariance, amu_scale_freq_tick() caps the scale
at SCHED_CAPACITY_SCALE, so the scheduler cannot tell a boosted CPU from
one at the sustained maximum, and arch_freq_get_on_cpu() cannot report
above the reference, so cpuinfo_avg_freq is pinned to it.

CPPC based systems do not have the problem: their reference comes from
highest_perf, and a disabled boost shows up as cpufreq pressure.  Do the
same for frequency table drivers and take the highest valid table entry,
boost entries included, as the reference.  With boost off
cpufreq_update_pressure() then reports the difference as pressure.

On a Snapdragon X2 Elite (Glymur) laptop with a 4032000 kHz sustained
and a 4723200 kHz boost OPP this moves the reference of the big cores
from 4032000 to 4723200 kHz.

Signed-off-by: Oleg Keri <okerixx@gmail.com>
---
 drivers/base/arch_topology.c | 16 +++++++++++++++-
 1 file changed, 15 insertions(+), 1 deletion(-)

diff --git a/drivers/base/arch_topology.c b/drivers/base/arch_topology.c
index 8c5e47c28d9a..300f64ffea2f 100644
--- a/drivers/base/arch_topology.c
+++ b/drivers/base/arch_topology.c
@@ -386,6 +386,20 @@ static cpumask_var_t cpus_to_visit;
 static void parsing_done_workfn(struct work_struct *work);
 static DECLARE_WORK(parsing_done_work, parsing_done_workfn);
 
+static unsigned int topology_policy_max_freq(struct cpufreq_policy *policy)
+{
+	unsigned int max_freq = policy->cpuinfo.max_freq;
+	struct cpufreq_frequency_table *pos;
+
+	if (!policy->freq_table)
+		return max_freq;
+
+	cpufreq_for_each_valid_entry(pos, policy->freq_table)
+		max_freq = max(max_freq, pos->frequency);
+
+	return max_freq;
+}
+
 static int
 init_cpu_capacity_callback(struct notifier_block *nb,
 			   unsigned long val,
@@ -404,7 +418,7 @@ init_cpu_capacity_callback(struct notifier_block *nb,
 	cpumask_andnot(cpus_to_visit, cpus_to_visit, policy->related_cpus);
 
 	for_each_cpu(cpu, policy->related_cpus) {
-		per_cpu(capacity_freq_ref, cpu) = policy->cpuinfo.max_freq;
+		per_cpu(capacity_freq_ref, cpu) = topology_policy_max_freq(policy);
 		freq_inv_set_max_ratio(cpu,
 				       per_cpu(capacity_freq_ref, cpu) * HZ_PER_KHZ);
 	}
-- 
2.55.0

Re: [PATCH v4 2/2] arch_topology: use the boost frequencies for capacity_freq_ref

[Dietmar Eggemann]

On 17.09.26 14:51, Oleg Keri wrote:
> capacity_freq_ref is latched from policy->cpuinfo.max_freq by
> init_cpu_capacity_callback() on CPUFREQ_CREATE_POLICY.
> cpufreq_frequency_table_cpuinfo() excludes CPUFREQ_BOOST_FREQ entries
> while boost is disabled, so on a system that boots with boost off the
> reference is the non-boost maximum and stays there.
> 
> Once boost is enabled the CPU runs above its reference.  On arm64, where
> the AMU drives frequency invariance, amu_scale_freq_tick() caps the scale
> at SCHED_CAPACITY_SCALE, so the scheduler cannot tell a boosted CPU from
> one at the sustained maximum, and arch_freq_get_on_cpu() cannot report
> above the reference, so cpuinfo_avg_freq is pinned to it.
> 
> CPPC based systems do not have the problem: their reference comes from
> highest_perf, and a disabled boost shows up as cpufreq pressure.  Do the
> same for frequency table drivers and take the highest valid table entry,
> boost entries included, as the reference.  With boost off
> cpufreq_update_pressure() then reports the difference as pressure.

Ah, you want to fix this for 'frequency table' drivers.

IMHO,
https://lore.kernel.org/r/20260908-schedutil-boost-frequency-handling-v2-0-25312a713699@oss.qualcomm.com
is trying the same at the moment.

[...]

Re: [PATCH v4 2/2] arch_topology: use the boost frequencies for capacity_freq_ref

[Oleg Keri]

Hi Dietmar,

On Thu, Sep 17, 2026, Dietmar Eggemann wrote:
> Ah, you want to fix this for 'frequency table' drivers.
>
> IMHO,
> https://lore.kernel.org/r/20260908-schedutil-boost-frequency-handling-v2-0-25312a713699@oss.qualcomm.com
> is trying the same at the moment.

Thanks, I had missed that series. Please drop my 2/2 in favour of it.

Patch 1/2 here is still needed, and becomes a prerequisite for his series
on this SoC: with the reference at 4723200 kHz from boot,
arch_freq_get_on_cpu() wraps and cpuinfo_avg_freq reads about 524283 kHz
unless the multiply stays in 64 bits.

I will test his series on the Yoga Slim 7x Gen 11 and reply there.

Thanks,
Oleg