Understanding cpufreq?

Understanding cpufreq?

[James Pearson]

I have a number of dual CPU and dual CPU/dual core Opteron systems that 
are used as compute servers. In an effort to reduce power consumption 
and reduce heat output, I would like to make use of the PowerNow! 
capabilities to clock back the CPUs when the machines are idle.

These machines are running a 2.6.9-42 RHEL4 kernel with the powernow-k8 
module loaded - which I believe have backported cpufreq support from 
more recent mainline kernels.

In trying to achieve what I want, I've become rather confused as to how 
cpufreq in a multi-CPU environment works:

There is a directory under /sys/devices/system/cpu/cpu*/cpufreq for each 
CPU, which seems to imply that each CPU speed can be controlled 
separately - can this really be the case? Can separate CPU cores run at 
different speeds?

e.g. I can echo 4 different governor names to the scaling_governor file 
in each /sys/devices/system/cpu/cpu[0-3]/cpufreq directory on a 4 core 
machine - and the resulting scaling_cur_freq file can contain a 
different value.

However, the "cpu MHz" fields in /proc/cpuinfo are all the same for each 
each CPU - I assume the values in /proc/cpuinfo are the 'correct' values ??

Also, if I set all the governors to userspace, and then set each CPU's 
speed via scaling_setspeed to a different (allowed) value, then it 
appears quite random as to which value is then reflected in 
/proc/cpuinfo i.e. sometimes it will take the value given to CPU 0, 
other times it will be CPU 1 etc.

If I set all the governors to ondemand, the CPUs will from time to time, 
clock back their speed in situations where one or more CPUs are being 
heavily used. i.e it appears that each CPU is treated separately, and if 
one CPU is deemed to be idle enough by its given metrics, then it can 
reduce the speed of all CPUs, regardless of other CPUs being 'busy' ...

I've also tried a couple of userspace daemons (cpuspeed and powernowd) - 
again, these treat each CPU separately and will also reduce the speed of 
an 'idle' CPU - and hence reduce the speed of all the CPUs, again, 
regardless of other CPUs being 'busy'.

Essentially what I want to achieve is something like: if _any_ CPU is 
'busy' (usage over some threshold over some sampling period), then run 
at full speed and if _all_ CPUs are 'idle' (all below some threshold 
over some sampling period) then clock back the CPUs.

Is there something/some setting(s) that can do this in a multi-CPU machine?

Thanks

James Pearson

RE: Understanding cpufreq?

[Langsdorf, Mark]

> These machines are running a 2.6.9-42 RHEL4 kernel with the 
> powernow-k8 module loaded - which I believe have backported
> cpufreq support from more recent mainline kernels.

To an extent.  The problem is not the powernow-k8 driver,
but the cpufreq infrastructure.  See below.

> In trying to achieve what I want, I've become rather confused 
> as to how cpufreq in a multi-CPU environment works:
> 
> There is a directory under 
> /sys/devices/system/cpu/cpu*/cpufreq for each 
> CPU, which seems to imply that each CPU speed can be controlled 
> separately - can this really be the case? Can separate CPU 
> cores run at different speeds?

No, but prior to 2.6.10, cpufreq discovers each core
independently, and creates a directory independently.
The powernow-k8 knows which cores are on the same processors,
and handles them together.

After 2.6.10, cpufreq understands multicore devices and
creates 1 directory for the entire processor and then
creates symbolic links for all the cores that share the
same pstates.

> e.g. I can echo 4 different governor names to the 
> scaling_governor file in each
> /sys/devices/system/cpu/cpu[0-3]/cpufreq directory on 
> a 4 core machine - and the resulting scaling_cur_freq
> file can contain a different value.
> 
> However, the "cpu MHz" fields in /proc/cpuinfo are all the 
> same for each each CPU - I assume the values in /proc/cpuinfo
> are the 'correct' values ??
> 
> Also, if I set all the governors to userspace, and then set 
> each CPU's speed via scaling_setspeed to a different
> (allowed) value, then it appears quite random as to which
> value is then reflected in /proc/cpuinfo i.e. sometimes
> it will take the value given to CPU 0, 
> other times it will be CPU 1 etc.
 
It's always the highest requested value of all cores on
a processor.  powernow-k8 keeps track of the requested
frequencies, though, so you can have some weird situations.

ie, from boot up:
	core0 800 MHz	core1 800 MHz	CPU 800 MHz
	core0 to 2000 MHz	core1 800 MHz	CPU 2000 MHz
	core0 2000 MHz	core1 to 1800 MHz	CPU 2000 MHz
	core0 to 800 MHz	core1 1800 MHz	CPU 1800 MHz

> If I set all the governors to ondemand, the CPUs will from 
> time to time, clock back their speed in situations where
> one or more CPUs are being heavily used. i.e it appears
> that each CPU is treated separately, and if 
> one CPU is deemed to be idle enough by its given metrics,
> then it can reduce the speed of all CPUs, regardless of
> other CPUs being 'busy' ...

That shouldn't happen, unless ondemand requests a low
frequency to all cores.

> Essentially what I want to achieve is something like:
> if _any_ CPU is 'busy' (usage over some threshold over
> some sampling period), then run at full speed and
> if _all_ CPUs are 'idle' (all below some threshold 
> over some sampling period) then clock back the CPUs.

This should be happening automatically in RHEL4.

Note that the 2.6.10 kernel changed cpufreq behavior
dramatically, and now the most recent frequency request 
wins.

-Mark Langsdorf
powernow-k8 maintainer
AMD, Inc.