Understanding cpufreq?

[James Pearson <james-p@moving-picture.com>]

I have a number of dual CPU and dual CPU/dual core Opteron systems that 
are used as compute servers. In an effort to reduce power consumption 
and reduce heat output, I would like to make use of the PowerNow! 
capabilities to clock back the CPUs when the machines are idle.

These machines are running a 2.6.9-42 RHEL4 kernel with the powernow-k8 
module loaded - which I believe have backported cpufreq support from 
more recent mainline kernels.

In trying to achieve what I want, I've become rather confused as to how 
cpufreq in a multi-CPU environment works:

There is a directory under /sys/devices/system/cpu/cpu*/cpufreq for each 
CPU, which seems to imply that each CPU speed can be controlled 
separately - can this really be the case? Can separate CPU cores run at 
different speeds?

e.g. I can echo 4 different governor names to the scaling_governor file 
in each /sys/devices/system/cpu/cpu[0-3]/cpufreq directory on a 4 core 
machine - and the resulting scaling_cur_freq file can contain a 
different value.

However, the "cpu MHz" fields in /proc/cpuinfo are all the same for each 
each CPU - I assume the values in /proc/cpuinfo are the 'correct' values ??

Also, if I set all the governors to userspace, and then set each CPU's 
speed via scaling_setspeed to a different (allowed) value, then it 
appears quite random as to which value is then reflected in 
/proc/cpuinfo i.e. sometimes it will take the value given to CPU 0, 
other times it will be CPU 1 etc.

If I set all the governors to ondemand, the CPUs will from time to time, 
clock back their speed in situations where one or more CPUs are being 
heavily used. i.e it appears that each CPU is treated separately, and if 
one CPU is deemed to be idle enough by its given metrics, then it can 
reduce the speed of all CPUs, regardless of other CPUs being 'busy' ...

I've also tried a couple of userspace daemons (cpuspeed and powernowd) - 
again, these treat each CPU separately and will also reduce the speed of 
an 'idle' CPU - and hence reduce the speed of all the CPUs, again, 
regardless of other CPUs being 'busy'.

Essentially what I want to achieve is something like: if _any_ CPU is 
'busy' (usage over some threshold over some sampling period), then run 
at full speed and if _all_ CPUs are 'idle' (all below some threshold 
over some sampling period) then clock back the CPUs.

Is there something/some setting(s) that can do this in a multi-CPU machine?

Thanks

James Pearson