Re: Hotplug CPU support for Oprofile

Re: Hotplug CPU support for Oprofile

[Mike Travis]

[adding some possibly interested parties to the cc line.]

Chris J Arges wrote:
> Mike,
> I was wondering what your rationale behind using per_cpu variables for
> the cpu buffers in oprofile versus using percpu_alloc, percpu_free?
> For instance:
> 
>         DEFINE_PER_CPU_SHARED_ALIGNED(struct oprofile_cpu_buffer,
>         cpu_buffer);
> 
>         void free_cpu_buffers(void)
>         {
>                 int i;
>         
>                 for_each_online_cpu(i)
>                         vfree(per_cpu(cpu_buffer, i).buffer);
>         
>         }
> 
> Could it be possible to write it like this:
>         
>         struct oprofile_cpu_buffer *cpu_buffer;
>         
>         void free_cpu_buffers(void)
>         {
>         	percpu_free(cpu_buffer);
>         }
> 

        
Hi Chris,

There was no particular reason for one or the other.  We did have a
patch a few months ago that added CPU_ALLOC functionality that was
much cleaner than percpu_alloc (and replaced it), but due to
circumstances it didn't actually make it into the source base.
The percpu approach was the fallback.

> 
> Phillipe, Mike,
> I am currently working on a patch for enabling hotplug cpu support for
> oprofile as currently oprofile crashes whenever cpus are onlined of
> offlined. So far my patch registers a callback function for hotplug
> events, and checked if a cpu has been onlined/offlined and then
> allocs/frees a cpu buffer accordingly. 
> 
> I am wondering if you have any further recommendations as to approaching
> adding hotplug support. I am still fairly new to oprofile/linux kernel
> but I would like to contribute, please forgive any newbie mistakes.
> Thanks!
> 

Other than insuring that any routines for bringing up/tearing down
processors are in the __cpuinit/data sections and test as many
different configs as you can.

Also, increasing NR_CPUS to 4096 is one thing but I've been testing
with "possible" cpus == 4096 and have hit some snags.  The biggest
right now is the enormous amount of memory that the kernel tracing
facility requires.  So trying out your code with a healthy amount of
possible cpus will expose problems like this.

Another thing that I've encountered is that you can add "additional_cpus"
to the boot line, and the BIOS can signal that there are cpus present
but currently disabled, but there's no way to bring these cpus online
after bootup.

One thought would be to register the cpu but make it offline.  This
has some advantages but it was felt that instead something more like:

echo '32-63' > /sys/devices/system/cpu/online

would startup and register cpus 32-63.  This is orthogonal to the
showing of which cpu's are online:

# cat /sys/devices/system/cpu/online
0-31

Any suggestions along this area are welcome!

Thanks,
Mike

Re: Hotplug CPU support for Oprofile

[Christoph Lameter]

The allocpercpu approach creates a pointer array and results in objects 
distributed over all of memory. A cpu alloc approach would place 
dynamically allocated per cpu objects together which is better for memory 
consumption (no need to align) and performance (same TLB entry, neigboring 
memory cells, possible shared cachelines).