Re:smp_send_reschedule vs. smp_migrate_task

[James Bottomley <James.Bottomley@SteelEye.com>]

> I am looking at the updates for PPC that are needed because of the
> changes to the scheduler in 2.5.x.  I need to implement
> smp_migrate_task(), but I do not have another IPI easily available;
> the Open PIC interrupt controller used in a lot of SMP PPC machines
> supports 4 IPIs in hardware and we are already using all of them.

I have this problem with the older voyager architectures that effectively have 
only one IPI.  I solved it by using a per-cpu area mailbox for IPIs (to send 
an IPI, set the bit in the per CPU area for all CPUs you want to send it to 
and then send of the global IPI.  Each receiving CPU clears the bit in their 
area before they begin processing the particular IPI).  It's not very 
efficient through (especially with cache transfers from sender to recipient), 
so using what you have is probably better.

> Thus I was thinking of using the same IPI for smp_migrate_task and
> smp_send_reschedule.  The idea is that smp_send_reschedule(cpu) will
> be effectively smp_migrate_task(cpu, NULL), and the code that receives
> that IPI will check for the NULL and do set_need_resched() instead of
> sched_task_migrated().

I wouldn't necessarily do this.  smp_send_reschedule is used a lot by the 
scheduler and is designed to be fast execution (i.e. you just send out the IPI 
and continue).  Adding spinlocks to this code will slow it down and add cache 
thrashing contention.

smp_migrate_task is designed to be fast executing, but it is only used in 
set_cpus_allowed(), which is rarely called and is not time critical.  Why not 
use the smp_call_function interface instead?  The semantics aren't entirely 
the same, but I don't believe the impact to set_cpus_allowed() will be felt at 
all.

James Bottomley