David, thanks for talking this through. I attach the latest patch that
I think corrects the problems identified earlier and should serve as the base
discussion for our discussion.


> Sent by:  davidel@ewok.dev.mcafeelabs.com
>
> To:   Hubertus Franke/Watson/IBM@IBMUS
> cc:   mkravetz@sequent.com, ak@suse.de
> Subject:  Re: [Lse-tech] Re: CPU affinity & IPI latency (FIX)_
>
> On 18-Jul-2001 Hubertus Franke wrote:
> >
> > The only reason one might want to put it outside is to force a shorter
> > schedule function, as this one only takes cpu_schedule(cpu) rather
> > then running the whole list.
> > I moved it inside the function.
>
> Maybe we're talking about two different things :)
> What's the line number you refer to ?

I am talking about this line in the fast track of reschedule_idle().
The current thing that we seem to agree on is
this (and that's in the atached whole patch)

@@ -229,8 +233,11 @@
                         */
                        need_resched = tsk->need_resched;
                        tsk->need_resched = 1;
-                       if ((best_cpu != this_cpu) && !need_resched)
+                       if ((best_cpu != this_cpu) && !need_resched) {
+                               p->has_cpu = 1;
+                               cpu_resched(best_cpu) = p;
                                smp_send_reschedule(best_cpu);
+                       }
                        return;
                }
        }

What could be done is the following. If we haven't
issued a need_resched event yet, then lock down <p>
to the best_cpu and if necessary issue an IPI.
This would ensure due to the fast check in schedule()
that we wouldn't run through the whole list in this
particular case either.

@@ -229,8 +233,11 @@
                         */
                        need_resched = tsk->need_resched;
                        tsk->need_resched = 1;
-                       if ((best_cpu != this_cpu) && !need_resched)
+                       if (!need_resched) {
+                               p->has_cpu = 1;
+                               cpu_resched(best_cpu) = p;
+                               if (best_cpu != this_cpu)
                                          smp_send_reschedule(best_cpu);
+                       }
                        return;
                }
        }

>
> >
> > In general is this something we should push for in the kernel scheduler
> > or should your quote be taken as the remedy for this problem ?
> > "If You want to have a better reschedule latency use the poll idle."
>
> The poll idle is power expensive but it has a better latency.
> Right now is boot command line settable.
>
> - Davide

I know, my question was, whether you think this whole discussion is useless
because somebody can always switch over to poll and should do so if he/she
has a problem with the current implementation ?

Also somebody contacted me with concerns to hot-plug cpus. Not a problem
One merely needs to do something  like this before yanking the cpu.

if (cpu_resched(yankcpu)) {
        cpu_resched(yankcpu)->has_cpu = 0;
        cpu_resched(yankcpu) = NULL;
}

and the temp-bound   task is free to be scheduable from anywhere else again.

-- Hubertus Franke
(frankeh@us.ibm.com)