Re: [ANNOUNCE] 3.0.1-rt11

[Frank Rowand <frank.rowand@am.sony.com>]

On 09/07/11 07:01, Russell King - ARM Linux wrote:
> On Wed, Sep 07, 2011 at 12:57:44PM +0200, Thomas Gleixner wrote:
>> The problem is that if you enable interrupts on the CPU _BEFORE_ it is
>> set online AND active, then you can end up waking up kernel threads
>> which are bound to that CPU and the scheduler will happily schedule
>> them on an online CPU. That makes them lose the cpu affinity to the
>> CPU as well and hell breaks lose.
> 
> How can that happen?
> 
> 1. The only interrupts we're likely to receive are the local timer
>    interrupts - we have not routed any other interrupts to this CPU.

Yes, it is the local timer interrupt.

> 
> 2. We will not schedule on this CPU except at explicit scheduling
>    points (such as contended mutexes or explicit calls to schedule)
>    as we have a call to preempt_disable().

It is not a schedule.  It is wake_up_process():

   wake_up_process()
      try_to_wake_up()
         select_task_rq()
            if (... || !cpu_online(cpu))
               select_fallback_rq(task_cpu(p), p)
                  ...
                  /* No more Mr. Nice Guy. */
                  dest_cpu = cpuset_cpus_allowed_fallback(p)
                     do_set_cpus_allowed(p, cpu_possible_mask)
                        #  Thus ksoftirqd can now run on any cpu...


> 
>> Frank has observed this with softirq threads, but the same thing is
>> true for any other CPU bound thread like the worker stuff.
> 
> So who is scheduling a workqueue from the local timer?

   do_local_timer()
      ipi_timer()
         irq_exit()
            invoke_softirq()
               wakeup_softirqd()
                  wake_up_process()

> 
>> So moving the online, active thing BEFORE enabling interrupt is the
>> only sensible solution.
> 
> Yes, that'll be why even x86 enables interrupts before setting the CPU
> online for the delay calibration.

-Frank