softirq

[Julia Lawall <julia.lawall@inria.fr>]

Hello,

I'm not sure to understand how soft irqs work.  I see the code:

open_softirq(SCHED_SOFTIRQ, sched_balance_softirq);

Intuitively, I would expect that sched_balance_softirq would be run by
ksoftirqd.  That is, I would expect ksoftirqd to be scheduled
(sched_switch event), then the various actions of sched_balance_softirq to
be executed, and the ksoftirqd to be unscheduled (another ksoftirqd)
event.

But in practice, I see the code of sched_balance_softirq being executed
by the idle task, before the ksoftirqd is scheduled (see core 40):

          <idle>-0     [040]  3611.432554: softirq_entry:        vec=7 [action=SCHED]
          <idle>-0     [040]  3611.432554: bputs:                sched_balance_softirq: starting nohz
          <idle>-0     [040]  3611.432554: bputs:                sched_balance_softirq: starting _nohz_idle_balance
          bt.B.x-12022 [047]  3611.432554: softirq_entry:        vec=1 [action=TIMER]
          <idle>-0     [040]  3611.432554: bputs:                _nohz_idle_balance.isra.0: searching for a cpu
          bt.B.x-12033 [003]  3611.432554: softirq_entry:        vec=7 [action=SCHED]
          <idle>-0     [040]  3611.432554: bputs:                sched_balance_softirq: ending _nohz_idle_balance
          bt.B.x-12052 [011]  3611.432554: softirq_entry:        vec=7 [action=SCHED]
          <idle>-0     [040]  3611.432554: bputs:                sched_balance_softirq: nohz returns true ending soft irq
          <idle>-0     [040]  3611.432554: softirq_exit:         vec=7 [action=SCHED]

For example, idle seems to be running the code in _nohz_idle_balance.

I updated the code of _nohz_idle_balance as follows:

trace_printk("searching for a cpu\n");
        for_each_cpu_wrap(balance_cpu,  nohz.idle_cpus_mask, this_cpu+1) {
                if (!idle_cpu(balance_cpu))
                        continue;
trace_printk("found an idle cpu\n");

It prints searching for a cpu, but not found an idle cpu, because the
ksoftirqd on the core's runqueue makes the core not idle.  This makes the
whole softirq seem fairly useless when the only idle core is the one
raising the soft irq.

This is all for the same scenario that I have discussed previously, where
there are two sockets and an overload of on thread on one and an underload
of on thread on the other, and all the thread have been marked by numa
balancing as preferring to be where they are.  Now I am trying Prateek's
patch series.

thanks,
julia