RE: [PATCH 0/6] x86/cpu hotplug: Wake up offline CPU via mwait or nmi

[Thomas Gleixner <tglx@linutronix.de>]

On Tue, 5 Jun 2012, Peter Zijlstra wrote:
> On Tue, 2012-06-05 at 21:43 +0200, Thomas Gleixner wrote:
> > Vs. the interrupt/timer/other crap madness:
> > 
> >  - We really don't want to have an interrupt balancer in the kernel
> >    again, but we need a mechanism to prevent the user space balancer
> >    trainwreck from ruining the power saving party.
> 
> What's wrong with having an interrupt balancer tied to the scheduler
> which optimistically tries to avoid interrupting nohz/isolated/idle
> cpus?

You want to run through a boatload of interrupts and change their
affinity from the load balancer or something related? Not really.

> >  - The timer issue is mostly solved by the existing nohz stuff
> >    (plus/minus the few bugs in there).
> 
> Its not.. if you create an isolated domain there's no way to expel
> existing timers from there.

Yep, that's one of the problems which need to be fixed independent of
the solution we come up with.

> >  - The other details (silly IPIs) and cross CPU timer arming) are way
> >    easier to solve by a proper prohibitive state than by chasing that
> >    nonsense all over the tree forever. 
> 
> But we need to solve all that without a prohibitibe state anyway for the
> isolation stuff to be useful.

And what is preventing us to use a prohibitive state for that purpose?
The isolation stuff Frederic is working on is nothing else than
dynamically switching in and out of a prohibitive state.

So do we really need to make the world and some more aware of those
states, instead of having a facility which lets us control what's
allowed/applicable in a given situation? Whether that's controlled by
the load-balancer or by user space or partially by both or something
else is a totally different issue.

I completely understand your reasoning, but I seriously doubt that we
can educate the whole crowd to understand the problems at hand. My
experience in the last 10+ years tells me that if you do not restrict
stuff you enter a never ending "chase the human stupidity^Wcreativity"
game. Even if you restrict it massively you end up observing a patch
which does:

+       d->core_internal_state__do_not_mess_with_it |= SOME_CONSTANT;

So do you really want to promote a solution which requires brain
sanity of all involved parties?

What's wrong with making a 'hotplug' model which provides the
following states:

  Fully functional

  Isolated functional

  Isolated idle

  <the physical hotplug mess>

where you have the ability to control the transitions of the upper 3
(or maybe more) states from the load balancer and/or user space or
whatever instance we come up with?

That puts the burden on the core facility design, but it removes the
maintainence burden to chase a gazillion of instances doing IPIs,
cross cpu function calls, add_timer_on, add_work_on and whatever
nonsense.

Note, that these upper states are not 'hotplug' by definition, but
they have to be traversed by hot(un)plug as well. So why not making
them explicit states which we can exploit for the other problems we
want to solve?

Your idea of tying everything to the scheduler and the load balancer
is just introducing the exacly same states again, just in a different
context.

Thanks,

	tglx