Re: [RFC PATCH 0/4] Gang scheduling in CFS

[Avi Kivity <avi@redhat.com>]

On 01/04/2012 11:31 PM, Peter Zijlstra wrote:
> On Wed, 2012-01-04 at 19:16 +0200, Avi Kivity wrote:
> > 
> > 
> > I think we can solve it at the guest level.  The paravirt ticketlock
> > stuff introduces wait/wake calls (actually wait is just a HLT
> > instruction); we could spin for a while, then HLT until the other side
> > wakes us.  We should do this for all sites that busy wait.
> > 
> This is all TLB invalidates, right?
>
> So why wait for non-running vcpus at all? That is, why not paravirt the
> TLB flush such that the invalidate marks the non-running VCPU's state so
> that on resume it will first flush its TLBs. That way you don't have to
> wake it up and wait for it to invalidate its TLBs.

That's what Xen does, but it's tricky.  For example
get_user_pages_fast() depends on the IPI to hold off page freeing, if we
paravirt it we have to take that into consideration.

> Or am I like totally missing the point (I am after all reading the
> thread backwards and I haven't yet fully paged the kernel stuff back
> into my brain).

You aren't, and I bet those kernel pages are unswappable anyway.

> I guess tagging remote VCPU state like that might be somewhat tricky..
> but it seems worth considering, the whole wake and wait for flush thing
> seems daft.

It's nasty, but then so is paravirt.  It's hard to get right, and it has
a tendency to cause performance regressions as hardware improves.

-- 
I have a truly marvellous patch that fixes the bug which this
signature is too narrow to contain.