Re: [PATCH] sched,kvm: Fix KVM preempt_notifier usage

[Peter Zijlstra <peterz@infradead.org>]

On Fri, Jul 03, 2015 at 02:31:25PM +0200, Paolo Bonzini wrote:
> On 03/07/2015 14:19, Peter Zijlstra wrote:
> > On Fri, Jul 03, 2015 at 01:12:11PM +0200, Paolo Bonzini wrote:
> >> In fact you shouldn't have just tested the patch on a case _without_
> >> preemption notifiers, you should have also benchmarked the impact that
> >> static keys have _with_ preemption notifiers.  In a
> >> not-really-artificial case (one single-processor guest running on the
> >> host), the static key patch adds a static_key_slow_inc on a relatively
> >> hot path for KVM, which is not acceptable.
> > 
> > Spawning the first vcpu is a hot path?
> 
> This is not *spawning* the first VCPU.  Basically any critical section
> for vcpu->mutex includes a preempt_notifier_register/unregister pair:
> 
> /*
>  * Switches to specified vcpu, until a matching vcpu_put()
>  */
> int vcpu_load(struct kvm_vcpu *vcpu)
> {
>         int cpu;
> 
>         if (mutex_lock_killable(&vcpu->mutex))
>                 return -EINTR;
>         cpu = get_cpu();
>         preempt_notifier_register(&vcpu->preempt_notifier);
>         kvm_arch_vcpu_load(vcpu, cpu);
>         put_cpu();
>         return 0;
> }
> 
> void vcpu_put(struct kvm_vcpu *vcpu)
> {
>         preempt_disable();
>         kvm_arch_vcpu_put(vcpu);
>         preempt_notifier_unregister(&vcpu->preempt_notifier);
>         preempt_enable();
>         mutex_unlock(&vcpu->mutex);
> }
> 
> So basically you're adding at least one static_key_slow_inc/dec pair to
> every userspace exit.

Ugh, ok that is not what I was expecting to happen. I'll ask Ingo to
queue a revert until we can fix this better.

I thought these were vcpu create/destroy functions.

That said, the slow_inc/dec are really only slow on the 0<->!0
transitions.

But, could we rework the code so that you register the preempt notifier
when creating the vcpu thread and leave it installed forevermore?