From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <linux-kernel-owner@vger.kernel.org>
Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand
        id S1757360AbdKOJye (ORCPT <rfc822;w@1wt.eu>);
        Wed, 15 Nov 2017 04:54:34 -0500
Received: from bombadil.infradead.org ([65.50.211.133]:38901 "EHLO
        bombadil.infradead.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org
        with ESMTP id S1752787AbdKOJya (ORCPT
        <rfc822;linux-kernel@vger.kernel.org>);
        Wed, 15 Nov 2017 04:54:30 -0500
Date: Wed, 15 Nov 2017 10:54:25 +0100
From: Peter Zijlstra <peterz@infradead.org>
To: Wanpeng Li <kernellwp@gmail.com>
Cc: "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        kvm <kvm@vger.kernel.org>, Radim Kr??m???? <rkrcmar@redhat.com>,
        Wanpeng Li <wanpeng.li@hotmail.com>,
        Paolo Bonzini <pbonzini@redhat.com>
Subject: Re: [PATCH v2 2/4] KVM: Add paravirt remote TLB flush
Message-ID: <20171115095425.2hsgpfomdmdru7ke@hirez.programming.kicks-ass.net>
References: <1510297497-10063-1-git-send-email-wanpeng.li@hotmail.com>
 <1510297497-10063-3-git-send-email-wanpeng.li@hotmail.com>
 <1a42f32f-3f2a-0c05-bd50-e6c3c710be85@redhat.com>
 <CANRm+CyoR9Nkjrz_HSkUbz0w2rzG_Rmufx48A0kw4OdxHYi3mQ@mail.gmail.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <CANRm+CyoR9Nkjrz_HSkUbz0w2rzG_Rmufx48A0kw4OdxHYi3mQ@mail.gmail.com>
User-Agent: NeoMutt/20170609 (1.8.3)
Sender: linux-kernel-owner@vger.kernel.org
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On Wed, Nov 15, 2017 at 04:43:32PM +0800, Wanpeng Li wrote:
> Hi Peterz,
> 
> I found big performance difference as I discuss with you several days ago.
> 
> ebizzy -M
>                         vanilla    static/local cpumask     per-cpu cpumask
>  8 vCPUs       10152            10083                          10117
> 16 vCPUs        1224              4866                          10008
> 24 vCPUs        1109              3871                            9928
> 32 vCPUs        1025              3375                            9811
> 
> In addition, I can observe ~50% perf top time is occupied by
> smp_call_function_many(), ~30% perf top time is occupied by
> call_function_interrupt() in the guest when running ebizzy for
> static/local cpumask variable. However, I almost can't observe these
> IPI stuffs after changing to per-cpu variable. Any opinions?

That doesn't really make sense.. :/

So a single static variable is broken (multiple CPUs can call
flush_tlb_others() concurrently and overwrite each others masks). But I
don't see why a per-cpu variable would be much slower than an on-stack
variable.