From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.3 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,NICE_REPLY_A,SPF_HELO_NONE, SPF_PASS,URIBL_BLOCKED,USER_AGENT_SANE_1 autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8B680C0018C for ; Wed, 16 Dec 2020 08:46:43 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 4410F21D81 for ; Wed, 16 Dec 2020 08:46:43 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726143AbgLPIqh (ORCPT ); Wed, 16 Dec 2020 03:46:37 -0500 Received: from szxga01-in.huawei.com ([45.249.212.187]:2094 "EHLO szxga01-in.huawei.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1725972AbgLPIqh (ORCPT ); Wed, 16 Dec 2020 03:46:37 -0500 Received: from dggeme761-chm.china.huawei.com (unknown [172.30.72.57]) by szxga01-in.huawei.com (SkyGuard) with ESMTP id 4CwpbP6N4MzVfx0; Wed, 16 Dec 2020 16:44:49 +0800 (CST) Received: from [10.174.177.7] (10.174.177.7) by dggeme761-chm.china.huawei.com (10.3.19.107) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P256) id 15.1.1913.5; Wed, 16 Dec 2020 16:45:52 +0800 Subject: Re: [PATCH] use x86 cpu park to speedup smp_init in kexec situation To: Thomas Gleixner , Andy Lutomirski CC: LKML , Ingo Molnar , Borislav Petkov , X86 ML , "H. Peter Anvin" , , , , References: <87eejqu5q5.fsf@nanos.tec.linutronix.de> From: "shenkai (D)" Message-ID: Date: Wed, 16 Dec 2020 16:45:34 +0800 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:68.0) Gecko/20100101 Thunderbird/68.9.0 MIME-Version: 1.0 In-Reply-To: <87eejqu5q5.fsf@nanos.tec.linutronix.de> Content-Type: text/plain; charset="gbk"; format=flowed Content-Transfer-Encoding: 8bit X-Originating-IP: [10.174.177.7] X-ClientProxiedBy: dggeme707-chm.china.huawei.com (10.1.199.103) To dggeme761-chm.china.huawei.com (10.3.19.107) X-CFilter-Loop: Reflected Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org ÔÚ 2020/12/16 5:20, Thomas Gleixner дµÀ: > On Tue, Dec 15 2020 at 08:31, Andy Lutomirski wrote: >> On Tue, Dec 15, 2020 at 6:46 AM shenkai (D) wrote: >>> From: shenkai >>> Date: Tue, 15 Dec 2020 01:58:06 +0000 >>> Subject: [PATCH] use x86 cpu park to speedup smp_init in kexec situation >>> >>> In kexec reboot on x86 machine, APs will be halted and then waked up >>> by the apic INIT and SIPI interrupt. Here we can let APs spin instead >>> of being halted and boot APs by writing to specific address. In this way >>> we can accelerate smp_init procedure for we don't need to pull APs up >>> from a deep C-state. >>> >>> This is meaningful in many situations where users are sensitive to reboot >>> time cost. >> I like the concept. > No. This is the wrong thing to do. We are not optimizing for _one_ > special case. > > We can optimize it for all operations where all the non boot CPUs have > to brought up, be it cold boot, hibernation resume or kexec. > > Aside of that this is not a magic X86 special problem. Pretty much all > architectures have the same issue and it can be solved very simple, > which has been discussed before and I outlined the solution years ago, > but nobody sat down and actually made it work. > > Since the rewrite of the CPU hotplug infrastructure to a state machine > it's pretty obvious that the bringup of APs can changed from the fully > serialized: > > for_each_present_cpu(cpu) { > if (!cpu_online(cpu)) > cpu_up(cpu, CPUHP_ONLINE); > } > > to > > for_each_present_cpu(cpu) { > if (!cpu_online(cpu)) > cpu_up(cpu, CPUHP_KICK_CPU); > } > > for_each_present_cpu(cpu) { > if (!cpu_active(cpu)) > cpu_up(cpu, CPUHP_ONLINE); > } > > The CPUHP_KICK_CPU state does not exist today, but it's just the logical > consequence of the state machine. It's basically splitting __cpu_up() > into: > > __cpu_kick() > { > prepare(); > arch_kick_remote_cpu(); -> Send IPI/NMI, Firmware call ..... > } > > __cpu_wait_online() > { > wait_until_cpu_online(); > do_further_stuff(); > } > > There is some more to it than just blindly splitting it up at the > architecture level. > > All __cpu_up() implementations across arch/ have a lot of needlessly > duplicated and pointlessly differently implemented code which can move > completely into the core. > > So actually we want to split this further up: > > CPUHP_PREPARE_CPU_UP: Generic preparation step where all > the magic cruft which is duplicated > across architectures goes to > > CPUHP_KICK_CPU: Architecture specific prepare and kick > > CPUHP_WAIT_ONLINE: Generic wait function for CPU coming > online: wait_for_completion_timeout() > which releases the upcoming CPU and > invokes an optional arch_sync_cpu_up() > function which finalizes the bringup. > and on the AP side: > > CPU comes up, does all the low level setup, sets online, calls > complete() and the spinwaits for release. > > Once the control CPU comes out of the completion it releases the > spinwait. > > That works for all bringup situations and not only for kexec and the > simple trick is that by the time the last CPU has been kicked in the > first step, the first kicked CPU is already spinwaiting for release. > > By the time the first kicked CPU has completed the process, i.e. reached > the active state, then the next CPU is spinwaiting and so on. > > If you look at the provided time saving: > > Mainline: 210ms > Patched: 80ms > ----------------------------- > Delta 130ms > > i.e. it takes ~ 1.8ms to kick and wait for the AP to come up and ~ 1.1ms > per CPU for the whole bringup. It does not completly add up, but it has > a clear benefit for everything. > > Also the changelog says that the delay is related to CPUs in deep > C-states. If CPUs are brought down for kexec then it's trivial enough to > limit the C-states or just not use mwait() at all. > > It would be interesting to see the numbers just with play_dead() using > hlt() or mwait(eax=0, 0) for the kexec case and no other change at all. > > Thanks, > > tglx > Thanks for your and Andy's precious comments. I would like to take a try on reconstructing this patch to make it more decent and generic. Thanks again Kai