From: Usama Arif <usama.arif@bytedance.com>
To: David Woodhouse <dwmw2@infradead.org>,
Oleksandr Natalenko <oleksandr@natalenko.name>,
tglx@linutronix.de, kim.phillips@amd.com, brgerst@gmail.com
Cc: piotrgorski@cachyos.org, arjan@linux.intel.com, mingo@redhat.com,
bp@alien8.de, dave.hansen@linux.intel.com, hpa@zytor.com,
x86@kernel.org, pbonzini@redhat.com, paulmck@kernel.org,
linux-kernel@vger.kernel.org, kvm@vger.kernel.org,
rcu@vger.kernel.org, mimoja@mimoja.de, hewenliang4@huawei.com,
thomas.lendacky@amd.com, seanjc@google.com,
pmenzel@molgen.mpg.de, fam.zheng@bytedance.com,
punit.agrawal@bytedance.com, simon.evans@bytedance.com,
liangma@liangbit.com
Subject: Re: [External] Re: [PATCH v12 00/11] Parallel CPU bringup for x86_64
Date: Mon, 27 Feb 2023 06:25:17 +0000 [thread overview]
Message-ID: <c795e967-b69d-33fd-b333-b0f8e0636b5b@bytedance.com> (raw)
In-Reply-To: <EB27C1C4-7246-4557-A6B3-EBBC9E8FC69F@infradead.org>
On 27/02/2023 06:13, David Woodhouse wrote:
>
>
> On 26 February 2023 20:59:17 GMT, Usama Arif <usama.arif@bytedance.com> wrote:
>>
>>
>> On 26/02/2023 18:31, Oleksandr Natalenko wrote:
>>> Hello.
>>>
>>> On neděle 26. února 2023 12:07:51 CET Usama Arif wrote:
>>>> The main code change over v11 is the build error fix by Brian Gerst and
>>>> acquiring tr_lock in trampoline_64.S whenever the stack is setup.
>>>>
>>>> The git history is also rewritten to move the commits that removed
>>>> initial_stack, early_gdt_descr and initial_gs earlier in the patchset.
>>>>
>>>> Thanks,
>>>> Usama
>>>>
>>>> Changes across versions:
>>>> v2: Cut it back to just INIT/SIPI/SIPI in parallel for now, nothing more
>>>> v3: Clean up x2apic patch, add MTRR optimisation, lock topology update
>>>> in preparation for more parallelisation.
>>>> v4: Fixes to the real mode parallelisation patch spotted by SeanC, to
>>>> avoid scribbling on initial_gs in common_cpu_up(), and to allow all
>>>> 24 bits of the physical X2APIC ID to be used. That patch still needs
>>>> a Signed-off-by from its original author, who once claimed not to
>>>> remember writing it at all. But now we've fixed it, hopefully he'll
>>>> admit it now :)
>>>> v5: rebase to v6.1 and remeasure performance, disable parallel bringup
>>>> for AMD CPUs.
>>>> v6: rebase to v6.2-rc6, disabled parallel boot on amd as a cpu bug and
>>>> reused timer calibration for secondary CPUs.
>>>> v7: [David Woodhouse] iterate over all possible CPUs to find any existing
>>>> cluster mask in alloc_clustermask. (patch 1/9)
>>>> Keep parallel AMD support enabled in AMD, using APIC ID in CPUID leaf
>>>> 0x0B (for x2APIC mode) or CPUID leaf 0x01 where 8 bits are sufficient.
>>>> Included sanity checks for APIC id from 0x0B. (patch 6/9)
>>>> Removed patch for reusing timer calibration for secondary CPUs.
>>>> commit message and code improvements.
>>>> v8: Fix CPU0 hotplug by setting up the initial_gs, initial_stack and
>>>> early_gdt_descr.
>>>> Drop trampoline lock and bail if APIC ID not found in find_cpunr.
>>>> Code comments improved and debug prints added.
>>>> v9: Drop patch to avoid repeated saves of MTRR at boot time.
>>>> rebased and retested at v6.2-rc8.
>>>> added kernel doc for no_parallel_bringup and made do_parallel_bringup
>>>> __ro_after_init.
>>>> v10: Fixed suspend/resume not working with parallel smpboot.
>>>> rebased and retested to 6.2.
>>>> fixed checkpatch errors.
>>>> v11: Added patches from Brian Gerst to remove the global variables initial_gs,
>>>> initial_stack, and early_gdt_descr from the 64-bit boot code
>>>> (https://lore.kernel.org/all/20230222221301.245890-1-brgerst@gmail.com/).
>>>> v12: Fixed compilation errors, acquire tr_lock for every stack setup in
>>>> trampoline_64.S.
>>>> Rearranged commits for a cleaner git history.
>>>>
>>>> Brian Gerst (3):
>>>> x86/smpboot: Remove initial_stack on 64-bit
>>>> x86/smpboot: Remove early_gdt_descr on 64-bit
>>>> x86/smpboot: Remove initial_gs
>>>>
>>>> David Woodhouse (8):
>>>> x86/apic/x2apic: Allow CPU cluster_mask to be populated in parallel
>>>> cpu/hotplug: Move idle_thread_get() to <linux/smpboot.h>
>>>> cpu/hotplug: Add dynamic parallel bringup states before
>>>> CPUHP_BRINGUP_CPU
>>>> x86/smpboot: Reference count on smpboot_setup_warm_reset_vector()
>>>> x86/smpboot: Split up native_cpu_up into separate phases and document
>>>> them
>>>> x86/smpboot: Support parallel startup of secondary CPUs
>>>> x86/smpboot: Send INIT/SIPI/SIPI to secondary CPUs in parallel
>>>> x86/smpboot: Serialize topology updates for secondary bringup
>>>>
>>>> .../admin-guide/kernel-parameters.txt | 3 +
>>>> arch/x86/include/asm/processor.h | 6 +-
>>>> arch/x86/include/asm/realmode.h | 4 +-
>>>> arch/x86/include/asm/smp.h | 15 +-
>>>> arch/x86/include/asm/topology.h | 2 -
>>>> arch/x86/kernel/acpi/sleep.c | 15 +-
>>>> arch/x86/kernel/apic/apic.c | 2 +-
>>>> arch/x86/kernel/apic/x2apic_cluster.c | 126 ++++---
>>>> arch/x86/kernel/asm-offsets.c | 1 +
>>>> arch/x86/kernel/cpu/common.c | 6 +-
>>>> arch/x86/kernel/head_64.S | 129 +++++--
>>>> arch/x86/kernel/smpboot.c | 350 +++++++++++++-----
>>>> arch/x86/realmode/init.c | 3 +
>>>> arch/x86/realmode/rm/trampoline_64.S | 27 +-
>>>> arch/x86/xen/smp_pv.c | 4 +-
>>>> arch/x86/xen/xen-head.S | 2 +-
>>>> include/linux/cpuhotplug.h | 2 +
>>>> include/linux/smpboot.h | 7 +
>>>> kernel/cpu.c | 31 +-
>>>> kernel/smpboot.h | 2 -
>>>> 20 files changed, 537 insertions(+), 200 deletions(-)
>>>
>>> With `CONFIG_FORCE_NR_CPUS=y` this results in:
>>>
>>> ```
>>> ld: vmlinux.o: in function `secondary_startup_64_no_verify':
>>> (.head.text+0x10c): undefined reference to `nr_cpu_ids'
>>> ```
>>>
>>> That's because in `arch/x86/kernel/head_64.S` `secondary_startup_64_no_verify()` refers to `nr_cpu_ids` under `#ifdef CONFIG_SMP`, but this symbol is available under the following conditions:
>>>
>>> ```
>>> 38 #if (NR_CPUS == 1) || defined(CONFIG_FORCE_NR_CPUS)
>>> 39 #define nr_cpu_ids ((unsigned int)NR_CPUS)
>>> 40 #else
>>> 41 extern unsigned int nr_cpu_ids;
>>> 42 #endif
>>>
>>> 1090 #if (NR_CPUS > 1) && !defined(CONFIG_FORCE_NR_CPUS)
>>> 1091 /* Setup number of possible processor ids */
>>> 1092 unsigned int nr_cpu_ids __read_mostly = NR_CPUS;
>>> 1093 EXPORT_SYMBOL(nr_cpu_ids);
>>> 1094 #endif
>>> ```
>>>
>>> So having `CONFIG_SMP=y` and, for instance, `CONFIG_NR_CPUS=320`, it is possible to compile out `EXPORT_SYMBOL(nr_cpu_ids);` if `CONFIG_FORCE_NR_CPUS=y` is set.
>>>
>>
>> I think something like below diff should work in all scenarios?
>
> I'd've changed the asm side to use the constant limit.
Yup, just needed the morning coffee :) Had sent the proper fix in
https://lore.kernel.org/all/5e8ad90a-1dc6-95c2-e020-5e95da6f9eda@bytedance.com/#t
I guess the diff is still small over v12 (including the cosmetic
changes) to send out a new version so soon, probably better to wait a
couple of days incase something else comes up as well?
Thanks,
Usama
next prev parent reply other threads:[~2023-02-27 6:25 UTC|newest]
Thread overview: 49+ messages / expand[flat|nested] mbox.gz Atom feed top
2023-02-26 11:07 Usama Arif
2023-02-26 11:07 ` [PATCH v12 01/11] x86/apic/x2apic: Allow CPU cluster_mask to be populated in parallel Usama Arif
2023-02-26 11:07 ` [PATCH v12 02/11] cpu/hotplug: Move idle_thread_get() to <linux/smpboot.h> Usama Arif
2023-02-26 11:07 ` [PATCH v12 03/11] cpu/hotplug: Add dynamic parallel bringup states before CPUHP_BRINGUP_CPU Usama Arif
2023-02-26 11:07 ` [PATCH v12 04/11] x86/smpboot: Reference count on smpboot_setup_warm_reset_vector() Usama Arif
2023-02-26 11:07 ` [PATCH v12 05/11] x86/smpboot: Split up native_cpu_up into separate phases and document them Usama Arif
2023-02-26 11:07 ` [PATCH v12 06/11] x86/smpboot: Remove initial_stack on 64-bit Usama Arif
2023-02-28 16:13 ` Thomas Gleixner
2023-02-28 16:25 ` David Woodhouse
2023-02-28 20:32 ` Thomas Gleixner
2023-02-28 17:09 ` David Woodhouse
2023-02-28 20:17 ` Thomas Gleixner
2023-02-28 20:43 ` David Woodhouse
2023-03-01 10:18 ` Thomas Gleixner
2023-03-01 10:42 ` David Woodhouse
2023-02-28 22:22 ` Brian Gerst
2023-02-26 11:07 ` [PATCH v12 07/11] x86/smpboot: Remove early_gdt_descr " Usama Arif
2023-02-28 21:01 ` Thomas Gleixner
2023-02-28 21:02 ` Arjan van de Ven
2023-03-01 20:32 ` Josh Triplett
2023-03-01 22:16 ` Paul E. McKenney
2023-03-01 22:25 ` Josh Triplett
2023-03-02 0:28 ` Paul E. McKenney
2023-03-02 1:05 ` Randy Dunlap
2023-03-02 1:37 ` Paul E. McKenney
2023-02-28 21:57 ` David Woodhouse
2023-02-28 22:41 ` David Woodhouse
2023-02-28 22:48 ` Brian Gerst
2023-02-28 23:42 ` David Woodhouse
2023-03-01 0:02 ` H. Peter Anvin
2023-03-01 7:39 ` David Woodhouse
2023-03-01 0:03 ` Brian Gerst
2023-03-01 10:27 ` Thomas Gleixner
2023-02-28 22:09 ` Brian Gerst
2023-02-26 11:07 ` [PATCH v12 08/11] x86/smpboot: Remove initial_gs Usama Arif
2023-02-26 11:08 ` [PATCH v12 09/11] x86/smpboot: Support parallel startup of secondary CPUs Usama Arif
2023-02-26 11:08 ` [PATCH v12 10/11] x86/smpboot: Send INIT/SIPI/SIPI to secondary CPUs in parallel Usama Arif
2023-02-26 11:08 ` [PATCH v12 11/11] x86/smpboot: Serialize topology updates for secondary bringup Usama Arif
2023-02-26 18:31 ` [PATCH v12 00/11] Parallel CPU bringup for x86_64 Oleksandr Natalenko
2023-02-26 20:59 ` [External] " Usama Arif
2023-02-27 6:13 ` David Woodhouse
2023-02-27 6:25 ` Usama Arif [this message]
2023-02-27 6:14 ` Usama Arif
2023-02-27 15:29 ` David Woodhouse
2023-02-27 16:32 ` Usama Arif
2023-02-27 21:39 ` Guilherme G. Piccoli
2023-02-28 9:07 ` David Woodhouse
2023-02-28 10:13 ` Paul Menzel
2023-02-28 12:04 ` Guilherme G. Piccoli
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=c795e967-b69d-33fd-b333-b0f8e0636b5b@bytedance.com \
--to=usama.arif@bytedance.com \
--cc=arjan@linux.intel.com \
--cc=bp@alien8.de \
--cc=brgerst@gmail.com \
--cc=dave.hansen@linux.intel.com \
--cc=dwmw2@infradead.org \
--cc=fam.zheng@bytedance.com \
--cc=hewenliang4@huawei.com \
--cc=hpa@zytor.com \
--cc=kim.phillips@amd.com \
--cc=kvm@vger.kernel.org \
--cc=liangma@liangbit.com \
--cc=linux-kernel@vger.kernel.org \
--cc=mimoja@mimoja.de \
--cc=mingo@redhat.com \
--cc=oleksandr@natalenko.name \
--cc=paulmck@kernel.org \
--cc=pbonzini@redhat.com \
--cc=piotrgorski@cachyos.org \
--cc=pmenzel@molgen.mpg.de \
--cc=punit.agrawal@bytedance.com \
--cc=rcu@vger.kernel.org \
--cc=seanjc@google.com \
--cc=simon.evans@bytedance.com \
--cc=tglx@linutronix.de \
--cc=thomas.lendacky@amd.com \
--cc=x86@kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®