mirror of https://lore.kernel.org/lkml/
 help / color / mirror / Atom feed
* [PATCH v8 0/12] x86: Arbitrary CPU hot(un)plug support
@ 2012-08-25 17:41 Fenghua Yu
  2012-08-25 17:41 ` [PATCH v8 01/12] doc: Add x86 CPU0 online/offline feature Fenghua Yu
                   ` (12 more replies)
  0 siblings, 13 replies; 28+ messages in thread
From: Fenghua Yu @ 2012-08-25 17:41 UTC (permalink / raw)
  To: Ingo Molnar, Thomas Gleixner, H Peter Anvin, Linus Torvalds,
	Andrew Morton, Asit K Mallick, Tony Luck, Arjan Dan De Ven,
	Suresh B Siddha, Len Brown, Srivatssa S. Bhat, Randy Dunlap,
	Chen Gong, linux-kernel, linux-pm, x86
  Cc: Fenghua Yu

From: Fenghua Yu <fenghua.yu@intel.com>

CPU0 or BSP (Bootstrap Processor) has been the last processor that can not be
hot removed on x86. This patchset implements CPU0 or BSP online and offline
and removes this obstacle to CPU hotplug.

RAS needs the feature. If socket0 needs to be hotplugged for any reason (any
thread on socket0 is bad, shared cache issue, uncore issue, etc), CPU0 is
required to be offline or hot replaced to keep the system run. For example,
starting with Core Duo, if you have a system that is reporting cache problems
via the "yellow" status in the MCi_STATUS msr, then there is benefit in simply
soft off-lining the cores that share that cache - assuming that leaves you at
least one online core. A single socket system with L3 cache troubles is not
helped - but problems in L1/L2 cache, or on multi-socket systems can be avoided.
They are already being avoided for the cases where CPU0 is not involved.
This patchset can help L1/L2 cache problem in CPU0 or L3 cache problem on the
socket with CPU0 in a multi-socket system.

v8: Add smp_store_boot_cpu_info() and change smp_store_cpu_info() back to avoid
a compilation error. Fix a few messy subject issues.

v7: Change smp_store_cpu_info() to allow store cpuinfo for CPU0 when online.
Change PIC irq detection in native_cpu_disable().

v6: If CPU0 is offlined during boot time in CPU0 hotplug debug mode, put CPU0
online again before resuming from hibernation and disable x2apic and xapic.Don't
set __CPUINIT for start_cpu0() in head_32.S. Clean up CPU0 wake up nmi handler
after callin and callout sync. In a period (3 seconds), check if CPU0 wake up
NMI is handled after offlined CPU0 exits from mwait.

v5: Add CONFIG_BOOTPARAM_HOTPLUG_CPU0 and CONFIG_DEBUG_HOTPLUG_CPU0. Simplify
duplicate xstate_size init check. Wakeup CPU0 via nmi instead INITs. Add
mcheck_cpu_init when CPU0 online. Change variable bsp_hotpluggable to
cpu0_hotpluggable with __initdata qualifier.

v4: Add __read_mostly for internal bsp_hotpluggable variable. Add my email
address in cpu-hotplug.txt document. A wording change in comment.

v3: Register a pm notifier to check if CPU0 is online before hibernate/suspend.
Small wording changes in document and print info.

v2: Add locking changes between cpu hotplug and hibernate/suspend. Change PIC
irq bound to CPU0 detection.

Fenghua Yu (12):
  doc: Add x86 CPU0 online/offline feature
  x86, Kconfig: Add config switch for CPU0 hotplug
  x86, topology: Support functions for CPU0 online/offline
  x86, hotplug: Don't offline CPU0 if any PIC irq can not be migrated
    out of it
  x86, hotplug, suspend: Online CPU0 for suspend or hibernate
  x86-64, hotplug: Add start_cpu0() entry point to head_64.S
  x86-32, hotplug: Add start_cpu0() entry point to head_32.S
  x86, hotplug: Wake up CPU0 via NMI instead of INIT, SIPI, SIPI
  x86, hotplug: During CPU0 online, enable x2apic, set_numa_node.
  x86, hotplug: The first online processor saves the MTRR state
  x86/i387.c: Initialize thread xstate only on CPU0 only once
  x86, topology: Debug CPU00 hotplug

 Documentation/cpu-hotplug.txt       |   24 ++++++
 Documentation/kernel-parameters.txt |   14 +++
 arch/x86/Kconfig                    |   44 ++++++++++
 arch/x86/include/asm/cpu.h          |    4 +
 arch/x86/include/asm/smp.h          |    1 +
 arch/x86/kernel/cpu/common.c        |    5 +-
 arch/x86/kernel/cpu/mtrr/main.c     |    9 ++-
 arch/x86/kernel/head_32.S           |   12 +++
 arch/x86/kernel/head_64.S           |   15 ++++
 arch/x86/kernel/i387.c              |    6 +-
 arch/x86/kernel/smpboot.c           |  150 +++++++++++++++++++++++++++++------
 arch/x86/kernel/topology.c          |   92 ++++++++++++++++++++--
 arch/x86/power/cpu.c                |   82 +++++++++++++++++++
 13 files changed, 419 insertions(+), 39 deletions(-)

-- 
1.7.2


^ permalink raw reply	[flat|nested] 28+ messages in thread
* [PATCH v10 09/14] x86, hotplug: Wake up CPU0 via NMI instead of INIT, SIPI, SIPI
@ 2012-11-13 19:32 Fenghua Yu
  2012-11-14 21:29 ` [tip:x86/bsp-hotplug] " tip-bot for Fenghua Yu
  0 siblings, 1 reply; 28+ messages in thread
From: Fenghua Yu @ 2012-11-13 19:32 UTC (permalink / raw)
  To: Ingo Molnar, Thomas Gleixner, H Peter Anvin, Linus Torvalds,
	Andrew Morton, Asit K Mallick, Tony Luck, Arjan Dan De Ven,
	Suresh B Siddha, Len Brown, Srivatssa S. Bhat, Randy Dunlap,
	Rafael J. Wysocki, Chen Gong, linux-kernel, linux-pm, x86
  Cc: Fenghua Yu

From: Fenghua Yu <fenghua.yu@intel.com>

Instead of waiting for STARTUP after INITs, BSP will execute the BIOS boot-strap
code which is not a desired behavior for waking up BSP. To avoid the boot-strap
code, wake up CPU0 by NMI instead.

This works to wake up soft offlined CPU0 only. If CPU0 is hard offlined (i.e.
physically hot removed and then hot added), NMI won't wake it up. We'll change
this code in the future to wake up hard offlined CPU0 if real platform and
request are available.

AP is still waken up as before by INIT, SIPI, SIPI sequence.

Signed-off-by: Fenghua Yu <fenghua.yu@intel.com>
---
 arch/x86/include/asm/cpu.h |    1 +
 arch/x86/kernel/smpboot.c  |  113 +++++++++++++++++++++++++++++++++++++++++---
 2 files changed, 107 insertions(+), 7 deletions(-)

diff --git a/arch/x86/include/asm/cpu.h b/arch/x86/include/asm/cpu.h
index 4564c8e..a119572 100644
--- a/arch/x86/include/asm/cpu.h
+++ b/arch/x86/include/asm/cpu.h
@@ -28,6 +28,7 @@ struct x86_cpu {
 #ifdef CONFIG_HOTPLUG_CPU
 extern int arch_register_cpu(int num);
 extern void arch_unregister_cpu(int);
+extern void __cpuinit start_cpu0(void);
 #endif
 
 DECLARE_PER_CPU(int, cpu_state);
diff --git a/arch/x86/kernel/smpboot.c b/arch/x86/kernel/smpboot.c
index c297907..31854bc 100644
--- a/arch/x86/kernel/smpboot.c
+++ b/arch/x86/kernel/smpboot.c
@@ -138,15 +138,17 @@ static void __cpuinit smp_callin(void)
 	 * we may get here before an INIT-deassert IPI reaches
 	 * our local APIC.  We have to wait for the IPI or we'll
 	 * lock up on an APIC access.
+	 *
+	 * Since CPU0 is not wakened up by INIT, it doesn't wait for the IPI.
 	 */
-	if (apic->wait_for_init_deassert)
+	cpuid = smp_processor_id();
+	if (apic->wait_for_init_deassert && cpuid != 0)
 		apic->wait_for_init_deassert(&init_deasserted);
 
 	/*
 	 * (This works even if the APIC is not enabled.)
 	 */
 	phys_id = read_apic_id();
-	cpuid = smp_processor_id();
 	if (cpumask_test_cpu(cpuid, cpu_callin_mask)) {
 		panic("%s: phys CPU#%d, CPU#%d already present??\n", __func__,
 					phys_id, cpuid);
@@ -228,6 +230,8 @@ static void __cpuinit smp_callin(void)
 	cpumask_set_cpu(cpuid, cpu_callin_mask);
 }
 
+static int cpu0_logical_apicid;
+static int enable_start_cpu0;
 /*
  * Activate a secondary processor.
  */
@@ -243,6 +247,8 @@ notrace static void __cpuinit start_secondary(void *unused)
 	preempt_disable();
 	smp_callin();
 
+	enable_start_cpu0 = 0;
+
 #ifdef CONFIG_X86_32
 	/* switch away from the initial page table */
 	load_cr3(swapper_pg_dir);
@@ -492,7 +498,7 @@ void __inquire_remote_apic(int apicid)
  * won't ... remember to clear down the APIC, etc later.
  */
 int __cpuinit
-wakeup_secondary_cpu_via_nmi(int logical_apicid, unsigned long start_eip)
+wakeup_secondary_cpu_via_nmi(int apicid, unsigned long start_eip)
 {
 	unsigned long send_status, accept_status = 0;
 	int maxlvt;
@@ -500,7 +506,7 @@ wakeup_secondary_cpu_via_nmi(int logical_apicid, unsigned long start_eip)
 	/* Target chip */
 	/* Boot on the stack */
 	/* Kick the second */
-	apic_icr_write(APIC_DM_NMI | apic->dest_logical, logical_apicid);
+	apic_icr_write(APIC_DM_NMI | apic->dest_logical, apicid);
 
 	pr_debug("Waiting for send to finish...\n");
 	send_status = safe_apic_wait_icr_idle();
@@ -660,6 +666,63 @@ static void __cpuinit announce_cpu(int cpu, int apicid)
 			node, cpu, apicid);
 }
 
+static int wakeup_cpu0_nmi(unsigned int cmd, struct pt_regs *regs)
+{
+	int cpu;
+
+	cpu = smp_processor_id();
+	if (cpu == 0 && !cpu_online(cpu) && enable_start_cpu0)
+		return NMI_HANDLED;
+
+	return NMI_DONE;
+}
+
+/*
+ * Wake up AP by INIT, INIT, STARTUP sequence.
+ *
+ * Instead of waiting for STARTUP after INITs, BSP will execute the BIOS
+ * boot-strap code which is not a desired behavior for waking up BSP. To
+ * void the boot-strap code, wake up CPU0 by NMI instead.
+ *
+ * This works to wake up soft offlined CPU0 only. If CPU0 is hard offlined
+ * (i.e. physically hot removed and then hot added), NMI won't wake it up.
+ * We'll change this code in the future to wake up hard offlined CPU0 if
+ * real platform and request are available.
+ */
+static int __cpuinit
+wakeup_cpu_via_init_nmi(int cpu, unsigned long start_ip, int apicid,
+	       int *cpu0_nmi_registered)
+{
+	int id;
+	int boot_error;
+
+	/*
+	 * Wake up AP by INIT, INIT, STARTUP sequence.
+	 */
+	if (cpu)
+		return wakeup_secondary_cpu_via_init(apicid, start_ip);
+
+	/*
+	 * Wake up BSP by nmi.
+	 *
+	 * Register a NMI handler to help wake up CPU0.
+	 */
+	boot_error = register_nmi_handler(NMI_LOCAL,
+					  wakeup_cpu0_nmi, 0, "wake_cpu0");
+
+	if (!boot_error) {
+		enable_start_cpu0 = 1;
+		*cpu0_nmi_registered = 1;
+		if (apic->dest_logical == APIC_DEST_LOGICAL)
+			id = cpu0_logical_apicid;
+		else
+			id = apicid;
+		boot_error = wakeup_secondary_cpu_via_nmi(id, start_ip);
+	}
+
+	return boot_error;
+}
+
 /*
  * NOTE - on most systems this is a PHYSICAL apic ID, but on multiquad
  * (ie clustered apic addressing mode), this is a LOGICAL apic ID.
@@ -675,6 +738,7 @@ static int __cpuinit do_boot_cpu(int apicid, int cpu, struct task_struct *idle)
 
 	unsigned long boot_error = 0;
 	int timeout;
+	int cpu0_nmi_registered = 0;
 
 	/* Just in case we booted with a single CPU. */
 	alternatives_enable_smp();
@@ -722,13 +786,16 @@ static int __cpuinit do_boot_cpu(int apicid, int cpu, struct task_struct *idle)
 	}
 
 	/*
-	 * Kick the secondary CPU. Use the method in the APIC driver
-	 * if it's defined - or use an INIT boot APIC message otherwise:
+	 * Wake up a CPU in difference cases:
+	 * - Use the method in the APIC driver if it's defined
+	 * Otherwise,
+	 * - Use an INIT boot APIC message for APs or NMI for BSP.
 	 */
 	if (apic->wakeup_secondary_cpu)
 		boot_error = apic->wakeup_secondary_cpu(apicid, start_ip);
 	else
-		boot_error = wakeup_secondary_cpu_via_init(apicid, start_ip);
+		boot_error = wakeup_cpu_via_init_nmi(cpu, start_ip, apicid,
+						     &cpu0_nmi_registered);
 
 	if (!boot_error) {
 		/*
@@ -793,6 +860,13 @@ static int __cpuinit do_boot_cpu(int apicid, int cpu, struct task_struct *idle)
 		 */
 		smpboot_restore_warm_reset_vector();
 	}
+	/*
+	 * Clean up the nmi handler. Do this after the callin and callout sync
+	 * to avoid impact of possible long unregister time.
+	 */
+	if (cpu0_nmi_registered)
+		unregister_nmi_handler(NMI_LOCAL, "wake_cpu0");
+
 	return boot_error;
 }
 
@@ -1037,6 +1111,11 @@ void __init native_smp_prepare_cpus(unsigned int max_cpus)
 	 */
 	setup_local_APIC();
 
+	if (x2apic_mode)
+		cpu0_logical_apicid = apic_read(APIC_LDR);
+	else
+		cpu0_logical_apicid = GET_APIC_LOGICAL_ID(apic_read(APIC_LDR));
+
 	/*
 	 * Enable IO APIC before setting up error vector
 	 */
@@ -1264,6 +1343,16 @@ void play_dead_common(void)
 	local_irq_disable();
 }
 
+static bool wakeup_cpu0(void)
+{
+	unsigned int timeout;
+
+	if (smp_processor_id() == 0 && enable_start_cpu0)
+		return true;
+
+	return false;
+}
+
 /*
  * We need to flush the caches before going to sleep, lest we have
  * dirty data in our caches when we come back up.
@@ -1327,6 +1416,11 @@ static inline void mwait_play_dead(void)
 		__monitor(mwait_ptr, 0, 0);
 		mb();
 		__mwait(eax, 0);
+		/*
+		 * If NMI wants to wake up CPU0, start CPU0.
+		 */
+		if (wakeup_cpu0())
+			start_cpu0();
 	}
 }
 
@@ -1337,6 +1431,11 @@ static inline void hlt_play_dead(void)
 
 	while (1) {
 		native_halt();
+		/*
+		 * If NMI wants to wake up CPU0, start CPU0.
+		 */
+		if (wakeup_cpu0())
+			start_cpu0();
 	}
 }
 
-- 
1.7.2


^ permalink raw reply	[flat|nested] 28+ messages in thread
[parent not found: <1352896613-25957-1-git-send-email-fenghua.yu@intel.com>]

end of thread, other threads:[~2012-11-15  0:24 UTC | newest]

Thread overview: 28+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2012-08-25 17:41 [PATCH v8 0/12] x86: Arbitrary CPU hot(un)plug support Fenghua Yu
2012-08-25 17:41 ` [PATCH v8 01/12] doc: Add x86 CPU0 online/offline feature Fenghua Yu
2012-08-27 17:07   ` [tip:x86/bsp-hotplug] " tip-bot for Fenghua Yu
2012-08-25 17:41 ` [PATCH v8 02/12] x86, Kconfig: Add config switch for CPU0 hotplug Fenghua Yu
2012-08-27 17:08   ` [tip:x86/bsp-hotplug] " tip-bot for Fenghua Yu
2012-08-25 17:41 ` [PATCH v8 03/12] x86, topology: Don't offline CPU0 if any PIC irq can not be migrated out of it Fenghua Yu
2012-08-27 17:09   ` [tip:x86/bsp-hotplug] x86, topology: Don' t " tip-bot for Fenghua Yu
2012-08-25 17:41 ` [PATCH v8 04/12] x86, hotplug: Support functions for CPU0 online/offline Fenghua Yu
2012-08-27 17:10   ` [tip:x86/bsp-hotplug] " tip-bot for Fenghua Yu
2012-08-25 17:41 ` [PATCH v8 05/12] x86, hotplug, suspend: Online CPU0 for suspend or hibernate Fenghua Yu
2012-08-27 17:10   ` [tip:x86/bsp-hotplug] " tip-bot for Fenghua Yu
2012-08-25 17:41 ` [PATCH v8 06/12] x86-64, hotplug: Add start_cpu0() entry point to head_64.S Fenghua Yu
2012-08-27 17:11   ` [tip:x86/bsp-hotplug] " tip-bot for Fenghua Yu
2012-08-25 17:41 ` [PATCH v8 07/12] x86-32, hotplug: Add start_cpu0() entry point to head_32.S Fenghua Yu
2012-08-27 17:12   ` [tip:x86/bsp-hotplug] " tip-bot for Fenghua Yu
2012-08-25 17:41 ` [PATCH v8 08/12] x86, hotplug: Wake up CPU0 via NMI instead of INIT, SIPI, SIPI Fenghua Yu
2012-08-27 17:13   ` [tip:x86/bsp-hotplug] " tip-bot for Fenghua Yu
2012-08-25 17:41 ` [PATCH v8 09/12] x86, hotplug: During CPU0 online, enable x2apic, set_numa_node Fenghua Yu
2012-08-27 17:14   ` [tip:x86/bsp-hotplug] " tip-bot for Fenghua Yu
2012-08-25 17:41 ` [PATCH v8 10/12] x86, hotplug: The first online processor saves the MTRR state Fenghua Yu
2012-08-27 17:15   ` [tip:x86/bsp-hotplug] " tip-bot for Fenghua Yu
2012-08-25 17:41 ` [PATCH v8 11/12] x86/i387.c: Initialize thread xstate only on CPU0 only once Fenghua Yu
2012-08-27 17:16   ` [tip:x86/bsp-hotplug] " tip-bot for Fenghua Yu
2012-08-25 17:41 ` [PATCH v8 12/12] x86, topology: Debug CPU00 hotplug Fenghua Yu
2012-08-27 17:17   ` [tip:x86/bsp-hotplug] x86, topology: Debug CPU0 hotplug tip-bot for Fenghua Yu
2012-09-10  5:50 ` [PATCH v8 0/12] x86: Arbitrary CPU hot(un)plug support Ingo Molnar
2012-11-13 19:32 [PATCH v10 09/14] x86, hotplug: Wake up CPU0 via NMI instead of INIT, SIPI, SIPI Fenghua Yu
2012-11-14 21:29 ` [tip:x86/bsp-hotplug] " tip-bot for Fenghua Yu
     [not found] <1352896613-25957-1-git-send-email-fenghua.yu@intel.com>
2012-11-15  0:23 ` tip-bot for Fenghua Yu

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox

Powered by JetHome