mirror of https://lore.kernel.org/lkml/
 help / color / mirror / Atom feed
From: "Pallipadi, Venkatesh" <venkatesh.pallipadi@intel.com>
To: "Linux Kernel" <linux-kernel@vger.kernel.org>
Cc: "Martin Bligh" <mbligh@us.ibm.com>,
	"John Stultz" <johnstul@us.ibm.com>,
	"Nakajima, Jun" <jun.nakajima@intel.com>, <jamesclv@us.ibm.com>,
	"Mallick, Asit K" <asit.k.mallick@intel.com>,
	"Saxena, Sunil" <sunil.saxena@intel.com>
Subject: [PATCH][2.4]  generic cluster APIC support for systems with more than 8 CPUs
Date: Wed, 18 Dec 2002 14:36:20 -0800	[thread overview]
Message-ID: <C8C38546F90ABF408A5961FC01FDBF1912E18E@fmsmsx405.fm.intel.com> (raw)

[-- Attachment #1: Type: text/plain, Size: 4237 bytes --]


2.4.21-pre1 (i386) based patch to fix the issues with systems having more than
8 CPUs, in a generic way.


Motivation:
The current APIC destination mode ("Flat Logical") used in linux kernel has 
an upper limit of 8 CPUs. For more than 8 CPUs, either "Clustered Logical" or
"Physical" mode has to be used.
There is already some code in current kernel (2.4.21-pre1), to support such 
conditions. Specifically, IBM Summit, uses Physical mode, and IBM NUMAQ 
uses clustered mode. But, there are some issues too:
- it is not generic enough to support any other more than 8 CPU system 
  out of the box. Supporting different systems may need more hacks in the code. 
- clustered mode setup is tightly coupled with NUMA system. Whereas, in reality,
  we can as well have logical clusters in a non-NUMA system as well.
- physical mode setup is somewhat tightly coupled with xAPIC. But, xAPIC doesn't
  necessarily imply physical mode. You can as well have clustered mode with xAPIC
- APIC destination mode is selected based on particular OEM string.

These reasons together led to panics on other OEM systems with > 8 CPUS. The 
patch tries to fix this issue in a generic way (in place of having multiple 
hacks for different OEMs). Note, the patch only intends to change the 
initialization of systems with more than 8 CPUs and it will not affect
other systems (apart from possible bugs in my code itself).


Description:
The basic idea is to use the number of processors detected on the system, to 
decide on which APIC destination mode is to be used. Once all the CPU info, is 
collected either from ACPI or MP table, we can check the total number of 
processors in the system.
If the number of processors in less than equal to 8,
   then no change is required, and we can use the default, "Flat Logical" set up.
If the number of processors is more than 8
   we can switch to clustered logical setup. 
   The logical clusters set up as follows.
   Cluster 0 (CPU 0-3), Cluster 1 (CPU 4-7), Cluster 2 (CPU 8-11) and so on..

The other things that are done in the patch include:
- Separate out the NUMA specific stuff from APIC setup in cluster mode. Also, 
  NUMA has its own way of setting up the clusters, and doesn't follow the 
  logical cluster mapping defined above.
- Separate out xAPIC stuff from APIC destination setup. And the availability of 
  xAPIC support can actually  be determined from the LAPIC version.
- physical mode support _removed_, as we can use clustered logical setup to 
  support can support upto a max of 60 CPUs. This is mainly because of the 
  advantage of being able to setup IOAPICs in LowestPrio, when using clustered mode.

The whole stuff is protected by 'Clustered APIC (> 8 CPUs) support
(CONFIG_X86_APIC_CLUSTER)' config option under Processor Type and Features. 
But going forward, we can actually make this as default, as this doesn't 
affect the systems with less than equal to 8 CPUs (Apart from minor increase 
in code size and couple of additional checks during boot up), but gives the 
default support to more than 8 CPU systems.


Please let me know your comments/criticisms about this. 
I was able to test this successfully on an 8-way with HT(16 logical) 
CPU systems that I have access to. But, I haven't tested it on x440, or NUMAQ 
systems. Would love to hear about the effect of this patch on these systems.

Thanks,
-Venkatesh



> -----Original Message-----
> From: Nakajima, Jun 
> Sent: Thursday, December 12, 2002 7:06 PM
> To: jamesclv@us.ibm.com; Zwane Mwaikambo
> Cc: Martin Bligh; John Stultz; Linux Kernel
> Subject: RE: [PATCH][2.5][RFC] Using xAPIC apic address space 
> on !Summit
> 
> 
> BTW, we are working on a xAPIC patch that supports more than 
> 8 CPUs in a
> generic fashion (don't use hardcode OEM checking). We already 
> tested it on
> two OEM systems with 16 CPUs. 
> - It uses clustered mode. We don't want to use physical mode 
> because it does
> not support lowest priority delivery mode.
> - We also check the version of the local APIC if it's xAPIC 
> or not. It's
> possible that some other x86 architecture (other than P4P) uses xAPIC.
> 
> Stay tuned.
> 
> Jun

[-- Attachment #2: cluster-2.4.21-pre1.patch --]
[-- Type: application/octet-stream, Size: 25995 bytes --]

diff -urN linux-2.4.21-pre1.org/Documentation/Configure.help linux-test1/Documentation/Configure.help
--- linux-2.4.21-pre1.org/Documentation/Configure.help	2002-12-13 17:53:58.000000000 -0800
+++ linux-test1/Documentation/Configure.help	2002-12-14 15:00:39.000000000 -0800
@@ -246,6 +246,13 @@
 
   If unsure, say N.
 
+Clustered APIC support for x86 systems
+CONFIG_X86_APIC_CLUSTER
+  This option is used for getting Linux to run on more than 8 CPU system.
+  This dynamically changes the way that processors are bootstrapped,
+  and uses Clustered Logical APIC addressing mode instead of Flat Logical, for
+  more than 8 CPU system. It doesn't effect the systems with less than 8 CPUs.
+
 Multiquad support for NUMAQ systems
 CONFIG_X86_NUMAQ
   This option is used for getting Linux to run on a (IBM/Sequent) NUMA 
diff -urN linux-2.4.21-pre1.org/arch/i386/config.in linux-test1/arch/i386/config.in
--- linux-2.4.21-pre1.org/arch/i386/config.in	2002-12-13 17:53:58.000000000 -0800
+++ linux-test1/arch/i386/config.in	2002-12-14 14:59:22.000000000 -0800
@@ -216,17 +216,13 @@
       define_bool CONFIG_X86_IO_APIC y
    fi
 else
-   bool 'Multi-node NUMA system support' CONFIG_X86_NUMA
-   if [ "$CONFIG_X86_NUMA" = "y" ]; then
+   bool 'Clustered APIC (> 8 CPUs) support' CONFIG_X86_APIC_CLUSTER
+   if [ "$CONFIG_X86_APIC_CLUSTER" = "y" ]; then
+      define_bool CONFIG_X86_CLUSTERED_APIC y
       #Platform Choices
       bool ' Multiquad (IBM/Sequent) NUMAQ support' CONFIG_X86_NUMAQ
       if [ "$CONFIG_X86_NUMAQ" = "y" ]; then
-         define_bool CONFIG_X86_CLUSTERED_APIC y
-		 define_bool CONFIG_MULTIQUAD y
-      fi
-      bool ' IBM x440 (Summit/EXA) support' CONFIG_X86_SUMMIT
-      if [ "$CONFIG_X86_SUMMIT" = "y" ]; then
-         define_bool CONFIG_X86_CLUSTERED_APIC y
+                 define_bool CONFIG_MULTIQUAD y
       fi
    fi
 fi
diff -urN linux-2.4.21-pre1.org/arch/i386/defconfig linux-test1/arch/i386/defconfig
--- linux-2.4.21-pre1.org/arch/i386/defconfig	2002-11-28 15:53:09.000000000 -0800
+++ linux-test1/arch/i386/defconfig	2002-12-14 14:59:52.000000000 -0800
@@ -62,6 +62,7 @@
 # CONFIG_MATH_EMULATION is not set
 # CONFIG_MTRR is not set
 CONFIG_SMP=y
+CONFIG_X86_APIC_CLUSTER=y
 # CONFIG_MULTIQUAD is not set
 CONFIG_HAVE_DEC_LOCK=y
 
diff -urN linux-2.4.21-pre1.org/arch/i386/kernel/acpitable.c linux-test1/arch/i386/kernel/acpitable.c
--- linux-2.4.21-pre1.org/arch/i386/kernel/acpitable.c	2002-08-02 17:39:42.000000000 -0700
+++ linux-test1/arch/i386/kernel/acpitable.c	2002-12-13 23:25:09.000000000 -0800
@@ -314,12 +314,15 @@
 int have_acpi_tables;
 
 extern void __init MP_processor_info(struct mpc_config_processor *);
+extern unsigned int xapic_support;
 
 static void __init
 acpi_parse_lapic(struct acpi_table_lapic *local_apic)
 {
 	struct mpc_config_processor proc_entry;
 	int ix = 0;
+	static unsigned long apic_ver;
+	static int first_time = 1;
 
 	if (!local_apic)
 		return;
@@ -357,7 +360,16 @@
 		proc_entry.mpc_featureflag = boot_cpu_data.x86_capability[0];
 		proc_entry.mpc_reserved[0] = 0;
 		proc_entry.mpc_reserved[1] = 0;
-		proc_entry.mpc_apicver = 0x10;	/* integrated APIC */
+		if (first_time) {
+			first_time = 0;
+			set_fixmap(FIX_APIC_BASE, APIC_DEFAULT_PHYS_BASE);
+			Dprintk("Local APIC ID %lx\n", apic_read(APIC_ID));
+			apic_ver = apic_read(APIC_LVR);
+			Dprintk("Local APIC Version %lx\n", apic_ver);
+			if (APIC_XAPIC_SUPPORT(apic_ver))
+				xapic_support = 1;
+		}
+		proc_entry.mpc_apicver = apic_ver; 
 		MP_processor_info(&proc_entry);
 	} else {
 		printk(" disabled");
diff -urN linux-2.4.21-pre1.org/arch/i386/kernel/apic.c linux-test1/arch/i386/kernel/apic.c
--- linux-2.4.21-pre1.org/arch/i386/kernel/apic.c	2002-12-13 17:53:58.000000000 -0800
+++ linux-test1/arch/i386/kernel/apic.c	2002-12-13 23:25:09.000000000 -0800
@@ -264,8 +264,8 @@
 static unsigned long calculate_ldr(unsigned long old)
 {
 	unsigned long id;
-	if(clustered_apic_mode == CLUSTERED_APIC_XAPIC)
-		id = physical_to_logical_apicid(hard_smp_processor_id());
+	if(clustered_apic_mode)
+		id = cpu_2_logical_apicid[smp_processor_id()];
 	else
 		id = 1UL << smp_processor_id();
 	return (old & ~APIC_LDR_MASK)|SET_APIC_LOGICAL_ID(id);
@@ -302,22 +302,26 @@
 	 * an APIC.  See e.g. "AP-388 82489DX User's Manual" (Intel
 	 * document number 292116).  So here it goes...
 	 */
-	if (clustered_apic_mode != CLUSTERED_APIC_NUMAQ) {
+	if (configured_platform_type != CONFIGURED_PLATFORM_NUMA) {
+		unsigned int dfr;
 		/*
 		 * For NUMA-Q (clustered apic logical), the firmware does this
 		 * for us. Otherwise put the APIC into clustered or flat
 		 * delivery mode. Must be "all ones" explicitly for 82489DX.
 		 */
-		if(clustered_apic_mode == CLUSTERED_APIC_XAPIC)
-			apic_write_around(APIC_DFR, APIC_DFR_CLUSTER);
+		if(clustered_apic_mode)
+			dfr = APIC_DFR_CLUSTER;
 		else
-			apic_write_around(APIC_DFR, APIC_DFR_FLAT);
+			dfr = APIC_DFR_FLAT;
 
+		apic_write_around(APIC_DFR, dfr);
 		/*
 		 * Set up the logical destination ID.
 		 */
 		value = apic_read(APIC_LDR);
 		apic_write_around(APIC_LDR, calculate_ldr(value));
+		Dprintk("setup_local_APIC: CPU#%d LDR 0x%lx\n", smp_processor_id(), calculate_ldr(value));
+		Dprintk("setup_local_APIC: CPU#%d DFR 0x%lx\n", smp_processor_id(), dfr);
 	}
 
 	/*
diff -urN linux-2.4.21-pre1.org/arch/i386/kernel/io_apic.c linux-test1/arch/i386/kernel/io_apic.c
--- linux-2.4.21-pre1.org/arch/i386/kernel/io_apic.c	2002-12-13 17:53:58.000000000 -0800
+++ linux-test1/arch/i386/kernel/io_apic.c	2002-12-13 23:25:09.000000000 -0800
@@ -40,8 +40,9 @@
 
 static spinlock_t ioapic_lock = SPIN_LOCK_UNLOCKED;
 
-unsigned int int_dest_addr_mode = APIC_DEST_LOGICAL;
-unsigned char int_delivery_mode = dest_LowestPrio;
+extern unsigned int int_dest_addr_mode; /* Default = APIC_DEST_LOGICAL */
+extern unsigned char int_delivery_mode; /* Default = dest_LowestPrio */
+extern unsigned int xapic_support;
 
 
 /*
@@ -658,7 +659,8 @@
 		 * skip adding the timer int on secondary nodes, which causes
 		 * a small but painful rift in the time-space continuum
 		 */
-		if ((clustered_apic_mode == CLUSTERED_APIC_NUMAQ) 
+		if (clustered_apic_mode &&
+			(configured_platform_type == CONFIGURED_PLATFORM_NUMA) 
 			&& (apic != 0) && (irq == 0))
 			continue;
 		else
@@ -1067,7 +1069,8 @@
 		
 		old_id = mp_ioapics[apic].mpc_apicid;
 
-		if (mp_ioapics[apic].mpc_apicid >= apic_broadcast_id) {
+		if ( !xapic_support && 
+			(mp_ioapics[apic].mpc_apicid >= apic_broadcast_id)) {
 			printk(KERN_ERR "BIOS bug, IO-APIC#%d ID is %d in the MPC table!...\n",
 				apic, mp_ioapics[apic].mpc_apicid);
 			printk(KERN_ERR "... fixing up to %d. (tell your hw vendor)\n",
@@ -1081,22 +1084,23 @@
 		 * 'stuck on smp_invalidate_needed IPI wait' messages.
 		 * I/O APIC IDs no longer have any meaning for xAPICs and SAPICs.
 		 */
-		if ((clustered_apic_mode != CLUSTERED_APIC_XAPIC) &&
-		    (phys_id_present_map & (1 << mp_ioapics[apic].mpc_apicid))) {
-			printk(KERN_ERR "BIOS bug, IO-APIC#%d ID %d is already used!...\n",
-				apic, mp_ioapics[apic].mpc_apicid);
-			for (i = 0; i < 0xf; i++)
-				if (!(phys_id_present_map & (1 << i)))
-					break;
-			if (i >= apic_broadcast_id)
-				panic("Max APIC ID exceeded!\n");
-			printk(KERN_ERR "... fixing up to %d. (tell your hw vendor)\n",
-				i);
-			phys_id_present_map |= 1 << i;
-			mp_ioapics[apic].mpc_apicid = i;
-		} else {
-			printk("Setting %d in the phys_id_present_map\n", mp_ioapics[apic].mpc_apicid);
-			phys_id_present_map |= 1 << mp_ioapics[apic].mpc_apicid;
+		if ( !xapic_support ) {
+			if (phys_id_present_map & (1 << mp_ioapics[apic].mpc_apicid)) {
+				printk(KERN_ERR "BIOS bug, IO-APIC#%d ID %d is already used!...\n",
+					apic, mp_ioapics[apic].mpc_apicid);
+				for (i = 0; i < 0xf; i++)
+					if (!(phys_id_present_map & (1 << i)))
+						break;
+				if (i >= 0xf)
+					panic("Max APIC ID exceeded!\n");
+				printk(KERN_ERR "... fixing up to %d. (tell your hw vendor)\n",
+					i);
+				phys_id_present_map |= 1 << i;
+				mp_ioapics[apic].mpc_apicid = i;
+			} else {
+				printk("Setting %d in the phys_id_present_map\n", mp_ioapics[apic].mpc_apicid);
+				phys_id_present_map |= 1 << mp_ioapics[apic].mpc_apicid;
+			}
 		}
 
 
diff -urN linux-2.4.21-pre1.org/arch/i386/kernel/mpparse.c linux-test1/arch/i386/kernel/mpparse.c
--- linux-2.4.21-pre1.org/arch/i386/kernel/mpparse.c	2002-12-13 17:53:58.000000000 -0800
+++ linux-test1/arch/i386/kernel/mpparse.c	2002-12-13 23:25:09.000000000 -0800
@@ -10,6 +10,8 @@
  *		Alan Cox	:	Added EBDA scanning
  *		Ingo Molnar	:	various cleanups and rewrites
  *	Maciej W. Rozycki	:	Bits for default MP configurations
+ *	Venkatesh Pallipadi	:	Added generic support for Clustered
+ *					int. dest. modes
  */
 
 #include <linux/mm.h>
@@ -67,11 +69,17 @@
 unsigned long phys_cpu_present_map;
 unsigned long logical_cpu_present_map;
 
+/* Default values are for Logical Flat destination set up */
 #ifdef CONFIG_X86_CLUSTERED_APIC
 unsigned char esr_disable = 0;
-unsigned char clustered_apic_mode = CLUSTERED_APIC_NONE;
+unsigned char clustered_apic_mode = CONFIGURED_APIC_NONE;
+unsigned char configured_platform_type = CONFIGURED_PLATFORM_NONE;
 unsigned int apic_broadcast_id = APIC_BROADCAST_ID_APIC;
 #endif
+unsigned int xapic_support=0;
+unsigned int int_dest_addr_mode = APIC_DEST_LOGICAL;
+unsigned char int_delivery_mode = dest_LowestPrio;
+
 unsigned char raw_phys_apicid[NR_CPUS] = { [0 ... NR_CPUS-1] = BAD_APICID };
 
 /*
@@ -156,14 +164,15 @@
 		return;
 
 	logical_apicid = m->mpc_apicid;
-	if (clustered_apic_mode == CLUSTERED_APIC_NUMAQ) {
+	if (clustered_apic_mode &&
+		(configured_platform_type == CONFIGURED_PLATFORM_NUMA) ) {
 		quad = translation_table[mpc_record]->trans_quad;
 		logical_apicid = (quad << 4) + 
 			(m->mpc_apicid ? m->mpc_apicid << 1 : 1);
 		printk("Processor #%d %s APIC version %d (quad %d, apic %d)\n",
 			m->mpc_apicid,
 			mpc_family((m->mpc_cpufeature & CPU_FAMILY_MASK)>>8 ,
-				   (m->mpc_cpufeature & CPU_MODEL_MASK)>>4),
+				    (m->mpc_cpufeature & CPU_MODEL_MASK)>>4),
 			m->mpc_apicver, quad, logical_apicid);
 	} else {
 		printk("Processor #%d %s APIC version %d\n",
@@ -236,6 +245,8 @@
 		return;
 	}
 	ver = m->mpc_apicver;
+	if (APIC_XAPIC_SUPPORT(ver))
+		xapic_support = 1;
 
 	logical_cpu_present_map |= 1 << (num_processors-1);
  	phys_cpu_present_map |= apicid_to_phys_cpu_present(m->mpc_apicid);
@@ -259,7 +270,8 @@
 	memcpy(str, m->mpc_bustype, 6);
 	str[6] = 0;
 	
-	if (clustered_apic_mode == CLUSTERED_APIC_NUMAQ) {
+	if (clustered_apic_mode &&
+		(configured_platform_type == CONFIGURED_PLATFORM_NUMA) ) {
 		quad = translation_table[mpc_record]->trans_quad;
 		mp_bus_id_to_node[m->mpc_busid] = quad;
 		mp_bus_id_to_local[m->mpc_busid] = translation_table[mpc_record]->trans_local;
@@ -446,7 +458,9 @@
 	if (!have_acpi_tables)
 		mp_lapic_addr = mpc->mpc_lapic;
 
-	if ((clustered_apic_mode == CLUSTERED_APIC_NUMAQ) && mpc->mpc_oemptr) {
+	if (clustered_apic_mode &&
+		(configured_platform_type == CONFIGURED_PLATFORM_NUMA) && 
+		mpc->mpc_oemptr) {
 		/* We need to process the oem mpc tables to tell us which quad things are in ... */
 		mpc_record = 0;
 		smp_read_mpc_oem((struct mp_config_oemtable *) mpc->mpc_oemptr, mpc->mpc_oemsize);
@@ -516,20 +530,11 @@
 		++mpc_record;
 	}
 
-	if (clustered_apic_mode){
+	if (clustered_apic_mode &&
+	    (configured_platform_type==CONFIGURED_PLATFORM_NUMA)){
 		phys_cpu_present_map = logical_cpu_present_map;
 	}
 
-
-	printk("Enabling APIC mode: ");
-	if(clustered_apic_mode == CLUSTERED_APIC_NUMAQ)
-		printk("Clustered Logical.	");
-	else if(clustered_apic_mode == CLUSTERED_APIC_XAPIC)
-		printk("Physical.	");
-	else
-		printk("Flat.	");
-	printk("Using %d I/O APICs\n",nr_ioapics);
-
 	if (!num_processors)
 		printk(KERN_ERR "SMP mptable: no processors registered!\n");
 	return num_processors;
@@ -712,7 +717,7 @@
 	 */
 	config_acpi_tables();
 #endif
-	
+
 	printk("Intel MultiProcessor Specification v1.%d\n", mpf->mpf_specification);
 	if (mpf->mpf_feature2 & (1<<7)) {
 		printk("    IMCR and PIC compatibility mode.\n");
@@ -764,6 +769,29 @@
 		BUG();
 
 	printk("Processors: %d\n", num_processors);
+
+#ifdef CONFIG_X86_CLUSTERED_APIC
+	/* Default is Logical Flat destination mode */
+	apic_broadcast_id = xapic_support?APIC_BROADCAST_ID_XAPIC:APIC_BROADCAST_ID_APIC;
+	if ((clustered_apic_mode == CONFIGURED_APIC_NONE) && 
+	    (num_processors > FLAT_APIC_CPU_MAX)) {
+		/*Clustered Logical destination mode*/
+		configured_platform_type = CONFIGURED_PLATFORM_NONE;
+		clustered_apic_mode = CONFIGURED_APIC_CLUSTERED;
+		int_dest_addr_mode = APIC_DEST_LOGICAL;
+		int_delivery_mode = dest_LowestPrio;
+		esr_disable = 1;
+	}
+#endif //CONFIG_X86_CLUSTERED_APIC
+
+	printk("Enabling APIC mode: ");
+	if(clustered_apic_mode == CONFIGURED_APIC_CLUSTERED)
+		printk("Clustered Logical.	");
+	else
+		printk("Flat Logical.	");
+	printk("Using %d I/O APICs\n",nr_ioapics);
+	printk("xAPIC support %s\n", (xapic_support?"Enabled":"Disabled"));
+
 	/*
 	 * Only use the first configuration found.
 	 */
diff -urN linux-2.4.21-pre1.org/arch/i386/kernel/pci-pc.c linux-test1/arch/i386/kernel/pci-pc.c
--- linux-2.4.21-pre1.org/arch/i386/kernel/pci-pc.c	2002-11-28 15:53:09.000000000 -0800
+++ linux-test1/arch/i386/kernel/pci-pc.c	2002-12-13 23:25:09.000000000 -0800
@@ -478,7 +478,7 @@
 
 #ifdef CONFIG_MULTIQUAD			
 			/* Multi-Quad has an extended PCI Conf1 */
-			if(clustered_apic_mode == CLUSTERED_APIC_NUMAQ)
+			if(configured_platform_type == CONFIGURED_PLATFORM_NUMA)
 				return &pci_direct_mq_conf1;
 #endif				
 			return &pci_direct_conf1;
@@ -1407,7 +1407,7 @@
 
 	printk(KERN_INFO "PCI: Probing PCI hardware\n");
 	pci_root_bus = pci_scan_bus(0, pci_root_ops, NULL);
-	if (clustered_apic_mode && (numnodes > 1)) {
+	if ( (configured_platform_type==CONFIGURED_PLATFORM_NUMA) && (numnodes > 1)) {
 		for (quad = 1; quad < numnodes; ++quad) {
 			printk("Scanning PCI bus %d for quad %d\n", 
 				QUADLOCAL2BUS(quad,0), quad);
diff -urN linux-2.4.21-pre1.org/arch/i386/kernel/smp.c linux-test1/arch/i386/kernel/smp.c
--- linux-2.4.21-pre1.org/arch/i386/kernel/smp.c	2002-12-13 17:53:58.000000000 -0800
+++ linux-test1/arch/i386/kernel/smp.c	2002-12-13 23:25:09.000000000 -0800
@@ -214,10 +214,7 @@
 			/*
 			 * prepare target chip field
 			 */
-			if(clustered_apic_mode == CLUSTERED_APIC_XAPIC)
-				cfg = __prepare_ICR2(cpu_to_physical_apicid(query_cpu));
-			else
-				cfg = __prepare_ICR2(cpu_to_logical_apicid(query_cpu));
+			cfg = __prepare_ICR2(cpu_to_logical_apicid(query_cpu));
 			apic_write_around(APIC_ICR2, cfg);
 		
 			/*
diff -urN linux-2.4.21-pre1.org/arch/i386/kernel/smpboot.c linux-test1/arch/i386/kernel/smpboot.c
--- linux-2.4.21-pre1.org/arch/i386/kernel/smpboot.c	2002-12-13 17:53:58.000000000 -0800
+++ linux-test1/arch/i386/kernel/smpboot.c	2002-12-13 23:25:09.000000000 -0800
@@ -30,6 +30,8 @@
  *		Tigran Aivazian	:	fixed "0.00 in /proc/uptime on SMP" bug.
  *	Maciej W. Rozycki	:	Bits for genuine 82489DX APICs
  *		Martin J. Bligh	: 	Added support for multi-quad systems
+ *	Venkatesh Pallipadi	: 	Added generic support for Clustered
+ *					int. dest. modes
  */
 
 #include <linux/config.h>
@@ -358,7 +360,7 @@
 	 * our local APIC.  We have to wait for the IPI or we'll
 	 * lock up on an APIC access.
 	 */
-	if (!clustered_apic_mode) 
+	if (configured_platform_type != CONFIGURED_PLATFORM_NUMA) 
 		while (!atomic_read(&init_deasserted));
 
 	/*
@@ -412,7 +414,7 @@
 	 * Because we use NMIs rather than the INIT-STARTUP sequence to
 	 * bootstrap the CPUs, the APIC may be in a wierd state. Kick it.
 	 */
-	if (clustered_apic_mode)
+	if (configured_platform_type == CONFIGURED_PLATFORM_NUMA)
 		clear_local_APIC();
 	setup_local_APIC();
 
@@ -540,9 +542,10 @@
  * else physical apic ids
  */
 {
-	if (clustered_apic_mode == CLUSTERED_APIC_NUMAQ) {
-		logical_apicid_2_cpu[apicid] = cpu;	
+	Dprintk("cpu %d, apicid %d, clustered %d\n", cpu, apicid, clustered_apic_mode);
+	if (clustered_apic_mode) {
 		cpu_2_logical_apicid[cpu] = apicid;
+		logical_apicid_2_cpu[apicid] = cpu;	
 	} else {
 		physical_apicid_2_cpu[apicid] = cpu;	
 		cpu_2_physical_apicid[cpu] = apicid;
@@ -555,7 +558,7 @@
  * else physical apic ids
  */
 {
-	if (clustered_apic_mode == CLUSTERED_APIC_NUMAQ) {
+	if (clustered_apic_mode) {
 		logical_apicid_2_cpu[apicid] = BAD_APICID;
 		cpu_2_logical_apicid[cpu] = BAD_APICID;
 	} else {
@@ -702,6 +705,7 @@
 	 * Determine this based on the APIC version.
 	 * If we don't have an integrated APIC, don't send the STARTUP IPIs.
 	 */
+	Dprintk("###phys_apicid: %d.\n", phys_apicid);
 	if (APIC_INTEGRATED(apic_version[phys_apicid]))
 		num_starts = 2;
 	else
@@ -778,7 +782,7 @@
 static void __init do_boot_cpu (int apicid) 
 /*
  * NOTE - on most systems this is a PHYSICAL apic ID, but on multiquad
- * (ie clustered apic addressing mode), this is a LOGICAL apic ID.
+ * (ie NUMA ), this is a LOGICAL apic ID.
  */
 {
 	struct task_struct *idle;
@@ -806,7 +810,17 @@
 	idle->processor = cpu;
 	idle->cpus_runnable = 1 << cpu; /* we schedule the first task manually */
 
-	map_cpu_to_boot_apicid(cpu, apicid);
+	/*
+	 * In Clustered mode interrupt delivery, without NUMA, we 
+	 * initialize all the CPUs in normal IPI fashion and use their
+	 * logical_apicid while setting the local apics. Thats when clustering
+	 * is enabled. So, setup their logical_apicid here.
+	 */
+	if (clustered_apic_mode &&
+	    (configured_platform_type != CONFIGURED_PLATFORM_NUMA))
+		map_cpu_to_boot_apicid(cpu, physical_to_logical_apicid(apicid));
+	else
+		map_cpu_to_boot_apicid(cpu, apicid);
 
 	idle->thread.eip = (unsigned long) start_secondary;
 
@@ -830,7 +844,7 @@
 
 	Dprintk("Setting warm reset code and vector.\n");
 
-	if (clustered_apic_mode == CLUSTERED_APIC_NUMAQ) {
+	if (configured_platform_type == CONFIGURED_PLATFORM_NUMA) {
 		/* stash the current NMI vector, so we can put things back */
 		nmi_high = *((volatile unsigned short *) TRAMPOLINE_HIGH);
 		nmi_low = *((volatile unsigned short *) TRAMPOLINE_LOW);
@@ -847,7 +861,7 @@
 	/*
 	 * Be paranoid about clearing APIC errors.
 	 */
-	if (!clustered_apic_mode && APIC_INTEGRATED(apic_version[apicid])) {
+	if ((configured_platform_type!=CONFIGURED_PLATFORM_NUMA) && APIC_INTEGRATED(apic_version[apicid])) {
 		apic_read_around(APIC_SPIV);
 		apic_write(APIC_ESR, 0);
 		apic_read(APIC_ESR);
@@ -862,7 +876,7 @@
 	 * Starting actual IPI sequence...
 	 */
 
-	if (clustered_apic_mode == CLUSTERED_APIC_NUMAQ)
+	if (configured_platform_type == CONFIGURED_PLATFORM_NUMA)
 		boot_error = wakeup_secondary_via_NMI(apicid);
 	else 
 		boot_error = wakeup_secondary_via_INIT(apicid, start_eip);
@@ -917,7 +931,7 @@
 	/* mark "stuck" area as not stuck */
 	*((volatile unsigned long *)phys_to_virt(8192)) = 0;
 
-	if(clustered_apic_mode == CLUSTERED_APIC_NUMAQ) {
+	if(configured_platform_type == CONFIGURED_PLATFORM_NUMA) {
 		printk("Restoring NMI vector\n");
 		*((volatile unsigned short *) TRAMPOLINE_HIGH) = nmi_high;
 		*((volatile unsigned short *) TRAMPOLINE_LOW) = nmi_low;
@@ -981,7 +995,7 @@
 {
 	int apicid, cpu, bit;
 
-        if ((clustered_apic_mode == CLUSTERED_APIC_NUMAQ) && (numnodes > 1)) {
+        if ((configured_platform_type == CONFIGURED_PLATFORM_NUMA) && (numnodes > 1)) {
                 printk("Remapping cross-quad port I/O for %d quads\n",
 			numnodes);
                 printk("xquad_portio vaddr 0x%08lx, len %08lx\n",
@@ -1019,11 +1033,22 @@
 	 * We have the boot CPU online for sure.
 	 */
 	set_bit(0, &cpu_online_map);
-	if (clustered_apic_mode == CLUSTERED_APIC_XAPIC)
+	if (clustered_apic_mode)
 		boot_cpu_logical_apicid = physical_to_logical_apicid(boot_cpu_physical_apicid);
 	else
 		boot_cpu_logical_apicid = logical_smp_processor_id();
-	map_cpu_to_boot_apicid(0, boot_cpu_apicid);
+
+	/*
+	 * In Clustered mode interrupt delivery, without NUMA, we
+	 * initialize all the CPUs in normal IPI fashion and use their
+	 * logical_apicid while setting the local apics. Thats when clustering 
+	 * is enabled. So, setup their logical_apicid here.
+	 */
+	if (clustered_apic_mode &&
+	    (configured_platform_type != CONFIGURED_PLATFORM_NUMA))
+		map_cpu_to_boot_apicid(0, physical_to_logical_apicid(boot_cpu_apicid));
+	else
+		map_cpu_to_boot_apicid(0, boot_cpu_apicid);
 
 	global_irq_holder = 0;
 	current->processor = 0;
@@ -1111,6 +1136,7 @@
 		/*
 		 * Don't even attempt to start the boot CPU!
 		 */
+		Dprintk("apicid %d, boot_cpu_apicid %d, bit %d\n", apicid, boot_cpu_apicid, bit);
 		if (apicid == boot_cpu_apicid)
 			continue;
 
@@ -1201,7 +1227,7 @@
 			}
 		}
 	}
-	     
+	    
 #ifndef CONFIG_VISWS
 	/*
 	 * Here we can be sure that there is an IO-APIC in the system. Let's
diff -urN linux-2.4.21-pre1.org/include/asm-i386/apicdef.h linux-test1/include/asm-i386/apicdef.h
--- linux-2.4.21-pre1.org/include/asm-i386/apicdef.h	2002-12-13 17:53:58.000000000 -0800
+++ linux-test1/include/asm-i386/apicdef.h	2002-12-13 23:26:22.000000000 -0800
@@ -18,6 +18,7 @@
 #define			GET_APIC_VERSION(x)	((x)&0xFF)
 #define			GET_APIC_MAXLVT(x)	(((x)>>16)&0xFF)
 #define			APIC_INTEGRATED(x)	((x)&0xF0)
+#define			APIC_XAPIC_SUPPORT(x)	(x >= 0x14)
 #define		APIC_TASKPRI	0x80
 #define			APIC_TPRI_MASK		0xFF
 #define		APIC_ARBPRI	0x90
diff -urN linux-2.4.21-pre1.org/include/asm-i386/smpboot.h linux-test1/include/asm-i386/smpboot.h
--- linux-2.4.21-pre1.org/include/asm-i386/smpboot.h	2002-12-13 17:53:58.000000000 -0800
+++ linux-test1/include/asm-i386/smpboot.h	2002-12-13 23:26:40.000000000 -0800
@@ -3,14 +3,22 @@
 
 /*emum for clustered_apic_mode values*/
 enum{
-	CLUSTERED_APIC_NONE = 0,
-	CLUSTERED_APIC_XAPIC,
-	CLUSTERED_APIC_NUMAQ
+	CONFIGURED_APIC_NONE = 0,
+	CONFIGURED_APIC_CLUSTERED
 };
 
+/*emum for configured_platform_type values*/
+enum{
+	CONFIGURED_PLATFORM_NONE = 0,
+	CONFIGURED_PLATFORM_NUMA
+};
+
+#define FLAT_APIC_CPU_MAX	8
+
 #ifdef CONFIG_X86_CLUSTERED_APIC
 extern unsigned int apic_broadcast_id;
 extern unsigned char clustered_apic_mode;
+extern unsigned char configured_platform_type;
 extern unsigned char esr_disable;
 extern unsigned char int_delivery_mode;
 extern unsigned int int_dest_addr_mode;
@@ -20,14 +28,15 @@
 	 * Can't recognize Summit xAPICs at present, so use the OEM ID.
 	 */
 	if (!strncmp(oem, "IBM ENSW", 8) && !strncmp(prod, "VIGIL SMP", 9)){
-		clustered_apic_mode = CLUSTERED_APIC_XAPIC;
-		apic_broadcast_id = APIC_BROADCAST_ID_XAPIC;
-		int_dest_addr_mode = APIC_DEST_PHYSICAL;
-		int_delivery_mode = dest_Fixed;
+		clustered_apic_mode = CONFIGURED_APIC_CLUSTERED;
+		apic_broadcast_id = APIC_BROADCAST_ID_APIC;
+		int_dest_addr_mode = APIC_DEST_LOGICAL;
+		int_delivery_mode = dest_LowestPrio;
 		esr_disable = 1;
 	}
 	else if (!strncmp(oem, "IBM NUMA", 8)){
-		clustered_apic_mode = CLUSTERED_APIC_NUMAQ;
+		configured_platform_type = CONFIGURED_PLATFORM_NUMA;
+		clustered_apic_mode = CONFIGURED_APIC_CLUSTERED;
 		apic_broadcast_id = APIC_BROADCAST_ID_APIC;
 		int_dest_addr_mode = APIC_DEST_LOGICAL;
 		int_delivery_mode = dest_LowestPrio;
@@ -38,7 +47,8 @@
 #define	INT_DELIVERY_MODE (int_delivery_mode)
 #else /* CONFIG_X86_CLUSTERED_APIC */
 #define apic_broadcast_id (APIC_BROADCAST_ID_APIC)
-#define clustered_apic_mode (CLUSTERED_APIC_NONE)
+#define configured_platform_type (CONFIGURED_PLATFORM_NONE)
+#define clustered_apic_mode (CONFIGURED_APIC_NONE)
 #define esr_disable (0)
 #define detect_clustered_apic(x,y)
 #define INT_DEST_ADDR_MODE (APIC_DEST_LOGICAL)	/* logical delivery */
@@ -46,10 +56,10 @@
 #endif /* CONFIG_X86_CLUSTERED_APIC */
 #define BAD_APICID 0xFFu
 
-#define TRAMPOLINE_LOW phys_to_virt((clustered_apic_mode == CLUSTERED_APIC_NUMAQ)?0x8:0x467)
-#define TRAMPOLINE_HIGH phys_to_virt((clustered_apic_mode == CLUSTERED_APIC_NUMAQ)?0xa:0x469)
+#define TRAMPOLINE_LOW phys_to_virt((configured_platform_type == CONFIGURED_PLATFORM_NUMA)?0x8:0x467)
+#define TRAMPOLINE_HIGH phys_to_virt((configured_platform_type == CONFIGURED_PLATFORM_NUMA)?0xa:0x469)
 
-#define boot_cpu_apicid ((clustered_apic_mode == CLUSTERED_APIC_NUMAQ)?boot_cpu_logical_apicid:boot_cpu_physical_apicid)
+#define boot_cpu_apicid ((configured_platform_type == CONFIGURED_PLATFORM_NUMA)? boot_cpu_logical_apicid: boot_cpu_physical_apicid)
 
 extern unsigned char raw_phys_apicid[NR_CPUS];
 
@@ -58,21 +68,30 @@
  */
 static inline int cpu_present_to_apicid(int mps_cpu)
 {
-	if (clustered_apic_mode == CLUSTERED_APIC_XAPIC)
-		return raw_phys_apicid[mps_cpu];
-	if(clustered_apic_mode == CLUSTERED_APIC_NUMAQ)
+	if (clustered_apic_mode &&
+		(configured_platform_type == CONFIGURED_PLATFORM_NUMA))
 		return (mps_cpu/4)*16 + (1<<(mps_cpu%4));
 	return mps_cpu;
 }
 
 static inline unsigned long apicid_to_phys_cpu_present(int apicid)
 {
-	if(clustered_apic_mode)
+	if (clustered_apic_mode &&
+		(configured_platform_type == CONFIGURED_PLATFORM_NUMA))
 		return 1UL << (((apicid >> 4) << 2) + (apicid & 0x3));
 	return 1UL << apicid;
 }
 
-#define physical_to_logical_apicid(phys_apic) ( (1ul << (phys_apic & 0x3)) | (phys_apic & 0xF0u) )
+static inline unsigned long physical_to_logical_apicid(int apicid)
+{
+	if (clustered_apic_mode) {
+		if (configured_platform_type == CONFIGURED_PLATFORM_NUMA)
+			return ((1ul << (apicid & 0x3)) | (apicid & 0xF0u));
+		else
+			return ((1ul << (apicid & 0x3)) + ((apicid&(~0x3))<<2));
+	}
+	return apicid;
+}
 
 /*
  * Mappings between logical cpu number and logical / physical apicid
@@ -100,13 +119,12 @@
 {
 	static int cpu;
 	switch(clustered_apic_mode){
-		case CLUSTERED_APIC_NUMAQ:
+		case CONFIGURED_APIC_CLUSTERED:
 			/* Broadcast intrs to local quad only. */
-			return APIC_BROADCAST_ID_APIC;
-		case CLUSTERED_APIC_XAPIC:
-			/*round robin the interrupts*/
-			cpu = (cpu+1)%smp_num_cpus;
-			return cpu_to_physical_apicid(cpu);
+#define APIC_BROADCAST_CLUSTER	0xf
+			if (configured_platform_type == CONFIGURED_PLATFORM_NUMA)
+				return APIC_BROADCAST_CLUSTER;
+			return (cpu_online_map&APIC_BROADCAST_CLUSTER);
 		default:
 	}
 	return cpu_online_map;

             reply	other threads:[~2002-12-18 22:28 UTC|newest]

Thread overview: 20+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2002-12-18 22:36 Pallipadi, Venkatesh [this message]
2002-12-18 23:26 ` Christoph Hellwig
2002-12-18 23:41   ` William Lee Irwin III
2002-12-18 23:59   ` Martin J. Bligh
2002-12-19  0:24 ` Martin J. Bligh
2002-12-20  2:04 ` James Cleverdon
2002-12-20  8:00   ` Christoph Hellwig
2002-12-20 11:24     ` William Lee Irwin III
2002-12-20 11:29       ` William Lee Irwin III
2002-12-19  1:05 Pallipadi, Venkatesh
2002-12-19  1:32 ` James Cleverdon
2002-12-19  2:35 Pallipadi, Venkatesh
2002-12-19  3:10 ` Martin J. Bligh
2002-12-19  2:45 Pallipadi, Venkatesh
2002-12-19  4:14 ` James Cleverdon
2002-12-21  3:27 Nakajima, Jun
2002-12-21  4:43 ` Martin J. Bligh
2003-01-07 22:42 Andrew Theurer
2003-01-11  4:28 ` James Cleverdon
2003-01-08  3:19 Kamble, Nitin A

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=C8C38546F90ABF408A5961FC01FDBF1912E18E@fmsmsx405.fm.intel.com \
    --to=venkatesh.pallipadi@intel.com \
    --cc=asit.k.mallick@intel.com \
    --cc=jamesclv@us.ibm.com \
    --cc=johnstul@us.ibm.com \
    --cc=jun.nakajima@intel.com \
    --cc=linux-kernel@vger.kernel.org \
    --cc=mbligh@us.ibm.com \
    --cc=sunil.saxena@intel.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox

all inboxes | Powered by JetHome®