mirror of https://lore.kernel.org/lkml/
 help / color / mirror / Atom feed
* [PATCH][2.5] avoid scribbling in IDT with high interrupt count.
@ 2003-04-07  6:03 Zwane Mwaikambo
  2003-04-07 15:27 ` Linus Torvalds
  0 siblings, 1 reply; 4+ messages in thread
From: Zwane Mwaikambo @ 2003-04-07  6:03 UTC (permalink / raw)
  To: Linux Kernel; +Cc: Linus Torvalds

On systems with high interrupt counts we end up overshooting the 
interrupt[] array and dumping garbage entry points into the idt. This 
patch checks for out of bounds access during ioapic setup as well as 
returning -ENOSPC when we run out of vectors to assign in 
assign_irq_vector, thus allowing us to remove the panic and boot a 320 
interrupt source system with 2.5.66. Patch appended

(Hardware courtesy of OSDL)
Tested and booted on failure case system;
32 processor NUMAQ 16 IOAPICs, 320 interrupt sources

and regression tested on
8 processor 1 IOAPIC, 63 interrupt sources

Total of 32 processors activated (31453.18 BogoMIPS).
ENABLING IO-APIC IRQs
 printing eip:
c010c954
*pde = 00000000
Oops: 0002
CPU:    0
EIP:    0060:[<c010c954>]    Not tainted
EFLAGS: 00010202
EIP is at set_intr_gate+0x14/0x30
eax: 00606f20   ebx: 07070707   ecx: 07070707   edx: c0138e00
esi: c0136f20   edi: 0000000c   ebp: 00000127   esp: c3c9ff40
ds: 007b   es: 007b   ss: 0068
Process swapper (pid: 1, threadinfo=c3c9e000 task=c3c9c040)
Stack: c037cd34 07070707 c0136f20 c3c9ff60 07070707 00000000 00000007 0000000c 
       0001a107 0f000000 000072ed 00000286 c03ee396 00000246 00000000 00eff800 
       00000010 00000000 c037d662 c02c6181 000072b4 00000296 c03ee3b9 00000246 
Call Trace:
 [<c0136f20>] sys_setregid16+0x0/0x30
 [<c0122f28>] printk+0x1d8/0x230
 [<c01050ec>] init+0x6c/0x220
 [<c0105080>] init+0x0/0x220
 [<c0108bb5>] kernel_thread_helper+0x5/0x10

Code: 89 04 cd 00 10 35 c0 89 14 cd 04 10 35 c0 c3 8d b6 00 00 00 
 <0>Kernel panic: Attempted to kill init!


Index: linux-2.5.66/arch/i386/kernel/io_apic.c
===================================================================
RCS file: /build/cvsroot/linux-2.5.66/arch/i386/kernel/io_apic.c,v
retrieving revision 1.1.1.1
diff -u -p -B -r1.1.1.1 io_apic.c
--- linux-2.5.66/arch/i386/kernel/io_apic.c	24 Mar 2003 23:40:27 -0000	1.1.1.1
+++ linux-2.5.66/arch/i386/kernel/io_apic.c	3 Apr 2003 08:25:48 -0000
@@ -1105,6 +1105,9 @@ static int __init assign_irq_vector(int 
 	if (IO_APIC_VECTOR(irq) > 0)
 		return IO_APIC_VECTOR(irq);
 next:
+	if (current_vector == -ENOSPC)
+		goto out;
+
 	current_vector += 8;
 	if (current_vector == SYSCALL_VECTOR)
 		goto next;
@@ -1114,10 +1117,14 @@ next:
 		current_vector = FIRST_DEVICE_VECTOR + offset;
 	}
 
-	if (current_vector == FIRST_SYSTEM_VECTOR)
-		panic("ran out of interrupt sources!");
+	if (current_vector == FIRST_SYSTEM_VECTOR) {
+		printk(KERN_WARNING "ran out of interrupt vectors!\n");
+		current_vector = -ENOSPC;
+		goto out;
+	}
 
 	IO_APIC_VECTOR(irq) = current_vector;
+out:
 	return current_vector;
 }
 
@@ -1164,6 +1171,9 @@ void __init setup_IO_APIC_irqs(void)
 		}
 
 		irq = pin_2_irq(idx, apic, pin);
+		if (irq > NR_IRQS)
+			continue;
+
 		/*
 		 * skip adding the timer int on secondary nodes, which causes
 		 * a small but painful rift in the time-space continuum
@@ -1178,8 +1188,10 @@ void __init setup_IO_APIC_irqs(void)
 
 		if (IO_APIC_IRQ(irq)) {
 			vector = assign_irq_vector(irq);
-			entry.vector = vector;
+			if (vector < 0)
+				continue;
 
+			entry.vector = vector;
 			if (IO_APIC_irq_trigger(irq))
 				irq_desc[irq].handler = &ioapic_level_irq_type;
 			else
@@ -2274,6 +2286,10 @@ int io_apic_set_pci_routing (int ioapic,
 {
 	struct IO_APIC_route_entry entry;
 	unsigned long flags;
+	int vector;
+
+	if (irq > NR_IRQS)
+		return -ENOSPC;
 
 	if (!IO_APIC_IRQ(irq)) {
 		printk(KERN_ERR "IOAPIC[%d]: Invalid reference to IRQ 0/n", 
@@ -2298,8 +2314,11 @@ int io_apic_set_pci_routing (int ioapic,
 
 	add_pin_to_irq(irq, ioapic, pin);
 
-	entry.vector = assign_irq_vector(irq);
+	vector = assign_irq_vector(irq);
+	if (vector < 0)
+		return -ENOSPC;
 
+	entry.vector = vector;
 	printk(KERN_DEBUG "IOAPIC[%d]: Set PCI routing entry (%d-%d -> 0x%x -> "
 		"IRQ %d)\n", ioapic, 
 		mp_ioapics[ioapic].mpc_apicid, pin, entry.vector, irq);

^ permalink raw reply	[flat|nested] 4+ messages in thread

* Re: [PATCH][2.5] avoid scribbling in IDT with high interrupt count.
  2003-04-07  6:03 [PATCH][2.5] avoid scribbling in IDT with high interrupt count Zwane Mwaikambo
@ 2003-04-07 15:27 ` Linus Torvalds
  2003-04-08  2:04   ` Zwane Mwaikambo
  0 siblings, 1 reply; 4+ messages in thread
From: Linus Torvalds @ 2003-04-07 15:27 UTC (permalink / raw)
  To: Zwane Mwaikambo; +Cc: Linux Kernel


On Mon, 7 Apr 2003, Zwane Mwaikambo wrote:
>
> On systems with high interrupt counts we end up overshooting the 
> interrupt[] array and dumping garbage entry points into the idt. This 
> patch checks for out of bounds access during ioapic setup as well as 
> returning -ENOSPC when we run out of vectors to assign in 
> assign_irq_vector, thus allowing us to remove the panic and boot a 320 
> interrupt source system with 2.5.66.

Zwane - is there any reason we couldn't just start re-using irq vector
offsets when this happens? We already re-use the vectors themselves, so 
restarting the offset pointer shouldn't really _change_ anything.

In other words, I'm wondering if this simpler patch wouldn't be sufficient 
instead?

Can you please test this, and re-submit (and if you can explain why your 
patch is better, please do so - I have nothing fundamentally against it, I 
just want to understand _why_ the complexity is needed).

Thanks,

		Linus

--- 1.58/arch/i386/kernel/io_apic.c	Thu Mar 20 03:11:41 2003
+++ edited/arch/i386/kernel/io_apic.c	Mon Apr  7 08:25:48 2003
@@ -1110,12 +1110,9 @@
 		goto next;
 
 	if (current_vector > FIRST_SYSTEM_VECTOR) {
-		offset++;
+		offset = (offset+1) & 7;
 		current_vector = FIRST_DEVICE_VECTOR + offset;
 	}
-
-	if (current_vector == FIRST_SYSTEM_VECTOR)
-		panic("ran out of interrupt sources!");
 
 	IO_APIC_VECTOR(irq) = current_vector;
 	return current_vector;


^ permalink raw reply	[flat|nested] 4+ messages in thread

* Re: [PATCH][2.5] avoid scribbling in IDT with high interrupt count.
  2003-04-07 15:27 ` Linus Torvalds
@ 2003-04-08  2:04   ` Zwane Mwaikambo
  2003-04-08  2:31     ` Martin J. Bligh
  0 siblings, 1 reply; 4+ messages in thread
From: Zwane Mwaikambo @ 2003-04-08  2:04 UTC (permalink / raw)
  To: Linus Torvalds; +Cc: Linux Kernel

On Mon, 7 Apr 2003, Linus Torvalds wrote:

> Zwane - is there any reason we couldn't just start re-using irq vector
> offsets when this happens? We already re-use the vectors themselves, so 
> restarting the offset pointer shouldn't really _change_ anything.

My patch skips irq numbers > NR_IRQS and simply return error when we run 
out of vectors, although mainline doesn't assign duplicates and cause 
collisions, it does waste vector space. e.g. for a system with 
NR_IRQS = 224 we have 23 vectors free, 0 collisions and 167 useable irqs 
and assign_irq_vectors states that we're out of vectors.

> In other words, I'm wondering if this simpler patch wouldn't be sufficient 
> instead?
> 
> Can you please test this, and re-submit (and if you can explain why your 
> patch is better, please do so - I have nothing fundamentally against it, I 
> just want to understand _why_ the complexity is needed).

Your patch booted the system but there are vector collisions resulting in 
lost irq routing when we program the IOAPIC with the duplicated vector. So 
with your patch and NR_IRQS = 224 we have 1 vectors free (0x80), 34 collisions 
and 225 irqs. However that isn't a fault of your patch but a fault with 
the NR_IRQS definition. This is what the vector space looks like on i386 
at present.

0________0x31__________________________0xef____________0xff
  system	   io interrupts            resvd vectors

0xef - 0x31 = 190 useable io interrupt vectors

So perhaps we should, apply your patch, add a NR_IRQ_VECTORS define 
and also add commentary in irq_vectors.h how does the following look? I 
had to readd the NR_IRQS checks to protect against overrunning NR_IRQS sized 
arrays and i added nr_assigned to track how many vectors were allocated so 
taht we can bail out when we're out.

Patch has been tested on a 320 interrupt system and had a maximum useable 
irq line of 211 (ethernet).

Thanks,
	Zwane

Index: linux-2.5.67/include/asm-i386/mach-default/irq_vectors.h
===================================================================
RCS file: /build/cvsroot/linux-2.5.67/include/asm-i386/mach-default/irq_vectors.h,v
retrieving revision 1.1.1.1
diff -u -p -B -r1.1.1.1 irq_vectors.h
--- linux-2.5.67/include/asm-i386/mach-default/irq_vectors.h	8 Apr 2003 01:15:29 -0000	1.1.1.1
+++ linux-2.5.67/include/asm-i386/mach-default/irq_vectors.h	8 Apr 2003 01:34:54 -0000
@@ -68,15 +68,22 @@
 #define TIMER_IRQ 0
 
 /*
- * 16 8259A IRQ's, 208 potential APIC interrupt sources.
+ * 16 8259A IRQ's, MAX_IRQ_SOURCES-16 potential APIC interrupt sources.
  * Right now the APIC is mostly only used for SMP.
  * 256 vectors is an architectural limit. (we can have
  * more than 256 devices theoretically, but they will
  * have to use shared interrupts)
  * Since vectors 0x00-0x1f are used/reserved for the CPU,
- * the usable vector space is 0x20-0xff (224 vectors)
+ * the usable vector space is 0x20-0xff (224 vectors).
+ * Linux currently makes 190 vectors available for io interrupts
+ * starting at FIRST_DEVICE_VECTOR till FIRST_SYSTEM_VECTOR
+ * 
+ * 0________0x31__________________________0xef______0xff
+ *   system           io interrupts           resvd
+ *
  */
 #ifdef CONFIG_X86_IO_APIC
+#define NR_IRQ_VECTORS	190
 #define NR_IRQS 224
 #else
 #define NR_IRQS 16
Index: linux-2.5.67/arch/i386/kernel/io_apic.c
===================================================================
RCS file: /build/cvsroot/linux-2.5.67/arch/i386/kernel/io_apic.c,v
retrieving revision 1.1.1.1
diff -u -p -B -r1.1.1.1 io_apic.c
--- linux-2.5.67/arch/i386/kernel/io_apic.c	8 Apr 2003 01:15:35 -0000	1.1.1.1
+++ linux-2.5.67/arch/i386/kernel/io_apic.c	8 Apr 2003 01:36:06 -0000
@@ -1107,9 +1107,15 @@ int irq_vector[NR_IRQS] = { FIRST_DEVICE
 
 static int __init assign_irq_vector(int irq)
 {
-	static int current_vector = FIRST_DEVICE_VECTOR, offset = 0;
+	static int current_vector = FIRST_DEVICE_VECTOR, offset = 0,
+		nr_assigned = 1;
+
 	if (IO_APIC_VECTOR(irq) > 0)
 		return IO_APIC_VECTOR(irq);
+
+	if (++nr_assigned > NR_IRQ_VECTORS)
+		return -ENOSPC;
+
 next:
 	current_vector += 8;
 	if (current_vector == SYSCALL_VECTOR)
@@ -1167,6 +1173,8 @@ void __init setup_IO_APIC_irqs(void)
 		}
 
 		irq = pin_2_irq(idx, apic, pin);
+		if (irq >= NR_IRQS)
+			continue;
 		/*
 		 * skip adding the timer int on secondary nodes, which causes
 		 * a small but painful rift in the time-space continuum
@@ -1181,6 +1189,9 @@ void __init setup_IO_APIC_irqs(void)
 
 		if (IO_APIC_IRQ(irq)) {
 			vector = assign_irq_vector(irq);
+			if (vector < 0)
+				continue;
+
 			entry.vector = vector;
 
 			if (IO_APIC_irq_trigger(irq))
@@ -2277,6 +2288,10 @@ int io_apic_set_pci_routing (int ioapic,
 {
 	struct IO_APIC_route_entry entry;
 	unsigned long flags;
+	int vector;
+
+	if (irq >= NR_IRQS)
+		return -ENOSPC;
 
 	if (!IO_APIC_IRQ(irq)) {
 		printk(KERN_ERR "IOAPIC[%d]: Invalid reference to IRQ 0/n", 
@@ -2301,8 +2316,11 @@ int io_apic_set_pci_routing (int ioapic,
 
 	add_pin_to_irq(irq, ioapic, pin);
 
-	entry.vector = assign_irq_vector(irq);
+	vector = assign_irq_vector(irq);
+	if (vector < 0)
+		return -ENOSPC;
 
+	entry.vector = vector;
 	printk(KERN_DEBUG "IOAPIC[%d]: Set PCI routing entry (%d-%d -> 0x%x -> "
 		"IRQ %d)\n", ioapic, 
 		mp_ioapics[ioapic].mpc_apicid, pin, entry.vector, irq);

-- 
function.linuxpower.ca

^ permalink raw reply	[flat|nested] 4+ messages in thread

* Re: [PATCH][2.5] avoid scribbling in IDT with high interrupt count.
  2003-04-08  2:04   ` Zwane Mwaikambo
@ 2003-04-08  2:31     ` Martin J. Bligh
  0 siblings, 0 replies; 4+ messages in thread
From: Martin J. Bligh @ 2003-04-08  2:31 UTC (permalink / raw)
  To: Zwane Mwaikambo, Linus Torvalds; +Cc: Linux Kernel

> My patch skips irq numbers > NR_IRQS and simply return error when we run 
> out of vectors, although mainline doesn't assign duplicates and cause 
> collisions, it does waste vector space. e.g. for a system with 
> NR_IRQS = 224 we have 23 vectors free, 0 collisions and 167 useable irqs 
> and assign_irq_vectors states that we're out of vectors.
> 
>> In other words, I'm wondering if this simpler patch wouldn't be sufficient 
>> instead?
>> 
>> Can you please test this, and re-submit (and if you can explain why your 
>> patch is better, please do so - I have nothing fundamentally against it, I 
>> just want to understand _why_ the complexity is needed).
> 
> Your patch booted the system but there are vector collisions resulting in 
> lost irq routing when we program the IOAPIC with the duplicated vector. So 
> with your patch and NR_IRQS = 224 we have 1 vectors free (0x80), 34 collisions 
> and 225 irqs. However that isn't a fault of your patch but a fault with 
> the NR_IRQS definition. This is what the vector space looks like on i386 
> at present.
> 
> 0________0x31__________________________0xef____________0xff
>   system	   io interrupts            resvd vectors
> 
> 0xef - 0x31 = 190 useable io interrupt vectors
> 
> So perhaps we should, apply your patch, add a NR_IRQ_VECTORS define 
> and also add commentary in irq_vectors.h how does the following look? I 
> had to readd the NR_IRQS checks to protect against overrunning NR_IRQS sized 
> arrays and i added nr_assigned to track how many vectors were allocated so 
> taht we can bail out when we're out.
> 
> Patch has been tested on a 320 interrupt system and had a maximum useable 
> irq line of 211 (ethernet).

We're still allocating interrupts to a bunch 'o stuff that isn't actually
being used ... I'd prefer it if we could change that, and allocate them
as they're actually used, rather than (what I believe is) still a fixed
number per IO-APIC. Is there some smart way to do that?

If not, the overflow protection is still (obviously) a good thing to do.

M.


^ permalink raw reply	[flat|nested] 4+ messages in thread

end of thread, other threads:[~2003-04-08  2:23 UTC | newest]

Thread overview: 4+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2003-04-07  6:03 [PATCH][2.5] avoid scribbling in IDT with high interrupt count Zwane Mwaikambo
2003-04-07 15:27 ` Linus Torvalds
2003-04-08  2:04   ` Zwane Mwaikambo
2003-04-08  2:31     ` Martin J. Bligh

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox

all inboxes | Powered by JetHome®