* [PATCH][2.5] avoid scribbling in IDT with high interrupt count.
@ 2003-04-07 6:03 Zwane Mwaikambo
2003-04-07 15:27 ` Linus Torvalds
0 siblings, 1 reply; 4+ messages in thread
From: Zwane Mwaikambo @ 2003-04-07 6:03 UTC (permalink / raw)
To: Linux Kernel; +Cc: Linus Torvalds
On systems with high interrupt counts we end up overshooting the
interrupt[] array and dumping garbage entry points into the idt. This
patch checks for out of bounds access during ioapic setup as well as
returning -ENOSPC when we run out of vectors to assign in
assign_irq_vector, thus allowing us to remove the panic and boot a 320
interrupt source system with 2.5.66. Patch appended
(Hardware courtesy of OSDL)
Tested and booted on failure case system;
32 processor NUMAQ 16 IOAPICs, 320 interrupt sources
and regression tested on
8 processor 1 IOAPIC, 63 interrupt sources
Total of 32 processors activated (31453.18 BogoMIPS).
ENABLING IO-APIC IRQs
printing eip:
c010c954
*pde = 00000000
Oops: 0002
CPU: 0
EIP: 0060:[<c010c954>] Not tainted
EFLAGS: 00010202
EIP is at set_intr_gate+0x14/0x30
eax: 00606f20 ebx: 07070707 ecx: 07070707 edx: c0138e00
esi: c0136f20 edi: 0000000c ebp: 00000127 esp: c3c9ff40
ds: 007b es: 007b ss: 0068
Process swapper (pid: 1, threadinfo=c3c9e000 task=c3c9c040)
Stack: c037cd34 07070707 c0136f20 c3c9ff60 07070707 00000000 00000007 0000000c
0001a107 0f000000 000072ed 00000286 c03ee396 00000246 00000000 00eff800
00000010 00000000 c037d662 c02c6181 000072b4 00000296 c03ee3b9 00000246
Call Trace:
[<c0136f20>] sys_setregid16+0x0/0x30
[<c0122f28>] printk+0x1d8/0x230
[<c01050ec>] init+0x6c/0x220
[<c0105080>] init+0x0/0x220
[<c0108bb5>] kernel_thread_helper+0x5/0x10
Code: 89 04 cd 00 10 35 c0 89 14 cd 04 10 35 c0 c3 8d b6 00 00 00
<0>Kernel panic: Attempted to kill init!
Index: linux-2.5.66/arch/i386/kernel/io_apic.c
===================================================================
RCS file: /build/cvsroot/linux-2.5.66/arch/i386/kernel/io_apic.c,v
retrieving revision 1.1.1.1
diff -u -p -B -r1.1.1.1 io_apic.c
--- linux-2.5.66/arch/i386/kernel/io_apic.c 24 Mar 2003 23:40:27 -0000 1.1.1.1
+++ linux-2.5.66/arch/i386/kernel/io_apic.c 3 Apr 2003 08:25:48 -0000
@@ -1105,6 +1105,9 @@ static int __init assign_irq_vector(int
if (IO_APIC_VECTOR(irq) > 0)
return IO_APIC_VECTOR(irq);
next:
+ if (current_vector == -ENOSPC)
+ goto out;
+
current_vector += 8;
if (current_vector == SYSCALL_VECTOR)
goto next;
@@ -1114,10 +1117,14 @@ next:
current_vector = FIRST_DEVICE_VECTOR + offset;
}
- if (current_vector == FIRST_SYSTEM_VECTOR)
- panic("ran out of interrupt sources!");
+ if (current_vector == FIRST_SYSTEM_VECTOR) {
+ printk(KERN_WARNING "ran out of interrupt vectors!\n");
+ current_vector = -ENOSPC;
+ goto out;
+ }
IO_APIC_VECTOR(irq) = current_vector;
+out:
return current_vector;
}
@@ -1164,6 +1171,9 @@ void __init setup_IO_APIC_irqs(void)
}
irq = pin_2_irq(idx, apic, pin);
+ if (irq > NR_IRQS)
+ continue;
+
/*
* skip adding the timer int on secondary nodes, which causes
* a small but painful rift in the time-space continuum
@@ -1178,8 +1188,10 @@ void __init setup_IO_APIC_irqs(void)
if (IO_APIC_IRQ(irq)) {
vector = assign_irq_vector(irq);
- entry.vector = vector;
+ if (vector < 0)
+ continue;
+ entry.vector = vector;
if (IO_APIC_irq_trigger(irq))
irq_desc[irq].handler = &ioapic_level_irq_type;
else
@@ -2274,6 +2286,10 @@ int io_apic_set_pci_routing (int ioapic,
{
struct IO_APIC_route_entry entry;
unsigned long flags;
+ int vector;
+
+ if (irq > NR_IRQS)
+ return -ENOSPC;
if (!IO_APIC_IRQ(irq)) {
printk(KERN_ERR "IOAPIC[%d]: Invalid reference to IRQ 0/n",
@@ -2298,8 +2314,11 @@ int io_apic_set_pci_routing (int ioapic,
add_pin_to_irq(irq, ioapic, pin);
- entry.vector = assign_irq_vector(irq);
+ vector = assign_irq_vector(irq);
+ if (vector < 0)
+ return -ENOSPC;
+ entry.vector = vector;
printk(KERN_DEBUG "IOAPIC[%d]: Set PCI routing entry (%d-%d -> 0x%x -> "
"IRQ %d)\n", ioapic,
mp_ioapics[ioapic].mpc_apicid, pin, entry.vector, irq);
^ permalink raw reply [flat|nested] 4+ messages in thread* Re: [PATCH][2.5] avoid scribbling in IDT with high interrupt count. 2003-04-07 6:03 [PATCH][2.5] avoid scribbling in IDT with high interrupt count Zwane Mwaikambo @ 2003-04-07 15:27 ` Linus Torvalds 2003-04-08 2:04 ` Zwane Mwaikambo 0 siblings, 1 reply; 4+ messages in thread From: Linus Torvalds @ 2003-04-07 15:27 UTC (permalink / raw) To: Zwane Mwaikambo; +Cc: Linux Kernel On Mon, 7 Apr 2003, Zwane Mwaikambo wrote: > > On systems with high interrupt counts we end up overshooting the > interrupt[] array and dumping garbage entry points into the idt. This > patch checks for out of bounds access during ioapic setup as well as > returning -ENOSPC when we run out of vectors to assign in > assign_irq_vector, thus allowing us to remove the panic and boot a 320 > interrupt source system with 2.5.66. Zwane - is there any reason we couldn't just start re-using irq vector offsets when this happens? We already re-use the vectors themselves, so restarting the offset pointer shouldn't really _change_ anything. In other words, I'm wondering if this simpler patch wouldn't be sufficient instead? Can you please test this, and re-submit (and if you can explain why your patch is better, please do so - I have nothing fundamentally against it, I just want to understand _why_ the complexity is needed). Thanks, Linus --- 1.58/arch/i386/kernel/io_apic.c Thu Mar 20 03:11:41 2003 +++ edited/arch/i386/kernel/io_apic.c Mon Apr 7 08:25:48 2003 @@ -1110,12 +1110,9 @@ goto next; if (current_vector > FIRST_SYSTEM_VECTOR) { - offset++; + offset = (offset+1) & 7; current_vector = FIRST_DEVICE_VECTOR + offset; } - - if (current_vector == FIRST_SYSTEM_VECTOR) - panic("ran out of interrupt sources!"); IO_APIC_VECTOR(irq) = current_vector; return current_vector; ^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: [PATCH][2.5] avoid scribbling in IDT with high interrupt count. 2003-04-07 15:27 ` Linus Torvalds @ 2003-04-08 2:04 ` Zwane Mwaikambo 2003-04-08 2:31 ` Martin J. Bligh 0 siblings, 1 reply; 4+ messages in thread From: Zwane Mwaikambo @ 2003-04-08 2:04 UTC (permalink / raw) To: Linus Torvalds; +Cc: Linux Kernel On Mon, 7 Apr 2003, Linus Torvalds wrote: > Zwane - is there any reason we couldn't just start re-using irq vector > offsets when this happens? We already re-use the vectors themselves, so > restarting the offset pointer shouldn't really _change_ anything. My patch skips irq numbers > NR_IRQS and simply return error when we run out of vectors, although mainline doesn't assign duplicates and cause collisions, it does waste vector space. e.g. for a system with NR_IRQS = 224 we have 23 vectors free, 0 collisions and 167 useable irqs and assign_irq_vectors states that we're out of vectors. > In other words, I'm wondering if this simpler patch wouldn't be sufficient > instead? > > Can you please test this, and re-submit (and if you can explain why your > patch is better, please do so - I have nothing fundamentally against it, I > just want to understand _why_ the complexity is needed). Your patch booted the system but there are vector collisions resulting in lost irq routing when we program the IOAPIC with the duplicated vector. So with your patch and NR_IRQS = 224 we have 1 vectors free (0x80), 34 collisions and 225 irqs. However that isn't a fault of your patch but a fault with the NR_IRQS definition. This is what the vector space looks like on i386 at present. 0________0x31__________________________0xef____________0xff system io interrupts resvd vectors 0xef - 0x31 = 190 useable io interrupt vectors So perhaps we should, apply your patch, add a NR_IRQ_VECTORS define and also add commentary in irq_vectors.h how does the following look? I had to readd the NR_IRQS checks to protect against overrunning NR_IRQS sized arrays and i added nr_assigned to track how many vectors were allocated so taht we can bail out when we're out. Patch has been tested on a 320 interrupt system and had a maximum useable irq line of 211 (ethernet). Thanks, Zwane Index: linux-2.5.67/include/asm-i386/mach-default/irq_vectors.h =================================================================== RCS file: /build/cvsroot/linux-2.5.67/include/asm-i386/mach-default/irq_vectors.h,v retrieving revision 1.1.1.1 diff -u -p -B -r1.1.1.1 irq_vectors.h --- linux-2.5.67/include/asm-i386/mach-default/irq_vectors.h 8 Apr 2003 01:15:29 -0000 1.1.1.1 +++ linux-2.5.67/include/asm-i386/mach-default/irq_vectors.h 8 Apr 2003 01:34:54 -0000 @@ -68,15 +68,22 @@ #define TIMER_IRQ 0 /* - * 16 8259A IRQ's, 208 potential APIC interrupt sources. + * 16 8259A IRQ's, MAX_IRQ_SOURCES-16 potential APIC interrupt sources. * Right now the APIC is mostly only used for SMP. * 256 vectors is an architectural limit. (we can have * more than 256 devices theoretically, but they will * have to use shared interrupts) * Since vectors 0x00-0x1f are used/reserved for the CPU, - * the usable vector space is 0x20-0xff (224 vectors) + * the usable vector space is 0x20-0xff (224 vectors). + * Linux currently makes 190 vectors available for io interrupts + * starting at FIRST_DEVICE_VECTOR till FIRST_SYSTEM_VECTOR + * + * 0________0x31__________________________0xef______0xff + * system io interrupts resvd + * */ #ifdef CONFIG_X86_IO_APIC +#define NR_IRQ_VECTORS 190 #define NR_IRQS 224 #else #define NR_IRQS 16 Index: linux-2.5.67/arch/i386/kernel/io_apic.c =================================================================== RCS file: /build/cvsroot/linux-2.5.67/arch/i386/kernel/io_apic.c,v retrieving revision 1.1.1.1 diff -u -p -B -r1.1.1.1 io_apic.c --- linux-2.5.67/arch/i386/kernel/io_apic.c 8 Apr 2003 01:15:35 -0000 1.1.1.1 +++ linux-2.5.67/arch/i386/kernel/io_apic.c 8 Apr 2003 01:36:06 -0000 @@ -1107,9 +1107,15 @@ int irq_vector[NR_IRQS] = { FIRST_DEVICE static int __init assign_irq_vector(int irq) { - static int current_vector = FIRST_DEVICE_VECTOR, offset = 0; + static int current_vector = FIRST_DEVICE_VECTOR, offset = 0, + nr_assigned = 1; + if (IO_APIC_VECTOR(irq) > 0) return IO_APIC_VECTOR(irq); + + if (++nr_assigned > NR_IRQ_VECTORS) + return -ENOSPC; + next: current_vector += 8; if (current_vector == SYSCALL_VECTOR) @@ -1167,6 +1173,8 @@ void __init setup_IO_APIC_irqs(void) } irq = pin_2_irq(idx, apic, pin); + if (irq >= NR_IRQS) + continue; /* * skip adding the timer int on secondary nodes, which causes * a small but painful rift in the time-space continuum @@ -1181,6 +1189,9 @@ void __init setup_IO_APIC_irqs(void) if (IO_APIC_IRQ(irq)) { vector = assign_irq_vector(irq); + if (vector < 0) + continue; + entry.vector = vector; if (IO_APIC_irq_trigger(irq)) @@ -2277,6 +2288,10 @@ int io_apic_set_pci_routing (int ioapic, { struct IO_APIC_route_entry entry; unsigned long flags; + int vector; + + if (irq >= NR_IRQS) + return -ENOSPC; if (!IO_APIC_IRQ(irq)) { printk(KERN_ERR "IOAPIC[%d]: Invalid reference to IRQ 0/n", @@ -2301,8 +2316,11 @@ int io_apic_set_pci_routing (int ioapic, add_pin_to_irq(irq, ioapic, pin); - entry.vector = assign_irq_vector(irq); + vector = assign_irq_vector(irq); + if (vector < 0) + return -ENOSPC; + entry.vector = vector; printk(KERN_DEBUG "IOAPIC[%d]: Set PCI routing entry (%d-%d -> 0x%x -> " "IRQ %d)\n", ioapic, mp_ioapics[ioapic].mpc_apicid, pin, entry.vector, irq); -- function.linuxpower.ca ^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: [PATCH][2.5] avoid scribbling in IDT with high interrupt count. 2003-04-08 2:04 ` Zwane Mwaikambo @ 2003-04-08 2:31 ` Martin J. Bligh 0 siblings, 0 replies; 4+ messages in thread From: Martin J. Bligh @ 2003-04-08 2:31 UTC (permalink / raw) To: Zwane Mwaikambo, Linus Torvalds; +Cc: Linux Kernel > My patch skips irq numbers > NR_IRQS and simply return error when we run > out of vectors, although mainline doesn't assign duplicates and cause > collisions, it does waste vector space. e.g. for a system with > NR_IRQS = 224 we have 23 vectors free, 0 collisions and 167 useable irqs > and assign_irq_vectors states that we're out of vectors. > >> In other words, I'm wondering if this simpler patch wouldn't be sufficient >> instead? >> >> Can you please test this, and re-submit (and if you can explain why your >> patch is better, please do so - I have nothing fundamentally against it, I >> just want to understand _why_ the complexity is needed). > > Your patch booted the system but there are vector collisions resulting in > lost irq routing when we program the IOAPIC with the duplicated vector. So > with your patch and NR_IRQS = 224 we have 1 vectors free (0x80), 34 collisions > and 225 irqs. However that isn't a fault of your patch but a fault with > the NR_IRQS definition. This is what the vector space looks like on i386 > at present. > > 0________0x31__________________________0xef____________0xff > system io interrupts resvd vectors > > 0xef - 0x31 = 190 useable io interrupt vectors > > So perhaps we should, apply your patch, add a NR_IRQ_VECTORS define > and also add commentary in irq_vectors.h how does the following look? I > had to readd the NR_IRQS checks to protect against overrunning NR_IRQS sized > arrays and i added nr_assigned to track how many vectors were allocated so > taht we can bail out when we're out. > > Patch has been tested on a 320 interrupt system and had a maximum useable > irq line of 211 (ethernet). We're still allocating interrupts to a bunch 'o stuff that isn't actually being used ... I'd prefer it if we could change that, and allocate them as they're actually used, rather than (what I believe is) still a fixed number per IO-APIC. Is there some smart way to do that? If not, the overflow protection is still (obviously) a good thing to do. M. ^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2003-04-08 2:23 UTC | newest] Thread overview: 4+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2003-04-07 6:03 [PATCH][2.5] avoid scribbling in IDT with high interrupt count Zwane Mwaikambo 2003-04-07 15:27 ` Linus Torvalds 2003-04-08 2:04 ` Zwane Mwaikambo 2003-04-08 2:31 ` Martin J. Bligh
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®