[PATCH] drop note_interrupt() for per-CPU for proper scaling

[PATCH] drop note_interrupt() for per-CPU for proper scaling

[John Hawkes]

The "unhandled interrupts" catcher, note_interrupt(), increments a global
desc->irq_count and grossly damages scaling of very large systems, e.g.,
>192p ia64 Altix, because of this highly contented cacheline, especially
for timer interrupts.  384p is severely crippled, and 512p is unuseable.

All calls to note_interrupt() can be disabled by booting with "noirqdebug",
but this disables the useful interrupt checking for all interrupts.

I propose eliminating note_interrupt() for all per-CPU interrupts.
This was the behavior of linux-2.6.10 and earlier, but in 2.6.11 a
code restructuring added a call to note_interrupt() for per-CPU interrupts.
Besides, note_interrupt() is a bit racy for concurrent CPU calls anyway,
as the desc->irq_count++ increment isn't atomic (which, if done, would make
scaling even worse).

Signed-off-by: John Hawkes <hawkes@sgi.com>
---

Index: linux/kernel/irq/handle.c
===================================================================
--- linux.orig/kernel/irq/handle.c	2005-03-01 23:38:37.000000000 -0800
+++ linux/kernel/irq/handle.c	2005-05-18 11:02:18.000000000 -0700
@@ -118,8 +118,6 @@
 		 */
 		desc->handler->ack(irq);
 		action_ret = handle_IRQ_event(irq, regs, desc->action);
-		if (!noirqdebug)
-			note_interrupt(irq, desc, action_ret);
 		desc->handler->end(irq);
 		return 1;
 	}

Re: [Lse-tech] [PATCH] drop note_interrupt() for per-CPU for proper scaling

[Paul E. McKenney]

On Wed, May 25, 2005 at 04:19:55PM -0700, John Hawkes wrote:
> The "unhandled interrupts" catcher, note_interrupt(), increments a global
> desc->irq_count and grossly damages scaling of very large systems, e.g.,
> >192p ia64 Altix, because of this highly contented cacheline, especially
> for timer interrupts.  384p is severely crippled, and 512p is unuseable.
> 
> All calls to note_interrupt() can be disabled by booting with "noirqdebug",
> but this disables the useful interrupt checking for all interrupts.
> 
> I propose eliminating note_interrupt() for all per-CPU interrupts.
> This was the behavior of linux-2.6.10 and earlier, but in 2.6.11 a
> code restructuring added a call to note_interrupt() for per-CPU interrupts.
> Besides, note_interrupt() is a bit racy for concurrent CPU calls anyway,
> as the desc->irq_count++ increment isn't atomic (which, if done, would make
> scaling even worse).

Would it be reasonable for note_interrupt() to operate on per-CPU or
per-group-of-CPUs counters?  Each CPU (or group) could either separately
count and check, or, if accuracy is important (can't see why it would
be), the per-CPU (or group) counters could be added to a global counter
every 100 (or 1,000 or whatever) counts, and this global counter then
checked against the limit.

							Thanx, Paul

> Signed-off-by: John Hawkes <hawkes@sgi.com>
> ---
> 
> Index: linux/kernel/irq/handle.c
> ===================================================================
> --- linux.orig/kernel/irq/handle.c	2005-03-01 23:38:37.000000000 -0800
> +++ linux/kernel/irq/handle.c	2005-05-18 11:02:18.000000000 -0700
> @@ -118,8 +118,6 @@
>  		 */
>  		desc->handler->ack(irq);
>  		action_ret = handle_IRQ_event(irq, regs, desc->action);
> -		if (!noirqdebug)
> -			note_interrupt(irq, desc, action_ret);
>  		desc->handler->end(irq);
>  		return 1;
>  	}
> 
> 
> -------------------------------------------------------
> SF.Net email is sponsored by: GoToMeeting - the easiest way to collaborate
> online with coworkers and clients while avoiding the high cost of travel and
> communications. There is no equipment to buy and you can meet as often as
> you want. Try it free.http://ads.osdn.com/?ad_id=7402&alloc_id=16135&op=click
> _______________________________________________
> Lse-tech mailing list
> Lse-tech@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/lse-tech
>