mirror of https://lore.kernel.org/lkml/
 help / color / mirror / Atom feed
* RE: [RFC][2.6.12.3] IRQ compression/sharing patch
@ 2005-08-11  0:21 Protasevich, Natalie
  2005-08-11  3:14 ` James Cleverdon
  0 siblings, 1 reply; 21+ messages in thread
From: Protasevich, Natalie @ 2005-08-11  0:21 UTC (permalink / raw)
  To: jamesclv; +Cc: Andi Kleen, Brown, Len, zwane, linux-kernel

> > int gsi_irq_sharing(int gsi)
> > {
> >         int i, irq, vector;
> >
> >         BUG_ON(gsi >= NR_IRQ_VECTORS);
> >
> >         if (platform_legacy_irq(gsi)) {
> >                 gsi_2_irq[gsi] = gsi;
> >                 return gsi;
> >         }
> >
> >         if (gsi_2_irq[gsi] != 0xFF)
> >                 return (int)gsi_2_irq[gsi];
> >
> >         vector = assign_irq_vector(gsi); // this part here==========
> 
> I thought I had this case covered earlier, given that in both i386 and
> x86_64:
> 
> #define platform_legacy_irq(irq)      ((irq) < 16)

Yes, you are absolutely correct, I don't need the (gsi<16) part, this
takes care of PCI IRQs that happened to be <16.

> In the deleted vector sharing code, I also check 
> platform_legacy_irq, to avoid inadvertently sharing vectors 
> already assigned to legacy IRQs.
> 
> Am I missing your point here?
> 
> >         if (gsi < 16) {
> >                 irq = gsi;
> >                 gsi_2_irq[gsi] = irq;
> >         } else {
> >                 irq = next_irq++;
> >                 gsi_2_irq[gsi] = irq;
> >         }
> > //====================
> 
> I can't explain the gaps in the numbers with the original 
> version.  I'll give your variant a try.

The only problem is here:

+	if (i < NR_IRQS) {
+		gsi_2_irq[gsi] = i;
+		printk(KERN_INFO "GSI %d sharing vector 0x%02X and IRQ
%d\n",
+				gsi, vector, i);
+		return i;
+	}
+
+	i = next_irq++;
 
That means for any IRQ < NR_IRQS you allow it to be identity mapped,
with all the gaps, and only for ones exceeding 224 you'll assign
consecutive next_irqs++, whereas you can do it for all PCI IRQs above
15. So, the alternative clause should probably come down to just:
                 irq = next_irq++;
                 gsi_2_irq[gsi] = irq; - which means just removing the
one above...
(although we better test that :)...I will definitely test vector sharing
when manage to get on max configuration partition here.

Regards,
--Natalie
 

^ permalink raw reply	[flat|nested] 21+ messages in thread
* RE: [RFC][2.6.12.3] IRQ compression/sharing patch
@ 2005-08-15  4:35 Protasevich, Natalie
  2005-08-15 17:11 ` James Cleverdon
  0 siblings, 1 reply; 21+ messages in thread
From: Protasevich, Natalie @ 2005-08-15  4:35 UTC (permalink / raw)
  To: jamesclv, Andi Kleen; +Cc: Russ Weight, linux-kernel

> On Thursday 04 August 2005 02:22 am, Andi Kleen wrote:
> > On Thu, Aug 04, 2005 at 12:05:50AM -0700, James Cleverdon wrote:
> > > diff -pruN 2.6.12.3/arch/i386/kernel/acpi/boot.c
> > > n12.3/arch/i386/kernel/acpi/boot.c ---
> > > 2.6.12.3/arch/i386/kernel/acpi/boot.c	2005-07-15 
> 14:18:57.000000000
> > > -0700 +++ n12.3/arch/i386/kernel/acpi/boot.c	2005-08-04
> > > 00:01:10.199710211 -0700 @@ -42,6 +42,7 @@  static inline void  
> > > acpi_madt_oem_check(char *oem_id, char
> > > *oem_table_id) { } extern void __init 
> clustered_apic_check(void);  
> > > static inline int ioapic_setup_disabled(void) { return 0; }
> > > +extern int gsi_irq_sharing(int gsi);
> > >  #include <asm/proto.h>
> > >
> > >  #else	/* X86 */
> > > @@ -51,6 +52,9 @@ static inline int 
> ioapic_setup_disabled(  #include 
> > > <mach_mpparse.h>
> > >  #endif	/* CONFIG_X86_LOCAL_APIC */
> > >
> > > +static inline int gsi_irq_sharing(int gsi) { return gsi; }
> >
> > Why is this different for i386/x86-64? It shouldn't.
> 
> True.  Have added code for i386.  Unfortunately, I didn't see 
> one file that is shared by both architectures and which is 
> included when building with I/O APIC support.  So, I 
> duplicated the function into io_apic.c
> 
> > As a unrelated note we really need to get rid of this whole ifdef 
> > block.
> >
> > > +++ n12.3/arch/x86_64/Kconfig	2005-08-03 
> 21:31:07.487451167 -0700
> > > @@ -280,13 +280,13 @@ config HAVE_DEC_LOCK
> > >  	default y
> > >
> > >  config NR_CPUS
> > > -	int "Maximum number of CPUs (2-256)"
> > > -	range 2 256
> > > +	int "Maximum number of CPUs (2-255)"
> > > +	range 2 255
> > >  	depends on SMP
> > > -	default "8"
> > > +	default "16"
> >
> > Don't change the default please.
> >
> > > +static int next_irq = 16;
> >
> > Won't this need a lock for hotplug later?
> 
> That's what I thought originally, but maybe not.  We 
> initialize all RTEs and assign IRQs+vectors fairly early in 
> boot, plus store the results in arrays.  Thereafter the 
> functions just return the preallocated values.
> 
> Hmmm...  Since the I/O APIC init comes after the other CPUs 
> are brought online, and since I don't understand all that the 
> MSI driver is trying to accomplish, it might be safer to use 
> a spin lock anyway.
> 
> > > +
> > > + retry_vector:
> > > +	vector = assign_irq_vector(gsi);
> > > +
> > > +	/*
> > > +	 * Sharing vectors means sharing IRQs, so scan irq_vectors for
> > > previous +	 * use of vector and if found, return that IRQ. 
> > > However, we never want +	 * to share legacy IRQs, which usually
> > > have a different trigger mode +	 * than PCI.
> > > +	 */
> >
> > Can we perhaps force such sharing early temporarily even when the 
> > table is not filled up?  This way we would get better test 
> coverage of 
> > all of  this.
> >
> > That would be later disabled of course.
> 
> Suppose I added a static counter and pretended that every 
> third non-legacy IRQ needed to be shared?
> 
> > Rest looks ok to me.
> >
> > -Andi
> 
> Sigh.  Have to attach the file again.  Sorry about that.
> 
> Signed-off-by:  James Cleverdon <jamesclv@us.ibm.com>

I think you were going to change this line, which fixed the jumps in the
irq distribution:

--- io_apic.c	2005-08-11 10:14:33.564748923 -0700
+++ io_apic.c.new	2005-08-11 10:15:55.412331115 -0700
@@ -617,7 +617,7 @@ int gsi_irq_sharing(int gsi)
  	 * than PCI.
  	 */
  	for (i = 0; i < NR_IRQS; i++)
 -		if (IO_APIC_VECTOR(i) == vector) {
 +		if (IO_APIC_VECTOR(i) == vector && i != gsi) {
  			if (!platform_legacy_irq(i))
  				break;			/* got one */
  			IO_APIC_VECTOR(gsi) = 0;
But it's not in this version of the patch.
Thanks,
--Natalie 
> --
> James Cleverdon
> IBM LTC (xSeries Linux Solutions)
> {jamesclv(Unix, preferred), cleverdj(Notes)} at us dot ibm dot comm
> 
> 

^ permalink raw reply	[flat|nested] 21+ messages in thread
* RE: [RFC][2.6.12.3] IRQ compression/sharing patch
@ 2005-08-11 22:02 Protasevich, Natalie
  2005-08-11 22:34 ` Zwane Mwaikambo
  0 siblings, 1 reply; 21+ messages in thread
From: Protasevich, Natalie @ 2005-08-11 22:02 UTC (permalink / raw)
  To: Zwane Mwaikambo; +Cc: Andi Kleen, Brown, Len, linux-kernel

> On Wed, 10 Aug 2005, Protasevich, Natalie wrote:
> 
> > our systems we are just about to use up all 224 interrupts, but not 
> > quiet.
> > I have to mention that as far as I know Zwane is about to 
> release his 
> > vector sharing mechanism, he had it implemented and working 
> for i386 
> > (I tested it on ES7000 successfully, by itself and combined with 
> > compression patch too), and was planning implementing it 
> for x86_64. I 
> > am officially volunteering for testing it in its present state, for 
> > both
> > i386 and x86_64 (I can still do this on our systems by removing the 
> > IRQ compression code :), hope this will help Zwane and Andi 
> to release 
> > it as soon as possible.
> 
> I added some of the suggestions brought forward (dynamically 
> allocated IDTs, percpu IDT) last night, all that's left is 
> MSI, which does work right now, but gets all its vectors 
> allocated on the first irq handling domain. I should be done 
> soon, time permitting.

Zwane, please let me know when I can try it on ES7000, even work in
progress if you need it (see above about volunteering :)

Regards,
--Natalie 
 

^ permalink raw reply	[flat|nested] 21+ messages in thread
* RE: [RFC][2.6.12.3] IRQ compression/sharing patch
@ 2005-08-11 21:55 Protasevich, Natalie
  2005-08-12  1:07 ` James Cleverdon
  2005-08-12  2:59 ` James Cleverdon
  0 siblings, 2 replies; 21+ messages in thread
From: Protasevich, Natalie @ 2005-08-11 21:55 UTC (permalink / raw)
  To: jamesclv; +Cc: Andi Kleen, Russ Weight, linux-kernel

> After sleeping on it, maybe the original code can be patched 
> without having to hack assign_irq_vector(), etc.  How about:
> 
> --- io_apic.c	2005-08-11 10:14:33.564748923 -0700
> +++ io_apic.c.new	2005-08-11 10:15:55.412331115 -0700
> @@ -617,7 +617,7 @@ int gsi_irq_sharing(int gsi)
>  	 * than PCI.
>  	 */
>  	for (i = 0; i < NR_IRQS; i++)
> -		if (IO_APIC_VECTOR(i) == vector) {
> +		if (IO_APIC_VECTOR(i) == vector && i != gsi) {
>  			if (!platform_legacy_irq(i))
>  				break;			/* got one */
>  			IO_APIC_VECTOR(gsi) = 0;
> 
> 
Yes that did it, on my small system it looked just right:

<7>IRQ to pin mappings:
<7>IRQ0 -> 0:2
<7>IRQ1 -> 0:1
<7>IRQ3 -> 0:3
<7>IRQ4 -> 0:4
<7>IRQ5 -> 0:5
<7>IRQ6 -> 0:6
<7>IRQ7 -> 0:7
<7>IRQ8 -> 0:8
<7>IRQ9 -> 0:9
<7>IRQ10 -> 0:10
<7>IRQ11 -> 0:11
<7>IRQ12 -> 0:12
<7>IRQ14 -> 0:14
<7>IRQ15 -> 0:15
<7>IRQ16 -> 0:16
<7>IRQ17 -> 0:17
<7>IRQ18 -> 0:18
<7>IRQ19 -> 0:19
<7>IRQ20 -> 0:20
<7>IRQ21 -> 0:23
<7>IRQ22 -> 1:2
<7>IRQ23 -> 1:3
<7>IRQ24 -> 1:4
<7>IRQ25 -> 1:5
<7>IRQ26 -> 2:0
<7>IRQ27 -> 2:1
<7>IRQ28 -> 2:2
<7>IRQ29 -> 2:3
<7>IRQ30 -> 2:4
<7>IRQ31 -> 2:5
<7>IRQ32 -> 2:6
<7>IRQ33 -> 2:7
<7>IRQ34 -> 2:8
:!cat /proc/interrupts
           CPU0       CPU1       CPU2       CPU3
  0:      12621      15007      12781      20921    IO-APIC-edge  timer
  1:         72          0          2        175    IO-APIC-edge  i8042
  2:          0          0          0          0          XT-PIC
cascade
  8:          0          0          0          1    IO-APIC-edge  rtc
  9:          0          0          0          0    IO-APIC-edge  acpi
 12:          4        272          0        110    IO-APIC-edge  i8042
 15:          4          0          0         39    IO-APIC-edge  ide1
 16:          0          0          0          0   IO-APIC-level
uhci_hcd:usb1, uhci_hcd:usb4
 17:          0          0          0          2   IO-APIC-level
ohci1394
 18:        730       2407        932       2083   IO-APIC-level
libata, uhci_hcd:usb3
 19:          0          0          0          0   IO-APIC-level
uhci_hcd:usb2
 21:          0          0          0          0   IO-APIC-level
ehci_hcd:usb5
 26:        416          0          0          4   IO-APIC-level  eth0
NMI:        116         71         73         51
LOC:      61280      61258      61236      61214
ERR:          3
MIS:          0

Looks good! I will try the patch also on the ES7000 hopefully big enough
to exercise some vector sharing.

Regards,
--Natalie

^ permalink raw reply	[flat|nested] 21+ messages in thread
* RE: [RFC][2.6.12.3] IRQ compression/sharing patch
@ 2005-08-11 13:15 Protasevich, Natalie
  2005-08-11 17:24 ` James Cleverdon
  0 siblings, 1 reply; 21+ messages in thread
From: Protasevich, Natalie @ 2005-08-11 13:15 UTC (permalink / raw)
  To: jamesclv; +Cc: Andi Kleen, Russ Weight, linux-kernel

> > The only problem is here:
> >
> > +	if (i < NR_IRQS) {
> > +		gsi_2_irq[gsi] = i;
> > +		printk(KERN_INFO "GSI %d sharing vector 0x%02X and IRQ
> > %d\n",
> > +				gsi, vector, i);
> > +		return i;
> > +	}
> > +
> > +	i = next_irq++;
> >
> > That means for any IRQ < NR_IRQS you allow it to be 
> identity mapped, 
> > with all the gaps, and only for ones exceeding 224 you'll assign 
> > consecutive next_irqs++, whereas you can do it for all PCI 
> IRQs above 
> > 15. So, the alternative clause should probably come down to just:
> >                  irq = next_irq++;
> >                  gsi_2_irq[gsi] = irq; - which means just 
> removing the 
> > one above...
> > (although we better test that :)...I will definitely test vector 
> > sharing when manage to get on max configuration partition here.
> >
> > Regards,
> > --Natalie
> 
> Blast.  You're right, because i is the results of scanning 
> irq_vector (AKA IO_APIC_VECTOR()) for a previous use of the 
> vector.  If i < NR_IRQS then it has found a previous use and 
> would substitute the other IRQ number rather than allocate a new one.
> 
> Of course, one side effect of calling assign_irq_vector(gsi) 
> is that the vector number is stored at position gsi in 
> irq_vector.  So, for IRQs 0 to NR_IRQS - 1, it will always 
> find itself.
> 
> OK, how about not fighting that side effect and only 
> reassigning IRQs that are >= NR_IRQS?  Maybe something like this:

I am wondering where sharing_vectors gets set, it was not in the patch
originally.

> @@ -579,4 +589,57 @@ static inline int irq_trigger(int idx)
>  	return MPBIOS_trigger(idx);
>  }
>  
> +/*
> + * gsi_irq_sharing -- Name overload!  "irq" can be either a 
> legacy IRQ
> + * in the range 0-15, a linux IRQ in the range 0-223, or a GSI number
> + * from ACPI, which can easily reach 800.
> + *
> + * Compact the sparse GSI space into a available IRQs and reuse
> + * vectors if possible.
> + */
> +int gsi_irq_sharing(int gsi)
> +{
> +	int i, tries, vector;
> +
> +	BUG_ON(gsi >= NR_IRQ_VECTORS);
> +
> +	/*
> +	 * Don't share legacy IRQs.  Their trigger modes are 
> usually edge
> +	 * and PCI is level.  Mixed modes are trouble.  Only 
> big boxes are
> +	 * likely to overflow IRQs or to share vectors.
> +	 */
> +	if (likely(gsi < NR_IRQS && !sharing_vectors) || 
> platform_legacy_irq(gsi)) {
> +		gsi_2_irq[gsi] = gsi;
> +		return gsi;
> +	}
> +
> +	if (gsi_2_irq[gsi] != 0xFF)
> +		return gsi_2_irq[gsi];
> +	/*
> +	 * Ran out of vectors or IRQ >= NR_IRQS.  Sharing vectors
> +	 * means sharing IRQs, so scan irq_vectors for previous use
> +	 * of vector and return that IRQ.
> +	 */
> +	tries = NR_IRQS;
> +  try_again:
> +	vector = assign_irq_vector(gsi);
> +
> +	/* Find the first IRQ using vector. */
> +	for (i = 0; i < NR_IRQS; i++)
> +		if (IO_APIC_VECTOR(i) == vector)
> +			break;
> +
> +	if (i >= NR_IRQS || platform_legacy_irq(i)) {
> +		if (--tries >= 0) {
> +			IO_APIC_VECTOR(gsi) = 0;
> +			goto try_again;
> +		}
> +		panic("gsi_irq_sharing: didn't find an IRQ 
> using vector 0x%02X for GSI %d", vector, gsi);
> +	}
> +	printk("GSI %d assigned vector 0x%02X and IRQ %d\n", 
> gsi, vector, i);
> +	gsi_2_irq[gsi] = i;
> +	return i;
> +}
> +
>  static int pin_2_irq(int idx, int apic, int pin)  {
>  	int irq, i;
> 
> 
> --
> James Cleverdon
> IBM LTC (xSeries Linux Solutions)
> {jamesclv(Unix, preferred), cleverdj(Notes)} at us dot ibm dot comm
> 

^ permalink raw reply	[flat|nested] 21+ messages in thread
* RE: [RFC][2.6.12.3] IRQ compression/sharing patch
@ 2005-08-10 21:03 Protasevich, Natalie
  2005-08-10 23:55 ` James Cleverdon
  2005-08-11 17:52 ` Zwane Mwaikambo
  0 siblings, 2 replies; 21+ messages in thread
From: Protasevich, Natalie @ 2005-08-10 21:03 UTC (permalink / raw)
  To: jamesclv, Andi Kleen; +Cc: Brown, Len, zwane, linux-kernel

> Due to some device driver issues, I built this iteration of 
> the patch vs. 2.6.12.3.
> 
> (Sorry about the attachment, but KMail is still word wrapping inserted
> files.)
> 
> Background:
> 
> Here's a patch that builds on Natalie Protasevich's IRQ 
> compression patch and tries to work for MPS boots as well as 
> ACPI.  It is meant for a 4-node IBM x460 NUMA box, which was 
> dying because it had interrupt pins with GSI numbers > 
> NR_IRQS and thus overflowed irq_desc.
> 
> The problem is that this system has 280 GSIs (which are 1:1 
> mapped with I/O APIC RTEs) and an 8-node box would have 560.  
> This is much bigger than NR_IRQS (224 for both i386 and 
> x86_64).  Also, there aren't enough vectors to go around.  
> There are about 190 usable vectors, not counting the reserved 
> ones and the unused vectors at 0x20 to 0x2F.  So, my patch 
> attempts to compress the GSI range and share vectors by sharing IRQs.
> 
Hi James, 
I tested your patch today (sorry it took a while, was out of town), and
in general it worked just fine. It was a small system with 3 IO-APICs,
will hopefully try it on a large partition with 64 of them tonight.
One thing I noticed: I think the patch is going for shared vectors way
before exhausting available NR_IRQS, so I suggest a small modification
to it, in gsi_irq_sharing():
int gsi_irq_sharing(int gsi)
{
        int i, irq, vector;

        BUG_ON(gsi >= NR_IRQ_VECTORS);

        if (platform_legacy_irq(gsi)) {
                gsi_2_irq[gsi] = gsi;
                return gsi;
        }

        if (gsi_2_irq[gsi] != 0xFF)
                return (int)gsi_2_irq[gsi];

        vector = assign_irq_vector(gsi);
// this part here==========
        if (gsi < 16) {
                irq = gsi;
                gsi_2_irq[gsi] = irq;
        } else {
                irq = next_irq++;
                gsi_2_irq[gsi] = irq;
        }
//====================
        IO_APIC_VECTOR(irq) = vector;
        printk(KERN_INFO "GSI %d assigned vector 0x%02X and IRQ %d\n",
                        gsi, vector, irq);

        return irq;
}

(I took out the vector sharing part for clarity, just to concentrate on
compression, and I didn't do any boundary checks). The (gsi<16) takes
care of the recent problem with my ACPI IRQ compression patch breaking
VIA chipset that doesn't tolerate PCI IRQ numbers above 15.

I think this way we are saving more IRQs and place them denser.
Here is back-to-back comparison of IRQ distribution with the original
and modified patch:

Original:
           CPU0       CPU1       CPU2       CPU3
  0:      18758      20011      20008      28294    IO-APIC-edge  timer
  1:         97         18         79         16    IO-APIC-edge  i8042
  2:          0          0          0          0          XT-PIC
cascade
  8:          1          0          0          1    IO-APIC-edge  rtc
  9:          0          0          0          0    IO-APIC-edge  acpi
 12:          0        708          0        110    IO-APIC-edge  i8042
 15:          4          0          0         39    IO-APIC-edge  ide1
 16:          0          0          0          0   IO-APIC-level
uhci_hcd:usb1, uhci_hcd:usb4
 17:          0          0          0          3   IO-APIC-level
ohci1394
 18:        670       2253        836       1981   IO-APIC-level
libata, uhci_hcd:usb3
 19:          0          0          0          0   IO-APIC-level
uhci_hcd:usb2
 23:          0          0          0          0   IO-APIC-level
ehci_hcd:usb5
 48:        212          0          0          4   IO-APIC-level  eth0
<== gap on the 3nd io-apic
NMI:        117         71         73         51
LOC:      87020      86997      86975      86952
ERR:          3
MIS:          0

<7>IRQ to pin mappings:
<7>IRQ0 -> 0:2
<7>IRQ1 -> 0:1
<7>IRQ3 -> 0:3
<7>IRQ4 -> 0:4
<7>IRQ5 -> 0:5
<7>IRQ6 -> 0:6
<7>IRQ7 -> 0:7
<7>IRQ8 -> 0:8
<7>IRQ9 -> 0:9
<7>IRQ10 -> 0:10
<7>IRQ11 -> 0:11
<7>IRQ12 -> 0:12
<7>IRQ14 -> 0:14
<7>IRQ15 -> 0:15
<7>IRQ16 -> 0:16
<7>IRQ17 -> 0:17
<7>IRQ18 -> 0:18
<7>IRQ19 -> 0:19
<7>IRQ20 -> 0:20
<7>IRQ23 -> 0:23
<7>IRQ26 -> 1:2 <=======jump on the 2nd io-apic
<7>IRQ27 -> 1:3
<7>IRQ28 -> 1:4
<7>IRQ29 -> 1:5
<7>IRQ48 -> 2:0 <=======jump on the 3rd io-apic
<7>IRQ49 -> 2:1
<7>IRQ50 -> 2:2
<7>IRQ51 -> 2:3
<7>IRQ52 -> 2:4
<7>IRQ53 -> 2:5
<7>IRQ54 -> 2:6
<7>IRQ55 -> 2:7
<7>IRQ56 -> 2:8

Modified:
           CPU0       CPU1       CPU2       CPU3
  0:      15125      17509      17507      25592    IO-APIC-edge  timer
  1:        187         66        280        140    IO-APIC-edge  i8042
  2:          0          0          0          0          XT-PIC
cascade
  8:          1          0          0          1    IO-APIC-edge  rtc
  9:          0          0          0          0    IO-APIC-edge  acpi
 12:          0          0          0        110    IO-APIC-edge  i8042
 15:          4          0          0         39    IO-APIC-edge  ide1
 16:          0          0          0          0   IO-APIC-level
uhci_hcd:usb1, uhci_hcd:usb4
 17:          0          0          0          2   IO-APIC-level
ohci1394
 18:        753       2070        925       2035   IO-APIC-level
libata, uhci_hcd:usb3
 19:          0          0          0          0   IO-APIC-level
uhci_hcd:usb2
 21:          0          0          0          0   IO-APIC-level
ehci_hcd:usb5
 26:        164          0          0          4   IO-APIC-level  eth0
NMI:        117         72         73         52
LOC:      75682      75659      75638      75615
ERR:          3
MIS:          0

<7>IRQ to pin mappings:
<7>IRQ0 -> 0:2
<7>IRQ1 -> 0:1
<7>IRQ3 -> 0:3
<7>IRQ4 -> 0:4
<7>IRQ5 -> 0:5
<7>IRQ6 -> 0:6
<7>IRQ7 -> 0:7
<7>IRQ8 -> 0:8
<7>IRQ9 -> 0:9
<7>IRQ10 -> 0:10
<7>IRQ11 -> 0:11
<7>IRQ12 -> 0:12
<7>IRQ14 -> 0:14
<7>IRQ15 -> 0:15
<7>IRQ16 -> 0:16
<7>IRQ17 -> 0:17
<7>IRQ18 -> 0:18
<7>IRQ19 -> 0:19
<7>IRQ20 -> 0:20
<7>IRQ21 -> 0:23
<7>IRQ22 -> 1:2
<7>IRQ23 -> 1:3
<7>IRQ24 -> 1:4
<7>IRQ25 -> 1:5
<7>IRQ26 -> 2:0
<7>IRQ27 -> 2:1
<7>IRQ28 -> 2:2
<7>IRQ29 -> 2:3
<7>IRQ30 -> 2:4
<7>IRQ31 -> 2:5
<7>IRQ32 -> 2:6
<7>IRQ33 -> 2:7
<7>IRQ34 -> 2:8

Unfortunately, I cannot test the vector sharing part properly, since on
our systems we are just about to use up all 224 interrupts, but not
quiet. 
I have to mention that as far as I know Zwane is about to release his
vector sharing mechanism, he had it implemented and working for i386 (I
tested it on ES7000 successfully, by itself and combined with
compression patch too), and was planning implementing it for x86_64. I
am officially volunteering for testing it in its present state, for both
i386 and x86_64 (I can still do this on our systems by removing the IRQ
compression code :), hope this will help Zwane and Andi to release it as
soon as possible.

Regards,
--Natalie

^ permalink raw reply	[flat|nested] 21+ messages in thread
* [RFC][2.6.13-rc3-mm1] IRQ compression/sharing patch
@ 2005-07-26  7:12 James Cleverdon
  2005-07-26 16:03 ` Andi Kleen
  0 siblings, 1 reply; 21+ messages in thread
From: James Cleverdon @ 2005-07-26  7:12 UTC (permalink / raw)
  To: linux-kernel; +Cc: Andi Kleen, Protasevich, Natalie

[-- Attachment #1: Type: text/plain, Size: 1129 bytes --]

Here's a patch that builds on Natalie Protasevich's IRQ compression 
patch and tries to work for MPS boots as well as ACPI.  It is meant for 
a 4-node IBM x460 NUMA box, which was dying because it had interrupt 
pins with GSI numbers > NR_IRQS and thus overflowed irq_desc.

The problem is that this system has 270 GSIs (which are 1:1 mapped with 
I/O APIC RTEs) and an 8-node box would have 540.  This is much bigger 
than NR_IRQS (224 for both i386 and x86_64).  Also, there aren't enough 
vectors to go around.  There are about 190 usable vectors, not counting 
the reserved ones and the unused vectors at 0x20 to 0x2F.  So, my patch 
attempts to compress the GSI range and share vectors by sharing IRQs.

Important safety note:  While the SLES 9 version of this patch works, I 
haven't been able to test the -rc3-mm1 patch enclosed.  I keep getting 
errors from the adp94xx driver.  8-(

(Sorry about doing an attachment, but KMail is steadfastly word wrapping 
inserted files.  I need to upgrade....)

-- 
James Cleverdon
IBM LTC (xSeries Linux Solutions)
{jamesclv(Unix, preferred), cleverdj(Notes)} at us dot ibm dot comm

[-- Attachment #2: vect_share_irq_2005-07-23_2.6.13-rc3-mm1 --]
[-- Type: text/x-diff, Size: 10949 bytes --]

diff -pru 2.6.13-rc3-mm1/arch/i386/kernel/acpi/boot.c j13-rc3-mm1/arch/i386/kernel/acpi/boot.c
--- 2.6.13-rc3-mm1/arch/i386/kernel/acpi/boot.c	2005-07-22 18:32:35.000000000 -0700
+++ j13-rc3-mm1/arch/i386/kernel/acpi/boot.c	2005-07-23 18:32:06.000000000 -0700
@@ -40,9 +40,10 @@
 
 #ifdef	CONFIG_X86_64
 
-static inline void  acpi_madt_oem_check(char *oem_id, char *oem_table_id) { }
+extern void  acpi_madt_oem_check(char *oem_id, char *oem_table_id);
 extern void __init clustered_apic_check(void);
 static inline int ioapic_setup_disabled(void) { return 0; }
+extern int gsi_irq_sharing(int gsi);
 #include <asm/proto.h>
 
 #else	/* X86 */
@@ -52,6 +53,10 @@ static inline int ioapic_setup_disabled(
 #include <mach_mpparse.h>
 #endif	/* CONFIG_X86_LOCAL_APIC */
 
+static inline int ioapic_setup_disabled(void) { return 0; }
+static inline int gsi_irq_sharing(int gsi) { return gsi; }
+
+
 #endif	/* X86 */
 
 #define BAD_MADT_ENTRY(entry, end) (					    \
@@ -480,7 +485,7 @@ int acpi_gsi_to_irq(u32 gsi, unsigned in
  		*irq = IO_APIC_VECTOR(gsi);
 	else
 #endif
-		*irq = gsi;
+		*irq = gsi_irq_sharing(gsi);
 	return 0;
 }
 
diff -pru 2.6.13-rc3-mm1/arch/x86_64/Kconfig j13-rc3-mm1/arch/x86_64/Kconfig
--- 2.6.13-rc3-mm1/arch/x86_64/Kconfig	2005-07-22 18:32:35.000000000 -0700
+++ j13-rc3-mm1/arch/x86_64/Kconfig	2005-07-22 18:34:08.000000000 -0700
@@ -276,13 +276,13 @@ config HAVE_DEC_LOCK
 	default y
 
 config NR_CPUS
-	int "Maximum number of CPUs (2-256)"
-	range 2 256
+	int "Maximum number of CPUs (2-255)"
+	range 2 255
 	depends on SMP
-	default "8"
+	default "64"
 	help
 	  This allows you to specify the maximum number of CPUs which this
-	  kernel will support. Current maximum is 256 CPUs due to
+	  kernel will support. Current maximum is 255 CPUs due to
 	  APIC addressing limits. Less depending on the hardware.
 
 	  This is purely to save memory - each supported CPU requires
diff -pru 2.6.13-rc3-mm1/arch/x86_64/kernel/genapic.c j13-rc3-mm1/arch/x86_64/kernel/genapic.c
--- 2.6.13-rc3-mm1/arch/x86_64/kernel/genapic.c	2005-07-12 21:46:46.000000000 -0700
+++ j13-rc3-mm1/arch/x86_64/kernel/genapic.c	2005-07-22 18:34:08.000000000 -0700
@@ -103,3 +103,19 @@ void send_IPI_self(int vector)
 {
 	__send_IPI_shortcut(APIC_DEST_SELF, vector, APIC_DEST_PHYSICAL);
 }
+
+/*
+ * Check the APIC IDs in MADT table header and choose the APIC mode.
+ */
+void acpi_madt_oem_check(char *oem_id, char *oem_table_id)
+{
+	/* May need to check OEM strings in the future. */
+}
+
+/*
+ * Check the IDs in MPS header and choose the APIC mode.
+ */
+void mps_oem_check(struct mp_config_table *mpc, char *oem, char *productid)
+{
+	/* May need to check OEM strings in the future. */
+}
diff -pru 2.6.13-rc3-mm1/arch/x86_64/kernel/io_apic.c j13-rc3-mm1/arch/x86_64/kernel/io_apic.c
--- 2.6.13-rc3-mm1/arch/x86_64/kernel/io_apic.c	2005-07-22 18:32:35.000000000 -0700
+++ j13-rc3-mm1/arch/x86_64/kernel/io_apic.c	2005-07-23 18:32:40.000000000 -0700
@@ -56,7 +56,7 @@ int nr_ioapic_registers[MAX_IO_APICS];
  * Rough estimation of how many shared IRQs there are, can
  * be changed anytime.
  */
-#define MAX_PLUS_SHARED_IRQS NR_IRQS
+#define MAX_PLUS_SHARED_IRQS NR_IRQ_VECTORS
 #define PIN_MAP_SIZE (MAX_PLUS_SHARED_IRQS + NR_IRQS)
 
 /*
@@ -84,6 +84,7 @@ int vector_irq[NR_VECTORS] = { [0 ... NR
 	int pin;							\
 	struct irq_pin_list *entry = irq_2_pin + irq;			\
 									\
+	BUG_ON(irq >= NR_IRQS);						\
 	for (;;) {							\
 		unsigned int reg;					\
 		pin = entry->pin;					\
@@ -126,6 +127,8 @@ static void set_ioapic_affinity_irq(unsi
 }
 #endif
 
+static u8 gsi_2_irq[NR_IRQ_VECTORS] = { [0 ... NR_IRQ_VECTORS-1] = 0xFF };
+
 /*
  * The common case is 1:1 IRQ<->pin mappings. Sometimes there are
  * shared ISA-space IRQs, so we have to support them. We are super
@@ -136,6 +139,7 @@ static void add_pin_to_irq(unsigned int 
 	static int first_free_entry = NR_IRQS;
 	struct irq_pin_list *entry = irq_2_pin + irq;
 
+	BUG_ON(irq >= NR_IRQS);
 	while (entry->next)
 		entry = irq_2_pin + entry->next;
 
@@ -143,7 +147,7 @@ static void add_pin_to_irq(unsigned int 
 		entry->next = first_free_entry;
 		entry = irq_2_pin + entry->next;
 		if (++first_free_entry >= PIN_MAP_SIZE)
-			panic("io_apic.c: whoops");
+			panic("io_apic.c: ran out of irq_2_pin entries!");
 	}
 	entry->apic = apic;
 	entry->pin = pin;
@@ -419,6 +423,7 @@ int IO_APIC_get_PCI_irq_vector(int bus, 
 				best_guess = irq;
 		}
 	}
+	BUG_ON(best_guess >= NR_IRQS);
 	return best_guess;
 }
 
@@ -609,6 +614,64 @@ static inline int irq_trigger(int idx)
 	return MPBIOS_trigger(idx);
 }
 
+static int next_irq = 16;
+
+/*
+ * gsi_irq_sharing -- Name overload!  "irq" can be either a legacy IRQ
+ * in the range 0-15, a linux IRQ in the range 0-223, or a GSI number
+ * from ACPI, which can reach 800 in large boxen.
+ *
+ * Compact the sparse GSI space into a sequential IRQ series and reuse
+ * vectors if possible.
+ */
+int gsi_irq_sharing(int gsi)
+{
+	int i, tries, vector;
+
+	BUG_ON(gsi >= NR_IRQ_VECTORS);
+
+	if (platform_legacy_irq(gsi))
+		return gsi;
+
+	if (gsi_2_irq[gsi] != 0xFF)
+		return (int)gsi_2_irq[gsi];
+
+	tries = NR_IRQS;
+  try_again:
+	vector = assign_irq_vector(gsi);
+
+	/*
+	 * Sharing vectors means sharing IRQs, so scan irq_vectors for previous
+	 * use of vector and if found, return that IRQ.  However, we never want
+	 * to share legacy IRQs, which usually have a different trigger mode
+	 * than PCI.
+	 */
+	for (i = 0; i < NR_IRQS; i++)
+		if (IO_APIC_VECTOR(i) == vector)
+			break;
+	if (platform_legacy_irq(i)) {
+		if (--tries >= 0) {
+			IO_APIC_VECTOR(i) = 0;
+			goto try_again;
+		}
+		panic("gsi_irq_sharing: didn't find an IRQ using vector 0x%02X for GSI %d", vector, gsi);
+	}
+	if (i < NR_IRQS) {
+		gsi_2_irq[gsi] = i;
+		printk(KERN_INFO "GSI %d sharing vector 0x%02X and IRQ %d\n",
+				gsi, vector, i);
+		return i;
+	}
+
+	i = next_irq++;
+	BUG_ON(i >= NR_IRQS);
+	gsi_2_irq[gsi] = i;
+	IO_APIC_VECTOR(i) = vector;
+	printk(KERN_INFO "GSI %d assigned vector 0x%02X and IRQ %d\n",
+			gsi, vector, i);
+	return i;
+}
+
 static int pin_2_irq(int idx, int apic, int pin)
 {
 	int irq, i;
@@ -638,6 +701,7 @@ static int pin_2_irq(int idx, int apic, 
 			while (i < apic)
 				irq += nr_ioapic_registers[i++];
 			irq += pin;
+			irq = gsi_irq_sharing(irq);
 			break;
 		}
 		default:
@@ -647,6 +711,7 @@ static int pin_2_irq(int idx, int apic, 
 			break;
 		}
 	}
+	BUG_ON(irq >= NR_IRQS);
 
 	/*
 	 * PCI IRQ command line redirection. Yes, limits are hardcoded.
@@ -662,6 +727,7 @@ static int pin_2_irq(int idx, int apic, 
 			}
 		}
 	}
+	BUG_ON(irq >= NR_IRQS);
 	return irq;
 }
 
@@ -689,8 +755,8 @@ int assign_irq_vector(int irq)
 {
 	static int current_vector = FIRST_DEVICE_VECTOR, offset = 0;
 
-	BUG_ON(irq >= NR_IRQ_VECTORS);
-	if (IO_APIC_VECTOR(irq) > 0)
+	BUG_ON(irq != AUTO_ASSIGN && (unsigned)irq >= NR_IRQ_VECTORS);
+	if (irq != AUTO_ASSIGN && IO_APIC_VECTOR(irq) > 0)
 		return IO_APIC_VECTOR(irq);
 next:
 	current_vector += 8;
@@ -698,9 +764,8 @@ next:
 		goto next;
 
 	if (current_vector >= FIRST_SYSTEM_VECTOR) {
-		offset++;
-		if (!(offset%8))
-			return -ENOSPC;
+		/* If we run out of vectors on large boxen, must share them. */
+		offset = (offset + 1) % 8;
 		current_vector = FIRST_DEVICE_VECTOR + offset;
 	}
 
@@ -1920,6 +1985,7 @@ int io_apic_set_pci_routing (int ioapic,
 	entry.polarity = active_high_low;
 	entry.mask = 1;					 /* Disabled (masked) */
 
+	irq = gsi_irq_sharing(irq);
 	/*
 	 * IRQs < 16 are already in the irq_2_pin[] map
 	 */
diff -pru 2.6.13-rc3-mm1/arch/x86_64/kernel/irq.c j13-rc3-mm1/arch/x86_64/kernel/irq.c
--- 2.6.13-rc3-mm1/arch/x86_64/kernel/irq.c	2005-07-22 18:32:35.000000000 -0700
+++ j13-rc3-mm1/arch/x86_64/kernel/irq.c	2005-07-22 18:34:08.000000000 -0700
@@ -99,7 +99,7 @@ asmlinkage unsigned int do_IRQ(struct pt
 	unsigned irq = regs->orig_rax & 0xff;
 
 	irq_enter();
-	BUG_ON(irq > 256);
+	BUG_ON(irq > NR_IRQS);
 
 	__do_IRQ(irq, regs);
 	irq_exit();
diff -pru 2.6.13-rc3-mm1/arch/x86_64/kernel/mpparse.c j13-rc3-mm1/arch/x86_64/kernel/mpparse.c
--- 2.6.13-rc3-mm1/arch/x86_64/kernel/mpparse.c	2005-07-22 18:32:35.000000000 -0700
+++ j13-rc3-mm1/arch/x86_64/kernel/mpparse.c	2005-07-22 18:34:08.000000000 -0700
@@ -216,7 +216,7 @@ static void __init MP_intsrc_info (struc
 			m->mpc_irqtype, m->mpc_irqflag & 3,
 			(m->mpc_irqflag >> 2) & 3, m->mpc_srcbus,
 			m->mpc_srcbusirq, m->mpc_dstapic, m->mpc_dstirq);
-	if (++mp_irq_entries == MAX_IRQ_SOURCES)
+	if (++mp_irq_entries >= MAX_IRQ_SOURCES)
 		panic("Max # of irq sources exceeded!!\n");
 }
 
@@ -248,9 +248,10 @@ static void __init MP_lintsrc_info (stru
 
 static int __init smp_read_mpc(struct mp_config_table *mpc)
 {
-	char str[16];
+	char oem[9], str[16];
 	int count=sizeof(*mpc);
 	unsigned char *mpt=((unsigned char *)mpc)+count;
+	extern void mps_oem_check(struct mp_config_table *mpc, char *oem, char *productid);
 
 	if (memcmp(mpc->mpc_signature,MPC_SIGNATURE,4)) {
 		printk("SMP mptable: bad signature [%c%c%c%c]!\n",
@@ -273,14 +274,16 @@ static int __init smp_read_mpc(struct mp
 		printk(KERN_ERR "SMP mptable: null local APIC address!\n");
 		return 0;
 	}
-	memcpy(str,mpc->mpc_oem,8);
-	str[8]=0;
-	printk(KERN_INFO "OEM ID: %s ",str);
+	memcpy(oem, mpc->mpc_oem, 8);
+	oem[8] = 0;
+	printk(KERN_INFO "OEM ID: %s ", oem);
 
 	memcpy(str,mpc->mpc_productid,12);
 	str[12]=0;
 	printk(KERN_INFO "Product ID: %s ",str);
 
+	mps_oem_check(mpc, oem, str);
+
 	printk(KERN_INFO "APIC at: 0x%X\n",mpc->mpc_lapic);
 
 	/* save the local APIC address, it might be non-default */
diff -pru 2.6.13-rc3-mm1/include/asm-x86_64/desc.h j13-rc3-mm1/include/asm-x86_64/desc.h
--- 2.6.13-rc3-mm1/include/asm-x86_64/desc.h	2005-07-12 21:46:46.000000000 -0700
+++ j13-rc3-mm1/include/asm-x86_64/desc.h	2005-07-22 18:34:08.000000000 -0700
@@ -95,16 +95,19 @@ static inline void _set_gate(void *adr, 
 
 static inline void set_intr_gate(int nr, void *func) 
 { 
+	BUG_ON((unsigned)nr > 0xFF);
 	_set_gate(&idt_table[nr], GATE_INTERRUPT, (unsigned long) func, 0, 0); 
 } 
 
 static inline void set_intr_gate_ist(int nr, void *func, unsigned ist) 
 { 
+	BUG_ON((unsigned)nr > 0xFF);
 	_set_gate(&idt_table[nr], GATE_INTERRUPT, (unsigned long) func, 0, ist); 
 } 
 
 static inline void set_system_gate(int nr, void *func) 
 { 
+	BUG_ON((unsigned)nr > 0xFF);
 	_set_gate(&idt_table[nr], GATE_INTERRUPT, (unsigned long) func, 3, 0); 
 } 
 
diff -pru 2.6.13-rc3-mm1/include/asm-x86_64/mpspec.h j13-rc3-mm1/include/asm-x86_64/mpspec.h
--- 2.6.13-rc3-mm1/include/asm-x86_64/mpspec.h	2005-07-12 21:46:46.000000000 -0700
+++ j13-rc3-mm1/include/asm-x86_64/mpspec.h	2005-07-22 18:34:08.000000000 -0700
@@ -157,7 +157,8 @@ struct mpc_config_lintsrc
  */
 
 #define MAX_MP_BUSSES 256
-#define MAX_IRQ_SOURCES 256
+/* Each PCI slot may be a combo card with its own bus.  4 IRQ pins per slot. */
+#define MAX_IRQ_SOURCES (MAX_MP_BUSSES * 4)
 enum mp_bustype {
 	MP_BUS_ISA = 1,
 	MP_BUS_EISA,

^ permalink raw reply	[flat|nested] 21+ messages in thread
end of thread, other threads:[~2005-08-16  6:58 UTC | newest]

Thread overview: 21+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2005-08-11  0:21 [RFC][2.6.12.3] IRQ compression/sharing patch Protasevich, Natalie
2005-08-11  3:14 ` James Cleverdon
  -- strict thread matches above, loose matches on Subject: below --
2005-08-15  4:35 Protasevich, Natalie
2005-08-15 17:11 ` James Cleverdon
2005-08-11 22:02 Protasevich, Natalie
2005-08-11 22:34 ` Zwane Mwaikambo
2005-08-11 21:55 Protasevich, Natalie
2005-08-12  1:07 ` James Cleverdon
2005-08-12  2:59 ` James Cleverdon
2005-08-11 13:15 Protasevich, Natalie
2005-08-11 17:24 ` James Cleverdon
2005-08-10 21:03 Protasevich, Natalie
2005-08-10 23:55 ` James Cleverdon
2005-08-11 17:52 ` Zwane Mwaikambo
2005-07-26  7:12 [RFC][2.6.13-rc3-mm1] " James Cleverdon
2005-07-26 16:03 ` Andi Kleen
2005-08-04  7:05   ` [RFC][2.6.12.3] " James Cleverdon
2005-08-04  9:22     ` Andi Kleen
2005-08-15  2:57       ` James Cleverdon
2005-08-15  5:55         ` Zwane Mwaikambo
2005-08-15 17:44         ` Andi Kleen
2005-08-16  3:24           ` James Cleverdon
2005-08-16  6:58             ` Andi Kleen

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
Powered by JetHome