[PATCH] pci/intr_remapping: allocate irq_iommu on node

[PATCH] pci/intr_remapping: allocate irq_iommu on node

[Yinghai Lu]

make it use node from irq_desc.

Signed-off-by: Yinghai Lu <yinghai@kernel.org>

---
 drivers/pci/intr_remapping.c |    4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

Index: linux-2.6/drivers/pci/intr_remapping.c
===================================================================
--- linux-2.6.orig/drivers/pci/intr_remapping.c
+++ linux-2.6/drivers/pci/intr_remapping.c
@@ -201,7 +201,7 @@ int alloc_irte(struct intel_iommu *iommu
 	for (i = index; i < index + count; i++)
 		table->base[i].present = 1;
 
-	irq_iommu = irq_2_iommu_alloc(irq);
+	irq_iommu = irq_2_iommu_alloc_node(irq, irq_to_desc(irq)->node);
 	if (!irq_iommu) {
 		spin_unlock_irqrestore(&irq_2_ir_lock, flags);
 		printk(KERN_ERR "can't allocate irq_2_iommu\n");
@@ -255,7 +255,7 @@ int set_irte_irq(int irq, struct intel_i
 
 	spin_lock_irqsave(&irq_2_ir_lock, flags);
 
-	irq_iommu = irq_2_iommu_alloc(irq);
+	irq_iommu = irq_2_iommu_alloc_node(irq, irq_to_desc(irq)->node);
 
 	if (!irq_iommu) {
 		spin_unlock_irqrestore(&irq_2_ir_lock, flags);

Re: [PATCH] pci/intr_remapping: allocate irq_iommu on node

[Suresh Siddha]

On Tue, 2009-08-04 at 09:04 -0700, Yinghai Lu wrote:
> make it use node from irq_desc.
> 
> Signed-off-by: Yinghai Lu <yinghai@kernel.org>
> 
> ---
>  drivers/pci/intr_remapping.c |    4 ++--
>  1 file changed, 2 insertions(+), 2 deletions(-)
> 
> Index: linux-2.6/drivers/pci/intr_remapping.c
> ===================================================================
> --- linux-2.6.orig/drivers/pci/intr_remapping.c
> +++ linux-2.6/drivers/pci/intr_remapping.c
> @@ -201,7 +201,7 @@ int alloc_irte(struct intel_iommu *iommu
>  	for (i = index; i < index + count; i++)
>  		table->base[i].present = 1;
>  
> -	irq_iommu = irq_2_iommu_alloc(irq);
> +	irq_iommu = irq_2_iommu_alloc_node(irq, irq_to_desc(irq)->node);

As far as I can see, this node will always be boot_cpu_id's node. We
allocate the IRTE at the time of the create_irq() and at that time,
desc->node will always be of the boot_cpu_id's.

Can you please elaborate when this node will be different to make this
an effective patch?

thanks,
suresh

Re: [PATCH] pci/intr_remapping: allocate irq_iommu on node

[Yinghai Lu]

Suresh Siddha wrote:
> On Tue, 2009-08-04 at 09:04 -0700, Yinghai Lu wrote:
>> make it use node from irq_desc.
>>
>> Signed-off-by: Yinghai Lu <yinghai@kernel.org>
>>
>> ---
>>  drivers/pci/intr_remapping.c |    4 ++--
>>  1 file changed, 2 insertions(+), 2 deletions(-)
>>
>> Index: linux-2.6/drivers/pci/intr_remapping.c
>> ===================================================================
>> --- linux-2.6.orig/drivers/pci/intr_remapping.c
>> +++ linux-2.6/drivers/pci/intr_remapping.c
>> @@ -201,7 +201,7 @@ int alloc_irte(struct intel_iommu *iommu
>>  	for (i = index; i < index + count; i++)
>>  		table->base[i].present = 1;
>>  
>> -	irq_iommu = irq_2_iommu_alloc(irq);
>> +	irq_iommu = irq_2_iommu_alloc_node(irq, irq_to_desc(irq)->node);
> 
> As far as I can see, this node will always be boot_cpu_id's node. We
> allocate the IRTE at the time of the create_irq() and at that time,
> desc->node will always be of the boot_cpu_id's.
> 
> Can you please elaborate when this node will be different to make this
> an effective patch?

without this patch

[  299.175436] igbvf 0000:41:10.0: enabling device (0000 -> 0002)
[  299.180009] igbvf 0000:41:10.0: using 64bit DMA mask
[  299.183077] igbvf 0000:41:10.0: using 64bit consistent DMA mask
[  299.199870] igbvf 0000:41:10.0: enabling bus mastering
[  299.205534] igbvf 0000:41:10.0: setting latency timer to 64
[  299.219654] reserve_memtype added 0xcfc00000-0xcfc04000, track uncached-minus, req uncached-minus, ret uncached-minus
[  299.239615] reserve_memtype added 0xcfc20000-0xcfc21000, track uncached-minus, req uncached-minus, ret uncached-minus
[  299.260181]   alloc irq_desc for 217 on node 2
[  299.263501]   alloc kstat_irqs on node 2
[  299.276276] alloc irq_2_iommu on node 0
[  299.281847] igbvf 0000:41:10.0: irq 217 for MSI/MSI-X
[  299.285511]   alloc irq_desc for 218 on node 2
[  299.297813]   alloc kstat_irqs on node 2
[  299.302543] alloc irq_2_iommu on node 0
[  299.315358] igbvf 0000:41:10.0: irq 218 for MSI/MSI-X
[  299.321102]   alloc irq_desc for 219 on node 2
[  299.325932]   alloc kstat_irqs on node 2
[  299.338616] alloc irq_2_iommu on node 0
[  299.340789] igbvf 0000:41:10.0: irq 219 for MSI/MSI-X
[  299.356532] igbvf 0000:41:10.0: PF still in reset state, assigning new address
[  299.365128] igbvf 0000:41:10.0: PF still resetting
[  299.377625] igbvf 0000:41:10.0: Intel(R) 82576 Virtual Function
[  299.385619] igbvf 0000:41:10.0: Address: c6:e3:de:62:de:eb
[  299.395957] igbvf 0000:41:10.0: MAC: 1

thought we could put irq_2_iommu to same node too.

YH

Re: [PATCH] pci/intr_remapping: allocate irq_iommu on node

[Suresh Siddha]

On Wed, 2009-08-05 at 02:14 -0700, Yinghai Lu wrote:
> without this patch
> 
> [  299.175436] igbvf 0000:41:10.0: enabling device (0000 -> 0002)
> [  299.180009] igbvf 0000:41:10.0: using 64bit DMA mask
> [  299.183077] igbvf 0000:41:10.0: using 64bit consistent DMA mask
> [  299.199870] igbvf 0000:41:10.0: enabling bus mastering
> [  299.205534] igbvf 0000:41:10.0: setting latency timer to 64
> [  299.219654] reserve_memtype added 0xcfc00000-0xcfc04000, track uncached-minus, req uncached-minus, ret uncached-minus
> [  299.239615] reserve_memtype added 0xcfc20000-0xcfc21000, track uncached-minus, req uncached-minus, ret uncached-minus
> [  299.260181]   alloc irq_desc for 217 on node 2
> [  299.263501]   alloc kstat_irqs on node 2
> [  299.276276] alloc irq_2_iommu on node 0
> [  299.281847] igbvf 0000:41:10.0: irq 217 for MSI/MSI-X
> [  299.285511]   alloc irq_desc for 218 on node 2
> [  299.297813]   alloc kstat_irqs on node 2
> [  299.302543] alloc irq_2_iommu on node 0
> [  299.315358] igbvf 0000:41:10.0: irq 218 for MSI/MSI-X
> [  299.321102]   alloc irq_desc for 219 on node 2
> [  299.325932]   alloc kstat_irqs on node 2
> [  299.338616] alloc irq_2_iommu on node 0
> [  299.340789] igbvf 0000:41:10.0: irq 219 for MSI/MSI-X
> [  299.356532] igbvf 0000:41:10.0: PF still in reset state, assigning new address
> [  299.365128] igbvf 0000:41:10.0: PF still resetting
> [  299.377625] igbvf 0000:41:10.0: Intel(R) 82576 Virtual Function
> [  299.385619] igbvf 0000:41:10.0: Address: c6:e3:de:62:de:eb
> [  299.395957] igbvf 0000:41:10.0: MAC: 1
> 
> thought we could put irq_2_iommu to same node too.

Yinghai, even with this patch, we will still be allocating from node-0.
No?

Re: [PATCH] pci/intr_remapping: allocate irq_iommu on node

[Yinghai Lu]

Suresh Siddha wrote:
> On Wed, 2009-08-05 at 02:14 -0700, Yinghai Lu wrote:
>> without this patch
>>
>> [  299.175436] igbvf 0000:41:10.0: enabling device (0000 -> 0002)
>> [  299.180009] igbvf 0000:41:10.0: using 64bit DMA mask
>> [  299.183077] igbvf 0000:41:10.0: using 64bit consistent DMA mask
>> [  299.199870] igbvf 0000:41:10.0: enabling bus mastering
>> [  299.205534] igbvf 0000:41:10.0: setting latency timer to 64
>> [  299.219654] reserve_memtype added 0xcfc00000-0xcfc04000, track uncached-minus, req uncached-minus, ret uncached-minus
>> [  299.239615] reserve_memtype added 0xcfc20000-0xcfc21000, track uncached-minus, req uncached-minus, ret uncached-minus
>> [  299.260181]   alloc irq_desc for 217 on node 2
>> [  299.263501]   alloc kstat_irqs on node 2
>> [  299.276276] alloc irq_2_iommu on node 0
>> [  299.281847] igbvf 0000:41:10.0: irq 217 for MSI/MSI-X
>> [  299.285511]   alloc irq_desc for 218 on node 2
>> [  299.297813]   alloc kstat_irqs on node 2
>> [  299.302543] alloc irq_2_iommu on node 0
>> [  299.315358] igbvf 0000:41:10.0: irq 218 for MSI/MSI-X
>> [  299.321102]   alloc irq_desc for 219 on node 2
>> [  299.325932]   alloc kstat_irqs on node 2
>> [  299.338616] alloc irq_2_iommu on node 0
>> [  299.340789] igbvf 0000:41:10.0: irq 219 for MSI/MSI-X
>> [  299.356532] igbvf 0000:41:10.0: PF still in reset state, assigning new address
>> [  299.365128] igbvf 0000:41:10.0: PF still resetting
>> [  299.377625] igbvf 0000:41:10.0: Intel(R) 82576 Virtual Function
>> [  299.385619] igbvf 0000:41:10.0: Address: c6:e3:de:62:de:eb
>> [  299.395957] igbvf 0000:41:10.0: MAC: 1
>>
>> thought we could put irq_2_iommu to same node too.
> 
> Yinghai, even with this patch, we will still be allocating from node-0.
> No?

with this patch will get 
alloc irq_2_iommu on node 2

YH

Re: [PATCH] pci/intr_remapping: allocate irq_iommu on node

[Suresh Siddha]

On Wed, 2009-08-05 at 18:19 -0700, Yinghai Lu wrote:
> Suresh Siddha wrote:
> > 
> > Yinghai, even with this patch, we will still be allocating from node-0.
> > No?
> 
> with this patch will get 
> alloc irq_2_iommu on node 2

Looking closer, I missed the fact that we are now using the
create_irq_nr() which has the node information.

Acked-by: Suresh Siddha <suresh.b.siddha@intel.com>

Thanks.

[tip:irq/core] pci/intr_remapping: Allocate irq_iommu on node

[tip-bot for Yinghai Lu]

Commit-ID:  94fb9a6b49d3a63831c4e752f6f0ab8da0a99eb5
Gitweb:     http://git.kernel.org/tip/94fb9a6b49d3a63831c4e752f6f0ab8da0a99eb5
Author:     Yinghai Lu <yinghai@kernel.org>
AuthorDate: Tue, 4 Aug 2009 09:04:39 -0700
Committer:  Ingo Molnar <mingo@elte.hu>
CommitDate: Sat, 8 Aug 2009 17:07:37 +0200

pci/intr_remapping: Allocate irq_iommu on node

Make it use node from irq_desc - this results in more
optimal memory layout.

Signed-off-by: Yinghai Lu <yinghai@kernel.org>
Cc: Jesse Barnes <jbarnes@virtuousgeek.org>
Acked-by: Suresh Siddha <suresh.b.siddha@intel.com>
LKML-Reference: <4A785C17.3030104@kernel.org>
Signed-off-by: Ingo Molnar <mingo@elte.hu>


---
 drivers/pci/intr_remapping.c |    4 ++--
 1 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/drivers/pci/intr_remapping.c b/drivers/pci/intr_remapping.c
index 4f5b871..e27a28c 100644
--- a/drivers/pci/intr_remapping.c
+++ b/drivers/pci/intr_remapping.c
@@ -201,7 +201,7 @@ int alloc_irte(struct intel_iommu *iommu, int irq, u16 count)
 	for (i = index; i < index + count; i++)
 		table->base[i].present = 1;
 
-	irq_iommu = irq_2_iommu_alloc(irq);
+	irq_iommu = irq_2_iommu_alloc_node(irq, irq_to_desc(irq)->node);
 	if (!irq_iommu) {
 		spin_unlock_irqrestore(&irq_2_ir_lock, flags);
 		printk(KERN_ERR "can't allocate irq_2_iommu\n");
@@ -255,7 +255,7 @@ int set_irte_irq(int irq, struct intel_iommu *iommu, u16 index, u16 subhandle)
 
 	spin_lock_irqsave(&irq_2_ir_lock, flags);
 
-	irq_iommu = irq_2_iommu_alloc(irq);
+	irq_iommu = irq_2_iommu_alloc_node(irq, irq_to_desc(irq)->node);
 
 	if (!irq_iommu) {
 		spin_unlock_irqrestore(&irq_2_ir_lock, flags);

Re: [tip:irq/core] pci/intr_remapping: Allocate irq_iommu on node

[Ingo Molnar]

* tip-bot for Yinghai Lu <yinghai@kernel.org> wrote:

> Commit-ID:  94fb9a6b49d3a63831c4e752f6f0ab8da0a99eb5
> Gitweb:     http://git.kernel.org/tip/94fb9a6b49d3a63831c4e752f6f0ab8da0a99eb5
> Author:     Yinghai Lu <yinghai@kernel.org>
> AuthorDate: Tue, 4 Aug 2009 09:04:39 -0700
> Committer:  Ingo Molnar <mingo@elte.hu>
> CommitDate: Sat, 8 Aug 2009 17:07:37 +0200
> 
> pci/intr_remapping: Allocate irq_iommu on node

-tip testing found that this patch broke the x86 build:

drivers/pci/intr_remapping.c: In function ‘alloc_irte’:
drivers/pci/intr_remapping.c:204: error: ‘struct irq_desc’ has no member named ‘node’
drivers/pci/intr_remapping.c: In function ‘set_irte_irq’:
drivers/pci/intr_remapping.c:258: error: ‘struct irq_desc’ has no member named ‘node’

	Ingo

Re: [tip:irq/core] pci/intr_remapping: Allocate irq_iommu on node

[Jesse Barnes]

On Sun, 9 Aug 2009 12:44:00 +0200
Ingo Molnar <mingo@elte.hu> wrote:

> 
> * tip-bot for Yinghai Lu <yinghai@kernel.org> wrote:
> 
> > Commit-ID:  94fb9a6b49d3a63831c4e752f6f0ab8da0a99eb5
> > Gitweb:
> > http://git.kernel.org/tip/94fb9a6b49d3a63831c4e752f6f0ab8da0a99eb5
> > Author:     Yinghai Lu <yinghai@kernel.org> AuthorDate: Tue, 4 Aug
> > 2009 09:04:39 -0700 Committer:  Ingo Molnar <mingo@elte.hu>
> > CommitDate: Sat, 8 Aug 2009 17:07:37 +0200
> > 
> > pci/intr_remapping: Allocate irq_iommu on node
> 
> -tip testing found that this patch broke the x86 build:
> 
> drivers/pci/intr_remapping.c: In function ‘alloc_irte’:
> drivers/pci/intr_remapping.c:204: error: ‘struct irq_desc’ has no
> member named ‘node’ drivers/pci/intr_remapping.c: In function
> ‘set_irte_irq’: drivers/pci/intr_remapping.c:258: error: ‘struct
> irq_desc’ has no member named ‘node’

Looks like we need a wrapper function for this with proper CONFIG_NUMA
checks...  I'll apply it to my linux-next tree when someone sends me a
new version.

Thanks,
-- 
Jesse Barnes, Intel Open Source Technology Center

[PATCH] pci/intr_remapping: allocate irq_iommu on node -v2

[Yinghai Lu]

make it use node from irq_desc.

also need to make sure legacy irq node is right, when there is no ram on node 0

Signed-off-by: Yinghai Lu <yinghai@kernel.org>

---
 drivers/pci/intr_remapping.c |   10 +++++++++-
 kernel/irq/handle.c          |    5 ++++-
 2 files changed, 13 insertions(+), 2 deletions(-)

Index: linux-2.6/drivers/pci/intr_remapping.c
===================================================================
--- linux-2.6.orig/drivers/pci/intr_remapping.c
+++ linux-2.6/drivers/pci/intr_remapping.c
@@ -79,7 +79,15 @@ static struct irq_2_iommu *irq_2_iommu_a
 
 static struct irq_2_iommu *irq_2_iommu_alloc(unsigned int irq)
 {
-	return irq_2_iommu_alloc_node(irq, cpu_to_node(boot_cpu_id));
+	int node;
+
+#ifdef CONFIG_SMP
+	node = irq_to_desc(irq)->node;
+#else
+	node = cpu_to_node(boot_cpu_id);
+#endif
+
+	return irq_2_iommu_alloc_node(irq, node);
 }
 
 #else /* !CONFIG_SPARSE_IRQ */
Index: linux-2.6/kernel/irq/handle.c
===================================================================
--- linux-2.6.orig/kernel/irq/handle.c
+++ linux-2.6/kernel/irq/handle.c
@@ -161,7 +161,7 @@ int __init early_irq_init(void)
 
 	desc = irq_desc_legacy;
 	legacy_count = ARRAY_SIZE(irq_desc_legacy);
- 	node = first_online_node;
+	node = first_online_node;
 
 	/* allocate irq_desc_ptrs array based on nr_irqs */
 	irq_desc_ptrs = kcalloc(nr_irqs, sizeof(void *), GFP_NOWAIT);
@@ -172,6 +172,9 @@ int __init early_irq_init(void)
 
 	for (i = 0; i < legacy_count; i++) {
 		desc[i].irq = i;
+#ifdef CONFIG_SMP
+		desc[i].node = node;
+#endif
 		desc[i].kstat_irqs = kstat_irqs_legacy + i * nr_cpu_ids;
 		lockdep_set_class(&desc[i].lock, &irq_desc_lock_class);
 		alloc_desc_masks(&desc[i], node, true);

Re: [PATCH] pci/intr_remapping: allocate irq_iommu on node -v2

[Ingo Molnar]

* Yinghai Lu <yinghai@kernel.org> wrote:

>  static struct irq_2_iommu *irq_2_iommu_alloc(unsigned int irq)
>  {
> -	return irq_2_iommu_alloc_node(irq, cpu_to_node(boot_cpu_id));
> +	int node;
> +
> +#ifdef CONFIG_SMP
> +	node = irq_to_desc(irq)->node;
> +#else
> +	node = cpu_to_node(boot_cpu_id);
> +#endif
> +
> +	return irq_2_iommu_alloc_node(irq, node);
>  }

To avoid the ifdef, shouldnt there be an new irq_node(irq) primitive 
that maps to desc->node on SMP and does the right thing on UP?

	Ingo

[PATCH] pci/intr_remapping: allocate irq_iommu on node -v3

[Yinghai Lu]

make it use node from irq_desc.

Signed-off-by: Yinghai Lu <yinghai@kernel.org>
Acked-by: Jesse Barnes <jbarnes@virtuousgeek.org>

---
 drivers/pci/intr_remapping.c |   14 +++-----------
 1 file changed, 3 insertions(+), 11 deletions(-)

Index: linux-2.6/drivers/pci/intr_remapping.c
===================================================================
--- linux-2.6.orig/drivers/pci/intr_remapping.c
+++ linux-2.6/drivers/pci/intr_remapping.c
@@ -55,15 +55,12 @@ static struct irq_2_iommu *irq_2_iommu(u
 	return desc->irq_2_iommu;
 }
 
-static struct irq_2_iommu *irq_2_iommu_alloc_node(unsigned int irq, int node)
+static struct irq_2_iommu *irq_2_iommu_alloc(unsigned int irq)
 {
 	struct irq_desc *desc;
 	struct irq_2_iommu *irq_iommu;
 
-	/*
-	 * alloc irq desc if not allocated already.
-	 */
-	desc = irq_to_desc_alloc_node(irq, node);
+	desc = irq_to_desc(irq);
 	if (!desc) {
 		printk(KERN_INFO "can not get irq_desc for %d\n", irq);
 		return NULL;
@@ -72,16 +69,11 @@ static struct irq_2_iommu *irq_2_iommu_a
 	irq_iommu = desc->irq_2_iommu;
 
 	if (!irq_iommu)
-		desc->irq_2_iommu = get_one_free_irq_2_iommu(node);
+		desc->irq_2_iommu = get_one_free_irq_2_iommu(irq_node(irq));
 
 	return desc->irq_2_iommu;
 }
 
-static struct irq_2_iommu *irq_2_iommu_alloc(unsigned int irq)
-{
-	return irq_2_iommu_alloc_node(irq, cpu_to_node(boot_cpu_id));
-}
-
 #else /* !CONFIG_SPARSE_IRQ */
 
 static struct irq_2_iommu irq_2_iommuX[NR_IRQS];

[tip:irq/core] pci/intr_remapping: Allocate irq_iommu on node

[tip-bot for Yinghai Lu]

Commit-ID:  70590ea75b7f9ef4846b0b0f4400e8338dbcc7eb
Gitweb:     http://git.kernel.org/tip/70590ea75b7f9ef4846b0b0f4400e8338dbcc7eb
Author:     Yinghai Lu <yinghai@kernel.org>
AuthorDate: Wed, 26 Aug 2009 16:21:54 -0700
Committer:  Ingo Molnar <mingo@elte.hu>
CommitDate: Sat, 29 Aug 2009 15:53:01 +0200

pci/intr_remapping: Allocate irq_iommu on node

make it use the node from irq_desc.

Signed-off-by: Yinghai Lu <yinghai@kernel.org>
Acked-by: Jesse Barnes <jbarnes@virtuousgeek.org>
Cc: Andrew Morton <akpm@linux-foundation.org>
LKML-Reference: <4A95C392.5050903@kernel.org>
Signed-off-by: Ingo Molnar <mingo@elte.hu>


---
 drivers/pci/intr_remapping.c |   14 +++-----------
 1 files changed, 3 insertions(+), 11 deletions(-)

diff --git a/drivers/pci/intr_remapping.c b/drivers/pci/intr_remapping.c
index 4f5b871..4480364 100644
--- a/drivers/pci/intr_remapping.c
+++ b/drivers/pci/intr_remapping.c
@@ -55,15 +55,12 @@ static struct irq_2_iommu *irq_2_iommu(unsigned int irq)
 	return desc->irq_2_iommu;
 }
 
-static struct irq_2_iommu *irq_2_iommu_alloc_node(unsigned int irq, int node)
+static struct irq_2_iommu *irq_2_iommu_alloc(unsigned int irq)
 {
 	struct irq_desc *desc;
 	struct irq_2_iommu *irq_iommu;
 
-	/*
-	 * alloc irq desc if not allocated already.
-	 */
-	desc = irq_to_desc_alloc_node(irq, node);
+	desc = irq_to_desc(irq);
 	if (!desc) {
 		printk(KERN_INFO "can not get irq_desc for %d\n", irq);
 		return NULL;
@@ -72,16 +69,11 @@ static struct irq_2_iommu *irq_2_iommu_alloc_node(unsigned int irq, int node)
 	irq_iommu = desc->irq_2_iommu;
 
 	if (!irq_iommu)
-		desc->irq_2_iommu = get_one_free_irq_2_iommu(node);
+		desc->irq_2_iommu = get_one_free_irq_2_iommu(irq_node(irq));
 
 	return desc->irq_2_iommu;
 }
 
-static struct irq_2_iommu *irq_2_iommu_alloc(unsigned int irq)
-{
-	return irq_2_iommu_alloc_node(irq, cpu_to_node(boot_cpu_id));
-}
-
 #else /* !CONFIG_SPARSE_IRQ */
 
 static struct irq_2_iommu irq_2_iommuX[NR_IRQS];