Re: [discuss] Re: x86_64: 2.6.14-rc4 swiotlb broken

[Ravikiran G Thirumalai <kiran@scalex86.org>]

On Tue, Oct 18, 2005 at 04:37:03PM -0600, Alex Williamson wrote:
> On Tue, 2005-10-18 at 16:04 -0600, Alex Williamson wrote:
> 
>    Nope, it breaks with a current git-2.6.14.  Here's what my extra
> printk says:
> 
> Node 0: 0xe000074104e67200
> Node 1: 0xe000082080722000
> Node 2: 0xe000000101532000
> Placing software IO TLB between 0x74108e68000 - 0x7410ce68000
> 

Hope the following works.   Using __alloc_bootmem_node now with a hard coded
goal to avoid 16MB DMA zone.  It is ugly :( and hope it works this time
<fingers crossed>. 

--

Patch to ensure low32 mem allocation for x86_64 swiotlb

Signed-off-by: Ravikiran Thirumalai <kiran@scalex86.org>

Index: linux-2.6.14-rc4/arch/ia64/lib/swiotlb.c
===================================================================
--- linux-2.6.14-rc4.orig/arch/ia64/lib/swiotlb.c	2005-10-18 14:14:12.000000000 -0700
+++ linux-2.6.14-rc4/arch/ia64/lib/swiotlb.c	2005-10-18 16:09:51.000000000 -0700
@@ -106,6 +106,8 @@
 __setup("swiotlb=", setup_io_tlb_npages);
 /* make io_tlb_overflow tunable too? */
 
+#define IS_LOWPAGES(paddr, size) ((paddr < 0xffffffff) && ((paddr+size) < 0xffffffff))
+
 /*
  * Statically reserve bounce buffer space and initialize bounce buffer data
  * structures for the software IO TLB used to implement the PCI DMA API.
@@ -114,17 +116,46 @@
 swiotlb_init_with_default_size (size_t default_size)
 {
 	unsigned long i;
+	unsigned long iotlbsz;
+	int node;
 
 	if (!io_tlb_nslabs) {
 		io_tlb_nslabs = (default_size >> IO_TLB_SHIFT);
 		io_tlb_nslabs = ALIGN(io_tlb_nslabs, IO_TLB_SEGSIZE);
 	}
 
+	iotlbsz = io_tlb_nslabs * (1 << IO_TLB_SHIFT);	
+
 	/*
-	 * Get IO TLB memory from the low pages
+	 * Get IO TLB memory from the 0-4G range
 	 */
-	io_tlb_start = alloc_bootmem_low_pages(io_tlb_nslabs *
-					       (1 << IO_TLB_SHIFT));
+	
+	for_each_online_node(node) {
+		/* Ugly, hate it.  To be gone post 2.6.14 */
+		io_tlb_start = __alloc_bootmem_node(NODE_DATA(node), 
+						    iotlbsz, PAGE_SIZE, 
+						    0x1000000);
+		if (io_tlb_start) {
+			if (IS_LOWPAGES(virt_to_phys(io_tlb_start), iotlbsz))
+				break;
+			free_bootmem_node(NODE_DATA(node), 
+					  virt_to_phys(io_tlb_start), iotlbsz);
+			io_tlb_start = NULL;
+		}
+	}
+
+	/* 
+	 * FIXME: This should go away when the bootmem allocator is fixed to
+	 * guarantee lowmem32 allocations somehow, and the swiotlb mess is 
+	 * cleaned.  The alloc_bootmem_low_pages fall back is to ensure 
+         * boxes like amd64  which donot use swiotlb but still have 
+	 * swiotlb compiled in, falls back to the 2.6.13 behaviour instead
+	 * of panicking, when proper low32 pages are not available
+	 */
+	if (!io_tlb_start)
+		io_tlb_start = alloc_bootmem_low_pages(io_tlb_nslabs *
+							(1 << IO_TLB_SHIFT));
+
 	if (!io_tlb_start)
 		panic("Cannot allocate SWIOTLB buffer");
 	io_tlb_end = io_tlb_start + io_tlb_nslabs * (1 << IO_TLB_SHIFT);