Re: [discuss] Re: x86_64: 2.6.14-rc4 swiotlb broken

[Yasunori Goto <y-goto@jp.fujitsu.com>]

I tested your patch, but unfortunately, it doesn't work IA64.
alloc_bootmem_node() requires bigger area than MAX_DMA_ADDRESS.
It is defined as 4GB for ia64. (arch/ia64/mm/init.c)
But this patch require smaller area than 4GB. 
So they are exclusive each other.

I'm convinced that a new interface like alloc_bootmem_low32() is
necessary after all. ;-)

Thanks.


> On Tue, Oct 18, 2005 at 01:28:20PM +0900, Yasunori Goto wrote:
> > > > So, just "use NODE(0)" is not enough hack for our machine.
> > > > If "use NODE(0)" is selected, kernel must sort pgdat link and
> > > > node id by memory address. I think that hot add code will be a 
> > > > bit messy instead.
> > > 
> > > Yasunori-san,
> > > Does this patch work on your boxes instead? (For 2.6.14)
> > > http://marc.theaimsgroup.com/?l=linux-kernel&m=112959469914681&w=2
> > 
> > Not yet. But could you change this line at least?
> > 
> > +	
> > +	for_each_node(node) {
> > +		io_tlb_start = alloc_bootmem_node(NODE_DATA(node), iotlbsz);
> > 
> > for_each_node() loop walks around node_possible_map which includes
> > "offlined" node. 
> > Please use for_each_online_node() instead. Then, I'll check it. :-)
> 
> Since swiotlb is is allocated even before APs are brought up, I thought,
> if the node containing lowmem32 was not the boot node, it would not be
> online.   But on closer look, my assumption was wrong.  Here is the patch
> which iterates through online nodes to allocate lowmem32 bootmem for
> swiotlb.
> 
> --
> 
> Patch to ensure low32 mem allocation for x86_64 swiotlb 
> 
> Signed-off-by: Ravikiran Thirumalai <kiran@scalex86.org>
> 
> Index: linux-2.6.14-rc4/arch/ia64/lib/swiotlb.c
> ===================================================================
> --- linux-2.6.14-rc4.orig/arch/ia64/lib/swiotlb.c	2005-10-17 13:27:35.000000000 -0700
> +++ linux-2.6.14-rc4/arch/ia64/lib/swiotlb.c	2005-10-17 16:00:44.000000000 -0700
> @@ -106,6 +106,8 @@
>  __setup("swiotlb=", setup_io_tlb_npages);
>  /* make io_tlb_overflow tunable too? */
>  
> +#define IS_LOWPAGES(paddr, size) ((paddr < 0xffffffff) && ((paddr+size) < 0xffffffff))
> +
>  /*
>   * Statically reserve bounce buffer space and initialize bounce buffer data
>   * structures for the software IO TLB used to implement the PCI DMA API.
> @@ -114,17 +116,32 @@
>  swiotlb_init_with_default_size (size_t default_size)
>  {
>  	unsigned long i;
> +	unsigned long iotlbsz;
> +	int node;
>  
>  	if (!io_tlb_nslabs) {
>  		io_tlb_nslabs = (default_size >> IO_TLB_SHIFT);
>  		io_tlb_nslabs = ALIGN(io_tlb_nslabs, IO_TLB_SEGSIZE);
>  	}
>  
> +	iotlbsz = io_tlb_nslabs * (1 << IO_TLB_SHIFT);	
> +
>  	/*
> -	 * Get IO TLB memory from the low pages
> +	 * Get IO TLB memory from the 0-4G range
>  	 */
> -	io_tlb_start = alloc_bootmem_low_pages(io_tlb_nslabs *
> -					       (1 << IO_TLB_SHIFT));
> +	
> +	for_each_online_node(node) {
> +		io_tlb_start = alloc_bootmem_node(NODE_DATA(node), iotlbsz);
> +		if (io_tlb_start) {
> +			if (IS_LOWPAGES(virt_to_phys(io_tlb_start), iotlbsz))
> +				break;
> +			free_bootmem_node(NODE_DATA(node), 
> +					  virt_to_phys(io_tlb_start), iotlbsz);
> +			io_tlb_start = NULL;
> +		}
> +	}
> +
> +	
>  	if (!io_tlb_start)
>  		panic("Cannot allocate SWIOTLB buffer");
>  	io_tlb_end = io_tlb_start + io_tlb_nslabs * (1 << IO_TLB_SHIFT);

-- 
Yasunori Goto