Re: [PATCH] TTM DMA pool v2.2 or [GIT PULL] (stable/ttm.dma_pool.v2.3) for 3.3

[Jerome Glisse <jglisse@redhat.com>]

On Fri, Nov 04, 2011 at 03:41:03PM -0400, Konrad Rzeszutek Wilk wrote:
> On Fri, Nov 04, 2011 at 03:24:51PM -0400, Jerome Glisse wrote:
> > On Fri, Nov 04, 2011 at 02:44:53PM -0400, Konrad Rzeszutek Wilk wrote:
> > > > > git://git.kernel.org/pub/scm/linux/kernel/git/konrad/xen.git stable/ttm.dma_pool.v2.3
> > > > > 
> > > > 
> > > > On what hw did you tested ? With and without xen ? Here radeon
> > > 
> > > On AMD and Intel. And with both Nvidia and Radeon cards.
> > > 64-bit cards (I have a patch where I forced the 64-bit card to use
> > > the TTM DMA pool code to test) and 32-bit cards (ATI ES1000)
> > > 
> > > On baremetal and Xen. Um, Fedora Core 16 as distro.
> > > 
> > > Oh, and I also tried PPC (Power Mac 4) but could not get it to boot
> > > the 3.1 kernel. Something with the LILO grub loader did not work.
> > > 
> > > > that doesn't need dma32 doesn't work when forcing swiotlb which
> > > > kind of expected i guess. Should we expose if swiotlb is enabled
> > > 
> > > You did 'swiotlb=force' ?
> > > > forced so we use dma pool in such case ?
> > 
> > Issue is that when booted without force swiotlb_nr_tlb still return
> > positive thus we endup using the dma pool path.
> 
> Did " Using software bounce buffering for IO (SWIOTLB)" or "software IO TLB at"
> show up in the dmesg output? You might have to run it with 'debug loglevel=8'?
> Presumarily yes, since the code "swiotlb: Expose.." sets
>   io_tlb_nslabs = 0
> 
> if there is no need for it. And since io_tlb_nslabs is set, then the code
> did start.
> 
> Some AMD boxes with the AMD-Vi chipset enable the SWIOTLB b/c not all of
> the PCI devices are on the IOMMU chipset path. The Intel VT-d does the same
> thing.
> 
> Hmm, I think the reason for those devices to turn SWIOTLB on is that they
> are not comfortable handling 32-bit devices, and the TTM DMA pool code would
> only turn itself on when the radeon|nouveau was 32-bit _and_ SWIOTLB was enabled.
> 
> .. Testing the patches you posted on my AMD box.

Yes, swiotlb was enabled (bounce buffer message) thing is when force is
not set usual ttm memory page allocation works properly ie on pci map
no bounce buffer is use, but when force is set it does use a bounce
This is due to :
if (dma_capable(dev, dev_addr, size) && !swiotlb_force)
in swiotlb_map_page, i am not sure how much the force argument is
usefull.

Anyway i did few benchmark and it seems that the dma allocator isn't
slower than the other page allocator. But this is limited testing
(openarena, nexuiz running on composited desktop with firefox).

Cheers,
Jerome