Re: DMA API issues

Re: DMA API issues

[James Bottomley]

    
    
    My colleagues and I are encountering a number of difficulties with the
    DMA API, to which a generic solution is required (or risk multiple
    architectures, busses, and devices going their own way...)
    
    Here is an example system that illustrates these problems:
    
    I have a System On Chip device which, among other functions, contains an
    OHCI controller and 32K of SRAM.
    
    heres the catch:- The OHCI controller has a different address space than
    the host bus, and worse, can *only* DMA data from its internal SRAM.
    
    The architecture is not broken, merely unusual.
    
    This causes the following problems:
    
    1) The DMA API provides no methods to set up a mapping between the host
       memory map and the devices view of the space
            example:
               the OHCI controller above would see its 32K of SRAM as
               mapped from 0x10000 - 0x1ffff and not 0xXXX10000 - 0xXXX1ffff
               which is the address the CPU sees.

Erm, well this isn't unusual.  A lot of devices have on board memory to
offload accesses to.  All the later Symbios SCSI chips for instance.  If
you look at the drivers, you'll see they ioremap the region and then use
it via the normal memory accessors.

    2) The DMA API assumes the device can access SDRAM
            example:
               the OHCI controller base is mapped at 0x10000000 on my platform.
               this is NOT is SDRAM, its in IO space.

The usual thing for devices to do is maintain their own mapping of the
the regions, since the device physical locations has no meaning at all
to the platform systems (other than it better not clash with real
memory).
    
    If these points are possible to be addressed, it would allow at LEAST three chips *in \
    use* in linux devices able to use mainline OHCI code directly - TC6393XB (in toshiba \
    PDAs), SAMCOP (Ipaqs), and mediaQ (dell axims).
    
    I am told HPPA has some similar problems also.
    
I don't understand what you're asking for beyond what we currently do. 
These devices should be able to operate using ioremap and memcpy_toio,
what more do you want?

There was one extension to the DMA API that I considered for a Q720 SCSI
card which has 2MB of onboard memory.  That was to allow the device to
declare the memory region to the platform so the platform could hand it
out as coherent memory (this is platform dependent, some platforms
simply cannot allow direct memory access to bus space like this). 
However, something with a memory space as tiny as 32kB is unlikely to
benefit from this.

James

Re: DMA API issues

[Ian Molton]

On 18 Jun 2004 13:20:43 -0500
James Bottomley <James.Bottomley@SteelEye.com> wrote:

> 
> Erm, well this isn't unusual.  A lot of devices have on board memory
> to offload accesses to.  All the later Symbios SCSI chips for
> instance.  If you look at the drivers, you'll see they ioremap the
> region and then use it via the normal memory accessors.

Thats all well and good for devices which have their own drivers, but
thats not the case always.

the device I described is an OHCI controller, and in theory, it should
be able to use the OHCI driver in the kernel without any modification,
*as long as* the DMA API returns valid device and virtual addresses,
which, at present, it does not.

Re: DMA API issues

[James Bottomley]

On Fri, 2004-06-18 at 13:35, Ian Molton wrote:
> Thats all well and good for devices which have their own drivers, but
> thats not the case always.
> 
> the device I described is an OHCI controller, and in theory, it should
> be able to use the OHCI driver in the kernel without any modification,
> *as long as* the DMA API returns valid device and virtual addresses,
> which, at present, it does not.

Yes, this sounds similar to the Q720 problem.  I wanted to use the
generic ncr53c8xx driver (being lazy) but I wanted to persuade the
driver to use my onboard memory.  This sounds like your issue because
the ncr driver has been sliced apart to become simply a chip driver and
I supply a small skeleton NCR_Q720.c to glue it on to the bus.

You still haven't explained what you want to do though.  Apart from the
occasional brush with usbstorage, I don't have a good knowledge of the
layout of the USB drivers.  I assume you simply want to persuade the
ohci driver to use your memory area somehow, but what do you actually
want the ohci driver to do with it?  And how much leeway do you get to
customise the driver.

The reason I'm asking is beause it's still unclear whether this is a DMA
API issue or an ohci one.  I could solve my Q720 issue simply by
exporting an interface from the ncr driver to supply alternative memory
allocation use and descriptors.

James

Re: DMA API issues

[Ian Molton]

On 18 Jun 2004 13:52:46 -0500
James Bottomley <James.Bottomley@SteelEye.com> wrote:

> 
> You still haven't explained what you want to do though.  Apart from the
> occasional brush with usbstorage, I don't have a good knowledge of the
> layout of the USB drivers.  I assume you simply want to persuade the
> ohci driver to use your memory area somehow, but what do you actually
> want the ohci driver to do with it?  And how much leeway do you get to
> customise the driver.


In *theory* the OHCI driver is doing everything right - its asking for DMAable memory and using it. if the DMA api simply understood the device in question, and alocated accordingly, it would just work.

there are two solutions:

1) Break up the OHCI driver and make it into a chip driver as you describe
2) Make the DMA API do the right thing with these devices

1) means everyone gets to write their own allocator - not pretty
2) means we get to share code and it all just works.

Re: DMA API issues

[David Brownell]

Ian Molton wrote:
> On 18 Jun 2004 13:52:46 -0500
> James Bottomley <James.Bottomley@SteelEye.com> wrote:
> 
> 
>>You still haven't explained what you want to do though.  Apart from the
>>occasional brush with usbstorage, I don't have a good knowledge of the
>>layout of the USB drivers.  I assume you simply want to persuade the
>>ohci driver to use your memory area somehow, but what do you actually
>>want the ohci driver to do with it?  And how much leeway do you get to
>>customise the driver.

A fair amount, so long as drivers above it don't need to
change much at all (this is "stable").  That includes
usbcore and all the usb drivers.

> 
> In *theory* the OHCI driver is doing everything right - its asking for DMAable memory and using it. if the DMA api simply understood the device in question, and alocated accordingly, it would just work.
> 
> there are two solutions:
> 
> 1) Break up the OHCI driver and make it into a chip driver as you describe

It's heading that way already ... breaking along the lines
where standard APIs solve problems, such as this.

> 2) Make the DMA API do the right thing with these devices
> 
> 1) means everyone gets to write their own allocator - not pretty
> 2) means we get to share code and it all just works.
> 

For example, if usbaudio uses usb_buffer_alloc to stream data,
that eliminates dma bouncing.  That's dma_alloc_coherent at
its core ... it should allocate from that 32K region.

- Dave

Re: DMA API issues

[Jamey Hicks]

James Bottomley wrote:

>You still haven't explained what you want to do though.  Apart from the
>occasional brush with usbstorage, I don't have a good knowledge of the
>layout of the USB drivers.  I assume you simply want to persuade the
>ohci driver to use your memory area somehow, but what do you actually
>want the ohci driver to do with it?  And how much leeway do you get to
>customise the driver.
>
>  
>
It's really not a question of laziness.  The ASICs we are interested in 
implement OHCI, so I think the core OHCI driver should work unmodified.  
OHCI driver allocates dma_pools for managing endpoint descriptors (ED) 
and transaction descriptors (TD).  I expect that the driver wrapper that 
initializes the OHCI controller driver will create dma_pools drawing 
from the ASIC's private SRAM.  The OHCI driver uses 
dma_{alloc,free}_coherent to manage the space used for the top level 
control structure shared between the driver and the controller 
hardware.  This also needs to be allocated in the SRAM.  Finally, in 
drivers/usb/core/usb.c, the USB drivers call dma_map_single and 
dma_unmap_single given pointers to transfer buffers allocated by the USB 
device drivers.  If the USB device is a network device (as it is on the 
iPAQ), the transfer buffers are allocated via dev_alloc_skb. 

>The reason I'm asking is beause it's still unclear whether this is a DMA
>API issue or an ohci one.  I could solve my Q720 issue simply by
>exporting an interface from the ncr driver to supply alternative memory
>allocation use and descriptors.
>
>  
>
I really think this is a DMA API implementation issue.  The problem 
touches more than the USB drivers.  I say implementation because the DMA 
API already takes struct device, so the public interface would not have 
to change or would not have to change much.  However, we would like to 
be able to provide device-specific implementations of the dma 
operations.  One way to implement this would be a pointer to 
dma_operations from struct device.

Jamey

Re: DMA API issues

[James Bottomley]

On Fri, 2004-06-18 at 13:57, Ian Molton wrote:
> In *theory* the OHCI driver is doing everything right - its asking for DMAable memory and using it. if the DMA api simply understood the device in question, and alocated accordingly, it would just work.
> 
> there are two solutions:
> 
> 1) Break up the OHCI driver and make it into a chip driver as you describe
> 2) Make the DMA API do the right thing with these devices

Could you please just describe what the problem actually is.

The ohci driver looks to be reasonably modular already, with a chip
piece and a bus attachment piece.

Is your problem that you'd like the dma pools it uses to come out of the
on chip buffer?

James

Re: DMA API issues

[James Bottomley]

On Fri, 2004-06-18 at 14:22, Jamey Hicks wrote:
> It's really not a question of laziness.  The ASICs we are interested in 
> implement OHCI, so I think the core OHCI driver should work unmodified.  
> OHCI driver allocates dma_pools for managing endpoint descriptors (ED) 
> and transaction descriptors (TD).  I expect that the driver wrapper that 
> initializes the OHCI controller driver will create dma_pools drawing 
> from the ASIC's private SRAM.  The OHCI driver uses 
> dma_{alloc,free}_coherent to manage the space used for the top level 
> control structure shared between the driver and the controller 
> hardware.  This also needs to be allocated in the SRAM.  Finally, in 
> drivers/usb/core/usb.c, the USB drivers call dma_map_single and 
> dma_unmap_single given pointers to transfer buffers allocated by the USB 
> device drivers.  If the USB device is a network device (as it is on the 
> iPAQ), the transfer buffers are allocated via dev_alloc_skb. 

Well, I thought it was something like that.  So the problem could be
solved simply by rejigging ohci to export td_alloc and td_free as
overrideable methods?

Your map and unmap single could also be handled this way: with usb
specific overrides that default to dma_map_single.  None of this would
cause much perturbation in usb, and it would give you everthing you
need.

I assume your implementation of dma_map_single is simply to copy the
memory into the on chip area?

> I really think this is a DMA API implementation issue.  The problem 
> touches more than the USB drivers.  I say implementation because the DMA 
> API already takes struct device, so the public interface would not have 
> to change or would not have to change much.  However, we would like to 
> be able to provide device-specific implementations of the dma 
> operations.  One way to implement this would be a pointer to 
> dma_operations from struct device.

The DMA API is highly platform specific.  It basically embodies a
contract between the platform and its attached busses.  It wasn't
designed to embody a contract between two busses (or in this case, a bus
and its implementing driver).

James

Re: DMA API issues

[Ian Molton]

On Fri, 18 Jun 2004 12:20:24 -0700
David Brownell <david-b@pacbell.net> wrote:

> For example, if usbaudio uses usb_buffer_alloc to stream data,
> that eliminates dma bouncing.  That's dma_alloc_coherent at
> its core ... it should allocate from that 32K region.

Agreed.

Re: DMA API issues

[Ian Molton]

On 18 Jun 2004 14:30:01 -0500
James Bottomley <James.Bottomley@SteelEye.com> wrote:

> Is your problem that you'd like the dma pools it uses to come out of the
> on chip buffer?

Correct. its the right way to do this.

Re: DMA API issues

[James Bottomley]

On Fri, 2004-06-18 at 14:44, Ian Molton wrote:
> > For example, if usbaudio uses usb_buffer_alloc to stream data,
> > that eliminates dma bouncing.  That's dma_alloc_coherent at
> > its core ... it should allocate from that 32K region.
> 
> Agreed.

There are complications to this: not all platforms can access PCI memory
directly.  That's why ioremap and memcpy_toio and friends exist.  What
should happen on these platforms?

James

Re: DMA API issues

[Oliver Neukum]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Am Freitag, 18. Juni 2004 21:41 schrieb James Bottomley:
> Well, I thought it was something like that.  So the problem could be
> solved simply by rejigging ohci to export td_alloc and td_free as
> overrideable methods?

Unfortunately no. Usb_buffer_alloc() needs to know about the restriction,
too.

	Regards
		Oliver

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)

iD8DBQFA00pDbuJ1a+1Sn8oRAoqjAKDVMBJCgjrysIZlQYdLDFCTEic6JgCfQ6t/
g4B4/fqQwvFNelxVo4sQO3o=
=q4nt
-----END PGP SIGNATURE-----

Re: DMA API issues

[James Bottomley]

On Fri, 2004-06-18 at 15:02, Oliver Neukum wrote:
> Am Freitag, 18. Juni 2004 21:41 schrieb James Bottomley:
> > Well, I thought it was something like that.  So the problem could be
> > solved simply by rejigging ohci to export td_alloc and td_free as
> > overrideable methods?
> 
> Unfortunately no. Usb_buffer_alloc() needs to know about the restriction,
> too.

But usb_buffer_alloc is already an indirected operation, it looks like
it can be easily overridden to do precisely what you want.

James

Re: DMA API issues

[Benjamin Herrenschmidt]

> I really think this is a DMA API implementation issue.  The problem 
> touches more than the USB drivers.  I say implementation because the DMA 
> API already takes struct device, so the public interface would not have 
> to change or would not have to change much.  However, we would like to 
> be able to provide device-specific implementations of the dma 
> operations.  One way to implement this would be a pointer to 
> dma_operations from struct device.

I wanted to do just that a while ago, and ended up doing things a bit
differently, but still, I agree that would help. The thing is, you
can do that in your platform code. just use the platform data pointer
in struct device to stuff a ptr to the structure with your "ops"

Ben.

Re: DMA API issues

[James Bottomley]

On Fri, 2004-06-18 at 15:14, Benjamin Herrenschmidt wrote:
> I wanted to do just that a while ago, and ended up doing things a bit
> differently, but still, I agree that would help. The thing is, you
> can do that in your platform code. just use the platform data pointer
> in struct device to stuff a ptr to the structure with your "ops"

Yes, we do this on parisc too.  We actually have a hidden method pointer
(per platform) and cache the iommu (we have more than one) accessors in
platform_data.

The problem is, though, that I really don't think this interface would
benefit from being made public (or part of the API).  There are too many
nasty quirks in the platform for this (at least in our case).

James

Re: DMA API issues

[David Brownell]

James Bottomley wrote:
> On Fri, 2004-06-18 at 14:44, Ian Molton wrote:
> 
>>>For example, if usbaudio uses usb_buffer_alloc to stream data,
>>>that eliminates dma bouncing.  That's dma_alloc_coherent at
>>>its core ... it should allocate from that 32K region.
>>
>>Agreed.
> 
> 
> There are complications to this: not all platforms can access PCI memory
> directly.  That's why ioremap and memcpy_toio and friends exist.  What
> should happen on these platforms?

I'm not following you.  This isn't using the PCI DMA calls.
These dots don't connect:  different hardware needs different
solutions.  How would those calls make dma_alloc_coherent work?

- Dave

Re: DMA API issues

[James Bottomley]

On Fri, 2004-06-18 at 16:08, David Brownell wrote:
> I'm not following you.  This isn't using the PCI DMA calls.
> These dots don't connect:  different hardware needs different
> solutions.  How would those calls make dma_alloc_coherent work?


The statement was "That's dma_alloc_coherent at its core ... it should
allocate from that 32K region." and what I was pointing out is that not
all platforms can treat an on-chip memory region as a real memory area. 
That's why we have the iomem accessor functions.

James

Re: DMA API issues

[Russell King]

On Fri, Jun 18, 2004 at 03:24:45PM -0500, James Bottomley wrote:
> On Fri, 2004-06-18 at 15:14, Benjamin Herrenschmidt wrote:
> > I wanted to do just that a while ago, and ended up doing things a bit
> > differently, but still, I agree that would help. The thing is, you
> > can do that in your platform code. just use the platform data pointer
> > in struct device to stuff a ptr to the structure with your "ops"
> 
> Yes, we do this on parisc too.  We actually have a hidden method pointer
> (per platform) and cache the iommu (we have more than one) accessors in
> platform_data.

Except that platform_data already has multiple other uses, especially for
platform devices.

-- 
Russell King
 Linux kernel    2.6 ARM Linux   - http://www.arm.linux.org.uk/
 maintainer of:  2.6 PCMCIA      - http://pcmcia.arm.linux.org.uk/
                 2.6 Serial core

Re: DMA API issues

[David Brownell]

James Bottomley wrote:
> On Fri, 2004-06-18 at 16:08, David Brownell wrote:
> 
>>I'm not following you.  This isn't using the PCI DMA calls.
>>These dots don't connect:  different hardware needs different
>>solutions.  How would those calls make dma_alloc_coherent work?
> 
> 
> 
> The statement was "That's dma_alloc_coherent at its core ... it should
> allocate from that 32K region." and what I was pointing out is that not
> all platforms can treat an on-chip memory region as a real memory area. 

But this one can, and it sure seems like the appropriate
solution.  For reasons like the one not quoted above:  it's
a good way to eliminate what would otherwise be a case
where a dmabounce is needed.  And hey wow, it even uses
the API designed to reduce such DMA "mapping" costs, and
there are drivers already using it for such purposes.


> That's why we have the iomem accessor functions.

You mentioned ioremap(), which doesn't help here since
the need is for a block of memory, not just address space,
and also memcpy_toio(), which just another tool to implement
the dma bouncing (which is on the "strongly avoid!" list).

As I said, those still don't make dma_alloc_coherent() work.

- Dave


> James
> 
> 
> 

Re: DMA API issues

[James Bottomley]

On Fri, 2004-06-18 at 17:38, David Brownell wrote:
> James Bottomley wrote:
> > The statement was "That's dma_alloc_coherent at its core ... it should
> > allocate from that 32K region." and what I was pointing out is that not
> > all platforms can treat an on-chip memory region as a real memory area. 
> 
> But this one can, and it sure seems like the appropriate
> solution.  For reasons like the one not quoted above:  it's
> a good way to eliminate what would otherwise be a case
> where a dmabounce is needed.  And hey wow, it even uses
> the API designed to reduce such DMA "mapping" costs, and
> there are drivers already using it for such purposes.

Well, yes, but the problem: chips have onboard memory is generic.  The
proposed solution in the DMA API can't only work on certain platforms.

> 
> > That's why we have the iomem accessor functions.
> 
> You mentioned ioremap(), which doesn't help here since
> the need is for a block of memory, not just address space,
> and also memcpy_toio(), which just another tool to implement
> the dma bouncing (which is on the "strongly avoid!" list).
> 
> As I said, those still don't make dma_alloc_coherent() work.

Right, that's rather the point.  The memory you get by doing an ioremap
on this chip area may have to be treated differently from real memory on
some platforms.

That's the fundamental problem of trying to treat it as memory obtained
from dma_alloc_coherent().

James

Re: DMA API issues

[Ian Molton]

On Fri, 18 Jun 2004 22:20:14 +0100
Russell King <rmk+lkml@arm.linux.org.uk> wrote:

> > Yes, we do this on parisc too.  We actually have a hidden method
> > pointer(per platform) and cache the iommu (we have more than one)
> > accessors in platform_data.
> 
> Except that platform_data already has multiple other uses, especially
> for platform devices.

In the case of the SOC devices I described, its actually appropriate to
make the allocator system tied to the bus - as several devices end up
sharing the same 32K pool in the device. at the device level the
allocator would be useless in these cases.

Re: DMA API issues

[Ian Molton]

On 18 Jun 2004 14:57:01 -0500
James Bottomley <James.Bottomley@SteelEye.com> wrote:

> 
> There are complications to this: not all platforms can access PCI memory
> directly.  That's why ioremap and memcpy_toio and friends exist.  What
> should happen on these platforms?

I wasnt talking about a PCI system here.

Re: DMA API issues

[James Bottomley]

On Fri, 2004-06-18 at 18:25, Ian Molton wrote:
> On 18 Jun 2004 14:57:01 -0500
> James Bottomley <James.Bottomley@SteelEye.com> wrote:
> > There are complications to this: not all platforms can access PCI memory
> > directly.  That's why ioremap and memcpy_toio and friends exist.  What
> > should happen on these platforms?
> 
> I wasnt talking about a PCI system here.

ioremap is used for all bus remote MMIO regions, not just PCI.

James

Re: DMA API issues

[Ian Molton]

On 18 Jun 2004 18:07:30 -0500
James Bottomley <James.Bottomley@SteelEye.com> wrote:

> 
> Well, yes, but the problem: chips have onboard memory is generic.  The
> proposed solution in the DMA API can't only work on certain platforms.

It will work on any platform where the memory is directly visible to the
CPU...

if this sint the case its not *D* MA (ie. *direct* memory access is it?

Re: DMA API issues

[Ian Molton]

On 18 Jun 2004 18:29:22 -0500
James Bottomley <James.Bottomley@SteelEye.com> wrote:

> > 
> > I wasnt talking about a PCI system here.
> 
> ioremap is used for all bus remote MMIO regions, not just PCI.

Im aware of that but the OHCI core doesnt do that, it uses the DMA API,
which is entirely reasonable, given its tring to access a DMA-able chunk
of memory.

I *could* write a new driver ohci-ioremapped for all these chips but its
needless duplication which is going to result in bugs bveing fixed in
one ohci driver and not the other.

why not simply expand the DMA API to allow DMA to these easily DMA-able
chips ?

Re: DMA API issues

[James Bottomley]

On Fri, 2004-06-18 at 18:51, Ian Molton wrote:
> Im aware of that but the OHCI core doesnt do that, it uses the DMA API,
> which is entirely reasonable, given its tring to access a DMA-able chunk
> of memory.
> 
> I *could* write a new driver ohci-ioremapped for all these chips but its
> needless duplication which is going to result in bugs bveing fixed in
> one ohci driver and not the other.
> 
> why not simply expand the DMA API to allow DMA to these easily DMA-able
> chips ?

Because the piece of memory you wish to access is bus remote.  Our
current API for bus remote memory requires the use of ioremap and the
accessor functions.  Just because it could be accessed directly on ARM
doesn't mean it can be on all platforms.  The DMA API is a platform
generic API.  To propose an addition to it to make use of this memory,
it would have to work on *all* platforms.

James

Re: DMA API issues

[Ian Molton]

On 18 Jun 2004 19:04:11 -0500
James Bottomley <James.Bottomley@SteelEye.com> wrote:

> Because the piece of memory you wish to access is bus remote. 

No, its *not*

my CPU can write there directly.

no strings attached.

the DMA API just only understands how to map from RAM, not anything
else.

Re: DMA API issues

[James Bottomley]

On Fri, 2004-06-18 at 19:14, Ian Molton wrote:
> On 18 Jun 2004 19:04:11 -0500
> James Bottomley <James.Bottomley@SteelEye.com> wrote:
> 
> > Because the piece of memory you wish to access is bus remote. 
> 
> No, its *not*
> 
> my CPU can write there directly.
> 
> no strings attached.
> 
> the DMA API just only understands how to map from RAM, not anything
> else.

I think you'll actually find that it is.  OHCI is a device (representing
a USB hub), it's attached to the system by some interface that
constitutes a bus (the bus interface transforming the CPU access cycles
to device access cycles, translating interrupts etc.).

But even if you've somehow managed to glue an OHCI directly on to the
system memory controller, from the point of view of the DMA API, the
memory the device contains is still bus remote.  To be useful, the API
has to deal with bus remote memory in all its forms.

James

DMA API issues... summary

[Ian Molton]

Ok, heres a summary of the problems we have (feel free to add any more problems).

We have two types of "device": single function devices and 'system on chip' devices which have multiple functions.

Single chip devices may be able to either access system memory directly, or may only be able to access their internal SRAM pool. in the case of the latter the system can either directly access the SRAM or not, depending on the device/bus setup. Its possible the devices may have more than one non-continuous SRAM mapping.

The same goes for SOC devices, however they could come in two 'classes'. In one type, we would essentially have multiple independant devices in a single chip. In another case (which appears to be fairly common) we can have multiple devices sharing a common SRAM pool. its also possible to have some devices sharing the pool and some having their own in the same chip.

Can anyone describe another type of chip we need to accomodate?

Re: DMA API issues

[David Brownell]

James Bottomley wrote:
> On Fri, 2004-06-18 at 17:38, David Brownell wrote:

>>You mentioned ioremap(), which doesn't help here since
>>the need is for a block of memory, not just address space,
>>and also memcpy_toio(), which just another tool to implement
>>the dma bouncing (which is on the "strongly avoid!" list).
>>
>>As I said, those still don't make dma_alloc_coherent() work.
> 
> 
> Right, that's rather the point.  The memory you get by doing an ioremap
> on this chip area may have to be treated differently from real memory on
> some platforms.

And that point/difference would be ... what?  It IS real memory.
Not "main memory", so the device's DMA access never consumes
bandwidth on the "main" memory bus, but real nonetheless.

I'm having to guess at your point here, even from other emails.
You've asserted a difference, but not what it is.  Maybe it's
something to do with the problem's NUMA nature?  Are you for
some reason applying DMA _mapping_ requirements (main-memory
only) to the DMA memory _allocation_ problem?


> That's the fundamental problem of trying to treat it as memory obtained
> from dma_alloc_coherent().

Well, no other memory in the entire system meets the requirements
for the dma_alloc_coherent() API, since _only that_ chunk of memory
is works with that device's DMA hardware.  Which is the fundamental
problem that needs to be solved.  It can clearly done at the platform
level, using device- or bus-specific implementations.

- Dave

Re: DMA API issues

[Russell King]

On Sat, Jun 19, 2004 at 11:23:23AM -0700, David Brownell wrote:
> I'm having to guess at your point here, even from other emails.
> You've asserted a difference, but not what it is.  Maybe it's
> something to do with the problem's NUMA nature?  Are you for
> some reason applying DMA _mapping_ requirements (main-memory
> only) to the DMA memory _allocation_ problem?

I suspect the problem is concerning how Linux backs the memory and
what is assumed to be backing memory returned from the DMA coherent
API.

Currently, there are drivers which assume that it's possible that
dma_alloc_coherent memory is backed by system memory, which has
page structures associated with each page.  For this "new" memory,
there are no such page structures, so things like bus_to_virt()
don't work on them (not that they were guaranteed to work on a
dma_addr_t _anyway_ but that doesn't stop driver authors thinking
that they do work - and as code presently stands, they do indeed
work.)

In addition, the ARM implementation of dma_alloc_coherent()
implicitly believes in struct page pointers - they're a fundamental
properly of the way that has been implemented, so any deviation from
"memory with struct page" means more or less a rewrite this.

I would say that, yes, from a perfectly objective view, if you are
unable to do coherent DMA from system memory, but your system provides
you with a totally separate memory system which does indeed provide
coherent DMA, it seems logical to allow dma_alloc_coherent() to use
it - at risk of breaking some drivers making incorrect assumptions.

And I don't see _that_ case as being vastly different from Ian's
case.

So, I think as long as we can ensure that drivers do not make bad
assumptions about dma_alloc_coherent() _and_ we have a suitable DMA
MMAP API to solve the cases where device drivers want to mmap DMA
buffers (and they _do_ want to do this) it should be possible.

Depending on how I look at the problem, I'm oscillating between "yes
it should be done" (if its an overall system thing like DMA memory
on a PCI north bridge separate from your normal system non-DMA
memory) and "no it's out of the question."

> Well, no other memory in the entire system meets the requirements
> for the dma_alloc_coherent() API, since _only that_ chunk of memory
> is works with that device's DMA hardware.  Which is the fundamental
> problem that needs to be solved.  It can clearly done at the platform
> level, using device- or bus-specific implementations.

The counter-argument here is to consider the case of video cards.
They do almost continuous DMA from on-board RAM, yet the DMA API
doesn't get involved...

So I'm afraid I'm sitting on the fence between the two sides not
really knowing which side to fall off to.  It sounds to me like
other people here are in a similar situation.

-- 
Russell King
 Linux kernel    2.6 ARM Linux   - http://www.arm.linux.org.uk/
 maintainer of:  2.6 PCMCIA      - http://pcmcia.arm.linux.org.uk/
                 2.6 Serial core

Re: DMA API issues

[James Bottomley]

On Sat, 2004-06-19 at 15:41, Russell King wrote:
> I suspect the problem is concerning how Linux backs the memory and
> what is assumed to be backing memory returned from the DMA coherent
> API.

More or less, yes.  The basic problem is platforms that simply cannot
make this type of bus remote memory visible in the CPU page tables at
all (the IBM AS/400 apparently falls into that).  Then there are the
ones that could be persuaded to do this with great difficulty and a lot
of restrictions (sparc and parisc).

> Currently, there are drivers which assume that it's possible that
> dma_alloc_coherent memory is backed by system memory, which has
> page structures associated with each page.  For this "new" memory,
> there are no such page structures, so things like bus_to_virt()
> don't work on them (not that they were guaranteed to work on a
> dma_addr_t _anyway_ but that doesn't stop driver authors thinking
> that they do work - and as code presently stands, they do indeed
> work.)
> 
> In addition, the ARM implementation of dma_alloc_coherent()
> implicitly believes in struct page pointers - they're a fundamental
> properly of the way that has been implemented, so any deviation from
> "memory with struct page" means more or less a rewrite this.

Yes, I think everyone has more or less a struct page backed allocator,
although we could first consult an exception table hanging off the
device somehow to get around this.

> I would say that, yes, from a perfectly objective view, if you are
> unable to do coherent DMA from system memory, but your system provides
> you with a totally separate memory system which does indeed provide
> coherent DMA, it seems logical to allow dma_alloc_coherent() to use
> it - at risk of breaking some drivers making incorrect assumptions.
> 
> And I don't see _that_ case as being vastly different from Ian's
> case.
> 
> So, I think as long as we can ensure that drivers do not make bad
> assumptions about dma_alloc_coherent() _and_ we have a suitable DMA
> MMAP API to solve the cases where device drivers want to mmap DMA
> buffers (and they _do_ want to do this) it should be possible.

But we still need some sort of fallback where the platform really cannot
do this.  And that fallback is going to be ioremap and all the other
paraphenalia.  So, the thing that bothers me is that if we have to have
the fallback which is identical to what every other driver that uses
on-chip memory does anyway, is there any point to placing this in the
DMA API?

James

Re: DMA API issues

[Ian Molton]

On 19 Jun 2004 16:46:42 -0500
James Bottomley <James.Bottomley@SteelEye.com> wrote:

> 
> But we still need some sort of fallback where the platform really
> cannot do this.  And that fallback is going to be ioremap and all the
> other paraphenalia.  So, the thing that bothers me is that if we have
> to have the fallback which is identical to what every other driver
> that uses on-chip memory does anyway, is there any point to placing
> this in the DMA API?

Can you describe a system where its impossible to use the DMA API or one
of the modifications proposed here? what sort of hardware does this and
why?

Re: DMA API issues

[James Bottomley]

On Sat, 2004-06-19 at 17:49, Ian Molton wrote:
> James Bottomley <James.Bottomley@SteelEye.com> wrote:
> > But we still need some sort of fallback where the platform really
> > cannot do this.  And that fallback is going to be ioremap and all the
> > other paraphenalia.  So, the thing that bothers me is that if we have
> > to have the fallback which is identical to what every other driver
> > that uses on-chip memory does anyway, is there any point to placing
> > this in the DMA API?
> 
> Can you describe a system where its impossible to use the DMA API or one
> of the modifications proposed here? what sort of hardware does this and
> why?

There's no architecture currently that can't use the DMA API.

The modification you propose, to make on chip memory visible as normal
memory can't be done on the IBM iserie, AS/400 as I said in the the
email you quote:

On Sat, 2004-06-19 at 16:46, James Bottomley wrote:
> More or less, yes.  The basic problem is platforms that simply cannot
> make this type of bus remote memory visible in the CPU page tables at
> all (the IBM AS/400 apparently falls into that).  Then there are the
> ones that could be persuaded to do this with great difficulty and a lot
> of restrictions (sparc and parisc).

The iseries can't because the PCI bus sits behind the hypervisor and has
to use accessors to get at the on chip memory, it can't simply be mapped
into the address space like it can on ARM.

James

Re: DMA API issues

[Ian Molton]

On 20 Jun 2004 08:37:58 -0500
James Bottomley <James.Bottomley@SteelEye.com> wrote:

> 
> There's no architecture currently that can't use the DMA API.
> 
> The modification you propose, to make on chip memory visible as normal
> memory can't be done on the IBM iserie, AS/400 as I said in the the
> email you quote:

Those two statements are contradictory. clearly the iseries cant use the
DMA API *now* so I dont see how that makes any difference. We're talking
about adding propper support for *addresssable* memory mapped devices
with limited size DMA-able windows to the DMA API, not adding support
for a whole new weird way of talking to devices. These devices work the
same way as all the other devices that use the DMA API but are simply
restricted in the range of addresses they can DMA from. they require no
special 'accessors'.

iseries cant work the usual way now and wont with these modifications -
so nothing is made worse.

Re: DMA API issues

[Jeff Garzik]

On Sun, Jun 20, 2004 at 04:50:42PM +0100, Ian Molton wrote:
> Those two statements are contradictory. clearly the iseries cant use the
> DMA API *now* so I dont see how that makes any difference. We're talking
> about adding propper support for *addresssable* memory mapped devices
> with limited size DMA-able windows to the DMA API, not adding support
> for a whole new weird way of talking to devices. These devices work the
> same way as all the other devices that use the DMA API but are simply
> restricted in the range of addresses they can DMA from. they require no
> special 'accessors'.
> 
> iseries cant work the usual way now and wont with these modifications -
> so nothing is made worse.

Are you purposefully ignoring James?

He is saying the DMA API must be uniform across all platforms.  Your
proposal 

1) breaks this

2) is unneeded, as many other drivers in this same situation simply use
ioremap

Re: DMA API issues

[James Bottomley]

On Sun, 2004-06-20 at 10:50, Ian Molton wrote:
> Those two statements are contradictory. clearly the iseries cant use the
> DMA API *now* so I dont see how that makes any difference. We're talking
> about adding propper support for *addresssable* memory mapped devices
> with limited size DMA-able windows to the DMA API, not adding support
> for a whole new weird way of talking to devices. These devices work the
> same way as all the other devices that use the DMA API but are simply
> restricted in the range of addresses they can DMA from. they require no
> special 'accessors'.
> 
> iseries cant work the usual way now and wont with these modifications -
> so nothing is made worse.

OK, let's try and make this as simple as I know how.  The system looks
like this


       +-----+
       | CPU |
       +--+--+
          |
    <-----+-----+----------------------+-----> Central Bus
                |                      |
          +-----+------+        +------+-----+
          |   Memory   |        |    I/O     |
          | Controller |        | Controller |
          +-----+------+        +------+-----+
                |                      |
            +---+-----+             +--+---+    +--------+
            | Memory  |             | OHCI |----| Memory |
            +---------+             +------+    +--------+

In order to access this OHCI memory, both the I/O controller and the
OHCI have to respond to the memory access cycles, rather than the memory
controller.  This is why such memory is termed "bus remote".

Even though ARM can programm the I/O controller and the OHCI device to
access this memory as though it were behind the memory controller (i.e.
using normal CPU memory cycles), you'll find that even on ARM there's
probably special page table trickery involved (probably to do with cache
coherency issues).  Next, you'll find that no other device can see this
memory without some type of i2o support, so it can't be the target of a
DMA transaction. So even on ARM, you can't treat it as "normal" memory.

On iSeries, the I/O controller sits behind the hypervisor and can't be
the target of normal memory cycles, that's why the CPU can't address
this bus remote memory normally. As I've explained.

The DMA API is about allowing devices to transact directly with memory
behind the memory controller, it's an API that essentially allows the
I/O controller and memory controller to communicate without CPU
intervention.  This is still possible through the hypervisor, so the
iSeries currently fully implements the DMA API.

James

Re: DMA API issues

[Ian Molton]

On Sun, 20 Jun 2004 12:26:11 -0400
Jeff Garzik <jgarzik@pobox.com> wrote:

> 
> Are you purposefully ignoring James?

No, Im failing to see his point which appears to be that if we modify
the DMA API it has to work on all platforms, even if it doesnt work on
them currently.
 
> He is saying the DMA API must be uniform across all platforms.  Your
> proposal 
> 
> 1) breaks this

How? I havent proposed any alteration to the API, just the allocation
system inside it. No drivers would need modification and any platform
not using these allocators would continue to not use them just as it
currently does(nt).

> 2) is unneeded, as many other drivers in this same situation simply
> use ioremap

Except that the OHCI driver (for example) doesnt. it uses the DMA API
(quite rightly) and to use ioremap would mean creating two ohci drivers,
one using the DMA API and one not, which is wasteful and asking for
trouble.

Re: DMA API issues

[Oliver Neukum]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1


>        +-----+
>        | CPU |
>        +--+--+
>           |
>     <-----+-----+----------------------+-----> Central Bus
>                 |                      |
>           +-----+------+        +------+-----+
>           |   Memory   |        |    I/O     |
>           | Controller |        | Controller |
>           +-----+------+        +------+-----+
>                 |                      |
>             +---+-----+             +--+---+    +--------+
>             | Memory  |             | OHCI |----| Memory |
>             +---------+             +------+    +--------+
> 
> In order to access this OHCI memory, both the I/O controller and the
> OHCI have to respond to the memory access cycles, rather than the memory
> controller.  This is why such memory is termed "bus remote".

Why in an abstract API would we care how the memory is connected to
the system? Isn't the only relevant issue whether the memory is in the
CPU's normal address space?
 
> Even though ARM can programm the I/O controller and the OHCI device to
> access this memory as though it were behind the memory controller (i.e.
> using normal CPU memory cycles), you'll find that even on ARM there's
> probably special page table trickery involved (probably to do with cache
> coherency issues).  Next, you'll find that no other device can see this
> memory without some type of i2o support, so it can't be the target of a
> DMA transaction. So even on ARM, you can't treat it as "normal" memory.

How is it relevant whether memory can be DMAed to by devices other
than the device in question? Even on regular x86 there's the low 16MB,
isn't there?
 
> The DMA API is about allowing devices to transact directly with memory
> behind the memory controller, it's an API that essentially allows the
> I/O controller and memory controller to communicate without CPU
> intervention.  This is still possible through the hypervisor, so the
> iSeries currently fully implements the DMA API.

Then what's the problem?

	Regards
		Oliver
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)

iD8DBQFA1dFEbuJ1a+1Sn8oRAiFKAKCJZ9mDNzeDRie7xyLAFD+nv1gpSwCfWQke
TISWtAZS6nw8J4dabkk+8lo=
=yTfH
-----END PGP SIGNATURE-----

Re: DMA API issues

[Deepak Saxena]

On Jun 19 2004, at 00:20, Ian Molton was caught saying:
> On Fri, 18 Jun 2004 22:20:14 +0100
> Russell King <rmk+lkml@arm.linux.org.uk> wrote:
> 
> > > Yes, we do this on parisc too.  We actually have a hidden method
> > > pointer(per platform) and cache the iommu (we have more than one)
> > > accessors in platform_data.
> > 
> > Except that platform_data already has multiple other uses, especially
> > for platform devices.
> 
> In the case of the SOC devices I described, its actually appropriate to
> make the allocator system tied to the bus - as several devices end up
> sharing the same 32K pool in the device. at the device level the
> allocator would be useless in these cases.

I've followed the whole thread and I still think what you are trying
to do can be currently accomplished with the existing API by simply 
overriding the generic ARM implementation and providing one specific
to your platform. All the dma_* APIs take a dev structure as a
parameter, so you simply have to look at that and if it's your OHCI
device, call your specific allocator/deallocator/mapping function/etc.
Your dma_alloc_coherent() would simply ioremap() and return the
ioremap'd address as the virtual address and the OHCI-bus address 
as the DMA_ADDR. Your dma_map_single() simply bounces data between
the SRAM and system memory.  Having per-bus or per-device allocators 
as you are proposing is very possibly the correct way to go, but I don't 
think it's a simple enough change that we want to do it on 2.6.

~Deepak

-- 
Deepak Saxena - dsaxena at plexity dot net - http://www.plexity.net/

"Unlike me, many of you have accepted the situation of your imprisonment and
 will die here like rotten cabbages." - Number 6

Re: DMA API issues

[James Bottomley]

On Sun, 2004-06-20 at 13:02, Oliver Neukum wrote:
> > The DMA API is about allowing devices to transact directly with memory
> > behind the memory controller, it's an API that essentially allows the
> > I/O controller and memory controller to communicate without CPU
> > intervention.  This is still possible through the hypervisor, so the
> > iSeries currently fully implements the DMA API.
> 
> Then what's the problem?

If you look at the diagram, you'll see that the OHCI memory isn't behind
the memory controller...

James

Re: DMA API issues

[Oliver Neukum]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Am Sonntag, 20. Juni 2004 21:27 schrieb James Bottomley:
> On Sun, 2004-06-20 at 13:02, Oliver Neukum wrote:
> > > The DMA API is about allowing devices to transact directly with memory
> > > behind the memory controller, it's an API that essentially allows the
> > > I/O controller and memory controller to communicate without CPU
> > > intervention.  This is still possible through the hypervisor, so the
> > > iSeries currently fully implements the DMA API.
> > 
> > Then what's the problem?
> 
> If you look at the diagram, you'll see that the OHCI memory isn't behind
> the memory controller...

And that means what? We don't care about how memory is implemented
in the machines we run on. All we care about is the consequences of the
implementation.
We have an area of memory here that the CPU and a subset of devices
can directly access. What to do about it?

	Regards
		Oliver
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)

iD8DBQFA1ebgbuJ1a+1Sn8oRAnFdAJ45v2h3fUe7NIPVxyS9CmH69vEHLgCghgQe
qXQVLQOlkEabmPVqmCOaSq8=
=35je
-----END PGP SIGNATURE-----

Re: DMA API issues

[David Brownell]

Russell King wrote:
> On Sat, Jun 19, 2004 at 11:23:23AM -0700, David Brownell wrote:
> 
>>I'm having to guess at your point here, even from other emails.
>>You've asserted a difference, but not what it is.  Maybe it's
>>something to do with the problem's NUMA nature?  Are you for
>>some reason applying DMA _mapping_ requirements (main-memory
>>only) to the DMA memory _allocation_ problem?
> 
> ...
> 
> Currently, there are drivers which assume that it's possible that
> dma_alloc_coherent memory is backed by system memory, which has
> page structures associated with each page.  For this "new" memory,
> there are no such page structures, so things like bus_to_virt()
> don't work on them (not that they were guaranteed to work on a

This shouldn't include the USB stack, FWIW; nothing in drivers/usb
makes such calls (on 2.6).  The device drivers don't have any
reason to use such calls, either.


> In addition, the ARM implementation of dma_alloc_coherent()
> implicitly believes in struct page pointers - they're a fundamental
> properly of the way that has been implemented, so any deviation from
> "memory with struct page" means more or less a rewrite this.

Or just bypassing it before it gets to __dma_alloc(), which would
be ugly but functional.  Or flagging the devices in question as
needing to use that particular memory zone ... so for example a
dma_zone(dev) macro, used with alloc_pages_node(), might be a
cleaner approach.


> I would say that, yes, from a perfectly objective view, if you are
> unable to do coherent DMA from system memory, but your system provides
> you with a totally separate memory system which does indeed provide
> coherent DMA, it seems logical to allow dma_alloc_coherent() to use
> it - at risk of breaking some drivers making incorrect assumptions.
> 
> And I don't see _that_ case as being vastly different from Ian's
> case.

That's hardly a vast difference, no!


> So, I think as long as we can ensure that drivers do not make bad
> assumptions about dma_alloc_coherent() _and_ we have a suitable DMA
> MMAP API to solve the cases where device drivers want to mmap DMA
> buffers (and they _do_ want to do this) it should be possible.

Right, and the whole USB stack should "just work" once that's done.
Subject to memory cramping for some uses!  :)


> Depending on how I look at the problem, I'm oscillating between "yes
> it should be done" (if its an overall system thing like DMA memory
> on a PCI north bridge separate from your normal system non-DMA
> memory) and "no it's out of the question."

I'm clearly on the "yes" side, though I suspect Deepak is right
that doing it very cleanly as an all-arch extension may not be
a near-term 2.6 option.

- Dave

Re: DMA API issues

[David Brownell]

James Bottomley wrote:

> The DMA API is about allowing devices to transact directly with memory
> behind the memory controller, ...

Nope, that's certainly not in the API spec.  Never has been,
and I'd have objected if it had been ... because I knew this
was one of the types of hardware the API needed to support as
Linux ran on more systems.  The call syntax just returns two
addresses, usable by host and by device:

     void *
     dma_alloc_coherent(struct device *dev, size_t size,
                 dma_addr_t *dma_handle, int flag);

That doesn't say ANYTHING about where that memory lives, or
who may or may not have set up a "struct page" (or what the
page size is for that component).  It certainly doesn't make
assumptions about which busses are used by CPU or device for
memory or control access, or how they're linked.

- Dave

Re: DMA API issues

[David Brownell]

Jeff Garzik wrote:

> 2) is unneeded, as many other drivers in this same situation simply use
> ioremap

But ioremap() only solves the problem of allocating virtual address
space to match this memory.  It doesn't solve the problem of then
allocating/freeing such memory through dma_alloc_coherent(), so that
the drivers that need to share access to that memory can do so.

Re: DMA API issues

[David Brownell]

James Bottomley wrote:

> The iseries can't because the PCI bus sits behind the hypervisor and has
> to use accessors to get at the on chip memory, it can't simply be mapped
> into the address space like it can on ARM.

So you're saying that the iSeries can't implement dma_alloc_coherent()
for such hardware, yes?  In that case, it should return a polite failure
status.  Meanwhile, hardware that _can_ implement it should do so.

- Dave

Re: DMA API issues... summary

[Joshua Wise]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Jumping into the discussion from the middle of nowhere ......

> Single chip devices may be able to either access system memory directly, or
> may only be able to access their internal SRAM pool. in the case of the
> latter the system can either directly access the SRAM or not, depending on
> the device/bus setup. Its possible the devices may have more than one
> non-continuous SRAM mapping.
>
> The same goes for SOC devices, however they could come in two 'classes'. In
> one type, we would essentially have multiple independant devices in a
> single chip. In another case (which appears to be fairly common) we can
> have multiple devices sharing a common SRAM pool. its also possible to have
> some devices sharing the pool and some having their own in the same chip.

First off ... Why just say the internal SRAM pool? I think it woudl be better 
to say that the devices can only access their own address space, and can only 
DMA from a subset of that (which may be the full original set, QED)

Second... Remember that the SOC can also be the CPU! At least on XScale, there 
are some portions that we can DMA from main memory (I think USB is one).

Of course this might not be at all relevant to the discussion, and of course I 
could be using my rectally-based knowledge, but I _THINK_ that this is 
correct.

joshua
- -- 
Joshua Wise | www.joshuawise.com
GPG Key     | 0xEA80E0B3
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)

iD8DBQFA1fhdPn9tWOqA4LMRAuwVAKCq2kcu0V1nnBtZZgwSkTAM2a/izACbBAKu
0ef8cBr9EfxqeYILtdzrgvw=
=B2Yf
-----END PGP SIGNATURE-----

Re: DMA API issues

[Jamey Hicks]

James Bottomley wrote:

>On Fri, 2004-06-18 at 19:14, Ian Molton wrote:
>  
>
>>On 18 Jun 2004 19:04:11 -0500
>>James Bottomley <James.Bottomley@SteelEye.com> wrote:
>>
>>    
>>
>>>Because the piece of memory you wish to access is bus remote. 
>>>      
>>>
>>No, its *not*
>>
>>my CPU can write there directly.
>>
>>no strings attached.
>>
>>the DMA API just only understands how to map from RAM, not anything
>>else.
>>    
>>
>
>I think you'll actually find that it is.  OHCI is a device (representing
>a USB hub), it's attached to the system by some interface that
>constitutes a bus (the bus interface transforming the CPU access cycles
>to device access cycles, translating interrupts etc.).
>
>  
>
Bus remote is a red herring in this case.  The only difference between 
this case and the ones supported by the coherent_dma_mask is that the 
constraint on placement of the allocated memory cannot be encoded as a 
bitmask.

Jamey