Re: PCI DMA to small buffers on cache-incoherent arch

[Paul Mackerras <paulus@samba.org>]

David S. Miller writes:

>    From: Roland Dreier <roland@topspin.com>
>    Date: 08 Jun 2002 18:26:12 -0700
> 
>    Just to make sure I'm reading this correctly, you're saying that as
>    long as a buffer is OK for DMA, it should be OK to use a
>    sub-cache-line chunk as a DMA buffer via pci_map_single(), and
>    accessing the rest of the cache line should be OK at any time before,
>    during and after the DMA.
>    
> Yes.

No.  Not on processors where the cache doesn't snoop DMA accesses to
memory.

>    What alternate implementation are you proposing?
> 
> For non-cacheline aligned chunks in the range "start" to "end" you
> must perform a cache writeback and invalidate. To preserve the data
> outside of the DMA range.

This is the problem scenario.  Suppose we are doing DMA to a buffer B
and also independently accessing a variable X which is not part of B.
Suppose that X and the beginning of B are both in cache line C.

	CPU				I/O device

1.	write back & invalidate cache
	line(s) containing B
2.	start DMA
3.	read X: cache line C now
	present in cache
4.					DMA write to B:
					cache line C updated in memory
5.	write X: cache line C now
	dirty in cache
6.					Signal DMA completion

Now at this point the driver calls pci_unmap_single or whatever.  What
is pci_unmap_single to do?  If it does nothing, or does a writeback,
we lose the DMA data.  If it does an invalidate we lose the value
written to X.  Clearly, neither is correct.

The bottom line is that we can only have one writer to any given cache
line at a time.  If a buffer is being used for DMA from a device to
memory, the cpu MUST NOT write to any cache line that overlaps the
buffer.

Fundamentally, this is because the hardware lacks the ability to
transfer the "ownership" of the cache line between the cpu and the DMA
device on demand, and so we must manage that in software.  The only
safe way to allow the cpu to write to the cache line during a DMA
transfer is active is to pause the DMA, invalidate the cache line, do
the write, write back and invalidate the cache line, and restart the
DMA.

Paul.