Re: Opinion on ordering of writel vs. stores to RAM

[Paul Mackerras <paulus@samba.org>]

Linus Torvalds writes:

> I don't think anyting but a few SCSI drivers that are used on some ia64 
> boxes use mmiowb(). And it's currently a no-op not only on x86 but also on 

The examples in Documentation/memory-barriers.txt and
Documentation/DocBook/deviceiobook.tmpl only seem to say that mmiowb
is needed between a writel and a spin_unlock.

> powerpc. Probably because it's defined to be a barrier between _two_ MMIO 
> operations, while we should probably have things like
> 
>  a)
> 	.. regular mem store ..
> 	mem_to_io_barrier();
> 	.. IOMEM store ..
> 
>  b)
> 	.. IOMEM store ..
> 	io_to_mem_barrier();
> 	.. regular mem store ..
> 
> although it's quite possible that (a) never makes any sense at all.

Do you mean (b) never makes sense?  (a) is the classic scenario for a
device doing DMA to read a descriptor we've just constructed.

> That said, it's also entirely possible that what you _should_ do is to 
> just make sure that the	"sync" is always _before_ the IO op. But that 
> would require that you have one before an IO load too. Do you? I'm too 
> lazy to check.

Not at present; readX() have no barrier before the load.  They have a
barrier after the load that waits for the data to arrive before
allowing any following instructions to execute.

> (Keeping it inside a spinlock, I don't know. Spinlocks aren't _supposed_ 
> to order IO, so I don't _think_ that's necessarily an argument for doing 
> so. So your rationale seems strange. Even on x86, a spinlock release by 
> _no_ means would mean that an IO write would be "done").

Well, it's about ordering between multiple CPUs rather than the write
being "done".  I think the powerpc readX/writeX got to their current
form before the ia64 guys invented mmiowb().

I suspect the best thing at this point is to move the sync in writeX()
before the store, as you suggest, and add an "eieio" before the load
in readX().  That does mean that we are then relying on driver writers
putting in the mmiowb() between a writeX() and a spin_unlock, but at
least that is documented.

Paul.