[RFC] MMIO accessors & barriers documentation #2

[RFC] MMIO accessors & barriers documentation #2

[Benjamin Herrenschmidt]

Hi !

Here's a second version of my proposed documentation for MMIO accessors
and proposal for relaxed versions and associated barriers. I've removed
the questions as those have been answered most of the time (and when
they haven't directly, I've decided on a response myself :)

I'd like to have this submited for 2.6.19, along with an implementation
of the relaxed ops for PowerPC (it would be nice if ia64 could provide
one too), a generic implementation defining the relxaed ops based on the
normal one for other archs and maybe changes to a few drivers to use the
relaxed ops in some hot path.

Ben.

------


Definitions of MMIO accessors and IO related barriers semantics. First
we define some ordering requirements. Then, we define accessors and
which of these requirements they implement. Note that anything that
isn't explicitly mentioned in that list is left free to the
implementation, and thus isn't defined.


[Note] Those semantics are only currently defined for non-cacheable
MMIO mappings obtained from ioremap() or pci_iomap(). Anything mapped
cacheable via some architecture specific mecanisms will have
architecture specific ordering semantics.

[Note] Those ordering requirements and barriers are strictly specific
to MMIO/PIO and MMIO/PIO vs. memory ordering. Ordering between memory
accesses not directly involving MMIO is not covered in this document
and is the domain of the "traditional" memory barriers (wmb,mb,rmb)
which are to be used to provide explicit ordering of accesses to
consistent memory shared with the IO adapter.

[Note] About write combining. Some architectures can provide an
ioremap_wc() that enables write-combining globally over a given
mapping. However, on such a mapping, only relaxed or partially relaxed
accessors are defined as allowing write combining. It's yet to be
defined wether we can provide global write combining semantics, a lot
in that area is still left to the architecture.


* I * Ordering requirements:
============================

First, let's define 4 types of ordering requirements that can be
provided by MMIO accessors:

 1- MMIO + MMIO: This type of ordering means that two consecutive MMIO
accesses performed by one processors are issued in program order on the
bus. Reads can't cross writes. Writes can't be re-oredered vs. each
other. There is no implication on MMIOs issued by different CPUs nor
non-MMIO accesses.

 2- memory W + MMIO R/W: This type of ordering means that an store to main
memory that is performed in program order before an MMIO load or store to a
device, must be visible to that device before the MMIO access reaches it.
For example: updating of a DMA descriptor in memory is visible to the
chip before the MMIO write that cause that chip to go fetch it.

 3- MMIO R + memory R: This type of ordering means that an MMIO read will
be effectively performed (the result returned by the device to the
processor) before a following read for memory. That is, the value
returned by that following read is what was present in the coherency
domain after the MMIO read is complete. For example: reading a DMA
"pointer" from a device with an MMIO read, and then fetching the data in
memory up to that pointer.

 4- MMIO W + spin_unlock: This type of ordering means that MMIO stores
followed by a spin unlock will have all reached a host IO bridge
before the unlocking is visible to other CPUs. For example, two CPUs
have a locked section (same spinlock) issuing some MMIO stores to the
same device. Such ordering means that both sets of MMIO stores will not
be interleaved when reaching the host IO controller (PCI typically)
and thus the device. All MMIO stores from one locked section will be
performed before all MMIO stores from the other.

[Note] Rule #4 is strictly specific to MMIO stores followed by a
spin_unlock(). There is no ordering requirement provided by Linux to
ensure ordering of an MMIO store followed by a generic memory store
unless an explicit barrier is used. This limitation allows platforms
to limit the cost of such a barrier by implementing a per-cpu flag set
by MMIO stores, and test it in spin_unlock() instead of doing the
actual barrier on every MMIO store access which can be very expensive.


* II * Accessors:
=================

We provide 3 classes of accessors:

Class 1: Ordered accessors
--------------------------

 1- {read,write}{b,w,l,q} : MMIO accessors. Those accessors provide
all MMIO ordering requirements. They are thus called "fully ordered".
That is #1, #2 and #4 for writes and #1 and #3 for reads. 

 2- {in,out}{b,w,l,sb,sw,sl} " PIO accessors Those accessors provide
all MMIO ordering requirements. They are thus called "fully ordered".
That is #1, #2 and #4 for writes and #1 and #3 for reads.


 3- memcpy_to_io, memcpy_from_io: #1+#2+#4 (stores) or #1+#3 (loads)
semantics apply to the operation as a whole. That is memcpy_to_io() is
equivalent to a single writel() vs. preceding or subsequent operations
and memcpy_from_io() is equivalent to a single readl(). There is no
definition of the order or size of accesses within the actual memcpy
operation which is left to the implementation.

 4- io{read,write}{8,16,32}[be]: Those have the same semantics as 1
and 2.

Class 2: Partially relaxed accessors
------------------------------------

 1- __{read,write}{b,w,l,q} : Those accessors provide only ordering rule
#1. That is, MMIOs are ordered vs. each other as issued by one CPU.
Barriers are required to ensure ordering vs. memory and vs. locks (see
"Barriers" section). 

 2- __io{read,write}{8,16,32}[be] (optional ?) : Those have the same
semantic as 1 for MMIO, and provide full ordering requirements as
defined in Classe 1 for PIO.

 3- __memcpy_to_io, __memcpy_from_io. Those provide only requirement #1,
that is the MMIOs within the copy are performed in order and are in
order vs. preceeding and subsequent MMIOs executed on the same CPU.

Class 3: Fully relaxed accessors
--------------------------------
 
 1- __raw_{read,write}{b,w,l,q} : Those accessors provide no ordering
rule whatsoever. They also provide no endian swapping. They are
essentially the equivalent of a direct load/store instruction to/from
the MMIO space. Access is done in platform native endian.


* III * IO related barriers
===========================

Some of the above accessors do not provide all ordering rules define in
* I *, thus explicit barriers are provided to enforce those ordering
rules:

 1- io_to_io_barrier() : This barrier provides ordering requirement #1
between two MMIO accesses. It's to be used in conjuction with fully
relaxed accessors of Class 3.

 2- memory_to_io_barrier() : This barrier provides ordering
requirement #2 between a memory store and an MMIO access. It can be
used in conjunction with MMIO accessors of Class 2 and 3.

 3- memory_to_io_wb() : This barrier is a subset of memory_to_io_barrier()
(and thus can be implemented as an equivalent if the architecture
doesn't provide the facility). It provides a subset of ordering
requirement #2 by ordering a memory store followed by an MMIO store
only. It doesn't provide any ordering guarantee vs. a subsequent
memory load.

 4- io_to_memory_rb(value) : This barrier provides ordering requirement
#3 between an MMIO read and a subsequent read from memory. For
implementation purposes on some architectures, the value actually read
by the MMIO read shall be passed as an argument to this barrier. (This
allows to generate the appropriate CPU instruction magic to force the
CPU to consider the value as being "used" and thus force the read to be
performed immediately). It can be used in conjunction with read
accessors of Class 2 and 3

 5- io_to_lock_wb() : This barrier provides ordering requirement #4
between an MMIO store and a subsequent spin_unlock(). It can be used in
conjunction with write accessors of Class 2 and 3. It's the equivalent
of the old mmiowb().


* IV * Mixing of accessors
==========================

There are few rules concerning the mixing of accessors of the different
ordering Classes. Basically, when accessors of different classes share
an ordering rule, then that rule apply. If not, it doesn't apply. For
example:

writel followed by __writel : both accesors provide rule #1, thus it
applies and stores are visible in order. Since the previous writel will
have ordered previous stores to memory, the second __writel naturally
benefits from this despite the fact that __writel doesn't normally
provide that semantic.

Ben.


Questions:
==========

[* Question 1] Should Rule #4 be generalized to MMIO store followed by a
memory store ? (as spin_unlock are essentially a wmb followed by a
memory store) or do we need to keep a rule specific for locks to avoid
arch specific pitfalls on some architecture ? In that case, do we need a
specific barrier to provide MMIO store followed by a memory store ? That
sort of ordering is not generally useful and is generally expensive as
it requires to access the PCI host bridge to enforce that the previous
MMIO stores have reached the bus. Drivers generally don't need such a
rule or a barrier, as they have to deal with write posting anyway, and
thus use an MMIO read to provide the necessary synchronisation when it
makes sense.

[* Question 2] : Do we actually want the "ordered" accessors to also provide
ordering rule #4 in the general case ? This can be very expensive on
some architectures like ia64 where, I think, it has to actually access
the PCI host bridge to provide the guarantee that the previous MMIO
stores have reached it before the unlock is made visible to the
coherency domain. If we decide not to, then an explicit barrier will
still be needed in most drivers before spin_unlock(). This is the
current mmiowb() barrier that I'm proposing to rename (section * III *).
A way to provide that ordering requirement with less performance impact
is to instead set a per-cpu flag in writeX(), and test it in
spin_unlock() which would then do the barrier only if the flag is set.
It's to be measured whether the impact on unrelated spin_unlock() is low
enough to make that solution realstic.
If we decide to not enforce rule #4 for ordered accessors, and thus
require the barrier before spin_unlock, the above trick, could still be
implemented as a debug option to "detect" the lack of appropriate
barriers.

[* Question 3] If we decide that accessors of Class 1 do not provide rule
#4, then this barrier is to be used for all classes of accessors, except
maybe PIO which should always be fully ordered.

[* Question 4] Would it be a useful optimisation on archs like ia64 to
require this accessor to take the struct device of the device as an
argument (with can NULL for a "generic" barrier) or it doesn't matter ?

[* Question 5] Should we document the rules for memory-memory barriers
here as well ? (and give examples, like live updating of a network
driver ring descriptor entry)
 

Re: [RFC] MMIO accessors & barriers documentation #2

[Linus Torvalds]

On Mon, 18 Sep 2006, Benjamin Herrenschmidt wrote:
> 
> Class 1: Ordered accessors
> --------------------------
> 
>  1- {read,write}{b,w,l,q} : MMIO accessors. Those accessors provide
> all MMIO ordering requirements. They are thus called "fully ordered".
> That is #1, #2 and #4 for writes and #1 and #3 for reads. 

Well, it's already not defined to be #4 right now on SGI boxes, and we 
have that (badly named) mmiowb() thing to enforce #4, so I think we should 
just accept that write[bwl]() it's _that_ ordered.

And on x86, we _already_ depend on "wmb()" to be a "normal write to MMIO 
write" barrier, which is technically wrong and bad. Again, thanks to 
mmiowb(), normal memory accesses and MMIO accesses have already been 
defined to not be in the same "ordering domain", so "wmb()" is technically 
wrong and may not order a regular write wrt a MMIO (because it doesn't do 
so for the other order: MMIO->spin_unlock).

So I think we should just admit that at least MMIO _stores_ are already 
not entirely ordered, and not try to strengthen the rules for the current 
setup (and just try to clarify the currently accepted semantics).

		Linus

Re: [RFC] MMIO accessors & barriers documentation #2

[Benjamin Herrenschmidt]

On Sun, 2006-09-17 at 19:24 -0700, Linus Torvalds wrote:

> >  1- {read,write}{b,w,l,q} : MMIO accessors. Those accessors provide
> > all MMIO ordering requirements. They are thus called "fully ordered".
> > That is #1, #2 and #4 for writes and #1 and #3 for reads. 
> 
> Well, it's already not defined to be #4 right now on SGI boxes, and we 
> have that (badly named) mmiowb() thing to enforce #4, so I think we should 
> just accept that write[bwl]() it's _that_ ordered.
>
> And on x86, we _already_ depend on "wmb()" to be a "normal write to MMIO 
> write" barrier, which is technically wrong and bad. Again, thanks to 
> mmiowb(), normal memory accesses and MMIO accesses have already been 
> defined to not be in the same "ordering domain", so "wmb()" is technically 
> wrong and may not order a regular write wrt a MMIO (because it doesn't do 
> so for the other order: MMIO->spin_unlock).
> 
> So I think we should just admit that at least MMIO _stores_ are already 
> not entirely ordered, and not try to strengthen the rules for the current 
> setup (and just try to clarify the currently accepted semantics).

I'm a little bit confused as Alan seemed to imply that memory store vs.
MMIO were fully ordered on x86 and thus no barrier was needed (which was
the start of the discussion in the first place since on PowerPC, they
were not ordered until Paul's latest patch).

I tend to agree with Alan and others though that non-ordered accessors
will be a problem with driver writers who don't understand what it's all
about. David Miller even said that assuming anything else but strongly
ordered accessor was basically shooting ourselves in the foot :)

There have been two main cases biting us so far out of my list of 4,
which is #2 and #4 in my list of ordering requirements: memory store vs.
mmio (read or store) and mmio vs. spinlock. The former was/is handled in
some drivers with wmb() (which I dislike as wmb() should really only be
defined for the coherency domain) and not in others. The later was/is
handled in some drivers with mmiowb() (which I agree is badly named).

Thus we though it might be a more reasonable path to simply define
existing accessors as strongly ordered (and make them so on archs where
they aren't yet) and add some partially relaxed ones with semantics
precise enough to make them useful for drivers in non-arch specific
ways.

Now, you seem to say (unless I misunderstood) that x86 -does- have the
problem of storing to memory followed by an MMIO write not being in
order (wich means that tg3 would have been broken on x86 as well) which
I though wasn't the case.

So what do you recommend we do now ? Go back to step #1 and simply
define readl,writel and friends as not being ordered vs. memory loads
and then define a bunch of explicit barriers, which basically boils down
to using the semantics I defined in my document for class 2, partially
relaxed accessors (__readl/__writel/...) ? Or do we consider that it's
too much of a risk vs. driver writers and do implement my proposal,
making the current accessors slower on some archs but safer, and
starting to convert some drivers hot path to use the relaxed ones ?

Ben.