From: Paul Mackerras <paulus@samba.org>
To: Jesse Barnes <jbarnes@virtuousgeek.org>
Cc: akpm@osdl.org, linux-arch@vger.kernel.org,
linux-kernel@vger.kernel.org, Linus Torvalds <torvalds@osdl.org>,
mingo@redhat.com, Alan Cox <alan@redhat.com>,
linuxppc64-dev@ozlabs.org
Subject: Re: [PATCH] Document Linux's memory barriers [try #2]
Date: Thu, 9 Mar 2006 12:57:27 +1100 [thread overview]
Message-ID: <17423.35719.758492.297725@cargo.ozlabs.ibm.com> (raw)
In-Reply-To: <200603081655.13672.jbarnes@virtuousgeek.org>
Jesse Barnes writes:
> On Wednesday, March 8, 2006 4:35 pm, Paul Mackerras wrote:
> > David Howells writes:
> > > On NUMA PowerPC, should mmiowb() be a SYNC or an EIEIO instruction
> > > then? Those do inter-component synchronisation.
> >
> > We actually have quite heavy synchronization in read*/write* on PPC,
> > and mmiowb can safely be a no-op. It would be nice to be able to have
> > lighter-weight synchronization, but I'm sure we would see lots of
> > subtle driver bugs cropping up if we did. write* do a full memory
> > barrier (sync) after the store, and read* explicitly wait for the data
> > to come back before.
> >
> > If you ask me, the need for mmiowb on some platforms merely shows that
> > those platforms' implementations of spinlocks and read*/write* are
> > buggy...
>
> Or maybe they just wanted to keep them fast. I don't know why you
> compromised so much in your implementation of read/write and
> lock/unlock, but given how expensive synchronization is, I'd think it
> would be better in the long run to make the barrier types explicit (or
> at least a subset of them) to maximize performance.
The PPC read*/write* and in*/out* aim to implement x86 semantics, in
order to minimize the number of subtle driver bugs that only show up
under heavy load.
I agree that in the long run making the barriers more explicit is a
good thing.
> The rules for using
> the barriers really aren't that bad... for mmiowb() you basically want
> to do it before an unlock in any critical section where you've done PIO
> writes.
Do you mean just PIO, or do you mean PIO or MMIO writes?
> Of course, that doesn't mean there isn't confusion about existing
> barriers. There was a long thread a few years ago (Jes worked it all
> out, iirc) regarding some subtle memory ordering bugs in the tty layer
> that ended up being due to ia64's very weak spin_unlock ordering
> guarantees (one way memory barrier only), but I think that's mainly an
> artifact of how ill defined the semantics of the various arch specific
> routines are in some cases.
Yes, there is a lot of confusion, unfortunately. There is also some
difficulty in defining things to be any different from what x86 does.
> That's why I suggested in an earlier thread that you enumerate all the
> memory ordering combinations on ppc and see if we can't define them all.
The main difficulty we strike on PPC is that cacheable accesses tend
to get ordered independently of noncacheable accesses. The only
instruction we have that orders cacheable accesses with respect to
noncacheable accesses is the sync instruction, which is a heavyweight
"synchronize everything" operation. It acts as a full memory barrier
for both cacheable and noncacheable loads and stores.
The other barriers we have are the lwsync instruction and the eieio
instruction. The lwsync instruction (light-weight sync) acts as a
memory barrier for cacheable loads and stores except that it allows a
following load to go before a preceding store.
The eieio instruction has two separate and independent effects. It
acts as a full barrier for accesses to noncacheable nonprefetchable
memory (i.e. MMIO or PIO registers), and it acts as a write barrier
for accesses to cacheable memory. It doesn't do any ordering between
cacheable and noncacheable accesses though.
There is also the isync (instruction synchronize) instruction, which
isn't explicitly a memory barrier. It prevents any following
instructions from executing until the outcome of any previous
conditional branches are known, and until it is known that no previous
instruction can generate an exception. Thus it can be used to create
a one-way barrier in spin_lock and read*.
> Then David can roll the implicit ones up into his document, or we can
> add the appropriate new operations to the kernel. Really getting
> barriers right shouldn't be much harder than getting DMA mapping right,
> from a driver writers POV (though people often get that wrong I guess).
Unfortunately, if you get the barriers wrong your driver will still
work most of the time on pretty much any machine, whereas if you get
the DMA mapping wrong your driver won't work at all on some machines.
Nevertheless, we should get these things defined properly and then try
to make sure drivers do the right things.
Paul.
next prev parent reply other threads:[~2006-03-09 1:57 UTC|newest]
Thread overview: 98+ messages / expand[flat|nested] mbox.gz Atom feed top
2006-03-07 17:40 [PATCH] Document Linux's memory barriers David Howells
2006-03-07 10:34 ` Andi Kleen
2006-03-07 17:47 ` Stephen Hemminger
2006-03-07 18:30 ` David Howells
2006-03-07 11:13 ` Andi Kleen
2006-03-07 18:46 ` Jesse Barnes
2006-03-07 19:23 ` Bryan O'Sullivan
2006-03-07 11:57 ` Andi Kleen
2006-03-07 20:01 ` Jesse Barnes
2006-03-07 21:14 ` Bryan O'Sullivan
2006-03-07 21:24 ` Andi Kleen
2006-03-08 0:36 ` Alan Cox
2006-03-08 0:35 ` Alan Cox
2006-03-07 19:24 ` David Howells
2006-03-07 19:46 ` Stephen Hemminger
2006-03-07 18:40 ` Alan Cox
2006-03-07 18:54 ` linux-os (Dick Johnson)
2006-03-07 19:06 ` Matthew Wilcox
2006-03-07 19:15 ` linux-os (Dick Johnson)
2006-03-09 11:26 ` Sergei Organov
2006-03-07 19:33 ` Alan Cox
2006-03-07 20:09 ` David Howells
2006-03-08 0:32 ` Alan Cox
2006-03-08 8:25 ` Duncan Sands
2006-03-08 22:06 ` Paul Mackerras
2006-03-08 22:24 ` David S. Miller
2006-03-08 22:31 ` Linus Torvalds
2006-03-08 22:42 ` Alan Cox
2006-03-08 2:07 ` Nick Piggin
2006-03-08 3:10 ` Paul Mackerras
2006-03-08 3:30 ` Linus Torvalds
2006-03-08 7:41 ` Nick Piggin
2006-03-08 12:34 ` David Howells
2006-03-08 16:40 ` Bryan O'Sullivan
2006-03-08 13:19 ` David Howells
2006-03-08 21:49 ` Paul Mackerras
2006-03-08 22:05 ` Alan Cox
2006-03-10 0:49 ` H. Peter Anvin
2006-03-08 14:37 ` [PATCH] Document Linux's memory barriers [try #2] David Howells
2006-03-08 14:55 ` Alan Cox
2006-03-08 15:41 ` Matthew Wilcox
2006-03-08 17:19 ` David Howells
2006-03-08 22:10 ` Paul Mackerras
2006-03-08 23:08 ` Ivan Kokshaysky
2006-03-09 1:01 ` Paul Mackerras
2006-03-09 16:02 ` Ivan Kokshaysky
2006-03-08 22:01 ` Paul Mackerras
2006-03-08 22:23 ` David S. Miller
2006-03-08 17:04 ` David Howells
2006-03-08 17:36 ` Alan Cox
2006-03-08 18:35 ` David Howells
2006-03-08 18:45 ` Alan Cox
2006-03-08 18:59 ` David Howells
2006-03-08 11:38 ` Andi Kleen
2006-03-08 19:08 ` David Howells
2006-03-08 19:26 ` Linus Torvalds
2006-03-08 19:40 ` Matthew Wilcox
2006-03-09 0:37 ` Paul Mackerras
2006-03-09 0:59 ` Jesse Barnes
2006-03-09 1:36 ` Paul Mackerras
2006-03-09 4:18 ` Jesse Barnes
2006-03-08 19:54 ` Jesse Barnes
2006-03-08 20:02 ` Alan Cox
2006-03-08 19:31 ` David Howells
2006-03-09 0:35 ` Paul Mackerras
2006-03-09 0:54 ` Linus Torvalds
2006-03-09 1:08 ` Paul Mackerras
2006-03-09 1:27 ` Linus Torvalds
2006-03-09 2:38 ` Nick Piggin
2006-03-09 3:45 ` Paul Mackerras
2006-03-09 4:36 ` Jesse Barnes
2006-03-09 7:41 ` Paul Mackerras
2006-03-09 5:38 ` Linus Torvalds
2006-03-09 11:44 ` Michael Buesch
2006-03-09 12:27 ` David Howells
2006-03-09 4:34 ` Jesse Barnes
2006-03-09 4:43 ` Paul Mackerras
2006-03-09 10:05 ` Jes Sorensen
2006-03-09 0:55 ` Jesse Barnes
2006-03-09 1:57 ` Paul Mackerras [this message]
2006-03-09 4:26 ` Jesse Barnes
2006-03-09 12:02 ` Sergei Organov
2006-03-08 16:18 ` [PATCH] Document Linux's memory barriers Pavel Machek
2006-03-08 16:26 ` Christoph Lameter
2006-03-08 17:35 ` David Howells
2006-03-08 17:46 ` Christoph Lameter
2006-03-08 17:59 ` Alan Cox
2006-03-08 19:37 ` [PATCH] Document Linux's memory barriers [try #3] David Howells
2006-03-08 20:16 ` [PATCH] Document Linux's memory barriers David Howells
2006-03-08 22:01 ` Alan Cox
2006-03-09 11:41 ` David Howells
2006-03-09 12:28 ` Alan Cox
2006-03-09 13:02 ` David Howells
2006-03-09 16:32 ` Linus Torvalds
2006-03-09 17:39 ` David Howells
2006-03-09 17:54 ` Linus Torvalds
2006-03-09 17:56 ` Linus Torvalds
2006-03-09 14:01 ` [PATCH] Document Linux's memory barriers [try #3] David Howells
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=17423.35719.758492.297725@cargo.ozlabs.ibm.com \
--to=paulus@samba.org \
--cc=akpm@osdl.org \
--cc=alan@redhat.com \
--cc=jbarnes@virtuousgeek.org \
--cc=linux-arch@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linuxppc64-dev@ozlabs.org \
--cc=mingo@redhat.com \
--cc=torvalds@osdl.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®