Re: [PATCH 2/2] [PATCH] sched: Add smp_rmb() in task rq locking cycles

["Paul E. McKenney" <paulmck@linux.vnet.ibm.com>]

On Wed, Feb 18, 2015 at 02:47:06PM +0100, Peter Zijlstra wrote:
> On Tue, Feb 17, 2015 at 01:52:31PM -0800, Paul E. McKenney wrote:
> > I could do a table per communication style.  For example, message
> > passing looks like this (give or take likely errors in the table):
> > 
> > 	Side CPU	Top CPU
> > 	--------	-------
> > 	X = 1;		r1 = Y;
> > 	<some barrier>	<some barrier>
> > 	Y = 1;		r2 = X;
> > 
> > 	assert(r1 == 0 || r2 == 1);
> > 
> > 
> >       |   mb  |  wmb  |  rmb  |  rbd  |  acq  |  rel  |  ctl  |
> >  -----+-------+-------+-------+-------+-------+-------+-------+
> >    mb |   Y   |       |   Y   |   y   |   Y   |       |   Y   +
> >  -----+-------+-------+-------+-------+-------+-------+-------+
> >   wmb |   Y   |       |   Y   |   y   |   Y   |       |   Y   +
> >  -----+-------+-------+-------+-------+-------+-------+-------+
> >   rmb |       |       |       |       |       |       |       +
> >  -----+-------+-------+-------+-------+-------+-------+-------+
> >   rbd |       |       |       |       |       |       |       +
> >  -----+-------+-------+-------+-------+-------+-------+-------+
> >   acq |       |       |       |       |       |       |       +
> >  -----+-------+-------+-------+-------+-------+-------+-------+
> >   rel |   Y   |       |   Y   |   y   |   Y   |       |   Y   +
> >  -----+-------+-------+-------+-------+-------+-------+-------+
> >   ctl |       |       |       |       |       |       |       +
> >  -----+-------+-------+-------+-------+-------+-------+-------+
> > 
> > Here "Y" says that the barrier pair works, "y" says that it can
> > work but requires an artificial dependency, and " " says that
> > it does not work.
> 
> I would maybe do s/artificial/additional/, the pointer deref in RCU is
> not really artificial, is it?

Ah, but RCU is a slightly different pattern because you do have to have
a dependency, so you need a different litmus test:

	Initial state: X = 0, Y = &Z, Z = 2

	Side CPU		Top CPU
	--------		-------
	X = 1;			r1 = READ_ONCE(Y);
	<some barrier>		<some_barrier>
	WRITE_ONCE(Y, &X);	r2 = *r1;

	assert(r1 == &Z || r2 == 1);

      |   mb  |  wmb  |  rmb  |  rbd  |  acq  |  rel  |  ctl  |
 -----+-------+-------+-------+-------+-------+-------+-------+
   mb |   Y   |       |   Y   |   Y   |   Y   |       |   Y   +
 -----+-------+-------+-------+-------+-------+-------+-------+
  wmb |   Y   |       |   Y   |   Y   |   Y   |       |   Y   +
 -----+-------+-------+-------+-------+-------+-------+-------+
  rmb |       |       |       |       |       |       |       +
 -----+-------+-------+-------+-------+-------+-------+-------+
  rbd |       |       |       |       |       |       |       +
 -----+-------+-------+-------+-------+-------+-------+-------+
  acq |       |       |       |       |       |       |       +
 -----+-------+-------+-------+-------+-------+-------+-------+
  rel |   Y   |       |   Y   |   Y   |   Y   |       |   Y   +
 -----+-------+-------+-------+-------+-------+-------+-------+
  ctl |       |       |       |       |       |       |       +
 -----+-------+-------+-------+-------+-------+-------+-------+

But of course you should instead use rcu_assign_pointer() and
rcu_dereference().  And I should also have used READ_ONCE() and
WRITE_ONCE() in my earlier example.

> Also, how many communication styles do you envision to enumerate?

As few as possible, but yes, that is likely to be a bit of a problem.
To get an idea of how big it could get if not contained somehow, take
a look at http://www.cl.cam.ac.uk/~pes20/ppc-supplemental/test6.pdf.
20 of them on the first page alone!  ;-)

We most definitely need to stick to the lowest common denominator
if we are taking this approach.

							Thanx, Paul