Re: [PATCH] percpu: add __percpu to SHIFT_PERCPU_PTR

[Luc Van Oostenryck <luc.vanoostenryck@gmail.com>]

On Mon, Nov 25, 2019 at 05:41:19PM -0500, Dennis Zhou wrote:
> On Thu, Oct 17, 2019 at 02:13:01PM -0400, Dennis Zhou wrote:
> > On Tue, Oct 15, 2019 at 11:26:15AM +0100, Ben Dooks wrote:
> > > The SHIFT_PERCPU_PTR() returns a pointer used by a number
> > > of functions that expect the pointer to be __percpu annotated
> > > (sparse address space 3). Adding __percpu to this makes the
> > > following sparse warnings go away.
> > > 
> > > Note, this then creates the problem the __percup is marked
> > > as noderef, which may need removing for some of the internal
> > > functions, or to remove other warnings.
> > > 
> > > diff --git a/include/linux/percpu-defs.h b/include/linux/percpu-defs.h
> > > index a6fabd865211..a49b6c702598 100644
> > > --- a/include/linux/percpu-defs.h
> > > +++ b/include/linux/percpu-defs.h
> > > @@ -229,7 +229,7 @@ do {									\
> > >   * pointer value.  The weird cast keeps both GCC and sparse happy.
> > >   */
> > >  #define SHIFT_PERCPU_PTR(__p, __offset)					\
> > > -	RELOC_HIDE((typeof(*(__p)) __kernel __force *)(__p), (__offset))
> > > +	RELOC_HIDE((typeof(*(__p)) __kernel __percpu __force *)(__p), (__offset))
> > >  
> > >  #define per_cpu_ptr(ptr, cpu)						\
> > >  ({									\
> > > -- 
> > > 2.23.0
> > > 
> 
> Hi Ben,
> 
> I've reverted this commit. After spending a little more time on it, I
> don't think this is the fix we want because after we call RELOC_HIDE,
> we are shifting from __percpu to __kernel address spaces as we're adding
> the offset here.

I agree. SHIFT_PERCPU_PTR()'s type is as designed.
Also, it should be noted that this warning doesn't exist on x86.

The problem lies in include/asm-generic/percpu.h, in this case within:
	#define raw_cpu_generic_add_return(pcp, val)		\
	({							\
		typeof(&(pcp)) __p = raw_cpu_ptr(&(pcp));	\
								\
		*__p += val;					\
		*__p;						\
	})

There, the variable __p is declared as a __percpu pointer but:
1) the value assigned to it, the return value of raw_cpu_ptr(), is not
   a plain (__kernel) pointer.
2) the variable is dereferenced just after when a __percpu pointer
   is implicitly __noderef.
It's thus not SHIFT_PERCPU_PTR() or raw_cpu_ptr() that must have
their return type changed but the declaration of __p.

Here is a quick patch, only compile & sparse tested on mm/vmstat.c:

diff --git a/include/asm-generic/percpu.h b/include/asm-generic/percpu.h
index c2de013b2..757b667ca 100644
--- a/include/asm-generic/percpu.h
+++ b/include/asm-generic/percpu.h
@@ -74,7 +74,7 @@ do {									\
 
 #define raw_cpu_generic_add_return(pcp, val)				\
 ({									\
-	typeof(&(pcp)) __p = raw_cpu_ptr(&(pcp));			\
+	typeof(pcp) __kernel __force *__p = raw_cpu_ptr(&(pcp));	\
 									\
 	*__p += val;							\
 	*__p;								\


Note: the same problem is present in raw_cpu_generic_xchg(),
      raw_cpu_generic_cmpxchg() & raw_cpu_generic_cmpxchg_double()


I need more testing and to evaluate possible side-effects before
sending a proper patch.

Best regards,
-- Luc Van Oostenryck