[PATCH 0/4] s390/percpu: More small improvements

[PATCH 0/4] s390/percpu: More small improvements

[Heiko Carstens]

Some more improvements for the preemptible percpu code. The missing
asm_inline conversion addressed with the first patch is the most
important one. While I first thought that it is a compiler glitch that
some simple this_cpu operations are not inlined anymore it was a simple
asm_inline vs asm in front of percpu inline assemblies which caused this.
With this addressed inlining works again like expected.

Two more patches reduce register pressure for inline assemblies. The last
patch implements this_cpu_try_cmpxchg() for s390 so the compiler can
generate slightly better code similar to try_cmpxchg().

Series applies on top of linux-next 20261008.

Thanks,
Heiko

Heiko Carstens (4):
  s390/percpu: Use asm_inline for arch_this_cpu_add()
  s390/percpu: Drop unused output register from arch_this_cpu_add()
  s390/percpu: Drop unused output register from arch_this_cpu_to_op()
  s390/percpu: Implement this_cpu_try_cmpxchg()

 arch/s390/include/asm/percpu.h | 59 ++++++++++++++++++++++++++++------
 1 file changed, 49 insertions(+), 10 deletions(-)

base-commit: aac26bee2287c88af5be5a5ff96d783b19a28790
-- 
2.53.0

[PATCH 1/4] s390/percpu: Use asm_inline for arch_this_cpu_add()

[Heiko Carstens]

When converting to preemptible percpu operations the arch_this_cpu_add()
inline assemblies were not converted to use asm_inline instead of a plain
asm statement.

With all the macro usage the inline assemblies grow significantly, and
the compiler may decide to not inline even small functions. This can be
observed with gcc16 and a defconfig build where many calls to
__srcu_read_lock_fast() are not inlined anymore, even though it is
basically only a this_cpu_inc().

Use asm_inline to address this.

Fixes: 18ec6c5d7ec6 ("s390/percpu: Use new percpu code section for arch_this_cpu_add()")
Signed-off-by: Heiko Carstens <hca@linux.ibm.com>
---
 arch/s390/include/asm/percpu.h | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/arch/s390/include/asm/percpu.h b/arch/s390/include/asm/percpu.h
index c3995bd8d1de..dfab41b43b8b 100644
--- a/arch/s390/include/asm/percpu.h
+++ b/arch/s390/include/asm/percpu.h
@@ -193,7 +193,7 @@ do {										\
 	ptr__ = PERCPU_PTR(&(pcp));						\
 	if (__builtin_constant_p(val__) &&					\
 	    ((szcast)val__ > -129) && ((szcast)val__ < 128)) {			\
-		asm volatile(							\
+		asm_inline volatile(						\
 			__PCPU_BEGIN("%[lcreg]","%[lcoff]","%[ptr__]",		\
 				      "%[pair__]","%N[pair__]")			\
 			op2 "   0(%N[pair__]),%[val__]\n"			\
@@ -206,7 +206,7 @@ do {										\
 			  "m" (((struct lowcore *)0)->percpu_offset)		\
 			: "cc");						\
 	} else {								\
-		asm volatile(							\
+		asm_inline volatile(						\
 			__PCPU_BEGIN("%[lcreg]","%[lcoff]","%[ptr__]",		\
 				     "%[pair__]","%N[pair__]")			\
 			op1 "   %[old__],%[val__],0(%N[pair__])\n"		\
-- 
2.53.0

[PATCH 2/4] s390/percpu: Drop unused output register from arch_this_cpu_add()

[Heiko Carstens]

The laa/laag instructions return the old memory content, but
arch_this_cpu_add() does not need it. Instead of allocating a dedicated
output register old__ just to discard the returned value, change the
val__ constraint from "d" to "+&d" so it is used as input/output
register.

This reduces register pressure.

Signed-off-by: Heiko Carstens <hca@linux.ibm.com>
---
 arch/s390/include/asm/percpu.h | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/arch/s390/include/asm/percpu.h b/arch/s390/include/asm/percpu.h
index dfab41b43b8b..fd0a3a9951ce 100644
--- a/arch/s390/include/asm/percpu.h
+++ b/arch/s390/include/asm/percpu.h
@@ -188,7 +188,7 @@ do {										\
 	typedef typeof(pcp) pcp_op_T__;						\
 	union register_pair rp__;						\
 	pcp_op_T__ val__ = (val);						\
-	pcp_op_T__ old__, *ptr__;						\
+	pcp_op_T__ *ptr__;							\
 										\
 	ptr__ = PERCPU_PTR(&(pcp));						\
 	if (__builtin_constant_p(val__) &&					\
@@ -209,12 +209,12 @@ do {										\
 		asm_inline volatile(						\
 			__PCPU_BEGIN("%[lcreg]","%[lcoff]","%[ptr__]",		\
 				     "%[pair__]","%N[pair__]")			\
-			op1 "   %[old__],%[val__],0(%N[pair__])\n"		\
+			op1 "   %[val__],%[val__],0(%N[pair__])\n"		\
 			__PCPU_END("%[lcreg]")					\
-			: [old__] "=&d" (old__), [pair__] "=&a" (rp__.pair),	\
+			: [val__] "+&d" (val__), [pair__] "=&a" (rp__.pair),	\
 			  "+m" (*ptr__),					\
 			  "=m" (((struct lowcore *)0)->percpu_register)		\
-			: [val__] "d" (val__), [ptr__] "a" (ptr__),		\
+			: [ptr__] "a" (ptr__),					\
 			  [lcreg] "i" (LC_PERCPU_REGISTER),			\
 			  [lcoff] "i" (LC_PERCPU_OFFSET),			\
 			  "m" (((struct lowcore *)0)->percpu_offset)		\
-- 
2.53.0

[PATCH 3/4] s390/percpu: Drop unused output register from arch_this_cpu_to_op()

[Heiko Carstens]

The lan/lang/lao/laog instructions return the old memory content, but
arch_this_cpu_to_op() does not need it. Instead of allocating a dedicated
output register old__ just to discard the returned value, change the
val__ constraint from "d" to "+&d" so it is used as input/output
register.

This reduces register pressure.

Signed-off-by: Heiko Carstens <hca@linux.ibm.com>
---
 arch/s390/include/asm/percpu.h | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/arch/s390/include/asm/percpu.h b/arch/s390/include/asm/percpu.h
index fd0a3a9951ce..7d87e0a262c3 100644
--- a/arch/s390/include/asm/percpu.h
+++ b/arch/s390/include/asm/percpu.h
@@ -257,18 +257,18 @@ do {									\
 	typedef typeof(pcp) pcp_op_T__; 				\
 	union register_pair rp__;					\
 	pcp_op_T__ val__ = (val);					\
-	pcp_op_T__ old__, *ptr__;					\
+	pcp_op_T__ *ptr__;						\
 									\
 	ptr__ = PERCPU_PTR(&(pcp));					\
 	asm_inline volatile(						\
 		__PCPU_BEGIN("%[lcreg]","%[lcoff]","%[ptr__]",		\
 			     "%[pair__]","%N[pair__]")			\
-		op "    %[old__],%[val__],0(%N[pair__])\n"		\
+		op "    %[val__],%[val__],0(%N[pair__])\n"		\
 		__PCPU_END("%[lcreg]")					\
-		: [old__] "=&d" (old__), [pair__] "=&a" (rp__.pair),	\
+		: [val__] "+&d" (val__), [pair__] "=&a" (rp__.pair),	\
 		  "+m" (*ptr__),					\
 		  "=m" (((struct lowcore *)0)->percpu_register)		\
-		: [val__] "d" (val__), [ptr__] "a" (ptr__),		\
+		: [ptr__] "a" (ptr__),					\
 		  [lcreg] "i" (LC_PERCPU_REGISTER),			\
 		  [lcoff] "i" (LC_PERCPU_OFFSET),			\
 		  "m" (((struct lowcore *)0)->percpu_offset)		\
-- 
2.53.0

[PATCH 4/4] s390/percpu: Implement this_cpu_try_cmpxchg()

[Heiko Carstens]

Provide an s390 specific implementation of this_cpu_try_cmpxchg() for
the four and eight byte cases, if the compiler supports flag output
operands.

This variant uses the condition code set by cs/csg, which avoids a
compare instruction required by the generic variant and generates
slightly better code.

Signed-off-by: Heiko Carstens <hca@linux.ibm.com>
---
 arch/s390/include/asm/percpu.h | 39 ++++++++++++++++++++++++++++++++++
 1 file changed, 39 insertions(+)

diff --git a/arch/s390/include/asm/percpu.h b/arch/s390/include/asm/percpu.h
index 7d87e0a262c3..7b7a8cb51daa 100644
--- a/arch/s390/include/asm/percpu.h
+++ b/arch/s390/include/asm/percpu.h
@@ -5,6 +5,7 @@
 #include <linux/preempt.h>
 #include <asm/cmpxchg.h>
 #include <asm/march.h>
+#include <asm/asm.h>
 
 /*
  * s390 uses its own implementation for per cpu data, the offset of
@@ -404,6 +405,44 @@ do {									\
 	old__;								\
 })
 
+#ifdef __HAVE_ASM_FLAG_OUTPUTS__
+
+#define arch_this_cpu_try_cmpxchg(pcp, ovalp, nval, op)			\
+({									\
+	typedef typeof(pcp) pcp_op_T__;					\
+	typeof(ovalp) ovalp__ = (ovalp);				\
+	pcp_op_T__ old__ = *ovalp__;					\
+	pcp_op_T__ new__ = (nval);					\
+	union register_pair rp__;					\
+	pcp_op_T__ *ptr__;						\
+	int cc__;							\
+									\
+	ptr__ = PERCPU_PTR(&(pcp));					\
+	asm_inline volatile(						\
+		__PCPU_BEGIN("%[lcreg]","%[lcoff]","%[ptr__]",		\
+			     "%[pair__]","%N[pair__]")			\
+		op "	%[old__],%[new__],0(%N[pair__])\n"		\
+		__PCPU_END("%[lcreg]")					\
+		: [old__] "+&d" (old__), [pair__] "=&a" (rp__.pair),	\
+		  "+m" (*ptr__), "=@cc" (cc__),				\
+		  "=m" (((struct lowcore *)0)->percpu_register)		\
+		: [new__] "d" (new__), [ptr__] "a" (ptr__),		\
+		  [lcreg] "i" (LC_PERCPU_REGISTER),			\
+		  [lcoff] "i" (LC_PERCPU_OFFSET),			\
+		  "m" (((struct lowcore *)0)->percpu_offset)		\
+		: "memory");						\
+	if (unlikely(cc__))						\
+		*ovalp__ = old__;					\
+	likely(cc__ == 0);						\
+})
+
+#define this_cpu_try_cmpxchg_4(pcp, ovalp, nval) arch_this_cpu_try_cmpxchg(pcp, ovalp, nval, "cs")
+#define this_cpu_try_cmpxchg_8(pcp, ovalp, nval) arch_this_cpu_try_cmpxchg(pcp, ovalp, nval, "csg")
+
+#define this_cpu_try_cmpxchg64(pcp, ovalp, nval) this_cpu_try_cmpxchg_8(pcp, ovalp, nval)
+
+#endif /* __HAVE_ASM_FLAG_OUTPUTS__ */
+
 #define arch_this_cpu_xchg_simple(pcp, nval)				\
 ({									\
 	typeof(pcp) *ptr__;						\
-- 
2.53.0

Re: [PATCH 1/4] s390/percpu: Use asm_inline for arch_this_cpu_add()

[Alexander Gordeev]

On Mon, Oct 12, 2026 at 01:42:46PM +0200, Heiko Carstens wrote:
> When converting to preemptible percpu operations the arch_this_cpu_add()
> inline assemblies were not converted to use asm_inline instead of a plain
> asm statement.
> 
> With all the macro usage the inline assemblies grow significantly, and
> the compiler may decide to not inline even small functions. This can be
> observed with gcc16 and a defconfig build where many calls to
> __srcu_read_lock_fast() are not inlined anymore, even though it is
> basically only a this_cpu_inc().
> 
> Use asm_inline to address this.
> 
> Fixes: 18ec6c5d7ec6 ("s390/percpu: Use new percpu code section for arch_this_cpu_add()")
> Signed-off-by: Heiko Carstens <hca@linux.ibm.com>
> ---
>  arch/s390/include/asm/percpu.h | 4 ++--
>  1 file changed, 2 insertions(+), 2 deletions(-)

Acked-by: Alexander Gordeev <agordeev@linux.ibm.com>

Re: [PATCH 2/4] s390/percpu: Drop unused output register from arch_this_cpu_add()

[Alexander Gordeev]

On Mon, Oct 12, 2026 at 01:42:47PM +0200, Heiko Carstens wrote:
> The laa/laag instructions return the old memory content, but
> arch_this_cpu_add() does not need it. Instead of allocating a dedicated
> output register old__ just to discard the returned value, change the
> val__ constraint from "d" to "+&d" so it is used as input/output
> register.
> 
> This reduces register pressure.

Do you know whether the compiler is may be smart enough to select the
same register for old and val himself?

My concern is if we force the compiler to do so we might lose optimiztions
where the val could be reused after the op.

(CC-ed Ilya)

> Signed-off-by: Heiko Carstens <hca@linux.ibm.com>
> ---
>  arch/s390/include/asm/percpu.h | 8 ++++----
>  1 file changed, 4 insertions(+), 4 deletions(-)
> 
> diff --git a/arch/s390/include/asm/percpu.h b/arch/s390/include/asm/percpu.h
> index dfab41b43b8b..fd0a3a9951ce 100644
> --- a/arch/s390/include/asm/percpu.h
> +++ b/arch/s390/include/asm/percpu.h
> @@ -188,7 +188,7 @@ do {										\
>  	typedef typeof(pcp) pcp_op_T__;						\
>  	union register_pair rp__;						\
>  	pcp_op_T__ val__ = (val);						\
> -	pcp_op_T__ old__, *ptr__;						\
> +	pcp_op_T__ *ptr__;							\
>  										\
>  	ptr__ = PERCPU_PTR(&(pcp));						\
>  	if (__builtin_constant_p(val__) &&					\
> @@ -209,12 +209,12 @@ do {										\
>  		asm_inline volatile(						\
>  			__PCPU_BEGIN("%[lcreg]","%[lcoff]","%[ptr__]",		\
>  				     "%[pair__]","%N[pair__]")			\
> -			op1 "   %[old__],%[val__],0(%N[pair__])\n"		\
> +			op1 "   %[val__],%[val__],0(%N[pair__])\n"		\
>  			__PCPU_END("%[lcreg]")					\
> -			: [old__] "=&d" (old__), [pair__] "=&a" (rp__.pair),	\
> +			: [val__] "+&d" (val__), [pair__] "=&a" (rp__.pair),	\
>  			  "+m" (*ptr__),					\
>  			  "=m" (((struct lowcore *)0)->percpu_register)		\
> -			: [val__] "d" (val__), [ptr__] "a" (ptr__),		\
> +			: [ptr__] "a" (ptr__),					\
>  			  [lcreg] "i" (LC_PERCPU_REGISTER),			\
>  			  [lcoff] "i" (LC_PERCPU_OFFSET),			\
>  			  "m" (((struct lowcore *)0)->percpu_offset)		\
> -- 
> 2.53.0
> 

Re: [PATCH 2/4] s390/percpu: Drop unused output register from arch_this_cpu_add()

[Heiko Carstens]

On Mon, Oct 12, 2026 at 03:24:20PM +0200, Alexander Gordeev wrote:
> On Mon, Oct 12, 2026 at 01:42:47PM +0200, Heiko Carstens wrote:
> > The laa/laag instructions return the old memory content, but
> > arch_this_cpu_add() does not need it. Instead of allocating a dedicated
> > output register old__ just to discard the returned value, change the
> > val__ constraint from "d" to "+&d" so it is used as input/output
> > register.
> > 
> > This reduces register pressure.
> 
> Do you know whether the compiler is may be smart enough to select the
> same register for old and val himself?

In theory, if the constraints would allow that, yes. But for the preemptible
percpu inline asm all output operands need to be early clobbered. Therefore
gcc is forced to not do that.

> My concern is if we force the compiler to do so we might lose optimiztions
> where the val could be reused after the op.

The overall size change of the kernel image is six bytes :).
I guess we shouldn't spend too much time with this.

Re: [PATCH 2/4] s390/percpu: Drop unused output register from arch_this_cpu_add()

[Alexander Gordeev]

On Mon, Oct 12, 2026 at 01:42:47PM +0200, Heiko Carstens wrote:
> The laa/laag instructions return the old memory content, but
> arch_this_cpu_add() does not need it. Instead of allocating a dedicated
> output register old__ just to discard the returned value, change the
> val__ constraint from "d" to "+&d" so it is used as input/output
> register.
> 
> This reduces register pressure.
> 
> Signed-off-by: Heiko Carstens <hca@linux.ibm.com>
> ---
>  arch/s390/include/asm/percpu.h | 8 ++++----
>  1 file changed, 4 insertions(+), 4 deletions(-)

Acked-by: Alexander Gordeev <agordeev@linux.ibm.com>

Re: [PATCH 3/4] s390/percpu: Drop unused output register from arch_this_cpu_to_op()

[Alexander Gordeev]

On Mon, Oct 12, 2026 at 01:42:48PM +0200, Heiko Carstens wrote:
> The lan/lang/lao/laog instructions return the old memory content, but
> arch_this_cpu_to_op() does not need it. Instead of allocating a dedicated
> output register old__ just to discard the returned value, change the
> val__ constraint from "d" to "+&d" so it is used as input/output
> register.
> 
> This reduces register pressure.
> 
> Signed-off-by: Heiko Carstens <hca@linux.ibm.com>
> ---
>  arch/s390/include/asm/percpu.h | 8 ++++----
>  1 file changed, 4 insertions(+), 4 deletions(-)

Acked-by: Alexander Gordeev <agordeev@linux.ibm.com>

Re: [PATCH 4/4] s390/percpu: Implement this_cpu_try_cmpxchg()

[Alexander Gordeev]

On Mon, Oct 12, 2026 at 01:42:49PM +0200, Heiko Carstens wrote:
> Provide an s390 specific implementation of this_cpu_try_cmpxchg() for
> the four and eight byte cases, if the compiler supports flag output
> operands.
> 
> This variant uses the condition code set by cs/csg, which avoids a
> compare instruction required by the generic variant and generates
> slightly better code.
> 
> Signed-off-by: Heiko Carstens <hca@linux.ibm.com>
> ---
>  arch/s390/include/asm/percpu.h | 39 ++++++++++++++++++++++++++++++++++
>  1 file changed, 39 insertions(+)

Acked-by: Alexander Gordeev <agordeev@linux.ibm.com>