* [PATCH v3 01/11] s390/percpu: Fix comment typo
2026-09-21 8:39 [PATCH v3 00/11] s390: More this_cpu_*() changes Heiko Carstens
@ 2026-09-21 8:39 ` Heiko Carstens
2026-09-21 8:39 ` [PATCH v3 02/11] s390/percpu: Add sanity check to GEN_MVIY macro Heiko Carstens
` (10 subsequent siblings)
11 siblings, 0 replies; 16+ messages in thread
From: Heiko Carstens @ 2026-09-21 8:39 UTC (permalink / raw)
To: Alexander Gordeev, Sven Schnelle, Vasily Gorbik,
Christian Borntraeger, Mete Durlu, Peter Zijlstra, Mark Rutland,
Juergen Christ, Ilya Leoshkevich
Cc: linux-kernel, linux-s390
This is all preempt_disable() / preempt_enable() pairs and not
preempt_disable() / preempt_disable() pairs.
Signed-off-by: Heiko Carstens <hca@linux.ibm.com>
---
arch/s390/include/asm/percpu.h | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/arch/s390/include/asm/percpu.h b/arch/s390/include/asm/percpu.h
index feb58acf87d6..31012abe8292 100644
--- a/arch/s390/include/asm/percpu.h
+++ b/arch/s390/include/asm/percpu.h
@@ -63,7 +63,7 @@
/*
* Macros to be used for percpu code section based on atomic instructions.
*
- * Avoid the need to use preempt_disable() / preempt_disable() pairs and the
+ * Avoid the need to use preempt_disable() / preempt_enable() pairs and the
* conditional preempt_schedule_notrace() function calls which come with
* this. The idea is that this_cpu operations based on atomic instructions are
* guarded with mviy instructions:
--
2.53.0
^ permalink raw reply [flat|nested] 16+ messages in thread* [PATCH v3 02/11] s390/percpu: Add sanity check to GEN_MVIY macro
2026-09-21 8:39 [PATCH v3 00/11] s390: More this_cpu_*() changes Heiko Carstens
2026-09-21 8:39 ` [PATCH v3 01/11] s390/percpu: Fix comment typo Heiko Carstens
@ 2026-09-21 8:39 ` Heiko Carstens
2026-09-21 8:39 ` [PATCH v3 03/11] s390/lowcore: Remove _AC() from LOWCORE_ALT_ADDRESS Heiko Carstens
` (9 subsequent siblings)
11 siblings, 0 replies; 16+ messages in thread
From: Heiko Carstens @ 2026-09-21 8:39 UTC (permalink / raw)
To: Alexander Gordeev, Sven Schnelle, Vasily Gorbik,
Christian Borntraeger, Mete Durlu, Peter Zijlstra, Mark Rutland,
Juergen Christ, Ilya Leoshkevich
Cc: linux-kernel, linux-s390
Emit an error if the GEN_MVIY macro fails to parse the passed register
string. This will make it more obvious if something similar which was
fixed with [1] happens again.
[1] commit 101782f8945a ("s390/percpu: Fix MVIY_PERCPU() with older binutils")
Signed-off-by: Heiko Carstens <hca@linux.ibm.com>
---
arch/s390/include/asm/percpu.h | 9 +++++++--
1 file changed, 7 insertions(+), 2 deletions(-)
diff --git a/arch/s390/include/asm/percpu.h b/arch/s390/include/asm/percpu.h
index 31012abe8292..9888b38dedab 100644
--- a/arch/s390/include/asm/percpu.h
+++ b/arch/s390/include/asm/percpu.h
@@ -100,12 +100,17 @@
*/
#define MVIY_PERCPU(disp, dispalt, reg) \
- ".macro GEN_MVIY disp reg\n" \
+ ".macro GEN_MVIY disp, reg\n" \
+ ".set .Lreg,255\n" \
".irp rs,0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15\n" \
" .ifc \\reg,%%r\\rs\n" \
- " mviy \\disp(%%r0),\\rs\n" \
+ " .set .Lreg,\\rs\n" \
" .endif\n" \
".endr\n" \
+ ".if .Lreg == 255\n" \
+ " .error \"Illegal register number\"\n" \
+ ".endif\n" \
+ "mviy \\disp(%%r0),.Lreg\n" \
".endm\n" \
ALTERNATIVE("GEN_MVIY " disp ", " reg "\n", \
"GEN_MVIY " dispalt ", " reg "\n", \
--
2.53.0
^ permalink raw reply [flat|nested] 16+ messages in thread* [PATCH v3 03/11] s390/lowcore: Remove _AC() from LOWCORE_ALT_ADDRESS
2026-09-21 8:39 [PATCH v3 00/11] s390: More this_cpu_*() changes Heiko Carstens
2026-09-21 8:39 ` [PATCH v3 01/11] s390/percpu: Fix comment typo Heiko Carstens
2026-09-21 8:39 ` [PATCH v3 02/11] s390/percpu: Add sanity check to GEN_MVIY macro Heiko Carstens
@ 2026-09-21 8:39 ` Heiko Carstens
2026-09-21 8:39 ` [PATCH v3 04/11] s390/percpu: Let MVIY_PERCPU() calculate alternative displacement Heiko Carstens
` (8 subsequent siblings)
11 siblings, 0 replies; 16+ messages in thread
From: Heiko Carstens @ 2026-09-21 8:39 UTC (permalink / raw)
To: Alexander Gordeev, Sven Schnelle, Vasily Gorbik,
Christian Borntraeger, Mete Durlu, Peter Zijlstra, Mark Rutland,
Juergen Christ, Ilya Leoshkevich
Cc: linux-kernel, linux-s390
Remove the _AC() conversion from LOWCORE_ALT_ADDRESS. This changes the
type of LOWCORE_ALT_ADDRESS from unsigned long to int in C code, however
looking at the current users this should not cause any changes.
This allows to use __stringify(LOWCORE_ALT_ADDRESS) for inline
assemblies, which otherwise wouldn't work because of the UL suffix.
Signed-off-by: Heiko Carstens <hca@linux.ibm.com>
---
arch/s390/include/asm/lowcore.h | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/arch/s390/include/asm/lowcore.h b/arch/s390/include/asm/lowcore.h
index 5cef215d30e7..f0364a9f8a19 100644
--- a/arch/s390/include/asm/lowcore.h
+++ b/arch/s390/include/asm/lowcore.h
@@ -21,7 +21,7 @@
#define LC_ORDER 1
#define LC_PAGES 2
-#define LOWCORE_ALT_ADDRESS _AC(0x70000, UL)
+#define LOWCORE_ALT_ADDRESS 0x70000
#ifndef __ASSEMBLER__
--
2.53.0
^ permalink raw reply [flat|nested] 16+ messages in thread* [PATCH v3 04/11] s390/percpu: Let MVIY_PERCPU() calculate alternative displacement
2026-09-21 8:39 [PATCH v3 00/11] s390: More this_cpu_*() changes Heiko Carstens
` (2 preceding siblings ...)
2026-09-21 8:39 ` [PATCH v3 03/11] s390/lowcore: Remove _AC() from LOWCORE_ALT_ADDRESS Heiko Carstens
@ 2026-09-21 8:39 ` Heiko Carstens
2026-09-21 8:39 ` [PATCH v3 05/11] s390/percpu/lowcore: Add and use LC_PERCPU lowcore offset defines Heiko Carstens
` (7 subsequent siblings)
11 siblings, 0 replies; 16+ messages in thread
From: Heiko Carstens @ 2026-09-21 8:39 UTC (permalink / raw)
To: Alexander Gordeev, Sven Schnelle, Vasily Gorbik,
Christian Borntraeger, Mete Durlu, Peter Zijlstra, Mark Rutland,
Juergen Christ, Ilya Leoshkevich
Cc: linux-kernel, linux-s390
Let MVIY_PERCPU() calculate the alternative lowcore displacement instead
of passing the same value(s) again and again in all percpu inline
assemblies.
This shortens the inline assemblies and makes them a bit more readable.
Signed-off-by: Heiko Carstens <hca@linux.ibm.com>
---
arch/s390/include/asm/percpu.h | 68 +++++++++++++++-------------------
1 file changed, 29 insertions(+), 39 deletions(-)
diff --git a/arch/s390/include/asm/percpu.h b/arch/s390/include/asm/percpu.h
index 9888b38dedab..3bc448c4c2dd 100644
--- a/arch/s390/include/asm/percpu.h
+++ b/arch/s390/include/asm/percpu.h
@@ -99,7 +99,9 @@
* MVIY_ALT(...) <- end of percpu code section
*/
-#define MVIY_PERCPU(disp, dispalt, reg) \
+#define LC_ALT_ADDR __stringify(LOWCORE_ALT_ADDRESS)
+
+#define MVIY_PERCPU(disp, reg) \
".macro GEN_MVIY disp, reg\n" \
".set .Lreg,255\n" \
".irp rs,0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15\n" \
@@ -112,19 +114,19 @@
".endif\n" \
"mviy \\disp(%%r0),.Lreg\n" \
".endm\n" \
- ALTERNATIVE("GEN_MVIY " disp ", " reg "\n", \
- "GEN_MVIY " dispalt ", " reg "\n", \
+ ALTERNATIVE("GEN_MVIY " disp " , " reg "\n", \
+ "GEN_MVIY " disp "+" LC_ALT_ADDR " , " reg "\n", \
ALT_FEATURE(MFEATURE_LOWCORE)) \
".purgem GEN_MVIY\n"
-#define MVIY_ALT(disp, dispalt) \
- ALTERNATIVE(" mviy " disp "(%%r0),0\n", \
- " mviy " dispalt "(%%r0),0\n", \
+#define MVIY_ALT(disp) \
+ ALTERNATIVE(" mviy " disp "(%%r0),0\n", \
+ " mviy " disp "+" LC_ALT_ADDR "(%%r0),0\n", \
ALT_FEATURE(MFEATURE_LOWCORE))
-#define AG_ALT(disp, dispalt, reg) \
- ALTERNATIVE(" ag " reg ", " disp "(%%r0)\n", \
- " ag " reg ", " dispalt "(%%r0)\n", \
+#define AG_ALT(disp, reg) \
+ ALTERNATIVE(" ag " reg ", " disp "(%%r0)\n", \
+ " ag " reg ", " disp "+" LC_ALT_ADDR "(%%r0)\n", \
ALT_FEATURE(MFEATURE_LOWCORE))
#ifndef MARCH_HAS_Z196_FEATURES
@@ -153,33 +155,29 @@ do { \
if (__builtin_constant_p(val__) && \
((szcast)val__ > -129) && ((szcast)val__ < 128)) { \
asm volatile( \
- MVIY_PERCPU("%[disppcpr]", "%[dispaltpcpr]", "%[ptr__]")\
- AG_ALT("%[disppcpo]", "%[dispaltpcpo]", "%[ptr__]") \
+ MVIY_PERCPU("%[disppcpr]","%[ptr__]") \
+ AG_ALT("%[disppcpo]", "%[ptr__]") \
op2 " 0(%[ptr__]),%[val__]\n" \
- MVIY_ALT("%[disppcpr]", "%[dispaltpcpr]") \
+ MVIY_ALT("%[disppcpr]") \
: [ptr__] "+&a" (ptr__), "+m" (*ptr__), \
"=m" (((struct lowcore *)0)->percpu_register) \
: [val__] "i" ((szcast)val__), \
[disppcpr] "i" (lc_pcpr), \
[disppcpo] "i" (lc_pcpo), \
- [dispaltpcpr] "i" (lc_pcpr + LOWCORE_ALT_ADDRESS), \
- [dispaltpcpo] "i" (lc_pcpo + LOWCORE_ALT_ADDRESS), \
"m" (((struct lowcore *)0)->percpu_offset) \
: "cc"); \
} else { \
asm volatile( \
- MVIY_PERCPU("%[disppcpr]", "%[dispaltpcpr]", "%[ptr__]")\
- AG_ALT("%[disppcpo]", "%[dispaltpcpo]", "%[ptr__]") \
+ MVIY_PERCPU("%[disppcpr]", "%[ptr__]") \
+ AG_ALT("%[disppcpo]", "%[ptr__]") \
op1 " %[old__],%[val__],0(%[ptr__])\n" \
- MVIY_ALT("%[disppcpr]", "%[dispaltpcpr]") \
+ MVIY_ALT("%[disppcpr]") \
: [old__] "=&d" (old__), \
[ptr__] "+&a" (ptr__), "+m" (*ptr__), \
"=m" (((struct lowcore *)0)->percpu_register) \
: [val__] "d" (val__), \
[disppcpr] "i" (lc_pcpr), \
[disppcpo] "i" (lc_pcpo), \
- [dispaltpcpr] "i" (lc_pcpr + LOWCORE_ALT_ADDRESS), \
- [dispaltpcpo] "i" (lc_pcpo + LOWCORE_ALT_ADDRESS), \
"m" (((struct lowcore *)0)->percpu_offset) \
: "cc"); \
} \
@@ -199,18 +197,16 @@ do { \
lc_pcpo = offsetof(struct lowcore, percpu_offset); \
ptr__ = PERCPU_PTR(&(pcp)); \
asm_inline volatile( \
- MVIY_PERCPU("%[disppcpr]", "%[dispaltpcpr]", "%[ptr__]")\
- AG_ALT("%[disppcpo]", "%[dispaltpcpo]", "%[ptr__]") \
+ MVIY_PERCPU("%[disppcpr]","%[ptr__]") \
+ AG_ALT("%[disppcpo]","%[ptr__]") \
op " %[old__],%[val__],0(%[ptr__])\n" \
- MVIY_ALT("%[disppcpr]", "%[dispaltpcpr]") \
+ MVIY_ALT("%[disppcpr]") \
: [old__] "=&d" (old__), \
[ptr__] "+&a" (ptr__), "+m" (*ptr__), \
"=m" (((struct lowcore *)0)->percpu_register) \
: [val__] "d" (val__), \
[disppcpr] "i" (lc_pcpr), \
[disppcpo] "i" (lc_pcpo), \
- [dispaltpcpr] "i" (lc_pcpr + LOWCORE_ALT_ADDRESS), \
- [dispaltpcpo] "i" (lc_pcpo + LOWCORE_ALT_ADDRESS), \
"m" (((struct lowcore *)0)->percpu_offset) \
: "cc"); \
old__ + val__; \
@@ -230,18 +226,16 @@ do { \
lc_pcpo = offsetof(struct lowcore, percpu_offset); \
ptr__ = PERCPU_PTR(&(pcp)); \
asm_inline volatile( \
- MVIY_PERCPU("%[disppcpr]", "%[dispaltpcpr]", "%[ptr__]")\
- AG_ALT("%[disppcpo]", "%[dispaltpcpo]", "%[ptr__]") \
+ MVIY_PERCPU("%[disppcpr]","%[ptr__]") \
+ AG_ALT("%[disppcpo]","%[ptr__]") \
op " %[old__],%[val__],0(%[ptr__])\n" \
- MVIY_ALT("%[disppcpr]", "%[dispaltpcpr]") \
+ MVIY_ALT("%[disppcpr]") \
: [old__] "=&d" (old__), \
[ptr__] "+&a" (ptr__), "+m" (*ptr__), \
"=m" (((struct lowcore *)0)->percpu_register) \
: [val__] "d" (val__), \
[disppcpr] "i" (lc_pcpr), \
[disppcpo] "i" (lc_pcpo), \
- [dispaltpcpr] "i" (lc_pcpr + LOWCORE_ALT_ADDRESS), \
- [dispaltpcpo] "i" (lc_pcpo + LOWCORE_ALT_ADDRESS), \
"m" (((struct lowcore *)0)->percpu_offset) \
: "cc"); \
} while (0)
@@ -263,16 +257,14 @@ do { \
lc_pcpo = offsetof(struct lowcore, percpu_offset); \
ptr__ = PERCPU_PTR(&(pcp)); \
asm_inline volatile( \
- MVIY_PERCPU("%[disppcpr]", "%[dispaltpcpr]", "%[ptr__]")\
- AG_ALT("%[disppcpo]", "%[dispaltpcpo]", "%[ptr__]") \
+ MVIY_PERCPU("%[disppcpr]","%[ptr__]") \
+ AG_ALT("%[disppcpo]","%[ptr__]") \
op " %[res__],0(%[ptr__])\n" \
- MVIY_ALT("%[disppcpr]", "%[dispaltpcpr]") \
+ MVIY_ALT("%[disppcpr]") \
: [res__] "=&d" (res__), [ptr__] "+&a" (ptr__), \
"=m" (((struct lowcore *)0)->percpu_register) \
: [disppcpr] "i" (lc_pcpr), \
[disppcpo] "i" (lc_pcpo), \
- [dispaltpcpr] "i" (lc_pcpr + LOWCORE_ALT_ADDRESS), \
- [dispaltpcpo] "i" (lc_pcpo + LOWCORE_ALT_ADDRESS), \
"m" (*ptr__), \
"m" (((struct lowcore *)0)->percpu_offset) \
: "cc"); \
@@ -294,17 +286,15 @@ do { \
lc_pcpo = offsetof(struct lowcore, percpu_offset); \
ptr__ = PERCPU_PTR(&(pcp)); \
asm_inline volatile( \
- MVIY_PERCPU("%[disppcpr]", "%[dispaltpcpr]", "%[ptr__]")\
- AG_ALT("%[disppcpo]", "%[dispaltpcpo]", "%[ptr__]") \
+ MVIY_PERCPU("%[disppcpr]","%[ptr__]") \
+ AG_ALT("%[disppcpo]","%[ptr__]") \
op " %[val__],0(%[ptr__])\n" \
- MVIY_ALT("%[disppcpr]", "%[dispaltpcpr]") \
+ MVIY_ALT("%[disppcpr]") \
: [ptr__] "+&a" (ptr__), "=m" (*ptr__), \
"=m" (((struct lowcore *)0)->percpu_register) \
: [val__] "d" (val__), \
[disppcpr] "i" (lc_pcpr), \
[disppcpo] "i" (lc_pcpo), \
- [dispaltpcpr] "i" (lc_pcpr + LOWCORE_ALT_ADDRESS), \
- [dispaltpcpo] "i" (lc_pcpo + LOWCORE_ALT_ADDRESS), \
"m" (((struct lowcore *)0)->percpu_offset) \
: "cc"); \
} while (0)
--
2.53.0
^ permalink raw reply [flat|nested] 16+ messages in thread* [PATCH v3 05/11] s390/percpu/lowcore: Add and use LC_PERCPU lowcore offset defines
2026-09-21 8:39 [PATCH v3 00/11] s390: More this_cpu_*() changes Heiko Carstens
` (3 preceding siblings ...)
2026-09-21 8:39 ` [PATCH v3 04/11] s390/percpu: Let MVIY_PERCPU() calculate alternative displacement Heiko Carstens
@ 2026-09-21 8:39 ` Heiko Carstens
2026-09-21 8:40 ` [PATCH v3 06/11] s390/percpu: Rename inline assembly symbolic names Heiko Carstens
` (6 subsequent siblings)
11 siblings, 0 replies; 16+ messages in thread
From: Heiko Carstens @ 2026-09-21 8:39 UTC (permalink / raw)
To: Alexander Gordeev, Sven Schnelle, Vasily Gorbik,
Christian Borntraeger, Mete Durlu, Peter Zijlstra, Mark Rutland,
Juergen Christ, Ilya Leoshkevich
Cc: linux-kernel, linux-s390
Define LC_PERCPU_OFFSET and LC_PERCPU_REGISTER offset defines and use
them in percpu code. This shortens the percpu inline assemblies and makes
them a bit more readable.
Note that because of header dependencies it is not possible to use
asm-offsets. However to avoid adding more dependencies to asm-offsets
it seems to be preferable to use the current approach in any way.
Signed-off-by: Heiko Carstens <hca@linux.ibm.com>
---
arch/s390/include/asm/entry-percpu.h | 2 +-
arch/s390/include/asm/lowcore.h | 3 ++
arch/s390/include/asm/percpu.h | 47 ++++++++++------------------
3 files changed, 20 insertions(+), 32 deletions(-)
diff --git a/arch/s390/include/asm/entry-percpu.h b/arch/s390/include/asm/entry-percpu.h
index 4ef47265cfce..89c9e44f6f83 100644
--- a/arch/s390/include/asm/entry-percpu.h
+++ b/arch/s390/include/asm/entry-percpu.h
@@ -49,7 +49,7 @@ static __always_inline bool percpu_code_check(struct pt_regs *regs)
}
if ((insn & 0xff0f) != 0xe300)
return true;
- disp = offsetof(struct lowcore, percpu_offset);
+ disp = LC_PERCPU_OFFSET;
if (machine_has_relocated_lowcore())
disp += LOWCORE_ALT_ADDRESS;
insn = (disp & 0xff000) >> 4 | (disp & 0x00fff) << 16 | 0x8;
diff --git a/arch/s390/include/asm/lowcore.h b/arch/s390/include/asm/lowcore.h
index f0364a9f8a19..f4517bbe405b 100644
--- a/arch/s390/include/asm/lowcore.h
+++ b/arch/s390/include/asm/lowcore.h
@@ -222,6 +222,9 @@ struct lowcore {
__u8 pad_0x1900[0x2000-0x1900]; /* 0x1900 */
} __packed __aligned(8192);
+#define LC_PERCPU_OFFSET offsetof(struct lowcore, percpu_offset)
+#define LC_PERCPU_REGISTER offsetof(struct lowcore, percpu_register)
+
static __always_inline struct lowcore *get_lowcore(void)
{
struct lowcore *lc;
diff --git a/arch/s390/include/asm/percpu.h b/arch/s390/include/asm/percpu.h
index 3bc448c4c2dd..770fe4be65dc 100644
--- a/arch/s390/include/asm/percpu.h
+++ b/arch/s390/include/asm/percpu.h
@@ -14,17 +14,16 @@
#define arch_raw_cpu_ptr(_ptr) \
({ \
- unsigned long lc_percpu, tcp_ptr__; \
+ unsigned long tcp_ptr__; \
\
tcp_ptr__ = (__force unsigned long)(_ptr); \
- lc_percpu = offsetof(struct lowcore, percpu_offset); \
asm_inline volatile( \
ALTERNATIVE("ag %[__ptr__],%[offzero](%%r0)\n", \
"ag %[__ptr__],%[offalt](%%r0)\n", \
ALT_FEATURE(MFEATURE_LOWCORE)) \
: [__ptr__] "+d" (tcp_ptr__) \
- : [offzero] "i" (lc_percpu), \
- [offalt] "i" (lc_percpu + LOWCORE_ALT_ADDRESS), \
+ : [offzero] "i" (LC_PERCPU_OFFSET), \
+ [offalt] "i" (LC_PERCPU_OFFSET + LOWCORE_ALT_ADDRESS), \
"m" (((struct lowcore *)0)->percpu_offset) \
: "cc"); \
(TYPEOF_UNQUAL(*(_ptr)) __force __kernel *)tcp_ptr__; \
@@ -144,13 +143,10 @@
#define arch_this_cpu_add(pcp, val, op1, op2, szcast) \
do { \
- unsigned long lc_pcpr, lc_pcpo; \
typedef typeof(pcp) pcp_op_T__; \
pcp_op_T__ val__ = (val); \
pcp_op_T__ old__, *ptr__; \
\
- lc_pcpr = offsetof(struct lowcore, percpu_register); \
- lc_pcpo = offsetof(struct lowcore, percpu_offset); \
ptr__ = PERCPU_PTR(&(pcp)); \
if (__builtin_constant_p(val__) && \
((szcast)val__ > -129) && ((szcast)val__ < 128)) { \
@@ -162,8 +158,8 @@ do { \
: [ptr__] "+&a" (ptr__), "+m" (*ptr__), \
"=m" (((struct lowcore *)0)->percpu_register) \
: [val__] "i" ((szcast)val__), \
- [disppcpr] "i" (lc_pcpr), \
- [disppcpo] "i" (lc_pcpo), \
+ [disppcpr] "i" (LC_PERCPU_REGISTER), \
+ [disppcpo] "i" (LC_PERCPU_OFFSET), \
"m" (((struct lowcore *)0)->percpu_offset) \
: "cc"); \
} else { \
@@ -176,8 +172,8 @@ do { \
[ptr__] "+&a" (ptr__), "+m" (*ptr__), \
"=m" (((struct lowcore *)0)->percpu_register) \
: [val__] "d" (val__), \
- [disppcpr] "i" (lc_pcpr), \
- [disppcpo] "i" (lc_pcpo), \
+ [disppcpr] "i" (LC_PERCPU_REGISTER), \
+ [disppcpo] "i" (LC_PERCPU_OFFSET), \
"m" (((struct lowcore *)0)->percpu_offset) \
: "cc"); \
} \
@@ -188,13 +184,10 @@ do { \
#define arch_this_cpu_add_return(pcp, val, op) \
({ \
- unsigned long lc_pcpr, lc_pcpo; \
typedef typeof(pcp) pcp_op_T__; \
pcp_op_T__ val__ = (val); \
pcp_op_T__ old__, *ptr__; \
\
- lc_pcpr = offsetof(struct lowcore, percpu_register); \
- lc_pcpo = offsetof(struct lowcore, percpu_offset); \
ptr__ = PERCPU_PTR(&(pcp)); \
asm_inline volatile( \
MVIY_PERCPU("%[disppcpr]","%[ptr__]") \
@@ -205,8 +198,8 @@ do { \
[ptr__] "+&a" (ptr__), "+m" (*ptr__), \
"=m" (((struct lowcore *)0)->percpu_register) \
: [val__] "d" (val__), \
- [disppcpr] "i" (lc_pcpr), \
- [disppcpo] "i" (lc_pcpo), \
+ [disppcpr] "i" (LC_PERCPU_REGISTER), \
+ [disppcpo] "i" (LC_PERCPU_OFFSET), \
"m" (((struct lowcore *)0)->percpu_offset) \
: "cc"); \
old__ + val__; \
@@ -217,13 +210,10 @@ do { \
#define arch_this_cpu_to_op(pcp, val, op) \
do { \
- unsigned long lc_pcpr, lc_pcpo; \
typedef typeof(pcp) pcp_op_T__; \
pcp_op_T__ val__ = (val); \
pcp_op_T__ old__, *ptr__; \
\
- lc_pcpr = offsetof(struct lowcore, percpu_register); \
- lc_pcpo = offsetof(struct lowcore, percpu_offset); \
ptr__ = PERCPU_PTR(&(pcp)); \
asm_inline volatile( \
MVIY_PERCPU("%[disppcpr]","%[ptr__]") \
@@ -234,8 +224,8 @@ do { \
[ptr__] "+&a" (ptr__), "+m" (*ptr__), \
"=m" (((struct lowcore *)0)->percpu_register) \
: [val__] "d" (val__), \
- [disppcpr] "i" (lc_pcpr), \
- [disppcpo] "i" (lc_pcpo), \
+ [disppcpr] "i" (LC_PERCPU_REGISTER), \
+ [disppcpo] "i" (LC_PERCPU_OFFSET), \
"m" (((struct lowcore *)0)->percpu_offset) \
: "cc"); \
} while (0)
@@ -249,12 +239,10 @@ do { \
#define arch_this_cpu_read(pcp, op) \
({ \
- unsigned long lc_pcpr, lc_pcpo, res__; \
typedef typeof(pcp) pcp_op_T__; \
+ unsigned long res__; \
pcp_op_T__ *ptr__; \
\
- lc_pcpr = offsetof(struct lowcore, percpu_register); \
- lc_pcpo = offsetof(struct lowcore, percpu_offset); \
ptr__ = PERCPU_PTR(&(pcp)); \
asm_inline volatile( \
MVIY_PERCPU("%[disppcpr]","%[ptr__]") \
@@ -263,8 +251,8 @@ do { \
MVIY_ALT("%[disppcpr]") \
: [res__] "=&d" (res__), [ptr__] "+&a" (ptr__), \
"=m" (((struct lowcore *)0)->percpu_register) \
- : [disppcpr] "i" (lc_pcpr), \
- [disppcpo] "i" (lc_pcpo), \
+ : [disppcpr] "i" (LC_PERCPU_REGISTER), \
+ [disppcpo] "i" (LC_PERCPU_OFFSET), \
"m" (*ptr__), \
"m" (((struct lowcore *)0)->percpu_offset) \
: "cc"); \
@@ -278,12 +266,9 @@ do { \
#define arch_this_cpu_write(pcp, val, op) \
do { \
- unsigned long lc_pcpr, lc_pcpo; \
typedef typeof(pcp) pcp_op_T__; \
pcp_op_T__ *ptr__, val__ = (val); \
\
- lc_pcpr = offsetof(struct lowcore, percpu_register); \
- lc_pcpo = offsetof(struct lowcore, percpu_offset); \
ptr__ = PERCPU_PTR(&(pcp)); \
asm_inline volatile( \
MVIY_PERCPU("%[disppcpr]","%[ptr__]") \
@@ -293,8 +278,8 @@ do { \
: [ptr__] "+&a" (ptr__), "=m" (*ptr__), \
"=m" (((struct lowcore *)0)->percpu_register) \
: [val__] "d" (val__), \
- [disppcpr] "i" (lc_pcpr), \
- [disppcpo] "i" (lc_pcpo), \
+ [disppcpr] "i" (LC_PERCPU_REGISTER), \
+ [disppcpo] "i" (LC_PERCPU_OFFSET), \
"m" (((struct lowcore *)0)->percpu_offset) \
: "cc"); \
} while (0)
--
2.53.0
^ permalink raw reply [flat|nested] 16+ messages in thread* [PATCH v3 06/11] s390/percpu: Rename inline assembly symbolic names
2026-09-21 8:39 [PATCH v3 00/11] s390: More this_cpu_*() changes Heiko Carstens
` (4 preceding siblings ...)
2026-09-21 8:39 ` [PATCH v3 05/11] s390/percpu/lowcore: Add and use LC_PERCPU lowcore offset defines Heiko Carstens
@ 2026-09-21 8:40 ` Heiko Carstens
2026-09-21 8:40 ` [PATCH v3 07/11] s390/percpu: Use __PCPU_BEGIN() and __PCPU_END() for inline assemblies Heiko Carstens
` (5 subsequent siblings)
11 siblings, 0 replies; 16+ messages in thread
From: Heiko Carstens @ 2026-09-21 8:40 UTC (permalink / raw)
To: Alexander Gordeev, Sven Schnelle, Vasily Gorbik,
Christian Borntraeger, Mete Durlu, Peter Zijlstra, Mark Rutland,
Juergen Christ, Ilya Leoshkevich
Cc: linux-kernel, linux-s390
Rename percpu inline assembly symbolic names so they match (more) what is
specified with input operands. A trivial change which is supposed to make
the code a bit more readable.
Signed-off-by: Heiko Carstens <hca@linux.ibm.com>
---
arch/s390/include/asm/percpu.h | 60 +++++++++++++++++-----------------
1 file changed, 30 insertions(+), 30 deletions(-)
diff --git a/arch/s390/include/asm/percpu.h b/arch/s390/include/asm/percpu.h
index 770fe4be65dc..0df47ec2e6c7 100644
--- a/arch/s390/include/asm/percpu.h
+++ b/arch/s390/include/asm/percpu.h
@@ -151,29 +151,29 @@ do { \
if (__builtin_constant_p(val__) && \
((szcast)val__ > -129) && ((szcast)val__ < 128)) { \
asm volatile( \
- MVIY_PERCPU("%[disppcpr]","%[ptr__]") \
- AG_ALT("%[disppcpo]", "%[ptr__]") \
+ MVIY_PERCPU("%[lcreg]","%[ptr__]") \
+ AG_ALT("%[lcoff]", "%[ptr__]") \
op2 " 0(%[ptr__]),%[val__]\n" \
- MVIY_ALT("%[disppcpr]") \
+ MVIY_ALT("%[lcreg]") \
: [ptr__] "+&a" (ptr__), "+m" (*ptr__), \
"=m" (((struct lowcore *)0)->percpu_register) \
: [val__] "i" ((szcast)val__), \
- [disppcpr] "i" (LC_PERCPU_REGISTER), \
- [disppcpo] "i" (LC_PERCPU_OFFSET), \
+ [lcreg] "i" (LC_PERCPU_REGISTER), \
+ [lcoff] "i" (LC_PERCPU_OFFSET), \
"m" (((struct lowcore *)0)->percpu_offset) \
: "cc"); \
} else { \
asm volatile( \
- MVIY_PERCPU("%[disppcpr]", "%[ptr__]") \
- AG_ALT("%[disppcpo]", "%[ptr__]") \
+ MVIY_PERCPU("%[lcreg]", "%[ptr__]") \
+ AG_ALT("%[lcoff]", "%[ptr__]") \
op1 " %[old__],%[val__],0(%[ptr__])\n" \
- MVIY_ALT("%[disppcpr]") \
+ MVIY_ALT("%[lcreg]") \
: [old__] "=&d" (old__), \
[ptr__] "+&a" (ptr__), "+m" (*ptr__), \
"=m" (((struct lowcore *)0)->percpu_register) \
: [val__] "d" (val__), \
- [disppcpr] "i" (LC_PERCPU_REGISTER), \
- [disppcpo] "i" (LC_PERCPU_OFFSET), \
+ [lcreg] "i" (LC_PERCPU_REGISTER), \
+ [lcoff] "i" (LC_PERCPU_OFFSET), \
"m" (((struct lowcore *)0)->percpu_offset) \
: "cc"); \
} \
@@ -190,16 +190,16 @@ do { \
\
ptr__ = PERCPU_PTR(&(pcp)); \
asm_inline volatile( \
- MVIY_PERCPU("%[disppcpr]","%[ptr__]") \
- AG_ALT("%[disppcpo]","%[ptr__]") \
+ MVIY_PERCPU("%[lcreg]","%[ptr__]") \
+ AG_ALT("%[lcoff]","%[ptr__]") \
op " %[old__],%[val__],0(%[ptr__])\n" \
- MVIY_ALT("%[disppcpr]") \
+ MVIY_ALT("%[lcreg]") \
: [old__] "=&d" (old__), \
[ptr__] "+&a" (ptr__), "+m" (*ptr__), \
"=m" (((struct lowcore *)0)->percpu_register) \
: [val__] "d" (val__), \
- [disppcpr] "i" (LC_PERCPU_REGISTER), \
- [disppcpo] "i" (LC_PERCPU_OFFSET), \
+ [lcreg] "i" (LC_PERCPU_REGISTER), \
+ [lcoff] "i" (LC_PERCPU_OFFSET), \
"m" (((struct lowcore *)0)->percpu_offset) \
: "cc"); \
old__ + val__; \
@@ -216,16 +216,16 @@ do { \
\
ptr__ = PERCPU_PTR(&(pcp)); \
asm_inline volatile( \
- MVIY_PERCPU("%[disppcpr]","%[ptr__]") \
- AG_ALT("%[disppcpo]","%[ptr__]") \
+ MVIY_PERCPU("%[lcreg]","%[ptr__]") \
+ AG_ALT("%[lcoff]","%[ptr__]") \
op " %[old__],%[val__],0(%[ptr__])\n" \
- MVIY_ALT("%[disppcpr]") \
+ MVIY_ALT("%[lcreg]") \
: [old__] "=&d" (old__), \
[ptr__] "+&a" (ptr__), "+m" (*ptr__), \
"=m" (((struct lowcore *)0)->percpu_register) \
: [val__] "d" (val__), \
- [disppcpr] "i" (LC_PERCPU_REGISTER), \
- [disppcpo] "i" (LC_PERCPU_OFFSET), \
+ [lcreg] "i" (LC_PERCPU_REGISTER), \
+ [lcoff] "i" (LC_PERCPU_OFFSET), \
"m" (((struct lowcore *)0)->percpu_offset) \
: "cc"); \
} while (0)
@@ -245,14 +245,14 @@ do { \
\
ptr__ = PERCPU_PTR(&(pcp)); \
asm_inline volatile( \
- MVIY_PERCPU("%[disppcpr]","%[ptr__]") \
- AG_ALT("%[disppcpo]","%[ptr__]") \
+ MVIY_PERCPU("%[lcreg]","%[ptr__]") \
+ AG_ALT("%[lcoff]","%[ptr__]") \
op " %[res__],0(%[ptr__])\n" \
- MVIY_ALT("%[disppcpr]") \
+ MVIY_ALT("%[lcreg]") \
: [res__] "=&d" (res__), [ptr__] "+&a" (ptr__), \
"=m" (((struct lowcore *)0)->percpu_register) \
- : [disppcpr] "i" (LC_PERCPU_REGISTER), \
- [disppcpo] "i" (LC_PERCPU_OFFSET), \
+ : [lcreg] "i" (LC_PERCPU_REGISTER), \
+ [lcoff] "i" (LC_PERCPU_OFFSET), \
"m" (*ptr__), \
"m" (((struct lowcore *)0)->percpu_offset) \
: "cc"); \
@@ -271,15 +271,15 @@ do { \
\
ptr__ = PERCPU_PTR(&(pcp)); \
asm_inline volatile( \
- MVIY_PERCPU("%[disppcpr]","%[ptr__]") \
- AG_ALT("%[disppcpo]","%[ptr__]") \
+ MVIY_PERCPU("%[lcreg]","%[ptr__]") \
+ AG_ALT("%[lcoff]","%[ptr__]") \
op " %[val__],0(%[ptr__])\n" \
- MVIY_ALT("%[disppcpr]") \
+ MVIY_ALT("%[lcreg]") \
: [ptr__] "+&a" (ptr__), "=m" (*ptr__), \
"=m" (((struct lowcore *)0)->percpu_register) \
: [val__] "d" (val__), \
- [disppcpr] "i" (LC_PERCPU_REGISTER), \
- [disppcpo] "i" (LC_PERCPU_OFFSET), \
+ [lcreg] "i" (LC_PERCPU_REGISTER), \
+ [lcoff] "i" (LC_PERCPU_OFFSET), \
"m" (((struct lowcore *)0)->percpu_offset) \
: "cc"); \
} while (0)
--
2.53.0
^ permalink raw reply [flat|nested] 16+ messages in thread* [PATCH v3 07/11] s390/percpu: Use __PCPU_BEGIN() and __PCPU_END() for inline assemblies
2026-09-21 8:39 [PATCH v3 00/11] s390: More this_cpu_*() changes Heiko Carstens
` (5 preceding siblings ...)
2026-09-21 8:40 ` [PATCH v3 06/11] s390/percpu: Rename inline assembly symbolic names Heiko Carstens
@ 2026-09-21 8:40 ` Heiko Carstens
2026-09-21 8:40 ` [PATCH v3 08/11] s390/percpu: Use percpu code section for this_cpu_cmpxchg128() Heiko Carstens
` (4 subsequent siblings)
11 siblings, 0 replies; 16+ messages in thread
From: Heiko Carstens @ 2026-09-21 8:40 UTC (permalink / raw)
To: Alexander Gordeev, Sven Schnelle, Vasily Gorbik,
Christian Borntraeger, Mete Durlu, Peter Zijlstra, Mark Rutland,
Juergen Christ, Ilya Leoshkevich
Cc: linux-kernel, linux-s390
Replace the use of MVIY_PERCPU(), AG_ALT(), and MVIY_ALT() within all
percpu inline assemblies with __PCPU_BEGIN() and __PCPU_END(), since
the begin and end of each inline assembly is the same.
Also use consistent names: e.g. __PCPU_MVIY() and __PCPU_AG() instead of
the previous MVIY_PERCPU() and AG_ALT().
Furthermore extract DEFINE_GR_NUM and UNDEF_GR_NUM which are used to
convert register strings to register numbers.
All of this should make the code more readable. No functional change.
Signed-off-by: Heiko Carstens <hca@linux.ibm.com>
---
arch/s390/include/asm/percpu.h | 83 +++++++++++++++++-----------------
1 file changed, 42 insertions(+), 41 deletions(-)
diff --git a/arch/s390/include/asm/percpu.h b/arch/s390/include/asm/percpu.h
index 0df47ec2e6c7..78c3aaa138f7 100644
--- a/arch/s390/include/asm/percpu.h
+++ b/arch/s390/include/asm/percpu.h
@@ -92,42 +92,49 @@
*
* Inline assemblies making use of this typically have a code sequence like:
*
- * MVIY_PERCPU(...) <- start of percpu code section
- * AG_ALT(...) <- add percpu offset; must be the second instruction
- * atomic_op <- atomic op
- * MVIY_ALT(...) <- end of percpu code section
+ * __PCPU_BEGIN(...) <- start of percpu code section
+ * atomic_op <- atomic op
+ * __PCPU_END(...) <- end of percpu code section
*/
#define LC_ALT_ADDR __stringify(LOWCORE_ALT_ADDRESS)
-#define MVIY_PERCPU(disp, reg) \
- ".macro GEN_MVIY disp, reg\n" \
- ".set .Lreg,255\n" \
- ".irp rs,0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15\n" \
- " .ifc \\reg,%%r\\rs\n" \
- " .set .Lreg,\\rs\n" \
+#define DEFINE_GR_NUM \
+ ".macro _GR_NUM opd, gr\n" \
+ ".set \\opd,255\n" \
+ ".irp rs,0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15\n" \
+ " .ifc \\gr,%%r\\rs\n" \
+ " \\opd = \\rs\n" \
" .endif\n" \
".endr\n" \
- ".if .Lreg == 255\n" \
+ ".if \\opd == 255\n" \
" .error \"Illegal register number\"\n" \
".endif\n" \
- "mviy \\disp(%%r0),.Lreg\n" \
- ".endm\n" \
- ALTERNATIVE("GEN_MVIY " disp " , " reg "\n", \
- "GEN_MVIY " disp "+" LC_ALT_ADDR " , " reg "\n", \
- ALT_FEATURE(MFEATURE_LOWCORE)) \
- ".purgem GEN_MVIY\n"
+ ".endm\n"
-#define MVIY_ALT(disp) \
- ALTERNATIVE(" mviy " disp "(%%r0),0\n", \
- " mviy " disp "+" LC_ALT_ADDR "(%%r0),0\n", \
+#define UNDEF_GR_NUM \
+ ".purgem _GR_NUM\n"
+
+#define __PCPU_MVIY(lcreg, imm) \
+ ALTERNATIVE(" mviy " lcreg "(%%r0)," imm "\n", \
+ " mviy " lcreg "+" LC_ALT_ADDR "(%%r0)," imm "\n", \
ALT_FEATURE(MFEATURE_LOWCORE))
-#define AG_ALT(disp, reg) \
- ALTERNATIVE(" ag " reg ", " disp "(%%r0)\n", \
- " ag " reg ", " disp "+" LC_ALT_ADDR "(%%r0)\n", \
+#define __PCPU_AG(reg, lcoff) \
+ ALTERNATIVE(" ag " reg ", " lcoff "(%%r0)\n", \
+ " ag " reg ", " lcoff "+" LC_ALT_ADDR "(%%r0)\n", \
ALT_FEATURE(MFEATURE_LOWCORE))
+#define __PCPU_BEGIN(lcreg, lcoff, reg) \
+ DEFINE_GR_NUM \
+ "_GR_NUM .Lreg, " reg "\n" \
+ UNDEF_GR_NUM \
+ __PCPU_MVIY(lcreg, ".Lreg") \
+ __PCPU_AG(reg, lcoff)
+
+#define __PCPU_END(lcreg) \
+ __PCPU_MVIY(lcreg, "0")
+
#ifndef MARCH_HAS_Z196_FEATURES
#define this_cpu_add_4(pcp, val) arch_this_cpu_to_op_simple(pcp, val, +)
@@ -151,10 +158,9 @@ do { \
if (__builtin_constant_p(val__) && \
((szcast)val__ > -129) && ((szcast)val__ < 128)) { \
asm volatile( \
- MVIY_PERCPU("%[lcreg]","%[ptr__]") \
- AG_ALT("%[lcoff]", "%[ptr__]") \
+ __PCPU_BEGIN("%[lcreg]","%[lcoff]","%[ptr__]") \
op2 " 0(%[ptr__]),%[val__]\n" \
- MVIY_ALT("%[lcreg]") \
+ __PCPU_END("%[lcreg]") \
: [ptr__] "+&a" (ptr__), "+m" (*ptr__), \
"=m" (((struct lowcore *)0)->percpu_register) \
: [val__] "i" ((szcast)val__), \
@@ -164,10 +170,9 @@ do { \
: "cc"); \
} else { \
asm volatile( \
- MVIY_PERCPU("%[lcreg]", "%[ptr__]") \
- AG_ALT("%[lcoff]", "%[ptr__]") \
+ __PCPU_BEGIN("%[lcreg]","%[lcoff]","%[ptr__]") \
op1 " %[old__],%[val__],0(%[ptr__])\n" \
- MVIY_ALT("%[lcreg]") \
+ __PCPU_END("%[lcreg]") \
: [old__] "=&d" (old__), \
[ptr__] "+&a" (ptr__), "+m" (*ptr__), \
"=m" (((struct lowcore *)0)->percpu_register) \
@@ -190,10 +195,9 @@ do { \
\
ptr__ = PERCPU_PTR(&(pcp)); \
asm_inline volatile( \
- MVIY_PERCPU("%[lcreg]","%[ptr__]") \
- AG_ALT("%[lcoff]","%[ptr__]") \
+ __PCPU_BEGIN("%[lcreg]","%[lcoff]","%[ptr__]") \
op " %[old__],%[val__],0(%[ptr__])\n" \
- MVIY_ALT("%[lcreg]") \
+ __PCPU_END("%[lcreg]") \
: [old__] "=&d" (old__), \
[ptr__] "+&a" (ptr__), "+m" (*ptr__), \
"=m" (((struct lowcore *)0)->percpu_register) \
@@ -216,10 +220,9 @@ do { \
\
ptr__ = PERCPU_PTR(&(pcp)); \
asm_inline volatile( \
- MVIY_PERCPU("%[lcreg]","%[ptr__]") \
- AG_ALT("%[lcoff]","%[ptr__]") \
+ __PCPU_BEGIN("%[lcreg]","%[lcoff]","%[ptr__]") \
op " %[old__],%[val__],0(%[ptr__])\n" \
- MVIY_ALT("%[lcreg]") \
+ __PCPU_END("%[lcreg]") \
: [old__] "=&d" (old__), \
[ptr__] "+&a" (ptr__), "+m" (*ptr__), \
"=m" (((struct lowcore *)0)->percpu_register) \
@@ -245,10 +248,9 @@ do { \
\
ptr__ = PERCPU_PTR(&(pcp)); \
asm_inline volatile( \
- MVIY_PERCPU("%[lcreg]","%[ptr__]") \
- AG_ALT("%[lcoff]","%[ptr__]") \
+ __PCPU_BEGIN("%[lcreg]","%[lcoff]","%[ptr__]") \
op " %[res__],0(%[ptr__])\n" \
- MVIY_ALT("%[lcreg]") \
+ __PCPU_END("%[lcreg]") \
: [res__] "=&d" (res__), [ptr__] "+&a" (ptr__), \
"=m" (((struct lowcore *)0)->percpu_register) \
: [lcreg] "i" (LC_PERCPU_REGISTER), \
@@ -271,10 +273,9 @@ do { \
\
ptr__ = PERCPU_PTR(&(pcp)); \
asm_inline volatile( \
- MVIY_PERCPU("%[lcreg]","%[ptr__]") \
- AG_ALT("%[lcoff]","%[ptr__]") \
+ __PCPU_BEGIN("%[lcreg]","%[lcoff]","%[ptr__]") \
op " %[val__],0(%[ptr__])\n" \
- MVIY_ALT("%[lcreg]") \
+ __PCPU_END("%[lcreg]") \
: [ptr__] "+&a" (ptr__), "=m" (*ptr__), \
"=m" (((struct lowcore *)0)->percpu_register) \
: [val__] "d" (val__), \
--
2.53.0
^ permalink raw reply [flat|nested] 16+ messages in thread* [PATCH v3 08/11] s390/percpu: Use percpu code section for this_cpu_cmpxchg128()
2026-09-21 8:39 [PATCH v3 00/11] s390: More this_cpu_*() changes Heiko Carstens
` (6 preceding siblings ...)
2026-09-21 8:40 ` [PATCH v3 07/11] s390/percpu: Use __PCPU_BEGIN() and __PCPU_END() for inline assemblies Heiko Carstens
@ 2026-09-21 8:40 ` Heiko Carstens
2026-09-21 8:40 ` [PATCH v3 09/11] s390/percpu: Use percpu code section for this_cpu_xchg() Heiko Carstens
` (3 subsequent siblings)
11 siblings, 0 replies; 16+ messages in thread
From: Heiko Carstens @ 2026-09-21 8:40 UTC (permalink / raw)
To: Alexander Gordeev, Sven Schnelle, Vasily Gorbik,
Christian Borntraeger, Mete Durlu, Peter Zijlstra, Mark Rutland,
Juergen Christ, Ilya Leoshkevich
Cc: linux-kernel, linux-s390
Convert this_cpu_cmpxchg128() to make use of percpu code section
functionality. This is just for the sake of completeness, since
currently there is no in-kernel user of this_cpu_cmpxchg128().
Signed-off-by: Heiko Carstens <hca@linux.ibm.com>
---
arch/s390/include/asm/percpu.h | 24 ++++++++++++++++--------
1 file changed, 16 insertions(+), 8 deletions(-)
diff --git a/arch/s390/include/asm/percpu.h b/arch/s390/include/asm/percpu.h
index 78c3aaa138f7..0eae65937c0d 100644
--- a/arch/s390/include/asm/percpu.h
+++ b/arch/s390/include/asm/percpu.h
@@ -312,15 +312,23 @@ do { \
#define this_cpu_cmpxchg128(pcp, oval, nval) \
({ \
typedef typeof(pcp) pcp_op_T__; \
- u128 old__, new__, ret__; \
+ u128 old__ = (oval), new__ = (nval); \
pcp_op_T__ *ptr__; \
- old__ = oval; \
- new__ = nval; \
- preempt_disable_notrace(); \
- ptr__ = raw_cpu_ptr(&(pcp)); \
- ret__ = cmpxchg128((void *)ptr__, old__, new__); \
- preempt_enable_notrace(); \
- ret__; \
+ \
+ ptr__ = PERCPU_PTR(&(pcp)); \
+ asm_inline volatile( \
+ __PCPU_BEGIN("%[lcreg]","%[lcoff]","%[ptr__]") \
+ " cdsg %[old__],%[new__],0(%[ptr__])\n" \
+ __PCPU_END("%[lcreg]") \
+ : [old__] "+&d" (old__), [ptr__] "+&a" (ptr__), \
+ "+m" (*ptr__), \
+ "=m" (((struct lowcore *)0)->percpu_register) \
+ : [new__] "d" (new__), \
+ [lcreg] "i" (LC_PERCPU_REGISTER), \
+ [lcoff] "i" (LC_PERCPU_OFFSET), \
+ "m" (((struct lowcore *)0)->percpu_offset) \
+ : "memory", "cc"); \
+ old__; \
})
#define arch_this_cpu_xchg(pcp, nval) \
--
2.53.0
^ permalink raw reply [flat|nested] 16+ messages in thread* [PATCH v3 09/11] s390/percpu: Use percpu code section for this_cpu_xchg()
2026-09-21 8:39 [PATCH v3 00/11] s390: More this_cpu_*() changes Heiko Carstens
` (7 preceding siblings ...)
2026-09-21 8:40 ` [PATCH v3 08/11] s390/percpu: Use percpu code section for this_cpu_cmpxchg128() Heiko Carstens
@ 2026-09-21 8:40 ` Heiko Carstens
2026-09-21 8:40 ` [PATCH v3 10/11] s390/percpu: Use percpu code section for this_cpu_cmpxchg() Heiko Carstens
` (2 subsequent siblings)
11 siblings, 0 replies; 16+ messages in thread
From: Heiko Carstens @ 2026-09-21 8:40 UTC (permalink / raw)
To: Alexander Gordeev, Sven Schnelle, Vasily Gorbik,
Christian Borntraeger, Mete Durlu, Peter Zijlstra, Mark Rutland,
Juergen Christ, Ilya Leoshkevich
Cc: linux-kernel, linux-s390
Convert this_cpu_xchg() to make use of percpu code section
functionality in order to get rid of preempt_disable() and
preempt_enable() calls.
Consider only four and eight byte usages, since there are no compare and
swap instructions for one and two bytes available. For one and two byte
use the previous simple implementation.
Signed-off-by: Heiko Carstens <hca@linux.ibm.com>
---
arch/s390/include/asm/percpu.h | 34 +++++++++++++++++++++++++++++-----
1 file changed, 29 insertions(+), 5 deletions(-)
diff --git a/arch/s390/include/asm/percpu.h b/arch/s390/include/asm/percpu.h
index 0eae65937c0d..237bf8c80843 100644
--- a/arch/s390/include/asm/percpu.h
+++ b/arch/s390/include/asm/percpu.h
@@ -331,7 +331,7 @@ do { \
old__; \
})
-#define arch_this_cpu_xchg(pcp, nval) \
+#define arch_this_cpu_xchg_simple(pcp, nval) \
({ \
typeof(pcp) *ptr__; \
typeof(pcp) ret__; \
@@ -342,10 +342,34 @@ do { \
ret__; \
})
-#define this_cpu_xchg_1(pcp, nval) arch_this_cpu_xchg(pcp, nval)
-#define this_cpu_xchg_2(pcp, nval) arch_this_cpu_xchg(pcp, nval)
-#define this_cpu_xchg_4(pcp, nval) arch_this_cpu_xchg(pcp, nval)
-#define this_cpu_xchg_8(pcp, nval) arch_this_cpu_xchg(pcp, nval)
+#define arch_this_cpu_xchg(pcp, nval, ldop, csop) \
+({ \
+ typedef typeof(pcp) pcp_op_T__; \
+ pcp_op_T__ old__, new__ = (nval); \
+ pcp_op_T__ *ptr__; \
+ \
+ ptr__ = PERCPU_PTR(&(pcp)); \
+ asm_inline volatile( \
+ __PCPU_BEGIN("%[lcreg]","%[lcoff]","%[ptr__]") \
+ " " ldop " %[old__],0(%[ptr__])\n" \
+ "0: " csop " %[old__],%[new__],0(%[ptr__])\n"\
+ " jnz 0b\n" \
+ __PCPU_END("%[lcreg]") \
+ : [old__] "=&d" (old__), \
+ [ptr__] "+&a" (ptr__), "+m" (*ptr__), \
+ "=m" (((struct lowcore *)0)->percpu_register) \
+ : [new__] "d" (new__), \
+ [lcreg] "i" (LC_PERCPU_REGISTER), \
+ [lcoff] "i" (LC_PERCPU_OFFSET), \
+ "m" (((struct lowcore *)0)->percpu_offset) \
+ : "memory", "cc"); \
+ old__; \
+})
+
+#define this_cpu_xchg_1(pcp, nval) arch_this_cpu_xchg_simple(pcp, nval)
+#define this_cpu_xchg_2(pcp, nval) arch_this_cpu_xchg_simple(pcp, nval)
+#define this_cpu_xchg_4(pcp, nval) arch_this_cpu_xchg(pcp, nval, "l", "cs")
+#define this_cpu_xchg_8(pcp, nval) arch_this_cpu_xchg(pcp, nval, "lg", "csg")
#include <asm-generic/percpu.h>
--
2.53.0
^ permalink raw reply [flat|nested] 16+ messages in thread* [PATCH v3 10/11] s390/percpu: Use percpu code section for this_cpu_cmpxchg()
2026-09-21 8:39 [PATCH v3 00/11] s390: More this_cpu_*() changes Heiko Carstens
` (8 preceding siblings ...)
2026-09-21 8:40 ` [PATCH v3 09/11] s390/percpu: Use percpu code section for this_cpu_xchg() Heiko Carstens
@ 2026-09-21 8:40 ` Heiko Carstens
2026-09-21 8:40 ` [PATCH v3 11/11] s390/percpu: Rework to simplify percpu_entry() and percpu_exit() Heiko Carstens
2026-09-21 9:15 ` [PATCH v3 00/11] s390: More this_cpu_*() changes Heiko Carstens
11 siblings, 0 replies; 16+ messages in thread
From: Heiko Carstens @ 2026-09-21 8:40 UTC (permalink / raw)
To: Alexander Gordeev, Sven Schnelle, Vasily Gorbik,
Christian Borntraeger, Mete Durlu, Peter Zijlstra, Mark Rutland,
Juergen Christ, Ilya Leoshkevich
Cc: linux-kernel, linux-s390
Convert this_cpu_cmpxchg() to make use of percpu code section
functionality in order to get rid of preempt_disable() and
preempt_enable() calls.
Consider only four and eight byte usages, since there are no compare
and swap instructions for one and two bytes available. For one and
two bytes use the previous simple implementation.
Signed-off-by: Heiko Carstens <hca@linux.ibm.com>
---
arch/s390/include/asm/percpu.h | 32 +++++++++++++++++++++++++++-----
1 file changed, 27 insertions(+), 5 deletions(-)
diff --git a/arch/s390/include/asm/percpu.h b/arch/s390/include/asm/percpu.h
index 237bf8c80843..46368131b932 100644
--- a/arch/s390/include/asm/percpu.h
+++ b/arch/s390/include/asm/percpu.h
@@ -290,7 +290,7 @@ do { \
#define this_cpu_write_4(pcp, val) arch_this_cpu_write(pcp, val, "st")
#define this_cpu_write_8(pcp, val) arch_this_cpu_write(pcp, val, "stg")
-#define arch_this_cpu_cmpxchg(pcp, oval, nval) \
+#define arch_this_cpu_cmpxchg_simple(pcp, oval, nval) \
({ \
typedef typeof(pcp) pcp_op_T__; \
pcp_op_T__ ret__; \
@@ -302,10 +302,32 @@ do { \
ret__; \
})
-#define this_cpu_cmpxchg_1(pcp, oval, nval) arch_this_cpu_cmpxchg(pcp, oval, nval)
-#define this_cpu_cmpxchg_2(pcp, oval, nval) arch_this_cpu_cmpxchg(pcp, oval, nval)
-#define this_cpu_cmpxchg_4(pcp, oval, nval) arch_this_cpu_cmpxchg(pcp, oval, nval)
-#define this_cpu_cmpxchg_8(pcp, oval, nval) arch_this_cpu_cmpxchg(pcp, oval, nval)
+#define arch_this_cpu_cmpxchg(pcp, oval, nval, op) \
+({ \
+ typedef typeof(pcp) pcp_op_T__; \
+ pcp_op_T__ old__ = (oval), new__ = (nval); \
+ pcp_op_T__ *ptr__; \
+ \
+ ptr__ = PERCPU_PTR(&(pcp)); \
+ asm_inline volatile( \
+ __PCPU_BEGIN("%[lcreg]","%[lcoff]","%[ptr__]") \
+ op " %[old__],%[new__],0(%[ptr__])\n" \
+ __PCPU_END("%[lcreg]") \
+ : [old__] "+&d" (old__), \
+ [ptr__] "+&a" (ptr__), "+m" (*ptr__), \
+ "=m" (((struct lowcore *)0)->percpu_register) \
+ : [new__] "d" (new__), \
+ [lcreg] "i" (LC_PERCPU_REGISTER), \
+ [lcoff] "i" (LC_PERCPU_OFFSET), \
+ "m" (((struct lowcore *)0)->percpu_offset) \
+ : "memory", "cc"); \
+ old__; \
+})
+
+#define this_cpu_cmpxchg_1(pcp, oval, nval) arch_this_cpu_cmpxchg_simple(pcp, oval, nval)
+#define this_cpu_cmpxchg_2(pcp, oval, nval) arch_this_cpu_cmpxchg_simple(pcp, oval, nval)
+#define this_cpu_cmpxchg_4(pcp, oval, nval) arch_this_cpu_cmpxchg(pcp, oval, nval, "cs")
+#define this_cpu_cmpxchg_8(pcp, oval, nval) arch_this_cpu_cmpxchg(pcp, oval, nval, "csg")
#define this_cpu_cmpxchg64(pcp, o, n) this_cpu_cmpxchg_8(pcp, o, n)
--
2.53.0
^ permalink raw reply [flat|nested] 16+ messages in thread* [PATCH v3 11/11] s390/percpu: Rework to simplify percpu_entry() and percpu_exit()
2026-09-21 8:39 [PATCH v3 00/11] s390: More this_cpu_*() changes Heiko Carstens
` (9 preceding siblings ...)
2026-09-21 8:40 ` [PATCH v3 10/11] s390/percpu: Use percpu code section for this_cpu_cmpxchg() Heiko Carstens
@ 2026-09-21 8:40 ` Heiko Carstens
2026-09-21 9:55 ` David Laight
2026-09-21 9:15 ` [PATCH v3 00/11] s390: More this_cpu_*() changes Heiko Carstens
11 siblings, 1 reply; 16+ messages in thread
From: Heiko Carstens @ 2026-09-21 8:40 UTC (permalink / raw)
To: Alexander Gordeev, Sven Schnelle, Vasily Gorbik,
Christian Borntraeger, Mete Durlu, Peter Zijlstra, Mark Rutland,
Juergen Christ, Ilya Leoshkevich
Cc: linux-kernel, linux-s390
The percpu code section functionality uses a rather complex method to
figure out if the register, which contains the address of the current
cpu's percpu variable, needs to be adjusted.
If an interrupt happens within a percpu code section (indicated by a
lowcore field), the instruction at the interrupted location is
checked. If it is not a specific AG instruction, the register needs to
be updated. This mechanism needs to take kprobes into account, and
enforces a specific instruction ordering.
Mark Rutland provided a different solution for arm64 [1] which comes
without such limitations, but requires to use one more instruction, and
two more registers. Given that this simplifies percpu_entry() and
percpu_exit() it seems to be worth to go that route.
Change s390 to implement a similar approach. This requires to encode
three register numbers into the "percpu_register" field, which is used
to indicate if a percpu code section is executed.
The used mviy instruction can write only one byte, which allows to
encode only two register numbers. Use a register pair for the inline
assemblies, and only encode the even register number of the register
pair to work around this.
The generated code changes like this for e.g. a simple this_cpu_inc():
Old:
c0 20 00 00 00 00 larl %r2,c6 <foo+0x6> <-- load address of percpu var
b9 04 00 32 lgr %r3,%r2 <-- pointless copy of address
eb 03 03 c0 00 52 mviy 960,3 <-- start of percpu code section
- gpr 3 contains percpu var address
e3 30 03 b8 00 08 ag %r3,952 <-- add percpu offset
eb 01 30 00 00 7a agsi 0(%r3),1 <-- atomic inc
eb 00 03 c0 00 52 mviy 960,0 <-- end of percpu code section
New:
c0 10 00 00 00 00 larl %r1,c6 <foo+0x6> <-- load address of percpu var
eb 21 03 c0 00 52 mviy 960,33 <-- start of percpu code section
33 == 0x21:
- gpr 1 contains percpu var address
- gpr 2 used for percpu offset
- gpr 2+1 == 3 used for current cpu's percpu var address
e3 20 03 b8 00 04 lg %r2,952 <-- load percpu offset
41 32 10 00 la %r3,0(%r2,%r1) <-- generate current cpu's percpu var address
eb 01 30 00 00 7a agsi 0(%r3),1 <-- atomic inc
eb 00 03 c0 00 52 mviy 960,0 <-- end of percpu code section
In the above "new" code example 33 (0x21) is used as indicator value to
mark that a percpu code section is executed. This value implies that
registers 2 and 3 will (only) hold the percpu offset and the current
cpu's percpu var address. Those registers will be updated by
percpu_exit() if the process was migrated to a different cpu to contain
the percpu offset and percpu var address of the new cpu.
Note that because of the pointless lgr instruction in the "old" code
example the two code sequences have identical size, and it looks like
only one more register is used. However this is because of suboptimal
gcc code generation.
[1] https://lore.kernel.org/all/20260908151741.394589-1-mark.rutland@arm.com/
Signed-off-by: Heiko Carstens <hca@linux.ibm.com>
---
arch/s390/include/asm/entry-percpu.h | 71 +++--------
arch/s390/include/asm/percpu.h | 177 +++++++++++++++++----------
arch/s390/kernel/irq.c | 10 +-
arch/s390/kernel/nmi.c | 4 +-
arch/s390/kernel/traps.c | 4 +-
5 files changed, 138 insertions(+), 128 deletions(-)
diff --git a/arch/s390/include/asm/entry-percpu.h b/arch/s390/include/asm/entry-percpu.h
index 89c9e44f6f83..f6f79ec4bee3 100644
--- a/arch/s390/include/asm/entry-percpu.h
+++ b/arch/s390/include/asm/entry-percpu.h
@@ -2,79 +2,42 @@
#ifndef ARCH_S390_ENTRY_PERCPU_H
#define ARCH_S390_ENTRY_PERCPU_H
-#include <linux/kprobes.h>
+#include <linux/bitfield.h>
#include <linux/percpu.h>
#include <asm/lowcore.h>
#include <asm/ptrace.h>
-#include <asm/asm-offsets.h>
static __always_inline void percpu_entry(struct pt_regs *regs)
{
struct lowcore *lc = get_lowcore();
- if (user_mode(regs))
- return;
regs->cpu = lc->cpu_nr;
regs->percpu_register = lc->percpu_register;
lc->percpu_register = 0;
}
-static __always_inline bool percpu_code_check(struct pt_regs *regs)
-{
- unsigned int insn, disp;
- struct kprobe *p;
-
- if (likely(user_mode(regs) || !regs->percpu_register))
- return false;
- /*
- * Within a percpu code section - check if the percpu base register
- * needs to be updated. This is the case if the PSW does not point to
- * the ADD instruction within the section.
- * - AG %rx,percpu_offset_in_lowcore(%r0,%r0)
- * which adds the percpu offset to the percpu base register.
- */
- lockdep_assert_preemption_disabled();
-again:
- insn = READ_ONCE(*(u16 *)psw_bits(regs->psw).ia);
- if (unlikely(insn == BREAKPOINT_INSTRUCTION)) {
- p = get_kprobe((void *)psw_bits(regs->psw).ia);
- /*
- * If the kprobe is concurrently removed on a different CPU
- * it might not be found anymore. However text must have
- * been restored - try again.
- */
- if (!p)
- goto again;
- insn = p->opcode;
- }
- if ((insn & 0xff0f) != 0xe300)
- return true;
- disp = LC_PERCPU_OFFSET;
- if (machine_has_relocated_lowcore())
- disp += LOWCORE_ALT_ADDRESS;
- insn = (disp & 0xff000) >> 4 | (disp & 0x00fff) << 16 | 0x8;
- if (*(u32 *)(psw_bits(regs->psw).ia + 2) != insn)
- return true;
- return false;
-}
-
-static __always_inline void percpu_exit(struct pt_regs *regs, bool needs_fixup)
+static __always_inline void percpu_exit(struct pt_regs *regs)
{
+ unsigned char regval, regpcp, regoff, regptr;
struct lowcore *lc = get_lowcore();
- unsigned char reg;
- if (user_mode(regs))
+ if (!regs->percpu_register)
return;
- reg = regs->percpu_register;
- lc->percpu_register = reg;
- if (likely(!needs_fixup))
- return;
- /* Check if process has been migrated to a different CPU. */
+ regval = regs->percpu_register;
+ lc->percpu_register = regval;
+ /* Migrated to a different CPU? */
if (regs->cpu == lc->cpu_nr)
return;
- /* Fixup percpu base register */
- regs->gprs[reg] -= __per_cpu_offset[regs->cpu];
- regs->gprs[reg] += lc->percpu_offset;
+ regpcp = FIELD_GET(PCPU_REG_PCP, regval);
+ regoff = FIELD_GET(PCPU_REG_OFF, regval);
+ regptr = regoff + 1;
+ /*
+ * Update register 'regoff' with the current CPU's percpu offset,
+ * and recalculate and update current CPU's percpu variable address
+ * contained in register 'regptr'.
+ */
+ regs->gprs[regoff] = lc->percpu_offset;
+ regs->gprs[regptr] = regs->gprs[regpcp] + regs->gprs[regoff];
}
#endif
diff --git a/arch/s390/include/asm/percpu.h b/arch/s390/include/asm/percpu.h
index 46368131b932..db30f5ac4042 100644
--- a/arch/s390/include/asm/percpu.h
+++ b/arch/s390/include/asm/percpu.h
@@ -67,28 +67,32 @@
* this. The idea is that this_cpu operations based on atomic instructions are
* guarded with mviy instructions:
*
- * - The first mviy instruction writes the register number, which contains the
- * percpu address variable to lowcore. This also indicates that a percpu
- * code section is executed.
+ * - The first mviy instruction writes the register number of the percpu address
+ * variable and the even register number of a register pair (which encodes two
+ * registers: the even register for the percpu offset and the odd register for
+ * the percpu pointer) to lowcore. This also indicates that a percpu code
+ * section is executed.
*
- * - The first mviy instruction following the mviy instruction must be the ag
- * instruction which adds the percpu offset to the percpu address register.
+ * - The mviy instruction is followed by the lg instruction which loads the
+ * percpu offset into the even register of the pair and the la instruction
+ * which adds the percpu offset and the percpu address into the odd register
+ * of the pair (the percpu pointer register).
*
* - Afterwards the atomic percpu operation follows.
*
* - Then a second mviy instruction writes a zero to lowcore, which indicates
* the end of the percpu code section.
*
- * - In case of an interrupt/exception/nmi the register number which was
- * written to lowcore is copied to the exception frame (pt_regs), and a zero
- * is written to lowcore.
+ * - In case of an interrupt/exception/nmi the encoded register numbers which
+ * were written to lowcore are copied to the exception frame (pt_regs), and a
+ * zero is written to lowcore.
*
* - On return to the previous context it is checked if a percpu code section
- * was executed (saved register number not zero), and if the process was
- * migrated to a different cpu. If the percpu offset was already added to
- * the percpu address register (instruction address does _not_ point to the
- * ag instruction) the content of the percpu address register is adjusted so
- * it points to percpu variable of the new cpu.
+ * was executed (saved register value not zero), and if the process was
+ * migrated to a different cpu. The content of the percpu offset register
+ * (even register of the pair) is reloaded with the current cpu's percpu
+ * offset and the percpu pointer register (odd register of the pair) is
+ * recalculated so it points to the percpu variable of the new cpu.
*
* Inline assemblies making use of this typically have a code sequence like:
*
@@ -115,22 +119,53 @@
#define UNDEF_GR_NUM \
".purgem _GR_NUM\n"
+#define PCPU_REG_PCP_SHIFT 0
+#define PCPU_REG_PCP GENMASK(3, 0)
+#define PCPU_REG_OFF_SHIFT 4
+#define PCPU_REG_OFF GENMASK(7, 4)
+
#define __PCPU_MVIY(lcreg, imm) \
ALTERNATIVE(" mviy " lcreg "(%%r0)," imm "\n", \
" mviy " lcreg "+" LC_ALT_ADDR "(%%r0)," imm "\n", \
ALT_FEATURE(MFEATURE_LOWCORE))
-#define __PCPU_AG(reg, lcoff) \
- ALTERNATIVE(" ag " reg ", " lcoff "(%%r0)\n", \
- " ag " reg ", " lcoff "+" LC_ALT_ADDR "(%%r0)\n", \
+#define __PCPU_LG(regoff, lcoff) \
+ ALTERNATIVE(" lg " regoff ", " lcoff "(%%r0)\n", \
+ " lg " regoff ", " lcoff "+" LC_ALT_ADDR "(%%r0)\n", \
ALT_FEATURE(MFEATURE_LOWCORE))
-#define __PCPU_BEGIN(lcreg, lcoff, reg) \
+#define __PCPU_LA(regptr, regoff, regpcp) \
+ " la " regptr ",0(" regoff "," regpcp ")\n"
+
+#define __PCPU_CALC_REGVAL(regpcp, regoff) \
+ "(" regpcp " << " __stringify(PCPU_REG_PCP_SHIFT) ") |" \
+ "(" regoff " << " __stringify(PCPU_REG_OFF_SHIFT) ")"
+
+#define __PCPU_CHECK_REGS(regpcp, regoff, regptr) \
+ ".if " regoff " & 1\n" \
+ " .error \"Percpu offset register must be even\"\n" \
+ ".endif\n" \
+ ".if " regptr " != " regoff " + 1\n" \
+ " .error \"Percpu pointer and offset register must be a pair\"\n" \
+ ".endif\n" \
+ ".if " regptr " == 0\n" \
+ " .error \"Percpu pointer register must be non-zero\"\n" \
+ ".endif\n" \
+ ".if (" regpcp " == " regoff ") || (" regpcp " == " regptr ")\n" \
+ " .error \"Percpu registers must be distinct\"\n" \
+ ".endif\n"
+
+#define __PCPU_BEGIN(lcreg, lcoff, regpcp, regoff, regptr) \
DEFINE_GR_NUM \
- "_GR_NUM .Lreg, " reg "\n" \
+ "_GR_NUM .Lregpcp, " regpcp "\n" \
+ "_GR_NUM .Lregoff, " regoff "\n" \
+ "_GR_NUM .Lregptr, " regptr "\n" \
UNDEF_GR_NUM \
- __PCPU_MVIY(lcreg, ".Lreg") \
- __PCPU_AG(reg, lcoff)
+ __PCPU_CHECK_REGS(".Lregpcp", ".Lregoff", ".Lregptr") \
+ ".set .Lregval, " __PCPU_CALC_REGVAL(".Lregpcp", ".Lregoff") "\n" \
+ __PCPU_MVIY(lcreg, ".Lregval") \
+ __PCPU_LG(regoff, lcoff) \
+ __PCPU_LA(regptr, regoff, regpcp)
#define __PCPU_END(lcreg) \
__PCPU_MVIY(lcreg, "0")
@@ -151,6 +186,7 @@
#define arch_this_cpu_add(pcp, val, op1, op2, szcast) \
do { \
typedef typeof(pcp) pcp_op_T__; \
+ union register_pair rp__; \
pcp_op_T__ val__ = (val); \
pcp_op_T__ old__, *ptr__; \
\
@@ -158,25 +194,27 @@ do { \
if (__builtin_constant_p(val__) && \
((szcast)val__ > -129) && ((szcast)val__ < 128)) { \
asm volatile( \
- __PCPU_BEGIN("%[lcreg]","%[lcoff]","%[ptr__]") \
- op2 " 0(%[ptr__]),%[val__]\n" \
+ __PCPU_BEGIN("%[lcreg]","%[lcoff]","%[ptr__]", \
+ "%[pair__]","%N[pair__]") \
+ op2 " 0(%N[pair__]),%[val__]\n" \
__PCPU_END("%[lcreg]") \
- : [ptr__] "+&a" (ptr__), "+m" (*ptr__), \
+ : [pair__] "=&d" (rp__.pair), "+m" (*ptr__), \
"=m" (((struct lowcore *)0)->percpu_register) \
- : [val__] "i" ((szcast)val__), \
+ : [val__] "i" ((szcast)val__), [ptr__] "a" (ptr__), \
[lcreg] "i" (LC_PERCPU_REGISTER), \
[lcoff] "i" (LC_PERCPU_OFFSET), \
"m" (((struct lowcore *)0)->percpu_offset) \
: "cc"); \
} else { \
asm volatile( \
- __PCPU_BEGIN("%[lcreg]","%[lcoff]","%[ptr__]") \
- op1 " %[old__],%[val__],0(%[ptr__])\n" \
+ __PCPU_BEGIN("%[lcreg]","%[lcoff]","%[ptr__]", \
+ "%[pair__]","%N[pair__]") \
+ op1 " %[old__],%[val__],0(%N[pair__])\n" \
__PCPU_END("%[lcreg]") \
- : [old__] "=&d" (old__), \
- [ptr__] "+&a" (ptr__), "+m" (*ptr__), \
+ : [old__] "=&d" (old__), [pair__] "=&d" (rp__.pair), \
+ "+m" (*ptr__), \
"=m" (((struct lowcore *)0)->percpu_register) \
- : [val__] "d" (val__), \
+ : [val__] "d" (val__), [ptr__] "a" (ptr__), \
[lcreg] "i" (LC_PERCPU_REGISTER), \
[lcoff] "i" (LC_PERCPU_OFFSET), \
"m" (((struct lowcore *)0)->percpu_offset) \
@@ -190,18 +228,20 @@ do { \
#define arch_this_cpu_add_return(pcp, val, op) \
({ \
typedef typeof(pcp) pcp_op_T__; \
+ union register_pair rp__; \
pcp_op_T__ val__ = (val); \
pcp_op_T__ old__, *ptr__; \
\
ptr__ = PERCPU_PTR(&(pcp)); \
asm_inline volatile( \
- __PCPU_BEGIN("%[lcreg]","%[lcoff]","%[ptr__]") \
- op " %[old__],%[val__],0(%[ptr__])\n" \
+ __PCPU_BEGIN("%[lcreg]","%[lcoff]","%[ptr__]", \
+ "%[pair__]","%N[pair__]") \
+ op " %[old__],%[val__],0(%N[pair__])\n" \
__PCPU_END("%[lcreg]") \
- : [old__] "=&d" (old__), \
- [ptr__] "+&a" (ptr__), "+m" (*ptr__), \
+ : [old__] "=&d" (old__), [pair__] "=&d" (rp__.pair), \
+ "+m" (*ptr__), \
"=m" (((struct lowcore *)0)->percpu_register) \
- : [val__] "d" (val__), \
+ : [val__] "d" (val__), [ptr__] "a" (ptr__), \
[lcreg] "i" (LC_PERCPU_REGISTER), \
[lcoff] "i" (LC_PERCPU_OFFSET), \
"m" (((struct lowcore *)0)->percpu_offset) \
@@ -215,18 +255,20 @@ do { \
#define arch_this_cpu_to_op(pcp, val, op) \
do { \
typedef typeof(pcp) pcp_op_T__; \
+ union register_pair rp__; \
pcp_op_T__ val__ = (val); \
pcp_op_T__ old__, *ptr__; \
\
ptr__ = PERCPU_PTR(&(pcp)); \
asm_inline volatile( \
- __PCPU_BEGIN("%[lcreg]","%[lcoff]","%[ptr__]") \
- op " %[old__],%[val__],0(%[ptr__])\n" \
+ __PCPU_BEGIN("%[lcreg]","%[lcoff]","%[ptr__]", \
+ "%[pair__]","%N[pair__]") \
+ op " %[old__],%[val__],0(%N[pair__])\n" \
__PCPU_END("%[lcreg]") \
- : [old__] "=&d" (old__), \
- [ptr__] "+&a" (ptr__), "+m" (*ptr__), \
+ : [old__] "=&d" (old__), [pair__] "=&d" (rp__.pair), \
+ "+m" (*ptr__), \
"=m" (((struct lowcore *)0)->percpu_register) \
- : [val__] "d" (val__), \
+ : [val__] "d" (val__), [ptr__] "a" (ptr__), \
[lcreg] "i" (LC_PERCPU_REGISTER), \
[lcoff] "i" (LC_PERCPU_OFFSET), \
"m" (((struct lowcore *)0)->percpu_offset) \
@@ -243,17 +285,20 @@ do { \
#define arch_this_cpu_read(pcp, op) \
({ \
typedef typeof(pcp) pcp_op_T__; \
+ union register_pair rp__; \
unsigned long res__; \
pcp_op_T__ *ptr__; \
\
ptr__ = PERCPU_PTR(&(pcp)); \
asm_inline volatile( \
- __PCPU_BEGIN("%[lcreg]","%[lcoff]","%[ptr__]") \
- op " %[res__],0(%[ptr__])\n" \
+ __PCPU_BEGIN("%[lcreg]","%[lcoff]","%[ptr__]", \
+ "%[pair__]","%N[pair__]") \
+ op " %[res__],0(%N[pair__])\n" \
__PCPU_END("%[lcreg]") \
- : [res__] "=&d" (res__), [ptr__] "+&a" (ptr__), \
+ : [res__] "=&d" (res__), [pair__] "=&d" (rp__.pair), \
"=m" (((struct lowcore *)0)->percpu_register) \
- : [lcreg] "i" (LC_PERCPU_REGISTER), \
+ : [ptr__] "a" (ptr__), \
+ [lcreg] "i" (LC_PERCPU_REGISTER), \
[lcoff] "i" (LC_PERCPU_OFFSET), \
"m" (*ptr__), \
"m" (((struct lowcore *)0)->percpu_offset) \
@@ -269,16 +314,18 @@ do { \
#define arch_this_cpu_write(pcp, val, op) \
do { \
typedef typeof(pcp) pcp_op_T__; \
+ union register_pair rp__; \
pcp_op_T__ *ptr__, val__ = (val); \
\
ptr__ = PERCPU_PTR(&(pcp)); \
asm_inline volatile( \
- __PCPU_BEGIN("%[lcreg]","%[lcoff]","%[ptr__]") \
- op " %[val__],0(%[ptr__])\n" \
+ __PCPU_BEGIN("%[lcreg]","%[lcoff]","%[ptr__]", \
+ "%[pair__]","%N[pair__]") \
+ op " %[val__],0(%N[pair__])\n" \
__PCPU_END("%[lcreg]") \
- : [ptr__] "+&a" (ptr__), "=m" (*ptr__), \
+ : [pair__] "=&d" (rp__.pair), "=m" (*ptr__), \
"=m" (((struct lowcore *)0)->percpu_register) \
- : [val__] "d" (val__), \
+ : [val__] "d" (val__), [ptr__] "a" (ptr__), \
[lcreg] "i" (LC_PERCPU_REGISTER), \
[lcoff] "i" (LC_PERCPU_OFFSET), \
"m" (((struct lowcore *)0)->percpu_offset) \
@@ -306,17 +353,19 @@ do { \
({ \
typedef typeof(pcp) pcp_op_T__; \
pcp_op_T__ old__ = (oval), new__ = (nval); \
+ union register_pair rp__; \
pcp_op_T__ *ptr__; \
\
ptr__ = PERCPU_PTR(&(pcp)); \
asm_inline volatile( \
- __PCPU_BEGIN("%[lcreg]","%[lcoff]","%[ptr__]") \
- op " %[old__],%[new__],0(%[ptr__])\n" \
+ __PCPU_BEGIN("%[lcreg]","%[lcoff]","%[ptr__]", \
+ "%[pair__]","%N[pair__]") \
+ op " %[old__],%[new__],0(%N[pair__])\n" \
__PCPU_END("%[lcreg]") \
- : [old__] "+&d" (old__), \
- [ptr__] "+&a" (ptr__), "+m" (*ptr__), \
+ : [old__] "+&d" (old__), [pair__] "=&d" (rp__.pair), \
+ "+m" (*ptr__), \
"=m" (((struct lowcore *)0)->percpu_register) \
- : [new__] "d" (new__), \
+ : [new__] "d" (new__), [ptr__] "a" (ptr__), \
[lcreg] "i" (LC_PERCPU_REGISTER), \
[lcoff] "i" (LC_PERCPU_OFFSET), \
"m" (((struct lowcore *)0)->percpu_offset) \
@@ -335,17 +384,19 @@ do { \
({ \
typedef typeof(pcp) pcp_op_T__; \
u128 old__ = (oval), new__ = (nval); \
+ union register_pair rp__; \
pcp_op_T__ *ptr__; \
\
ptr__ = PERCPU_PTR(&(pcp)); \
asm_inline volatile( \
- __PCPU_BEGIN("%[lcreg]","%[lcoff]","%[ptr__]") \
- " cdsg %[old__],%[new__],0(%[ptr__])\n" \
+ __PCPU_BEGIN("%[lcreg]","%[lcoff]","%[ptr__]", \
+ "%[pair__]","%N[pair__]") \
+ " cdsg %[old__],%[new__],0(%N[pair__])\n" \
__PCPU_END("%[lcreg]") \
- : [old__] "+&d" (old__), [ptr__] "+&a" (ptr__), \
+ : [old__] "+&d" (old__), [pair__] "=&d" (rp__.pair), \
"+m" (*ptr__), \
"=m" (((struct lowcore *)0)->percpu_register) \
- : [new__] "d" (new__), \
+ : [new__] "d" (new__), [ptr__] "a" (ptr__), \
[lcreg] "i" (LC_PERCPU_REGISTER), \
[lcoff] "i" (LC_PERCPU_OFFSET), \
"m" (((struct lowcore *)0)->percpu_offset) \
@@ -368,19 +419,21 @@ do { \
({ \
typedef typeof(pcp) pcp_op_T__; \
pcp_op_T__ old__, new__ = (nval); \
+ union register_pair rp__; \
pcp_op_T__ *ptr__; \
\
ptr__ = PERCPU_PTR(&(pcp)); \
asm_inline volatile( \
- __PCPU_BEGIN("%[lcreg]","%[lcoff]","%[ptr__]") \
- " " ldop " %[old__],0(%[ptr__])\n" \
- "0: " csop " %[old__],%[new__],0(%[ptr__])\n"\
+ __PCPU_BEGIN("%[lcreg]","%[lcoff]","%[ptr__]", \
+ "%[pair__]","%N[pair__]") \
+ " " ldop " %[old__],0(%N[pair__])\n" \
+ "0: " csop " %[old__],%[new__],0(%N[pair__])\n"\
" jnz 0b\n" \
__PCPU_END("%[lcreg]") \
- : [old__] "=&d" (old__), \
- [ptr__] "+&a" (ptr__), "+m" (*ptr__), \
+ : [old__] "=&d" (old__), [pair__] "=&d" (rp__.pair), \
+ "+m" (*ptr__), \
"=m" (((struct lowcore *)0)->percpu_register) \
- : [new__] "d" (new__), \
+ : [new__] "d" (new__), [ptr__] "a" (ptr__), \
[lcreg] "i" (LC_PERCPU_REGISTER), \
[lcoff] "i" (LC_PERCPU_OFFSET), \
"m" (((struct lowcore *)0)->percpu_offset) \
diff --git a/arch/s390/kernel/irq.c b/arch/s390/kernel/irq.c
index c923496aa7b4..8f21bea349a6 100644
--- a/arch/s390/kernel/irq.c
+++ b/arch/s390/kernel/irq.c
@@ -144,9 +144,9 @@ static int irq_pending(struct pt_regs *regs)
void noinstr do_io_irq(struct pt_regs *regs)
{
- bool from_idle, percpu_needs_fixup;
struct pt_regs *old_regs;
irqentry_state_t state;
+ bool from_idle;
percpu_entry(regs);
state = irqentry_enter(regs);
@@ -174,21 +174,20 @@ void noinstr do_io_irq(struct pt_regs *regs)
do_irq_async(regs, IO_INTERRUPT);
} while (machine_is_lpar() && irq_pending(regs));
- percpu_needs_fixup = percpu_code_check(regs);
irq_exit_rcu();
set_irq_regs(old_regs);
irqentry_exit(regs, state);
if (from_idle)
regs->psw.mask &= ~(PSW_MASK_EXT | PSW_MASK_IO | PSW_MASK_WAIT);
- percpu_exit(regs, percpu_needs_fixup);
+ percpu_exit(regs);
}
void noinstr do_ext_irq(struct pt_regs *regs)
{
- bool from_idle, percpu_needs_fixup;
struct pt_regs *old_regs;
irqentry_state_t state;
+ bool from_idle;
percpu_entry(regs);
state = irqentry_enter(regs);
@@ -214,14 +213,13 @@ void noinstr do_ext_irq(struct pt_regs *regs)
do_irq_async(regs, EXT_INTERRUPT);
- percpu_needs_fixup = percpu_code_check(regs);
irq_exit_rcu();
set_irq_regs(old_regs);
irqentry_exit(regs, state);
if (from_idle)
regs->psw.mask &= ~(PSW_MASK_EXT | PSW_MASK_IO | PSW_MASK_WAIT);
- percpu_exit(regs, percpu_needs_fixup);
+ percpu_exit(regs);
}
static void show_msi_interrupt(struct seq_file *p, int irq)
diff --git a/arch/s390/kernel/nmi.c b/arch/s390/kernel/nmi.c
index 17297a8b63d9..56a15edc65d7 100644
--- a/arch/s390/kernel/nmi.c
+++ b/arch/s390/kernel/nmi.c
@@ -361,7 +361,6 @@ NOKPROBE_SYMBOL(s390_backup_mcck_info);
*/
void notrace s390_do_machine_check(struct pt_regs *regs)
{
- bool percpu_needs_fixup;
static int ipd_count;
static DEFINE_SPINLOCK(ipd_lock);
static unsigned long long last_ipd;
@@ -495,9 +494,8 @@ void notrace s390_do_machine_check(struct pt_regs *regs)
if (mcck_pending)
schedule_mcck_handler();
- percpu_needs_fixup = percpu_code_check(regs);
irqentry_nmi_exit(regs, irq_state);
- percpu_exit(regs, percpu_needs_fixup);
+ percpu_exit(regs);
}
NOKPROBE_SYMBOL(s390_do_machine_check);
diff --git a/arch/s390/kernel/traps.c b/arch/s390/kernel/traps.c
index b6ba4465f59d..dd051046170b 100644
--- a/arch/s390/kernel/traps.c
+++ b/arch/s390/kernel/traps.c
@@ -338,7 +338,6 @@ static void (*pgm_check_table[128])(struct pt_regs *regs);
void noinstr __do_pgm_check(struct pt_regs *regs, unsigned long flags)
{
struct lowcore *lc = get_lowcore();
- bool percpu_needs_fixup;
irqentry_state_t state;
struct pgm_stat *stat;
unsigned int trapnr;
@@ -400,9 +399,8 @@ void noinstr __do_pgm_check(struct pt_regs *regs, unsigned long flags)
pgm_check_table[trapnr](regs);
out:
local_irq_disable();
- percpu_needs_fixup = percpu_code_check(regs);
irqentry_exit(regs, state);
- percpu_exit(regs, percpu_needs_fixup);
+ percpu_exit(regs);
}
static int pgm_check_stat_show(struct seq_file *p, void *v)
--
2.53.0
^ permalink raw reply [flat|nested] 16+ messages in thread* Re: [PATCH v3 11/11] s390/percpu: Rework to simplify percpu_entry() and percpu_exit()
2026-09-21 8:40 ` [PATCH v3 11/11] s390/percpu: Rework to simplify percpu_entry() and percpu_exit() Heiko Carstens
@ 2026-09-21 9:55 ` David Laight
2026-09-21 10:07 ` Mark Rutland
0 siblings, 1 reply; 16+ messages in thread
From: David Laight @ 2026-09-21 9:55 UTC (permalink / raw)
To: Heiko Carstens
Cc: Alexander Gordeev, Sven Schnelle, Vasily Gorbik,
Christian Borntraeger, Mete Durlu, Peter Zijlstra, Mark Rutland,
Juergen Christ, Ilya Leoshkevich, linux-kernel, linux-s390
On Mon, 21 Sep 2026 10:40:05 +0200
Heiko Carstens <hca@linux.ibm.com> wrote:
> The percpu code section functionality uses a rather complex method to
> figure out if the register, which contains the address of the current
> cpu's percpu variable, needs to be adjusted.
>
> If an interrupt happens within a percpu code section (indicated by a
> lowcore field), the instruction at the interrupted location is
> checked. If it is not a specific AG instruction, the register needs to
> be updated. This mechanism needs to take kprobes into account, and
> enforces a specific instruction ordering.
>
> Mark Rutland provided a different solution for arm64 [1] which comes
> without such limitations, but requires to use one more instruction, and
> two more registers. Given that this simplifies percpu_entry() and
> percpu_exit() it seems to be worth to go that route.
>
> Change s390 to implement a similar approach. This requires to encode
> three register numbers into the "percpu_register" field, which is used
> to indicate if a percpu code section is executed.
>
> The used mviy instruction can write only one byte, which allows to
> encode only two register numbers. Use a register pair for the inline
> assemblies, and only encode the even register number of the register
> pair to work around this.
>
> The generated code changes like this for e.g. a simple this_cpu_inc():
>
> Old:
>
> c0 20 00 00 00 00 larl %r2,c6 <foo+0x6> <-- load address of percpu var
> b9 04 00 32 lgr %r3,%r2 <-- pointless copy of address
> eb 03 03 c0 00 52 mviy 960,3 <-- start of percpu code section
> - gpr 3 contains percpu var address
> e3 30 03 b8 00 08 ag %r3,952 <-- add percpu offset
> eb 01 30 00 00 7a agsi 0(%r3),1 <-- atomic inc
> eb 00 03 c0 00 52 mviy 960,0 <-- end of percpu code section
>
> New:
>
> c0 10 00 00 00 00 larl %r1,c6 <foo+0x6> <-- load address of percpu var
> eb 21 03 c0 00 52 mviy 960,33 <-- start of percpu code section
> 33 == 0x21:
> - gpr 1 contains percpu var address
> - gpr 2 used for percpu offset
> - gpr 2+1 == 3 used for current cpu's percpu var address
> e3 20 03 b8 00 04 lg %r2,952 <-- load percpu offset
> 41 32 10 00 la %r3,0(%r2,%r1) <-- generate current cpu's percpu var address
> eb 01 30 00 00 7a agsi 0(%r3),1 <-- atomic inc
> eb 00 03 c0 00 52 mviy 960,0 <-- end of percpu code section
>
> In the above "new" code example 33 (0x21) is used as indicator value to
> mark that a percpu code section is executed. This value implies that
> registers 2 and 3 will (only) hold the percpu offset and the current
> cpu's percpu var address. Those registers will be updated by
> percpu_exit() if the process was migrated to a different cpu to contain
> the percpu offset and percpu var address of the new cpu.
>
> Note that because of the pointless lgr instruction in the "old" code
> example the two code sequences have identical size, and it looks like
> only one more register is used. However this is because of suboptimal
> gcc code generation.
In both cases the register dependency chain length is 3 (ignoring the
pointless lgr) so the execution time is likely to be the same.
...
> + regpcp = FIELD_GET(PCPU_REG_PCP, regval);
> + regoff = FIELD_GET(PCPU_REG_OFF, regval);
...
> +#define PCPU_REG_PCP_SHIFT 0
> +#define PCPU_REG_PCP GENMASK(3, 0)
> +#define PCPU_REG_OFF_SHIFT 4
> +#define PCPU_REG_OFF GENMASK(7, 4)
...
> +#define __PCPU_CALC_REGVAL(regpcp, regoff) \
> + "(" regpcp " << " __stringify(PCPU_REG_PCP_SHIFT) ") |" \
> + "(" regoff " << " __stringify(PCPU_REG_OFF_SHIFT) ")"
I'm not a big fan of GENMASK() + FIELD_GET() and I'm not at all sure it
really helps here.
Maybe:
regpcp = (regval >> PCPU_REG_PCP_SHIFT) & 15;
regoff = (regval >> PCPU_REG_OFF_SHIFT) & 15;
Which at least uses the same constants for encode and decode.
But I might just comment that the pcp register is in the high nibble
(twice) and remove the 'crud'.
#define __PCPU_CALC_REGVAL(regpcp, regoff) "(" regpcp " << 4 ) | " regoff
regpcp = regval >> 4;
regoff = regval & 15;
David
^ permalink raw reply [flat|nested] 16+ messages in thread* Re: [PATCH v3 11/11] s390/percpu: Rework to simplify percpu_entry() and percpu_exit()
2026-09-21 9:55 ` David Laight
@ 2026-09-21 10:07 ` Mark Rutland
2026-09-21 10:17 ` Heiko Carstens
0 siblings, 1 reply; 16+ messages in thread
From: Mark Rutland @ 2026-09-21 10:07 UTC (permalink / raw)
To: David Laight
Cc: Heiko Carstens, Alexander Gordeev, Sven Schnelle, Vasily Gorbik,
Christian Borntraeger, Mete Durlu, Peter Zijlstra,
Juergen Christ, Ilya Leoshkevich, linux-kernel, linux-s390
On Mon, Sep 21, 2026 at 10:55:18AM +0100, David Laight wrote:
> On Mon, 21 Sep 2026 10:40:05 +0200
> Heiko Carstens <hca@linux.ibm.com> wrote:
> > + regpcp = FIELD_GET(PCPU_REG_PCP, regval);
> > + regoff = FIELD_GET(PCPU_REG_OFF, regval);
> ...
> > +#define PCPU_REG_PCP_SHIFT 0
> > +#define PCPU_REG_PCP GENMASK(3, 0)
> > +#define PCPU_REG_OFF_SHIFT 4
> > +#define PCPU_REG_OFF GENMASK(7, 4)
> ...
> > +#define __PCPU_CALC_REGVAL(regpcp, regoff) \
> > + "(" regpcp " << " __stringify(PCPU_REG_PCP_SHIFT) ") |" \
> > + "(" regoff " << " __stringify(PCPU_REG_OFF_SHIFT) ")"
>
> I'm not a big fan of GENMASK() + FIELD_GET() and I'm not at all sure it
> really helps here.
It's the same pattern already used by the extable code.
While _you_ don't like it, that's not a universal opinion (and FWIW, I
prefer the FIELD_GET() approach).
I'll leave it to Heiko and co to choose whatever style they prefer, as
it's their architecture port...
Mark.
^ permalink raw reply [flat|nested] 16+ messages in thread* Re: [PATCH v3 11/11] s390/percpu: Rework to simplify percpu_entry() and percpu_exit()
2026-09-21 10:07 ` Mark Rutland
@ 2026-09-21 10:17 ` Heiko Carstens
0 siblings, 0 replies; 16+ messages in thread
From: Heiko Carstens @ 2026-09-21 10:17 UTC (permalink / raw)
To: Mark Rutland
Cc: David Laight, Alexander Gordeev, Sven Schnelle, Vasily Gorbik,
Christian Borntraeger, Mete Durlu, Peter Zijlstra,
Juergen Christ, Ilya Leoshkevich, linux-kernel, linux-s390
On Mon, Sep 21, 2026 at 11:07:04AM +0100, Mark Rutland wrote:
> On Mon, Sep 21, 2026 at 10:55:18AM +0100, David Laight wrote:
> > On Mon, 21 Sep 2026 10:40:05 +0200
> > Heiko Carstens <hca@linux.ibm.com> wrote:
>
> > > + regpcp = FIELD_GET(PCPU_REG_PCP, regval);
> > > + regoff = FIELD_GET(PCPU_REG_OFF, regval);
> > ...
> > > +#define PCPU_REG_PCP_SHIFT 0
> > > +#define PCPU_REG_PCP GENMASK(3, 0)
> > > +#define PCPU_REG_OFF_SHIFT 4
> > > +#define PCPU_REG_OFF GENMASK(7, 4)
> > ...
> > > +#define __PCPU_CALC_REGVAL(regpcp, regoff) \
> > > + "(" regpcp " << " __stringify(PCPU_REG_PCP_SHIFT) ") |" \
> > > + "(" regoff " << " __stringify(PCPU_REG_OFF_SHIFT) ")"
> >
> > I'm not a big fan of GENMASK() + FIELD_GET() and I'm not at all sure it
> > really helps here.
>
> It's the same pattern already used by the extable code.
>
> While _you_ don't like it, that's not a universal opinion (and FWIW, I
> prefer the FIELD_GET() approach).
>
> I'll leave it to Heiko and co to choose whatever style they prefer, as
> it's their architecture port...
As Mark said: it is already used, and in addition I would like to have s390
architecture code quite similar to other architectures whenever possible.
This simplifies cross architecture changes, even though it seems unlikely that
somebody would ever touch this particular code across architectures.
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [PATCH v3 00/11] s390: More this_cpu_*() changes
2026-09-21 8:39 [PATCH v3 00/11] s390: More this_cpu_*() changes Heiko Carstens
` (10 preceding siblings ...)
2026-09-21 8:40 ` [PATCH v3 11/11] s390/percpu: Rework to simplify percpu_entry() and percpu_exit() Heiko Carstens
@ 2026-09-21 9:15 ` Heiko Carstens
11 siblings, 0 replies; 16+ messages in thread
From: Heiko Carstens @ 2026-09-21 9:15 UTC (permalink / raw)
To: Heiko Carstens
Cc: Alexander Gordeev, Sven Schnelle, Vasily Gorbik,
Christian Borntraeger, Mete Durlu, Peter Zijlstra, Mark Rutland,
Juergen Christ, Ilya Leoshkevich, linux-kernel, linux-s390
On Mon, Sep 21, 2026 at 10:39:54AM +0200, Heiko Carstens wrote:
> v3:
> - Add various sanity checks to enforce correct register usage
> [Mark Rutland]
...
> s390/percpu: Rework to simplify percpu_entry() and percpu_exit()
>
> arch/s390/include/asm/entry-percpu.h | 71 ++----
> arch/s390/include/asm/lowcore.h | 5 +-
> arch/s390/include/asm/percpu.h | 354 +++++++++++++++++----------
> arch/s390/kernel/irq.c | 10 +-
> arch/s390/kernel/nmi.c | 4 +-
> arch/s390/kernel/traps.c | 4 +-
> 6 files changed, 248 insertions(+), 200 deletions(-)
Please ignore this version.
The checks are still not sufficient, and there is indeed a bug with
register allocation in the last patch which Sashiko reported. The
incorrect register allocation happened exactly once in my test build,
which explains why everything worked for me (tm).
I will need to send yet another version - sorry for the noise.
^ permalink raw reply [flat|nested] 16+ messages in thread