* [PATCH 01/12] cyrpto/b128ops: Remove struct u128
2023-05-31 13:08 [PATCH 00/12] Introduce cmpxchg128() -- aka. the demise of cmpxchg_double() Peter Zijlstra
@ 2023-05-31 13:08 ` Peter Zijlstra
2023-06-05 7:42 ` [tip: locking/core] " tip-bot2 for Peter Zijlstra
2023-05-31 13:08 ` [PATCH 02/12] types: Introduce [us]128 Peter Zijlstra
` (11 subsequent siblings)
12 siblings, 1 reply; 47+ messages in thread
From: Peter Zijlstra @ 2023-05-31 13:08 UTC (permalink / raw)
To: torvalds
Cc: corbet, will, peterz, boqun.feng, mark.rutland, catalin.marinas,
dennis, tj, cl, hca, gor, agordeev, borntraeger, svens, tglx,
mingo, bp, dave.hansen, x86, hpa, joro, suravee.suthikulpanit,
robin.murphy, dwmw2, baolu.lu, Arnd Bergmann, Herbert Xu, davem,
penberg, rientjes, iamjoonsoo.kim, Andrew Morton, vbabka,
roman.gushchin, 42.hyeyoo, linux-doc, linux-kernel, linux-mm,
linux-s390, iommu, linux-arch, linux-crypto, sfr, mpe,
James.Bottomley, deller, linux-parisc
Per git-grep u128_xor() and its related struct u128 are unused except
to implement {be,le}128_xor(). Remove them to free up the namespace.
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Reviewed-by: Arnd Bergmann <arnd@arndb.de>
Acked-by: Herbert Xu <herbert@gondor.apana.org.au>
---
include/crypto/b128ops.h | 14 +++-----------
1 file changed, 3 insertions(+), 11 deletions(-)
--- a/include/crypto/b128ops.h
+++ b/include/crypto/b128ops.h
@@ -50,10 +50,6 @@
#include <linux/types.h>
typedef struct {
- u64 a, b;
-} u128;
-
-typedef struct {
__be64 a, b;
} be128;
@@ -61,20 +57,16 @@ typedef struct {
__le64 b, a;
} le128;
-static inline void u128_xor(u128 *r, const u128 *p, const u128 *q)
+static inline void be128_xor(be128 *r, const be128 *p, const be128 *q)
{
r->a = p->a ^ q->a;
r->b = p->b ^ q->b;
}
-static inline void be128_xor(be128 *r, const be128 *p, const be128 *q)
-{
- u128_xor((u128 *)r, (u128 *)p, (u128 *)q);
-}
-
static inline void le128_xor(le128 *r, const le128 *p, const le128 *q)
{
- u128_xor((u128 *)r, (u128 *)p, (u128 *)q);
+ r->a = p->a ^ q->a;
+ r->b = p->b ^ q->b;
}
#endif /* _CRYPTO_B128OPS_H */
^ permalink raw reply [flat|nested] 47+ messages in thread* [tip: locking/core] cyrpto/b128ops: Remove struct u128
2023-05-31 13:08 ` [PATCH 01/12] cyrpto/b128ops: Remove struct u128 Peter Zijlstra
@ 2023-06-05 7:42 ` tip-bot2 for Peter Zijlstra
0 siblings, 0 replies; 47+ messages in thread
From: tip-bot2 for Peter Zijlstra @ 2023-06-05 7:42 UTC (permalink / raw)
To: linux-tip-commits
Cc: Peter Zijlstra (Intel),
Arnd Bergmann, Mark Rutland, Herbert Xu, x86, linux-kernel
The following commit has been merged into the locking/core branch of tip:
Commit-ID: f413e724818c6482146218b3bcaf3d75b1317fc4
Gitweb: https://git.kernel.org/tip/f413e724818c6482146218b3bcaf3d75b1317fc4
Author: Peter Zijlstra <peterz@infradead.org>
AuthorDate: Wed, 31 May 2023 15:08:34 +02:00
Committer: Peter Zijlstra <peterz@infradead.org>
CommitterDate: Mon, 05 Jun 2023 09:36:35 +02:00
cyrpto/b128ops: Remove struct u128
Per git-grep u128_xor() and its related struct u128 are unused except
to implement {be,le}128_xor(). Remove them to free up the namespace.
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Reviewed-by: Arnd Bergmann <arnd@arndb.de>
Reviewed-by: Mark Rutland <mark.rutland@arm.com>
Acked-by: Herbert Xu <herbert@gondor.apana.org.au>
Tested-by: Mark Rutland <mark.rutland@arm.com>
Link: https://lore.kernel.org/r/20230531132323.314826687@infradead.org
---
include/crypto/b128ops.h | 14 +++-----------
1 file changed, 3 insertions(+), 11 deletions(-)
diff --git a/include/crypto/b128ops.h b/include/crypto/b128ops.h
index 0b8e6bc..f3b37cb 100644
--- a/include/crypto/b128ops.h
+++ b/include/crypto/b128ops.h
@@ -50,10 +50,6 @@
#include <linux/types.h>
typedef struct {
- u64 a, b;
-} u128;
-
-typedef struct {
__be64 a, b;
} be128;
@@ -61,20 +57,16 @@ typedef struct {
__le64 b, a;
} le128;
-static inline void u128_xor(u128 *r, const u128 *p, const u128 *q)
+static inline void be128_xor(be128 *r, const be128 *p, const be128 *q)
{
r->a = p->a ^ q->a;
r->b = p->b ^ q->b;
}
-static inline void be128_xor(be128 *r, const be128 *p, const be128 *q)
-{
- u128_xor((u128 *)r, (u128 *)p, (u128 *)q);
-}
-
static inline void le128_xor(le128 *r, const le128 *p, const le128 *q)
{
- u128_xor((u128 *)r, (u128 *)p, (u128 *)q);
+ r->a = p->a ^ q->a;
+ r->b = p->b ^ q->b;
}
#endif /* _CRYPTO_B128OPS_H */
^ permalink raw reply [flat|nested] 47+ messages in thread
* [PATCH 02/12] types: Introduce [us]128
2023-05-31 13:08 [PATCH 00/12] Introduce cmpxchg128() -- aka. the demise of cmpxchg_double() Peter Zijlstra
2023-05-31 13:08 ` [PATCH 01/12] cyrpto/b128ops: Remove struct u128 Peter Zijlstra
@ 2023-05-31 13:08 ` Peter Zijlstra
2023-06-05 7:42 ` [tip: locking/core] " tip-bot2 for Peter Zijlstra
2023-05-31 13:08 ` [PATCH 03/12] arch: Introduce arch_{,try_}_cmpxchg128{,_local}() Peter Zijlstra
` (10 subsequent siblings)
12 siblings, 1 reply; 47+ messages in thread
From: Peter Zijlstra @ 2023-05-31 13:08 UTC (permalink / raw)
To: torvalds
Cc: corbet, will, peterz, boqun.feng, mark.rutland, catalin.marinas,
dennis, tj, cl, hca, gor, agordeev, borntraeger, svens, tglx,
mingo, bp, dave.hansen, x86, hpa, joro, suravee.suthikulpanit,
robin.murphy, dwmw2, baolu.lu, Arnd Bergmann, Herbert Xu, davem,
penberg, rientjes, iamjoonsoo.kim, Andrew Morton, vbabka,
roman.gushchin, 42.hyeyoo, linux-doc, linux-kernel, linux-mm,
linux-s390, iommu, linux-arch, linux-crypto, sfr, mpe,
James.Bottomley, deller, linux-parisc
Introduce [us]128 (when available). Unlike [us]64, ensure they are
always naturally aligned.
This also enables 128bit wide atomics (which require natural
alignment) such as cmpxchg128().
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Reviewed-by: Arnd Bergmann <arnd@arndb.de>
Acked-by: Herbert Xu <herbert@gondor.apana.org.au>
---
include/linux/types.h | 5 +++++
include/uapi/linux/types.h | 4 ++++
lib/crypto/curve25519-hacl64.c | 2 --
lib/crypto/poly1305-donna64.c | 2 --
4 files changed, 9 insertions(+), 4 deletions(-)
--- a/include/linux/types.h
+++ b/include/linux/types.h
@@ -10,6 +10,11 @@
#define DECLARE_BITMAP(name,bits) \
unsigned long name[BITS_TO_LONGS(bits)]
+#ifdef __SIZEOF_INT128__
+typedef __s128 s128;
+typedef __u128 u128;
+#endif
+
typedef u32 __kernel_dev_t;
typedef __kernel_fd_set fd_set;
--- a/include/uapi/linux/types.h
+++ b/include/uapi/linux/types.h
@@ -13,6 +13,10 @@
#include <linux/posix_types.h>
+#ifdef __SIZEOF_INT128__
+typedef __signed__ __int128 __s128 __attribute__((aligned(16)));
+typedef unsigned __int128 __u128 __attribute__((aligned(16)));
+#endif
/*
* Below are truly Linux-specific types that should never collide with
--- a/lib/crypto/curve25519-hacl64.c
+++ b/lib/crypto/curve25519-hacl64.c
@@ -14,8 +14,6 @@
#include <crypto/curve25519.h>
#include <linux/string.h>
-typedef __uint128_t u128;
-
static __always_inline u64 u64_eq_mask(u64 a, u64 b)
{
u64 x = a ^ b;
--- a/lib/crypto/poly1305-donna64.c
+++ b/lib/crypto/poly1305-donna64.c
@@ -10,8 +10,6 @@
#include <asm/unaligned.h>
#include <crypto/internal/poly1305.h>
-typedef __uint128_t u128;
-
void poly1305_core_setkey(struct poly1305_core_key *key,
const u8 raw_key[POLY1305_BLOCK_SIZE])
{
^ permalink raw reply [flat|nested] 47+ messages in thread* [tip: locking/core] types: Introduce [us]128
2023-05-31 13:08 ` [PATCH 02/12] types: Introduce [us]128 Peter Zijlstra
@ 2023-06-05 7:42 ` tip-bot2 for Peter Zijlstra
0 siblings, 0 replies; 47+ messages in thread
From: tip-bot2 for Peter Zijlstra @ 2023-06-05 7:42 UTC (permalink / raw)
To: linux-tip-commits
Cc: Peter Zijlstra (Intel),
Arnd Bergmann, Mark Rutland, Herbert Xu, x86, linux-kernel
The following commit has been merged into the locking/core branch of tip:
Commit-ID: 224d80c584d3016cb8d83d1c33914fdd3508aa8c
Gitweb: https://git.kernel.org/tip/224d80c584d3016cb8d83d1c33914fdd3508aa8c
Author: Peter Zijlstra <peterz@infradead.org>
AuthorDate: Wed, 31 May 2023 15:08:35 +02:00
Committer: Peter Zijlstra <peterz@infradead.org>
CommitterDate: Mon, 05 Jun 2023 09:36:35 +02:00
types: Introduce [us]128
Introduce [us]128 (when available). Unlike [us]64, ensure they are
always naturally aligned.
This also enables 128bit wide atomics (which require natural
alignment) such as cmpxchg128().
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Reviewed-by: Arnd Bergmann <arnd@arndb.de>
Reviewed-by: Mark Rutland <mark.rutland@arm.com>
Acked-by: Herbert Xu <herbert@gondor.apana.org.au>
Tested-by: Mark Rutland <mark.rutland@arm.com>
Link: https://lore.kernel.org/r/20230531132323.385005581@infradead.org
---
include/linux/types.h | 5 +++++
include/uapi/linux/types.h | 4 ++++
lib/crypto/curve25519-hacl64.c | 2 --
lib/crypto/poly1305-donna64.c | 2 --
4 files changed, 9 insertions(+), 4 deletions(-)
diff --git a/include/linux/types.h b/include/linux/types.h
index 688fb94..becb8cd 100644
--- a/include/linux/types.h
+++ b/include/linux/types.h
@@ -10,6 +10,11 @@
#define DECLARE_BITMAP(name,bits) \
unsigned long name[BITS_TO_LONGS(bits)]
+#ifdef __SIZEOF_INT128__
+typedef __s128 s128;
+typedef __u128 u128;
+#endif
+
typedef u32 __kernel_dev_t;
typedef __kernel_fd_set fd_set;
diff --git a/include/uapi/linux/types.h b/include/uapi/linux/types.h
index 308433b..6375a06 100644
--- a/include/uapi/linux/types.h
+++ b/include/uapi/linux/types.h
@@ -13,6 +13,10 @@
#include <linux/posix_types.h>
+#ifdef __SIZEOF_INT128__
+typedef __signed__ __int128 __s128 __attribute__((aligned(16)));
+typedef unsigned __int128 __u128 __attribute__((aligned(16)));
+#endif
/*
* Below are truly Linux-specific types that should never collide with
diff --git a/lib/crypto/curve25519-hacl64.c b/lib/crypto/curve25519-hacl64.c
index 771d82d..c40e5d9 100644
--- a/lib/crypto/curve25519-hacl64.c
+++ b/lib/crypto/curve25519-hacl64.c
@@ -14,8 +14,6 @@
#include <crypto/curve25519.h>
#include <linux/string.h>
-typedef __uint128_t u128;
-
static __always_inline u64 u64_eq_mask(u64 a, u64 b)
{
u64 x = a ^ b;
diff --git a/lib/crypto/poly1305-donna64.c b/lib/crypto/poly1305-donna64.c
index d34cf40..988702c 100644
--- a/lib/crypto/poly1305-donna64.c
+++ b/lib/crypto/poly1305-donna64.c
@@ -10,8 +10,6 @@
#include <asm/unaligned.h>
#include <crypto/internal/poly1305.h>
-typedef __uint128_t u128;
-
void poly1305_core_setkey(struct poly1305_core_key *key,
const u8 raw_key[POLY1305_BLOCK_SIZE])
{
^ permalink raw reply [flat|nested] 47+ messages in thread
* [PATCH 03/12] arch: Introduce arch_{,try_}_cmpxchg128{,_local}()
2023-05-31 13:08 [PATCH 00/12] Introduce cmpxchg128() -- aka. the demise of cmpxchg_double() Peter Zijlstra
2023-05-31 13:08 ` [PATCH 01/12] cyrpto/b128ops: Remove struct u128 Peter Zijlstra
2023-05-31 13:08 ` [PATCH 02/12] types: Introduce [us]128 Peter Zijlstra
@ 2023-05-31 13:08 ` Peter Zijlstra
2023-06-05 7:42 ` [tip: locking/core] " tip-bot2 for Peter Zijlstra
2023-05-31 13:08 ` [PATCH 04/12] instrumentation: Wire up cmpxchg128() Peter Zijlstra
` (9 subsequent siblings)
12 siblings, 1 reply; 47+ messages in thread
From: Peter Zijlstra @ 2023-05-31 13:08 UTC (permalink / raw)
To: torvalds
Cc: corbet, will, peterz, boqun.feng, mark.rutland, catalin.marinas,
dennis, tj, cl, hca, gor, agordeev, borntraeger, svens, tglx,
mingo, bp, dave.hansen, x86, hpa, joro, suravee.suthikulpanit,
robin.murphy, dwmw2, baolu.lu, Arnd Bergmann, Herbert Xu, davem,
penberg, rientjes, iamjoonsoo.kim, Andrew Morton, vbabka,
roman.gushchin, 42.hyeyoo, linux-doc, linux-kernel, linux-mm,
linux-s390, iommu, linux-arch, linux-crypto, sfr, mpe,
James.Bottomley, deller, linux-parisc
For all architectures that currently support cmpxchg_double()
implement the cmpxchg128() family of functions that is basically the
same but with a saner interface.
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Reviewed-by: Arnd Bergmann <arnd@arndb.de>
Acked-by: Heiko Carstens <hca@linux.ibm.com>
Acked-by: Mark Rutland <mark.rutland@arm.com>
---
arch/arm64/include/asm/atomic_ll_sc.h | 41 +++++++++++++++++++++
arch/arm64/include/asm/atomic_lse.h | 31 ++++++++++++++++
arch/arm64/include/asm/cmpxchg.h | 26 +++++++++++++
arch/s390/include/asm/cmpxchg.h | 14 +++++++
arch/x86/include/asm/cmpxchg_32.h | 3 +
arch/x86/include/asm/cmpxchg_64.h | 64 +++++++++++++++++++++++++++++++++-
6 files changed, 177 insertions(+), 2 deletions(-)
--- a/arch/arm64/include/asm/atomic_ll_sc.h
+++ b/arch/arm64/include/asm/atomic_ll_sc.h
@@ -326,6 +326,47 @@ __CMPXCHG_DBL( , , , )
__CMPXCHG_DBL(_mb, dmb ish, l, "memory")
#undef __CMPXCHG_DBL
+
+union __u128_halves {
+ u128 full;
+ struct {
+ u64 low, high;
+ };
+};
+
+#define __CMPXCHG128(name, mb, rel, cl...) \
+static __always_inline u128 \
+__ll_sc__cmpxchg128##name(volatile u128 *ptr, u128 old, u128 new) \
+{ \
+ union __u128_halves r, o = { .full = (old) }, \
+ n = { .full = (new) }; \
+ unsigned int tmp; \
+ \
+ asm volatile("// __cmpxchg128" #name "\n" \
+ " prfm pstl1strm, %[v]\n" \
+ "1: ldxp %[rl], %[rh], %[v]\n" \
+ " cmp %[rl], %[ol]\n" \
+ " ccmp %[rh], %[oh], 0, eq\n" \
+ " b.ne 2f\n" \
+ " st" #rel "xp %w[tmp], %[nl], %[nh], %[v]\n" \
+ " cbnz %w[tmp], 1b\n" \
+ " " #mb "\n" \
+ "2:" \
+ : [v] "+Q" (*(u128 *)ptr), \
+ [rl] "=&r" (r.low), [rh] "=&r" (r.high), \
+ [tmp] "=&r" (tmp) \
+ : [ol] "r" (o.low), [oh] "r" (o.high), \
+ [nl] "r" (n.low), [nh] "r" (n.high) \
+ : "cc", ##cl); \
+ \
+ return r.full; \
+}
+
+__CMPXCHG128( , , )
+__CMPXCHG128(_mb, dmb ish, l, "memory")
+
+#undef __CMPXCHG128
+
#undef K
#endif /* __ASM_ATOMIC_LL_SC_H */
--- a/arch/arm64/include/asm/atomic_lse.h
+++ b/arch/arm64/include/asm/atomic_lse.h
@@ -317,4 +317,35 @@ __CMPXCHG_DBL(_mb, al, "memory")
#undef __CMPXCHG_DBL
+#define __CMPXCHG128(name, mb, cl...) \
+static __always_inline u128 \
+__lse__cmpxchg128##name(volatile u128 *ptr, u128 old, u128 new) \
+{ \
+ union __u128_halves r, o = { .full = (old) }, \
+ n = { .full = (new) }; \
+ register unsigned long x0 asm ("x0") = o.low; \
+ register unsigned long x1 asm ("x1") = o.high; \
+ register unsigned long x2 asm ("x2") = n.low; \
+ register unsigned long x3 asm ("x3") = n.high; \
+ register unsigned long x4 asm ("x4") = (unsigned long)ptr; \
+ \
+ asm volatile( \
+ __LSE_PREAMBLE \
+ " casp" #mb "\t%[old1], %[old2], %[new1], %[new2], %[v]\n"\
+ : [old1] "+&r" (x0), [old2] "+&r" (x1), \
+ [v] "+Q" (*(u128 *)ptr) \
+ : [new1] "r" (x2), [new2] "r" (x3), [ptr] "r" (x4), \
+ [oldval1] "r" (o.low), [oldval2] "r" (o.high) \
+ : cl); \
+ \
+ r.low = x0; r.high = x1; \
+ \
+ return r.full; \
+}
+
+__CMPXCHG128( , )
+__CMPXCHG128(_mb, al, "memory")
+
+#undef __CMPXCHG128
+
#endif /* __ASM_ATOMIC_LSE_H */
--- a/arch/arm64/include/asm/cmpxchg.h
+++ b/arch/arm64/include/asm/cmpxchg.h
@@ -146,6 +146,19 @@ __CMPXCHG_DBL(_mb)
#undef __CMPXCHG_DBL
+#define __CMPXCHG128(name) \
+static inline u128 __cmpxchg128##name(volatile u128 *ptr, \
+ u128 old, u128 new) \
+{ \
+ return __lse_ll_sc_body(_cmpxchg128##name, \
+ ptr, old, new); \
+}
+
+__CMPXCHG128( )
+__CMPXCHG128(_mb)
+
+#undef __CMPXCHG128
+
#define __CMPXCHG_GEN(sfx) \
static __always_inline unsigned long __cmpxchg##sfx(volatile void *ptr, \
unsigned long old, \
@@ -228,6 +241,19 @@ __CMPXCHG_GEN(_mb)
__ret; \
})
+/* cmpxchg128 */
+#define system_has_cmpxchg128() 1
+
+#define arch_cmpxchg128(ptr, o, n) \
+({ \
+ __cmpxchg128_mb((ptr), (o), (n)); \
+})
+
+#define arch_cmpxchg128_local(ptr, o, n) \
+({ \
+ __cmpxchg128((ptr), (o), (n)); \
+})
+
#define __CMPWAIT_CASE(w, sfx, sz) \
static inline void __cmpwait_case_##sz(volatile void *ptr, \
unsigned long val) \
--- a/arch/s390/include/asm/cmpxchg.h
+++ b/arch/s390/include/asm/cmpxchg.h
@@ -224,4 +224,18 @@ static __always_inline int __cmpxchg_dou
(unsigned long)(n1), (unsigned long)(n2)); \
})
+#define system_has_cmpxchg128() 1
+
+static __always_inline u128 arch_cmpxchg128(volatile u128 *ptr, u128 old, u128 new)
+{
+ asm volatile(
+ " cdsg %[old],%[new],%[ptr]\n"
+ : [old] "+d" (old), [ptr] "+QS" (*ptr)
+ : [new] "d" (new)
+ : "memory", "cc");
+ return old;
+}
+
+#define arch_cmpxchg128 arch_cmpxchg128
+
#endif /* __ASM_CMPXCHG_H */
--- a/arch/x86/include/asm/cmpxchg_32.h
+++ b/arch/x86/include/asm/cmpxchg_32.h
@@ -103,6 +103,7 @@ static inline bool __try_cmpxchg64(volat
#endif
-#define system_has_cmpxchg_double() boot_cpu_has(X86_FEATURE_CX8)
+#define system_has_cmpxchg_double() boot_cpu_has(X86_FEATURE_CX8)
+#define system_has_cmpxchg64() boot_cpu_has(X86_FEATURE_CX8)
#endif /* _ASM_X86_CMPXCHG_32_H */
--- a/arch/x86/include/asm/cmpxchg_64.h
+++ b/arch/x86/include/asm/cmpxchg_64.h
@@ -20,6 +20,68 @@
arch_try_cmpxchg((ptr), (po), (n)); \
})
-#define system_has_cmpxchg_double() boot_cpu_has(X86_FEATURE_CX16)
+union __u128_halves {
+ u128 full;
+ struct {
+ u64 low, high;
+ };
+};
+
+#define __arch_cmpxchg128(_ptr, _old, _new, _lock) \
+({ \
+ union __u128_halves o = { .full = (_old), }, \
+ n = { .full = (_new), }; \
+ \
+ asm volatile(_lock "cmpxchg16b %[ptr]" \
+ : [ptr] "+m" (*(_ptr)), \
+ "+a" (o.low), "+d" (o.high) \
+ : "b" (n.low), "c" (n.high) \
+ : "memory"); \
+ \
+ o.full; \
+})
+
+static __always_inline u128 arch_cmpxchg128(volatile u128 *ptr, u128 old, u128 new)
+{
+ return __arch_cmpxchg128(ptr, old, new, LOCK_PREFIX);
+}
+
+static __always_inline u128 arch_cmpxchg128_local(volatile u128 *ptr, u128 old, u128 new)
+{
+ return __arch_cmpxchg128(ptr, old, new,);
+}
+
+#define __arch_try_cmpxchg128(_ptr, _oldp, _new, _lock) \
+({ \
+ union __u128_halves o = { .full = *(_oldp), }, \
+ n = { .full = (_new), }; \
+ bool ret; \
+ \
+ asm volatile(_lock "cmpxchg16b %[ptr]" \
+ CC_SET(e) \
+ : CC_OUT(e) (ret), \
+ [ptr] "+m" (*ptr), \
+ "+a" (o.low), "+d" (o.high) \
+ : "b" (n.low), "c" (n.high) \
+ : "memory"); \
+ \
+ if (unlikely(!ret)) \
+ *(_oldp) = o.full; \
+ \
+ likely(ret); \
+})
+
+static __always_inline bool arch_try_cmpxchg128(volatile u128 *ptr, u128 *oldp, u128 new)
+{
+ return __arch_try_cmpxchg128(ptr, oldp, new, LOCK_PREFIX);
+}
+
+static __always_inline bool arch_try_cmpxchg128_local(volatile u128 *ptr, u128 *oldp, u128 new)
+{
+ return __arch_try_cmpxchg128(ptr, oldp, new,);
+}
+
+#define system_has_cmpxchg_double() boot_cpu_has(X86_FEATURE_CX16)
+#define system_has_cmpxchg128() boot_cpu_has(X86_FEATURE_CX16)
#endif /* _ASM_X86_CMPXCHG_64_H */
^ permalink raw reply [flat|nested] 47+ messages in thread* [tip: locking/core] arch: Introduce arch_{,try_}_cmpxchg128{,_local}()
2023-05-31 13:08 ` [PATCH 03/12] arch: Introduce arch_{,try_}_cmpxchg128{,_local}() Peter Zijlstra
@ 2023-06-05 7:42 ` tip-bot2 for Peter Zijlstra
0 siblings, 0 replies; 47+ messages in thread
From: tip-bot2 for Peter Zijlstra @ 2023-06-05 7:42 UTC (permalink / raw)
To: linux-tip-commits
Cc: Peter Zijlstra (Intel),
Arnd Bergmann, Mark Rutland, Heiko Carstens, x86, linux-kernel
The following commit has been merged into the locking/core branch of tip:
Commit-ID: b23e139d0b66c0216e7e9361a5021290395f504c
Gitweb: https://git.kernel.org/tip/b23e139d0b66c0216e7e9361a5021290395f504c
Author: Peter Zijlstra <peterz@infradead.org>
AuthorDate: Wed, 31 May 2023 15:08:36 +02:00
Committer: Peter Zijlstra <peterz@infradead.org>
CommitterDate: Mon, 05 Jun 2023 09:36:35 +02:00
arch: Introduce arch_{,try_}_cmpxchg128{,_local}()
For all architectures that currently support cmpxchg_double()
implement the cmpxchg128() family of functions that is basically the
same but with a saner interface.
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Reviewed-by: Arnd Bergmann <arnd@arndb.de>
Reviewed-by: Mark Rutland <mark.rutland@arm.com>
Acked-by: Heiko Carstens <hca@linux.ibm.com>
Acked-by: Mark Rutland <mark.rutland@arm.com>
Tested-by: Mark Rutland <mark.rutland@arm.com>
Link: https://lore.kernel.org/r/20230531132323.452120708@infradead.org
---
arch/arm64/include/asm/atomic_ll_sc.h | 41 +++++++++++++++++-
arch/arm64/include/asm/atomic_lse.h | 31 +++++++++++++-
arch/arm64/include/asm/cmpxchg.h | 26 +++++++++++-
arch/s390/include/asm/cmpxchg.h | 14 ++++++-
arch/x86/include/asm/cmpxchg_32.h | 3 +-
arch/x86/include/asm/cmpxchg_64.h | 64 +++++++++++++++++++++++++-
6 files changed, 177 insertions(+), 2 deletions(-)
diff --git a/arch/arm64/include/asm/atomic_ll_sc.h b/arch/arm64/include/asm/atomic_ll_sc.h
index cbb3d96..b100a49 100644
--- a/arch/arm64/include/asm/atomic_ll_sc.h
+++ b/arch/arm64/include/asm/atomic_ll_sc.h
@@ -326,6 +326,47 @@ __CMPXCHG_DBL( , , , )
__CMPXCHG_DBL(_mb, dmb ish, l, "memory")
#undef __CMPXCHG_DBL
+
+union __u128_halves {
+ u128 full;
+ struct {
+ u64 low, high;
+ };
+};
+
+#define __CMPXCHG128(name, mb, rel, cl...) \
+static __always_inline u128 \
+__ll_sc__cmpxchg128##name(volatile u128 *ptr, u128 old, u128 new) \
+{ \
+ union __u128_halves r, o = { .full = (old) }, \
+ n = { .full = (new) }; \
+ unsigned int tmp; \
+ \
+ asm volatile("// __cmpxchg128" #name "\n" \
+ " prfm pstl1strm, %[v]\n" \
+ "1: ldxp %[rl], %[rh], %[v]\n" \
+ " cmp %[rl], %[ol]\n" \
+ " ccmp %[rh], %[oh], 0, eq\n" \
+ " b.ne 2f\n" \
+ " st" #rel "xp %w[tmp], %[nl], %[nh], %[v]\n" \
+ " cbnz %w[tmp], 1b\n" \
+ " " #mb "\n" \
+ "2:" \
+ : [v] "+Q" (*(u128 *)ptr), \
+ [rl] "=&r" (r.low), [rh] "=&r" (r.high), \
+ [tmp] "=&r" (tmp) \
+ : [ol] "r" (o.low), [oh] "r" (o.high), \
+ [nl] "r" (n.low), [nh] "r" (n.high) \
+ : "cc", ##cl); \
+ \
+ return r.full; \
+}
+
+__CMPXCHG128( , , )
+__CMPXCHG128(_mb, dmb ish, l, "memory")
+
+#undef __CMPXCHG128
+
#undef K
#endif /* __ASM_ATOMIC_LL_SC_H */
diff --git a/arch/arm64/include/asm/atomic_lse.h b/arch/arm64/include/asm/atomic_lse.h
index 319958b..c781281 100644
--- a/arch/arm64/include/asm/atomic_lse.h
+++ b/arch/arm64/include/asm/atomic_lse.h
@@ -317,4 +317,35 @@ __CMPXCHG_DBL(_mb, al, "memory")
#undef __CMPXCHG_DBL
+#define __CMPXCHG128(name, mb, cl...) \
+static __always_inline u128 \
+__lse__cmpxchg128##name(volatile u128 *ptr, u128 old, u128 new) \
+{ \
+ union __u128_halves r, o = { .full = (old) }, \
+ n = { .full = (new) }; \
+ register unsigned long x0 asm ("x0") = o.low; \
+ register unsigned long x1 asm ("x1") = o.high; \
+ register unsigned long x2 asm ("x2") = n.low; \
+ register unsigned long x3 asm ("x3") = n.high; \
+ register unsigned long x4 asm ("x4") = (unsigned long)ptr; \
+ \
+ asm volatile( \
+ __LSE_PREAMBLE \
+ " casp" #mb "\t%[old1], %[old2], %[new1], %[new2], %[v]\n"\
+ : [old1] "+&r" (x0), [old2] "+&r" (x1), \
+ [v] "+Q" (*(u128 *)ptr) \
+ : [new1] "r" (x2), [new2] "r" (x3), [ptr] "r" (x4), \
+ [oldval1] "r" (o.low), [oldval2] "r" (o.high) \
+ : cl); \
+ \
+ r.low = x0; r.high = x1; \
+ \
+ return r.full; \
+}
+
+__CMPXCHG128( , )
+__CMPXCHG128(_mb, al, "memory")
+
+#undef __CMPXCHG128
+
#endif /* __ASM_ATOMIC_LSE_H */
diff --git a/arch/arm64/include/asm/cmpxchg.h b/arch/arm64/include/asm/cmpxchg.h
index c6bc5d8..097b832 100644
--- a/arch/arm64/include/asm/cmpxchg.h
+++ b/arch/arm64/include/asm/cmpxchg.h
@@ -146,6 +146,19 @@ __CMPXCHG_DBL(_mb)
#undef __CMPXCHG_DBL
+#define __CMPXCHG128(name) \
+static inline u128 __cmpxchg128##name(volatile u128 *ptr, \
+ u128 old, u128 new) \
+{ \
+ return __lse_ll_sc_body(_cmpxchg128##name, \
+ ptr, old, new); \
+}
+
+__CMPXCHG128( )
+__CMPXCHG128(_mb)
+
+#undef __CMPXCHG128
+
#define __CMPXCHG_GEN(sfx) \
static __always_inline unsigned long __cmpxchg##sfx(volatile void *ptr, \
unsigned long old, \
@@ -228,6 +241,19 @@ __CMPXCHG_GEN(_mb)
__ret; \
})
+/* cmpxchg128 */
+#define system_has_cmpxchg128() 1
+
+#define arch_cmpxchg128(ptr, o, n) \
+({ \
+ __cmpxchg128_mb((ptr), (o), (n)); \
+})
+
+#define arch_cmpxchg128_local(ptr, o, n) \
+({ \
+ __cmpxchg128((ptr), (o), (n)); \
+})
+
#define __CMPWAIT_CASE(w, sfx, sz) \
static inline void __cmpwait_case_##sz(volatile void *ptr, \
unsigned long val) \
diff --git a/arch/s390/include/asm/cmpxchg.h b/arch/s390/include/asm/cmpxchg.h
index 06e0e42..7517376 100644
--- a/arch/s390/include/asm/cmpxchg.h
+++ b/arch/s390/include/asm/cmpxchg.h
@@ -224,4 +224,18 @@ static __always_inline int __cmpxchg_double(unsigned long p1, unsigned long p2,
(unsigned long)(n1), (unsigned long)(n2)); \
})
+#define system_has_cmpxchg128() 1
+
+static __always_inline u128 arch_cmpxchg128(volatile u128 *ptr, u128 old, u128 new)
+{
+ asm volatile(
+ " cdsg %[old],%[new],%[ptr]\n"
+ : [old] "+d" (old), [ptr] "+QS" (*ptr)
+ : [new] "d" (new)
+ : "memory", "cc");
+ return old;
+}
+
+#define arch_cmpxchg128 arch_cmpxchg128
+
#endif /* __ASM_CMPXCHG_H */
diff --git a/arch/x86/include/asm/cmpxchg_32.h b/arch/x86/include/asm/cmpxchg_32.h
index 6ba80ce..d12f9ed 100644
--- a/arch/x86/include/asm/cmpxchg_32.h
+++ b/arch/x86/include/asm/cmpxchg_32.h
@@ -103,6 +103,7 @@ static inline bool __try_cmpxchg64(volatile u64 *ptr, u64 *pold, u64 new)
#endif
-#define system_has_cmpxchg_double() boot_cpu_has(X86_FEATURE_CX8)
+#define system_has_cmpxchg_double() boot_cpu_has(X86_FEATURE_CX8)
+#define system_has_cmpxchg64() boot_cpu_has(X86_FEATURE_CX8)
#endif /* _ASM_X86_CMPXCHG_32_H */
diff --git a/arch/x86/include/asm/cmpxchg_64.h b/arch/x86/include/asm/cmpxchg_64.h
index 0d3beb2..fe29107 100644
--- a/arch/x86/include/asm/cmpxchg_64.h
+++ b/arch/x86/include/asm/cmpxchg_64.h
@@ -20,6 +20,68 @@
arch_try_cmpxchg((ptr), (po), (n)); \
})
-#define system_has_cmpxchg_double() boot_cpu_has(X86_FEATURE_CX16)
+union __u128_halves {
+ u128 full;
+ struct {
+ u64 low, high;
+ };
+};
+
+#define __arch_cmpxchg128(_ptr, _old, _new, _lock) \
+({ \
+ union __u128_halves o = { .full = (_old), }, \
+ n = { .full = (_new), }; \
+ \
+ asm volatile(_lock "cmpxchg16b %[ptr]" \
+ : [ptr] "+m" (*(_ptr)), \
+ "+a" (o.low), "+d" (o.high) \
+ : "b" (n.low), "c" (n.high) \
+ : "memory"); \
+ \
+ o.full; \
+})
+
+static __always_inline u128 arch_cmpxchg128(volatile u128 *ptr, u128 old, u128 new)
+{
+ return __arch_cmpxchg128(ptr, old, new, LOCK_PREFIX);
+}
+
+static __always_inline u128 arch_cmpxchg128_local(volatile u128 *ptr, u128 old, u128 new)
+{
+ return __arch_cmpxchg128(ptr, old, new,);
+}
+
+#define __arch_try_cmpxchg128(_ptr, _oldp, _new, _lock) \
+({ \
+ union __u128_halves o = { .full = *(_oldp), }, \
+ n = { .full = (_new), }; \
+ bool ret; \
+ \
+ asm volatile(_lock "cmpxchg16b %[ptr]" \
+ CC_SET(e) \
+ : CC_OUT(e) (ret), \
+ [ptr] "+m" (*ptr), \
+ "+a" (o.low), "+d" (o.high) \
+ : "b" (n.low), "c" (n.high) \
+ : "memory"); \
+ \
+ if (unlikely(!ret)) \
+ *(_oldp) = o.full; \
+ \
+ likely(ret); \
+})
+
+static __always_inline bool arch_try_cmpxchg128(volatile u128 *ptr, u128 *oldp, u128 new)
+{
+ return __arch_try_cmpxchg128(ptr, oldp, new, LOCK_PREFIX);
+}
+
+static __always_inline bool arch_try_cmpxchg128_local(volatile u128 *ptr, u128 *oldp, u128 new)
+{
+ return __arch_try_cmpxchg128(ptr, oldp, new,);
+}
+
+#define system_has_cmpxchg_double() boot_cpu_has(X86_FEATURE_CX16)
+#define system_has_cmpxchg128() boot_cpu_has(X86_FEATURE_CX16)
#endif /* _ASM_X86_CMPXCHG_64_H */
^ permalink raw reply [flat|nested] 47+ messages in thread
* [PATCH 04/12] instrumentation: Wire up cmpxchg128()
2023-05-31 13:08 [PATCH 00/12] Introduce cmpxchg128() -- aka. the demise of cmpxchg_double() Peter Zijlstra
` (2 preceding siblings ...)
2023-05-31 13:08 ` [PATCH 03/12] arch: Introduce arch_{,try_}_cmpxchg128{,_local}() Peter Zijlstra
@ 2023-05-31 13:08 ` Peter Zijlstra
2023-06-05 7:42 ` [tip: locking/core] " tip-bot2 for Peter Zijlstra
2023-05-31 13:08 ` [PATCH 05/12] percpu: Add {raw,this}_cpu_try_cmpxchg() Peter Zijlstra
` (8 subsequent siblings)
12 siblings, 1 reply; 47+ messages in thread
From: Peter Zijlstra @ 2023-05-31 13:08 UTC (permalink / raw)
To: torvalds
Cc: corbet, will, peterz, boqun.feng, mark.rutland, catalin.marinas,
dennis, tj, cl, hca, gor, agordeev, borntraeger, svens, tglx,
mingo, bp, dave.hansen, x86, hpa, joro, suravee.suthikulpanit,
robin.murphy, dwmw2, baolu.lu, Arnd Bergmann, Herbert Xu, davem,
penberg, rientjes, iamjoonsoo.kim, Andrew Morton, vbabka,
roman.gushchin, 42.hyeyoo, linux-doc, linux-kernel, linux-mm,
linux-s390, iommu, linux-arch, linux-crypto, sfr, mpe,
James.Bottomley, deller, linux-parisc
Wire up the cmpxchg128 family in the atomic wrapper scripts.
These provide the generic cmpxchg128 family of functions from the
arch_ prefixed version, adding explicit instrumentation where needed.
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Reviewed-by: Arnd Bergmann <arnd@arndb.de>
Acked-by: Mark Rutland <mark.rutland@arm.com>
---
include/linux/atomic/atomic-arch-fallback.h | 95 +++++++++++++++++++++++++++-
include/linux/atomic/atomic-instrumented.h | 86 +++++++++++++++++++++++++
scripts/atomic/gen-atomic-fallback.sh | 4 -
scripts/atomic/gen-atomic-instrumented.sh | 4 -
4 files changed, 183 insertions(+), 6 deletions(-)
--- a/include/linux/atomic/atomic-arch-fallback.h
+++ b/include/linux/atomic/atomic-arch-fallback.h
@@ -77,6 +77,29 @@
#endif /* arch_cmpxchg64_relaxed */
+#ifndef arch_cmpxchg128_relaxed
+#define arch_cmpxchg128_acquire arch_cmpxchg128
+#define arch_cmpxchg128_release arch_cmpxchg128
+#define arch_cmpxchg128_relaxed arch_cmpxchg128
+#else /* arch_cmpxchg128_relaxed */
+
+#ifndef arch_cmpxchg128_acquire
+#define arch_cmpxchg128_acquire(...) \
+ __atomic_op_acquire(arch_cmpxchg128, __VA_ARGS__)
+#endif
+
+#ifndef arch_cmpxchg128_release
+#define arch_cmpxchg128_release(...) \
+ __atomic_op_release(arch_cmpxchg128, __VA_ARGS__)
+#endif
+
+#ifndef arch_cmpxchg128
+#define arch_cmpxchg128(...) \
+ __atomic_op_fence(arch_cmpxchg128, __VA_ARGS__)
+#endif
+
+#endif /* arch_cmpxchg128_relaxed */
+
#ifndef arch_try_cmpxchg_relaxed
#ifdef arch_try_cmpxchg
#define arch_try_cmpxchg_acquire arch_try_cmpxchg
@@ -217,6 +240,76 @@
#endif /* arch_try_cmpxchg64_relaxed */
+#ifndef arch_try_cmpxchg128_relaxed
+#ifdef arch_try_cmpxchg128
+#define arch_try_cmpxchg128_acquire arch_try_cmpxchg128
+#define arch_try_cmpxchg128_release arch_try_cmpxchg128
+#define arch_try_cmpxchg128_relaxed arch_try_cmpxchg128
+#endif /* arch_try_cmpxchg128 */
+
+#ifndef arch_try_cmpxchg128
+#define arch_try_cmpxchg128(_ptr, _oldp, _new) \
+({ \
+ typeof(*(_ptr)) *___op = (_oldp), ___o = *___op, ___r; \
+ ___r = arch_cmpxchg128((_ptr), ___o, (_new)); \
+ if (unlikely(___r != ___o)) \
+ *___op = ___r; \
+ likely(___r == ___o); \
+})
+#endif /* arch_try_cmpxchg128 */
+
+#ifndef arch_try_cmpxchg128_acquire
+#define arch_try_cmpxchg128_acquire(_ptr, _oldp, _new) \
+({ \
+ typeof(*(_ptr)) *___op = (_oldp), ___o = *___op, ___r; \
+ ___r = arch_cmpxchg128_acquire((_ptr), ___o, (_new)); \
+ if (unlikely(___r != ___o)) \
+ *___op = ___r; \
+ likely(___r == ___o); \
+})
+#endif /* arch_try_cmpxchg128_acquire */
+
+#ifndef arch_try_cmpxchg128_release
+#define arch_try_cmpxchg128_release(_ptr, _oldp, _new) \
+({ \
+ typeof(*(_ptr)) *___op = (_oldp), ___o = *___op, ___r; \
+ ___r = arch_cmpxchg128_release((_ptr), ___o, (_new)); \
+ if (unlikely(___r != ___o)) \
+ *___op = ___r; \
+ likely(___r == ___o); \
+})
+#endif /* arch_try_cmpxchg128_release */
+
+#ifndef arch_try_cmpxchg128_relaxed
+#define arch_try_cmpxchg128_relaxed(_ptr, _oldp, _new) \
+({ \
+ typeof(*(_ptr)) *___op = (_oldp), ___o = *___op, ___r; \
+ ___r = arch_cmpxchg128_relaxed((_ptr), ___o, (_new)); \
+ if (unlikely(___r != ___o)) \
+ *___op = ___r; \
+ likely(___r == ___o); \
+})
+#endif /* arch_try_cmpxchg128_relaxed */
+
+#else /* arch_try_cmpxchg128_relaxed */
+
+#ifndef arch_try_cmpxchg128_acquire
+#define arch_try_cmpxchg128_acquire(...) \
+ __atomic_op_acquire(arch_try_cmpxchg128, __VA_ARGS__)
+#endif
+
+#ifndef arch_try_cmpxchg128_release
+#define arch_try_cmpxchg128_release(...) \
+ __atomic_op_release(arch_try_cmpxchg128, __VA_ARGS__)
+#endif
+
+#ifndef arch_try_cmpxchg128
+#define arch_try_cmpxchg128(...) \
+ __atomic_op_fence(arch_try_cmpxchg128, __VA_ARGS__)
+#endif
+
+#endif /* arch_try_cmpxchg128_relaxed */
+
#ifndef arch_try_cmpxchg_local
#define arch_try_cmpxchg_local(_ptr, _oldp, _new) \
({ \
@@ -2668,4 +2761,4 @@ arch_atomic64_dec_if_positive(atomic64_t
#endif
#endif /* _LINUX_ATOMIC_FALLBACK_H */
-// ad2e2b4d168dbc60a73922616047a9bfa446af36
+// 52dfc6fe4a2e7234bbd2aa3e16a377c1db793a53
--- a/include/linux/atomic/atomic-instrumented.h
+++ b/include/linux/atomic/atomic-instrumented.h
@@ -2034,6 +2034,36 @@ atomic_long_dec_if_positive(atomic_long_
arch_cmpxchg64_relaxed(__ai_ptr, __VA_ARGS__); \
})
+#define cmpxchg128(ptr, ...) \
+({ \
+ typeof(ptr) __ai_ptr = (ptr); \
+ kcsan_mb(); \
+ instrument_atomic_read_write(__ai_ptr, sizeof(*__ai_ptr)); \
+ arch_cmpxchg128(__ai_ptr, __VA_ARGS__); \
+})
+
+#define cmpxchg128_acquire(ptr, ...) \
+({ \
+ typeof(ptr) __ai_ptr = (ptr); \
+ instrument_atomic_read_write(__ai_ptr, sizeof(*__ai_ptr)); \
+ arch_cmpxchg128_acquire(__ai_ptr, __VA_ARGS__); \
+})
+
+#define cmpxchg128_release(ptr, ...) \
+({ \
+ typeof(ptr) __ai_ptr = (ptr); \
+ kcsan_release(); \
+ instrument_atomic_read_write(__ai_ptr, sizeof(*__ai_ptr)); \
+ arch_cmpxchg128_release(__ai_ptr, __VA_ARGS__); \
+})
+
+#define cmpxchg128_relaxed(ptr, ...) \
+({ \
+ typeof(ptr) __ai_ptr = (ptr); \
+ instrument_atomic_read_write(__ai_ptr, sizeof(*__ai_ptr)); \
+ arch_cmpxchg128_relaxed(__ai_ptr, __VA_ARGS__); \
+})
+
#define try_cmpxchg(ptr, oldp, ...) \
({ \
typeof(ptr) __ai_ptr = (ptr); \
@@ -2110,6 +2140,44 @@ atomic_long_dec_if_positive(atomic_long_
arch_try_cmpxchg64_relaxed(__ai_ptr, __ai_oldp, __VA_ARGS__); \
})
+#define try_cmpxchg128(ptr, oldp, ...) \
+({ \
+ typeof(ptr) __ai_ptr = (ptr); \
+ typeof(oldp) __ai_oldp = (oldp); \
+ kcsan_mb(); \
+ instrument_atomic_read_write(__ai_ptr, sizeof(*__ai_ptr)); \
+ instrument_read_write(__ai_oldp, sizeof(*__ai_oldp)); \
+ arch_try_cmpxchg128(__ai_ptr, __ai_oldp, __VA_ARGS__); \
+})
+
+#define try_cmpxchg128_acquire(ptr, oldp, ...) \
+({ \
+ typeof(ptr) __ai_ptr = (ptr); \
+ typeof(oldp) __ai_oldp = (oldp); \
+ instrument_atomic_read_write(__ai_ptr, sizeof(*__ai_ptr)); \
+ instrument_read_write(__ai_oldp, sizeof(*__ai_oldp)); \
+ arch_try_cmpxchg128_acquire(__ai_ptr, __ai_oldp, __VA_ARGS__); \
+})
+
+#define try_cmpxchg128_release(ptr, oldp, ...) \
+({ \
+ typeof(ptr) __ai_ptr = (ptr); \
+ typeof(oldp) __ai_oldp = (oldp); \
+ kcsan_release(); \
+ instrument_atomic_read_write(__ai_ptr, sizeof(*__ai_ptr)); \
+ instrument_read_write(__ai_oldp, sizeof(*__ai_oldp)); \
+ arch_try_cmpxchg128_release(__ai_ptr, __ai_oldp, __VA_ARGS__); \
+})
+
+#define try_cmpxchg128_relaxed(ptr, oldp, ...) \
+({ \
+ typeof(ptr) __ai_ptr = (ptr); \
+ typeof(oldp) __ai_oldp = (oldp); \
+ instrument_atomic_read_write(__ai_ptr, sizeof(*__ai_ptr)); \
+ instrument_read_write(__ai_oldp, sizeof(*__ai_oldp)); \
+ arch_try_cmpxchg128_relaxed(__ai_ptr, __ai_oldp, __VA_ARGS__); \
+})
+
#define cmpxchg_local(ptr, ...) \
({ \
typeof(ptr) __ai_ptr = (ptr); \
@@ -2124,6 +2192,13 @@ atomic_long_dec_if_positive(atomic_long_
arch_cmpxchg64_local(__ai_ptr, __VA_ARGS__); \
})
+#define cmpxchg128_local(ptr, ...) \
+({ \
+ typeof(ptr) __ai_ptr = (ptr); \
+ instrument_atomic_read_write(__ai_ptr, sizeof(*__ai_ptr)); \
+ arch_cmpxchg128_local(__ai_ptr, __VA_ARGS__); \
+})
+
#define sync_cmpxchg(ptr, ...) \
({ \
typeof(ptr) __ai_ptr = (ptr); \
@@ -2150,6 +2225,15 @@ atomic_long_dec_if_positive(atomic_long_
arch_try_cmpxchg64_local(__ai_ptr, __ai_oldp, __VA_ARGS__); \
})
+#define try_cmpxchg128_local(ptr, oldp, ...) \
+({ \
+ typeof(ptr) __ai_ptr = (ptr); \
+ typeof(oldp) __ai_oldp = (oldp); \
+ instrument_atomic_read_write(__ai_ptr, sizeof(*__ai_ptr)); \
+ instrument_read_write(__ai_oldp, sizeof(*__ai_oldp)); \
+ arch_try_cmpxchg128_local(__ai_ptr, __ai_oldp, __VA_ARGS__); \
+})
+
#define cmpxchg_double(ptr, ...) \
({ \
typeof(ptr) __ai_ptr = (ptr); \
@@ -2167,4 +2251,4 @@ atomic_long_dec_if_positive(atomic_long_
})
#endif /* _LINUX_ATOMIC_INSTRUMENTED_H */
-// 6b513a42e1a1b5962532a019b7fc91eaa044ad5e
+// 82d1be694fab30414527d0877c29fa75ed5a0b74
--- a/scripts/atomic/gen-atomic-fallback.sh
+++ b/scripts/atomic/gen-atomic-fallback.sh
@@ -217,11 +217,11 @@ cat << EOF
EOF
-for xchg in "arch_xchg" "arch_cmpxchg" "arch_cmpxchg64"; do
+for xchg in "arch_xchg" "arch_cmpxchg" "arch_cmpxchg64" "arch_cmpxchg128"; do
gen_xchg_fallbacks "${xchg}"
done
-for cmpxchg in "cmpxchg" "cmpxchg64"; do
+for cmpxchg in "cmpxchg" "cmpxchg64" "cmpxchg128"; do
gen_try_cmpxchg_fallbacks "${cmpxchg}"
done
--- a/scripts/atomic/gen-atomic-instrumented.sh
+++ b/scripts/atomic/gen-atomic-instrumented.sh
@@ -166,14 +166,14 @@ grep '^[a-z]' "$1" | while read name met
done
-for xchg in "xchg" "cmpxchg" "cmpxchg64" "try_cmpxchg" "try_cmpxchg64"; do
+for xchg in "xchg" "cmpxchg" "cmpxchg64" "cmpxchg128" "try_cmpxchg" "try_cmpxchg64" "try_cmpxchg128"; do
for order in "" "_acquire" "_release" "_relaxed"; do
gen_xchg "${xchg}" "${order}" ""
printf "\n"
done
done
-for xchg in "cmpxchg_local" "cmpxchg64_local" "sync_cmpxchg" "try_cmpxchg_local" "try_cmpxchg64_local" ; do
+for xchg in "cmpxchg_local" "cmpxchg64_local" "cmpxchg128_local" "sync_cmpxchg" "try_cmpxchg_local" "try_cmpxchg64_local" "try_cmpxchg128_local"; do
gen_xchg "${xchg}" "" ""
printf "\n"
done
^ permalink raw reply [flat|nested] 47+ messages in thread* [tip: locking/core] instrumentation: Wire up cmpxchg128()
2023-05-31 13:08 ` [PATCH 04/12] instrumentation: Wire up cmpxchg128() Peter Zijlstra
@ 2023-06-05 7:42 ` tip-bot2 for Peter Zijlstra
0 siblings, 0 replies; 47+ messages in thread
From: tip-bot2 for Peter Zijlstra @ 2023-06-05 7:42 UTC (permalink / raw)
To: linux-tip-commits
Cc: Peter Zijlstra (Intel), Arnd Bergmann, Mark Rutland, x86, linux-kernel
The following commit has been merged into the locking/core branch of tip:
Commit-ID: 8c8b096a23d12fedf3c0f50524f30113ef97aa8c
Gitweb: https://git.kernel.org/tip/8c8b096a23d12fedf3c0f50524f30113ef97aa8c
Author: Peter Zijlstra <peterz@infradead.org>
AuthorDate: Wed, 31 May 2023 15:08:37 +02:00
Committer: Peter Zijlstra <peterz@infradead.org>
CommitterDate: Mon, 05 Jun 2023 09:36:36 +02:00
instrumentation: Wire up cmpxchg128()
Wire up the cmpxchg128 family in the atomic wrapper scripts.
These provide the generic cmpxchg128 family of functions from the
arch_ prefixed version, adding explicit instrumentation where needed.
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Reviewed-by: Arnd Bergmann <arnd@arndb.de>
Reviewed-by: Mark Rutland <mark.rutland@arm.com>
Acked-by: Mark Rutland <mark.rutland@arm.com>
Tested-by: Mark Rutland <mark.rutland@arm.com>
Link: https://lore.kernel.org/r/20230531132323.519237070@infradead.org
---
include/linux/atomic/atomic-arch-fallback.h | 95 +++++++++++++++++++-
include/linux/atomic/atomic-instrumented.h | 86 +++++++++++++++++-
scripts/atomic/gen-atomic-fallback.sh | 4 +-
scripts/atomic/gen-atomic-instrumented.sh | 4 +-
4 files changed, 183 insertions(+), 6 deletions(-)
diff --git a/include/linux/atomic/atomic-arch-fallback.h b/include/linux/atomic/atomic-arch-fallback.h
index a6e4437..1722ddb 100644
--- a/include/linux/atomic/atomic-arch-fallback.h
+++ b/include/linux/atomic/atomic-arch-fallback.h
@@ -77,6 +77,29 @@
#endif /* arch_cmpxchg64_relaxed */
+#ifndef arch_cmpxchg128_relaxed
+#define arch_cmpxchg128_acquire arch_cmpxchg128
+#define arch_cmpxchg128_release arch_cmpxchg128
+#define arch_cmpxchg128_relaxed arch_cmpxchg128
+#else /* arch_cmpxchg128_relaxed */
+
+#ifndef arch_cmpxchg128_acquire
+#define arch_cmpxchg128_acquire(...) \
+ __atomic_op_acquire(arch_cmpxchg128, __VA_ARGS__)
+#endif
+
+#ifndef arch_cmpxchg128_release
+#define arch_cmpxchg128_release(...) \
+ __atomic_op_release(arch_cmpxchg128, __VA_ARGS__)
+#endif
+
+#ifndef arch_cmpxchg128
+#define arch_cmpxchg128(...) \
+ __atomic_op_fence(arch_cmpxchg128, __VA_ARGS__)
+#endif
+
+#endif /* arch_cmpxchg128_relaxed */
+
#ifndef arch_try_cmpxchg_relaxed
#ifdef arch_try_cmpxchg
#define arch_try_cmpxchg_acquire arch_try_cmpxchg
@@ -217,6 +240,76 @@
#endif /* arch_try_cmpxchg64_relaxed */
+#ifndef arch_try_cmpxchg128_relaxed
+#ifdef arch_try_cmpxchg128
+#define arch_try_cmpxchg128_acquire arch_try_cmpxchg128
+#define arch_try_cmpxchg128_release arch_try_cmpxchg128
+#define arch_try_cmpxchg128_relaxed arch_try_cmpxchg128
+#endif /* arch_try_cmpxchg128 */
+
+#ifndef arch_try_cmpxchg128
+#define arch_try_cmpxchg128(_ptr, _oldp, _new) \
+({ \
+ typeof(*(_ptr)) *___op = (_oldp), ___o = *___op, ___r; \
+ ___r = arch_cmpxchg128((_ptr), ___o, (_new)); \
+ if (unlikely(___r != ___o)) \
+ *___op = ___r; \
+ likely(___r == ___o); \
+})
+#endif /* arch_try_cmpxchg128 */
+
+#ifndef arch_try_cmpxchg128_acquire
+#define arch_try_cmpxchg128_acquire(_ptr, _oldp, _new) \
+({ \
+ typeof(*(_ptr)) *___op = (_oldp), ___o = *___op, ___r; \
+ ___r = arch_cmpxchg128_acquire((_ptr), ___o, (_new)); \
+ if (unlikely(___r != ___o)) \
+ *___op = ___r; \
+ likely(___r == ___o); \
+})
+#endif /* arch_try_cmpxchg128_acquire */
+
+#ifndef arch_try_cmpxchg128_release
+#define arch_try_cmpxchg128_release(_ptr, _oldp, _new) \
+({ \
+ typeof(*(_ptr)) *___op = (_oldp), ___o = *___op, ___r; \
+ ___r = arch_cmpxchg128_release((_ptr), ___o, (_new)); \
+ if (unlikely(___r != ___o)) \
+ *___op = ___r; \
+ likely(___r == ___o); \
+})
+#endif /* arch_try_cmpxchg128_release */
+
+#ifndef arch_try_cmpxchg128_relaxed
+#define arch_try_cmpxchg128_relaxed(_ptr, _oldp, _new) \
+({ \
+ typeof(*(_ptr)) *___op = (_oldp), ___o = *___op, ___r; \
+ ___r = arch_cmpxchg128_relaxed((_ptr), ___o, (_new)); \
+ if (unlikely(___r != ___o)) \
+ *___op = ___r; \
+ likely(___r == ___o); \
+})
+#endif /* arch_try_cmpxchg128_relaxed */
+
+#else /* arch_try_cmpxchg128_relaxed */
+
+#ifndef arch_try_cmpxchg128_acquire
+#define arch_try_cmpxchg128_acquire(...) \
+ __atomic_op_acquire(arch_try_cmpxchg128, __VA_ARGS__)
+#endif
+
+#ifndef arch_try_cmpxchg128_release
+#define arch_try_cmpxchg128_release(...) \
+ __atomic_op_release(arch_try_cmpxchg128, __VA_ARGS__)
+#endif
+
+#ifndef arch_try_cmpxchg128
+#define arch_try_cmpxchg128(...) \
+ __atomic_op_fence(arch_try_cmpxchg128, __VA_ARGS__)
+#endif
+
+#endif /* arch_try_cmpxchg128_relaxed */
+
#ifndef arch_try_cmpxchg_local
#define arch_try_cmpxchg_local(_ptr, _oldp, _new) \
({ \
@@ -2668,4 +2761,4 @@ arch_atomic64_dec_if_positive(atomic64_t *v)
#endif
#endif /* _LINUX_ATOMIC_FALLBACK_H */
-// ad2e2b4d168dbc60a73922616047a9bfa446af36
+// 52dfc6fe4a2e7234bbd2aa3e16a377c1db793a53
diff --git a/include/linux/atomic/atomic-instrumented.h b/include/linux/atomic/atomic-instrumented.h
index 03a232a..8583720 100644
--- a/include/linux/atomic/atomic-instrumented.h
+++ b/include/linux/atomic/atomic-instrumented.h
@@ -2034,6 +2034,36 @@ atomic_long_dec_if_positive(atomic_long_t *v)
arch_cmpxchg64_relaxed(__ai_ptr, __VA_ARGS__); \
})
+#define cmpxchg128(ptr, ...) \
+({ \
+ typeof(ptr) __ai_ptr = (ptr); \
+ kcsan_mb(); \
+ instrument_atomic_read_write(__ai_ptr, sizeof(*__ai_ptr)); \
+ arch_cmpxchg128(__ai_ptr, __VA_ARGS__); \
+})
+
+#define cmpxchg128_acquire(ptr, ...) \
+({ \
+ typeof(ptr) __ai_ptr = (ptr); \
+ instrument_atomic_read_write(__ai_ptr, sizeof(*__ai_ptr)); \
+ arch_cmpxchg128_acquire(__ai_ptr, __VA_ARGS__); \
+})
+
+#define cmpxchg128_release(ptr, ...) \
+({ \
+ typeof(ptr) __ai_ptr = (ptr); \
+ kcsan_release(); \
+ instrument_atomic_read_write(__ai_ptr, sizeof(*__ai_ptr)); \
+ arch_cmpxchg128_release(__ai_ptr, __VA_ARGS__); \
+})
+
+#define cmpxchg128_relaxed(ptr, ...) \
+({ \
+ typeof(ptr) __ai_ptr = (ptr); \
+ instrument_atomic_read_write(__ai_ptr, sizeof(*__ai_ptr)); \
+ arch_cmpxchg128_relaxed(__ai_ptr, __VA_ARGS__); \
+})
+
#define try_cmpxchg(ptr, oldp, ...) \
({ \
typeof(ptr) __ai_ptr = (ptr); \
@@ -2110,6 +2140,44 @@ atomic_long_dec_if_positive(atomic_long_t *v)
arch_try_cmpxchg64_relaxed(__ai_ptr, __ai_oldp, __VA_ARGS__); \
})
+#define try_cmpxchg128(ptr, oldp, ...) \
+({ \
+ typeof(ptr) __ai_ptr = (ptr); \
+ typeof(oldp) __ai_oldp = (oldp); \
+ kcsan_mb(); \
+ instrument_atomic_read_write(__ai_ptr, sizeof(*__ai_ptr)); \
+ instrument_read_write(__ai_oldp, sizeof(*__ai_oldp)); \
+ arch_try_cmpxchg128(__ai_ptr, __ai_oldp, __VA_ARGS__); \
+})
+
+#define try_cmpxchg128_acquire(ptr, oldp, ...) \
+({ \
+ typeof(ptr) __ai_ptr = (ptr); \
+ typeof(oldp) __ai_oldp = (oldp); \
+ instrument_atomic_read_write(__ai_ptr, sizeof(*__ai_ptr)); \
+ instrument_read_write(__ai_oldp, sizeof(*__ai_oldp)); \
+ arch_try_cmpxchg128_acquire(__ai_ptr, __ai_oldp, __VA_ARGS__); \
+})
+
+#define try_cmpxchg128_release(ptr, oldp, ...) \
+({ \
+ typeof(ptr) __ai_ptr = (ptr); \
+ typeof(oldp) __ai_oldp = (oldp); \
+ kcsan_release(); \
+ instrument_atomic_read_write(__ai_ptr, sizeof(*__ai_ptr)); \
+ instrument_read_write(__ai_oldp, sizeof(*__ai_oldp)); \
+ arch_try_cmpxchg128_release(__ai_ptr, __ai_oldp, __VA_ARGS__); \
+})
+
+#define try_cmpxchg128_relaxed(ptr, oldp, ...) \
+({ \
+ typeof(ptr) __ai_ptr = (ptr); \
+ typeof(oldp) __ai_oldp = (oldp); \
+ instrument_atomic_read_write(__ai_ptr, sizeof(*__ai_ptr)); \
+ instrument_read_write(__ai_oldp, sizeof(*__ai_oldp)); \
+ arch_try_cmpxchg128_relaxed(__ai_ptr, __ai_oldp, __VA_ARGS__); \
+})
+
#define cmpxchg_local(ptr, ...) \
({ \
typeof(ptr) __ai_ptr = (ptr); \
@@ -2124,6 +2192,13 @@ atomic_long_dec_if_positive(atomic_long_t *v)
arch_cmpxchg64_local(__ai_ptr, __VA_ARGS__); \
})
+#define cmpxchg128_local(ptr, ...) \
+({ \
+ typeof(ptr) __ai_ptr = (ptr); \
+ instrument_atomic_read_write(__ai_ptr, sizeof(*__ai_ptr)); \
+ arch_cmpxchg128_local(__ai_ptr, __VA_ARGS__); \
+})
+
#define sync_cmpxchg(ptr, ...) \
({ \
typeof(ptr) __ai_ptr = (ptr); \
@@ -2150,6 +2225,15 @@ atomic_long_dec_if_positive(atomic_long_t *v)
arch_try_cmpxchg64_local(__ai_ptr, __ai_oldp, __VA_ARGS__); \
})
+#define try_cmpxchg128_local(ptr, oldp, ...) \
+({ \
+ typeof(ptr) __ai_ptr = (ptr); \
+ typeof(oldp) __ai_oldp = (oldp); \
+ instrument_atomic_read_write(__ai_ptr, sizeof(*__ai_ptr)); \
+ instrument_read_write(__ai_oldp, sizeof(*__ai_oldp)); \
+ arch_try_cmpxchg128_local(__ai_ptr, __ai_oldp, __VA_ARGS__); \
+})
+
#define cmpxchg_double(ptr, ...) \
({ \
typeof(ptr) __ai_ptr = (ptr); \
@@ -2167,4 +2251,4 @@ atomic_long_dec_if_positive(atomic_long_t *v)
})
#endif /* _LINUX_ATOMIC_INSTRUMENTED_H */
-// 6b513a42e1a1b5962532a019b7fc91eaa044ad5e
+// 82d1be694fab30414527d0877c29fa75ed5a0b74
diff --git a/scripts/atomic/gen-atomic-fallback.sh b/scripts/atomic/gen-atomic-fallback.sh
index 6e853f0..a70acd5 100755
--- a/scripts/atomic/gen-atomic-fallback.sh
+++ b/scripts/atomic/gen-atomic-fallback.sh
@@ -217,11 +217,11 @@ cat << EOF
EOF
-for xchg in "arch_xchg" "arch_cmpxchg" "arch_cmpxchg64"; do
+for xchg in "arch_xchg" "arch_cmpxchg" "arch_cmpxchg64" "arch_cmpxchg128"; do
gen_xchg_fallbacks "${xchg}"
done
-for cmpxchg in "cmpxchg" "cmpxchg64"; do
+for cmpxchg in "cmpxchg" "cmpxchg64" "cmpxchg128"; do
gen_try_cmpxchg_fallbacks "${cmpxchg}"
done
diff --git a/scripts/atomic/gen-atomic-instrumented.sh b/scripts/atomic/gen-atomic-instrumented.sh
index d9ffd74..4f80347 100755
--- a/scripts/atomic/gen-atomic-instrumented.sh
+++ b/scripts/atomic/gen-atomic-instrumented.sh
@@ -166,14 +166,14 @@ grep '^[a-z]' "$1" | while read name meta args; do
done
-for xchg in "xchg" "cmpxchg" "cmpxchg64" "try_cmpxchg" "try_cmpxchg64"; do
+for xchg in "xchg" "cmpxchg" "cmpxchg64" "cmpxchg128" "try_cmpxchg" "try_cmpxchg64" "try_cmpxchg128"; do
for order in "" "_acquire" "_release" "_relaxed"; do
gen_xchg "${xchg}" "${order}" ""
printf "\n"
done
done
-for xchg in "cmpxchg_local" "cmpxchg64_local" "sync_cmpxchg" "try_cmpxchg_local" "try_cmpxchg64_local" ; do
+for xchg in "cmpxchg_local" "cmpxchg64_local" "cmpxchg128_local" "sync_cmpxchg" "try_cmpxchg_local" "try_cmpxchg64_local" "try_cmpxchg128_local"; do
gen_xchg "${xchg}" "" ""
printf "\n"
done
^ permalink raw reply [flat|nested] 47+ messages in thread
* [PATCH 05/12] percpu: Add {raw,this}_cpu_try_cmpxchg()
2023-05-31 13:08 [PATCH 00/12] Introduce cmpxchg128() -- aka. the demise of cmpxchg_double() Peter Zijlstra
` (3 preceding siblings ...)
2023-05-31 13:08 ` [PATCH 04/12] instrumentation: Wire up cmpxchg128() Peter Zijlstra
@ 2023-05-31 13:08 ` Peter Zijlstra
2023-06-05 7:42 ` [tip: locking/core] " tip-bot2 for Peter Zijlstra
2023-06-09 16:10 ` [PATCH v2 5/12] " Konrad Dybcio
2023-05-31 13:08 ` [PATCH 06/12] percpu: Wire up cmpxchg128 Peter Zijlstra
` (7 subsequent siblings)
12 siblings, 2 replies; 47+ messages in thread
From: Peter Zijlstra @ 2023-05-31 13:08 UTC (permalink / raw)
To: torvalds
Cc: corbet, will, peterz, boqun.feng, mark.rutland, catalin.marinas,
dennis, tj, cl, hca, gor, agordeev, borntraeger, svens, tglx,
mingo, bp, dave.hansen, x86, hpa, joro, suravee.suthikulpanit,
robin.murphy, dwmw2, baolu.lu, Arnd Bergmann, Herbert Xu, davem,
penberg, rientjes, iamjoonsoo.kim, Andrew Morton, vbabka,
roman.gushchin, 42.hyeyoo, linux-doc, linux-kernel, linux-mm,
linux-s390, iommu, linux-arch, linux-crypto, sfr, mpe,
James.Bottomley, deller, linux-parisc
Add the try_cmpxchg() form to the per-cpu ops.
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
---
include/asm-generic/percpu.h | 113 +++++++++++++++++++++++++++++++++++++++++--
include/linux/percpu-defs.h | 19 +++++++
2 files changed, 128 insertions(+), 4 deletions(-)
--- a/include/asm-generic/percpu.h
+++ b/include/asm-generic/percpu.h
@@ -89,16 +89,37 @@ do { \
__ret; \
})
-#define raw_cpu_generic_cmpxchg(pcp, oval, nval) \
+#define __cpu_fallback_try_cmpxchg(pcp, ovalp, nval, _cmpxchg) \
+({ \
+ typeof(pcp) __val, __old = *(ovalp); \
+ __val = _cmpxchg(pcp, __old, nval); \
+ if (__val != __old) \
+ *(ovalp) = __val; \
+ __val == __old; \
+})
+
+#define raw_cpu_generic_try_cmpxchg(pcp, ovalp, nval) \
({ \
typeof(pcp) *__p = raw_cpu_ptr(&(pcp)); \
- typeof(pcp) __ret; \
- __ret = *__p; \
- if (__ret == (oval)) \
+ typeof(pcp) __val = *__p, __old = *(ovalp); \
+ bool __ret; \
+ if (__val == __old) { \
*__p = nval; \
+ __ret = true; \
+ } else { \
+ *(ovalp) = __val; \
+ __ret = false; \
+ } \
__ret; \
})
+#define raw_cpu_generic_cmpxchg(pcp, oval, nval) \
+({ \
+ typeof(pcp) __old = (oval); \
+ raw_cpu_generic_try_cmpxchg(pcp, &__old, nval); \
+ __old; \
+})
+
#define raw_cpu_generic_cmpxchg_double(pcp1, pcp2, oval1, oval2, nval1, nval2) \
({ \
typeof(pcp1) *__p1 = raw_cpu_ptr(&(pcp1)); \
@@ -170,6 +191,16 @@ do { \
__ret; \
})
+#define this_cpu_generic_try_cmpxchg(pcp, ovalp, nval) \
+({ \
+ bool __ret; \
+ unsigned long __flags; \
+ raw_local_irq_save(__flags); \
+ __ret = raw_cpu_generic_try_cmpxchg(pcp, ovalp, nval); \
+ raw_local_irq_restore(__flags); \
+ __ret; \
+})
+
#define this_cpu_generic_cmpxchg(pcp, oval, nval) \
({ \
typeof(pcp) __ret; \
@@ -282,6 +313,43 @@ do { \
#define raw_cpu_xchg_8(pcp, nval) raw_cpu_generic_xchg(pcp, nval)
#endif
+#ifndef raw_cpu_try_cmpxchg_1
+#ifdef raw_cpu_cmpxchg_1
+#define raw_cpu_try_cmpxchg_1(pcp, ovalp, nval) \
+ __cpu_fallback_try_cmpxchg(pcp, ovalp, nval, raw_cpu_cmpxchg_1)
+#else
+#define raw_cpu_try_cmpxchg_1(pcp, ovalp, nval) \
+ raw_cpu_generic_try_cmpxchg(pcp, ovalp, nval)
+#endif
+#endif
+#ifndef raw_cpu_try_cmpxchg_2
+#ifdef raw_cpu_cmpxchg_2
+#define raw_cpu_try_cmpxchg_2(pcp, ovalp, nval) \
+ __cpu_fallback_try_cmpxchg(pcp, ovalp, nval, raw_cpu_cmpxchg_2)
+#else
+#define raw_cpu_try_cmpxchg_2(pcp, ovalp, nval) \
+ raw_cpu_generic_try_cmpxchg(pcp, ovalp, nval)
+#endif
+#endif
+#ifndef raw_cpu_try_cmpxchg_4
+#ifdef raw_cpu_cmpxchg_4
+#define raw_cpu_try_cmpxchg_4(pcp, ovalp, nval) \
+ __cpu_fallback_try_cmpxchg(pcp, ovalp, nval, raw_cpu_cmpxchg_4)
+#else
+#define raw_cpu_try_cmpxchg_4(pcp, ovalp, nval) \
+ raw_cpu_generic_try_cmpxchg(pcp, ovalp, nval)
+#endif
+#endif
+#ifndef raw_cpu_try_cmpxchg_8
+#ifdef raw_cpu_cmpxchg_8
+#define raw_cpu_try_cmpxchg_8(pcp, ovalp, nval) \
+ __cpu_fallback_try_cmpxchg(pcp, ovalp, nval, raw_cpu_cmpxchg_8)
+#else
+#define raw_cpu_try_cmpxchg_8(pcp, ovalp, nval) \
+ raw_cpu_generic_try_cmpxchg(pcp, ovalp, nval)
+#endif
+#endif
+
#ifndef raw_cpu_cmpxchg_1
#define raw_cpu_cmpxchg_1(pcp, oval, nval) \
raw_cpu_generic_cmpxchg(pcp, oval, nval)
@@ -407,6 +475,43 @@ do { \
#define this_cpu_xchg_8(pcp, nval) this_cpu_generic_xchg(pcp, nval)
#endif
+#ifndef this_cpu_try_cmpxchg_1
+#ifdef this_cpu_cmpxchg_1
+#define this_cpu_try_cmpxchg_1(pcp, ovalp, nval) \
+ __cpu_fallback_try_cmpxchg(pcp, ovalp, nval, this_cpu_cmpxchg_1)
+#else
+#define this_cpu_try_cmpxchg_1(pcp, ovalp, nval) \
+ this_cpu_generic_try_cmpxchg(pcp, ovalp, nval)
+#endif
+#endif
+#ifndef this_cpu_try_cmpxchg_2
+#ifdef this_cpu_cmpxchg_2
+#define this_cpu_try_cmpxchg_2(pcp, ovalp, nval) \
+ __cpu_fallback_try_cmpxchg(pcp, ovalp, nval, this_cpu_cmpxchg_2)
+#else
+#define this_cpu_try_cmpxchg_2(pcp, ovalp, nval) \
+ this_cpu_generic_try_cmpxchg(pcp, ovalp, nval)
+#endif
+#endif
+#ifndef this_cpu_try_cmpxchg_4
+#ifdef this_cpu_cmpxchg_4
+#define this_cpu_try_cmpxchg_4(pcp, ovalp, nval) \
+ __cpu_fallback_try_cmpxchg(pcp, ovalp, nval, this_cpu_cmpxchg_4)
+#else
+#define this_cpu_try_cmpxchg_4(pcp, ovalp, nval) \
+ this_cpu_generic_try_cmpxchg(pcp, ovalp, nval)
+#endif
+#endif
+#ifndef this_cpu_try_cmpxchg_8
+#ifdef this_cpu_cmpxchg_8
+#define this_cpu_try_cmpxchg_8(pcp, ovalp, nval) \
+ __cpu_fallback_try_cmpxchg(pcp, ovalp, nval, this_cpu_cmpxchg_8)
+#else
+#define this_cpu_try_cmpxchg_8(pcp, ovalp, nval) \
+ this_cpu_generic_try_cmpxchg(pcp, ovalp, nval)
+#endif
+#endif
+
#ifndef this_cpu_cmpxchg_1
#define this_cpu_cmpxchg_1(pcp, oval, nval) \
this_cpu_generic_cmpxchg(pcp, oval, nval)
--- a/include/linux/percpu-defs.h
+++ b/include/linux/percpu-defs.h
@@ -343,6 +343,21 @@ static __always_inline void __this_cpu_p
pscr2_ret__; \
})
+#define __pcpu_size_call_return2bool(stem, variable, ...) \
+({ \
+ bool pscr2_ret__; \
+ __verify_pcpu_ptr(&(variable)); \
+ switch(sizeof(variable)) { \
+ case 1: pscr2_ret__ = stem##1(variable, __VA_ARGS__); break; \
+ case 2: pscr2_ret__ = stem##2(variable, __VA_ARGS__); break; \
+ case 4: pscr2_ret__ = stem##4(variable, __VA_ARGS__); break; \
+ case 8: pscr2_ret__ = stem##8(variable, __VA_ARGS__); break; \
+ default: \
+ __bad_size_call_parameter(); break; \
+ } \
+ pscr2_ret__; \
+})
+
/*
* Special handling for cmpxchg_double. cmpxchg_double is passed two
* percpu variables. The first has to be aligned to a double word
@@ -426,6 +441,8 @@ do { \
#define raw_cpu_xchg(pcp, nval) __pcpu_size_call_return2(raw_cpu_xchg_, pcp, nval)
#define raw_cpu_cmpxchg(pcp, oval, nval) \
__pcpu_size_call_return2(raw_cpu_cmpxchg_, pcp, oval, nval)
+#define raw_cpu_try_cmpxchg(pcp, ovalp, nval) \
+ __pcpu_size_call_return2bool(raw_cpu_try_cmpxchg_, pcp, ovalp, nval)
#define raw_cpu_cmpxchg_double(pcp1, pcp2, oval1, oval2, nval1, nval2) \
__pcpu_double_call_return_bool(raw_cpu_cmpxchg_double_, pcp1, pcp2, oval1, oval2, nval1, nval2)
@@ -513,6 +530,8 @@ do { \
#define this_cpu_xchg(pcp, nval) __pcpu_size_call_return2(this_cpu_xchg_, pcp, nval)
#define this_cpu_cmpxchg(pcp, oval, nval) \
__pcpu_size_call_return2(this_cpu_cmpxchg_, pcp, oval, nval)
+#define this_cpu_try_cmpxchg(pcp, ovalp, nval) \
+ __pcpu_size_call_return2bool(this_cpu_try_cmpxchg_, pcp, ovalp, nval)
#define this_cpu_cmpxchg_double(pcp1, pcp2, oval1, oval2, nval1, nval2) \
__pcpu_double_call_return_bool(this_cpu_cmpxchg_double_, pcp1, pcp2, oval1, oval2, nval1, nval2)
^ permalink raw reply [flat|nested] 47+ messages in thread* [tip: locking/core] percpu: Add {raw,this}_cpu_try_cmpxchg()
2023-05-31 13:08 ` [PATCH 05/12] percpu: Add {raw,this}_cpu_try_cmpxchg() Peter Zijlstra
@ 2023-06-05 7:42 ` tip-bot2 for Peter Zijlstra
2023-06-09 16:10 ` [PATCH v2 5/12] " Konrad Dybcio
1 sibling, 0 replies; 47+ messages in thread
From: tip-bot2 for Peter Zijlstra @ 2023-06-05 7:42 UTC (permalink / raw)
To: linux-tip-commits; +Cc: Peter Zijlstra (Intel), Mark Rutland, x86, linux-kernel
The following commit has been merged into the locking/core branch of tip:
Commit-ID: c5c0ba953b8c969c5d51bf1c57f239866a97c47c
Gitweb: https://git.kernel.org/tip/c5c0ba953b8c969c5d51bf1c57f239866a97c47c
Author: Peter Zijlstra <peterz@infradead.org>
AuthorDate: Wed, 31 May 2023 15:08:38 +02:00
Committer: Peter Zijlstra <peterz@infradead.org>
CommitterDate: Mon, 05 Jun 2023 09:36:36 +02:00
percpu: Add {raw,this}_cpu_try_cmpxchg()
Add the try_cmpxchg() form to the per-cpu ops.
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Reviewed-by: Mark Rutland <mark.rutland@arm.com>
Tested-by: Mark Rutland <mark.rutland@arm.com>
Link: https://lore.kernel.org/r/20230531132323.587480729@infradead.org
---
include/asm-generic/percpu.h | 113 ++++++++++++++++++++++++++++++++--
include/linux/percpu-defs.h | 19 ++++++-
2 files changed, 128 insertions(+), 4 deletions(-)
diff --git a/include/asm-generic/percpu.h b/include/asm-generic/percpu.h
index 6432a7f..96af32c 100644
--- a/include/asm-generic/percpu.h
+++ b/include/asm-generic/percpu.h
@@ -89,16 +89,37 @@ do { \
__ret; \
})
-#define raw_cpu_generic_cmpxchg(pcp, oval, nval) \
+#define __cpu_fallback_try_cmpxchg(pcp, ovalp, nval, _cmpxchg) \
+({ \
+ typeof(pcp) __val, __old = *(ovalp); \
+ __val = _cmpxchg(pcp, __old, nval); \
+ if (__val != __old) \
+ *(ovalp) = __val; \
+ __val == __old; \
+})
+
+#define raw_cpu_generic_try_cmpxchg(pcp, ovalp, nval) \
({ \
typeof(pcp) *__p = raw_cpu_ptr(&(pcp)); \
- typeof(pcp) __ret; \
- __ret = *__p; \
- if (__ret == (oval)) \
+ typeof(pcp) __val = *__p, __old = *(ovalp); \
+ bool __ret; \
+ if (__val == __old) { \
*__p = nval; \
+ __ret = true; \
+ } else { \
+ *(ovalp) = __val; \
+ __ret = false; \
+ } \
__ret; \
})
+#define raw_cpu_generic_cmpxchg(pcp, oval, nval) \
+({ \
+ typeof(pcp) __old = (oval); \
+ raw_cpu_generic_try_cmpxchg(pcp, &__old, nval); \
+ __old; \
+})
+
#define raw_cpu_generic_cmpxchg_double(pcp1, pcp2, oval1, oval2, nval1, nval2) \
({ \
typeof(pcp1) *__p1 = raw_cpu_ptr(&(pcp1)); \
@@ -170,6 +191,16 @@ do { \
__ret; \
})
+#define this_cpu_generic_try_cmpxchg(pcp, ovalp, nval) \
+({ \
+ bool __ret; \
+ unsigned long __flags; \
+ raw_local_irq_save(__flags); \
+ __ret = raw_cpu_generic_try_cmpxchg(pcp, ovalp, nval); \
+ raw_local_irq_restore(__flags); \
+ __ret; \
+})
+
#define this_cpu_generic_cmpxchg(pcp, oval, nval) \
({ \
typeof(pcp) __ret; \
@@ -282,6 +313,43 @@ do { \
#define raw_cpu_xchg_8(pcp, nval) raw_cpu_generic_xchg(pcp, nval)
#endif
+#ifndef raw_cpu_try_cmpxchg_1
+#ifdef raw_cpu_cmpxchg_1
+#define raw_cpu_try_cmpxchg_1(pcp, ovalp, nval) \
+ __cpu_fallback_try_cmpxchg(pcp, ovalp, nval, raw_cpu_cmpxchg_1)
+#else
+#define raw_cpu_try_cmpxchg_1(pcp, ovalp, nval) \
+ raw_cpu_generic_try_cmpxchg(pcp, ovalp, nval)
+#endif
+#endif
+#ifndef raw_cpu_try_cmpxchg_2
+#ifdef raw_cpu_cmpxchg_2
+#define raw_cpu_try_cmpxchg_2(pcp, ovalp, nval) \
+ __cpu_fallback_try_cmpxchg(pcp, ovalp, nval, raw_cpu_cmpxchg_2)
+#else
+#define raw_cpu_try_cmpxchg_2(pcp, ovalp, nval) \
+ raw_cpu_generic_try_cmpxchg(pcp, ovalp, nval)
+#endif
+#endif
+#ifndef raw_cpu_try_cmpxchg_4
+#ifdef raw_cpu_cmpxchg_4
+#define raw_cpu_try_cmpxchg_4(pcp, ovalp, nval) \
+ __cpu_fallback_try_cmpxchg(pcp, ovalp, nval, raw_cpu_cmpxchg_4)
+#else
+#define raw_cpu_try_cmpxchg_4(pcp, ovalp, nval) \
+ raw_cpu_generic_try_cmpxchg(pcp, ovalp, nval)
+#endif
+#endif
+#ifndef raw_cpu_try_cmpxchg_8
+#ifdef raw_cpu_cmpxchg_8
+#define raw_cpu_try_cmpxchg_8(pcp, ovalp, nval) \
+ __cpu_fallback_try_cmpxchg(pcp, ovalp, nval, raw_cpu_cmpxchg_8)
+#else
+#define raw_cpu_try_cmpxchg_8(pcp, ovalp, nval) \
+ raw_cpu_generic_try_cmpxchg(pcp, ovalp, nval)
+#endif
+#endif
+
#ifndef raw_cpu_cmpxchg_1
#define raw_cpu_cmpxchg_1(pcp, oval, nval) \
raw_cpu_generic_cmpxchg(pcp, oval, nval)
@@ -407,6 +475,43 @@ do { \
#define this_cpu_xchg_8(pcp, nval) this_cpu_generic_xchg(pcp, nval)
#endif
+#ifndef this_cpu_try_cmpxchg_1
+#ifdef this_cpu_cmpxchg_1
+#define this_cpu_try_cmpxchg_1(pcp, ovalp, nval) \
+ __cpu_fallback_try_cmpxchg(pcp, ovalp, nval, this_cpu_cmpxchg_1)
+#else
+#define this_cpu_try_cmpxchg_1(pcp, ovalp, nval) \
+ this_cpu_generic_try_cmpxchg(pcp, ovalp, nval)
+#endif
+#endif
+#ifndef this_cpu_try_cmpxchg_2
+#ifdef this_cpu_cmpxchg_2
+#define this_cpu_try_cmpxchg_2(pcp, ovalp, nval) \
+ __cpu_fallback_try_cmpxchg(pcp, ovalp, nval, this_cpu_cmpxchg_2)
+#else
+#define this_cpu_try_cmpxchg_2(pcp, ovalp, nval) \
+ this_cpu_generic_try_cmpxchg(pcp, ovalp, nval)
+#endif
+#endif
+#ifndef this_cpu_try_cmpxchg_4
+#ifdef this_cpu_cmpxchg_4
+#define this_cpu_try_cmpxchg_4(pcp, ovalp, nval) \
+ __cpu_fallback_try_cmpxchg(pcp, ovalp, nval, this_cpu_cmpxchg_4)
+#else
+#define this_cpu_try_cmpxchg_4(pcp, ovalp, nval) \
+ this_cpu_generic_try_cmpxchg(pcp, ovalp, nval)
+#endif
+#endif
+#ifndef this_cpu_try_cmpxchg_8
+#ifdef this_cpu_cmpxchg_8
+#define this_cpu_try_cmpxchg_8(pcp, ovalp, nval) \
+ __cpu_fallback_try_cmpxchg(pcp, ovalp, nval, this_cpu_cmpxchg_8)
+#else
+#define this_cpu_try_cmpxchg_8(pcp, ovalp, nval) \
+ this_cpu_generic_try_cmpxchg(pcp, ovalp, nval)
+#endif
+#endif
+
#ifndef this_cpu_cmpxchg_1
#define this_cpu_cmpxchg_1(pcp, oval, nval) \
this_cpu_generic_cmpxchg(pcp, oval, nval)
diff --git a/include/linux/percpu-defs.h b/include/linux/percpu-defs.h
index e60727b..cbbf6d1 100644
--- a/include/linux/percpu-defs.h
+++ b/include/linux/percpu-defs.h
@@ -343,6 +343,21 @@ static __always_inline void __this_cpu_preempt_check(const char *op) { }
pscr2_ret__; \
})
+#define __pcpu_size_call_return2bool(stem, variable, ...) \
+({ \
+ bool pscr2_ret__; \
+ __verify_pcpu_ptr(&(variable)); \
+ switch(sizeof(variable)) { \
+ case 1: pscr2_ret__ = stem##1(variable, __VA_ARGS__); break; \
+ case 2: pscr2_ret__ = stem##2(variable, __VA_ARGS__); break; \
+ case 4: pscr2_ret__ = stem##4(variable, __VA_ARGS__); break; \
+ case 8: pscr2_ret__ = stem##8(variable, __VA_ARGS__); break; \
+ default: \
+ __bad_size_call_parameter(); break; \
+ } \
+ pscr2_ret__; \
+})
+
/*
* Special handling for cmpxchg_double. cmpxchg_double is passed two
* percpu variables. The first has to be aligned to a double word
@@ -426,6 +441,8 @@ do { \
#define raw_cpu_xchg(pcp, nval) __pcpu_size_call_return2(raw_cpu_xchg_, pcp, nval)
#define raw_cpu_cmpxchg(pcp, oval, nval) \
__pcpu_size_call_return2(raw_cpu_cmpxchg_, pcp, oval, nval)
+#define raw_cpu_try_cmpxchg(pcp, ovalp, nval) \
+ __pcpu_size_call_return2bool(raw_cpu_try_cmpxchg_, pcp, ovalp, nval)
#define raw_cpu_cmpxchg_double(pcp1, pcp2, oval1, oval2, nval1, nval2) \
__pcpu_double_call_return_bool(raw_cpu_cmpxchg_double_, pcp1, pcp2, oval1, oval2, nval1, nval2)
@@ -513,6 +530,8 @@ do { \
#define this_cpu_xchg(pcp, nval) __pcpu_size_call_return2(this_cpu_xchg_, pcp, nval)
#define this_cpu_cmpxchg(pcp, oval, nval) \
__pcpu_size_call_return2(this_cpu_cmpxchg_, pcp, oval, nval)
+#define this_cpu_try_cmpxchg(pcp, ovalp, nval) \
+ __pcpu_size_call_return2bool(this_cpu_try_cmpxchg_, pcp, ovalp, nval)
#define this_cpu_cmpxchg_double(pcp1, pcp2, oval1, oval2, nval1, nval2) \
__pcpu_double_call_return_bool(this_cpu_cmpxchg_double_, pcp1, pcp2, oval1, oval2, nval1, nval2)
^ permalink raw reply [flat|nested] 47+ messages in thread* Re: [PATCH v2 5/12] percpu: Add {raw,this}_cpu_try_cmpxchg()
2023-05-31 13:08 ` [PATCH 05/12] percpu: Add {raw,this}_cpu_try_cmpxchg() Peter Zijlstra
2023-06-05 7:42 ` [tip: locking/core] " tip-bot2 for Peter Zijlstra
@ 2023-06-09 16:10 ` Konrad Dybcio
2023-06-09 16:13 ` Nathan Chancellor
1 sibling, 1 reply; 47+ messages in thread
From: Konrad Dybcio @ 2023-06-09 16:10 UTC (permalink / raw)
To: Peter Zijlstra
Cc: corbet, will, peterz, boqun.feng, mark.rutland, catalin.marinas,
dennis, tj, cl, hca, gor, agordeev, borntraeger, svens, tglx,
mingo, bp, dave.hansen, x86, hpa, joro, suravee.suthikulpanit,
robin.murphy, dwmw2, baolu.lu, Arnd Bergmann, Herbert Xu, davem,
penberg, rientjes, iamjoonsoo.kim, Andrew Morton, vbabka,
roman.gushchin, 42.hyeyoo, linux-doc, linux-kernel, linux-mm,
linux-s390, iommu, linux-arch, linux-crypto, sfr, mpe,
James.Bottomley, deller, linux-parisc, Nathan Chancellor, llvm
On 31.05.2023 15:08, Peter Zijlstra wrote:
> Add the try_cmpxchg() form to the per-cpu ops.
>
> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
> ---
+CC Nathan, llvm list
Hi all, this patch seems to break booting on Qualcomm ARM64 platforms
when compiled with clang (GCC works fine) for some reason..:
next-20230605 - works
next-20230606 - doesn't
grev -m 1 dc4e51fd9846 on next-20230606 - works again
b4 shazam <this_msgid> -P 1-4 - still works
b4 shazam <this_msgid> -P 5 - breaks
Confirmed on at least Qualcomm QCM2290, SM8250.
Checking the serial console, it hits a BUG_ON:
[ 0.000000] ------------[ cut here ]------------
[ 0.000000] kernel BUG at mm/vmalloc.c:1638!
[ 0.000000] Internal error: Oops - BUG: 00000000f2000800 [#1] SMP
[ 0.000000] Modules linked in:
[ 0.000000] CPU: 0 PID: 0 Comm: swapper/0 Not tainted [snip]
[ 0.000000] Hardware name: Qualcomm Technologies, Inc. Robotics RB1 (DT)
[ 0.000000] pstate: 000000c5 (nzcv daIF -PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[ 0.000000] pc : alloc_vmap_area+0xafc/0xb08
[ 0.000000] lr : alloc_vmap_area+0x9e4/0xb08
[ 0.000000] sp : ffffa50137f53c20
[ 0.000000] x29: ffffa50137f53c60 x28: ffffa50137f30c18 x27: 0000000000000000
[ 0.000000] x26: 0000000000007fff x25: ffff800080000000 x24: 000000000000cfff
[ 0.000000] x23: ffffffffffff8000 x22: ffffa50137fef970 x21: fffffbfff0000000
[ 0.000000] x20: ffff022982003208 x19: ffff0229820031f8 x18: ffffa50137f64f70
[ 0.000000] x17: ffffa50137fef980 x16: ffffa501375e6d08 x15: 0000000000000001
[ 0.000000] x14: ffffa5013831e1a0 x13: ffffa50137f30c18 x12: 0000000000402dc2
[ 0.000000] x11: 0000000000000000 x10: ffff022982003018 x9 : ffffa5013831e188
[ 0.000000] x8 : ffffcb55ff003228 x7 : 0000000000000000 x6 : 0000000000000048
[ 0.000000] x5 : 0000000000000000 x4 : ffffa50137f53bd0 x3 : ffffa50136490000
[ 0.000000] x2 : 0000000000000001 x1 : ffffa5013831e190 x0 : ffff022982003208
[ 0.000000] Call trace:
[ 0.000000] alloc_vmap_area+0xafc/0xb08
[ 0.000000] __get_vm_area_node+0x108/0x1e8
[ 0.000000] __vmalloc_node_range+0x1fc/0x728
[ 0.000000] __vmalloc_node+0x5c/0x70
[ 0.000000] init_IRQ+0x90/0x11c
[ 0.000000] start_kernel+0x1ac/0x3bc
[ 0.000000] __primary_switched+0xc4/0xcc
[ 0.000000] Code: f000e300 91062000 943bd9ba 17ffff8f (d4210000)
[ 0.000000] ---[ end trace 0000000000000000 ]---
[ 0.000000] Kernel panic - not syncing: Attempted to kill the idle task!
Compiled with clang 15.0.7 from Arch repos, with
make ARCH=arm64 LLVM=1
Konrad
> include/asm-generic/percpu.h | 113 +++++++++++++++++++++++++++++++++++++++++--
> include/linux/percpu-defs.h | 19 +++++++
> 2 files changed, 128 insertions(+), 4 deletions(-)
>
> --- a/include/asm-generic/percpu.h
> +++ b/include/asm-generic/percpu.h
> @@ -89,16 +89,37 @@ do { \
> __ret; \
> })
>
> -#define raw_cpu_generic_cmpxchg(pcp, oval, nval) \
> +#define __cpu_fallback_try_cmpxchg(pcp, ovalp, nval, _cmpxchg) \
> +({ \
> + typeof(pcp) __val, __old = *(ovalp); \
> + __val = _cmpxchg(pcp, __old, nval); \
> + if (__val != __old) \
> + *(ovalp) = __val; \
> + __val == __old; \
> +})
> +
> +#define raw_cpu_generic_try_cmpxchg(pcp, ovalp, nval) \
> ({ \
> typeof(pcp) *__p = raw_cpu_ptr(&(pcp)); \
> - typeof(pcp) __ret; \
> - __ret = *__p; \
> - if (__ret == (oval)) \
> + typeof(pcp) __val = *__p, __old = *(ovalp); \
> + bool __ret; \
> + if (__val == __old) { \
> *__p = nval; \
> + __ret = true; \
> + } else { \
> + *(ovalp) = __val; \
> + __ret = false; \
> + } \
> __ret; \
> })
>
> +#define raw_cpu_generic_cmpxchg(pcp, oval, nval) \
> +({ \
> + typeof(pcp) __old = (oval); \
> + raw_cpu_generic_try_cmpxchg(pcp, &__old, nval); \
> + __old; \
> +})
> +
> #define raw_cpu_generic_cmpxchg_double(pcp1, pcp2, oval1, oval2, nval1, nval2) \
> ({ \
> typeof(pcp1) *__p1 = raw_cpu_ptr(&(pcp1)); \
> @@ -170,6 +191,16 @@ do { \
> __ret; \
> })
>
> +#define this_cpu_generic_try_cmpxchg(pcp, ovalp, nval) \
> +({ \
> + bool __ret; \
> + unsigned long __flags; \
> + raw_local_irq_save(__flags); \
> + __ret = raw_cpu_generic_try_cmpxchg(pcp, ovalp, nval); \
> + raw_local_irq_restore(__flags); \
> + __ret; \
> +})
> +
> #define this_cpu_generic_cmpxchg(pcp, oval, nval) \
> ({ \
> typeof(pcp) __ret; \
> @@ -282,6 +313,43 @@ do { \
> #define raw_cpu_xchg_8(pcp, nval) raw_cpu_generic_xchg(pcp, nval)
> #endif
>
> +#ifndef raw_cpu_try_cmpxchg_1
> +#ifdef raw_cpu_cmpxchg_1
> +#define raw_cpu_try_cmpxchg_1(pcp, ovalp, nval) \
> + __cpu_fallback_try_cmpxchg(pcp, ovalp, nval, raw_cpu_cmpxchg_1)
> +#else
> +#define raw_cpu_try_cmpxchg_1(pcp, ovalp, nval) \
> + raw_cpu_generic_try_cmpxchg(pcp, ovalp, nval)
> +#endif
> +#endif
> +#ifndef raw_cpu_try_cmpxchg_2
> +#ifdef raw_cpu_cmpxchg_2
> +#define raw_cpu_try_cmpxchg_2(pcp, ovalp, nval) \
> + __cpu_fallback_try_cmpxchg(pcp, ovalp, nval, raw_cpu_cmpxchg_2)
> +#else
> +#define raw_cpu_try_cmpxchg_2(pcp, ovalp, nval) \
> + raw_cpu_generic_try_cmpxchg(pcp, ovalp, nval)
> +#endif
> +#endif
> +#ifndef raw_cpu_try_cmpxchg_4
> +#ifdef raw_cpu_cmpxchg_4
> +#define raw_cpu_try_cmpxchg_4(pcp, ovalp, nval) \
> + __cpu_fallback_try_cmpxchg(pcp, ovalp, nval, raw_cpu_cmpxchg_4)
> +#else
> +#define raw_cpu_try_cmpxchg_4(pcp, ovalp, nval) \
> + raw_cpu_generic_try_cmpxchg(pcp, ovalp, nval)
> +#endif
> +#endif
> +#ifndef raw_cpu_try_cmpxchg_8
> +#ifdef raw_cpu_cmpxchg_8
> +#define raw_cpu_try_cmpxchg_8(pcp, ovalp, nval) \
> + __cpu_fallback_try_cmpxchg(pcp, ovalp, nval, raw_cpu_cmpxchg_8)
> +#else
> +#define raw_cpu_try_cmpxchg_8(pcp, ovalp, nval) \
> + raw_cpu_generic_try_cmpxchg(pcp, ovalp, nval)
> +#endif
> +#endif
> +
> #ifndef raw_cpu_cmpxchg_1
> #define raw_cpu_cmpxchg_1(pcp, oval, nval) \
> raw_cpu_generic_cmpxchg(pcp, oval, nval)
> @@ -407,6 +475,43 @@ do { \
> #define this_cpu_xchg_8(pcp, nval) this_cpu_generic_xchg(pcp, nval)
> #endif
>
> +#ifndef this_cpu_try_cmpxchg_1
> +#ifdef this_cpu_cmpxchg_1
> +#define this_cpu_try_cmpxchg_1(pcp, ovalp, nval) \
> + __cpu_fallback_try_cmpxchg(pcp, ovalp, nval, this_cpu_cmpxchg_1)
> +#else
> +#define this_cpu_try_cmpxchg_1(pcp, ovalp, nval) \
> + this_cpu_generic_try_cmpxchg(pcp, ovalp, nval)
> +#endif
> +#endif
> +#ifndef this_cpu_try_cmpxchg_2
> +#ifdef this_cpu_cmpxchg_2
> +#define this_cpu_try_cmpxchg_2(pcp, ovalp, nval) \
> + __cpu_fallback_try_cmpxchg(pcp, ovalp, nval, this_cpu_cmpxchg_2)
> +#else
> +#define this_cpu_try_cmpxchg_2(pcp, ovalp, nval) \
> + this_cpu_generic_try_cmpxchg(pcp, ovalp, nval)
> +#endif
> +#endif
> +#ifndef this_cpu_try_cmpxchg_4
> +#ifdef this_cpu_cmpxchg_4
> +#define this_cpu_try_cmpxchg_4(pcp, ovalp, nval) \
> + __cpu_fallback_try_cmpxchg(pcp, ovalp, nval, this_cpu_cmpxchg_4)
> +#else
> +#define this_cpu_try_cmpxchg_4(pcp, ovalp, nval) \
> + this_cpu_generic_try_cmpxchg(pcp, ovalp, nval)
> +#endif
> +#endif
> +#ifndef this_cpu_try_cmpxchg_8
> +#ifdef this_cpu_cmpxchg_8
> +#define this_cpu_try_cmpxchg_8(pcp, ovalp, nval) \
> + __cpu_fallback_try_cmpxchg(pcp, ovalp, nval, this_cpu_cmpxchg_8)
> +#else
> +#define this_cpu_try_cmpxchg_8(pcp, ovalp, nval) \
> + this_cpu_generic_try_cmpxchg(pcp, ovalp, nval)
> +#endif
> +#endif
> +
> #ifndef this_cpu_cmpxchg_1
> #define this_cpu_cmpxchg_1(pcp, oval, nval) \
> this_cpu_generic_cmpxchg(pcp, oval, nval)
> --- a/include/linux/percpu-defs.h
> +++ b/include/linux/percpu-defs.h
> @@ -343,6 +343,21 @@ static __always_inline void __this_cpu_p
> pscr2_ret__; \
> })
>
> +#define __pcpu_size_call_return2bool(stem, variable, ...) \
> +({ \
> + bool pscr2_ret__; \
> + __verify_pcpu_ptr(&(variable)); \
> + switch(sizeof(variable)) { \
> + case 1: pscr2_ret__ = stem##1(variable, __VA_ARGS__); break; \
> + case 2: pscr2_ret__ = stem##2(variable, __VA_ARGS__); break; \
> + case 4: pscr2_ret__ = stem##4(variable, __VA_ARGS__); break; \
> + case 8: pscr2_ret__ = stem##8(variable, __VA_ARGS__); break; \
> + default: \
> + __bad_size_call_parameter(); break; \
> + } \
> + pscr2_ret__; \
> +})
> +
> /*
> * Special handling for cmpxchg_double. cmpxchg_double is passed two
> * percpu variables. The first has to be aligned to a double word
> @@ -426,6 +441,8 @@ do { \
> #define raw_cpu_xchg(pcp, nval) __pcpu_size_call_return2(raw_cpu_xchg_, pcp, nval)
> #define raw_cpu_cmpxchg(pcp, oval, nval) \
> __pcpu_size_call_return2(raw_cpu_cmpxchg_, pcp, oval, nval)
> +#define raw_cpu_try_cmpxchg(pcp, ovalp, nval) \
> + __pcpu_size_call_return2bool(raw_cpu_try_cmpxchg_, pcp, ovalp, nval)
> #define raw_cpu_cmpxchg_double(pcp1, pcp2, oval1, oval2, nval1, nval2) \
> __pcpu_double_call_return_bool(raw_cpu_cmpxchg_double_, pcp1, pcp2, oval1, oval2, nval1, nval2)
>
> @@ -513,6 +530,8 @@ do { \
> #define this_cpu_xchg(pcp, nval) __pcpu_size_call_return2(this_cpu_xchg_, pcp, nval)
> #define this_cpu_cmpxchg(pcp, oval, nval) \
> __pcpu_size_call_return2(this_cpu_cmpxchg_, pcp, oval, nval)
> +#define this_cpu_try_cmpxchg(pcp, ovalp, nval) \
> + __pcpu_size_call_return2bool(this_cpu_try_cmpxchg_, pcp, ovalp, nval)
> #define this_cpu_cmpxchg_double(pcp1, pcp2, oval1, oval2, nval1, nval2) \
> __pcpu_double_call_return_bool(this_cpu_cmpxchg_double_, pcp1, pcp2, oval1, oval2, nval1, nval2)
>
^ permalink raw reply [flat|nested] 47+ messages in thread* Re: [PATCH v2 5/12] percpu: Add {raw,this}_cpu_try_cmpxchg()
2023-06-09 16:10 ` [PATCH v2 5/12] " Konrad Dybcio
@ 2023-06-09 16:13 ` Nathan Chancellor
2023-06-09 16:20 ` Konrad Dybcio
0 siblings, 1 reply; 47+ messages in thread
From: Nathan Chancellor @ 2023-06-09 16:13 UTC (permalink / raw)
To: Konrad Dybcio
Cc: Peter Zijlstra, corbet, will, boqun.feng, mark.rutland,
catalin.marinas, dennis, tj, cl, hca, gor, agordeev, borntraeger,
svens, tglx, mingo, bp, dave.hansen, x86, hpa, joro,
suravee.suthikulpanit, robin.murphy, dwmw2, baolu.lu,
Arnd Bergmann, Herbert Xu, davem, penberg, rientjes,
iamjoonsoo.kim, Andrew Morton, vbabka, roman.gushchin, 42.hyeyoo,
linux-doc, linux-kernel, linux-mm, linux-s390, iommu, linux-arch,
linux-crypto, sfr, mpe, James.Bottomley, deller, linux-parisc,
llvm
Hi Konrad,
On Fri, Jun 09, 2023 at 06:10:38PM +0200, Konrad Dybcio wrote:
>
>
> On 31.05.2023 15:08, Peter Zijlstra wrote:
> > Add the try_cmpxchg() form to the per-cpu ops.
> >
> > Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
> > ---
> +CC Nathan, llvm list
>
> Hi all, this patch seems to break booting on Qualcomm ARM64 platforms
> when compiled with clang (GCC works fine) for some reason..:
>
> next-20230605 - works
> next-20230606 - doesn't
>
> grev -m 1 dc4e51fd9846 on next-20230606 - works again
> b4 shazam <this_msgid> -P 1-4 - still works
> b4 shazam <this_msgid> -P 5 - breaks
>
> Confirmed on at least Qualcomm QCM2290, SM8250.
>
> Checking the serial console, it hits a BUG_ON:
>
> [ 0.000000] ------------[ cut here ]------------
> [ 0.000000] kernel BUG at mm/vmalloc.c:1638!
> [ 0.000000] Internal error: Oops - BUG: 00000000f2000800 [#1] SMP
> [ 0.000000] Modules linked in:
> [ 0.000000] CPU: 0 PID: 0 Comm: swapper/0 Not tainted [snip]
> [ 0.000000] Hardware name: Qualcomm Technologies, Inc. Robotics RB1 (DT)
> [ 0.000000] pstate: 000000c5 (nzcv daIF -PAN -UAO -TCO -DIT -SSBS BTYPE=--)
> [ 0.000000] pc : alloc_vmap_area+0xafc/0xb08
> [ 0.000000] lr : alloc_vmap_area+0x9e4/0xb08
> [ 0.000000] sp : ffffa50137f53c20
> [ 0.000000] x29: ffffa50137f53c60 x28: ffffa50137f30c18 x27: 0000000000000000
> [ 0.000000] x26: 0000000000007fff x25: ffff800080000000 x24: 000000000000cfff
> [ 0.000000] x23: ffffffffffff8000 x22: ffffa50137fef970 x21: fffffbfff0000000
> [ 0.000000] x20: ffff022982003208 x19: ffff0229820031f8 x18: ffffa50137f64f70
> [ 0.000000] x17: ffffa50137fef980 x16: ffffa501375e6d08 x15: 0000000000000001
> [ 0.000000] x14: ffffa5013831e1a0 x13: ffffa50137f30c18 x12: 0000000000402dc2
> [ 0.000000] x11: 0000000000000000 x10: ffff022982003018 x9 : ffffa5013831e188
> [ 0.000000] x8 : ffffcb55ff003228 x7 : 0000000000000000 x6 : 0000000000000048
> [ 0.000000] x5 : 0000000000000000 x4 : ffffa50137f53bd0 x3 : ffffa50136490000
> [ 0.000000] x2 : 0000000000000001 x1 : ffffa5013831e190 x0 : ffff022982003208
> [ 0.000000] Call trace:
> [ 0.000000] alloc_vmap_area+0xafc/0xb08
> [ 0.000000] __get_vm_area_node+0x108/0x1e8
> [ 0.000000] __vmalloc_node_range+0x1fc/0x728
> [ 0.000000] __vmalloc_node+0x5c/0x70
> [ 0.000000] init_IRQ+0x90/0x11c
> [ 0.000000] start_kernel+0x1ac/0x3bc
> [ 0.000000] __primary_switched+0xc4/0xcc
> [ 0.000000] Code: f000e300 91062000 943bd9ba 17ffff8f (d4210000)
> [ 0.000000] ---[ end trace 0000000000000000 ]---
> [ 0.000000] Kernel panic - not syncing: Attempted to kill the idle task!
>
> Compiled with clang 15.0.7 from Arch repos, with
> make ARCH=arm64 LLVM=1
Thanks a lot for testing with LLVM, submitting this report, and doing a
bisect. I sent a patch to fix this a couple of days ago and Peter pushed
it to -tip today, so it should be in the next -next release:
https://git.kernel.org/tip/093d9b240a1fa261ff8aeb7c7cc484dedacfda53
Cheers,
Nathan
^ permalink raw reply [flat|nested] 47+ messages in thread* Re: [PATCH v2 5/12] percpu: Add {raw,this}_cpu_try_cmpxchg()
2023-06-09 16:13 ` Nathan Chancellor
@ 2023-06-09 16:20 ` Konrad Dybcio
0 siblings, 0 replies; 47+ messages in thread
From: Konrad Dybcio @ 2023-06-09 16:20 UTC (permalink / raw)
To: Nathan Chancellor
Cc: Peter Zijlstra, corbet, will, boqun.feng, mark.rutland,
catalin.marinas, dennis, tj, cl, hca, gor, agordeev, borntraeger,
svens, tglx, mingo, bp, dave.hansen, x86, hpa, joro,
suravee.suthikulpanit, robin.murphy, dwmw2, baolu.lu,
Arnd Bergmann, Herbert Xu, davem, penberg, rientjes,
iamjoonsoo.kim, Andrew Morton, vbabka, roman.gushchin, 42.hyeyoo,
linux-doc, linux-kernel, linux-mm, linux-s390, iommu, linux-arch,
linux-crypto, sfr, mpe, James.Bottomley, deller, linux-parisc,
llvm
On 9.06.2023 18:13, Nathan Chancellor wrote:
> Hi Konrad,
>
> On Fri, Jun 09, 2023 at 06:10:38PM +0200, Konrad Dybcio wrote:
>>
>>
>> On 31.05.2023 15:08, Peter Zijlstra wrote:
>>> Add the try_cmpxchg() form to the per-cpu ops.
>>>
>>> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
>>> ---
>> +CC Nathan, llvm list
>>
>> Hi all, this patch seems to break booting on Qualcomm ARM64 platforms
>> when compiled with clang (GCC works fine) for some reason..:
>>
>> next-20230605 - works
>> next-20230606 - doesn't
>>
>> grev -m 1 dc4e51fd9846 on next-20230606 - works again
>> b4 shazam <this_msgid> -P 1-4 - still works
>> b4 shazam <this_msgid> -P 5 - breaks
>>
>> Confirmed on at least Qualcomm QCM2290, SM8250.
>>
>> Checking the serial console, it hits a BUG_ON:
>>
>> [ 0.000000] ------------[ cut here ]------------
>> [ 0.000000] kernel BUG at mm/vmalloc.c:1638!
>> [ 0.000000] Internal error: Oops - BUG: 00000000f2000800 [#1] SMP
>> [ 0.000000] Modules linked in:
>> [ 0.000000] CPU: 0 PID: 0 Comm: swapper/0 Not tainted [snip]
>> [ 0.000000] Hardware name: Qualcomm Technologies, Inc. Robotics RB1 (DT)
>> [ 0.000000] pstate: 000000c5 (nzcv daIF -PAN -UAO -TCO -DIT -SSBS BTYPE=--)
>> [ 0.000000] pc : alloc_vmap_area+0xafc/0xb08
>> [ 0.000000] lr : alloc_vmap_area+0x9e4/0xb08
>> [ 0.000000] sp : ffffa50137f53c20
>> [ 0.000000] x29: ffffa50137f53c60 x28: ffffa50137f30c18 x27: 0000000000000000
>> [ 0.000000] x26: 0000000000007fff x25: ffff800080000000 x24: 000000000000cfff
>> [ 0.000000] x23: ffffffffffff8000 x22: ffffa50137fef970 x21: fffffbfff0000000
>> [ 0.000000] x20: ffff022982003208 x19: ffff0229820031f8 x18: ffffa50137f64f70
>> [ 0.000000] x17: ffffa50137fef980 x16: ffffa501375e6d08 x15: 0000000000000001
>> [ 0.000000] x14: ffffa5013831e1a0 x13: ffffa50137f30c18 x12: 0000000000402dc2
>> [ 0.000000] x11: 0000000000000000 x10: ffff022982003018 x9 : ffffa5013831e188
>> [ 0.000000] x8 : ffffcb55ff003228 x7 : 0000000000000000 x6 : 0000000000000048
>> [ 0.000000] x5 : 0000000000000000 x4 : ffffa50137f53bd0 x3 : ffffa50136490000
>> [ 0.000000] x2 : 0000000000000001 x1 : ffffa5013831e190 x0 : ffff022982003208
>> [ 0.000000] Call trace:
>> [ 0.000000] alloc_vmap_area+0xafc/0xb08
>> [ 0.000000] __get_vm_area_node+0x108/0x1e8
>> [ 0.000000] __vmalloc_node_range+0x1fc/0x728
>> [ 0.000000] __vmalloc_node+0x5c/0x70
>> [ 0.000000] init_IRQ+0x90/0x11c
>> [ 0.000000] start_kernel+0x1ac/0x3bc
>> [ 0.000000] __primary_switched+0xc4/0xcc
>> [ 0.000000] Code: f000e300 91062000 943bd9ba 17ffff8f (d4210000)
>> [ 0.000000] ---[ end trace 0000000000000000 ]---
>> [ 0.000000] Kernel panic - not syncing: Attempted to kill the idle task!
>>
>> Compiled with clang 15.0.7 from Arch repos, with
>> make ARCH=arm64 LLVM=1
>
> Thanks a lot for testing with LLVM, submitting this report, and doing a
> bisect.
No, thank *you* for making it even possible ;)
I sent a patch to fix this a couple of days ago and Peter pushed
> it to -tip today, so it should be in the next -next release:
>
> https://git.kernel.org/tip/093d9b240a1fa261ff8aeb7c7cc484dedacfda53
Amazing, I can boot the most recent next-20230609 with it again!
Konrad
>
> Cheers,
> Nathan
^ permalink raw reply [flat|nested] 47+ messages in thread
* [PATCH 06/12] percpu: Wire up cmpxchg128
2023-05-31 13:08 [PATCH 00/12] Introduce cmpxchg128() -- aka. the demise of cmpxchg_double() Peter Zijlstra
` (4 preceding siblings ...)
2023-05-31 13:08 ` [PATCH 05/12] percpu: Add {raw,this}_cpu_try_cmpxchg() Peter Zijlstra
@ 2023-05-31 13:08 ` Peter Zijlstra
2023-06-05 7:42 ` [tip: locking/core] " tip-bot2 for Peter Zijlstra
2023-05-31 13:08 ` [PATCH 07/12] percpu: #ifndef __SIZEOF_INT128__ Peter Zijlstra
` (6 subsequent siblings)
12 siblings, 1 reply; 47+ messages in thread
From: Peter Zijlstra @ 2023-05-31 13:08 UTC (permalink / raw)
To: torvalds
Cc: corbet, will, peterz, boqun.feng, mark.rutland, catalin.marinas,
dennis, tj, cl, hca, gor, agordeev, borntraeger, svens, tglx,
mingo, bp, dave.hansen, x86, hpa, joro, suravee.suthikulpanit,
robin.murphy, dwmw2, baolu.lu, Arnd Bergmann, Herbert Xu, davem,
penberg, rientjes, iamjoonsoo.kim, Andrew Morton, vbabka,
roman.gushchin, 42.hyeyoo, linux-doc, linux-kernel, linux-mm,
linux-s390, iommu, linux-arch, linux-crypto, sfr, mpe,
James.Bottomley, deller, linux-parisc
In order to replace cmpxchg_double() with the newly minted
cmpxchg128() family of functions, wire it up in this_cpu_cmpxchg().
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
---
arch/arm64/include/asm/percpu.h | 20 ++++++++++
arch/s390/include/asm/percpu.h | 16 ++++++++
arch/x86/include/asm/percpu.h | 74 ++++++++++++++++++++++++++++++++++++----
arch/x86/lib/Makefile | 3 +
arch/x86/lib/cmpxchg16b_emu.S | 43 +++++++++++++----------
arch/x86/lib/cmpxchg8b_emu.S | 67 ++++++++++++++++++++++++++++--------
include/asm-generic/percpu.h | 56 ++++++++++++++++++++++++++++++
7 files changed, 240 insertions(+), 39 deletions(-)
--- a/arch/arm64/include/asm/percpu.h
+++ b/arch/arm64/include/asm/percpu.h
@@ -140,6 +140,10 @@ PERCPU_RET_OP(add, add, ldadd)
* re-enabling preemption for preemptible kernels, but doing that in a way
* which builds inside a module would mean messing directly with the preempt
* count. If you do this, peterz and tglx will hunt you down.
+ *
+ * Not to mention it'll break the actual preemption model for missing a
+ * preemption point when TIF_NEED_RESCHED gets set while preemption is
+ * disabled.
*/
#define this_cpu_cmpxchg_double_8(ptr1, ptr2, o1, o2, n1, n2) \
({ \
@@ -240,6 +244,22 @@ PERCPU_RET_OP(add, add, ldadd)
#define this_cpu_cmpxchg_8(pcp, o, n) \
_pcp_protect_return(cmpxchg_relaxed, pcp, o, n)
+#define this_cpu_cmpxchg64(pcp, o, n) this_cpu_cmpxchg_8(pcp, o, n)
+
+#define this_cpu_cmpxchg128(pcp, o, n) \
+({ \
+ typedef typeof(pcp) pcp_op_T__; \
+ u128 old__, new__, ret__; \
+ pcp_op_T__ *ptr__; \
+ old__ = o; \
+ new__ = n; \
+ preempt_disable_notrace(); \
+ ptr__ = raw_cpu_ptr(&(pcp)); \
+ ret__ = cmpxchg128_local((void *)ptr__, old__, new__); \
+ preempt_enable_notrace(); \
+ ret__; \
+})
+
#ifdef __KVM_NVHE_HYPERVISOR__
extern unsigned long __hyp_per_cpu_offset(unsigned int cpu);
#define __per_cpu_offset
--- a/arch/s390/include/asm/percpu.h
+++ b/arch/s390/include/asm/percpu.h
@@ -148,6 +148,22 @@
#define this_cpu_cmpxchg_4(pcp, oval, nval) arch_this_cpu_cmpxchg(pcp, oval, nval)
#define this_cpu_cmpxchg_8(pcp, oval, nval) arch_this_cpu_cmpxchg(pcp, oval, nval)
+#define this_cpu_cmpxchg64(pcp, o, n) this_cpu_cmpxchg_8(pcp, o, n)
+
+#define this_cpu_cmpxchg128(pcp, oval, nval) \
+({ \
+ typedef typeof(pcp) pcp_op_T__; \
+ u128 old__, new__, ret__; \
+ pcp_op_T__ *ptr__; \
+ old__ = oval; \
+ new__ = nval; \
+ preempt_disable_notrace(); \
+ ptr__ = raw_cpu_ptr(&(pcp)); \
+ ret__ = cmpxchg128((void *)ptr__, old__, new__); \
+ preempt_enable_notrace(); \
+ ret__; \
+})
+
#define arch_this_cpu_xchg(pcp, nval) \
({ \
typeof(pcp) *ptr__; \
--- a/arch/x86/include/asm/percpu.h
+++ b/arch/x86/include/asm/percpu.h
@@ -210,6 +210,67 @@ do { \
(typeof(_var))(unsigned long) pco_old__; \
})
+#if defined(CONFIG_X86_32) && !defined(CONFIG_UML)
+#define percpu_cmpxchg64_op(size, qual, _var, _oval, _nval) \
+({ \
+ union { \
+ u64 var; \
+ struct { \
+ u32 low, high; \
+ }; \
+ } old__, new__; \
+ \
+ old__.var = _oval; \
+ new__.var = _nval; \
+ \
+ asm qual (ALTERNATIVE("leal %P[var], %%esi; call this_cpu_cmpxchg8b_emu", \
+ "cmpxchg8b " __percpu_arg([var]), X86_FEATURE_CX8) \
+ : [var] "+m" (_var), \
+ "+a" (old__.low), \
+ "+d" (old__.high) \
+ : "b" (new__.low), \
+ "c" (new__.high) \
+ : "memory", "esi"); \
+ \
+ old__.var; \
+})
+
+#define raw_cpu_cmpxchg64(pcp, oval, nval) percpu_cmpxchg64_op(8, , pcp, oval, nval)
+#define this_cpu_cmpxchg64(pcp, oval, nval) percpu_cmpxchg64_op(8, volatile, pcp, oval, nval)
+#endif
+
+#ifdef CONFIG_X86_64
+#define raw_cpu_cmpxchg64(pcp, oval, nval) percpu_cmpxchg_op(8, , pcp, oval, nval);
+#define this_cpu_cmpxchg64(pcp, oval, nval) percpu_cmpxchg_op(8, volatile, pcp, oval, nval);
+
+#define percpu_cmpxchg128_op(size, qual, _var, _oval, _nval) \
+({ \
+ union { \
+ u128 var; \
+ struct { \
+ u64 low, high; \
+ }; \
+ } old__, new__; \
+ \
+ old__.var = _oval; \
+ new__.var = _nval; \
+ \
+ asm qual (ALTERNATIVE("leaq %P[var], %%rsi; call this_cpu_cmpxchg16b_emu", \
+ "cmpxchg16b " __percpu_arg([var]), X86_FEATURE_CX16) \
+ : [var] "+m" (_var), \
+ "+a" (old__.low), \
+ "+d" (old__.high) \
+ : "b" (new__.low), \
+ "c" (new__.high) \
+ : "memory", "rsi"); \
+ \
+ old__.var; \
+})
+
+#define raw_cpu_cmpxchg128(pcp, oval, nval) percpu_cmpxchg128_op(16, , pcp, oval, nval)
+#define this_cpu_cmpxchg128(pcp, oval, nval) percpu_cmpxchg128_op(16, volatile, pcp, oval, nval)
+#endif
+
/*
* this_cpu_read() makes gcc load the percpu variable every time it is
* accessed while this_cpu_read_stable() allows the value to be cached.
@@ -341,12 +402,13 @@ do { \
bool __ret; \
typeof(pcp1) __o1 = (o1), __n1 = (n1); \
typeof(pcp2) __o2 = (o2), __n2 = (n2); \
- alternative_io("leaq %P1,%%rsi\n\tcall this_cpu_cmpxchg16b_emu\n\t", \
- "cmpxchg16b " __percpu_arg(1) "\n\tsetz %0\n\t", \
- X86_FEATURE_CX16, \
- ASM_OUTPUT2("=a" (__ret), "+m" (pcp1), \
- "+m" (pcp2), "+d" (__o2)), \
- "b" (__n1), "c" (__n2), "a" (__o1) : "rsi"); \
+ asm volatile (ALTERNATIVE("leaq %P1, %%rsi; call this_cpu_cmpxchg16b_emu", \
+ "cmpxchg16b " __percpu_arg(1), X86_FEATURE_CX16) \
+ "setz %0" \
+ : "=a" (__ret), "+m" (pcp1) \
+ : "b" (__n1), "c" (__n2), \
+ "a" (__o1), "d" (__o2) \
+ : "memory", "rsi"); \
__ret; \
})
--- a/arch/x86/lib/Makefile
+++ b/arch/x86/lib/Makefile
@@ -61,8 +61,9 @@ ifeq ($(CONFIG_X86_32),y)
lib-y += strstr_32.o
lib-y += string_32.o
lib-y += memmove_32.o
+ lib-y += cmpxchg8b_emu.o
ifneq ($(CONFIG_X86_CMPXCHG64),y)
- lib-y += cmpxchg8b_emu.o atomic64_386_32.o
+ lib-y += atomic64_386_32.o
endif
else
obj-y += iomap_copy_64.o
--- a/arch/x86/lib/cmpxchg16b_emu.S
+++ b/arch/x86/lib/cmpxchg16b_emu.S
@@ -1,47 +1,54 @@
/* SPDX-License-Identifier: GPL-2.0-only */
#include <linux/linkage.h>
#include <asm/percpu.h>
+#include <asm/processor-flags.h>
.text
/*
+ * Emulate 'cmpxchg16b %gs:(%rsi)'
+ *
* Inputs:
* %rsi : memory location to compare
* %rax : low 64 bits of old value
* %rdx : high 64 bits of old value
* %rbx : low 64 bits of new value
* %rcx : high 64 bits of new value
- * %al : Operation successful
+ *
+ * Notably this is not LOCK prefixed and is not safe against NMIs
*/
SYM_FUNC_START(this_cpu_cmpxchg16b_emu)
-#
-# Emulate 'cmpxchg16b %gs:(%rsi)' except we return the result in %al not
-# via the ZF. Caller will access %al to get result.
-#
-# Note that this is only useful for a cpuops operation. Meaning that we
-# do *not* have a fully atomic operation but just an operation that is
-# *atomic* on a single cpu (as provided by the this_cpu_xx class of
-# macros).
-#
pushfq
cli
- cmpq PER_CPU_VAR((%rsi)), %rax
- jne .Lnot_same
- cmpq PER_CPU_VAR(8(%rsi)), %rdx
- jne .Lnot_same
+ /* if (*ptr == old) */
+ cmpq PER_CPU_VAR(0(%rsi)), %rax
+ jne .Lnot_same
+ cmpq PER_CPU_VAR(8(%rsi)), %rdx
+ jne .Lnot_same
+
+ /* *ptr = new */
+ movq %rbx, PER_CPU_VAR(0(%rsi))
+ movq %rcx, PER_CPU_VAR(8(%rsi))
- movq %rbx, PER_CPU_VAR((%rsi))
- movq %rcx, PER_CPU_VAR(8(%rsi))
+ /* set ZF in EFLAGS to indicate success */
+ orl $X86_EFLAGS_ZF, (%rsp)
popfq
- mov $1, %al
RET
.Lnot_same:
+ /* *ptr != old */
+
+ /* old = *ptr */
+ movq PER_CPU_VAR(0(%rsi)), %rax
+ movq PER_CPU_VAR(8(%rsi)), %rdx
+
+ /* clear ZF in EFLAGS to indicate failure */
+ andl $(~X86_EFLAGS_ZF), (%rsp)
+
popfq
- xor %al,%al
RET
SYM_FUNC_END(this_cpu_cmpxchg16b_emu)
--- a/arch/x86/lib/cmpxchg8b_emu.S
+++ b/arch/x86/lib/cmpxchg8b_emu.S
@@ -2,10 +2,16 @@
#include <linux/linkage.h>
#include <asm/export.h>
+#include <asm/percpu.h>
+#include <asm/processor-flags.h>
.text
+#ifndef CONFIG_X86_CMPXCHG64
+
/*
+ * Emulate 'cmpxchg8b (%esi)' on UP
+ *
* Inputs:
* %esi : memory location to compare
* %eax : low 32 bits of old value
@@ -15,32 +21,65 @@
*/
SYM_FUNC_START(cmpxchg8b_emu)
-#
-# Emulate 'cmpxchg8b (%esi)' on UP except we don't
-# set the whole ZF thing (caller will just compare
-# eax:edx with the expected value)
-#
pushfl
cli
- cmpl (%esi), %eax
- jne .Lnot_same
- cmpl 4(%esi), %edx
- jne .Lhalf_same
+ cmpl 0(%esi), %eax
+ jne .Lnot_same
+ cmpl 4(%esi), %edx
+ jne .Lnot_same
+
+ movl %ebx, 0(%esi)
+ movl %ecx, 4(%esi)
- movl %ebx, (%esi)
- movl %ecx, 4(%esi)
+ orl $X86_EFLAGS_ZF, (%esp)
popfl
RET
.Lnot_same:
- movl (%esi), %eax
-.Lhalf_same:
- movl 4(%esi), %edx
+ movl 0(%esi), %eax
+ movl 4(%esi), %edx
+
+ andl $(~X86_EFLAGS_ZF), (%esp)
popfl
RET
SYM_FUNC_END(cmpxchg8b_emu)
EXPORT_SYMBOL(cmpxchg8b_emu)
+
+#endif
+
+#ifndef CONFIG_UML
+
+SYM_FUNC_START(this_cpu_cmpxchg8b_emu)
+
+ pushfl
+ cli
+
+ cmpl PER_CPU_VAR(0(%esi)), %eax
+ jne .Lnot_same2
+ cmpl PER_CPU_VAR(4(%esi)), %edx
+ jne .Lnot_same2
+
+ movl %ebx, PER_CPU_VAR(0(%esi))
+ movl %ecx, PER_CPU_VAR(4(%esi))
+
+ orl $X86_EFLAGS_ZF, (%esp)
+
+ popfl
+ RET
+
+.Lnot_same2:
+ movl PER_CPU_VAR(0(%esi)), %eax
+ movl PER_CPU_VAR(4(%esi)), %edx
+
+ andl $(~X86_EFLAGS_ZF), (%esp)
+
+ popfl
+ RET
+
+SYM_FUNC_END(this_cpu_cmpxchg8b_emu)
+
+#endif
--- a/include/asm-generic/percpu.h
+++ b/include/asm-generic/percpu.h
@@ -350,6 +350,25 @@ do { \
#endif
#endif
+#ifndef raw_cpu_try_cmpxchg64
+#ifdef raw_cpu_cmpxchg64
+#define raw_cpu_try_cmpxchg64(pcp, ovalp, nval) \
+ __cpu_fallback_try_cmpxchg(pcp, ovalp, nval, raw_cpu_cmpxchg64)
+#else
+#define raw_cpu_try_cmpxchg64(pcp, ovalp, nval) \
+ raw_cpu_generic_try_cmpxchg(pcp, ovalp, nval)
+#endif
+#endif
+#ifndef raw_cpu_try_cmpxchg128
+#ifdef raw_cpu_cmpxchg128
+#define raw_cpu_try_cmpxchg128(pcp, ovalp, nval) \
+ __cpu_fallback_try_cmpxchg(pcp, ovalp, nval, raw_cpu_cmpxchg128)
+#else
+#define raw_cpu_try_cmpxchg128(pcp, ovalp, nval) \
+ raw_cpu_generic_try_cmpxchg(pcp, ovalp, nval)
+#endif
+#endif
+
#ifndef raw_cpu_cmpxchg_1
#define raw_cpu_cmpxchg_1(pcp, oval, nval) \
raw_cpu_generic_cmpxchg(pcp, oval, nval)
@@ -367,6 +386,15 @@ do { \
raw_cpu_generic_cmpxchg(pcp, oval, nval)
#endif
+#ifndef raw_cpu_cmpxchg64
+#define raw_cpu_cmpxchg64(pcp, oval, nval) \
+ raw_cpu_generic_cmpxchg(pcp, oval, nval)
+#endif
+#ifndef raw_cpu_cmpxchg128
+#define raw_cpu_cmpxchg128(pcp, oval, nval) \
+ raw_cpu_generic_cmpxchg(pcp, oval, nval)
+#endif
+
#ifndef raw_cpu_cmpxchg_double_1
#define raw_cpu_cmpxchg_double_1(pcp1, pcp2, oval1, oval2, nval1, nval2) \
raw_cpu_generic_cmpxchg_double(pcp1, pcp2, oval1, oval2, nval1, nval2)
@@ -512,6 +540,25 @@ do { \
#endif
#endif
+#ifndef this_cpu_try_cmpxchg64
+#ifdef this_cpu_cmpxchg64
+#define this_cpu_try_cmpxchg64(pcp, ovalp, nval) \
+ __cpu_fallback_try_cmpxchg(pcp, ovalp, nval, this_cpu_cmpxchg64)
+#else
+#define this_cpu_try_cmpxchg64(pcp, ovalp, nval) \
+ this_cpu_generic_try_cmpxchg(pcp, ovalp, nval)
+#endif
+#endif
+#ifndef this_cpu_try_cmpxchg128
+#ifdef this_cpu_cmpxchg128
+#define this_cpu_try_cmpxchg128(pcp, ovalp, nval) \
+ __cpu_fallback_try_cmpxchg(pcp, ovalp, nval, this_cpu_cmpxchg128)
+#else
+#define this_cpu_try_cmpxchg128(pcp, ovalp, nval) \
+ this_cpu_generic_try_cmpxchg(pcp, ovalp, nval)
+#endif
+#endif
+
#ifndef this_cpu_cmpxchg_1
#define this_cpu_cmpxchg_1(pcp, oval, nval) \
this_cpu_generic_cmpxchg(pcp, oval, nval)
@@ -529,6 +576,15 @@ do { \
this_cpu_generic_cmpxchg(pcp, oval, nval)
#endif
+#ifndef this_cpu_cmpxchg64
+#define this_cpu_cmpxchg64(pcp, oval, nval) \
+ this_cpu_generic_cmpxchg(pcp, oval, nval)
+#endif
+#ifndef this_cpu_cmpxchg128
+#define this_cpu_cmpxchg128(pcp, oval, nval) \
+ this_cpu_generic_cmpxchg(pcp, oval, nval)
+#endif
+
#ifndef this_cpu_cmpxchg_double_1
#define this_cpu_cmpxchg_double_1(pcp1, pcp2, oval1, oval2, nval1, nval2) \
this_cpu_generic_cmpxchg_double(pcp1, pcp2, oval1, oval2, nval1, nval2)
^ permalink raw reply [flat|nested] 47+ messages in thread* [tip: locking/core] percpu: Wire up cmpxchg128
2023-05-31 13:08 ` [PATCH 06/12] percpu: Wire up cmpxchg128 Peter Zijlstra
@ 2023-06-05 7:42 ` tip-bot2 for Peter Zijlstra
0 siblings, 0 replies; 47+ messages in thread
From: tip-bot2 for Peter Zijlstra @ 2023-06-05 7:42 UTC (permalink / raw)
To: linux-tip-commits; +Cc: Peter Zijlstra (Intel), Mark Rutland, x86, linux-kernel
The following commit has been merged into the locking/core branch of tip:
Commit-ID: 6d12c8d308e68b9b0fa98ca2df4f83db4b4c965d
Gitweb: https://git.kernel.org/tip/6d12c8d308e68b9b0fa98ca2df4f83db4b4c965d
Author: Peter Zijlstra <peterz@infradead.org>
AuthorDate: Wed, 31 May 2023 15:08:39 +02:00
Committer: Peter Zijlstra <peterz@infradead.org>
CommitterDate: Mon, 05 Jun 2023 09:36:37 +02:00
percpu: Wire up cmpxchg128
In order to replace cmpxchg_double() with the newly minted
cmpxchg128() family of functions, wire it up in this_cpu_cmpxchg().
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Reviewed-by: Mark Rutland <mark.rutland@arm.com>
Tested-by: Mark Rutland <mark.rutland@arm.com>
Link: https://lore.kernel.org/r/20230531132323.654945124@infradead.org
---
arch/arm64/include/asm/percpu.h | 20 +++++++++-
arch/s390/include/asm/percpu.h | 16 +++++++-
arch/x86/include/asm/percpu.h | 74 +++++++++++++++++++++++++++++---
arch/x86/lib/Makefile | 3 +-
arch/x86/lib/cmpxchg16b_emu.S | 43 +++++++++++--------
arch/x86/lib/cmpxchg8b_emu.S | 67 ++++++++++++++++++++++-------
include/asm-generic/percpu.h | 56 ++++++++++++++++++++++++-
7 files changed, 240 insertions(+), 39 deletions(-)
diff --git a/arch/arm64/include/asm/percpu.h b/arch/arm64/include/asm/percpu.h
index b9ba19d..2549829 100644
--- a/arch/arm64/include/asm/percpu.h
+++ b/arch/arm64/include/asm/percpu.h
@@ -140,6 +140,10 @@ PERCPU_RET_OP(add, add, ldadd)
* re-enabling preemption for preemptible kernels, but doing that in a way
* which builds inside a module would mean messing directly with the preempt
* count. If you do this, peterz and tglx will hunt you down.
+ *
+ * Not to mention it'll break the actual preemption model for missing a
+ * preemption point when TIF_NEED_RESCHED gets set while preemption is
+ * disabled.
*/
#define this_cpu_cmpxchg_double_8(ptr1, ptr2, o1, o2, n1, n2) \
({ \
@@ -240,6 +244,22 @@ PERCPU_RET_OP(add, add, ldadd)
#define this_cpu_cmpxchg_8(pcp, o, n) \
_pcp_protect_return(cmpxchg_relaxed, pcp, o, n)
+#define this_cpu_cmpxchg64(pcp, o, n) this_cpu_cmpxchg_8(pcp, o, n)
+
+#define this_cpu_cmpxchg128(pcp, o, n) \
+({ \
+ typedef typeof(pcp) pcp_op_T__; \
+ u128 old__, new__, ret__; \
+ pcp_op_T__ *ptr__; \
+ old__ = o; \
+ new__ = n; \
+ preempt_disable_notrace(); \
+ ptr__ = raw_cpu_ptr(&(pcp)); \
+ ret__ = cmpxchg128_local((void *)ptr__, old__, new__); \
+ preempt_enable_notrace(); \
+ ret__; \
+})
+
#ifdef __KVM_NVHE_HYPERVISOR__
extern unsigned long __hyp_per_cpu_offset(unsigned int cpu);
#define __per_cpu_offset
diff --git a/arch/s390/include/asm/percpu.h b/arch/s390/include/asm/percpu.h
index 081837b..5603402 100644
--- a/arch/s390/include/asm/percpu.h
+++ b/arch/s390/include/asm/percpu.h
@@ -148,6 +148,22 @@
#define this_cpu_cmpxchg_4(pcp, oval, nval) arch_this_cpu_cmpxchg(pcp, oval, nval)
#define this_cpu_cmpxchg_8(pcp, oval, nval) arch_this_cpu_cmpxchg(pcp, oval, nval)
+#define this_cpu_cmpxchg64(pcp, o, n) this_cpu_cmpxchg_8(pcp, o, n)
+
+#define this_cpu_cmpxchg128(pcp, oval, nval) \
+({ \
+ typedef typeof(pcp) pcp_op_T__; \
+ u128 old__, new__, ret__; \
+ pcp_op_T__ *ptr__; \
+ old__ = oval; \
+ new__ = nval; \
+ preempt_disable_notrace(); \
+ ptr__ = raw_cpu_ptr(&(pcp)); \
+ ret__ = cmpxchg128((void *)ptr__, old__, new__); \
+ preempt_enable_notrace(); \
+ ret__; \
+})
+
#define arch_this_cpu_xchg(pcp, nval) \
({ \
typeof(pcp) *ptr__; \
diff --git a/arch/x86/include/asm/percpu.h b/arch/x86/include/asm/percpu.h
index 13c0d63..4232fb2 100644
--- a/arch/x86/include/asm/percpu.h
+++ b/arch/x86/include/asm/percpu.h
@@ -210,6 +210,67 @@ do { \
(typeof(_var))(unsigned long) pco_old__; \
})
+#if defined(CONFIG_X86_32) && !defined(CONFIG_UML)
+#define percpu_cmpxchg64_op(size, qual, _var, _oval, _nval) \
+({ \
+ union { \
+ u64 var; \
+ struct { \
+ u32 low, high; \
+ }; \
+ } old__, new__; \
+ \
+ old__.var = _oval; \
+ new__.var = _nval; \
+ \
+ asm qual (ALTERNATIVE("leal %P[var], %%esi; call this_cpu_cmpxchg8b_emu", \
+ "cmpxchg8b " __percpu_arg([var]), X86_FEATURE_CX8) \
+ : [var] "+m" (_var), \
+ "+a" (old__.low), \
+ "+d" (old__.high) \
+ : "b" (new__.low), \
+ "c" (new__.high) \
+ : "memory", "esi"); \
+ \
+ old__.var; \
+})
+
+#define raw_cpu_cmpxchg64(pcp, oval, nval) percpu_cmpxchg64_op(8, , pcp, oval, nval)
+#define this_cpu_cmpxchg64(pcp, oval, nval) percpu_cmpxchg64_op(8, volatile, pcp, oval, nval)
+#endif
+
+#ifdef CONFIG_X86_64
+#define raw_cpu_cmpxchg64(pcp, oval, nval) percpu_cmpxchg_op(8, , pcp, oval, nval);
+#define this_cpu_cmpxchg64(pcp, oval, nval) percpu_cmpxchg_op(8, volatile, pcp, oval, nval);
+
+#define percpu_cmpxchg128_op(size, qual, _var, _oval, _nval) \
+({ \
+ union { \
+ u128 var; \
+ struct { \
+ u64 low, high; \
+ }; \
+ } old__, new__; \
+ \
+ old__.var = _oval; \
+ new__.var = _nval; \
+ \
+ asm qual (ALTERNATIVE("leaq %P[var], %%rsi; call this_cpu_cmpxchg16b_emu", \
+ "cmpxchg16b " __percpu_arg([var]), X86_FEATURE_CX16) \
+ : [var] "+m" (_var), \
+ "+a" (old__.low), \
+ "+d" (old__.high) \
+ : "b" (new__.low), \
+ "c" (new__.high) \
+ : "memory", "rsi"); \
+ \
+ old__.var; \
+})
+
+#define raw_cpu_cmpxchg128(pcp, oval, nval) percpu_cmpxchg128_op(16, , pcp, oval, nval)
+#define this_cpu_cmpxchg128(pcp, oval, nval) percpu_cmpxchg128_op(16, volatile, pcp, oval, nval)
+#endif
+
/*
* this_cpu_read() makes gcc load the percpu variable every time it is
* accessed while this_cpu_read_stable() allows the value to be cached.
@@ -341,12 +402,13 @@ do { \
bool __ret; \
typeof(pcp1) __o1 = (o1), __n1 = (n1); \
typeof(pcp2) __o2 = (o2), __n2 = (n2); \
- alternative_io("leaq %P1,%%rsi\n\tcall this_cpu_cmpxchg16b_emu\n\t", \
- "cmpxchg16b " __percpu_arg(1) "\n\tsetz %0\n\t", \
- X86_FEATURE_CX16, \
- ASM_OUTPUT2("=a" (__ret), "+m" (pcp1), \
- "+m" (pcp2), "+d" (__o2)), \
- "b" (__n1), "c" (__n2), "a" (__o1) : "rsi"); \
+ asm volatile (ALTERNATIVE("leaq %P1, %%rsi; call this_cpu_cmpxchg16b_emu", \
+ "cmpxchg16b " __percpu_arg(1), X86_FEATURE_CX16) \
+ "setz %0" \
+ : "=a" (__ret), "+m" (pcp1) \
+ : "b" (__n1), "c" (__n2), \
+ "a" (__o1), "d" (__o2) \
+ : "memory", "rsi"); \
__ret; \
})
diff --git a/arch/x86/lib/Makefile b/arch/x86/lib/Makefile
index 01932af..ea3a28e 100644
--- a/arch/x86/lib/Makefile
+++ b/arch/x86/lib/Makefile
@@ -61,8 +61,9 @@ ifeq ($(CONFIG_X86_32),y)
lib-y += strstr_32.o
lib-y += string_32.o
lib-y += memmove_32.o
+ lib-y += cmpxchg8b_emu.o
ifneq ($(CONFIG_X86_CMPXCHG64),y)
- lib-y += cmpxchg8b_emu.o atomic64_386_32.o
+ lib-y += atomic64_386_32.o
endif
else
obj-y += iomap_copy_64.o
diff --git a/arch/x86/lib/cmpxchg16b_emu.S b/arch/x86/lib/cmpxchg16b_emu.S
index 33c70c0..6962df3 100644
--- a/arch/x86/lib/cmpxchg16b_emu.S
+++ b/arch/x86/lib/cmpxchg16b_emu.S
@@ -1,47 +1,54 @@
/* SPDX-License-Identifier: GPL-2.0-only */
#include <linux/linkage.h>
#include <asm/percpu.h>
+#include <asm/processor-flags.h>
.text
/*
+ * Emulate 'cmpxchg16b %gs:(%rsi)'
+ *
* Inputs:
* %rsi : memory location to compare
* %rax : low 64 bits of old value
* %rdx : high 64 bits of old value
* %rbx : low 64 bits of new value
* %rcx : high 64 bits of new value
- * %al : Operation successful
+ *
+ * Notably this is not LOCK prefixed and is not safe against NMIs
*/
SYM_FUNC_START(this_cpu_cmpxchg16b_emu)
-#
-# Emulate 'cmpxchg16b %gs:(%rsi)' except we return the result in %al not
-# via the ZF. Caller will access %al to get result.
-#
-# Note that this is only useful for a cpuops operation. Meaning that we
-# do *not* have a fully atomic operation but just an operation that is
-# *atomic* on a single cpu (as provided by the this_cpu_xx class of
-# macros).
-#
pushfq
cli
- cmpq PER_CPU_VAR((%rsi)), %rax
- jne .Lnot_same
- cmpq PER_CPU_VAR(8(%rsi)), %rdx
- jne .Lnot_same
+ /* if (*ptr == old) */
+ cmpq PER_CPU_VAR(0(%rsi)), %rax
+ jne .Lnot_same
+ cmpq PER_CPU_VAR(8(%rsi)), %rdx
+ jne .Lnot_same
- movq %rbx, PER_CPU_VAR((%rsi))
- movq %rcx, PER_CPU_VAR(8(%rsi))
+ /* *ptr = new */
+ movq %rbx, PER_CPU_VAR(0(%rsi))
+ movq %rcx, PER_CPU_VAR(8(%rsi))
+
+ /* set ZF in EFLAGS to indicate success */
+ orl $X86_EFLAGS_ZF, (%rsp)
popfq
- mov $1, %al
RET
.Lnot_same:
+ /* *ptr != old */
+
+ /* old = *ptr */
+ movq PER_CPU_VAR(0(%rsi)), %rax
+ movq PER_CPU_VAR(8(%rsi)), %rdx
+
+ /* clear ZF in EFLAGS to indicate failure */
+ andl $(~X86_EFLAGS_ZF), (%rsp)
+
popfq
- xor %al,%al
RET
SYM_FUNC_END(this_cpu_cmpxchg16b_emu)
diff --git a/arch/x86/lib/cmpxchg8b_emu.S b/arch/x86/lib/cmpxchg8b_emu.S
index 6a912d5..4980525 100644
--- a/arch/x86/lib/cmpxchg8b_emu.S
+++ b/arch/x86/lib/cmpxchg8b_emu.S
@@ -2,10 +2,16 @@
#include <linux/linkage.h>
#include <asm/export.h>
+#include <asm/percpu.h>
+#include <asm/processor-flags.h>
.text
+#ifndef CONFIG_X86_CMPXCHG64
+
/*
+ * Emulate 'cmpxchg8b (%esi)' on UP
+ *
* Inputs:
* %esi : memory location to compare
* %eax : low 32 bits of old value
@@ -15,32 +21,65 @@
*/
SYM_FUNC_START(cmpxchg8b_emu)
-#
-# Emulate 'cmpxchg8b (%esi)' on UP except we don't
-# set the whole ZF thing (caller will just compare
-# eax:edx with the expected value)
-#
pushfl
cli
- cmpl (%esi), %eax
- jne .Lnot_same
- cmpl 4(%esi), %edx
- jne .Lhalf_same
+ cmpl 0(%esi), %eax
+ jne .Lnot_same
+ cmpl 4(%esi), %edx
+ jne .Lnot_same
+
+ movl %ebx, 0(%esi)
+ movl %ecx, 4(%esi)
- movl %ebx, (%esi)
- movl %ecx, 4(%esi)
+ orl $X86_EFLAGS_ZF, (%esp)
popfl
RET
.Lnot_same:
- movl (%esi), %eax
-.Lhalf_same:
- movl 4(%esi), %edx
+ movl 0(%esi), %eax
+ movl 4(%esi), %edx
+
+ andl $(~X86_EFLAGS_ZF), (%esp)
popfl
RET
SYM_FUNC_END(cmpxchg8b_emu)
EXPORT_SYMBOL(cmpxchg8b_emu)
+
+#endif
+
+#ifndef CONFIG_UML
+
+SYM_FUNC_START(this_cpu_cmpxchg8b_emu)
+
+ pushfl
+ cli
+
+ cmpl PER_CPU_VAR(0(%esi)), %eax
+ jne .Lnot_same2
+ cmpl PER_CPU_VAR(4(%esi)), %edx
+ jne .Lnot_same2
+
+ movl %ebx, PER_CPU_VAR(0(%esi))
+ movl %ecx, PER_CPU_VAR(4(%esi))
+
+ orl $X86_EFLAGS_ZF, (%esp)
+
+ popfl
+ RET
+
+.Lnot_same2:
+ movl PER_CPU_VAR(0(%esi)), %eax
+ movl PER_CPU_VAR(4(%esi)), %edx
+
+ andl $(~X86_EFLAGS_ZF), (%esp)
+
+ popfl
+ RET
+
+SYM_FUNC_END(this_cpu_cmpxchg8b_emu)
+
+#endif
diff --git a/include/asm-generic/percpu.h b/include/asm-generic/percpu.h
index 96af32c..5c66e44 100644
--- a/include/asm-generic/percpu.h
+++ b/include/asm-generic/percpu.h
@@ -350,6 +350,25 @@ do { \
#endif
#endif
+#ifndef raw_cpu_try_cmpxchg64
+#ifdef raw_cpu_cmpxchg64
+#define raw_cpu_try_cmpxchg64(pcp, ovalp, nval) \
+ __cpu_fallback_try_cmpxchg(pcp, ovalp, nval, raw_cpu_cmpxchg64)
+#else
+#define raw_cpu_try_cmpxchg64(pcp, ovalp, nval) \
+ raw_cpu_generic_try_cmpxchg(pcp, ovalp, nval)
+#endif
+#endif
+#ifndef raw_cpu_try_cmpxchg128
+#ifdef raw_cpu_cmpxchg128
+#define raw_cpu_try_cmpxchg128(pcp, ovalp, nval) \
+ __cpu_fallback_try_cmpxchg(pcp, ovalp, nval, raw_cpu_cmpxchg128)
+#else
+#define raw_cpu_try_cmpxchg128(pcp, ovalp, nval) \
+ raw_cpu_generic_try_cmpxchg(pcp, ovalp, nval)
+#endif
+#endif
+
#ifndef raw_cpu_cmpxchg_1
#define raw_cpu_cmpxchg_1(pcp, oval, nval) \
raw_cpu_generic_cmpxchg(pcp, oval, nval)
@@ -367,6 +386,15 @@ do { \
raw_cpu_generic_cmpxchg(pcp, oval, nval)
#endif
+#ifndef raw_cpu_cmpxchg64
+#define raw_cpu_cmpxchg64(pcp, oval, nval) \
+ raw_cpu_generic_cmpxchg(pcp, oval, nval)
+#endif
+#ifndef raw_cpu_cmpxchg128
+#define raw_cpu_cmpxchg128(pcp, oval, nval) \
+ raw_cpu_generic_cmpxchg(pcp, oval, nval)
+#endif
+
#ifndef raw_cpu_cmpxchg_double_1
#define raw_cpu_cmpxchg_double_1(pcp1, pcp2, oval1, oval2, nval1, nval2) \
raw_cpu_generic_cmpxchg_double(pcp1, pcp2, oval1, oval2, nval1, nval2)
@@ -512,6 +540,25 @@ do { \
#endif
#endif
+#ifndef this_cpu_try_cmpxchg64
+#ifdef this_cpu_cmpxchg64
+#define this_cpu_try_cmpxchg64(pcp, ovalp, nval) \
+ __cpu_fallback_try_cmpxchg(pcp, ovalp, nval, this_cpu_cmpxchg64)
+#else
+#define this_cpu_try_cmpxchg64(pcp, ovalp, nval) \
+ this_cpu_generic_try_cmpxchg(pcp, ovalp, nval)
+#endif
+#endif
+#ifndef this_cpu_try_cmpxchg128
+#ifdef this_cpu_cmpxchg128
+#define this_cpu_try_cmpxchg128(pcp, ovalp, nval) \
+ __cpu_fallback_try_cmpxchg(pcp, ovalp, nval, this_cpu_cmpxchg128)
+#else
+#define this_cpu_try_cmpxchg128(pcp, ovalp, nval) \
+ this_cpu_generic_try_cmpxchg(pcp, ovalp, nval)
+#endif
+#endif
+
#ifndef this_cpu_cmpxchg_1
#define this_cpu_cmpxchg_1(pcp, oval, nval) \
this_cpu_generic_cmpxchg(pcp, oval, nval)
@@ -529,6 +576,15 @@ do { \
this_cpu_generic_cmpxchg(pcp, oval, nval)
#endif
+#ifndef this_cpu_cmpxchg64
+#define this_cpu_cmpxchg64(pcp, oval, nval) \
+ this_cpu_generic_cmpxchg(pcp, oval, nval)
+#endif
+#ifndef this_cpu_cmpxchg128
+#define this_cpu_cmpxchg128(pcp, oval, nval) \
+ this_cpu_generic_cmpxchg(pcp, oval, nval)
+#endif
+
#ifndef this_cpu_cmpxchg_double_1
#define this_cpu_cmpxchg_double_1(pcp1, pcp2, oval1, oval2, nval1, nval2) \
this_cpu_generic_cmpxchg_double(pcp1, pcp2, oval1, oval2, nval1, nval2)
^ permalink raw reply [flat|nested] 47+ messages in thread
* [PATCH 07/12] percpu: #ifndef __SIZEOF_INT128__
2023-05-31 13:08 [PATCH 00/12] Introduce cmpxchg128() -- aka. the demise of cmpxchg_double() Peter Zijlstra
` (5 preceding siblings ...)
2023-05-31 13:08 ` [PATCH 06/12] percpu: Wire up cmpxchg128 Peter Zijlstra
@ 2023-05-31 13:08 ` Peter Zijlstra
2023-05-31 14:21 ` Arnd Bergmann
2023-05-31 13:08 ` [PATCH 08/12] x86,amd_iommu: Replace cmpxchg_double() Peter Zijlstra
` (5 subsequent siblings)
12 siblings, 1 reply; 47+ messages in thread
From: Peter Zijlstra @ 2023-05-31 13:08 UTC (permalink / raw)
To: torvalds
Cc: corbet, will, peterz, boqun.feng, mark.rutland, catalin.marinas,
dennis, tj, cl, hca, gor, agordeev, borntraeger, svens, tglx,
mingo, bp, dave.hansen, x86, hpa, joro, suravee.suthikulpanit,
robin.murphy, dwmw2, baolu.lu, Arnd Bergmann, Herbert Xu, davem,
penberg, rientjes, iamjoonsoo.kim, Andrew Morton, vbabka,
roman.gushchin, 42.hyeyoo, linux-doc, linux-kernel, linux-mm,
linux-s390, iommu, linux-arch, linux-crypto, sfr, mpe,
James.Bottomley, deller, linux-parisc
Some 64bit architectures do not advertise __SIZEOF_INT128__ on all
supported compiler versions. Notably the HPPA64 only started doing
with GCC-11.
Since the per-cpu ops are universally availably, and
this_cpu_{,try_}cmpxchg128() is expected to be available on all 64bit
architectures a wee bodge is in order.
Sadly, while C reverts to memcpy() for assignment of POD types, it does
not revert to memcmp() for for equality. Therefore frob that manually.
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
---
include/asm-generic/percpu.h | 56 +++++++++++++++++++++++++++++++++++++++++++
include/linux/types.h | 7 +++++
2 files changed, 63 insertions(+)
--- a/include/asm-generic/percpu.h
+++ b/include/asm-generic/percpu.h
@@ -313,6 +313,35 @@ do { \
#define raw_cpu_xchg_8(pcp, nval) raw_cpu_generic_xchg(pcp, nval)
#endif
+#ifndef __SIZEOF_INT128__
+#define raw_cpu_generic_try_cmpxchg_memcmp(pcp, ovalp, nval) \
+({ \
+ typeof(pcp) *__p = raw_cpu_ptr(&(pcp)); \
+ typeof(pcp) __val = *__p, __old = *(ovalp); \
+ bool __ret; \
+ if (!__builtin_memcmp(&__val, &__old, sizeof(pcp))) { \
+ *__p = nval; \
+ __ret = true; \
+ } else { \
+ *(ovalp) = __val; \
+ __ret = false; \
+ } \
+ __ret; \
+})
+
+#define raw_cpu_generic_cmpxchg_memcmp(pcp, oval, nval) \
+({ \
+ typeof(pcp) __old = (oval); \
+ raw_cpu_generic_try_cmpxchg_memcpy(pcp, &__old, nval); \
+ __old; \
+})
+
+#define raw_cpu_cmpxchg128(pcp, oval, nval) \
+ raw_cpu_generic_cmpxchg_memcmp(pcp, oval, nval)
+#define raw_cpu_try_cmpxchg128(pcp, ovalp, nval) \
+ raw_cpu_generic_try_cmpxchg_memcmp(pcp, ovalp, nval)
+#endif
+
#ifndef raw_cpu_try_cmpxchg_1
#ifdef raw_cpu_cmpxchg_1
#define raw_cpu_try_cmpxchg_1(pcp, ovalp, nval) \
@@ -503,6 +532,33 @@ do { \
#define this_cpu_xchg_8(pcp, nval) this_cpu_generic_xchg(pcp, nval)
#endif
+#ifndef __SIZEOF_INT128__
+#define this_cpu_generic_try_cmpxchg_memcmp(pcp, ovalp, nval) \
+({ \
+ bool __ret; \
+ unsigned long __flags; \
+ raw_local_irq_save(__flags); \
+ __ret = raw_cpu_generic_try_cmpxchg_memcmp(pcp, ovalp, nval); \
+ raw_local_irq_restore(__flags); \
+ __ret; \
+})
+
+#define this_cpu_generic_cmpxchg_memcmp(pcp, oval, nval) \
+({ \
+ typeof(pcp) __ret; \
+ unsigned long __flags; \
+ raw_local_irq_save(__flags); \
+ __ret = raw_cpu_generic_cmpxchg_memcmp(pcp, oval, nval); \
+ raw_local_irq_restore(__flags); \
+ __ret; \
+})
+
+#define this_cpu_cmpxchg128(pcp, oval, nval) \
+ this_cpu_generic_cmpxchg_memcmp(pcp, oval, nval)
+#define this_cpu_try_cmpxchg128(pcp, ovalp, nval) \
+ this_cpu_generic_try_cmpxchg_memcmp(pcp, ovalp, nval)
+#endif
+
#ifndef this_cpu_try_cmpxchg_1
#ifdef this_cpu_cmpxchg_1
#define this_cpu_try_cmpxchg_1(pcp, ovalp, nval) \
--- a/include/linux/types.h
+++ b/include/linux/types.h
@@ -13,6 +13,13 @@
#ifdef __SIZEOF_INT128__
typedef __s128 s128;
typedef __u128 u128;
+#else
+#ifdef CONFIG_64BIT
+/* hack for this_cpu_cmpxchg128 */
+typedef struct {
+ u64 a, b;
+} u128 __attribute__((aligned(16)));
+#endif
#endif
typedef u32 __kernel_dev_t;
^ permalink raw reply [flat|nested] 47+ messages in thread* Re: [PATCH 07/12] percpu: #ifndef __SIZEOF_INT128__
2023-05-31 13:08 ` [PATCH 07/12] percpu: #ifndef __SIZEOF_INT128__ Peter Zijlstra
@ 2023-05-31 14:21 ` Arnd Bergmann
2023-05-31 15:50 ` Peter Zijlstra
2023-06-01 10:14 ` [PATCH v2 07/12] parisc/percpu: Work around the lack of __SIZEOF_INT128__ Peter Zijlstra
0 siblings, 2 replies; 47+ messages in thread
From: Arnd Bergmann @ 2023-05-31 14:21 UTC (permalink / raw)
To: Peter Zijlstra, Linus Torvalds
Cc: Jonathan Corbet, Will Deacon, Boqun Feng, Mark Rutland,
Catalin Marinas, dennis, Tejun Heo, Christoph Lameter,
Heiko Carstens, gor, Alexander Gordeev, borntraeger,
Sven Schnelle, Thomas Gleixner, Ingo Molnar, Borislav Petkov,
Dave Hansen, x86, H. Peter Anvin, Joerg Roedel,
suravee.suthikulpanit, Robin Murphy, David Woodhouse, Baolu Lu,
Herbert Xu, David S . Miller, Pekka Enberg, David Rientjes,
Joonsoo Kim, Andrew Morton, Vlastimil Babka, Roman Gushchin,
Hyeonggon Yoo, linux-doc, linux-kernel, linux-mm, linux-s390,
iommu, Linux-Arch, linux-crypto, Stephen Rothwell,
Michael Ellerman, James E . J . Bottomley, Helge Deller,
linux-parisc
On Wed, May 31, 2023, at 15:08, Peter Zijlstra wrote:
> Some 64bit architectures do not advertise __SIZEOF_INT128__ on all
> supported compiler versions. Notably the HPPA64 only started doing
> with GCC-11.
I checked the other compilers to be sure that anything else
we support (gcc-5.1 and up) across all 64-bit architectures
does support int128.
It would be nice to have the hack more localized to parisc
and guarded with a CONFIG_GCC_VERSION check so we can kill
it off in the future, once we drop either gcc-10 or parisc
support.
> +#ifndef __SIZEOF_INT128__
> +#define raw_cpu_generic_try_cmpxchg_memcmp(pcp, ovalp, nval) \
> +({ \
> + typeof(pcp) *__p = raw_cpu_ptr(&(pcp)); \
> + typeof(pcp) __val = *__p, __old = *(ovalp); \
> + bool __ret; \
> + if (!__builtin_memcmp(&__val, &__old, sizeof(pcp))) { \
> + *__p = nval; \
> + __ret = true; \
> + } else { \
> + *(ovalp) = __val; \
> + __ret = false; \
> + } \
> + __ret; \
> +})
> +
> +#define raw_cpu_generic_cmpxchg_memcmp(pcp, oval, nval) \
> +({ \
> + typeof(pcp) __old = (oval); \
> + raw_cpu_generic_try_cmpxchg_memcpy(pcp, &__old, nval); \
> + __old; \
> +})
Instead of having this in include/asm-generic under
!__SIZEOF_INT128__, could you just move this into the parisc
files with a compiler version check?
Arnd
^ permalink raw reply [flat|nested] 47+ messages in thread* Re: [PATCH 07/12] percpu: #ifndef __SIZEOF_INT128__
2023-05-31 14:21 ` Arnd Bergmann
@ 2023-05-31 15:50 ` Peter Zijlstra
2023-06-01 10:14 ` [PATCH v2 07/12] parisc/percpu: Work around the lack of __SIZEOF_INT128__ Peter Zijlstra
1 sibling, 0 replies; 47+ messages in thread
From: Peter Zijlstra @ 2023-05-31 15:50 UTC (permalink / raw)
To: Arnd Bergmann
Cc: Linus Torvalds, Jonathan Corbet, Will Deacon, Boqun Feng,
Mark Rutland, Catalin Marinas, dennis, Tejun Heo,
Christoph Lameter, Heiko Carstens, gor, Alexander Gordeev,
borntraeger, Sven Schnelle, Thomas Gleixner, Ingo Molnar,
Borislav Petkov, Dave Hansen, x86, H. Peter Anvin, Joerg Roedel,
suravee.suthikulpanit, Robin Murphy, David Woodhouse, Baolu Lu,
Herbert Xu, David S . Miller, Pekka Enberg, David Rientjes,
Joonsoo Kim, Andrew Morton, Vlastimil Babka, Roman Gushchin,
Hyeonggon Yoo, linux-doc, linux-kernel, linux-mm, linux-s390,
iommu, Linux-Arch, linux-crypto, Stephen Rothwell,
Michael Ellerman, James E . J . Bottomley, Helge Deller,
linux-parisc
On Wed, May 31, 2023 at 04:21:22PM +0200, Arnd Bergmann wrote:
> On Wed, May 31, 2023, at 15:08, Peter Zijlstra wrote:
> > Some 64bit architectures do not advertise __SIZEOF_INT128__ on all
> > supported compiler versions. Notably the HPPA64 only started doing
> > with GCC-11.
>
> I checked the other compilers to be sure that anything else
> we support (gcc-5.1 and up) across all 64-bit architectures
> does support int128.
Oh excellent -- I didn't have sufficient old cross compilers to verify,
hence me not doing what you suggest below.
If HPPA64 really is the only one so affected, then yes, I can move this
hack into arch/parisc.
^ permalink raw reply [flat|nested] 47+ messages in thread
* [PATCH v2 07/12] parisc/percpu: Work around the lack of __SIZEOF_INT128__
2023-05-31 14:21 ` Arnd Bergmann
2023-05-31 15:50 ` Peter Zijlstra
@ 2023-06-01 10:14 ` Peter Zijlstra
2023-06-01 10:32 ` Helge Deller
1 sibling, 1 reply; 47+ messages in thread
From: Peter Zijlstra @ 2023-06-01 10:14 UTC (permalink / raw)
To: Arnd Bergmann
Cc: Linus Torvalds, Jonathan Corbet, Will Deacon, Boqun Feng,
Mark Rutland, Catalin Marinas, dennis, Tejun Heo,
Christoph Lameter, Heiko Carstens, gor, Alexander Gordeev,
borntraeger, Sven Schnelle, Thomas Gleixner, Ingo Molnar,
Borislav Petkov, Dave Hansen, x86, H. Peter Anvin, Joerg Roedel,
suravee.suthikulpanit, Robin Murphy, David Woodhouse, Baolu Lu,
Herbert Xu, David S . Miller, Pekka Enberg, David Rientjes,
Joonsoo Kim, Andrew Morton, Vlastimil Babka, Roman Gushchin,
Hyeonggon Yoo, linux-doc, linux-kernel, linux-mm, linux-s390,
iommu, Linux-Arch, linux-crypto, Stephen Rothwell,
Michael Ellerman, James E . J . Bottomley, Helge Deller,
linux-parisc
On Wed, May 31, 2023 at 04:21:22PM +0200, Arnd Bergmann wrote:
> It would be nice to have the hack more localized to parisc
> and guarded with a CONFIG_GCC_VERSION check so we can kill
> it off in the future, once we drop either gcc-10 or parisc
> support.
I vote for dropping parisc -- it's the only 64bit arch that doesn't have
sane atomics.
Anyway, the below seems to work -- build tested with GCC-10.1
---
Subject: parisc/percpu: Work around the lack of __SIZEOF_INT128__
From: Peter Zijlstra <peterz@infradead.org>
Date: Tue May 30 22:27:40 CEST 2023
HPPA64 is unique in not providing __SIZEOF_INT128__ across all
supported compilers, specifically it only started doing this with
GCC-11.
Since the per-cpu ops are universally availably, and
this_cpu_{,try_}cmpxchg128() is expected to be available on all 64bit
architectures a wee bodge is in order.
Sadly, while C reverts to memcpy() for assignment of POD types, it does
not revert to memcmp() for for equality. Therefore frob that manually.
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
---
arch/parisc/include/asm/percpu.h | 77 +++++++++++++++++++++++++++++++++++++++
1 file changed, 77 insertions(+)
--- /dev/null
+++ b/arch/parisc/include/asm/percpu.h
@@ -0,0 +1,77 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef _ASM_PARISC_PERCPU_H
+#define _ASM_PARISC_PERCPU_H
+
+#include <linux/types.h>
+
+#if defined(CONFIG_64BIT) && CONFIG_GCC_VERSION < 1100000
+
+/*
+ * GCC prior to 11 does not provide __SIZEOF_INT128__ on HPPA64
+ * as such we need to provide an alternative implementation of
+ * {raw,this}_cpu_{,try_}cmpxchg128().
+ *
+ * This obviously doesn't function as u128 should, but for the purpose
+ * of per-cpu cmpxchg128 it might just do.
+ */
+typedef struct {
+ u64 a, b;
+} u128 __attribute__((aligned(16)));
+
+#define raw_cpu_generic_try_cmpxchg_memcmp(pcp, ovalp, nval) \
+({ \
+ typeof(pcp) *__p = raw_cpu_ptr(&(pcp)); \
+ typeof(pcp) __val = *__p, __old = *(ovalp); \
+ bool __ret; \
+ if (!__builtin_memcmp(&__val, &__old, sizeof(pcp))) { \
+ *__p = nval; \
+ __ret = true; \
+ } else { \
+ *(ovalp) = __val; \
+ __ret = false; \
+ } \
+ __ret; \
+})
+
+#define raw_cpu_generic_cmpxchg_memcmp(pcp, oval, nval) \
+({ \
+ typeof(pcp) __old = (oval); \
+ raw_cpu_generic_try_cmpxchg_memcpy(pcp, &__old, nval); \
+ __old; \
+})
+
+#define raw_cpu_cmpxchg128(pcp, oval, nval) \
+ raw_cpu_generic_cmpxchg_memcmp(pcp, oval, nval)
+#define raw_cpu_try_cmpxchg128(pcp, ovalp, nval) \
+ raw_cpu_generic_try_cmpxchg_memcmp(pcp, ovalp, nval)
+
+#define this_cpu_generic_try_cmpxchg_memcmp(pcp, ovalp, nval) \
+({ \
+ bool __ret; \
+ unsigned long __flags; \
+ raw_local_irq_save(__flags); \
+ __ret = raw_cpu_generic_try_cmpxchg_memcmp(pcp, ovalp, nval); \
+ raw_local_irq_restore(__flags); \
+ __ret; \
+})
+
+#define this_cpu_generic_cmpxchg_memcmp(pcp, oval, nval) \
+({ \
+ typeof(pcp) __ret; \
+ unsigned long __flags; \
+ raw_local_irq_save(__flags); \
+ __ret = raw_cpu_generic_cmpxchg_memcmp(pcp, oval, nval); \
+ raw_local_irq_restore(__flags); \
+ __ret; \
+})
+
+#define this_cpu_cmpxchg128(pcp, oval, nval) \
+ this_cpu_generic_cmpxchg_memcmp(pcp, oval, nval)
+#define this_cpu_try_cmpxchg128(pcp, ovalp, nval) \
+ this_cpu_generic_try_cmpxchg_memcmp(pcp, ovalp, nval)
+
+#endif /* !__SIZEOF_INT128__ */
+
+#include <asm-generic/percpu.h>
+
+#endif /* _ASM_PARISC_PERCPU_H */
^ permalink raw reply [flat|nested] 47+ messages in thread* Re: [PATCH v2 07/12] parisc/percpu: Work around the lack of __SIZEOF_INT128__
2023-06-01 10:14 ` [PATCH v2 07/12] parisc/percpu: Work around the lack of __SIZEOF_INT128__ Peter Zijlstra
@ 2023-06-01 10:32 ` Helge Deller
2023-06-01 10:50 ` Peter Zijlstra
2023-06-01 13:29 ` Linus Torvalds
0 siblings, 2 replies; 47+ messages in thread
From: Helge Deller @ 2023-06-01 10:32 UTC (permalink / raw)
To: Peter Zijlstra, Arnd Bergmann
Cc: Linus Torvalds, Jonathan Corbet, Will Deacon, Boqun Feng,
Mark Rutland, Catalin Marinas, dennis, Tejun Heo,
Christoph Lameter, Heiko Carstens, gor, Alexander Gordeev,
borntraeger, Sven Schnelle, Thomas Gleixner, Ingo Molnar,
Borislav Petkov, Dave Hansen, x86, H. Peter Anvin, Joerg Roedel,
suravee.suthikulpanit, Robin Murphy, David Woodhouse, Baolu Lu,
Herbert Xu, David S . Miller, Pekka Enberg, David Rientjes,
Joonsoo Kim, Andrew Morton, Vlastimil Babka, Roman Gushchin,
Hyeonggon Yoo, linux-doc, linux-kernel, linux-mm, linux-s390,
iommu, Linux-Arch, linux-crypto, Stephen Rothwell,
Michael Ellerman, James E . J . Bottomley, linux-parisc,
John David Anglin, Sam James
On 6/1/23 12:14, Peter Zijlstra wrote:
> On Wed, May 31, 2023 at 04:21:22PM +0200, Arnd Bergmann wrote:
>
>> It would be nice to have the hack more localized to parisc
>> and guarded with a CONFIG_GCC_VERSION check so we can kill
>> it off in the future, once we drop either gcc-10 or parisc
>> support.
>
> I vote for dropping parisc -- it's the only 64bit arch that doesn't have
> sane atomics.
Of course I'm against dropping parisc.
> Anyway, the below seems to work -- build tested with GCC-10.1
I don't think we need to care about gcc-10 on parisc.
Debian and Gentoo are the only supported distributions, while Debian
requires gcc-12 to build > 6.x kernels, and I assume Gentoo uses at least
gcc-12 as well.
So raising the gcc limit for parisc only (at least temporarily for now)
should be fine and your workaround below wouldn't be necessary, right?
Helge
> ---
> Subject: parisc/percpu: Work around the lack of __SIZEOF_INT128__
> From: Peter Zijlstra <peterz@infradead.org>
> Date: Tue May 30 22:27:40 CEST 2023
>
> HPPA64 is unique in not providing __SIZEOF_INT128__ across all
> supported compilers, specifically it only started doing this with
> GCC-11.
>
> Since the per-cpu ops are universally availably, and
> this_cpu_{,try_}cmpxchg128() is expected to be available on all 64bit
> architectures a wee bodge is in order.
>
> Sadly, while C reverts to memcpy() for assignment of POD types, it does
> not revert to memcmp() for for equality. Therefore frob that manually.
>
> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
> ---
> arch/parisc/include/asm/percpu.h | 77 +++++++++++++++++++++++++++++++++++++++
> 1 file changed, 77 insertions(+)
>
> --- /dev/null
> +++ b/arch/parisc/include/asm/percpu.h
> @@ -0,0 +1,77 @@
> +/* SPDX-License-Identifier: GPL-2.0 */
> +#ifndef _ASM_PARISC_PERCPU_H
> +#define _ASM_PARISC_PERCPU_H
> +
> +#include <linux/types.h>
> +
> +#if defined(CONFIG_64BIT) && CONFIG_GCC_VERSION < 1100000
> +
> +/*
> + * GCC prior to 11 does not provide __SIZEOF_INT128__ on HPPA64
> + * as such we need to provide an alternative implementation of
> + * {raw,this}_cpu_{,try_}cmpxchg128().
> + *
> + * This obviously doesn't function as u128 should, but for the purpose
> + * of per-cpu cmpxchg128 it might just do.
> + */
> +typedef struct {
> + u64 a, b;
> +} u128 __attribute__((aligned(16)));
> +
> +#define raw_cpu_generic_try_cmpxchg_memcmp(pcp, ovalp, nval) \
> +({ \
> + typeof(pcp) *__p = raw_cpu_ptr(&(pcp)); \
> + typeof(pcp) __val = *__p, __old = *(ovalp); \
> + bool __ret; \
> + if (!__builtin_memcmp(&__val, &__old, sizeof(pcp))) { \
> + *__p = nval; \
> + __ret = true; \
> + } else { \
> + *(ovalp) = __val; \
> + __ret = false; \
> + } \
> + __ret; \
> +})
> +
> +#define raw_cpu_generic_cmpxchg_memcmp(pcp, oval, nval) \
> +({ \
> + typeof(pcp) __old = (oval); \
> + raw_cpu_generic_try_cmpxchg_memcpy(pcp, &__old, nval); \
> + __old; \
> +})
> +
> +#define raw_cpu_cmpxchg128(pcp, oval, nval) \
> + raw_cpu_generic_cmpxchg_memcmp(pcp, oval, nval)
> +#define raw_cpu_try_cmpxchg128(pcp, ovalp, nval) \
> + raw_cpu_generic_try_cmpxchg_memcmp(pcp, ovalp, nval)
> +
> +#define this_cpu_generic_try_cmpxchg_memcmp(pcp, ovalp, nval) \
> +({ \
> + bool __ret; \
> + unsigned long __flags; \
> + raw_local_irq_save(__flags); \
> + __ret = raw_cpu_generic_try_cmpxchg_memcmp(pcp, ovalp, nval); \
> + raw_local_irq_restore(__flags); \
> + __ret; \
> +})
> +
> +#define this_cpu_generic_cmpxchg_memcmp(pcp, oval, nval) \
> +({ \
> + typeof(pcp) __ret; \
> + unsigned long __flags; \
> + raw_local_irq_save(__flags); \
> + __ret = raw_cpu_generic_cmpxchg_memcmp(pcp, oval, nval); \
> + raw_local_irq_restore(__flags); \
> + __ret; \
> +})
> +
> +#define this_cpu_cmpxchg128(pcp, oval, nval) \
> + this_cpu_generic_cmpxchg_memcmp(pcp, oval, nval)
> +#define this_cpu_try_cmpxchg128(pcp, ovalp, nval) \
> + this_cpu_generic_try_cmpxchg_memcmp(pcp, ovalp, nval)
> +
> +#endif /* !__SIZEOF_INT128__ */
> +
> +#include <asm-generic/percpu.h>
> +
> +#endif /* _ASM_PARISC_PERCPU_H */
^ permalink raw reply [flat|nested] 47+ messages in thread* Re: [PATCH v2 07/12] parisc/percpu: Work around the lack of __SIZEOF_INT128__
2023-06-01 10:32 ` Helge Deller
@ 2023-06-01 10:50 ` Peter Zijlstra
2023-06-01 21:08 ` Sam James
2023-06-01 13:29 ` Linus Torvalds
1 sibling, 1 reply; 47+ messages in thread
From: Peter Zijlstra @ 2023-06-01 10:50 UTC (permalink / raw)
To: Helge Deller
Cc: Arnd Bergmann, Linus Torvalds, Jonathan Corbet, Will Deacon,
Boqun Feng, Mark Rutland, Catalin Marinas, dennis, Tejun Heo,
Christoph Lameter, Heiko Carstens, gor, Alexander Gordeev,
borntraeger, Sven Schnelle, Thomas Gleixner, Ingo Molnar,
Borislav Petkov, Dave Hansen, x86, H. Peter Anvin, Joerg Roedel,
suravee.suthikulpanit, Robin Murphy, David Woodhouse, Baolu Lu,
Herbert Xu, David S . Miller, Pekka Enberg, David Rientjes,
Joonsoo Kim, Andrew Morton, Vlastimil Babka, Roman Gushchin,
Hyeonggon Yoo, linux-doc, linux-kernel, linux-mm, linux-s390,
iommu, Linux-Arch, linux-crypto, Stephen Rothwell,
Michael Ellerman, James E . J . Bottomley, linux-parisc,
John David Anglin, Sam James
On Thu, Jun 01, 2023 at 12:32:38PM +0200, Helge Deller wrote:
> On 6/1/23 12:14, Peter Zijlstra wrote:
> > On Wed, May 31, 2023 at 04:21:22PM +0200, Arnd Bergmann wrote:
> >
> > > It would be nice to have the hack more localized to parisc
> > > and guarded with a CONFIG_GCC_VERSION check so we can kill
> > > it off in the future, once we drop either gcc-10 or parisc
> > > support.
> >
> > I vote for dropping parisc -- it's the only 64bit arch that doesn't have
> > sane atomics.
>
> Of course I'm against dropping parisc.
:-)
> > Anyway, the below seems to work -- build tested with GCC-10.1
>
> I don't think we need to care about gcc-10 on parisc.
> Debian and Gentoo are the only supported distributions, while Debian
> requires gcc-12 to build > 6.x kernels, and I assume Gentoo uses at least
> gcc-12 as well.
>
> So raising the gcc limit for parisc only (at least temporarily for now)
> should be fine and your workaround below wouldn't be necessary, right?
Correct, if you're willing to set minimum GCC version to 11 for parisc
all is well and this patch can go play in the bit bucket.
^ permalink raw reply [flat|nested] 47+ messages in thread
* Re: [PATCH v2 07/12] parisc/percpu: Work around the lack of __SIZEOF_INT128__
2023-06-01 10:50 ` Peter Zijlstra
@ 2023-06-01 21:08 ` Sam James
0 siblings, 0 replies; 47+ messages in thread
From: Sam James @ 2023-06-01 21:08 UTC (permalink / raw)
To: Peter Zijlstra
Cc: Helge Deller, Arnd Bergmann, Linus Torvalds, Jonathan Corbet,
Will Deacon, Boqun Feng, Mark Rutland, Catalin Marinas, dennis,
Tejun Heo, Christoph Lameter, Heiko Carstens, gor,
Alexander Gordeev, borntraeger, Sven Schnelle, Thomas Gleixner,
Ingo Molnar, Borislav Petkov, Dave Hansen, x86, H. Peter Anvin,
Joerg Roedel, suravee.suthikulpanit, Robin Murphy,
David Woodhouse, Baolu Lu, Herbert Xu, David S . Miller,
Pekka Enberg, David Rientjes, Joonsoo Kim, Andrew Morton,
Vlastimil Babka, Roman Gushchin, Hyeonggon Yoo, linux-doc,
linux-kernel, linux-mm, linux-s390, iommu, Linux-Arch,
linux-crypto, Stephen Rothwell, Michael Ellerman,
James E . J . Bottomley, linux-parisc, John David Anglin
[-- Attachment #1: Type: text/plain, Size: 1255 bytes --]
Peter Zijlstra <peterz@infradead.org> writes:
> On Thu, Jun 01, 2023 at 12:32:38PM +0200, Helge Deller wrote:
>> On 6/1/23 12:14, Peter Zijlstra wrote:
>> > On Wed, May 31, 2023 at 04:21:22PM +0200, Arnd Bergmann wrote:
>> >
>> > > It would be nice to have the hack more localized to parisc
>> > > and guarded with a CONFIG_GCC_VERSION check so we can kill
>> > > it off in the future, once we drop either gcc-10 or parisc
>> > > support.
>> >
>> > I vote for dropping parisc -- it's the only 64bit arch that doesn't have
>> > sane atomics.
>>
>> Of course I'm against dropping parisc.
>
> :-)
>
>> > Anyway, the below seems to work -- build tested with GCC-10.1
>>
>> I don't think we need to care about gcc-10 on parisc.
>> Debian and Gentoo are the only supported distributions, while Debian
>> requires gcc-12 to build > 6.x kernels, and I assume Gentoo uses at least
>> gcc-12 as well.
>>
>> So raising the gcc limit for parisc only (at least temporarily for now)
>> should be fine and your workaround below wouldn't be necessary, right?
>
> Correct, if you're willing to set minimum GCC version to 11 for parisc
> all is well and this patch can go play in the bit bucket.
It's fine for us in Gentoo, thanks!
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 377 bytes --]
^ permalink raw reply [flat|nested] 47+ messages in thread
* Re: [PATCH v2 07/12] parisc/percpu: Work around the lack of __SIZEOF_INT128__
2023-06-01 10:32 ` Helge Deller
2023-06-01 10:50 ` Peter Zijlstra
@ 2023-06-01 13:29 ` Linus Torvalds
2023-06-01 17:38 ` Masahiro Yamada
2023-06-02 14:39 ` Peter Zijlstra
1 sibling, 2 replies; 47+ messages in thread
From: Linus Torvalds @ 2023-06-01 13:29 UTC (permalink / raw)
To: Helge Deller, Masahiro Yamada
Cc: Peter Zijlstra, Arnd Bergmann, Jonathan Corbet, Will Deacon,
Boqun Feng, Mark Rutland, Catalin Marinas, dennis, Tejun Heo,
Christoph Lameter, Heiko Carstens, gor, Alexander Gordeev,
borntraeger, Sven Schnelle, Thomas Gleixner, Ingo Molnar,
Borislav Petkov, Dave Hansen, x86, H. Peter Anvin, Joerg Roedel,
suravee.suthikulpanit, Robin Murphy, David Woodhouse, Baolu Lu,
Herbert Xu, David S . Miller, Pekka Enberg, David Rientjes,
Joonsoo Kim, Andrew Morton, Vlastimil Babka, Roman Gushchin,
Hyeonggon Yoo, linux-doc, linux-kernel, linux-mm, linux-s390,
iommu, Linux-Arch, linux-crypto, Stephen Rothwell,
Michael Ellerman, James E . J . Bottomley, linux-parisc,
John David Anglin, Sam James
On Thu, Jun 1, 2023 at 6:32 AM Helge Deller <deller@gmx.de> wrote:
>
> I don't think we need to care about gcc-10 on parisc.
> Debian and Gentoo are the only supported distributions, while Debian
> requires gcc-12 to build > 6.x kernels, and I assume Gentoo uses at least
> gcc-12 as well.
>
> So raising the gcc limit for parisc only (at least temporarily for now)
> should be fine and your workaround below wouldn't be necessary, right?
This absolutely sounds like the right option. Let's simplify the
problem space by just saying that parisc needs the newer compiler.
Right now we have that "minimum gcc version" in a somewhat annoying
place: it's in the ./scripts/min-tool-version.sh file as a shell
script.
I wonder if we could move the gcc minimum version check into the
Kconfig file instead, and make it easier to let architectures override
the minimum version.
I don't quite know how to do that sanely, though. I don't think we
have a sane way to error out at Kconfig time (except by forcing some
syntax error inside an 'if' statement or something horrendously hacky
like that).
Added Masahiro to the (already overlong) participants list.
Linus
^ permalink raw reply [flat|nested] 47+ messages in thread* Re: [PATCH v2 07/12] parisc/percpu: Work around the lack of __SIZEOF_INT128__
2023-06-01 13:29 ` Linus Torvalds
@ 2023-06-01 17:38 ` Masahiro Yamada
2023-06-02 14:39 ` Peter Zijlstra
1 sibling, 0 replies; 47+ messages in thread
From: Masahiro Yamada @ 2023-06-01 17:38 UTC (permalink / raw)
To: Linus Torvalds
Cc: Helge Deller, Peter Zijlstra, Arnd Bergmann, Jonathan Corbet,
Will Deacon, Boqun Feng, Mark Rutland, Catalin Marinas, dennis,
Tejun Heo, Christoph Lameter, Heiko Carstens, gor,
Alexander Gordeev, borntraeger, Sven Schnelle, Thomas Gleixner,
Ingo Molnar, Borislav Petkov, Dave Hansen, x86, H. Peter Anvin,
Joerg Roedel, suravee.suthikulpanit, Robin Murphy,
David Woodhouse, Baolu Lu, Herbert Xu, David S . Miller,
Pekka Enberg, David Rientjes, Joonsoo Kim, Andrew Morton,
Vlastimil Babka, Roman Gushchin, Hyeonggon Yoo, linux-doc,
linux-kernel, linux-mm, linux-s390, iommu, Linux-Arch,
linux-crypto, Stephen Rothwell, Michael Ellerman,
James E . J . Bottomley, linux-parisc, John David Anglin,
Sam James
On Thu, Jun 1, 2023 at 10:29 PM Linus Torvalds
<torvalds@linux-foundation.org> wrote:
>
> On Thu, Jun 1, 2023 at 6:32 AM Helge Deller <deller@gmx.de> wrote:
> >
> > I don't think we need to care about gcc-10 on parisc.
> > Debian and Gentoo are the only supported distributions, while Debian
> > requires gcc-12 to build > 6.x kernels, and I assume Gentoo uses at least
> > gcc-12 as well.
> >
> > So raising the gcc limit for parisc only (at least temporarily for now)
> > should be fine and your workaround below wouldn't be necessary, right?
>
> This absolutely sounds like the right option. Let's simplify the
> problem space by just saying that parisc needs the newer compiler.
>
> Right now we have that "minimum gcc version" in a somewhat annoying
> place: it's in the ./scripts/min-tool-version.sh file as a shell
> script.
>
> I wonder if we could move the gcc minimum version check into the
> Kconfig file instead, and make it easier to let architectures override
> the minimum version.
Currently, it is invoked in the Kconfig time,
but not directly in Kconfig files.
scripts/Kconfig.include
-> scripts/cc-version.sh
-> scripts/min-tool-version.sh
It would be ugly if we wrote the equivalent code
directly in Kconfig files.
>
> I don't quite know how to do that sanely, though. I don't think we
> have a sane way to error out at Kconfig time (except by forcing some
> syntax error inside an 'if' statement or something horrendously hacky
> like that).
The parse stage can fail by $(error-if ) macro, but
the evaluation stage never fails. I think it is a design.
I think checking the compiler version during the parse stage
makes sense given the current situation. The compiler version is
fixed when Kconfig starts. If the compiler is found to be too old,
there is no meaning to proceed.
You suggested to choose a compiler in the Kconfig time:
https://lore.kernel.org/lkml/CAHk-=whdrvCkSWh=BRrwZwNo3=yLBXXM88NGx8VEpP1VTgmkyQ@mail.gmail.com/
When we achieve that, moving the min version to Kconfig files will be
the right thing to do. Then, everything will be evaluated dynamically.
>
> Added Masahiro to the (already overlong) participants list.
>
> Linus
--
Best Regards
Masahiro Yamada
^ permalink raw reply [flat|nested] 47+ messages in thread* Re: [PATCH v2 07/12] parisc/percpu: Work around the lack of __SIZEOF_INT128__
2023-06-01 13:29 ` Linus Torvalds
2023-06-01 17:38 ` Masahiro Yamada
@ 2023-06-02 14:39 ` Peter Zijlstra
2023-06-02 14:50 ` Mark Rutland
` (3 more replies)
1 sibling, 4 replies; 47+ messages in thread
From: Peter Zijlstra @ 2023-06-02 14:39 UTC (permalink / raw)
To: Linus Torvalds
Cc: Helge Deller, Masahiro Yamada, Arnd Bergmann, Jonathan Corbet,
Will Deacon, Boqun Feng, Mark Rutland, Catalin Marinas, dennis,
Tejun Heo, Christoph Lameter, Heiko Carstens, gor,
Alexander Gordeev, borntraeger, Sven Schnelle, Thomas Gleixner,
Ingo Molnar, Borislav Petkov, Dave Hansen, x86, H. Peter Anvin,
Joerg Roedel, suravee.suthikulpanit, Robin Murphy,
David Woodhouse, Baolu Lu, Herbert Xu, David S . Miller,
Pekka Enberg, David Rientjes, Joonsoo Kim, Andrew Morton,
Vlastimil Babka, Roman Gushchin, Hyeonggon Yoo, linux-doc,
linux-kernel, linux-mm, linux-s390, iommu, Linux-Arch,
linux-crypto, Stephen Rothwell, Michael Ellerman,
James E . J . Bottomley, linux-parisc, John David Anglin,
Sam James
On Thu, Jun 01, 2023 at 09:29:18AM -0400, Linus Torvalds wrote:
> Right now we have that "minimum gcc version" in a somewhat annoying
> place: it's in the ./scripts/min-tool-version.sh file as a shell
> script.
Something like so then?
---
Subject: parisc: Raise minimal GCC version
From: Peter Zijlstra <peterz@infradead.org>
Date: Fri Jun 2 16:33:54 CEST 2023
With 64bit builds depending on __SIZEOF_INT128__ raise the parisc
minimum compiler version to gcc-11.0.0.
All other 64bit architectures provide this from GCC-5.1.0 (and
probably before), except hppa64 which only started advertising this
with GCC-11.
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
---
scripts/min-tool-version.sh | 6 +++++-
1 file changed, 5 insertions(+), 1 deletion(-)
--- a/scripts/min-tool-version.sh
+++ b/scripts/min-tool-version.sh
@@ -17,7 +17,11 @@ binutils)
echo 2.25.0
;;
gcc)
- echo 5.1.0
+ if [ "$SRCARCH" = parisc ]; then
+ echo 11.0.0
+ else
+ echo 5.1.0
+ fi
;;
llvm)
if [ "$SRCARCH" = s390 ]; then
^ permalink raw reply [flat|nested] 47+ messages in thread* Re: [PATCH v2 07/12] parisc/percpu: Work around the lack of __SIZEOF_INT128__
2023-06-02 14:39 ` Peter Zijlstra
@ 2023-06-02 14:50 ` Mark Rutland
2023-06-02 16:09 ` Linus Torvalds
` (2 subsequent siblings)
3 siblings, 0 replies; 47+ messages in thread
From: Mark Rutland @ 2023-06-02 14:50 UTC (permalink / raw)
To: Peter Zijlstra
Cc: Linus Torvalds, Helge Deller, Masahiro Yamada, Arnd Bergmann,
Jonathan Corbet, Will Deacon, Boqun Feng, Catalin Marinas,
dennis, Tejun Heo, Christoph Lameter, Heiko Carstens, gor,
Alexander Gordeev, borntraeger, Sven Schnelle, Thomas Gleixner,
Ingo Molnar, Borislav Petkov, Dave Hansen, x86, H. Peter Anvin,
Joerg Roedel, suravee.suthikulpanit, Robin Murphy,
David Woodhouse, Baolu Lu, Herbert Xu, David S . Miller,
Pekka Enberg, David Rientjes, Joonsoo Kim, Andrew Morton,
Vlastimil Babka, Roman Gushchin, Hyeonggon Yoo, linux-doc,
linux-kernel, linux-mm, linux-s390, iommu, Linux-Arch,
linux-crypto, Stephen Rothwell, Michael Ellerman,
James E . J . Bottomley, linux-parisc, John David Anglin,
Sam James
On Fri, Jun 02, 2023 at 04:39:12PM +0200, Peter Zijlstra wrote:
> On Thu, Jun 01, 2023 at 09:29:18AM -0400, Linus Torvalds wrote:
>
> > Right now we have that "minimum gcc version" in a somewhat annoying
> > place: it's in the ./scripts/min-tool-version.sh file as a shell
> > script.
>
> Something like so then?
>
> ---
> Subject: parisc: Raise minimal GCC version
> From: Peter Zijlstra <peterz@infradead.org>
> Date: Fri Jun 2 16:33:54 CEST 2023
>
> With 64bit builds depending on __SIZEOF_INT128__ raise the parisc
> minimum compiler version to gcc-11.0.0.
>
> All other 64bit architectures provide this from GCC-5.1.0 (and
> probably before), except hppa64 which only started advertising this
> with GCC-11.
>
> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
> ---
> scripts/min-tool-version.sh | 6 +++++-
> 1 file changed, 5 insertions(+), 1 deletion(-)
>
> --- a/scripts/min-tool-version.sh
> +++ b/scripts/min-tool-version.sh
> @@ -17,7 +17,11 @@ binutils)
> echo 2.25.0
> ;;
> gcc)
> - echo 5.1.0
> + if [ "$SRCARCH" = parisc ]; then
> + echo 11.0.0
> + else
> + echo 5.1.0
> + fi
> ;;
> llvm)
> if [ "$SRCARCH" = s390 ]; then
I gave this a spin and it looks good to me:
[mark@lakrids:~/src/linux]% usekorg 10.3.0 make ARCH=arm64 CROSS_COMPILE=aarch64-linux- defconfig
HOSTCC scripts/basic/fixdep
HOSTCC scripts/kconfig/conf.o
HOSTCC scripts/kconfig/confdata.o
HOSTCC scripts/kconfig/expr.o
LEX scripts/kconfig/lexer.lex.c
YACC scripts/kconfig/parser.tab.[ch]
HOSTCC scripts/kconfig/lexer.lex.o
HOSTCC scripts/kconfig/menu.o
HOSTCC scripts/kconfig/parser.tab.o
HOSTCC scripts/kconfig/preprocess.o
HOSTCC scripts/kconfig/symbol.o
HOSTCC scripts/kconfig/util.o
HOSTLD scripts/kconfig/conf
*** Default configuration is based on 'defconfig'
#
# configuration written to .config
#
[mark@lakrids:~/src/linux]% usekorg 10.3.0 make ARCH=parisc CROSS_COMPILE=hppa64-linux- generic-64bit_defconfig
***
*** C compiler is too old.
*** Your GCC version: 10.3.0
*** Minimum GCC version: 11.0.0
***
scripts/Kconfig.include:44: Sorry, this C compiler is not supported.
make[1]: *** [scripts/kconfig/Makefile:94: generic-64bit_defconfig] Error 1
make: *** [Makefile:692: generic-64bit_defconfig] Error 2
[mark@lakrids:~/src/linux]% usekorg 11.3.0 make ARCH=parisc CROSS_COMPILE=hppa64-linux- generic-64bit_defconfig
#
# configuration written to .config
#
FWIW:
Tested-by: Mark Rutland <mark.rutland@arm.com>
Mark.
^ permalink raw reply [flat|nested] 47+ messages in thread* Re: [PATCH v2 07/12] parisc/percpu: Work around the lack of __SIZEOF_INT128__
2023-06-02 14:39 ` Peter Zijlstra
2023-06-02 14:50 ` Mark Rutland
@ 2023-06-02 16:09 ` Linus Torvalds
2023-06-02 17:00 ` H. Peter Anvin
2023-06-02 20:42 ` Helge Deller
3 siblings, 0 replies; 47+ messages in thread
From: Linus Torvalds @ 2023-06-02 16:09 UTC (permalink / raw)
To: Peter Zijlstra
Cc: Helge Deller, Masahiro Yamada, Arnd Bergmann, Jonathan Corbet,
Will Deacon, Boqun Feng, Mark Rutland, Catalin Marinas, dennis,
Tejun Heo, Christoph Lameter, Heiko Carstens, gor,
Alexander Gordeev, borntraeger, Sven Schnelle, Thomas Gleixner,
Ingo Molnar, Borislav Petkov, Dave Hansen, x86, H. Peter Anvin,
Joerg Roedel, suravee.suthikulpanit, Robin Murphy,
David Woodhouse, Baolu Lu, Herbert Xu, David S . Miller,
Pekka Enberg, David Rientjes, Joonsoo Kim, Andrew Morton,
Vlastimil Babka, Roman Gushchin, Hyeonggon Yoo, linux-doc,
linux-kernel, linux-mm, linux-s390, iommu, Linux-Arch,
linux-crypto, Stephen Rothwell, Michael Ellerman,
James E . J . Bottomley, linux-parisc, John David Anglin,
Sam James
On Fri, Jun 2, 2023 at 10:40 AM Peter Zijlstra <peterz@infradead.org> wrote:
>
> Something like so then?
Ack. I think it would be much cleaner if we would have it as part of
the Kconfig file and architectures could just override some
GCC_MIN_VERSION value, but that's not the universe we currently have,
so your patch looks like the best thing to do.
Linus
^ permalink raw reply [flat|nested] 47+ messages in thread* Re: [PATCH v2 07/12] parisc/percpu: Work around the lack of __SIZEOF_INT128__
2023-06-02 14:39 ` Peter Zijlstra
2023-06-02 14:50 ` Mark Rutland
2023-06-02 16:09 ` Linus Torvalds
@ 2023-06-02 17:00 ` H. Peter Anvin
2023-06-02 19:10 ` Peter Zijlstra
2023-06-02 20:42 ` Helge Deller
3 siblings, 1 reply; 47+ messages in thread
From: H. Peter Anvin @ 2023-06-02 17:00 UTC (permalink / raw)
To: Peter Zijlstra, Linus Torvalds
Cc: Helge Deller, Masahiro Yamada, Arnd Bergmann, Jonathan Corbet,
Will Deacon, Boqun Feng, Mark Rutland, Catalin Marinas, dennis,
Tejun Heo, Christoph Lameter, Heiko Carstens, gor,
Alexander Gordeev, borntraeger, Sven Schnelle, Thomas Gleixner,
Ingo Molnar, Borislav Petkov, Dave Hansen, x86, Joerg Roedel,
suravee.suthikulpanit, Robin Murphy, David Woodhouse, Baolu Lu,
Herbert Xu, David S . Miller, Pekka Enberg, David Rientjes,
Joonsoo Kim, Andrew Morton, Vlastimil Babka, Roman Gushchin,
Hyeonggon Yoo, linux-doc, linux-kernel, linux-mm, linux-s390,
iommu, Linux-Arch, linux-crypto, Stephen Rothwell,
Michael Ellerman, James E . J . Bottomley, linux-parisc,
John David Anglin, Sam James
On June 2, 2023 7:39:12 AM PDT, Peter Zijlstra <peterz@infradead.org> wrote:
>On Thu, Jun 01, 2023 at 09:29:18AM -0400, Linus Torvalds wrote:
>
>> Right now we have that "minimum gcc version" in a somewhat annoying
>> place: it's in the ./scripts/min-tool-version.sh file as a shell
>> script.
>
>Something like so then?
>
>---
>Subject: parisc: Raise minimal GCC version
>From: Peter Zijlstra <peterz@infradead.org>
>Date: Fri Jun 2 16:33:54 CEST 2023
>
>With 64bit builds depending on __SIZEOF_INT128__ raise the parisc
>minimum compiler version to gcc-11.0.0.
>
>All other 64bit architectures provide this from GCC-5.1.0 (and
>probably before), except hppa64 which only started advertising this
>with GCC-11.
>
>Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
>---
> scripts/min-tool-version.sh | 6 +++++-
> 1 file changed, 5 insertions(+), 1 deletion(-)
>
>--- a/scripts/min-tool-version.sh
>+++ b/scripts/min-tool-version.sh
>@@ -17,7 +17,11 @@ binutils)
> echo 2.25.0
> ;;
> gcc)
>- echo 5.1.0
>+ if [ "$SRCARCH" = parisc ]; then
>+ echo 11.0.0
>+ else
>+ echo 5.1.0
>+ fi
> ;;
> llvm)
> if [ "$SRCARCH" = s390 ]; then
Dumb question: is this only about the cpp macro or is it about __int128 existing at all?
^ permalink raw reply [flat|nested] 47+ messages in thread
* Re: [PATCH v2 07/12] parisc/percpu: Work around the lack of __SIZEOF_INT128__
2023-06-02 17:00 ` H. Peter Anvin
@ 2023-06-02 19:10 ` Peter Zijlstra
2023-06-02 19:20 ` H. Peter Anvin
0 siblings, 1 reply; 47+ messages in thread
From: Peter Zijlstra @ 2023-06-02 19:10 UTC (permalink / raw)
To: H. Peter Anvin
Cc: Linus Torvalds, Helge Deller, Masahiro Yamada, Arnd Bergmann,
Jonathan Corbet, Will Deacon, Boqun Feng, Mark Rutland,
Catalin Marinas, dennis, Tejun Heo, Christoph Lameter,
Heiko Carstens, gor, Alexander Gordeev, borntraeger,
Sven Schnelle, Thomas Gleixner, Ingo Molnar, Borislav Petkov,
Dave Hansen, x86, Joerg Roedel, suravee.suthikulpanit,
Robin Murphy, David Woodhouse, Baolu Lu, Herbert Xu,
David S . Miller, Pekka Enberg, David Rientjes, Joonsoo Kim,
Andrew Morton, Vlastimil Babka, Roman Gushchin, Hyeonggon Yoo,
linux-doc, linux-kernel, linux-mm, linux-s390, iommu, Linux-Arch,
linux-crypto, Stephen Rothwell, Michael Ellerman,
James E . J . Bottomley, linux-parisc, John David Anglin,
Sam James
On Fri, Jun 02, 2023 at 10:00:17AM -0700, H. Peter Anvin wrote:
> Dumb question: is this only about the cpp macro or is it about __int128 existing at all?
It's mostly about __int128 being there, __SIZEOF_INT128__ is just the
way we go about detecting if it's there or not.
^ permalink raw reply [flat|nested] 47+ messages in thread
* Re: [PATCH v2 07/12] parisc/percpu: Work around the lack of __SIZEOF_INT128__
2023-06-02 19:10 ` Peter Zijlstra
@ 2023-06-02 19:20 ` H. Peter Anvin
2023-06-02 19:40 ` Peter Zijlstra
0 siblings, 1 reply; 47+ messages in thread
From: H. Peter Anvin @ 2023-06-02 19:20 UTC (permalink / raw)
To: Peter Zijlstra
Cc: Linus Torvalds, Helge Deller, Masahiro Yamada, Arnd Bergmann,
Jonathan Corbet, Will Deacon, Boqun Feng, Mark Rutland,
Catalin Marinas, dennis, Tejun Heo, Christoph Lameter,
Heiko Carstens, gor, Alexander Gordeev, borntraeger,
Sven Schnelle, Thomas Gleixner, Ingo Molnar, Borislav Petkov,
Dave Hansen, x86, Joerg Roedel, suravee.suthikulpanit,
Robin Murphy, David Woodhouse, Baolu Lu, Herbert Xu,
David S . Miller, Pekka Enberg, David Rientjes, Joonsoo Kim,
Andrew Morton, Vlastimil Babka, Roman Gushchin, Hyeonggon Yoo,
linux-doc, linux-kernel, linux-mm, linux-s390, iommu, Linux-Arch,
linux-crypto, Stephen Rothwell, Michael Ellerman,
James E . J . Bottomley, linux-parisc, John David Anglin,
Sam James
Ok. So the patch description needs to be fixed. Otherwise the solution would be far simpler :)
^ permalink raw reply [flat|nested] 47+ messages in thread
* Re: [PATCH v2 07/12] parisc/percpu: Work around the lack of __SIZEOF_INT128__
2023-06-02 19:20 ` H. Peter Anvin
@ 2023-06-02 19:40 ` Peter Zijlstra
2023-06-02 19:57 ` Linus Torvalds
0 siblings, 1 reply; 47+ messages in thread
From: Peter Zijlstra @ 2023-06-02 19:40 UTC (permalink / raw)
To: H. Peter Anvin
Cc: Linus Torvalds, Helge Deller, Masahiro Yamada, Arnd Bergmann,
Jonathan Corbet, Will Deacon, Boqun Feng, Mark Rutland,
Catalin Marinas, dennis, Tejun Heo, Christoph Lameter,
Heiko Carstens, gor, Alexander Gordeev, borntraeger,
Sven Schnelle, Thomas Gleixner, Ingo Molnar, Borislav Petkov,
Dave Hansen, x86, Joerg Roedel, suravee.suthikulpanit,
Robin Murphy, David Woodhouse, Baolu Lu, Herbert Xu,
David S . Miller, Pekka Enberg, David Rientjes, Joonsoo Kim,
Andrew Morton, Vlastimil Babka, Roman Gushchin, Hyeonggon Yoo,
linux-doc, linux-kernel, linux-mm, linux-s390, iommu, Linux-Arch,
linux-crypto, Stephen Rothwell, Michael Ellerman,
James E . J . Bottomley, linux-parisc, John David Anglin,
Sam James
On Fri, Jun 02, 2023 at 12:20:05PM -0700, H. Peter Anvin wrote:
> Ok. So the patch description needs to be fixed. Otherwise the solution would be far simpler :)
"With 64bit builds depending on __SIZEOF_INT128__ to detect the
presence of __int128 raise the parisc minimum compiler version to
gcc-11.0.0."
better?
^ permalink raw reply [flat|nested] 47+ messages in thread
* Re: [PATCH v2 07/12] parisc/percpu: Work around the lack of __SIZEOF_INT128__
2023-06-02 19:40 ` Peter Zijlstra
@ 2023-06-02 19:57 ` Linus Torvalds
0 siblings, 0 replies; 47+ messages in thread
From: Linus Torvalds @ 2023-06-02 19:57 UTC (permalink / raw)
To: Peter Zijlstra
Cc: H. Peter Anvin, Helge Deller, Masahiro Yamada, Arnd Bergmann,
Jonathan Corbet, Will Deacon, Boqun Feng, Mark Rutland,
Catalin Marinas, dennis, Tejun Heo, Christoph Lameter,
Heiko Carstens, gor, Alexander Gordeev, borntraeger,
Sven Schnelle, Thomas Gleixner, Ingo Molnar, Borislav Petkov,
Dave Hansen, x86, Joerg Roedel, suravee.suthikulpanit,
Robin Murphy, David Woodhouse, Baolu Lu, Herbert Xu,
David S . Miller, Pekka Enberg, David Rientjes, Joonsoo Kim,
Andrew Morton, Vlastimil Babka, Roman Gushchin, Hyeonggon Yoo,
linux-doc, linux-kernel, linux-mm, linux-s390, iommu, Linux-Arch,
linux-crypto, Stephen Rothwell, Michael Ellerman,
James E . J . Bottomley, linux-parisc, John David Anglin,
Sam James
On Fri, Jun 2, 2023 at 3:40 PM Peter Zijlstra <peterz@infradead.org> wrote:
>
> "With 64bit builds depending on __SIZEOF_INT128__ to detect the
> presence of __int128 raise the parisc minimum compiler version to
> gcc-11.0.0."
>
> better?
I'd just say "64-bit targets need the __int128 type, which for pa-risc
means raising the minimum gcc version to 11".
The __SIZEOF_INT128__ part isn't the important part. That's just the symptom.
Linus
^ permalink raw reply [flat|nested] 47+ messages in thread
* Re: [PATCH v2 07/12] parisc/percpu: Work around the lack of __SIZEOF_INT128__
2023-06-02 14:39 ` Peter Zijlstra
` (2 preceding siblings ...)
2023-06-02 17:00 ` H. Peter Anvin
@ 2023-06-02 20:42 ` Helge Deller
3 siblings, 0 replies; 47+ messages in thread
From: Helge Deller @ 2023-06-02 20:42 UTC (permalink / raw)
To: Peter Zijlstra, Linus Torvalds
Cc: Masahiro Yamada, Arnd Bergmann, Jonathan Corbet, Will Deacon,
Boqun Feng, Mark Rutland, Catalin Marinas, dennis, Tejun Heo,
Christoph Lameter, Heiko Carstens, gor, Alexander Gordeev,
borntraeger, Sven Schnelle, Thomas Gleixner, Ingo Molnar,
Borislav Petkov, Dave Hansen, x86, H. Peter Anvin, Joerg Roedel,
suravee.suthikulpanit, Robin Murphy, David Woodhouse, Baolu Lu,
Herbert Xu, David S . Miller, Pekka Enberg, David Rientjes,
Joonsoo Kim, Andrew Morton, Vlastimil Babka, Roman Gushchin,
Hyeonggon Yoo, linux-doc, linux-kernel, linux-mm, linux-s390,
iommu, Linux-Arch, linux-crypto, Stephen Rothwell,
Michael Ellerman, James E . J . Bottomley, linux-parisc,
John David Anglin, Sam James
On 6/2/23 16:39, Peter Zijlstra wrote:
> On Thu, Jun 01, 2023 at 09:29:18AM -0400, Linus Torvalds wrote:
>
>> Right now we have that "minimum gcc version" in a somewhat annoying
>> place: it's in the ./scripts/min-tool-version.sh file as a shell
>> script.
>
> Something like so then?
>
> ---
> Subject: parisc: Raise minimal GCC version
> From: Peter Zijlstra <peterz@infradead.org>
> Date: Fri Jun 2 16:33:54 CEST 2023
>
> With 64bit builds depending on __SIZEOF_INT128__ raise the parisc
> minimum compiler version to gcc-11.0.0.
>
> All other 64bit architectures provide this from GCC-5.1.0 (and
> probably before), except hppa64 which only started advertising this
> with GCC-11.
>
> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
The patch raises the compiler for 32- and 64-bit parisc builds, but that's OK.
So:
Acked-by: Helge Deller <deller@gmx.de>
Thank you!
Helge
> ---
> scripts/min-tool-version.sh | 6 +++++-
> 1 file changed, 5 insertions(+), 1 deletion(-)
>
> --- a/scripts/min-tool-version.sh
> +++ b/scripts/min-tool-version.sh
> @@ -17,7 +17,11 @@ binutils)
> echo 2.25.0
> ;;
> gcc)
> - echo 5.1.0
> + if [ "$SRCARCH" = parisc ]; then
> + echo 11.0.0
> + else
> + echo 5.1.0
> + fi
> ;;
> llvm)
> if [ "$SRCARCH" = s390 ]; then
^ permalink raw reply [flat|nested] 47+ messages in thread
* [PATCH 08/12] x86,amd_iommu: Replace cmpxchg_double()
2023-05-31 13:08 [PATCH 00/12] Introduce cmpxchg128() -- aka. the demise of cmpxchg_double() Peter Zijlstra
` (6 preceding siblings ...)
2023-05-31 13:08 ` [PATCH 07/12] percpu: #ifndef __SIZEOF_INT128__ Peter Zijlstra
@ 2023-05-31 13:08 ` Peter Zijlstra
2023-06-05 7:42 ` [tip: locking/core] " tip-bot2 for Peter Zijlstra
2023-05-31 13:08 ` [PATCH 09/12] x86,intel_iommu: " Peter Zijlstra
` (4 subsequent siblings)
12 siblings, 1 reply; 47+ messages in thread
From: Peter Zijlstra @ 2023-05-31 13:08 UTC (permalink / raw)
To: torvalds
Cc: corbet, will, peterz, boqun.feng, mark.rutland, catalin.marinas,
dennis, tj, cl, hca, gor, agordeev, borntraeger, svens, tglx,
mingo, bp, dave.hansen, x86, hpa, joro, suravee.suthikulpanit,
robin.murphy, dwmw2, baolu.lu, Arnd Bergmann, Herbert Xu, davem,
penberg, rientjes, iamjoonsoo.kim, Andrew Morton, vbabka,
roman.gushchin, 42.hyeyoo, linux-doc, linux-kernel, linux-mm,
linux-s390, iommu, linux-arch, linux-crypto, sfr, mpe,
James.Bottomley, deller, linux-parisc, Vasant Hegde
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Reviewed-by: Arnd Bergmann <arnd@arndb.de>
Tested-by: Vasant Hegde <vasant.hegde@amd.com>
---
drivers/iommu/amd/amd_iommu_types.h | 9 +++++++--
drivers/iommu/amd/iommu.c | 10 ++++------
2 files changed, 11 insertions(+), 8 deletions(-)
--- a/drivers/iommu/amd/amd_iommu_types.h
+++ b/drivers/iommu/amd/amd_iommu_types.h
@@ -986,8 +986,13 @@ union irte_ga_hi {
};
struct irte_ga {
- union irte_ga_lo lo;
- union irte_ga_hi hi;
+ union {
+ struct {
+ union irte_ga_lo lo;
+ union irte_ga_hi hi;
+ };
+ u128 irte;
+ };
};
struct irq_2_irte {
--- a/drivers/iommu/amd/iommu.c
+++ b/drivers/iommu/amd/iommu.c
@@ -3003,10 +3003,10 @@ static int alloc_irq_index(struct amd_io
static int modify_irte_ga(struct amd_iommu *iommu, u16 devid, int index,
struct irte_ga *irte, struct amd_ir_data *data)
{
- bool ret;
struct irq_remap_table *table;
- unsigned long flags;
struct irte_ga *entry;
+ unsigned long flags;
+ u128 old;
table = get_irq_table(iommu, devid);
if (!table)
@@ -3017,16 +3017,14 @@ static int modify_irte_ga(struct amd_iom
entry = (struct irte_ga *)table->table;
entry = &entry[index];
- ret = cmpxchg_double(&entry->lo.val, &entry->hi.val,
- entry->lo.val, entry->hi.val,
- irte->lo.val, irte->hi.val);
/*
* We use cmpxchg16 to atomically update the 128-bit IRTE,
* and it cannot be updated by the hardware or other processors
* behind us, so the return value of cmpxchg16 should be the
* same as the old value.
*/
- WARN_ON(!ret);
+ old = entry->irte;
+ WARN_ON(!try_cmpxchg128(&entry->irte, &old, irte->irte));
if (data)
data->ref = entry;
^ permalink raw reply [flat|nested] 47+ messages in thread* [tip: locking/core] x86,amd_iommu: Replace cmpxchg_double()
2023-05-31 13:08 ` [PATCH 08/12] x86,amd_iommu: Replace cmpxchg_double() Peter Zijlstra
@ 2023-06-05 7:42 ` tip-bot2 for Peter Zijlstra
0 siblings, 0 replies; 47+ messages in thread
From: tip-bot2 for Peter Zijlstra @ 2023-06-05 7:42 UTC (permalink / raw)
To: linux-tip-commits
Cc: Peter Zijlstra (Intel),
Arnd Bergmann, Mark Rutland, Vasant Hegde, x86, linux-kernel
The following commit has been merged into the locking/core branch of tip:
Commit-ID: 0a0a6800b02489c1288f963ad290b6a4876a2cc8
Gitweb: https://git.kernel.org/tip/0a0a6800b02489c1288f963ad290b6a4876a2cc8
Author: Peter Zijlstra <peterz@infradead.org>
AuthorDate: Wed, 31 May 2023 15:08:41 +02:00
Committer: Peter Zijlstra <peterz@infradead.org>
CommitterDate: Mon, 05 Jun 2023 09:36:38 +02:00
x86,amd_iommu: Replace cmpxchg_double()
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Reviewed-by: Arnd Bergmann <arnd@arndb.de>
Reviewed-by: Mark Rutland <mark.rutland@arm.com>
Tested-by: Vasant Hegde <vasant.hegde@amd.com>
Tested-by: Mark Rutland <mark.rutland@arm.com>
Link: https://lore.kernel.org/r/20230531132323.788955257@infradead.org
---
drivers/iommu/amd/amd_iommu_types.h | 9 +++++++--
drivers/iommu/amd/iommu.c | 10 ++++------
2 files changed, 11 insertions(+), 8 deletions(-)
diff --git a/drivers/iommu/amd/amd_iommu_types.h b/drivers/iommu/amd/amd_iommu_types.h
index 2ddbda3..ab8aa8f 100644
--- a/drivers/iommu/amd/amd_iommu_types.h
+++ b/drivers/iommu/amd/amd_iommu_types.h
@@ -986,8 +986,13 @@ union irte_ga_hi {
};
struct irte_ga {
- union irte_ga_lo lo;
- union irte_ga_hi hi;
+ union {
+ struct {
+ union irte_ga_lo lo;
+ union irte_ga_hi hi;
+ };
+ u128 irte;
+ };
};
struct irq_2_irte {
diff --git a/drivers/iommu/amd/iommu.c b/drivers/iommu/amd/iommu.c
index 4a31464..1e9f85e 100644
--- a/drivers/iommu/amd/iommu.c
+++ b/drivers/iommu/amd/iommu.c
@@ -3003,10 +3003,10 @@ out:
static int modify_irte_ga(struct amd_iommu *iommu, u16 devid, int index,
struct irte_ga *irte, struct amd_ir_data *data)
{
- bool ret;
struct irq_remap_table *table;
- unsigned long flags;
struct irte_ga *entry;
+ unsigned long flags;
+ u128 old;
table = get_irq_table(iommu, devid);
if (!table)
@@ -3017,16 +3017,14 @@ static int modify_irte_ga(struct amd_iommu *iommu, u16 devid, int index,
entry = (struct irte_ga *)table->table;
entry = &entry[index];
- ret = cmpxchg_double(&entry->lo.val, &entry->hi.val,
- entry->lo.val, entry->hi.val,
- irte->lo.val, irte->hi.val);
/*
* We use cmpxchg16 to atomically update the 128-bit IRTE,
* and it cannot be updated by the hardware or other processors
* behind us, so the return value of cmpxchg16 should be the
* same as the old value.
*/
- WARN_ON(!ret);
+ old = entry->irte;
+ WARN_ON(!try_cmpxchg128(&entry->irte, &old, irte->irte));
if (data)
data->ref = entry;
^ permalink raw reply [flat|nested] 47+ messages in thread
* [PATCH 09/12] x86,intel_iommu: Replace cmpxchg_double()
2023-05-31 13:08 [PATCH 00/12] Introduce cmpxchg128() -- aka. the demise of cmpxchg_double() Peter Zijlstra
` (7 preceding siblings ...)
2023-05-31 13:08 ` [PATCH 08/12] x86,amd_iommu: Replace cmpxchg_double() Peter Zijlstra
@ 2023-05-31 13:08 ` Peter Zijlstra
2023-06-05 7:42 ` [tip: locking/core] " tip-bot2 for Peter Zijlstra
2023-05-31 13:08 ` [PATCH 10/12] slub: " Peter Zijlstra
` (3 subsequent siblings)
12 siblings, 1 reply; 47+ messages in thread
From: Peter Zijlstra @ 2023-05-31 13:08 UTC (permalink / raw)
To: torvalds
Cc: corbet, will, peterz, boqun.feng, mark.rutland, catalin.marinas,
dennis, tj, cl, hca, gor, agordeev, borntraeger, svens, tglx,
mingo, bp, dave.hansen, x86, hpa, joro, suravee.suthikulpanit,
robin.murphy, dwmw2, baolu.lu, Arnd Bergmann, Herbert Xu, davem,
penberg, rientjes, iamjoonsoo.kim, Andrew Morton, vbabka,
roman.gushchin, 42.hyeyoo, linux-doc, linux-kernel, linux-mm,
linux-s390, iommu, linux-arch, linux-crypto, sfr, mpe,
James.Bottomley, deller, linux-parisc
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Reviewed-by: Lu Baolu <baolu.lu@linux.intel.com>
Reviewed-by: Arnd Bergmann <arnd@arndb.de>
---
drivers/iommu/intel/irq_remapping.c | 8 --
include/linux/dmar.h | 125 +++++++++++++++++++-----------------
2 files changed, 68 insertions(+), 65 deletions(-)
--- a/drivers/iommu/intel/irq_remapping.c
+++ b/drivers/iommu/intel/irq_remapping.c
@@ -175,18 +175,14 @@ static int modify_irte(struct irq_2_iomm
irte = &iommu->ir_table->base[index];
if ((irte->pst == 1) || (irte_modified->pst == 1)) {
- bool ret;
-
- ret = cmpxchg_double(&irte->low, &irte->high,
- irte->low, irte->high,
- irte_modified->low, irte_modified->high);
/*
* We use cmpxchg16 to atomically update the 128-bit IRTE,
* and it cannot be updated by the hardware or other processors
* behind us, so the return value of cmpxchg16 should be the
* same as the old value.
*/
- WARN_ON(!ret);
+ u128 old = irte->irte;
+ WARN_ON(!try_cmpxchg128(&irte->irte, &old, irte_modified->irte));
} else {
WRITE_ONCE(irte->low, irte_modified->low);
WRITE_ONCE(irte->high, irte_modified->high);
--- a/include/linux/dmar.h
+++ b/include/linux/dmar.h
@@ -202,67 +202,74 @@ static inline void detect_intel_iommu(vo
struct irte {
union {
- /* Shared between remapped and posted mode*/
struct {
- __u64 present : 1, /* 0 */
- fpd : 1, /* 1 */
- __res0 : 6, /* 2 - 6 */
- avail : 4, /* 8 - 11 */
- __res1 : 3, /* 12 - 14 */
- pst : 1, /* 15 */
- vector : 8, /* 16 - 23 */
- __res2 : 40; /* 24 - 63 */
+ union {
+ /* Shared between remapped and posted mode*/
+ struct {
+ __u64 present : 1, /* 0 */
+ fpd : 1, /* 1 */
+ __res0 : 6, /* 2 - 6 */
+ avail : 4, /* 8 - 11 */
+ __res1 : 3, /* 12 - 14 */
+ pst : 1, /* 15 */
+ vector : 8, /* 16 - 23 */
+ __res2 : 40; /* 24 - 63 */
+ };
+
+ /* Remapped mode */
+ struct {
+ __u64 r_present : 1, /* 0 */
+ r_fpd : 1, /* 1 */
+ dst_mode : 1, /* 2 */
+ redir_hint : 1, /* 3 */
+ trigger_mode : 1, /* 4 */
+ dlvry_mode : 3, /* 5 - 7 */
+ r_avail : 4, /* 8 - 11 */
+ r_res0 : 4, /* 12 - 15 */
+ r_vector : 8, /* 16 - 23 */
+ r_res1 : 8, /* 24 - 31 */
+ dest_id : 32; /* 32 - 63 */
+ };
+
+ /* Posted mode */
+ struct {
+ __u64 p_present : 1, /* 0 */
+ p_fpd : 1, /* 1 */
+ p_res0 : 6, /* 2 - 7 */
+ p_avail : 4, /* 8 - 11 */
+ p_res1 : 2, /* 12 - 13 */
+ p_urgent : 1, /* 14 */
+ p_pst : 1, /* 15 */
+ p_vector : 8, /* 16 - 23 */
+ p_res2 : 14, /* 24 - 37 */
+ pda_l : 26; /* 38 - 63 */
+ };
+ __u64 low;
+ };
+
+ union {
+ /* Shared between remapped and posted mode*/
+ struct {
+ __u64 sid : 16, /* 64 - 79 */
+ sq : 2, /* 80 - 81 */
+ svt : 2, /* 82 - 83 */
+ __res3 : 44; /* 84 - 127 */
+ };
+
+ /* Posted mode*/
+ struct {
+ __u64 p_sid : 16, /* 64 - 79 */
+ p_sq : 2, /* 80 - 81 */
+ p_svt : 2, /* 82 - 83 */
+ p_res3 : 12, /* 84 - 95 */
+ pda_h : 32; /* 96 - 127 */
+ };
+ __u64 high;
+ };
};
-
- /* Remapped mode */
- struct {
- __u64 r_present : 1, /* 0 */
- r_fpd : 1, /* 1 */
- dst_mode : 1, /* 2 */
- redir_hint : 1, /* 3 */
- trigger_mode : 1, /* 4 */
- dlvry_mode : 3, /* 5 - 7 */
- r_avail : 4, /* 8 - 11 */
- r_res0 : 4, /* 12 - 15 */
- r_vector : 8, /* 16 - 23 */
- r_res1 : 8, /* 24 - 31 */
- dest_id : 32; /* 32 - 63 */
- };
-
- /* Posted mode */
- struct {
- __u64 p_present : 1, /* 0 */
- p_fpd : 1, /* 1 */
- p_res0 : 6, /* 2 - 7 */
- p_avail : 4, /* 8 - 11 */
- p_res1 : 2, /* 12 - 13 */
- p_urgent : 1, /* 14 */
- p_pst : 1, /* 15 */
- p_vector : 8, /* 16 - 23 */
- p_res2 : 14, /* 24 - 37 */
- pda_l : 26; /* 38 - 63 */
- };
- __u64 low;
- };
-
- union {
- /* Shared between remapped and posted mode*/
- struct {
- __u64 sid : 16, /* 64 - 79 */
- sq : 2, /* 80 - 81 */
- svt : 2, /* 82 - 83 */
- __res3 : 44; /* 84 - 127 */
- };
-
- /* Posted mode*/
- struct {
- __u64 p_sid : 16, /* 64 - 79 */
- p_sq : 2, /* 80 - 81 */
- p_svt : 2, /* 82 - 83 */
- p_res3 : 12, /* 84 - 95 */
- pda_h : 32; /* 96 - 127 */
- };
- __u64 high;
+#ifdef CONFIG_IRQ_REMAP
+ __u128 irte;
+#endif
};
};
^ permalink raw reply [flat|nested] 47+ messages in thread* [tip: locking/core] x86,intel_iommu: Replace cmpxchg_double()
2023-05-31 13:08 ` [PATCH 09/12] x86,intel_iommu: " Peter Zijlstra
@ 2023-06-05 7:42 ` tip-bot2 for Peter Zijlstra
0 siblings, 0 replies; 47+ messages in thread
From: tip-bot2 for Peter Zijlstra @ 2023-06-05 7:42 UTC (permalink / raw)
To: linux-tip-commits
Cc: Peter Zijlstra (Intel),
Lu Baolu, Arnd Bergmann, Mark Rutland, x86, linux-kernel
The following commit has been merged into the locking/core branch of tip:
Commit-ID: b1fe7f2cda2a003afe316ce8dfe8d3645694a67e
Gitweb: https://git.kernel.org/tip/b1fe7f2cda2a003afe316ce8dfe8d3645694a67e
Author: Peter Zijlstra <peterz@infradead.org>
AuthorDate: Wed, 31 May 2023 15:08:42 +02:00
Committer: Peter Zijlstra <peterz@infradead.org>
CommitterDate: Mon, 05 Jun 2023 09:36:38 +02:00
x86,intel_iommu: Replace cmpxchg_double()
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Reviewed-by: Lu Baolu <baolu.lu@linux.intel.com>
Reviewed-by: Arnd Bergmann <arnd@arndb.de>
Reviewed-by: Mark Rutland <mark.rutland@arm.com>
Tested-by: Mark Rutland <mark.rutland@arm.com>
Link: https://lore.kernel.org/r/20230531132323.855976804@infradead.org
---
drivers/iommu/intel/irq_remapping.c | 8 +--
include/linux/dmar.h | 125 ++++++++++++++-------------
2 files changed, 68 insertions(+), 65 deletions(-)
diff --git a/drivers/iommu/intel/irq_remapping.c b/drivers/iommu/intel/irq_remapping.c
index a1b9873..08f5632 100644
--- a/drivers/iommu/intel/irq_remapping.c
+++ b/drivers/iommu/intel/irq_remapping.c
@@ -175,18 +175,14 @@ static int modify_irte(struct irq_2_iommu *irq_iommu,
irte = &iommu->ir_table->base[index];
if ((irte->pst == 1) || (irte_modified->pst == 1)) {
- bool ret;
-
- ret = cmpxchg_double(&irte->low, &irte->high,
- irte->low, irte->high,
- irte_modified->low, irte_modified->high);
/*
* We use cmpxchg16 to atomically update the 128-bit IRTE,
* and it cannot be updated by the hardware or other processors
* behind us, so the return value of cmpxchg16 should be the
* same as the old value.
*/
- WARN_ON(!ret);
+ u128 old = irte->irte;
+ WARN_ON(!try_cmpxchg128(&irte->irte, &old, irte_modified->irte));
} else {
WRITE_ONCE(irte->low, irte_modified->low);
WRITE_ONCE(irte->high, irte_modified->high);
diff --git a/include/linux/dmar.h b/include/linux/dmar.h
index 725d5e6..27dbd4c 100644
--- a/include/linux/dmar.h
+++ b/include/linux/dmar.h
@@ -202,67 +202,74 @@ static inline void detect_intel_iommu(void)
struct irte {
union {
- /* Shared between remapped and posted mode*/
struct {
- __u64 present : 1, /* 0 */
- fpd : 1, /* 1 */
- __res0 : 6, /* 2 - 6 */
- avail : 4, /* 8 - 11 */
- __res1 : 3, /* 12 - 14 */
- pst : 1, /* 15 */
- vector : 8, /* 16 - 23 */
- __res2 : 40; /* 24 - 63 */
+ union {
+ /* Shared between remapped and posted mode*/
+ struct {
+ __u64 present : 1, /* 0 */
+ fpd : 1, /* 1 */
+ __res0 : 6, /* 2 - 6 */
+ avail : 4, /* 8 - 11 */
+ __res1 : 3, /* 12 - 14 */
+ pst : 1, /* 15 */
+ vector : 8, /* 16 - 23 */
+ __res2 : 40; /* 24 - 63 */
+ };
+
+ /* Remapped mode */
+ struct {
+ __u64 r_present : 1, /* 0 */
+ r_fpd : 1, /* 1 */
+ dst_mode : 1, /* 2 */
+ redir_hint : 1, /* 3 */
+ trigger_mode : 1, /* 4 */
+ dlvry_mode : 3, /* 5 - 7 */
+ r_avail : 4, /* 8 - 11 */
+ r_res0 : 4, /* 12 - 15 */
+ r_vector : 8, /* 16 - 23 */
+ r_res1 : 8, /* 24 - 31 */
+ dest_id : 32; /* 32 - 63 */
+ };
+
+ /* Posted mode */
+ struct {
+ __u64 p_present : 1, /* 0 */
+ p_fpd : 1, /* 1 */
+ p_res0 : 6, /* 2 - 7 */
+ p_avail : 4, /* 8 - 11 */
+ p_res1 : 2, /* 12 - 13 */
+ p_urgent : 1, /* 14 */
+ p_pst : 1, /* 15 */
+ p_vector : 8, /* 16 - 23 */
+ p_res2 : 14, /* 24 - 37 */
+ pda_l : 26; /* 38 - 63 */
+ };
+ __u64 low;
+ };
+
+ union {
+ /* Shared between remapped and posted mode*/
+ struct {
+ __u64 sid : 16, /* 64 - 79 */
+ sq : 2, /* 80 - 81 */
+ svt : 2, /* 82 - 83 */
+ __res3 : 44; /* 84 - 127 */
+ };
+
+ /* Posted mode*/
+ struct {
+ __u64 p_sid : 16, /* 64 - 79 */
+ p_sq : 2, /* 80 - 81 */
+ p_svt : 2, /* 82 - 83 */
+ p_res3 : 12, /* 84 - 95 */
+ pda_h : 32; /* 96 - 127 */
+ };
+ __u64 high;
+ };
};
-
- /* Remapped mode */
- struct {
- __u64 r_present : 1, /* 0 */
- r_fpd : 1, /* 1 */
- dst_mode : 1, /* 2 */
- redir_hint : 1, /* 3 */
- trigger_mode : 1, /* 4 */
- dlvry_mode : 3, /* 5 - 7 */
- r_avail : 4, /* 8 - 11 */
- r_res0 : 4, /* 12 - 15 */
- r_vector : 8, /* 16 - 23 */
- r_res1 : 8, /* 24 - 31 */
- dest_id : 32; /* 32 - 63 */
- };
-
- /* Posted mode */
- struct {
- __u64 p_present : 1, /* 0 */
- p_fpd : 1, /* 1 */
- p_res0 : 6, /* 2 - 7 */
- p_avail : 4, /* 8 - 11 */
- p_res1 : 2, /* 12 - 13 */
- p_urgent : 1, /* 14 */
- p_pst : 1, /* 15 */
- p_vector : 8, /* 16 - 23 */
- p_res2 : 14, /* 24 - 37 */
- pda_l : 26; /* 38 - 63 */
- };
- __u64 low;
- };
-
- union {
- /* Shared between remapped and posted mode*/
- struct {
- __u64 sid : 16, /* 64 - 79 */
- sq : 2, /* 80 - 81 */
- svt : 2, /* 82 - 83 */
- __res3 : 44; /* 84 - 127 */
- };
-
- /* Posted mode*/
- struct {
- __u64 p_sid : 16, /* 64 - 79 */
- p_sq : 2, /* 80 - 81 */
- p_svt : 2, /* 82 - 83 */
- p_res3 : 12, /* 84 - 95 */
- pda_h : 32; /* 96 - 127 */
- };
- __u64 high;
+#ifdef CONFIG_IRQ_REMAP
+ __u128 irte;
+#endif
};
};
^ permalink raw reply [flat|nested] 47+ messages in thread
* [PATCH 10/12] slub: Replace cmpxchg_double()
2023-05-31 13:08 [PATCH 00/12] Introduce cmpxchg128() -- aka. the demise of cmpxchg_double() Peter Zijlstra
` (8 preceding siblings ...)
2023-05-31 13:08 ` [PATCH 09/12] x86,intel_iommu: " Peter Zijlstra
@ 2023-05-31 13:08 ` Peter Zijlstra
2023-06-05 7:42 ` [tip: locking/core] " tip-bot2 for Peter Zijlstra
2023-05-31 13:08 ` [PATCH 11/12] arch: Remove cmpxchg_double Peter Zijlstra
` (2 subsequent siblings)
12 siblings, 1 reply; 47+ messages in thread
From: Peter Zijlstra @ 2023-05-31 13:08 UTC (permalink / raw)
To: torvalds
Cc: corbet, will, peterz, boqun.feng, mark.rutland, catalin.marinas,
dennis, tj, cl, hca, gor, agordeev, borntraeger, svens, tglx,
mingo, bp, dave.hansen, x86, hpa, joro, suravee.suthikulpanit,
robin.murphy, dwmw2, baolu.lu, Arnd Bergmann, Herbert Xu, davem,
penberg, rientjes, iamjoonsoo.kim, Andrew Morton, vbabka,
roman.gushchin, 42.hyeyoo, linux-doc, linux-kernel, linux-mm,
linux-s390, iommu, linux-arch, linux-crypto, sfr, mpe,
James.Bottomley, deller, linux-parisc
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Reviewed-by: Arnd Bergmann <arnd@arndb.de>
Acked-by: Vlastimil Babka <vbabka@suse.cz>
Acked-by: Hyeonggon Yoo <42.hyeyoo@gmail.com>
---
include/linux/slub_def.h | 12 +++-
mm/slab.h | 53 +++++++++++++++--
mm/slub.c | 139 +++++++++++++++++++++++++++--------------------
3 files changed, 137 insertions(+), 67 deletions(-)
--- a/include/linux/slub_def.h
+++ b/include/linux/slub_def.h
@@ -39,7 +39,8 @@ enum stat_item {
CPU_PARTIAL_FREE, /* Refill cpu partial on free */
CPU_PARTIAL_NODE, /* Refill cpu partial from node partial */
CPU_PARTIAL_DRAIN, /* Drain cpu partial to node partial */
- NR_SLUB_STAT_ITEMS };
+ NR_SLUB_STAT_ITEMS
+};
#ifndef CONFIG_SLUB_TINY
/*
@@ -47,8 +48,13 @@ enum stat_item {
* with this_cpu_cmpxchg_double() alignment requirements.
*/
struct kmem_cache_cpu {
- void **freelist; /* Pointer to next available object */
- unsigned long tid; /* Globally unique transaction id */
+ union {
+ struct {
+ void **freelist; /* Pointer to next available object */
+ unsigned long tid; /* Globally unique transaction id */
+ };
+ freelist_aba_t freelist_tid;
+ };
struct slab *slab; /* The slab from which we are allocating */
#ifdef CONFIG_SLUB_CPU_PARTIAL
struct slab *partial; /* Partially allocated frozen slabs */
--- a/mm/slab.h
+++ b/mm/slab.h
@@ -6,6 +6,38 @@
*/
void __init kmem_cache_init(void);
+#ifdef CONFIG_64BIT
+# ifdef system_has_cmpxchg128
+# define system_has_freelist_aba() system_has_cmpxchg128()
+# define try_cmpxchg_freelist try_cmpxchg128
+# endif
+#define this_cpu_try_cmpxchg_freelist this_cpu_try_cmpxchg128
+typedef u128 freelist_full_t;
+#else /* CONFIG_64BIT */
+# ifdef system_has_cmpxchg64
+# define system_has_freelist_aba() system_has_cmpxchg64()
+# define try_cmpxchg_freelist try_cmpxchg64
+# endif
+#define this_cpu_try_cmpxchg_freelist this_cpu_try_cmpxchg64
+typedef u64 freelist_full_t;
+#endif /* CONFIG_64BIT */
+
+#if defined(system_has_freelist_aba) && !defined(CONFIG_HAVE_ALIGNED_STRUCT_PAGE)
+#undef system_has_freelist_aba
+#endif
+
+/*
+ * Freelist pointer and counter to cmpxchg together, avoids the typical ABA
+ * problems with cmpxchg of just a pointer.
+ */
+typedef union {
+ struct {
+ void *freelist;
+ unsigned long counter;
+ };
+ freelist_full_t full;
+} freelist_aba_t;
+
/* Reuses the bits in struct page */
struct slab {
unsigned long __page_flags;
@@ -38,14 +70,21 @@ struct slab {
#endif
};
/* Double-word boundary */
- void *freelist; /* first free object */
union {
- unsigned long counters;
struct {
- unsigned inuse:16;
- unsigned objects:15;
- unsigned frozen:1;
+ void *freelist; /* first free object */
+ union {
+ unsigned long counters;
+ struct {
+ unsigned inuse:16;
+ unsigned objects:15;
+ unsigned frozen:1;
+ };
+ };
};
+#ifdef system_has_freelist_aba
+ freelist_aba_t freelist_counter;
+#endif
};
};
struct rcu_head rcu_head;
@@ -72,8 +111,8 @@ SLAB_MATCH(memcg_data, memcg_data);
#endif
#undef SLAB_MATCH
static_assert(sizeof(struct slab) <= sizeof(struct page));
-#if defined(CONFIG_HAVE_CMPXCHG_DOUBLE) && defined(CONFIG_SLUB)
-static_assert(IS_ALIGNED(offsetof(struct slab, freelist), 2*sizeof(void *)));
+#if defined(system_has_freelist_aba) && defined(CONFIG_SLUB)
+static_assert(IS_ALIGNED(offsetof(struct slab, freelist), sizeof(freelist_aba_t)));
#endif
/**
--- a/mm/slub.c
+++ b/mm/slub.c
@@ -292,7 +292,12 @@ static inline bool kmem_cache_has_cpu_pa
/* Poison object */
#define __OBJECT_POISON ((slab_flags_t __force)0x80000000U)
/* Use cmpxchg_double */
+
+#ifdef system_has_freelist_aba
#define __CMPXCHG_DOUBLE ((slab_flags_t __force)0x40000000U)
+#else
+#define __CMPXCHG_DOUBLE ((slab_flags_t __force)0U)
+#endif
/*
* Tracking user of a slab.
@@ -512,6 +517,40 @@ static __always_inline void slab_unlock(
__bit_spin_unlock(PG_locked, &page->flags);
}
+static inline bool
+__update_freelist_fast(struct slab *slab,
+ void *freelist_old, unsigned long counters_old,
+ void *freelist_new, unsigned long counters_new)
+{
+#ifdef system_has_freelist_aba
+ freelist_aba_t old = { .freelist = freelist_old, .counter = counters_old };
+ freelist_aba_t new = { .freelist = freelist_new, .counter = counters_new };
+
+ return try_cmpxchg_freelist(&slab->freelist_counter.full, &old.full, new.full);
+#else
+ return false;
+#endif
+}
+
+static inline bool
+__update_freelist_slow(struct slab *slab,
+ void *freelist_old, unsigned long counters_old,
+ void *freelist_new, unsigned long counters_new)
+{
+ bool ret = false;
+
+ slab_lock(slab);
+ if (slab->freelist == freelist_old &&
+ slab->counters == counters_old) {
+ slab->freelist = freelist_new;
+ slab->counters = counters_new;
+ ret = true;
+ }
+ slab_unlock(slab);
+
+ return ret;
+}
+
/*
* Interrupts must be disabled (for the fallback code to work right), typically
* by an _irqsave() lock variant. On PREEMPT_RT the preempt_disable(), which is
@@ -519,33 +558,25 @@ static __always_inline void slab_unlock(
* allocation/ free operation in hardirq context. Therefore nothing can
* interrupt the operation.
*/
-static inline bool __cmpxchg_double_slab(struct kmem_cache *s, struct slab *slab,
+static inline bool __slab_update_freelist(struct kmem_cache *s, struct slab *slab,
void *freelist_old, unsigned long counters_old,
void *freelist_new, unsigned long counters_new,
const char *n)
{
+ bool ret;
+
if (USE_LOCKLESS_FAST_PATH())
lockdep_assert_irqs_disabled();
-#if defined(CONFIG_HAVE_CMPXCHG_DOUBLE) && \
- defined(CONFIG_HAVE_ALIGNED_STRUCT_PAGE)
+
if (s->flags & __CMPXCHG_DOUBLE) {
- if (cmpxchg_double(&slab->freelist, &slab->counters,
- freelist_old, counters_old,
- freelist_new, counters_new))
- return true;
- } else
-#endif
- {
- slab_lock(slab);
- if (slab->freelist == freelist_old &&
- slab->counters == counters_old) {
- slab->freelist = freelist_new;
- slab->counters = counters_new;
- slab_unlock(slab);
- return true;
- }
- slab_unlock(slab);
+ ret = __update_freelist_fast(slab, freelist_old, counters_old,
+ freelist_new, counters_new);
+ } else {
+ ret = __update_freelist_slow(slab, freelist_old, counters_old,
+ freelist_new, counters_new);
}
+ if (likely(ret))
+ return true;
cpu_relax();
stat(s, CMPXCHG_DOUBLE_FAIL);
@@ -557,36 +588,26 @@ static inline bool __cmpxchg_double_slab
return false;
}
-static inline bool cmpxchg_double_slab(struct kmem_cache *s, struct slab *slab,
+static inline bool slab_update_freelist(struct kmem_cache *s, struct slab *slab,
void *freelist_old, unsigned long counters_old,
void *freelist_new, unsigned long counters_new,
const char *n)
{
-#if defined(CONFIG_HAVE_CMPXCHG_DOUBLE) && \
- defined(CONFIG_HAVE_ALIGNED_STRUCT_PAGE)
+ bool ret;
+
if (s->flags & __CMPXCHG_DOUBLE) {
- if (cmpxchg_double(&slab->freelist, &slab->counters,
- freelist_old, counters_old,
- freelist_new, counters_new))
- return true;
- } else
-#endif
- {
+ ret = __update_freelist_fast(slab, freelist_old, counters_old,
+ freelist_new, counters_new);
+ } else {
unsigned long flags;
local_irq_save(flags);
- slab_lock(slab);
- if (slab->freelist == freelist_old &&
- slab->counters == counters_old) {
- slab->freelist = freelist_new;
- slab->counters = counters_new;
- slab_unlock(slab);
- local_irq_restore(flags);
- return true;
- }
- slab_unlock(slab);
+ ret = __update_freelist_slow(slab, freelist_old, counters_old,
+ freelist_new, counters_new);
local_irq_restore(flags);
}
+ if (likely(ret))
+ return true;
cpu_relax();
stat(s, CMPXCHG_DOUBLE_FAIL);
@@ -2228,7 +2249,7 @@ static inline void *acquire_slab(struct
VM_BUG_ON(new.frozen);
new.frozen = 1;
- if (!__cmpxchg_double_slab(s, slab,
+ if (!__slab_update_freelist(s, slab,
freelist, counters,
new.freelist, new.counters,
"acquire_slab"))
@@ -2554,7 +2575,7 @@ static void deactivate_slab(struct kmem_
}
- if (!cmpxchg_double_slab(s, slab,
+ if (!slab_update_freelist(s, slab,
old.freelist, old.counters,
new.freelist, new.counters,
"unfreezing slab")) {
@@ -2611,7 +2632,7 @@ static void __unfreeze_partials(struct k
new.frozen = 0;
- } while (!__cmpxchg_double_slab(s, slab,
+ } while (!__slab_update_freelist(s, slab,
old.freelist, old.counters,
new.freelist, new.counters,
"unfreezing slab"));
@@ -3008,6 +3029,18 @@ static inline bool pfmemalloc_match(stru
}
#ifndef CONFIG_SLUB_TINY
+static inline bool
+__update_cpu_freelist_fast(struct kmem_cache *s,
+ void *freelist_old, void *freelist_new,
+ unsigned long tid)
+{
+ freelist_aba_t old = { .freelist = freelist_old, .counter = tid };
+ freelist_aba_t new = { .freelist = freelist_new, .counter = next_tid(tid) };
+
+ return this_cpu_try_cmpxchg_freelist(s->cpu_slab->freelist_tid.full,
+ &old.full, new.full);
+}
+
/*
* Check the slab->freelist and either transfer the freelist to the
* per cpu freelist or deactivate the slab.
@@ -3034,7 +3067,7 @@ static inline void *get_freelist(struct
new.inuse = slab->objects;
new.frozen = freelist != NULL;
- } while (!__cmpxchg_double_slab(s, slab,
+ } while (!__slab_update_freelist(s, slab,
freelist, counters,
NULL, new.counters,
"get_freelist"));
@@ -3359,11 +3392,7 @@ static __always_inline void *__slab_allo
* against code executing on this cpu *not* from access by
* other cpus.
*/
- if (unlikely(!this_cpu_cmpxchg_double(
- s->cpu_slab->freelist, s->cpu_slab->tid,
- object, tid,
- next_object, next_tid(tid)))) {
-
+ if (unlikely(!__update_cpu_freelist_fast(s, object, next_object, tid))) {
note_cmpxchg_failure("slab_alloc", s, tid);
goto redo;
}
@@ -3631,7 +3660,7 @@ static void __slab_free(struct kmem_cach
}
}
- } while (!cmpxchg_double_slab(s, slab,
+ } while (!slab_update_freelist(s, slab,
prior, counters,
head, new.counters,
"__slab_free"));
@@ -3736,11 +3765,7 @@ static __always_inline void do_slab_free
set_freepointer(s, tail_obj, freelist);
- if (unlikely(!this_cpu_cmpxchg_double(
- s->cpu_slab->freelist, s->cpu_slab->tid,
- freelist, tid,
- head, next_tid(tid)))) {
-
+ if (unlikely(!__update_cpu_freelist_fast(s, freelist, head, tid))) {
note_cmpxchg_failure("slab_free", s, tid);
goto redo;
}
@@ -4505,11 +4530,11 @@ static int kmem_cache_open(struct kmem_c
}
}
-#if defined(CONFIG_HAVE_CMPXCHG_DOUBLE) && \
- defined(CONFIG_HAVE_ALIGNED_STRUCT_PAGE)
- if (system_has_cmpxchg_double() && (s->flags & SLAB_NO_CMPXCHG) == 0)
+#ifdef system_has_freelist_aba
+ if (system_has_freelist_aba() && !(s->flags & SLAB_NO_CMPXCHG)) {
/* Enable fast mode */
s->flags |= __CMPXCHG_DOUBLE;
+ }
#endif
/*
^ permalink raw reply [flat|nested] 47+ messages in thread* [tip: locking/core] slub: Replace cmpxchg_double()
2023-05-31 13:08 ` [PATCH 10/12] slub: " Peter Zijlstra
@ 2023-06-05 7:42 ` tip-bot2 for Peter Zijlstra
0 siblings, 0 replies; 47+ messages in thread
From: tip-bot2 for Peter Zijlstra @ 2023-06-05 7:42 UTC (permalink / raw)
To: linux-tip-commits
Cc: Peter Zijlstra (Intel),
Arnd Bergmann, Mark Rutland, Vlastimil Babka, Hyeonggon Yoo, x86,
linux-kernel
The following commit has been merged into the locking/core branch of tip:
Commit-ID: 6801be4f2653e5fdddca73b527cf0728284ba8a3
Gitweb: https://git.kernel.org/tip/6801be4f2653e5fdddca73b527cf0728284ba8a3
Author: Peter Zijlstra <peterz@infradead.org>
AuthorDate: Wed, 31 May 2023 15:08:43 +02:00
Committer: Peter Zijlstra <peterz@infradead.org>
CommitterDate: Mon, 05 Jun 2023 09:36:39 +02:00
slub: Replace cmpxchg_double()
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Reviewed-by: Arnd Bergmann <arnd@arndb.de>
Reviewed-by: Mark Rutland <mark.rutland@arm.com>
Acked-by: Vlastimil Babka <vbabka@suse.cz>
Acked-by: Hyeonggon Yoo <42.hyeyoo@gmail.com>
Tested-by: Mark Rutland <mark.rutland@arm.com>
Link: https://lore.kernel.org/r/20230531132323.924677086@infradead.org
---
include/linux/slub_def.h | 12 ++-
mm/slab.h | 53 ++++++++++++--
mm/slub.c | 139 ++++++++++++++++++++++----------------
3 files changed, 137 insertions(+), 67 deletions(-)
diff --git a/include/linux/slub_def.h b/include/linux/slub_def.h
index f6df03f..deb90cf 100644
--- a/include/linux/slub_def.h
+++ b/include/linux/slub_def.h
@@ -39,7 +39,8 @@ enum stat_item {
CPU_PARTIAL_FREE, /* Refill cpu partial on free */
CPU_PARTIAL_NODE, /* Refill cpu partial from node partial */
CPU_PARTIAL_DRAIN, /* Drain cpu partial to node partial */
- NR_SLUB_STAT_ITEMS };
+ NR_SLUB_STAT_ITEMS
+};
#ifndef CONFIG_SLUB_TINY
/*
@@ -47,8 +48,13 @@ enum stat_item {
* with this_cpu_cmpxchg_double() alignment requirements.
*/
struct kmem_cache_cpu {
- void **freelist; /* Pointer to next available object */
- unsigned long tid; /* Globally unique transaction id */
+ union {
+ struct {
+ void **freelist; /* Pointer to next available object */
+ unsigned long tid; /* Globally unique transaction id */
+ };
+ freelist_aba_t freelist_tid;
+ };
struct slab *slab; /* The slab from which we are allocating */
#ifdef CONFIG_SLUB_CPU_PARTIAL
struct slab *partial; /* Partially allocated frozen slabs */
diff --git a/mm/slab.h b/mm/slab.h
index f01ac25..bc36edd 100644
--- a/mm/slab.h
+++ b/mm/slab.h
@@ -6,6 +6,38 @@
*/
void __init kmem_cache_init(void);
+#ifdef CONFIG_64BIT
+# ifdef system_has_cmpxchg128
+# define system_has_freelist_aba() system_has_cmpxchg128()
+# define try_cmpxchg_freelist try_cmpxchg128
+# endif
+#define this_cpu_try_cmpxchg_freelist this_cpu_try_cmpxchg128
+typedef u128 freelist_full_t;
+#else /* CONFIG_64BIT */
+# ifdef system_has_cmpxchg64
+# define system_has_freelist_aba() system_has_cmpxchg64()
+# define try_cmpxchg_freelist try_cmpxchg64
+# endif
+#define this_cpu_try_cmpxchg_freelist this_cpu_try_cmpxchg64
+typedef u64 freelist_full_t;
+#endif /* CONFIG_64BIT */
+
+#if defined(system_has_freelist_aba) && !defined(CONFIG_HAVE_ALIGNED_STRUCT_PAGE)
+#undef system_has_freelist_aba
+#endif
+
+/*
+ * Freelist pointer and counter to cmpxchg together, avoids the typical ABA
+ * problems with cmpxchg of just a pointer.
+ */
+typedef union {
+ struct {
+ void *freelist;
+ unsigned long counter;
+ };
+ freelist_full_t full;
+} freelist_aba_t;
+
/* Reuses the bits in struct page */
struct slab {
unsigned long __page_flags;
@@ -38,14 +70,21 @@ struct slab {
#endif
};
/* Double-word boundary */
- void *freelist; /* first free object */
union {
- unsigned long counters;
struct {
- unsigned inuse:16;
- unsigned objects:15;
- unsigned frozen:1;
+ void *freelist; /* first free object */
+ union {
+ unsigned long counters;
+ struct {
+ unsigned inuse:16;
+ unsigned objects:15;
+ unsigned frozen:1;
+ };
+ };
};
+#ifdef system_has_freelist_aba
+ freelist_aba_t freelist_counter;
+#endif
};
};
struct rcu_head rcu_head;
@@ -72,8 +111,8 @@ SLAB_MATCH(memcg_data, memcg_data);
#endif
#undef SLAB_MATCH
static_assert(sizeof(struct slab) <= sizeof(struct page));
-#if defined(CONFIG_HAVE_CMPXCHG_DOUBLE) && defined(CONFIG_SLUB)
-static_assert(IS_ALIGNED(offsetof(struct slab, freelist), 2*sizeof(void *)));
+#if defined(system_has_freelist_aba) && defined(CONFIG_SLUB)
+static_assert(IS_ALIGNED(offsetof(struct slab, freelist), sizeof(freelist_aba_t)));
#endif
/**
diff --git a/mm/slub.c b/mm/slub.c
index c87628c..7529626 100644
--- a/mm/slub.c
+++ b/mm/slub.c
@@ -292,7 +292,12 @@ static inline bool kmem_cache_has_cpu_partial(struct kmem_cache *s)
/* Poison object */
#define __OBJECT_POISON ((slab_flags_t __force)0x80000000U)
/* Use cmpxchg_double */
+
+#ifdef system_has_freelist_aba
#define __CMPXCHG_DOUBLE ((slab_flags_t __force)0x40000000U)
+#else
+#define __CMPXCHG_DOUBLE ((slab_flags_t __force)0U)
+#endif
/*
* Tracking user of a slab.
@@ -512,6 +517,40 @@ static __always_inline void slab_unlock(struct slab *slab)
__bit_spin_unlock(PG_locked, &page->flags);
}
+static inline bool
+__update_freelist_fast(struct slab *slab,
+ void *freelist_old, unsigned long counters_old,
+ void *freelist_new, unsigned long counters_new)
+{
+#ifdef system_has_freelist_aba
+ freelist_aba_t old = { .freelist = freelist_old, .counter = counters_old };
+ freelist_aba_t new = { .freelist = freelist_new, .counter = counters_new };
+
+ return try_cmpxchg_freelist(&slab->freelist_counter.full, &old.full, new.full);
+#else
+ return false;
+#endif
+}
+
+static inline bool
+__update_freelist_slow(struct slab *slab,
+ void *freelist_old, unsigned long counters_old,
+ void *freelist_new, unsigned long counters_new)
+{
+ bool ret = false;
+
+ slab_lock(slab);
+ if (slab->freelist == freelist_old &&
+ slab->counters == counters_old) {
+ slab->freelist = freelist_new;
+ slab->counters = counters_new;
+ ret = true;
+ }
+ slab_unlock(slab);
+
+ return ret;
+}
+
/*
* Interrupts must be disabled (for the fallback code to work right), typically
* by an _irqsave() lock variant. On PREEMPT_RT the preempt_disable(), which is
@@ -519,33 +558,25 @@ static __always_inline void slab_unlock(struct slab *slab)
* allocation/ free operation in hardirq context. Therefore nothing can
* interrupt the operation.
*/
-static inline bool __cmpxchg_double_slab(struct kmem_cache *s, struct slab *slab,
+static inline bool __slab_update_freelist(struct kmem_cache *s, struct slab *slab,
void *freelist_old, unsigned long counters_old,
void *freelist_new, unsigned long counters_new,
const char *n)
{
+ bool ret;
+
if (USE_LOCKLESS_FAST_PATH())
lockdep_assert_irqs_disabled();
-#if defined(CONFIG_HAVE_CMPXCHG_DOUBLE) && \
- defined(CONFIG_HAVE_ALIGNED_STRUCT_PAGE)
+
if (s->flags & __CMPXCHG_DOUBLE) {
- if (cmpxchg_double(&slab->freelist, &slab->counters,
- freelist_old, counters_old,
- freelist_new, counters_new))
- return true;
- } else
-#endif
- {
- slab_lock(slab);
- if (slab->freelist == freelist_old &&
- slab->counters == counters_old) {
- slab->freelist = freelist_new;
- slab->counters = counters_new;
- slab_unlock(slab);
- return true;
- }
- slab_unlock(slab);
+ ret = __update_freelist_fast(slab, freelist_old, counters_old,
+ freelist_new, counters_new);
+ } else {
+ ret = __update_freelist_slow(slab, freelist_old, counters_old,
+ freelist_new, counters_new);
}
+ if (likely(ret))
+ return true;
cpu_relax();
stat(s, CMPXCHG_DOUBLE_FAIL);
@@ -557,36 +588,26 @@ static inline bool __cmpxchg_double_slab(struct kmem_cache *s, struct slab *slab
return false;
}
-static inline bool cmpxchg_double_slab(struct kmem_cache *s, struct slab *slab,
+static inline bool slab_update_freelist(struct kmem_cache *s, struct slab *slab,
void *freelist_old, unsigned long counters_old,
void *freelist_new, unsigned long counters_new,
const char *n)
{
-#if defined(CONFIG_HAVE_CMPXCHG_DOUBLE) && \
- defined(CONFIG_HAVE_ALIGNED_STRUCT_PAGE)
+ bool ret;
+
if (s->flags & __CMPXCHG_DOUBLE) {
- if (cmpxchg_double(&slab->freelist, &slab->counters,
- freelist_old, counters_old,
- freelist_new, counters_new))
- return true;
- } else
-#endif
- {
+ ret = __update_freelist_fast(slab, freelist_old, counters_old,
+ freelist_new, counters_new);
+ } else {
unsigned long flags;
local_irq_save(flags);
- slab_lock(slab);
- if (slab->freelist == freelist_old &&
- slab->counters == counters_old) {
- slab->freelist = freelist_new;
- slab->counters = counters_new;
- slab_unlock(slab);
- local_irq_restore(flags);
- return true;
- }
- slab_unlock(slab);
+ ret = __update_freelist_slow(slab, freelist_old, counters_old,
+ freelist_new, counters_new);
local_irq_restore(flags);
}
+ if (likely(ret))
+ return true;
cpu_relax();
stat(s, CMPXCHG_DOUBLE_FAIL);
@@ -2228,7 +2249,7 @@ static inline void *acquire_slab(struct kmem_cache *s,
VM_BUG_ON(new.frozen);
new.frozen = 1;
- if (!__cmpxchg_double_slab(s, slab,
+ if (!__slab_update_freelist(s, slab,
freelist, counters,
new.freelist, new.counters,
"acquire_slab"))
@@ -2554,7 +2575,7 @@ redo:
}
- if (!cmpxchg_double_slab(s, slab,
+ if (!slab_update_freelist(s, slab,
old.freelist, old.counters,
new.freelist, new.counters,
"unfreezing slab")) {
@@ -2611,7 +2632,7 @@ static void __unfreeze_partials(struct kmem_cache *s, struct slab *partial_slab)
new.frozen = 0;
- } while (!__cmpxchg_double_slab(s, slab,
+ } while (!__slab_update_freelist(s, slab,
old.freelist, old.counters,
new.freelist, new.counters,
"unfreezing slab"));
@@ -3008,6 +3029,18 @@ static inline bool pfmemalloc_match(struct slab *slab, gfp_t gfpflags)
}
#ifndef CONFIG_SLUB_TINY
+static inline bool
+__update_cpu_freelist_fast(struct kmem_cache *s,
+ void *freelist_old, void *freelist_new,
+ unsigned long tid)
+{
+ freelist_aba_t old = { .freelist = freelist_old, .counter = tid };
+ freelist_aba_t new = { .freelist = freelist_new, .counter = next_tid(tid) };
+
+ return this_cpu_try_cmpxchg_freelist(s->cpu_slab->freelist_tid.full,
+ &old.full, new.full);
+}
+
/*
* Check the slab->freelist and either transfer the freelist to the
* per cpu freelist or deactivate the slab.
@@ -3034,7 +3067,7 @@ static inline void *get_freelist(struct kmem_cache *s, struct slab *slab)
new.inuse = slab->objects;
new.frozen = freelist != NULL;
- } while (!__cmpxchg_double_slab(s, slab,
+ } while (!__slab_update_freelist(s, slab,
freelist, counters,
NULL, new.counters,
"get_freelist"));
@@ -3359,11 +3392,7 @@ redo:
* against code executing on this cpu *not* from access by
* other cpus.
*/
- if (unlikely(!this_cpu_cmpxchg_double(
- s->cpu_slab->freelist, s->cpu_slab->tid,
- object, tid,
- next_object, next_tid(tid)))) {
-
+ if (unlikely(!__update_cpu_freelist_fast(s, object, next_object, tid))) {
note_cmpxchg_failure("slab_alloc", s, tid);
goto redo;
}
@@ -3631,7 +3660,7 @@ static void __slab_free(struct kmem_cache *s, struct slab *slab,
}
}
- } while (!cmpxchg_double_slab(s, slab,
+ } while (!slab_update_freelist(s, slab,
prior, counters,
head, new.counters,
"__slab_free"));
@@ -3736,11 +3765,7 @@ redo:
set_freepointer(s, tail_obj, freelist);
- if (unlikely(!this_cpu_cmpxchg_double(
- s->cpu_slab->freelist, s->cpu_slab->tid,
- freelist, tid,
- head, next_tid(tid)))) {
-
+ if (unlikely(!__update_cpu_freelist_fast(s, freelist, head, tid))) {
note_cmpxchg_failure("slab_free", s, tid);
goto redo;
}
@@ -4505,11 +4530,11 @@ static int kmem_cache_open(struct kmem_cache *s, slab_flags_t flags)
}
}
-#if defined(CONFIG_HAVE_CMPXCHG_DOUBLE) && \
- defined(CONFIG_HAVE_ALIGNED_STRUCT_PAGE)
- if (system_has_cmpxchg_double() && (s->flags & SLAB_NO_CMPXCHG) == 0)
+#ifdef system_has_freelist_aba
+ if (system_has_freelist_aba() && !(s->flags & SLAB_NO_CMPXCHG)) {
/* Enable fast mode */
s->flags |= __CMPXCHG_DOUBLE;
+ }
#endif
/*
^ permalink raw reply [flat|nested] 47+ messages in thread
* [PATCH 11/12] arch: Remove cmpxchg_double
2023-05-31 13:08 [PATCH 00/12] Introduce cmpxchg128() -- aka. the demise of cmpxchg_double() Peter Zijlstra
` (9 preceding siblings ...)
2023-05-31 13:08 ` [PATCH 10/12] slub: " Peter Zijlstra
@ 2023-05-31 13:08 ` Peter Zijlstra
2023-06-05 7:42 ` [tip: locking/core] " tip-bot2 for Peter Zijlstra
2023-05-31 13:08 ` [PATCH 12/12] s390/cpum_sf: Convert to cmpxchg128() Peter Zijlstra
2023-05-31 14:47 ` [PATCH 00/12] Introduce cmpxchg128() -- aka. the demise of cmpxchg_double() Mark Rutland
12 siblings, 1 reply; 47+ messages in thread
From: Peter Zijlstra @ 2023-05-31 13:08 UTC (permalink / raw)
To: torvalds
Cc: corbet, will, peterz, boqun.feng, mark.rutland, catalin.marinas,
dennis, tj, cl, hca, gor, agordeev, borntraeger, svens, tglx,
mingo, bp, dave.hansen, x86, hpa, joro, suravee.suthikulpanit,
robin.murphy, dwmw2, baolu.lu, Arnd Bergmann, Herbert Xu, davem,
penberg, rientjes, iamjoonsoo.kim, Andrew Morton, vbabka,
roman.gushchin, 42.hyeyoo, linux-doc, linux-kernel, linux-mm,
linux-s390, iommu, linux-arch, linux-crypto, sfr, mpe,
James.Bottomley, deller, linux-parisc
No moar users, remove the monster.
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Reviewed-by: Arnd Bergmann <arnd@arndb.de>
Acked-by: Heiko Carstens <hca@linux.ibm.com>
---
Documentation/core-api/this_cpu_ops.rst | 2 -
arch/arm64/include/asm/atomic_ll_sc.h | 33 ----------------
arch/arm64/include/asm/atomic_lse.h | 36 ------------------
arch/arm64/include/asm/cmpxchg.h | 46 -----------------------
arch/arm64/include/asm/percpu.h | 10 -----
arch/s390/include/asm/cmpxchg.h | 34 -----------------
arch/s390/include/asm/percpu.h | 18 ---------
arch/x86/include/asm/cmpxchg.h | 25 ------------
arch/x86/include/asm/cmpxchg_32.h | 1
arch/x86/include/asm/cmpxchg_64.h | 1
arch/x86/include/asm/percpu.h | 42 ---------------------
include/asm-generic/percpu.h | 58 -----------------------------
include/linux/atomic/atomic-instrumented.h | 17 --------
include/linux/percpu-defs.h | 38 -------------------
scripts/atomic/gen-atomic-instrumented.sh | 15 ++-----
15 files changed, 5 insertions(+), 371 deletions(-)
--- a/Documentation/core-api/this_cpu_ops.rst
+++ b/Documentation/core-api/this_cpu_ops.rst
@@ -53,7 +53,6 @@ are defined. These operations can be use
this_cpu_add_return(pcp, val)
this_cpu_xchg(pcp, nval)
this_cpu_cmpxchg(pcp, oval, nval)
- this_cpu_cmpxchg_double(pcp1, pcp2, oval1, oval2, nval1, nval2)
this_cpu_sub(pcp, val)
this_cpu_inc(pcp)
this_cpu_dec(pcp)
@@ -242,7 +241,6 @@ modifies the variable, then RMW actions
__this_cpu_add_return(pcp, val)
__this_cpu_xchg(pcp, nval)
__this_cpu_cmpxchg(pcp, oval, nval)
- __this_cpu_cmpxchg_double(pcp1, pcp2, oval1, oval2, nval1, nval2)
__this_cpu_sub(pcp, val)
__this_cpu_inc(pcp)
__this_cpu_dec(pcp)
--- a/arch/arm64/include/asm/atomic_ll_sc.h
+++ b/arch/arm64/include/asm/atomic_ll_sc.h
@@ -294,39 +294,6 @@ __CMPXCHG_CASE( , , mb_, 64, dmb ish,
#undef __CMPXCHG_CASE
-#define __CMPXCHG_DBL(name, mb, rel, cl) \
-static __always_inline long \
-__ll_sc__cmpxchg_double##name(unsigned long old1, \
- unsigned long old2, \
- unsigned long new1, \
- unsigned long new2, \
- volatile void *ptr) \
-{ \
- unsigned long tmp, ret; \
- \
- asm volatile("// __cmpxchg_double" #name "\n" \
- " prfm pstl1strm, %2\n" \
- "1: ldxp %0, %1, %2\n" \
- " eor %0, %0, %3\n" \
- " eor %1, %1, %4\n" \
- " orr %1, %0, %1\n" \
- " cbnz %1, 2f\n" \
- " st" #rel "xp %w0, %5, %6, %2\n" \
- " cbnz %w0, 1b\n" \
- " " #mb "\n" \
- "2:" \
- : "=&r" (tmp), "=&r" (ret), "+Q" (*(__uint128_t *)ptr) \
- : "r" (old1), "r" (old2), "r" (new1), "r" (new2) \
- : cl); \
- \
- return ret; \
-}
-
-__CMPXCHG_DBL( , , , )
-__CMPXCHG_DBL(_mb, dmb ish, l, "memory")
-
-#undef __CMPXCHG_DBL
-
union __u128_halves {
u128 full;
struct {
--- a/arch/arm64/include/asm/atomic_lse.h
+++ b/arch/arm64/include/asm/atomic_lse.h
@@ -281,42 +281,6 @@ __CMPXCHG_CASE(x, , mb_, 64, al, "memo
#undef __CMPXCHG_CASE
-#define __CMPXCHG_DBL(name, mb, cl...) \
-static __always_inline long \
-__lse__cmpxchg_double##name(unsigned long old1, \
- unsigned long old2, \
- unsigned long new1, \
- unsigned long new2, \
- volatile void *ptr) \
-{ \
- unsigned long oldval1 = old1; \
- unsigned long oldval2 = old2; \
- register unsigned long x0 asm ("x0") = old1; \
- register unsigned long x1 asm ("x1") = old2; \
- register unsigned long x2 asm ("x2") = new1; \
- register unsigned long x3 asm ("x3") = new2; \
- register unsigned long x4 asm ("x4") = (unsigned long)ptr; \
- \
- asm volatile( \
- __LSE_PREAMBLE \
- " casp" #mb "\t%[old1], %[old2], %[new1], %[new2], %[v]\n"\
- " eor %[old1], %[old1], %[oldval1]\n" \
- " eor %[old2], %[old2], %[oldval2]\n" \
- " orr %[old1], %[old1], %[old2]" \
- : [old1] "+&r" (x0), [old2] "+&r" (x1), \
- [v] "+Q" (*(__uint128_t *)ptr) \
- : [new1] "r" (x2), [new2] "r" (x3), [ptr] "r" (x4), \
- [oldval1] "r" (oldval1), [oldval2] "r" (oldval2) \
- : cl); \
- \
- return x0; \
-}
-
-__CMPXCHG_DBL( , )
-__CMPXCHG_DBL(_mb, al, "memory")
-
-#undef __CMPXCHG_DBL
-
#define __CMPXCHG128(name, mb, cl...) \
static __always_inline u128 \
__lse__cmpxchg128##name(volatile u128 *ptr, u128 old, u128 new) \
--- a/arch/arm64/include/asm/cmpxchg.h
+++ b/arch/arm64/include/asm/cmpxchg.h
@@ -130,22 +130,6 @@ __CMPXCHG_CASE(mb_, 64)
#undef __CMPXCHG_CASE
-#define __CMPXCHG_DBL(name) \
-static inline long __cmpxchg_double##name(unsigned long old1, \
- unsigned long old2, \
- unsigned long new1, \
- unsigned long new2, \
- volatile void *ptr) \
-{ \
- return __lse_ll_sc_body(_cmpxchg_double##name, \
- old1, old2, new1, new2, ptr); \
-}
-
-__CMPXCHG_DBL( )
-__CMPXCHG_DBL(_mb)
-
-#undef __CMPXCHG_DBL
-
#define __CMPXCHG128(name) \
static inline u128 __cmpxchg128##name(volatile u128 *ptr, \
u128 old, u128 new) \
@@ -211,36 +195,6 @@ __CMPXCHG_GEN(_mb)
#define arch_cmpxchg64 arch_cmpxchg
#define arch_cmpxchg64_local arch_cmpxchg_local
-/* cmpxchg_double */
-#define system_has_cmpxchg_double() 1
-
-#define __cmpxchg_double_check(ptr1, ptr2) \
-({ \
- if (sizeof(*(ptr1)) != 8) \
- BUILD_BUG(); \
- VM_BUG_ON((unsigned long *)(ptr2) - (unsigned long *)(ptr1) != 1); \
-})
-
-#define arch_cmpxchg_double(ptr1, ptr2, o1, o2, n1, n2) \
-({ \
- int __ret; \
- __cmpxchg_double_check(ptr1, ptr2); \
- __ret = !__cmpxchg_double_mb((unsigned long)(o1), (unsigned long)(o2), \
- (unsigned long)(n1), (unsigned long)(n2), \
- ptr1); \
- __ret; \
-})
-
-#define arch_cmpxchg_double_local(ptr1, ptr2, o1, o2, n1, n2) \
-({ \
- int __ret; \
- __cmpxchg_double_check(ptr1, ptr2); \
- __ret = !__cmpxchg_double((unsigned long)(o1), (unsigned long)(o2), \
- (unsigned long)(n1), (unsigned long)(n2), \
- ptr1); \
- __ret; \
-})
-
/* cmpxchg128 */
#define system_has_cmpxchg128() 1
--- a/arch/arm64/include/asm/percpu.h
+++ b/arch/arm64/include/asm/percpu.h
@@ -145,16 +145,6 @@ PERCPU_RET_OP(add, add, ldadd)
* preemption point when TIF_NEED_RESCHED gets set while preemption is
* disabled.
*/
-#define this_cpu_cmpxchg_double_8(ptr1, ptr2, o1, o2, n1, n2) \
-({ \
- int __ret; \
- preempt_disable_notrace(); \
- __ret = cmpxchg_double_local( raw_cpu_ptr(&(ptr1)), \
- raw_cpu_ptr(&(ptr2)), \
- o1, o2, n1, n2); \
- preempt_enable_notrace(); \
- __ret; \
-})
#define _pcp_protect(op, pcp, ...) \
({ \
--- a/arch/s390/include/asm/cmpxchg.h
+++ b/arch/s390/include/asm/cmpxchg.h
@@ -190,40 +190,6 @@ static __always_inline unsigned long __c
#define arch_cmpxchg_local arch_cmpxchg
#define arch_cmpxchg64_local arch_cmpxchg
-#define system_has_cmpxchg_double() 1
-
-static __always_inline int __cmpxchg_double(unsigned long p1, unsigned long p2,
- unsigned long o1, unsigned long o2,
- unsigned long n1, unsigned long n2)
-{
- union register_pair old = { .even = o1, .odd = o2, };
- union register_pair new = { .even = n1, .odd = n2, };
- int cc;
-
- asm volatile(
- " cdsg %[old],%[new],%[ptr]\n"
- " ipm %[cc]\n"
- " srl %[cc],28\n"
- : [cc] "=&d" (cc), [old] "+&d" (old.pair)
- : [new] "d" (new.pair),
- [ptr] "QS" (*(unsigned long *)p1), "Q" (*(unsigned long *)p2)
- : "memory", "cc");
- return !cc;
-}
-
-#define arch_cmpxchg_double(p1, p2, o1, o2, n1, n2) \
-({ \
- typeof(p1) __p1 = (p1); \
- typeof(p2) __p2 = (p2); \
- \
- BUILD_BUG_ON(sizeof(*(p1)) != sizeof(long)); \
- BUILD_BUG_ON(sizeof(*(p2)) != sizeof(long)); \
- VM_BUG_ON((unsigned long)((__p1) + 1) != (unsigned long)(__p2));\
- __cmpxchg_double((unsigned long)__p1, (unsigned long)__p2, \
- (unsigned long)(o1), (unsigned long)(o2), \
- (unsigned long)(n1), (unsigned long)(n2)); \
-})
-
#define system_has_cmpxchg128() 1
static __always_inline u128 arch_cmpxchg128(volatile u128 *ptr, u128 old, u128 new)
--- a/arch/s390/include/asm/percpu.h
+++ b/arch/s390/include/asm/percpu.h
@@ -180,24 +180,6 @@
#define this_cpu_xchg_4(pcp, nval) arch_this_cpu_xchg(pcp, nval)
#define this_cpu_xchg_8(pcp, nval) arch_this_cpu_xchg(pcp, nval)
-#define arch_this_cpu_cmpxchg_double(pcp1, pcp2, o1, o2, n1, n2) \
-({ \
- typeof(pcp1) *p1__; \
- typeof(pcp2) *p2__; \
- int ret__; \
- \
- preempt_disable_notrace(); \
- p1__ = raw_cpu_ptr(&(pcp1)); \
- p2__ = raw_cpu_ptr(&(pcp2)); \
- ret__ = __cmpxchg_double((unsigned long)p1__, (unsigned long)p2__, \
- (unsigned long)(o1), (unsigned long)(o2), \
- (unsigned long)(n1), (unsigned long)(n2)); \
- preempt_enable_notrace(); \
- ret__; \
-})
-
-#define this_cpu_cmpxchg_double_8 arch_this_cpu_cmpxchg_double
-
#include <asm-generic/percpu.h>
#endif /* __ARCH_S390_PERCPU__ */
--- a/arch/x86/include/asm/cmpxchg.h
+++ b/arch/x86/include/asm/cmpxchg.h
@@ -239,29 +239,4 @@ extern void __add_wrong_size(void)
#define __xadd(ptr, inc, lock) __xchg_op((ptr), (inc), xadd, lock)
#define xadd(ptr, inc) __xadd((ptr), (inc), LOCK_PREFIX)
-#define __cmpxchg_double(pfx, p1, p2, o1, o2, n1, n2) \
-({ \
- bool __ret; \
- __typeof__(*(p1)) __old1 = (o1), __new1 = (n1); \
- __typeof__(*(p2)) __old2 = (o2), __new2 = (n2); \
- BUILD_BUG_ON(sizeof(*(p1)) != sizeof(long)); \
- BUILD_BUG_ON(sizeof(*(p2)) != sizeof(long)); \
- VM_BUG_ON((unsigned long)(p1) % (2 * sizeof(long))); \
- VM_BUG_ON((unsigned long)((p1) + 1) != (unsigned long)(p2)); \
- asm volatile(pfx "cmpxchg%c5b %1" \
- CC_SET(e) \
- : CC_OUT(e) (__ret), \
- "+m" (*(p1)), "+m" (*(p2)), \
- "+a" (__old1), "+d" (__old2) \
- : "i" (2 * sizeof(long)), \
- "b" (__new1), "c" (__new2)); \
- __ret; \
-})
-
-#define arch_cmpxchg_double(p1, p2, o1, o2, n1, n2) \
- __cmpxchg_double(LOCK_PREFIX, p1, p2, o1, o2, n1, n2)
-
-#define arch_cmpxchg_double_local(p1, p2, o1, o2, n1, n2) \
- __cmpxchg_double(, p1, p2, o1, o2, n1, n2)
-
#endif /* ASM_X86_CMPXCHG_H */
--- a/arch/x86/include/asm/cmpxchg_32.h
+++ b/arch/x86/include/asm/cmpxchg_32.h
@@ -103,7 +103,6 @@ static inline bool __try_cmpxchg64(volat
#endif
-#define system_has_cmpxchg_double() boot_cpu_has(X86_FEATURE_CX8)
#define system_has_cmpxchg64() boot_cpu_has(X86_FEATURE_CX8)
#endif /* _ASM_X86_CMPXCHG_32_H */
--- a/arch/x86/include/asm/cmpxchg_64.h
+++ b/arch/x86/include/asm/cmpxchg_64.h
@@ -81,7 +81,6 @@ static __always_inline bool arch_try_cmp
return __arch_try_cmpxchg128(ptr, oldp, new,);
}
-#define system_has_cmpxchg_double() boot_cpu_has(X86_FEATURE_CX16)
#define system_has_cmpxchg128() boot_cpu_has(X86_FEATURE_CX16)
#endif /* _ASM_X86_CMPXCHG_64_H */
--- a/arch/x86/include/asm/percpu.h
+++ b/arch/x86/include/asm/percpu.h
@@ -351,23 +351,6 @@ do { \
#define this_cpu_cmpxchg_2(pcp, oval, nval) percpu_cmpxchg_op(2, volatile, pcp, oval, nval)
#define this_cpu_cmpxchg_4(pcp, oval, nval) percpu_cmpxchg_op(4, volatile, pcp, oval, nval)
-#ifdef CONFIG_X86_CMPXCHG64
-#define percpu_cmpxchg8b_double(pcp1, pcp2, o1, o2, n1, n2) \
-({ \
- bool __ret; \
- typeof(pcp1) __o1 = (o1), __n1 = (n1); \
- typeof(pcp2) __o2 = (o2), __n2 = (n2); \
- asm volatile("cmpxchg8b "__percpu_arg(1) \
- CC_SET(z) \
- : CC_OUT(z) (__ret), "+m" (pcp1), "+m" (pcp2), "+a" (__o1), "+d" (__o2) \
- : "b" (__n1), "c" (__n2)); \
- __ret; \
-})
-
-#define raw_cpu_cmpxchg_double_4 percpu_cmpxchg8b_double
-#define this_cpu_cmpxchg_double_4 percpu_cmpxchg8b_double
-#endif /* CONFIG_X86_CMPXCHG64 */
-
/*
* Per cpu atomic 64 bit operations are only available under 64 bit.
* 32 bit must fall back to generic operations.
@@ -390,31 +373,6 @@ do { \
#define this_cpu_add_return_8(pcp, val) percpu_add_return_op(8, volatile, pcp, val)
#define this_cpu_xchg_8(pcp, nval) percpu_xchg_op(8, volatile, pcp, nval)
#define this_cpu_cmpxchg_8(pcp, oval, nval) percpu_cmpxchg_op(8, volatile, pcp, oval, nval)
-
-/*
- * Pretty complex macro to generate cmpxchg16 instruction. The instruction
- * is not supported on early AMD64 processors so we must be able to emulate
- * it in software. The address used in the cmpxchg16 instruction must be
- * aligned to a 16 byte boundary.
- */
-#define percpu_cmpxchg16b_double(pcp1, pcp2, o1, o2, n1, n2) \
-({ \
- bool __ret; \
- typeof(pcp1) __o1 = (o1), __n1 = (n1); \
- typeof(pcp2) __o2 = (o2), __n2 = (n2); \
- asm volatile (ALTERNATIVE("leaq %P1, %%rsi; call this_cpu_cmpxchg16b_emu", \
- "cmpxchg16b " __percpu_arg(1), X86_FEATURE_CX16) \
- "setz %0" \
- : "=a" (__ret), "+m" (pcp1) \
- : "b" (__n1), "c" (__n2), \
- "a" (__o1), "d" (__o2) \
- : "memory", "rsi"); \
- __ret; \
-})
-
-#define raw_cpu_cmpxchg_double_8 percpu_cmpxchg16b_double
-#define this_cpu_cmpxchg_double_8 percpu_cmpxchg16b_double
-
#endif
static __always_inline bool x86_this_cpu_constant_test_bit(unsigned int nr,
--- a/include/asm-generic/percpu.h
+++ b/include/asm-generic/percpu.h
@@ -120,19 +120,6 @@ do { \
__old; \
})
-#define raw_cpu_generic_cmpxchg_double(pcp1, pcp2, oval1, oval2, nval1, nval2) \
-({ \
- typeof(pcp1) *__p1 = raw_cpu_ptr(&(pcp1)); \
- typeof(pcp2) *__p2 = raw_cpu_ptr(&(pcp2)); \
- int __ret = 0; \
- if (*__p1 == (oval1) && *__p2 == (oval2)) { \
- *__p1 = nval1; \
- *__p2 = nval2; \
- __ret = 1; \
- } \
- (__ret); \
-})
-
#define __this_cpu_generic_read_nopreempt(pcp) \
({ \
typeof(pcp) ___ret; \
@@ -211,17 +198,6 @@ do { \
__ret; \
})
-#define this_cpu_generic_cmpxchg_double(pcp1, pcp2, oval1, oval2, nval1, nval2) \
-({ \
- int __ret; \
- unsigned long __flags; \
- raw_local_irq_save(__flags); \
- __ret = raw_cpu_generic_cmpxchg_double(pcp1, pcp2, \
- oval1, oval2, nval1, nval2); \
- raw_local_irq_restore(__flags); \
- __ret; \
-})
-
#ifndef raw_cpu_read_1
#define raw_cpu_read_1(pcp) raw_cpu_generic_read(pcp)
#endif
@@ -424,23 +400,6 @@ do { \
raw_cpu_generic_cmpxchg(pcp, oval, nval)
#endif
-#ifndef raw_cpu_cmpxchg_double_1
-#define raw_cpu_cmpxchg_double_1(pcp1, pcp2, oval1, oval2, nval1, nval2) \
- raw_cpu_generic_cmpxchg_double(pcp1, pcp2, oval1, oval2, nval1, nval2)
-#endif
-#ifndef raw_cpu_cmpxchg_double_2
-#define raw_cpu_cmpxchg_double_2(pcp1, pcp2, oval1, oval2, nval1, nval2) \
- raw_cpu_generic_cmpxchg_double(pcp1, pcp2, oval1, oval2, nval1, nval2)
-#endif
-#ifndef raw_cpu_cmpxchg_double_4
-#define raw_cpu_cmpxchg_double_4(pcp1, pcp2, oval1, oval2, nval1, nval2) \
- raw_cpu_generic_cmpxchg_double(pcp1, pcp2, oval1, oval2, nval1, nval2)
-#endif
-#ifndef raw_cpu_cmpxchg_double_8
-#define raw_cpu_cmpxchg_double_8(pcp1, pcp2, oval1, oval2, nval1, nval2) \
- raw_cpu_generic_cmpxchg_double(pcp1, pcp2, oval1, oval2, nval1, nval2)
-#endif
-
#ifndef this_cpu_read_1
#define this_cpu_read_1(pcp) this_cpu_generic_read(pcp)
#endif
@@ -641,21 +600,4 @@ do { \
this_cpu_generic_cmpxchg(pcp, oval, nval)
#endif
-#ifndef this_cpu_cmpxchg_double_1
-#define this_cpu_cmpxchg_double_1(pcp1, pcp2, oval1, oval2, nval1, nval2) \
- this_cpu_generic_cmpxchg_double(pcp1, pcp2, oval1, oval2, nval1, nval2)
-#endif
-#ifndef this_cpu_cmpxchg_double_2
-#define this_cpu_cmpxchg_double_2(pcp1, pcp2, oval1, oval2, nval1, nval2) \
- this_cpu_generic_cmpxchg_double(pcp1, pcp2, oval1, oval2, nval1, nval2)
-#endif
-#ifndef this_cpu_cmpxchg_double_4
-#define this_cpu_cmpxchg_double_4(pcp1, pcp2, oval1, oval2, nval1, nval2) \
- this_cpu_generic_cmpxchg_double(pcp1, pcp2, oval1, oval2, nval1, nval2)
-#endif
-#ifndef this_cpu_cmpxchg_double_8
-#define this_cpu_cmpxchg_double_8(pcp1, pcp2, oval1, oval2, nval1, nval2) \
- this_cpu_generic_cmpxchg_double(pcp1, pcp2, oval1, oval2, nval1, nval2)
-#endif
-
#endif /* _ASM_GENERIC_PERCPU_H_ */
--- a/include/linux/atomic/atomic-instrumented.h
+++ b/include/linux/atomic/atomic-instrumented.h
@@ -2234,21 +2234,6 @@ atomic_long_dec_if_positive(atomic_long_
arch_try_cmpxchg128_local(__ai_ptr, __ai_oldp, __VA_ARGS__); \
})
-#define cmpxchg_double(ptr, ...) \
-({ \
- typeof(ptr) __ai_ptr = (ptr); \
- kcsan_mb(); \
- instrument_atomic_read_write(__ai_ptr, 2 * sizeof(*__ai_ptr)); \
- arch_cmpxchg_double(__ai_ptr, __VA_ARGS__); \
-})
-
-
-#define cmpxchg_double_local(ptr, ...) \
-({ \
- typeof(ptr) __ai_ptr = (ptr); \
- instrument_atomic_read_write(__ai_ptr, 2 * sizeof(*__ai_ptr)); \
- arch_cmpxchg_double_local(__ai_ptr, __VA_ARGS__); \
-})
#endif /* _LINUX_ATOMIC_INSTRUMENTED_H */
-// 82d1be694fab30414527d0877c29fa75ed5a0b74
+// 3611991b015450e119bcd7417a9431af7f3ba13c
--- a/include/linux/percpu-defs.h
+++ b/include/linux/percpu-defs.h
@@ -358,33 +358,6 @@ static __always_inline void __this_cpu_p
pscr2_ret__; \
})
-/*
- * Special handling for cmpxchg_double. cmpxchg_double is passed two
- * percpu variables. The first has to be aligned to a double word
- * boundary and the second has to follow directly thereafter.
- * We enforce this on all architectures even if they don't support
- * a double cmpxchg instruction, since it's a cheap requirement, and it
- * avoids breaking the requirement for architectures with the instruction.
- */
-#define __pcpu_double_call_return_bool(stem, pcp1, pcp2, ...) \
-({ \
- bool pdcrb_ret__; \
- __verify_pcpu_ptr(&(pcp1)); \
- BUILD_BUG_ON(sizeof(pcp1) != sizeof(pcp2)); \
- VM_BUG_ON((unsigned long)(&(pcp1)) % (2 * sizeof(pcp1))); \
- VM_BUG_ON((unsigned long)(&(pcp2)) != \
- (unsigned long)(&(pcp1)) + sizeof(pcp1)); \
- switch(sizeof(pcp1)) { \
- case 1: pdcrb_ret__ = stem##1(pcp1, pcp2, __VA_ARGS__); break; \
- case 2: pdcrb_ret__ = stem##2(pcp1, pcp2, __VA_ARGS__); break; \
- case 4: pdcrb_ret__ = stem##4(pcp1, pcp2, __VA_ARGS__); break; \
- case 8: pdcrb_ret__ = stem##8(pcp1, pcp2, __VA_ARGS__); break; \
- default: \
- __bad_size_call_parameter(); break; \
- } \
- pdcrb_ret__; \
-})
-
#define __pcpu_size_call(stem, variable, ...) \
do { \
__verify_pcpu_ptr(&(variable)); \
@@ -443,9 +416,6 @@ do { \
__pcpu_size_call_return2(raw_cpu_cmpxchg_, pcp, oval, nval)
#define raw_cpu_try_cmpxchg(pcp, ovalp, nval) \
__pcpu_size_call_return2bool(raw_cpu_try_cmpxchg_, pcp, ovalp, nval)
-#define raw_cpu_cmpxchg_double(pcp1, pcp2, oval1, oval2, nval1, nval2) \
- __pcpu_double_call_return_bool(raw_cpu_cmpxchg_double_, pcp1, pcp2, oval1, oval2, nval1, nval2)
-
#define raw_cpu_sub(pcp, val) raw_cpu_add(pcp, -(val))
#define raw_cpu_inc(pcp) raw_cpu_add(pcp, 1)
#define raw_cpu_dec(pcp) raw_cpu_sub(pcp, 1)
@@ -505,11 +475,6 @@ do { \
raw_cpu_cmpxchg(pcp, oval, nval); \
})
-#define __this_cpu_cmpxchg_double(pcp1, pcp2, oval1, oval2, nval1, nval2) \
-({ __this_cpu_preempt_check("cmpxchg_double"); \
- raw_cpu_cmpxchg_double(pcp1, pcp2, oval1, oval2, nval1, nval2); \
-})
-
#define __this_cpu_sub(pcp, val) __this_cpu_add(pcp, -(typeof(pcp))(val))
#define __this_cpu_inc(pcp) __this_cpu_add(pcp, 1)
#define __this_cpu_dec(pcp) __this_cpu_sub(pcp, 1)
@@ -532,9 +497,6 @@ do { \
__pcpu_size_call_return2(this_cpu_cmpxchg_, pcp, oval, nval)
#define this_cpu_try_cmpxchg(pcp, ovalp, nval) \
__pcpu_size_call_return2bool(this_cpu_try_cmpxchg_, pcp, ovalp, nval)
-#define this_cpu_cmpxchg_double(pcp1, pcp2, oval1, oval2, nval1, nval2) \
- __pcpu_double_call_return_bool(this_cpu_cmpxchg_double_, pcp1, pcp2, oval1, oval2, nval1, nval2)
-
#define this_cpu_sub(pcp, val) this_cpu_add(pcp, -(typeof(pcp))(val))
#define this_cpu_inc(pcp) this_cpu_add(pcp, 1)
#define this_cpu_dec(pcp) this_cpu_sub(pcp, 1)
--- a/scripts/atomic/gen-atomic-instrumented.sh
+++ b/scripts/atomic/gen-atomic-instrumented.sh
@@ -84,7 +84,6 @@ gen_xchg()
{
local xchg="$1"; shift
local order="$1"; shift
- local mult="$1"; shift
kcsan_barrier=""
if [ "${xchg%_local}" = "${xchg}" ]; then
@@ -104,8 +103,8 @@ cat <<EOF
EOF
[ -n "$kcsan_barrier" ] && printf "\t${kcsan_barrier}; \\\\\n"
cat <<EOF
- instrument_atomic_read_write(__ai_ptr, ${mult}sizeof(*__ai_ptr)); \\
- instrument_read_write(__ai_oldp, ${mult}sizeof(*__ai_oldp)); \\
+ instrument_atomic_read_write(__ai_ptr, sizeof(*__ai_ptr)); \\
+ instrument_read_write(__ai_oldp, sizeof(*__ai_oldp)); \\
arch_${xchg}${order}(__ai_ptr, __ai_oldp, __VA_ARGS__); \\
})
EOF
@@ -168,22 +167,16 @@ done
for xchg in "xchg" "cmpxchg" "cmpxchg64" "cmpxchg128" "try_cmpxchg" "try_cmpxchg64" "try_cmpxchg128"; do
for order in "" "_acquire" "_release" "_relaxed"; do
- gen_xchg "${xchg}" "${order}" ""
+ gen_xchg "${xchg}" "${order}"
printf "\n"
done
done
for xchg in "cmpxchg_local" "cmpxchg64_local" "cmpxchg128_local" "sync_cmpxchg" "try_cmpxchg_local" "try_cmpxchg64_local" "try_cmpxchg128_local"; do
- gen_xchg "${xchg}" "" ""
+ gen_xchg "${xchg}" ""
printf "\n"
done
-gen_xchg "cmpxchg_double" "" "2 * "
-
-printf "\n\n"
-
-gen_xchg "cmpxchg_double_local" "" "2 * "
-
cat <<EOF
#endif /* _LINUX_ATOMIC_INSTRUMENTED_H */
^ permalink raw reply [flat|nested] 47+ messages in thread* [tip: locking/core] arch: Remove cmpxchg_double
2023-05-31 13:08 ` [PATCH 11/12] arch: Remove cmpxchg_double Peter Zijlstra
@ 2023-06-05 7:42 ` tip-bot2 for Peter Zijlstra
0 siblings, 0 replies; 47+ messages in thread
From: tip-bot2 for Peter Zijlstra @ 2023-06-05 7:42 UTC (permalink / raw)
To: linux-tip-commits
Cc: Peter Zijlstra (Intel),
Arnd Bergmann, Mark Rutland, Heiko Carstens, x86, linux-kernel
The following commit has been merged into the locking/core branch of tip:
Commit-ID: febe950dbfb464799beb0339cc6fb10699f4a5da
Gitweb: https://git.kernel.org/tip/febe950dbfb464799beb0339cc6fb10699f4a5da
Author: Peter Zijlstra <peterz@infradead.org>
AuthorDate: Wed, 31 May 2023 15:08:44 +02:00
Committer: Peter Zijlstra <peterz@infradead.org>
CommitterDate: Mon, 05 Jun 2023 09:36:39 +02:00
arch: Remove cmpxchg_double
No moar users, remove the monster.
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Reviewed-by: Arnd Bergmann <arnd@arndb.de>
Reviewed-by: Mark Rutland <mark.rutland@arm.com>
Acked-by: Heiko Carstens <hca@linux.ibm.com>
Tested-by: Mark Rutland <mark.rutland@arm.com>
Link: https://lore.kernel.org/r/20230531132323.991907085@infradead.org
---
Documentation/core-api/this_cpu_ops.rst | 2 +-
arch/arm64/include/asm/atomic_ll_sc.h | 33 +------------
arch/arm64/include/asm/atomic_lse.h | 36 +-------------
arch/arm64/include/asm/cmpxchg.h | 46 +-----------------
arch/arm64/include/asm/percpu.h | 10 +----
arch/s390/include/asm/cmpxchg.h | 34 +------------
arch/s390/include/asm/percpu.h | 18 +-------
arch/x86/include/asm/cmpxchg.h | 25 +---------
arch/x86/include/asm/cmpxchg_32.h | 1 +-
arch/x86/include/asm/cmpxchg_64.h | 1 +-
arch/x86/include/asm/percpu.h | 42 +---------------
include/asm-generic/percpu.h | 58 +---------------------
include/linux/atomic/atomic-instrumented.h | 17 +------
include/linux/percpu-defs.h | 38 +--------------
scripts/atomic/gen-atomic-instrumented.sh | 15 +----
15 files changed, 5 insertions(+), 371 deletions(-)
diff --git a/Documentation/core-api/this_cpu_ops.rst b/Documentation/core-api/this_cpu_ops.rst
index 5cb8b88..91acbcf 100644
--- a/Documentation/core-api/this_cpu_ops.rst
+++ b/Documentation/core-api/this_cpu_ops.rst
@@ -53,7 +53,6 @@ preemption and interrupts::
this_cpu_add_return(pcp, val)
this_cpu_xchg(pcp, nval)
this_cpu_cmpxchg(pcp, oval, nval)
- this_cpu_cmpxchg_double(pcp1, pcp2, oval1, oval2, nval1, nval2)
this_cpu_sub(pcp, val)
this_cpu_inc(pcp)
this_cpu_dec(pcp)
@@ -242,7 +241,6 @@ safe::
__this_cpu_add_return(pcp, val)
__this_cpu_xchg(pcp, nval)
__this_cpu_cmpxchg(pcp, oval, nval)
- __this_cpu_cmpxchg_double(pcp1, pcp2, oval1, oval2, nval1, nval2)
__this_cpu_sub(pcp, val)
__this_cpu_inc(pcp)
__this_cpu_dec(pcp)
diff --git a/arch/arm64/include/asm/atomic_ll_sc.h b/arch/arm64/include/asm/atomic_ll_sc.h
index b100a49..89d2ba2 100644
--- a/arch/arm64/include/asm/atomic_ll_sc.h
+++ b/arch/arm64/include/asm/atomic_ll_sc.h
@@ -294,39 +294,6 @@ __CMPXCHG_CASE( , , mb_, 64, dmb ish, , l, "memory", L)
#undef __CMPXCHG_CASE
-#define __CMPXCHG_DBL(name, mb, rel, cl) \
-static __always_inline long \
-__ll_sc__cmpxchg_double##name(unsigned long old1, \
- unsigned long old2, \
- unsigned long new1, \
- unsigned long new2, \
- volatile void *ptr) \
-{ \
- unsigned long tmp, ret; \
- \
- asm volatile("// __cmpxchg_double" #name "\n" \
- " prfm pstl1strm, %2\n" \
- "1: ldxp %0, %1, %2\n" \
- " eor %0, %0, %3\n" \
- " eor %1, %1, %4\n" \
- " orr %1, %0, %1\n" \
- " cbnz %1, 2f\n" \
- " st" #rel "xp %w0, %5, %6, %2\n" \
- " cbnz %w0, 1b\n" \
- " " #mb "\n" \
- "2:" \
- : "=&r" (tmp), "=&r" (ret), "+Q" (*(__uint128_t *)ptr) \
- : "r" (old1), "r" (old2), "r" (new1), "r" (new2) \
- : cl); \
- \
- return ret; \
-}
-
-__CMPXCHG_DBL( , , , )
-__CMPXCHG_DBL(_mb, dmb ish, l, "memory")
-
-#undef __CMPXCHG_DBL
-
union __u128_halves {
u128 full;
struct {
diff --git a/arch/arm64/include/asm/atomic_lse.h b/arch/arm64/include/asm/atomic_lse.h
index c781281..87f568a 100644
--- a/arch/arm64/include/asm/atomic_lse.h
+++ b/arch/arm64/include/asm/atomic_lse.h
@@ -281,42 +281,6 @@ __CMPXCHG_CASE(x, , mb_, 64, al, "memory")
#undef __CMPXCHG_CASE
-#define __CMPXCHG_DBL(name, mb, cl...) \
-static __always_inline long \
-__lse__cmpxchg_double##name(unsigned long old1, \
- unsigned long old2, \
- unsigned long new1, \
- unsigned long new2, \
- volatile void *ptr) \
-{ \
- unsigned long oldval1 = old1; \
- unsigned long oldval2 = old2; \
- register unsigned long x0 asm ("x0") = old1; \
- register unsigned long x1 asm ("x1") = old2; \
- register unsigned long x2 asm ("x2") = new1; \
- register unsigned long x3 asm ("x3") = new2; \
- register unsigned long x4 asm ("x4") = (unsigned long)ptr; \
- \
- asm volatile( \
- __LSE_PREAMBLE \
- " casp" #mb "\t%[old1], %[old2], %[new1], %[new2], %[v]\n"\
- " eor %[old1], %[old1], %[oldval1]\n" \
- " eor %[old2], %[old2], %[oldval2]\n" \
- " orr %[old1], %[old1], %[old2]" \
- : [old1] "+&r" (x0), [old2] "+&r" (x1), \
- [v] "+Q" (*(__uint128_t *)ptr) \
- : [new1] "r" (x2), [new2] "r" (x3), [ptr] "r" (x4), \
- [oldval1] "r" (oldval1), [oldval2] "r" (oldval2) \
- : cl); \
- \
- return x0; \
-}
-
-__CMPXCHG_DBL( , )
-__CMPXCHG_DBL(_mb, al, "memory")
-
-#undef __CMPXCHG_DBL
-
#define __CMPXCHG128(name, mb, cl...) \
static __always_inline u128 \
__lse__cmpxchg128##name(volatile u128 *ptr, u128 old, u128 new) \
diff --git a/arch/arm64/include/asm/cmpxchg.h b/arch/arm64/include/asm/cmpxchg.h
index 097b832..d7a5407 100644
--- a/arch/arm64/include/asm/cmpxchg.h
+++ b/arch/arm64/include/asm/cmpxchg.h
@@ -130,22 +130,6 @@ __CMPXCHG_CASE(mb_, 64)
#undef __CMPXCHG_CASE
-#define __CMPXCHG_DBL(name) \
-static inline long __cmpxchg_double##name(unsigned long old1, \
- unsigned long old2, \
- unsigned long new1, \
- unsigned long new2, \
- volatile void *ptr) \
-{ \
- return __lse_ll_sc_body(_cmpxchg_double##name, \
- old1, old2, new1, new2, ptr); \
-}
-
-__CMPXCHG_DBL( )
-__CMPXCHG_DBL(_mb)
-
-#undef __CMPXCHG_DBL
-
#define __CMPXCHG128(name) \
static inline u128 __cmpxchg128##name(volatile u128 *ptr, \
u128 old, u128 new) \
@@ -211,36 +195,6 @@ __CMPXCHG_GEN(_mb)
#define arch_cmpxchg64 arch_cmpxchg
#define arch_cmpxchg64_local arch_cmpxchg_local
-/* cmpxchg_double */
-#define system_has_cmpxchg_double() 1
-
-#define __cmpxchg_double_check(ptr1, ptr2) \
-({ \
- if (sizeof(*(ptr1)) != 8) \
- BUILD_BUG(); \
- VM_BUG_ON((unsigned long *)(ptr2) - (unsigned long *)(ptr1) != 1); \
-})
-
-#define arch_cmpxchg_double(ptr1, ptr2, o1, o2, n1, n2) \
-({ \
- int __ret; \
- __cmpxchg_double_check(ptr1, ptr2); \
- __ret = !__cmpxchg_double_mb((unsigned long)(o1), (unsigned long)(o2), \
- (unsigned long)(n1), (unsigned long)(n2), \
- ptr1); \
- __ret; \
-})
-
-#define arch_cmpxchg_double_local(ptr1, ptr2, o1, o2, n1, n2) \
-({ \
- int __ret; \
- __cmpxchg_double_check(ptr1, ptr2); \
- __ret = !__cmpxchg_double((unsigned long)(o1), (unsigned long)(o2), \
- (unsigned long)(n1), (unsigned long)(n2), \
- ptr1); \
- __ret; \
-})
-
/* cmpxchg128 */
#define system_has_cmpxchg128() 1
diff --git a/arch/arm64/include/asm/percpu.h b/arch/arm64/include/asm/percpu.h
index 2549829..9abcc8e 100644
--- a/arch/arm64/include/asm/percpu.h
+++ b/arch/arm64/include/asm/percpu.h
@@ -145,16 +145,6 @@ PERCPU_RET_OP(add, add, ldadd)
* preemption point when TIF_NEED_RESCHED gets set while preemption is
* disabled.
*/
-#define this_cpu_cmpxchg_double_8(ptr1, ptr2, o1, o2, n1, n2) \
-({ \
- int __ret; \
- preempt_disable_notrace(); \
- __ret = cmpxchg_double_local( raw_cpu_ptr(&(ptr1)), \
- raw_cpu_ptr(&(ptr2)), \
- o1, o2, n1, n2); \
- preempt_enable_notrace(); \
- __ret; \
-})
#define _pcp_protect(op, pcp, ...) \
({ \
diff --git a/arch/s390/include/asm/cmpxchg.h b/arch/s390/include/asm/cmpxchg.h
index 7517376..aae0315 100644
--- a/arch/s390/include/asm/cmpxchg.h
+++ b/arch/s390/include/asm/cmpxchg.h
@@ -190,40 +190,6 @@ static __always_inline unsigned long __cmpxchg(unsigned long address,
#define arch_cmpxchg_local arch_cmpxchg
#define arch_cmpxchg64_local arch_cmpxchg
-#define system_has_cmpxchg_double() 1
-
-static __always_inline int __cmpxchg_double(unsigned long p1, unsigned long p2,
- unsigned long o1, unsigned long o2,
- unsigned long n1, unsigned long n2)
-{
- union register_pair old = { .even = o1, .odd = o2, };
- union register_pair new = { .even = n1, .odd = n2, };
- int cc;
-
- asm volatile(
- " cdsg %[old],%[new],%[ptr]\n"
- " ipm %[cc]\n"
- " srl %[cc],28\n"
- : [cc] "=&d" (cc), [old] "+&d" (old.pair)
- : [new] "d" (new.pair),
- [ptr] "QS" (*(unsigned long *)p1), "Q" (*(unsigned long *)p2)
- : "memory", "cc");
- return !cc;
-}
-
-#define arch_cmpxchg_double(p1, p2, o1, o2, n1, n2) \
-({ \
- typeof(p1) __p1 = (p1); \
- typeof(p2) __p2 = (p2); \
- \
- BUILD_BUG_ON(sizeof(*(p1)) != sizeof(long)); \
- BUILD_BUG_ON(sizeof(*(p2)) != sizeof(long)); \
- VM_BUG_ON((unsigned long)((__p1) + 1) != (unsigned long)(__p2));\
- __cmpxchg_double((unsigned long)__p1, (unsigned long)__p2, \
- (unsigned long)(o1), (unsigned long)(o2), \
- (unsigned long)(n1), (unsigned long)(n2)); \
-})
-
#define system_has_cmpxchg128() 1
static __always_inline u128 arch_cmpxchg128(volatile u128 *ptr, u128 old, u128 new)
diff --git a/arch/s390/include/asm/percpu.h b/arch/s390/include/asm/percpu.h
index 5603402..264095d 100644
--- a/arch/s390/include/asm/percpu.h
+++ b/arch/s390/include/asm/percpu.h
@@ -180,24 +180,6 @@
#define this_cpu_xchg_4(pcp, nval) arch_this_cpu_xchg(pcp, nval)
#define this_cpu_xchg_8(pcp, nval) arch_this_cpu_xchg(pcp, nval)
-#define arch_this_cpu_cmpxchg_double(pcp1, pcp2, o1, o2, n1, n2) \
-({ \
- typeof(pcp1) *p1__; \
- typeof(pcp2) *p2__; \
- int ret__; \
- \
- preempt_disable_notrace(); \
- p1__ = raw_cpu_ptr(&(pcp1)); \
- p2__ = raw_cpu_ptr(&(pcp2)); \
- ret__ = __cmpxchg_double((unsigned long)p1__, (unsigned long)p2__, \
- (unsigned long)(o1), (unsigned long)(o2), \
- (unsigned long)(n1), (unsigned long)(n2)); \
- preempt_enable_notrace(); \
- ret__; \
-})
-
-#define this_cpu_cmpxchg_double_8 arch_this_cpu_cmpxchg_double
-
#include <asm-generic/percpu.h>
#endif /* __ARCH_S390_PERCPU__ */
diff --git a/arch/x86/include/asm/cmpxchg.h b/arch/x86/include/asm/cmpxchg.h
index 540573f..d536365 100644
--- a/arch/x86/include/asm/cmpxchg.h
+++ b/arch/x86/include/asm/cmpxchg.h
@@ -239,29 +239,4 @@ extern void __add_wrong_size(void)
#define __xadd(ptr, inc, lock) __xchg_op((ptr), (inc), xadd, lock)
#define xadd(ptr, inc) __xadd((ptr), (inc), LOCK_PREFIX)
-#define __cmpxchg_double(pfx, p1, p2, o1, o2, n1, n2) \
-({ \
- bool __ret; \
- __typeof__(*(p1)) __old1 = (o1), __new1 = (n1); \
- __typeof__(*(p2)) __old2 = (o2), __new2 = (n2); \
- BUILD_BUG_ON(sizeof(*(p1)) != sizeof(long)); \
- BUILD_BUG_ON(sizeof(*(p2)) != sizeof(long)); \
- VM_BUG_ON((unsigned long)(p1) % (2 * sizeof(long))); \
- VM_BUG_ON((unsigned long)((p1) + 1) != (unsigned long)(p2)); \
- asm volatile(pfx "cmpxchg%c5b %1" \
- CC_SET(e) \
- : CC_OUT(e) (__ret), \
- "+m" (*(p1)), "+m" (*(p2)), \
- "+a" (__old1), "+d" (__old2) \
- : "i" (2 * sizeof(long)), \
- "b" (__new1), "c" (__new2)); \
- __ret; \
-})
-
-#define arch_cmpxchg_double(p1, p2, o1, o2, n1, n2) \
- __cmpxchg_double(LOCK_PREFIX, p1, p2, o1, o2, n1, n2)
-
-#define arch_cmpxchg_double_local(p1, p2, o1, o2, n1, n2) \
- __cmpxchg_double(, p1, p2, o1, o2, n1, n2)
-
#endif /* ASM_X86_CMPXCHG_H */
diff --git a/arch/x86/include/asm/cmpxchg_32.h b/arch/x86/include/asm/cmpxchg_32.h
index d12f9ed..b5731c5 100644
--- a/arch/x86/include/asm/cmpxchg_32.h
+++ b/arch/x86/include/asm/cmpxchg_32.h
@@ -103,7 +103,6 @@ static inline bool __try_cmpxchg64(volatile u64 *ptr, u64 *pold, u64 new)
#endif
-#define system_has_cmpxchg_double() boot_cpu_has(X86_FEATURE_CX8)
#define system_has_cmpxchg64() boot_cpu_has(X86_FEATURE_CX8)
#endif /* _ASM_X86_CMPXCHG_32_H */
diff --git a/arch/x86/include/asm/cmpxchg_64.h b/arch/x86/include/asm/cmpxchg_64.h
index fe29107..3e6e3ee 100644
--- a/arch/x86/include/asm/cmpxchg_64.h
+++ b/arch/x86/include/asm/cmpxchg_64.h
@@ -81,7 +81,6 @@ static __always_inline bool arch_try_cmpxchg128_local(volatile u128 *ptr, u128 *
return __arch_try_cmpxchg128(ptr, oldp, new,);
}
-#define system_has_cmpxchg_double() boot_cpu_has(X86_FEATURE_CX16)
#define system_has_cmpxchg128() boot_cpu_has(X86_FEATURE_CX16)
#endif /* _ASM_X86_CMPXCHG_64_H */
diff --git a/arch/x86/include/asm/percpu.h b/arch/x86/include/asm/percpu.h
index 4232fb2..34734d7 100644
--- a/arch/x86/include/asm/percpu.h
+++ b/arch/x86/include/asm/percpu.h
@@ -351,23 +351,6 @@ do { \
#define this_cpu_cmpxchg_2(pcp, oval, nval) percpu_cmpxchg_op(2, volatile, pcp, oval, nval)
#define this_cpu_cmpxchg_4(pcp, oval, nval) percpu_cmpxchg_op(4, volatile, pcp, oval, nval)
-#ifdef CONFIG_X86_CMPXCHG64
-#define percpu_cmpxchg8b_double(pcp1, pcp2, o1, o2, n1, n2) \
-({ \
- bool __ret; \
- typeof(pcp1) __o1 = (o1), __n1 = (n1); \
- typeof(pcp2) __o2 = (o2), __n2 = (n2); \
- asm volatile("cmpxchg8b "__percpu_arg(1) \
- CC_SET(z) \
- : CC_OUT(z) (__ret), "+m" (pcp1), "+m" (pcp2), "+a" (__o1), "+d" (__o2) \
- : "b" (__n1), "c" (__n2)); \
- __ret; \
-})
-
-#define raw_cpu_cmpxchg_double_4 percpu_cmpxchg8b_double
-#define this_cpu_cmpxchg_double_4 percpu_cmpxchg8b_double
-#endif /* CONFIG_X86_CMPXCHG64 */
-
/*
* Per cpu atomic 64 bit operations are only available under 64 bit.
* 32 bit must fall back to generic operations.
@@ -390,31 +373,6 @@ do { \
#define this_cpu_add_return_8(pcp, val) percpu_add_return_op(8, volatile, pcp, val)
#define this_cpu_xchg_8(pcp, nval) percpu_xchg_op(8, volatile, pcp, nval)
#define this_cpu_cmpxchg_8(pcp, oval, nval) percpu_cmpxchg_op(8, volatile, pcp, oval, nval)
-
-/*
- * Pretty complex macro to generate cmpxchg16 instruction. The instruction
- * is not supported on early AMD64 processors so we must be able to emulate
- * it in software. The address used in the cmpxchg16 instruction must be
- * aligned to a 16 byte boundary.
- */
-#define percpu_cmpxchg16b_double(pcp1, pcp2, o1, o2, n1, n2) \
-({ \
- bool __ret; \
- typeof(pcp1) __o1 = (o1), __n1 = (n1); \
- typeof(pcp2) __o2 = (o2), __n2 = (n2); \
- asm volatile (ALTERNATIVE("leaq %P1, %%rsi; call this_cpu_cmpxchg16b_emu", \
- "cmpxchg16b " __percpu_arg(1), X86_FEATURE_CX16) \
- "setz %0" \
- : "=a" (__ret), "+m" (pcp1) \
- : "b" (__n1), "c" (__n2), \
- "a" (__o1), "d" (__o2) \
- : "memory", "rsi"); \
- __ret; \
-})
-
-#define raw_cpu_cmpxchg_double_8 percpu_cmpxchg16b_double
-#define this_cpu_cmpxchg_double_8 percpu_cmpxchg16b_double
-
#endif
static __always_inline bool x86_this_cpu_constant_test_bit(unsigned int nr,
diff --git a/include/asm-generic/percpu.h b/include/asm-generic/percpu.h
index 5c66e44..68c410e 100644
--- a/include/asm-generic/percpu.h
+++ b/include/asm-generic/percpu.h
@@ -120,19 +120,6 @@ do { \
__old; \
})
-#define raw_cpu_generic_cmpxchg_double(pcp1, pcp2, oval1, oval2, nval1, nval2) \
-({ \
- typeof(pcp1) *__p1 = raw_cpu_ptr(&(pcp1)); \
- typeof(pcp2) *__p2 = raw_cpu_ptr(&(pcp2)); \
- int __ret = 0; \
- if (*__p1 == (oval1) && *__p2 == (oval2)) { \
- *__p1 = nval1; \
- *__p2 = nval2; \
- __ret = 1; \
- } \
- (__ret); \
-})
-
#define __this_cpu_generic_read_nopreempt(pcp) \
({ \
typeof(pcp) ___ret; \
@@ -211,17 +198,6 @@ do { \
__ret; \
})
-#define this_cpu_generic_cmpxchg_double(pcp1, pcp2, oval1, oval2, nval1, nval2) \
-({ \
- int __ret; \
- unsigned long __flags; \
- raw_local_irq_save(__flags); \
- __ret = raw_cpu_generic_cmpxchg_double(pcp1, pcp2, \
- oval1, oval2, nval1, nval2); \
- raw_local_irq_restore(__flags); \
- __ret; \
-})
-
#ifndef raw_cpu_read_1
#define raw_cpu_read_1(pcp) raw_cpu_generic_read(pcp)
#endif
@@ -395,23 +371,6 @@ do { \
raw_cpu_generic_cmpxchg(pcp, oval, nval)
#endif
-#ifndef raw_cpu_cmpxchg_double_1
-#define raw_cpu_cmpxchg_double_1(pcp1, pcp2, oval1, oval2, nval1, nval2) \
- raw_cpu_generic_cmpxchg_double(pcp1, pcp2, oval1, oval2, nval1, nval2)
-#endif
-#ifndef raw_cpu_cmpxchg_double_2
-#define raw_cpu_cmpxchg_double_2(pcp1, pcp2, oval1, oval2, nval1, nval2) \
- raw_cpu_generic_cmpxchg_double(pcp1, pcp2, oval1, oval2, nval1, nval2)
-#endif
-#ifndef raw_cpu_cmpxchg_double_4
-#define raw_cpu_cmpxchg_double_4(pcp1, pcp2, oval1, oval2, nval1, nval2) \
- raw_cpu_generic_cmpxchg_double(pcp1, pcp2, oval1, oval2, nval1, nval2)
-#endif
-#ifndef raw_cpu_cmpxchg_double_8
-#define raw_cpu_cmpxchg_double_8(pcp1, pcp2, oval1, oval2, nval1, nval2) \
- raw_cpu_generic_cmpxchg_double(pcp1, pcp2, oval1, oval2, nval1, nval2)
-#endif
-
#ifndef this_cpu_read_1
#define this_cpu_read_1(pcp) this_cpu_generic_read(pcp)
#endif
@@ -585,21 +544,4 @@ do { \
this_cpu_generic_cmpxchg(pcp, oval, nval)
#endif
-#ifndef this_cpu_cmpxchg_double_1
-#define this_cpu_cmpxchg_double_1(pcp1, pcp2, oval1, oval2, nval1, nval2) \
- this_cpu_generic_cmpxchg_double(pcp1, pcp2, oval1, oval2, nval1, nval2)
-#endif
-#ifndef this_cpu_cmpxchg_double_2
-#define this_cpu_cmpxchg_double_2(pcp1, pcp2, oval1, oval2, nval1, nval2) \
- this_cpu_generic_cmpxchg_double(pcp1, pcp2, oval1, oval2, nval1, nval2)
-#endif
-#ifndef this_cpu_cmpxchg_double_4
-#define this_cpu_cmpxchg_double_4(pcp1, pcp2, oval1, oval2, nval1, nval2) \
- this_cpu_generic_cmpxchg_double(pcp1, pcp2, oval1, oval2, nval1, nval2)
-#endif
-#ifndef this_cpu_cmpxchg_double_8
-#define this_cpu_cmpxchg_double_8(pcp1, pcp2, oval1, oval2, nval1, nval2) \
- this_cpu_generic_cmpxchg_double(pcp1, pcp2, oval1, oval2, nval1, nval2)
-#endif
-
#endif /* _ASM_GENERIC_PERCPU_H_ */
diff --git a/include/linux/atomic/atomic-instrumented.h b/include/linux/atomic/atomic-instrumented.h
index 8583720..a55b5b7 100644
--- a/include/linux/atomic/atomic-instrumented.h
+++ b/include/linux/atomic/atomic-instrumented.h
@@ -2234,21 +2234,6 @@ atomic_long_dec_if_positive(atomic_long_t *v)
arch_try_cmpxchg128_local(__ai_ptr, __ai_oldp, __VA_ARGS__); \
})
-#define cmpxchg_double(ptr, ...) \
-({ \
- typeof(ptr) __ai_ptr = (ptr); \
- kcsan_mb(); \
- instrument_atomic_read_write(__ai_ptr, 2 * sizeof(*__ai_ptr)); \
- arch_cmpxchg_double(__ai_ptr, __VA_ARGS__); \
-})
-
-
-#define cmpxchg_double_local(ptr, ...) \
-({ \
- typeof(ptr) __ai_ptr = (ptr); \
- instrument_atomic_read_write(__ai_ptr, 2 * sizeof(*__ai_ptr)); \
- arch_cmpxchg_double_local(__ai_ptr, __VA_ARGS__); \
-})
#endif /* _LINUX_ATOMIC_INSTRUMENTED_H */
-// 82d1be694fab30414527d0877c29fa75ed5a0b74
+// 3611991b015450e119bcd7417a9431af7f3ba13c
diff --git a/include/linux/percpu-defs.h b/include/linux/percpu-defs.h
index cbbf6d1..ec35731 100644
--- a/include/linux/percpu-defs.h
+++ b/include/linux/percpu-defs.h
@@ -358,33 +358,6 @@ static __always_inline void __this_cpu_preempt_check(const char *op) { }
pscr2_ret__; \
})
-/*
- * Special handling for cmpxchg_double. cmpxchg_double is passed two
- * percpu variables. The first has to be aligned to a double word
- * boundary and the second has to follow directly thereafter.
- * We enforce this on all architectures even if they don't support
- * a double cmpxchg instruction, since it's a cheap requirement, and it
- * avoids breaking the requirement for architectures with the instruction.
- */
-#define __pcpu_double_call_return_bool(stem, pcp1, pcp2, ...) \
-({ \
- bool pdcrb_ret__; \
- __verify_pcpu_ptr(&(pcp1)); \
- BUILD_BUG_ON(sizeof(pcp1) != sizeof(pcp2)); \
- VM_BUG_ON((unsigned long)(&(pcp1)) % (2 * sizeof(pcp1))); \
- VM_BUG_ON((unsigned long)(&(pcp2)) != \
- (unsigned long)(&(pcp1)) + sizeof(pcp1)); \
- switch(sizeof(pcp1)) { \
- case 1: pdcrb_ret__ = stem##1(pcp1, pcp2, __VA_ARGS__); break; \
- case 2: pdcrb_ret__ = stem##2(pcp1, pcp2, __VA_ARGS__); break; \
- case 4: pdcrb_ret__ = stem##4(pcp1, pcp2, __VA_ARGS__); break; \
- case 8: pdcrb_ret__ = stem##8(pcp1, pcp2, __VA_ARGS__); break; \
- default: \
- __bad_size_call_parameter(); break; \
- } \
- pdcrb_ret__; \
-})
-
#define __pcpu_size_call(stem, variable, ...) \
do { \
__verify_pcpu_ptr(&(variable)); \
@@ -443,9 +416,6 @@ do { \
__pcpu_size_call_return2(raw_cpu_cmpxchg_, pcp, oval, nval)
#define raw_cpu_try_cmpxchg(pcp, ovalp, nval) \
__pcpu_size_call_return2bool(raw_cpu_try_cmpxchg_, pcp, ovalp, nval)
-#define raw_cpu_cmpxchg_double(pcp1, pcp2, oval1, oval2, nval1, nval2) \
- __pcpu_double_call_return_bool(raw_cpu_cmpxchg_double_, pcp1, pcp2, oval1, oval2, nval1, nval2)
-
#define raw_cpu_sub(pcp, val) raw_cpu_add(pcp, -(val))
#define raw_cpu_inc(pcp) raw_cpu_add(pcp, 1)
#define raw_cpu_dec(pcp) raw_cpu_sub(pcp, 1)
@@ -505,11 +475,6 @@ do { \
raw_cpu_cmpxchg(pcp, oval, nval); \
})
-#define __this_cpu_cmpxchg_double(pcp1, pcp2, oval1, oval2, nval1, nval2) \
-({ __this_cpu_preempt_check("cmpxchg_double"); \
- raw_cpu_cmpxchg_double(pcp1, pcp2, oval1, oval2, nval1, nval2); \
-})
-
#define __this_cpu_sub(pcp, val) __this_cpu_add(pcp, -(typeof(pcp))(val))
#define __this_cpu_inc(pcp) __this_cpu_add(pcp, 1)
#define __this_cpu_dec(pcp) __this_cpu_sub(pcp, 1)
@@ -532,9 +497,6 @@ do { \
__pcpu_size_call_return2(this_cpu_cmpxchg_, pcp, oval, nval)
#define this_cpu_try_cmpxchg(pcp, ovalp, nval) \
__pcpu_size_call_return2bool(this_cpu_try_cmpxchg_, pcp, ovalp, nval)
-#define this_cpu_cmpxchg_double(pcp1, pcp2, oval1, oval2, nval1, nval2) \
- __pcpu_double_call_return_bool(this_cpu_cmpxchg_double_, pcp1, pcp2, oval1, oval2, nval1, nval2)
-
#define this_cpu_sub(pcp, val) this_cpu_add(pcp, -(typeof(pcp))(val))
#define this_cpu_inc(pcp) this_cpu_add(pcp, 1)
#define this_cpu_dec(pcp) this_cpu_sub(pcp, 1)
diff --git a/scripts/atomic/gen-atomic-instrumented.sh b/scripts/atomic/gen-atomic-instrumented.sh
index 4f80347..a2ef735 100755
--- a/scripts/atomic/gen-atomic-instrumented.sh
+++ b/scripts/atomic/gen-atomic-instrumented.sh
@@ -84,7 +84,6 @@ gen_xchg()
{
local xchg="$1"; shift
local order="$1"; shift
- local mult="$1"; shift
kcsan_barrier=""
if [ "${xchg%_local}" = "${xchg}" ]; then
@@ -104,8 +103,8 @@ cat <<EOF
EOF
[ -n "$kcsan_barrier" ] && printf "\t${kcsan_barrier}; \\\\\n"
cat <<EOF
- instrument_atomic_read_write(__ai_ptr, ${mult}sizeof(*__ai_ptr)); \\
- instrument_read_write(__ai_oldp, ${mult}sizeof(*__ai_oldp)); \\
+ instrument_atomic_read_write(__ai_ptr, sizeof(*__ai_ptr)); \\
+ instrument_read_write(__ai_oldp, sizeof(*__ai_oldp)); \\
arch_${xchg}${order}(__ai_ptr, __ai_oldp, __VA_ARGS__); \\
})
EOF
@@ -168,22 +167,16 @@ done
for xchg in "xchg" "cmpxchg" "cmpxchg64" "cmpxchg128" "try_cmpxchg" "try_cmpxchg64" "try_cmpxchg128"; do
for order in "" "_acquire" "_release" "_relaxed"; do
- gen_xchg "${xchg}" "${order}" ""
+ gen_xchg "${xchg}" "${order}"
printf "\n"
done
done
for xchg in "cmpxchg_local" "cmpxchg64_local" "cmpxchg128_local" "sync_cmpxchg" "try_cmpxchg_local" "try_cmpxchg64_local" "try_cmpxchg128_local"; do
- gen_xchg "${xchg}" "" ""
+ gen_xchg "${xchg}" ""
printf "\n"
done
-gen_xchg "cmpxchg_double" "" "2 * "
-
-printf "\n\n"
-
-gen_xchg "cmpxchg_double_local" "" "2 * "
-
cat <<EOF
#endif /* _LINUX_ATOMIC_INSTRUMENTED_H */
^ permalink raw reply [flat|nested] 47+ messages in thread
* [PATCH 12/12] s390/cpum_sf: Convert to cmpxchg128()
2023-05-31 13:08 [PATCH 00/12] Introduce cmpxchg128() -- aka. the demise of cmpxchg_double() Peter Zijlstra
` (10 preceding siblings ...)
2023-05-31 13:08 ` [PATCH 11/12] arch: Remove cmpxchg_double Peter Zijlstra
@ 2023-05-31 13:08 ` Peter Zijlstra
2023-06-05 7:42 ` [tip: locking/core] " tip-bot2 for Peter Zijlstra
2023-05-31 14:47 ` [PATCH 00/12] Introduce cmpxchg128() -- aka. the demise of cmpxchg_double() Mark Rutland
12 siblings, 1 reply; 47+ messages in thread
From: Peter Zijlstra @ 2023-05-31 13:08 UTC (permalink / raw)
To: torvalds
Cc: corbet, will, peterz, boqun.feng, mark.rutland, catalin.marinas,
dennis, tj, cl, hca, gor, agordeev, borntraeger, svens, tglx,
mingo, bp, dave.hansen, x86, hpa, joro, suravee.suthikulpanit,
robin.murphy, dwmw2, baolu.lu, Arnd Bergmann, Herbert Xu, davem,
penberg, rientjes, iamjoonsoo.kim, Andrew Morton, vbabka,
roman.gushchin, 42.hyeyoo, linux-doc, linux-kernel, linux-mm,
linux-s390, iommu, linux-arch, linux-crypto, sfr, mpe,
James.Bottomley, deller, linux-parisc
Now that there is a cross arch u128 and cmpxchg128(), use those
instead of the custom CDSG helper.
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Reviewed-by: Arnd Bergmann <arnd@arndb.de>
Acked-by: Heiko Carstens <hca@linux.ibm.com>
---
arch/s390/include/asm/cpu_mf.h | 2 +-
arch/s390/kernel/perf_cpum_sf.c | 16 +++-------------
2 files changed, 4 insertions(+), 14 deletions(-)
--- a/arch/s390/include/asm/cpu_mf.h
+++ b/arch/s390/include/asm/cpu_mf.h
@@ -140,7 +140,7 @@ union hws_trailer_header {
unsigned int dsdes:16; /* 48-63: size of diagnostic SDE */
unsigned long long overflow; /* 64 - Overflow Count */
};
- __uint128_t val;
+ u128 val;
};
struct hws_trailer_entry {
--- a/arch/s390/kernel/perf_cpum_sf.c
+++ b/arch/s390/kernel/perf_cpum_sf.c
@@ -1271,16 +1271,6 @@ static void hw_collect_samples(struct pe
}
}
-static inline __uint128_t __cdsg(__uint128_t *ptr, __uint128_t old, __uint128_t new)
-{
- asm volatile(
- " cdsg %[old],%[new],%[ptr]\n"
- : [old] "+d" (old), [ptr] "+QS" (*ptr)
- : [new] "d" (new)
- : "memory", "cc");
- return old;
-}
-
/* hw_perf_event_update() - Process sampling buffer
* @event: The perf event
* @flush_all: Flag to also flush partially filled sample-data-blocks
@@ -1352,7 +1342,7 @@ static void hw_perf_event_update(struct
new.f = 0;
new.a = 1;
new.overflow = 0;
- prev.val = __cdsg(&te->header.val, old.val, new.val);
+ prev.val = cmpxchg128(&te->header.val, old.val, new.val);
} while (prev.val != old.val);
/* Advance to next sample-data-block */
@@ -1562,7 +1552,7 @@ static bool aux_set_alert(struct aux_buf
}
new.a = 1;
new.overflow = 0;
- prev.val = __cdsg(&te->header.val, old.val, new.val);
+ prev.val = cmpxchg128(&te->header.val, old.val, new.val);
} while (prev.val != old.val);
return true;
}
@@ -1636,7 +1626,7 @@ static bool aux_reset_buffer(struct aux_
new.a = 1;
else
new.a = 0;
- prev.val = __cdsg(&te->header.val, old.val, new.val);
+ prev.val = cmpxchg128(&te->header.val, old.val, new.val);
} while (prev.val != old.val);
*overflow += orig_overflow;
}
^ permalink raw reply [flat|nested] 47+ messages in thread* [tip: locking/core] s390/cpum_sf: Convert to cmpxchg128()
2023-05-31 13:08 ` [PATCH 12/12] s390/cpum_sf: Convert to cmpxchg128() Peter Zijlstra
@ 2023-06-05 7:42 ` tip-bot2 for Peter Zijlstra
0 siblings, 0 replies; 47+ messages in thread
From: tip-bot2 for Peter Zijlstra @ 2023-06-05 7:42 UTC (permalink / raw)
To: linux-tip-commits
Cc: Peter Zijlstra (Intel),
Arnd Bergmann, Mark Rutland, Heiko Carstens, x86, linux-kernel
The following commit has been merged into the locking/core branch of tip:
Commit-ID: 497cc42bf53b55185ab3d39c634fbf09eb6681ae
Gitweb: https://git.kernel.org/tip/497cc42bf53b55185ab3d39c634fbf09eb6681ae
Author: Peter Zijlstra <peterz@infradead.org>
AuthorDate: Wed, 31 May 2023 15:08:45 +02:00
Committer: Peter Zijlstra <peterz@infradead.org>
CommitterDate: Mon, 05 Jun 2023 09:36:40 +02:00
s390/cpum_sf: Convert to cmpxchg128()
Now that there is a cross arch u128 and cmpxchg128(), use those
instead of the custom CDSG helper.
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Reviewed-by: Arnd Bergmann <arnd@arndb.de>
Reviewed-by: Mark Rutland <mark.rutland@arm.com>
Acked-by: Heiko Carstens <hca@linux.ibm.com>
Tested-by: Mark Rutland <mark.rutland@arm.com>
Link: https://lore.kernel.org/r/20230531132324.058821078@infradead.org
---
arch/s390/include/asm/cpu_mf.h | 2 +-
arch/s390/kernel/perf_cpum_sf.c | 16 +++-------------
2 files changed, 4 insertions(+), 14 deletions(-)
diff --git a/arch/s390/include/asm/cpu_mf.h b/arch/s390/include/asm/cpu_mf.h
index 7e417d7..a0de5b9 100644
--- a/arch/s390/include/asm/cpu_mf.h
+++ b/arch/s390/include/asm/cpu_mf.h
@@ -140,7 +140,7 @@ union hws_trailer_header {
unsigned int dsdes:16; /* 48-63: size of diagnostic SDE */
unsigned long long overflow; /* 64 - Overflow Count */
};
- __uint128_t val;
+ u128 val;
};
struct hws_trailer_entry {
diff --git a/arch/s390/kernel/perf_cpum_sf.c b/arch/s390/kernel/perf_cpum_sf.c
index 7ef72f5..8ecfbce 100644
--- a/arch/s390/kernel/perf_cpum_sf.c
+++ b/arch/s390/kernel/perf_cpum_sf.c
@@ -1271,16 +1271,6 @@ static void hw_collect_samples(struct perf_event *event, unsigned long *sdbt,
}
}
-static inline __uint128_t __cdsg(__uint128_t *ptr, __uint128_t old, __uint128_t new)
-{
- asm volatile(
- " cdsg %[old],%[new],%[ptr]\n"
- : [old] "+d" (old), [ptr] "+QS" (*ptr)
- : [new] "d" (new)
- : "memory", "cc");
- return old;
-}
-
/* hw_perf_event_update() - Process sampling buffer
* @event: The perf event
* @flush_all: Flag to also flush partially filled sample-data-blocks
@@ -1352,7 +1342,7 @@ static void hw_perf_event_update(struct perf_event *event, int flush_all)
new.f = 0;
new.a = 1;
new.overflow = 0;
- prev.val = __cdsg(&te->header.val, old.val, new.val);
+ prev.val = cmpxchg128(&te->header.val, old.val, new.val);
} while (prev.val != old.val);
/* Advance to next sample-data-block */
@@ -1562,7 +1552,7 @@ static bool aux_set_alert(struct aux_buffer *aux, unsigned long alert_index,
}
new.a = 1;
new.overflow = 0;
- prev.val = __cdsg(&te->header.val, old.val, new.val);
+ prev.val = cmpxchg128(&te->header.val, old.val, new.val);
} while (prev.val != old.val);
return true;
}
@@ -1636,7 +1626,7 @@ static bool aux_reset_buffer(struct aux_buffer *aux, unsigned long range,
new.a = 1;
else
new.a = 0;
- prev.val = __cdsg(&te->header.val, old.val, new.val);
+ prev.val = cmpxchg128(&te->header.val, old.val, new.val);
} while (prev.val != old.val);
*overflow += orig_overflow;
}
^ permalink raw reply [flat|nested] 47+ messages in thread
* Re: [PATCH 00/12] Introduce cmpxchg128() -- aka. the demise of cmpxchg_double()
2023-05-31 13:08 [PATCH 00/12] Introduce cmpxchg128() -- aka. the demise of cmpxchg_double() Peter Zijlstra
` (11 preceding siblings ...)
2023-05-31 13:08 ` [PATCH 12/12] s390/cpum_sf: Convert to cmpxchg128() Peter Zijlstra
@ 2023-05-31 14:47 ` Mark Rutland
12 siblings, 0 replies; 47+ messages in thread
From: Mark Rutland @ 2023-05-31 14:47 UTC (permalink / raw)
To: Peter Zijlstra
Cc: torvalds, corbet, will, boqun.feng, catalin.marinas, dennis, tj,
cl, hca, gor, agordeev, borntraeger, svens, tglx, mingo, bp,
dave.hansen, x86, hpa, joro, suravee.suthikulpanit, robin.murphy,
dwmw2, baolu.lu, Arnd Bergmann, Herbert Xu, davem, penberg,
rientjes, iamjoonsoo.kim, Andrew Morton, vbabka, roman.gushchin,
42.hyeyoo, linux-doc, linux-kernel, linux-mm, linux-s390, iommu,
linux-arch, linux-crypto, sfr, mpe, James.Bottomley, deller,
linux-parisc
On Wed, May 31, 2023 at 03:08:33PM +0200, Peter Zijlstra wrote:
> Hi!
>
> After much breaking of things, find here the improved version.
>
> Since v3:
>
> - unbreak everything that does *NOT* have cmpxchg128()
>
> Notably this_cpu_cmpxchg_double() is used unconditionally by SLUB
> which means that this_cpu_try_cmpxchg128() needs to be unconditionally
> available on all 64bit architectures.
>
> - fixed up x86/x86_64 cmpxchg{8,16}b emulation for this_cpu_cmpxchg{64,128}()
>
> - introduce {raw,this}_cpu_try_cmpxchg*()
>
> - add fallback for !__SIZEOF_INT128__ 64bit architectures
>
> Sadly there are supported 64bit architecture/compiler combinations that do
> not have __SIZEOF_INT128__, specifically it was found that HPPA64 only added
> this with GCC-11.
>
> this is yuck, and ideally we'd simply raise compiler requirements, but this
> 'works'.
The patches look good to me, and I used my local cross-build script to build
test this with the kernel.org GCC 10.3.0 cross toolchain for all of the
following arch/triplet/config combinations:
alpha alpha-linux defconfig
arc arc-linux defconfig
arm arm-linux-gnueabi multi_v4t_defconfig
arm arm-linux-gnueabi multi_v5_defconfig
arm arm-linux-gnueabi multi_v7_defconfig
arm arm-linux-gnueabi omap1_defconfig
arm64 aarch64-linux defconfig
csky csky-linux defconfig
i386 i386-linux defconfig
ia64 ia64-linux defconfig
m68k m68k-linux defconfig
microblaze microblaze-linux defconfig
mips mips-linux 32r1_defconfig
mips mips-linux 32r2_defconfig
mips mips-linux 32r6_defconfig
mips mips64-linux 64r1_defconfig
mips mips64-linux 64r2_defconfig
mips mips64-linux 64r6_defconfig
nios2 nios2-linux defconfig
openrisc or1k-linux defconfig
parisc hppa-linux generic-32bit_defconfig
parisc hppa64-linux generic-64bit_defconfig
powerpc powerpc-linux ppc40x_defconfig
powerpc powerpc64-linux ppc64_defconfig
powerpc powerpc64-linux ppc64e_defconfig
riscv riscv32-linux rv32_defconfig
riscv riscv64-linux defconfig
s390 s390-linux defconfig
sh sh4-linux defconfig
sparc sparc-linux sparc32_defconfig
sparc sparc64-linux sparc64_defconfig
x86_64 x86_64-linux defconfig
xtensa xtensa-linux defconfig
... and everything seemed happy.
I've also boot-tested arm64 defconfig.
So FWIW, for the series:
Reviewed-by: Mark Rutland <mark.rutland@arm.com>
Tested-by: Mark Rutland <mark.rutland@arm.com>
> My plan is to re-add this to tip/locking/core and thus -next later this week.
I'll need to rebase my kerneldoc series atop this, so getting this into a
stable branch soon would be great!
Thanks,
Mark.
>
> Also available at:
>
> git://git.kernel.org/pub/scm/linux/kernel/git/peterz/queue.git locking/core
>
> ---
> Documentation/core-api/this_cpu_ops.rst | 2 -
> arch/arm64/include/asm/atomic_ll_sc.h | 56 +++---
> arch/arm64/include/asm/atomic_lse.h | 39 ++---
> arch/arm64/include/asm/cmpxchg.h | 48 ++----
> arch/arm64/include/asm/percpu.h | 30 ++--
> arch/s390/include/asm/cmpxchg.h | 32 +---
> arch/s390/include/asm/cpu_mf.h | 2 +-
> arch/s390/include/asm/percpu.h | 34 ++--
> arch/s390/kernel/perf_cpum_sf.c | 16 +-
> arch/x86/include/asm/cmpxchg.h | 25 ---
> arch/x86/include/asm/cmpxchg_32.h | 2 +-
> arch/x86/include/asm/cmpxchg_64.h | 63 ++++++-
> arch/x86/include/asm/percpu.h | 102 ++++++-----
> arch/x86/lib/Makefile | 3 +-
> arch/x86/lib/cmpxchg16b_emu.S | 43 +++--
> arch/x86/lib/cmpxchg8b_emu.S | 67 ++++++--
> drivers/iommu/amd/amd_iommu_types.h | 9 +-
> drivers/iommu/amd/iommu.c | 10 +-
> drivers/iommu/intel/irq_remapping.c | 8 +-
> include/asm-generic/percpu.h | 257 ++++++++++++++++++++++------
> include/crypto/b128ops.h | 14 +-
> include/linux/atomic/atomic-arch-fallback.h | 95 +++++++++-
> include/linux/atomic/atomic-instrumented.h | 93 ++++++++--
> include/linux/dmar.h | 125 +++++++-------
> include/linux/percpu-defs.h | 45 ++---
> include/linux/slub_def.h | 12 +-
> include/linux/types.h | 12 ++
> include/uapi/linux/types.h | 4 +
> lib/crypto/curve25519-hacl64.c | 2 -
> lib/crypto/poly1305-donna64.c | 2 -
> mm/slab.h | 53 +++++-
> mm/slub.c | 139 +++++++++------
> scripts/atomic/gen-atomic-fallback.sh | 4 +-
> scripts/atomic/gen-atomic-instrumented.sh | 19 +-
> 34 files changed, 952 insertions(+), 515 deletions(-)
>
^ permalink raw reply [flat|nested] 47+ messages in thread