[PATCH] asm-generic/atomic: Add try_cmpxchg() fallbacks

[PATCH] asm-generic/atomic: Add try_cmpxchg() fallbacks

[Peter Zijlstra]

On Tue, Feb 18, 2020 at 02:13:10PM +0100, Peter Zijlstra wrote:
> (with the caveat that try_cmpxchg() doesn't seem available on !x86 -- I
> should go fix that)

Completely untested (lemme go do that shortly), but something like so I
suppose.

---
Subject: asm-generic/atomic: Add try_cmpxchg() fallbacks

Only x86 provides try_cmpxchg() outside of the atomic_t interfaces,
provide generic fallbacks to create this interface from the widely
available cmpxchg() function.

Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
---
diff --git a/include/linux/atomic-fallback.h b/include/linux/atomic-fallback.h
index 656b5489b673..243f61d6c35f 100644
--- a/include/linux/atomic-fallback.h
+++ b/include/linux/atomic-fallback.h
@@ -77,6 +77,50 @@
 
 #endif /* cmpxchg64_relaxed */
 
+#ifndef try_cmpxchg
+#define try_cmpxchg(_ptr, _oldp, _new) \
+({ \
+	typeof(*ptr) ___r, ___o = *(_oldp); \
+	___r = cmpxchg((_ptr), ___o, (_new)); \
+	if (unlikely(___r != ___o)) \
+		*(_old) = ___r; \
+	likely(___r == ___o); \
+})
+#endif /* try_cmpxchg */
+
+#ifndef try_cmpxchg_acquire
+#define try_cmpxchg_acquire(_ptr, _oldp, _new) \
+({ \
+	typeof(*ptr) ___r, ___o = *(_oldp); \
+	___r = cmpxchg_acquire((_ptr), ___o, (_new)); \
+	if (unlikely(___r != ___o)) \
+		*(_old) = ___r; \
+	likely(___r == ___o); \
+})
+#endif /* try_cmpxchg_acquire */
+
+#ifndef try_cmpxchg_release
+#define try_cmpxchg_release(_ptr, _oldp, _new) \
+({ \
+	typeof(*ptr) ___r, ___o = *(_oldp); \
+	___r = cmpxchg_release((_ptr), ___o, (_new)); \
+	if (unlikely(___r != ___o)) \
+		*(_old) = ___r; \
+	likely(___r == ___o); \
+})
+#endif /* try_cmpxchg_release */
+
+#ifndef try_cmpxchg_relaxed
+#define try_cmpxchg_relaxed(_ptr, _oldp, _new) \
+({ \
+	typeof(*ptr) ___r, ___o = *(_oldp); \
+	___r = cmpxchg_relaxed((_ptr), ___o, (_new)); \
+	if (unlikely(___r != ___o)) \
+		*(_old) = ___r; \
+	likely(___r == ___o); \
+})
+#endif /* try_cmpxchg_relaxed */
+
 #ifndef atomic_read_acquire
 static __always_inline int
 atomic_read_acquire(const atomic_t *v)
@@ -2294,4 +2338,4 @@ atomic64_dec_if_positive(atomic64_t *v)
 #define atomic64_cond_read_relaxed(v, c) smp_cond_load_relaxed(&(v)->counter, (c))
 
 #endif /* _LINUX_ATOMIC_FALLBACK_H */
-// baaf45f4c24ed88ceae58baca39d7fd80bb8101b
+// 2dfbc767ce308d2edd67a49bd7b764dd07f62f6c
diff --git a/scripts/atomic/gen-atomic-fallback.sh b/scripts/atomic/gen-atomic-fallback.sh
index b6c6f5d306a7..3c9be8d550e0 100755
--- a/scripts/atomic/gen-atomic-fallback.sh
+++ b/scripts/atomic/gen-atomic-fallback.sh
@@ -140,6 +140,32 @@ cat <<EOF
 EOF
 }
 
+gen_try_cmpxchg_fallback()
+{
+	local order="$1"; shift;
+
+cat <<EOF
+#ifndef try_cmpxchg${order}
+#define try_cmpxchg${order}(_ptr, _oldp, _new) \\
+({ \\
+	typeof(*ptr) ___r, ___o = *(_oldp); \\
+	___r = cmpxchg${order}((_ptr), ___o, (_new)); \\
+	if (unlikely(___r != ___o)) \\
+		*(_old) = ___r; \\
+	likely(___r == ___o); \\
+})
+#endif /* try_cmpxchg${order} */
+
+EOF
+}
+
+gen_try_cmpxchg_fallbacks()
+{
+	for order in "" "_acquire" "_release" "_relaxed"; do
+		gen_try_cmpxchg_fallback "${order}"
+	done
+}
+
 cat << EOF
 // SPDX-License-Identifier: GPL-2.0
 
@@ -157,6 +183,8 @@ for xchg in "xchg" "cmpxchg" "cmpxchg64"; do
 	gen_xchg_fallbacks "${xchg}"
 done
 
+gen_try_cmpxchg_fallbacks
+
 grep '^[a-z]' "$1" | while read name meta args; do
 	gen_proto "${meta}" "${name}" "atomic" "int" ${args}
 done

Re: [PATCH] asm-generic/atomic: Add try_cmpxchg() fallbacks

[Peter Zijlstra]

On Tue, Feb 18, 2020 at 03:27:00PM +0100, Peter Zijlstra wrote:
> diff --git a/scripts/atomic/gen-atomic-fallback.sh b/scripts/atomic/gen-atomic-fallback.sh
> index b6c6f5d306a7..3c9be8d550e0 100755
> --- a/scripts/atomic/gen-atomic-fallback.sh
> +++ b/scripts/atomic/gen-atomic-fallback.sh
> @@ -140,6 +140,32 @@ cat <<EOF
>  EOF
>  }
>  
> +gen_try_cmpxchg_fallback()
> +{
> +	local order="$1"; shift;
> +
> +cat <<EOF
> +#ifndef try_cmpxchg${order}
> +#define try_cmpxchg${order}(_ptr, _oldp, _new) \\
> +({ \\
> +	typeof(*ptr) ___r, ___o = *(_oldp); \\
               *(_ptr)
> +	___r = cmpxchg${order}((_ptr), ___o, (_new)); \\
> +	if (unlikely(___r != ___o)) \\
> +		*(_old) = ___r; \\
		*(_oldp)
> +	likely(___r == ___o); \\
> +})
> +#endif /* try_cmpxchg${order} */

And it actually builds, as tested on an ARM build.

Re: [PATCH] asm-generic/atomic: Add try_cmpxchg() fallbacks

[Will Deacon]

On Tue, Feb 18, 2020 at 03:27:00PM +0100, Peter Zijlstra wrote:
> On Tue, Feb 18, 2020 at 02:13:10PM +0100, Peter Zijlstra wrote:
> > (with the caveat that try_cmpxchg() doesn't seem available on !x86 -- I
> > should go fix that)
> 
> Completely untested (lemme go do that shortly), but something like so I
> suppose.
> 
> ---
> Subject: asm-generic/atomic: Add try_cmpxchg() fallbacks
> 
> Only x86 provides try_cmpxchg() outside of the atomic_t interfaces,
> provide generic fallbacks to create this interface from the widely
> available cmpxchg() function.
> 
> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
> ---
> diff --git a/include/linux/atomic-fallback.h b/include/linux/atomic-fallback.h
> index 656b5489b673..243f61d6c35f 100644
> --- a/include/linux/atomic-fallback.h
> +++ b/include/linux/atomic-fallback.h
> @@ -77,6 +77,50 @@
>  
>  #endif /* cmpxchg64_relaxed */
>  
> +#ifndef try_cmpxchg
> +#define try_cmpxchg(_ptr, _oldp, _new) \
> +({ \
> +	typeof(*ptr) ___r, ___o = *(_oldp); \

Probably worth pointing out that this will have the nasty behaviour
for volatile pointers that we're tackling for READ_ONCE. Obviously no
need to hold this up, but just mentioning it here in the hope that one
of us remembers to fix it later on.

> diff --git a/scripts/atomic/gen-atomic-fallback.sh b/scripts/atomic/gen-atomic-fallback.sh
> index b6c6f5d306a7..3c9be8d550e0 100755
> --- a/scripts/atomic/gen-atomic-fallback.sh
> +++ b/scripts/atomic/gen-atomic-fallback.sh
> @@ -140,6 +140,32 @@ cat <<EOF
>  EOF
>  }
>  
> +gen_try_cmpxchg_fallback()
> +{
> +	local order="$1"; shift;
> +
> +cat <<EOF
> +#ifndef try_cmpxchg${order}
> +#define try_cmpxchg${order}(_ptr, _oldp, _new) \\
> +({ \\
> +	typeof(*ptr) ___r, ___o = *(_oldp); \\
> +	___r = cmpxchg${order}((_ptr), ___o, (_new)); \\
> +	if (unlikely(___r != ___o)) \\
> +		*(_old) = ___r; \\

This doesn't compile because _old isn't declared. Probably best to avoid
evaluating _oldp twice though.

Will

Re: [PATCH] asm-generic/atomic: Add try_cmpxchg() fallbacks

[Peter Zijlstra]

On Thu, Feb 20, 2020 at 10:30:45AM +0000, Will Deacon wrote:
> On Tue, Feb 18, 2020 at 03:27:00PM +0100, Peter Zijlstra wrote:
> > On Tue, Feb 18, 2020 at 02:13:10PM +0100, Peter Zijlstra wrote:
> > > (with the caveat that try_cmpxchg() doesn't seem available on !x86 -- I
> > > should go fix that)
> > 
> > Completely untested (lemme go do that shortly), but something like so I
> > suppose.
> > 
> > ---
> > Subject: asm-generic/atomic: Add try_cmpxchg() fallbacks
> > 
> > Only x86 provides try_cmpxchg() outside of the atomic_t interfaces,
> > provide generic fallbacks to create this interface from the widely
> > available cmpxchg() function.
> > 
> > Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
> > ---
> > diff --git a/include/linux/atomic-fallback.h b/include/linux/atomic-fallback.h
> > index 656b5489b673..243f61d6c35f 100644
> > --- a/include/linux/atomic-fallback.h
> > +++ b/include/linux/atomic-fallback.h
> > @@ -77,6 +77,50 @@
> >  
> >  #endif /* cmpxchg64_relaxed */
> >  
> > +#ifndef try_cmpxchg
> > +#define try_cmpxchg(_ptr, _oldp, _new) \
> > +({ \
> > +	typeof(*ptr) ___r, ___o = *(_oldp); \
> 
> Probably worth pointing out that this will have the nasty behaviour
> for volatile pointers that we're tackling for READ_ONCE. Obviously no
> need to hold this up, but just mentioning it here in the hope that one
> of us remembers to fix it later on.

Right :/

> > diff --git a/scripts/atomic/gen-atomic-fallback.sh b/scripts/atomic/gen-atomic-fallback.sh
> > index b6c6f5d306a7..3c9be8d550e0 100755
> > --- a/scripts/atomic/gen-atomic-fallback.sh
> > +++ b/scripts/atomic/gen-atomic-fallback.sh
> > @@ -140,6 +140,32 @@ cat <<EOF
> >  EOF
> >  }
> >  
> > +gen_try_cmpxchg_fallback()
> > +{
> > +	local order="$1"; shift;
> > +
> > +cat <<EOF
> > +#ifndef try_cmpxchg${order}
> > +#define try_cmpxchg${order}(_ptr, _oldp, _new) \\
> > +({ \\
> > +	typeof(*ptr) ___r, ___o = *(_oldp); \\
> > +	___r = cmpxchg${order}((_ptr), ___o, (_new)); \\
> > +	if (unlikely(___r != ___o)) \\
> > +		*(_old) = ___r; \\
> 
> This doesn't compile because _old isn't declared. Probably best to avoid
> evaluating _oldp twice though.

Compiler had already spotted that, I'll make it something like:

	typeof(*ptr) *___op = (_oldp), ___o = *___op;

	...

		*___op = ___r;

Which avoids the double eval.

Re: [PATCH] asm-generic/atomic: Add try_cmpxchg() fallbacks

[Will Deacon]

On Thu, Feb 20, 2020 at 11:37:27AM +0100, Peter Zijlstra wrote:
> On Thu, Feb 20, 2020 at 10:30:45AM +0000, Will Deacon wrote:
> > On Tue, Feb 18, 2020 at 03:27:00PM +0100, Peter Zijlstra wrote:
> > > diff --git a/scripts/atomic/gen-atomic-fallback.sh b/scripts/atomic/gen-atomic-fallback.sh
> > > index b6c6f5d306a7..3c9be8d550e0 100755
> > > --- a/scripts/atomic/gen-atomic-fallback.sh
> > > +++ b/scripts/atomic/gen-atomic-fallback.sh
> > > @@ -140,6 +140,32 @@ cat <<EOF
> > >  EOF
> > >  }
> > >  
> > > +gen_try_cmpxchg_fallback()
> > > +{
> > > +	local order="$1"; shift;
> > > +
> > > +cat <<EOF
> > > +#ifndef try_cmpxchg${order}
> > > +#define try_cmpxchg${order}(_ptr, _oldp, _new) \\
> > > +({ \\
> > > +	typeof(*ptr) ___r, ___o = *(_oldp); \\
> > > +	___r = cmpxchg${order}((_ptr), ___o, (_new)); \\
> > > +	if (unlikely(___r != ___o)) \\
> > > +		*(_old) = ___r; \\
> > 
> > This doesn't compile because _old isn't declared. Probably best to avoid
> > evaluating _oldp twice though.
> 
> Compiler had already spotted that, I'll make it something like:
> 
> 	typeof(*ptr) *___op = (_oldp), ___o = *___op;
> 
> 	...
> 
> 		*___op = ___r;
> 
> Which avoids the double eval.

Cool, you can stick my Ack on the patch with that change.

Will