[PATCH -tip] x86/mm: Use %RIP-relative address in untagged_addr()

[PATCH -tip] x86/mm: Use %RIP-relative address in untagged_addr()

[Uros Bizjak]

%RIP-relative addresses are nowadays correctly handled in alternative
instructions, so remove misleading comment and improve assembly to
use %RIP-relative address.

Also, explicitly using %gs: prefix will segfault for non-SMP builds.
Use macros from percpu.h which will DTRT with segment prefix register
as far as SMP/non-SMP builds are concerned.

Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Ingo Molnar <mingo@kernel.org>
Cc: Borislav Petkov <bp@alien8.de>
Cc: Dave Hansen <dave.hansen@linux.intel.com>
Cc: "H. Peter Anvin" <hpa@zytor.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Signed-off-by: Uros Bizjak <ubizjak@gmail.com>
---
 arch/x86/include/asm/uaccess_64.h | 11 ++++-------
 1 file changed, 4 insertions(+), 7 deletions(-)

diff --git a/arch/x86/include/asm/uaccess_64.h b/arch/x86/include/asm/uaccess_64.h
index f2c02e4469cc..01455c0b070c 100644
--- a/arch/x86/include/asm/uaccess_64.h
+++ b/arch/x86/include/asm/uaccess_64.h
@@ -11,6 +11,7 @@
 #include <asm/alternative.h>
 #include <asm/cpufeatures.h>
 #include <asm/page.h>
+#include <asm/percpu.h>
 
 #ifdef CONFIG_ADDRESS_MASKING
 /*
@@ -18,14 +19,10 @@
  */
 static inline unsigned long __untagged_addr(unsigned long addr)
 {
-	/*
-	 * Refer tlbstate_untag_mask directly to avoid RIP-relative relocation
-	 * in alternative instructions. The relocation gets wrong when gets
-	 * copied to the target place.
-	 */
 	asm (ALTERNATIVE("",
-			 "and %%gs:tlbstate_untag_mask, %[addr]\n\t", X86_FEATURE_LAM)
-	     : [addr] "+r" (addr) : "m" (tlbstate_untag_mask));
+			 "and " __percpu_arg([mask]) ", %[addr]", X86_FEATURE_LAM)
+	     : [addr] "+r" (addr)
+	     : [mask] "m" (__my_cpu_var(tlbstate_untag_mask)));
 
 	return addr;
 }
-- 
2.41.0

Re: [PATCH -tip] x86/mm: Use %RIP-relative address in untagged_addr()

[Peter Zijlstra]

On Thu, Nov 16, 2023 at 08:10:59PM +0100, Uros Bizjak wrote:
> %RIP-relative addresses are nowadays correctly handled in alternative
> instructions, so remove misleading comment and improve assembly to
> use %RIP-relative address.

Ha!, it might've been this exact case (and Kirill grumbling) that got me
to fix the alternative code :-)

> Also, explicitly using %gs: prefix will segfault for non-SMP builds.
> Use macros from percpu.h which will DTRT with segment prefix register
> as far as SMP/non-SMP builds are concerned.

> Signed-off-by: Uros Bizjak <ubizjak@gmail.com>

Acked-byL Peter Zijlstra (Intel) <peterz@infradaed.org>

> ---
>  arch/x86/include/asm/uaccess_64.h | 11 ++++-------
>  1 file changed, 4 insertions(+), 7 deletions(-)
> 
> diff --git a/arch/x86/include/asm/uaccess_64.h b/arch/x86/include/asm/uaccess_64.h
> index f2c02e4469cc..01455c0b070c 100644
> --- a/arch/x86/include/asm/uaccess_64.h
> +++ b/arch/x86/include/asm/uaccess_64.h
> @@ -11,6 +11,7 @@
>  #include <asm/alternative.h>
>  #include <asm/cpufeatures.h>
>  #include <asm/page.h>
> +#include <asm/percpu.h>
>  
>  #ifdef CONFIG_ADDRESS_MASKING
>  /*
> @@ -18,14 +19,10 @@
>   */
>  static inline unsigned long __untagged_addr(unsigned long addr)
>  {
> -	/*
> -	 * Refer tlbstate_untag_mask directly to avoid RIP-relative relocation
> -	 * in alternative instructions. The relocation gets wrong when gets
> -	 * copied to the target place.
> -	 */
>  	asm (ALTERNATIVE("",
> -			 "and %%gs:tlbstate_untag_mask, %[addr]\n\t", X86_FEATURE_LAM)
> -	     : [addr] "+r" (addr) : "m" (tlbstate_untag_mask));
> +			 "and " __percpu_arg([mask]) ", %[addr]", X86_FEATURE_LAM)
> +	     : [addr] "+r" (addr)
> +	     : [mask] "m" (__my_cpu_var(tlbstate_untag_mask)));
>  
>  	return addr;
>  }
> -- 
> 2.41.0
> 

Re: [PATCH -tip] x86/mm: Use %RIP-relative address in untagged_addr()

[kirill.shutemov]

On Fri, Nov 17, 2023 at 10:41:03AM +0100, Peter Zijlstra wrote:
> On Thu, Nov 16, 2023 at 08:10:59PM +0100, Uros Bizjak wrote:
> > %RIP-relative addresses are nowadays correctly handled in alternative
> > instructions, so remove misleading comment and improve assembly to
> > use %RIP-relative address.
> 
> Ha!, it might've been this exact case (and Kirill grumbling) that got me
> to fix the alternative code :-)

Nice! :)

> > Also, explicitly using %gs: prefix will segfault for non-SMP builds.
> > Use macros from percpu.h which will DTRT with segment prefix register
> > as far as SMP/non-SMP builds are concerned.
> 
> > Signed-off-by: Uros Bizjak <ubizjak@gmail.com>
> 
> Acked-byL Peter Zijlstra (Intel) <peterz@infradaed.org>

Acked-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com>

-- 
  Kiryl Shutsemau / Kirill A. Shutemov

Re: [PATCH -tip] x86/mm: Use %RIP-relative address in untagged_addr()

[H. Peter Anvin]

On 11/16/23 11:10, Uros Bizjak wrote:
> %RIP-relative addresses are nowadays correctly handled in alternative
> instructions, so remove misleading comment and improve assembly to
> use %RIP-relative address.
> 
> Also, explicitly using %gs: prefix will segfault for non-SMP builds.
> Use macros from percpu.h which will DTRT with segment prefix register
> as far as SMP/non-SMP builds are concerned.

OK, this is starting to feel silly. One could seriously question the use 
case for supporting !SMP builds x86-64. It isn't like our performance 
for SMP builds on UP systems is significantly worse, it is mostly just a 
matter of code size, and the difference isn't huge, either, especially 
considering that on systems of the x86-64 era the kernel is a rather 
small part of system memory (unlike the very early i386 era, for those 
of us who remember those ancient times.)

The number of UP x86-64 systems is really very small (since 
multicore/SMT became ubiquitous at roughly the same time x86-64 was 
introduced), and as far as I know none of them lack APIC which is really 
the most fundamental difference between SMP and !SMP on x86.

Why don't we simply have %gs_base == 0 as an invariant for !SMP? If we 
*REALLY* care to skip SWAPGS on !SMP systems, we could use alternatives 
to patch out %gs: and lock (wouldn't even have to be explicit: this is 
the kind of thing that objtool does really well.) We can use 
alternatives without anything special, since it only matters after we 
have entered user spae for the first time and would be concurrent with 
patching out SWAPGS itself.

If we really *do* care about UP builds, we could teach objtool to do 
this patching at compile time for the !SMP builds.

Also, didn't we at least use to have a way to mark a function as "init 
on UP" so that it could be jettisoned with the init code if we find 
ourselves on a uniprocessor system?

	-hpa

Re: [PATCH -tip] x86/mm: Use %RIP-relative address in untagged_addr()

[Brian Gerst]

On Fri, Nov 17, 2023 at 1:16 PM H. Peter Anvin <hpa@zytor.com> wrote:
>
> On 11/16/23 11:10, Uros Bizjak wrote:
> > %RIP-relative addresses are nowadays correctly handled in alternative
> > instructions, so remove misleading comment and improve assembly to
> > use %RIP-relative address.
> >
> > Also, explicitly using %gs: prefix will segfault for non-SMP builds.
> > Use macros from percpu.h which will DTRT with segment prefix register
> > as far as SMP/non-SMP builds are concerned.
>
> OK, this is starting to feel silly. One could seriously question the use
> case for supporting !SMP builds x86-64. It isn't like our performance
> for SMP builds on UP systems is significantly worse, it is mostly just a
> matter of code size, and the difference isn't huge, either, especially
> considering that on systems of the x86-64 era the kernel is a rather
> small part of system memory (unlike the very early i386 era, for those
> of us who remember those ancient times.)
>
> The number of UP x86-64 systems is really very small (since
> multicore/SMT became ubiquitous at roughly the same time x86-64 was
> introduced), and as far as I know none of them lack APIC which is really
> the most fundamental difference between SMP and !SMP on x86.
>
> Why don't we simply have %gs_base == 0 as an invariant for !SMP?

The reason is stack protector, which is still stuck at %gs:40.  So
GSBASE has to point at fixed_percpu_data, even on a UP build.  That is
corrected by the patch series I recently posted, though.

> If we
> *REALLY* care to skip SWAPGS on !SMP systems, we could use alternatives
> to patch out %gs: and lock (wouldn't even have to be explicit: this is
> the kind of thing that objtool does really well.) We can use
> alternatives without anything special, since it only matters after we
> have entered user spae for the first time and would be concurrent with
> patching out SWAPGS itself.

There is already support to patch out LOCK prefixes when running an
SMP build on a single CPU (.smp_locks section).  Patching out the GS
prefix would only work if the initial percpu area is not freed.
Beyond that I don't think other optimizations are worth the effort,
and would get very little testing.


Brian Gerst

Re: [PATCH -tip] x86/mm: Use %RIP-relative address in untagged_addr()

[H. Peter Anvin]

On 11/17/23 11:43, Brian Gerst wrote:>>
>> Why don't we simply have %gs_base == 0 as an invariant for !SMP?
> 
> The reason is stack protector, which is still stuck at %gs:40.  So
> GSBASE has to point at fixed_percpu_data, even on a UP build.  That is
> corrected by the patch series I recently posted, though.
> 

Right, that problem is gone.

>> If we
>> *REALLY* care to skip SWAPGS on !SMP systems, we could use alternativesYep, that is 
>> to patch out %gs: and lock (wouldn't even have to be explicit: this is
>> the kind of thing that objtool does really well.) We can use
>> alternatives without anything special, since it only matters after we
>> have entered user spae for the first time and would be concurrent with
>> patching out SWAPGS itself.
> 
> There is already support to patch out LOCK prefixes when running an
> SMP build on a single CPU (.smp_locks section).  Patching out the GS
> prefix would only work if the initial percpu area is not freed.
> Beyond that I don't think other optimizations are worth the effort,
> and would get very little testing.

Yes, that is basically my point.

	-hpa