hardening and other opts in kernel config used for benchmarking?

hardening and other opts in kernel config used for benchmarking?

[Mateusz Guzik]

Hello,

I have no idea who should be in To: or Cc:, I grabbed names from an
e-mail I got regarding one of previous changes.

Recently I benchmarked a change which added unconditional file
position locking and found a minor regression from it (with profiler
output to justify it):
https://lore.kernel.org/linux-fsdevel/CAHk-=whJtLkYwEFTS9LcRiMjSqq_xswDeXo7hYNWT0Em6nL4Sw@mail.gmail.com/T/#m7c0cd6e913c6295732daea3c88f502bd4724ffb3

However, according to Christian the change was benchmarked by your
machinery and no difference was found.

I briefly poked around and found that used configs have:
CONFIG_RANDOMIZE_KSTACK_OFFSET=y

This is an optional and very expensive hardening feature, my question
is if it was enabled on purpose. The cost comes from adding rdtsc to
every syscall.

Looking at the rest of the config you have a mixed bag (e.g., hardened
usercopy but *no* init_on_alloc) so I genuinely don't know.

Given the high cost of the opt I would suggest removing it, as it
avoidably muddles the waters for single-threaded changes (one way or
the other -- slowdowns can hide and speed ups go unnoticed).

I did not review the whole config.

Any comments? :)

Thanks,
-- 
Mateusz Guzik <mjguzik gmail.com>

Re: hardening and other opts in kernel config used for benchmarking?

[Yin Fengwei]

Add more LKP people.

On 8/8/23 23:25, Mateusz Guzik wrote:
> Hello,
> 
> I have no idea who should be in To: or Cc:, I grabbed names from an
> e-mail I got regarding one of previous changes.
> 
> Recently I benchmarked a change which added unconditional file
> position locking and found a minor regression from it (with profiler
> output to justify it):
> https://lore.kernel.org/linux-fsdevel/CAHk-=whJtLkYwEFTS9LcRiMjSqq_xswDeXo7hYNWT0Em6nL4Sw@mail.gmail.com/T/#m7c0cd6e913c6295732daea3c88f502bd4724ffb3
> 
> However, according to Christian the change was benchmarked by your
> machinery and no difference was found.
> 
> I briefly poked around and found that used configs have:
> CONFIG_RANDOMIZE_KSTACK_OFFSET=y
> 
> This is an optional and very expensive hardening feature, my question
> is if it was enabled on purpose. The cost comes from adding rdtsc to
> every syscall.
> 
> Looking at the rest of the config you have a mixed bag (e.g., hardened
> usercopy but *no* init_on_alloc) so I genuinely don't know.
We used a kernel config from Redhat Enterprise Linux distribution as
base config (it was from redhat kernel-4.18.0-187). And then use the
default value of new configs.


> 
> Given the high cost of the opt I would suggest removing it, as it
From arch/Kconfig:

config RANDOMIZE_KSTACK_OFFSET                                                  
        bool "Support for randomizing kernel stack offset on syscall entry" if EXPERT
        default y                                                               
        depends on HAVE_ARCH_RANDOMIZE_KSTACK_OFFSET                            
        depends on INIT_STACK_NONE || !CC_IS_CLANG || CLANG_VERSION >= 140000   
        help                                                                    
          The kernel stack offset can be randomized (after pt_regs) by          
          roughly 5 bits of entropy, frustrating memory corruption              
          attacks that depend on stack address determinism or                   
          cross-syscall address exposures.                                      
                                                                                
          The feature is controlled via the "randomize_kstack_offset=on/off"    
          kernel boot param, and if turned off has zero overhead due to its use 
          of static branches (see JUMP_LABEL).                                  
                                                                                
          If unsure, say Y.                                                  

I don't think we want to disable this kernel config in LKP.


> avoidably muddles the waters for single-threaded changes (one way or
> the other -- slowdowns can hide and speed ups go unnoticed).
> 
> I did not review the whole config.
> 
> Any comments? :)
IMHO, LKP cares more about the possible performance downgrade users could
observe and assume users are using distribution. So LKP sticks to a distribution
config and using the default value for new configs.


I did test the will-it-scale.read1 against the commit
   20ea1e7d13c1b544fe67c4a8dc3943bb1ab33e6f
and its parent commit:
   9e0ee0c7545c7ec012a53878e7687e05b87abc75
by using "read1_process -s 4 -t 1" on a IceLake platform with 48C/96T + 192G
memory. I don't have available Sapphire Rapid to run same test.

The config is with the CONFIG_RANDOMIZE_KSTACK_OFFSET and CONFIG_HARDENED_USERCOPY
disabled/enabled.


I found the result is not stable. stddev is around 7% ~ 8%. larger or close
the the performance difference. For average: there is no regression with
these two kernel configs disabled. Around 9% regression with these two kernel
configs enabled.

But from LKP perspective (Oliver, Yujie and Philip, correct me if I am wrong
here), this test result is not reported as regression because the stddev is
very close to the regression.


The specific result is as following:
parent without random/hardened_usercopy:      child without random/hardened_usercopy:
	4050865                                	3570609
	3732989                                	3971604
	4043509                                	3577899
	3644475                                	4106721
	4156399                                	3432816
	3633719                                	4207917
	3508474                                	3581210
	3363729                                	3548164
	3794146                                	3490865
	3449113                                	4088112
	
avg:	3737741                                avg:	3757591
stddev:	7%                                     stddev:	7%


parent with random/hardened_usercopy:         child with random/hardened_usercopy:
	3770330                               	4279949
	3824055                               	3377960
	3497524                               	3359246
	4244985                               	3462981
	3442711                               	3520535
	4418889                               	3436251
	4255512                               	3884103
	3605148                               	3480065
	3965194                               	3230780
	3654892                               	3372904

avg:	3867924                               avg:	3540477     9% regression. But with 8.7% stddev
stddev:	8.8%                                  stddev:	8.7%


Regards
Yin, Fengwei

> 
> Thanks,