[linus:master] [x86/bugs] a3af84b0fa: stress-ng.seccomp.ops_per_sec 95.7% regression

[linus:master] [x86/bugs] a3af84b0fa: stress-ng.seccomp.ops_per_sec 95.7% regression

[kernel test robot]

Hello,

kernel test robot noticed a 95.7% regression of stress-ng.seccomp.ops_per_sec on:

commit: a3af84b0fa00ead01fcd0e28b5d773ff25990a0d ("x86/bugs: Enable IBPB flush on BPF JIT allocation")
https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git master

testcase: stress-ng
config: x86_64-rhel-9.4
compiler: gcc-14
test machine: 256 threads 2 sockets Intel(R) Xeon(R) 6768P  CPU @ 2.4GHz (Granite Rapids) with 64G memory
parameters:

	nr_threads: 100%
	testtime: 60s
	test: seccomp
	cpufreq_governor: performance

If you fix the issue in a separate patch/commit (i.e. not just a new version of
the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <yi1.lai@intel.com>
| Closes: https://lore.kernel.org/oe-lkp/202608191340.98574fe2-lkp@intel.com


Details are as below:
-------------------------------------------------------------------------------------------------->


The kernel config and materials to reproduce are available at:
https://download.01.org/0day-ci/archive/20260819/202608191340.98574fe2-lkp@intel.com

=========================================================================================
compiler/cpufreq_governor/kconfig/nr_threads/rootfs/tbox_group/test/testcase/testtime:
  gcc-14/performance/x86_64-rhel-9.4/100%/debian-13-x86_64-20250902.cgz/lkp-gnr-2sp4/seccomp/stress-ng/60s

commit: 
  96cce16e26 ("bpf: Support for hardening against JIT spraying")
  a3af84b0fa ("x86/bugs: Enable IBPB flush on BPF JIT allocation")
  b72e29e0f7 (bpf: Prefer dirty packs for eBPF allocations)

96cce16e26dd02a8 a3af84b0fa00ead01fcd0e28b5d b72e29e0f7ee329d89f86db8700
---------------- --------------------------- ---------------------------
         %stddev     %change         %stddev     %change         %stddev
             \          |                \          |                \
    351130           -95.7%      15233           -92.8%      25169        stress-ng.seccomp.ops
      5866           -95.7%     252.80           -92.9%     418.07        stress-ng.seccomp.ops_per_sec
    403577           -92.1%      31710           -88.5%      46510        stress-ng.time.involuntary_context_switches
  34288719 ±  2%     -95.4%    1594121           -92.5%    2566499        stress-ng.time.minor_page_faults
      4244           -67.9%       1363 ±  2%     -71.9%       1193        stress-ng.time.percent_of_cpu_this_job_got
      2515           -67.1%     826.95 ±  2%     -71.3%     721.20        stress-ng.time.system_time
     41.17 ±  2%     -94.9%       2.10 ±  3%     -91.7%       3.40 ±  2%  stress-ng.time.user_time
   1119583 ±  2%     -96.3%      40891           -94.5%      61287        stress-ng.time.voluntary_context_switches

Commit b72e29e0f7 slightly recovers performance but remains far below baseline.

Disclaimer:
Results have been estimated based on internal Intel analysis and are provided
for informational purposes only. Any difference in system hardware or software
design or configuration may affect actual performance.


-- 
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki

Re: [linus:master] [x86/bugs] a3af84b0fa: stress-ng.seccomp.ops_per_sec 95.7% regression

[Dave Hansen]

On 8/18/26 23:20, kernel test robot wrote:
> kernel test robot noticed a 95.7% regression of stress-ng.seccomp.ops_per_sec on:

One one level this is a "vulnerability mitigations cause performance
loss, news at 11" kind of thing. Like, Duh!

The microbenchmark here does seem to be a pretty worse-case scenario
where it sits in a loop loading and running a seccomp filter program. In
other words, it's just about as pathological of a test as you can have.

I think Pawan had some more ideas about ways to optimize this. Is there
any appetite for that? Or is the test just too synthetic to worry about?

Re: [linus:master] [x86/bugs] a3af84b0fa: stress-ng.seccomp.ops_per_sec 95.7% regression

[Pawan Gupta]

On Wed, Aug 19, 2026 at 08:26:53AM -0700, Dave Hansen wrote:
> On 8/18/26 23:20, kernel test robot wrote:
> > kernel test robot noticed a 95.7% regression of stress-ng.seccomp.ops_per_sec on:
> 
> One one level this is a "vulnerability mitigations cause performance
> loss, news at 11" kind of thing. Like, Duh!
> 
> The microbenchmark here does seem to be a pretty worse-case scenario
> where it sits in a loop loading and running a seccomp filter program. In
> other words, it's just about as pathological of a test as you can have.
> 
> I think Pawan had some more ideas about ways to optimize this. Is there
> any appetite for that? Or is the test just too synthetic to worry about?

One of the optimization is to do fine-grained tracking, i.e. track reuse
per-chunk instead of per-pack. This requires an additional bitmap per-pack.

Another option is to track the CPUs where the cBPF program ran, and only
send IPIs to those CPUs. This mainly reduces the overhead on workloads on
other CPUs, not necessarily the workload allocating programs. Seems more
relevant on servers with many CPUs.

Not sure if these are worth the effort yet, given that I am not aware of a
real-world workload with a significant impact.

Re: [linus:master] [x86/bugs] a3af84b0fa: stress-ng.seccomp.ops_per_sec 95.7% regression

[Chengfeng Lin]

[-- Attachment #1: Type: text/plain, Size: 3485 bytes --]

Hi Pawan, Dave and Daniel,

I found another workload affected by a3af84b0fa00 while running MMTests:
Redis ZPOPMIN throughput on v7.3 is about 13% lower than on v7.1.3.
Unlike the repeated filter loading discussed here, the slowdown remains
after the tasks that triggered the predictor flush (IBPB) have exited.
Can this cost be reduced while preserving the security protection?

The machine is an i7-12700KF with 32 GiB RAM. All runs used CPU microcode
0x3e, and I did not disable any security mitigations. The test uses
unmodified Redis 6.2.6 and MMTests' redis-benchmark-small recipe:

  redis-benchmark --csv -c 1 -r 50000 -n 100000 -P 256

There is one client, no CPU pinning, and all 20 logical CPUs are available.
After each boot, I ran the full MMTests recipe nine times. Each run kept
all 20 commands, including ZADD before ZPOPMIN, and the recipe's five
internal repetitions:

                           v7.1.3 first       v7.3      v7.1.3 return
  ZPOPMIN (requests/s)        999,366         871,397      1,000,947
  GET (requests/s)          3,548,521       3,551,132      3,547,933
  Complete test time (s)       61.396          61.704         61.301

ZPOPMIN is down 12.87% relative to the mean of the two v7.1.3 baselines.
Within each boot, its coefficient of variation is below 0.75%; the two
baselines differ by 0.16%.
GET is essentially unchanged (+0.08%). The complete test, including
startup and cleanup, takes 0.58% longer. The ZPOPMIN sequence drains the
set without refilling it, so its request rate includes empty results,
not just successful member removals.

Bisection identified a3af84b0fa00. To test the effect of a flush without
changing kernel code, I used its parent, 96cce16e26dd, where Redis was
still fast. On each CPU in turn, a short-lived helper enabled
indirect-branch protection for itself using:

  prctl(PR_SET_SPECULATION_CTRL, PR_SPEC_INDIRECT_BRANCH,
        PR_SPEC_FORCE_DISABLE, 0L, 0L)

After all helpers exited, I started Redis from a controller whose
protection setting was unchanged. Redis did not inherit the helpers'
setting.

ZPOPMIN throughput fell by 11.06% and 11.69% on two separate boots.
After another reboot, I ran the same helpers without changing their
protection settings. Throughput returned to baseline (a 1.33% difference).
Separate diagnostic runs found no BPF JIT allocations or associated IBPB
calls during the Redis test. They did show more samples of mispredicted
returns in kernel code, as on the slow kernel.

The tested v7.3 already includes a9b1f19a6a67, which reduces how often
BPF code allocation triggers an IBPB. This result concerns the cost
after an IBPB, rather than repeatedly loading filters during the
benchmark. I have not determined why the return-prediction penalty
persists after the helpers exit.

The helper used in these tests is attached as ibpb-pulse.c. On a dedicated
test machine, build it before rebooting:

  cc -O2 -Wall -Wextra -o ibpb-pulse ibpb-pulse.c

Boot the parent kernel, 96cce16e26dd, then run as a non-root user:

  for cpu in $(seq 0 19); do ./ibpb-pulse "$cpu" protect || exit 1; done

Then run the Redis test above against a fresh, dedicated Redis instance.
For the control, reboot into the same kernel and replace protect with
sham. The separate boot avoids carrying over the effect of the protect run.
The helper is limited to CPUs 0-19 and checks the initial protection state.
I would be happy to test changes that preserve the security protection.

Thanks,
Chengfeng

[-- Attachment #2: ibpb-pulse.c --]
[-- Type: text/x-csrc, Size: 1808 bytes --]

#define _GNU_SOURCE
#include <errno.h>
#include <sched.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/prctl.h>
#include <unistd.h>

/* Opt IN to protection for this disposable process only. Never enable
 * speculation, touch an MSR, change a system setting or run a benchmark. */
int main(int argc, char **argv)
{
    char *end;
    cpu_set_t mask, actual;
    if (argc != 3 || (strcmp(argv[2], "sham") && strcmp(argv[2], "protect")) ||
        getuid() == 0)
        return 2;
    errno = 0;
    long cpu = strtol(argv[1], &end, 10);
    if (errno || *end || cpu < 0 || cpu >= 20)
        return 3;
    CPU_ZERO(&mask);
    CPU_SET(cpu, &mask);
    if (sched_setaffinity(0, sizeof(mask), &mask) ||
        sched_getaffinity(0, sizeof(actual), &actual) ||
        CPU_COUNT(&actual) != 1 || !CPU_ISSET(cpu, &actual) || sched_getcpu() != cpu)
        return 4;
    int before = prctl(PR_GET_SPECULATION_CTRL, PR_SPEC_INDIRECT_BRANCH, 0L, 0L, 0L);
    if (before != (PR_SPEC_PRCTL | PR_SPEC_ENABLE)) {
        fprintf(stderr, "unexpected initial speculation mask %d errno %d\n", before, errno);
        return 5;
    }
    int protect = !strcmp(argv[2], "protect");
    if (protect && prctl(PR_SET_SPECULATION_CTRL, PR_SPEC_INDIRECT_BRANCH,
                         PR_SPEC_FORCE_DISABLE, 0L, 0L)) {
        perror("protection opt-in");
        return 6;
    }
    int after = prctl(PR_GET_SPECULATION_CTRL, PR_SPEC_INDIRECT_BRANCH, 0L, 0L, 0L);
    if (after != (protect ? (int)(PR_SPEC_PRCTL | PR_SPEC_FORCE_DISABLE) : before) ||
        sched_getcpu() != cpu)
        return 7;
    printf("{\"cpu\":%ld,\"pid\":%ld,\"uid\":%ld,\"mode\":\"%s\","
           "\"before\":%d,\"after\":%d}\n",
           cpu, (long)getpid(), (long)getuid(), argv[2], before, after);
    return 0;
}

Re: [linus:master] [x86/bugs] a3af84b0fa: stress-ng.seccomp.ops_per_sec 95.7% regression

[Pawan Gupta]

On Mon, Nov 02, 2026 at 06:43:58PM +0800, Chengfeng Lin wrote:
> Hi Pawan, Dave and Daniel,
> 
> I found another workload affected by a3af84b0fa00 while running MMTests:
> Redis ZPOPMIN throughput on v7.3 is about 13% lower than on v7.1.3.
> Unlike the repeated filter loading discussed here, the slowdown remains
> after the tasks that triggered the predictor flush (IBPB) have exited.
> Can this cost be reduced while preserving the security protection?
> 
> The machine is an i7-12700KF with 32 GiB RAM. All runs used CPU microcode
> 0x3e, and I did not disable any security mitigations. The test uses
> unmodified Redis 6.2.6 and MMTests' redis-benchmark-small recipe:
> 
>   redis-benchmark --csv -c 1 -r 50000 -n 100000 -P 256
> 
> There is one client, no CPU pinning, and all 20 logical CPUs are available.
> After each boot, I ran the full MMTests recipe nine times. Each run kept
> all 20 commands, including ZADD before ZPOPMIN, and the recipe's five
> internal repetitions:
> 
>                            v7.1.3 first       v7.3      v7.1.3 return
>   ZPOPMIN (requests/s)        999,366         871,397      1,000,947
>   GET (requests/s)          3,548,521       3,551,132      3,547,933
>   Complete test time (s)       61.396          61.704         61.301
> 
> ZPOPMIN is down 12.87% relative to the mean of the two v7.1.3 baselines.
> Within each boot, its coefficient of variation is below 0.75%; the two
> baselines differ by 0.16%.
> GET is essentially unchanged (+0.08%). The complete test, including
> startup and cleanup, takes 0.58% longer. The ZPOPMIN sequence drains the
> set without refilling it, so its request rate includes empty results,
> not just successful member removals.
> 
> Bisection identified a3af84b0fa00. To test the effect of a flush without
> changing kernel code, I used its parent, 96cce16e26dd, where Redis was
> still fast. On each CPU in turn, a short-lived helper enabled
> indirect-branch protection for itself using:
> 
>   prctl(PR_SET_SPECULATION_CTRL, PR_SPEC_INDIRECT_BRANCH,
>         PR_SPEC_FORCE_DISABLE, 0L, 0L)
> 
> After all helpers exited, I started Redis from a controller whose
> protection setting was unchanged. Redis did not inherit the helpers'
> setting.
> 
> ZPOPMIN throughput fell by 11.06% and 11.69% on two separate boots.
> After another reboot, I ran the same helpers without changing their
> protection settings. Throughput returned to baseline (a 1.33% difference).

So am I understanding it correctly that with the same kernel(without BPF
JIT mitigation) and same workload configuration produced different
performance results accross boots?

Is it possible for you to repeat the test with a different microcode? If
the microcode is late-loaded in your setup you can use dis_ucode_ldr kernel
parameter to skip loading the microcode.

Re: [linus:master] [x86/bugs] a3af84b0fa: stress-ng.seccomp.ops_per_sec 95.7% regression

[Dave Hansen]

On 11/2/26 02:43, Chengfeng Lin wrote:
> I found another workload affected by a3af84b0fa00 while running MMTests:
> Redis ZPOPMIN throughput on v7.3 is about 13% lower than on v7.1.3.
> Unlike the repeated filter loading discussed here, the slowdown remains
> after the tasks that triggered the predictor flush (IBPB) have exited.
> Can this cost be reduced while preserving the security protection?

I'm puzzled by this result. Any chance you could collect some profiles
and see where the difference is?