futex: private hash can stay on the shared atomic refcount indefinitely after auto-scaling

futex: private hash can stay on the shared atomic refcount indefinitely after auto-scaling

[Nikita Taranov]

Hi,

A threaded process doing sustained futex work can end up using
mm_struct::futex_atomic for reference counting for its entire life,
rather than the per-CPU counters introduced by

  56180dd20c19 ("futex: Use RCU-based per-CPU reference counting instead
                 of rcuref_t")

i.e. it silently falls back to the bottleneck that commit was written to
remove.  Introduced in 6.17 and still present in 7.0.0-1010-aws.
On a 96-core Granite Rapids with SNC=3 this costs ~90x
throughput on a futex-heavy microbenchmark and ~11x wall time on a
contended-mutex benchmark.  It is a threshold effect: the same reproducer
on two other 192-CPU machines (AMD Turin, Graviton4) does not stall at
all, and even on the Intel box 64 threads is fine while 96 is not.

The commit message anticipates a slow transition, but as latency:

  "The side effects would be that on auto scaling the new hash is used
   later and the SET_SLOTS prctl() will block longer."

What I am reporting is that under load it may not complete at all for
tens of seconds.

The trigger needs futex activity to overlap a hash growth, which is just
an ordinary thread-pool ramp: the growth is requested while the earlier
threads are already working.  That is not specific to startup -- a pool
that grows later under load is hit at least as hard, see below.  If the
process goes quiet for a moment afterwards the count drains and everything
is fine, which is why this is easy to miss and why the symptom is
bimodal.

I could not find this specific case in the archives -
apologies if this is already known.

#regzbot introduced: v6.16..v6.17


Reproducer
----------

96 threads, each issuing FUTEX_WAKE_PRIVATE on a private futex with no
waiters, so the syscall does little besides take and drop a hash
reference.  Threads start work as they are created -- deliberately no
start barrier, since an idle window after thread creation lets the pivot
complete and hides the problem.

Build with

  gcc -O2 -pthread futex_hash_repro.c -o futex_hash_repro

  # let the hash auto-scale (default)
  $ numactl --membind=0,1,2 taskset -c 0-95 ./futex_hash_repro 96 5 0
  threads=96 slots(start=0 set=0 end=512) 5.01s   13.0 Mops/s
  threads=96 slots(start=0 set=0 end=512) 5.01s    9.4 Mops/s
  threads=96 slots(start=0 set=0 end=512) 5.01s   11.7 Mops/s

  # pre-size the hash before any thread exists
  $ numactl --membind=0,1,2 taskset -c 0-95 ./futex_hash_repro 96 5 512
  threads=96 slots(start=0 set=512 end=512) 5.00s 1049.0 Mops/s
  threads=96 slots(start=0 set=512 end=512) 5.00s 1050.2 Mops/s
  threads=96 slots(start=0 set=512 end=512) 5.00s 1049.8 Mops/s

Both end with a 512-bucket hash, so this is not about the hash size or
collisions -- only about when it was sized.  Pre-sizing via
prctl(PR_FUTEX_HASH, PR_FUTEX_HASH_SET_SLOTS, 512) marks the hash custom,
futex_hash_allocate_default() then returns early, no pivot is requested,
and the per-CPU path stays in use.


Evidence that the hash is stuck in FR_ATOMIC
--------------------------------------------

A kprobe on futex_ref_get() reading fph->state, aggregated in-kernel over
a 3 s run:

  auto-scaled (default):
    { state:          0 } hitcount:    1333243      FR_PERCPU    7.5%
    { state:          1 } hitcount:   16461834      FR_ATOMIC   92.5%

  pre-sized:
    { state:          0 } hitcount:   37513027      FR_PERCPU    100%


Effect on a realistic workload
------------------------------

A contended std::mutex benchmark (96 threads over 32 mutexes, glibc-style
futex mutex with an unconditional FUTEX_WAKE on unlock), same binary,
differing only in whether the hash is pre-sized:

  auto-scaled:  52.8 46.1 36.3 51.7 49.6 s     median 49.6
  pre-sized:     4.2  4.5  4.0  5.1  4.6 s     median  4.5

The magnitude depends on how far mm->futex_atomic has to travel.  Across
the three SNC clusters of this socket it is ~11x; confining the same 96
threads to one cluster brings the run to ~3.

It is a regression against older kernels
----------------------------------------

Same machine, same binaries, only the kernel differs.  6.14 is the newest
pre-private-hash kernel I found available on EC2, so it is the
last-good point I actually tested; on it prctl(PR_FUTEX_HASH,
PR_FUTEX_HASH_GET_SLOTS) returns -1, i.e. the feature is absent.

                                    v6.14.0-1018-aws   v6.17.0-1019-aws
  reproducer, 96 threads              1082 Mops/s        11-14 Mops/s
  mutex bench 96T/32L store_wake      3.81-3.84 s        36-53 s
  mutex bench 16T/32L xchg             971 ms             960 ms
  mutex bench 16T/32L store_wake       620 ms             696 ms

The 16-thread rows matter as a control: below the stall threshold the two
kernels agree to within about 12%, so the 96-thread difference is the
stall and not a general slowdown between these versions.

It is not fixed as of 7.0.  On a second machine of the same type -- Xeon
6975P-C, 96 cores, SNC=3, running 7.0.0-1010-aws -- the reproducer behaves
exactly as on 6.17, and does so without any pinning or numactl:

  $ ./futex_hash_repro 96 3 512
  threads=96 slots(start=0 set=512 end=512) 3.00s 1024.9 Mops/s
  $ ./futex_hash_repro 96 3
  threads=96 slots(start=0 set=0 end=512) 3.01s   11.6 Mops/s

That is 88x, from one prctl.

The same conclusion can be reached without changing kernels, which may be
easier to reproduce: selecting the global hash on 6.17 with
prctl(PR_FUTEX_HASH, PR_FUTEX_HASH_SET_SLOTS, 0) restores 1020 Mops/s
against 11-14 for the default.


It is not limited to process startup
------------------------------------

A pool that grows later, while already serving futex load, is hit at
least as hard -- and there the hash is left undersized as well, because
the pivot that would install the bigger one never completes.  Threads
hammer futexes throughout; the pool is grown with no pause:

  start -> final     phase 1 (start threads)   phase 2 (final threads)
   8 -> 96     79.7 Mops/s 9.96/thr s=32    5.2 Mops/s 0.05/thr s=32
  32 -> 96    310.7 Mops/s 9.71/thr s=128   5.9 Mops/s 0.06/thr s=128
  64 -> 96    619.8 Mops/s 9.69/thr s=256   6.0 Mops/s 0.06/thr s=256

s = PR_FUTEX_HASH_GET_SLOTS.


Environment
-----------

  kernel   6.17.0-1019-aws (Ubuntu, x86_64)
  CPU      Intel Xeon 6975P-C, 96 cores / 192 threads, SNC=3
           (3 NUMA nodes per socket)
  glibc    Ubuntu 2.41
  governor performance, threads pinned, memory bound with numactl


Reproducer source
-----------------

--- 8< --- futex_hash_repro.c --- 8< ---
// Minimal reproducer: a process whose private futex hash is auto-scaled during
// thread creation can remain on the shared mm_struct::futex_atomic reference
// count for the rest of its futex-heavy life, instead of returning to the
// per-CPU counters that commit 56180dd20c19 ("futex: Use RCU-based per-CPU
// reference counting instead of rcuref_t") introduced.
//
// Each thread issues FUTEX_WAKE on a private futex with no waiters, so the
// syscall does essentially nothing except take and drop a reference on the
// process's private futex hash.
//
//   $ gcc -O2 -pthread futex_hash_repro.c -o futex_hash_repro
//   $ ./futex_hash_repro <threads> <seconds> [presize_slots]
//
// Pass presize_slots to call prctl(PR_FUTEX_HASH, PR_FUTEX_HASH_SET_SLOTS, n)
// before any thread is created, which marks the hash "custom" and disables the
// auto-scaling that requests the pivot.
//
// Pin to cores spread across NUMA/sub-NUMA boundaries to see the full effect;
// the cost is dominated by mm_struct::futex_atomic moving between them.

#define _GNU_SOURCE
#include <errno.h>
#include <linux/futex.h>
#include <pthread.h>
#include <stdatomic.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/prctl.h>
#include <sys/syscall.h>
#include <time.h>
#include <unistd.h>

#ifndef PR_FUTEX_HASH
#define PR_FUTEX_HASH 78
#define PR_FUTEX_HASH_SET_SLOTS 1
#define PR_FUTEX_HASH_GET_SLOTS 2
#endif

#define NWORDS 32
static int words[NWORDS * 16];          /* padded, one per cache line */
static atomic_long total_ops;
static int run_seconds;

static double now_s(void)
{
    struct timespec ts;
    clock_gettime(CLOCK_MONOTONIC, &ts);
    return (double)ts.tv_sec + (double)ts.tv_nsec / 1e9;
}

static void *worker(void *arg)
{
    long id = (long)arg;
    int *w = &words[(id % NWORDS) * 16];

    /*
     * Deliberately NO start barrier: each thread begins issuing futex
     * operations the moment it exists. That is what keeps a reference on the
     * private hash while the remaining threads are still being created, so the
     * auto-scale pivot requested at the final thread count is deferred. Adding
     * an idle window here lets the refcount drain and the pivot complete, and
     * the problem disappears -- which is itself part of the report.
     */
    long ops = 0;
    double deadline = now_s() + run_seconds;
    do {
        for (int i = 0; i < 1000; i++)
            syscall(SYS_futex, w, FUTEX_WAKE_PRIVATE, 1, NULL, NULL, 0);
        ops += 1000;
    } while (now_s() < deadline);

    atomic_fetch_add(&total_ops, ops);
    return NULL;
}

int main(int argc, char **argv)
{
    int nthreads = argc > 1 ? atoi(argv[1]) : 96;
    run_seconds  = argc > 2 ? atoi(argv[2]) : 5;
    int presize  = argc > 3 ? atoi(argv[3]) : 0;

    long before = prctl(PR_FUTEX_HASH, PR_FUTEX_HASH_GET_SLOTS, 0, 0, 0);
    /*
     * presize < 0 asks for slots=0, which selects the GLOBAL hash:
     * fph->hash_mask becomes 0 and __futex_hash() falls back to the global
     * table. That is the pre-6.17 world, where CONFIG_FUTEX_PRIVATE_HASH
     * was "depends on BROKEN".
     */
    if (presize) {
        int slots = presize < 0 ? 0 : presize;
        if (prctl(PR_FUTEX_HASH, PR_FUTEX_HASH_SET_SLOTS, slots, 0, 0))
            fprintf(stderr, "prctl(SET_SLOTS, %d): %s\n",
                    slots, strerror(errno));
        presize = slots;
    }
    long after = prctl(PR_FUTEX_HASH, PR_FUTEX_HASH_GET_SLOTS, 0, 0, 0);

    pthread_t *t = calloc(nthreads, sizeof(*t));
    for (long i = 0; i < nthreads; i++)
        pthread_create(&t[i], NULL, worker, (void *)i);

    double t0 = now_s();
    for (int i = 0; i < nthreads; i++)
        pthread_join(t[i], NULL);
    double el = now_s() - t0;

    long ops = atomic_load(&total_ops);
    long at_end = prctl(PR_FUTEX_HASH, PR_FUTEX_HASH_GET_SLOTS, 0, 0, 0);
    printf("threads=%d slots(start=%ld set=%ld end=%ld) "
           "%.2fs %.1f Mops/s\n",
           nthreads, before, after, at_end,
           el, (double)ops / el / 1e6);
    return 0;
}
--- >8 --- futex_hash_repro.c --- >8 ---

Re: futex: private hash can stay on the shared atomic refcount indefinitely after auto-scaling

[Sebastian Andrzej Siewior]

On 2026-08-23 15:00:43 [+0000], Nikita Taranov wrote:
> Hi,
Hi,

> The commit message anticipates a slow transition, but as latency:
> 
>   "The side effects would be that on auto scaling the new hash is used
>    later and the SET_SLOTS prctl() will block longer."
> 
> What I am reporting is that under load it may not complete at all for
> tens of seconds.

Hmm. There are two things that can an effect how quick the transition
is:
- The completion of an RCU grace period to ensure all futex users moved
  from per-CPU reference counting to atomic
- The possibility of all FUTEX participants (all threads of the task) to
  not use futex for a while.

The latter means, if you multiple FUTEX users all the time (as seen in
your example) then the reference counter never drops to 0. As such, the
transition to the new private hash bucket can not happen.

> The trigger needs futex activity to overlap a hash growth, which is just
> an ordinary thread-pool ramp: the growth is requested while the earlier
> threads are already working.  That is not specific to startup -- a pool
> that grows later under load is hit at least as hard, see below.  If the
> process goes quiet for a moment afterwards the count drains and everything
> is fine, which is why this is easy to miss and why the symptom is
> bimodal.

Right. As long as you allow it to settle then everything will be fine.
Once the reference counter was allowed to drop to 0, the first user will
install the new private-hash and all other threads wait until it is
done.

…
> 96 threads, each issuing FUTEX_WAKE_PRIVATE on a private futex with no
> waiters, so the syscall does little besides take and drop a hash
> reference.  Threads start work as they are created -- deliberately no
> start barrier, since an idle window after thread creation lets the pivot
> complete and hides the problem.
> 
> Build with
> 
>   gcc -O2 -pthread futex_hash_repro.c -o futex_hash_repro
> 
>   # let the hash auto-scale (default)
>   $ numactl --membind=0,1,2 taskset -c 0-95 ./futex_hash_repro 96 5 0
>   threads=96 slots(start=0 set=0 end=512) 5.01s   13.0 Mops/s
>   threads=96 slots(start=0 set=0 end=512) 5.01s    9.4 Mops/s
>   threads=96 slots(start=0 set=0 end=512) 5.01s   11.7 Mops/s
> 
>   # pre-size the hash before any thread exists
>   $ numactl --membind=0,1,2 taskset -c 0-95 ./futex_hash_repro 96 5 512
>   threads=96 slots(start=0 set=512 end=512) 5.00s 1049.0 Mops/s
>   threads=96 slots(start=0 set=512 end=512) 5.00s 1050.2 Mops/s
>   threads=96 slots(start=0 set=512 end=512) 5.00s 1049.8 Mops/s

You might have hit the sweet spot with your CPU. I have here a 144 CPU
box and 

| bigeasy@herakles:~$ ./futex_hash_repro 140 5 0
| threads=140 slots(start=0 set=0 end=1024) 5.00s 659.2 Mops/s
| bigeasy@herakles:~$ ./futex_hash_repro 140 5 4096
| threads=140 slots(start=0 set=4096 end=4096) 5.00s 789.1 Mops/s
| bigeasy@herakles:~$ ./futex_hash_repro 140 5 4096
| threads=140 slots(start=0 set=4096 end=4096) 5.00s 786.4 Mops/s
| bigeasy@herakles:~$ ./futex_hash_repro 140 5 0
| threads=140 slots(start=0 set=0 end=1024) 5.00s 631.1 Mops/s
|
| bigeasy@herakles:~$ taskset -c 0-95 ./futex_hash_repro 96 5 0
| threads=96 slots(start=0 set=0 end=512) 5.00s 704.4 Mops/s
| bigeasy@herakles:~$ taskset -c 0-95 ./futex_hash_repro 96 5 512
| threads=96 slots(start=0 set=512 end=512) 5.00s 737.4 Mops/s
| bigeasy@herakles:~$ taskset -c 0-95 ./futex_hash_repro 96 5 512
| threads=96 slots(start=0 set=512 end=512) 5.00s 733.1 Mops/s
| bigeasy@herakles:~$ taskset -c 0-95 ./futex_hash_repro 96 5 0
| threads=96 slots(start=0 set=0 end=512) 5.00s 690.6 Mops/s

It is not as bad as you describe. Could you try v7.2 which has commit
a734d9fca84e1 ("futex: Optimize futex hash bucket access patterns")
which might help to settle the counter.

> The magnitude depends on how far mm->futex_atomic has to travel.  Across
> the three SNC clusters of this socket it is ~11x; confining the same 96
> threads to one cluster brings the run to ~3.

Not sure what we could do here. One idea might be to block further
futex syscalls so they don't acquire a new reference on the existing
hash and allow a transition to the new hash more quickly.

Sebastian

Re: futex: private hash can stay on the shared atomic refcount indefinitely after auto-scaling

[Nikita Taranov]

On 2026-08-24 18:03:37 [+0200], Sebastian Andrzej Siewior wrote:
> It is not as bad as you describe. Could you try v7.2 which has commit
> a734d9fca84e1 ("futex: Optimize futex hash bucket access patterns")
> which might help to settle the counter.

I built v7.2 from the release tarball and booted it on the same machine.
It does not change the behaviour:

  # v7.2, 96 threads x 5 s, numactl --membind=0,1,2 taskset -c 0-95
  threads=96 slots(start=0 set=0   end=512) 5.01s   12.0 Mops/s
  threads=96 slots(start=0 set=0   end=512) 5.01s    8.1 Mops/s
  threads=96 slots(start=0 set=0   end=512) 5.01s    9.8 Mops/s
  threads=96 slots(start=0 set=512 end=512) 5.00s 1128.7 Mops/s
  threads=96 slots(start=0 set=512 end=512) 5.00s 1128.1 Mops/s
  threads=96 slots(start=0 set=512 end=512) 5.00s 1128.2 Mops/s

Unpinned, as you ran it, is the same: 8.7 / 11.5 / 11.4 auto-scaled vs
1128.2 / 1127.3 / 1127.6 pre-sized.

I rebooted the same box back to 7.0 and re-ran the identical commands, so
the two are directly comparable: 9.8 / 10.2 / 9.4 auto-scaled against
1023.2 / 1022.8 / 1022.7 pre-sized.

> You might have hit the sweet spot with your CPU. I have here a 144 CPU
> box and

Yes, this Intel host is the only one where the repro worked for me:

  ratio = pre-sized / auto-scaled; 4 s runs, threads pinned to cores,
  pre-sized to 512 slots; kernel 6.17

  threads   Xeon 6975P-C     EPYC 9R45      Graviton4
            96c/192t SNC=3   96c/192t       96c/192t
      2       1.0x             1.0x           1.0x
      4       1.0x             1.0x           1.0x
      8       1.1x             1.1x           1.1x
     16       1.1x             1.1x           1.1x
     32       1.1x             1.1x           1.1x
     48       1.1x             1.1x           1.1x
     64       1.1x             1.1x           1.1x
     96      81.8x             1.1x           1.1x

For reference, the same test with different thread count and placement:

  placement of 96 threads              auto-scaled   pre-sized   ratio
  96 cores, socket 0, mem interleaved       14.3       1048.2     73.3x
  96 threads on 32 cores, one node         334.8        349.2      1.0x
  96 threads on  8 cores, one node          85.6         87.2      1.0x

One more thing that may be useful: whether the pivot completes is a race.
Ten runs at each thread count, 3 s each, on v7.2 on this box
(stalled = under 100 Mops/s):

  T=24   stalled  0/10    259.3 249.3 261.3 250.0 251.6 ...
  T=32   stalled  0/10    334.6 332.0 335.3 326.5 335.3 ...
  T=40   stalled  0/10    415.3 300.5 415.7 389.1 416.3 ...
  T=48   stalled  1/10    251.2 423.8 482.1   6.5 410.8 ...
  T=56   stalled  8/10    526.9   6.2   7.0   7.0 540.0 ...
  T=64   stalled 10/10      9.4   8.6   8.0   8.5   9.3 ...
  T=72   stalled  9/10     11.7  10.7  12.2 331.0  10.1 ...
  T=96   stalled 10/10     11.8   9.4  12.1  12.9  13.4 ...

> Not sure what we could do here. One idea might be to block further
> futex syscalls so they don't acquire a new reference on the existing
> hash and allow a transition to the new hash more quickly.

I am happy to test a patch on this machine if that is useful; it reproduces
in about five seconds and the difference is 100x, so it should be an easy
signal to read.

Thanks for looking at this.

Regards.

Re: futex: private hash can stay on the shared atomic refcount indefinitely after auto-scaling

[Nikita Taranov]

Following up with three things: a better reproducer, results from two more
vendors, and a prototype of the approach you suggested.

## A new repro

The code could be found at https://gist.github.com/nickitat/a7a46b61d6769ceec493881f82e7b344 .

This one is an ordinary producer/consumer: threads park in plain FUTEX_WAIT
like the idle workers of a thread pool, others issue plain FUTEX_WAKE on the
same futexes, so waiters are woken, loop and re-park. Only plain FUTEX_WAIT and
FUTEX_WAKE. The kernel's own auto-scaling during thread creation is the only
thing that ever asks for a pivot.

The control is one prctl(PR_FUTEX_HASH, PR_FUTEX_HASH_SET_SLOTS, n) before any
thread exists, which marks the hash custom and disables auto-scaling, so no
pivot is ever requested and the stall cannot occur. Basically, the same story
as in the first repro.

v7.2, Xeon 6975P-C 96c/192t SNC=3, 8 runs of 4 s per point, clock pinned:

  threads   auto   pre-sized   stalled
       16   25.8        42.3      0/8
       32   34.8        41.1      0/8
       48   28.7        31.8      0/8
       64   29.9        34.2      0/8
       96    3.7        46.3      8/8
      128    3.6        53.6      8/8
      192    4.0        38.7      8/8

Throughput is far lower than the numbers in my earlier mails because the work
is real; the comparison is the point. The control (pre-sized) is almost flat
across the whole range, so the work does not get harder with thread count
-- only the arm that requests a pivot collapses, and only above a threshold.

## It is not only this Intel box

> You might have hit the sweet spot with your CPU. I have here a 144 CPU
> box and

I wrote previously that this Intel host was the only one where the repro worked,
based on a 96-thread sweep with the old reproducer. At 192 threads, with the
reproducer above, Graviton4 does reach the same state -- less often,
but permanently when it does.

192 threads, 15 s runs, per-second traces:

                        latched permanently   transient   never
  Xeon 6975P-C                  8 / 8              -        0
  Graviton4 Neoverse-V2         1 / 15           2 / 15    12 / 15
  AMD EPYC 9R45                 0 / 14          14 / 14      -

Graviton4 will stall at a higher rate if the wakers are created gradually
rather than all at once -- the `ramp_ms` parameter spreads their creation
over that many milliseconds in total, which is what a thread pool growing
while already under load does. I saw 5/16 stalls on one run.
AMD still never latched in ~200 runs.

So the state is reachable on more than one vendor's hardware and usually permanent
once entered. What differs is how often a process falls in.
Arguably, this flakiness is worse than a deterministic stall for a real production.

## Prototype of your suggestion

> Not sure what we could do here. One idea might be to block further
> futex syscalls so they don't acquire a new reference on the existing
> hash and allow a transition to the new hash more quickly.

The only analogy available to me due to lack of any knowledge is rw-locks. I asked
Claude to adapt the percpu-rwsem code to futex, and after a few iterations we
produced a hopefully viable patch. I suppose it implements plus-minus your suggestion.

The results (same box and all the rest of the environment):

  - the reproducer: 0/8 stalled at every thread count, against 8/8 on stock at
    96 and above; 192 threads run at 54 Mops/s instead of 4.
  - futex kselftests: 14 passed, 0 failed, with the mechanism enabled and
    disabled; futex_numa and futex_numa_mpol pass.
  - requeue-PI selftests looped under a continuous pivot storm, to exercise the
    drop_fph handoff against the phase: all rounds passed.
  - stress-ng: 64 futex stressors, and a 40-way futex/clone/mmap mix, 0 failures.
  - PROVE_LOCKING and KASAN builds: clean.
  - measured overhead ~0.6% on a pthread_mutex/pthread_cond benchmark with pinned clock.

Makes sense to note that the patch was applied to v7.2 and all testing was done on that build.

The code could be found at https://gist.github.com/nickitat/0b1cc68deff5fe2b27209b150da6fb00 .
If you find it useful, I can prepare a proper patch for the kernel tree.

Regards.

Re: futex: private hash can stay on the shared atomic refcount indefinitely after auto-scaling

[Sebastian Andrzej Siewior]

On 2026-09-01 23:05:59 [+0000], Nikita Taranov wrote:
> Following up with three things: a better reproducer, results from two more
> vendors, and a prototype of the approach you suggested.
> 
> ## A new repro
> 
> The code could be found at https://gist.github.com/nickitat/a7a46b61d6769ceec493881f82e7b344 .
> 
> This one is an ordinary producer/consumer: threads park in plain FUTEX_WAIT
> like the idle workers of a thread pool, others issue plain FUTEX_WAKE on the
> same futexes, so waiters are woken, loop and re-park. Only plain FUTEX_WAIT and
> FUTEX_WAKE. The kernel's own auto-scaling during thread creation is the only
> thing that ever asks for a pivot.
> 
> The control is one prctl(PR_FUTEX_HASH, PR_FUTEX_HASH_SET_SLOTS, n) before any
> thread exists, which marks the hash custom and disables auto-scaling, so no
> pivot is ever requested and the stall cannot occur. Basically, the same story
> as in the first repro.

Depends. That PR_FUTEX_HASH_SET_SLOTS will block the requesting thread
until the change is active. The auto-scaling does not block the
requesting thread (the one creating the thread).
If you have already 20 threads which are busy on FUTEX_WAIT/WAKE, the
PR_FUTEX_HASH_SET_SLOTS request can also take some time if the active
counter never drops to zero (because multiple threads always own a
reference).

> v7.2, Xeon 6975P-C 96c/192t SNC=3, 8 runs of 4 s per point, clock pinned:
> 
>   threads   auto   pre-sized   stalled
>        16   25.8        42.3      0/8
>        32   34.8        41.1      0/8
>        48   28.7        31.8      0/8
>        64   29.9        34.2      0/8
>        96    3.7        46.3      8/8
>       128    3.6        53.6      8/8
>       192    4.0        38.7      8/8
> 
> Throughput is far lower than the numbers in my earlier mails because the work
> is real; the comparison is the point. The control (pre-sized) is almost flat
> across the whole range, so the work does not get harder with thread count
> -- only the arm that requests a pivot collapses, and only above a threshold.

How bad is this in general? Do you have a workload where you fire a
process, create 128 threads and then seconds later kill the whole thing?
I would imagine that in general it is long living process and once the
"final" privat hash gets installed, it remains…

> ## It is not only this Intel box
> 
> > You might have hit the sweet spot with your CPU. I have here a 144 CPU
> > box and
> 
> I wrote previously that this Intel host was the only one where the repro worked,
> based on a 96-thread sweep with the old reproducer. At 192 threads, with the
> reproducer above, Graviton4 does reach the same state -- less often,
> but permanently when it does.
> 
> 192 threads, 15 s runs, per-second traces:
> 
>                         latched permanently   transient   never
>   Xeon 6975P-C                  8 / 8              -        0
>   Graviton4 Neoverse-V2         1 / 15           2 / 15    12 / 15
>   AMD EPYC 9R45                 0 / 14          14 / 14      -
> 
> Graviton4 will stall at a higher rate if the wakers are created gradually
> rather than all at once -- the `ramp_ms` parameter spreads their creation
> over that many milliseconds in total, which is what a thread pool growing
> while already under load does. I saw 5/16 stalls on one run.
> AMD still never latched in ~200 runs.
> 
> So the state is reachable on more than one vendor's hardware and usually permanent
> once entered. What differs is how often a process falls in.
> Arguably, this flakiness is worse than a deterministic stall for a real production.
> 
> ## Prototype of your suggestion
> 
> > Not sure what we could do here. One idea might be to block further
> > futex syscalls so they don't acquire a new reference on the existing
> > hash and allow a transition to the new hash more quickly.
> 
> The only analogy available to me due to lack of any knowledge is rw-locks. I asked
> Claude to adapt the percpu-rwsem code to futex, and after a few iterations we
> produced a hopefully viable patch. I suppose it implements plus-minus your suggestion.
> 
> The results (same box and all the rest of the environment):
> 
>   - the reproducer: 0/8 stalled at every thread count, against 8/8 on stock at
>     96 and above; 192 threads run at 54 Mops/s instead of 4.
>   - futex kselftests: 14 passed, 0 failed, with the mechanism enabled and
>     disabled; futex_numa and futex_numa_mpol pass.
>   - requeue-PI selftests looped under a continuous pivot storm, to exercise the
>     drop_fph handoff against the phase: all rounds passed.
>   - stress-ng: 64 futex stressors, and a 40-way futex/clone/mmap mix, 0 failures.
>   - PROVE_LOCKING and KASAN builds: clean.
>   - measured overhead ~0.6% on a pthread_mutex/pthread_cond benchmark with pinned clock.
> 
> Makes sense to note that the patch was applied to v7.2 and all testing was done on that build.
> 
> The code could be found at https://gist.github.com/nickitat/0b1cc68deff5fe2b27209b150da6fb00 .
> If you find it useful, I can prepare a proper patch for the kernel tree.

So you create a new state and try to not deadlock. I'm too sure about
the PI bits.
But what about

diff --git a/kernel/futex/core.c b/kernel/futex/core.c
index 51ba5e1257c04..b370b4cf168eb 100644
--- a/kernel/futex/core.c
+++ b/kernel/futex/core.c
@@ -1975,6 +1975,39 @@ static int futex_hash_allocate(unsigned int hash_slots, unsigned int flags)
 	return 0;
 }
 
+void futex_hash_wait_new(void)
+{
+	struct wait_bit_queue_entry __wbq_entry;
+	struct mm_struct *mm = current->mm;
+	struct wait_queue_head *__wq_head;
+
+	if (!mm->futex.phash.hash_new)
+		return;
+
+	__wq_head = __var_waitqueue(mm);
+	init_wait_var_entry(&__wbq_entry, mm, 0);
+	__wbq_entry.wq_entry.func = woken_wake_bit_function;
+	add_wait_queue(__wq_head, &__wbq_entry.wq_entry);
+
+	/*
+	 * add_wait_queue()		futex_ref_put()
+	 * MB (this)			MB (implied)
+	 * futex_pivot_pending()	wake_up_var()
+	 *                                waitqueue_active()
+	 *
+	 * Notably, it must not be possible to see
+	 * !futex_pivot_pending() && !waitqueue_active().
+	 */
+	smp_mb();
+
+	while (!futex_pivot_pending(mm) &&
+	       wait_woken(&__wbq_entry.wq_entry, TASK_UNINTERRUPTIBLE,
+			  MAX_SCHEDULE_TIMEOUT))
+		/* empty */;
+
+	remove_wait_queue(__wq_head, &__wbq_entry.wq_entry);
+}
+
 int futex_hash_allocate_default(void)
 {
 	unsigned int threads, buckets, current_buckets = 0;
diff --git a/kernel/futex/futex.h b/kernel/futex/futex.h
index f00f0863ed441..d538e8e15c336 100644
--- a/kernel/futex/futex.h
+++ b/kernel/futex/futex.h
@@ -246,10 +246,11 @@ struct futex_bucket_ref {
 #ifdef CONFIG_FUTEX_PRIVATE_HASH
 extern struct futex_private_hash *futex_private_hash(struct mm_struct *mm);
 extern void futex_private_hash_put(struct futex_private_hash *fph);
-
+extern void futex_hash_wait_new(void);
 #else /* !CONFIG_FUTEX_PRIVATE_HASH */
 static inline struct futex_private_hash *futex_private_hash(struct mm_struct *mm) { return NULL; }
 static inline void futex_private_hash_put(struct futex_private_hash *fph) { }
+static inline void futex_hash_wait_new(void) { }
 #endif
 
 extern struct futex_bucket_ref futex_hash(union futex_key *key);
diff --git a/kernel/futex/waitwake.c b/kernel/futex/waitwake.c
index d4483d15d30a5..7ad8a060371c6 100644
--- a/kernel/futex/waitwake.c
+++ b/kernel/futex/waitwake.c
@@ -195,6 +195,8 @@ int futex_wake(u32 __user *uaddr, unsigned int flags, void __user *pop, int nr_w
 	if ((flags & FLAGS_STRICT) && !nr_wake)
 		return 0;
 
+	futex_hash_wait_new();
+
 	CLASS(hbr, hbr)(&key);
 	auto hb = hbr.hb;
 
@@ -283,6 +285,7 @@ int futex_wake_op(u32 __user *uaddr1, unsigned int flags, u32 __user *uaddr2,
 	int ret, op_ret;
 	DEFINE_WAKE_Q(wake_q);
 
+	futex_hash_wait_new();
 retry:
 	ret = get_futex_key(uaddr1, flags, &key1, FUTEX_READ);
 	if (unlikely(ret != 0))
@@ -744,6 +747,8 @@ int futex_wait(u32 __user *uaddr, unsigned int flags, u32 val, ktime_t *abs_time
 	struct restart_block *restart;
 	int ret;
 
+	futex_hash_wait_new();
+
 	to = futex_setup_timer(abs_time, &timeout, flags,
 			       current->timer_slack_ns);
 

It depends on v7.3-rc1 and should keep new waiters blocked until the new
hash is installed. I also added wake path assuming they will continue to
matter what. It might make sense to do this on the wake side only.
Does this do anything?

> Regards.

Sebastian

Re: futex: private hash can stay on the shared atomic refcount indefinitely after auto-scaling

[Nikita Taranov]

> How bad is this in general? Do you have a workload where you fire a
> process, create 128 threads and then seconds later kill the whole thing?
> I would imagine that in general it is long living process and once the
> "final" privat hash gets installed, it remains…

The problem was originally observed on a synthetic benchmark that pretty much
did what the first repro does. On that benchmark, it was really bad as you saw.
I've never seen it on a real workload, so cannot comment on that.

Speaking of the benchmarks again, by "never recovers" I meant time ranges
comparable with the duration of the benchmark itself (few seconds).
For example, running the load continuously and printing throughput every second,
v7.2, 192 threads:

  9.0 3.4 3.4 3.2 3.4 ... [169 seconds at ~3.4] ... 3.5 24.8 34.6 34.7 34.8

So it recovers - after 169 seconds. Six further runs of 300s each, same
configuration, time to recovery:

  14s, 33s, 165s, 280s, ~300s, and one run didn't recover within 300s
  (it peaked at 6.9 Mops/s against 35-65 in the others)

The pre-sized control run at the same thread count is flat for the full 300s.

So the accurate statement is not "it never recovers" but "it recovers after an
unpredictable interval, sometimes seconds, sometimes several minutes, and the
process runs ~10x slower until it does".

> It depends on v7.3-rc1 and should keep new waiters blocked until the new
> hash is installed. I also added wake path assuming they will continue to
> matter what. It might make sense to do this on the wake side only.
> Does this do anything?

Yes. I built v7.3-rc1 twice on each machine - stock and with your patch, same
config, same LOCALVERSION.

Stalled runs out of 8, per thread count (auto-scaled vs a pre-sized control):

                          stock v7.3-rc1        + your patch
  Xeon 6975P-C 96c/192t   8/8 at 96,128,192     0/8 everywhere
  Graviton4 Neoverse-V2   see below             0/8 everywhere
  AMD EPYC 9R45           0/8 (never stalls)    0/8

On the Intel box the 300s continuous run is flat at ~36 Mops/s with your patch,
against 4 Mops/s stalled on stock. No hung-task splats in dmesg on any machine.

Graviton4 needs more than 8 runs to say anything, because it only reaches the
state a few percent of the time. At 192 threads, 24 runs per configuration:

                          stock v7.3-rc1        + your patch
  threads all at once      0/24                  0/24
  waker creation over 3s   4/24                  0/24

I'm guessing your patch was an early stage draft, but just in case let me flag
one problem I saw: on the Intel box the futex selftests regress:

  stock v7.3-rc1     14 passed, 0 failed
  + your patch       12 passed, 2 failed

  futex_requeue      requeue_multiple:
                       expected 10 == futex_cmp_requeue(f1, 0, &f2, 3, 7, 0), got 0
                       expected  7 == futex_wake(&f2, INT_MAX, 0), got 0
  futex_requeue_pi   2 of 24 fail: broadcast_owner, owner
                       "Test terminated by timeout"

stress-ng futex was clean, so it is specific to requeue.

Happy to test any following revision on these three machines.

Regards.

Re: futex: private hash can stay on the shared atomic refcount indefinitely after auto-scaling

[Sebastian Andrzej Siewior]

On 2026-09-02 21:31:42 [+0000], Nikita Taranov wrote:
> The problem was originally observed on a synthetic benchmark that pretty much
> did what the first repro does. On that benchmark, it was really bad as you saw.
> I've never seen it on a real workload, so cannot comment on that.

Okay. In that case I need to figure if we want to care about a
regression on micro benchmark.

>                           stock v7.3-rc1        + your patch
>   Xeon 6975P-C 96c/192t   8/8 at 96,128,192     0/8 everywhere
>   Graviton4 Neoverse-V2   see below             0/8 everywhere
>   AMD EPYC 9R45           0/8 (never stalls)    0/8

So this means my quick hacked fixed your problem?

> I'm guessing your patch was an early stage draft, but just in case let me flag
> one problem I saw: on the Intel box the futex selftests regress:
> 
>   stock v7.3-rc1     14 passed, 0 failed
>   + your patch       12 passed, 2 failed
> 
>   futex_requeue      requeue_multiple:
>                        expected 10 == futex_cmp_requeue(f1, 0, &f2, 3, 7, 0), got 0
>                        expected  7 == futex_wake(&f2, INT_MAX, 0), got 0
>   futex_requeue_pi   2 of 24 fail: broadcast_owner, owner
>                        "Test terminated by timeout"

Yeah, I might have got something wrong. It was just an idea but might
look into this if we need this fixed. Thanks for the feedback.

> Regards.

Sebastian