Re: futex: private hash can stay on the shared atomic refcount indefinitely after auto-scaling

[Nikita Taranov <nickita.taranov@gmail.com>]

> How bad is this in general? Do you have a workload where you fire a
> process, create 128 threads and then seconds later kill the whole thing?
> I would imagine that in general it is long living process and once the
> "final" privat hash gets installed, it remains…

The problem was originally observed on a synthetic benchmark that pretty much
did what the first repro does. On that benchmark, it was really bad as you saw.
I've never seen it on a real workload, so cannot comment on that.

Speaking of the benchmarks again, by "never recovers" I meant time ranges
comparable with the duration of the benchmark itself (few seconds).
For example, running the load continuously and printing throughput every second,
v7.2, 192 threads:

  9.0 3.4 3.4 3.2 3.4 ... [169 seconds at ~3.4] ... 3.5 24.8 34.6 34.7 34.8

So it recovers - after 169 seconds. Six further runs of 300s each, same
configuration, time to recovery:

  14s, 33s, 165s, 280s, ~300s, and one run didn't recover within 300s
  (it peaked at 6.9 Mops/s against 35-65 in the others)

The pre-sized control run at the same thread count is flat for the full 300s.

So the accurate statement is not "it never recovers" but "it recovers after an
unpredictable interval, sometimes seconds, sometimes several minutes, and the
process runs ~10x slower until it does".

> It depends on v7.3-rc1 and should keep new waiters blocked until the new
> hash is installed. I also added wake path assuming they will continue to
> matter what. It might make sense to do this on the wake side only.
> Does this do anything?

Yes. I built v7.3-rc1 twice on each machine - stock and with your patch, same
config, same LOCALVERSION.

Stalled runs out of 8, per thread count (auto-scaled vs a pre-sized control):

                          stock v7.3-rc1        + your patch
  Xeon 6975P-C 96c/192t   8/8 at 96,128,192     0/8 everywhere
  Graviton4 Neoverse-V2   see below             0/8 everywhere
  AMD EPYC 9R45           0/8 (never stalls)    0/8

On the Intel box the 300s continuous run is flat at ~36 Mops/s with your patch,
against 4 Mops/s stalled on stock. No hung-task splats in dmesg on any machine.

Graviton4 needs more than 8 runs to say anything, because it only reaches the
state a few percent of the time. At 192 threads, 24 runs per configuration:

                          stock v7.3-rc1        + your patch
  threads all at once      0/24                  0/24
  waker creation over 3s   4/24                  0/24

I'm guessing your patch was an early stage draft, but just in case let me flag
one problem I saw: on the Intel box the futex selftests regress:

  stock v7.3-rc1     14 passed, 0 failed
  + your patch       12 passed, 2 failed

  futex_requeue      requeue_multiple:
                       expected 10 == futex_cmp_requeue(f1, 0, &f2, 3, 7, 0), got 0
                       expected  7 == futex_wake(&f2, INT_MAX, 0), got 0
  futex_requeue_pi   2 of 24 fail: broadcast_owner, owner
                       "Test terminated by timeout"

stress-ng futex was clean, so it is specific to requeue.

Happy to test any following revision on these three machines.

Regards.