Re: [RFC PATCH v24 00/10] Hierarchical Percpu Counters for RSS

[Mathieu Desnoyers <mathieu.desnoyers@efficios.com>]

Subject: Re: [RFC PATCH v24 00/10] Hierarchical Percpu Counters for RSS
In-Reply-To: <88a96126-11be-41cc-ab0d-372d32189e79@kernel.org>

On 2026-11-09 15:35, David Hildenbrand (Arm) wrote:
> On 11/5/26 21:36, Mathieu Desnoyers wrote:
[...]
> I think we discussed other possible applications (IIRC Michal expressed concern
> if a single user is sufficient motivation for these additional LOC). Did
> anything result from that discussion that is relevant to v24?

Yes. The next round adds a second user: the tmpfs size limit (the size=
mount option).

Today, each page charged to a tmpfs mount checks a struct percpu_counter
with percpu_counter_limited_add(). That check takes no lock only while
the mount is more than batch * nr_cpus pages below its size. At 384
CPUs, this is 1.1 GB. A smaller mount, such as a 64 MB /dev/shm, takes
the counter spinlock and sums all CPUs on every page.

The next round adds a limit counter built on the tree. It keeps the
limit strict, without a lock. fallocate() of one page at a time on a
64 MB mount, in a 384-vCPU guest, in thousands of calls per second:

   processes   before    after
           1      209      481
         384      124    53944

Mounts of 2 GB and above lose 1 to 2% with 384 processes. These are
patches 8 and 9 of the next round, right after the RSS patches.

A later follow-up would use the comparison functions for the OOM killer
task selection.

[...]
>> Since commit f1a7941243c1 ("mm: convert mm's rss stats into
>> percpu_counter"), the RSS counters are struct percpu_counter with a
>> batch size of max(32, 2 * nr_cpus): their approximation error grows with
>> O(nr_cpus^2), up to 1.1GB per counter at 384 logical CPUs.
>
> Okay, so that's a per-MM problem, not a per-system problem. On a large system
> with many CPUs I'd have said "likel we don't care" :)

Right. The error is per process, so a large system does not make it
small compared to the process.

>> /proc/PID/status, /proc/PID/statm and the OOM killer moved to precise
>> sums (commits 82241a83cd15 and 5898aa8f9a0b), but the other readers
>> still use the approximation: the RSS high watermark (VmHWM, the
>> ru_maxrss reported as "Maximum resident set size" by /usr/bin/time -v,
>> taskstats), /proc/PID/stat (ps, top, monitoring agents), the taskstats
>> RSS-time integral (coremem, sampled at each tick), NUMA balancing and
>> the mark_victim tracepoint.  On the test machine below, the
>> /proc/PID/stat RSS of a test process was off by 17MB while
>> single-threaded, and by up to 134MB with 64 threads.
>
> What's the resulting impact of that (temporary) imprecision?

It depends on the reader:

- The high watermark keeps it. update_hiwater_rss() stores the maximum
   of the approximation, so a temporary overshoot stays in VmHWM and
   ru_maxrss. /usr/bin/time -v and taskstats report that value.
- The taskstats coremem integral adds the approximate RSS at each tick,
   so the error accumulates.
- /proc/PID/stat shows it as is to ps, top and monitoring agents. In
   testing at 384 CPUs, it was off by 17 MB for a single-threaded
   process, and by up to 134 MB with 64 threads.
- NUMA balancing makes its decisions on it.

>> Precise sums cannot fix the high watermark or the RSS-time integral:
>> update_hiwater_rss() samples the RSS on every unmap path, and the
>
> As a side-note, I absolutely hate update_hiwater_rss() on the unmap path.

:) It is also why a precise sum is not an option there.

[...]
>> Patches 1 to 5 replace struct percpu_counter for the RSS counters with
>> hierarchical per-cpu counters: per-cpu bytes carry into a tree of atomic
>> counters shaped at boot from nr_cpu_ids, with a fixed batch of 32, an
>> error bound in O(n*log(n)) and no global lock.
>
> How does this interact with CPU hotplug?

Like the per-cpu data of struct percpu_counter, the tree is shaped at
boot from nr_cpu_ids. It is not reshaped on hotplug. The residual of an
offline CPU stays in place. The precise sum still reads it, and the
bound still holds. Patch 1 has a "CPU hotplug" section on this.

[...]
>> Patches 6 and 7 keep the RSS counters of single-threaded processes exact
>> and without per-cpu memory: until a second thread shares the mm, the
>> task running on the mm adds to the bias of each counter without atomic
>> instructions, and other tasks add to its atomic bias, both read in O(1).
[...]
>
> What was the particular reason to special-case single-threaded apps? Do we
> expect them in practice to really be that RSS heavy+dynamic such that it would
> matter?

The main reason is the cost of creating a process, more than accuracy.
Every exec creates a new mm. Upstream allocates per-cpu memory for its
RSS counters there, under the per-cpu allocator mutex. It also sums all
CPUs when the mm goes away. Most processes are single-threaded and
never need this per-cpu memory.

With 64 processes spawning a static program on a 384-CPU host, 19% of
the CPU time goes to these allocations and sums. Here is posix_spawn()
of a static program which exits, in a 384-vCPU guest. The gain is over
patches 1 to 5, to isolate the single-threaded mode, as the median of 3
to 5 runs:

   processes   gain
           1   +22%
          16   +26%
          64   +42%
         384   +22%

In UnixBench with one copy, Process Creation gains 15%, Execl Throughput
4% and Shell Scripts 6 to 9% (one sample each). With 384 copies, the
results are within 5% of upstream, in both directions. A kernel build at
-j384 does not change.

It also saves memory: at 384 CPUs, a single-threaded mm uses 1928 bytes
instead of 8072. And a zeroed counter is a valid counter in this mode,
so a static mm such as init_mm has working RSS counters without setup.

>> Patches 8 to 10 are the tools to tune the tree shape on other hardware:
>> the percpu_counter_tree.arity= and rss_stat_batch= boot parameters, and
>> a KUnit benchmark.  The default shapes were measured on one machine,
>> with up to 384 logical CPUs, so results from other hardware are welcome.
>
> Hm, so that means one might have to fine tune these toggles to get the best
> performance? That's a bit unfortunate.

They are not meant to be needed. The tree shape is chosen at boot from
nr_cpu_ids, and the RSS batch size is fixed. The parameters and the
benchmark are there to check these defaults on other hardware, because
I only measured them on one machine.

[...]
>> 1. Is mm->rss_owner acceptable, or would you prefer current->mm == mm
>>    (without PF_KTHREAD) and an "activated" flag?
>
> I'm afraid I'm the wrong person to ask that question. I'd have to look into the
> details. If it's just a simple member, we likely don't care about the addition
> memory footprint (if that's your main concern).

OK, thanks.

[...]
>> 4. Should the rss_stat tracepoint report the approximation, stay
>>    precise, or report both?
>
> No strong optinion on that. Do we know who uses these and for which purpose?

It was added for Android, to track the RSS of each process and detect
memory spikes: commit b3d1411b6726 ("mm: emit tracepoint when RSS
changes"). As far as I know, Perfetto reads it. Martin, would the
approximation be fine for your use?

>> 5. Should the tuning tools be merged, built only with the benchmark
>>    config, or dropped later?  Who could run them on non-AMD, 4-socket,
>>    arm64 or POWER machines?
>
> Is there some way to have it be an auto-tuning instead of requiring this tooling?

The defaults are already automatic: the tree shape only depends on
nr_cpu_ids. A finer heuristic could use the cache topology. I would
rather get results from other hardware first, to see whether the
defaults need it.

>> 6. Is tools/testing/hpcc the right home for the topology test?
> Good question, I'd assume so.

OK, it stays there.

Thanks,

Mathieu