[PATCH] perf jevents: Limit the number of JSON reading workers

[PATCH] perf jevents: Limit the number of JSON reading workers

[Sandipan Das]

The ProcessPoolExecutor is created without max_workers, so
concurrent.futures defaults to one worker per CPU and forks all of them
when the first task is submitted. Each worker costs the parent a few
file descriptors. For systems with high core-count processors like the
AMD EPYC 9996, the pool can easily exhaust RLIMIT_NOFILE as seen below.

  File ".../multiprocessing/popen_fork.py", line 65, in _launch
    child_r, parent_w = os.pipe()
  OSError: [Errno 24] Too many open files

The error is reported, but jevents.py does not exit. The workers that
did start stay blocked waiting for work and are never joined, so the
build eventually hangs. Other targets keep building in the meantime,
leaving the traceback several lines above the point where the build
stalls.

Limit the pool instead of sizing it against the descriptor limit, which
would mean relying on how many descriptors concurrent.futures needs for
each worker. At least for recent AMD EPYC processors, reading the JSON
files does not need many workers to be fast and things typically stop
scaling beyond 32 workers. Hence, a small upper bound keeps the speedup
and stays well inside any usable limit.

Fixes: eaab2eb09dc2 ("perf pmu-events: Parallelize JSON and metric pre-computation in jevents.py")
Signed-off-by: Sandipan Das <sandipan.das@amd.com>
---
 tools/perf/pmu-events/jevents.py | 11 ++++++++++-
 1 file changed, 10 insertions(+), 1 deletion(-)

diff --git a/tools/perf/pmu-events/jevents.py b/tools/perf/pmu-events/jevents.py
index 860027bb71b2..d81fe6757915 100755
--- a/tools/perf/pmu-events/jevents.py
+++ b/tools/perf/pmu-events/jevents.py
@@ -1450,6 +1450,13 @@ const char *describe_metricgroup(const char *group)
 }
 """)
 
+# Upper bound on the number of workers reading JSON files in parallel.
+_max_parallel_workers = 32
+
+def _parallel_worker_count(num_tasks: int) -> int:
+  return max(1, min(getattr(os, 'process_cpu_count', os.cpu_count)() or 1,
+                    _max_parallel_workers, num_tasks))
+
 def _parallel_read_json_events(task: Tuple[str, str]) -> Tuple[str, str, Sequence[JsonEvent]]:
   path, topic = task
   return path, topic, _read_json_events_impl(path, topic)
@@ -1549,7 +1556,9 @@ struct pmu_table_entry {
     preprocess_arch_std_files(arch_path)
     ftw(arch_path, [], collect_json)
 
-  with concurrent.futures.ProcessPoolExecutor(initializer=_init_worker, initargs=(_arch_std_events,)) as executor:
+  with concurrent.futures.ProcessPoolExecutor(max_workers=_parallel_worker_count(len(tasks)),
+                                              initializer=_init_worker,
+                                              initargs=(_arch_std_events,)) as executor:
     for path, topic, events in executor.map(_parallel_read_json_events, tasks):
       _json_cache[(path, topic)] = events
 
-- 
2.53.0