From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-dy1-f202.google.com (mail-dy1-f202.google.com [74.125.82.202]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9D4EF31B838 for ; Sun, 31 May 2026 05:27:57 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=74.125.82.202 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780205279; cv=none; b=BrygdIRZzCYxbLYEn8ITAKuBCUZzpVr2qFkMmgWiqhOnGEn7ujKB7emR6iuaD2kRNgPAyxmO+WA/jITeLaL7Rw/UFhvtG11nGg4OFjZqgDBnxBG4oJ3MelsFYZD/OoBI2A+46cKqQ1V0dRCiI0cyf1nXYKT3OFax5gNL6mfTDLQ= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780205279; c=relaxed/simple; bh=RYOAQ7i8l+HnhrtSX3dXUYJUZe+fXy/boK0G24VBIis=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=IQFxnnvUyq/XIhIMIr+j7WL9mwJXSVIwHmkrObtTwUIxRxJh7Xcm1+gS/lqkdv6yBUIaKmEah4xILFoWIxYQfgqG97k7AHlTzKaJ/1gJhbH13fBDkgnWqW+NG7fDl5w/VU1o1YGZk+zb5k5R1C8dQusznPBEDfGJpaesA5ZPr8Y= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--irogers.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=GnaNY/bG; arc=none smtp.client-ip=74.125.82.202 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--irogers.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="GnaNY/bG" Received: by mail-dy1-f202.google.com with SMTP id 5a478bee46e88-304ee7d1368so1757021eec.0 for ; Sat, 30 May 2026 22:27:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1780205277; x=1780810077; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=/PJvMIeavvXr05kLBSGmt9R6Igq4yiHfNpH4eLkjMrw=; b=GnaNY/bGgOFhNAbu7YAGWaa+4cwHJYA/RB6t3jujuG5DDOchX9ur1Yj0AoVNA+BErw yw/d+jIUonJ1hOt/KAb0h+30xlEHb+ocPrwJVao/YEKwnZYEqOLouSFdAQVOBmYN4U4E ZlWfkXJ3/5SF+YoPobGYQCKEx+ebnMlPF8YEftZu/oqzkPDLgSIsahS71oCYTYuRKLP3 IRu22Xzsf9XHwmJwQqcOHWU/thZ3mv7baFZqDfhBgZDmkDF91ncDsm2Z48E0+zJrsgpt vuOFrOlMHbz74LXwWDfWIR2FFRJTAB9t3zOk5xTZ8dNdTd2gcMQf3BXZQr+koOLJtpQZ GsTQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1780205277; x=1780810077; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=/PJvMIeavvXr05kLBSGmt9R6Igq4yiHfNpH4eLkjMrw=; b=hfnRJxU6l5Qi8Jm+85ZzAWHlCNkjg8ck940y/bRES51ksXTjjQXCGpJnQjF9hcoZBh BAE4qc392IXNYeA/6j27UBXsb6IBJVMCaNADN/BFGSqsqLuMxnIeDnYXj+bsAf0aSL+s sb02y3/NK6AOm+D3fcpKxrX/7smz4HYEvGHsO9k/q2NA4mLcbbU/K+Szg/VZPPVbiZ2e Vh4TGR741duG1HNEcSXj1p0ouprWlVlTY4E3B4LhfMDCCE/Nvz5Vo028hEYrVUF6kDmp ypIzj7dof0c5bufXOw0ueDa8vVcaHwDsrmcCxNRneCZcrve/8F+RyllLOWZWdXCV108O tl1g== X-Forwarded-Encrypted: i=1; AFNElJ+seY2DZgsOmMFZriyfRcQ6JdovbwHtF/rUh8Px6Of50h0BpXSvfkaO3dQjZ5EGCk4hNsO9x7sRJssIxU0=@vger.kernel.org X-Gm-Message-State: AOJu0YzGBPzdgk++oYG1FBma1OBqxhZKq2nxJ9RRPmsMrH1tIWmNv32G OTnyEvDwRrkxm24L8HjppwGI8cLTUqu7f47ZHb9gSz9dENwIj9iODKvVVT2ecGZvolRV7EI5lGx XqAK3bWgVuA== X-Received: from dlbpt6.prod.google.com ([2002:a05:7022:e806:b0:135:1edf:c813]) (user=irogers job=prod-delivery.src-stubby-dispatcher) by 2002:a05:7022:61b:b0:128:d20a:2f40 with SMTP id a92af1059eb24-137d4026c04mr2510126c88.8.1780205276529; Sat, 30 May 2026 22:27:56 -0700 (PDT) Date: Sat, 30 May 2026 22:27:32 -0700 In-Reply-To: <20260531052740.796087-1-irogers@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20260513230450.529380-1-irogers@google.com> <20260531052740.796087-1-irogers@google.com> X-Mailer: git-send-email 2.54.0.823.g6e5bcc1fc9-goog Message-ID: <20260531052740.796087-7-irogers@google.com> Subject: [PATCH v2 06/14] perf test: Refactor parallel poll loop to drain all pipes simultaneously From: Ian Rogers To: irogers@google.com, acme@kernel.org, adrian.hunter@intel.com, namhyung@kernel.org Cc: alexander.shishkin@linux.intel.com, james.clark@linaro.org, jolsa@kernel.org, linux-kernel@vger.kernel.org, linux-perf-users@vger.kernel.org, mingo@redhat.com, peterz@infradead.org Content-Type: text/plain; charset="UTF-8" When running tests in parallel with verbose output (-v), child processes write to pipes. If a test produces significant output (e.g. Granite Rapids metric parsing printing hundreds of lines), it fills the 64KB pipe buffer and blocks. Previously, the parent harness (finish_test) only polled the pipe of the "current" test waiting to be printed. Other children blocked indefinitely until the parent reached them, severely sequentializing execution. Address this by implementing finish_tests_parallel() to poll and drain output pipes from all running children simultaneously into per-child buffers. Reaping occurs out of order as children finish, while final result printing remains strictly in order. This drops parallel verbose execution time for the PMU events suite from ~35 seconds down to ~5.9 seconds. Assisted-by: Gemini-CLI:Google Gemini 3 Signed-off-by: Ian Rogers --- tools/lib/subcmd/run-command.c | 14 ++- tools/perf/tests/builtin-test.c | 194 +++++++++++++++++++++++++++++++- 2 files changed, 204 insertions(+), 4 deletions(-) diff --git a/tools/lib/subcmd/run-command.c b/tools/lib/subcmd/run-command.c index b7510f83209a..4f8ec001b948 100644 --- a/tools/lib/subcmd/run-command.c +++ b/tools/lib/subcmd/run-command.c @@ -241,8 +241,18 @@ int check_if_command_finished(struct child_process *cmd) sprintf(filename, "/proc/%u/status", cmd->pid); status_file = fopen(filename, "r"); if (status_file == NULL) { - /* Open failed assume finish_command was called. */ - return true; + /* + * fopen() can fail with ENOENT if the process has been reaped. + * It can also fail with EMFILE/ENFILE if RLIMIT_NOFILE is reached, + * or with EINTR/ENOMEM. Use kill(pid, 0) as a robust fallback + * to distinguish between active processes and dead ones without + * consuming file descriptors. + */ + if (errno == ENOENT) + return 1; + if (kill(cmd->pid, 0) == -1 && errno == ESRCH) + return 1; + return 0; } while (fgets(status_line, sizeof(status_line), status_file) != NULL) { char *p; diff --git a/tools/perf/tests/builtin-test.c b/tools/perf/tests/builtin-test.c index 2ccb52a776cc..87c6a67beb10 100644 --- a/tools/perf/tests/builtin-test.c +++ b/tools/perf/tests/builtin-test.c @@ -302,6 +302,9 @@ struct child_test { struct test_suite *test; int suite_num; int test_case_num; + struct strbuf err_output; + int result; + bool done; }; static jmp_buf run_test_jmp_buf; @@ -509,6 +512,192 @@ static void finish_test(struct child_test **child_tests, int running_test, int c zfree(&child_tests[running_test]); } +static void drain_child_process_err(struct child_test *child) +{ + char buf[512]; + ssize_t len; + + while ((len = read(child->process.err, buf, sizeof(buf) - 1)) > 0) { + buf[len] = '\0'; + strbuf_addstr(&child->err_output, buf); + } +} + +static int finish_tests_parallel(struct child_test **child_tests, size_t num_tests, int width) +{ + size_t next_to_print = 0; + struct pollfd *pfds; + size_t *pfd_indices; + size_t num_pfds = 0; + int last_running = -1; + size_t i; + int last_suite_printed = -1; + + pfds = calloc(num_tests, sizeof(*pfds)); + pfd_indices = calloc(num_tests, sizeof(*pfd_indices)); + if (!pfds || !pfd_indices) { + free(pfds); + free(pfd_indices); + return -ENOMEM; + } + + for (i = 0; i < num_tests; i++) { + struct child_test *child = child_tests[i]; + + if (!child) + continue; + strbuf_init(&child->err_output, 0); + if (child->process.err > 0) + fcntl(child->process.err, F_SETFL, O_NONBLOCK); + } + + while (next_to_print < num_tests) { + size_t running_count = 0; + size_t p; + + while (next_to_print < num_tests && + (!child_tests[next_to_print] || child_tests[next_to_print]->done)) + next_to_print++; + + if (next_to_print >= num_tests) + break; + + num_pfds = 0; + + for (i = next_to_print; i < num_tests; i++) { + struct child_test *child = child_tests[i]; + + if (!child || child->done) + continue; + + if (!check_if_command_finished(&child->process)) + running_count++; + + if (child->process.err > 0) { + pfds[num_pfds].fd = child->process.err; + pfds[num_pfds].events = POLLIN | POLLERR | POLLHUP | POLLNVAL; + pfd_indices[num_pfds] = i; + num_pfds++; + } + } + + if (perf_use_color_default && running_count != (size_t)last_running) { + struct child_test *next_child = child_tests[next_to_print]; + + if (last_running != -1) + fprintf(debug_file(), PERF_COLOR_DELETE_LINE); + + if (next_child) { + if (test_suite__num_test_cases(next_child->test) > 1 && + last_suite_printed != next_child->suite_num) { + pr_info("%3d: %-*s:\n", next_child->suite_num + 1, width, + test_description(next_child->test, -1)); + last_suite_printed = next_child->suite_num; + } + print_test_result(next_child->test, next_child->suite_num, + next_child->test_case_num, TEST_RUNNING, width, + running_count); + } + last_running = running_count; + } + + if (num_pfds == 0) { + if (running_count > 0) + usleep(10 * 1000); + } else { + int pret = poll(pfds, num_pfds, 100); + + if (pret > 0) { + for (p = 0; p < num_pfds; p++) { + if (pfds[p].revents) { + size_t idx = pfd_indices[p]; + struct child_test *child = child_tests[idx]; + + drain_child_process_err(child); + /* + * If the child closed its end of the pipe (EOF) or encountered + * an error, close the file descriptor immediately and set it + * to -1. This removes it from the pfds array for subsequent + * iterations, preventing a tight CPU busy-loop while waiting + * for the process itself to exit. + */ + if (pfds[p].revents & (POLLHUP | POLLERR | POLLNVAL)) { + close(child->process.err); + child->process.err = -1; + } + } + } + } + } + + for (i = next_to_print; i < num_tests; i++) { + struct child_test *child = child_tests[i]; + + if (!child || child->done) + continue; + + if (check_if_command_finished(&child->process)) { + if (child->process.err > 0) { + drain_child_process_err(child); + close(child->process.err); + child->process.err = -1; + } + child->result = finish_command(&child->process); + child->done = true; + } + } + + while (next_to_print < num_tests) { + struct child_test *child = child_tests[next_to_print]; + + if (!child) { + next_to_print++; + continue; + } + if (!child->done) + break; + + if (perf_use_color_default && last_running != -1) { + fprintf(debug_file(), PERF_COLOR_DELETE_LINE); + last_running = -1; + } + + if (test_suite__num_test_cases(child->test) > 1 && + last_suite_printed != child->suite_num) { + pr_info("%3d: %-*s:\n", child->suite_num + 1, width, + test_description(child->test, -1)); + last_suite_printed = child->suite_num; + } + + if (verbose > 1) { + if (test_suite__num_test_cases(child->test) > 1) { + pr_info("%3d.%1d: %s:\n", child->suite_num + 1, + child->test_case_num + 1, + test_description(child->test, + child->test_case_num)); + } else { + pr_info("%3d: %s:\n", child->suite_num + 1, + test_description(child->test, -1)); + } + } + + if (verbose > 1 || (verbose == 1 && child->result == TEST_FAIL)) + fprintf(stderr, "%s", child->err_output.buf); + + print_test_result(child->test, child->suite_num, child->test_case_num, + child->result, width, 0); + strbuf_release(&child->err_output); + child_tests[next_to_print] = NULL; + zfree(&child); + next_to_print++; + } + } + + free(pfds); + free(pfd_indices); + return 0; +} + static int start_test(struct test_suite *test, int curr_suite, int curr_test_case, struct child_test **child, int width, int pass) { @@ -671,8 +860,9 @@ static int __cmd_test(struct test_suite **suites, int argc, const char *argv[], } if (!sequential) { /* Parallel mode starts tests but doesn't finish them. Do that now. */ - for (size_t x = 0; x < num_tests; x++) - finish_test(child_tests, x, num_tests, width); + err = finish_tests_parallel(child_tests, num_tests, width); + if (err) + goto err_out; } } err_out: -- 2.54.0.823.g6e5bcc1fc9-goog