From: Tejun Heo <tj@kernel.org>
To: K Prateek Nayak <kprateek.nayak@amd.com>
Cc: Sandeep Dhavale <dhavale@google.com>,
jiangshanlai@gmail.com, torvalds@linux-foundation.org,
peterz@infradead.org, linux-kernel@vger.kernel.org,
kernel-team@meta.com, joshdon@google.com, brho@google.com,
briannorris@chromium.org, nhuck@google.com, agk@redhat.com,
snitzer@kernel.org, void@manifault.com, kernel-team@android.com
Subject: Re: [PATCH 14/24] workqueue: Generalize unbound CPU pods
Date: Mon, 7 Aug 2023 15:08:56 -1000 [thread overview]
Message-ID: <ZNGVqL8nXOO1rXlk@slm.duckdns.org> (raw)
In-Reply-To: <ZMhJNZLWQHInDf4e@slm.duckdns.org>
Hello,
On Mon, Jul 31, 2023 at 01:52:21PM -1000, Tejun Heo wrote:
> On Tue, Jul 11, 2023 at 08:32:27AM +0530, K Prateek Nayak wrote:
> > > Yeah, that's a bit surprising given that in terms of affinity behavior
> > > "numa" should be identical to base. The only meaningful differences that I
> > > can think of is when the work item is assigned to its worker and maybe how
> > > pwq max_active limit is applied. Hmm... can you monitor the number of
> > > kworker kthreads while running the benchmark? No need to do the whole
> > > matrix, just comparing base against numa should be enough.
> >
> > Sure. I'll get back to you with the data soon.
>
> Any updates? I'd like to proceed with the patchset as it helps resolving
> problems others are reporting. I can try to reproduce the results too if you
> can share more details on how they're run.
Prateek sent me how he tested along with workqueue traces. I tried to
reproduce on an AMD zen2 machine and here are the findings:
* The test has a high run-to-run variance. Even with cpufreq boost turned
off, the numbers reported every second within each run is relatively
stable but adjacent runs can report signficantly variable numbers. Maybe
initial thread placement has lingering effects?
On ryzen 3900x, 15 runs of `./tbench -c ./client.txt -t 60 32 127.0.0.1`:
Before: AVG=9066.43 STDEV=42.65
After : AVG=9076.11 STDEV=60.50
Given the stdev, I don't think this is indicating any meaningful
difference.
* I looked at what were consuming CPUs during the benchmark runs and also
Prateek's workqueue traces. None of the operations that tbench is doing
directly involves workqueue. I couldn't find a mechanism how workqueue
differences would cause any meaningful performance differences.
At least for tbench results, I couldn't find any signal.
Thanks.
--
tejun
next prev parent reply other threads:[~2023-08-08 1:09 UTC|newest]
Thread overview: 73+ messages / expand[flat|nested] mbox.gz Atom feed top
2023-05-19 0:16 [PATCHSET v1 wq/for-6.5] workqueue: Improve unbound workqueue execution locality Tejun Heo
2023-05-19 0:16 ` [PATCH 01/24] workqueue: Drop the special locking rule for worker->flags and worker_pool->flags Tejun Heo
2023-05-19 0:16 ` [PATCH 02/24] workqueue: Cleanups around process_scheduled_works() Tejun Heo
2023-05-19 0:16 ` [PATCH 03/24] workqueue: Not all work insertion needs to wake up a worker Tejun Heo
2023-05-23 9:54 ` Lai Jiangshan
2023-05-23 21:37 ` Tejun Heo
2023-08-08 1:15 ` [PATCH v2 " Tejun Heo
2023-05-19 0:16 ` [PATCH 04/24] workqueue: Rename wq->cpu_pwqs to wq->cpu_pwq Tejun Heo
2023-05-19 0:16 ` [PATCH 05/24] workqueue: Relocate worker and work management functions Tejun Heo
2023-05-19 0:16 ` [PATCH 06/24] workqueue: Remove module param disable_numa and sysfs knobs pool_ids and numa Tejun Heo
2023-05-19 0:16 ` [PATCH 07/24] workqueue: Use a kthread_worker to release pool_workqueues Tejun Heo
2023-05-19 0:16 ` [PATCH 08/24] workqueue: Make per-cpu pool_workqueues allocated and released like unbound ones Tejun Heo
2023-05-19 0:16 ` [PATCH 09/24] workqueue: Make unbound workqueues to use per-cpu pool_workqueues Tejun Heo
2023-05-22 6:41 ` Leon Romanovsky
2023-05-22 12:27 ` Dennis Dalessandro
2023-05-19 0:16 ` [PATCH 10/24] workqueue: Rename workqueue_attrs->no_numa to ->ordered Tejun Heo
2023-05-19 0:16 ` [PATCH 11/24] workqueue: Rename NUMA related names to use pod instead Tejun Heo
2023-05-19 0:16 ` [PATCH 12/24] workqueue: Move wq_pod_init() below workqueue_init() Tejun Heo
2023-05-19 0:16 ` [PATCH 13/24] workqueue: Initialize unbound CPU pods later in the boot Tejun Heo
2023-05-19 0:16 ` [PATCH 14/24] workqueue: Generalize unbound CPU pods Tejun Heo
2023-05-30 8:06 ` K Prateek Nayak
2023-06-07 1:50 ` Tejun Heo
2023-05-30 21:18 ` Sandeep Dhavale
2023-05-31 12:14 ` K Prateek Nayak
2023-06-07 22:13 ` Tejun Heo
2023-06-08 3:01 ` K Prateek Nayak
2023-06-08 22:50 ` Tejun Heo
2023-06-09 3:43 ` K Prateek Nayak
2023-06-14 18:49 ` Sandeep Dhavale
2023-06-21 20:14 ` Tejun Heo
2023-06-19 4:30 ` Swapnil Sapkal
2023-06-21 20:38 ` Tejun Heo
2023-07-05 7:04 ` K Prateek Nayak
2023-07-05 18:39 ` Tejun Heo
2023-07-11 3:02 ` K Prateek Nayak
2023-07-31 23:52 ` Tejun Heo
2023-08-08 1:08 ` Tejun Heo [this message]
2023-05-19 0:17 ` [PATCH 15/24] workqueue: Add tools/workqueue/wq_dump.py which prints out workqueue configuration Tejun Heo
2023-05-19 0:17 ` [PATCH 16/24] workqueue: Modularize wq_pod_type initialization Tejun Heo
2023-05-19 0:17 ` [PATCH 17/24] workqueue: Add multiple affinity scopes and interface to select them Tejun Heo
2023-05-19 0:17 ` [PATCH 18/24] workqueue: Factor out work to worker assignment and collision handling Tejun Heo
2023-05-19 0:17 ` [PATCH 19/24] workqueue: Factor out need_more_worker() check and worker wake-up Tejun Heo
2023-05-19 0:17 ` [PATCH 20/24] workqueue: Add workqueue_attrs->__pod_cpumask Tejun Heo
2023-05-19 0:17 ` [PATCH 21/24] workqueue: Implement non-strict affinity scope for unbound workqueues Tejun Heo
2023-05-19 0:17 ` [PATCH 22/24] workqueue: Add "Affinity Scopes and Performance" section to documentation Tejun Heo
2023-05-19 0:17 ` [PATCH 23/24] workqueue: Add pool_workqueue->cpu Tejun Heo
2023-05-19 0:17 ` [PATCH 24/24] workqueue: Implement localize-to-issuing-CPU for unbound workqueues Tejun Heo
2023-05-19 0:41 ` [PATCHSET v1 wq/for-6.5] workqueue: Improve unbound workqueue execution locality Linus Torvalds
2023-05-19 22:35 ` Tejun Heo
2023-05-19 23:03 ` Tejun Heo
2023-05-23 1:51 ` Linus Torvalds
2023-05-23 17:59 ` Linus Torvalds
2023-05-23 20:08 ` Rik van Riel
2023-05-23 21:36 ` Sandeep Dhavale
2023-05-23 11:18 ` Peter Zijlstra
2023-05-23 16:12 ` Vincent Guittot
2023-05-24 7:34 ` Peter Zijlstra
2023-05-24 13:15 ` Vincent Guittot
2023-06-05 4:46 ` Gautham R. Shenoy
2023-06-07 14:42 ` Libo Chen
2023-05-26 1:12 ` Tejun Heo
2023-05-30 11:32 ` Peter Zijlstra
2023-06-12 23:56 ` Brian Norris
2023-06-13 2:48 ` Tejun Heo
2023-06-13 9:26 ` Pin-yen Lin
2023-06-21 19:16 ` Tejun Heo
2023-06-21 19:31 ` Linus Torvalds
2023-06-29 9:49 ` Pin-yen Lin
2023-07-03 21:47 ` Tejun Heo
2023-08-08 1:22 ` Tejun Heo
2023-08-08 2:58 ` K Prateek Nayak
2023-08-08 7:59 ` Tejun Heo
2023-08-18 4:05 ` K Prateek Nayak
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=ZNGVqL8nXOO1rXlk@slm.duckdns.org \
--to=tj@kernel.org \
--cc=agk@redhat.com \
--cc=brho@google.com \
--cc=briannorris@chromium.org \
--cc=dhavale@google.com \
--cc=jiangshanlai@gmail.com \
--cc=joshdon@google.com \
--cc=kernel-team@android.com \
--cc=kernel-team@meta.com \
--cc=kprateek.nayak@amd.com \
--cc=linux-kernel@vger.kernel.org \
--cc=nhuck@google.com \
--cc=peterz@infradead.org \
--cc=snitzer@kernel.org \
--cc=torvalds@linux-foundation.org \
--cc=void@manifault.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®