From: Peter Zijlstra <peterz@infradead.org>
To: Jens Axboe <axboe@kernel.dk>
Cc: Valentin Schneider <valentin.schneider@arm.com>,
Thomas Gleixner <tglx@linutronix.de>,
Lai Jiangshan <jiangshanlai@gmail.com>,
linux-kernel@vger.kernel.org, Qian Cai <cai@redhat.com>,
Vincent Donnefort <vincent.donnefort@arm.com>,
Dexuan Cui <decui@microsoft.com>,
Lai Jiangshan <laijs@linux.alibaba.com>,
Paul McKenney <paulmck@kernel.org>,
Vincent Guittot <vincent.guittot@linaro.org>,
Steven Rostedt <rostedt@goodmis.org>
Subject: Re: [PATCH -tip V3 0/8] workqueue: break affinity initiatively
Date: Tue, 12 Jan 2021 16:51:06 +0100 [thread overview]
Message-ID: <X/3FaoV0K/U7ZuY4@hirez.programming.kicks-ass.net> (raw)
In-Reply-To: <f5b0f650-5dc1-c67d-f548-340e7ca54d41@kernel.dk>
On Tue, Jan 12, 2021 at 07:57:26AM -0700, Jens Axboe wrote:
> On 1/11/21 12:21 PM, Valentin Schneider wrote:
> > On 11/01/21 18:16, Peter Zijlstra wrote:
> >> Sadly it appears like io_uring() uses kthread_create_on_cpu() without
> >> then having any hotplug crud on, so that needs additinoal frobbing.
> >>
> >
> > I noticed that as well sometime ago, and I believed then (still do) this
> > usage is broken. I don't think usage of kthread_create_on_cpu() outside
> > of smpboot makes sense, because without any hotplug step to park the
> > thread, its affinity can end up being reset after its dedicated CPU gets
> > offlined.
> >
> > I'm clueless about io_uring, but if it *actually* has a good reason to
> > use some pcpu kthreads (it seems it doesn't have to be on all CPUs?),
> > then it needs to register some hotplug step to park them / do something
> > sensible on hotplug.
>
> For io_uring, it's purely used by the SQPOLL mode, which sets aside a
> kernel thread for submissions so the application doesn't have to do
> anything but write new SQE entries to submit. The thread then notices
> these, and submits them. There's an option to affinitize that thread
> to a single CPU, which often makes sense for setups like that. Think
> of it like a strong hint.
OK, that matches what I could make of it earlier today.
> Things aren't going to break if this CPU goes away and we end up being
> affinitized to some other CPU, though it is suboptimal. So I guess we
> might need some notifiers to ensure that we reset the CPU back again
> if it's gone offline+online again? I can take a look at that.
Indeed, that would make sense.
Thanks!
prev parent reply other threads:[~2021-01-12 15:53 UTC|newest]
Thread overview: 43+ messages / expand[flat|nested] mbox.gz Atom feed top
2020-12-26 2:51 Lai Jiangshan
2020-12-26 2:51 ` [PATCH -tip V3 1/8] workqueue: use cpu_possible_mask instead of cpu_active_mask to break affinity Lai Jiangshan
2020-12-26 2:51 ` [PATCH -tip V3 2/8] workqueue: Manually break affinity on pool detachment Lai Jiangshan
2020-12-26 2:51 ` [PATCH -tip V3 3/8] workqueue: introduce wq_online_cpumask Lai Jiangshan
2021-01-04 13:56 ` Peter Zijlstra
2021-01-05 2:41 ` Lai Jiangshan
2021-01-05 2:53 ` Lai Jiangshan
2021-01-05 8:23 ` Lai Jiangshan
2021-01-05 13:17 ` Peter Zijlstra
2021-01-05 14:37 ` Lai Jiangshan
2021-01-05 14:40 ` Lai Jiangshan
2021-01-05 16:24 ` Peter Zijlstra
2020-12-26 2:51 ` [PATCH -tip V3 4/8] workqueue: use wq_online_cpumask in restore_unbound_workers_cpumask() Lai Jiangshan
2020-12-26 2:51 ` [PATCH -tip V3 5/8] workqueue: Manually break affinity on hotplug for unbound pool Lai Jiangshan
[not found] ` <20201226101631.5448-1-hdanton@sina.com>
2020-12-27 14:04 ` Lai Jiangshan
2020-12-26 2:51 ` [PATCH -tip V3 6/8] workqueue: reorganize workqueue_online_cpu() Lai Jiangshan
2020-12-26 2:51 ` [PATCH -tip V3 7/8] workqueue: reorganize workqueue_offline_cpu() unbind_workers() Lai Jiangshan
2020-12-26 2:51 ` [PATCH -tip V3 8/8] workqueue: Fix affinity of kworkers when attaching into pool Lai Jiangshan
[not found] ` <20201229100639.2086-1-hdanton@sina.com>
2020-12-29 10:13 ` Lai Jiangshan
2021-01-08 11:46 ` [PATCH -tip V3 0/8] workqueue: break affinity initiatively Peter Zijlstra
2021-01-11 10:07 ` Thomas Gleixner
2021-01-11 11:01 ` Peter Zijlstra
2021-01-11 15:00 ` Paul E. McKenney
2021-01-11 17:16 ` Peter Zijlstra
2021-01-11 18:09 ` Paul E. McKenney
2021-01-11 21:50 ` Paul E. McKenney
2021-01-12 17:14 ` Paul E. McKenney
2021-01-12 23:53 ` Paul E. McKenney
2021-01-15 9:11 ` Peter Zijlstra
2021-01-15 13:04 ` Peter Zijlstra
2021-01-16 6:00 ` Lai Jiangshan
2021-01-11 19:21 ` Valentin Schneider
2021-01-11 20:23 ` Peter Zijlstra
2021-01-11 22:47 ` Valentin Schneider
2021-01-12 4:33 ` Lai Jiangshan
2021-01-12 14:53 ` Peter Zijlstra
2021-01-12 15:38 ` Lai Jiangshan
2021-01-13 11:10 ` Peter Zijlstra
2021-01-13 12:00 ` Lai Jiangshan
2021-01-13 12:57 ` Lai Jiangshan
2021-01-12 17:52 ` Valentin Schneider
2021-01-12 14:57 ` Jens Axboe
2021-01-12 15:51 ` Peter Zijlstra [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=X/3FaoV0K/U7ZuY4@hirez.programming.kicks-ass.net \
--to=peterz@infradead.org \
--cc=axboe@kernel.dk \
--cc=cai@redhat.com \
--cc=decui@microsoft.com \
--cc=jiangshanlai@gmail.com \
--cc=laijs@linux.alibaba.com \
--cc=linux-kernel@vger.kernel.org \
--cc=paulmck@kernel.org \
--cc=rostedt@goodmis.org \
--cc=tglx@linutronix.de \
--cc=valentin.schneider@arm.com \
--cc=vincent.donnefort@arm.com \
--cc=vincent.guittot@linaro.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®