From: Bernd Schubert <bschubert@ddn.com>
To: Peter Zijlstra <peterz@infradead.org>
Cc: Ingo Molnar <mingo@redhat.com>,
Miklos Szeredi <miklos@szeredi.hu>,
Dharmendra Singh <dsingh@ddn.com>,
"linux-fsdevel@vger.kernel.org" <linux-fsdevel@vger.kernel.org>,
Amir Goldstein <amir73il@gmail.com>,
"linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
K Prateek Nayak <kprateek.nayak@amd.com>,
Hillf Danton <hdanton@sina.com>, Andrei Vagin <avagin@gmail.com>
Subject: Re: fuse uring / wake_up on the same core
Date: Wed, 26 Apr 2023 22:40:32 +0000 [thread overview]
Message-ID: <6213ca93-a14c-7900-f6dc-4b8bd8ae2e74@ddn.com> (raw)
In-Reply-To: <20230327102845.GB7701@hirez.programming.kicks-ass.net>
On 3/27/23 12:28, Peter Zijlstra wrote:
> On Fri, Mar 24, 2023 at 07:50:12PM +0000, Bernd Schubert wrote:
>
>> With the fuse-uring patches that part is basically solved - the waitq
>> that that thread is about is not used anymore. But as per above,
>> remaining is the waitq of the incoming workq (not mentioned in the
>> thread above). As I wrote, I have tried
>> __wake_up_sync((x), TASK_NORMAL), but it does not make a difference for
>> me - similar to Miklos' testing before. I have also tried struct
>> completion / swait - does not make a difference either.
>> I can see task_struct has wake_cpu, but there doesn't seem to be a good
>> interface to set it.
>>
>> Any ideas?
>
> Does the stuff from:
>
> https://lkml.kernel.org/r/20230308073201.3102738-1-avagin@google.com
Thanks Peter, I have already replied in that thread - using
__wake_up_on_current_cpu() helps to avoid cpu migrations. Well, some
update since my last mail in that thread (a few hours ago), more logging
reveals that I still see a few cpu switches, but nothing compared to
what I had before.
My issue is now that these patches are not enough and contrary to
previous testing, forcefully disabling cpu migration using
migrate_disable() before wait_event_* in fuse's request_wait_answer()
and enabling it after does not help either - my process to create files
(bonnie++) somewhere migrates to another cpu at a later time.
The only workaround I currently have is to set the ring thread
processing vfs/fuse events in userspace to SCHED_IDLE. In combination
with WF_CURRENT_CPU performance then goes from ~2200 to ~9000 file
creates/s for a single thread in the latest branch (should be scalable).
Which is very close to binding the bonnie++ process to a single core
(~9400 creates/s).
Is there something available to mark ring threads as IO processing and
that there is no need to migrate away the submitting thread from IO
threads?
* application sends request -> forwards to ring and wake ring -> wait
* ring wakes up (core bound) -> process request -> sends completion ->
wake up application -> wait for next request
* application wakes up with request result
==> I don't understand why the application is moved to another process
at all, after the wake issue is eliminated.
I also only see SCHED_IDLE only as workaround, as it would likely have
side effects if there is anything else running on the system and would
consume cpus while another process is doing IO.
Is there a way to trace where and why a process is migrated away?
Thanks,
Bernd
next prev parent reply other threads:[~2023-04-26 22:43 UTC|newest]
Thread overview: 11+ messages / expand[flat|nested] mbox.gz Atom feed top
2023-03-24 19:50 Bernd Schubert
2023-03-24 22:44 ` Bernd Schubert
[not found] ` <20230325002815.1703-1-hdanton@sina.com>
2023-03-25 7:08 ` K Prateek Nayak
2023-03-27 14:35 ` Bernd Schubert
2023-03-27 10:28 ` Peter Zijlstra
2023-04-26 22:40 ` Bernd Schubert [this message]
[not found] ` <20230427122417.2452-1-hdanton@sina.com>
2023-04-27 13:35 ` Bernd Schubert
[not found] ` <20230428014443.2539-1-hdanton@sina.com>
2023-04-28 21:54 ` Bernd Schubert
2023-05-01 21:44 ` Bernd Schubert
[not found] ` <20230502003335.3253-1-hdanton@sina.com>
2023-05-03 17:04 ` Bernd Schubert
[not found] ` <20230504021656.3823-1-hdanton@sina.com>
2023-05-05 13:10 ` Bernd Schubert
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=6213ca93-a14c-7900-f6dc-4b8bd8ae2e74@ddn.com \
--to=bschubert@ddn.com \
--cc=amir73il@gmail.com \
--cc=avagin@gmail.com \
--cc=dsingh@ddn.com \
--cc=hdanton@sina.com \
--cc=kprateek.nayak@amd.com \
--cc=linux-fsdevel@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=miklos@szeredi.hu \
--cc=mingo@redhat.com \
--cc=peterz@infradead.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®