mirror of https://lore.kernel.org/lkml/
 help / color / mirror / Atom feed
From: Bernd Schubert <bschubert@ddn.com>
To: Peter Zijlstra <peterz@infradead.org>
Cc: Ingo Molnar <mingo@redhat.com>,
	Miklos Szeredi <miklos@szeredi.hu>,
	Dharmendra Singh <dsingh@ddn.com>,
	"linux-fsdevel@vger.kernel.org" <linux-fsdevel@vger.kernel.org>,
	Amir Goldstein <amir73il@gmail.com>,
	"linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
	K Prateek Nayak <kprateek.nayak@amd.com>,
	Hillf Danton <hdanton@sina.com>, Andrei Vagin <avagin@gmail.com>
Subject: Re: fuse uring / wake_up on the same core
Date: Wed, 26 Apr 2023 22:40:32 +0000	[thread overview]
Message-ID: <6213ca93-a14c-7900-f6dc-4b8bd8ae2e74@ddn.com> (raw)
In-Reply-To: <20230327102845.GB7701@hirez.programming.kicks-ass.net>

On 3/27/23 12:28, Peter Zijlstra wrote:
> On Fri, Mar 24, 2023 at 07:50:12PM +0000, Bernd Schubert wrote:
> 
>> With the fuse-uring patches that part is basically solved - the waitq
>> that that thread is about is not used anymore. But as per above,
>> remaining is the waitq of the incoming workq (not mentioned in the
>> thread above). As I wrote, I have tried
>> __wake_up_sync((x), TASK_NORMAL), but it does not make a difference for
>> me - similar to Miklos' testing before. I have also tried struct
>> completion / swait - does not make a difference either.
>> I can see task_struct has wake_cpu, but there doesn't seem to be a good
>> interface to set it.
>>
>> Any ideas?
> 
> Does the stuff from:
> 
>    https://lkml.kernel.org/r/20230308073201.3102738-1-avagin@google.com

Thanks Peter, I have already replied in that thread - using 
__wake_up_on_current_cpu() helps to avoid cpu migrations. Well, some 
update since my last mail in that thread (a few hours ago), more logging 
reveals that I still see a few cpu switches, but nothing compared to 
what I had before.
My issue is now that these patches are not enough and contrary to 
previous testing, forcefully disabling cpu migration using 
migrate_disable() before wait_event_* in fuse's request_wait_answer()
and enabling it after does not help either - my process to create files
(bonnie++) somewhere migrates to another cpu at a later time.
The only workaround I currently have is to set the ring thread 
processing vfs/fuse events in userspace to SCHED_IDLE. In combination 
with WF_CURRENT_CPU performance then goes from ~2200 to ~9000 file 
creates/s for a single thread in the latest branch (should be scalable). 
Which is very close to binding the bonnie++ process to a single core 
(~9400 creates/s).

Is there something available to mark ring threads as IO processing and 
that there is no need to migrate away the submitting thread from IO 
threads?

* application sends request -> forwards to ring and wake ring  -> wait
* ring wakes up (core bound) -> process request -> sends completion -> 
wake up application -> wait for next request
* application wakes up with request result

==> I don't understand why the application is moved to another process 
at all, after the wake issue is eliminated.

I also only see SCHED_IDLE only as workaround, as it would likely have 
side effects if there is anything else running on the system and would 
consume cpus while another process is doing IO.
Is there a way to trace where and why a process is migrated away?


Thanks,
Bernd


  reply	other threads:[~2023-04-26 22:43 UTC|newest]

Thread overview: 11+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2023-03-24 19:50 Bernd Schubert
2023-03-24 22:44 ` Bernd Schubert
     [not found] ` <20230325002815.1703-1-hdanton@sina.com>
2023-03-25  7:08   ` K Prateek Nayak
2023-03-27 14:35     ` Bernd Schubert
2023-03-27 10:28 ` Peter Zijlstra
2023-04-26 22:40   ` Bernd Schubert [this message]
     [not found]   ` <20230427122417.2452-1-hdanton@sina.com>
2023-04-27 13:35     ` Bernd Schubert
     [not found]     ` <20230428014443.2539-1-hdanton@sina.com>
2023-04-28 21:54       ` Bernd Schubert
2023-05-01 21:44         ` Bernd Schubert
     [not found]         ` <20230502003335.3253-1-hdanton@sina.com>
2023-05-03 17:04           ` Bernd Schubert
     [not found]           ` <20230504021656.3823-1-hdanton@sina.com>
2023-05-05 13:10             ` Bernd Schubert

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=6213ca93-a14c-7900-f6dc-4b8bd8ae2e74@ddn.com \
    --to=bschubert@ddn.com \
    --cc=amir73il@gmail.com \
    --cc=avagin@gmail.com \
    --cc=dsingh@ddn.com \
    --cc=hdanton@sina.com \
    --cc=kprateek.nayak@amd.com \
    --cc=linux-fsdevel@vger.kernel.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=miklos@szeredi.hu \
    --cc=mingo@redhat.com \
    --cc=peterz@infradead.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox

all inboxes | Powered by JetHome®