From: Song Zhang <zhangsong34@huawei.com>
To: Vincent Guittot <vincent.guittot@linaro.org>
Cc: <mingo@redhat.com>, <peterz@infradead.org>,
<juri.lelli@redhat.com>, <mcgrof@kernel.org>,
<keescook@chromium.org>, <yzaikin@google.com>,
<dietmar.eggemann@arm.com>, <rostedt@goodmis.org>,
<bsegall@google.com>, <mgorman@suse.de>, <bristot@redhat.com>,
<vschneid@redhat.com>, <linux-kernel@vger.kernel.org>,
<linux-fsdevel@vger.kernel.org>
Subject: Re: [PATCH] sched/fair: Introduce priority load balance for CFS
Date: Wed, 16 Nov 2022 15:37:18 +0800 [thread overview]
Message-ID: <241e837b-056a-4fde-0673-205bd7400e82@huawei.com> (raw)
In-Reply-To: <CAKfTPtAMdQD9S-mbLszeu2pjB4YB2A+1OM5NUV_2xDzCTCc7Qw@mail.gmail.com>
On 2022/11/15 15:18, Vincent Guittot wrote:
> On Mon, 14 Nov 2022 at 17:42, Vincent Guittot
> <vincent.guittot@linaro.org> wrote:
>>
>> On Sat, 12 Nov 2022 at 03:51, Song Zhang <zhangsong34@huawei.com> wrote:
>>>
>>> Hi, Vincent
>>>
>>> On 2022/11/3 17:22, Vincent Guittot wrote:
>>>> On Thu, 3 Nov 2022 at 10:20, Song Zhang <zhangsong34@huawei.com> wrote:
>>>>>
>>>>>
>>>>>
>>>>> On 2022/11/3 16:33, Vincent Guittot wrote:
>>>>>> On Thu, 3 Nov 2022 at 04:01, Song Zhang <zhangsong34@huawei.com> wrote:
>>>>>>>
>>>>>>> Thanks for your reply!
>>>>>>>
>>>>>>> On 2022/11/3 2:01, Vincent Guittot wrote:
>>>>>>>> On Wed, 2 Nov 2022 at 04:54, Song Zhang <zhangsong34@huawei.com> wrote:
>>>>>>>>>
>>>>>>>>
>>>>>>>> This really looks like a v3 of
>>>>>>>> https://lore.kernel.org/all/20220810015636.3865248-1-zhangsong34@huawei.com/
>>>>>>>>
>>>>>>>> Please keep versioning.
>>>>>>>>
>>>>>>>>> Add a new sysctl interface:
>>>>>>>>> /proc/sys/kernel/sched_prio_load_balance_enabled
>>>>>>>>
>>>>>>>> We don't want to add more sysctl knobs for the scheduler, we even
>>>>>>>> removed some. Knob usually means that you want to fix your use case
>>>>>>>> but the solution doesn't make sense for all cases.
>>>>>>>>
>>>>>>>
>>>>>>> OK, I will remove this knobs later.
>>>>>>>
>>>>>>>>>
>>>>>>>>> 0: default behavior
>>>>>>>>> 1: enable priority load balance for CFS
>>>>>>>>>
>>>>>>>>> For co-location with idle and non-idle tasks, when CFS do load balance,
>>>>>>>>> it is reasonable to prefer migrating non-idle tasks and migrating idle
>>>>>>>>> tasks lastly. This will reduce the interference by SCHED_IDLE tasks
>>>>>>>>> as much as possible.
>>>>>>>>
>>>>>>>> I don't agree that it's always the best choice to migrate a non-idle task 1st.
>>>>>>>>
>>>>>>>> CPU0 has 1 non idle task and CPU1 has 1 non idle task and hundreds of
>>>>>>>> idle task and there is an imbalance between the 2 CPUS: migrating the
>>>>>>>> non idle task from CPU1 to CPU0 is not the best choice
>>>>>>>>
>>>>>>>
>>>>>>> If the non idle task on CPU1 is running or cache hot, it cannot be
>>>>>>> migrated and idle tasks can also be migrated from CPU1 to CPU0. So I
>>>>>>> think it does not matter.
>>>>>>
>>>>>> What I mean is that migrating non idle tasks first is not a universal
>>>>>> win and not always what we want.
>>>>>>
>>>>>
>>>>> But migrating online tasks first is mostly a trade-off that
>>>>> non-idle(Latency Sensitive) tasks can obtain more CPU time and minimize
>>>>> the interference caused by IDLE tasks. I think this makes sense in most
>>>>> cases, or you can point out what else I need to think about it ?
>>>>>
>>>>> Best regards.
>>>>>
>>>>>>>
>>>>>>>>>
>>>>>>>>> Testcase:
>>>>>>>>> - Spawn large number of idle(SCHED_IDLE) tasks occupy CPUs
>>>>>>>>
>>>>>>>> What do you mean by a large number ?
>>>>>>>>
>>>>>>>>> - Let non-idle tasks compete with idle tasks for CPU time.
>>>>>>>>>
>>>>>>>>> Using schbench to test non-idle tasks latency:
>>>>>>>>> $ ./schbench -m 1 -t 10 -r 30 -R 200
>>>>>>>>
>>>>>>>> How many CPUs do you have ?
>>>>>>>>
>>>>>>>
>>>>>>> OK, some details may not be mentioned.
>>>>>>> My virtual machine has 8 CPUs running with a schbench process and 5000
>>>>>>> idle tasks. The idle task is a while dead loop process below:
>>>>>>
>>>>>> How can you care about latency when you start 10 workers on 8 vCPUs
>>>>>> with 5000 non idle threads ?
>>>>>>
>>>>>
>>>>> No no no... spawn 5000 idle(SCHED_IDLE) processes not 5000 non-idle
>>>>> threads, and with 10 non-idle schbench workers on 8 vCPUs.
>>>>
>>>> yes spawn 5000 idle tasks but my point remains the same
>>>>
>>>
>>> I am so sorry that I have not received your reply for a long time, and I
>>> am still waiting for it anxiously. In fact, migrating non-idle tasks 1st
>>> works well in most scenarios, so it maybe possible to add a
>>> sched_feat(LB_PRIO) to enable or disable that. Finally, I really hope
>>> you can give me some better advice.
>>
>> I have seen that you posted a v4 5 days ago which is on my list to be reviewed.
>>
>> My concern here remains that selecting non idle task 1st is not always
>> the best choices as for example when you have 1 non idle task per cpu
>> and thousands of idle tasks moving around. Then regarding your use
>> case, the weight of the 5000 idle threads is around twice more than
>> the weight of your non idle bench: sum weight of idle threads is 15k
>> whereas the weight of your bench is around 6k IIUC how RPS run. This
>> also means that the idle threads will take a significant times of the
>> system: 5000 / 7000 ticks. I don't understand how you can care about
>> latency in such extreme case and I'm interested to get the real use
>> case where you can have such situation.
>>
>> All that to say that idle task remains cfs task with a small but not
>> null weight and we should not make them special other than by not
>> preempting at wakeup.
>
> Also, as mentioned for a previous version, a task with nice prio 19
> has a weight of 15 so if you replace the 5k idle threads with 1k cfs
> w/ nice prio 19 threads, you will face a similar problem. So you can't
> really care only on the idle property of a task
>
Well, my original idea was to consider interference between tasks of
different priorities when doing CFS load balancing to ensure that
non-idle tasks get more CPU scheduler time without changing the native
CFS load balancing policy.
Consider a simple scenario. Assume that CPU 0 has two non-idle tasks
whose weight is 1024 * 2 = 2048, also CPU 0 has 1000 idle tasks whose
weight is 1K x 15 = 15K. CPU 1 is idle. Therefore, IDLE load balance is
triggered. CPU 1 needs to pull a certain number of tasks from CPU 0. If
we do not considerate task priorities and interference between tasks,
more than 600 idle tasks on CPU 0 may be migrated to CPU 1. As a result,
two non-idle tasks still compete on CPU 0. However, CPU 1 is running
with all idle but not non-idle tasks.
Let's calculate the percentage of CPU time gained by non-idle tasks in a
scheduling period:
CPU 1: time_percent(non-idle tasks) = 0
CPU 0: time_percent(non-idle tasks) = 2048 * 2 / (2048 + 15000) = 24%
On the other hand, if we consider the interference between different
task priorities, we change the migration policy to firstly migrate an
non-idle task on CPU 0 to CPU 1. Migrating idle tasks on CPU 0 maybe
interfered with the non-idle task on CPU 1. So we decide to migrate idle
tasks on CPU 0 after non-idle tasks on CPU 1 are completed or exited.
Now the percentage of the CPU time obtained by the non-idle tasks in a
scheduling period is as follows:
CPU 1: time_percent(non-idle tasks) = 1024 / 1024 = 100%
CPU 0: time_percent(non-idle tasks) = 1024 / (1024 + 15000) = 6.4%
Obviously, if load balance migration tasks prefer migrate non-idle tasks
and suppress the interference of idle tasks migration on non-idle tasks,
the latency of non-idle tasks can be significantly reduced. Although
this will cause some idle tasks imbalance between different CPUs and
reduce throughput of idle tasks., I think this strategy is feasible in
some real-time business scenarios for latency tasks.
>>
>>>
>>> Best regards.
>>>
>>> Song Zhang
> .
next prev parent reply other threads:[~2022-11-16 7:37 UTC|newest]
Thread overview: 14+ messages / expand[flat|nested] mbox.gz Atom feed top
2022-11-02 3:53 Song Zhang
2022-11-02 18:01 ` Vincent Guittot
2022-11-03 3:01 ` Song Zhang
2022-11-03 8:33 ` Vincent Guittot
2022-11-03 9:20 ` Song Zhang
2022-11-03 9:22 ` Vincent Guittot
2022-11-03 11:45 ` Song Zhang
2022-11-04 2:05 ` Song Zhang
2022-11-12 2:51 ` Song Zhang
2022-11-14 16:42 ` Vincent Guittot
2022-11-15 7:18 ` Vincent Guittot
2022-11-16 7:37 ` Song Zhang [this message]
2022-11-16 14:38 ` Vincent Guittot
2022-11-17 9:07 ` Song Zhang
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=241e837b-056a-4fde-0673-205bd7400e82@huawei.com \
--to=zhangsong34@huawei.com \
--cc=bristot@redhat.com \
--cc=bsegall@google.com \
--cc=dietmar.eggemann@arm.com \
--cc=juri.lelli@redhat.com \
--cc=keescook@chromium.org \
--cc=linux-fsdevel@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=mcgrof@kernel.org \
--cc=mgorman@suse.de \
--cc=mingo@redhat.com \
--cc=peterz@infradead.org \
--cc=rostedt@goodmis.org \
--cc=vincent.guittot@linaro.org \
--cc=vschneid@redhat.com \
--cc=yzaikin@google.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®