From: Abel Wu <wuyun.abel@bytedance.com>
To: Peter Zijlstra <peterz@infradead.org>,
Ingo Molnar <mingo@kernel.org>,
Vincent Guittot <vincent.guittot@linaro.org>,
Dietmar Eggemann <dietmar.eggemann@arm.com>,
Valentin Schneider <valentin.schneider@arm.com>
Cc: Barry Song <21cnbao@gmail.com>,
Benjamin Segall <bsegall@google.com>,
Chen Yu <yu.c.chen@intel.com>,
Daniel Jordan <daniel.m.jordan@oracle.com>,
"Gautham R . Shenoy" <gautham.shenoy@amd.com>,
Joel Fernandes <joel@joelfernandes.org>,
K Prateek Nayak <kprateek.nayak@amd.com>,
Mike Galbraith <efault@gmx.de>, Qais Yousef <qyousef@layalina.io>,
Tim Chen <tim.c.chen@linux.intel.com>,
Yicong Yang <yangyicong@huawei.com>,
Youssef Esmat <youssefesmat@chromium.org>,
linux-kernel@vger.kernel.org, Abel Wu <wuyun.abel@bytedance.com>
Subject: [PATCH v2 3/4] sched/eevdf: O(1) fastpath for task selection
Date: Wed, 15 Nov 2023 11:36:46 +0800 [thread overview]
Message-ID: <20231115033647.80785-4-wuyun.abel@bytedance.com> (raw)
In-Reply-To: <20231115033647.80785-1-wuyun.abel@bytedance.com>
Since the RB-tree is now sorted by deadline, let's first try the
leftmost entity which has the earliest virtual deadline. I've done
some benchmarks to see its effectiveness.
All the benchmarks are done inside a normal cpu cgroup in a clean
environment with cpu turbo disabled, on a dual-CPU Intel Xeon(R)
Platinum 8260 with 2 NUMA nodes each of which has 24C/48T.
hackbench: process/thread + pipe/socket + 1/2/4/8 groups
netperf: TCP/UDP + STREAM/RR + 24/48/72/96/192 threads
tbench: loopback 24/48/72/96/192 threads
schbench: 1/2/4/8 mthreads
direct: cfs_rq has only one entity
parity: RUN_TO_PARITY
fast: O(1) fastpath
slow: heap search
(%) direct parity fast slow
hackbench 92.95 2.02 4.91 0.12
netperf 68.08 6.60 24.18 1.14
tbench 67.55 11.22 20.61 0.62
schbench 69.91 2.65 25.73 1.71
The above results indicate that this fastpath really makes task
selection more efficient.
Signed-off-by: Abel Wu <wuyun.abel@bytedance.com>
---
kernel/sched/fair.c | 14 +++++++++++---
1 file changed, 11 insertions(+), 3 deletions(-)
diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index e1d686196528..4197258b76ab 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -878,6 +878,7 @@ struct sched_entity *__pick_first_entity(struct cfs_rq *cfs_rq)
static struct sched_entity *pick_eevdf(struct cfs_rq *cfs_rq)
{
struct rb_node *node = cfs_rq->tasks_timeline.rb_root.rb_node;
+ struct sched_entity *se = __pick_first_entity(cfs_rq);
struct sched_entity *curr = cfs_rq->curr;
struct sched_entity *best = NULL;
@@ -886,7 +887,7 @@ static struct sched_entity *pick_eevdf(struct cfs_rq *cfs_rq)
* in this cfs_rq, saving some cycles.
*/
if (cfs_rq->nr_running == 1)
- return curr && curr->on_rq ? curr : __node_2_se(node);
+ return curr && curr->on_rq ? curr : se;
if (curr && (!curr->on_rq || !entity_eligible(cfs_rq, curr)))
curr = NULL;
@@ -898,9 +899,14 @@ static struct sched_entity *pick_eevdf(struct cfs_rq *cfs_rq)
if (sched_feat(RUN_TO_PARITY) && curr && curr->vlag == curr->deadline)
return curr;
+ /* Pick the leftmost entity if it's eligible */
+ if (se && entity_eligible(cfs_rq, se)) {
+ best = se;
+ goto found;
+ }
+
/* Heap search for the EEVD entity */
while (node) {
- struct sched_entity *se = __node_2_se(node);
struct rb_node *left = node->rb_left;
/*
@@ -913,6 +919,8 @@ static struct sched_entity *pick_eevdf(struct cfs_rq *cfs_rq)
continue;
}
+ se = __node_2_se(node);
+
/*
* The left subtree either is empty or has no eligible
* entity, so check the current node since it is the one
@@ -925,7 +933,7 @@ static struct sched_entity *pick_eevdf(struct cfs_rq *cfs_rq)
node = node->rb_right;
}
-
+found:
if (!best || (curr && entity_before(curr, best)))
best = curr;
--
2.37.3
next prev parent reply other threads:[~2023-11-15 3:38 UTC|newest]
Thread overview: 7+ messages / expand[flat|nested] mbox.gz Atom feed top
2023-11-15 3:36 [PATCH v2 0/4] sched/eevdf: Optimize reweight and pick Abel Wu
2023-11-15 3:36 ` [PATCH v2 1/4] sched/eevdf: Fix vruntime adjustment on reweight Abel Wu
2023-11-15 3:36 ` [PATCH v2 2/4] sched/eevdf: Sort the rbtree by virtual deadline Abel Wu
2023-11-15 9:04 ` [tip: sched/core] " tip-bot2 for Abel Wu
2023-11-15 3:36 ` Abel Wu [this message]
2023-11-15 9:04 ` [tip: sched/core] sched/eevdf: O(1) fastpath for task selection tip-bot2 for Abel Wu
2023-11-15 3:36 ` [PATCH v2 4/4] sched/stats: Add statistics for pick_eevdf() Abel Wu
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20231115033647.80785-4-wuyun.abel@bytedance.com \
--to=wuyun.abel@bytedance.com \
--cc=21cnbao@gmail.com \
--cc=bsegall@google.com \
--cc=daniel.m.jordan@oracle.com \
--cc=dietmar.eggemann@arm.com \
--cc=efault@gmx.de \
--cc=gautham.shenoy@amd.com \
--cc=joel@joelfernandes.org \
--cc=kprateek.nayak@amd.com \
--cc=linux-kernel@vger.kernel.org \
--cc=mingo@kernel.org \
--cc=peterz@infradead.org \
--cc=qyousef@layalina.io \
--cc=tim.c.chen@linux.intel.com \
--cc=valentin.schneider@arm.com \
--cc=vincent.guittot@linaro.org \
--cc=yangyicong@huawei.com \
--cc=youssefesmat@chromium.org \
--cc=yu.c.chen@intel.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®