From: Vinicius Costa Gomes <vinicius.gomes@intel.com>
To: Ingo Molnar <mingo@redhat.com>,
Peter Zijlstra <peterz@infradead.org>,
Juri Lelli <juri.lelli@redhat.com>,
Vincent Guittot <vincent.guittot@linaro.org>,
Dietmar Eggemann <dietmar.eggemann@arm.com>,
Steven Rostedt <rostedt@goodmis.org>,
Ben Segall <bsegall@google.com>, Mel Gorman <mgorman@suse.de>,
Valentin Schneider <vschneid@redhat.com>,
K Prateek Nayak <kprateek.nayak@amd.com>
Cc: Chen Yu <yu.c.chen@intel.com>,
Tim Chen <tim.c.chen@linux.intel.com>,
Srikar Dronamraju <srikar@linux.ibm.com>,
Shubhang Kaushik <sh@gentwo.org>,
Christoph Lameter <cl@gentwo.org>,
linux-kernel@vger.kernel.org,
Vinicius Costa Gomes <vinicius.gomes@intel.com>
Subject: [PATCH RFC] sched/fair: decline WF_SYNC stacking when waker LLC is the busier share
Date: Tue, 04 Aug 2026 16:14:05 -0700 [thread overview]
Message-ID: <20260804-epoll-wasync-fix-v1-v1-1-a83b4c8e08ba@intel.com> (raw)
Since commit 900bbaae67e9 ("epoll: Add synchronous wakeup support for
ep_poll_callback"), epoll driven WF_SYNC wakeups have been "too
strong" and could cause tasks to stack on a busy NUMA node while other
nodes are relatively idle.
As commit 900bbaae67e9 ("epoll: Add synchronous wakeup support for
ep_poll_callback") improves real workloads a revert is not the answer.
The fix is to make the WF_SYNC "stack on waker" shortcut take into
account the load on this and prev's LLC, rejecting the shortcut only
when the waker (this) LLC is fully loaded and prev's LLC is less
loaded than the waker's.
Signed-off-by: Vinicius Costa Gomes <vinicius.gomes@intel.com>
---
We received a report of a regression on a openresty based
workload (the main metric being tail latencies) on a CWF SNC3 single
socket system, the main symptom that we could measure was one node
being overloaded while the other nodes were relatively idle.
Further investigation showed that spreading the NIC RX interrupts over
all NUMA nodes helped. Reverting commit 900bbaae67e9 ("epoll: Add
synchronous wakeup support for ep_poll_callback") also helped.
Commit 900bbaae67e9 gave the hint, that perhaps for the WF_SYNC wakeup
path the 'sync' flag was too strong and was not taking the load into
account, which for workloads that are too quick for the NUMA load
balancer could perhaps cause this "over stacking".
The idea of this patch is to reject the stacking if it would overload
a LLC. I couldn't find a way to make it NUMA-only and only measured on
NUMA-like machines. One more reason for sending it as a RFC.
(I just became aware of [1], I haven't tested the series, but seems
that it's the same problem but approached from the "other side"; And
from [2], it's interesting, for this workload WF_SYNC is "too strong"
for others, it's too weak. Further documentation about the expected
behavior might be good indeed.)
To try and emulate the workload, which the wakeups happen mostly on
one node, I tried memcached + memtier_benchmark, with memcached
unpinned and memtier pinned to one of the NUMA nodes. As the customer
machine was single socket one, I offlined half the cores of the
machine I have access. (AI helped with this)
Kernels
-------
base master fc46aed51f62 (post-v7.2-rc5) vanilla
rfc base + sched/fair: decline WF_SYNC stacking when waker LLC is the
busier share
Results (memcached + memtier_benchmark, 10 repetitions, medians +/- stdev)
---------------------------------------------------------------
Results (memcached + memtier_benchmark, 10 repetitions, mean +/- stddev)
------------------------------------------------------------------------
Client (memtier, the wakers) pinned to one SNC node; memcached worker
threads span the whole socket. Latencies in milliseconds.
Load 64x32 (memtier --threads=64 --clients=32)
Throughput (higher is better)
metric base rfc delta
-------------------------------------------------------------
ops/sec 3.179M +/- 0.027 3.072M +/- 0.039 -3.4%
Latency (lower is better)
pctile base rfc delta
-------------------------------------------------------------
p50 (ms) 0.488 +/- 0.006 0.493 +/- 0.013 +1.0%
p75 0.783 +/- 0.009 0.809 +/- 0.011 +3.3%
p90 1.265 +/- 0.010 1.312 +/- 0.017 +3.7%
p99 2.756 +/- 0.026 2.897 +/- 0.028 +5.1%
p99.9 3.753 +/- 0.064 4.005 +/- 0.117 +6.7%
p99.99 5.993 +/- 0.308 5.938 +/- 0.322 -0.9%
Load 96x32 (memtier --threads=96 --clients=32)
Throughput (higher is better)
metric base rfc delta
-------------------------------------------------------------
ops/sec 2.788M +/- 0.028 2.693M +/- 0.040 -3.4%
Latency (lower is better)
pctile base rfc delta
-------------------------------------------------------------
p50 (ms) 0.773 +/- 0.020 1.109 +/- 0.013 +43.5%
p75 1.321 +/- 0.013 1.135 +/- 0.016 -14.1%
p90 2.442 +/- 0.028 1.188 +/- 0.070 -51.4%
p99 4.857 +/- 0.020 1.974 +/- 0.141 -59.4%
p99.9 7.145 +/- 0.048 2.722 +/- 0.699 -61.9%
p99.99 9.637 +/- 0.230 4.324 +/- 0.888 -55.1%
[1] https://lore.kernel.org/all/20260714013940.4068189-4-srikar@linux.ibm.com/
[2] https://lore.kernel.org/all/20260803-b4-sched-sync-wakeup-v4-1-52333b0cfb79@gentwo.org/
---
kernel/sched/fair.c | 55 +++++++++++++++++++++++++++++++++++++++++++++++++++--
1 file changed, 53 insertions(+), 2 deletions(-)
diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index d78467ec6ee1..c0c6cdaefda4 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -8185,6 +8185,56 @@ static int wake_wide(struct task_struct *p)
return 1;
}
+/*
+ * Decline WF_SYNC "stack on waker" wakeup when it would overload this LLC.
+ *
+ * Only decline when prev and this do not share a cache, this LLC has
+ * run out of idle headroom and prev is actually less loaded.
+ *
+ * Returns true when WF_SYNC should be declined.
+ */
+static bool sched_llc_over_commited(int this_cpu, int prev_cpu)
+{
+ struct sched_domain_shared *sds;
+ int this_busy, this_size, prev_busy, prev_size;
+ int headroom;
+
+ /* Same LLC: stacking cannot spread anywhere. */
+ if (cpus_share_cache(this_cpu, prev_cpu))
+ return false;
+
+ sds = rcu_dereference_all(per_cpu(sd_llc_shared, this_cpu));
+ if (!sds)
+ return false;
+ /* FIXME: if NO_HZ nr_busy_cpus is not updated, always accept? */
+ this_busy = atomic_read(&sds->nr_busy_cpus);
+ this_size = per_cpu(sd_llc_size, this_cpu);
+
+ /*
+ * Decline only once the waker LLC has no spare CPU beyond the
+ * waker's own, which is going to be replaced by this sync
+ * wakeup (the caller already made sure that nr_running == 1).
+ * If at least another idle CPU remains, keep stacking.
+ */
+ headroom = 1;
+
+ if (this_size - this_busy > headroom)
+ return false;
+
+ sds = rcu_dereference_all(per_cpu(sd_llc_shared, prev_cpu));
+ if (!sds)
+ return false;
+ prev_busy = atomic_read(&sds->nr_busy_cpus);
+ prev_size = per_cpu(sd_llc_size, prev_cpu);
+
+ /*
+ * Waker LLC is near saturation: spread only if prev is the less
+ * busy fraction (cross multiplied to take into account different
+ * LLC sizes).
+ */
+ return (u64)this_busy * prev_size > (u64)prev_busy * this_size;
+}
+
/*
* The purpose of wake_affine() is to quickly determine on which CPU we can run
* soonest. For the purpose of speed we only consider the waking and previous
@@ -8218,7 +8268,8 @@ wake_affine_idle(int this_cpu, int prev_cpu, int sync)
if (sync) {
struct rq *rq = cpu_rq(this_cpu);
- if ((rq->nr_running - cfs_h_nr_delayed(rq)) == 1)
+ if ((rq->nr_running - cfs_h_nr_delayed(rq)) == 1 &&
+ !sched_llc_over_commited(this_cpu, prev_cpu))
return this_cpu;
}
@@ -8237,7 +8288,7 @@ wake_affine_weight(struct sched_domain *sd, struct task_struct *p,
this_eff_load = cpu_load(cpu_rq(this_cpu));
- if (sync) {
+ if (sync && !sched_llc_over_commited(this_cpu, prev_cpu)) {
unsigned long current_load = task_h_load(current);
if (current_load > this_eff_load)
---
base-commit: c21bb4193868a8de71fc4693fa741e195fdf5d86
change-id: 20260723-epoll-wasync-fix-v1-28d2531033c7
Best regards,
--
Vinicius
next reply other threads:[~2026-08-04 23:14 UTC|newest]
Thread overview: 6+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-08-04 23:14 Vinicius Costa Gomes [this message]
2026-08-05 2:14 ` Hillf Danton
2026-08-06 17:44 ` Vinicius Costa Gomes
2026-08-06 22:53 ` Hillf Danton
2026-08-06 23:22 ` Vinicius Costa Gomes
2026-08-07 5:17 ` Hillf Danton
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20260804-epoll-wasync-fix-v1-v1-1-a83b4c8e08ba@intel.com \
--to=vinicius.gomes@intel.com \
--cc=bsegall@google.com \
--cc=cl@gentwo.org \
--cc=dietmar.eggemann@arm.com \
--cc=juri.lelli@redhat.com \
--cc=kprateek.nayak@amd.com \
--cc=linux-kernel@vger.kernel.org \
--cc=mgorman@suse.de \
--cc=mingo@redhat.com \
--cc=peterz@infradead.org \
--cc=rostedt@goodmis.org \
--cc=sh@gentwo.org \
--cc=srikar@linux.ibm.com \
--cc=tim.c.chen@linux.intel.com \
--cc=vincent.guittot@linaro.org \
--cc=vschneid@redhat.com \
--cc=yu.c.chen@intel.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®