From: Chen Yu <yu.c.chen@intel.com>
To: Jirka Hladky <jhladky@redhat.com>
Cc: Chen Yu <chen.yu@linux.dev>,
Tim Chen <tim.c.chen@linux.intel.com>,
"Peter Zijlstra" <peterz@infradead.org>,
Ingo Molnar <mingo@redhat.com>,
"K Prateek Nayak" <kprateek.nayak@amd.com>,
"Gautham R . Shenoy" <gautham.shenoy@amd.com>,
Vincent Guittot <vincent.guittot@linaro.org>,
"Juri Lelli" <juri.lelli@redhat.com>,
Dietmar Eggemann <dietmar.eggemann@arm.com>,
Steven Rostedt <rostedt@goodmis.org>,
Ben Segall <bsegall@google.com>, "Mel Gorman" <mgorman@suse.de>,
Valentin Schneider <vschneid@redhat.com>,
"Madadi Vineeth Reddy" <vineethr@linux.ibm.com>,
Hillf Danton <hdanton@sina.com>,
Shrikanth Hegde <sshegde@linux.ibm.com>,
Jianyong Wu <jianyong.wu@outlook.com>,
Yangyu Chen <cyy@cyyself.name>,
Tingyin Duan <tingyin.duan@gmail.com>,
Vern Hao <vernhao@tencent.com>, Vern Hao <haoxing990@gmail.com>,
Len Brown <len.brown@intel.com>, Aubrey Li <aubrey.li@intel.com>,
Zhao Liu <zhao1.liu@intel.com>, Chen Yu <yu.chen.surf@gmail.com>,
Adam Li <adamli@os.amperecomputing.com>,
Aaron Lu <ziqianlu@bytedance.com>,
Tim Chen <tim.c.chen@intel.com>, Josh Don <joshdon@google.com>,
Gavin Guo <gavinguo@igalia.com>,
Qais Yousef <qyousef@layalina.io>,
Libo Chen <libchen@purestorage.com>,
<linux-kernel@vger.kernel.org>,
"Luck, Tony" <tony.luck@intel.com>,
"Reinette Chatre" <reinette.chatre@intel.com>,
Philip Auld <pauld@redhat.com>, Joe Mario <jmario@redhat.com>
Subject: Re: [Patch v4 17/22] sched/cache: Avoid cache-aware scheduling for memory-heavy processes
Date: Wed, 18 Nov 2026 21:51:49 +0800 [thread overview]
Message-ID: <av2tdXOhCjMgLHsq@chenyu-dev> (raw)
In-Reply-To: <CAE4VaGD2idBi35zDQWCY2ybB0spz+tKEX5=59O5U5tBbjGuYAw@mail.gmail.com>
Hi Jirka,
On Wed, Nov 18, 2026 at 04:06:55AM +0100, Jirka Hladky wrote:
> Hi Tim, Chen Yu,
>
> Thank you both for the patches and the detailed design discussion.
> Two updates: the trace Tim asked for, and test results for the two
> patches.
>
> PART 1: is NUMA balancing preventing the re-spread?
>
> > Wonder if NUMA balancing is preventing the re-spread?
> > If your traces have numa_preferred_nid and task_numa_migrate activity
> > after the trip, that would confirm or rule out out.
>
> I traced it on the Fix-2 test kernel (mg.D 1x64 CAS-on, two runs that
> aggregated on different nodes, same structure in both): per-thread
> numa_preferred_nid sampled at every context switch, sched_setnuma()
> and migrate_task_to() in the task's own context, every
> sched_migrate_task with origin and destination, and wakeup target
> CPUs, alongside the group state per epoch.
>
> The short answer: yes, but only late, and it is not the main
> obstacle. The run decomposes into three phases:
>
> 1. 0 - 4.9 s: consolidation. About 30 cross-node migrations toward
> the aggregated node; the footprint crosses the threshold at
> 4.2 - 4.9 s and grp->cpu is cleared.
>
> 2. Trip to sweep completion (4.9 - 35.8 s in the traced run): the
> re-spread runs at about 0.5 net cross-node migrations per second
> (thread placement 2/63 across the two nodes at t=4 s, 18/47 at
> t=28 s), and numa_preferred_nid is -1 for every thread during this
> whole window: sched_setnuma() first fires only when the sweep
> completes and placement runs, at t=35.8 s. So the NUMA preference
> cannot be what holds the re-spread back here. What the trace shows
> instead: the wakeup placement keeps directing threads to the
> aggregated node at roughly 2:1 to 3:1 throughout the window
> (1264 vs 87 wakeups per 2 s at t=8 s, still 639 vs 380 at t=32 s;
> these are OMP barrier wakeups landing near their wakers), while
> the periodic balancer moves tasks out at only about 2 per 4 s.
> The net drift is the difference.
>
> 3. Sweep completion to end (35.8 - 42 s): placement sets
> numa_preferred_nid to the aggregated node for 53 of 66 threads
> (the pages have migrated there by then, so the fault statistics
> point there, exactly your reinforcing loop). From this moment the
> pull-back engages: migrate_task_to() toward the preferred node
> appears, and the cross-node churn in the last 6 s is 8 moves out
> vs 10 moves back, i.e. the spread stalls.
>
> So NUMA balancing does prevent the re-spread, but only in the last
> 15 percent of the run. For the first 85 percent the preferred node is
> simply not set, and what holds it back is wake placement plus the
> conservative pace of cross-node load balancing.
I created a test case that can reproduce this issue on a SRF. Actually, it is not
the wakeup that aggregates the wakee on the local LLC in wake_affine(), that
is the expected behavior for fast wakeup. Rather, because the tasks are first
aggregated onto the same LLC by cache-aware scheduling, the wakeup then tries
to wake the wakee on its previous/target CPU on the same LLC. The problem is
that load balancing spreads the task too slowly: because there is only one running
task on each CPU, active load balancing has to be triggered to spread the task
to a different LLC. Active load balancing has to exceed an upper limit of 5
failures: cache_nice_tries + 2 = 4, with a min_interval of 192 ms because
sd_weight = 2 NUMA nodes = 192 on my SRF, so roughly every 5 * 192 = 960 ms it
will pull a task out. In your case, there are 288 CPUs for 2 nodes, so in theory
it should take 288 * 5 = 1.4 seconds to pull one task out.
The test code is a single process with several threads just doing a while(1) loop,
without waking up each other. First, bind all threads to LLC0 (socket0), then release
their CPU affinity, and watch how they are spread to LLC1.
So to avoid this from happening, we should avoid aggregating the tasks at the first
place.
> Resetting NUMA preferences at withdrawal would therefore not help this workload:
> the sweep outlives the run, so there is nothing to reset during the
> window that matters. And the wake-placement observation suggests the
> re-spread would stay slow even with preferences reset; an active
> re-spread may need to move the tasks and let the wake placement
> follow.
>
> PART 2: test results for the two patches
>
> I built a kernel from the same 7.4.0-0.rc2.19.eln159 source, with
> Chen Yu's sketch (as tested before) and both patches applied. Both
> applied cleanly. Same test battery as the earlier rounds, n=5
> medians for the NAS runs:
>
> workload stock OFF stock ON patched OFF patched ON
> NAS MG D, 1x64thr 26.45 s 41.78 s 26.38 s 26.47 s (+0.3%)
> (+58%)
> NAS MG C, 1x4thr 9.34 s 11.99 s 9.38 s 9.36 s (-0.2%)
> (+28%)
>
> Both reported regressions are gone. The mechanism trace shows why:
> grp->cpu = -1 in all 42,566 task_cache_work() samples over one
> CAS-on MG run. The aggregation never settles, not even during the
> warm-up: the RSS bound refuses within ~0.1 s of the allocation phase,
> before first-touch can skew anything. The hysteresis from PART 1
> never develops because there is nothing to unwind.
>
> The VM reproducer [3], the workload that benefits from packing,
> shows one cost of the patches and one way around it:
>
> VM runtime: stock off 22.12 s, stock on 17.66 s (-20%)
> patched off 21.81 s
> patched on, tol=1: 22.47 s (+3.0%, win lost)
> patched on, tol=4: 17.70 s (-18.9%, win back)
>
> cache-locality: tol=1: -4.6%, tol=4: +24.6%
> mem-bandwidth: tol=1: +2.5%, tol=4: +30.6%
>
> The win is lost at the default tolerance: qemu's RSS (guest memory,
> all anon) far exceeds the 108 MiB LLC, and the first sweep of its
> ~25 GB address space takes ~100 s while a reproducer pass runs ~20 s,
> so the bound governs throughout and aggregation is refused. At
> aggr_tolerance=4 the bound is (1 + 3*256) x 108 MiB = 83 GiB, qemu
> fits, and the win returns. The tolerance knob thus doubles as the
> opt-in for large-RSS workloads that benefit from aggregation.
> Splitting the two, so the tolerance scales only the estimate and
> not the bound, may be worth considering; either way the coupling
> deserves a line in the documentation.
>
> On Chen Yu's question, whether Fix 1 is still needed if Fix 2
> accumulates the RSS bound: the data says yes for the regression
> class. The bound is what refuses aggregation for MG D and MG C,
> because their fault-based estimate never exists within a run (the
> sweep is too slow), and Fix 2 alone recovered only one sixth of the
> regression in the earlier test.
>
Actually what I mean was if the RSS(not footprint accumulation during
numa page fault) is used as the last resort if the footprint is 0,
will this change fix the issue anyway? I guess the answer is yes,
because you mentioned that the major fix is from the RSS gating.
> Fix 2 correcting the initial placement does happen for a
> long-running VM, and I traced it. I ran the VM reproducer with the
> in-VM workloads scaled 10x (about 100 s per phase, 4 minutes per
> pass). The group state over the CAS-on pass at the default
> tolerance:
>
> t=0 VM boots; rss_bound is 0 for the first epoch, CAS
> aggregates for ~0.1 s until the first RSS sample lands
> (435k pages), then refuses.
> t=16 s the first scan sweep folds: the history becomes 27,298
> pages, the VM's actual hot set, within 1.3 percent of
> the 27,648-page LLC. The estimate takes over from the
> RSS bound, the check passes, and the aggregation is
> granted: grp->cpu is set for the first time since boot.
> t=19 s the bandwidth phase grows the current window to 23,576
> pages; max(history, history/2 + window) crosses the
> threshold and the aggregation is withdrawn.
> rest stably refused; the estimate stays above the threshold.
>
> So the mechanism works exactly as designed end to end: the bound
> guards the window before the first sweep, the estimate corrects the
> decision afterward, and the flat combination catches the growth
> within the sweep. This VM is a boundary case by accident rather
> than design: its workload data is tiny, but the guest kernel and
> initramfs dominate the hot set, which was never sized against the
> LLC. Either way it is precisely the flip territory your max()
> smoothing addresses.
>
> The aggregate numbers reflect how little of the run the granted
> window covers: the long pass ends at -2.9% runtime, +2.4%
> cache-locality, +6.0% bandwidth, against -20% for the stock kernel
> and +3.0% for the same patched kernel with the short (~20 s)
> workload. A persistent VM partially recovers the benefit.
>
> One race worth noting: rss_bound is sampled at the first
> task_cache_work() epoch after group creation, so for about 0.1 s
> the bound is 0 and CAS can aggregate. In the traced run that blip
> was enough to set the initial vCPU placement (the threads clustered
> and never moved afterward; the box is idle and nothing pulls 8
> threads back). Sampling rss_bound at group creation in
> mm_init_sched(), or withholding aggregation until the first sample,
> would close it.
>
> Putting the three rounds of testing together, my recommendation:
>
> I vote for keeping both fixes. They do different jobs: the RSS
> bound is the only thing that protects the window before the first
> sweep, which is where the memory-bound workloads take their damage,
> and the within-sweep accumulation is what makes the decision
> correct afterward and recovers the beneficial cases. The cost
> asymmetry settles the default: a wrongly granted aggregation cost
> us 58 percent, a wrongly refused one forfeits a 20 percent gain.
>
> Two refinements seem worth considering on top:
>
> - Hysteresis on the grant/withdraw decision. The VM trace showed a
> boundary case as it sits close to the threshold, so the aggregation
> flips as the window grows and the history decays. Withdrawing above
> the threshold and re-granting below half of it, the pattern imb_pct
> and overaggr_pct already use, would stabilize it.
>
Agree, this is a good point that we should also apply hysteresis for
footprint/RSS.
> - The tolerance splitting. aggr_tolerance was designed to scale the
> footprint threshold, and it now scales the RSS bound too, so
> tolerance 2 jumps the bound from 108 MiB to 27.5 GiB. That works
> as an opt-in but is a blunt one; the per-group control from the
> prctl RFC seems the better tool for "this large-RSS workload
> wants aggregation", with the tolerance kept for the estimate.
>
Agree, BTW, if the workload has large-RSS but does not share data
between each threads, this is an indicator that cache-aware-scheduling
should not take care of. This information can be given either by the
userspace via prctl, or maybe by leveraging numa_group in the NUMA balancing
mechanism.
thanks,
Chenyu
> Happy to share the probe scripts or re-run anything.
>
> Thanks,
> Jirka
>
> [3] The VM test boots a small VM directly off the host's own kernel
> and initramfs, with no guest OS install: 1 VM, 8 vCPUs, 25 GB of
> guest memory. Two workloads run inside it, in this order: a
> cache-locality test (process pairs exchanging data through FIFO
> pipes, reported as ops/s) and a memory-bandwidth test (cp over a
> 216 MB buffer, twice this host's LLC, reported as MB/s). The
> test records which physical CPUs the vCPU threads land on, which
> is how the packing effect is measured. The long variant scales
> the in-VM iterations 10x, to about 100 s per workload phase.
>
> On Mon, Nov 17, 2026 at 12:39 AM Tim Chen <tim.c.chen@linux.intel.com> wrote:
>
next prev parent reply other threads:[~2026-11-18 14:03 UTC|newest]
Thread overview: 83+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-04-01 21:52 [Patch v4 00/22] Cache aware scheduling Tim Chen
2026-04-01 21:52 ` [Patch v4 01/22] sched/cache: Introduce infrastructure for cache-aware load balancing Tim Chen
2026-04-09 12:41 ` Peter Zijlstra
2026-04-09 19:21 ` Tim Chen
2026-04-09 23:00 ` Peter Zijlstra
2026-04-10 6:30 ` Chen, Yu C
2026-04-15 2:06 ` Vern Hao
2026-04-15 3:34 ` Chen, Yu C
2026-09-14 17:50 ` Zenghui Yu
2026-09-14 23:12 ` Tim Chen
2026-09-18 7:14 ` Zenghui Yu
2026-04-01 21:52 ` [Patch v4 02/22] sched/cache: Limit the scan number of CPUs when calculating task occupancy Tim Chen
2026-04-09 13:17 ` Luo Gengkun
2026-04-09 13:41 ` Peter Zijlstra
2026-04-10 10:12 ` Luo Gengkun
2026-04-10 7:29 ` Chen, Yu C
2026-04-10 10:20 ` Luo Gengkun
2026-04-10 17:12 ` Tim Chen
2026-04-10 17:27 ` Chen, Yu C
2026-04-13 7:23 ` [RFC PATCH] sched/fair: dynamically scale the period of cache work Jianyong Wu
2026-04-13 8:38 ` Chen, Yu C
2026-04-13 11:27 ` Jianyong Wu
2026-04-15 3:31 ` Chen, Yu C
2026-04-16 3:39 ` Jianyong Wu
2026-04-15 17:22 ` Tim Chen
2026-04-16 6:50 ` Jianyong Wu
2026-04-14 15:07 ` [PATCH v2] sched/cache: Reduce the overhead of task_cache_work by only scan the visisted cpus Luo Gengkun
2026-04-15 3:10 ` Chen, Yu C
2026-04-18 9:01 ` Luo Gengkun
2026-04-20 7:53 ` Chen, Yu C
2026-04-23 8:54 ` [PATCH v3] " Luo Gengkun
2026-04-01 21:52 ` [Patch v4 03/22] sched/cache: Record per LLC utilization to guide cache aware scheduling decisions Tim Chen
2026-04-01 21:52 ` [Patch v4 04/22] sched/cache: Introduce helper functions to enforce LLC migration policy Tim Chen
2026-04-01 21:52 ` [Patch v4 05/22] sched/cache: Make LLC id continuous Tim Chen
2026-04-01 21:52 ` [Patch v4 06/22] sched/cache: Assign preferred LLC ID to processes Tim Chen
2026-04-01 21:52 ` [Patch v4 07/22] sched/cache: Track LLC-preferred tasks per runqueue Tim Chen
2026-04-01 21:52 ` [Patch v4 08/22] sched/cache: Introduce per CPU's tasks LLC preference counter Tim Chen
2026-04-01 21:52 ` [Patch v4 09/22] sched/cache: Calculate the percpu sd task LLC preference Tim Chen
2026-04-01 21:52 ` [Patch v4 10/22] sched/cache: Count tasks prefering destination LLC in a sched group Tim Chen
2026-04-01 21:52 ` [Patch v4 11/22] sched/cache: Check local_group only once in update_sg_lb_stats() Tim Chen
2026-04-01 21:52 ` [Patch v4 12/22] sched/cache: Prioritize tasks preferring destination LLC during balancing Tim Chen
2026-04-01 21:52 ` [Patch v4 13/22] sched/cache: Add migrate_llc_task migration type for cache-aware balancing Tim Chen
2026-04-01 21:52 ` [Patch v4 14/22] sched/cache: Handle moving single tasks to/from their preferred LLC Tim Chen
2026-04-01 21:52 ` [Patch v4 15/22] sched/cache: Respect LLC preference in task migration and detach Tim Chen
2026-04-01 21:52 ` [Patch v4 16/22] sched/cache: Disable cache aware scheduling for processes with high thread counts Tim Chen
2026-04-09 12:43 ` Peter Zijlstra
2026-04-09 19:27 ` Tim Chen
2026-04-01 21:52 ` [Patch v4 17/22] sched/cache: Avoid cache-aware scheduling for memory-heavy processes Tim Chen
2026-04-09 12:46 ` Peter Zijlstra
2026-04-09 12:55 ` Peter Zijlstra
2026-04-10 8:59 ` Chen, Yu C
2026-04-10 9:20 ` Peter Zijlstra
2026-11-13 16:29 ` Jirka Hladky
2026-11-13 23:06 ` Tim Chen
2026-11-14 0:55 ` Jirka Hladky
2026-11-16 5:46 ` Chen Yu
2026-11-16 14:35 ` Jirka Hladky
2026-11-16 23:16 ` Tim Chen
2026-11-17 12:07 ` Chen Yu
2026-11-17 12:20 ` Chen Yu
2026-11-18 3:06 ` Jirka Hladky
2026-11-18 13:41 ` Phil Auld
2026-11-18 13:51 ` Chen Yu [this message]
2026-04-01 21:52 ` [Patch v4 18/22] sched/cache: Enable cache aware scheduling for multi LLCs NUMA node Tim Chen
2026-04-09 13:37 ` Peter Zijlstra
2026-04-09 19:39 ` Tim Chen
2026-04-01 21:52 ` [Patch v4 19/22] sched/cache: Allow the user space to turn on and off cache aware scheduling Tim Chen
2026-04-01 21:52 ` [Patch v4 20/22] sched/cache: Add user control to adjust the aggressiveness of cache-aware scheduling Tim Chen
2026-04-01 21:52 ` [Patch v4 21/22] -- DO NOT APPLY!!! -- sched/cache/debug: Display the per LLC occupancy for each process via proc fs Tim Chen
2026-04-01 21:52 ` [Patch v4 22/22] -- DO NOT APPLY!!! -- sched/cache/debug: Add ftrace to track the load balance statistics Tim Chen
2026-04-09 13:54 ` [Patch v4 00/22] Cache aware scheduling Peter Zijlstra
2026-04-09 20:02 ` Tim Chen
2026-04-14 3:20 ` Duan Tingyin
2026-04-15 17:35 ` Tim Chen
2026-04-16 0:27 ` Qais Yousef
2026-04-20 9:01 ` Chen, Yu C
2026-04-21 0:34 ` Qais Yousef
2026-04-21 20:57 ` Tim Chen
2026-04-23 15:06 ` Qais Yousef
2026-04-23 16:48 ` Chen, Yu C
2026-04-25 0:05 ` Qais Yousef
2026-04-23 17:17 ` Chen, Yu C
2026-04-25 0:14 ` Qais Yousef
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=av2tdXOhCjMgLHsq@chenyu-dev \
--to=yu.c.chen@intel.com \
--cc=adamli@os.amperecomputing.com \
--cc=aubrey.li@intel.com \
--cc=bsegall@google.com \
--cc=chen.yu@linux.dev \
--cc=cyy@cyyself.name \
--cc=dietmar.eggemann@arm.com \
--cc=gautham.shenoy@amd.com \
--cc=gavinguo@igalia.com \
--cc=haoxing990@gmail.com \
--cc=hdanton@sina.com \
--cc=jhladky@redhat.com \
--cc=jianyong.wu@outlook.com \
--cc=jmario@redhat.com \
--cc=joshdon@google.com \
--cc=juri.lelli@redhat.com \
--cc=kprateek.nayak@amd.com \
--cc=len.brown@intel.com \
--cc=libchen@purestorage.com \
--cc=linux-kernel@vger.kernel.org \
--cc=mgorman@suse.de \
--cc=mingo@redhat.com \
--cc=pauld@redhat.com \
--cc=peterz@infradead.org \
--cc=qyousef@layalina.io \
--cc=reinette.chatre@intel.com \
--cc=rostedt@goodmis.org \
--cc=sshegde@linux.ibm.com \
--cc=tim.c.chen@intel.com \
--cc=tim.c.chen@linux.intel.com \
--cc=tingyin.duan@gmail.com \
--cc=tony.luck@intel.com \
--cc=vernhao@tencent.com \
--cc=vincent.guittot@linaro.org \
--cc=vineethr@linux.ibm.com \
--cc=vschneid@redhat.com \
--cc=yu.chen.surf@gmail.com \
--cc=zhao1.liu@intel.com \
--cc=ziqianlu@bytedance.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®