[PATCH v3] sched/cache: Remove the old cache group footprint on exec

[PATCH v3] sched/cache: Remove the old cache group footprint on exec

[Jemmy Wong]

Exec replaces the task's cache group before resetting its NUMA fault
statistics, but only the exit path subtracts the task's contribution
from the old group's footprint.

Although de_thread() removes other members of the executing task's
thread group, tasks sharing the mm through CLONE_VM without CLONE_THREAD
can retain the old mm and cache group. The executing task's contribution
then remains without further updates or decay, potentially suppressing
cache-aware aggregation through the LLC capacity check. A vfork() child
rarely accumulates NUMA faults before exec because scanning starts late;
the issue mainly concerns longer-lived CLONE_VM tasks. This was found by
code inspection, not a workload reproducer.

Simply subtracting on exec is insufficient: the unlocked read/modify/write
updates can race with NUMA placement by another task sharing the group,
which can overwrite the subtraction. Convert footprint to atomic_long_t
and use non-returning atomic add/sub operations, which require no ordering
against unrelated accesses. Apply the net change in total_numa_faults once
per placement pass to limit shared cache-line traffic.

Rename sched_cache_replace_grp() to sched_cache_switch_grp() to reflect
its expanded responsibility: subtract the task's contribution when leaving
the old group, publish the new pointer, and return the old reference for
the caller to release. Use it for both exec and exit, and document that
exec subtraction must precede task_numa_free() resetting the NUMA total.

With atomic updates, placement deltas balance the departing task's total,
so no write-side clamp is needed, provided the accounting fits in signed
long. Retain a defensive read-side clamp and warn on a negative footprint
observed by the LLC capacity check.

Keep the existing NUMA reset ordering. If exec fails after switching groups
but before resetting the statistics, exit can subtract the old total from
the new group. This mismatch is confined to the execing task's private
group: the task is past the point of no return and the group disappears on
exit. The surviving old shared group remains correctly accounted.

Fixes: b636fef85bda ("sched/cache: Introduce task_struct->sched_cache_grp to fix UAF")
Suggested-by: Chen Yu <yu.c.chen@intel.com>
Signed-off-by: Jemmy Wong <jemmywong512@gmail.com>
---
v1: https://lore.kernel.org/all/20261009151215.62878-1-jemmywong512@gmail.com/
v2: https://lore.kernel.org/all/20261010111612.31345-1-jemmywong512@gmail.com/

Changes in v3:
- Convert grp->footprint to atomic_long_t and use atomic add/sub to avoid
  losing placement updates or exec/exit subtractions, as Chen Yu suggested.
- Update the group footprint once per NUMA placement pass, using the net
  change in p->total_numa_faults rather than updating each fault bucket.
- Rename sched_cache_replace_grp() to sched_cache_switch_grp() and move
  leave-group accounting into it, as agreed with Chen Yu. Exec and exit
  share the subtraction, guarded by old && old != new.
- Remove the write-side clamp now that updates are atomic. Use an atomic
  read, WARN_ON_ONCE(), and a defensive zero clamp in the capacity check.
- Explain the failed-exec over-subtraction case and why it is confined to
  the dying task's private group; leave NUMA reset ordering unchanged.
- Add Suggested-by: Chen Yu. Drop Tim Chen's Reviewed-by tag because the
  accounting changes are substantial and need renewed review.

Changes in v2:
- Mention vfork(), explain that longer-lived CLONE_VM tasks are the main
  concern, and state that the issue was found by code inspection.
- Add the Fixes tag and document that exec footprint subtraction must
  precede task_numa_free() resetting p->total_numa_faults.
- Add Tim Chen's Reviewed-by tag (not carried forward to v3).

Validation:
- Strict checkpatch and whitespace checks passed.
- Built kernel/sched/{fair,core,build_utility}.o on x86-64 with:
  - CONFIG_SCHED_CACHE=y, CONFIG_NUMA_BALANCING=y
  - CONFIG_SCHED_CACHE=y, CONFIG_NUMA_BALANCING=n
  - CONFIG_SCHED_CACHE=n, CONFIG_NUMA_BALANCING=y
- Runtime and performance testing have not been performed.

 include/linux/sched.h |  3 +-
 kernel/sched/fair.c   | 84 +++++++++++++++++--------------------------
 2 files changed, 35 insertions(+), 52 deletions(-)

diff --git a/include/linux/sched.h b/include/linux/sched.h
index d35ae49a991f..4a02a47c988e 100644
--- a/include/linux/sched.h
+++ b/include/linux/sched.h
@@ -2412,7 +2412,8 @@ struct sched_cache_group {
 	unsigned long epoch;
 	u64 nr_running_avg;
 	unsigned long next_scan;
-	unsigned long footprint;
+	/* Sum of the member tasks' accounted footprints, in pages. */
+	atomic_long_t footprint;
 	int cpu;
 	refcount_t refcnt;
 	struct rcu_head rcu;
diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index 57360f5cdde4..47faa0bf12c7 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -1483,6 +1483,7 @@ static bool exceed_llc_capacity(struct sched_cache_group *grp, int cpu)
 #ifdef CONFIG_NUMA_BALANCING
 	unsigned long llc, footprint;
 	struct sched_domain *sd;
+	long fp;
 	int scale;

 	guard(rcu)();
@@ -1497,7 +1498,9 @@ static bool exceed_llc_capacity(struct sched_cache_group *grp, int cpu)
 		 * excluded.
 		 */
 		llc = sd->llc_bytes;
-		footprint = READ_ONCE(grp->footprint);
+		fp = atomic_long_read(&grp->footprint);
+		WARN_ON_ONCE(fp < 0);
+		footprint = max(fp, 0L);

 		/*
 		 * Scale the LLC size by 256*llc_aggr_tolerance
@@ -1677,7 +1680,7 @@ int mm_init_sched(struct mm_struct *mm,
 	grp->cpu = -1;
 	grp->next_scan = jiffies;
 	grp->nr_running_avg = 0;
-	grp->footprint = 0;
+	atomic_long_set(&grp->footprint, 0);
 	refcount_set(&grp->refcnt, 1);
 	/*
 	 * The update to grp->pcpu_sched should not be reordered
@@ -1719,12 +1722,26 @@ DEFINE_FREE(sched_cache_group_put, struct sched_cache_group *,
 #define rcu_deref_sched_cache_grp(tsk) \
 	rcu_dereference_check((tsk)->sched_cache_grp, (tsk) == current)

-static struct sched_cache_group *sched_cache_replace_grp(struct task_struct *p,
-							 struct sched_cache_group *new)
+static inline void sched_cache_footprint_update(struct task_struct *p, long delta)
+{
+	struct sched_cache_group *grp = rcu_deref_sched_cache_grp(p);
+
+	if (grp && delta)
+		atomic_long_add(delta, &grp->footprint);
+}
+
+/* Remove the old contribution and return the old group for the caller to put. */
+static struct sched_cache_group *sched_cache_switch_grp(struct task_struct *p,
+							struct sched_cache_group *new)
 {
 	struct sched_cache_group *old;

 	old = rcu_deref_sched_cache_grp(p);
+#ifdef CONFIG_NUMA_BALANCING
+	/* Other tasks sharing the old mm can keep its group alive after exec. */
+	if (old && old != new)
+		atomic_long_sub(p->total_numa_faults, &old->footprint);
+#endif
 	rcu_assign_pointer(p->sched_cache_grp, new);

 	return old;
@@ -1778,28 +1795,18 @@ void sched_cache_exec_mmap(struct task_struct *p, struct mm_struct *mm)
 	/*
 	 * Acquire the new reference before publishing the pointer, then drop
 	 * the old one. @p is current and the only writer of its own pointer.
+	 *
+	 * Footprint subtraction must precede task_numa_free() in bprm_execve(),
+	 * which clears p->total_numa_faults. Nothing here enforces that order.
 	 */
-	old = sched_cache_replace_grp(p, sched_cache_group_get(mm->sched_cache_grp));
+	old = sched_cache_switch_grp(p, sched_cache_group_get(mm->sched_cache_grp));
 	sched_cache_group_put(old);
 }

 void sched_cache_exit_mm(struct task_struct *p)
 {
-	struct sched_cache_group *grp = sched_cache_replace_grp(p, NULL);
-
-#ifdef CONFIG_NUMA_BALANCING
-	/*
-	 * Subtract this task's footprint from the group before dropping the
-	 * reference, so the group footprint converges as its threads exit.
-	 * Unlocked for performance; clamp to avoid underflow.
-	 */
-	if (grp && p->total_numa_faults) {
-		unsigned long fp = READ_ONCE(grp->footprint);
-		unsigned long sub = min(fp, p->total_numa_faults);
+	struct sched_cache_group *grp = sched_cache_switch_grp(p, NULL);

-		WRITE_ONCE(grp->footprint, fp - sub);
-	}
-#endif
 	sched_cache_group_put(grp);
 }

@@ -2189,6 +2196,9 @@ void init_sched_mm(struct task_struct *p)
 static inline void account_mm_sched(struct rq *rq, struct task_struct *p,
 				    s64 delta_exec) { }

+static inline void sched_cache_footprint_update(struct task_struct *p,
+						long delta) { }
+
 void init_sched_mm(struct task_struct *p) { }

 static void task_tick_cache(struct rq *rq, struct task_struct *p) { }
@@ -3883,14 +3893,13 @@ static int preferred_group_nid(struct task_struct *p, int nid)
 static void task_numa_placement(struct task_struct *p)
 	__context_unsafe(/* conditional locking */)
 {
-	struct sched_cache_group __maybe_unused *grp;
 	int seq, nid, max_nid = NUMA_NO_NODE;
 	unsigned long max_faults = 0;
 	unsigned long fault_types[2] = { 0, 0 };
+	unsigned long old_faults = p->total_numa_faults;
 	unsigned long total_faults;
 	u64 runtime, period;
 	spinlock_t *group_lock = NULL;
-	long __maybe_unused new_fp;
 	struct numa_group *ng;

 	/*
@@ -3965,36 +3974,6 @@ static void task_numa_placement(struct task_struct *p)
 				ng->total_faults += diff;
 				group_faults += ng->faults[mem_idx];
 			}
-#ifdef CONFIG_SCHED_CACHE
-			/*
-			 * Per task p->numa_faults[mem_idx] converges,
-			 * so the accumulation of each task's faults
-			 * converges too - Given the number of threads,
-			 * it cannot overflow an unsigned long.
-			 * Racy with concurrent updates from other threads
-			 * sharing this mm. Acceptable since footprint is a
-			 * heuristic and occasional lost updates are tolerable.
-			 *
-			 * If a task exits, its corresponding footprint must
-			 * be subtracted from p->sched_cache_grp->footprint,
-			 * otherwise the footprint will not converge: the
-			 * exiting thread's footprint remains unchanged/undecayed.
-			 * See exit_mm().
-			 *
-			 * Lost updates and unsynchronized subtraction
-			 * in exit_mm() can cause footprint + diff to
-			 * go negative. Clamp to zero to prevent the
-			 * unsigned footprint from wrapping.
-			 */
-			scoped_guard(rcu) {
-				grp = rcu_dereference(p->sched_cache_grp);
-
-				if (grp) {
-					new_fp = (long)READ_ONCE(grp->footprint) + diff;
-					WRITE_ONCE(grp->footprint, max(new_fp, 0L));
-				}
-			}
-#endif
 		}

 		if (!ng) {
@@ -4017,6 +3996,9 @@ static void task_numa_placement(struct task_struct *p)
 		max_nid = preferred_group_nid(p, max_nid);
 	}

+	/* Apply the net change once per placement pass to limit cache-line traffic. */
+	sched_cache_footprint_update(p, p->total_numa_faults - old_faults);
+
 	if (max_faults) {
 		/* Set the new preferred node */
 		if (max_nid != p->numa_preferred_nid)
--
2.25.1

Re: [PATCH v3] sched/cache: Remove the old cache group footprint on exec

[Chen Yu]

On Thu, Oct 15, 2026 at 01:36:04PM +0800, Jemmy Wong wrote:

[ ... ]

> @@ -3965,36 +3974,6 @@ static void task_numa_placement(struct task_struct *p)
>  				ng->total_faults += diff;
>  				group_faults += ng->faults[mem_idx];
>  			}
> -#ifdef CONFIG_SCHED_CACHE
> -			/*
> -			 * Per task p->numa_faults[mem_idx] converges,
> -			 * so the accumulation of each task's faults
> -			 * converges too - Given the number of threads,
> -			 * it cannot overflow an unsigned long.

Above comments can be kept for future reference, with minor adjustment:

 s/it cannot overflow an unsigned long/it cannot overflow an atomic_long_t

I launched a full hackbench test on a Xeon with multiple LLCs per node. There
is no obvious difference except for some false-positive regress with large
run-to-run variance. This patch looks good to me in terms of turning the
footprint into an accurate counter. Feel free to add:

Reviewed-by: Chen Yu <yu.c.chen@intel.com>
Tested-by: Chen Yu <yu.c.chen@intel.com>

thanks,
Chenyu



First run:
=========================================
Hackbench comparison
BASE: cas_on_baseline
TEST: cas_on_patched
=========================================
MODE     G  FD |         BASE(s) |         TEST(s) |   DIFF% | RESULT
------- -- ----+-----------------+-----------------+---------+----------
process 16   1 |      0.178/1.7% |      0.166/0.6% |   6.74% | IMPROVED
process 16  16 |      4.871/9.1% |      5.230/3.8% |  -7.37% | REGRESSED
process 16   2 |      1.574/3.0% |     1.296/27.5% |  17.66% | IMPROVED
process 16  20 |      6.502/7.8% |      6.262/1.5% |   3.69% | IMPROVED
process 16   4 |      2.296/5.0% |     2.138/13.0% |   6.88% | IMPROVED
process 16   8 |      2.056/1.1% |     1.982/12.3% |   3.60% | IMPROVED
process  1   1 |      2.040/0.8% |      2.010/4.5% |   1.47% | IMPROVED
process  1  16 |      4.837/3.5% |     4.189/12.7% |  13.40% | IMPROVED
process  1   2 |      3.708/5.0% |     2.768/14.3% |  25.35% | IMPROVED
process  1  20 |      4.552/3.5% |      4.505/2.2% |   1.03% | IMPROVED
process  1   4 |     2.696/17.0% |     3.092/21.8% | -14.69% | REGRESSED
process  1   8 |      5.474/3.4% |      5.372/9.3% |   1.86% | IMPROVED
process  2   1 |      1.089/3.4% |      1.115/2.3% |  -2.39% | REGRESSED
process  2  16 |     4.985/13.9% |      5.246/5.1% |  -5.24% | REGRESSED
process  2   2 |      1.063/7.2% |      0.998/3.2% |   6.11% | IMPROVED
process  2  20 |     5.091/13.6% |      5.081/3.1% |   0.20% | IMPROVED
process  2   4 |     3.935/18.2% |     3.438/23.2% |  12.63% | IMPROVED
process  2   8 |      4.791/7.0% |      4.925/3.8% |  -2.80% | REGRESSED
process  4   1 |     0.690/18.3% |      0.586/1.9% |  15.07% | IMPROVED
process  4  16 |      5.878/3.5% |      6.119/1.9% |  -4.10% | REGRESSED
process  4   2 |      0.534/6.4% |      0.512/2.7% |   4.12% | IMPROVED
process  4  20 |      5.839/3.1% |      5.644/5.0% |   3.34% | IMPROVED
process  4   4 |      4.012/1.5% |     3.386/12.0% |  15.60% | IMPROVED
process  4   8 |     4.737/11.0% |      4.703/3.9% |   0.72% | IMPROVED
process  8   1 |      0.317/0.9% |      0.322/0.3% |  -1.58% | REGRESSED
process  8  16 |      3.671/5.4% |      3.922/4.8% |  -6.84% | REGRESSED
process  8   2 |     0.921/35.8% |     0.812/36.0% |  11.83% | IMPROVED
process  8  20 |      4.878/6.9% |      4.769/1.8% |   2.23% | IMPROVED
process  8   4 |     2.969/10.2% |     3.500/16.5% | -17.88% | REGRESSED
process  8   8 |      4.698/2.6% |      4.263/4.7% |   9.26% | IMPROVED
threads 16   1 |      0.192/0.5% |      0.189/0.0% |   1.56% | IMPROVED
threads 16  16 |      5.430/4.7% |      5.685/1.2% |  -4.70% | REGRESSED
threads 16   2 |     0.362/17.4% |     0.577/29.6% | -59.39% | REGRESSED
threads 16  20 |      6.210/3.6% |      5.523/6.2% |  11.06% | IMPROVED
threads 16   4 |     2.678/11.0% |      3.173/5.6% | -18.48% | REGRESSED
threads 16   8 |      2.347/3.3% |      2.394/8.7% |  -2.00% | REGRESSED
threads  1   1 |      2.131/1.5% |      2.260/2.5% |  -6.05% | REGRESSED
threads  1  16 |      0.379/2.4% |      0.353/2.5% |   6.86% | IMPROVED
threads  1   2 |      2.050/0.9% |      2.003/0.6% |   2.29% | IMPROVED
threads  1  20 |      0.347/2.6% |      0.324/0.9% |   6.63% | IMPROVED
threads  1   4 |      1.342/1.0% |      1.333/1.1% |   0.67% | IMPROVED
threads  1   8 |      0.730/2.5% |      0.732/1.4% |  -0.27% | REGRESSED
threads  2   1 |      1.144/1.2% |      1.167/1.2% |  -2.01% | REGRESSED
threads  2  16 |      0.286/1.4% |     0.987/71.3% | -245.10% | REGRESSED
threads  2   2 |      1.035/2.4% |      1.040/0.5% |  -0.48% | REGRESSED
threads  2  20 |     1.016/73.2% |     2.038/43.5% | -100.59% | REGRESSED
threads  2   4 |      0.647/0.9% |      0.647/1.2% |   0.00% | SAME
threads  2   8 |      0.338/1.8% |      0.335/1.5% |   0.89% | IMPROVED
threads  4   1 |      0.658/1.4% |      0.614/4.4% |   6.69% | IMPROVED
threads  4  16 |      6.434/1.4% |      6.194/1.5% |   3.73% | IMPROVED
threads  4   2 |      0.522/2.1% |      0.521/1.7% |   0.19% | IMPROVED
threads  4  20 |      5.361/1.4% |      6.060/1.6% | -13.04% | REGRESSED
threads  4   4 |      0.324/1.5% |      0.335/1.8% |  -3.40% | REGRESSED
threads  4   8 |     0.946/35.4% |     0.854/61.1% |   9.73% | IMPROVED
threads  8   1 |      0.351/5.7% |      0.364/1.9% |  -3.70% | REGRESSED
threads  8  16 |      3.818/4.8% |      4.109/4.9% |  -7.62% | REGRESSED
threads  8   2 |      0.294/6.5% |      0.276/1.4% |   6.12% | IMPROVED
threads  8  20 |      4.640/5.2% |      4.788/9.1% |  -3.19% | REGRESSED
threads  8   4 |     1.548/57.0% |     0.880/11.0% |  43.15% | IMPROVED
threads  8   8 |      5.028/2.2% |      5.161/2.9% |  -2.65% | REGRESSED

Then I pick the following test, which seems to have a big regression with a
large stderr%:
threads 2 16 | 0.286/1.4% | 0.987/71.3% | -245.10% | REGRESSED
threads 2 20 | 1.016/73.2% | 2.038/43.5% | -100.59% | REGRESSED
threads 16 2 | 0.362/17.4% | 0.577/29.6% | -59.39% | REGRESSED
threads 16 4 | 2.678/11.0% | 3.173/5.6% | -18.48% | REGRESSED


The following is the second run:
=========================================
MODE     G  FD |         BASE(s) |         TEST(s) |   DIFF% | RESULT
------- -- ----+-----------------+-----------------+---------+----------
threads  2  16 |     0.915/29.6% |     1.376/66.1% | -50.38% | REGRESSED
threads  2  20 |     2.154/23.0% |     1.299/50.7% |  39.69% | IMPROVED
threads 16   2 |     0.320/18.8% |     0.756/26.3% | -136.25% | REGRESSED
threads 16   4 |      4.139/1.9% |     3.174/18.2% |  23.31% | IMPROVED

So the data seems to be quite unstable when the system is saturated, which
might be another issue that I'll investigate soon.

Re: [PATCH v3] sched/cache: Remove the old cache group footprint on exec

[Jemmy Wong]

> On Oct 21, 2026, at 6:12 PM, Chen Yu <yu.c.chen@intel.com> wrote:
> 
> On Thu, Oct 15, 2026 at 01:36:04PM +0800, Jemmy Wong wrote:
> 
> [ ... ]
> 
>> @@ -3965,36 +3974,6 @@ static void task_numa_placement(struct task_struct *p)
>> ng->total_faults += diff;
>> group_faults += ng->faults[mem_idx];
>> }
>> -#ifdef CONFIG_SCHED_CACHE
>> - /*
>> - * Per task p->numa_faults[mem_idx] converges,
>> - * so the accumulation of each task's faults
>> - * converges too - Given the number of threads,
>> - * it cannot overflow an unsigned long.
> 
> Above comments can be kept for future reference, with minor adjustment:
> 
> s/it cannot overflow an unsigned long/it cannot overflow an atomic_long_t
> 
> I launched a full hackbench test on a Xeon with multiple LLCs per node. There
> is no obvious difference except for some false-positive regress with large
> run-to-run variance. This patch looks good to me in terms of turning the
> footprint into an accurate counter. Feel free to add:
> 
> Reviewed-by: Chen Yu <yu.c.chen@intel.com>
> Tested-by: Chen Yu <yu.c.chen@intel.com>
> 
> thanks,
> Chenyu

Hi Chenyu,

Thanks for the review and testing.

For v4, I've restored the convergence comment next to the batched
footprint update, changed the type to atomic_long_t as suggested, and
rewrapped the text:


+ /*
+ * Per task p->numa_faults[mem_idx] converges, so the accumulation of each
+ * task's faults converges too. Given the number of threads, it cannot
+ * overflow an atomic_long_t.
+ *
+ * Apply the net change once per placement pass to limit cache-line traffic.
+ */
+ sched_cache_footprint_update(p, p->total_numa_faults - old_faults);
+

Does this wording and placement look fine to you?

I'll also include your Reviewed-by and Tested-by tags in v4.

Thanks,
Jemmy

Re: [PATCH v3] sched/cache: Remove the old cache group footprint on exec

[Chen Yu]

On Wed, Oct 21, 2026 at 08:50:52PM +0800, Jemmy Wong wrote:
> > On Oct 21, 2026, at 6:12 PM, Chen Yu <yu.c.chen@intel.com> wrote:
> > 
> > On Thu, Oct 15, 2026 at 01:36:04PM +0800, Jemmy Wong wrote:
> > 
> > [ ... ]
> > 
> >> @@ -3965,36 +3974,6 @@ static void task_numa_placement(struct task_struct *p)
> >> ng->total_faults += diff;
> >> group_faults += ng->faults[mem_idx];
> >> }
> >> -#ifdef CONFIG_SCHED_CACHE
> >> - /*
> >> - * Per task p->numa_faults[mem_idx] converges,
> >> - * so the accumulation of each task's faults
> >> - * converges too - Given the number of threads,
> >> - * it cannot overflow an unsigned long.
> > 
> > Above comments can be kept for future reference, with minor adjustment:
> > 
> > s/it cannot overflow an unsigned long/it cannot overflow an atomic_long_t
> > 
> > I launched a full hackbench test on a Xeon with multiple LLCs per node. There
> > is no obvious difference except for some false-positive regress with large
> > run-to-run variance. This patch looks good to me in terms of turning the
> > footprint into an accurate counter. Feel free to add:
> > 
> > Reviewed-by: Chen Yu <yu.c.chen@intel.com>
> > Tested-by: Chen Yu <yu.c.chen@intel.com>
> > 
> > thanks,
> > Chenyu
> 
> Hi Chenyu,
> 
> Thanks for the review and testing.
> 
> For v4, I've restored the convergence comment next to the batched
> footprint update, changed the type to atomic_long_t as suggested, and
> rewrapped the text:
> 
> 
> + /*
> + * Per task p->numa_faults[mem_idx] converges, so the accumulation of each
> + * task's faults converges too. Given the number of threads, it cannot
> + * overflow an atomic_long_t.
> + *
> + * Apply the net change once per placement pass to limit cache-line traffic.
> + */
> + sched_cache_footprint_update(p, p->total_numa_faults - old_faults);
> +
> 
> Does this wording and placement look fine to you?
>

Yes,  it looks good to me.

thanks,
Chenyu 

> I'll also include your Reviewed-by and Tested-by tags in v4.
> 
> Thanks,
> Jemmy