mirror of https://lore.kernel.org/lkml/
 help / color / mirror / Atom feed
* [RFC PATCH 0/3] fork,arm,x86,fs/resctrl: Fix stale resource group assignment on fork
@ 2026-11-20 18:39 Reinette Chatre
  2026-11-20 18:39 ` [RFC PATCH 1/3] fork,arm,x86,fs/resctrl: Fix stale resource group assignment vs copy_process() Reinette Chatre
                   ` (2 more replies)
  0 siblings, 3 replies; 4+ messages in thread
From: Reinette Chatre @ 2026-11-20 18:39 UTC (permalink / raw)
  To: kees, mingo, peterz, juri.lelli, vincent.guittot,
	dietmar.eggemann, rostedt, bsegall, mgorman, vschneid,
	kprateek.nayak, david, ljs, liam, vbabka, rppt, surenb, mhocko,
	bp, tglx, dave.hansen, tony.luck, ben.horgan, Dave.Martin,
	james.morse, babu.moger, fustini, jedrzej.wasiukiewicz
  Cc: fenghuay, peternewman, yu.c.chen, x86, linux-mm, linux-kernel,
	patches, reinette.chatre

Hi Everybody,

Jedrzej reported that a newly created resource group can start out with tasks
in it, before any task has been assigned to it. He created the reproducer in
patch 3 to demonstrate this.

Hitting this needs a fork() to overlap the removal of the resource group the
forking task belongs to. A system that forks heavily while monitor groups are
created and removed around it will eventually do both at once.

Removing a resctrl resource group moves the group's tasks to another group by
walking the task list, and then frees the group's CLOSID and RMID. A task
created while that walk runs escapes it: fork() copies the parent's CLOSID
and RMID early in copy_process(), before the new task is linked onto the task
list, and the walk holds tasklist_lock for read so the new task cannot join
the list until the walk has finished.

The new task is thus left with the CLOSID and RMID of a resource group that no
longer exists, and passes them on to every task it creates.

Once the freed CLOSID and RMID are handed to a resource group created later the
strays appear in that group's tasks file, their memory accesses are counted in
its monitoring data, and they are subject to its allocation policy, while the
group intended for them does not control or monitor them.

Series layout
=============

Patch 1 depicts the race.

resctrl has no fork() hook today. Patch 1 adds one, resctrl_copy_process(),
called from copy_process() while tasklist_lock is held for write, which is the
lock the task list walk is serialized against. Considering klp_copy_process(),
this seems to match the custom used to address an issue with this shape.

  1   Add resctrl_copy_process() and assign a task's CLOSID and RMID
      with tasklist_lock held
  2   Record the new tasklist_lock requirement with lockdep
  3   Reproducer, not for inclusion

Why is this an RFC?
===================

- resctrl_copy_process() runs on every fork(), so
  resctrl_arch_copy_closid_rmid() is a static inline on x86 but is
  out-of-line for MPAM. The note below patch 1 describes what was
  attempted to make it inline for MPAM and asks the MPAM maintainers
  what shape it should take there.

- Patch 1 touches kernel/fork.c. It adds a single call inside the
  existing write_lock_irq(&tasklist_lock) region of copy_process(),
  immediately after klp_copy_process() on which it is modelled.

How was this tested?
====================
The issue can be reproduced using Jedrzej's script found in patch #3. On
my test system the issue is encountered on a single run. With these fixes
the reproducer did not encounter the issue in 100 runs.

What remains?
=============
On x86 the CLOSID and RMID are separate values in struct task_struct and
__resctrl_sched_in() reads them without a lock, so a task can still be
labelled with a mismatched pair for the duration of a move. Unlike the
fork case, which is permanent and invisible to every later walk of the
task list, that window is transient and the pair is corrected at the next
context switch. MPAM stores the two as a single value.

The series is based on v7.4-rc3.

Any feedback is appreciated.

Thank you.

Reinette

Jedrzej Wasiukiewicz (1):
  NOT_FOR_INCLUSION: selftests/resctrl: Tasks keep using removed RMID
    after fork() reproducer

Reinette Chatre (2):
  fork,arm,x86,fs/resctrl: Fix stale resource group assignment vs
    copy_process()
  arm,x86/resctrl: Assert tasklist_lock is held when setting CLOSID and
    RMID

 arch/arm64/include/asm/mpam.h                 |   6 +
 arch/x86/include/asm/resctrl.h                |  11 ++
 drivers/resctrl/mpam_resctrl.c                |   8 ++
 fs/resctrl/rdtgroup.c                         |  21 +++
 include/linux/arm_mpam.h                      |   1 +
 include/linux/resctrl.h                       |  28 ++++
 kernel/fork.c                                 |   3 +
 .../selftests/resctrl/stray_tasks_repro.sh    | 122 ++++++++++++++++++
 8 files changed, 200 insertions(+)
 create mode 100755 tools/testing/selftests/resctrl/stray_tasks_repro.sh


base-commit: aebb7274476403040d582c61478df8f4a7bf9cd3
-- 
2.56.0


^ permalink raw reply	[flat|nested] 4+ messages in thread

* [RFC PATCH 1/3] fork,arm,x86,fs/resctrl: Fix stale resource group assignment vs copy_process()
  2026-11-20 18:39 [RFC PATCH 0/3] fork,arm,x86,fs/resctrl: Fix stale resource group assignment on fork Reinette Chatre
@ 2026-11-20 18:39 ` Reinette Chatre
  2026-11-20 18:39 ` [RFC PATCH 2/3] arm,x86/resctrl: Assert tasklist_lock is held when setting CLOSID and RMID Reinette Chatre
  2026-11-20 18:39 ` [RFC PATCH 3/3] NOT_FOR_INCLUSION: selftests/resctrl: Tasks keep using removed RMID after fork() reproducer Reinette Chatre
  2 siblings, 0 replies; 4+ messages in thread
From: Reinette Chatre @ 2026-11-20 18:39 UTC (permalink / raw)
  To: kees, mingo, peterz, juri.lelli, vincent.guittot,
	dietmar.eggemann, rostedt, bsegall, mgorman, vschneid,
	kprateek.nayak, david, ljs, liam, vbabka, rppt, surenb, mhocko,
	bp, tglx, dave.hansen, tony.luck, ben.horgan, Dave.Martin,
	james.morse, babu.moger, fustini, jedrzej.wasiukiewicz
  Cc: fenghuay, peternewman, yu.c.chen, x86, linux-mm, linux-kernel,
	patches, reinette.chatre

During fork a task's CLOSID and RMID are copied into the new task by
dup_task_struct(), early in copy_process() without holding tasklist_lock
or any lock that resctrl takes. The new task becomes visible to a walk of
the task list later in copy_process(), after it is linked onto the task
list while the tasklist_lock is held for write.

Removing a resource group moves all of its tasks to another group by
walking the task list while holding the tasklist_lock for read, and then
frees the CLOSID and RMID of the removed group. A task that copies its
parent's CLOSID and RMID before the walk reaches the parent is missed by
the walk: it is not on the task list yet, and it cannot get there while
the walk holds the tasklist_lock. By the time it is added to the task list
the CLOSID and RMID it was initialized with have been freed.

    CPU0                                CPU1
    ====                                ====

  fork()
    copy_process()
      dup_task_struct()
        /*
         * New task inherits the
         * parent's CLOSID and RMID.
         */
                                    rdtgroup_rmdir_{ctrl,mon}()
                                      rdt_move_group_tasks()
                                        read_lock(&tasklist_lock)
                                        for_each_process_thread()
                                          /*
                                           * Parent is moved out of
                                           * the group. New task is
                                           * not on the task list
                                           * and is not moved.
                                           */
      write_lock_irq(&tasklist_lock)
        /* Waits for the walk. */
                                        read_unlock(&tasklist_lock)
      /* Lock acquired. */
      list_add_tail_rcu(&p->tasks, ...)
      write_unlock_irq(&tasklist_lock)
                                      free_rmid()
                                      closid_free() /* ctrl only */

    New task and every task it creates use the freed CLOSID and RMID.
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

The new task and every task it goes on to create keep using a CLOSID
and RMID that resctrl believes are unused. Once those identifiers are
handed to a resource group created later, the strays appear in that
group's tasks file, their memory accesses are counted in its monitoring
data, and they are subject to its allocation policy, while the group
intended for them does not control or monitor them.

Introduce resctrl_copy_process() to copy the parent's CLOSID and RMID
into the new task from copy_process(), while the tasklist_lock is held
for write. The walk holds the same lock for read for its full duration,
so the new task is either already on the task list and assigned by the
walk, or the copy happens after the walk completed and observes the
parent's new resource group.

resctrl_copy_process() reads two values that some architectures store
separately. __rdtgroup_move_task() assigns them with two stores while
holding only rdtgroup_mutex, which resctrl_copy_process() cannot take,
so those reads can fall between the two stores and leave the new task
with the CLOSID of one resource group and the RMID of another. Assign
them from __rdtgroup_move_task() while holding tasklist_lock for read
as well, so that resctrl_copy_process() observes both as intended.

Add an architecture call to perform the copy because CLOSID and RMID are
stored per architecture and cannot be read and written as independent
values on all of them.

Fixes: f3cbeacaa06e ("x86/intel_rdt/cqm: Add rmdir support")
Reported-by: Jedrzej Wasiukiewicz <jedrzej.wasiukiewicz@intel.com>
Signed-off-by: Reinette Chatre <reinette.chatre@intel.com>
---
Note to MPAM maintainers:

resctrl_copy_process() runs on every fork(), so
resctrl_arch_copy_closid_rmid() is a static inline on x86. Doing so is
perhaps not required as fork() is already expensive.

Even so, I attempted doing same for MPAM but ended up with solutions
that do not seem ideal:

- A static inline in arch/arm64/include/asm/mpam.h, next to the other
  accessors of thread_info::mpam_partid_pmg. Nothing on the path from
  include/linux/resctrl.h reaches that header, so the definition is not
  visible to resctrl_copy_process(). Making it visible means
  arch/arm64/include/asm/resctrl.h has to include asm/mpam.h, which
  does not seem to match MPAM's intended design?

- A static inline in include/linux/arm_mpam.h, beside the
  resctrl_arch_*() inlines already there. asm/mpam.h includes
  linux/arm_mpam.h, so the reverse include would be circular and
  mpam_get_regval() cannot be called. The copy would have to read and
  write thread_info::mpam_partid_pmg directly, which no code outside
  asm/mpam.h does today, and would need a CONFIG_ARM64_MPAM guard
  because the field is only present when it is set.

The out-of-line definition this ended up with matches
resctrl_arch_sched_in(). Please let me know (if this is a change you
agree with) what kind of shape it needs to take for MPAM. We could also
just make x86 and MPAM both be out-of-line - doing so may not be felt by
the heavy copy_process().
---
 arch/arm64/include/asm/mpam.h  |  6 ++++++
 arch/x86/include/asm/resctrl.h |  7 +++++++
 drivers/resctrl/mpam_resctrl.c |  6 ++++++
 fs/resctrl/rdtgroup.c          | 21 +++++++++++++++++++++
 include/linux/arm_mpam.h       |  1 +
 include/linux/resctrl.h        | 28 ++++++++++++++++++++++++++++
 kernel/fork.c                  |  3 +++
 7 files changed, 72 insertions(+)

diff --git a/arch/arm64/include/asm/mpam.h b/arch/arm64/include/asm/mpam.h
index 70d396e7b6da..0746e5d7cc2d 100644
--- a/arch/arm64/include/asm/mpam.h
+++ b/arch/arm64/include/asm/mpam.h
@@ -63,6 +63,12 @@ static inline void mpam_set_task_partid_pmg(struct task_struct *tsk,
 	WRITE_ONCE(task_thread_info(tsk)->mpam_partid_pmg, regval);
 }
 
+static inline void mpam_copy_task_partid_pmg(struct task_struct *dst,
+					     struct task_struct *src)
+{
+	WRITE_ONCE(task_thread_info(dst)->mpam_partid_pmg, mpam_get_regval(src));
+}
+
 static inline void mpam_thread_switch(struct task_struct *tsk)
 {
 	u64 oldregval;
diff --git a/arch/x86/include/asm/resctrl.h b/arch/x86/include/asm/resctrl.h
index 8f6edcdcfd87..199ca9af0089 100644
--- a/arch/x86/include/asm/resctrl.h
+++ b/arch/x86/include/asm/resctrl.h
@@ -153,6 +153,13 @@ static inline void resctrl_arch_set_closid_rmid(struct task_struct *tsk,
 	WRITE_ONCE(tsk->rmid, rmid);
 }
 
+static inline void resctrl_arch_copy_closid_rmid(struct task_struct *dst,
+						 struct task_struct *src)
+{
+	WRITE_ONCE(dst->closid, READ_ONCE(src->closid));
+	WRITE_ONCE(dst->rmid, READ_ONCE(src->rmid));
+}
+
 static inline bool resctrl_arch_match_closid(struct task_struct *tsk, u32 closid)
 {
 	return READ_ONCE(tsk->closid) == closid;
diff --git a/drivers/resctrl/mpam_resctrl.c b/drivers/resctrl/mpam_resctrl.c
index 5468367776e2..7aea58fca85b 100644
--- a/drivers/resctrl/mpam_resctrl.c
+++ b/drivers/resctrl/mpam_resctrl.c
@@ -345,6 +345,12 @@ void resctrl_arch_set_closid_rmid(struct task_struct *tsk, u32 closid, u32 rmid)
 	}
 }
 
+void resctrl_arch_copy_closid_rmid(struct task_struct *dst,
+				   struct task_struct *src)
+{
+	mpam_copy_task_partid_pmg(dst, src);
+}
+
 bool resctrl_arch_match_closid(struct task_struct *tsk, u32 closid)
 {
 	u64 regval = mpam_get_regval(tsk);
diff --git a/fs/resctrl/rdtgroup.c b/fs/resctrl/rdtgroup.c
index 57ab090072c2..ebf780fc5fe9 100644
--- a/fs/resctrl/rdtgroup.c
+++ b/fs/resctrl/rdtgroup.c
@@ -657,12 +657,28 @@ static int __rdtgroup_move_task(struct task_struct *tsk,
 		return -EINVAL;
 	}
 
+	/*
+	 * Hold tasklist_lock for read so that resctrl_copy_process(), which
+	 * holds it for write, cannot read the CLOSID and the RMID of @tsk
+	 * while only one of the two has been updated.
+	 *
+	 * Hold tasklist_lock just for the @tsk update. A CPU running @tsk
+	 * keeps using the CLOSID and RMID that resctrl_arch_sched_in()
+	 * applied when @tsk was last scheduled in, so
+	 * update_task_closid_rmid() below has to interrupt that CPU for the
+	 * new resource group to take effect. Leave that until after
+	 * tasklist_lock has been dropped: a CPU waiting for tasklist_lock
+	 * for write waits with interrupts disabled, and @tsk's CPU is the
+	 * one that may be waiting, because @tsk may be forking.
+	 */
+	read_lock(&tasklist_lock);
 	if (rdtgrp->type == RDTMON_GROUP)
 		resctrl_arch_set_closid_rmid(tsk, rdtgrp->mon.parent->closid,
 					     rdtgrp->mon.rmid);
 	else
 		resctrl_arch_set_closid_rmid(tsk, rdtgrp->closid,
 					     rdtgrp->mon.rmid);
+	read_unlock(&tasklist_lock);
 
 	/*
 	 * Ensure the task's closid and rmid are written before determining if
@@ -2959,6 +2975,11 @@ static void schemata_list_destroy(void)
  * If @mask is not NULL the cpus on which moved tasks are running are set
  * in that mask so the update smp function call is restricted to affected
  * cpus.
+ *
+ * Tasks created while this walks the task list are handled by
+ * resctrl_copy_process(), which tasklist_lock serializes against this walk.
+ * A new task is either visible to the walk and moved with the others, or
+ * resctrl_copy_process() assigns it after the walk completes.
  */
 static void rdt_move_group_tasks(struct rdtgroup *from, struct rdtgroup *to,
 				 struct cpumask *mask)
diff --git a/include/linux/arm_mpam.h b/include/linux/arm_mpam.h
index 002f56e15362..b605e9771c32 100644
--- a/include/linux/arm_mpam.h
+++ b/include/linux/arm_mpam.h
@@ -56,6 +56,7 @@ bool resctrl_arch_mon_capable(void);
 void resctrl_arch_set_cpu_default_closid(int cpu, u32 closid);
 void resctrl_arch_set_closid_rmid(struct task_struct *tsk, u32 closid, u32 rmid);
 void resctrl_arch_set_cpu_default_closid_rmid(int cpu, u32 closid, u32 rmid);
+void resctrl_arch_copy_closid_rmid(struct task_struct *dst, struct task_struct *src);
 void resctrl_arch_sched_in(struct task_struct *tsk);
 bool resctrl_arch_match_closid(struct task_struct *tsk, u32 closid);
 bool resctrl_arch_match_rmid(struct task_struct *tsk, u32 closid, u32 rmid);
diff --git a/include/linux/resctrl.h b/include/linux/resctrl.h
index 10dfdca7f4bf..32d8bf9ca9ba 100644
--- a/include/linux/resctrl.h
+++ b/include/linux/resctrl.h
@@ -7,6 +7,8 @@
 #include <linux/list.h>
 #include <linux/pid.h>
 #include <linux/resctrl_types.h>
+#include <linux/sched.h>
+#include <linux/sched/task.h>
 
 #ifdef CONFIG_ARCH_HAS_CPU_RESCTRL
 #include <asm/resctrl.h>
@@ -729,6 +731,32 @@ extern unsigned int resctrl_rmid_realloc_limit;
 int resctrl_init(void);
 void resctrl_exit(void);
 
+#ifdef CONFIG_RESCTRL_FS
+/**
+ * resctrl_copy_process() - Give a new task the resource group of its parent.
+ * @child:	The new task, which is not yet on the task list.
+ *
+ * Call from copy_process() while tasklist_lock is held for write. Serialized
+ * against rdt_move_group_tasks() and __rdtgroup_move_task() that hold
+ * tasklist_lock for read.
+ *
+ * dup_task_struct() copies the parent's CLOSID and RMID into @child before
+ * @child is linked onto the task list. If the parent is moved out of a
+ * resource group that is being removed, that group's CLOSID and RMID are
+ * freed once the walk completes, leaving @child with stale assignments.
+ *
+ * Bring @child up to date with the parent.
+ */
+static inline void resctrl_copy_process(struct task_struct *child)
+{
+	lockdep_assert_held_write(&tasklist_lock);
+
+	resctrl_arch_copy_closid_rmid(child, current);
+}
+#else
+static inline void resctrl_copy_process(struct task_struct *child) { }
+#endif
+
 #ifdef CONFIG_RESCTRL_FS_PSEUDO_LOCK
 u64 resctrl_arch_get_prefetch_disable_bits(void);
 int resctrl_arch_pseudo_lock_fn(void *_plr);
diff --git a/kernel/fork.c b/kernel/fork.c
index 5efceb8d58a7..8344e0d27b78 100644
--- a/kernel/fork.c
+++ b/kernel/fork.c
@@ -105,6 +105,7 @@
 #include <linux/stackprotector.h>
 #include <linux/user_events.h>
 #include <linux/iommu.h>
+#include <linux/resctrl.h>
 #include <linux/rseq.h>
 #include <uapi/linux/pidfd.h>
 #include <linux/pidfs.h>
@@ -2491,6 +2492,8 @@ __latent_entropy struct task_struct *copy_process(
 
 	klp_copy_process(p);
 
+	resctrl_copy_process(p);
+
 	sched_core_fork(p);
 
 	spin_lock(&current->sighand->siglock);
-- 
2.56.0


^ permalink raw reply	[flat|nested] 4+ messages in thread

* [RFC PATCH 2/3] arm,x86/resctrl: Assert tasklist_lock is held when setting CLOSID and RMID
  2026-11-20 18:39 [RFC PATCH 0/3] fork,arm,x86,fs/resctrl: Fix stale resource group assignment on fork Reinette Chatre
  2026-11-20 18:39 ` [RFC PATCH 1/3] fork,arm,x86,fs/resctrl: Fix stale resource group assignment vs copy_process() Reinette Chatre
@ 2026-11-20 18:39 ` Reinette Chatre
  2026-11-20 18:39 ` [RFC PATCH 3/3] NOT_FOR_INCLUSION: selftests/resctrl: Tasks keep using removed RMID after fork() reproducer Reinette Chatre
  2 siblings, 0 replies; 4+ messages in thread
From: Reinette Chatre @ 2026-11-20 18:39 UTC (permalink / raw)
  To: kees, mingo, peterz, juri.lelli, vincent.guittot,
	dietmar.eggemann, rostedt, bsegall, mgorman, vschneid,
	kprateek.nayak, david, ljs, liam, vbabka, rppt, surenb, mhocko,
	bp, tglx, dave.hansen, tony.luck, ben.horgan, Dave.Martin,
	james.morse, babu.moger, fustini, jedrzej.wasiukiewicz
  Cc: fenghuay, peternewman, yu.c.chen, x86, linux-mm, linux-kernel,
	patches, reinette.chatre

resctrl_arch_set_closid_rmid() assigns a task's CLOSID and RMID. All of
its callers hold tasklist_lock for read, which is what lets
resctrl_copy_process() read both values as intended.

Each architecture declares resctrl_arch_set_closid_rmid() itself, so
there is no shared declaration in which to record that requirement.
Record it in the implementations instead.

Signed-off-by: Reinette Chatre <reinette.chatre@intel.com>
---
 arch/x86/include/asm/resctrl.h | 4 ++++
 drivers/resctrl/mpam_resctrl.c | 2 ++
 2 files changed, 6 insertions(+)

diff --git a/arch/x86/include/asm/resctrl.h b/arch/x86/include/asm/resctrl.h
index 199ca9af0089..48bf22fb6b47 100644
--- a/arch/x86/include/asm/resctrl.h
+++ b/arch/x86/include/asm/resctrl.h
@@ -5,9 +5,11 @@
 #ifdef CONFIG_X86_CPU_RESCTRL
 
 #include <linux/jump_label.h>
+#include <linux/lockdep.h>
 #include <linux/percpu.h>
 #include <linux/resctrl_types.h>
 #include <linux/sched.h>
+#include <linux/sched/task.h>
 
 #include <asm/msr.h>
 
@@ -149,6 +151,8 @@ static inline void resctrl_arch_set_cpu_default_closid_rmid(int cpu, u32 closid,
 static inline void resctrl_arch_set_closid_rmid(struct task_struct *tsk,
 						u32 closid, u32 rmid)
 {
+	lockdep_assert_held(&tasklist_lock);
+
 	WRITE_ONCE(tsk->closid, closid);
 	WRITE_ONCE(tsk->rmid, rmid);
 }
diff --git a/drivers/resctrl/mpam_resctrl.c b/drivers/resctrl/mpam_resctrl.c
index 7aea58fca85b..4e6ffe273863 100644
--- a/drivers/resctrl/mpam_resctrl.c
+++ b/drivers/resctrl/mpam_resctrl.c
@@ -332,6 +332,8 @@ void resctrl_arch_sync_cpu_closid_rmid(void *info)
 
 void resctrl_arch_set_closid_rmid(struct task_struct *tsk, u32 closid, u32 rmid)
 {
+	lockdep_assert_held(&tasklist_lock);
+
 	WARN_ON_ONCE(closid > U16_MAX);
 	WARN_ON_ONCE(rmid > U8_MAX);
 
-- 
2.56.0


^ permalink raw reply	[flat|nested] 4+ messages in thread

* [RFC PATCH 3/3] NOT_FOR_INCLUSION: selftests/resctrl: Tasks keep using removed RMID after fork() reproducer
  2026-11-20 18:39 [RFC PATCH 0/3] fork,arm,x86,fs/resctrl: Fix stale resource group assignment on fork Reinette Chatre
  2026-11-20 18:39 ` [RFC PATCH 1/3] fork,arm,x86,fs/resctrl: Fix stale resource group assignment vs copy_process() Reinette Chatre
  2026-11-20 18:39 ` [RFC PATCH 2/3] arm,x86/resctrl: Assert tasklist_lock is held when setting CLOSID and RMID Reinette Chatre
@ 2026-11-20 18:39 ` Reinette Chatre
  2 siblings, 0 replies; 4+ messages in thread
From: Reinette Chatre @ 2026-11-20 18:39 UTC (permalink / raw)
  To: kees, mingo, peterz, juri.lelli, vincent.guittot,
	dietmar.eggemann, rostedt, bsegall, mgorman, vschneid,
	kprateek.nayak, david, ljs, liam, vbabka, rppt, surenb, mhocko,
	bp, tglx, dave.hansen, tony.luck, ben.horgan, Dave.Martin,
	james.morse, babu.moger, fustini, jedrzej.wasiukiewicz
  Cc: fenghuay, peternewman, yu.c.chen, x86, linux-mm, linux-kernel,
	patches, reinette.chatre

From: Jedrzej Wasiukiewicz <jedrzej.wasiukiewicz@intel.com>

Introduce stray_tasks_repro.sh that demonstrates how a task forked while
its monitor group is being removed may continue to use the removed RMID.

What this script does:
  1. Starts 8 processes inside a new monitor group "m" that fork sleeping
     children as fast as they can.
  2. Removes "m" while they are forking.
  3. Creates new monitor groups until no RMID is free, so the RMID freed in
     step 2 gets a new owner. Nothing is ever moved into these groups.
  4. Reports every task found in them. Each one carries the removed group's
     RMID and is a stray. The expected result is none.

To avoid the freed RMID landing in limbo, the script requires that LLC
occupancy monitoring is disabled (on x86, boot with "rdt=!cmt").

Example of issue encountered:

  monitor groups created: 566, stopped on: No space left on device
  tasks in new monitor groups (expect none): 1
  /sys/fs/resctrl/c/mon_groups/n565/tasks
  RESULT: 1 stray task(s) kept the removed group's RMID

See script header for more detail.

Signed-off-by: Jedrzej Wasiukiewicz <jedrzej.wasiukiewicz@intel.com>
Signed-off-by: Reinette Chatre <reinette.chatre@intel.com>
---
 .../selftests/resctrl/stray_tasks_repro.sh    | 122 ++++++++++++++++++
 1 file changed, 122 insertions(+)
 create mode 100755 tools/testing/selftests/resctrl/stray_tasks_repro.sh

diff --git a/tools/testing/selftests/resctrl/stray_tasks_repro.sh b/tools/testing/selftests/resctrl/stray_tasks_repro.sh
new file mode 100755
index 000000000000..42f65d7d7bb4
--- /dev/null
+++ b/tools/testing/selftests/resctrl/stray_tasks_repro.sh
@@ -0,0 +1,122 @@
+#!/bin/bash
+# SPDX-License-Identifier: GPL-2.0-only
+#
+# resctrl: a task forked while its monitor group is being removed keeps the
+# removed group's RMID.
+#
+# rdt_move_group_tasks() moves a removed group's tasks to the parent group
+# under read_lock(&tasklist_lock). That lock only delays a new child from
+# joining the task list. The child copies its parent's closid/rmid earlier,
+# in dup_task_struct(), without the lock. A child whose fork() is in flight
+# when the walk starts therefore joins the list afterwards with the removed
+# group's RMID, and is never moved.
+#
+# What this script does:
+#   1. Starts 8 processes inside a new monitor group "m" that fork sleeping
+#      children as fast as they can.
+#   2. Removes "m" while they are forking.
+#   3. Creates new monitor groups until no RMID is free, so the RMID freed in
+#      step 2 gets a new owner. Nothing is ever moved into these groups.
+#   4. Reports every task found in them. Each one carries the removed group's
+#      RMID and is a stray. The expected result is none.
+#
+# Requirements: bash, root, resctrl mounted with monitoring, and the L3
+# occupancy event (llc_occupancy) off. Memory bandwidth events (mbm_*) are
+# enough.
+#   mount -t resctrl resctrl /sys/fs/resctrl
+# With llc_occupancy a freed RMID waits in "limbo" until the cache lines tagged
+# with it are gone, so it may not be reused and the race cannot be seen. Without
+# that event a freed RMID is reusable at once. To turn the event off, boot with
+# the kernel parameter rdt=!cmt. The script refuses to run when llc_occupancy is
+# listed in info/L3_MON/mon_features.
+#
+# Output (example of a hit):
+#   monitor groups created: 566, stopped on: No space left on device
+#   tasks in new monitor groups (expect none): 1
+#   /sys/fs/resctrl/c/mon_groups/n565/tasks
+#   RESULT: 1 stray task(s) kept the removed group's RMID
+#
+# The race is probabilistic: run it several times. "no stray this run" is not
+# a pass. If the run stops on anything but "No space left on device", the
+# freed RMID was never reused, so strays cannot be seen. The script then
+# reports RESULT: INCONCLUSIVE and the run counts as no data.
+#
+# Side effects: creates and removes the control group "c" and its monitor
+# groups (it refuses to run if "c" exists) and starts thousands of short-lived
+# sleeping processes. Other resctrl groups are not touched, but while the RMID
+# pool is full anyone else creating a monitor group gets ENOSPC.
+
+R=/sys/fs/resctrl
+C=$R/c                      # control group used for the test
+M=$C/mon_groups/m           # monitor group that gets removed
+MF=$R/info/L3_MON/mon_features
+
+[ "$(id -u)" = 0 ] || { echo "run as root"; exit 1; }
+[ -f $MF ] || { echo "resctrl with monitoring is not mounted at $R"; exit 1; }
+if grep -q llc_occupancy $MF; then
+	echo "llc_occupancy is enabled: freed RMIDs can sit in limbo."
+	echo "Boot with the kernel parameter rdt=!cmt."
+	exit 1
+fi
+[ ! -e $C ] || { echo "$C already exists, remove it first"; exit 1; }
+
+FORKERS=()                  # PIDs of the forker processes
+KIDS=""                     # PIDs of their children
+
+# Runs on every exit, including Ctrl-C: stop all test processes and remove the
+# test groups.
+cleanup() {
+	trap - EXIT INT TERM
+	[ -n "$KIDS" ] || [ ${#FORKERS[@]} -eq 0 ] ||
+		KIDS=$(pgrep -P "$(IFS=,; echo "${FORKERS[*]}")")
+	[ ${#FORKERS[@]} -eq 0 ] || kill "${FORKERS[@]}" 2>/dev/null
+	[ -z "$KIDS" ] || kill $KIDS 2>/dev/null
+	sleep 1
+	rmdir $C/mon_groups/* 2>/dev/null
+	rmdir $C 2>/dev/null
+}
+trap cleanup EXIT
+trap 'exit 130' INT TERM
+
+mkdir $C $M || { echo "cannot create $C and $M"; exit 1; }
+
+# 8 forkers. Each joins c, then m, and then forks in a loop: a sleeping child
+# that lives 20 s, plus a very short one that keeps the fork rate high.
+for i in 1 2 3 4 5 6 7 8; do
+	( echo $BASHPID > $C/tasks; echo $BASHPID > $M/tasks
+	  while :; do sleep 20 & sleep 0.0003; done ) &
+	FORKERS+=($!)
+done
+sleep 0.3                   # let them start forking
+
+# The race window is while this walks the task list.
+rmdir $M
+
+# Remember the children for cleanup, then stop the forkers. The children stay.
+KIDS=$(pgrep -P "$(IFS=,; echo "${FORKERS[*]}")")
+kill "${FORKERS[@]}" 2>/dev/null
+sleep 2                     # let forks that are still in flight finish
+
+# Take every free RMID. Stop on the first error, normally ENOSPC (pool full).
+n=0
+while err=$(mkdir $C/mon_groups/n$n 2>&1); do n=$((n+1)); done
+echo "monitor groups created: $n, stopped on: ${err##*: }"
+
+# Nothing was ever put into n*, so any task listed here is a stray.
+strays=$(cat $C/mon_groups/n*/tasks 2>/dev/null | wc -l)
+echo "tasks in new monitor groups (expect none): $strays"
+if [ "$strays" -gt 0 ]; then
+	grep -l . $C/mon_groups/n*/tasks        # which group holds them
+	echo "RESULT: $strays stray task(s) kept the removed group's RMID"
+	exit 0
+fi
+
+case $err in
+*"No space left"*)
+	echo "RESULT: no stray this run, run again"
+	exit 0 ;;
+esac
+
+# Not ENOSPC: the pool was never full, so the freed RMID was not reused and a
+# stray cannot be seen in this run.
+echo "RESULT: INCONCLUSIVE, the freed RMID was never reused (${err##*: })"
-- 
2.56.0


^ permalink raw reply	[flat|nested] 4+ messages in thread

end of thread, other threads:[~2026-11-20 18:39 UTC | newest]

Thread overview: 4+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2026-11-20 18:39 [RFC PATCH 0/3] fork,arm,x86,fs/resctrl: Fix stale resource group assignment on fork Reinette Chatre
2026-11-20 18:39 ` [RFC PATCH 1/3] fork,arm,x86,fs/resctrl: Fix stale resource group assignment vs copy_process() Reinette Chatre
2026-11-20 18:39 ` [RFC PATCH 2/3] arm,x86/resctrl: Assert tasklist_lock is held when setting CLOSID and RMID Reinette Chatre
2026-11-20 18:39 ` [RFC PATCH 3/3] NOT_FOR_INCLUSION: selftests/resctrl: Tasks keep using removed RMID after fork() reproducer Reinette Chatre

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox

all inboxes | Powered by JetHome®