[BUG] sched/isolation: suspicious RCU usage with CONFIG_CPUSETS=n

[Mathieu Desnoyers <mathieu.desnoyers@efficios.com>]

Hi,

With CONFIG_CPUSETS=n and isolcpus=domain, a read of
/proc/sys/kernel/sched_rt_runtime_us gives a "suspicious RCU usage"
lockdep report. A write gives the same report. I first saw it while
testing a SCHED_DEADLINE series [1]. I then reproduced it in a VM on tip
sched/urgent (006991aa8997), without the series. A proposed fix follows
the scissors line.

Trigger
=======

Build with CONFIG_PROVE_RCU=y and CONFIG_CPUSETS=n. Boot 5 CPUs with
isolcpus=domain,1,2. Then:

  cat /proc/sys/kernel/sched_rt_runtime_us

isolcpus=1,2 gives the same report, since the domain flag is the
default. Without isolcpus, there is no report.

Log
===

  =============================
  WARNING: suspicious RCU usage
  7.4.0-rc1-00267-g006991aa8997 #1 Not tainted
  -----------------------------
  kernel/sched/isolation.c:62 suspicious rcu_dereference_check() usage!
  other info that might help us debug this:
  rcu_scheduler_active = 2, debug_locks = 1
  locks held by cat/91: 3, last CPU#3:
   #0: ff1100000613e260 (&pipe->mutex){+.+.}-{4:4}, at: splice_file_to_pipe+0x23/0xd0
   #1: ffffffff8306c930 (cpu_hotplug_lock){++++}-{0:0}, at: sched_rt_handler+0xb5/0xf0
   #2: ffffffff8307aac0 (sched_domains_mutex){+.+.}-{4:4}, at: partition_sched_domains+0x37/0x990
  stack backtrace:
  CPU: 3 UID: 0 PID: 91 Comm: cat Not tainted 7.4.0-rc1-00267-g006991aa8997 #1 PREEMPT(lazy)
  Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.3-debian-1.16.3-2 04/01/2014
  Call Trace:
   <TASK>
   dump_stack_lvl+0x5d/0x80
   lockdep_rcu_suspicious.cold+0x4e/0x9f
   housekeeping_cpumask+0xaa/0xb0
   partition_sched_domains+0x8e6/0x990
   sched_rt_handler+0xc3/0xf0
   proc_sys_call_handler+0x1cf/0x250
   copy_splice_read+0x217/0x3b0
   splice_file_to_pipe+0x60/0xd0
   do_sendfile+0x161/0x480
   __x64_sys_sendfile64+0xba/0xd0
   do_syscall_64+0x109/0x5d0
   entry_SYSCALL_64_after_hwframe+0x77/0x7f
   </TASK>

Analysis
========

sched_rt_handler() calls rebuild_sched_domains() after each read or
write of sched_rt_period_us and sched_rt_runtime_us. Without
CONFIG_CPUSETS, rebuild_sched_domains() is this stub in
include/linux/cpuset.h:

	static inline void rebuild_sched_domains(void)
	{
		guard(cpus_read_lock)();
		partition_sched_domains(1, NULL, NULL);
	}

With no doms_new, partition_sched_domains_locked() builds one domain
from the housekeeping mask:

	cpumask_and(doms_new[0], cpu_active_mask,
		    housekeeping_cpumask(HK_TYPE_DOMAIN));

housekeeping_cpumask() reads housekeeping.cpumasks[HK_TYPE_DOMAIN] with
rcu_dereference_all_check(). Its condition is
housekeeping_dereference_check():

	/* Cpuset isn't even writable yet? */
	if (system_state <= SYSTEM_SCHEDULING)
		return true;

	/* CPU hotplug write locked, so cpuset partition can't be overwritten */
	if (IS_ENABLED(CONFIG_HOTPLUG_CPU) && lockdep_is_cpus_write_held())
		return true;

	/* Cpuset lock held, partitions not writable */
	if (IS_ENABLED(CONFIG_CPUSETS) && lockdep_is_cpuset_held())
		return true;

	return false;

The caller holds cpus_read_lock(), not the write lock. It holds no RCU
read lock. The cpuset case is compiled out. So the check fails, and
lockdep warns.

The report is a false positive. Only cpuset calls housekeeping_update().
Without CONFIG_CPUSETS, housekeeping_init() sets the pointer at boot,
and nothing changes it after that. Without isolcpus, the domain flag is
not set. housekeeping_cpumask() then returns cpu_possible_mask without
reading the pointer.

Versions:
- The check comes from 03ff73510169 ("cpuset: Update HK_TYPE_DOMAIN
  cpumask from cpuset"), in v7.0. It is the same in tip sched/urgent.
- 666a32836c8f ("sched/topology: Add a cpus_read_lock to
  rebuild_sched_domains()"), in v7.3, fixed another warning on the same
  read path. Before it, the stub held no lock at all. So the check also
  fails from v7.0 to v7.2. This is from reading.

Other callers of rebuild_sched_domains() would warn the same way with
CONFIG_CPUSETS=n. Examples are the x86 sched_itmt_enabled debugfs file
and rebuild_sched_domains_energy(). This is from reading. I did not
run them.

Where to fix
============

The patch adds one case to housekeeping_dereference_check(). It accepts
all readers when CONFIG_CPUSETS is off. This covers every caller of the
stub. With CONFIG_CPUSETS=y, the new case is compiled out.

Another option is an RCU read-side section around the two reads in
partition_sched_domains_locked(). It would fix this path only.

Testing
=======

I ran tip sched/urgent (006991aa8997) in a VM with 5 CPUs. The kernel
had CONFIG_PROVE_RCU=y and CONFIG_CPUSETS=n. Without the patch:
- isolcpus=domain,1,2: one report from a read alone. In another boot,
  one report from a write alone.
- isolcpus=1,2: one report from a read.
- No isolcpus: no report from reads and writes.

With the patch, there is no report:
- isolcpus=domain,1,2: a read alone. In another boot, a write alone.
  In a third boot, reads and writes of sched_rt_runtime_us and
  sched_rt_period_us.
- isolcpus=1,2: reads and writes.
- No isolcpus: reads and writes.

debug_locks was still 1 at the end of each boot. So lockdep was still
checking.

With CONFIG_CPUSETS=y, the patch also builds. A boot with
isolcpus=domain,1,2 and the same reads and writes gives no report. I
did not run other tests.

An open question, from reading only. Since a84097e625f2
("cgroup/cpuset: Call housekeeping_update() without holding
cpus_read_lock"), cpuset calls housekeeping_update() with only
cpuset_top_mutex held. The check accepts the hotplug write lock or
cpuset_mutex. Do these two locks still keep the mask from changing
under a reader? I did not look at a reader that holds only
cpuset_mutex.

The analysis and the patch were written by an LLM coding assistant
under my direction and review.

Thanks,

Mathieu

[1] https://lore.kernel.org/all/20261107014451.1487825-1-mathieu.desnoyers@efficios.com/

-- >8 --
From: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Subject: sched/isolation: Fix suspicious RCU usage without CONFIG_CPUSETS

Since commit 03ff73510169 ("cpuset: Update HK_TYPE_DOMAIN cpumask from
cpuset"), cpuset can change the HK_TYPE_DOMAIN mask at run time. Readers
use RCU. housekeeping_dereference_check() also accepts readers that hold
a lock which keeps cpuset partitions from changing: the CPU hotplug
write lock or cpuset_mutex. During boot, before cpuset is writable, it
accepts all readers.

Without CONFIG_CPUSETS, nothing changes the mask after boot. But the
check still asks for one of these locks. In that configuration,
rebuild_sched_domains() holds only cpus_read_lock(). It calls
partition_sched_domains(1, NULL, NULL). That reads
housekeeping_cpumask(HK_TYPE_DOMAIN). sched_rt_handler() calls
rebuild_sched_domains() on each read or write of sched_rt_period_us and
sched_rt_runtime_us. So with isolcpus=domain, a read of
/proc/sys/kernel/sched_rt_runtime_us gives:

  WARNING: suspicious RCU usage
  kernel/sched/isolation.c:62 suspicious rcu_dereference_check() usage!
  [...]
  locks held by cat/91: 3, last CPU#3:
   [...]
   #1: (cpu_hotplug_lock){++++}-{0:0}, at: sched_rt_handler+0xb5/0xf0
   #2: (sched_domains_mutex){+.+.}-{4:4}, at: partition_sched_domains+0x37/0x990
  [...]
  Call Trace:
   lockdep_rcu_suspicious.cold+0x4e/0x9f
   housekeeping_cpumask+0xaa/0xb0
   partition_sched_domains+0x8e6/0x990
   sched_rt_handler+0xc3/0xf0
   proc_sys_call_handler+0x1cf/0x250

The report is a false positive. It needs CONFIG_PROVE_RCU and isolcpus=
with the domain flag, which is also the default when no flag is given.
Test runs that check the kernel log for lockdep reports flag it.

Fix this by accepting all readers when CONFIG_CPUSETS is off.

This change was developed with assistance from an LLM.

Fixes: 03ff73510169 ("cpuset: Update HK_TYPE_DOMAIN cpumask from cpuset")
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
---
 kernel/sched/isolation.c | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/kernel/sched/isolation.c b/kernel/sched/isolation.c
index 156025ef81b7..dac9a1735198 100644
--- a/kernel/sched/isolation.c
+++ b/kernel/sched/isolation.c
@@ -43,6 +43,10 @@ static bool housekeeping_dereference_check(enum hk_type type)
 		if (system_state <= SYSTEM_SCHEDULING)
 			return true;
 
+		/* No cpuset, so the mask never changes after boot */
+		if (!IS_ENABLED(CONFIG_CPUSETS))
+			return true;
+
 		/* CPU hotplug write locked, so cpuset partition can't be overwritten */
 		if (IS_ENABLED(CONFIG_HOTPLUG_CPU) && lockdep_is_cpus_write_held())
 			return true;