[PATCH 0/6] mm, sched, cgroup: LRU Cache never drains when isolcpus are used with spinning SCHED_FIFO tasks causing page migration stalls

[PATCH 0/6] mm, sched, cgroup: LRU Cache never drains when isolcpus are used with spinning SCHED_FIFO tasks causing page migration stalls

[Matthew Ruffell]

Dear maintainers of the scheduler, memory management, and cgroup subsystems,

A user, Caron Wills, ran into this problem in production and authored
the patch series, which we have iterated on, tested, and verified on bare
metal with the included reproducer together.

Caron Wills has been running into a problem where their low latency 
workloads are causing systemd / PID 1 to be stuck in D state for extended
periods of time, up to 10 minutes in their production environment, and I have
seen 3-5 minutes in testing my synthetic reproducers. This prevents user 
sessions from being created / ssh logins fail / and system services cannot be
restarted or containers cannot be killed or launched.

Downstream bug:
https://bugs.launchpad.net/ubuntu/+source/linux-aws/+bug/2165410

The user's system uses isolcpus on a large system with 192 CPUs, with 2x
AMD EPYC 9R45 CPUs across 2 NUMA nodes. 

The cpus are pinned like so:

isolcpus=nohz,18-95,108-178 nohz_full=18-95,108-178 rcu_nocbs=18-95,108-178 
rcu_nocb_poll skew_tick=1 nosmt=force idle=poll numa_balancing=disable 
transparent_hugepage=never audit=0 nmi_watchdog=0 nowatchdog mce=ignore_ce 
tsc=reliable clocksource=tsc

Each of the isolated cpus are running CPU intensive workloads, spinning at
100% as SCHED_FIFO tasks at priority 80.

The issue happens when a significant amount of pages are migrated between NUMA
nodes. For example, 50GB of pages being migrated from node 1 to node 0.

Systemd enters D state for 3-5+ minutes with the following stack:

kernel: INFO: task systemd:1 blocked for more than 122 seconds.
kernel:       Tainted: G           OE       7.0.0-1004-aws #4-Ubuntu
kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
kernel: task:systemd         state:D stack:0     pid:1     tgid:1     ppid:0      task_flags:0x40     0100 flags:0x00080000
kernel: Call Trace:
kernel:  <TASK>
kernel:  __schedule+0x2b2/0x630
kernel:  schedule+0x27/0x90
kernel:  schedule_timeout+0xcf/0x110
kernel:  wait_for_completion+0x7c/0x140
kernel:  __flush_workqueue+0x14e/0x430
kernel:  ? raw_spin_rq_unlock+0x14/0x50
kernel:  flush_migrate_mm_task_workfn+0x19/0x30
kernel:  task_work_run+0x62/0x90
kernel:  exit_to_user_mode_loop+0x14e/0x500
kernel:  do_syscall_64+0x265/0x5a0
kernel:  ? putname+0x41/0x90
kernel:  ? do_sys_openat2+0x99/0xe0
kernel:  ? __x64_sys_openat+0x5f/0xa0
kernel:  ? __ct_user_enter+0x2c/0x110
kernel:  ? do_syscall_64+0x21b/0x5a0
kernel:  ? __ct_user_enter+0x2c/0x110
kernel:  ? __ct_user_enter+0x2c/0x110
kernel:  ? do_syscall_64+0x21b/0x5a0
kernel:  ? __ct_user_enter+0x2c/0x110
kernel:  ? irqentry_exit+0x17f/0x5a0
kernel:  ? exc_page_fault+0x94/0x1e0
kernel:  entry_SYSCALL_64_after_hwframe+0x76/0x7e
kernel:  </TASK>
kernel: INFO: task kworker/u768:1:1765 blocked for more than 122 seconds.
kernel:       Tainted: G           OE       7.0.0-1004-aws #4-Ubuntu
kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
kernel: task:kworker/u768:1  state:D stack:0     pid:1765  tgid:1765  ppid:2      task_flags:0x42     08060 flags:0x00080000
kernel: Workqueue: cpuset_migrate_mm cpuset_migrate_mm_workfn
kernel: Call Trace:
kernel:  <TASK>
kernel:  __schedule+0x2b2/0x630
kernel:  schedule+0x27/0x90
kernel:  schedule_timeout+0xcf/0x110
kernel:  wait_for_completion+0x7c/0x140
kernel:  __flush_work+0x7d/0xd0
kernel:  ? __pfx_wq_barrier_func+0x10/0x10
kernel:  flush_work+0x10/0x20
kernel:  __lru_add_drain_all+0x19c/0x2c0
kernel:  lru_cache_disable+0x1f/0x30
kernel:  do_migrate_pages+0x4e/0x2b0
kernel:  cpuset_migrate_mm_workfn+0x26/0x50
kernel:  process_one_work+0x1a2/0x380
kernel:  worker_thread+0x1b8/0x360
kernel:  ? _raw_spin_lock_irqsave+0xe/0x20
kernel:  ? __pfx_worker_thread+0x10/0x10
kernel:  kthread+0xf7/0x130
kernel:  ? __pfx_kthread+0x10/0x10
kernel:  ret_from_fork+0x195/0x2a0
kernel:  ? __pfx_kthread+0x10/0x10
kernel:  ? __pfx_kthread+0x10/0x10
kernel:  ret_from_fork_asm+0x1a/0x30
kernel:  </TASK>

What happens is that the spinning SCHED_FIFO processes running on isolcpus are
constantly dirtying the LRU cache, and when we attempt NUMA migration of the
50GB of pages, systemd calls flush_migrate_mm_task_workfn() which creates a
kworker that calls cpuset_migrate_mm_workfn() then do_migrate_pages() then 
lru_cache_disable(), and then __lru_add_drain_all(). __lru_add_drain_all() then
creates a kworker on each CPU to drain the LRU cache, and systemd and the
initial kworker wait until this is done on all CPUs.

Since the isolcpus are running SCHED_FIFO tasks, they never get preempted,
and kworkers never get a chance to run until they get an opportunity to by
returning from a syscall, and it takes a very long time before all 192 CPUs
actually drain their LRU cache before the page migration can continue.

This stalls page migration, systemd gets stuck in D state, and causes all sorts
of problems.

Reproducer:

This was all tested on c8a.metal-48xl on AWS, but this instance type is not
really necessary. You need a large system with 2 NUMA nodes, and many CPUs.
c8a.metal-48xl has 2 NUMA nodes, and 192 cpus, so something similar will be okay.

In this reproducer, CPUs 0-17, 96-107, 179-191 are housekeeping threads, while
18-95 and 108-178 are isolated with isolcpus.

All of this was tested with Ubuntu resolute 26.04 userspace.

```
sudo apt update
sudo apt upgrade
sudo apt install build-essential
```

Setup isolcpus, nohz_full and rcu_nocb by setting the following grub command
line in `/etc/default/grub.d/99-repro.cfg`:

``` 
cat << 'EOF' | sudo tee /etc/default/grub.d/99-repro.cfg
GRUB_CMDLINE_LINUX_DEFAULT="$GRUB_CMDLINE_LINUX_DEFAULT isolcpus=nohz,18-95,108-178 nohz_full=18-95,108-178 rcu_nocbs=18-95,108-178 rcu_nocb_poll skew_tick=1 nosmt=force idle=poll numa_balancing=disable transparent_hugepage=never audit=0 nmi_watchdog=0 nowatchdog mce=ignore_ce tsc=reliable clocksource=tsc"
EOF
```

Run 

```
sudo update-grub
sudo reboot
```

Once the system is up again, check the cmdline:

```
cat /proc/cmdline
```

Save the following to files:

1. Churn Worker (`repro_worker.c`)

This program pins a `SCHED_FIFO/80` thread to an isolated core, continuously 
dirtying and discarding a 24 MB private buffer to generate un-drained local
per-CPU LRU folio batches.

```c
// repro_worker.c
#define _GNU_SOURCE
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>
#include <sched.h>
#include <sys/mman.h>

#define REGION_SIZE (24 * 1024 * 1024) // 24 MB

int main(int argc, char *argv[]) {
    if (argc < 2) {
        fprintf(stderr, "Usage: %s <cpu_id>\n", argv[0]);
        return 1;
    }

    int cpu = atoi(argv[1]);

    // Pin thread to target isolated CPU
    cpu_set_t cpuset;
    CPU_ZERO(&cpuset);
    CPU_SET(cpu, &cpuset);
    if (sched_setaffinity(0, sizeof(cpuset), &cpuset) != 0) {
        perror("sched_setaffinity");
        return 1;
    }

    // Elevate priority to SCHED_FIFO 80
    struct sched_param param = { .sched_priority = 80 };
    if (sched_setscheduler(0, SCHED_FIFO, &param) != 0) {
        perror("sched_setscheduler (run as root)");
        return 1;
    }

    // Allocate 24MB private anonymous region
    char *ptr = mmap(NULL, REGION_SIZE, PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
    if (ptr == MAP_FAILED) {
        perror("mmap");
        return 1;
    }

    // Churn loop: refault pages, then trigger MADV_DONTNEED
    while (1) {
        for (size_t i = 0; i < REGION_SIZE; i += 4096) {
            ptr[i] = (char)i;
        }
        madvise(ptr, REGION_SIZE, MADV_DONTNEED);
    }

    return 0;
}

```

2. Anchor Memory Allocator (`anchor_alloc.c`)

This program allocates and holds 50 GB of memory bound to NUMA node 1 inside
the target cgroup slice.

```c
// anchor_alloc.c
#define _GNU_SOURCE
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>
#include <sys/mman.h>

#define GIB_50 (50ULL * 1024 * 1024 * 1024)
#define PAGE_SIZE 4096

int main() {
    char *ptr = mmap(NULL, GIB_50, PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
    if (ptr == MAP_FAILED) {
        perror("mmap anchor");
        return 1;
    }

    printf("[+] Touching full 50 GB anchor memory (4KB page stride)...\n");
    fflush(stdout);

    // Touch every 4KB page so all 13,107,200 pages fault into physical RAM
    for (size_t i = 0; i < GIB_50; i += PAGE_SIZE) {
        ptr[i] = 1;
    }

    printf("[+] 50 GB anchor memory fully allocated on NUMA Node 1.\n");
    fflush(stdout);

    while (1) {
        sleep(3600);
    }
    return 0;
}

```

3. Orchestration Script (`run_repro.sh`)

This script sets kernel panic thresholds, spawns worker load on isolated CPUs 
`18-95` and `108-178` inside `repro.slice`, allocates node 1 anchor memory, and
fires the `cpuset.mems` flip via systemd.

```bash
#!/usr/bin/env bash
set -euo pipefail

# 1. Compile binaries
gcc -O2 repro_worker.c -o repro_worker
gcc -O2 anchor_alloc.c -o anchor_alloc

# 2. Configure capture sysctls
echo "[+] Configuring panic and hung task sysctls..."
#sysctl -w kernel.hung_task_panic=1
sysctl -w kernel.hung_task_timeout_secs=60
sysctl -w fs.inotify.max_user_watches=524288
sysctl -w fs.inotify.max_user_instances=8192
sysctl -w fs.file-max=2097152
ulimit -n 65536


# 3. Create target systemd slice
SLICE="repro.slice"
echo "[+] Resetting ${SLICE}..."
systemctl stop ${SLICE} 2>/dev/null || true
systemctl reset-failed ${SLICE} 2>/dev/null || true

# Initially assign allowed memory nodes to both 0 and 1
systemctl set-property ${SLICE} AllowedMemoryNodes=0,1

# 4. Generate list of 149 isolated CPUs (18-95 and 108-178)
systemd-run --slice=${SLICE} --scope --unit=repro-workers bash -c '
CPUS=($(seq 18 95) $(seq 108 178))
for cpu in "$${CPUS[@]}"; do
    chrt -f 80 taskset -c "$${cpu}" ./repro_worker "$${cpu}" &
done
wait
' &

sleep 5

# 5. Allocate 50GB anchor memory bound strictly to NUMA Node 1
echo "[+] Spawning 50GB anchor process bound to NUMA Node 1..."
systemd-run --slice=${SLICE} --scope --unit=repro-anchor numactl --membind=1 ./anchor_alloc &

# Wait for anchor allocation to complete
sleep 25

# Verify memory node distribution
echo "[+] Verifying memory location:"
numastat -c anchor_alloc || true

(
    while [ "$(awk '{print $3}' /proc/1/stat)" != "D" ]; do sleep 0.01; done
    t1=$(date +%s.%N)
    echo -e "\n[*] PID 1 entered D state at $(date +%H:%M:%S)..."

    while [ "$(awk '{print $3}' /proc/1/stat)" = "D" ]; do sleep 0.02; done
    t2=$(date +%s.%N)

    awk -v t1="$t1" -v t2="$t2" 'BEGIN {printf "\n[+] PID 1 recovered. Total duration in D state: %.3fs\n", t2 - t1}'
) &
MONITOR_PID=$!

# 6. Trigger cpuset.mems write via systemd (forces cpuset_migrate_mm_workfn)
echo "[!] FIRING TRIGGER: Restricting ${SLICE} AllowedMemoryNodes to Node 0..."
systemctl set-property ${SLICE} AllowedMemoryNodes=0 2>/dev/null || true

echo "[*] Trigger sent. PID 1 should block in task_work -> lru_cache_disable()."

# Wait for the background timer to complete measurement
wait "${MONITOR_PID}"
```

Instructions

1. Log in as `root`.
2. Save the files to a local directory.
3. Make the script executable and run:
```bash
chmod +x run_repro.sh
./run_repro.sh
```

The reproducer script attempts to migrate the 50GB of pages from NUMA node 1
to node 0, which stalls on draining the LRU cache, and systemd hangs for 3-5 
minutes.

The following patches fix this problem by:

1) Adding HK_TYPE_KERNEL_NOISE back to the test in cpu_is_isolated().
2) Keeping unbound kthreads off nohz_full CPUs when fetching kthread affinity.
3) Bypassing per-CPU LRU/mlock batching on isolated CPUs, committing folios
   directly to the LRU lists instead.
4) Draining per-CPU LRU batches when CPUs become isolated at runtime.
5) Allowing cpuset.mems writes opened with O_NONBLOCK to migrate memory
   asynchronously without blocking the writer in task_work.
6) Dropping the redundant workqueue flush in cpuset_fork() so forking tasks
   do not block on asynchronous memory migrations.

Please have a look, and please let me know if you have any questions or need
any more information.

Thanks,
Matthew

Caron Wills (6):
  sched/isolation: Consider nohz_full CPUs isolated in cpu_is_isolated()
    again
  kthread: Keep unbound kthreads off nohz_full CPUs
  mm/folio, mm/mlock: Don't batch folios in per-CPU LRU caches on
    isolated CPUs
  sched/isolation: Drain per-CPU LRU batches when CPUs become isolated
  cgroup/cpuset: Don't wait for memory migration when cpuset.mems is
    written with O_NONBLOCK
  cgroup/cpuset: Don't make a forking task wait for memory migration

 include/linux/sched/isolation.h | 10 ++++++-
 kernel/cgroup/cpuset.c          | 50 +++++++++++++++++++++++----------
 kernel/kthread.c                | 30 +++++++++++++++++---
 kernel/sched/isolation.c        |  8 ++++++
 mm/folio.c                      | 21 +++++++++++++-
 mm/internal.h                   | 28 ++++++++++++++++++
 mm/mlock.c                      | 28 ++++++++++++++----
 7 files changed, 148 insertions(+), 27 deletions(-)

-- 
2.55.0

[PATCH 1/6] sched/isolation: Consider nohz_full CPUs isolated in cpu_is_isolated() again

[Matthew Ruffell]

From: Caron Wills <caron.wills@pulsar.com>

Commit 1c3329df409b ("sched/isolation: Remove HK_TYPE_TICK test from
cpu_is_isolated()") reduced cpu_is_isolated() to a HK_TYPE_DOMAIN test on
the assumption that "it doesn't make sense to use nohz_full without also
isolating the related CPUs from the domain topology", i.e. that every
nohz_full CPU is also domain isolated.

The kernel does not enforce that assumption. "nohz_full=<list>" and
"isolcpus=nohz,<list>" are accepted and documented on their own, and
deployments exist that deliberately keep the nohz_full CPUs inside the
scheduler domains, for example the AWS c8a.metal-48xl systems of
LP: #2165410, booted with

  isolcpus=nohz,18-95,108-178 nohz_full=18-95,108-178 rcu_nocbs=18-95,108-178

On such a system cpu_is_isolated() is now false for every CPU and all of
its users silently lost their isolation awareness compared to v6.8, which
also tested HK_TYPE_TICK:

  - fs/buffer.c bh_lru_install() caches buffer_heads on the nohz_full
    CPUs again, so has_bh_in_lru() makes __lru_add_drain_all() queue
    drain work on them;
  - mm/vmstat.c vmstat_shepherd() queues vmstat_update on them again;
  - mm/memcontrol.c schedule_drain_work() queues memcg stock drains on
    them again;
  - block/blk-mq.c keeps them in hctx->cpumask.

Per-CPU work queued on a nohz_full CPU that runs a SCHED_FIFO busy loop
never executes, and the flush_work() callers hang behind it. This is one
half of the PID 1 D-state stall described in LP: #2165410.

Test HK_TYPE_KERNEL_NOISE (the housekeeping type set by nohz_full= and
isolcpus=nohz, of which HK_TYPE_TICK is an alias) in addition to
HK_TYPE_DOMAIN, and use the housekeeping_cpu() wrapper so that systems
without any CPU isolation keep paying nothing but a patched-out static
branch. On systems where the nohz_full CPUs are also domain isolated
nothing changes.

Fixes: 1c3329df409b ("sched/isolation: Remove HK_TYPE_TICK test from cpu_is_isolated()")
Link: https://bugs.launchpad.net/bugs/2165410
Signed-off-by: Caron Wills <caron.wills@pulsar.com>
Signed-off-by: Matthew Ruffell <matthew.ruffell@canonical.com>
---
 include/linux/sched/isolation.h | 10 +++++++++-
 1 file changed, 9 insertions(+), 1 deletion(-)

diff --git a/include/linux/sched/isolation.h b/include/linux/sched/isolation.h
index cf0fd03dd7a2..294c508b0018 100644
--- a/include/linux/sched/isolation.h
+++ b/include/linux/sched/isolation.h
@@ -84,9 +84,17 @@ static inline bool housekeeping_cpu(int cpu, enum hk_type type)
 	return true;
 }
 
+/*
+ * A CPU is isolated if it has been taken out of the scheduler domains
+ * (isolcpus=domain or a cpuset isolated partition) or out of kernel noise
+ * (nohz_full= or isolcpus=nohz). The latter does not imply the former:
+ * the kernel accepts nohz_full CPUs that are still part of the scheduler
+ * domains, and deferred per-CPU work must not target those CPUs either.
+ */
 static inline bool cpu_is_isolated(int cpu)
 {
-	return !housekeeping_test_cpu(cpu, HK_TYPE_DOMAIN);
+	return !housekeeping_cpu(cpu, HK_TYPE_DOMAIN) ||
+	       !housekeeping_cpu(cpu, HK_TYPE_KERNEL_NOISE);
 }
 
 #endif /* _LINUX_SCHED_ISOLATION_H */
-- 
2.55.0

[PATCH 2/6] kthread: Keep unbound kthreads off nohz_full CPUs

[Matthew Ruffell]

From: Caron Wills <caron.wills@pulsar.com>

Since commit 041ee6f3727a ("kthread: Rely on HK_TYPE_DOMAIN for
preferred affinity management") kthread_fetch_affinity() only removes
domain isolated CPUs from a kthread's affinity, assuming that nohz_full
CPUs are always domain isolated as well. Up to v6.13 every kthread
inherited kthreadd's affinity to housekeeping_cpumask(HK_TYPE_KTHREAD),
and from v6.14 the preferred affinity was filtered with HK_TYPE_KTHREAD,
so nohz_full CPUs were always excluded.

On a system that keeps its nohz_full CPUs in the scheduler domains, e.g.

  isolcpus=nohz,18-95,108-178 nohz_full=18-95,108-178

HK_TYPE_DOMAIN covers every CPU, and the RCU expedited grace period
workers rcu_exp_par_gp_kthread_worker/N, the RCU boost kthreads rcub/N
and every NUMA node affine kthread can now land on nohz_full CPUs.
rcu_thread_affine_rnp() prefers the CPUs of the rcu_node, so for a leaf
rcu_node made entirely of nohz_full CPUs the worker is pinned to nothing
but nohz_full CPUs.

Those workers are SCHED_NORMAL (CONFIG_RCU_BOOST=n leaves kthread_prio
at 0), so a SCHED_FIFO userspace busy loop on such a CPU starves them
indefinitely. Every synchronize_rcu_expedited() in the system then
blocks in sync_rcu_exp_select_cpus() -> kthread_flush_work() until the
FIFO task yields. LP: #2165410 captured exactly that on three 7.0
kernels: the cpuset_migrate_mm kworker sits in

  synchronize_rcu_expedited+0x1b2/0x220
  lru_cache_disable+0x15/0x30
  do_migrate_pages+0x4e/0x2b0
  cpuset_migrate_mm_workfn+0x26/0x50

with rcu_exp_gp_kthread_worker in D state in kthread_flush_work(), and
PID 1 waits behind that migration in flush_workqueue() for minutes.

Intersect the affinity with the HK_TYPE_KERNEL_NOISE housekeeping mask
as well and fall back to the CPUs that are housekeeping for both types.
On systems where nohz_full implies domain isolation the two masks are
identical and nothing changes.

HK_TYPE_KERNEL_NOISE is named explicitly rather than HK_TYPE_KTHREAD:
commit 8f78b749f3da ("sched/isolation: Make HK_TYPE_KTHREAD an alias of
HK_TYPE_DOMAIN") (v7.1-rc3) turned the latter into a synonym of the domain
mask, which is precisely the assumption being corrected here, and made a
first version of this patch a no-op on 7.1+ kernels.

Fixes: 041ee6f3727a ("kthread: Rely on HK_TYPE_DOMAIN for preferred affinity management")
Link: https://bugs.launchpad.net/bugs/2165410
Signed-off-by: Caron Wills <caron.wills@pulsar.com>
Signed-off-by: Matthew Ruffell <matthew.ruffell@canonical.com>
---
 kernel/kthread.c | 30 ++++++++++++++++++++++++++----
 1 file changed, 26 insertions(+), 4 deletions(-)

diff --git a/kernel/kthread.c b/kernel/kthread.c
index fdb8fb07f40b..b9d8dfe1b6ca 100644
--- a/kernel/kthread.c
+++ b/kernel/kthread.c
@@ -327,24 +327,46 @@ void __noreturn kthread_complete_and_exit(struct completion *comp, long code)
 }
 EXPORT_SYMBOL(kthread_complete_and_exit);
 
+/*
+ * Unbound kthreads must run neither on domain isolated CPUs (isolcpus=domain,
+ * cpuset isolated partitions) nor on CPUs isolated from kernel noise
+ * (nohz_full=, isolcpus=nohz). The two sets usually coincide, but the kernel
+ * accepts nohz_full CPUs that remain in the scheduler domains, so both
+ * housekeeping types have to be honoured.
+ */
 static void kthread_fetch_affinity(struct kthread *kthread, struct cpumask *cpumask)
 {
-	const struct cpumask *pref;
+	const struct cpumask *pref, *hk_domain, *hk_kthread;
 
 	guard(rcu)();
 
+	hk_domain = housekeeping_cpumask(HK_TYPE_DOMAIN);
+	/*
+	 * Deliberately not HK_TYPE_KTHREAD: since v7.1 that is an alias of
+	 * HK_TYPE_DOMAIN (commit 8f78b749f3da), which is the very assumption
+	 * being corrected here.
+	 */
+	hk_kthread = housekeeping_cpumask(HK_TYPE_KERNEL_NOISE);
+
 	if (kthread->preferred_affinity) {
 		pref = kthread->preferred_affinity;
 	} else {
 		if (kthread->node == NUMA_NO_NODE)
-			pref = housekeeping_cpumask(HK_TYPE_DOMAIN);
+			pref = hk_domain;
 		else
 			pref = cpumask_of_node(kthread->node);
 	}
 
-	cpumask_and(cpumask, pref, housekeeping_cpumask(HK_TYPE_DOMAIN));
+	cpumask_and(cpumask, pref, hk_domain);
+	cpumask_and(cpumask, cpumask, hk_kthread);
+	if (cpumask_empty(cpumask))
+		cpumask_and(cpumask, hk_domain, hk_kthread);
+	/*
+	 * Both the boot parameters and cpuset guarantee that at least one
+	 * CPU stays housekeeping for every type, but don't rely on it.
+	 */
 	if (!cpumask_intersects(cpumask, cpu_online_mask))
-		cpumask_copy(cpumask, housekeeping_cpumask(HK_TYPE_DOMAIN));
+		cpumask_copy(cpumask, hk_domain);
 }
 
 static void kthread_affine_node(void)
-- 
2.55.0

[PATCH 3/6] mm/folio, mm/mlock: Don't batch folios in per-CPU LRU caches on isolated CPUs

[Matthew Ruffell]

From: Caron Wills <caron.wills@pulsar.com>

Folios are collected in per-CPU folio batches (cpu_fbatches and
mlock_fbatch) before being added to or moved on the LRU lists. Another
CPU can only empty such a batch by queueing lru_add_drain_per_cpu() on
the owning CPU and waiting for it in __lru_add_drain_all(), which is
what every lru_add_drain_all() caller (compaction, khugepaged, memory
hotplug, memfd, drop_caches, ...) and every lru_cache_disable() caller
(page migration through cpuset.mems, mbind() and move_pages(),
alloc_contig_range(), memory offlining) does.

On an isolated CPU that per-CPU kworker may not run for a very long
time, or ever: low latency deployments pin a SCHED_FIFO busy-polling
thread to each isolated CPU, and since RT bandwidth throttling is gone
nothing forces such a thread to yield to a SCHED_NORMAL kworker. Any
folio left in that CPU's batch then blocks the waiting caller.

LP: #2165410 hits exactly this. Writing cpuset.mems for a cgroup on a
192-CPU AWS instance with nohz_full CPUs 18-95,108-178 running FIFO
spinners makes cpuset_migrate_mm_workfn() -> do_migrate_pages() ->
lru_cache_disable() -> __lru_add_drain_all() -> flush_work() wait for
the kworkers on those CPUs, and PID 1 (systemd), which flushes the
migration workqueue from task_work before returning to userspace, stays
in D state for 2 to 10 minutes. logind cannot create sessions meanwhile
and SSH logins fail. The reproducer, a FIFO/80 thread per isolated CPU
that touches and MADV_DONTNEEDs a 24 MB buffer in a loop, keeps 1 to 14
folios in the lru_add batch of every isolated CPU at all times.

Stop batching on isolated CPUs: when the current CPU is isolated, move
the folio to the LRU immediately through an on-stack batch, without
touching the per-CPU batch at all. The latter matters: __lru_add_drain_all()
samples the per-CPU counters without the owning CPU's local_lock, so a
folio that sits in the batch even for the few instructions between
folio_batch_add() and an immediate drain is enough to get work queued on
that CPU. A first version of this patch reused the lru_cache_disabled()
add-then-drain path and was caught doing exactly that in testing.
cpu_needs_drain() now stays false for isolated CPUs and
__lru_add_drain_all() never queues work on them. The per-CPU buffer_head
LRU has been skipping isolated CPUs the same way since commit 8a237adf213d 
("fs/buffer.c: disable per-CPU buffer_head cache for isolated CPUs"),
and mm/memcontrol.c schedule_drain_work() does not queue memcg stock drains
on them either.

The cost is one lruvec lock acquisition per folio instead of one per
batch of 15 on isolated CPUs, which are not expected to do bulk memory
work. Nothing changes on housekeeping CPUs or on systems without CPU
isolation, where cpu_is_isolated() is a static-key guarded test.

Link: https://bugs.launchpad.net/bugs/2165410
Signed-off-by: Caron Wills <caron.wills@pulsar.com>
Signed-off-by: Matthew Ruffell <matthew.ruffell@canonical.com>
---
 mm/folio.c    | 21 ++++++++++++++++++++-
 mm/internal.h | 28 ++++++++++++++++++++++++++++
 mm/mlock.c    | 28 ++++++++++++++++++++++------
 3 files changed, 70 insertions(+), 7 deletions(-)

diff --git a/mm/folio.c b/mm/folio.c
index 35e242b48870..7f85747910d7 100644
--- a/mm/folio.c
+++ b/mm/folio.c
@@ -208,6 +208,16 @@ static void folio_batch_move_lru(struct folio_batch *fbatch, move_fn_t move_fn)
 	folios_put(fbatch);
 }
 
+/* Move a single folio to the LRU without touching the per-CPU batches. */
+static void folio_move_lru_now(struct folio *folio, move_fn_t move_fn)
+{
+	struct folio_batch fbatch;
+
+	folio_batch_init(&fbatch);
+	folio_batch_add(&fbatch, folio);
+	folio_batch_move_lru(&fbatch, move_fn);
+}
+
 static void __folio_batch_add_and_move(struct folio_batch __percpu *fbatch,
 		struct folio *folio, move_fn_t move_fn, bool disable_irq)
 {
@@ -220,7 +230,9 @@ static void __folio_batch_add_and_move(struct folio_batch __percpu *fbatch,
 	else
 		local_lock(&cpu_fbatches.lock);
 
-	if (!folio_batch_add(this_cpu_ptr(fbatch), folio) ||
+	if (lru_batching_bypassed())
+		folio_move_lru_now(folio, move_fn);
+	else if (!folio_batch_add(this_cpu_ptr(fbatch), folio) ||
 			!folio_may_be_lru_cached(folio) || lru_cache_disabled())
 		folio_batch_move_lru(this_cpu_ptr(fbatch), move_fn);
 
@@ -856,6 +868,13 @@ static inline void __lru_add_drain_all(bool force_all_cpus)
 	WRITE_ONCE(lru_drain_gen, lru_drain_gen + 1);
 	smp_mb();
 
+	/*
+	 * Isolated CPUs don't batch folios (see lru_batching_bypassed()), so
+	 * their counters stay at zero and no work gets queued to them.  This
+	 * matters because the per-CPU kworker of an isolated CPU may not get
+	 * to run for a very long time, if ever, and flush_work() below would
+	 * wait for it.
+	 */
 	cpumask_clear(&has_work);
 	for_each_online_cpu(cpu) {
 		struct work_struct *work = &per_cpu(lru_add_drain_work, cpu);
diff --git a/mm/internal.h b/mm/internal.h
index 566cbd2dce48..5888f7b5ea5c 100644
--- a/mm/internal.h
+++ b/mm/internal.h
@@ -17,6 +17,7 @@
 #include <linux/pagewalk.h>
 #include <linux/rmap.h>
 #include <linux/swap.h>
+#include <linux/sched/isolation.h>
 #include <linux/leafops.h>
 #include <linux/tracepoint-defs.h>
 
@@ -1006,6 +1007,33 @@ bool need_mlock_drain(int cpu);
 void mlock_drain_local(void);
 void mlock_drain_remote(int cpu);
 
+/*
+ * Folios are normally parked in per-CPU batches (cpu_fbatches and
+ * mlock_fbatch) before they are moved to the LRU lists. A remote CPU can
+ * only empty such a batch by queueing lru_add_drain_per_cpu() on the owning
+ * CPU and waiting for it, see __lru_add_drain_all(). On an isolated CPU that
+ * kworker may not get to run for a long time, e.g. because a SCHED_FIFO task
+ * busy-loops there, and every lru_add_drain_all() or lru_cache_disable()
+ * caller in the system then blocks behind it.
+ *
+ * Isolated CPUs therefore bypass the per-CPU batches entirely and move
+ * folios to the LRU lists right away, through an on-stack batch. The folio
+ * must not even transiently sit in the per-CPU batch: __lru_add_drain_all()
+ * samples the batch counters without the owning CPU's lock, so the few
+ * instructions between folio_batch_add() and an immediate drain are enough
+ * to get work queued on the isolated CPU. The per-CPU buffer_head LRU
+ * (bh_lru_install()) and the memcg stock drain (schedule_drain_work())
+ * already stay away from isolated CPUs the same way.
+ *
+ * Callers hold the batch's local_lock, so smp_processor_id() is stable. The
+ * housekeeping cpumask consulted by cpu_is_isolated() is RCU protected.
+ */
+static inline bool lru_batching_bypassed(void)
+{
+	guard(rcu)();
+	return cpu_is_isolated(smp_processor_id());
+}
+
 extern pmd_t maybe_pmd_mkwrite(pmd_t pmd, struct vm_area_struct *vma);
 
 static inline unsigned long __vma_address(const struct vm_area_struct *vma,
diff --git a/mm/mlock.c b/mm/mlock.c
index 67ae9b18b923..3bb8195e5c5f 100644
--- a/mm/mlock.c
+++ b/mm/mlock.c
@@ -209,6 +209,16 @@ static void mlock_folio_batch(struct folio_batch *fbatch)
 	folios_put(fbatch);
 }
 
+/* Handle a single folio right away, without touching the per-CPU batch. */
+static void mlock_folio_now(struct folio *folio)
+{
+	struct folio_batch fbatch;
+
+	folio_batch_init(&fbatch);
+	folio_batch_add(&fbatch, folio);
+	mlock_folio_batch(&fbatch);
+}
+
 void mlock_drain_local(void)
 {
 	struct folio_batch *fbatch;
@@ -254,8 +264,10 @@ void mlock_folio(struct folio *folio)
 	}
 
 	folio_get(folio);
-	if (!folio_batch_add(fbatch, mlock_lru(folio)) ||
-	    !folio_may_be_lru_cached(folio) || lru_cache_disabled())
+	if (lru_batching_bypassed())
+		mlock_folio_now(mlock_lru(folio));
+	else if (!folio_batch_add(fbatch, mlock_lru(folio)) ||
+		 !folio_may_be_lru_cached(folio) || lru_cache_disabled())
 		mlock_folio_batch(fbatch);
 	local_unlock(&mlock_fbatch.lock);
 }
@@ -277,8 +289,10 @@ void mlock_new_folio(struct folio *folio)
 	__count_vm_events(UNEVICTABLE_PGMLOCKED, nr_pages);
 
 	folio_get(folio);
-	if (!folio_batch_add(fbatch, mlock_new(folio)) ||
-	    !folio_may_be_lru_cached(folio) || lru_cache_disabled())
+	if (lru_batching_bypassed())
+		mlock_folio_now(mlock_new(folio));
+	else if (!folio_batch_add(fbatch, mlock_new(folio)) ||
+		 !folio_may_be_lru_cached(folio) || lru_cache_disabled())
 		mlock_folio_batch(fbatch);
 	local_unlock(&mlock_fbatch.lock);
 }
@@ -298,8 +312,10 @@ void munlock_folio(struct folio *folio)
 	 * which will check whether the folio is multiply mlocked.
 	 */
 	folio_get(folio);
-	if (!folio_batch_add(fbatch, folio) ||
-	    !folio_may_be_lru_cached(folio) || lru_cache_disabled())
+	if (lru_batching_bypassed())
+		mlock_folio_now(folio);
+	else if (!folio_batch_add(fbatch, folio) ||
+		 !folio_may_be_lru_cached(folio) || lru_cache_disabled())
 		mlock_folio_batch(fbatch);
 	local_unlock(&mlock_fbatch.lock);
 }
-- 
2.55.0

[PATCH 4/6] sched/isolation: Drain per-CPU LRU batches when CPUs become isolated

[Matthew Ruffell]

From: Caron Wills <caron.wills@pulsar.com>

Isolated CPUs no longer batch folios in their per-CPU LRU caches, so
that __lru_add_drain_all() never has to queue work on them. Boot time
isolation (isolcpus=, nohz_full=) takes effect before any folio is
batched, but a cpuset isolated partition can be created at runtime and
the affected CPUs may still hold folios in their batches from before.
Those would linger until the CPU happens to drain locally, and the next
lru_add_drain_all() would fall back to queueing work on the now isolated
CPU, which is exactly what must be avoided.

Drain all batches once from housekeeping_update(), right after the
HK_TYPE_DOMAIN mask has been updated and next to the vmstat and memcg
workqueue flushes that exist for the same reason, so that newly isolated
CPUs start out with empty batches.

Link: https://bugs.launchpad.net/bugs/2165410
Signed-off-by: Caron Wills <caron.wills@pulsar.com>
Signed-off-by: Matthew Ruffell <matthew.ruffell@canonical.com>
---
 kernel/sched/isolation.c | 8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/kernel/sched/isolation.c b/kernel/sched/isolation.c
index 156025ef81b7..17d8171fe97b 100644
--- a/kernel/sched/isolation.c
+++ b/kernel/sched/isolation.c
@@ -10,6 +10,7 @@
 #include <linux/sched/isolation.h>
 #include <linux/llist.h>
 #include <linux/pci.h>
+#include <linux/swap.h>
 #include "sched.h"
 
 enum hk_flags {
@@ -149,6 +150,13 @@ int housekeeping_update(struct cpumask *isol_mask)
 	pci_probe_flush_workqueue();
 	mem_cgroup_flush_workqueue();
 	vmstat_flush_workqueue();
+	/*
+	 * Folios batched on the newly isolated CPUs before the update would
+	 * otherwise linger in their per-CPU LRU batches, since isolated CPUs
+	 * stop batching from now on (see lru_batching_bypassed()) and remote
+	 * draining is exactly what must be avoided on them.
+	 */
+	lru_add_drain_all();
 
 	err = workqueue_unbound_housekeeping_update(housekeeping_cpumask(HK_TYPE_DOMAIN));
 	WARN_ON_ONCE(err < 0);
-- 
2.55.0

[PATCH 5/6] cgroup/cpuset: Don't wait for memory migration when cpuset.mems is written with O_NONBLOCK

[Matthew Ruffell]

From: Caron Wills <caron.wills@pulsar.com>

Writing cpuset.mems migrates the memory of every task in the cpuset on
the ordered cpuset_migrate_mm_wq, and since commit 3514309e0322
("cpuset: Defer flushing of the cpuset_migrate_mm_wq to task_work") the
writer flushes that workqueue from task_work before returning to
userspace, so that the write appears synchronous (the design of commit
e93ad19d0564 ("cpuset: make mm migration asynchronous")). The writer
therefore sleeps in TASK_UNINTERRUPTIBLE for as long as all queued
migrations take. On a system with isolated CPUs running SCHED_FIFO tasks
that was measured at 2 to 10 minutes (LP: #2165410), and it is unbounded
in principle: a migration blocked behind a starved kworker or a slow
expedited grace period holds up every migration queued after it.

The writer is usually systemd. It applies AllowedMemoryNodes= from PID 1
itself, and since v259 it opens every cgroup v2 attribute file with
O_NONBLOCK precisely to opt into asynchronous semantics wherever the
kernel offers them (src/basic/cgroup-util.c, cg_set_attribute(), citing
the memory.max/memory.high behaviour of commit c8e6002bd611 ("memcg:
introduce non-blocking limit setting option")). While PID 1 is stuck in
the flush, logind cannot create sessions and SSH logins fail.

Honour O_NONBLOCK the same way memcg does: skip the task_work flush when
the file was opened non-blocking and let the migration complete in the
background. Blocking writers keep the synchronous behaviour and the
cgroup.procs attach path is unchanged.

This does not make the migration any faster; it only stops the writer,
in practice init, from being held hostage by it.

Link: https://bugs.launchpad.net/bugs/2165410
Signed-off-by: Caron Wills <caron.wills@pulsar.com>
Signed-off-by: Matthew Ruffell <matthew.ruffell@canonical.com>
---
 kernel/cgroup/cpuset.c | 11 ++++++++++-
 1 file changed, 10 insertions(+), 1 deletion(-)

diff --git a/kernel/cgroup/cpuset.c b/kernel/cgroup/cpuset.c
index 3219f927c35e..439b55a310fe 100644
--- a/kernel/cgroup/cpuset.c
+++ b/kernel/cgroup/cpuset.c
@@ -3482,7 +3482,16 @@ ssize_t cpuset_write_resmask(struct kernfs_open_file *of,
 	free_cpuset(trialcs);
 out_unlock:
 	cpuset_update_sd_hk_unlock();
-	if (of_cft(of)->private == FILE_MEMLIST)
+	/*
+	 * Memory migration runs on cpuset_migrate_mm_wq and the writer
+	 * normally waits for it before returning to userspace, so that the
+	 * write appears synchronous. A writer that opened the file with
+	 * O_NONBLOCK asks not to wait: let the migration finish in the
+	 * background, like memory.max and memory.high skip synchronous
+	 * reclaim for non-blocking writers.
+	 */
+	if (of_cft(of)->private == FILE_MEMLIST &&
+	    !(of->file->f_flags & O_NONBLOCK))
 		schedule_flush_migrate_mm();
 	return retval ?: nbytes;
 }
-- 
2.55.0

[PATCH 6/6] cgroup/cpuset: Don't make a forking task wait for memory migration

[Matthew Ruffell]

From: Caron Wills <caron.wills@pulsar.com>

Since commit 65e510cd30d0 ("cgroup/cpuset: Move mpol_rebind_mm/cpuset_migrate_mm()
calls inside cpuset_attach_task()") a CLONE_INTO_CGROUP fork into a cpuset that
differs from the parent's goes through cpuset_attach_task(), which queues a
memory migration of the child's mm and sets attach_ctx.task_work_queued, and
cpuset_fork() then schedules flush_migrate_mm_task_workfn() on the *parent*.
Two things are wrong with that:

 - task_work_queued is set whenever is_memory_migrate(cs) is true, which
   on cgroup v2 is always, even when cpuset_migrate_mm() queued nothing
   because the nodemasks were equal. The parent then flushes
   cpuset_migrate_mm_wq for no reason of its own.

 - cpuset_migrate_mm_wq is a single ordered workqueue, so the flush waits
   for every migration queued by anyone, not just this fork's. systemd
   forks services with CLONE_INTO_CGROUP, and as soon as one slice carries
   a cpuset setting every top-level slice has a cpuset of its own, so PID 1
   blocks in flush_workqueue() on every service start for as long as some
   unrelated cpuset.mems migration takes. On the AWS systems of
   LP: #2165410 that is minutes, during which logind cannot create
   sessions and SSH logins fail.

Make cpuset_migrate_mm() report whether it queued work and only set
task_work_queued in that case, and don't schedule the flush from
cpuset_fork() at all: the forking task is neither the cpuset.mems writer
nor the cgroup.procs writer for whom the synchronous semantics were
introduced by commit e93ad19d0564 ("cpuset: make mm migration asynchronous"),
and nothing depends on the child's pages having moved before the parent
returns from fork(). The migration itself still runs, in the background.

Fixes: 65e510cd30d0 ("cgroup/cpuset: Move mpol_rebind_mm/cpuset_migrate_mm() calls inside cpuset_attach_task()")
Link: https://bugs.launchpad.net/bugs/2165410
Signed-off-by: Caron Wills <caron.wills@pulsar.com>
Signed-off-by: Matthew Ruffell <matthew.ruffell@canonical.com>
---
 kernel/cgroup/cpuset.c | 39 +++++++++++++++++++++++++--------------
 1 file changed, 25 insertions(+), 14 deletions(-)

diff --git a/kernel/cgroup/cpuset.c b/kernel/cgroup/cpuset.c
index 439b55a310fe..b331689eab24 100644
--- a/kernel/cgroup/cpuset.c
+++ b/kernel/cgroup/cpuset.c
@@ -2704,26 +2704,32 @@ static void cpuset_migrate_mm_workfn(struct work_struct *work)
 	kfree(mwork);
 }
 
-static void cpuset_migrate_mm(struct mm_struct *mm, const nodemask_t *from,
+/*
+ * Returns true if migration work was queued, in which case the caller may
+ * want to flush cpuset_migrate_mm_wq before returning to userspace.
+ */
+static bool cpuset_migrate_mm(struct mm_struct *mm, const nodemask_t *from,
 							const nodemask_t *to)
 {
 	struct cpuset_migrate_mm_work *mwork;
 
 	if (nodes_equal(*from, *to)) {
 		mmput(mm);
-		return;
+		return false;
 	}
 
 	mwork = kzalloc_obj(*mwork);
-	if (mwork) {
-		mwork->mm = mm;
-		mwork->from = *from;
-		mwork->to = *to;
-		INIT_WORK(&mwork->work, cpuset_migrate_mm_workfn);
-		queue_work(cpuset_migrate_mm_wq, &mwork->work);
-	} else {
+	if (!mwork) {
 		mmput(mm);
+		return false;
 	}
+
+	mwork->mm = mm;
+	mwork->from = *from;
+	mwork->to = *to;
+	INIT_WORK(&mwork->work, cpuset_migrate_mm_workfn);
+	queue_work(cpuset_migrate_mm_wq, &mwork->work);
+	return true;
 }
 
 static void flush_migrate_mm_task_workfn(struct callback_head *head)
@@ -3376,9 +3382,9 @@ static void cpuset_attach_task(struct cpuset *cs, struct task_struct *task)
 		 * migrate mm from.
 		 */
 		if (is_memory_migrate(cs)) {
-			cpuset_migrate_mm(mm, &oldcs->old_mems_allowed,
-					  &attach_ctx.nodemask_to);
-			attach_ctx.task_work_queued = true;
+			if (cpuset_migrate_mm(mm, &oldcs->old_mems_allowed,
+					      &attach_ctx.nodemask_to))
+				attach_ctx.task_work_queued = true;
 		} else {
 			mmput(mm);
 		}
@@ -3887,8 +3893,13 @@ static void cpuset_fork(struct task_struct *task)
 	attach_ctx.task_work_queued = false;
 	attach_ctx.old_cs = oldcs;
 	cpuset_attach_task(cs, task);
-	if (attach_ctx.task_work_queued)
-		schedule_flush_migrate_mm();
+	/*
+	 * Unlike a cpuset.mems or cgroup.procs writer, the forking task has
+	 * no business waiting for the child's pages to be migrated, and a
+	 * flush of the ordered cpuset_migrate_mm_wq would make it wait for
+	 * every migration anyone queued. Let the migration run in the
+	 * background.
+	 */
 
 	dec_attach_in_progress_locked();
 	mutex_unlock(&cpuset_mutex);
-- 
2.55.0

Re: [PATCH 0/6] mm, sched, cgroup: LRU Cache never drains when isolcpus are used with spinning SCHED_FIFO tasks causing page migration stalls

[Michal Hocko]

On Tue 03-11-26 20:59:34, Matthew Ruffell wrote:
> What happens is that the spinning SCHED_FIFO processes running on isolcpus are
> constantly dirtying the LRU cache, and when we attempt NUMA migration of the
> 50GB of pages, systemd calls flush_migrate_mm_task_workfn() which creates a
> kworker that calls cpuset_migrate_mm_workfn() then do_migrate_pages() then 
> lru_cache_disable(), and then __lru_add_drain_all(). __lru_add_drain_all() then
> creates a kworker on each CPU to drain the LRU cache, and systemd and the
> initial kworker wait until this is done on all CPUs.
> 
> Since the isolcpus are running SCHED_FIFO tasks, they never get preempted,
> and kworkers never get a chance to run until they get an opportunity to by
> returning from a syscall, and it takes a very long time before all 192 CPUs
> actually drain their LRU cache before the page migration can continue.

The idea is that they are running on isolated cpus should they shouldn't
be preempted by unrelated work. But at the same time those
isolated/cached pages shouldn't be blocking the rest of the system. IIRC
there was a proposal to flush all pages cached on pcp on return to the
userspace. What has happened with that work? I do not think this has
been merged. This should deal with the underlying problem without
introducing lock contention that might be a result dopped caching.
-- 
Michal Hocko
SUSE Labs

Re: [PATCH 3/6] mm/folio, mm/mlock: Don't batch folios in per-CPU LRU caches on isolated CPUs

[kernel test robot]

Hi Matthew,

kernel test robot noticed the following build errors:

[auto build test ERROR on akpm-mm/mm-everything]

url:    https://github.com/intel-lab-lkp/linux/commits/Matthew-Ruffell/sched-isolation-Consider-nohz_full-CPUs-isolated-in-cpu_is_isolated-again/20261103-205935
base:   https://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm.git mm-everything
patch link:    https://lore.kernel.org/r/20261103075940.110308-4-matthew.ruffell%40canonical.com
patch subject: [PATCH 3/6] mm/folio, mm/mlock: Don't batch folios in per-CPU LRU caches on isolated CPUs
config: um-allnoconfig (https://download.01.org/0day-ci/archive/20261103/202611032053.I3Ud8d11-lkp@intel.com/config)
compiler: clang version 17.0.6 (https://github.com/llvm/llvm-project 6009708b4367171ccdbf4b5905cb6a803753fe18)
reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20261103/202611032053.I3Ud8d11-lkp@intel.com/reproduce)

If you fix the issue in a separate patch/commit (i.e. not just a new version of
the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <lkp@intel.com>
| Closes: https://lore.kernel.org/oe-kbuild-all/202611032053.I3Ud8d11-lkp@intel.com/

All errors (new ones prefixed by >>):

>> mm/folio.c:293:6: error: call to undeclared function 'lru_batching_bypassed'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]
     293 |         if (lru_batching_bypassed())
         |             ^
   1 error generated.


vim +/lru_batching_bypassed +293 mm/folio.c

   280	
   281	static void __folio_batch_add_and_move(struct folio_batch __percpu *fbatch,
   282			struct folio *folio, move_fn_t move_fn, bool disable_irq)
   283	{
   284		unsigned long flags;
   285	
   286		folio_get(folio);
   287	
   288		if (disable_irq)
   289			local_lock_irqsave(&cpu_fbatches.lock_irq, flags);
   290		else
   291			local_lock(&cpu_fbatches.lock);
   292	
 > 293		if (lru_batching_bypassed())
   294			folio_move_lru_now(folio, move_fn);
   295		else if (!folio_batch_add(this_cpu_ptr(fbatch), folio) ||
   296				!folio_may_be_lru_cached(folio) || lru_cache_disabled())
   297			folio_batch_move_lru(this_cpu_ptr(fbatch), move_fn);
   298	
   299		if (disable_irq)
   300			local_unlock_irqrestore(&cpu_fbatches.lock_irq, flags);
   301		else
   302			local_unlock(&cpu_fbatches.lock);
   303	}
   304	

--
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki

Re: [PATCH 3/6] mm/folio, mm/mlock: Don't batch folios in per-CPU LRU caches on isolated CPUs

[kernel test robot]

Hi Matthew,

kernel test robot noticed the following build errors:

[auto build test ERROR on akpm-mm/mm-everything]

url:    https://github.com/intel-lab-lkp/linux/commits/Matthew-Ruffell/sched-isolation-Consider-nohz_full-CPUs-isolated-in-cpu_is_isolated-again/20261103-205935
base:   https://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm.git mm-everything
patch link:    https://lore.kernel.org/r/20261103075940.110308-4-matthew.ruffell%40canonical.com
patch subject: [PATCH 3/6] mm/folio, mm/mlock: Don't batch folios in per-CPU LRU caches on isolated CPUs
config: sh-allnoconfig (https://download.01.org/0day-ci/archive/20261103/202611032138.4bxGcYLC-lkp@intel.com/config)
compiler: sh4-linux-gcc (GCC) 16.2.0
reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20261103/202611032138.4bxGcYLC-lkp@intel.com/reproduce)

If you fix the issue in a separate patch/commit (i.e. not just a new version of
the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <lkp@intel.com>
| Closes: https://lore.kernel.org/oe-kbuild-all/202611032138.4bxGcYLC-lkp@intel.com/

All errors (new ones prefixed by >>):

   mm/folio.c: In function '__folio_batch_add_and_move':
>> mm/folio.c:293:13: error: implicit declaration of function 'lru_batching_bypassed' [-Wimplicit-function-declaration]
     293 |         if (lru_batching_bypassed())
         |             ^~~~~~~~~~~~~~~~~~~~~


vim +/lru_batching_bypassed +293 mm/folio.c

   280	
   281	static void __folio_batch_add_and_move(struct folio_batch __percpu *fbatch,
   282			struct folio *folio, move_fn_t move_fn, bool disable_irq)
   283	{
   284		unsigned long flags;
   285	
   286		folio_get(folio);
   287	
   288		if (disable_irq)
   289			local_lock_irqsave(&cpu_fbatches.lock_irq, flags);
   290		else
   291			local_lock(&cpu_fbatches.lock);
   292	
 > 293		if (lru_batching_bypassed())
   294			folio_move_lru_now(folio, move_fn);
   295		else if (!folio_batch_add(this_cpu_ptr(fbatch), folio) ||
   296				!folio_may_be_lru_cached(folio) || lru_cache_disabled())
   297			folio_batch_move_lru(this_cpu_ptr(fbatch), move_fn);
   298	
   299		if (disable_irq)
   300			local_unlock_irqrestore(&cpu_fbatches.lock_irq, flags);
   301		else
   302			local_unlock(&cpu_fbatches.lock);
   303	}
   304	

--
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki

Re: [PATCH 2/6] kthread: Keep unbound kthreads off nohz_full CPUs

[Frederic Weisbecker]

Le Tue, Nov 03, 2026 at 08:59:36PM +1300, Matthew Ruffell a écrit :
> From: Caron Wills <caron.wills@pulsar.com>
> 
> Since commit 041ee6f3727a ("kthread: Rely on HK_TYPE_DOMAIN for
> preferred affinity management") kthread_fetch_affinity() only removes
> domain isolated CPUs from a kthread's affinity, assuming that nohz_full
> CPUs are always domain isolated as well. Up to v6.13 every kthread
> inherited kthreadd's affinity to housekeeping_cpumask(HK_TYPE_KTHREAD),
> and from v6.14 the preferred affinity was filtered with HK_TYPE_KTHREAD,
> so nohz_full CPUs were always excluded.

This was done on purpose because on all setup we have met so far, it didn't
make sense to run nohz_full with load balancing. After all the point is to
run a pinned single task on a single CPU. We have seen too many misconfigured
CPU isolation workloads due to that constraint not being enforced since the
origins.

I'm open to counter-arguments of course but you better have a good case.

Thanks.

-- 
Frederic Weisbecker
SUSE Labs

Re: [PATCH 4/6] sched/isolation: Drain per-CPU LRU batches when CPUs become isolated

[Frederic Weisbecker]

Le Tue, Nov 03, 2026 at 08:59:38PM +1300, Matthew Ruffell a écrit :
> From: Caron Wills <caron.wills@pulsar.com>
> 
> Isolated CPUs no longer batch folios in their per-CPU LRU caches, so
> that __lru_add_drain_all() never has to queue work on them. Boot time
> isolation (isolcpus=, nohz_full=) takes effect before any folio is
> batched, but a cpuset isolated partition can be created at runtime and
> the affected CPUs may still hold folios in their batches from before.
> Those would linger until the CPU happens to drain locally, and the next
> lru_add_drain_all() would fall back to queueing work on the now isolated
> CPU, which is exactly what must be avoided.
> 
> Drain all batches once from housekeeping_update(), right after the
> HK_TYPE_DOMAIN mask has been updated and next to the vmstat and memcg
> workqueue flushes that exist for the same reason, so that newly isolated
> CPUs start out with empty batches.
> 
> Link: https://bugs.launchpad.net/bugs/2165410
> Signed-off-by: Caron Wills <caron.wills@pulsar.com>
> Signed-off-by: Matthew Ruffell <matthew.ruffell@canonical.com>

There was this patchset:
https://lore.kernel.org/lkml/20260519012754.240804-1-leobras.c@gmail.com/

Leonardo, do you plan a v5?

Thanks.

-- 
Frederic Weisbecker
SUSE Labs

Re: [PATCH 6/6] cgroup/cpuset: Don't make a forking task wait for memory migration

[Tejun Heo]

Hello,

On Tue, Nov 03, 2026 at 08:59:40PM +1300, Matthew Ruffell wrote:
...
>  - task_work_queued is set whenever is_memory_migrate(cs) is true, which
>    on cgroup v2 is always, even when cpuset_migrate_mm() queued nothing
>    because the nodemasks were equal. The parent then flushes
>    cpuset_migrate_mm_wq for no reason of its own.
> 
>  - cpuset_migrate_mm_wq is a single ordered workqueue, so the flush waits
>    for every migration queued by anyone, not just this fork's. systemd
>    forks services with CLONE_INTO_CGROUP, and as soon as one slice carries
>    a cpuset setting every top-level slice has a cpuset of its own, so PID 1
>    blocks in flush_workqueue() on every service start for as long as some
>    unrelated cpuset.mems migration takes. On the AWS systems of
>    LP: #2165410 that is minutes, during which logind cannot create
>    sessions and SSH logins fail.

The end result looks fine to me but can you please separate the two changes
into separate patches? The two changes can't interact with each other, so
the patch as a whole looks puzzling.

Thanks.

-- 
tejun

Re: [PATCH 5/6] cgroup/cpuset: Don't wait for memory migration when cpuset.mems is written with O_NONBLOCK

[Tejun Heo]

On Tue, Nov 03, 2026 at 08:59:39PM +1300, Matthew Ruffell wrote:
> From: Caron Wills <caron.wills@pulsar.com>
> 
> Writing cpuset.mems migrates the memory of every task in the cpuset on
> the ordered cpuset_migrate_mm_wq, and since commit 3514309e0322
> ("cpuset: Defer flushing of the cpuset_migrate_mm_wq to task_work") the
> writer flushes that workqueue from task_work before returning to
> userspace, so that the write appears synchronous (the design of commit
> e93ad19d0564 ("cpuset: make mm migration asynchronous")). The writer
> therefore sleeps in TASK_UNINTERRUPTIBLE for as long as all queued
> migrations take. On a system with isolated CPUs running SCHED_FIFO tasks
> that was measured at 2 to 10 minutes (LP: #2165410), and it is unbounded
> in principle: a migration blocked behind a starved kworker or a slow
> expedited grace period holds up every migration queued after it.
> 
> The writer is usually systemd. It applies AllowedMemoryNodes= from PID 1
> itself, and since v259 it opens every cgroup v2 attribute file with
> O_NONBLOCK precisely to opt into asynchronous semantics wherever the
> kernel offers them (src/basic/cgroup-util.c, cg_set_attribute(), citing
> the memory.max/memory.high behaviour of commit c8e6002bd611 ("memcg:
> introduce non-blocking limit setting option")). While PID 1 is stuck in
> the flush, logind cannot create sessions and SSH logins fail.
> 
> Honour O_NONBLOCK the same way memcg does: skip the task_work flush when
> the file was opened non-blocking and let the migration complete in the
> background. Blocking writers keep the synchronous behaviour and the
> cgroup.procs attach path is unchanged.
> 
> This does not make the migration any faster; it only stops the writer,
> in practice init, from being held hostage by it.
> 
> Link: https://bugs.launchpad.net/bugs/2165410
> Signed-off-by: Caron Wills <caron.wills@pulsar.com>
> Signed-off-by: Matthew Ruffell <matthew.ruffell@canonical.com>

This makes sense to me. Waiman?

Thanks.

-- 
tejun

Re: [PATCH 0/6] mm, sched, cgroup: LRU Cache never drains when isolcpus are used with spinning SCHED_FIFO tasks causing page migration stalls

[Tejun Heo]

Hello,

On Tue, Nov 03, 2026 at 08:59:34PM +1300, Matthew Ruffell wrote:
...
> isolcpus=nohz,18-95,108-178 nohz_full=18-95,108-178 rcu_nocbs=18-95,108-178 
> rcu_nocb_poll skew_tick=1 nosmt=force idle=poll numa_balancing=disable 
> transparent_hugepage=never audit=0 nmi_watchdog=0 nowatchdog mce=ignore_ce 
> tsc=reliable clocksource=tsc

We should get rid of fine grained isolation control. This is such a footgun
and ties implementation details to interface too tightly.

...
> What happens is that the spinning SCHED_FIFO processes running on isolcpus are
> constantly dirtying the LRU cache, and when we attempt NUMA migration of the
> 50GB of pages, systemd calls flush_migrate_mm_task_workfn() which creates a
> kworker that calls cpuset_migrate_mm_workfn() then do_migrate_pages() then 
> lru_cache_disable(), and then __lru_add_drain_all(). __lru_add_drain_all() then
> creates a kworker on each CPU to drain the LRU cache, and systemd and the
> initial kworker wait until this is done on all CPUs.

I suppose you have fair deadline server disabled?

Thanks.

-- 
tejun

Re: [PATCH 2/6] kthread: Keep unbound kthreads off nohz_full CPUs

[Waiman Long]

On 11/3/26 9:01 AM, Frederic Weisbecker wrote:
> Le Tue, Nov 03, 2026 at 08:59:36PM +1300, Matthew Ruffell a écrit :
>> From: Caron Wills <caron.wills@pulsar.com>
>>
>> Since commit 041ee6f3727a ("kthread: Rely on HK_TYPE_DOMAIN for
>> preferred affinity management") kthread_fetch_affinity() only removes
>> domain isolated CPUs from a kthread's affinity, assuming that nohz_full
>> CPUs are always domain isolated as well. Up to v6.13 every kthread
>> inherited kthreadd's affinity to housekeeping_cpumask(HK_TYPE_KTHREAD),
>> and from v6.14 the preferred affinity was filtered with HK_TYPE_KTHREAD,
>> so nohz_full CPUs were always excluded.
> This was done on purpose because on all setup we have met so far, it didn't
> make sense to run nohz_full with load balancing. After all the point is to
> run a pinned single task on a single CPU. We have seen too many misconfigured
> CPU isolation workloads due to that constraint not being enforced since the
> origins.
>
> I'm open to counter-arguments of course but you better have a good case.

I do see configurations where nohz_full/managed_irq CPUs are set at boot 
time without isolcpus=domain. But these nohz_full CPUs are there as a 
backup in case some latency critical workloads need a set of fully 
isolated CPUs to run on. Right now, isolcpus=domain can be set 
dynamically by using the cpuset isolated partition feature and they will 
be used when the need arises.

Matthew, is it possible for users to adapt their workloads to 
dynamically domain-isolated the CPUs before running RT tasks on them?

Cheers,
Longman

Re: [PATCH 5/6] cgroup/cpuset: Don't wait for memory migration when cpuset.mems is written with O_NONBLOCK

[Waiman Long]

On 11/3/26 9:59 AM, Tejun Heo wrote:
> On Tue, Nov 03, 2026 at 08:59:39PM +1300, Matthew Ruffell wrote:
>> From: Caron Wills <caron.wills@pulsar.com>
>>
>> Writing cpuset.mems migrates the memory of every task in the cpuset on
>> the ordered cpuset_migrate_mm_wq, and since commit 3514309e0322
>> ("cpuset: Defer flushing of the cpuset_migrate_mm_wq to task_work") the
>> writer flushes that workqueue from task_work before returning to
>> userspace, so that the write appears synchronous (the design of commit
>> e93ad19d0564 ("cpuset: make mm migration asynchronous")). The writer
>> therefore sleeps in TASK_UNINTERRUPTIBLE for as long as all queued
>> migrations take. On a system with isolated CPUs running SCHED_FIFO tasks
>> that was measured at 2 to 10 minutes (LP: #2165410), and it is unbounded
>> in principle: a migration blocked behind a starved kworker or a slow
>> expedited grace period holds up every migration queued after it.
>>
>> The writer is usually systemd. It applies AllowedMemoryNodes= from PID 1
>> itself, and since v259 it opens every cgroup v2 attribute file with
>> O_NONBLOCK precisely to opt into asynchronous semantics wherever the
>> kernel offers them (src/basic/cgroup-util.c, cg_set_attribute(), citing
>> the memory.max/memory.high behaviour of commit c8e6002bd611 ("memcg:
>> introduce non-blocking limit setting option")). While PID 1 is stuck in
>> the flush, logind cannot create sessions and SSH logins fail.
>>
>> Honour O_NONBLOCK the same way memcg does: skip the task_work flush when
>> the file was opened non-blocking and let the migration complete in the
>> background. Blocking writers keep the synchronous behaviour and the
>> cgroup.procs attach path is unchanged.
>>
>> This does not make the migration any faster; it only stops the writer,
>> in practice init, from being held hostage by it.
>>
>> Link: https://bugs.launchpad.net/bugs/2165410
>> Signed-off-by: Caron Wills <caron.wills@pulsar.com>
>> Signed-off-by: Matthew Ruffell <matthew.ruffell@canonical.com>
> This makes sense to me. Waiman?
>
> Thanks.

Yes, that looks good to me too.

Maybe, we should also update the cgroup-v2.rst to list the condition 
when synchronous waiting will be skipped when updating cpuset.mems as it 
is a behavioral change.

Cheers,
Longman

>