[RFC] sched/fair: avoid redundant updates for unchanged default slices

[RFC] sched/fair: avoid redundant updates for unchanged default slices

[Chengfeng Lin]

[-- Attachment #1: Type: text/plain, Size: 6823 bytes --]

Hi,

Repeated sched_setparam() calls with unchanged parameters still take the
parameter-update path for a SCHED_OTHER task using the default slice.
A small prototype reduced this cost from 618.18 to 163.23 ns per call on
v7.2, saving about 455 ns (73.6%). I would appreciate feedback on this
optimization.

I found this while investigating a 6.5-7.1% increase in this no-change
workload's cost between the tested v6.12.95 and v7.2 builds. The patch
test below reuses the same benchmark source, binary and parameters.
It tests an optimization on v7.2; it does not establish the cause of the
earlier version difference.

The test task uses SCHED_OTHER, nice 0 and the default slice. It repeatedly
calls sched_setparam() on itself with pid 0 and sched_priority 0, without
requesting a parameter change.

The legacy wrapper _sched_setscheduler() leaves attr.sched_runtime at zero
for a default slice. Zero requests the default, not a zero-length slice.
However, p->se.slice stores the actual duration. __sched_setscheduler()
compares these values and continues with the update even when the task
already uses the current default. For a custom slice, the wrapper copies
the actual duration, so the existing no-change shortcut works.

The prototype below extends that shortcut to unchanged default slices.
The new case requires the actual fair scheduling class, unchanged nice,
default-slice mode, and a slice matching the current base value. It keeps
the full update when an implicit uclamp request needs resetting. This
matters after an RT priority-inheritance boost has ended. Existing
permission and LSM checks remain before the shortcut; explicit uclamp
updates and reset-on-fork handling are unchanged.

I tested original v7.2 -> prototype -> original v7.2 on an i7-12700KF.
Mean loop cost per call, in ns:

  slice mode          original A    prototype    original B
  default                619.40       163.23        616.97
  custom, 10 ms          162.51       162.64        162.68

The custom control made the same calls after setting a 10 ms slice.
Both cases ran as non-root, SCHED_OTHER, nice-0 tasks pinned to CPU 2.
Each boot had three warmup rounds and 15 measured rounds of 500,000 calls
per case. Both kernels used GCC 15.2.0 and matching configurations except for
LOCALVERSION, with full preemption, performance governor/EPP and Turbo
off. Maximum CV was below 0.19%; original-boot drift stayed below 0.40%.

Separate probes confirmed that the default case stopped entering the
fair parameter-update path. Permission failures, changed parameters,
custom/default transitions, and bounded concurrent base-slice updates
passed the checks. RT/DL priority-inheritance tests retained the observed
error returns and required implicit uclamp reset after unboosting.
No probes were active during timing.

This is a synthetic syscall microbenchmark, not an application speedup.
In a separate sched_setattr() A/B/A test, I compared the prototype with
the midpoint of the two original-kernel runs. Loop cost was 0.10% higher
when alternating between 4 ms and 10 ms custom slices. It was 0.72% higher
when alternating between the default slice and a 10 ms custom slice.
Both comparisons passed the stability checks. The increases were below
1% in this bounded test.
I have not tested all parameter changes, concurrent states or a running
sched_ext scheduler.

In an October 8 source check, the unchanged diff also applied to mainline
0c2669a9f4a1 and tip/sched/core 4a3b51aab6e2. Their syscalls.c matched the
tested v7.2 file. These were not additional kernel tests.

The standalone benchmark is attached as sched_fair_control_v2_bench.c.
With the settings described above, build it and run both cases from a
non-root shell using SCHED_OTHER and nice 0:

  gcc-15 -O2 -g0 -std=gnu11 -Wall -Wextra -Werror -pthread \
    sched_fair_control_v2_bench.c -o sched-fair-bench
  ./sched-fair-bench --profile setparam-current-v2 --cpu-a 2 \
    --ops 500000 --warmups 3 --rounds 15
  ./sched-fair-bench --profile setparam-fast-control-v2 --cpu-a 2 \
    --ops 500000 --warmups 3 --rounds 15

Use an available CPU on your machine; CPU 2 was used here. The program
sets the requested slice mode before timing, checks return values and
final state, and restores the default slice and original CPU affinity
before exit. It does not change the machine's power settings.

Claude proposed the workload. Codex assisted with the prototype,
validation scripts, result analysis and this message. The performance
figures come from recorded bare-metal runs, not model estimates.

Would this be a useful shortcut, or are there other updates that this
unchanged-parameter case must retain?

Thanks,
Chengfeng

Assisted-by: LLM

---
diff --git a/kernel/sched/syscalls.c b/kernel/sched/syscalls.c
--- a/kernel/sched/syscalls.c
+++ b/kernel/sched/syscalls.c
@@ -491,6 +491,39 @@ req_priv:
        return 0;
 }

+static bool fair_params_changed(struct task_struct *p,
+                               const struct sched_attr *attr)
+{
+       if (attr->sched_nice != task_nice(p))
+               return true;
+
+       if (attr->sched_runtime == p->se.slice)
+               return false;
+
+       /* Preserve the existing path for PI-boosted and sched_ext tasks. */
+       if (p->sched_class != &fair_sched_class)
+               return true;
+
+#ifdef CONFIG_UCLAMP_TASK
+       /*
+        * Unchanged fair parameters can still require an implicit clamp reset,
+        * for example after a task was PI-boosted into the RT class.
+        */
+       if ((!p->uclamp_req[UCLAMP_MIN].user_defined &&
+            p->uclamp_req[UCLAMP_MIN].value != uclamp_none(UCLAMP_MIN)) ||
+           (!p->uclamp_req[UCLAMP_MAX].user_defined &&
+            p->uclamp_req[UCLAMP_MAX].value != uclamp_none(UCLAMP_MAX)))
+               return true;
+#endif
+
+       /*
+        * A zero runtime requests the default slice, whereas se.slice stores
+        * its actual duration. Keep the update if the mode or default changed.
+        */
+       return attr->sched_runtime || p->se.custom_slice ||
+               p->se.slice != READ_ONCE(sysctl_sched_base_slice);
+}
+
 int __sched_setscheduler(struct task_struct *p,
                         const struct sched_attr *attr,
                         bool user, bool pi)
@@ -588,9 +621,7 @@ recheck:
         * but store a possible modification of reset_on_fork.
         */
        if (unlikely(policy == p->policy)) {
-               if (fair_policy(policy) &&
-                   (attr->sched_nice != task_nice(p) ||
-                    (attr->sched_runtime != p->se.slice)))
+               if (fair_policy(policy) && fair_params_changed(p, attr))
                        goto change;
                if (rt_policy(policy) && attr->sched_priority != p->rt_priority)
                        goto change;

[-- Attachment #2: sched_fair_control_v2_bench.c --]
[-- Type: text/x-csrc, Size: 23270 bytes --]

#define _GNU_SOURCE

#include <errno.h>
#include <limits.h>
#include <pthread.h>
#include <sched.h>
#include <stdatomic.h>
#include <stdbool.h>
#include <stdint.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/resource.h>
#include <sys/syscall.h>
#include <sys/types.h>
#include <time.h>
#include <unistd.h>

#define NS_PER_MS 1000000ULL
#define DEFAULT_WARMUPS 3U
#define DEFAULT_ROUNDS 15U
#define DEFAULT_OPS 100000ULL
#define DEFAULT_AFFINITY_OPS 20000ULL
#define WAIT_TIMEOUT_NS (10ULL * 1000000000ULL)

struct ks_sched_attr {
	uint32_t size;
	uint32_t sched_policy;
	uint64_t sched_flags;
	int32_t sched_nice;
	uint32_t sched_priority;
	uint64_t sched_runtime;
	uint64_t sched_deadline;
	uint64_t sched_period;
	uint32_t sched_util_min;
	uint32_t sched_util_max;
};

enum profile {
	PROFILE_YIELD_DUAL,
	PROFILE_YIELD_SOLO,
	PROFILE_SETPARAM_CURRENT,
	PROFILE_SETPARAM_FAST_CONTROL,
	PROFILE_AFFINITY_QUEUED,
	PROFILE_AFFINITY_SAME_MASK,
};

struct options {
	enum profile profile;
	const char *profile_name;
	int cpu_a;
	int cpu_b;
	uint64_t ops;
	unsigned int warmups;
	unsigned int rounds;
};

struct result_row {
	uint64_t requested_ops;
	uint64_t success_ops;
	uint64_t elapsed_ns;
	uint64_t call_ns_sum;
	uint64_t errors;
	pid_t caller_tid;
	pid_t peer_tid;
	pid_t target_tid;
	uint64_t heartbeat_before;
	uint64_t heartbeat_after;
	int final_cpu;
};

struct yield_peer_state {
	atomic_uint_fast64_t ready_generation;
	atomic_uint_fast64_t go_generation;
	atomic_uint_fast64_t peer_done_generation;
	atomic_uint_fast64_t main_done_generation;
	atomic_uint_fast64_t cumulative_errors;
	atomic_int thread_error;
	atomic_int peer_tid;
	uint64_t ops;
	unsigned int total_rounds;
	int cpu;
};

struct affinity_state {
	atomic_uint_fast64_t turn;
	atomic_uint_fast64_t target_heartbeat;
	atomic_uint_fast64_t controller_b_call_ns;
	atomic_uint_fast64_t controller_b_success;
	atomic_uint_fast64_t controller_b_errors;
	atomic_int target_ready;
	atomic_int controller_b_ready;
	atomic_int controller_b_tid;
	atomic_int target_last_cpu;
	atomic_int thread_error;
	atomic_int stop_target;
	pid_t target_tid;
	uint64_t ops;
	int cpu_a;
	int cpu_b;
	int target_cpu;
};

static void die(const char *what)
{
	fprintf(stderr, "%s: %s\n", what, strerror(errno));
	exit(EXIT_FAILURE);
}

static void fail(const char *what)
{
	fprintf(stderr, "%s\n", what);
	exit(EXIT_FAILURE);
}

static void set_thread_name(const char *name)
{
	int error = pthread_setname_np(pthread_self(), name);

	if (error) {
		errno = error;
		die("pthread_setname_np");
	}
}

static uint64_t now_ns(void)
{
	struct timespec ts;

	if (clock_gettime(CLOCK_MONOTONIC_RAW, &ts))
		die("clock_gettime");
	return (uint64_t)ts.tv_sec * 1000000000ULL + (uint64_t)ts.tv_nsec;
}

static void cpu_relax(void)
{
#if defined(__x86_64__) || defined(__i386__)
	__asm__ __volatile__("pause" ::: "memory");
#else
	atomic_signal_fence(memory_order_seq_cst);
#endif
}

static uint64_t parse_u64(const char *value, const char *name, bool allow_zero)
{
	char *end = NULL;
	unsigned long long parsed;

	errno = 0;
	parsed = strtoull(value, &end, 10);
	if (errno || !end || *end || (!allow_zero && parsed == 0)) {
		fprintf(stderr, "invalid %s: %s\n", name, value);
		exit(EXIT_FAILURE);
	}
	return (uint64_t)parsed;
}

static int parse_cpu(const char *value, const char *name)
{
	uint64_t parsed = parse_u64(value, name, true);

	if (parsed >= CPU_SETSIZE || parsed > INT_MAX) {
		fprintf(stderr, "invalid %s: %s\n", name, value);
		exit(EXIT_FAILURE);
	}
	return (int)parsed;
}

static enum profile parse_profile(const char *name)
{
	if (!strcmp(name, "yield-dual"))
		return PROFILE_YIELD_DUAL;
	if (!strcmp(name, "yield-solo"))
		return PROFILE_YIELD_SOLO;
	if (!strcmp(name, "setparam-current-v2"))
		return PROFILE_SETPARAM_CURRENT;
	if (!strcmp(name, "setparam-fast-control-v2"))
		return PROFILE_SETPARAM_FAST_CONTROL;
	if (!strcmp(name, "affinity-queued-v2"))
		return PROFILE_AFFINITY_QUEUED;
	if (!strcmp(name, "affinity-remote-same-mask-v2"))
		return PROFILE_AFFINITY_SAME_MASK;
	fail("unknown profile");
	return PROFILE_YIELD_SOLO;
}

static bool is_affinity_profile(enum profile profile)
{
	return profile == PROFILE_AFFINITY_QUEUED ||
	       profile == PROFILE_AFFINITY_SAME_MASK;
}

static void usage(const char *argv0)
{
	fprintf(stderr,
		"usage: %s --profile yield-dual|yield-solo|setparam-current-v2|"
		"setparam-fast-control-v2|affinity-queued-v2|"
		"affinity-remote-same-mask-v2 "
		"--cpu-a N [--cpu-b N] [--ops N] [--warmups N] [--rounds N]\n",
		argv0);
	exit(EXIT_FAILURE);
}

static struct options parse_options(int argc, char **argv)
{
	struct options opts = {
		.cpu_a = -1,
		.cpu_b = -1,
		.ops = 0,
		.warmups = DEFAULT_WARMUPS,
		.rounds = DEFAULT_ROUNDS,
	};
	int i;

	for (i = 1; i < argc; i++) {
		if (!strcmp(argv[i], "--profile") && i + 1 < argc) {
			opts.profile_name = argv[++i];
			opts.profile = parse_profile(opts.profile_name);
		} else if (!strcmp(argv[i], "--cpu-a") && i + 1 < argc) {
			opts.cpu_a = parse_cpu(argv[++i], "CPU A");
		} else if (!strcmp(argv[i], "--cpu-b") && i + 1 < argc) {
			opts.cpu_b = parse_cpu(argv[++i], "CPU B");
		} else if (!strcmp(argv[i], "--ops") && i + 1 < argc) {
			opts.ops = parse_u64(argv[++i], "ops", false);
		} else if (!strcmp(argv[i], "--warmups") && i + 1 < argc) {
			uint64_t parsed = parse_u64(argv[++i], "warmups", true);

			if (parsed > UINT_MAX)
				fail("warmups is too large");
			opts.warmups = (unsigned int)parsed;
		} else if (!strcmp(argv[i], "--rounds") && i + 1 < argc) {
			uint64_t parsed = parse_u64(argv[++i], "rounds", false);

			if (parsed > UINT_MAX)
				fail("rounds is too large");
			opts.rounds = (unsigned int)parsed;
		} else {
			usage(argv[0]);
		}
	}
	if (!opts.profile_name || opts.cpu_a < 0)
		usage(argv[0]);
	if (opts.ops == 0)
		opts.ops = is_affinity_profile(opts.profile) ?
			DEFAULT_AFFINITY_OPS : DEFAULT_OPS;
	if (is_affinity_profile(opts.profile)) {
		if (opts.cpu_b < 0 || opts.cpu_a == opts.cpu_b)
			fail("affinity profiles need two different CPUs");
	}
	if (opts.profile == PROFILE_AFFINITY_QUEUED) {
		if (opts.ops & 1)
			fail("affinity-queued ops must be even");
	}
	return opts;
}

static void make_single_cpu_mask(cpu_set_t *mask, int cpu)
{
	CPU_ZERO(mask);
	CPU_SET(cpu, mask);
}

static void pin_current(int cpu)
{
	cpu_set_t mask;

	make_single_cpu_mask(&mask, cpu);
	if (sched_setaffinity(0, sizeof(mask), &mask))
		die("sched_setaffinity current");
}

static void require_cpu_allowed(const cpu_set_t *mask, int cpu)
{
	if (!CPU_ISSET(cpu, mask)) {
		fprintf(stderr, "CPU %d is outside the original affinity mask\n", cpu);
		exit(EXIT_FAILURE);
	}
}

static int current_nice(void)
{
	int value;

	errno = 0;
	value = getpriority(PRIO_PROCESS, 0);
	if (value == -1 && errno)
		die("getpriority");
	return value;
}

static void verify_current_state(int cpu)
{
	struct sched_param param;

	if (sched_getscheduler(0) != SCHED_OTHER)
		fail("current policy is not SCHED_OTHER");
	if (sched_getparam(0, &param))
		die("sched_getparam");
	if (param.sched_priority != 0)
		fail("current sched_priority is not zero");
	if (current_nice() != 0)
		fail("current nice is not zero");
	if (sched_getcpu() != cpu)
		fail("current CPU does not match the requested CPU");
}

static int ks_sched_setattr(pid_t pid, uint64_t runtime)
{
	struct ks_sched_attr attr = {
		.size = sizeof(attr),
		.sched_policy = SCHED_OTHER,
		.sched_nice = 0,
		.sched_runtime = runtime,
	};

	return (int)syscall(SYS_sched_setattr, pid, &attr, 0);
}

static void wait_atomic_u64(atomic_uint_fast64_t *value, uint64_t expected,
			    atomic_int *thread_error, const char *what)
{
	uint64_t start = now_ns();

	while (atomic_load_explicit(value, memory_order_acquire) < expected) {
		if (thread_error && atomic_load_explicit(thread_error,
						       memory_order_acquire))
			fail(what);
		if (now_ns() - start > WAIT_TIMEOUT_NS)
			fail(what);
		cpu_relax();
	}
}

static void wait_atomic_int(atomic_int *value, int expected,
			    atomic_int *thread_error, const char *what)
{
	uint64_t start = now_ns();

	while (atomic_load_explicit(value, memory_order_acquire) != expected) {
		if (thread_error && atomic_load_explicit(thread_error,
						       memory_order_acquire))
			fail(what);
		if (now_ns() - start > WAIT_TIMEOUT_NS)
			fail(what);
		cpu_relax();
	}
}

static void *yield_peer_main(void *opaque)
{
	struct yield_peer_state *state = opaque;
	unsigned int round;

	set_thread_name("sf-yield-peer");
	atomic_store_explicit(&state->peer_tid, (int)syscall(SYS_gettid),
			      memory_order_release);
	{
		cpu_set_t mask;

		make_single_cpu_mask(&mask, state->cpu);
		if (sched_setaffinity(0, sizeof(mask), &mask)) {
			atomic_store_explicit(&state->thread_error, errno ?: EIO,
					      memory_order_release);
			return NULL;
		}
	}
	for (round = 1; round <= state->total_rounds; round++) {
		uint64_t i, errors = 0;

		atomic_store_explicit(&state->ready_generation, round,
				      memory_order_release);
		wait_atomic_u64(&state->go_generation, round,
				&state->thread_error, "yield peer start timeout");
		for (i = 0; i < state->ops; i++) {
			if (syscall(SYS_sched_yield) != 0)
				errors++;
		}
		atomic_fetch_add_explicit(&state->cumulative_errors, errors,
					  memory_order_relaxed);
		atomic_store_explicit(&state->peer_done_generation, round,
				      memory_order_release);
		wait_atomic_u64(&state->main_done_generation, round,
				&state->thread_error, "yield main completion timeout");
	}
	return NULL;
}

static struct result_row run_yield_solo(uint64_t ops)
{
	struct result_row row = {
		.requested_ops = ops,
		.caller_tid = (pid_t)syscall(SYS_gettid),
	};
	uint64_t i, start = now_ns();

	for (i = 0; i < ops; i++) {
		if (syscall(SYS_sched_yield) == 0)
			row.success_ops++;
		else
			row.errors++;
	}
	row.elapsed_ns = now_ns() - start;
	row.call_ns_sum = row.elapsed_ns;
	row.final_cpu = sched_getcpu();
	return row;
}

static void run_yield_dual(const struct options *opts, unsigned int total_rounds,
			   struct result_row *rows)
{
	struct yield_peer_state state;
	pthread_t peer;
	unsigned int round;
	uint64_t previous_peer_errors = 0;

	memset(&state, 0, sizeof(state));
	atomic_init(&state.ready_generation, 0);
	atomic_init(&state.go_generation, 0);
	atomic_init(&state.peer_done_generation, 0);
	atomic_init(&state.main_done_generation, 0);
	atomic_init(&state.cumulative_errors, 0);
	atomic_init(&state.thread_error, 0);
	atomic_init(&state.peer_tid, 0);
	state.ops = opts->ops;
	state.total_rounds = total_rounds;
	state.cpu = opts->cpu_a;
	if (pthread_create(&peer, NULL, yield_peer_main, &state))
		die("pthread_create yield peer");
	pin_current(opts->cpu_a);
	for (round = 1; round <= total_rounds; round++) {
		struct result_row *row = &rows[round - 1];
		uint64_t i, start, peer_errors;

		wait_atomic_u64(&state.ready_generation, round,
				&state.thread_error, "yield peer readiness timeout");
		start = now_ns();
		atomic_store_explicit(&state.go_generation, round,
				      memory_order_release);
		for (i = 0; i < opts->ops; i++) {
			if (syscall(SYS_sched_yield) == 0)
				row->success_ops++;
			else
				row->errors++;
		}
		atomic_store_explicit(&state.main_done_generation, round,
				      memory_order_release);
		wait_atomic_u64(&state.peer_done_generation, round,
				&state.thread_error, "yield peer completion timeout");
		row->elapsed_ns = now_ns() - start;
		row->call_ns_sum = row->elapsed_ns;
		peer_errors = atomic_load_explicit(&state.cumulative_errors,
						  memory_order_acquire);
		row->errors += peer_errors - previous_peer_errors;
		previous_peer_errors = peer_errors;
		row->requested_ops = 2 * opts->ops;
		row->success_ops = row->requested_ops - row->errors;
		row->caller_tid = (pid_t)syscall(SYS_gettid);
		row->peer_tid = (pid_t)atomic_load_explicit(&state.peer_tid,
							 memory_order_acquire);
		row->final_cpu = sched_getcpu();
	}
	if (pthread_join(peer, NULL))
		die("pthread_join yield peer");
	if (atomic_load_explicit(&state.thread_error, memory_order_acquire))
		fail("yield peer failed");
}

static struct result_row run_setparam(uint64_t ops)
{
	struct result_row row = {
		.requested_ops = ops,
		.caller_tid = (pid_t)syscall(SYS_gettid),
	};
	struct sched_param param = { .sched_priority = 0 };
	uint64_t i, start = now_ns();

	for (i = 0; i < ops; i++) {
		if (syscall(SYS_sched_setparam, 0, &param) == 0)
			row.success_ops++;
		else
			row.errors++;
	}
	row.elapsed_ns = now_ns() - start;
	row.call_ns_sum = row.elapsed_ns;
	row.final_cpu = sched_getcpu();
	return row;
}

static void *affinity_target_main(void *opaque)
{
	struct affinity_state *state = opaque;
	cpu_set_t mask;
	uint64_t heartbeat = 0;

	set_thread_name("sf-target");
	make_single_cpu_mask(&mask, state->target_cpu);
	if (sched_setaffinity(0, sizeof(mask), &mask)) {
		atomic_store_explicit(&state->thread_error, errno ?: EIO,
				      memory_order_release);
		return NULL;
	}
	state->target_tid = (pid_t)syscall(SYS_gettid);
	atomic_store_explicit(&state->target_last_cpu, sched_getcpu(),
			      memory_order_release);
	atomic_store_explicit(&state->target_ready, 1, memory_order_release);
	while (!atomic_load_explicit(&state->stop_target, memory_order_acquire)) {
		heartbeat++;
		if ((heartbeat & 1023) == 0) {
			atomic_store_explicit(&state->target_heartbeat, heartbeat,
					      memory_order_release);
			atomic_store_explicit(&state->target_last_cpu,
					      sched_getcpu(), memory_order_release);
		}
		cpu_relax();
	}
	atomic_store_explicit(&state->target_heartbeat, heartbeat,
			      memory_order_release);
	atomic_store_explicit(&state->target_last_cpu, sched_getcpu(),
			      memory_order_release);
	return NULL;
}

static int set_target_cpu(pid_t tid, int cpu)
{
	cpu_set_t mask;

	make_single_cpu_mask(&mask, cpu);
	return (int)syscall(SYS_sched_setaffinity, tid, sizeof(mask), &mask);
}

static void *affinity_controller_b_main(void *opaque)
{
	struct affinity_state *state = opaque;
	cpu_set_t mask;
	uint64_t i, call_ns = 0, success = 0, errors = 0;

	set_thread_name("sf-ctl-b");
	atomic_store_explicit(&state->controller_b_tid,
			      (int)syscall(SYS_gettid), memory_order_release);
	make_single_cpu_mask(&mask, state->cpu_b);
	if (sched_setaffinity(0, sizeof(mask), &mask)) {
		atomic_store_explicit(&state->thread_error, errno ?: EIO,
				      memory_order_release);
		return NULL;
	}
	atomic_store_explicit(&state->controller_b_ready, 1, memory_order_release);
	for (i = 1; i < state->ops; i += 2) {
		uint64_t start;

		wait_atomic_u64(&state->turn, i, &state->thread_error,
				"affinity controller B turn timeout");
		start = now_ns();
		if (set_target_cpu(state->target_tid, state->cpu_a) == 0)
			success++;
		else
			errors++;
		call_ns += now_ns() - start;
		atomic_store_explicit(&state->turn, i + 1, memory_order_release);
	}
	atomic_store_explicit(&state->controller_b_call_ns, call_ns,
			      memory_order_release);
	atomic_store_explicit(&state->controller_b_success, success,
			      memory_order_release);
	atomic_store_explicit(&state->controller_b_errors, errors,
			      memory_order_release);
	return NULL;
}

static void wait_target_on_cpu(struct affinity_state *state, int cpu,
			       uint64_t heartbeat_before)
{
	uint64_t start = now_ns();

	while (atomic_load_explicit(&state->target_last_cpu,
				   memory_order_acquire) != cpu ||
	       atomic_load_explicit(&state->target_heartbeat,
				   memory_order_acquire) <= heartbeat_before) {
		if (atomic_load_explicit(&state->thread_error,
					memory_order_acquire))
			fail("affinity target failed");
		if (now_ns() - start > WAIT_TIMEOUT_NS)
			fail("affinity target CPU acknowledgement timeout");
		syscall(SYS_sched_yield);
	}
}

static void verify_target_mask(pid_t tid, int cpu)
{
	cpu_set_t expected, observed;

	make_single_cpu_mask(&expected, cpu);
	if (sched_getaffinity(tid, sizeof(observed), &observed))
		die("sched_getaffinity target");
	if (!CPU_EQUAL(&expected, &observed))
		fail("target affinity mask does not match the expected singleton");
}

static struct result_row run_affinity_round(const struct options *opts,
				     bool queued)
{
	struct affinity_state state;
	pthread_t target, controller_b;
	struct result_row row = {
		.requested_ops = opts->ops,
		.caller_tid = (pid_t)syscall(SYS_gettid),
	};
	uint64_t i, start;
	bool has_controller_b = false;

	memset(&state, 0, sizeof(state));
	atomic_init(&state.turn, 0);
	atomic_init(&state.target_heartbeat, 0);
	atomic_init(&state.controller_b_call_ns, 0);
	atomic_init(&state.controller_b_success, 0);
	atomic_init(&state.controller_b_errors, 0);
	atomic_init(&state.target_ready, 0);
	atomic_init(&state.controller_b_ready, 0);
	atomic_init(&state.controller_b_tid, 0);
	atomic_init(&state.target_last_cpu, -1);
	atomic_init(&state.thread_error, 0);
	atomic_init(&state.stop_target, 0);
	state.ops = opts->ops;
	state.cpu_a = opts->cpu_a;
	state.cpu_b = opts->cpu_b;
	state.target_cpu = queued ? opts->cpu_a : opts->cpu_b;
	if (pthread_create(&target, NULL, affinity_target_main, &state))
		die("pthread_create affinity target");
	wait_atomic_int(&state.target_ready, 1, &state.thread_error,
			"affinity target readiness timeout");
	row.target_tid = state.target_tid;
	if (queued) {
		if (pthread_create(&controller_b, NULL,
				   affinity_controller_b_main, &state))
			die("pthread_create affinity controller B");
		has_controller_b = true;
		wait_atomic_int(&state.controller_b_ready, 1,
				&state.thread_error,
				"affinity controller B readiness timeout");
		row.peer_tid = (pid_t)atomic_load_explicit(&state.controller_b_tid,
							memory_order_acquire);
	}
	pin_current(opts->cpu_a);
	row.heartbeat_before = atomic_load_explicit(&state.target_heartbeat,
						    memory_order_acquire);
	start = now_ns();
	if (queued) {
		uint64_t call_ns = 0, success = 0, errors = 0;

		for (i = 0; i < opts->ops; i += 2) {
			uint64_t call_start;

			wait_atomic_u64(&state.turn, i, &state.thread_error,
					"affinity controller A turn timeout");
			call_start = now_ns();
			if (set_target_cpu(state.target_tid, opts->cpu_b) == 0)
				success++;
			else
				errors++;
			call_ns += now_ns() - call_start;
			atomic_store_explicit(&state.turn, i + 1,
					      memory_order_release);
		}
		wait_atomic_u64(&state.turn, opts->ops, &state.thread_error,
				"affinity final turn timeout");
		row.call_ns_sum = call_ns +
			atomic_load_explicit(&state.controller_b_call_ns,
					     memory_order_acquire);
		row.success_ops = success +
			atomic_load_explicit(&state.controller_b_success,
					     memory_order_acquire);
		row.errors = errors +
			atomic_load_explicit(&state.controller_b_errors,
					     memory_order_acquire);
	} else {
		for (i = 0; i < opts->ops; i++) {
			if (set_target_cpu(state.target_tid, state.target_cpu) == 0)
				row.success_ops++;
			else
				row.errors++;
		}
		row.call_ns_sum = now_ns() - start;
	}
	row.elapsed_ns = now_ns() - start;
	if (has_controller_b && pthread_join(controller_b, NULL))
		die("pthread_join affinity controller B");
	verify_target_mask(state.target_tid, state.target_cpu);
	wait_target_on_cpu(&state, state.target_cpu, row.heartbeat_before);
	row.heartbeat_after = atomic_load_explicit(&state.target_heartbeat,
						   memory_order_acquire);
	row.final_cpu = sched_getcpu();
	atomic_store_explicit(&state.stop_target, 1, memory_order_release);
	if (pthread_join(target, NULL))
		die("pthread_join affinity target");
	if (atomic_load_explicit(&state.thread_error, memory_order_acquire))
		fail("affinity worker failed");
	return row;
}

static void validate_row(const struct options *opts, const struct result_row *row)
{
	if (row->errors || row->success_ops != row->requested_ops)
		fail("timed syscall semantic error");
	if (row->final_cpu != opts->cpu_a)
		fail("profile did not finish on CPU A");
	if (is_affinity_profile(opts->profile) &&
	    row->heartbeat_after <= row->heartbeat_before)
		fail("affinity target heartbeat did not advance");
	verify_current_state(opts->cpu_a);
}

static void print_header(void)
{
	printf("profile\tphase\tround\trequested_ops\tsuccess_ops\t"
	       "elapsed_ns\tcall_ns_sum\tns_per_op\terrors\tcpu_a\tcpu_b\t"
	       "caller_tid\tpeer_tid\ttarget_tid\theartbeat_before\t"
	       "heartbeat_after\tfinal_cpu\t"
	       "state_ok\n");
}

static void print_row(const struct options *opts, const char *phase,
		      unsigned int round, const struct result_row *row)
{
	double ns_per_op = (double)row->call_ns_sum /
			   (double)row->success_ops;

	printf("%s\t%s\t%u\t%llu\t%llu\t%llu\t%llu\t%.3f\t%llu\t"
	       "%d\t%d\t%d\t%d\t%d\t%llu\t%llu\t%d\t1\n",
	       opts->profile_name, phase, round,
	       (unsigned long long)row->requested_ops,
	       (unsigned long long)row->success_ops,
	       (unsigned long long)row->elapsed_ns,
	       (unsigned long long)row->call_ns_sum, ns_per_op,
	       (unsigned long long)row->errors, opts->cpu_a, opts->cpu_b,
	       row->caller_tid, row->peer_tid, row->target_tid,
	       (unsigned long long)row->heartbeat_before,
	       (unsigned long long)row->heartbeat_after, row->final_cpu);
	fflush(stdout);
}

int main(int argc, char **argv)
{
	struct options opts = parse_options(argc, argv);
	cpu_set_t original_affinity;
	unsigned int total_rounds = opts.warmups + opts.rounds;
	struct result_row *dual_rows = NULL;
	unsigned int round;

	set_thread_name("sf-main");

	if (sched_getaffinity(0, sizeof(original_affinity), &original_affinity))
		die("sched_getaffinity original");
	require_cpu_allowed(&original_affinity, opts.cpu_a);
	if (opts.cpu_b >= 0)
		require_cpu_allowed(&original_affinity, opts.cpu_b);
	if (opts.profile == PROFILE_YIELD_DUAL) {
		dual_rows = calloc(total_rounds, sizeof(*dual_rows));
		if (!dual_rows)
			die("calloc yield rows");
		run_yield_dual(&opts, total_rounds, dual_rows);
	} else {
		pin_current(opts.cpu_a);
	}
	verify_current_state(opts.cpu_a);
	if (opts.profile == PROFILE_SETPARAM_CURRENT) {
		if (ks_sched_setattr(0, 0))
			die("restore default slice before setparam-current");
	} else if (opts.profile == PROFILE_SETPARAM_FAST_CONTROL) {
		if (ks_sched_setattr(0, 10 * NS_PER_MS))
			die("set custom slice before setparam control");
	}

	print_header();
	for (round = 0; round < total_rounds; round++) {
		struct result_row row;
		const char *phase = round < opts.warmups ? "warmup" : "measured";
		unsigned int visible_round = round < opts.warmups ? round :
					     round - opts.warmups;

		if (opts.profile == PROFILE_YIELD_DUAL) {
			row = dual_rows[round];
		} else if (opts.profile == PROFILE_YIELD_SOLO) {
			row = run_yield_solo(opts.ops);
		} else if (opts.profile == PROFILE_SETPARAM_CURRENT ||
			   opts.profile == PROFILE_SETPARAM_FAST_CONTROL) {
			row = run_setparam(opts.ops);
		} else {
			row = run_affinity_round(&opts,
				opts.profile == PROFILE_AFFINITY_QUEUED);
		}
		validate_row(&opts, &row);
		print_row(&opts, phase, visible_round, &row);
	}
	if (opts.profile == PROFILE_SETPARAM_CURRENT ||
	    opts.profile == PROFILE_SETPARAM_FAST_CONTROL) {
		if (ks_sched_setattr(0, 0))
			die("restore default slice after setparam profile");
	}
	if (sched_setaffinity(0, sizeof(original_affinity), &original_affinity))
		die("restore original affinity");
	free(dual_rows);
	return EXIT_SUCCESS;
}

Re: [RFC] sched/fair: avoid redundant updates for unchanged default slices

[Chengfeng Lin]

[-- Attachment #1: Type: text/plain, Size: 279 bytes --]

Hi,

The inline patch in my previous message lost its tab indentation when
sent. I have attached the original patch to preserve the formatting. It
passes git apply --check against v7.2.

The code and test results are unchanged. Sorry for the formatting issue.

Thanks,
Chengfeng

[-- Attachment #2: default-slice-noop-v3.patch --]
[-- Type: text/x-patch, Size: 1751 bytes --]

diff --git a/kernel/sched/syscalls.c b/kernel/sched/syscalls.c
--- a/kernel/sched/syscalls.c
+++ b/kernel/sched/syscalls.c
@@ -491,6 +491,39 @@ req_priv:
 	return 0;
 }
 
+static bool fair_params_changed(struct task_struct *p,
+				const struct sched_attr *attr)
+{
+	if (attr->sched_nice != task_nice(p))
+		return true;
+
+	if (attr->sched_runtime == p->se.slice)
+		return false;
+
+	/* Preserve the existing path for PI-boosted and sched_ext tasks. */
+	if (p->sched_class != &fair_sched_class)
+		return true;
+
+#ifdef CONFIG_UCLAMP_TASK
+	/*
+	 * Unchanged fair parameters can still require an implicit clamp reset,
+	 * for example after a task was PI-boosted into the RT class.
+	 */
+	if ((!p->uclamp_req[UCLAMP_MIN].user_defined &&
+	     p->uclamp_req[UCLAMP_MIN].value != uclamp_none(UCLAMP_MIN)) ||
+	    (!p->uclamp_req[UCLAMP_MAX].user_defined &&
+	     p->uclamp_req[UCLAMP_MAX].value != uclamp_none(UCLAMP_MAX)))
+		return true;
+#endif
+
+	/*
+	 * A zero runtime requests the default slice, whereas se.slice stores
+	 * its actual duration. Keep the update if the mode or default changed.
+	 */
+	return attr->sched_runtime || p->se.custom_slice ||
+		p->se.slice != READ_ONCE(sysctl_sched_base_slice);
+}
+
 int __sched_setscheduler(struct task_struct *p,
 			 const struct sched_attr *attr,
 			 bool user, bool pi)
@@ -588,9 +621,7 @@ recheck:
 	 * but store a possible modification of reset_on_fork.
 	 */
 	if (unlikely(policy == p->policy)) {
-		if (fair_policy(policy) &&
-		    (attr->sched_nice != task_nice(p) ||
-		     (attr->sched_runtime != p->se.slice)))
+		if (fair_policy(policy) && fair_params_changed(p, attr))
 			goto change;
 		if (rt_policy(policy) && attr->sched_priority != p->rt_priority)
 			goto change;

Re: [RFC] sched/fair: avoid redundant updates for unchanged default slices

[Peter Zijlstra]

On Fri, Oct 09, 2026 at 11:49:40PM +0800, Chengfeng Lin wrote:
> Hi,
> 
> Repeated sched_setparam() calls with unchanged parameters still take the
> parameter-update path for a SCHED_OTHER task using the default slice.
> A small prototype reduced this cost from 618.18 to 163.23 ns per call on
> v7.2, saving about 455 ns (73.6%). I would appreciate feedback on this
> optimization.
> 

> This
> matters after an RT priority-inheritance boost has ended. Existing
> permission and LSM checks remain before the shortcut; explicit uclamp
> updates and reset-on-fork handling are unchanged.

Pff, so much text to filter just to find the issue. So you're saying
that you saw PI cost increase due to this change?

> Would this be a useful shortcut, or are there other updates that this
> unchanged-parameter case must retain?
> 
> Thanks,
> Chengfeng
> 
> Assisted-by: LLM
> 
> ---
> diff --git a/kernel/sched/syscalls.c b/kernel/sched/syscalls.c
> --- a/kernel/sched/syscalls.c
> +++ b/kernel/sched/syscalls.c
> @@ -491,6 +491,39 @@ req_priv:
>         return 0;
>  }
> 
> +static bool fair_params_changed(struct task_struct *p,
> +                               const struct sched_attr *attr)
> +{
> +       if (attr->sched_nice != task_nice(p))
> +               return true;
> +
> +       if (attr->sched_runtime == p->se.slice)
> +               return false;
> +
> +       /* Preserve the existing path for PI-boosted and sched_ext tasks. */
> +       if (p->sched_class != &fair_sched_class)
> +               return true;
> +
> +#ifdef CONFIG_UCLAMP_TASK
> +       /*
> +        * Unchanged fair parameters can still require an implicit clamp reset,
> +        * for example after a task was PI-boosted into the RT class.
> +        */
> +       if ((!p->uclamp_req[UCLAMP_MIN].user_defined &&
> +            p->uclamp_req[UCLAMP_MIN].value != uclamp_none(UCLAMP_MIN)) ||
> +           (!p->uclamp_req[UCLAMP_MAX].user_defined &&
> +            p->uclamp_req[UCLAMP_MAX].value != uclamp_none(UCLAMP_MAX)))
> +               return true;
> +#endif
> +
> +       /*
> +        * A zero runtime requests the default slice, whereas se.slice stores
> +        * its actual duration. Keep the update if the mode or default changed.
> +        */
> +       return attr->sched_runtime || p->se.custom_slice ||
> +               p->se.slice != READ_ONCE(sysctl_sched_base_slice);
> +}

Egads, this is unreadable. Also, perhaps wrong. Suppose the case where
attr->sched_runtime == se->slice, but !se->custom_slice. Or suppose the
slice matches, but you still need the change due to uclamp.

Anyway, are you trying to say something like so?

	if (!!attr->sched_runtime != !!se->custom_slice)
		return true;

	if (attr->sched_runtime && attr->sched_runtime != se->slice)
		return true;

Re: [RFC] sched/fair: avoid redundant updates for unchanged default slices

[Chengfeng Lin]

[-- Attachment #1: Type: text/plain, Size: 2343 bytes --]

Hi Peter,

Sorry for the long and confusing explanation. The performance test was
not a PI test: it repeatedly called sched_setparam(0, priority=0) from an
ordinary SCHED_OTHER task using the default slice. PI was only part of
the correctness checks.

You are right that equal runtimes alone are not enough. Both problems
occur in unmodified v7.2 and my previous prototype. The attached revision
uses your mode-first comparison and fixes both in the tests:

- Requesting a custom slice now switches the task to custom mode, even
  if its duration equals the current default. Later changes to the
  default no longer change that task's slice.
- In the RT-PI test, sched_setparam() resets the implicit uclamp_min
  from 1024 to 0 after the boost ends, even though the slice duration
  is unchanged.

The revision and unmodified v7.2 also passed the basic syscall checks
and the other controls: RT/DL PI, default-duration changes, explicit
uclamp requests and bounded concurrent default changes.
The revision keeps the update path when the default
duration has changed, an implicit clamp needs resetting, or the task is
not currently in the fair class. Explicit uclamp requests still take the
existing update path.

I reran the benchmark with the same parameters on unmodified v7.2,
the revision, and unmodified v7.2 again. These are new measurements,
not the previous prototype's numbers (ns per sched_setparam call):

                         Original A    Revision    Original B
  Unchanged default slice    615.98      162.39        617.03
  Unchanged custom slice     162.69      163.22        162.45

The default-slice case saves about 454 ns (73.66%); the existing custom
fast path costs about 0.65 ns more (0.40%), relative to the midpoint of
the two original runs. The runs used the same GCC 15.2 build settings
apart from the release suffix, CPU 2 on the same bare-metal machine,
full preemption, performance mode with turbo disabled, and 15 samples
of 500,000 calls per case per boot. Control drift was below 0.18% and
within-run CV below 0.30%. Probes were used separately, not during timing.

The attached default-slice-peter-v4.patch is the revision tested here,
against unmodified v7.2. Does this look like a reasonable way to keep
the unchanged-parameter fast path while handling these cases?

Thanks,
Chengfeng

Assisted-by: LLM

[-- Attachment #2: default-slice-peter-v4.patch --]
[-- Type: text/x-patch, Size: 1865 bytes --]

Subject: [RFC prototype] sched/fair: distinguish slice mode before skipping updates

Use separate checks for default/custom mode and requested duration.
Keep stale-default refreshes, implicit clamp resets and the full
update path outside the actual fair class. Equal numeric runtimes
alone must not skip a requested mode change or clamp reset.

This experimental prototype was prepared with LLM assistance.
It is not a signed-off submission.

---
--- a/kernel/sched/syscalls.c
+++ b/kernel/sched/syscalls.c
@@ -488,6 +488,38 @@
 		return -EPERM;
 
 	return 0;
+}
+
+static bool fair_params_changed(struct task_struct *p,
+				const struct sched_attr *attr)
+{
+	if (attr->sched_nice != task_nice(p))
+		return true;
+
+	/* Keep the full update path for PI-boosted and sched_ext tasks. */
+	if (p->sched_class != &fair_sched_class)
+		return true;
+
+	if (!!attr->sched_runtime != !!p->se.custom_slice)
+		return true;
+
+	if (attr->sched_runtime && attr->sched_runtime != p->se.slice)
+		return true;
+
+	if (!attr->sched_runtime &&
+	    p->se.slice != READ_ONCE(sysctl_sched_base_slice))
+		return true;
+
+#ifdef CONFIG_UCLAMP_TASK
+	/* Equal fair parameters do not rule out an implicit clamp reset. */
+	if ((!p->uclamp_req[UCLAMP_MIN].user_defined &&
+	     p->uclamp_req[UCLAMP_MIN].value != uclamp_none(UCLAMP_MIN)) ||
+	    (!p->uclamp_req[UCLAMP_MAX].user_defined &&
+	     p->uclamp_req[UCLAMP_MAX].value != uclamp_none(UCLAMP_MAX)))
+		return true;
+#endif
+
+	return false;
 }
 
 int __sched_setscheduler(struct task_struct *p,
@@ -589,8 +621,7 @@
 	 */
 	if (unlikely(policy == p->policy)) {
 		if (fair_policy(policy) &&
-		    (attr->sched_nice != task_nice(p) ||
-		     (attr->sched_runtime != p->se.slice)))
+		    fair_params_changed(p, attr))
 			goto change;
 		if (rt_policy(policy) && attr->sched_priority != p->rt_priority)
 			goto change;

Re: [RFC] sched/fair: avoid redundant updates for unchanged default slices

[Peter Zijlstra]

On Tue, Oct 13, 2026 at 05:58:17PM +0800, Chengfeng Lin wrote:

> The attached default-slice-peter-v4.patch is the revision tested here,
> against unmodified v7.2. Does this look like a reasonable way to keep
> the unchanged-parameter fast path while handling these cases?

Yes, this looks reasonable.