Announcing Sched QoS v0.1-alpha

Announcing Sched QoS v0.1-alpha

[Qais Yousef]

Hi everyone

This is the first announcement of Sched QoS 0.1-alpha release. This is still at
PoC stage and NOT production ready.

	https://github.com/qais-yousef/schedqos

This is a follow of LPC 2025 discussion about Userspace Assissted Scheduling via Sched QoS

	https://lpc.events/event/19/contributions/2089/


Background and Concepts
=======================

The world has changed a _little_ bit in the past 30 years..

Modern systems have sophisticated hardware that comes in all shapes and colors.
How software is written to interact with the modern hardware hasn't changed
though. Kernel had to keep up with hardware, but userspace didn't. POSIX is
ancient and didn't evolve to help OS and application writers how to deal with
these changes.

A major problem faced by many workloads is how to manage resources better so
that important workloads can get the latency or throughput they need, while
still co-exist alongside each others, and while being oblivious of the hardware
they are running on or kernel (scheduler) implementation details.

Many discussion in this area focused on what kernel interface should look like,
but I think this is the wrong approach. The same way we have libc, pthread etc
to define interface to do common operations that are detached from the OS, we
need to detach how workloads are described from OS specifics. And of course
from hardware by implication.

Sched QoS utility aims to do that. And it tries to take the unique approach of
Zero-API based approach.

One of the biggest challenges to adding any new API is the adoption time. It
can easily span 18-14 months if not more. Rolling things out for users to see
benefits has a substantial delay, iterating and reacting to problems will take
similar long time. All in all to reach a maturity point this can end up far
ahead in the future that adoption can never materialize. And it makes the
burden and the par to getting things right the first time very high that
progress can stifle at discussion point.

Another major challenge is trusting application to provide the right hints.
Managing potential abusers can be difficult (doable, but back to previous point
can lead to stifling discussion to agree on the 'right' approach). But also the
potential ABI implications that can make easy evolution and fast iteration
hard.

With config based hinting we eliminate these. Admins (which on many systems is
the user, because it's Linux..) will have to choose and approve the config to
be applied for any app.  It also provides additional flexibility where
a workload is 'important' on one system, but is actually a 'background' on
another. Group control (discussed at LPC) is better way to manage this, but
generally it inherently better handles potential perf/power trade-offs
variations if tagging could be done differently to suite different needs. e.g:
you can have more than one way to tag an application to suite potential
different needs rather than being stuck with a single one imposed on you.

By using NETLINK to listen to when new tasks are forked and processes are
created, we can easily create a config based system to auto tag tasks based on
userspace description.

For example the following config

	{
		"chrome": {
			"qos": "QOS_USER_INTERACTIVE"
		},
		"gcc": {
			"qos": "QOS_BACKGROUND"
		}
	}

is a poor man's way to tell the system the user wants to treat Chrome as
a user_interactive application and gcc (compiler) as a background one.

This is a provisional way for 'easy' tagging. The real intended use is for
specific tasks within an application (process) to be tagged to describe their
individual role

	{
		"chrome": {
			"thread_qos": {
				"chrome": "QOS_USER_INTERACTIVE",
				"some_bg_task": "QOS_BACKGROUND"
			}
		}
	}

so that real interactive tasks are given the resource they need, and the noise
is reduced to just that. Note by default everything will be treated as
QOS_DEFAULT, which is set to match QOS_UTILITY. IOW everything is assumed to be
random 'noise' by default. Which makes delivering better beset-effort QoS
easier.

Roles
=====

This model is based on existing one shipped in the industry [1] that its users
are happy with. It breaks down the tasks' role into 4 classes:

* USER_INTERACTIVE:	Requires immediate response
* USER_INITIATED:	Tolerates short latencies, but must get work done quickly still
* UTILITY:		Tolerates long delays, but not prolonged ones
* BACKGROUND:		Doesn't mind prolonged delays
* DEFAULT:		All untagged tasks will get this category which will map to utility.

EEVDF should allow us to describe these different levels via specifying
different runtime (custom slice) to each class. Shortest slice should still be
long enough not to sacrifice throughput. Nice values will operate as bandwidth
control so that long running user_interactive tasks can't be starved by long
running background ones if they had to run on the same CPU under overloaded
scenarios. uclamp_max help constraint power impact and access to expensive
highest performance levels.

Mapping
-------

	{
		"QOS_USER_INTERACTIVE": {
			"sched_policy": "SCHED_NORMAL",
			"sched_nice": -4,
			"sched_runtime": 8000000,
			"sched_util_max": 1024
		},
		"QOS_USER_INITIATED": {
			"sched_policy": "SCHED_NORMAL",
			"sched_nice": -2,
			"sched_runtime": 12000000,
			"sched_util_max": 768
		},
		"QOS_UTILITY": {
			"sched_policy": "SCHED_BATCH",
			"sched_nice": 2,
			"sched_runtime": 16000000,
			"sched_util_max": 512
		},
		"QOS_BACKGROUND": {
			"sched_policy": "SCHED_BATCH",
			"sched_nice": 4,
			"sched_runtime": 20000000,
			"sched_util_max": 256
		},
		"QOS_DEFAULT": {
			"sched_policy": "SCHED_BATCH",
			"sched_nice": 2,
			"sched_runtime": 16000000,
			"sched_util_max": 512
		}
	}


Caveats
=======

AUTOGROUP and cgroup cpu controller must be disabled for maximum effectiveness.
We assume a flat hierarchy and per-task description to keep the system under
control.

If they are enabled, it is hard to distinguish between background and user
interactive tasks  across processes due to imposed fairness at group level.
Most notably under loaded scenarios with a lot of long running tasks.

It is believed that a flat hierarchy is best approach and using per-task
tagging combined with simple group control to ensure roles of process and tasks
are described simply, yet sufficiently, to get the desired behavior with the
least complexity and maximum portability/flexibility.


Next steps
==========

The current setup is usable and should provide tangible results for those
interested. Corner cases where it fails will be visible though under
comprehensive testing. See schbench+kernel build results below for example.

I won't repeat our LPC discussion, but we need multi modal wake up path and
coherent decision making between wake up and load balancer. Both items are
being worked on already. Push based load balance for fair is on the list [2].

For best perf/watt under schedutil, we need to introduce the concept of rampup
multiplier to help counter DVFS related latencies. Also this is WIP and there
were patches in the past [3] that will be rebased and reposted.

Idle states can be a problem for performance and power and scheduler today
doesn't take them into account at all.

Performance Inversion and Inheritance issues are common in practice and require
Proxy Execution and teaching libc and languages to move to futex_pi by default.
In practice folks will see latency/perf issues in their P99 and max at least
- severity of which will depend on the use case. See discussion on Enable PI by
default in userspace [4].

We also need to add a new unfair pi lock that doesn't operate on strict order
which is critical for performance [5]. I have gathered some results privately
that using futex_pi causes performance regressions.

The utility itself needs to handle group control still. We need to extend
NETLINK to send events when task move group and introduce a group level control
of what QoS is allowed or not. The goal is to piggy-back on cgroup but provide
user space annotation of QoS. Which is a simple allow/disallow to prevent
user_interactive tasks for instance when a task is in a 'background' group.
Think an app that is minimized or a browser tab that is hidden. This group
control might need integration with window managers so that this control is
transparently handled for all apps.

There are also problems that are not seen outside of Linux ecosystem that might
require an extra dimension to annotate tasks that can be memory sensitive
(particularly NUMA). I call this new dimension Execution Profile. But this is
an area that requires further discussion in one of the upcoming conferences and
a dedicate thread on the list. Cache Aware Scheduling can particularly be
simplified if such annotation can be provided by userspace - which this
approach should make it easy to implement and start experimenting with.

We also need to ensure sched_attr is locked and can only be modified by
schedqos utility so that it can be the sole orchestrator for managing the
behavior of the tasks.

The project would hopefully move to kernel.org and get contributions as part of
usual kernel/scheduler development process.


Results
=======

Based on tip/sched/core: 8d16e3c6f844

Cyclictest + hackbench
----------------------

	hackbench -T -p -l 60000 -g 2 &
	sleep 1
	sudo nice -n 0 cyclictest -t 1 -i 1777 -D 30 -h 20000 -q

	{
		"hackbench": {
			"qos": "QOS_BACKGROUND"
		},
		"cyclictest": {
			"qos": "QOS_USER_INTERACTIVE"
		}
	}

Default:

	# Min Latencies: 00004
	# Avg Latencies: 00062
	# Max Latencies: 04426

With schedqos:

	# Min Latencies: 00003
	# Avg Latencies: 00053
	# Max Latencies: 01246

schbench + kernel build
-----------------------

AUTOGROUP and CPU controller were disabled otherwise you won't see a difference
due to imposed fairness at group level

	{
		"make": {
			"qos": "QOS_BACKGROUND"
		},
		"gcc": {
			"qos": "QOS_BACKGROUND"
		},
		"cc1": {
			"qos": "QOS_BACKGROUND"
		},
		"schbench": {
			"thread_qos": {
				"schbench-msg": "QOS_USER_INTERACTIVE",
				"schbench-worker": "QOS_USER_INITIATED"
			}
		}
	}

Default:

	Wakeup Latencies percentiles (usec) runtime 30 (s) (45018 total samples)
		  50.0th: 1618       (11870 samples)
		  90.0th: 3580       (18081 samples)
		* 99.0th: 4952       (3973 samples)
		  99.9th: 6760       (405 samples)
		  min=1, max=11092
	Request Latencies percentiles (usec) runtime 30 (s) (45042 total samples)
		  50.0th: 12464      (13518 samples)
		  90.0th: 22496      (18009 samples)
		* 99.0th: 36032      (4020 samples)
		  99.9th: 75904      (405 samples)
		  min=3860, max=144284
	RPS percentiles (requests) runtime 30 (s) (31 total samples)
		  20.0th: 1458       (7 samples)
		* 50.0th: 1486       (9 samples)
		  90.0th: 1566       (12 samples)
		  min=1420, max=1603
	average rps: 1501.40

With schedqos:

	Wakeup Latencies percentiles (usec) runtime 30 (s) (67556 total samples)
		  50.0th: 10         (15337 samples)
		  90.0th: 6488       (26386 samples)
		* 99.0th: 13168      (5961 samples)
		  99.9th: 19232      (607 samples)
		  min=1, max=32126
	Request Latencies percentiles (usec) runtime 30 (s) (67618 total samples)
		  50.0th: 6568       (21537 samples)
		  90.0th: 13744      (25740 samples)
		* 99.0th: 37312      (6064 samples)
		  99.9th: 65472      (602 samples)
		  min=3506, max=153046
	RPS percentiles (requests) runtime 30 (s) (31 total samples)
		  20.0th: 2084       (7 samples)
		* 50.0th: 2268       (9 samples)
		  90.0th: 2412       (12 samples)
		  min=1904, max=2523
	average rps: 2253.93

Notes
-----

Throughput is better, but latencies get worse due to a number of bugs Vincent
is addressing and patches for which are already on the list [6][7][8].

One observation is disabling RUN_TO_PARITY yields better latencies. But
hopefully this won't be necessary.

Also schbench can suffer from bad task placement where two worker threads end
up on the same CPU. Once multi-modal wake up path is ready, USER_INTERACTIVE
tasks should be spread across CPUs; current wake up behavior spreads based on
load only which means we can end up with these accidental bad placements that
require the scheduler to understand that for better latencies, it is best not
to place two tasks with short deadlines on the same CPU.


With schedqos + NO_RUN_TO_PARITY + [6][7][8] patches:

	Wakeup Latencies percentiles (usec) runtime 30 (s) (69260 total samples)
		  50.0th: 9          (26985 samples)
		  90.0th: 1342       (19692 samples)
		* 99.0th: 1686       (6280 samples)
		  99.9th: 2428       (570 samples)
		  min=1, max=5710
	Request Latencies percentiles (usec) runtime 30 (s) (69338 total samples)
		  50.0th: 8104       (20785 samples)
		  90.0th: 17696      (27798 samples)
		* 99.0th: 23648      (6173 samples)
		  99.9th: 35904      (623 samples)
		  min=3835, max=90821
	RPS percentiles (requests) runtime 30 (s) (31 total samples)
		  20.0th: 2228       (7 samples)
		* 50.0th: 2300       (9 samples)
		  90.0th: 2404       (12 samples)
		  min=2171, max=2556
	average rps: 2311.27


Contribute
==========

Help us get it better! See list of areas we need help with in CONTRIBUTE.md [9]

Or just give it a go and report any corner cases that doesn't work so we can
look at it and make sure we have a plan to get it fixed :)


[1] https://developer.apple.com/library/archive/documentation/Performance/Conceptual/EnergyGuide-iOS/PrioritizeWorkWithQoS.html#//apple_ref/doc/uid/TP40015243-CH39-SW1
[2] https://lore.kernel.org/lkml/20251202181242.1536213-1-vincent.guittot@linaro.org/
[3] https://lore.kernel.org/lkml/20240820163512.1096301-1-qyousef@layalina.io/
[4] https://lpc.events/event/19/contributions/2244/
[5] https://developer.apple.com/documentation/os/os_unfair_lock_lock
[6] https://lore.kernel.org/lkml/20260331162352.551501-1-vincent.guittot@linaro.org/
[7] https://lore.kernel.org/lkml/20260410132321.2897789-1-vincent.guittot@linaro.org/
[8] https://lore.kernel.org/lkml/20260410144808.2943278-1-vincent.guittot@linaro.org/
[9] https://github.com/qais-yousef/schedqos/blob/main/CONTRIBUTE.md

Re: Announcing Sched QoS v0.1-alpha

[Christian Loehle]

On 4/15/26 01:09, Qais Yousef wrote:
> Hi everyone
> 
> This is the first announcement of Sched QoS 0.1-alpha release. This is still at
> PoC stage and NOT production ready.
> 
> 	https://github.com/qais-yousef/schedqos
> 
> This is a follow of LPC 2025 discussion about Userspace Assissted Scheduling via Sched QoS
> 
> 	https://lpc.events/event/19/contributions/2089/
> 
> 
> Background and Concepts
> =======================
> 
> The world has changed a _little_ bit in the past 30 years..
> 
> Modern systems have sophisticated hardware that comes in all shapes and colors.
> How software is written to interact with the modern hardware hasn't changed
> though. Kernel had to keep up with hardware, but userspace didn't. POSIX is
> ancient and didn't evolve to help OS and application writers how to deal with
> these changes.
> 
> A major problem faced by many workloads is how to manage resources better so
> that important workloads can get the latency or throughput they need, while
> still co-exist alongside each others, and while being oblivious of the hardware
> they are running on or kernel (scheduler) implementation details.
> 
> Many discussion in this area focused on what kernel interface should look like,
> but I think this is the wrong approach. The same way we have libc, pthread etc
> to define interface to do common operations that are detached from the OS, we
> need to detach how workloads are described from OS specifics. And of course
> from hardware by implication.
> 
> Sched QoS utility aims to do that. And it tries to take the unique approach of
> Zero-API based approach.
> 
> One of the biggest challenges to adding any new API is the adoption time. It
> can easily span 18-14 months if not more. Rolling things out for users to see
> benefits has a substantial delay, iterating and reacting to problems will take
> similar long time. All in all to reach a maturity point this can end up far
> ahead in the future that adoption can never materialize. And it makes the
> burden and the par to getting things right the first time very high that
> progress can stifle at discussion point.
> 
> Another major challenge is trusting application to provide the right hints.
> Managing potential abusers can be difficult (doable, but back to previous point
> can lead to stifling discussion to agree on the 'right' approach). But also the
> potential ABI implications that can make easy evolution and fast iteration
> hard.
> 
> With config based hinting we eliminate these. Admins (which on many systems is
> the user, because it's Linux..) will have to choose and approve the config to
> be applied for any app.  It also provides additional flexibility where
> a workload is 'important' on one system, but is actually a 'background' on
> another. Group control (discussed at LPC) is better way to manage this, but
> generally it inherently better handles potential perf/power trade-offs
> variations if tagging could be done differently to suite different needs. e.g:
> you can have more than one way to tag an application to suite potential
> different needs rather than being stuck with a single one imposed on you.
> 
> By using NETLINK to listen to when new tasks are forked and processes are
> created, we can easily create a config based system to auto tag tasks based on
> userspace description.
> 
> For example the following config
> 
> 	{
> 		"chrome": {
> 			"qos": "QOS_USER_INTERACTIVE"
> 		},
> 		"gcc": {
> 			"qos": "QOS_BACKGROUND"
> 		}
> 	}
> 
> is a poor man's way to tell the system the user wants to treat Chrome as
> a user_interactive application and gcc (compiler) as a background one.
> 
> This is a provisional way for 'easy' tagging. The real intended use is for
> specific tasks within an application (process) to be tagged to describe their
> individual role
> 
> 	{
> 		"chrome": {
> 			"thread_qos": {
> 				"chrome": "QOS_USER_INTERACTIVE",
> 				"some_bg_task": "QOS_BACKGROUND"
> 			}
> 		}
> 	}
> 
> so that real interactive tasks are given the resource they need, and the noise
> is reduced to just that. Note by default everything will be treated as
> QOS_DEFAULT, which is set to match QOS_UTILITY. IOW everything is assumed to be
> random 'noise' by default. Which makes delivering better beset-effort QoS
> easier.
> 
> Roles
> =====
> 
> This model is based on existing one shipped in the industry [1] that its users
> are happy with. It breaks down the tasks' role into 4 classes:
> 
> * USER_INTERACTIVE:	Requires immediate response
> * USER_INITIATED:	Tolerates short latencies, but must get work done quickly still
> * UTILITY:		Tolerates long delays, but not prolonged ones
> * BACKGROUND:		Doesn't mind prolonged delays
> * DEFAULT:		All untagged tasks will get this category which will map to utility.
> 
> EEVDF should allow us to describe these different levels via specifying
> different runtime (custom slice) to each class. Shortest slice should still be
> long enough not to sacrifice throughput. Nice values will operate as bandwidth
> control so that long running user_interactive tasks can't be starved by long
> running background ones if they had to run on the same CPU under overloaded
> scenarios. uclamp_max help constraint power impact and access to expensive
> highest performance levels.
> 
> Mapping
> -------
> 
> 	{
> 		"QOS_USER_INTERACTIVE": {
> 			"sched_policy": "SCHED_NORMAL",
> 			"sched_nice": -4,
> 			"sched_runtime": 8000000,
> 			"sched_util_max": 1024
> 		},
> 		"QOS_USER_INITIATED": {
> 			"sched_policy": "SCHED_NORMAL",
> 			"sched_nice": -2,
> 			"sched_runtime": 12000000,
> 			"sched_util_max": 768
> 		},
> 		"QOS_UTILITY": {
> 			"sched_policy": "SCHED_BATCH",
> 			"sched_nice": 2,
> 			"sched_runtime": 16000000,
> 			"sched_util_max": 512
> 		},
> 		"QOS_BACKGROUND": {
> 			"sched_policy": "SCHED_BATCH",
> 			"sched_nice": 4,
> 			"sched_runtime": 20000000,
> 			"sched_util_max": 256
> 		},
> 		"QOS_DEFAULT": {
> 			"sched_policy": "SCHED_BATCH",
> 			"sched_nice": 2,
> 			"sched_runtime": 16000000,
> 			"sched_util_max": 512
> 		}
> 	}
> 
Is my understand correct that this is device-agnostic?
In particular sched_util_max seems very platform-dependent?
Also these could very well all land on the same big cluster and be effectively
void then.
And I don't think I generally buy the argument that uclamp_max is even a good
generic way to save power, not with these fixed (and arbitrary) values (e.g.
256 might be just the threshold to allow/require very inefficient OPPs that
happen frequently, in particular with the instability of utilization values
under sched_util_max / restricted compute capacity).

Re: Announcing Sched QoS v0.1-alpha

[Qais Yousef]

On 04/15/26 09:57, Christian Loehle wrote:

> > Roles
> > =====
> > 
> > This model is based on existing one shipped in the industry [1] that its users
> > are happy with. It breaks down the tasks' role into 4 classes:
> > 
> > * USER_INTERACTIVE:	Requires immediate response
> > * USER_INITIATED:	Tolerates short latencies, but must get work done quickly still
> > * UTILITY:		Tolerates long delays, but not prolonged ones
> > * BACKGROUND:		Doesn't mind prolonged delays
> > * DEFAULT:		All untagged tasks will get this category which will map to utility.
> > 
> > EEVDF should allow us to describe these different levels via specifying
> > different runtime (custom slice) to each class. Shortest slice should still be
> > long enough not to sacrifice throughput. Nice values will operate as bandwidth
> > control so that long running user_interactive tasks can't be starved by long
> > running background ones if they had to run on the same CPU under overloaded
> > scenarios. uclamp_max help constraint power impact and access to expensive
> > highest performance levels.
> > 
> > Mapping
> > -------
> > 
> > 	{
> > 		"QOS_USER_INTERACTIVE": {
> > 			"sched_policy": "SCHED_NORMAL",
> > 			"sched_nice": -4,
> > 			"sched_runtime": 8000000,
> > 			"sched_util_max": 1024
> > 		},
> > 		"QOS_USER_INITIATED": {
> > 			"sched_policy": "SCHED_NORMAL",
> > 			"sched_nice": -2,
> > 			"sched_runtime": 12000000,
> > 			"sched_util_max": 768
> > 		},
> > 		"QOS_UTILITY": {
> > 			"sched_policy": "SCHED_BATCH",
> > 			"sched_nice": 2,
> > 			"sched_runtime": 16000000,
> > 			"sched_util_max": 512
> > 		},
> > 		"QOS_BACKGROUND": {
> > 			"sched_policy": "SCHED_BATCH",
> > 			"sched_nice": 4,
> > 			"sched_runtime": 20000000,
> > 			"sched_util_max": 256
> > 		},
> > 		"QOS_DEFAULT": {
> > 			"sched_policy": "SCHED_BATCH",
> > 			"sched_nice": 2,
> > 			"sched_runtime": 16000000,
> > 			"sched_util_max": 512
> > 		}
> > 	}
> > 
> Is my understand correct that this is device-agnostic?

Ideally, yes.

> In particular sched_util_max seems very platform-dependent?

How come?

> Also these could very well all land on the same big cluster and be effectively
> void then.

Who said we are trying to avoid the big cluster? This setup is meaningful for
SMP systems too FWIW.

> And I don't think I generally buy the argument that uclamp_max is even a good
> generic way to save power, not with these fixed (and arbitrary) values (e.g.

Who said they are arbitrary? What is the right generic way to save power?

> 256 might be just the threshold to allow/require very inefficient OPPs that
> happen frequently, in particular with the instability of utilization values
> under sched_util_max / restricted compute capacity).

Have you actually encountered this scenario frequently? What was the power
impact?


Anyways. It seems you're jumping to a number of wrong conclusions.

Let me see if I can hopefully provide a helpful comprehensive answer.

The mapping is a simple config file that is meant to address as many systems as
possible so that it just works. But if there's a case where an admin thinks
they know better, it is a simple update to this config file locally with their
'fine tuned' values and `sudo schedqos restart --daemon` for the changes to
take effect. Not sure if you read USAGE.md, but running applications will see
the updated changes without having to restart them by simply restarting the
daemon.  There's an option, --configs-path, to provide a path to your own
config files.

uclamp_max values are not arbitrary because we have 4 levels, hence the 25%
increment (100%/4). Since we are told USER_INTERACTIVE are the most impactful
tasks, they are the only ones allowed to reach max perf. Given the exponential
nature of power/perf curve; cutting these top frequencies will be tangible for
all tasks that are deemed 'tolerating of some delays'.

Can someone do better for a specific system and use case? Sure. Do we block
them? No, as already answered above.

We don't care about capping to any particular CPU. This is scheduler detail
that must be handled transparently. The power saving should be impactful still
compared to what you get today since now only very few select tasks can drive
the system to the top, and very expensive, performance level.

For the inefficient OPPs. It seems you're referring to potentially uclamp_max
not allowing us to go to next frequency if it is more efficient. I need to
reinspect the code to confirm this will happen, but assuming it do, I doubt
that 256 will frequently land on those frequencies. And when it does, it would
be NOT really bad in practice. Why? Because today tasks are free to consume max
frequencies and we are doing okay. By capping them to 25% and 50% this is
much better than what you get today. The power difference between efficient and
inefficient OPPs is surely much (much!) smaller than the difference of running
at max vs 50% and 25%. So it is a question of minor fine tuning rather than
spilling power left and right. Ie, it is still a very impactful generic setup.

As for the platform dependent, I hope the above answered it already, but if
not, we really don't need to be precise. The goal is to restrict lower classes
from accessing higher freqs. Cutting 25% of the performance level (ie
frequencies) at each class is good enough of a description to shift frequency
usage towards the lower ones on any system.

Evolving this to do better is not hard if you have better ideas. And this is
what we are enabling now. You can tweak these mappings and run experiments and
iterate very quickly easily. And if you have a better way to do the mapping, we
can just update it and let everyone else reap the benefit ;-)

For a starter, I think these 25% increments should do well initially and might
not require fancier logic. But time will tell I guess :-)

For the record, I am worried about USER_INITIATED being capped to 75%. But
I think it is important to deliver meaningful perf and power difference between
classes. Since iterating and improving is 'cheap', I'd rather push my luck
first and get data to demonstrate how we need to do better.

Go ahead and give it a go. And if you hit corner cases it fails, let us know so
we can look at how we can do better.

Re: Announcing Sched QoS v0.1-alpha

[Barry Song]

On Wed, Apr 15, 2026 at 8:10 AM Qais Yousef <qyousef@layalina.io> wrote:
>
> Hi everyone
>
> This is the first announcement of Sched QoS 0.1-alpha release. This is still at
> PoC stage and NOT production ready.
>
>         https://github.com/qais-yousef/schedqos

Thanks for releasing this code—I’ve been looking forward to
it for quite some time.

I tried running it on arm64, but unfortunately it crashed at schedqos.c:144.

Program received signal SIGSEGV, Segmentation fault.
Download failed: Invalid argument.  Continuing without source file
./string/../sysdeps/aarch64/strcmp.S.
__GI_strcmp () at ../sysdeps/aarch64/strcmp.S:78
warning: 78 ../sysdeps/aarch64/strcmp.S: No such file or directory
(gdb) bt
#0  __GI_strcmp () at ../sysdeps/aarch64/strcmp.S:78
#1  0x0000aaaaaaaa2458 in main (argc=<optimized out>, argv=<optimized
out>) at schedqos.c:144

Then it seems that running "./schedqos start" is fine.

>
> This is a follow of LPC 2025 discussion about Userspace Assissted Scheduling via Sched QoS
>
>         https://lpc.events/event/19/contributions/2089/
>
>
> Background and Concepts
> =======================
>
> The world has changed a _little_ bit in the past 30 years..
>
> Modern systems have sophisticated hardware that comes in all shapes and colors.
> How software is written to interact with the modern hardware hasn't changed
> though. Kernel had to keep up with hardware, but userspace didn't. POSIX is
> ancient and didn't evolve to help OS and application writers how to deal with
> these changes.
>
> A major problem faced by many workloads is how to manage resources better so
> that important workloads can get the latency or throughput they need, while
> still co-exist alongside each others, and while being oblivious of the hardware
> they are running on or kernel (scheduler) implementation details.
>
> Many discussion in this area focused on what kernel interface should look like,
> but I think this is the wrong approach. The same way we have libc, pthread etc
> to define interface to do common operations that are detached from the OS, we
> need to detach how workloads are described from OS specifics. And of course
> from hardware by implication.
>
> Sched QoS utility aims to do that. And it tries to take the unique approach of
> Zero-API based approach.
>
> One of the biggest challenges to adding any new API is the adoption time. It
> can easily span 18-14 months if not more. Rolling things out for users to see
> benefits has a substantial delay, iterating and reacting to problems will take
> similar long time. All in all to reach a maturity point this can end up far
> ahead in the future that adoption can never materialize. And it makes the
> burden and the par to getting things right the first time very high that
> progress can stifle at discussion point.
>
> Another major challenge is trusting application to provide the right hints.
> Managing potential abusers can be difficult (doable, but back to previous point
> can lead to stifling discussion to agree on the 'right' approach). But also the
> potential ABI implications that can make easy evolution and fast iteration
> hard.
>
> With config based hinting we eliminate these. Admins (which on many systems is
> the user, because it's Linux..) will have to choose and approve the config to
> be applied for any app.  It also provides additional flexibility where
> a workload is 'important' on one system, but is actually a 'background' on
> another. Group control (discussed at LPC) is better way to manage this, but
> generally it inherently better handles potential perf/power trade-offs
> variations if tagging could be done differently to suite different needs. e.g:
> you can have more than one way to tag an application to suite potential
> different needs rather than being stuck with a single one imposed on you.
>
> By using NETLINK to listen to when new tasks are forked and processes are
> created, we can easily create a config based system to auto tag tasks based on
> userspace description.

We have a long history of relying on netlink for the device driver model
and udev. For threads, I guess the events would be much more frequent
than device hotplug/unplug—one app might create hundreds of threads.
We may end up relying on some ring-buffer-based event mechanism if we
eventually find that netlink itself becomes a bottleneck :-)

Now I see you are monitoring both PROC_EVENT_FORK and
PROC_EVENT_COMM. Wouldn’t this be sometimes duplicated, since
PR_SET_NAME may come soon after a thread is created?

>
> For example the following config
>
>         {
>                 "chrome": {
>                         "qos": "QOS_USER_INTERACTIVE"
>                 },
>                 "gcc": {
>                         "qos": "QOS_BACKGROUND"
>                 }
>         }
>
> is a poor man's way to tell the system the user wants to treat Chrome as
> a user_interactive application and gcc (compiler) as a background one.
>
> This is a provisional way for 'easy' tagging. The real intended use is for
> specific tasks within an application (process) to be tagged to describe their
> individual role
>
>         {
>                 "chrome": {
>                         "thread_qos": {
>                                 "chrome": "QOS_USER_INTERACTIVE",
>                                 "some_bg_task": "QOS_BACKGROUND"
>                         }
>                 }
>         }
>

I assume this depends on user code setting the thread name via
prctl(PR_SET_NAME, "name"), based on the code below?

static void iterate_threads(pid_t tgid)
{
        char task_path[256];
        snprintf(task_path, sizeof(task_path), "/proc/%d/task", tgid);

        DIR *tdir = opendir(task_path);
        struct dirent *tentry;

        while (tdir && (tentry = readdir(tdir)) != NULL) {
                if (is_numeric(tentry->d_name)) {
                        pid_t pid = atoi(tentry->d_name);
                        char comm[TASK_COMM_LEN];

                        if (get_comm_by_pid(pid, comm))
                                apply_thread_qos(pid, tgid, comm);
                }
        }
        if (tdir) closedir(tdir);
}

I assume this is quite common on Android, but in Linux
distributions, almost no applications set names for their
threads?

So, the project is trying to encourage people to set proper
names for their threads, right?

> so that real interactive tasks are given the resource they need, and the noise
> is reduced to just that. Note by default everything will be treated as
> QOS_DEFAULT, which is set to match QOS_UTILITY. IOW everything is assumed to be
> random 'noise' by default. Which makes delivering better beset-effort QoS
> easier.
>
> Roles
> =====
>
> This model is based on existing one shipped in the industry [1] that its users
> are happy with. It breaks down the tasks' role into 4 classes:
>
> * USER_INTERACTIVE:     Requires immediate response
> * USER_INITIATED:       Tolerates short latencies, but must get work done quickly still
> * UTILITY:              Tolerates long delays, but not prolonged ones
> * BACKGROUND:           Doesn't mind prolonged delays
> * DEFAULT:              All untagged tasks will get this category which will map to utility.
>
> EEVDF should allow us to describe these different levels via specifying
> different runtime (custom slice) to each class. Shortest slice should still be
> long enough not to sacrifice throughput. Nice values will operate as bandwidth
> control so that long running user_interactive tasks can't be starved by long
> running background ones if they had to run on the same CPU under overloaded
> scenarios. uclamp_max help constraint power impact and access to expensive
> highest performance levels.
>
> Mapping
> -------
>
>         {
>                 "QOS_USER_INTERACTIVE": {
>                         "sched_policy": "SCHED_NORMAL",
>                         "sched_nice": -4,
>                         "sched_runtime": 8000000,
>                         "sched_util_max": 1024
>                 },
>                 "QOS_USER_INITIATED": {
>                         "sched_policy": "SCHED_NORMAL",
>                         "sched_nice": -2,
>                         "sched_runtime": 12000000,
>                         "sched_util_max": 768
>                 },
>                 "QOS_UTILITY": {
>                         "sched_policy": "SCHED_BATCH",
>                         "sched_nice": 2,
>                         "sched_runtime": 16000000,
>                         "sched_util_max": 512
>                 },
>                 "QOS_BACKGROUND": {
>                         "sched_policy": "SCHED_BATCH",
>                         "sched_nice": 4,
>                         "sched_runtime": 20000000,
>                         "sched_util_max": 256
>                 },
>                 "QOS_DEFAULT": {
>                         "sched_policy": "SCHED_BATCH",
>                         "sched_nice": 2,
>                         "sched_runtime": 16000000,
>                         "sched_util_max": 512
>                 }
>         }
>

I now see a fairly simple call stack like
apply_thread_qos_tag() → sched_setattr(), which seems quite readable to me.

>
> Caveats
> =======
>
> AUTOGROUP and cgroup cpu controller must be disabled for maximum effectiveness.
> We assume a flat hierarchy and per-task description to keep the system under
> control.
>
> If they are enabled, it is hard to distinguish between background and user
> interactive tasks  across processes due to imposed fairness at group level.
> Most notably under loaded scenarios with a lot of long running tasks.
>
> It is believed that a flat hierarchy is best approach and using per-task
> tagging combined with simple group control to ensure roles of process and tasks
> are described simply, yet sufficiently, to get the desired behavior with the
> least complexity and maximum portability/flexibility.
>

I don’t see how this is related to your schedqos, since you’re just calling
sched_setattr(). That is fully consistent with the Linux kernel model.

Am I right in assuming that whatever you set via sched_setattr() will always
take effect even in a cgroup-based system?

>
> Next steps
> ==========
>
> The current setup is usable and should provide tangible results for those
> interested. Corner cases where it fails will be visible though under
> comprehensive testing. See schbench+kernel build results below for example.
>
> I won't repeat our LPC discussion, but we need multi modal wake up path and
> coherent decision making between wake up and load balancer. Both items are
> being worked on already. Push based load balance for fair is on the list [2].
>
> For best perf/watt under schedutil, we need to introduce the concept of rampup
> multiplier to help counter DVFS related latencies. Also this is WIP and there
> were patches in the past [3] that will be rebased and reposted.
>
> Idle states can be a problem for performance and power and scheduler today
> doesn't take them into account at all.
>
> Performance Inversion and Inheritance issues are common in practice and require
> Proxy Execution and teaching libc and languages to move to futex_pi by default.
> In practice folks will see latency/perf issues in their P99 and max at least
> - severity of which will depend on the use case. See discussion on Enable PI by
> default in userspace [4].
>
> We also need to add a new unfair pi lock that doesn't operate on strict order
> which is critical for performance [5]. I have gathered some results privately
> that using futex_pi causes performance regressions.

I can imagine that if we have too much priority inheritance, we could
end up hurting the system. Priority inheritance raises the priority of
tasks that should not necessarily be in the high-priority class. This is
especially problematic when the dependency chain is long and complex.

>
> The utility itself needs to handle group control still. We need to extend
> NETLINK to send events when task move group and introduce a group level control
> of what QoS is allowed or not. The goal is to piggy-back on cgroup but provide
> user space annotation of QoS. Which is a simple allow/disallow to prevent
> user_interactive tasks for instance when a task is in a 'background' group.
> Think an app that is minimized or a browser tab that is hidden. This group
> control might need integration with window managers so that this control is
> transparently handled for all apps.
>
> There are also problems that are not seen outside of Linux ecosystem that might
> require an extra dimension to annotate tasks that can be memory sensitive
> (particularly NUMA). I call this new dimension Execution Profile. But this is
> an area that requires further discussion in one of the upcoming conferences and
> a dedicate thread on the list. Cache Aware Scheduling can particularly be
> simplified if such annotation can be provided by userspace - which this
> approach should make it easy to implement and start experimenting with.
>
> We also need to ensure sched_attr is locked and can only be modified by
> schedqos utility so that it can be the sole orchestrator for managing the
> behavior of the tasks.
>
> The project would hopefully move to kernel.org and get contributions as part of
> usual kernel/scheduler development process.
>
>
> Results
> =======
>
> Based on tip/sched/core: 8d16e3c6f844
>
> Cyclictest + hackbench
> ----------------------
>
>         hackbench -T -p -l 60000 -g 2 &
>         sleep 1
>         sudo nice -n 0 cyclictest -t 1 -i 1777 -D 30 -h 20000 -q
>
>         {
>                 "hackbench": {
>                         "qos": "QOS_BACKGROUND"
>                 },
>                 "cyclictest": {
>                         "qos": "QOS_USER_INTERACTIVE"
>                 }
>         }
>

This raises a question: do you want app developers to write the
configuration file for their own apps, or do you want system
administrators to define a global configuration file?

Why is hackbench considered background and cyclictest considered
USER_INTERACTIVE? Who is supposed to know this?

App developers may not do this, and system administrators may lack
knowledge of specific apps.

BTW, assuming we have a phone with 300 installed applications, do we
need to configure each one individually, or can we use some shared
configuration? For example, on Android, threads may have the same name
across different applications, such as RenderThread, HeapTaskDaemon,
Binder:xxx etc.

> Default:
>
>         # Min Latencies: 00004
>         # Avg Latencies: 00062
>         # Max Latencies: 04426
>
> With schedqos:
>
>         # Min Latencies: 00003
>         # Avg Latencies: 00053
>         # Max Latencies: 01246
>
> schbench + kernel build
> -----------------------
>
> AUTOGROUP and CPU controller were disabled otherwise you won't see a difference
> due to imposed fairness at group level
>
>         {
>                 "make": {
>                         "qos": "QOS_BACKGROUND"
>                 },
>                 "gcc": {
>                         "qos": "QOS_BACKGROUND"
>                 },
>                 "cc1": {
>                         "qos": "QOS_BACKGROUND"
>                 },
>                 "schbench": {
>                         "thread_qos": {
>                                 "schbench-msg": "QOS_USER_INTERACTIVE",
>                                 "schbench-worker": "QOS_USER_INITIATED"
>                         }

I assume you modified the schbench? In my experiments, I always get the name
-schbench.

/proc/2892/task$ ls
2892  2893  2894  2895  2896  2897
/proc/2892/task$ cat 2892/comm
schbench
/proc/2892/task$ cat 2893/comm
schbench
/proc/2892/task$ cat 2894/comm
schbench
/proc/2892/task$ cat 2895/comm
schbench
/proc/2892/task$ cat 2896/comm
schbench
/proc/2892/task$ cat 2897/comm
schbench


>                 }
>         }
>
> Default:
>
>         Wakeup Latencies percentiles (usec) runtime 30 (s) (45018 total samples)
>                   50.0th: 1618       (11870 samples)
>                   90.0th: 3580       (18081 samples)
>                 * 99.0th: 4952       (3973 samples)
>                   99.9th: 6760       (405 samples)
>                   min=1, max=11092
>         Request Latencies percentiles (usec) runtime 30 (s) (45042 total samples)
>                   50.0th: 12464      (13518 samples)
>                   90.0th: 22496      (18009 samples)
>                 * 99.0th: 36032      (4020 samples)
>                   99.9th: 75904      (405 samples)
>                   min=3860, max=144284
>         RPS percentiles (requests) runtime 30 (s) (31 total samples)
>                   20.0th: 1458       (7 samples)
>                 * 50.0th: 1486       (9 samples)
>                   90.0th: 1566       (12 samples)
>                   min=1420, max=1603
>         average rps: 1501.40
>
> With schedqos:
>
>         Wakeup Latencies percentiles (usec) runtime 30 (s) (67556 total samples)
>                   50.0th: 10         (15337 samples)
>                   90.0th: 6488       (26386 samples)
>                 * 99.0th: 13168      (5961 samples)
>                   99.9th: 19232      (607 samples)
>                   min=1, max=32126
>         Request Latencies percentiles (usec) runtime 30 (s) (67618 total samples)
>                   50.0th: 6568       (21537 samples)
>                   90.0th: 13744      (25740 samples)
>                 * 99.0th: 37312      (6064 samples)
>                   99.9th: 65472      (602 samples)
>                   min=3506, max=153046
>         RPS percentiles (requests) runtime 30 (s) (31 total samples)
>                   20.0th: 2084       (7 samples)
>                 * 50.0th: 2268       (9 samples)
>                   90.0th: 2412       (12 samples)
>                   min=1904, max=2523
>         average rps: 2253.93
>
> Notes
> -----
>
> Throughput is better, but latencies get worse due to a number of bugs Vincent
> is addressing and patches for which are already on the list [6][7][8].
>
> One observation is disabling RUN_TO_PARITY yields better latencies. But
> hopefully this won't be necessary.
>
> Also schbench can suffer from bad task placement where two worker threads end
> up on the same CPU. Once multi-modal wake up path is ready, USER_INTERACTIVE
> tasks should be spread across CPUs; current wake up behavior spreads based on
> load only which means we can end up with these accidental bad placements that
> require the scheduler to understand that for better latencies, it is best not
> to place two tasks with short deadlines on the same CPU.
>
>
> With schedqos + NO_RUN_TO_PARITY + [6][7][8] patches:
>
>         Wakeup Latencies percentiles (usec) runtime 30 (s) (69260 total samples)
>                   50.0th: 9          (26985 samples)
>                   90.0th: 1342       (19692 samples)
>                 * 99.0th: 1686       (6280 samples)
>                   99.9th: 2428       (570 samples)
>                   min=1, max=5710
>         Request Latencies percentiles (usec) runtime 30 (s) (69338 total samples)
>                   50.0th: 8104       (20785 samples)
>                   90.0th: 17696      (27798 samples)
>                 * 99.0th: 23648      (6173 samples)
>                   99.9th: 35904      (623 samples)
>                   min=3835, max=90821
>         RPS percentiles (requests) runtime 30 (s) (31 total samples)
>                   20.0th: 2228       (7 samples)
>                 * 50.0th: 2300       (9 samples)
>                   90.0th: 2404       (12 samples)
>                   min=2171, max=2556
>         average rps: 2311.27
>
>
> Contribute
> ==========
>
> Help us get it better! See list of areas we need help with in CONTRIBUTE.md [9]
>
> Or just give it a go and report any corner cases that doesn't work so we can
> look at it and make sure we have a plan to get it fixed :)
>
>
> [1] https://developer.apple.com/library/archive/documentation/Performance/Conceptual/EnergyGuide-iOS/PrioritizeWorkWithQoS.html#//apple_ref/doc/uid/TP40015243-CH39-SW1
> [2] https://lore.kernel.org/lkml/20251202181242.1536213-1-vincent.guittot@linaro.org/
> [3] https://lore.kernel.org/lkml/20240820163512.1096301-1-qyousef@layalina.io/
> [4] https://lpc.events/event/19/contributions/2244/
> [5] https://developer.apple.com/documentation/os/os_unfair_lock_lock
> [6] https://lore.kernel.org/lkml/20260331162352.551501-1-vincent.guittot@linaro.org/
> [7] https://lore.kernel.org/lkml/20260410132321.2897789-1-vincent.guittot@linaro.org/
> [8] https://lore.kernel.org/lkml/20260410144808.2943278-1-vincent.guittot@linaro.org/
> [9] https://github.com/qais-yousef/schedqos/blob/main/CONTRIBUTE.md


Best Regards
Barry

Re: Announcing Sched QoS v0.1-alpha

[Qais Yousef]

On 04/17/26 19:26, Barry Song wrote:
> On Wed, Apr 15, 2026 at 8:10 AM Qais Yousef <qyousef@layalina.io> wrote:
> >
> > Hi everyone
> >
> > This is the first announcement of Sched QoS 0.1-alpha release. This is still at
> > PoC stage and NOT production ready.
> >
> >         https://github.com/qais-yousef/schedqos
> 
> Thanks for releasing this code—I’ve been looking forward to
> it for quite some time.

Thanks for trying it out.

> 
> I tried running it on arm64, but unfortunately it crashed at schedqos.c:144.
> 
> Program received signal SIGSEGV, Segmentation fault.
> Download failed: Invalid argument.  Continuing without source file
> ./string/../sysdeps/aarch64/strcmp.S.
> __GI_strcmp () at ../sysdeps/aarch64/strcmp.S:78
> warning: 78 ../sysdeps/aarch64/strcmp.S: No such file or directory
> (gdb) bt
> #0  __GI_strcmp () at ../sysdeps/aarch64/strcmp.S:78
> #1  0x0000aaaaaaaa2458 in main (argc=<optimized out>, argv=<optimized
> out>) at schedqos.c:144
> 
> Then it seems that running "./schedqos start" is fine.

Oh, I thought argp will throw an error and print usage. Pushed a quick fix.

> 
> >
> > This is a follow of LPC 2025 discussion about Userspace Assissted Scheduling via Sched QoS
> >
> >         https://lpc.events/event/19/contributions/2089/
> >
> >
> > Background and Concepts
> > =======================
> >
> > The world has changed a _little_ bit in the past 30 years..
> >
> > Modern systems have sophisticated hardware that comes in all shapes and colors.
> > How software is written to interact with the modern hardware hasn't changed
> > though. Kernel had to keep up with hardware, but userspace didn't. POSIX is
> > ancient and didn't evolve to help OS and application writers how to deal with
> > these changes.
> >
> > A major problem faced by many workloads is how to manage resources better so
> > that important workloads can get the latency or throughput they need, while
> > still co-exist alongside each others, and while being oblivious of the hardware
> > they are running on or kernel (scheduler) implementation details.
> >
> > Many discussion in this area focused on what kernel interface should look like,
> > but I think this is the wrong approach. The same way we have libc, pthread etc
> > to define interface to do common operations that are detached from the OS, we
> > need to detach how workloads are described from OS specifics. And of course
> > from hardware by implication.
> >
> > Sched QoS utility aims to do that. And it tries to take the unique approach of
> > Zero-API based approach.
> >
> > One of the biggest challenges to adding any new API is the adoption time. It
> > can easily span 18-14 months if not more. Rolling things out for users to see
> > benefits has a substantial delay, iterating and reacting to problems will take
> > similar long time. All in all to reach a maturity point this can end up far
> > ahead in the future that adoption can never materialize. And it makes the
> > burden and the par to getting things right the first time very high that
> > progress can stifle at discussion point.
> >
> > Another major challenge is trusting application to provide the right hints.
> > Managing potential abusers can be difficult (doable, but back to previous point
> > can lead to stifling discussion to agree on the 'right' approach). But also the
> > potential ABI implications that can make easy evolution and fast iteration
> > hard.
> >
> > With config based hinting we eliminate these. Admins (which on many systems is
> > the user, because it's Linux..) will have to choose and approve the config to
> > be applied for any app.  It also provides additional flexibility where
> > a workload is 'important' on one system, but is actually a 'background' on
> > another. Group control (discussed at LPC) is better way to manage this, but
> > generally it inherently better handles potential perf/power trade-offs
> > variations if tagging could be done differently to suite different needs. e.g:
> > you can have more than one way to tag an application to suite potential
> > different needs rather than being stuck with a single one imposed on you.
> >
> > By using NETLINK to listen to when new tasks are forked and processes are
> > created, we can easily create a config based system to auto tag tasks based on
> > userspace description.
> 
> We have a long history of relying on netlink for the device driver model
> and udev. For threads, I guess the events would be much more frequent
> than device hotplug/unplug—one app might create hundreds of threads.
> We may end up relying on some ring-buffer-based event mechanism if we
> eventually find that netlink itself becomes a bottleneck :-)

Yeah efficiency keeps coming up with many of the decisions. The thinking is
that at userspace we can iterate and nail down the details faster. Once it is
stable and these become a real and annoying bottleneck, we can address them.
But for a starter, it works and allows us to build up and iterate :)

> 
> Now I see you are monitoring both PROC_EVENT_FORK and
> PROC_EVENT_COMM. Wouldn’t this be sometimes duplicated, since
> PR_SET_NAME may come soon after a thread is created?

It's not a duplicate. There are several scenarios to cater for:

* New processes can be created from FORK, not just EXEC
* If a task is not tagged, we want to enforce QOS_DEFAULT on all created tasks
* If a task is tagged and forks, we want to apply the same tag to its forkee
  (think worker threads spawning multiple forks)

You probably noted, but for the sake of sharing more widely, we iterate through
all existing tasks on start and tag them. There's an exception of PIDs below
1000 as I was scared to mess up all kernel threads and seemed they are all
below 1000, but this needs a massage.

> 
> >
> > For example the following config
> >
> >         {
> >                 "chrome": {
> >                         "qos": "QOS_USER_INTERACTIVE"
> >                 },
> >                 "gcc": {
> >                         "qos": "QOS_BACKGROUND"
> >                 }
> >         }
> >
> > is a poor man's way to tell the system the user wants to treat Chrome as
> > a user_interactive application and gcc (compiler) as a background one.
> >
> > This is a provisional way for 'easy' tagging. The real intended use is for
> > specific tasks within an application (process) to be tagged to describe their
> > individual role
> >
> >         {
> >                 "chrome": {
> >                         "thread_qos": {
> >                                 "chrome": "QOS_USER_INTERACTIVE",
> >                                 "some_bg_task": "QOS_BACKGROUND"
> >                         }
> >                 }
> >         }
> >
> 
> I assume this depends on user code setting the thread name via
> prctl(PR_SET_NAME, "name"), based on the code below?

It's pthread_setname_np() too - which I think what most folks use.

> 
> static void iterate_threads(pid_t tgid)
> {
>         char task_path[256];
>         snprintf(task_path, sizeof(task_path), "/proc/%d/task", tgid);
> 
>         DIR *tdir = opendir(task_path);
>         struct dirent *tentry;
> 
>         while (tdir && (tentry = readdir(tdir)) != NULL) {
>                 if (is_numeric(tentry->d_name)) {
>                         pid_t pid = atoi(tentry->d_name);
>                         char comm[TASK_COMM_LEN];
> 
>                         if (get_comm_by_pid(pid, comm))
>                                 apply_thread_qos(pid, tgid, comm);
>                 }
>         }
>         if (tdir) closedir(tdir);
> }
> 
> I assume this is quite common on Android, but in Linux
> distributions, almost no applications set names for their
> threads?

I don't think there's a difference between the two. You can monitor
/var/log/schedqos.log to see what gets created. I was trying to collect data
for gaming and could see different task name for steam and came up with a tag
to test, unfortunately my rig was giving me 10 FPS to produce any usable data.
I needed a new machine, not QoS to help :D

(side note, what's going on with game sizes? 90 and 100+ GiB :-O)

> 
> So, the project is trying to encourage people to set proper
> names for their threads, right?

I think this already exist, but generally yes. Also to encourage how to divide
the roles of the tasks when creating them.

> 
> > so that real interactive tasks are given the resource they need, and the noise
> > is reduced to just that. Note by default everything will be treated as
> > QOS_DEFAULT, which is set to match QOS_UTILITY. IOW everything is assumed to be
> > random 'noise' by default. Which makes delivering better beset-effort QoS
> > easier.
> >
> > Roles
> > =====
> >
> > This model is based on existing one shipped in the industry [1] that its users
> > are happy with. It breaks down the tasks' role into 4 classes:
> >
> > * USER_INTERACTIVE:     Requires immediate response
> > * USER_INITIATED:       Tolerates short latencies, but must get work done quickly still
> > * UTILITY:              Tolerates long delays, but not prolonged ones
> > * BACKGROUND:           Doesn't mind prolonged delays
> > * DEFAULT:              All untagged tasks will get this category which will map to utility.
> >
> > EEVDF should allow us to describe these different levels via specifying
> > different runtime (custom slice) to each class. Shortest slice should still be
> > long enough not to sacrifice throughput. Nice values will operate as bandwidth
> > control so that long running user_interactive tasks can't be starved by long
> > running background ones if they had to run on the same CPU under overloaded
> > scenarios. uclamp_max help constraint power impact and access to expensive
> > highest performance levels.
> >
> > Mapping
> > -------
> >
> >         {
> >                 "QOS_USER_INTERACTIVE": {
> >                         "sched_policy": "SCHED_NORMAL",
> >                         "sched_nice": -4,
> >                         "sched_runtime": 8000000,
> >                         "sched_util_max": 1024
> >                 },
> >                 "QOS_USER_INITIATED": {
> >                         "sched_policy": "SCHED_NORMAL",
> >                         "sched_nice": -2,
> >                         "sched_runtime": 12000000,
> >                         "sched_util_max": 768
> >                 },
> >                 "QOS_UTILITY": {
> >                         "sched_policy": "SCHED_BATCH",
> >                         "sched_nice": 2,
> >                         "sched_runtime": 16000000,
> >                         "sched_util_max": 512
> >                 },
> >                 "QOS_BACKGROUND": {
> >                         "sched_policy": "SCHED_BATCH",
> >                         "sched_nice": 4,
> >                         "sched_runtime": 20000000,
> >                         "sched_util_max": 256
> >                 },
> >                 "QOS_DEFAULT": {
> >                         "sched_policy": "SCHED_BATCH",
> >                         "sched_nice": 2,
> >                         "sched_runtime": 16000000,
> >                         "sched_util_max": 512
> >                 }
> >         }
> >
> 
> I now see a fairly simple call stack like
> apply_thread_qos_tag() → sched_setattr(), which seems quite readable to me.

Yeah I kept the json description faithful to sched_attr including the values.

> 
> >
> > Caveats
> > =======
> >
> > AUTOGROUP and cgroup cpu controller must be disabled for maximum effectiveness.
> > We assume a flat hierarchy and per-task description to keep the system under
> > control.
> >
> > If they are enabled, it is hard to distinguish between background and user
> > interactive tasks  across processes due to imposed fairness at group level.
> > Most notably under loaded scenarios with a lot of long running tasks.
> >
> > It is believed that a flat hierarchy is best approach and using per-task
> > tagging combined with simple group control to ensure roles of process and tasks
> > are described simply, yet sufficiently, to get the desired behavior with the
> > least complexity and maximum portability/flexibility.
> >
> 
> I don’t see how this is related to your schedqos, since you’re just calling
> sched_setattr(). That is fully consistent with the Linux kernel model.
> 
> Am I right in assuming that whatever you set via sched_setattr() will always
> take effect even in a cgroup-based system?

Sort of. Nice values will get lost across groups. Throughput of schbench+make
would be the same as default for example if you don't disable AUTOGROUP
particularly as I think that what was causing my specific test conditions to
trip. But the same issue exist for process in different groups. We can't
differentiate between BACKGROUND and INTERACTIVE tasks across groups from
bandwidth PoV. So a busy background tasks in a different process can cause
problems for interactive task on another process given they are in different
groups.

> 
> >
> > Next steps
> > ==========
> >
> > The current setup is usable and should provide tangible results for those
> > interested. Corner cases where it fails will be visible though under
> > comprehensive testing. See schbench+kernel build results below for example.
> >
> > I won't repeat our LPC discussion, but we need multi modal wake up path and
> > coherent decision making between wake up and load balancer. Both items are
> > being worked on already. Push based load balance for fair is on the list [2].
> >
> > For best perf/watt under schedutil, we need to introduce the concept of rampup
> > multiplier to help counter DVFS related latencies. Also this is WIP and there
> > were patches in the past [3] that will be rebased and reposted.
> >
> > Idle states can be a problem for performance and power and scheduler today
> > doesn't take them into account at all.
> >
> > Performance Inversion and Inheritance issues are common in practice and require
> > Proxy Execution and teaching libc and languages to move to futex_pi by default.
> > In practice folks will see latency/perf issues in their P99 and max at least
> > - severity of which will depend on the use case. See discussion on Enable PI by
> > default in userspace [4].
> >
> > We also need to add a new unfair pi lock that doesn't operate on strict order
> > which is critical for performance [5]. I have gathered some results privately
> > that using futex_pi causes performance regressions.
> 
> I can imagine that if we have too much priority inheritance, we could
> end up hurting the system. Priority inheritance raises the priority of
> tasks that should not necessarily be in the high-priority class. This is
> especially problematic when the dependency chain is long and complex.

It shouldn't be a problem. With proxy execution we don't really inherit
attributes, but the lock holder will run in the context of the blocked task(s)
until it releases the lock.

> 
> >
> > The utility itself needs to handle group control still. We need to extend
> > NETLINK to send events when task move group and introduce a group level control
> > of what QoS is allowed or not. The goal is to piggy-back on cgroup but provide
> > user space annotation of QoS. Which is a simple allow/disallow to prevent
> > user_interactive tasks for instance when a task is in a 'background' group.
> > Think an app that is minimized or a browser tab that is hidden. This group
> > control might need integration with window managers so that this control is
> > transparently handled for all apps.
> >
> > There are also problems that are not seen outside of Linux ecosystem that might
> > require an extra dimension to annotate tasks that can be memory sensitive
> > (particularly NUMA). I call this new dimension Execution Profile. But this is
> > an area that requires further discussion in one of the upcoming conferences and
> > a dedicate thread on the list. Cache Aware Scheduling can particularly be
> > simplified if such annotation can be provided by userspace - which this
> > approach should make it easy to implement and start experimenting with.
> >
> > We also need to ensure sched_attr is locked and can only be modified by
> > schedqos utility so that it can be the sole orchestrator for managing the
> > behavior of the tasks.
> >
> > The project would hopefully move to kernel.org and get contributions as part of
> > usual kernel/scheduler development process.
> >
> >
> > Results
> > =======
> >
> > Based on tip/sched/core: 8d16e3c6f844
> >
> > Cyclictest + hackbench
> > ----------------------
> >
> >         hackbench -T -p -l 60000 -g 2 &
> >         sleep 1
> >         sudo nice -n 0 cyclictest -t 1 -i 1777 -D 30 -h 20000 -q
> >
> >         {
> >                 "hackbench": {
> >                         "qos": "QOS_BACKGROUND"
> >                 },
> >                 "cyclictest": {
> >                         "qos": "QOS_USER_INTERACTIVE"
> >                 }
> >         }
> >
> 
> This raises a question: do you want app developers to write the
> configuration file for their own apps, or do you want system
> administrators to define a global configuration file?

Good question. And the answer is both.

I'm hoping apps will install a config file in /etc/schedqos.

But there are systems that are run by an army of engineers to tune them that
they might decide they want to take control and define the behavior of
everything that is supposed to run on their system.

I consider the flexibility part of the advantage of this approach. Clueless
users can rely on app developers (or community) to give them something that
works out of the box. Power users can choose to fine tune everything. We can
cater for both.

I think I mentioned this elsewhere, but to change anything with mappings or
tagging by adding/omitting/modifying, a simple restart of the daemon is suffice
for the changes to take effect.

I hope we can start collecting config files as part of the service. At least
until this picks up where we no longer have to.

> 
> Why is hackbench considered background and cyclictest considered
> USER_INTERACTIVE? Who is supposed to know this?

This is synthetic setup to demonstrate the benefit. We treat hackbench as
something noisy that can interfere with cyclictest and demonstrate how the QoS
can help against this interference.

In real life both should be USER_INTERACTIVE.

This brings an interesting point that I couldn't expand on properly in the
announcement. How can an app be forced to behave like background?

This is where group control (or access control, not sure what's a better name)
comes into play.

The plan is to introduce a new netlink to see when tasks are moving between
groups and use that to implement QoS control at group level (all in userspace).

The goal is to allow/disallow a QoS level, forcing it to demote to the next
allowed level.

For example

{
	"background": {
		"QOS_USER_INTERACTIVE": false,
		"QOS_USER_INITIATED": false,
		"QOS_UTILITY": false,
		"QOS_BACKGROUND": true
	},
	"utility": {
		"QOS_USER_INTERACTIVE": false,
		"QOS_USER_INITIATED": false,
		"QOS_UTILITY": true,
		"QOS_BACKGROUND": true
	},
	"foreground": {
		"QOS_USER_INTERACTIVE": false,
		"QOS_USER_INITIATED": true,
		"QOS_UTILITY": true,
		"QOS_BACKGROUND": true
	},
	"focused": {
		"QOS_USER_INTERACTIVE": true,
		"QOS_USER_INITIATED": true,
		"QOS_UTILITY": true,
		"QOS_BACKGROUND": true
	}
}

When a task is in background group, we will enforce all tasks to be
QOS_BACKGROUND since any task that is tagged at higher level is not allowed.

"focused" group is what android calls top-app or the app that the user is
currently interacting with where everything is allowed.

This is just an example.

The anticipation is to add integration with window managers to help manage
these groups so when an app is minimized or hidden from view is automatically
moved to background etc.

If you wonder, the window manager can create its own group description and
install it into /etc/schedqos. We can come up with a single convention, but
they can do whatever they want too. I think we need to walk this walk with one
integration to see. We need to fill up more details still.

Also we can give users the ability to put apps to specific group (as part of
window manager integration). For example, I want to preserve battery on my
laptop so I'd like gcc/make to be treated as utility to save resources. But
allow them to run as interactive on my proper build server. We shouldn't change
how the app is described, but this group control will allow to shuffle things
around to meet user needs, UI perception, or even to add handling based on low
battery, wall plugged or any other cue (location? compile fast in office but
slow while at home ;-)).

> 
> App developers may not do this, and system administrators may lack
> knowledge of specific apps.

Actually by looking at /var/log/schedqos.log you can try to take a guess by
monitor an app of interest. Which I think for popular apps can easily gather
community contributions to create such tags.

Hopefully app developers will opt in so users can get the best out of the box.

I see admins having to do this for super fine control and special niche cases.

> 
> BTW, assuming we have a phone with 300 installed applications, do we
> need to configure each one individually, or can we use some shared
> configuration? For example, on Android, threads may have the same name
> across different applications, such as RenderThread, HeapTaskDaemon,
> Binder:xxx etc.

I didn't get a chance to do so, and I hope someone will help with
a contribution as I can't scale. But the plan is to support globbing at process
and task levels so you can match more widely when necessary. I had a quick stab
but I ended up abandoning it to get this to a usable initial stage that I can
share.

> 
> > Default:
> >
> >         # Min Latencies: 00004
> >         # Avg Latencies: 00062
> >         # Max Latencies: 04426
> >
> > With schedqos:
> >
> >         # Min Latencies: 00003
> >         # Avg Latencies: 00053
> >         # Max Latencies: 01246
> >
> > schbench + kernel build
> > -----------------------
> >
> > AUTOGROUP and CPU controller were disabled otherwise you won't see a difference
> > due to imposed fairness at group level
> >
> >         {
> >                 "make": {
> >                         "qos": "QOS_BACKGROUND"
> >                 },
> >                 "gcc": {
> >                         "qos": "QOS_BACKGROUND"
> >                 },
> >                 "cc1": {
> >                         "qos": "QOS_BACKGROUND"
> >                 },
> >                 "schbench": {
> >                         "thread_qos": {
> >                                 "schbench-msg": "QOS_USER_INTERACTIVE",
> >                                 "schbench-worker": "QOS_USER_INITIATED"
> >                         }
> 
> I assume you modified the schbench? In my experiments, I always get the name
> -schbench.

You must be on old version, the code sets these names

	https://github.com/masoncl/schbench/blob/main/schbench.c#L1547

IIRC setting both as USER_INTERACTIVE yielded slightly worse results than what
I am proposing here - but I might have added uclamp_max to initiated later, so
not sure what the current state will yield :). Maybe I shouldn't cap these
performance sensitive levels..

Thanks for the excellent questions and I hope more will come ;-)

I'd love to see results too. I know I hit some corner cases when trying to
collect data for this announcement. Trying it with gaming would be interesting
too. Would be great to know what it helps and doesn't help with in general.


Thanks!

--
Qais Yousef

> 
> /proc/2892/task$ ls
> 2892  2893  2894  2895  2896  2897
> /proc/2892/task$ cat 2892/comm
> schbench
> /proc/2892/task$ cat 2893/comm
> schbench
> /proc/2892/task$ cat 2894/comm
> schbench
> /proc/2892/task$ cat 2895/comm
> schbench
> /proc/2892/task$ cat 2896/comm
> schbench
> /proc/2892/task$ cat 2897/comm
> schbench
> 
> 
> >                 }
> >         }
> >
> > Default:
> >
> >         Wakeup Latencies percentiles (usec) runtime 30 (s) (45018 total samples)
> >                   50.0th: 1618       (11870 samples)
> >                   90.0th: 3580       (18081 samples)
> >                 * 99.0th: 4952       (3973 samples)
> >                   99.9th: 6760       (405 samples)
> >                   min=1, max=11092
> >         Request Latencies percentiles (usec) runtime 30 (s) (45042 total samples)
> >                   50.0th: 12464      (13518 samples)
> >                   90.0th: 22496      (18009 samples)
> >                 * 99.0th: 36032      (4020 samples)
> >                   99.9th: 75904      (405 samples)
> >                   min=3860, max=144284
> >         RPS percentiles (requests) runtime 30 (s) (31 total samples)
> >                   20.0th: 1458       (7 samples)
> >                 * 50.0th: 1486       (9 samples)
> >                   90.0th: 1566       (12 samples)
> >                   min=1420, max=1603
> >         average rps: 1501.40
> >
> > With schedqos:
> >
> >         Wakeup Latencies percentiles (usec) runtime 30 (s) (67556 total samples)
> >                   50.0th: 10         (15337 samples)
> >                   90.0th: 6488       (26386 samples)
> >                 * 99.0th: 13168      (5961 samples)
> >                   99.9th: 19232      (607 samples)
> >                   min=1, max=32126
> >         Request Latencies percentiles (usec) runtime 30 (s) (67618 total samples)
> >                   50.0th: 6568       (21537 samples)
> >                   90.0th: 13744      (25740 samples)
> >                 * 99.0th: 37312      (6064 samples)
> >                   99.9th: 65472      (602 samples)
> >                   min=3506, max=153046
> >         RPS percentiles (requests) runtime 30 (s) (31 total samples)
> >                   20.0th: 2084       (7 samples)
> >                 * 50.0th: 2268       (9 samples)
> >                   90.0th: 2412       (12 samples)
> >                   min=1904, max=2523
> >         average rps: 2253.93
> >
> > Notes
> > -----
> >
> > Throughput is better, but latencies get worse due to a number of bugs Vincent
> > is addressing and patches for which are already on the list [6][7][8].
> >
> > One observation is disabling RUN_TO_PARITY yields better latencies. But
> > hopefully this won't be necessary.
> >
> > Also schbench can suffer from bad task placement where two worker threads end
> > up on the same CPU. Once multi-modal wake up path is ready, USER_INTERACTIVE
> > tasks should be spread across CPUs; current wake up behavior spreads based on
> > load only which means we can end up with these accidental bad placements that
> > require the scheduler to understand that for better latencies, it is best not
> > to place two tasks with short deadlines on the same CPU.
> >
> >
> > With schedqos + NO_RUN_TO_PARITY + [6][7][8] patches:
> >
> >         Wakeup Latencies percentiles (usec) runtime 30 (s) (69260 total samples)
> >                   50.0th: 9          (26985 samples)
> >                   90.0th: 1342       (19692 samples)
> >                 * 99.0th: 1686       (6280 samples)
> >                   99.9th: 2428       (570 samples)
> >                   min=1, max=5710
> >         Request Latencies percentiles (usec) runtime 30 (s) (69338 total samples)
> >                   50.0th: 8104       (20785 samples)
> >                   90.0th: 17696      (27798 samples)
> >                 * 99.0th: 23648      (6173 samples)
> >                   99.9th: 35904      (623 samples)
> >                   min=3835, max=90821
> >         RPS percentiles (requests) runtime 30 (s) (31 total samples)
> >                   20.0th: 2228       (7 samples)
> >                 * 50.0th: 2300       (9 samples)
> >                   90.0th: 2404       (12 samples)
> >                   min=2171, max=2556
> >         average rps: 2311.27
> >
> >
> > Contribute
> > ==========
> >
> > Help us get it better! See list of areas we need help with in CONTRIBUTE.md [9]
> >
> > Or just give it a go and report any corner cases that doesn't work so we can
> > look at it and make sure we have a plan to get it fixed :)
> >
> >
> > [1] https://developer.apple.com/library/archive/documentation/Performance/Conceptual/EnergyGuide-iOS/PrioritizeWorkWithQoS.html#//apple_ref/doc/uid/TP40015243-CH39-SW1
> > [2] https://lore.kernel.org/lkml/20251202181242.1536213-1-vincent.guittot@linaro.org/
> > [3] https://lore.kernel.org/lkml/20240820163512.1096301-1-qyousef@layalina.io/
> > [4] https://lpc.events/event/19/contributions/2244/
> > [5] https://developer.apple.com/documentation/os/os_unfair_lock_lock
> > [6] https://lore.kernel.org/lkml/20260331162352.551501-1-vincent.guittot@linaro.org/
> > [7] https://lore.kernel.org/lkml/20260410132321.2897789-1-vincent.guittot@linaro.org/
> > [8] https://lore.kernel.org/lkml/20260410144808.2943278-1-vincent.guittot@linaro.org/
> > [9] https://github.com/qais-yousef/schedqos/blob/main/CONTRIBUTE.md
> 
> 
> Best Regards
> Barry