[PATCH 1/2] sched/doc: add a preemption model overview

[PATCH 1/2] sched/doc: add a preemption model overview

[Quchaosheng]

The scheduler documentation describes the individual scheduling classes
and various tuning knobs, but there is nothing that describes the
preemption models themselves.  The only place where they are documented
is the kernel-parameters entry for "preempt=", which explains the boot
time parameter but not the models it selects.

Add Documentation/scheduler/sched-preemption.rst, covering the four
models, which of them can be selected at runtime, and the mechanism
behind PREEMPT_LAZY.  The lazy model in particular is easy to
misunderstand: a lazy reschedule does not send a cross-CPU reschedule
IPI, so it is only committed on the return to user space or on the next
tick.  That makes the tick an upper bound on lazy preemption latency,
and it means the usual real-time latency tools, which wake a pinned
task on its own CPU, do not exercise it at all.

Signed-off-by: Quchaosheng <quchaosheng000406@163.com>
---
 Documentation/scheduler/index.rst            |   1 +
 Documentation/scheduler/sched-preemption.rst | 120 +++++++++++++++++++
 2 files changed, 121 insertions(+)
 create mode 100644 Documentation/scheduler/sched-preemption.rst

diff --git a/Documentation/scheduler/index.rst b/Documentation/scheduler/index.rst
index 17ce8d76befc..d6d75421756a 100644
--- a/Documentation/scheduler/index.rst
+++ b/Documentation/scheduler/index.rst
@@ -23,5 +23,6 @@ Scheduler
     sched-stats
     sched-ext
     sched-debug
+    sched-preemption
 
     text_files
diff --git a/Documentation/scheduler/sched-preemption.rst b/Documentation/scheduler/sched-preemption.rst
new file mode 100644
index 000000000000..fc6fd89789cf
--- /dev/null
+++ b/Documentation/scheduler/sched-preemption.rst
@@ -0,0 +1,120 @@
+=====================
+Scheduler preemption
+=====================
+
+The kernel can be built to run kernel code either uninterruptibly, or with
+varying degrees of preemptibility.  These are the *preemption models*.
+
+When CONFIG_PREEMPT_DYNAMIC is enabled the preemption model can additionally
+be selected at boot time with the ``preempt=`` command line parameter, without
+rebuilding the kernel.  See
+Documentation/admin-guide/kernel-parameters.txt for the parameter itself.
+
+The models selectable at runtime are:
+
+	=========	====================================================
+	none		No preemption of kernel code other than at explicit
+			``cond_resched()`` / blocking points.
+	voluntary	As ``none``, plus ``might_sleep()`` sites.
+	full		Any section that is not explicitly preempt disabled
+			may be preempted at any time.  Tasks also yield
+			contended spinlocks.
+	lazy		As ``full``, except that a reschedule requested by
+			the fair scheduler does not interrupt the target
+			CPU.  It is committed at the next return to user
+			space or at the next tick, whichever comes first.
+	=========	====================================================
+
+Not every model is available on every kernel.  On a kernel that selects
+ARCH_HAS_PREEMPT_LAZY, only ``full`` and ``lazy`` are offered and ``none`` and
+``voluntary`` are rejected.  A PREEMPT_RT kernel likewise does not offer
+``none`` or ``voluntary``.
+
+The model that is actually active is reported in the boot log::
+
+	Dynamic Preempt: full
+
+PREEMPT_LAZY
+============
+
+``lazy`` is the interesting one, because its behaviour cannot be understood
+from the "can this be preempted" question alone.
+
+When a wakeup makes a running CFS task preemptible, the fair scheduler calls
+``resched_curr_lazy()``.  With the lazy model inactive that is just
+``resched_curr()``; with ``lazy`` active it sets ``TIF_NEED_RESCHED_LAZY``
+instead of ``TIF_NEED_RESCHED``.  The consequence is in ``__resched_curr()``:
+the cross-CPU reschedule IPI is only sent for ``TIF_NEED_RESCHED``.
+
+A lazy reschedule therefore does *not* interrupt the target CPU.  It is
+committed later, in one of two ways:
+
+1. On the next return to user mode.  ``__exit_to_user_mode_loop()`` treats
+   ``_TIF_NEED_RESCHED | _TIF_NEED_RESCHED_LAZY`` as a reschedule request, and
+   ``xfer_to_guest_mode_work()`` does the same for a vCPU returning to guest
+   mode.
+
+2. In ``scheduler_tick()``, which promotes a pending
+   ``TIF_NEED_RESCHED_LAZY`` to a full ``TIF_NEED_RESCHED`` once per tick.
+
+The tick is thus an upper bound on lazy preemption latency for CFS tasks: a
+runnable CFS task that has been passed over is picked up within one tick, not
+immediately.  The tick is guaranteed to run in the case that matters: while
+more than one CFS task is runnable on the CPU, ``sched_can_stop_tick()``
+refuses to stop the tick.  A lazy reschedule aimed at the idle task is promoted
+to a full one instead.
+
+That is the intended trade-off -- it avoids sending an IPI to every CPU that
+has a runnable CFS task -- and it is why ``lazy`` gives up latency in exchange
+for fewer inter-processor interrupts.
+
+The idle task is never delayed: ``__resched_curr()`` promotes a lazy
+reschedule to a full one when the target is the idle task.
+
+Only the fair scheduler issues lazy reschedules.  RT and deadline tasks are
+still preempted immediately.  ``lazy`` is therefore not a replacement for
+PREEMPT_RT; it is a way to keep most of the responsiveness of ``full`` while
+removing a large part of its IPI traffic.
+
+Debugging
+=========
+
+The scheduler debugfs directory provides ``/sys/kernel/debug/sched/preempt``,
+which lists the models that may be switched to; the active one is enclosed in
+parentheses::
+
+	# cat /sys/kernel/debug/sched/preempt
+	full (lazy)
+
+Writing a model name to the file switches to it.  A name that the kernel
+cannot select (either because the model is not built in, or because it is not
+available on this kernel) is rejected with EINVAL.  The ``preempt=``
+boot parameter rejects such a value as well, but only prints an ``unsupported
+mode`` warning and keeps the default model.
+
+Measuring
+=========
+
+Two things routinely trip people up when measuring preemption latency.
+
+First, the classic real-time latency tools do not exercise lazy preemption at
+all.  Lazy only applies to the fair class, so waking a SCHED_FIFO task takes
+``resched_curr()`` and preempts immediately, whichever model is active.  And
+when the woken task runs on the CPU the wakeup happens on, even a fair lazy
+reschedule is committed as soon as that CPU returns to user mode, which for an
+already runnable task is immediate.  For both reasons such workloads measure
+the same under ``lazy`` and ``full``.
+
+The difference only shows up when the woken task is *not* running on the CPU
+where the wakeup happens.  In that case the waker requests the reschedule, and
+under ``lazy`` that request is not turned into an IPI; the target CPU only
+notices at its next tick.  A measurement that cannot distinguish ``none`` from
+``full`` is usually not measuring the preemption model at all: in the absence
+of ``lazy``, the reschedule IPI is sent regardless of the model and provides a
+preemption point even on PREEMPT_NONE.
+
+Second, when driving cross-CPU wakeups from a periodic source, keep the wakeup
+period away from integer multiples of the tick.  A period that is phase-locked
+to the tick will hide (or invent) exactly the effect that is being measured.
+Add jitter to the wakeup period, and confirm the active model from the
+``Dynamic Preempt:`` line rather than assuming that ``preempt=`` took effect.
-- 
2.43.0

[PATCH 2/2] docs: kernel-parameters: fix a truncated sentence for preempt=lazy

[Quchaosheng]

The "lazy" entry ends with:

  One preemption is when the task returns to user space.

which is not a sentence.  It is a leftover from shortening "One such
preemption is when the task returns to user space."  Add the missing
word back.

Fixes: f66e4a996582 ("sched/core: Update kernel boot parameters for LAZY preempt.")
Signed-off-by: Quchaosheng <quchaosheng000406@163.com>
---
 Documentation/admin-guide/kernel-parameters.txt | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt
index 33cd30996e47..6c1bfba180a2 100644
--- a/Documentation/admin-guide/kernel-parameters.txt
+++ b/Documentation/admin-guide/kernel-parameters.txt
@@ -5453,8 +5453,8 @@ Kernel parameters
 			lazy - Scheduler controlled. Similar to full but instead
 			       of preempting the task immediately, the task gets
 			       one HZ tick time to yield itself before the
-			       preemption will be forced. One preemption is when the
-			       task returns to user space.
+			       preemption will be forced. One such preemption is
+			       when the task returns to user space.
 
 	print-fatal-signals=
 			[KNL] debug: print fatal signals
-- 
2.43.0

Re: [PATCH 2/2] docs: kernel-parameters: fix a truncated sentence for preempt=lazy

[Sebastian Andrzej Siewior]

On 2026-09-20 10:20:03 [+0800], Quchaosheng wrote:
> The "lazy" entry ends with:
> 
>   One preemption is when the task returns to user space.
> 
> which is not a sentence.  It is a leftover from shortening "One such
> preemption is when the task returns to user space."  Add the missing
> word back.
> 
> Fixes: f66e4a996582 ("sched/core: Update kernel boot parameters for LAZY preempt.")
> Signed-off-by: Quchaosheng <quchaosheng000406@163.com>
> ---
>  Documentation/admin-guide/kernel-parameters.txt | 4 ++--
>  1 file changed, 2 insertions(+), 2 deletions(-)
> 
> diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt
> index 33cd30996e47..6c1bfba180a2 100644
> --- a/Documentation/admin-guide/kernel-parameters.txt
> +++ b/Documentation/admin-guide/kernel-parameters.txt
> @@ -5453,8 +5453,8 @@ Kernel parameters
>  			lazy - Scheduler controlled. Similar to full but instead
>  			       of preempting the task immediately, the task gets
>  			       one HZ tick time to yield itself before the
> -			       preemption will be forced. One preemption is when the
> -			       task returns to user space.
> +			       preemption will be forced. One such preemption is
> +			       when the task returns to user space.

I think if you do "preemption point" the sentence becomes English again.

>  
>  	print-fatal-signals=
>  			[KNL] debug: print fatal signals

Sebastian

Re: [PATCH 2/2] docs: kernel-parameters: fix a truncated sentence for preempt=lazy

[Quchaosheng]

On 2026-09-21 11:33:26 [+0200], Sebastian Andrzej Siewior wrote:

> I think if you do "preemption point" the sentence becomes English again.

You are right, and it is a better fix than mine. "One preemption is when
the task returns to user space" becomes:

  One such preemption point is when the task returns to user space.

I will use that if I resend. Holding the patch with the other one in the
series for now.

Quchaosheng

Re: [PATCH 1/2] sched/doc: add a preemption model overview

[Sebastian Andrzej Siewior]

On 2026-09-20 10:20:02 [+0800], Quchaosheng wrote:
> The scheduler documentation describes the individual scheduling classes
> and various tuning knobs, but there is nothing that describes the
> preemption models themselves.  The only place where they are documented
> is the kernel-parameters entry for "preempt=", which explains the boot
> time parameter but not the models it selects.
> 
> Add Documentation/scheduler/sched-preemption.rst, covering the four
> models, which of them can be selected at runtime, and the mechanism
> behind PREEMPT_LAZY.  The lazy model in particular is easy to
> misunderstand: a lazy reschedule does not send a cross-CPU reschedule
> IPI, so it is only committed on the return to user space or on the next
> tick.  That makes the tick an upper bound on lazy preemption latency,
> and it means the usual real-time latency tools, which wake a pinned
> task on its own CPU, do not exercise it at all.
> 
> Signed-off-by: Quchaosheng <quchaosheng000406@163.com>
> ---
>  Documentation/scheduler/index.rst            |   1 +
>  Documentation/scheduler/sched-preemption.rst | 120 +++++++++++++++++++
>  2 files changed, 121 insertions(+)
>  create mode 100644 Documentation/scheduler/sched-preemption.rst
> 
> diff --git a/Documentation/scheduler/index.rst b/Documentation/scheduler/index.rst
> index 17ce8d76befc..d6d75421756a 100644
> --- a/Documentation/scheduler/index.rst
> +++ b/Documentation/scheduler/index.rst
> @@ -23,5 +23,6 @@ Scheduler
>      sched-stats
>      sched-ext
>      sched-debug
> +    sched-preemption
>  
>      text_files
> diff --git a/Documentation/scheduler/sched-preemption.rst b/Documentation/scheduler/sched-preemption.rst
> new file mode 100644
> index 000000000000..fc6fd89789cf
> --- /dev/null
> +++ b/Documentation/scheduler/sched-preemption.rst
> @@ -0,0 +1,120 @@
> +=====================
> +Scheduler preemption
> +=====================
> +
> +The kernel can be built to run kernel code either uninterruptibly, or with
> +varying degrees of preemptibility.  These are the *preemption models*.

  These preemption models apply only to the kernel - a user-space task is
  always preempted by the scheduler regardless of the selected model.

> +
> +When CONFIG_PREEMPT_DYNAMIC is enabled the preemption model can additionally
> +be selected at boot time with the ``preempt=`` command line parameter, without
> +rebuilding the kernel.  See
> +Documentation/admin-guide/kernel-parameters.txt for the parameter itself.

  Additionally the preemption model can be inspected and changed via the
  /sys/kernel/debug/sched/preempt file (which is available for debugging
  purposes and may change).

> +
> +The models selectable at runtime are:
> +
> +	=========	====================================================
> +	none		No preemption of kernel code other than at explicit
> +			``cond_resched()`` / blocking points.
> +	voluntary	As ``none``, plus ``might_sleep()`` sites.
> +	full		Any section that is not explicitly preempt disabled
> +			may be preempted at any time.  Tasks also yield
> +			contended spinlocks.

What does "Tasks also yield contended spinlocks" mean?

> +	lazy		As ``full``, except that a reschedule requested by
> +			the fair scheduler does not interrupt the target
> +			CPU.  It is committed at the next return to user
> +			space or at the next tick, whichever comes first.

  Same as ``full` except that the scheduling request is delayed until the
  return to user space or the next tick, whichever comes first. This delay
  does not apply to real-time tasks.


> +	=========	====================================================
> +
> +Not every model is available on every kernel.  On a kernel that selects
> +ARCH_HAS_PREEMPT_LAZY, only ``full`` and ``lazy`` are offered and ``none`` and
> +``voluntary`` are rejected.  A PREEMPT_RT kernel likewise does not offer
> +``none`` or ``voluntary``.

I would rephrase it to
  On an architecture, that provides support lazy-preempt, only …

and maybe 
  A PREEMPT_RT kernel offers always ``full`` and ``lazy`` if provided by
  the architecture.

> +The model that is actually active is reported in the boot log::
> +
> +	Dynamic Preempt: full
> +
> +PREEMPT_LAZY
> +============
> +
> +``lazy`` is the interesting one, because its behaviour cannot be understood
> +from the "can this be preempted" question alone.
> +
> +When a wakeup makes a running CFS task preemptible, the fair scheduler calls
> +``resched_curr_lazy()``.  With the lazy model inactive that is just
> +``resched_curr()``; with ``lazy`` active it sets ``TIF_NEED_RESCHED_LAZY``
> +instead of ``TIF_NEED_RESCHED``.  The consequence is in ``__resched_curr()``:
> +the cross-CPU reschedule IPI is only sent for ``TIF_NEED_RESCHED``.
> +

This is a technical description of the code. From a bird's eye view:

  The scheduler becomes a wake up request for a task and marks it
  immediately as eligible for running. If there is an idle CPU which is
  suitable then it will move the task there. Otherwise the scheduler must
  decide which task to preempt. Once a decision has been made, the task,
  that needs to be removed from the CPU, gets a "NEED RESCHED" flag set.
  In tracing this is visible in the "need-resched" column and is denoted
  by a ``N`` for "NEED_RESCHED" or ``l`` for "NEED_RESCHED_LAZY" request.
  The next step for kernel is to honour this scheduling request. 
  A task executing in userland can always be preempted. A task executing
  in kernel can only be preempted when it is safe to do so.
  
  - No Forced Preemption
    Code paths with loops have often a "cond_resched()" function which
    performs scheduling.
  
  - Voluntary Kernel Preemption
    Same as in the previous model and additionally functions, which are
    known that they may block and schedule, have also such a
    "cond_resched()".
  
  - Preemptible Kernel
    The previously mention "cond_resched()" mechanics is not involved.
    Instead the kernel tracks preemptiblity and schedules after a
    scheduling request (NEED_RESCHED) once possible.
  
  - Scheduler controlled preemption model
    Same as in the previous model except that the default scheduling
    request is "NEED_RESCHED_LAZY". This scheduling request is not
    honoured immediately, even if possible, but delayed until the task is
    returning back to userland (allowing running to completion).
    This "NEED_RESCHED_LAZY" scheduling request will be turned into a
    "NEED_RESCHED" request on the next HZ tick should the task not
    schedule on its own before that. This is denoted in the tracing output
    by a "B".
    Real-time tasks use always the "NEED_RESCHED" mechanism to avoid any
    delays.

Wouldn't this do the job?

> +A lazy reschedule therefore does *not* interrupt the target CPU.  It is
> +committed later, in one of two ways:
> +
> +1. On the next return to user mode.  ``__exit_to_user_mode_loop()`` treats
> +   ``_TIF_NEED_RESCHED | _TIF_NEED_RESCHED_LAZY`` as a reschedule request, and
> +   ``xfer_to_guest_mode_work()`` does the same for a vCPU returning to guest
> +   mode.
> +
> +2. In ``scheduler_tick()``, which promotes a pending
> +   ``TIF_NEED_RESCHED_LAZY`` to a full ``TIF_NEED_RESCHED`` once per tick.
> +
> +The tick is thus an upper bound on lazy preemption latency for CFS tasks: a
> +runnable CFS task that has been passed over is picked up within one tick, not
> +immediately.  The tick is guaranteed to run in the case that matters: while
> +more than one CFS task is runnable on the CPU, ``sched_can_stop_tick()``
> +refuses to stop the tick.  A lazy reschedule aimed at the idle task is promoted
> +to a full one instead.

This is an implementation detail I think.

> +That is the intended trade-off -- it avoids sending an IPI to every CPU that
> +has a runnable CFS task -- and it is why ``lazy`` gives up latency in exchange
> +for fewer inter-processor interrupts.

Maybe I miss the point but a wakeup never sends an IPI to every CPU. It
only sends one to the CPU which should run task in a cross-CPU wake-up.

> +The idle task is never delayed: ``__resched_curr()`` promotes a lazy
> +reschedule to a full one when the target is the idle task.

You never explained what an idle task and this is not visible in ps
either. Also an implementation detail.

> +Only the fair scheduler issues lazy reschedules.  RT and deadline tasks are
> +still preempted immediately.  ``lazy`` is therefore not a replacement for
> +PREEMPT_RT; it is a way to keep most of the responsiveness of ``full`` while
> +removing a large part of its IPI traffic.

Not sure it is the IPI traffic. Of course the IPI is absent on cross-CPU
wake-ups. The main point however is to allow "run-to-completion" for the
task. By pushing a scheduling request to the point where it returns to
userland, the "in-kernel-work" has been done. That means all locks have
been dropped so another task will not ask for a mutex which is held by
preempted task. Also it will not write-back cache hot memory which needs
to be dragged back once it schedules back-in.

> +
> +Debugging
> +=========
> +
> +The scheduler debugfs directory provides ``/sys/kernel/debug/sched/preempt``,
> +which lists the models that may be switched to; the active one is enclosed in
> +parentheses::
> +
> +	# cat /sys/kernel/debug/sched/preempt
> +	full (lazy)
> +
> +Writing a model name to the file switches to it.  A name that the kernel
> +cannot select (either because the model is not built in, or because it is not
> +available on this kernel) is rejected with EINVAL.  The ``preempt=``
> +boot parameter rejects such a value as well, but only prints an ``unsupported
> +mode`` warning and keeps the default model.

I added a small point upstairs, you have a bit more here, not sure what
to do about here. Reading and writing is sort of obvious.

> +Measuring
> +=========
> +
> +Two things routinely trip people up when measuring preemption latency.
> +
> +First, the classic real-time latency tools do not exercise lazy preemption at
> +all.  Lazy only applies to the fair class, so waking a SCHED_FIFO task takes
> +``resched_curr()`` and preempts immediately, whichever model is active.  And

The scheduling request happens immediately for SCHED_FIFO, yes. But
preemption does not happen immediately on NONE for instance.

> +when the woken task runs on the CPU the wakeup happens on, even a fair lazy
> +reschedule is committed as soon as that CPU returns to user mode, which for an
> +already runnable task is immediate.  For both reasons such workloads measure
> +the same under ``lazy`` and ``full``.

I don't comprehend that one.

> +
> +The difference only shows up when the woken task is *not* running on the CPU
> +where the wakeup happens.  In that case the waker requests the reschedule, and
> +under ``lazy`` that request is not turned into an IPI; the target CPU only
> +notices at its next tick.  A measurement that cannot distinguish ``none`` from
> +``full`` is usually not measuring the preemption model at all: in the absence
> +of ``lazy``, the reschedule IPI is sent regardless of the model and provides a
> +preemption point even on PREEMPT_NONE.

This is a detail that could be mentioned. However, despite a wakeup (as
of try_to_wake_up()) the scheduler can decide to keep the current task
on CPU (because it did not consume enough of its time slice) and not
schedule the runnable SCHED_OTHER task immediately. There is no
requirement that a freshly woke up task must be scheduled right away.

> +
> +Second, when driving cross-CPU wakeups from a periodic source, keep the wakeup
> +period away from integer multiples of the tick.  A period that is phase-locked
> +to the tick will hide (or invent) exactly the effect that is being measured.
> +Add jitter to the wakeup period, and confirm the active model from the
> +``Dynamic Preempt:`` line rather than assuming that ``preempt=`` took effect.

If you do timer-based wake ups for SCHED_OTHER tasks you also have the
"slack" value. Also you can have jitter from preemption or interrupts
disabled regions.

Sebastian

Re: [PATCH 1/2] sched/doc: add a preemption model overview

[Quchaosheng]

Sebastian,

Thanks for going through it that closely. You found real errors, not just
wording, so here is what I am doing with each.

The "Measuring" section is wrong and comes out entirely. A wakeup does not
send an IPI to every CPU with a runnable CFS task; it targets one CPU. The
rest of that section reasons from that false premise, so there is nothing
in it worth keeping. Your point that a freshly woken SCHED_OTHER task is
not required to run immediately, and that timer slack and preemption or
interrupt disabled regions add their own jitter, was the useful content
and I did not have it.

The rationale is also wrong. I wrote that lazy exists to cut IPI traffic;
the point is run-to-completion, so that in-kernel work finishes before the
switch and no lock is held across it. I will use your wording for the
model table, including the part about real-time tasks not being delayed.

The tick promotion and the idle-task case are implementation details and
go, and so does the debugfs section, for the reason you gave.

That leaves the four models, runtime selection, the note that this applies
to kernel code only, and /sys/kernel/debug/sched/preempt as another way to
see and change the model.

Both patches are held. If what is left is too thin to be worth a file,
say so and I will drop the series rather than post a trimmed version of
something you did not think should exist.

Quchaosheng

Re: [PATCH 1/2] sched/doc: add a preemption model overview

[Sebastian Andrzej Siewior]

On 2026-09-22 15:06:19 [+0800], Quchaosheng wrote:
> Sebastian,
Hi Quchaosheng,

> Both patches are held. If what is left is too thin to be worth a file,
> say so and I will drop the series rather than post a trimmed version of
> something you did not think should exist.

If you think the documentation is worth having then it submit it. The
only thing we have is the documentation for preempt= in
kernel-parameters.txt.

> Quchaosheng

Sebastian