[PATCH 0/3] Stop the idle tick on polling idle on nohz_full CPUs

[PATCH 0/3] Stop the idle tick on polling idle on nohz_full CPUs

[Marco Crivellari]

Hi,

On nohz_full CPUs booted with idle=poll, the tick interrupts the task once
after every wakeup from idle. do_idle() restarts the tick before polling,
and on wakeup tick_nohz_idle_exit() only re-evaluates the tick if it is
stopped, so it keeps running and fires in userspace before
tick_nohz_irq_exit() stops it. Without idle=poll this doesn't happen:
since commit a5183862e76f ("tick/nohz: Conditionally restart tick on idle
exit") a nohz_full CPU leaves idle with the tick still stopped.

Marcelo Tosatti proposed stopping the tick on polling idle entry:
  https://lore.kernel.org/all/aQiWfnnSzxsnwa2o@tpad/

Following Frederic's review, this series does it only on nohz_full CPUs,
and handles on the target CPU the cases where stopping the tick in polling
idle would leave a timer unprogrammed.

[ P1 ] tick/nohz: Re-evaluate the tick in polling idle and on user entry

  On nohz_full CPUs the tick is re-evaluated on IRQ exit and on idle exit.
  In idle, IRQ exit only does accounting and relies on the idle loop
  running again, which cpu_idle_poll() doesn't do: it only returns on
  TIF_NEED_RESCHED. P1 closes three gaps:

  - A nohz_full kick handled in idle, e.g. after a timer is enqueued from
    an interrupt on the polling CPU: nohz_full_kick_func() sets
    TIF_NEED_RESCHED so that the polling loop exits and
    tick_nohz_idle_exit() re-evaluates the tick.

  - The expiry of a stopped tick in idle: a stopped tick is programmed for
    the next timer only, and reprogramming it for the following one is
    left to the idle loop. tick_nohz_handler() sets TIF_NEED_RESCHED so
    that timers already queued are programmed in turn.

  - The return to userspace: if the tick was left running in the kernel,
    e.g. because a tick dependency was cleared (which doesn't kick the
    CPU), stop it from context tracking instead of letting it fire once
    in userspace.

  The first two only matter once P2 stops the tick in polling idle, so P1
  must come first.

[ P2 ] sched/idle: Stop the tick on polling idle on nohz_full CPUs

  With idle=poll, nohz_full CPUs stop the tick on idle entry instead of
  restarting it. Housekeeping CPUs keep restarting it, and the
  tick_check_broadcast_expired() polling case is unchanged.

[ P3 ] Documentation: no_hz: Add tick stop in polling idle

  Document the above in the idle=poll item of "Known Issues".

With big effort on the Claude side, I made a test suite to help testing
the work:

  https://github.com/DispatchCode/nohz-full-polling-idle-tick-test-suite

The following is a short summary of the results.

On x86_64 guest, HZ=1000, nohz_full=2-3 idle=poll), a task alternating
100us sleeps with 10ms of userspace work, 200 times:
  - ticks in userspace:              200/200 -> 0/200
  - ticks while polling:             35 -> 0
  - housekeeping CPUs:               unchanged (tick runs while polling)
  - P2 without the kick change:      15 of 20 timers enqueued from an
                                     interrupt on the polling CPU lost
  - P2 without the stopped tick fix: the second of two queued timers up
                                     to 1.9s late
Tick dependencies (POSIX CPU timers, two tasks on the CPU) and CPU hotplug
were also checked.

Thanks,

Marco Crivellari (3):
  tick/nohz: Re-evaluate the tick on polling idle and on user entry
  sched/idle: Stop the tick on polling idle on nohz_full CPUs
  Documentation: no_hz: Add tick stop in polling idle

 Documentation/timers/no_hz.rst |  3 ++
 include/linux/tick.h           | 10 ++++++
 kernel/context_tracking.c      |  9 ++++++
 kernel/sched/idle.c            |  5 ++-
 kernel/time/tick-sched.c       | 59 +++++++++++++++++++++++++++++++++-
 5 files changed, 84 insertions(+), 2 deletions(-)

-- 
2.56.0

[PATCH 1/3] tick/nohz: Re-evaluate the tick on polling idle and on user entry

[Marco Crivellari]

On nohz_full CPUs the tick is re-evaluated on IRQ exit and on idle exit.
There are three places where this is not enough:

- A kick handled while the CPU is in polling idle: in idle,
  tick_nohz_irq_exit() only does accounting and relies on the idle loop
  iterating again. HLT and MWAIT return on any IRQ, but cpu_idle_poll()
  only returns on TIF_NEED_RESCHED. The tick always runs in polling idle
  for now, but once the next patch stops it there, a timer enqueued from
  an interrupt on the polling CPU would not be programmed until something
  else wakes the CPU up.

  Set TIF_NEED_RESCHED on the idle task when the kick is handled in idle,
  so that cpu_idle_poll() returns and tick_nohz_idle_exit() re-evaluates
  the tick.

- The return to userspace: if the tick is left running in the kernel,
  for example because a tick dependency was cleared (tick_nohz_dep_clear*()
  don't kick the CPU) or a timer that kept it running was cancelled, it
  fires at least once in userspace before tick_nohz_irq_exit() stops it.

  Stop it from context tracking right before resuming userspace. When the
  tick is already stopped, the common case, this costs a few flag tests
  on each return to userspace.

- The expiry of a stopped tick in polling idle: a stopped tick is
  programmed for the next timer only, and reprogramming it for the
  following one is left to the idle loop, which doesn't run again while
  cpu_idle_poll() spins. Timers that were already queued would then wait
  for an unrelated wakeup.

  Set TIF_NEED_RESCHED in tick_nohz_handler() when the stopped tick
  fires in idle on a nohz_full CPU.

Cc: Anna-Maria Behnsen <anna-maria@linutronix.de>
Cc: Thomas Gleixner <tglx@kernel.org>
Signed-off-by: Marco Crivellari <marco.crivellari@suse.com>
---
 include/linux/tick.h      |  8 ++++++++
 kernel/context_tracking.c |  9 +++++++++
 kernel/time/tick-sched.c  | 41 ++++++++++++++++++++++++++++++++++++++-
 3 files changed, 57 insertions(+), 1 deletion(-)

diff --git a/include/linux/tick.h b/include/linux/tick.h
index b121c5d53e30..8bfbb053f4fd 100644
--- a/include/linux/tick.h
+++ b/include/linux/tick.h
@@ -269,6 +269,7 @@ static inline void tick_dep_clear_signal(struct signal_struct *signal,
 extern void tick_nohz_full_kick_cpu(int cpu);
 extern void __tick_nohz_task_switch(void);
 extern void __init tick_nohz_full_setup(cpumask_var_t cpumask);
+extern void __tick_nohz_user_enter(void);
 #else
 static inline bool tick_nohz_full_enabled(void) { return false; }
 static inline bool tick_nohz_full_cpu(int cpu) { return false; }
@@ -294,6 +295,7 @@ static inline void tick_dep_clear_signal(struct signal_struct *signal,
 static inline void tick_nohz_full_kick_cpu(int cpu) { }
 static inline void __tick_nohz_task_switch(void) { }
 static inline void tick_nohz_full_setup(cpumask_var_t cpumask) { }
+static inline void __tick_nohz_user_enter(void) { }
 #endif
 
 static inline void tick_nohz_task_switch(void)
@@ -302,6 +304,12 @@ static inline void tick_nohz_task_switch(void)
 		__tick_nohz_task_switch();
 }
 
+static inline void tick_nohz_user_enter(void)
+{
+	if (tick_nohz_full_enabled())
+		__tick_nohz_user_enter();
+}
+
 static inline void tick_nohz_user_enter_prepare(void)
 {
 	if (tick_nohz_full_cpu(smp_processor_id()))
diff --git a/kernel/context_tracking.c b/kernel/context_tracking.c
index a743e7ffa6c0..d4274698d6f3 100644
--- a/kernel/context_tracking.c
+++ b/kernel/context_tracking.c
@@ -24,6 +24,7 @@
 #include <linux/export.h>
 #include <linux/kprobes.h>
 #include <trace/events/rcu.h>
+#include <linux/tick.h>
 
 
 DEFINE_PER_CPU(struct context_tracking, context_tracking) = {
@@ -466,10 +467,18 @@ void noinstr __ct_user_enter(enum ctx_state state)
 			 */
 			if (state == CT_STATE_USER) {
 				instrumentation_begin();
+				/*
+				 * On nohz_full CPUs the tick may have been left running in the
+				 * kernel, e.g. kept by a tick dependency that is now gone.
+				 * Stop it now rather than let it interrupt userspace once
+				 * more before the next IRQ exit notices.
+				 */
+				tick_nohz_user_enter();
 				trace_user_enter(0);
 				vtime_user_enter(current);
 				instrumentation_end();
 			}
+
 			/*
 			 * Other than generic entry implementation, we may be past the last
 			 * rescheduling opportunity in the entry code. Trigger a self IPI
diff --git a/kernel/time/tick-sched.c b/kernel/time/tick-sched.c
index c6d86e4da8a9..a9f622fdeb12 100644
--- a/kernel/time/tick-sched.c
+++ b/kernel/time/tick-sched.c
@@ -323,8 +323,17 @@ static enum hrtimer_restart tick_nohz_handler(struct hrtimer *timer)
 	 * - to the idle task if in dynticks-idle
 	 * - to IRQ exit if in full-dynticks.
 	 */
-	if (unlikely(tick_sched_flag_test(ts, TS_FLAG_STOPPED)))
+	if (unlikely(tick_sched_flag_test(ts, TS_FLAG_STOPPED))) {
+		/*
+		 * In polling idle on nohz_full the idle loop doesn't iterate on
+		 * interrupts, so it would never reprogram the tick for the next
+		 * timer. Make cpu_idle_poll() return.
+		 */
+		if (tick_sched_flag_test(ts, TS_FLAG_INIDLE) &&
+		    tick_nohz_full_cpu(smp_processor_id()))
+			set_tsk_need_resched(current);
 		return HRTIMER_NORESTART;
+	}
 
 	hrtimer_forward(timer, now, TICK_NSEC);
 
@@ -414,6 +423,12 @@ static void nohz_full_kick_func(struct irq_work *work)
 		if (WARN_ON_ONCE(tick_sched_flag_test(ts, TS_FLAG_INIDLE)))
 			return;
 		tick_nohz_full_update_tick(ts);
+	} else if (tick_sched_flag_test(ts, TS_FLAG_INIDLE)) {
+		/*
+		 * In idle, tick_nohz_irq_exit() doesn't re-evaluate the tick, and
+		 * cpu_idle_poll() only returns on need_resched().
+		 */
+		set_tsk_need_resched(current);
 	}
 }
 
@@ -1071,6 +1086,30 @@ static void tick_nohz_full_update_tick(struct tick_sched *ts)
 	__tick_nohz_full_update_tick(ts, ktime_get());
 }
 
+#ifdef CONFIG_NO_HZ_FULL
+void __tick_nohz_user_enter(void)
+{
+	struct tick_sched *ts;
+	int cpu;
+
+	lockdep_assert_irqs_disabled();
+
+	cpu = smp_processor_id();
+	if (!tick_nohz_full_cpu(cpu))
+		return;
+
+	ts = this_cpu_ptr(&tick_cpu_sched);
+	if (!tick_sched_flag_test(ts, TS_FLAG_NOHZ))
+		return;
+
+	if (tick_sched_flag_test(ts, TS_FLAG_STOPPED))
+		return;
+
+	if (can_stop_full_tick(cpu, ts))
+		tick_nohz_full_stop_tick(ts, cpu);
+}
+#endif
+
 /*
  * A pending softirq outside an IRQ (or softirq disabled section) context
  * should be waiting for ksoftirqd to handle it. Therefore we shouldn't
-- 
2.56.0

[PATCH 2/3] sched/idle: Stop the tick on polling idle on nohz_full CPUs

[Marco Crivellari]

With idle=poll, the tick fires while in user space, interrupting the task
running there. This is a problem when the CPU is nohz_full, as it
introduces noise into the workload.

To fix this, stop the tick when entering polling idle on nohz_full CPUs,
while on other CPUs keep restarting it.

Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Juri Lelli <juri.lelli@redhat.com>
Cc: Vincent Guittot <vincent.guittot@linaro.org>
Cc: Dietmar Eggemann <dietmar.eggemann@arm.com>
Cc: Steven Rostedt <rostedt@goodmis.org>
Cc: Ben Segall <bsegall@google.com>
Cc: Mel Gorman <mgorman@suse.de>
Cc: Valentin Schneider <vschneid@redhat.com>
Cc: K Prateek Nayak <kprateek.nayak@amd.com>
Link: https://lore.kernel.org/all/aQiWfnnSzxsnwa2o@tpad/
Signed-off-by: Marco Crivellari <marco.crivellari@suse.com>
---
 include/linux/tick.h     |  2 ++
 kernel/sched/idle.c      |  5 ++++-
 kernel/time/tick-sched.c | 18 ++++++++++++++++++
 3 files changed, 24 insertions(+), 1 deletion(-)

diff --git a/include/linux/tick.h b/include/linux/tick.h
index 8bfbb053f4fd..be4940f76777 100644
--- a/include/linux/tick.h
+++ b/include/linux/tick.h
@@ -129,6 +129,7 @@ extern bool tick_nohz_is_active(void);
 extern bool tick_nohz_tick_stopped(void);
 extern bool tick_nohz_tick_stopped_cpu(int cpu);
 extern void tick_nohz_idle_stop_tick(void);
+extern void tick_nohz_full_idle_stop_tick(void);
 extern void tick_nohz_idle_retain_tick(void);
 extern void tick_nohz_idle_restart_tick(void);
 extern void tick_nohz_idle_enter(void);
@@ -144,6 +145,7 @@ static inline bool tick_nohz_is_active(void) { return false; }
 static inline int tick_nohz_tick_stopped(void) { return 0; }
 static inline int tick_nohz_tick_stopped_cpu(int cpu) { return 0; }
 static inline void tick_nohz_idle_stop_tick(void) { }
+static inline void tick_nohz_full_idle_stop_tick(void) { }
 static inline void tick_nohz_idle_retain_tick(void) { }
 static inline void tick_nohz_idle_restart_tick(void) { }
 static inline void tick_nohz_idle_enter(void) { }
diff --git a/kernel/sched/idle.c b/kernel/sched/idle.c
index e0c10a5dbda2..226d02d24c73 100644
--- a/kernel/sched/idle.c
+++ b/kernel/sched/idle.c
@@ -349,7 +349,10 @@ static void do_idle(void)
 		 * idle as we know that the IPI is going to arrive right away.
 		 */
 		if (cpu_idle_force_poll || tick_check_broadcast_expired()) {
-			tick_nohz_idle_restart_tick();
+			if (cpu_idle_force_poll)
+				tick_nohz_full_idle_stop_tick();
+			else
+				tick_nohz_idle_restart_tick();
 			cpu_idle_poll();
 		} else {
 			cpuidle_idle_call(got_tick);
diff --git a/kernel/time/tick-sched.c b/kernel/time/tick-sched.c
index a9f622fdeb12..3625ad9d2268 100644
--- a/kernel/time/tick-sched.c
+++ b/kernel/time/tick-sched.c
@@ -1179,6 +1179,24 @@ static bool can_stop_idle_tick(int cpu, struct tick_sched *ts)
 	return true;
 }
 
+/**
+ * tick_nohz_full_idle_stop_tick - stop the tick on polling idle on NOHZ
+ *
+ * On nohz_full CPUs the tick restarted for polling idle would keep
+ * running after idle exit and interrupt userspace once more; so stop it
+ * instaed. On other CPUs, keep restarting it as before. Must be called
+ * with interrupts disabled.
+ */
+void tick_nohz_full_idle_stop_tick(void)
+{
+	lockdep_assert_irqs_disabled();
+
+	if (tick_nohz_full_cpu(smp_processor_id()))
+		tick_nohz_idle_stop_tick();
+	else
+		tick_nohz_idle_restart_tick();
+}
+
 /**
  * tick_nohz_idle_stop_tick - stop the idle tick from the idle task
  *
-- 
2.56.0

Re: [PATCH 2/3] sched/idle: Stop the tick on polling idle on nohz_full CPUs

[Frederic Weisbecker]

Le Fri, Nov 06, 2026 at 05:06:14PM +0100, Marco Crivellari a écrit :
> With idle=poll, the tick fires while in user space, interrupting the task
> running there. This is a problem when the CPU is nohz_full, as it
> introduces noise into the workload.
> 
> To fix this, stop the tick when entering polling idle on nohz_full CPUs,
> while on other CPUs keep restarting it.
> 
> Cc: Peter Zijlstra <peterz@infradead.org>
> Cc: Juri Lelli <juri.lelli@redhat.com>
> Cc: Vincent Guittot <vincent.guittot@linaro.org>
> Cc: Dietmar Eggemann <dietmar.eggemann@arm.com>
> Cc: Steven Rostedt <rostedt@goodmis.org>
> Cc: Ben Segall <bsegall@google.com>
> Cc: Mel Gorman <mgorman@suse.de>
> Cc: Valentin Schneider <vschneid@redhat.com>
> Cc: K Prateek Nayak <kprateek.nayak@amd.com>
> Link: https://lore.kernel.org/all/aQiWfnnSzxsnwa2o@tpad/
> Signed-off-by: Marco Crivellari <marco.crivellari@suse.com>
> ---
>  include/linux/tick.h     |  2 ++
>  kernel/sched/idle.c      |  5 ++++-
>  kernel/time/tick-sched.c | 18 ++++++++++++++++++
>  3 files changed, 24 insertions(+), 1 deletion(-)
> 
> diff --git a/include/linux/tick.h b/include/linux/tick.h
> index 8bfbb053f4fd..be4940f76777 100644
> --- a/include/linux/tick.h
> +++ b/include/linux/tick.h
> @@ -129,6 +129,7 @@ extern bool tick_nohz_is_active(void);
>  extern bool tick_nohz_tick_stopped(void);
>  extern bool tick_nohz_tick_stopped_cpu(int cpu);
>  extern void tick_nohz_idle_stop_tick(void);
> +extern void tick_nohz_full_idle_stop_tick(void);
>  extern void tick_nohz_idle_retain_tick(void);
>  extern void tick_nohz_idle_restart_tick(void);
>  extern void tick_nohz_idle_enter(void);
> @@ -144,6 +145,7 @@ static inline bool tick_nohz_is_active(void) { return false; }
>  static inline int tick_nohz_tick_stopped(void) { return 0; }
>  static inline int tick_nohz_tick_stopped_cpu(int cpu) { return 0; }
>  static inline void tick_nohz_idle_stop_tick(void) { }
> +static inline void tick_nohz_full_idle_stop_tick(void) { }
>  static inline void tick_nohz_idle_retain_tick(void) { }
>  static inline void tick_nohz_idle_restart_tick(void) { }
>  static inline void tick_nohz_idle_enter(void) { }
> diff --git a/kernel/sched/idle.c b/kernel/sched/idle.c
> index e0c10a5dbda2..226d02d24c73 100644
> --- a/kernel/sched/idle.c
> +++ b/kernel/sched/idle.c
> @@ -349,7 +349,10 @@ static void do_idle(void)
>  		 * idle as we know that the IPI is going to arrive right away.
>  		 */
>  		if (cpu_idle_force_poll || tick_check_broadcast_expired()) {
> -			tick_nohz_idle_restart_tick();
> +			if (cpu_idle_force_poll)
> +				tick_nohz_full_idle_stop_tick();
> +			else
> +				tick_nohz_idle_restart_tick();
>  			cpu_idle_poll();
>  		} else {
>  			cpuidle_idle_call(got_tick);
> diff --git a/kernel/time/tick-sched.c b/kernel/time/tick-sched.c
> index a9f622fdeb12..3625ad9d2268 100644
> --- a/kernel/time/tick-sched.c
> +++ b/kernel/time/tick-sched.c
> @@ -1179,6 +1179,24 @@ static bool can_stop_idle_tick(int cpu, struct tick_sched *ts)
>  	return true;
>  }
>  
> +/**
> + * tick_nohz_full_idle_stop_tick - stop the tick on polling idle on NOHZ
> + *
> + * On nohz_full CPUs the tick restarted for polling idle would keep
> + * running after idle exit and interrupt userspace once more; so stop it
> + * instaed. On other CPUs, keep restarting it as before. Must be called

*instead

Also it should mention that we want to stop the tick before polling
and not after because we don't want the tick stop induced latency after
the task wake-up.

> + * with interrupts disabled.
> + */
> +void tick_nohz_full_idle_stop_tick(void)

I would rather call that tick_nohz_idle_poll_enter() because it is
very specific and it's not just about stopping the nohz_full tick
in idle, it's also restarting the tick if in dynticks-idle.

Thanks.

> +{
> +	lockdep_assert_irqs_disabled();
> +
> +	if (tick_nohz_full_cpu(smp_processor_id()))
> +		tick_nohz_idle_stop_tick();
> +	else
> +		tick_nohz_idle_restart_tick();
> +}
> +
>  /**
>   * tick_nohz_idle_stop_tick - stop the idle tick from the idle task
>   *
> -- 
> 2.56.0
> 

-- 
Frederic Weisbecker
SUSE Labs

[PATCH 3/3] Documentation: no_hz: Add tick stop in polling idle

[Marco Crivellari]

On nohz_full CPUs the scheduling-clock tick is stopped while polling.
Update the documentation accordingly.

Cc: Anna-Maria Behnsen <anna-maria@linutronix.de>
Cc: Thomas Gleixner <tglx@kernel.org>
Cc: Jonathan Corbet <corbet@lwn.net>
Cc: Shuah Khan <skhan@linuxfoundation.org>
Cc: Randy Dunlap <rdunlap@infradead.org>
Cc: linux-doc@vger.kernel.org
Signed-off-by: Marco Crivellari <marco.crivellari@suse.com>
---
 Documentation/timers/no_hz.rst | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/Documentation/timers/no_hz.rst b/Documentation/timers/no_hz.rst
index 4006cab64079..d3ada4a0ebea 100644
--- a/Documentation/timers/no_hz.rst
+++ b/Documentation/timers/no_hz.rst
@@ -246,6 +246,9 @@ Known Issues
 		be even worse than that of dyntick-idle.  Furthermore,
 		this parameter effectively disables Turbo Mode on Intel
 		CPUs, which can significantly reduce maximum performance.
+		Note also that on nohz_full CPUs the scheduling-clock tick
+		is stopped while polling, so that it does not keep running
+		and interrupt the task once it resumes after idle.
 
 *	Adaptive-ticks slows user/kernel transitions slightly.
 	This is not expected to be a problem for computationally intensive
-- 
2.56.0