[PATCH 2/3] sched/idle: Stop the tick on polling idle on nohz_full CPUs

[Marco Crivellari <marco.crivellari@suse.com>]

With idle=poll, the tick fires while in user space, interrupting the task
running there. This is a problem when the CPU is nohz_full, as it
introduces noise into the workload.

To fix this, stop the tick when entering polling idle on nohz_full CPUs,
while on other CPUs keep restarting it.

Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Juri Lelli <juri.lelli@redhat.com>
Cc: Vincent Guittot <vincent.guittot@linaro.org>
Cc: Dietmar Eggemann <dietmar.eggemann@arm.com>
Cc: Steven Rostedt <rostedt@goodmis.org>
Cc: Ben Segall <bsegall@google.com>
Cc: Mel Gorman <mgorman@suse.de>
Cc: Valentin Schneider <vschneid@redhat.com>
Cc: K Prateek Nayak <kprateek.nayak@amd.com>
Link: https://lore.kernel.org/all/aQiWfnnSzxsnwa2o@tpad/
Signed-off-by: Marco Crivellari <marco.crivellari@suse.com>
---
 include/linux/tick.h     |  2 ++
 kernel/sched/idle.c      |  5 ++++-
 kernel/time/tick-sched.c | 18 ++++++++++++++++++
 3 files changed, 24 insertions(+), 1 deletion(-)

diff --git a/include/linux/tick.h b/include/linux/tick.h
index 8bfbb053f4fd..be4940f76777 100644
--- a/include/linux/tick.h
+++ b/include/linux/tick.h
@@ -129,6 +129,7 @@ extern bool tick_nohz_is_active(void);
 extern bool tick_nohz_tick_stopped(void);
 extern bool tick_nohz_tick_stopped_cpu(int cpu);
 extern void tick_nohz_idle_stop_tick(void);
+extern void tick_nohz_full_idle_stop_tick(void);
 extern void tick_nohz_idle_retain_tick(void);
 extern void tick_nohz_idle_restart_tick(void);
 extern void tick_nohz_idle_enter(void);
@@ -144,6 +145,7 @@ static inline bool tick_nohz_is_active(void) { return false; }
 static inline int tick_nohz_tick_stopped(void) { return 0; }
 static inline int tick_nohz_tick_stopped_cpu(int cpu) { return 0; }
 static inline void tick_nohz_idle_stop_tick(void) { }
+static inline void tick_nohz_full_idle_stop_tick(void) { }
 static inline void tick_nohz_idle_retain_tick(void) { }
 static inline void tick_nohz_idle_restart_tick(void) { }
 static inline void tick_nohz_idle_enter(void) { }
diff --git a/kernel/sched/idle.c b/kernel/sched/idle.c
index e0c10a5dbda2..226d02d24c73 100644
--- a/kernel/sched/idle.c
+++ b/kernel/sched/idle.c
@@ -349,7 +349,10 @@ static void do_idle(void)
 		 * idle as we know that the IPI is going to arrive right away.
 		 */
 		if (cpu_idle_force_poll || tick_check_broadcast_expired()) {
-			tick_nohz_idle_restart_tick();
+			if (cpu_idle_force_poll)
+				tick_nohz_full_idle_stop_tick();
+			else
+				tick_nohz_idle_restart_tick();
 			cpu_idle_poll();
 		} else {
 			cpuidle_idle_call(got_tick);
diff --git a/kernel/time/tick-sched.c b/kernel/time/tick-sched.c
index a9f622fdeb12..3625ad9d2268 100644
--- a/kernel/time/tick-sched.c
+++ b/kernel/time/tick-sched.c
@@ -1179,6 +1179,24 @@ static bool can_stop_idle_tick(int cpu, struct tick_sched *ts)
 	return true;
 }
 
+/**
+ * tick_nohz_full_idle_stop_tick - stop the tick on polling idle on NOHZ
+ *
+ * On nohz_full CPUs the tick restarted for polling idle would keep
+ * running after idle exit and interrupt userspace once more; so stop it
+ * instaed. On other CPUs, keep restarting it as before. Must be called
+ * with interrupts disabled.
+ */
+void tick_nohz_full_idle_stop_tick(void)
+{
+	lockdep_assert_irqs_disabled();
+
+	if (tick_nohz_full_cpu(smp_processor_id()))
+		tick_nohz_idle_stop_tick();
+	else
+		tick_nohz_idle_restart_tick();
+}
+
 /**
  * tick_nohz_idle_stop_tick - stop the idle tick from the idle task
  *
-- 
2.56.0