Re: [PATCH v4] hrtimers: Force migrate away hrtimers queued after CPUHP_AP_HRTIMERS_DYING

["Walter Chang (張維哲)" <Walter.Chang@mediatek.com>]

On Wed, 2025-03-26 at 17:44 +0100, Frederic Weisbecker wrote:
> 
> It's not the first time I get such a report on an out of tree
> kernel. The problem is I don't know if the tainted modules are
> involved. But something is probably making an offline CPU visible
> within
> the hierarchy on get_nohz_timer_target(). And that new warning made
> that visible.
> 
> Can you try this and tell us if the warning fires?
> 
> Thanks.
> 
> diff --git a/include/linux/sched/nohz.h b/include/linux/sched/nohz.h
> index 6d67e9a5af6b..f49512628269 100644
> --- a/include/linux/sched/nohz.h
> +++ b/include/linux/sched/nohz.h
> @@ -9,6 +9,7 @@
>  #if defined(CONFIG_SMP) && defined(CONFIG_NO_HZ_COMMON)
>  extern void nohz_balance_enter_idle(int cpu);
>  extern int get_nohz_timer_target(void);
> +extern void assert_domain_online(void);
>  #else
>  static inline void nohz_balance_enter_idle(int cpu) { }
>  #endif
> diff --git a/kernel/cpu.c b/kernel/cpu.c
> index 07455d25329c..98c8f8408403 100644
> --- a/kernel/cpu.c
> +++ b/kernel/cpu.c
> @@ -13,6 +13,7 @@
>  #include <linux/sched/isolation.h>
>  #include <linux/sched/task.h>
>  #include <linux/sched/smt.h>
> +#include <linux/sched/nohz.h>
>  #include <linux/unistd.h>
>  #include <linux/cpu.h>
>  #include <linux/oom.h>
> @@ -1277,6 +1278,7 @@ static int take_cpu_down(void *_param)
>         if (err < 0)
>                 return err;
> 
> +       assert_domain_online();
>         /*
>          * Must be called from CPUHP_TEARDOWN_CPU, which means, as we
> are going
>          * down, that the current state is CPUHP_TEARDOWN_CPU - 1.
> diff --git a/kernel/sched/core.c b/kernel/sched/core.c
> index 175a5a7ac107..88157b1645cc 100644
> --- a/kernel/sched/core.c
> +++ b/kernel/sched/core.c
> @@ -1163,6 +1163,20 @@ void resched_cpu(int cpu)
> 
>  #ifdef CONFIG_SMP
>  #ifdef CONFIG_NO_HZ_COMMON
> +void assert_domain_online(void)
> +{
> +       int cpu = smp_processor_id();
> +       int i;
> +       struct sched_domain *sd;
> +
> +       guard(rcu)();
> +
> +       for_each_domain(cpu, sd) {
> +               for_each_cpu(i, sched_domain_span(sd)) {
> +                       WARN_ON_ONCE(cpu_is_offline(i));
> +               }
> +       }
> +}
>  /*
>   * In the semi idle case, use the nearest busy CPU for migrating
> timers
>   * from an idle CPU.  This is good for power-savings.

Hi Frederic,

Thank you for providing the patch to debug the hrtimer warning issue.

I have applied the patch and conducted stress testing over the weekend.
And the warning provided in the patch did not occur during this period.

Additionally, after a thorough review of our internal tainted modules,
I can confirm that you are correct in your assessment. The
get_nohz_timer_target() with our tainted modules may indeed return a
CPU that is offline, leading to the hrtimer warning issue. We are
working on fixing this within our tainted modules.

Thanks again for your help in debugging this issue.

Best regards,
Walter Chang