Re: [PATCH] clocksource: Use the clocksource under test for the skew check timeout

["Aithal, Srikanth" <Srikanth.Aithal@amd.com>]

On 11/4/2026 1:50 AM, Thomas Gleixner wrote:
> On Mon, Nov 02 2026 at 16:45, Mani Tofigh via wrote:
>> watchdog_check_skew() waits for the remote CPU with interrupts disabled
>> and gives up after a timeout measured with ktime_get_raw_fast_ns().
>> While the clocksource is still jiffies during boot (or with
>> clocksource=jiffies) that timeout can't expire, because only the boot
>> CPU's tick advances jiffies and that's the CPU spinning with interrupts
>> disabled.
>>
>> The boot then hangs when the remote CPU can't answer, eg in the
>> stop_machine() of the switch away from jiffies:
>>
>>    CPU0                                  CPU1
>>    multi_cpu_stop()                      multi_cpu_stop()
>>                                            local_irq_disable()
>>                                            /* waits for CPU0 */
>>      <timer interrupt>
>>        clocksource_watchdog()
>>          /* sends CSD to CPU1 */
>>          watchdog_check_skew()
>>            /* waits for CPU1 forever */
>>
>> Nothing gets printed.  With the hard lockup detector on in a KVM guest:
> The above explanation is enough. No need for this backtrace.
>
>> Measure the timeout with the clocksource under test instead, which
>> keeps counting with interrupts disabled.  With the fix, the watchdog
>> gives up after the timeout and the boot goes on.  A 2-vCPU guest on one
>> host core hung in 4 of 150 boots without it, and it didn't hang at all
>> in 400 tests with the fix applied.
>>
>> This bug was identified and confirmed with the help of a deterministic
>> hypervisor currently under development.  The C reproducer below however
>> hits it within about 80s on a kernel booted with clocksource=jiffies.
>>
>> Fixes: 763aacf86f1b ("clocksource: Rewrite watchdog code completely")
>> Cc: stable@vger.kernel.org
>> Signed-off-by: Mani Tofigh <mt4022@columbia.edu>
>> ---
>> Srikanth reported silent early boot hangs of large KVM guests that he
>> bisected to the same commit [1].  I think it's the same bug but it was
>> never confirmed.  Srikanth, would you mind giving this a try on your
>> setup?
> Yes. I gave him a similar patch which simply uses sched_clock()
> instead. That solves the problem, but I was reluctant to post it due to
> the lack of explanation why this ends up in a situation which does not
> allow to make progress.
>
> So your reproducer does:
>
>    while true; do echo 0 >/sys/devices/system/cpu/cpu1/online; echo 1 >/sys/devices/system/cpu/cpu1/online; done
>
> right?
>
> Unfortunately it does not reproduce here within a reasonable amount of
> time (still going strong after one hour...).
>
> But yes, your explanation makes sense and triggering it with the hotplug
> code is obviously a matter of probability.
>
> Though one thing I thought about before looking at your mail while
> reasoning over the possible causes of Srikanth's report and the possible
> consequences is that such a timeout might in the worst case result in a
> stale CPU state which is silently ignored. The consequence is that the
> watchdog wont run on a CPU with such state ever again.
>
> So something like the below delta patch on top of your fix (or the
> sched_clock()) variant might be appropriate.
>
> Can you and Srikanth please run that on top to validate that we don't
> end up with that?
>
> Thanks,
>
>          tglx
> ---
> --- a/kernel/time/clocksource.c
> +++ b/kernel/time/clocksource.c
> @@ -250,6 +250,7 @@ enum wd_result {
>   	WD_FREQ_RESET,
>   	WD_FREQ_SKEWED,
>   	WD_CPU_TIMEOUT,
> +	WD_CPU_STALLED,
>   	WD_CPU_SKEWED,
>   };
>   
> @@ -398,7 +399,7 @@ static void __watchdog_check_cpu_skew(st
>   
>   	wd = per_cpu_ptr(&watchdog_cpu_data, cpu);
>   	if (atomic_read(&wd->remote_inprogress) || wd_csd_locked(wd)) {
> -		watchdog_data.result = WD_CPU_TIMEOUT;
> +		watchdog_data.result = WD_CPU_STALLED;
>   		return;
>   	}
>   
> @@ -590,11 +591,16 @@ static void watchdog_print_freq_skew(str
>   	pr_warn("Clocksource %20s interval: %16lluns\n", cs->name, watchdog_data.cs_delta);
>   }
>   
> -static void watchdog_handle_remote_timeout(struct clocksource *cs)
> +static void watchdog_print_remote_timeout(struct clocksource *cs)
>   {
>   	pr_info_once("Watchdog remote CPU %u read timed out\n", watchdog_data.curr_cpu);
>   }
>   
> +static void watchdog_print_remote_stall(struct clocksource *cs)
> +{
> +	pr_warn("Watchdog remote CPU %u stalled\n", watchdog_data.curr_cpu);
> +}
> +
>   static void watchdog_print_remote_skew(struct clocksource *cs)
>   {
>   	pr_warn("Marking clocksource %s unstable due to inter CPU skew\n", cs->name);
> @@ -635,9 +641,13 @@ static void watchdog_check_result(struct
>   
>   	case WD_CPU_TIMEOUT:
>   		/* Remote check timed out. Try again next cycle. */
> -		watchdog_handle_remote_timeout(cs);
> +		watchdog_print_remote_timeout(cs);
>   		return;
>   
> +	case WD_CPU_STALLED:
> +		watchdog_print_remote_stall(cs);
> +		break;
> +
>   	case WD_CPU_SKEWED:
>   		watchdog_print_remote_skew(cs);
>   		break;


Tested on 6628e7d4c6b25 / v7.4-rc1:

1. Mani's skew-timeout change + your WD_CPU_STALLED delta
2. Mani's patch alone (without the STALLED delta)
Both passed on my large-KVM recreation setup (5× smp=255 SVM guests, 
~2500 reboot stress).

Without these patches, the issue is still recreatable on 6628e7d4c6b25 / 
v7.4-rc1.