Re: [PATCH] x86/nmi: Ignore negative NMI handler durations

[Rik van Riel <riel@surriel.com>]

On Tue, 2026-10-27 at 03:43 -0700, Breno Leitao wrote:
> I am seeing NMI handlers in the Meta fleet reported as running for
> about 584 years. Over 30 days, 11 AMD and Intel machines logged 637
> such lines, for example:
> 
>   AMD EPYC 9D25 (family 0x1a, model 0x11, stepping 0x0):
>   [ 1650.616627] INFO: NMI handler
> (nmi_cpu_backtrace_handler.llvm.9643156322683129580) took too long to
> run: 18446744073709.551 msecs
> 
>   Intel Xeon Platinum 8321HC (family 0x6, model 0x55, stepping 0xb):
>   [360448.230129] INFO: NMI handler (perf_event_nmi_handler) took too
> long to run: 18446744071018.936 msecs
> 
>   Intel Xeon D-2191A (family 0x6, model 0x55, stepping 0x4):
>   [81787.188959] INFO: NMI handler (ghes_notify_nmi) took too long to
> run: 18446744073709.545 msecs
> 
> nmi_handle() times each handler with two sched_clock() reads. When
> the
> second read is lower than the first, the u64 subtraction wraps, and
> nmi_check_duration() treats the result as a duration. In the three
> examples, the clock went back by less than 1 us, by 2.7 s and by
> about
> 6 us.
> 
The comment above ktime_get_mono_fast_ns() makes
me wonder whether for time deltas we should read
raw TSC values, and calculate the time delta between
them afterward.

The reason is that another CPU might adjust the time
slope between the first and second timestamp read.
I pasted the comment below.

Of course, that still leaves the actual misbehaving
CPUs. A two second time difference is not explained
by a clock conversion adjustment, so I suspect we 
will need this patch either way...

/**
 * ktime_get_mono_fast_ns - Fast NMI safe access to clock monotonic
 *
 * This timestamp is not guaranteed to be monotonic across an update.
 * The timestamp is calculated by:
 *
 *      now = base_mono + clock_delta * slope
 *
 * So if the update lowers the slope, readers who are forced to the
 * not yet updated second array are still using the old steeper slope.
 *
 * tmono
 * ^
 * |    o  n
 * |   o n
 * |  u
 * | o
 * |o
 * |12345678---> reader order
 *
 * o = old slope
 * u = update
 * n = new slope
 *
 * So reader 6 will observe time going backwards versus reader 5.
 *
 * While other CPUs are likely to be able to observe that, the only way
 * for a CPU local observation is when an NMI hits in the middle of
 * the update. Timestamps taken from that NMI context might be ahead
 * of the following timestamps. Callers need to be aware of that and
 * deal with it.
 */




-- 
All Rights Reversed.