[QUESTION] sched/clock: Potential timestamp jump due to inter-CPU counter skew?

[QUESTION] sched/clock: Potential timestamp jump due to inter-CPU counter skew?

[Zhang Qiao]

Hi all,

While reading kernel/time/sched_clock.c, I had a question about counter
synchronization requirements across CPUs.

update_sched_clock() updates the shared epoch_cyc using a counter value read on
the CPU performing the update. Other CPUs subsequently call sched_clock() and
subtract this shared epoch from the counter value they read:

cyc = (rd->read_sched_clock() - rd->epoch_cyc) & rd->sched_clock_mask;
res = rd->epoch_ns + cyc_to_ns(cyc, rd->mult, rd->shift);


If CPU1 observes a new epoch_cyc published by CPU0, but its subsequent counter
read returns a value strictly less than that epoch (assuming no normal counter
wrap-around), the unsigned subtraction would underflow. After masking, this
produces a very large delta and potentially causes sched_clock() to return an
incorrect, massive timestamp.

For example, consider this hypothetical sequence:
1. CPU0 reads the counter and gets 1000.
2. CPU0 completes the epoch update, publishing {epoch_cyc = 1000, epoch_ns = N}.
3. CPU1 observes the new epoch, but its subsequent counter read returns 993.

Assuming a 56-bit counter mask, CPU1 would calculate:
(993 - 1000) & ((1ULL << 56) - 1) = 2^56 - 7

Although the counter read on CPU1 is only seven ticks behind the epoch, the
calculated delta is close to a full counter period. Passing this delta to
cyc_to_ns() could cause a multiplication overflow or yield a completely
incorrect timestamp.

This scenario assumes the inter-CPU counter skew exceeds the time it takes for
CPU0 to publish the new epoch and CPU1 to observe it and read its own counter.
(All values in this example are hypothetical).

My questions are:
1. Are there hardware or architecture specifications (e.g., ARM64 architected
timers, RISC-V) requiring counters on all CPUs—including those on different
dies/sockets—to be perfectly synchronized? If skew is permitted, is there a
specified maximum allowed skew?

2. If the sequence above is theoretically possible, does generic sched_clock()
already have a mechanism to handle a counter read that lags behind the shared
epoch_cyc?

Any comments or suggestions are welcome.

Best regards,
Zhang Qiao

Re: [QUESTION] sched/clock: Potential timestamp jump due to inter-CPU counter skew?

[Zhan Xusheng]

Hi Qiao,

On Mon, Oct 26, 2026 at 02:53:25PM +0800, Zhang Qiao wrote:
> 1. Are there hardware or architecture specifications (e.g., ARM64 architected
> timers, RISC-V) requiring counters on all CPUs—including those on different
> dies/sockets—to be perfectly synchronized? If skew is permitted, is there a
> specified maximum allowed skew?

RISC-V's Zicntr extension requires the real-time clocks of all harts to
be synchronized to within one tick.

On arm64 the system counter is meant to give all PEs a common view of
time, and keeping the copies on separate dies in step is the
implementation's job.  The kernel has no tolerance for skew.  The counter
errata handled in arm_arch_timer.c (Freescale A-008585, HiSilicon
161010101, Cortex-A73 858921, Allwinner A64) are all reads on a single
CPU returning a wrong value, worked around by reading twice; none of them
is skew between CPUs.

> 2. If the sequence above is theoretically possible, does generic sched_clock()
> already have a mechanism to handle a counter read that lags behind the shared
> epoch_cyc?

No.  __sched_clock() masks the difference and converts it directly.
Timekeeping's clocksource_delta() returns 0 when the masked delta exceeds
max_delta, but sched_clock has no such check, so a read behind epoch_cyc
gives a delta of nearly a full period and cyc * mult overflows.

Thanks,
Zhan Xusheng

Re: [QUESTION] sched/clock: Potential timestamp jump due to inter-CPU counter skew?

[Marc Zyngier]

On Mon, 26 Oct 2026 08:27:30 +0000,
Zhan Xusheng <zhanxusheng1024@gmail.com> wrote:
> 
> On arm64 the system counter is meant to give all PEs a common view of
> time, and keeping the copies on separate dies in step is the
> implementation's job.  The kernel has no tolerance for skew.  The counter
> errata handled in arm_arch_timer.c (Freescale A-008585, HiSilicon
> 161010101, Cortex-A73 858921, Allwinner A64) are all reads on a single
> CPU returning a wrong value, worked around by reading twice; none of them
> is skew between CPUs.

Indeed. If the implementation cannot give any two arbitrary CPUs
anywhere in the system the exact same view of the counter, then this
implementation is broken, and unsupported. No discussion.

	M.

-- 
Without deviation from the norm, progress is not possible.

Re: [QUESTION] sched/clock: Potential timestamp jump due to inter-CPU counter skew?

[Peter Zijlstra]

On Mon, Oct 26, 2026 at 09:05:00AM +0000, Marc Zyngier wrote:
> On Mon, 26 Oct 2026 08:27:30 +0000,
> Zhan Xusheng <zhanxusheng1024@gmail.com> wrote:
> > 
> > On arm64 the system counter is meant to give all PEs a common view of
> > time, and keeping the copies on separate dies in step is the
> > implementation's job.  The kernel has no tolerance for skew.  The counter
> > errata handled in arm_arch_timer.c (Freescale A-008585, HiSilicon
> > 161010101, Cortex-A73 858921, Allwinner A64) are all reads on a single
> > CPU returning a wrong value, worked around by reading twice; none of them
> > is skew between CPUs.
> 
> Indeed. If the implementation cannot give any two arbitrary CPUs
> anywhere in the system the exact same view of the counter, then this
> implementation is broken, and unsupported. No discussion.

The limit is the cross-cpu latency.

CPU-0			CPU-1

			A = clock();
t1 = A;
t2 = clock();

BUG_ON(t2-t1 < 0);


CPUs must never be able to tell time goes backwards. This is not to say
the clocks must be in perfect sync, only up to the point where the error
is smaller than the observation limit.

So effectively, if software can tell the clock is wobbly, you've lost.