[PATCH v4 0/4] vdso: Keep the CLOCK_AUX base at full precision

[PATCH v4 0/4] vdso: Keep the CLOCK_AUX base at full precision

[Zhan Xusheng]

v3: https://lore.kernel.org/all/20260901020636.1821993-1-zhanxusheng@xiaomi.com

Changes in v4:

 - New 1/4 converts __iter_div_u64_rem() to OPTIMIZER_HIDE_VAR() as Thomas
   asked, so the two helpers do not diverge.  Measured on the x86 vDSO with
   clang 18: vdso64 text -64 bytes, vdso32 -112.  gcc 13 is unchanged.
   Neither compiler turns the loop into a division with the new form,
   checked at both widths.
 - 2/4 uses the macro too and drops the comment about the memory
   alternative.
 - 2/4 no longer claims the vDSO cannot link a division.  It cannot, but
   this code is on the kernel side, so that was the wrong reason.  The
   reason the loop is right is that the quotient is never more than one,
   which also answers David: there is nothing for a reciprocal multiply to
   save.
 - 3/4 drops the sentence about the advertised granularity, and says
   instead that only the sub-second field changes.
 - Continuation lines aligned under the opening paren.
 - Picked up the Reviewed-by on 3/4.

Sorry about the pace of v2 and v3.

The problem 3/4 fixes: the CLOCK_AUX basetime is shifted down to
nanoseconds and back up, which drops the fractional nanoseconds of
xtime_nsec, so the vDSO floors the base and the cycle delta separately
where ktime_get_aux() floors their sum.  The vDSO reading ends up 0 or 1 ns
below the syscall for the same clock.

Checked under QEMU with an auxiliary clock enabled through
/sys/kernel/time/aux_clocks/0/aux_clock_enable, comparing CLOCK_AUX via the
vDSO against the raw syscall at offset 0, at +5.123456789 s, and with
offs_aux driven negative.  The two agree within read latency in all three,
and over 100000 interleaved pairs each the vDSO reading is never ahead of a
syscall reading taken after it.  The 1 ns bias itself is not measurable
way, a pair of reads costing several hundred nanoseconds.  The argument
for it is the algebra in 3/4, plus a sweep over (xtime_nsec, delta) which
puts the 1 ns case at 60% for shift 24.

Zhan Xusheng (4):
  vdso/math64: Use OPTIMIZER_HIDE_VAR() in __iter_div_u64_rem()
  vdso/math64: Add and use __iter_div64_u64_rem()
  vdso/vsyscall: Keep the CLOCK_AUX base scaled
  vdso/gettimeofday: Assert that the clock id fits the dispatch mask

 include/vdso/math64.h   | 29 ++++++++++++++++++++++++++---
 kernel/time/vsyscall.c  | 26 ++++++++++----------------
 lib/vdso/gettimeofday.c |  2 ++
 3 files changed, 38 insertions(+), 19 deletions(-)


base-commit: 89a312991dc6e638a36adc43ccb91dbc25504c04
-- 
2.43.0

[PATCH v4 1/4] vdso/math64: Use OPTIMIZER_HIDE_VAR() in __iter_div_u64_rem()

[Zhan Xusheng]

The loop subtracts because the quotient is tiny at every caller, and the
asm() keeps the compiler from turning it into a division instead.  It
offers a memory alternative for the value:

	asm("" : "+rm"(dividend));

clang takes that alternative and spills the value inside the loop, on
64-bit as well as 32-bit.  OPTIMIZER_HIDE_VAR() is the register-only form
of the same barrier and is what the rest of the kernel uses for this.

The helper sits on the clock_gettime() path through vdso_set_timespec(),
so this is measurable: x86 vDSO built with clang 18 loses 64 bytes of
vdso64 text and 112 bytes of vdso32.  gcc 13 emits the same code either
way.  Neither compiler turns the loop into a division with the new form,
checked at both widths.

Signed-off-by: Zhan Xusheng <zhanxusheng@xiaomi.com>
---
 include/vdso/math64.h | 8 +++++---
 1 file changed, 5 insertions(+), 3 deletions(-)

diff --git a/include/vdso/math64.h b/include/vdso/math64.h
index 22ae212f8b28..83ebac2e5c1b 100644
--- a/include/vdso/math64.h
+++ b/include/vdso/math64.h
@@ -8,9 +8,11 @@ __iter_div_u64_rem(u64 dividend, u32 divisor, u64 *remainder)
 	u32 ret = 0;
 
 	while (dividend >= divisor) {
-		/* The following asm() prevents the compiler from
-		   optimising this loop into a modulo operation.  */
-		asm("" : "+rm"(dividend));
+		/*
+		 * Prevent the compiler from optimising this loop into a
+		 * modulo operation.
+		 */
+		OPTIMIZER_HIDE_VAR(dividend);
 
 		dividend -= divisor;
 		ret++;
-- 
2.43.0

[PATCH v4 2/4] vdso/math64: Add and use __iter_div64_u64_rem()

[Zhan Xusheng]

The vDSO basetimes for CLOCK_MONOTONIC and CLOCK_BOOTTIME are kept in the
scaled nanoseconds of tkr_mono, so normalising them means dividing by
NSEC_PER_SEC << shift, which does not fit the u32 divisor of
__iter_div_u64_rem().  update_vdso_time_data() therefore open-codes the
iterative division twice.

Subtracting is the right shape here rather than dividing, because the
quotient is never more than one: xtime_nsec is kept below one scaled
second by accumulate_nsecs_to_secs(), and the offset added to it is a
normalised timespec64 fraction, so the dividend stays below twice the
divisor.  That holds on every architecture, which matters more than what
any one of them charges for a division.

Add __iter_div64_u64_rem(), the u64-divisor counterpart of
__iter_div_u64_rem(), and use it for both.  The remainder goes straight
into the basetime as the coarse clocks already do, which also makes the
copy of the CLOCK_MONOTONIC values for CLOCK_BOOTTIME take both of them
from the same place.

The quotient is a u32 like the u32-divisor version returns.  It keeps
32-bit from carrying the counter in a register pair: vsyscall.o loses 16
bytes of text on x86-64 and the loop drops from 36 to 28 instructions on
32-bit gcc.

No functional change.

Suggested-by: David Laight <david.laight.linux@gmail.com>
Signed-off-by: Zhan Xusheng <zhanxusheng@xiaomi.com>
---
 include/vdso/math64.h  | 21 +++++++++++++++++++++
 kernel/time/vsyscall.c | 16 +++++-----------
 2 files changed, 26 insertions(+), 11 deletions(-)

diff --git a/include/vdso/math64.h b/include/vdso/math64.h
index 83ebac2e5c1b..0c91410e5f69 100644
--- a/include/vdso/math64.h
+++ b/include/vdso/math64.h
@@ -23,6 +23,27 @@ __iter_div_u64_rem(u64 dividend, u32 divisor, u64 *remainder)
 	return ret;
 }
 
+static __always_inline u32
+__iter_div64_u64_rem(u64 dividend, u64 divisor, u64 *remainder)
+{
+	u32 ret = 0;
+
+	while (dividend >= divisor) {
+		/*
+		 * Prevent the compiler from optimising this loop into a
+		 * modulo operation.
+		 */
+		OPTIMIZER_HIDE_VAR(dividend);
+
+		dividend -= divisor;
+		ret++;
+	}
+
+	*remainder = dividend;
+
+	return ret;
+}
+
 #if defined(CONFIG_ARCH_SUPPORTS_INT128) && defined(__SIZEOF_INT128__)
 
 #ifndef mul_u64_u32_add_u64_shr
diff --git a/kernel/time/vsyscall.c b/kernel/time/vsyscall.c
index aa59919b8f2c..f43dd3f4744b 100644
--- a/kernel/time/vsyscall.c
+++ b/kernel/time/vsyscall.c
@@ -41,14 +41,12 @@ static inline void update_vdso_time_data(struct vdso_time_data *vdata, struct ti
 
 	nsec = tk->tkr_mono.xtime_nsec;
 	nsec += ((u64)tk->wall_to_monotonic.tv_nsec << tk->tkr_mono.shift);
-	while (nsec >= (((u64)NSEC_PER_SEC) << tk->tkr_mono.shift)) {
-		nsec -= (((u64)NSEC_PER_SEC) << tk->tkr_mono.shift);
-		vdso_ts->sec++;
-	}
-	vdso_ts->nsec	= nsec;
+	vdso_ts->sec	+= __iter_div64_u64_rem(nsec, (u64)NSEC_PER_SEC << tk->tkr_mono.shift,
+						&vdso_ts->nsec);
 
 	/* Copy MONOTONIC time for BOOTTIME */
 	sec	= vdso_ts->sec;
+	nsec	= vdso_ts->nsec;
 	/* Add the boot offset */
 	sec	+= tk->monotonic_to_boot.tv_sec;
 	nsec	+= (u64)tk->monotonic_to_boot.tv_nsec << tk->tkr_mono.shift;
@@ -56,12 +54,8 @@ static inline void update_vdso_time_data(struct vdso_time_data *vdata, struct ti
 	/* CLOCK_BOOTTIME */
 	vdso_ts		= &vc[CS_HRES_COARSE].basetime[CLOCK_BOOTTIME];
 	vdso_ts->sec	= sec;
-
-	while (nsec >= (((u64)NSEC_PER_SEC) << tk->tkr_mono.shift)) {
-		nsec -= (((u64)NSEC_PER_SEC) << tk->tkr_mono.shift);
-		vdso_ts->sec++;
-	}
-	vdso_ts->nsec	= nsec;
+	vdso_ts->sec	+= __iter_div64_u64_rem(nsec, (u64)NSEC_PER_SEC << tk->tkr_mono.shift,
+						&vdso_ts->nsec);
 
 	/* CLOCK_MONOTONIC_RAW */
 	vdso_ts		= &vc[CS_RAW].basetime[CLOCK_MONOTONIC_RAW];
-- 
2.43.0

[PATCH v4 3/4] vdso/vsyscall: Keep the CLOCK_AUX base scaled

[Zhan Xusheng]

The vDSO basetime of a clock is stored in the scaled nanoseconds of
tkr_mono, so that the reader can floor the base and the cycle delta
together in vdso_calc_ns().

vdso_time_update_aux() instead shifts the base down to nanoseconds, adds
the offset, and shifts it back up, which zeroes the fractional nanoseconds
of xtime_nsec.  The reader then floors the base and the delta separately:

  ktime_get_aux():  base + ((delta * mult + xtime_nsec) >> shift)
  vdso:             base + (xtime_nsec >> shift)
                         + ((delta * mult) >> shift)

Since floor(a) + floor(b) <= floor(a + b), the vDSO reports 0 or 1 ns
below the syscall for the same clock.  It is not a monotonicity problem:
across an update the step is floor(a + d) - floor(a) - floor(d), which is
0 or 1, never negative.

Add the offset in scaled nanoseconds as the other high resolution clocks
do, and normalise with __iter_div64_u64_rem() so that the stored base
stays below one second and the userspace fast-path does not iterate more
in __iter_div_u64_rem().

Only the sub-second field changes.  (a + (b << shift)) >> shift is exactly
(a >> shift) + b, so the seconds carried out of the normalisation are the
same as before; what the old form dropped was the low shift bits of the
remainder.

monotonic_to_aux.tv_nsec is a normalised timespec64 fraction, so it stays
below NSEC_PER_SEC even for a negative offset, and the sum stays below
2 * (NSEC_PER_SEC << shift).  The largest shift clocks_calc_mult_shift()
can pick is 32, which makes that 8.6e18 against a u64 limit of 1.8e19.

Fixes: 380b84e168e5 ("vdso/vsyscall: Update auxiliary clock data in the datapage")
Signed-off-by: Zhan Xusheng <zhanxusheng@xiaomi.com>
Reviewed-by: Thomas Weißschuh <thomas.weissschuh@linutronix.de>
---
 kernel/time/vsyscall.c | 10 +++++-----
 1 file changed, 5 insertions(+), 5 deletions(-)

diff --git a/kernel/time/vsyscall.c b/kernel/time/vsyscall.c
index f43dd3f4744b..0e4b499328c0 100644
--- a/kernel/time/vsyscall.c
+++ b/kernel/time/vsyscall.c
@@ -155,11 +155,11 @@ void vdso_time_update_aux(struct timekeeper *tk)
 
 		vdso_ts->sec = tk->xtime_sec + tk->monotonic_to_aux.tv_sec;
 
-		nsec = tk->tkr_mono.xtime_nsec >> tk->tkr_mono.shift;
-		nsec += tk->monotonic_to_aux.tv_nsec;
-		vdso_ts->sec += __iter_div_u64_rem(nsec, NSEC_PER_SEC, &nsec);
-		nsec = nsec << tk->tkr_mono.shift;
-		vdso_ts->nsec = nsec;
+		nsec = tk->tkr_mono.xtime_nsec;
+		nsec += (u64)tk->monotonic_to_aux.tv_nsec << tk->tkr_mono.shift;
+		vdso_ts->sec += __iter_div64_u64_rem(nsec,
+						     (u64)NSEC_PER_SEC << tk->tkr_mono.shift,
+						     &vdso_ts->nsec);
 	}
 
 	__arch_update_vdso_clock(vc);
-- 
2.43.0

[PATCH v4 4/4] vdso/gettimeofday: Assert that the clock id fits the dispatch mask

[Zhan Xusheng]

From: Zhan Xusheng <zhanxusheng1024@gmail.com>

From: Zhan Xusheng <zhanxusheng@xiaomi.com>

The clock id dispatch turns the id into a bit in a u32:

  if (!vdso_clockid_valid(clock))
      return false;
  msk = 1U << clock;

vdso_clockid_valid() admits everything up to CLOCK_AUX_LAST, which is 23,
so the shift is in range.  Nothing states the dependency though, and
raising MAX_AUX_CLOCKS past 16 would take CLOCK_AUX_LAST to 32 or beyond
and make the shift undefined.

Assert it at both dispatch sites.  The condition is on a parameter rather
than a constant, so it relies on the compiler deriving the range from the
vdso_clockid_valid() bail-out above it.  gcc 13 and clang 18 both do: x86
vdso64 and vdso32 build clean, and raising MAX_AUX_CLOCKS to 17 fails the
assert as intended.

Suggested-by: Thomas Weißschuh <thomas.weissschuh@linutronix.de>
Signed-off-by: Zhan Xusheng <zhanxusheng@xiaomi.com>
Reviewed-by: Thomas Weißschuh <thomas.weissschuh@linutronix.de>
---
No change since v1.

 lib/vdso/gettimeofday.c | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/lib/vdso/gettimeofday.c b/lib/vdso/gettimeofday.c
index f7a591aba59f..ef4dcc614489 100644
--- a/lib/vdso/gettimeofday.c
+++ b/lib/vdso/gettimeofday.c
@@ -285,6 +285,7 @@ __cvdso_clock_gettime_common(const struct vdso_time_data *vd, clockid_t clock,
 	 * Convert the clockid to a bitmask and use it to check which
 	 * clocks are handled in the VDSO directly.
 	 */
+	BUILD_BUG_ON(clock >= BITS_PER_TYPE(msk));
 	msk = 1U << clock;
 	if (likely(msk & VDSO_HRES))
 		vc = &vc[CS_HRES_COARSE];
@@ -438,6 +439,7 @@ bool __cvdso_clock_getres_common(const struct vdso_time_data *vd, clockid_t cloc
 	 * Convert the clockid to a bitmask and use it to check which
 	 * clocks are handled in the VDSO directly.
 	 */
+	BUILD_BUG_ON(clock >= BITS_PER_TYPE(msk));
 	msk = 1U << clock;
 	if (msk & (VDSO_HRES | VDSO_RAW)) {
 		/*
-- 
2.43.0

Re: [PATCH v4 2/4] vdso/math64: Add and use __iter_div64_u64_rem()

[Thomas Weißschuh]

Hi Zhan,

This was still quite a fast resend, please slow down a bit more.

On Wed, Sep 02, 2026 at 11:37:59AM +0800, Zhan Xusheng wrote:
(...)

> Suggested-by: David Laight <david.laight.linux@gmail.com>
> Signed-off-by: Zhan Xusheng <zhanxusheng@xiaomi.com>

Reviewed-by: Thomas Weißschuh <thomas.weissschuh@linutronix.de>

> ---
>  include/vdso/math64.h  | 21 +++++++++++++++++++++
>  kernel/time/vsyscall.c | 16 +++++-----------
>  2 files changed, 26 insertions(+), 11 deletions(-)

(...)

Re: [PATCH v4 1/4] vdso/math64: Use OPTIMIZER_HIDE_VAR() in __iter_div_u64_rem()

[Thomas Gleixner]

On Wed, Sep 02 2026 at 11:37, Zhan Xusheng wrote:
> The loop subtracts because the quotient is tiny at every caller, and the

Which loop?

Care to read and follow:

  https://docs.kernel.org/process/maintainer-tip.html#patch-submission-notes

That's not optional.

Re: [PATCH v4 4/4] vdso/gettimeofday: Assert that the clock id fits the dispatch mask

[Thomas Gleixner]

On Wed, Sep 02 2026 at 11:38, Zhan Xusheng wrote:

> From: Zhan Xusheng <zhanxusheng1024@gmail.com>
>
> From: Zhan Xusheng <zhanxusheng@xiaomi.com>

Can you please slow down and decide which of your multiple personalities
actually wrote that patch?

> The clock id dispatch turns the id into a bit in a u32:

What is a a clock_id dispatch? Please explain things in the actual
context as I pointed out to you before. Making things up is not helpful.

Re: [PATCH v4 2/4] vdso/math64: Add and use __iter_div64_u64_rem()

[Thomas Gleixner]

On Wed, Sep 02 2026 at 11:37, Zhan Xusheng wrote:
>  
> +static __always_inline u32
> +__iter_div64_u64_rem(u64 dividend, u64 divisor, u64 *remainder)

What's this line break for?

That's unreadable gunk. You have 100 characters.