mirror of https://lore.kernel.org/lkml/
 help / color / mirror / Atom feed
* [PATCH v4 0/7] timekeeping/ntp: Fix drift tracking precision
@ 2026-05-25 13:54 David Woodhouse
  2026-05-25 13:54 ` [PATCH v4 1/7] MAINTAINERS: Add Miroslav as timekeeping reviewer David Woodhouse
                   ` (6 more replies)
  0 siblings, 7 replies; 10+ messages in thread
From: David Woodhouse @ 2026-05-25 13:54 UTC (permalink / raw)
  To: Richard Cochran, Wen Gu, David Woodhouse, Andrew Lunn,
	David S. Miller, Eric Dumazet, Jakub Kicinski, Paolo Abeni,
	John Stultz, Thomas Gleixner, Stephen Boyd, Anna-Maria Behnsen,
	Frederic Weisbecker, Shuah Khan, Peter Zijlstra,
	Thomas Weißschuh, Arnd Bergmann, Miroslav Lichvar,
	Julien Ridoux, Ryan Luu, linux-kernel

This is the bugfix subset of the RFC series last posted at
https://lore.kernel.org/all/20260520135207.37826-1-dwmw2@infradead.org/

These patches stand alone and fix several long-standing precision issues 
in the kernel's NTP drift tracking. The feed-forward clock discipline
which was the reason I was *looking*, is dropped for now.

Patch 1: MAINTAINERS update
  Add Miroslav Lichvar as timekeeping reviewer.

Patches 2-3: Timekeeping bugfixes
  2. Remove stale xtime_remainder from ntp_error accumulation.
  3. Account for monotonicity adjustment in ntp_error.

  These fix systematic drift in ntp_error tracking that caused the
  timekeeping dithering to fight against NTP corrections rather than
  assist them.

Patch 4: Independent bugfix
  4. Guard against divide-by-zero during clocksource recalibration.

  Prevents an oops on KVM guests with clocksource=tsc when the TSC
  frequency is recalibrated during boot.

Patches 5-7: NTP rework — eliminate tick_length_base
  5. Drive time_offset skew via per-tick ntp_error transfer instead of
     tick_length inflation, with mult adjustment for dithering bandwidth.
  6. Convert adjtime() to use time_offset directly instead of inflating
     tick_length, removing the rounding loss that prevented convergence.
  7. Remove tick_length_base entirely — tick_length is now always the
     NTP-disciplined value with no per-tick inflation.

  This eliminates the dual-accounting between tick_length and
  tick_length_base that was a source of rounding errors and made the
  code harder to reason about.

The net effect is cleaner accounting in the timekeeping code: the
dual tick_length/tick_length_base bookkeeping is eliminated, and the
ntp_error tracking no longer accumulates systematic bias from stale
remainders or unaccounted monotonicity adjustments.

I've tested this with added debugging tracking the absolute values of
the three time references:
 • (A): xtime, reported as CLOCK_REALTIME.
 • (B): xtime+ntp_error, the time NTP *wants* to report right now.
 • (C): xtime+ntp_error+time_offset, the time NTP wants to skew to.

With the fixes, the ntp_error and time_offset deltas remain entirely
consistent and the kernel perfectly skews to and tracks the reference
setting that it's asked to (although the actual *setting* is now
absent from this series; it's just using the existing tick_length
and time_offset parameters).

Changes since RFC v3:
 - Dropped patches 8-10 (feed-forward infrastructure, ptp_vmclock
   integration, vmclock_host) — those will be reposted separately
   once Thomas's timekeeping branch lands.
 - No code changes to patches 1-7 vs the RFC.

Changes since RFC v2:
 • Renamed "clawback" to "monotonicity adjustment" throughout (patch 2).
 • Drop the exponential tail clamping (v2 patch 3).
 • Convert adjtime() to use time_offset to deliver skew too, and remove
   the separate 'tick_length_base' as adjusting tick_length directly is
   no longer used to skew the clock. The skew_delta basically does the
   same thing, but is easier to get the accounting right.
 • The timekeeping_set_reference() API (patch 8) now takes tk_core.lock
   and computes the phase offset internally, eliminating the race window
   that existed in v2 between setting the reference and the tick code
   consuming it.
 • vmclock_host (patch 10) is no longer marked WIP — it has a selftest
   and proper locking (but is still RFC).
 • Added MAINTAINERS entry for Miroslav Lichvar as timekeeping reviewer
   (patch 1).

David Woodhouse (7):
      MAINTAINERS: Add Miroslav as timekeeping reviewer
      timekeeping: Remove xtime_remainder from ntp_error accumulation
      timekeeping: Account for monotonicity adjustment in ntp_error
      timekeeping: Guard against divide-by-zero in timekeeping_adjust
      timekeeping: Drive time_offset skew via per-tick ntp_error transfer
      ntp: Convert adjtime() to use time_offset instead of tick_length inflation
      ntp: Remove tick_length_base, use tick_length directly

 MAINTAINERS                         |  1 +
 include/linux/timekeeper_internal.h |  4 +-
 kernel/time/ntp.c                   | 99 ++++++++++++++++++++++++++-----------
 kernel/time/ntp_internal.h          |  3 ++
 kernel/time/timekeeping.c           | 41 ++++++++++++---


^ permalink raw reply	[flat|nested] 10+ messages in thread

* [PATCH v4 1/7] MAINTAINERS: Add Miroslav as timekeeping reviewer
  2026-05-25 13:54 [PATCH v4 0/7] timekeeping/ntp: Fix drift tracking precision David Woodhouse
@ 2026-05-25 13:54 ` David Woodhouse
  2026-05-25 13:54 ` [PATCH v4 2/7] timekeeping: Remove xtime_remainder from ntp_error accumulation David Woodhouse
                   ` (5 subsequent siblings)
  6 siblings, 0 replies; 10+ messages in thread
From: David Woodhouse @ 2026-05-25 13:54 UTC (permalink / raw)
  To: Richard Cochran, Wen Gu, David Woodhouse, Andrew Lunn,
	David S. Miller, Eric Dumazet, Jakub Kicinski, Paolo Abeni,
	John Stultz, Thomas Gleixner, Stephen Boyd, Anna-Maria Behnsen,
	Frederic Weisbecker, Shuah Khan, Peter Zijlstra,
	Thomas Weißschuh, Arnd Bergmann, Miroslav Lichvar,
	Julien Ridoux, Ryan Luu, linux-kernel

From: David Woodhouse <dwmw@amazon.co.uk>

If Thomas is going to nudge me on IRC to add Miroslav to Cc on
timekeeping patches, then he might as well actually be listed in the
MAINTAINERS file.

Signed-off-by: David Woodhouse <dwmw@amazon.co.uk>
Acked-by: John Stultz <jstultz@google.com>
---
 MAINTAINERS | 1 +
 1 file changed, 1 insertion(+)

diff --git a/MAINTAINERS b/MAINTAINERS
index 882214b0e7db..de39cc3e7e68 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -26877,6 +26877,7 @@ TIMEKEEPING, CLOCKSOURCE CORE, NTP, ALARMTIMER
 M:	John Stultz <jstultz@google.com>
 M:	Thomas Gleixner <tglx@kernel.org>
 R:	Stephen Boyd <sboyd@kernel.org>
+R:	Miroslav Lichvar <mlichvar@redhat.com>
 L:	linux-kernel@vger.kernel.org
 S:	Supported
 T:	git git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git timers/core
-- 
2.54.0


^ permalink raw reply	[flat|nested] 10+ messages in thread

* [PATCH v4 2/7] timekeeping: Remove xtime_remainder from ntp_error accumulation
  2026-05-25 13:54 [PATCH v4 0/7] timekeeping/ntp: Fix drift tracking precision David Woodhouse
  2026-05-25 13:54 ` [PATCH v4 1/7] MAINTAINERS: Add Miroslav as timekeeping reviewer David Woodhouse
@ 2026-05-25 13:54 ` David Woodhouse
  2026-05-29 14:41   ` David Woodhouse
  2026-05-25 13:54 ` [PATCH v4 3/7] timekeeping: Account for monotonicity adjustment in ntp_error David Woodhouse
                   ` (4 subsequent siblings)
  6 siblings, 1 reply; 10+ messages in thread
From: David Woodhouse @ 2026-05-25 13:54 UTC (permalink / raw)
  To: Richard Cochran, Wen Gu, David Woodhouse, Andrew Lunn,
	David S. Miller, Eric Dumazet, Jakub Kicinski, Paolo Abeni,
	John Stultz, Thomas Gleixner, Stephen Boyd, Anna-Maria Behnsen,
	Frederic Weisbecker, Shuah Khan, Peter Zijlstra,
	Thomas Weißschuh, Arnd Bergmann, Miroslav Lichvar,
	Julien Ridoux, Ryan Luu, linux-kernel

From: David Woodhouse <dwmw@amazon.co.uk>

The ntp_error accumulator tracks the difference between the time actually
reported to consumers in xtime, and the *intended* time. The former is
subject to a sawtooth effect due to the quantisation of 'mult', which
means that it actually advances by 'xtime_interval' each tick, while
the intended clock advances by 'ntp_tick'.

By dithering between adjacent integer values of 'mult' which result in
an 'xtime_interval' slightly higher/lower than the intended tick length,
the advancement of xtime is kept on average to the intended rate.

The accounting should therefore adjust ntp_error by adding ntp_tick and
subtracting xtime_interval on each tick.

Since commit a386b5af8edd ("time: Compensate for rounding on
odd-frequency clocksources") the value subtracted has been
(xtime_interval + xtime_remainder), which is wrong. The effect is a
systematic drift whose magnitude depends on the value of xtime_remainder
and the NTP frequency correction. NTP masks this by continuously
adjusting the frequency to compensate, but with a fixed frequency (or an
external reference clock like vmclock), the drift is exposed.

The value of xtime_remainder actually does represent the difference
between the tick period and xtime_interval, so simply adding it instead
of (+ tick length - xtime_remainder) might have made sense... except
that it's only calculated once at boot time, so it's inaccurate anyway.
So just kill it with fire.

Also remove it from the mult computation in timekeeping_adjust(), which
used it to offset the division for the same (incorrect) reason.

Fixes: a386b5af8edd ("time: Compensate for rounding on odd-frequency clocksources")
Signed-off-by: David Woodhouse <dwmw@amazon.co.uk>
Assisted-by: Kiro:claude-opus-4.6-1m
---
 include/linux/timekeeper_internal.h | 3 ---
 kernel/time/timekeeping.c           | 8 +++-----
 2 files changed, 3 insertions(+), 8 deletions(-)

diff --git a/include/linux/timekeeper_internal.h b/include/linux/timekeeper_internal.h
index e36d11e33e0c..da6cf383bedc 100644
--- a/include/linux/timekeeper_internal.h
+++ b/include/linux/timekeeper_internal.h
@@ -84,8 +84,6 @@ struct tk_read_base {
  * @cycle_interval:		Number of clock cycles in one NTP interval
  * @xtime_interval:		Number of clock shifted nano seconds in one NTP
  *				interval.
- * @xtime_remainder:		Shifted nano seconds left over when rounding
- *				@cycle_interval
  * @raw_interval:		Shifted raw nano seconds accumulated per NTP interval.
  * @next_leap_ktime:		CLOCK_MONOTONIC time value of a pending leap-second
  * @ntp_tick:			The ntp_tick_length() value currently being
@@ -178,7 +176,6 @@ struct timekeeper {
 
 	u64			cycle_interval;
 	u64			xtime_interval;
-	s64			xtime_remainder;
 	u64			raw_interval;
 
 	ktime_t			next_leap_ktime;
diff --git a/kernel/time/timekeeping.c b/kernel/time/timekeeping.c
index b53225b91b4b..9b20f2101c45 100644
--- a/kernel/time/timekeeping.c
+++ b/kernel/time/timekeeping.c
@@ -365,7 +365,6 @@ static void tk_setup_internals(struct timekeeper *tk, struct clocksource *clock)
 
 	/* Go back from cycles -> shifted ns */
 	tk->xtime_interval = interval * clock->mult;
-	tk->xtime_remainder = ntpinterval - tk->xtime_interval;
 	tk->raw_interval = interval * clock->mult;
 
 	 /* if changing clocks, convert xtime_nsec shift units */
@@ -2400,8 +2399,8 @@ static void timekeeping_adjust(struct timekeeper *tk, s64 offset)
 		mult = tk->tkr_mono.mult - tk->ntp_err_mult;
 	} else {
 		tk->ntp_tick = ntp_tl;
-		mult = div64_u64((tk->ntp_tick >> tk->ntp_error_shift) -
-				 tk->xtime_remainder, tk->cycle_interval);
+		mult = div64_u64(tk->ntp_tick >> tk->ntp_error_shift,
+				 tk->cycle_interval);
 	}
 
 	/*
@@ -2526,8 +2525,7 @@ static u64 logarithmic_accumulation(struct timekeeper *tk, u64 offset,
 
 	/* Accumulate error between NTP and clock interval */
 	tk->ntp_error += tk->ntp_tick << shift;
-	tk->ntp_error -= (tk->xtime_interval + tk->xtime_remainder) <<
-						(tk->ntp_error_shift + shift);
+	tk->ntp_error -= tk->xtime_interval << (tk->ntp_error_shift + shift);
 
 	return offset;
 }
-- 
2.54.0


^ permalink raw reply	[flat|nested] 10+ messages in thread

* [PATCH v4 3/7] timekeeping: Account for monotonicity adjustment in ntp_error
  2026-05-25 13:54 [PATCH v4 0/7] timekeeping/ntp: Fix drift tracking precision David Woodhouse
  2026-05-25 13:54 ` [PATCH v4 1/7] MAINTAINERS: Add Miroslav as timekeeping reviewer David Woodhouse
  2026-05-25 13:54 ` [PATCH v4 2/7] timekeeping: Remove xtime_remainder from ntp_error accumulation David Woodhouse
@ 2026-05-25 13:54 ` David Woodhouse
  2026-05-25 13:54 ` [PATCH v4 4/7] timekeeping: Guard against divide-by-zero in timekeeping_adjust David Woodhouse
                   ` (3 subsequent siblings)
  6 siblings, 0 replies; 10+ messages in thread
From: David Woodhouse @ 2026-05-25 13:54 UTC (permalink / raw)
  To: Richard Cochran, Wen Gu, David Woodhouse, Andrew Lunn,
	David S. Miller, Eric Dumazet, Jakub Kicinski, Paolo Abeni,
	John Stultz, Thomas Gleixner, Stephen Boyd, Anna-Maria Behnsen,
	Frederic Weisbecker, Shuah Khan, Peter Zijlstra,
	Thomas Weißschuh, Arnd Bergmann, Miroslav Lichvar,
	Julien Ridoux, Ryan Luu, linux-kernel

From: David Woodhouse <dwmw@amazon.co.uk>

timekeeping_apply_adjustment() modifies xtime_nsec to maintain vDSO
monotonicity when mult changes:

    xtime_nsec -= offset

This ensures that the time reported to userspace does not jump when the
multiplier is adjusted from one tick to the next. However, the ntp_error
accumulator which tracks the difference between intended and actual
clock position was not being updated updated to reflect this additional
discrepancy.

An earlier attempt at this compensation existed as:

    ntp_error -= (interval - offset) << ntp_error_shift

but was removed in commit c2cda2a5bda9 ("timekeeping/ntp: Don't align
NTP frequency adjustments to ticks") because it was a major source of
NTP error. That's because (interval - offset) was wrong: the subtraction
of "interval" prematurely accounted for the changed xtime_interval of
the next tick, which would be correctly accounted in the next
accumulation anyway — a double subtraction.

What is actually needed is just the "offset" part: ntp_error must be
told that xtime_nsec moved by "offset" without a corresponding change
in the intended position. For the normal ±1 mult dithering this is
negligible (the adjustments cancel over time), but for larger mult
changes — such as when an external reference clock sets a new
frequency — the one-time uncompensated offset is significant.

Fix by adjusting ntp_error by the correct amount:

    ntp_error += offset << ntp_error_shift

This keeps ntp_error consistent with the actual xtime_nsec position
after the adjustment.

Fixes: c2cda2a5bda9 ("timekeeping/ntp: Don't align NTP frequency adjustments to ticks")
Signed-off-by: David Woodhouse <dwmw@amazon.co.uk>
Assisted-by: Kiro:claude-opus-4.6-1m
Acked-by: John Stultz <jstultz@google.com>
---
 kernel/time/timekeeping.c | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/kernel/time/timekeeping.c b/kernel/time/timekeeping.c
index 9b20f2101c45..3473a91b11df 100644
--- a/kernel/time/timekeeping.c
+++ b/kernel/time/timekeeping.c
@@ -2370,6 +2370,11 @@ static __always_inline void timekeeping_apply_adjustment(struct timekeeper *tk,
 	 *	xtime_nsec_2 = xtime_nsec_1 - offset
 	 * Which simplifies to:
 	 *	xtime_nsec -= offset
+	 *
+	 * When subtracting offset from xtime_nsec, the same amount
+	 * (in appropriate units) has to be added to ntp_error, in
+	 * order to correctly track the delta between the time
+	 * reported in xtime_nsec, and the intended time.
 	 */
 	if ((mult_adj > 0) && (tk->tkr_mono.mult + mult_adj < mult_adj)) {
 		/* NTP adjustment caused clocksource mult overflow */
@@ -2380,6 +2385,7 @@ static __always_inline void timekeeping_apply_adjustment(struct timekeeper *tk,
 	tk->tkr_mono.mult += mult_adj;
 	tk->xtime_interval += interval;
 	tk->tkr_mono.xtime_nsec -= offset;
+	tk->ntp_error += offset << tk->ntp_error_shift;
 }
 
 /*
-- 
2.54.0


^ permalink raw reply	[flat|nested] 10+ messages in thread

* [PATCH v4 4/7] timekeeping: Guard against divide-by-zero in timekeeping_adjust
  2026-05-25 13:54 [PATCH v4 0/7] timekeeping/ntp: Fix drift tracking precision David Woodhouse
                   ` (2 preceding siblings ...)
  2026-05-25 13:54 ` [PATCH v4 3/7] timekeeping: Account for monotonicity adjustment in ntp_error David Woodhouse
@ 2026-05-25 13:54 ` David Woodhouse
  2026-05-25 13:54 ` [PATCH v4 5/7] timekeeping: Drive time_offset skew via per-tick ntp_error transfer David Woodhouse
                   ` (2 subsequent siblings)
  6 siblings, 0 replies; 10+ messages in thread
From: David Woodhouse @ 2026-05-25 13:54 UTC (permalink / raw)
  To: Richard Cochran, Wen Gu, David Woodhouse, Andrew Lunn,
	David S. Miller, Eric Dumazet, Jakub Kicinski, Paolo Abeni,
	John Stultz, Thomas Gleixner, Stephen Boyd, Anna-Maria Behnsen,
	Frederic Weisbecker, Shuah Khan, Peter Zijlstra,
	Thomas Weißschuh, Arnd Bergmann, Miroslav Lichvar,
	Julien Ridoux, Ryan Luu, linux-kernel

From: David Woodhouse <dwmw@amazon.co.uk>

When the TSC clocksource is recalibrated (e.g. on KVM guests with
clocksource=tsc), cycle_interval can momentarily be zero during the
transition. Guard the div64_u64 in timekeeping_adjust() to prevent a
divide-by-zero oops.

This can be triggered on KVM guests that force clocksource=tsc when
the guest's measured TSC frequency doesn't match what KVM reported,
causing a recalibration during boot.

Signed-off-by: David Woodhouse <dwmw@amazon.co.uk>
Assisted-by: Kiro:claude-opus-4.6-1m
Acked-by: John Stultz <jstultz@google.com>
---
 kernel/time/timekeeping.c | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/kernel/time/timekeeping.c b/kernel/time/timekeeping.c
index 3473a91b11df..75397f5caace 100644
--- a/kernel/time/timekeeping.c
+++ b/kernel/time/timekeeping.c
@@ -2404,6 +2404,8 @@ static void timekeeping_adjust(struct timekeeper *tk, s64 offset)
 	if (likely(tk->ntp_tick == ntp_tl)) {
 		mult = tk->tkr_mono.mult - tk->ntp_err_mult;
 	} else {
+		if (unlikely(!tk->cycle_interval))
+			return;
 		tk->ntp_tick = ntp_tl;
 		mult = div64_u64(tk->ntp_tick >> tk->ntp_error_shift,
 				 tk->cycle_interval);
-- 
2.54.0


^ permalink raw reply	[flat|nested] 10+ messages in thread

* [PATCH v4 5/7] timekeeping: Drive time_offset skew via per-tick ntp_error transfer
  2026-05-25 13:54 [PATCH v4 0/7] timekeeping/ntp: Fix drift tracking precision David Woodhouse
                   ` (3 preceding siblings ...)
  2026-05-25 13:54 ` [PATCH v4 4/7] timekeeping: Guard against divide-by-zero in timekeeping_adjust David Woodhouse
@ 2026-05-25 13:54 ` David Woodhouse
  2026-05-27 18:02   ` David Woodhouse
  2026-05-25 13:54 ` [PATCH v4 6/7] ntp: Convert adjtime() to use time_offset instead of tick_length inflation David Woodhouse
  2026-05-25 13:54 ` [PATCH v4 7/7] ntp: Remove tick_length_base, use tick_length directly David Woodhouse
  6 siblings, 1 reply; 10+ messages in thread
From: David Woodhouse @ 2026-05-25 13:54 UTC (permalink / raw)
  To: Richard Cochran, Wen Gu, David Woodhouse, Andrew Lunn,
	David S. Miller, Eric Dumazet, Jakub Kicinski, Paolo Abeni,
	John Stultz, Thomas Gleixner, Stephen Boyd, Anna-Maria Behnsen,
	Frederic Weisbecker, Shuah Khan, Peter Zijlstra,
	Thomas Weißschuh, Arnd Bergmann, Miroslav Lichvar,
	Julien Ridoux, Ryan Luu, linux-kernel

From: David Woodhouse <dwmw@amazon.co.uk>

Instead of inflating tick_length to effect the time_offset slew,
transfer the skew to ntp_error per-tick and drain time_offset at the
equivalent per-tick rate:

 - ntp_error += skew_delta << shift (biases dithering to deliver skew)
 - time_offset -= skew_delta / NTP_INTERVAL_FREQ (per-tick drain)

Compute mult from (ntp_tick + skew_delta) so the dithering has enough
bandwidth to deliver the skew rate by selecting between mult and mult+1.
This is equivalent to the old tick_length += delta approach but without
modifying tick_length, and with exact per-tick accounting of the
time_offset drain.

To eliminate remainder error in the per-tick division, skew_delta is
rounded to a multiple of NTP_INTERVAL_FREQ in second_overflow().

second_overflow() computes skew_delta (the exponential decay rate)
but no longer drains time_offset or inflates tick_length directly.

Signed-off-by: David Woodhouse <dwmw@amazon.co.uk>
Assisted-by: Kiro:claude-opus-4.6-1m
---
 include/linux/timekeeper_internal.h |  1 +
 kernel/time/ntp.c                   | 35 +++++++++++++++++++++++++++--
 kernel/time/ntp_internal.h          |  2 ++
 kernel/time/timekeeping.c           | 29 +++++++++++++++++++-----
 4 files changed, 60 insertions(+), 7 deletions(-)

diff --git a/include/linux/timekeeper_internal.h b/include/linux/timekeeper_internal.h
index da6cf383bedc..9de6b5b94dc0 100644
--- a/include/linux/timekeeper_internal.h
+++ b/include/linux/timekeeper_internal.h
@@ -184,6 +184,7 @@ struct timekeeper {
 	u32			ntp_error_shift;
 	u32			ntp_err_mult;
 	u32			skip_second_overflow;
+	s64			skew_delta;
 	s32			tai_offset;
 };
 
diff --git a/kernel/time/ntp.c b/kernel/time/ntp.c
index 97fa99b96dd0..3c1e287eb384 100644
--- a/kernel/time/ntp.c
+++ b/kernel/time/ntp.c
@@ -63,6 +63,7 @@ struct ntp_data {
 	int			time_state;
 	int			time_status;
 	s64			time_offset;
+	s64			skew_delta;
 	long			time_constant;
 	long			time_maxerror;
 	long			time_esterror;
@@ -364,6 +365,31 @@ u64 ntp_tick_length(unsigned int tkid)
 	return tk_ntp_data[tkid].tick_length;
 }
 
+s64 ntp_get_skew_delta(unsigned int tkid)
+{
+	return tk_ntp_data[tkid].skew_delta;
+}
+
+s64 ntp_drain_time_offset(unsigned int tkid, s64 amount)
+{
+	struct ntp_data *ntpdata = &tk_ntp_data[tkid];
+
+	/* Only drain if amount and time_offset have the same sign */
+	if (!amount || (amount > 0) != (ntpdata->time_offset > 0))
+		return amount;
+
+	/* Clamp: don't overshoot zero */
+	if (abs(amount) > abs(ntpdata->time_offset)) {
+		s64 undrained = amount - ntpdata->time_offset;
+
+		ntpdata->time_offset = 0;
+		return undrained;
+	}
+
+	ntpdata->time_offset -= amount;
+	return 0;
+}
+
 /**
  * ntp_get_next_leap - Returns the next leapsecond in CLOCK_REALTIME ktime_t
  * @tkid:	Timekeeper ID
@@ -460,9 +486,14 @@ int second_overflow(unsigned int tkid, time64_t secs)
 	/* Compute the phase adjustment for the next second */
 	ntpdata->tick_length	 = ntpdata->tick_length_base;
 
+	/*
+	 * Set the per-tick skew rate for the tick code. This is in the
+	 * same units as tick_length (ns << NTP_SCALE_SHIFT), and is
+	 * rounded to a multiple of NTP_INTERVAL_FREQ so that the per-tick
+	 * division in the tick code is exact.
+	 */
 	delta			 = ntp_offset_chunk(ntpdata, ntpdata->time_offset);
-	ntpdata->time_offset	-= delta;
-	ntpdata->tick_length	+= delta;
+	ntpdata->skew_delta	 = div_s64(delta, NTP_INTERVAL_FREQ) * NTP_INTERVAL_FREQ;
 
 	/* Check PPS signal */
 	pps_dec_valid(ntpdata);
diff --git a/kernel/time/ntp_internal.h b/kernel/time/ntp_internal.h
index 7084d839c207..05e5dd5e1b70 100644
--- a/kernel/time/ntp_internal.h
+++ b/kernel/time/ntp_internal.h
@@ -6,6 +6,8 @@ extern void ntp_init(void);
 extern void ntp_clear(unsigned int tkid);
 /* Returns how long ticks are at present, in ns / 2^NTP_SCALE_SHIFT. */
 extern u64 ntp_tick_length(unsigned int tkid);
+extern s64 ntp_get_skew_delta(unsigned int tkid);
+extern s64 ntp_drain_time_offset(unsigned int tkid, s64 amount);
 extern ktime_t ntp_get_next_leap(unsigned int tkid);
 extern int second_overflow(unsigned int tkid, time64_t secs);
 extern int ntp_adjtimex(unsigned int tkid, struct __kernel_timex *txc, const struct timespec64 *ts,
diff --git a/kernel/time/timekeeping.c b/kernel/time/timekeeping.c
index 75397f5caace..780f66b12916 100644
--- a/kernel/time/timekeeping.c
+++ b/kernel/time/timekeeping.c
@@ -2395,20 +2395,23 @@ static __always_inline void timekeeping_apply_adjustment(struct timekeeper *tk,
 static void timekeeping_adjust(struct timekeeper *tk, s64 offset)
 {
 	u64 ntp_tl = ntp_tick_length(tk->id);
+	s64 skew = ntp_get_skew_delta(tk->id);
 	u32 mult;
 
 	/*
-	 * Determine the multiplier from the current NTP tick length.
-	 * Avoid expensive division when the tick length doesn't change.
+	 * Determine the multiplier from the current NTP tick length plus
+	 * skew_delta. The skew biases mult so that ±1 dithering can deliver
+	 * the time_offset slew rate. Recompute when either changes.
 	 */
-	if (likely(tk->ntp_tick == ntp_tl)) {
+	if (likely(tk->ntp_tick == ntp_tl && tk->skew_delta == skew)) {
 		mult = tk->tkr_mono.mult - tk->ntp_err_mult;
 	} else {
 		if (unlikely(!tk->cycle_interval))
 			return;
 		tk->ntp_tick = ntp_tl;
-		mult = div64_u64(tk->ntp_tick >> tk->ntp_error_shift,
-				 tk->cycle_interval);
+		tk->skew_delta = skew;
+		mult = div64_u64((tk->ntp_tick + skew) >> tk->ntp_error_shift,
+				  tk->cycle_interval);
 	}
 
 	/*
@@ -2535,6 +2538,22 @@ static u64 logarithmic_accumulation(struct timekeeper *tk, u64 offset,
 	tk->ntp_error += tk->ntp_tick << shift;
 	tk->ntp_error -= tk->xtime_interval << (tk->ntp_error_shift + shift);
 
+	/*
+	 * During clock skew driven by ntpdata->time_offset, transfer a
+	 * *portion* of the requested total delta into ntp_error from
+	 * time_offset each tick. The second_overflow() function sets
+	 * the rate of skew, and the value of 'mult' has been selected
+	 * in order to allow the dithering to keep ntp_error around zero
+	 * even while this adjustment is being applied.
+	 */
+	if (tk->skew_delta) {
+		s64 drain = div_s64(tk->skew_delta << shift,
+				    NTP_INTERVAL_FREQ);
+
+		tk->ntp_error += (tk->skew_delta << shift) -
+				 ntp_drain_time_offset(tk->id, drain);
+	}
+
 	return offset;
 }
 
-- 
2.54.0


^ permalink raw reply	[flat|nested] 10+ messages in thread

* [PATCH v4 6/7] ntp: Convert adjtime() to use time_offset instead of tick_length inflation
  2026-05-25 13:54 [PATCH v4 0/7] timekeeping/ntp: Fix drift tracking precision David Woodhouse
                   ` (4 preceding siblings ...)
  2026-05-25 13:54 ` [PATCH v4 5/7] timekeeping: Drive time_offset skew via per-tick ntp_error transfer David Woodhouse
@ 2026-05-25 13:54 ` David Woodhouse
  2026-05-25 13:54 ` [PATCH v4 7/7] ntp: Remove tick_length_base, use tick_length directly David Woodhouse
  6 siblings, 0 replies; 10+ messages in thread
From: David Woodhouse @ 2026-05-25 13:54 UTC (permalink / raw)
  To: Richard Cochran, Wen Gu, David Woodhouse, Andrew Lunn,
	David S. Miller, Eric Dumazet, Jakub Kicinski, Paolo Abeni,
	John Stultz, Thomas Gleixner, Stephen Boyd, Anna-Maria Behnsen,
	Frederic Weisbecker, Shuah Khan, Peter Zijlstra,
	Thomas Weißschuh, Arnd Bergmann, Miroslav Lichvar,
	Julien Ridoux, Ryan Luu, linux-kernel

From: David Woodhouse <dwmw@amazon.co.uk>

The legacy adjtime() syscall (ADJ_ADJTIME/ADJ_OFFSET_SINGLESHOT) used
to slew the clock by inflating tick_length directly via time_adjust.
This was the last remaining user of tick_length != tick_length_base.

Convert it to fold time_adjust into time_offset each second (up to
MAX_TICKADJ per second, same rate limit as before). The existing
time_offset skew mechanism then delivers it via the per-tick ntp_error
transfer and mult adjustment.

Introduce ntp_set_time_offset() helper for setting time_offset from a
nanosecond value, and refactor ntp_update_offset() to use it. This
helper will also be used by the feed-forward reference clock API in a
subsequent commit.

This eliminates the last source of tick_length inflation, making
tick_length always equal tick_length_base.

Signed-off-by: David Woodhouse <dwmw@amazon.co.uk>
Assisted-by: Kiro:claude-opus-4.6-1m
---
 kernel/time/ntp.c          | 52 ++++++++++++++++++++++++--------------
 kernel/time/ntp_internal.h |  1 +
 2 files changed, 34 insertions(+), 19 deletions(-)

diff --git a/kernel/time/ntp.c b/kernel/time/ntp.c
index 3c1e287eb384..9470ac1f597b 100644
--- a/kernel/time/ntp.c
+++ b/kernel/time/ntp.c
@@ -329,7 +329,7 @@ static void ntp_update_offset(struct ntp_data *ntpdata, long offset)
 
 	ntpdata->time_freq   = max(freq_adj, -MAXFREQ_SCALED);
 
-	ntpdata->time_offset = div_s64(offset64 << NTP_SCALE_SHIFT, NTP_INTERVAL_FREQ);
+	ntp_set_time_offset(ntpdata - tk_ntp_data, offset64);
 }
 
 static void __ntp_clear(struct ntp_data *ntpdata)
@@ -390,6 +390,24 @@ s64 ntp_drain_time_offset(unsigned int tkid, s64 amount)
 	return 0;
 }
 
+/**
+ * ntp_set_time_offset - Set the NTP time offset (phase correction)
+ * @tkid:	Timekeeper ID
+ * @offset_ns:	Desired offset in nanoseconds
+ *
+ * Converts nanoseconds to internal time_offset units and stores it.
+ * Also clears time_adjust since a new offset supersedes any pending
+ * adjtime() slew.
+ */
+void ntp_set_time_offset(unsigned int tkid, s64 offset_ns)
+{
+	struct ntp_data *ntpdata = &tk_ntp_data[tkid];
+
+	ntpdata->time_offset = div_s64((s64)offset_ns << NTP_SCALE_SHIFT,
+				       NTP_INTERVAL_FREQ);
+	ntpdata->time_adjust = 0;
+}
+
 /**
  * ntp_get_next_leap - Returns the next leapsecond in CLOCK_REALTIME ktime_t
  * @tkid:	Timekeeper ID
@@ -498,26 +516,22 @@ int second_overflow(unsigned int tkid, time64_t secs)
 	/* Check PPS signal */
 	pps_dec_valid(ntpdata);
 
-	if (!ntpdata->time_adjust)
-		goto out;
-
-	if (ntpdata->time_adjust > MAX_TICKADJ) {
-		ntpdata->time_adjust -= MAX_TICKADJ;
-		ntpdata->tick_length += MAX_TICKADJ_SCALED;
-		goto out;
-	}
-
-	if (ntpdata->time_adjust < -MAX_TICKADJ) {
-		ntpdata->time_adjust += MAX_TICKADJ;
-		ntpdata->tick_length -= MAX_TICKADJ_SCALED;
-		goto out;
+	/*
+	 * Fold any pending time_adjust (from adjtime()) into time_offset.
+	 * This used to inflate tick_length directly; now it uses the same
+	 * per-tick skew mechanism as NTP's time_offset. Rate-limited to
+	 * MAX_TICKADJ (500µs) per second.
+	 */
+	if (ntpdata->time_adjust) {
+		long adj = clamp(ntpdata->time_adjust,
+				 (long)-MAX_TICKADJ, (long)MAX_TICKADJ);
+
+		ntpdata->time_adjust -= adj;
+		ntpdata->time_offset += div_s64(
+			(s64)adj * NSEC_PER_USEC << NTP_SCALE_SHIFT,
+			NTP_INTERVAL_FREQ);
 	}
 
-	ntpdata->tick_length += (s64)(ntpdata->time_adjust * NSEC_PER_USEC / NTP_INTERVAL_FREQ)
-				<< NTP_SCALE_SHIFT;
-	ntpdata->time_adjust = 0;
-
-out:
 	return leap;
 }
 
diff --git a/kernel/time/ntp_internal.h b/kernel/time/ntp_internal.h
index 05e5dd5e1b70..639860ff2baf 100644
--- a/kernel/time/ntp_internal.h
+++ b/kernel/time/ntp_internal.h
@@ -8,6 +8,7 @@ extern void ntp_clear(unsigned int tkid);
 extern u64 ntp_tick_length(unsigned int tkid);
 extern s64 ntp_get_skew_delta(unsigned int tkid);
 extern s64 ntp_drain_time_offset(unsigned int tkid, s64 amount);
+extern void ntp_set_time_offset(unsigned int tkid, s64 offset_ns);
 extern ktime_t ntp_get_next_leap(unsigned int tkid);
 extern int second_overflow(unsigned int tkid, time64_t secs);
 extern int ntp_adjtimex(unsigned int tkid, struct __kernel_timex *txc, const struct timespec64 *ts,
-- 
2.54.0


^ permalink raw reply	[flat|nested] 10+ messages in thread

* [PATCH v4 7/7] ntp: Remove tick_length_base, use tick_length directly
  2026-05-25 13:54 [PATCH v4 0/7] timekeeping/ntp: Fix drift tracking precision David Woodhouse
                   ` (5 preceding siblings ...)
  2026-05-25 13:54 ` [PATCH v4 6/7] ntp: Convert adjtime() to use time_offset instead of tick_length inflation David Woodhouse
@ 2026-05-25 13:54 ` David Woodhouse
  6 siblings, 0 replies; 10+ messages in thread
From: David Woodhouse @ 2026-05-25 13:54 UTC (permalink / raw)
  To: Richard Cochran, Wen Gu, David Woodhouse, Andrew Lunn,
	David S. Miller, Eric Dumazet, Jakub Kicinski, Paolo Abeni,
	John Stultz, Thomas Gleixner, Stephen Boyd, Anna-Maria Behnsen,
	Frederic Weisbecker, Shuah Khan, Peter Zijlstra,
	Thomas Weißschuh, Arnd Bergmann, Miroslav Lichvar,
	Julien Ridoux, Ryan Luu, linux-kernel

From: David Woodhouse <dwmw@amazon.co.uk>

Now that nothing inflates tick_length beyond tick_length_base (the
adjtime path was converted to use time_offset in the previous commit),
the two fields are always equal.

Remove tick_length_base and keep tick_length as the single field.
Remove the per-second reset and the delta update in
ntp_update_frequency() since there is no separate base to track.

No functional change intended.

Signed-off-by: David Woodhouse <dwmw@amazon.co.uk>
Assisted-by: Kiro:claude-opus-4.6-1m
---
 kernel/time/ntp.c | 12 +++---------
 1 file changed, 3 insertions(+), 9 deletions(-)

diff --git a/kernel/time/ntp.c b/kernel/time/ntp.c
index 9470ac1f597b..2aada891aead 100644
--- a/kernel/time/ntp.c
+++ b/kernel/time/ntp.c
@@ -26,8 +26,7 @@
 /**
  * struct ntp_data - Structure holding all NTP related state
  * @tick_usec:		USER_HZ period in microseconds
- * @tick_length:	Adjusted tick length
- * @tick_length_base:	Base value for @tick_length
+ * @tick_length:	Tick length in ns << NTP_SCALE_SHIFT
  * @time_state:		State of the clock synchronization
  * @time_status:	Clock status bits
  * @time_offset:	Time adjustment in nanoseconds
@@ -59,7 +58,6 @@
 struct ntp_data {
 	unsigned long		tick_usec;
 	u64			tick_length;
-	u64			tick_length_base;
 	int			time_state;
 	int			time_status;
 	s64			time_offset;
@@ -246,8 +244,7 @@ static inline void pps_fill_timex(struct ntp_data *ntpdata, struct __kernel_time
 #endif /* CONFIG_NTP_PPS */
 
 /*
- * Update tick_length and tick_length_base, based on tick_usec, ntp_tick_adj and
- * time_freq:
+ * Update tick_length based on tick_usec, ntp_tick_adj and time_freq:
  */
 static void ntp_update_frequency(struct ntp_data *ntpdata)
 {
@@ -264,8 +261,7 @@ static void ntp_update_frequency(struct ntp_data *ntpdata)
 	 * Don't wait for the next second_overflow, apply the change to the
 	 * tick length immediately:
 	 */
-	ntpdata->tick_length		+= new_base - ntpdata->tick_length_base;
-	ntpdata->tick_length_base	 = new_base;
+	ntpdata->tick_length	 = new_base;
 }
 
 static inline s64 ntp_update_offset_fll(struct ntp_data *ntpdata, s64 offset64, long secs)
@@ -342,7 +338,6 @@ static void __ntp_clear(struct ntp_data *ntpdata)
 
 	ntp_update_frequency(ntpdata);
 
-	ntpdata->tick_length	= ntpdata->tick_length_base;
 	ntpdata->time_offset	= 0;
 
 	ntpdata->ntp_next_leap_sec = TIME64_MAX;
@@ -502,7 +497,6 @@ int second_overflow(unsigned int tkid, time64_t secs)
 	}
 
 	/* Compute the phase adjustment for the next second */
-	ntpdata->tick_length	 = ntpdata->tick_length_base;
 
 	/*
 	 * Set the per-tick skew rate for the tick code. This is in the
-- 
2.54.0


^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: [PATCH v4 5/7] timekeeping: Drive time_offset skew via per-tick ntp_error transfer
  2026-05-25 13:54 ` [PATCH v4 5/7] timekeeping: Drive time_offset skew via per-tick ntp_error transfer David Woodhouse
@ 2026-05-27 18:02   ` David Woodhouse
  0 siblings, 0 replies; 10+ messages in thread
From: David Woodhouse @ 2026-05-27 18:02 UTC (permalink / raw)
  To: Richard Cochran, Wen Gu, Andrew Lunn, David S. Miller,
	Eric Dumazet, Jakub Kicinski, Paolo Abeni, John Stultz,
	Thomas Gleixner, Stephen Boyd, Anna-Maria Behnsen,
	Frederic Weisbecker, Shuah Khan, Peter Zijlstra,
	Thomas Weißschuh, Arnd Bergmann, Miroslav Lichvar,
	Julien Ridoux, Ryan Luu, linux-kernel

[-- Attachment #1: Type: text/plain, Size: 1399 bytes --]

On Mon, 2026-05-25 at 14:54 +0100, David Woodhouse wrote:
> 
> @@ -460,9 +486,14 @@ int second_overflow(unsigned int tkid, time64_t secs)
>  	/* Compute the phase adjustment for the next second */
>  	ntpdata->tick_length	 = ntpdata->tick_length_base;
>  
> +	/*
> +	 * Set the per-tick skew rate for the tick code. This is in the
> +	 * same units as tick_length (ns << NTP_SCALE_SHIFT), and is
> +	 * rounded to a multiple of NTP_INTERVAL_FREQ so that the per-tick
> +	 * division in the tick code is exact.
> +	 */
>  	delta			 = ntp_offset_chunk(ntpdata, ntpdata->time_offset);
> -	ntpdata->time_offset	-= delta;
> -	ntpdata->tick_length	+= delta;
> +	ntpdata->skew_delta	 = div_s64(delta, NTP_INTERVAL_FREQ) * NTP_INTERVAL_FREQ;
>  
>  	/* Check PPS signal */
>  	pps_dec_valid(ntpdata);

It turns out the infinite impulse response really *is* infinite because
once time_offset gets low enough that ntp_offset_chunk() returns zero,
that last bit of time_offset is *never* drained.

And rounding down to a multiple of NTP_INTERVAL_FREQ as I do in this
patch, just increases the threshold below which it stops draining.

If I repost this series I'll set skew_delta to ±NTP_INTERVAL_FREQ if it
ends up zero when time_offset isn't. Otherwise I'll do it in a follow-
up.

In practice it barely matters; it's *fractions* of a nanosecond. But
it's annoying :)

[-- Attachment #2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 5069 bytes --]

^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: [PATCH v4 2/7] timekeeping: Remove xtime_remainder from ntp_error accumulation
  2026-05-25 13:54 ` [PATCH v4 2/7] timekeeping: Remove xtime_remainder from ntp_error accumulation David Woodhouse
@ 2026-05-29 14:41   ` David Woodhouse
  0 siblings, 0 replies; 10+ messages in thread
From: David Woodhouse @ 2026-05-29 14:41 UTC (permalink / raw)
  To: Richard Cochran, Wen Gu, Andrew Lunn, David S. Miller,
	Eric Dumazet, Jakub Kicinski, Paolo Abeni, John Stultz,
	Thomas Gleixner, Stephen Boyd, Anna-Maria Behnsen,
	Frederic Weisbecker, Shuah Khan, Peter Zijlstra,
	Thomas Weißschuh, Arnd Bergmann, Miroslav Lichvar,
	Julien Ridoux, Ryan Luu, linux-kernel

[-- Attachment #1: Type: text/plain, Size: 3295 bytes --]

On Mon, 2026-05-25 at 14:54 +0100, David Woodhouse wrote:
> From: David Woodhouse <dwmw@amazon.co.uk>
> 
> The ntp_error accumulator tracks the difference between the time actually
> reported to consumers in xtime, and the *intended* time. The former is
> subject to a sawtooth effect due to the quantisation of 'mult', which
> means that it actually advances by 'xtime_interval' each tick, while
> the intended clock advances by 'ntp_tick'.
> 
> By dithering between adjacent integer values of 'mult' which result in
> an 'xtime_interval' slightly higher/lower than the intended tick length,
> the advancement of xtime is kept on average to the intended rate.
> 
> The accounting should therefore adjust ntp_error by adding ntp_tick and
> subtracting xtime_interval on each tick.
> 
> Since commit a386b5af8edd ("time: Compensate for rounding on
> odd-frequency clocksources") the value subtracted has been
> (xtime_interval + xtime_remainder), which is wrong. The effect is a
> systematic drift whose magnitude depends on the value of xtime_remainder
> and the NTP frequency correction. NTP masks this by continuously
> adjusting the frequency to compensate, but with a fixed frequency (or an
> external reference clock like vmclock), the drift is exposed.
> 
> The value of xtime_remainder actually does represent the difference
> between the tick period and xtime_interval, so simply adding it instead
> of (+ tick length - xtime_remainder) might have made sense... except
> that it's only calculated once at boot time, so it's inaccurate anyway.
> So just kill it with fire.
> 
> Also remove it from the mult computation in timekeeping_adjust(), which
> used it to offset the division for the same (incorrect) reason.
> 
> Fixes: a386b5af8edd ("time: Compensate for rounding on odd-frequency clocksources")
> Signed-off-by: David Woodhouse <dwmw@amazon.co.uk>
> Assisted-by: Kiro:claude-opus-4.6-1m
> ---

Sashiko asks, "Does this removal cause tk->ntp_error to diverge towards
negative infinity when NTP is inactive and cycle_interval rounds up?"

And Sashiko is right; it *does*. But it's also *correct* to do so.

Because tk_setup_internals() currently rounds 'mult' to the *nearest*
value, instead of rounding *down* like timekeeping_adjust() does.

So the dithering that timekeeping_adjust() does between mult and mult+1
can't actually achieve the desired result, and we *do* just drift
further and further away from where we should be, because 'mult' is one
too many than it should be. The values chosen for mult/mult+1 are
supposed to be either *side* of the desired average period.

Any adjtimex() call which sets the frequency will fix this, because
mult *is* correctly calculated in timekeeping_adjust().

I'll fix that pre-existing bug in a separate patch, in v5. Something
like this in tk_setup_internals() should do it:

-	/* Go back from cycles -> shifted ns */
-	tk->xtime_interval = interval * clock->mult;
-	tk->raw_interval = interval * clock->mult;

-	tk->tkr_mono.mult = clock->mult;
+	tk->tkr_mono.mult = div64_u64(ntpinterval, tk-
>cycle_interval);

+	/* Go back from cycles -> shifted ns */
+	tk->xtime_interval = interval * tk->tkr_mono.mult;
+	tk->raw_interval = interval * tk->tkr_raw.mult;



[-- Attachment #2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 5069 bytes --]

^ permalink raw reply	[flat|nested] 10+ messages in thread

end of thread, other threads:[~2026-05-29 14:41 UTC | newest]

Thread overview: 10+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2026-05-25 13:54 [PATCH v4 0/7] timekeeping/ntp: Fix drift tracking precision David Woodhouse
2026-05-25 13:54 ` [PATCH v4 1/7] MAINTAINERS: Add Miroslav as timekeeping reviewer David Woodhouse
2026-05-25 13:54 ` [PATCH v4 2/7] timekeeping: Remove xtime_remainder from ntp_error accumulation David Woodhouse
2026-05-29 14:41   ` David Woodhouse
2026-05-25 13:54 ` [PATCH v4 3/7] timekeeping: Account for monotonicity adjustment in ntp_error David Woodhouse
2026-05-25 13:54 ` [PATCH v4 4/7] timekeeping: Guard against divide-by-zero in timekeeping_adjust David Woodhouse
2026-05-25 13:54 ` [PATCH v4 5/7] timekeeping: Drive time_offset skew via per-tick ntp_error transfer David Woodhouse
2026-05-27 18:02   ` David Woodhouse
2026-05-25 13:54 ` [PATCH v4 6/7] ntp: Convert adjtime() to use time_offset instead of tick_length inflation David Woodhouse
2026-05-25 13:54 ` [PATCH v4 7/7] ntp: Remove tick_length_base, use tick_length directly David Woodhouse

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox

all inboxes | Powered by JetHome®