* [PATCH 1/2] arm64: Support inlined clocksource reads for the arch counter
2026-10-02 15:19 [PATCH 0/2] Optimise ktime_get_snapshot_id() for capture latency David Woodhouse
@ 2026-10-02 15:19 ` David Woodhouse
2026-10-02 16:36 ` Mark Rutland
2026-10-02 15:19 ` [PATCH 2/2] timekeeping: Use inlined counter read in ktime_get_snapshot_id() David Woodhouse
2026-10-02 16:41 ` [PATCH 0/2] Optimise ktime_get_snapshot_id() for capture latency Mark Rutland
2 siblings, 1 reply; 6+ messages in thread
From: David Woodhouse @ 2026-10-02 15:19 UTC (permalink / raw)
To: Thomas Gleixner, John Stultz
Cc: Stephen Boyd, Miroslav Lichvar, Catalin Marinas, Will Deacon,
Mark Rutland, Marc Zyngier, Daniel Lezcano, Rodolfo Giometti,
linux-arm-kernel, linux-kernel, David Woodhouse, David Woodhouse
From: David Woodhouse <dwmw@amazon.co.uk>
Commit 2e27beeb66e43 ("timekeeping: Allow inlining clocksource::read()")
allows an architecture's clocksource::read() to be inlined, and was only
wired up for x86_64 thus far in commit b27801189f7fc ("x86: Inline TSC
reads in timekeeping").
Add the corresponding ARCH_WANTS_CLOCKSOURCE_READ_INLINE support for the
arm64 arch counter, allowing the indirect clock->read() call in the fast
paths to be replaced with a direct sysreg read. A static key selects
CNTPCT vs CNTVCT as appropriate, and the inline support is only
advertised for the plain (no erratum workaround) read paths.
Measured on a Cortex-A53 at 1.35GHz (12.5MHz arch counter): the
interval from a raw counter read in the caller to the counter read
inside ktime_get_real_ts64() drops from 30.2ns to 23.4ns, mean of 1M
iterations. Every ktime_get*() user of tk_clock_read() benefits
similarly. Except ktime_get_snapshot_id(), which I shall get to shortly.
Signed-off-by: David Woodhouse <dwmw@amazon.co.uk>
Assisted-by: LLM
---
arch/arm64/Kconfig | 1 +
arch/arm64/include/asm/clock_inlined.h | 26 ++++++++++++++++++++++++++
drivers/clocksource/arm_arch_timer.c | 21 +++++++++++++++++++++
3 files changed, 48 insertions(+)
create mode 100644 arch/arm64/include/asm/clock_inlined.h
diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig
index b5a51b0ef944..25795b87cd41 100644
--- a/arch/arm64/Kconfig
+++ b/arch/arm64/Kconfig
@@ -94,6 +94,7 @@ config ARM64
select ARCH_WANT_FRAME_POINTERS
select ARCH_WANT_HUGE_PMD_SHARE if ARM64_4K_PAGES || (ARM64_16K_PAGES && !ARM64_VA_BITS_36)
select ARCH_WANT_LD_ORPHAN_WARN
+ select ARCH_WANTS_CLOCKSOURCE_READ_INLINE
select ARCH_WANTS_EXECMEM_LATE
select ARCH_WANTS_NO_INSTR
select ARCH_WANTS_THP_SWAP if ARM64_4K_PAGES
diff --git a/arch/arm64/include/asm/clock_inlined.h b/arch/arm64/include/asm/clock_inlined.h
new file mode 100644
index 000000000000..fe997412bc7d
--- /dev/null
+++ b/arch/arm64/include/asm/clock_inlined.h
@@ -0,0 +1,26 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef _ASM_ARM64_CLOCK_INLINED_H
+#define _ASM_ARM64_CLOCK_INLINED_H
+
+#include <linux/jump_label.h>
+#include <asm/arch_timer.h>
+
+struct clocksource;
+
+/*
+ * Set by the arch timer driver when it selects the physical counter
+ * (kernel booted at EL2); otherwise the virtual counter is used. The
+ * driver only advertises CLOCK_SOURCE_CAN_INLINE_READ when it selected
+ * a plain (no erratum workaround) sysreg read matching this function.
+ */
+DECLARE_STATIC_KEY_FALSE(arch_timer_inline_read_pct);
+
+static __always_inline u64 arch_inlined_clocksource_read(struct clocksource *cs)
+{
+ if (static_branch_likely(&arch_timer_inline_read_pct))
+ return __arch_counter_get_cntpct();
+
+ return __arch_counter_get_cntvct();
+}
+
+#endif
diff --git a/drivers/clocksource/arm_arch_timer.c b/drivers/clocksource/arm_arch_timer.c
index cc9a8b399004..1c3d503d881a 100644
--- a/drivers/clocksource/arm_arch_timer.c
+++ b/drivers/clocksource/arm_arch_timer.c
@@ -133,6 +133,13 @@ static noinstr u64 arch_counter_get_cntvct(void)
u64 (*arch_timer_read_counter)(void) __ro_after_init = arch_counter_get_cntvct;
EXPORT_SYMBOL_GPL(arch_timer_read_counter);
+#ifdef CONFIG_ARM64
+#include <asm/clock_inlined.h>
+
+/* Selects CNTPCT in arch_inlined_clocksource_read() */
+DEFINE_STATIC_KEY_FALSE(arch_timer_inline_read_pct);
+#endif
+
static u64 arch_counter_read(struct clocksource *cs)
{
return arch_timer_read_counter();
@@ -917,6 +924,15 @@ static void __init arch_counter_register(void)
} else {
rd = arch_counter_get_cntvct;
scr = arch_counter_get_cntvct;
+ /*
+ * A plain sysreg read with no erratum workaround
+ * matches arch_inlined_clocksource_read(); allow the
+ * core timekeeping code to bypass the indirect call.
+ */
+#ifdef CONFIG_ARM64
+ clocksource_counter.flags |=
+ CLOCK_SOURCE_CAN_INLINE_READ;
+#endif
}
} else {
if (arch_timer_counter_has_wa()) {
@@ -925,6 +941,11 @@ static void __init arch_counter_register(void)
} else {
rd = arch_counter_get_cntpct;
scr = arch_counter_get_cntpct;
+#ifdef CONFIG_ARM64
+ static_branch_enable(&arch_timer_inline_read_pct);
+ clocksource_counter.flags |=
+ CLOCK_SOURCE_CAN_INLINE_READ;
+#endif
}
}
--
2.43.0
^ permalink raw reply [flat|nested] 6+ messages in thread* Re: [PATCH 1/2] arm64: Support inlined clocksource reads for the arch counter
2026-10-02 15:19 ` [PATCH 1/2] arm64: Support inlined clocksource reads for the arch counter David Woodhouse
@ 2026-10-02 16:36 ` Mark Rutland
0 siblings, 0 replies; 6+ messages in thread
From: Mark Rutland @ 2026-10-02 16:36 UTC (permalink / raw)
To: David Woodhouse
Cc: Thomas Gleixner, John Stultz, Stephen Boyd, Miroslav Lichvar,
Catalin Marinas, Will Deacon, Marc Zyngier, Daniel Lezcano,
Rodolfo Giometti, linux-arm-kernel, linux-kernel,
David Woodhouse
Hi David,
On Fri, Oct 02, 2026 at 04:19:37PM +0100, David Woodhouse wrote:
> From: David Woodhouse <dwmw@amazon.co.uk>
>
> Commit 2e27beeb66e43 ("timekeeping: Allow inlining clocksource::read()")
> allows an architecture's clocksource::read() to be inlined, and was only
> wired up for x86_64 thus far in commit b27801189f7fc ("x86: Inline TSC
> reads in timekeeping").
>
> Add the corresponding ARCH_WANTS_CLOCKSOURCE_READ_INLINE support for the
> arm64 arch counter, allowing the indirect clock->read() call in the fast
> paths to be replaced with a direct sysreg read. A static key selects
> CNTPCT vs CNTVCT as appropriate, and the inline support is only
> advertised for the plain (no erratum workaround) read paths.
>
> Measured on a Cortex-A53 at 1.35GHz (12.5MHz arch counter): the
> interval from a raw counter read in the caller to the counter read
> inside ktime_get_real_ts64() drops from 30.2ns to 23.4ns, mean of 1M
> iterations. Every ktime_get*() user of tk_clock_read() benefits
> similarly. Except ktime_get_snapshot_id(), which I shall get to shortly.
>
> Signed-off-by: David Woodhouse <dwmw@amazon.co.uk>
> Assisted-by: LLM
On IRC you mentioned that the saving here was dominated by other costs
(in the microseconds range). Given that, as I mentioned there, I'd
prefer to avoid this.
The arm_arch_timer driver is complicated enough with all the errata
handling, with which we've had more than one painful problem. I don't
want more complication there just to game a microbenchmark.
Mark.
> ---
> arch/arm64/Kconfig | 1 +
> arch/arm64/include/asm/clock_inlined.h | 26 ++++++++++++++++++++++++++
> drivers/clocksource/arm_arch_timer.c | 21 +++++++++++++++++++++
> 3 files changed, 48 insertions(+)
> create mode 100644 arch/arm64/include/asm/clock_inlined.h
>
> diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig
> index b5a51b0ef944..25795b87cd41 100644
> --- a/arch/arm64/Kconfig
> +++ b/arch/arm64/Kconfig
> @@ -94,6 +94,7 @@ config ARM64
> select ARCH_WANT_FRAME_POINTERS
> select ARCH_WANT_HUGE_PMD_SHARE if ARM64_4K_PAGES || (ARM64_16K_PAGES && !ARM64_VA_BITS_36)
> select ARCH_WANT_LD_ORPHAN_WARN
> + select ARCH_WANTS_CLOCKSOURCE_READ_INLINE
> select ARCH_WANTS_EXECMEM_LATE
> select ARCH_WANTS_NO_INSTR
> select ARCH_WANTS_THP_SWAP if ARM64_4K_PAGES
> diff --git a/arch/arm64/include/asm/clock_inlined.h b/arch/arm64/include/asm/clock_inlined.h
> new file mode 100644
> index 000000000000..fe997412bc7d
> --- /dev/null
> +++ b/arch/arm64/include/asm/clock_inlined.h
> @@ -0,0 +1,26 @@
> +/* SPDX-License-Identifier: GPL-2.0 */
> +#ifndef _ASM_ARM64_CLOCK_INLINED_H
> +#define _ASM_ARM64_CLOCK_INLINED_H
> +
> +#include <linux/jump_label.h>
> +#include <asm/arch_timer.h>
> +
> +struct clocksource;
> +
> +/*
> + * Set by the arch timer driver when it selects the physical counter
> + * (kernel booted at EL2); otherwise the virtual counter is used. The
> + * driver only advertises CLOCK_SOURCE_CAN_INLINE_READ when it selected
> + * a plain (no erratum workaround) sysreg read matching this function.
> + */
> +DECLARE_STATIC_KEY_FALSE(arch_timer_inline_read_pct);
> +
> +static __always_inline u64 arch_inlined_clocksource_read(struct clocksource *cs)
> +{
> + if (static_branch_likely(&arch_timer_inline_read_pct))
> + return __arch_counter_get_cntpct();
> +
> + return __arch_counter_get_cntvct();
> +}
> +
> +#endif
> diff --git a/drivers/clocksource/arm_arch_timer.c b/drivers/clocksource/arm_arch_timer.c
> index cc9a8b399004..1c3d503d881a 100644
> --- a/drivers/clocksource/arm_arch_timer.c
> +++ b/drivers/clocksource/arm_arch_timer.c
> @@ -133,6 +133,13 @@ static noinstr u64 arch_counter_get_cntvct(void)
> u64 (*arch_timer_read_counter)(void) __ro_after_init = arch_counter_get_cntvct;
> EXPORT_SYMBOL_GPL(arch_timer_read_counter);
>
> +#ifdef CONFIG_ARM64
> +#include <asm/clock_inlined.h>
> +
> +/* Selects CNTPCT in arch_inlined_clocksource_read() */
> +DEFINE_STATIC_KEY_FALSE(arch_timer_inline_read_pct);
> +#endif
> +
> static u64 arch_counter_read(struct clocksource *cs)
> {
> return arch_timer_read_counter();
> @@ -917,6 +924,15 @@ static void __init arch_counter_register(void)
> } else {
> rd = arch_counter_get_cntvct;
> scr = arch_counter_get_cntvct;
> + /*
> + * A plain sysreg read with no erratum workaround
> + * matches arch_inlined_clocksource_read(); allow the
> + * core timekeeping code to bypass the indirect call.
> + */
> +#ifdef CONFIG_ARM64
> + clocksource_counter.flags |=
> + CLOCK_SOURCE_CAN_INLINE_READ;
> +#endif
> }
> } else {
> if (arch_timer_counter_has_wa()) {
> @@ -925,6 +941,11 @@ static void __init arch_counter_register(void)
> } else {
> rd = arch_counter_get_cntpct;
> scr = arch_counter_get_cntpct;
> +#ifdef CONFIG_ARM64
> + static_branch_enable(&arch_timer_inline_read_pct);
> + clocksource_counter.flags |=
> + CLOCK_SOURCE_CAN_INLINE_READ;
> +#endif
> }
> }
>
> --
> 2.43.0
>
^ permalink raw reply [flat|nested] 6+ messages in thread
* [PATCH 2/2] timekeeping: Use inlined counter read in ktime_get_snapshot_id()
2026-10-02 15:19 [PATCH 0/2] Optimise ktime_get_snapshot_id() for capture latency David Woodhouse
2026-10-02 15:19 ` [PATCH 1/2] arm64: Support inlined clocksource reads for the arch counter David Woodhouse
@ 2026-10-02 15:19 ` David Woodhouse
2026-10-02 16:41 ` [PATCH 0/2] Optimise ktime_get_snapshot_id() for capture latency Mark Rutland
2 siblings, 0 replies; 6+ messages in thread
From: David Woodhouse @ 2026-10-02 15:19 UTC (permalink / raw)
To: Thomas Gleixner, John Stultz
Cc: Stephen Boyd, Miroslav Lichvar, Catalin Marinas, Will Deacon,
Mark Rutland, Marc Zyngier, Daniel Lezcano, Rodolfo Giometti,
linux-arm-kernel, linux-kernel, David Woodhouse, David Woodhouse
From: David Woodhouse <dwmw@amazon.co.uk>
Every instruction between a PPS/hwstamp caller's triggering event and
the counter read inside ktime_get_snapshot_id() displaces the returned
timestamp from that event. When the clocksource supports inlined
reads, read the counter directly in tk_clock_read_snapshot(), avoiding
the indirect call. The inlined read cannot provide a hardware
snapshot, so clear the chs struct after the read instead of
beforehand.
Also bypass the clock_id case statement for CLOCK_REALTIME, which is the
common and latency-critical caller.
Measured on a Cortex-A53 at 1.35GHz (12.5MHz arch counter), with
inlined clocksource reads enabled: the interval from a raw counter
read in the caller to the counter read inside the function drops from
49.9ns to 41.2ns with the inline read, and to 38.8ns with the dispatch
bias, mean of 1M iterations.
Assisted-by: LLM
Signed-off-by: David Woodhouse <dwmw@amazon.co.uk>
---
kernel/time/timekeeping.c | 110 +++++++++++++++++++++++++-------------
1 file changed, 73 insertions(+), 37 deletions(-)
diff --git a/kernel/time/timekeeping.c b/kernel/time/timekeeping.c
index ea2e6e55f37b..f0768efb8cff 100644
--- a/kernel/time/timekeeping.c
+++ b/kernel/time/timekeeping.c
@@ -304,6 +304,30 @@ static __always_inline u64 tk_clock_read(const struct tk_read_base *tkr)
return clock->read(clock);
}
+static inline u64 tk_clock_read_snapshot(const struct tk_read_base *tkr,
+ struct clocksource_hw_snapshot *chs)
+{
+ struct clocksource *clock = READ_ONCE(tkr->clock);
+ u64 count;
+
+ if (static_branch_likely(&clocksource_read_inlined)) {
+ count = arch_inlined_clocksource_read(clock);
+ } else if (unlikely(clock->read_snapshot)) {
+ *chs = (struct clocksource_hw_snapshot){ };
+ return clock->read_snapshot(clock, chs);
+ } else {
+ count = clock->read(clock);
+ }
+
+ /*
+ * Unless it's *used* by ->read_snapshot(), this can be cleared
+ * *after* the clocksource is read, allowing the clocksource to
+ * be sampled sooner.
+ */
+ *chs = (struct clocksource_hw_snapshot){ };
+ return count;
+}
+
static inline void clocksource_disable_inline_read(void)
{
static_branch_disable(&clocksource_read_inlined);
@@ -321,6 +345,22 @@ static __always_inline u64 tk_clock_read(const struct tk_read_base *tkr)
return clock->read(clock);
}
+static inline u64 tk_clock_read_snapshot(const struct tk_read_base *tkr,
+ struct clocksource_hw_snapshot *chs)
+{
+ struct clocksource *clock = READ_ONCE(tkr->clock);
+ u64 count;
+
+ if (unlikely(clock->read_snapshot)) {
+ *chs = (struct clocksource_hw_snapshot){ };
+ return clock->read_snapshot(clock, chs);
+ }
+
+ count = clock->read(clock);
+ *chs = (struct clocksource_hw_snapshot){ };
+ return count;
+}
+
static inline void clocksource_disable_inline_read(void) { }
static inline void clocksource_enable_inline_read(void) { }
#endif
@@ -1219,18 +1259,6 @@ noinstr time64_t __ktime_get_real_seconds(void)
return READ_ONCE(tk->xtime_sec);
}
-static inline u64 tk_clock_read_snapshot(const struct tk_read_base *tkr,
- struct clocksource_hw_snapshot *chs)
-{
- struct clocksource *clock = READ_ONCE(tkr->clock);
-
- if (unlikely(clock->read_snapshot))
- return clock->read_snapshot(clock, chs);
-
- return clock->read(clock);
-}
-
-
/**
* ktime_get_snapshot_id - Simultaneously snapshot a given clock ID with
* the corresponding monotonic raw and the underlying
@@ -1263,36 +1291,41 @@ void ktime_get_snapshot_id(clockid_t clock_id, struct system_time_snapshot *syst
if (WARN_ON_ONCE(timekeeping_suspended))
return;
- switch (clock_id) {
- case CLOCK_REALTIME:
- tkd = &tk_core;
- offs = &tk_core.timekeeper.offs_real;
- break;
- /* Map RAW to MONOTONIC so the loop below is trivial */
- case CLOCK_MONOTONIC_RAW:
- case CLOCK_MONOTONIC:
- tkd = &tk_core;
- offs = &offs_zero;
- break;
- case CLOCK_BOOTTIME:
- tkd = &tk_core;
- offs = &tk_core.timekeeper.offs_boot;
- break;
- case CLOCK_AUX ... CLOCK_AUX_LAST:
- tkd = aux_get_tk_data(clock_id);
- if (!tkd)
+ /*
+ * Bias the dispatch for CLOCK_REALTIME: it is the overwhelmingly
+ * common caller (PPS, PTP cross-timestamps) and the most
+ * latency-critical -- everything ahead of the counter read
+ * displaces those callers' timestamps from their triggering event.
+ */
+ tkd = &tk_core;
+ offs = &tk_core.timekeeper.offs_real;
+
+ if (unlikely(clock_id != CLOCK_REALTIME)) {
+ switch (clock_id) {
+ /* Map RAW to MONOTONIC so the loop below is trivial */
+ case CLOCK_MONOTONIC_RAW:
+ case CLOCK_MONOTONIC:
+ offs = &offs_zero;
+ break;
+ case CLOCK_BOOTTIME:
+ offs = &tk_core.timekeeper.offs_boot;
+ break;
+ case CLOCK_AUX ... CLOCK_AUX_LAST:
+ tkd = aux_get_tk_data(clock_id);
+ if (!tkd)
+ return;
+ offs = &tkd->timekeeper.offs_aux;
+ break;
+ default:
+ WARN_ON_ONCE(1);
return;
- offs = &tkd->timekeeper.offs_aux;
- break;
- default:
- WARN_ON_ONCE(1);
- return;
+ }
}
tk = &tkd->timekeeper;
do {
- struct clocksource_hw_snapshot chs = { };
+ struct clocksource_hw_snapshot chs;
seq = read_seqcount_begin(&tkd->seq);
@@ -1911,8 +1944,11 @@ int timekeeping_notify(struct clocksource *clock)
* If the clocksource has been selected and supports inlined reads
* enable the branch.
*/
- if (tk->tkr_mono.clock == clock && clock->flags & CLOCK_SOURCE_CAN_INLINE_READ)
+ if (tk->tkr_mono.clock == clock && clock->flags & CLOCK_SOURCE_CAN_INLINE_READ) {
+ /* The inlined read cannot provide a hardware snapshot */
+ WARN_ON_ONCE(clock->read_snapshot);
clocksource_enable_inline_read();
+ }
tick_clock_notify();
return tk->tkr_mono.clock == clock ? 0 : -1;
--
2.43.0
^ permalink raw reply [flat|nested] 6+ messages in thread* Re: [PATCH 0/2] Optimise ktime_get_snapshot_id() for capture latency
2026-10-02 15:19 [PATCH 0/2] Optimise ktime_get_snapshot_id() for capture latency David Woodhouse
2026-10-02 15:19 ` [PATCH 1/2] arm64: Support inlined clocksource reads for the arch counter David Woodhouse
2026-10-02 15:19 ` [PATCH 2/2] timekeeping: Use inlined counter read in ktime_get_snapshot_id() David Woodhouse
@ 2026-10-02 16:41 ` Mark Rutland
2026-10-02 17:11 ` David Woodhouse
2 siblings, 1 reply; 6+ messages in thread
From: Mark Rutland @ 2026-10-02 16:41 UTC (permalink / raw)
To: David Woodhouse
Cc: Thomas Gleixner, John Stultz, Stephen Boyd, Miroslav Lichvar,
Catalin Marinas, Will Deacon, Marc Zyngier, Daniel Lezcano,
Rodolfo Giometti, linux-arm-kernel, linux-kernel
On Fri, Oct 02, 2026 at 04:19:36PM +0100, David Woodhouse wrote:
> Commit 2e27beeb66e4 ("timekeeping: Allow inlining clocksource::read()")
> allows an architecture's primary implementation of clocksource::read()
> to be inlined, apparently because the out-of-line function calls get
> expensive on x86 with certain speculation mitigations. Use the inline
> read in tk_clock_read_snapshot(), and wire it up for arm64 too.
>
> Also bias the clock_id switch() in ktime_get_snapshot_id() for the
> likely() case of CLOCK_REALTIME.
>
> Measured on a Cortex-A53 at 1.35GHz (12.5MHz arch counter), with
> inlined clocksource reads enabled: the interval from a raw counter
> read in the caller to the counter read in ktime_get_snapshot_id()
> drops from 49.9ns to 41.2ns with the inline read, and to 38.8ns with
> the dispatch bias, mean of 1M iterations.
>
> This is unashamedly a microbenchmark but it actually captures a real
> world use case: The time it takes for ktime_get_snapshot_id() to read
> the counter is critical to some users, especially 1PPS signal
> capture. Every nanosecond helps, and for the inline clocksource read,
> the actually *complex* part is already in place; this is just using
> the infrastructure that's already there.
When I mentioned this on IRC, my complaint about complexity wasn't about
the core code, My conern was with the subtle interactions *within* the
timer driver, and between the timer driver and arch code.
I don't think it's fair to say that the core code is the singular
complex part.
Does this show up on any top-level workload, or is the benefit purely
limited to microbenchmarks and timer synchronization?
Mark.
^ permalink raw reply [flat|nested] 6+ messages in thread* Re: [PATCH 0/2] Optimise ktime_get_snapshot_id() for capture latency
2026-10-02 16:41 ` [PATCH 0/2] Optimise ktime_get_snapshot_id() for capture latency Mark Rutland
@ 2026-10-02 17:11 ` David Woodhouse
0 siblings, 0 replies; 6+ messages in thread
From: David Woodhouse @ 2026-10-02 17:11 UTC (permalink / raw)
To: Mark Rutland
Cc: Thomas Gleixner, John Stultz, Stephen Boyd, Miroslav Lichvar,
Catalin Marinas, Will Deacon, Marc Zyngier, Daniel Lezcano,
Rodolfo Giometti, linux-arm-kernel, linux-kernel
[-- Attachment #1: Type: text/plain, Size: 3218 bytes --]
On Fri, 2026-10-02 at 17:41 +0100, Mark Rutland wrote:
> On Fri, Oct 02, 2026 at 04:19:36PM +0100, David Woodhouse wrote:
> > Commit 2e27beeb66e4 ("timekeeping: Allow inlining clocksource::read()")
> > allows an architecture's primary implementation of clocksource::read()
> > to be inlined, apparently because the out-of-line function calls get
> > expensive on x86 with certain speculation mitigations. Use the inline
> > read in tk_clock_read_snapshot(), and wire it up for arm64 too.
> >
> > Also bias the clock_id switch() in ktime_get_snapshot_id() for the
> > likely() case of CLOCK_REALTIME.
> >
> > Measured on a Cortex-A53 at 1.35GHz (12.5MHz arch counter), with
> > inlined clocksource reads enabled: the interval from a raw counter
> > read in the caller to the counter read in ktime_get_snapshot_id()
> > drops from 49.9ns to 41.2ns with the inline read, and to 38.8ns with
> > the dispatch bias, mean of 1M iterations.
> >
> > This is unashamedly a microbenchmark but it actually captures a real
> > world use case: The time it takes for ktime_get_snapshot_id() to read
> > the counter is critical to some users, especially 1PPS signal
> > capture. Every nanosecond helps, and for the inline clocksource read,
> > the actually *complex* part is already in place; this is just using
> > the infrastructure that's already there.
>
> When I mentioned this on IRC, my complaint about complexity wasn't about
> the core code, My conern was with the subtle interactions *within* the
> timer driver, and between the timer driver and arch code.
>
> I don't think it's fair to say that the core code is the singular
> complex part.
This patch doesn't touch the *complex* part of the arch_timer support
with a barge-pole. The inline path is only enabled for the trivial case
where !arch_timer_counter_has_wa() and thus the indirect function
pointer is guaranteed to do *precisely* the same thing.
> Does this show up on any top-level workload, or is the benefit purely
> limited to microbenchmarks and timer synchronization?
For PPS, my latest experiments show that even with a tight loop
*polling* the GPIO directly via MMIO on my test board, the MMIO read
takes ~400ns. So I don't think we can claim that the ~8ns I get from
the inlining is critical to the PPS use case, no.
Rodolfo was concerned about the time taken in ktime_get_snapshot_id();
I responded that I think it's in the noise but there's some low-hanging
fruit for making it faster... I think I've fairly much shown that it's
in the noise *however* hard we try to reduce the rest of the latency.
I think think it's low-hanging fruit though; I guess it comes down to
how we view the complexity.
As also discussed on IRC before you pointed out that it's a long way to
April 1st, the most effective answer for PPS by *far* is to sample the
counter as early as possible in the exception handler, and for the PPS
IRQ handler to pull it out of pt_regs.
https://lore.kernel.org/all/dd7a873caecab6c912e2c0f910d1e46818ea455d.camel@infradead.org/
I did say I'd post the inline timer patch for posterity and let you
shoot it down, and here it is. I'm still not posting the entry.S one :)
[-- Attachment #2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 6179 bytes --]
^ permalink raw reply [flat|nested] 6+ messages in thread