[PATCH net 0/2] tcp: correct timestamp echo for accepted old ACKs

[PATCH net 0/2] tcp: correct timestamp echo for accepted old ACKs

[Jeff Jo]

Linux can acknowledge newly received data while echoing an outdated
TCP timestamp. This happens when a reordered packet fills a receive gap
but carries an older acknowledgment for traffic in the other direction.
If the sender uses this echo to measure round-trip time after a long idle
period, the stale timestamp can inflate its estimate and slow its sending.

In this example, S sends the reordered data and R is the Linux receiver
being patched. All packets shown belong to the same TCP connection, with
overlapping requests in both directions. Each illustrated request fits in
one TCP packet; request/response names describe application
messages, while ACK numbers acknowledge TCP bytes. The numbers are
illustrative: TS and echo use S's millisecond clock, and byte numbers are
relative to the first post-idle byte in each direction. ACK=N acknowledges
bytes before N.

  Before idle:
    S -> R: sender request 1, TS=999
    R -> S: response to sender request 1, echo=999
    S -> R: TCP ACK, TS=1000             (R saves timestamp 1000)

                   ... 300 seconds idle ...

  After idle (byte ranges include both ends):
    S -> R: sender request 2, bytes 1-17, ACK=1, TS=301000
            (delayed in the network)
    R -> S: receiver request 1, bytes 1-17, ACK=1
            (initiated by R while sender request 2 is still in flight)
    S -> R: sender request 3, bytes 18-34, ACK=18, TS=301005
            (arrives before sender request 2)
    R -> S: TCP ACK=1, SACK for sender request 3, echo=1000
    S -> R: original sender request 2 arrives, still ACK=1, TS=301000
    R -> S: TCP ACK=35, echo=1000 (bug) or echo=301000 (fixed)

R initiates receiver request 1 while sender request 2 is still in flight;
its ACK=1 means it has not received sender request 2. S receives R's
request before sending sender request 3, so that packet carries ACK=18.
Sender request 3 reaches R first, making sender request 2's ACK=1 old.
The earlier ACK with SACK correctly echoes 1000 while the gap is open.
The bug is retaining 1000 in ACK=35 after the gap closes, instead of
echoing sender request 2's timestamp, 301000.

If S falls back to timestamp-based RTT measurement for ACK=35, subtracting
echo=1000 from its current timestamp (about 301000) produces a roughly
300-second RTT sample, mistakenly counting the idle period. The inflated
smoothed RTT lowers the sender's calculated pacing rate and, when pacing
is enforced, unnecessarily delays outgoing packets and slows the transfer.

Patch 1 refreshes the saved timestamp while preserving the existing
validation and handshake behavior. Patch 2 adds a regression test for
IPv4, IPv6 and IPv4-mapped IPv6. A shell wrapper checks the timestamp
echo because packetdrill does not currently compare that field.

Testing on net dd47bcf279f1:
- ARM64/QEMU, normal and KASAN/UBSAN/lockdep: all 68 focused/control
  cases and three selftest address modes pass with the fix; baseline
  reproduces the stale echoes.
- ARM64 W=1 allyesconfig/allmodconfig builds and Sparse: no new diagnostics
  (patched builds incremental; existing warnings require -Wno-error).

Earlier C-socket repro on net 46bc52d13594: after 300 seconds idle, with
controlled reordering, retransmission and fq pacing, a 1 MiB transfer takes
22.02 seconds without the fix versus 0.38 seconds with it (one run per arm).

Limits: previous broad-suite timing failures and feature gaps remain
unresolved; those suites were not rerun on this base.

AI assistance: Codex generated and revised the fix, reproducers, selftest,
analysis and patch messages. The user directed the investigation, asked
for real-socket and upstream-kernel comparisons, and requested broader
testing. A separate Codex reviewer challenged the code and evidence.
Sparse supplied static analysis.

Jeff Jo (2):
  tcp: refresh TS.Recent for accepted old ACKs
  selftests: net: check timestamp echo after an old ACK

 net/ipv4/tcp_input.c                          |  7 +++
 .../selftests/net/packetdrill/Makefile        |  5 +-
 .../net/packetdrill/tcp_old_ack_ts.pkt        | 23 ++++++++
 .../net/packetdrill/tcp_old_ack_ts.sh         | 57 +++++++++++++++++++
 4 files changed, 91 insertions(+), 1 deletion(-)
 create mode 100644 tools/testing/selftests/net/packetdrill/tcp_old_ack_ts.pkt
 create mode 100755 tools/testing/selftests/net/packetdrill/tcp_old_ack_ts.sh


base-commit: dd47bcf279f1083f09bf5266890b26263361022b
-- 
2.55.0

[PATCH net 1/2] tcp: refresh TS.Recent for accepted old ACKs

[Jeff Jo]

A TCP packet can carry new data while acknowledging traffic in the
opposite direction. With overlapping traffic in both directions, a
delayed packet's acknowledgment can be older than one Linux has already
accepted, even when that packet fills a gap in the received data.

Linux accepts the data, but tcp_ack() takes the old_ack path and skips
updating TS.Recent, the timestamp saved for outgoing acknowledgments.
The reply therefore echoes an older timestamp. If the sender uses this
echo to measure round-trip time after a long idle period, its estimate
includes the idle time and can reduce its sending rate.

Update TS.Recent in old_ack using tcp_replace_ts_recent(), before SACK
processing can trigger a transmission. This reuses the existing timestamp
and sequence checks, including PAWS protection against old duplicate
packets. Leave SYN_RECV unchanged because its caller rejects old ACKs;
established and closing connections accept them.

Echoing the timestamp of the packet that fills the receive gap follows
RFC 7323 section 4.3. In a socket reproduction with 300 seconds idle,
controlled reordering and retransmission to exercise timestamp-based RTT
sampling, the sender's smoothed round-trip time was 37.5 seconds without
the fix and 15.5 ms with it.

Fixes: 12fb3dd9dc3c ("tcp: call tcp_replace_ts_recent() from tcp_ack()")
Assisted-by: LLM sparse
Signed-off-by: Jeff Jo <jeffjo@openai.com>
---
 net/ipv4/tcp_input.c | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/net/ipv4/tcp_input.c b/net/ipv4/tcp_input.c
index 92bc60716f33..f3f4c905ce00 100644
--- a/net/ipv4/tcp_input.c
+++ b/net/ipv4/tcp_input.c
@@ -4465,6 +4465,13 @@ static int tcp_ack(struct sock *sk, const struct sk_buff *skb, int flag)
 	return 1;
 
 old_ack:
+	/* Closing connections also accept old ACKs. SYN_RECV rejects them in
+	 * the caller, so leave its timestamp unchanged. Update before SACK
+	 * processing can trigger a retransmission.
+	 */
+	if (sk->sk_state != TCP_SYN_RECV && (flag & FLAG_UPDATE_TS_RECENT))
+		flag |= tcp_replace_ts_recent(tp, TCP_SKB_CB(skb)->seq);
+
 	/* If data was SACKed, tag it and see if we should send more data.
 	 * If data was DSACKed, see if we can undo a cwnd reduction.
 	 */
-- 
2.55.0

Re: [PATCH net 1/2] tcp: refresh TS.Recent for accepted old ACKs

[Eric Dumazet]

On Tue, Sep 22, 2026 at 12:26 AM Jeff Jo <jeffjo@openai.com> wrote:
>
> A TCP packet can carry new data while acknowledging traffic in the
> opposite direction. With overlapping traffic in both directions, a
> delayed packet's acknowledgment can be older than one Linux has already
> accepted, even when that packet fills a gap in the received data.
>
> Linux accepts the data, but tcp_ack() takes the old_ack path and skips
> updating TS.Recent, the timestamp saved for outgoing acknowledgments.
> The reply therefore echoes an older timestamp. If the sender uses this
> echo to measure round-trip time after a long idle period, its estimate
> includes the idle time and can reduce its sending rate.
>
> Update TS.Recent in old_ack using tcp_replace_ts_recent(), before SACK
> processing can trigger a transmission. This reuses the existing timestamp
> and sequence checks, including PAWS protection against old duplicate
> packets. Leave SYN_RECV unchanged because its caller rejects old ACKs;
> established and closing connections accept them.
>
> Echoing the timestamp of the packet that fills the receive gap follows
> RFC 7323 section 4.3. In a socket reproduction with 300 seconds idle,
> controlled reordering and retransmission to exercise timestamp-based RTT
> sampling, the sender's smoothed round-trip time was 37.5 seconds without
> the fix and 15.5 ms with it.
>
> Fixes: 12fb3dd9dc3c ("tcp: call tcp_replace_ts_recent() from tcp_ack()")
> Assisted-by: LLM sparse
> Signed-off-by: Jeff Jo <jeffjo@openai.com>
> ---
>  net/ipv4/tcp_input.c | 7 +++++++
>  1 file changed, 7 insertions(+)
>
> diff --git a/net/ipv4/tcp_input.c b/net/ipv4/tcp_input.c
> index 92bc60716f33..f3f4c905ce00 100644
> --- a/net/ipv4/tcp_input.c
> +++ b/net/ipv4/tcp_input.c
> @@ -4465,6 +4465,13 @@ static int tcp_ack(struct sock *sk, const struct sk_buff *skb, int flag)
>         return 1;
>
>  old_ack:
> +       /* Closing connections also accept old ACKs. SYN_RECV rejects them in
> +        * the caller, so leave its timestamp unchanged. Update before SACK
> +        * processing can trigger a retransmission.
> +        */
> +       if (sk->sk_state != TCP_SYN_RECV && (flag & FLAG_UPDATE_TS_RECENT))
> +               flag |= tcp_replace_ts_recent(tp, TCP_SKB_CB(skb)->seq);
> +

I think you missed 3d501dd326fb ("tcp: do not accept ACK of bytes we
never sent")

sk->sk_state can not be TCP_SYN_RECV at this point.

Therefore your patch could simply be:

if (flag & FLAG_UPDATE_TS_RECENT)
        tcp_replace_ts_recent(tp, TCP_SKB_CB(skb)->seq);

Thanks.

pw-bot: cr

[PATCH net 2/2] selftests: net: check timestamp echo after an old ACK

[Jeff Jo]

Add a regression test for a gap-filling packet whose acknowledgment has
become old. Linux first sends data to the peer. Deliver two peer data
packets out of order: the later packet acknowledges Linux's data, while
the delayed packet still carries the earlier acknowledgment. Require
the final reply to echo the delayed packet's timestamp.

Use a large jump in peer timestamps to represent the idle interval.
The test checks the echo directly, without waiting for that interval
and without packet loss or retransmission.

Packetdrill does not currently compare the timestamp-echo field (TSecr),
so the packet script alone can pass on a broken kernel. Add a shell
wrapper to check all four outgoing timestamp echoes in packetdrill's
output. Register the wrapper as the runnable test and keep the packet
script as a supporting file.

Run the test in a separate network namespace for each of IPv4, IPv6
and IPv4-mapped IPv6.

Assisted-by: LLM
Signed-off-by: Jeff Jo <jeffjo@openai.com>
---
 .../selftests/net/packetdrill/Makefile        |  5 +-
 .../net/packetdrill/tcp_old_ack_ts.pkt        | 23 ++++++++
 .../net/packetdrill/tcp_old_ack_ts.sh         | 57 +++++++++++++++++++
 3 files changed, 84 insertions(+), 1 deletion(-)
 create mode 100644 tools/testing/selftests/net/packetdrill/tcp_old_ack_ts.pkt
 create mode 100755 tools/testing/selftests/net/packetdrill/tcp_old_ack_ts.sh

diff --git a/tools/testing/selftests/net/packetdrill/Makefile b/tools/testing/selftests/net/packetdrill/Makefile
index ff54641493e9..4695d8258aa1 100644
--- a/tools/testing/selftests/net/packetdrill/Makefile
+++ b/tools/testing/selftests/net/packetdrill/Makefile
@@ -7,6 +7,9 @@ TEST_INCLUDES := \
 	../../kselftest/ktap_helpers.sh \
 # end of TEST_INCLUDES
 
-TEST_PROGS := $(wildcard *.pkt)
+# This fixture requires its wrapper to validate outbound timestamp echoes.
+TEST_PROGS := $(filter-out tcp_old_ack_ts.pkt,$(wildcard *.pkt))
+TEST_PROGS += tcp_old_ack_ts.sh
+TEST_FILES := tcp_old_ack_ts.pkt
 
 include ../../lib.mk
diff --git a/tools/testing/selftests/net/packetdrill/tcp_old_ack_ts.pkt b/tools/testing/selftests/net/packetdrill/tcp_old_ack_ts.pkt
new file mode 100644
index 000000000000..d13f5ea4ac12
--- /dev/null
+++ b/tools/testing/selftests/net/packetdrill/tcp_old_ack_ts.pkt
@@ -0,0 +1,23 @@
+// SPDX-License-Identifier: GPL-2.0
+// Run through tcp_old_ack_ts.sh: packetdrill alone does not check TSecr.
+// An out-of-order segment advances SND.UNA. The gap filler still carries
+// its original, older ACK; accepting its data must refresh TS.Recent.
+--tcp_ts_tick_usecs=1000
+
+`./defaults.sh`
+
+0.000 socket(..., SOCK_STREAM, IPPROTO_TCP) = 3
++0 setsockopt(3, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0
++0 bind(3, ..., ...) = 0
++0 listen(3, 1) = 0
+0.100 < S 0:0(0) win 20000 <mss 1000,sackOK,TS val 900 ecr 0>
++0 > S. 0:0(0) ack 1 <mss 1460,sackOK,TS val 100 ecr 900>
+0.200 < . 1:1(0) ack 1 win 20000 <nop,nop,TS val 1000 ecr 100>
++0 accept(3, ..., ...) = 4
+0.300 write(4, ..., 17) = 17
++0 > P. 1:18(17) ack 1 <nop,nop,TS val 300 ecr 1000>
+0.310 < P. 18:35(17) ack 18 win 20000 <nop,nop,TS val 301001 ecr 300>
++0 > . 18:18(0) ack 1 <nop,nop,TS val 310 ecr 1000,nop,nop,sack 18:35>
+0.320 < P. 1:18(17) ack 1 win 20000 <nop,nop,TS val 301000 ecr 100>
++0 > . 18:18(0) ack 35 <nop,nop,TS val 320 ecr 301000>
++0 read(4, ..., 34) = 34
diff --git a/tools/testing/selftests/net/packetdrill/tcp_old_ack_ts.sh b/tools/testing/selftests/net/packetdrill/tcp_old_ack_ts.sh
new file mode 100755
index 000000000000..4c4534a81e78
--- /dev/null
+++ b/tools/testing/selftests/net/packetdrill/tcp_old_ack_ts.sh
@@ -0,0 +1,57 @@
+#!/bin/bash
+# SPDX-License-Identifier: GPL-2.0
+
+cd "$(dirname "$(realpath "$0")")" || exit 1
+source ../../kselftest/ktap_helpers.sh
+
+if ! command -v packetdrill >/dev/null; then
+	ktap_skip_all "packetdrill not found in PATH"
+	exit "$KSFT_SKIP"
+fi
+
+log=$(mktemp) || exit "$KSFT_FAIL"
+trap 'rm -f "$log"' EXIT
+optargs=()
+if [[ -n "${KSFT_MACHINE_SLOW}" ]]; then
+	optargs+=(--tolerance_usecs=14000)
+fi
+
+ktap_print_header
+ktap_set_plan 3
+
+for family in ipv4 ipv6 ipv4-mapped-ipv6; do
+	mtu=1500
+	[[ "$family" == ipv6 ]] && mtu=1520
+	if ! unshare -n packetdrill --verbose --ip_version="$family" \
+		--mtu="$mtu" "${optargs[@]}" tcp_old_ack_ts.pkt >"$log" 2>&1; then
+		sed 's/^/# /' "$log"
+		ktap_test_fail "$family: packetdrill"
+		continue
+	fi
+
+	# Packetdrill checks packet structure and timing but ignores outbound
+	# TSecr. Require all four observed echoes, including the gap-filling ACK.
+	if awk '
+	/outbound sniffed packet:/ {
+		count++
+		if (!match($0, /ecr [0-9]+/)) {
+			bad = 1
+			next
+		}
+		echoes = echoes (count == 1 ? "" : ",") \
+			substr($0, RSTART + 4, RLENGTH - 4)
+	}
+	END {
+		if (bad || count != 4 || echoes != "900,1000,1000,301000") {
+			print "# unexpected timestamp echoes: " echoes
+			exit 1
+		}
+	}' "$log"; then
+		ktap_test_pass "$family"
+	else
+		sed 's/^/# /' "$log"
+		ktap_test_fail "$family: timestamp echo"
+	fi
+done
+
+ktap_finished
-- 
2.55.0

Re: [PATCH net 2/2] selftests: net: check timestamp echo after an old ACK

[Eric Dumazet]

On Tue, Sep 22, 2026 at 12:26 AM Jeff Jo <jeffjo@openai.com> wrote:
>
> Add a regression test for a gap-filling packet whose acknowledgment has
> become old. Linux first sends data to the peer. Deliver two peer data
> packets out of order: the later packet acknowledges Linux's data, while
> the delayed packet still carries the earlier acknowledgment. Require
> the final reply to echo the delayed packet's timestamp.
>
> Use a large jump in peer timestamps to represent the idle interval.
> The test checks the echo directly, without waiting for that interval
> and without packet loss or retransmission.
>
> Packetdrill does not currently compare the timestamp-echo field (TSecr),
> so the packet script alone can pass on a broken kernel. Add a shell
> wrapper to check all four outgoing timestamp echoes in packetdrill's
> output. Register the wrapper as the runnable test and keep the packet
> script as a supporting file.
>
> Run the test in a separate network namespace for each of IPv4, IPv6
> and IPv4-mapped IPv6.
>
> Assisted-by: LLM
> Signed-off-by: Jeff Jo <jeffjo@openai.com>

Thanks for your patches. But really we must not work around
packetdrill bugs like this.

pw-bot:cr

In the original 2013 packetdrill (commit 77954df72a78),
verify_outbound_live_tcp_options()
checked TS val against tolerance_usecs, then temporarily overwrote
actual_packet's TS val
with script_ts_val and called same_tcp_options(actual_packet,
script_packet) (memcmp),
which did verify TS ecr.

When commit 9a0ade62b7c8 ("net-test: packetdrill: merge Google
packetdrill changes
through April 2018") refactored option verification into a per-option
switch (actual_option->kind),
case TCPOPT_TIMESTAMP: checked actual_ts_val vs script_ts_val and
ended with break;
instead of checking actual_option->data.time_stamp.ecr (or rewriting
actual_ts_val to script_ts_val
and falling through to default:).

Fixing packetdrill in run_packet.c is just:

diff --git a/gtests/net/packetdrill/run_packet.c
b/gtests/net/packetdrill/run_packet.c
index 4aa235d35585..f5fdc8699970 100644
--- a/gtests/net/packetdrill/run_packet.c
+++ b/gtests/net/packetdrill/run_packet.c
@@ -1483,6 +1483,10 @@ static int verify_outbound_tcp_option(
                        asprintf(error, "bad outbound TCP timestamp
value, tolerance %ld", tolerance_usecs);
                        return STATUS_ERR;
                }
+               if (check_field("tcp_ts_ecr",
+                               packet_tcp_ts_ecr(script_packet),
+                               packet_tcp_ts_ecr(actual_packet), error))
+                       return STATUS_ERR;
                break;

        default:

Please work with Neal Cardwell to get packetdrill fixed, then ask
netdev maintainers to
upgrade packetdrill on their test servers.



Note: Even with existing packetdrill binaries in the wild that don't
check TS ecr,
tp->rx_opt.ts_recent is still directly testable in pure .pkt via PAWS
(tcp_paws_discard()),
the same way tcp_ts_recent_invalid_ack.pkt works:


// SPDX-License-Identifier: GPL-2.0
// An out-of-order segment advances SND.UNA. The gap filler still carries
// its original, older ACK; accepting its data must refresh TS.Recent.
--tcp_ts_tick_usecs=1000
`./defaults.sh`
0.000 socket(..., SOCK_STREAM, IPPROTO_TCP) = 3
+0 setsockopt(3, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0
+0 bind(3, ..., ...) = 0
+0 listen(3, 1) = 0
0.100 < S 0:0(0) win 20000 <mss 1000,sackOK,TS val 900 ecr 0>
+0 > S. 0:0(0) ack 1 <mss 1460,sackOK,TS val 100 ecr 900>
0.200 < . 1:1(0) ack 1 win 20000 <nop,nop,TS val 1000 ecr 100>
+0 accept(3, ..., ...) = 4
0.300 write(4, ..., 17) = 17
+0 > P. 1:18(17) ack 1 <nop,nop,TS val 300 ecr 1000>
0.310 < P. 18:35(17) ack 18 win 20000 <nop,nop,TS val 301001 ecr 300>
+0 > . 18:18(0) ack 1 <nop,nop,TS val 310 ecr 1000,nop,nop,sack 18:35>
0.320 < P. 1:18(17) ack 1 win 20000 <nop,nop,TS val 301000 ecr 100>
+0 > . 18:18(0) ack 35 <nop,nop,TS val 320 ecr 301000>
// Verify TS.Recent was updated to 301000: a segment with TSval < 301000
// (but > old TS.Recent 1000) must be rejected by PAWS and trigger dupack 35.
0.330 < P. 35:52(17) ack 18 win 20000 <nop,nop,TS val 300999 ecr 320>
+0 > . 18:18(0) ack 35 <nop,nop,TS val 330 ecr 301000>
+0 read(4, ..., 34) = 34

Re: [PATCH net 2/2] selftests: net: check timestamp echo after an old ACK

[Jeff Jo]

On Tue, Sep 22, 2026 at 02:08:47AM +0200, Eric Dumazet wrote:
> Note: Even with existing packetdrill binaries in the wild that don't
> check TS ecr,
> tp->rx_opt.ts_recent is still directly testable in pure .pkt via PAWS
> (tcp_paws_discard()),
> the same way tcp_ts_recent_invalid_ack.pkt works:

Thanks for the feedback. I'll send a v2 series tomorrow removing the
redundant SYN_RECV check and replacing the test with a plain .pkt test
that uses PAWS for the assertion. I've verified that the revised test
detects the bug with unmodified packetdrill and passes with the
simplified kernel fix.

For the PAWS probe, I used TSval 300998 rather than 300999, since
Linux's one-tick PAWS tolerance allows the latter through.

I'll follow up with a separate packetdrill fix. How does that sound?

Re: [PATCH net 2/2] selftests: net: check timestamp echo after an old ACK

[Eric Dumazet]

On Tue, Sep 22, 2026 at 6:33 AM Jeff Jo <jeffjo@openai.com> wrote:
>
> On Tue, Sep 22, 2026 at 02:08:47AM +0200, Eric Dumazet wrote:
> > Note: Even with existing packetdrill binaries in the wild that don't
> > check TS ecr,
> > tp->rx_opt.ts_recent is still directly testable in pure .pkt via PAWS
> > (tcp_paws_discard()),
> > the same way tcp_ts_recent_invalid_ack.pkt works:
>
> Thanks for the feedback. I'll send a v2 series tomorrow removing the
> redundant SYN_RECV check and replacing the test with a plain .pkt test
> that uses PAWS for the assertion. I've verified that the revised test
> detects the bug with unmodified packetdrill and passes with the
> simplified kernel fix.
>
> For the PAWS probe, I used TSval 300998 rather than 300999, since
> Linux's one-tick PAWS tolerance allows the latter through.
>
> I'll follow up with a separate packetdrill fix. How does that sound?

I saw you sent the packetdrill patch for formal inclusion.

Let's wait for it to be merged and our runners to be updated, as the
kernel bug is minor.
This will allow us to get a nicer packetdrill test that exposes the
issue directly instead of via a workaround.

Thanks!