thunderbolt_net: GRO header-length ordering defect on Proxmox 7.0.14

[<brianrobertjames@gmail.com>]

Hello Mika and Yehezkel,

I would appreciate your assessment of a thunderbolt_net GRO ordering defect
observed on a direct USB4 link between two AMD Proxmox hosts. We have
receiver-only original/corrected/original comparisons and packet-order
traces supporting the Ethernet-header/headroom correction already published
by Shun Li.

Credit for that prior published correction belongs to Shun Li:
https://github.com/riverscn/thunderbolt-net-dkms/commit/b18a196bb5ca54246443
e1a29924d6007aa16036 

We independently investigated the same defect and applied the minimal change
to pinned official Proxmox source. We did not install the project's broader
DKMS/RX-normalization modifications.

Tested environment:
- Two AMD x86-64 hosts with one direct USB4 cable; native thunderbolt0
traffic, no bridge, VM or Ceph in the path.
- Receiver kernel 7.0.14-28-pve, #2 SMP PREEMPT_DYNAMIC PMX 7.0.14-28
(2026-11-02T11:54Z). Also reproduced historically on 7.0.2-6-pve.
- MTU 9000; TSO/GSO/GRO enabled; gso_max_size=65536; unpaced single-stream
IPv4 TCP iperf3.
- Stock thunderbolt controller driver unchanged. Receiver comparisons
changed only the network module's header/headroom metadata. Cable, ports,
offloads and CPU/IRQ placement stayed constant within each comparison.
Secure Boot remained enabled with signed test modules.

Receiver-only comparison, original -> corrected -> original, 20 seconds per
arm:
  Original:          30.599221 Gb/s, 620 TCP retransmissions
  GRO corrected:     29.779541 Gb/s,   0 TCP retransmissions
  Original restored: 29.994158 Gb/s, 540 TCP retransmissions

Endpoint interface error/drop increments and receiver FQ drops were zero in
all three arms. The unchanged sender in this historical receiver-isolation
test was still on 7.0.2-6-pve with TX E2E enabled; receiver RX flags stayed
FRAME|E2E (6). Both hosts now remain on 7.0.14-28-pve. This is not a
two-latest-kernel GRO-only comparison.

Minimal reproduction outline: establish direct USB4 IP connectivity with the
above settings, run an iperf3 server bound to the receiver's USB4 address,
and run one 20-second TCP stream from the peer:
  receiver: iperf3 -s -B <receiver-USB4-IP>
  sender: iperf3 -c <receiver-USB4-IP> -B <sender-USB4-IP> -P 1 -t 20 -J
Repeat after changing only the receiver's header/headroom line, then restore
the original receiver module and repeat. Module reload interrupts the data
link; our management connection was separate.

Mechanism and minimal correction in tbnet_probe():
- dev->hard_header_len += sizeof(struct thunderbolt_ip_frame_header);
+ dev->needed_headroom = sizeof(struct thunderbolt_ip_frame_header);

The 12-byte transport header is not part of the Ethernet MAC header
presented to eth_type_trans()/GRO. The original hard_header_len is 26 rather
than ETH_HLEN (14), so gro_list_prepare() can compare 12 IP-header bytes as
link-layer bytes and fail to recognize packets belonging to the same flow.
Runtime tracing confirmed original hard_header_len/needed_headroom=26/0
versus corrected 14/12.

Historical paired receive hooks on 7.0.2 covered the same 18,871,120,863 TCP
payload bytes and 303,590 packets before GRO and at TCP ingress, on the same
CPU. Forward sequence gaps were 0 before GRO and 1,219 at TCP ingress, with
no packet-length parsing mismatches. A corrected capture had equal byte
coverage and zero forward gaps. Retransmitted/overlapping sequence ranges
were treated separately. This localizes the observed ordering defect to the
receive software path; it is not proof of physical USB4 packet loss.

Full reproduction details, source provenance and the one-line patch:
https://bugzilla.proxmox.com/show_bug.cgi?id=8306 

Independent hardware validation shared with the prior fix's author:
https://github.com/riverscn/thunderbolt-net-dkms/issues/27 

Separate context: with both hosts on 7.0.14-28-pve and GRO already
corrected, we also observed TX-credit-dependent fragment-validation errors.
A local negotiated TX E2E opt-in gave 18,604 -> 1 -> 18,601 TCP
retransmissions in off/on/off 20-second arms. This is a separate diagnostic
change, excluded from the GRO patch. Its lower-level cause is unresolved; we
are not proposing globally re-enabling TX E2E given the known controller
compatibility concerns. The related discussion and limitations are here:
https://github.com/hellas-ai/thunderbolt-ibverbs/pull/78#issuecomment-672616
9650 

We have not reproduced these observations on the latest vanilla mainline
kernel, and upstream acceptance of Shun Li's GRO correction has not been
verified. Could you assess whether the GRO correction is appropriate for
upstream/backporting, or point us to an existing fix or discussion? Raw
benchmark and trace evidence is retained locally.

AI assistance disclosure: source review, diagnostic automation, analysis and
preparation of this email were AI-assisted using OpenAI Codex (GPT-6 model
family; a more specific serving-model identifier was not provided).
Measurements came from actual hosts and are not simulated results.

Thank you,
Brian James