mlx5e:zero-prefix corruption during TCP DMA-BUF RX - help requested

mlx5e:zero-prefix corruption during TCP DMA-BUF RX - help requested

[jiabin deng]

Hello mlx5e and networking maintainers,

We are investigating reproducible data corruption: with SHAMPO enabled,
some TCP data is corrupted when received through mlx5e directly into
CUDA-exported GPU DMA-BUF memory.

The full expected number of bytes is received, but the prefixes of some
payloads become zero. The corrupted ranges end exactly at the next
64-byte boundary in GPU memory.

We have not determined whether this is a driver bug, a configuration
issue, or incorrect API usage in the reproducer. We would appreciate
your guidance in narrowing down the cause.

Receiver environment:

- OS: Ubuntu 26.04 LTS
- Kernel: 7.0.0-30-generic
- /proc/version_signature:
  Ubuntu 7.0.0-30.30-generic 7.0.12
- NIC: NVIDIA BlueField-3 B3210E E-Series DPU
- NIC driver: Ubuntu-packaged mlx5_core
- NIC firmware: 32.43.2400 (MT_0000001117)
- GPU: NVIDIA H20
- NVIDIA GPU driver: 595.71.05
- Receiver uses the CUDA Driver API, built with CUDA 12.4 headers
- Architecture: x86_64, 4 KiB base pages
- Link: 100 GbE, MTU 1500
- Receive configuration: tcp-data-split on, rx-gro-hw on

Tracing confirms that the receive path executes
mlx5e_handle_rx_cqe_mpwrq_shampo().

The sender is a standalone Python TCP program that sends reproducible
nonzero data from host memory. The receiver is a standalone C++ program
using Linux device-memory TCP interfaces and the CUDA Driver API.

Test procedure:

1. Allocate 4 GiB of GPU memory, export it as a DMA-BUF, and bind it
   to RX queues 8-15. Direct all test traffic to RX queue 8.

2. Before each iteration, fill the GPU memory with a nonzero value
   and call cuCtxSynchronize(). Then transmit the test data, in which
   every byte is in the range 1-255.

3. Receive the complete message using MSG_SOCK_DEVMEM and confirm EOF.
   Record the returned DMA-BUF offsets, lengths and tokens.

4. Keep all receive tokens outstanding. First read the data back with
   cuMemcpyDtoH(), then call cuFlushGPUDirectRDMAWrites() with
   CURRENT_CTX and TO_ALL_DEVICES, and read the data back again.
   These two reads produce the snapshots before and after the flush.

5. Compare the two snapshots with each other and the first snapshot
   against the expected data. Return the tokens through
   SO_DEVMEM_DONTNEED only after both snapshots have been saved and
   the comparisons have completed.

Each iteration uses a new TCP connection. Test connections do not run
concurrently.

Observed results:

We tested eight message sizes from 32 KiB to 4 MiB, with 50 iterations
per size:

- 400 iterations completed.
- 354 iterations contained at least one corrupted range.
- 20,707 corrupted ranges were recorded.
- All recorded corrupted ranges were zero-filled prefixes ending
  at the next 64-byte boundary in GPU memory.
- Corruption was present in both snapshots, before and after the
  explicit GPUDirect write flush.

A specific example:

- Packet payload length: 1448 bytes.
- Offset of the first payload byte from the start of the GPU buffer:
  2149169912 bytes.
- This byte offset modulo 64 bytes is 56 bytes.
- First 8 bytes: zero.
- Remaining 1440 bytes: correct.

Related source observations:

The receiver's drivers/net/ethernet/mellanox/mlx5/ directory and headers
under include/linux/mlx5/ are byte-for-byte identical to upstream
v7.0.12. The running kernel is an Ubuntu kernel; we have not yet run
this test on a current vanilla mainline kernel.

One possible lead is an interaction between SHAMPO end padding and
Relaxed Ordering writes. In this version, rq_end_pad_mode() disables
end padding only when LRO is enabled and write RO is supported.
SHAMPO therefore selects ALIGN; at the same time, write RO is enabled
on the payload MKey when the device advertises write RO support.

Commit e2351e517068 ("net/mlx5e: RX, Avoid possible data corruption
when relaxed ordering and LRO combined") describes a related issue
with consecutively placed LRO payloads. We would like to understand
whether SHAMPO might also need to account for a similar situation.
We do not yet have results from padding/RO diagnostic experiments.

Is our receive and synchronization sequence appropriate? Are there
known issues or fixes relevant to this configuration? Please also
advise which additional logs or targeted tests would be most useful
in identifying the cause.

Thank you for your time,
Jiabin Deng

Re: mlx5e:zero-prefix corruption during TCP DMA-BUF RX - help requested

[Dragos Tatulea]

Hi Jiabin,

On 15.09.26 03:49, jiabin deng wrote:
> Hello mlx5e and networking maintainers,
> 
> We are investigating reproducible data corruption: with SHAMPO enabled,
> some TCP data is corrupted when received through mlx5e directly into
> CUDA-exported GPU DMA-BUF memory.
> 
> The full expected number of bytes is received, but the prefixes of some
> payloads become zero. The corrupted ranges end exactly at the next
> 64-byte boundary in GPU memory.
>> [...]
>
> Each iteration uses a new TCP connection. Test connections do not run
> concurrently.
> 
> Observed results:
> 
> We tested eight message sizes from 32 KiB to 4 MiB, with 50 iterations
> per size:
> 
> - 400 iterations completed.
> - 354 iterations contained at least one corrupted range.
> - 20,707 corrupted ranges were recorded.
> - All recorded corrupted ranges were zero-filled prefixes ending
>   at the next 64-byte boundary in GPU memory.
> - Corruption was present in both snapshots, before and after the
>   explicit GPUDirect write flush.
> 
> A specific example:
> 
> - Packet payload length: 1448 bytes.
> - Offset of the first payload byte from the start of the GPU buffer:
>   2149169912 bytes.
> - This byte offset modulo 64 bytes is 56 bytes.
> - First 8 bytes: zero.
> - Remaining 1440 bytes: correct.
> 
This sounds a bit like the ethernet padding issue that was recently
fixed [1]. Did you have this patch when testing?

>
> [...]
>

[1] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit?id=e2466392a0b8496000e12181cb1ee1535eb0da25

Thanks,
Dragos

Re: mlx5e:zero-prefix corruption during TCP DMA-BUF RX - help requested

[jiabin deng]

Hi Dragos,

Thank you for pointing us to this fix.

Our original test used Ubuntu kernel 7.0.0-30-generic, based on
v7.0.12, and did not include that change. We have now repeated the
test using Ubuntu's mainline build of Linux 7.2.6
(7.2.6-070206-generic), with its in-tree mlx5_core driver.

I checked the v7.2.6 source: it contains the small-frame guard from
commit e2466392a0b8496000e12181cb1ee1535eb0da25  In
mlx5e_handle_rx_cqe_mpwrq_shampo(), frames with
cqe_bcnt <= ETH_ZLEN + 2 * VLAN_HLEN set match = false and
flush = true [2].

The zero-filled corruption ending at the next 64-byte boundary in
GPU memory still reproduces with this kernel.

We kept the same sender and receiver data-checking logic, with
tcp-data-split on and rx-gro-hw on. One additional environment
change is that the NVIDIA GPU driver was updated from 595.71.05 to
595.91.07. The NIC firmware remains 32.43.2400 (MT_0000001117).

We repeated the same eight message sizes, from 32 KiB to 4 MiB,
with 50 iterations per size:

- All 400 iterations completed.
- 105 iterations passed; 295 contained corrupted data.
- 21,599 corrupted ranges were recorded, all zero-filled and
  ending at the next 64-byte boundary in GPU memory.
- For each size from 256 KiB through 4 MiB, all 50 iterations
  contained corruption.
- The full-message snapshots taken before and after
  cuFlushGPUDirectRDMAWrites() were identical in all iterations.

For example, in the first 32 KiB iteration, an 8-byte corrupted
range starts at offset 2147495416 bytes from the beginning of the
GPU buffer. This offset is 56 bytes into a 64-byte-aligned block,
so the eight zero bytes end exactly at the next 64-byte boundary.

These results come from the reproducer's runtime byte comparisons.

Could you suggest the next targeted checks or additional debug
information that would help narrow this down?

Thanks,
Jiabin Deng


On Tue, Sep 15, 2026 at 8:21 PM Dragos Tatulea <dtatulea@nvidia.com> wrote:
>
> Hi Jiabin,
>
> On 15.09.26 03:49, jiabin deng wrote:
> > Hello mlx5e and networking maintainers,
> >
> > We are investigating reproducible data corruption: with SHAMPO enabled,
> > some TCP data is corrupted when received through mlx5e directly into
> > CUDA-exported GPU DMA-BUF memory.
> >
> > The full expected number of bytes is received, but the prefixes of some
> > payloads become zero. The corrupted ranges end exactly at the next
> > 64-byte boundary in GPU memory.
> >> [...]
> >
> > Each iteration uses a new TCP connection. Test connections do not run
> > concurrently.
> >
> > Observed results:
> >
> > We tested eight message sizes from 32 KiB to 4 MiB, with 50 iterations
> > per size:
> >
> > - 400 iterations completed.
> > - 354 iterations contained at least one corrupted range.
> > - 20,707 corrupted ranges were recorded.
> > - All recorded corrupted ranges were zero-filled prefixes ending
> >   at the next 64-byte boundary in GPU memory.
> > - Corruption was present in both snapshots, before and after the
> >   explicit GPUDirect write flush.
> >
> > A specific example:
> >
> > - Packet payload length: 1448 bytes.
> > - Offset of the first payload byte from the start of the GPU buffer:
> >   2149169912 bytes.
> > - This byte offset modulo 64 bytes is 56 bytes.
> > - First 8 bytes: zero.
> > - Remaining 1440 bytes: correct.
> >
> This sounds a bit like the ethernet padding issue that was recently
> fixed [1]. Did you have this patch when testing?
>
> >
> > [...]
> >
>
> [1] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit?id=e2466392a0b8496000e12181cb1ee1535eb0da25
>
> Thanks,
> Dragos

Re: mlx5e:zero-prefix corruption during TCP DMA-BUF RX - help requested

[Dragos Tatulea]

On 18.09.26 08:06, jiabin deng wrote:
> Hi Dragos,
> 
> Thank you for pointing us to this fix.
> 
> Our original test used Ubuntu kernel 7.0.0-30-generic, based on
> v7.0.12, and did not include that change. We have now repeated the
> test using Ubuntu's mainline build of Linux 7.2.6
> (7.2.6-070206-generic), with its in-tree mlx5_core driver.
> 
> I checked the v7.2.6 source: it contains the small-frame guard from
> commit e2466392a0b8496000e12181cb1ee1535eb0da25  In
> mlx5e_handle_rx_cqe_mpwrq_shampo(), frames with
> cqe_bcnt <= ETH_ZLEN + 2 * VLAN_HLEN set match = false and
> flush = true [2].
> 
> The zero-filled corruption ending at the next 64-byte boundary in
> GPU memory still reproduces with this kernel.
> 
> We kept the same sender and receiver data-checking logic, with
> tcp-data-split on and rx-gro-hw on. One additional environment
> change is that the NVIDIA GPU driver was updated from 595.71.05 to
> 595.91.07. The NIC firmware remains 32.43.2400 (MT_0000001117).
> 
> We repeated the same eight message sizes, from 32 KiB to 4 MiB,
> with 50 iterations per size:
> 
> - All 400 iterations completed.
> - 105 iterations passed; 295 contained corrupted data.
> - 21,599 corrupted ranges were recorded, all zero-filled and
>   ending at the next 64-byte boundary in GPU memory.
> - For each size from 256 KiB through 4 MiB, all 50 iterations
>   contained corruption.
> - The full-message snapshots taken before and after
>   cuFlushGPUDirectRDMAWrites() were identical in all iterations.
> 
> For example, in the first 32 KiB iteration, an 8-byte corrupted
> range starts at offset 2147495416 bytes from the beginning of the
> GPU buffer. This offset is 56 bytes into a 64-byte-aligned block,
> so the eight zero bytes end exactly at the next 64-byte boundary.
> 
> These results come from the reproducer's runtime byte comparisons.
> 
> Could you suggest the next targeted checks or additional debug
> information that would help narrow this down?
> 
Did you also try CPU buffers? From the NIC perspective it shouldn't
be any different.

Could you share a program + reproduction steps?

Thanks,
Dragos