mlx5e:zero-prefix corruption during TCP DMA-BUF RX - help requested

mlx5e:zero-prefix corruption during TCP DMA-BUF RX - help requested

[jiabin deng]

Hello mlx5e and networking maintainers,

We are investigating reproducible data corruption: with SHAMPO enabled,
some TCP data is corrupted when received through mlx5e directly into
CUDA-exported GPU DMA-BUF memory.

The full expected number of bytes is received, but the prefixes of some
payloads become zero. The corrupted ranges end exactly at the next
64-byte boundary in GPU memory.

We have not determined whether this is a driver bug, a configuration
issue, or incorrect API usage in the reproducer. We would appreciate
your guidance in narrowing down the cause.

Receiver environment:

- OS: Ubuntu 26.04 LTS
- Kernel: 7.0.0-30-generic
- /proc/version_signature:
  Ubuntu 7.0.0-30.30-generic 7.0.12
- NIC: NVIDIA BlueField-3 B3210E E-Series DPU
- NIC driver: Ubuntu-packaged mlx5_core
- NIC firmware: 32.43.2400 (MT_0000001117)
- GPU: NVIDIA H20
- NVIDIA GPU driver: 595.71.05
- Receiver uses the CUDA Driver API, built with CUDA 12.4 headers
- Architecture: x86_64, 4 KiB base pages
- Link: 100 GbE, MTU 1500
- Receive configuration: tcp-data-split on, rx-gro-hw on

Tracing confirms that the receive path executes
mlx5e_handle_rx_cqe_mpwrq_shampo().

The sender is a standalone Python TCP program that sends reproducible
nonzero data from host memory. The receiver is a standalone C++ program
using Linux device-memory TCP interfaces and the CUDA Driver API.

Test procedure:

1. Allocate 4 GiB of GPU memory, export it as a DMA-BUF, and bind it
   to RX queues 8-15. Direct all test traffic to RX queue 8.

2. Before each iteration, fill the GPU memory with a nonzero value
   and call cuCtxSynchronize(). Then transmit the test data, in which
   every byte is in the range 1-255.

3. Receive the complete message using MSG_SOCK_DEVMEM and confirm EOF.
   Record the returned DMA-BUF offsets, lengths and tokens.

4. Keep all receive tokens outstanding. First read the data back with
   cuMemcpyDtoH(), then call cuFlushGPUDirectRDMAWrites() with
   CURRENT_CTX and TO_ALL_DEVICES, and read the data back again.
   These two reads produce the snapshots before and after the flush.

5. Compare the two snapshots with each other and the first snapshot
   against the expected data. Return the tokens through
   SO_DEVMEM_DONTNEED only after both snapshots have been saved and
   the comparisons have completed.

Each iteration uses a new TCP connection. Test connections do not run
concurrently.

Observed results:

We tested eight message sizes from 32 KiB to 4 MiB, with 50 iterations
per size:

- 400 iterations completed.
- 354 iterations contained at least one corrupted range.
- 20,707 corrupted ranges were recorded.
- All recorded corrupted ranges were zero-filled prefixes ending
  at the next 64-byte boundary in GPU memory.
- Corruption was present in both snapshots, before and after the
  explicit GPUDirect write flush.

A specific example:

- Packet payload length: 1448 bytes.
- Offset of the first payload byte from the start of the GPU buffer:
  2149169912 bytes.
- This byte offset modulo 64 bytes is 56 bytes.
- First 8 bytes: zero.
- Remaining 1440 bytes: correct.

Related source observations:

The receiver's drivers/net/ethernet/mellanox/mlx5/ directory and headers
under include/linux/mlx5/ are byte-for-byte identical to upstream
v7.0.12. The running kernel is an Ubuntu kernel; we have not yet run
this test on a current vanilla mainline kernel.

One possible lead is an interaction between SHAMPO end padding and
Relaxed Ordering writes. In this version, rq_end_pad_mode() disables
end padding only when LRO is enabled and write RO is supported.
SHAMPO therefore selects ALIGN; at the same time, write RO is enabled
on the payload MKey when the device advertises write RO support.

Commit e2351e517068 ("net/mlx5e: RX, Avoid possible data corruption
when relaxed ordering and LRO combined") describes a related issue
with consecutively placed LRO payloads. We would like to understand
whether SHAMPO might also need to account for a similar situation.
We do not yet have results from padding/RO diagnostic experiments.

Is our receive and synchronization sequence appropriate? Are there
known issues or fixes relevant to this configuration? Please also
advise which additional logs or targeted tests would be most useful
in identifying the cause.

Thank you for your time,
Jiabin Deng

Re: mlx5e:zero-prefix corruption during TCP DMA-BUF RX - help requested

[Dragos Tatulea]

Hi Jiabin,

On 15.09.26 03:49, jiabin deng wrote:
> Hello mlx5e and networking maintainers,
> 
> We are investigating reproducible data corruption: with SHAMPO enabled,
> some TCP data is corrupted when received through mlx5e directly into
> CUDA-exported GPU DMA-BUF memory.
> 
> The full expected number of bytes is received, but the prefixes of some
> payloads become zero. The corrupted ranges end exactly at the next
> 64-byte boundary in GPU memory.
>> [...]
>
> Each iteration uses a new TCP connection. Test connections do not run
> concurrently.
> 
> Observed results:
> 
> We tested eight message sizes from 32 KiB to 4 MiB, with 50 iterations
> per size:
> 
> - 400 iterations completed.
> - 354 iterations contained at least one corrupted range.
> - 20,707 corrupted ranges were recorded.
> - All recorded corrupted ranges were zero-filled prefixes ending
>   at the next 64-byte boundary in GPU memory.
> - Corruption was present in both snapshots, before and after the
>   explicit GPUDirect write flush.
> 
> A specific example:
> 
> - Packet payload length: 1448 bytes.
> - Offset of the first payload byte from the start of the GPU buffer:
>   2149169912 bytes.
> - This byte offset modulo 64 bytes is 56 bytes.
> - First 8 bytes: zero.
> - Remaining 1440 bytes: correct.
> 
This sounds a bit like the ethernet padding issue that was recently
fixed [1]. Did you have this patch when testing?

>
> [...]
>

[1] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit?id=e2466392a0b8496000e12181cb1ee1535eb0da25

Thanks,
Dragos