Re: [PATCH net-next 00/10] net/mlx5e: Add netdev support for data direct

[Dragos Tatulea <dtatulea@nvidia.com>]

On 11.10.26 18:25, Mina Almasry wrote:
> On Sat, Oct 10, 2026 at 7:16 AM Dragos Tatulea <dtatulea@nvidia.com> wrote:
>>
>>
>>
>> On 10.10.26 01:54, Mina Almasry wrote:
>>> On Fri, Oct 9, 2026 at 9:42 AM Mina Almasry <almasrymina@google.com> wrote:
>>>>
>>>> On Thu, Oct 8, 2026 at 6:28 AM Tariq Toukan <tariqt@nvidia.com> wrote:
>>>>>
>>>>> Hi,
>>>>>
>>>>> This patch series by Dragos adds data direct support for netdev devices,
>>>>> allowing a mlx5 netdev to issue DMA commands through multiple paths.
>>>>> This feature is critical for improving performance and reaching line
>>>>> rate in certain environments where issuing PCI transactions over one
>>>>> path may be significantly faster than over another. These differences
>>>>> can arise from various PCI generations in the system or the specific
>>>>> system topology.
>>>>>
>>>>> Data direct for netdev will work with devmem by returning the data
>>>>> direct DMA device instead of the DMA device of the NIC. Each PF netdev
>>>>> registers its own data direct device and support for RX and TX datapath
>>>>> is added. The feature is selected through a new data direct priv flag.
>>>>>
>>>>> The feature is enabled though a ethtool private flag. When this flag
>>>>> is enabled, the driver will return the data direct device in the
>>>>> ndo_queue_get_dma_dev op.
>>>>>
>>>>> During data_direct device unbind, a re-creation of the channels is
>>>>> triggered.
>>>>>
>>>>> A note about the unbind: it does not does not tear down dmabuf bindings
>>>>> attached to the data direct device; the recreated channels will still
>>>>> pick up the binding via rxq->mp_params and post DMA addresses from the
>>>>> data direct IOMMU domain to the PF, causing IOMMU faults. A devmem
>>>>> revoke hook is needed to close those bindings before the switch. This is
>>>>> out of scope for this series.
>>>>>
>>>>
>>>> Elaborate on this please. Are you referring to the fact that on
>>>> page_pool_scrub we unmap the netmems dma-mapped by the page-pool, but
>>>> we don't unmap the netmems dma-mapped by the memory provider?
>>>>
>>>> What's the impact of leaving this unfixed? Is the machine going to
>>>> crash if the device goes away but we don't unmap the dmabuf?
>>>
>> IOMMU faults because the dma device is invalid. Either because the
>> DD dev is still in use and was unbound. Or because the non-DD
>> one is used and is invalid (due to channel reopen).
>>
>>> Oh, you're referring to an even different bug than the one I had in
>>> mind. (I'll see if I can submit a fix for the bug I had in mind).
>>>
>>> But in general I think not addressing the issue you're referring to is
>>> way too messy. I think we do indeed need to unbind the dma-buf if the
>>> dma-dev is going away. The invariant should be that the
>>> binding->attachment->dev should not change at all during the entire
>>> lifetime of the binding. We already 'revoke' the dma-buf binding if
>>> the netdev is going away, sorta (there is a bug there I need to fix.
>>> The uninstall function in the provider doesn't actually unmap the
>>> dma-buf :sad face:).
>>>
>>> I've worked with my LLM to suggest some changes that (we think) make
>>> this work, but you may disagree. I'll post the suggestions. But please
>>> I think this should be handled one way or another.
>>>
>> My idea was to handle the unbind part in a subsequent series because
>> it probably needs some more back and forth to get this cleanup path
>> right.
>>
>> If not acceptable I'll work on adding it to this series.
>>
> 
> Since these are mostly driver-local changes I don't want to be so
> forceful to block it by saying it's unacceptable, but if at all
> possible lets try not to exacerbate the current mess. It looks like we
> already a bunch of lifetime bugs with regards to the device going
> away. I've posted this series, which - I think - gives you the devmem
> 'revoke' mechanism that you want:
> 
> https://lore.kernel.org/netdev/20261010025532.839559-1-almasrymina@google.com/
> 
> Also working on that, I realized we don't use binding->vdev and
> binding->dev correctly on binding rx. binding->vdev is meant to be the
> device we do the bind to, and binding->dev is meant to be the
> underlying dma_device (if they're different). That fits into your
> model where you have the netdev you want to bind to, but have an
> underlying device that you want to dma-map to.
> 
> I've gotten some feedback on my series and I'm working on v2. If it
> makes sense for you to wait on my devmem core changes so you can
> integrate into them better and avoid these lifetime bugs, lets do
> that.
> 

Thanks for your patches Mina! This made it easier to implement the
unbind. I will add a dependency in the series to your patches. That's
if they don't get merged before.

Thanks,
Dragos