Re: [PATCH rdma-next v3 00/11] RDMA/core: Introduce FRMR pools infrastructure

[Edward Srouji <edwards@nvidia.com>]

On 2/25/2026 1:47 PM, Leon Romanovsky wrote:
> External email: Use caution opening links or attachments
> 
> 
> On Mon, Feb 02, 2026 at 05:59:52PM +0200, Edward Srouji wrote:
>>  From Michael:
>>
>> This patch series introduces a new FRMR (Fast Registration Memory Region)
>> pool infrastructure for the RDMA core subsystem. The goal is to provide
>> efficient management and allow reuse of MRs (Memory Regions) for RDMA
>> device drivers.
>>
>> Background
>> ==========
>>
>> Memory registration and deregistration can be a significant bottleneck in
>> RDMA applications that need to register memory regions dynamically in
>> their data path or must re-register memory on application restart.
>> Repeatedly allocating and freeing these resources introduces overhead,
>> particularly in high-throughput or latency-sensitive environments where
>> memory regions are frequently cycled. Notably, the mlx5_ib driver has
>> already adopted memory registration reuse mechanisms and has demonstrated
>> notable performance improvements as a result.
>>
>> FRMR pools will store handles of the reusable objects, giving drivers
>> the flexibility to choose what to store (e.g: pointers or indexes).
>> Device driver integration requires the ability to modify the hardware
>> objects underlying MRs when reusing FRMR handles, allowing the update
>> of pre-allocated handles to fit the parameters of requested MR
>> registrations. The FRMR pools manage memory region handles with respect
>> to attributes that cannot be changed after allocation such as access flags,
>> ATS capabilities, vendor keys, and DMA block size so each pool is uniquely
>> characterized by these non-modifiable attributes.
>> This ensures compatibility and correctness while allowing drivers
>> flexibility in managing other aspects of the MR lifecycle.
>>
>> Solution Overview
>> =================
>>
>> This patch series introduces a centralized, per-device FRMR pooling
>> infrastructure that provides:
>>
>> 1. Pool Organization: Uses an RB-tree to organize pools by FRMR
>>     characteristics (ATS support, access flags, vendor-specific keys,
>>     and DMA block count). This allows efficient lookup and reuse of
>>     compatible FRMR handles.
>>
>> 2. Dynamic Allocation: Pools grow dynamically on demand when no cached
>>     handles are available, ensuring optimal memory usage without
>>     sacrificing performance.
>>
>> 3. Aging Mechanism: Implements an aging system. Unused handles are
>>     gradually moved to the freed after a configurable aging period
>>     (default: 60 seconds), preventing memory bloat during idle periods.
>>
>> 4. Pinned Handles: Supports pinning a minimum number of handles per
>>     pool to maintain performance for latency-sensitive workloads, avoiding
>>     allocation overhead on critical paths.
>>
>> 5. Driver Flexibility: Provides a callback-based interface
>>     (ib_frmr_pool_ops) that allows drivers to implement their own FRMR
>>     creation/destruction logic while leveraging the common pooling
>>     infrastructure.
>>
>> API
>> ===
>>
>> The infrastructure exposes the following APIs:
>>
>> - ib_frmr_pools_init(): Initialize FRMR pools for a device
>> - ib_frmr_pools_cleanup(): Clean up all pools for a device
>> - ib_frmr_pool_pop(): Get an FRMR handle from the pool
>> - ib_frmr_pool_push(): Return an FRMR handle to the pool
>> - ib_frmr_pools_set_aging_period(): Configure aging period
>> - ib_frmr_pools_set_pinned(): Set minimum pinned handles per pool
>>
>> mlx5_ib
>> =======
>>
>> The partial control and visability we had only over the 'persistent'
>> cache entries through debugfs is replaced by the netlink FRMR API that
>> allows showing and setting properties of all available pools.
>> This series also changes the default behavior MR cache had for PFs
>> (Physical Functions) by dropping the pre-allocation of MKEYs that was
>> costing 100MB of memory per PF and slowing down the loading and
>> unloading of the driver.
>>
>> Signed-off-by: Michael Guralnik <michaelgur@nvidia.com>
>> Signed-off-by: Edward Srouji <edwards@nvidia.com>
>> ---
>> Changes in v3:
>> - Use rbtree helpers for pool find and find_add operations.
>> - Use cmp_int() for pool key comparison.
>> - Make key comparison inline.
>> - Link to v2: https://lore.kernel.org/r/20251222-frmr_pools-v2-0-f06a99caa538@nvidia.com
>>
>> Changes in v2:
>> - Fix stack size warning in netlink set_pinned flow.
>> - Add commit to move async command context init and cleanup out of MR
>>    cache logic.
>> - Add enforcement of access flags in set_pinned flow and enforce used
>>    bits in vendor specific fields to ensure old kernels fail if any
>>    unknown parameter is passed.
>> - Add an option to expose kernel-internal pools through netlink.
>> - Link to v1: https://lore.kernel.org/r/20251116-frmr_pools-v1-0-5eb3c8f5c9c4@nvidia.com
>>
>> ---
>> Chiara Meiohas (1):
>>        RDMA/mlx5: Move device async_ctx initialization
>>
>> Michael Guralnik (10):
>>        IB/core: Introduce FRMR pools
>>        RDMA/core: Add aging to FRMR pools
>>        RDMA/core: Add FRMR pools statistics
>>        RDMA/core: Add pinned handles to FRMR pools
>>        RDMA/mlx5: Switch from MR cache to FRMR pools
>>        net/mlx5: Drop MR cache related code
>>        RDMA/nldev: Add command to get FRMR pools
>>        RDMA/core: Add netlink command to modify FRMR aging
>>        RDMA/nldev: Add command to set pinned FRMR handles
>>        RDMA/nldev: Expose kernel-internal FRMR pools in netlink
> 
> 
> There is a need to rebase this series, it doesn't apply.

NP, I'll send a rebased v4
Thanks.

> 
> Thanks