[PATCH rdma-next 0/2] RDMA: Fix DEVX UMEM mailbox size overflow

[Edward Srouji <edwards@nvidia.com>]

Registering a DEVX UMEM builds a CREATE_UMEM mailbox whose size grows
with the number of MTTs, i.e. with the user-supplied length and
pgsz_bitmap. The size is stored in a u32, so a UMEM of about 2TB with
4KB pages wraps it, the mailbox is under-allocated and
mlx5_ib_populate_pas() writes past its end.

Patch 1 computes the size with overflow checks into an int, the in_size
type of mlx5_cmd_exec(), and rejects UMEMs whose mailbox does not fit.

Patch 2 handles the remaining corner: _uverbs_alloc() adds a small
header to the request before calling kvmalloc(), so a mailbox just
under INT_MAX can still push kvmalloc() over its INT_MAX limit and
trigger a WARN. Since these sizes come from userspace, the allocation
now uses __GFP_NOWARN and fails with -ENOMEM.

Sizes just below INT_MAX also overflow the block count in
mlx5_calc_cmd_blocks(). That predates this series and fails safely
with -ENOMEM before the command reaches firmware, so it is left for a
separate mlx5_core change.

The overflow was found with an AI-assisted static analysis tool and
confirmed by code review.

Patch 1 relies on size_mul() and mixed-type check_add_overflow(), so
stable trees older than 6.1 will need a manual backport.

Signed-off-by: Edward Srouji <edwards@nvidia.com>
---
Uri Dar (2):
      RDMA/mlx5: Fix DEVX UMEM mailbox size truncation
      RDMA/core: Don't warn on oversized bundle allocations

 drivers/infiniband/core/ib_core_uverbs.c |  2 +-
 drivers/infiniband/hw/mlx5/devx.c        | 15 +++++++++------
 2 files changed, 10 insertions(+), 7 deletions(-)
---
base-commit: e50525a486300e5ff8014c9985b8477b2fca0368
change-id: 20261109-devx-umem-mailbox-overflow-c741bc73deb4

Best regards,
-- 
Edward Srouji <edwards@nvidia.com>