[PATCH rdma-next v2 0/3] RDMA: Fix DEVX UMEM mailbox size overflow

[Edward Srouji <edwards@nvidia.com>]

Registering a DEVX UMEM builds a CREATE_UMEM mailbox whose size grows
with the number of MTTs, i.e. with the user-supplied length and
pgsz_bitmap. The size is stored in a u32, so a UMEM of about 2TB with
4KB pages wraps it, the mailbox is under-allocated and
mlx5_ib_populate_pas() writes past its end.

Patch 1 fixes a pre-existing issue in the same path, reported by Sashiko
on v1: devx_umem_find_best_pgsize() can return a page size that is not
in the user's pgsz_bitmap.

Patch 2 computes the mailbox size with the saturating size_mul() and
size_add() helpers and rejects UMEMs whose mailbox exceeds INT_MAX, the
largest in_size mlx5_cmd_exec() accepts.

Patch 3 handles the remaining corner: _uverbs_alloc() adds a small
header to the request before calling kvmalloc(), so a mailbox just
under INT_MAX can still push kvmalloc() over its INT_MAX limit and
trigger a WARN. Since these sizes come from userspace, the allocation
now uses __GFP_NOWARN and fails with -ENOMEM.

Mailbox sizes within 512 bytes of INT_MAX also overflow the block count
in mlx5_calc_cmd_blocks(). That predates this series and fails safely
with -ENOMEM before the command reaches firmware, so it is left for a
separate mlx5_core change.

The overflow was found with an AI-assisted static analysis tool and
confirmed by code review.

Patch 2 relies on size_mul() and size_add(), so stable trees older than
5.18 will need a manual backport.

Signed-off-by: Edward Srouji <edwards@nvidia.com>
---
Changes in v2:
- Keep cmd->inlen unsigned and bound the mailbox size explicitly
- Add patch 1 to respect pgsz_bitmap when reducing the page size
  (Sashiko)
- Link to v1: https://lore.kernel.org/r/20261109-devx-umem-mailbox-overflow-v1-0-4a63baab25dc@nvidia.com

---
Edward Srouji (1):
      RDMA/mlx5: Respect pgsz_bitmap when reducing the DEVX UMEM page size

Uri Dar (2):
      RDMA/mlx5: Fix DEVX UMEM mailbox size truncation
      RDMA/core: Don't warn on oversized bundle allocations

 drivers/infiniband/core/ib_core_uverbs.c |  2 +-
 drivers/infiniband/hw/mlx5/devx.c        | 23 ++++++++++++++++-------
 2 files changed, 17 insertions(+), 8 deletions(-)
---
base-commit: e50525a486300e5ff8014c9985b8477b2fca0368
change-id: 20261109-devx-umem-mailbox-overflow-c741bc73deb4

Best regards,
-- 
Edward Srouji <edwards@nvidia.com>