[PATCH rdma-next 0/2] RDMA: Fix DEVX UMEM mailbox size overflow

[PATCH rdma-next 0/2] RDMA: Fix DEVX UMEM mailbox size overflow

[Edward Srouji]

Registering a DEVX UMEM builds a CREATE_UMEM mailbox whose size grows
with the number of MTTs, i.e. with the user-supplied length and
pgsz_bitmap. The size is stored in a u32, so a UMEM of about 2TB with
4KB pages wraps it, the mailbox is under-allocated and
mlx5_ib_populate_pas() writes past its end.

Patch 1 computes the size with overflow checks into an int, the in_size
type of mlx5_cmd_exec(), and rejects UMEMs whose mailbox does not fit.

Patch 2 handles the remaining corner: _uverbs_alloc() adds a small
header to the request before calling kvmalloc(), so a mailbox just
under INT_MAX can still push kvmalloc() over its INT_MAX limit and
trigger a WARN. Since these sizes come from userspace, the allocation
now uses __GFP_NOWARN and fails with -ENOMEM.

Sizes just below INT_MAX also overflow the block count in
mlx5_calc_cmd_blocks(). That predates this series and fails safely
with -ENOMEM before the command reaches firmware, so it is left for a
separate mlx5_core change.

The overflow was found with an AI-assisted static analysis tool and
confirmed by code review.

Patch 1 relies on size_mul() and mixed-type check_add_overflow(), so
stable trees older than 6.1 will need a manual backport.

Signed-off-by: Edward Srouji <edwards@nvidia.com>
---
Uri Dar (2):
      RDMA/mlx5: Fix DEVX UMEM mailbox size truncation
      RDMA/core: Don't warn on oversized bundle allocations

 drivers/infiniband/core/ib_core_uverbs.c |  2 +-
 drivers/infiniband/hw/mlx5/devx.c        | 15 +++++++++------
 2 files changed, 10 insertions(+), 7 deletions(-)
---
base-commit: e50525a486300e5ff8014c9985b8477b2fca0368
change-id: 20261109-devx-umem-mailbox-overflow-c741bc73deb4

Best regards,
-- 
Edward Srouji <edwards@nvidia.com>

[PATCH rdma-next 1/2] RDMA/mlx5: Fix DEVX UMEM mailbox size truncation

[Edward Srouji]

From: Uri Dar <udar@nvidia.com>

devx_umem_reg_cmd_alloc() stores the CREATE_UMEM mailbox size, a header
plus 8 bytes per MTT, in the u32 cmd->inlen. The number of MTTs is
derived from the user-supplied length and pgsz_bitmap, so a UMEM of about
2TB registered with 4KB pages wraps the size, the mailbox is
under-allocated and mlx5_ib_populate_pas() writes past its end.

Store the size in an int, the in_size type of mlx5_cmd_exec(), where a
larger value would turn negative, and compute it with overflow checks so
a UMEM whose mailbox does not fit is rejected.

Mailbox sizes within 512 bytes of INT_MAX still overflow the block count
in mlx5_calc_cmd_blocks(), but the command then fails with -ENOMEM
before reaching firmware. This predates the change.

Fixes: aeae94579caf ("IB/mlx5: Add DEVX support for memory registration")
Cc: stable@vger.kernel.org
Signed-off-by: Uri Dar <udar@nvidia.com>
Reviewed-by: Michael Gur <michaelgur@nvidia.com>
Signed-off-by: Edward Srouji <edwards@nvidia.com>
---
 drivers/infiniband/hw/mlx5/devx.c | 15 +++++++++------
 1 file changed, 9 insertions(+), 6 deletions(-)

diff --git a/drivers/infiniband/hw/mlx5/devx.c b/drivers/infiniband/hw/mlx5/devx.c
index fc6f793dcc6584e6a22b40c57e2df8f24579eb14..1da6c0e440d6e19d8791b012a148f479cfdddad7 100644
--- a/drivers/infiniband/hw/mlx5/devx.c
+++ b/drivers/infiniband/hw/mlx5/devx.c
@@ -113,7 +113,7 @@ struct devx_umem {
 
 struct devx_umem_reg_cmd {
 	void				*in;
-	u32				inlen;
+	int				inlen;
 	u32				out[MLX5_ST_SZ_DW(create_umem_out)];
 };
 
@@ -2334,6 +2334,7 @@ static int devx_umem_reg_cmd_alloc(struct mlx5_ib_dev *dev,
 {
 	unsigned long pgsz_bitmap;
 	unsigned int page_size;
+	size_t num_mtt;
 	__be64 *mtt;
 	void *umem;
 	int ret;
@@ -2361,9 +2362,12 @@ static int devx_umem_reg_cmd_alloc(struct mlx5_ib_dev *dev,
 	if (!page_size)
 		return -EINVAL;
 
-	cmd->inlen = MLX5_ST_SZ_BYTES(create_umem_in) +
-		     (MLX5_ST_SZ_BYTES(mtt) *
-		      ib_umem_num_dma_blocks(obj->umem, page_size));
+	num_mtt = ib_umem_num_dma_blocks(obj->umem, page_size);
+	if (check_add_overflow(MLX5_ST_SZ_BYTES(create_umem_in),
+			       size_mul(MLX5_ST_SZ_BYTES(mtt), num_mtt),
+			       &cmd->inlen))
+		return -EINVAL;
+
 	cmd->in = uverbs_zalloc(attrs, cmd->inlen);
 	if (IS_ERR(cmd->in))
 		return PTR_ERR(cmd->in);
@@ -2372,8 +2376,7 @@ static int devx_umem_reg_cmd_alloc(struct mlx5_ib_dev *dev,
 	mtt = (__be64 *)MLX5_ADDR_OF(umem, umem, mtt);
 
 	MLX5_SET(create_umem_in, cmd->in, opcode, MLX5_CMD_OP_CREATE_UMEM);
-	MLX5_SET64(umem, umem, num_of_mtt,
-		   ib_umem_num_dma_blocks(obj->umem, page_size));
+	MLX5_SET64(umem, umem, num_of_mtt, num_mtt);
 	MLX5_SET(umem, umem, log_page_size,
 		 order_base_2(page_size) - MLX5_ADAPTER_PAGE_SHIFT);
 	MLX5_SET(umem, umem, page_offset,

-- 
2.49.0

Re: [PATCH rdma-next 1/2] RDMA/mlx5: Fix DEVX UMEM mailbox size truncation

[Jason Gunthorpe]

On Mon, Nov 09, 2026 at 04:42:32PM +0200, Edward Srouji wrote:
> From: Uri Dar <udar@nvidia.com>
> 
> devx_umem_reg_cmd_alloc() stores the CREATE_UMEM mailbox size, a header
> plus 8 bytes per MTT, in the u32 cmd->inlen. The number of MTTs is
> derived from the user-supplied length and pgsz_bitmap, so a UMEM of about
> 2TB registered with 4KB pages wraps the size, the mailbox is
> under-allocated and mlx5_ib_populate_pas() writes past its end.
> 
> Store the size in an int, the in_size type of mlx5_cmd_exec(), where a
> larger value would turn negative, and compute it with overflow checks so
> a UMEM whose mailbox does not fit is rejected.
> 
> Mailbox sizes within 512 bytes of INT_MAX still overflow the block count
> in mlx5_calc_cmd_blocks(), but the command then fails with -ENOMEM
> before reaching firmware. This predates the change.
> 
> Fixes: aeae94579caf ("IB/mlx5: Add DEVX support for memory registration")
> Cc: stable@vger.kernel.org
> Signed-off-by: Uri Dar <udar@nvidia.com>
> Reviewed-by: Michael Gur <michaelgur@nvidia.com>
> Signed-off-by: Edward Srouji <edwards@nvidia.com>
> ---
>  drivers/infiniband/hw/mlx5/devx.c | 15 +++++++++------
>  1 file changed, 9 insertions(+), 6 deletions(-)
> 
> diff --git a/drivers/infiniband/hw/mlx5/devx.c b/drivers/infiniband/hw/mlx5/devx.c
> index fc6f793dcc6584e6a22b40c57e2df8f24579eb14..1da6c0e440d6e19d8791b012a148f479cfdddad7 100644
> --- a/drivers/infiniband/hw/mlx5/devx.c
> +++ b/drivers/infiniband/hw/mlx5/devx.c
> @@ -113,7 +113,7 @@ struct devx_umem {
>  
>  struct devx_umem_reg_cmd {
>  	void				*in;
> -	u32				inlen;
> +	int				inlen;

Do not change things which must be unsigned into signed values.

Jason

Re: [PATCH rdma-next 1/2] RDMA/mlx5: Fix DEVX UMEM mailbox size truncation

[Edward Srouji]

On 11/9/2026 4:48 PM, Jason Gunthorpe wrote:
> External email: Use caution opening links or attachments
> 
> 
> On Mon, Nov 09, 2026 at 04:42:32PM +0200, Edward Srouji wrote:
>> From: Uri Dar <udar@nvidia.com>
>>
>> devx_umem_reg_cmd_alloc() stores the CREATE_UMEM mailbox size, a header
>> plus 8 bytes per MTT, in the u32 cmd->inlen. The number of MTTs is
>> derived from the user-supplied length and pgsz_bitmap, so a UMEM of about
>> 2TB registered with 4KB pages wraps the size, the mailbox is
>> under-allocated and mlx5_ib_populate_pas() writes past its end.
>>
>> Store the size in an int, the in_size type of mlx5_cmd_exec(), where a
>> larger value would turn negative, and compute it with overflow checks so
>> a UMEM whose mailbox does not fit is rejected.
>>
>> Mailbox sizes within 512 bytes of INT_MAX still overflow the block count
>> in mlx5_calc_cmd_blocks(), but the command then fails with -ENOMEM
>> before reaching firmware. This predates the change.
>>
>> Fixes: aeae94579caf ("IB/mlx5: Add DEVX support for memory registration")
>> Cc: stable@vger.kernel.org
>> Signed-off-by: Uri Dar <udar@nvidia.com>
>> Reviewed-by: Michael Gur <michaelgur@nvidia.com>
>> Signed-off-by: Edward Srouji <edwards@nvidia.com>
>> ---
>>   drivers/infiniband/hw/mlx5/devx.c | 15 +++++++++------
>>   1 file changed, 9 insertions(+), 6 deletions(-)
>>
>> diff --git a/drivers/infiniband/hw/mlx5/devx.c b/drivers/infiniband/hw/mlx5/devx.c
>> index fc6f793dcc6584e6a22b40c57e2df8f24579eb14..1da6c0e440d6e19d8791b012a148f479cfdddad7 100644
>> --- a/drivers/infiniband/hw/mlx5/devx.c
>> +++ b/drivers/infiniband/hw/mlx5/devx.c
>> @@ -113,7 +113,7 @@ struct devx_umem {
>>
>>   struct devx_umem_reg_cmd {
>>        void                            *in;
>> -     u32                             inlen;
>> +     int                             inlen;
> 
> Do not change things which must be unsigned into signed values.

I asked to use check_add_overflow() but it was dumb to ignore this from 
my side. Will fix in V2.

> 
> Jason

[PATCH rdma-next 2/2] RDMA/core: Don't warn on oversized bundle allocations

[Edward Srouji]

From: Uri Dar <udar@nvidia.com>

_uverbs_alloc() serves requests that do not fit in the bundle's internal
buffer from kvmalloc(), prepending a small list header. The requested
size can be derived from userspace input, and since commit 7661809d493b
("mm: don't allow oversized kvmalloc() calls") kvmalloc() WARNs on any
request above INT_MAX unless __GFP_NOWARN is given.

mlx5 DEVX UMEM registration is one such caller: its CREATE_UMEM mailbox
scales with the size of the user's UMEM, and a mailbox that passes the
driver's INT_MAX check can still exceed INT_MAX once the header is
added.

As with memdup_user(), an allocation sized by userspace should fail
quietly rather than warn. Pass __GFP_NOWARN so such requests fail with
-ENOMEM.

Fixes: 461bb2eee4e1 ("IB/uverbs: Add a simple allocator to uverbs_attr_bundle")
Signed-off-by: Uri Dar <udar@nvidia.com>
Signed-off-by: Edward Srouji <edwards@nvidia.com>
---
 drivers/infiniband/core/ib_core_uverbs.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/drivers/infiniband/core/ib_core_uverbs.c b/drivers/infiniband/core/ib_core_uverbs.c
index 74d9c006bad5aa5d8366b1b75843c9873c62c344..c41b77d7084034613678e8b4b25bc95f72c8e241 100644
--- a/drivers/infiniband/core/ib_core_uverbs.c
+++ b/drivers/infiniband/core/ib_core_uverbs.c
@@ -554,7 +554,7 @@ __malloc void *_uverbs_alloc(struct uverbs_attr_bundle *bundle, size_t size,
 	if (new_used > pbundle->internal_avail) {
 		struct bundle_alloc_head *buf;
 
-		buf = kvmalloc_flex(*buf, data, size, flags);
+		buf = kvmalloc_flex(*buf, data, size, flags | __GFP_NOWARN);
 		if (!buf)
 			return ERR_PTR(-ENOMEM);
 		buf->next = pbundle->allocated_mem;

-- 
2.49.0