[PATCH v3 0/2] RDMA/rxe: fix ADVISE_MR prefetch on non-ODP MRs

[PATCH v3 0/2] RDMA/rxe: fix ADVISE_MR prefetch on non-ODP MRs

[Norbert Szetei via B4 Relay]

ADVISE_MR prefetch on a plain MR runs to_ib_umem_odp() on a struct
ib_umem, giving a KASAN slab-out-of-bounds read in
ib_umem_odp_map_dma_and_lock() (splat in patch 2).

v1 checked is_odp_mr() after lookup_mr(). Leon asked for lookup_mr() to
do the check itself, off mr->access. That only holds once mr->access
cannot disagree with the umem, so patch 1 stops the two paths that
assign mr->access after registration from touching IB_ACCESS_ON_DEMAND,
and patch 2 passes the flag to lookup_mr() in both prefetch arms.

Based on for-next. In rxe_rereg_user_mr() the new check rejects the
access flags before the mr->ibmr.pd swap, where for-rc's ae36a5b609ae
("RDMA/rxe: validate access flags before swapping the MR's PD") also
moved the existing validation.

v2 -> v3:
 - rebased on for-next, no functional change
v2: https://lore.kernel.org/all/20260913-rxe-advise-mr-v2-v2-0-b806c789871c@doyensec.com/
v1: https://lore.kernel.org/all/521D5E74-89E3-43C0-81C7-AC0BE52591E7@doyensec.com/

Signed-off-by: Norbert Szetei <norbert@doyensec.com>
---
Norbert Szetei (2):
      RDMA/rxe: Reject IB_ACCESS_ON_DEMAND changes after MR creation
      RDMA/rxe: Reject prefetch of a non-ODP MR

 drivers/infiniband/sw/rxe/rxe_mr.c    | 6 ++++++
 drivers/infiniband/sw/rxe/rxe_odp.c   | 4 ++--
 drivers/infiniband/sw/rxe/rxe_verbs.c | 6 ++++++
 3 files changed, 14 insertions(+), 2 deletions(-)
---
base-commit: 9dcacdc41083cccff74292851bf9fde999e0731a
change-id: 20260923-rxe-advise-mr-v3-355a808e851f

Best regards,
--  
Norbert Szetei <norbert@doyensec.com>

[PATCH v3 1/2] RDMA/rxe: Reject IB_ACCESS_ON_DEMAND changes after MR creation

[Norbert Szetei via B4 Relay]

From: Norbert Szetei <norbert@doyensec.com>

Whether an MR is an ODP MR is decided once, at registration time:
rxe_reg_user_mr() picks rxe_odp_mr_init_user() over rxe_mr_init_user()
based on IB_ACCESS_ON_DEMAND, and only the former builds an ib_umem_odp
via ib_umem_odp_get(). The umem cannot change type afterwards, and
is_odp_mr() reads mr->umem->is_odp.

Two paths assign mr->access after that point and can leave it
describing an MR type the umem does not have:

rxe_rereg_user_mr() with IB_MR_REREG_ACCESS overwrites mr->access with
the caller's value, and IB_ACCESS_ON_DEMAND is part of
RXE_ACCESS_SUPPORTED_MR, so userspace can set the flag on a plain MR or
clear it on an ODP MR while the umem stays what it was.

rxe_reg_fast_mr() takes mr->access from the REG_MR work request
unmasked and moves the MR to RXE_MR_STATE_VALID, on an MR that
rxe_mr_init_fast() left with a NULL umem.

Reject IB_ACCESS_ON_DEMAND in both, so mr->access carries the flag only
for an MR that has an ODP umem and the flag can be used to identify one.

Fixes: 544c7f62cf32 ("RDMA/rxe: Implement rereg_user_mr")
Cc: stable@vger.kernel.org
Signed-off-by: Norbert Szetei <norbert@doyensec.com>
---
 drivers/infiniband/sw/rxe/rxe_mr.c    | 6 ++++++
 drivers/infiniband/sw/rxe/rxe_verbs.c | 6 ++++++
 2 files changed, 12 insertions(+)

diff --git a/drivers/infiniband/sw/rxe/rxe_mr.c b/drivers/infiniband/sw/rxe/rxe_mr.c
index 875eceb55fdf..2afda5154dfb 100644
--- a/drivers/infiniband/sw/rxe/rxe_mr.c
+++ b/drivers/infiniband/sw/rxe/rxe_mr.c
@@ -795,6 +795,12 @@ int rxe_reg_fast_mr(struct rxe_qp *qp, struct rxe_send_wqe *wqe)
 		return -EINVAL;
 	}
 
+	/* an MR with no umem is never an ODP MR */
+	if (unlikely(access & IB_ACCESS_ON_DEMAND)) {
+		rxe_dbg_mr(mr, "access = 0x%x requests ODP\n", access);
+		return -EINVAL;
+	}
+
 	mr->access = access;
 	mr->lkey = key;
 	mr->rkey = key;
diff --git a/drivers/infiniband/sw/rxe/rxe_verbs.c b/drivers/infiniband/sw/rxe/rxe_verbs.c
index 96c7716057fe..21855274f63a 100644
--- a/drivers/infiniband/sw/rxe/rxe_verbs.c
+++ b/drivers/infiniband/sw/rxe/rxe_verbs.c
@@ -1331,6 +1331,12 @@ static struct ib_mr *rxe_rereg_user_mr(struct ib_mr *ibmr, int flags,
 	if (err)
 		return ERR_PTR(err);
 
+	if ((flags & IB_MR_REREG_ACCESS) &&
+	    ((access ^ mr->access) & IB_ACCESS_ON_DEMAND)) {
+		rxe_err_mr(mr, "cannot change IB_ACCESS_ON_DEMAND\n");
+		return ERR_PTR(-EOPNOTSUPP);
+	}
+
 	if (flags & IB_MR_REREG_PD) {
 		rxe_put(old_pd);
 		rxe_get(pd);

-- 
2.55.0

Re: [PATCH v3 1/2] RDMA/rxe: Reject IB_ACCESS_ON_DEMAND changes after MR creation

[Zhu Yanjun]

在 2026/9/23 7:40, Norbert Szetei via B4 Relay 写道:
> From: Norbert Szetei <norbert@doyensec.com>
> 
> Whether an MR is an ODP MR is decided once, at registration time:
> rxe_reg_user_mr() picks rxe_odp_mr_init_user() over rxe_mr_init_user()
> based on IB_ACCESS_ON_DEMAND, and only the former builds an ib_umem_odp
> via ib_umem_odp_get(). The umem cannot change type afterwards, and
> is_odp_mr() reads mr->umem->is_odp.
> 
> Two paths assign mr->access after that point and can leave it
> describing an MR type the umem does not have:
> 
> rxe_rereg_user_mr() with IB_MR_REREG_ACCESS overwrites mr->access with
> the caller's value, and IB_ACCESS_ON_DEMAND is part of
> RXE_ACCESS_SUPPORTED_MR, so userspace can set the flag on a plain MR or
> clear it on an ODP MR while the umem stays what it was.
> 
> rxe_reg_fast_mr() takes mr->access from the REG_MR work request
> unmasked and moves the MR to RXE_MR_STATE_VALID, on an MR that
> rxe_mr_init_fast() left with a NULL umem.
> 
> Reject IB_ACCESS_ON_DEMAND in both, so mr->access carries the flag only
> for an MR that has an ODP umem and the flag can be used to identify one.
> 
> Fixes: 544c7f62cf32 ("RDMA/rxe: Implement rereg_user_mr")
> Cc: stable@vger.kernel.org
> Signed-off-by: Norbert Szetei <norbert@doyensec.com>

I have already reviewed this commit. I am fine with this commit.
So
Reviewed-by: Zhu Yanjun <yanjun.zhu@linux.dev>

Zhu Yanjun

> ---
>   drivers/infiniband/sw/rxe/rxe_mr.c    | 6 ++++++
>   drivers/infiniband/sw/rxe/rxe_verbs.c | 6 ++++++
>   2 files changed, 12 insertions(+)
> 
> diff --git a/drivers/infiniband/sw/rxe/rxe_mr.c b/drivers/infiniband/sw/rxe/rxe_mr.c
> index 875eceb55fdf..2afda5154dfb 100644
> --- a/drivers/infiniband/sw/rxe/rxe_mr.c
> +++ b/drivers/infiniband/sw/rxe/rxe_mr.c
> @@ -795,6 +795,12 @@ int rxe_reg_fast_mr(struct rxe_qp *qp, struct rxe_send_wqe *wqe)
>   		return -EINVAL;
>   	}
>   
> +	/* an MR with no umem is never an ODP MR */
> +	if (unlikely(access & IB_ACCESS_ON_DEMAND)) {
> +		rxe_dbg_mr(mr, "access = 0x%x requests ODP\n", access);
> +		return -EINVAL;
> +	}
> +
>   	mr->access = access;
>   	mr->lkey = key;
>   	mr->rkey = key;
> diff --git a/drivers/infiniband/sw/rxe/rxe_verbs.c b/drivers/infiniband/sw/rxe/rxe_verbs.c
> index 96c7716057fe..21855274f63a 100644
> --- a/drivers/infiniband/sw/rxe/rxe_verbs.c
> +++ b/drivers/infiniband/sw/rxe/rxe_verbs.c
> @@ -1331,6 +1331,12 @@ static struct ib_mr *rxe_rereg_user_mr(struct ib_mr *ibmr, int flags,
>   	if (err)
>   		return ERR_PTR(err);
>   
> +	if ((flags & IB_MR_REREG_ACCESS) &&
> +	    ((access ^ mr->access) & IB_ACCESS_ON_DEMAND)) {
> +		rxe_err_mr(mr, "cannot change IB_ACCESS_ON_DEMAND\n");
> +		return ERR_PTR(-EOPNOTSUPP);
> +	}
> +
>   	if (flags & IB_MR_REREG_PD) {
>   		rxe_put(old_pd);
>   		rxe_get(pd);
> 

[PATCH v3 2/2] RDMA/rxe: Reject prefetch of a non-ODP MR

[Norbert Szetei via B4 Relay]

From: Norbert Szetei <norbert@doyensec.com>

rxe_ib_advise_mr_prefetch() and rxe_ib_prefetch_sg_list() look up the MR
by lkey and hand it to rxe_odp_do_pagefault_and_lock() without checking
that it is an ODP MR. That path runs to_ib_umem_odp() on mr->umem, and
for a non-ODP MR mr->umem is a plain struct ib_umem from ib_umem_get(),
so the container_of() in to_ib_umem_odp() lands past the end of the
object and ib_umem_odp_map_dma_and_lock() reads its ib_umem_odp fields
out of bounds.

lookup_mr() validates the lkey, PD, access and state but not the MR
type, and IB_UVERBS_ADVISE_MR_ADVICE_PREFETCH is accepted for any MR.

 BUG: KASAN: slab-out-of-bounds in ib_umem_odp_map_dma_and_lock+0x884/0x8a0
 Read of size 8 at addr ffff88810a3ebcf0 by task advi/921
  ib_umem_odp_map_dma_and_lock+0x884/0x8a0
  rxe_ib_advise_mr+0x543/0xad0
  ib_uverbs_handler_UVERBS_METHOD_ADVISE_MR+0x446/0x530
  ib_uverbs_cmd_verbs+0x2b3c/0x3b20
  ib_uverbs_ioctl+0x1e3/0x310
 Allocated by task 921:
  __ib_umem_get_va+0x13e/0xae0
  rxe_mr_init_user+0x2ae/0xb00
  rxe_reg_user_mr+0x337/0x510
 The buggy address belongs to the object at ffff88810a3ebc80
  which belongs to the cache kmalloc-96 of size 96

Ask lookup_mr() for IB_ACCESS_ON_DEMAND in both the synchronous and
the asynchronous prefetch arm. mr->access carries that flag only for
an MR registered as ODP, so the existing
(access & mr->access) != access test rejects a plain MR and the
prefetch fails with -EINVAL.

Fixes: 3576b0df1588 ("RDMA/rxe: Implement synchronous prefetch for ODP MRs")
Cc: stable@vger.kernel.org
Signed-off-by: Norbert Szetei <norbert@doyensec.com>
---
 drivers/infiniband/sw/rxe/rxe_odp.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/drivers/infiniband/sw/rxe/rxe_odp.c b/drivers/infiniband/sw/rxe/rxe_odp.c
index e870efa7a0a3..f8ffd6a3219a 100644
--- a/drivers/infiniband/sw/rxe/rxe_odp.c
+++ b/drivers/infiniband/sw/rxe/rxe_odp.c
@@ -463,7 +463,7 @@ static int rxe_ib_prefetch_sg_list(struct ib_pd *ibpd,
 		struct rxe_mr *mr;
 		struct ib_umem_odp *umem_odp;
 
-		mr = lookup_mr(pd, IB_ACCESS_LOCAL_WRITE,
+		mr = lookup_mr(pd, IB_ACCESS_LOCAL_WRITE | IB_ACCESS_ON_DEMAND,
 			       sg_list[i].lkey, RXE_LOOKUP_LOCAL);
 
 		if (!mr) {
@@ -529,7 +529,7 @@ static int rxe_ib_advise_mr_prefetch(struct ib_pd *ibpd,
 
 	for (i = 0; i < num_sge; ++i) {
 		/* Takes a reference, which will be released in the queued work */
-		mr = lookup_mr(pd, IB_ACCESS_LOCAL_WRITE,
+		mr = lookup_mr(pd, IB_ACCESS_LOCAL_WRITE | IB_ACCESS_ON_DEMAND,
 			       sg_list[i].lkey, RXE_LOOKUP_LOCAL);
 		if (!mr) {
 			mr = ERR_PTR(-EINVAL);

-- 
2.55.0

Re: [PATCH v3 2/2] RDMA/rxe: Reject prefetch of a non-ODP MR

[Zhu Yanjun]

在 2026/9/23 7:40, Norbert Szetei via B4 Relay 写道:
> From: Norbert Szetei <norbert@doyensec.com>
> 
> rxe_ib_advise_mr_prefetch() and rxe_ib_prefetch_sg_list() look up the MR
> by lkey and hand it to rxe_odp_do_pagefault_and_lock() without checking
> that it is an ODP MR. That path runs to_ib_umem_odp() on mr->umem, and
> for a non-ODP MR mr->umem is a plain struct ib_umem from ib_umem_get(),
> so the container_of() in to_ib_umem_odp() lands past the end of the
> object and ib_umem_odp_map_dma_and_lock() reads its ib_umem_odp fields
> out of bounds.
> 
> lookup_mr() validates the lkey, PD, access and state but not the MR
> type, and IB_UVERBS_ADVISE_MR_ADVICE_PREFETCH is accepted for any MR.
> 
>   BUG: KASAN: slab-out-of-bounds in ib_umem_odp_map_dma_and_lock+0x884/0x8a0
>   Read of size 8 at addr ffff88810a3ebcf0 by task advi/921
>    ib_umem_odp_map_dma_and_lock+0x884/0x8a0
>    rxe_ib_advise_mr+0x543/0xad0
>    ib_uverbs_handler_UVERBS_METHOD_ADVISE_MR+0x446/0x530
>    ib_uverbs_cmd_verbs+0x2b3c/0x3b20
>    ib_uverbs_ioctl+0x1e3/0x310
>   Allocated by task 921:
>    __ib_umem_get_va+0x13e/0xae0
>    rxe_mr_init_user+0x2ae/0xb00
>    rxe_reg_user_mr+0x337/0x510
>   The buggy address belongs to the object at ffff88810a3ebc80
>    which belongs to the cache kmalloc-96 of size 96
> 
> Ask lookup_mr() for IB_ACCESS_ON_DEMAND in both the synchronous and
> the asynchronous prefetch arm. mr->access carries that flag only for
> an MR registered as ODP, so the existing
> (access & mr->access) != access test rejects a plain MR and the
> prefetch fails with -EINVAL.
> 
> Fixes: 3576b0df1588 ("RDMA/rxe: Implement synchronous prefetch for ODP MRs")
> Cc: stable@vger.kernel.org
> Signed-off-by: Norbert Szetei <norbert@doyensec.com>

Thanks a lot.
It is fine with me.

Reviewed-by: Zhu Yanjun <yanjun.zhu@linux.dev>

Zhu Yanjun

> ---
>   drivers/infiniband/sw/rxe/rxe_odp.c | 4 ++--
>   1 file changed, 2 insertions(+), 2 deletions(-)
> 
> diff --git a/drivers/infiniband/sw/rxe/rxe_odp.c b/drivers/infiniband/sw/rxe/rxe_odp.c
> index e870efa7a0a3..f8ffd6a3219a 100644
> --- a/drivers/infiniband/sw/rxe/rxe_odp.c
> +++ b/drivers/infiniband/sw/rxe/rxe_odp.c
> @@ -463,7 +463,7 @@ static int rxe_ib_prefetch_sg_list(struct ib_pd *ibpd,
>   		struct rxe_mr *mr;
>   		struct ib_umem_odp *umem_odp;
>   
> -		mr = lookup_mr(pd, IB_ACCESS_LOCAL_WRITE,
> +		mr = lookup_mr(pd, IB_ACCESS_LOCAL_WRITE | IB_ACCESS_ON_DEMAND,
>   			       sg_list[i].lkey, RXE_LOOKUP_LOCAL);
>   
>   		if (!mr) {
> @@ -529,7 +529,7 @@ static int rxe_ib_advise_mr_prefetch(struct ib_pd *ibpd,
>   
>   	for (i = 0; i < num_sge; ++i) {
>   		/* Takes a reference, which will be released in the queued work */
> -		mr = lookup_mr(pd, IB_ACCESS_LOCAL_WRITE,
> +		mr = lookup_mr(pd, IB_ACCESS_LOCAL_WRITE | IB_ACCESS_ON_DEMAND,
>   			       sg_list[i].lkey, RXE_LOOKUP_LOCAL);
>   		if (!mr) {
>   			mr = ERR_PTR(-EINVAL);
>