mirror of https://lore.kernel.org/lkml/
 help / color / mirror / Atom feed
* [PATCH rdma-next v2] RDMA/rxe: Require a responder resource for FLUSH
@ 2026-11-22 17:05 Andrew Stellman
  2026-11-22 19:42 ` Zhu Yanjun
  0 siblings, 1 reply; 4+ messages in thread
From: Andrew Stellman @ 2026-11-22 17:05 UTC (permalink / raw)
  To: Zhu Yanjun; +Cc: Jason Gunthorpe, Leon Romanovsky, linux-rdma, linux-kernel

process_flush() keeps its state in a responder resource from
rxe_prepare_res(), as READ and ATOMIC requests do. check_resource()
refuses those requests when max_dest_rd_atomic is 0, but it does not
check FLUSH. With max_dest_rd_atomic 0, qp->resp.resources is
ZERO_SIZE_PTR, and rxe_prepare_res() writes through it in
free_rd_atomic_resource().

A FLUSH sent to an RC QP with max_dest_rd_atomic 0 oopses the
responder:

  Oops: general protection fault, probably for non-canonical address 0xdffffc0000000002: 0000 [#1] SMP KASAN NOPTI
  KASAN: null-ptr-deref in range [0x0000000000000010-0x0000000000000017]
  CPU: 0 UID: 0 PID: 13 Comm: kworker/u8:0 Not tainted 7.4.0-rc3-gaebb72744764 #1 PREEMPT(lazy)
  Hardware name: QEMU Ubuntu 24.04 PC v2 (i440FX + PIIX, arch_caps fix, 1996), BIOS 1.16.3-debian-1.16.3-2 04/01/2014
  Workqueue: rxe_wq do_work
  RIP: 0010:free_rd_atomic_resource+0x15/0x50
  Code: 00 00 00 00 00 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 0f 1f 40 d6 48 b8 00 00 00 00 00 fc ff df 48 89 fa 48 c1 ea 03 <0f> b6 04 02 84 c0 74 04 3c 03 7e 0b c7 07 00 00 00 00 c3 cc cc cc
  RSP: 0018:ff110000013479f8 EFLAGS: 00000202
  RAX: dffffc0000000000 RBX: 0000000000000010 RCX: ff110000059f9760
  RDX: 0000000000000002 RSI: 0000000000000000 RDI: 0000000000000010
  RBP: 0000000000000001 R08: ff110000024a2180 R09: ffe21c0000268f2c
  R10: 0000000000000003 R11: ff110000059f9298 R12: ff1100000592dc00
  R13: 0000000000000040 R14: ff110000059f9000 R15: ff110000024a2168
  FS:  0000000000000000(0000) GS:ff110000e18da000(0000) knlGS:0000000000000000
  CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
  CR2: dffffc0000000002 CR3: 00000000052ca000 CR4: 0000000000751ef0
  PKRU: 55555554
  Call Trace:
   <TASK>
   rxe_receiver+0x5f6e/0x9a20
   ? flush_workqueue_prep_pwqs+0x1a4/0xe80
   ? kthread+0x2b0/0x3b0
   ? __pfx_rxe_receiver+0x10/0x10
   ? update_cfs_rq_load_avg+0x5a/0x560
   ? pick_task_fair+0x468/0x1a60
   ? __pfx_rxe_receiver+0x10/0x10
   ? _raw_spin_lock_irqsave+0x86/0xe0
   ? __pfx_rxe_receiver+0x10/0x10
   do_work+0x144/0x470
   process_one_work+0x6fc/0x1180
   ? __pfx___schedule+0x10/0x10
   ? __pfx_process_one_work+0x10/0x10
   ? _raw_spin_lock_irq+0x81/0xe0
   ? __pfx_do_work+0x10/0x10
   ? assign_work+0x11d/0x410
   worker_thread+0x45b/0xd10
   ? __pfx_worker_thread+0x10/0x10
   kthread+0x2c8/0x3b0
   ? recalc_sigpending+0x15c/0x1e0
   ? __pfx_kthread+0x10/0x10
   ret_from_fork+0x36e/0x5a0
   ? __pfx_ret_from_fork+0x10/0x10
   ? __switch_to+0x572/0xdc0
   ? __switch_to_asm+0x33/0x70
   ? __pfx_kthread+0x10/0x10
   ret_from_fork_asm+0x1a/0x30
   </TASK>
  Modules linked in:
  ---[ end trace 0000000000000000 ]---
  RIP: 0010:free_rd_atomic_resource+0x15/0x50
  Code: 00 00 00 00 00 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 0f 1f 40 d6 48 b8 00 00 00 00 00 fc ff df 48 89 fa 48 c1 ea 03 <0f> b6 04 02 84 c0 74 04 3c 03 7e 0b c7 07 00 00 00 00 c3 cc cc cc
  RSP: 0018:ff110000013479f8 EFLAGS: 00000202
  RAX: dffffc0000000000 RBX: 0000000000000010 RCX: ff110000059f9760
  RDX: 0000000000000002 RSI: 0000000000000000 RDI: 0000000000000010
  RBP: 0000000000000001 R08: ff110000024a2180 R09: ffe21c0000268f2c
  R10: 0000000000000003 R11: ff110000059f9298 R12: ff1100000592dc00
  R13: 0000000000000040 R14: ff110000059f9000 R15: ff110000024a2168
  FS:  0000000000000000(0000) GS:ff110000e18da000(0000) knlGS:0000000000000000
  CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
  CR2: dffffc0000000002 CR3: 00000000052ca000 CR4: 0000000000751ef0
  PKRU: 55555554

  ------------[ cut here ]------------
  WARNING: drivers/infiniband/sw/rxe/rxe_pool.c:212 at __rxe_cleanup+0x21b/0x290, CPU#0: t028/87
  Modules linked in:
  CPU: 0 UID: 0 PID: 87 Comm: t028 Tainted: G      D             7.4.0-rc3-gaebb72744764 #1 PREEMPT(lazy)
  Tainted: [D]=DIE
  Hardware name: QEMU Ubuntu 24.04 PC v2 (i440FX + PIIX, arch_caps fix, 1996), BIOS 1.16.3-debian-1.16.3-2 04/01/2014
  RIP: 0010:__rxe_cleanup+0x21b/0x290
  Code: f7 fe ff ff 4c 89 e7 e8 43 99 ad fd 84 c0 0f 85 6f ff ff ff be 50 c3 00 00 4c 89 e7 e8 9e 22 f3 00 85 c0 0f 85 5a ff ff ff 90 <0f> 0b 90 eb 04 90 0f 0b 90 bd 92 ff ff ff e9 48 ff ff ff 90 0f 0b
  RSP: 0018:ff110000053f78f8 EFLAGS: 00000246
  RAX: 0000000000000000 RBX: ff110000059f9160 RCX: ffe21c0000a7ef08
  RDX: 0000000000000000 RSI: ff110000053f7860 RDI: ff110000059f9190
  RBP: 0000000000000001 R08: 0000000000000001 R09: ffe21c0000a7eef5
  R10: 0000000000000003 R11: ffffffff89c14700 R12: ff110000059f9188
  R13: ff110000059f9170 R14: ff11000004794ea0 R15: 1fe2200000a7ef1f
  FS:  0000000000000000(0000) GS:ff110000e18da000(0000) knlGS:0000000000000000
  CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
  CR2: dffffc0000000002 CR3: 000000001b872000 CR4: 0000000000751ef0
  PKRU: 55555554
  Call Trace:
   <TASK>
   ? __pfx___rxe_cleanup+0x10/0x10
   ? rxe_qp_chk_destroy+0x61/0xc0
   rxe_destroy_qp+0x2f/0x180
   ib_destroy_qp_user+0x30f/0x4f0
   ? __pfx_ib_destroy_qp_user+0x10/0x10
   ? stack_trace_save+0x8e/0xc0
   uverbs_free_qp+0x7c/0x150
   ? put_files_struct+0x181/0x310
   destroy_hw_idr_uobject+0x7e/0x130
   ? entry_SYSCALL_64_after_hwframe+0x77/0x7f
   uverbs_destroy_uobject+0xdf/0x770
   __uverbs_cleanup_ufile+0x18b/0x390
   ? __pfx___uverbs_cleanup_ufile+0x10/0x10
   ? __pfx_down_write+0x10/0x10
   uverbs_destroy_ufile_hw+0x6d/0x260
   ib_uverbs_close+0x46/0x270
   __fput+0x2f8/0x9e0
   fput_close_sync+0xe2/0x190
   ? __pfx_fput_close_sync+0x10/0x10
   filp_close_sync+0x18/0x30
   put_files_struct+0x181/0x310
   do_exit+0x6ab/0x25f0
   ? __pfx_do_exit+0x10/0x10
   ? zap_other_threads+0xb9/0x300
   ? __pfx_zap_other_threads+0x10/0x10
   ? handle_mm_fault+0x13d/0x420
   do_group_exit+0xab/0x220
   __x64_sys_exit_group+0x39/0x40
   x64_sys_call+0x1962/0x1970
   do_syscall_64+0xdf/0x4c0
   entry_SYSCALL_64_after_hwframe+0x77/0x7f
  RIP: 0033:0x7f34f928a21d
  Code: Unable to access opcode bytes at 0x7f34f928a1f3.
  RSP: 002b:00007ffeed584088 EFLAGS: 00000202 ORIG_RAX: 00000000000000e7
  RAX: ffffffffffffffda RBX: 00007f34f93a0fa8 RCX: 00007f34f928a21d
  RDX: 00000000000000e7 RSI: ffffffffffffff88 RDI: 0000000000000000
  RBP: 00007ffeed5840e0 R08: 00007ffeed584030 R09: 0000000000000000
  R10: 000055d8b71c93e0 R11: 0000000000000202 R12: 0000000000000001
  R13: 0000000000000000 R14: 0000000000000000 R15: 00007f34f93a0fc0
   </TASK>
  ---[ end trace 0000000000000000 ]---

The responder task dies, and when the test exits, destroying the QP
warns on a timeout in __rxe_cleanup() (the WARNING above) and then
hangs.

Add RXE_FLUSH_MASK to the check, so a FLUSH gets the same Invalid
Request NAK a READ would, and the requester sees IB_WC_REM_INV_REQ_ERR.

The issue was found by Claude Opus 5.5 running Quality Playbook, an
LLM-driven code review tool:
https://github.com/andrewstellman/quality-playbook
The fix and this changelog were drafted with Claude Opus 5.5 and
reviewed by me.

Fixes: ea1bb00ee9a5 ("RDMA/rxe: Implement flush execution in responder side")
Signed-off-by: Andrew Stellman <astellman@stellman-greene.com>
---

v2: the full kernel log of the oops and the later WARNING, as Leon
asked (timestamps removed). The code change is the same as v1.
v1: https://lore.kernel.org/r/20261119151332.95466-1-astellman@stellman-greene.com

Tested in QEMU with KASAN on v7.4-rc3, and on v7.4-rc3 with the RDMA
code from rdma for-next (7404c82cbf12). The reproducer is a short
libibverbs program; I can send it off-list.

 drivers/infiniband/sw/rxe/rxe_resp.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/drivers/infiniband/sw/rxe/rxe_resp.c b/drivers/infiniband/sw/rxe/rxe_resp.c
index 1b7f31737225..38b5b0c2abe3 100644
--- a/drivers/infiniband/sw/rxe/rxe_resp.c
+++ b/drivers/infiniband/sw/rxe/rxe_resp.c
@@ -340,7 +340,8 @@ static enum resp_states check_resource(struct rxe_qp *qp,
 {
 	struct rxe_srq *srq = qp->srq;
 
-	if (pkt->mask & (RXE_READ_OR_ATOMIC_MASK | RXE_ATOMIC_WRITE_MASK)) {
+	if (pkt->mask & (RXE_READ_OR_ATOMIC_MASK | RXE_ATOMIC_WRITE_MASK |
+			 RXE_FLUSH_MASK)) {
 		/* it is the requesters job to not send
 		 * too many read/atomic ops, we just
 		 * recycle the responder resource queue

base-commit: 7404c82cbf1246e0e84e2ec21834d23c26f9a8ed
-- 
2.43.0


^ permalink raw reply	[flat|nested] 4+ messages in thread

* Re: [PATCH rdma-next v2] RDMA/rxe: Require a responder resource for FLUSH
  2026-11-22 17:05 [PATCH rdma-next v2] RDMA/rxe: Require a responder resource for FLUSH Andrew Stellman
@ 2026-11-22 19:42 ` Zhu Yanjun
  2026-11-22 19:51   ` Andrew Stellman
  0 siblings, 1 reply; 4+ messages in thread
From: Zhu Yanjun @ 2026-11-22 19:42 UTC (permalink / raw)
  To: Andrew Stellman, Zhu Yanjun, yanjun.zhu
  Cc: Jason Gunthorpe, Leon Romanovsky, linux-rdma, linux-kernel

在 2026/11/22 9:05, Andrew Stellman 写道:
> process_flush() keeps its state in a responder resource from
> rxe_prepare_res(), as READ and ATOMIC requests do. check_resource()
> refuses those requests when max_dest_rd_atomic is 0, but it does not
> check FLUSH. With max_dest_rd_atomic 0, qp->resp.resources is
> ZERO_SIZE_PTR, and rxe_prepare_res() writes through it in
> free_rd_atomic_resource().
> 
> A FLUSH sent to an RC QP with max_dest_rd_atomic 0 oopses the
> responder:
> 
>    Oops: general protection fault, probably for non-canonical address 0xdffffc0000000002: 0000 [#1] SMP KASAN NOPTI
>    KASAN: null-ptr-deref in range [0x0000000000000010-0x0000000000000017]
>    CPU: 0 UID: 0 PID: 13 Comm: kworker/u8:0 Not tainted 7.4.0-rc3-gaebb72744764 #1 PREEMPT(lazy)
>    Hardware name: QEMU Ubuntu 24.04 PC v2 (i440FX + PIIX, arch_caps fix, 1996), BIOS 1.16.3-debian-1.16.3-2 04/01/2014
>    Workqueue: rxe_wq do_work
>    RIP: 0010:free_rd_atomic_resource+0x15/0x50
>    Code: 00 00 00 00 00 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 0f 1f 40 d6 48 b8 00 00 00 00 00 fc ff df 48 89 fa 48 c1 ea 03 <0f> b6 04 02 84 c0 74 04 3c 03 7e 0b c7 07 00 00 00 00 c3 cc cc cc
>    RSP: 0018:ff110000013479f8 EFLAGS: 00000202
>    RAX: dffffc0000000000 RBX: 0000000000000010 RCX: ff110000059f9760
>    RDX: 0000000000000002 RSI: 0000000000000000 RDI: 0000000000000010
>    RBP: 0000000000000001 R08: ff110000024a2180 R09: ffe21c0000268f2c
>    R10: 0000000000000003 R11: ff110000059f9298 R12: ff1100000592dc00
>    R13: 0000000000000040 R14: ff110000059f9000 R15: ff110000024a2168
>    FS:  0000000000000000(0000) GS:ff110000e18da000(0000) knlGS:0000000000000000
>    CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>    CR2: dffffc0000000002 CR3: 00000000052ca000 CR4: 0000000000751ef0
>    PKRU: 55555554
>    Call Trace:
>     <TASK>
>     rxe_receiver+0x5f6e/0x9a20
>     ? flush_workqueue_prep_pwqs+0x1a4/0xe80
>     ? kthread+0x2b0/0x3b0
>     ? __pfx_rxe_receiver+0x10/0x10
>     ? update_cfs_rq_load_avg+0x5a/0x560
>     ? pick_task_fair+0x468/0x1a60
>     ? __pfx_rxe_receiver+0x10/0x10
>     ? _raw_spin_lock_irqsave+0x86/0xe0
>     ? __pfx_rxe_receiver+0x10/0x10
>     do_work+0x144/0x470
>     process_one_work+0x6fc/0x1180
>     ? __pfx___schedule+0x10/0x10
>     ? __pfx_process_one_work+0x10/0x10
>     ? _raw_spin_lock_irq+0x81/0xe0
>     ? __pfx_do_work+0x10/0x10
>     ? assign_work+0x11d/0x410
>     worker_thread+0x45b/0xd10
>     ? __pfx_worker_thread+0x10/0x10
>     kthread+0x2c8/0x3b0
>     ? recalc_sigpending+0x15c/0x1e0
>     ? __pfx_kthread+0x10/0x10
>     ret_from_fork+0x36e/0x5a0
>     ? __pfx_ret_from_fork+0x10/0x10
>     ? __switch_to+0x572/0xdc0
>     ? __switch_to_asm+0x33/0x70
>     ? __pfx_kthread+0x10/0x10
>     ret_from_fork_asm+0x1a/0x30
>     </TASK>
>    Modules linked in:
>    ---[ end trace 0000000000000000 ]---
>    RIP: 0010:free_rd_atomic_resource+0x15/0x50
>    Code: 00 00 00 00 00 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 0f 1f 40 d6 48 b8 00 00 00 00 00 fc ff df 48 89 fa 48 c1 ea 03 <0f> b6 04 02 84 c0 74 04 3c 03 7e 0b c7 07 00 00 00 00 c3 cc cc cc
>    RSP: 0018:ff110000013479f8 EFLAGS: 00000202
>    RAX: dffffc0000000000 RBX: 0000000000000010 RCX: ff110000059f9760
>    RDX: 0000000000000002 RSI: 0000000000000000 RDI: 0000000000000010
>    RBP: 0000000000000001 R08: ff110000024a2180 R09: ffe21c0000268f2c
>    R10: 0000000000000003 R11: ff110000059f9298 R12: ff1100000592dc00
>    R13: 0000000000000040 R14: ff110000059f9000 R15: ff110000024a2168
>    FS:  0000000000000000(0000) GS:ff110000e18da000(0000) knlGS:0000000000000000
>    CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>    CR2: dffffc0000000002 CR3: 00000000052ca000 CR4: 0000000000751ef0
>    PKRU: 55555554
> 
>    ------------[ cut here ]------------
>    WARNING: drivers/infiniband/sw/rxe/rxe_pool.c:212 at __rxe_cleanup+0x21b/0x290, CPU#0: t028/87
>    Modules linked in:
>    CPU: 0 UID: 0 PID: 87 Comm: t028 Tainted: G      D             7.4.0-rc3-gaebb72744764 #1 PREEMPT(lazy)
>    Tainted: [D]=DIE
>    Hardware name: QEMU Ubuntu 24.04 PC v2 (i440FX + PIIX, arch_caps fix, 1996), BIOS 1.16.3-debian-1.16.3-2 04/01/2014
>    RIP: 0010:__rxe_cleanup+0x21b/0x290
>    Code: f7 fe ff ff 4c 89 e7 e8 43 99 ad fd 84 c0 0f 85 6f ff ff ff be 50 c3 00 00 4c 89 e7 e8 9e 22 f3 00 85 c0 0f 85 5a ff ff ff 90 <0f> 0b 90 eb 04 90 0f 0b 90 bd 92 ff ff ff e9 48 ff ff ff 90 0f 0b
>    RSP: 0018:ff110000053f78f8 EFLAGS: 00000246
>    RAX: 0000000000000000 RBX: ff110000059f9160 RCX: ffe21c0000a7ef08
>    RDX: 0000000000000000 RSI: ff110000053f7860 RDI: ff110000059f9190
>    RBP: 0000000000000001 R08: 0000000000000001 R09: ffe21c0000a7eef5
>    R10: 0000000000000003 R11: ffffffff89c14700 R12: ff110000059f9188
>    R13: ff110000059f9170 R14: ff11000004794ea0 R15: 1fe2200000a7ef1f
>    FS:  0000000000000000(0000) GS:ff110000e18da000(0000) knlGS:0000000000000000
>    CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>    CR2: dffffc0000000002 CR3: 000000001b872000 CR4: 0000000000751ef0
>    PKRU: 55555554
>    Call Trace:
>     <TASK>
>     ? __pfx___rxe_cleanup+0x10/0x10
>     ? rxe_qp_chk_destroy+0x61/0xc0
>     rxe_destroy_qp+0x2f/0x180
>     ib_destroy_qp_user+0x30f/0x4f0
>     ? __pfx_ib_destroy_qp_user+0x10/0x10
>     ? stack_trace_save+0x8e/0xc0
>     uverbs_free_qp+0x7c/0x150
>     ? put_files_struct+0x181/0x310
>     destroy_hw_idr_uobject+0x7e/0x130
>     ? entry_SYSCALL_64_after_hwframe+0x77/0x7f
>     uverbs_destroy_uobject+0xdf/0x770
>     __uverbs_cleanup_ufile+0x18b/0x390
>     ? __pfx___uverbs_cleanup_ufile+0x10/0x10
>     ? __pfx_down_write+0x10/0x10
>     uverbs_destroy_ufile_hw+0x6d/0x260
>     ib_uverbs_close+0x46/0x270
>     __fput+0x2f8/0x9e0
>     fput_close_sync+0xe2/0x190
>     ? __pfx_fput_close_sync+0x10/0x10
>     filp_close_sync+0x18/0x30
>     put_files_struct+0x181/0x310
>     do_exit+0x6ab/0x25f0
>     ? __pfx_do_exit+0x10/0x10
>     ? zap_other_threads+0xb9/0x300
>     ? __pfx_zap_other_threads+0x10/0x10
>     ? handle_mm_fault+0x13d/0x420
>     do_group_exit+0xab/0x220
>     __x64_sys_exit_group+0x39/0x40
>     x64_sys_call+0x1962/0x1970
>     do_syscall_64+0xdf/0x4c0
>     entry_SYSCALL_64_after_hwframe+0x77/0x7f
>    RIP: 0033:0x7f34f928a21d
>    Code: Unable to access opcode bytes at 0x7f34f928a1f3.
>    RSP: 002b:00007ffeed584088 EFLAGS: 00000202 ORIG_RAX: 00000000000000e7
>    RAX: ffffffffffffffda RBX: 00007f34f93a0fa8 RCX: 00007f34f928a21d
>    RDX: 00000000000000e7 RSI: ffffffffffffff88 RDI: 0000000000000000
>    RBP: 00007ffeed5840e0 R08: 00007ffeed584030 R09: 0000000000000000
>    R10: 000055d8b71c93e0 R11: 0000000000000202 R12: 0000000000000001
>    R13: 0000000000000000 R14: 0000000000000000 R15: 00007f34f93a0fc0
>     </TASK>
>    ---[ end trace 0000000000000000 ]---
> 
> The responder task dies, and when the test exits, destroying the QP
> warns on a timeout in __rxe_cleanup() (the WARNING above) and then
> hangs.
> 
> Add RXE_FLUSH_MASK to the check, so a FLUSH gets the same Invalid
> Request NAK a READ would, and the requester sees IB_WC_REM_INV_REQ_ERR.
> 
> The issue was found by Claude Opus 5.5 running Quality Playbook, an
> LLM-driven code review tool:
> https://github.com/andrewstellman/quality-playbook
> The fix and this changelog were drafted with Claude Opus 5.5 and
> reviewed by me.
> 
> Fixes: ea1bb00ee9a5 ("RDMA/rxe: Implement flush execution in responder side")
> Signed-off-by: Andrew Stellman <astellman@stellman-greene.com>
> ---
> 
> v2: the full kernel log of the oops and the later WARNING, as Leon
> asked (timestamps removed). The code change is the same as v1.
> v1: https://lore.kernel.org/r/20261119151332.95466-1-astellman@stellman-greene.com
> 
> Tested in QEMU with KASAN on v7.4-rc3, and on v7.4-rc3 with the RDMA
> code from rdma for-next (7404c82cbf12). The reproducer is a short
> libibverbs program; I can send it off-list.
> 
>   drivers/infiniband/sw/rxe/rxe_resp.c | 3 ++-
>   1 file changed, 2 insertions(+), 1 deletion(-)
> 
> diff --git a/drivers/infiniband/sw/rxe/rxe_resp.c b/drivers/infiniband/sw/rxe/rxe_resp.c
> index 1b7f31737225..38b5b0c2abe3 100644
> --- a/drivers/infiniband/sw/rxe/rxe_resp.c
> +++ b/drivers/infiniband/sw/rxe/rxe_resp.c
> @@ -340,7 +340,8 @@ static enum resp_states check_resource(struct rxe_qp *qp,
>   {
>   	struct rxe_srq *srq = qp->srq;
>   
> -	if (pkt->mask & (RXE_READ_OR_ATOMIC_MASK | RXE_ATOMIC_WRITE_MASK)) {
> +	if (pkt->mask & (RXE_READ_OR_ATOMIC_MASK | RXE_ATOMIC_WRITE_MASK |
> +			 RXE_FLUSH_MASK)) {

This looks sensible to me—adding RXE_FLUSH_MASK here avoids 
rxe_prepare_res() / free_rd_atomic_resource() touching ZERO_SIZE_PTR 
when max_dest_rd_atomic == 0 and returns the expected Invalid Request NAK.

Just worth confirming:
(1) RXE_FLUSH_MASK takes the same error/NAK path here without later 
responder states assuming a valid qp->resp.res or non-ZERO_SIZE_PTR 
qp->resp.resources, and

(2) no other RXE_READ_OR_ATOMIC_MASK / RXE_ATOMIC_WRITE_MASK checks in 
rxe_resp.c for max_dest_rd_atomic or responder resources missed 
RXE_FLUSH_MASK.

Thanks a lot.

Zhu Yanjun

>   		/* it is the requesters job to not send
>   		 * too many read/atomic ops, we just
>   		 * recycle the responder resource queue
> 
> base-commit: 7404c82cbf1246e0e84e2ec21834d23c26f9a8ed


^ permalink raw reply	[flat|nested] 4+ messages in thread

* Re: [PATCH rdma-next v2] RDMA/rxe: Require a responder resource for FLUSH
  2026-11-22 19:42 ` Zhu Yanjun
@ 2026-11-22 19:51   ` Andrew Stellman
  2026-11-22 20:13     ` Zhu Yanjun
  0 siblings, 1 reply; 4+ messages in thread
From: Andrew Stellman @ 2026-11-22 19:51 UTC (permalink / raw)
  To: Zhu Yanjun
  Cc: Zhu Yanjun, Jason Gunthorpe, Leon Romanovsky, linux-rdma, linux-kernel

Hi Zhu,

Thanks for looking at it.

> (1) RXE_FLUSH_MASK takes the same error/NAK path here without later
> responder states assuming a valid qp->resp.res or non-ZERO_SIZE_PTR
> qp->resp.resources, and

Yes. With the change, check_resource() returns
RESPST_ERR_TOO_MANY_RDMA_ATM_REQ for a FLUSH, the same state READ and
ATOMIC use. That state calls do_class_ac_error() and moves to
RESPST_COMPLETE and then acknowledge(), which sends the NAK because
aeth_syndrome is set. None of those read qp->resp.res or
qp->resp.resources, and process_flush() and rxe_prepare_res() are not
reached. A retransmitted FLUSH goes to duplicate_request(), where
find_resource() loops over max_dest_rd_atomic entries, so with 0 it
returns NULL and the packet is dropped without touching the array.

In the QEMU test with the fix, the requester's FLUSH completed with
status 9 (IB_WC_REM_INV_REQ_ERR), and the test program then exited
normally (closing its QPs) with no oops or warning. Without the fix,
closing the QP hit the __rxe_cleanup() timeout warning and hung.

> (2) no other RXE_READ_OR_ATOMIC_MASK / RXE_ATOMIC_WRITE_MASK checks in
> rxe_resp.c for max_dest_rd_atomic or responder resources missed
> RXE_FLUSH_MASK.

check_resource() is the only place in rxe_resp.c that checks those
masks against max_dest_rd_atomic. The other places that test them
(check_op_valid(), rxe_resp_check_length(), execute() and
acknowledge()) already handle RXE_FLUSH_MASK, and the only users of
qp->resp.resources are rxe_prepare_res() and find_resource().

Andrew


On Sun, Nov 22, 2026 at 2:42 PM Zhu Yanjun <yanjun.zhu@linux.dev> wrote:
>
> 在 2026/11/22 9:05, Andrew Stellman 写道:
> > process_flush() keeps its state in a responder resource from
> > rxe_prepare_res(), as READ and ATOMIC requests do. check_resource()
> > refuses those requests when max_dest_rd_atomic is 0, but it does not
> > check FLUSH. With max_dest_rd_atomic 0, qp->resp.resources is
> > ZERO_SIZE_PTR, and rxe_prepare_res() writes through it in
> > free_rd_atomic_resource().
> >
> > A FLUSH sent to an RC QP with max_dest_rd_atomic 0 oopses the
> > responder:
> >
> >    Oops: general protection fault, probably for non-canonical address 0xdffffc0000000002: 0000 [#1] SMP KASAN NOPTI
> >    KASAN: null-ptr-deref in range [0x0000000000000010-0x0000000000000017]
> >    CPU: 0 UID: 0 PID: 13 Comm: kworker/u8:0 Not tainted 7.4.0-rc3-gaebb72744764 #1 PREEMPT(lazy)
> >    Hardware name: QEMU Ubuntu 24.04 PC v2 (i440FX + PIIX, arch_caps fix, 1996), BIOS 1.16.3-debian-1.16.3-2 04/01/2014
> >    Workqueue: rxe_wq do_work
> >    RIP: 0010:free_rd_atomic_resource+0x15/0x50
> >    Code: 00 00 00 00 00 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 0f 1f 40 d6 48 b8 00 00 00 00 00 fc ff df 48 89 fa 48 c1 ea 03 <0f> b6 04 02 84 c0 74 04 3c 03 7e 0b c7 07 00 00 00 00 c3 cc cc cc
> >    RSP: 0018:ff110000013479f8 EFLAGS: 00000202
> >    RAX: dffffc0000000000 RBX: 0000000000000010 RCX: ff110000059f9760
> >    RDX: 0000000000000002 RSI: 0000000000000000 RDI: 0000000000000010
> >    RBP: 0000000000000001 R08: ff110000024a2180 R09: ffe21c0000268f2c
> >    R10: 0000000000000003 R11: ff110000059f9298 R12: ff1100000592dc00
> >    R13: 0000000000000040 R14: ff110000059f9000 R15: ff110000024a2168
> >    FS:  0000000000000000(0000) GS:ff110000e18da000(0000) knlGS:0000000000000000
> >    CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> >    CR2: dffffc0000000002 CR3: 00000000052ca000 CR4: 0000000000751ef0
> >    PKRU: 55555554
> >    Call Trace:
> >     <TASK>
> >     rxe_receiver+0x5f6e/0x9a20
> >     ? flush_workqueue_prep_pwqs+0x1a4/0xe80
> >     ? kthread+0x2b0/0x3b0
> >     ? __pfx_rxe_receiver+0x10/0x10
> >     ? update_cfs_rq_load_avg+0x5a/0x560
> >     ? pick_task_fair+0x468/0x1a60
> >     ? __pfx_rxe_receiver+0x10/0x10
> >     ? _raw_spin_lock_irqsave+0x86/0xe0
> >     ? __pfx_rxe_receiver+0x10/0x10
> >     do_work+0x144/0x470
> >     process_one_work+0x6fc/0x1180
> >     ? __pfx___schedule+0x10/0x10
> >     ? __pfx_process_one_work+0x10/0x10
> >     ? _raw_spin_lock_irq+0x81/0xe0
> >     ? __pfx_do_work+0x10/0x10
> >     ? assign_work+0x11d/0x410
> >     worker_thread+0x45b/0xd10
> >     ? __pfx_worker_thread+0x10/0x10
> >     kthread+0x2c8/0x3b0
> >     ? recalc_sigpending+0x15c/0x1e0
> >     ? __pfx_kthread+0x10/0x10
> >     ret_from_fork+0x36e/0x5a0
> >     ? __pfx_ret_from_fork+0x10/0x10
> >     ? __switch_to+0x572/0xdc0
> >     ? __switch_to_asm+0x33/0x70
> >     ? __pfx_kthread+0x10/0x10
> >     ret_from_fork_asm+0x1a/0x30
> >     </TASK>
> >    Modules linked in:
> >    ---[ end trace 0000000000000000 ]---
> >    RIP: 0010:free_rd_atomic_resource+0x15/0x50
> >    Code: 00 00 00 00 00 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 0f 1f 40 d6 48 b8 00 00 00 00 00 fc ff df 48 89 fa 48 c1 ea 03 <0f> b6 04 02 84 c0 74 04 3c 03 7e 0b c7 07 00 00 00 00 c3 cc cc cc
> >    RSP: 0018:ff110000013479f8 EFLAGS: 00000202
> >    RAX: dffffc0000000000 RBX: 0000000000000010 RCX: ff110000059f9760
> >    RDX: 0000000000000002 RSI: 0000000000000000 RDI: 0000000000000010
> >    RBP: 0000000000000001 R08: ff110000024a2180 R09: ffe21c0000268f2c
> >    R10: 0000000000000003 R11: ff110000059f9298 R12: ff1100000592dc00
> >    R13: 0000000000000040 R14: ff110000059f9000 R15: ff110000024a2168
> >    FS:  0000000000000000(0000) GS:ff110000e18da000(0000) knlGS:0000000000000000
> >    CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> >    CR2: dffffc0000000002 CR3: 00000000052ca000 CR4: 0000000000751ef0
> >    PKRU: 55555554
> >
> >    ------------[ cut here ]------------
> >    WARNING: drivers/infiniband/sw/rxe/rxe_pool.c:212 at __rxe_cleanup+0x21b/0x290, CPU#0: t028/87
> >    Modules linked in:
> >    CPU: 0 UID: 0 PID: 87 Comm: t028 Tainted: G      D             7.4.0-rc3-gaebb72744764 #1 PREEMPT(lazy)
> >    Tainted: [D]=DIE
> >    Hardware name: QEMU Ubuntu 24.04 PC v2 (i440FX + PIIX, arch_caps fix, 1996), BIOS 1.16.3-debian-1.16.3-2 04/01/2014
> >    RIP: 0010:__rxe_cleanup+0x21b/0x290
> >    Code: f7 fe ff ff 4c 89 e7 e8 43 99 ad fd 84 c0 0f 85 6f ff ff ff be 50 c3 00 00 4c 89 e7 e8 9e 22 f3 00 85 c0 0f 85 5a ff ff ff 90 <0f> 0b 90 eb 04 90 0f 0b 90 bd 92 ff ff ff e9 48 ff ff ff 90 0f 0b
> >    RSP: 0018:ff110000053f78f8 EFLAGS: 00000246
> >    RAX: 0000000000000000 RBX: ff110000059f9160 RCX: ffe21c0000a7ef08
> >    RDX: 0000000000000000 RSI: ff110000053f7860 RDI: ff110000059f9190
> >    RBP: 0000000000000001 R08: 0000000000000001 R09: ffe21c0000a7eef5
> >    R10: 0000000000000003 R11: ffffffff89c14700 R12: ff110000059f9188
> >    R13: ff110000059f9170 R14: ff11000004794ea0 R15: 1fe2200000a7ef1f
> >    FS:  0000000000000000(0000) GS:ff110000e18da000(0000) knlGS:0000000000000000
> >    CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> >    CR2: dffffc0000000002 CR3: 000000001b872000 CR4: 0000000000751ef0
> >    PKRU: 55555554
> >    Call Trace:
> >     <TASK>
> >     ? __pfx___rxe_cleanup+0x10/0x10
> >     ? rxe_qp_chk_destroy+0x61/0xc0
> >     rxe_destroy_qp+0x2f/0x180
> >     ib_destroy_qp_user+0x30f/0x4f0
> >     ? __pfx_ib_destroy_qp_user+0x10/0x10
> >     ? stack_trace_save+0x8e/0xc0
> >     uverbs_free_qp+0x7c/0x150
> >     ? put_files_struct+0x181/0x310
> >     destroy_hw_idr_uobject+0x7e/0x130
> >     ? entry_SYSCALL_64_after_hwframe+0x77/0x7f
> >     uverbs_destroy_uobject+0xdf/0x770
> >     __uverbs_cleanup_ufile+0x18b/0x390
> >     ? __pfx___uverbs_cleanup_ufile+0x10/0x10
> >     ? __pfx_down_write+0x10/0x10
> >     uverbs_destroy_ufile_hw+0x6d/0x260
> >     ib_uverbs_close+0x46/0x270
> >     __fput+0x2f8/0x9e0
> >     fput_close_sync+0xe2/0x190
> >     ? __pfx_fput_close_sync+0x10/0x10
> >     filp_close_sync+0x18/0x30
> >     put_files_struct+0x181/0x310
> >     do_exit+0x6ab/0x25f0
> >     ? __pfx_do_exit+0x10/0x10
> >     ? zap_other_threads+0xb9/0x300
> >     ? __pfx_zap_other_threads+0x10/0x10
> >     ? handle_mm_fault+0x13d/0x420
> >     do_group_exit+0xab/0x220
> >     __x64_sys_exit_group+0x39/0x40
> >     x64_sys_call+0x1962/0x1970
> >     do_syscall_64+0xdf/0x4c0
> >     entry_SYSCALL_64_after_hwframe+0x77/0x7f
> >    RIP: 0033:0x7f34f928a21d
> >    Code: Unable to access opcode bytes at 0x7f34f928a1f3.
> >    RSP: 002b:00007ffeed584088 EFLAGS: 00000202 ORIG_RAX: 00000000000000e7
> >    RAX: ffffffffffffffda RBX: 00007f34f93a0fa8 RCX: 00007f34f928a21d
> >    RDX: 00000000000000e7 RSI: ffffffffffffff88 RDI: 0000000000000000
> >    RBP: 00007ffeed5840e0 R08: 00007ffeed584030 R09: 0000000000000000
> >    R10: 000055d8b71c93e0 R11: 0000000000000202 R12: 0000000000000001
> >    R13: 0000000000000000 R14: 0000000000000000 R15: 00007f34f93a0fc0
> >     </TASK>
> >    ---[ end trace 0000000000000000 ]---
> >
> > The responder task dies, and when the test exits, destroying the QP
> > warns on a timeout in __rxe_cleanup() (the WARNING above) and then
> > hangs.
> >
> > Add RXE_FLUSH_MASK to the check, so a FLUSH gets the same Invalid
> > Request NAK a READ would, and the requester sees IB_WC_REM_INV_REQ_ERR.
> >
> > The issue was found by Claude Opus 5.5 running Quality Playbook, an
> > LLM-driven code review tool:
> > https://github.com/andrewstellman/quality-playbook
> > The fix and this changelog were drafted with Claude Opus 5.5 and
> > reviewed by me.
> >
> > Fixes: ea1bb00ee9a5 ("RDMA/rxe: Implement flush execution in responder side")
> > Signed-off-by: Andrew Stellman <astellman@stellman-greene.com>
> > ---
> >
> > v2: the full kernel log of the oops and the later WARNING, as Leon
> > asked (timestamps removed). The code change is the same as v1.
> > v1: https://lore.kernel.org/r/20261119151332.95466-1-astellman@stellman-greene.com
> >
> > Tested in QEMU with KASAN on v7.4-rc3, and on v7.4-rc3 with the RDMA
> > code from rdma for-next (7404c82cbf12). The reproducer is a short
> > libibverbs program; I can send it off-list.
> >
> >   drivers/infiniband/sw/rxe/rxe_resp.c | 3 ++-
> >   1 file changed, 2 insertions(+), 1 deletion(-)
> >
> > diff --git a/drivers/infiniband/sw/rxe/rxe_resp.c b/drivers/infiniband/sw/rxe/rxe_resp.c
> > index 1b7f31737225..38b5b0c2abe3 100644
> > --- a/drivers/infiniband/sw/rxe/rxe_resp.c
> > +++ b/drivers/infiniband/sw/rxe/rxe_resp.c
> > @@ -340,7 +340,8 @@ static enum resp_states check_resource(struct rxe_qp *qp,
> >   {
> >       struct rxe_srq *srq = qp->srq;
> >
> > -     if (pkt->mask & (RXE_READ_OR_ATOMIC_MASK | RXE_ATOMIC_WRITE_MASK)) {
> > +     if (pkt->mask & (RXE_READ_OR_ATOMIC_MASK | RXE_ATOMIC_WRITE_MASK |
> > +                      RXE_FLUSH_MASK)) {
>
> This looks sensible to me—adding RXE_FLUSH_MASK here avoids
> rxe_prepare_res() / free_rd_atomic_resource() touching ZERO_SIZE_PTR
> when max_dest_rd_atomic == 0 and returns the expected Invalid Request NAK.
>
> Just worth confirming:
> (1) RXE_FLUSH_MASK takes the same error/NAK path here without later
> responder states assuming a valid qp->resp.res or non-ZERO_SIZE_PTR
> qp->resp.resources, and
>
> (2) no other RXE_READ_OR_ATOMIC_MASK / RXE_ATOMIC_WRITE_MASK checks in
> rxe_resp.c for max_dest_rd_atomic or responder resources missed
> RXE_FLUSH_MASK.
>
> Thanks a lot.
>
> Zhu Yanjun
>
> >               /* it is the requesters job to not send
> >                * too many read/atomic ops, we just
> >                * recycle the responder resource queue
> >
> > base-commit: 7404c82cbf1246e0e84e2ec21834d23c26f9a8ed
>

^ permalink raw reply	[flat|nested] 4+ messages in thread

* Re: [PATCH rdma-next v2] RDMA/rxe: Require a responder resource for FLUSH
  2026-11-22 19:51   ` Andrew Stellman
@ 2026-11-22 20:13     ` Zhu Yanjun
  0 siblings, 0 replies; 4+ messages in thread
From: Zhu Yanjun @ 2026-11-22 20:13 UTC (permalink / raw)
  To: Andrew Stellman, yanjun.zhu
  Cc: Zhu Yanjun, Jason Gunthorpe, Leon Romanovsky, linux-rdma, linux-kernel


在 2026/11/22 11:51, Andrew Stellman 写道:
> Hi Zhu,
>
> Thanks for looking at it.
>
>> (1) RXE_FLUSH_MASK takes the same error/NAK path here without later
>> responder states assuming a valid qp->resp.res or non-ZERO_SIZE_PTR
>> qp->resp.resources, and
> Yes. With the change, check_resource() returns
> RESPST_ERR_TOO_MANY_RDMA_ATM_REQ for a FLUSH, the same state READ and
> ATOMIC use. That state calls do_class_ac_error() and moves to
> RESPST_COMPLETE and then acknowledge(), which sends the NAK because
> aeth_syndrome is set. None of those read qp->resp.res or
> qp->resp.resources, and process_flush() and rxe_prepare_res() are not
> reached. A retransmitted FLUSH goes to duplicate_request(), where
> find_resource() loops over max_dest_rd_atomic entries, so with 0 it
> returns NULL and the packet is dropped without touching the array.
>
> In the QEMU test with the fix, the requester's FLUSH completed with
> status 9 (IB_WC_REM_INV_REQ_ERR), and the test program then exited
> normally (closing its QPs) with no oops or warning. Without the fix,
> closing the QP hit the __rxe_cleanup() timeout warning and hung.
>
>> (2) no other RXE_READ_OR_ATOMIC_MASK / RXE_ATOMIC_WRITE_MASK checks in
>> rxe_resp.c for max_dest_rd_atomic or responder resources missed
>> RXE_FLUSH_MASK.
> check_resource() is the only place in rxe_resp.c that checks those
> masks against max_dest_rd_atomic. The other places that test them
> (check_op_valid(), rxe_resp_check_length(), execute() and
> acknowledge()) already handle RXE_FLUSH_MASK, and the only users of
> qp->resp.resources are rxe_prepare_res() and find_resource().

Thanks a lot for your quick reply. The above explanations answer my 
questions.

I am fine with this commit.

Reviewed-by: Zhu Yanjun <yanjun.zhu@linux.dev>

Zhu Yanjun

>
> Andrew
>
>
> On Sun, Nov 22, 2026 at 2:42 PM Zhu Yanjun <yanjun.zhu@linux.dev> wrote:
>> 在 2026/11/22 9:05, Andrew Stellman 写道:
>>> process_flush() keeps its state in a responder resource from
>>> rxe_prepare_res(), as READ and ATOMIC requests do. check_resource()
>>> refuses those requests when max_dest_rd_atomic is 0, but it does not
>>> check FLUSH. With max_dest_rd_atomic 0, qp->resp.resources is
>>> ZERO_SIZE_PTR, and rxe_prepare_res() writes through it in
>>> free_rd_atomic_resource().
>>>
>>> A FLUSH sent to an RC QP with max_dest_rd_atomic 0 oopses the
>>> responder:
>>>
>>>     Oops: general protection fault, probably for non-canonical address 0xdffffc0000000002: 0000 [#1] SMP KASAN NOPTI
>>>     KASAN: null-ptr-deref in range [0x0000000000000010-0x0000000000000017]
>>>     CPU: 0 UID: 0 PID: 13 Comm: kworker/u8:0 Not tainted 7.4.0-rc3-gaebb72744764 #1 PREEMPT(lazy)
>>>     Hardware name: QEMU Ubuntu 24.04 PC v2 (i440FX + PIIX, arch_caps fix, 1996), BIOS 1.16.3-debian-1.16.3-2 04/01/2014
>>>     Workqueue: rxe_wq do_work
>>>     RIP: 0010:free_rd_atomic_resource+0x15/0x50
>>>     Code: 00 00 00 00 00 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 0f 1f 40 d6 48 b8 00 00 00 00 00 fc ff df 48 89 fa 48 c1 ea 03 <0f> b6 04 02 84 c0 74 04 3c 03 7e 0b c7 07 00 00 00 00 c3 cc cc cc
>>>     RSP: 0018:ff110000013479f8 EFLAGS: 00000202
>>>     RAX: dffffc0000000000 RBX: 0000000000000010 RCX: ff110000059f9760
>>>     RDX: 0000000000000002 RSI: 0000000000000000 RDI: 0000000000000010
>>>     RBP: 0000000000000001 R08: ff110000024a2180 R09: ffe21c0000268f2c
>>>     R10: 0000000000000003 R11: ff110000059f9298 R12: ff1100000592dc00
>>>     R13: 0000000000000040 R14: ff110000059f9000 R15: ff110000024a2168
>>>     FS:  0000000000000000(0000) GS:ff110000e18da000(0000) knlGS:0000000000000000
>>>     CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>>>     CR2: dffffc0000000002 CR3: 00000000052ca000 CR4: 0000000000751ef0
>>>     PKRU: 55555554
>>>     Call Trace:
>>>      <TASK>
>>>      rxe_receiver+0x5f6e/0x9a20
>>>      ? flush_workqueue_prep_pwqs+0x1a4/0xe80
>>>      ? kthread+0x2b0/0x3b0
>>>      ? __pfx_rxe_receiver+0x10/0x10
>>>      ? update_cfs_rq_load_avg+0x5a/0x560
>>>      ? pick_task_fair+0x468/0x1a60
>>>      ? __pfx_rxe_receiver+0x10/0x10
>>>      ? _raw_spin_lock_irqsave+0x86/0xe0
>>>      ? __pfx_rxe_receiver+0x10/0x10
>>>      do_work+0x144/0x470
>>>      process_one_work+0x6fc/0x1180
>>>      ? __pfx___schedule+0x10/0x10
>>>      ? __pfx_process_one_work+0x10/0x10
>>>      ? _raw_spin_lock_irq+0x81/0xe0
>>>      ? __pfx_do_work+0x10/0x10
>>>      ? assign_work+0x11d/0x410
>>>      worker_thread+0x45b/0xd10
>>>      ? __pfx_worker_thread+0x10/0x10
>>>      kthread+0x2c8/0x3b0
>>>      ? recalc_sigpending+0x15c/0x1e0
>>>      ? __pfx_kthread+0x10/0x10
>>>      ret_from_fork+0x36e/0x5a0
>>>      ? __pfx_ret_from_fork+0x10/0x10
>>>      ? __switch_to+0x572/0xdc0
>>>      ? __switch_to_asm+0x33/0x70
>>>      ? __pfx_kthread+0x10/0x10
>>>      ret_from_fork_asm+0x1a/0x30
>>>      </TASK>
>>>     Modules linked in:
>>>     ---[ end trace 0000000000000000 ]---
>>>     RIP: 0010:free_rd_atomic_resource+0x15/0x50
>>>     Code: 00 00 00 00 00 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 0f 1f 40 d6 48 b8 00 00 00 00 00 fc ff df 48 89 fa 48 c1 ea 03 <0f> b6 04 02 84 c0 74 04 3c 03 7e 0b c7 07 00 00 00 00 c3 cc cc cc
>>>     RSP: 0018:ff110000013479f8 EFLAGS: 00000202
>>>     RAX: dffffc0000000000 RBX: 0000000000000010 RCX: ff110000059f9760
>>>     RDX: 0000000000000002 RSI: 0000000000000000 RDI: 0000000000000010
>>>     RBP: 0000000000000001 R08: ff110000024a2180 R09: ffe21c0000268f2c
>>>     R10: 0000000000000003 R11: ff110000059f9298 R12: ff1100000592dc00
>>>     R13: 0000000000000040 R14: ff110000059f9000 R15: ff110000024a2168
>>>     FS:  0000000000000000(0000) GS:ff110000e18da000(0000) knlGS:0000000000000000
>>>     CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>>>     CR2: dffffc0000000002 CR3: 00000000052ca000 CR4: 0000000000751ef0
>>>     PKRU: 55555554
>>>
>>>     ------------[ cut here ]------------
>>>     WARNING: drivers/infiniband/sw/rxe/rxe_pool.c:212 at __rxe_cleanup+0x21b/0x290, CPU#0: t028/87
>>>     Modules linked in:
>>>     CPU: 0 UID: 0 PID: 87 Comm: t028 Tainted: G      D             7.4.0-rc3-gaebb72744764 #1 PREEMPT(lazy)
>>>     Tainted: [D]=DIE
>>>     Hardware name: QEMU Ubuntu 24.04 PC v2 (i440FX + PIIX, arch_caps fix, 1996), BIOS 1.16.3-debian-1.16.3-2 04/01/2014
>>>     RIP: 0010:__rxe_cleanup+0x21b/0x290
>>>     Code: f7 fe ff ff 4c 89 e7 e8 43 99 ad fd 84 c0 0f 85 6f ff ff ff be 50 c3 00 00 4c 89 e7 e8 9e 22 f3 00 85 c0 0f 85 5a ff ff ff 90 <0f> 0b 90 eb 04 90 0f 0b 90 bd 92 ff ff ff e9 48 ff ff ff 90 0f 0b
>>>     RSP: 0018:ff110000053f78f8 EFLAGS: 00000246
>>>     RAX: 0000000000000000 RBX: ff110000059f9160 RCX: ffe21c0000a7ef08
>>>     RDX: 0000000000000000 RSI: ff110000053f7860 RDI: ff110000059f9190
>>>     RBP: 0000000000000001 R08: 0000000000000001 R09: ffe21c0000a7eef5
>>>     R10: 0000000000000003 R11: ffffffff89c14700 R12: ff110000059f9188
>>>     R13: ff110000059f9170 R14: ff11000004794ea0 R15: 1fe2200000a7ef1f
>>>     FS:  0000000000000000(0000) GS:ff110000e18da000(0000) knlGS:0000000000000000
>>>     CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>>>     CR2: dffffc0000000002 CR3: 000000001b872000 CR4: 0000000000751ef0
>>>     PKRU: 55555554
>>>     Call Trace:
>>>      <TASK>
>>>      ? __pfx___rxe_cleanup+0x10/0x10
>>>      ? rxe_qp_chk_destroy+0x61/0xc0
>>>      rxe_destroy_qp+0x2f/0x180
>>>      ib_destroy_qp_user+0x30f/0x4f0
>>>      ? __pfx_ib_destroy_qp_user+0x10/0x10
>>>      ? stack_trace_save+0x8e/0xc0
>>>      uverbs_free_qp+0x7c/0x150
>>>      ? put_files_struct+0x181/0x310
>>>      destroy_hw_idr_uobject+0x7e/0x130
>>>      ? entry_SYSCALL_64_after_hwframe+0x77/0x7f
>>>      uverbs_destroy_uobject+0xdf/0x770
>>>      __uverbs_cleanup_ufile+0x18b/0x390
>>>      ? __pfx___uverbs_cleanup_ufile+0x10/0x10
>>>      ? __pfx_down_write+0x10/0x10
>>>      uverbs_destroy_ufile_hw+0x6d/0x260
>>>      ib_uverbs_close+0x46/0x270
>>>      __fput+0x2f8/0x9e0
>>>      fput_close_sync+0xe2/0x190
>>>      ? __pfx_fput_close_sync+0x10/0x10
>>>      filp_close_sync+0x18/0x30
>>>      put_files_struct+0x181/0x310
>>>      do_exit+0x6ab/0x25f0
>>>      ? __pfx_do_exit+0x10/0x10
>>>      ? zap_other_threads+0xb9/0x300
>>>      ? __pfx_zap_other_threads+0x10/0x10
>>>      ? handle_mm_fault+0x13d/0x420
>>>      do_group_exit+0xab/0x220
>>>      __x64_sys_exit_group+0x39/0x40
>>>      x64_sys_call+0x1962/0x1970
>>>      do_syscall_64+0xdf/0x4c0
>>>      entry_SYSCALL_64_after_hwframe+0x77/0x7f
>>>     RIP: 0033:0x7f34f928a21d
>>>     Code: Unable to access opcode bytes at 0x7f34f928a1f3.
>>>     RSP: 002b:00007ffeed584088 EFLAGS: 00000202 ORIG_RAX: 00000000000000e7
>>>     RAX: ffffffffffffffda RBX: 00007f34f93a0fa8 RCX: 00007f34f928a21d
>>>     RDX: 00000000000000e7 RSI: ffffffffffffff88 RDI: 0000000000000000
>>>     RBP: 00007ffeed5840e0 R08: 00007ffeed584030 R09: 0000000000000000
>>>     R10: 000055d8b71c93e0 R11: 0000000000000202 R12: 0000000000000001
>>>     R13: 0000000000000000 R14: 0000000000000000 R15: 00007f34f93a0fc0
>>>      </TASK>
>>>     ---[ end trace 0000000000000000 ]---
>>>
>>> The responder task dies, and when the test exits, destroying the QP
>>> warns on a timeout in __rxe_cleanup() (the WARNING above) and then
>>> hangs.
>>>
>>> Add RXE_FLUSH_MASK to the check, so a FLUSH gets the same Invalid
>>> Request NAK a READ would, and the requester sees IB_WC_REM_INV_REQ_ERR.
>>>
>>> The issue was found by Claude Opus 5.5 running Quality Playbook, an
>>> LLM-driven code review tool:
>>> https://github.com/andrewstellman/quality-playbook
>>> The fix and this changelog were drafted with Claude Opus 5.5 and
>>> reviewed by me.
>>>
>>> Fixes: ea1bb00ee9a5 ("RDMA/rxe: Implement flush execution in responder side")
>>> Signed-off-by: Andrew Stellman <astellman@stellman-greene.com>
>>> ---
>>>
>>> v2: the full kernel log of the oops and the later WARNING, as Leon
>>> asked (timestamps removed). The code change is the same as v1.
>>> v1: https://lore.kernel.org/r/20261119151332.95466-1-astellman@stellman-greene.com
>>>
>>> Tested in QEMU with KASAN on v7.4-rc3, and on v7.4-rc3 with the RDMA
>>> code from rdma for-next (7404c82cbf12). The reproducer is a short
>>> libibverbs program; I can send it off-list.
>>>
>>>    drivers/infiniband/sw/rxe/rxe_resp.c | 3 ++-
>>>    1 file changed, 2 insertions(+), 1 deletion(-)
>>>
>>> diff --git a/drivers/infiniband/sw/rxe/rxe_resp.c b/drivers/infiniband/sw/rxe/rxe_resp.c
>>> index 1b7f31737225..38b5b0c2abe3 100644
>>> --- a/drivers/infiniband/sw/rxe/rxe_resp.c
>>> +++ b/drivers/infiniband/sw/rxe/rxe_resp.c
>>> @@ -340,7 +340,8 @@ static enum resp_states check_resource(struct rxe_qp *qp,
>>>    {
>>>        struct rxe_srq *srq = qp->srq;
>>>
>>> -     if (pkt->mask & (RXE_READ_OR_ATOMIC_MASK | RXE_ATOMIC_WRITE_MASK)) {
>>> +     if (pkt->mask & (RXE_READ_OR_ATOMIC_MASK | RXE_ATOMIC_WRITE_MASK |
>>> +                      RXE_FLUSH_MASK)) {
>> This looks sensible to me—adding RXE_FLUSH_MASK here avoids
>> rxe_prepare_res() / free_rd_atomic_resource() touching ZERO_SIZE_PTR
>> when max_dest_rd_atomic == 0 and returns the expected Invalid Request NAK.
>>
>> Just worth confirming:
>> (1) RXE_FLUSH_MASK takes the same error/NAK path here without later
>> responder states assuming a valid qp->resp.res or non-ZERO_SIZE_PTR
>> qp->resp.resources, and
>>
>> (2) no other RXE_READ_OR_ATOMIC_MASK / RXE_ATOMIC_WRITE_MASK checks in
>> rxe_resp.c for max_dest_rd_atomic or responder resources missed
>> RXE_FLUSH_MASK.
>>
>> Thanks a lot.
>>
>> Zhu Yanjun
>>
>>>                /* it is the requesters job to not send
>>>                 * too many read/atomic ops, we just
>>>                 * recycle the responder resource queue
>>>
>>> base-commit: 7404c82cbf1246e0e84e2ec21834d23c26f9a8ed

-- 
Best Regards,
Yanjun.Zhu


^ permalink raw reply	[flat|nested] 4+ messages in thread

end of thread, other threads:[~2026-11-22 20:13 UTC | newest]

Thread overview: 4+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2026-11-22 17:05 [PATCH rdma-next v2] RDMA/rxe: Require a responder resource for FLUSH Andrew Stellman
2026-11-22 19:42 ` Zhu Yanjun
2026-11-22 19:51   ` Andrew Stellman
2026-11-22 20:13     ` Zhu Yanjun

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox

all inboxes | Powered by JetHome®