[PATCH 0/2] smbdirect: don't hang on netdev reconfiguration

[PATCH 0/2] smbdirect: don't hang on netdev reconfiguration

[Ammar Ratnani]

When I mount a CIFS share using SMB Direct over RoCEv2, I observe a kernel
thread hang if I "configure" its slave network device by taking its link
down and bringing it back up. Attempting to just `ls` the mount point in
this state gives EHOSTDOWN.

I believe the following is the root-cause of the hang: When the link is
taken down, the corresponding GID Table Entry is marked as pending deletion
and has its slave ndev set to NULL. All sends on RDMA connections still
using that GID Table Entry fail at MAD creation. SMB Direct eventually
detects this and tries to disconnect / reconnect to recover. Unfortunately,
disconnecting requires successfully sending either a DREQ or a DREP. Since
neither of them even post, the connection remains in the RDMA_CM_CONNECT
state, and no callback moves it out. The SMB Direct layer never gets the
RDMA_CM_EVENT_DISCONNECTED it's waiting for, and hangs.

Fix this in the CMA. If we call `rdma_disconnect` on a connected connection
and we fail to send both a DREP and a DREQ; disconnect, and thereby send
the `RDMA_CM_EVENT_DISCONNECTED` event to SMB Direct.

I tested this change in QEMU using RXE. I ran Ubuntu 26.04.1 with a
mainline kernel. On commit 9f0346dcbea3 ("Merge tag 'driver-core-7.3-rc2'
of git://git.kernel.org/pub/scm/linux/kernel/git/driver-core/driver-core"),
I reproduce the hang. With this patch applied, SMB Direct immediately
disconnects and reconnects when the slave device is brought down then up.
Listing and reading files from the mount point also work afterwards.

Artifacts for reproducing the hang and testing my fix are available at

    https://github.com/ammrat13/linux-cifs

There, you can find: logs from the "bad" and "good" kernels, the kernel
configuration, the VMs' configuration, and a playbook to trigger the hang.

Ammar Ratnani (2):
  infiniband: force disconnect if DREP and DREQ fail
  infiniband: trace force disconnections

 drivers/infiniband/core/cma.c       | 41 ++++++++++++++++++++++++++---
 drivers/infiniband/core/cma_trace.h |  1 +
 2 files changed, 38 insertions(+), 4 deletions(-)

-- 
2.55.0

[PATCH 1/2] infiniband: force disconnect if DREP and DREQ fail

[Ammar Ratnani]

When we call `rdma_disconnect`, it's possible that we fail to send both
a DREQ and a DREP. In that case, we currently either do nothing or move
into the `IB_CM_TIMEWAIT` state. Either way, we remain in the
`RDMA_CM_CONNECT` state. Anyone waiting for us to disconnect hangs.

Instead, while we're connected, when we call `rdma_disconnect`, and fail
to send both a DREQ and a DREP, mark the connection as disconnected.
Call the upper layer's handler for the disconnect, and destroy the
connection if instructed.

Cc: Leon Romanovsky <leon@kernel.org>
Cc: Jason Gunthorpe <jgg@ziepe.ca>
Cc: Namjae Jeon <linkinjeon@kernel.org>
Cc: Paulo Alcantara <pc@manguebit.org>
Cc: Stefan Metzmacher <metze@samba.org>
Cc: Tom Talpey <tom@talpey.com>
Cc: linux-rdma@vger.kernel.org
Cc: linux-cifs@vger.kernel.org
Cc: samba-technical@lists.samba.org
Cc: linux-kernel@vger.kernel.org
Signed-off-by: Ammar Ratnani <ammrat13@gmail.com>
---
 drivers/infiniband/core/cma.c | 39 +++++++++++++++++++++++++++++++----
 1 file changed, 35 insertions(+), 4 deletions(-)

diff --git a/drivers/infiniband/core/cma.c b/drivers/infiniband/core/cma.c
index 73170b15fc3d..1f1a8b4e160a 100644
--- a/drivers/infiniband/core/cma.c
+++ b/drivers/infiniband/core/cma.c
@@ -4823,6 +4823,34 @@ int rdma_reject(struct rdma_cm_id *id, const void *private_data,
 }
 EXPORT_SYMBOL(rdma_reject);
 
+/*
+ * Called from rdma_disconnect() when we know we're not receiving a callback.
+ * Emulate what that callback (in cma_ib_handler()) would have done.
+ *
+ * NOTE: Despite the name, this function will only actually disconnect when
+ * we're in the RDMA_CM_CONNECT state. Else, it does nothing.
+ */
+static int cma_force_disconnect(struct rdma_id_private *id_priv)
+{
+	struct cma_work *work;
+
+	work = kzalloc_obj(*work);
+	if (!work)
+		return -ENOMEM;
+
+	INIT_WORK(&work->work, cma_work_handler);
+	work->old_state = RDMA_CM_CONNECT;
+	work->new_state = RDMA_CM_DISCONNECT;
+	work->event.event = RDMA_CM_EVENT_DISCONNECTED;
+	work->event.status = -ECONNABORTED;
+
+	cma_id_get(id_priv);
+	work->id = id_priv;
+
+	queue_work(cma_wq, &work->work);
+	return 0;
+}
+
 int rdma_disconnect(struct rdma_cm_id *id)
 {
 	struct rdma_id_private *id_priv;
@@ -4838,12 +4866,15 @@ int rdma_disconnect(struct rdma_cm_id *id)
 			goto out;
 		/* Initiate or respond to a disconnect. */
 		trace_cm_disconnect(id_priv);
-		if (ib_send_cm_dreq(id_priv->cm_id.ib, NULL, 0)) {
-			if (!ib_send_cm_drep(id_priv->cm_id.ib, NULL, 0))
-				trace_cm_sent_drep(id_priv);
-		} else {
+		if (!ib_send_cm_dreq(id_priv->cm_id.ib, NULL, 0)) {
 			trace_cm_sent_dreq(id_priv);
+			goto out;
+		}
+		if (!ib_send_cm_drep(id_priv->cm_id.ib, NULL, 0)) {
+			trace_cm_sent_drep(id_priv);
+			goto out;
 		}
+		ret = cma_force_disconnect(id_priv);
 	} else if (rdma_cap_iw_cm(id->device, id->port_num)) {
 		ret = iw_cm_disconnect(id_priv->cm_id.iw, 0);
 	} else
-- 
2.55.0

[PATCH 2/2] infiniband: trace force disconnections

[Ammar Ratnani]

When we failed to send both a DREQ and a DREP, we have to infer that by
noting the absence of the `sent_dreq` and `sent_drep` trace events. It
would be better if we could see this case by the presence of a trace
event instead. Implement that.

Cc: Leon Romanovsky <leon@kernel.org>
Cc: Jason Gunthorpe <jgg@ziepe.ca>
Cc: Namjae Jeon <linkinjeon@kernel.org>
Cc: Paulo Alcantara <pc@manguebit.org>
Cc: Stefan Metzmacher <metze@samba.org>
Cc: Tom Talpey <tom@talpey.com>
Cc: linux-rdma@vger.kernel.org
Cc: linux-cifs@vger.kernel.org
Cc: samba-technical@lists.samba.org
Cc: linux-kernel@vger.kernel.org
Signed-off-by: Ammar Ratnani <ammrat13@gmail.com>
---
 drivers/infiniband/core/cma.c       | 2 ++
 drivers/infiniband/core/cma_trace.h | 1 +
 2 files changed, 3 insertions(+)

diff --git a/drivers/infiniband/core/cma.c b/drivers/infiniband/core/cma.c
index 1f1a8b4e160a..c4eab48a4a9f 100644
--- a/drivers/infiniband/core/cma.c
+++ b/drivers/infiniband/core/cma.c
@@ -4834,6 +4834,8 @@ static int cma_force_disconnect(struct rdma_id_private *id_priv)
 {
 	struct cma_work *work;
 
+	trace_cm_force_disconnect(id_priv);
+
 	work = kzalloc_obj(*work);
 	if (!work)
 		return -ENOMEM;
diff --git a/drivers/infiniband/core/cma_trace.h b/drivers/infiniband/core/cma_trace.h
index 3456d5f3aa47..3d05d8e80856 100644
--- a/drivers/infiniband/core/cma_trace.h
+++ b/drivers/infiniband/core/cma_trace.h
@@ -61,6 +61,7 @@ DEFINE_CMA_FSM_EVENT(send_sidr_rep);
 DEFINE_CMA_FSM_EVENT(disconnect);
 DEFINE_CMA_FSM_EVENT(sent_drep);
 DEFINE_CMA_FSM_EVENT(sent_dreq);
+DEFINE_CMA_FSM_EVENT(force_disconnect);
 DEFINE_CMA_FSM_EVENT(id_destroy);
 
 TRACE_EVENT(cm_id_attach,
-- 
2.55.0

Re: [PATCH 0/2] smbdirect: don't hang on netdev reconfiguration

[Stefan Metzmacher]

Hi Ammar,

thanks for the fixes!
I let the rdma maintainers comment on the patches in detail,
but it's good to fix these problems where they happen.

> When I mount a CIFS share using SMB Direct over RoCEv2, I observe a kernel
> thread hang if I "configure" its slave network device by taking its link
> down and bringing it back up. Attempting to just `ls` the mount point in
> this state gives EHOSTDOWN.
> 
> I believe the following is the root-cause of the hang: When the link is
> taken down, the corresponding GID Table Entry is marked as pending deletion
> and has its slave ndev set to NULL. All sends on RDMA connections still
> using that GID Table Entry fail at MAD creation. SMB Direct eventually
> detects this and tries to disconnect / reconnect to recover. Unfortunately,
> disconnecting requires successfully sending either a DREQ or a DREP. Since
> neither of them even post, the connection remains in the RDMA_CM_CONNECT
> state, and no callback moves it out. The SMB Direct layer never gets the
> RDMA_CM_EVENT_DISCONNECTED it's waiting for, and hangs.
> 
> Fix this in the CMA. If we call `rdma_disconnect` on a connected connection
> and we fail to send both a DREP and a DREQ; disconnect, and thereby send
> the `RDMA_CM_EVENT_DISCONNECTED` event to SMB Direct.
> 
> I tested this change in QEMU using RXE. I ran Ubuntu 26.04.1 with a
> mainline kernel. On commit 9f0346dcbea3 ("Merge tag 'driver-core-7.3-rc2'
> of git://git.kernel.org/pub/scm/linux/kernel/git/driver-core/driver-core"),
> I reproduce the hang. With this patch applied, SMB Direct immediately
> disconnects and reconnects when the slave device is brought down then up.
> Listing and reading files from the mount point also work afterwards.
> 
> Artifacts for reproducing the hang and testing my fix are available at
> 
>      https://github.com/ammrat13/linux-cifs
Given you have some automation to reproduce it I'm
wondering if you could also test the iwarp case, see
https://lore.kernel.org/linux-rdma/20260805000159.321645-2-yunseong.kim@est.tech/

That was reported as fix for ksmbd, but I guess it
will also happen for the case your're seeing with rxe.
And your fixes are likely also good for ksmbd.

Thanks!
metze

Re: [PATCH 0/2] smbdirect: don't hang on netdev reconfiguration

[Ammar Ratnani]

Thank you for the very quick response, Stefan!

> Given you have some automation to reproduce it I'm
> wondering if you could also test the iwarp case, see
> https://lore.kernel.org/linux-rdma/20260805000159.321645-2-yunseong.kim@est.tech/

I tried running my setup for iWARP. I used the old "bad" kernel, and the
only thing I changed was that I used an siw device instead of an rxe. I
do not get a hang in that case. To be more precise, I see the following
at the end of dmesg:

    [   40.006460] siw0: Port: 1 Link DOWN
    [   40.012912] siw0: Port: 1 Link ACTIVE

But I don't see any message of the form "INFO: task ... blocked for more
than ... seconds." Additionally, I'm able to list and read a file off
the CIFS mount, even after bringing the interface down and up.

For this test, I used commit 2beb1b31a12b ("Merge tag 'bpf-fixes' of
git://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf"). This is a later
tree than what I used in my last email, but I confirmed that I get the
same behavior I described in that email.

Thank you,
- Ammar Ratnani