From: Bryam Vargas via B4 Relay <devnull+hexlabsecurity.proton.me@kernel.org>
To: Mahanta Jambigi <mjambigi@linux.ibm.com>,
Dust Li <dust.li@linux.alibaba.com>,
Eric Dumazet <edumazet@kernel.org>,
Paolo Abeni <pabeni@redhat.com>,
"David S. Miller" <davem@davemloft.net>,
Jakub Kicinski <kuba@kernel.org>,
"D. Wythe" <alibuda@linux.alibaba.com>,
Sidraya Jayagond <sidraya@linux.ibm.com>
Cc: Ursula Braun <ubraun@linux.ibm.com>,
Wenjia Zhang <wenjia@linux.ibm.com>,
Stefan Raspl <raspl@linux.ibm.com>,
linux-s390@vger.kernel.org, linux-kernel@vger.kernel.org,
Wen Gu <guwen@linux.alibaba.com>,
Tony Lu <tonylu@linux.alibaba.com>,
Simon Horman <horms@kernel.org>,
Guangguan Wang <guangguan.wang@linux.alibaba.com>,
linux-rdma@vger.kernel.org, netdev@vger.kernel.org,
Hidayath Khan <hidayath@linux.ibm.com>
Subject: [PATCH net v9 6/6] net/smc: bound the link failover cursor advance to the send buffer
Date: Thu, 12 Nov 2026 07:59:40 -0500 [thread overview]
Message-ID: <20261112-b4-disp-0287c16e-v9-6-e5e8fa688177@proton.me> (raw)
In-Reply-To: <20261112-b4-disp-0287c16e-v9-0-e5e8fa688177@proton.me>
From: Bryam Vargas <hexlabsecurity@proton.me>
On SMC-R link failover, smc_switch_cursor() rewinds the send and
producer cursors to the last completed CDC send, then advances them by
the distance the peer's consumer cursor is ahead. That cursor comes
from the wire: a forged one adds an arbitrary amount to sndbuf_space
and to the cursors, and the next RDMA write takes its source past the
send buffer.
Advance only for a count inside the peer's RMBE, and no further than
tx_curs_prep; a conforming peer's cursor is inside both. For any other
count the write that follows can end past the peer's RMBE and needs
"net/smc: keep an RDMA write inside the peer's RMBE" to bound it. Set
tx_curs_fin and local_tx_ctrl_fin from the advanced cursors instead of
adding to them: smc_cdc_tx_handler() can rewrite both while the switch
runs. The switch is still not serialized against the handler: a
completion handled inside it is still credited more than once and can
make the next write hand the peer bytes it has already read.
Fixes: c6f02ebeea3a ("net/smc: switch connections to alternate link")
Cc: stable@vger.kernel.org # see patch description, needs another patch
Signed-off-by: Bryam Vargas <hexlabsecurity@proton.me>
---
v9: tx_curs_fin and local_tx_ctrl_fin are copied from the advanced
tx_curs_sent and local_tx_ctrl.prod, which change only under send_lock,
instead of being added to (second point of [1]). The write past the
peer's RMBE that v8 introduced for a count past the RMBE is bounded in
5/6. The cap is tx_curs_prep and not the pre-failover tx_curs_sent: an
earlier failover may have rewound that one below the peer's cursor.
rxe, two links (v7.4-rc2 plus d7900fa51642, KASAN), send buffer 64K,
peer RMBE 16K: one byte and then 4096 confirmed; 12000 and 5000 sent and
not confirmed, the 5000 in two writes, 287 up to the end of the RMBE and
4713; all read by a conforming peer. A hook holds the pending
completions and hands the first one to smc_cdc_tx_handler() inside
smc_switch_cursor(), after it has read tx_curs_fin and
local_tx_ctrl_fin: unpatched and v8 leave local_tx_ctrl_fin at count
16713 (the RMBE is 16384 bytes) and tx_curs_fin 12000 bytes past
tx_curs_prep; v9 leaves tx_curs_fin on tx_curs_prep and
local_tx_ctrl_fin on prod. Without the hook, v8 ends the same run with
the cursors v9 leaves.
The first case of the note under v8's 4/4 (cursor forged one wrap
ahead at count 64, with 4096 bytes unconfirmed in one run and corked
in another) gives on v9 what that note reports for v8: the advance
stops at tx_curs_prep and nothing is written after the switch. v8 and
unpatched were not rerun for it.
Not changed: the switch is not serialized against the handler. The
handler takes send_lock inside the socket spinlock (smc_tx_pending())
and the switch holds send_lock only, so keeping it out means taking the
socket spinlock around send_lock in smc_switch_conns(). What remains,
on unpatched, v8 and v9 alike unless noted:
- a completion handled inside the switch is credited to sndbuf_space
by the handler and again by the advance: in the run above
sndbuf_space is 77536 after the switch, for a 65536-byte send
buffer. The old-link completions that follow set tx_curs_fin back
and credit 4713 bytes once more: 82249. 3/6 keeps send() inside the
buffer, but send() can then overwrite data the peer has not received
(by inspection).
- a completion between the first two cursor copies of the switch
leaves tx_curs_sent behind local_tx_ctrl.prod. The next write starts
at tx_curs_sent and lands at prod, that far past its place in the
RMBE, so the peer is handed that many bytes a second time. In the
first case under 4/6 the peer reads 600 bytes twice on all three. In
the fourth case under 5/6, where the peer has read across the end of
its RMBE before the resend, unpatched and v8 have the write refused
and the connection lost; v9's goes out and the peer gets 600 bytes
twice there too.
- v9 only: the first case under 4/6 ends with sndbuf_space 1200 bytes
over the buffer, against 600 on unpatched and v8. v9 sets
tx_curs_fin back to tx_curs_sent, and the next old-link completion,
when it is handled before the validation message's, credits those
600 once more. Handled after it, all three end 600 over (by
inspection).
- the completion of the switch's validation message carries
tx_curs_sent and steps tx_curs_fin back to it whenever a completion
handled before it left tx_curs_fin ahead; across the end of the ring
that step is credited as more than a whole ring (the mechanism is in
3/6's message). In the fourth case under 5/6 it is credited 124472
bytes for a 65536-byte send buffer on every variant, and on v9,
where the last write goes out, sndbuf_space ends at 3x the buffer.
v9's smc_curs_copy() removes one route to that state, the switch
itself leaving tx_curs_fin ahead (by inspection); late old-link
completions are still one.
[1] https://lore.kernel.org/all/179364387382.3131.13471253186660322977@kernel.org/
---
net/smc/smc_core.c | 26 ++++++++++++++++----------
1 file changed, 16 insertions(+), 10 deletions(-)
diff --git a/net/smc/smc_core.c b/net/smc/smc_core.c
index 0edbe6106c1a..b5625f1a8792 100644
--- a/net/smc/smc_core.c
+++ b/net/smc/smc_core.c
@@ -1034,7 +1034,7 @@ static int smc_switch_cursor(struct smc_sock *smc, struct smc_cdc_tx_pend *pend,
struct smc_wr_buf *wr_buf)
{
struct smc_connection *conn = &smc->conn;
- union smc_host_cursor cons, fin;
+ union smc_host_cursor cons, fin, prep;
int rc = 0;
int diff;
@@ -1043,16 +1043,22 @@ static int smc_switch_cursor(struct smc_sock *smc, struct smc_cdc_tx_pend *pend,
/* set prod cursor to old state, enforce tx_rdma_writes() */
smc_curs_copy(&conn->local_tx_ctrl.prod, &fin, conn);
smc_curs_copy(&cons, &conn->local_rx_ctrl.cons, conn);
-
- if (smc_curs_comp(conn->peer_rmbe_size, &cons, &fin) < 0) {
- /* cons cursor advanced more than fin, and prod was set
- * fin above, so now prod is smaller than cons. Fix that.
+ smc_curs_copy(&prep, &conn->tx_curs_prep, conn);
+
+ if (cons.count < conn->peer_rmbe_size &&
+ smc_curs_comp(conn->peer_rmbe_size, &cons, &fin) < 0) {
+ /* cons cursor advanced more than fin, and prod was set fin
+ * above: catch up. cons comes off the wire, so only for a
+ * count inside the peer's RMBE and no further than
+ * tx_curs_prep. smc_cdc_tx_handler() is not excluded and can
+ * rewrite the fin cursors meanwhile: set them, do not add.
*/
- diff = smc_curs_diff(conn->peer_rmbe_size, &fin, &cons);
+ diff = min(smc_curs_diff(conn->peer_rmbe_size, &fin, &cons),
+ smc_curs_diff(conn->sndbuf_desc->len,
+ &conn->tx_curs_sent, &prep));
smc_curs_add(conn->sndbuf_desc->len,
&conn->tx_curs_sent, diff);
- smc_curs_add(conn->sndbuf_desc->len,
- &conn->tx_curs_fin, diff);
+ smc_curs_copy(&conn->tx_curs_fin, &conn->tx_curs_sent, conn);
smp_mb__before_atomic();
atomic_add(diff, &conn->sndbuf_space);
@@ -1060,8 +1066,8 @@ static int smc_switch_cursor(struct smc_sock *smc, struct smc_cdc_tx_pend *pend,
smc_curs_add(conn->peer_rmbe_size,
&conn->local_tx_ctrl.prod, diff);
- smc_curs_add(conn->peer_rmbe_size,
- &conn->local_tx_ctrl_fin, diff);
+ smc_curs_copy(&conn->local_tx_ctrl_fin,
+ &conn->local_tx_ctrl.prod, conn);
}
/* recalculate, value is used by tx_rdma_writes() */
atomic_set(&smc->conn.peer_rmbe_space, smc_write_space(conn));
--
2.56.0
next prev parent reply other threads:[~2026-11-12 12:59 UTC|newest]
Thread overview: 11+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-11-12 12:59 [PATCH net v9 0/6] net/smc: bound wire-controlled CDC cursors against the local buffers Bryam Vargas via B4 Relay
2026-11-12 12:59 ` [PATCH net v9 1/6] net/smc: bound the wire-controlled producer cursor to the RMB Bryam Vargas via B4 Relay
2026-11-12 12:59 ` [PATCH net v9 2/6] net/smc: bound the receive length " Bryam Vargas via B4 Relay
2026-11-16 13:00 ` netdev-bot+sashiko
2026-11-12 12:59 ` [PATCH net v9 3/6] net/smc: bound the send length to the send buffer in smc_tx_sendmsg() Bryam Vargas via B4 Relay
2026-11-16 13:00 ` netdev-bot+sashiko
2026-11-12 12:59 ` [PATCH net v9 4/6] net/smc: keep the source of an RDMA write inside the send buffer Bryam Vargas via B4 Relay
2026-11-12 12:59 ` [PATCH net v9 5/6] net/smc: keep an RDMA write inside the peer's RMBE Bryam Vargas via B4 Relay
2026-11-16 13:00 ` netdev-bot+sashiko
2026-11-12 12:59 ` Bryam Vargas via B4 Relay [this message]
2026-11-16 13:00 ` [PATCH net v9 6/6] net/smc: bound the link failover cursor advance to the send buffer netdev-bot+sashiko
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20261112-b4-disp-0287c16e-v9-6-e5e8fa688177@proton.me \
--to=devnull+hexlabsecurity.proton.me@kernel.org \
--cc=alibuda@linux.alibaba.com \
--cc=davem@davemloft.net \
--cc=dust.li@linux.alibaba.com \
--cc=edumazet@kernel.org \
--cc=guangguan.wang@linux.alibaba.com \
--cc=guwen@linux.alibaba.com \
--cc=hexlabsecurity@proton.me \
--cc=hidayath@linux.ibm.com \
--cc=horms@kernel.org \
--cc=kuba@kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-rdma@vger.kernel.org \
--cc=linux-s390@vger.kernel.org \
--cc=mjambigi@linux.ibm.com \
--cc=netdev@vger.kernel.org \
--cc=pabeni@redhat.com \
--cc=raspl@linux.ibm.com \
--cc=sidraya@linux.ibm.com \
--cc=tonylu@linux.alibaba.com \
--cc=ubraun@linux.ibm.com \
--cc=wenjia@linux.ibm.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®