From: Paolo Abeni <pabeni@redhat.com>
To: hexlabsecurity@proton.me, Dust Li <dust.li@linux.alibaba.com>,
Mahanta Jambigi <mjambigi@linux.ibm.com>,
"D. Wythe" <alibuda@linux.alibaba.com>,
Jakub Kicinski <kuba@kernel.org>,
Eric Dumazet <edumazet@kernel.org>,
"David S. Miller" <davem@davemloft.net>,
Sidraya Jayagond <sidraya@linux.ibm.com>
Cc: Guangguan Wang <guangguan.wang@linux.alibaba.com>,
Tony Lu <tonylu@linux.alibaba.com>,
netdev@vger.kernel.org, Wenjia Zhang <wenjia@linux.ibm.com>,
Ursula Braun <ubraun@linux.ibm.com>,
Hidayath Khan <hidayath@linux.ibm.com>,
linux-s390@vger.kernel.org, linux-kernel@vger.kernel.org,
Simon Horman <horms@kernel.org>, Wen Gu <guwen@linux.alibaba.com>,
Stefan Raspl <raspl@linux.ibm.com>,
linux-rdma@vger.kernel.org
Subject: Re: [PATCH net 0/4] net/smc: fix the failover cursor switch and wrap rollover
Date: Thu, 19 Nov 2026 10:08:20 +0100 [thread overview]
Message-ID: <9f3269af-64e5-4752-825b-5747cf8ddd86@redhat.com> (raw)
In-Reply-To: <20261113-b4-disp-1a9ab9b0-v1-0-4a7c8d85d849@proton.me>
On 11/14/26 05:14, Bryam Vargas via B4 Relay wrote:
> smc_switch_cursor() runs at SMC-R link failover under send_lock only,
> while smc_cdc_tx_handler() and the rx path update the same cursors under
> the socket spinlock. Its catch-up orders the peer's consumer cursor
> against local_tx_ctrl_fin with smc_curs_comp(), which misorders two
> cursors across the u16 wrap rollover. The v9 cover of the CDC cursor
> bounds series [1] lists the race against smc_cdc_tx_handler() and the
> rollover as not fixed there; Sashiko's review of v8 [2] reported both.
> The same switch has a third problem: a successful completion on the old
> link handled after the switch still credits bytes that are being resent.
>
> 1/4 takes the socket spinlock around the switch. 2/4 tags each CDC pend
> with a switch generation and credits a completion only from the current
> generation and never backwards; both checks rely on 1/4's lock. 3/4
> computes the catch-up from the signed cursor distance, bounds it by the
> bytes sent and advances the RMB cursors across wraps; it uses
> smc_curs_sdiff() from 2/4. 4/4 uses the same helper for the urgent mark
> in smc_rx_update_consumer(), which has the same rollover problem.
>
> Tested in an x86 KVM guest on a v7.4-rc2 KASAN kernel, with net/smc of
> the base below built as a module, over soft-RoCE on a veth pair between
> two network namespaces. Unless noted otherwise the RMB is 16 KiB, the
> send buffer 64 KiB, and the failover is triggered by deleting an rdma
> device. The module had test counters and test-only knobs, named where
> they are used below. Three controls pass with and without the series: a
> failover with no knob, one after 1 GiB with both cursors on the same
> wrap (there the series is 1/4 to 3/4), and an urgent byte on the
> reader's wrap. Leaving out 1/4, either of 2/4's two checks, 3/4 or 4/4
> makes its test fail. No KASAN splat or WARN in any run. On a
> PROVE_LOCKING build of the same kernel, a failover with no knob and the
> first test below give no lockdep report with 1/4 to 3/4 (4/4 takes no
> lock); with the two locks of 1/4 taken in the other order, lockdep
> reports the inversion.
>
> - Send buffer full. A completion handler held by a knob at its credit
> decision while a test hook takes the link down on another CPU, and a
> second knob holding the resend until the application has written: on
> net the switch runs while the handler is held, sndbuf_space ends 4096
> over the buffer size, and the application's new data replaces bytes
> still to be resent; the reader gets wrong data. With 1/4 the switch
> waits for the socket spinlock and the stream is intact.
> - Two completions on the old link held back by a knob and handed back
> after the switch, send buffer full, the resend held as above: on net
> they credit 8192 bytes that are still to be resent, and the reader
> gets wrong data. With 2/4 they credit nothing.
> - The peer shut down its write side, so the switch moves the link
> without a rewind; an old-link completion held back by a knob is
> handed back after one on the new link: on net tx_curs_fin moves back
> and sndbuf_space ends 4096 over the buffer size. With 2/4 nothing
> moves.
> - 1 GiB through the RMB, so that at the failover local_tx_ctrl_fin is
> on wrap 0 and the peer's consumer cursor 12288 bytes behind it on
> wrap 65535: on net the catch-up runs, sndbuf_space goes 20480 over,
> an RDMA write is refused (remote access error) and the reader gets EOF
> 65536 bytes short while every send() succeeded. Also on net, with a
> 16 KiB send buffer and one 48 KiB send() after the failover, send()
> takes 36864 bytes and copies 12288 of them past the end of the send
> buffer (a test counter; KASAN does not see it, the buffer is
> page-allocated). With 3/4 there is no catch-up, all data arrives,
> and send() takes 16384.
> - Old-link completions held back by a knob while the peer reads 24576
> bytes past local_tx_ctrl_fin, then the failover: on net the catch-up
> is 8192, 16384 short, and the connection stalls with no write space.
> With 3/4 it is 24576 and all data arrives.
> - 1 GiB through the RMB with the reader stopped on wrap 65535 and an
> urgent byte sent so that it lands on wrap 0, no SO_OOBINLINE: on net
> recv(MSG_OOB) fails with EINVAL, the urgent byte is never consumed,
> and the sender, which waits for that, sends nothing more (its send()
> times out with 32768 bytes unsent). With 4/4 the byte is read OOB and
> the stream is intact.
>
> The series is on net and does not depend on [1]. Patches 1/4, 2/4 and
> 4/4 apply on top of [1] as they are; 3/4 conflicts with [1] 6/6 in
> smc_switch_cursor(), where 3/4's version replaces 6/6's: 3/4 bounds the
> catch-up by the bytes sent, which is tighter than 6/6's bound,
> tx_curs_prep (by inspection). With [1] alone, [1] 3/6 keeps send()
> inside the send buffer, but the misordered catch-up still runs: it
> advances the cursors by up to tx_curs_prep - tx_curs_fin bytes without
> sending them on the new link and tells the peer they are in its RMB (by
> inspection).
>
> Not run, by inspection only: a CDC received during the switch (1/4), a
> consumer cursor that is ahead across the rollover (3/4) and one past the
> bytes sent, where 3/4's bound applies. Not run with 1/4: a completion
> handled between the switch's first two cursor copies, which leaves
> tx_curs_sent and the producer cursor out of step; [1] measured that case
> without this series.
>
> Test patch and logs on request.
>
> [1] https://lore.kernel.org/all/20261112-b4-disp-0287c16e-v9-0-e5e8fa688177@proton.me/
> [2] https://lore.kernel.org/all/179364387382.3131.13471253186660322977@kernel.org/
SMC maintainers: please have a look.
Thanks,
Paolo
prev parent reply other threads:[~2026-11-19 9:08 UTC|newest]
Thread overview: 6+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-11-14 4:14 Bryam Vargas via B4 Relay
2026-11-14 4:14 ` [PATCH net 1/4] net/smc: take the socket spinlock across the failover cursor switch Bryam Vargas via B4 Relay
2026-11-14 4:14 ` [PATCH net 2/4] net/smc: do not credit stale CDC completions after a link switch Bryam Vargas via B4 Relay
2026-11-14 4:14 ` [PATCH net 3/4] net/smc: fix the failover catch-up across the cursor wrap rollover Bryam Vargas via B4 Relay
2026-11-14 4:14 ` [PATCH net 4/4] net/smc: order the urgent mark " Bryam Vargas via B4 Relay
2026-11-19 9:08 ` Paolo Abeni [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=9f3269af-64e5-4752-825b-5747cf8ddd86@redhat.com \
--to=pabeni@redhat.com \
--cc=alibuda@linux.alibaba.com \
--cc=davem@davemloft.net \
--cc=dust.li@linux.alibaba.com \
--cc=edumazet@kernel.org \
--cc=guangguan.wang@linux.alibaba.com \
--cc=guwen@linux.alibaba.com \
--cc=hexlabsecurity@proton.me \
--cc=hidayath@linux.ibm.com \
--cc=horms@kernel.org \
--cc=kuba@kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-rdma@vger.kernel.org \
--cc=linux-s390@vger.kernel.org \
--cc=mjambigi@linux.ibm.com \
--cc=netdev@vger.kernel.org \
--cc=raspl@linux.ibm.com \
--cc=sidraya@linux.ibm.com \
--cc=tonylu@linux.alibaba.com \
--cc=ubraun@linux.ibm.com \
--cc=wenjia@linux.ibm.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®