mirror of https://lore.kernel.org/lkml/
 help / color / mirror / Atom feed
From: Paolo Abeni <pabeni@redhat.com>
To: hexlabsecurity@proton.me, Dust Li <dust.li@linux.alibaba.com>,
	Mahanta Jambigi <mjambigi@linux.ibm.com>,
	"D. Wythe" <alibuda@linux.alibaba.com>,
	Jakub Kicinski <kuba@kernel.org>,
	Eric Dumazet <edumazet@kernel.org>,
	"David S. Miller" <davem@davemloft.net>,
	Sidraya Jayagond <sidraya@linux.ibm.com>
Cc: Guangguan Wang <guangguan.wang@linux.alibaba.com>,
	Tony Lu <tonylu@linux.alibaba.com>,
	netdev@vger.kernel.org, Wenjia Zhang <wenjia@linux.ibm.com>,
	Ursula Braun <ubraun@linux.ibm.com>,
	Hidayath Khan <hidayath@linux.ibm.com>,
	linux-s390@vger.kernel.org, linux-kernel@vger.kernel.org,
	Simon Horman <horms@kernel.org>, Wen Gu <guwen@linux.alibaba.com>,
	Stefan Raspl <raspl@linux.ibm.com>,
	linux-rdma@vger.kernel.org
Subject: Re: [PATCH net 0/4] net/smc: fix the failover cursor switch and wrap rollover
Date: Thu, 19 Nov 2026 10:08:20 +0100	[thread overview]
Message-ID: <9f3269af-64e5-4752-825b-5747cf8ddd86@redhat.com> (raw)
In-Reply-To: <20261113-b4-disp-1a9ab9b0-v1-0-4a7c8d85d849@proton.me>

On 11/14/26 05:14, Bryam Vargas via B4 Relay wrote:
> smc_switch_cursor() runs at SMC-R link failover under send_lock only,
> while smc_cdc_tx_handler() and the rx path update the same cursors under
> the socket spinlock. Its catch-up orders the peer's consumer cursor
> against local_tx_ctrl_fin with smc_curs_comp(), which misorders two
> cursors across the u16 wrap rollover. The v9 cover of the CDC cursor
> bounds series [1] lists the race against smc_cdc_tx_handler() and the
> rollover as not fixed there; Sashiko's review of v8 [2] reported both.
> The same switch has a third problem: a successful completion on the old
> link handled after the switch still credits bytes that are being resent.
> 
> 1/4 takes the socket spinlock around the switch. 2/4 tags each CDC pend
> with a switch generation and credits a completion only from the current
> generation and never backwards; both checks rely on 1/4's lock. 3/4
> computes the catch-up from the signed cursor distance, bounds it by the
> bytes sent and advances the RMB cursors across wraps; it uses
> smc_curs_sdiff() from 2/4. 4/4 uses the same helper for the urgent mark
> in smc_rx_update_consumer(), which has the same rollover problem.
> 
> Tested in an x86 KVM guest on a v7.4-rc2 KASAN kernel, with net/smc of
> the base below built as a module, over soft-RoCE on a veth pair between
> two network namespaces. Unless noted otherwise the RMB is 16 KiB, the
> send buffer 64 KiB, and the failover is triggered by deleting an rdma
> device. The module had test counters and test-only knobs, named where
> they are used below. Three controls pass with and without the series: a
> failover with no knob, one after 1 GiB with both cursors on the same
> wrap (there the series is 1/4 to 3/4), and an urgent byte on the
> reader's wrap. Leaving out 1/4, either of 2/4's two checks, 3/4 or 4/4
> makes its test fail. No KASAN splat or WARN in any run. On a
> PROVE_LOCKING build of the same kernel, a failover with no knob and the
> first test below give no lockdep report with 1/4 to 3/4 (4/4 takes no
> lock); with the two locks of 1/4 taken in the other order, lockdep
> reports the inversion.
> 
> - Send buffer full. A completion handler held by a knob at its credit
>    decision while a test hook takes the link down on another CPU, and a
>    second knob holding the resend until the application has written: on
>    net the switch runs while the handler is held, sndbuf_space ends 4096
>    over the buffer size, and the application's new data replaces bytes
>    still to be resent; the reader gets wrong data. With 1/4 the switch
>    waits for the socket spinlock and the stream is intact.
> - Two completions on the old link held back by a knob and handed back
>    after the switch, send buffer full, the resend held as above: on net
>    they credit 8192 bytes that are still to be resent, and the reader
>    gets wrong data. With 2/4 they credit nothing.
> - The peer shut down its write side, so the switch moves the link
>    without a rewind; an old-link completion held back by a knob is
>    handed back after one on the new link: on net tx_curs_fin moves back
>    and sndbuf_space ends 4096 over the buffer size. With 2/4 nothing
>    moves.
> - 1 GiB through the RMB, so that at the failover local_tx_ctrl_fin is
>    on wrap 0 and the peer's consumer cursor 12288 bytes behind it on
>    wrap 65535: on net the catch-up runs, sndbuf_space goes 20480 over,
>    an RDMA write is refused (remote access error) and the reader gets EOF
>    65536 bytes short while every send() succeeded. Also on net, with a
>    16 KiB send buffer and one 48 KiB send() after the failover, send()
>    takes 36864 bytes and copies 12288 of them past the end of the send
>    buffer (a test counter; KASAN does not see it, the buffer is
>    page-allocated). With 3/4 there is no catch-up, all data arrives,
>    and send() takes 16384.
> - Old-link completions held back by a knob while the peer reads 24576
>    bytes past local_tx_ctrl_fin, then the failover: on net the catch-up
>    is 8192, 16384 short, and the connection stalls with no write space.
>    With 3/4 it is 24576 and all data arrives.
> - 1 GiB through the RMB with the reader stopped on wrap 65535 and an
>    urgent byte sent so that it lands on wrap 0, no SO_OOBINLINE: on net
>    recv(MSG_OOB) fails with EINVAL, the urgent byte is never consumed,
>    and the sender, which waits for that, sends nothing more (its send()
>    times out with 32768 bytes unsent). With 4/4 the byte is read OOB and
>    the stream is intact.
> 
> The series is on net and does not depend on [1]. Patches 1/4, 2/4 and
> 4/4 apply on top of [1] as they are; 3/4 conflicts with [1] 6/6 in
> smc_switch_cursor(), where 3/4's version replaces 6/6's: 3/4 bounds the
> catch-up by the bytes sent, which is tighter than 6/6's bound,
> tx_curs_prep (by inspection). With [1] alone, [1] 3/6 keeps send()
> inside the send buffer, but the misordered catch-up still runs: it
> advances the cursors by up to tx_curs_prep - tx_curs_fin bytes without
> sending them on the new link and tells the peer they are in its RMB (by
> inspection).
> 
> Not run, by inspection only: a CDC received during the switch (1/4), a
> consumer cursor that is ahead across the rollover (3/4) and one past the
> bytes sent, where 3/4's bound applies. Not run with 1/4: a completion
> handled between the switch's first two cursor copies, which leaves
> tx_curs_sent and the producer cursor out of step; [1] measured that case
> without this series.
> 
> Test patch and logs on request.
> 
> [1] https://lore.kernel.org/all/20261112-b4-disp-0287c16e-v9-0-e5e8fa688177@proton.me/
> [2] https://lore.kernel.org/all/179364387382.3131.13471253186660322977@kernel.org/
SMC maintainers: please have a look.

Thanks,

Paolo


      parent reply	other threads:[~2026-11-19  9:08 UTC|newest]

Thread overview: 6+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-11-14  4:14 Bryam Vargas via B4 Relay
2026-11-14  4:14 ` [PATCH net 1/4] net/smc: take the socket spinlock across the failover cursor switch Bryam Vargas via B4 Relay
2026-11-14  4:14 ` [PATCH net 2/4] net/smc: do not credit stale CDC completions after a link switch Bryam Vargas via B4 Relay
2026-11-14  4:14 ` [PATCH net 3/4] net/smc: fix the failover catch-up across the cursor wrap rollover Bryam Vargas via B4 Relay
2026-11-14  4:14 ` [PATCH net 4/4] net/smc: order the urgent mark " Bryam Vargas via B4 Relay
2026-11-19  9:08 ` Paolo Abeni [this message]

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=9f3269af-64e5-4752-825b-5747cf8ddd86@redhat.com \
    --to=pabeni@redhat.com \
    --cc=alibuda@linux.alibaba.com \
    --cc=davem@davemloft.net \
    --cc=dust.li@linux.alibaba.com \
    --cc=edumazet@kernel.org \
    --cc=guangguan.wang@linux.alibaba.com \
    --cc=guwen@linux.alibaba.com \
    --cc=hexlabsecurity@proton.me \
    --cc=hidayath@linux.ibm.com \
    --cc=horms@kernel.org \
    --cc=kuba@kernel.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=linux-rdma@vger.kernel.org \
    --cc=linux-s390@vger.kernel.org \
    --cc=mjambigi@linux.ibm.com \
    --cc=netdev@vger.kernel.org \
    --cc=raspl@linux.ibm.com \
    --cc=sidraya@linux.ibm.com \
    --cc=tonylu@linux.alibaba.com \
    --cc=ubraun@linux.ibm.com \
    --cc=wenjia@linux.ibm.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox

all inboxes | Powered by JetHome®