From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from acj35aaf85.lhr1.oracleemaildelivery.com (acj35aaf85.lhr1.oracleemaildelivery.com [130.35.116.85]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0C2F63D0BF3 for ; Thu, 14 May 2026 21:55:01 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=130.35.116.85 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1778795704; cv=none; b=ews2mix9oX7VFAjxbZvezYTvuv59Klp+Dan8fkNJErRsxprZZ2W/N6RbT8bnJzw//UFu3MLz0JJWtn2zG00ju6QZ/Dk+bHMHGcr6Z6Dab79/vCj2qF3TQzQHEb26dBpcyShdjfhfhpYTnqynpsaN1nHDup+4RDmz13hKmf+lw0o= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1778795704; c=relaxed/simple; bh=X5Fn3ohZpoJSJjP5QpCzv40uCoD93rfUnEoLX3OoLMk=; h=From:To:Cc:Subject:Date:Message-id:In-reply-to:References: MIME-version; b=aEp1POJSn0yRSQ5W6DUt++UeH+O8aojNPG76Bnh3pzJkKV6ReHEj07724ob5kfH3bIqYwrLqQZt6Zahsr3swSDt4GKOpWmq0qBPoyUrXpGlYyg8mMpUeRn19DTU2IYvMxz2yQST/8mnAcMAEp0KGgx8YkceP2OlD7QZGJebQWh0= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=raczylo.com; spf=pass smtp.mailfrom=lhr1.rp.oracleemaildelivery.com; dkim=pass (2048-bit key) header.d=raczylo.com header.i=@raczylo.com header.b=FAhNf7As; dkim=pass (2048-bit key) header.d=lhr1.rp.oracleemaildelivery.com header.i=@lhr1.rp.oracleemaildelivery.com header.b=LYBvD2cU; arc=none smtp.client-ip=130.35.116.85 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=raczylo.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=lhr1.rp.oracleemaildelivery.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=raczylo.com header.i=@raczylo.com header.b="FAhNf7As"; dkim=pass (2048-bit key) header.d=lhr1.rp.oracleemaildelivery.com header.i=@lhr1.rp.oracleemaildelivery.com header.b="LYBvD2cU" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; s=oracle-uk-012026; d=raczylo.com; h=Date:To:From:Subject:Message-Id:MIME-Version:Sender:List-Unsubscribe:List-Unsubscribe-Post; bh=bJPU9678+y1cp2+iUvq03ogPthNxCP++IVi7e5ZAngg=; b=FAhNf7AsEpRLxEqzs17NM43ou6ZFgcGoq/5nh3DFfZ3U2w9R92X4UbPZZAm8wTcsKyRgty1X4Vid BiWYr/PJMRT20CGoyZnk444y6sUe2LgI1PVgE/AwUSbftEmjYP7//LR9DceaCjx3SUh/7GJ0eSA5 7lVA3bnIK0xUeWPumpT3sljvk6/0qV7Mt1Vo8TGJ1ENi0Ehe+b5tIUuNiOZHI+U341Xm2XDQQliV BN6KidUduCJnTDATOCUgMH5K6o6cdWDcWjivJQnuSJ1D32X/4pBJGeeswQQ4/0a2untt+MkBMNzb Mbt0+/atYK6h03UVdq9vliZFm6oZWo8Pa4o1yw== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; s=prod-lhr-20191104; d=lhr1.rp.oracleemaildelivery.com; h=Date:To:From:Subject:Message-Id:MIME-Version:Sender:List-Unsubscribe:List-Unsubscribe-Post; bh=bJPU9678+y1cp2+iUvq03ogPthNxCP++IVi7e5ZAngg=; b=LYBvD2cUU08weNhGcTkSsIalu60bz7znFBJNsqL2NbJXw+DepCGshZefbg13tf5YEPv8Z0WBUujL 1cr+EfjIjZ6vUxM82VEGHY+vWpoAJh/k3nJz2lQp/0LuOjm7URfAJ/YxlNn2wgVALbg6i/dFNOoy 7J/R5jKY77V/iSpzdV/dC2G6OeVQDmLmxxadMwxpX//btCmadEDlX2xsvek+lt5MiYIluBBUUGjk qC+JJ4PcQtyzibMvV+UVKJELajz9f4LTadtY8zjuCk3FS+r4JZbD9oaOzzz1FURxDCDkwHCZoSXy fboI1PUX3x6f95Zp1BCfTI+LRJaym6y9DB1pAw== Received: by omta-ad2-fd2-1402-uk-london-1.omtaad2.vcndplhr.oraclevcn.com (Oracle Communications Messaging Server 8.1.0.1.20260212 64bit (built Feb 12 2026)) with ESMTPS id <0TF10888CRJO6EE0@omta-ad2-fd2-1402-uk-london-1.omtaad2.vcndplhr.oraclevcn.com> for linux-kernel@vger.kernel.org; Thu, 14 May 2026 21:55:00 +0000 (GMT) List-Unsubscribe-Post: List-Unsubscribe=One-Click From: Lukasz Raczylo To: netdev@vger.kernel.org Cc: Theo Lebrun , Andrea della Porta , Nicolas Ferre , Claudiu Beznea , Andrew Lunn , "David S . Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-rpi-kernel@lists.infradead.org Subject: [PATCH net-next v2 1/3] net: macb: flush PCIe posted write after TSTART doorbell (PCIe-only) Date: Thu, 14 May 2026 22:54:57 +0100 Message-id: <20260514215459.36109-2-lukasz@raczylo.com> X-Mailer: git-send-email 2.54.0 In-reply-to: <20260514215459.36109-1-lukasz@raczylo.com> References: <20260514215459.36109-1-lukasz@raczylo.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-version: 1.0 Content-transfer-encoding: 8bit Reporting-Meta: AAHf+Z0YX8zBMsksb6gSGL9ZV3VCbo9am2PhbSO1iipMaVnbSZPP4U7YWpbXzRHK g669lhZ1JQctwueluqlxYAuNG1sS6Qw9c23Nylz9ZwRKP9DN1xZQVrBXs/PRmKX+ 57Z7+3KVfPvBEMqMElvOdde04oGSQqxDaKC4ztkUEAkKkbwynlh3GDzNrGVMNpSD FW969w2GE7LNbzbXzrC6Kt3OFZB9eoVCgyJBjTXXp7zVJO5P9AfY60xnjCz+dW4n nnfxxY7Cwr0jtbyDlpVOGzwhDNe4LrnKVQSGw45Wwq8TjUoY+sn5PbJMfGBP61zR u06JCLNJx0ldWK1V/bnySSzNJm67gUxyoq9gGnoLzMl9hCpzIT2ZTll4+tphfxWi xAa6px+mXfFVR68FBaZ2Fbt3kzpcdzeNA6LrqPunMkKkBLYLTOdmvi43BPn5yUNu ZQ== macb_start_xmit() and macb_tx_restart() kick transmission by OR-ing MACB_BIT(TSTART) into NCR. On PCIe-attached macb instances (BCM2712 + RP1 PCIe south bridge on Raspberry Pi 5 is the case I have in front of me), writes to NCR are posted PCIe writes: they are not guaranteed to reach the device before the issuing CPU returns. If the TSTART doorbell does not reach the MAC, no TX begins, no TCOMP completion arrives, and the ring remains quiescent without any kernel-visible indication. Add a read-back of NCR after each TSTART write. The read is an architected PCIe read barrier for earlier posted writes on the same path; it ensures the doorbell has reached the MAC before the function returns. As a side effect on macb_start_xmit() it also flushes the preceding macb_tx_lpi_wake() NCR write -- not just TSTART -- since the barrier applies to all prior posted writes by the same requester. The cost is one non-posted PCIe read per TSTART. To avoid imposing this on SoC-integrated macb variants (Atmel, Microchip, SiFive, Xilinx), where NCR is on-chip MMIO and no fabric posted-write concern exists, gate the readback behind a new MACB_CAPS_PCIE_POSTED_WRITES capability set only on raspberrypi_rp1_config. Note that the raspberrypi/linux vendor fork carries a local patch around the TSTART site (a queue->tx_pending breadcrumb that is promoted to queue->txubr_pending by the next TCOMP interrupt, triggering macb_tx_restart()). That workaround makes the loss recoverable under traffic, but it cannot help if TCOMP itself is not raised because no TX started -- which is exactly the case targeted here. The handshake is not present in mainline. Link: https://github.com/cilium/cilium/issues/43198 Link: https://bugs.launchpad.net/ubuntu/+source/linux-raspi/+bug/2133877 Signed-off-by: Lukasz Raczylo --- drivers/net/ethernet/cadence/macb.h | 4 ++++ drivers/net/ethernet/cadence/macb_main.c | 15 +++++++++++++++ 2 files changed, 19 insertions(+) diff --git a/drivers/net/ethernet/cadence/macb.h b/drivers/net/ethernet/cadence/macb.h index 2de56017e..ce9037f9e 100644 --- a/drivers/net/ethernet/cadence/macb.h +++ b/drivers/net/ethernet/cadence/macb.h @@ -791,6 +791,10 @@ #define MACB_CAPS_USRIO_HAS_MII BIT(26) #define MACB_CAPS_USRIO_HAS_REFCLK_SOURCE BIT(27) #define MACB_CAPS_USRIO_HAS_TSUCLK_SOURCE BIT(28) +/* Register writes are posted on the parent fabric and need a non-posted + * read-back to guarantee delivery. Currently set only on RP1. + */ +#define MACB_CAPS_PCIE_POSTED_WRITES BIT(29) /* LSO settings */ #define MACB_LSO_UFO_ENABLE 0x01 diff --git a/drivers/net/ethernet/cadence/macb_main.c b/drivers/net/ethernet/cadence/macb_main.c index a12aa2124..6879f3458 100644 --- a/drivers/net/ethernet/cadence/macb_main.c +++ b/drivers/net/ethernet/cadence/macb_main.c @@ -1922,6 +1922,14 @@ static void macb_tx_restart(struct macb_queue *queue) spin_lock(&bp->lock); macb_writel(bp, NCR, macb_readl(bp, NCR) | MACB_BIT(TSTART)); + /* + * On PCIe-attached parts, flush the posted-write queue so the + * TSTART doorbell reliably reaches the MAC. Without this the + * write can sit in the fabric and the MAC never advances, + * causing a silent TX stall. + */ + if (bp->caps & MACB_CAPS_PCIE_POSTED_WRITES) + (void)macb_readl(bp, NCR); spin_unlock(&bp->lock); out_tx_ptr_unlock: @@ -2560,6 +2568,12 @@ static netdev_tx_t macb_start_xmit(struct sk_buff *skb, struct net_device *dev) spin_lock(&bp->lock); macb_tx_lpi_wake(bp); macb_writel(bp, NCR, macb_readl(bp, NCR) | MACB_BIT(TSTART)); + /* + * Flush PCIe posted-write queue; see comment in macb_tx_restart(). + * Also flushes the preceding macb_tx_lpi_wake() NCR write. + */ + if (bp->caps & MACB_CAPS_PCIE_POSTED_WRITES) + (void)macb_readl(bp, NCR); spin_unlock(&bp->lock); if (CIRC_SPACE(queue->tx_head, queue->tx_tail, bp->tx_ring_size) < 1) @@ -5674,6 +5688,7 @@ static const struct macb_config raspberrypi_rp1_config = { .caps = MACB_CAPS_GIGABIT_MODE_AVAILABLE | MACB_CAPS_CLK_HW_CHG | MACB_CAPS_JUMBO | MACB_CAPS_GEM_HAS_PTP | + MACB_CAPS_PCIE_POSTED_WRITES | MACB_CAPS_EEE | MACB_CAPS_USRIO_HAS_MII, .dma_burst_length = 16, -- 2.54.0