From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-wm1-f46.google.com (mail-wm1-f46.google.com [209.85.128.46]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 82BB42E0B5F for ; Thu, 23 Oct 2025 07:55:45 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.46 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1761206148; cv=none; b=WNKrwJT5wZ21KqtAOLSPXnV4Ujrv0zW9ELfVVNcCTw59/0HbwTcKxTug/8LkjeccSMSknTasiP9jwaZR3pAFkglAkiklNgQQNIF3wGtCPmdUnIjJHbwZtEsykqKorMsmAWcW9olrRM1yhmb+Luejnvv2FWv5cMXqsKLXJvWa5TQ= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1761206148; c=relaxed/simple; bh=2mW2ufPwK26AhdRx9EYoLmILverODcdS8Sg1ER/tufo=; h=From:To:Cc:Subject:In-Reply-To:References:Date:Message-ID: MIME-Version:Content-Type; b=AOyY737s++9e8PkIrqhVtPod2Dge+VAw97QZ4QSQgqDHB4qeV6OCxBEbVr+UAa9hhtzmcm+uGaJqYVfnCWIpAq71F4wBSIPdo1eC0odmoXzwa/7GIH/nBfyyjbhpnYl+N8xDhyrAh+axRMJMe6ztEWbwJOfMEqpJ1NL30DZND2M= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=baylibre.com; spf=pass smtp.mailfrom=baylibre.com; dkim=pass (2048-bit key) header.d=baylibre-com.20230601.gappssmtp.com header.i=@baylibre-com.20230601.gappssmtp.com header.b=ds3iu9Mi; arc=none smtp.client-ip=209.85.128.46 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=baylibre.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=baylibre.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=baylibre-com.20230601.gappssmtp.com header.i=@baylibre-com.20230601.gappssmtp.com header.b="ds3iu9Mi" Received: by mail-wm1-f46.google.com with SMTP id 5b1f17b1804b1-47112a73785so3018035e9.3 for ; Thu, 23 Oct 2025 00:55:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=baylibre-com.20230601.gappssmtp.com; s=20230601; t=1761206144; x=1761810944; darn=vger.kernel.org; h=mime-version:message-id:date:user-agent:references:in-reply-to :subject:cc:to:from:from:to:cc:subject:date:message-id:reply-to; bh=yP7Zvul1aJ+pzBJTLArL0ELIUpNFf5csxKan8A8VPjY=; b=ds3iu9MiwPN8FWrUAyOoTJAWq5Z3+cMg4KukmyjXLcQ1z2DYr1YYWunxfQuqGXEAvo loaHrqBDy2ZH7YlhZsClxrnLOT2Ixdiwg3EuilwcqJHiXbsz4chRbN4qOiJrFWnPjCCE glgL5OdYy0FPoHsgtmH18nKAKHr/xYjXGiGNdinca8fFvMPWniFuup13qlZOKnibplfn J2+si7gNzmW+U2dIb1n0wqnQqIJXuMFUqsgrp8kwz14DMgpPw6SJmhnqo4kI+f1Vrzsq a9AwCy/uiuZaGfPVPgin90hHzOXJ1FcZvAdWZbCrlg92m2kucqSygBzaIu6FuMk/t6p+ oaOg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1761206144; x=1761810944; h=mime-version:message-id:date:user-agent:references:in-reply-to :subject:cc:to:from:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=yP7Zvul1aJ+pzBJTLArL0ELIUpNFf5csxKan8A8VPjY=; b=AAh2kHcHgd4AlU8TwZ/opkvOtwXuAeKn7NILStreDgvgRJ3szEnHL24SHSq4u0BXa+ V/44UBIBb43SiSifJ0sAet92oMoZwoxWSog/sRHKhCt4gN/Hl/EGB0jGDVEUIo066+IZ 4+wjQdWVRPwcDe0nD/iuDdOwDhelIloF6iUxKwyhpTGgjww48BTYU/HSsehH7HuGg/Gh Xdlvtcga94Dsxno1AbR27c6CmZ9jda3YF1Jb46VPixnCVN+9wGu6ijeddvdjw30rqbNd +SFxv3+umUYR8SgODrVpn12mWRkFDUevTf3ZKJJrPQE3YC4zNbZ++vufN62foyaffuBQ SjwA== X-Forwarded-Encrypted: i=1; AJvYcCV6QOS3PDABimy0MgvRcwTQ/4swq/9gKcBn0v8NX2Fb4QCl2fZs71GzNWaSGKY0KvFuL73GXnLiZB1UB0w=@vger.kernel.org X-Gm-Message-State: AOJu0Yzi9xEh+vfFZ+ZcGrLhHlrRaaApxuL1VlCk909Z136SbGndQuH0 YxAsJcCfkEmyb7M1zAZNU60DbimzyRlWRN7mfKNiQUQLcpK6KVvz92WvRXPBz/sekwk= X-Gm-Gg: ASbGncvIVLMa/3X1gk3AbplIKeze75sglGX+xYQ8tROm+JoSS8XVZr+e7XTAG9HbB2x bOs9KNrpnpaAO3qTm3kI4gtgTuMvmr0TGvi8R0q4hIUJjljTq4FFA+B1z22kvFvjoCOcT22hSn8 6pQ38h0gGKZAl0GocF76+XrS1NqhN5szFJZ4JeolpYvLYwhHCisGn+YJGX2p1OJE1Eq9OtfsX3a b8Lc3zKISCdCsvwR7bVX8q9El9A3ixCwFecGBfoL7EyT17k0WXRfE+C6i3w0v/zHoppP6I4kLs8 Q8cAydv68V2UOWi6iV7oYt0Ar4Za/2ebyXaP1n2QT5Tknx/qafbAlnPpjI0lY3YgOFhIQAo1x+6 5boNlEqqoeOFVNTcciWxqEJuUJ9g9tftgkCh6m6sicw0MXm9T01Ltoadc5JPq+6sMdIfzB+Fq2W L6HWuIDEuY X-Google-Smtp-Source: AGHT+IHfejwjiA0rmTrIPIQL9pQttVcomeiavvXgPCYnLp3S1n4oYYBb/LewOnrxM9mg10WKWzhDig== X-Received: by 2002:a05:600c:3e07:b0:471:13fc:e356 with SMTP id 5b1f17b1804b1-471178760f8mr174411675e9.3.1761206143582; Thu, 23 Oct 2025 00:55:43 -0700 (PDT) Received: from localhost ([2a01:e0a:3c5:5fb1:6aef:b8f:aa92:c859]) by smtp.gmail.com with UTF8SMTPSA id 5b1f17b1804b1-475c4342946sm81833535e9.10.2025.10.23.00.55.43 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 23 Oct 2025 00:55:43 -0700 (PDT) From: Jerome Brunet To: Koichiro Den Cc: ntb@lists.linux.dev, linux-pci@vger.kernel.org, dmaengine@vger.kernel.org, linux-kernel@vger.kernel.org, mani@kernel.org, kwilczynski@kernel.org, kishon@kernel.org, bhelgaas@google.com, corbet@lwn.net, vkoul@kernel.org, jdmason@kudzu.us, dave.jiang@intel.com, allenbh@gmail.com, Basavaraj.Natikar@amd.com, Shyam-sundar.S-k@amd.com, kurt.schwemmer@microsemi.com, logang@deltatee.com, jingoohan1@gmail.com, lpieralisi@kernel.org, robh@kernel.org, Frank.Li@nxp.com, fancer.lancer@gmail.com, arnd@arndb.de, pstanner@redhat.com, elfring@users.sourceforge.net Subject: Re: [RFC PATCH 00/25] NTB/PCI: Add DW eDMA intr fallback and BAR MW offsets In-Reply-To: <20251023071916.901355-1-den@valinux.co.jp> (Koichiro Den's message of "Thu, 23 Oct 2025 16:18:51 +0900") References: <20251023071916.901355-1-den@valinux.co.jp> User-Agent: mu4e 1.12.9; emacs 30.1 Date: Thu, 23 Oct 2025 09:55:42 +0200 Message-ID: <1jqzuu2gsh.fsf@starbuckisacylon.baylibre.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain On Thu 23 Oct 2025 at 16:18, Koichiro Den wrote: > Hi all, > > Motivation > ========== > > On Renesas R-Car S4 the PCIe Endpoint is DesignWare-based and the platform > does not allow mapping GITS_TRANSLATER as an inbound iATU target. As a > result, forwarding MSI writes from the Root Complex (RC) to the Endpoint > (EP) is not possible even if we would add implementation to create a MSI > domain for the vNTB device to use existing drivers/ntb/msi.c, and NTB > traffic must fall back to doorbells (polling). In addition, BAR resources > are scarce, which makes it difficult to dedicate a BAR solely to an > NTB/msi window. > > This RFC introduces a generic interrupt backend for NTB. The existing MSI > path is converted to a backend, and a new DW eDMA test-interrupt backend > provides an RC-to-EP interrupt fallback when MSI cannot be used. In > parallel, EPC/DWC gains inbound subrange mapping so multiple NTB memory > windows (MWs) can share a single BAR at arbitrary offsets (via mwN_offset). > The vNTB EPF and ntb_transport are taught about offsets. > > Backend selection is automatic: if MSI is available we use the MSI backend. > Otherwise, if enabled, the DW eDMA backend is used. If neither is > available, we continue to use doorbells. Existing systems remain unaffected > unless use_intr=1 is set. > > Example layout (R-Car S4): > > BAR0: Config/Spad > BAR2 [0x00000-0xF0000]: MW1 (data) > BAR2 [0xF0000-0xF8000]: MW2 (interrupts) > BAR4: Doorbell Have you considered putting the doorbell in BAR0 along Config/SPAD instead ? Doorbells already have an offset in the config and it would allow the following setup BAR0 : Config/Spad/Doorbell BAR2 : MW1 BAR4 : MW2 If MW2 handle the IRQs, I suppose the size requirement is rather limited so it should fit ? The modification to allow this setup is minimal and you would not need all the offset related changes below ... This is something I was experimenting on. I can share that if you are interested. > > # The corresponding configfs settings (see Patch #25): > echo 0xF0000 > ./mw1 > echo 0x8000 > ./mw2 > echo 0xF0000 > ./mw2_offset > echo 2 > ./mw1_bar > echo 2 > ./mw2_bar > > Summary of changes > ================== > > * NTB core/transport > - Introduce struct ntb_intr_backend and convert MSI to the new backend. > - Add DW eDMA interrupt backend (CONFIG_NTB_DW_EDMA) as MSI-less fallback. > - Rename module parameter to use_intr (keep use_msi as deprecated alias). > - Support offsetted partial MWs in ntb_transport. > - Hardening for peer-reported interrupt values and minor cleanups. > > * PCI Endpoint core and DWC EP controller > - Add EPC ops map_inbound()/unmap_inbound() for BAR subrange mapping. > - Implement inbound mapping for DesignWare EP (Address Match mode), with > tracking of multiple inbound iATU entries per BAR and proper teardown. > > * EPF vNTB > - Add mwN_offset configfs attributes and propagate offsets to inbound maps. ... then you would not need this with and it would remove significant part of the necessary changes below > - Prefer pci_epc_map_inbound() when supported. Otherwise fall back to > set_bar(). > - Provide .get_pci_epc() so backends can locate the common eDMA instance. > > * DW eDMA > - Add self-interrupt registration and expose test-IRQ register offsets. > - Provide dw_edma_find_by_child(). > > * Renesas R-Car > - Place MW2 in BAR2 to host the interrupt window alongside the data MW. > > * Documentation > > Patch layout > ============ > > * Patches 01-11 : BAR subrange and MW offsets (EPC/DWC EP, vNTB, core helpers) > * Patches 12-14 : Interrupt handling hardening in ntb_transport/MSI > * Patches 15-17 : DW eDMA: self-IRQ API, offsets, lookup helper > * Patches 18-19 : NTB/EPF glue (.get_pci_epc()) > * Patch 20 : Module param name change (use_msi->use_intr, alias preserved) > * Patches 21-23 : Generic interrupt backend + MSI conversion + DW eDMA backend > * Patch 24 : R-Car: add MW2 in BAR2 for interrupts > * Patch 25 : Documentation updates > > Tested on > ========= > > * Renesas R-Car S4 Spider > * Kernel base: commit 68113d260674 ("NTB/msi: Remove unused functions") (ntb-driver-core/ntb-next) > > Performance measurement > ======================= > > Even without the DMA acceleration patches for R-Car S4 (which I keep > separate from this RFC patch series), enabling RC-to-EP interrupts > dramatically improves NTB latency on R-Car S4: > > * Before this patch series (NB. use_msi doesn't work on R-Car S4) > > # Server: sockperf server -i 0.0.0.0 > # Client: sockperf ping-pong -i $SERVER_IP > ========= Printing statistics for Server No: 0 > [Valid Duration] RunTime=0.540 sec; SentMessages=45; ReceivedMessages=45 > ====> avg-latency=5995.680 (std-dev=70.258, mean-ad=57.478, median-ad=85.978,\ > siqr=59.698, cv=0.012, std-error=10.473, 99.0% ci=[5968.702, 6022.658]) > # dropped messages = 0; # duplicated messages = 0; # out-of-order messages = 0 > Summary: Latency is 5995.680 usec > Total 45 observations; each percentile contains 0.45 observations > ---> observation = 6121.137 > ---> percentile 99.999 = 6121.137 > ---> percentile 99.990 = 6121.137 > ---> percentile 99.900 = 6121.137 > ---> percentile 99.000 = 6121.137 > ---> percentile 90.000 = 6099.178 > ---> percentile 75.000 = 6054.418 > ---> percentile 50.000 = 5993.040 > ---> percentile 25.000 = 5935.021 > ---> observation = 5883.362 > > * With this series (use_intr=1) > > # Server: sockperf server -i 0.0.0.0 > # Client: sockperf ping-pong -i $SERVER_IP > ========= Printing statistics for Server No: 0 > [Valid Duration] RunTime=0.550 sec; SentMessages=2145; ReceivedMessages=2145 > ====> avg-latency=127.677 (std-dev=21.719, mean-ad=11.759, median-ad=3.779,\ > siqr=2.699, cv=0.170, std-error=0.469, 99.0% ci=[126.469, 128.885]) > # dropped messages = 0; # duplicated messages = 0; # out-of-order messages = 0 > Summary: Latency is 127.677 usec > Total 2145 observations; each percentile contains 21.45 observations > ---> observation = 446.691 > ---> percentile 99.999 = 446.691 > ---> percentile 99.990 = 446.691 > ---> percentile 99.900 = 291.234 > ---> percentile 99.000 = 221.515 > ---> percentile 90.000 = 149.277 > ---> percentile 75.000 = 124.497 > ---> percentile 50.000 = 121.137 > ---> percentile 25.000 = 119.037 > ---> observation = 113.637 > > Feedback welcome on both the approach and the splitting/routing preference. > > (The series spans NTB, PCI EP/DWC and dmaengine/dw-edma. I'm happy to split > later if preferred.) > > Thanks for reviewing. > > > Koichiro Den (25): > PCI: endpoint: pci-epf-vntb: Use array_index_nospec() on mws_size[] > access > PCI: endpoint: pci-epf-vntb: Add mwN_offset configfs attributes > NTB: epf: Handle mwN_offset for inbound MW regions > PCI: endpoint: Add inbound mapping ops to EPC core > PCI: dwc: ep: Implement EPC inbound mapping support > PCI: endpoint: pci-epf-vntb: Use pci_epc_map_inbound() for MW mapping > NTB: Add offset parameter to MW translation APIs > PCI: endpoint: pci-epf-vntb: Propagate MW offset from configfs when > present > NTB: ntb_transport: Support offsetted partial memory windows > NTB/msi: Support offsetted partial memory window for MSI > NTB/msi: Do not force MW to its maximum possible size > NTB: ntb_transport: Stricter checks for peer-reported interrupt values > NTB/msi: Skip mw_set_trans() if already configured > NTB/msi: Add a inner loop for PCI-MSI cases > dmaengine: dw-edma: Add self-interrupt registration API > dmaengine: dw-edma: Expose self-IRQ register offsets > dmaengine: dw-edma: Add dw_edma_find_by_child() helper > NTB: core: Add .get_pci_epc() to ntb_dev_ops > NTB: epf: vntb: Implement .get_pci_epc() callback > NTB: ntb_transport: Rename use_msi to use_intr (keep alias) > NTB: Introduce generic interrupt backend abstraction and convert MSI > NTB: ntb_transport: Rename MSI symbols to generic interrupt form > NTB: intr_dw_edma: Add DW eDMA emulated interrupt backend > NTB: epf: Add MW2 for interrupt use on Renesas R-Car > Documentation: PCI: endpoint: pci-epf-vntb: Update and add mwN_offset > usage > > Documentation/PCI/endpoint/pci-vntb-howto.rst | 16 +- > drivers/dma/dw-edma/dw-edma-core.c | 109 ++++++++ > drivers/dma/dw-edma/dw-edma-core.h | 18 ++ > drivers/dma/dw-edma/dw-edma-v0-core.c | 15 ++ > drivers/ntb/Kconfig | 15 ++ > drivers/ntb/Makefile | 6 +- > drivers/ntb/hw/amd/ntb_hw_amd.c | 6 +- > drivers/ntb/hw/epf/ntb_hw_epf.c | 46 ++-- > drivers/ntb/hw/idt/ntb_hw_idt.c | 3 +- > drivers/ntb/hw/intel/ntb_hw_gen1.c | 6 +- > drivers/ntb/hw/intel/ntb_hw_gen1.h | 2 +- > drivers/ntb/hw/intel/ntb_hw_gen3.c | 3 +- > drivers/ntb/hw/intel/ntb_hw_gen4.c | 6 +- > drivers/ntb/hw/mscc/ntb_hw_switchtec.c | 6 +- > drivers/ntb/intr_common.c | 61 +++++ > drivers/ntb/intr_dw_edma.c | 253 ++++++++++++++++++ > drivers/ntb/msi.c | 186 +++++++------ > drivers/ntb/ntb_transport.c | 155 ++++++----- > drivers/ntb/test/ntb_msi_test.c | 26 +- > drivers/ntb/test/ntb_perf.c | 4 +- > drivers/ntb/test/ntb_tool.c | 6 +- > .../pci/controller/dwc/pcie-designware-ep.c | 242 +++++++++++++++-- > drivers/pci/controller/dwc/pcie-designware.c | 1 + > drivers/pci/controller/dwc/pcie-designware.h | 2 + > drivers/pci/endpoint/functions/pci-epf-vntb.c | 197 ++++++++++++-- > drivers/pci/endpoint/pci-epc-core.c | 44 +++ > include/linux/dma/edma.h | 31 +++ > include/linux/ntb.h | 134 +++++++--- > include/linux/pci-epc.h | 11 + > 29 files changed, 1310 insertions(+), 300 deletions(-) > create mode 100644 drivers/ntb/intr_common.c > create mode 100644 drivers/ntb/intr_dw_edma.c -- Jerome