From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-wr1-f41.google.com (mail-wr1-f41.google.com [209.85.221.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id BD203242928 for ; Tue, 1 Jul 2025 07:37:04 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.221.41 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1751355427; cv=none; b=agLxfP6g4HMIMXD9PN81lhKYIOBSvvy8455DH+b7eE/D6JF5AqEsVEpjDb7+z0OPgQ7HZUElU6uby8jsteuHqZAJ0T0K7K/60SftVQ7wlwt45Stu/jKxvFUnELKZjcZ4aEncq6khrYIAxJHJRhBk2Nj5mwehz2ttoIFHg69IOG8= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1751355427; c=relaxed/simple; bh=cCndqnCEV0L2x9DQeAD8r7njlV2XxiUC+AhkoZAasQc=; h=From:To:Cc:Subject:In-Reply-To:References:Date:Message-ID: MIME-Version:Content-Type; b=Wz4uFUO41E0jxFEkk8FxH1/BpEHIn/4nT4yJNF39jn3viwUaHrwpz3vq5H0qvvej1JKhURdhIMiLkqxXYX0J/f1KUw+IR1rYX5pkG0kmyds0h7hKjLeCGXgQAbVyxBTVPXLae791x9PtZYqKQdBgymRRf3Wf4tOFLzEBNVRPqEQ= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=baylibre.com; spf=pass smtp.mailfrom=baylibre.com; dkim=pass (2048-bit key) header.d=baylibre-com.20230601.gappssmtp.com header.i=@baylibre-com.20230601.gappssmtp.com header.b=btcweVMR; arc=none smtp.client-ip=209.85.221.41 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=baylibre.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=baylibre.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=baylibre-com.20230601.gappssmtp.com header.i=@baylibre-com.20230601.gappssmtp.com header.b="btcweVMR" Received: by mail-wr1-f41.google.com with SMTP id ffacd0b85a97d-3a522224582so2738141f8f.3 for ; Tue, 01 Jul 2025 00:37:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=baylibre-com.20230601.gappssmtp.com; s=20230601; t=1751355423; x=1751960223; darn=vger.kernel.org; h=mime-version:message-id:date:user-agent:references:in-reply-to :subject:cc:to:from:from:to:cc:subject:date:message-id:reply-to; bh=84nXSAp/Z1PnLoZAVeblOFLGSiCihYkJz9aJ4vUdZ0A=; b=btcweVMRO27TmOsgGr9C54wQCsAJZoYuF/GBsZmdd14UOkzvSl7yBBjAYsxoVbO51W VEUDnbU933Hlcahiw01+YgMf/n21GBI8rPvl5Q8yf2UMQ7Cq7Twyf0YYGB25XAEx9XtZ pE0jKja2EK5eeSPYLfd8JZneSbrUswcJINkSMxpn6DrZvqaYK0CmgwJzenm06CvAsjJe 4T4aHzNbw0XfaGG2AgJNV2EF62K1C22lc5mgTk/1+8HBBStxa+I8+Z0e/7dGWu5QEUG2 vENF+yUWzSBdQLqblyFiKHl4qWnxuEbvyDATX/zGIKC7SqIKK7dOeghwsO0ImQ4JB6dS JdWA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1751355423; x=1751960223; h=mime-version:message-id:date:user-agent:references:in-reply-to :subject:cc:to:from:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=84nXSAp/Z1PnLoZAVeblOFLGSiCihYkJz9aJ4vUdZ0A=; b=t0jKAhZUfPDIlbh6m8e52uKPnc/k7eHHZvY6s4xcEWx1zWCRr46eXOAbVvRLFPydu9 GOhqQ79WdftztttSGjyv8ciBCS2DS/9nehcPRuqN0GA4OAsgqvvQfqhKZUGim6krW4+X MYh0fP07x1bHKnDoVkOLo+PV4XoQbLeuA3jBISHo+52A1qaOE1UeOYTEOWqyVf0eJfHh kVgSUqVK7+RnFUdSFkHmMamyw83MpOSId84IlP044w0rmf98RavXcbcwPJbMDyvZGEKl Cgh+jSWdW3U/buQON+tBgiWja/lLCGJSh1CepJniMslFADt3mAR6zibnTgtoF+9/5Lf+ VJ0g== X-Forwarded-Encrypted: i=1; AJvYcCW111dTEU9eXAD2xbe8X8UBt/fXFSRHf9k3tWEofBzyd9yxAqKKYWYTOOZFC8ueL36hSiKUPk3uw1Vmc40=@vger.kernel.org X-Gm-Message-State: AOJu0YxrMl+qQ0rXlXqPFappeAYu0OEZoL5lW+9caw+DkO4VOxIpcg4B qhc5Xc1RygjGQ0E5j+kqBYXF/7uo5A1M/VM5Db8UhZ+Yh12eMLgR6LZKQRjyEXg5rgU= X-Gm-Gg: ASbGnctISw+sF/ftz95mkkOu2GUcdxHFSxIUtpV7OTpJcpGw6aIpLSTtynlFF9Sz4rB Mprzelb0TW19dt1b4NWcbyBGmzBE2cwb1cTMrAMI0Cg9+prT5ej2mRtw0xr2kjp3zr8qF7hdXHj qj/6DNlvchv3awL1fmYh7RfZSLhEqYvz0k2bGDmBhMy3kyI4aLlKM2uABavb2eq8UVZyPAVbV7P mFJ0/kCgwOcgvd1lQ18sKUUhKoNA+/tDIcjxIlwOfdTExhZ2ENn6vekeOg1cLeCnSnL0lZ6A5nF c0FGXDXk74Jq0dkILilUZY36Wn08pGp11Kp52RDsamY8VpiP1vY9RmMaZluWjg== X-Google-Smtp-Source: AGHT+IFjJP+l9Owhcenlw9SLs9bvnjDG/zUDl7SGdhqFX4q/YS2+JYr3fDMHgOByMEELdFSvV8KjEQ== X-Received: by 2002:a05:6000:4009:b0:3a8:2f65:373f with SMTP id ffacd0b85a97d-3aa9fc2002dmr12597509f8f.16.1751355422716; Tue, 01 Jul 2025 00:37:02 -0700 (PDT) Received: from localhost ([2a01:e0a:3c5:5fb1:fbe8:19b9:f06d:d64c]) by smtp.gmail.com with UTF8SMTPSA id 5b1f17b1804b1-4538a3fe0efsm154899435e9.24.2025.07.01.00.37.02 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 01 Jul 2025 00:37:02 -0700 (PDT) From: Jerome Brunet To: Bjorn Helgaas Cc: Jon Mason , Dave Jiang , Allen Hubbe , Manivannan Sadhasivam , Kishon Vijay Abraham I , Bjorn Helgaas , Frank Li , Krzysztof =?utf-8?Q?Wilczy=C5=84ski?= , ntb@lists.linux.dev, linux-pci@vger.kernel.org, linux-kernel@vger.kernel.org Subject: Re: [PATCH v2 3/3] PCI: endpoint: pci-epf-vntb: Allow BAR assignment via configfs In-Reply-To: <20250630203842.GA1800194@bhelgaas> (Bjorn Helgaas's message of "Mon, 30 Jun 2025 15:38:42 -0500") References: <20250630203842.GA1800194@bhelgaas> User-Agent: mu4e 1.12.9; emacs 30.1 Date: Tue, 01 Jul 2025 09:37:01 +0200 Message-ID: <1jikkce50i.fsf@starbuckisacylon.baylibre.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain On Mon 30 Jun 2025 at 15:38, Bjorn Helgaas wrote: > On Tue, Jun 03, 2025 at 07:03:40PM +0200, Jerome Brunet wrote: >> The current BAR configuration for the PCI vNTB endpoint function allocates >> BARs in order, which lacks flexibility and does not account for >> platform-specific quirks. This is problematic on Renesas platforms, where >> BAR_4 is a fixed 256B region that ends up being used for MW1, despite being >> better suited for doorbells. >> >> Add new configfs attributes to allow users to specify arbitrary BAR >> assignments. If no configuration is provided, the driver retains its >> original behavior of sequential BAR allocation, preserving compatibility >> with existing userspace setups. >> >> This enables use cases such as assigning BAR_2 for MW1 and using the >> limited BAR_4 for doorbells on Renesas platforms. > > Is there any documentation for how to use this new feature? Indeed no. Thanks for the reminder. Section 9.8.2 is no longer relevant with that change, I'll try to add some explanation there. > >> Signed-off-by: Jerome Brunet >> --- >> drivers/pci/endpoint/functions/pci-epf-vntb.c | 131 ++++++++++++++++++++++++-- >> 1 file changed, 124 insertions(+), 7 deletions(-) >> >> diff --git a/drivers/pci/endpoint/functions/pci-epf-vntb.c b/drivers/pci/endpoint/functions/pci-epf-vntb.c >> index 2198282a80a40774047502a37f0288ca396bdb0e..7475d87659b1c70aa41b0999eabfa661f4ceed39 100644 >> --- a/drivers/pci/endpoint/functions/pci-epf-vntb.c >> +++ b/drivers/pci/endpoint/functions/pci-epf-vntb.c >> @@ -73,6 +73,8 @@ enum epf_ntb_bar { >> BAR_MW1, >> BAR_MW2, >> BAR_MW3, >> + BAR_MW4, >> + VNTB_BAR_NUM, >> }; >> >> /* >> @@ -132,7 +134,7 @@ struct epf_ntb { >> bool linkup; >> u32 spad_size; >> >> - enum pci_barno epf_ntb_bar[6]; >> + enum pci_barno epf_ntb_bar[VNTB_BAR_NUM]; >> >> struct epf_ntb_ctrl *reg; >> >> @@ -654,6 +656,62 @@ static void epf_ntb_epc_destroy(struct epf_ntb *ntb) >> pci_epc_put(ntb->epf->epc); >> } >> >> + >> +/** >> + * epf_ntb_is_bar_used() - Check if a bar is used in the ntb configuration >> + * @ntb: NTB device that facilitates communication between HOST and VHOST >> + * @barno: Checked bar number >> + * >> + * Returns: true if used, false if free. >> + */ >> +static bool epf_ntb_is_bar_used(struct epf_ntb *ntb, >> + enum pci_barno barno) >> +{ >> + int i; >> + >> + for (i = 0; i < VNTB_BAR_NUM; i++) { >> + if (ntb->epf_ntb_bar[i] == barno) >> + return true; >> + } >> + >> + return false; >> +} >> + >> +/** >> + * epf_ntb_find_bar() - Assign BAR number when no configuration is provided >> + * @epc_features: The features provided by the EPC specific to this EPF >> + * @ntb: NTB device that facilitates communication between HOST and VHOST >> + * @barno: Bar start index >> + * >> + * When the BAR configuration was not provided through the userspace >> + * configuration, automatically assign BAR as it has been historically >> + * done by this endpoint function. >> + * >> + * Returns: the BAR number found, if any. -1 otherwise >> + */ >> +static int epf_ntb_find_bar(struct epf_ntb *ntb, >> + const struct pci_epc_features *epc_features, >> + enum epf_ntb_bar bar, >> + enum pci_barno barno) >> +{ >> + while (ntb->epf_ntb_bar[bar] < 0) { >> + barno = pci_epc_get_next_free_bar(epc_features, barno); >> + if (barno < 0) >> + break; /* No more BAR available */ >> + >> + /* >> + * Verify if the BAR found is not already assigned >> + * through the provided configuration >> + */ >> + if (!epf_ntb_is_bar_used(ntb, barno)) >> + ntb->epf_ntb_bar[bar] = barno; >> + >> + barno += 1; >> + } >> + >> + return barno; >> +} >> + >> /** >> * epf_ntb_init_epc_bar() - Identify BARs to be used for each of the NTB >> * constructs (scratchpad region, doorbell, memorywindow) >> @@ -676,23 +734,21 @@ static int epf_ntb_init_epc_bar(struct epf_ntb *ntb) >> epc_features = pci_epc_get_features(ntb->epf->epc, ntb->epf->func_no, ntb->epf->vfunc_no); >> >> /* These are required BARs which are mandatory for NTB functionality */ >> - for (bar = BAR_CONFIG; bar <= BAR_MW1; bar++, barno++) { >> - barno = pci_epc_get_next_free_bar(epc_features, barno); >> + for (bar = BAR_CONFIG; bar <= BAR_MW1; bar++) { >> + barno = epf_ntb_find_bar(ntb, epc_features, bar, barno); >> if (barno < 0) { >> dev_err(dev, "Fail to get NTB function BAR\n"); >> return -EINVAL; >> } >> - ntb->epf_ntb_bar[bar] = barno; >> } >> >> /* These are optional BARs which don't impact NTB functionality */ >> - for (bar = BAR_MW1, i = 1; i < num_mws; bar++, barno++, i++) { >> - barno = pci_epc_get_next_free_bar(epc_features, barno); >> + for (bar = BAR_MW1, i = 1; i < num_mws; bar++, i++) { >> + barno = epf_ntb_find_bar(ntb, epc_features, bar, barno); >> if (barno < 0) { >> ntb->num_mws = i; >> dev_dbg(dev, "BAR not available for > MW%d\n", i + 1); >> } >> - ntb->epf_ntb_bar[bar] = barno; >> } >> >> return 0; >> @@ -860,6 +916,37 @@ static ssize_t epf_ntb_##_name##_store(struct config_item *item, \ >> return len; \ >> } >> >> +#define EPF_NTB_BAR_R(_name, _id) \ >> + static ssize_t epf_ntb_##_name##_show(struct config_item *item, \ >> + char *page) \ >> + { \ >> + struct config_group *group = to_config_group(item); \ >> + struct epf_ntb *ntb = to_epf_ntb(group); \ >> + \ >> + return sprintf(page, "%d\n", ntb->epf_ntb_bar[_id]); \ >> + } >> + >> +#define EPF_NTB_BAR_W(_name, _id) \ >> + static ssize_t epf_ntb_##_name##_store(struct config_item *item, \ >> + const char *page, size_t len) \ >> + { \ >> + struct config_group *group = to_config_group(item); \ >> + struct epf_ntb *ntb = to_epf_ntb(group); \ >> + int val; \ >> + int ret; \ >> + \ >> + ret = kstrtoint(page, 0, &val); \ >> + if (ret) \ >> + return ret; \ >> + \ >> + if (val < NO_BAR || val > BAR_5) \ >> + return -EINVAL; \ >> + \ >> + ntb->epf_ntb_bar[_id] = val; \ >> + \ >> + return len; \ >> + } >> + >> static ssize_t epf_ntb_num_mws_store(struct config_item *item, >> const char *page, size_t len) >> { >> @@ -899,6 +986,18 @@ EPF_NTB_MW_R(mw3) >> EPF_NTB_MW_W(mw3) >> EPF_NTB_MW_R(mw4) >> EPF_NTB_MW_W(mw4) >> +EPF_NTB_BAR_R(ctrl_bar, BAR_CONFIG) >> +EPF_NTB_BAR_W(ctrl_bar, BAR_CONFIG) >> +EPF_NTB_BAR_R(db_bar, BAR_DB) >> +EPF_NTB_BAR_W(db_bar, BAR_DB) >> +EPF_NTB_BAR_R(mw1_bar, BAR_MW1) >> +EPF_NTB_BAR_W(mw1_bar, BAR_MW1) >> +EPF_NTB_BAR_R(mw2_bar, BAR_MW1) >> +EPF_NTB_BAR_W(mw2_bar, BAR_MW1) >> +EPF_NTB_BAR_R(mw3_bar, BAR_MW3) >> +EPF_NTB_BAR_W(mw3_bar, BAR_MW3) >> +EPF_NTB_BAR_R(mw4_bar, BAR_MW4) >> +EPF_NTB_BAR_W(mw4_bar, BAR_MW4) >> >> CONFIGFS_ATTR(epf_ntb_, spad_count); >> CONFIGFS_ATTR(epf_ntb_, db_count); >> @@ -910,6 +1009,12 @@ CONFIGFS_ATTR(epf_ntb_, mw4); >> CONFIGFS_ATTR(epf_ntb_, vbus_number); >> CONFIGFS_ATTR(epf_ntb_, vntb_pid); >> CONFIGFS_ATTR(epf_ntb_, vntb_vid); >> +CONFIGFS_ATTR(epf_ntb_, ctrl_bar); >> +CONFIGFS_ATTR(epf_ntb_, db_bar); >> +CONFIGFS_ATTR(epf_ntb_, mw1_bar); >> +CONFIGFS_ATTR(epf_ntb_, mw2_bar); >> +CONFIGFS_ATTR(epf_ntb_, mw3_bar); >> +CONFIGFS_ATTR(epf_ntb_, mw4_bar); >> >> static struct configfs_attribute *epf_ntb_attrs[] = { >> &epf_ntb_attr_spad_count, >> @@ -922,6 +1027,12 @@ static struct configfs_attribute *epf_ntb_attrs[] = { >> &epf_ntb_attr_vbus_number, >> &epf_ntb_attr_vntb_pid, >> &epf_ntb_attr_vntb_vid, >> + &epf_ntb_attr_ctrl_bar, >> + &epf_ntb_attr_db_bar, >> + &epf_ntb_attr_mw1_bar, >> + &epf_ntb_attr_mw2_bar, >> + &epf_ntb_attr_mw3_bar, >> + &epf_ntb_attr_mw4_bar, >> NULL, >> }; >> >> @@ -1379,6 +1490,7 @@ static int epf_ntb_probe(struct pci_epf *epf, >> { >> struct epf_ntb *ntb; >> struct device *dev; >> + int i; >> >> dev = &epf->dev; >> >> @@ -1389,6 +1501,11 @@ static int epf_ntb_probe(struct pci_epf *epf, >> epf->header = &epf_ntb_header; >> ntb->epf = epf; >> ntb->vbus_number = 0xff; >> + >> + /* Initially, no bar is assigned */ >> + for (i = 0; i < VNTB_BAR_NUM; i++) >> + ntb->epf_ntb_bar[i] = NO_BAR; >> + >> epf_set_drvdata(epf, ntb); >> >> dev_info(dev, "pci-ep epf driver loaded\n"); >> >> -- >> 2.47.2 >> -- Jerome