From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mx0b-001b2d01.pphosted.com (mx0b-001b2d01.pphosted.com [148.163.158.5]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0669134A3B1; Wed, 7 Jan 2026 14:26:11 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=148.163.158.5 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1767795974; cv=none; b=GmcH71OrrfUF8tbwwkWMhO/MhT49I+LbRjBBO0KcgXjbppVICi1i/Q9bleX7PAwBrONmZ3IjlQC5ZOw2r0v/XqaamRI/Spm3sNeMbgdbRUefHSed98wkEHluK125wPnzuoDTBoJF7yA/iQYrgTq2BCCksnMR+H4oZZywolU0tC0= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1767795974; c=relaxed/simple; bh=dQ4CMqeJrzP/9taAHssRr0hqqLzpP6FlnL0p8NKYkVA=; h=Message-ID:Date:MIME-Version:Subject:To:Cc:References:From: In-Reply-To:Content-Type; b=LERwORAhAkYvfiyNzFRNfqOsqG1fd+2fz1MxoX9lA2wjzWjxxtDyaGquH9qfjvgynWwHiIeHME0cTlXC9fPNrcd74fjWFlJTLQ80fowHfn64flYJLAR1WwfGKx1nqOrh5SgDox01Tp0kvDfFfOefOR+Bj1D4lD7E4VtNrUx6XQw= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.ibm.com; spf=pass smtp.mailfrom=linux.ibm.com; dkim=pass (2048-bit key) header.d=ibm.com header.i=@ibm.com header.b=OxiffGlr; arc=none smtp.client-ip=148.163.158.5 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.ibm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.ibm.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=ibm.com header.i=@ibm.com header.b="OxiffGlr" Received: from pps.filterd (m0356516.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.18.1.2/8.18.1.2) with ESMTP id 607EKEbd016642; Wed, 7 Jan 2026 14:25:46 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=cc :content-transfer-encoding:content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to; s=pp1; bh=JWyHqh 0cbDpqwkYQwJwz2YOUGHQ98e9j1ks+F+GVVmM=; b=OxiffGlr0ulcQeNaRhIB5I KJevXDrykTyGvGb8oh7u3TMqxrO8wSojZkzOeLZwQp3qy8P/icyzmxIGairK6aHT DKULFawFfxSAypdsubTAYNreOnoxaeUqG+e8ESzSrtB0wdQ7tNntotOx1/CBcFOR inlziGFIy/VairHPQE7pqCJ/TlN5v7rg/AYfEVpfLuJkxZtEnCyzFBJJMIE23IX2 +1RvEifLn1nAerEzfl5c30QHDTb2FpED7VHsz6QIMWDVARD0an9dHluO4g01LfDb JnmyPZjXAGKhNVjP1xBx7H7DawQZnMsSZIoWtAHYsoRnbuP5s5HbU66JiTHE03Bg == Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 4berhk8fb3-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Wed, 07 Jan 2026 14:25:46 +0000 (GMT) Received: from m0356516.ppops.net (m0356516.ppops.net [127.0.0.1]) by pps.reinject (8.18.1.12/8.18.0.8) with ESMTP id 607EBFoh011138; Wed, 7 Jan 2026 14:25:45 GMT Received: from ppma23.wdc07v.mail.ibm.com (5d.69.3da9.ip4.static.sl-reverse.com [169.61.105.93]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 4berhk8fb1-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Wed, 07 Jan 2026 14:25:45 +0000 (GMT) Received: from pps.filterd (ppma23.wdc07v.mail.ibm.com [127.0.0.1]) by ppma23.wdc07v.mail.ibm.com (8.18.1.2/8.18.1.2) with ESMTP id 607C09WP005264; Wed, 7 Jan 2026 14:25:45 GMT Received: from smtprelay06.dal12v.mail.ibm.com ([172.16.1.8]) by ppma23.wdc07v.mail.ibm.com (PPS) with ESMTPS id 4bfexk9e1c-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Wed, 07 Jan 2026 14:25:45 +0000 Received: from smtpav05.wdc07v.mail.ibm.com (smtpav05.wdc07v.mail.ibm.com [10.39.53.232]) by smtprelay06.dal12v.mail.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 607EPhDD30081540 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 7 Jan 2026 14:25:43 GMT Received: from smtpav05.wdc07v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 710FD58043; Wed, 7 Jan 2026 14:25:43 +0000 (GMT) Received: from smtpav05.wdc07v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 02CE058053; Wed, 7 Jan 2026 14:25:38 +0000 (GMT) Received: from [9.111.20.18] (unknown [9.111.20.18]) by smtpav05.wdc07v.mail.ibm.com (Postfix) with ESMTP; Wed, 7 Jan 2026 14:25:37 +0000 (GMT) Message-ID: <1fda7021-ec31-40ed-bfd8-e0e9b657662f@linux.ibm.com> Date: Wed, 7 Jan 2026 19:55:35 +0530 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [PATCH v2] powerpc/pseries: Fix MSI-X allocation failure when quota is exceeded To: Nam Cao , Madhavan Srinivasan , Michael Ellerman , Nicholas Piggin , Christophe Leroy , Thomas Gleixner , maz@kernel.org, gautam@linux.ibm.com, Gregory Joyce , linuxppc-dev@lists.ozlabs.org, linux-kernel@vger.kernel.org Cc: stable@vger.kernel.org References: <20260107100230.1466093-1-namcao@linutronix.de> Content-Language: en-US From: Nilay Shroff In-Reply-To: <20260107100230.1466093-1-namcao@linutronix.de> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-TM-AS-GCONF: 00 X-Authority-Analysis: v=2.4 cv=P4s3RyAu c=1 sm=1 tr=0 ts=695e6cea cx=c_pps a=3Bg1Hr4SwmMryq2xdFQyZA==:117 a=3Bg1Hr4SwmMryq2xdFQyZA==:17 a=IkcTkHD0fZMA:10 a=vUbySO9Y5rIA:10 a=VkNPw1HP01LnGYTKEx00:22 a=VwQbUJbxAAAA:8 a=VnNF1IyMAAAA:8 a=zwuvnbuwBpirbEp69l4A:9 a=QEXdDO2ut3YA:10 X-Proofpoint-ORIG-GUID: r-9MBfHGL3XoHCVF5R9Qs5_jkm7TC-BC X-Proofpoint-Spam-Details-Enc: AW1haW4tMjYwMTA3MDExMCBTYWx0ZWRfXztqz+xZmX+/C D+TIHiCmy25+0M4v9ObRPDf15giyJ4ZxQxF2Lo/xWzRf2HoUV2So33ToOIUiop3pHbks+hJM44R 4eNRNvUURtR67HSFCXcLHNvOC8GoB/liLxWcKV8tc3c09bkoDuJXSLuCfYEzuO2zufl9fzL20xU W2S12lOrQppMrKq1ah44xPu6wNOatqeq7pnWltYwjm7OFPsTVLXx/K5vVkRiilLir+lzuI8ZRJm d3kYWHZMIhnCFM0lR1H2AdCG96s2XAgfGJQa9Pwgauzf1vNhiNXxu2QbGkZ6U1KBnrZU2e22VGz WXp4B80vFsK0P97FzZYpsTYWKYeF6/pjFWzWO6AUms2zG+p8T/19gEEVsAbPMqJ7Ca496vgovMA pUEn8mZa8JLtYtG6ArS92OP0BrnY9G92ADQgXPKQ+oiduCK0Mq7/iB5m54ylIDtZEYsimjhGNFI plbWtpo7u98dS1TCVKA== X-Proofpoint-GUID: ZiQACeOSl4pDbMqbBy8nv-oApTDhYEly X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1121,Hydra:6.1.9,FMLib:17.12.100.49 definitions=2026-01-07_02,2026-01-06_01,2025-10-01_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 malwarescore=0 bulkscore=0 priorityscore=1501 clxscore=1015 suspectscore=0 phishscore=0 adultscore=0 spamscore=0 impostorscore=0 lowpriorityscore=0 classifier=typeunknown authscore=0 authtc= authcc= route=outbound adjust=0 reason=mlx scancount=1 engine=8.19.0-2512120000 definitions=main-2601070110 On 1/7/26 3:32 PM, Nam Cao wrote: > Nilay reported that since commit daaa574aba6f ("powerpc/pseries/msi: Switch > to msi_create_parent_irq_domain()"), the NVMe driver cannot enable MSI-X > when the device's MSI-X table size is larger than the firmware's MSI quota > for the device. > > This is because the commit changes how rtas_prepare_msi_irqs() is called: > > - Before, it is called when interrupts are allocated at the global > interrupt domain with nvec_in being the number of allocated interrupts. > rtas_prepare_msi_irqs() can return a positive number and the allocation > will be retried. > > - Now, it is called at the creation of per-device interrupt domain with > nvec_in being the number of interrupts that the device supports. If > rtas_prepare_msi_irqs() returns positive, domain creation just fails. > > For Nilay's NVMe driver case, rtas_prepare_msi_irqs() returns a positive > number (the quota). This causes per-device interrupt domain creation to > fail and thus the NVMe driver cannot enable MSI-X. > > Rework to make this scenario works again: > > - pseries_msi_ops_prepare() only prepares as many interrupts as the quota > permit. > > - pseries_irq_domain_alloc() fails if the device's quota is exceeded. > > Now, if the quota is exceeded, pseries_msi_ops_prepare() will only prepare > as allowed by the quota. If device drivers attempt to allocate more > interrupts than the quota permits, pseries_irq_domain_alloc() will return > an error code and msi_handle_pci_fail() will allow device drivers a retry. > > Reported-by: Nilay Shroff > Closes: https://lore.kernel.org/linuxppc-dev/6af2c4c2-97f6-4758-be33-256638ef39e5@linux.ibm.com/ > Fixes: daaa574aba6f ("powerpc/pseries/msi: Switch to msi_create_parent_irq_domain()") > Signed-off-by: Nam Cao > Acked-by: Nilay Shroff > Cc: stable@vger.kernel.org > --- > v2: > - change pseries_msi_ops_prepare()'s allocation logic to match the > original logic in __pci_enable_msix_range() > > - fix up Nilay's email address > --- > arch/powerpc/platforms/pseries/msi.c | 44 ++++++++++++++++++++++++++-- > 1 file changed, 41 insertions(+), 3 deletions(-) > > diff --git a/arch/powerpc/platforms/pseries/msi.c b/arch/powerpc/platforms/pseries/msi.c > index a82aaa786e9e..edc30cda5dbc 100644 > --- a/arch/powerpc/platforms/pseries/msi.c > +++ b/arch/powerpc/platforms/pseries/msi.c > @@ -19,6 +19,11 @@ > > #include "pseries.h" > > +struct pseries_msi_device { > + unsigned int msi_quota; > + unsigned int msi_used; > +}; > + > static int query_token, change_token; > > #define RTAS_QUERY_FN 0 > @@ -433,8 +438,28 @@ static int pseries_msi_ops_prepare(struct irq_domain *domain, struct device *dev > struct msi_domain_info *info = domain->host_data; > struct pci_dev *pdev = to_pci_dev(dev); > int type = (info->flags & MSI_FLAG_PCI_MSIX) ? PCI_CAP_ID_MSIX : PCI_CAP_ID_MSI; > + int ret; > + > + struct pseries_msi_device *pseries_dev __free(kfree) > + = kmalloc(sizeof(*pseries_dev), GFP_KERNEL); > + if (!pseries_dev) > + return -ENOMEM; > + > + while (1) { > + ret = rtas_prepare_msi_irqs(pdev, nvec, type, arg); > + if (!ret) > + break; > + else if (ret > 0) > + nvec = ret; > + else > + return ret; > + } > > - return rtas_prepare_msi_irqs(pdev, nvec, type, arg); > + pseries_dev->msi_quota = nvec; > + pseries_dev->msi_used = 0; > + > + arg->scratchpad[0].ptr = no_free_ptr(pseries_dev); > + return 0; > } > > /* > @@ -443,9 +468,13 @@ static int pseries_msi_ops_prepare(struct irq_domain *domain, struct device *dev > */ > static void pseries_msi_ops_teardown(struct irq_domain *domain, msi_alloc_info_t *arg) > { > + struct pseries_msi_device *pseries_dev = arg->scratchpad[0].ptr; > struct pci_dev *pdev = to_pci_dev(domain->dev); > > rtas_disable_msi(pdev); > + > + WARN_ON(pseries_dev->msi_used); > + kfree(pseries_dev); > } > > static void pseries_msi_shutdown(struct irq_data *d) > @@ -546,12 +575,18 @@ static int pseries_irq_domain_alloc(struct irq_domain *domain, unsigned int virq > unsigned int nr_irqs, void *arg) > { > struct pci_controller *phb = domain->host_data; > + struct pseries_msi_device *pseries_dev; > msi_alloc_info_t *info = arg; > struct msi_desc *desc = info->desc; > struct pci_dev *pdev = msi_desc_to_pci_dev(desc); > int hwirq; > int i, ret; > > + pseries_dev = info->scratchpad[0].ptr; > + > + if (pseries_dev->msi_used + nr_irqs > pseries_dev->msi_quota) > + return -ENOSPC; > + > hwirq = rtas_query_irq_number(pci_get_pdn(pdev), desc->msi_index); > if (hwirq < 0) { > dev_err(&pdev->dev, "Failed to query HW IRQ: %d\n", hwirq); > @@ -567,9 +602,10 @@ static int pseries_irq_domain_alloc(struct irq_domain *domain, unsigned int virq > goto out; > > irq_domain_set_hwirq_and_chip(domain, virq + i, hwirq + i, > - &pseries_msi_irq_chip, domain->host_data); > + &pseries_msi_irq_chip, pseries_dev); > } > > + pseries_dev->msi_used++; > return 0; > > out: > @@ -582,9 +618,11 @@ static void pseries_irq_domain_free(struct irq_domain *domain, unsigned int virq > unsigned int nr_irqs) > { > struct irq_data *d = irq_domain_get_irq_data(domain, virq); > - struct pci_controller *phb = irq_data_get_irq_chip_data(d); > + struct pseries_msi_device *pseries_dev = irq_data_get_irq_chip_data(d); > + struct pci_controller *phb = domain->host_data; > > pr_debug("%s bridge %pOF %d #%d\n", __func__, phb->dn, virq, nr_irqs); > + pseries_dev->msi_used -= nr_irqs; > irq_domain_free_irqs_parent(domain, virq, nr_irqs); > } > I just tested this change on my system using the latest mainline kernel and it works well for me. So with that, please fell free to add, Tested-by: Nilay Shroff