From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-alma10-1.taild15c8.ts.net [100.103.45.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B0D7A344DA0 for ; Tue, 4 Aug 2026 14:56:04 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=100.103.45.18 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1785855365; cv=none; b=me84dntQVQc/lrspwZjymqJp3AcIZdaQoLTyUzwr+Y7uzPYxSjE6OKd0qN5034ehwvAZ/cAldWaIr03/uPfw4nQusH3GhZlpV7YD1PdFBpe/ZvXuuQxdt9o1GilnBHIsi8pVPwyd6Yg4aZAoZPx1e1OmtfYchAUTRzkQrD++Yb4= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1785855365; c=relaxed/simple; bh=hVHkV9aMg9yUW0mE9AWpDcdDH2wpg7L3aVybOI9xbTw=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=pkuJIn8NJpkoH3J+Mp0Vd3J4/PX1zIf0CdkNsjweY3Gmgb4Vn8CXei5Orqh810myPcX6LydMsEkeTz1EfP+nVjGU6DlyAYAyj6uFZXq5Zpskwtf/SSYs6wHDrGerf2lIObV00nr8KKxnCno68LRwEYsr4fT6pRZLVbFdTCtboMU= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=TtknuONO; arc=none smtp.client-ip=100.103.45.18 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="TtknuONO" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 9E2751F000E9; Tue, 4 Aug 2026 14:56:03 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1785855364; bh=pN5T6SWfef15BLljX0crFCY/mrHy2E+vETSnOrvY9gY=; h=Date:From:To:Cc:Subject:References:In-Reply-To; b=TtknuONO0JjvVkp7qqVz/WI97Qa/DAHZZ2RyNd47QMzCc/KtOSqGiflG1bQ7w72pf RSBfERS2yyg1Vn29Hka+b+jMRn4Zz6FcorLGCJJ0YghyplnjC8gkHTVEozEd2MEKbs 3UBTN/12TMKGJ6kBZi7gy0QfOm611amUKHJG3B6emznaqvSvR0QmCaWOMzCuL+E6tp gEH7uRMPTn0HVlg/EqllPskOXRomKoDCOceDoJWIhgbNr2HX2dw4RlSZWIT7DaeEYA XDQJyd1HmUoMHfdVXLTp/Pr6Fl2XhjTGNayvAJRDLc7eCKana4Ju4Xuusm49NHmtR7 THmGxYdwy+FJg== Date: Tue, 4 Aug 2026 04:56:01 -1000 From: Keith Busch To: Christoph Hellwig Cc: Thomas Gleixner , santhosh kumar , linux-kernel@vger.kernel.org, Ming Lei , x86@kernel.org Subject: Re: [RFC] PCI_IRQ_AFFINITY limits MSI-X allocation on 384 CPU / 1000+ NVMe system Message-ID: References: <87cxw6hgli.ffs@fw13> <87ldasfwht.ffs@fw13> <87ik5wfeju.ffs@fw13> <20260804141023.GA9171@lst.de> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20260804141023.GA9171@lst.de> On Tue, Aug 04, 2026 at 04:10:23PM +0200, Christoph Hellwig wrote: > On Mon, Aug 03, 2026 at 05:43:59AM -1000, Keith Busch wrote: > > On Thu, Jul 30, 2026 at 09:44:21PM +0200, Thomas Gleixner wrote: > > > On Thu, Jul 30 2026 at 07:28, Keith Busch wrote: > > > > We can introduce a module parameter to throttle down the maximum number > > > > of IO queues to allocate per controller. I don't think the driver can > > > > automatically reason out what the correct number should be because it > > > > doesn't know how many devices it's going to see. > > > > > > I might be missing something here as usual, but the number of io queues > > > per controller does not affect that at all as I explained before. > > > > Oh, you did explain that. I skipped to the end as I had too many > > distractions last week. Sorry about that, and thanks for the > > re-explanation. > > > > I mistakenly was thinking the effective_affinity was the only hardware > > resource used. Back to the drawing board. > > But given that we don't use multiple tag_set by default this can't > be the issue of the reported unless they didnt report enabling > multiple tag sets. I believe the use of additional sets makes the problem worse, but I think we still have a problem even if we have only the one default set. This is the quote from Thomas that I'm reading into: "the per CPU vector consumption is always _one_ per set." So just the default set with only 1 NVMe IO queue managed IRQ will still consume 384 vectors for the reported system even though the effective affinity is pinned to 1 CPU.