From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from galois.linutronix.de (Galois.linutronix.de [193.142.43.55]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 790EC1DB356 for ; Fri, 14 Aug 2026 00:52:36 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=193.142.43.55 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1786668758; cv=none; b=ENFTOUMqXYYtNZAic+hJXBVo4NNOetzCApW8zM7oC4XjCufBCo1UoimVbKEha64BKFfE7GSu8mTv2t3nKsx4HXsXP9fPoLVzieevhnVB4bAHmZex05Ox3idzGZMGsnRhGMEU2kX+yJE62Ywpt55J/dnPuTBUsXudnfneGAu9EQ4= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1786668758; c=relaxed/simple; bh=NhOrt/zZLFgivfATzyxoaf6NJXPijPpq87k+OCeDaf0=; h=From:To:Cc:Subject:In-Reply-To:References:Date:Message-ID: MIME-Version:Content-Type; b=Trn20wtf+jsY2Cm9hZzVzPtvmfT0V0V7x0ZG/1oBsMbHq5R2JK3FWmGP6Wb1neTFOXkfrajGvcjLnzYFHi1lvGc5IwjfNw+3vSpKVVgfJf3uDyPsgvFtLG1PwzIzf53tGflncXAERdRuIA3r7u+QKhXdeBTRag16luid+2PkvuE= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linutronix.de; spf=pass smtp.mailfrom=linutronix.de; dkim=pass (2048-bit key) header.d=linutronix.de header.i=@linutronix.de header.b=Ewh0nvi8; dkim=permerror (0-bit key) header.d=linutronix.de header.i=@linutronix.de header.b=BUbVY29t; arc=none smtp.client-ip=193.142.43.55 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linutronix.de Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linutronix.de Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="Ewh0nvi8"; dkim=permerror (0-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="BUbVY29t" From: Thomas Gleixner DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1786668754; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=cnpx1Bsn1WxUqSZzDmW+kJbYAApcN8pKthV1MnaZRzQ=; b=Ewh0nvi8a8BPAxG5MgjcE18fttvdeToip4OGQC2kCw5zOw1oxFPDgagbfEHXlmAh9BPc+s s18MPQUDnOY3odQCZ13+d2EEEwx2JXo1cl9QeBRyOzOVPNGb+xd0LPMfAPRkeXxVjG68hK dIqsAi1cVdgBuro9dMSw7Z5/GmHEwTtyeZkVOEsCF0ATOkfPcF/iWHoJANolton8DQMGSZ rPVee4nslJ3FKGX9JBUi+n4U4igNCI8PWLtumHMbepRlx0bWH0qdSX9goWvw9heOxP3oL3 xcXw2wQBSp/n3Zqh0IHXZzI6QNVkJOhkeRaAe4s8rfbzOj+6C5BYoPW54J+ppA== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1786668754; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=cnpx1Bsn1WxUqSZzDmW+kJbYAApcN8pKthV1MnaZRzQ=; b=BUbVY29t6FDpVO/uJkjQ8rxjZ1NQ54vrFhWy8+6ju97WvL14oZTquxpPIsxK6YXh2fuWev NXpSl//n2WRtm9Bg== To: Keith Busch Cc: Christoph Hellwig , santhosh kumar , linux-kernel@vger.kernel.org, Ming Lei , x86@kernel.org Subject: Re: [RFC] PCI_IRQ_AFFINITY limits MSI-X allocation on 384 CPU / 1000+ NVMe system In-Reply-To: References: <87cxw6hgli.ffs@fw13> <87ldasfwht.ffs@fw13> <87ik5wfeju.ffs@fw13> <20260804141023.GA9171@lst.de> <871pcdfufr.ffs@fw13> Date: Fri, 14 Aug 2026 02:52:33 +0200 Message-ID: <87ecg1msku.ffs@fw13> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain On Mon, Aug 10 2026 at 15:11, Keith Busch wrote: > On Tue, Aug 04, 2026 at 11:27:04PM +0200, Thomas Gleixner wrote: >> We could come up with some less restrictive mechanism, but that would at >> the end run into the vector limitation on hotplug/hibernate because you >> can't fit more than ~200 vectors into the last online CPU. >> >> OTOH. With 1000 devices which consume also one non-managed interrupt for >> their management queues, i.e. a total of 1000, that's not going to work >> anyway. > > I think we'd have to at some point declare that extremely mismatched > setups just can't be reasonably supported. :) Then let the guy who wants 1000+ NVME devices have a parameter to turn off managed and multiqueue mode and leave everything else alone. :) > My understanding is you've got the managed IRQ's allocating a vector up > front for every possible CPU they could be migrated to primarily to > ensure CPU unplug can always move the effective affinity to an active > CPU. Until the last CPU in the assigned affinity mask of a particular interrupt goes offline, which shuts down the interrupt after the multiqueue management drained and silenced the queue. > I totally get that not having this guarantee can cause a device to > appear unresponsive, but maybe this is unlikely enough to accept? Or Unlikely does not exist at scale, so that's a non-argument to begin with. I have exactly ZERO interest to debug such nonsense. > maybe not, I am not sure; it just seems like a fringe case that trying > to mitigate causes a more likely problem. Which fringe case? CPU hotplug and hibernation are actively used for dynamic scaling and it's not so fringe as you think. Just because your use case does not need it does not mean there are no relevant users who depend on it. Throwing 1000+ NVME devices at a machine is more fringe to me and as I explained before that also means that CPU hotplug down to a single CPU is not possible at all. The real fringe case is x86 because any other sane and relevant architecture just has a large enough vector space to work with. Don't tell me about loongarch as those clowns just repeated the x86 nonsense. > Anyway, if we can tolerate dynamic vector allocation that only happens > on effective CPU assigment, here's a PoC I did some basic sanity testing > with: That breaks the basic guarantees of managed interrupts and the multi-queue magic: The interrupt and the queue are local to the auto-configured affinity set which got established by the spreading algorithm. You can't guarantee that anymore. A lot of people really want to have that guarantee and worked hard to make the spreading algorithm work accross various topologies for optimal performance. This needs a lot more thoughts than just ripping it out and pretending that it works by some definition of works in order to support a use case which has not even demonstrated any real world relevance. Just because someone want's to instantiate 1000+ devices does not mean that's something actually useful and practical. I used Qemu myself to push the limits just to verify that stuff works as designed. You cannot rip out the managed code on x86 and the related matrix bits without taking care of the consequences for CPU hotplug under vector pressure. With your change, failing to move the interrupt during hotplug within the given affinity mask will result in a non-working interrupt. Of course you can "fix" that in the interrupt hotplug handling and break affinity instead of failing, but that will permanently screw up the affinity mask and therefore the relationship between queues and interrupts and you might end up with a queue on one and the related interrupt on another node. The problem here is that the interrupt affinity handling is done late in the hotplug operations, i.e. after the CPU marked itself offline, which is the point of no return. There have been discussions to move that earlier, but that's a non trivial effort. That's why the hotplug machinery validates that the actual required affinity movement can be handled _before_ this stage is reached and if not the hotplug operation is aborted. But that code does not take managed interrupts into account under the assumption that there is a guaranteed free vector for those interrupts within the associated affinity mask. To make this work you have to do something like this: 1) Change the managed code in the matrix allocator to reserve a vector only on one CPU within the affinity mask of each queue and ensure that they are balanced out between the CPUs properly, which the matrix allocator already supports. The resulting single CPU reservation needs to be returned and stored e.g. in irq_cfg 2) Let the set_affinity() callback, which is invoked when the interrupt is activated, use that cached CPU information to actually allocate the interrupt on that CPU. 3) Make lapic_can_unplug_cpu() aware of the managed magic. That means you have to: - Walk the per CPU vector descriptor array - Find the managed interrupts - Check their affinity mask and validate that they can be moved within the mask unless it's the last CPU going down - If that's not possible, reject the hotplug operation after cleaning up the already made reservations. - If it's not the last CPU going down the reserve a vector on one of the remaining online CPUs, which is the same as #1) So if #3 succeeds the vector space is not exhausted and then migrate_one_irq() should just do the right thing. The reservation magic is required because otherwise the allocator could exhaust the vector space on one of the still online CPUs by moving non-managed interrupts to them instead of moving them to some other CPU as the actual hotplug code just walks the interrupt space linearly from 0 to max and does not pick the managed ones first. That should pretty much preserve the current behavior and guarantees and if exceeded actually fail the hotplug operations (e.g. on hibernate) gracefully and not rely on insane "unlikely to happen" assumptions. With that the only limiting factor for the non-hotplug case will be the number of queues per device which you spread assuming an even spreading, which is not guaranteed on assymetric node topologies. And of course ignoring other devices in the system which consume interrupt vectors. Ndev = Nvec * NCPUs / (M_per_device + Nqueues_per_device * Nsets_per_device) Where 'M_per_device' stands for the non-managed 'management' interrupts per device, which is usually one. So assuming 256 CPUs with 200 available vectors (Nvec) you get the following limits: Nqueues_per_device | Nsets_per_device | M_per_device | Ndev 1 | 1 | 1 | 25600 2 | 1 | 1 | 17067 4 | 1 | 1 | 10240 8 | 1 | 1 | 5689 16 | 1 | 1 | 3012 32 | 1 | 1 | 1552 64 | 1 | 1 | 788 128 | 1 | 1 | 397 256 | 1 | 1 | 199 With two sets per device that obviously just shifts the whole business up by one row and the 199 devices limit happens already at 128 write queues plus 128 read qeues per device and the 256 queues limit is reached with 100 devices. The point where hibernation remains possible reduces that number significantly, but that's obviously not at all different from todays limitations: Ndev = Nvec / (Nqueues_per_device * Nsets_per_device) Nqueues_per_device | Nsets_per_device | M_per_device | Ndev X | 1 | 1 | 100 X | 2 | 1 | 50 As I told the original poster before, it all just boils down to trivial math :) I have zero objections to make that change, but I'm not going to accept anything which does not completely address the points I made because I have _ZERO_ interrest to deal with the fallout. Correctness first is still the only valid engineering principle. Thanks, tglx