From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id AB3A52DF12F for ; Sun, 26 Apr 2026 20:52:47 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1777236767; cv=none; b=FsFwYOsmccV/OAONNVQkRmCvlxedj2E9QKcTVUcVzS+htOi39nFxFafNjdVF9mvw8qyegkC+CYb4IVrQOx983yzQZHnoRPhMg1nuawg2xL8x3hLcgXwV4QBnbs5Uj60lf/5FMPifPuj6pNzNmADtUQSBpHOFuOFifKBs8XMBrkA= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1777236767; c=relaxed/simple; bh=1ma1ybPQrkEGIy/7AZgec1dYWXv9poAvKBhZ+gJ0LKE=; h=Date:From:To:Cc:Subject:Message-Id:In-Reply-To:References: Mime-Version:Content-Type; b=CKXMT1834sSpdou5Rj640owTBdpNF0FqCJFTE7Wkqd3b0nNHtiUixy7ijOECdsnqJytbGzug0Iezavh0RJQOZ7Y4Oxk8huP9Gb1up+944Vk7bf2dbWeuh2aBdRy71cz44YUErfQfwfTHsCiz9+r+Ubq0CLHqm6xOP1vM5QkzXFw= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux-foundation.org header.i=@linux-foundation.org header.b=UEKtmT4R; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux-foundation.org header.i=@linux-foundation.org header.b="UEKtmT4R" Received: by smtp.kernel.org (Postfix) with ESMTPSA id EB00DC2BCB4; Sun, 26 Apr 2026 20:52:46 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linux-foundation.org; s=korg; t=1777236767; bh=1ma1ybPQrkEGIy/7AZgec1dYWXv9poAvKBhZ+gJ0LKE=; h=Date:From:To:Cc:Subject:In-Reply-To:References:From; b=UEKtmT4RDlL/UzRBSh4Se1cQjE+8To+USCtz4vPBDznMVAXa+3soVfp7PMcDQgfjQ aExEemQJ7VUhpV+NA10qVTmeMy3/bT2C69qbqpzbDoass8M/aJ8f+6pYCvckw6gZzz 7MM+ze+ddifeINdH78a6R/frnmudfmJK7mr+jO40= Date: Sun, 26 Apr 2026 13:52:46 -0700 From: Andrew Morton To: Naman Jain Cc: Thomas Gleixner , Tianyou Li , Wangyang Guo , Tim Chen , linux-kernel@vger.kernel.org, Long Li , Ming Lei , Ming Lei Subject: Re: [PATCH] lib/group_cpus: rotate extra groups to avoid IRQ stacking Message-Id: <20260426135246.0a52d9f4e04fd84a304b2339@linux-foundation.org> In-Reply-To: <20260324075352.2326972-1-namjain@linux.microsoft.com> References: <20260324075352.2326972-1-namjain@linux.microsoft.com> X-Mailer: Sylpheed 3.8.0beta1 (GTK+ 2.24.33; x86_64-pc-linux-gnu) Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit On Tue, 24 Mar 2026 07:53:52 +0000 Naman Jain wrote: > When multiple devices call group_cpus_evenly() with the same number of > groups, the cluster-aware path in __try_group_cluster_cpus() assigns > extra groups to the same set of clusters every time, producing identical > affinity masks for every caller. CPUs in clusters that receive two > groups (and thus get single-CPU dedicated masks) end up handling > interrupts from ALL devices, creating an IRQ imbalance. > > For example, on a 96-CPU / 2-NUMA-node system with 24 clusters of > 2 CPUs each and 6 NVMe disks each requesting 62 vectors: > alloc_groups_to_nodes() distributes 31 groups across 24 clusters, > giving 7 clusters 2 groups (single-CPU mask = dedicated) and 17 > clusters 1 group (2-CPU mask = shared). Because the assignment is > deterministic, all 6 disks produce the same mapping and the same 14 > CPUs each accumulate 6 dedicated IRQs -- roughly twice the interrupt > load of other CPUs -- causing up to 11% per-disk throughput degradation > on IRQ-heavy CPUs. > > Fix this by introducing a per-caller rotation offset via a static > atomic counter. After alloc_groups_to_nodes() determines each > cluster's group count, collect the extras (groups above the per-cluster > minimum), then redistribute them starting from a rotated position with > a stride of ncluster/extras so that successive callers scatter their > extra groups across different clusters. A capacity check > (cpumask_weight_and) ensures no cluster is assigned more groups than it > has CPUs, with a fallback loop for any extras that could not be placed > in the strided pass. > > For systems without cluster topology, the same rotation is applied in > assign_cpus_to_groups() at the per-group level: the modular expression > (v + spread_offset) % nv->ngroups selects which groups receive the > extra CPU, replacing the previous sequential decrement. Thanks. AI review asked a couple of questions. https://sashiko.dev/#/patchset/20260324075352.2326972-1-namjain@linux.microsoft.com group_cpus.c is difficult. It's tricky code and few people seem to be familiar with it - I certainly don't feel competent to review changes. The original author (Ming Lei) is still around, but wasn't cc'ed on this change. Let me add. (I'm seeing two Ming Lei's - apologies if they aren't the same person ;))