From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from linux.microsoft.com (linux.microsoft.com [13.77.154.182]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 783D439F18B for ; Mon, 27 Apr 2026 08:45:12 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=13.77.154.182 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1777279513; cv=none; b=LU42OOyN0TolGlEXTOU8KdMUBWX6F1EspwgU5Q4P7t1h//fD66JN/qJmJAmOr4fTcqHlSLQ6aLgJDBjR1yqDeY2q3/Fu60CmOvNvztPqDBJJ7b4E/Vk0p296ITPyBg6NvqSIbMV4+VfJBYQPVjm+DC1bhpMRaj0pK/+KVSil6YE= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1777279513; c=relaxed/simple; bh=1H67xYzOiH7Fz3pAyT6Qxgm16LSMGNWebf4ZmFVLTH8=; h=Message-ID:Date:MIME-Version:Subject:To:Cc:References:From: In-Reply-To:Content-Type; b=dyceor7QpM34y+LxHLzmuninOZBDCY58P5DSADN2m01eovUywQPE9p50qtKM1CxcIHvXkdZ0jMk8OO76YuSD6X5IoaDorYGaZ4y91MV40jb5AYgHcAiybKUVDJpaUjyTs3cPVSqouBLFa7gmdWSNE+YWAz9n22Oq0Oyg1dJBcVU= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.microsoft.com; spf=pass smtp.mailfrom=linux.microsoft.com; dkim=pass (1024-bit key) header.d=linux.microsoft.com header.i=@linux.microsoft.com header.b=PAKcjLk1; arc=none smtp.client-ip=13.77.154.182 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.microsoft.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.microsoft.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.microsoft.com header.i=@linux.microsoft.com header.b="PAKcjLk1" Received: from [10.95.65.38] (unknown [167.220.238.198]) by linux.microsoft.com (Postfix) with ESMTPSA id 97FAF20B7169; Mon, 27 Apr 2026 01:45:09 -0700 (PDT) DKIM-Filter: OpenDKIM Filter v2.11.0 linux.microsoft.com 97FAF20B7169 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.microsoft.com; s=default; t=1777279512; bh=mN1bb8tSsp6OzfO0fPRg6W+gNzwAIj30QL4w/cSjqFs=; h=Date:Subject:To:Cc:References:From:In-Reply-To:From; b=PAKcjLk1VveQNZnTqPjI4qIrHIoommfCPNJIu6RKsIhFWjbb7g7xdC8VybV1FP2nj n+RgWPflKN8HRvoYLQ/v42lXpJp4p7dALRA3OrC/3nKvNkEOUEwMUmu+AaByNe/7TE gVwni8eliXWsfgUhEAYoqlNGYylIq6gPq8BRnsSc= Message-ID: <16c5687c-f906-4299-8ff1-59e2b66a0bbb@linux.microsoft.com> Date: Mon, 27 Apr 2026 14:15:05 +0530 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [PATCH] lib/group_cpus: rotate extra groups to avoid IRQ stacking To: Andrew Morton Cc: Thomas Gleixner , Tianyou Li , Wangyang Guo , Tim Chen , linux-kernel@vger.kernel.org, Long Li , Ming Lei , Ming Lei References: <20260324075352.2326972-1-namjain@linux.microsoft.com> <20260426135246.0a52d9f4e04fd84a304b2339@linux-foundation.org> Content-Language: en-US From: Naman Jain In-Reply-To: <20260426135246.0a52d9f4e04fd84a304b2339@linux-foundation.org> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit On 4/27/2026 2:22 AM, Andrew Morton wrote: > On Tue, 24 Mar 2026 07:53:52 +0000 Naman Jain wrote: > >> When multiple devices call group_cpus_evenly() with the same number of >> groups, the cluster-aware path in __try_group_cluster_cpus() assigns >> extra groups to the same set of clusters every time, producing identical >> affinity masks for every caller. CPUs in clusters that receive two >> groups (and thus get single-CPU dedicated masks) end up handling >> interrupts from ALL devices, creating an IRQ imbalance. >> >> For example, on a 96-CPU / 2-NUMA-node system with 24 clusters of >> 2 CPUs each and 6 NVMe disks each requesting 62 vectors: >> alloc_groups_to_nodes() distributes 31 groups across 24 clusters, >> giving 7 clusters 2 groups (single-CPU mask = dedicated) and 17 >> clusters 1 group (2-CPU mask = shared). Because the assignment is >> deterministic, all 6 disks produce the same mapping and the same 14 >> CPUs each accumulate 6 dedicated IRQs -- roughly twice the interrupt >> load of other CPUs -- causing up to 11% per-disk throughput degradation >> on IRQ-heavy CPUs. >> >> Fix this by introducing a per-caller rotation offset via a static >> atomic counter. After alloc_groups_to_nodes() determines each >> cluster's group count, collect the extras (groups above the per-cluster >> minimum), then redistribute them starting from a rotated position with >> a stride of ncluster/extras so that successive callers scatter their >> extra groups across different clusters. A capacity check >> (cpumask_weight_and) ensures no cluster is assigned more groups than it >> has CPUs, with a fallback loop for any extras that could not be placed >> in the strided pass. >> >> For systems without cluster topology, the same rotation is applied in >> assign_cpus_to_groups() at the per-group level: the modular expression >> (v + spread_offset) % nv->ngroups selects which groups receive the >> extra CPU, replacing the previous sequential decrement. > > Thanks. AI review asked a couple of questions. > > https://sashiko.dev/#/patchset/20260324075352.2326972-1-namjain@linux.microsoft.com > > > group_cpus.c is difficult. It's tricky code and few people seem to be > familiar with it - I certainly don't feel competent to review changes. > > The original author (Ming Lei) is still around, but wasn't cc'ed on > this change. Let me add. (I'm seeing two Ming Lei's - apologies if > they aren't the same person ;)) Thank you Andrew. The points raised in the Sashiko's AI review seems to be valid, with respect to asymmetric topology. I'll address them in v2. Regards, Naman