From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 25D282E6CD3 for ; Fri, 9 Jan 2026 19:13:36 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.133.124 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1767986018; cv=none; b=YkSOltuZHq0wWL9aFdLh9X5WIZHgf/ZzBSzTWBmJkkDKxfsDnfzm9ufOu7SylNtxazmT9ObpQjKjo8h0WZG7p5AR/7o1a4qGu7t80cvWD7hjyL7ajnTLuh29LAC/laLNkV/0WEOFAdY3qnOITDgh7aMZx6apHuNjLNWWlP1jtZ8= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1767986018; c=relaxed/simple; bh=3xLqK7dbo+LhMyJSpdWdW9PWsAG/N3YiIzr7UtMPamc=; h=Message-ID:Subject:From:To:Cc:Date:In-Reply-To:References: Content-Type:MIME-Version; b=pkQZWsVgd04/BlHYc8fHWzMPbqU4SPOUY2njKneYwNfqPSdZZwgK1grjr8CjiUIFPLmLPnzfVGYTZvvsUK+ymvezeLZ5/NgWa/o6CtqpKaw/D9BY9AnKrneaLd6PVSZ3/fm6RVzmONFXVZGZmBENiqj1jgBDjK/3WlVUZB4Rogs= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=TR/akbNX; dkim=pass (2048-bit key) header.d=redhat.com header.i=@redhat.com header.b=ql4f7B/7; arc=none smtp.client-ip=170.10.133.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="TR/akbNX"; dkim=pass (2048-bit key) header.d=redhat.com header.i=@redhat.com header.b="ql4f7B/7" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1767986016; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=4A3HmcRWbH7dATR6POUlRaQ+ovECxyBvO9jLQXYEt34=; b=TR/akbNXAc+C7mxkEoqZQddUww7TmC5ECidYiAy4C7fG0HlO3Bp3+xzXnGiXbymTVFJW/3 kEhEyor5ewrXPugTfmjiEeXjKftSw7QlKHzWVi3sNPhpPn4DpkAnybySDLXMZoL+oxsQkt 3JyajLU9WZ7QUlabe6Nz++pDu36tg6E= Received: from mail-qv1-f71.google.com (mail-qv1-f71.google.com [209.85.219.71]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-654-F5-l6Vh0ODCyp80XGy7zjw-1; Fri, 09 Jan 2026 14:13:35 -0500 X-MC-Unique: F5-l6Vh0ODCyp80XGy7zjw-1 X-Mimecast-MFC-AGG-ID: F5-l6Vh0ODCyp80XGy7zjw_1767986014 Received: by mail-qv1-f71.google.com with SMTP id 6a1803df08f44-88a3a4af701so119022906d6.1 for ; Fri, 09 Jan 2026 11:13:35 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=google; t=1767986014; x=1768590814; darn=vger.kernel.org; h=mime-version:user-agent:content-transfer-encoding:references :in-reply-to:date:cc:to:from:subject:message-id:from:to:cc:subject :date:message-id:reply-to; bh=4A3HmcRWbH7dATR6POUlRaQ+ovECxyBvO9jLQXYEt34=; b=ql4f7B/7w5r278P/PQvyHbDzeOqv0y3SneOjTkJVRx1wb3exptEqsk6alELSWmr+vB M2ojQSPjz6LlyEOkG4eyB+ZC3ZT5eRsxsSTMnRc3N35wJ+h3M9ABQk2Zm5rx55UtknEc YSuy7h7LcrmSrKGfwkz/cr3LjTOsThrIEAE1rkoagFPVjawEcl0bBg0h8Fckaeyh4JlO 7MEE/4zErcCu/GbdpviMzlD/sTsbqtW7wLBzNIjX/FShXXqqRVvkM2SytHWlPWysuwQ9 qacMMhJLHcbO3igjYc9ZuPSDMJ7cRrPla0HjXEPv1rzJKm5Bmas9BHVD3ToS3T0Sf0dB OA4w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1767986014; x=1768590814; h=mime-version:user-agent:content-transfer-encoding:references :in-reply-to:date:cc:to:from:subject:message-id:x-gm-gg :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=4A3HmcRWbH7dATR6POUlRaQ+ovECxyBvO9jLQXYEt34=; b=GXVSBlmqibQ5ECK7N02yj7RsNkey1e1k5SwFbZpPmPCjA8A4GwjFujlsX7M8FQxUOE +sninYaDQ2+I0L7L5CQ18cBmToTShTDGls/uLuCrsI2Qr8vlkZpXs21+kBN0Zxeev0Yc jTiHk0E2qtnp5IPMl++kvgNJVfijF+K7hxfWNItWh8FQ5VSXHz+diG99dhYCEIdQ6jXd U4anfQrm6irV6Aa4FyZP3Huk/q2bSrYwkmSIFzg6lzzMDGCPzj9yO3rIA4SZZ2sqSKqX aa2x4DjjLDcwgk21Ec2YybvGjJTPu6LeC/nLlQXAt8H+TYcXU8KlsD4d5EJ2lxKfOFjp obDA== X-Forwarded-Encrypted: i=1; AJvYcCU9d1fdIUvOxFmsjlDU7WVk5so9sr5kaUO+4Rpt0WWvywRutFZWQdkyyel2Bbj2i6FY/USGRi+XgL/+1/0=@vger.kernel.org X-Gm-Message-State: AOJu0Yw1gBqsBqtWPXYBuYfA0QQ/Q+zh9JSekTqwpg5jgQimzrKCFff5 oZNbkaW2JVYVFkQpqHZfks1x9gFYs88uHf1GLs1MoFUsTtrY5IC9mcTTKBqSNSWYfSMB30UvMEZ FUyGcAXxQarXidHEU9ho+CAbyM4EPBvQlNdtLXvwRGeGjUCQeKmGBK8FzCfQj0t8W/ShFk8QQ7Q == X-Gm-Gg: AY/fxX6GgnzJufxAg/cBW5hj73Lcx2T0C5BWY413/qWQaN+NpY19oG8GrC4i+EWc3cg dWNKYthVhSvHIG7a1stz27hI2QlbW+dyeDgxthKxj5euRraQ5Vp12GOT4tUbnCRZKD7wtXfWkPn GJW3tQ6IN/HxVniGdxTvADoj/Owpwa5WD7K9sHhCsObO3fw9p+hPrBeUDh5dlTrf3DCfzBc3QtU JPbF6rbWDF0YY7udCKAEwcyva8D8zQzCCaApneKROjxo32j5VCcKBS35pXCBNCpkuN9vy4pvIgt xNGxD/3FsEMScAbTYZG17PTo2Qf4VSO95SMGmmwiE05iiN2oMld2ic1u3b1IKnvSG0jaZ0MPevn vDKkPb369UGDo/97vRg6bin2+Y9+P90l0qToQTUYyaEWvJe+smwNkMLIaagN80BQ6cPUvnPyDtD hLNw== X-Received: by 2002:a05:6214:2608:b0:880:471f:2ecc with SMTP id 6a1803df08f44-890841ae647mr169022846d6.29.1767986014158; Fri, 09 Jan 2026 11:13:34 -0800 (PST) X-Google-Smtp-Source: AGHT+IEFfc5418alOChLAhPBEt9xQwcY0cf5ms8n7ipMlidtgqmWrxpkIb9c9xGvi9uFSHGwdW6m1w== X-Received: by 2002:a05:6214:2608:b0:880:471f:2ecc with SMTP id 6a1803df08f44-890841ae647mr169022346d6.29.1767986013756; Fri, 09 Jan 2026 11:13:33 -0800 (PST) Received: from thinkpad-p1.localdomain (pool-174-112-193-187.cpe.net.cable.rogers.com. [174.112.193.187]) by smtp.gmail.com with ESMTPSA id 6a1803df08f44-890770e2aa5sm82084876d6.19.2026.01.09.11.13.32 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 09 Jan 2026 11:13:33 -0800 (PST) Message-ID: Subject: Re: [PATCH] lib/group_cpus: make group CPU cluster aware From: Radu Rendec To: "Guo, Wangyang" , Andrew Morton Cc: Thomas Gleixner , linux-kernel@vger.kernel.org, Tianyou Li , Tim Chen , Dan Liang Date: Fri, 09 Jan 2026 14:13:32 -0500 In-Reply-To: References: <20251024023038.872616-1-wangyang.guo@intel.com> <20251221111047.597248db9868d278c7786f6b@linux-foundation.org> Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable User-Agent: Evolution 3.56.2 (3.56.2-2.fc42) Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Hi all, On Mon, 2025-12-22 at 11:03 +0800, Guo, Wangyang wrote: > On 12/22/2025 3:10 AM, Andrew Morton wrote: > > On Fri, 24 Oct 2025 10:30:38 +0800 Wangyang Guo wrote: > >=20 > > > As CPU core counts increase, the number of NVMe IRQs may be smaller t= han > > > the total number of CPUs. This forces multiple CPUs to share the same > > > IRQ. If the IRQ affinity and the CPU=E2=80=99s cluster do not align, = a > > > performance penalty can be observed on some platforms. > >=20 > > It would be helpful to quantify "performance penalty".=C2=A0 At least g= ive > > readers some approximate understanding of how serious this issue is, > > please. > >=20 > Thanks for your reminder, will update changelog in next version. We see= =20 > 15%+ performance difference in FIO libaio/randread/bs=3D8k. >=20 > > > This patch improves IRQ affinity by grouping CPUs by cluster within e= ach > > > NUMA domain, ensuring better locality between CPUs and their assigned > > > NVMe IRQs. > > >=20 > > > Reviewed-by: Tianyou Li > > > Reviewed-by: Tim Chen > > > Tested-by: Dan Liang > > > Signed-off-by: Wangyang Guo > >=20 > > Patch hasn't attracted additional review so I'll queue this version for > > some testing in mm.git's mm-nonmm-unstable branch.=C2=A0 I'll add a > > note-to-self that a changelog addition is desirable. >=20 >=20 > Thanks a lot for your time and support! Please let me know if you have= =20 > any further comments or guidance. Any feedback would be appreciated. With this patch applied, I see a weird issue in a qemu x86_64 vm if I start it with a higher number of max CPUs than active CPUs, for example `-smp 4,maxcpus=3D8` on the qemu command line. What I see is the `while (1)` loop in alloc_cluster_groups() spinning forever. Removing the `maxcpus=3D8` from the qemu command line fixes the issue but so does reverting the patch :) FWIW, this is my full qemu command line, in case it helps reproduce the issue. qemu-system-x86_64 -nographic -m 1G -accel kvm -machine q35,hpet=3Doff \ -cpu host -smp 4,maxcpus=3D8 \ -netdev bridge,br=3Dvbr-nat,id=3Dnet0 -device virtio-net-pci,netdev= =3Dnet0 \ -drive file=3Droot.img,format=3Draw,if=3Dvirtio \ -kernel bzImage \ -append "console=3DttyS0 root=3D/dev/vda1 rw nokaslr" -s And this is a stack trace extracted from gdb attached to qemu's gdb server. #0 check_region_inline (addr=3D0xffffffff833b7d40 <__per_cpu_offset+32>,= =20 size=3D8, write=3Dfalse, ret_ip=3D18446744071582127502) at mm/kasan/generic.c:185 #1 __asan_load8 (addr=3D0xffffffff833b7d40 <__per_cpu_offset+32>) at mm/kasan/generic.c:273 #2 0xffffffff8132158e in cpu_l2c_shared_mask (cpu=3Dcpu@entry=3D4) at arch/x86/include/asm/smp.h:149 #3 cpu_clustergroup_mask (cpu=3Dcpu@entry=3D4) at arch/x86/kernel/smpboot.c:672 #4 0xffffffff81e9d6b6 in alloc_cluster_groups (ncpus=3Dncpus@entry=3D4,= =20 ngroups=3Dngroups@entry=3D4,=20 node_cpumask=3Dnode_cpumask@entry=3D0xffff88800d7a28e0,=20 msk=3D0xffff88800d7a24e0,=20 clusters_ptr=3Dclusters_ptr@entry=3D0xffffc9000001eef0,=20 cluster_groups_ptr=3Dcluster_groups_ptr@entry=3D0xffffc9000001eed0) at lib/group_cpus.c:310 #5 0xffffffff81e9da22 in __try_group_cluster_cpus (ncpus=3Dncpus@entry=3D4= ,=20 ngroups=3D4, node_cpumask=3Dnode_cpumask@entry=3D0xffff88800d7a28e0,= =20 masks=3Dmasks@entry=3D0xffff88800e9a8000,=20 curgrp=3Dcurgrp@entry=3D0xffffc9000001f000, last_grp=3Dlast_grp@entry= =3D4) at lib/group_cpus.c:374 #6 0xffffffff81e9decf in __group_cpus_evenly (startgrp=3Dstartgrp@entry=3D= 0,=20 numgrps=3Dnumgrps@entry=3D4,=20 node_to_cpumask=3Dnode_to_cpumask@entry=3D0xffff88800d7a28a0,=20 cpu_mask=3D, nmsk=3D,=20 masks=3Dmasks@entry=3D0xffff88800e9a8000) at lib/group_cpus.c:457 #7 0xffffffff81e9e406 in group_cpus_evenly (numgrps=3Dnumgrps@entry=3D4,= =20 nummasks=3Dnummasks@entry=3D0xffffc9000001f248) at lib/group_cpus.c:547 #8 0xffffffff8121690c in irq_create_affinity_masks (nvecs=3Dnvecs@entry=3D= 5,=20 affd=3Daffd@entry=3D0xffffc9000001f738) at kernel/irq/affinity.c:74 #9 0xffffffff81efc253 in msix_setup_interrupts (dev=3D,=20 entries=3D, nvec=3D5, affd=3D0xffffc9000001f738) at drivers/pci/msi/msi.c:694 #10 msix_capability_init (dev=3D0xffff88800e014000, entries=3D,=20 nvec=3D5, affd=3D) at drivers/pci/msi/msi.c:738 #11 __pci_enable_msix_range (dev=3Ddev@entry=3D0xffff88800e014000,=20 entries=3Dentries@entry=3D0x0, minvec=3Dminvec@entry=3D5, maxvec=3Dmaxv= ec@entry=3D5,=20 affd=3Daffd@entry=3D0xffffc9000001f738, flags=3Dflags@entry=3D12) at drivers/pci/msi/msi.c:846 #12 0xffffffff81ef9ef3 in pci_alloc_irq_vectors_affinity ( dev=3D0xffff88800e014000, min_vecs=3Dmin_vecs@entry=3D5,=20 max_vecs=3Dmax_vecs@entry=3D5, flags=3Dflags@entry=3D12,=20 affd=3Daffd@entry=3D0xffffc9000001f738) at drivers/pci/msi/api.c:268 #13 0xffffffff82039d1f in vp_request_msix_vectors ( vdev=3Dvdev@entry=3D0xffff88800e86a000, nvectors=3D5,=20 per_vq_vectors=3Dper_vq_vectors@entry=3Dtrue, desc=3D,= =20 desc@entry=3D0xffffc9000001f738) at drivers/virtio/virtio_pci_common.c:160 #14 0xffffffff8203a615 in vp_find_vqs_msix ( vdev=3Dvdev@entry=3D0xffff88800e86a000, nvqs=3Dnvqs@entry=3D4,=20 vqs=3Dvqs@entry=3D0xffff88800d739ac0,=20 vqs_info=3Dvqs_info@entry=3D0xffff88800d761100,=20 vector_policy=3Dvector_policy@entry=3DVP_VQ_VECTOR_POLICY_EACH,=20 desc=3Ddesc@entry=3D0xffffc9000001f738) at drivers/virtio/virtio_pci_common.c:417 #15 0xffffffff8203ac0a in vp_find_vqs (vdev=3Dvdev@entry=3D0xffff88800e86a0= 00,=20 nvqs=3D4, vqs=3D0xffff88800d739ac0, vqs_info=3D0xffff88800d761100,=20 desc=3D0xffffc9000001f738) at drivers/virtio/virtio_pci_common.c:525 #16 0xffffffff82036553 in vp_modern_find_vqs (vdev=3D0xffff88800e86a000,= =20 nvqs=3D, vqs=3D, vqs_info=3D,=20 desc=3D) at drivers/virtio/virtio_pci_modern.c:751 #17 0xffffffff82152ea9 in virtio_find_vqs (vdev=3D0xffff88800e86a000, nvqs= =3D4,=20 vqs=3D0xffff88800d739ac0, vqs_info=3D0xffff88800d761100,=20 desc=3D0xffffc9000001f738) at include/linux/virtio_config.h:298 #18 init_vq (vblk=3Dvblk@entry=3D0xffff88800e7fe800) at drivers/block/virtio_blk.c:1017 #19 0xffffffff82153212 in virtblk_probe (vdev=3D0xffff88800e86a000) at drivers/block/virtio_blk.c:1470 #20 0xffffffff82024133 in virtio_dev_probe (_d=3D0xffff88800e86a098) at drivers/virtio/virtio.c:347 #21 0xffffffff82114166 in call_driver_probe (dev=3Ddev@entry=3D0xffff88800e= 86a098,=20 drv=3Ddrv@entry=3D0xffffffff840a5020 ) at drivers/base/dd.c:581 #22 0xffffffff82115038 in really_probe (dev=3Ddev@entry=3D0xffff88800e86a09= 8,=20 drv=3Ddrv@entry=3D0xffffffff840a5020 ) at drivers/base/dd.c:659 #23 0xffffffff821153d9 in __driver_probe_device ( drv=3Ddrv@entry=3D0xffffffff840a5020 ,=20 dev=3Ddev@entry=3D0xffff88800e86a098) at drivers/base/dd.c:801 #24 0xffffffff8211590a in driver_probe_device ( drv=3Ddrv@entry=3D0xffffffff840a5020 ,=20 dev=3Ddev@entry=3D0xffff88800e86a098) at drivers/base/dd.c:831 #25 0xffffffff82115c46 in __driver_attach (dev=3D0xffff88800e86a098,=20 data=3D0xffffffff840a5020 ) at drivers/base/dd.c:1225 #26 0xffffffff82111700 in bus_for_each_dev (bus=3D,=20 start=3Dstart@entry=3D0x0, data=3Ddata@entry=3D0xffffffff840a5020 ,=20 fn=3Dfn@entry=3D0xffffffff82115b30 <__driver_attach>) at drivers/base/bus.c:383 #27 0xffffffff82114738 in driver_attach ( drv=3Ddrv@entry=3D0xffffffff840a5020 ) at drivers/base/dd.c:1243 #28 0xffffffff82113860 in bus_add_driver ( drv=3Ddrv@entry=3D0xffffffff840a5020 ) at drivers/base/bus.c:715 #29 0xffffffff821171f6 in driver_register ( drv=3Ddrv@entry=3D0xffffffff840a5020 ) at drivers/base/driver.c:249 #30 0xffffffff8202306b in __register_virtio_driver ( driver=3Ddriver@entry=3D0xffffffff840a5020 , owner=3Downer@= entry=3D0x0) at drivers/virtio/virtio.c:456 #31 0xffffffff84b19134 in virtio_blk_init () at drivers/block/virtio_blk.c:1707 #32 0xffffffff812884eb in do_one_initcall ( fn=3D0xffffffff84b190e0 ) at init/main.c:1379 #33 0xffffffff84a82ba7 in do_initcall_level (level=3D6,=20 command_line=3D0xffff8880090e1880 "console") at init/main.c:1441 #34 do_initcalls () at init/main.c:1457 #35 0xffffffff84a82f24 in do_basic_setup () at init/main.c:1476 #36 kernel_init_freeable () at init/main.c:1689 #37 0xffffffff8277ad98 in kernel_init (unused=3D) at init/main.c:1579 #38 0xffffffff812e11fb in ret_from_fork (prev=3D,=20 regs=3D0xffffc9000001ff48, fn=3D0xffffffff8277ad80 , fn_ar= g=3D0x0) at arch/x86/kernel/process.c:158 #39 0xffffffff8129390a in ret_from_fork_asm () at arch/x86/entry/entry_64.S:246 --=20 Best regards, Radu