From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.15]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D6127330641 for ; Sat, 20 Jun 2026 06:27:53 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=192.198.163.15 ARC-Seal:i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1781936876; cv=fail; b=TuWt1cWMWfH/t6zhjS1vBExDsKJ4yc+xbZ7DpSj2Q/k8GVDItBbRUbNb52TqnZCwA6/Be5wKwZZznmkjMdKcU6lD741GHJoajVUFg93RVfI9UBByMUMq+DuaUw/i7OXeYXjM1lrCiXi+cns4OpngFRY1TPuwcd0K3zQEMmffRSQ= ARC-Message-Signature:i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1781936876; c=relaxed/simple; bh=LOMLp3rdIGH+vpJloi52yN75vtaz2K7WDhe7FbQiI9I=; h=Message-ID:Date:Subject:To:CC:References:From:In-Reply-To: Content-Type:MIME-Version; b=mNZqVvHKqVboSwjFdT7LRndD+grvk1Pa4M6kOxd87oUSvT6WlmAsmYJgaRCfCj8ER/HwTXO8vkxoRb+fVnH+FF20wpY1FXGrS04fRZ8cpeip6O32UJINH5FtQNZIPxb9D3XtHgA97wd19G9+1VOMDyo4+N0+A0vvBZXdJLkciJk= ARC-Authentication-Results:i=2; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com; spf=pass smtp.mailfrom=intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=SBSTVwYe; arc=fail smtp.client-ip=192.198.163.15 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="SBSTVwYe" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1781936874; x=1813472874; h=message-id:date:subject:to:cc:references:from: in-reply-to:content-transfer-encoding:mime-version; bh=LOMLp3rdIGH+vpJloi52yN75vtaz2K7WDhe7FbQiI9I=; b=SBSTVwYenZMaRNLbznpBq+zlLGbNgExXnP7FyVR3xXGebOG/CvPK6gKl XKP0isp+L+Oi0deOczhcCUc+Miea3e6WUhxlOAvzB35Ir04buZ1AltPgT ImHIUFLXcpm3fo79r4B4272BPe8DzOIccSKDo3W/c65KnwIBBO7ZXFEwR 5KDvFaIoVSKR+kGb1TfWyFA1n3lg+4dgRaiDUf/hFiCFNJRn7oYgec1xr l1WbOVlCYZe8PUCfJt6q3Z35rbXXdPk21UMsRsVDllyC0p2McSFskTL5i SgUaYkrCgzrBQG0b5hoFfDLuRfQpIftzv0JLtNaMUYw2aDzT0h44n+Hl4 A==; X-CSE-ConnectionGUID: qGJADoABTUWdYZS7K6Iyng== X-CSE-MsgGUID: GqAq2UR+RE+rTWdIQZrHCA== X-IronPort-AV: E=McAfee;i="6800,10657,11822"; a="82867318" X-IronPort-AV: E=Sophos;i="6.24,214,1774335600"; d="scan'208";a="82867318" Received: from fmviesa005.fm.intel.com ([10.60.135.145]) by fmvoesa109.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 19 Jun 2026 23:27:53 -0700 X-CSE-ConnectionGUID: 3RaQCoEsQciaN1xggbaa2A== X-CSE-MsgGUID: R9oTMCFQQm+LWPWLqN6RpA== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.24,214,1774335600"; d="scan'208";a="253870603" Received: from fmsmsx902.amr.corp.intel.com ([10.18.126.91]) by fmviesa005.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 19 Jun 2026 23:27:53 -0700 Received: from FMSMSX903.amr.corp.intel.com (10.18.126.92) by fmsmsx902.amr.corp.intel.com (10.18.126.91) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.37; Fri, 19 Jun 2026 23:27:52 -0700 Received: from fmsedg902.ED.cps.intel.com (10.1.192.144) by FMSMSX903.amr.corp.intel.com (10.18.126.92) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.37 via Frontend Transport; Fri, 19 Jun 2026 23:27:52 -0700 Received: from SA9PR02CU001.outbound.protection.outlook.com (40.93.196.26) by edgegateway.intel.com (192.55.55.82) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.37; Fri, 19 Jun 2026 23:27:52 -0700 ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=onrSqEABmFxMKzrbg7u50T/JHa4U9liadUrzFvdedy2lYqeoQ1IjiW1xf0zty+vl9OCshiNgukJSA1sdERWRfzXISP5PbXR7i7ks07xMULhPnFrGPZMWHKQ4joayh7BOSnH+mMA6m0HaLlUBUWQVT55dqKy0dg0climF7Edwot/laOo43/eRD3T7RBPYsscGadQYnmGbt0of2ZN/Ubg6ruIQJrgFnlSXQwWCAvLIlmQkwSV3T3Os397tyJrIoFRiSKDdZdO3HDAZV/R2qIXKuBg08+8ZsBw5yc0u2BmgfSuMP072EvjLKm1ZzhsQPAc7gKVUobhx8lRn6ORlY1x3xw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=VnFjoOfnogWAT/c37aQMXv5dI44VHMewXNL24+VmKLE=; b=K1sc0lM4UsXG+H846+hgUz1Yecb8WOApOj28cKgyKqeo8ouuIZCMb8C+QVKgUOdIl3rhp3akTIHUe8rq4XMoCXSqf3k608KqlwNJqC3Q4Tr5Nllmq+7d1Ne7o9ErTll9mfIGbnkAjHCU40sj0isca1xy7ZLZlx6/LKypNYafI/R7NnOVxWqBcdCZJYUrS8fHY5D7/t92/YCL/qXWVnpJDgt5MVIA3fkSBagVFKhv1yvh+gqQ1EQzH6Qh7FTGOd2IQV04fJ9EEnqGNZ77QcTw6YK2Pj05fKmApSv7kQr+OznnnDoroiilaTtcLe1UorCk+2z2j9uvpB0/sk3UL4CxgQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=intel.com; dmarc=pass action=none header.from=intel.com; dkim=pass header.d=intel.com; arc=none Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=intel.com; Received: from DM4PR11MB6020.namprd11.prod.outlook.com (2603:10b6:8:61::19) by SJ5PPFA0B9CD929.namprd11.prod.outlook.com (2603:10b6:a0f:fc02::84a) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.139.11; Sat, 20 Jun 2026 06:27:49 +0000 Received: from DM4PR11MB6020.namprd11.prod.outlook.com ([fe80::3058:1480:e4ac:5765]) by DM4PR11MB6020.namprd11.prod.outlook.com ([fe80::3058:1480:e4ac:5765%4]) with mapi id 15.21.0139.009; Sat, 20 Jun 2026 06:27:49 +0000 Message-ID: <3e24db1b-eeb0-4794-b011-0a38f2ff284f@intel.com> Date: Sat, 20 Jun 2026 14:27:38 +0800 User-Agent: Mozilla Thunderbird Subject: Re: [PATCH v4 1/2] sched/cache: Reduce the overhead of task_cache_work by only scan the visisted cpus To: K Prateek Nayak , Luo Gengkun CC: , , , , , , , , , , References: <20260618064300.80072-1-luogengkun2@huawei.com> <20260618064300.80072-2-luogengkun2@huawei.com> Content-Language: en-US From: "Chen, Yu C" In-Reply-To: Content-Type: text/plain; charset="UTF-8"; format=flowed Content-Transfer-Encoding: 8bit X-ClientProxiedBy: KUZPR04CA0015.apcprd04.prod.outlook.com (2603:1096:d10:25::7) To DM4PR11MB6020.namprd11.prod.outlook.com (2603:10b6:8:61::19) Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DM4PR11MB6020:EE_|SJ5PPFA0B9CD929:EE_ X-MS-Office365-Filtering-Correlation-Id: f4a5c5de-0937-4b62-2fc4-08dece950cd5 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|366016|23010399003|1800799024|376014|7416014|22082099003|18002099003|56012099006|4143699003|5023799004|11063799006; X-Microsoft-Antispam-Message-Info: QwAmKmfH/iqwrkNi6NAx0S2JW7VDhfat52dQhOsJCuCvU1olXv1wAoj/Dr2TeRDqkxVk49zMTmCE+5MtsB3yP0X4KN2UwJzd+M6oej+5w5QUz5nNzYsrYASoXvKc+qVM3npG+mWkJTr2LaXh6BwkjwMWCtUug7inYYLXFndB/hwyvgpaFV3MFvuTnNa6TzNMUuQVJYTLll9JLQDqm5rnwA1wrmbviRCN+TCfn+HcMwrFCvBQb0D3SC1WRfuVhYF9EKug+uxTzqnLUVWzPBP8PTRXcx8SJ0eSWXLHuoghD2jCMEhc8qEF4BOxm1zFfui98hsSsCFeYYjYrSU7M7RQxw3d2ezXNu45BpZBLuDZmu7uOZyeJJG2YAP3uJgrdWFcmixkgb17VNEhDyyh/7zS0IQyFi3x/Ou20RDvNyjrDYar3I59meQFyQhAy7NGYfONN/PaXA/vPgxddams++VqaVT1v4alyR4PfcBzLpCcKjT42QLdPkc4gueEEmZ7rGtJ0wejc7V/CwsReyAo/7GkowFVWDljiwGKtdIGH35S30zPKrz9rogaMVt6XPUco0pKi6ZyjXKqslmXNtBSBWhV7NqR7jUHM2wY+yr1mYZxMDhf/9R6z8w5vnN9s7YN628gTiWacl8yLHufLsZHEqYR5Izis5m6xb5/7UErs+At0dk= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DM4PR11MB6020.namprd11.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(366016)(23010399003)(1800799024)(376014)(7416014)(22082099003)(18002099003)(56012099006)(4143699003)(5023799004)(11063799006);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?NmNLQ01uaEZkZHdjTWRoWlBNbVF3NTBqQ1dnYkNhSHcxV2FtN2ZZYWR5K0tM?= =?utf-8?B?SGs3UmprblA1RFdNd0wycW5aZXc2Wkxza1ZGcU51eVd3eUk1aTNSTFN4UkJY?= =?utf-8?B?TzdFVDVKRHJXSGhpRC9oZitZd0hjdlRPM0gxRmovK1pQZTFGaWI0UTJFYndV?= =?utf-8?B?UjRvd0VRajUzdkRBalJKeURIZExSN0xjbWQ0M2xQano2ZE1QUXdyUk1GUnFP?= =?utf-8?B?UG16dWtlOTZzYS9RRThxN2xWRU9VMUtVSjFqeE5HdXZBbVVCT1hCUXVxSHRX?= =?utf-8?B?VkZ0QkdJY21KWTlFNWIxdkorb3g0bUV0TU9kS1JMNnBVUGUxUWlBcFp4TTRw?= =?utf-8?B?SmFmRzBiZXdZWFk0b3M1d09sRis0b1pISUtIWTVXSmJTMmJ0QUpxQzJybElo?= =?utf-8?B?M09KRisyVmdkcVVOSlhhWm4rQ1RhQVhtMTBjNWowbFZJK24xZXZob1BCMlZU?= =?utf-8?B?VW9QZGNUcFNMcU94RDBualRlTWNqem5ReGlnNFNNREJkellLbDlnaWRtMEgz?= =?utf-8?B?dnhBMEZXRUFGb3M2Uk9lck8xVmlrYjBNUEtSMks3c3N6Qm8yZ0JpVkgzbUJz?= =?utf-8?B?cjFpOElRRGprYS9nR3cxeG1UR2JPd3RDOHBLMERBMmFoMmNlT1VyTDkyZWhh?= =?utf-8?B?VUtMQnpwRXQySlFNMzV0Tkc1OExoVmF1UGNQRHgzYTR3V2FWUlRPOE1wTkFJ?= =?utf-8?B?c2ovQk8vd0NOVFFLaW54bDErWEdyMFpYY085SVRPclhzcTZxN2p3Y0hjc3Zt?= =?utf-8?B?UFJKNHhQaHpOSVlXRlFUaTRwMFgzbDI1KzJmYVlXU0o2aFRNQkFWV3hkcmtq?= =?utf-8?B?S3ZPU08wN3lzdmNwQnBSbE5SbUt2UHR0VWprcWI5RnFnUVQrKzgrQWxCWFIr?= =?utf-8?B?RmF3ZUVKc0pDL3QrM3lqQW5VRnBjS2hHdzhCQmtKSEFwdC96ZXBtSlBKTTVU?= =?utf-8?B?K1NJdDJCMnh1QlpIS1MrM0xycDRqNmZCM04zbnRsQVUyNmppRzBxbHlRU2FW?= =?utf-8?B?b29IcmIyQk5zZmc0cmJGdVlWSzA0R0IxL3E1Znh1UnU5SmFiR1J6N1lTeFhK?= =?utf-8?B?eklNUDRLamlGY0RhR05EVjhCSmxJVnZKYlRkTHpkT3pRaGVRdGs2L00vMUE1?= =?utf-8?B?MWN1OHphdzdmV0l0Sk1yaUFxdjlzS1YyZGpvOEZITTZJVEJwL0sySWlpOUhB?= =?utf-8?B?aXZJWkQ1UEw0N1QzM3dkWGRab3lpRTg5eUZzTWZIMExsZDc5YmNQZVBVV2hy?= =?utf-8?B?VmU4MTN2d3VSc1F4dmhDUXNOMmtYUCtFZTBTaDIxSW9ENEJDY0h3TFRXdml4?= =?utf-8?B?V0ZxQW5HM2szY1JWUlQraHh3cllVKzh5VlBDMVJYOWNkZ0xNOXc2Z29lSXlK?= =?utf-8?B?QjVzODhXaTRia3hhUGdXQ2NiaUo5Rks4WUdUc2pnbUU1dHMwaWhvQzJWL0ZB?= =?utf-8?B?YjExMVdUbndNSTdwTE9LTlJZVXhJaG01aTFTTHV3MXJFRVZhRkhBRC8zcGRY?= =?utf-8?B?ZWpOclo5VmQxV3NWZmRpeGpod1JnRlVrdlkzSjFObHdrWHlxRU5ZUVRVU3Ev?= =?utf-8?B?U1FsNzg4aG13amR4VFEwNDJXVzZMaG5pVDh5MUd1SFk5M3dYWHpabHRCdkZh?= =?utf-8?B?YU1ORGNQZUQ5WW1qcWo2enFVNFEzb1gzL20rL2hlOUoreTFFTkRUeElVclhN?= =?utf-8?B?Q1BTcjNiSnozZlNkcHFOUHIrNHhFVTNDcUMrUHQrWXNRV1FXTzc0YTNjVmdY?= =?utf-8?B?SDVLOTNxWjAzbHFXdWhiZ1V2aUZpMjJKL2UrRFBIdDFkRmxYZmlFZ2tleFFu?= =?utf-8?B?bUN5NVB5VHYyci95dGpRa1NLckUySHRYZVl3RkJReXJhZlpzaElCeGZJZS9a?= =?utf-8?B?N1ZIUUR5U1BER3ZaSGxsMWlrV0plb3hrL3dQZTA1NEl4Um9aWGpKMWpQNGpT?= =?utf-8?B?ampvL21QbUlOY1cralZyK2dySFBlME9LcUpIREkwZlk1aXhHa0hBTE1adXVx?= =?utf-8?B?TUk0a3FtNk5oNHNPQWlwLzF3ZmQ2TlRNc0ZsUnowbVV1WndxSDlUazI1NUhM?= =?utf-8?B?c21nL0p1WHFnU0tNRVU5KzladDdNWU5RU2I4M2hUUVM1ZXdtTk52RTJDbDNB?= =?utf-8?B?bFJ3Y0I2ajBHTjJpbjFkaHNvK3BLS3FDTUlYdmFZZ3NNYjNqUkpWL0w1Qys0?= =?utf-8?B?MldXVXc3UDFxK05jcDRxaERsWGc2NmVwM3NrYmtZd3RSeVMzcWdYbDJXSGZD?= =?utf-8?B?NXFFSTdQWDNQVjltcnhXR2crUWpOTFQrQjR4TTI0eitUaXdWZjBGeHV1T0U4?= =?utf-8?B?aHEzN0E3emZqa2ZFUEhJMHplYmNRMnFrRlRoN0lSZm5pcXo4UEZ1QT09?= X-Exchange-RoutingPolicyChecked: EViH2/Zl2v6x9K8UwA+vD6yIipSI3Ju11jyK9GWM2NpLVGIbvuTbbM13iw5Unm3AwCzO9BKRB+y0l3+OdyXi8jF5nbRR8erhiwKY8EjHyN1S2UsYZF07aGcY2lFheVYK5Urt53wLBGLog7/JeZ/YqMH2DOompKwBnc4BysIXAS7T/1PSRLpBI+0eZt8crUXJ5TLvrqHM1fekG8tZSgdEKqLnSfjuioyXI6k+ajQiXOfHnmfsoiVhFqsZSjk4DuRdBzY0By9uqV0oxY9ls58z+siE3GmQh8PAVRC0bUD35MDEfBCyQ9iHzGXeKtnD/gEvHSf3VPwH7i/UlUGCcIYZrA== X-MS-Exchange-CrossTenant-Network-Message-Id: f4a5c5de-0937-4b62-2fc4-08dece950cd5 X-MS-Exchange-CrossTenant-AuthSource: DM4PR11MB6020.namprd11.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 20 Jun 2026 06:27:49.5339 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: QR9hWVEPYX3jMFPgvxIeVIQfJt2qLc4iTRW/JRkdPSLicVh8n2adpeOPCybafFLzoiNsUOq8SGK08LQxSbRqXA== X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ5PPFA0B9CD929 X-OriginatorOrg: intel.com On 6/19/2026 2:54 PM, K Prateek Nayak wrote: > Hello Luo, > > On 6/18/2026 12:12 PM, Luo Gengkun wrote: >> The overhead of task_cache_work() is high, especially in multi-NUMA >> systems. Currently, task_cache_work() tries to find the pref_llc by >> scanning all CPUs in the system. However, most of these scans are >> meaningless, such as those for CPUs that have never been visited or were >> accessed a long time ago. >> >> To address this problem, introduce visited_cpus to track the visited CPUs >> and evict them once they have not been accessed for a duration exceeding >> llc_epoch_affinity_timeout. With this patch, get_scan_cpumasks() is no >> longer need and is therefore removed. > > Please include performance numbers here. > > I'm not convinced by the amount of benefits this brings. For hackbench, > the only stable improvement I can see from the cover letter is: > > threads 1 4 | 27.758 (1.64%) | 25.711 (1.32%) | 7.37% | IMPROVED > > Since you mention Redis may benefit from this, do you actually have > numbers for Redis, or any other real world workload on your system? > I agree that metrics for Redis and other workloads would be quite helpful. Gengkun seems to share the test data I sent over earlier, yet I haven’t observed a noticeable difference in scores between current node-based scans and visit_cpu scans. Hi Gengkun, I wonder if you conducted Redis tests on an AMD machine, if the Redis performance figures were captured from that environment, it would be helpful if you could share the corresponding dataset. Should there be no noticeable score variance, could you verify whether the total scan count has been reduced? > [..snip..] > >> @@ -1635,11 +1637,21 @@ static inline void __update_mm_sched(struct rq *rq, >> } >> } >> >> -static unsigned long fraction_mm_sched(struct rq *rq, >> - struct sched_cache_time *pcpu_sched) >> +static unsigned long fraction_mm_sched(int cpu, >> + struct mm_struct *mm) >> { >> + struct sched_cache_time *pcpu_sched = >> + per_cpu_ptr(mm->sc_stat.pcpu_sched, cpu); >> + struct rq *rq = cpu_rq(cpu); >> + >> guard(raw_spinlock_irqsave)(&rq->cpu_epoch_lock); >> >> + /* Skip the rq that has not been hit for a long time */ >> + if ((rq->cpu_epoch - pcpu_sched->epoch_timeout) > llc_epoch_affinity_timeout) { >> + cpumask_clear_cpu(cpu, &mm->sc_stat.visited_cpus); > > This makes me think your issue is more with pcpu_sched->runtime > not decaying fast enough? > > If you haven't run in 50us, the pcpu_sched->runtime would already be > decayed by >> 5 but that is not enough? > If I understanding correctly, Gengkun intended to reduce the number of CPUs being scanned. If pcpu_sched->runtime has decayed by 32, the existing code may still check that CPU. And in the visited_cpu proposed here, I suppose those CPUs will be skipped? >> >> for_each_cpu(cpu, cpus) { >> /* XXX sched_cluster_active */ >> @@ -1878,18 +1848,21 @@ static void task_cache_work(struct callback_head *work) >> continue; >> >> for_each_cpu(i, sched_domain_span(sd)) { > > What does your system topology look like? We visit all CPUs of LLC here > so even if a single CPU of LLC is set in visited, you'll still visit all > CPUs of LLC here ... > Yeah, maybe for_each_cpu_and(i, sched_domain_span(sd), cpus) thanks, Chenyu