From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from CO1PR03CU002.outbound.protection.outlook.com (mail-westus2azon11010001.outbound.protection.outlook.com [52.101.46.1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4011F283C89 for ; Mon, 19 Jan 2026 07:17:53 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=52.101.46.1 ARC-Seal:i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1768807074; cv=fail; b=H1SaKOnAgbSKprUD1kgYPG9VYuS9m/4dTPC2EznNkZuA3AwYajhkH9M+8d6WfhYXiHD4EmkiphZeF1dfEppjAVr7n3FkP+BtaryU3s1M5UCslPiYi9AAYfdW+8OW/0dAxgINChYQjXxgHFx1A41zAlzA1lzONuKPD23oLpWtV04= ARC-Message-Signature:i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1768807074; c=relaxed/simple; bh=DgqNLjSnGP0NCO/ESFzZnwfuIY4jCLr/gdsv8oM11G8=; h=Message-ID:Date:MIME-Version:Subject:To:CC:References:From: In-Reply-To:Content-Type; b=ASqCt22vkJuHVUAd/YPdEubWVtDkPpAaalu5KNLWGiy4BYWvP4AsdM1EutwsOzFqRu8OdTWf8W6w1voOZPPW1jhnPdLJyt+B8N/bsbzOaMtQisHNlV6/cBEZlyQ+XKDXKaTnGgDHEiu6JAJewPHrM36DOTUAhJwFmiIVWluX8ng= ARC-Authentication-Results:i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=T4UGoXw8; arc=fail smtp.client-ip=52.101.46.1 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="T4UGoXw8" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=FsYE7B3JyNTjt1OleJRgtfpP1jgw2rUrLUAohYDBu6wYWZGPsknZIh2QIo6q+Qtw5Fmc5mYxrBLs6Nm7IqwvQ2363e2d1iF4NEs3eieDPM3YwDIJKdTiCZ+HuhvUSyP2PDhM/Qv8xeQTo0Qbr20IXE0DL5RJ6TMdHs96gYWk5SePaOWU8zKiFBboBPhD6oxZet7KTeShpI8qaDIoo65UHN3HfSGqrcrCodnvrRdvStqyg825jemTa4QSHlkEF2vuA6W758XpkZ5+9QADuTiKTEf1MSXXHr3k24JVjqq8CmCsDRkuzoqKdy7vvWfZXbN3m0gmhV6qjr8SDzIAwrrz3w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=Rb8vCfkhYThNZUCNnLMqvE57rE5KFhet/PDXXGMVkns=; b=wB6otUb2iC3QdpkLU/ZkBh34acb6YvgeuUph9uH0SkZUJ+nTeZvRhZj9QWveLoiF/uaeZgQuThxSpvyb//mxdWwR/kMGAPac8TYaAsB7017G7lS8/g9vLhPJWzUY/d1Vx1my537t9ScqnZtp1G5RKOdBb+XkpcFHZXEJ4nbxRFtIbE7htkNp3fUBvpZOOYLWPhCTrsOU0onfDosyBNF3TFVfMdvLQ4YxWD1+u4/UR5+mwJbqKPOHNcFRs+A8YfLALw8hBwy6djV1eFOMRDIuhnPixVCn0Inc/xOomkppnCfmFO3BF47729j9vfEYSVZrjS/b1sgYGnJHDDqoTfwWpg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=ncsu.edu smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=Rb8vCfkhYThNZUCNnLMqvE57rE5KFhet/PDXXGMVkns=; b=T4UGoXw88vd5L6hBe5IkXUYzqKiF6UQZn9tEyF8wrYkl067yLD/l/LhOJZlrtbUhuugpDaXiA9MCCfROKo0fY8LpR78SJAocai+FEwqXVhY2FmMKfWhWhdre9eU62bTF1SuwCKCwKSBLsyoSnP90PwwX0ahHxKP61n7RhvHegE0= Received: from MN2PR04CA0010.namprd04.prod.outlook.com (2603:10b6:208:d4::23) by BY5PR12MB4083.namprd12.prod.outlook.com (2603:10b6:a03:20d::18) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.9520.12; Mon, 19 Jan 2026 07:17:48 +0000 Received: from BL6PEPF00022572.namprd02.prod.outlook.com (2603:10b6:208:d4:cafe::10) by MN2PR04CA0010.outlook.office365.com (2603:10b6:208:d4::23) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.9520.10 via Frontend Transport; Mon, 19 Jan 2026 07:17:45 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=satlexmb07.amd.com; pr=C Received: from satlexmb07.amd.com (165.204.84.17) by BL6PEPF00022572.mail.protection.outlook.com (10.167.249.40) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.9542.4 via Frontend Transport; Mon, 19 Jan 2026 07:17:47 +0000 Received: from Satlexmb09.amd.com (10.181.42.218) by satlexmb07.amd.com (10.181.42.216) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.17; Mon, 19 Jan 2026 01:17:47 -0600 Received: from satlexmb07.amd.com (10.181.42.216) by satlexmb09.amd.com (10.181.42.218) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.17; Sun, 18 Jan 2026 23:17:47 -0800 Received: from [10.85.36.78] (10.180.168.240) by satlexmb07.amd.com (10.181.42.216) with Microsoft SMTP Server id 15.2.2562.17 via Frontend Transport; Sun, 18 Jan 2026 23:17:42 -0800 Message-ID: <47e98557-e898-4302-b89d-c2f62b84ce38@amd.com> Date: Mon, 19 Jan 2026 12:47:36 +0530 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [PATCH v7 3/3] sched/fair: Allocate both cfs_tg_state with percpu allocator To: Zecheng Li , Ingo Molnar , "Peter Zijlstra" , Juri Lelli , "Vincent Guittot" CC: Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Valentin Schneider , Rik van Riel , Chris Mason , Madadi Vineeth Reddy , Xu Liu , Blake Jones , Josh Don , Nilay Vaish , , Zecheng Li References: <20260118033431.158464-1-zli94@ncsu.edu> <20260118033431.158464-4-zli94@ncsu.edu> Content-Language: en-US From: K Prateek Nayak In-Reply-To: <20260118033431.158464-4-zli94@ncsu.edu> Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BL6PEPF00022572:EE_|BY5PR12MB4083:EE_ X-MS-Office365-Filtering-Correlation-Id: 8998fdc1-87e9-47d5-f98e-08de572ad95a X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|82310400026|7416014|376014|1800799024|36860700013; X-Microsoft-Antispam-Message-Info: =?utf-8?B?bVlkeDh0LzhOTzk1UkRRaFYyb3FpMDArSC8rODFUM3E1L3Myb2huWG5GN3Za?= =?utf-8?B?VG4yblppQXZOUnpXRUFmSzlwdlcwUEREQk1pSlZ1THpxbzBmckhDRjBmdnBW?= =?utf-8?B?V0hMQ2VNTGd4VVRtaGJ3WXhTS3c1bEIzcEFjSzVDSnRwZ0pQSWtQWnI5QlNQ?= =?utf-8?B?K1hBeVovcmkvUklxa1l3TWFQbjJsdFRUS01vSWpPWVQ0bTkyQW1yMkllbXov?= =?utf-8?B?bnVHUkxRZCtLemVkUXF1dTg5T3M5UytCbU1MYTBublFaczR2dGN2OXBUL3JS?= =?utf-8?B?RGgvcFh3WGcvTzQ4dCtFMm4yUHpDZ0s5UVhjWkR3MmNEVmJmSlY3cXlPVFZX?= =?utf-8?B?UTZDMERMWXBCT1F0S2JieWhQaVkzYldqenlwWnhxMUd2aU9xRjVzbTZRdlRz?= =?utf-8?B?U2JWakdwQ284SndnSVNadjhWSWRSdWtLUDZxNGJHY1NIS0hzRVVSbE5mYWF6?= =?utf-8?B?ZVBkK0dYUmw4OUo1ZWZuSEE2ZW5EazczdVpKdEpKUFd6ZE5GTFVBWUs0VWpP?= =?utf-8?B?TlFrMUNZeGVyRkhlUUxmeFZ6SnBaeFRvQlJHazBGYlRJN2lyYncyaHF0RWdw?= =?utf-8?B?anVobnY2TFFDM2JEUStaSFZRamtvWFY1b3htQlJQTldzYmJPYTZUSllyM1hm?= =?utf-8?B?VVZOWHY3enVPVUtXd1ZGa1JiUi9HWklHSCtxK0NyM3J1dm0vV2l4NzBnNzZV?= =?utf-8?B?bVRhRWpXUWFYOGl1bjAxajgzeExSSnA0TXdVdDhTVUttaU9FOW9COEJYV1Bu?= =?utf-8?B?UEp6YmVOV2tqZzZNR1hSZXQwNVVlc0hwQlI1cTBiWDhhbUo1L1ovTXpNWnhW?= =?utf-8?B?QysvVVBQRldJNWRtMy9LellJOWFaQmFMN1B1c2pTWjlnYkExVk50bTEzRGlw?= =?utf-8?B?WmJ1Y1VVQWllVHpSaFczRGp5VG93bXIzb3NWQzQ0L3BscTBSeXIweWtKYTFQ?= =?utf-8?B?MnJSNTljWW1aMWpFYWl4UU5KUG5EeWlPeEFhUFpXYnc1N0syYm96b3IzZ2c0?= =?utf-8?B?MVBUejRJbGNud002RjgxTnN1OE5WUHBzTUFmM01EakpJU1lHRzBrbUZaTFhV?= =?utf-8?B?MFowV0xldmx6MGZoQzRWUnhzM1J2YXFJcnFhRUY1UHFQa3JIdTR1K2QwcEJ4?= =?utf-8?B?czEwNURLaTR1SGNMbUJjTStra0J0cjA4cHZ0K0tOM0oyZ0NRNWUvS1VNcXNt?= =?utf-8?B?bkJROEVnbmNvRFNtRVNxZGZkTWdpdFJRcGczNHQ5YzZQRnFPQUNjaTJZTWlO?= =?utf-8?B?N3phK0VjbkpReEZjQTY3bTZMVVp1Ui81bmhybTBtd0FNQXplS2lMbVd1SE4z?= =?utf-8?B?V29UZkNnbmFaODh5Q1FXbllGa1pjeElOZm5sY01uYUs1QURvYWl1dTAva0dt?= =?utf-8?B?YkhYVmNzR3JVK0U3MkU4U29ENU43RGtrM1FNWlQrYVdCREtVVENIWU1GcVZK?= =?utf-8?B?dW5kSmhFQWNJWndVREhHcmREMFpzczJraHdVZStkSmtweForMzBrSy9DTmxV?= =?utf-8?B?Y21sWFZ3UlhTcHZuak4yYmlnd2tFVnFZU2tvK2ZGVW5aMzZyVG9ObXJYNWhI?= =?utf-8?B?TnlEc0pJZE9XL3o4M3ZzQkRrOERqUDhWTEpBSm9ET1YyN2pyVGxmbU5qaWp4?= =?utf-8?B?ZHRzZUlJSDU4NTlQandtclhVcWM5eFJFM1hLU2lORG1Va0VIbEZKTVNCWUls?= =?utf-8?B?a2l0aWYyZmp2VjB6ZzA1TWhnV2pPZC9pRGpzclpKcjFZSXU2dmI2dGFNWnFR?= =?utf-8?B?d0RPS3lHWDVRQTlEcWRDQzY0akgxN1FCbXJwV2VuMEhaemI1bHdSUkgyakg1?= =?utf-8?B?Qkx5RHhNVERtYkQ4R1pETXlPcURZSDBmNFFaM3hRZ2dTaUNmd0U0MjZvLzI0?= =?utf-8?B?YkpjTVNmc3dNelF4Z2hraERNSVRkTDRmNGJ6ditNTm1JeVhjWnIxVWdZWnVX?= =?utf-8?B?Rmp3ZkJ1RGZPZnVTMkU2Z0NKcUpUZ2lad0RRZy9YOUE3akVaQXAwMUVNdHFa?= =?utf-8?B?UUVJMEVRdnBqZkp6VmdWK2QrZFdjVThMdW5ZUUIzanRaUG5xRUU0dWJ6YUxp?= =?utf-8?B?U0Jzb044RU8wN2dQTnpsVFlKSjdxTk9wZHhiSzgzRmVGS2c2aGVFbTYyRk1m?= =?utf-8?B?Z2JaTHNaMGJmRncrYnVOTDczZ1VPZ3hOaHVNSzJXQ0NqTm1pMC91dnp1RXl0?= =?utf-8?Q?rO9arNbV2gyWL1kYb5jxLG8bD1nAO9n2Z5H07YOQZGEe?= X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:satlexmb07.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(82310400026)(7416014)(376014)(1800799024)(36860700013);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 19 Jan 2026 07:17:47.9624 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 8998fdc1-87e9-47d5-f98e-08de572ad95a X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[satlexmb07.amd.com] X-MS-Exchange-CrossTenant-AuthSource: BL6PEPF00022572.namprd02.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: BY5PR12MB4083 Hello Zecheng, On 1/18/2026 9:04 AM, Zecheng Li wrote: > --- a/kernel/sched/core.c > +++ b/kernel/sched/core.c > @@ -8549,7 +8549,7 @@ static struct kmem_cache *task_group_cache __ro_after_init; > > void __init sched_init(void) > { > - unsigned long ptr = 0; > + unsigned long __maybe_unused ptr = 0; Since "ptr" is now only used for CONFIG_RT_GROUP_SCHED ... > int i; > > /* Make sure the linker didn't screw up */ > @@ -8565,33 +8565,24 @@ void __init sched_init(void) > wait_bit_init(); > > #ifdef CONFIG_FAIR_GROUP_SCHED > - ptr += nr_cpu_ids * sizeof(void **); > -#endif > -#ifdef CONFIG_RT_GROUP_SCHED > - ptr += 2 * nr_cpu_ids * sizeof(void **); > -#endif > - if (ptr) { > - ptr = (unsigned long)kzalloc(ptr, GFP_NOWAIT); > + root_task_group.cfs_rq = &runqueues.cfs; > > -#ifdef CONFIG_FAIR_GROUP_SCHED > - root_task_group.cfs_rq = (struct cfs_rq **)ptr; > - ptr += nr_cpu_ids * sizeof(void **); > - > - root_task_group.shares = ROOT_TASK_GROUP_LOAD; > - init_cfs_bandwidth(&root_task_group.cfs_bandwidth, NULL); > + root_task_group.shares = ROOT_TASK_GROUP_LOAD; > + init_cfs_bandwidth(&root_task_group.cfs_bandwidth, NULL); > #endif /* CONFIG_FAIR_GROUP_SCHED */ > #ifdef CONFIG_EXT_GROUP_SCHED > - scx_tg_init(&root_task_group); > + scx_tg_init(&root_task_group); > #endif /* CONFIG_EXT_GROUP_SCHED */ > #ifdef CONFIG_RT_GROUP_SCHED > - root_task_group.rt_se = (struct sched_rt_entity **)ptr; > - ptr += nr_cpu_ids * sizeof(void **); > + ptr += 2 * nr_cpu_ids * sizeof(void **); > + ptr = (unsigned long)kzalloc(ptr, GFP_NOWAIT); > + root_task_group.rt_se = (struct sched_rt_entity **)ptr; > + ptr += nr_cpu_ids * sizeof(void **); > > - root_task_group.rt_rq = (struct rt_rq **)ptr; > - ptr += nr_cpu_ids * sizeof(void **); > + root_task_group.rt_rq = (struct rt_rq **)ptr; > + ptr += nr_cpu_ids * sizeof(void **); Can we also optimize the CONFIG_RT_GROUP_SCHED stuff in the same way although I'm assuming the benefit is far less since they aren't accessed as frequently as the cfs bits. Something like: (Only build and boot tested on top of your series) diff --git a/kernel/sched/autogroup.c b/kernel/sched/autogroup.c index 954137775f38..7135f19d5fa2 100644 --- a/kernel/sched/autogroup.c +++ b/kernel/sched/autogroup.c @@ -52,7 +52,6 @@ static inline void autogroup_destroy(struct kref *kref) #ifdef CONFIG_RT_GROUP_SCHED /* We've redirected RT tasks to the root task group... */ - ag->tg->rt_se = NULL; ag->tg->rt_rq = NULL; #endif sched_release_group(ag->tg); @@ -109,7 +108,6 @@ static inline struct autogroup *autogroup_create(void) * the policy change to proceed. */ free_rt_sched_group(tg); - tg->rt_se = root_task_group.rt_se; tg->rt_rq = root_task_group.rt_rq; #endif /* CONFIG_RT_GROUP_SCHED */ tg->autogroup = ag; diff --git a/kernel/sched/core.c b/kernel/sched/core.c index 80e8f4eb3f87..cad8e8a1f519 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -8549,7 +8549,6 @@ static struct kmem_cache *task_group_cache __ro_after_init; void __init sched_init(void) { - unsigned long __maybe_unused ptr = 0; int i; /* Make sure the linker didn't screw up */ @@ -8574,14 +8573,7 @@ void __init sched_init(void) scx_tg_init(&root_task_group); #endif /* CONFIG_EXT_GROUP_SCHED */ #ifdef CONFIG_RT_GROUP_SCHED - ptr += 2 * nr_cpu_ids * sizeof(void **); - ptr = (unsigned long)kzalloc(ptr, GFP_NOWAIT); - root_task_group.rt_se = (struct sched_rt_entity **)ptr; - ptr += nr_cpu_ids * sizeof(void **); - - root_task_group.rt_rq = (struct rt_rq **)ptr; - ptr += nr_cpu_ids * sizeof(void **); - + root_task_group.rt_rq = &runqueues.rt; #endif /* CONFIG_RT_GROUP_SCHED */ init_defrootdomain(); diff --git a/kernel/sched/rt.c b/kernel/sched/rt.c index 0a9b2cd6da72..963004905a7d 100644 --- a/kernel/sched/rt.c +++ b/kernel/sched/rt.c @@ -201,26 +201,16 @@ void unregister_rt_sched_group(struct task_group *tg) if (!rt_group_sched_enabled()) return; - if (tg->rt_se) + if (!is_root_task_group(tg)) destroy_rt_bandwidth(&tg->rt_bandwidth); } void free_rt_sched_group(struct task_group *tg) { - int i; - if (!rt_group_sched_enabled()) return; - for_each_possible_cpu(i) { - if (tg->rt_rq) - kfree(tg->rt_rq[i]); - if (tg->rt_se) - kfree(tg->rt_se[i]); - } - - kfree(tg->rt_rq); - kfree(tg->rt_se); + free_percpu(tg->rt_rq); } void init_tg_rt_entry(struct task_group *tg, struct rt_rq *rt_rq, @@ -234,9 +224,6 @@ void init_tg_rt_entry(struct task_group *tg, struct rt_rq *rt_rq, rt_rq->rq = rq; rt_rq->tg = tg; - tg->rt_rq[cpu] = rt_rq; - tg->rt_se[cpu] = rt_se; - if (!rt_se) return; @@ -252,42 +239,32 @@ void init_tg_rt_entry(struct task_group *tg, struct rt_rq *rt_rq, int alloc_rt_sched_group(struct task_group *tg, struct task_group *parent) { - struct rt_rq *rt_rq; + struct rt_tg_state __percpu *state; struct sched_rt_entity *rt_se; + struct rt_rq *rt_rq; int i; if (!rt_group_sched_enabled()) return 1; - tg->rt_rq = kcalloc(nr_cpu_ids, sizeof(rt_rq), GFP_KERNEL); - if (!tg->rt_rq) - goto err; - tg->rt_se = kcalloc(nr_cpu_ids, sizeof(rt_se), GFP_KERNEL); - if (!tg->rt_se) + state = alloc_percpu_gfp(struct rt_tg_state, GFP_KERNEL); + if (!state) goto err; + tg->rt_rq = &state->rt_rq; init_rt_bandwidth(&tg->rt_bandwidth, ktime_to_ns(global_rt_period()), 0); for_each_possible_cpu(i) { - rt_rq = kzalloc_node(sizeof(struct rt_rq), - GFP_KERNEL, cpu_to_node(i)); - if (!rt_rq) - goto err; - - rt_se = kzalloc_node(sizeof(struct sched_rt_entity), - GFP_KERNEL, cpu_to_node(i)); - if (!rt_se) - goto err_free_rq; + rt_rq = tg_rt_rq(tg, i); + rt_se = rt_rq_se(rt_rq); init_rt_rq(rt_rq); rt_rq->rt_runtime = tg->rt_bandwidth.rt_runtime; - init_tg_rt_entry(tg, rt_rq, rt_se, i, parent->rt_se[i]); + init_tg_rt_entry(tg, rt_rq, rt_se, i, tg_rt_se(parent, i)); } return 1; -err_free_rq: - kfree(rt_rq); err: return 0; } @@ -510,7 +487,7 @@ static inline struct task_group *next_task_group(struct task_group *tg) #define for_each_rt_rq(rt_rq, iter, rq) \ for (iter = &root_task_group; \ - iter && (rt_rq = iter->rt_rq[cpu_of(rq)]); \ + iter && (rt_rq = tg_rt_rq(iter, cpu_of(rq))); \ iter = next_task_group(iter)) #define for_each_sched_rt_entity(rt_se) \ @@ -528,11 +505,7 @@ static void sched_rt_rq_enqueue(struct rt_rq *rt_rq) { struct task_struct *donor = rq_of_rt_rq(rt_rq)->donor; struct rq *rq = rq_of_rt_rq(rt_rq); - struct sched_rt_entity *rt_se; - - int cpu = cpu_of(rq); - - rt_se = rt_rq->tg->rt_se[cpu]; + struct sched_rt_entity *rt_se = rt_rq_se(rt_rq); if (rt_rq->rt_nr_running) { if (!rt_se) @@ -547,10 +520,7 @@ static void sched_rt_rq_enqueue(struct rt_rq *rt_rq) static void sched_rt_rq_dequeue(struct rt_rq *rt_rq) { - struct sched_rt_entity *rt_se; - int cpu = cpu_of(rq_of_rt_rq(rt_rq)); - - rt_se = rt_rq->tg->rt_se[cpu]; + struct sched_rt_entity *rt_se = rt_rq_se(rt_rq); if (!rt_se) { dequeue_top_rt_rq(rt_rq, rt_rq->rt_nr_running); @@ -586,7 +556,7 @@ static inline const struct cpumask *sched_rt_period_mask(void) static inline struct rt_rq *sched_rt_period_rt_rq(struct rt_bandwidth *rt_b, int cpu) { - return container_of(rt_b, struct task_group, rt_bandwidth)->rt_rq[cpu]; + return tg_rt_rq(container_of(rt_b, struct task_group, rt_bandwidth), cpu); } static inline struct rt_bandwidth *sched_rt_bandwidth(struct rt_rq *rt_rq) @@ -2563,7 +2533,7 @@ static int task_is_throttled_rt(struct task_struct *p, int cpu) struct rt_rq *rt_rq; #ifdef CONFIG_RT_GROUP_SCHED // XXX maybe add task_rt_rq(), see also sched_rt_period_rt_rq - rt_rq = task_group(p)->rt_rq[cpu]; + rt_rq = tg_rt_rq(task_group(p), cpu); WARN_ON(!rt_group_sched_enabled() && rt_rq->tg != &root_task_group); #else rt_rq = &cpu_rq(cpu)->rt; @@ -2752,7 +2722,7 @@ static int tg_set_rt_bandwidth(struct task_group *tg, tg->rt_bandwidth.rt_runtime = rt_runtime; for_each_possible_cpu(i) { - struct rt_rq *rt_rq = tg->rt_rq[i]; + struct rt_rq *rt_rq = tg_rt_rq(tg, i); raw_spin_lock(&rt_rq->rt_runtime_lock); rt_rq->rt_runtime = rt_runtime; diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h index 8ad1780edfe8..62d7c89c74d8 100644 --- a/kernel/sched/sched.h +++ b/kernel/sched/sched.h @@ -488,8 +488,7 @@ struct task_group { #endif /* CONFIG_FAIR_GROUP_SCHED */ #ifdef CONFIG_RT_GROUP_SCHED - struct sched_rt_entity **rt_se; - struct rt_rq **rt_rq; + struct rt_rq __percpu *rt_rq; struct rt_bandwidth rt_bandwidth; #endif @@ -2228,6 +2227,41 @@ static inline struct sched_entity *cfs_rq_se(struct cfs_rq *cfs_rq) } #endif +#ifdef CONFIG_RT_GROUP_SCHED +struct rt_tg_state { + struct rt_rq rt_rq; + struct sched_rt_entity rt_se; +} __no_randomize_layout; + +/* Access a specific CPU's rt_rq from a task group */ +static inline struct rt_rq *tg_rt_rq(struct task_group *tg, int cpu) +{ + return per_cpu_ptr(tg->rt_rq, cpu); +} + +static inline struct sched_rt_entity *tg_rt_se(struct task_group *tg, int cpu) +{ + struct rt_tg_state *state; + + if (is_root_task_group(tg)) + return NULL; + + state = container_of(tg_rt_rq(tg, cpu), struct rt_tg_state, rt_rq); + return &state->rt_se; +} + +static inline struct sched_rt_entity *rt_rq_se(struct rt_rq *rt_rq) +{ + struct rt_tg_state *state; + + if (is_root_task_group(rt_rq->tg)) + return NULL; + + state = container_of(rt_rq, struct rt_tg_state, rt_rq); + return &state->rt_se; +} +#endif /* CONFIG_RT_GROUP_SCHED */ + /* Change a task's cfs_rq and parent entity if it moves across CPUs/groups */ static inline void set_task_rq(struct task_struct *p, unsigned int cpu) { @@ -2250,8 +2284,8 @@ static inline void set_task_rq(struct task_struct *p, unsigned int cpu) */ if (!rt_group_sched_enabled()) tg = &root_task_group; - p->rt.rt_rq = tg->rt_rq[cpu]; - p->rt.parent = tg->rt_se[cpu]; + p->rt.rt_rq = tg_rt_rq(tg, cpu); + p->rt.parent = tg_rt_se(tg, cpu); #endif /* CONFIG_RT_GROUP_SCHED */ } -- Thanks and Regards, Prateek