From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-pj1-f45.google.com (mail-pj1-f45.google.com [209.85.216.45]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9A01220F06E for ; Tue, 14 Jan 2025 23:01:08 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.45 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1736895671; cv=none; b=ibwpbUEfiFKtOlGsnzDYeVCKNwjIDKZHA7tvlQviylV84HIBJzJctPi2nKW9e3Z0BSu759TNRNmgQlky1CNCnMLwILr4BlmxVrc0iUTg4P6ZuFJTprapFc6U97/cTwvJo5NHmwcRkD5SghSD7a4RUKtpY/vttZh1xEYVMUVUFss= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1736895671; c=relaxed/simple; bh=o6BxDKen0JwrAtMbBRVCbj++vv91HJR6QAAKLkID72k=; h=Message-ID:Date:MIME-Version:Subject:To:Cc:References:From: In-Reply-To:Content-Type; b=kVsbbppvgC5ElI9YweAn5U80aQ5rjqbQNtWBicakX6xk+n9iezu99JBPNa1G3pis//DP/vOnYt4IynNeB+j2yS4A4d4l2tIbpccJ3V975wmvqAeD+m6vxmNy/Q0ddAXzGntcqXS2AanLmHuAkH6vS/oxS4jiYVa+goKa7jDzC7c= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=UK3BvMZs; arc=none smtp.client-ip=209.85.216.45 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="UK3BvMZs" Received: by mail-pj1-f45.google.com with SMTP id 98e67ed59e1d1-2eed82ca5b4so10035451a91.2 for ; Tue, 14 Jan 2025 15:01:08 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1736895668; x=1737500468; darn=vger.kernel.org; h=content-transfer-encoding:in-reply-to:from:content-language :references:cc:to:subject:user-agent:mime-version:date:message-id :from:to:cc:subject:date:message-id:reply-to; bh=+gLXVPwvkeIrx6hClcLUXBm6KrPEc09QiotrThJYX3w=; b=UK3BvMZsNlJOFJ7/ayptGbFk8RaGX+RIxJr4iW3y/7mttbOe1eL9rq6tlTgIBBpZm3 XqbTwiJSXlnEuW+3JDwcAfHIi7y/pxdY9CGZAYW1sRnHrIyMoPn3rjsTI1ZS1Ike7+St WVrN9QMRepu4g2SQuGnj4KNlsw8IJ4Nu8929J5bvQofyveFMSXx5cqD5pONyqm2XQ/go cu7NkjhPtDKGF9fzEtzSyC6MAEkJSIjhUiQ7I2qHlmlHKO8jDHmZtO+Po/KBIrgRWdTV qEaDnVAfQ4RMbJUUiuq/rRFFy5rQgA1qkpOu5lzgTamk1erYfjS8TAys7tx9FFSxcidl Y8HQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1736895668; x=1737500468; h=content-transfer-encoding:in-reply-to:from:content-language :references:cc:to:subject:user-agent:mime-version:date:message-id :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=+gLXVPwvkeIrx6hClcLUXBm6KrPEc09QiotrThJYX3w=; b=DBBID+CvPayDYTJUHSwVZDVnP1g2ZEYGY1fiGpsXdcUtgVKNdimOHtIMJQOg2B8Tjl R3Qbfn0LLH1grPBCC8MQ7aO9NbGDKoUAqFGf0efuOR5MLtnLHUQ3XtxyuMNxB9ZVOi3W ConWaeK/dMHxhN2C0DO1IcvjXOCOTeOV/+m1NVoZEr4d6BuTxgvTpZsJyeRc8A0QchBX opcrTdnt76tNgGT4CRfAt4QKDP62I87dF0QNUHC8xtVhOem37Kz1cxTMVDIGfbD45fsh lDrQ4iaD598s9LSTAOGLE/eh209c5PrWpfzqVRItkbhaQgOinwVjeSfQqMqr/6qB5gpQ wyjg== X-Forwarded-Encrypted: i=1; AJvYcCXooZzfDyR+FS01pZuS/itPl8230aoRbJIN4gX2Z5P0dHgOCDU/456i3H5DUEgeG4vQSfclA3iJBp/dn+Q=@vger.kernel.org X-Gm-Message-State: AOJu0YwltR74vSXbBt//iSYHuYBn4XQUrNML8M2xPasUZdfrbDrSZJqe ttKx/gSCWyH8YGJgpI6jMV2BxN4XonrLxAhVoIXsAfTdYyUgkNpo X-Gm-Gg: ASbGncuMHrqt7pBCMbRV34urCOlOL4CXLaP+5la5mo2aXe3eHmckm8AkpwftFWAkbhL LoGTbHcpeum/sjTGexg5mtXp5WuiGuB8sJL+Yfr1iZtzT4FKj5oVd7KhtOYzbQi12ggQYRBDMsg mecHj7uiR03WKZ38rKjKyQ2wFMIH4pRhVPZAZRU/Icszjo6guHweMdgnh2wdGbE6Tx8E3TPfpfs nuoevuMXepQ+D3lsdLAGjTzlRdzzWw0BUAvCvBt88WEJyvzwg2g9WW6hGUWbyRAQp+C X-Google-Smtp-Source: AGHT+IFXoKTCrZavrzoyfOcgmMIva36WyhJKMzQd+d+fQ1H2EI+Cq1/T0WVereFo+HcyBOlTD4jQXw== X-Received: by 2002:a17:90b:3cd0:b0:2ee:5bc9:75c3 with SMTP id 98e67ed59e1d1-2f548f09e88mr35278897a91.5.1736895667695; Tue, 14 Jan 2025 15:01:07 -0800 (PST) Received: from [10.69.47.104] ([192.19.223.252]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-21a9f252518sm70786735ad.214.2025.01.14.15.01.06 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Tue, 14 Jan 2025 15:01:07 -0800 (PST) Message-ID: <0e69b462-1666-4a17-bde3-758d0d81a7d5@gmail.com> Date: Tue, 14 Jan 2025 15:01:04 -0800 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [RFC] sched/deadline: only mark active cpu as free To: Juri Lelli Cc: Ingo Molnar , Peter Zijlstra , Vincent Guittot , Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Valentin Schneider , Florian Fainelli , linux-kernel@vger.kernel.org References: <20250110233010.2339521-1-opendmb@gmail.com> Content-Language: en-US From: Doug Berger In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit On 1/13/2025 2:54 AM, Juri Lelli wrote: > Hi Doug, > > On 10/01/25 15:30, Doug Berger wrote: >> There is a hazard in the deadline scheduler where an offlined CPU >> can have its free_cpus bit left set in the def_root_domain when >> the schedutil cpufreq governor is used. This can allow a deadline >> thread to be pushed to the runqueue of a powered down CPU which >> breaks scheduling. >> >> This commit works around the issue by only setting the free_cpus >> bit for a CPU when it is "active". It is likely that the ordering >> of sched_set_rq_online() and set_cpu_active() at the end of the >> sched_cpu_deactivate() function should be revisited if this >> approach has merit. >> >> Signed-off-by: Doug Berger >> --- >> This problem appears to have been initially introduced by commit >> 120455c514f7 ("sched: Fix hotplug vs CPU bandwidth control") which >> moved the set_rq_offline() handling from sched_cpu_dying() to >> sched_cpu_deactivate(). The original sequence allowed the free_cpus >> bit to be forcibly cleared in the def_root_domains after all of the >> scheduler dust settled. The new location makes the >> sched_set_rq_offline() essentially meaningless for the deadline >> scheduler since the managing of changed scheduling domains happens >> later. >> >> There are likely many different approaches to address this issue >> and I'm hopeful that somone more familiar with the scheduler than >> I can propose a better solution than the one suggested here. >> >> Thank you for reading this far. Any advice is appreciated. > > Thanks a lot for the detailed analysis! Thanks for the review! > > I actually fear that the issue is due to the cpudl_clear_freecpu() call > in rq_offline_dl() being racy, as we don't hold cp->lock while calling > that. So, I think your solution below might be almost correct. I am > thinking we should do something similar in cpudl_set() and remove cpudl_ > {set,clear}_freecpu() calls altogether. > > What do you think? If agree, care to update your patch please? :) As I noted, I'm not real keen on what I proposed here. In particular I am wary of introducing any extra processing to normal deadline scheduler execution paths. I am about to submit an alternative that only affects paths when the topology is changing that I hope you will take another look at ;). I am still very much open to someone saying something like "You know we could probably bring the def_root_domain of the onlining CPU online earlier so that set_rq_offline() gets called for it when attaching to the new scheduling domain." I just don't feel comfortable enough yet with my understanding of the scheduler to know what all of the implications of such a change would be. > > Best, > Juri Thanks again! Doug > >> kernel/sched/cpudeadline.c | 3 ++- >> 1 file changed, 2 insertions(+), 1 deletion(-) >> >> diff --git a/kernel/sched/cpudeadline.c b/kernel/sched/cpudeadline.c >> index 95baa12a1029..6896bbe0e9ae 100644 >> --- a/kernel/sched/cpudeadline.c >> +++ b/kernel/sched/cpudeadline.c >> @@ -195,7 +195,8 @@ void cpudl_clear(struct cpudl *cp, int cpu) >> cp->elements[cpu].idx = IDX_INVALID; >> cpudl_heapify(cp, old_idx); >> >> - cpumask_set_cpu(cpu, cp->free_cpus); >> + if (cpu_active(cpu)) >> + cpumask_set_cpu(cpu, cp->free_cpus); >> } >> raw_spin_unlock_irqrestore(&cp->lock, flags); >> } >> -- >> 2.34.1 >> >