From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 2D2B13E8C62 for ; Tue, 14 Apr 2026 15:43:10 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=217.140.110.172 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1776181393; cv=none; b=HWueJZsO3r9SPjG9v+Oyw5OBxONHfbcevhvFhEMVE9AtO8E5uKuasOHId6bQt1UHRgZxJq1lEzSXcAqC4Qykiw5/86SJVSyu3NxggtFwLFdeZZ0uYmNEQIu3tSNvE0lHTS8Wwh6RXLtHsqXj2mn+xuo07C4Yh6t9KNxhDrZIKmc= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1776181393; c=relaxed/simple; bh=9EFJwuj6bPxMsIIjcLQlzbHgMH7X2rMiZZNl7Wz5vxY=; h=Message-ID:Date:MIME-Version:Subject:To:Cc:References:From: In-Reply-To:Content-Type; b=gzX1d6HSNFASRjik8lveL0LK4j8deU1RX9kyt7IqRGc3U5sXjQiej0+GizINXhxwwo0pKwb6MK1Ssj+cIRHqOE2EVsub8yBZX5U+geqNidffthbNEGlIxZfy+nNSFmNcRxs+StKZXSFWGRl04qIaGEW3brJ9MeV3haN4uX5Y3rw= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com; spf=pass smtp.mailfrom=arm.com; dkim=pass (1024-bit key) header.d=arm.com header.i=@arm.com header.b=sXVaau8n; arc=none smtp.client-ip=217.140.110.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=arm.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=arm.com header.i=@arm.com header.b="sXVaau8n" Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id ADE192E98; Tue, 14 Apr 2026 08:43:04 -0700 (PDT) Received: from [10.1.27.133] (R40MDXPJLG.cambridge.arm.com [10.1.27.133]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 091523F7B4; Tue, 14 Apr 2026 08:43:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=arm.com; s=foss; t=1776181390; bh=9EFJwuj6bPxMsIIjcLQlzbHgMH7X2rMiZZNl7Wz5vxY=; h=Date:Subject:To:Cc:References:From:In-Reply-To:From; b=sXVaau8nUq/F+f2NFH9pGdxtMSeYOOqbpWziffldeWzDTBAxt+lkwpSzyEP2ESNnp xtFG8LGZtrpR3me46pFRsoulv8suL2MfdWgTvrrnAfJg6IPiJ5+1Ng66PjlqHqAItm DjaHfZk7YA/vWD2oOyVJynD0KJgJhH/g+oAFB41E= Message-ID: <9ca931c1-8d4a-4c98-a962-2e36072db04e@arm.com> Date: Tue, 14 Apr 2026 16:43:06 +0100 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [Question] Sched: Severe scheduling latency (>10s) observed on kernel 6.12 with specific workload To: John Stultz , Xuewen Yan Cc: Peter Zijlstra , Ingo Molnar , Juri Lelli , Vincent Guittot , Steven Rostedt , Benjamin Segall , Mel Gorman , Valentin Schneider , linux-kernel , =?UTF-8?B?546L56eRIChLZSBXYW5nKQ==?= , Xuewen Yan , "hongyu.jin@unisoc.com" , guohua.yan@unisoc.com References: Content-Language: en-GB From: Dietmar Eggemann In-Reply-To: Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit On 13.04.26 21:44, John Stultz wrote: > On Thu, Apr 9, 2026 at 9:01 PM John Stultz wrote: >> >> On Thu, Apr 9, 2026 at 8:31 PM Xuewen Yan wrote: [...] > > So unfortuantely, it looks like I'm going to have to eat my words here. :( > > In the android16-6.12 tree, the behavior has been isolated down to a > backport of commit 6d71a9c61604 ("sched/fair: Fix EEVDF entity > placement bug causing scheduling lag"). > https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=6d71a9c6160479899ee744d2c6d6602a191deb1f > > Specifically the `place_entity(cfs_rq, se, 0);` addition in `reweight_entity()` > > I was assuming android16-6.12 was missing other related changes > causing the bad behavior, but Xuewen pointed out similar problems > could be seen on android17-6.18. I aligned that tree to the > 6.18-stable branch, and could also reproduce it. Additionally, I > tested with our andorid-mainline branch (as of 6.19 the base) and it > showed the same issue. So this does "look" to be upstream related. > > Removing the place_entity() line added in commit 6d71a9c61604 from > reweight_entity() seems to prevent the behavior. > > I've been able to trigger this "bad behavior" on devices using the > rt-app with the configuration[1] Xuewen first provided (putting 10 > spinners per cpu on the bottom 4 cpus in a background v1 cpu cgroup). > Then I run `cyclictest -m -t -a --policy=SCHED_OTHER -b 1000000 -D > 120s` in the root cgroup, and (usually well) within two minutes I'll > see > 1second delays in cyclictest. > > If I remove the cgroup from the rt-app config, the issue doesn't reproduce. > > Unfortuantely I've only been able to reproduce this on device, which > requires the android kernel tree. I've installed an old debian11 image > on x86 QEMU to be able to utilize cpu v1 cgroup support, but I haven't > reproduced the issue there, which is a bit confounding. > > The issue doesn't immediatley trigger, but usually after a few seconds > of normal behavior I'll start to see cyclictest on one or two of the > cpus start to trip larger 100ms+ latencies until it hits the 1second > boundary. > > Late last week I went digging into place_entity() to try to understand > why it was tripping, but wasn't very succesful in narrowing down what > might be going wrong. I did see that the place_entity() call seems to > always be on a non-task se, and its almost always exiting at the `if > (se->rel_deadline)` case. It does go through the lag calculation > conditional, but not always. > > Anyway, I'm going to continue digging into this, but I just wanted to > give folks a heads up in case there were any ideas to explore. > > thanks > -john > > > [1] Xuewen's rt-app config: > { > "tasks" : { > "t0" : { > "instance" : 40, > "priority" : 0, > "cpus" : [ 0, 1, 2, 3 ], > "taskgroup" : "/background", > "loop" : -1, > "run" : 200, > "sleep" : 50 > } > } > } Just guessing ... does Android do more task reweights (set_load_weight(p, true)) ? on tip sched/core: [PATCH v2 0/7] sched: Various reweight_entity() fixes https://lore.kernel.org/r/20260219075840.162631716@infradead.org db4551e2ba34 - sched/fair: Use full weight to __calc_delta() (2026-02-23 Peter Zijlstra) 101f3498b4bd - sched/fair: Revert 6d71a9c61604 ("sched/fair: Fix EEVDF entity placement bug causing scheduling lag") (2026-02-23 Peter Zijlstra) <-- !!! 4823725d9d1d - sched/fair: Increase weight bits for avg_vruntime (2026-02-23 Peter Zijlstra) 9fe89f022c05 - sched/fair: More complex proportional newidle balance (2026-02-23 Peter Zijlstra) v7.0 6e3c0a4e1ad1 - sched/fair: Fix lag clamp (2026-02-23 Peter Zijlstra) bcd74b2ffdd0 - sched/fair: Only set slice protection at pick time (2026-02-23 Peter Zijlstra) b3d99f43c72b - sched/fair: Fix zero_vruntime tracking (2026-02-23 Peter Zijlstra)