From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 287473ECBCE for ; Thu, 2 Apr 2026 14:58:05 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=217.140.110.172 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1775141887; cv=none; b=N03fdisbPLAWgXBew5gkqTNv9jc3YbV4Q/SLm7yLPsLkQFohi2mljJKW+uOn+s6zJCHPkJKvsRish+BFtam5N3XPPLmoq9SeYKrYQfqyPJQzAsKtdmoBFYSZtCr2csC5aBRLQ7w/NRP4IPrp3DcqRY3E5kFZSRrAmdjF8GuEYw0= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1775141887; c=relaxed/simple; bh=R6jU49z59sDMy0ety05rl4ce3tuYUOMgaLkpzWKubQo=; h=Message-ID:Date:MIME-Version:Subject:To:Cc:References:From: In-Reply-To:Content-Type; b=QivcSwwVCLBxYWF9eBSMlr2Mtko0F2LnHjHSeCkv/oDThPtsoRUxOSCh2MEXzMGY3R5lSwhExBeDpGYPYDkgQWg5bguVe2/o46/GF9TEfi4iIpe/2F5fBRrFX7nvTRXEGL5791k+fh5HbQgdvUeY6FiKVzag2aqfnvwnBLfgE9A= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com; spf=pass smtp.mailfrom=arm.com; dkim=pass (1024-bit key) header.d=arm.com header.i=@arm.com header.b=ZTJeN1Xt; arc=none smtp.client-ip=217.140.110.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=arm.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=arm.com header.i=@arm.com header.b="ZTJeN1Xt" Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id AA6BC32E3; Thu, 2 Apr 2026 07:57:59 -0700 (PDT) Received: from [192.168.178.100] (usa-sjc-mx-foss1.foss.arm.com [172.31.20.19]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 71B5C3F915; Thu, 2 Apr 2026 07:58:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=arm.com; s=foss; t=1775141885; bh=R6jU49z59sDMy0ety05rl4ce3tuYUOMgaLkpzWKubQo=; h=Date:Subject:To:Cc:References:From:In-Reply-To:From; b=ZTJeN1XtD6rQCg+d6X8jEjR2PbP1Ml3IDVFG+yaICSWA4T4qF7rifEDvxivLvz1ij dwAEssNWjWZWi2M+aE8pPHDNW89uMvxDdOC3/plFdLxqEJCUxlCt0PQbqfds09Yb8H edDcUyBgaD3J39kE5YYBZEcRKcQ7oUpyJSyE5WB4= Message-ID: <89c464b5-908d-4817-87f4-d8cb905495a4@arm.com> Date: Thu, 2 Apr 2026 16:58:02 +0200 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [Question] Sched: Severe scheduling latency (>10s) observed on kernel 6.12 with specific workload To: Xuewen Yan Cc: Vincent Guittot , John Stultz , Peter Zijlstra , Ingo Molnar , Juri Lelli , Steven Rostedt , Benjamin Segall , Mel Gorman , Valentin Schneider , linux-kernel , =?UTF-8?B?546L56eRIChLZSBXYW5nKQ==?= , Xuewen Yan , "hongyu.jin@unisoc.com" , guohua.yan@unisoc.com References: <1c0e5a70-22b7-4cbd-8ca9-0a7da27ed7a9@arm.com> From: Dietmar Eggemann Content-Language: en-GB In-Reply-To: Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit On 02.04.26 07:16, Xuewen Yan wrote: > On Wed, Apr 1, 2026 at 9:00 PM Dietmar Eggemann > wrote: >> >> On 01.04.26 12:48, Xuewen Yan wrote: >>> On Wed, Apr 1, 2026 at 6:05 PM Vincent Guittot >>> wrote: >>>> >>>> On Wed, 1 Apr 2026 at 08:04, Xuewen Yan wrote: >>>>> >>>>> On Wed, Apr 1, 2026 at 12:25 PM John Stultz wrote: >>>>>> >>>>>> On Tue, Mar 31, 2026 at 7:32 PM Xuewen Yan wrote: >>>>>>> >>>>>>> Dear Linux maintainers and reviewers, >>>>>>> >>>>>>> I am writing to report a severe scheduling latency issue we recently >>>>>>> discovered on Linux Kernel 6.12. >>>>>>> >>>>>>> Issue Description >>>>>>> >>>>>>> We observed that when running a specific background workload pattern, >>>>>>> certain tasks experience excessive scheduling latency. The delay from >>>>>>> the runnable state to running on the CPU exceeds 10 seconds, and in >>>>>>> extreme cases, it reaches up to 100 seconds. >>>>>>> >>>>>>> Environment Details >>>>>>> >>>>>>> Kernel Version: 6.12.58-android16-6-g3835fd28159d-ab000018-4k >>>>>>> Architecture: [ ARM64] >>>>>>> Hardware: T7300 >> >> Is this 4 big & 4 little CPUs? > > 6 little + 2big. > On our devices, background tasks are bound to cores 0-3. To mimic the > behavior of these background tasks, we also bound rt-app to cores 0-3. > >> >>>>>>> Config: gki_defconfig >>>>>>> >>>>>>> RT-app‘s workload Pattern: >>>>>>> >>>>>>> { >>>>>>> "tasks" : { >>>>>>> "t0" : { >>>>>>> "instance" : 40, >>>>>>> "priority" : 0, >>>>>>> "cpus" : [ 0, 1, 2, 3 ], >>>>>>> "taskgroup" : "/background", >>>>>>> "loop" : -1, >>>>>>> "run" : 200, >>>>>>> "sleep" : 50 >>>>>>> } >>>>>>> } >>>>>>> } >>>>>>> >>>>>>> And we have applied the following patchs: >>>>>>> >>>>>>> https://lore.kernel.org/all/20251216111321.966709786@linuxfoundation.org/ >>>>>>> https://lore.kernel.org/all/20260106170509.413636243@linuxfoundation.org/ >>>>>>> https://lore.kernel.org/all/20260323134533.805879358@linuxfoundation.org/ >> >> Does the issue happen on v6.12.58 plain (android) or only when those 3 >> additional patches are applied on top? > > The issue was discovered on android16-6.12.58. We applied the > following three patches, but the issue is still reproducible. > >> >> d5843e1530d8 - sched/fair: Forfeit vruntime on yield (2025-12-18 Fernand >> Sieber) v6.12.63 >> >> bddd95054e33 - sched/eevdf: Fix min_vruntime vs avg_vruntime (2026-01-08 >> Peter Zijlstra) v6.12.64 >> >> d2fc2dcfce47 - sched/fair: Fix zero_vruntime tracking (2026-03-25 Peter >> Zijlstra) v6.12.78 > > Thanks! I tried to recreate your env as much as possible on qemu and ran your rt-app file but I can't spot anything suspicious either. This is with defconfig and cgroupv2. $ cat /sys/devices/system/cpu/cpu*/cpu_capacity 512 512 512 512 512 512 1024 1024 10 highest wu_lat values: v6.6 0.024601000 task0-9:881 0.019151000 task0-13:885 0.018344000 task0-27:899 0.017332000 task0-5:876 0.010613000 task0-21:893 0.010356000 task0-20:892 0.007796000 task0-15:887 0.007550000 task0-13:885 0.007292000 task0-2:872 0.006718000 task0-15:887 6.12.58 0.029507000 task0-32:1211 0.027374000 task0-37:1216 0.027294000 task0-12:1191 0.027063000 task0-11:1190 0.026612000 task0-28:1207 0.024829000 task0-38:1217 0.024472000 task0-18:1197 0.024396000 task0-34:1213 0.024303000 task0-10:1189 0.023317000 task0-26:1205 tip sched/core (7.0.0-rc4-00030-g265439eb88fd) 0.025000000 task0-32:851 0.020467000 task0-5:824 0.017190000 task0-16:835 0.015365000 task0-8:827 0.011591000 task0-32:851 0.010153000 task0-34:853 0.009932000 task0-4:823 0.008972000 task0-24:843 0.008564000 task0-39:858 0.007591000 task0-25:844