From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from serv15.avernis.de (serv15.avernis.de [176.9.89.163]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id BAA933B7760 for ; Tue, 26 May 2026 04:10:50 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=176.9.89.163 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1779768658; cv=none; b=fJDiI0RDtQ32aj3jqFI+5XoUiEaa8+L/HPyHT6aZ3PHW3YutNdJdTW/J4OUmOWLMX6EKd7k0MpzhGjHj8kGqvXY+IuSuVq1UzALuRYOGzvF0TgVRtnvLUSeUEGYBWWbTx4Z/RwVnFm+4bNJZjRimeBcdxOrEhbCXV1JKyR0r9MI= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1779768658; c=relaxed/simple; bh=OSK9B7V4ZzJLQLo0FUl+Tm7nlS4BP83S0Mve/sIRLt0=; h=MIME-Version:Date:From:To:Cc:Subject:In-Reply-To:References: Message-ID:Content-Type; b=KAcPPVHSDzeU87h0DKChkNrCRpq4dZQuA3FpYrgjr1FJME6Cs6l4QnkMXDE6aeDVg1OiVulS638wFgpLGYOsYcnI/G01WBwVZQPdq4C22dQqDLc5x5CKhwbBBf1/3B3DieJfOK1+rq3PeoFVQGnTwB2P/pzsi8Uq0nzYbwpy8IA= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=umbiko.net; spf=none smtp.mailfrom=umbiko.net; dkim=pass (1024-bit key) header.d=umbiko.net header.i=@umbiko.net header.b=vZXy8MeH; arc=none smtp.client-ip=176.9.89.163 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=umbiko.net Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=umbiko.net Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=umbiko.net header.i=@umbiko.net header.b="vZXy8MeH" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=umbiko.net; s=mail; t=1779768135; bh=OSK9B7V4ZzJLQLo0FUl+Tm7nlS4BP83S0Mve/sIRLt0=; h=Date:From:To:Cc:Subject:In-Reply-To:References; b=vZXy8MeHXbIYcdLT21Gy4Aew4FBGagYDWJZAGOyruVVC2noRev8hLEkREYGGSN2gi XYO82+COHpAhHkl8WHuUnjJdjDHwCWW5vi7WuZysBkKxqSQ/Hqx3s65XLDOIJFakwr Wy6iMS4UtoCPEPDgBmP8RPvd6V7o5VIM79SfqVDI= Received: by serv15.avernis.de (Postfix) with ESMTPSA id CD577BDE0CBD; Tue, 26 May 2026 06:02:14 +0200 (CEST) Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Date: Tue, 26 May 2026 04:02:14 +0000 From: Andreas Ziegler To: Christian Loehle Cc: Peter Zijlstra , Juri Lelli , linux-kernel@vger.kernel.org, Dietmar Eggemann , John Stultz , Gabriele Monaco Subject: Re: sched/deadline: Use revised wakeup rule for dl_server In-Reply-To: <8b82ef04-6b8c-4e39-ae9a-8cdaa1e97cf1@arm.com> References: <496e4b3329fe258da9618b9f05b18fcf@umbiko.net> <97d9e04fd9d222f1a64f1ecfda8b81d7@umbiko.net> <50156878-265d-4025-9b36-c819c80b7493@arm.com> <8b82ef04-6b8c-4e39-ae9a-8cdaa1e97cf1@arm.com> Message-ID: <5ec17184870e6966daa60f76c0694364@umbiko.net> X-Sender: br025@umbiko.net Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Virus-Scanned: clamav-milter 1.4.3 at serv15.avernis.de X-Virus-Status: Clean On 2026-05-25 07:25, Christian Loehle wrote: > On 5/11/26 10:47, Christian Loehle wrote: >> On 5/9/26 12:42, Andreas Ziegler wrote: >>> Hi Christian, Everyone, >>> >>> On 2026-05-08 14:13, Christian Loehle wrote: >>>> On 5/8/26 13:06, Andreas Ziegler wrote: >>>>> Hi Christian, >>>>> >>>>> On 2026-05-08 09:20, Christian Loehle wrote: >>>>>> On 5/8/26 09:09, Andreas Ziegler wrote: >>>>>>> Linux kernel version: 6.12 >>>>>>>   CONFIG_PREEMPT_RT (w/ PREEMPT_RT patch applied) >>>>>>> Architecture: aarch64 >>>>>>> Platform: Raspberry Pi 4 >>>>>>> >>>>>>> Hi everyone, >>>>>>> >>>>>>> Commit d66792919d4f (sched/deadline: Use revised wakeup rule for >>>>>>> dl_server) [1] introduced a marked degradation in scheduling >>>>>>> latency for real-time tasks in the presence of heavy I/O load. >>>>>>> >>>>>>> --- a/kernel/sched/deadline.c >>>>>>> +++ b/kernel/sched/deadline.c >>>>>>> @@ -1079,7 +1079,7 @@ static void update_dl_entity(struct >>>>>>> sched_dl_entity *dl_se) >>>>>>>      if (dl_time_before(dl_se->deadline, rq_clock(rq)) || >>>>>>>          dl_entity_overflow(dl_se, rq_clock(rq))) { >>>>>>> >>>>>>> -        if (unlikely(!dl_is_implicit(dl_se) && >>>>>>> +        if (unlikely((!dl_is_implicit(dl_se) || dl_se->dl_defer) >>>>>>> && >>>>>>>                   !dl_time_before(dl_se->deadline, rq_clock(rq)) >>>>>>> && >>>>>>>                   !is_dl_boosted(dl_se))) { >>>>>>>              update_dl_revised_wakeup(dl_se, rq); >>>>>>> >>>>>>> This was observed using a modified version of Con Kolivas' >>>>>>> interactivity benchmark [2]; kernel bisection eventually pointed >>>>>>> to the above mentioned commit. >>>>>>> >>>>>>> Benchmark results before d66792919d4f: >>>>>>> >>>>>>> --- Benchmarking simulated cpu of Audio real time in the presence >>>>>>> of simulated --- >>>>>>> Load    Latency +/- SD   median  max [100n]    Desired CPU  >>>>>>> Deadlines met [%] >>>>>>> None      76.6 +/- 8.3654    76  166 >>>>>>> Video      78.5 +/- 3.9433    78  107 >>>>>>> X      76.4 +/- 8.123     75  157 >>>>>>> Burn      72.0 +/- 6.4733    71  127 >>>>>>> Write     255.3 +/- 26.627   252  331 >>>>>>> Read     226.6 +/- 12.38    227  262 >>>>>>> Ring      84.2 +/- 6.6207    83  125 >>>>>>> Compile     225.3 +/- 23.949   222  328 >>>>>>> >>>>>>>      136.8 +/- 78.462        331 >>>>>>> >>>>>>> Benchmark results after d66792919d4f: >>>>>>> >>>>>>> --- Benchmarking simulated cpu of Audio real time in the presence >>>>>>> of simulated --- >>>>>>> Load    Latency +/- SD   median  max [100n]    Desired CPU  >>>>>>> Deadlines met [%] >>>>>>> None      68.4 +/- 9.7864    67  169 >>>>>>> Video      74.4 +/- 3.724     74   97 >>>>>>> X      72.0 +/- 6.5681    71  129 >>>>>>> Burn      66.9 +/- 5.9059    66  117 >>>>>>> Write    9576.9 +/- 67639    250500418        98.1         98.1 >>>>>>> Read     209.3 +/- 11.018   209  267 >>>>>>> Ring      80.5 +/- 8.0993    78  125 >>>>>>> Compile     239.0 +/- 29.447   234  372 >>>>>>> >>>>>>>     1298.4 +/- 24118       500418 >>>>>>> >>>>>>> Reverting this commit obviously solves the issue for me. I have >>>>>>> no idea why this issue appears exclusively with heavy write loads >>>>>>> in the background. >>>>>>> >>>>>>> Is this a scheduler issue, or rather something in the background? >>>>>>> >>>>>> >>>>>> Hi Andreas, >>>>>> You're using cpufreq schedutil for your tests I'm assuming? >>>>>> Is there a difference in cpufreq behavior (avg cpufreq or OPP >>>>>> residencies?) >>>>>> Does the regression also happen on powersave/performance governor? >>>>> >>>>> Actually this is a very stripped-down system. The 'performance' >>>>> cpufreq governor is the only one compiled in, the processor cores >>>>> run on a fixed frequency. CONFIG_PM_OPP is not set. >>>> >>>> That certainly makes the analysis easier. >>>> I couldn't reproduce the issue so far on my system but it does seem >>>> like the dl server >>>> would get potentially unbounded running time with very frequent >>>> starting and stopping of the dlserver (which presumably happens >>>> because of >>>> the writeback) reset the runtime, which then leads to your 25s >>>> observed latency. >>>> Peter, how is the revised wakeup rule supposed to behave here? >>>> >>>>> [snip] >>> >>> This seems to be a case of runtime starvation. If I change >>> sched_rt_runtime_us to a smaller value, the benchmark returns >>> reasonable latency values. >>> >>> # echo "980000" > /proc/sys/kernel/sched_rt_runtime_us >>> >>> I could live with this workaround, since it seems not to impact >>> overall latency values in a noticeable way. >>> >> >> Not a very stable workaround unfortunately :/ >> While I try to reproduce this, what you're observing should imply that >> the >> background SCHED_NORMAL work is enough to fully utilize the system, >> right? >> interbench Write does 4k (buffered) writes of a 1GB file and then >> close+open >> and repeat, nothing fancy really. Does this actually produce >> significant CPU >> utilization for you? Can you just run the background work and see what >> that >> looks like? >> (What you're seeing looks like a bug in any case, just so I'm not >> going down >> a wrong path when trying to reproduce here). > > I'd be interested if you can still reproduce with this fix: > https://lore.kernel.org/lkml/20260522125833.264145-1-gmonaco@redhat.com/ The current 6.12.91 kernel with this patch applied shows normal latencies without 50ms outliers. There is a backport in planning for three commits that preceded d66792919d4f, that also fixes the issue: https://marc.info/?l=linux-rt-users&m=177948576595996&w=2 -> https://lore.kernel.org/stable/20260522213120.1205100-1-lbckmnn@mailbox.org Applying the patch above on top of the proposed backport also tested without issues.