mirror of https://lore.kernel.org/lkml/
 help / color / mirror / Atom feed
From: K Prateek Nayak <kprateek.nayak@amd.com>
To: Peter Zijlstra <peterz@infradead.org>
Cc: Vincent Guittot <vincent.guittot@linaro.org>,
	Doug Smythies <dsmythies@telus.net>, <mingo@kernel.org>,
	<juri.lelli@redhat.com>, <dietmar.eggemann@arm.com>,
	<rostedt@goodmis.org>, <bsegall@google.com>, <mgorman@suse.de>,
	<vschneid@redhat.com>, <linux-kernel@vger.kernel.org>,
	<wangtao554@huawei.com>, <quzicheng@huawei.com>,
	<wuyun.abel@bytedance.com>
Subject: Re: [PATCH 0/4] sched: Various reweight_entity() fixes
Date: Fri, 13 Feb 2026 11:34:56 +0530	[thread overview]
Message-ID: <914c1bc5-e0e3-46d9-85e9-03dcbdbf0278@amd.com> (raw)
In-Reply-To: <20260212192911.GM1395416@noisy.programming.kicks-ass.net>

Hello Peter,

On 2/13/2026 12:59 AM, Peter Zijlstra wrote:
> On Thu, Feb 12, 2026 at 06:16:11PM +0100, Peter Zijlstra wrote:
>> On Thu, Feb 12, 2026 at 12:59:43PM +0100, Peter Zijlstra wrote:
>>> On Thu, Feb 12, 2026 at 01:13:30PM +0530, K Prateek Nayak wrote:
>>
>>>>     Enqueue cfs_rq: depth(1) weight(2144337920) nr_queued(2045) sum_w_vruntime(18343811230990336)    sum_weight(2144337920) zero_vruntime(18337852274) sum_shift(0) avg_vruntime(18337852274)
>>>>     Dequeue cfs_rq: depth(1) weight(2144337920) nr_queued(2045) sum_w_vruntime(1303379968)           sum_weight(2143289344) zero_vruntime(18354971669) sum_shift(0) avg_vruntime(18354971669)
>>
>> After staring at waaay too many traces of my own, which confirm what
>> you're seeing, but didn't want to make any sense either...
>>
>> ... I thinks I found it ...
>>
>> Note that sum_weight, that has _just_ flipped bit 31 and thus turned
>> negative *if* it were a 32bit number. Every time, exactly at this point
>> things started to go sideways.
>>
>> So I went looking for where this might be and I found the below.
>>
>> My latest run has passed 0x8000000/(1024*1024) = 2048 tasks and is still
>> running. Numbers are still sane, fingers crossed.
>>
> 
> I made it all the way to 3k tasks on the one CPU and all is well.

So the current peterz:sched/core refused to boot to prompt for (hopefully
it is just you forgetting to push out the latest branch ;-)

I have the following situation:

    Pick failed to find eligible entities. Dumping cfs_rq
    cfs_rq: depth(0) weight(3145728) nr_queued(3) sum_w_vruntime(-3074459814163644416) sum_weight(3145728) zero_vruntime(18446688365342623754) sum_shift(0)
    cfs_rq after avg_vruntime(): sum_w_vruntime(1048576) sum_weight(3145728) zero_vruntime(18446687387998169890) avg_vruntime(18446687387998169890)
    se: weight(1048576) vruntime(18446691297375985345) slice(700000) deadline(18446691297376644150) curr?(1) task?(1) se_depth(0) eligible?(1)
    se: weight(1048576) vruntime(18446682501282949076) slice(700000) deadline(18446682501283649076) curr?(0) task?(1) se_depth(0) eligible?(0)
    se: weight(1048576) vruntime(18446688365342622487) slice(700000) deadline(18446688365342972487) curr?(0) task?(1) se_depth(0) eligible?(0)
    se: weight(1048576) vruntime(18446691297368938108) slice(700000) deadline(18446691297369619108) curr?(0) task?(1) se_depth(0) eligible?(1)


Note: sum_w_vruntime grows pretty large before but on calling
avg_vruntime(), things are back to normal.

Since this is so early into the boot I traced the heck out of
it and one of the observation was how the sum_weight() keeps
growing from the initial zero_vruntime:

    Legend: '|' is a place_entity(), "->" is enqueue, "<-" is dequeue.
            If the line contains "*" it is "before" the action.
            Without the "*" is "after" the action.
            Line with only "cfs_rq" line before and after avg_vruntime() in entity_tick()

    vruntime: CPU(0)  se vruntime(18446744073708503040) vlag(0) deadline(0) | cfs_rq sum_w_vruntime(0) sum_weight(0) zero_vruntime(18446744073708503040)
    vruntime: CPU(0) *se vruntime(18446744073708503040) vlag(0) deadline(18446744073708853040) -> cfs_rq sum_w_vruntime(0) sum_weight(0) zero_vruntime(18446744073708503040)
    vruntime: CPU(0)  se vruntime(18446744073708503040) vlag(0) deadline(18446744073708853040) -> cfs_rq sum_w_vruntime(0) sum_weight(1048576) zero_vruntime(18446744073708503040)
    vruntime: CPU(0)  se vruntime(18446744073708503040) vlag(0) deadline(0) | cfs_rq sum_w_vruntime(0) sum_weight(1048576) zero_vruntime(18446744073708503040)
    vruntime: CPU(0) *se vruntime(18446744073708503040) vlag(0) deadline(18446744073708853040) -> cfs_rq sum_w_vruntime(0) sum_weight(1048576) zero_vruntime(18446744073708503040)
    vruntime: CPU(0)  se vruntime(18446744073708503040) vlag(0) deadline(18446744073708853040) -> cfs_rq sum_w_vruntime(0) sum_weight(2097152) zero_vruntime(18446744073708503040)
    vruntime: CPU(0) *se vruntime(18446744073708503040) vlag(0) deadline(18446744073708853040) <- cfs_rq sum_w_vruntime(0) sum_weight(2097152) zero_vruntime(18446744073708503040)
    vruntime: CPU(0)  se vruntime(18446744073708503040) vlag(0) deadline(18446744073708853040) <- cfs_rq sum_w_vruntime(0) sum_weight(1048576) zero_vruntime(18446744073708503040)
    vruntime: CPU(0) *cfs_rq: cfs_rq sum_w_vruntime(0) sum_weight(1048576) zero_vruntime(18446744073708503040)

        At this point:

        curr->vruntime = 951329; /* should be same as computed avg */
        u64 zero_vruntime = (u64)(-1048576LL); /* From init */
        s64 sum_w_vruntime = 0;
        
        s64 delta = curr->vruntime - cfs_rq->zero_vruntime = 1999905; /* Net positive. */
        sum_w_vruntime -= 1999905 * sum_weight;

        sum_w_vruntime = -2097052385280; /* Checks out! */

    vruntime: CPU(0)  cfs_rq: cfs_rq sum_w_vruntime(-2097052385280) sum_weight(1048576) zero_vruntime(951329) avg_vruntime(951329)

        After this point all delta are positive and we keep
        subtracting from sum_w_vruntime.

    vruntime: CPU(0) *cfs_rq: cfs_rq sum_w_vruntime(-2097052385280) sum_weight(1048576) zero_vruntime(951329)
    vruntime: CPU(0)  cfs_rq: cfs_rq sum_w_vruntime(-4194182365184) sum_weight(1048576) zero_vruntime(2951308) avg_vruntime(2951308)
    vruntime: CPU(0) *cfs_rq: cfs_rq sum_w_vruntime(-4194182365184) sum_weight(1048576) zero_vruntime(2951308)
    vruntime: CPU(0)  cfs_rq: cfs_rq sum_w_vruntime(-6291298713600) sum_weight(1048576) zero_vruntime(4951274) avg_vruntime(4951274)
    vruntime: CPU(0) *cfs_rq: cfs_rq sum_w_vruntime(-6291298713600) sum_weight(1048576) zero_vruntime(4951274)
    vruntime: CPU(0)  cfs_rq: cfs_rq sum_w_vruntime(-8388566056960) sum_weight(1048576) zero_vruntime(6951384) avg_vruntime(6951384)
    vruntime: CPU(0) *cfs_rq: cfs_rq sum_w_vruntime(-8388566056960) sum_weight(1048576) zero_vruntime(6951384)
    vruntime: CPU(0)  cfs_rq: cfs_rq sum_w_vruntime(-10485571256320) sum_weight(1048576) zero_vruntime(8951244) avg_vruntime(8951244)
    vruntime: CPU(0) *cfs_rq: cfs_rq sum_w_vruntime(-10485571256320) sum_weight(1048576) zero_vruntime(8951244)
    vruntime: CPU(0)  cfs_rq: cfs_rq sum_w_vruntime(-12582749470720) sum_weight(1048576) zero_vruntime(10951269) avg_vruntime(10951269)
    vruntime: CPU(0) *cfs_rq: cfs_rq sum_w_vruntime(-12582749470720) sum_weight(1048576) zero_vruntime(10951269)
    vruntime: CPU(0)  cfs_rq: cfs_rq sum_w_vruntime(-14680032542720) sum_weight(1048576) zero_vruntime(12951394) avg_vruntime(12951394)


I still haven't found how we end up at a situation where avg_vruntime
is close to -56685711381726LL at the time of crash :-(

Will update if I find something.

-- 
Thanks and Regards,
Prateek


  parent reply	other threads:[~2026-02-13  6:05 UTC|newest]

Thread overview: 70+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-01-30  9:34 Peter Zijlstra
2026-01-30  9:34 ` [PATCH 1/4] sched/fair: Only set slice protection at pick time Peter Zijlstra
2026-01-30 15:52   ` Vincent Guittot
2026-01-30  9:34 ` [PATCH 2/4] sched/eevdf: Update se->vprot in reweight_entity() Peter Zijlstra
2026-01-30 16:20   ` Vincent Guittot
2026-01-30  9:34 ` [PATCH 3/4] sched/fair: Increase weight bits for avg_vruntime Peter Zijlstra
2026-01-30  9:34 ` [PATCH 4/4] sched/fair: Revert 6d71a9c61604 ("sched/fair: Fix EEVDF entity placement bug causing scheduling lag") Peter Zijlstra
2026-01-31  1:47   ` Zhang Qiao
2026-01-31 15:21     ` Peter Zijlstra
2026-02-02  9:12       ` Peter Zijlstra
2026-02-02  9:24         ` Peter Zijlstra
2026-02-02 11:23         ` Zhang Qiao
2026-02-01 17:13 ` [PATCH 0/4] sched: Various reweight_entity() fixes Doug Smythies
2026-02-03  6:45 ` K Prateek Nayak
2026-02-03 11:11   ` Peter Zijlstra
2026-02-03 12:19     ` K Prateek Nayak
2026-02-03 16:36       ` Doug Smythies
2026-02-10 18:13         ` Peter Zijlstra
2026-02-11  5:51           ` Doug Smythies
2026-02-04 10:15       ` K Prateek Nayak
2026-02-09 15:47         ` Peter Zijlstra
2026-02-09 16:52           ` K Prateek Nayak
2026-02-10  5:16           ` K Prateek Nayak
2026-02-10 10:29             ` Peter Zijlstra
2026-02-10 15:41           ` Doug Smythies
2026-02-10 18:09             ` K Prateek Nayak
2026-02-10 18:35               ` Peter Zijlstra
2026-02-10 20:04                 ` K Prateek Nayak
2026-02-11  6:28                   ` K Prateek Nayak
2026-02-11  8:50                     ` K Prateek Nayak
2026-02-11 23:09               ` Doug Smythies
2026-02-10 18:52             ` Peter Zijlstra
2026-02-10 20:52           ` Vincent Guittot
2026-02-11  5:21             ` Doug Smythies
2026-02-11  8:49               ` Vincent Guittot
2026-02-11  9:01                 ` Peter Zijlstra
2026-02-11 10:48                   ` Peter Zijlstra
2026-02-11 10:49                     ` Peter Zijlstra
2026-02-11 11:15                     ` Vincent Guittot
2026-02-11 16:28                       ` Peter Zijlstra
2026-02-12  7:43                         ` K Prateek Nayak
2026-02-12 11:59                           ` Peter Zijlstra
2026-02-12 17:16                             ` Peter Zijlstra
2026-02-12 17:24                               ` Vincent Guittot
2026-02-12 19:31                                 ` Peter Zijlstra
2026-02-13  5:22                                   ` K Prateek Nayak
2026-02-13  6:44                                     ` Peter Zijlstra
2026-02-13 10:50                                       ` Peter Zijlstra
2026-02-13 14:29                                         ` K Prateek Nayak
2026-02-14  6:31                                         ` Doug Smythies
2026-02-21 22:51                                           ` Doug Smythies
2026-02-12 19:29                               ` Peter Zijlstra
2026-02-12 19:37                                 ` Doug Smythies
2026-02-13  6:04                                 ` K Prateek Nayak [this message]
2026-02-11 16:21                     ` Peter Zijlstra
2026-02-12  5:54                       ` Doug Smythies
2026-02-12  7:51                         ` Peter Zijlstra
2026-02-12 15:47                           ` Doug Smythies
2026-02-12  7:46                       ` Peter Zijlstra
2026-02-11 23:25                     ` Doug Smythies
2026-02-11  8:48             ` Peter Zijlstra
2026-02-04 10:44       ` Peter Zijlstra
2026-02-14  7:20 ` Shubhang Kaushik
2026-02-16  3:14   ` K Prateek Nayak
2026-02-16 10:59     ` Dietmar Eggemann
2026-02-17 14:37       ` Dietmar Eggemann
2026-02-17 22:02     ` Shubhang Kaushik
2026-02-17  4:20 ` K Prateek Nayak
2026-02-18 18:37 ` Shubhang Kaushik
2026-02-19  7:53   ` Peter Zijlstra

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=914c1bc5-e0e3-46d9-85e9-03dcbdbf0278@amd.com \
    --to=kprateek.nayak@amd.com \
    --cc=bsegall@google.com \
    --cc=dietmar.eggemann@arm.com \
    --cc=dsmythies@telus.net \
    --cc=juri.lelli@redhat.com \
    --cc=linux-kernel@vger.kernel.org \
    --cc=mgorman@suse.de \
    --cc=mingo@kernel.org \
    --cc=peterz@infradead.org \
    --cc=quzicheng@huawei.com \
    --cc=rostedt@goodmis.org \
    --cc=vincent.guittot@linaro.org \
    --cc=vschneid@redhat.com \
    --cc=wangtao554@huawei.com \
    --cc=wuyun.abel@bytedance.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox

all inboxes | Powered by JetHome®