mirror of https://lore.kernel.org/lkml/
 help / color / mirror / Atom feed
From: K Prateek Nayak <kprateek.nayak@amd.com>
To: Peter Zijlstra <peterz@infradead.org>
Cc: <mingo@kernel.org>, <juri.lelli@redhat.com>,
	<vincent.guittot@linaro.org>, <dietmar.eggemann@arm.com>,
	<rostedt@goodmis.org>, <bsegall@google.com>, <mgorman@suse.de>,
	<vschneid@redhat.com>, <linux-kernel@vger.kernel.org>,
	<wangtao554@huawei.com>, <quzicheng@huawei.com>,
	<wuyun.abel@bytedance.com>, <dsmythies@telus.net>
Subject: Re: [PATCH 0/4] sched: Various reweight_entity() fixes
Date: Tue, 3 Feb 2026 17:49:16 +0530	[thread overview]
Message-ID: <38ef3462-4c4e-4f40-8d63-84dd71cbd043@amd.com> (raw)
In-Reply-To: <20260203111134.GL1282955@noisy.programming.kicks-ass.net>

Hello Peter,

On 2/3/2026 4:41 PM, Peter Zijlstra wrote:
> On Tue, Feb 03, 2026 at 12:15:56PM +0530, K Prateek Nayak wrote:
>> Hello Peter,
>>
>> On 1/30/2026 3:04 PM, Peter Zijlstra wrote:
>>> Two issues related to reweight_entity() were raised; poking at all that got me
>>> these patches.
>>>
>>> They're in queue.git/sched/core and I spend most of yesterday staring at traces
>>> trying to find anything wrong. So far, so good.
>>>
>>> Please test.
>>
>> I put this on top of tip:sched/urgent + tip:sched/core which contains Ingo's
>> cleanup of removing the union and at some point in the benchmark run I hit:
>>
>>     BUG: kernel NULL pointer dereference, address: 0000000000000051
> 
> :-(
> 
>>
>> so something went sideways with the avg_vruntime calculation I presume.
>> I'm rerunning with the PARANOID_AVG feat now.
>>
>> Just re-running the particular schbench variant hasn't crashed the kernel
>> in the half hour it has been running so I've re-triggered the same set of
>> benchmarks to see if flipping PARANOID_AVG makes any difference.
> 
> If you run with PARANOID_AVG, the condition ends up visible as:
> 
>   grep shift /debug/sched/debug
> 
> If any of the fields are !0, you tripped an overflow.

Yup I see a few !0 values. Some inching closer to the BUG_ON()

 grep "shift.*: [^0]$" /sys/kernel/debug/sched/debug
  .sum_shift                     : 4
  .sum_shift                     : 3
  .sum_shift                     : 5
  .sum_shift                     : 1
  .sum_shift                     : 2
  .sum_shift                     : 3

> 
> Once its !0, you can't get it back to 0 (except perhaps if its cgroup
> things, in which case you can destroy and re-create the cgroups I
> suppose) other than reboot.
> 
> Anyway, if you can reproduce without PARANOID_AVG (or indeed have
> tripped overflow) could you share the specific schbench invocation you
> used?

This trips when I'm running a (very) old version of schbench at commit
e4aa540 ("Make sure rps isn't zero in auto_rps mode.")

I'm running the following on a 512 CPU server:

#!/bin/bash

DIR=$1
MESSENGERS=1
MAX_ITERS=2
SCHBENCH=./schbench

for i in 1 2 4 8 16 32 64 128 256 512 768 1024;
do
    THISDIR=$DIR/$i-workers
    if [ ! -d $THISDIR ]
    then
        mkdir -p $THISDIR
    fi
    for j in `seq 0 $MAX_ITERS`
    do
        echo "===== Worker $i : Iter $j ======";
        $SCHBENCH -m $MESSENGERS -t $i  |& tee $THISDIR/iter-$j.log;
    sleep 2
    done
done


Fails when it is running with 768 workers. Standalone runs didn't
fail - have to run a cumulative runner that runs sched-messaging,
stream, tbench, netperf, first before running schbench :-(

> 
> I'm not sure I have valuable tracing patches, I just stick random
> trace_printk()s in.

I'll plop those in and update once the I get a log for sum_shift++.

-- 
Thanks and Regards,
Prateek


  reply	other threads:[~2026-02-03 12:19 UTC|newest]

Thread overview: 70+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-01-30  9:34 Peter Zijlstra
2026-01-30  9:34 ` [PATCH 1/4] sched/fair: Only set slice protection at pick time Peter Zijlstra
2026-01-30 15:52   ` Vincent Guittot
2026-01-30  9:34 ` [PATCH 2/4] sched/eevdf: Update se->vprot in reweight_entity() Peter Zijlstra
2026-01-30 16:20   ` Vincent Guittot
2026-01-30  9:34 ` [PATCH 3/4] sched/fair: Increase weight bits for avg_vruntime Peter Zijlstra
2026-01-30  9:34 ` [PATCH 4/4] sched/fair: Revert 6d71a9c61604 ("sched/fair: Fix EEVDF entity placement bug causing scheduling lag") Peter Zijlstra
2026-01-31  1:47   ` Zhang Qiao
2026-01-31 15:21     ` Peter Zijlstra
2026-02-02  9:12       ` Peter Zijlstra
2026-02-02  9:24         ` Peter Zijlstra
2026-02-02 11:23         ` Zhang Qiao
2026-02-01 17:13 ` [PATCH 0/4] sched: Various reweight_entity() fixes Doug Smythies
2026-02-03  6:45 ` K Prateek Nayak
2026-02-03 11:11   ` Peter Zijlstra
2026-02-03 12:19     ` K Prateek Nayak [this message]
2026-02-03 16:36       ` Doug Smythies
2026-02-10 18:13         ` Peter Zijlstra
2026-02-11  5:51           ` Doug Smythies
2026-02-04 10:15       ` K Prateek Nayak
2026-02-09 15:47         ` Peter Zijlstra
2026-02-09 16:52           ` K Prateek Nayak
2026-02-10  5:16           ` K Prateek Nayak
2026-02-10 10:29             ` Peter Zijlstra
2026-02-10 15:41           ` Doug Smythies
2026-02-10 18:09             ` K Prateek Nayak
2026-02-10 18:35               ` Peter Zijlstra
2026-02-10 20:04                 ` K Prateek Nayak
2026-02-11  6:28                   ` K Prateek Nayak
2026-02-11  8:50                     ` K Prateek Nayak
2026-02-11 23:09               ` Doug Smythies
2026-02-10 18:52             ` Peter Zijlstra
2026-02-10 20:52           ` Vincent Guittot
2026-02-11  5:21             ` Doug Smythies
2026-02-11  8:49               ` Vincent Guittot
2026-02-11  9:01                 ` Peter Zijlstra
2026-02-11 10:48                   ` Peter Zijlstra
2026-02-11 10:49                     ` Peter Zijlstra
2026-02-11 11:15                     ` Vincent Guittot
2026-02-11 16:28                       ` Peter Zijlstra
2026-02-12  7:43                         ` K Prateek Nayak
2026-02-12 11:59                           ` Peter Zijlstra
2026-02-12 17:16                             ` Peter Zijlstra
2026-02-12 17:24                               ` Vincent Guittot
2026-02-12 19:31                                 ` Peter Zijlstra
2026-02-13  5:22                                   ` K Prateek Nayak
2026-02-13  6:44                                     ` Peter Zijlstra
2026-02-13 10:50                                       ` Peter Zijlstra
2026-02-13 14:29                                         ` K Prateek Nayak
2026-02-14  6:31                                         ` Doug Smythies
2026-02-21 22:51                                           ` Doug Smythies
2026-02-12 19:29                               ` Peter Zijlstra
2026-02-12 19:37                                 ` Doug Smythies
2026-02-13  6:04                                 ` K Prateek Nayak
2026-02-11 16:21                     ` Peter Zijlstra
2026-02-12  5:54                       ` Doug Smythies
2026-02-12  7:51                         ` Peter Zijlstra
2026-02-12 15:47                           ` Doug Smythies
2026-02-12  7:46                       ` Peter Zijlstra
2026-02-11 23:25                     ` Doug Smythies
2026-02-11  8:48             ` Peter Zijlstra
2026-02-04 10:44       ` Peter Zijlstra
2026-02-14  7:20 ` Shubhang Kaushik
2026-02-16  3:14   ` K Prateek Nayak
2026-02-16 10:59     ` Dietmar Eggemann
2026-02-17 14:37       ` Dietmar Eggemann
2026-02-17 22:02     ` Shubhang Kaushik
2026-02-17  4:20 ` K Prateek Nayak
2026-02-18 18:37 ` Shubhang Kaushik
2026-02-19  7:53   ` Peter Zijlstra

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=38ef3462-4c4e-4f40-8d63-84dd71cbd043@amd.com \
    --to=kprateek.nayak@amd.com \
    --cc=bsegall@google.com \
    --cc=dietmar.eggemann@arm.com \
    --cc=dsmythies@telus.net \
    --cc=juri.lelli@redhat.com \
    --cc=linux-kernel@vger.kernel.org \
    --cc=mgorman@suse.de \
    --cc=mingo@kernel.org \
    --cc=peterz@infradead.org \
    --cc=quzicheng@huawei.com \
    --cc=rostedt@goodmis.org \
    --cc=vincent.guittot@linaro.org \
    --cc=vschneid@redhat.com \
    --cc=wangtao554@huawei.com \
    --cc=wuyun.abel@bytedance.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox

Powered by JetHome