mirror of https://lore.kernel.org/lkml/
 help / color / mirror / Atom feed
From: K Prateek Nayak <kprateek.nayak@amd.com>
To: Shrikanth Hegde <sshegde@linux.ibm.com>, <mingo@kernel.org>,
	<peterz@infradead.org>, <vincent.guittot@linaro.org>,
	<linux-kernel@vger.kernel.org>
Cc: <juri.lelli@redhat.com>, <vschneid@redhat.com>, <tglx@kernel.org>,
	<dietmar.eggemann@arm.com>, <anna-maria@linutronix.de>,
	<frederic@kernel.org>, <wangyang.guo@intel.com>
Subject: Re: [PATCH v4 0/3] sched/fair: Improve nohz fields for large systems
Date: Tue, 13 Jan 2026 12:15:46 +0530	[thread overview]
Message-ID: <3298a289-22c4-42c4-a7c1-39d9519f6223@amd.com> (raw)
In-Reply-To: <20260112050442.138446-1-sshegde@linux.ibm.com>

Hello Shrikanth,

On 1/12/2026 10:34 AM, Shrikanth Hegde wrote:
> Running on large systems nohz.nr_cpus cacheline was seen as contended.
> There is atomic inc/dec and read happening on many
> CPUs at a time and it is possible for this line to bounce often.
> 
> 1st and 2nd patch are minor ones. Looks like correct things to do.
> Not very important ones.
> 
> 3rd patch: Main patch which is to get rid of nr_cpus.Instead, use the cpumask
> which is always updated alongside with it. Functionally it should serve
> the same purpose. Rest of the fields aren't updated that often. So this
> line shouldn't bounce that often.
> 
> Contention issue with nohz.idle_cpus_mask still remains. Mostly it is in
> separate cacheline than nohz. There are ongoing efforts to mitigate it. It
> is not addressed by this series.
> 
> v3 -> v4:
> - Added to changelog on one less cacheline being dirtied on idle
>   entry/exit (Valentin Schneider)

I tested the v3 over the weekend and didn't spot any regressions
(at least none that I can reproduce consistently) so feel free to
include:

Reviewed-and-tested-by: K Prateek Nayak <kprateek.nayak@amd.com>

If anyone is curious, following are results from my setup
(3rd Generation EPYC, 2 socket x 64/128T, boost on, C2 disabled):

Note: tbench hit some insane luck on higher utilization runs. I
haven't been able to reproduce those regressions reliably.
Most data points that show regression also have high run to run
variance on both tip and tip + patch making them unreliable.

  ==================================================================
  Test          : hackbench
  Units         : Normalized time in seconds
  Interpretation: Lower is better
  Statistic     : AMean
  ==================================================================
  Case:           tip[pct imp](CV)    nohz_no_nr_cpus[pct imp](CV)
   1-groups     1.00 [ -0.00]( 6.43)     1.04 [ -3.60](15.15)
   2-groups     1.00 [ -0.00]( 5.42)     1.02 [ -2.17]( 3.57)
   4-groups     1.00 [ -0.00]( 2.72)     0.99 [  0.84]( 3.11)
   8-groups     1.00 [ -0.00]( 3.65)     1.00 [  0.31]( 2.50)
  16-groups     1.00 [ -0.00]( 2.26)     1.02 [ -1.67]( 2.92)
  
  
  ==================================================================
  Test          : tbench
  Units         : Normalized throughput
  Interpretation: Higher is better
  Statistic     : AMean
  ==================================================================
  Clients:    tip[pct imp](CV)    nohz_no_nr_cpus[pct imp](CV)
      1     1.00 [  0.00]( 0.40)     1.00 [ -0.25]( 1.22)
      2     1.00 [  0.00]( 1.33)     0.99 [ -0.57]( 0.37)
      4     1.00 [  0.00]( 0.27)     1.00 [  0.07]( 0.89)
      8     1.00 [  0.00]( 0.53)     0.99 [ -0.83]( 0.32)
     16     1.00 [  0.00]( 1.39)     1.00 [  0.11]( 1.92)
     32     1.00 [  0.00]( 1.85)     0.99 [ -1.44]( 3.08)
     64     1.00 [  0.00]( 1.55)     0.98 [ -2.17]( 2.51)
    128     1.00 [  0.00]( 1.05)     0.94 [ -6.11]( 0.28)
    256     1.00 [  0.00]( 0.68)     0.94 [ -5.58]( 3.77)
    512     1.00 [  0.00]( 0.30)     0.95 [ -4.91]( 0.22)
   1024     1.00 [  0.00]( 0.19)     0.95 [ -4.86]( 0.21)
  
  
  ==================================================================
  Test          : stream-10
  Units         : Normalized Bandwidth, MB/s
  Interpretation: Higher is better
  Statistic     : HMean
  ==================================================================
  Test:       tip[pct imp](CV)    nohz_no_nr_cpus[pct imp](CV)
   Copy     1.00 [  0.00]( 8.08)     1.03 [  2.91]( 4.84)
  Scale     1.00 [  0.00]( 5.43)     1.04 [  3.56]( 3.32)
    Add     1.00 [  0.00]( 5.96)     1.04 [  4.10]( 2.96)
  Triad     1.00 [  0.00]( 6.36)     0.99 [ -1.23]( 5.83)
  
  
  ==================================================================
  Test          : stream-100
  Units         : Normalized Bandwidth, MB/s
  Interpretation: Higher is better
  Statistic     : HMean
  ==================================================================
  Test:       tip[pct imp](CV)    nohz_no_nr_cpus[pct imp](CV)
   Copy     1.00 [  0.00]( 3.78)     1.03 [  3.17]( 1.90)
  Scale     1.00 [  0.00]( 4.17)     1.02 [  1.79]( 0.91)
    Add     1.00 [  0.00]( 1.97)     1.01 [  0.52]( 1.66)
  Triad     1.00 [  0.00]( 2.28)     0.99 [ -1.49]( 4.44)
  
  
  ==================================================================
  Test          : schbench
  Units         : Normalized 99th percentile latency in us
  Interpretation: Lower is better
  Statistic     : Median
  ==================================================================
  #workers: tip[pct imp](CV)    nohz_no_nr_cpus[pct imp](CV)
    1     1.00 [ -0.00](33.02)     1.21 [-20.59]( 8.06)
    2     1.00 [ -0.00](14.30)     1.14 [-14.29]( 6.45)
    4     1.00 [ -0.00]( 2.22)     0.98 [  2.22]( 4.55)
    8     1.00 [ -0.00]( 4.63)     0.94 [  5.56]( 1.96)
   16     1.00 [ -0.00]( 1.67)     1.07 [ -6.67]( 1.82)
   32     1.00 [ -0.00]( 5.58)     0.99 [  1.04]( 2.11)
   64     1.00 [ -0.00]( 6.03)     0.99 [  0.52]( 5.25)
  128     1.00 [ -0.00]( 7.09)     1.00 [ -0.49]( 5.11)
  256     1.00 [ -0.00]( 3.14)     0.94 [  6.06](13.53)
  512     1.00 [ -0.00]( 0.86)     0.98 [  2.23]( 1.53)
  
  
  ==================================================================
  Test          : new-schbench-requests-per-second
  Units         : Normalized Requests per second
  Interpretation: Higher is better
  Statistic     : Median
  ==================================================================
  #workers: tip[pct imp](CV)    nohz_no_nr_cpus[pct imp](CV)
    1     1.00 [  0.00]( 0.14)     1.00 [  0.00]( 0.52)
    2     1.00 [  0.00]( 0.14)     1.00 [  0.28]( 0.00)
    4     1.00 [  0.00]( 0.14)     1.00 [  0.00]( 0.00)
    8     1.00 [  0.00]( 0.00)     1.00 [  0.00]( 0.14)
   16     1.00 [  0.00]( 0.00)     1.00 [  0.00]( 0.00)
   32     1.00 [  0.00]( 5.05)     0.97 [ -3.11]( 1.91)
   64     1.00 [  0.00](10.41)     1.06 [  5.60]( 3.79)
  128     1.00 [  0.00]( 0.30)     0.98 [ -2.38]( 0.31)
  256     1.00 [  0.00]( 1.43)     0.98 [ -1.73]( 1.38)
  512     1.00 [  0.00]( 1.45)     0.97 [ -3.33]( 1.48)
  
  
  ==================================================================
  Test          : new-schbench-wakeup-latency
  Units         : Normalized 99th percentile latency in us
  Interpretation: Lower is better
  Statistic     : Median
  ==================================================================
  #workers: tip[pct imp](CV)    nohz_no_nr_cpus[pct imp](CV)
    1     1.00 [ -0.00](24.99)     1.08 [ -8.33](16.90)
    2     1.00 [ -0.00]( 0.00)     1.40 [-40.00](18.20)
    4     1.00 [ -0.00](12.06)     1.27 [-27.27]( 7.75)
    8     1.00 [ -0.00](14.13)     0.90 [ 10.00](23.66)
   16     1.00 [ -0.00](15.96)     1.09 [ -9.09]( 7.45)
   32     1.00 [ -0.00](12.06)     0.91 [  9.09](18.23)
   64     1.00 [ -0.00](15.78)     1.06 [ -6.25](13.18)
  128     1.00 [ -0.00](10.57)     1.03 [ -3.41]( 5.15)
  256     1.00 [ -0.00]( 0.32)     1.00 [ -0.00]( 0.21)
  512     1.00 [ -0.00]( 0.00)     1.00 [  0.38]( 0.20)
  
  
  ==================================================================
  Test          : new-schbench-request-latency
  Units         : Normalized 99th percentile latency in us
  Interpretation: Lower is better
  Statistic     : Median
  ==================================================================
  #workers: tip[pct imp](CV)    nohz_no_nr_cpus[pct imp](CV)
    1     1.00 [ -0.00]( 0.00)     1.00 [ -0.27]( 1.79)
    2     1.00 [ -0.00]( 0.74)     0.96 [  4.07]( 1.90)
    4     1.00 [ -0.00]( 0.37)     0.96 [  3.83]( 1.91)
    8     1.00 [ -0.00]( 1.02)     1.00 [ -0.28]( 1.52)
   16     1.00 [ -0.00]( 1.61)     1.00 [  0.28]( 1.86)
   32     1.00 [ -0.00]( 9.22)     1.04 [ -3.52]( 6.84)
   64     1.00 [ -0.00]( 6.39)     1.06 [ -5.96](22.58)
  128     1.00 [ -0.00]( 1.08)     1.12 [-12.43]( 4.61)
  256     1.00 [ -0.00]( 6.10)     1.01 [ -0.77]( 4.87)
  512     1.00 [ -0.00]( 1.41)     1.01 [ -1.03]( 1.27) 

-- 
Thanks and Regards,
Prateek


      parent reply	other threads:[~2026-01-13  6:45 UTC|newest]

Thread overview: 13+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-01-12  5:04 Shrikanth Hegde
2026-01-12  5:04 ` [PATCH v4 1/3] sched/fair: Move checking for nohz cpus after time check Shrikanth Hegde
2026-01-13  9:07   ` Vincent Guittot
2026-01-13  9:23     ` Shrikanth Hegde
2026-01-13 10:21       ` Vincent Guittot
2026-01-13 11:18         ` Shrikanth Hegde
2026-01-13 13:20           ` Vincent Guittot
2026-01-12  5:04 ` [PATCH v4 2/3] sched/fair: Change likelyhood of nohz.nr_cpus Shrikanth Hegde
2026-01-12  5:04 ` [PATCH v4 3/3] sched/fair: Remove nohz.nr_cpus and use weight of cpumask instead Shrikanth Hegde
2026-01-12 11:49   ` Valentin Schneider
2026-01-13  9:23   ` Vincent Guittot
2026-01-13  9:30     ` Shrikanth Hegde
2026-01-13  6:45 ` K Prateek Nayak [this message]

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=3298a289-22c4-42c4-a7c1-39d9519f6223@amd.com \
    --to=kprateek.nayak@amd.com \
    --cc=anna-maria@linutronix.de \
    --cc=dietmar.eggemann@arm.com \
    --cc=frederic@kernel.org \
    --cc=juri.lelli@redhat.com \
    --cc=linux-kernel@vger.kernel.org \
    --cc=mingo@kernel.org \
    --cc=peterz@infradead.org \
    --cc=sshegde@linux.ibm.com \
    --cc=tglx@kernel.org \
    --cc=vincent.guittot@linaro.org \
    --cc=vschneid@redhat.com \
    --cc=wangyang.guo@intel.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox

all inboxes | Powered by JetHome®