From: "Paul E. McKenney" <paulmck@kernel.org>
To: Feng Tang <feng.tang@intel.com>
Cc: kernel test robot <oliver.sang@intel.com>,
0day robot <lkp@intel.com>, Thomas Gleixner <tglx@linutronix.de>,
John Stultz <john.stultz@linaro.org>,
Stephen Boyd <sboyd@kernel.org>, Jonathan Corbet <corbet@lwn.net>,
Mark Rutland <Mark.Rutland@arm.com>,
Marc Zyngier <maz@kernel.org>, Andi Kleen <ak@linux.intel.com>,
Xing Zhengjun <zhengjun.xing@linux.intel.com>,
LKML <linux-kernel@vger.kernel.org>,
lkp@lists.01.org, ying.huang@intel.com, zhengjun.xing@intel.com,
kernel-team@fb.com, neeraju@codeaurora.org
Subject: Re: [clocksource] 388450c708: netperf.Throughput_tps -65.1% regression
Date: Fri, 14 May 2021 10:49:08 -0700 [thread overview]
Message-ID: <20210514174908.GI975577@paulmck-ThinkPad-P17-Gen-1> (raw)
In-Reply-To: <20210514074314.GB5384@shbuild999.sh.intel.com>
On Fri, May 14, 2021 at 03:43:14PM +0800, Feng Tang wrote:
> Hi Paul,
>
> On Thu, May 13, 2021 at 10:07:07AM -0700, Paul E. McKenney wrote:
> > On Thu, May 13, 2021 at 11:55:15PM +0800, kernel test robot wrote:
> > >
> > >
> > > Greeting,
> > >
> > > FYI, we noticed a -65.1% regression of netperf.Throughput_tps due to commit:
> > >
> > >
> > > commit: 388450c7081ded73432e2b7148c1bb9a0b039963 ("[PATCH v12 clocksource 4/5] clocksource: Reduce clocksource-skew threshold for TSC")
> > > url: https://github.com/0day-ci/linux/commits/Paul-E-McKenney/Do-not-mark-clocks-unstable-due-to-delays-for-v5-13/20210501-083404
> > > base: https://git.kernel.org/cgit/linux/kernel/git/tip/tip.git 2d036dfa5f10df9782f5278fc591d79d283c1fad
> > >
> > > in testcase: netperf
> > > on test machine: 96 threads 2 sockets Ice Lake with 256G memory
> > > with following parameters:
> > >
> > > ip: ipv4
> > > runtime: 300s
> > > nr_threads: 25%
> > > cluster: cs-localhost
> > > test: UDP_RR
> > > cpufreq_governor: performance
> > > ucode: 0xb000280
> > >
> > > test-description: Netperf is a benchmark that can be use to measure various aspect of networking performance.
> > > test-url: http://www.netperf.org/netperf/
> > >
> > >
> > >
> > > If you fix the issue, kindly add following tag
> > > Reported-by: kernel test robot <oliver.sang@intel.com>
> > >
> > >
> > > also as Feng Tang checked, this is a "unstable clocksource" case.
> > > attached dmesg FYI.
> >
> > Agreed, given the clock-skew event and the resulting switch to HPET,
> > performance regressions are expected behavior.
> >
> > That dmesg output does demonstrate the value of Feng Tang's patch!
> >
> > I don't see how to obtain the values of ->mult and ->shift that would
> > allow me to compute the delta. So if you don't tell me otherwise, I
> > will assume that the skew itself was expected on this hardware, perhaps
> > somehow due to the tpm_tis_status warning immediately preceding the
> > clock-skew event. If my assumption is incorrect, please let me know.
>
> I run the case with the debug patch applied, the info is:
>
> [ 13.796429] clocksource: timekeeping watchdog on CPU19: Marking clocksource 'tsc' as unstable because the skew is too large:
> [ 13.797413] clocksource: 'hpet' wd_nesc: 505192062 wd_now: 10657158 wd_last: fac6f97 mask: ffffffff
> [ 13.797413] clocksource: 'tsc' cs_nsec: 504008008 cs_now: 3445570292aa5 cs_last: 344551f0cad6f mask: ffffffffffffffff
> [ 13.797413] clocksource: 'tsc' is current clocksource.
> [ 13.797413] tsc: Marking TSC unstable due to clocksource watchdog
> [ 13.844513] clocksource: Checking clocksource tsc synchronization from CPU 50 to CPUs 0-1,12,22,32-33,60,65.
> [ 13.855080] clocksource: Switched to clocksource hpet
>
> So the delta is 1184 us (505192062 - 504008008), and I agree with
> you that it should be related with the tpm_tis_status warning stuff.
>
> But this re-trigger my old concerns, that if the margins calculated
> for tsc, hpet are too small?
If the error really did disturb either tsc or hpet, then we really
do not have a false positive, and nothing should change (aside from
perhaps documenting that TPM issues can disturb the clocks, or better
yet treating that perturbation as a separate bug that should be fixed).
But if this is yet another way to get a confused measurement, then it
would be better to work out a way to reject the confusion and keep the
tighter margins. I cannot think right off of a way that this could
cause measurement confusion, but you never know.
So any thoughts on exactly how the tpm_tis_status warning might have
resulted in the skew?
> With current math algorithm, the 'uncertainty_margin' is
> calculated against the frequency, and those tsc/hpet/acpi_pm
> timer is multiple of MHz or GHz, which gives them to have margin of
> 100 us. It works with normal systems. But in the wild world, there
> could be some sparkles due to some immature HW components, their
> firmwares or drivers etc, just like this case.
Isn't diagnosing issues from immature hardware, firmware, and drivers
actually a benefit? It would after all be quite unfortunate if some issue
that was visible only due to clock skew were to escape into production.
Thanx, Paul
next prev parent reply other threads:[~2021-05-14 17:49 UTC|newest]
Thread overview: 18+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-05-01 0:32 [PATCH v12 clocksource 0/5] Do not mark clocks unstable due to delays for v5.13 Paul E. McKenney
2021-05-01 0:32 ` [PATCH v12 clocksource 1/5] clocksource: Retry clock read if long delays detected Paul E. McKenney
2021-05-01 0:32 ` [PATCH v12 clocksource 2/5] clocksource: Check per-CPU clock synchronization when marked unstable Paul E. McKenney
2021-05-01 0:32 ` [PATCH v12 clocksource 3/5] clocksource: Limit number of CPUs checked for clock synchronization Paul E. McKenney
2021-05-01 0:32 ` [PATCH v12 clocksource 4/5] clocksource: Reduce clocksource-skew threshold for TSC Paul E. McKenney
2021-05-13 15:55 ` [clocksource] 388450c708: netperf.Throughput_tps -65.1% regression kernel test robot
2021-05-13 17:07 ` Paul E. McKenney
2021-05-14 7:43 ` Feng Tang
2021-05-14 17:49 ` Paul E. McKenney [this message]
2021-05-16 6:34 ` Feng Tang
2021-05-17 2:30 ` Paul E. McKenney
2021-05-19 6:09 ` Feng Tang
2021-05-19 18:05 ` Paul E. McKenney
2021-05-20 0:52 ` Feng Tang
2021-05-20 4:55 ` Paul E. McKenney
2021-05-01 0:32 ` [PATCH v12 clocksource 5/5] clocksource: Provide kernel module to test clocksource watchdog Paul E. McKenney
2021-05-01 9:49 ` kernel test robot
2021-05-01 16:01 ` Paul E. McKenney
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20210514174908.GI975577@paulmck-ThinkPad-P17-Gen-1 \
--to=paulmck@kernel.org \
--cc=Mark.Rutland@arm.com \
--cc=ak@linux.intel.com \
--cc=corbet@lwn.net \
--cc=feng.tang@intel.com \
--cc=john.stultz@linaro.org \
--cc=kernel-team@fb.com \
--cc=linux-kernel@vger.kernel.org \
--cc=lkp@intel.com \
--cc=lkp@lists.01.org \
--cc=maz@kernel.org \
--cc=neeraju@codeaurora.org \
--cc=oliver.sang@intel.com \
--cc=sboyd@kernel.org \
--cc=tglx@linutronix.de \
--cc=ying.huang@intel.com \
--cc=zhengjun.xing@intel.com \
--cc=zhengjun.xing@linux.intel.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®