Re: [RFC PATCH] tsc: synchronize TSCs on buggy Intel Xeon E5 CPUs with offset error

[Gratian Crisan <gratian.crisan@ni.com>]

Peter Zijlstra writes:

> On Wed, Nov 11, 2015 at 09:41:25AM -0600, Gratian Crisan wrote:
>> I also wrote a small C utility[1], with a bit of code borrowed from the
>> kernel, for reading the TSC on all CPUs. It starts a high priority
>> thread per CPU, tries to synchronize them and prints out the TSC values
>> and their offset with regards to CPU0.
>> It can be called from a SysV init shell script[2] at the beginning of
>> the boot process and right before a reboot to save the values in a file.
>
> Could you also read and print TSC_ADJUST (msr 0x3b) ? This would tell us
> if for example your BIOS messed it up.

Good call on the TSC_ADJUST. The BIOS seems to set it for CPU0 but not
for any of the other ones. I'll bug our BIOS guys about it.
Here's how the data looks after a couple reboots:

stop      : 127385698358[0] 741784252175365[741656866477007] 741784252175432[741656866477074] 741784252175471[741656866477113] 741784252175349[741656866476991] 741784252175458[741656866477100] 741784252175285[741656866476927] 741784252175501[741656866477143] 
TSC_ADJUST: -741656866477048 0 0 0 0 0 0 0 

start     : 47601069657[0] 741849504816842[741801903747185] 741849504816884[741801903747227] 741849504817004[741801903747347] 741849504817113[741801903747456] 741849504817051[741801903747394] 741849504816746[741801903747089] 741849504816962[741801903747305] 
TSC_ADJUST: -741801903747272 0 0 0 0 0 0 0 

stop      : 127495422447[0] 741929399169793[741801903747346] 741929399169821[741801903747374] 741929399169739[741801903747292] 741929399169767[741801903747320] 741929399169657[741801903747210] 741929399169612[741801903747165] 741929399169679[741801903747232] 
TSC_ADJUST: -741801903747272 0 0 0 0 0 0 0 

start     : 47522880051[0] 741994508088208[741946985208157] 741994508088258[741946985208207] 741994508088305[741946985208254] 741994508088110[741946985208059] 741994508088052[741946985208001] 741994508088020[741946985207969] 741994508087930[741946985207879] 
TSC_ADJUST: -741946985208111 0 0 0 0 0 0 0

Thanks a lot for helping with this,
-Gratian