From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752454AbdI2Rjb (ORCPT ); Fri, 29 Sep 2017 13:39:31 -0400 Received: from g4t3426.houston.hpe.com ([15.241.140.75]:36056 "EHLO g4t3426.houston.hpe.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752113AbdI2Rja (ORCPT ); Fri, 29 Sep 2017 13:39:30 -0400 X-Greylist: delayed 8406 seconds by postgrey-1.27 at vger.kernel.org; Fri, 29 Sep 2017 13:39:30 EDT Subject: Re: [PATCH 0/4] x86/platform/UV: Update TSC support To: Peter Zijlstra Cc: Ingo Molnar , "H. Peter Anvin" , Thomas Gleixner , Bin Gao , Prarit Bhargava , Dimitri Sivanich , Andrew Banman , Russ Anderson , linux-kernel@vger.kernel.org, x86@kernel.org, kevin.b.stanton@intel.com References: <20170928180339.010141096@stormcage.americas.sgi.com> <20170929084632.gfwugbeaonraadp3@hirez.programming.kicks-ass.net> <20170929162350.adakhwxsqyq2ff2f@hirez.programming.kicks-ass.net> From: Mike Travis Message-ID: Date: Fri, 29 Sep 2017 10:39:28 -0700 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:52.0) Gecko/20100101 Thunderbird/52.3.0 MIME-Version: 1.0 In-Reply-To: <20170929162350.adakhwxsqyq2ff2f@hirez.programming.kicks-ass.net> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On 9/29/2017 9:23 AM, Peter Zijlstra wrote: > On Fri, Sep 29, 2017 at 08:19:22AM -0700, Mike Travis wrote: >>> So I would still like to get clarification on how ART works (or likely >>> doesn't) on your systems. I think for now its fairly prudent to kill >>> detect_art() on UV. >> >> I tested with both detect_art enabled and disabled and didn't notice a >> difference though I wasn't sure what test to run to verify whether it was >> being used or not. (I'd be glad to run some specific test if one can be >> suggested?) The num/denom setting for a 2100Mhz CPU was 168/2 if that >> information helps? > > While ART has a ratio to TSC, it too has an absolute relation to it. > Given an ART time stamp we can compute a TSC value and vice versa, this > allows correlating device timestamps (Network, Audio/Video etc..) with > CPU time stamps. > > Per detect_art() we have a single system wide offset, namely: > > rdmsrl(MSR_IA32_TSC_ADJUST, art_to_tsc_offset); > > But you use TSC_ADJUST to sync between your cabinets, this cannot ever > be right. The ART clock of the other cabinets (those that did not run > detect_art) will have a different offset. > > Currently there are only two device drivers that use ART: > > drivers/net/ethernet/intel/e1000e/ptp.c: *system = convert_art_to_tsc(sys_cycles); > sound/pci/hda/hda_controller.c: *system = convert_art_to_tsc(tsc_counter); > > Outside of that nobody cares, _for_now_. I'm checking with the hardware/firmware designers but your mention of e1000e reminded me that I did see this but didn't quite connect the meaning. If it's really a system wide constant, then yes we cannot provide a single value that would apply to all CPU's. > > I'm not sure if there's a means for the CPU to read ART in order to test > this correlation. > > Intel SDM Vol 3B 17.17.4 speaks of 'K' with a footnote about TSC_ADJUST > and the VMCS TSC fields. But basically both TSC and ART start at 0 on > power on and given the frequency ratio 'K' is a known for native system > agents. > > Again, I would suggest killing detect_art() (and the setting of > X86_FEATURE_ART) on UV systems until things are worked out. Also, given > you have your own distributed clock, I'm thinking you use that on your > own devices, obviating the immediate need for ART. > >>> Also, while indeed not strictly required, that TSC_ADJUST==0 test on >>> bootcpu is nice for consumer systems, BIOS did something 'weird' if that >>> is not true. Is something like is_uv_system() available early enough? >> >> My previous version of the patches had me setting a flag that could be >> checked by the tsc_sanitize_first_cpu() function and disable the requirement >> of "TSC == 0 on socket 0" for any arch that specified it. >> (And UV did set that flag.) >> >> But Thomas said it was "hackery" and that TSC being 0 on socket 0 was no >> longer a requirement. So I took it out for this version and made the "TSC >> == 0 on socket 0" no longer the default for any arch. > > That's where it comes from. But normal systems really _should_ have it > at 0 and its a useful sanity check IMO. We really want to know when the > BIOS does a funny behind our backs. > How about a more generic flag, such as "multi_tsc_sync_sources"? That could trigger both disabling the "TSC == 0 on socket 0" check as well as disabling X86_FEATURE_ART where appropriate? Or I could clear the feature ART cap separately in the UV system init code if they are not really related?