From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1756165AbbI3KFe (ORCPT ); Wed, 30 Sep 2015 06:05:34 -0400 Received: from mail-qg0-f44.google.com ([209.85.192.44]:36733 "EHLO mail-qg0-f44.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S932247AbbI3KED convert rfc822-to-8bit (ORCPT ); Wed, 30 Sep 2015 06:04:03 -0400 Date: Wed, 30 Sep 2015 06:03:59 -0400 From: Jeff Layton To: "Huang\, Ying" Cc: kernel test robot , , LKML , bfields@fieldses.org Subject: Re: [lkp] [nfsd] 4aac1bf05b: -2.9% fsmark.files_per_sec Message-ID: <20150930060359.070dba46@tlielax.poochiereds.net> In-Reply-To: <871tdg718x.fsf@yhuang-dev.intel.com> References: <87h9mfhwcj.fsf@yhuang-dev.intel.com> <20150929074151.0fe32fc9@tlielax.poochiereds.net> <87lhbokdqd.fsf@yhuang-dev.intel.com> <20150929200613.66c3ab4f@synchrony.poochiereds.net> <871tdg718x.fsf@yhuang-dev.intel.com> X-Mailer: Claws Mail 3.11.1 (GTK+ 2.24.28; x86_64-redhat-linux-gnu) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8BIT Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Wed, 30 Sep 2015 16:35:58 +0800 "Huang\, Ying" wrote: > Jeff Layton writes: > > > On Wed, 30 Sep 2015 07:27:54 +0800 > > "Huang\, Ying" wrote: > > > >> Jeff Layton writes: > >> > >> > On Mon, 28 Sep 2015 14:49:32 +0800 > >> > kernel test robot wrote: > >> > > >> >> FYI, we noticed the below changes on > >> >> > >> >> ========================================================================================= > >> >> tbox_group/testcase/rootfs/kconfig/compiler/cpufreq_governor/iterations/nr_threads/disk/fs/fs2/filesize/test_size/sync_method/nr_directories/nr_files_per_directory: > >> >> lkp-ne04/fsmark/debian-x86_64-2015-02-07.cgz/x86_64-rhel/gcc-4.9/performance/1x/32t/1HDD/xfs/nfsv4/5K/400M/fsyncBeforeClose/16d/256fpd > >> >> > >> >> commit: > >> >> cd2d35ff27c4fda9ba73b0aa84313e8e20ce4d2c > >> >> 4aac1bf05b053a201a4b392dd9a684fb2b7e6103 > >> >> > >> > > >> > A question... > >> > > >> > I think my tree should now contain a fix for this, but with a > >> > performance regression like this it's difficult to know for sure. > >> > > >> > Is there some (automated) way to request that the KTR redo this test? > >> > If not, will I get a note saying "problem seems to now be fixed" or do > >> > I just take a lack of further emails from the KTR about this as a sign > >> > that it's resolved? > >> > >> Can you provide the branch name and commit ID for your tree with fix? I > >> can confirm whether it is fixed for you. > >> > > Sure: > > > > git://git.samba.org/jlayton/linux.git nfsd-4.4 > > > > The tip commit is ed3d7c1e01a76f5ecc7444067704a82af4c2f76e. > > > > It seems that the regression is fixed at that commit. Thanks! > > ========================================================================================= > tbox_group/testcase/rootfs/kconfig/compiler/cpufreq_governor/iterations/nr_threads/disk/fs/fs2/filesize/test_size/sync_method/nr_directories/nr_files_per_directory: > lkp-ne04/fsmark/debian-x86_64-2015-02-07.cgz/x86_64-rhel/gcc-4.9/performance/1x/32t/1HDD/xfs/nfsv4/5K/400M/fsyncBeforeClose/16d/256fpd > > commit: > cd2d35ff27c4fda9ba73b0aa84313e8e20ce4d2c > 4aac1bf05b053a201a4b392dd9a684fb2b7e6103 > ed3d7c1e01a76f5ecc7444067704a82af4c2f76e > > cd2d35ff27c4fda9 4aac1bf05b053a201a4b392dd9 ed3d7c1e01a76f5ecc74440677 > ---------------- -------------------------- -------------------------- > %stddev %change %stddev %change %stddev > \ | \ | \ > 14415356 ± 0% +2.6% 14788625 ± 1% +4.1% 15008301 ± 0% fsmark.app_overhead > 441.60 ± 0% -2.9% 428.80 ± 0% -0.4% 439.68 ± 0% fsmark.files_per_sec > 185.78 ± 0% +2.9% 191.26 ± 0% +0.3% 186.37 ± 0% fsmark.time.elapsed_time > 185.78 ± 0% +2.9% 191.26 ± 0% +0.3% 186.37 ± 0% fsmark.time.elapsed_time.max > 97472 ± 0% -2.8% 94713 ± 0% -0.8% 96657 ± 0% fsmark.time.involuntary_context_switches > > Best Regards, > Huang, Ying Thanks for testing it and catching the problem in the first place! FWIW, the problem seems to have been bad hash distribution generated by hash_ptr on struct inode pointers. When the cache had ~10000 entries in it total, one of the hash chains had almost 2000 entries. When I switched to hashing on inode->i_ino, the distribution was much better. I'm not sure if it was just rotten luck or there is something about inode pointers that makes hash_ptr generate a lot of duplicates. That really could use more investigation... -- Jeff Layton