From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751324AbdH1S7T convert rfc822-to-8bit (ORCPT ); Mon, 28 Aug 2017 14:59:19 -0400 Received: from mx1.redhat.com ([209.132.183.28]:33944 "EHLO mx1.redhat.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751203AbdH1S7R (ORCPT ); Mon, 28 Aug 2017 14:59:17 -0400 DMARC-Filter: OpenDMARC Filter v1.3.2 mx1.redhat.com D205737E85 Authentication-Results: ext-mx05.extmail.prod.ext.phx2.redhat.com; dmarc=none (p=none dis=none) header.from=redhat.com Authentication-Results: ext-mx05.extmail.prod.ext.phx2.redhat.com; spf=fail smtp.mailfrom=longman@redhat.com Subject: Re: [PATCH v3 0/5] fs/dcache: Limit # of negative dentries From: Waiman Long To: Alexander Viro , Jonathan Corbet Cc: linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-fsdevel@vger.kernel.org, "Paul E. McKenney" , Andrew Morton , Ingo Molnar , Miklos Szeredi , Matthew Wilcox , Larry Woodman , James Bottomley , "Wangkai (Kevin,C)" References: <1501266880-26288-1-git-send-email-longman@redhat.com> <35820b85-b312-9329-656a-ac781abd9f11@redhat.com> Organization: Red Hat Message-ID: <55b05fda-e320-0059-0e2b-8ffaef0be0e8@redhat.com> Date: Mon, 28 Aug 2017 14:59:15 -0400 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Thunderbird/52.2.0 MIME-Version: 1.0 In-Reply-To: <35820b85-b312-9329-656a-ac781abd9f11@redhat.com> Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8BIT Content-Language: en-US X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.29]); Mon, 28 Aug 2017 18:59:17 +0000 (UTC) Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On 08/28/2017 01:58 PM, Waiman Long wrote: > On 07/28/2017 02:34 PM, Waiman Long wrote: >> v2->v3: >> - Add a faster pruning rate when the free pool is closed to depletion. >> - As suggested by James Bottomley, add an artificial delay waiting >> loop before killing a negative dentry and properly clear the >> DCACHE_KILL_NEGATIVE flag if killing doesn't happen. >> - Add a new patch to track number of negative dentries that are >> forcifully killed. >> >> v1->v2: >> - Move the new nr_negative field to the end of dentry_stat_t structure >> as suggested by Matthew Wilcox. >> - With the help of Miklos Szeredi, fix incorrect locking order in >> dentry_kill() by using lock_parent() instead of locking the parent's >> d_lock directly. >> - Correctly account for positive to negative dentry transitions. >> - Automatic pruning of negative dentries will now ignore the reference >> bit in negative dentries but not the regular shrinking. >> >> A rogue application can potentially create a large number of negative >> dentries in the system consuming most of the memory available. This >> can impact performance of other applications running on the system. >> >> This patchset introduces changes to the dcache subsystem to limit >> the number of negative dentries allowed to be created thus limiting >> the amount of memory that can be consumed by negative dentries. >> >> Patch 1 tracks the number of negative dentries used and disallow >> the creation of more when the limit is reached. >> >> Patch 2 enables /proc/sys/fs/dentry-state to report the number of >> negative dentries in the system. >> >> Patch 3 enables automatic pruning of negative dentries when it is >> close to the limit so that we won't end up killing recently used >> negative dentries. >> >> Patch 4 prevents racing between negative dentry pruning and umount >> operation. >> >> Patch 5 shows the number of forced negative dentry killings in >> /proc/sys/fs/dentry-state. End users can then tune the neg_dentry_pc= >> kernel boot parameter if they want to reduce forced negative dentry >> killings. >> >> Waiman Long (5): >> fs/dcache: Limit numbers of negative dentries >> fs/dcache: Report negative dentry number in dentry-state >> fs/dcache: Enable automatic pruning of negative dentries >> fs/dcache: Protect negative dentry pruning from racing with umount >> fs/dcache: Track count of negative dentries forcibly killed >> >> Documentation/admin-guide/kernel-parameters.txt | 7 + >> fs/dcache.c | 451 ++++++++++++++++++++++-- >> include/linux/dcache.h | 8 +- >> include/linux/list_lru.h | 1 + >> mm/list_lru.c | 4 +- >> 5 files changed, 435 insertions(+), 36 deletions(-) >> > With a 4.13 based kernel, the positive & negative dentries lookup rates > (lookups per second) after initial boot on a 32GB memory VM with and > without the patch were as follows: > > Metric w/o patch with patch > ------ --------- ---------- > Positive dentry lookup 844881 842618 > Negative dentry lookup 1865158 1901875 > Negative dentry creation 609887 617215 > > The last row refers to the creation rate of 10 millions negative > dentries with the negative dentry limit set to 50% (> 80M dentries). > Ignoring some inherent noise in the test results, there wasn't any > noticeable difference in term of lookup and negative dentry creation > performance with or without this patch. > > If the limit was set to 5% (the default), the 10M negative dentry > creation rate dropped to 199565 and the dentry-state was: > > 2344754 2326486 45 0 2316533 7494261 > > This was expected as negative dentry creation throttling with forced > dentry deletion happened in this case. > > IOW, this patch does not cause any regression in term of lookup and > negative dentry creation performance as long as the limit hasn't been > reached. Another performance data point about running the AIM7 highsystime workload on a 36-core 32G VM is as follows: Running the AIM7 high-systime workload on the VM, the baseline performance was 186770 jobs/min. By running a single-thread rogue negative dentry creation program in the background until the patched kernel with 5% limit started throttling, the performance was 183746 jobs/min. On an unpatched kernel with memory almost exhausted and memory shrinker was kicked in, the performance was 148997 jobs/min. So the patch does protect the system from suffering significant performance degradation in case a negative dentry creation rogue program is runninig in the background. Cheers, Longman