From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1755350AbdK2ORx (ORCPT ); Wed, 29 Nov 2017 09:17:53 -0500 Received: from mx1.redhat.com ([209.132.183.28]:42328 "EHLO mx1.redhat.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752486AbdK2ORu (ORCPT ); Wed, 29 Nov 2017 09:17:50 -0500 From: Waiman Long To: Andrew Morton , Vladimir Davydov , Johannes Weiner , Dave Chinner Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, Waiman Long Subject: [PATCH] list_lru: Prefetch neighboring list entries before acquiring lock Date: Wed, 29 Nov 2017 09:17:34 -0500 Message-Id: <1511965054-6328-1-git-send-email-longman@redhat.com> X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.39]); Wed, 29 Nov 2017 14:17:50 +0000 (UTC) Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org The list_lru_del() function removes the given item from the LRU list. The operation looks simple, but it involves writing into the cachelines of the two neighboring list entries in order to get the deletion done. That can take a while if the cachelines aren't there yet, thus prolonging the lock hold time. To reduce the lock hold time, the cachelines of the two neighboring list entries are now prefetched before acquiring the list_lru_node's lock. Using a multi-threaded test program that created a large number of dentries and then killed them, the execution time was reduced from 38.5s to 36.6s after applying the patch on a 2-socket 36-core 72-thread x86-64 system. Signed-off-by: Waiman Long --- mm/list_lru.c | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/mm/list_lru.c b/mm/list_lru.c index f141f0c..65aae44 100644 --- a/mm/list_lru.c +++ b/mm/list_lru.c @@ -132,8 +132,16 @@ bool list_lru_del(struct list_lru *lru, struct list_head *item) struct list_lru_node *nlru = &lru->node[nid]; struct list_lru_one *l; + /* + * Prefetch the neighboring list entries to reduce lock hold time. + */ + if (unlikely(list_empty(item))) + return false; + prefetchw(item->prev); + prefetchw(item->next); + spin_lock(&nlru->lock); - if (!list_empty(item)) { + if (likely(!list_empty(item))) { l = list_lru_from_kmem(nlru, item); list_del_init(item); l->nr_items--; -- 1.8.3.1