From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3A15FC433F5 for ; Thu, 24 Mar 2022 20:33:17 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1353424AbiCXUer convert rfc822-to-8bit (ORCPT ); Thu, 24 Mar 2022 16:34:47 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38500 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1345637AbiCXUeq (ORCPT ); Thu, 24 Mar 2022 16:34:46 -0400 X-Greylist: delayed 1232 seconds by postgrey-1.37 at lindbergh.monkeyblade.net; Thu, 24 Mar 2022 13:33:13 PDT Received: from mail2.intersystems.com (mail2.intersystems.com [38.105.105.84]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8270FDF7F for ; Thu, 24 Mar 2022 13:33:11 -0700 (PDT) X-InterSystems: Sent from InterSystems X-InterSystems: Sent from InterSystems X-InterSystems: Sent from InterSystems X-InterSystems: Sent from InterSystems From: Ray Fucillo To: "linux-kernel@vger.kernel.org" Subject: scalability regressions related to hugetlb_fault() changes Thread-Topic: scalability regressions related to hugetlb_fault() changes Thread-Index: AQHYP7uAL1vA/rQijE+hDIc6LhXGuA== Date: Thu, 24 Mar 2022 20:12:35 +0000 Message-ID: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [172.17.254.204] x-c2processedorg: 5d7e5ca7-6395-445f-80da-8568a4fc58e5 Content-Type: text/plain; charset="us-ascii" Content-ID: <48F2A26C80B440459846FEED0F086F05@exchangemail.iscinternal.com> Content-Transfer-Encoding: 8BIT MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org In moving to newer versions of the kernel, our customers have experienced dramatic new scalability problems in our database application, InterSystems IRIS. Our research has narrowed this down to new processes that attach to the database's shared memory segment taking very long delays (in some cases ~100ms!) acquiring the i_mmap_lock_read() in hugetlb_fault() as they fault in the huge page for the first time. The addition of this lock in hugetlb_fault() matches the versions where we see this problem. It's not just slowing the new process that incurs the delay, but backing up other processes if the page fault occurs inside a critical section within the database application. Is there something that can be improved here? The read locks in hugetlb_fault() contend with write locks that seem to be taken in very common application code paths: shmat(), process exit, fork() (not vfork()), shmdt(), presumably others. So hugetlb_fault() contending to read turns out to be common. When the system is loaded, there will be many new processes faulting in pages that may blocks the write lock, which in turn blocks more readers in fault behind it, and so on... I don't think there's any support for shared page tables in hugetlb to avoid the faults altogether. Switching to 1GB huge pages instead of 2MB is a good mitigation in reducing the frequency of fault, but not a complete solution. Thanks for considering. Ray