From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 41BD4C77B7C for ; Fri, 12 May 2023 19:01:56 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S239210AbjELTBy (ORCPT ); Fri, 12 May 2023 15:01:54 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35662 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237651AbjELTBw (ORCPT ); Fri, 12 May 2023 15:01:52 -0400 Received: from mx0a-00069f02.pphosted.com (mx0a-00069f02.pphosted.com [205.220.165.32]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 30770268D for ; Fri, 12 May 2023 12:01:51 -0700 (PDT) Received: from pps.filterd (m0246627.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 34CIoQkV023033; Fri, 12 May 2023 19:01:35 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : content-transfer-encoding : content-type : mime-version; s=corp-2023-03-30; bh=6UMyziWN70NXLZyyndT1rOtXcbcEJJFo/akBpk6OYoU=; b=sKtxrqlK4XuJaMKIYgggmujXB1CJaGFMAhm8Uw+IomNByENVEWpoJ2/UGroCLsAl0d4L SLPiJbRNoU0wRDoMxSVIXnyFoAYk3R9BJqPYpf+WScebSn17Oiy/WWC13ryzlFdZQD88 ItA1I806V1Lt8tWwBA3Ru3ArYG+J7CMTl1MtaRf+3OCsadnyVBdYMFEKIcgqhchRnt1u bt0WHjmQNTBjO7KrAxqQ/mFEa3mdUAna+FhE9qyISuNVlKOkPAifjYHd2DPhaE9jpCan TaPVqAsz8u/Zqn0LOEQLYd/HQh4zrDX0kKci24Nme3RpjObFs2c+GQwkTorUH9nXUpZS Dg== Received: from phxpaimrmta03.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta03.appoci.oracle.com [138.1.37.129]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3qf77gbuk6-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 12 May 2023 19:01:35 +0000 Received: from pps.filterd (phxpaimrmta03.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by phxpaimrmta03.imrmtpd1.prodappphxaev1.oraclevcn.com (8.17.1.19/8.17.1.19) with ESMTP id 34CI8S1S011705; Fri, 12 May 2023 19:01:34 GMT Received: from nam10-bn7-obe.outbound.protection.outlook.com (mail-bn7nam10lp2109.outbound.protection.outlook.com [104.47.70.109]) by phxpaimrmta03.imrmtpd1.prodappphxaev1.oraclevcn.com (PPS) with ESMTPS id 3qf8158c3a-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 12 May 2023 19:01:34 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=HC3T7C/D9V33AfO8TIMUcmit9eCSqemjmM2FLTB/nWPpEYqi2Va4ZvYapkoXXnU5+YsJXbrxortUgQ1PKYaiElnlIL5IPne0zSZgfORm1PpLDfEE6vTK1BQcNDJnVvZfD56GamVfXnXiAywir4Dzb42WEwQv4QIe+MW9dNX6zNtj/74arSFhqU6WmFcCM8qnaIQODyoR1zkgLdjWf9hCvbmMCQFY0ZpRFXF1lQXXwIDpJNdb9loNlz5lYozpUbwEYzk0OMyHI41DB+wFGGjnDoRIh+XbHe1gS4pQY5W/DfNBqIM3ugDdwgQhSWglb7V34/K1McSk1c8WWB0CncrAkw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=6UMyziWN70NXLZyyndT1rOtXcbcEJJFo/akBpk6OYoU=; b=foQfHOc0L6PXnjPnK36koOODzdV5wgJ/AL5cL4yCsLKMeuy2MYgYPiGLeAwX8UR2cC3c2wP7pyuNbahvpbQriiPZ2LFQiKzkfYYgydSnULFY4P/+kkCF1wX6mqygd7XIsjFNwF/SxDjWwilRgHUk0elGYg3qF1D7037M3ymXPRNeLghK0c6RcX85/twwuLDEJ0NYHdVBemrWJj+zG0p//KDFaYo/wMq4JHpP9jUPL8pAhDouEL6KCOlqLtpaS8PmI6Ol9Ig2X6U2O+0Ndbe4R2fukrOGjTLa+tICNpzgdcPZyqhseVCWPHG2hc7sini5SNAraCLpTDSNgQ6xGo8bsA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=6UMyziWN70NXLZyyndT1rOtXcbcEJJFo/akBpk6OYoU=; b=J/vVwhMUGSFy+0jkezklHyCSKyE46qQqsoTz86ueeoN8YsHBZy6Hume8iU/ewGcQtKlgbmY5ekU9d4FLsrSzwHPKIl5j9uRC5fTvQq0RzmYJW4VspzlL/CqscYMdGnCJor0Hs4Qfwiyz3cRVwDEPjIAkddnETK5RrCzukJPe3+g= Received: from CH3PR10MB6810.namprd10.prod.outlook.com (2603:10b6:610:140::15) by DS0PR10MB6973.namprd10.prod.outlook.com (2603:10b6:8:14d::17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6387.22; Fri, 12 May 2023 19:01:31 +0000 Received: from CH3PR10MB6810.namprd10.prod.outlook.com ([fe80::ba49:e5dd:6a80:37ae]) by CH3PR10MB6810.namprd10.prod.outlook.com ([fe80::ba49:e5dd:6a80:37ae%6]) with mapi id 15.20.6387.018; Fri, 12 May 2023 19:01:31 +0000 From: Khalid Aziz To: akpm@linux-foundation.org Cc: Khalid Aziz , willy@infradead.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Steve Sistare , Khalid Aziz Subject: [PATCH v2] mm, compaction: Skip all non-migratable pages during scan Date: Fri, 12 May 2023 13:00:00 -0600 Message-Id: <20230512190000.103737-1-khalid.aziz@oracle.com> X-Mailer: git-send-email 2.37.2 Content-Transfer-Encoding: 8bit Content-Type: text/plain X-ClientProxiedBy: BY5PR17CA0003.namprd17.prod.outlook.com (2603:10b6:a03:1b8::16) To CH3PR10MB6810.namprd10.prod.outlook.com (2603:10b6:610:140::15) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CH3PR10MB6810:EE_|DS0PR10MB6973:EE_ X-MS-Office365-Filtering-Correlation-Id: 46ab2c8d-ae33-48e8-05cd-08db531b4c72 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: h1HQS4ghQ65iwxJ1igXAG5yRZ1DqqOSlZvr2Tbz77TEcR5ZU/TYUizVd/k4VF4Fax4dGMJcweBssPVBtAUWyr0pfnsGTOWlnucri9gzizq66bvcc+vmrEBkCL3j9pUUfdSGafs+/IZFiBoAZXGIAfb73f9tGUplcmGjp4Sjy+S9RphFV40xCtaxsYvp1JwBMG7CC5BzwEP7ox6FsY5b1kL9U+bH5yufn2wiqFdy1ir0JyUCRynZoRqSRREv00MLMSCroXLnDdhv/BEzJDv2kyaDoh5WvXfY4l3amXvilmI4evUcfJAzK1oz781G6XfoYKZqj0ZbgNvVdNCQFxZxC5hz/y4LQgjLK8xx12KNA4+LVdigZYn0Vi63RrHRgkQVw5yoGYq53u1A62m4vStAQ/c7kKkMGHdz+Dqpzi8LcxbRIRddGh33tzL7//PgYnGICBOYe3qo4grTb8XuyZ4OCBJciOAOpPmfbBKo3HhFp/bNu1ttRZWs+Bq6arpswbQIBbAvjiuzwu6dUJ1fwYOhNx3UQtC+lgE2SZ/pGYBAbtyntt0ksrrKiB0Qrf6k5iELC X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:CH3PR10MB6810.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230028)(396003)(346002)(376002)(39860400002)(136003)(366004)(451199021)(26005)(2616005)(1076003)(6512007)(6506007)(41300700001)(38100700002)(83380400001)(6486002)(186003)(6666004)(478600001)(54906003)(4326008)(6916009)(66556008)(66476007)(66946007)(5660300002)(44832011)(2906002)(86362001)(8936002)(36756003)(8676002)(316002);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?VZ7d8BBzvuBq1cPg3kTgBmUTGn+NKmvNOFzT7NGomSEArmvbs6aJVY+aHvwl?= =?us-ascii?Q?mtOlJPH7N4z+f7THDPLjsfAt0s/o7eQIed65jkxMaMfSSOlQnIRJHqOYsADy?= =?us-ascii?Q?Ig8YeZEnxcjyII6Tb6FE7aMD2yJUEdAkY7JUDg3uN3OYDPr52wYPCjz5B/qS?= =?us-ascii?Q?TAkL5XZKXPrPsDwNN7Zg1kudEUUXTcUEMNKxLj0010dHRU5pQVo3IsVNEU8k?= =?us-ascii?Q?0JFasqSgdJXiK8oyJappUSuOC3kIQs9D946UgtE9BMExlQg8eA0qVWmSthKl?= =?us-ascii?Q?+jTejkA4C8D/T3W0JaVc4nVz8a9hi25WfxuC9IEQLEffkpq6K+s+mD/lwKNJ?= =?us-ascii?Q?tZhJh+WUDlRW4CaSEnu30ZQKIm0mKkP3+22085wl3JgqYygTC12xCjBgXtVx?= =?us-ascii?Q?wwkuGKM8xHTCHzFQ93yyYVqpgTczM5wKPlXAjZzvJxDGDlxsAvYQn42Btlqb?= =?us-ascii?Q?xtRmI4vzsJe+MhWW/X/OxDySwW4A0QCKpf54GzzILh1iLp6rR+Z9fpXakrEl?= =?us-ascii?Q?U+GHdyL865iYsvlCROSSNFVUL7G/8jY2p3WAtKMYyH5v+AZ7m8/9IIhXMSkZ?= =?us-ascii?Q?d883KKHFEkKNCTMD9+Ou81fcaOveEtzLQjk1JBooWdWldhKWcfcodIIlYHRq?= =?us-ascii?Q?yvR5AvYnpb+9O1zXR2cg0aIkw0Y9N3vC/0IMHV8KIs79Uf2RKrLcR67yU/RC?= =?us-ascii?Q?+WC7roKy/HYVBbo5S8QCjIt8BoDXYsF0HYr0xKp1gxp+ivpTvJwrInjf27sJ?= =?us-ascii?Q?zjjb8tWOSqJeDM8qih3hLmCZnGF70b6IdQ4zCNv5YmmJdbWO4GNb5Y0gMFus?= =?us-ascii?Q?hr99I1hvbbnNl1g/bCWHxiV7CrObfdFHRDDur6Ax8XGywKjMdWWSqAjxA5B6?= =?us-ascii?Q?Sh6hQPN9S2k0tzVlT6YvZvloAcyd8IoSg26IvTlm5yD49rqnKYREKOqeqovP?= =?us-ascii?Q?x/uQFKvUpiR/9dXDvnEXcqqt8c0ne1cL980EicV43LSmgzzJXnfOIZM/Wz4P?= =?us-ascii?Q?31Iid0b+Fl7zQxK8m01hC6lhG5KQhH9X/halXksTKR3muz1s+Osm4i5V581R?= =?us-ascii?Q?QG+7PrHzinZyJbzgRHXd9ewGK9eInCbUJG+nL7eUVn8t/a+MsTxguykatQjF?= =?us-ascii?Q?H2nw6Cu1vOS46LxSY1dy17PhJbZWVZlepE39v1ERO8UB4zi4k1ZoAzMCNCt8?= =?us-ascii?Q?4MebMSMdAhvf2Zc1h7aNzQhQPqyrdbkFPCdwnFCNoogO41/pD+SgA0DEZn2K?= =?us-ascii?Q?deKE4UhGIFjNXT6BxwlWnH8YxklVvYI2lPTCOBwmiEH3sDQiKeDCqFqaUfrG?= =?us-ascii?Q?vOKlJnNjdRDEJ+59vFUXbjedrSW/rZ1OC2xsviEet/jPGyoFmTDCvLIf00zG?= =?us-ascii?Q?iTuG50awKbBtUfC7YyJ4RsGhOnlYs7GubXVbv3tjt4vGY9BvDixbjZkXm9nW?= =?us-ascii?Q?YWPdPjnyRNiLAul9ox3LJvwBCzWqVbe+o0fNI6HGtUL8RlOHV5sxNPRjTMOH?= =?us-ascii?Q?5Zx0QQiO1Ahc7lyzePTuTkO3y8rho7tGnTl72M4+Ef74Nz6VCJIK8F/7VQAr?= =?us-ascii?Q?/pVJDGP/+IRoFQYO06R+XORE036kUCUAE4GVDg0Q?= X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: 3fTvXp05SrAIL/tYlWu1fGYUJ/NlDK4AKWqMODeMcfzEFdiGy6VsGQMcYZJgxPuV5eYKJx2/30XAcMa0/HYDkIfg4DGzuMk9TUxkF3ROBlnv1moO04WlhhPmpd/Y4TkkQssQ11orTNBA5DfrlINxx9IEHtJuzoB67HnbWgED+RnGqp0Vs5h9bjVKGg0qCamunNk5WmAWMGn5EYvGLaLWa1clGjtrogzNopO9V4l1Gmp/HaNaRy0TcZmbRaveNGHfooIxFbezwcRCS9kZafnf4qyp5IC7Vlm0siUuJZdQv2LENTFMFQjOfDjIuBzHa+Ur9wlGtmSvcwRgouiBSdqcCN4IneEDS3d2zRtzPILsa7lKHOOU4XVMlBclJa+eJorPkJpM7Ncm4Qs0jgo4n93Q6EcMUKZmEtXLN+AQ9IcLrNu2WU18FJvmg/jsJh0q+6TA8BaUF7Ze0xME8kshbjdudgdC3u42SCvK4oXbI20BcrnpjzOISFYKntY5mf/4/4T5shwXVNhfgCtqBkjGYafm6qwWWGJxMnqA85sSptxcvunR1ZMhvhuFqIOKofSxzUSJbax2WSn45iq1nY+GKpQG01i/hye0I6DbpxcFghen6Hh5ZEryIfAYi3fbDJQt9BGmyRfKE1glKSYbcZgXtg9LVni2mY2Wt7U/FgrEqk97MgQEVKjfG96omz2EU4hSb6XUBrmVBCds6Qxm+AZx+JS6J2l7r6jQJmx19seHAkqGtWeIyihzcVux84UjgvypYFmZNzNELvUqfpptO5O8FPh/II8u3W6TJgBeAZGurI2AoBaCtWyDerpPTkbcNCisMJ/rjpVTilJpNp0I78J4PawNqwxjKv9+boLR4mjo/PnXaTeNqYGi5nPWZV6VJDUP/pIzSFPSU22oCDWwL/jiVvYeiyPES1mdQFwc0s622OmrUXM= X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 46ab2c8d-ae33-48e8-05cd-08db531b4c72 X-MS-Exchange-CrossTenant-AuthSource: CH3PR10MB6810.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 12 May 2023 19:01:31.7358 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: LqzOoL2XbqZT6aw3CwwFt0ZXB1SZPnUvSE9VmSLah2r7S6GlQyFod3aEfPp/51jjZba18ehgjS6x2qAssKMeaA== X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS0PR10MB6973 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.254,Aquarius:18.0.942,Hydra:6.0.573,FMLib:17.11.170.22 definitions=2023-05-12_11,2023-05-05_01,2023-02-09_01 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 mlxlogscore=830 phishscore=0 adultscore=0 spamscore=0 suspectscore=0 malwarescore=0 bulkscore=0 mlxscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2304280000 definitions=main-2305120159 X-Proofpoint-GUID: ftQTMl3oQCJlx8smriZW_67PA91T-Dcj X-Proofpoint-ORIG-GUID: ftQTMl3oQCJlx8smriZW_67PA91T-Dcj Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Pages pinned in memory through extra refcounts can not be migrated. Currently as isolate_migratepages_block() scans pages for compaction, it skips any pinned anonymous pages. All non-migratable pages should be skipped and not just the anonymous pinned pages. This patch adds a check for extra refcounts on a page to determine if the page can be migrated. This was seen as a real issue on a customer workload where a large number of pages were pinned by vfio on the host and any attempts to allocate hugepages resulted in significant amount of cpu time spent in either direct compaction or in kcompatd scanning vfio pinned pages over and over again that can not be migrated. Signed-off-by: Khalid Aziz Suggested-by: Steve Sistare Cc: Khalid Aziz --- v2: - Update comments in the code (Suggested by Andrew) - Use PagePrivate() instead of page_has_private() (Suggested by Matthew) - Pass mapping to page_has_extrarefs() (Suggested by Matthew) - Use page_ref_count() (Suggested by Matthew) - Rename is_pinned_page() to reflect its function more accurately (Suggested by Matthew) mm/compaction.c | 36 ++++++++++++++++++++++++++++++++---- 1 file changed, 32 insertions(+), 4 deletions(-) diff --git a/mm/compaction.c b/mm/compaction.c index 5a9501e0ae01..837f20df2bbb 100644 --- a/mm/compaction.c +++ b/mm/compaction.c @@ -764,6 +764,34 @@ static bool too_many_isolated(pg_data_t *pgdat) return too_many; } +/* + * Check if this base page should be skipped from isolation because + * it has extra refcounts that will prevent it from being migrated. + * This function is called for regular pages only, and not + * for THP or hugetlbfs pages. This code is inspired by similar code + * in migrate_vma_check_page(), can_split_folio() and + * folio_migrate_mapping() + */ +static inline bool page_has_extrarefs(struct page *page, + struct address_space *mapping) +{ + unsigned long extra_refs; + + /* anonymous page can have extra ref from swap cache */ + if (mapping) + extra_refs = 1 + PagePrivate(page); + else + extra_refs = PageSwapCache(page) ? 1 : 0; + + /* + * This is an admittedly racy check but good enough to determine + * if a page is pinned and can not be migrated + */ + if ((page_ref_count(page) - extra_refs) > page_mapcount(page)) + return true; + return false; +} + /** * isolate_migratepages_block() - isolate all migrate-able pages within * a single pageblock @@ -992,12 +1020,12 @@ isolate_migratepages_block(struct compact_control *cc, unsigned long low_pfn, goto isolate_fail; /* - * Migration will fail if an anonymous page is pinned in memory, - * so avoid taking lru_lock and isolating it unnecessarily in an - * admittedly racy check. + * Migration will fail if a page has extra refcounts + * preventing it from migrating, so avoid taking + * lru_lock and isolating it unnecessarily */ mapping = page_mapping(page); - if (!mapping && (page_count(page) - 1) > total_mapcount(page)) + if (page_has_extrarefs(page, mapping)) goto isolate_fail_put; /* -- 2.37.2