From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from m16.mail.126.com (m16.mail.126.com [220.197.31.8]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 0905B234987 for ; Mon, 13 Jan 2025 11:24:14 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=220.197.31.8 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1736767458; cv=none; b=fOQAz+oX7J8tXTirtfH3ZYvPB4AXbzn8FFdmAebka4qmfojKKOml8JtX2G/TtXUYWd+BHC+lYFFsZBwtrGxsy2sKpJWVZ7RX49qqyO4yPwxhsl8p4dckL3SDOIm4OJ/utLTq0WrupxVcW1YMt9eK5ASAG4rnZBufiaiLldZW+hk= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1736767458; c=relaxed/simple; bh=yPqfawszzlZtmy9n1cfjrNgziCix7sBcLrgonOMvBM4=; h=Message-ID:Date:MIME-Version:Subject:To:Cc:References:From: In-Reply-To:Content-Type; b=WTWCHko9XbOYvbH3W9dThWBtMZxfuzySix32SxUl7jb8wrvK8IlXsJCuOurPXUJhCdxJOKhKUG3A1CSH7S0e7xLQeMcix2k9p4hdrIiZOFErwmOqbamP6Ppo9TAUsuMGM4aA378pIcZ/mhp9qDLnTaxWb1S8qXfWgaAPoyUdIHM= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=126.com; spf=pass smtp.mailfrom=126.com; dkim=pass (1024-bit key) header.d=126.com header.i=@126.com header.b=C8m1vER/; arc=none smtp.client-ip=220.197.31.8 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=126.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=126.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=126.com header.i=@126.com header.b="C8m1vER/" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=126.com; s=s110527; h=Message-ID:Date:MIME-Version:Subject:From: Content-Type; bh=Db0pLgV0yMFTyp7VrRJl9wkQ82DP+5RZQzKhO56UXEs=; b=C8m1vER/ItONgmzlPXmx2zPOTxZjk+Mi1A6F9Vg0QMI8BE9IHEVtwNLiZrxfDL D6b0kl/Q6ICsigZ+2nqIkorWmHlMA++qEKZuLSY0VNMgLlHwjvO0bBSJT1GGLRMU yKZSfBYM+hOVaY8bSkAtVDW+uCRchNuP4K/Qs3/9xgtdo= Received: from [172.19.20.199] (unknown []) by gzga-smtp-mtada-g1-1 (Coremail) with SMTP id _____wD31wq894Rn6tyPBA--.11799S2; Mon, 13 Jan 2025 19:23:40 +0800 (CST) Message-ID: Date: Mon, 13 Jan 2025 19:23:40 +0800 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [PATCH V3] mm: compaction: skip memory compaction when there are not enough migratable pages To: Barry Song <21cnbao@gmail.com> Cc: akpm@linux-foundation.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, david@redhat.com, baolin.wang@linux.alibaba.com, hannes@cmpxchg.org, liuzixing@hygon.cn References: <1736335854-548-1-git-send-email-yangge1116@126.com> From: Ge Yang In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-CM-TRANSID:_____wD31wq894Rn6tyPBA--.11799S2 X-Coremail-Antispam: 1Uf129KBjvJXoWxKF1xAw45CFWDuFW7Cr1fCrg_yoW7Gry3pr y8GFnxKF4DXFZIyr1Iq3ZYvFnxt3yfKF4UXr9rtF97u3sI9F1Iyr47tryUuF1kZr1Utr4Y vw4DX3sruan8Za7anT9S1TB71UUUUU7qnTZGkaVYY2UrUUUUjbIjqfuFe4nvWSU5nxnvy2 9KBjDUYxBIdaVFxhVjvjDU0xZFpf9x07jYpB-UUUUU= X-CM-SenderInfo: 51dqwwjhrrila6rslhhfrp/1tbifhvTG2eE44fx0wAAsI 在 2025/1/13 18:05, Barry Song 写道: > On Mon, Jan 13, 2025 at 10:04 PM Ge Yang wrote: >> >> >> >> 在 2025/1/13 16:47, Barry Song 写道: >>> On Thu, Jan 9, 2025 at 12:31 AM wrote: >>>> >>>> From: yangge >>>> >>>> There are 4 NUMA nodes on my machine, and each NUMA node has 32GB >>>> of memory. I have configured 16GB of CMA memory on each NUMA node, >>>> and starting a 32GB virtual machine with device passthrough is >>>> extremely slow, taking almost an hour. >>>> >>>> During the start-up of the virtual machine, it will call >>>> pin_user_pages_remote(..., FOLL_LONGTERM, ...) to allocate memory. >>>> Long term GUP cannot allocate memory from CMA area, so a maximum of >>>> 16 GB of no-CMA memory on a NUMA node can be used as virtual machine >>>> memory. There is 16GB of free CMA memory on a NUMA node, which is >>>> sufficient to pass the order-0 watermark check, causing the >>>> __compaction_suitable() function to consistently return true. >>>> However, if there aren't enough migratable pages available, performing >>>> memory compaction is also meaningless. Besides checking whether >>>> the order-0 watermark is met, __compaction_suitable() also needs >>>> to determine whether there are sufficient migratable pages available >>>> for memory compaction. >>>> >>>> For costly allocations, because __compaction_suitable() always >>>> returns true, __alloc_pages_slowpath() can't exit at the appropriate >>>> place, resulting in excessively long virtual machine startup times. >>>> Call trace: >>>> __alloc_pages_slowpath >>>> if (compact_result == COMPACT_SKIPPED || >>>> compact_result == COMPACT_DEFERRED) >>>> goto nopage; // should exit __alloc_pages_slowpath() from here >>>> >>>> When the 16G of non-CMA memory on a single node is exhausted, we will >>>> fallback to allocating memory on other nodes. In order to quickly >>>> fallback to remote nodes, we should skip memory compaction when >>>> migratable pages are insufficient. After this fix, it only takes a >>>> few tens of seconds to start a 32GB virtual machine with device >>>> passthrough functionality. >>>> >>>> Signed-off-by: yangge >>>> --- >>>> >>>> V3: >>>> - fix build error >>>> >>>> V2: >>>> - consider unevictable folios >>>> >>>> mm/compaction.c | 20 ++++++++++++++++++++ >>>> 1 file changed, 20 insertions(+) >>>> >>>> diff --git a/mm/compaction.c b/mm/compaction.c >>>> index 07bd227..a9f1261 100644 >>>> --- a/mm/compaction.c >>>> +++ b/mm/compaction.c >>>> @@ -2383,7 +2383,27 @@ static bool __compaction_suitable(struct zone *zone, int order, >>>> int highest_zoneidx, >>>> unsigned long wmark_target) >>>> { >>>> + pg_data_t __maybe_unused *pgdat = zone->zone_pgdat; >>>> + unsigned long sum, nr_pinned; >>>> unsigned long watermark; >>>> + >>>> + sum = node_page_state(pgdat, NR_INACTIVE_FILE) + >>>> + node_page_state(pgdat, NR_INACTIVE_ANON) + >>>> + node_page_state(pgdat, NR_ACTIVE_FILE) + >>>> + node_page_state(pgdat, NR_ACTIVE_ANON) + >>>> + node_page_state(pgdat, NR_UNEVICTABLE); >>>> + >>>> + nr_pinned = node_page_state(pgdat, NR_FOLL_PIN_ACQUIRED) - >>>> + node_page_state(pgdat, NR_FOLL_PIN_RELEASED); >>>> + >>> >>> Does the sum of all LRU pages equal non-CMA memory? >>> I'm quite confused for two reasons: >>> 1. CMA pages can be LRU pages. >>> 2. Free pages might not belong to any LRUs. >> NO. >> >> If all the pages in the LRU are pinned, it seems unnecessary to perform >> memory compaction, as the migration of pinned pages is unlikely to succeed. >> Besides checking whether the order-0 watermark is met, >> __compaction_suitable() also needs to determine whether there are >> sufficient migratable pages available for memory compaction. > > Ok, but I am not convinced that this is a correct patch. If all your > CMA pages are > used by userspace—in other words, they are in LRUs—the sum could become > quite large, and `nr_pinned` might include non-CMA pages. In that case, > `sum - nr_pinned` would also be quite large. The "return false" logic wouldn't > work as intended. > > I suspect the issue seems to have disappeared simply because your CMA is > not being used at all. > Part of the CMA has been used. Due to __compaction_suitable() always returning true, it triggers swapping, which evicts the already-used CMA pages to disk, ultimately resulting in only pinned pages remaining in the LRU (Least Recently Used) list. >>> >>> >>>> + /* >>>> + * Gup-pinned pages are non-migratable. After subtracting these pages, >>>> + * we need to check if the remaining pages are sufficient for memory >>>> + * compaction. >>>> + */ >>>> + if ((sum - nr_pinned) < (1 << order)) >>>> + return false; >>>> + >>>> /* >>>> * Watermarks for order-0 must be met for compaction to be able to >>>> * isolate free pages for migration targets. This means that the >>>> -- >>>> 2.7.4 >>>> >>>> >>> > > Thanks > Barry