From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from m16.mail.126.com (m16.mail.126.com [117.135.210.9]) by smtp.subspace.kernel.org (Postfix) with ESMTP id EE8191A8419 for ; Mon, 13 Jan 2025 09:04:34 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=117.135.210.9 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1736759079; cv=none; b=iMEKxmvxhCr+QXEIgObTNpo8AH9i1deyEfXSNyZsDpqjr7j1RYMdUF09sFiXhZ6WZzEqmh4HviFrT3xjXRyFn2qEwxFHRDJqDQRRMMPu6nKr9lzGGSbW/WoWMMF28SHchTTklbkfAA51WeupGPTt6suyjTi5HiM7q+OvrNCBdjA= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1736759079; c=relaxed/simple; bh=EgLdVsQ/ePt1zdP03KzFJILTLnNyjz7o3B5SpmbHQSM=; h=Message-ID:Date:MIME-Version:Subject:To:Cc:References:From: In-Reply-To:Content-Type; b=DDQOaypp66kgLDdwyu+cHYA7B5oT0EMxLXZxX4nIJYKxpjKT9NF1syUWIVj/J+DqYF5MpF9hlOAFuO20IXcWiZMnLQhbbnCAIJMoVSEFu/dzaDA5aXz9xa4F6zi+3XH75QGnUAjA20d+x4+WwoUV6c6amIhX1xuXtoUZAtJLiMA= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=126.com; spf=pass smtp.mailfrom=126.com; dkim=pass (1024-bit key) header.d=126.com header.i=@126.com header.b=qTjNONdI; arc=none smtp.client-ip=117.135.210.9 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=126.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=126.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=126.com header.i=@126.com header.b="qTjNONdI" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=126.com; s=s110527; h=Message-ID:Date:MIME-Version:Subject:From: Content-Type; bh=RGwztksnAIIz3KzDSysrdFJS1Cs/SJDQuD5QdAHGeDk=; b=qTjNONdIF419a4cuXjeYSp1tW+6KN2tTn8wM6UWuWhLqyf4neiYngOK9J5CTaX kFe6QnYMtukGMPejsYSBv+8Hx7PeSdMmiooPlixENObBX74L9Oy4f1qRLf0YCCHd gwAKqrTtRSSFHonwGdR410pWS7QzChzoVOXp5SnXndfYw= Received: from [172.19.20.199] (unknown []) by gzga-smtp-mtada-g1-3 (Coremail) with SMTP id _____wD3V7+g1oRn1ql1BA--.42667S2; Mon, 13 Jan 2025 17:02:25 +0800 (CST) Message-ID: Date: Mon, 13 Jan 2025 17:02:24 +0800 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [PATCH V3] mm: compaction: skip memory compaction when there are not enough migratable pages To: Barry Song <21cnbao@gmail.com> Cc: akpm@linux-foundation.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, david@redhat.com, baolin.wang@linux.alibaba.com, hannes@cmpxchg.org, liuzixing@hygon.cn References: <1736335854-548-1-git-send-email-yangge1116@126.com> From: Ge Yang In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-CM-TRANSID:_____wD3V7+g1oRn1ql1BA--.42667S2 X-Coremail-Antispam: 1Uf129KBjvJXoWxAF4kCFWxuF13WFy7uw15Jwb_yoWrXF4kpr y8CFnxKa1DXFZFkw1Iq3WvyF9Iyw4fKF4UJF9FyF97uasI9FySyw17KryUCa1Dury0qr4Y va1Uu3srCan8Za7anT9S1TB71UUUUU7qnTZGkaVYY2UrUUUUjbIjqfuFe4nvWSU5nxnvy2 9KBjDUYxBIdaVFxhVjvjDU0xZFpf9x07jYGQhUUUUU= X-CM-SenderInfo: 51dqwwjhrrila6rslhhfrp/1tbiOg7TG2eEyGL04gAAsj 在 2025/1/13 16:47, Barry Song 写道: > On Thu, Jan 9, 2025 at 12:31 AM wrote: >> >> From: yangge >> >> There are 4 NUMA nodes on my machine, and each NUMA node has 32GB >> of memory. I have configured 16GB of CMA memory on each NUMA node, >> and starting a 32GB virtual machine with device passthrough is >> extremely slow, taking almost an hour. >> >> During the start-up of the virtual machine, it will call >> pin_user_pages_remote(..., FOLL_LONGTERM, ...) to allocate memory. >> Long term GUP cannot allocate memory from CMA area, so a maximum of >> 16 GB of no-CMA memory on a NUMA node can be used as virtual machine >> memory. There is 16GB of free CMA memory on a NUMA node, which is >> sufficient to pass the order-0 watermark check, causing the >> __compaction_suitable() function to consistently return true. >> However, if there aren't enough migratable pages available, performing >> memory compaction is also meaningless. Besides checking whether >> the order-0 watermark is met, __compaction_suitable() also needs >> to determine whether there are sufficient migratable pages available >> for memory compaction. >> >> For costly allocations, because __compaction_suitable() always >> returns true, __alloc_pages_slowpath() can't exit at the appropriate >> place, resulting in excessively long virtual machine startup times. >> Call trace: >> __alloc_pages_slowpath >> if (compact_result == COMPACT_SKIPPED || >> compact_result == COMPACT_DEFERRED) >> goto nopage; // should exit __alloc_pages_slowpath() from here >> >> When the 16G of non-CMA memory on a single node is exhausted, we will >> fallback to allocating memory on other nodes. In order to quickly >> fallback to remote nodes, we should skip memory compaction when >> migratable pages are insufficient. After this fix, it only takes a >> few tens of seconds to start a 32GB virtual machine with device >> passthrough functionality. >> >> Signed-off-by: yangge >> --- >> >> V3: >> - fix build error >> >> V2: >> - consider unevictable folios >> >> mm/compaction.c | 20 ++++++++++++++++++++ >> 1 file changed, 20 insertions(+) >> >> diff --git a/mm/compaction.c b/mm/compaction.c >> index 07bd227..a9f1261 100644 >> --- a/mm/compaction.c >> +++ b/mm/compaction.c >> @@ -2383,7 +2383,27 @@ static bool __compaction_suitable(struct zone *zone, int order, >> int highest_zoneidx, >> unsigned long wmark_target) >> { >> + pg_data_t __maybe_unused *pgdat = zone->zone_pgdat; >> + unsigned long sum, nr_pinned; >> unsigned long watermark; >> + >> + sum = node_page_state(pgdat, NR_INACTIVE_FILE) + >> + node_page_state(pgdat, NR_INACTIVE_ANON) + >> + node_page_state(pgdat, NR_ACTIVE_FILE) + >> + node_page_state(pgdat, NR_ACTIVE_ANON) + >> + node_page_state(pgdat, NR_UNEVICTABLE); >> + >> + nr_pinned = node_page_state(pgdat, NR_FOLL_PIN_ACQUIRED) - >> + node_page_state(pgdat, NR_FOLL_PIN_RELEASED); >> + > > Does the sum of all LRU pages equal non-CMA memory? > I'm quite confused for two reasons: > 1. CMA pages can be LRU pages. > 2. Free pages might not belong to any LRUs. NO. If all the pages in the LRU are pinned, it seems unnecessary to perform memory compaction, as the migration of pinned pages is unlikely to succeed. Besides checking whether the order-0 watermark is met, __compaction_suitable() also needs to determine whether there are sufficient migratable pages available for memory compaction. > > >> + /* >> + * Gup-pinned pages are non-migratable. After subtracting these pages, >> + * we need to check if the remaining pages are sufficient for memory >> + * compaction. >> + */ >> + if ((sum - nr_pinned) < (1 << order)) >> + return false; >> + >> /* >> * Watermarks for order-0 must be met for compaction to be able to >> * isolate free pages for migration targets. This means that the >> -- >> 2.7.4 >> >> > > Thanks > barry