From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from out30-124.freemail.mail.aliyun.com (out30-124.freemail.mail.aliyun.com [115.124.30.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D5A7F7E765 for ; Wed, 8 Jan 2025 02:50:46 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=115.124.30.124 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1736304650; cv=none; b=jbYtInAIDfpzMnPO6Idj0J82H4zAILIB719U6Ca3iJ3ZcGwUBvK9bJI3Ei/Ow1kQd7zDihQf5Lp8wexkGWTAefNFg9A2gkab8zhAGBpb+6qkwfbVdiKdQQ6heJLZmZXzQW+muQ8qW1o2XpU3fVq29Gt++F85SPbKjZ9URwzN8l0= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1736304650; c=relaxed/simple; bh=gV54JTwe0AbLbxG7MFIjhh3vtC/aM7+0KN1eFN4XOmE=; h=Message-ID:Date:MIME-Version:Subject:To:Cc:References:From: In-Reply-To:Content-Type; b=D5xLXgP/gplFK5ApJOlbnBxNsD+Jikzbp32dHRhTZqFEpmbXI2Sm9bGg5Jqqi8lnnxl1KtSVT3bD9dI1ltoCq0RbknGfHkB/oDjPQfIi4LA1CQXeyjs5H/RbJvlFLHbNWkmm6NfsdxsQX1WCcPM+TIHUw/6MwlfbaOpd+pMfjUI= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.alibaba.com; spf=pass smtp.mailfrom=linux.alibaba.com; dkim=pass (1024-bit key) header.d=linux.alibaba.com header.i=@linux.alibaba.com header.b=U88+Eh73; arc=none smtp.client-ip=115.124.30.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.alibaba.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.alibaba.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.alibaba.com header.i=@linux.alibaba.com header.b="U88+Eh73" DKIM-Signature:v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.alibaba.com; s=default; t=1736304644; h=Message-ID:Date:MIME-Version:Subject:To:From:Content-Type; bh=zFfSp6cBrgkyr25i0Ve6IFs3/rxGOE6Qs6SHi3jjjCI=; b=U88+Eh73fRM0k3JefiQk5ISy6C4eb227N2p5xBn4J9eb0PaY7pgOC7rOO/SR98MKZUhAIP3TJxsslvabNaqrpDcgq/y8lGzCzS19yMP7Oca+Ya4RbjwNi8OaWv/l0ZxVos8teklwVut9KuxmqBaD4nk3DC4yjsJV43Q/6qI2oPc= Received: from 30.74.144.127(mailfrom:baolin.wang@linux.alibaba.com fp:SMTPD_---0WNCOuUI_1736304642 cluster:ay36) by smtp.aliyun-inc.com; Wed, 08 Jan 2025 10:50:43 +0800 Message-ID: <180269be-f344-49e8-86da-23dda0bb31a0@linux.alibaba.com> Date: Wed, 8 Jan 2025 10:50:42 +0800 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [PATCH] mm: compaction: skip memory compaction when there are not enough migratable pages To: Ge Yang , akpm@linux-foundation.org Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, 21cnbao@gmail.com, david@redhat.com, hannes@cmpxchg.org, liuzixing@hygon.cn References: <1735981122-2085-1-git-send-email-yangge1116@126.com> <2889f0bf-b0ae-4f1a-b91c-fb4b59eb2d97@126.com> From: Baolin Wang In-Reply-To: <2889f0bf-b0ae-4f1a-b91c-fb4b59eb2d97@126.com> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit On 2025/1/6 16:49, Ge Yang wrote: > > > 在 2025/1/6 16:12, Baolin Wang 写道: >> >> >> On 2025/1/4 16:58, yangge1116@126.com wrote: >>> From: yangge >>> >>> There are 4 NUMA nodes on my machine, and each NUMA node has 32GB >>> of memory. I have configured 16GB of CMA memory on each NUMA node, >>> and starting a 32GB virtual machine with device passthrough is >>> extremely slow, taking almost an hour. >>> >>> During the start-up of the virtual machine, it will call >>> pin_user_pages_remote(..., FOLL_LONGTERM, ...) to allocate memory. >>> Long term GUP cannot allocate memory from CMA area, so a maximum of >>> 16 GB of no-CMA memory on a NUMA node can be used as virtual machine >>> memory. There is 16GB of free CMA memory on a NUMA node, which is >>> sufficient to pass the order-0 watermark check, causing the >>> __compaction_suitable() function to  consistently return true. >>> However, if there aren't enough migratable pages available, performing >>> memory compaction is also meaningless. Besides checking whether >>> the order-0 watermark is met, __compaction_suitable() also needs >>> to determine whether there are sufficient migratable pages available >>> for memory compaction. >>> >>> For costly allocations, because __compaction_suitable() always >>> returns true, __alloc_pages_slowpath() can't exit at the appropriate >>> place, resulting in excessively long virtual machine startup times. >>> Call trace: >>> __alloc_pages_slowpath >>>      if (compact_result == COMPACT_SKIPPED || >>>          compact_result == COMPACT_DEFERRED) >>>          goto nopage; // should exit __alloc_pages_slowpath() from here >>> >>> When the 16G of non-CMA memory on a single node is exhausted, we will >>> fallback to allocating memory on other nodes. In order to quickly >>> fallback to remote nodes, we should skip memory compaction when >>> migratable pages are insufficient. After this fix, it only takes a >>> few tens of seconds to start a 32GB virtual machine with device >>> passthrough functionality. >>> >>> Signed-off-by: yangge >>> --- >>>   mm/compaction.c | 19 +++++++++++++++++++ >>>   1 file changed, 19 insertions(+) >>> >>> diff --git a/mm/compaction.c b/mm/compaction.c >>> index 07bd227..1c469b3 100644 >>> --- a/mm/compaction.c >>> +++ b/mm/compaction.c >>> @@ -2383,7 +2383,26 @@ static bool __compaction_suitable(struct zone >>> *zone, int order, >>>                     int highest_zoneidx, >>>                     unsigned long wmark_target) >>>   { >>> +    pg_data_t *pgdat = zone->zone_pgdat; >>> +    unsigned long sum, nr_pinned; >>>       unsigned long watermark; >>> + >>> +    sum = node_page_state(pgdat, NR_INACTIVE_FILE) + >>> +        node_page_state(pgdat, NR_INACTIVE_ANON) + >>> +        node_page_state(pgdat, NR_ACTIVE_FILE) + >>> +        node_page_state(pgdat, NR_ACTIVE_ANON); >>> + >>> +    nr_pinned = node_page_state(pgdat, NR_FOLL_PIN_ACQUIRED) - >>> +        node_page_state(pgdat, NR_FOLL_PIN_RELEASED); >>> + >>> +    /* >>> +     * Gup-pinned pages are non-migratable. After subtracting these >>> pages, >>> +     * we need to check if the remaining pages are sufficient for >>> memory >>> +     * compaction. >>> +     */ >>> +    if ((sum - nr_pinned) < (1 << order)) >>> +        return false; >>> + >> >> IMO, using the node's statistics to determine whether the zone is >> suitable for compaction doesn't make sense. It is possible that even >> though the normal zone has long-term pinned pages, the movable zone >> can still be suitable for compaction. > If all the memory used on a node is pinned, then this memory cannot be > migrated anymore, and memory compaction operations would not succeed. > I haven't used movable zone before, can you explain why memory > compaction is still necessary? Thank you. Please consider unevictable folios that are not in the active/inactive file/anon LRU lists, yet can still be migrated.