From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from m16.mail.126.com (m16.mail.126.com [117.135.210.6]) by smtp.subspace.kernel.org (Postfix) with ESMTP id A776F1A0BE0 for ; Wed, 8 Jan 2025 08:37:35 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=117.135.210.6 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1736325463; cv=none; b=Kfn7nDTKhweldFk47jiSXscr+gG5eZnbvWv0rnARihnVNRtdPPzfMOwspA+73ZmQWI+rMXg+HUaRC0gubbVYU9+XH2lbmnPNXkvyU988DPGgdU6cP9LGe7vhDQDP4VGNpMbFeMXWrV0niBVEUj3C4LPZQhVYl3cDB3xiD6J8cyQ= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1736325463; c=relaxed/simple; bh=5UP9jZojZYxLWQX4U0G6M7nEh33IOK1Blj3MnoqY1eQ=; h=Message-ID:Date:MIME-Version:Subject:To:Cc:References:From: In-Reply-To:Content-Type; b=uwmbmKhnDTdmdYErla3SIuDxAiI6Ll8qPv7vSXk+4rBE1KNRn9rj2LdLGpt2lmo22XG3VqyDvQ8woUk9/GnjGs7v7v4uUNUe7s23mp71xoupio4337eZl7DnFzbQuVY2BA8wjG9WTts/lOTs1QCgm9zFnNHcIN07KC3eb/XabF0= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=126.com; spf=pass smtp.mailfrom=126.com; dkim=pass (1024-bit key) header.d=126.com header.i=@126.com header.b=Wb9gxMij; arc=none smtp.client-ip=117.135.210.6 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=126.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=126.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=126.com header.i=@126.com header.b="Wb9gxMij" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=126.com; s=s110527; h=Message-ID:Date:MIME-Version:Subject:From: Content-Type; bh=l2/5Dt1bINDIBXXfQ6MMwdwckdqbsFeMj4hTlS6F0Dk=; b=Wb9gxMijQIdAh6NbGqb2V2eCYuU464B64J7L4H2IRdSSdqEK7r9eLY7nG5oqzC 9OZRLJ8zeCKFum8K2vFd9bBpQ6F6VUz+qg56+rVcB2pWwstoxgtWsVm58Tt5+4xt lj8LqB/r0TF7G2aFCjMLIFddNOrFakWeLsJADDk4YzJHc= Received: from [172.19.20.199] (unknown []) by gzga-smtp-mtada-g1-3 (Coremail) with SMTP id _____wD3J1DPOH5nYlyYAw--.51883S2; Wed, 08 Jan 2025 16:35:27 +0800 (CST) Message-ID: <32d4b46a-cf7c-4fd2-a86f-7e82b9b1a5e3@126.com> Date: Wed, 8 Jan 2025 16:35:27 +0800 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [PATCH] mm: compaction: skip memory compaction when there are not enough migratable pages To: Baolin Wang , akpm@linux-foundation.org Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, 21cnbao@gmail.com, david@redhat.com, hannes@cmpxchg.org, liuzixing@hygon.cn References: <1735981122-2085-1-git-send-email-yangge1116@126.com> <2889f0bf-b0ae-4f1a-b91c-fb4b59eb2d97@126.com> <180269be-f344-49e8-86da-23dda0bb31a0@linux.alibaba.com> From: Ge Yang In-Reply-To: <180269be-f344-49e8-86da-23dda0bb31a0@linux.alibaba.com> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-CM-TRANSID:_____wD3J1DPOH5nYlyYAw--.51883S2 X-Coremail-Antispam: 1Uf129KBjvJXoWxCFy8Ww15Aw48CF1fZF4DXFb_yoWrWrW3pr y8G3ZxKr4DJr9Fkr1Iq3Z0vFnxtw4fGFWUXr9Fyr97uasI9F1Syr47t34UC3W8Zr1jqF4j vr1Du3sxuan8Za7anT9S1TB71UUUUU7qnTZGkaVYY2UrUUUUjbIjqfuFe4nvWSU5nxnvy2 9KBjDUYxBIdaVFxhVjvjDU0xZFpf9x07jYGQhUUUUU= X-CM-SenderInfo: 51dqwwjhrrila6rslhhfrp/1tbiOg3OG2d+M7lRZgAAsG 在 2025/1/8 10:50, Baolin Wang 写道: > > > On 2025/1/6 16:49, Ge Yang wrote: >> >> >> 在 2025/1/6 16:12, Baolin Wang 写道: >>> >>> >>> On 2025/1/4 16:58, yangge1116@126.com wrote: >>>> From: yangge >>>> >>>> There are 4 NUMA nodes on my machine, and each NUMA node has 32GB >>>> of memory. I have configured 16GB of CMA memory on each NUMA node, >>>> and starting a 32GB virtual machine with device passthrough is >>>> extremely slow, taking almost an hour. >>>> >>>> During the start-up of the virtual machine, it will call >>>> pin_user_pages_remote(..., FOLL_LONGTERM, ...) to allocate memory. >>>> Long term GUP cannot allocate memory from CMA area, so a maximum of >>>> 16 GB of no-CMA memory on a NUMA node can be used as virtual machine >>>> memory. There is 16GB of free CMA memory on a NUMA node, which is >>>> sufficient to pass the order-0 watermark check, causing the >>>> __compaction_suitable() function to  consistently return true. >>>> However, if there aren't enough migratable pages available, performing >>>> memory compaction is also meaningless. Besides checking whether >>>> the order-0 watermark is met, __compaction_suitable() also needs >>>> to determine whether there are sufficient migratable pages available >>>> for memory compaction. >>>> >>>> For costly allocations, because __compaction_suitable() always >>>> returns true, __alloc_pages_slowpath() can't exit at the appropriate >>>> place, resulting in excessively long virtual machine startup times. >>>> Call trace: >>>> __alloc_pages_slowpath >>>>      if (compact_result == COMPACT_SKIPPED || >>>>          compact_result == COMPACT_DEFERRED) >>>>          goto nopage; // should exit __alloc_pages_slowpath() from here >>>> >>>> When the 16G of non-CMA memory on a single node is exhausted, we will >>>> fallback to allocating memory on other nodes. In order to quickly >>>> fallback to remote nodes, we should skip memory compaction when >>>> migratable pages are insufficient. After this fix, it only takes a >>>> few tens of seconds to start a 32GB virtual machine with device >>>> passthrough functionality. >>>> >>>> Signed-off-by: yangge >>>> --- >>>>   mm/compaction.c | 19 +++++++++++++++++++ >>>>   1 file changed, 19 insertions(+) >>>> >>>> diff --git a/mm/compaction.c b/mm/compaction.c >>>> index 07bd227..1c469b3 100644 >>>> --- a/mm/compaction.c >>>> +++ b/mm/compaction.c >>>> @@ -2383,7 +2383,26 @@ static bool __compaction_suitable(struct zone >>>> *zone, int order, >>>>                     int highest_zoneidx, >>>>                     unsigned long wmark_target) >>>>   { >>>> +    pg_data_t *pgdat = zone->zone_pgdat; >>>> +    unsigned long sum, nr_pinned; >>>>       unsigned long watermark; >>>> + >>>> +    sum = node_page_state(pgdat, NR_INACTIVE_FILE) + >>>> +        node_page_state(pgdat, NR_INACTIVE_ANON) + >>>> +        node_page_state(pgdat, NR_ACTIVE_FILE) + >>>> +        node_page_state(pgdat, NR_ACTIVE_ANON); >>>> + >>>> +    nr_pinned = node_page_state(pgdat, NR_FOLL_PIN_ACQUIRED) - >>>> +        node_page_state(pgdat, NR_FOLL_PIN_RELEASED); >>>> + >>>> +    /* >>>> +     * Gup-pinned pages are non-migratable. After subtracting these >>>> pages, >>>> +     * we need to check if the remaining pages are sufficient for >>>> memory >>>> +     * compaction. >>>> +     */ >>>> +    if ((sum - nr_pinned) < (1 << order)) >>>> +        return false; >>>> + >>> >>> IMO, using the node's statistics to determine whether the zone is >>> suitable for compaction doesn't make sense. It is possible that even >>> though the normal zone has long-term pinned pages, the movable zone >>> can still be suitable for compaction. >> If all the memory used on a node is pinned, then this memory cannot be >> migrated anymore, and memory compaction operations would not succeed. >> I haven't used movable zone before, can you explain why memory >> compaction is still necessary? Thank you. > > Please consider unevictable folios that are not in the active/inactive > file/anon LRU lists, yet can still be migrated. Ok, thanks.