From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-qk1-f181.google.com (mail-qk1-f181.google.com [209.85.222.181]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9FCE53101DF for ; Sun, 16 Nov 2025 18:12:24 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.222.181 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1763316746; cv=none; b=cYUkYfplvTo3/0Dopv5t/uIBOKu7jDxUvZkMfcrqhk4GOj+dmuc4oeOleUKcNLWlWfWn9b1NUoN4yT4jN5Mb6SXtZVrfXWhGr0mWFTKL7sGZGhK30fqXdt0dXFoms5u2vO1/QwWMGBlkHhdXTIDP4txgt6YWmOAUF+fRG6Vg20k= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1763316746; c=relaxed/simple; bh=SqrP6ZdA/QV2DBe1puUM0aReRKVf0TM2tVXx+rbKAZs=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=GktRqA7RcWE6YJJ563CpJjfUvUKh6AeKHBfXHyDXuF8mu6pjMpSDLgoi7pCckV55H7ju/hpmtTfolJXg2QbipWLeGrGzo5JshXzQ3Jcy8TNcfRaz+bxUuDHU/xcPGl2XuATBEN0k0HPSr43QdqtfrADZVrUqSldjUHzkxfKmrrk= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=Npe//Zsq; arc=none smtp.client-ip=209.85.222.181 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="Npe//Zsq" Received: by mail-qk1-f181.google.com with SMTP id af79cd13be357-8b21fa7b91bso339815385a.1 for ; Sun, 16 Nov 2025 10:12:24 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1763316743; x=1763921543; darn=vger.kernel.org; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :mime-version:subject:date:from:from:to:cc:subject:date:message-id :reply-to; bh=ZPlKpDo9gotjMa7idR5+EmYa4a/7gtSNuBeBwlWj2sc=; b=Npe//ZsqHnD8PqvE8ktVK5TlzLaD6f2uOy8GKddVAEXTvryc1MB5GvTu38VyimTedO trZiY3hND+a504S7wG/qAznFw8/qX419pCGOkby9hVirmRlTEBds84kzhYBpjRssFKiK Q9qwD8J5WMiLPeulKdyzPoT51wAilUVLb986HKJFxwmGIMqWJiV5xaYr16/pOU57WVp0 jYkW/jhxymKDJqpbnuUxZTbPkHQ6cL0GO19fh34NVQXnTMkZ6Udoi2vzW7KkrfGcYq8x BSq8iKil8Nrkcqmtk3o1xiesL6XVsa+n4XdtyFbwkLnfMP3kGvpJlQ0ZzMTJa6WL9jWL g5fA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1763316743; x=1763921543; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :mime-version:subject:date:from:x-gm-gg:x-gm-message-state:from:to :cc:subject:date:message-id:reply-to; bh=ZPlKpDo9gotjMa7idR5+EmYa4a/7gtSNuBeBwlWj2sc=; b=CKPCBNX8TEr7FN9njNp5/XDT37Tw2/YcWnu1q4HpAnsxIMtVXWMlLpxrit3dqJzbQW 65I32nBy/m1Jccda+vQ2l80c1AEUZxNkN+xMdjVRc6JwJjaRJXF42kaNGguNYVizTsBo /4WmZOD8nO3/2K+x6zbTHwyaDg98cq/b0uI9jZLDe9IOlZDr8AtS06kvt68kgFvd4Hye 0m15XNuvGJkaKQPG/DlGVaYjB9qhevWj0p0XuOL2o1TQzXcYWgf2GONLv0eynTs8AV8u P5QR4BK/+5QfGtLq5M4Ly8bbPoP9lNIcsVno0OmJrnib/7ZhfhPxm8F4DJKR+CA8GzFB j8KA== X-Forwarded-Encrypted: i=1; AJvYcCU0k9A7sm66+0Jb6E2TktKyiDVU2P99LHqHSb5mwyEwuouNxne4AH4DjLA+1NlxHZntu+4Qy5xbwvfuCdA=@vger.kernel.org X-Gm-Message-State: AOJu0YxiFvcvqtrtdNVncUXFNgyUr+UDMnEDO+b6HiL5EdbUEPM36vJR KPJ/Z0ecyB8G6Ktojl4gl5kV9lgNeQVqmPgiKlPntzKcq4rCq3cSfHvN X-Gm-Gg: ASbGncsOvUN6PEHqdRNhCUoqOVqwVpq2oiQZxn6u2aBlJDVra3TpHSw7GQ2Onwx21Q5 4bq6f5pqfltfTrQmLutqUqDfVAX6Shjw8JjECNcuhiLJmWaw6pcRZ1tOAg5Omz2iDzaGb0Oa213 hAGzit9lDeR5Ez2Xs7k4E4L9WJ1r6xF2yjU3oorSAKXIAjuXz6q60EK4+18V8ji8fIztG1g/VaM IeMm5ImtDwT0q3aySnmMhPRcCqzjypXNpjYrFKS1wV7ojemW3HTNFgRyiki7hDN/tUFbQt2tx3a scUgAZmVGSYFgQWRJxo267o93bYAc3QZR0XJY4H1px4Ct8gCWHKbk18tJnIWuI+PFX2hXg3LzNT C7gZCZU9lM6tHAP9FLcjYJWRzUgu+GfiZmd0AuMk/CWtPv03ZmBchGiC+EUjQpAj8Aog/u2fI5W yFzRJxkoEuOjWT8ckLKhQuDu8YDCy8Qs9i/Gne67v9dgyOyifa5cYvIzw5P/FMcys= X-Google-Smtp-Source: AGHT+IGkbTJU2oHCVXdPX8fPVA4HU5P92X5uQ8K3KICJSntE3qu2/JqofRMQqk1a6qpQAjGSkmVGEw== X-Received: by 2002:a05:620a:284b:b0:8b1:ac18:acce with SMTP id af79cd13be357-8b2c31597bamr1131656285a.28.1763316743439; Sun, 16 Nov 2025 10:12:23 -0800 (PST) Received: from [127.0.0.1] ([101.32.222.185]) by smtp.gmail.com with ESMTPSA id af79cd13be357-8b2e089b436sm305447785a.45.2025.11.16.10.12.17 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 16 Nov 2025 10:12:23 -0800 (PST) From: Kairui Song Date: Mon, 17 Nov 2025 02:11:45 +0800 Subject: [PATCH v2 04/19] mm, swap: always try to free swap cache for SWP_SYNCHRONOUS_IO devices Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 7bit Message-Id: <20251117-swap-table-p2-v2-4-37730e6ea6d5@tencent.com> References: <20251117-swap-table-p2-v2-0-37730e6ea6d5@tencent.com> In-Reply-To: <20251117-swap-table-p2-v2-0-37730e6ea6d5@tencent.com> To: linux-mm@kvack.org Cc: Andrew Morton , Baoquan He , Barry Song , Chris Li , Nhat Pham , Yosry Ahmed , David Hildenbrand , Johannes Weiner , Youngjun Park , Hugh Dickins , Baolin Wang , Ying Huang , Kemeng Shi , Lorenzo Stoakes , "Matthew Wilcox (Oracle)" , linux-kernel@vger.kernel.org, Kairui Song X-Mailer: b4 0.14.3 X-Developer-Signature: v=1; a=ed25519-sha256; t=1763316712; l=3125; i=kasong@tencent.com; s=kasong-sign-tencent; h=from:subject:message-id; bh=dC0xGmdnooLJYCyKsR/Xnrwvtqrz0t3aztFZKkG5Dw8=; b=pflnVKkaKIxmedAM1ePGW6UrPehZ5NknpGxCbuxr1z7BnS4l8okau0y9BWPAFi5zLB13h5p+a 6iZq12lUcfHDIhzKgQQsEbjORT7j6GAx47J65++9w/LtL/+uj0NoWf9 X-Developer-Key: i=kasong@tencent.com; a=ed25519; pk=kCdoBuwrYph+KrkJnrr7Sm1pwwhGDdZKcKrqiK8Y1mI= From: Kairui Song Now SWP_SYNCHRONOUS_IO devices are also using swap cache. One side effect is that a folio may stay in swap cache for a longer time due to lazy freeing (vm_swap_full()). This can help save some CPU / IO if folios are being swapped out very frequently right after swapin, hence improving the performance. But the long pinning of swap slots also increases the fragmentation rate of the swap device significantly, and currently, all in-tree SWP_SYNCHRONOUS_IO devices are RAM disks, so it also causes the backing memory to be pinned, increasing the memory pressure. So drop the swap cache immediately for SWP_SYNCHRONOUS_IO devices after swapin finishes. Swap cache has served its role as a synchronization layer to prevent any parallel swap-in from wasting CPU or memory allocation, and the redundant IO is not a major concern for SWP_SYNCHRONOUS_IO devices. Worth noting, without this patch, this series so far can provide a ~30% performance gain for certain workloads like MySQL or kernel compilation, but causes significant regression or OOM when under extreme global pressure. With this patch, we still have a nice performance gain for most workloads, and without introducing any observable regressions. This is a hint that further optimization can be done based on the new unified swapin with swap cache, but for now, just keep the behaviour consistent with before. Signed-off-by: Kairui Song --- mm/memory.c | 18 ++++++++++++++++-- 1 file changed, 16 insertions(+), 2 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index c95c36efc26a..806becd6bad5 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -4352,12 +4352,26 @@ static vm_fault_t remove_device_exclusive_entry(struct vm_fault *vmf) return 0; } -static inline bool should_try_to_free_swap(struct folio *folio, +/* + * Check if we should call folio_free_swap to free the swap cache. + * folio_free_swap only frees the swap cache to release the slot if swap + * count is zero, so we don't need to check the swap count here. + */ +static inline bool should_try_to_free_swap(struct swap_info_struct *si, + struct folio *folio, struct vm_area_struct *vma, unsigned int fault_flags) { if (!folio_test_swapcache(folio)) return false; + /* + * Always try to free swap cache for SWP_SYNCHRONOUS_IO devices. Swap + * cache can help save some IO or memory overhead, but these devices + * are fast, and meanwhile, swap cache pinning the slot deferring the + * release of metadata or fragmentation is a more critical issue. + */ + if (data_race(si->flags & SWP_SYNCHRONOUS_IO)) + return true; if (mem_cgroup_swap_full(folio) || (vma->vm_flags & VM_LOCKED) || folio_test_mlocked(folio)) return true; @@ -4928,7 +4942,7 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) * yet. */ swap_free_nr(entry, nr_pages); - if (should_try_to_free_swap(folio, vma, vmf->flags)) + if (should_try_to_free_swap(si, folio, vma, vmf->flags)) folio_free_swap(folio); add_mm_counter(vma->vm_mm, MM_ANONPAGES, nr_pages); -- 2.51.2