From mboxrd@z Thu Jan 1 00:00:00 1970 Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.remote-ip=95.215.58.85 ARC-Seal: i=1; d=subspace.kernel.org; s=arc20261015; a=rsa-sha256; cv=none; t=1794296724; b=G6VGl2l/Fougp2BN61DfnUWMIkdipEF4cHrUhrxx7sJjhaWWUlBqK9zwxFe5R1H25b94 dQsH4e1GFj4kTOUZB2mjzpBj4+Fv4aaczTEHD5X01LnIU9TAbVsq6tx5ivteuaAZi4tM9 ydOmI/J8rYM9S6bDR6rJiUAhWjXPs5/un4CtrrgIWkW38J1rz+kMfiZ5hLyWrZYcAOeGI BhppM309B7+Ri/aEVDkiT3fS5W8QaNBgkPBco7IoRUfI9TeeVNX6554Ris11U1mRYakNV sfHrJE/N4JrZpPTf37zk7+6cx/A8zIPhDVHwH+lZtssSoNWqQXkzqf356tkfEuH+Tzg== ARC-Message-Signature: i=1; d=subspace.kernel.org; s=arc20261015; a=rsa-sha256; c=relaxed/relaxed; t=1794296724; h=DMARC-Filter:X-Envelope-To:DKIM-Signature:X-Envelope-To:Received: X-Mizu-Trace-ID:X-Migadu-Flow:From:To:Cc:Subject:Date:Message-Id: X-Mailer:In-Reply-To:References:MIME-Version: Content-Transfer-Encoding; bh=bYjIYJm+n4nfDEAiZsWWktm7zzoyMcwgpbOm01w6spo=; b=e3RvsTwF47CywYVd/czDVm6/1w53T0x+jUlanU9OKb6RzIxZ+SGRq6iFTgenSUTQb2MV dEUHZqtZQO5QrzVJen6DgTYxx1HxMV4rD5zU+cFSGLYlpGxsGV7eKuTFptwtXBjPkKAyl Btd/II7RXLNCzL9gbbxdHrdGkysKTQNFNVbukajqOx50QurKeFlbUzWQ9o3SEMuFQ6rQC 3RAfKjatOWMz77DdtHS8YcKixD0xaxYSeeKXypsgJfsbf1IeFcURTnLUZMgqisjeRqo+v VB5C1SPkya+yTwrYr3TZKOqfEaOj6yo66ZIwNe9nwgD3GoMIGPSKerCMaDkZRblUqmQ== ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass header.from=linux.dev; spf=pass smtp.mailfrom=linux.dev; dkim=pass header.d=linux.dev header.i=@linux.dev header.a=rsa-sha256 header.s=key1 header.b=vlqCJjlJ; arc=none smtp.remote-ip=95.215.58.85 Received: from mta1.migadu.com (out-85.mta1.migadu.com [95.215.58.85]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature ECDSA (prime256v1) server-digest SHA256) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5DFC76075572 for ; Tue, 10 Nov 2026 07:45:23 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.2 smtp.subspace.kernel.org 5DFC76075572 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.dev Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key, unprotected) header.d=linux.dev header.i=@linux.dev header.a=rsa-sha256 header.s=key1 header.b=vlqCJjlJ X-Envelope-To: linux-kernel@vger.kernel.org DKIM-Signature: a=rsa-sha256; bh=eY4VzD6wuof/JGdxP9nVKDJYmUFOxcoqpj+Y4Rp/P/w=; c=simple/simple; d=linux.dev; h=from:to:subject:date:message-id:mime-version:content-type; s=key1; t=1794296721; v=1; x=1794901521; b=vlqCJjlJyhHO7JNbh8wajhkg5NO2nXDKM909lNOFUfkvgpW5VMjfwx25/Ij7r4IAoc3qN9jO pvKoFU0KTLNqAH5FS6p7MEbW8dboYwgKqMDNyHnEDT077oFgIzFKurSY4FboJMkAIQaAG2+6jjm Wzn+O01jdKSBSvqiJwnfISLo= X-Envelope-To: linux-kernel@vger.kernel.org Received: by mta10.migadu.com with ESMTPS id d048f83d76d25cf6; Tue, 10 Nov 2026 07:45:20 +0000 X-Mizu-Trace-ID: d048f83d76d25cf6 X-Migadu-Flow: FLOW_OUT From: "Xu Xin (vivo)" To: xu.xin@linux.dev Cc: akpm@linux-foundation.org, chengming.zhou@linux.dev, corbet@lwn.net, david@kernel.org, link@vivo.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-kselftest@vger.kernel.org, linux-mm@kvack.org, ljs@kernel.org, odys.yuan@gmail.com, rppt@kernel.org, surenb@google.com, tujinjiang@huawei.com, vbabka@kernel.org, xiao.wang@vivo.com Subject: [PATCH 2/4] ksm: skip rmap_item allocation for non-zero pages under only_merge_zero_pages Date: Tue, 10 Nov 2026 15:44:49 +0800 Message-Id: <20261110074451.220524-2-xu.xin@linux.dev> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20261110074451.220524-1-xu.xin@linux.dev> References: <20261110074312.219847-1-xu.xin@linux.dev> <20261110074451.220524-1-xu.xin@linux.dev> X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Post: List-Help: List-Subscribe: List-Unsubscribe: List-Owner: Precedence: list MIME-Version: 1.0 Content-Transfer-Encoding: 8bit When only_merge_zero_pages is enabled, ksmd only ever merges empty (zeroed) anonymous pages with the kernel zero page. A page that carries data can never be merged, yet the current code still allocates an rmap_item for it in scan_get_next_rmap_item() before cmp_and_merge_page() decides to skip it. On workloads with a large amount of non-mergeable anonymous memory this is a real cost: every scanned non-zero page pays for an rmap_item allocation, and the rmap_items accumulate in the per-mm pending list. For example, a MySQL instance with hundreds of gigabytes of anonymous memory (mostly buffer pool pages that cannot be merged) ends up with ksmd creating rmap_item metadata for each candidate page; the metadata overhead can wipe out any gain from the zero-page merging and even turn the net benefit negative. Add should_skip_non_zero_page(), called in scan_get_next_rmap_item() before get_next_rmap_item(), which, when only_merge_zero_pages is enabled, checks whether the candidate page is all zeros. A non-zero page is skipped entirely: no rmap_item is allocated for it, and any stale pending rmap_items below it are freed (as get_next_rmap_item() would). The page is reported back to ksm_do_scan() as ERR_PTR(-EAGAIN), which counts it as scanned and continues, so the pages_to_scan budget is preserved. The all-zero check (pages_identical() against the kernel zero page) reads the whole page. For a non-zero page this is a net win: the extra page read is cheaper than the rmap_item allocation and the stable/unstable tree operations it would otherwise trigger. A page that is zero now but is written before cmp_and_merge_page() runs is still handled correctly, because the checksum-based decision in cmp_and_merge_page() (added by the previous patch) is the authoritative one. Signed-off-by: Xu Xin (vivo) Co-developed-by: Jinjiang Tu Signed-off-by: Jinjiang Tu --- mm/ksm.c | 82 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 82 insertions(+) diff --git a/mm/ksm.c b/mm/ksm.c index 49f3728a3d03..e641c37ee096 100644 --- a/mm/ksm.c +++ b/mm/ksm.c @@ -2535,6 +2535,69 @@ static bool should_skip_rmap_item(struct folio *folio, return true; } +/* + * Determines whether a candidate page should be skipped entirely when + * only_merge_zero_pages is enabled, before an rmap_item is allocated for + * it. + * + * A page that is not all zeros carries data and can never be merged with + * the kernel zero page, so there is no point in allocating an rmap_item + * for it. Skipping it here avoids the rmap_item metadata cost that full + * KSM would otherwise pay for every non-mergeable page. + * + * @rmap_list: head of the per-mm pending rmap_item list + * @addr: address of the candidate page + * @folio: folio containing the candidate page + * @page: the candidate page + * + * Returns true if the page should be skipped (and any stale pending + * rmap_items below @addr are freed), false if it should be processed. + */ +static bool should_skip_non_zero_page(struct ksm_rmap_item **rmap_list, + unsigned long addr, struct folio *folio, + struct page *page) +{ + struct ksm_rmap_item *rmap_item; + + if (!ksm_only_merge_zero_pages) + return false; + + /* + * Pages already merged by KSM are handled by cmp_and_merge_page(); + * do not skip them here. + */ + if (folio_test_ksm(folio)) + return false; + + /* + * An all-zero page is exactly what we want to merge with the kernel + * zero page; process it. + */ + if (pages_identical(page, ZERO_PAGE(0))) + return false; + + /* + * The page carries data: free any pending rmap_items that are below + * it (they are stale, as get_next_rmap_item() would), and skip it so + * no rmap_item is allocated for it. + * + * Compare the masked address, unlike get_next_rmap_item() which + * compares the raw rmap_item->address (which may carry the + * UNSTABLE/STABLE_FLAG bits). That is intentional: a pending + * rmap_item at the same address as a non-zero page is stale and must + * be freed here, not returned for processing. + */ + while (*rmap_list) { + rmap_item = *rmap_list; + if ((rmap_item->address & PAGE_MASK) > addr) + break; + *rmap_list = rmap_item->rmap_list; + remove_rmap_item_from_tree(rmap_item); + free_rmap_item(rmap_item); + } + return true; +} + struct ksm_next_page_arg { struct folio *folio; struct page *page; @@ -2737,6 +2800,16 @@ static struct ksm_rmap_item *scan_get_next_rmap_item(struct page **page) if (tmp_page) { flush_anon_page(vma, tmp_page, ksm_scan.address); flush_dcache_page(tmp_page); + + if (should_skip_non_zero_page(ksm_scan.rmap_list, + ksm_scan.address, folio, + tmp_page)) { + folio_put(folio); + ksm_scan.address += PAGE_SIZE; + mmap_read_unlock(mm); + return ERR_PTR(-EAGAIN); + } + rmap_item = get_next_rmap_item(mm_slot, ksm_scan.rmap_list, ksm_scan.address); if (rmap_item) { @@ -2839,6 +2912,15 @@ static void ksm_do_scan(unsigned int scan_npages) rmap_item = scan_get_next_rmap_item(&page); if (!rmap_item) return; + if (rmap_item == ERR_PTR(-EAGAIN)) { + /* + * A non-zero page was skipped under + * only_merge_zero_pages: it was examined but not + * merged, so count it as scanned and keep going. + */ + ksm_pages_scanned++; + continue; + } cmp_and_merge_page(page, rmap_item); put_page(page); ksm_pages_scanned++; -- 2.34.1