From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-pf1-f176.google.com (mail-pf1-f176.google.com [209.85.210.176]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B14833B3BE2 for ; Mon, 27 Jul 2026 15:40:35 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.176 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1785166837; cv=none; b=fXec+ewnD10xQBnXhlQUBKXnKps7n0taWF02Vt2rtJm4XDREr7f0AXqA9Fj0GRNOCpCGnyciwXb/M0MRbslbnEKZm5EbpW/0qh5S7MuC+vXhNyGYV0719yGUBfioZR6H+XYePHiFPTXb6N1/x3mOVhEY8DOJ2MCFifXFz0Vk+F0= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1785166837; c=relaxed/simple; bh=7TbmGkj5XctrvRQfI98zd+79MAqRuuIUULa9r12HdqQ=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=QyKsAUMccPNxYGlvBZEBE6jxLh/gNtZa0N8yXTbspjZy5uQ6gUJAUyzimCrRWbBgo79vs5wdqkCcch8Po3upoYxUewnkN0NjXNL/8vI3ZYZbiWrnVCoOlLuXpPZ6oFZAHs1brmIdjKAwB9jJ6/DJV48L8eQJqHezGqKsV3g7Lb8= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=rkMo3iy2; arc=none smtp.client-ip=209.85.210.176 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="rkMo3iy2" Received: by mail-pf1-f176.google.com with SMTP id d2e1a72fcca58-84864086bfeso2757724b3a.1 for ; Mon, 27 Jul 2026 08:40:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1785166835; x=1785771635; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=oZNNFqmiKhCxMpepdTGRnKCzuc4JoIea+/gUgOwbSgQ=; b=rkMo3iy2CSGC/gBh/Wn6tw9ruFufrGgHTWvzCmSTurhhFyrhz0juFCW/1QR+5bhNEP CKDD4L3QVuzh0NKSnsAKHv8PA0PeKm2UAmcNIi2RScTVCSeDDje/K8R8LTMMbJAQKAAW JExwfc4gMefDi4HBSJ6e340oCjf52pgs4zfoc5E3M/Nl030FUyYnf5Rz3Y/3O9G94yIT mza5HmF3V+qDjq0qL8Y9d/rs+A3sYLk+wjfMvOxVX2LbIVbc2jLQnY/xZXOPu0aOVVYO AAXUnIDDAvhyEkL7ysfANcCf7kawbr7Cjv446jMMmUbn5jBSdel3HkC6Ga8cH60ybukk tOVQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1785166835; x=1785771635; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=oZNNFqmiKhCxMpepdTGRnKCzuc4JoIea+/gUgOwbSgQ=; b=JysMrUevuOd2Dcr7/BAG13IooQbABh957xWnCsb9P2IEqF1qVFlYP3jnfGvMWcq7Z4 RsDZRNvfDJ8nDkYUNiJ7geaM6nnfd3zbRXhynSHzpMSJI+A+QYnfgLdzXCUlChtq81wO OfoASg2Jv+GFnjuMGsJB9jjSOwR6yBZDIg2XNCHhieSM02XwGWVLDPlC9TxgmD+8X8ks iiVq1A7UPxpKirp4QcDxMGtpx/TBBNsvcZMyIdobNZcJStzArC0UuHW1hjcAFkpWY9+q 5NpDkjh9WH8ZNd6iAIrRDcAcA64pNSE8FLPvkbWZjvP3OZbjVGWs409xc81eWt0C+Sa9 KTog== X-Forwarded-Encrypted: i=1; AHgh+RpTNFRp20tlTqUg6UFKT2Gnxf6YDlTck7xYazvb6uIKUkqw2tDd67pBMNKkyZ4tADD2Es/9bJ78dU72IfI=@vger.kernel.org X-Gm-Message-State: AOJu0Yw7aYAp3C8ll0r55Zk6TPdIhGO2OhPH7l9RFNSzavdLlfhj9XSO cFegvDY09K+++H4jt0C+3fk3oCkIdXhUINm1LSzqEASoVbPMv0PfksSP X-Gm-Gg: AR+sD11ip5Y27KZlX7M+YZBVIAs7l3yHjaQVAHRayp4lrAgXBszfiJvcc6ral734qLt hZCLbxAFsh0uWIhM8L01qFue8LKsanltBnmv2S8QxPfEWIrjiJetrS7Gwv1DTuj/Hry35yQU04n jXhNfJWXhs8Fr7fYnuJzIswMrHJAaEqCt0x2V4IGaqD1WVX6AftNaAUXReOP9jK3PGCl1jJ/6ll 8b1w+c8n/pk3b20wQvuD1AzjTQjVxESpFbaqr0YZqIvsRYp4EQxm53DwXXq7balGg///f2CJxWZ ZLKwqqqxkiGPpbjRFNe+LULOatl8Apj1+EbAe30pF17bUJPzX+hjIPP2LsRCGM3wKAN5iykuszP GGwlUTQ4yEubblO9MHSvfE+Z2Yemr4e3dExYx9wIv/jZleWWOiFzDvwihi/yI8OFGT8FPhEJI0y ny6rOP9uyvW5A/9CBD4vnc X-Received: by 2002:a05:6a00:3393:b0:845:40d1:87fa with SMTP id d2e1a72fcca58-84e8cf89b3amr16485b3a.25.1785166834932; Mon, 27 Jul 2026 08:40:34 -0700 (PDT) Received: from spr1.ipads-lab.se.sjtu.edu.cn ([202.120.40.82]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-84e5327f3fbsm3332996b3a.19.2026.07.27.08.40.30 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 27 Jul 2026 08:40:34 -0700 (PDT) From: Hengbin Zhang To: Andrew Morton , David Hildenbrand Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, stable@vger.kernel.org, Hengbin Zhang , Lorenzo Stoakes , Zi Yan , Baolin Wang , "Liam R. Howlett" , Nico Pache , Ryan Roberts , Dev Jain , Barry Song , Lance Yang , Usama Arif Subject: [RFC PATCH v3] mm/thp: serialize huge-zero folio state transitions Date: Mon, 27 Jul 2026 15:40:01 +0000 Message-Id: <20260727154001.4102341-1-uqbarz@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <4146d4f5-c8fb-42cf-9dc8-a9d4c26055da@kernel.org> References: <4146d4f5-c8fb-42cf-9dc8-a9d4c26055da@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit The nonpersistent huge-zero shrinker clears huge_zero_folio and then invalidates huge_zero_pfn. A concurrent fault can publish a replacement folio and PFN between those updates, after which the old shrinker invalidates the replacement generation's PFN identity. A later partial mprotect() can then misidentify the live special PMD and enter the ordinary anonymous THP split path. A writer-side spinlock keeps huge_zero_folio and huge_zero_pfn updates from different generations together. Lockless getters still use the refcount fast path. atomic_set_release() publishes an initialized new generation before a successful atomic_inc_not_zero() can admit a getter on weakly ordered architectures. Suggested-by: David Hildenbrand Link: https://lore.kernel.org/r/20260724100509.2300200-1-uqbarz@gmail.com Signed-off-by: Hengbin Zhang --- mm/huge_memory.c | 22 +++++++++++++++------- 1 file changed, 15 insertions(+), 7 deletions(-) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index b5d1e9d4463d..79bec4495540 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -78,6 +78,7 @@ static unsigned long deferred_split_scan(struct shrinker *shrink, static bool split_underused_thp = true; static atomic_t huge_zero_refcount; +static DEFINE_SPINLOCK(huge_zero_lock); struct folio *huge_zero_folio __read_mostly; unsigned long huge_zero_pfn __read_mostly = ~0UL; unsigned long huge_anon_orders_always __read_mostly; @@ -237,17 +238,18 @@ static bool get_huge_zero_folio(void) } /* Ensure zero folio won't have large_rmappable flag set. */ folio_clear_large_rmappable(zero_folio); - preempt_disable(); - if (cmpxchg(&huge_zero_folio, NULL, zero_folio)) { - preempt_enable(); + spin_lock(&huge_zero_lock); + if (READ_ONCE(huge_zero_folio)) { + spin_unlock(&huge_zero_lock); folio_put(zero_folio); goto retry; } WRITE_ONCE(huge_zero_pfn, folio_pfn(zero_folio)); + WRITE_ONCE(huge_zero_folio, zero_folio); + spin_unlock(&huge_zero_lock); - /* We take additional reference here. It will be put back by shrinker */ - atomic_set(&huge_zero_refcount, 2); - preempt_enable(); + /* Publish the identity before admitting lockless getters. */ + atomic_set_release(&huge_zero_refcount, 2); count_vm_event(THP_ZERO_PAGE_ALLOC); return true; } @@ -298,9 +300,15 @@ static unsigned long shrink_huge_zero_folio_scan(struct shrinker *shrink, struct shrink_control *sc) { if (atomic_cmpxchg(&huge_zero_refcount, 1, 0) == 1) { - struct folio *zero_folio = xchg(&huge_zero_folio, NULL); + struct folio *zero_folio; + + spin_lock(&huge_zero_lock); + zero_folio = READ_ONCE(huge_zero_folio); BUG_ON(zero_folio == NULL); WRITE_ONCE(huge_zero_pfn, ~0UL); + WRITE_ONCE(huge_zero_folio, NULL); + spin_unlock(&huge_zero_lock); + folio_put(zero_folio); return HPAGE_PMD_NR; } -- 2.34.1