From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-qk1-f178.google.com (mail-qk1-f178.google.com [209.85.222.178]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1CA8724DCF6 for ; Thu, 13 Aug 2026 02:21:30 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.222.178 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1786587693; cv=none; b=U13cq74EuJrTEj5ur5AlVr2GB6sNO+V8/XKZEi14TusJ6WZOn0J9X6GeRkU5EqALPUieN3FIs1cX6AflWxvI04VZxIt//lCVsqCIT3nbrjIETmbQqHmkluxr0cof3ae/6WpJk1oKyCs0LxLIwJtCaXKO/7HapUGmeZ1Rqqt8PKI= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1786587693; c=relaxed/simple; bh=sM8g8CQTZ/S7evFo640TGiMrGK09TXpM2NF2AJgRwmQ=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=E6AarfyfkCvn1Q7Mvq1Qj5nZJh34Rw6xf2Kve/LNdV9i+jVH37SQJdcIGy/B/zIEvJdl6ssyq5YUEFLPmvwPkd/qdDJLTWszwfTq1f60xGr3AtORgRc55OVwhOJNwi46SrKbLEmD/T4oplYeAibaxBR6j3oJ9IInAhTpfzAy1n4= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=gourry.net; spf=pass smtp.mailfrom=gourry.net; dkim=pass (2048-bit key) header.d=gourry.net header.i=@gourry.net header.b=uV+Z7Hfd; arc=none smtp.client-ip=209.85.222.178 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=gourry.net Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gourry.net Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gourry.net header.i=@gourry.net header.b="uV+Z7Hfd" Received: by mail-qk1-f178.google.com with SMTP id af79cd13be357-930f618435cso8760585a.3 for ; Wed, 12 Aug 2026 19:21:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gourry.net; s=google; t=1786587690; x=1787192490; darn=vger.kernel.org; h=in-reply-to:content-disposition:content-type:mime-version :references:message-id:subject:cc:to:from:date:from:to:cc:subject :date:message-id:reply-to:content-type; bh=Lmq76BWyH9k4izhQxstGYjoZH3LaJ0p93Xt9P+A1osw=; b=uV+Z7HfdcwsBnbpiAfCVQ0CtiazZv9fO1vb0pIWkxn/cKn/uL4+5UGT6MZMOAM4G5R AJJ5p4U6pkKZB3ZUR+DGE29GK60isjZUap2uV1GX60/zepBIBBpB1V9Ixz9OHcvrlkq0 jVDEXX40MOIjaa1sNXPBo57Yrl9WKBOYEIaqxXKwb/Xbh6U1VSF/rnfMtKYjRTiuwDCr 7PQ7nk/wtSKunTewLBajsOi/Obvbsw6tu/Xq8fyl8GzUnobIwr3+a9w3L+QI06USLuHr Sok6qLE5N0Xwk8l2rDsE74rLU4nPInZ/1KYAXbXpgdnKHgG5AQ9KOaLDEMppd/98xb+V +bXA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1786587690; x=1787192490; h=in-reply-to:content-disposition:content-type:mime-version :references:message-id:subject:cc:to:from:date:x-gm-gg :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to :content-type; bh=Lmq76BWyH9k4izhQxstGYjoZH3LaJ0p93Xt9P+A1osw=; b=VCdzWsfxYSDLCjiDl3hx0tG9qOdtCHGHd9h6VRbSHcNvireBeZlvyEceD44c09u+3U BwsVvWyQntflozXO15k69Ei5ZV5gJH6aQFv8jZHi4em3q1qG+ocmeqKJO3bm01F13kzC mxEpcftnhngbiqFlMNRPZdnYaPwT+gmGC3fUZ2uDA1nblVVuGYJ6WbSOMAY4nPSLPx0/ WfgOW43C7kT5/AVBTV9HtJxd6bdPIod30NRMVk1JylZtlNywWtquEje6rmaHBBDnt0jk wCwXxBDZzlSaArhzeQ0zBJEQ19rcfXEa2JQtZgjqxY92AZXiaUM6XDS/FbboBnqcFKuE ocZQ== X-Forwarded-Encrypted: i=1; AHgh+Rr/XkvJ8xytqBezcXVm4UqPkGzj3iPyWdkE9jr4TGr9K7KnIoa6J1aUprgZnkVwJ97OcFCqLM8S4MpM8cc=@vger.kernel.org X-Gm-Message-State: AOJu0YwM8Fhr0GGHGP2N9emNSUXgursWuUcQXBU4j22sT+jl2Ni2H/Wt sfOo5Xf02almVrXM7AAnkXB1WjJRlONCAMjyYcYGOgIdWd8U/QLfp6z8wyyp5a0kyQUW3/rRNte TwxY5DEA= X-Gm-Gg: AR+sD11F/5+3i5NZOt5g51f4xIVWojyjDtQG86S9JJF1leFTava3+5V2c9/QVZMugnt jrD1J+osex0RYPPzndPh28L2wA3yh8+bxbq3pSQTQE/MXviXES6OggJOFMgdWvvtcJr/3zlxAjV Frc6V2dsdvSEGx1tjZ6bzLDEwBHS0LfZ8VQ8qfmUStP+MyMIK6BHu76fZ6zrOxIiMNe+FSyjczi DdRhP4tDyg2DiOzlcW6+Zs9HmmQrHvAivVbZ5qnlZafZcwMtn/35G7MlvEeyj27eTPk183xYDWa 0yoXCwjUxXEJWcWiJZztP1TAerzLJ/iJIbHLToSgc6vwu1yqvKIKmId7H77+bLrJilGVEmR5mDU mHuSOPouBqbJn5bBdVps6viFCi4pYsBz4r++w/n0tJGkakK/8amA42nL/b+RpRskVGz72JYDDUU hzR6z8seGfOu8TivA6F2SX5MZ4IqLE/u6Idfk= X-Received: by 2002:a05:620a:2059:b0:92e:e3eb:de54 with SMTP id af79cd13be357-936bf90138bmr201611585a.11.1786587689928; Wed, 12 Aug 2026 19:21:29 -0700 (PDT) Received: from fedora ([2601:19e:8500:d3c0::28fe]) by smtp.gmail.com with ESMTPSA id af79cd13be357-936c1b1234fsm45066485a.27.2026.08.12.19.21.27 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 12 Aug 2026 19:21:29 -0700 (PDT) Date: Wed, 12 Aug 2026 22:21:26 -0400 From: Gregory Price To: Matthew Wilcox Cc: Yongting Lin , Jonathan.Cameron@huawei.com, akpm@linux-foundation.org, alok.rathore@samsung.com, balbirs@nvidia.com, bharata@amd.com, byungchul@sk.com, dave.hansen@intel.com, dave@stgolabs.net, david@kernel.org, donettom@linux.ibm.com, joshua.hahnjy@gmail.com, kinseyho@google.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org, mgorman@techsingularity.net, mingo@redhat.com, nifan.cxl@gmail.com, peterz@infradead.org, raghavendra.kt@amd.com, riel@surriel.com, rientjes@google.com, shivankg@amd.com, sj@kernel.org, weixugc@google.com, xuezhengchu@huawei.com, yiannis@zptcorp.com, ying.huang@linux.alibaba.com, yuanchu@google.com, ziy@nvidia.com Subject: Re: [PATCH v8 0/8] mm: Hot page tracking and promotion infrastructure Message-ID: References: <20260810033814.83951-1-linyongting@bytedance.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: On Mon, Aug 10, 2026 at 05:16:39AM +0100, Matthew Wilcox wrote: > On Mon, Aug 10, 2026 at 11:38:14AM +0800, Yongting Lin wrote: > > On Tue, Jul 28, 2026 at 07:24:46PM +0100, Matthew Wilcox wrote: > > > > From our perspective, this is not merely a hypothetical use case. We > > are actively developing and evaluating CXL tiered-memory systems, and > > expect promoting CXL-resident pages that become hot back to DRAM to be > > a practical requirement. > > But what *is* your use case? Truly hot data ends up in L1/L2/L3 no > matter whether it came across the ridiculously high latency CXL link or > from regular DRAM. So we're talking about "warm" data that's presumably > measured in gigabytes (since current server CPUs have about half a > gigabyte of LLC) > Speaking for myself (not Yongting Lin) - our use case is pretty straight forward and deployed at scale today, with plans to expand on it in future generations. We have systems with 1TB+ of memory, with a 768GB/256GB DRAM/CXL split. We're expecting these numbers to grow, along with density of work. On these systems we use both scaled and stacked workloads (single and multi-container) where we do have a decent amount of warm data landing on CXL. In fact, we're looking to make that a more explicit scenario to reduce per-workload variance (higher floors, lower ceilings). See Joshua's Tiered Memcg Limits work [0] [0] https://lore.kernel.org/all/20260807202059.2620949-1-joshua.hahnjy@gmail.com/ Right now this works when mixing workloads which largely use mapped memory, but for workloads using unmapped pagecache, they end up permanently demoted to CXL because NUMA Balancing can't promote page cache (but reclaim can demote it, and the page allocator can fallback to CXL). PGHot at least gives us the *full* solution that NUMA balancing does not. > I see the basis for saying "this task is low priority, it gets its memory > allocated on CXL" and "this task is high priority, it gets its memory > allocated on DRAM". I don't see the use case where we're trying to figure > out that region A of this task is sufficiently warmer than region B, and so > we want to demote region B back to CXL and promote region A back to DRAM. > We find the vast, vast majority of "low priority" work ends up pushed out to swap / fully invalidated anyway, and not even taking up much of any memory in the first place. This work is not the problem - it's usually things like system updates or monitors etc. Actual user work wanting X-GB of memory is almost never willing to take the latency hit of 100% CXL - and even if you did that, you actually cause bandwidth issues for the rest of the system due to limited bandwidth compared to local DRAM. Something to remember: Demotion does not respect cpusets or mempolicy, and page cache is owned by the first-toucher, so cpuset limitations break for page cache in general. (This is one of many reasons I've been exploring the private nodes series). === For actual tangible problems: We've found fallback allocations landing on CXL is preferable to forcing reclaim to run and demote to CXL before a new DRAM allocation can be made. e.g. zone_reclaim_mode=0x0 is better than 0xf. ZRM induced stalls are just always worse than eating a CXL page and moving it later. This directly results in hot memory (in fact many gbs of it) on CXL. This is especially prevelent when workloads spinup after the top tier is already / near full. The tier-aware memcg stuff should help with this, but it still requires both demotion and promotion to level out the usage between tiers to an acceptable ratio. I'm not looking for a system that can predict the future and maximize performance. At best I'd like a system that nudges placement in the right direction over a long period for better workload consistency. > I particularly don't see a case for trying to do it on single-page > granularity. There's just too much damn information to track. Maybe at > a 2MB or 1GB boundary, but not per page. I largely agree with you here. If you look at the other work my colleagues have been doing - it's been heavily focused on making THPs more reliable and even pushing towards 1GB THPs. We're getting to a point where 4kb just doesn't make sense anymore for many things. ~Gregory