From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-ot1-f49.google.com (mail-ot1-f49.google.com [209.85.210.49]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B23703EB81A for ; Mon, 6 Jul 2026 20:45:09 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.49 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1783370711; cv=none; b=B76QUslSHmp/UfM90HQI5FzY7MJ5W3CsNm+S7I299UoUohkk7t2YvbviDRXNEQmtMI9MBO1fZXiJMH2oE1SWKJuOxLh8qGTJcQrKer8mIHGWwPuxWmJmwy74EfC74qyiNesfc+jJlFXlw13bVL590muEU+fv2v5qgNSoZy+CJfs= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1783370711; c=relaxed/simple; bh=GuAFEjNWdWhY/bbVGLeHhU6fWp0hZegv1XJn6Mx54RY=; h=From:To:Cc:Subject:Date:Message-ID:MIME-Version; b=F18aPoI5y0Qp7SjqCyAWjnYNtftNq2+B86RLOqc9Ln/TzB0MZaftG7Ubm6U0BeiyQks3MAZd7riJJX5jts01IqiWgiW5F+fDmZuHE/K2xheAMZRT4oNnEnaur6LzbkAddbMDVFvDFus8d2Pgb7OzRk89xryB/Lc18RgK0kTWBRc= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=F+PrQpSd; arc=none smtp.client-ip=209.85.210.49 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="F+PrQpSd" Received: by mail-ot1-f49.google.com with SMTP id 46e09a7af769-7e9ecd7216cso1480421a34.3 for ; Mon, 06 Jul 2026 13:45:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1783370708; x=1783975508; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date:message-id:reply-to:content-type; bh=XVLOymukG8mEHjr57PGE+70BcqfkinJsntMYx+ElCgI=; b=F+PrQpSdRZLtEA14ojlmktcMQU0VP3JDMog13Kr+kHBkNuKckXsixzcNLv5SdvI9Cs 7t3Zx3d0ykSeAdm/zbYWYqsEEdmGkwFsGQ/c4o6h2eObhhPPfpncIdrgBSLU2qLQybbV GqhMeVskTHLoVwaj2qvr0ZvwQ1hRfVzPTv+4sZ0+X2OYErGzIYbzqMhoVCvociCiE5xR /t9tGp+ht/BXdJHSqhXAzQJ8rIJ1YQUn0xN0sqD5NenEVOnLFlKpgGk0DTv+Hn6swTFx cWBp6pOhifjA4MpR0Vfty4GEkrOAn+9M23DuW0CF9hH3JE5lLFsj6J6w7dqXVKcsggo1 drUQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1783370708; x=1783975508; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-gg:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to:content-type; bh=XVLOymukG8mEHjr57PGE+70BcqfkinJsntMYx+ElCgI=; b=KKgxymY4nUtsNr8QxIhDCEH8wT02b5LaTRLx1j8dkeH4OhPlEwH47SOTyIyol945EN 4AQ+DTDludX8btecP7cJD7gzEGNYo2Lx4HB6dJwcJCvFlCU8SL0tLnsWDCMdYGDphWXk ONjajX7TCKWu0J5sb08dbxi3Mq1k6BKn05GF0UoixR4ZrD27x61z2pHGXh7JHcvVsQgw FPTIZX0ggMSoGB9uF+5vvKWUrXZBCifk7KaopAvhq7yco4X2OpbDHUjRgQBHAXPF4NWW HbPAskBAxDgN/OLc2PjHeG4NQhDH7i37j/LjvG7pd05qBvjmgyCjFe5lkVpVzqZEDBCU JEBg== X-Forwarded-Encrypted: i=1; AFNElJ/H8XXX0UdGiGqxtxx3zJTokDflksxjaaky5da0CQrzE+eiQypDAGUySt2cyo7L+1KE9WyNqbr3Jd231NY=@vger.kernel.org X-Gm-Message-State: AOJu0YxE7aNqThuwon9x3nUm3pIAQor9Y03XF2iItnEyofyBl9gvsYWu d4zTzvUPNT4n6hG07bmtLz4B9+54x0lYUWW+f4RaubZ6bax0OyzKDz/h X-Gm-Gg: AfdE7clfaJydjRLgLlJuOcwoeYUjSk75oHHj84Z645kQ6WCYf1Nc2/pAeaKmaUrNpmB z4VROsUCXxrkFIyUZUO24F9VRlS487m8Eo3mRVoqRFFw/F32QFuDDdTg20WfRbt1KF6aOh3ZkYw b52b4ORJmU27gGqpLQMj+U069VnMssmVwGG0vrIHmfOo1JaAKj40cgxVU/x/DRz+NbogewCkvSL 0V2JQX++7gcl3L1Zfk4Hv2PXKYDpDDzHd1JjAfr36Kpz5BxhCC+76EvXQAGb81kFYzrdoENLisD UXrZW3XVFn/iY4QKFgby9hxpz5K9Gnj2rsAnNnVbESL6nxQxSFkKHc9z3CavyfnHiKpg7VB7owQ l45RfRziE+65xQlcs5n2bTlO7Bd541/5F9RFfDaBxQeC0hrY5ab0Jb6EvZk+sllsjXfXfLg8O9q rQfqr8hXjAHdwpZq2slaSLmMBBw1g5F6/Nsu4Hx3daKas= X-Received: by 2002:a05:6830:7307:b0:7d7:e142:2ead with SMTP id 46e09a7af769-7ebb232166bmr1568070a34.20.1783370708494; Mon, 06 Jul 2026 13:45:08 -0700 (PDT) Received: from localhost ([2a03:2880:10ff:70::]) by smtp.gmail.com with ESMTPSA id 46e09a7af769-7eb8998850fsm5937937a34.11.2026.07.06.13.45.08 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 06 Jul 2026 13:45:08 -0700 (PDT) From: Joshua Hahn To: linux-mm@kvack.org Cc: Andrew Morton , David Hildenbrand , Lorenzo Stoakes , "Liam R. Howlett" , Vlastimil Babka , Mike Rapoport , Suren Baghdasaryan , Michal Hocko , linux-kernel@vger.kernel.org, kernel-team@meta.com Subject: [PATCH] mm/memory-tiers: cache top tier nodes Date: Mon, 6 Jul 2026 13:45:06 -0700 Message-ID: <20260706204507.98522-1-joshua.hahnjy@gmail.com> X-Mailer: git-send-email 2.53.0 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit node_is_toptier() is called in a few hot paths: task_numa_fault(), should_numa_migrate_memory(), folio_migrate_flags(), etc. Each call takes an RCU read section and performs the tier distance check again. Tieredness for all nodes only changes on memory node hotplugs. Instead of recomputing toptier nodes inside each of the hot paths above, compute it only on memory node hotplug events and cache the results. Signed-off-by: Joshua Hahn --- mm/memory-tiers.c | 36 ++++++++++-------------------------- 1 file changed, 10 insertions(+), 26 deletions(-) diff --git a/mm/memory-tiers.c b/mm/memory-tiers.c index 54851d8a195b0..2e6e02ec1fce4 100644 --- a/mm/memory-tiers.c +++ b/mm/memory-tiers.c @@ -71,6 +71,7 @@ bool folio_use_access_time(struct folio *folio) #ifdef CONFIG_NUMA_MIGRATION static int top_tier_adistance; +static nodemask_t toptier_nodes __read_mostly = NODE_MASK_ALL; /* * node_demotion[] examples: * @@ -276,27 +277,7 @@ static struct memory_tier *__node_get_memory_tier(int node) #ifdef CONFIG_NUMA_MIGRATION bool node_is_toptier(int node) { - bool toptier; - pg_data_t *pgdat; - struct memory_tier *memtier; - - pgdat = NODE_DATA(node); - if (!pgdat) - return false; - - rcu_read_lock(); - memtier = rcu_dereference(pgdat->memtier); - if (!memtier) { - toptier = true; - goto out; - } - if (memtier->adistance_start <= top_tier_adistance) - toptier = true; - else - toptier = false; -out: - rcu_read_unlock(); - return toptier; + return node_isset(node, toptier_nodes); } void node_get_allowed_targets(pg_data_t *pgdat, nodemask_t *targets) @@ -497,19 +478,22 @@ static void establish_demotion_targets(void) } } /* - * Now build the lower_tier mask for each node collecting node mask from - * all memory tier below it. This allows us to fallback demotion page - * allocation to a set of nodes that is closer the above selected - * preferred node. + * A node stays toptier unless it belongs to a tier below + * top_tier_adistance, while each tier's lower_tier_mask collects the + * nodes of every tier below it so demotion page allocation can fall + * back to nodes closer to the selected preferred node. */ + toptier_nodes = node_states[N_MEMORY]; lower_tier = node_states[N_MEMORY]; list_for_each_entry(memtier, &memory_tiers, list) { + tier_nodes = get_memtier_nodemask(memtier); + if (memtier->adistance_start > top_tier_adistance) + nodes_andnot(toptier_nodes, toptier_nodes, tier_nodes); /* * Keep removing current tier from lower_tier nodes, * This will remove all nodes in current and above * memory tier from the lower_tier mask. */ - tier_nodes = get_memtier_nodemask(memtier); nodes_andnot(lower_tier, lower_tier, tier_nodes); memtier->lower_tier_mask = lower_tier; } -- 2.53.0-Meta