From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mx0a-0031df01.pphosted.com (mx0a-0031df01.pphosted.com [205.220.168.131]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B7AC72F7EE7 for ; Sat, 13 Jun 2026 17:21:32 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=205.220.168.131 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1781371294; cv=none; b=C77F/bXe6QGDyHJ+qDKpmTLsyPJEkzKc389v79+FWBmLmcPGZvw5bL1RnJ2HlK1O++74jzHsxqA5Y33xYUw9WptpcxHNQm+HWOKqv47goi1VhmQWFqMcoanoQjROZmWdkJROrDMERQ1VJZ7WBZJ+PqqhGZEdd0zmnLD0Z01+DO8= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1781371294; c=relaxed/simple; bh=JrEBDVSR2nkTeNdCYAhWPyaNXTAWBRV/Ax7hvp9FT5w=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=GsWMJs5LXOadZ3VZKQ+lI0p791wHOQYpxkvcbc2bmml/qxpzL1fuD4xhJEzyZfQWLjHSz4JOdT1jEX6ASU14YMBJM1iyhToj6a+f7Y/M69B9iCicAGWgvN0P/RThH2+ZPo5l7LPQ/s7QlVigjiz5NNPy0UqtvE40wpWqGe8Ytmo= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=oss.qualcomm.com; spf=pass smtp.mailfrom=oss.qualcomm.com; dkim=pass (2048-bit key) header.d=qualcomm.com header.i=@qualcomm.com header.b=Hs6ygKrH; dkim=pass (2048-bit key) header.d=oss.qualcomm.com header.i=@oss.qualcomm.com header.b=bma4WZRf; arc=none smtp.client-ip=205.220.168.131 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=oss.qualcomm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=oss.qualcomm.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=qualcomm.com header.i=@qualcomm.com header.b="Hs6ygKrH"; dkim=pass (2048-bit key) header.d=oss.qualcomm.com header.i=@oss.qualcomm.com header.b="bma4WZRf" Received: from pps.filterd (m0279865.ppops.net [127.0.0.1]) by mx0a-0031df01.pphosted.com (8.18.1.11/8.18.1.11) with ESMTP id 65DF9HEe1235339 for ; Sat, 13 Jun 2026 17:21:32 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=qualcomm.com; h= cc:content-transfer-encoding:content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to; s=qcppdkim1; bh= gmDg6wpviGfici5wFw4rRp3TOW9sHNaqJUgICxRvP6Q=; b=Hs6ygKrHsfqhLSA/ y3v+FTKnv6YyTHSt4kUez393vsyDxS9tMSuz7GJlMqGy9X1VFhqDtl+fYRVqU/UL 4oCC1wpP/zYPctUNGzpUo1WMQrRzlYQlYlkSgMdiEGYNR8cdDr+xiJXR2fRBCp/r 7EwC+PVQ28p+bs9+38ypxUawOxqMwe+KtoqTCtQ3nNnfwysj5V8iZz08/v/nfRvq vZ/LAGISPOExJyT/FRXVXjLROgN1aVQmgY4MqlBOYgFBM1w2nUdykpLO0eYcOHb6 l1ISlJz08W1pRK78IR8smpIQF/fC0irKLZybRQfjq5LzjL1vnKzaUfBGo81Bm+4T Z+MXTw== Received: from mail-pf1-f198.google.com (mail-pf1-f198.google.com [209.85.210.198]) by mx0a-0031df01.pphosted.com (PPS) with ESMTPS id 4eryffhkvh-1 (version=TLSv1.3 cipher=TLS_AES_128_GCM_SHA256 bits=128 verify=NOT) for ; Sat, 13 Jun 2026 17:21:31 +0000 (GMT) Received: by mail-pf1-f198.google.com with SMTP id d2e1a72fcca58-84233efcaadso1358563b3a.1 for ; Sat, 13 Jun 2026 10:21:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oss.qualcomm.com; s=google; t=1781371291; x=1781976091; darn=vger.kernel.org; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :mime-version:subject:date:from:from:to:cc:subject:date:message-id :reply-to; bh=gmDg6wpviGfici5wFw4rRp3TOW9sHNaqJUgICxRvP6Q=; b=bma4WZRfQBkzbLC2FX2aCTBLI3OnZCSIb/zvmTzGCUneh725FNgWumyQK0bTZLUoAo rWnXZXo+GkSYP8e2ImjLoyNsZjVnFIPnV214Ft7qiSTBgumDCWAhE1+vdWPkmoVjO0po RmbdTz8xVFfvuTF7E9sVqYPdFJiI/beTcTFmSX7JBcfM6YeptrmqYcVR4ZDu78KYv/LF CTGY7vcBm0FzmDqEYQPFVX0vv9qmzaV6SCCCIuYbXPV8YWFvxgXRNaHDi6Qorkr6gvUs oshgirqxdRugm5LY3KdYMizO9QlDKdYQbVuQMCgg0sNKPkMjvaAtGYK5veTCwVBqeeMH mm3g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1781371291; x=1781976091; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :mime-version:subject:date:from:x-gm-gg:x-gm-message-state:from:to :cc:subject:date:message-id:reply-to; bh=gmDg6wpviGfici5wFw4rRp3TOW9sHNaqJUgICxRvP6Q=; b=nEEXD5Y1MZ0m1Di6Tf3e++EzsOP7gV2wKMWU5jjvHATx6Z0evAAX9of4iaW6P883pZ m06amWmzdPIaQQ89rknexnEdWuTCo92fUfdw+hWMaevU9vB1onedL9GDTPIg2ifM93lb i+TxwlX8V/ktMxOHhhDzobnltAzijTOYX7NoMVa63biNpm4W8vNeO8g3ElugkP+OhNNe /lolW7dPRJJ9xEE+ROWiPw67CQB0vljqq4vXfjPk7cpmUeoD79qDD7ELrADmMUTue+sD noz8eKbZFud8qr8sAZ6KwZ0HFx4i8XkE/WyaK9i4CNExTglZyBhWpapw4ZWcB4kK0m3H oCQA== X-Forwarded-Encrypted: i=1; AFNElJ9jBh+XT12z6iOURzyrlzUH7ISYb++mbAKqdTlFbOZpeqsYB6+DLf/FsEElsysWO4y1Lw1WaubfEWle/TY=@vger.kernel.org X-Gm-Message-State: AOJu0YwGzVFVE6G/E+0//vsGly7H7XRBmCREBOyxgVkpRkGePZFhRb/9 6GDfb5z+ZIo9iyzA5XK2ealhajZc6gj+2YxSwSjW8GzmJOxj54gzkgDHm0ATftfNs0v8aeAAxjD 70m2iRYMBp4Vfex6wIFArCg0Lfp9sQ/IQCvrF0X1ldXvCgy573djYJHwWaI4tvj8NuFs= X-Gm-Gg: Acq92OHV4fmYY289Cx5mJGrUKjXUA5LOG300F/e+au1QGGn7fNL0ubgZu6KvNmdEJLg Fy/Hq0lb8loSZPh8dVmYHJq2C6tg9nP2nmAQKqqT9CoD4j9EhzYcdsUfveo2qDpLbUv2Mn61fg0 6ZrmXLmUgHJkf6KtqR5I81bYv3yWtgDV9WRgyaeiRlMQnydysmg9Rgsk4zCSOBhn8d7fSFfsQN0 mge8RdxrBcXamg7+QlHjIplueRocTUvERmHfzTei8khmrbXZ4pTj6W7h/sXDvW1nFZT7lW57cFq 2okEHDWCl9qLys1TwERd03W9W4wC+JyY0J+ZNlfo685pHSK2pMhWEiiIvyz/5iFZzY7n+GvuoHz OyQwiYJqmmh6JzrwbbKPpJUB4pxja3b75+ge/55pAeJWdzeDabiOXvg== X-Received: by 2002:a05:6a00:3d48:b0:842:3a98:b34d with SMTP id d2e1a72fcca58-8434ce83f17mr8390911b3a.31.1781371291158; Sat, 13 Jun 2026 10:21:31 -0700 (PDT) X-Received: by 2002:a05:6a00:3d48:b0:842:3a98:b34d with SMTP id d2e1a72fcca58-8434ce83f17mr8390867b3a.31.1781371290674; Sat, 13 Jun 2026 10:21:30 -0700 (PDT) Received: from hu-pranarya-hyd.qualcomm.com ([202.46.22.19]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-8434accbec5sm5390913b3a.16.2026.06.13.10.21.22 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 13 Jun 2026 10:21:30 -0700 (PDT) From: Pranjal Arya Date: Sat, 13 Jun 2026 22:49:50 +0530 Subject: [PATCH RFC 08/12] mm/vmalloc: track lazy-purge queue as a list_head Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 8bit Message-Id: <20260613-vmalloc_maple-v1-8-0aa740bb944b@oss.qualcomm.com> References: <20260613-vmalloc_maple-v1-0-0aa740bb944b@oss.qualcomm.com> In-Reply-To: <20260613-vmalloc_maple-v1-0-0aa740bb944b@oss.qualcomm.com> To: Andrew Morton , Uladzislau Rezki , "Liam R. Howlett" , Alice Ryhl , Andrew Ballance Cc: linux-arm-msm@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, maple-tree@lists.infradead.org, Lorenzo Stoakes , Pranjal Shrivastava , Will Deacon , Suzuki K Poulose , Neil Armstrong , Mostafa Saleh , Balbir Singh , Suren Baghdasaryan , Marco Elver , Dmitry Vyukov , Alexander Potapenko , Shuah Khan , Dev Jain , Brendan Jackman , Puranjay Mohan , Santosh Shukla , Wyes Karny , Pranjal Arya , Sudeep Holla X-Mailer: b4 0.15.2 X-Developer-Signature: v=1; a=ed25519-sha256; t=1781371215; l=7702; i=pranjal.arya@oss.qualcomm.com; s=20260516; h=from:subject:message-id; bh=JrEBDVSR2nkTeNdCYAhWPyaNXTAWBRV/Ax7hvp9FT5w=; b=HlA43z7W1BgFHiQdUCJ9NYyv6guSh8B+257WOVr1UNUmQ3utJWN5wVgMVBRX9p2ibCldFaTWa 7E0Iaid+Gw+AfDte7XbFrrnh/UdH/cFriQdiXtcPQ+ualCVfPKMbz1S X-Developer-Key: i=pranjal.arya@oss.qualcomm.com; a=ed25519; pk=ymtcTlccEIDsi3ErhpjIoZZHKdPBYWGWW0Lchs5MsbE= X-Authority-Analysis: v=2.4 cv=HuxG3UTS c=1 sm=1 tr=0 ts=6a2d919b cx=c_pps a=m5Vt/hrsBiPMCU0y4gIsQw==:117 a=fChuTYTh2wq5r3m49p7fHw==:17 a=IkcTkHD0fZMA:10 a=FelO9ux0wxsA:10 a=s4-Qcg_JpJYA:10 a=VkNPw1HP01LnGYTKEx00:22 a=u7WPNUs3qKkmUXheDGA7:22 a=Um2Pa8k9VHT-vaBCBUpS:22 a=EUspDBNiAAAA:8 a=LbbEsUlp8jkdcCOz7_EA:9 a=3ZKOabzyN94A:10 a=QEXdDO2ut3YA:10 a=IoOABgeZipijB_acs4fv:22 X-Proofpoint-ORIG-GUID: WoCV72kNTZI2ux378kKNe0zzMjEy-spi X-Proofpoint-Spam-Info: AW1haW4tMjYwNjEzMDE4MCBTYWx0ZWRfX9VlKvdJWfXh7 GtjLX47UZY++rhlqrT6CYy/p6l4TLPab/ExcEqqsKgQEcq/s+xfh+jCdP11HZkZ/elldV0i+Ax/ It4oGsg/wk2SgvATDKhOVlRfcvZNG98= X-Proofpoint-GUID: WoCV72kNTZI2ux378kKNe0zzMjEy-spi X-Proofpoint-Spam-Details-Enc: AW1haW4tMjYwNjEzMDE4MCBTYWx0ZWRfXxvrGl/xzls84 l/hZgfYhxkbe8K4FKSx9hOg4ab+/OxqfA3XgmO6vmGGWfQ/DLE8ksdPR1uNmFvyAZMNdfXOXuZK bXTluaXHRbMBmn33WairQrQ9npyOez6ppn1abOpWsrFREix5l7Pdj0kXOALOMu/D1IdfRZs6gB+ 6DhQ4uuvyKoSihTvxTyT4lhY8jIzB7a2tp0aScq8CB8wPM5VYiWgtIK/CYTUNkKGpDZQ4g0m7eD krpLG1ebgKkN8yimT/5cTOoAFKLurmiXkC5f5H+2+Cj5DAE/ispDrji5DppxBj8Jh87CnTLFY4n eApkGzLUk3AXHtzN/P3G4xyaRiv/RJ/dAM9icJUYCujokKlOsJP3CoL++4HkbBdeETMbSA05App K8vlKkSULFONJsAXsHZKsfQ9+qVeiHnTcVvvWu9ueXBGTrqOp8BFgTSQyOj/+hzmgbJkJTjL5zA dNO6TD8KYahkXD2frEw== X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1143,Hydra:6.1.125,FMLib:17.12.100.49 definitions=2026-06-13_03,2026-06-12_03,2025-10-01_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 lowpriorityscore=0 bulkscore=0 adultscore=0 impostorscore=0 priorityscore=1501 phishscore=0 clxscore=1011 spamscore=0 suspectscore=0 malwarescore=0 classifier=typeunknown authscore=0 authtc= authcc= route=outbound adjust=0 reason=mlx scancount=1 engine=8.22.0-2606040000 definitions=main-2606130180 The lazy queue is bulk-drained from the purge worker; nothing queries it by address. publish_vmap_area_lazy() inserts into the queue and purge_vmap_areas_lazy() walks it linearly. A list_head expresses the actual usage and saves the per-publish maple insert. Per-node vn->lazy.mt becomes vn->lazy_list. The locking discipline (vn->lazy.lock still serialises inserts) is unchanged. Signed-off-by: Pranjal Arya --- mm/vmalloc.c | 133 +++++++++++++++++++++++++---------------------------------- 1 file changed, 57 insertions(+), 76 deletions(-) diff --git a/mm/vmalloc.c b/mm/vmalloc.c index 73a40a88dbf6..1b73001e197e 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -942,6 +942,16 @@ static struct vmap_node { struct mt_list busy; struct mt_list lazy; + /* + * Lazy list. The lazy index is no longer queried by address on the + * hot path: free_vmap_area_noflush() pushes the VA via list_add and + * purge drains it via list_splice. Keeping a list head sidesteps a + * mas_store on every vfree and a mas_for_each + per-entry + * mas_store(NULL) during purge. lazy.mt is retained for the rare + * non-perf_mode rollback path inside publish_vmap_area_lazy(). + */ + struct list_head lazy_list; + /* * Ready-to-free areas. */ @@ -1510,52 +1520,37 @@ unlink_vmap_area_busy_locked(struct vmap_area *va, struct vmap_node *vn) static __always_inline bool insert_vmap_area_lazy_locked(struct vmap_area *va, struct vmap_node *vn) { - int err; - lockdep_assert_held(&vn->lazy.lock); - try_init_lazy_mt_locked(vn); - if (WARN_ON_ONCE(!vn->lazy.mt_enabled)) - return false; - - if (!validate_vmap_area_range_insert_mt_locked(&vn->lazy.mt, - va->va_start, - va->va_end)) + /* + * The maple-tree lazy index is bypassed in the hot path: a simple + * list saves one mas_store per vfree and one mas_for_each + N + * mas_store(NULL) during purge. lazy.mt is left untouched here so + * the non-perf_mode publish_vmap_area_lazy() rollback can still + * unlink the VA via unlink_vmap_area_lazy_locked() if it inserted + * one — that path is unreachable in steady state with perf_mode on. + */ + if (WARN_ON_ONCE(!list_empty(&va->list))) return false; - INIT_LIST_HEAD(&va->list); - - MA_STATE(mas, &vn->lazy.mt, va->va_start, va->va_end - 1); - - err = mas_preallocate(&mas, va, GFP_NOWAIT | __GFP_NOWARN); - if (!err) { - mas_store_prealloc(&mas, va); - mas_destroy(&mas); - return true; - } - - err = mas_store_gfp(&mas, va, GFP_ATOMIC | __GFP_NOWARN); - return !WARN_ON_ONCE(err); + list_add_tail(&va->list, &vn->lazy_list); + return true; } static __always_inline bool unlink_vmap_area_lazy_locked(struct vmap_area *va, struct vmap_node *vn) { - int err; - lockdep_assert_held(&vn->lazy.lock); - try_init_lazy_mt_locked(vn); - if (WARN_ON_ONCE(!vn->lazy.mt_enabled)) - return false; - - MA_STATE(mas, &vn->lazy.mt, va->va_start, va->va_end - 1); - - err = mas_store_gfp(&mas, NULL, GFP_ATOMIC | __GFP_NOWARN); - if (WARN_ON_ONCE(err)) + /* + * Match insert_vmap_area_lazy_locked()'s list-based fast path. Used + * only by publish_vmap_area_lazy() rollback, which is unreachable in + * steady state but kept for the non-perf_mode early-boot window. + */ + if (list_empty(&va->list)) return false; - INIT_LIST_HEAD(&va->list); + list_del_init(&va->list); return true; } @@ -1610,48 +1605,22 @@ lazy_vmap_areas_empty_locked(struct vmap_node *vn) { lockdep_assert_held(&vn->lazy.lock); - try_init_lazy_mt_locked(vn); - if (WARN_ON_ONCE(!vn->lazy.mt_enabled)) - return true; - - return mtree_empty(&vn->lazy.mt); + return list_empty(&vn->lazy_list); } static __always_inline void move_lazy_vmap_areas_to_purge_locked(struct vmap_node *vn) { - LIST_HEAD(move_list); - struct vmap_area *va, *n_va; - int err; - lockdep_assert_held(&vn->lazy.lock); - try_init_lazy_mt_locked(vn); - if (WARN_ON_ONCE(!vn->lazy.mt_enabled)) - return; - - MA_STATE(mas, &vn->lazy.mt, 0, 0); - - mas_for_each(&mas, va, ULONG_MAX) - list_add_tail(&va->list, &move_list); - /* - * Erase ranges one-by-one and move only successfully erased entries to - * purge_list. This avoids destroy/reinit churn and keeps lazy index - * coherence if an erase operation fails under pressure. + * Move every queued VA to purge_list with a single splice. The + * sort-by-address property that the maple-tree lazy index used to + * provide is no longer used by purge_vmap_node(); kasan_release + * computes its own min/max over the resulting purge_list when + * needed. */ - list_for_each_entry_safe(va, n_va, &move_list, list) { - MA_STATE(mas_erase, &vn->lazy.mt, va->va_start, va->va_end - 1); - - err = mas_store_gfp(&mas_erase, NULL, GFP_ATOMIC | __GFP_NOWARN); - if (unlikely(err)) { - WARN_ON_ONCE(err); - list_del_init(&va->list); - continue; - } - - list_move_tail(&va->list, &vn->purge_list); - } + list_splice_tail_init(&vn->lazy_list, &vn->purge_list); } static __always_inline bool @@ -2806,13 +2775,18 @@ static void kasan_release_vmalloc_node(struct vmap_node *vn) { struct vmap_area *va; - unsigned long start, end; + unsigned long start = ULONG_MAX, end = 0; unsigned int batch_count = 0; - start = list_first_entry(&vn->purge_list, struct vmap_area, list)->va_start; - end = list_last_entry(&vn->purge_list, struct vmap_area, list)->va_end; - + /* + * purge_list is no longer sorted by address (lazy_list is built in + * insertion order via list_add_tail). Compute the bounding range + * inline with the per-VA shadow-release loop to avoid a second walk. + */ list_for_each_entry(va, &vn->purge_list, list) { + start = min(start, va->va_start); + end = max(end, va->va_end); + if (is_vmalloc_or_module_addr((void *) va->va_start)) kasan_release_vmalloc(va->va_start, va->va_end, va->va_start, va->va_end, @@ -2824,7 +2798,9 @@ kasan_release_vmalloc_node(struct vmap_node *vn) } } - kasan_release_vmalloc(start, end, start, end, KASAN_VMALLOC_TLB_FLUSH); + if (start < end) + kasan_release_vmalloc(start, end, start, end, + KASAN_VMALLOC_TLB_FLUSH); } static void purge_vmap_node(struct work_struct *work) @@ -2938,6 +2914,7 @@ static bool __purge_vmap_area_lazy(unsigned long start, unsigned long end, static cpumask_t purge_nodes; unsigned int nr_purge_nodes; struct vmap_node *vn; + struct vmap_area *va; int i; lockdep_assert_held(&vmap_purge_lock); @@ -2964,11 +2941,14 @@ static bool __purge_vmap_area_lazy(unsigned long start, unsigned long end, move_lazy_vmap_areas_to_purge_locked(vn); spin_unlock(&vn->lazy.lock); - start = min(start, list_first_entry(&vn->purge_list, - struct vmap_area, list)->va_start); - - end = max(end, list_last_entry(&vn->purge_list, - struct vmap_area, list)->va_end); + /* + * lazy_list (and therefore purge_list) is no longer sorted by + * address. Compute the bounding range by walking purge_list. + */ + list_for_each_entry(va, &vn->purge_list, list) { + start = min(start, va->va_start); + end = max(end, va->va_end); + } cpumask_set_cpu(node_to_id(vn), &purge_nodes); } @@ -6153,6 +6133,7 @@ static void vmap_init_nodes(void) mt_init_flags(&vn->lazy.mt, MT_FLAGS_LOCK_EXTERN); mt_set_external_lock(&vn->lazy.mt, &vn->lazy.lock); vn->lazy.mt_enabled = true; + INIT_LIST_HEAD(&vn->lazy_list); for (i = 0; i < MAX_VA_SIZE_PAGES; i++) { INIT_LIST_HEAD(&vn->pool[i].head); -- 2.34.1