From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-pl1-f199.google.com (mail-pl1-f199.google.com [209.85.214.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id CDA163DCD9B for ; Thu, 27 Aug 2026 23:31:26 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.199 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787873488; cv=none; b=dYrD23iJalIBXRS33akKyv71X4BqNEHE7T2xAkEgme47D/2KCPck589O41gX9FdDmdKxgJ7tKK2eCqZ/I2ATUap9LE1jn5Dls5ZND50whpRZzkjWwo3HSHe4i38kCb3W8WQhWXqOMzSOtmq3zaEbfKfEDbAl9NR5YeaPTQELTYU= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787873488; c=relaxed/simple; bh=OL5kyzzH3pevyaGpCfzUthvi5UyUYi/GZsUUb63MbgA=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=qWutvfyBIBNMYFZf5xxW/zE2CzyiqrxtcOzPdY6ZaIddv9UFK6z862+7IJQcgyI9uLOTofOhJ1S6d4/rF2y/b21aSuXDvp2akOvtdFmdSwNqR4T61bwd4zZnc31yXp+3hwRPn3qVSt/G6AgXJITGIK06d9QnnUtENp5j+DMRmho= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--stevensd.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=EnnqF4V/; arc=none smtp.client-ip=209.85.214.199 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--stevensd.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="EnnqF4V/" Received: by mail-pl1-f199.google.com with SMTP id d9443c01a7336-2ccb687f82eso3260405ad.3 for ; Thu, 27 Aug 2026 16:31:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1787873486; x=1788478286; darn=vger.kernel.org; h=content-type:cc:to:from:subject:message-id:references:mime-version :in-reply-to:date:from:to:cc:subject:date:message-id:reply-to :content-type; bh=/lOzSmEZ51hzHprpWRTTa0ItgwWLWckqKQo+nDAEYGw=; b=EnnqF4V/hmj76dHndqgFoNkxdLWMWu2xhJC+n1smXB2RMIUla8CfUp+jl59LRjmCm4 3CUJ2nQi+zv4eRxpJ0qoyiObwFRb9KLA+QTYtvxdJVvT0NANAyP6/3YouiUTmk/ZU+QI OvqFvYX879aqrmkafCTOuACsXXmec7qshPzdi3Ppo2Y5GwrQE7Mp17gKh2YdrEUhf+Ct scsb1RrPw1jxfpu6fWaZ4KtAj9pWNhpkC4xVrmQSl0hoytRF/qsWNnuGg7i4HCLcv/+W sZjSxk1BCHmgfwu77n/GFhVDI5pupAU14RaeNBbvM4DwmH+wV+yIVtcUul5yaaUMhLki iowQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1787873486; x=1788478286; h=content-type:cc:to:from:subject:message-id:references:mime-version :in-reply-to:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to:content-type; bh=/lOzSmEZ51hzHprpWRTTa0ItgwWLWckqKQo+nDAEYGw=; b=TgoduZh3lyLsfNXvbRC+fVq92g5WIzbOP28xijwzUcZ1SNx0BdpRwkh2aeR6GPnN+C MR1Y8Lyd0ThWG+NqHZeNaHtNT0b9kpEAQD/owfkHLvBeLeFhHrKEeE8oer6PRL5gDUkF yoVTT0BjpmSEhIZ5nq0K8a4BYI4/ARoaczx9Si3UUPIOdISV/1dYeGWbyj4j+wtf4B6o lwLZKo2d5/0vkRoAYLhEbpi2IQ5/2pkK2MvFOn3hkt68ba/Z9jZys0iTv6fFxcOiAa82 gd5sDqGGGL+GrpfrR4PV6ySFuoOH15PlymKMKsLORIC4dwF2afrmZLWaNDD8hCbeV58o MSwg== X-Gm-Message-State: AFuF++lqQZ8JD3EbiSjz3+pRON2MXNRW32pcPjQksKpYAl7v3bcEZTH0 g3zynk5xn86RtS2MO90LKRC2jwqzJg6KBSKtRqR8+HLfMg0yjxHIRD1XIGrRLfxh7j/3MJcx5+E OfZd1iFtjp/iiQQ== X-Received: from dybud10.prod.google.com ([2002:a05:7300:f60a:b0:311:6ce7:736d]) (user=stevensd job=prod-delivery.src-stubby-dispatcher) by 2002:a17:903:458b:b0:2d6:f3f0:3e7a with SMTP id d9443c01a7336-2d74dc2ade3mr36340345ad.3.1787873485885; Thu, 27 Aug 2026 16:31:25 -0700 (PDT) Date: Thu, 27 Aug 2026 16:29:43 -0700 In-Reply-To: <20260827232948.2520558-1-stevensd@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20260827232948.2520558-1-stevensd@google.com> X-Mailer: git-send-email 2.55.0.897.gb25b4bd76c-goog Message-ID: <20260827232948.2520558-6-stevensd@google.com> Subject: [RFC 05/10] fork: allocate reclaimable stacks with VM_SPARSE From: David Stevens To: Catalin Marinas , Will Deacon , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , x86@kernel.org, "H . Peter Anvin" , Andrew Morton , Dave Chinner , Qi Zheng , Roman Gushchin , Muchun Song , Peter Zijlstra , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Valentin Schneider , K Prateek Nayak , Uladzislau Rezki , David Hildenbrand , Lorenzo Stoakes , "Liam R . Howlett" , Vlastimil Babka , Mike Rapoport , Suren Baghdasaryan , Michal Hocko , Kees Cook , Sebastian Andrzej Siewior , Clark Williams , suleiman@google.com Cc: linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-mm@kvack.org, linux-rt-devel@lists.linux.dev, David Stevens Content-Type: text/plain; charset="UTF-8" Since the pages of reclaimable stacks will not always be populated, we need to set VM_SPARSE on their vm_structs to avoid crashes when vread_iter sees a partially reclaimed stack. Note that stacks will only be partially reclaimed when they are associated with a live task, so the stack management and caching code in fork.c won't actually ever see a partially reclaimed stack. Directly managing the vm_structs instead of going through vmalloc also requires doing the freeing of the stack on a work queue instead of in an RCU callback. Previously, we were relying on deferred vfree work to move much of the cleanup work from softirq to process context. The fact that vmap stack pages are included in vmalloc's vmstat is well known. We should continue that to avoid changing procfs. Signed-off-by: David Stevens --- kernel/fork.c | 92 +++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 92 insertions(+) diff --git a/kernel/fork.c b/kernel/fork.c index 10bd76f6dd20..6acad0038b78 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -267,6 +267,97 @@ static bool try_release_thread_stack_to_cache(struct vm_struct *vm_area) return false; } +#ifdef CONFIG_RECLAIMABLE_STACK +static void free_vmap_stack(struct vm_struct *vm_area) +{ + int i; + + remove_vm_area(vm_area->addr); + + for (i = 0; i < vm_area->nr_pages; i++) { + mod_node_page_state(page_pgdat(vm_area->pages[i]), NR_VMALLOC, -1); + __free_page(vm_area->pages[i]); + } + + kfree(vm_area->pages); + kfree(vm_area); +} + +static struct vm_struct *alloc_vmap_stack(int node) +{ + struct vm_struct *vm_area; + int ret; + + vm_area = get_vm_area_node(THREAD_SIZE, THREAD_ALIGN, VM_MAP | VM_SPARSE, node); + if (!vm_area) + return NULL; + + vm_area->pages = kcalloc_node(THREAD_SIZE >> PAGE_SHIFT, sizeof(*vm_area->pages), + GFP_KERNEL | __GFP_ZERO, node); + if (!vm_area->pages) + goto alloc_failure; + + while (vm_area->nr_pages < THREAD_SIZE >> PAGE_SHIFT) { + struct page *page; + gfp_t gfp = GFP_VMAP_STACK | __GFP_HIGHMEM; + + if (node == NUMA_NO_NODE) + page = alloc_pages(gfp, 0); + else + page = alloc_pages_node(node, gfp, 0); + + if (!page) + goto alloc_failure; + + /* + * Non-reclaimable vmap stacks pages aren't charged against an + * memcg until account_kernel_stack(), but they are added to + * the node's NR_VMALLOC counter. Copy that behavior to avoid + * confusing userspace. + */ + mod_node_page_state(page_pgdat(page), NR_VMALLOC, 1); + vm_area->pages[vm_area->nr_pages++] = page; + } + + ret = vmap_pages_range((unsigned long)vm_area->addr, + (unsigned long)vm_area->addr + THREAD_SIZE, + PAGE_KERNEL, vm_area->pages, PAGE_SHIFT); + if (ret) + goto alloc_failure; + + return vm_area; + +alloc_failure: + free_vmap_stack(vm_area); + return NULL; +} + +struct vm_stack { + struct rcu_work work; + struct vm_struct *stack_vm_area; +}; + +static void thread_stack_free_work(struct work_struct *work) +{ + struct vm_stack *vm_stack = container_of(to_rcu_work(work), struct vm_stack, work); + struct vm_struct *vm_area = vm_stack->stack_vm_area; + + if (try_release_thread_stack_to_cache(vm_stack->stack_vm_area)) + return; + + free_vmap_stack(vm_area); +} + +static void thread_stack_delayed_free(struct task_struct *tsk) +{ + struct vm_stack *vm_stack = tsk->stack; + + vm_stack->stack_vm_area = tsk->stack_vm_area; + INIT_RCU_WORK(&vm_stack->work, thread_stack_free_work); + queue_rcu_work(system_wq, &vm_stack->work); +} + +#else /* !CONFIG_RECLAIMABLE_STACK */ static void free_vmap_stack(struct vm_struct *vm_area) { vfree(vm_area->addr); @@ -305,6 +396,7 @@ static void thread_stack_delayed_free(struct task_struct *tsk) vm_stack->stack_vm_area = tsk->stack_vm_area; call_rcu(&vm_stack->rcu, thread_stack_free_rcu); } +#endif /* CONFIG_RECLAIMABLE_STACK */ static int free_vm_stack_cache(unsigned int cpu) { -- 2.55.0.897.gb25b4bd76c-goog