From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-10.0 required=3.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH,MAILING_LIST_MULTI,SIGNED_OFF_BY, SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 95734C0650E for ; Wed, 3 Jul 2019 15:59:55 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 629F62189E for ; Wed, 3 Jul 2019 15:59:55 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="qWymVMwW" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727049AbfGCP7y (ORCPT ); Wed, 3 Jul 2019 11:59:54 -0400 Received: from mail-lj1-f195.google.com ([209.85.208.195]:42240 "EHLO mail-lj1-f195.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726473AbfGCP7x (ORCPT ); Wed, 3 Jul 2019 11:59:53 -0400 Received: by mail-lj1-f195.google.com with SMTP id t28so2997260lje.9 for ; Wed, 03 Jul 2019 08:59:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id; bh=gFrhzCDbKN3dXJqOCjxaWjlxE64lMzIuw0MAKOdMJPo=; b=qWymVMwWBiebY3jqm1IZFIvKbFwtFLiQBcstlPbjsEjP1XWeeND9r9xO1Gv+nq8hMl KMjBNXPgyz9R2PCG624S838/3s2HNifSySYIKpD85drn4YzChNIDbXnNFqcb/+Yr9dxU w+qz2mE3begIDz0gBU9B+lSKVeoz3DPpfkts4wBr7zcVU3NSEeEwE2Igkn++2ZOww0dt VW9zD5ez59Nzn3H2cAlqNWYEHi9c7yeTv5q8tfkcVNqas3NCu5le3eEb/p1zEpN6x0J5 MWuKxAyKJGUwJLYqYeaT0ronfXLCgUI/sZj3wH9ateZX1SVSOVFpQgea8ix3pvwAle5H CtUA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id; bh=gFrhzCDbKN3dXJqOCjxaWjlxE64lMzIuw0MAKOdMJPo=; b=a7bUrnQxd3lCPjhY5geJQ7+Adg6jpvneVNVH32ae7dpyqIuk32UHEASVmiuKxExYdY TuIvceT4Rnbebt5gDbRGoYFLHBLd0X+cD0nn0JAgwx9grRx7/uDFFYH33yM7/9cJ2boo vWe4wNgpgsxqbA+lBipi31EAGTf16qjwe50gJcRUAfRvKqxqmkqi8joHYiBkolwqSYL4 FIYkjvzI6HIg8XZQrnv6JjQ2SoLEeUazm8qZCVWwGsTxrzV0V0FJ4qubVxkXBRt3Sxwn tJZQWGHY6i4yCD/8U75Bo+LBtLYthW23M5T8KagH44s5VQRy5NIo3bF1JLs9/cHqHDNX UpCg== X-Gm-Message-State: APjAAAXXUqV5HsUe07BHp6NUy7Op/FJSxlih6PoqWQKhfhcpeJOGVxGK t9x05E+vhUGKooBDZcbaIpERAYYV+YPwiQ== X-Google-Smtp-Source: APXvYqxQN0IjsHvsXPGL1/dmxNkT7MMV5tTjkmdNONnvAy7DjCdslgHJ1eS+uFVKcySrprZrSgDZXA== X-Received: by 2002:a2e:9ad1:: with SMTP id p17mr22136975ljj.34.1562169590522; Wed, 03 Jul 2019 08:59:50 -0700 (PDT) Received: from pc636.semobile.internal ([37.139.158.167]) by smtp.gmail.com with ESMTPSA id t21sm441121lfd.85.2019.07.03.08.59.49 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 03 Jul 2019 08:59:49 -0700 (PDT) From: "Uladzislau Rezki (Sony)" To: Andrew Morton Cc: linux-mm@kvack.org, LKML , Roman Gushchin , Uladzislau Rezki , Hillf Danton , Michal Hocko , Matthew Wilcox , Oleksiy Avramchenko , Steven Rostedt Subject: [PATCH 1/1] mm/vmalloc: do not keep unpurged areas in the busy tree Date: Wed, 3 Jul 2019 17:59:42 +0200 Message-Id: <20190703155942.13571-1-urezki@gmail.com> X-Mailer: git-send-email 2.11.0 Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org The busy tree can be quite big, even though the area is freed or unmapped it still stays there until "purge" logic removes it. 1) Optimize and reduce the size of "busy" tree by removing a node from it right away as soon as user triggers free paths. It is possible to do so, because the allocation is done using another augmented tree. The vmalloc test driver shows the difference, for example the "fix_size_alloc_test" is ~11% better comparing with default configuration: sudo ./test_vmalloc.sh performance Summary: fix_size_alloc_test loops: 1000000 avg: 993985 usec Summary: full_fit_alloc_test loops: 1000000 avg: 973554 usec Summary: long_busy_list_alloc_test loops: 1000000 avg: 12617652 usec Summary: fix_size_alloc_test loops: 1000000 avg: 882263 usec Summary: full_fit_alloc_test loops: 1000000 avg: 973407 usec Summary: long_busy_list_alloc_test loops: 1000000 avg: 12593929 usec 2) Since the busy tree now contains allocated areas only and does not interfere with lazily free nodes, introduce the new function show_purge_info() that dumps "unpurged" areas that is propagated through "/proc/vmallocinfo". 3) Eliminate VM_LAZY_FREE flag. Signed-off-by: Uladzislau Rezki (Sony) --- mm/vmalloc.c | 51 ++++++++++++++++++++++++++++++++++++++++++--------- 1 file changed, 42 insertions(+), 9 deletions(-) diff --git a/mm/vmalloc.c b/mm/vmalloc.c index edb212298c8a..1219152e60b1 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -329,7 +329,6 @@ EXPORT_SYMBOL(vmalloc_to_pfn); #define DEBUG_AUGMENT_PROPAGATE_CHECK 0 #define DEBUG_AUGMENT_LOWEST_MATCH_CHECK 0 -#define VM_LAZY_FREE 0x02 #define VM_VM_AREA 0x04 static DEFINE_SPINLOCK(vmap_area_lock); @@ -534,7 +533,7 @@ link_va(struct vmap_area *va, struct rb_root *root, static __always_inline void unlink_va(struct vmap_area *va, struct rb_root *root) { - if (WARN_ON(RB_EMPTY_NODE(&va->rb_node))) + if (RB_EMPTY_NODE(&va->rb_node)) return; if (root == &free_vmap_area_root) @@ -1160,7 +1159,11 @@ EXPORT_SYMBOL_GPL(unregister_vmap_purge_notifier); static void __free_vmap_area(struct vmap_area *va) { /* - * Remove from the busy tree/list. + * In most cases VA is not attached to the tree, but there + * are a few exceptions: + * + * - is linked only in case of pcpu, recovery part; + * - if radix_tree_preload gets failed, see new_vmap_block(). */ unlink_va(va, &vmap_area_root); @@ -1311,6 +1314,10 @@ static void free_vmap_area_noflush(struct vmap_area *va) { unsigned long nr_lazy; + spin_lock(&vmap_area_lock); + unlink_va(va, &vmap_area_root); + spin_unlock(&vmap_area_lock); + nr_lazy = atomic_long_add_return((va->va_end - va->va_start) >> PAGE_SHIFT, &vmap_lazy_nr); @@ -2130,14 +2137,13 @@ struct vm_struct *remove_vm_area(const void *addr) might_sleep(); - va = find_vmap_area((unsigned long)addr); + spin_lock(&vmap_area_lock); + va = __find_vmap_area((unsigned long)addr); if (va && va->flags & VM_VM_AREA) { struct vm_struct *vm = va->vm; - spin_lock(&vmap_area_lock); va->vm = NULL; va->flags &= ~VM_VM_AREA; - va->flags |= VM_LAZY_FREE; spin_unlock(&vmap_area_lock); kasan_free_shadow(vm); @@ -2145,6 +2151,8 @@ struct vm_struct *remove_vm_area(const void *addr) return vm; } + + spin_unlock(&vmap_area_lock); return NULL; } @@ -3421,6 +3429,22 @@ static void show_numa_info(struct seq_file *m, struct vm_struct *v) } } +static void show_purge_info(struct seq_file *m) +{ + struct llist_node *head; + struct vmap_area *va; + + head = READ_ONCE(vmap_purge_list.first); + if (head == NULL) + return; + + llist_for_each_entry(va, head, purge_list) { + seq_printf(m, "0x%pK-0x%pK %7ld unpurged vm_area\n", + (void *)va->va_start, (void *)va->va_end, + va->va_end - va->va_start); + } +} + static int s_show(struct seq_file *m, void *p) { struct vmap_area *va; @@ -3433,10 +3457,9 @@ static int s_show(struct seq_file *m, void *p) * behalf of vmap area is being tear down or vm_map_ram allocation. */ if (!(va->flags & VM_VM_AREA)) { - seq_printf(m, "0x%pK-0x%pK %7ld %s\n", + seq_printf(m, "0x%pK-0x%pK %7ld vm_map_ram\n", (void *)va->va_start, (void *)va->va_end, - va->va_end - va->va_start, - va->flags & VM_LAZY_FREE ? "unpurged vm_area" : "vm_map_ram"); + va->va_end - va->va_start); return 0; } @@ -3472,6 +3495,16 @@ static int s_show(struct seq_file *m, void *p) show_numa_info(m, v); seq_putc(m, '\n'); + + /* + * As a final step, dump "unpurged" areas. Note, + * that entire "/proc/vmallocinfo" output will not + * be address sorted, because the purge list is not + * sorted. + */ + if (list_is_last(&va->list, &vmap_area_list)) + show_purge_info(m); + return 0; } -- 2.11.0