From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 381ECC38A02 for ; Fri, 28 Oct 2022 09:34:15 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230235AbiJ1JeN (ORCPT ); Fri, 28 Oct 2022 05:34:13 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58088 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230103AbiJ1JeL (ORCPT ); Fri, 28 Oct 2022 05:34:11 -0400 Received: from smtp-relay-canonical-0.canonical.com (smtp-relay-canonical-0.canonical.com [185.125.188.120]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 82F281C77E6; Fri, 28 Oct 2022 02:34:09 -0700 (PDT) Received: from [192.168.43.182] (unknown [62.168.35.125]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by smtp-relay-canonical-0.canonical.com (Postfix) with ESMTPSA id 90583422D7; Fri, 28 Oct 2022 09:34:07 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=canonical.com; s=20210705; t=1666949647; bh=bQ/1OS6mvFzzu0LHm35vKXK9IjpkGW7uJvvale0LiEA=; h=Message-ID:Date:MIME-Version:Subject:To:Cc:References:From: In-Reply-To:Content-Type; b=qz+RM5EGaOqTeHePR4rNeD8+yp2Ra/rDO8x9BpOQePi81kSwziLnZEtQTI0eh+se2 L3rWzFUnEe2PkXNlcdMC0eVweXYHtZSn0IpMS8PShKCE30p0va/SDyHZY1xsVhqtL8 E8uryNzzZBPrjbZoVPZgyasWGxgbmslHCEiQJeX79ZyB5nBJHAblzi69LPq3x5OcOK FYWrU9OyJadAmSD/AmiF4+36f1iqQt13BRUY3TjSE2/arFmVfMeWdHpQ8pUpsy2JaO 8N99tthUbFzKIZzZ8Uaw2+vIni0JxsYPT6XyNG2xiaIuOXpDDcY0bykYi/8XWPNtb6 97Fn7uhb30iOg== Message-ID: Date: Fri, 28 Oct 2022 02:34:07 -0700 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Thunderbird/102.2.2 Subject: Re: apparmor: global buffers spin lock may get contended Content-Language: en-US To: Sergey Senozhatsky , Sebastian Andrzej Siewior Cc: Peter Zijlstra , Tomasz Figa , linux-kernel@vger.kernel.org, linux-security-module@vger.kernel.org References: From: John Johansen Organization: Canonical In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On 7/13/21 06:19, Sergey Senozhatsky wrote: > Hi, > > We've notices that apparmor has switched from using per-CPU buffer pool > and per-CPU spin_lock to a global spin_lock in df323337e507a0009d3db1ea. > > This seems to be causing some contention on our build machines (with > quite a bit of cores). Because that global spin lock is a part of the > stat() sys call (and perhaps some other) > > E.g. > > - 9.29% 0.00% clang++ [kernel.vmlinux] > - 9.28% entry_SYSCALL_64_after_hwframe > - 8.98% do_syscall_64 > - 7.43% __do_sys_newlstat > - 7.43% vfs_statx > - 7.18% security_inode_getattr > - 7.15% apparmor_inode_getattr > - aa_path_perm > - 3.53% aa_get_buffer > - 3.47% _raw_spin_lock > 3.44% native_queued_spin_lock_slowpath > - 3.49% aa_put_buffer.part.0 > - 3.45% _raw_spin_lock > 3.43% native_queued_spin_lock_slowpath > > Can we fix this contention? sorry for the delay on this. Below is a proposed patch that I have been testing to deal with this issue. From d026988196fdbda7234fb87bc3e4aea22edcbaf9 Mon Sep 17 00:00:00 2001 From: John Johansen Date: Tue, 25 Oct 2022 01:18:41 -0700 Subject: [PATCH] apparmor: cache buffers on percpu list if there is lock contention On a heavily loaded machine there can be lock contention on the global buffers lock. Add a percpu list to cache buffers on when lock contention is encountered. When allocating buffers attempt to use cached buffers first, before taking the global buffers lock. When freeing buffers try to put them back to the global list but if contention is encountered, put the buffer on the percpu list. The length of time a buffer is held on the percpu list is dynamically adjusted based on lock contention. The amount of hold time is rapidly increased and slow ramped down. Signed-off-by: John Johansen --- security/apparmor/lsm.c | 74 ++++++++++++++++++++++++++++++++++++++--- 1 file changed, 69 insertions(+), 5 deletions(-) diff --git a/security/apparmor/lsm.c b/security/apparmor/lsm.c index 25114735bc11..0ab70171bdb6 100644 --- a/security/apparmor/lsm.c +++ b/security/apparmor/lsm.c @@ -49,12 +49,19 @@ union aa_buffer { char buffer[1]; }; +struct aa_local_cache { + unsigned int contention; + unsigned int hold; + struct list_head head; +}; + #define RESERVE_COUNT 2 static int reserve_count = RESERVE_COUNT; static int buffer_count; static LIST_HEAD(aa_global_buffers); static DEFINE_SPINLOCK(aa_buffers_lock); +static DEFINE_PER_CPU(struct aa_local_cache, aa_local_buffers); /* * LSM hook functions @@ -1622,14 +1629,44 @@ static int param_set_mode(const char *val, const struct kernel_param *kp) return 0; } +static void update_contention(struct aa_local_cache *cache) +{ + cache->contention += 3; + if (cache->contention > 9) + cache->contention = 9; + cache->hold += 1 << cache->contention; /* 8, 64, 512 */ +} + char *aa_get_buffer(bool in_atomic) { union aa_buffer *aa_buf; + struct aa_local_cache *cache; bool try_again = true; gfp_t flags = (GFP_KERNEL | __GFP_RETRY_MAYFAIL | __GFP_NOWARN); + /* use per cpu cached buffers first */ + cache = get_cpu_ptr(&aa_local_buffers); + if (!list_empty(&cache->head)) { + aa_buf = list_first_entry(&cache->head, union aa_buffer, list); + list_del(&aa_buf->list); + cache->hold--; + put_cpu_ptr(&aa_local_buffers); + return &aa_buf->buffer[0]; + } + put_cpu_ptr(&aa_local_buffers); + + if (!spin_trylock(&aa_buffers_lock)) { + cache = get_cpu_ptr(&aa_local_buffers); + update_contention(cache); + put_cpu_ptr(&aa_local_buffers); + spin_lock(&aa_buffers_lock); + } else { + cache = get_cpu_ptr(&aa_local_buffers); + if (cache->contention) + cache->contention--; + put_cpu_ptr(&aa_local_buffers); + } retry: - spin_lock(&aa_buffers_lock); if (buffer_count > reserve_count || (in_atomic && !list_empty(&aa_global_buffers))) { aa_buf = list_first_entry(&aa_global_buffers, union aa_buffer, @@ -1655,6 +1692,7 @@ char *aa_get_buffer(bool in_atomic) if (!aa_buf) { if (try_again) { try_again = false; + spin_lock(&aa_buffers_lock); goto retry; } pr_warn_once("AppArmor: Failed to allocate a memory buffer.\n"); @@ -1666,15 +1704,32 @@ char *aa_get_buffer(bool in_atomic) void aa_put_buffer(char *buf) { union aa_buffer *aa_buf; + struct aa_local_cache *cache; if (!buf) return; aa_buf = container_of(buf, union aa_buffer, buffer[0]); - spin_lock(&aa_buffers_lock); - list_add(&aa_buf->list, &aa_global_buffers); - buffer_count++; - spin_unlock(&aa_buffers_lock); + cache = get_cpu_ptr(&aa_local_buffers); + if (!cache->hold) { + put_cpu_ptr(&aa_local_buffers); + if (spin_trylock(&aa_buffers_lock)) { + list_add(&aa_buf->list, &aa_global_buffers); + buffer_count++; + spin_unlock(&aa_buffers_lock); + cache = get_cpu_ptr(&aa_local_buffers); + if (cache->contention) + cache->contention--; + put_cpu_ptr(&aa_local_buffers); + return; + } + cache = get_cpu_ptr(&aa_local_buffers); + update_contention(cache); + } + + /* cache in percpu list */ + list_add(&aa_buf->list, &cache->head); + put_cpu_ptr(&aa_local_buffers); } /* @@ -1716,6 +1771,15 @@ static int __init alloc_buffers(void) union aa_buffer *aa_buf; int i, num; + /* + * per cpu set of cached allocated buffers used to help reduce + * lock contention + */ + for_each_possible_cpu(i) { + per_cpu(aa_local_buffers, i).contention = 0; + per_cpu(aa_local_buffers, i).hold = 0; + INIT_LIST_HEAD(&per_cpu(aa_local_buffers, i).head); + } /* * A function may require two buffers at once. Usually the buffers are * used for a short period of time and are shared. On UP kernel buffers -- 2.34.1