From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-20.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,MENTIONS_GIT_HOSTING,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0DCA8C433B4 for ; Thu, 8 Apr 2021 12:13:56 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id CD71E6115B for ; Thu, 8 Apr 2021 12:13:55 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231592AbhDHMOF (ORCPT ); Thu, 8 Apr 2021 08:14:05 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50198 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231485AbhDHMNz (ORCPT ); Thu, 8 Apr 2021 08:13:55 -0400 Received: from galois.linutronix.de (Galois.linutronix.de [IPv6:2a0a:51c0:0:12e:550::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 74F1BC061764; Thu, 8 Apr 2021 05:13:44 -0700 (PDT) Date: Thu, 08 Apr 2021 12:13:41 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1617884022; h=from:from:sender:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=UCnuNm1hTO8lkFEWJXV+ybB+XTQ7k04SlSSa+8IV+AY=; b=gqrmd1XtIT/jY2+M2pBpBvI7bnrMMerecUa1PA4gaz3ptFYh8Z8u6NtzGVNG5TfF2u55wm /l130lv+T+wT2NvvLYWDU5AWb91JUxsEqVJp8sjtfVWvd1OMblSWz3jt1IpRgqNyXeNIoH 0PzKmTttInorNuEp+euS4Cbbj6yuZzmskiDbpsAQ1d9XvRwpFQudlPiSudDFPFS6NnMcP5 vkEgkeI8K19kl6ltL01JHobm4BXqeOEw7V9q12hgzG6B0fgcKNREQ27CdgKzbjPvOK9709 if88ENCaMkKmd2pgQbNiTaxGGIu66pJndfxSHg2Fk7DF8ROiAIDc06dHvNbQoA== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1617884022; h=from:from:sender:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=UCnuNm1hTO8lkFEWJXV+ybB+XTQ7k04SlSSa+8IV+AY=; b=YgkOkCnTw2/RXyULMI/qVWyzzbEXrOye1d8ezhYwDVx7aoJ3siVUxb/hRVJ3ei2oFTJXkK Y9APbCIT9DC5GZBA== From: "tip-bot2 for Kees Cook" Sender: tip-bot2@linutronix.de Reply-to: linux-kernel@vger.kernel.org To: linux-tip-commits@vger.kernel.org Subject: [tip: x86/entry] init_on_alloc: Optimize static branches Cc: Kees Cook , Thomas Gleixner , Alexander Potapenko , Vlastimil Babka , x86@kernel.org, linux-kernel@vger.kernel.org In-Reply-To: <20210401232347.2791257-3-keescook@chromium.org> References: <20210401232347.2791257-3-keescook@chromium.org> MIME-Version: 1.0 Message-ID: <161788402167.29796.5452552471704996304.tip-bot2@tip-bot2> Robot-ID: Robot-Unsubscribe: Contact to get blacklisted from these emails Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 7bit Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org The following commit has been merged into the x86/entry branch of tip: Commit-ID: 51cba1ebc60df9c4ce034a9f5441169c0d0956c0 Gitweb: https://git.kernel.org/tip/51cba1ebc60df9c4ce034a9f5441169c0d0956c0 Author: Kees Cook AuthorDate: Thu, 01 Apr 2021 16:23:43 -07:00 Committer: Thomas Gleixner CommitterDate: Thu, 08 Apr 2021 14:05:19 +02:00 init_on_alloc: Optimize static branches The state of CONFIG_INIT_ON_ALLOC_DEFAULT_ON (and ...ON_FREE...) did not change the assembly ordering of the static branches: they were always out of line. Use the new jump_label macros to check the CONFIG settings to default to the "expected" state, which slightly optimizes the resulting assembly code. Signed-off-by: Kees Cook Signed-off-by: Thomas Gleixner Reviewed-by: Alexander Potapenko Acked-by: Vlastimil Babka Link: https://lore.kernel.org/r/20210401232347.2791257-3-keescook@chromium.org --- include/linux/mm.h | 10 ++++++---- mm/page_alloc.c | 4 ++-- mm/slab.h | 6 ++++-- 3 files changed, 12 insertions(+), 8 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 8ba4342..616dcaf 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -2904,18 +2904,20 @@ static inline void kernel_poison_pages(struct page *page, int numpages) { } static inline void kernel_unpoison_pages(struct page *page, int numpages) { } #endif -DECLARE_STATIC_KEY_FALSE(init_on_alloc); +DECLARE_STATIC_KEY_MAYBE(CONFIG_INIT_ON_ALLOC_DEFAULT_ON, init_on_alloc); static inline bool want_init_on_alloc(gfp_t flags) { - if (static_branch_unlikely(&init_on_alloc)) + if (static_branch_maybe(CONFIG_INIT_ON_ALLOC_DEFAULT_ON, + &init_on_alloc)) return true; return flags & __GFP_ZERO; } -DECLARE_STATIC_KEY_FALSE(init_on_free); +DECLARE_STATIC_KEY_MAYBE(CONFIG_INIT_ON_FREE_DEFAULT_ON, init_on_free); static inline bool want_init_on_free(void) { - return static_branch_unlikely(&init_on_free); + return static_branch_maybe(CONFIG_INIT_ON_FREE_DEFAULT_ON, + &init_on_free); } extern bool _debug_pagealloc_enabled_early; diff --git a/mm/page_alloc.c b/mm/page_alloc.c index cfc7287..e2f19bf 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -167,10 +167,10 @@ unsigned long totalcma_pages __read_mostly; int percpu_pagelist_fraction; gfp_t gfp_allowed_mask __read_mostly = GFP_BOOT_MASK; -DEFINE_STATIC_KEY_FALSE(init_on_alloc); +DEFINE_STATIC_KEY_MAYBE(CONFIG_INIT_ON_ALLOC_DEFAULT_ON, init_on_alloc); EXPORT_SYMBOL(init_on_alloc); -DEFINE_STATIC_KEY_FALSE(init_on_free); +DEFINE_STATIC_KEY_MAYBE(CONFIG_INIT_ON_FREE_DEFAULT_ON, init_on_free); EXPORT_SYMBOL(init_on_free); static bool _init_on_alloc_enabled_early __read_mostly diff --git a/mm/slab.h b/mm/slab.h index 076582f..774c722 100644 --- a/mm/slab.h +++ b/mm/slab.h @@ -601,7 +601,8 @@ static inline void cache_random_seq_destroy(struct kmem_cache *cachep) { } static inline bool slab_want_init_on_alloc(gfp_t flags, struct kmem_cache *c) { - if (static_branch_unlikely(&init_on_alloc)) { + if (static_branch_maybe(CONFIG_INIT_ON_ALLOC_DEFAULT_ON, + &init_on_alloc)) { if (c->ctor) return false; if (c->flags & (SLAB_TYPESAFE_BY_RCU | SLAB_POISON)) @@ -613,7 +614,8 @@ static inline bool slab_want_init_on_alloc(gfp_t flags, struct kmem_cache *c) static inline bool slab_want_init_on_free(struct kmem_cache *c) { - if (static_branch_unlikely(&init_on_free)) + if (static_branch_maybe(CONFIG_INIT_ON_FREE_DEFAULT_ON, + &init_on_free)) return !(c->ctor || (c->flags & (SLAB_TYPESAFE_BY_RCU | SLAB_POISON))); return false;