From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1754030AbcHWQTR (ORCPT ); Tue, 23 Aug 2016 12:19:17 -0400 Received: from mail-lf0-f67.google.com ([209.85.215.67]:34720 "EHLO mail-lf0-f67.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1753837AbcHWQTP (ORCPT ); Tue, 23 Aug 2016 12:19:15 -0400 Date: Tue, 23 Aug 2016 19:19:21 +0300 From: Alexey Dobriyan To: x86@kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH 1/3] x86: branchless clear_page() Message-ID: <20160823161920.GA2286@p183.telecom.by> References: <20160822205229.GA2638@p183.telecom.by> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20160822205229.GA2638@p183.telecom.by> User-Agent: Mutt/1.5.24 (2015-08-30) Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Apply alternatives at the call site instead of function body. Save branch per clean page. Bonus: tell gcc to not flush whole shebang of registers, just RDI, RAX, RCX. Rename individual clear_page() functions closer to implementation details. Signed-off-by: Alexey Dobriyan --- Export individual functions. arch/x86/include/asm/page_64.h | 16 +++++++++++++++- arch/x86/kernel/x8664_ksyms_64.c | 4 +++- arch/x86/lib/clear_page_64.S | 18 ++++++------------ 3 files changed, 24 insertions(+), 14 deletions(-) --- a/arch/x86/include/asm/page_64.h +++ b/arch/x86/include/asm/page_64.h @@ -4,6 +4,7 @@ #include #ifndef __ASSEMBLY__ +#include /* duplicated to the one in bootmem.h */ extern unsigned long max_pfn; @@ -34,7 +35,20 @@ extern unsigned long __phys_addr_symbol(unsigned long); #define pfn_valid(pfn) ((pfn) < max_pfn) #endif -void clear_page(void *page); +void clear_page_mov(void *page); +void clear_page_rep_stosq(void *page); +void clear_page_rep_stosb(void *page); +static __always_inline void clear_page(void *page) +{ + alternative_call_2( + clear_page_mov, + clear_page_rep_stosq, X86_FEATURE_REP_GOOD, + clear_page_rep_stosb, X86_FEATURE_ERMS, + "=D" (page), + "0" (page) + : "rax", "rcx", "memory" + ); +} void copy_page(void *to, void *from); #endif /* !__ASSEMBLY__ */ --- a/arch/x86/kernel/x8664_ksyms_64.c +++ b/arch/x86/kernel/x8664_ksyms_64.c @@ -41,7 +41,9 @@ EXPORT_SYMBOL(_copy_to_user); EXPORT_SYMBOL_GPL(memcpy_mcsafe); EXPORT_SYMBOL(copy_page); -EXPORT_SYMBOL(clear_page); +EXPORT_SYMBOL(clear_page_mov); +EXPORT_SYMBOL(clear_page_rep_stosq); +EXPORT_SYMBOL(clear_page_rep_stosb); EXPORT_SYMBOL(csum_partial); --- a/arch/x86/lib/clear_page_64.S +++ b/arch/x86/lib/clear_page_64.S @@ -1,6 +1,4 @@ #include -#include -#include /* * Most CPUs support enhanced REP MOVSB/STOSB instructions. It is @@ -13,18 +11,14 @@ * Zero a page. * %rdi - page */ -ENTRY(clear_page) - - ALTERNATIVE_2 "jmp clear_page_orig", "", X86_FEATURE_REP_GOOD, \ - "jmp clear_page_c_e", X86_FEATURE_ERMS - +ENTRY(clear_page_rep_stosq) movl $4096/8,%ecx xorl %eax,%eax rep stosq ret -ENDPROC(clear_page) +ENDPROC(clear_page_rep_stosq) -ENTRY(clear_page_orig) +ENTRY(clear_page_mov) xorl %eax,%eax movl $4096/64,%ecx @@ -44,11 +38,11 @@ ENTRY(clear_page_orig) jnz .Lloop nop ret -ENDPROC(clear_page_orig) +ENDPROC(clear_page_mov) -ENTRY(clear_page_c_e) +ENTRY(clear_page_rep_stosb) movl $4096,%ecx xorl %eax,%eax rep stosb ret -ENDPROC(clear_page_c_e) +ENDPROC(clear_page_rep_stosb)