* [PATCH v4 0/5] LoongArch: Add kernel relocation and KASLR support
@ 2023-02-10 8:47 Youling Tang
2023-02-10 8:47 ` [PATCH v4 1/5] LoongArch: Use la.pcrel instead of la.abs when it's trivially possible Youling Tang
` (4 more replies)
0 siblings, 5 replies; 22+ messages in thread
From: Youling Tang @ 2023-02-10 8:47 UTC (permalink / raw)
To: Huacai Chen, Xi Ruoyao, Jinyang He; +Cc: Xuerui Wang, loongarch, linux-kernel
This patch series to support kernel relocation and KASLR (only 64bit).
Tested the kernel images built with new toolchain (Binutils-2.40 + patched
GCC-12.2) and old toolchain (kernel.org cross toolchain [1]) on a
3A5000-7A2000-EVB.
With CONFIG_RANDOMIZE_BASE=y, the results are:
1. first boot, new toolchain:
$ sudo cat /proc/iomem | grep Kernel
01080000-0189ffff : Kernel code
018a0000-01deb5ff : Kernel data
01deb600-01ef6e9f : Kernel bss
2. second boot, new toolchain:
$ sudo cat /proc/iomem | grep Kernel
012f0000-01b0ffff : Kernel code
01b10000-0205b5ff : Kernel data
0205b600-02166e9f : Kernel bss
3. first boot, old toolchain:
010e0000-018fffff : Kernel code
01900000-01e591ff : Kernel data
01e59200-01f56dcf : Kernel bss
4. second boot, old toolchain:
010b0000-018cffff : Kernel code
018d0000-01e291ff : Kernel data
01e29200-01f26dcf : Kernel bss
Changes from v3:
- JUMP_LINK_ADDR renamed to JUMP_VIRT_ADDR, and use the way of parameter
passing.
- Reimplement kernel relocation, when the link address and load address
are different, realize the effect of adaptive relocation (one of the
usage scenarios is kdump operation).
- Reimplement KASLR.
Changes from v2:
- Correctly fixup pcaddi12i/ori/lu32i.d/lu52i.d sequence generated by
GNU as <= 2.39 for la.pcrel.
Changes from v1 to v2:
- Relocate the handlers instead of using a trampoline, to avoid
performance issue on NUMA systems.
- Fix compiler warnings.
Xi Ruoyao (2):
LoongArch: Use la.pcrel instead of la.abs when it's trivially possible
LoongArch: Use la.pcrel instead of la.abs for exception handlers
Youling Tang (3):
LoongArch: Add JUMP_VIRT_ADDR macro implementation to avoid using
la.abs
LoongArch: Add support for kernel relocation
LoongArch: Add support for kernel address space layout randomization
(KASLR)
arch/loongarch/Kconfig | 38 +++++
arch/loongarch/Makefile | 5 +
arch/loongarch/include/asm/inst.h | 1 +
arch/loongarch/include/asm/setup.h | 6 +-
arch/loongarch/include/asm/stackframe.h | 14 +-
arch/loongarch/include/asm/uaccess.h | 1 -
arch/loongarch/kernel/Makefile | 2 +
arch/loongarch/kernel/entry.S | 2 +-
arch/loongarch/kernel/genex.S | 40 ++++-
arch/loongarch/kernel/head.S | 31 +++-
arch/loongarch/kernel/relocate.c | 215 ++++++++++++++++++++++++
arch/loongarch/kernel/traps.c | 158 ++++++++++++++---
arch/loongarch/kernel/vmlinux.lds.S | 11 +-
arch/loongarch/mm/tlb.c | 23 +--
arch/loongarch/mm/tlbex.S | 72 +++++++-
arch/loongarch/power/suspend_asm.S | 5 +-
16 files changed, 560 insertions(+), 64 deletions(-)
create mode 100644 arch/loongarch/kernel/relocate.c
--
2.37.3
^ permalink raw reply [flat|nested] 22+ messages in thread* [PATCH v4 1/5] LoongArch: Use la.pcrel instead of la.abs when it's trivially possible 2023-02-10 8:47 [PATCH v4 0/5] LoongArch: Add kernel relocation and KASLR support Youling Tang @ 2023-02-10 8:47 ` Youling Tang 2023-02-10 9:04 ` Huacai Chen 2023-02-10 8:47 ` [PATCH v4 2/5] LoongArch: Use la.pcrel instead of la.abs for exception handlers Youling Tang ` (3 subsequent siblings) 4 siblings, 1 reply; 22+ messages in thread From: Youling Tang @ 2023-02-10 8:47 UTC (permalink / raw) To: Huacai Chen, Xi Ruoyao, Jinyang He; +Cc: Xuerui Wang, loongarch, linux-kernel From: Xi Ruoyao <xry111@xry111.site> Let's start to kill la.abs inpreparation for the subsequent support of the PIE kernel. Signed-off-by: Xi Ruoyao <xry111@xry111.site> --- arch/loongarch/include/asm/stackframe.h | 2 +- arch/loongarch/include/asm/uaccess.h | 1 - arch/loongarch/kernel/entry.S | 2 +- arch/loongarch/kernel/head.S | 2 +- arch/loongarch/mm/tlbex.S | 3 +-- 5 files changed, 4 insertions(+), 6 deletions(-) diff --git a/arch/loongarch/include/asm/stackframe.h b/arch/loongarch/include/asm/stackframe.h index 4ca953062b5b..7deb043ce387 100644 --- a/arch/loongarch/include/asm/stackframe.h +++ b/arch/loongarch/include/asm/stackframe.h @@ -90,7 +90,7 @@ .endm .macro set_saved_sp stackp temp temp2 - la.abs \temp, kernelsp + la.pcrel \temp, kernelsp #ifdef CONFIG_SMP LONG_ADD \temp, \temp, u0 #endif diff --git a/arch/loongarch/include/asm/uaccess.h b/arch/loongarch/include/asm/uaccess.h index 255899d4a7c3..0d22991ae430 100644 --- a/arch/loongarch/include/asm/uaccess.h +++ b/arch/loongarch/include/asm/uaccess.h @@ -22,7 +22,6 @@ extern u64 __ua_limit; #define __UA_ADDR ".dword" -#define __UA_LA "la.abs" #define __UA_LIMIT __ua_limit /* diff --git a/arch/loongarch/kernel/entry.S b/arch/loongarch/kernel/entry.S index d53b631c9022..2566977f2f68 100644 --- a/arch/loongarch/kernel/entry.S +++ b/arch/loongarch/kernel/entry.S @@ -20,7 +20,7 @@ .align 5 SYM_FUNC_START(handle_syscall) csrrd t0, PERCPU_BASE_KS - la.abs t1, kernelsp + la.pcrel t1, kernelsp add.d t1, t1, t0 move t2, sp ld.d sp, t1, 0 diff --git a/arch/loongarch/kernel/head.S b/arch/loongarch/kernel/head.S index 57bada6b4e93..aa6181714ec3 100644 --- a/arch/loongarch/kernel/head.S +++ b/arch/loongarch/kernel/head.S @@ -117,7 +117,7 @@ SYM_CODE_START(smpboot_entry) li.w t0, 0x00 # FPE=0, SXE=0, ASXE=0, BTE=0 csrwr t0, LOONGARCH_CSR_EUEN - la.abs t0, cpuboot_data + la.pcrel t0, cpuboot_data ld.d sp, t0, CPU_BOOT_STACK ld.d tp, t0, CPU_BOOT_TINFO diff --git a/arch/loongarch/mm/tlbex.S b/arch/loongarch/mm/tlbex.S index 58781c6e4191..3dd2a9615cd9 100644 --- a/arch/loongarch/mm/tlbex.S +++ b/arch/loongarch/mm/tlbex.S @@ -24,8 +24,7 @@ move a0, sp REG_S a2, sp, PT_BVADDR li.w a1, \write - la.abs t0, do_page_fault - jirl ra, t0, 0 + bl do_page_fault RESTORE_ALL_AND_RET SYM_FUNC_END(tlb_do_page_fault_\write) .endm -- 2.37.3 ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [PATCH v4 1/5] LoongArch: Use la.pcrel instead of la.abs when it's trivially possible 2023-02-10 8:47 ` [PATCH v4 1/5] LoongArch: Use la.pcrel instead of la.abs when it's trivially possible Youling Tang @ 2023-02-10 9:04 ` Huacai Chen 0 siblings, 0 replies; 22+ messages in thread From: Huacai Chen @ 2023-02-10 9:04 UTC (permalink / raw) To: Youling Tang; +Cc: Xi Ruoyao, Jinyang He, Xuerui Wang, loongarch, linux-kernel Hi, Youling, On Fri, Feb 10, 2023 at 4:47 PM Youling Tang <tangyouling@loongson.cn> wrote: > > From: Xi Ruoyao <xry111@xry111.site> > > Let's start to kill la.abs inpreparation for the subsequent support of the > PIE kernel. > > Signed-off-by: Xi Ruoyao <xry111@xry111.site> > --- > arch/loongarch/include/asm/stackframe.h | 2 +- > arch/loongarch/include/asm/uaccess.h | 1 - > arch/loongarch/kernel/entry.S | 2 +- > arch/loongarch/kernel/head.S | 2 +- > arch/loongarch/mm/tlbex.S | 3 +-- > 5 files changed, 4 insertions(+), 6 deletions(-) > > diff --git a/arch/loongarch/include/asm/stackframe.h b/arch/loongarch/include/asm/stackframe.h > index 4ca953062b5b..7deb043ce387 100644 > --- a/arch/loongarch/include/asm/stackframe.h > +++ b/arch/loongarch/include/asm/stackframe.h > @@ -90,7 +90,7 @@ > .endm > > .macro set_saved_sp stackp temp temp2 > - la.abs \temp, kernelsp > + la.pcrel \temp, kernelsp > #ifdef CONFIG_SMP > LONG_ADD \temp, \temp, u0 > #endif > diff --git a/arch/loongarch/include/asm/uaccess.h b/arch/loongarch/include/asm/uaccess.h > index 255899d4a7c3..0d22991ae430 100644 > --- a/arch/loongarch/include/asm/uaccess.h > +++ b/arch/loongarch/include/asm/uaccess.h > @@ -22,7 +22,6 @@ > extern u64 __ua_limit; > > #define __UA_ADDR ".dword" > -#define __UA_LA "la.abs" > #define __UA_LIMIT __ua_limit > > /* > diff --git a/arch/loongarch/kernel/entry.S b/arch/loongarch/kernel/entry.S > index d53b631c9022..2566977f2f68 100644 > --- a/arch/loongarch/kernel/entry.S > +++ b/arch/loongarch/kernel/entry.S > @@ -20,7 +20,7 @@ > .align 5 > SYM_FUNC_START(handle_syscall) > csrrd t0, PERCPU_BASE_KS > - la.abs t1, kernelsp > + la.pcrel t1, kernelsp Retab the whole function to align the first parameter, please. Huacai > add.d t1, t1, t0 > move t2, sp > ld.d sp, t1, 0 > diff --git a/arch/loongarch/kernel/head.S b/arch/loongarch/kernel/head.S > index 57bada6b4e93..aa6181714ec3 100644 > --- a/arch/loongarch/kernel/head.S > +++ b/arch/loongarch/kernel/head.S > @@ -117,7 +117,7 @@ SYM_CODE_START(smpboot_entry) > li.w t0, 0x00 # FPE=0, SXE=0, ASXE=0, BTE=0 > csrwr t0, LOONGARCH_CSR_EUEN > > - la.abs t0, cpuboot_data > + la.pcrel t0, cpuboot_data > ld.d sp, t0, CPU_BOOT_STACK > ld.d tp, t0, CPU_BOOT_TINFO > > diff --git a/arch/loongarch/mm/tlbex.S b/arch/loongarch/mm/tlbex.S > index 58781c6e4191..3dd2a9615cd9 100644 > --- a/arch/loongarch/mm/tlbex.S > +++ b/arch/loongarch/mm/tlbex.S > @@ -24,8 +24,7 @@ > move a0, sp > REG_S a2, sp, PT_BVADDR > li.w a1, \write > - la.abs t0, do_page_fault > - jirl ra, t0, 0 > + bl do_page_fault > RESTORE_ALL_AND_RET > SYM_FUNC_END(tlb_do_page_fault_\write) > .endm > -- > 2.37.3 > > ^ permalink raw reply [flat|nested] 22+ messages in thread
* [PATCH v4 2/5] LoongArch: Use la.pcrel instead of la.abs for exception handlers 2023-02-10 8:47 [PATCH v4 0/5] LoongArch: Add kernel relocation and KASLR support Youling Tang 2023-02-10 8:47 ` [PATCH v4 1/5] LoongArch: Use la.pcrel instead of la.abs when it's trivially possible Youling Tang @ 2023-02-10 8:47 ` Youling Tang 2023-02-10 9:09 ` Huacai Chen 2023-02-10 8:47 ` [PATCH v4 3/5] LoongArch: Add JUMP_VIRT_ADDR macro implementation to avoid using la.abs Youling Tang ` (2 subsequent siblings) 4 siblings, 1 reply; 22+ messages in thread From: Youling Tang @ 2023-02-10 8:47 UTC (permalink / raw) To: Huacai Chen, Xi Ruoyao, Jinyang He; +Cc: Xuerui Wang, loongarch, linux-kernel From: Xi Ruoyao <xry111@xry111.site> It's needed to build the kernel as a PIE, or the linker will complain. For the consideration about performance, we copy the exception handlers to a dedicated 64 KB area for each CPU. So, the PC-relative offset calculated at link time will be incorrect and we need to relocate the exception handlers after copying them. For the simplicity, we don't use the ELF R_LARCH_* relocations, but code an relocation entry as simply (offset_in_the_handler, symbol_addr). For each exception handler, we also code the number of relocation entries. Then we can use the relocation information to fix up the handlers after copying them. Signed-off-by: Xi Ruoyao <xry111@xry111.site> --- arch/loongarch/include/asm/inst.h | 1 + arch/loongarch/include/asm/setup.h | 6 +- arch/loongarch/include/asm/stackframe.h | 3 +- arch/loongarch/kernel/genex.S | 40 +++++- arch/loongarch/kernel/traps.c | 158 ++++++++++++++++++++---- arch/loongarch/mm/tlb.c | 23 ++-- arch/loongarch/mm/tlbex.S | 69 +++++++++-- 7 files changed, 255 insertions(+), 45 deletions(-) diff --git a/arch/loongarch/include/asm/inst.h b/arch/loongarch/include/asm/inst.h index 7eedd83fd0d7..426054518a3d 100644 --- a/arch/loongarch/include/asm/inst.h +++ b/arch/loongarch/include/asm/inst.h @@ -32,6 +32,7 @@ enum reg1i20_op { lu12iw_op = 0x0a, lu32id_op = 0x0b, pcaddi_op = 0x0c, + pcalau12i_op = 0x0d, pcaddu12i_op = 0x0e, pcaddu18i_op = 0x0f, }; diff --git a/arch/loongarch/include/asm/setup.h b/arch/loongarch/include/asm/setup.h index 72ead58039f3..f0a2b34365f1 100644 --- a/arch/loongarch/include/asm/setup.h +++ b/arch/loongarch/include/asm/setup.h @@ -11,6 +11,9 @@ #define VECSIZE 0x200 +struct handler_reloc; + +extern struct handler_reloc *eentry_reloc[]; extern unsigned long eentry; extern unsigned long tlbrentry; extern char init_command_line[COMMAND_LINE_SIZE]; @@ -18,7 +21,8 @@ extern void tlb_init(int cpu); extern void cpu_cache_init(void); extern void cache_error_setup(void); extern void per_cpu_trap_init(int cpu); -extern void set_handler(unsigned long offset, void *addr, unsigned long len); +extern void set_handler(unsigned long exccode, void *addr); extern void set_merr_handler(unsigned long offset, void *addr, unsigned long len); +extern void reloc_handler(unsigned long handler, struct handler_reloc *rel); #endif /* __SETUP_H */ diff --git a/arch/loongarch/include/asm/stackframe.h b/arch/loongarch/include/asm/stackframe.h index 7deb043ce387..bbec1e56b61b 100644 --- a/arch/loongarch/include/asm/stackframe.h +++ b/arch/loongarch/include/asm/stackframe.h @@ -77,7 +77,8 @@ * new value in sp. */ .macro get_saved_sp docfi=0 - la.abs t1, kernelsp + /* The label is used for generating reloc tables for handlers */ +514: la.pcrel t1, t0, kernelsp #ifdef CONFIG_SMP csrrd t0, PERCPU_BASE_KS LONG_ADD t1, t1, t0 diff --git a/arch/loongarch/kernel/genex.S b/arch/loongarch/kernel/genex.S index 7e5c293ed89f..005a10fe5a50 100644 --- a/arch/loongarch/kernel/genex.S +++ b/arch/loongarch/kernel/genex.S @@ -34,7 +34,7 @@ SYM_FUNC_END(__arch_cpu_idle) SYM_FUNC_START(handle_vint) BACKUP_T0T1 SAVE_ALL - la.abs t1, __arch_cpu_idle +0: la.pcrel t1, t2, __arch_cpu_idle LONG_L t0, sp, PT_ERA /* 32 byte rollback region */ ori t0, t0, 0x1f @@ -43,11 +43,25 @@ SYM_FUNC_START(handle_vint) LONG_S t0, sp, PT_ERA 1: move a0, sp move a1, sp - la.abs t0, do_vint +2: la.pcrel t0, t2, do_vint jirl ra, t0, 0 RESTORE_ALL_AND_RET SYM_FUNC_END(handle_vint) +SYM_DATA_START(rel_handle_vint) +LONG 3 + +LONG 514b - handle_vint +LONG kernelsp + +LONG 0b - handle_vint +LONG __arch_cpu_idle + +LONG 2b - handle_vint +LONG do_vint + +SYM_DATA_END(rel_handle_vint) + SYM_FUNC_START(except_vec_cex) b cache_parity_error SYM_FUNC_END(except_vec_cex) @@ -72,12 +86,24 @@ SYM_FUNC_END(except_vec_cex) SAVE_ALL build_prep_\prep move a0, sp - la.abs t0, do_\handler + 667: + la.pcrel t0, t1, do_\handler jirl ra, t0, 0 668: RESTORE_ALL_AND_RET SYM_FUNC_END(handle_\exception) SYM_DATA(unwind_hint_\exception, .word 668b - 666b) + + SYM_DATA_START(rel_handle_\exception) + LONG 2 + + LONG 514b - 666b + LONG kernelsp + + LONG 667b - 666b + LONG do_\handler + + SYM_DATA_END(rel_handle_\exception) .endm BUILD_HANDLER ade ade badv @@ -93,6 +119,12 @@ SYM_FUNC_END(except_vec_cex) BUILD_HANDLER reserved reserved none /* others */ SYM_FUNC_START(handle_sys) - la.abs t0, handle_syscall + la.pcrel t0, t1, handle_syscall jr t0 SYM_FUNC_END(handle_sys) + +SYM_DATA_START(rel_handle_sys) +LONG 1 +LONG 0 +LONG handle_syscall +SYM_DATA_END(rel_handle_sys) diff --git a/arch/loongarch/kernel/traps.c b/arch/loongarch/kernel/traps.c index c38a146a973b..7e073854f493 100644 --- a/arch/loongarch/kernel/traps.c +++ b/arch/loongarch/kernel/traps.c @@ -62,6 +62,127 @@ extern asmlinkage void handle_reserved(void); extern asmlinkage void handle_watch(void); extern asmlinkage void handle_vint(void); +struct handler_reloc_entry { + unsigned long offset; + unsigned long sym; +}; + +struct handler_reloc { + unsigned long cnt; + struct handler_reloc_entry entries[]; +}; + +extern struct handler_reloc rel_handle_tlb_load; +extern struct handler_reloc rel_handle_tlb_store; +extern struct handler_reloc rel_handle_tlb_modify; +extern struct handler_reloc rel_handle_tlb_protect; +extern struct handler_reloc rel_handle_ade; +extern struct handler_reloc rel_handle_ale; +extern struct handler_reloc rel_handle_sys; +extern struct handler_reloc rel_handle_bp; +extern struct handler_reloc rel_handle_ri; +extern struct handler_reloc rel_handle_fpu; +extern struct handler_reloc rel_handle_lsx; +extern struct handler_reloc rel_handle_lasx; +extern struct handler_reloc rel_handle_fpe; +extern struct handler_reloc rel_handle_lbt; +extern struct handler_reloc rel_handle_watch; +extern struct handler_reloc rel_handle_reserved; +extern struct handler_reloc rel_handle_vint; + +struct handler_reloc *eentry_reloc[128] = { + [0] = NULL, /* merr handler */ + [EXCCODE_TLBL] = &rel_handle_tlb_load, + [EXCCODE_TLBS] = &rel_handle_tlb_store, + [EXCCODE_TLBI] = &rel_handle_tlb_load, + [EXCCODE_TLBM] = &rel_handle_tlb_modify, + [EXCCODE_TLBNR] = &rel_handle_tlb_protect, + [EXCCODE_TLBNX] = &rel_handle_tlb_protect, + [EXCCODE_TLBPE] = &rel_handle_tlb_protect, + [EXCCODE_ADE] = &rel_handle_ade, + [EXCCODE_ALE] = &rel_handle_ale, + [EXCCODE_SYS] = &rel_handle_sys, + [EXCCODE_BP] = &rel_handle_bp, + [EXCCODE_INE] = &rel_handle_ri, + [EXCCODE_IPE] = &rel_handle_ri, + [EXCCODE_FPDIS] = &rel_handle_fpu, + [EXCCODE_LSXDIS] = &rel_handle_lsx, + [EXCCODE_LASXDIS] = &rel_handle_lasx, + [EXCCODE_FPE] = &rel_handle_fpe, + [EXCCODE_BTDIS] = &rel_handle_lbt, + [EXCCODE_WATCH] = &rel_handle_watch, + [(EXCCODE_WATCH + 1) ... (EXCCODE_INT_START - 1)] = &rel_handle_reserved, + [EXCCODE_INT_START ... (EXCCODE_INT_END - 1)] = &rel_handle_vint, +}; + +void reloc_handler(unsigned long handler, struct handler_reloc *rel) +{ + if (!rel) + return; + + for (unsigned long i = 0; i < rel->cnt; i++) { + unsigned long pc = handler + rel->entries[i].offset; + union loongarch_instruction *insn = + (union loongarch_instruction *)pc; + u32 imm[4]; + unsigned long v = rel->entries[i].sym; + + /* GNU as >= 2.40 uses pcalau12i for la.pcrel, but GNU ld <= 2.39 + * uses pcaddu12i. + */ + if (insn->reg1i20_format.opcode == pcalau12i_op) { + /* Use s32 deliberately for sign extension. */ + s32 offset_hi20 = ((v + 0x800) & ~0xfff) - + (pc & ~0xfff); + unsigned long anchor = (pc & ~0xfff) + offset_hi20; + unsigned long offset_rem = v - anchor; + + imm[0] = (offset_hi20 >> 12) & 0xfffff; + imm[1] = v & 0xfff; + imm[2] = (offset_rem >> 32) & 0xfffff; + imm[3] = offset_rem >> 52; + } else if (insn->reg1i20_format.opcode == pcaddu12i_op) { + /* Use s32 deliberately for sign extension. */ + s32 offset_lo = v - pc; + unsigned long offset_hi = v - pc - offset_lo; + + imm[0] = (offset_lo >> 12) & 0xfffff; + imm[1] = offset_lo & 0xfff; + imm[2] = (offset_hi >> 32) & 0xfffff; + imm[3] = offset_hi >> 52; + } else + panic("Cannot fixup la.pcrel for exception handler at %lu: unexpected instruction %d!", + pc, insn->word); + + insn[0].reg1i20_format.immediate = imm[0]; + insn[1].reg2i12_format.immediate = imm[1]; + insn[2].reg1i20_format.immediate = imm[2]; + insn[3].reg2i12_format.immediate = imm[3]; + } +} + +/* Install CPU exception handler */ +static void do_set_handler(unsigned long exccode, void *addr, + struct handler_reloc *rel) +{ + unsigned long dest_addr = eentry + exccode * VECSIZE; + + memcpy((void *)dest_addr, addr, VECSIZE); + reloc_handler(dest_addr, rel); + local_flush_icache_range(dest_addr, dest_addr + VECSIZE); +} + +/* Install CPU exception handler, with the reloc table from eentry_reloc */ +void set_handler(unsigned long exccode, void *addr) +{ + do_set_handler(exccode, addr, eentry_reloc[exccode]); +} + +static void set_handler_reserved(unsigned long exccode) +{ + do_set_handler(exccode, handle_reserved, &rel_handle_reserved); +} + static void show_backtrace(struct task_struct *task, const struct pt_regs *regs, const char *loglvl, bool user) { @@ -704,19 +825,12 @@ void per_cpu_trap_init(int cpu) /* Initialise exception handlers */ if (cpu == 0) for (i = 0; i < 64; i++) - set_handler(i * VECSIZE, handle_reserved, VECSIZE); + set_handler_reserved(i); tlb_init(cpu); cpu_cache_init(); } -/* Install CPU exception handler */ -void set_handler(unsigned long offset, void *addr, unsigned long size) -{ - memcpy((void *)(eentry + offset), addr, size); - local_flush_icache_range(eentry + offset, eentry + offset + size); -} - static const char panic_null_cerr[] = "Trying to set NULL cache error exception handler\n"; @@ -741,20 +855,20 @@ void __init trap_init(void) /* Set interrupt vector handler */ for (i = EXCCODE_INT_START; i < EXCCODE_INT_END; i++) - set_handler(i * VECSIZE, handle_vint, VECSIZE); - - set_handler(EXCCODE_ADE * VECSIZE, handle_ade, VECSIZE); - set_handler(EXCCODE_ALE * VECSIZE, handle_ale, VECSIZE); - set_handler(EXCCODE_SYS * VECSIZE, handle_sys, VECSIZE); - set_handler(EXCCODE_BP * VECSIZE, handle_bp, VECSIZE); - set_handler(EXCCODE_INE * VECSIZE, handle_ri, VECSIZE); - set_handler(EXCCODE_IPE * VECSIZE, handle_ri, VECSIZE); - set_handler(EXCCODE_FPDIS * VECSIZE, handle_fpu, VECSIZE); - set_handler(EXCCODE_LSXDIS * VECSIZE, handle_lsx, VECSIZE); - set_handler(EXCCODE_LASXDIS * VECSIZE, handle_lasx, VECSIZE); - set_handler(EXCCODE_FPE * VECSIZE, handle_fpe, VECSIZE); - set_handler(EXCCODE_BTDIS * VECSIZE, handle_lbt, VECSIZE); - set_handler(EXCCODE_WATCH * VECSIZE, handle_watch, VECSIZE); + set_handler(i, handle_vint); + + set_handler(EXCCODE_ADE, handle_ade); + set_handler(EXCCODE_ALE, handle_ale); + set_handler(EXCCODE_SYS, handle_sys); + set_handler(EXCCODE_BP, handle_bp); + set_handler(EXCCODE_INE, handle_ri); + set_handler(EXCCODE_IPE, handle_ri); + set_handler(EXCCODE_FPDIS, handle_fpu); + set_handler(EXCCODE_LSXDIS, handle_lsx); + set_handler(EXCCODE_LASXDIS, handle_lasx); + set_handler(EXCCODE_FPE, handle_fpe); + set_handler(EXCCODE_BTDIS, handle_lbt); + set_handler(EXCCODE_WATCH, handle_watch); cache_error_setup(); diff --git a/arch/loongarch/mm/tlb.c b/arch/loongarch/mm/tlb.c index 8bad6b0cff59..6f70aab7202a 100644 --- a/arch/loongarch/mm/tlb.c +++ b/arch/loongarch/mm/tlb.c @@ -253,7 +253,6 @@ static void output_pgtable_bits_defines(void) #ifdef CONFIG_NUMA unsigned long pcpu_handlers[NR_CPUS]; #endif -extern long exception_handlers[VECSIZE * 128 / sizeof(long)]; void setup_tlb_handler(int cpu) { @@ -264,19 +263,20 @@ void setup_tlb_handler(int cpu) if (cpu == 0) { memcpy((void *)tlbrentry, handle_tlb_refill, 0x80); local_flush_icache_range(tlbrentry, tlbrentry + 0x80); - set_handler(EXCCODE_TLBI * VECSIZE, handle_tlb_load, VECSIZE); - set_handler(EXCCODE_TLBL * VECSIZE, handle_tlb_load, VECSIZE); - set_handler(EXCCODE_TLBS * VECSIZE, handle_tlb_store, VECSIZE); - set_handler(EXCCODE_TLBM * VECSIZE, handle_tlb_modify, VECSIZE); - set_handler(EXCCODE_TLBNR * VECSIZE, handle_tlb_protect, VECSIZE); - set_handler(EXCCODE_TLBNX * VECSIZE, handle_tlb_protect, VECSIZE); - set_handler(EXCCODE_TLBPE * VECSIZE, handle_tlb_protect, VECSIZE); + set_handler(EXCCODE_TLBI, handle_tlb_load); + set_handler(EXCCODE_TLBL, handle_tlb_load); + set_handler(EXCCODE_TLBS, handle_tlb_store); + set_handler(EXCCODE_TLBM, handle_tlb_modify); + set_handler(EXCCODE_TLBNR, handle_tlb_protect); + set_handler(EXCCODE_TLBNX, handle_tlb_protect); + set_handler(EXCCODE_TLBPE, handle_tlb_protect); } #ifdef CONFIG_NUMA else { void *addr; + unsigned long addr_ul; struct page *page; - const int vec_sz = sizeof(exception_handlers); + const int vec_sz = VECSIZE * 128; if (pcpu_handlers[cpu]) return; @@ -286,8 +286,11 @@ void setup_tlb_handler(int cpu) return; addr = page_address(page); + addr_ul = (unsigned long)addr; pcpu_handlers[cpu] = (unsigned long)addr; - memcpy((void *)addr, (void *)eentry, vec_sz); + memcpy(addr, (void *)eentry, vec_sz); + for (unsigned long i = 0; i < 128; i++) + reloc_handler(addr_ul + i * VECSIZE, eentry_reloc[i]); local_flush_icache_range((unsigned long)addr, (unsigned long)addr + vec_sz); csr_write64(pcpu_handlers[cpu], LOONGARCH_CSR_EENTRY); csr_write64(pcpu_handlers[cpu], LOONGARCH_CSR_MERRENTRY); diff --git a/arch/loongarch/mm/tlbex.S b/arch/loongarch/mm/tlbex.S index 3dd2a9615cd9..044c2190771a 100644 --- a/arch/loongarch/mm/tlbex.S +++ b/arch/loongarch/mm/tlbex.S @@ -39,11 +39,21 @@ SYM_FUNC_START(handle_tlb_protect) move a1, zero csrrd a2, LOONGARCH_CSR_BADV REG_S a2, sp, PT_BVADDR - la.abs t0, do_page_fault +1: la.pcrel t0, t1, do_page_fault jirl ra, t0, 0 RESTORE_ALL_AND_RET SYM_FUNC_END(handle_tlb_protect) +SYM_DATA_START(rel_handle_tlb_protect) + LONG 2 + + LONG 514b - handle_tlb_protect + LONG kernelsp + + LONG 1b - handle_tlb_protect + LONG do_page_fault +SYM_DATA_END(rel_handle_tlb_protect) + SYM_FUNC_START(handle_tlb_load) csrwr t0, EXCEPTION_KS0 csrwr t1, EXCEPTION_KS1 @@ -115,7 +125,8 @@ smp_pgtable_change_load: #ifdef CONFIG_64BIT vmalloc_load: - la.abs t1, swapper_pg_dir + /* The first insn of vmalloc_done_load overwrites ra */ +1: la.pcrel t1, ra, swapper_pg_dir b vmalloc_done_load #endif @@ -186,10 +197,24 @@ tlb_huge_update_load: nopage_tlb_load: dbar 0 csrrd ra, EXCEPTION_KS2 - la.abs t0, tlb_do_page_fault_0 +2: la.pcrel t0, t1, tlb_do_page_fault_0 jr t0 SYM_FUNC_END(handle_tlb_load) +SYM_DATA_START(rel_handle_tlb_load) +#ifdef CONFIG_64BIT + LONG 2 + + LONG 1b - handle_tlb_load + LONG swapper_pg_dir +#else + LONG 1 +#endif + + LONG 2b - handle_tlb_load + LONG tlb_do_page_fault_0 +SYM_DATA_END(rel_handle_tlb_load) + SYM_FUNC_START(handle_tlb_store) csrwr t0, EXCEPTION_KS0 csrwr t1, EXCEPTION_KS1 @@ -262,7 +287,8 @@ smp_pgtable_change_store: #ifdef CONFIG_64BIT vmalloc_store: - la.abs t1, swapper_pg_dir + /* The first insn of vmalloc_done_store overwrites ra */ +1: la.pcrel t1, ra, swapper_pg_dir b vmalloc_done_store #endif @@ -335,10 +361,24 @@ tlb_huge_update_store: nopage_tlb_store: dbar 0 csrrd ra, EXCEPTION_KS2 - la.abs t0, tlb_do_page_fault_1 +2: la.pcrel t0, t1, tlb_do_page_fault_1 jr t0 SYM_FUNC_END(handle_tlb_store) +SYM_DATA_START(rel_handle_tlb_store) +#ifdef CONFIG_64BIT + LONG 2 + + LONG 1b - handle_tlb_store + LONG swapper_pg_dir +#else + LONG 1 +#endif + + LONG 2b - handle_tlb_store + LONG tlb_do_page_fault_1 +SYM_DATA_END(rel_handle_tlb_store) + SYM_FUNC_START(handle_tlb_modify) csrwr t0, EXCEPTION_KS0 csrwr t1, EXCEPTION_KS1 @@ -410,7 +450,8 @@ smp_pgtable_change_modify: #ifdef CONFIG_64BIT vmalloc_modify: - la.abs t1, swapper_pg_dir + /* The first insn of vmalloc_done_modify overwrites ra */ +1: la.pcrel t1, ra, swapper_pg_dir b vmalloc_done_modify #endif @@ -482,10 +523,24 @@ tlb_huge_update_modify: nopage_tlb_modify: dbar 0 csrrd ra, EXCEPTION_KS2 - la.abs t0, tlb_do_page_fault_1 +2: la.pcrel t0, t1, tlb_do_page_fault_1 jr t0 SYM_FUNC_END(handle_tlb_modify) +SYM_DATA_START(rel_handle_tlb_modify) +#ifdef CONFIG_64BIT + LONG 2 + + LONG 1b - handle_tlb_modify + LONG swapper_pg_dir +#else + LONG 1 +#endif + + LONG 2b - handle_tlb_modify + LONG tlb_do_page_fault_1 +SYM_DATA_END(rel_handle_tlb_modify) + SYM_FUNC_START(handle_tlb_refill) csrwr t0, LOONGARCH_CSR_TLBRSAVE csrrd t0, LOONGARCH_CSR_PGD -- 2.37.3 ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [PATCH v4 2/5] LoongArch: Use la.pcrel instead of la.abs for exception handlers 2023-02-10 8:47 ` [PATCH v4 2/5] LoongArch: Use la.pcrel instead of la.abs for exception handlers Youling Tang @ 2023-02-10 9:09 ` Huacai Chen 2023-02-10 9:18 ` Youling Tang 0 siblings, 1 reply; 22+ messages in thread From: Huacai Chen @ 2023-02-10 9:09 UTC (permalink / raw) To: Youling Tang; +Cc: Xi Ruoyao, Jinyang He, Xuerui Wang, loongarch, linux-kernel Hi, Youling and Ruoyao, Thank you very much for implementing the per-node exceptions. But I want to know if the per-node solution is really worthy for a PIE kernel. So, could you please test the performance? Maybe we can reduce the complexity if we give up the per-node solution. Huacai On Fri, Feb 10, 2023 at 4:47 PM Youling Tang <tangyouling@loongson.cn> wrote: > > From: Xi Ruoyao <xry111@xry111.site> > > It's needed to build the kernel as a PIE, or the linker will complain. > > For the consideration about performance, we copy the exception handlers > to a dedicated 64 KB area for each CPU. So, the PC-relative offset > calculated at link time will be incorrect and we need to relocate the > exception handlers after copying them. > > For the simplicity, we don't use the ELF R_LARCH_* relocations, but code > an relocation entry as simply (offset_in_the_handler, symbol_addr). For > each exception handler, we also code the number of relocation entries. > Then we can use the relocation information to fix up the handlers after > copying them. > > Signed-off-by: Xi Ruoyao <xry111@xry111.site> > --- > arch/loongarch/include/asm/inst.h | 1 + > arch/loongarch/include/asm/setup.h | 6 +- > arch/loongarch/include/asm/stackframe.h | 3 +- > arch/loongarch/kernel/genex.S | 40 +++++- > arch/loongarch/kernel/traps.c | 158 ++++++++++++++++++++---- > arch/loongarch/mm/tlb.c | 23 ++-- > arch/loongarch/mm/tlbex.S | 69 +++++++++-- > 7 files changed, 255 insertions(+), 45 deletions(-) > > diff --git a/arch/loongarch/include/asm/inst.h b/arch/loongarch/include/asm/inst.h > index 7eedd83fd0d7..426054518a3d 100644 > --- a/arch/loongarch/include/asm/inst.h > +++ b/arch/loongarch/include/asm/inst.h > @@ -32,6 +32,7 @@ enum reg1i20_op { > lu12iw_op = 0x0a, > lu32id_op = 0x0b, > pcaddi_op = 0x0c, > + pcalau12i_op = 0x0d, > pcaddu12i_op = 0x0e, > pcaddu18i_op = 0x0f, > }; > diff --git a/arch/loongarch/include/asm/setup.h b/arch/loongarch/include/asm/setup.h > index 72ead58039f3..f0a2b34365f1 100644 > --- a/arch/loongarch/include/asm/setup.h > +++ b/arch/loongarch/include/asm/setup.h > @@ -11,6 +11,9 @@ > > #define VECSIZE 0x200 > > +struct handler_reloc; > + > +extern struct handler_reloc *eentry_reloc[]; > extern unsigned long eentry; > extern unsigned long tlbrentry; > extern char init_command_line[COMMAND_LINE_SIZE]; > @@ -18,7 +21,8 @@ extern void tlb_init(int cpu); > extern void cpu_cache_init(void); > extern void cache_error_setup(void); > extern void per_cpu_trap_init(int cpu); > -extern void set_handler(unsigned long offset, void *addr, unsigned long len); > +extern void set_handler(unsigned long exccode, void *addr); > extern void set_merr_handler(unsigned long offset, void *addr, unsigned long len); > +extern void reloc_handler(unsigned long handler, struct handler_reloc *rel); > > #endif /* __SETUP_H */ > diff --git a/arch/loongarch/include/asm/stackframe.h b/arch/loongarch/include/asm/stackframe.h > index 7deb043ce387..bbec1e56b61b 100644 > --- a/arch/loongarch/include/asm/stackframe.h > +++ b/arch/loongarch/include/asm/stackframe.h > @@ -77,7 +77,8 @@ > * new value in sp. > */ > .macro get_saved_sp docfi=0 > - la.abs t1, kernelsp > + /* The label is used for generating reloc tables for handlers */ > +514: la.pcrel t1, t0, kernelsp > #ifdef CONFIG_SMP > csrrd t0, PERCPU_BASE_KS > LONG_ADD t1, t1, t0 > diff --git a/arch/loongarch/kernel/genex.S b/arch/loongarch/kernel/genex.S > index 7e5c293ed89f..005a10fe5a50 100644 > --- a/arch/loongarch/kernel/genex.S > +++ b/arch/loongarch/kernel/genex.S > @@ -34,7 +34,7 @@ SYM_FUNC_END(__arch_cpu_idle) > SYM_FUNC_START(handle_vint) > BACKUP_T0T1 > SAVE_ALL > - la.abs t1, __arch_cpu_idle > +0: la.pcrel t1, t2, __arch_cpu_idle > LONG_L t0, sp, PT_ERA > /* 32 byte rollback region */ > ori t0, t0, 0x1f > @@ -43,11 +43,25 @@ SYM_FUNC_START(handle_vint) > LONG_S t0, sp, PT_ERA > 1: move a0, sp > move a1, sp > - la.abs t0, do_vint > +2: la.pcrel t0, t2, do_vint > jirl ra, t0, 0 > RESTORE_ALL_AND_RET > SYM_FUNC_END(handle_vint) > > +SYM_DATA_START(rel_handle_vint) > +LONG 3 > + > +LONG 514b - handle_vint > +LONG kernelsp > + > +LONG 0b - handle_vint > +LONG __arch_cpu_idle > + > +LONG 2b - handle_vint > +LONG do_vint > + > +SYM_DATA_END(rel_handle_vint) > + > SYM_FUNC_START(except_vec_cex) > b cache_parity_error > SYM_FUNC_END(except_vec_cex) > @@ -72,12 +86,24 @@ SYM_FUNC_END(except_vec_cex) > SAVE_ALL > build_prep_\prep > move a0, sp > - la.abs t0, do_\handler > + 667: > + la.pcrel t0, t1, do_\handler > jirl ra, t0, 0 > 668: > RESTORE_ALL_AND_RET > SYM_FUNC_END(handle_\exception) > SYM_DATA(unwind_hint_\exception, .word 668b - 666b) > + > + SYM_DATA_START(rel_handle_\exception) > + LONG 2 > + > + LONG 514b - 666b > + LONG kernelsp > + > + LONG 667b - 666b > + LONG do_\handler > + > + SYM_DATA_END(rel_handle_\exception) > .endm > > BUILD_HANDLER ade ade badv > @@ -93,6 +119,12 @@ SYM_FUNC_END(except_vec_cex) > BUILD_HANDLER reserved reserved none /* others */ > > SYM_FUNC_START(handle_sys) > - la.abs t0, handle_syscall > + la.pcrel t0, t1, handle_syscall > jr t0 > SYM_FUNC_END(handle_sys) > + > +SYM_DATA_START(rel_handle_sys) > +LONG 1 > +LONG 0 > +LONG handle_syscall > +SYM_DATA_END(rel_handle_sys) > diff --git a/arch/loongarch/kernel/traps.c b/arch/loongarch/kernel/traps.c > index c38a146a973b..7e073854f493 100644 > --- a/arch/loongarch/kernel/traps.c > +++ b/arch/loongarch/kernel/traps.c > @@ -62,6 +62,127 @@ extern asmlinkage void handle_reserved(void); > extern asmlinkage void handle_watch(void); > extern asmlinkage void handle_vint(void); > > +struct handler_reloc_entry { > + unsigned long offset; > + unsigned long sym; > +}; > + > +struct handler_reloc { > + unsigned long cnt; > + struct handler_reloc_entry entries[]; > +}; > + > +extern struct handler_reloc rel_handle_tlb_load; > +extern struct handler_reloc rel_handle_tlb_store; > +extern struct handler_reloc rel_handle_tlb_modify; > +extern struct handler_reloc rel_handle_tlb_protect; > +extern struct handler_reloc rel_handle_ade; > +extern struct handler_reloc rel_handle_ale; > +extern struct handler_reloc rel_handle_sys; > +extern struct handler_reloc rel_handle_bp; > +extern struct handler_reloc rel_handle_ri; > +extern struct handler_reloc rel_handle_fpu; > +extern struct handler_reloc rel_handle_lsx; > +extern struct handler_reloc rel_handle_lasx; > +extern struct handler_reloc rel_handle_fpe; > +extern struct handler_reloc rel_handle_lbt; > +extern struct handler_reloc rel_handle_watch; > +extern struct handler_reloc rel_handle_reserved; > +extern struct handler_reloc rel_handle_vint; > + > +struct handler_reloc *eentry_reloc[128] = { > + [0] = NULL, /* merr handler */ > + [EXCCODE_TLBL] = &rel_handle_tlb_load, > + [EXCCODE_TLBS] = &rel_handle_tlb_store, > + [EXCCODE_TLBI] = &rel_handle_tlb_load, > + [EXCCODE_TLBM] = &rel_handle_tlb_modify, > + [EXCCODE_TLBNR] = &rel_handle_tlb_protect, > + [EXCCODE_TLBNX] = &rel_handle_tlb_protect, > + [EXCCODE_TLBPE] = &rel_handle_tlb_protect, > + [EXCCODE_ADE] = &rel_handle_ade, > + [EXCCODE_ALE] = &rel_handle_ale, > + [EXCCODE_SYS] = &rel_handle_sys, > + [EXCCODE_BP] = &rel_handle_bp, > + [EXCCODE_INE] = &rel_handle_ri, > + [EXCCODE_IPE] = &rel_handle_ri, > + [EXCCODE_FPDIS] = &rel_handle_fpu, > + [EXCCODE_LSXDIS] = &rel_handle_lsx, > + [EXCCODE_LASXDIS] = &rel_handle_lasx, > + [EXCCODE_FPE] = &rel_handle_fpe, > + [EXCCODE_BTDIS] = &rel_handle_lbt, > + [EXCCODE_WATCH] = &rel_handle_watch, > + [(EXCCODE_WATCH + 1) ... (EXCCODE_INT_START - 1)] = &rel_handle_reserved, > + [EXCCODE_INT_START ... (EXCCODE_INT_END - 1)] = &rel_handle_vint, > +}; > + > +void reloc_handler(unsigned long handler, struct handler_reloc *rel) > +{ > + if (!rel) > + return; > + > + for (unsigned long i = 0; i < rel->cnt; i++) { > + unsigned long pc = handler + rel->entries[i].offset; > + union loongarch_instruction *insn = > + (union loongarch_instruction *)pc; > + u32 imm[4]; > + unsigned long v = rel->entries[i].sym; > + > + /* GNU as >= 2.40 uses pcalau12i for la.pcrel, but GNU ld <= 2.39 > + * uses pcaddu12i. > + */ > + if (insn->reg1i20_format.opcode == pcalau12i_op) { > + /* Use s32 deliberately for sign extension. */ > + s32 offset_hi20 = ((v + 0x800) & ~0xfff) - > + (pc & ~0xfff); > + unsigned long anchor = (pc & ~0xfff) + offset_hi20; > + unsigned long offset_rem = v - anchor; > + > + imm[0] = (offset_hi20 >> 12) & 0xfffff; > + imm[1] = v & 0xfff; > + imm[2] = (offset_rem >> 32) & 0xfffff; > + imm[3] = offset_rem >> 52; > + } else if (insn->reg1i20_format.opcode == pcaddu12i_op) { > + /* Use s32 deliberately for sign extension. */ > + s32 offset_lo = v - pc; > + unsigned long offset_hi = v - pc - offset_lo; > + > + imm[0] = (offset_lo >> 12) & 0xfffff; > + imm[1] = offset_lo & 0xfff; > + imm[2] = (offset_hi >> 32) & 0xfffff; > + imm[3] = offset_hi >> 52; > + } else > + panic("Cannot fixup la.pcrel for exception handler at %lu: unexpected instruction %d!", > + pc, insn->word); > + > + insn[0].reg1i20_format.immediate = imm[0]; > + insn[1].reg2i12_format.immediate = imm[1]; > + insn[2].reg1i20_format.immediate = imm[2]; > + insn[3].reg2i12_format.immediate = imm[3]; > + } > +} > + > +/* Install CPU exception handler */ > +static void do_set_handler(unsigned long exccode, void *addr, > + struct handler_reloc *rel) > +{ > + unsigned long dest_addr = eentry + exccode * VECSIZE; > + > + memcpy((void *)dest_addr, addr, VECSIZE); > + reloc_handler(dest_addr, rel); > + local_flush_icache_range(dest_addr, dest_addr + VECSIZE); > +} > + > +/* Install CPU exception handler, with the reloc table from eentry_reloc */ > +void set_handler(unsigned long exccode, void *addr) > +{ > + do_set_handler(exccode, addr, eentry_reloc[exccode]); > +} > + > +static void set_handler_reserved(unsigned long exccode) > +{ > + do_set_handler(exccode, handle_reserved, &rel_handle_reserved); > +} > + > static void show_backtrace(struct task_struct *task, const struct pt_regs *regs, > const char *loglvl, bool user) > { > @@ -704,19 +825,12 @@ void per_cpu_trap_init(int cpu) > /* Initialise exception handlers */ > if (cpu == 0) > for (i = 0; i < 64; i++) > - set_handler(i * VECSIZE, handle_reserved, VECSIZE); > + set_handler_reserved(i); > > tlb_init(cpu); > cpu_cache_init(); > } > > -/* Install CPU exception handler */ > -void set_handler(unsigned long offset, void *addr, unsigned long size) > -{ > - memcpy((void *)(eentry + offset), addr, size); > - local_flush_icache_range(eentry + offset, eentry + offset + size); > -} > - > static const char panic_null_cerr[] = > "Trying to set NULL cache error exception handler\n"; > > @@ -741,20 +855,20 @@ void __init trap_init(void) > > /* Set interrupt vector handler */ > for (i = EXCCODE_INT_START; i < EXCCODE_INT_END; i++) > - set_handler(i * VECSIZE, handle_vint, VECSIZE); > - > - set_handler(EXCCODE_ADE * VECSIZE, handle_ade, VECSIZE); > - set_handler(EXCCODE_ALE * VECSIZE, handle_ale, VECSIZE); > - set_handler(EXCCODE_SYS * VECSIZE, handle_sys, VECSIZE); > - set_handler(EXCCODE_BP * VECSIZE, handle_bp, VECSIZE); > - set_handler(EXCCODE_INE * VECSIZE, handle_ri, VECSIZE); > - set_handler(EXCCODE_IPE * VECSIZE, handle_ri, VECSIZE); > - set_handler(EXCCODE_FPDIS * VECSIZE, handle_fpu, VECSIZE); > - set_handler(EXCCODE_LSXDIS * VECSIZE, handle_lsx, VECSIZE); > - set_handler(EXCCODE_LASXDIS * VECSIZE, handle_lasx, VECSIZE); > - set_handler(EXCCODE_FPE * VECSIZE, handle_fpe, VECSIZE); > - set_handler(EXCCODE_BTDIS * VECSIZE, handle_lbt, VECSIZE); > - set_handler(EXCCODE_WATCH * VECSIZE, handle_watch, VECSIZE); > + set_handler(i, handle_vint); > + > + set_handler(EXCCODE_ADE, handle_ade); > + set_handler(EXCCODE_ALE, handle_ale); > + set_handler(EXCCODE_SYS, handle_sys); > + set_handler(EXCCODE_BP, handle_bp); > + set_handler(EXCCODE_INE, handle_ri); > + set_handler(EXCCODE_IPE, handle_ri); > + set_handler(EXCCODE_FPDIS, handle_fpu); > + set_handler(EXCCODE_LSXDIS, handle_lsx); > + set_handler(EXCCODE_LASXDIS, handle_lasx); > + set_handler(EXCCODE_FPE, handle_fpe); > + set_handler(EXCCODE_BTDIS, handle_lbt); > + set_handler(EXCCODE_WATCH, handle_watch); > > cache_error_setup(); > > diff --git a/arch/loongarch/mm/tlb.c b/arch/loongarch/mm/tlb.c > index 8bad6b0cff59..6f70aab7202a 100644 > --- a/arch/loongarch/mm/tlb.c > +++ b/arch/loongarch/mm/tlb.c > @@ -253,7 +253,6 @@ static void output_pgtable_bits_defines(void) > #ifdef CONFIG_NUMA > unsigned long pcpu_handlers[NR_CPUS]; > #endif > -extern long exception_handlers[VECSIZE * 128 / sizeof(long)]; > > void setup_tlb_handler(int cpu) > { > @@ -264,19 +263,20 @@ void setup_tlb_handler(int cpu) > if (cpu == 0) { > memcpy((void *)tlbrentry, handle_tlb_refill, 0x80); > local_flush_icache_range(tlbrentry, tlbrentry + 0x80); > - set_handler(EXCCODE_TLBI * VECSIZE, handle_tlb_load, VECSIZE); > - set_handler(EXCCODE_TLBL * VECSIZE, handle_tlb_load, VECSIZE); > - set_handler(EXCCODE_TLBS * VECSIZE, handle_tlb_store, VECSIZE); > - set_handler(EXCCODE_TLBM * VECSIZE, handle_tlb_modify, VECSIZE); > - set_handler(EXCCODE_TLBNR * VECSIZE, handle_tlb_protect, VECSIZE); > - set_handler(EXCCODE_TLBNX * VECSIZE, handle_tlb_protect, VECSIZE); > - set_handler(EXCCODE_TLBPE * VECSIZE, handle_tlb_protect, VECSIZE); > + set_handler(EXCCODE_TLBI, handle_tlb_load); > + set_handler(EXCCODE_TLBL, handle_tlb_load); > + set_handler(EXCCODE_TLBS, handle_tlb_store); > + set_handler(EXCCODE_TLBM, handle_tlb_modify); > + set_handler(EXCCODE_TLBNR, handle_tlb_protect); > + set_handler(EXCCODE_TLBNX, handle_tlb_protect); > + set_handler(EXCCODE_TLBPE, handle_tlb_protect); > } > #ifdef CONFIG_NUMA > else { > void *addr; > + unsigned long addr_ul; > struct page *page; > - const int vec_sz = sizeof(exception_handlers); > + const int vec_sz = VECSIZE * 128; > > if (pcpu_handlers[cpu]) > return; > @@ -286,8 +286,11 @@ void setup_tlb_handler(int cpu) > return; > > addr = page_address(page); > + addr_ul = (unsigned long)addr; > pcpu_handlers[cpu] = (unsigned long)addr; > - memcpy((void *)addr, (void *)eentry, vec_sz); > + memcpy(addr, (void *)eentry, vec_sz); > + for (unsigned long i = 0; i < 128; i++) > + reloc_handler(addr_ul + i * VECSIZE, eentry_reloc[i]); > local_flush_icache_range((unsigned long)addr, (unsigned long)addr + vec_sz); > csr_write64(pcpu_handlers[cpu], LOONGARCH_CSR_EENTRY); > csr_write64(pcpu_handlers[cpu], LOONGARCH_CSR_MERRENTRY); > diff --git a/arch/loongarch/mm/tlbex.S b/arch/loongarch/mm/tlbex.S > index 3dd2a9615cd9..044c2190771a 100644 > --- a/arch/loongarch/mm/tlbex.S > +++ b/arch/loongarch/mm/tlbex.S > @@ -39,11 +39,21 @@ SYM_FUNC_START(handle_tlb_protect) > move a1, zero > csrrd a2, LOONGARCH_CSR_BADV > REG_S a2, sp, PT_BVADDR > - la.abs t0, do_page_fault > +1: la.pcrel t0, t1, do_page_fault > jirl ra, t0, 0 > RESTORE_ALL_AND_RET > SYM_FUNC_END(handle_tlb_protect) > > +SYM_DATA_START(rel_handle_tlb_protect) > + LONG 2 > + > + LONG 514b - handle_tlb_protect > + LONG kernelsp > + > + LONG 1b - handle_tlb_protect > + LONG do_page_fault > +SYM_DATA_END(rel_handle_tlb_protect) > + > SYM_FUNC_START(handle_tlb_load) > csrwr t0, EXCEPTION_KS0 > csrwr t1, EXCEPTION_KS1 > @@ -115,7 +125,8 @@ smp_pgtable_change_load: > > #ifdef CONFIG_64BIT > vmalloc_load: > - la.abs t1, swapper_pg_dir > + /* The first insn of vmalloc_done_load overwrites ra */ > +1: la.pcrel t1, ra, swapper_pg_dir > b vmalloc_done_load > #endif > > @@ -186,10 +197,24 @@ tlb_huge_update_load: > nopage_tlb_load: > dbar 0 > csrrd ra, EXCEPTION_KS2 > - la.abs t0, tlb_do_page_fault_0 > +2: la.pcrel t0, t1, tlb_do_page_fault_0 > jr t0 > SYM_FUNC_END(handle_tlb_load) > > +SYM_DATA_START(rel_handle_tlb_load) > +#ifdef CONFIG_64BIT > + LONG 2 > + > + LONG 1b - handle_tlb_load > + LONG swapper_pg_dir > +#else > + LONG 1 > +#endif > + > + LONG 2b - handle_tlb_load > + LONG tlb_do_page_fault_0 > +SYM_DATA_END(rel_handle_tlb_load) > + > SYM_FUNC_START(handle_tlb_store) > csrwr t0, EXCEPTION_KS0 > csrwr t1, EXCEPTION_KS1 > @@ -262,7 +287,8 @@ smp_pgtable_change_store: > > #ifdef CONFIG_64BIT > vmalloc_store: > - la.abs t1, swapper_pg_dir > + /* The first insn of vmalloc_done_store overwrites ra */ > +1: la.pcrel t1, ra, swapper_pg_dir > b vmalloc_done_store > #endif > > @@ -335,10 +361,24 @@ tlb_huge_update_store: > nopage_tlb_store: > dbar 0 > csrrd ra, EXCEPTION_KS2 > - la.abs t0, tlb_do_page_fault_1 > +2: la.pcrel t0, t1, tlb_do_page_fault_1 > jr t0 > SYM_FUNC_END(handle_tlb_store) > > +SYM_DATA_START(rel_handle_tlb_store) > +#ifdef CONFIG_64BIT > + LONG 2 > + > + LONG 1b - handle_tlb_store > + LONG swapper_pg_dir > +#else > + LONG 1 > +#endif > + > + LONG 2b - handle_tlb_store > + LONG tlb_do_page_fault_1 > +SYM_DATA_END(rel_handle_tlb_store) > + > SYM_FUNC_START(handle_tlb_modify) > csrwr t0, EXCEPTION_KS0 > csrwr t1, EXCEPTION_KS1 > @@ -410,7 +450,8 @@ smp_pgtable_change_modify: > > #ifdef CONFIG_64BIT > vmalloc_modify: > - la.abs t1, swapper_pg_dir > + /* The first insn of vmalloc_done_modify overwrites ra */ > +1: la.pcrel t1, ra, swapper_pg_dir > b vmalloc_done_modify > #endif > > @@ -482,10 +523,24 @@ tlb_huge_update_modify: > nopage_tlb_modify: > dbar 0 > csrrd ra, EXCEPTION_KS2 > - la.abs t0, tlb_do_page_fault_1 > +2: la.pcrel t0, t1, tlb_do_page_fault_1 > jr t0 > SYM_FUNC_END(handle_tlb_modify) > > +SYM_DATA_START(rel_handle_tlb_modify) > +#ifdef CONFIG_64BIT > + LONG 2 > + > + LONG 1b - handle_tlb_modify > + LONG swapper_pg_dir > +#else > + LONG 1 > +#endif > + > + LONG 2b - handle_tlb_modify > + LONG tlb_do_page_fault_1 > +SYM_DATA_END(rel_handle_tlb_modify) > + > SYM_FUNC_START(handle_tlb_refill) > csrwr t0, LOONGARCH_CSR_TLBRSAVE > csrrd t0, LOONGARCH_CSR_PGD > -- > 2.37.3 > ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [PATCH v4 2/5] LoongArch: Use la.pcrel instead of la.abs for exception handlers 2023-02-10 9:09 ` Huacai Chen @ 2023-02-10 9:18 ` Youling Tang 2023-02-16 2:32 ` Youling Tang 0 siblings, 1 reply; 22+ messages in thread From: Youling Tang @ 2023-02-10 9:18 UTC (permalink / raw) To: Huacai Chen; +Cc: Xi Ruoyao, Jinyang He, Xuerui Wang, loongarch, linux-kernel On 02/10/2023 05:09 PM, Huacai Chen wrote: > Hi, Youling and Ruoyao, > > Thank you very much for implementing the per-node exceptions. But I > want to know if the per-node solution is really worthy for a PIE > kernel. So, could you please test the performance? Maybe we can reduce > the complexity if we give up the per-node solution. I will test performance on NUMA machines based on v2 and v3 patch sets. Youling. > > Huacai > > On Fri, Feb 10, 2023 at 4:47 PM Youling Tang <tangyouling@loongson.cn> wrote: >> >> From: Xi Ruoyao <xry111@xry111.site> >> >> It's needed to build the kernel as a PIE, or the linker will complain. >> >> For the consideration about performance, we copy the exception handlers >> to a dedicated 64 KB area for each CPU. So, the PC-relative offset >> calculated at link time will be incorrect and we need to relocate the >> exception handlers after copying them. >> >> For the simplicity, we don't use the ELF R_LARCH_* relocations, but code >> an relocation entry as simply (offset_in_the_handler, symbol_addr). For >> each exception handler, we also code the number of relocation entries. >> Then we can use the relocation information to fix up the handlers after >> copying them. >> >> Signed-off-by: Xi Ruoyao <xry111@xry111.site> >> --- >> arch/loongarch/include/asm/inst.h | 1 + >> arch/loongarch/include/asm/setup.h | 6 +- >> arch/loongarch/include/asm/stackframe.h | 3 +- >> arch/loongarch/kernel/genex.S | 40 +++++- >> arch/loongarch/kernel/traps.c | 158 ++++++++++++++++++++---- >> arch/loongarch/mm/tlb.c | 23 ++-- >> arch/loongarch/mm/tlbex.S | 69 +++++++++-- >> 7 files changed, 255 insertions(+), 45 deletions(-) >> >> diff --git a/arch/loongarch/include/asm/inst.h b/arch/loongarch/include/asm/inst.h >> index 7eedd83fd0d7..426054518a3d 100644 >> --- a/arch/loongarch/include/asm/inst.h >> +++ b/arch/loongarch/include/asm/inst.h >> @@ -32,6 +32,7 @@ enum reg1i20_op { >> lu12iw_op = 0x0a, >> lu32id_op = 0x0b, >> pcaddi_op = 0x0c, >> + pcalau12i_op = 0x0d, >> pcaddu12i_op = 0x0e, >> pcaddu18i_op = 0x0f, >> }; >> diff --git a/arch/loongarch/include/asm/setup.h b/arch/loongarch/include/asm/setup.h >> index 72ead58039f3..f0a2b34365f1 100644 >> --- a/arch/loongarch/include/asm/setup.h >> +++ b/arch/loongarch/include/asm/setup.h >> @@ -11,6 +11,9 @@ >> >> #define VECSIZE 0x200 >> >> +struct handler_reloc; >> + >> +extern struct handler_reloc *eentry_reloc[]; >> extern unsigned long eentry; >> extern unsigned long tlbrentry; >> extern char init_command_line[COMMAND_LINE_SIZE]; >> @@ -18,7 +21,8 @@ extern void tlb_init(int cpu); >> extern void cpu_cache_init(void); >> extern void cache_error_setup(void); >> extern void per_cpu_trap_init(int cpu); >> -extern void set_handler(unsigned long offset, void *addr, unsigned long len); >> +extern void set_handler(unsigned long exccode, void *addr); >> extern void set_merr_handler(unsigned long offset, void *addr, unsigned long len); >> +extern void reloc_handler(unsigned long handler, struct handler_reloc *rel); >> >> #endif /* __SETUP_H */ >> diff --git a/arch/loongarch/include/asm/stackframe.h b/arch/loongarch/include/asm/stackframe.h >> index 7deb043ce387..bbec1e56b61b 100644 >> --- a/arch/loongarch/include/asm/stackframe.h >> +++ b/arch/loongarch/include/asm/stackframe.h >> @@ -77,7 +77,8 @@ >> * new value in sp. >> */ >> .macro get_saved_sp docfi=0 >> - la.abs t1, kernelsp >> + /* The label is used for generating reloc tables for handlers */ >> +514: la.pcrel t1, t0, kernelsp >> #ifdef CONFIG_SMP >> csrrd t0, PERCPU_BASE_KS >> LONG_ADD t1, t1, t0 >> diff --git a/arch/loongarch/kernel/genex.S b/arch/loongarch/kernel/genex.S >> index 7e5c293ed89f..005a10fe5a50 100644 >> --- a/arch/loongarch/kernel/genex.S >> +++ b/arch/loongarch/kernel/genex.S >> @@ -34,7 +34,7 @@ SYM_FUNC_END(__arch_cpu_idle) >> SYM_FUNC_START(handle_vint) >> BACKUP_T0T1 >> SAVE_ALL >> - la.abs t1, __arch_cpu_idle >> +0: la.pcrel t1, t2, __arch_cpu_idle >> LONG_L t0, sp, PT_ERA >> /* 32 byte rollback region */ >> ori t0, t0, 0x1f >> @@ -43,11 +43,25 @@ SYM_FUNC_START(handle_vint) >> LONG_S t0, sp, PT_ERA >> 1: move a0, sp >> move a1, sp >> - la.abs t0, do_vint >> +2: la.pcrel t0, t2, do_vint >> jirl ra, t0, 0 >> RESTORE_ALL_AND_RET >> SYM_FUNC_END(handle_vint) >> >> +SYM_DATA_START(rel_handle_vint) >> +LONG 3 >> + >> +LONG 514b - handle_vint >> +LONG kernelsp >> + >> +LONG 0b - handle_vint >> +LONG __arch_cpu_idle >> + >> +LONG 2b - handle_vint >> +LONG do_vint >> + >> +SYM_DATA_END(rel_handle_vint) >> + >> SYM_FUNC_START(except_vec_cex) >> b cache_parity_error >> SYM_FUNC_END(except_vec_cex) >> @@ -72,12 +86,24 @@ SYM_FUNC_END(except_vec_cex) >> SAVE_ALL >> build_prep_\prep >> move a0, sp >> - la.abs t0, do_\handler >> + 667: >> + la.pcrel t0, t1, do_\handler >> jirl ra, t0, 0 >> 668: >> RESTORE_ALL_AND_RET >> SYM_FUNC_END(handle_\exception) >> SYM_DATA(unwind_hint_\exception, .word 668b - 666b) >> + >> + SYM_DATA_START(rel_handle_\exception) >> + LONG 2 >> + >> + LONG 514b - 666b >> + LONG kernelsp >> + >> + LONG 667b - 666b >> + LONG do_\handler >> + >> + SYM_DATA_END(rel_handle_\exception) >> .endm >> >> BUILD_HANDLER ade ade badv >> @@ -93,6 +119,12 @@ SYM_FUNC_END(except_vec_cex) >> BUILD_HANDLER reserved reserved none /* others */ >> >> SYM_FUNC_START(handle_sys) >> - la.abs t0, handle_syscall >> + la.pcrel t0, t1, handle_syscall >> jr t0 >> SYM_FUNC_END(handle_sys) >> + >> +SYM_DATA_START(rel_handle_sys) >> +LONG 1 >> +LONG 0 >> +LONG handle_syscall >> +SYM_DATA_END(rel_handle_sys) >> diff --git a/arch/loongarch/kernel/traps.c b/arch/loongarch/kernel/traps.c >> index c38a146a973b..7e073854f493 100644 >> --- a/arch/loongarch/kernel/traps.c >> +++ b/arch/loongarch/kernel/traps.c >> @@ -62,6 +62,127 @@ extern asmlinkage void handle_reserved(void); >> extern asmlinkage void handle_watch(void); >> extern asmlinkage void handle_vint(void); >> >> +struct handler_reloc_entry { >> + unsigned long offset; >> + unsigned long sym; >> +}; >> + >> +struct handler_reloc { >> + unsigned long cnt; >> + struct handler_reloc_entry entries[]; >> +}; >> + >> +extern struct handler_reloc rel_handle_tlb_load; >> +extern struct handler_reloc rel_handle_tlb_store; >> +extern struct handler_reloc rel_handle_tlb_modify; >> +extern struct handler_reloc rel_handle_tlb_protect; >> +extern struct handler_reloc rel_handle_ade; >> +extern struct handler_reloc rel_handle_ale; >> +extern struct handler_reloc rel_handle_sys; >> +extern struct handler_reloc rel_handle_bp; >> +extern struct handler_reloc rel_handle_ri; >> +extern struct handler_reloc rel_handle_fpu; >> +extern struct handler_reloc rel_handle_lsx; >> +extern struct handler_reloc rel_handle_lasx; >> +extern struct handler_reloc rel_handle_fpe; >> +extern struct handler_reloc rel_handle_lbt; >> +extern struct handler_reloc rel_handle_watch; >> +extern struct handler_reloc rel_handle_reserved; >> +extern struct handler_reloc rel_handle_vint; >> + >> +struct handler_reloc *eentry_reloc[128] = { >> + [0] = NULL, /* merr handler */ >> + [EXCCODE_TLBL] = &rel_handle_tlb_load, >> + [EXCCODE_TLBS] = &rel_handle_tlb_store, >> + [EXCCODE_TLBI] = &rel_handle_tlb_load, >> + [EXCCODE_TLBM] = &rel_handle_tlb_modify, >> + [EXCCODE_TLBNR] = &rel_handle_tlb_protect, >> + [EXCCODE_TLBNX] = &rel_handle_tlb_protect, >> + [EXCCODE_TLBPE] = &rel_handle_tlb_protect, >> + [EXCCODE_ADE] = &rel_handle_ade, >> + [EXCCODE_ALE] = &rel_handle_ale, >> + [EXCCODE_SYS] = &rel_handle_sys, >> + [EXCCODE_BP] = &rel_handle_bp, >> + [EXCCODE_INE] = &rel_handle_ri, >> + [EXCCODE_IPE] = &rel_handle_ri, >> + [EXCCODE_FPDIS] = &rel_handle_fpu, >> + [EXCCODE_LSXDIS] = &rel_handle_lsx, >> + [EXCCODE_LASXDIS] = &rel_handle_lasx, >> + [EXCCODE_FPE] = &rel_handle_fpe, >> + [EXCCODE_BTDIS] = &rel_handle_lbt, >> + [EXCCODE_WATCH] = &rel_handle_watch, >> + [(EXCCODE_WATCH + 1) ... (EXCCODE_INT_START - 1)] = &rel_handle_reserved, >> + [EXCCODE_INT_START ... (EXCCODE_INT_END - 1)] = &rel_handle_vint, >> +}; >> + >> +void reloc_handler(unsigned long handler, struct handler_reloc *rel) >> +{ >> + if (!rel) >> + return; >> + >> + for (unsigned long i = 0; i < rel->cnt; i++) { >> + unsigned long pc = handler + rel->entries[i].offset; >> + union loongarch_instruction *insn = >> + (union loongarch_instruction *)pc; >> + u32 imm[4]; >> + unsigned long v = rel->entries[i].sym; >> + >> + /* GNU as >= 2.40 uses pcalau12i for la.pcrel, but GNU ld <= 2.39 >> + * uses pcaddu12i. >> + */ >> + if (insn->reg1i20_format.opcode == pcalau12i_op) { >> + /* Use s32 deliberately for sign extension. */ >> + s32 offset_hi20 = ((v + 0x800) & ~0xfff) - >> + (pc & ~0xfff); >> + unsigned long anchor = (pc & ~0xfff) + offset_hi20; >> + unsigned long offset_rem = v - anchor; >> + >> + imm[0] = (offset_hi20 >> 12) & 0xfffff; >> + imm[1] = v & 0xfff; >> + imm[2] = (offset_rem >> 32) & 0xfffff; >> + imm[3] = offset_rem >> 52; >> + } else if (insn->reg1i20_format.opcode == pcaddu12i_op) { >> + /* Use s32 deliberately for sign extension. */ >> + s32 offset_lo = v - pc; >> + unsigned long offset_hi = v - pc - offset_lo; >> + >> + imm[0] = (offset_lo >> 12) & 0xfffff; >> + imm[1] = offset_lo & 0xfff; >> + imm[2] = (offset_hi >> 32) & 0xfffff; >> + imm[3] = offset_hi >> 52; >> + } else >> + panic("Cannot fixup la.pcrel for exception handler at %lu: unexpected instruction %d!", >> + pc, insn->word); >> + >> + insn[0].reg1i20_format.immediate = imm[0]; >> + insn[1].reg2i12_format.immediate = imm[1]; >> + insn[2].reg1i20_format.immediate = imm[2]; >> + insn[3].reg2i12_format.immediate = imm[3]; >> + } >> +} >> + >> +/* Install CPU exception handler */ >> +static void do_set_handler(unsigned long exccode, void *addr, >> + struct handler_reloc *rel) >> +{ >> + unsigned long dest_addr = eentry + exccode * VECSIZE; >> + >> + memcpy((void *)dest_addr, addr, VECSIZE); >> + reloc_handler(dest_addr, rel); >> + local_flush_icache_range(dest_addr, dest_addr + VECSIZE); >> +} >> + >> +/* Install CPU exception handler, with the reloc table from eentry_reloc */ >> +void set_handler(unsigned long exccode, void *addr) >> +{ >> + do_set_handler(exccode, addr, eentry_reloc[exccode]); >> +} >> + >> +static void set_handler_reserved(unsigned long exccode) >> +{ >> + do_set_handler(exccode, handle_reserved, &rel_handle_reserved); >> +} >> + >> static void show_backtrace(struct task_struct *task, const struct pt_regs *regs, >> const char *loglvl, bool user) >> { >> @@ -704,19 +825,12 @@ void per_cpu_trap_init(int cpu) >> /* Initialise exception handlers */ >> if (cpu == 0) >> for (i = 0; i < 64; i++) >> - set_handler(i * VECSIZE, handle_reserved, VECSIZE); >> + set_handler_reserved(i); >> >> tlb_init(cpu); >> cpu_cache_init(); >> } >> >> -/* Install CPU exception handler */ >> -void set_handler(unsigned long offset, void *addr, unsigned long size) >> -{ >> - memcpy((void *)(eentry + offset), addr, size); >> - local_flush_icache_range(eentry + offset, eentry + offset + size); >> -} >> - >> static const char panic_null_cerr[] = >> "Trying to set NULL cache error exception handler\n"; >> >> @@ -741,20 +855,20 @@ void __init trap_init(void) >> >> /* Set interrupt vector handler */ >> for (i = EXCCODE_INT_START; i < EXCCODE_INT_END; i++) >> - set_handler(i * VECSIZE, handle_vint, VECSIZE); >> - >> - set_handler(EXCCODE_ADE * VECSIZE, handle_ade, VECSIZE); >> - set_handler(EXCCODE_ALE * VECSIZE, handle_ale, VECSIZE); >> - set_handler(EXCCODE_SYS * VECSIZE, handle_sys, VECSIZE); >> - set_handler(EXCCODE_BP * VECSIZE, handle_bp, VECSIZE); >> - set_handler(EXCCODE_INE * VECSIZE, handle_ri, VECSIZE); >> - set_handler(EXCCODE_IPE * VECSIZE, handle_ri, VECSIZE); >> - set_handler(EXCCODE_FPDIS * VECSIZE, handle_fpu, VECSIZE); >> - set_handler(EXCCODE_LSXDIS * VECSIZE, handle_lsx, VECSIZE); >> - set_handler(EXCCODE_LASXDIS * VECSIZE, handle_lasx, VECSIZE); >> - set_handler(EXCCODE_FPE * VECSIZE, handle_fpe, VECSIZE); >> - set_handler(EXCCODE_BTDIS * VECSIZE, handle_lbt, VECSIZE); >> - set_handler(EXCCODE_WATCH * VECSIZE, handle_watch, VECSIZE); >> + set_handler(i, handle_vint); >> + >> + set_handler(EXCCODE_ADE, handle_ade); >> + set_handler(EXCCODE_ALE, handle_ale); >> + set_handler(EXCCODE_SYS, handle_sys); >> + set_handler(EXCCODE_BP, handle_bp); >> + set_handler(EXCCODE_INE, handle_ri); >> + set_handler(EXCCODE_IPE, handle_ri); >> + set_handler(EXCCODE_FPDIS, handle_fpu); >> + set_handler(EXCCODE_LSXDIS, handle_lsx); >> + set_handler(EXCCODE_LASXDIS, handle_lasx); >> + set_handler(EXCCODE_FPE, handle_fpe); >> + set_handler(EXCCODE_BTDIS, handle_lbt); >> + set_handler(EXCCODE_WATCH, handle_watch); >> >> cache_error_setup(); >> >> diff --git a/arch/loongarch/mm/tlb.c b/arch/loongarch/mm/tlb.c >> index 8bad6b0cff59..6f70aab7202a 100644 >> --- a/arch/loongarch/mm/tlb.c >> +++ b/arch/loongarch/mm/tlb.c >> @@ -253,7 +253,6 @@ static void output_pgtable_bits_defines(void) >> #ifdef CONFIG_NUMA >> unsigned long pcpu_handlers[NR_CPUS]; >> #endif >> -extern long exception_handlers[VECSIZE * 128 / sizeof(long)]; >> >> void setup_tlb_handler(int cpu) >> { >> @@ -264,19 +263,20 @@ void setup_tlb_handler(int cpu) >> if (cpu == 0) { >> memcpy((void *)tlbrentry, handle_tlb_refill, 0x80); >> local_flush_icache_range(tlbrentry, tlbrentry + 0x80); >> - set_handler(EXCCODE_TLBI * VECSIZE, handle_tlb_load, VECSIZE); >> - set_handler(EXCCODE_TLBL * VECSIZE, handle_tlb_load, VECSIZE); >> - set_handler(EXCCODE_TLBS * VECSIZE, handle_tlb_store, VECSIZE); >> - set_handler(EXCCODE_TLBM * VECSIZE, handle_tlb_modify, VECSIZE); >> - set_handler(EXCCODE_TLBNR * VECSIZE, handle_tlb_protect, VECSIZE); >> - set_handler(EXCCODE_TLBNX * VECSIZE, handle_tlb_protect, VECSIZE); >> - set_handler(EXCCODE_TLBPE * VECSIZE, handle_tlb_protect, VECSIZE); >> + set_handler(EXCCODE_TLBI, handle_tlb_load); >> + set_handler(EXCCODE_TLBL, handle_tlb_load); >> + set_handler(EXCCODE_TLBS, handle_tlb_store); >> + set_handler(EXCCODE_TLBM, handle_tlb_modify); >> + set_handler(EXCCODE_TLBNR, handle_tlb_protect); >> + set_handler(EXCCODE_TLBNX, handle_tlb_protect); >> + set_handler(EXCCODE_TLBPE, handle_tlb_protect); >> } >> #ifdef CONFIG_NUMA >> else { >> void *addr; >> + unsigned long addr_ul; >> struct page *page; >> - const int vec_sz = sizeof(exception_handlers); >> + const int vec_sz = VECSIZE * 128; >> >> if (pcpu_handlers[cpu]) >> return; >> @@ -286,8 +286,11 @@ void setup_tlb_handler(int cpu) >> return; >> >> addr = page_address(page); >> + addr_ul = (unsigned long)addr; >> pcpu_handlers[cpu] = (unsigned long)addr; >> - memcpy((void *)addr, (void *)eentry, vec_sz); >> + memcpy(addr, (void *)eentry, vec_sz); >> + for (unsigned long i = 0; i < 128; i++) >> + reloc_handler(addr_ul + i * VECSIZE, eentry_reloc[i]); >> local_flush_icache_range((unsigned long)addr, (unsigned long)addr + vec_sz); >> csr_write64(pcpu_handlers[cpu], LOONGARCH_CSR_EENTRY); >> csr_write64(pcpu_handlers[cpu], LOONGARCH_CSR_MERRENTRY); >> diff --git a/arch/loongarch/mm/tlbex.S b/arch/loongarch/mm/tlbex.S >> index 3dd2a9615cd9..044c2190771a 100644 >> --- a/arch/loongarch/mm/tlbex.S >> +++ b/arch/loongarch/mm/tlbex.S >> @@ -39,11 +39,21 @@ SYM_FUNC_START(handle_tlb_protect) >> move a1, zero >> csrrd a2, LOONGARCH_CSR_BADV >> REG_S a2, sp, PT_BVADDR >> - la.abs t0, do_page_fault >> +1: la.pcrel t0, t1, do_page_fault >> jirl ra, t0, 0 >> RESTORE_ALL_AND_RET >> SYM_FUNC_END(handle_tlb_protect) >> >> +SYM_DATA_START(rel_handle_tlb_protect) >> + LONG 2 >> + >> + LONG 514b - handle_tlb_protect >> + LONG kernelsp >> + >> + LONG 1b - handle_tlb_protect >> + LONG do_page_fault >> +SYM_DATA_END(rel_handle_tlb_protect) >> + >> SYM_FUNC_START(handle_tlb_load) >> csrwr t0, EXCEPTION_KS0 >> csrwr t1, EXCEPTION_KS1 >> @@ -115,7 +125,8 @@ smp_pgtable_change_load: >> >> #ifdef CONFIG_64BIT >> vmalloc_load: >> - la.abs t1, swapper_pg_dir >> + /* The first insn of vmalloc_done_load overwrites ra */ >> +1: la.pcrel t1, ra, swapper_pg_dir >> b vmalloc_done_load >> #endif >> >> @@ -186,10 +197,24 @@ tlb_huge_update_load: >> nopage_tlb_load: >> dbar 0 >> csrrd ra, EXCEPTION_KS2 >> - la.abs t0, tlb_do_page_fault_0 >> +2: la.pcrel t0, t1, tlb_do_page_fault_0 >> jr t0 >> SYM_FUNC_END(handle_tlb_load) >> >> +SYM_DATA_START(rel_handle_tlb_load) >> +#ifdef CONFIG_64BIT >> + LONG 2 >> + >> + LONG 1b - handle_tlb_load >> + LONG swapper_pg_dir >> +#else >> + LONG 1 >> +#endif >> + >> + LONG 2b - handle_tlb_load >> + LONG tlb_do_page_fault_0 >> +SYM_DATA_END(rel_handle_tlb_load) >> + >> SYM_FUNC_START(handle_tlb_store) >> csrwr t0, EXCEPTION_KS0 >> csrwr t1, EXCEPTION_KS1 >> @@ -262,7 +287,8 @@ smp_pgtable_change_store: >> >> #ifdef CONFIG_64BIT >> vmalloc_store: >> - la.abs t1, swapper_pg_dir >> + /* The first insn of vmalloc_done_store overwrites ra */ >> +1: la.pcrel t1, ra, swapper_pg_dir >> b vmalloc_done_store >> #endif >> >> @@ -335,10 +361,24 @@ tlb_huge_update_store: >> nopage_tlb_store: >> dbar 0 >> csrrd ra, EXCEPTION_KS2 >> - la.abs t0, tlb_do_page_fault_1 >> +2: la.pcrel t0, t1, tlb_do_page_fault_1 >> jr t0 >> SYM_FUNC_END(handle_tlb_store) >> >> +SYM_DATA_START(rel_handle_tlb_store) >> +#ifdef CONFIG_64BIT >> + LONG 2 >> + >> + LONG 1b - handle_tlb_store >> + LONG swapper_pg_dir >> +#else >> + LONG 1 >> +#endif >> + >> + LONG 2b - handle_tlb_store >> + LONG tlb_do_page_fault_1 >> +SYM_DATA_END(rel_handle_tlb_store) >> + >> SYM_FUNC_START(handle_tlb_modify) >> csrwr t0, EXCEPTION_KS0 >> csrwr t1, EXCEPTION_KS1 >> @@ -410,7 +450,8 @@ smp_pgtable_change_modify: >> >> #ifdef CONFIG_64BIT >> vmalloc_modify: >> - la.abs t1, swapper_pg_dir >> + /* The first insn of vmalloc_done_modify overwrites ra */ >> +1: la.pcrel t1, ra, swapper_pg_dir >> b vmalloc_done_modify >> #endif >> >> @@ -482,10 +523,24 @@ tlb_huge_update_modify: >> nopage_tlb_modify: >> dbar 0 >> csrrd ra, EXCEPTION_KS2 >> - la.abs t0, tlb_do_page_fault_1 >> +2: la.pcrel t0, t1, tlb_do_page_fault_1 >> jr t0 >> SYM_FUNC_END(handle_tlb_modify) >> >> +SYM_DATA_START(rel_handle_tlb_modify) >> +#ifdef CONFIG_64BIT >> + LONG 2 >> + >> + LONG 1b - handle_tlb_modify >> + LONG swapper_pg_dir >> +#else >> + LONG 1 >> +#endif >> + >> + LONG 2b - handle_tlb_modify >> + LONG tlb_do_page_fault_1 >> +SYM_DATA_END(rel_handle_tlb_modify) >> + >> SYM_FUNC_START(handle_tlb_refill) >> csrwr t0, LOONGARCH_CSR_TLBRSAVE >> csrrd t0, LOONGARCH_CSR_PGD >> -- >> 2.37.3 >> ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [PATCH v4 2/5] LoongArch: Use la.pcrel instead of la.abs for exception handlers 2023-02-10 9:18 ` Youling Tang @ 2023-02-16 2:32 ` Youling Tang 2023-02-16 6:56 ` Huacai Chen 2023-02-16 6:59 ` Jinyang He 0 siblings, 2 replies; 22+ messages in thread From: Youling Tang @ 2023-02-16 2:32 UTC (permalink / raw) To: Huacai Chen, Xi Ruoyao Cc: Jinyang He, Xuerui Wang, loongarch, linux-kernel, Xuefeng Li, Jianmin lv, Tiezhu Yang Hi folks, On 02/10/2023 05:18 PM, Youling Tang wrote: > > > On 02/10/2023 05:09 PM, Huacai Chen wrote: >> Hi, Youling and Ruoyao, >> >> Thank you very much for implementing the per-node exceptions. But I >> want to know if the per-node solution is really worthy for a PIE >> kernel. So, could you please test the performance? Maybe we can reduce >> the complexity if we give up the per-node solution. Tested on Loongson-3C5000L-LL machine, using CLFS7.3 system. - nopernode: Based on the v1 patch method, and remove the else branch process in setup_tlb_handler(). - pernode: Based on the v4 patch method. - pie: Enable RANDOMIZE_BASE (KASLR). - nopie: Disable RANDOMIZE_BASE and RELOCATABLE. The UnixBench test results are as follows: - nopernode-nopie: 3938.7 - pernode-nopie: 4062.2 - nopernode-pie: 4009.7 - pernode-pie: 4028.7 In general, `pernode` is higher than `nopernode`, and `nopie` is higher than `pie`. (except that nopernode-pie is higher than nopernode-nopie, which is not as expected, which may be caused by the instability of the machine). Everyone is more inclined to use `pernode` or `nopernode` to implement in the exception handling process? Youling. ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [PATCH v4 2/5] LoongArch: Use la.pcrel instead of la.abs for exception handlers 2023-02-16 2:32 ` Youling Tang @ 2023-02-16 6:56 ` Huacai Chen 2023-02-16 6:59 ` Jinyang He 1 sibling, 0 replies; 22+ messages in thread From: Huacai Chen @ 2023-02-16 6:56 UTC (permalink / raw) To: Youling Tang Cc: Xi Ruoyao, Jinyang He, Xuerui Wang, loongarch, linux-kernel, Xuefeng Li, Jianmin lv, Tiezhu Yang On Thu, Feb 16, 2023 at 10:32 AM Youling Tang <tangyouling@loongson.cn> wrote: > > Hi folks, > > On 02/10/2023 05:18 PM, Youling Tang wrote: > > > > > > On 02/10/2023 05:09 PM, Huacai Chen wrote: > >> Hi, Youling and Ruoyao, > >> > >> Thank you very much for implementing the per-node exceptions. But I > >> want to know if the per-node solution is really worthy for a PIE > >> kernel. So, could you please test the performance? Maybe we can reduce > >> the complexity if we give up the per-node solution. > > Tested on Loongson-3C5000L-LL machine, using CLFS7.3 system. > > - nopernode: > Based on the v1 patch method, and remove the else branch process in > setup_tlb_handler(). > > - pernode: Based on the v4 patch method. > > - pie: Enable RANDOMIZE_BASE (KASLR). > > - nopie: Disable RANDOMIZE_BASE and RELOCATABLE. > > > The UnixBench test results are as follows: > > - nopernode-nopie: 3938.7 > > - pernode-nopie: 4062.2 > > - nopernode-pie: 4009.7 > > - pernode-pie: 4028.7 > > In general, `pernode` is higher than `nopernode`, and `nopie` is higher > than `pie`. (except that nopernode-pie is higher than nopernode-nopie, > which is not as expected, which may be caused by the instability of the > machine). > > Everyone is more inclined to use `pernode` or `nopernode` to implement > in the exception handling process? From my point of view, for the PIE kernel the performance difference between pernode and nopoernode is negligible. On the other hand, pernode implementation needs some compiler hackings and makes the logic significantly complex. So I prefer to remove the pernode exception support. Huacai > > Youling. > > ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [PATCH v4 2/5] LoongArch: Use la.pcrel instead of la.abs for exception handlers 2023-02-16 2:32 ` Youling Tang 2023-02-16 6:56 ` Huacai Chen @ 2023-02-16 6:59 ` Jinyang He 2023-02-16 7:10 ` Xi Ruoyao 1 sibling, 1 reply; 22+ messages in thread From: Jinyang He @ 2023-02-16 6:59 UTC (permalink / raw) To: Youling Tang, Huacai Chen, Xi Ruoyao Cc: Xuerui Wang, loongarch, linux-kernel, Xuefeng Li, Jianmin lv, Tiezhu Yang On 2023-02-16 10:32, Youling Tang wrote: > Hi folks, > > On 02/10/2023 05:18 PM, Youling Tang wrote: >> >> >> On 02/10/2023 05:09 PM, Huacai Chen wrote: >>> Hi, Youling and Ruoyao, >>> >>> Thank you very much for implementing the per-node exceptions. But I >>> want to know if the per-node solution is really worthy for a PIE >>> kernel. So, could you please test the performance? Maybe we can reduce >>> the complexity if we give up the per-node solution. > > Tested on Loongson-3C5000L-LL machine, using CLFS7.3 system. > > - nopernode: > Based on the v1 patch method, and remove the else branch process in > setup_tlb_handler(). > > - pernode: Based on the v4 patch method. > > - pie: Enable RANDOMIZE_BASE (KASLR). > > - nopie: Disable RANDOMIZE_BASE and RELOCATABLE. > > > The UnixBench test results are as follows: > > - nopernode-nopie: 3938.7 > > - pernode-nopie: 4062.2 > > - nopernode-pie: 4009.7 > > - pernode-pie: 4028.7 > > In general, `pernode` is higher than `nopernode`, and `nopie` is higher > than `pie`. (except that nopernode-pie is higher than nopernode-nopie, > which is not as expected, which may be caused by the instability of the > machine). > > Everyone is more inclined to use `pernode` or `nopernode` to implement > in the exception handling process? > > Youling. Hi, Youling, Thanks for your test results. I did an informal patch to keep la.abs, which think la.abs as a macro. just qemu test. To test this patch, patch the [PATCH v4 1/5] [PATCH v4 3/5] as prediction. This following patch just provides a method. I'm busy with other things. Hopefully it will help you simplify [PATCH v4 2/5]. Thanks, Jinyang diff --git a/arch/loongarch/include/asm/asmmacro.h b/arch/loongarch/include/asm/asmmacro.h index 328bb956f241..6ebad458d662 100644 --- a/arch/loongarch/include/asm/asmmacro.h +++ b/arch/loongarch/include/asm/asmmacro.h @@ -667,4 +667,19 @@ nor \dst, \src, zero .endm +.macro la.abs reg, sym +766: + nop + nop + nop + nop + .pushsection ".laabs", "aw", %progbits +768: + .word 768b-766b + parse_r regno, \reg + .word regno + .dword \sym + .popsection +.endm + #endif /* _ASM_ASMMACRO_H */ diff --git a/arch/loongarch/kernel/head.S b/arch/loongarch/kernel/head.S index d2ac26b5b22b..3b273f05be8c 100644 --- a/arch/loongarch/kernel/head.S +++ b/arch/loongarch/kernel/head.S @@ -86,6 +86,7 @@ SYM_CODE_START(kernel_entry) # kernel entry point PTR_ADD sp, sp, tp set_saved_sp sp, t0, t1 + bl relocate_laabs bl start_kernel ASM_BUG() diff --git a/arch/loongarch/kernel/setup.c b/arch/loongarch/kernel/setup.c index 4344502c0b31..9f8833a2524a 100644 --- a/arch/loongarch/kernel/setup.c +++ b/arch/loongarch/kernel/setup.c @@ -582,3 +582,30 @@ void __init setup_arch(char **cmdline_p) paging_init(); } + +void __init relocate_laabs(void) +{ + extern void *__laabs_begin; + extern void *__laabs_end; + struct laabs { + int offset; + int reg; + long symvalue; + } *p; + + for (p = (void *)&__laabs_begin; (void *)p < (void *)&__laabs_end; p++) + { + int lu12iw, ori, lu32id, lu52id; + long v = p->symvalue; + int reg = p->reg; + int *insn = (void *)p - p->offset; + lu12iw = 0x14000000 | reg | (((v & 0xfffff000) >> 12) << 5); + ori = 0x03800000 | reg | (reg<<5) | ((v & 0xfff) << 10); + lu32id = 0x16000000 | reg | (((v & 0x000fffff00000000) >> 32) << 5); + lu52id = 0x03000000 | reg | (reg<<5) | (((v >> 52) & 0xfff) << 10); + insn[0] = lu12iw; + insn[1] = ori; + insn[2] = lu32id; + insn[3] = lu52id; + } +} diff --git a/arch/loongarch/kernel/vmlinux.lds.S b/arch/loongarch/kernel/vmlinux.lds.S index 733b16e8d55d..4d128e089393 100644 --- a/arch/loongarch/kernel/vmlinux.lds.S +++ b/arch/loongarch/kernel/vmlinux.lds.S @@ -66,6 +66,13 @@ SECTIONS __alt_instructions_end = .; } + . = ALIGN(4); + .laabs : AT(ADDR(.laabs) - LOAD_OFFSET) { + __laabs_begin = .; + *(.laabs) + __laabs_end = .; + } + .got : ALIGN(16) { *(.got) } .plt : ALIGN(16) { *(.plt) } .got.plt : ALIGN(16) { *(.got.plt) } ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [PATCH v4 2/5] LoongArch: Use la.pcrel instead of la.abs for exception handlers 2023-02-16 6:59 ` Jinyang He @ 2023-02-16 7:10 ` Xi Ruoyao 2023-02-16 8:03 ` Youling Tang 0 siblings, 1 reply; 22+ messages in thread From: Xi Ruoyao @ 2023-02-16 7:10 UTC (permalink / raw) To: Jinyang He, Youling Tang, Huacai Chen Cc: Xuerui Wang, loongarch, linux-kernel, Xuefeng Li, Jianmin lv, Tiezhu Yang On Thu, 2023-02-16 at 14:59 +0800, Jinyang He wrote: > +.macro la.abs reg, sym > +766: > + nop > + nop > + nop > + nop In the "formal" version we can code lu12i.w reg, 0 ori reg, reg, 0 lu32i.d reg, 0 lu52i.d reg, reg, 0 here. Then we only need to fixup the immediate slot so we can avoid using parse_r. > + .pushsection ".laabs", "aw", %progbits > +768: > + .word 768b-766b > + parse_r regno, \reg > + .word regno > + .dword \sym > + .popsection > +.endm -- Xi Ruoyao <xry111@xry111.site> School of Aerospace Science and Technology, Xidian University ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [PATCH v4 2/5] LoongArch: Use la.pcrel instead of la.abs for exception handlers 2023-02-16 7:10 ` Xi Ruoyao @ 2023-02-16 8:03 ` Youling Tang 2023-02-16 11:18 ` Youling Tang 0 siblings, 1 reply; 22+ messages in thread From: Youling Tang @ 2023-02-16 8:03 UTC (permalink / raw) To: Xi Ruoyao, Jinyang He, Huacai Chen Cc: Xuerui Wang, loongarch, linux-kernel, Xuefeng Li, Jianmin lv, Tiezhu Yang On 02/16/2023 03:10 PM, Xi Ruoyao wrote: > On Thu, 2023-02-16 at 14:59 +0800, Jinyang He wrote: >> +.macro la.abs reg, sym >> +766: >> + nop >> + nop >> + nop >> + nop > > In the "formal" version we can code > > lu12i.w reg, 0 > ori reg, reg, 0 > lu32i.d reg, 0 > lu52i.d reg, reg, 0 > > here. Then we only need to fixup the immediate slot so we can avoid > using parse_r. > > >> + .pushsection ".laabs", "aw", %progbits >> +768: >> + .word 768b-766b >> + parse_r regno, \reg >> + .word regno >> + .dword \sym >> + .popsection >> +.endm I will try to modify a version for testing, using the following definition, when the RELOCATABLE is turned on, the "la.abs macro" is used, otherwise the "la.abs pseudo instruction" is still used as before. #ifdef CONFIG_RELOCATABLE .macro la.abs reg, sym lu12i.w reg, 0 ori reg, reg, 0 lu32i.d reg, 0 lu52i.d reg, reg, 0 .endm #endif Youling. ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [PATCH v4 2/5] LoongArch: Use la.pcrel instead of la.abs for exception handlers 2023-02-16 8:03 ` Youling Tang @ 2023-02-16 11:18 ` Youling Tang 2023-02-16 11:29 ` Youling Tang 0 siblings, 1 reply; 22+ messages in thread From: Youling Tang @ 2023-02-16 11:18 UTC (permalink / raw) To: Xi Ruoyao, Jinyang He, Huacai Chen Cc: Xuerui Wang, loongarch, linux-kernel, Xuefeng Li, Jianmin lv, Tiezhu Yang On 02/16/2023 04:03 PM, Youling Tang wrote: > > > On 02/16/2023 03:10 PM, Xi Ruoyao wrote: >> On Thu, 2023-02-16 at 14:59 +0800, Jinyang He wrote: >>> +.macro la.abs reg, sym >>> +766: >>> + nop >>> + nop >>> + nop >>> + nop >> >> In the "formal" version we can code >> >> lu12i.w reg, 0 >> ori reg, reg, 0 >> lu32i.d reg, 0 >> lu52i.d reg, reg, 0 >> >> here. Then we only need to fixup the immediate slot so we can avoid >> using parse_r. >> >> >>> + .pushsection ".laabs", "aw", %progbits >>> +768: >>> + .word 768b-766b >>> + parse_r regno, \reg >>> + .word regno >>> + .dword \sym >>> + .popsection >>> +.endm > > I will try to modify a version for testing, using the following > definition, when the RELOCATABLE is turned on, the "la.abs macro" is > used, otherwise the "la.abs pseudo instruction" is still used as before. > > #ifdef CONFIG_RELOCATABLE > .macro la.abs reg, sym > lu12i.w reg, 0 > ori reg, reg, 0 > lu32i.d reg, 0 > lu52i.d reg, reg, 0 > .endm > #endif On the basis of the v4 patch set, remove patch2, and then add the following patches, and the test is successful on qemu. If this method is more acceptable to everyone, I will send v5. diff --git a/arch/loongarch/include/asm/asmmacro.h b/arch/loongarch/include/asm/asmmacro.h index 328bb956f241..adb04ae6b208 100644 --- a/arch/loongarch/include/asm/asmmacro.h +++ b/arch/loongarch/include/asm/asmmacro.h @@ -667,4 +667,19 @@ nor \dst, \src, zero .endm +#ifdef CONFIG_RELOCATABLE +.macro la.abs reg, sym +766: + lu12i.w \reg, 0 + ori \reg, \reg, 0 + lu32i.d \reg, 0 + lu52i.d \reg, \reg, 0 + .pushsection ".laabs", "aw", %progbits +768: + .dword 768b-766b + .dword \sym + .popsection +.endm +#endif + #endif /* _ASM_ASMMACRO_H */ diff --git a/arch/loongarch/kernel/relocate.c b/arch/loongarch/kernel/relocate.c index 7d19cc0d2185..7ad327a554f9 100644 --- a/arch/loongarch/kernel/relocate.c +++ b/arch/loongarch/kernel/relocate.c @@ -12,6 +12,7 @@ #include <linux/start_kernel.h> #include <asm/bootinfo.h> #include <asm/early_ioremap.h> +#include <asm/inst.h> #include <asm/sections.h> #define RELOCATED(x) ((void *)((long)x + reloc_offset)) @@ -45,6 +46,32 @@ static inline __init void relocate_relative(void) } } +static inline void __init relocate_laabs(long offset) +{ + extern void *__laabs_begin; + extern void *__laabs_end; + struct laabs { + long offset; + long symvalue; + } *p; + + for (p = (void *)&__laabs_begin; (void *)p < (void *)&__laabs_end; p++) { + long v = p->symvalue + reloc_offset; + union loongarch_instruction *insn = (void *)p - p->offset + offset; + u32 lu12iw, ori, lu32id, lu52id; + + lu12iw = (v >> 12) & 0xfffff; + ori = v & 0xfff; + lu32id = (v >> 32) & 0xfffff; + lu52id = v >> 52; + + insn[0].reg1i20_format.immediate = lu12iw; + insn[1].reg2i12_format.immediate = ori; + insn[2].reg1i20_format.immediate = lu32id; + insn[3].reg2i12_format.immediate = lu52id; + } +} + #ifdef CONFIG_RANDOMIZE_BASE static inline __init unsigned long rotate_xor(unsigned long hash, const void *area, size_t size) @@ -168,8 +195,10 @@ void *__init do_kaslr(void) update_reloc_offset(&reloc_offset, offset); } - if (reloc_offset) + if (reloc_offset) { relocate_relative(); + relocate_laabs(offset); + } return kernel_entry; } @@ -181,6 +210,8 @@ void __init relocate_kernel(void) if (reloc_offset) relocate_relative(); + + relocate_laabs(0); } /* diff --git a/arch/loongarch/kernel/vmlinux.lds.S b/arch/loongarch/kernel/vmlinux.lds.S index aec0b6567d24..0e58c68bf427 100644 --- a/arch/loongarch/kernel/vmlinux.lds.S +++ b/arch/loongarch/kernel/vmlinux.lds.S @@ -66,6 +66,13 @@ SECTIONS __alt_instructions_end = .; } + . = ALIGN(8); + .laabs : AT(ADDR(.laabs) - LOAD_OFFSET) { + __laabs_begin = .; + *(.laabs) + __laabs_end = .; + } + .got : ALIGN(16) { *(.got) } Youling. ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [PATCH v4 2/5] LoongArch: Use la.pcrel instead of la.abs for exception handlers 2023-02-16 11:18 ` Youling Tang @ 2023-02-16 11:29 ` Youling Tang 0 siblings, 0 replies; 22+ messages in thread From: Youling Tang @ 2023-02-16 11:29 UTC (permalink / raw) To: Xi Ruoyao, Jinyang He, Huacai Chen Cc: Xuerui Wang, loongarch, linux-kernel, Xuefeng Li, Jianmin lv, Tiezhu Yang On 02/16/2023 07:18 PM, Youling Tang wrote: > > On 02/16/2023 04:03 PM, Youling Tang wrote: >> >> >> On 02/16/2023 03:10 PM, Xi Ruoyao wrote: >>> On Thu, 2023-02-16 at 14:59 +0800, Jinyang He wrote: >>>> +.macro la.abs reg, sym >>>> +766: >>>> + nop >>>> + nop >>>> + nop >>>> + nop >>> >>> In the "formal" version we can code >>> >>> lu12i.w reg, 0 >>> ori reg, reg, 0 >>> lu32i.d reg, 0 >>> lu52i.d reg, reg, 0 >>> >>> here. Then we only need to fixup the immediate slot so we can avoid >>> using parse_r. >>> >>> >>>> + .pushsection ".laabs", "aw", %progbits >>>> +768: >>>> + .word 768b-766b >>>> + parse_r regno, \reg >>>> + .word regno >>>> + .dword \sym >>>> + .popsection >>>> +.endm >> >> I will try to modify a version for testing, using the following >> definition, when the RELOCATABLE is turned on, the "la.abs macro" is >> used, otherwise the "la.abs pseudo instruction" is still used as before. >> >> #ifdef CONFIG_RELOCATABLE >> .macro la.abs reg, sym >> lu12i.w reg, 0 >> ori reg, reg, 0 >> lu32i.d reg, 0 >> lu52i.d reg, reg, 0 >> .endm >> #endif > > On the basis of the v4 patch set, remove patch2, and then add the > following patches, and the test is successful on qemu. > > If this method is more acceptable to everyone, I will send v5. > > diff --git a/arch/loongarch/include/asm/asmmacro.h > b/arch/loongarch/include/asm/asmmacro.h > index 328bb956f241..adb04ae6b208 100644 > --- a/arch/loongarch/include/asm/asmmacro.h > +++ b/arch/loongarch/include/asm/asmmacro.h > @@ -667,4 +667,19 @@ > nor \dst, \src, zero > .endm > > +#ifdef CONFIG_RELOCATABLE > +.macro la.abs reg, sym > +766: > + lu12i.w \reg, 0 > + ori \reg, \reg, 0 > + lu32i.d \reg, 0 > + lu52i.d \reg, \reg, 0 > + .pushsection ".laabs", "aw", %progbits > +768: > + .dword 768b-766b > + .dword \sym > + .popsection > +.endm > +#endif > + > #endif /* _ASM_ASMMACRO_H */ > diff --git a/arch/loongarch/kernel/relocate.c > b/arch/loongarch/kernel/relocate.c > index 7d19cc0d2185..7ad327a554f9 100644 > --- a/arch/loongarch/kernel/relocate.c > +++ b/arch/loongarch/kernel/relocate.c > @@ -12,6 +12,7 @@ > #include <linux/start_kernel.h> > #include <asm/bootinfo.h> > #include <asm/early_ioremap.h> > +#include <asm/inst.h> > #include <asm/sections.h> > > #define RELOCATED(x) ((void *)((long)x + reloc_offset)) > @@ -45,6 +46,32 @@ static inline __init void relocate_relative(void) > } > } > > +static inline void __init relocate_laabs(long offset) > +{ > + extern void *__laabs_begin; > + extern void *__laabs_end; > + struct laabs { > + long offset; > + long symvalue; > + } *p; > + > + for (p = (void *)&__laabs_begin; (void *)p < (void > *)&__laabs_end; p++) { > + long v = p->symvalue + reloc_offset; > + union loongarch_instruction *insn = (void *)p - > p->offset + offset; > + u32 lu12iw, ori, lu32id, lu52id; > + > + lu12iw = (v >> 12) & 0xfffff; > + ori = v & 0xfff; > + lu32id = (v >> 32) & 0xfffff; > + lu52id = v >> 52; > + > + insn[0].reg1i20_format.immediate = lu12iw; > + insn[1].reg2i12_format.immediate = ori; > + insn[2].reg1i20_format.immediate = lu32id; > + insn[3].reg2i12_format.immediate = lu52id; > + } > +} > + > #ifdef CONFIG_RANDOMIZE_BASE > static inline __init unsigned long rotate_xor(unsigned long hash, > const void *area, size_t > size) > @@ -168,8 +195,10 @@ void *__init do_kaslr(void) > update_reloc_offset(&reloc_offset, offset); > } > > - if (reloc_offset) > + if (reloc_offset) { > relocate_relative(); > + relocate_laabs(offset); > + } Self review: if (reloc_offset) relocate_relative(); relocate_laabs(offset); > > return kernel_entry; > } > @@ -181,6 +210,8 @@ void __init relocate_kernel(void) > > if (reloc_offset) > relocate_relative(); > + > + relocate_laabs(0); > } > > /* > diff --git a/arch/loongarch/kernel/vmlinux.lds.S > b/arch/loongarch/kernel/vmlinux.lds.S > index aec0b6567d24..0e58c68bf427 100644 > --- a/arch/loongarch/kernel/vmlinux.lds.S > +++ b/arch/loongarch/kernel/vmlinux.lds.S > @@ -66,6 +66,13 @@ SECTIONS > __alt_instructions_end = .; > } > > + . = ALIGN(8); > + .laabs : AT(ADDR(.laabs) - LOAD_OFFSET) { > + __laabs_begin = .; > + *(.laabs) > + __laabs_end = .; > + } > + > .got : ALIGN(16) { *(.got) } > > > Youling. > ^ permalink raw reply [flat|nested] 22+ messages in thread
* [PATCH v4 3/5] LoongArch: Add JUMP_VIRT_ADDR macro implementation to avoid using la.abs 2023-02-10 8:47 [PATCH v4 0/5] LoongArch: Add kernel relocation and KASLR support Youling Tang 2023-02-10 8:47 ` [PATCH v4 1/5] LoongArch: Use la.pcrel instead of la.abs when it's trivially possible Youling Tang 2023-02-10 8:47 ` [PATCH v4 2/5] LoongArch: Use la.pcrel instead of la.abs for exception handlers Youling Tang @ 2023-02-10 8:47 ` Youling Tang 2023-02-10 8:47 ` [PATCH v4 4/5] LoongArch: Add support for kernel relocation Youling Tang 2023-02-10 8:47 ` [PATCH v4 5/5] LoongArch: Add support for kernel address space layout randomization (KASLR) Youling Tang 4 siblings, 0 replies; 22+ messages in thread From: Youling Tang @ 2023-02-10 8:47 UTC (permalink / raw) To: Huacai Chen, Xi Ruoyao, Jinyang He; +Cc: Xuerui Wang, loongarch, linux-kernel Add JUMP_VIRT_ADDR macro implementation to avoid using la.abs. Signed-off-by: Youling Tang <tangyouling@loongson.cn> --- arch/loongarch/include/asm/stackframe.h | 9 +++++++++ arch/loongarch/kernel/head.S | 12 ++++-------- arch/loongarch/power/suspend_asm.S | 5 ++--- 3 files changed, 15 insertions(+), 11 deletions(-) diff --git a/arch/loongarch/include/asm/stackframe.h b/arch/loongarch/include/asm/stackframe.h index bbec1e56b61b..9a14bbb4cf36 100644 --- a/arch/loongarch/include/asm/stackframe.h +++ b/arch/loongarch/include/asm/stackframe.h @@ -10,6 +10,7 @@ #include <asm/asm.h> #include <asm/asmmacro.h> #include <asm/asm-offsets.h> +#include <asm/addrspace.h> #include <asm/loongarch.h> #include <asm/thread_info.h> @@ -217,4 +218,12 @@ RESTORE_SP_AND_RET \docfi .endm +/* Jump to the virtual address. */ + .macro JUMP_VIRT_ADDR temp1 temp2 + li.d \temp1, CACHE_BASE + pcaddi \temp2, 0 + or \temp1, \temp1, \temp2 + jirl zero, \temp1, 0xc + .endm + #endif /* _ASM_STACKFRAME_H */ diff --git a/arch/loongarch/kernel/head.S b/arch/loongarch/kernel/head.S index aa6181714ec3..d2ac26b5b22b 100644 --- a/arch/loongarch/kernel/head.S +++ b/arch/loongarch/kernel/head.S @@ -50,11 +50,8 @@ SYM_CODE_START(kernel_entry) # kernel entry point li.d t0, CSR_DMW1_INIT # CA, PLV0, 0x9000 xxxx xxxx xxxx csrwr t0, LOONGARCH_CSR_DMWIN1 - /* We might not get launched at the address the kernel is linked to, - so we jump there. */ - la.abs t0, 0f - jr t0 -0: + JUMP_VIRT_ADDR t0, t1 + /* Enable PG */ li.w t0, 0xb0 # PLV=0, IE=0, PG=1 csrwr t0, LOONGARCH_CSR_CRMD @@ -106,9 +103,8 @@ SYM_CODE_START(smpboot_entry) li.d t0, CSR_DMW1_INIT # CA, PLV0 csrwr t0, LOONGARCH_CSR_DMWIN1 - la.abs t0, 0f - jr t0 -0: + JUMP_VIRT_ADDR t0, t1 + /* Enable PG */ li.w t0, 0xb0 # PLV=0, IE=0, PG=1 csrwr t0, LOONGARCH_CSR_CRMD diff --git a/arch/loongarch/power/suspend_asm.S b/arch/loongarch/power/suspend_asm.S index eb2675642f9f..90da899c06a1 100644 --- a/arch/loongarch/power/suspend_asm.S +++ b/arch/loongarch/power/suspend_asm.S @@ -78,9 +78,8 @@ SYM_INNER_LABEL(loongarch_wakeup_start, SYM_L_GLOBAL) li.d t0, CSR_DMW1_INIT # CA, PLV0 csrwr t0, LOONGARCH_CSR_DMWIN1 - la.abs t0, 0f - jr t0 -0: + JUMP_VIRT_ADDR t0, t1 + la.pcrel t0, acpi_saved_sp ld.d sp, t0, 0 SETUP_WAKEUP -- 2.37.3 ^ permalink raw reply [flat|nested] 22+ messages in thread
* [PATCH v4 4/5] LoongArch: Add support for kernel relocation 2023-02-10 8:47 [PATCH v4 0/5] LoongArch: Add kernel relocation and KASLR support Youling Tang ` (2 preceding siblings ...) 2023-02-10 8:47 ` [PATCH v4 3/5] LoongArch: Add JUMP_VIRT_ADDR macro implementation to avoid using la.abs Youling Tang @ 2023-02-10 8:47 ` Youling Tang 2023-02-10 9:05 ` Huacai Chen 2023-02-10 18:01 ` kernel test robot 2023-02-10 8:47 ` [PATCH v4 5/5] LoongArch: Add support for kernel address space layout randomization (KASLR) Youling Tang 4 siblings, 2 replies; 22+ messages in thread From: Youling Tang @ 2023-02-10 8:47 UTC (permalink / raw) To: Huacai Chen, Xi Ruoyao, Jinyang He; +Cc: Xuerui Wang, loongarch, linux-kernel This config allows to compile kernel as PIE and to relocate it at any virtual address at runtime: this paves the way to KASLR. Runtime relocation is possible since relocation metadata are embedded into the kernel. Signed-off-by: Youling Tang <tangyouling@loongson.cn> Signed-off-by: Xi Ruoyao <xry111@xry111.site> # Use arch_initcall --- arch/loongarch/Kconfig | 15 ++++++ arch/loongarch/Makefile | 5 ++ arch/loongarch/kernel/Makefile | 2 + arch/loongarch/kernel/head.S | 5 ++ arch/loongarch/kernel/relocate.c | 78 +++++++++++++++++++++++++++++ arch/loongarch/kernel/vmlinux.lds.S | 11 +++- 6 files changed, 114 insertions(+), 2 deletions(-) create mode 100644 arch/loongarch/kernel/relocate.c diff --git a/arch/loongarch/Kconfig b/arch/loongarch/Kconfig index 9cc8b84f7eb0..089a4695b1b3 100644 --- a/arch/loongarch/Kconfig +++ b/arch/loongarch/Kconfig @@ -48,6 +48,7 @@ config LOONGARCH select ARCH_SUPPORTS_ATOMIC_RMW select ARCH_SUPPORTS_HUGETLBFS select ARCH_SUPPORTS_NUMA_BALANCING + select SYS_SUPPORTS_RELOCATABLE select ARCH_USE_BUILTIN_BSWAP select ARCH_USE_CMPXCHG_LOCKREF select ARCH_USE_QUEUED_RWLOCKS @@ -229,6 +230,11 @@ config SCHED_OMIT_FRAME_POINTER config AS_HAS_EXPLICIT_RELOCS def_bool $(as-instr,x:pcalau12i \$t0$(comma)%pc_hi20(x)) +config SYS_SUPPORTS_RELOCATABLE + bool + help + Selected if the platform supports relocating the kernel. + menu "Kernel type and options" source "kernel/Kconfig.hz" @@ -474,6 +480,15 @@ config PHYSICAL_START specified in the "crashkernel=YM@XM" command line boot parameter passed to the panic-ed kernel). +config RELOCATABLE + bool "Relocatable kernel" + depends on SYS_SUPPORTS_RELOCATABLE + help + This builds the kernel as a Position Independent Executable (PIE), + which retains all relocation metadata required to relocate the + kernel binary at runtime to a different virtual address than the + address it was linked at. + config SECCOMP bool "Enable seccomp to safely compute untrusted bytecode" depends on PROC_FS diff --git a/arch/loongarch/Makefile b/arch/loongarch/Makefile index 4402387d2755..27b5a70ff31c 100644 --- a/arch/loongarch/Makefile +++ b/arch/loongarch/Makefile @@ -71,6 +71,11 @@ KBUILD_AFLAGS_MODULE += -Wa,-mla-global-with-abs KBUILD_CFLAGS_MODULE += -fplt -Wa,-mla-global-with-abs,-mla-local-with-abs endif +ifeq ($(CONFIG_RELOCATABLE),y) +LDFLAGS_vmlinux += -static -pie --no-dynamic-linker -z notext +KBUILD_CFLAGS_KERNEL += -fPIE +endif + cflags-y += -ffreestanding cflags-y += $(call cc-option, -mno-check-zero-division) diff --git a/arch/loongarch/kernel/Makefile b/arch/loongarch/kernel/Makefile index c8cfbd562921..3341dd5f0926 100644 --- a/arch/loongarch/kernel/Makefile +++ b/arch/loongarch/kernel/Makefile @@ -31,6 +31,8 @@ endif obj-$(CONFIG_MODULES) += module.o module-sections.o obj-$(CONFIG_STACKTRACE) += stacktrace.o +obj-$(CONFIG_RELOCATABLE) += relocate.o + obj-$(CONFIG_PROC_FS) += proc.o obj-$(CONFIG_SMP) += smp.o diff --git a/arch/loongarch/kernel/head.S b/arch/loongarch/kernel/head.S index d2ac26b5b22b..499edc80d8ab 100644 --- a/arch/loongarch/kernel/head.S +++ b/arch/loongarch/kernel/head.S @@ -86,6 +86,11 @@ SYM_CODE_START(kernel_entry) # kernel entry point PTR_ADD sp, sp, tp set_saved_sp sp, t0, t1 +#ifdef CONFIG_RELOCATABLE + /* Apply the relocations */ + bl relocate_kernel +#endif + bl start_kernel ASM_BUG() diff --git a/arch/loongarch/kernel/relocate.c b/arch/loongarch/kernel/relocate.c new file mode 100644 index 000000000000..91ce92433bab --- /dev/null +++ b/arch/loongarch/kernel/relocate.c @@ -0,0 +1,78 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * Support for Kernel relocation at boot time + * + * Copyright (C) 2023 Loongson Technology Corporation Limited + */ + +#include <linux/elf.h> +#include <linux/kernel.h> +#include <linux/printk.h> +#include <linux/panic_notifier.h> +#include <asm/sections.h> + +#define RELOCATED(x) ((void *)((long)x + reloc_offset)) + +extern long __rela_dyn_start; +extern long __rela_dyn_end; + +static unsigned long reloc_offset; + +void __init relocate_kernel(void) +{ + reloc_offset = (unsigned long)_text - VMLINUX_LOAD_ADDRESS; + + if (reloc_offset) { + Elf64_Rela *rela, *rela_end; + rela = (Elf64_Rela *)&__rela_dyn_start; + rela_end = (Elf64_Rela *)&__rela_dyn_end; + + for ( ; rela < rela_end; rela++) { + Elf64_Addr addr = rela->r_offset; + Elf64_Addr relocated_addr = rela->r_addend; + + if (rela->r_info != R_LARCH_RELATIVE) + continue; + + if (relocated_addr >= VMLINUX_LOAD_ADDRESS) + relocated_addr = + (Elf64_Addr)RELOCATED(relocated_addr); + + *(Elf64_Addr *)RELOCATED(addr) = relocated_addr; + } + } +} + +/* + * Show relocation information on panic. + */ +static void show_kernel_relocation(const char *level) +{ + if (reloc_offset > 0) { + printk(level); + pr_cont("Kernel relocated offset @ 0x%lx\n", reloc_offset); + pr_cont(" .text @ 0x%lx\n", (unsigned long)&_text); + pr_cont(" .data @ 0x%lx\n", (unsigned long)&_sdata); + pr_cont(" .bss @ 0x%lx\n", (unsigned long)&__bss_start); + } +} + +static int kernel_location_notifier_fn(struct notifier_block *self, + unsigned long v, void *p) +{ + show_kernel_relocation(KERN_EMERG); + return NOTIFY_DONE; +} + +static struct notifier_block kernel_location_notifier = { + .notifier_call = kernel_location_notifier_fn +}; + +static int __init register_kernel_offset_dumper(void) +{ + atomic_notifier_chain_register(&panic_notifier_list, + &kernel_location_notifier); + return 0; +} + +arch_initcall(register_kernel_offset_dumper); diff --git a/arch/loongarch/kernel/vmlinux.lds.S b/arch/loongarch/kernel/vmlinux.lds.S index 733b16e8d55d..aec0b6567d24 100644 --- a/arch/loongarch/kernel/vmlinux.lds.S +++ b/arch/loongarch/kernel/vmlinux.lds.S @@ -70,6 +70,8 @@ SECTIONS .plt : ALIGN(16) { *(.plt) } .got.plt : ALIGN(16) { *(.got.plt) } + .data.rel : { *(.data.rel*) } + . = ALIGN(PECOFF_SEGMENT_ALIGN); __init_begin = .; __inittext_begin = .; @@ -93,8 +95,6 @@ SECTIONS PERCPU_SECTION(1 << CONFIG_L1_CACHE_SHIFT) #endif - .rela.dyn : ALIGN(8) { *(.rela.dyn) *(.rela*) } - .init.bss : { *(.init.bss) } @@ -107,6 +107,12 @@ SECTIONS RO_DATA(4096) RW_DATA(1 << CONFIG_L1_CACHE_SHIFT, PAGE_SIZE, THREAD_SIZE) + .rela.dyn : ALIGN(8) { + __rela_dyn_start = .; + *(.rela.dyn) *(.rela*) + __rela_dyn_end = .; + } + .sdata : { *(.sdata) } @@ -133,6 +139,7 @@ SECTIONS DISCARDS /DISCARD/ : { + *(.dynamic .dynsym .dynstr .hash .gnu.hash) *(.gnu.attributes) *(.options) *(.eh_frame) -- 2.37.3 ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [PATCH v4 4/5] LoongArch: Add support for kernel relocation 2023-02-10 8:47 ` [PATCH v4 4/5] LoongArch: Add support for kernel relocation Youling Tang @ 2023-02-10 9:05 ` Huacai Chen 2023-02-10 18:01 ` kernel test robot 1 sibling, 0 replies; 22+ messages in thread From: Huacai Chen @ 2023-02-10 9:05 UTC (permalink / raw) To: Youling Tang; +Cc: Xi Ruoyao, Jinyang He, Xuerui Wang, loongarch, linux-kernel Hi, Youling, On Fri, Feb 10, 2023 at 4:48 PM Youling Tang <tangyouling@loongson.cn> wrote: > > This config allows to compile kernel as PIE and to relocate it at > any virtual address at runtime: this paves the way to KASLR. > Runtime relocation is possible since relocation metadata are embedded > into the kernel. > > Signed-off-by: Youling Tang <tangyouling@loongson.cn> > Signed-off-by: Xi Ruoyao <xry111@xry111.site> # Use arch_initcall > --- > arch/loongarch/Kconfig | 15 ++++++ > arch/loongarch/Makefile | 5 ++ > arch/loongarch/kernel/Makefile | 2 + > arch/loongarch/kernel/head.S | 5 ++ > arch/loongarch/kernel/relocate.c | 78 +++++++++++++++++++++++++++++ > arch/loongarch/kernel/vmlinux.lds.S | 11 +++- > 6 files changed, 114 insertions(+), 2 deletions(-) > create mode 100644 arch/loongarch/kernel/relocate.c > > diff --git a/arch/loongarch/Kconfig b/arch/loongarch/Kconfig > index 9cc8b84f7eb0..089a4695b1b3 100644 > --- a/arch/loongarch/Kconfig > +++ b/arch/loongarch/Kconfig > @@ -48,6 +48,7 @@ config LOONGARCH > select ARCH_SUPPORTS_ATOMIC_RMW > select ARCH_SUPPORTS_HUGETLBFS > select ARCH_SUPPORTS_NUMA_BALANCING > + select SYS_SUPPORTS_RELOCATABLE We don't need such a Kconfig option since it is always true. Huacai > select ARCH_USE_BUILTIN_BSWAP > select ARCH_USE_CMPXCHG_LOCKREF > select ARCH_USE_QUEUED_RWLOCKS > @@ -229,6 +230,11 @@ config SCHED_OMIT_FRAME_POINTER > config AS_HAS_EXPLICIT_RELOCS > def_bool $(as-instr,x:pcalau12i \$t0$(comma)%pc_hi20(x)) > > +config SYS_SUPPORTS_RELOCATABLE > + bool > + help > + Selected if the platform supports relocating the kernel. > + > menu "Kernel type and options" > > source "kernel/Kconfig.hz" > @@ -474,6 +480,15 @@ config PHYSICAL_START > specified in the "crashkernel=YM@XM" command line boot parameter > passed to the panic-ed kernel). > > +config RELOCATABLE > + bool "Relocatable kernel" > + depends on SYS_SUPPORTS_RELOCATABLE > + help > + This builds the kernel as a Position Independent Executable (PIE), > + which retains all relocation metadata required to relocate the > + kernel binary at runtime to a different virtual address than the > + address it was linked at. > + > config SECCOMP > bool "Enable seccomp to safely compute untrusted bytecode" > depends on PROC_FS > diff --git a/arch/loongarch/Makefile b/arch/loongarch/Makefile > index 4402387d2755..27b5a70ff31c 100644 > --- a/arch/loongarch/Makefile > +++ b/arch/loongarch/Makefile > @@ -71,6 +71,11 @@ KBUILD_AFLAGS_MODULE += -Wa,-mla-global-with-abs > KBUILD_CFLAGS_MODULE += -fplt -Wa,-mla-global-with-abs,-mla-local-with-abs > endif > > +ifeq ($(CONFIG_RELOCATABLE),y) > +LDFLAGS_vmlinux += -static -pie --no-dynamic-linker -z notext > +KBUILD_CFLAGS_KERNEL += -fPIE > +endif > + > cflags-y += -ffreestanding > cflags-y += $(call cc-option, -mno-check-zero-division) > > diff --git a/arch/loongarch/kernel/Makefile b/arch/loongarch/kernel/Makefile > index c8cfbd562921..3341dd5f0926 100644 > --- a/arch/loongarch/kernel/Makefile > +++ b/arch/loongarch/kernel/Makefile > @@ -31,6 +31,8 @@ endif > obj-$(CONFIG_MODULES) += module.o module-sections.o > obj-$(CONFIG_STACKTRACE) += stacktrace.o > > +obj-$(CONFIG_RELOCATABLE) += relocate.o > + > obj-$(CONFIG_PROC_FS) += proc.o > > obj-$(CONFIG_SMP) += smp.o > diff --git a/arch/loongarch/kernel/head.S b/arch/loongarch/kernel/head.S > index d2ac26b5b22b..499edc80d8ab 100644 > --- a/arch/loongarch/kernel/head.S > +++ b/arch/loongarch/kernel/head.S > @@ -86,6 +86,11 @@ SYM_CODE_START(kernel_entry) # kernel entry point > PTR_ADD sp, sp, tp > set_saved_sp sp, t0, t1 > > +#ifdef CONFIG_RELOCATABLE > + /* Apply the relocations */ > + bl relocate_kernel > +#endif > + > bl start_kernel > ASM_BUG() > > diff --git a/arch/loongarch/kernel/relocate.c b/arch/loongarch/kernel/relocate.c > new file mode 100644 > index 000000000000..91ce92433bab > --- /dev/null > +++ b/arch/loongarch/kernel/relocate.c > @@ -0,0 +1,78 @@ > +// SPDX-License-Identifier: GPL-2.0 > +/* > + * Support for Kernel relocation at boot time > + * > + * Copyright (C) 2023 Loongson Technology Corporation Limited > + */ > + > +#include <linux/elf.h> > +#include <linux/kernel.h> > +#include <linux/printk.h> > +#include <linux/panic_notifier.h> > +#include <asm/sections.h> > + > +#define RELOCATED(x) ((void *)((long)x + reloc_offset)) > + > +extern long __rela_dyn_start; > +extern long __rela_dyn_end; > + > +static unsigned long reloc_offset; > + > +void __init relocate_kernel(void) > +{ > + reloc_offset = (unsigned long)_text - VMLINUX_LOAD_ADDRESS; > + > + if (reloc_offset) { > + Elf64_Rela *rela, *rela_end; > + rela = (Elf64_Rela *)&__rela_dyn_start; > + rela_end = (Elf64_Rela *)&__rela_dyn_end; > + > + for ( ; rela < rela_end; rela++) { > + Elf64_Addr addr = rela->r_offset; > + Elf64_Addr relocated_addr = rela->r_addend; > + > + if (rela->r_info != R_LARCH_RELATIVE) > + continue; > + > + if (relocated_addr >= VMLINUX_LOAD_ADDRESS) > + relocated_addr = > + (Elf64_Addr)RELOCATED(relocated_addr); > + > + *(Elf64_Addr *)RELOCATED(addr) = relocated_addr; > + } > + } > +} > + > +/* > + * Show relocation information on panic. > + */ > +static void show_kernel_relocation(const char *level) > +{ > + if (reloc_offset > 0) { > + printk(level); > + pr_cont("Kernel relocated offset @ 0x%lx\n", reloc_offset); > + pr_cont(" .text @ 0x%lx\n", (unsigned long)&_text); > + pr_cont(" .data @ 0x%lx\n", (unsigned long)&_sdata); > + pr_cont(" .bss @ 0x%lx\n", (unsigned long)&__bss_start); > + } > +} > + > +static int kernel_location_notifier_fn(struct notifier_block *self, > + unsigned long v, void *p) > +{ > + show_kernel_relocation(KERN_EMERG); > + return NOTIFY_DONE; > +} > + > +static struct notifier_block kernel_location_notifier = { > + .notifier_call = kernel_location_notifier_fn > +}; > + > +static int __init register_kernel_offset_dumper(void) > +{ > + atomic_notifier_chain_register(&panic_notifier_list, > + &kernel_location_notifier); > + return 0; > +} > + > +arch_initcall(register_kernel_offset_dumper); > diff --git a/arch/loongarch/kernel/vmlinux.lds.S b/arch/loongarch/kernel/vmlinux.lds.S > index 733b16e8d55d..aec0b6567d24 100644 > --- a/arch/loongarch/kernel/vmlinux.lds.S > +++ b/arch/loongarch/kernel/vmlinux.lds.S > @@ -70,6 +70,8 @@ SECTIONS > .plt : ALIGN(16) { *(.plt) } > .got.plt : ALIGN(16) { *(.got.plt) } > > + .data.rel : { *(.data.rel*) } > + > . = ALIGN(PECOFF_SEGMENT_ALIGN); > __init_begin = .; > __inittext_begin = .; > @@ -93,8 +95,6 @@ SECTIONS > PERCPU_SECTION(1 << CONFIG_L1_CACHE_SHIFT) > #endif > > - .rela.dyn : ALIGN(8) { *(.rela.dyn) *(.rela*) } > - > .init.bss : { > *(.init.bss) > } > @@ -107,6 +107,12 @@ SECTIONS > RO_DATA(4096) > RW_DATA(1 << CONFIG_L1_CACHE_SHIFT, PAGE_SIZE, THREAD_SIZE) > > + .rela.dyn : ALIGN(8) { > + __rela_dyn_start = .; > + *(.rela.dyn) *(.rela*) > + __rela_dyn_end = .; > + } > + > .sdata : { > *(.sdata) > } > @@ -133,6 +139,7 @@ SECTIONS > > DISCARDS > /DISCARD/ : { > + *(.dynamic .dynsym .dynstr .hash .gnu.hash) > *(.gnu.attributes) > *(.options) > *(.eh_frame) > -- > 2.37.3 > ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [PATCH v4 4/5] LoongArch: Add support for kernel relocation 2023-02-10 8:47 ` [PATCH v4 4/5] LoongArch: Add support for kernel relocation Youling Tang 2023-02-10 9:05 ` Huacai Chen @ 2023-02-10 18:01 ` kernel test robot 1 sibling, 0 replies; 22+ messages in thread From: kernel test robot @ 2023-02-10 18:01 UTC (permalink / raw) To: Youling Tang, Huacai Chen, Xi Ruoyao, Jinyang He Cc: oe-kbuild-all, Xuerui Wang, loongarch, linux-kernel Hi Youling, Thank you for the patch! Perhaps something to improve: [auto build test WARNING on linus/master] [also build test WARNING on v6.2-rc7 next-20230210] [If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in https://git-scm.com/docs/git-format-patch#_base_tree_information] url: https://github.com/intel-lab-lkp/linux/commits/Youling-Tang/LoongArch-Use-la-pcrel-instead-of-la-abs-when-it-s-trivially-possible/20230210-165022 patch link: https://lore.kernel.org/r/1676018856-26520-5-git-send-email-tangyouling%40loongson.cn patch subject: [PATCH v4 4/5] LoongArch: Add support for kernel relocation config: loongarch-allmodconfig (https://download.01.org/0day-ci/archive/20230211/202302110150.tIuRIiTp-lkp@intel.com/config) compiler: loongarch64-linux-gcc (GCC) 12.1.0 reproduce (this is a W=1 build): wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross chmod +x ~/bin/make.cross # https://github.com/intel-lab-lkp/linux/commit/26dc7750408c7f232632db44fab905df7b48d83c git remote add linux-review https://github.com/intel-lab-lkp/linux git fetch --no-tags linux-review Youling-Tang/LoongArch-Use-la-pcrel-instead-of-la-abs-when-it-s-trivially-possible/20230210-165022 git checkout 26dc7750408c7f232632db44fab905df7b48d83c # save the config file mkdir build_dir && cp config build_dir/.config COMPILER_INSTALL_PATH=$HOME/0day COMPILER=gcc-12.1.0 make.cross W=1 O=build_dir ARCH=loongarch olddefconfig COMPILER_INSTALL_PATH=$HOME/0day COMPILER=gcc-12.1.0 make.cross W=1 O=build_dir ARCH=loongarch SHELL=/bin/bash arch/loongarch/kernel/ If you fix the issue, kindly add following tag where applicable | Reported-by: kernel test robot <lkp@intel.com> | Link: https://lore.kernel.org/oe-kbuild-all/202302110150.tIuRIiTp-lkp@intel.com/ Note: functions only called from assembly code should be annotated with the asmlinkage attribute All warnings (new ones prefixed by >>): >> arch/loongarch/kernel/relocate.c:21:13: warning: no previous prototype for 'relocate_kernel' [-Wmissing-prototypes] 21 | void __init relocate_kernel(void) | ^~~~~~~~~~~~~~~ vim +/relocate_kernel +21 arch/loongarch/kernel/relocate.c 20 > 21 void __init relocate_kernel(void) 22 { 23 reloc_offset = (unsigned long)_text - VMLINUX_LOAD_ADDRESS; 24 25 if (reloc_offset) { 26 Elf64_Rela *rela, *rela_end; 27 rela = (Elf64_Rela *)&__rela_dyn_start; 28 rela_end = (Elf64_Rela *)&__rela_dyn_end; 29 30 for ( ; rela < rela_end; rela++) { 31 Elf64_Addr addr = rela->r_offset; 32 Elf64_Addr relocated_addr = rela->r_addend; 33 34 if (rela->r_info != R_LARCH_RELATIVE) 35 continue; 36 37 if (relocated_addr >= VMLINUX_LOAD_ADDRESS) 38 relocated_addr = 39 (Elf64_Addr)RELOCATED(relocated_addr); 40 41 *(Elf64_Addr *)RELOCATED(addr) = relocated_addr; 42 } 43 } 44 } 45 -- 0-DAY CI Kernel Test Service https://github.com/intel/lkp-tests ^ permalink raw reply [flat|nested] 22+ messages in thread
* [PATCH v4 5/5] LoongArch: Add support for kernel address space layout randomization (KASLR) 2023-02-10 8:47 [PATCH v4 0/5] LoongArch: Add kernel relocation and KASLR support Youling Tang ` (3 preceding siblings ...) 2023-02-10 8:47 ` [PATCH v4 4/5] LoongArch: Add support for kernel relocation Youling Tang @ 2023-02-10 8:47 ` Youling Tang 2023-02-10 9:06 ` Youling Tang 2023-02-17 8:09 ` Xi Ruoyao 4 siblings, 2 replies; 22+ messages in thread From: Youling Tang @ 2023-02-10 8:47 UTC (permalink / raw) To: Huacai Chen, Xi Ruoyao, Jinyang He; +Cc: Xuerui Wang, loongarch, linux-kernel This patch adds support for relocating the kernel to a random address. Entropy is derived from the banner, which will change every build and random_get_entropy() which should provide additional runtime entropy. The kernel is relocated by up to RANDOMIZE_BASE_MAX_OFFSET bytes from its link address. Because relocation happens so early in the kernel boot, the amount of physical memory has not yet been determined. This means the only way to limit relocation within the available memory is via Kconfig. Limit the maximum value of RANDOMIZE_BASE_MAX_OFFSET to 256M(0x10000000) because our memory layout has many holes. Signed-off-by: Youling Tang <tangyouling@loongson.cn> Signed-off-by: Xi Ruoyao <xry111@xry111.site> # Fix compiler warnings --- arch/loongarch/Kconfig | 23 +++++ arch/loongarch/kernel/head.S | 14 ++- arch/loongarch/kernel/relocate.c | 143 ++++++++++++++++++++++++++++++- 3 files changed, 176 insertions(+), 4 deletions(-) diff --git a/arch/loongarch/Kconfig b/arch/loongarch/Kconfig index 089a4695b1b3..f0a070bd7254 100644 --- a/arch/loongarch/Kconfig +++ b/arch/loongarch/Kconfig @@ -489,6 +489,29 @@ config RELOCATABLE kernel binary at runtime to a different virtual address than the address it was linked at. +config RANDOMIZE_BASE + bool "Randomize the address of the kernel image (KASLR)" + depends on RELOCATABLE + help + Randomizes the physical and virtual address at which the + kernel image is loaded, as a security feature that + deters exploit attempts relying on knowledge of the location + of kernel internals. + + The kernel will be offset by up to RANDOMIZE_BASE_MAX_OFFSET. + + If unsure, say N. + +config RANDOMIZE_BASE_MAX_OFFSET + hex "Maximum KASLR offset" if EXPERT + depends on RANDOMIZE_BASE + range 0x0 0x10000000 if 64BIT + default "0x01000000" + help + When KASLR is active, this provides the maximum offset that will + be applied to the kernel image. + + config SECCOMP bool "Enable seccomp to safely compute untrusted bytecode" depends on PROC_FS diff --git a/arch/loongarch/kernel/head.S b/arch/loongarch/kernel/head.S index 499edc80d8ab..b12f459ad73a 100644 --- a/arch/loongarch/kernel/head.S +++ b/arch/loongarch/kernel/head.S @@ -87,10 +87,22 @@ SYM_CODE_START(kernel_entry) # kernel entry point set_saved_sp sp, t0, t1 #ifdef CONFIG_RELOCATABLE +#ifdef CONFIG_RANDOMIZE_BASE + bl do_kaslr + + /* Repoint the sp into the new kernel image */ + PTR_LI sp, (_THREAD_SIZE - PT_SIZE) + PTR_ADD sp, sp, tp + set_saved_sp sp, t0, t1 + + /* do_kaslr returns the new kernel image entry point */ + jr a0 + ASM_BUG() +#else /* Apply the relocations */ bl relocate_kernel #endif - +#endif bl start_kernel ASM_BUG() diff --git a/arch/loongarch/kernel/relocate.c b/arch/loongarch/kernel/relocate.c index 91ce92433bab..5266f23a3006 100644 --- a/arch/loongarch/kernel/relocate.c +++ b/arch/loongarch/kernel/relocate.c @@ -9,19 +9,21 @@ #include <linux/kernel.h> #include <linux/printk.h> #include <linux/panic_notifier.h> +#include <linux/start_kernel.h> +#include <asm/bootinfo.h> +#include <asm/early_ioremap.h> #include <asm/sections.h> #define RELOCATED(x) ((void *)((long)x + reloc_offset)) +#define RELOCATED_KASLR(x) ((void *)((long)x + offset)) extern long __rela_dyn_start; extern long __rela_dyn_end; static unsigned long reloc_offset; -void __init relocate_kernel(void) +static inline __init void relocate_relative(void) { - reloc_offset = (unsigned long)_text - VMLINUX_LOAD_ADDRESS; - if (reloc_offset) { Elf64_Rela *rela, *rela_end; rela = (Elf64_Rela *)&__rela_dyn_start; @@ -43,6 +45,141 @@ void __init relocate_kernel(void) } } +#ifdef CONFIG_RANDOMIZE_BASE +static inline __init unsigned long rotate_xor(unsigned long hash, + const void *area, size_t size) +{ + size_t i; + unsigned long *ptr = (unsigned long *)area; + + for (i = 0; i < size / sizeof(hash); i++) { + /* Rotate by odd number of bits and XOR. */ + hash = (hash << ((sizeof(hash) * 8) - 7)) | (hash >> 7); + hash ^= ptr[i]; + } + + return hash; +} + +static inline __init unsigned long get_random_boot(void) +{ + unsigned long entropy = random_get_entropy(); + unsigned long hash = 0; + + /* Attempt to create a simple but unpredictable starting entropy. */ + hash = rotate_xor(hash, linux_banner, strlen(linux_banner)); + + /* Add in any runtime entropy we can get */ + hash = rotate_xor(hash, &entropy, sizeof(entropy)); + + return hash; +} + +static inline __init bool kaslr_disabled(void) +{ + char *str; + + str = strstr(boot_command_line, "nokaslr"); + if (str == boot_command_line || (str > boot_command_line && *(str - 1) == ' ')) + return true; + + return false; +} + +/* Choose a new address for the kernel */ +static inline void __init *determine_relocation_address(void) +{ + unsigned long kernel_length; + void *dest = _text; + unsigned long offset; + + if (kaslr_disabled()) + return dest; + + kernel_length = (long)_end - (long)_text; + + offset = get_random_boot() << 16; + offset &= (CONFIG_RANDOMIZE_BASE_MAX_OFFSET - 1); + if (offset < kernel_length) + offset += ALIGN(kernel_length, 0xffff); + + return RELOCATED_KASLR(dest); +} + +static inline int __init relocation_addr_valid(void *loc_new) +{ + if ((unsigned long)loc_new & 0x00000ffff) { + /* Inappropriately aligned new location */ + return 0; + } + if ((unsigned long)loc_new < (unsigned long)_end) { + /* New location overlaps original kernel */ + return 0; + } + return 1; +} + +static inline void __init update_reloc_offset(unsigned long *addr, long offset) +{ + unsigned long *new_addr = (unsigned long *)RELOCATED_KASLR(addr); + + *new_addr = (unsigned long)offset; +} + +void *__init do_kaslr(void) +{ + void *loc_new; + unsigned long kernel_length; + long offset = 0; + /* Default to original kernel entry point */ + void *kernel_entry = start_kernel; + char *cmdline = early_ioremap(fw_arg1, COMMAND_LINE_SIZE); + + /* Boot command line was passed in fw_arg1 */ + strscpy(boot_command_line, cmdline, COMMAND_LINE_SIZE); + + kernel_length = (long)(_end) - (long)(_text); + + loc_new = determine_relocation_address(); + + /* Sanity check relocation address */ + if (relocation_addr_valid(loc_new)) + offset = (unsigned long)loc_new - (unsigned long)(_text); + + if (offset) { + /* Copy the kernel to it's new location */ + memcpy(loc_new, _text, kernel_length); + + /* Sync the caches ready for execution of new kernel */ + __asm__ __volatile__ ( + "ibar 0 \t\n" + "dbar 0 \t\n"); + + reloc_offset = (unsigned long)_text - VMLINUX_LOAD_ADDRESS; + reloc_offset += offset; + + relocate_relative(); + + /* The current thread is now within the relocated image */ + __current_thread_info = RELOCATED_KASLR(__current_thread_info); + + /* Return the new kernel's entry point */ + kernel_entry = RELOCATED_KASLR(start_kernel); + + update_reloc_offset(&reloc_offset, offset); + } + + return kernel_entry; +} +#endif + +void __init relocate_kernel(void) +{ + reloc_offset = (unsigned long)_text - VMLINUX_LOAD_ADDRESS; + + relocate_relative(); +} + /* * Show relocation information on panic. */ -- 2.37.3 ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [PATCH v4 5/5] LoongArch: Add support for kernel address space layout randomization (KASLR) 2023-02-10 8:47 ` [PATCH v4 5/5] LoongArch: Add support for kernel address space layout randomization (KASLR) Youling Tang @ 2023-02-10 9:06 ` Youling Tang 2023-02-10 9:37 ` Youling Tang 2023-02-17 8:09 ` Xi Ruoyao 1 sibling, 1 reply; 22+ messages in thread From: Youling Tang @ 2023-02-10 9:06 UTC (permalink / raw) To: Huacai Chen; +Cc: Xi Ruoyao, Jinyang He, Xuerui Wang, loongarch, linux-kernel On 02/10/2023 04:47 PM, Youling Tang wrote: > This patch adds support for relocating the kernel to a random address. > > Entropy is derived from the banner, which will change every build and > random_get_entropy() which should provide additional runtime entropy. > > The kernel is relocated by up to RANDOMIZE_BASE_MAX_OFFSET bytes from > its link address. Because relocation happens so early in the kernel boot, > the amount of physical memory has not yet been determined. This means > the only way to limit relocation within the available memory is via > Kconfig. Limit the maximum value of RANDOMIZE_BASE_MAX_OFFSET to > 256M(0x10000000) because our memory layout has many holes. > > Signed-off-by: Youling Tang <tangyouling@loongson.cn> > Signed-off-by: Xi Ruoyao <xry111@xry111.site> # Fix compiler warnings > --- > arch/loongarch/Kconfig | 23 +++++ > arch/loongarch/kernel/head.S | 14 ++- > arch/loongarch/kernel/relocate.c | 143 ++++++++++++++++++++++++++++++- > 3 files changed, 176 insertions(+), 4 deletions(-) > > diff --git a/arch/loongarch/Kconfig b/arch/loongarch/Kconfig > index 089a4695b1b3..f0a070bd7254 100644 > --- a/arch/loongarch/Kconfig > +++ b/arch/loongarch/Kconfig > @@ -489,6 +489,29 @@ config RELOCATABLE > kernel binary at runtime to a different virtual address than the > address it was linked at. > > +config RANDOMIZE_BASE > + bool "Randomize the address of the kernel image (KASLR)" > + depends on RELOCATABLE > + help > + Randomizes the physical and virtual address at which the > + kernel image is loaded, as a security feature that > + deters exploit attempts relying on knowledge of the location > + of kernel internals. > + > + The kernel will be offset by up to RANDOMIZE_BASE_MAX_OFFSET. > + > + If unsure, say N. > + > +config RANDOMIZE_BASE_MAX_OFFSET > + hex "Maximum KASLR offset" if EXPERT > + depends on RANDOMIZE_BASE > + range 0x0 0x10000000 if 64BIT > + default "0x01000000" > + help > + When KASLR is active, this provides the maximum offset that will > + be applied to the kernel image. > + > + > config SECCOMP > bool "Enable seccomp to safely compute untrusted bytecode" > depends on PROC_FS > diff --git a/arch/loongarch/kernel/head.S b/arch/loongarch/kernel/head.S > index 499edc80d8ab..b12f459ad73a 100644 > --- a/arch/loongarch/kernel/head.S > +++ b/arch/loongarch/kernel/head.S > @@ -87,10 +87,22 @@ SYM_CODE_START(kernel_entry) # kernel entry point > set_saved_sp sp, t0, t1 > > #ifdef CONFIG_RELOCATABLE > +#ifdef CONFIG_RANDOMIZE_BASE > + bl do_kaslr > + > + /* Repoint the sp into the new kernel image */ > + PTR_LI sp, (_THREAD_SIZE - PT_SIZE) > + PTR_ADD sp, sp, tp > + set_saved_sp sp, t0, t1 > + > + /* do_kaslr returns the new kernel image entry point */ > + jr a0 > + ASM_BUG() > +#else > /* Apply the relocations */ > bl relocate_kernel > #endif > - > +#endif > bl start_kernel > ASM_BUG() > > diff --git a/arch/loongarch/kernel/relocate.c b/arch/loongarch/kernel/relocate.c > index 91ce92433bab..5266f23a3006 100644 > --- a/arch/loongarch/kernel/relocate.c > +++ b/arch/loongarch/kernel/relocate.c > @@ -9,19 +9,21 @@ > #include <linux/kernel.h> > #include <linux/printk.h> > #include <linux/panic_notifier.h> > +#include <linux/start_kernel.h> > +#include <asm/bootinfo.h> > +#include <asm/early_ioremap.h> > #include <asm/sections.h> > > #define RELOCATED(x) ((void *)((long)x + reloc_offset)) > +#define RELOCATED_KASLR(x) ((void *)((long)x + offset)) > > extern long __rela_dyn_start; > extern long __rela_dyn_end; > > static unsigned long reloc_offset; > > -void __init relocate_kernel(void) > +static inline __init void relocate_relative(void) > { > - reloc_offset = (unsigned long)_text - VMLINUX_LOAD_ADDRESS; > - > if (reloc_offset) { > Elf64_Rela *rela, *rela_end; > rela = (Elf64_Rela *)&__rela_dyn_start; > @@ -43,6 +45,141 @@ void __init relocate_kernel(void) > } > } > > +#ifdef CONFIG_RANDOMIZE_BASE > +static inline __init unsigned long rotate_xor(unsigned long hash, > + const void *area, size_t size) > +{ > + size_t i; > + unsigned long *ptr = (unsigned long *)area; > + > + for (i = 0; i < size / sizeof(hash); i++) { > + /* Rotate by odd number of bits and XOR. */ > + hash = (hash << ((sizeof(hash) * 8) - 7)) | (hash >> 7); > + hash ^= ptr[i]; > + } > + > + return hash; > +} > + > +static inline __init unsigned long get_random_boot(void) > +{ > + unsigned long entropy = random_get_entropy(); > + unsigned long hash = 0; > + > + /* Attempt to create a simple but unpredictable starting entropy. */ > + hash = rotate_xor(hash, linux_banner, strlen(linux_banner)); > + > + /* Add in any runtime entropy we can get */ > + hash = rotate_xor(hash, &entropy, sizeof(entropy)); > + > + return hash; > +} > + > +static inline __init bool kaslr_disabled(void) > +{ > + char *str; > + > + str = strstr(boot_command_line, "nokaslr"); > + if (str == boot_command_line || (str > boot_command_line && *(str - 1) == ' ')) > + return true; > + > + return false; > +} > + > +/* Choose a new address for the kernel */ > +static inline void __init *determine_relocation_address(void) > +{ > + unsigned long kernel_length; > + void *dest = _text; > + unsigned long offset; > + > + if (kaslr_disabled()) > + return dest; > + > + kernel_length = (long)_end - (long)_text; > + > + offset = get_random_boot() << 16; > + offset &= (CONFIG_RANDOMIZE_BASE_MAX_OFFSET - 1); > + if (offset < kernel_length) > + offset += ALIGN(kernel_length, 0xffff); > + > + return RELOCATED_KASLR(dest); > +} > + > +static inline int __init relocation_addr_valid(void *loc_new) > +{ > + if ((unsigned long)loc_new & 0x00000ffff) { > + /* Inappropriately aligned new location */ > + return 0; > + } > + if ((unsigned long)loc_new < (unsigned long)_end) { > + /* New location overlaps original kernel */ > + return 0; > + } > + return 1; > +} > + > +static inline void __init update_reloc_offset(unsigned long *addr, long offset) > +{ > + unsigned long *new_addr = (unsigned long *)RELOCATED_KASLR(addr); > + > + *new_addr = (unsigned long)offset; > +} > + > +void *__init do_kaslr(void) > +{ > + void *loc_new; > + unsigned long kernel_length; > + long offset = 0; > + /* Default to original kernel entry point */ > + void *kernel_entry = start_kernel; > + char *cmdline = early_ioremap(fw_arg1, COMMAND_LINE_SIZE); > + > + /* Boot command line was passed in fw_arg1 */ > + strscpy(boot_command_line, cmdline, COMMAND_LINE_SIZE); > + > + kernel_length = (long)(_end) - (long)(_text); > + > + loc_new = determine_relocation_address(); > + > + /* Sanity check relocation address */ > + if (relocation_addr_valid(loc_new)) > + offset = (unsigned long)loc_new - (unsigned long)(_text); > + > + if (offset) { > + /* Copy the kernel to it's new location */ > + memcpy(loc_new, _text, kernel_length); > + > + /* Sync the caches ready for execution of new kernel */ > + __asm__ __volatile__ ( > + "ibar 0 \t\n" > + "dbar 0 \t\n"); > + > + reloc_offset = (unsigned long)_text - VMLINUX_LOAD_ADDRESS; > + reloc_offset += offset; > + > + relocate_relative(); > + > + /* The current thread is now within the relocated image */ > + __current_thread_info = RELOCATED_KASLR(__current_thread_info); > + > + /* Return the new kernel's entry point */ > + kernel_entry = RELOCATED_KASLR(start_kernel); > + > + update_reloc_offset(&reloc_offset, offset); > + } Self review: There is a problem with do_kaslr implementation, which will be fixed in the next version. > + > + return kernel_entry; > +} > +#endif > + > +void __init relocate_kernel(void) > +{ > + reloc_offset = (unsigned long)_text - VMLINUX_LOAD_ADDRESS; > + > + relocate_relative(); > +} > + > /* > * Show relocation information on panic. > */ > ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [PATCH v4 5/5] LoongArch: Add support for kernel address space layout randomization (KASLR) 2023-02-10 9:06 ` Youling Tang @ 2023-02-10 9:37 ` Youling Tang 0 siblings, 0 replies; 22+ messages in thread From: Youling Tang @ 2023-02-10 9:37 UTC (permalink / raw) To: Huacai Chen; +Cc: Xi Ruoyao, Jinyang He, Xuerui Wang, loongarch, linux-kernel On 02/10/2023 05:06 PM, Youling Tang wrote: > > > On 02/10/2023 04:47 PM, Youling Tang wrote: >> This patch adds support for relocating the kernel to a random address. >> >> Entropy is derived from the banner, which will change every build and >> random_get_entropy() which should provide additional runtime entropy. >> >> The kernel is relocated by up to RANDOMIZE_BASE_MAX_OFFSET bytes from >> its link address. Because relocation happens so early in the kernel boot, >> the amount of physical memory has not yet been determined. This means >> the only way to limit relocation within the available memory is via >> Kconfig. Limit the maximum value of RANDOMIZE_BASE_MAX_OFFSET to >> 256M(0x10000000) because our memory layout has many holes. >> >> Signed-off-by: Youling Tang <tangyouling@loongson.cn> >> Signed-off-by: Xi Ruoyao <xry111@xry111.site> # Fix compiler warnings >> --- >> arch/loongarch/Kconfig | 23 +++++ >> arch/loongarch/kernel/head.S | 14 ++- >> arch/loongarch/kernel/relocate.c | 143 ++++++++++++++++++++++++++++++- >> 3 files changed, 176 insertions(+), 4 deletions(-) >> >> diff --git a/arch/loongarch/Kconfig b/arch/loongarch/Kconfig >> index 089a4695b1b3..f0a070bd7254 100644 >> --- a/arch/loongarch/Kconfig >> +++ b/arch/loongarch/Kconfig >> @@ -489,6 +489,29 @@ config RELOCATABLE >> kernel binary at runtime to a different virtual address than the >> address it was linked at. >> >> +config RANDOMIZE_BASE >> + bool "Randomize the address of the kernel image (KASLR)" >> + depends on RELOCATABLE >> + help >> + Randomizes the physical and virtual address at which the >> + kernel image is loaded, as a security feature that >> + deters exploit attempts relying on knowledge of the location >> + of kernel internals. >> + >> + The kernel will be offset by up to RANDOMIZE_BASE_MAX_OFFSET. >> + >> + If unsure, say N. >> + >> +config RANDOMIZE_BASE_MAX_OFFSET >> + hex "Maximum KASLR offset" if EXPERT >> + depends on RANDOMIZE_BASE >> + range 0x0 0x10000000 if 64BIT >> + default "0x01000000" >> + help >> + When KASLR is active, this provides the maximum offset that will >> + be applied to the kernel image. >> + >> + >> config SECCOMP >> bool "Enable seccomp to safely compute untrusted bytecode" >> depends on PROC_FS >> diff --git a/arch/loongarch/kernel/head.S b/arch/loongarch/kernel/head.S >> index 499edc80d8ab..b12f459ad73a 100644 >> --- a/arch/loongarch/kernel/head.S >> +++ b/arch/loongarch/kernel/head.S >> @@ -87,10 +87,22 @@ SYM_CODE_START(kernel_entry) # kernel >> entry point >> set_saved_sp sp, t0, t1 >> >> #ifdef CONFIG_RELOCATABLE >> +#ifdef CONFIG_RANDOMIZE_BASE >> + bl do_kaslr >> + >> + /* Repoint the sp into the new kernel image */ >> + PTR_LI sp, (_THREAD_SIZE - PT_SIZE) >> + PTR_ADD sp, sp, tp >> + set_saved_sp sp, t0, t1 >> + >> + /* do_kaslr returns the new kernel image entry point */ >> + jr a0 >> + ASM_BUG() >> +#else >> /* Apply the relocations */ >> bl relocate_kernel >> #endif >> - >> +#endif >> bl start_kernel >> ASM_BUG() >> >> diff --git a/arch/loongarch/kernel/relocate.c >> b/arch/loongarch/kernel/relocate.c >> index 91ce92433bab..5266f23a3006 100644 >> --- a/arch/loongarch/kernel/relocate.c >> +++ b/arch/loongarch/kernel/relocate.c >> @@ -9,19 +9,21 @@ >> #include <linux/kernel.h> >> #include <linux/printk.h> >> #include <linux/panic_notifier.h> >> +#include <linux/start_kernel.h> >> +#include <asm/bootinfo.h> >> +#include <asm/early_ioremap.h> >> #include <asm/sections.h> >> >> #define RELOCATED(x) ((void *)((long)x + reloc_offset)) >> +#define RELOCATED_KASLR(x) ((void *)((long)x + offset)) >> >> extern long __rela_dyn_start; >> extern long __rela_dyn_end; >> >> static unsigned long reloc_offset; >> >> -void __init relocate_kernel(void) >> +static inline __init void relocate_relative(void) >> { >> - reloc_offset = (unsigned long)_text - VMLINUX_LOAD_ADDRESS; >> - >> if (reloc_offset) { >> Elf64_Rela *rela, *rela_end; >> rela = (Elf64_Rela *)&__rela_dyn_start; >> @@ -43,6 +45,141 @@ void __init relocate_kernel(void) >> } >> } >> >> +#ifdef CONFIG_RANDOMIZE_BASE >> +static inline __init unsigned long rotate_xor(unsigned long hash, >> + const void *area, size_t size) >> +{ >> + size_t i; >> + unsigned long *ptr = (unsigned long *)area; >> + >> + for (i = 0; i < size / sizeof(hash); i++) { >> + /* Rotate by odd number of bits and XOR. */ >> + hash = (hash << ((sizeof(hash) * 8) - 7)) | (hash >> 7); >> + hash ^= ptr[i]; >> + } >> + >> + return hash; >> +} >> + >> +static inline __init unsigned long get_random_boot(void) >> +{ >> + unsigned long entropy = random_get_entropy(); >> + unsigned long hash = 0; >> + >> + /* Attempt to create a simple but unpredictable starting entropy. */ >> + hash = rotate_xor(hash, linux_banner, strlen(linux_banner)); >> + >> + /* Add in any runtime entropy we can get */ >> + hash = rotate_xor(hash, &entropy, sizeof(entropy)); >> + >> + return hash; >> +} >> + >> +static inline __init bool kaslr_disabled(void) >> +{ >> + char *str; >> + >> + str = strstr(boot_command_line, "nokaslr"); >> + if (str == boot_command_line || (str > boot_command_line && *(str >> - 1) == ' ')) >> + return true; >> + >> + return false; >> +} >> + >> +/* Choose a new address for the kernel */ >> +static inline void __init *determine_relocation_address(void) >> +{ >> + unsigned long kernel_length; >> + void *dest = _text; >> + unsigned long offset; >> + >> + if (kaslr_disabled()) >> + return dest; >> + >> + kernel_length = (long)_end - (long)_text; >> + >> + offset = get_random_boot() << 16; >> + offset &= (CONFIG_RANDOMIZE_BASE_MAX_OFFSET - 1); >> + if (offset < kernel_length) >> + offset += ALIGN(kernel_length, 0xffff); >> + >> + return RELOCATED_KASLR(dest); >> +} >> + >> +static inline int __init relocation_addr_valid(void *loc_new) >> +{ >> + if ((unsigned long)loc_new & 0x00000ffff) { >> + /* Inappropriately aligned new location */ >> + return 0; >> + } >> + if ((unsigned long)loc_new < (unsigned long)_end) { >> + /* New location overlaps original kernel */ >> + return 0; >> + } >> + return 1; >> +} >> + >> +static inline void __init update_reloc_offset(unsigned long *addr, >> long offset) >> +{ >> + unsigned long *new_addr = (unsigned long *)RELOCATED_KASLR(addr); >> + >> + *new_addr = (unsigned long)offset; >> +} >> + >> +void *__init do_kaslr(void) >> +{ >> + void *loc_new; >> + unsigned long kernel_length; >> + long offset = 0; >> + /* Default to original kernel entry point */ >> + void *kernel_entry = start_kernel; >> + char *cmdline = early_ioremap(fw_arg1, COMMAND_LINE_SIZE); >> + >> + /* Boot command line was passed in fw_arg1 */ >> + strscpy(boot_command_line, cmdline, COMMAND_LINE_SIZE); >> + >> + kernel_length = (long)(_end) - (long)(_text); >> + >> + loc_new = determine_relocation_address(); >> + >> + /* Sanity check relocation address */ >> + if (relocation_addr_valid(loc_new)) >> + offset = (unsigned long)loc_new - (unsigned long)(_text); >> + >> + if (offset) { >> + /* Copy the kernel to it's new location */ >> + memcpy(loc_new, _text, kernel_length); >> + >> + /* Sync the caches ready for execution of new kernel */ >> + __asm__ __volatile__ ( >> + "ibar 0 \t\n" >> + "dbar 0 \t\n"); >> + >> + reloc_offset = (unsigned long)_text - VMLINUX_LOAD_ADDRESS; >> + reloc_offset += offset; >> + >> + relocate_relative(); >> + >> + /* The current thread is now within the relocated image */ >> + __current_thread_info = RELOCATED_KASLR(__current_thread_info); >> + >> + /* Return the new kernel's entry point */ >> + kernel_entry = RELOCATED_KASLR(start_kernel); >> + >> + update_reloc_offset(&reloc_offset, offset); >> + } > > Self review: > > There is a problem with do_kaslr implementation, which will be fixed in > the next version. When offset is 0, but reloc_offset is not 0, relocate_relative() also needs to be executed. It will be modified as follows: --- a/arch/loongarch/kernel/relocate.c +++ b/arch/loongarch/kernel/relocate.c @@ -146,6 +146,8 @@ void *__init do_kaslr(void) if (relocation_addr_valid(loc_new)) offset = (unsigned long)loc_new - (unsigned long)(_text); + reloc_offset = (unsigned long)_text - VMLINUX_LOAD_ADDRESS; + if (offset) { /* Copy the kernel to it's new location */ memcpy(loc_new, _text, kernel_length); @@ -155,11 +157,8 @@ void *__init do_kaslr(void) "ibar 0 \t\n" "dbar 0 \t\n"); - reloc_offset = (unsigned long)_text - VMLINUX_LOAD_ADDRESS; reloc_offset += offset; - relocate_relative(); - /* The current thread is now within the relocated image */ __current_thread_info = RELOCATED_KASLR(__current_thread_info); @@ -169,6 +168,9 @@ void *__init do_kaslr(void) update_reloc_offset(&reloc_offset, offset); } + if (reloc_offset) + relocate_relative(); + return kernel_entry; } #endif @@ -177,7 +179,8 @@ void __init relocate_kernel(void) { reloc_offset = (unsigned long)_text - VMLINUX_LOAD_ADDRESS; - relocate_relative(); + if (reloc_offset) + relocate_relative(); } > >> + >> + return kernel_entry; >> +} >> +#endif >> + >> +void __init relocate_kernel(void) >> +{ >> + reloc_offset = (unsigned long)_text - VMLINUX_LOAD_ADDRESS; >> + >> + relocate_relative(); >> +} >> + >> /* >> * Show relocation information on panic. >> */ >> > ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [PATCH v4 5/5] LoongArch: Add support for kernel address space layout randomization (KASLR) 2023-02-10 8:47 ` [PATCH v4 5/5] LoongArch: Add support for kernel address space layout randomization (KASLR) Youling Tang 2023-02-10 9:06 ` Youling Tang @ 2023-02-17 8:09 ` Xi Ruoyao 2023-02-17 8:24 ` Youling Tang 1 sibling, 1 reply; 22+ messages in thread From: Xi Ruoyao @ 2023-02-17 8:09 UTC (permalink / raw) To: Youling Tang, Huacai Chen, Jinyang He Cc: Xuerui Wang, loongarch, linux-kernel On Fri, 2023-02-10 at 16:47 +0800, Youling Tang wrote: > + /* Sync the caches ready for execution of new kernel > */ > + __asm__ __volatile__ ( > + "ibar 0 \t\n" > + "dbar 0 \t\n"); I think we should add ::: "memory" here to prevent a future compiler from being too smart. Otherwise LGTM. -- Xi Ruoyao <xry111@xry111.site> School of Aerospace Science and Technology, Xidian University ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [PATCH v4 5/5] LoongArch: Add support for kernel address space layout randomization (KASLR) 2023-02-17 8:09 ` Xi Ruoyao @ 2023-02-17 8:24 ` Youling Tang 0 siblings, 0 replies; 22+ messages in thread From: Youling Tang @ 2023-02-17 8:24 UTC (permalink / raw) To: Xi Ruoyao; +Cc: Huacai Chen, Jinyang He, Xuerui Wang, loongarch, linux-kernel On 02/17/2023 04:09 PM, Xi Ruoyao wrote: > On Fri, 2023-02-10 at 16:47 +0800, Youling Tang wrote: > >> + /* Sync the caches ready for execution of new kernel >> */ >> + __asm__ __volatile__ ( >> + "ibar 0 \t\n" >> + "dbar 0 \t\n"); > > I think we should add ::: "memory" here to prevent a future compiler > from being too smart. Got it. Thanks, Youling > > Otherwise LGTM. > ^ permalink raw reply [flat|nested] 22+ messages in thread
end of thread, other threads:[~2023-02-17 8:24 UTC | newest] Thread overview: 22+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2023-02-10 8:47 [PATCH v4 0/5] LoongArch: Add kernel relocation and KASLR support Youling Tang 2023-02-10 8:47 ` [PATCH v4 1/5] LoongArch: Use la.pcrel instead of la.abs when it's trivially possible Youling Tang 2023-02-10 9:04 ` Huacai Chen 2023-02-10 8:47 ` [PATCH v4 2/5] LoongArch: Use la.pcrel instead of la.abs for exception handlers Youling Tang 2023-02-10 9:09 ` Huacai Chen 2023-02-10 9:18 ` Youling Tang 2023-02-16 2:32 ` Youling Tang 2023-02-16 6:56 ` Huacai Chen 2023-02-16 6:59 ` Jinyang He 2023-02-16 7:10 ` Xi Ruoyao 2023-02-16 8:03 ` Youling Tang 2023-02-16 11:18 ` Youling Tang 2023-02-16 11:29 ` Youling Tang 2023-02-10 8:47 ` [PATCH v4 3/5] LoongArch: Add JUMP_VIRT_ADDR macro implementation to avoid using la.abs Youling Tang 2023-02-10 8:47 ` [PATCH v4 4/5] LoongArch: Add support for kernel relocation Youling Tang 2023-02-10 9:05 ` Huacai Chen 2023-02-10 18:01 ` kernel test robot 2023-02-10 8:47 ` [PATCH v4 5/5] LoongArch: Add support for kernel address space layout randomization (KASLR) Youling Tang 2023-02-10 9:06 ` Youling Tang 2023-02-10 9:37 ` Youling Tang 2023-02-17 8:09 ` Xi Ruoyao 2023-02-17 8:24 ` Youling Tang
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®