* [PATCH v2 0/9] x86/kexec: Fixes for tip/x86/boot
@ 2025-01-09 14:04 David Woodhouse
2025-01-09 14:04 ` [PATCH v2 1/9] x86/kexec: Disable global pages before writing to control page David Woodhouse
` (9 more replies)
0 siblings, 10 replies; 23+ messages in thread
From: David Woodhouse @ 2025-01-09 14:04 UTC (permalink / raw)
To: Thomas Gleixner, Ingo Molnar, Borislav Petkov, Dave Hansen, x86,
H . Peter Anvin, Eric Biederman, David Woodhouse, Sourabh Jain,
Hari Bathini, Michael Ellerman, Thomas Zimmermann, Andrew Morton,
Baoquan He, Yuntao Wang, David Kaplan, Tao Liu,
Kirill A . Shutemov, Kai Huang, Ard Biesheuvel, Josh Poimboeuf,
Breno Leitao, Wei Yang, Rong Xu, Thomas Weißschuh,
linux-kernel, kexec, Simon Horman, Dave Young, Peter Zijlstra,
bsz, nathan
Fix a few bugs introduced by the recent rework of the relocate_kernel()
code, and one which has been there in the kjump code from the beginning.
• Global read-only TLB entries made relocate_kernel() crash when writing
to its own page after the %cr3 switch should have made it writeable.
• The preserve_context flag was lost when invoking swap_pages on the way
back, causing pages not to be swapped.
• The wrong page was being used as the swap page.
• The ABI for the kjump call asks the callee to put its entry address for
next time at %rsp+8 before returning, but we set %rsp to the top of the
available page, such that the entry address is at the start of some
other page.
• The relocate_kernel() function lacked Clang CFI information, but is now
called via a function pointer, leading to a crash¹.
• The relocate_kernel() code and data could end up being linked into the
wrong place for a LTO / -ffunction-sections build.
Thanks to Nathan for reporting many of the above.
Also a few minor cleanups, including a comments-only patch from Rafael
on the suspend-like part of kjump as a prelude to actually cleaning that
up.
¹ Fixed by just adding __nocfi. Actually providing the CFI information
for relocate_kernel() will need a bit more work, so let's just do the
simple fix for now.
v2:
• Rebase
• Save a whole byte on a 'movq $1, %r11' instruction.
• Fix duplicated word in comment.
• Avoid absolute relocations against __relocate_kernel_start
David Woodhouse (7):
x86/kexec: Disable global pages before writing to control page
x86/kexec: Ensure preserve_context flag is set on return to kernel
x86/kexec: Use correct swap page in swap_pages function
x86/kexec: Fix stack and handling of re-entry point for ::preserve_context
x86/kexec: Mark machine_kexec() with __nocfi
x86/kexec: Cope with relocate_kernel() not being at the start of the page
x86/kexec: Use typedef for relocate_kernel_fn function prototype
Nathan Chancellor (1):
x86/kexec: Fix location of relocate_kernel with -ffunction-sections
Rafael J. Wysocki (1):
kexec_core: Add and update comments regarding the KEXEC_JUMP flow
arch/x86/include/asm/kexec.h | 26 ++++++++--------
arch/x86/kernel/machine_kexec_32.c | 7 +----
arch/x86/kernel/machine_kexec_64.c | 8 ++---
arch/x86/kernel/relocate_kernel_64.S | 57 ++++++++++++++++++++++++------------
arch/x86/kernel/vmlinux.lds.S | 4 +--
kernel/kexec_core.c | 23 +++++++++++----
6 files changed, 74 insertions(+), 51 deletions(-)
^ permalink raw reply [flat|nested] 23+ messages in thread
* [PATCH v2 1/9] x86/kexec: Disable global pages before writing to control page
2025-01-09 14:04 [PATCH v2 0/9] x86/kexec: Fixes for tip/x86/boot David Woodhouse
@ 2025-01-09 14:04 ` David Woodhouse
2025-01-15 9:17 ` [tip: x86/boot] " tip-bot2 for David Woodhouse
2025-01-09 14:04 ` [PATCH v2 2/9] x86/kexec: Ensure preserve_context flag is set on return to kernel David Woodhouse
` (8 subsequent siblings)
9 siblings, 1 reply; 23+ messages in thread
From: David Woodhouse @ 2025-01-09 14:04 UTC (permalink / raw)
To: Thomas Gleixner, Ingo Molnar, Borislav Petkov, Dave Hansen, x86,
H . Peter Anvin, Eric Biederman, David Woodhouse, Sourabh Jain,
Hari Bathini, Michael Ellerman, Thomas Zimmermann, Andrew Morton,
Baoquan He, Yuntao Wang, David Kaplan, Tao Liu,
Kirill A . Shutemov, Kai Huang, Ard Biesheuvel, Josh Poimboeuf,
Breno Leitao, Wei Yang, Rong Xu, Thomas Weißschuh,
linux-kernel, kexec, Simon Horman, Dave Young, Peter Zijlstra,
bsz, nathan
From: David Woodhouse <dwmw@amazon.co.uk>
The kernel switches to a new set of page tables during kexec. The global
mappings (_PAGE_GLOBAL==1) can remain in the TLB after this switch. This
is generally not a problem because the new page tables use a different
portion of the virtual address space than the normal kernel mappings.
The critical exception to that generalisation (and the only mapping
which isn't an identity mapping) is the kexec control page itself —
which was ROX in the original kernel mapping, but should be RWX in the
new page tables. If there is a global TLB entry for that in its prior
read-only state, it definitely needs to be flushed before attempting to
write through that virtual mapping.
It would be possible to just avoid writing to the virtual address of the
page and defer all writes until they can be done through the identity
mapping. But there's no good reason to keep the old TLB entries around,
as they can cause nothing but trouble.
Clear the PGE bit in %cr4 early, before storing data in the control page.
Fixes: 5a82223e0743 ("x86/kexec: Mark relocate_kernel page as ROX instead of RWX")
Co-authored-by: Dave Hansen <dave.hansen@linux.intel.com>
Reported-by: Nathan Chancellor <nathan@kernel.org>
Reported-by: "Ning, Hongyu" <hongyu.ning@linux.intel.com>
Closes: https://bugzilla.kernel.org/show_bug.cgi?id=219592
Signed-off-by: Dave Hansen <dave.hansen@linux.intel.com>
Signed-off-by: David Woodhouse <dwmw@amazon.co.uk>
Tested-by: Nathan Chancellor <nathan@kernel.org>
Tested-by: "Ning, Hongyu" <hongyu.ning@linux.intel.com>
---
arch/x86/kernel/relocate_kernel_64.S | 12 +++++++++---
1 file changed, 9 insertions(+), 3 deletions(-)
diff --git a/arch/x86/kernel/relocate_kernel_64.S b/arch/x86/kernel/relocate_kernel_64.S
index 8bc86a1e056a..9bd601dd8659 100644
--- a/arch/x86/kernel/relocate_kernel_64.S
+++ b/arch/x86/kernel/relocate_kernel_64.S
@@ -70,14 +70,20 @@ SYM_CODE_START_NOALIGN(relocate_kernel)
movq kexec_pa_table_page(%rip), %r9
movq %r9, %cr3
+ /* Leave CR4 in %r13 to enable the right paging mode later. */
+ movq %cr4, %r13
+
+ /* Disable global pages immediately to ensure this mapping is RWX */
+ movq %r13, %r12
+ andq $~(X86_CR4_PGE), %r12
+ movq %r12, %cr4
+
/* Save %rsp and CRs. */
+ movq %r13, saved_cr4(%rip)
movq %rsp, saved_rsp(%rip)
movq %rax, saved_cr3(%rip)
movq %cr0, %rax
movq %rax, saved_cr0(%rip)
- /* Leave CR4 in %r13 to enable the right paging mode later. */
- movq %cr4, %r13
- movq %r13, saved_cr4(%rip)
/* save indirection list for jumping back */
movq %rdi, pa_backup_pages_map(%rip)
base-commit: 893930143440eb5e3ea8f69cb51ab2e61e15c4e1
--
2.47.0
^ permalink raw reply [flat|nested] 23+ messages in thread
* [PATCH v2 2/9] x86/kexec: Ensure preserve_context flag is set on return to kernel
2025-01-09 14:04 [PATCH v2 0/9] x86/kexec: Fixes for tip/x86/boot David Woodhouse
2025-01-09 14:04 ` [PATCH v2 1/9] x86/kexec: Disable global pages before writing to control page David Woodhouse
@ 2025-01-09 14:04 ` David Woodhouse
2025-01-15 9:17 ` [tip: x86/boot] " tip-bot2 for David Woodhouse
2025-01-09 14:04 ` [PATCH v2 3/9] x86/kexec: Use correct swap page in swap_pages function David Woodhouse
` (7 subsequent siblings)
9 siblings, 1 reply; 23+ messages in thread
From: David Woodhouse @ 2025-01-09 14:04 UTC (permalink / raw)
To: Thomas Gleixner, Ingo Molnar, Borislav Petkov, Dave Hansen, x86,
H . Peter Anvin, Eric Biederman, David Woodhouse, Sourabh Jain,
Hari Bathini, Michael Ellerman, Thomas Zimmermann, Andrew Morton,
Baoquan He, Yuntao Wang, David Kaplan, Tao Liu,
Kirill A . Shutemov, Kai Huang, Ard Biesheuvel, Josh Poimboeuf,
Breno Leitao, Wei Yang, Rong Xu, Thomas Weißschuh,
linux-kernel, kexec, Simon Horman, Dave Young, Peter Zijlstra,
bsz, nathan
From: David Woodhouse <dwmw@amazon.co.uk>
The swap_pages function will only actually *swap*, as its name implies,
if the preserve_context flag in the %r11 register is non-zero. On the
way back from a ::preserve_context kexec, ensure that the %r11 register
is non-zero so that the pages get swapped back.
Fixes: 9e5683e2d0b5 ("x86/kexec: Only swap pages for ::preserve_context mode")
Signed-off-by: David Woodhouse <dwmw@amazon.co.uk>
---
arch/x86/kernel/relocate_kernel_64.S | 1 +
1 file changed, 1 insertion(+)
diff --git a/arch/x86/kernel/relocate_kernel_64.S b/arch/x86/kernel/relocate_kernel_64.S
index 9bd601dd8659..6fce4b459652 100644
--- a/arch/x86/kernel/relocate_kernel_64.S
+++ b/arch/x86/kernel/relocate_kernel_64.S
@@ -220,6 +220,7 @@ SYM_CODE_START_LOCAL_NOALIGN(identity_mapped)
movq kexec_pa_table_page(%rip), %rax
movq %rax, %cr3
lea PAGE_SIZE(%r8), %rsp
+ movl $1, %r11d /* Ensure preserve_context flag is set */
call swap_pages
movq kexec_va_control_page(%rip), %rax
addq $(virtual_mapped - relocate_kernel), %rax
--
2.47.0
^ permalink raw reply [flat|nested] 23+ messages in thread
* [PATCH v2 3/9] x86/kexec: Use correct swap page in swap_pages function
2025-01-09 14:04 [PATCH v2 0/9] x86/kexec: Fixes for tip/x86/boot David Woodhouse
2025-01-09 14:04 ` [PATCH v2 1/9] x86/kexec: Disable global pages before writing to control page David Woodhouse
2025-01-09 14:04 ` [PATCH v2 2/9] x86/kexec: Ensure preserve_context flag is set on return to kernel David Woodhouse
@ 2025-01-09 14:04 ` David Woodhouse
2025-01-15 9:17 ` [tip: x86/boot] " tip-bot2 for David Woodhouse
2025-01-09 14:04 ` [PATCH v2 4/9] x86/kexec: Fix stack and handling of re-entry point for ::preserve_context David Woodhouse
` (6 subsequent siblings)
9 siblings, 1 reply; 23+ messages in thread
From: David Woodhouse @ 2025-01-09 14:04 UTC (permalink / raw)
To: Thomas Gleixner, Ingo Molnar, Borislav Petkov, Dave Hansen, x86,
H . Peter Anvin, Eric Biederman, David Woodhouse, Sourabh Jain,
Hari Bathini, Michael Ellerman, Thomas Zimmermann, Andrew Morton,
Baoquan He, Yuntao Wang, David Kaplan, Tao Liu,
Kirill A . Shutemov, Kai Huang, Ard Biesheuvel, Josh Poimboeuf,
Breno Leitao, Wei Yang, Rong Xu, Thomas Weißschuh,
linux-kernel, kexec, Simon Horman, Dave Young, Peter Zijlstra,
bsz, nathan
From: David Woodhouse <dwmw@amazon.co.uk>
The swap_pages function expects the swap page to be in %r10, but there
was no documentation to that effect. Once upon a time the setup code
used to load its value from a kernel virtual address and save it to an
address which is accessible in the identity-mapped page tables, and
*happened* to use %r10 to do so, with no comment that it was left there
on *purpose* instead of just being a scratch register. Once that was no
longer necessary, %r10 just holds whatever the kernel happened to leave
in it.
Now that the original value passed by the kernel is accessible via
%rip-relative addressing, load directly from there instead of using %r10
for it. But document the other parameters that the swap_pages function
*does* expect in registers.
Fixes: b3adabae8a96 ("x86/kexec: Drop page_list argument from relocate_kernel()")
Signed-off-by: David Woodhouse <dwmw@amazon.co.uk>
---
arch/x86/kernel/relocate_kernel_64.S | 8 ++++++--
1 file changed, 6 insertions(+), 2 deletions(-)
diff --git a/arch/x86/kernel/relocate_kernel_64.S b/arch/x86/kernel/relocate_kernel_64.S
index 6fce4b459652..3ca3bf6b3f49 100644
--- a/arch/x86/kernel/relocate_kernel_64.S
+++ b/arch/x86/kernel/relocate_kernel_64.S
@@ -264,6 +264,10 @@ SYM_CODE_END(virtual_mapped)
/* Do the copies */
SYM_CODE_START_LOCAL_NOALIGN(swap_pages)
UNWIND_HINT_END_OF_STACK
+ /*
+ * %rdi indirection page
+ * %r11 preserve_context
+ */
movq %rdi, %rcx /* Put the indirection_page in %rcx */
xorl %edi, %edi
xorl %esi, %esi
@@ -302,7 +306,7 @@ SYM_CODE_START_LOCAL_NOALIGN(swap_pages)
jz .Lnoswap
/* copy source page to swap page */
- movq %r10, %rdi
+ movq kexec_pa_swap_page(%rip), %rdi
movl $512, %ecx
rep ; movsq
@@ -314,7 +318,7 @@ SYM_CODE_START_LOCAL_NOALIGN(swap_pages)
/* copy swap page to destination page */
movq %rdx, %rdi
- movq %r10, %rsi
+ movq kexec_pa_swap_page(%rip), %rsi
.Lnoswap:
movl $512, %ecx
rep ; movsq
--
2.47.0
^ permalink raw reply [flat|nested] 23+ messages in thread
* [PATCH v2 4/9] x86/kexec: Fix stack and handling of re-entry point for ::preserve_context
2025-01-09 14:04 [PATCH v2 0/9] x86/kexec: Fixes for tip/x86/boot David Woodhouse
` (2 preceding siblings ...)
2025-01-09 14:04 ` [PATCH v2 3/9] x86/kexec: Use correct swap page in swap_pages function David Woodhouse
@ 2025-01-09 14:04 ` David Woodhouse
2025-01-15 9:17 ` [tip: x86/boot] " tip-bot2 for David Woodhouse
2025-01-09 14:04 ` [PATCH v2 5/9] x86/kexec: Fix location of relocate_kernel with -ffunction-sections David Woodhouse
` (5 subsequent siblings)
9 siblings, 1 reply; 23+ messages in thread
From: David Woodhouse @ 2025-01-09 14:04 UTC (permalink / raw)
To: Thomas Gleixner, Ingo Molnar, Borislav Petkov, Dave Hansen, x86,
H . Peter Anvin, Eric Biederman, David Woodhouse, Sourabh Jain,
Hari Bathini, Michael Ellerman, Thomas Zimmermann, Andrew Morton,
Baoquan He, Yuntao Wang, David Kaplan, Tao Liu,
Kirill A . Shutemov, Kai Huang, Ard Biesheuvel, Josh Poimboeuf,
Breno Leitao, Wei Yang, Rong Xu, Thomas Weißschuh,
linux-kernel, kexec, Simon Horman, Dave Young, Peter Zijlstra,
bsz, nathan
From: David Woodhouse <dwmw@amazon.co.uk>
A ::preserve_context kimage can be invoked more than once, and the entry
point can be different every time. When the callee returns to the kernel,
it leaves the address of its entry point for next time on the stack.
That being the case, one might reasonably assume that the caller would
allocate space for it on the stack fram before actually performing the
'call' into the callee.
Apparently not, though. Ever since the kjump code was first added in
2009, it has set up a *new* stack at the top of the swap_page scratch
page, then just performed the 'call' without allocating any space for
the re-entry address to be returned. It then reads the re-entry point
for next time from 0(%rsp) which is actually the first qword of the page
*after* the swap page, which might not exist at all! And if the callee
has written to that, then it will have corrupted memory it doesn't own.
Correct this by pushing the entry point of the callee onto the stack
before calling it. The callee may then adjust it, or not, as it sees fit,
and subsequent invocations should work correctly either way.
Remove a stray push of zero to the *relocate_kernel* stack, which may
have been intended for this purpose, but which was actually just noise.
Also, loading the stack for the callee relied on the address of the swap
page being in %r10 without ever documenting that fact. Recent code
changes made that no longer true, so load it directly from the local
kexec_pa_swap_page variable instead.
Fixes: b3adabae8a96 ("x86/kexec: Drop page_list argument from relocate_kernel()")
Signed-off-by: David Woodhouse <dwmw@amazon.co.uk>
---
arch/x86/kernel/relocate_kernel_64.S | 12 +++++++++---
1 file changed, 9 insertions(+), 3 deletions(-)
diff --git a/arch/x86/kernel/relocate_kernel_64.S b/arch/x86/kernel/relocate_kernel_64.S
index 3ca3bf6b3f49..a95691b42c5c 100644
--- a/arch/x86/kernel/relocate_kernel_64.S
+++ b/arch/x86/kernel/relocate_kernel_64.S
@@ -113,8 +113,6 @@ SYM_CODE_START_LOCAL_NOALIGN(identity_mapped)
* %r13 original CR4 when relocate_kernel() was invoked
*/
- /* set return address to 0 if not preserving context */
- pushq $0
/* store the start address on the stack */
pushq %rdx
@@ -208,12 +206,19 @@ SYM_CODE_START_LOCAL_NOALIGN(identity_mapped)
.Lrelocate:
popq %rdx
+
+ /* Use the swap page for the callee's stack */
+ movq kexec_pa_swap_page(%rip), %r10
leaq PAGE_SIZE(%r10), %rsp
+
+ /* push the existing entry point onto the callee's stack */
+ pushq %rdx
+
ANNOTATE_RETPOLINE_SAFE
call *%rdx
/* get the re-entry point of the peer system */
- movq 0(%rsp), %rbp
+ popq %rbp
leaq relocate_kernel(%rip), %r8
movq kexec_pa_swap_page(%rip), %r10
movq pa_backup_pages_map(%rip), %rdi
@@ -247,6 +252,7 @@ SYM_CODE_START_LOCAL_NOALIGN(virtual_mapped)
lgdt saved_context_gdt_desc(%rax)
#endif
+ /* relocate_kernel() returns the re-entry point for next time */
movq %rbp, %rax
popf
--
2.47.0
^ permalink raw reply [flat|nested] 23+ messages in thread
* [PATCH v2 5/9] x86/kexec: Fix location of relocate_kernel with -ffunction-sections
2025-01-09 14:04 [PATCH v2 0/9] x86/kexec: Fixes for tip/x86/boot David Woodhouse
` (3 preceding siblings ...)
2025-01-09 14:04 ` [PATCH v2 4/9] x86/kexec: Fix stack and handling of re-entry point for ::preserve_context David Woodhouse
@ 2025-01-09 14:04 ` David Woodhouse
2025-01-15 9:17 ` [tip: x86/boot] " tip-bot2 for Nathan Chancellor
2025-01-09 14:04 ` [PATCH v2 6/9] x86/kexec: Mark machine_kexec() with __nocfi David Woodhouse
` (4 subsequent siblings)
9 siblings, 1 reply; 23+ messages in thread
From: David Woodhouse @ 2025-01-09 14:04 UTC (permalink / raw)
To: Thomas Gleixner, Ingo Molnar, Borislav Petkov, Dave Hansen, x86,
H . Peter Anvin, Eric Biederman, David Woodhouse, Sourabh Jain,
Hari Bathini, Michael Ellerman, Thomas Zimmermann, Andrew Morton,
Baoquan He, Yuntao Wang, David Kaplan, Tao Liu,
Kirill A . Shutemov, Kai Huang, Ard Biesheuvel, Josh Poimboeuf,
Breno Leitao, Wei Yang, Rong Xu, Thomas Weißschuh,
linux-kernel, kexec, Simon Horman, Dave Young, Peter Zijlstra,
bsz, nathan
From: Nathan Chancellor <nathan@kernel.org>
After commit cb33ff9e063c ("x86/kexec: Move relocate_kernel to kernel
.data section"), kernels configured with an option that uses
-ffunction-sections, such as CONFIG_LTO_CLANG, crash when kexecing
because the value of relocate_kernel does not match the value of
__relocate_kernel_start so incorrect code gets copied via
machine_kexec_prepare().
$ llvm-nm good-vmlinux &| rg relocate_kernel
ffffffff83280d41 T __relocate_kernel_end
ffffffff83280b00 T __relocate_kernel_start
ffffffff83280b00 T relocate_kernel
$ llvm-nm bad-vmlinux &| rg relocate_kernel
ffffffff83266100 D __relocate_kernel_end
ffffffff83266100 D __relocate_kernel_start
ffffffff8120b0d8 T relocate_kernel
When -ffunction-sections is enabled, TEXT_MAIN matches on
'.text.[0-9a-zA-Z_]*' to coalesce the function specific functions back
into .text during link time after they have been optimized. Due to the
placement of TEXT_TEXT before KEXEC_RELOCATE_KERNEL in the x86 linker
script, the .text.relocate_kernel section ends up in .text instead of
.data.
Use a second dot in the relocate_kernel section name to avoid matching
on TEXT_MAIN, which matches a similar situation that happened in
commit 79cd2a11224e ("x86/retpoline,kprobes: Fix position of thunk
sections with CONFIG_LTO_CLANG"), which allows kexec to function
properly.
While .data.relocate_kernel still ends up in the .data section via
DATA_MAIN -> DATA_DATA, ensure it is located with the
.text.relocate_kernel section as intended by performing the same
transformation.
Fixes: cb33ff9e063c ("x86/kexec: Move relocate_kernel to kernel .data section")
Fixes: 8dbec5c77bc3 ("x86/kexec: Add data section to relocate_kernel")
Signed-off-by: Nathan Chancellor <nathan@kernel.org>
Signed-off-by: David Woodhouse <dwmw@amazon.co.uk>
---
arch/x86/kernel/relocate_kernel_64.S | 6 +++---
arch/x86/kernel/vmlinux.lds.S | 4 ++--
2 files changed, 5 insertions(+), 5 deletions(-)
diff --git a/arch/x86/kernel/relocate_kernel_64.S b/arch/x86/kernel/relocate_kernel_64.S
index a95691b42c5c..14ed40b7f322 100644
--- a/arch/x86/kernel/relocate_kernel_64.S
+++ b/arch/x86/kernel/relocate_kernel_64.S
@@ -23,11 +23,11 @@
#define PAGE_ATTR (_PAGE_PRESENT | _PAGE_RW | _PAGE_ACCESSED | _PAGE_DIRTY)
/*
- * The .text.relocate_kernel and .data.relocate_kernel sections are copied
+ * The .text..relocate_kernel and .data..relocate_kernel sections are copied
* into the control page, and the remainder of the page is used as the stack.
*/
- .section .data.relocate_kernel,"a";
+ .section .data..relocate_kernel,"a";
/* Minimal CPU state */
SYM_DATA_LOCAL(saved_rsp, .quad 0)
SYM_DATA_LOCAL(saved_cr0, .quad 0)
@@ -39,7 +39,7 @@ SYM_DATA(kexec_pa_table_page, .quad 0)
SYM_DATA(kexec_pa_swap_page, .quad 0)
SYM_DATA_LOCAL(pa_backup_pages_map, .quad 0)
- .section .text.relocate_kernel,"ax";
+ .section .text..relocate_kernel,"ax";
.code64
SYM_CODE_START_NOALIGN(relocate_kernel)
UNWIND_HINT_END_OF_STACK
diff --git a/arch/x86/kernel/vmlinux.lds.S b/arch/x86/kernel/vmlinux.lds.S
index 0c893997f023..63ff60a11be5 100644
--- a/arch/x86/kernel/vmlinux.lds.S
+++ b/arch/x86/kernel/vmlinux.lds.S
@@ -100,8 +100,8 @@ const_pcpu_hot = pcpu_hot;
#define KEXEC_RELOCATE_KERNEL \
. = ALIGN(0x100); \
__relocate_kernel_start = .; \
- *(.text.relocate_kernel); \
- *(.data.relocate_kernel); \
+ *(.text..relocate_kernel); \
+ *(.data..relocate_kernel); \
__relocate_kernel_end = .;
ASSERT(__relocate_kernel_end - __relocate_kernel_start <= KEXEC_CONTROL_CODE_MAX_SIZE,
--
2.47.0
^ permalink raw reply [flat|nested] 23+ messages in thread
* [PATCH v2 6/9] x86/kexec: Mark machine_kexec() with __nocfi
2025-01-09 14:04 [PATCH v2 0/9] x86/kexec: Fixes for tip/x86/boot David Woodhouse
` (4 preceding siblings ...)
2025-01-09 14:04 ` [PATCH v2 5/9] x86/kexec: Fix location of relocate_kernel with -ffunction-sections David Woodhouse
@ 2025-01-09 14:04 ` David Woodhouse
2025-01-15 9:17 ` [tip: x86/boot] " tip-bot2 for David Woodhouse
2025-01-09 14:04 ` [PATCH v2 7/9] kexec_core: Add and update comments regarding the KEXEC_JUMP flow David Woodhouse
` (3 subsequent siblings)
9 siblings, 1 reply; 23+ messages in thread
From: David Woodhouse @ 2025-01-09 14:04 UTC (permalink / raw)
To: Thomas Gleixner, Ingo Molnar, Borislav Petkov, Dave Hansen, x86,
H . Peter Anvin, Eric Biederman, David Woodhouse, Sourabh Jain,
Hari Bathini, Michael Ellerman, Thomas Zimmermann, Andrew Morton,
Baoquan He, Yuntao Wang, David Kaplan, Tao Liu,
Kirill A . Shutemov, Kai Huang, Ard Biesheuvel, Josh Poimboeuf,
Breno Leitao, Wei Yang, Rong Xu, Thomas Weißschuh,
linux-kernel, kexec, Simon Horman, Dave Young, Peter Zijlstra,
bsz, nathan
From: David Woodhouse <dwmw@amazon.co.uk>
A recent commit caused the relocate_kernel() function to be invoked
through a function pointer, but it does not have CFI information. The
resulting trap occurs after the IDT and GDT have been invalidated,
leading to a triple-fault if CONFIG_CFI_CLANG is enabled.
Using SYM_TYPED_FUNC_START() to provide the CFI information looks like
it will require a prolonged battle with objtool. And is fairly pointless
anyway, as the actual signature comes from a __kcfi_typeid_… symbol
emitted from the C code based on the function prototype it thinks that
relocate_kernel has, rendering the check somewhat tautological.
The simple fix is just to mark machine_kexec() with __nocfi.
Reported-by: Nathan Chancellor <nathan@kernel.org>
Suggested-by: Nathan Chancellor <nathan@kernel.org>
Fixes: eeebbde57113 ("x86/kexec: Invoke copy of relocate_kernel() instead of
the original")
Signed-off-by: David Woodhouse <dwmw@amazon.co.uk>
---
arch/x86/kernel/machine_kexec_64.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/arch/x86/kernel/machine_kexec_64.c b/arch/x86/kernel/machine_kexec_64.c
index 9232ad1562c8..1440f792a86d 100644
--- a/arch/x86/kernel/machine_kexec_64.c
+++ b/arch/x86/kernel/machine_kexec_64.c
@@ -342,7 +342,7 @@ void machine_kexec_cleanup(struct kimage *image)
* Do not allocate memory (or fail in any way) in machine_kexec().
* We are past the point of no return, committed to rebooting now.
*/
-void machine_kexec(struct kimage *image)
+void __nocfi machine_kexec(struct kimage *image)
{
unsigned long (*relocate_kernel_ptr)(unsigned long indirection_page,
unsigned long pa_control_page,
--
2.47.0
^ permalink raw reply [flat|nested] 23+ messages in thread
* [PATCH v2 7/9] kexec_core: Add and update comments regarding the KEXEC_JUMP flow
2025-01-09 14:04 [PATCH v2 0/9] x86/kexec: Fixes for tip/x86/boot David Woodhouse
` (5 preceding siblings ...)
2025-01-09 14:04 ` [PATCH v2 6/9] x86/kexec: Mark machine_kexec() with __nocfi David Woodhouse
@ 2025-01-09 14:04 ` David Woodhouse
2025-01-15 9:17 ` [tip: x86/boot] " tip-bot2 for Rafael J. Wysocki
2025-01-09 14:04 ` [PATCH v2 8/9] x86/kexec: Cope with relocate_kernel() not being at the start of the page David Woodhouse
` (2 subsequent siblings)
9 siblings, 1 reply; 23+ messages in thread
From: David Woodhouse @ 2025-01-09 14:04 UTC (permalink / raw)
To: Thomas Gleixner, Ingo Molnar, Borislav Petkov, Dave Hansen, x86,
H . Peter Anvin, Eric Biederman, David Woodhouse, Sourabh Jain,
Hari Bathini, Michael Ellerman, Thomas Zimmermann, Andrew Morton,
Baoquan He, Yuntao Wang, David Kaplan, Tao Liu,
Kirill A . Shutemov, Kai Huang, Ard Biesheuvel, Josh Poimboeuf,
Breno Leitao, Wei Yang, Rong Xu, Thomas Weißschuh,
linux-kernel, kexec, Simon Horman, Dave Young, Peter Zijlstra,
bsz, nathan
From: "Rafael J. Wysocki" <rafael.j.wysocki@intel.com>
The KEXEC_JUMP flow is analogous to hibernation flows occurring before
and after creating an image and before and after jumping from the
restore kernel to the image one, which is why it uses the same device
callbacks as those hibernation flows.
Add comments explaining that to the code in question and update an
existing comment in it which appears a bit out of context.
No functional changes.
Signed-off-by: Rafael J. Wysocki <rafael.j.wysocki@intel.com>
Signed-off-by: David Woodhouse <dwmw@amazon.co.uk>
---
kernel/kexec_core.c | 23 +++++++++++++++++------
1 file changed, 17 insertions(+), 6 deletions(-)
diff --git a/kernel/kexec_core.c b/kernel/kexec_core.c
index c0caa14880c3..464ab059c58d 100644
--- a/kernel/kexec_core.c
+++ b/kernel/kexec_core.c
@@ -1001,6 +1001,12 @@ int kernel_kexec(void)
#ifdef CONFIG_KEXEC_JUMP
if (kexec_image->preserve_context) {
+ /*
+ * This flow is analogous to hibernation flows that occur
+ * before creating an image and before jumping from the
+ * restore kernel to the image one, so it uses the same
+ * device callbacks as those two flows.
+ */
pm_prepare_console();
error = freeze_processes();
if (error) {
@@ -1011,12 +1017,10 @@ int kernel_kexec(void)
error = dpm_suspend_start(PMSG_FREEZE);
if (error)
goto Resume_console;
- /* At this point, dpm_suspend_start() has been called,
- * but *not* dpm_suspend_end(). We *must* call
- * dpm_suspend_end() now. Otherwise, drivers for
- * some devices (e.g. interrupt controllers) become
- * desynchronized with the actual state of the
- * hardware at resume time, and evil weirdness ensues.
+ /*
+ * dpm_suspend_end() must be called after dpm_suspend_start()
+ * to complete the transition, like in the hibernation flows
+ * mentioned above.
*/
error = dpm_suspend_end(PMSG_FREEZE);
if (error)
@@ -1052,6 +1056,13 @@ int kernel_kexec(void)
#ifdef CONFIG_KEXEC_JUMP
if (kexec_image->preserve_context) {
+ /*
+ * This flow is analogous to hibernation flows that occur after
+ * creating an image and after the image hernel has got control
+ * back, and in case the devices have been reset or otherwise
+ * manipulated in the meantime, it uses the device callbacks
+ * used by the latter.
+ */
syscore_resume();
Enable_irqs:
local_irq_enable();
--
2.47.0
^ permalink raw reply [flat|nested] 23+ messages in thread
* [PATCH v2 8/9] x86/kexec: Cope with relocate_kernel() not being at the start of the page
2025-01-09 14:04 [PATCH v2 0/9] x86/kexec: Fixes for tip/x86/boot David Woodhouse
` (6 preceding siblings ...)
2025-01-09 14:04 ` [PATCH v2 7/9] kexec_core: Add and update comments regarding the KEXEC_JUMP flow David Woodhouse
@ 2025-01-09 14:04 ` David Woodhouse
2025-01-15 9:17 ` [tip: x86/boot] " tip-bot2 for David Woodhouse
2025-01-09 14:04 ` [PATCH v2 9/9] x86/kexec: Use typedef for relocate_kernel_fn function prototype David Woodhouse
2025-01-14 11:34 ` [PATCH v2 0/9] x86/kexec: Fixes for tip/x86/boot David Woodhouse
9 siblings, 1 reply; 23+ messages in thread
From: David Woodhouse @ 2025-01-09 14:04 UTC (permalink / raw)
To: Thomas Gleixner, Ingo Molnar, Borislav Petkov, Dave Hansen, x86,
H . Peter Anvin, Eric Biederman, David Woodhouse, Sourabh Jain,
Hari Bathini, Michael Ellerman, Thomas Zimmermann, Andrew Morton,
Baoquan He, Yuntao Wang, David Kaplan, Tao Liu,
Kirill A . Shutemov, Kai Huang, Ard Biesheuvel, Josh Poimboeuf,
Breno Leitao, Wei Yang, Rong Xu, Thomas Weißschuh,
linux-kernel, kexec, Simon Horman, Dave Young, Peter Zijlstra,
bsz, nathan
From: David Woodhouse <dwmw@amazon.co.uk>
A few places in the kexec control code page make the assumption that the
first instruction of relocate_kernel is at the very start of the page.
To allow for Clang CFI information to be added to relocate_kernel(), as
well as the general principle of removing unwarranted assumptions, fix
them to use the external __relocate_kernel_start symbol that the linker
adds. This means using a separate addq and subq for calculating offsets,
as the assembler can no longer calculate the delta directly for itself
and relocations aren't that versatile. But we can at least use values
relative to a local label to avoid absolute relocations.
Turn the jump from relocate_kernel() to identity_mapped() into a real
indirect 'jmp *%rsi' too, while touching it. There was no real reason
for it to be a push+ret in the first place, and adding Clang CFI info
will also give objtool enough visibility to start complaining 'return
with modified stack frame' about it.
Signed-off-by: David Woodhouse <dwmw@amazon.co.uk>
---
arch/x86/kernel/relocate_kernel_64.S | 18 +++++++++++-------
1 file changed, 11 insertions(+), 7 deletions(-)
diff --git a/arch/x86/kernel/relocate_kernel_64.S b/arch/x86/kernel/relocate_kernel_64.S
index 14ed40b7f322..af2cd06ff318 100644
--- a/arch/x86/kernel/relocate_kernel_64.S
+++ b/arch/x86/kernel/relocate_kernel_64.S
@@ -95,11 +95,10 @@ SYM_CODE_START_NOALIGN(relocate_kernel)
lea PAGE_SIZE(%rsi), %rsp
/* jump to identity mapped page */
- addq $(identity_mapped - relocate_kernel), %rsi
- pushq %rsi
- ANNOTATE_UNRET_SAFE
- ret
- int3
+0: addq $identity_mapped - 0b, %rsi
+ subq $__relocate_kernel_start - 0b, %rsi
+ ANNOTATE_RETPOLINE_SAFE
+ jmp *%rsi
SYM_CODE_END(relocate_kernel)
SYM_CODE_START_LOCAL_NOALIGN(identity_mapped)
@@ -219,16 +218,21 @@ SYM_CODE_START_LOCAL_NOALIGN(identity_mapped)
/* get the re-entry point of the peer system */
popq %rbp
- leaq relocate_kernel(%rip), %r8
movq kexec_pa_swap_page(%rip), %r10
movq pa_backup_pages_map(%rip), %rdi
movq kexec_pa_table_page(%rip), %rax
movq %rax, %cr3
+
+ /* Find start (and end) of this physical mapping of control page */
+ leaq (%rip), %r8
+ ANNOTATE_NOENDBR
+ andq $PAGE_MASK, %r8
lea PAGE_SIZE(%r8), %rsp
movl $1, %r11d /* Ensure preserve_context flag is set */
call swap_pages
movq kexec_va_control_page(%rip), %rax
- addq $(virtual_mapped - relocate_kernel), %rax
+0: addq $virtual_mapped - 0b, %rax
+ subq $__relocate_kernel_start - 0b, %rax
pushq %rax
ANNOTATE_UNRET_SAFE
ret
--
2.47.0
^ permalink raw reply [flat|nested] 23+ messages in thread
* [PATCH v2 9/9] x86/kexec: Use typedef for relocate_kernel_fn function prototype
2025-01-09 14:04 [PATCH v2 0/9] x86/kexec: Fixes for tip/x86/boot David Woodhouse
` (7 preceding siblings ...)
2025-01-09 14:04 ` [PATCH v2 8/9] x86/kexec: Cope with relocate_kernel() not being at the start of the page David Woodhouse
@ 2025-01-09 14:04 ` David Woodhouse
2025-01-15 9:17 ` [tip: x86/boot] " tip-bot2 for David Woodhouse
2025-01-14 11:34 ` [PATCH v2 0/9] x86/kexec: Fixes for tip/x86/boot David Woodhouse
9 siblings, 1 reply; 23+ messages in thread
From: David Woodhouse @ 2025-01-09 14:04 UTC (permalink / raw)
To: Thomas Gleixner, Ingo Molnar, Borislav Petkov, Dave Hansen, x86,
H . Peter Anvin, Eric Biederman, David Woodhouse, Sourabh Jain,
Hari Bathini, Michael Ellerman, Thomas Zimmermann, Andrew Morton,
Baoquan He, Yuntao Wang, David Kaplan, Tao Liu,
Kirill A . Shutemov, Kai Huang, Ard Biesheuvel, Josh Poimboeuf,
Breno Leitao, Wei Yang, Rong Xu, Thomas Weißschuh,
linux-kernel, kexec, Simon Horman, Dave Young, Peter Zijlstra,
bsz, nathan
From: David Woodhouse <dwmw@amazon.co.uk>
Both i386 and x86_64 now copy the relocate_kernel function into the control
page and execute it from there, using an open-coded function pointer.
Use a typedef for it instead.
Signed-off-by: David Woodhouse <dwmw@amazon.co.uk>
---
arch/x86/include/asm/kexec.h | 26 +++++++++++++-------------
arch/x86/kernel/machine_kexec_32.c | 7 +------
arch/x86/kernel/machine_kexec_64.c | 6 +-----
3 files changed, 15 insertions(+), 24 deletions(-)
diff --git a/arch/x86/include/asm/kexec.h b/arch/x86/include/asm/kexec.h
index 48e4f44f794f..8ad187462b68 100644
--- a/arch/x86/include/asm/kexec.h
+++ b/arch/x86/include/asm/kexec.h
@@ -111,21 +111,21 @@ static inline void crash_setup_regs(struct pt_regs *newregs,
}
#ifdef CONFIG_X86_32
-asmlinkage unsigned long
-relocate_kernel(unsigned long indirection_page,
- unsigned long control_page,
- unsigned long start_address,
- unsigned int has_pae,
- unsigned int preserve_context);
+typedef asmlinkage unsigned long
+relocate_kernel_fn(unsigned long indirection_page,
+ unsigned long control_page,
+ unsigned long start_address,
+ unsigned int has_pae,
+ unsigned int preserve_context);
#else
-unsigned long
-relocate_kernel(unsigned long indirection_page,
- unsigned long pa_control_page,
- unsigned long start_address,
- unsigned int preserve_context,
- unsigned int host_mem_enc_active);
+typedef unsigned long
+relocate_kernel_fn(unsigned long indirection_page,
+ unsigned long pa_control_page,
+ unsigned long start_address,
+ unsigned int preserve_context,
+ unsigned int host_mem_enc_active);
#endif
-
+extern relocate_kernel_fn relocate_kernel;
#define ARCH_HAS_KIMAGE_ARCH
#ifdef CONFIG_X86_32
diff --git a/arch/x86/kernel/machine_kexec_32.c b/arch/x86/kernel/machine_kexec_32.c
index 1b373d79cedc..80265162aeff 100644
--- a/arch/x86/kernel/machine_kexec_32.c
+++ b/arch/x86/kernel/machine_kexec_32.c
@@ -160,15 +160,10 @@ void machine_kexec_cleanup(struct kimage *image)
*/
void machine_kexec(struct kimage *image)
{
+ relocate_kernel_fn *relocate_kernel_ptr;
unsigned long page_list[PAGES_NR];
void *control_page;
int save_ftrace_enabled;
- asmlinkage unsigned long
- (*relocate_kernel_ptr)(unsigned long indirection_page,
- unsigned long control_page,
- unsigned long start_address,
- unsigned int has_pae,
- unsigned int preserve_context);
#ifdef CONFIG_KEXEC_JUMP
if (image->preserve_context)
diff --git a/arch/x86/kernel/machine_kexec_64.c b/arch/x86/kernel/machine_kexec_64.c
index 1440f792a86d..dd75a51463a2 100644
--- a/arch/x86/kernel/machine_kexec_64.c
+++ b/arch/x86/kernel/machine_kexec_64.c
@@ -344,12 +344,8 @@ void machine_kexec_cleanup(struct kimage *image)
*/
void __nocfi machine_kexec(struct kimage *image)
{
- unsigned long (*relocate_kernel_ptr)(unsigned long indirection_page,
- unsigned long pa_control_page,
- unsigned long start_address,
- unsigned int preserve_context,
- unsigned int host_mem_enc_active);
unsigned long reloc_start = (unsigned long)__relocate_kernel_start;
+ relocate_kernel_fn *relocate_kernel_ptr;
unsigned int host_mem_enc_active;
int save_ftrace_enabled;
void *control_page;
--
2.47.0
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [PATCH v2 0/9] x86/kexec: Fixes for tip/x86/boot
2025-01-09 14:04 [PATCH v2 0/9] x86/kexec: Fixes for tip/x86/boot David Woodhouse
` (8 preceding siblings ...)
2025-01-09 14:04 ` [PATCH v2 9/9] x86/kexec: Use typedef for relocate_kernel_fn function prototype David Woodhouse
@ 2025-01-14 11:34 ` David Woodhouse
2025-01-14 11:51 ` Borislav Petkov
9 siblings, 1 reply; 23+ messages in thread
From: David Woodhouse @ 2025-01-14 11:34 UTC (permalink / raw)
To: Thomas Gleixner, Ingo Molnar, Borislav Petkov, Dave Hansen, x86,
H . Peter Anvin, Eric Biederman, Sourabh Jain, Hari Bathini,
Michael Ellerman, Thomas Zimmermann, Andrew Morton, Baoquan He,
Yuntao Wang, David Kaplan, Tao Liu, Kirill A . Shutemov,
Kai Huang, Ard Biesheuvel, Josh Poimboeuf, Breno Leitao,
Wei Yang, Rong Xu, Thomas Weißschuh, linux-kernel, kexec,
Simon Horman, Dave Young, Peter Zijlstra, bsz, nathan
[-- Attachment #1: Type: text/plain, Size: 219 bytes --]
On Thu, 2025-01-09 at 14:04 +0000, David Woodhouse wrote:
> Fix a few bugs introduced by the recent rework of the relocate_kernel()
> code, and one which has been there in the kjump code from the beginning.
Ping?
[-- Attachment #2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 5069 bytes --]
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [PATCH v2 0/9] x86/kexec: Fixes for tip/x86/boot
2025-01-14 11:34 ` [PATCH v2 0/9] x86/kexec: Fixes for tip/x86/boot David Woodhouse
@ 2025-01-14 11:51 ` Borislav Petkov
2025-01-14 11:53 ` David Woodhouse
0 siblings, 1 reply; 23+ messages in thread
From: Borislav Petkov @ 2025-01-14 11:51 UTC (permalink / raw)
To: David Woodhouse
Cc: Thomas Gleixner, Ingo Molnar, Dave Hansen, x86, H . Peter Anvin,
Eric Biederman, Sourabh Jain, Hari Bathini, Michael Ellerman,
Thomas Zimmermann, Andrew Morton, Baoquan He, Yuntao Wang,
David Kaplan, Tao Liu, Kirill A . Shutemov, Kai Huang,
Ard Biesheuvel, Josh Poimboeuf, Breno Leitao, Wei Yang, Rong Xu,
Thomas Weißschuh, linux-kernel, kexec, Simon Horman,
Dave Young, Peter Zijlstra, bsz, nathan
On Tue, Jan 14, 2025 at 12:34:29PM +0100, David Woodhouse wrote:
> On Thu, 2025-01-09 at 14:04 +0000, David Woodhouse wrote:
> > Fix a few bugs introduced by the recent rework of the relocate_kernel()
> > code, and one which has been there in the kjump code from the beginning.
>
> Ping?
I'm going through them as we speak.
--
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquette
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [PATCH v2 0/9] x86/kexec: Fixes for tip/x86/boot
2025-01-14 11:51 ` Borislav Petkov
@ 2025-01-14 11:53 ` David Woodhouse
2025-01-14 12:11 ` Borislav Petkov
0 siblings, 1 reply; 23+ messages in thread
From: David Woodhouse @ 2025-01-14 11:53 UTC (permalink / raw)
To: Borislav Petkov
Cc: Thomas Gleixner, Ingo Molnar, Dave Hansen, x86, H . Peter Anvin,
Eric Biederman, Sourabh Jain, Hari Bathini, Michael Ellerman,
Thomas Zimmermann, Andrew Morton, Baoquan He, Yuntao Wang,
David Kaplan, Tao Liu, Kirill A . Shutemov, Kai Huang,
Ard Biesheuvel, Josh Poimboeuf, Breno Leitao, Wei Yang, Rong Xu,
Thomas Weißschuh, linux-kernel, kexec, Simon Horman,
Dave Young, Peter Zijlstra, bsz, nathan
On 14 January 2025 12:51:15 CET, Borislav Petkov <bp@alien8.de> wrote:
>On Tue, Jan 14, 2025 at 12:34:29PM +0100, David Woodhouse wrote:
>> On Thu, 2025-01-09 at 14:04 +0000, David Woodhouse wrote:
>> > Fix a few bugs introduced by the recent rework of the relocate_kernel()
>> > code, and one which has been there in the kjump code from the beginning.
>>
>> Ping?
>
>I'm going through them as we speak.
>
Thank you. Once the dust settles on these I can resend the debugging stuff which is actually what I came here to do.
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [PATCH v2 0/9] x86/kexec: Fixes for tip/x86/boot
2025-01-14 11:53 ` David Woodhouse
@ 2025-01-14 12:11 ` Borislav Petkov
0 siblings, 0 replies; 23+ messages in thread
From: Borislav Petkov @ 2025-01-14 12:11 UTC (permalink / raw)
To: David Woodhouse
Cc: Thomas Gleixner, Ingo Molnar, Dave Hansen, x86, H . Peter Anvin,
Eric Biederman, Sourabh Jain, Hari Bathini, Michael Ellerman,
Thomas Zimmermann, Andrew Morton, Baoquan He, Yuntao Wang,
David Kaplan, Tao Liu, Kirill A . Shutemov, Kai Huang,
Ard Biesheuvel, Josh Poimboeuf, Breno Leitao, Wei Yang, Rong Xu,
Thomas Weißschuh, linux-kernel, kexec, Simon Horman,
Dave Young, Peter Zijlstra, bsz, nathan
On Tue, Jan 14, 2025 at 12:53:45PM +0100, David Woodhouse wrote:
> Thank you. Once the dust settles on these I can resend the debugging stuff
> which is actually what I came here to do.
Sure, let's carry over the merge window successfully first. :)
--
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquette
^ permalink raw reply [flat|nested] 23+ messages in thread
* [tip: x86/boot] x86/kexec: Use typedef for relocate_kernel_fn function prototype
2025-01-09 14:04 ` [PATCH v2 9/9] x86/kexec: Use typedef for relocate_kernel_fn function prototype David Woodhouse
@ 2025-01-15 9:17 ` tip-bot2 for David Woodhouse
0 siblings, 0 replies; 23+ messages in thread
From: tip-bot2 for David Woodhouse @ 2025-01-15 9:17 UTC (permalink / raw)
To: linux-tip-commits
Cc: David Woodhouse, Borislav Petkov (AMD), x86, linux-kernel
The following commit has been merged into the x86/boot branch of tip:
Commit-ID: 7c61a3d8f7a8cf0f0cbdc8459dd86782ba7285f1
Gitweb: https://git.kernel.org/tip/7c61a3d8f7a8cf0f0cbdc8459dd86782ba7285f1
Author: David Woodhouse <dwmw@amazon.co.uk>
AuthorDate: Thu, 09 Jan 2025 14:04:21
Committer: Borislav Petkov (AMD) <bp@alien8.de>
CommitterDate: Tue, 14 Jan 2025 13:09:08 +01:00
x86/kexec: Use typedef for relocate_kernel_fn function prototype
Both i386 and x86_64 now copy the relocate_kernel() function into the control
page and execute it from there, using an open-coded function pointer.
Use a typedef for it instead.
[ bp: Put relocate_kernel_ptr ptr arithmetic on a single line. ]
Signed-off-by: David Woodhouse <dwmw@amazon.co.uk>
Signed-off-by: Borislav Petkov (AMD) <bp@alien8.de>
Link: https://lore.kernel.org/r/20250109140757.2841269-10-dwmw2@infradead.org
---
arch/x86/include/asm/kexec.h | 26 +++++++++++++-------------
arch/x86/kernel/machine_kexec_32.c | 7 +------
arch/x86/kernel/machine_kexec_64.c | 9 ++-------
3 files changed, 16 insertions(+), 26 deletions(-)
diff --git a/arch/x86/include/asm/kexec.h b/arch/x86/include/asm/kexec.h
index 48e4f44..8ad1874 100644
--- a/arch/x86/include/asm/kexec.h
+++ b/arch/x86/include/asm/kexec.h
@@ -111,21 +111,21 @@ static inline void crash_setup_regs(struct pt_regs *newregs,
}
#ifdef CONFIG_X86_32
-asmlinkage unsigned long
-relocate_kernel(unsigned long indirection_page,
- unsigned long control_page,
- unsigned long start_address,
- unsigned int has_pae,
- unsigned int preserve_context);
+typedef asmlinkage unsigned long
+relocate_kernel_fn(unsigned long indirection_page,
+ unsigned long control_page,
+ unsigned long start_address,
+ unsigned int has_pae,
+ unsigned int preserve_context);
#else
-unsigned long
-relocate_kernel(unsigned long indirection_page,
- unsigned long pa_control_page,
- unsigned long start_address,
- unsigned int preserve_context,
- unsigned int host_mem_enc_active);
+typedef unsigned long
+relocate_kernel_fn(unsigned long indirection_page,
+ unsigned long pa_control_page,
+ unsigned long start_address,
+ unsigned int preserve_context,
+ unsigned int host_mem_enc_active);
#endif
-
+extern relocate_kernel_fn relocate_kernel;
#define ARCH_HAS_KIMAGE_ARCH
#ifdef CONFIG_X86_32
diff --git a/arch/x86/kernel/machine_kexec_32.c b/arch/x86/kernel/machine_kexec_32.c
index 1b373d7..8026516 100644
--- a/arch/x86/kernel/machine_kexec_32.c
+++ b/arch/x86/kernel/machine_kexec_32.c
@@ -160,15 +160,10 @@ void machine_kexec_cleanup(struct kimage *image)
*/
void machine_kexec(struct kimage *image)
{
+ relocate_kernel_fn *relocate_kernel_ptr;
unsigned long page_list[PAGES_NR];
void *control_page;
int save_ftrace_enabled;
- asmlinkage unsigned long
- (*relocate_kernel_ptr)(unsigned long indirection_page,
- unsigned long control_page,
- unsigned long start_address,
- unsigned int has_pae,
- unsigned int preserve_context);
#ifdef CONFIG_KEXEC_JUMP
if (image->preserve_context)
diff --git a/arch/x86/kernel/machine_kexec_64.c b/arch/x86/kernel/machine_kexec_64.c
index 1440f79..a68f5a0 100644
--- a/arch/x86/kernel/machine_kexec_64.c
+++ b/arch/x86/kernel/machine_kexec_64.c
@@ -344,12 +344,8 @@ void machine_kexec_cleanup(struct kimage *image)
*/
void __nocfi machine_kexec(struct kimage *image)
{
- unsigned long (*relocate_kernel_ptr)(unsigned long indirection_page,
- unsigned long pa_control_page,
- unsigned long start_address,
- unsigned int preserve_context,
- unsigned int host_mem_enc_active);
unsigned long reloc_start = (unsigned long)__relocate_kernel_start;
+ relocate_kernel_fn *relocate_kernel_ptr;
unsigned int host_mem_enc_active;
int save_ftrace_enabled;
void *control_page;
@@ -391,8 +387,7 @@ void __nocfi machine_kexec(struct kimage *image)
* Allow for the possibility that relocate_kernel might not be at
* the very start of the page.
*/
- relocate_kernel_ptr = control_page + (unsigned long)relocate_kernel -
- reloc_start;
+ relocate_kernel_ptr = control_page + (unsigned long)relocate_kernel - reloc_start;
/*
* The segment registers are funny things, they have both a
^ permalink raw reply [flat|nested] 23+ messages in thread
* [tip: x86/boot] kexec_core: Add and update comments regarding the KEXEC_JUMP flow
2025-01-09 14:04 ` [PATCH v2 7/9] kexec_core: Add and update comments regarding the KEXEC_JUMP flow David Woodhouse
@ 2025-01-15 9:17 ` tip-bot2 for Rafael J. Wysocki
0 siblings, 0 replies; 23+ messages in thread
From: tip-bot2 for Rafael J. Wysocki @ 2025-01-15 9:17 UTC (permalink / raw)
To: linux-tip-commits
Cc: Rafael J. Wysocki, David Woodhouse, Borislav Petkov (AMD),
x86, linux-kernel
The following commit has been merged into the x86/boot branch of tip:
Commit-ID: dc6ffa6cd52d2cd8fa25b25e42c80faf17f5fe33
Gitweb: https://git.kernel.org/tip/dc6ffa6cd52d2cd8fa25b25e42c80faf17f5fe33
Author: Rafael J. Wysocki <rafael.j.wysocki@intel.com>
AuthorDate: Thu, 09 Jan 2025 14:04:19
Committer: Borislav Petkov (AMD) <bp@alien8.de>
CommitterDate: Tue, 14 Jan 2025 13:03:34 +01:00
kexec_core: Add and update comments regarding the KEXEC_JUMP flow
The KEXEC_JUMP flow is analogous to hibernation flows occurring before
and after creating an image and before and after jumping from the
restore kernel to the image one, which is why it uses the same device
callbacks as those hibernation flows.
Add comments explaining that to the code in question and update an
existing comment in it which appears a bit out of context.
No functional changes.
Signed-off-by: Rafael J. Wysocki <rafael.j.wysocki@intel.com>
Signed-off-by: David Woodhouse <dwmw@amazon.co.uk>
Signed-off-by: Borislav Petkov (AMD) <bp@alien8.de>
Link: https://lore.kernel.org/r/20250109140757.2841269-8-dwmw2@infradead.org
---
kernel/kexec_core.c | 23 +++++++++++++++++------
1 file changed, 17 insertions(+), 6 deletions(-)
diff --git a/kernel/kexec_core.c b/kernel/kexec_core.c
index c0caa14..b424a5c 100644
--- a/kernel/kexec_core.c
+++ b/kernel/kexec_core.c
@@ -1001,6 +1001,12 @@ int kernel_kexec(void)
#ifdef CONFIG_KEXEC_JUMP
if (kexec_image->preserve_context) {
+ /*
+ * This flow is analogous to hibernation flows that occur
+ * before creating an image and before jumping from the
+ * restore kernel to the image one, so it uses the same
+ * device callbacks as those two flows.
+ */
pm_prepare_console();
error = freeze_processes();
if (error) {
@@ -1011,12 +1017,10 @@ int kernel_kexec(void)
error = dpm_suspend_start(PMSG_FREEZE);
if (error)
goto Resume_console;
- /* At this point, dpm_suspend_start() has been called,
- * but *not* dpm_suspend_end(). We *must* call
- * dpm_suspend_end() now. Otherwise, drivers for
- * some devices (e.g. interrupt controllers) become
- * desynchronized with the actual state of the
- * hardware at resume time, and evil weirdness ensues.
+ /*
+ * dpm_suspend_end() must be called after dpm_suspend_start()
+ * to complete the transition, like in the hibernation flows
+ * mentioned above.
*/
error = dpm_suspend_end(PMSG_FREEZE);
if (error)
@@ -1052,6 +1056,13 @@ int kernel_kexec(void)
#ifdef CONFIG_KEXEC_JUMP
if (kexec_image->preserve_context) {
+ /*
+ * This flow is analogous to hibernation flows that occur after
+ * creating an image and after the image kernel has got control
+ * back, and in case the devices have been reset or otherwise
+ * manipulated in the meantime, it uses the device callbacks
+ * used by the latter.
+ */
syscore_resume();
Enable_irqs:
local_irq_enable();
^ permalink raw reply [flat|nested] 23+ messages in thread
* [tip: x86/boot] x86/kexec: Cope with relocate_kernel() not being at the start of the page
2025-01-09 14:04 ` [PATCH v2 8/9] x86/kexec: Cope with relocate_kernel() not being at the start of the page David Woodhouse
@ 2025-01-15 9:17 ` tip-bot2 for David Woodhouse
0 siblings, 0 replies; 23+ messages in thread
From: tip-bot2 for David Woodhouse @ 2025-01-15 9:17 UTC (permalink / raw)
To: linux-tip-commits
Cc: David Woodhouse, Borislav Petkov (AMD), x86, linux-kernel
The following commit has been merged into the x86/boot branch of tip:
Commit-ID: e536057543791fbfa0d979f4e782933ea312c38c
Gitweb: https://git.kernel.org/tip/e536057543791fbfa0d979f4e782933ea312c38c
Author: David Woodhouse <dwmw@amazon.co.uk>
AuthorDate: Thu, 09 Jan 2025 14:04:20
Committer: Borislav Petkov (AMD) <bp@alien8.de>
CommitterDate: Tue, 14 Jan 2025 13:05:14 +01:00
x86/kexec: Cope with relocate_kernel() not being at the start of the page
A few places in the kexec control code page make the assumption that the first
instruction of relocate_kernel is at the very start of the page.
To allow for Clang CFI information to be added to relocate_kernel(), as well
as the general principle of removing unwarranted assumptions, fix them to use
the external __relocate_kernel_start symbol that the linker adds. This means
using a separate addq and subq for calculating offsets, as the assembler can
no longer calculate the delta directly for itself and relocations aren't that
versatile. But those values can at least be used relative to a local label to
avoid absolute relocations.
Turn the jump from relocate_kernel() to identity_mapped() into a real indirect
'jmp *%rsi' too, while touching it. There was no real reason for it to be
a push+ret in the first place, and adding Clang CFI info will also give
objtool enough visibility to start complaining 'return with modified stack
frame' about it.
[ bp: Massage commit message. ]
Signed-off-by: David Woodhouse <dwmw@amazon.co.uk>
Signed-off-by: Borislav Petkov (AMD) <bp@alien8.de>
Link: https://lore.kernel.org/r/20250109140757.2841269-9-dwmw2@infradead.org
---
arch/x86/kernel/relocate_kernel_64.S | 18 +++++++++++-------
1 file changed, 11 insertions(+), 7 deletions(-)
diff --git a/arch/x86/kernel/relocate_kernel_64.S b/arch/x86/kernel/relocate_kernel_64.S
index 14ed40b..af2cd06 100644
--- a/arch/x86/kernel/relocate_kernel_64.S
+++ b/arch/x86/kernel/relocate_kernel_64.S
@@ -95,11 +95,10 @@ SYM_CODE_START_NOALIGN(relocate_kernel)
lea PAGE_SIZE(%rsi), %rsp
/* jump to identity mapped page */
- addq $(identity_mapped - relocate_kernel), %rsi
- pushq %rsi
- ANNOTATE_UNRET_SAFE
- ret
- int3
+0: addq $identity_mapped - 0b, %rsi
+ subq $__relocate_kernel_start - 0b, %rsi
+ ANNOTATE_RETPOLINE_SAFE
+ jmp *%rsi
SYM_CODE_END(relocate_kernel)
SYM_CODE_START_LOCAL_NOALIGN(identity_mapped)
@@ -219,16 +218,21 @@ SYM_CODE_START_LOCAL_NOALIGN(identity_mapped)
/* get the re-entry point of the peer system */
popq %rbp
- leaq relocate_kernel(%rip), %r8
movq kexec_pa_swap_page(%rip), %r10
movq pa_backup_pages_map(%rip), %rdi
movq kexec_pa_table_page(%rip), %rax
movq %rax, %cr3
+
+ /* Find start (and end) of this physical mapping of control page */
+ leaq (%rip), %r8
+ ANNOTATE_NOENDBR
+ andq $PAGE_MASK, %r8
lea PAGE_SIZE(%r8), %rsp
movl $1, %r11d /* Ensure preserve_context flag is set */
call swap_pages
movq kexec_va_control_page(%rip), %rax
- addq $(virtual_mapped - relocate_kernel), %rax
+0: addq $virtual_mapped - 0b, %rax
+ subq $__relocate_kernel_start - 0b, %rax
pushq %rax
ANNOTATE_UNRET_SAFE
ret
^ permalink raw reply [flat|nested] 23+ messages in thread
* [tip: x86/boot] x86/kexec: Fix location of relocate_kernel with -ffunction-sections
2025-01-09 14:04 ` [PATCH v2 5/9] x86/kexec: Fix location of relocate_kernel with -ffunction-sections David Woodhouse
@ 2025-01-15 9:17 ` tip-bot2 for Nathan Chancellor
0 siblings, 0 replies; 23+ messages in thread
From: tip-bot2 for Nathan Chancellor @ 2025-01-15 9:17 UTC (permalink / raw)
To: linux-tip-commits
Cc: Nathan Chancellor, David Woodhouse, Borislav Petkov (AMD),
x86, linux-kernel
The following commit has been merged into the x86/boot branch of tip:
Commit-ID: eeed9150411a63dd0611490cf31fdae681427918
Gitweb: https://git.kernel.org/tip/eeed9150411a63dd0611490cf31fdae681427918
Author: Nathan Chancellor <nathan@kernel.org>
AuthorDate: Thu, 09 Jan 2025 14:04:17
Committer: Borislav Petkov (AMD) <bp@alien8.de>
CommitterDate: Tue, 14 Jan 2025 13:00:18 +01:00
x86/kexec: Fix location of relocate_kernel with -ffunction-sections
After commit
cb33ff9e063c ("x86/kexec: Move relocate_kernel to kernel .data section"),
kernels configured with an option that uses -ffunction-sections, such as
CONFIG_LTO_CLANG, crash when kexecing because the value of relocate_kernel
does not match the value of __relocate_kernel_start so incorrect code gets
copied via machine_kexec_prepare().
$ llvm-nm good-vmlinux &| rg relocate_kernel
ffffffff83280d41 T __relocate_kernel_end
ffffffff83280b00 T __relocate_kernel_start
ffffffff83280b00 T relocate_kernel
$ llvm-nm bad-vmlinux &| rg relocate_kernel
ffffffff83266100 D __relocate_kernel_end
ffffffff83266100 D __relocate_kernel_start
ffffffff8120b0d8 T relocate_kernel
When -ffunction-sections is enabled, TEXT_MAIN matches on
'.text.[0-9a-zA-Z_]*' to coalesce the function specific functions back
into .text during link time after they have been optimized. Due to the
placement of TEXT_TEXT before KEXEC_RELOCATE_KERNEL in the x86 linker
script, the .text.relocate_kernel section ends up in .text instead of
.data.
Use a second dot in the relocate_kernel section name to avoid matching
on TEXT_MAIN, which matches a similar situation that happened in
commit
79cd2a11224e ("x86/retpoline,kprobes: Fix position of thunk sections with CONFIG_LTO_CLANG"),
which allows kexec to function properly.
While .data.relocate_kernel still ends up in the .data section via
DATA_MAIN -> DATA_DATA, ensure it is located with the
.text.relocate_kernel section as intended by performing the same
transformation.
Fixes: cb33ff9e063c ("x86/kexec: Move relocate_kernel to kernel .data section")
Fixes: 8dbec5c77bc3 ("x86/kexec: Add data section to relocate_kernel")
Signed-off-by: Nathan Chancellor <nathan@kernel.org>
Signed-off-by: David Woodhouse <dwmw@amazon.co.uk>
Signed-off-by: Borislav Petkov (AMD) <bp@alien8.de>
Link: https://lore.kernel.org/r/20250109140757.2841269-6-dwmw2@infradead.org
---
arch/x86/kernel/relocate_kernel_64.S | 6 +++---
arch/x86/kernel/vmlinux.lds.S | 4 ++--
2 files changed, 5 insertions(+), 5 deletions(-)
diff --git a/arch/x86/kernel/relocate_kernel_64.S b/arch/x86/kernel/relocate_kernel_64.S
index a95691b..14ed40b 100644
--- a/arch/x86/kernel/relocate_kernel_64.S
+++ b/arch/x86/kernel/relocate_kernel_64.S
@@ -23,11 +23,11 @@
#define PAGE_ATTR (_PAGE_PRESENT | _PAGE_RW | _PAGE_ACCESSED | _PAGE_DIRTY)
/*
- * The .text.relocate_kernel and .data.relocate_kernel sections are copied
+ * The .text..relocate_kernel and .data..relocate_kernel sections are copied
* into the control page, and the remainder of the page is used as the stack.
*/
- .section .data.relocate_kernel,"a";
+ .section .data..relocate_kernel,"a";
/* Minimal CPU state */
SYM_DATA_LOCAL(saved_rsp, .quad 0)
SYM_DATA_LOCAL(saved_cr0, .quad 0)
@@ -39,7 +39,7 @@ SYM_DATA(kexec_pa_table_page, .quad 0)
SYM_DATA(kexec_pa_swap_page, .quad 0)
SYM_DATA_LOCAL(pa_backup_pages_map, .quad 0)
- .section .text.relocate_kernel,"ax";
+ .section .text..relocate_kernel,"ax";
.code64
SYM_CODE_START_NOALIGN(relocate_kernel)
UNWIND_HINT_END_OF_STACK
diff --git a/arch/x86/kernel/vmlinux.lds.S b/arch/x86/kernel/vmlinux.lds.S
index 0c89399..63ff60a 100644
--- a/arch/x86/kernel/vmlinux.lds.S
+++ b/arch/x86/kernel/vmlinux.lds.S
@@ -100,8 +100,8 @@ const_pcpu_hot = pcpu_hot;
#define KEXEC_RELOCATE_KERNEL \
. = ALIGN(0x100); \
__relocate_kernel_start = .; \
- *(.text.relocate_kernel); \
- *(.data.relocate_kernel); \
+ *(.text..relocate_kernel); \
+ *(.data..relocate_kernel); \
__relocate_kernel_end = .;
ASSERT(__relocate_kernel_end - __relocate_kernel_start <= KEXEC_CONTROL_CODE_MAX_SIZE,
^ permalink raw reply [flat|nested] 23+ messages in thread
* [tip: x86/boot] x86/kexec: Mark machine_kexec() with __nocfi
2025-01-09 14:04 ` [PATCH v2 6/9] x86/kexec: Mark machine_kexec() with __nocfi David Woodhouse
@ 2025-01-15 9:17 ` tip-bot2 for David Woodhouse
0 siblings, 0 replies; 23+ messages in thread
From: tip-bot2 for David Woodhouse @ 2025-01-15 9:17 UTC (permalink / raw)
To: linux-tip-commits
Cc: Nathan Chancellor, David Woodhouse, Borislav Petkov (AMD),
x86, linux-kernel
The following commit has been merged into the x86/boot branch of tip:
Commit-ID: 2114796ca041f0d3e79e5dd165219b940b23c540
Gitweb: https://git.kernel.org/tip/2114796ca041f0d3e79e5dd165219b940b23c540
Author: David Woodhouse <dwmw@amazon.co.uk>
AuthorDate: Thu, 09 Jan 2025 14:04:18
Committer: Borislav Petkov (AMD) <bp@alien8.de>
CommitterDate: Tue, 14 Jan 2025 13:02:40 +01:00
x86/kexec: Mark machine_kexec() with __nocfi
A recent commit caused the relocate_kernel() function to be invoked through
a function pointer, but it does not have CFI information. The resulting trap
occurs after the IDT and GDT have been invalidated, leading to a triple-fault
if CONFIG_CFI_CLANG is enabled.
Using SYM_TYPED_FUNC_START() to provide the CFI information looks like it will
require a prolonged battle with objtool. And is fairly pointless anyway, as
the actual signature comes from a __kcfi_typeid_… symbol emitted from the
C code based on the function prototype it thinks that relocate_kernel has,
rendering the check somewhat tautological.
The simple fix is just to mark machine_kexec() with __nocfi.
Fixes: eeebbde57113 ("x86/kexec: Invoke copy of relocate_kernel() instead of the original")
Reported-by: Nathan Chancellor <nathan@kernel.org>
Suggested-by: Nathan Chancellor <nathan@kernel.org>
Signed-off-by: David Woodhouse <dwmw@amazon.co.uk>
Signed-off-by: Borislav Petkov (AMD) <bp@alien8.de>
Link: https://lore.kernel.org/r/20250109140757.2841269-7-dwmw2@infradead.org
---
arch/x86/kernel/machine_kexec_64.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/arch/x86/kernel/machine_kexec_64.c b/arch/x86/kernel/machine_kexec_64.c
index 9232ad1..1440f79 100644
--- a/arch/x86/kernel/machine_kexec_64.c
+++ b/arch/x86/kernel/machine_kexec_64.c
@@ -342,7 +342,7 @@ void machine_kexec_cleanup(struct kimage *image)
* Do not allocate memory (or fail in any way) in machine_kexec().
* We are past the point of no return, committed to rebooting now.
*/
-void machine_kexec(struct kimage *image)
+void __nocfi machine_kexec(struct kimage *image)
{
unsigned long (*relocate_kernel_ptr)(unsigned long indirection_page,
unsigned long pa_control_page,
^ permalink raw reply [flat|nested] 23+ messages in thread
* [tip: x86/boot] x86/kexec: Fix stack and handling of re-entry point for ::preserve_context
2025-01-09 14:04 ` [PATCH v2 4/9] x86/kexec: Fix stack and handling of re-entry point for ::preserve_context David Woodhouse
@ 2025-01-15 9:17 ` tip-bot2 for David Woodhouse
0 siblings, 0 replies; 23+ messages in thread
From: tip-bot2 for David Woodhouse @ 2025-01-15 9:17 UTC (permalink / raw)
To: linux-tip-commits
Cc: David Woodhouse, Borislav Petkov (AMD), x86, linux-kernel
The following commit has been merged into the x86/boot branch of tip:
Commit-ID: 2cacf7f23a024ab1fdc603ca6a4f4c8b2de9f64e
Gitweb: https://git.kernel.org/tip/2cacf7f23a024ab1fdc603ca6a4f4c8b2de9f64e
Author: David Woodhouse <dwmw@amazon.co.uk>
AuthorDate: Thu, 09 Jan 2025 14:04:16
Committer: Borislav Petkov (AMD) <bp@alien8.de>
CommitterDate: Tue, 14 Jan 2025 12:57:43 +01:00
x86/kexec: Fix stack and handling of re-entry point for ::preserve_context
A ::preserve_context kimage can be invoked more than once, and the entry point
can be different every time. When the callee returns to the kernel, it leaves
the address of its entry point for next time on the stack.
That being the case, one might reasonably assume that the caller would
allocate space for it on the stack frame before actually performing the 'call'
into the callee.
Apparently not, though. Ever since the kjump code was first added in 2009, it
has set up a *new* stack at the top of the swap_page scratch page, then just
performed the 'call' without allocating any space for the re-entry address to
be returned. It then reads the re-entry point for next time from 0(%rsp) which
is actually the first qword of the page *after* the swap page, which might not
exist at all! And if the callee has written to that, then it will have
corrupted memory it doesn't own.
Correct this by pushing the entry point of the callee onto the stack before
calling it. The callee may then adjust it, or not, as it sees fit, and
subsequent invocations should work correctly either way.
Remove a stray push of zero to the *relocate_kernel* stack, which may have
been intended for this purpose, but which was actually just noise.
Also, loading the stack for the callee relied on the address of the swap page
being in %r10 without ever documenting that fact. Recent code changes made
that no longer true, so load it directly from the local kexec_pa_swap_page
variable instead.
Fixes: b3adabae8a96 ("x86/kexec: Drop page_list argument from relocate_kernel()")
Signed-off-by: David Woodhouse <dwmw@amazon.co.uk>
Signed-off-by: Borislav Petkov (AMD) <bp@alien8.de>
Link: https://lore.kernel.org/r/20250109140757.2841269-5-dwmw2@infradead.org
---
arch/x86/kernel/relocate_kernel_64.S | 12 +++++++++---
1 file changed, 9 insertions(+), 3 deletions(-)
diff --git a/arch/x86/kernel/relocate_kernel_64.S b/arch/x86/kernel/relocate_kernel_64.S
index 3ca3bf6..a95691b 100644
--- a/arch/x86/kernel/relocate_kernel_64.S
+++ b/arch/x86/kernel/relocate_kernel_64.S
@@ -113,8 +113,6 @@ SYM_CODE_START_LOCAL_NOALIGN(identity_mapped)
* %r13 original CR4 when relocate_kernel() was invoked
*/
- /* set return address to 0 if not preserving context */
- pushq $0
/* store the start address on the stack */
pushq %rdx
@@ -208,12 +206,19 @@ SYM_CODE_START_LOCAL_NOALIGN(identity_mapped)
.Lrelocate:
popq %rdx
+
+ /* Use the swap page for the callee's stack */
+ movq kexec_pa_swap_page(%rip), %r10
leaq PAGE_SIZE(%r10), %rsp
+
+ /* push the existing entry point onto the callee's stack */
+ pushq %rdx
+
ANNOTATE_RETPOLINE_SAFE
call *%rdx
/* get the re-entry point of the peer system */
- movq 0(%rsp), %rbp
+ popq %rbp
leaq relocate_kernel(%rip), %r8
movq kexec_pa_swap_page(%rip), %r10
movq pa_backup_pages_map(%rip), %rdi
@@ -247,6 +252,7 @@ SYM_CODE_START_LOCAL_NOALIGN(virtual_mapped)
lgdt saved_context_gdt_desc(%rax)
#endif
+ /* relocate_kernel() returns the re-entry point for next time */
movq %rbp, %rax
popf
^ permalink raw reply [flat|nested] 23+ messages in thread
* [tip: x86/boot] x86/kexec: Ensure preserve_context flag is set on return to kernel
2025-01-09 14:04 ` [PATCH v2 2/9] x86/kexec: Ensure preserve_context flag is set on return to kernel David Woodhouse
@ 2025-01-15 9:17 ` tip-bot2 for David Woodhouse
0 siblings, 0 replies; 23+ messages in thread
From: tip-bot2 for David Woodhouse @ 2025-01-15 9:17 UTC (permalink / raw)
To: linux-tip-commits
Cc: David Woodhouse, Borislav Petkov (AMD), x86, linux-kernel
The following commit has been merged into the x86/boot branch of tip:
Commit-ID: 4d5f1da98f5e94ecc39d6e7a990db2efe0ae3810
Gitweb: https://git.kernel.org/tip/4d5f1da98f5e94ecc39d6e7a990db2efe0ae3810
Author: David Woodhouse <dwmw@amazon.co.uk>
AuthorDate: Thu, 09 Jan 2025 14:04:14
Committer: Borislav Petkov (AMD) <bp@alien8.de>
CommitterDate: Tue, 14 Jan 2025 12:52:47 +01:00
x86/kexec: Ensure preserve_context flag is set on return to kernel
The swap_pages() function will only actually *swap*, as its name implies, if
the preserve_context flag in the %r11 register is non-zero. On the way back
from a ::preserve_context kexec, ensure that the %r11 register is non-zero so
that the pages get swapped back.
Fixes: 9e5683e2d0b5 ("x86/kexec: Only swap pages for ::preserve_context mode")
Signed-off-by: David Woodhouse <dwmw@amazon.co.uk>
Signed-off-by: Borislav Petkov (AMD) <bp@alien8.de>
Link: https://lore.kernel.org/r/20250109140757.2841269-3-dwmw2@infradead.org
---
arch/x86/kernel/relocate_kernel_64.S | 1 +
1 file changed, 1 insertion(+)
diff --git a/arch/x86/kernel/relocate_kernel_64.S b/arch/x86/kernel/relocate_kernel_64.S
index 9bd601d..6fce4b4 100644
--- a/arch/x86/kernel/relocate_kernel_64.S
+++ b/arch/x86/kernel/relocate_kernel_64.S
@@ -220,6 +220,7 @@ SYM_CODE_START_LOCAL_NOALIGN(identity_mapped)
movq kexec_pa_table_page(%rip), %rax
movq %rax, %cr3
lea PAGE_SIZE(%r8), %rsp
+ movl $1, %r11d /* Ensure preserve_context flag is set */
call swap_pages
movq kexec_va_control_page(%rip), %rax
addq $(virtual_mapped - relocate_kernel), %rax
^ permalink raw reply [flat|nested] 23+ messages in thread
* [tip: x86/boot] x86/kexec: Use correct swap page in swap_pages function
2025-01-09 14:04 ` [PATCH v2 3/9] x86/kexec: Use correct swap page in swap_pages function David Woodhouse
@ 2025-01-15 9:17 ` tip-bot2 for David Woodhouse
0 siblings, 0 replies; 23+ messages in thread
From: tip-bot2 for David Woodhouse @ 2025-01-15 9:17 UTC (permalink / raw)
To: linux-tip-commits
Cc: David Woodhouse, Borislav Petkov (AMD), x86, linux-kernel
The following commit has been merged into the x86/boot branch of tip:
Commit-ID: 85d724df8c82c060dcdeb8d0de0bd986e6c37b72
Gitweb: https://git.kernel.org/tip/85d724df8c82c060dcdeb8d0de0bd986e6c37b72
Author: David Woodhouse <dwmw@amazon.co.uk>
AuthorDate: Thu, 09 Jan 2025 14:04:15
Committer: Borislav Petkov (AMD) <bp@alien8.de>
CommitterDate: Tue, 14 Jan 2025 12:54:36 +01:00
x86/kexec: Use correct swap page in swap_pages function
The swap_pages function expects the swap page to be in %r10, but there
was no documentation to that effect. Once upon a time the setup code
used to load its value from a kernel virtual address and save it to an
address which is accessible in the identity-mapped page tables, and
*happened* to use %r10 to do so, with no comment that it was left there
on *purpose* instead of just being a scratch register. Once that was no
longer necessary, %r10 just holds whatever the kernel happened to leave
in it.
Now that the original value passed by the kernel is accessible via
%rip-relative addressing, load directly from there instead of using %r10
for it. But document the other parameters that the swap_pages function
*does* expect in registers.
Fixes: b3adabae8a96 ("x86/kexec: Drop page_list argument from relocate_kernel()")
Signed-off-by: David Woodhouse <dwmw@amazon.co.uk>
Signed-off-by: Borislav Petkov (AMD) <bp@alien8.de>
Link: https://lore.kernel.org/r/20250109140757.2841269-4-dwmw2@infradead.org
---
arch/x86/kernel/relocate_kernel_64.S | 8 ++++++--
1 file changed, 6 insertions(+), 2 deletions(-)
diff --git a/arch/x86/kernel/relocate_kernel_64.S b/arch/x86/kernel/relocate_kernel_64.S
index 6fce4b4..3ca3bf6 100644
--- a/arch/x86/kernel/relocate_kernel_64.S
+++ b/arch/x86/kernel/relocate_kernel_64.S
@@ -264,6 +264,10 @@ SYM_CODE_END(virtual_mapped)
/* Do the copies */
SYM_CODE_START_LOCAL_NOALIGN(swap_pages)
UNWIND_HINT_END_OF_STACK
+ /*
+ * %rdi indirection page
+ * %r11 preserve_context
+ */
movq %rdi, %rcx /* Put the indirection_page in %rcx */
xorl %edi, %edi
xorl %esi, %esi
@@ -302,7 +306,7 @@ SYM_CODE_START_LOCAL_NOALIGN(swap_pages)
jz .Lnoswap
/* copy source page to swap page */
- movq %r10, %rdi
+ movq kexec_pa_swap_page(%rip), %rdi
movl $512, %ecx
rep ; movsq
@@ -314,7 +318,7 @@ SYM_CODE_START_LOCAL_NOALIGN(swap_pages)
/* copy swap page to destination page */
movq %rdx, %rdi
- movq %r10, %rsi
+ movq kexec_pa_swap_page(%rip), %rsi
.Lnoswap:
movl $512, %ecx
rep ; movsq
^ permalink raw reply [flat|nested] 23+ messages in thread
* [tip: x86/boot] x86/kexec: Disable global pages before writing to control page
2025-01-09 14:04 ` [PATCH v2 1/9] x86/kexec: Disable global pages before writing to control page David Woodhouse
@ 2025-01-15 9:17 ` tip-bot2 for David Woodhouse
0 siblings, 0 replies; 23+ messages in thread
From: tip-bot2 for David Woodhouse @ 2025-01-15 9:17 UTC (permalink / raw)
To: linux-tip-commits
Cc: Nathan Chancellor, Ning, Hongyu, Dave Hansen, David Woodhouse,
Borislav Petkov (AMD),
x86, linux-kernel
The following commit has been merged into the x86/boot branch of tip:
Commit-ID: d144d8a65286fb4a9f06ca80dea6e02e7d846558
Gitweb: https://git.kernel.org/tip/d144d8a65286fb4a9f06ca80dea6e02e7d846558
Author: David Woodhouse <dwmw@amazon.co.uk>
AuthorDate: Thu, 09 Jan 2025 14:04:13
Committer: Borislav Petkov (AMD) <bp@alien8.de>
CommitterDate: Tue, 14 Jan 2025 12:46:17 +01:00
x86/kexec: Disable global pages before writing to control page
The kernel switches to a new set of page tables during kexec. The global
mappings (_PAGE_GLOBAL==1) can remain in the TLB after this switch. This
is generally not a problem because the new page tables use a different
portion of the virtual address space than the normal kernel mappings.
The critical exception to that generalisation (and the only mapping
which isn't an identity mapping) is the kexec control page itself —
which was ROX in the original kernel mapping, but should be RWX in the
new page tables. If there is a global TLB entry for that in its prior
read-only state, it definitely needs to be flushed before attempting to
write through that virtual mapping.
It would be possible to just avoid writing to the virtual address of the
page and defer all writes until they can be done through the identity
mapping. But there's no good reason to keep the old TLB entries around,
as they can cause nothing but trouble.
Clear the PGE bit in %cr4 early, before storing data in the control page.
Fixes: 5a82223e0743 ("x86/kexec: Mark relocate_kernel page as ROX instead of RWX")
Closes: https://bugzilla.kernel.org/show_bug.cgi?id=219592
Reported-by: Nathan Chancellor <nathan@kernel.org>
Reported-by: "Ning, Hongyu" <hongyu.ning@linux.intel.com>
Co-developed-by: Dave Hansen <dave.hansen@linux.intel.com>
Signed-off-by: Dave Hansen <dave.hansen@linux.intel.com>
Signed-off-by: David Woodhouse <dwmw@amazon.co.uk>
Signed-off-by: Borislav Petkov (AMD) <bp@alien8.de>
Tested-by: Nathan Chancellor <nathan@kernel.org>
Tested-by: "Ning, Hongyu" <hongyu.ning@linux.intel.com>
Link: https://lore.kernel.org/r/20250109140757.2841269-2-dwmw2@infradead.org
---
arch/x86/kernel/relocate_kernel_64.S | 12 +++++++++---
1 file changed, 9 insertions(+), 3 deletions(-)
diff --git a/arch/x86/kernel/relocate_kernel_64.S b/arch/x86/kernel/relocate_kernel_64.S
index 8bc86a1..9bd601d 100644
--- a/arch/x86/kernel/relocate_kernel_64.S
+++ b/arch/x86/kernel/relocate_kernel_64.S
@@ -70,14 +70,20 @@ SYM_CODE_START_NOALIGN(relocate_kernel)
movq kexec_pa_table_page(%rip), %r9
movq %r9, %cr3
+ /* Leave CR4 in %r13 to enable the right paging mode later. */
+ movq %cr4, %r13
+
+ /* Disable global pages immediately to ensure this mapping is RWX */
+ movq %r13, %r12
+ andq $~(X86_CR4_PGE), %r12
+ movq %r12, %cr4
+
/* Save %rsp and CRs. */
+ movq %r13, saved_cr4(%rip)
movq %rsp, saved_rsp(%rip)
movq %rax, saved_cr3(%rip)
movq %cr0, %rax
movq %rax, saved_cr0(%rip)
- /* Leave CR4 in %r13 to enable the right paging mode later. */
- movq %cr4, %r13
- movq %r13, saved_cr4(%rip)
/* save indirection list for jumping back */
movq %rdi, pa_backup_pages_map(%rip)
^ permalink raw reply [flat|nested] 23+ messages in thread
end of thread, other threads:[~2025-01-15 9:17 UTC | newest]
Thread overview: 23+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2025-01-09 14:04 [PATCH v2 0/9] x86/kexec: Fixes for tip/x86/boot David Woodhouse
2025-01-09 14:04 ` [PATCH v2 1/9] x86/kexec: Disable global pages before writing to control page David Woodhouse
2025-01-15 9:17 ` [tip: x86/boot] " tip-bot2 for David Woodhouse
2025-01-09 14:04 ` [PATCH v2 2/9] x86/kexec: Ensure preserve_context flag is set on return to kernel David Woodhouse
2025-01-15 9:17 ` [tip: x86/boot] " tip-bot2 for David Woodhouse
2025-01-09 14:04 ` [PATCH v2 3/9] x86/kexec: Use correct swap page in swap_pages function David Woodhouse
2025-01-15 9:17 ` [tip: x86/boot] " tip-bot2 for David Woodhouse
2025-01-09 14:04 ` [PATCH v2 4/9] x86/kexec: Fix stack and handling of re-entry point for ::preserve_context David Woodhouse
2025-01-15 9:17 ` [tip: x86/boot] " tip-bot2 for David Woodhouse
2025-01-09 14:04 ` [PATCH v2 5/9] x86/kexec: Fix location of relocate_kernel with -ffunction-sections David Woodhouse
2025-01-15 9:17 ` [tip: x86/boot] " tip-bot2 for Nathan Chancellor
2025-01-09 14:04 ` [PATCH v2 6/9] x86/kexec: Mark machine_kexec() with __nocfi David Woodhouse
2025-01-15 9:17 ` [tip: x86/boot] " tip-bot2 for David Woodhouse
2025-01-09 14:04 ` [PATCH v2 7/9] kexec_core: Add and update comments regarding the KEXEC_JUMP flow David Woodhouse
2025-01-15 9:17 ` [tip: x86/boot] " tip-bot2 for Rafael J. Wysocki
2025-01-09 14:04 ` [PATCH v2 8/9] x86/kexec: Cope with relocate_kernel() not being at the start of the page David Woodhouse
2025-01-15 9:17 ` [tip: x86/boot] " tip-bot2 for David Woodhouse
2025-01-09 14:04 ` [PATCH v2 9/9] x86/kexec: Use typedef for relocate_kernel_fn function prototype David Woodhouse
2025-01-15 9:17 ` [tip: x86/boot] " tip-bot2 for David Woodhouse
2025-01-14 11:34 ` [PATCH v2 0/9] x86/kexec: Fixes for tip/x86/boot David Woodhouse
2025-01-14 11:51 ` Borislav Petkov
2025-01-14 11:53 ` David Woodhouse
2025-01-14 12:11 ` Borislav Petkov
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®