BUG: Sporadic crashes with current Linus tree

BUG: Sporadic crashes with current Linus tree

[Thomas Gleixner]

Hi!

I've seen the following crash sporadically with commit 46c1e79fee:

Have not seen that with 3882a734c19b, though I saw the PCID warnings on
that machine.

I have no idea how to reproduce so bisecting is pretty much pointless. Any
idea what to do?

Thanks,

	tglx


BUG: unable to handle kernel paging request at ffffffffffffffd8
IP: __memmove+0x24/0x1a0
PGD 1e17c0c067 P4D 1e17c0c067 PUD 1e17c0e067 PMD 0
Oops: 0002 [#1] SMP
CPU: 43 PID: 1958 Comm: rsyslogd Tainted: G        W       4.13.0+ #184
task: ffff9eb5e8780080 task.stack: ffffb648a08cc000
RIP: 0010:__memmove+0x24/0x1a0
RSP: 0018:ffffb648a08cfef0 EFLAGS: 00252097
RAX: ffffffffffffffd8 RBX: ffffffffffffff50 RCX: 0000000000000028
RDX: 0000000000000028 RSI: ffffb648a08cffd8 RDI: ffffffffffffffd8
RBP: ffffb648a08cff10 R08: ffffb648a08d0000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000293 R12: 0000000000000000
R13: ffffb648a08cff20 R14: 00007fffef2ed9a8 R15: 0000000000000000
FS:  00007f05281eff80(0000) GS:ffff9eb5ff5c0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: ffffffffffffffd8 CR3: 000000302857c002 CR4: 00000000001606e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
? fixup_bad_iret+0x3c/0x60
  error_entry+0xb4/0xc0
? general_protection+0xc/0x30
Code: 90 90 90 90 90 90 90 48 89 f8 48 83 fa 20 0f 82 03 01 00 00 48 39 fe 7d 0f 49 89 f0 49 01 d0 49 39 f8 0f 8f 9f 00 00 00 48 89 d1 <f3> a4 c3 48 81 fa a8 02 00 00 72 05 40 38 fe 74 3b 48 83 ea 20
RIP: __memmove+0x24/0x1a0 RSP: ffffb648a08cfef0
CR2: ffffffffffffffd8
[ end trace 72d77f3e20344f94 ]---

Re: BUG: Sporadic crashes with current Linus tree

[Andy Lutomirski]

On Thu, Sep 14, 2017 at 12:38 AM, Thomas Gleixner <tglx@linutronix.de> wrote:
> Hi!
>
> I've seen the following crash sporadically with commit 46c1e79fee:
>
> Have not seen that with 3882a734c19b, though I saw the PCID warnings on
> that machine.
>
> I have no idea how to reproduce so bisecting is pretty much pointless. Any
> idea what to do?

Does tools/testing/selftests/x86/sigreturn_64 reproduce it?

>
> Thanks,
>
>         tglx
>
>
> BUG: unable to handle kernel paging request at ffffffffffffffd8
> IP: __memmove+0x24/0x1a0
> PGD 1e17c0c067 P4D 1e17c0c067 PUD 1e17c0e067 PMD 0
> Oops: 0002 [#1] SMP
> CPU: 43 PID: 1958 Comm: rsyslogd Tainted: G        W       4.13.0+ #184
> task: ffff9eb5e8780080 task.stack: ffffb648a08cc000
> RIP: 0010:__memmove+0x24/0x1a0
> RSP: 0018:ffffb648a08cfef0 EFLAGS: 00252097
> RAX: ffffffffffffffd8 RBX: ffffffffffffff50 RCX: 0000000000000028
> RDX: 0000000000000028 RSI: ffffb648a08cffd8 RDI: ffffffffffffffd8
> RBP: ffffb648a08cff10 R08: ffffb648a08d0000 R09: 0000000000000000
> R10: 0000000000000000 R11: 0000000000000293 R12: 0000000000000000
> R13: ffffb648a08cff20 R14: 00007fffef2ed9a8 R15: 0000000000000000
> FS:  00007f05281eff80(0000) GS:ffff9eb5ff5c0000(0000) knlGS:0000000000000000
> CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> CR2: ffffffffffffffd8 CR3: 000000302857c002 CR4: 00000000001606e0
> DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
> Call Trace:
> ? fixup_bad_iret+0x3c/0x60
>   error_entry+0xb4/0xc0
> ? general_protection+0xc/0x30
> Code: 90 90 90 90 90 90 90 48 89 f8 48 83 fa 20 0f 82 03 01 00 00 48 39 fe 7d 0f 49 89 f0 49 01 d0 49 39 f8 0f 8f 9f 00 00 00 48 89 d1 <f3> a4 c3 48 81 fa a8 02 00 00 72 05 40 38 fe 74 3b 48 83 ea 20
> RIP: __memmove+0x24/0x1a0 RSP: ffffb648a08cfef0
> CR2: ffffffffffffffd8

Ugh, weird.  It kind of looks like current->thread.sp0 == NULL.  I
have a patch series that changes a bunch of that code in my git tree,
but that's definitely not in Linus' tree.

Re: BUG: Sporadic crashes with current Linus tree

[Thomas Gleixner]

On Thu, 14 Sep 2017, Andy Lutomirski wrote:
> On Thu, Sep 14, 2017 at 12:38 AM, Thomas Gleixner <tglx@linutronix.de> wrote:
> > Hi!
> >
> > I've seen the following crash sporadically with commit 46c1e79fee:
> >
> > Have not seen that with 3882a734c19b, though I saw the PCID warnings on
> > that machine.
> >
> > I have no idea how to reproduce so bisecting is pretty much pointless. Any
> > idea what to do?
> 
> Does tools/testing/selftests/x86/sigreturn_64 reproduce it?

Will try tomorrow once I figured out how to compile that stuff. Invoking a
simple make in that directory fails.

Built it manually and when I run it tells: stack16 is too high

> Ugh, weird.  It kind of looks like current->thread.sp0 == NULL.  I
> have a patch series that changes a bunch of that code in my git tree,
> but that's definitely not in Linus' tree.

Right. The stupid thing is that the machine did not throw up all day
neither idle nor loaded. Still the same kernel which barfed tonight several
times.

Thanks,

	tglx

Re: BUG: Sporadic crashes with current Linus tree

[Andy Lutomirski]

[-- Attachment #1: Type: text/plain, Size: 1530 bytes --]

On Thu, Sep 14, 2017 at 9:00 AM, Thomas Gleixner <tglx@linutronix.de> wrote:
> On Thu, 14 Sep 2017, Andy Lutomirski wrote:
>> On Thu, Sep 14, 2017 at 12:38 AM, Thomas Gleixner <tglx@linutronix.de> wrote:
>> > Hi!
>> >
>> > I've seen the following crash sporadically with commit 46c1e79fee:
>> >
>> > Have not seen that with 3882a734c19b, though I saw the PCID warnings on
>> > that machine.
>> >
>> > I have no idea how to reproduce so bisecting is pretty much pointless. Any
>> > idea what to do?
>>
>> Does tools/testing/selftests/x86/sigreturn_64 reproduce it?
>
> Will try tomorrow once I figured out how to compile that stuff. Invoking a
> simple make in that directory fails.

What's the error?  It works for me.

>
> Built it manually and when I run it tells: stack16 is too high
>
>> Ugh, weird.  It kind of looks like current->thread.sp0 == NULL.  I
>> have a patch series that changes a bunch of that code in my git tree,
>> but that's definitely not in Linus' tree.
>
> Right. The stupid thing is that the machine did not throw up all day
> neither idle nor loaded. Still the same kernel which barfed tonight several
> times.

This is weird.  The crashing process is rsyslogd, which should have
been running for a long time and shouldn't have any strange state.  I
wonder if this is some kind of memory corruption.  There would have to
be corruption of thread_struct *and* some kind of issue causing IRET
to fail, though.

The attached patch could plausibly give some useful hint.

>
> Thanks,
>
>         tglx
>
>
>
>

[-- Attachment #2: fixup_bad_iret.patch --]
[-- Type: text/x-patch, Size: 701 bytes --]

diff --git a/arch/x86/kernel/traps.c b/arch/x86/kernel/traps.c
index 34ea3651362e..fe95ea1da5cd 100644
--- a/arch/x86/kernel/traps.c
+++ b/arch/x86/kernel/traps.c
@@ -628,6 +628,12 @@ struct bad_iret_stack *fixup_bad_iret(struct bad_iret_stack *s)
 		container_of(task_pt_regs(current),
 			     struct bad_iret_stack, regs);
 
+	struct pt_regs *old = container_of((unsigned long *)s->regs.sp, struct pt_regs, ip);
+
+	pr_err("fixup_bad_iret.  my sp0 = %lx\n", current->thread.sp0);
+	pr_err("RIP = %lx:%lx RSP = %lx:%lx FLAGS=%lx\n", old->cs, old->ip,
+	       old->ss, old->sp, old->flags);
+
 	/* Copy the IRET target to the new stack. */
 	memmove(&new_stack->regs.ip, (void *)s->regs.sp, 5*8);
 

Re: BUG: Sporadic crashes with current Linus tree

[Thomas Gleixner]

On Thu, 14 Sep 2017, Andy Lutomirski wrote:
> On Thu, Sep 14, 2017 at 9:00 AM, Thomas Gleixner <tglx@linutronix.de> wrote:
> > On Thu, 14 Sep 2017, Andy Lutomirski wrote:
> >> On Thu, Sep 14, 2017 at 12:38 AM, Thomas Gleixner <tglx@linutronix.de> wrote:
> >> > Hi!
> >> >
> >> > I've seen the following crash sporadically with commit 46c1e79fee:
> >> >
> >> > Have not seen that with 3882a734c19b, though I saw the PCID warnings on
> >> > that machine.
> >> >
> >> > I have no idea how to reproduce so bisecting is pretty much pointless. Any
> >> > idea what to do?
> >>
> >> Does tools/testing/selftests/x86/sigreturn_64 reproduce it?
> >
> > Will try tomorrow once I figured out how to compile that stuff. Invoking a
> > simple make in that directory fails.
> 
> What's the error?  It works for me.

gcc -m64 -o /home/tglx/work/kernel/linus/linux/tools/testing/selftests/x86/sysret_ss_attrs_64 -O2 -g -std=gnu99 -pthread -Wall  sysret_ss_attrs.c thunks.S -lrt -ldl
/usr/bin/ld: /tmp/cco4vSkU.o: relocation R_X86_64_32S against `.text' can not be used when making a shared object; recompile with -fPIC
/usr/bin/ld: final link failed: Nonrepresentable section on output
collect2: error: ld returned 1 exit status

> >
> > Built it manually and when I run it tells: stack16 is too high
> >
> >> Ugh, weird.  It kind of looks like current->thread.sp0 == NULL.  I
> >> have a patch series that changes a bunch of that code in my git tree,
> >> but that's definitely not in Linus' tree.
> >
> > Right. The stupid thing is that the machine did not throw up all day
> > neither idle nor loaded. Still the same kernel which barfed tonight several
> > times.
> 
> This is weird.  The crashing process is rsyslogd, which should have
> been running for a long time and shouldn't have any strange state.  I
> wonder if this is some kind of memory corruption.  There would have to
> be corruption of thread_struct *and* some kind of issue causing IRET
> to fail, though.
> 
> The attached patch could plausibly give some useful hint.

I'll put it on that machine and hope it will reproduce. Didn't die since
yesterday moring ....

Thanks,

	tglx

Re: BUG: Sporadic crashes with current Linus tree

[Dmitry Safonov]

Hi Tglx, Andy,

Sorry for old-posting,

2017-09-15 8:09 GMT+01:00 Thomas Gleixner <tglx@linutronix.de>:
> On Thu, 14 Sep 2017, Andy Lutomirski wrote:
>> On Thu, Sep 14, 2017 at 9:00 AM, Thomas Gleixner <tglx@linutronix.de> wrote:
>> > On Thu, 14 Sep 2017, Andy Lutomirski wrote:
>> >> On Thu, Sep 14, 2017 at 12:38 AM, Thomas Gleixner <tglx@linutronix.de> wrote:
>> >> > Hi!
>> >> >
>> >> > I've seen the following crash sporadically with commit 46c1e79fee:
>> >> >
>> >> > Have not seen that with 3882a734c19b, though I saw the PCID warnings on
>> >> > that machine.
>> >> >
>> >> > I have no idea how to reproduce so bisecting is pretty much pointless. Any
>> >> > idea what to do?
>> >>
>> >> Does tools/testing/selftests/x86/sigreturn_64 reproduce it?
>> >
>> > Will try tomorrow once I figured out how to compile that stuff. Invoking a
>> > simple make in that directory fails.
>>
>> What's the error?  It works for me.
>
> gcc -m64 -o /home/tglx/work/kernel/linus/linux/tools/testing/selftests/x86/sysret_ss_attrs_64 -O2 -g -std=gnu99 -pthread -Wall  sysret_ss_attrs.c thunks.S -lrt -ldl
> /usr/bin/ld: /tmp/cco4vSkU.o: relocation R_X86_64_32S against `.text' can not be used when making a shared object; recompile with -fPIC
> /usr/bin/ld: final link failed: Nonrepresentable section on output
> collect2: error: ld returned 1 exit status

Had the same issue in copied thunks.S helper to CRIU.
As I wanted to compile CRIU as pie also (which is the default now
in some distributions), I fixed it up by using %rip-relative addressing
and pushing segment descriptor to stack before long-jumping.
Not sure if that's the issue for the selftest as it can be just always
compiled as pic.

Anyway, here is the commit if you want to look:
https://github.com/0x7f454c46/criu/commit/cf36ea5d8408bf1e42c3bd21b9594369ea7123fb

-- 
             Dmitry

Re: BUG: Sporadic crashes with current Linus tree

[Andy Lutomirski]

On Fri, Oct 13, 2017 at 8:54 AM, Dmitry Safonov <0x7f454c46@gmail.com> wrote:
> Hi Tglx, Andy,
>
> Sorry for old-posting,
>
> 2017-09-15 8:09 GMT+01:00 Thomas Gleixner <tglx@linutronix.de>:
>> On Thu, 14 Sep 2017, Andy Lutomirski wrote:
>>> On Thu, Sep 14, 2017 at 9:00 AM, Thomas Gleixner <tglx@linutronix.de> wrote:
>>> > On Thu, 14 Sep 2017, Andy Lutomirski wrote:
>>> >> On Thu, Sep 14, 2017 at 12:38 AM, Thomas Gleixner <tglx@linutronix.de> wrote:
>>> >> > Hi!
>>> >> >
>>> >> > I've seen the following crash sporadically with commit 46c1e79fee:
>>> >> >
>>> >> > Have not seen that with 3882a734c19b, though I saw the PCID warnings on
>>> >> > that machine.
>>> >> >
>>> >> > I have no idea how to reproduce so bisecting is pretty much pointless. Any
>>> >> > idea what to do?
>>> >>
>>> >> Does tools/testing/selftests/x86/sigreturn_64 reproduce it?
>>> >
>>> > Will try tomorrow once I figured out how to compile that stuff. Invoking a
>>> > simple make in that directory fails.
>>>
>>> What's the error?  It works for me.
>>
>> gcc -m64 -o /home/tglx/work/kernel/linus/linux/tools/testing/selftests/x86/sysret_ss_attrs_64 -O2 -g -std=gnu99 -pthread -Wall  sysret_ss_attrs.c thunks.S -lrt -ldl
>> /usr/bin/ld: /tmp/cco4vSkU.o: relocation R_X86_64_32S against `.text' can not be used when making a shared object; recompile with -fPIC
>> /usr/bin/ld: final link failed: Nonrepresentable section on output
>> collect2: error: ld returned 1 exit status
>
> Had the same issue in copied thunks.S helper to CRIU.
> As I wanted to compile CRIU as pie also (which is the default now
> in some distributions), I fixed it up by using %rip-relative addressing
> and pushing segment descriptor to stack before long-jumping.
> Not sure if that's the issue for the selftest as it can be just always
> compiled as pic.
>
> Anyway, here is the commit if you want to look:
> https://github.com/0x7f454c46/criu/commit/cf36ea5d8408bf1e42c3bd21b9594369ea7123fb

Thanks!

The problem is that I need to have some text below 4 GB, and that's
currently guaranteed by a non-PIE build.  And I'm lazy :)

>
> --
>              Dmitry