Re: Oops in VMA code

[Alexander Graf <agraf@suse.de>]

On 16.06.2011, at 07:59, Linus Torvalds wrote:

> On Wed, Jun 15, 2011 at 10:32 PM, Alexander Graf <agraf@suse.de> wrote:
>> 
>> 0xc000000000190580 <find_vma_prev+44>:  ld      r9,16(r9)
>> 0xc000000000190584 <find_vma_prev+48>:  mr      r26,r11
>> 0xc000000000190588 <find_vma_prev+52>:  cmpdi   cr7,r9,0
>> 0xc00000000019058c <find_vma_prev+56>:  mr      r11,r26
>> 0xc000000000190590 <find_vma_prev+60>:  beq     cr7,0xc0000000001905c4 <find_vma_prev+112>
>> 0xc000000000190594 <find_vma_prev+64>:  addi    r26,r9,-56
>> 0xc000000000190598 <find_vma_prev+68>:  ld      r0,16(r26)
>> 0xc00000000019059c <find_vma_prev+72>:  cmpld   cr7,r31,r0
>> 0xc0000000001905a0 <find_vma_prev+76>:  blt     cr7,0xc000000000190580 <find_vma_prev+44>
> 
> That's the inner loop in find_vma_prev(), and yes, it was inlined into
> do_munmap.
> 
> And the fault happens in that "ld r0,16(r26)", and it looks like you
> have memory corruption.
> 
> r26 has the value 0xc00090026236bbb0, and that "90" byte in the middle
> there looks bogus. It's not a valid pointer any more, but if that "9"
> had been a zero, it would have been.

Please see my reply to Ben here.

> So it looks like the rbtree has become corrupt, and it _looks_ like
> it's just a couple of bits that are set in what otherwise looks like a
> reasonable pointer. It *could* be a two-bit error that wasn't
> corrected (I assume you have ECC or parity on your RAM or caches), so
> it's theoretically possible that it's hardware, but generally memory
> corruption is due to software bugs, so that's a pretty far-fetched
> thing.

I'm not running on ECC memory IIRC, but this really doesn't look like a memory bit flip. Maybe somewhere else which resulted in that code to overwrite memory here, but I tend to not want to blame hardware for failures. Usually these bugs are software made :)

> At a guess, there's not a lot more to be had from the oops. The
> corruption probably came from some totally unrelated code. Without
> more of a pattern, it's pretty much impossible to even guess.
> 
> It may be that somebody can see something I'm missing, but unless you
> can find an ECC error report in your logs and say "oh, that's it", I
> suspect that you're better off ignoring it, and hoping that it will
> happen again (and again) so that we'd get enough of a pattern to start
> making any educated guesses about what's going on.
> 
> That's why I often google oops reports - one report may not give much
> of a pattern, but if google finds lots of them that all look roughly
> similar, you end up possibly seeing what the common issue is.

Yup, so let's keep this documented for now. Actually, the more I think about it the more it looks like simple random memory corruption by someone else in the kernel - and that's basically impossible to track and will give completely different bugs next time around :(.

Either way, thanks a lot for looking at it!


Alex