Re: BUG: soft lockup on all kernels after 2.6.3x

[Eric Dumazet <eric.dumazet@gmail.com>]

On Sat, 2013-02-09 at 18:10 +0400, Alexey Vlasov wrote:
> Hello.
> 
> I used 2.6.2x kernel for a long time on my shared hosting and I didn't
> have any problems. Kernels worked well and server uptime was about 2-3
> years.
> 
> But investigating some strange hangings of my clients' sites I came to
> this:
> http://bugs.mysql.com/bug.php?id=50399
> from this bug it is clear that on kernels younger than 2.6.32 (
> unfortunately I can't remember if it is true with 2.6.30-31) happens
> mysql client hanging.
> 
> It is not clear whether it is a bug of kernel or libc or mysql-client, I
> didn't manage to find it out. I decided to do simpler (as it seemed to
> me at that moment) to start using 2.6.3x kernels. And that caused
> greater problems. By trying to use new kernels on my working servers
> with peak load I got an uptime from an hour to 1-3 months.
> 
> I even got some statistics for how long can every kernel from version
> 2.6.32 work in peak load situations. It sounds funny but my clients are
> not happy with all these reboots.
> 
> From all the variety of servers from 2.6.32 to 3.7.4 I can say that
> 2.6.35 is the most stabil, I got about 30 servers on it. But they hang
> usually once in 1-3 months.
> 
> Returning to the problem of kernels >= 2.6.32, as I have noticed they
> hang totally alike, giving the console:
> 
> ...
> Feb  8 10:27:45 10.2.0.7 [470393.417168] BUG: soft lockup - CPU#2 stuck for 61s! [vsftpd:29013]
> ...
> [see the attachment]
> 
> it doesn't happen on an empty server, only on loaded ones. Unfortunately
> I don't know how to provoke such hanging artificially.
> 
> I' ve given a trace attached. In fact I don't know what to do with all
> these bugs, I can't use 2.6.2x because of MySQL hanging and = >2.6.3
> start hanging themselves.
> 

Did you compile the kernel yourself, or is it a standard kernel (distro
provided) ?

Your traces dont contain symbols, its quite hard to guess the issue.