2.6.35.x: acpi+no_hz+turboboost causing 3ware I/O controller resets

2.6.35.x: acpi+no_hz+turboboost causing 3ware I/O controller resets

[Justin Piszcz]

Hello,

There has been discussion going on here:
http://forums.storagereview.com/index.php/topic/28920-3ware-9650se-controller-resets-under-load-on-linux/page__st__30__gopid__264286&#entry264286

Member JGC posted:
======================================================================
I've been stuggling with these reset problems for a year now. Problems started to appear when we upgraded our servers running Debian Etch to Debian Lenny. When we did this upgrade, we changed our kernel from a custom compiled 2.6.24 kernel to 2.6.27. When 2.6.32 was declared as long time supported stable kernel, we switched to this kernel, but problems still appeared.
Recently I reconfigured the kernel (2.6.32.21) a bit:
- disable NOHZ
- disable speedstepping
- apply two patches from 2.6.35:
http://git.kernel.or...405aa31bcbb7091
http://git.kernel.or...e2d9fcf50aa04be
======================================================================
After those changes, I haven't seen reset problems anymore. We've seen this bug on several servers and contacted LSI and our vendor about this problem. The vendor doesn't know about the problems, and LSI blames the mainboard of some servers, while they blame the disks for other servers (yes, we have some servers with desktop drives).

I've been stresstesting this kernel with the "stress" utility on a 9690SA and 9650SE-based machine, both work fine with the modified kernel and haven't shown any resets ever since. I could even enable NCQ again without hanging up the servers completely.
======================================================================

I always have had NO_HZ enabled along with CPU Frequency Scaling so I could
utilize turbo boost.  However, with 2.6.35, as soon as there was any I/O on
the system, the system would "freeze," - it would respond to ping, but all
SSH windows would "lockup" until the controller resets (like below):

[  593.967176] 3w-9xxx: scsi0: WARNING: (0x06:0x0037): Character ioctl (0x108)
timed out, resetting card.
[  730.483812] 3w-9xxx: scsi0: WARNING: (0x06:0x0037): Character ioctl (0x108)
timed out, resetting card.

However, after disabling:

[*] Tickless System (Dynamic Ticks)
<*> Processor or CPU Frequency Scaling
[*] CPU Frequency scaling

Re-compiling, booting into 2.6.35.4 again (have not changed the kernel
version).

I have been running heavy I/O processes that can cause the problem pretty much
immediately when those CPU options are enabled-- and the problem appears to
be resolved, there have been no freezing events yet.

I'll let it run for a while longer but from what I can tell disabling
turboboost options (tickless/scaling) seems to solve the problem!

The next questions are:

1. what changed with CPU/frequency scaling from 2.6.34 -> 2.6.35?
2. when disabling the options above, it seems to stop the reset issue, why?

I've had a case (a couple cases open with 3ware/LSI) and they have my
configuration/etc and have not been able to re-produce the problem; however,
unless they have a CPU that is turbo-boost capable/matching the hardware,
I am not sure they will be able to reproduce the problem.

I'll keep running tests to see if I can re-produce the problem when CPU 
frequency scaling is turned off, but so far, the problem has not recurred.

Justin.

Re: 2.6.35.x: acpi+no_hz+turboboost causing 3ware I/O controller resets

[Justin Piszcz]

On Tue, 21 Sep 2010, Justin Piszcz wrote:

> Hello,
>
> There has been discussion going on here:
> http://forums.storagereview.com/index.php/topic/28920-3ware-9650se-controller-resets-under-load-on-linux/page__st__30__gopid__264286&#entry264286
>

Some additional details from the poster on storage review:

----------------------------------------------------------------------
It's not specific for turbo boost, and it's not specific for 2.6.34. We had the
freezes with 2.6.27 and 2.6.32, with both being the first kernels where we
enabled NOHZ and cpufreq. The older machines are running 2.6.24 without
frequency scaling, one even with the original firmware that was shipped with the
9690SA when it came to the market. (yes, we tried downgrading a spare machine
having issues to that firmware also, but no luck either).
Our servers are still core2 based, some are just Core2Duo on Intel 3210 chipset,
others are dual Xeon L5420 on Intel 5000P chipsets. All suffer the same problem:
resetting 3Ware controllers when cpufreq and nohz were enabled.

It's quite funny to read your issues here. When I contacted my vendor about
this, they told me that I was the only one having these issues, while they're
selling several hundreds of these controllers every month. LSI also gives me
that impression when I contact their tech support.
----------------------------------------------------------------------

Adding Dave to CC list for CPU FREQ driver:

CPU FREQUENCY DRIVERS
M:  Dave Jones
L:  cpufreq

Per: kernel/time/tick-sched.c
  *  Started by: Thomas Gleixner and Ingo Molnar

Maybe someone can shed some light on what is going on here in ref to the 
CPU FREQ/NO_HZ issue.  In the past I've never used CPU scaling until I got 
a CPU that could make use of turboboost.  Using NO_HZ will improve 
turboboost performance (basically makes it work correctly)..

For my machine, the problem is immediately apparent between:
2.6.34.1 -> 2.6.35.4

With the options enabled; however, (for my case) it only seems to affect
2.6.35.4 primarily.

With the options mentioned earlier (disabled) I can now run 2.6.35.4 and 
2.6.35.5 without any problems:

-# CONFIG_NO_HZ is not set
-# CONFIG_ACPI_PROCESSOR is not set
-# CONFIG_CPU_FREQ is not set

However, now I can no longer use turbo boost for my i7 CPU.

Opt 1: Enable parameters above with 2.6.34.x (there is no I/O locking, but 
over several months there may be an occasional reset)
Opt 2: Run any kernel with the parameters above disabled as shown and 
forget about turbo boost, but then my system would not lockup/have the 
controllers reset.
Opt 3: Try to find out what is triggering this bug so turboboost/freq 
scaling can be used with 3ware controllers?

Justin.