Hi LKML folks, I have a curious problem with a Tyan S8228-based, two-node system (data sheet: http://www.tyan.com/datasheets/YR190-B8228_DataSheet.pdf or http://www.tyan.com/product_SKU_spec.aspx?ProductType=MB&pid=678&SKU=600000191), and I hope that this list can help me determine the root of the problem, if not work around or even fix it... We have two Opteron 4272 with 32GB DDR3 ECC UDIMM in each of those boards, and their configs are 100% identical - same BIOS release, same CMOS sletettings, same BMC firmware release - the only thing that differs from one machine to the other is the IP address of the IPMI mgmt interface. Now, when we first got the system, I put it through our usual testing routine, which includes a load test over the weekend, running prime95 (http://mersenne.org/freesoft/) in torture test mode. That's usually enough to rule out any kind of CPU, memory or thermal problems. When torture-testing the new setup unter Debian Squeeze (with the 3.2.0 kernel from bpo, backported from Wheezy) we quickly found out that one node kept failing (that is, became unresponsive to any kind of user input) after a few minutes into the test, while the other node was happy to run flawlessly for several days, with prime95 not reporting any errors. We gradually swapped all components from one node to the other to identify which part was at fault - in the end, when we had swapped all components except for the motherboard and chassis (so even PSUs and the harddisks with their OS installations on them, etc.), the initially faulty node still kept failing, which told us that, probably, the board had some kind of defect. No problem, we got a replacent from the manufacturer three days ago, and went on to put that new machine through the same testing routine. Things is - it keeps failing in the exact same way as before, despite both chassis and mainboard having been replaced by new hardware. The problem is 100% reproducible with Squeeze's default kernel (2.6.32) and a recent-ish test-build of 3.8.0 that I did. I also tried if selecting a different clocksource for the system would make any kind of difference, but no matter if I use tsc, hpet or acpi_pm (there's no more alternatives for the kernel image I tested that with), the failure does happen after at most 20 minutes into the test. Setting the "performance" instead of "ondemand" cpufreq governor doesn't help either. System temperatures are fine. When failing, the machine doesn't outright die, but reports CPU soft-lockups on the console - networking goes down, the USB ports die, and there's no way to interact with the machine any more apart from a hard reboot. I've logged the console output using the serial console support in the kernel (that does work even after we trigger the error, though a shell on the serial console won't accept any more input once that happens), and attached it for your viewing pleasure. Maybe you have an idea what we can try to determine what's the all-important difference between the silbling systems, or what we could do to get the other, second node to work? If you need me to provide any other info than what's already in this mail, please let me know. Please keep me CC'd, as I'm a subscriber of this list at present. Thanks a bunch for reading, have a nice day :) -- Mit freundlichen Grüßen Johannes Truschnigg Senior System Administrator -- mailto:johannes.truschnigg@geizhals.at (in dringenden Fällen bitte an info@geizhals.at) Geizhals(R) - Preisvergleich Internet Services AG Obere Donaustrasse 63/2 A-1020 Wien Tel: +43 1 5811609/87 Fax: +43 1 5811609/55 http://geizhals.at => Preisvergleich für Österreich http://geizhals.de => Preisvergleich für Deutschland http://geizhals.eu => Preisvergleich EU-weit Handelsgericht Wien | FN 197241K | Firmensitz Wien