2.6.39, 3.0.0-rc7+: complete system freeze with rsync on big directory

2.6.39, 3.0.0-rc7+: complete system freeze with rsync on big directory

[Michael Brade]

Hi,

this is perfectly reproducible: with vanilla 2.6.39.x and 3.0.0 from 
yesterday's git my system completely freezes after a minute or so when I do

   rsync -cavSH -n /source/ /dest/

Watching htop, it freezes only after the caches are full but not immediately 
when the caches are full. No Alt-SysRq-magic is working anymore, nothing 
responds at all after the freeze. This is on tty1, no X server running.

Note the -n which means dry run. I also mounted both directories read-only, so 
it's not the write commands. I am using XFS on both directories, one is 
mounted with iscsi, the other is local and encrypted with loop-aes. The 
directory size is >20 GB, 4000-6000 files. I first noticed it when a 
cp -a /source /dest froze the system.

When I go back to 2.6.37.6 it works fine. I didn't try 2.6.38 yet.

I am running this on a Dell Vostro, amd64, 6 GB RAM, 8x Intel Core i7 CPU Q 
740 @ 1.73GHz.

Any ideas how to debug this despite the freeze? Do you need my kernel config? 
Do you need any other info? Please CC me.

thanks,
   Michael

Re: 2.6.39, 3.0.0-rc7+: complete system freeze with rsync on big directory

[Lars Täuber]

Hi Michael,

I'm just curious, because I have troubles with storage IO too.

Am Thu, 21 Jul 2011 11:19:15 +0200
Michael Brade <brade@informatik.uni-muenchen.de> schrieb:
> Hi,
> 
> this is perfectly reproducible: with vanilla 2.6.39.x and 3.0.0 from 
> yesterday's git my system completely freezes after a minute or so when I do
> 
>    rsync -cavSH -n /source/ /dest/
> 
> Watching htop, it freezes only after the caches are full but not
> immediately when the caches are full. No Alt-SysRq-magic is working
> anymore, nothing responds at all after the freeze. This is on tty1, no X
> server running.
> 
> Note the -n which means dry run. I also mounted both directories read-only,
> so it's not the write commands. I am using XFS on both directories, one is 
> mounted with iscsi, the other is local and encrypted with loop-aes. The 
> directory size is >20 GB, 4000-6000 files. I first noticed it when a 
> cp -a /source /dest froze the system.
> 
> When I go back to 2.6.37.6 it works fine. I didn't try 2.6.38 yet.
> 
> I am running this on a Dell Vostro, amd64, 6 GB RAM, 8x Intel Core i7 CPU Q 
> 740 @ 1.73GHz.
> 
> Any ideas how to debug this despite the freeze? Do you need my kernel
> config? Do you need any other info? Please CC me.

Could you try to capture the log via serial console?
(linux/Documentation/serial-console.txt)

What kind of disks/controller/driver do you use?



Lars

Re: 2.6.39, 3.0.0-rc7+: complete system freeze with rsync on big directory

[Michael Brade]

Hi Lars,

> > Any ideas how to debug this despite the freeze? Do you need my kernel
> > config? Do you need any other info? Please CC me.
> 
> Could you try to capture the log via serial console?
> (linux/Documentation/serial-console.txt)

Hm. I don't have a serial port on my laptop and I don't have a USB dongle, so 
I don't know how to do this. And really, when it freezes, everything freezes. 
Even the HDD light just stops right where it is - either on of off all the 
time. None of the buttons work anymore, the CPU fan keeps going the speed it 
had before, no network responses anymore. Only holding the power button for 4 
seconds switches it off.

But watching the normal console I get loads of

r8169 0000:13:00.0: eth0: link up
r8169 0000:13:00.0: eth0: link up
[...]

one message about every 1-2 seconds while rsync is running on 2.6.37.6. 

On 2.6.39 and 3.0.0-rc7 I get about 2-3 of those per second - until it 
freezes. The source partition is iSCSI so it does use the net.
 
> What kind of disks/controller/driver do you use?

Source HD is iSCSI over ethernet. I am using open-iscsi on the client.

Dest HD is on the laptop:
  block device: bus = scsi, bus_id = 0:0:0:0 driver = sd
  model = WDC WD5000BEKT-7
  Driver: "ahci", "sd"
  SATA controller

lspci:

00:1f.2 SATA controller: Intel Corporation 5 Series/3400 Series Chipset 6 port 
SATA AHCI Controller (rev 06)
13:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168B PCI 
Express Gigabit Ethernet controller (rev 03)

I guess my .config could also be helpful - if so, do you need all of it, part 
of it, and how should I post this? Attach it packed with 7z?

cheers,
  Michael

Re: 2.6.39, 3.0.0-rc7+: complete system freeze with rsync on big directory

[Michael Brade]

oh dear... this isn't true:

> > When I go back to 2.6.37.6 it works fine. I didn't try 2.6.38 yet.

I just froze the system three times in a row on 2.6.37.6 as well. However, it 
seems it takes longer, since I was able to copy 32 GB and 26 GB in two of 
those attempts until it froze.

-- Michael

[SOLVED: r8169 is the culprit] Re: 2.6.39, 3.0.0-rc7+: complete system freeze with rsync on big directory

[Michael Brade]

Hi,

finally I found the reason for my system freeze: the r8169 module. The freeze 
happens only on heavy network loads, i.e. > 70 MB/s. iSCSI is really fast, 
that's why I only saw the freezes with iSCSI.

> But watching the normal console I get loads of
> 
> r8169 0000:13:00.0: eth0: link up
> r8169 0000:13:00.0: eth0: link up

This does not happen with 2.6.32.28, and the system doesn't freeze either with 
this kernel. So the r8169 is fine on 2.6.32.x. 2.6.37 and above freeze the 
system. I haven't tried any kernel in between .32 and .37.

> [...]
> 
> lspci:
> 
> 13:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168B
> PCI Express Gigabit Ethernet controller (rev 03)

The solution with 2.6.37 and above: use the r8168 module from the realtek 
website. I have tested it with >30 GB at rates of 112 MB/s and experienced no 
freeze anymore.

I don't know if you are interested in fixing r8169 but if so and you need 
anything, let me know how I can help.

cheers,
  Michael

Re: [SOLVED: r8169 is the culprit] Re: 2.6.39, 3.0.0-rc7+: complete system freeze with rsync on big directory

[David Miller]

From: Michael Brade <brade@informatik.uni-muenchen.de>
Date: Sun, 14 Aug 2011 00:17:30 +0200

> I don't know if you are interested in fixing r8169 but if so and you need 
> anything, let me know how I can help.

The first thing you need to do is send your report to
netdev@vger.kernel.org and the listed r8169 maintainers
(nic_swsd@realtek.com and romieu@fr.zoreil.com), nobody on
linux-kernel will be able to fix this.