* OOM kiler/load problems with RAID/LVM and AoE
@ 2006-03-14 1:02 Joshua Kugler
2006-03-14 1:16 ` Lee Revell
2006-03-16 22:23 ` Rik van Riel
0 siblings, 2 replies; 6+ messages in thread
From: Joshua Kugler @ 2006-03-14 1:02 UTC (permalink / raw)
To: linux-kernel; +Cc: sah
Right off: This e-mail is going to be long. I'm first going to explain how I
first discovered this bug, then explain how I set up a "smaller scale" test
case. And I apologize for the wandering nature of this message as I am
trying to debug things and find the actual root cause of my problem. I've
read through the entire LKML FAQ, so I hope all that follows is proper form.
Initial setup:
I am using two Coraid Etherdrive units each with 15 400GB drives. Each unit
is configured internally as a RAID5 device, which presents a 5.1TB drive to
the OS. I wanted to mirror these units via RAID1, but discovered the 2TB
RAID component limit.
So, to get around this, here is what I did:
Put each unit into it's own LVM volume group (vg-shelf0 and vg-shelf1). Next,
I split each VG up into 5 1TB logical volumes (p1 through p5). Then, I
mirrored each 1TB LV against its corresponding LV in the other VG (e.g.
vg-shelf0/p1 and vg-shelf1/p1). I then took those 5 RAID1 devices and
assembled them into a RAID-Linear device to obtain a 5.1TB drive. All that
worked fine.
However, when I try to run stress tests against the system (using spew or
bonnie++), after a while, the load will start increasing, and eventually the
system will just hang. I've done this to make sure I don't run out of memory
(because oom-killer was firing off):
echo 5 > /proc/sys/vm/dirty_ratio
echo 5 > /proc/sys/vm/dirty_background_ratio
echo 2 > /proc/sys/vm/overcommit_memory
echo 0 > /proc/sys/vm/overcommit_ratio
That didn't seem to have any positive effect.
No, to take out two variables. I created a similar setup with two internal
400GB SATA hard drives (same brand, incidentally, as are in the Etherdrives),
so we got rid of AoE and the network drivers.
Here's what I did to test this process on the internal drives:
Prep the drives to be added to a VG
pvcreate /dev/sdc
pvcreate /dev/sdd
First, put each shelf in its own volume group
vgcreate /dev/vg-test0 /dev/sdc
vgcreate /dev/vg-test1 /dev/sdd
vgchange -ay /dev/vg-test0
vgchange -ay /dev/vg-test1
We now split each VG into 4 100GB chunks
Total PE = 95387 or 23846/23846/23846/23849
lvcreate -l 23846 -n p1 vg-test0
lvcreate -l 23846 -n p2 vg-test0
lvcreate -l 23846 -n p3 vg-test0
lvcreate -l 23849 -n p4 vg-test0
lvcreate -l 23846 -n p1 vg-test1
lvcreate -l 23846 -n p2 vg-test1
lvcreate -l 23846 -n p3 vg-test1
lvcreate -l 23849 -n p4 vg-test1
Now, create 4 100GB RAID1 md devices:
mdadm -C /dev/md7 --auto -l raid1 -n 2 /dev/vg-test0/p1 /dev/vg-test1/p1
mdadm -C /dev/md8 --auto -l raid1 -n 2 /dev/vg-test0/p2 /dev/vg-test1/p2
mdadm -C /dev/md9 --auto -l raid1 -n 2 /dev/vg-test0/p3 /dev/vg-test1/p3
mdadm -C /dev/md10 --auto -l raid1 -n 2 /dev/vg-test0/p4 /dev/vg-test1/p4
Now, we create a 400GB Linear device that is spread across the five RAID1
devices.
mdadm -C /dev/md11 --auto -l linear -n 4 /dev/md7 /dev/md8 /dev/md9 /dev/md10
Now, running this command:
bonnie++ -d /xtesting/bonnie -s 50g -u joshua:joshua
is where I have problems. It does the "putc()" section fine, but when it
starts "Writing intelligently," load goes through the roof, and the system
eventually hangs. It exhibits this behavior on the Etherdrive setup above
but *not* in the internal hard drive setup immediately above.
I assume the LVM/RAID code is pretty well tested. Any hints as to where I
might start debugging? syslog does not report anything useful (even
on-screen syslog). The AoE driver could be at fault, but seeing as I copied
95GB of files yesterday, I'm wondering if it has something to do with the
network drivers. These are Broadcoms using the tg3 driver...I have heard of
problems with these, correct? I put in a Intel Pro/1000 Dual NIC network
card and connected to the Etherdrives using this interface (I'm using Cat6
crossover cables, one Etherdrive per port). I re-ran the tests and had the
same results.
RAID or LVM problem? AoE drivers? Network driver badness (for both of them)?
I shall attempt to include all useful info:
Distribution: Mandriva 2006
Kernel: Linux community.dist-ed.uaf.edu 2.6.12-14mdksmp #1 SMP Tue Dec 20
13:45:20 MST 2005 i686 AMD Opteron(tm) Processor 246 unknown GNU/Linux
AoE drivers: aoe module version 23
AoE units are running on SS20060302 firmware
Note: AMD Opterons running in 32bit mode
Motherboard: Tyan S2882 Dual Opteron
RAM: 8GB
CPU's:
processor : 0
vendor_id : AuthenticAMD
cpu family : 15
model : 5
model name : AMD Opteron(tm) Processor 246
stepping : 10
cpu MHz : 1992.068
cache size : 1024 KB
processor : 1
vendor_id : AuthenticAMD
cpu family : 15
model : 5
model name : AMD Opteron(tm) Processor 246
stepping : 10
cpu MHz : 1992.068
cache size : 1024 KB
GCC 4.0.1 was used to compile (I think), binutils-2.16.91.0.2-3mdk (again, I
think). Not 100% sure as this is an RPM installed kernel, so I don't know
what versions were used to compile it.
[root@community /xtesting/bonnie]# lspcidrake
unknown : Advanced Micro Devices|AMD-8111 PCI [BRIDGE_PCI]
amd76xrom : Advanced Micro Devices|AMD-8111 LPC [BRIDGE_ISA]
amd74xx : Advanced Micro Devices|AMD-8111 IDE [STORAGE_IDE]
i2c-amd8111 : Advanced Micro Devices|AMD-8111 SMBus 2.0 [SERIAL_SMBUS]
hw_random : Advanced Micro Devices|AMD-8111 ACPI [BRIDGE_OTHER]
unknown : Advanced Micro Devices|AMD-8131 PCI-X Bridge [BRIDGE_PCI]
unknown : Advanced Micro Devices|AMD-8131 PCI-X Bridge [BRIDGE_PCI]
unknown : Advanced Micro Devices|AMD-8131 PCI-X APIC [SYSTEM_PIC]
unknown : Advanced Micro Devices|AMD-8131 PCI-X APIC [SYSTEM_PIC]
unknown : Advanced Micro Devices|K8 [Athlon64/Opteron] HyperTransport
Technology Configuration [BRIDGE_HOST]
unknown : Advanced Micro Devices|K8 [Athlon64/Opteron] HyperTransport
Technology Configuration [BRIDGE_HOST]
unknown : Advanced Micro Devices|K8 [Athlon64/Opteron] Address Map
[BRIDGE_HOST]
unknown : Advanced Micro Devices|K8 [Athlon64/Opteron] Address Map
[BRIDGE_HOST]
unknown : Advanced Micro Devices|K8 [Athlon64/Opteron] DRAM Controller
[BRIDGE_HOST]
unknown : Advanced Micro Devices|K8 [Athlon64/Opteron] DRAM Controller
[BRIDGE_HOST]
amd64-agp : Advanced Micro Devices|K8 [Athlon64/Opteron] Miscellaneous
Control [BRIDGE_HOST]
amd64-agp : Advanced Micro Devices|K8 [Athlon64/Opteron] Miscellaneous
Control [BRIDGE_HOST]
usb-ohci : Advanced Micro Devices|AMD-8111 USB [SERIAL_USB]
usb-ohci : Advanced Micro Devices|AMD-8111 USB [SERIAL_USB]
sata_sil : Silicon Image|Sil3114A Serial ATA [STORAGE_OTHER]
Card:ATI Rage XL: ATI|Rage XL [DISPLAY_VGA]
tg3 : Broadcom Corp.|BCM5704 CIOB-E 1000BaseTX [NETWORK_ETHERNET]
tg3 : Broadcom Corp.|BCM5704 CIOB-E 1000BaseTX [NETWORK_ETHERNET]
e1000 : Intel Corp.|82544EI Gigabit Ethernet Controller
[NETWORK_ETHERNET]
hub : Linux 2.6.12-14mdksmp ohci_hcd|OHCI Host Controller [Hub|
Unused]
hub : Linux 2.6.12-14mdksmp ohci_hcd|OHCI Host Controller [Hub|
Unused]
j----- k-----
--
Joshua Kugler PGP Key: http://pgp.mit.edu/
CDE System Administrator ID 0xDB26D7CE
http://distance.uaf.edu/
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: OOM kiler/load problems with RAID/LVM and AoE
2006-03-14 1:02 OOM kiler/load problems with RAID/LVM and AoE Joshua Kugler
@ 2006-03-14 1:16 ` Lee Revell
2006-03-14 1:20 ` Neil Brown
2006-03-14 21:14 ` Joshua Kugler
2006-03-16 22:23 ` Rik van Riel
1 sibling, 2 replies; 6+ messages in thread
From: Lee Revell @ 2006-03-14 1:16 UTC (permalink / raw)
To: Joshua Kugler; +Cc: linux-kernel, sah
On Mon, 2006-03-13 at 16:02 -0900, Joshua Kugler wrote:
> Kernel: Linux community.dist-ed.uaf.edu 2.6.12-14mdksmp #1 SMP Tue Dec
> 20
You'll have to try the latest kernel, 2.6.15.x or the latest 2.6.16
release candidate.
Lee
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: OOM kiler/load problems with RAID/LVM and AoE
2006-03-14 1:16 ` Lee Revell
@ 2006-03-14 1:20 ` Neil Brown
2006-03-14 21:14 ` Joshua Kugler
1 sibling, 0 replies; 6+ messages in thread
From: Neil Brown @ 2006-03-14 1:20 UTC (permalink / raw)
To: Lee Revell; +Cc: Joshua Kugler, linux-kernel, sah
On Monday March 13, rlrevell@joe-job.com wrote:
> On Mon, 2006-03-13 at 16:02 -0900, Joshua Kugler wrote:
> > Kernel: Linux community.dist-ed.uaf.edu 2.6.12-14mdksmp #1 SMP Tue Dec
> > 20
>
> You'll have to try the latest kernel, 2.6.15.x or the latest 2.6.16
> release candidate.
>
... and you'll probably find the 2G limit on raid1s have been removed
(certainly in the 2.6.16, not sure about 2.6.15). You'll need mdadm
2.3.
NeilBrown
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: OOM kiler/load problems with RAID/LVM and AoE
2006-03-14 1:16 ` Lee Revell
2006-03-14 1:20 ` Neil Brown
@ 2006-03-14 21:14 ` Joshua Kugler
1 sibling, 0 replies; 6+ messages in thread
From: Joshua Kugler @ 2006-03-14 21:14 UTC (permalink / raw)
To: linux-kernel; +Cc: Lee Revell, sah
On Monday 13 March 2006 16:16, Lee Revell wrote:
> You'll have to try the latest kernel, 2.6.15.x or the latest 2.6.16
> release candidate.
OK, compiled and booted with 2.6.16-rc5
On Monday 13 March 2006 21:53, Andrew Morton wrote:
> It it happens again, please get it to the stage where it's close
> to going oom, then capture:
>
> - The contents of /proc/meminfo
> - the contents of /proc/slabinfo
> - the oom-killer message (if poss)
This time around it seems OOM kiler didn't fire off, but bonnie "died," that
is, went to 0% CPU, and one CPU when to 100% wa (the other one is 100% idle),
there is no I/O going on (the Etherdrive units are showing no activity).
BUT! There is still the issue of massive load. Right now top, which is still
running, shows load at 33.42, 29.33, 20.28.
Next stop: take out the LVM layer and make a direct RAID1 device since md now
supports components > 2TB. I'll report those results as well.
Here is a recent output of meminfo and slabinfo for what it's worth:
MemTotal: 8252100 kB
MemFree: 7197868 kB
Buffers: 0 kB
Cached: 974740 kB
SwapCached: 0 kB
Active: 39124 kB
Inactive: 963964 kB
HighTotal: 7405504 kB
HighFree: 6395892 kB
LowTotal: 846596 kB
LowFree: 801976 kB
SwapTotal: 4393736 kB
SwapFree: 4391136 kB
Dirty: 16 kB
Writeback: 0 kB
Mapped: 36688 kB
Slab: 35344 kB
CommitLimit: 4393736 kB
Committed_AS: 101672 kB
PageTables: 1588 kB
VmallocTotal: 118776 kB
VmallocUsed: 8280 kB
VmallocChunk: 110172 kB
slabinfo - version: 2.1
# name <active_objs> <num_objs> <objsize> <objperslab>
<pagesperslab> : tunables <limit> <batchcount> <sharedfactor> : slabdata
<active_slabs> <num_slabs> <sharedavail>
aoe_bufs 16 78 48 78 1 : tunables 120 60 8 :
slabdata 1 1 0
fib6_nodes 9 113 32 113 1 : tunables 120 60 8 :
slabdata 1 1 0
ip6_dst_cache 10 15 256 15 1 : tunables 120 60 8 :
slabdata 1 1 0
ndisc_cache 1 20 192 20 1 : tunables 120 60 8 :
slabdata 1 1 0
RAWv6 5 6 640 6 1 : tunables 54 27 8 :
slabdata 1 1 0
UDPv6 1 6 640 6 1 : tunables 54 27 8 :
slabdata 1 1 0
tw_sock_TCPv6 0 0 128 30 1 : tunables 120 60 8 :
slabdata 0 0 0
request_sock_TCPv6 0 0 128 30 1 : tunables 120 60 8 :
slabdata 0 0 0
TCPv6 11 12 1216 3 1 : tunables 24 12 8 :
slabdata 4 4 0
ip_conntrack_expect 0 0 84 46 1 : tunables 120 60 8 :
slabdata 0 0 0
ip_conntrack 10 40 192 20 1 : tunables 120 60 8 :
slabdata 2 2 0
ip_fib_alias 9 113 32 113 1 : tunables 120 60 8 :
slabdata 1 1 0
ip_fib_hash 9 113 32 113 1 : tunables 120 60 8 :
slabdata 1 1 0
dm_tio 4608 4669 16 203 1 : tunables 120 60 8 :
slabdata 23 23 0
dm_io 4608 4732 20 169 1 : tunables 120 60 8 :
slabdata 28 28 0
xfs_acl 0 0 304 13 1 : tunables 54 27 8 :
slabdata 0 0 0
xfs_chashlist 167 845 20 169 1 : tunables 120 60 8 :
slabdata 5 5 0
xfs_ili 93 252 140 28 1 : tunables 120 60 8 :
slabdata 9 9 0
xfs_ifork 0 0 56 67 1 : tunables 120 60 8 :
slabdata 0 0 0
xfs_efi_item 0 0 260 15 1 : tunables 54 27 8 :
slabdata 0 0 0
xfs_efd_item 0 0 260 15 1 : tunables 54 27 8 :
slabdata 0 0 0
xfs_buf_item 0 0 148 26 1 : tunables 120 60 8 :
slabdata 0 0 0
xfs_dabuf 0 0 16 203 1 : tunables 120 60 8 :
slabdata 0 0 0
xfs_da_state 0 0 336 11 1 : tunables 54 27 8 :
slabdata 0 0 0
xfs_trans 1 13 596 13 2 : tunables 54 27 8 :
slabdata 1 1 0
xfs_inode 403 1320 368 10 1 : tunables 54 27 8 :
slabdata 132 132 0
xfs_btree_cur 0 0 140 28 1 : tunables 120 60 8 :
slabdata 0 0 0
xfs_bmap_free_item 0 0 16 203 1 : tunables 120 60 8 :
slabdata 0 0 0
xfs_buf 62 108 220 18 1 : tunables 120 60 8 :
slabdata 6 6 0
xfs_ioend 32 44 88 44 1 : tunables 120 60 8 :
slabdata 1 1 0
xfs_vnode 403 1269 408 9 1 : tunables 54 27 8 :
slabdata 141 141 0
scsi_cmd_cache 4 10 384 10 1 : tunables 54 27 8 :
slabdata 1 1 0
sgpool-128 32 33 2560 3 2 : tunables 24 12 8 :
slabdata 11 11 0
sgpool-64 32 33 1280 3 1 : tunables 24 12 8 :
slabdata 11 11 0
sgpool-32 32 36 640 6 1 : tunables 54 27 8 :
slabdata 6 6 0
sgpool-16 32 36 320 12 1 : tunables 54 27 8 :
slabdata 3 3 0
sgpool-8 32 40 192 20 1 : tunables 120 60 8 :
slabdata 2 2 0
scsi_io_context 0 0 104 37 1 : tunables 120 60 8 :
slabdata 0 0 0
UNIX 21 60 384 10 1 : tunables 54 27 8 :
slabdata 6 6 0
ip_mrt_cache 0 0 128 30 1 : tunables 120 60 8 :
slabdata 0 0 0
tcp_bind_bucket 8 203 16 203 1 : tunables 120 60 8 :
slabdata 1 1 0
inet_peer_cache 0 0 64 59 1 : tunables 120 60 8 :
slabdata 0 0 0
secpath_cache 0 0 128 30 1 : tunables 120 60 8 :
slabdata 0 0 0
xfrm_dst_cache 0 0 320 12 1 : tunables 54 27 8 :
slabdata 0 0 0
ip_dst_cache 16 45 256 15 1 : tunables 120 60 8 :
slabdata 3 3 0
arp_cache 1 20 192 20 1 : tunables 120 60 8 :
slabdata 1 1 0
RAW 3 7 512 7 1 : tunables 54 27 8 :
slabdata 1 1 0
UDP 9 14 512 7 1 : tunables 54 27 8 :
slabdata 2 2 0
tw_sock_TCP 0 0 128 30 1 : tunables 120 60 8 :
slabdata 0 0 0
request_sock_TCP 0 0 64 59 1 : tunables 120 60 8 :
slabdata 0 0 0
TCP 7 14 1088 7 2 : tunables 24 12 8 :
slabdata 2 2 0
flow_cache 0 0 128 30 1 : tunables 120 60 8 :
slabdata 0 0 0
cfq_ioc_pool 0 0 48 78 1 : tunables 120 60 8 :
slabdata 0 0 0
cfq_pool 0 0 96 40 1 : tunables 120 60 8 :
slabdata 0 0 0
crq_pool 0 0 48 78 1 : tunables 120 60 8 :
slabdata 0 0 0
deadline_drq 0 0 52 72 1 : tunables 120 60 8 :
slabdata 0 0 0
as_arq 20 59 64 59 1 : tunables 120 60 8 :
slabdata 1 1 0
mqueue_inode_cache 1 7 576 7 1 : tunables 54 27 8 :
slabdata 1 1 0
ext2_inode_cache 0 0 512 8 1 : tunables 54 27 8 :
slabdata 0 0 0
ext2_xattr 0 0 48 78 1 : tunables 120 60 8 :
slabdata 0 0 0
dnotify_cache 0 0 20 169 1 : tunables 120 60 8 :
slabdata 0 0 0
dquot 0 0 128 30 1 : tunables 120 60 8 :
slabdata 0 0 0
eventpoll_pwq 0 0 36 101 1 : tunables 120 60 8 :
slabdata 0 0 0
eventpoll_epi 0 0 128 30 1 : tunables 120 60 8 :
slabdata 0 0 0
inotify_event_cache 0 0 28 127 1 : tunables 120 60 8 :
slabdata 0 0 0
inotify_watch_cache 1 101 36 101 1 : tunables 120 60 8 :
slabdata 1 1 0
kioctx 0 0 192 20 1 : tunables 120 60 8 :
slabdata 0 0 0
kiocb 0 0 128 30 1 : tunables 120 60 8 :
slabdata 0 0 0
fasync_cache 0 0 16 203 1 : tunables 120 60 8 :
slabdata 0 0 0
shmem_inode_cache 4174 4176 472 8 1 : tunables 54 27 8 :
slabdata 522 522 0
posix_timers_cache 0 0 100 39 1 : tunables 120 60 8 :
slabdata 0 0 0
uid_cache 4 59 64 59 1 : tunables 120 60 8 :
slabdata 1 1 0
blkdev_ioc 33 127 28 127 1 : tunables 120 60 8 :
slabdata 1 1 0
blkdev_queue 54 56 940 4 1 : tunables 54 27 8 :
slabdata 14 14 0
blkdev_requests 20 22 176 22 1 : tunables 120 60 8 :
slabdata 1 1 0
biovec-(256) 260 260 3072 2 2 : tunables 24 12 8 :
slabdata 130 130 0
biovec-128 264 265 1536 5 2 : tunables 24 12 8 :
slabdata 53 53 0
biovec-64 272 290 768 5 1 : tunables 54 27 8 :
slabdata 58 58 0
biovec-16 272 300 192 20 1 : tunables 120 60 8 :
slabdata 15 15 0
biovec-4 272 295 64 59 1 : tunables 120 60 8 :
slabdata 5 5 0
biovec-1 272 406 16 203 1 : tunables 120 60 8 :
slabdata 2 2 0
bio 272 300 128 30 1 : tunables 120 60 8 :
slabdata 10 10 0
sock_inode_cache 73 90 448 9 1 : tunables 54 27 8 :
slabdata 10 10 0
skbuff_fclone_cache 43 60 384 10 1 : tunables 54 27 8 :
slabdata 6 6 13
skbuff_head_cache 1254 1320 192 20 1 : tunables 120 60 8 :
slabdata 66 66 0
file_lock_cache 18 42 92 42 1 : tunables 120 60 8 :
slabdata 1 1 0
proc_inode_cache 523 550 392 10 1 : tunables 54 27 8 :
slabdata 55 55 0
sigqueue 75 135 144 27 1 : tunables 120 60 8 :
slabdata 5 5 0
radix_tree_node 4079 4256 276 14 1 : tunables 54 27 8 :
slabdata 304 304 0
bdev_cache 36 42 512 7 1 : tunables 54 27 8 :
slabdata 6 6 0
sysfs_dir_cache 4822 4876 40 92 1 : tunables 120 60 8 :
slabdata 53 53 0
mnt_cache 25 30 128 30 1 : tunables 120 60 8 :
slabdata 1 1 0
inode_cache 1159 1280 376 10 1 : tunables 54 27 8 :
slabdata 128 128 0
dentry_cache 6289 10208 132 29 1 : tunables 120 60 8 :
slabdata 352 352 0
filp 1480 1480 192 20 1 : tunables 120 60 8 :
slabdata 74 74 300
names_cache 18 18 4096 1 1 : tunables 24 12 8 :
slabdata 18 18 0
key_jar 8 30 128 30 1 : tunables 120 60 8 :
slabdata 1 1 0
idr_layer_cache 115 116 136 29 1 : tunables 120 60 8 :
slabdata 4 4 0
buffer_head 240959 240984 52 72 1 : tunables 120 60 8 :
slabdata 3347 3347 0
mm_struct 144 144 448 9 1 : tunables 54 27 8 :
slabdata 16 16 0
vm_area_struct 3693 3738 92 42 1 : tunables 120 60 8 :
slabdata 89 89 0
fs_cache 236 236 64 59 1 : tunables 120 60 8 :
slabdata 4 4 0
files_cache 162 162 448 9 1 : tunables 54 27 8 :
slabdata 18 18 0
signal_cache 190 190 384 10 1 : tunables 54 27 8 :
slabdata 19 19 0
sighand_cache 177 177 1344 3 1 : tunables 24 12 8 :
slabdata 59 59 0
task_struct 180 180 1328 3 1 : tunables 24 12 8 :
slabdata 60 60 0
anon_vma 936 1016 12 254 1 : tunables 120 60 8 :
slabdata 4 4 0
pgd 226 226 32 113 1 : tunables 120 60 8 :
slabdata 2 2 0
pmd 279 279 4096 1 1 : tunables 24 12 8 :
slabdata 279 279 0
size-131072(DMA) 0 0 131072 1 32 : tunables 8 4 0 :
slabdata 0 0 0
size-131072 1 1 131072 1 32 : tunables 8 4 0 :
slabdata 1 1 0
size-65536(DMA) 0 0 65536 1 16 : tunables 8 4 0 :
slabdata 0 0 0
size-65536 4 4 65536 1 16 : tunables 8 4 0 :
slabdata 4 4 0
size-32768(DMA) 0 0 32768 1 8 : tunables 8 4 0 :
slabdata 0 0 0
size-32768 50 50 32768 1 8 : tunables 8 4 0 :
slabdata 50 50 0
size-16384(DMA) 0 0 16384 1 4 : tunables 8 4 0 :
slabdata 0 0 0
size-16384 18 18 16384 1 4 : tunables 8 4 0 :
slabdata 18 18 0
size-8192(DMA) 0 0 8192 1 2 : tunables 8 4 0 :
slabdata 0 0 0
size-8192 181 181 8192 1 2 : tunables 8 4 0 :
slabdata 181 181 0
size-4096(DMA) 0 0 4096 1 1 : tunables 24 12 8 :
slabdata 0 0 0
size-4096 879 879 4096 1 1 : tunables 24 12 8 :
slabdata 879 879 0
size-2048(DMA) 0 0 2048 2 1 : tunables 24 12 8 :
slabdata 0 0 0
size-2048 566 582 2048 2 1 : tunables 24 12 8 :
slabdata 291 291 0
size-1024(DMA) 0 0 1024 4 1 : tunables 54 27 8 :
slabdata 0 0 0
size-1024 324 324 1024 4 1 : tunables 54 27 8 :
slabdata 81 81 0
size-512(DMA) 0 0 512 8 1 : tunables 54 27 8 :
slabdata 0 0 0
size-512 448 448 512 8 1 : tunables 54 27 8 :
slabdata 56 56 0
size-256(DMA) 0 0 256 15 1 : tunables 120 60 8 :
slabdata 0 0 0
size-256 240 240 256 15 1 : tunables 120 60 8 :
slabdata 16 16 0
size-192(DMA) 0 0 192 20 1 : tunables 120 60 8 :
slabdata 0 0 0
size-192 756 760 192 20 1 : tunables 120 60 8 :
slabdata 38 38 0
size-128(DMA) 0 0 128 30 1 : tunables 120 60 8 :
slabdata 0 0 0
size-128 426 510 128 30 1 : tunables 120 60 8 :
slabdata 17 17 0
size-96(DMA) 0 0 128 30 1 : tunables 120 60 8 :
slabdata 0 0 0
size-96 869 930 128 30 1 : tunables 120 60 8 :
slabdata 31 31 0
size-64(DMA) 0 0 64 59 1 : tunables 120 60 8 :
slabdata 0 0 0
size-32(DMA) 0 0 32 113 1 : tunables 120 60 8 :
slabdata 0 0 0
size-64 2525 3127 64 59 1 : tunables 120 60 8 :
slabdata 53 53 0
size-32 3390 3390 32 113 1 : tunables 120 60 8 :
slabdata 30 30 0
kmem_cache 165 165 256 15 1 : tunables 120 60 8 :
slabdata 11 11 0
j----- k-----
--
Joshua Kugler PGP Key: http://pgp.mit.edu/
CDE System Administrator ID 0xDB26D7CE
http://distance.uaf.edu/
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: OOM kiler/load problems with RAID/LVM and AoE
2006-03-14 1:02 OOM kiler/load problems with RAID/LVM and AoE Joshua Kugler
2006-03-14 1:16 ` Lee Revell
@ 2006-03-16 22:23 ` Rik van Riel
2006-03-16 22:35 ` Joshua Kugler
1 sibling, 1 reply; 6+ messages in thread
From: Rik van Riel @ 2006-03-16 22:23 UTC (permalink / raw)
To: Joshua Kugler; +Cc: linux-kernel, sah
On Mon, 13 Mar 2006, Joshua Kugler wrote:
> RAID or LVM problem? AoE drivers? Network driver badness (for both of them)?
You could simply be hitting a fundamental problem that's present
on most operating systems. It happens roughly like this:
1) free memory gets low, so kswapd starts evicting pages
2) in order to write pages out over the network, the kernel
needs to allocate memory to compose network packets,
headers, etc...
3) if kswapd writes out a bunch of pages at once, or simply
if memory was low to begin with when we hit (1), there
may not be enough free memory left to receive the ACK
packets from the NAS box that acknowledge that the data
was received, nor the packets that indicate that the
data was written to disk and the kernel can complete
the IO
Locally attached disks do not have this problem because the
kernel keeps a number of reserved buffer heads around to get
us out of this deadlock problem.
Networking will need something similar. Because this is
slowly turning into an FAQ, I've written down the problem
and a proposed solution:
http://linux-mm.org/NetworkStorageDeadlock
--
All Rights Reversed
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: OOM kiler/load problems with RAID/LVM and AoE
2006-03-16 22:23 ` Rik van Riel
@ 2006-03-16 22:35 ` Joshua Kugler
0 siblings, 0 replies; 6+ messages in thread
From: Joshua Kugler @ 2006-03-16 22:35 UTC (permalink / raw)
To: linux-kernel; +Cc: Rik van Riel, sah
On Thursday 16 March 2006 13:23, Rik van Riel wrote:
> <Excellent answer snipped>
Thank you!! I wish I had gotten this answer weeks ago...it would have saved me
**hours** of troubleshooting, headaches, and debugging.
I've read over your proposed solution and it seems to make sense (but
IANAKH*). I hope to see it implemented soon.
j----- k-----
* I am not a kernel hacker
--
Joshua Kugler PGP Key: http://pgp.mit.edu/
CDE System Administrator ID 0xDB26D7CE
http://distance.uaf.edu/
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2006-03-16 22:35 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2006-03-14 1:02 OOM kiler/load problems with RAID/LVM and AoE Joshua Kugler
2006-03-14 1:16 ` Lee Revell
2006-03-14 1:20 ` Neil Brown
2006-03-14 21:14 ` Joshua Kugler
2006-03-16 22:23 ` Rik van Riel
2006-03-16 22:35 ` Joshua Kugler
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®