mounting 4000 disks

mounting 4000 disks

[Badari Pulavarty]

Hi,

I have been playing with testing 4000 disk support. (using 32bit dev_t).

I created 4000 disks using scsi_debug and mounted them.
We consumed 84MB of low memory by just mouting 4000 filesystems.
Out of 84MB lowmem 48MB is in slabs. 33 MB is used by buffers.

Here are the slabs which used more than 1MB. 
32MB is used by size-8192 slab. (super block ?)

Anything else interesting ? I guess i can try nobh mount option.

- Badari 

ext2_inode_cache        before:136 after:4136 diff:4000 size:480 incr:1920000
ext3_inode_cache        before:2776 after:10808 diff:8032 size:504 incr:4048128
radix_tree_node         before:932 after:5026 diff:4094 size:272 incr:1113568
inode_cache     before:144258 after:148260 diff:4002 size:364 incr:1456728
dentry_cache    before:148573 after:164611 diff:16038 size:172 incr:2758536
buffer_head     before:9027 after:48970 diff:39943 size:64 incr:2556352
size-8192       before:9 after:4009 diff:4000 size:8192 incr:32768000
size-512        before:12375 after:16408 diff:4033 size:524 incr:2113292

Before:
--------
MemTotal:      3883236 kB
MemFree:       3452404 kB
Buffers:          7012 kB
Cached:         140668 kB
SwapCached:          0 kB
Active:          47564 kB
Inactive:       107040 kB
HighTotal:     3014616 kB
HighFree:      2839104 kB
LowTotal:       868620 kB
LowFree:        613300 kB
SwapTotal:     2040244 kB
SwapFree:      2040244 kB
Dirty:              92 kB
Writeback:           0 kB
Mapped:          14052 kB
Slab:           236780 kB
Committed_AS:    12132 kB
PageTables:        352 kB
ReverseMaps:      1679

After:
------
MemTotal:      3883236 kB
MemFree:       3361564 kB
Buffers:         40512 kB
Cached:         147468 kB
SwapCached:          0 kB
Active:          49564 kB
Inactive:       145560 kB
HighTotal:     3014616 kB
HighFree:      2832064 kB
LowTotal:       868620 kB
LowFree:        529500 kB
SwapTotal:     2040244 kB
SwapFree:      2040244 kB
Dirty:            6252 kB
Writeback:           0 kB
Mapped:          14060 kB
Slab:           286952 kB
Committed_AS:    12136 kB
PageTables:        352 kB
ReverseMaps:      1681

Re: mounting 4000 disks

[Andrew Morton]

Badari Pulavarty <pbadari@us.ibm.com> wrote:
>
> Hi,
> 
> I have been playing with testing 4000 disk support. (using 32bit dev_t).
> 
> I created 4000 disks using scsi_debug and mounted them.
> We consumed 84MB of low memory by just mouting 4000 filesystems.
> Out of 84MB lowmem 48MB is in slabs. 33 MB is used by buffers.

20 kbytes per disk+filesystem seems reasonable.

The "Buffers" will be the superblock (4k) and the group descriptor blocks
(256 bytes per gigabyte of disk).  These are pinned memory.

And the ext2 superblock got bigger in -mm due to the recent scalability
patches.

What is using the "size-8192" allocation there?  It could be the ext2 superblock
if you have NR_CPUS=32.

Re: mounting 4000 disks

[Badari Pulavarty]

On Thursday 03 April 2003 03:43 pm, Andrew Morton wrote:
> Badari Pulavarty <pbadari@us.ibm.com> wrote:
> > Hi,
> >
> > I have been playing with testing 4000 disk support. (using 32bit dev_t).
> >
> > I created 4000 disks using scsi_debug and mounted them.
> > We consumed 84MB of low memory by just mouting 4000 filesystems.
> > Out of 84MB lowmem 48MB is in slabs. 33 MB is used by buffers.
>
> 20 kbytes per disk+filesystem seems reasonable.
>
> The "Buffers" will be the superblock (4k) and the group descriptor blocks
> (256 bytes per gigabyte of disk).  These are pinned memory.
>
> And the ext2 superblock got bigger in -mm due to the recent scalability
> patches.
>
> What is using the "size-8192" allocation there?  It could be the ext2
> superblock if you have NR_CPUS=32.

I think so..

Apr  3 17:21:46 elm3b78 kernel:  [<c013f71b>] kmalloc+0x5b/0x1d0
Apr  3 17:21:46 elm3b78 kernel:  [<c0208d62>] vsnprintf+0x2a2/0x430
Apr  3 17:21:46 elm3b78 kernel:  [<c01aae3b>] ext2_fill_super+0x2b/0x8f0
Apr  3 17:21:46 elm3b78 kernel:  [<c0208f09>] snprintf+0x19/0x20
Apr  3 17:21:46 elm3b78 kernel:  [<c025ad88>] __bdevname+0xc8/0x110
Apr  3 17:21:46 elm3b78 kernel:  [<c015cd60>] get_sb_bdev+0xe0/0x130
Apr  3 17:21:46 elm3b78 kernel:  [<c0172579>] alloc_vfsmnt+0x79/0xb0
Apr  3 17:21:46 elm3b78 kernel:  [<c016330e>] real_lookup+0x5e/0xd0
Apr  3 17:21:46 elm3b78 kernel:  [<c01aba8e>] ext2_get_sb+0x1e/0x30
Apr  3 17:21:46 elm3b78 kernel:  [<c01aae10>] ext2_fill_super+0x0/0x8f0
Apr  3 17:21:46 elm3b78 kernel:  [<c015cf34>] do_kern_mount+0x44/0xc0
Apr  3 17:21:46 elm3b78 kernel:  [<c0173ba9>] do_add_mount+0x79/0x170
Apr  3 17:21:46 elm3b78 kernel:  [<c0109d8d>] error_code+0x2d/0x38
Apr  3 17:21:46 elm3b78 kernel:  [<c0173ebb>] do_mount+0x14b/0x170
Apr  3 17:21:46 elm3b78 kernel:  [<c0173d13>] copy_mount_options+0x73/0xd0
Apr  3 17:21:46 elm3b78 kernel:  [<c017455d>] sys_mount+0xed/0x180
Apr  3 17:21:46 elm3b78 kernel:  [<c0109303>] syscall_call+0x7/0xb