Re: [Ext2-devel] Re: fsck out of memory

Re: [Ext2-devel] Re: fsck out of memory

[Peter Chubb]

>>>>> "Stephen" == Stephen C Tweedie <sct@redhat.com> writes:

Stephen> Hi, On Tue, 2003-02-11 at 13:11, Stephan van Hienen wrote:

Stephen> I've no idea.  Ben has some lb patches up at

Stephen>   http://people.redhat.com/bcrl/lb/

Stephen> but there's nothing broken out against the latest lbd diffs.


Ben's patches are against a very old version of the kernel (2.4.6-pre8)
and require linking against libgcc to get 64-bit division.


The main issue is 64-bit division.  In the limited time I had I
couldn't convince myself that I could rely on all divisors being less
than 2^31 in the raid4/5 code.  If you can convince yourself of that,
then t's a straightforward but tedious task to make raid1, raid4 and
raid5 LBD-safe.

--
Dr Peter Chubb				    peterc@gelato.unsw.edu.au
You are lost in a maze of BitKeeper repositories, all almost the same.

Re: [Ext2-devel] Re: fsck out of memory

[Randy.Dunlap]

>>>>>> "Stephen" == Stephen C Tweedie <sct@redhat.com> writes:
>
> Stephen> Hi, On Tue, 2003-02-11 at 13:11, Stephan van Hienen wrote:
>
> Stephen> I've no idea.  Ben has some lb patches up at
>
> Stephen>   http://people.redhat.com/bcrl/lb/
>
> Stephen> but there's nothing broken out against the latest lbd diffs.
>
>
> Ben's patches are against a very old version of the kernel (2.4.6-pre8) and
> require linking against libgcc to get 64-bit division.
>
>
> The main issue is 64-bit division.  In the limited time I had I
> couldn't convince myself that I could rely on all divisors being less than
> 2^31 in the raid4/5 code.  If you can convince yourself of that, then t's a
> straightforward but tedious task to make raid1, raid4 and raid5 LBD-safe.
> --

Hi,
Is this in a speed-sensitive area?
If not, you could consider 64-bit software divide functions, such as found at
http://nemesis.sourceforge.net/browse/lib/static/intmath/ix86/intmath.c.html,
which would only be used on 32-bit CPUs, of course.

--
~Randy

fsck out of memory

[kernel]

i'm trying to run e2fsk after a system hang
after 1 hour running (70%) which had a memory usage for about 128M
i get these errors in the dmesg :

Out of Memory: Killed process 732 (fsck.ext2).
Out of Memory: Killed process 732 (fsck.ext2).
Out of Memory: Killed process 732 (fsck.ext2).
Out of Memory: Killed process 732 (fsck.ext2).

and this some pages

top gives me this info for fsck.ext2 :

  732 root       9   0  592M 465M  2068 S    64.7 92.6   6:31 fsck.ext2

Mem:   514360K av,  512176K used,    2184K free,       0K shrd,     564K
buff
Swap:  136544K av,  136544K used,       0K free                    3120K
cache

system has 512Megabyte memory (and 128mb swap (only fileserver, never
needed more swap)

I really wonder if there is something wrong with e2fsk ?
does it really need that much memory ?
(fsck on 2.2TB /dev/md0)

it was putting a lot of info on the screen (for some minutes) :
Duplicate/bad block in inode ... / ... ...  ... ... ...
(and scrolling in real fast speed)

e2fsprogs version 1.27 with kernel 2.4.20 (+lbd patch)
i tried upgrading e2fsutils to 1.32 (latest version), but this doesn't
help

any hints ? (maybe a way to disable the enormous output from
'Duplicate/bad block in inode ..' ?)

(also why does it tell, killed when it stays running (otherwise it can't
kill multiple times...))

Re: fsck out of memory

[Stephan van Hienen]

ok added some swap space (4 gigabyte)

usage was about 2.5GB

till aborted :

d0:  64450554/dev/md0:  64450555/dev/md0:  64450556/dev/md0:
64450557/dev/md0:  64450558/dev/md0:  64450559/dev/md0:  64450560/dev/md0:
64450561/dev/md0:  64450562/dev/md0:  64450563/dev/md0:  64450564/dev/md0:
64450565/dev/md0:  64450566/dev/md0:  64450567/dev/md0:  64450568/dev/md0:
64450569/dev/md0:  64450570/dev/md0:  64450571/dev/md0:  64450572/dev/md0:
64450573/dev/md0:  64450574/dev/md0:  64450575/dev/md0:  64450576/dev/md0:
64450577/dev/md0:  64450578/dev/md0:  64450579/dev/md0:  64450580/dev/md0:
64450581/dev/md0:  64450582/dev/md0:  64450583/dev/md0:  64450584/dev/md0:
64450585/dev/md0:  64450586/dev/md0:  64450587/dev/md0:  64450588/dev/md0:
64450589/dev/md0:  64450590e2fsck: Can't allocate block element

e2fsck: aborted
/dev/md0: 153834/76922880 files (9.3% non-contiguous), 181680730/615381536
blocks

any hints ?
(i really would like to get back a clean fs (with ext3 journal))

Re: fsck out of memory

[Andreas Dilger]

On Feb 07, 2003  18:07 +0100, Stephan van Hienen wrote:
> ok added some swap space (4 gigabyte)
> 
> usage was about 2.5GB
> 
> till aborted :
> 
> d0:  64450554/dev/md0:  64450555/dev/md0:  64450556/dev/md0:
> 64450557/dev/md0:  64450558/dev/md0:  64450559/dev/md0:  64450560/dev/md0:
> 64450561/dev/md0:  64450562/dev/md0:  64450563/dev/md0:  64450564/dev/md0:
> 64450565/dev/md0:  64450566/dev/md0:  64450567/dev/md0:  64450568/dev/md0:
> 64450569/dev/md0:  64450570/dev/md0:  64450571/dev/md0:  64450572/dev/md0:
> 64450573/dev/md0:  64450574/dev/md0:  64450575/dev/md0:  64450576/dev/md0:
> 64450577/dev/md0:  64450578/dev/md0:  64450579/dev/md0:  64450580/dev/md0:
> 64450581/dev/md0:  64450582/dev/md0:  64450583/dev/md0:  64450584/dev/md0:
> 64450585/dev/md0:  64450586/dev/md0:  64450587/dev/md0:  64450588/dev/md0:
> 64450589/dev/md0:  64450590e2fsck: Can't allocate block element
> 
> e2fsck: aborted
> /dev/md0: 153834/76922880 files (9.3% non-contiguous), 181680730/615381536
> blocks
> 
> any hints ?
> (i really would like to get back a clean fs (with ext3 journal))

Hmm, I don't think that will be easy...  By default e2fsck will load all
of the inode blocks into memory (pretty sure at least), and if you have
76922880 inodes that is 9.6GB of memory, which you can't allocate from a
single process on i386 no matter how much swap you have.  2.5GB sounds
about right for the maximum amount of memory one can allocate.

Ted, any suggestions?

Cheers, Andreas
--
Andreas Dilger
http://sourceforge.net/projects/ext2resize/
http://www-mddsp.enel.ucalgary.ca/People/adilger/

Re: fsck out of memory

[Stephan van Hienen]

On Fri, 7 Feb 2003, Andreas Dilger wrote:

> Hmm, I don't think that will be easy...  By default e2fsck will load all
> of the inode blocks into memory (pretty sure at least), and if you have
> 76922880 inodes that is 9.6GB of memory, which you can't allocate from a
> single process on i386 no matter how much swap you have.  2.5GB sounds
> about right for the maximum amount of memory one can allocate.

hmms the data is not critical yet (i was just testing this server)
i really wonder why the crash was there in the first place

thing i found in /var/log/messages :

Feb  7 04:18:15 storage kernel: EXT3-fs error (device md(9,0)):
ext3_new_block:
Allocating block in system zone - block = 536875638
Feb  7 04:18:15 storage kernel: EXT3-fs error (device md(9,0)):
ext3_new_block:
Allocating block in system zone - block = 536875639

doesn't look ok to me (and explains the crash?)

makes me wonder if this can have todo with the lbd (to allow 2TB+ devices)
patch ? or is this something else?
(if it can be related to the lbd patch, i will remove 2 hd's from the
array (but i don't prefer this option))

also for not getting this thing again (that i can't fsck my filesystem)
what are the setting i can use for creating a large filesystem on /dev/md0
? (what is the maximum workable inodes?)

i did this :

mke2fs -j -m 0  -b 4096 -i 4096 -R stride=16

Re: [Ext2-devel] Re: fsck out of memory

[Stephen C. Tweedie]

Hi,

On Sun, 2003-02-09 at 10:08, Stephan van Hienen wrote:

> Feb  7 04:18:15 storage kernel: EXT3-fs error (device md(9,0)):
> ext3_new_block:
> Allocating block in system zone - block = 536875638

That looks like it could be a block wrap, amongst other possible causes.

> makes me wonder if this can have todo with the lbd (to allow 2TB+ devices)
> patch ? or is this something else?

Well, that's the most likely candidate, because it's the least tested
component.  Are you using Ben LaHaise's LBD fixes for the md devices? 
Without those, md and lvm are not LBD-safe.

Cheers,
 Stephen

Re: [Ext2-devel] Re: fsck out of memory

[Stephan van Hienen]

On Mon, 10 Feb 2003, Stephen C. Tweedie wrote:

> On Sun, 2003-02-09 at 10:08, Stephan van Hienen wrote:
>
> > Feb  7 04:18:15 storage kernel: EXT3-fs error (device md(9,0)):
> > ext3_new_block:
> > Allocating block in system zone - block = 536875638
>
> That looks like it could be a block wrap, amongst other possible causes.
hmms and this means ?


>
> > makes me wonder if this can have todo with the lbd (to allow 2TB+ devices)
> > patch ? or is this something else?
>
> Well, that's the most likely candidate, because it's the least tested
> component.  Are you using Ben LaHaise's LBD fixes for the md devices?
> Without those, md and lvm are not LBD-safe.
where can i find this lbd fixes for md ?

Re: [Ext2-devel] Re: fsck out of memory

[Stephen C. Tweedie]

Hi,

On Tue, 2003-02-11 at 13:11, Stephan van Hienen wrote:
> > On Sun, 2003-02-09 at 10:08, Stephan van Hienen wrote:
> >
> > > Feb  7 04:18:15 storage kernel: EXT3-fs error (device md(9,0)):
> > > ext3_new_block:
> > > Allocating block in system zone - block = 536875638
> >
> > That looks like it could be a block wrap, amongst other possible causes.
> hmms and this means ?

One possible cause here is that some component of the system has wrapped
the block number round at 2TB, rather than correctly going beyond 2TB,
resulting in the wrong block being picked up as a bitmap block.

> > Well, that's the most likely candidate, because it's the least tested
> > component.  Are you using Ben LaHaise's LBD fixes for the md devices?
> > Without those, md and lvm are not LBD-safe.
> where can i find this lbd fixes for md ?

I've no idea.  Ben has some lb patches up at

  http://people.redhat.com/bcrl/lb/

but there's nothing broken out against the latest lbd diffs.

Cheers,
 Stephen