Re: Htree ate my hard drive, was: post-halloween 0.2

Re: Htree ate my hard drive, was: post-halloween 0.2

[Petr Vandrovec]

On 31 Oct 02 at 9:20, Duncan Sands wrote:
> > I wonder if there is still a bug in the e2fsck code for re-hashing
> > directories?  It shouldn't be possible to have e2fsck complete and
> > there still be an error in the filesystem (ok, sometimes it happens,
> > but in those cases it spews a lot of warnings about the filesystem
> > not being fixed yet and to run manually).
> 
> It is possible that the filesystem was fine when fsck completed, but
> was damaged afterwards, i.e. in the time between fsck completing
> and the reboot.

Just stupid idea. Two or three months ago I complained that if
my box crashes shortly after boot, following things happen:

(1) system for some reason reads /var/run directory to page cache
(2) fsck finds that /var/run/* entries points to invalid nodes, and
    removes them (through block device access)
(4) / is remounted read-write    
(5) because of page cache for block device and directory is not
    coherent (or what...), system still sees /var/run/* populated
(6) rm /var/run/* is run. FS is remounted read-only due to
    freeing inode already freeed...
(7) Reboot, run fsck again, reboot, fine...
    
Nobody answered it at that time, and it happened at least 5 times
again to me - until I modified initscripts to do unconditional
reboot if "fsck /" did ANY modifications to filesystem.

Maybe kernel still uses old directory indexes structure after
fsck created new one?
                                            Best regards,
                                                Petr Vandrovec
                                                vandrove@vc.cvut.cz
                                                

Re: [Ext2-devel] Re: Htree ate my hard drive, was: post-halloween 0.2

[chrisl]

On Thu, Oct 31, 2002 at 01:19:23PM +0200, Petr Vandrovec wrote:
> On 31 Oct 02 at 9:20, Duncan Sands wrote:
> > > I wonder if there is still a bug in the e2fsck code for re-hashing
> > > directories?  It shouldn't be possible to have e2fsck complete and
> > > there still be an error in the filesystem (ok, sometimes it happens,
> > > but in those cases it spews a lot of warnings about the filesystem
> > > not being fixed yet and to run manually).
> > 
> > It is possible that the filesystem was fine when fsck completed, but
> > was damaged afterwards, i.e. in the time between fsck completing
> > and the reboot.
> 
> Just stupid idea. Two or three months ago I complained that if
> my box crashes shortly after boot, following things happen:
> 
> (1) system for some reason reads /var/run directory to page cache
> (2) fsck finds that /var/run/* entries points to invalid nodes, and
>     removes them (through block device access)
> (4) / is remounted read-write    
> (5) because of page cache for block device and directory is not
>     coherent (or what...), system still sees /var/run/* populated
> (6) rm /var/run/* is run. FS is remounted read-only due to
>     freeing inode already freeed...
> (7) Reboot, run fsck again, reboot, fine...
>     
> Nobody answered it at that time, and it happened at least 5 times
> again to me - until I modified initscripts to do unconditional
> reboot if "fsck /" did ANY modifications to filesystem.
> 
> Maybe kernel still uses old directory indexes structure after
> fsck created new one?

File system needs to unmount and remount after e2fsck packed
directory index. Kernel need to trash all the dentry cache of
that file system. If you pack the "/". It'd better reboot.

Chris

Re: [Ext2-devel] Re: Htree ate my hard drive, was: post-halloween 0.2

[Theodore Ts'o]

On Thu, Oct 31, 2002 at 01:19:23PM +0200, Petr Vandrovec wrote:
>     
> Nobody answered it at that time, and it happened at least 5 times
> again to me - until I modified initscripts to do unconditional
> reboot if "fsck /" did ANY modifications to filesystem.
> 

In fact, e2fsck should return an exit code which indicates that the
systme should be rebooted if an fsck the root filesystem makes any
changes to the filesystem.  See the man page to fsck(8) for a
definition of fsck's exit codes, but if (exit_status & 2) is non-zero,
the init scripts **should** reboot.  

Unfortunately, not all distributions get this right.  However, your
analysis is right.  If fsck needs to make any modifications to the
root filesystem, which is mounted read-only, it is possible for the
corrupted filesystem elements to still be cached in memory, and then
written back out to disk when the filesystem is remounted read/write.  

This is one reason why I normally recommend that / be a small
filesystem of approximately 128 megs, with separate partitions for
/usr, and either using a separate partition for /var, or using a
symlink from /usr/var to /var.  (And doing something similar for /home
and and /opt, as necessary.)  It minimizes the chances that the root
filesystem will get corrupted, and makes running fsck on the root
filesystem take much less time (obviously, since the root filesystem
becomes quite small.)

						- Ted

Re: Htree ate my hard drive, was: post-halloween 0.2

[Martin Waitz]

[-- Attachment #1: Type: text/plain, Size: 1169 bytes --]

hi :)

after testing htree support, i ran into similar problems:

booted 2.5, tune2fs -O dir_index on all filesystems,
umounted home
fscked -D home, mounted home, all is well...
rebooted into 2.4.19 and everything is still fine

then i got a 'maximum mount count reached' on / while booting 2.4
on fscking, it optimized some directories.
afterwards i had some files missing all over the root fs

after removing dir_index all files were there again

lately my / got checked again, and fsck complained about some
hashed directory entries on a fs without dir_index...
i had to press return several times but did not run into problems...

-- 
CU,		  / Friedrich-Alexander University Erlangen, Germany
Martin Waitz	//  [Tali on IRCnet]  [tali.home.pages.de] _________
______________/// - - - - - - - - - - - - - - - - - - - - ///
dies ist eine manuell generierte mail, sie beinhaltet    //
tippfehler und ist auch ohne grossbuchstaben gueltig.   /
			    -
Wer bereit ist, grundlegende Freiheiten aufzugeben, um sich 
kurzfristige Sicherheit zu verschaffen, der hat weder Freiheit 
noch Sicherheit verdient.
			Benjamin Franklin  (1706 - 1790)

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: Htree ate my hard drive, was: post-halloween 0.2

[Mike Civil]

In article <200211032142.43432.baldrick@wanadoo.fr>,
Duncan Sands <baldrick@wanadoo.fr> wrote:
>
>On Friday 01 November 2002 00:05, Mike Civil wrote:
>> For info, I had this as well. Kernel 2.4.19 only. Using e2fsprogs 1.29.
>
>Do you mean that you did not run a 2.5 kernel, or that you tried several
>2.4 kernels and this was the only one that chewed on your hard drive?

No, at the moment I only run a 2.4.19 kernel.

Mike

Re: Htree ate my hard drive, was: post-halloween 0.2

[Duncan Sands]

On Friday 01 November 2002 00:05, Mike Civil wrote:
> For info, I had this as well. Kernel 2.4.19 only. Using e2fsprogs 1.29.

Do you mean that you did not run a 2.5 kernel, or that you tried several
2.4 kernels and this was the only one that chewed on your hard drive?

Ciao,

Duncan.

Re: Htree ate my hard drive, was: post-halloween 0.2

[Mike Civil]

In article <200210310727.52636.baldrick@wanadoo.fr> you write:
>
>> EXT3 Htree support.
>> ~~~~~~~~~~~~~~~~~~~
>> The ext3 filesystem has gained indexed directory support, which offers
>> considerable performance gains when used on filesystems with large
>> directories. In order to use the htree feature, you need at least version
>> 1.29 of e2fsprogs. Existing filesystems can be converted using the command
>> "tune2fs -O dir_index /dev/hdXXX" The latest e2fsprogs can be found at
>> http://prdownloads.sourceforge.net/e2fsprogs
>
>I ran this (tune2fs -O dir_index /dev/hdXXX).
>
>After a bit of switching back and forth between 2.4.19 and 2.5.44,
>fsck was run while booting 2.4.19 (the usual check because of >30
>mounts).  There was a message about optimizing directories.  Booting
>continued but (big surprise) X refused to run.  It turned out that some
>device files had vanished.  Very strange.  On rebooting, fsck found a
>gazillion bad inodes.  They all turned out to be from the 2.5.44 tree -
>poetic justice I suppose!  But this did not suffice.  Rebooting, I got
>"optimizing directories" again.  Next fsck showed up more dud inodes.

For info, I had this as well. Kernel 2.4.19 only. Using e2fsprogs 1.29.

I've not used tune2fs, just using -D switch to e2fsck. Using ext3 on
all filesystems.

No obvious complaints from e2fsck, although I wasn't really watching
it. No entries in /'s lost+found. No problems with other filesystems.

Only lost device files in /dev (about 120) with no discernible pattern.

Mike

dumpe2fs output:

Filesystem volume name:   root
Last mounted on:          /
Filesystem UUID:          6278d5f9-5583-4651-8d58-f12886a4e3a9
Filesystem magic number:  0xEF53
Filesystem revision #:    1 (dynamic)
Filesystem features:      has_journal dir_index filetype needs_recovery sparse_super
Filesystem state:         clean
Errors behavior:          Continue
Filesystem OS type:       Linux
Inode count:              122624
Block count:              244991
Reserved block count:     2449
Free blocks:              79421
Free inodes:              86789
First block:              0
Block size:               4096
Fragment size:            4096
Blocks per group:         32768
Fragments per group:      32768
Inodes per group:         15328
Inode blocks per group:   479
Last mount time:          Thu Oct 31 17:08:15 2002
Last write time:          Thu Oct 31 17:08:15 2002
Mount count:              4
Maximum mount count:      31
Last checked:             Wed Oct 30 18:57:46 2002
Check interval:           15552000 (6 months)
Next check after:         Mon Apr 28 19:57:46 2003
Reserved blocks uid:      0 (user root)
Reserved blocks gid:      0 (group root)
First inode:              11
Inode size:               128
Journal UUID:             <none>
Journal inode:            8
Journal device:           0x0000
First orphan inode:       0
Default directory hash:   tea
Directory Hash Seed:      c9f2f715-bc28-4302-a2e9-e53aa08576f7

Re: Htree ate my hard drive, was: post-halloween 0.2

[Duncan Sands]

On Thursday 31 October 2002 09:07, Andreas Dilger wrote:
> On Oct 31, 2002  07:27 +0100, Duncan Sands wrote:
> > > EXT3 Htree support.
> > > ~~~~~~~~~~~~~~~~~~~
> > > The ext3 filesystem has gained indexed directory support, which offers
> > > considerable performance gains when used on filesystems with large
> > > directories. In order to use the htree feature, you need at least
> > > version 1.29 of e2fsprogs. Existing filesystems can be converted using
> > > the command "tune2fs -O dir_index /dev/hdXXX" The latest e2fsprogs can
> > > be found at http://prdownloads.sourceforge.net/e2fsprogs
> >
> > I ran this (tune2fs -O dir_index /dev/hdXXX).
> >
> > After a bit of switching back and forth between 2.4.19 and 2.5.44,
> > fsck was run while booting 2.4.19 (the usual check because of >30
> > mounts).  There was a message about optimizing directories.  Booting
> > continued but (big surprise) X refused to run.  It turned out that some
> > device files had vanished.  Very strange.  On rebooting, fsck found a
> > gazillion bad inodes.  They all turned out to be from the 2.5.44 tree -
> > poetic justice I suppose!  But this did not suffice.  Rebooting, I got
> > "optimizing directories" again.  Next fsck showed up more dud inodes.
> > After a few cycles of this, I ran
> >
> > tune2fs -O ^dir_index /dev/hdXXX
> >
> > to remove htree support.  No problems since then.
> >
> > tune2fs 1.30-WIP (30-Sep-2002)
>
> I wonder if there is still a bug in the e2fsck code for re-hashing
> directories?  It shouldn't be possible to have e2fsck complete and
> there still be an error in the filesystem (ok, sometimes it happens,
> but in those cases it spews a lot of warnings about the filesystem
> not being fixed yet and to run manually).

It is possible that the filesystem was fine when fsck completed, but
was damaged afterwards, i.e. in the time between fsck completing
and the reboot.

Duncan.

> What else is strange (at least to me) is e2fsck "optimizing directories"
> on a reboot.  My understanding at least is that this would be done only
> when explicitly asked for, otherwise it might slow down booting a lot,
> and as you can see it adds to the possibility of corrupting the fs when
> e2fsck should only be fixing it.

Re: Htree ate my hard drive, was: post-halloween 0.2

[Andreas Dilger]

On Oct 31, 2002  07:27 +0100, Duncan Sands wrote:
> > EXT3 Htree support.
> > ~~~~~~~~~~~~~~~~~~~
> > The ext3 filesystem has gained indexed directory support, which offers
> > considerable performance gains when used on filesystems with large
> > directories. In order to use the htree feature, you need at least version
> > 1.29 of e2fsprogs. Existing filesystems can be converted using the command
> > "tune2fs -O dir_index /dev/hdXXX" The latest e2fsprogs can be found at
> > http://prdownloads.sourceforge.net/e2fsprogs
> 
> I ran this (tune2fs -O dir_index /dev/hdXXX).
> 
> After a bit of switching back and forth between 2.4.19 and 2.5.44,
> fsck was run while booting 2.4.19 (the usual check because of >30
> mounts).  There was a message about optimizing directories.  Booting
> continued but (big surprise) X refused to run.  It turned out that some
> device files had vanished.  Very strange.  On rebooting, fsck found a
> gazillion bad inodes.  They all turned out to be from the 2.5.44 tree -
> poetic justice I suppose!  But this did not suffice.  Rebooting, I got
> "optimizing directories" again.  Next fsck showed up more dud inodes.
> After a few cycles of this, I ran
> 
> tune2fs -O ^dir_index /dev/hdXXX
> 
> to remove htree support.  No problems since then.
> 
> tune2fs 1.30-WIP (30-Sep-2002)

I wonder if there is still a bug in the e2fsck code for re-hashing
directories?  It shouldn't be possible to have e2fsck complete and
there still be an error in the filesystem (ok, sometimes it happens,
but in those cases it spews a lot of warnings about the filesystem
not being fixed yet and to run manually).

What else is strange (at least to me) is e2fsck "optimizing directories"
on a reboot.  My understanding at least is that this would be done only
when explicitly asked for, otherwise it might slow down booting a lot,
and as you can see it adds to the possibility of corrupting the fs when
e2fsck should only be fixing it.

Cheers, Andreas
--
Andreas Dilger
http://www-mddsp.enel.ucalgary.ca/People/adilger/
http://sourceforge.net/projects/ext2resize/

Htree ate my hard drive, was: post-halloween 0.2

[Duncan Sands]

> EXT3 Htree support.
> ~~~~~~~~~~~~~~~~~~~
> The ext3 filesystem has gained indexed directory support, which offers
> considerable performance gains when used on filesystems with large
> directories. In order to use the htree feature, you need at least version
> 1.29 of e2fsprogs. Existing filesystems can be converted using the command
> "tune2fs -O dir_index /dev/hdXXX" The latest e2fsprogs can be found at
> http://prdownloads.sourceforge.net/e2fsprogs

I ran this (tune2fs -O dir_index /dev/hdXXX).

After a bit of switching back and forth between 2.4.19 and 2.5.44,
fsck was run while booting 2.4.19 (the usual check because of >30
mounts).  There was a message about optimizing directories.  Booting
continued but (big surprise) X refused to run.  It turned out that some
device files had vanished.  Very strange.  On rebooting, fsck found a
gazillion bad inodes.  They all turned out to be from the 2.5.44 tree -
poetic justice I suppose!  But this did not suffice.  Rebooting, I got
"optimizing directories" again.  Next fsck showed up more dud inodes.
After a few cycles of this, I ran

tune2fs -O ^dir_index /dev/hdXXX

to remove htree support.  No problems since then.

Duncan.

PS: UP, no preempt.

tune2fs 1.30-WIP (30-Sep-2002)
Filesystem volume name:   <none>
Last mounted on:          <not available>
Filesystem UUID:          ee433ceb-6b14-45b1-894c-2a8aad1e280f
Filesystem magic number:  0xEF53
Filesystem revision #:    1 (dynamic)
Filesystem features:      has_journal needs_recovery
Filesystem state:         clean
Errors behavior:          Unknown (continue)
Filesystem OS type:       Linux
Inode count:              290816
Block count:              2315368
Reserved block count:     115768
Free blocks:              871842
Free inodes:              36718
First block:              0
Block size:               4096
Fragment size:            4096
Blocks per group:         32768
Fragments per group:      32768
Inodes per group:         4096
Inode blocks per group:   128
Last mount time:          Thu Oct 31 06:37:46 2002
Last write time:          Thu Oct 31 06:37:46 2002
Mount count:              7
Maximum mount count:      30
Last checked:             Wed Oct 30 11:50:37 2002
Check interval:           0 (<none>)
Reserved blocks uid:      0 (user root)
Reserved blocks gid:      0 (group root)
First inode:              11
Inode size:               128
Journal UUID:             <none>
Journal inode:            493
Journal device:           0x0000
First orphan inode:       139500