Re: Wrong free inodes count in kernels 2.0 and 2.2

Re: Wrong free inodes count in kernels 2.0 and 2.2

[Collectively Unconscious]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: TEXT/PLAIN; charset=X-UNKNOWN, Size: 1191 bytes --]


On Wed, 2 May 2001, [iso-8859-1] Samuli Kärkkäinen wrote:

> I get repeatably both in 2.0 and 2.2 serieses of kernels the following kind
> of errors:
> 
> 2.2 kernels (several, including 2.2.18):
>   EXT2-fs error (device ide1(22,6)): ext2_check_inodes_bitmap: Wrong free inodes count in group 768, stored = 984, counted = 717
>   EXT2-fs error (device ide1(22,6)): ext2_check_inodes_bitmap: Wrong free inodes count in group 769, stored = 1005, counted = 717
>   EXT2-fs error (device ide1(22,6)): ext2_check_inodes_bitmap: Wrong free inodes count in group 777, stored = 998, counted = 901
>   [ many similar lines deleted ]
> 
> and sometimes with 2.2 kernel, soon after the errors above:
>   EXT2-fs error (device ide1(22,1)): ext2_new_inode: Free inodes count corrupted in group 414 
>   last message repeated 795 times

We were getting this error on 4 of our arrays. We weren't able to track
down the original error, but the continuing problem turned out to be that
in "correcting" the errors fsck did not restore the drive to a stable
state and it would eventually corrupt itself again. 

If this is the same problem, force fsck until it no longer finds any
errors on the drive.

Jay

Wrong free inodes count in kernels 2.0 and 2.2

[Samuli Kärkkäinen]

I get repeatably both in 2.0 and 2.2 serieses of kernels the following kind
of errors:

2.0 kernels (at least 2.0.39):
  EXT2-fs error (device 03:04): ext2_check_inodes_bitmap: Wrong free inodes count in group 576, stored = 1956, counted = 1942
  EXT2-fs error (device 03:04): ext2_check_inodes_bitmap: Wrong free inodes count in group 577, stored = 1951, counted = 1942
  EXT2-fs error (device 03:04): ext2_check_inodes_bitmap: Wrong free inodes count in group 94, stored = 1921, counted = 1919
  [ many similar lines deleted ]

2.2 kernels (several, including 2.2.18):
  EXT2-fs error (device ide1(22,6)): ext2_check_inodes_bitmap: Wrong free inodes count in group 768, stored = 984, counted = 717
  EXT2-fs error (device ide1(22,6)): ext2_check_inodes_bitmap: Wrong free inodes count in group 769, stored = 1005, counted = 717
  EXT2-fs error (device ide1(22,6)): ext2_check_inodes_bitmap: Wrong free inodes count in group 777, stored = 998, counted = 901
  [ many similar lines deleted ]

and sometimes with 2.2 kernel, soon after the errors above:
  EXT2-fs error (device ide1(22,1)): ext2_new_inode: Free inodes count corrupted in group 414 
  last message repeated 795 times

The hosts running 2.0 and 2.2 kernels are completely separate. Also, all
hardware in the 2.2 box have been changed at least once, most components
more than that, and that hasn't affected the situation. There has been both
P2 and Athlon CPU's.

These errors start appearing when I have run my backup script about 10
times. They never show up before the script has been ran for about that many
times, and they always show up eventually. I have recreated the filesystem
several times, when trying different things to fix the problem, and the
problem always comes back. The backup script is essentially like this:

initial run:
  cp -ax / /backup/backup1
  
next night first differential:
  cp -al /backup/backup1 /backup/backup2
  rsync --archive --hard-links --whole-file --sparse --one-file-system --delete --force / /backup/backup2

following night second differential:
  cp -al /backup/backup2 /backup/backup3
  rsync --archive --hard-links --whole-file --sparse --one-file-system --delete --force / /backup/backup3

And so on. The result is that directories /backup/backupX each contain a
snapshot of the filesystem to be backed up, in this case the root
filesystem, with unchanged files hardlinked to the previous backups.

The hosts where this occurs have only one thing in common, namely that the
backups are made on IDE disks with ext2 filesystem. I also run this on a
third box, which has 2.0 kernel and SCSI disks, and the problem doesn't
occur there. Both boxes where the errors occur have otherwise no problems.

dumpe2fs says about the 2.0 box:
Filesystem volume name:   <none>
Last mounted on:          <not available>
Filesystem UUID:          c98e333c-0dbd-11d5-8297-00e02909a9a6
Filesystem magic number:  0xEF53
Filesystem revision #:    0 (original)
Filesystem features:      (none)
Filesystem state:         clean with errors
Errors behavior:          Continue
Filesystem OS type:       Linux
Inode count:              10733568
Block count:              42933712
Reserved block count:     2146685
Free blocks:              23202851
Free inodes:              10052554
First block:              1
Block size:               1024
Fragment size:            1024
Blocks per group:         8192
Fragments per group:      8192
Inodes per group:         2048
Inode blocks per group:   256
Last mount time:          Wed May  2 00:00:02 2001
Last write time:          Wed May  2 00:41:50 2001
Mount count:              18
Maximum mount count:      180
Last checked:             Wed Apr 18 07:53:30 2001
Check interval:           15552000 (6 months)
Next check after:         Mon Oct 15 07:53:30 2001
Reserved blocks uid:      0 (user root)
Reserved blocks gid:      0 (group root)

and about the 2.2 box:
Filesystem volume name:   <none>
Last mounted on:          <not available>
Filesystem UUID:          e74e4331-4057-40a2-a62c-a4d195918c3a
Filesystem magic number:  0xEF53
Filesystem revision #:    1 (dynamic)
Filesystem features:      filetype sparse_super
Filesystem state:         clean with errors
Errors behavior:          Continue
Filesystem OS type:       Linux
Inode count:              2009088
Block count:              16064968
Reserved block count:     803248
Free blocks:              2213966
Free inodes:              1094493
First block:              1
Block size:               1024
Fragment size:            1024
Blocks per group:         8192
Fragments per group:      8192
Inodes per group:         1024
Inode blocks per group:   128
Last mount time:          Wed May  2 00:00:00 2001
Last write time:          Wed May  2 00:42:32 2001
Mount count:              31
Maximum mount count:      20
Last checked:             Tue Apr  3 08:25:35 2001
Check interval:           15552000 (6 months)
Next check after:         Sun Sep 30 08:25:35 2001
Reserved blocks uid:      0 (user root)
Reserved blocks gid:      0 (group root)
First inode:              11
Inode size:		  128

More sample errors at http://www.woods.iki.fi/t/ext2-error/.

Anything else you need to know? I'd like to emphasis that this is highly
repeatable and occurs with a lot of different hardware.

-- 
  Samuli Kärkkäinen                   |\      _,,,---,,_
 skarkkai@woods.iki.fi----------ZZZzz /,`.-'`'    -.  ;-;;,_------
http://www.woods.iki.fi              |,4-  ) )-,_. ,\ (  `'-'
                                     '---''(_/--'  `-'\_)

Re: Wrong free inodes count in kernels 2.0 and 2.2

[Ville Herva]

On Wed, May 02, 2001 at 07:46:21PM +0300, [Samuli Kärkkäinen] claimed:
> I get repeatably both in 2.0 and 2.2 serieses of kernels the following kind
> of errors:
> 
> 2.2 kernels (several, including 2.2.18):
>   EXT2-fs error (device ide1(22,6)): ext2_check_inodes_bitmap: Wrong free inodes count in group 768, stored = 984, counted = 717
>   EXT2-fs error (device ide1(22,6)): ext2_check_inodes_bitmap: Wrong free inodes count in group 769, stored = 1005, counted = 717
>   EXT2-fs error (device ide1(22,6)): ext2_check_inodes_bitmap: Wrong free inodes count in group 777, stored = 998, counted = 901
>   [ many similar lines deleted ]
> 
> and sometimes with 2.2 kernel, soon after the errors above:
>   EXT2-fs error (device ide1(22,1)): ext2_new_inode: Free inodes count corrupted in group 414 
>   last message repeated 795 times

I get these messages as well on 2.2.18pre19:

EXT2-fs error (device md(9,0)): ext2_new_inode: Free inodes count
corrupted in group 501                                                                  
EXT2-fs error (device md(9,0)): ext2_new_inode: Free inodes count
corrupted in group 501                                                                  

I've applied ide, raid and ext2compr patches. While they sound like strong
suspects for they cause of the problem (the fs in question is on ide,
softraid 1 and is ext2compr'ed), I understand Samuli sees these messages on
a stock kernel (ide and scsi), which makes me suspect this is a core kernel
issue.

I've been running fairly heavy rsync backups onto the fs in question for
more than a year, and only now I have begun to get these errors.

> problem always comes back. The backup script is essentially like this:
> 
> following night second differential:
>   cp -al /backup/backup2 /backup/backup3
>   rsync --archive --hard-links --whole-file --sparse --one-file-system --delete --force / /backup/backup3

My backup script is a tad different, but the principle is the same: do an
initial snapshot, then cp -al it as the next day's snapshot and rsync the new
snapshot up to date. This of course introduces huge number of dir entries
and lots of inodes with big hard link counts.

BTW: What happens if the hard link count for an inode overflows? 


-- v --

v@iki.fi

Re: Wrong free inodes count in kernels 2.0 and 2.2

[Ville Herva]

On Thu, Aug 09, 2001 at 01:06:01AM +0300, you [Ville Herva] claimed:
> On Wed, May 02, 2001 at 07:46:21PM +0300, [Samuli Kärkkäinen] claimed:
> > I get repeatably both in 2.0 and 2.2 serieses of kernels the following kind
> > of errors:
> > 
> > 2.2 kernels (several, including 2.2.18):
> >   EXT2-fs error (device ide1(22,6)): ext2_check_inodes_bitmap: Wrong free inodes count in group 768, stored = 984, counted = 717
> >   EXT2-fs error (device ide1(22,6)): ext2_check_inodes_bitmap: Wrong free inodes count in group 769, stored = 1005, counted = 717
> >   EXT2-fs error (device ide1(22,6)): ext2_check_inodes_bitmap: Wrong free inodes count in group 777, stored = 998, counted = 901
> >   [ many similar lines deleted ]
> > 
> > and sometimes with 2.2 kernel, soon after the errors above:
> >   EXT2-fs error (device ide1(22,1)): ext2_new_inode: Free inodes count corrupted in group 414 
> >   last message repeated 795 times
> 
> I get these messages as well on 2.2.18pre19:
> 
> EXT2-fs error (device md(9,0)): ext2_new_inode: Free inodes count
> corrupted in group 501                                                                  
> EXT2-fs error (device md(9,0)): ext2_new_inode: Free inodes count
> corrupted in group 501                                                                  

I was just wondering, could this be the same bug Daniel Phillips described
in thread "[PATCH] Re: 2.4.0-test11 ext2 fs corruption":

http://groups.google.com/groups?q=2.4.0-test11+ext2+fs+corruption+group:mlist.linux.kernel&hl=en&safe=off&scoring=r&rnum=1&selm=linux.kernel.news2mail-3A259959.89EAD4DE%40innominate.de
http://groups.google.com/groups?hl=en&safe=off&th=cbe5e86866187b4,4&rnum=1&selm=linux.kernel.Pine.GSO.4.21.0011301652130.21891-100000%40weyl.math.psu.edu

and that was fixed in 2.4.0-test time?

A google search shows a number of people are seeing this error so it would be
nice to get the fix for 2.2 (and even 2.0), too...

http://www.google.com/search?q=ext2_new_inode%3A+Free+inodes+count+corrupted+in+group&sourceid=opera&num=0


-- v --

v@iki.fi