Wrong free space reported for XFS filesystem

Wrong free space reported for XFS filesystem

[Jesper Juhl]

For your information;

I've been running a bunch of benchmarks on a 250GB XFS filesystem.
After the benchmarks had run for a few hours and almost filled up the
fs, I removed all the files and did a "df -h" with interresting
results :

/dev/mapper/Data1-test
                     250G  -64Z  251G 101% /mnt/test

"df -k"  reported this :

/dev/mapper/Data1-test
                     262144000 -73786976294838202960 262147504 101% /mnt/test

I then did an umount and remount of the filesystem and then things
look more sane :

"df -h" :
/dev/mapper/Data1-test
                      250G  126M  250G   1% /mnt/test

"df -k" :
/dev/mapper/Data1-test
                     262144000    128280 262015720   1% /mnt/test

The filesystem is mounted like this :

/dev/mapper/Data1-test on /mnt/test type xfs
(rw,noatime,ihashsize=64433,logdev=/dev/Log1/test_log,usrquota)


-- 
Jesper Juhl <jesper.juhl@gmail.com>
Don't top-post  http://www.catb.org/~esr/jargon/html/T/top-post.html
Plain text mails only, please      http://www.expita.com/nomime.html

Re: Wrong free space reported for XFS filesystem

[Richard Mittendorfer]

Also sprach "Jesper Juhl" <jesper.juhl@gmail.com> (Wed, 6 Sep 2006
10:54:34 +0200):
> For your information;
> 
> I've been running a bunch of benchmarks on a 250GB XFS filesystem.
> After the benchmarks had run for a few hours and almost filled up the
> fs, I removed all the files and did a "df -h" with interresting
> results :
> 
> /dev/mapper/Data1-test
>                      250G  -64Z  251G 101% /mnt/test
> 
[...] 
> I then did an umount and remount of the filesystem and then things
> look more sane :
> 
> "df -h" :
> /dev/mapper/Data1-test
>                       250G  126M  250G   1% /mnt/test
[...]
> The filesystem is mounted like this :
> 
> /dev/mapper/Data1-test on /mnt/test type xfs
> (rw,noatime,ihashsize=64433,logdev=/dev/Log1/test_log,usrquota)

I once (2.6.12?) had to copy a quite large directory to an XFS
partition. It "should" had fit onto it (by what df said), but I ran into
"disk full". I think the reason was related to a large xfsbufd_centisecs
or xfssyncd_centisecs and indeed I could watch free space to grow and
shrink in regulaer intervals (watch df -k). I may well be wrong here (as
I'm sure no XFS-expert), but it looked like old data gets some kind of
"comressed" or "ordered" by the XFS-driver while newly written data took
more place. A "slow" copy did it, as well as a later try to an reiserfs
or ext.

sl ritch

Re: Wrong free space reported for XFS filesystem

[David Chinner]

On Wed, Sep 06, 2006 at 10:54:34AM +0200, Jesper Juhl wrote:
> For your information;
> 
> I've been running a bunch of benchmarks on a 250GB XFS filesystem.
> After the benchmarks had run for a few hours and almost filled up the
> fs, I removed all the files and did a "df -h" with interresting
> results :
> 
> /dev/mapper/Data1-test
>                     250G  -64Z  251G 101% /mnt/test
> 
> "df -k"  reported this :
> 
> /dev/mapper/Data1-test
>                     262144000 -73786976294838202960 262147504 101% /mnt/test
....
> The filesystem is mounted like this :
> 
> /dev/mapper/Data1-test on /mnt/test type xfs
> (rw,noatime,ihashsize=64433,logdev=/dev/Log1/test_log,usrquota)

So the in-core accounting has underflowed by a small amount but the
on disk accounting is correct.

We've had a few reports of this that I know of over the past couple of years,
but we've never managed to find a reproducable test case for it.

Can you describe what benchmark you were runnin, wht kernel you were
using and whether any of the tests hit  an ENOSPC condition?

Also, in future can you cc xfs@oss.sgi.com on XFS bug reports?

Cheers,

Dave.
-- 
Dave Chinner
Principal Engineer
SGI Australian Software Group

Re: Wrong free space reported for XFS filesystem

[Jesper Juhl]

On 07/09/06, David Chinner <dgc@sgi.com> wrote:
> On Wed, Sep 06, 2006 at 10:54:34AM +0200, Jesper Juhl wrote:
> > For your information;
> >
> > I've been running a bunch of benchmarks on a 250GB XFS filesystem.
> > After the benchmarks had run for a few hours and almost filled up the
> > fs, I removed all the files and did a "df -h" with interresting
> > results :
> >
> > /dev/mapper/Data1-test
> >                     250G  -64Z  251G 101% /mnt/test
> >
> > "df -k"  reported this :
> >
> > /dev/mapper/Data1-test
> >                     262144000 -73786976294838202960 262147504 101% /mnt/test
> ....
> > The filesystem is mounted like this :
> >
> > /dev/mapper/Data1-test on /mnt/test type xfs
> > (rw,noatime,ihashsize=64433,logdev=/dev/Log1/test_log,usrquota)
>
> So the in-core accounting has underflowed by a small amount but the
> on disk accounting is correct.
>
> We've had a few reports of this that I know of over the past couple of years,
> but we've never managed to find a reproducable test case for it.
>
> Can you describe what benchmark you were runnin, wht kernel you were
> using

The kernel is 2.6.18-rc6 SMP

> and whether any of the tests hit  an ENOSPC condition?
>
That I don't know.

The script I was running is this one :

###-----[ cut here ]-----
#!/bin/bash

DIR=/mnt/test
DIR1=random
DIR2=bigfiles
DIR3=smallfiles
BONNIEDIR=bonnie
RACERDIR=racer
WRIGGLERDIR=wriggler
IOZONEDIR=iozone

cd $DIR
rm -rf -- *

mkdir $DIR1
cd $DIR1
while true; do
        cd $DIR/$DIR1
        for i in `seq 1 100`; do
                touch -- "`head --bytes 15 /dev/urandom`";
        done >/dev/null 2>&1
        find . | while read i; do mv -f -- "$i" "`head --bytes 16
/dev/urandom`"; done >/dev/null 2>&1
        rm -f -- *
        sleep 1
done >/dev/null 2>&1 &

while true; do
        cd $DIR/$DIR1
        for i in `seq 1 100`; do
                touch -- "`head --bytes 25 /dev/urandom`";
        done >/dev/null 2>&1
        find . | while read i; do mv -f -- "$i" "`head --bytes 30
/dev/urandom`"; done >/dev/null 2>&1
        rm -f -- *
        sleep 2
done >/dev/null 2>&1 &

cd $DIR
mkdir $DIR/$BONNIEDIR
mkdir $DIR/$RACERDIR
mkdir $DIR/$WRIGGLERDIR
mkdir $DIR/$IOZONEDIR
mkdir $DIR/$DIR2
mkdir $DIR/$DIR3

while true; do
(cd /usr/local/racer ; sh ./racer-lustre.sh  >/dev/null 2>&1); done
>/dev/null 2>&1 &

while true; do
        (cd $DIR/$IOZONEDIR ; /usr/local/iozone/iozone -a >/dev/null 2>&1);
done >/dev/null 2>&1 &

while true; do
        (chown nobody $DIR/$BONNIEDIR ; cd $DIR/$BONNIEDIR ;
/usr/local/bonnie++/sbin/bonnie++ -u nobody >/dev/null 2>&1);
done >/dev/null 2>&1 &

while true; do
        (cd $DIR/$WRIGGLERDIR ; /usr/local/diskWriggler/diskWriggler
-n 500 -2K -o $DIR/$WRIGGLERDIR  >/dev/null 2>&1);
done >/dev/null 2>&1 &

while true; do
        (cd $DIR/$DIR2 ; dd if=/dev/zero of=$DIR/$DIR2/biggie bs=10M
count=15000  >/dev/null 2>&1; sync ; rm -f $DIR/$DIR2/biggie );
done >/dev/null 2>&1 &

while true; do
        (cd $DIR/$DIR3 ; export SIZE=$(($RANDOM*200000/32767)) ; dd
if=/dev/urandom of=$DIR/$DIR3/$SIZE bs=2048 count=$SIZE  >/dev/null
2>&1; rm -f -- $DIR/$D
IR3/* );
done >/dev/null 2>&1 &

while true; do
        for i in `seq 60 300`; do sleep $i; sync; done ;
done >/dev/null 2>&1 &

while true; do
        for i in `seq 7 21`; do sleep $i; find $DIR; done ;
done >/dev/null 2>&1 &

while true; do date ; echo "tests running"; sleep 300; done

###-----[ cut here ]-----


/usr/local/racer/ holds this test script :
ftp://ftp.lustre.org/pub/benchmarks/racer-lustre.tar.gz
The script is modified to use /mnt/test/racer for its tests.

/usr/local/iozone/ holds iozone version 3.263

/usr/local/bonnie++/ holds bonnie++ version 1.03

/usr/local/diskWriggler/ holds diskWriggler version 1.0.1


> Also, in future can you cc xfs@oss.sgi.com on XFS bug reports?
>
I keep forgetting, sorry. I'll try harder to remember :)


-- 
Jesper Juhl <jesper.juhl@gmail.com>
Don't top-post  http://www.catb.org/~esr/jargon/html/T/top-post.html
Plain text mails only, please      http://www.expita.com/nomime.html

Re: Wrong free space reported for XFS filesystem

[David Chinner]

On Thu, Sep 07, 2006 at 04:17:53PM +0200, Jesper Juhl wrote:
> On 07/09/06, David Chinner <dgc@sgi.com> wrote:
> >On Wed, Sep 06, 2006 at 10:54:34AM +0200, Jesper Juhl wrote:
> >> For your information;
> >>
> >> I've been running a bunch of benchmarks on a 250GB XFS filesystem.
> >> After the benchmarks had run for a few hours and almost filled up the
> >> fs, I removed all the files and did a "df -h" with interresting
> >> results :
.....
> >So the in-core accounting has underflowed by a small amount but the
> >on disk accounting is correct.
> >
> >We've had a few reports of this that I know of over the past
> >couple of years, but we've never managed to find a reproducable
> >test case for it.
> >Can you describe what benchmark you were runnin, wht kernel you were
> >using
> 
> The kernel is 2.6.18-rc6 SMP

Ok, so it's a current problem....

> >and whether any of the tests hit  an ENOSPC condition?
> >
> That I don't know.
> 
> The script I was running is this one :

<snip>

That doesn't really narrow down the scope at all. All that script
tells me is that problem is <waves hands> somewhere inside XFS....  :/
Can you try to isolate which of the loads is causing the problem?

That being said, this looks like a good stress load - I'll pass it
onto our QA folks...

Cheers,

Dave.
-- 
Dave Chinner
Principal Engineer
SGI Australian Software Group