Disk quotas not staying in sync?

Disk quotas not staying in sync?

[Chris Meadors]

Maybe I'm missing something silly, but I can't seem to put my finger on
whats happening here.

I have a sever that holds the e-mail and webpages for our dial-up
customers.  Each user (for the most part) gets 20MB of space to hold their
files.

So people, for what ever reason, get so much e-mail they run up to their
quota.  They have trouble downloading 20MB of mail across a dial-up link,
and really don't want all the messages (picutues) anyway.  So one of the
techs here will just wipe out the messages.

We are using the Maildir format to store the mail, so each message is in a
seperate file.

The problem is sometimes after all the messages have been deleted they
still can't get any new mail.  Looking at their quota usage the machine
still thinks that they have their quota full, but `du` says otherwise.
Running `quotacheck` fixes things up.

I've tried to reproduce this with a test account.  Just creating a bunch
of files to run up to the quota and then deleting them.  But no matter
what I tried the system followed the test account's disk usage exactly.

This doesn't just happen when someone runs up to their quota.  I have also
seen their quota usage not correctly reflect what is on disk, but as the
usage goes up and down, both the actual disk usage and the what quota says
change by the same amount.

Any ideas?  This machine is pretty much standard.  ext2 file systems on
all partitions.  /home is the only one with user disk quotas.  Running
2.4.7 kernel (but I've seen it in all 2.4 kernels, I was running
2.4.0-prerelease on the machine when I first put it up, so I don't know
how 2.2 works).  To access the disks I'm using a sym53c8xx controller that
is talking to a hardware RAID controller that has the disks behind it.

Thanks,
Chris
-- 
Two penguins were walking on an iceberg.  The first penguin said to the
second, "you look like you are wearing a tuxedo."  The second penguin
said, "I might be..."                         --David Lynch, Twin Peaks

Re: Disk quotas not staying in sync?

[Andreas Dilger]

Chris writes:
> The problem is sometimes after all the messages have been deleted they
> still can't get any new mail.  Looking at their quota usage the machine
> still thinks that they have their quota full, but `du` says otherwise.
> 
> I've tried to reproduce this with a test account.  Just creating a bunch
> of files to run up to the quota and then deleting them.  But no matter
> what I tried the system followed the test account's disk usage exactly.

Maybe the problem is caused by root/admin deleting another users files?

> This doesn't just happen when someone runs up to their quota.  I have also
> seen their quota usage not correctly reflect what is on disk, but as the
> usage goes up and down, both the actual disk usage and the what quota says
> change by the same amount.

Do you run quotacheck at boot time?  It is possible with ext2 that the
on-disk quotas are not in sync with the actual files if there is a crash.
If you use ext3 then the quota updates and other filesystem updates are
kept atomic.

> Any ideas?  This machine is pretty much standard.  ext2 file systems on
> all partitions.  /home is the only one with user disk quotas.  Running
> 2.4.7 kernel (but I've seen it in all 2.4 kernels, I was running
> 2.4.0-prerelease on the machine when I first put it up, so I don't know
> how 2.2 works).

You should also try the -ac kernels.  I'm pretty sure that they have
some fixes to the quota code.  However, I _think_ the changes are not
compatible with the non-ac kernel quota on-disk data, so you will have
to re-build the quota files (a small price to pay if it actually works).

Cheers, Andreas
-- 
Andreas Dilger  \ "If a man ate a pound of pasta and a pound of antipasto,
                 \  would they cancel out, leaving him still hungry?"
http://www-mddsp.enel.ucalgary.ca/People/adilger/               -- Dogbert

Re: Disk quotas not staying in sync?

[Chris Meadors]

On Wed, 1 Aug 2001, Andreas Dilger wrote:

> Maybe the problem is caused by root/admin deleting another users files?

I thought of that first.  But the techs that were deleting mail were
actually `su`ing to the user and using pine to delete the messages.  So I
was wondering if pine was doing something strange.  But since we use
Maildirs when I was cleaning up an account I just `rm`ed their e-mails and
still ended up seeing the same problem.

> Do you run quotacheck at boot time?  It is possible with ext2 that the
> on-disk quotas are not in sync with the actual files if there is a crash.
> If you use ext3 then the quota updates and other filesystem updates are
> kept atomic.

I have run quotacheck once at boot to see if it would help.  It got the
stuff in sync for a little while, but soon I started to see the same
problem again.  quotacheck on a cold cache takes a LONG time to run.

So when is ext3 going in the kernel?  (ducks)  But really that is
something I will look at.

> > Any ideas?  This machine is pretty much standard.  ext2 file systems on
> > all partitions.  /home is the only one with user disk quotas.  Running
> > 2.4.7 kernel (but I've seen it in all 2.4 kernels, I was running
> > 2.4.0-prerelease on the machine when I first put it up, so I don't know
> > how 2.2 works).

One thing I forgot to mention about the machine, it is an SMP PIII box.  I
was really meaning to say it was SMP cause that might be something to look
at.

> You should also try the -ac kernels.  I'm pretty sure that they have
> some fixes to the quota code.  However, I _think_ the changes are not
> compatible with the non-ac kernel quota on-disk data, so you will have
> to re-build the quota files (a small price to pay if it actually works).

Yeah, when I was first putting this machine together I recall seeing there
was a new version of the quota utils and patches for the kernel to go with
them.  And yes, the on disk data is different, that is if those are the
patches Alan has in his kernel.

Do you know if the ext3 patches play well with the -ac kernels?  Might as
well try shooting myself in both feet while I'm at it.

> Cheers, Andreas

Thanks again,
Chris
-- 
Two penguins were walking on an iceberg.  The first penguin said to the
second, "you look like you are wearing a tuxedo."  The second penguin
said, "I might be..."                         --David Lynch, Twin Peaks

Re: Disk quotas not staying in sync?

[Jan Kara]

  Hello,

> > Maybe the problem is caused by root/admin deleting another users files?
> 
> I thought of that first.  But the techs that were deleting mail were
> actually `su`ing to the user and using pine to delete the messages.  So I
> was wondering if pine was doing something strange.  But since we use
> Maildirs when I was cleaning up an account I just `rm`ed their e-mails and
> still ended up seeing the same problem.
  And anyway if root deletes file of some user, allocation is removed from
user so it doesn't matter. Can you check if output of df(1) changes
appropriately to deleted files - in other case the files might just
remain somewhere unlinked but open..

> > Do you run quotacheck at boot time?  It is possible with ext2 that the
> > on-disk quotas are not in sync with the actual files if there is a crash.
> > If you use ext3 then the quota updates and other filesystem updates are
> > kept atomic.
> 
> I have run quotacheck once at boot to see if it would help.  It got the
> stuff in sync for a little while, but soon I started to see the same
> problem again.  quotacheck on a cold cache takes a LONG time to run.
> 
> So when is ext3 going in the kernel?  (ducks)  But really that is
> something I will look at.
> 
> > > Any ideas?  This machine is pretty much standard.  ext2 file systems on
> > > all partitions.  /home is the only one with user disk quotas.  Running
> > > 2.4.7 kernel (but I've seen it in all 2.4 kernels, I was running
> > > 2.4.0-prerelease on the machine when I first put it up, so I don't know
> > > how 2.2 works).
> 
> One thing I forgot to mention about the machine, it is an SMP PIII box.  I
> was really meaning to say it was SMP cause that might be something to look
> at.
> 
> > You should also try the -ac kernels.  I'm pretty sure that they have
> > some fixes to the quota code.  However, I _think_ the changes are not
> > compatible with the non-ac kernel quota on-disk data, so you will have
> > to re-build the quota files (a small price to pay if it actually works).
  You may also try also just some quota fixes from
ftp://atrey.karlin.mff.cuni.cz/pub/local/jack/quota/v2.4/quota-fix-2.4.7-2.diff.gz
  But they are mostly tested only with other patch implementing new quota
format (can be found in same directory as quota-patch-2.4.7-1.diff.gz).

							Honza

--
Jan Kara <jack@suse.cz>
SuSE Labs

Re: Disk quotas not staying in sync?

[Jan Kara]

  Hello,

> On Tue, 7 Aug 2001, Jan Kara wrote:
> 
> >   And anyway if root deletes file of some user, allocation is removed from
> > user so it doesn't matter. Can you check if output of df(1) changes
> > appropriately to deleted files - in other case the files might just
> > remain somewhere unlinked but open..
> 
> I figured that if anyone removed the file it would come out of the user's
> quota.  It wouldn't make sense any other way, but was just making sure.
> I didn't get a chance to look at the output of df.  But I know from before
> when I was updating my glibc that open but unlinked files can be a real
> pain.
  Yes they can :).

> There is a similar thread going on with someone else deleting files from a
> quota enabled partition and not seeing the space being freed up.  The
> files that were being deleted from the user's home directory shouldn't
> have been open by any program.  Week old mail, year old webpages.  But
> still deleting them was not freeing them from the total quota usage
> (immediately).  It did seem that it might have been possible that the
> quotas were freed later, but I don't have any good observed evidence
> either way.  Is it possible that the quota code itself in the kernel was
> keeping the files open?
  No. Quota has opened only quotafiles. It doesn't open any other files.
That's why I think quota has no direct impact on disk space not being freed
(what is - if I understood it well - the problem in that l-k thread).
I can imagine leak in accounting of used space for quotas but not anywhere
else...

> > > I have run quotacheck once at boot to see if it would help.  It got the
> > > stuff in sync for a little while, but soon I started to see the same
> > > problem again.  quotacheck on a cold cache takes a LONG time to run.
> 
> I just got done with a reboot and quotacheck, everything syched up nicely,
> I'm hoping it stays that way this time.
  Hope too but I'm curious about that leak :).


							Honza
--
Jan Kara <jack@suse.cz>
SuSE Labs

Re: Disk quotas not staying in sync?

[Chris Meadors]

I just completed the change to Alan's kernel with ext3 and of course the
new quota code.  Here are my wrapping up comments along with replying to
myself a little bit.

On Tue, 7 Aug 2001, Jan Kara wrote:

>   And anyway if root deletes file of some user, allocation is removed from
> user so it doesn't matter. Can you check if output of df(1) changes
> appropriately to deleted files - in other case the files might just
> remain somewhere unlinked but open..

I figured that if anyone removed the file it would come out of the user's
quota.  It wouldn't make sense any other way, but was just making sure.

I didn't get a chance to look at the output of df.  But I know from before
when I was updating my glibc that open but unlinked files can be a real
pain.

There is a similar thread going on with someone else deleting files from a
quota enabled partition and not seeing the space being freed up.  The
files that were being deleted from the user's home directory shouldn't
have been open by any program.  Week old mail, year old webpages.  But
still deleting them was not freeing them from the total quota usage
(immediately).  It did seem that it might have been possible that the
quotas were freed later, but I don't have any good observed evidence
either way.  Is it possible that the quota code itself in the kernel was
keeping the files open?

> > I have run quotacheck once at boot to see if it would help.  It got the
> > stuff in sync for a little while, but soon I started to see the same
> > problem again.  quotacheck on a cold cache takes a LONG time to run.

I just got done with a reboot and quotacheck, everything syched up nicely,
I'm hoping it stays that way this time.

> > So when is ext3 going in the kernel?  (ducks)  But really that is
> > something I will look at.

What do you know, the -ac patches now include ext3, how nice.  I guess
they do play well together.  :)

>   You may also try also just some quota fixes from
> ftp://atrey.karlin.mff.cuni.cz/pub/local/jack/quota/v2.4/quota-fix-2.4.7-2.diff.gz
>   But they are mostly tested only with other patch implementing new quota
> format (can be found in same directory as quota-patch-2.4.7-1.diff.gz).

Well I went with 2.4.7-ac8, as it included everything I was looking for in
one nice package (tested with -ac4 at home, figured I'd try out the
joural, hit the reset, had the kernel hang with the read-write mount bug,
spent the next hour or so beating my head trying to figure out how to get
back into my machine when I had no boot disks with newer e2fsutils, when I
finally got everything straight, I checked my mail to see the bug reported
and -ac5, well it was a learning experience. :)

> Jan Kara <jack@suse.cz>
> SuSE Labs

Ah, just the person I was looking for.  I probally should have made this a
seperate e-mail, but I'm on a roll.  If you just have comments about this
part, please take it off list, unless you think it is important to anyone
else.

Just 2 little things.

One, repquota in quota-tools 3.0-pre8 is many, many times slower than the
same command in the 2.0 package.  Is there anything that can be done about
that?  I used to use it to get a quick overview of people who were running
up on their quota (thank you for the longer username display along with
the ' ' before the --).  But now my interactive scripts take a long time
to run.

Two, the configure script in 3.0-pre8 looks for and fails to find
ext2fs.h, and says that direct support won't be compiled, but quotacheck
still links against libext2fs (I found out the hard way. :).


Thanks everyone, Andreas for the first reply, Jan for this one, and Alan
keep those patches coming the techs love it when I reboot the server,
-Chris
-- 
Two penguins were walking on an iceberg.  The first penguin said to the
second, "you look like you are wearing a tuxedo."  The second penguin
said, "I might be..."                         --David Lynch, Twin Peaks