[EXT3/JBD] Periodic journal flush not enough?

[EXT3/JBD] Periodic journal flush not enough?

[Herbert Xu]

Hi:

I've encountered a problem with the journal flush timer.  The problem
is that when a filesystem is short on space, relying on a timer-based
flushing mechanism is no longer adequate.  For example, on my P4 2GHz
I can trigger an ENOSPC error by doing

while :; do echo test > a; [ -s a ] || break; rm a; done; echo Out of space

on an ext3 file system with 12Mb of free space using the usual 5s
journal flush timer.

Of course, when you extend the flushing period as you do with laptop-mode,
this problem becomes a lot worse.

So would it be possible to have the flushing activated on demand?

Thanks,
-- 
Debian GNU/Linux 3.0 is out! ( http://www.debian.org/ )
Email:  Herbert Xu ~{PmV>HI~} <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

Re: [EXT3/JBD] Periodic journal flush not enough?

[Andrew Morton]

Herbert Xu <herbert@gondor.apana.org.au> wrote:
>
> I've encountered a problem with the journal flush timer.  The problem
> is that when a filesystem is short on space, relying on a timer-based
> flushing mechanism is no longer adequate.  For example, on my P4 2GHz
> I can trigger an ENOSPC error by doing
> 
> while :; do echo test > a; [ -s a ] || break; rm a; done; echo Out of space
> 
> on an ext3 file system with 12Mb of free space using the usual 5s
> journal flush timer.

I cannot reproduce this.  Please send more details.  Journalling mode,
kernel version, etc.

Re: [EXT3/JBD] Periodic journal flush not enough?

[Herbert Xu]

[-- Attachment #1: Type: text/plain, Size: 1397 bytes --]

On Fri, Mar 26, 2004 at 03:48:51PM -0800, Andrew Morton wrote:
> Herbert Xu <herbert@gondor.apana.org.au> wrote:
> >
> > I've encountered a problem with the journal flush timer.  The problem
> > is that when a filesystem is short on space, relying on a timer-based
> > flushing mechanism is no longer adequate.  For example, on my P4 2GHz
> > I can trigger an ENOSPC error by doing
> > 
> > while :; do echo test > a; [ -s a ] || break; rm a; done; echo Out of space
> > 
> > on an ext3 file system with 12Mb of free space using the usual 5s
> > journal flush timer.
> 
> I cannot reproduce this.  Please send more details.  Journalling mode,
> kernel version, etc.

OK.  I can reproduce this under both 2.4.25 and 2.6.4.

To prepare for the test, you need to arrange for an ext3 file system
which is short on space.  In the following example, I'm using one with
around 12Mb of free space.

You'll also need a way to write/delete files quickly.  The above shell
fragment probably doesn't in bash since it's very slow.

So I've attached a C program which does a similar thing.

I've also attached the script output of it running in my VMware machine
running 2.6.4.

Cheers,
-- 
Debian GNU/Linux 3.0 is out! ( http://www.debian.org/ )
Email:  Herbert Xu ~{PmV>HI~} <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

[-- Attachment #2: b.c --]
[-- Type: text/x-csrc, Size: 444 bytes --]

#include <unistd.h>
#include <stdio.h>
#include <strings.h>
#include <pwd.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>

int main(void)
{
	int fd;
	int cnt = 0;

	while (1) {
		char c = 'x';
		fd = open("a", O_WRONLY|O_CREAT|O_EXCL);
		if (fd < 0) {
			perror("open");
			break;
		}
		if (write(fd, &c, 1) != 1) {
			perror("write");
			break;
		}
		close(fd);
		unlink("a");
		cnt++;
	}
	printf("%d\n", cnt);
	return 0;
}

[-- Attachment #3: c --]
[-- Type: text/plain, Size: 319 bytes --]

Script started on Sun Feb 22 22:08:38 2004
angband:~# df .
Filesystem           1K-blocks      Used Available Use% Mounted on
/devfs/mapper/ren-root
                        253871    241542     12329  96% /
angband:~# /tmp/b
write: No space left on device
21603
angband:~# exit

Script done on Sun Feb 22 22:08:51 2004

Re: [EXT3/JBD] Periodic journal flush not enough?

[Stephen C. Tweedie]

Hi,

On Fri, 2004-03-26 at 23:48, Andrew Morton wrote:
> Herbert Xu <herbert@gondor.apana.org.au> wrote:
> >
> > I've encountered a problem with the journal flush timer.  The problem
> > is that when a filesystem is short on space, relying on a timer-based
> > flushing mechanism is no longer adequate.  For example, on my P4 2GHz
> > I can trigger an ENOSPC error by doing
> > 
> > while :; do echo test > a; [ -s a ] || break; rm a; done; echo Out of space
> > 
> > on an ext3 file system with 12Mb of free space using the usual 5s
> > journal flush timer.
> 
> I cannot reproduce this.  Please send more details.  Journalling mode,
> kernel version, etc.

Sounds like it's due to the "b_committed_data" avoidance code.  Ext3
cannot immediately reuse disk space after a delete, because of lazy
writeback --- until the final writeback of the delete hits disk, we have
to be able to undo it.  And because in non-data-journaled modes we allow
new disk writes to hit disk before a transaction commit, that means we
can't reuse deleted blocks until after they are committed.

I've never seen it reported as a problem outside of artificial test
scenarios, but if it is something we need to address, Andreas Dilger's
patch looks good.

Cheers,
 Stephen

Re: [EXT3/JBD] Periodic journal flush not enough?

[Chris Mason]

On Mon, 2004-03-29 at 13:56, Stephen C. Tweedie wrote:

> Sounds like it's due to the "b_committed_data" avoidance code.  Ext3
> cannot immediately reuse disk space after a delete, because of lazy
> writeback --- until the final writeback of the delete hits disk, we have
> to be able to undo it.  And because in non-data-journaled modes we allow
> new disk writes to hit disk before a transaction commit, that means we
> can't reuse deleted blocks until after they are committed.
> 
> I've never seen it reported as a problem outside of artificial test
> scenarios, but if it is something we need to address, Andreas Dilger's
> patch looks good.

Just FYI, reiserfs does something slightly different.  When
reiserfs_file_write and get_block routines see -ENOSPC, they get things
into a consistent state, commit the running transaction and try again
(once).  It didn't end up very complex...

-chris

Re: [EXT3/JBD] Periodic journal flush not enough?

[Andreas Dilger]

On Mar 27, 2004  10:19 +1100, Herbert Xu wrote:
> I've encountered a problem with the journal flush timer.  The problem
> is that when a filesystem is short on space, relying on a timer-based
> flushing mechanism is no longer adequate.  For example, on my P4 2GHz
> I can trigger an ENOSPC error by doing
> 
> while :; do echo test > a; [ -s a ] || break; rm a; done; echo Out of space
> 
> on an ext3 file system with 12Mb of free space using the usual 5s
> journal flush timer.
> 
> Of course, when you extend the flushing period as you do with laptop-mode,
> this problem becomes a lot worse.
> 
> So would it be possible to have the flushing activated on demand?

I had created a patch a while ago, but never really got any testing on it.
It would be great to get this problem fixed.  Patch against a 2.4.x kernel.

--- fs/ext3/balloc.c.orig	Fri Jul 25 19:55:34 2003
+++ fs/ext3/balloc.c	Tue Sep  2 16:27:51 2003
@@ -547,6 +547,8 @@ int ext3_new_block (handle_t *handle, st
 #ifdef EXT3FS_DEBUG
 	static int goal_hits = 0, goal_attempts = 0;
 #endif
+	int tried_commit = 0;
+
 	*errp = -ENOSPC;
 	sb = inode->i_sb;
 	if (!sb) {
@@ -643,6 +645,26 @@ repeat:
 		}
 	}
 
+	/* We can only try to commit the previous transaction, or we will
+	 * deadlock because the current op has a transaction handle open.
+	 * We also can't restart the current handle in a new transaction as
+	 * that might break the atomicity guarantees of this transaction.
+	 * Set current handle h_sync to allow it to be committed ASAP. */
+	if (!tried_commit) {
+		journal_t *journal = handle->h_transaction->t_journal;
+		transaction_t *prev_trans = journal->j_committing_transaction;
+
+		if (prev_trans) {
+			tid_t prev_tid = prev_trans->t_tid;
+			log_start_commit(journal, prev_trans);
+			log_wait_commit(journal, prev_tid);
+		}
+		handle->h_sync = 1;
+		tried_commit = 1;
+
+		goto repeat;
+	}
+
 	/* No space left on the device */
 	goto out;
 
Cheers, Andreas
--
Andreas Dilger
http://sourceforge.net/projects/ext2resize/
http://www-mddsp.enel.ucalgary.ca/People/adilger/

Re: [EXT3/JBD] Periodic journal flush not enough?

[Stephen C. Tweedie]

Hi,

On Fri, 2004-03-26 at 23:55, Andreas Dilger wrote:

> I had created a patch a while ago, but never really got any testing on it.
> It would be great to get this problem fixed.

We don't really want to turn ENOSPC into an operation which always
requires synchronous IO, though, which your patch would do.  We can
probably detect the case where we've failed the allocate due to a
b_committed_data collision, though, and perform the retry only in that
case.

Cheers,
 Stephen