corruption

corruption

[Andries.Brouwer]

I did again a large test comparing two identical trees.
Found again corruption, and, upon inspection, the disk
files did not differ - this is in-core corruption only.

A few days ago:

diff -r /c2/linux/linux-2.4.0-test10/linux/include/asm-sparc/ecc.h /g1/linux/li\
nux-2.4.0-test10/linux/include/asm-sparc/ecc.h
80,83c80,95
< #define ECC_FADDR0_CACHE     0x00000800
< #define ECC_FADDR0_SIZE      0x00000700
< #define ECC_FADDR0_TYPE      0x000000f0
< #define ECC_FADDR0_PADDR     0x0000000f
---
> #define ECC_FADDR0_Ccount << RATIO_SCALE_LOG;
>           if (db->bytes_out != 0)
>             {
>               new_ratio /= db->bytes_out;
>             }
>
>           if (new_ratio < db->ratio || new_ratio < 1 * RATIO_SCALE)
>             {
>               bsd_clear (db);
>               return 1;
>             }
>           db->ratio = new_ratio;
>         }
>       }
>     return 0;
> }

Here the corruption starts precisely 3072 bytes into the file
(which lives on a filesystem with 1024-byte blocks).

But the tail is a fragment of drivers/isdn/isdn_bsdcomp.c
starting at an offset of 7168 bytes.

The former lives on blocks 6373895 6373896 6373897 6373898 6373899,
the other on blocks 2475568...2475579,2475616...2475628.

Today:

diff -r /g1/linux/linux-2.4.0-test11vanilla/linux/net/sched/sch_cbq.c /c2/linux\
/linux-2.4.0-test11vanilla/linux/net/sched/sch_cbq.c
2000c2000,2115
<               cbq_destr\201XM^@\202XM^@^@^@^@^@^@^@^@^@^@^@...

(lots of nulls)

with corruption starting at an offset of 47104=46*1024 bytes.
Don't know where the corruption part is from.

Andries
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/

Re: corruption

[Linus Torvalds]

On Wed, 29 Nov 2000 Andries.Brouwer@cwi.nl wrote:
>
> I did again a large test comparing two identical trees.
> Found again corruption, and, upon inspection, the disk
> files did not differ - this is in-core corruption only.

Ok. It definitely looks like the 1kB thing has become broken somehow. 

The fact that it is in-core only doesn't mean that much - it could still
easily be just problems at read-time, and if you have an IDE disk I would
strongly suggest you try out the patch that Jens Axboe posted,
re-initializing the "head" pointer when doing a re-merge.

That said, the VM/ext2 angle should definitely be looked at too. Nothing
has really changed there in some time - can you give a rough estimate on
when you suspect you started seeing it? Ie is it new to one of the test11
pre-kernels, or does it happen so occasionally that you can't tell whether
it happened much earlier too?

		Linus

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/

Re: corruption

[Alexander Viro]

On Tue, 28 Nov 2000, Linus Torvalds wrote:

> The fact that it is in-core only doesn't mean that much - it could still
> easily be just problems at read-time, and if you have an IDE disk I would
> strongly suggest you try out the patch that Jens Axboe posted,
> re-initializing the "head" pointer when doing a re-merge.
> 
> That said, the VM/ext2 angle should definitely be looked at too. Nothing
> has really changed there in some time - can you give a rough estimate on
> when you suspect you started seeing it? Ie is it new to one of the test11
> pre-kernels, or does it happen so occasionally that you can't tell whether
> it happened much earlier too?

Problem fixed by Jens' patch had been there since March, so if it's a
mix of __make_request() screwing up and something else... Urgh.

I'ld really like to see details on the box with ext2 corruption on SCSI.
Tigran, IIRC you had it on SCSI boxen, right? Could you send me relevant
part of logs?


-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/

Re: corruption

[Tigran Aivazian]

On Wed, 29 Nov 2000, Alexander Viro wrote:
> 
> I'ld really like to see details on the box with ext2 corruption on SCSI.
> Tigran, IIRC you had it on SCSI boxen, right? Could you send me relevant
> part of logs?
> 

I definitely did have this very corruption on a 4xXeon SCSI-only box. But
the bad news is that I reinstalled redhat7 on it immediately after this
happened so I don't have the logs. _However_, I don't need that particular
root filesystem there anymore (since more disks arrive today and I'm
rearranging stuff) so I'll try and corrupt it for you right now. Using
test12-pre3, unless you have better suggestions on what to do to help.

Regards,
Tigran

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/

Re: corruption

[Alexander Viro]

On Wed, 29 Nov 2000, Tigran Aivazian wrote:

> On Wed, 29 Nov 2000, Alexander Viro wrote:
> > 
> > I'ld really like to see details on the box with ext2 corruption on SCSI.
> > Tigran, IIRC you had it on SCSI boxen, right? Could you send me relevant
> > part of logs?
> > 
> 
> I definitely did have this very corruption on a 4xXeon SCSI-only box. But

"This" as in "range of blocks duplicated onto another range", "random
crap in indirect blocks" or both?

> the bad news is that I reinstalled redhat7 on it immediately after this
> happened so I don't have the logs. _However_, I don't need that particular
> root filesystem there anymore (since more disks arrive today and I'm
> rearranging stuff) so I'll try and corrupt it for you right now. Using
> test12-pre3, unless you have better suggestions on what to do to help.

Could you look for duplicates too?
							TIA,
								Al

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/

Re: corruption

[Tigran Aivazian]

On Wed, 29 Nov 2000, Alexander Viro wrote:
> Could you look for duplicates too?

will do. One useful finding so far -- trying simultaneous mke2fs /dev/sdX1
for X = {b,c,d,e,f} deadlocks the machine dead (and without kdb such death
was in vain). (each disk is 37G, RAM is 6G)

I know this is offtopic for this thread but not for this list.

I am continuing to pursue the corruption. Nothing yet.

Regards,
Tigran

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/

Re: corruption

[Andrea Arcangeli]

On Wed, Nov 29, 2000 at 04:08:26AM -0500, Alexander Viro wrote:
> Problem fixed by Jens' patch had been there since March, so if it's a

No, it's there only since Jens fixed the request merging bug in test11 or so.

With previous kernel the head pointer couldn't change so that change
was unnecessary and initializing it outside the critical section was
a micro scalability optimization :).

Andrea
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/

Re: corruption

[Rik van Riel]

On Wed, 29 Nov 2000, Andrea Arcangeli wrote:
> On Wed, Nov 29, 2000 at 04:08:26AM -0500, Alexander Viro wrote:
> > Problem fixed by Jens' patch had been there since March, so if it's a
> 
> No, it's there only since Jens fixed the request merging bug in
> test11 or so.
> 
> With previous kernel the head pointer couldn't change so that
> change was unnecessary and initializing it outside the critical
> section was a micro scalability optimization :).

To be honest, I have a big problem with micro optimisations
that prevent the big optimisations from happening.

Would it be an idea to explicitly comment such dangerous
micro optimisations so people implementing the big optimisations
later on won't run into nasty surprises?

regards,

Rik
--
Hollywood goes for world dumbination,
	Trailer at 11.

		http://www.surriel.com/
http://www.conectiva.com/	http://distro.conectiva.com.br/

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/

Re: corruption

[Linus Torvalds]

On Wed, 29 Nov 2000, Alexander Viro wrote:
> 
> Problem fixed by Jens' patch had been there since March, so if it's a
> mix of __make_request() screwing up and something else... Urgh.

No, the bug really got introduced in test11 due to the request merging
stuff.

The patch may _look_ like it fixed a generic problem that has been there
forever, but we didn't actually need the spinlock for initializing "head"
at all. It's initialized to a constant offset within the unchaning request
queue, so we can happily do it outside the spinlock.

The reason the initialization was moved inside the spinlock was really
just that it had to be re-initialized for the case where we re-did the
merge, so it had to be moved down to inside the loop - and it just happens
to happen inside the spinlock now.

So the spinlock protection was never relevant to the bug - forgetting to
re-initialize a variable when a straight-line code was turned into a loop
was the bug.

> I'ld really like to see details on the box with ext2 corruption on SCSI.
> Tigran, IIRC you had it on SCSI boxen, right? Could you send me relevant
> part of logs?

I suspect that Tigran may have seen other instability (of which we had
lots back when he saw it), and that the current rash is for the IDE
problem only. 

Which is not to say that there might not be SCSI issues or other issues
too, but I'm also not convinced that the SCSI thing might not just be a
red herring at this point.

		Linus

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/

Re: corruption

[Andrea Arcangeli]

On Wed, Nov 29, 2000 at 05:05:20PM -0200, Rik van Riel wrote:
> To be honest, I have a big problem with micro optimisations
> that prevent the big optimisations from happening.
> 
> Would it be an idea to explicitly comment such dangerous
> micro optimisations so people implementing the big optimisations
> later on won't run into nasty surprises?

Did you read the code we're talking about?

Andrea
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/

Re: corruption

[Alexander Viro]

On Wed, 29 Nov 2000, Linus Torvalds wrote:

> 
> 
> On Wed, 29 Nov 2000, Alexander Viro wrote:
> > 
> > Problem fixed by Jens' patch had been there since March, so if it's a
> > mix of __make_request() screwing up and something else... Urgh.
> 
> No, the bug really got introduced in test11 due to the request merging
> stuff.
> 
> The patch may _look_ like it fixed a generic problem that has been there
> forever, but we didn't actually need the spinlock for initializing "head"

Sure.

> at all. It's initialized to a constant offset within the unchaning request
> queue, so we can happily do it outside the spinlock.

Actually, I was not thinking about spinlock. What I missed was the fact
that again: was quite recent. My apologies...

> > I'ld really like to see details on the box with ext2 corruption on SCSI.
> > Tigran, IIRC you had it on SCSI boxen, right? Could you send me relevant
> > part of logs?
> 
> I suspect that Tigran may have seen other instability (of which we had
> lots back when he saw it), and that the current rash is for the IDE
> problem only. 
> 
> Which is not to say that there might not be SCSI issues or other issues
> too, but I'm also not convinced that the SCSI thing might not just be a
> red herring at this point.

There are two quite distinct patterns: duplicated range vs. crap in metadata.
The former looks like a bug caught by Jens. The latter (especially in
bitmaps) seems to be older[1] and independent from elevator stuff. _That_ may
be a fs/buffer.c or fs/ext2/* bug. The former definitely lives below the
fs/buffer.c level.

[1] "older" may mean "shared with 2.2" here - ISTR bug reports looking like
that and IIRC they were never resolved. BTW, if you know some searchable
l-k archive... DN sucks coprolites through the straw these days ;-/

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/

Re: corruption

[Rik van Riel]

On Wed, 29 Nov 2000, Andrea Arcangeli wrote:
> On Wed, Nov 29, 2000 at 05:05:20PM -0200, Rik van Riel wrote:
> > To be honest, I have a big problem with micro optimisations
> > that prevent the big optimisations from happening.
> > 
> > Would it be an idea to explicitly comment such dangerous
> > micro optimisations so people implementing the big optimisations
> > later on won't run into nasty surprises?
> 
> Did you read the code we're talking about?

This particular piece of code may be a bad example of
my "complaint", but I guess we can just as easily take
something like shrink_mmap() as our example ...

regards,

Rik
--
Hollywood goes for world dumbination,
	Trailer at 11.

		http://www.surriel.com/
http://www.conectiva.com/	http://distro.conectiva.com.br/

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/

Re: corruption

[Andrea Arcangeli]

On Wed, Nov 29, 2000 at 02:57:11PM -0500, Alexander Viro wrote:
> that again: was quite recent. My apologies...

Never mind, strict patch reading was obviously misleading in this case.

> [1] "older" may mean "shared with 2.2" here - ISTR bug reports looking like
> that and IIRC they were never resolved. [..]

I don't recall any report after 2.2.1x series but I might have missed them.
Previous reports may be due the bugs that got fixed there.

Andrea
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/

Re: searchable l-k archive

[Andreas Dilger]

Al Viro writes:
> BTW, if you know some searchable l-k archive...
> DN sucks coprolites through the straw these days ;-/

There are archives of pretty much all of the linux- and oss-development
related mailing lists at:

http://marc.theaimsgroup.com/

For l-k in particular, I use
http://marc.theaimsgroup.com/?l=linux-kernel&r=1&w=4

Cheers, Andreas
-- 
Andreas Dilger  \ "If a man ate a pound of pasta and a pound of antipasto,
                 \  would they cancel out, leaving him still hungry?"
http://www-mddsp.enel.ucalgary.ca/People/adilger/               -- Dogbert
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/