mmap vs mtime in 2.6.26

mmap vs mtime in 2.6.26

[Ferenc Wagner]

Hi,

I've noticed that the last modification times of our RRD files got
stuck after upgrading from 2.6.24 to 2.6.26 (Debian Etch -> Lenny).
It has some literature, most notably kernel bug #2645, but that's
closed long ago and the resulting patch
http://lkml.org/lkml/2008/1/22/370 is present in my kernel.  Still,
the test program (version 3 from the bug report) gives failures:

$ ./mmap_test junk
Modifying junk...
Flushing data using sync()...
Failure: time not changed.
Not modifying junk...
Flushing data using msync()...
Success: time not changed.
Not modifying junk...
Flushing data using fsync()...
Success: time not changed.
Modifying junk...
Flushing data using msync()...
Failure: time not changed.
Modifying junk...
Flushing data using fsync()...
Failure: time not changed.

Is this as expected?  I reckon the bug wasn't solved fully, but it
seems to be worse that that.

I don't know if it's related or not, but another anomaly appeared
after the upgrade: minutes-long remount times.  The machine always was
under a fair load (10), but mount /usr -oremount,rw never took
noticeable time.  The IO load pushes two other filesystems.  Now
strace -tt shows:

[...]
01:24:08.463813 stat64("/sbin/mount.xfs", 0xbfa40eb0) = -1 ENOENT (No such file or directory)
01:24:08.463904 mount("/dev/mapper/noc7-usr", "/usr", 0x894a2c8, MS_MGC_VAL|MS_NODEV|MS_REMOUNT, NULL) = 0
01:26:58.705387 readlink("/dev", 0xbfa3ef3b, 4096) = -1 EINVAL (Invalid argument)
[...]

After this, remount ro and rw again are quick for some time.

I can not easily test newer kernels in this setup, so thought to ask
around before delving into it further.  Is this a known problem
perhaps?  Does it ring any bells?  If already fixed, can the fix be
backported to 2.6.26 with reasonable effort?  If not, how can I best
help investigate this?
-- 
Thanks,
Feri.

mmap vs mtime in 2.6.26 and up

[Ferenc Wagner]

Hi,

I've noticed that the last modification times of our RRD files got
stuck after upgrading from 2.6.24 to 2.6.26 (Debian Etch -> Lenny; I
also tested with 2.6.30-rc5, they are still stuck).  It has some
literature, most notably kernel bug #2645, but that's closed long ago
and the resulting patch http://lkml.org/lkml/2008/1/22/370 is present
in my kernels.  Still, the test program (version 3 from the bug report)
gives failures:

$ ./mmap_test junk
Modifying junk...
Flushing data using sync()...
Failure: time not changed.
Not modifying junk...
Flushing data using msync()...
Success: time not changed.
Not modifying junk...
Flushing data using fsync()...
Success: time not changed.
Modifying junk...
Flushing data using msync()...
Failure: time not changed.
Modifying junk...
Flushing data using fsync()...
Failure: time not changed.

Is this as expected?  I reckon the bug wasn't solved fully, but it
seems to be worse that that.

I don't know if it's related or not, but another anomaly appeared
after the upgrade: minutes-long remount times.  The machine always was
under a fair load (10), but mount /usr -oremount,rw never took
noticeable time.  The IO load pushes two other filesystems.  Now
strace -tt shows:

[...]
01:24:08.463813 stat64("/sbin/mount.xfs", 0xbfa40eb0) = -1 ENOENT (No such file or directory)
01:24:08.463904 mount("/dev/mapper/noc7-usr", "/usr", 0x894a2c8, MS_MGC_VAL|MS_NODEV|MS_REMOUNT, NULL) = 0
01:26:58.705387 readlink("/dev", 0xbfa3ef3b, 4096) = -1 EINVAL (Invalid argument)
[...]

After this, remount ro and rw again are quick for some time.
(I couldn't test this part with 2.6.30-rc5, but again, this may be
totally unrelated.)

This bug is quite serious in our environment, as it thwarts the backup
software.  The file system is xfs; rw,nosuid,nodev,nobarrier,noquota.
-- 
Regards,
Feri.

Re: mmap vs mtime in 2.6.26 and up

[Ray Lee]

(Adding Christoph Hellwig, xfs list, Andrew Morton to cc:)

On Tue, May 12, 2009 at 5:32 AM, Ferenc Wagner <wferi@niif.hu> wrote:
> Hi,
>
> I've noticed that the last modification times of our RRD files got
> stuck after upgrading from 2.6.24 to 2.6.26 (Debian Etch -> Lenny; I
> also tested with 2.6.30-rc5, they are still stuck).  It has some
> literature, most notably kernel bug #2645, but that's closed long ago
> and the resulting patch http://lkml.org/lkml/2008/1/22/370 is present
> in my kernels.  Still, the test program (version 3 from the bug report)
> gives failures:
>
> $ ./mmap_test junk
> Modifying junk...
> Flushing data using sync()...
> Failure: time not changed.
> Not modifying junk...
> Flushing data using msync()...
> Success: time not changed.
> Not modifying junk...
> Flushing data using fsync()...
> Success: time not changed.
> Modifying junk...
> Flushing data using msync()...
> Failure: time not changed.
> Modifying junk...
> Flushing data using fsync()...
> Failure: time not changed.
>
> Is this as expected?  I reckon the bug wasn't solved fully, but it
> seems to be worse that that.


I don't think that's expected. Further, the test program (v3) from
that bugzilla http://bugzilla.kernel.org/show_bug.cgi?id=2645 works
for me under ext3, so perhaps this is an xfs-specific issue.


>
> I don't know if it's related or not, but another anomaly appeared
> after the upgrade: minutes-long remount times.  The machine always was
> under a fair load (10), but mount /usr -oremount,rw never took
> noticeable time.  The IO load pushes two other filesystems.  Now
> strace -tt shows:
>
> [...]
> 01:24:08.463813 stat64("/sbin/mount.xfs", 0xbfa40eb0) = -1 ENOENT (No such file or directory)
> 01:24:08.463904 mount("/dev/mapper/noc7-usr", "/usr", 0x894a2c8, MS_MGC_VAL|MS_NODEV|MS_REMOUNT, NULL) = 0
> 01:26:58.705387 readlink("/dev", 0xbfa3ef3b, 4096) = -1 EINVAL (Invalid argument)
> [...]
>
> After this, remount ro and rw again are quick for some time.
> (I couldn't test this part with 2.6.30-rc5, but again, this may be
> totally unrelated.)
>
> This bug is quite serious in our environment, as it thwarts the backup
> software.  The file system is xfs; rw,nosuid,nodev,nobarrier,noquota.
> --
> Regards,
> Feri.

Re: mmap vs mtime in 2.6.26 and up

[Christoph Hellwig]

On Tue, May 12, 2009 at 08:37:43AM -0700, Ray Lee wrote:
> > I've noticed that the last modification times of our RRD files got
> > stuck after upgrading from 2.6.24 to 2.6.26 (Debian Etch -> Lenny; I
> > also tested with 2.6.30-rc5, they are still stuck). ??It has some
> > literature, most notably kernel bug #2645, but that's closed long ago
> > and the resulting patch http://lkml.org/lkml/2008/1/22/370 is present
> > in my kernels. ??Still, the test program (version 3 from the bug report)
> > gives failures:

The problem is pretty simple.  do_wp_page and __do_fault use
file_update_time to update ctime and mtime.  But this function is only
a helper for simply filesystems that have a binary inode dirty/non dirty
state and keep the m/ctime purely in the Linux inode.  It must not be
called from generic code as more complex filesystems need a notification
through ->setattr to update the timestamps.  This will also affect other
filesystems like ubifs.  I'm not entirely sure why it ever worked
before, we must have picked up those c/mtime updates by accident
somehow.

Re: mmap vs mtime in 2.6.26 and up

[Ferenc Wagner]

Christoph Hellwig <hch@infradead.org> writes:

> On Tue, May 12, 2009 at 08:37:43AM -0700, Ray Lee wrote:
>>> I've noticed that the last modification times of our RRD files got
>>> stuck after upgrading from 2.6.24 to 2.6.26 (Debian Etch -> Lenny; I
>>> also tested with 2.6.30-rc5, they are still stuck). ??It has some
>>> literature, most notably kernel bug #2645, but that's closed long ago
>>> and the resulting patch http://lkml.org/lkml/2008/1/22/370 is present
>>> in my kernels. ??Still, the test program (version 3 from the bug report)
>>> gives failures:
>
> The problem is pretty simple.  do_wp_page and __do_fault use
> file_update_time to update ctime and mtime.  But this function is only
> a helper for simply filesystems that have a binary inode dirty/non dirty
> state and keep the m/ctime purely in the Linux inode.  It must not be
> called from generic code as more complex filesystems need a notification
> through ->setattr to update the timestamps.  This will also affect other
> filesystems like ubifs.  I'm not entirely sure why it ever worked
> before, we must have picked up those c/mtime updates by accident
> somehow.

Thanks for the analysis.  Unfortunately I don't nearly know enough to
work on this issue, but would like to track it as it affects our
backup system.  So, shouldn't #2645 be reopened again?
-- 
Feri.

Re: mmap vs mtime in 2.6.26 and up

[Christoph Hellwig]

On Fri, May 15, 2009 at 06:40:29PM +0200, Ferenc Wagner wrote:
> Thanks for the analysis.  Unfortunately I don't nearly know enough to
> work on this issue, but would like to track it as it affects our
> backup system.  So, shouldn't #2645 be reopened again?

Yes, definitively as the current "fix" is incorrected.  I'll try to cook
up a correct version once I get some time.

Re: mmap vs mtime in 2.6.26 and up

[Christoph Hellwig]

On Fri, May 15, 2009 at 12:50:54PM -0400, Christoph Hellwig wrote:
> On Fri, May 15, 2009 at 06:40:29PM +0200, Ferenc Wagner wrote:
> > Thanks for the analysis.  Unfortunately I don't nearly know enough to
> > work on this issue, but would like to track it as it affects our
> > backup system.  So, shouldn't #2645 be reopened again?
> 
> Yes, definitively as the current "fix" is incorrected.  I'll try to cook
> up a correct version once I get some time.

Doing this correctly in the framework of the current codee is
unfortunately not so easy, as calling ->setattr requires taking i_mutex
which we can't in the pagefaul path.

To fix this properly we need to actually update the timestamps during
msync and co as done by the patches from Miklos:

	http://lkml.org/lkml/2007/2/28/166

and Peter:

	http://lkml.org/lkml/2006/5/31/176

----- End forwarded message -----

Re: mmap vs mtime in 2.6.26 and up

[Ferenc Wagner]

Christoph Hellwig <hch@infradead.org> writes:

> On Fri, May 15, 2009 at 12:50:54PM -0400, Christoph Hellwig wrote:
>> On Fri, May 15, 2009 at 06:40:29PM +0200, Ferenc Wagner wrote:
>>> Thanks for the analysis.  Unfortunately I don't nearly know enough to
>>> work on this issue, but would like to track it as it affects our
>>> backup system.  So, shouldn't #2645 be reopened again?
>> 
>> Yes, definitively as the current "fix" is incorrected.  I'll try to cook
>> up a correct version once I get some time.
>
> Doing this correctly in the framework of the current codee is
> unfortunately not so easy, as calling ->setattr requires taking i_mutex
> which we can't in the pagefaul path.
>
> To fix this properly we need to actually update the timestamps during
> msync and co as done by the patches from Miklos:
> 	http://lkml.org/lkml/2007/2/28/166
> and Peter:
> 	http://lkml.org/lkml/2006/5/31/176

Hi Christoph,

http://bugzilla.kernel.org/show_bug.cgi?id=2645#c53 shows that Anton
doesn't quite agree with you on this.  I really can't tell, would you
(or anybody from the accused XFS community) please comment?  Or did
you perhaps fix it meanwhile?  I can't easily test never kernels, but
I will if there's some chance.
-- 
Thanks,
Feri.

Re: mmap vs mtime in 2.6.26 and up

[Ferenc Wagner]

Ferenc Wagner <wferi@niif.hu> writes:

> Christoph Hellwig <hch@infradead.org> writes:
>
>> On Fri, May 15, 2009 at 12:50:54PM -0400, Christoph Hellwig wrote:
>>
>>> On Fri, May 15, 2009 at 06:40:29PM +0200, Ferenc Wagner wrote:
>>>
>>>> Thanks for the analysis.  Unfortunately I don't nearly know enough to
>>>> work on this issue, but would like to track it as it affects our
>>>> backup system.  So, shouldn't #2645 be reopened again?
>>> 
>>> Yes, definitively as the current "fix" is incorrected.  I'll try to cook
>>> up a correct version once I get some time.
>>
>> Doing this correctly in the framework of the current codee is
>> unfortunately not so easy, as calling ->setattr requires taking i_mutex
>> which we can't in the pagefaul path.
>>
>> To fix this properly we need to actually update the timestamps during
>> msync and co as done by the patches from Miklos:
>> 	http://lkml.org/lkml/2007/2/28/166
>> and Peter:
>> 	http://lkml.org/lkml/2006/5/31/176
>
> http://bugzilla.kernel.org/show_bug.cgi?id=2645#c53 shows that Anton
> doesn't quite agree with you on this.  I really can't tell, would you
> (or anybody from the accused XFS community) please comment?  Or did
> you perhaps fix it meanwhile?  I can't easily test newer kernels, but
> I will if there's some chance.

I added some fresh test results to
http://bugzilla.kernel.org/show_bug.cgi?id=2645.  In short, under
2.6.30-rc8 the test program reports full failure on XFS and RAMFS,
full success on EXT3, VFAT and ReiserFS, and mixed results on TMPFS:

$ ./mmap /dev/shm/testfile 
Modifying /dev/shm/testfile...
Flushing data using sync()...
Failure: time not changed.
Not modifying /dev/shm/testfile...
Flushing data using msync()...
Success: time not changed.
Not modifying /dev/shm/testfile...
Flushing data using fsync()...
Success: time not changed.
Modifying /dev/shm/testfile...
Flushing data using msync()...
Failure: time not changed.
Modifying /dev/shm/testfile...
Flushing data using fsync()...
Failure: time not changed.

This doesn't look like az XFS-specific issue, but XFS is definitely
affected.  Anybody's got an idea how to tackle this?
-- 
Regards,
Feri.

Re: mmap vs mtime in 2.6.26 and up

[Christoph Hellwig]

On Fri, Sep 25, 2009 at 02:47:42PM +0200, Ferenc Wagner wrote:
> I added some fresh test results to
> http://bugzilla.kernel.org/show_bug.cgi?id=2645.  In short, under
> 2.6.30-rc8 the test program reports full failure on XFS and RAMFS,
> full success on EXT3, VFAT and ReiserFS, and mixed results on TMPFS:

There's a patch queued up for inclusion in 2.6.32 to work around this
issue in XFS.  The lack of a proper callout from the VFS still makes
this a much less than ideal solution.

Re: mmap vs mtime in 2.6.26 and up

[Ferenc Wagner]

Christoph Hellwig <hch@infradead.org> writes:

> On Fri, Sep 25, 2009 at 02:47:42PM +0200, Ferenc Wagner wrote:
>
>> I added some fresh test results to
>> http://bugzilla.kernel.org/show_bug.cgi?id=2645.  In short, under
>> 2.6.30-rc8 the test program reports full failure on XFS and RAMFS,
>> full success on EXT3, VFAT and ReiserFS, and mixed results on TMPFS:
>
> There's a patch queued up for inclusion in 2.6.32 to work around this
> issue in XFS.  The lack of a proper callout from the VFS still makes
> this a much less than ideal solution.

Great, anyway!  Somehow I managed to gloss over this mail until now,
but happened to test 2.6.32-rc4 and found the issue fixed.  At least
for XFS, as my latest addition to the bug report details it.  Thank
you very much for taking care of this issue!
-- 
Regards,
Feri.