the remount problem [2.4.0] kind of solved [patch]

the remount problem [2.4.0] kind of solved [patch]

[Bernd Eckenfels]

Hello,

the following patch against 2.4.0 will allow the kernel to write a message
to the kernel log in case files are open for write or delete on a partition
which should be remounted.

I run my System with Read-Only /usr File System and this works fairly well.
I have a script to remount the different Filesystems if I want to upgrade
them (using the cool apt-get from Debian).

Sometimes after upgrades I noticed, that I am unable to remount the /usr
File system read only. Since I was unable to detect the Reason for it (no
file according to lsof or fuser was open for writing) i decided to patch the
kernel to debug the problem.

The Solution: some files are deleted but open. Most of the time this is due
to an upgrade of a shared lib without restarting the related daemon. I am
not sure why I dont see the open but deleted files in lsof (according to its
man page "lsof -aL1 /usr" should work, but anyway. Here is my patch, in case
you experience the same problems. Not sure if it is ok to include it that
way into mainstream kernel since it might produce quite a few lines of logs
and i am not sure if prinkt is save inside the file list lock anyway.

So my question: which user mode program can be used to detect those "open
but deleted" mmaped files?

With my patch i get the inode an can grep in maps...

calista:/usr/src/linux# grep 145429 /proc/*/maps
/proc/354/maps:40156000-4016d000 r-xp 00000000 08:05 145429
/usr/lib/jabber/jsm/jsm.so (deleted)
/proc/354/maps:4016d000-4016e000 rw-p 00016000 08:05 145429
/usr/lib/jabber/jsm/jsm.so (deleted)
/proc/366/maps:40156000-4016d000 r-xp 00000000 08:05 145429
/usr/lib/jabber/jsm/jsm.so (deleted)
/proc/366/maps:4016d000-4016e000 rw-p 00016000 08:05 145429
/usr/lib/jabber/jsm/jsm.so (deleted)

Perhaps it is enough to run "grep '(deleted)' /proc/*/maps" in cron.daily?

Well, for Debian, it would be nice if we can make sure that on shared lib
upgrade at least a list of programs which needs to be restarted is mailed to
root. We do restartes on libc upgrade, but I guess it is not possible to
restart for other shared libs (automatically).

Greetings
Bernd

--- /usr/src/linux/fs/file_table.corg   Sun Jan 21 12:29:04 2001
+++ /usr/src/linux/fs/file_table.c      Sun Jan 21 12:40:07 2001
@@ -182,12 +182,15 @@
                inode = file->f_dentry->d_inode;
 
                /* File with pending delete? */
-               if (inode->i_nlink == 0)
+               if (inode->i_nlink == 0) {
+                       printk("ro-remount failed: pending delete on %s, inode no %lu\n",kdevname(inode->i_dev),inode->i_ino);
                        goto too_bad;
-
+               }
                /* Writable file? */
-               if (S_ISREG(inode->i_mode) && (file->f_mode & FMODE_WRITE))
+               if (S_ISREG(inode->i_mode) && (file->f_mode & FMODE_WRITE)) {
+                       printk("ro-remount failed: file open for write on %s, inode no %lu\n",kdevname(inode->i_dev),inode->i_ino);
                        goto too_bad;
+               }
        }
        file_list_unlock();
        return 1; /* Tis' cool bro. */

Greetings
Bernd
-- 
  (OO)      -- Bernd_Eckenfels@Wendelinusstrasse39.76646Bruchsal.de --
 ( .. )  ecki@{inka.de,linux.de,debian.org} http://home.pages.de/~eckes/
  o--o     *plush*  2048/93600EFD  eckes@irc  +497257930613  BE5-RIPE
(O____O)  When cryptography is outlawed, bayl bhgynjf jvyy unir cevinpl!
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/

Re: the remount problem [2.4.0] kind of solved [patch]

[Goswin Brederlow]

>>>>> " " == Bernd Eckenfels <ecki@lina.inka.de> writes:

     > Hello, the following patch against 2.4.0 will allow the kernel
     > to write a message to the kernel log in case files are open for
     > write or delete on a partition which should be remounted.

     > I run my System with Read-Only /usr File System and this works
     > fairly well.  I have a script to remount the different
     > Filesystems if I want to upgrade them (using the cool apt-get
     > from Debian).

     > Sometimes after upgrades I noticed, that I am unable to remount
     > the /usr File system read only. Since I was unable to detect
     > the Reason for it (no file according to lsof or fuser was open
     > for writing) i decided to patch the kernel to debug the
     > problem.

     > The Solution: some files are deleted but open. Most of the time
     > this is due to an upgrade of a shared lib without restarting
     > the related daemon. I am not sure why I dont see the open but
     > deleted files in lsof (according to its man page "lsof -aL1
     > /usr" should work, but anyway. Here is my patch, in case you
     > experience the same problems. Not sure if it is ok to include
     > it that way into mainstream kernel since it might produce quite
     > a few lines of logs and i am not sure if prinkt is save inside
     > the file list lock anyway.

Make it optional. Add an entry in proc that turns the feature on and
of and an config option disabling the code alltogether. Then send the
patch in and see what they say.

     > So my question: which user mode program can be used to detect
     > those "open but deleted" mmaped files?

     > With my patch i get the inode an can grep in maps...

     > calista:/usr/src/linux# grep 145429 /proc/*/maps
     > /proc/354/maps:40156000-4016d000 r-xp 00000000 08:05 145429
     > /usr/lib/jabber/jsm/jsm.so (deleted)
     > /proc/354/maps:4016d000-4016e000 rw-p 00016000 08:05 145429
     > /usr/lib/jabber/jsm/jsm.so (deleted)
     > /proc/366/maps:40156000-4016d000 r-xp 00000000 08:05 145429
     > /usr/lib/jabber/jsm/jsm.so (deleted)
     > /proc/366/maps:4016d000-4016e000 rw-p 00016000 08:05 145429
     > /usr/lib/jabber/jsm/jsm.so (deleted)

Why in hell are library open for write? But it doesn't seem to be only
libraries:

% cat /proc/self/maps          
08048000-0804b000 r-xp 00000000 03:01 22109      /bin/cat
0804b000-0804c000 rw-p 00002000 03:01 22109      /bin/cat
40000000-40016000 r-xp 00000000 03:01 18160      /lib/ld-2.2.1.so
40016000-40017000 rw-p 00015000 03:01 18160      /lib/ld-2.2.1.so
40017000-40018000 rw-p 00000000 00:00 0
4001f000-40120000 r-xp 00000000 03:01 18171      /lib/libc-2.2.1.so
40120000-40127000 rw-p 00100000 03:01 18171      /lib/libc-2.2.1.so
40127000-4012b000 rw-p 00000000 00:00 0
bfffe000-c0000000 rwxp fffff000 00:00 0

     > Perhaps it is enough to run "grep '(deleted)' /proc/*/maps" in
     > cron.daily?

     > Well, for Debian, it would be nice if we can make sure that on
     > shared lib upgrade at least a list of programs which needs to
     > be restarted is mailed to root. We do restartes on libc
     > upgrade, but I guess it is not possible to restart for other
     > shared libs (automatically).

At boottime the filesystems are readonly, so any deamon that gets
startet can only have read and exec permissions on files. Deleting a
library and replacing it with a new one can't change that.

I think the problem comes from daemons that actually get restarted,
maybe before the library is updated (so they will load the old/deleted
one).

MfG
        Goswin
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/

Re: the remount problem [2.4.0] kind of solved [patch]

[Bernd Eckenfels]

Hello,

for Short: I had a mail exchange with Vic Abell, the lsof Author, and in the
next Version of lsof the open shared libs will be detected. So my Kernel
Patch is no longer needed:

# ~root/rw
# rm /usr/lib/jabber/jsm/libjsm.so
# ~root/ro
mount: /usr busy
# lsof_4.55A.linux/lsof -a +L1 /usr
COMMAND  PID   USER  FD   TYPE DEVICE SIZE NLINK   NODE NAME
jabberd 9657 daemon mem    DEL    8,5          0 145429 /usr/lib/jabber/jsm/jsm.so
jabberd 9658 daemon mem    DEL    8,5          0 145429 /usr/lib/jabber/jsm/jsm.so
#

Still we need to address the problem of upgraded libs (at least in a Great
Distribution like Debian is :)

> Why in hell are library open for write? But it doesn't seem to be only
> libraries:

They are not open for write. They are open for mmaped read. The Problem with
this is, that as long as the files are open, the filesystem cannot remove
them from disk. This means, that as long as you have files open, even for
read, which are deleted, a remount ro will fail.

The new lsof will find those mmaped files, so you can simply restart the
associated binary.

> At boottime the filesystems are readonly, so any deamon that gets
> startet can only have read and exec permissions on files. Deleting a
> library and replacing it with a new one can't change that.

Deleteing a lib which is open (which will happen if u upgrade a lib*.deb)
will leave a open-but-delted file on the system, which in turn will disable
you to remount the file system read only. In my case i switch from read-only
/usr to read-write user, use apt-get -ufm upgrade and then want to switch
back to read-only /usr. But exactly the later is not possible. It is also
wasting shared memory since freshly started programs will use the new lib,
the old ones will use the old, unlinked libs.

> I think the problem comes from daemons that actually get restarted,
> maybe before the library is updated (so they will load the old/deleted
> one).

Yes, all daemons will get started with the old libs, since a upgrade always
happens after system start :) But not only daemons. Think of Shells, getty,
login, ...

Greetings
Bernd
-- 
  (OO)      -- Bernd_Eckenfels@Wendelinusstrasse39.76646Bruchsal.de --
 ( .. )  ecki@{inka.de,linux.de,debian.org} http://home.pages.de/~eckes/
  o--o     *plush*  2048/93600EFD  eckes@irc  +497257930613  BE5-RIPE
(O____O)  When cryptography is outlawed, bayl bhgynjf jvyy unir cevinpl!
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/

Re: the remount problem [2.4.0] kind of solved [patch]

[Goswin Brederlow]

>>>>> " " == Bernd Eckenfels <lists@lina.inka.de> writes:
    >> Why in hell are library open for write? But it doesn't seem to
    >> be only libraries:

     > They are not open for write. They are open for mmaped read. The
     > Problem with this is, that as long as the files are open, the
     > filesystem cannot remove them from disk. This means, that as
     > long as you have files open, even for read, which are deleted,
     > a remount ro will fail.

     > The new lsof will find those mmaped files, so you can simply
     > restart the associated binary.

Ah, I see. Forgot about that.

Maybe the kernel coud swap in the deleted libraries and keep it in
memory or real swap from then on instead of blocking the fs.

     > Yes, all daemons will get started with the old libs, since a
     > upgrade always happens after system start :) But not only
     > daemons. Think of Shells, getty, login, ...

So, when you update glibc, / can't be remounted ro any more?
Ugly.

MfG
        Goswin
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/

Re: the remount problem [2.4.0] kind of solved [patch]

[Wichert Akkerman]

Previously Goswin Brederlow wrote:
> Maybe the kernel coud swap in the deleted libraries and keep it in
> memory or real swap from then on instead of blocking the fs.

No, you have no idea how large the file might grow and you need to
keep that data somewhere.

Wichert.

-- 
  _________________________________________________________________
 /       Nothing is fool-proof to a sufficiently talented fool     \
| wichert@cistron.nl                  http://www.liacs.nl/~wichert/ |
| 1024D/2FA3BC2D 576E 100B 518D 2F16 36B0  2805 3CB8 9250 2FA3 BC2D |
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/

Re: the remount problem [2.4.0] kind of solved [patch]

[Albert D. Cahalan]

Wichert Akkerman writes:
> Previously Goswin Brederlow wrote:

>> Maybe the kernel coud swap in the deleted libraries and keep it in
>> memory or real swap from then on instead of blocking the fs.
>
> No, you have no idea how large the file might grow and you need to
> keep that data somewhere.

Grow? It makes little difference, since you can run out of filesystem
space too.
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/