PROBLEM: Inotify leaks file descriptors.

PROBLEM: Inotify leaks file descriptors.

[David Turner]

I apologize for the slightly convoluted reproduction steps here,
but I was not easily able to find a simpler test case in the
time that I had available.

First, you'll need Facebook's watchman:
https://github.com/facebook/watchman

Build and install it.  Then run the attached Python script.
After a few hundred lines, you'll start to see errors of the form
inotify_init error: Too many open files.  That could just
indicate that watchman is leaking, but I think that's not what's
going on, because killing watchman does not fix the problem.

To demonstrate, kill the python script, then kill watchman.
Then run tail -f /etc/hosts.  You'll get "tail: inotify cannot be
used, reverting to polling: Too many open files" (you may need to
run a few tails to see the error).  In fact, the only way I have
found to get back to normal is to reboot.

I tried increasing the ulimit to 10000 (from the default 1024).
The error still happens, but it seems to take a bit longer.

I have tried on a couple of Ubuntu kernels:

Linux version 3.11.0-17-generic (buildd@toyol) (gcc version 4.6.3
(Ubuntu/Linaro 4.6.3-1ubuntu5) ) #31~precise1-Ubuntu SMP Tue Feb 4
21:25:43 UTC 2014

And Ubuntu's 3.8.0-36-generic (it's not running right now so I can't give
the full version).

I've also tried a stock kernel built from source (in a virtualbox):

Linux version 3.13.5 (dturner@dturner-virtualbox) (gcc version 4.8.1
(Ubuntu/Linaro 4.8.1-10ubuntu8) ) #1 SMP Mon Mar 3 20:41:51 EST 2014

I get the error on all of these.
There is no output in dmesg.

I was running these tests on ext4 filesystems:
(for the Ubuntu kernels)
/dev/mapper/stross--vg-root on / type ext4 (rw,errors=remount-ro)
(for the stock kernel, in the virtualbox)
/dev/sda1 on / type ext4 (rw,errors=remount-ro)

Please let me know if you need any more information.

FWIW, I did find this bug while googling, but it was on older kernels and
was allegedly fixed:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1101666

Re: PROBLEM: Inotify leaks file descriptors.

[David Turner]

[-- Attachment #1: Type: text/plain, Size: 2186 bytes --]

(script attached)
On Tue, March 4, 2014 2:09 pm, David Turner wrote:
> I apologize for the slightly convoluted reproduction steps here,
> but I was not easily able to find a simpler test case in the
> time that I had available.
>
> First, you'll need Facebook's watchman:
> https://github.com/facebook/watchman
>
> Build and install it.  Then run the attached Python script.
> After a few hundred lines, you'll start to see errors of the form
> inotify_init error: Too many open files.  That could just
> indicate that watchman is leaking, but I think that's not what's
> going on, because killing watchman does not fix the problem.
>
> To demonstrate, kill the python script, then kill watchman.
> Then run tail -f /etc/hosts.  You'll get "tail: inotify cannot be
> used, reverting to polling: Too many open files" (you may need to
> run a few tails to see the error).  In fact, the only way I have
> found to get back to normal is to reboot.
>
> I tried increasing the ulimit to 10000 (from the default 1024).
> The error still happens, but it seems to take a bit longer.
>
> I have tried on a couple of Ubuntu kernels:
>
> Linux version 3.11.0-17-generic (buildd@toyol) (gcc version 4.6.3
> (Ubuntu/Linaro 4.6.3-1ubuntu5) ) #31~precise1-Ubuntu SMP Tue Feb 4
> 21:25:43 UTC 2014
>
> And Ubuntu's 3.8.0-36-generic (it's not running right now so I can't give
> the full version).
>
> I've also tried a stock kernel built from source (in a virtualbox):
>
> Linux version 3.13.5 (dturner@dturner-virtualbox) (gcc version 4.8.1
> (Ubuntu/Linaro 4.8.1-10ubuntu8) ) #1 SMP Mon Mar 3 20:41:51 EST 2014
>
> I get the error on all of these.
> There is no output in dmesg.
>
> I was running these tests on ext4 filesystems:
> (for the Ubuntu kernels)
> /dev/mapper/stross--vg-root on / type ext4 (rw,errors=remount-ro)
> (for the stock kernel, in the virtualbox)
> /dev/sda1 on / type ext4 (rw,errors=remount-ro)
>
> Please let me know if you need any more information.
>
> FWIW, I did find this bug while googling, but it was on older kernels and
> was allegedly fixed:
> https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1101666
>
>
>

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: abuse-watchman.py --]
[-- Type: text/x-python; name="abuse-watchman.py", Size: 4319 bytes --]

#!/usr/bin/python

from atomicinteger import AtomicInteger
from json import loads, dumps
from random import random
from subprocess import call, check_output
from tempfile import mkdtemp
from time import sleep, time

import os
import socket
import stat
import threading

#from https://github.com/littlehedgehog/base/blob/master/atomicinteger.py
class AtomicInteger:
    def __init__(self, integer = 0):
        self.counter = integer
        self.lock = threading.RLock()
        return

    def increase(self, inc = 1):
        self.lock.acquire()
        self.counter = self.counter + inc
        self.lock.release()
        return

    def decrease(self, dec = 1):
        self.lock.acquire()
        self.counter = self.counter - dec
        self.lock.release()
        return
    
    def get(self):
        return self.counter


def get_sockname():
    result = check_output(["watchman", "get-sockname"])
    result = loads(result)
    return result['sockname']

def connect():
    sockname = get_sockname()
    sock = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
    sock.connect(sockname)
    sock.setblocking(False)
    return sock

def watch(sock, directory):
    watch = ['watch', directory]
    sock.sendall(dumps(watch) + "\n")
    result = readline(sock)
    result = loads(result)
    if not result.get("watch"):
        print result

def readline(sock):
    message = []
    start = time()
    while True:
        elapsed = time() - start
        if elapsed > 5:
            print "We have been waiting a very long time for data from watchman. We have so far: %s" % "".join(message)
        try:
            data = sock.recv(1024, socket.MSG_DONTWAIT)
            if "\n" in data:
                message.append(data[:data.index("\n")])
                break
        except socket.error:
            pass
        sleep(0.001)
    return "".join(message)

def since(sock, directory, since="c:1:2:3:4"):
    expression = {'since' : since}
    watch = ["query", directory, expression]
    sock.sendall(dumps(watch) + "\n")
    message = readline(sock)
    result = loads(message)
    if "error" in result:
        print "Error in since: %s (since = %s)" % (result["error"], since)
    return result

def create(sock, directory=None):
    directory = mkdtemp(dir=directory)
    watch(sock, directory)
    return directory

def touch(directory):
    f = open(os.path.join(directory, "file-%s" % random()), "w")
    f.write("x")
    f.close()

def isdir(f):
    result = os.lstat(f)
    return stat.S_ISDIR(result.st_mode)

def recursive_rmdir(directory):
    for f in os.listdir(directory):
        qualified = os.path.join(directory, f)
        if isdir(qualified):
            recursive_rmdir(qualified)
        else:
            os.unlink(qualified)
    os.rmdir(directory)

nthreads = AtomicInteger()
runs = AtomicInteger()

def run():
    nthreads.increase()
    runs.increase()
    directory = None
    sock = None
    try:
        print "RUN: %d %d" % (runs.get(), nthreads.get())
        sock = connect()
        directory = create(sock)

        result = since(sock, directory)
        assert "files" not in result or len(result["files"]) == 0
        clock = result.get("clock")
        if not clock:
            print "Failed since: %s" % result
        assert clock

#this stanza is only necesary on unbuntu 3.11; on 3.15, it can be skipped
        touch(directory)
        result = since(sock, directory, clock)
        assert result["clock"] != clock
        clock = result["clock"]
        assert len(result["files"]) == 1

        sleep(0.1)

#ditto
        for i in range(5):
            touch(directory)
        result = since(sock, directory, clock)
        assert result["clock"] != clock
        if len(result["files"]) < 5:
            print result

    finally:
        if sock:
            sock.close()
        if directory:
            recursive_rmdir(directory)
        nthreads.decrease()

def threaded(target, *args, **kwargs):
        thread = threading.Thread(target=target, args=args, kwargs=kwargs)
        thread.start()

while True:
    if nthreads.get() < 15:
        threaded(run)
    else:
        sleep(0.1)

Re: PROBLEM: Inotify leaks file descriptors.

[David Turner]

[-- Attachment #1: Type: text/plain, Size: 2186 bytes --]

(script attached)
On Tue, March 4, 2014 2:09 pm, David Turner wrote:
> I apologize for the slightly convoluted reproduction steps here,
> but I was not easily able to find a simpler test case in the
> time that I had available.
>
> First, you'll need Facebook's watchman:
> https://github.com/facebook/watchman
>
> Build and install it.  Then run the attached Python script.
> After a few hundred lines, you'll start to see errors of the form
> inotify_init error: Too many open files.  That could just
> indicate that watchman is leaking, but I think that's not what's
> going on, because killing watchman does not fix the problem.
>
> To demonstrate, kill the python script, then kill watchman.
> Then run tail -f /etc/hosts.  You'll get "tail: inotify cannot be
> used, reverting to polling: Too many open files" (you may need to
> run a few tails to see the error).  In fact, the only way I have
> found to get back to normal is to reboot.
>
> I tried increasing the ulimit to 10000 (from the default 1024).
> The error still happens, but it seems to take a bit longer.
>
> I have tried on a couple of Ubuntu kernels:
>
> Linux version 3.11.0-17-generic (buildd@toyol) (gcc version 4.6.3
> (Ubuntu/Linaro 4.6.3-1ubuntu5) ) #31~precise1-Ubuntu SMP Tue Feb 4
> 21:25:43 UTC 2014
>
> And Ubuntu's 3.8.0-36-generic (it's not running right now so I can't give
> the full version).
>
> I've also tried a stock kernel built from source (in a virtualbox):
>
> Linux version 3.13.5 (dturner@dturner-virtualbox) (gcc version 4.8.1
> (Ubuntu/Linaro 4.8.1-10ubuntu8) ) #1 SMP Mon Mar 3 20:41:51 EST 2014
>
> I get the error on all of these.
> There is no output in dmesg.
>
> I was running these tests on ext4 filesystems:
> (for the Ubuntu kernels)
> /dev/mapper/stross--vg-root on / type ext4 (rw,errors=remount-ro)
> (for the stock kernel, in the virtualbox)
> /dev/sda1 on / type ext4 (rw,errors=remount-ro)
>
> Please let me know if you need any more information.
>
> FWIW, I did find this bug while googling, but it was on older kernels and
> was allegedly fixed:
> https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1101666
>
>
>

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: abuse-watchman.py --]
[-- Type: text/x-python; name="abuse-watchman.py", Size: 4319 bytes --]

#!/usr/bin/python

from atomicinteger import AtomicInteger
from json import loads, dumps
from random import random
from subprocess import call, check_output
from tempfile import mkdtemp
from time import sleep, time

import os
import socket
import stat
import threading

#from https://github.com/littlehedgehog/base/blob/master/atomicinteger.py
class AtomicInteger:
    def __init__(self, integer = 0):
        self.counter = integer
        self.lock = threading.RLock()
        return

    def increase(self, inc = 1):
        self.lock.acquire()
        self.counter = self.counter + inc
        self.lock.release()
        return

    def decrease(self, dec = 1):
        self.lock.acquire()
        self.counter = self.counter - dec
        self.lock.release()
        return
    
    def get(self):
        return self.counter


def get_sockname():
    result = check_output(["watchman", "get-sockname"])
    result = loads(result)
    return result['sockname']

def connect():
    sockname = get_sockname()
    sock = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
    sock.connect(sockname)
    sock.setblocking(False)
    return sock

def watch(sock, directory):
    watch = ['watch', directory]
    sock.sendall(dumps(watch) + "\n")
    result = readline(sock)
    result = loads(result)
    if not result.get("watch"):
        print result

def readline(sock):
    message = []
    start = time()
    while True:
        elapsed = time() - start
        if elapsed > 5:
            print "We have been waiting a very long time for data from watchman. We have so far: %s" % "".join(message)
        try:
            data = sock.recv(1024, socket.MSG_DONTWAIT)
            if "\n" in data:
                message.append(data[:data.index("\n")])
                break
        except socket.error:
            pass
        sleep(0.001)
    return "".join(message)

def since(sock, directory, since="c:1:2:3:4"):
    expression = {'since' : since}
    watch = ["query", directory, expression]
    sock.sendall(dumps(watch) + "\n")
    message = readline(sock)
    result = loads(message)
    if "error" in result:
        print "Error in since: %s (since = %s)" % (result["error"], since)
    return result

def create(sock, directory=None):
    directory = mkdtemp(dir=directory)
    watch(sock, directory)
    return directory

def touch(directory):
    f = open(os.path.join(directory, "file-%s" % random()), "w")
    f.write("x")
    f.close()

def isdir(f):
    result = os.lstat(f)
    return stat.S_ISDIR(result.st_mode)

def recursive_rmdir(directory):
    for f in os.listdir(directory):
        qualified = os.path.join(directory, f)
        if isdir(qualified):
            recursive_rmdir(qualified)
        else:
            os.unlink(qualified)
    os.rmdir(directory)

nthreads = AtomicInteger()
runs = AtomicInteger()

def run():
    nthreads.increase()
    runs.increase()
    directory = None
    sock = None
    try:
        print "RUN: %d %d" % (runs.get(), nthreads.get())
        sock = connect()
        directory = create(sock)

        result = since(sock, directory)
        assert "files" not in result or len(result["files"]) == 0
        clock = result.get("clock")
        if not clock:
            print "Failed since: %s" % result
        assert clock

#this stanza is only necesary on unbuntu 3.11; on 3.15, it can be skipped
        touch(directory)
        result = since(sock, directory, clock)
        assert result["clock"] != clock
        clock = result["clock"]
        assert len(result["files"]) == 1

        sleep(0.1)

#ditto
        for i in range(5):
            touch(directory)
        result = since(sock, directory, clock)
        assert result["clock"] != clock
        if len(result["files"]) < 5:
            print result

    finally:
        if sock:
            sock.close()
        if directory:
            recursive_rmdir(directory)
        nthreads.decrease()

def threaded(target, *args, **kwargs):
        thread = threading.Thread(target=target, args=args, kwargs=kwargs)
        thread.start()

while True:
    if nthreads.get() < 15:
        threaded(run)
    else:
        sleep(0.1)

Re: PROBLEM: Inotify leaks file descriptors.

[David Turner]

I was running code related to the python script -- multiple threads doing
inotify things in parallel, using watchman, and I got the following Oops:

[152513.914195] watchman[4963]: segfault at 7ff04ddb09d0 ip
00007ff05b831f60 sp 00007ff04c5acce8 error 4 in
libpthread-2.15.so[7ff05b825000+18000]
[152516.577861] watchman[6138]: segfault at 7f1962e099d0 ip
00007f1970489f60 sp 00007f1962406ce8 error 4 in
libpthread-2.15.so[7f197047d000+18000]
[153010.703990] BUG: unable to handle kernel NULL pointer dereference at  
        (null)
[153010.704036] IP: [<          (null)>]           (null)
[153010.704060] PGD 1b1b4e067 PUD 1cc1f1067 PMD 0
[153010.704084] Oops: 0010 [#1] SMP
[153010.704103] Modules linked in: btrfs raid6_pq zlib_deflate xor ufs
qnx4 hfsplus hfs minix ntfs msdos jfs xfs reiserfs usb_storage cdc_acm
joydev pci_stub vboxpci(OF) vboxnetadp(OF) vboxnetflt(OF) vboxdrv(OF) bnep
rfcomm bluetooth parport_pc ppdev uvcvideo videobuf2_core binfmt_misc
videodev snd_hda_codec_hdmi snd_hda_codec_conexant videobuf2_vmalloc
videobuf2_memops snd_hda_intel snd_hda_codec snd_hwdep snd_pcm
snd_seq_midi arc4 snd_rawmidi iwldvm mac80211 snd_seq_midi_event snd_seq
psmouse thinkpad_acpi snd_timer snd_seq_device iwlwifi nvram serio_raw snd
tpm_tis cfg80211 soundcore snd_page_alloc mac_hid mei_me mei lpc_ich lp
ext2 parport dm_crypt i915 drm_kms_helper e1000e wmi drm ptp pps_core ahci
libahci sdhci_pci sdhci i2c_algo_bit video
[153010.704453] CPU: 1 PID: 3586 Comm: watchman Tainted: GF       W  O
3.11.0-17-generic #31~precise1-Ubuntu
[153010.704493] Hardware name: LENOVO 4177Q5U/4177Q5U, BIOS 83ET76WW (1.46
) 07/05/2013
[153010.704529] task: ffff8801b6ae0000 ti: ffff88009ed30000 task.ti:
ffff88009ed30000
[153010.704564] RIP: 0010:[<0000000000000000>]  [<          (null)>]      
    (null)
[153010.704600] RSP: 0018:ffff88009ed31dc0  EFLAGS: 00010246
[153010.704624] RAX: 00000000b98ab901 RBX: ffff88015a9b5228 RCX:
00000000000188d0
[153010.704655] RDX: 000000000000b98a RSI: ffff880100b95c00 RDI:
ffff88015a9b5228
[153010.704686] RBP: ffff88009ed31dd8 R08: 0000000000000001 R09:
ffffea0000d85640
[153010.704718] R10: ffffffff811f9628 R11: 0000000000000000 R12:
ffff88015a9b5228
[153010.704750] R13: ffff880100b95ca0 R14: 00000000ffffffff R15:
ffff880100b95c00
[153010.704783] FS:  00007f5b19087700(0000) GS:ffff88021e240000(0000)
knlGS:0000000000000000
[153010.704819] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[153010.704846] CR2: 0000000000000000 CR3: 00000001dd9d1000 CR4:
00000000000427e0
[153010.704877] Stack:
[153010.704888]  ffffffff811f7120 ffff88015a9b5228 ffff8801e460c7f0
ffff88009ed31e28
[153010.704926]  ffffffff811f7847 ffff88009ed31e18 ffff880100b95c70
ffff88009ed31f50
[153010.704965]  ffff880100b95c00 0000000000000010 ffff8802120433c0
ffff8802120433c0
[153010.705005] Call Trace:
[153010.705024]  [<ffffffff811f7120>] ? fsnotify_put_mark+0x30/0x40
[153010.705054]  [<ffffffff811f7847>]
fsnotify_clear_marks_by_group_flags+0x87/0xb0
[153010.705088]  [<ffffffff811f7883>] fsnotify_clear_marks_by_group+0x13/0x20
[153010.705119]  [<ffffffff811f68d6>] fsnotify_destroy_group+0x16/0x40
[153010.705150]  [<ffffffff811f8be6>] inotify_release+0x26/0x50
[153010.705177]  [<ffffffff811b640a>] __fput+0xba/0x240
[153010.705201]  [<ffffffff811b65de>] ____fput+0xe/0x10
[153010.705226]  [<ffffffff810859d8>] task_work_run+0xc8/0xf0
[153010.706464]  [<ffffffff81013dfc>] do_notify_resume+0xac/0xc0
[153010.707730]  [<ffffffff8175159a>] int_signal+0x12/0x17
[153010.708933] Code:  Bad RIP value.
[153010.710089] RIP  [<          (null)>]           (null)
[153010.711251]  RSP <ffff88009ed31dc0>
[153010.712319] CR2: 0000000000000000
[153010.718669] ---[ end trace 17ed2927fe522cd1 ]---

Re: PROBLEM: Inotify leaks file descriptors.

[Jan Kara]

On Tue 04-03-14 14:09:18, David Turner wrote:
> I apologize for the slightly convoluted reproduction steps here,
> but I was not easily able to find a simpler test case in the
> time that I had available.
> 
> First, you'll need Facebook's watchman:
> https://github.com/facebook/watchman
> 
> Build and install it.  Then run the attached Python script.
> After a few hundred lines, you'll start to see errors of the form
> inotify_init error: Too many open files.  That could just
> indicate that watchman is leaking, but I think that's not what's
> going on, because killing watchman does not fix the problem.
  Since opening other files clearly works, what is likely leaking somewhere
is 'inotify_devs' counter - that's a counter of inotify instances per user.
And that leak is likely happening because we leak a fsnotify group
reference count somewhere. Why that happens isn't clear to me but the
refcounting isn't quite simple so some bug seems possible. I guess I'll try
to reproduce this and see.

								Hpnza

> To demonstrate, kill the python script, then kill watchman.
> Then run tail -f /etc/hosts.  You'll get "tail: inotify cannot be
> used, reverting to polling: Too many open files" (you may need to
> run a few tails to see the error).  In fact, the only way I have
> found to get back to normal is to reboot.
> 
> I tried increasing the ulimit to 10000 (from the default 1024).
> The error still happens, but it seems to take a bit longer.
> 
> I have tried on a couple of Ubuntu kernels:
> 
> Linux version 3.11.0-17-generic (buildd@toyol) (gcc version 4.6.3
> (Ubuntu/Linaro 4.6.3-1ubuntu5) ) #31~precise1-Ubuntu SMP Tue Feb 4
> 21:25:43 UTC 2014
> 
> And Ubuntu's 3.8.0-36-generic (it's not running right now so I can't give
> the full version).
> 
> I've also tried a stock kernel built from source (in a virtualbox):
> 
> Linux version 3.13.5 (dturner@dturner-virtualbox) (gcc version 4.8.1
> (Ubuntu/Linaro 4.8.1-10ubuntu8) ) #1 SMP Mon Mar 3 20:41:51 EST 2014
> 
> I get the error on all of these.
> There is no output in dmesg.
> 
> I was running these tests on ext4 filesystems:
> (for the Ubuntu kernels)
> /dev/mapper/stross--vg-root on / type ext4 (rw,errors=remount-ro)
> (for the stock kernel, in the virtualbox)
> /dev/sda1 on / type ext4 (rw,errors=remount-ro)
> 
> Please let me know if you need any more information.
> 
> FWIW, I did find this bug while googling, but it was on older kernels and
> was allegedly fixed:
> https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1101666
> 
> 
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
-- 
Jan Kara <jack@suse.cz>
SUSE Labs, CR

Re: PROBLEM: Inotify leaks file descriptors.

[Jan Kara]

On Tue 04-03-14 14:18:08, David Turner wrote:
> (script attached)
> On Tue, March 4, 2014 2:09 pm, David Turner wrote:
> > I apologize for the slightly convoluted reproduction steps here,
> > but I was not easily able to find a simpler test case in the
> > time that I had available.
> >
> > First, you'll need Facebook's watchman:
> > https://github.com/facebook/watchman
> >
> > Build and install it.  Then run the attached Python script.
> > After a few hundred lines, you'll start to see errors of the form
> > inotify_init error: Too many open files.  That could just
> > indicate that watchman is leaking, but I think that's not what's
> > going on, because killing watchman does not fix the problem.
> >
> > To demonstrate, kill the python script, then kill watchman.
> > Then run tail -f /etc/hosts.  You'll get "tail: inotify cannot be
> > used, reverting to polling: Too many open files" (you may need to
> > run a few tails to see the error).  In fact, the only way I have
> > found to get back to normal is to reboot.
> >
> > I tried increasing the ulimit to 10000 (from the default 1024).
> > The error still happens, but it seems to take a bit longer.
> >
> > I have tried on a couple of Ubuntu kernels:
> >
> > Linux version 3.11.0-17-generic (buildd@toyol) (gcc version 4.6.3
> > (Ubuntu/Linaro 4.6.3-1ubuntu5) ) #31~precise1-Ubuntu SMP Tue Feb 4
> > 21:25:43 UTC 2014
> >
> > And Ubuntu's 3.8.0-36-generic (it's not running right now so I can't give
> > the full version).
> >
> > I've also tried a stock kernel built from source (in a virtualbox):
> >
> > Linux version 3.13.5 (dturner@dturner-virtualbox) (gcc version 4.8.1
> > (Ubuntu/Linaro 4.8.1-10ubuntu8) ) #1 SMP Mon Mar 3 20:41:51 EST 2014
> >
> > I get the error on all of these.
> > There is no output in dmesg.
> >
> > I was running these tests on ext4 filesystems:
> > (for the Ubuntu kernels)
> > /dev/mapper/stross--vg-root on / type ext4 (rw,errors=remount-ro)
> > (for the stock kernel, in the virtualbox)
> > /dev/sda1 on / type ext4 (rw,errors=remount-ro)
> >
> > Please let me know if you need any more information.
> >
> > FWIW, I did find this bug while googling, but it was on older kernels and
> > was allegedly fixed:
> > https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1101666
> >
> >
> >
  So when I run your script on 3.14-rc5, I don't see any problems. So maybe
the problem really got fixed? BTW, I'll note that
/proc/sys/fs/inotify/max_user_instances is 128 on my system. Not sure what
it is on your system...

								Honza

> #!/usr/bin/python
> 
> from atomicinteger import AtomicInteger
> from json import loads, dumps
> from random import random
> from subprocess import call, check_output
> from tempfile import mkdtemp
> from time import sleep, time
> 
> import os
> import socket
> import stat
> import threading
> 
> #from https://github.com/littlehedgehog/base/blob/master/atomicinteger.py
> class AtomicInteger:
>     def __init__(self, integer = 0):
>         self.counter = integer
>         self.lock = threading.RLock()
>         return
> 
>     def increase(self, inc = 1):
>         self.lock.acquire()
>         self.counter = self.counter + inc
>         self.lock.release()
>         return
> 
>     def decrease(self, dec = 1):
>         self.lock.acquire()
>         self.counter = self.counter - dec
>         self.lock.release()
>         return
>     
>     def get(self):
>         return self.counter
> 
> 
> def get_sockname():
>     result = check_output(["watchman", "get-sockname"])
>     result = loads(result)
>     return result['sockname']
> 
> def connect():
>     sockname = get_sockname()
>     sock = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
>     sock.connect(sockname)
>     sock.setblocking(False)
>     return sock
> 
> def watch(sock, directory):
>     watch = ['watch', directory]
>     sock.sendall(dumps(watch) + "\n")
>     result = readline(sock)
>     result = loads(result)
>     if not result.get("watch"):
>         print result
> 
> def readline(sock):
>     message = []
>     start = time()
>     while True:
>         elapsed = time() - start
>         if elapsed > 5:
>             print "We have been waiting a very long time for data from watchman. We have so far: %s" % "".join(message)
>         try:
>             data = sock.recv(1024, socket.MSG_DONTWAIT)
>             if "\n" in data:
>                 message.append(data[:data.index("\n")])
>                 break
>         except socket.error:
>             pass
>         sleep(0.001)
>     return "".join(message)
> 
> def since(sock, directory, since="c:1:2:3:4"):
>     expression = {'since' : since}
>     watch = ["query", directory, expression]
>     sock.sendall(dumps(watch) + "\n")
>     message = readline(sock)
>     result = loads(message)
>     if "error" in result:
>         print "Error in since: %s (since = %s)" % (result["error"], since)
>     return result
> 
> def create(sock, directory=None):
>     directory = mkdtemp(dir=directory)
>     watch(sock, directory)
>     return directory
> 
> def touch(directory):
>     f = open(os.path.join(directory, "file-%s" % random()), "w")
>     f.write("x")
>     f.close()
> 
> def isdir(f):
>     result = os.lstat(f)
>     return stat.S_ISDIR(result.st_mode)
> 
> def recursive_rmdir(directory):
>     for f in os.listdir(directory):
>         qualified = os.path.join(directory, f)
>         if isdir(qualified):
>             recursive_rmdir(qualified)
>         else:
>             os.unlink(qualified)
>     os.rmdir(directory)
> 
> nthreads = AtomicInteger()
> runs = AtomicInteger()
> 
> def run():
>     nthreads.increase()
>     runs.increase()
>     directory = None
>     sock = None
>     try:
>         print "RUN: %d %d" % (runs.get(), nthreads.get())
>         sock = connect()
>         directory = create(sock)
> 
>         result = since(sock, directory)
>         assert "files" not in result or len(result["files"]) == 0
>         clock = result.get("clock")
>         if not clock:
>             print "Failed since: %s" % result
>         assert clock
> 
> #this stanza is only necesary on unbuntu 3.11; on 3.15, it can be skipped
>         touch(directory)
>         result = since(sock, directory, clock)
>         assert result["clock"] != clock
>         clock = result["clock"]
>         assert len(result["files"]) == 1
> 
>         sleep(0.1)
> 
> #ditto
>         for i in range(5):
>             touch(directory)
>         result = since(sock, directory, clock)
>         assert result["clock"] != clock
>         if len(result["files"]) < 5:
>             print result
> 
>     finally:
>         if sock:
>             sock.close()
>         if directory:
>             recursive_rmdir(directory)
>         nthreads.decrease()
> 
> def threaded(target, *args, **kwargs):
>         thread = threading.Thread(target=target, args=args, kwargs=kwargs)
>         thread.start()
> 
> while True:
>     if nthreads.get() < 15:
>         threaded(run)
>     else:
>         sleep(0.1)
-- 
Jan Kara <jack@suse.cz>
SUSE Labs, CR