* Re: PROBLEM: Inotify leaks file descriptors.
2014-03-04 19:09 PROBLEM: Inotify leaks file descriptors David Turner
@ 2014-03-04 19:18 ` David Turner
2014-03-07 5:59 ` David Turner
2014-03-19 13:16 ` Jan Kara
2014-03-04 19:18 ` David Turner
2014-03-11 21:42 ` Jan Kara
2 siblings, 2 replies; 6+ messages in thread
From: David Turner @ 2014-03-04 19:18 UTC (permalink / raw)
To: David Turner; +Cc: John McCutchan, Robert Love, Eric Paris, linux-kernel
[-- Attachment #1: Type: text/plain, Size: 2186 bytes --]
(script attached)
On Tue, March 4, 2014 2:09 pm, David Turner wrote:
> I apologize for the slightly convoluted reproduction steps here,
> but I was not easily able to find a simpler test case in the
> time that I had available.
>
> First, you'll need Facebook's watchman:
> https://github.com/facebook/watchman
>
> Build and install it. Then run the attached Python script.
> After a few hundred lines, you'll start to see errors of the form
> inotify_init error: Too many open files. That could just
> indicate that watchman is leaking, but I think that's not what's
> going on, because killing watchman does not fix the problem.
>
> To demonstrate, kill the python script, then kill watchman.
> Then run tail -f /etc/hosts. You'll get "tail: inotify cannot be
> used, reverting to polling: Too many open files" (you may need to
> run a few tails to see the error). In fact, the only way I have
> found to get back to normal is to reboot.
>
> I tried increasing the ulimit to 10000 (from the default 1024).
> The error still happens, but it seems to take a bit longer.
>
> I have tried on a couple of Ubuntu kernels:
>
> Linux version 3.11.0-17-generic (buildd@toyol) (gcc version 4.6.3
> (Ubuntu/Linaro 4.6.3-1ubuntu5) ) #31~precise1-Ubuntu SMP Tue Feb 4
> 21:25:43 UTC 2014
>
> And Ubuntu's 3.8.0-36-generic (it's not running right now so I can't give
> the full version).
>
> I've also tried a stock kernel built from source (in a virtualbox):
>
> Linux version 3.13.5 (dturner@dturner-virtualbox) (gcc version 4.8.1
> (Ubuntu/Linaro 4.8.1-10ubuntu8) ) #1 SMP Mon Mar 3 20:41:51 EST 2014
>
> I get the error on all of these.
> There is no output in dmesg.
>
> I was running these tests on ext4 filesystems:
> (for the Ubuntu kernels)
> /dev/mapper/stross--vg-root on / type ext4 (rw,errors=remount-ro)
> (for the stock kernel, in the virtualbox)
> /dev/sda1 on / type ext4 (rw,errors=remount-ro)
>
> Please let me know if you need any more information.
>
> FWIW, I did find this bug while googling, but it was on older kernels and
> was allegedly fixed:
> https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1101666
>
>
>
[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: abuse-watchman.py --]
[-- Type: text/x-python; name="abuse-watchman.py", Size: 4319 bytes --]
#!/usr/bin/python
from atomicinteger import AtomicInteger
from json import loads, dumps
from random import random
from subprocess import call, check_output
from tempfile import mkdtemp
from time import sleep, time
import os
import socket
import stat
import threading
#from https://github.com/littlehedgehog/base/blob/master/atomicinteger.py
class AtomicInteger:
def __init__(self, integer = 0):
self.counter = integer
self.lock = threading.RLock()
return
def increase(self, inc = 1):
self.lock.acquire()
self.counter = self.counter + inc
self.lock.release()
return
def decrease(self, dec = 1):
self.lock.acquire()
self.counter = self.counter - dec
self.lock.release()
return
def get(self):
return self.counter
def get_sockname():
result = check_output(["watchman", "get-sockname"])
result = loads(result)
return result['sockname']
def connect():
sockname = get_sockname()
sock = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
sock.connect(sockname)
sock.setblocking(False)
return sock
def watch(sock, directory):
watch = ['watch', directory]
sock.sendall(dumps(watch) + "\n")
result = readline(sock)
result = loads(result)
if not result.get("watch"):
print result
def readline(sock):
message = []
start = time()
while True:
elapsed = time() - start
if elapsed > 5:
print "We have been waiting a very long time for data from watchman. We have so far: %s" % "".join(message)
try:
data = sock.recv(1024, socket.MSG_DONTWAIT)
if "\n" in data:
message.append(data[:data.index("\n")])
break
except socket.error:
pass
sleep(0.001)
return "".join(message)
def since(sock, directory, since="c:1:2:3:4"):
expression = {'since' : since}
watch = ["query", directory, expression]
sock.sendall(dumps(watch) + "\n")
message = readline(sock)
result = loads(message)
if "error" in result:
print "Error in since: %s (since = %s)" % (result["error"], since)
return result
def create(sock, directory=None):
directory = mkdtemp(dir=directory)
watch(sock, directory)
return directory
def touch(directory):
f = open(os.path.join(directory, "file-%s" % random()), "w")
f.write("x")
f.close()
def isdir(f):
result = os.lstat(f)
return stat.S_ISDIR(result.st_mode)
def recursive_rmdir(directory):
for f in os.listdir(directory):
qualified = os.path.join(directory, f)
if isdir(qualified):
recursive_rmdir(qualified)
else:
os.unlink(qualified)
os.rmdir(directory)
nthreads = AtomicInteger()
runs = AtomicInteger()
def run():
nthreads.increase()
runs.increase()
directory = None
sock = None
try:
print "RUN: %d %d" % (runs.get(), nthreads.get())
sock = connect()
directory = create(sock)
result = since(sock, directory)
assert "files" not in result or len(result["files"]) == 0
clock = result.get("clock")
if not clock:
print "Failed since: %s" % result
assert clock
#this stanza is only necesary on unbuntu 3.11; on 3.15, it can be skipped
touch(directory)
result = since(sock, directory, clock)
assert result["clock"] != clock
clock = result["clock"]
assert len(result["files"]) == 1
sleep(0.1)
#ditto
for i in range(5):
touch(directory)
result = since(sock, directory, clock)
assert result["clock"] != clock
if len(result["files"]) < 5:
print result
finally:
if sock:
sock.close()
if directory:
recursive_rmdir(directory)
nthreads.decrease()
def threaded(target, *args, **kwargs):
thread = threading.Thread(target=target, args=args, kwargs=kwargs)
thread.start()
while True:
if nthreads.get() < 15:
threaded(run)
else:
sleep(0.1)
^ permalink raw reply [flat|nested] 6+ messages in thread* Re: PROBLEM: Inotify leaks file descriptors.
2014-03-04 19:18 ` David Turner
@ 2014-03-07 5:59 ` David Turner
2014-03-19 13:16 ` Jan Kara
1 sibling, 0 replies; 6+ messages in thread
From: David Turner @ 2014-03-07 5:59 UTC (permalink / raw)
To: David Turner
Cc: David Turner, John McCutchan, Robert Love, Eric Paris, linux-kernel
I was running code related to the python script -- multiple threads doing
inotify things in parallel, using watchman, and I got the following Oops:
[152513.914195] watchman[4963]: segfault at 7ff04ddb09d0 ip
00007ff05b831f60 sp 00007ff04c5acce8 error 4 in
libpthread-2.15.so[7ff05b825000+18000]
[152516.577861] watchman[6138]: segfault at 7f1962e099d0 ip
00007f1970489f60 sp 00007f1962406ce8 error 4 in
libpthread-2.15.so[7f197047d000+18000]
[153010.703990] BUG: unable to handle kernel NULL pointer dereference at
(null)
[153010.704036] IP: [< (null)>] (null)
[153010.704060] PGD 1b1b4e067 PUD 1cc1f1067 PMD 0
[153010.704084] Oops: 0010 [#1] SMP
[153010.704103] Modules linked in: btrfs raid6_pq zlib_deflate xor ufs
qnx4 hfsplus hfs minix ntfs msdos jfs xfs reiserfs usb_storage cdc_acm
joydev pci_stub vboxpci(OF) vboxnetadp(OF) vboxnetflt(OF) vboxdrv(OF) bnep
rfcomm bluetooth parport_pc ppdev uvcvideo videobuf2_core binfmt_misc
videodev snd_hda_codec_hdmi snd_hda_codec_conexant videobuf2_vmalloc
videobuf2_memops snd_hda_intel snd_hda_codec snd_hwdep snd_pcm
snd_seq_midi arc4 snd_rawmidi iwldvm mac80211 snd_seq_midi_event snd_seq
psmouse thinkpad_acpi snd_timer snd_seq_device iwlwifi nvram serio_raw snd
tpm_tis cfg80211 soundcore snd_page_alloc mac_hid mei_me mei lpc_ich lp
ext2 parport dm_crypt i915 drm_kms_helper e1000e wmi drm ptp pps_core ahci
libahci sdhci_pci sdhci i2c_algo_bit video
[153010.704453] CPU: 1 PID: 3586 Comm: watchman Tainted: GF W O
3.11.0-17-generic #31~precise1-Ubuntu
[153010.704493] Hardware name: LENOVO 4177Q5U/4177Q5U, BIOS 83ET76WW (1.46
) 07/05/2013
[153010.704529] task: ffff8801b6ae0000 ti: ffff88009ed30000 task.ti:
ffff88009ed30000
[153010.704564] RIP: 0010:[<0000000000000000>] [< (null)>]
(null)
[153010.704600] RSP: 0018:ffff88009ed31dc0 EFLAGS: 00010246
[153010.704624] RAX: 00000000b98ab901 RBX: ffff88015a9b5228 RCX:
00000000000188d0
[153010.704655] RDX: 000000000000b98a RSI: ffff880100b95c00 RDI:
ffff88015a9b5228
[153010.704686] RBP: ffff88009ed31dd8 R08: 0000000000000001 R09:
ffffea0000d85640
[153010.704718] R10: ffffffff811f9628 R11: 0000000000000000 R12:
ffff88015a9b5228
[153010.704750] R13: ffff880100b95ca0 R14: 00000000ffffffff R15:
ffff880100b95c00
[153010.704783] FS: 00007f5b19087700(0000) GS:ffff88021e240000(0000)
knlGS:0000000000000000
[153010.704819] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[153010.704846] CR2: 0000000000000000 CR3: 00000001dd9d1000 CR4:
00000000000427e0
[153010.704877] Stack:
[153010.704888] ffffffff811f7120 ffff88015a9b5228 ffff8801e460c7f0
ffff88009ed31e28
[153010.704926] ffffffff811f7847 ffff88009ed31e18 ffff880100b95c70
ffff88009ed31f50
[153010.704965] ffff880100b95c00 0000000000000010 ffff8802120433c0
ffff8802120433c0
[153010.705005] Call Trace:
[153010.705024] [<ffffffff811f7120>] ? fsnotify_put_mark+0x30/0x40
[153010.705054] [<ffffffff811f7847>]
fsnotify_clear_marks_by_group_flags+0x87/0xb0
[153010.705088] [<ffffffff811f7883>] fsnotify_clear_marks_by_group+0x13/0x20
[153010.705119] [<ffffffff811f68d6>] fsnotify_destroy_group+0x16/0x40
[153010.705150] [<ffffffff811f8be6>] inotify_release+0x26/0x50
[153010.705177] [<ffffffff811b640a>] __fput+0xba/0x240
[153010.705201] [<ffffffff811b65de>] ____fput+0xe/0x10
[153010.705226] [<ffffffff810859d8>] task_work_run+0xc8/0xf0
[153010.706464] [<ffffffff81013dfc>] do_notify_resume+0xac/0xc0
[153010.707730] [<ffffffff8175159a>] int_signal+0x12/0x17
[153010.708933] Code: Bad RIP value.
[153010.710089] RIP [< (null)>] (null)
[153010.711251] RSP <ffff88009ed31dc0>
[153010.712319] CR2: 0000000000000000
[153010.718669] ---[ end trace 17ed2927fe522cd1 ]---
^ permalink raw reply [flat|nested] 6+ messages in thread* Re: PROBLEM: Inotify leaks file descriptors.
2014-03-04 19:18 ` David Turner
2014-03-07 5:59 ` David Turner
@ 2014-03-19 13:16 ` Jan Kara
1 sibling, 0 replies; 6+ messages in thread
From: Jan Kara @ 2014-03-19 13:16 UTC (permalink / raw)
To: David Turner; +Cc: John McCutchan, Robert Love, Eric Paris, linux-kernel
On Tue 04-03-14 14:18:08, David Turner wrote:
> (script attached)
> On Tue, March 4, 2014 2:09 pm, David Turner wrote:
> > I apologize for the slightly convoluted reproduction steps here,
> > but I was not easily able to find a simpler test case in the
> > time that I had available.
> >
> > First, you'll need Facebook's watchman:
> > https://github.com/facebook/watchman
> >
> > Build and install it. Then run the attached Python script.
> > After a few hundred lines, you'll start to see errors of the form
> > inotify_init error: Too many open files. That could just
> > indicate that watchman is leaking, but I think that's not what's
> > going on, because killing watchman does not fix the problem.
> >
> > To demonstrate, kill the python script, then kill watchman.
> > Then run tail -f /etc/hosts. You'll get "tail: inotify cannot be
> > used, reverting to polling: Too many open files" (you may need to
> > run a few tails to see the error). In fact, the only way I have
> > found to get back to normal is to reboot.
> >
> > I tried increasing the ulimit to 10000 (from the default 1024).
> > The error still happens, but it seems to take a bit longer.
> >
> > I have tried on a couple of Ubuntu kernels:
> >
> > Linux version 3.11.0-17-generic (buildd@toyol) (gcc version 4.6.3
> > (Ubuntu/Linaro 4.6.3-1ubuntu5) ) #31~precise1-Ubuntu SMP Tue Feb 4
> > 21:25:43 UTC 2014
> >
> > And Ubuntu's 3.8.0-36-generic (it's not running right now so I can't give
> > the full version).
> >
> > I've also tried a stock kernel built from source (in a virtualbox):
> >
> > Linux version 3.13.5 (dturner@dturner-virtualbox) (gcc version 4.8.1
> > (Ubuntu/Linaro 4.8.1-10ubuntu8) ) #1 SMP Mon Mar 3 20:41:51 EST 2014
> >
> > I get the error on all of these.
> > There is no output in dmesg.
> >
> > I was running these tests on ext4 filesystems:
> > (for the Ubuntu kernels)
> > /dev/mapper/stross--vg-root on / type ext4 (rw,errors=remount-ro)
> > (for the stock kernel, in the virtualbox)
> > /dev/sda1 on / type ext4 (rw,errors=remount-ro)
> >
> > Please let me know if you need any more information.
> >
> > FWIW, I did find this bug while googling, but it was on older kernels and
> > was allegedly fixed:
> > https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1101666
> >
> >
> >
So when I run your script on 3.14-rc5, I don't see any problems. So maybe
the problem really got fixed? BTW, I'll note that
/proc/sys/fs/inotify/max_user_instances is 128 on my system. Not sure what
it is on your system...
Honza
> #!/usr/bin/python
>
> from atomicinteger import AtomicInteger
> from json import loads, dumps
> from random import random
> from subprocess import call, check_output
> from tempfile import mkdtemp
> from time import sleep, time
>
> import os
> import socket
> import stat
> import threading
>
> #from https://github.com/littlehedgehog/base/blob/master/atomicinteger.py
> class AtomicInteger:
> def __init__(self, integer = 0):
> self.counter = integer
> self.lock = threading.RLock()
> return
>
> def increase(self, inc = 1):
> self.lock.acquire()
> self.counter = self.counter + inc
> self.lock.release()
> return
>
> def decrease(self, dec = 1):
> self.lock.acquire()
> self.counter = self.counter - dec
> self.lock.release()
> return
>
> def get(self):
> return self.counter
>
>
> def get_sockname():
> result = check_output(["watchman", "get-sockname"])
> result = loads(result)
> return result['sockname']
>
> def connect():
> sockname = get_sockname()
> sock = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
> sock.connect(sockname)
> sock.setblocking(False)
> return sock
>
> def watch(sock, directory):
> watch = ['watch', directory]
> sock.sendall(dumps(watch) + "\n")
> result = readline(sock)
> result = loads(result)
> if not result.get("watch"):
> print result
>
> def readline(sock):
> message = []
> start = time()
> while True:
> elapsed = time() - start
> if elapsed > 5:
> print "We have been waiting a very long time for data from watchman. We have so far: %s" % "".join(message)
> try:
> data = sock.recv(1024, socket.MSG_DONTWAIT)
> if "\n" in data:
> message.append(data[:data.index("\n")])
> break
> except socket.error:
> pass
> sleep(0.001)
> return "".join(message)
>
> def since(sock, directory, since="c:1:2:3:4"):
> expression = {'since' : since}
> watch = ["query", directory, expression]
> sock.sendall(dumps(watch) + "\n")
> message = readline(sock)
> result = loads(message)
> if "error" in result:
> print "Error in since: %s (since = %s)" % (result["error"], since)
> return result
>
> def create(sock, directory=None):
> directory = mkdtemp(dir=directory)
> watch(sock, directory)
> return directory
>
> def touch(directory):
> f = open(os.path.join(directory, "file-%s" % random()), "w")
> f.write("x")
> f.close()
>
> def isdir(f):
> result = os.lstat(f)
> return stat.S_ISDIR(result.st_mode)
>
> def recursive_rmdir(directory):
> for f in os.listdir(directory):
> qualified = os.path.join(directory, f)
> if isdir(qualified):
> recursive_rmdir(qualified)
> else:
> os.unlink(qualified)
> os.rmdir(directory)
>
> nthreads = AtomicInteger()
> runs = AtomicInteger()
>
> def run():
> nthreads.increase()
> runs.increase()
> directory = None
> sock = None
> try:
> print "RUN: %d %d" % (runs.get(), nthreads.get())
> sock = connect()
> directory = create(sock)
>
> result = since(sock, directory)
> assert "files" not in result or len(result["files"]) == 0
> clock = result.get("clock")
> if not clock:
> print "Failed since: %s" % result
> assert clock
>
> #this stanza is only necesary on unbuntu 3.11; on 3.15, it can be skipped
> touch(directory)
> result = since(sock, directory, clock)
> assert result["clock"] != clock
> clock = result["clock"]
> assert len(result["files"]) == 1
>
> sleep(0.1)
>
> #ditto
> for i in range(5):
> touch(directory)
> result = since(sock, directory, clock)
> assert result["clock"] != clock
> if len(result["files"]) < 5:
> print result
>
> finally:
> if sock:
> sock.close()
> if directory:
> recursive_rmdir(directory)
> nthreads.decrease()
>
> def threaded(target, *args, **kwargs):
> thread = threading.Thread(target=target, args=args, kwargs=kwargs)
> thread.start()
>
> while True:
> if nthreads.get() < 15:
> threaded(run)
> else:
> sleep(0.1)
--
Jan Kara <jack@suse.cz>
SUSE Labs, CR
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: PROBLEM: Inotify leaks file descriptors.
2014-03-04 19:09 PROBLEM: Inotify leaks file descriptors David Turner
2014-03-04 19:18 ` David Turner
@ 2014-03-04 19:18 ` David Turner
2014-03-11 21:42 ` Jan Kara
2 siblings, 0 replies; 6+ messages in thread
From: David Turner @ 2014-03-04 19:18 UTC (permalink / raw)
To: David Turner; +Cc: John McCutchan, Robert Love, Eric Paris, linux-kernel
[-- Attachment #1: Type: text/plain, Size: 2186 bytes --]
(script attached)
On Tue, March 4, 2014 2:09 pm, David Turner wrote:
> I apologize for the slightly convoluted reproduction steps here,
> but I was not easily able to find a simpler test case in the
> time that I had available.
>
> First, you'll need Facebook's watchman:
> https://github.com/facebook/watchman
>
> Build and install it. Then run the attached Python script.
> After a few hundred lines, you'll start to see errors of the form
> inotify_init error: Too many open files. That could just
> indicate that watchman is leaking, but I think that's not what's
> going on, because killing watchman does not fix the problem.
>
> To demonstrate, kill the python script, then kill watchman.
> Then run tail -f /etc/hosts. You'll get "tail: inotify cannot be
> used, reverting to polling: Too many open files" (you may need to
> run a few tails to see the error). In fact, the only way I have
> found to get back to normal is to reboot.
>
> I tried increasing the ulimit to 10000 (from the default 1024).
> The error still happens, but it seems to take a bit longer.
>
> I have tried on a couple of Ubuntu kernels:
>
> Linux version 3.11.0-17-generic (buildd@toyol) (gcc version 4.6.3
> (Ubuntu/Linaro 4.6.3-1ubuntu5) ) #31~precise1-Ubuntu SMP Tue Feb 4
> 21:25:43 UTC 2014
>
> And Ubuntu's 3.8.0-36-generic (it's not running right now so I can't give
> the full version).
>
> I've also tried a stock kernel built from source (in a virtualbox):
>
> Linux version 3.13.5 (dturner@dturner-virtualbox) (gcc version 4.8.1
> (Ubuntu/Linaro 4.8.1-10ubuntu8) ) #1 SMP Mon Mar 3 20:41:51 EST 2014
>
> I get the error on all of these.
> There is no output in dmesg.
>
> I was running these tests on ext4 filesystems:
> (for the Ubuntu kernels)
> /dev/mapper/stross--vg-root on / type ext4 (rw,errors=remount-ro)
> (for the stock kernel, in the virtualbox)
> /dev/sda1 on / type ext4 (rw,errors=remount-ro)
>
> Please let me know if you need any more information.
>
> FWIW, I did find this bug while googling, but it was on older kernels and
> was allegedly fixed:
> https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1101666
>
>
>
[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: abuse-watchman.py --]
[-- Type: text/x-python; name="abuse-watchman.py", Size: 4319 bytes --]
#!/usr/bin/python
from atomicinteger import AtomicInteger
from json import loads, dumps
from random import random
from subprocess import call, check_output
from tempfile import mkdtemp
from time import sleep, time
import os
import socket
import stat
import threading
#from https://github.com/littlehedgehog/base/blob/master/atomicinteger.py
class AtomicInteger:
def __init__(self, integer = 0):
self.counter = integer
self.lock = threading.RLock()
return
def increase(self, inc = 1):
self.lock.acquire()
self.counter = self.counter + inc
self.lock.release()
return
def decrease(self, dec = 1):
self.lock.acquire()
self.counter = self.counter - dec
self.lock.release()
return
def get(self):
return self.counter
def get_sockname():
result = check_output(["watchman", "get-sockname"])
result = loads(result)
return result['sockname']
def connect():
sockname = get_sockname()
sock = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
sock.connect(sockname)
sock.setblocking(False)
return sock
def watch(sock, directory):
watch = ['watch', directory]
sock.sendall(dumps(watch) + "\n")
result = readline(sock)
result = loads(result)
if not result.get("watch"):
print result
def readline(sock):
message = []
start = time()
while True:
elapsed = time() - start
if elapsed > 5:
print "We have been waiting a very long time for data from watchman. We have so far: %s" % "".join(message)
try:
data = sock.recv(1024, socket.MSG_DONTWAIT)
if "\n" in data:
message.append(data[:data.index("\n")])
break
except socket.error:
pass
sleep(0.001)
return "".join(message)
def since(sock, directory, since="c:1:2:3:4"):
expression = {'since' : since}
watch = ["query", directory, expression]
sock.sendall(dumps(watch) + "\n")
message = readline(sock)
result = loads(message)
if "error" in result:
print "Error in since: %s (since = %s)" % (result["error"], since)
return result
def create(sock, directory=None):
directory = mkdtemp(dir=directory)
watch(sock, directory)
return directory
def touch(directory):
f = open(os.path.join(directory, "file-%s" % random()), "w")
f.write("x")
f.close()
def isdir(f):
result = os.lstat(f)
return stat.S_ISDIR(result.st_mode)
def recursive_rmdir(directory):
for f in os.listdir(directory):
qualified = os.path.join(directory, f)
if isdir(qualified):
recursive_rmdir(qualified)
else:
os.unlink(qualified)
os.rmdir(directory)
nthreads = AtomicInteger()
runs = AtomicInteger()
def run():
nthreads.increase()
runs.increase()
directory = None
sock = None
try:
print "RUN: %d %d" % (runs.get(), nthreads.get())
sock = connect()
directory = create(sock)
result = since(sock, directory)
assert "files" not in result or len(result["files"]) == 0
clock = result.get("clock")
if not clock:
print "Failed since: %s" % result
assert clock
#this stanza is only necesary on unbuntu 3.11; on 3.15, it can be skipped
touch(directory)
result = since(sock, directory, clock)
assert result["clock"] != clock
clock = result["clock"]
assert len(result["files"]) == 1
sleep(0.1)
#ditto
for i in range(5):
touch(directory)
result = since(sock, directory, clock)
assert result["clock"] != clock
if len(result["files"]) < 5:
print result
finally:
if sock:
sock.close()
if directory:
recursive_rmdir(directory)
nthreads.decrease()
def threaded(target, *args, **kwargs):
thread = threading.Thread(target=target, args=args, kwargs=kwargs)
thread.start()
while True:
if nthreads.get() < 15:
threaded(run)
else:
sleep(0.1)
^ permalink raw reply [flat|nested] 6+ messages in thread* Re: PROBLEM: Inotify leaks file descriptors.
2014-03-04 19:09 PROBLEM: Inotify leaks file descriptors David Turner
2014-03-04 19:18 ` David Turner
2014-03-04 19:18 ` David Turner
@ 2014-03-11 21:42 ` Jan Kara
2 siblings, 0 replies; 6+ messages in thread
From: Jan Kara @ 2014-03-11 21:42 UTC (permalink / raw)
To: David Turner; +Cc: John McCutchan, Robert Love, Eric Paris, linux-kernel
On Tue 04-03-14 14:09:18, David Turner wrote:
> I apologize for the slightly convoluted reproduction steps here,
> but I was not easily able to find a simpler test case in the
> time that I had available.
>
> First, you'll need Facebook's watchman:
> https://github.com/facebook/watchman
>
> Build and install it. Then run the attached Python script.
> After a few hundred lines, you'll start to see errors of the form
> inotify_init error: Too many open files. That could just
> indicate that watchman is leaking, but I think that's not what's
> going on, because killing watchman does not fix the problem.
Since opening other files clearly works, what is likely leaking somewhere
is 'inotify_devs' counter - that's a counter of inotify instances per user.
And that leak is likely happening because we leak a fsnotify group
reference count somewhere. Why that happens isn't clear to me but the
refcounting isn't quite simple so some bug seems possible. I guess I'll try
to reproduce this and see.
Hpnza
> To demonstrate, kill the python script, then kill watchman.
> Then run tail -f /etc/hosts. You'll get "tail: inotify cannot be
> used, reverting to polling: Too many open files" (you may need to
> run a few tails to see the error). In fact, the only way I have
> found to get back to normal is to reboot.
>
> I tried increasing the ulimit to 10000 (from the default 1024).
> The error still happens, but it seems to take a bit longer.
>
> I have tried on a couple of Ubuntu kernels:
>
> Linux version 3.11.0-17-generic (buildd@toyol) (gcc version 4.6.3
> (Ubuntu/Linaro 4.6.3-1ubuntu5) ) #31~precise1-Ubuntu SMP Tue Feb 4
> 21:25:43 UTC 2014
>
> And Ubuntu's 3.8.0-36-generic (it's not running right now so I can't give
> the full version).
>
> I've also tried a stock kernel built from source (in a virtualbox):
>
> Linux version 3.13.5 (dturner@dturner-virtualbox) (gcc version 4.8.1
> (Ubuntu/Linaro 4.8.1-10ubuntu8) ) #1 SMP Mon Mar 3 20:41:51 EST 2014
>
> I get the error on all of these.
> There is no output in dmesg.
>
> I was running these tests on ext4 filesystems:
> (for the Ubuntu kernels)
> /dev/mapper/stross--vg-root on / type ext4 (rw,errors=remount-ro)
> (for the stock kernel, in the virtualbox)
> /dev/sda1 on / type ext4 (rw,errors=remount-ro)
>
> Please let me know if you need any more information.
>
> FWIW, I did find this bug while googling, but it was on older kernels and
> was allegedly fixed:
> https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1101666
>
>
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at http://www.tux.org/lkml/
--
Jan Kara <jack@suse.cz>
SUSE Labs, CR
^ permalink raw reply [flat|nested] 6+ messages in thread