PROBLEM: close(fd) doesn't wake up read(fd) or select() in another thread

PROBLEM: close(fd) doesn't wake up read(fd) or select() in another thread

[Michael Lindner]

[1.] One line summary of the problem:
	close(fd) doesn't wake up read(fd) or select() in another thread

[2.] Full description of the problem/report:
	In a multithreaded program, if a thread closes a file descriptor
	that another thread is reading from or selecting on, the thread
	doing the read() or select() does not see an EOF and does not
	rerturn.

	The context is a piece of code that farms work out to multiple
	worker threads via a pipe to each worker. When the main thread
	wants to shut down, it closes the pipes and joins the other threads.
	This code worked on Solaris 2.7, 8, & 9, but failed when we switched
	to Linux.

	In tests, the same behavior happens with other types of file
	descriptors (sockets and console - I did not test with a disk file
	for the obvious reason that the speed of a read would make
	the test very timing critical).

[3.] kernel

[4.] Kernel version (from /proc/version): Linux version 2.6.11-w4l (root@self) 
(gcc version 3.3.4 (Mandrakelinux 10.1 3.3.4-2mdk)) #2 Thu Nov 3 15:44:07 EST 
2005

[5.] Output of Oops.. message (if applicable)

[6.] A small shell script or example program which triggers the
     problem (if possible)

/*
 * Compile with:
 * 
 * gcc -o threadbug threadbug.c -lpthread
 * 
 * This code illustrates that then a thread reads from a file descriptor
 * and the file descriptor is closed by another thread, the read() never
 * returns. The same behavior happens with select().
 *
 * N.B. If the file descriptor is a socket and you call shutdown() for
 * reading, the read() does return.
 *
 */

#include <pthread.h>
#include <stdio.h>
#include <unistd.h>
#include <sys/socket.h>

void*
func(void* v)
{
    int n;
    char c;
    fprintf(stderr, "in thread function, reading from fd %d\n", 0);
    n = read(0, &c, 1);
    fprintf(stderr, "read returned %d\n", n);
}

main()
{
    pthread_t tid;
    pthread_create(&tid, 0, func, 0);
    sleep(2);
    fprintf(stderr, "in main thread closing fd %d\n", 0);
    close(0);
    fprintf(stderr, "in main thread waiting for thread function to return\n");
    fprintf(stderr, "...but it never will...\n");
    pthread_join(tid, 0);
}

[7.] Environment 

[7.1.] Software (add the output of the ver_linux script here)
Linux self 2.6.11-w4l #2 Thu Nov 3 15:44:07 EST 2005 i686 Unknown CPU Typ 
unknown GNU/Linux
 
Gnu C                  3.3.4
Gnu make               3.80
binutils               2.15.90.0.3
util-linux             2.12a
mount                  2.12a
module-init-tools      3.0
e2fsprogs              1.35
reiserfsprogs          3.6.17
reiser4progs           line
PPP                    2.4.2
nfs-utils              1.0.6
Linux C Library        2.3.4
Dynamic linker (ldd)   2.3.4
Procps                 3.2.3
Net-tools              1.60
Console-tools          0.2.3
Sh-utils               5.2.1
udev                   030
Modules Loaded         nls_iso8859_1 nls_cp850 vfat fat sd_mod tun sr_mod 
binfmt_misc eeprom w83l785ts asb100 i2c_sensor i2c_nforce2 i2c_core Win4Lin 
mki_adapter md5 ipv6 snd_pcm_oss snd_mixer_oss snd_cs46xx snd_rawmidi 
snd_seq_device snd_ac97_codec snd_pcm snd_timer snd soundcore snd_page_alloc 
gameport ppp_async ppp_generic slhc crc_ccitt af_packet floppy forcedeth 
ide_cd cdrom loop nvidia_agp agpgart usblp usb_storage scsi_mod ehci_hcd 
ohci_hcd usbcore reiserfs raid1

[7.2.] Processor information (from /proc/cpuinfo): 
processor       : 0
vendor_id       : AuthenticAMD
cpu family      : 6
model           : 10
model name      : Unknown CPU Typ
stepping        : 0
cpu MHz         : 2305.674
cache size      : 512 KB
fdiv_bug        : no
hlt_bug         : no
f00f_bug        : no
coma_bug        : no
fpu             : yes
fpu_exception   : yes
cpuid level     : 1
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 sep mtrr pge mca cmov pat 
pse36 mmx fxsr sse pni syscall mp mmxext 3dnowext 3dnow
bogomips        : 4571.13

[7.3.] Module information (from /proc/modules):
nls_iso8859_1 3840 0 - Live 0xf8caf000
nls_cp850 4608 0 - Live 0xf8d4b000
vfat 10880 0 - Live 0xf8b3a000
fat 35612 1 vfat, Live 0xf8d64000
sd_mod 15504 0 - Live 0xf8d53000
tun 8704 0 - Live 0xf8d1a000
sr_mod 15076 0 - Live 0xf8d4e000
binfmt_misc 8840 1 - Live 0xf8d16000
eeprom 5712 0 - Live 0xf8d13000
w83l785ts 5716 0 - Live 0xf8d09000
asb100 22036 0 - Live 0xf8d41000
i2c_sensor 2880 3 eeprom,w83l785ts,asb100, Live 0xf887e000
i2c_nforce2 5440 0 - Live 0xf8d06000
i2c_core 17808 5 eeprom,w83l785ts,asb100,i2c_sensor,i2c_nforce2, Live 
0xf8d0d000
Win4Lin 292936 1 - Live 0xf8e14000
mki_adapter 37736 1 Win4Lin, Live 0xf8cd2000
md5 3776 1 - Live 0xf8b4b000
ipv6 230848 29 - Live 0xf8d84000
snd_pcm_oss 48096 0 - Live 0xf8d34000
snd_mixer_oss 17088 1 snd_pcm_oss, Live 0xf8ccc000
snd_cs46xx 82056 0 - Live 0xf8d1e000
snd_rawmidi 19552 1 snd_cs46xx, Live 0xf8cc6000
snd_seq_device 6924 1 snd_rawmidi, Live 0xf8c5c000
snd_ac97_codec 73656 1 snd_cs46xx, Live 0xf8cf3000
snd_pcm 80200 3 snd_pcm_oss,snd_cs46xx,snd_ac97_codec, Live 0xf8cde000
snd_timer 20484 1 snd_pcm, Live 0xf8cb1000
snd 46436 8 
snd_pcm_oss,snd_mixer_oss,snd_cs46xx,snd_rawmidi,snd_seq_device,snd_ac97_codec,snd_pcm,snd_timer, 
Live 0xf8cb9000
soundcore 7136 1 snd, Live 0xf8c59000
snd_page_alloc 7556 2 snd_cs46xx,snd_pcm, Live 0xf8c56000
gameport 3520 1 snd_cs46xx, Live 0xf8859000
ppp_async 9216 0 - Live 0xf8c52000
ppp_generic 24532 1 ppp_async, Live 0xf8ca2000
slhc 6720 1 ppp_generic, Live 0xf8b3e000
crc_ccitt 1728 1 ppp_async, Live 0xf887c000
af_packet 16328 2 - Live 0xf8c4d000
floppy 53904 0 - Live 0xf8c71000
forcedeth 15872 0 - Live 0xf8b35000
ide_cd 36228 0 - Live 0xf8c43000
cdrom 36320 2 sr_mod,ide_cd, Live 0xf8b41000
loop 12616 0 - Live 0xf8b30000
nvidia_agp 5916 1 - Live 0xf8876000
agpgart 28136 1 nvidia_agp, Live 0xf8b28000
usblp 10752 0 - Live 0xf884d000
usb_storage 66304 0 - Live 0xf8b6b000
scsi_mod 115016 3 sd_mod,sr_mod,usb_storage, Live 0xf8b4d000
ehci_hcd 27848 0 - Live 0xf8851000
ohci_hcd 18696 0 - Live 0xf8842000
usbcore 105592 5 usblp,usb_storage,ehci_hcd,ohci_hcd, Live 0xf885b000
reiserfs 258932 1 - Live 0xf8a56000
raid1 13824 1 - Live 0xf8815000

[7.4.] Loaded driver and hardware information (/proc/ioports, /proc/iomem)
0000-001f : dma1
0020-0021 : pic1
0040-0043 : timer0
0050-0053 : timer1
0060-006f : keyboard
0080-008f : dma page reg
00a0-00a1 : pic2
00c0-00df : dma2
00f0-00ff : fpu
0170-0177 : ide1
01f0-01f7 : ide0
02f8-02ff : serial
0376-0376 : ide1
03c0-03df : vesafb
03f6-03f6 : ide0
03f8-03ff : serial
0cf8-0cff : PCI conf1
5000-5007 : nForce2 SMBus
5500-5507 : nForce2 SMBus
9000-afff : PCI Bus #01
  9000-9007 : 0000:01:0a.0
    9000-9007 : ide2
  9400-9403 : 0000:01:0a.0
    9402-9402 : ide2
  9800-9807 : 0000:01:0a.0
    9800-9807 : ide3
  9c00-9c03 : 0000:01:0a.0
    9c02-9c02 : ide3
  a000-a00f : 0000:01:0a.0
    a000-a007 : ide2
    a008-a00f : ide3
b000-b007 : 0000:00:04.0
  b000-b007 : forcedeth
b400-b4ff : 0000:00:06.0
b800-b87f : 0000:00:06.0
c400-c41f : 0000:00:01.1
f000-f00f : 0000:00:09.0
  f000-f007 : ide0
  f008-f00f : ide1

00000000-0009e7ff : System RAM
0009e800-0009ffff : reserved
000a0000-000bffff : Video RAM area
000c0000-000cf7ff : Video ROM
000d0000-000d17ff : Adapter ROM
000d2000-000d47ff : Adapter ROM
000f0000-000fffff : System ROM
00100000-3ffeffff : System RAM
  00100000-002d6d22 : Kernel code
  002d6d23-0036733f : Kernel data
3fff0000-3fff2fff : ACPI Non-volatile Storage
3fff3000-3fffffff : ACPI Tables
d0000000-d7ffffff : PCI Bus #02
  d0000000-d7ffffff : 0000:02:00.0
    d0000000-d7ffffff : vesafb
d8000000-dbffffff : 0000:00:00.0
dc000000-ddffffff : PCI Bus #02
  dc000000-dcffffff : 0000:02:00.0
de000000-dfffffff : PCI Bus #01
  df000000-df003fff : 0000:01:0a.0
e0000000-e0000fff : 0000:00:04.0
  e0000000-e0000fff : forcedeth
e0001000-e0001fff : 0000:00:06.0
e0003000-e0003fff : 0000:00:02.0
  e0003000-e0003fff : ohci_hcd
e0004000-e0004fff : 0000:00:02.1
  e0004000-e0004fff : ohci_hcd
e0005000-e00050ff : 0000:00:02.2
  e0005000-e00050ff : ehci_hcd
fec00000-fec00fff : reserved
fee00000-fee00fff : reserved
ffff0000-ffffffff : reserved

[7.5.] PCI information ('lspci -vvv' as root)

lots of stuff here - do you really need it?

[7.6.] SCSI information (from /proc/scsi/scsi)
Attached devices:

[7.7.] Other information that might be relevant to the problem

[X.] Other notes, patches, fixes, workarounds:

-- 
Michael Lindner

Re: PROBLEM: close(fd) doesn't wake up read(fd) or select() in another thread

[Michael Lindner]

Update: On advice from Nish Aravamudan I upgraded my kernel to 2.6.17.4 and 
verified that the problem still exists in that version.

Thanks,
-- 
Michael Lindner

Re: PROBLEM: close(fd) doesn't wake up read(fd) or select() in another thread

[Alistair John Strachan]

On Friday 14 July 2006 20:07, Michael Lindner wrote:
> Update: On advice from Nish Aravamudan I upgraded my kernel to 2.6.17.4 and
> verified that the problem still exists in that version.

This behaviour is POSIX compliant and should be expected. There is no bug.

If you want to work around it, use shutdown() as you suggested, and poll() 
instead of select() and wait for POLLNVAL or POLLHUP.

See the following Java "bug" which explains the situation a little better:

http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=4344135

-- 
Cheers,
Alistair.

Final year Computer Science undergraduate.
1F2 55 South Clerk Street, Edinburgh, UK.

RE: PROBLEM: close(fd) doesn't wake up read(fd) or select() in another thread

[David Schwartz]

> [1.] One line summary of the problem:
> 	close(fd) doesn't wake up read(fd) or select() in another thread

	If you delete a resource in one thread while another thread is using it,
anything can happen. This is as serious a bug as calling 'free' on a block
of memory while another thread is using it.

	You can never be sure the other thread was in 'read' or 'select' (as
opposed to being about to call it and then being pre-empted), so such code
will always have race conditions.

	DS