mirror of https://lore.kernel.org/lkml/
 help / color / mirror / Atom feed
* Re: [patch] voluntary-preempt-2.6.9-rc3-mm2-T1
@ 2004-10-06 20:01 Mark_H_Johnson
  2004-10-06 20:07 ` Ingo Molnar
  0 siblings, 1 reply; 12+ messages in thread
From: Mark_H_Johnson @ 2004-10-06 20:01 UTC (permalink / raw)
  To: linux-kernel; +Cc: Ingo Molnar, Rui Nuno Capela

>Meanwhile, I'm stuck with 2.6.9-rc2-mm4-S7 (SMP), but happy.
>
>Strange thing is, that on my laptop, 2.6.9-rc3-mm2-S9 (UP) is doing just
>fine. Guess that ohci_hcd now makes the difference here, against the
>former which makes uhci_hcd bad behaved atm.

I am having similar problems with -T1 and separately reported problems with
a build of rc3-mm1-S8 as well (no oops, but the USB mouse is dead).
Somewhere between those two versions (rc2-mm4-S7 and rc3-mm1-S8) is where
the problem appears to be introduced. For now I'll stay with my working -S0
kernel.

--Mark H Johnson
  <mailto:Mark_H_Johnson@raytheon.com>


^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: [patch] voluntary-preempt-2.6.9-rc3-mm2-T1
  2004-10-06 20:01 [patch] voluntary-preempt-2.6.9-rc3-mm2-T1 Mark_H_Johnson
@ 2004-10-06 20:07 ` Ingo Molnar
  0 siblings, 0 replies; 12+ messages in thread
From: Ingo Molnar @ 2004-10-06 20:07 UTC (permalink / raw)
  To: Mark_H_Johnson; +Cc: linux-kernel, Rui Nuno Capela


* Mark_H_Johnson@raytheon.com <Mark_H_Johnson@raytheon.com> wrote:

> >Meanwhile, I'm stuck with 2.6.9-rc2-mm4-S7 (SMP), but happy.
> >
> >Strange thing is, that on my laptop, 2.6.9-rc3-mm2-S9 (UP) is doing just
> >fine. Guess that ohci_hcd now makes the difference here, against the
> >former which makes uhci_hcd bad behaved atm.
> 
> I am having similar problems with -T1 and separately reported problems
> with a build of rc3-mm1-S8 as well (no oops, but the USB mouse is
> dead). Somewhere between those two versions (rc2-mm4-S7 and
> rc3-mm1-S8) is where the problem appears to be introduced. For now
> I'll stay with my working -S0 kernel.

disable USB for now - it's broken in -mm and unrelated to -VP. There are
hopes that in -rc3-mm3 USB will work again.

	Ingo

^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: [patch] voluntary-preempt-2.6.9-rc3-mm2-T1
  2004-10-05 13:47                   ` [patch] voluntary-preempt-2.6.9-rc3-mm2-T1 Ingo Molnar
  2004-10-05 16:37                     ` Rui Nuno Capela
@ 2004-10-08 11:16                     ` William Lee Irwin III
  1 sibling, 0 replies; 12+ messages in thread
From: William Lee Irwin III @ 2004-10-08 11:16 UTC (permalink / raw)
  To: Ingo Molnar
  Cc: linux-kernel, Lee Revell, K.R. Foley, Rui Nuno Capela, Florian Schmidt

On Tue, Oct 05, 2004 at 03:47:07PM +0200, Ingo Molnar wrote:
> i've released the -T1 VP patch:
>   http://redhat.com/~mingo/voluntary-preempt/voluntary-preempt-2.6.9-rc3-mm2-T1
> Changes since -T0:
>  - added the read-lock fix from Hugh that affects SMP systems. This 
>    could fix Rui's problem - i've checked -T1 on a P4/HT box and saw no 
>    problems, BYMMV.
>  - compilation fixes (for those who downloaded T0 early)
>  - small tracer improvement
> to build a -T1 tree from scratch the patching order is:
>    http://kernel.org/pub/linux/kernel/v2.6/linux-2.6.8.tar.bz2
>  + http://kernel.org/pub/linux/kernel/v2.6/testing/patch-2.6.9-rc3.bz2
>  + http://kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.9-rc3/2.6.9-rc3-mm2/2.6.9-rc3-mm2.bz2
>  + http://redhat.com/~mingo/voluntary-preempt/voluntary-preempt-2.6.9-rc3-mm2-T1

The version numbers are up to Chebyshev polynomials. Now I have to try it.


-- wli

^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: [patch] voluntary-preempt-2.6.9-rc3-mm2-T1
@ 2004-10-06 18:57 Kari Hameenaho
  0 siblings, 0 replies; 12+ messages in thread
From: Kari Hameenaho @ 2004-10-06 18:57 UTC (permalink / raw)
  To: linux-kernel

Ingo Molnar wrote:
 >
 >* Lee Revell <rlrevell@xxxxxxxxxxx> wrote:
 >
 > > > i believe Andrew said that these USB problems should be fixed in
 > > > the next -mm iteration.
 > > >
 > >
 > > FWIW, this one does not work for me either, I get a USB-related Oops
 > > on boot.
 >
 > by next -mm iteration i meant -rc3-mm3, which is not released yet.
 >
 > Ingo
 >

The usb-hubs problem patch in LKML by Greg KH seems to fix this problem.

The thread is 2.6.9-rc3-mm1, bk-pci patch, USB hubs

---
Kari Hämeenaho

^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: [patch] voluntary-preempt-2.6.9-rc3-mm2-T1
  2004-10-06  0:12                             ` Lee Revell
@ 2004-10-06  7:51                               ` Ingo Molnar
  0 siblings, 0 replies; 12+ messages in thread
From: Ingo Molnar @ 2004-10-06  7:51 UTC (permalink / raw)
  To: Lee Revell; +Cc: Rui Nuno Capela, linux-kernel, K.R. Foley, Florian Schmidt


* Lee Revell <rlrevell@joe-job.com> wrote:

> > i believe Andrew said that these USB problems should be fixed in the 
> > next -mm iteration.
> > 
> 
> FWIW, this one does not work for me either, I get a USB-related Oops
> on boot.

by next -mm iteration i meant -rc3-mm3, which is not released yet.

	Ingo

^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: [patch] voluntary-preempt-2.6.9-rc3-mm2-T1
  2004-10-05 19:44                           ` Ingo Molnar
  2004-10-05 20:01                             ` Rui Nuno Capela
@ 2004-10-06  0:12                             ` Lee Revell
  2004-10-06  7:51                               ` Ingo Molnar
  1 sibling, 1 reply; 12+ messages in thread
From: Lee Revell @ 2004-10-06  0:12 UTC (permalink / raw)
  To: Ingo Molnar; +Cc: Rui Nuno Capela, linux-kernel, K.R. Foley, Florian Schmidt

On Tue, 2004-10-05 at 15:44, Ingo Molnar wrote:
> * Rui Nuno Capela <rncbc@rncbc.org> wrote:
> 
> > OTOH, I've tested T1 with CONFIG_SCHED_SMT and/or CONFIG_SMP not set,
> > and got similar crashes too. So this seems to be some nasty bug
> > introduced by -mm{1,2}, not by VP on SMP/SMT.
> > 
> > Yes, I do have some critical USB devices around here. One is that
> > wacom tablet (mouse) and the other is a tascam us-224 audio/midi
> > control surface that a love very much :)
> > 
> > Don't know if this makes me feeling better, doh.
> 
> i believe Andrew said that these USB problems should be fixed in the 
> next -mm iteration.
> 

FWIW, this one does not work for me either, I get a USB-related Oops on
boot.

Lee 


^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: [patch] voluntary-preempt-2.6.9-rc3-mm2-T1
  2004-10-05 19:44                           ` Ingo Molnar
@ 2004-10-05 20:01                             ` Rui Nuno Capela
  2004-10-06  0:12                             ` Lee Revell
  1 sibling, 0 replies; 12+ messages in thread
From: Rui Nuno Capela @ 2004-10-05 20:01 UTC (permalink / raw)
  To: Ingo Molnar; +Cc: linux-kernel, Lee Revell, K.R. Foley, Florian Schmidt

Ingo Molnar wrote:
>
> * Rui Nuno Capela wrote:
>
>> OTOH, I've tested T1 with CONFIG_SCHED_SMT and/or CONFIG_SMP not set,
>> and got similar crashes too. So this seems to be some nasty bug
>> introduced by -mm{1,2}, not by VP on SMP/SMT.
>>
>> Yes, I do have some critical USB devices around here. One is that
>> wacom tablet (mouse) and the other is a tascam us-224 audio/midi
>> control surface that a love very much :)
>>
>> Don't know if this makes me feeling better, doh.
>
> i believe Andrew said that these USB problems should be fixed in the
> next -mm iteration.
>

Oh yes, I really do hope so :)

Meanwhile, I'm stuck with 2.6.9-rc2-mm4-S7 (SMP), but happy.

Strange thing is, that on my laptop, 2.6.9-rc3-mm2-S9 (UP) is doing just
fine. Guess that ohci_hcd now makes the difference here, against the
former which makes uhci_hcd bad behaved atm.

Thanks Ingo.
-- 
rncbc aka Rui Nuno Capela
rncbc@rncbc.org


^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: [patch] voluntary-preempt-2.6.9-rc3-mm2-T1
  2004-10-05 19:38                         ` Rui Nuno Capela
@ 2004-10-05 19:44                           ` Ingo Molnar
  2004-10-05 20:01                             ` Rui Nuno Capela
  2004-10-06  0:12                             ` Lee Revell
  0 siblings, 2 replies; 12+ messages in thread
From: Ingo Molnar @ 2004-10-05 19:44 UTC (permalink / raw)
  To: Rui Nuno Capela; +Cc: linux-kernel, Lee Revell, K.R. Foley, Florian Schmidt


* Rui Nuno Capela <rncbc@rncbc.org> wrote:

> OTOH, I've tested T1 with CONFIG_SCHED_SMT and/or CONFIG_SMP not set,
> and got similar crashes too. So this seems to be some nasty bug
> introduced by -mm{1,2}, not by VP on SMP/SMT.
> 
> Yes, I do have some critical USB devices around here. One is that
> wacom tablet (mouse) and the other is a tascam us-224 audio/midi
> control surface that a love very much :)
> 
> Don't know if this makes me feeling better, doh.

i believe Andrew said that these USB problems should be fixed in the 
next -mm iteration.

	Ingo

^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: [patch] voluntary-preempt-2.6.9-rc3-mm2-T1
  2004-10-05 18:42                       ` Ingo Molnar
@ 2004-10-05 19:38                         ` Rui Nuno Capela
  2004-10-05 19:44                           ` Ingo Molnar
  0 siblings, 1 reply; 12+ messages in thread
From: Rui Nuno Capela @ 2004-10-05 19:38 UTC (permalink / raw)
  To: Ingo Molnar; +Cc: linux-kernel, Lee Revell, K.R. Foley, Florian Schmidt

Ingo Molnar wrote:
>
> * Rui Nuno Capela wrote:
>
>> Still the same ugliness here with T1. As before, there goes some info
>> attached, which I could collect while barely up and running.
>
> the dmesg info shows you had a crash early on, in khubd:
>
>  Badness in remove_proc_entry at fs/proc/generic.c:688
>   [<c018c8e8>] remove_proc_entry+0x152/0x15a
>   [<f8b8e116>] uhci_hcd_init+0x116/0x133 [uhci_hcd]
>   [<c0135f0e>] sys_init_module+0x1df/0x2da
>   [<c01044ed>] sysenter_past_esp+0x52/0x71
>  usb 3-1: new low speed USB device using address 2
>  Unable to handle kernel paging request at virtual address a49c0e0c
>
> i believe this is a crash present in -mm too. In theory such a crash
> could mess up the kernel so best would be if you could try a kernel with
> USB disabled? Hopefully none of your critical devices is on USB ...
>

Yeah, I've found it the hard way :) One of my trials discovered just that,
by booting without any USB devices plugged in. It booted apparently fine.

Then, exacltly when I plug in my USB mouse (Wacom Graphire2 tablet), I
immediately get the following kernel oops (on dmesg):

IRQ#23 thread started up.
IRQ#19 thread started up.
usb 2-1: new low speed USB device using address 2
Unable to handle kernel paging request at virtual address aaf7ee8f
 printing eip:
c014284b
*pde = 00000000
Oops: 0000 [#1]
PREEMPT SMP
Modules linked in: mga ohci1394 ieee1394 ehci_hcd uhci_hcd intel_mch_agp
agpgart snd_usb_usx2y snd_usb_lib snd_rawmidi snd_seq_device snd_hwdep
snd_intel8x0 snd_ac97_codec snd_pcm snd_timer snd soundcore snd_page_alloc
evdev sk98lin realtime commoncap w83781d i2c_sensor i2c_isa i2c_i801
i2c_core wacom usbcore subfs dm_mod
CPU:    1
EIP:    0060:[<c014284b>]    Not tainted VLI
EFLAGS: 00010093   (2.6.9-rc3-mm2-T1.0smp)
EIP is at check_poison_obj+0x98/0x1d0
eax: 0000006b   ebx: f6e8a780   ecx: f6d254b8   edx: ffffffa5
esi: 00000000   edi: aaf7ee8f   ebp: f74d3d10   esp: f74d3ce8
ds: 007b   es: 007b   ss: 0068   preempt: 00000002
Process khubd (pid: 1306, threadinfo=f74d3000 task=f7df2a40)
Stack: f6e8a780 f74d3cfc c0130ce3 0000002b c0144658 00000000 0000002c
f6e8a780
       aaf7ee8b f6e8a780 f74d3d34 c01442f5 f6e8a780 aaf7ee8b f6d254b8
f6d254b8
       f6e8a780 00000046 00000020 f74d3d58 c0144658 f6e8a780 00000020
aaf7ee8b
Call Trace:
 [<c0130ce3>] __mcount+0x1d/0x1f
 [<c0144658>] kmem_cache_alloc+0x6b/0xa5
 [<c01442f5>] cache_alloc_debugcheck_after+0x129/0x162
 [<c0144658>] kmem_cache_alloc+0x6b/0xa5
 [<f8c35ff9>] uhci_alloc_urb_priv+0x26/0x81 [uhci_hcd]
 [<f8c35ff9>] uhci_alloc_urb_priv+0x26/0x81 [uhci_hcd]
 [<f8c36fbb>] uhci_urb_enqueue+0x5d/0x2bc [uhci_hcd]
 [<c0110350>] mcount+0x14/0x18
 [<f8b424f4>] hcd_submit_urb+0x127/0x194 [usbcore]
 [<f8b42e38>] usb_submit_urb+0x1e2/0x244 [usbcore]
 [<f8b430c7>] usb_start_wait_urb+0xe/0xe2 [usbcore]
 [<f8b4310c>] usb_start_wait_urb+0x53/0xe2 [usbcore]
 [<c0130ce3>] __mcount+0x1d/0x1f
 [<c01ea660>] kref_init+0x8/0x13
 [<f8b42ba8>] usb_init_urb+0x27/0x3c [usbcore]
 [<c0110350>] mcount+0x14/0x18
 [<f8b42bef>] usb_alloc_urb+0x32/0x52 [usbcore]
 [<f8b43215>] usb_internal_control_msg+0x7a/0x83 [usbcore]
 [<f8b432aa>] usb_control_msg+0x8c/0xa0 [usbcore]
 [<f8b3fe23>] hub_set_address+0x6d/0x90 [usbcore]
 [<f8b3fff7>] hub_port_init+0x1b1/0x39a [usbcore]
 [<f8b404d9>] hub_port_connect_change+0xfe/0x43a [usbcore]
 [<f8b409d3>] hub_events+0x1be/0x395 [usbcore]
 [<f8b40be1>] hub_thread+0x37/0x109 [usbcore]
 [<c0130845>] autoremove_wake_function+0x0/0x50
 [<c0104416>] ret_from_fork+0x6/0x14
 [<c0130845>] autoremove_wake_function+0x0/0x50
 [<f8b40baa>] hub_thread+0x0/0x109 [usbcore]
 [<c0102601>] kernel_thread_helper+0x5/0xb
Code: 89 44 24 08 e8 09 fe ff ff 83 45 ec 01 83 7d ec 05 7f 5b 83 c6 01 3b
75 f0 7d 53 3b 75 e4 b8 6b 00 00 00 ba a5 ff ff ff 0f 44 c2 <38> 04 3e 2e
74 e2 8b 45 ec 85 c0 75 a0 8b 55 f0 89 7c 24 04 c7
 <6>note: khubd[1306] exited with preempt_count 1


Guess what? It's right after this crash that ksoftirqd/1 pulls up to
99.99%CPU#1 and stays still on that figure, forever. Of course, as Ingo
noted, this was happening behind the scenes every time I was boot/initing.

OTOH, I've tested T1 with CONFIG_SCHED_SMT and/or CONFIG_SMP not set, and
got similar crashes too. So this seems to be some nasty bug introduced by
-mm{1,2}, not by VP on SMP/SMT.

Yes, I do have some critical USB devices around here. One is that wacom
tablet (mouse) and the other is a tascam us-224 audio/midi control surface
that a love very much :)

Don't know if this makes me feeling better, doh.

Cheers.
-- 
rncbc aka Rui Nuno Capela
rncbc@rncbc.org


^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: [patch] voluntary-preempt-2.6.9-rc3-mm2-T1
  2004-10-05 16:37                     ` Rui Nuno Capela
@ 2004-10-05 18:42                       ` Ingo Molnar
  2004-10-05 19:38                         ` Rui Nuno Capela
  0 siblings, 1 reply; 12+ messages in thread
From: Ingo Molnar @ 2004-10-05 18:42 UTC (permalink / raw)
  To: Rui Nuno Capela; +Cc: linux-kernel, Lee Revell, K.R. Foley, Florian Schmidt


* Rui Nuno Capela <rncbc@rncbc.org> wrote:

> Still the same ugliness here with T1. As before, there goes some info
> attached, which I could collect while barely up and running.

the dmesg info shows you had a crash early on, in khubd:

 Badness in remove_proc_entry at fs/proc/generic.c:688
  [<c018c8e8>] remove_proc_entry+0x152/0x15a
  [<f8b8e116>] uhci_hcd_init+0x116/0x133 [uhci_hcd]
  [<c0135f0e>] sys_init_module+0x1df/0x2da
  [<c01044ed>] sysenter_past_esp+0x52/0x71
 usb 3-1: new low speed USB device using address 2
 Unable to handle kernel paging request at virtual address a49c0e0c

i believe this is a crash present in -mm too. In theory such a crash
could mess up the kernel so best would be if you could try a kernel with
USB disabled? Hopefully none of your critical devices is on USB ...

	Ingo

^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: [patch] voluntary-preempt-2.6.9-rc3-mm2-T1
  2004-10-05 13:47                   ` [patch] voluntary-preempt-2.6.9-rc3-mm2-T1 Ingo Molnar
@ 2004-10-05 16:37                     ` Rui Nuno Capela
  2004-10-05 18:42                       ` Ingo Molnar
  2004-10-08 11:16                     ` William Lee Irwin III
  1 sibling, 1 reply; 12+ messages in thread
From: Rui Nuno Capela @ 2004-10-05 16:37 UTC (permalink / raw)
  To: Ingo Molnar; +Cc: linux-kernel, Lee Revell, K.R. Foley, Florian Schmidt

[-- Attachment #1: Type: text/plain, Size: 745 bytes --]

Ingo Molnar wrote:
>
> i've released the -T1 VP patch:
>
http://redhat.com/~mingo/voluntary-preempt/voluntary-preempt-2.6.9-rc3-mm2-T1
>
> Changes since -T0:
>
>  - added the read-lock fix from Hugh that affects SMP systems. This
>    could fix Rui's problem - i've checked -T1 on a P4/HT box and saw
>    no problems, BYMMV.
>

No go here. My milleage certainly varies :)

Still the same ugliness here with T1. As before, there goes some info
attached, which I could collect while barely up and running.

I don't wanna be a PITA, but changes are hardly noticed since T0.

Next I'll try building with CONFIG_SCHED_SMT off, and later with
CONFIG_SMP off too, to just try to make some point here.

CU
-- 
rncbc aka Rui Nuno Capela
rncbc@rncbc.org

[-- Attachment #2: dump-2.6.9-rc3-mm2-T1.0smp.tar.gz --]
[-- Type: application/x-tgz, Size: 16691 bytes --]

^ permalink raw reply	[flat|nested] 12+ messages in thread

* [patch] voluntary-preempt-2.6.9-rc3-mm2-T1
  2004-10-04 21:53                 ` [patch] voluntary-preempt-2.6.9-rc3-mm1-S9 Ingo Molnar
@ 2004-10-05 13:47                   ` Ingo Molnar
  2004-10-05 16:37                     ` Rui Nuno Capela
  2004-10-08 11:16                     ` William Lee Irwin III
  0 siblings, 2 replies; 12+ messages in thread
From: Ingo Molnar @ 2004-10-05 13:47 UTC (permalink / raw)
  To: linux-kernel; +Cc: Lee Revell, K.R. Foley, Rui Nuno Capela, Florian Schmidt


i've released the -T1 VP patch:

  http://redhat.com/~mingo/voluntary-preempt/voluntary-preempt-2.6.9-rc3-mm2-T1

Changes since -T0:

 - added the read-lock fix from Hugh that affects SMP systems. This 
   could fix Rui's problem - i've checked -T1 on a P4/HT box and saw no 
   problems, BYMMV.

 - compilation fixes (for those who downloaded T0 early)

 - small tracer improvement

to build a -T1 tree from scratch the patching order is:

   http://kernel.org/pub/linux/kernel/v2.6/linux-2.6.8.tar.bz2
 + http://kernel.org/pub/linux/kernel/v2.6/testing/patch-2.6.9-rc3.bz2
 + http://kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.9-rc3/2.6.9-rc3-mm2/2.6.9-rc3-mm2.bz2
 + http://redhat.com/~mingo/voluntary-preempt/voluntary-preempt-2.6.9-rc3-mm2-T1

	Ingo

^ permalink raw reply	[flat|nested] 12+ messages in thread

end of thread, other threads:[~2004-10-08 11:20 UTC | newest]

Thread overview: 12+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2004-10-06 20:01 [patch] voluntary-preempt-2.6.9-rc3-mm2-T1 Mark_H_Johnson
2004-10-06 20:07 ` Ingo Molnar
  -- strict thread matches above, loose matches on Subject: below --
2004-10-06 18:57 Kari Hameenaho
2004-09-21  2:07 BKL backtraces - was: Re: [patch] voluntary-preempt-2.6.9-rc2-mm1-S1 K.R. Foley
2004-09-21  7:18 ` Ingo Molnar
2004-09-21  7:44   ` [patch] voluntary-preempt-2.6.9-rc2-mm1-S2 Ingo Molnar
2004-09-22 10:33     ` [patch] voluntary-preempt-2.6.9-rc2-mm1-S3 Ingo Molnar
2004-09-23 12:28       ` [patch] voluntary-preempt-2.6.9-rc2-mm1-S4 Ingo Molnar
2004-09-23 21:12         ` [patch] voluntary-preempt-2.6.9-rc2-mm3-S5 Ingo Molnar
2004-09-24  7:44           ` [patch] voluntary-preempt-2.6.9-rc2-mm3-S6 Ingo Molnar
2004-09-28  0:05             ` [patch] voluntary-preempt-2.6.9-rc2-mm4-S7 Ingo Molnar
2004-10-03 21:09               ` [patch] voluntary-preempt-2.6.9-rc3-mm1-S8 Ingo Molnar
2004-10-04 21:53                 ` [patch] voluntary-preempt-2.6.9-rc3-mm1-S9 Ingo Molnar
2004-10-05 13:47                   ` [patch] voluntary-preempt-2.6.9-rc3-mm2-T1 Ingo Molnar
2004-10-05 16:37                     ` Rui Nuno Capela
2004-10-05 18:42                       ` Ingo Molnar
2004-10-05 19:38                         ` Rui Nuno Capela
2004-10-05 19:44                           ` Ingo Molnar
2004-10-05 20:01                             ` Rui Nuno Capela
2004-10-06  0:12                             ` Lee Revell
2004-10-06  7:51                               ` Ingo Molnar
2004-10-08 11:16                     ` William Lee Irwin III

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox

Powered by JetHome