* Re: [lm-sensors] 2.6.24-rc4 hwmon it87 probe fails
[not found] ` <fa.Xv3BmMxnCGLOeEijySte5mKDO5k@ifi.uio.no>
@ 2007-12-20 1:09 ` Robert Hancock
2008-01-12 9:56 ` Jean Delvare
0 siblings, 1 reply; 9+ messages in thread
From: Robert Hancock @ 2007-12-20 1:09 UTC (permalink / raw)
To: Carlos Corbacho
Cc: Bjorn Helgaas, Jean Delvare, Shaohua Li, Mike Houston,
Adrian Bunk, Elvis Pranskevichus, mhoffman, linux-kernel,
lm-sensors, Adam Belay, Zhao Yakui, Thomas Renninger, lenb,
linux-acpi
Carlos Corbacho wrote:
> On Thursday 20 December 2007 00:20:21 Bjorn Helgaas wrote:
>> I suspect the manufacturers would say "Oh, the sensors? The BIOS
>> isn't broken, you're just supposed to use WMI or some (undocumented)
>> ACPI device to get at those."
>
> It's quite possible - can we have DSDTs for the boards in question so we can
> quickly check if this is a possibility? (Basically, to see if they have
> PNP0C14 devices - if they don't, then I'm afraid it's nothing to do with
> WMI).
>
> -Carlos
It's quite possible that the BIOS accesses the device either from ACPI
AML or possibly even from SMI. In that case it would be quite reasonable
for the BIOS to reserve that region to prevent another driver from
loading and trying to take conflicting control of the device. One has to
be careful before assuming that any such reservation is bogus.
--
Robert Hancock Saskatoon, SK, Canada
To email, remove "nospam" from hancockr@nospamshaw.ca
Home Page: http://www.roberthancock.com/
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: 2.6.24-rc4 hwmon it87 probe fails
2007-12-20 1:09 ` [lm-sensors] 2.6.24-rc4 hwmon it87 probe fails Robert Hancock
@ 2008-01-12 9:56 ` Jean Delvare
0 siblings, 0 replies; 9+ messages in thread
From: Jean Delvare @ 2008-01-12 9:56 UTC (permalink / raw)
To: Robert Hancock
Cc: Carlos Corbacho, Bjorn Helgaas, Shaohua Li, Mike Houston,
Adrian Bunk, Elvis Pranskevichus, mhoffman, linux-kernel,
lm-sensors, Adam Belay, Zhao Yakui, Thomas Renninger, lenb,
linux-acpi
Hi Robert,
On Wed, 19 Dec 2007 19:09:54 -0600, Robert Hancock wrote:
> It's quite possible that the BIOS accesses the device either from ACPI
> AML or possibly even from SMI. In that case it would be quite reasonable
> for the BIOS to reserve that region to prevent another driver from
> loading and trying to take conflicting control of the device. One has to
> be careful before assuming that any such reservation is bogus.
Again I am all for honoring such BIOS requests so as to prevent
conflicts between ACPI or SMI and native drivers. The problem is that
no two BIOS out there do the same in this respect. I couldn't see any
correlation between machines declaring their hwmon device in PNP and
machines where ACPI or SMI access the device in question. Many boards
declare their device and seemingly never touch it so it's fine for
Linux to drive them. Some boards no not declare the devices but still
access them in our back.
Thomas' patches should deal with the ACPI AML case in most cases, but
not with SMI.
So either the PNP code in Linux isn't exporting enough details to
differentiate, or even the PNP code has no way to tell these cases
apart. In the latter case there's not much we can do. In the former
case, let's have the PNP code export the information so that hwmon
drivers can decide whether they should bind to the devices or not by
default.
--
Jean Delvare
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: 2.6.24-rc4 hwmon it87 probe fails
2008-01-02 18:30 ` Bjorn Helgaas
@ 2008-01-12 9:49 ` Jean Delvare
0 siblings, 0 replies; 9+ messages in thread
From: Jean Delvare @ 2008-01-12 9:49 UTC (permalink / raw)
To: Bjorn Helgaas
Cc: Shaohua Li, Mike Houston, Adrian Bunk, Elvis Pranskevichus,
Mark M. Hoffman, linux-kernel, lm-sensors, Adam Belay,
Zhao Yakui, Thomas Renninger, lenb, linux-acpi
Hi Bjorn,
Sorry for the late answer.
On Wed, 2 Jan 2008 11:30:55 -0700, Bjorn Helgaas wrote:
> Even if the BIOS does not declare an IT87xxF as an independent device,
> there may be AML that uses the chip internally. For example, the
> BIOS could declare a thermal zone with a _TMP method, and the _TMP
> method could use the IT87xxF to read the temperature. In that case,
> the BIOS would still need to declare the IT87xxF resources so the OS
> doesn't place another device on top of it.
Thomas Renninger's patch set should handle this case. It's in -mm at the
moment and I guess we'll merge most of it in 2.6.25.
> (...) An easy way to do this
> is with a PNP0C02 "motherboard" device, which just says "these
> resources are in use, but the OS needn't attach a driver to this
> device."
How do I see if a given motherboard does this or not? For example I
have a motherboard here that says:
0290-029f : pnp 00:04
How do I know if this is a "PNP0C02 motherboard device" I am supposed
not to touch, or a device that I can attach a native hwmon driver to?
And more importantly, how does the it87 driver (or any other hwmon
driver) know about it? I'm all for updating the hwmon drivers to
cooperate better with PNP in order to prevent clashes between the BIOS
and the OS, but I just don't know how I am supposed to do that.
> > And what "collision" are you talking about?
>
> I meant the problem where we place two devices at the same address,
> causing one to stop working. I wrote "even if it87 isn't present,"
> which is ambiguous -- I meant that we need something that works even
> when the it87 _driver_ isn't present.
I agree then. I don't think this has much to do with PNP though. What
this is calling for is a Super-I/O subsystem that detects the Super-I/O
and instantiate devices out of it. That way the drivers for these
devices (in particular hardware monitoring drivers, watchdog drivers,
some parport drivers...) would no longer need to instantiate their
devices themselves, thus matching the device driver model much better.
I've seen patches float around but so far I could never find the time
to look into them (and I fear it's unlikely to change anytime soon.)
> (...)
> Well, it's true that PNP does not need to describe ALL devices in the
> system. However, it should describe all devices to which the OS should
> bind a driver. The BIOS writer has the discretion to hide devices from
> the OS by omitting them from the ACPI namespace. If he does that, his
> assumption is that the OS will ignore the device (but of course, he's
> thinking about a Plug-and-Play-compliant OS like Windows).
Even under Windows, I pretty much doubt that any monitoring application
cares about what the BIOS says. Motherboard vendor applications have
enough information about each board model to know what they can do or
not without asking the BIOS. Third-party applications probably just
poke at known I/O locations (pretty much like the Linux hwmon drivers
do) and hope for the best.
> (...)
> I think my patch to request only the ports actually used by the it87
> driver is sufficient to solve the current problem. If we find that
> a BIOS neglects to reserve some of the ports decoded by the IT87xxF,
> we can add a PNP quirk to handle that. But I'm not aware of any
> problems like that yet.
I'm not aware of such a problem either.
--
Jean Delvare
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: 2.6.24-rc4 hwmon it87 probe fails
2007-12-25 21:31 ` Jean Delvare
@ 2008-01-02 18:30 ` Bjorn Helgaas
2008-01-12 9:49 ` Jean Delvare
0 siblings, 1 reply; 9+ messages in thread
From: Bjorn Helgaas @ 2008-01-02 18:30 UTC (permalink / raw)
To: Jean Delvare
Cc: Shaohua Li, Mike Houston, Adrian Bunk, Elvis Pranskevichus,
Mark M. Hoffman, linux-kernel, lm-sensors, Adam Belay,
Zhao Yakui, Thomas Renninger, lenb, linux-acpi
On Tuesday 25 December 2007 02:31:26 pm Jean Delvare wrote:
> Le 23/12/2007, "Bjorn Helgaas" <bjorn.helgaas@hp.com> a écrit:
> >On Sunday 23 December 2007 2:28:05 am Jean Delvare wrote:
> >> The problem is that the it87 driver is used on a variety of motherboards,
> >> some where the hardware monitoring device I/O ports are reserved by the
> >> BIOS, some where they aren't. How am I supposed to deal with this?
> >
> >I assume you mean some boards describe those ports in ACPI, and some
> >don't?
>
> Yes, that's what I meant.
>
> >(...) I think the ideal solution would be to figure out how the
> >BIOS writers intended the device to be used, and do that. But the
> >documentation may be lacking, and it degenerates into an OEM-specific
> >mess. Second-best seems like a PNP quirk that pokes enough to determine
> >that a Super I/O device is present, and then reserves resources for
> >it. Then we avoid the collision even if it87 isn't present.
>
> I'm not sure what you mean here exactly. We have two different cases:
> either the ACPI code did declare the PNP device, or it did not. How do
> you unify both?
Even if the BIOS does not declare an IT87xxF as an independent device,
there may be AML that uses the chip internally. For example, the
BIOS could declare a thermal zone with a _TMP method, and the _TMP
method could use the IT87xxF to read the temperature. In that case,
the BIOS would still need to declare the IT87xxF resources so the OS
doesn't place another device on top of it. An easy way to do this
is with a PNP0C02 "motherboard" device, which just says "these
resources are in use, but the OS needn't attach a driver to this
device."
> And what "collision" are you talking about?
I meant the problem where we place two devices at the same address,
causing one to stop working. I wrote "even if it87 isn't present,"
which is ambiguous -- I meant that we need something that works even
when the it87 _driver_ isn't present.
> >> PCI is an entirely different beast. With PCI you know the PCI device ID
> >> that is associated with the resources, and for a given device, the
> >> resources are always declared (if standard BARs are used) or never
> >> declared; there's no "may be". So it's much easier to handle the
> >> resources properly.
> >
> >That's the way PNP is supposed to work, too (more about this below).
>
> Not really. PNP devices may or may not be declared. PCI devices are
> always declared. This is a fundamental difference. I'm also not sure
> about the PNP IDs, see below.
Well, it's true that PNP does not need to describe ALL devices in the
system. However, it should describe all devices to which the OS should
bind a driver. The BIOS writer has the discretion to hide devices from
the OS by omitting them from the ACPI namespace. If he does that, his
assumption is that the OS will ignore the device (but of course, he's
thinking about a Plug-and-Play-compliant OS like Windows).
> >> That's not really safe, no. They may overlap with other device resources
> >> and prevent those drivers from loading.
> >
> >True. If that happens, I think we definitely need some kind of DMI-
> >based quirk to fix the resources. My points are (a) the quirk needs
> >to be at the PNP level; it can't be in a driver, and (b) I'm lazy and
> >would wait until a problem happens before implementing it, because I
> >know so little about PNP and the specific board, and by waiting, there's
> >a chance I'll learn enough to avoid a mistake :-)
>
> But we _do_ have a problem to solve! That's what started this thread.
> And we need to solve it before 2.6.24 is released. I'd rather (try to)
> do the right thing now rather than going in one direction today and
> doing something different next month.
I think my patch to request only the ports actually used by the it87
driver is sufficient to solve the current problem. If we find that
a BIOS neglects to reserve some of the ports decoded by the IT87xxF,
we can add a PNP quirk to handle that. But I'm not aware of any
problems like that yet.
> >it87 doesn't fit the model very well because usually the BIOS
> >doesn't describe the device explicitly. I think the expectation
> >is that the OS will get sensor readings some other way, such as
> >by evaluating ACPI "_TMP" methods, or by using some OEM-specific
> >ACPI device.
>
> If the IT87xxF device is declared, will it always the same PNP ID on
> every system? I fear that the answer is no, in which case I see no
> simple way to implement the it87 driver as a PNP driver.
PNP drivers can claim multiple PNP IDs, e.g., drivers/serial/8250_pnp.c
recognizes many IDs for 16550-compatible UARTs. But I doubt that
any BIOS actually describes an IT87xxF explicitly.
Bjorn
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: 2.6.24-rc4 hwmon it87 probe fails
2007-12-23 23:14 [lm-sensors] " Bjorn Helgaas
@ 2007-12-25 21:31 ` Jean Delvare
2008-01-02 18:30 ` Bjorn Helgaas
0 siblings, 1 reply; 9+ messages in thread
From: Jean Delvare @ 2007-12-25 21:31 UTC (permalink / raw)
To: bjorn.helgaas
Cc: Shaohua Li, Mike Houston, Adrian Bunk, Elvis Pranskevichus,
Mark M. Hoffman, linux-kernel, lm-sensors, Adam Belay,
Zhao Yakui, Thomas Renninger, lenb, linux-acpi
Hi Bjorn,
Le 23/12/2007, "Bjorn Helgaas" <bjorn.helgaas@hp.com> a écrit:
>On Sunday 23 December 2007 2:28:05 am Jean Delvare wrote:
>> The problem is that the it87 driver is used on a variety of motherboards,
>> some where the hardware monitoring device I/O ports are reserved by the
>> BIOS, some where they aren't. How am I supposed to deal with this?
>
>I assume you mean some boards describe those ports in ACPI, and some
>don't?
Yes, that's what I meant.
>(...) I think the ideal solution would be to figure out how the
>BIOS writers intended the device to be used, and do that. But the
>documentation may be lacking, and it degenerates into an OEM-specific
>mess. Second-best seems like a PNP quirk that pokes enough to determine
>that a Super I/O device is present, and then reserves resources for
>it. Then we avoid the collision even if it87 isn't present.
I'm not sure what you mean here exactly. We have two different cases:
either the ACPI code did declare the PNP device, or it did not. How do
you unify both? And what "collision" are you talking about?
>> PCI is an entirely different beast. With PCI you know the PCI device ID
>> that is associated with the resources, and for a given device, the
>> resources are always declared (if standard BARs are used) or never
>> declared; there's no "may be". So it's much easier to handle the
>> resources properly.
>
>That's the way PNP is supposed to work, too (more about this below).
Not really. PNP devices may or may not be declared. PCI devices are
always declared. This is a fundamental difference. I'm also not sure
about the PNP IDs, see below.
>> That's not really safe, no. They may overlap with other device resources
>> and prevent those drivers from loading.
>
>True. If that happens, I think we definitely need some kind of DMI-
>based quirk to fix the resources. My points are (a) the quirk needs
>to be at the PNP level; it can't be in a driver, and (b) I'm lazy and
>would wait until a problem happens before implementing it, because I
>know so little about PNP and the specific board, and by waiting, there's
>a chance I'll learn enough to avoid a mistake :-)
But we _do_ have a problem to solve! That's what started this thread.
And we need to solve it before 2.6.24 is released. I'd rather (try to)
do the right thing now rather than going in one direction today and
doing something different next month.
I do agree that the quirks should be done at the PNP level and not in the
it87 driver.
>> Another problem is how do I connect this specific I/O port range of the
>> PNP0C02 device with the it87 driver? I am by far no PNP expert but it
>> looks to me like this PNP device covers more than one actual device, and
>> I/O port ranges do not have labels nor identifiers that would let me
>> find the one that corresponds to the IT87xxF device (if it exists at
>> all.)
>
>The general rule is that you have a PNP device with an identifier
>like PNP0500 that means "16550 UART" and some resources underneath
>it. The PNP ID ("PNP0500") tells the OS which driver to bind to the
>device, and the resources tell the driver where the device is. The
>serial driver in drivers/serial/8250_pnp.c is a good example of the
>"normal" way PNP drivers work.
>
>it87 doesn't fit the model very well because usually the BIOS
>doesn't describe the device explicitly. I think the expectation
>is that the OS will get sensor readings some other way, such as
>by evaluating ACPI "_TMP" methods, or by using some OEM-specific
>ACPI device.
If the IT87xxF device is declared, will it always the same PNP ID on
every system? I fear that the answer is no, in which case I see no
simple way to implement the it87 driver as a PNP driver.
>It's very irritating when ACPI is used to take some device that
>would otherwise be nicely generic and machine-independent, and
>wrap it up into some abstract device that requires an OEM-specific
>driver, but I think that's what's happening here. I suspect it
>has to do with the fact that the BIOS wants to retain some
>control over the device so it can deal with things like thermal
>emergencies, even if the OS driver is missing or broken.
Yes, very irritating. Especially when the ACPI way is not only
system-specific, but also very poor in terms of features compared to
what a native driver can do. This is the main reason why we still write
native drivers for hardware monitoring drivers despite the conflicts
with ACPI that arise on some systems.
--
Jean Delvare
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: 2.6.24-rc4 hwmon it87 probe fails
2007-12-10 1:32 ` Ed Sweetman
@ 2007-12-10 14:55 ` Jean Delvare
0 siblings, 0 replies; 9+ messages in thread
From: Jean Delvare @ 2007-12-10 14:55 UTC (permalink / raw)
To: Ed Sweetman
Cc: Mike Houston, Elvis Pranskevichus, Adrian Bunk, mhoffman,
linux-kernel, lm-sensors, Bjorn Helgaas, Adam Belay
Hi Ed,
On Sun, 09 Dec 2007 20:32:41 -0500, Ed Sweetman wrote:
> I'm seeing this exact problem on an Asus Nforce4 based board. Prior to
> moving to 2.6.24-rc4 it worked just fine. No additional acpi options
> were selected in kernel config.
>
> So add Asus A8N-E to the list of broken pnpacpi
For the records, can you please paste the offending lines (from the
boot logs or /proc/ioports)?
Then please give a try to Shaohua's patch and report:
http://lkml.org/lkml/diff/2007/12/9/186/1
Thanks,
--
Jean Delvare
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: 2.6.24-rc4 hwmon it87 probe fails
2007-12-09 0:05 ` Adrian Bunk
@ 2007-12-09 2:22 ` Mike Houston
2007-12-09 9:50 ` [lm-sensors] " Jean Delvare
0 siblings, 1 reply; 9+ messages in thread
From: Mike Houston @ 2007-12-09 2:22 UTC (permalink / raw)
To: Adrian Bunk; +Cc: linux-kernel, mhoffman, lm-sensors
On Sun, 9 Dec 2007 01:05:54 +0100
Adrian Bunk <bunk@stusta.de> wrote:
> On Tue, Dec 04, 2007 at 09:51:54PM -0500, Mike Houston wrote:
> > I finally got around to testing Linux 2.6.24 (2.6.24-rc4) and
> > found that the it87 driver fails to probe and consequently, my
> > sensors no longer work. This was fine with Linux 2.6.23.8 (the
> > last kernel I was using)
> >
> > The necessary modules load, but:
> >
> > it87: Found IT8718F chip at 0x290, revision 2
> > it87: in3 is VCC (+5V)
> > it87 it87.656: Failed to request region 0x290-0x297
> > it87: probe of it87.656 failed with error -16
> >
> > Coretemp still works.
> >
> > It appears it has something to do with the ioport range being
> > reserved for some reason:
> >
> > system 00:01: ioport range 0x290-0x29f has been reserved
>
> Thanks for your report.
>
> Please also provide:
> - dmesg from 2.6.23.8
> - The output of "cat /proc/ioports" for both kernels
Thanks Adrian, here is the information you have requested, for
both kernels (I have 2.6.23.9 now though where it87 still works)
Linux 2.6.23.9:
http://www.mikeserv.com/temp/proc_ioports-2.6.23.9.txt
http://www.mikeserv.com/temp/dmesg-2.6.23.9.txt
http://www.mikeserv.com/temp/config-2.6.23.9.txt
Linux 2.6.24-rc4:
http://www.mikeserv.com/temp/proc_ioports-2.6.24-rc4.txt
http://www.mikeserv.com/temp/dmesg-2.6.24-rc4.txt
http://www.mikeserv.com/temp/config-2.6.24-rc4.txt
Mike Houston
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: 2.6.24-rc4 hwmon it87 probe fails
2007-12-05 2:51 Mike Houston
@ 2007-12-09 0:05 ` Adrian Bunk
2007-12-09 2:22 ` Mike Houston
0 siblings, 1 reply; 9+ messages in thread
From: Adrian Bunk @ 2007-12-09 0:05 UTC (permalink / raw)
To: Mike Houston; +Cc: linux-kernel, mhoffman, lm-sensors
On Tue, Dec 04, 2007 at 09:51:54PM -0500, Mike Houston wrote:
> I finally got around to testing Linux 2.6.24 (2.6.24-rc4) and found
> that the it87 driver fails to probe and consequently, my sensors no
> longer work. This was fine with Linux 2.6.23.8 (the last kernel I was
> using)
>
> The necessary modules load, but:
>
> it87: Found IT8718F chip at 0x290, revision 2
> it87: in3 is VCC (+5V)
> it87 it87.656: Failed to request region 0x290-0x297
> it87: probe of it87.656 failed with error -16
>
> Coretemp still works.
>
> It appears it has something to do with the ioport range being
> reserved for some reason:
>
> system 00:01: ioport range 0x290-0x29f has been reserved
>
> At least I don't see that happening on previous kernels.
>
> In any case, the changes to it87.c itself don't appear to have caused
> this. (CC'd hwmon maintainer anyways, my apologies if that was
> inappropriate)
>
> dmesg:
> http://www.mikeserv.com/temp/dmesg-2.6.24-rc4.txt
>
> config:
> http://www.mikeserv.com/temp/config-2.6.24-rc4.txt
Thanks for your report.
Please also provide:
- dmesg from 2.6.23.8
- The output of "cat /proc/ioports" for both kernels
> Thanks for any help or suggestions,
>
> Mike Houston
cu
Adrian
--
"Is there not promise of rain?" Ling Tan asked suddenly out
of the darkness. There had been need of rain for many days.
"Only a promise," Lao Er said.
Pearl S. Buck - Dragon Seed
^ permalink raw reply [flat|nested] 9+ messages in thread
* 2.6.24-rc4 hwmon it87 probe fails
@ 2007-12-05 2:51 Mike Houston
2007-12-09 0:05 ` Adrian Bunk
0 siblings, 1 reply; 9+ messages in thread
From: Mike Houston @ 2007-12-05 2:51 UTC (permalink / raw)
To: linux-kernel; +Cc: mhoffman
I finally got around to testing Linux 2.6.24 (2.6.24-rc4) and found
that the it87 driver fails to probe and consequently, my sensors no
longer work. This was fine with Linux 2.6.23.8 (the last kernel I was
using)
The necessary modules load, but:
it87: Found IT8718F chip at 0x290, revision 2
it87: in3 is VCC (+5V)
it87 it87.656: Failed to request region 0x290-0x297
it87: probe of it87.656 failed with error -16
Coretemp still works.
It appears it has something to do with the ioport range being
reserved for some reason:
system 00:01: ioport range 0x290-0x29f has been reserved
At least I don't see that happening on previous kernels.
In any case, the changes to it87.c itself don't appear to have caused
this. (CC'd hwmon maintainer anyways, my apologies if that was
inappropriate)
dmesg:
http://www.mikeserv.com/temp/dmesg-2.6.24-rc4.txt
config:
http://www.mikeserv.com/temp/config-2.6.24-rc4.txt
Thanks for any help or suggestions,
Mike Houston
^ permalink raw reply [flat|nested] 9+ messages in thread
end of thread, other threads:[~2008-01-12 9:57 UTC | newest]
Thread overview: 9+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
[not found] <fa.QPUBl9Xd2PDsImgWn6hbR+ShV1U@ifi.uio.no>
[not found] ` <fa.ri5Klmu4+MwjYC8x5nSms+xKXfI@ifi.uio.no>
[not found] ` <fa.B/J+j9kGGZ1+gW9FrfHky+cj0Eo@ifi.uio.no>
[not found] ` <fa.Xv3BmMxnCGLOeEijySte5mKDO5k@ifi.uio.no>
2007-12-20 1:09 ` [lm-sensors] 2.6.24-rc4 hwmon it87 probe fails Robert Hancock
2008-01-12 9:56 ` Jean Delvare
2007-12-23 23:14 [lm-sensors] " Bjorn Helgaas
2007-12-25 21:31 ` Jean Delvare
2008-01-02 18:30 ` Bjorn Helgaas
2008-01-12 9:49 ` Jean Delvare
-- strict thread matches above, loose matches on Subject: below --
2007-12-05 2:51 Mike Houston
2007-12-09 0:05 ` Adrian Bunk
2007-12-09 2:22 ` Mike Houston
2007-12-09 9:50 ` [lm-sensors] " Jean Delvare
2007-12-09 21:12 ` Elvis Pranskevichus
2007-12-09 22:42 ` Jean Delvare
2007-12-10 0:19 ` Mike Houston
2007-12-10 1:32 ` Ed Sweetman
2007-12-10 14:55 ` Jean Delvare
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
Powered by JetHome