* Re: Hard lockup debugging suggestions? APIC enabling suggestions?
@ 2001-05-30 10:56 Mikael Pettersson
0 siblings, 0 replies; 4+ messages in thread
From: Mikael Pettersson @ 2001-05-30 10:56 UTC (permalink / raw)
To: linux-kernel, shag-linux-kernel
On Tue, 29 May 2001 18:22:57 -0500 (CDT), Paul Walmsley wrote:
>I have an 700Mhz Pentium III HP Omnibook 6000 that has been locking up
>...
>In the hopes of getting an oops from the NMI watchdog, I attempted to
>enable the NMI watchdog with both 'nmi_watchdog=1' and 'nmi_watchdog=2'.
>Neither seems to work. (Support for APIC and IO-APIC is compiled into the
>kernel, although /proc/interrupts reveals that all interrupts are still
>being routed through the XT-PIC)
>
>It would seem that Linux is having trouble enabling the CPU's local APIC.
>>From the boot messages:
>
> Local APIC disabled by BIOS -- reenabling.
> Could not enable APIC!
Unfortunately Intel removed the local APIC from their Mobile P6 range,
so you won't be able to use the NMI watchdog on your Omnibook.
This is actually documented (for the Mobile PII at least) in some
obscure place, probably in the spec update.
Oh, and remove IO_APIC support too. Your machine doesn't have one.
/Mikael
^ permalink raw reply [flat|nested] 4+ messages in thread
* Hard lockup debugging suggestions? APIC enabling suggestions?
@ 2001-05-29 23:22 Paul Walmsley
2001-05-29 23:25 ` Alan Cox
0 siblings, 1 reply; 4+ messages in thread
From: Paul Walmsley @ 2001-05-29 23:22 UTC (permalink / raw)
To: linux-kernel
[-- Attachment #1: Type: TEXT/PLAIN, Size: 1178 bytes --]
Hello,
I have an 700Mhz Pentium III HP Omnibook 6000 that has been locking up
hard several times per day under Linux 2.4.4-ac11. This lockup occurs
during standard use of the system, e.g., web browsing or text editing.
(What's particularly strange about the lockup is that sometimes the system
will turn off the LCD backlight when it freezes -- but not the LCD panel
itself. Other times, it freezes with the backlight on.)
In the hopes of getting an oops from the NMI watchdog, I attempted to
enable the NMI watchdog with both 'nmi_watchdog=1' and 'nmi_watchdog=2'.
Neither seems to work. (Support for APIC and IO-APIC is compiled into the
kernel, although /proc/interrupts reveals that all interrupts are still
being routed through the XT-PIC)
It would seem that Linux is having trouble enabling the CPU's local APIC.
>From the boot messages:
Local APIC disabled by BIOS -- reenabling.
Could not enable APIC!
I was curious to know if there's anything I can do to enable the APIC.
Also, if anyone has any other suggestions for troubleshooting these hard
lockups, I would love to hear them :-)
(I've attached 'dmesg' output and 'lspci -vv' output.)
Thanks,
- Paul
[-- Attachment #2: dmesg --]
[-- Type: TEXT/PLAIN, Size: 6955 bytes --]
Linux version 2.4.4-ac11 (root@localhost.localdomain) (gcc version 2.95.3 19991030 (prerelease)) #8 Sun May 27 14:09:18 MDT 2001
BIOS-provided physical RAM map:
BIOS-e820: 0000000000000000 - 000000000009f800 (usable)
BIOS-e820: 000000000009f800 - 00000000000a0000 (reserved)
BIOS-e820: 00000000000e8400 - 0000000000100000 (reserved)
BIOS-e820: 0000000000100000 - 0000000009ff0000 (usable)
BIOS-e820: 0000000009ff0000 - 0000000009fffc00 (ACPI data)
BIOS-e820: 0000000009fffc00 - 000000000a000000 (ACPI NVS)
BIOS-e820: 00000000fff80000 - 0000000100000000 (reserved)
On node 0 totalpages: 40944
zone(0): 4096 pages.
zone(1): 36848 pages.
zone(2): 0 pages.
Local APIC disabled by BIOS -- reenabling.
Could not enable APIC!
Kernel command line: root=/dev/hda6 nmi_watchdog=2 ide1=autotune ide0=autotune mem=163776K
ide_setup: ide1=autotune
ide_setup: ide0=autotune
Initializing CPU#0
Detected 696.981 MHz processor.
Console: colour VGA+ 80x25
Calibrating delay loop... 1389.36 BogoMIPS
Memory: 158736k/163776k available (1067k kernel code, 4652k reserved, 402k data, 192k init, 0k highmem)
Dentry-cache hash table entries: 32768 (order: 6, 262144 bytes)
Buffer-cache hash table entries: 8192 (order: 3, 32768 bytes)
Page-cache hash table entries: 65536 (order: 6, 262144 bytes)
Inode-cache hash table entries: 16384 (order: 5, 131072 bytes)
CPU: Before vendor init, caps: 0383f9ff 00000000 00000000, vendor = 0
CPU: L1 I cache: 16K, L1 D cache: 16K
CPU: L2 cache: 256K
Intel machine check architecture supported.
Intel machine check reporting enabled on CPU#0.
CPU: After vendor init, caps: 0383f9ff 00000000 00000000 00000000
CPU: After generic, caps: 0383f9ff 00000000 00000000 00000000
CPU: Common caps: 0383f9ff 00000000 00000000 00000000
CPU: Intel Pentium III (Coppermine) stepping 06
Enabling fast FPU save and restore... done.
Enabling unmasked SIMD FPU exception support... done.
Checking 'hlt' instruction... OK.
POSIX conformance testing by UNIFIX
mtrr: v1.40 (20010327) Richard Gooch (rgooch@atnf.csiro.au)
mtrr: detected mtrr type: Intel
PCI: PCI BIOS revision 2.10 entry at 0xfd980, last bus=1
PCI: Using configuration type 1
PCI: Probing PCI hardware
Unknown bridge resource 2: assuming transparent
PCI: Using IRQ router PIIX [8086/7110] at 00:07.0
PCI: Found IRQ 10 for device 00:0a.0
PCI: The same IRQ used for device 00:0b.0
PCI: The same IRQ used for device 00:0b.1
PCI: Found IRQ 10 for device 00:0a.1
PCI: The same IRQ used for device 01:00.0
got res[a000000:a000fff] for resource 0 of Texas Instruments PCI1420
got res[a001000:a001fff] for resource 0 of Texas Instruments PCI1420 (#2)
Limiting direct PCI/PCI transfers.
PnP: PNP BIOS installation structure at 0xc00f67f0
PnP: PNP BIOS version 1.0, entry at f0000:9c4d, dseg at 400
Linux NET4.0 for Linux 2.4
Based upon Swansea University Computer Society NET3.039
Initializing RT netlink socket
apm: BIOS version 1.2 Flags 0x03 (Driver version 1.14)
Starting kswapd v1.8
block: queued sectors max/low 105274kB/35091kB, 320 slots per queue
Uniform Multi-Platform E-IDE driver Revision: 6.31
ide: Assuming 33MHz system bus speed for PIO modes; override with idebus=xx
PIIX4: IDE controller on PCI bus 00 dev 39
PIIX4: chipset revision 1
PIIX4: not 100% native mode: will probe irqs later
ide0: BM-DMA at 0x18a0-0x18a7, BIOS settings: hda:DMA, hdb:pio
ide1: BM-DMA at 0x18a8-0x18af, BIOS settings: hdc:pio, hdd:pio
hda: IBM-DJSA-230, ATA DISK drive
hdc: TOSHIBA DVD-ROM SD-C2402, ATAPI CD/DVD-ROM drive
ide0 at 0x1f0-0x1f7,0x3f6 on irq 14
ide1 at 0x170-0x177,0x376 on irq 15
hda: 58605120 sectors (30006 MB) w/1874KiB Cache, CHS=3876/240/63
hdc: ATAPI 24X DVD-ROM drive, 128kB Cache
Uniform CD-ROM driver Revision: 3.12
Partition check:
hda: hda1 hda2 hda3 hda4 < hda5 hda6 >
Floppy drive(s): fd0 is 1.44M
FDC 0 is a National Semiconductor PC87306
pty: 256 Unix98 ptys configured
Serial driver version 5.05b (2001-05-03) with MANY_PORTS SHARE_IRQ SERIAL_PCI enabled
ttyS00 at 0x03f8 (irq = 4) is a 16550A
ttyS01 at 0x02f8 (irq = 3) is a 16550A
Real Time Clock Driver v1.10d
Software Watchdog Timer: 0.05, timer margin: 60 sec
Linux PCMCIA Card Services 3.1.22
options: [pci] [cardbus] [pm]
PCI: Found IRQ 10 for device 00:0a.0
PCI: The same IRQ used for device 00:0b.0
PCI: The same IRQ used for device 00:0b.1
PCI: Found IRQ 10 for device 00:0a.1
PCI: The same IRQ used for device 01:00.0
Intel PCIC probe: not found.
NET4: Linux TCP/IP 1.0 for NET4.0
IP Protocols: ICMP, UDP, TCP, IGMP
IP: routing cache hash table of 1024 buckets, 8Kbytes
TCP: Hash tables configured (established 16384 bind 16384)
ip_conntrack (1279 buckets, 10232 max)
ip_tables: (c)2000 Netfilter core team
NET4: Unix domain sockets 1.0/SMP for Linux NET4.0.
Yenta IRQ list 0a98, PCI irq10
Socket status: 30000006
Yenta IRQ list 0a98, PCI irq10
Socket status: 30000010
fatfs: bogus cluster size
fatfs: bogus cluster size
reiserfs: checking transaction log (device 03:06) ...
Warning, log replay starting on readonly filesystem
reiserfs: replayed 3 transactions in 7 seconds
Using r5 hash to sort names
ReiserFS version 3.6.25
VFS: Mounted root (reiserfs filesystem) readonly.
Freeing unused kernel memory: 192k freed
VFS: Disk change detected on device ide1(22,0)
Adding Swap: 317480k swap-space (priority -1)
cs: IO port probe 0x0c00-0x0cff: clean.
cs: IO port probe 0x0800-0x08ff: clean.
cs: IO port probe 0x0100-0x04ff: excluding 0x378-0x37f 0x398-0x39f 0x4d0-0x4d7
cs: IO port probe 0x0a00-0x0aff: clean.
cs: memory probe 0xa0000000-0xa0ffffff: clean.
PCI: Found IRQ 5 for device 00:0d.0
hermes.c: 12 Dec 2000 David Gibson <hermes@gibson.dropbear.id.au>
orinoco.c 0.05d (David Gibson <hermes@gibson.dropbear.id.au> and others)
dldwd: David's Less Dodgy WaveLAN/IEEE Driver
eth0: Firmware ID 1F vendor 0x1 (Lucent) version 7.28
eth0: Ad-hoc demo mode supported.
eth0: IEEE standard IBSS ad-hoc mode supported.
eth0: WEP supported, "128"-bit key.
eth0: MAC address 00:02:2D:0D:12:9F
eth0: Station name "HERMES I"
eth0: ready
eth0: index 0x01: Vcc 5.0, irq 9, io 0x0100-0x013f
usb.c: registered new driver hub
PCI: Found IRQ 10 for device 00:07.2
uhci.c: USB UHCI at I/O 0x1880, IRQ 10
usb.c: new USB bus registered, assigned bus number 1
hub.c: USB hub found
hub.c: 2 ports detected
PnPBIOS: Parport found PNPBIOS PNP0401 at io=0378,0778 irq=7 dma=3
0x378: FIFO is 16 bytes
0x378: writeIntrThreshold is 8
0x378: readIntrThreshold is 8
0x378: PWord is 8 bits
0x378: Interrupts are ISA-Pulses
0x378: ECP port cfgA=0x14 cfgB=0x4b
0x378: ECP settings irq=7 dma=3
parport0: PC-style at 0x378 (0x778), irq 7, dma 3 [PCSPP,TRISTATE,COMPAT,ECP,DMA]
parport0: cpp_daisy: aa5500ff(38)
parport0: assign_addrs: aa5500ff(38)
parport0: cpp_daisy: aa5500ff(38)
parport0: assign_addrs: aa5500ff(38)
lp0: using parport0 (interrupt-driven).
[-- Attachment #3: lspci -vv --]
[-- Type: TEXT/PLAIN, Size: 7032 bytes --]
00:00.0 Host bridge: Intel Corporation 440BX/ZX - 82443BX/ZX Host bridge (rev 03)
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B-
Status: Cap+ 66Mhz- UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort+ >SERR- <PERR-
Latency: 64
Region 0: Memory at f8000000 (32-bit, prefetchable) [size=64M]
Capabilities: [a0] AGP version 1.0
Status: RQ=31 SBA+ 64bit- FW- Rate=x1,x2
Command: RQ=0 SBA- AGP- 64bit- FW- Rate=<none>
00:01.0 PCI bridge: Intel Corporation 440BX/ZX - 82443BX/ZX AGP bridge (rev 03) (prog-if 00 [Normal decode])
Control: I/O+ Mem+ BusMaster+ SpecCycle+ MemWINV+ VGASnoop- ParErr- Stepping- SERR- FastB2B-
Status: Cap- 66Mhz+ UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR-
Latency: 128
Bus: primary=00, secondary=01, subordinate=01, sec-latency=64
I/O behind bridge: 00009000-00009fff
Memory behind bridge: f4100000-f5ffffff
BridgeCtl: Parity- SERR- NoISA+ VGA+ MAbort- >Reset- FastB2B+
00:07.0 Bridge: Intel Corporation 82371AB PIIX4 ISA (rev 02)
Control: I/O+ Mem+ BusMaster+ SpecCycle+ MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B-
Status: Cap- 66Mhz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR-
Latency: 0
00:07.1 IDE interface: Intel Corporation 82371AB PIIX4 IDE (rev 01) (prog-if 80 [Master])
Control: I/O+ Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B-
Status: Cap- 66Mhz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR-
Latency: 64
Region 4: I/O ports at 18a0 [size=16]
00:07.2 USB Controller: Intel Corporation 82371AB PIIX4 USB (rev 01) (prog-if 00 [UHCI])
Control: I/O+ Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B-
Status: Cap- 66Mhz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR-
Latency: 64
Interrupt: pin D routed to IRQ 10
Region 4: I/O ports at 1880 [size=32]
00:07.3 Bridge: Intel Corporation 82371AB PIIX4 ACPI (rev 03)
Control: I/O+ Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B-
Status: Cap- 66Mhz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR-
Interrupt: pin ? routed to IRQ 9
00:0a.0 CardBus bridge: Texas Instruments PCI1420
Subsystem: Hewlett-Packard Company: Unknown device 0010
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B-
Status: Cap+ 66Mhz- UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR-
Latency: 168, cache line size 08
Interrupt: pin A routed to IRQ 10
Region 0: Memory at 0a000000 (32-bit, non-prefetchable) [size=4K]
Bus: primary=00, secondary=02, subordinate=02, sec-latency=176
Memory window 0: 0a400000-0a7ff000 (prefetchable)
Memory window 1: 0a800000-0abff000
I/O window 0: 00001c00-00001cff
I/O window 1: 00002000-000020ff
BridgeCtl: Parity- SERR- ISA- VGA- MAbort- >Reset+ 16bInt+ PostWrite+
16-bit legacy interface ports at 0001
00:0a.1 CardBus bridge: Texas Instruments PCI1420
Subsystem: Hewlett-Packard Company: Unknown device 0010
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B-
Status: Cap+ 66Mhz- UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR-
Latency: 168, cache line size 08
Interrupt: pin B routed to IRQ 10
Region 0: Memory at 0a001000 (32-bit, non-prefetchable) [size=4K]
Bus: primary=00, secondary=06, subordinate=06, sec-latency=176
Memory window 0: 0ac00000-0afff000 (prefetchable)
Memory window 1: 0b000000-0b3ff000
I/O window 0: 00002400-000024ff
I/O window 1: 00002800-000028ff
BridgeCtl: Parity- SERR- ISA- VGA- MAbort- >Reset+ 16bInt+ PostWrite+
16-bit legacy interface ports at 0001
00:0b.0 Ethernet controller: 3Com Corporation 3c556 Laptop Tornado (rev 10)
Subsystem: 3Com Corporation: Unknown device 6256
Control: I/O+ Mem+ BusMaster- SpecCycle- MemWINV+ VGASnoop- ParErr- Stepping- SERR- FastB2B-
Status: Cap+ 66Mhz- UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR-
Interrupt: pin A routed to IRQ 10
Region 0: I/O ports at 1800 [size=128]
Region 1: Memory at f4002400 (32-bit, non-prefetchable) [size=128]
Region 2: Memory at f4002000 (32-bit, non-prefetchable) [size=128]
Expansion ROM at <unassigned> [disabled] [size=128K]
Capabilities: [50] Power Management version 1
Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA PME(D0-,D1+,D2+,D3hot+,D3cold+)
Status: D0 PME-Enable- DSel=0 DScale=1 PME-
00:0b.1 Communication controller: 3Com Corporation: Unknown device 1007 (rev 10)
Subsystem: 3Com Corporation: Unknown device 6158
Control: I/O+ Mem+ BusMaster- SpecCycle- MemWINV+ VGASnoop- ParErr- Stepping- SERR- FastB2B-
Status: Cap+ 66Mhz- UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR-
Interrupt: pin A routed to IRQ 10
Region 0: I/O ports at 1000 [size=256]
Region 1: Memory at f4002c00 (32-bit, non-prefetchable) [size=256]
Region 2: Memory at f4002800 (32-bit, non-prefetchable) [size=128]
Capabilities: [50] Power Management version 1
Flags: PMEClk- DSI- D1- D2+ AuxCurrent=0mA PME(D0-,D1-,D2+,D3hot+,D3cold+)
Status: D0 PME-Enable- DSel=0 DScale=1 PME-
00:0d.0 Multimedia audio controller: ESS Technology ES1983S Maestro-3i PCI Audio Accelerator
Subsystem: Hewlett-Packard Company: Unknown device 0010
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B-
Status: Cap+ 66Mhz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR-
Latency: 64 (500ns min, 6000ns max)
Interrupt: pin A routed to IRQ 5
Region 0: I/O ports at 1400 [size=256]
Region 1: Memory at f4000000 (32-bit, non-prefetchable) [size=8K]
Capabilities: [c0] Power Management version 2
Flags: PMEClk- DSI+ D1+ D2+ AuxCurrent=0mA PME(D0-,D1+,D2+,D3hot+,D3cold-)
Status: D0 PME-Enable- DSel=0 DScale=0 PME-
01:00.0 VGA compatible controller: ATI Technologies Inc 3D Rage P/M Mobility AGP 2x (rev 64) (prog-if 00 [VGA])
Subsystem: Hewlett-Packard Company: Unknown device 0010
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping+ SERR- FastB2B-
Status: Cap+ 66Mhz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR-
Latency: 66 (2000ns min), cache line size 08
Interrupt: pin A routed to IRQ 10
Region 0: Memory at f5000000 (32-bit, non-prefetchable) [size=16M]
Region 1: I/O ports at 9000 [size=256]
Region 2: Memory at f4100000 (32-bit, non-prefetchable) [size=4K]
Expansion ROM at <unassigned> [disabled] [size=128K]
Capabilities: [50] AGP version 1.0
Status: RQ=255 SBA+ 64bit- FW- Rate=x1,x2
Command: RQ=0 SBA- AGP- 64bit- FW- Rate=<none>
Capabilities: [5c] Power Management version 1
Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 PME-Enable- DSel=0 DScale=0 PME-
^ permalink raw reply [flat|nested] 4+ messages in thread* Re: Hard lockup debugging suggestions? APIC enabling suggestions?
2001-05-29 23:22 Paul Walmsley
@ 2001-05-29 23:25 ` Alan Cox
2001-06-13 2:32 ` Paul Walmsley
0 siblings, 1 reply; 4+ messages in thread
From: Alan Cox @ 2001-05-29 23:25 UTC (permalink / raw)
To: Paul Walmsley; +Cc: linux-kernel
> hard several times per day under Linux 2.4.4-ac11. This lockup occurs
> during standard use of the system, e.g., web browsing or text editing.
> (What's particularly strange about the lockup is that sometimes the system
> will turn off the LCD backlight when it freezes -- but not the LCD panel
> itself. Other times, it freezes with the backlight on.)
First things to try: Can you make it die without X, can you make it die if
you compile without APM or ACPI support.
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: Hard lockup debugging suggestions? APIC enabling suggestions?
2001-05-29 23:25 ` Alan Cox
@ 2001-06-13 2:32 ` Paul Walmsley
0 siblings, 0 replies; 4+ messages in thread
From: Paul Walmsley @ 2001-06-13 2:32 UTC (permalink / raw)
To: Alan Cox; +Cc: linux-kernel
On Wed, 30 May 2001, Alan Cox wrote:
> > hard several times per day under Linux 2.4.4-ac11. This lockup occurs
> > during standard use of the system, e.g., web browsing or text editing.
> > (What's particularly strange about the lockup is that sometimes the system
> > will turn off the LCD backlight when it freezes -- but not the LCD panel
> > itself. Other times, it freezes with the backlight on.)
>
> First things to try: Can you make it die without X, can you make it die if
> you compile without APM or ACPI support.
Further exhaustive testing along the lines you describe above has led me
to believe this is an intermittent hardware problem on the notebook's
motherboard. I'm able to trigger the hard lockup both in Linux and in
Windows 2000. It appears to be triggered by stress on a certain part of
the notebook's case.
I'm having the motherboard replaced. If it turns out not to be the
problem, I'll follow up here.
Thanks for the help,
- Paul
^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2001-06-13 2:33 UTC | newest]
Thread overview: 4+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2001-05-30 10:56 Hard lockup debugging suggestions? APIC enabling suggestions? Mikael Pettersson
-- strict thread matches above, loose matches on Subject: below --
2001-05-29 23:22 Paul Walmsley
2001-05-29 23:25 ` Alan Cox
2001-06-13 2:32 ` Paul Walmsley
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®