[2.6.36-rc2 regression] X201s: resume-from-S2R hangs

[2.6.36-rc2 regression] X201s: resume-from-S2R hangs

[Andy Isaacson]

[-- Attachment #1: Type: text/plain, Size: 1574 bytes --]

I have a Thinkpad X201s on which suspend-to-RAM was quite reliable until
after 2.6.36-rc1; with problematic kernels, suspend seems to work OK
(the machine goes to sleep with the moon LED steadily lit) but on resume
the LCD never lights up (and it's not just the backlight, the screen
isn't drawing at all) and the keyboard and network are dead.  Nothing
makes it to syslog, either.  The moon LED is turned off though.

There seems to be some timing component to the problem -- some problem
kernels will suspend reliably 25% or 50% of the time and hang the rest.
There also seems to be some per-reboot state involved; one problem
kernel suspended successfully and survived 25 suspends in a row without
triggering a failure.  (But after a reboot, the first suspend hung...)
Other problem kernels fail on 4 out of 4 reboots.

Bisecting is pretty hard under these conditions, but it seems to point
to

commit ce17178094f368d9e3f39b2cb4303da5ed633dd4
Author: Zou Nan hai <nanhai.zou@intel.com>
Date:   Fri Jun 25 13:40:22 2010 +0800

    drm/i915: Enable RC6 on Ironlake.
    
    RC6 allows the GPU to enter a lower power state when the GPU is idle.
    
    Signed-off-by: Zou Nan hai <nanhai.zou@intel.com>
    [anholt: Fixed the !renderctx error path to actually not enable RC6.]
    Signed-off-by: Eric Anholt <eric@anholt.net>

Reverting that commit on top of -rc3 seems to fix the problem -- three
reboots so far, no failures yet.

The machine has a "Core(TM) i7 CPU L 640", 4GB of RAM, and BIOS 1.05.
I've attached lspci and dmesg.gz from a successful boot + S2R.

-andy

[-- Attachment #2: lspci.txt --]
[-- Type: text/plain, Size: 6189 bytes --]

00:00.0 Host bridge: Intel Corporation Core Processor DRAM Controller (rev 02)
	Subsystem: Lenovo Device 2193
	Flags: bus master, fast devsel, latency 0
	Capabilities: <access denied>
	Kernel driver in use: agpgart-intel

00:02.0 VGA compatible controller: Intel Corporation Core Processor Integrated Graphics Controller (rev 02) (prog-if 00 [VGA controller])
	Subsystem: Lenovo Device 215a
	Flags: bus master, fast devsel, latency 0, IRQ 46
	Memory at f2000000 (64-bit, non-prefetchable) [size=4M]
	Memory at d0000000 (64-bit, prefetchable) [size=256M]
	I/O ports at 1800 [size=8]
	Expansion ROM at <unassigned> [disabled]
	Capabilities: <access denied>
	Kernel driver in use: i915

00:16.0 Communication controller: Intel Corporation 5 Series/3400 Series Chipset HECI Controller (rev 06)
	Subsystem: Lenovo Device 215f
	Flags: bus master, fast devsel, latency 0, IRQ 11
	Memory at f2727800 (64-bit, non-prefetchable) [size=16]
	Capabilities: <access denied>

00:19.0 Ethernet controller: Intel Corporation 82577LM Gigabit Network Connection (rev 06)
	Subsystem: Lenovo Device 2153
	Flags: bus master, fast devsel, latency 0, IRQ 43
	Memory at f2500000 (32-bit, non-prefetchable) [size=128K]
	Memory at f2525000 (32-bit, non-prefetchable) [size=4K]
	I/O ports at 1820 [size=32]
	Capabilities: <access denied>
	Kernel driver in use: e1000e

00:1a.0 USB Controller: Intel Corporation 5 Series/3400 Series Chipset USB2 Enhanced Host Controller (rev 06) (prog-if 20 [EHCI])
	Subsystem: Lenovo Device 2163
	Flags: bus master, medium devsel, latency 0, IRQ 23
	Memory at f2728000 (32-bit, non-prefetchable) [size=1K]
	Capabilities: <access denied>
	Kernel driver in use: ehci_hcd

00:1b.0 Audio device: Intel Corporation 5 Series/3400 Series Chipset High Definition Audio (rev 06)
	Subsystem: Lenovo Device 215e
	Flags: bus master, fast devsel, latency 0, IRQ 47
	Memory at f2520000 (64-bit, non-prefetchable) [size=16K]
	Capabilities: <access denied>
	Kernel driver in use: HDA Intel

00:1c.0 PCI bridge: Intel Corporation 5 Series/3400 Series Chipset PCI Express Root Port 1 (rev 06) (prog-if 00 [Normal decode])
	Flags: bus master, fast devsel, latency 0
	Bus: primary=00, secondary=0d, subordinate=0d, sec-latency=0
	Capabilities: <access denied>
	Kernel driver in use: pcieport

00:1c.3 PCI bridge: Intel Corporation 5 Series/3400 Series Chipset PCI Express Root Port 4 (rev 06) (prog-if 00 [Normal decode])
	Flags: bus master, fast devsel, latency 0
	Bus: primary=00, secondary=05, subordinate=0c, sec-latency=0
	I/O behind bridge: 00002000-00002fff
	Memory behind bridge: f0000000-f1ffffff
	Prefetchable memory behind bridge: 00000000f2800000-00000000f28fffff
	Capabilities: <access denied>
	Kernel driver in use: pcieport

00:1c.4 PCI bridge: Intel Corporation 5 Series/3400 Series Chipset PCI Express Root Port 5 (rev 06) (prog-if 00 [Normal decode])
	Flags: bus master, fast devsel, latency 0
	Bus: primary=00, secondary=02, subordinate=02, sec-latency=0
	Memory behind bridge: f2400000-f24fffff
	Capabilities: <access denied>
	Kernel driver in use: pcieport

00:1d.0 USB Controller: Intel Corporation 5 Series/3400 Series Chipset USB2 Enhanced Host Controller (rev 06) (prog-if 20 [EHCI])
	Subsystem: Lenovo Device 2163
	Flags: bus master, medium devsel, latency 0, IRQ 19
	Memory at f2728400 (32-bit, non-prefetchable) [size=1K]
	Capabilities: <access denied>
	Kernel driver in use: ehci_hcd

00:1e.0 PCI bridge: Intel Corporation 82801 Mobile PCI Bridge (rev a6) (prog-if 01 [Subtractive decode])
	Flags: bus master, fast devsel, latency 0
	Bus: primary=00, secondary=0e, subordinate=0e, sec-latency=0
	Capabilities: <access denied>

00:1f.0 ISA bridge: Intel Corporation Mobile 5 Series Chipset LPC Interface Controller (rev 06)
	Subsystem: Lenovo Device 2166
	Flags: bus master, medium devsel, latency 0
	Capabilities: <access denied>

00:1f.2 SATA controller: Intel Corporation 5 Series/3400 Series Chipset 6 port SATA AHCI Controller (rev 06) (prog-if 01 [AHCI 1.0])
	Subsystem: Lenovo Device 2168
	Flags: bus master, 66MHz, medium devsel, latency 0, IRQ 44
	I/O ports at 1860 [size=8]
	I/O ports at 1814 [size=4]
	I/O ports at 1818 [size=8]
	I/O ports at 1810 [size=4]
	I/O ports at 1840 [size=32]
	Memory at f2727000 (32-bit, non-prefetchable) [size=2K]
	Capabilities: <access denied>
	Kernel driver in use: ahci

00:1f.3 SMBus: Intel Corporation 5 Series/3400 Series Chipset SMBus Controller (rev 06)
	Subsystem: Lenovo Device 2167
	Flags: medium devsel, IRQ 23
	Memory at f2728800 (64-bit, non-prefetchable) [size=256]
	I/O ports at 1880 [size=32]
	Kernel driver in use: i801_smbus

00:1f.6 Signal processing controller: Intel Corporation 5 Series/3400 Series Chipset Thermal Subsystem (rev 06)
	Subsystem: Lenovo Device 2190
	Flags: fast devsel, IRQ 19
	Memory at f2526000 (64-bit, non-prefetchable) [size=4K]
	Capabilities: <access denied>
	Kernel driver in use: intel ips

02:00.0 Network controller: Intel Corporation Centrino Advanced-N + WiMAX 6250 (rev 35)
	Subsystem: Intel Corporation Centrino Advanced-N + WiMAX 6250 2x2 AGN
	Flags: bus master, fast devsel, latency 0, IRQ 45
	Memory at f2400000 (64-bit, non-prefetchable) [size=8K]
	Capabilities: <access denied>
	Kernel driver in use: iwlagn

ff:00.0 Host bridge: Intel Corporation Core Processor QuickPath Architecture Generic Non-core Registers (rev 02)
	Subsystem: Lenovo Device 2196
	Flags: bus master, fast devsel, latency 0

ff:00.1 Host bridge: Intel Corporation Core Processor QuickPath Architecture System Address Decoder (rev 02)
	Subsystem: Lenovo Device 2196
	Flags: bus master, fast devsel, latency 0

ff:02.0 Host bridge: Intel Corporation Core Processor QPI Link 0 (rev 02)
	Subsystem: Lenovo Device 2196
	Flags: bus master, fast devsel, latency 0

ff:02.1 Host bridge: Intel Corporation Core Processor QPI Physical 0 (rev 02)
	Subsystem: Lenovo Device 2196
	Flags: bus master, fast devsel, latency 0

ff:02.2 Host bridge: Intel Corporation Core Processor Reserved (rev 02)
	Subsystem: Lenovo Device 2196
	Flags: bus master, fast devsel, latency 0

ff:02.3 Host bridge: Intel Corporation Core Processor Reserved (rev 02)
	Subsystem: Lenovo Device 2196
	Flags: bus master, fast devsel, latency 0


[-- Attachment #3: dmesg.gz --]
[-- Type: application/octet-stream, Size: 16815 bytes --]

[-- Attachment #4: config.gz --]
[-- Type: application/octet-stream, Size: 18985 bytes --]

Re: [2.6.36-rc2 regression] X201s: resume-from-S2R hangs

[Eric Anholt]

[-- Attachment #1: Type: text/plain, Size: 1183 bytes --]

On Mon, 6 Sep 2010 02:05:36 -0700, Andy Isaacson <adi@hexapodia.org> wrote:
> I have a Thinkpad X201s on which suspend-to-RAM was quite reliable until
> after 2.6.36-rc1; with problematic kernels, suspend seems to work OK
> (the machine goes to sleep with the moon LED steadily lit) but on resume
> the LCD never lights up (and it's not just the backlight, the screen
> isn't drawing at all) and the keyboard and network are dead.  Nothing
> makes it to syslog, either.  The moon LED is turned off though.
> 
> There seems to be some timing component to the problem -- some problem
> kernels will suspend reliably 25% or 50% of the time and hang the rest.
> There also seems to be some per-reboot state involved; one problem
> kernel suspended successfully and survived 25 suspends in a row without
> triggering a failure.  (But after a reboot, the first suspend hung...)
> Other problem kernels fail on 4 out of 4 reboots.
> 
> Bisecting is pretty hard under these conditions, but it seems to point
> to

Weird.  That's exactly the hardware that I tested RC6 on, and I haven't
seen a single resume failure on mine.  Sounds like time to revert for
this release.

[-- Attachment #2: Type: application/pgp-signature, Size: 197 bytes --]

Re: [2.6.36-rc2 regression] X201s: resume-from-S2R hangs

[Chris Wilson]

On Tue, 07 Sep 2010 10:12:34 -0700, Eric Anholt <eric@anholt.net> wrote:
> On Mon, 6 Sep 2010 02:05:36 -0700, Andy Isaacson <adi@hexapodia.org> wrote:
> > I have a Thinkpad X201s on which suspend-to-RAM was quite reliable until
> > after 2.6.36-rc1; with problematic kernels, suspend seems to work OK
> > (the machine goes to sleep with the moon LED steadily lit) but on resume
> > the LCD never lights up (and it's not just the backlight, the screen
> > isn't drawing at all) and the keyboard and network are dead.  Nothing
> > makes it to syslog, either.  The moon LED is turned off though.
> > 
> > There seems to be some timing component to the problem -- some problem
> > kernels will suspend reliably 25% or 50% of the time and hang the rest.
> > There also seems to be some per-reboot state involved; one problem
> > kernel suspended successfully and survived 25 suspends in a row without
> > triggering a failure.  (But after a reboot, the first suspend hung...)
> > Other problem kernels fail on 4 out of 4 reboots.

We introduced a timing regression which should be fixed by 
http://cgit.freedesktop.org/~ickle/drm-intel/ drm-intel-fixes
[http://cgit.freedesktop.org/~ickle/drm-intel/commit/?id=300387c0b57d75e5218e2881d6ad2720657a8bcf]

> > Bisecting is pretty hard under these conditions, but it seems to point
> > to
> 
> Weird.  That's exactly the hardware that I tested RC6 on, and I haven't
> seen a single resume failure on mine.  Sounds like time to revert for
> this release.

At the moment we have a second resume regression that has been difficult to
reproduce, https://bugs.freedesktop.org/show_bug.cgi?id=29406 :

commit d1b851fc0d105caa6b6e3e7c92d2987dfb52cbe0
Author: Zou Nan hai <nanhai.zou@intel.com>
Date:   Fri May 21 09:08:57 2010 +0800

    drm/i915: implement BSD ring buffer V2

The workaround for this is to #define HAS_BSD(dev) 0 [i915_drv.h].

Andy, can you try the workaround to see if this is the same issue in a
different guise?

Alternatively, we can be optimistic that it is just a timing issue fixed
in drm-intel-next. :)

Otherwise we will revert that patch. Thanks.
-Chris

-- 
Chris Wilson, Intel Open Source Technology Centre

RE: [2.6.36-rc2 regression] X201s: resume-from-S2R hangs

[Zou, Nanhai]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset="gb2312", Size: 2758 bytes --]

We've had tried every kind of G45 and G45M machines here for many times, 
But can not reproduce #29406, 
We will see if we can reproduce this issue on X201

Thanks
Zou Nan hai

-----Original Message-----
From: Chris Wilson [mailto:chris@chris-wilson.co.uk] 
Sent: 2010Äê9ÔÂ8ÈÕ 1:38
To: Eric Anholt; Andy Isaacson; linux-kernel@vger.kernel.org
Cc: Zou, Nanhai
Subject: Re: [2.6.36-rc2 regression] X201s: resume-from-S2R hangs

On Tue, 07 Sep 2010 10:12:34 -0700, Eric Anholt <eric@anholt.net> wrote:
> On Mon, 6 Sep 2010 02:05:36 -0700, Andy Isaacson <adi@hexapodia.org> wrote:
> > I have a Thinkpad X201s on which suspend-to-RAM was quite reliable until
> > after 2.6.36-rc1; with problematic kernels, suspend seems to work OK
> > (the machine goes to sleep with the moon LED steadily lit) but on resume
> > the LCD never lights up (and it's not just the backlight, the screen
> > isn't drawing at all) and the keyboard and network are dead.  Nothing
> > makes it to syslog, either.  The moon LED is turned off though.
> > 
> > There seems to be some timing component to the problem -- some problem
> > kernels will suspend reliably 25% or 50% of the time and hang the rest.
> > There also seems to be some per-reboot state involved; one problem
> > kernel suspended successfully and survived 25 suspends in a row without
> > triggering a failure.  (But after a reboot, the first suspend hung...)
> > Other problem kernels fail on 4 out of 4 reboots.

We introduced a timing regression which should be fixed by 
http://cgit.freedesktop.org/~ickle/drm-intel/ drm-intel-fixes
[http://cgit.freedesktop.org/~ickle/drm-intel/commit/?id=300387c0b57d75e5218e2881d6ad2720657a8bcf]

> > Bisecting is pretty hard under these conditions, but it seems to point
> > to
> 
> Weird.  That's exactly the hardware that I tested RC6 on, and I haven't
> seen a single resume failure on mine.  Sounds like time to revert for
> this release.

At the moment we have a second resume regression that has been difficult to
reproduce, https://bugs.freedesktop.org/show_bug.cgi?id=29406 :

commit d1b851fc0d105caa6b6e3e7c92d2987dfb52cbe0
Author: Zou Nan hai <nanhai.zou@intel.com>
Date:   Fri May 21 09:08:57 2010 +0800

    drm/i915: implement BSD ring buffer V2

The workaround for this is to #define HAS_BSD(dev) 0 [i915_drv.h].

Andy, can you try the workaround to see if this is the same issue in a
different guise?

Alternatively, we can be optimistic that it is just a timing issue fixed
in drm-intel-next. :)

Otherwise we will revert that patch. Thanks.
-Chris

-- 
Chris Wilson, Intel Open Source Technology Centre
ÿôèº{.nÇ+‰·Ÿ®‰­†+%ŠËÿ±éݶ\x17¥Šwÿº{.nÇ+‰·¥Š{±þG«éÿŠ{ayº\x1dʇڙë,j\a­¢f£¢·hšïêÿ‘êçz_è®\x03(­éšŽŠÝ¢j"ú\x1a¶^[m§ÿÿ¾\a«þG«éÿ¢¸?™¨è­Ú&£ø§~á¶iO•æ¬z·švØ^\x14\x04\x1a¶^[m§ÿÿÃ\fÿ¶ìÿ¢¸?–I¥

Re: [2.6.36-rc2 regression] X201s: resume-from-S2R hangs

[Andy Isaacson]

On Tue, Sep 07, 2010 at 06:37:55PM +0100, Chris Wilson wrote:
> On Tue, 07 Sep 2010 10:12:34 -0700, Eric Anholt <eric@anholt.net> wrote:
> > On Mon, 6 Sep 2010 02:05:36 -0700, Andy Isaacson <adi@hexapodia.org> wrote:
> > > I have a Thinkpad X201s on which suspend-to-RAM was quite reliable until
> > > after 2.6.36-rc1; with problematic kernels, suspend seems to work OK
> > > (the machine goes to sleep with the moon LED steadily lit) but on resume
> > > the LCD never lights up (and it's not just the backlight, the screen
> > > isn't drawing at all) and the keyboard and network are dead.  Nothing
> > > makes it to syslog, either.  The moon LED is turned off though.
> > > 
> > > There seems to be some timing component to the problem -- some problem
> > > kernels will suspend reliably 25% or 50% of the time and hang the rest.
> > > There also seems to be some per-reboot state involved; one problem
> > > kernel suspended successfully and survived 25 suspends in a row without
> > > triggering a failure.  (But after a reboot, the first suspend hung...)
> > > Other problem kernels fail on 4 out of 4 reboots.
>
> At the moment we have a second resume regression that has been difficult to
> reproduce, https://bugs.freedesktop.org/show_bug.cgi?id=29406 :
> 
> commit d1b851fc0d105caa6b6e3e7c92d2987dfb52cbe0
> Author: Zou Nan hai <nanhai.zou@intel.com>
> Date:   Fri May 21 09:08:57 2010 +0800
> 
>     drm/i915: implement BSD ring buffer V2
> 
> The workaround for this is to #define HAS_BSD(dev) 0 [i915_drv.h].

Setting HAS_BSD 0 does not fix the issue for me.

@@ -1203,7 +1203,7
...
-#define HAS_BSD(dev)            (IS_IRONLAKE(dev) || IS_G4X(dev))
+#define HAS_BSD(dev)            0

As previously reported, reverting ce171780 does seem to do the trick
(I'm a bit more confident having survived another day of use).

> We introduced a timing regression which should be fixed by 
> http://cgit.freedesktop.org/~ickle/drm-intel/ drm-intel-fixes
> [http://cgit.freedesktop.org/~ickle/drm-intel/commit/?id=300387c0b57d75e5218e2881d6ad2720657a8bcf]

Currently building d56557 which includes 300387, will let you know if
that fixes it.

-andy

Re: [2.6.36-rc2 regression] X201s: resume-from-S2R hangs

[Andy Isaacson]

On Tue, Sep 07, 2010 at 06:37:55PM +0100, Chris Wilson wrote:
> On Tue, 07 Sep 2010 10:12:34 -0700, Eric Anholt <eric@anholt.net> wrote:
> > On Mon, 6 Sep 2010 02:05:36 -0700, Andy Isaacson <adi@hexapodia.org> wrote:
> > > I have a Thinkpad X201s on which suspend-to-RAM was quite reliable until
> > > after 2.6.36-rc1; with problematic kernels, suspend seems to work OK
> > > (the machine goes to sleep with the moon LED steadily lit) but on resume
> > > the LCD never lights up (and it's not just the backlight, the screen
> > > isn't drawing at all) and the keyboard and network are dead.  Nothing
> > > makes it to syslog, either.  The moon LED is turned off though.
> > > 
> > > There seems to be some timing component to the problem -- some problem
> > > kernels will suspend reliably 25% or 50% of the time and hang the rest.
> > > There also seems to be some per-reboot state involved; one problem
> > > kernel suspended successfully and survived 25 suspends in a row without
> > > triggering a failure.  (But after a reboot, the first suspend hung...)
> > > Other problem kernels fail on 4 out of 4 reboots.
> 
> We introduced a timing regression which should be fixed by 
> http://cgit.freedesktop.org/~ickle/drm-intel/ drm-intel-fixes
> [http://cgit.freedesktop.org/~ickle/drm-intel/commit/?id=300387c0b57d75e5218e2881d6ad2720657a8bcf]
> 
> > > Bisecting is pretty hard under these conditions, but it seems to point
> > > to
> > 
> > Weird.  That's exactly the hardware that I tested RC6 on, and I haven't
> > seen a single resume failure on mine.

I have BIOS 1.05 (6QET35WW) and the 1440x900 panel, perhaps something is
slightly different?  I suspend by running pm-suspend from an xterm.

> >  Sounds like time to revert for
> > this release.
> 
> At the moment we have a second resume regression that has been difficult to
> reproduce, https://bugs.freedesktop.org/show_bug.cgi?id=29406 :
> 
> commit d1b851fc0d105caa6b6e3e7c92d2987dfb52cbe0
> Author: Zou Nan hai <nanhai.zou@intel.com>
> Date:   Fri May 21 09:08:57 2010 +0800
> 
>     drm/i915: implement BSD ring buffer V2
> 
> The workaround for this is to #define HAS_BSD(dev) 0 [i915_drv.h].
> 
> Andy, can you try the workaround to see if this is the same issue in a
> different guise?
> 
> Alternatively, we can be optimistic that it is just a timing issue fixed
> in drm-intel-next. :)
> 
> Otherwise we will revert that patch. Thanks.

Tried d56557a (which includes 300387c), same hang at resume.

Tried #define HAS_BSD(dev) 0, same hang at resume.

Reverted ce171780 on top of d56557a, resume works great.

-andy