[BUG] Problem with commit cf03c5dac83609f09d9f0e9fa3c09d86daed614d

[BUG] Problem with commit cf03c5dac83609f09d9f0e9fa3c09d86daed614d

[Dirk Gouders]

Hi Seth,

today, I noticed a problem with the current kernel
(3.5.0-04773-gbdc0077): my laptop hangs during boot -- the last message
on the console is printed just before invoking `udevadm trigger'.

I bisected the problem and ended at commit
cf03c5dac83609f09d9f0e9fa3c09d86daed614d (brcm80211: smac: inform
mac80211 of the X2 regulatory domain).  However, I'm not sure that this
commit really is the reason for my problems, because the behaviour is a
bit different: at that state the machine not just hangs but the console
shows a trace which I append to the end of this mail -- but the trace
happens at the same state during the boot process.

Best regards,

Dirk

------------------------------------------------------------------------

Freeing unused kernel memory: 516k freed
BUG: unable to handle kernel NULL pointer dereference at 000000000000096a
IP: [<ffffffffa0023410>] wlc_phy_chanspec_band_validch+0x26/0x6b [brcmsmac]
PGD 0 
Oops: 0000 [#1] PREEMPT SMP 
CPU 1 
Modules linked in: brcmsmac cordic brcmutil bcma

Pid: 982, comm: kworker/1:2 Not tainted 3.5.0-rc1-00538-gcf03c5d #38 LENOVO 08946MG/MoutCook
RIP: 0010:[<ffffffffa0023410>]  [<ffffffffa0023410>] wlc_phy_chanspec_band_validch+0x26/0x6b [brcmsmac]
RSP: 0018:ffff8800700c7cb8  EFLAGS: 00010202
RAX: 0000000000000094 RBX: ffff88006f2c9000 RCX: 0000000000000095
RDX: ffff8800700c7cc4 RSI: 0000000000000000 RDI: ffff8800700c7ce0
RBP: 0000000000000001 R08: 0000000000000000 R09: 0000000000000001
R10: 0000000000000040 R11: ffff88007048e240 R12: ffff88006f1021c0
R13: ffffffffa0046c30 R14: ffff88006f2c9000 R15: ffff880073114905
FS:  0000000000000000(0000) GS:ffff880073100000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 000000000000096a CR3: 0000000001923000 CR4: 00000000000007e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process kworker/1:2 (pid: 982, threadinfo ffff8800700c6000, task ffff880072faca40)
Stack:
 ffffffffa0016aa3 000000006f103d20 0000000000000000 0000000000000000
 0000000000000000 ffff88007048e2c3 ffff88006f103d20 ffff88006f102520
 ffff88006f103d20 0000000000000000 ffffffffa004666a 0000000000000011
Call Trace:
 [<ffffffffa0016aa3>] ? brcms_c_regd_init+0x3c/0xc5 [brcmsmac]
 [<ffffffffa004666a>] ? brcms_bcma_probe+0x548/0x5b0 [brcmsmac]
 [<ffffffff81315a1e>] ? driver_probe_device+0xa8/0x1ad
 [<ffffffff81315b72>] ? __driver_attach+0x4f/0x6f
 [<ffffffff81315b23>] ? driver_probe_device+0x1ad/0x1ad
 [<ffffffff81314474>] ? bus_for_each_dev+0x4a/0x75
 [<ffffffff81315216>] ? bus_add_driver+0xb7/0x207
 [<ffffffffa001399b>] ? brcms_isr+0x5d/0x5d [brcmsmac]
 [<ffffffff81315f87>] ? driver_register+0x85/0xed
 [<ffffffffa001399b>] ? brcms_isr+0x5d/0x5d [brcmsmac]
 [<ffffffffa00139af>] ? brcms_driver_init+0x14/0x32 [brcmsmac]
 [<ffffffff8106ade3>] ? process_one_work+0x1b0/0x2e7
 [<ffffffff8106b463>] ? worker_thread+0x158/0x215
 [<ffffffff8106b30b>] ? manage_workers.isra.26+0x15b/0x15b
 [<ffffffff8106eab6>] ? kthread+0x7d/0x85
 [<ffffffff8162ed14>] ? kernel_thread_helper+0x4/0x10
 [<ffffffff8106ea39>] ? kthread_freezable_should_stop+0x48/0x48
 [<ffffffff8162ed10>] ? gs_change+0xb/0xb
Code: ff 5b 5b 5d c3 31 c0 49 89 f8 b9 07 00 00 00 48 89 d7 41 b9 01 00 00 00 f3 ab 31 c0 0f b7 88 50 7d 04 a0 81 f9 94 00 00 00 76 12 <41> 80 b8 6a 09 00 00 00 74 08 81 f9 a5 00 00 00 76 26 83 f9 0e 
RIP  [<ffffffffa0023410>] wlc_phy_chanspec_band_validch+0x26/0x6b [brcmsmac]
 RSP <ffff8800700c7cb8>
CR2: 000000000000096a
---[ end trace 112507b89cd00789 ]---
BUG: unable to handle kernel paging request at fffffffffffffff8
IP: [<ffffffff8106ec81>] kthread_data+0x7/0xc
PGD 1925067 PUD 1926067 PMD 0 
Oops: 0000 [#2] PREEMPT SMP 
CPU 1 
Modules linked in: brcmsmac cordic brcmutil bcma

Pid: 982, comm: kworker/1:2 Tainted: G      D      3.5.0-rc1-00538-gcf03c5d #38 LENOVO 08946MG/MoutCook
RIP: 0010:[<ffffffff8106ec81>]  [<ffffffff8106ec81>] kthread_data+0x7/0xc
RSP: 0018:ffff8800700c78f0  EFLAGS: 00010002
RAX: 0000000000000000 RBX: ffff880073111040 RCX: 0000000000000001
RDX: ffff880072c04080 RSI: 0000000000000001 RDI: ffff880072faca40
RBP: 0000000000000001 R08: 0000000000000400 R09: dead000000200200
R10: dead000000200200 R11: ffff88006f576700 R12: ffff880072facd10
R13: 0000000000000246 R14: 0000000000000001 R15: 0000000000000001
FS:  0000000000000000(0000) GS:ffff880073100000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: fffffffffffffff8 CR3: 0000000001923000 CR4: 00000000000007e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process kworker/1:2 (pid: 982, threadinfo ffff8800700c6000, task ffff880072faca40)
Stack:
 ffffffff8106bb52 ffff880073111040 ffff880072faca40 ffff880072facd10
 ffffffff8162c007 ffff8800700c7fd8 ffff8800700c7fd8 0000000000011040
 0000000000004000 00000000000003d6 ffffffff810ff309 ffff8800700c798b
Call Trace:
 [<ffffffff8106bb52>] ? wq_worker_sleeping+0xb/0x6a
 [<ffffffff8162c007>] ? __schedule+0x137/0x71a
 [<ffffffff810ff309>] ? d_lookup+0x24/0x36
 [<ffffffff81131a5e>] ? proc_flush_task+0xe1/0x1a7
 [<ffffffff8105afef>] ? release_task+0x2f2/0x30a
 [<ffffffff8105c42b>] ? do_exit+0x769/0x775
 [<ffffffff8102becb>] ? oops_end+0x85/0x8a
 [<ffffffff81620dd9>] ? no_context+0x256/0x265
 [<ffffffff8122f82e>] ? ida_get_new_above+0x176/0x198
 [<ffffffff81045529>] ? do_page_fault+0x160/0x34b
 [<ffffffff81132115>] ? proc_register+0x193/0x1a9
 [<ffffffff811324ba>] ? proc_mkdir_mode+0x39/0x50
 [<ffffffff810a6d73>] ? register_handler_proc+0xcd/0xde
 [<ffffffff8162d71f>] ? page_fault+0x1f/0x30
 [<ffffffffa0023410>] ? wlc_phy_chanspec_band_validch+0x26/0x6b [brcmsmac]
 [<ffffffffa0016aa3>] ? brcms_c_regd_init+0x3c/0xc5 [brcmsmac]
 [<ffffffffa004666a>] ? brcms_bcma_probe+0x548/0x5b0 [brcmsmac]
 [<ffffffff81315a1e>] ? driver_probe_device+0xa8/0x1ad
 [<ffffffff81315b72>] ? __driver_attach+0x4f/0x6f
 [<ffffffff81315b23>] ? driver_probe_device+0x1ad/0x1ad
 [<ffffffff81314474>] ? bus_for_each_dev+0x4a/0x75
 [<ffffffff81315216>] ? bus_add_driver+0xb7/0x207
 [<ffffffffa001399b>] ? brcms_isr+0x5d/0x5d [brcmsmac]
 [<ffffffff81315f87>] ? driver_register+0x85/0xed
 [<ffffffffa001399b>] ? brcms_isr+0x5d/0x5d [brcmsmac]
 [<ffffffffa00139af>] ? brcms_driver_init+0x14/0x32 [brcmsmac]
 [<ffffffff8106ade3>] ? process_one_work+0x1b0/0x2e7
 [<ffffffff8106b463>] ? worker_thread+0x158/0x215
 [<ffffffff8106b30b>] ? manage_workers.isra.26+0x15b/0x15b
 [<ffffffff8106eab6>] ? kthread+0x7d/0x85
 [<ffffffff8162ed14>] ? kernel_thread_helper+0x4/0x10
 [<ffffffff8106ea39>] ? kthread_freezable_should_stop+0x48/0x48
 [<ffffffff8162ed10>] ? gs_change+0xb/0xb
Code: df 83 e0 3f 48 c1 e5 03 48 8d 34 c5 08 94 64 81 48 29 ee e8 7d 8a 00 00 81 4b 14 00 00 00 04 41 5b 5b 5d c3 48 8b 87 78 02 00 00 <48> 8b 40 f8 c3 48 3b 3d 4b 11 a5 00 75 08 0f bf 87 da 05 00 00 
RIP  [<ffffffff8106ec81>] kthread_data+0x7/0xc
 RSP <ffff8800700c78f0>
CR2: fffffffffffffff8
---[ end trace 112507b89cd0078a ]---
Fixing recursive fault but reboot is needed!

Re: [BUG] Problem with commit cf03c5dac83609f09d9f0e9fa3c09d86daed614d

[Seth Forshee]

On Thu, Jul 26, 2012 at 03:28:44PM +0200, Dirk Gouders wrote:
> Hi Seth,
> 
> today, I noticed a problem with the current kernel
> (3.5.0-04773-gbdc0077): my laptop hangs during boot -- the last message
> on the console is printed just before invoking `udevadm trigger'.
> 
> I bisected the problem and ended at commit
> cf03c5dac83609f09d9f0e9fa3c09d86daed614d (brcm80211: smac: inform
> mac80211 of the X2 regulatory domain).  However, I'm not sure that this
> commit really is the reason for my problems, because the behaviour is a
> bit different: at that state the machine not just hangs but the console
> shows a trace which I append to the end of this mail -- but the trace
> happens at the same state during the boot process.

Hi Dirk,

The NULL pointer dereference in wlc_phy_chanspec_band_validch is already
fixed in edc7651f3a4ffa7fca37d92401562694121512ad. There was also
another related crash that appeared after the introduction of 802.11ad
support that's fixed in 32c336a5dfd16b4645c8a0dbc534efd02f079e02.

So in all likelihood whatever problem your seeing in current -git is a
different issue. One easy way to test this would be to prevent the
brcmsmac module from loading and verify that you still get the hang,
then rerun the bisection that way to keep those bugs from affecting your
results.

Cheers,
Seth

Re: [BUG] Problem with commit cf03c5dac83609f09d9f0e9fa3c09d86daed614d

[Dirk Gouders]

Seth Forshee <seth.forshee@canonical.com> writes:

> On Thu, Jul 26, 2012 at 03:28:44PM +0200, Dirk Gouders wrote:
>> Hi Seth,
>> 
>> today, I noticed a problem with the current kernel
>> (3.5.0-04773-gbdc0077): my laptop hangs during boot -- the last message
>> on the console is printed just before invoking `udevadm trigger'.
>> 
>> I bisected the problem and ended at commit
>> cf03c5dac83609f09d9f0e9fa3c09d86daed614d (brcm80211: smac: inform
>> mac80211 of the X2 regulatory domain).  However, I'm not sure that this
>> commit really is the reason for my problems, because the behaviour is a
>> bit different: at that state the machine not just hangs but the console
>> shows a trace which I append to the end of this mail -- but the trace
>> happens at the same state during the boot process.
>
> Hi Dirk,
>
> The NULL pointer dereference in wlc_phy_chanspec_band_validch is already
> fixed in edc7651f3a4ffa7fca37d92401562694121512ad. There was also
> another related crash that appeared after the introduction of 802.11ad
> support that's fixed in 32c336a5dfd16b4645c8a0dbc534efd02f079e02.
>
> So in all likelihood whatever problem your seeing in current -git is a
> different issue. One easy way to test this would be to prevent the
> brcmsmac module from loading and verify that you still get the hang,
> then rerun the bisection that way to keep those bugs from affecting your
> results.

Hi Seth,

thanks for your reply and sorry for the noise.

I followed your advice and tried to boot with the WLAN interface turned
off, and the problem still exists.  I'll start a new bisect session,
probably with one of the commits you mentioned as the first good commit.

Best regards,

Dirk

Re: [BUG] Problem with commit cf03c5dac83609f09d9f0e9fa3c09d86daed614d

[Seth Forshee]

On Thu, Jul 26, 2012 at 05:07:57PM +0200, Dirk Gouders wrote:
> Hi Seth,
> 
> thanks for your reply and sorry for the noise.
> 
> I followed your advice and tried to boot with the WLAN interface turned
> off, and the problem still exists.  I'll start a new bisect session,
> probably with one of the commits you mentioned as the first good commit.

Just to make sure three's not any confusion ...

What I was suggesting was not just disabling the network interface but
completely preventing brcmsmac from being loaded. The oops you saw
happens in the context of the driver's probe function and would happen
regardless of whether or not the interface is enabled.

Seth

Re: [BUG] Problem with commit cf03c5dac83609f09d9f0e9fa3c09d86daed614d

[Dirk Gouders]

Seth Forshee <seth.forshee@canonical.com> writes:

> On Thu, Jul 26, 2012 at 05:07:57PM +0200, Dirk Gouders wrote:
>> Hi Seth,
>> 
>> thanks for your reply and sorry for the noise.
>> 
>> I followed your advice and tried to boot with the WLAN interface turned
>> off, and the problem still exists.  I'll start a new bisect session,
>> probably with one of the commits you mentioned as the first good commit.
>
> Just to make sure three's not any confusion ...
>
> What I was suggesting was not just disabling the network interface but
> completely preventing brcmsmac from being loaded. The oops you saw
> happens in the context of the driver's probe function and would happen
> regardless of whether or not the interface is enabled.

Sorry for the confusion.

I already started a new bisect session, with a bad commit I disabled
brcmsmac in the kernel config and the problem still exists.  Now, I
continue with brcmsmac disabled and will see where the bisect ends.

Dirk

Re: [BUG] Problem with commit cf03c5dac83609f09d9f0e9fa3c09d86daed614d

[Dirk Gouders]

Dirk Gouders <gouders@et.bocholt.fh-gelsenkirchen.de> writes:

> Seth Forshee <seth.forshee@canonical.com> writes:
>
>> On Thu, Jul 26, 2012 at 05:07:57PM +0200, Dirk Gouders wrote:
>>> Hi Seth,
>>> 
>>> thanks for your reply and sorry for the noise.
>>> 
>>> I followed your advice and tried to boot with the WLAN interface turned
>>> off, and the problem still exists.  I'll start a new bisect session,
>>> probably with one of the commits you mentioned as the first good commit.
>>
>> Just to make sure three's not any confusion ...
>>
>> What I was suggesting was not just disabling the network interface but
>> completely preventing brcmsmac from being loaded. The oops you saw
>> happens in the context of the driver's probe function and would happen
>> regardless of whether or not the interface is enabled.
>
> Sorry for the confusion.
>
> I already started a new bisect session, with a bad commit I disabled
> brcmsmac in the kernel config and the problem still exists.  Now, I
> continue with brcmsmac disabled and will see where the bisect ends.

Hi Seth,

I just want to let you know that the problem with my laptop has been
fixed.

Yesterday, I found commit b9562545ef0b to cause the problem
and after a pull this morning I noticed that the problem introduced by
it has been fixed by commit 1f03bf06e4e3 (bcma: fix invalid PMU chip
control masks).

Thanks again for your help.

Dirk