Re: exit_mmap BUG_ON in 2.6.23

Re: exit_mmap BUG_ON in 2.6.23

[Hugh Dickins]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 2476 bytes --]

On Fri, 18 May 2012, Sam Portolla wrote:
> [please cc samPortolla@yahoo.com on your replies, not subscribed to the linux-kernel mailer]
> 
> Hi, I have read the thread on same issue in 3.1:
> but this is happening on earlier GNU linux version 2.6.23 for x86_64,
> which does not have THP (I believe), nor it has huge_memory.c.
> Is there a fix one of you experts could supply?  Issue is not reproducible
> so far, but happened on a customer site. Some info below.
> 
> kernel BUG at .../bfc/linux/kernel-2.6.x/mm/mmap.c:2049!
> 
> Line 2049 is in exit_mmap():
> 
> BUG_ON(mm->nr_ptes > (FIRST_USER_ADDRESS+PMD_SIZE-1)>>PMD_SHIFT);
> 
>  RIP: 0010:[<ffffffff80277840>]  [<ffffffff80277840>] exit_mmap+0xf0/0x100 
> [snip]
>  Call Trace:
>  [<ffffffff8022ee14>] mmput+0x44/0xd0
>  [<ffffffff802340a1>] exit_mm+0x91/0x100
>  [<ffffffff802347ea>] do_exit+0x17a/0x960
>  [<ffffffff8023c4bc>] __dequeue_signal+0xec/0x1b0
>  [<ffffffff80235048>] do_group_exit+0x38/0x90
>  [<ffffffff8023e3c6>] get_signal_to_deliver+0x2d6/0x4b0
>  [<ffffffff8020b69a>] do_notify_resume+0xaa/0x760
>  [<ffffffff8020c818>] retint_signal+0x3d/0x85

I've checked back through old ChangeLogs, and (apart from a UserModeLinux
case) I don't see any fix for a BUG_ON(nr_ptes) issue in between 2.6.19
and the much later THP issue, which you're right to think cannot be yours.

But the 2.6.19 case, and one which a video driver writer had more recently,
were both caused by unrelated code zeroing beyond what it had allocated:
happening to zero part of a higher-level page table, making it impossible
for task exit to locate all the page tables (and pages) it had to free.

Though I can't be sure, these BUG_ON(nr_ptes) reports do seem perhaps
too infrequent to be caused by bad logic in mm itself: I suspect memory
corruption in your case too.

There's no clue here as to what the cause might be, I'm afraid.
Rebuilding your kernel with CONFIG_DEBUG_PAGEALLOC=y, and slab debugging
on, might shed more light: but that's probably not something you want to
get into on a customer site, for a problem only seen once or twice.

The best I can suggest is for you to change that BUG_ON to a WARN_ON,
so at least the kernel doesn't crash there, and you might gather more
information from each time it happens; but you'll probably leak pages,
and may very well crash soon for other reasons (e.g. when evicting an
inode cannot locate all the maps of its pages).

Hugh

Re: exit_mmap BUG_ON in 2.6.23 (and Add qdisc __NET_XMIT_STOLEN)

[Sam Portolla]

[pease cc samPortolla@yahoo.com on the replies; not a member of this mailer]

Hi Hugh,

Thank you!  It turns out our 2.6.23 kernel does not have this old patch, I am also adding Jarek, David and Patrick who were involved in the below fix for their insights:


commit 378a2f090f7a478704a372a4869b8a9ac206234e
Date:   Mon Aug 4 22:31:03 2008 -0700
net_sched: Add qdisc __NET_XMIT_STOLEN flag
In this failure case below, as well as some others, the ethernet driver printed a transmit timeout just before the crash.

It seems since we don't have the above patch, the kernel qdisc Tx packet path for fragmented packets can be messed up and corrupt the skb  it passes to drivers, which in the historic case that led to above fix, caused an skb NULL ptr de-ref in the driver itself (which we also saw once). 

Jarek, David or Patrick,

Could the lack of above patch cause the kernel to also falsely detect transmit timeouts on various drivers as it can not properly keep track of packets transmitted? Can you please elaborate so  a newbie like me can understand?

Is the above commit the sole one required for the kernel panic/skb NULL de-ref driver issue or is there more needed fixes later on that can be backported to an older kernel (2.6.23 GNU/Linux x86_64)?


Hugh, 

 I wonder if the lack of above patch in our code base could explain the exit_mmap() BUG_ON as well due to memory corruption causing MMU to not be able to locate the page(s) it had to free. NR_PTES keeps track of that? Could you explain that more?



Thank you ALL

----- Original Message -----
From: Hugh Dickins <hughd@google.com>
To: Sam Portolla <samportolla@yahoo.com>
Cc: "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>; "aarcange@redhat.com" <aarcange@redhat.com>
Sent: Saturday, May 19, 2012 1:45 PM
Subject: Re: exit_mmap BUG_ON in 2.6.23

On Fri, 18 May 2012, Sam Portolla wrote:
> [please cc samPortolla@yahoo.com on your replies, not subscribed to the linux-kernel mailer]
> 
> Hi, I have read the thread on same issue in 3.1:
> but this is happening on earlier GNU linux version 2.6.23 for x86_64,
> which does not have THP (I believe), nor it has huge_memory.c.
> Is there a fix one of you experts could supply?  Issue is not reproducible
> so far, but happened on a customer site. Some info below.
> 
> kernel BUG at .../bfc/linux/kernel-2.6.x/mm/mmap.c:2049!
> 
> Line 2049 is in exit_mmap():
> 
> BUG_ON(mm->nr_ptes > (FIRST_USER_ADDRESS+PMD_SIZE-1)>>PMD_SHIFT);
> 
>  RIP: 0010:[<ffffffff80277840>]  [<ffffffff80277840>] exit_mmap+0xf0/0x100 
> [snip]
>  Call Trace:
>  [<ffffffff8022ee14>] mmput+0x44/0xd0
>  [<ffffffff802340a1>] exit_mm+0x91/0x100
>  [<ffffffff802347ea>] do_exit+0x17a/0x960
>  [<ffffffff8023c4bc>] __dequeue_signal+0xec/0x1b0
>  [<ffffffff80235048>] do_group_exit+0x38/0x90
>  [<ffffffff8023e3c6>] get_signal_to_deliver+0x2d6/0x4b0
>  [<ffffffff8020b69a>] do_notify_resume+0xaa/0x760
>  [<ffffffff8020c818>] retint_signal+0x3d/0x85

I've checked back through old ChangeLogs, and (apart from a UserModeLinux
case) I don't see any fix for a BUG_ON(nr_ptes) issue in between 2.6.19
and the much later THP issue, which you're right to think cannot be yours.

But the 2.6.19 case, and one which a video driver writer had more recently,
were both caused by unrelated code zeroing beyond what it had allocated:
happening to zero part of a higher-level page table, making it impossible
for task exit to locate all the page tables (and pages) it had to free.

Though I can't be sure, these BUG_ON(nr_ptes) reports do seem perhaps
too infrequent to be caused by bad logic in mm itself: I suspect memory
corruption in your case too.

There's no clue here as to what the cause might be, I'm afraid.
Rebuilding your kernel with CONFIG_DEBUG_PAGEALLOC=y, and slab debugging
on, might shed more light: but that's probably not something you want to
get into on a customer site, for a problem only seen once or twice.

The best I can suggest is for you to change that BUG_ON to a WARN_ON,
so at least the kernel doesn't crash there, and you might gather more
information from each time it happens; but you'll probably leak pages,
and may very well crash soon for other reasons (e.g. when evicting an
inode cannot locate all the maps of its pages).

Hugh 

Re: exit_mmap BUG_ON in 2.6.23 (and Add qdisc __NET_XMIT_STOLEN)

[Eric Dumazet]

On Fri, 2012-05-25 at 17:28 -0700, Sam Portolla wrote:

Please don't top post on this list

> 
> [pease cc samPortolla@yahoo.com on the replies; not a member of this
> mailer]
> 
> Hi Hugh,
> 
> Thank you!  It turns out our 2.6.23 kernel does not have this old
> patch, I am also adding Jarek, David and Patrick who were involved in
> the below fix for their insights:
> 
> 
> commit 378a2f090f7a478704a372a4869b8a9ac206234e
> Date:   Mon Aug 4 22:31:03 2008 -0700
> net_sched: Add qdisc __NET_XMIT_STOLEN flag
> In this failure case below, as well as some others, the ethernet
> driver printed a transmit timeout just before the crash.
> 
> It seems since we don't have the above patch, the kernel qdisc Tx
> packet path for fragmented packets can be messed up and corrupt the
> skb  it passes to drivers, which in the historic case that led to
> above fix, caused an skb NULL ptr de-ref in the driver itself (which
> we also saw once). 
> 
> Jarek, David or Patrick,
> 
> Could the lack of above patch cause the kernel to also falsely detect
> transmit timeouts on various drivers as it can not properly keep track
> of packets transmitted? Can you please elaborate so  a newbie like me
> can understand?
> 
> Is the above commit the sole one required for the kernel panic/skb
> NULL de-ref driver issue or is there more needed fixes later on that
> can be backported to an older kernel (2.6.23 GNU/Linux x86_64)?
> 

Transmit timeouts are because of races in some network drivers.

The device stay in XOFF state for too long time (forever as a matter of
fact once the race triggered)

Since 2.6.23 we fixed a lot of them, but still races still exist.

Re: exit_mmap BUG_ON in 2.6.23 (and Add qdisc __NET_XMIT_STOLEN)

[Sam Portolla]

----- Original Message -----
From: Eric Dumazet <eric.dumazet@gmail.com>
To: Sam Portolla <samportolla@yahoo.com>
Cc: Hugh Dickins <hughd@google.com>; "kaber@trash.net" <kaber@trash.net>; "jarkao2@gmail.com" <jarkao2@gmail.com>; "davem@davemloft.net" <davem@davemloft.net>; "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>
Sent: Friday, May 25, 2012 9:25 PM
Subject: Re: exit_mmap BUG_ON in 2.6.23 (and Add qdisc __NET_XMIT_STOLEN)

On Fri, 2012-05-25 at 17:28 -0700, Sam Portolla wrote:

Please don't top post on this list

> 
> [pease cc samPortolla@yahoo.com on the replies; not a member of this
> mailer]
> 
> Hi Hugh,
> 
> Thank you!  It turns out our 2.6.23 kernel does not have this old
> patch, I am also adding Jarek, David and Patrick who were involved in
> the below fix for their insights:
> 
> 
> commit 378a2f090f7a478704a372a4869b8a9ac206234e
> Date:   Mon Aug 4 22:31:03 2008 -0700
> net_sched: Add qdisc __NET_XMIT_STOLEN flag
> In this failure case below, as well as some others, the ethernet
> driver printed a transmit timeout just before the crash.
> 
> It seems since we don't have the above patch, the kernel qdisc Tx
> packet path for fragmented packets can be messed up and corrupt the
> skb  it passes to drivers, which in the historic case that led to
> above fix, caused an skb NULL ptr de-ref in the driver itself (which
> we also saw once). 
> 
> Jarek, David or Patrick,
> 
> Could the lack of above patch cause the kernel to also falsely detect
> transmit timeouts on various drivers as it can not properly keep track
> of packets transmitted? Can you please elaborate so  a newbie like me
> can understand?
> 
> Is the above commit the sole one required for the kernel panic/skb
> NULL de-ref driver issue or is there more needed fixes later on that
> can be backported to an older kernel (2.6.23 GNU/Linux x86_64)?
> 

Transmit timeouts are because of races in some network drivers.

The device stay in XOFF state for too long time (forever as a matter of
fact once the race triggered)

Since 2.6.23 we fixed a lot of them, but still races still exist.

Yes, thanks I had looked at the kernel  code and know how transmit timeouts come to be in normal cases. The driver specifies a timeout period to the network layer, along with a callback function to call in case of Tx timeout so the driver can do error handling which is typically to reset the driver (and this happened in the case of the BNX2 linux driver our system uses as well). Above I had asked some specific questions with regards to whether a known bug w/ qdisc could stop the Tx Q's to the device and thereby cause traffic timeouts. Also it seems from the email thread on the patch I had mentioned above that the qdisc issue can cause memory corruption, which could then tie it in with the BUG_ON in exit_mmap() which Hugh had previously commented on. I am hoping the engineers who fixed the QDISC issue can comment on the former and Hugh can comment on the BUG_ON again. Regards. 

Re: exit_mmap BUG_ON in 2.6.23 (and Add qdisc __NET_XMIT_STOLEN)

[Eric Dumazet]

On Fri, 2012-05-25 at 22:27 -0700, Sam Portolla wrote:

> Yes, thanks I had looked at the kernel  code and know how transmit
> timeouts come to be in normal cases. The driver specifies a timeout
> period to the network layer, along with a callback function to call in
> case of Tx timeout so the driver can do error handling which is
> typically to reset the driver (and this happened in the case of the
> BNX2 linux driver our system uses as well). Above I had asked some
> specific questions with regards to whether a known bug w/ qdisc could
> stop the Tx Q's to the device and thereby cause traffic timeouts. Also
> it seems from the email thread on the patch I had mentioned above that
> the qdisc issue can cause memory corruption, which could then tie it
> in with the BUG_ON in exit_mmap() which Hugh had previously commented
> on. I am hoping the engineers who fixed the QDISC issue can comment on
> the former and Hugh can comment on the BUG_ON again. Regards. 


The commit you mention is about a very unusual use of qdiscs.
I really doubt it is your problem.
Most advanced tc users probably wont stick with 2.6.23 kernels.

Please post :

tc -s -d qdisc

And for all your network devices : 

for DEV in eth0 eth1 eth2
do
tc -s -d class show dev $DEV
done

Re: exit_mmap BUG_ON in 2.6.23 (and Add qdisc __NET_XMIT_STOLEN)

[Sam Portolla]

----- Original Message -----
From: Eric Dumazet <eric.dumazet@gmail.com>
To: Sam Portolla <samportolla@yahoo.com>
Cc: Hugh Dickins <hughd@google.com>; "kaber@trash.net" <kaber@trash.net>; "jarkao2@gmail.com" <jarkao2@gmail.com>; "davem@davemloft.net" <davem@davemloft.net>; "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>
Sent: Friday, May 25, 2012 11:16 PM
Subject: Re: exit_mmap BUG_ON in 2.6.23 (and Add qdisc __NET_XMIT_STOLEN)

On Fri, 2012-05-25 at 22:27 -0700, Sam Portolla wrote:

> Yes, thanks I had looked at the kernel  code and know how transmit
> timeouts come to be in normal cases. The driver specifies a timeout
> period to the network layer, along with a callback function to call in
> case of Tx timeout so the driver can do error handling which is
> typically to reset the driver (and this happened in the case of the
> BNX2 linux driver our system uses as well). Above I had asked some
> specific questions with regards to whether a known bug w/ qdisc could
> stop the Tx Q's to the device and thereby cause traffic timeouts. Also
> it seems from the email thread on the patch I had mentioned above that
> the qdisc issue can cause memory corruption, which could then tie it
> in with the BUG_ON in exit_mmap() which Hugh had previously commented
> on. I am hoping the engineers who fixed the QDISC issue can comment on
> the former and Hugh can comment on the BUG_ON again. Regards. 


The commit you mention is about a very unusual use of qdiscs.
I really doubt it is your problem.
Most advanced tc users probably wont stick with 2.6.23 kernels.

Please post :

tc -s -d qdisc

And for all your network devices : 

for DEV in eth0 eth1 eth2
do
tc -s -d class show dev $DEV
done

Hi Eric,

Can you please elaborate on what you mean by the commit is "about a very unusual use of qdiscs"?
The lack of this fix was at the time determined to cause the ether driver to do a NULL ptr de-ref on its  Tx ring SKB, which is what we saw in our case as well.  The qdisc code apparently was changing the "nr_frags" field in the SKB, while the driver was the owner of the SKB, causing the issue. 
 
Can't find the "tc" command mentioned above on our system. Tried from harddisk directory as well dev directory. 
What is "tc" and could you please paste how you run it on your system? Also the 2.6.23  GNU/linux we use is not fully compatible with the previously mentioned qdisc commit. For example, there is no qdisc_enqueue() function in our baseline and ditto for some of the other code. So seems risky to backport this patch. If we go w/ backporting it, i can post my diffs here, and would really appreciate a review from you. Regards.

Re: exit_mmap BUG_ON in 2.6.23 (and Add qdisc __NET_XMIT_STOLEN)

[Hugh Dickins]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 1694 bytes --]

On Fri, 25 May 2012, Sam Portolla wrote:
> 
> commit 378a2f090f7a478704a372a4869b8a9ac206234e
> Date:   Mon Aug 4 22:31:03 2008 -0700
> net_sched: Add qdisc __NET_XMIT_STOLEN flag
...
> 
>  I wonder if the lack of above patch in our code base could explain the
>  exit_mmap() BUG_ON as well due to memory corruption causing MMU to not
>  be able to locate the page(s) it had to free. NR_PTES keeps track of
>  that? Could you explain that more?

I concur with Eric in thinking it unlikely - though (unlike Eric)
I know far too little about networking to comment with authority.

I'd guess that there have been literally hundreds of fixes gone into
the kernel since 2.6.23, each more likely to be the fix to such memory
corruption than this one.  And I could also be wrong in attributing
your BUG to memory corruption: perhaps I'm forgetting an mm fix.

You ask me to explain more: mm->nr_ptes keeps track of the number of
page tables that have been allocated; when we free the mm, we should
be freeing exactly the number of page tables we allocated earlier,
but a bug in the code maintaining the vmas or the page tables might
break that, hence the BUG_ON to test.  But equally, if there has been
memory corruption of vmas or of higher-level page tables, we may now
be unable to locate all the page tables we allocated earlier, and so
hit the BUG_ON for that reason.

Would I be unfair to characterize this as a problem seen once at a
customer site in the 4.5 years since 2.6.23 was released?

As I said before, please just change that BUG_ON to WARN_ON, and
wait to see if more such issues come up: if they do, then you can
start to look for a pattern.

Hugh

Re: exit_mmap BUG_ON in 2.6.23 (and Add qdisc __NET_XMIT_STOLEN)

[Sam Portolla]

[please cc samPortolla@yahoo.com on the reply as not a member of this mailer]



----- Original Message -----
From: Hugh Dickins <hughd@google.com>
To: Sam Portolla <samportolla@yahoo.com>
Cc: Eric Dumazet <eric.dumazet@gmail.com>; "kaber@trash.net" <kaber@trash.net>; "jarkao2@gmail.com" <jarkao2@gmail.com>; "davem@davemloft.net" <davem@davemloft.net>; "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>
Sent: Saturday, May 26, 2012 11:06 AM
Subject: Re: exit_mmap BUG_ON in 2.6.23 (and Add qdisc __NET_XMIT_STOLEN)

On Fri, 25 May 2012, Sam Portolla wrote:
> 
> commit 378a2f090f7a478704a372a4869b8a9ac206234e
> Date:   Mon Aug 4 22:31:03 2008 -0700
> net_sched: Add qdisc __NET_XMIT_STOLEN flag
...
> 
>  I wonder if the lack of above patch in our code base could explain the
>  exit_mmap() BUG_ON as well due to memory corruption causing MMU to not
>  be able to locate the page(s) it had to free. NR_PTES keeps track of
>  that? Could you explain that more?

I concur with Eric in thinking it unlikely - though (unlike Eric)
I know far too little about networking to comment with authority.

I'd guess that there have been literally hundreds of fixes gone into
the kernel since 2.6.23, each more likely to be the fix to such memory
corruption than this one.  And I could also be wrong in attributing
your BUG to memory corruption: perhaps I'm forgetting an mm fix.

You ask me to explain more: mm->nr_ptes keeps track of the number of
page tables that have been allocated; when we free the mm, we should
be freeing exactly the number of page tables we allocated earlier,
but a bug in the code maintaining the vmas or the page tables might
break that, hence the BUG_ON to test.  But equally, if there has been
memory corruption of vmas or of higher-level page tables, we may now
be unable to locate all the page tables we allocated earlier, and so
hit the BUG_ON for that reason.

Would I be unfair to characterize this as a problem seen once at a
customer site in the 4.5 years since 2.6.23 was released?

As I said before, please just change that BUG_ON to WARN_ON, and
wait to see if more such issues come up: if they do, then you can
start to look for a pattern.

Hi Hugh, The concern I have with changing BUG_ON to WARN_ON, is one you had mentioned earlier in the thread.
If there is a memory corruption, BUG_ON causes system reboot and a clean start. WARN_ON won't and we might again end up crashing somewhere totally unrelated, possibly much later, i.e. unknown impact of this change. I know you know this area 100 times or more than me, but this is my concern and I am ready to be corrected, by all means.

Hugh and Eric, 

 Also, I can not get it out of my head that there was 3 instances of kernel crashes on the same system within 1 hour, all of them in different areas of the kernel and all of them just after the ethernet driver printed a transmit timeout message, which would come out if it the network layer saw the transmit Q's stopped. It seems really unlikely that 3 separate root causes exist in such a scenario. Therefore I keep thinking whatever caused the transmit timeout, also caused kernel memory corruption, which then manifested in different ways, namely: crash in the corresponding ethernet driver due to NULL ptr access, this BUG_ON and another NULL ptr access in buffer.c for the 3rd crash. The way I can think of unifying all these is that possibly the QDISC bug caused memory corruption AND also triggered the transmit timeout as it messed up the Tx Q to the device. We know when that QDISC issue happened originally couple of years ago, the same BNX2 driver had a
 NULL ptr access in its SKB area, because of what I explained in my reply to Eric yesterday. So, we may have a pure coincidence here, but in the absence of reproducibility, core file to analyze, and based on reasons above, I really am interested in this QDISC problem. Eric, could you kindly respond to the email I sent yesterday, with the above background in mind. Regards to both of you. 





Hugh