Re: do_IRQ: stack overflow: 872..

Re: do_IRQ: stack overflow: 872..

[Andi Kleen]

Crazy AMD K7 <snort2004@mail.ru> writes:

> Hi!
> I have found a few days ago strange messages in /var/log/messages
> More than 10 times there was do_IRQ: stack overflow: (nimber).... followed
> with code. If need I can send all this data. I have run
> ksymoops with only first 3 cases. Here is the first, the second and
> the third are in attachment.
> After that oopses my system continued to work.

It's not really an oops, just a warning that stack space got quiet tight.

The problem seems to be that the br netfilter code is nesting far too
deeply and recursing several times. Looks like a design bug to me,
it shouldn't do that.

> uname uname -a
> Linux linux 2.4.28 #2 ÷ÔÒ îÏÑ 30 15:43:35 MSK 2004 i686 unknown
> gcc -v
> Reading specs from /usr/lib/gcc-lib/i386-redhat-linux/2.96/specs
> gcc version 2.96 20000731 (Red Hat Linux 7.3 2.96-113)
> I have applies ebtables_brnf patch (http://bridge.sf.net) and a

Don't do that then or contact the author to fix it. Unfortunately
the code is also in 2.6 mainline.

-Andi

Re: do_IRQ: stack overflow: 872..

[Bart De Schuymer]

Op za, 18-12-2004 te 08:50 +0100, schreef Andi Kleen:
> Crazy AMD K7 <snort2004@mail.ru> writes:
> 
> > Hi!
> > I have found a few days ago strange messages in /var/log/messages
> > More than 10 times there was do_IRQ: stack overflow: (nimber).... followed
> > with code. If need I can send all this data. I have run
> > ksymoops with only first 3 cases. Here is the first, the second and
> > the third are in attachment.
> > After that oopses my system continued to work.
> 
> It's not really an oops, just a warning that stack space got quiet tight.
> 
> The problem seems to be that the br netfilter code is nesting far too
> deeply and recursing several times. Looks like a design bug to me,
> it shouldn't do that.
> 
> > uname uname -a
> > Linux linux 2.4.28 #2 ÷ÔÒ îÏÑ 30 15:43:35 MSK 2004 i686 unknown
> > gcc -v
> > Reading specs from /usr/lib/gcc-lib/i386-redhat-linux/2.96/specs
> > gcc version 2.96 20000731 (Red Hat Linux 7.3 2.96-113)
> > I have applies ebtables_brnf patch (http://bridge.sf.net) and a
> 
> Don't do that then or contact the author to fix it. Unfortunately
> the code is also in 2.6 mainline.
> 

Hi.

The bridge-nf code does not use recursive function calls and there is no
long consecutive function calling. Furthermore, there is no function in
the bridge-nf code that uses a large part of the stack.
Andi, if you make such statements then please point out the code part
you have (of course) read after which you decided to make the statement.
The bridge-nf code is used by quite a few people and by commercial
companies and I have never had a report like this. AMD has been having
all sorts of strange problems for weeks now, they're all somehow related
to bridge-nf, but I doubt he is using the bridge-nf patch on a clean 2.4
kernel.
AMD, is there any chance you can use the latest 2.6 kernel, without
extra patches ?

Bart

Re: do_IRQ: stack overflow: 872..

[Andi Kleen]

> The bridge-nf code does not use recursive function calls and there is no
> long consecutive function calling. Furthermore, there is no function in
> the bridge-nf code that uses a large part of the stack.

Just take a look at the backtrace in the original post. It clearly
shows a problem. And it points strongly towards br-netfilter.

Re: do_IRQ: stack overflow: 872..

[Bart De Schuymer]

Op za, 18-12-2004 te 12:14 +0100, schreef Andi Kleen:
> > The bridge-nf code does not use recursive function calls and there is no
> > long consecutive function calling. Furthermore, there is no function in
> > the bridge-nf code that uses a large part of the stack.
> 
> Just take a look at the backtrace in the original post. It clearly
> shows a problem. And it points strongly towards br-netfilter.

I don't doubt you are a much better reader of such backtraces than me.
However, let's count the number of times a function from
net/bridge/br_netfilter.c is in the backtrace:
1. br_nf*: 6 times
2. *sabotage*: 3 times
Seriously, out of 222 lines, only 9 from bridge-nf.
The function ip_queue_xmit, OTOH, is 8 times in the trace.

Anyway, as I already suspected weeks ago, AMD must be seeing some
incompatibility between ip_queue (he's using snort) and the bridge-nf
patch.

He is using the patch (I gave it to him) below on top of the bridge-nf
patch. Before using that patch he got a kernel panic occasionally.
However he seems not to get a message in his syslog.

Bart


--- linux-2.4.28-ebt-brnf/net/bridge/br_netfilter.c.old 2004-11-27 23:43:18.000000000 +0100
+++ linux-2.4.28-ebt-brnf/net/bridge/br_netfilter.c     2004-11-27
23:52:05.000000000 +0100
@@ -870,6 +870,10 @@ static unsigned int ip_sabotage_out(unsi
 {
        struct sk_buff *skb = *pskb;
 
+if (!skb) {
+       printk("TROUBLE IN IP_SABOTAGE_OUT: skb==NULL\n");
+       goto in_trouble;
+}
 #ifdef CONFIG_SYSCTL
        if (!skb->nf_bridge) {
                struct vlan_ethhdr *hdr =
@@ -884,6 +888,10 @@ static unsigned int ip_sabotage_out(unsi
        }
 #endif
 
+if (!out) {
+       printk("TROUBLE IN IP_SABOTAGE_OUT: out == NULL\n");
+       goto in_trouble;
+}
        if ((out->hard_start_xmit == br_dev_xmit &&
            okfn != br_nf_forward_finish &&
            okfn != br_nf_local_out_finish &&
@@ -920,6 +928,9 @@ static unsigned int ip_sabotage_out(unsi
        }
 
        return NF_ACCEPT;
+in_trouble:
+       dump_stack();
+       return NF_DROP;
 }
 
 /* For br_nf_local_out we need (prio = NF_BR_PRI_FIRST), to insure that
innocent

Re: do_IRQ: stack overflow: 872..

[Andi Kleen]

On Sat, Dec 18, 2004 at 12:51:30PM +0100, Bart De Schuymer wrote:
> > 
> > Just take a look at the backtrace in the original post. It clearly
> > shows a problem. And it points strongly towards br-netfilter.
> 
> I don't doubt you are a much better reader of such backtraces than me.
> However, let's count the number of times a function from
> net/bridge/br_netfilter.c is in the backtrace:
> 1. br_nf*: 6 times
> 2. *sabotage*: 3 times
> Seriously, out of 222 lines, only 9 from bridge-nf.
> The function ip_queue_xmit, OTOH, is 8 times in the trace.

Yep, but ip_queue_xmit doesn't call itself recursively. Someone 
must be doing it. And that's likely the bridge code.

BTW not all of these entries are probably true, there can 
be a lot of false positives.

> Anyway, as I already suspected weeks ago, AMD must be seeing some
> incompatibility between ip_queue (he's using snort) and the bridge-nf
> patch.
> 
> He is using the patch (I gave it to him) below on top of the bridge-nf
> patch. Before using that patch he got a kernel panic occasionally.
> However he seems not to get a message in his syslog.

Ok, since this report seems to be for a totally non standard
severly hacked up kernel I suppose nothing from it can be concluded for
the mainline kernel. Thanks for clearing this up.

Note to the original poster: when you report a bug with a patched
kernel always mention it.

-Andi

Re[2]: do_IRQ: stack overflow: 872..

[Crazy AMD K7]

> Note to the original poster: when you report a bug with a patched
> kernel always mention it.
I have mentioned earlier and Bart knows it.

I use 2.4.28
+ ebtables-brnf-8_vs_2.4.28.diff
+ U32 patch from patch-o-matic-ng-20040621.tar.bz2
+ patch for br_netfilter.c made by Bart to find out why kernel panic happens(it was a few
  letters ago)
  All patches has applies cleanly.
  U32 doesn't affect on br_netfilter.c

[root@linux kernel]# md5sum linux-2.4.28.tar.bz2
ac7735000d185bc7778c08288760a8a3  linux-2.4.28.tar.bz2
(taken from http://www.ru.kernel.org/pub/linux/kernel/v2.4/linux-2.4.28.tar.bz2)

[root@linux bridge]# md5sum ebtables-brnf-8_vs_2.4.28.diff.gz
30542b1a7a502593afb4d37055ec5e35  ebtables-brnf-8_vs_2.4.28.diff.gz

[root@linux iptables]# md5sum patch-o-matic-ng-20040621.tar.bz2
4fd3c744bf55f119fef6c7c3c4acc4b6  patch-o-matic-ng-20040621.tar.bz2

If the problem will continue appear and will not be solved an any
way, of course, I will use 2.6 kernel, now I am not ready to use it.

Pasha

Re: Re[2]: do_IRQ: stack overflow: 872..

[Bart De Schuymer]

Op za, 18-12-2004 te 19:07 +0300, schreef Crazy AMD K7:
> > Note to the original poster: when you report a bug with a patched
> > kernel always mention it.
> I have mentioned earlier and Bart knows it.
> 
> I use 2.4.28
> + ebtables-brnf-8_vs_2.4.28.diff
> + U32 patch from patch-o-matic-ng-20040621.tar.bz2
> + patch for br_netfilter.c made by Bart to find out why kernel panic happens(it was a few
>   letters ago)
>   All patches has applies cleanly.
>   U32 doesn't affect on br_netfilter.c

Sorry, I don't know the ip_queue mechanism and I don't know what could
possibly go wrong.
All we know is that you no longer have kernel panics with the simple
patch I gave you (which just drops packets when a kernel panic would
happen otherwise, and tells about this with a printk). However, you
state there are no entries in your syslog that tell about this dropping.
Is your syslog working right? Do you have a console open on which kernel
messages get printed?

I still secretly suspect the snort code of inserting packets back into
the kernel that don't have an output device (I don't know if that's
possible, though).

cheers,
Bart

Re: do_IRQ: stack overflow: 872..

[David Woodhouse]

On Sat, 2004-12-18 at 08:50 +0100, Andi Kleen wrote:
> It's not really an oops, just a warning that stack space got quiet
> tight.
> 
> The problem seems to be that the br netfilter code is nesting far too
> deeply and recursing several times. Looks like a design bug to me,
> it shouldn't do that.

I don't think it's recursing -- I think the stack trace is just a bit
noisy. The problem is that the bridge code, especially with br_netfilter
in the equation, is implicated in code paths which are just _too_ deep.
This happens when you're bridging packets received in an interrupt while
you were deep in journalling code, and it's also been seen with a call
trace something like nfs->sunrpc->ip->bridge->br_netfilter.

One option might be to make br_dev_xmit() just queue the packet rather
than trying to deliver it to all the slave devices immediately. Then the
actual retransmission can be handled from a context where we're _not_
short of stack; perhaps from a dedicated kernel thread. 

Unfortunately that approach would introduce a lot of latency on all
packets we pass. Another option would be to have all architectures
provide a stack_available() function and for br_dev_xmit() to queue the
packet only if we're short of stack, while still sending most packets
immediately. 

Proof of concept below; obviously the stack_available() is an evil hack
and would need to be done more sanely. Comments?

===== net/bridge/br_device.c 1.17 vs edited =====
--- 1.17/net/bridge/br_device.c	2004-07-29 22:40:51 +01:00
+++ edited/net/bridge/br_device.c	2005-01-07 16:54:26 +00:00
@@ -19,6 +19,13 @@
 #include <asm/uaccess.h>
 #include "br_private.h"
 
+static inline unsigned long stack_available(void)
+{
+	unsigned long esp;
+	asm volatile("movl %%esp,%0" : "=r"(esp));
+	return esp - (unsigned long)current - sizeof(struct thread_info);
+}
+
 static struct net_device_stats *br_dev_get_stats(struct net_device *dev)
 {
 	struct net_bridge *br;
@@ -34,6 +41,14 @@
 	const unsigned char *dest = skb->data;
 	struct net_bridge_fdb_entry *dst;
 
+	if (stack_available() < THREAD_SIZE/2) {
+		if (net_ratelimit()) {
+			printk(KERN_DEBUG "Bridge device %s queues packet due to stack shortage\n",
+			       dev->name);
+		}
+		return NETDEV_TX_BUSY;
+	}
+
 	br->statistics.tx_packets++;
 	br->statistics.tx_bytes += skb->len;
 
@@ -104,7 +119,7 @@
 	SET_MODULE_OWNER(dev);
 	dev->stop = br_dev_stop;
 	dev->accept_fastpath = br_dev_accept_fastpath;
-	dev->tx_queue_len = 0;
+	dev->tx_queue_len = 5;
 	dev->set_mac_address = NULL;
 	dev->priv_flags = IFF_EBRIDGE;
 }




-- 
dwmw2