kernel oops, fast ethernet bridge, 2.4.31

kernel oops, fast ethernet bridge, 2.4.31

[Lukasz Spaleniak]

[-- Attachment #1: Type: text/plain, Size: 919 bytes --]

Hello,

I have bridge firewall (linux box) with three fast ethernet cards (one
rtl8139 for management and two e100 for bridge). It is running 2.4.31
kernel and iptables v1.2.11. It works ok about one month. Few weeks
ago It started ooopsing. First thought was hardware, but it was
replaced with a new one and problem still exist. I'm attaching 
ksymoops analysis of ooops message. I hope I'll help.

Some facts:
Traffic over this bridge is about ~30 MBit/sec and every frame is
tagged with VID (802.1q). Problem exist also on 2.4.30 kernel.
Kernel is patched with ebtables patch. Ooops completeley freezes
machine, only hard reset takes effect. Nothing is placed into syslog.

The ksymooops analisis is attached as text file.

Anyone know this bug/problem ?

Regards,
Lukasz Spaleniak

-- 
spalek on wroc zigzag pl
GCM dpu s: a--- C++ UL++++ P+ L+++ E--- W+ N+ K- w O- M V-
PGP t--- 5 X+ R- tv-- b DI- D- G e-- h! r y+

[-- Attachment #2: lalala --]
[-- Type: application/octet-stream, Size: 6060 bytes --]

ksymoops 2.4.9 on i686 2.4.31.  Options used
     -V (default)
     -k ksyms (specified)
     -l moduly (specified)
     -o /lib/modules/2.4.31/ (default)
     -m /boot/System.map-2.4.31 (default)

skput:under: e08e7c48:1518 put:14 dev:eth2kernel BUG at skbuff.c:109!
invalid operand: 0000
CPU:    0
EIP:    0010:[<c01cd1cb>]    Not tainted
Using defaults from ksymoops -t elf32-i386 -a i386
EFLAGS: 00010282
eax: 0000002a   ebx: df3b8c80   ecx: c0286000   edx: df3c3f84
esi: ddbb3ba6   edi: ddf9b810   ebp: df3b8c80   esp: c0287b48
ds: 0018   es: 0018   ss: 0018
Process swapper (pid: 0, stackpage=c0287000)
Stack: c023b4e0 e08e7c48 000005ee 0000000e dfae8c00 e08e7c51 df3b8c80 0000000e
       e08e7c48 df3b8c80 ddf9b810 ddf9b824 000005c8 c01e634c df3b8c80 00000014
       ddf9b824 000005c8 000005dc 00000014 00000000 000005c8 00000009 000005c8
Call Trace:    [<e08e7c48>] [<e08e7c51>] [<e08e7c48>] [<c01e634c>] [<e08ecec0>]
  [<e08ed020>] [<e08e7bb0>] [<e0900684>] [<e08e7bb0>] [<e08e7bb0>] [<e09073d0>]
  [<c01d9c6a>] [<e08e7bb0>] [<c01d9fd9>] [<e08e7bb0>] [<e09073d0>] [<e08ec34d>]
  [<e08e7bb0>] [<e08ed140>] [<c01d9c6a>] [<e08e7bb0>] [<c01d9fd9>] [<e08e7bb0>]
  [<e08ed140>] [<e08e7cf3>] [<e08e7bb0>] [<e08ebd0c>] [<e08e7cb0>] [<e08ebe6f>]
  [<e08ebc80>] [<e08ed110>] [<c01d9c6a>] [<e08e7cb0>] [<c01d9fd9>] [<e08e7cb0>]
  [<e08ed110>] [<e08e7dea>] [<e08e7cb0>] [<e08e88df>] [<e08e8820>] [<e08eb362>]
  [<e08e8820>] [<e08eb250>] [<e08ebade>] [<e08eb250>] [<e08ed0e0>] [<c01d9c6a>]
  [<e08e8820>] [<c01d9fd9>] [<e08e8820>] [<e08ed0e0>] [<e08e8820>] [<e08e8aa2>]
  [<e08e8820>] [<c01d1289>] [<c01d1463>] [<c01d1564>] [<c0118b05>] [<c01088fb>]
  [<c01052c0>] [<c010ac78>] [<c01052c0>] [<c01052e3>] [<c0105372>] [<c0105000>]
Code: 0f 0b 6d 00 5f a5 23 c0 83 c4 14 c3 89 f6 8d bc 27 00 00 00


>>EIP; c01cd1cb <skb_under_panic+3b/50>   <=====

>>ebx; df3b8c80 <_end+1f0f4ffc/205f83dc>
>>ecx; c0286000 <init_task_union+0/2000>
>>edx; df3c3f84 <_end+1f100300/205f83dc>
>>esi; ddbb3ba6 <_end+1d8eff22/205f83dc>
>>edi; ddf9b810 <_end+1dcd7b8c/205f83dc>
>>ebp; df3b8c80 <_end+1f0f4ffc/205f83dc>
>>esp; c0287b48 <init_task_union+1b48/2000>

Trace; e08e7c48 <[bridge]br_dev_queue_push_xmit+98/100>
Trace; e08e7c51 <[bridge]br_dev_queue_push_xmit+a1/100>
Trace; e08e7c48 <[bridge]br_dev_queue_push_xmit+98/100>
Trace; c01e634c <ip_fragment+2ec/3e0>
Trace; e08ecec0 <[bridge]__fake_net_device+0/160>
Trace; e08ed020 <[bridge]__fake_rtable+0/c0>
Trace; e08e7bb0 <[bridge]br_dev_queue_push_xmit+0/100>
Trace; e0900684 <[ip_conntrack]ip_refrag+74/80>
Trace; e08e7bb0 <[bridge]br_dev_queue_push_xmit+0/100>
Trace; e08e7bb0 <[bridge]br_dev_queue_push_xmit+0/100>
Trace; e09073d0 <[ip_conntrack]ip_conntrack_out_ops+0/18>
Trace; c01d9c6a <nf_iterate+6a/b0>
Trace; e08e7bb0 <[bridge]br_dev_queue_push_xmit+0/100>
Trace; c01d9fd9 <nf_hook_slow+99/1f0>
Trace; e08e7bb0 <[bridge]br_dev_queue_push_xmit+0/100>
Trace; e09073d0 <[ip_conntrack]ip_conntrack_out_ops+0/18>
Trace; e08ec34d <[bridge]br_nf_post_routing+13d/240>
Trace; e08e7bb0 <[bridge]br_dev_queue_push_xmit+0/100>
Trace; e08ed140 <[bridge]br_nf_ops+60/140>
Trace; c01d9c6a <nf_iterate+6a/b0>
Trace; e08e7bb0 <[bridge]br_dev_queue_push_xmit+0/100>
Trace; c01d9fd9 <nf_hook_slow+99/1f0>
Trace; e08e7bb0 <[bridge]br_dev_queue_push_xmit+0/100>
Trace; e08ed140 <[bridge]br_nf_ops+60/140>
Trace; e08e7cf3 <[bridge]br_forward_finish+43/60>
Trace; e08e7bb0 <[bridge]br_dev_queue_push_xmit+0/100>
Trace; e08ebd0c <[bridge]br_nf_forward_finish+8c/e0>
Trace; e08e7cb0 <[bridge]br_forward_finish+0/60>
Trace; e08ebe6f <[bridge]br_nf_forward_ip+10f/160>
Trace; e08ebc80 <[bridge]br_nf_forward_finish+0/e0>
Trace; e08ed110 <[bridge]br_nf_ops+30/140>
Trace; c01d9c6a <nf_iterate+6a/b0>
Trace; e08e7cb0 <[bridge]br_forward_finish+0/60>
Trace; c01d9fd9 <nf_hook_slow+99/1f0>
Trace; e08e7cb0 <[bridge]br_forward_finish+0/60>
Trace; e08ed110 <[bridge]br_nf_ops+30/140>
Trace; e08e7dea <[bridge]__br_forward+5a/70>
Trace; e08e7cb0 <[bridge]br_forward_finish+0/60>
Trace; e08e88df <[bridge]br_handle_frame_finish+bf/160>
Trace; e08e8820 <[bridge]br_handle_frame_finish+0/160>
Trace; e08eb362 <[bridge]br_nf_pre_routing_finish+112/2b0>
Trace; e08e8820 <[bridge]br_handle_frame_finish+0/160>
Trace; e08eb250 <[bridge]br_nf_pre_routing_finish+0/2b0>
Trace; e08ebade <[bridge]br_nf_pre_routing+29e/410>
Trace; e08eb250 <[bridge]br_nf_pre_routing_finish+0/2b0>
Trace; e08ed0e0 <[bridge]br_nf_ops+0/140>
Trace; c01d9c6a <nf_iterate+6a/b0>
Trace; e08e8820 <[bridge]br_handle_frame_finish+0/160>
Trace; c01d9fd9 <nf_hook_slow+99/1f0>
Trace; e08e8820 <[bridge]br_handle_frame_finish+0/160>
Trace; e08ed0e0 <[bridge]br_nf_ops+0/140>
Trace; e08e8820 <[bridge]br_handle_frame_finish+0/160>
Trace; e08e8aa2 <[bridge]br_handle_frame+122/1e0>
Trace; e08e8820 <[bridge]br_handle_frame_finish+0/160>
Trace; c01d1289 <netif_receive_skb+b9/210>
Trace; c01d1463 <process_backlog+83/110>
Trace; c01d1564 <net_rx_action+74/110>
Trace; c0118b05 <do_softirq+95/a0>
Trace; c01088fb <do_IRQ+9b/a0>
Trace; c01052c0 <default_idle+0/40>
Trace; c010ac78 <call_do_IRQ+5/d>
Trace; c01052c0 <default_idle+0/40>
Trace; c01052e3 <default_idle+23/40>
Trace; c0105372 <cpu_idle+52/70>
Trace; c0105000 <_stext+0/0>

Code;  c01cd1cb <skb_under_panic+3b/50>
00000000 <_EIP>:
Code;  c01cd1cb <skb_under_panic+3b/50>   <=====
   0:   0f 0b                     ud2a      <=====
Code;  c01cd1cd <skb_under_panic+3d/50>
   2:   6d                        insl   (%dx),%es:(%edi)
Code;  c01cd1ce <skb_under_panic+3e/50>
   3:   00 5f a5                  add    %bl,0xffffffa5(%edi)
Code;  c01cd1d1 <skb_under_panic+41/50>
   6:   23 c0                     and    %eax,%eax
Code;  c01cd1d3 <skb_under_panic+43/50>
   8:   83 c4 14                  add    $0x14,%esp
Code;  c01cd1d6 <skb_under_panic+46/50>
   b:   c3                        ret    
Code;  c01cd1d7 <skb_under_panic+47/50>
   c:   89 f6                     mov    %esi,%esi
Code;  c01cd1d9 <skb_under_panic+49/50>
   e:   8d bc 27 00 00 00 00      lea    0x0(%edi),%edi

 <0>Kernel panic: Aiee, killing interrupt handler!

Re: kernel oops, fast ethernet bridge, 2.4.31

[Willy Tarreau]

Hello,

just some basic questions :
  - did your configuration change before the oopses started ? (eg: new
    matches, etc...)

  - did the traffic change recently (protocols, data rate) ? eg: new
    applications on the network, etc...

  - is it possible that it's being targetted by an attack where it is
    installed (unfiltered internet, holiday employees who like to play
    with the network, etc...) ?

I really find it strange that it suddenly started oopsing if nothing
changed. At least it should have been oopsing from day one.

Regards,
willy

On Wed, Jul 20, 2005 at 05:00:25PM +0200, Lukasz Spaleniak wrote:
> Hello,
> 
> I have bridge firewall (linux box) with three fast ethernet cards (one
> rtl8139 for management and two e100 for bridge). It is running 2.4.31
> kernel and iptables v1.2.11. It works ok about one month. Few weeks
> ago It started ooopsing. First thought was hardware, but it was
> replaced with a new one and problem still exist. I'm attaching 
> ksymoops analysis of ooops message. I hope I'll help.
> 
> Some facts:
> Traffic over this bridge is about ~30 MBit/sec and every frame is
> tagged with VID (802.1q). Problem exist also on 2.4.30 kernel.
> Kernel is patched with ebtables patch. Ooops completeley freezes
> machine, only hard reset takes effect. Nothing is placed into syslog.
> 
> The ksymooops analisis is attached as text file.
> 
> Anyone know this bug/problem ?
> 
> Regards,
> Lukasz Spaleniak
> 
> -- 
> spalek on wroc zigzag pl
> GCM dpu s: a--- C++ UL++++ P+ L+++ E--- W+ N+ K- w O- M V-
> PGP t--- 5 X+ R- tv-- b DI- D- G e-- h! r y+

Re[2]: kernel oops, fast ethernet bridge, 2.4.31

[Lukasz Spaleniak]

On Wednesday, July 20, 2005, 9:44:57 PM, Willy Tarreau wrote:

> Hello,
Hello Willy,

> just some basic questions :
>   - did your configuration change before the oopses started ? (eg: new
>     matches, etc...)
One new machine appears but it generates small traffic rate (by now
it's almost unused).

>   - did the traffic change recently (protocols, data rate) ? eg: new
>     applications on the network, etc...
No - firewall is bridging IPv4 only. There was no dramatic topology
change. Those VLANs which are going through this firewall were
untouched.

>   - is it possible that it's being targetted by an attack where it is
>     installed (unfiltered internet, holiday employees who like to play
>     with the network, etc...) ?
I don't think so that managed IP of firewall was targetet, maybe
machines behid firewall but problem appears on eth2 interface which
is:
internet <-trunk-> eth1(firewall/iptables)eth2<-trunk->(switch
ports) <-> servers
So it's after iptables ...

> I really find it strange that it suddenly started oopsing if nothing
> changed. At least it should have been oopsing from day one.
It is strange to me too. There is no dependency when it happens.
Sometimes traffic is small, sometimes it's normal. Packet rates are
around ~2000-3000 pkt/sec - so not so high.


Regards,
Lukasz

-- 
lspaleniak on wroc zigzag pl
GCM dpu s: a--- C++ UL++++ P+ L+++ E--- W+ N+ K- w O- M V-
PGP t--- 5 X+ R- tv-- b DI- D- G e-- h! r y+

Re: Re[2]: kernel oops, fast ethernet bridge, 2.4.31

[Joy Leima]

Lukasz Spaleniak <lspaleniak <at> wroc.zigzag.pl> writes:

> 
> On Wednesday, July 20, 2005, 9:44:57 PM, Willy Tarreau wrote:
 > changed. At least it should have been oopsing from day one.
> It is strange to me too. There is no dependency when it happens.
> Sometimes traffic is small, sometimes it's normal. Packet rates are
> around ~2000-3000 pkt/sec - so not so high.
> 
> Regards,
> Lukasz
> 
Lukasz,

I think I have a fix for you.  Verify for me that it is the same problem.  Send
a large UDP packet through the bridge.  I believe the problem is the ip_fragment
code is not taking into account the VLAN header that needs to be added to the
packet when it gets fragmented on the way out.   

Just send the large UDP packet through the bridge.  I use ttcp.  If it panics
then I can send you the fix.  There are further changed to ip_output.c

Re: Re[2]: kernel oops, fast ethernet bridge, 2.4.31

[Lukasz Spaleniak]

On Fri, 22 Jul 2005 15:13:33 +0000 (UTC)
Joy Leima <jleima@comcast.net> wrote:

> Lukasz,
> 
> I think I have a fix for you.  Verify for me that it is the same
> problem.  Send a large UDP packet through the bridge.  I believe the
> problem is the ip_fragment code is not taking into account the VLAN
> header that needs to be added to the packet when it gets fragmented
> on the way out.   
> 
> Just send the large UDP packet through the bridge.  I use ttcp.  If
> it panics then I can send you the fix.  There are further changed to
> ip_output.c

Hello Joy,

This is exactly this situation which you described.
Could you be so kind to send me this patch ?

Best regards,
Lukasz Spaleniak

-- 
lspaleniak on wroc zigzag pl
GCM dpu s: a--- C++ UL++++ P+ L+++ E--- W+ N+ K- w O- M V-
PGP t--- 5 X+ R- tv-- b DI- D- G e-- h! r y+