Re: Remotely triggered kernel panic on PPPoE + IPv6 enabled linux boxes

Re: Remotely triggered kernel panic on PPPoE + IPv6 enabled linux boxes

[Pasi Valminen]

> I can trigger a kernel panic from a remote host using tracepath6.
> First I connect to the internet using pppoe.
PPPoE is not needed to crash the kernel. Plain vanilla 2.6.7 will crash
just fine without it, seems like bringing up an ipv6 tunnel is enough.
Then just

$ tracepath6 <your tunnel ipv6 endpoint>

from a remote host and the kernel panics without any entry in the syslog.
But there is a quick and easy workaround. If you modprobe ip6table_filter,
the kernel won't panic!

Also when the ip6table_filter is not loaded, the tracepath never succeeds,
so I guess the first packet crashes the box. On the otherhand when
ip6table_filter is loaded, the tracepath succeeds as expected.

Re: Remotely triggered kernel panic on IPv6 enabled linux boxes (no more PPPoE)

[pp]

On Tue, 27 Jul 2004 17:38:23 +0300 (EEST)
Pasi Valminen wrote:

> > I can trigger a kernel panic from a remote host using tracepath6.
> > First I connect to the internet using pppoe.
> PPPoE is not needed to crash the kernel. Plain vanilla 2.6.7 will
> crash just fine without it, seems like bringing up an ipv6 tunnel is
> enough. Then just
> 
> $ tracepath6 <your tunnel ipv6 endpoint>
> 
> from a remote host and the kernel panics without any entry in the
> syslog. But there is a quick and easy workaround. If you modprobe
> ip6table_filter, the kernel won't panic!

I can confirm the whole thing.

I have:
cable modem & NIC, using DHCP, no PPPoE or anything like that
heartbeat tunnel from Sixxs.net
Gentoo Linux
Kernel 2.6.7 vanilla, gcc 3.3.3

I have worked with Pasi a couple of days now and he has crashed my
machine many times. With the help of a serial terminal I have gathered
some information about the crashes, including kernel oops/panic messages
and tcpdump output.
The information is here:
http://www.hack.fi/~pq/

My machine crashes immediately when someone runs a tracepath6 on me,
unless I have ip6table_filter module loaded.
To get to the vulnerable state all I have to do is to boot (to single
user mode), bring inet interface up and establish the tunnel.
We have not yet tested if it works with 6to4 connection also.

The data at the URL is somewhat censored, but I can provide the original
data and any other information if needed.
I am also willing to assist in solving the problem, try out patches etc.


I am not a subscriber to LKML.
-- 
Pekka "PQ" Paalanen

Re: Remotely triggered kernel panic on PPPoE + IPv6 enabled linux boxes

[Andrew Morton]

(Somebody has been editing email headers again.  Please do reply-to-all
when working on the kernel).

Pasi Valminen <okun@niksula.hut.fi> wrote:
>
> > I can trigger a kernel panic from a remote host using tracepath6.
>  > First I connect to the internet using pppoe.
>  PPPoE is not needed to crash the kernel. Plain vanilla 2.6.7 will crash
>  just fine without it, seems like bringing up an ipv6 tunnel is enough.
>  Then just
> 
>  $ tracepath6 <your tunnel ipv6 endpoint>

These problems were allegedly fixed post-2.6.7.  Please retest using the
latest kernel from ftp://ftp.kernel.org/pub/linux/kernel/v2.6/snapshots

Re: Remotely triggered kernel panic on PPPoE + IPv6 enabled linux boxes

[Pasi Valminen]

On Tue, 27 Jul 2004, Andrew Morton wrote:
> > Pasi Valminen <okun@niksula.hut.fi> wrote:
> >
> > Plain vanilla 2.6.7 will crash
> > just fine without it, seems like bringing up an ipv6 tunnel is enough.
> > Then just
> >
> > $ tracepath6 <your tunnel ipv6 endpoint>
>
> These problems were allegedly fixed post-2.6.7.  Please retest using the
> latest kernel from ftp://ftp.kernel.org/pub/linux/kernel/v2.6/snapshots

Confirmed, we tested today's snapshot (2.6.8-rc2-bk7) with Pekka Paalanen
and could not cause the kernel to panic.

Remotely triggered kernel panic on PPPoE + IPv6 enabled linux boxes

[Pasi Valminen]

Hello,

I can trigger a kernel panic from a remote host using tracepath6.
First I connect to the internet using pppoe.

Software:
rp-pppoe version 3.5, no patches
http://www.roaringpenguin.com/penguin/open_source_rp-pppoe.php
ppp version 2.4.2, no PPP packet filtering
ftp://ftp.samba.org/pub/ppp/

Then I configure an IPv6 tunnel to a tunnelbroker. I bring the tunnel up
and use tracepath6 from a remote host to trace some IPv6 address
configured to my net devices and the kernel panics.

# cat /proc/version
Linux version 2.6.7 (root@noc) (gcc version 3.3.3 (CRUX)) #1 SMP Wed Jul
21 14:38:22 UTC 2004

The output of the panic message obtained from a serial console, nothing
ends up in the syslog:

Unable to handle kernel paging request at virtual address edc8df7c
 printing eip:
*pde = 0046b067
*pte = 2dc8d000
Oops: 0000 [#1]
PREEMPT SMP DEBUG_PAGEALLOC
Modules linked in: uhci_hcd usbcore ipt_MASQUERADE iptable_nat ip_conntrack ip_d
CPU:    0
EIP:    0060:[<c02c8690>]    Not tainted
EFLAGS: 00010296   (2.6.7)
EIP is at ip6_output2+0x10/0x2c0
eax: edc8df50   ebx: ec50df40   ecx: ede8be20   edx: edc8df50
esi: 00000000   edi: 01000000   ebp: c0422d2c   esp: c0422d0c
ds: 007b   es: 007b   ss: 0068
Process pppoe (pid: 523, threadinfo=c0422000 task=ec6d8a60)
Stack: 00000292 c0465434 00000206 00000000 edc8df50 ec50df40 00000000 01000000
       c0422d98 c02c9d6d ec2a8000 00000286 ec2a8848 ef454b34 3a000400 000004d0
       000004d0 01000000 c026dd54 ec8becf0 00000028 000004d0 ec50df40
ec253f18
Call Trace:
Stack pointer is garbage, not printing trace
Code: 8b 40 2c 8b 58 10 66 c7 42 7c 86 dd 89 5a 18 8b 4d f0 8b 51
 <0>Kernel panic: Fatal exception in interrupt
In interrupt handler - not syncing

The output of the panic message run through ksymoops:
# ./ksymoops -v /usr/src/linux/vmlinux -k /proc/kallsyms < oops > ksymoops.out

ksymoops 2.4.9 on i686 2.6.7.  Options used
     -v /usr/src/linux/vmlinux (specified)
     -k /proc/kallsyms (specified)
     -l /proc/modules (default)
     -o /lib/modules/2.6.7/ (default)
     -m /usr/src/linux/System.map (default)

Warning (read_ksyms): no kernel symbols in ksyms, is /proc/kallsyms a
valid ksyms file?
No modules in ksyms, skipping objects
No ksyms, skipping lsmod
Unable to handle kernel paging request at virtual address edc8df7c
*pde = 0046b067
Oops: 0000 [#1]
CPU:    0
EIP:    0060:[<c02c8690>]    Not tainted
Using defaults from ksymoops -t elf32-i386 -a i386
EFLAGS: 00010296   (2.6.7)
eax: edc8df50   ebx: ec50df40   ecx: ede8be20   edx: edc8df50
esi: 00000000   edi: 01000000   ebp: c0422d2c   esp: c0422d0c
ds: 007b   es: 007b   ss: 0068
Stack: 00000292 c0465434 00000206 00000000 edc8df50 ec50df40 00000000 01000000
       c0422d98 c02c9d6d ec2a8000 00000286 ec2a8848 ef454b34 3a000400 000004d0
       000004d0 01000000 c026dd54 ec8becf0 00000028 000004d0 ec50df40 ec253f18
Call Trace:
Code: 8b 40 2c 8b 58 10 66 c7 42 7c 86 dd 89 5a 18 8b 4d f0 8b 51


>>EIP; c02c8690 <ip6_output2+10/2c0>   <=====

>>eax; edc8df50 <__crc_update_region+175f7a/2e64c7>
>>ebx; ec50df40 <__crc_proc_root+11c3bc/23c634>
>>ecx; ede8be20 <__crc_rtc_unregister+8d983/236a6b>
>>edx; edc8df50 <__crc_update_region+175f7a/2e64c7>
>>edi; 01000000 <__crc_class_device_get+6f7b6/e53a6>
>>ebp; c0422d2c <softirq_stack+d2c/2000>
>>esp; c0422d0c <softirq_stack+d0c/2000>

Code;  c02c8690 <ip6_output2+10/2c0>
00000000 <_EIP>:
Code;  c02c8690 <ip6_output2+10/2c0>   <=====
   0:   8b 40 2c                  mov    0x2c(%eax),%eax   <=====
Code;  c02c8693 <ip6_output2+13/2c0>
   3:   8b 58 10                  mov    0x10(%eax),%ebx
Code;  c02c8696 <ip6_output2+16/2c0>
   6:   66 c7 42 7c 86 dd         movw   $0xdd86,0x7c(%edx)
Code;  c02c869c <ip6_output2+1c/2c0>
   c:   89 5a 18                  mov    %ebx,0x18(%edx)
Code;  c02c869f <ip6_output2+1f/2c0>
   f:   8b 4d f0                  mov    0xfffffff0(%ebp),%ecx
Code;  c02c86a2 <ip6_output2+22/2c0>
  12:   8b 51 00                  mov    0x0(%ecx),%edx

 <0>Kernel panic: Fatal exception in interrupt

1 warning issued.  Results may not be reliable.

My net setup follows... The IP addresses has been obfuscated because I
need my box online and suspect that some malicious people might want to
crash it. :( If you think you need the real addresses, email me.

/sbin/modprobe 3c59x

/sbin/ifconfig lo 127.0.0.1

/usr/sbin/adsl-start # rp-pppoe script
/usr/sbin/ip tunnel add sit1 mode sit remote 1.2.3.4 local \
        `/usr/sbin/ip -o addr show dev ppp0 | /usr/bin/tail -n 1 | \
         /bin/sed -e "s:\(.*\) inet \(.*\) peer \(.*\):\2:"` \
        ttl 64 dev ppp0
# Setting MTU any lower fails with SIOCSIFMTU: Invalid argument :(
/usr/sbin/ip link set sit1 mtu 1280
/usr/sbin/ip link set sit1 up
/usr/sbin/ip -6 addr add 1::2/64 dev sit1 # Tunnel endpoint IPv6
/usr/sbin/ip -6 route add 2000::/3 dev sit1
# This enables the tunnel from the PoP side
/sbin/heartbeat-client /etc/heartbeat/heartbeat.conf
/usr/sbin/ip -6 addr add 2::3/48 dev sit1 # Additional IPv6
/usr/sbin/ip -6 addr add 3::4/48 dev sit1 # Additional IPv6

/usr/sbin/ip addr add 10.0.0.1/24 broadcast 10.0.0.255 dev eth1
/usr/sbin/ip -6 addr add 4::5/64 dev eth1
/usr/sbin/ip link set eth1 up

/bin/echo 1 > /proc/sys/net/ipv4/ip_forward
/usr/sbin/iptables -t nat -A POSTROUTING --proto ! 41 -o ppp0 -j MASQUERADE

After these steps I only need tracepath6 to crash the box from a remote
box.

$ tracepath6 4::5 # (IPv6 address obfuscated) crashes the box

System environment (same kernel, after the panic)

# cat /proc/cpuinfo
processor       : 0
vendor_id       : GenuineIntel
cpu family      : 6
model           : 7
model name      : Pentium III (Katmai)
stepping        : 3
cpu MHz         : 598.145
cache size      : 512 KB
fdiv_bug        : no
hlt_bug         : no
f00f_bug        : no
coma_bug        : no
fpu             : yes
fpu_exception   : yes
cpuid level     : 2
wp              : yes
flags           : fpu vme de tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat
pse36 mmx fxsr sse
bogomips        : 1179.64

processor       : 1
vendor_id       : GenuineIntel
cpu family      : 6
model           : 7
model name      : Pentium III (Katmai)
stepping        : 3
cpu MHz         : 598.145
cache size      : 512 KB
fdiv_bug        : no
hlt_bug         : no
f00f_bug        : no
coma_bug        : no
fpu             : yes
fpu_exception   : yes
cpuid level     : 2
wp              : yes
flags           : fpu vme de tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat
pse36 mmx fxsr sse
bogomips        : 1191.93

# cat /proc/modules
uhci_hcd 33616 0 - Live 0xf0970000
usbcore 118176 3 uhci_hcd, Live 0xf0952000
ipt_MASQUERADE 4576 1 - Live 0xf0998000
iptable_nat 24676 2 ipt_MASQUERADE, Live 0xf0931000
ip_conntrack 37040 2 ipt_MASQUERADE,iptable_nat, Live 0xf0a3a000
ip_tables 19008 2 ipt_MASQUERADE,iptable_nat, Live 0xf0894000
ppp_synctty 10400 0 - Live 0xf0890000
ppp_async 13568 1 - Live 0xf088b000
ppp_generic 31124 6 ppp_synctty,ppp_async, Live 0xf0882000
slhc 7392 1 ppp_generic, Live 0xf0870000
3c59x 40040 0 - Live 0xf0877000
aes 31712 2 - Live 0xf089d000
rd 6848 0 - Live 0xf0865000

# cat /proc/ioports
0000-001f : dma1
0020-0021 : pic1
0040-005f : timer
0060-006f : keyboard
0070-0077 : rtc
0080-008f : dma page reg
00a0-00a1 : pic2
00c0-00df : dma2
00f0-00ff : fpu
0170-0177 : ide1
01f0-01f7 : ide0
02f8-02ff : serial
0376-0376 : ide1
03c0-03df : vga+
03f6-03f6 : ide0
03f8-03ff : serial
0800-083f : 0000:00:07.3
0840-085f : 0000:00:07.3
0cf8-0cff : PCI conf1
cc00-cc7f : 0000:00:0e.0
  cc00-cc7f : 0000:00:0e.0
cce0-ccff : 0000:00:07.2
  cce0-ccff : uhci_hcd
d000-dfff : PCI Bus #02
  dc80-dcff : 0000:02:09.0
    dc80-dcff : 0000:02:09.0
e000-efff : PCI Bus #01
  ec00-ecff : 0000:01:00.0
ffa0-ffaf : 0000:00:07.1
  ffa0-ffa7 : ide0
  ffa8-ffaf : ide1

# cat /proc/iomem
00000000-0009ffff : System RAM
000a0000-000bffff : Video RAM area
000c0000-000c7fff : Video ROM
000c8000-000c87ff : Adapter ROM
000cc000-000cc7ff : Adapter ROM
000f0000-000fffff : System ROM
00100000-2fffdfff : System RAM
  00100000-002f88c6 : Kernel code
  002f88c7-003f529f : Kernel data
2fffe000-2fffffff : reserved
f0000000-f3ffffff : 0000:00:00.0
f5000000-f5ffffff : PCI Bus #02
f6000000-f6ffffff : PCI Bus #01
f9000000-faffffff : PCI Bus #02
  f9fffc00-f9fffc7f : 0000:02:09.0
fb000000-fdffffff : PCI Bus #01
  fbfff000-fbffffff : 0000:01:00.0
  fc000000-fcffffff : 0000:01:00.0
fe000000-fe00007f : 0000:00:0e.0
fec00000-fec0ffff : reserved
fee00000-fee0ffff : reserved
fff80000-ffffffff : reserved

The kernel config, the output of dmesg before the panic, and the output of
lspci -vvv are available here:
http://www.niksula.cs.hut.fi/~okun/{config,dmesg,lspci}

If you need additional information try something out or have a fix, please
email me. I'm not a subscriber of the lkml so I would appreciate CC.
Thanks.