2.6.10 : kernel BUG at mm/rmap.c:483!

2.6.10 : kernel BUG at mm/rmap.c:483!

[Erik de Castro Lopo]

Hi all,

I'm running a kernel compiled from pristine 2.6.10 sources
downloaded from kernel.org on a P4 with HT. I was originally
working on an SMP/CONFIG_PREEMPT kernel when I first hit this
problem. I downgraded to an SMP kernel without CONFIG_PREEMPT 
the problem remained. Same problem on a kernel compiled for a 
single CPU. The dmesg output below is for the single CPU, no 
CONFIG_PREEMPT case.

The problem is triggered by a user space application which 
mmaps memory space on a custom PCI card via /dev/mem, reads 
some data and munmaps it. The program works perfectly on a 
2.4.26 kernel. On 2.6.10 I get this on the console:

------------[ cut here ]------------
kernel BUG at mm/rmap.c:483!
invalid operand: 0000 [#1]
Modules linked in:
CPU:    0
EIP:    0060:[<c013b5b6>]    Not tainted VLI
EFLAGS: 00010286   (2.6.10) 
EIP is at page_remove_rmap+0x26/0x40
eax: ffffffff   ebx: 00003000   ecx: b5e86000   edx: c1000060
esi: dabd9a24   edi: c1000060   ebp: 0017a000   esp: da40febc
ds: 007b   es: 007b   ss: 0068
Process counters-erikd (pid: 1622, threadinfo=da40e000 task=da9d6a60)
Stack: c0135a38 dfd0f878 00000002 f277a8c0 00000000 00003027 b5e86000 c049e480 
       b6286000 da43eb60 b6000000 c049e480 c0135b9b 0017a000 00000000 de16f030 
       dfc82000 b5e86000 da43eb60 b6286000 c049e480 c0135bfd 00400000 00000000 
Call Trace:
 [<c0135a38>] zap_pte_range+0x138/0x250
 [<c0135b9b>] zap_pmd_range+0x4b/0x70
 [<c0135bfd>] unmap_page_range+0x3d/0x70
 [<c0135d2e>] unmap_vmas+0xfe/0x1b0
 [<c013955a>] unmap_region+0x6a/0xd0
 [<c01397f4>] do_munmap+0xf4/0x130
 [<c0139870>] sys_munmap+0x40/0x70
 [<c01022f3>] syscall_call+0x7/0xb
Code: 90 8d 74 26 00 89 c2 8b 00 f6 c4 08 75 27 83 42 08 ff 0f 98 c0 84 c0 74 11 8b 42 08 40 78 0c 9c 58 fa ff 0d 50 65 4a c0 50 9d c3 <0f> 0b e3 01 41 2c 38 c0 eb ea 0f 0b e0 01 41 2c 38 c0 eb cf 8d 
 
Is there anything I have missed? Does /dev/mem on a 2.6
kerel behave the same way as on a 2.4 kernel?

I can supply more info if needed.

Regards,
Erik
-- 
-------------------------------------------------------
[N] Erik de Castro Lopo, Senior Computer Engineer
[E] erik.de.castro.lopo@sensorynetworks.com
[W] http://www.sensorynetworks.com
[T] +61 2 83022726
[F] +61 2 94750316
[A] L6/140 William St, East Sydney NSW 2011, Australia
-------------------------------------------------------
A good debugger is no substitute for a good test suite.

Re: 2.6.10 : kernel BUG at mm/rmap.c:483!

[Hugh Dickins]

On Wed, 16 Feb 2005, Erik de Castro Lopo wrote:
> 
> The problem is triggered by a user space application which 
> mmaps memory space on a custom PCI card via /dev/mem, reads 
> some data and munmaps it. The program works perfectly on a 
> 2.4.26 kernel. On 2.6.10 I get this on the console:
> 
> kernel BUG at mm/rmap.c:483!
> EIP is at page_remove_rmap+0x26/0x40
>  [<c0135a38>] zap_pte_range+0x138/0x250 ...
>  [<c0139870>] sys_munmap+0x40/0x70
> 
> Is there anything I have missed? Does /dev/mem on a 2.6
> kerel behave the same way as on a 2.4 kernel?

Yes, I believe /dev/mem handling on 2.4 and 2.6 is much the same,
but the rmap.c BUG you're hitting in 2.6 had no equivalent in 2.4 -
maybe the 2.4 case has actually been unsafe too, or maybe not.

Though there have been other reports of this BUG, some remaining
mysterious, some down to bad ram, your case is almost certainly
due to mismatched PageReserved accounting.

mmap of /dev/mem only works usefully when the mapped pages are
outside of ordinary ram, or temporarily marked PageReserved,
normally by the relevant driver.

It seems that when the user space application mapped that part of
/dev/mem, the pages were marked PageReserved, and rmap accounting
skipped.  But when the user space application unmapped that part
of /dev/mem, the pages were no longer marked PageReserved, so
rmap accounting hit the BUG.

I may be wrong, but I think it's fair to say that usually when
there's to be cooperation between driver and user space of this
kind, the driver would offer its own device and mmap entry point,
user space would be mapping that rather than /dev/mem, and file
reference counting would avoid the possibility that driver clears
PageReserved while the page is mapped.

Beyond this particular BUG, clearing PageReserved too soon is
likely to lead to freeing an already freed page - noticed with
a warning if the page is still free, but mayhem if by then reused.

Which is the driver involved?

Or, might the user space application be (arguably) at fault,
mmap'ing an indeterminate uncontrolled area of /dev/mem, some
pages of which may or may not be PageReserved at any instant?

(Yes, PageReserved is an unsatisfactory notion we should have
weaned the kernel off long ago.)

Hugh

Re: 2.6.10 : kernel BUG at mm/rmap.c:483!

[Erik de Castro Lopo]

On Wed, 16 Feb 2005 08:07:11 +0000 (GMT)
Hugh Dickins <hugh@veritas.com> wrote:

> Yes, I believe /dev/mem handling on 2.4 and 2.6 is much the same,
> but the rmap.c BUG you're hitting in 2.6 had no equivalent in 2.4 -
> maybe the 2.4 case has actually been unsafe too, or maybe not.

OK, that makes sense.

> Though there have been other reports of this BUG, some remaining
> mysterious, some down to bad ram, your case is almost certainly
> due to mismatched PageReserved accounting.
> 
> mmap of /dev/mem only works usefully when the mapped pages are
> outside of ordinary ram, 

This is memory on a PCI card which is definitely not part of
ordinary ram.

> or temporarily marked PageReserved, normally by the relevant driver.
>
> I may be wrong, but I think it's fair to say that usually when
> there's to be cooperation between driver and user space of this
> kind, the driver would offer its own device and mmap entry point,
> user space would be mapping that rather than /dev/mem, and file

Since I wrote the driver, thats definitely a possibility.

> reference counting would avoid the possibility that driver clears
> PageReserved while the page is mapped.

Ahh, here's a clue. My driver uses ioremap_nocache() to get virtual
pointers to physical register/memory (addresses obtain using 
pci_resource_start etc) on the card and calls iounmap() when the 
driver is unloaded. However, while the driver is loaded, the user 
space program mmaps a superset of the same memory. Is it possible 
that this is screwing up the page accounting?

> Beyond this particular BUG, clearing PageReserved too soon is
> likely to lead to freeing an already freed page - noticed with
> a warning if the page is still free, but mayhem if by then reused.

The driver is very strict about mapping memory on load and
unmapping memory on unload. I'm pretty confident that this is
working correctly because the driver can be repeatedly loaded
and unloaded without errors. During normal operation, it does 
not fiddle with mmapping.

> Which is the driver involved?

Its a driver I wrote for a piece of hardware designed by the 
company I work for.

> Or, might the user space application be (arguably) at fault,
> mmap'ing an indeterminate uncontrolled area of /dev/mem, some
> pages of which may or may not be PageReserved at any instant?

Nope, very, very unlikely. My driver is not doing anything 
complicated with memory mapping and I know the user space code 
is reading the right memory location because the read data is 
valid.

Thanks Hugh. Your input was very helpful.

Erik
-- 
-------------------------------------------------------
[N] Erik de Castro Lopo, Senior Computer Engineer
[E] erik.de.castro.lopo@sensorynetworks.com
[W] http://www.sensorynetworks.com
[T] +61 2 83022726
[F] +61 2 94750316
[A] L6/140 William St, East Sydney NSW 2011, Australia
-------------------------------------------------------
A good debugger is no substitute for a good test suite.