Using remap_pfn_range causes system hang on app close in 2.6.15 & up

["Sam Abu-Nassar" <djanssen1@hotmail.com>]

Hello,

I currently have a module for a PCI device that maps system memory to user 
space with remap_pfn_range().  The system memory is acquired through 
dma_alloc_coherent (i.e. __get_free_pages).  The driver works fine on all 
2.4 kernels and all 2.6 kernels up to 2.6.14.  The failure seems to only 
occurs on 2.6.15 & up.

The issue occurs when the application exists and/or the file descriptor is 
closed if the application maps the RAM buffer.  Sometimes kernel log 
messages appear regarding the page _mapcount being invalid (-1).  So, it 
must be when the VMM attempts to release page mappings.  Generally, the 
system hangs and sometimes, after a moment or two, a kernel panic message 
arises (see below).

I am able to avoid the issue if my module does not call remap_pfn_range().  
I realize that there were some significant changes in this function in 
kernel 2.6.15, but I can't see what I am doing wrong.  I've tried various 
tests, all with the same result, including:

- Removing the VM_RESERVED flag (remap_pfn_range sets it anyway)
- Tried setting & not setting PG_reserved on the allocated pages
- Tried setting & not setting PG_locked on the allocated pages
- Tried holding the mmap semaphore during the call to remap_pfn_range

I've checked the parameters to remap_pfn_range and they seem to be ok.  
There are no alignment issues and the physical address is page shifted.  The 
VMA is passed pretty much directly from mmap().

I'm pretty sure the changes in 2.6.15 with remap_pfn_range are causing this. 
  I've read through the logs regarding the changes and about the new 
vm_insert_page() function, but it is my understanding that remap_pfn_range 
should still work ok for my case.  All my driver needs to do here is to 
simply map contiguous system pages to user space.

Has anyone else run into issues with using remap_pfn_range and kernels 
2.5.15 & up?  Are there any new special requirements for using this in a 
module.  I've looked at the /dev/mem mmap code and I do not see anything 
obviously different from my code.  I appreciate any comment/suggestions you 
may have

Thanks in advance,

-Sam Abu-Nassar

Here is some more info and the code snip:

KERNEL LOG
-------------------
Line 555 in rmap.c is where the kernel panic is:
http://www.kernel.org/hg/linux-2.6/?f=106a2be54bb7;file=mm/rmap.c

<0>Eeek! page_mapcount(page) went negative! (-1)
<0>  page->flags = 80000404
<0>  page->count = 1
<0>  page->mapping = 00000000
<0>------------[ cut here ]------------
<0>kernel BUG at mm/rmap.c:560!
<0>invalid opcode: 0000 [#1]
<1>last sysfs file: /devices/system/cpu/cpu0/cpufreq/scaling_setspeed
<4>Modules linked in: Pci9656_dbg(U) autofs4 hidp rfcomm l2cap
bluetooth sunrpc
dm_mirror dm_mod video button battery ac ipv6 lp parport_pc parport
floppy nvram
ehci_hcd uhci_hcd sky2 snd_via82xx gameport snd_ac97_codec
snd_ac97_bus snd_seq
_dummy snd_seq_oss snd_seq_midi_event snd_seq snd_pcm_oss snd_mixer_oss
snd_pcm
snd_timer snd_page_alloc snd_mpu401_uart snd_rawmidi snd_seq_device
i2c_viapro s
nd i2c_core soundcore ext3 jbd
<0>CPU:    0
<4>EIP:    0060:[<c014852d>]    Not tainted VLI
<4>EFLAGS: 00010286   (2.6.15-1.2054_FC5 #1)
<0>EIP is at page_remove_rmap+0x63/0x7b
<0>eax: ffffffff   ebx: c15e7a00   ecx: ef1fcee8   edx: c02fbeaf
<0>esi: b7f97000   edi: ee564e5c   ebp: 00000020   esp: ef1fcef4
<0>ds: 007b   es: 007b   ss: 0068
<0>Process Test (pid: 3493, threadinfo=ef1fc000 task=f7ca5000)
<0>Stack: <0>c15e7a00 c0142db5 00000000 f6b12cd4 ef1fcf60 00340efa
00000000 0000
0001
<0>       b7fa7000 ef49bb7c f6af1500 c03e022c 00000000 ffffffff
f6af1554 ef49bb7
c
<0>       b7fa7000 00000000 ef1fcf60 f6af690c f6af1500 ef1fcfa0
c0145644 fffffff
f
<0>Call Trace:
<0> [<c0142db5>] unmap_vmas+0x285/0x48d     [<c0145644>]
exit_mmap+0x50/0xc0
<0> [<c01184bd>] mmput+0x1c/0x8f     [<c011d012>] do_exit+0x1a5/0x6c8
<0> [<c011d5b9>] sys_exit_group+0x0/0xd     [<c0102bc1>]
syscall_call+0x7/0xb


CODE SNIP
---------------
Buffer Allocation Code
======================
    // Allocate system buffer
    pMemObject->pKernelVa =
        dma_alloc_coherent(
            pdx->pPciDevice->dev,
            pMemObject->Size,     // 64k
            &BusAddress,
            GFP_KERNEL | __GFP_NOWARN
            );

    // Store the bus address
    pMemObject->BusPhysical = (U64)BusAddress;

    // Tag all pages as reserved
    for (....)
        SetPageReserved( virt_to_page(virt_addr) );

    // Get CPU physical address of buffer
    pMemObject->CpuPhysical = virt_to_phys( pMemObject->pKernelVa );


Buffer Mapping Code in mmap()
=============================

    // Get CPU physical address of buffer
    AddressToMap = pMemObject->CpuPhysical;

    // Set the region as reserved
    vma->vm_flags |= VM_RESERVED;

    // Map system memory
    rc =
        remap_pfn_range(
            vma,
            vma->vm_start,
            AddressToMap >> PAGE_SHIFT,
            vma->vm_end - vma->vm_start,
            vma->vm_page_prot
            );