[HOWTO...] LUFS, readpage and large files

[HOWTO...] LUFS, readpage and large files

[Eduard Bloch]

Hello,

while writting a new LUFS plugin (to emulate large file support on
FAT32;) I stumbled over a problem with it's kernel communication module.
The symptoms were the same as with broken programs that use int or long
int instead of size_t (or even long long) for storing file offsets.
However, here the broken offsets (modulo 4GB) came from kernel so I
traced it down to this method:

static int lu_file_readpage(struct file *f, struct page *p)
{
    int res;
    struct iovec siov[3], riov;
    long long offset;
    unsigned long count;
    struct server_slot *slot;

    TRACE("in\n");

    if((slot = lu_getslot(GET_INFO(f->f_dentry->d_sb))) == NULL)
    	return -ERESTARTSYS;

    get_page(p);

    if((res = lu_getname(f->f_dentry, slot->s_buf, LU_MAXDATA)) < 0){
      WARN("lu_getname failed!\n");
      goto out;
    }

    offset = p->index << PAGE_CACHE_SHIFT;
    count = PAGE_SIZE;

The problem is, page->index indeed contains a short datatype for offset -
...but where to get the correct data? (the long long offset).

I tried to look at how other filesystems manage it but they become too
complex when it comes to details. Unfortunately, most documentation
about VFS and filesystems (found on Internet) simply sucks when it comes
to such details, especially for Large Files.
The autors simply refer to "block lockup methods" or similar things but
nobody gives an example explanation or correct description of how the
way of the data should like (complete - all steps between the request,
page allocation, translation of addresses etc.pp.).

Regards,
Eduard.
-- 
Eine Freude vertreibt hundert Sorgen.

Re: [HOWTO...] LUFS, readpage and large files

[Andrew Morton]

Eduard Bloch <blade@debian.org> wrote:
>
>     offset = p->index << PAGE_CACHE_SHIFT;
>      count = PAGE_SIZE;
> 
>  The problem is, page->index indeed contains a short datatype for offset -
>  ...but where to get the correct data? (the long long offset).

p->index is a 32-bit type.  So (p->index << PAGE_CACHE_SHIFT) is also
32-bit, and stuff overflows.  Do:

	offset = p->index;
	offset <<= PAGE_CACHE_SHIFT;

Re: [HOWTO...] LUFS, readpage and large files

[Miklos Szeredi]

> while writting a new LUFS plugin (to emulate large file support on
> FAT32;) I stumbled over a problem with it's kernel communication module.
> The symptoms were the same as with broken programs that use int or long
> int instead of size_t (or even long long) for storing file offsets.
> However, here the broken offsets (modulo 4GB) came from kernel so I
> traced it down to this method:

[...]

>     offset = p->index << PAGE_CACHE_SHIFT;

This should be:

     offset = (unsigned long long) p->index << PAGE_CACHE_SHIFT;

BTW, you can use FUSE + LUFIS for running LUFS filesystems with the
FUSE kernel module. Or even better: you can write your filesystem
natively with FUSE ;)

You can download LUFIS from http://sourceforge.net/projects/avf, FUSE
is already in debian testing.

Cheers,
Miklos