Re: Improved file handling mechanisms for 64-bit architectures

[mru@kth.se (Måns Rullgård)]

"Vijolicni.oblak" <un.info@gmx.net> writes:

> I would like to make a suggestion how to best use the new AMD64 architecture
> in the new kernel version for 64-bit architectures. You are making new sets
> of functions and optimizing new compilers anyway so I think that now is also
> the time to introduce new file handling mechanisms that can be fully
> implemented only in 64-bit environment.
>
> Currently, the AMD64 architecture defines a mechanism for translating 48-bit
> virtual addresses to 52-bit physical addresses.
>
> I will make a suggestion on how to improve file handling performance:
>
> Currently if you want to work with files, you have to:
>
> 1) assign file a handle; 2) read wanted portions of file into memory 3) if
> there is not enough physical memory, save the contents of the file to disk
>
> With AMD64 you are able to make 48-bit addresses, which amount to 256000
> gigabytes of virtual memory. When working with large (eg. 10GB) video or
> database files, Linux kernel could map the whole file into the virtual
> memory using processor's Page Translation Mechanism.
>
> Those 10GB would then be mapped to a certain memory range. If a portion of
> file that is currently requested is in physical RAM the processor would
> handle it without OS intervention; if not, then a page fault (#PF, 14)
> exception would occur and read a missing page (a missing portion of file).
>
> The application would see the file as a 10GB large array (or a string), or
> perhaps could map its own data structures into this memory space.
>
> When writing to file, i.e. modifying the data in the array, a #PF would
> occur and mark this page as dirty and write modified data to the disk (or
> schedule writing to a later time)

Did you have something different from mmap in mind?  What you are
saying look very much like the existing mmap functionality.

> There is also one addition benefit to this: when inserting data into the
> middle of the 10GB file - a new video frame or enlarging a table in SQL only
> a remapping of virtual memory and the cooperation of file system is needed.
>
> Let's say that AAAAAAAAAAAAAAAAAAAAAAAA is the original file. You want to
> insert B somewhere in the middle of the file. AAAAAAAAABAAAAAAAAAAAAAAA is
> done by calling a function increasefile(startoffset,length) that moves a few
> hundred bytes in Page Table or in Page directory table and by adding an
> additional segment to the file chain (eg. FAT chain). The length must just
> be a multiplier of 4KB.

This is non-trivial.  It would require support from all the various
filesystems.  Surely there's some filesystem using an on-disk layout
making such things difficult or impossible.

-- 
Måns Rullgård
mru@kth.se