Learning ext2 fs

Learning ext2 fs

[Paulo da Silva]

I am reading the ext2 fs code. One of my purposes
is to save the original data of a file to another file
just before it is changed by write/mmap/whatever.
Because of mmap (any other reasons?) I thought
of doing this at "ext2-writepage" or/and
"ext2-writepages".

Is this the right place?
Is there a lower level where I can read/write blocks
of data from/to hd instead of full pages?

How do I tell the really file data from other data?

I traced these functions but I only got
"ext2-writepages" to be called. "ext2-writepage"
was never called using the programs
I wrote to test this. When is "ext2-writepage" called?

Thanks for any help.
Any readings advice is also welcome.

Re: Learning ext2 fs

[Fawad Lateef]

On 10/27/05, Paulo da Silva <psdasilva@esoterica.pt> wrote:
> I am reading the ext2 fs code. One of my purposes
> is to save the original data of a file to another file
> just before it is changed by write/mmap/whatever.
> Because of mmap (any other reasons?) I thought
> of doing this at "ext2-writepage" or/and
> "ext2-writepages".
>
> Is this the right place?

ya, AFAIK ext2-writepage/s and ext2-readpage/s are the functions
actually dealing with the file data and you can also see
generic_file_write/read assigned to the fields in ext2_file_operations
variable because they also deals with the file data ...

> Is there a lower level where I can read/write blocks
> of data from/to hd instead of full pages?
>

I think at the lower-level you can deal with the block-layer but at
this layer AFAIK you won't be able to distinguish between file data
and file system data/structures because you will be dealing with just
the blocks/chunks of data for block devices.

> How do I tell the really file data from other data?
>

the functions readpage/writepage deals with the file data only not
with the file-system data/structures. Here I am considering by saying
other data you mean file-system data/structures


> I traced these functions but I only got
> "ext2-writepages" to be called. "ext2-writepage"
> was never called using the programs
> I wrote to test this. When is "ext2-writepage" called?
>

I think it doesn't matter that writepage or writepages are called
because both almost do the same thing and the writepages version deals
with more data and in your case the data may be big enough to handle
by the writepages.

> Thanks for any help.
> Any readings advice is also welcome.
>

I think you can first get some detailed understanding of VFS layer and
then you can easily understand the code of the file-system like EXT2 !

--
Fawad Lateef

Re: Learning ext2 fs

[Paulo da Silva]

Fawad Lateef wrote:

>On 10/27/05, Paulo da Silva <psdasilva@esoterica.pt> wrote:
>  
>
>>I am reading the ext2 fs code. One of my purposes
>>is to save the original data of a file to another file
>>just before it is changed by write/mmap/whatever.
>>Because of mmap (any other reasons?) I thought
>>of doing this at "ext2-writepage" or/and
>>"ext2-writepages".
>>
>>Is this the right place?
>>    
>>
>
>ya, AFAIK ext2-writepage/s and ext2-readpage/s are the functions
>actually dealing with the file data and you can also see
>generic_file_write/read assigned to the fields in ext2_file_operations
>variable because they also deals with the file data ...
>  
>
Yes, I thought to hook generic_file_write/read for my purpose
until I found mmap does not use them! Then I went down to
writepage(s). 

>  
>
>>Is there a lower level where I can read/write blocks
>>of data from/to hd instead of full pages?
>>
>>    
>>
>
>I think at the lower-level you can deal with the block-layer but at
>this layer AFAIK you won't be able to distinguish between file data
>and file system data/structures because you will be dealing with just
>the blocks/chunks of data for block devices.
>  
>
Yes, but it could be useful for another purpose I have
in mind.
So far, I have found sb_read that allows for blocks
reading using a buffer_head. Does all block readings
use it? What about writes?

>  
>
>>How do I tell the really file data from other data?
>>
>>    
>>
>
>the functions readpage/writepage deals with the file data only not
>with the file-system data/structures. Here I am considering by saying
>other data you mean file-system data/structures
>
>
>  
>
May I conclude that all pages refer to data from/to the file?
I am not sure here, because I noticed that writepages
gets called when I unmounted the device!

>>I traced these functions but I only got
>>"ext2-writepages" to be called. "ext2-writepage"
>>was never called using the programs
>>I wrote to test this. When is "ext2-writepage" called?
>>
>>    
>>
>
>I think it doesn't matter that writepage or writepages are called
>because both almost do the same thing and the writepages version deals
>with more data and in your case the data may be big enough to handle
>by the writepages.
>
>  
>
Not at all. I just used a python program to create a file
with less than 1kb and then I wrote a few bytes mixing
"write" and "mmap". That's why I found strange
writepage not being called. Only writepages.

>>Thanks for any help.
>>Any readings advice is also welcome.
>>
>>    
>>
>
>I think you can first get some detailed understanding of VFS layer and
>then you can easily understand the code of the file-system like EXT2 !
>
>  
>
Well, so far I have only a superficial understanding of VFS,
but I feel relatively comfortable with it.
The problems I found are more related to the lower levels.
How to "say" to write the data to the storage, how to
handle the file data structures. For example, in this
case, I have to reread the original data from the file,
write it to another file, and then commit the pages
to disc as requested to the writepage(s) functions.
But to reread them, I have to save the  "about to
be written pages".
There is no literature that I know of, about this ...
and understanding the code is a hard work.

It would be nice if someone could write information about FSs.
I could find lots of information about VFS level, even a
small fs example with only 1 file. Unfortunately it works
in memory and makes no access to the disc. So,
the lower level remains obscure ...

Thank you.
Paulo

Re: Learning ext2 fs

[Fawad Lateef]

On 10/28/05, Paulo da Silva <psdasilva@esoterica.pt> wrote:
> Fawad Lateef wrote:
>
> >On 10/27/05, Paulo da Silva <psdasilva@esoterica.pt> wrote:
> >
> >
> >>Is there a lower level where I can read/write blocks
> >>of data from/to hd instead of full pages?
> >>
> >>
> >>
> >
> >I think at the lower-level you can deal with the block-layer but at
> >this layer AFAIK you won't be able to distinguish between file data
> >and file system data/structures because you will be dealing with just
> >the blocks/chunks of data for block devices.
> >
> >
> Yes, but it could be useful for another purpose I have
> in mind.
> So far, I have found sb_read that allows for blocks
> reading using a buffer_head. Does all block readings
> use it? What about writes?
>

sb_bread is used to read the blocks containing file-system data not
the files, and for writing file-system data blocks back to storage
mark_buffer_dirty and sync_dirty_buffer functions are used as you can
see in the function ext2_sync_super (I am not FS expert, rather just
worked on it for understanding because i am going to use file-system's
stuff in my current project). sb_bread is actually reading the data
from the storage directly means with-out going through the
buffer_cache layer.


> >
> >
> >>How do I tell the really file data from other data?
> >>
> >>
> >>
> >
> >the functions readpage/writepage deals with the file data only not
> >with the file-system data/structures. Here I am considering by saying
> >other data you mean file-system data/structures
> >
> >
> >
> >
> May I conclude that all pages refer to data from/to the file?
> I am not sure here, because I noticed that writepages
> gets called when I unmounted the device!
>

I am pretty sure that pages here will be refering to the file data,
and as far as unmounting case is concern, synching of the file data
might be done so you are noticing them called by the VFS layer for
writing the un-synched data of file to storage.

> >>I traced these functions but I only got
> >>"ext2-writepages" to be called. "ext2-writepage"
> >>was never called using the programs
> >>I wrote to test this. When is "ext2-writepage" called?
> >>
> >
> >I think it doesn't matter that writepage or writepages are called
> >because both almost do the same thing and the writepages version deals
> >with more data and in your case the data may be big enough to handle
> >by the writepages.
> >
> >
> >
> Not at all. I just used a python program to create a file
> with less than 1kb and then I wrote a few bytes mixing
> "write" and "mmap". That's why I found strange
> writepage not being called. Only writepages.
>

I can't say much about this, but I think writepages are called by VFS
layer might be with nr_pages=1, so that a single page can be written
with small data ! Have you checked the nr_pages field when it is
called ?

>
> It would be nice if someone could write information about FSs.
> I could find lots of information about VFS level, even a
> small fs example with only 1 file. Unfortunately it works
> in memory and makes no access to the disc. So,
> the lower level remains obscure ...
>

AFAIR there are books which contain chapters explaining the
file-systems like EXT2/3 and you can also search for "Simple File
System" or "sfs" on the google for the code of small file-system which
works upto 64MB storage but on the real device not on the memory !


--
Fawad Lateef

Re: Learning ext2 fs

[Jörn Engel]

On Thu, 27 October 2005 19:05:16 +0100, Paulo da Silva wrote:
> 
> I am reading the ext2 fs code. One of my purposes
> is to save the original data of a file to another file
> just before it is changed by write/mmap/whatever.
> Because of mmap (any other reasons?) I thought
> of doing this at "ext2-writepage" or/and
> "ext2-writepages".
> 
> Is this the right place?

Maybe, but the error handling will drive you insane.  My approach was
to copy on open, not on write.  See
http://wohnheim.fh-wedel.de/~joern/cowlink/

Jörn

-- 
Happiness isn't having what you want, it's wanting what you have.
-- unknown