Accessing odd last partition sector (was: [Linux-NTFS-Dev] mkntfs dirty volume marking)

Accessing odd last partition sector (was: [Linux-NTFS-Dev] mkntfs dirty volume marking)

[Szakacsits Szabolcs]

On Fri, 9 Apr 2004, fledely wrote:

> TODO.ntfsprogs conatins the following TODO item under mkntfs:
>  - We don't know what the real last sector is, thus we mark the volume
> dirty and the subsequent chkdsk (which will happen on reboot into
> Windows automatically) recreates the backup boot sector if the Linux
> kernel lied to us about the number of sectors.

ntfsresize, ntfsclone and others have the same problem due to this kernel
limitation.

>  Now that we know about the extended/legacy BIOS geometry, and
> assuming kernel version 2.6.5+ Is it possible to finally get fixed, or
> totally unrelated?

I'm afraid unrelated. There are two main issues two consider if you want
to fix this:

 1) You must know the exact, _real_ partition size.

 2) You must be able to access it.

Firstly, I don't think EDD tells anything about the partitions but maybe
I'm wrong, I didn't have time to check it out.

Secondly the kernel doesn't always allow access to the last sector via the
partition (longstanding kernel bug). We could access it via the full
device (e.g. /dev/hda, etc) but that's quite messy, error-prone and this
should be fixed in the _kernel_.

Other people are also disturbed by this for a long time,

 - they can't make a full partition backup (dd)
	
 - just like NTFS, the new partitioning format, GPT, also stores
   backup data in the last sectors. 

There were attempts to fix this in 2.6 but I don't know if it was
completed. Maybe somebody knows or have the time to investigate it?

	Szaka

Re: Accessing odd last partition sector (was: [Linux-NTFS-Dev] mkntfs dirty volume marking)

[Andries Brouwer]

On Fri, Apr 09, 2004 at 01:38:51PM +0200, Szakacsits Szabolcs wrote:

> > TODO.ntfsprogs conatins the following TODO item under mkntfs:
> >  - We don't know what the real last sector is, thus we mark the volume
> > dirty and the subsequent chkdsk (which will happen on reboot into
> > Windows automatically) recreates the backup boot sector if the Linux
> > kernel lied to us about the number of sectors.

The ioctl BLKGETSIZE64 will tell you the size (in bytes) of a block device.

Andries

Re: Accessing odd last partition sector (was: [Linux-NTFS-Dev] mkntfs dirty volume marking)

[viro]

On Sat, Apr 10, 2004 at 10:29:37PM +0200, Andries Brouwer wrote:
> On Fri, Apr 09, 2004 at 01:38:51PM +0200, Szakacsits Szabolcs wrote:
> 
> > > TODO.ntfsprogs conatins the following TODO item under mkntfs:
> > >  - We don't know what the real last sector is, thus we mark the volume
> > > dirty and the subsequent chkdsk (which will happen on reboot into
> > > Windows automatically) recreates the backup boot sector if the Linux
> > > kernel lied to us about the number of sectors.
> 
> The ioctl BLKGETSIZE64 will tell you the size (in bytes) of a block device.

So will lseek() to SEEK_END, actually (both 2.4 and 2.6).
And yes, last sector _is_ accessible for dd(1) et.al.

Re: Accessing odd last partition sector (was: [Linux-NTFS-Dev] mkntfs dirty volume marking)

[Szakacsits Szabolcs]

On Sat, 10 Apr 2004 viro@parcelfarce.linux.theplanet.co.uk wrote:
> On Sat, Apr 10, 2004 at 10:29:37PM +0200, Andries Brouwer wrote:
> > On Fri, Apr 09, 2004 at 01:38:51PM +0200, Szakacsits Szabolcs wrote:
> > 
> > > > TODO.ntfsprogs conatins the following TODO item under mkntfs:
> > > >  - We don't know what the real last sector is, thus we mark the volume
> > > > dirty and the subsequent chkdsk (which will happen on reboot into
> > > > Windows automatically) recreates the backup boot sector if the Linux
> > > > kernel lied to us about the number of sectors.
> > 
> > The ioctl BLKGETSIZE64 will tell you the size (in bytes) of a block device.
> 
> So will lseek() to SEEK_END, actually (both 2.4 and 2.6).
> And yes, last sector _is_ accessible for dd(1) et.al.

In 2.6? Not for 2.4 when I tried (it wasn't the latest 2.4 kernel).

	Szaka

Re: Accessing odd last partition sector (was: [Linux-NTFS-Dev] mkntfs dirty volume marking)

[viro]

On Sat, Apr 10, 2004 at 11:04:45PM +0200, Szakacsits Szabolcs wrote:
> 
> On Sat, 10 Apr 2004 viro@parcelfarce.linux.theplanet.co.uk wrote:
> > On Sat, Apr 10, 2004 at 10:29:37PM +0200, Andries Brouwer wrote:
> > > On Fri, Apr 09, 2004 at 01:38:51PM +0200, Szakacsits Szabolcs wrote:
> > > 
> > > > > TODO.ntfsprogs conatins the following TODO item under mkntfs:
> > > > >  - We don't know what the real last sector is, thus we mark the volume
> > > > > dirty and the subsequent chkdsk (which will happen on reboot into
> > > > > Windows automatically) recreates the backup boot sector if the Linux
> > > > > kernel lied to us about the number of sectors.
> > > 
> > > The ioctl BLKGETSIZE64 will tell you the size (in bytes) of a block device.
> > 
> > So will lseek() to SEEK_END, actually (both 2.4 and 2.6).
> > And yes, last sector _is_ accessible for dd(1) et.al.
> 
> In 2.6? Not for 2.4 when I tried (it wasn't the latest 2.4 kernel).

2.4 logics around block size handling is broken; we probably could backport
that series of patches, though.  2.6 simply sets block size to GCD(page size,
device size), so we don't have to worry about all that crap.

Re: Accessing odd last partition sector (was: [Linux-NTFS-Dev] mkntfs dirty volume marking)

[Szakacsits Szabolcs]

On Sat, 10 Apr 2004 viro@parcelfarce.linux.theplanet.co.uk wrote:

> 2.4 logics around block size handling is broken; we probably could backport
> that series of patches, though.  2.6 simply sets block size to GCD(page size,
> device size), so we don't have to worry about all that crap.

That's great! 

Just one question, in the most common cases the block size ends up between
512 and 4096 bytes. Depending on how this block size used, it can have a
significant impact on performance (e.g. 512 vs 4096). Is this true or is
it used to be performance independent?

	Szaka

Re: Accessing odd last partition sector (was: [Linux-NTFS-Dev] mkntfs dirty volume marking)

[viro]

On Sun, Apr 11, 2004 at 12:23:47AM +0200, Szakacsits Szabolcs wrote:
 
> Just one question, in the most common cases the block size ends up between
> 512 and 4096 bytes. Depending on how this block size used, it can have a
> significant impact on performance (e.g. 512 vs 4096). Is this true or is
> it used to be performance independent?

Resulting requests are immediately merged anyway.  Yes, we get more bio
sitting on top of the merged request; however, it's heavily IO-dominated
and I would be surprised if you really saw any noticable overhead in that
situation.

Re: Accessing odd last partition sector (was: [Linux-NTFS-Dev] mkntfs dirty volume marking)

[Szakacsits Szabolcs]

On Sat, 10 Apr 2004 viro@parcelfarce.linux.theplanet.co.uk wrote:
> On Sun, Apr 11, 2004 at 12:23:47AM +0200, Szakacsits Szabolcs wrote:
>  
> > Just one question, in the most common cases the block size ends up between
> > 512 and 4096 bytes. Depending on how this block size used, it can have a
> > significant impact on performance (e.g. 512 vs 4096). Is this true or is
> > it used to be performance independent?
> 
> Resulting requests are immediately merged anyway.  Yes, we get more bio
> sitting on top of the merged request; however, it's heavily IO-dominated
> and I would be surprised if you really saw any noticable overhead in that
> situation.

Thanks, I'll test it in the near future unless somebody does it earlier. 

I have my test stuff but I'm interested of you could suggest specific ones
that might exhibit/trigger the overhead if it exists at all.

	Szaka

Re: Accessing odd last partition sector (was: [Linux-NTFS-Dev] mkntfs dirty volume marking)

[Szakacsits Szabolcs]

On Sat, 10 Apr 2004 viro@parcelfarce.linux.theplanet.co.uk wrote:
> > > > 
> > > > > > TODO.ntfsprogs conatins the following TODO item under mkntfs:
> > > > > >  - We don't know what the real last sector is, thus we mark the volume
> > > > > > dirty and the subsequent chkdsk (which will happen on reboot into
> > > > > > Windows automatically) recreates the backup boot sector if the Linux
> > > > > > kernel lied to us about the number of sectors.
> > > > 
> > > > The ioctl BLKGETSIZE64 will tell you the size (in bytes) of a
> > > > block device.

Unless kernel 2.4.1[567] used (they return the size in sectors) or no
ioctl conflict with the unofficial but used BLKSETLASTSECT (the issue was
also summarized at http://lwn.net/2001/0906/kernel.php3). The last one
could (did?) corrupt NTFS (NTFS keeps metadata there) when one tried to
get the device size by BLKGETSIZE64 ...

Unfortunately not many softwares get BLKGETSIZE64 right, but at least the
latest fdisk (2.12a) and e2fsprogs (1.36-WIP) are ok AFAIS, although they
workaround these issues differently.

> > > So will lseek() to SEEK_END, actually (both 2.4 and 2.6).
> > > And yes, last sector _is_ accessible for dd(1) et.al.

I checked these for 2.4.25 and 2.6.5. In 2.4 the last odd sector is
visible only by BLKGETSIZE64 and BLKGETSIZE, otherwise it can't be
accessed, as we agreed later on.

But 2.6 is ok in all cases, size is seen correctly by BLKGETSIZE64,
BLKGETSIZE and accessible by lseek.

I don't know how intrusive, risky would be the backport but I suspect 
it's not worth. 

Thank you for clarifying the issue,

	Szaka