Re: Sync option destroys flash!

Re: Sync option destroys flash!

[linux]

Alan the Hirsute spake unto the masses:
> All non-shite quality flash keys have an on media log structured file
> system and will take 100,000+ writes per sector or so. They decent ones
> also map out bad blocks and have spares. The "wear out the same sector"
> stuff is a myth except on ultra-crap devices.

I would have though so, but I can say from personal experience that
SanDisk brand CF cards respond to losing power during a write by producing
a bad sector.  I had assumed that a sensible implementation would take
advantage of the out-of-place writing by doing a two-phase commit at
write time, so writes would be atomic.

Does anyone know of a CF manufacturer that *does* guarantee atomic writes?
Obviously, if power is lost during a write, it's not clear whether
I'll get the old or the new contents, but I want one or ther other and
not -EIO.

Given that SanDisk first developed the CompactFlash card, you'd think they'd
be a fairly reputable brand...

Re: Sync option destroys flash!

[Lennart Sorensen]

On Sat, May 14, 2005 at 02:43:46AM -0000, linux@horizon.com wrote:
> Alan the Hirsute spake unto the masses:
> > All non-shite quality flash keys have an on media log structured file
> > system and will take 100,000+ writes per sector or so. They decent ones
> > also map out bad blocks and have spares. The "wear out the same sector"
> > stuff is a myth except on ultra-crap devices.
> 
> I would have though so, but I can say from personal experience that
> SanDisk brand CF cards respond to losing power during a write by producing
> a bad sector.  I had assumed that a sensible implementation would take
> advantage of the out-of-place writing by doing a two-phase commit at
> write time, so writes would be atomic.

It can also respond to loosing power during write by getting it's state
so mixed up the whole card is dead (it identifies but all sectors fail
to read).  The binary industrial grade CF cards (no longer in
production) had capacitors to be able to finish writing the block they
were doing to prevent problems.  Supposedly their new firmware now will
have a rollback system so that any partial write is just added back to
the free pool.  I had thought this was always how they did it, but no
apparently that is also something new.

> Does anyone know of a CF manufacturer that *does* guarantee atomic writes?
> Obviously, if power is lost during a write, it's not clear whether
> I'll get the old or the new contents, but I want one or ther other and
> not -EIO.

We were told by SanDisk when we asked about a dead card (it had power
loss during a write) and was told that is normal for the regular
multicell flash cards.  They told us the firmware in the generation of
cards they are currently launching does not have a problem with that
anymore since it essentially journals the writes and can roll back a
partial block write.  I imagine they have patents on that too along with
lots of other flash technology.  Unfortunately their next generation
cards aren't -40 to +85C operation so although everything else was
perfect about them, they are of no use to us.

> Given that SanDisk first developed the CompactFlash card, you'd think they'd
> be a fairly reputable brand...

Well they seem to finally be getting those features working as people
have expected them to work.

Len Sorensen

Re: Sync option destroys flash!

[linux]

> It can also respond to loosing power during write by getting it's state
> so mixed up the whole card is dead (it identifies but all sectors fail
> to read).

Gee, that just happened to me!  Well, actually, thanks to Linux's
*insistence* on reading the partition table, I haven't managed to
get I/O errors on anything bit sectors 0 through 7, but I am quite
sure I wasn't writing those sectors when I pulled the plug:

hda: read_intr: status=0x51 { DriveReady SeekComplete Error }
hda: read_intr: error=0x10 { SectorIdNotFound }, LBAsect=6, sector=6
hda: read_intr: status=0x51 { DriveReady SeekComplete Error }
hda: read_intr: error=0x10 { SectorIdNotFound }, LBAsect=6, sector=6
hda: read_intr: status=0x51 { DriveReady SeekComplete Error }
hda: read_intr: error=0x10 { SectorIdNotFound }, LBAsect=6, sector=6
hda: read_intr: status=0x51 { DriveReady SeekComplete Error }
hda: read_intr: error=0x10 { SectorIdNotFound }, LBAsect=6, sector=6
ide0: reset: success
hda: read_intr: status=0x51 { DriveReady SeekComplete Error }
hda: read_intr: error=0x10 { SectorIdNotFound }, LBAsect=6, sector=6
end_request: I/O error, dev 03:00 (hda), sector 6
 unable to read partition table

> The binary industrial grade CF cards (no longer in
> production) had capacitors to be able to finish writing the block they
> were doing to prevent problems.

Er... are you talking about the SanDisk SDCFI or SDCFJ series?
If they told me, I'd specify them instantly...

> Supposedly their new firmware now will have a rollback system so that any
> partial write is just added back to the free pool.  I had thought this
> was always how they did it, but no apparently that is also something new.

Indeed; I'm in a fix right now because this seemed to thunderingly obvious
to me I didn't check carefully before committing to a CF-based design.

> We were told by SanDisk when we asked about a dead card (it had power
> loss during a write) and was told that is normal for the regular
> multicell flash cards.  They told us the firmware in the generation of
> cards they are currently launching does not have a problem with that
> anymore since it essentially journals the writes and can roll back a
> partial block write.

You wouldn't happen to know what devices those are, would you?
The SDCFH "Ultra II" series, maybe?

I'm talking to them now, so perhaps I'll learn.  As I said, since it
specifically does out-of-place writes, a two-phase commit is
startlingly easy to do.  The basic procedure is:

- For those who don't know, Flash memory technology can only be "erased"
  to all 1 bits large blocks, but can be "programmed" to 0 bits on
  a bit-by-bit basis.  Also, high-density NAND flash often has bad bits,
  so ECC is required.
- All High-density NAND flash has 512+16=528-byte sectors.  The 16 bytes
  are a label area for ECC and information to identify the 512-byte payload.
- If a write is interrupted, it's possible that the affected bit will
  read unreliably.
- Reserve three bits (possibly each implemented with multiple physical
  bits for redundancy in the face of errors).  They mean, respectively:
  - "This sector has started being programmed",
  - "This sector has finished being programmed and its contents are valid", and
  - "This sector contains stale data and should be erased".

To execute a new write,
- Choose an erased sector (or erase an unused sector if your pool of
  pre-erased sectors has been used up).
- (Verify that the sector truly is erased.  If it's not, program the
  "stale data; to be erased" bits and go back to step 1.)
- Program the "write starting bit".  This is important so that it is
  possible to tell that the sector is no longer clear without having
  to check the entire data area.  Which is important when building the
  initial list of erased sectors when booting.
- Program the data, ECC bits, etc.
- (Verify the data was written properly.  Flash memory wears out
  eventually, so bad blocks may develop during operation.)
- Program the "finished programming bit".
- Program the stale-data bits of the previous version of the sector.

When booting, read all the label areas and build the initial
logical/physical sector map.

If you find one for which the "started programming" bit is set but the
"finished programming" one is not, read it and verify the checksums.
If all looks well, re-program the sector (to make sure there aren't
any half-programmed bits) and program the "finished programming" bit.
(This is required in case the finished programming bit was half-programmed
when power was lost; if you don't do it, it's possible that *this* time
you felt sure the sector wasn't finished but the next time the card is
booted, the bit *will* read as programmed, resulting in a confused user.)

Also find the pervious version of the same sector and program its
stale bits.  You need at least a 3-state sequence number to do this,
but that's not a requirement created by atomic writing.

If, on the other hand, reading the payload produces a CRC error, program
the "stale & to be erased" bit.

> I imagine they have patents on that too along with
> lots of other flash technology.  Unfortunately their next generation
> cards aren't -40 to +85C operation so although everything else was
> perfect about them, they are of no use to us.

Well, I can make do.  If you *are* talking about SDCFI or SDCFJ, they're
still for sale at
https://www.californiapc.com/products/sdflash_industrial.php3
at least...

Anyway, thanks for the information!

Re: Sync option destroys flash!

[Richard B. Johnson]

On Tue, 17 May 2005 linux@horizon.com wrote:

>> It can also respond to loosing power during write by getting it's state
>> so mixed up the whole card is dead (it identifies but all sectors fail
>> to read).
>
> Gee, that just happened to me!  Well, actually, thanks to Linux's
> *insistence* on reading the partition table, I haven't managed to
> get I/O errors on anything bit sectors 0 through 7, but I am quite
> sure I wasn't writing those sectors when I pulled the plug:
>
> hda: read_intr: status=0x51 { DriveReady SeekComplete Error }
> hda: read_intr: error=0x10 { SectorIdNotFound }, LBAsect=6, sector=6
> hda: read_intr: status=0x51 { DriveReady SeekComplete Error }
> hda: read_intr: error=0x10 { SectorIdNotFound }, LBAsect=6, sector=6
> hda: read_intr: status=0x51 { DriveReady SeekComplete Error }
> hda: read_intr: error=0x10 { SectorIdNotFound }, LBAsect=6, sector=6
> hda: read_intr: status=0x51 { DriveReady SeekComplete Error }
> hda: read_intr: error=0x10 { SectorIdNotFound }, LBAsect=6, sector=6
> ide0: reset: success
> hda: read_intr: status=0x51 { DriveReady SeekComplete Error }
> hda: read_intr: error=0x10 { SectorIdNotFound }, LBAsect=6, sector=6
> end_request: I/O error, dev 03:00 (hda), sector 6
> unable to read partition table
[SNIPPED...]

You can "fix" this by writing all sectors. Although the data is lost,
the flash-RAM isn't. This can (read will) happen if you pull the
flash-RAM out of its socket with the power ON.

Cheers,
Dick Johnson
Penguin : Linux version 2.6.11.9 on an i686 machine (5537.79 BogoMips).
  Notice : All mail here is now cached for review by Dictator Bush.
                  98.36% of all statistics are fiction.

Re: Sync option destroys flash!

[Lennart Sorensen]

On Tue, May 17, 2005 at 08:31:17PM -0000, linux@horizon.com wrote:
> Gee, that just happened to me!  Well, actually, thanks to Linux's
> *insistence* on reading the partition table, I haven't managed to
> get I/O errors on anything bit sectors 0 through 7, but I am quite
> sure I wasn't writing those sectors when I pulled the plug:
> 
> hda: read_intr: status=0x51 { DriveReady SeekComplete Error }
> hda: read_intr: error=0x10 { SectorIdNotFound }, LBAsect=6, sector=6
> hda: read_intr: status=0x51 { DriveReady SeekComplete Error }
> hda: read_intr: error=0x10 { SectorIdNotFound }, LBAsect=6, sector=6
> hda: read_intr: status=0x51 { DriveReady SeekComplete Error }
> hda: read_intr: error=0x10 { SectorIdNotFound }, LBAsect=6, sector=6
> hda: read_intr: status=0x51 { DriveReady SeekComplete Error }
> hda: read_intr: error=0x10 { SectorIdNotFound }, LBAsect=6, sector=6
> ide0: reset: success
> hda: read_intr: status=0x51 { DriveReady SeekComplete Error }
> hda: read_intr: error=0x10 { SectorIdNotFound }, LBAsect=6, sector=6
> end_request: I/O error, dev 03:00 (hda), sector 6
>  unable to read partition table

Yeah that is exactly how it responds after a powerloss during write when
you have no protection against that.

> > The binary industrial grade CF cards (no longer in
> > production) had capacitors to be able to finish writing the block they
> > were doing to prevent problems.
> 
> Er... are you talking about the SanDisk SDCFI or SDCFJ series?
> If they told me, I'd specify them instantly...

The model with the capacitors was the SDCFBI-size-201-80, but they are no
longer available for purchase (we were going to use them and are now
trying a few other brands).  The -80 Meant industrial grade which was
the flash with capacitors to keep power failures from killing a write
in progress.  None of our tests have killed a -80 yet.  The -00 we have
killed by have power off during writes.  They were also rated at 3 times
the write cycles of their regular grade.

We were told the new cards that they offer to OEMs have new firmware
that does some kind of journaling or timestamping or something to deal
with the problem instead.  I haven't tried the new cards and given they
don't offer the needed temperature range we need, I won't be trying them
either.

> > Supposedly their new firmware now will have a rollback system so that any
> > partial write is just added back to the free pool.  I had thought this
> > was always how they did it, but no apparently that is also something new.
> 
> Indeed; I'm in a fix right now because this seemed to thunderingly obvious
> to me I didn't check carefully before committing to a CF-based design.
> 
> > We were told by SanDisk when we asked about a dead card (it had power
> > loss during a write) and was told that is normal for the regular
> > multicell flash cards.  They told us the firmware in the generation of
> > cards they are currently launching does not have a problem with that
> > anymore since it essentially journals the writes and can roll back a
> > partial block write.
> 
> You wouldn't happen to know what devices those are, would you?
> The SDCFH "Ultra II" series, maybe?

Anything sold retail is anyone's guess (according to their rep) while
OEM cards you get what it says on the card.  The retail cards don't
carry the same model numbers either.  A retail sandisk card might not
even contain sandisk memory (only the controller is sure to be sandisk).

> I'm talking to them now, so perhaps I'll learn.  As I said, since it
> specifically does out-of-place writes, a two-phase commit is
> startlingly easy to do.  The basic procedure is:
> 
> - For those who don't know, Flash memory technology can only be "erased"
>   to all 1 bits large blocks, but can be "programmed" to 0 bits on
>   a bit-by-bit basis.  Also, high-density NAND flash often has bad bits,
>   so ECC is required.
> - All High-density NAND flash has 512+16=528-byte sectors.  The 16 bytes
>   are a label area for ECC and information to identify the 512-byte payload.
> - If a write is interrupted, it's possible that the affected bit will
>   read unreliably.
> - Reserve three bits (possibly each implemented with multiple physical
>   bits for redundancy in the face of errors).  They mean, respectively:
>   - "This sector has started being programmed",
>   - "This sector has finished being programmed and its contents are valid", and
>   - "This sector contains stale data and should be erased".
> 
> To execute a new write,
> - Choose an erased sector (or erase an unused sector if your pool of
>   pre-erased sectors has been used up).
> - (Verify that the sector truly is erased.  If it's not, program the
>   "stale data; to be erased" bits and go back to step 1.)
> - Program the "write starting bit".  This is important so that it is
>   possible to tell that the sector is no longer clear without having
>   to check the entire data area.  Which is important when building the
>   initial list of erased sectors when booting.
> - Program the data, ECC bits, etc.
> - (Verify the data was written properly.  Flash memory wears out
>   eventually, so bad blocks may develop during operation.)
> - Program the "finished programming bit".
> - Program the stale-data bits of the previous version of the sector.
> 
> When booting, read all the label areas and build the initial
> logical/physical sector map.
> 
> If you find one for which the "started programming" bit is set but the
> "finished programming" one is not, read it and verify the checksums.
> If all looks well, re-program the sector (to make sure there aren't
> any half-programmed bits) and program the "finished programming" bit.
> (This is required in case the finished programming bit was half-programmed
> when power was lost; if you don't do it, it's possible that *this* time
> you felt sure the sector wasn't finished but the next time the card is
> booted, the bit *will* read as programmed, resulting in a confused user.)
> 
> Also find the pervious version of the same sector and program its
> stale bits.  You need at least a 3-state sequence number to do this,
> but that's not a requirement created by atomic writing.
> 
> If, on the other hand, reading the payload produces a CRC error, program
> the "stale & to be erased" bit.

Well I suspect that is along the lines of what the sandisk 201 series'
replacement is doing in it's firmware.

The new ones must be either SDCFH or SDCFJ but I can't find anything
that says what the difference is between the two lines.  We were using
the SDCFBI-*-80 cards.

> Well, I can make do.  If you *are* talking about SDCFI or SDCFJ, they're
> still for sale at
> https://www.californiapc.com/products/sdflash_industrial.php3
> at least...

Well we were told last buy was about a week or two ago on the
SDCFBI-*-201-80 cards.  We are now playing with SLCF*JI cards from
SimpleTech and hopefully those will work out for us.  It sure is hard to
do indurstrial temperature when most people don't care.  Most companies
are happy to avoid the trouble since normal temperature suits 99% of the
market, so why bother with the trouble for the last 1% even if they are
willing to pay double. :)

Len Sorensen

Re: Sync option destroys flash! Now I'm confused...

[Michael H. Warfield]

[-- Attachment #1: Type: text/plain, Size: 5183 bytes --]

	All right...  Now I'm really confused.

	There are, obviously, some individuals on this list who are a LOT more
knowledgeable about the internal workings of flash, so I'm hoping for a
clear(er) understanding of just WHAT is going on here.

On Tue, 2005-05-17 at 16:43 -0400, Richard B. Johnson wrote:
> On Tue, 17 May 2005 linux@horizon.com wrote:

> >> It can also respond to loosing power during write by getting it's state
> >> so mixed up the whole card is dead (it identifies but all sectors fail
> >> to read).

> > Gee, that just happened to me!  Well, actually, thanks to Linux's
> > *insistence* on reading the partition table, I haven't managed to
> > get I/O errors on anything bit sectors 0 through 7, but I am quite
> > sure I wasn't writing those sectors when I pulled the plug:

> > hda: read_intr: status=0x51 { DriveReady SeekComplete Error }
> > hda: read_intr: error=0x10 { SectorIdNotFound }, LBAsect=6, sector=6
> > hda: read_intr: status=0x51 { DriveReady SeekComplete Error }
> > hda: read_intr: error=0x10 { SectorIdNotFound }, LBAsect=6, sector=6
> > hda: read_intr: status=0x51 { DriveReady SeekComplete Error }
> > hda: read_intr: error=0x10 { SectorIdNotFound }, LBAsect=6, sector=6
> > hda: read_intr: status=0x51 { DriveReady SeekComplete Error }
> > hda: read_intr: error=0x10 { SectorIdNotFound }, LBAsect=6, sector=6
> > ide0: reset: success
> > hda: read_intr: status=0x51 { DriveReady SeekComplete Error }
> > hda: read_intr: error=0x10 { SectorIdNotFound }, LBAsect=6, sector=6
> > end_request: I/O error, dev 03:00 (hda), sector 6
> > unable to read partition table
> [SNIPPED...]

> You can "fix" this by writing all sectors. Although the data is lost,
> the flash-RAM isn't. This can (read will) happen if you pull the
> flash-RAM out of its socket with the power ON.

	I'm the original poster and someone in another message remarked about
not having enough details on the damage to the card...  So I just did
some spot checking on the card for some details.

	Block checking with dd bs=512 if=/dev/sda gave me some indicators...

Blocks 0-7 DOA, hard read errors, 0+0 records in.

Blocks 8-31 would read 8 blocks at a time and then give me an error, but
the next 8 blocks would read fine.  So 24 consecutive blocks SEEMED to
read but strangely.

Blocks 32-39 DOA

Blocks 40-71 would read 8 blocks at a time.

Roughly 1/3 of the blocks seem to be dead in multiples of 8 blocks on 8
block boundries early on.  No real pattern to which ones were dead and
which ones would read 8 and then error.

Once past block 512, huge blocks would be readable but eventually give
me an error.

Dead fields (0 records read) were always multiples of 8 512 byte blocks,
4KBytes falling on an 4K boundry.

Reading with dd bs=4096 gave similar results for 4K blocks with skip
count less than 64.  Skip count 64 and greater gave me large swaths that
were readable.  No time did I see a partial record read (indicating a
failure off a 4K boundry).

	Basically, that re-enforced my option that it was block wear-out from
uneven wear leveling when copying that 700 Meg file and beating the
bejesus out of the FAT tables.  Front part of the flash was heavily
damaged with sporatic damage deeper in the flash.

	Now, I saw this message...  Well...  I didn't remove the key when it
was being written to but, what the hell...  The key is dead, I've got
nothing to loose, and it might yield some more information as to the
nature of the failure.  So I copied zeros to the entire key with "dd
if=/dev/zero of=/dev/sda bs=16M".  I'll be a son of a bitch but that key
recovered.  I've partitioned it and read the whole damn thing back end
to end and it's perfect.

	Ok...  So, WTF?  It wasn't (AFAICT) due to loss of power or pulling it
while writing.  What was this failure and why did overwriting it fix it?
Did the stick just flaw out all the burned out blocks or did it really
recover the ECC errors?  I'm really baffled now.

	BTW...  I've killed the "sync" option in hal (you just have to create
an XML policy file in the right location to specify that option as false
in all cases) and have been beating the crap out of several other keys
without a single failure.  I'm going to try this key again...

	Thank you very VERY much for this hint to recover the damaged key.
That's a trick I've used for damaged IDE & SCSI hard drives (recover
head drift and soft errors) and I never thought to try it with a flash
key.  I'll be damned if I understand just what has happened at this
point but I really appreciate that trick.

> Cheers,
> Dick Johnson
> Penguin : Linux version 2.6.11.9 on an i686 machine (5537.79 BogoMips).
>   Notice : All mail here is now cached for review by Dictator Bush.
>                   98.36% of all statistics are fiction.

	Regards,
	Mike
-- 
 Michael H. Warfield    |  (770) 985-6132   |  mhw@WittsEnd.com  
  /\/\|=mhw=|\/\/       |  (678) 463-0932   |  http://www.wittsend.com/mhw/
  NIC whois:  MHW9      |  An optimist believes we live in the best of all
 PGP Key: 0xDF1DD471    |  possible worlds.  A pessimist is sure of it!

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 307 bytes --]

Re: Sync option destroys flash! Now I'm confused...

[Richard B. Johnson]

On Wed, 18 May 2005, Michael H. Warfield wrote:

> 	All right...  Now I'm really confused.
>
> 	There are, obviously, some individuals on this list who are a LOT more
> knowledgeable about the internal workings of flash, so I'm hoping for a
> clear(er) understanding of just WHAT is going on here.
>
> On Tue, 2005-05-17 at 16:43 -0400, Richard B. Johnson wrote:
>> On Tue, 17 May 2005 linux@horizon.com wrote:
>
>>>> It can also respond to loosing power during write by getting it's state
>>>> so mixed up the whole card is dead (it identifies but all sectors fail
>>>> to read).
>
>>> Gee, that just happened to me!  Well, actually, thanks to Linux's
>>> *insistence* on reading the partition table, I haven't managed to
>>> get I/O errors on anything bit sectors 0 through 7, but I am quite
>>> sure I wasn't writing those sectors when I pulled the plug:
>
>>> hda: read_intr: status=0x51 { DriveReady SeekComplete Error }
>>> hda: read_intr: error=0x10 { SectorIdNotFound }, LBAsect=6, sector=6
>>> hda: read_intr: status=0x51 { DriveReady SeekComplete Error }
>>> hda: read_intr: error=0x10 { SectorIdNotFound }, LBAsect=6, sector=6
>>> hda: read_intr: status=0x51 { DriveReady SeekComplete Error }
>>> hda: read_intr: error=0x10 { SectorIdNotFound }, LBAsect=6, sector=6
>>> hda: read_intr: status=0x51 { DriveReady SeekComplete Error }
>>> hda: read_intr: error=0x10 { SectorIdNotFound }, LBAsect=6, sector=6
>>> ide0: reset: success
>>> hda: read_intr: status=0x51 { DriveReady SeekComplete Error }
>>> hda: read_intr: error=0x10 { SectorIdNotFound }, LBAsect=6, sector=6
>>> end_request: I/O error, dev 03:00 (hda), sector 6
>>> unable to read partition table
>> [SNIPPED...]
>
>> You can "fix" this by writing all sectors. Although the data is lost,
>> the flash-RAM isn't. This can (read will) happen if you pull the
>> flash-RAM out of its socket with the power ON.
>
> 	I'm the original poster and someone in another message remarked about
> not having enough details on the damage to the card...  So I just did
> some spot checking on the card for some details.
>
> 	Block checking with dd bs=512 if=/dev/sda gave me some indicators...
>
> Blocks 0-7 DOA, hard read errors, 0+0 records in.
>
> Blocks 8-31 would read 8 blocks at a time and then give me an error, but
> the next 8 blocks would read fine.  So 24 consecutive blocks SEEMED to
> read but strangely.
>
> Blocks 32-39 DOA
>
> Blocks 40-71 would read 8 blocks at a time.
>
> Roughly 1/3 of the blocks seem to be dead in multiples of 8 blocks on 8
> block boundries early on.  No real pattern to which ones were dead and
> which ones would read 8 and then error.
>
> Once past block 512, huge blocks would be readable but eventually give
> me an error.
>
> Dead fields (0 records read) were always multiples of 8 512 byte blocks,
> 4KBytes falling on an 4K boundry.
>
> Reading with dd bs=4096 gave similar results for 4K blocks with skip
> count less than 64.  Skip count 64 and greater gave me large swaths that
> were readable.  No time did I see a partial record read (indicating a
> failure off a 4K boundry).
>
> 	Basically, that re-enforced my option that it was block wear-out from
> uneven wear leveling when copying that 700 Meg file and beating the
> bejesus out of the FAT tables.  Front part of the flash was heavily
> damaged with sporatic damage deeper in the flash.
>
> 	Now, I saw this message...  Well...  I didn't remove the key when it
> was being written to but, what the hell...  The key is dead, I've got
> nothing to loose, and it might yield some more information as to the
> nature of the failure.  So I copied zeros to the entire key with "dd
> if=/dev/zero of=/dev/sda bs=16M".  I'll be a son of a bitch but that key
> recovered.  I've partitioned it and read the whole damn thing back end
> to end and it's perfect.
>
> 	Ok...  So, WTF?  It wasn't (AFAICT) due to loss of power or pulling it
> while writing.  What was this failure and why did overwriting it fix it?
> Did the stick just flaw out all the burned out blocks or did it really
> recover the ECC errors?  I'm really baffled now.
>
> 	BTW...  I've killed the "sync" option in hal (you just have to create
> an XML policy file in the right location to specify that option as false
> in all cases) and have been beating the crap out of several other keys
> without a single failure.  I'm going to try this key again...
>
> 	Thank you very VERY much for this hint to recover the damaged key.
> That's a trick I've used for damaged IDE & SCSI hard drives (recover
> head drift and soft errors) and I never thought to try it with a flash
> key.  I'll be damned if I understand just what has happened at this
> point but I really appreciate that trick.
>
>> Cheers,
>> Dick Johnson
>> Penguin : Linux version 2.6.11.9 on an i686 machine (5537.79 BogoMips).
>>   Notice : All mail here is now cached for review by Dictator Bush.
>>                   98.36% of all statistics are fiction.
>
> 	Regards,
> 	Mike

The problem is that it's a RAM disk that uses flash-RAM, plus a
little bit of SRAM for one page of I/O. Some devices use two
pages of SRAM to ping-pong for speed. The size of the 'pages'
might vary with the manufacturer. The only thing known is that
these pages will be a multiple of the de facto 512 byte 'sector'
size of a physical disk.

You fixed the device by writing to the whole device without
an intervening read. Writes work like this. The data written goes
into a SRAM page that is shadowed. When that page is filled,
before another page is switched, the flash-RAM page that was
shadowed is now written to the real flash-RAM. This is necessary
because flash-RAM can only be written by resetting bits, not
setting them. So first the page is erased which takes a lot of
time and sets all the bits high. The writing process sends an
unlock-sequence to the flash-RAM controller, followed by the
offset into the page, followed by the data byte. This also
takes time so flash-RAM without the SRAM random-access shadow
page is somewhat limited in value. This process continues util
you have written the whole device.

Normal random access works like this, the page to be accessed
is calculated by dividing the offset you want, by the real
page-size. The offset into that page is the remainder from
the division. Any unflushed data in the SRAM gets written to
the device as previously shown. The newly calculated page
is read into the SRAM. You do I/O from the SRAM. The chip
remembers if any writes occur. If a write occurs, the contents
of the SRAM is flushed to the calculated page any time the
page is about to be changed. There is also a "stale" alogrithm
that writes out a page that hasn't been accessed for some
time.

Now, if you interrupt this sequence by killing the power at
some 'bad' time, data will not be correct. In fact, you could
have an erased page with all bits set.

Now, when you have a file-system that has inodes scattered
all through it, any inodes that are on an erased page will
cause the next access to be at some offset (sector) that
doesn't exist. Since there are no 'sector IDs' as shown
in the errors reported, they must be created by the hard-disk
emulation. So, looking at errors with Sector ID=6, etc.,
simply means that the emulator was 'confused'. It was probably
the Nth wrap of some hardware variable.

Anyway, I've used the SanDisk and PNY flash-RAM that emulates
a 'type 3' IDE drive since they first became available. I haven't
killed any yet. But.... I've destroyed many file-systems by
unplugging them while accesses were occurring.

Cheers,
Dick Johnson
Penguin : Linux version 2.6.11.9 on an i686 machine (5537.79 BogoMips).
  Notice : All mail here is now cached for review by Dictator Bush.
                  98.36% of all statistics are fiction.