Re: IDE Hotswap

Re: IDE Hotswap

[Bartlomiej Zolnierkiewicz]

On Tuesday 28 September 2004 20:49, Suresh Grandhi wrote:
> Hi,
> My hardware has IDE hotswap support. I like to implement the hotswap driver
> over 2.6.8 kernel.
> Right now I am able to register and unregister the IDE devices successfully
> if they are not part of any raid or LVM.
> If I/O operation is going on the disk, then IDE unregister fails generating
> lot of I/O error messages.
> In this case, I want to make a work-around. 
> Instead of unregistering the IDE device, is there a way to re-register the
> IDE device with the already existing ports and irq when the disk is
> inserted?

No and such workaround won't work anyway because
re-register operation is nothing else but unregister+register.

hotswap is currently unsupported, some TODO items:
- convert IDE driver to use dynamic objects instead of static ones
- add needed refcounting and locking
- convert IDE driver to new driver-model
- add full sysfs support
- add hotswap

Any help/support is appreciated.

Regards,
Bartlomiej

Re: IDE Hotswap

[Alan Cox]

On Maw, 2004-09-28 at 22:38, Bartlomiej Zolnierkiewicz wrote:
> No and such workaround won't work anyway because
> re-register operation is nothing else but unregister+register.

If you grab the 2.6.8.1-ac patch you can do IDE controller hotplugging
and a few other things but not yet drive hotplugging. 2.4 can do drive
hotplug although you need a small -ac patch if you see wrong disk
geometry data.

For new controllers (ie SATA ones) use Jeff Garzik's serial ATA layer as
that is a lot cleaner and the SCSI layer already has a good basic
understanding of hotplug management.

> Any help/support is appreciated.

Except for the dynamic stuff I consider the problem solved. Its up to
you when and what you merge and I understand why you want to get stuff
like sysfs there. 

For drive level hotplug its actually a lot easier and I guess that is
the case most users care about. The changes done for 2.6 clean up stuff
like suspend mean the nasties in 2.4 for sequencing have gone away. No
refcounting needed since the block and fs layer are doing it all for
you. TTY layer, revoke(), and some other current critical bonfires first
before I can help with that however.

Alan

Re: IDE Hotswap

[Bartlomiej Zolnierkiewicz]

On Tuesday 28 September 2004 23:45, Alan Cox wrote:
> On Maw, 2004-09-28 at 22:38, Bartlomiej Zolnierkiewicz wrote:
> > No and such workaround won't work anyway because
> > re-register operation is nothing else but unregister+register.
> 
> If you grab the 2.6.8.1-ac patch you can do IDE controller hotplugging
> and a few other things but not yet drive hotplugging. 2.4 can do drive
> hotplug although you need a small -ac patch if you see wrong disk
> geometry data.
> 
> For new controllers (ie SATA ones) use Jeff Garzik's serial ATA layer as
> that is a lot cleaner and the SCSI layer already has a good basic
> understanding of hotplug management.
> 
> > Any help/support is appreciated.
> 
> Except for the dynamic stuff I consider the problem solved. Its up to
> you when and what you merge and I understand why you want to get stuff
> like sysfs there. 

Your patch is a nice start but it don't solve main issues, not to even
mention minor stuff like leaving /proc/ide/<chipset> around.

Merging it now is asking for problems.

> For drive level hotplug its actually a lot easier and I guess that is
> the case most users care about. The changes done for 2.6 clean up stuff

drive level hotplug is actually much harder
and it is _required_ for controller level hotplug, no? :)

> like suspend mean the nasties in 2.4 for sequencing have gone away. No
> refcounting needed since the block and fs layer are doing it all for

It helps but you still get bunch of races.  Refcounting is _really_ needed.

> you. TTY layer, revoke(), and some other current critical bonfires first
> before I can help with that however.

Fine.

Bartlomiej

Re: IDE Hotswap

[Alan Cox]

On Mer, 2004-09-29 at 02:54, Bartlomiej Zolnierkiewicz wrote:
> Your patch is a nice start but it don't solve main issues, not to even
> mention minor stuff like leaving /proc/ide/<chipset> around.
> Merging it now is asking for problems.

Oh I agree. There is a patch but it isn't the final answer. There is a
small resource bug (harmless but a bug) in the 2.6.8.1-ac patch. I
hadn't noticed proc/ide/<chipset> leaking but I'll take a look when I
get time to sort that out. 

> > For drive level hotplug its actually a lot easier and I guess that is
> > the case most users care about. The changes done for 2.6 clean up stuff
> 
> drive level hotplug is actually much harder
> and it is _required_ for controller level hotplug, no? :)

Actually its all a lot easier now than in 2.4 because the block layer is
designed to make it possible.

> > like suspend mean the nasties in 2.4 for sequencing have gone away. No
> > refcounting needed since the block and fs layer are doing it all for
> 
> It helps but you still get bunch of races.  Refcounting is _really_ needed.

Even in 2.4 ide drive hotplug was easy. The drive hotplug comes out
trivially because your controllers are fairly constant. As we all know
driver level hotplug is a bit trickier although the block layer has
really made this vastly easier in 2.6

For drive level hotplug you don't actually need refcounting at all
providing you've got a couple of locking issues dealt with.

Firstly the drive never goes away as a high level object (in fact you
don't want it to as then you can't ioctl it to make it come back!). That
means the upper layers don't know anything about it.

At the IDE layer the 2.4 code simply enforced the rule that you must be
the only opener of the device in order to hot unplug it. That means we
know its quiescent and not mounted. The only 2.4 race I know about is
suspend in parallel to hot unplug, and 2.6 has the mechanism to fix that
properly because suspend is a command state machine.

Providing hot unplug is about making drive->present stuff vary and
flipping to ide_default drivers the world is happy. The moment you want
to make /dev/hda 'disappear' to the block layer its fun, and on 2.4 its
as good as impossible.

Alan

Re: IDE Hotswap

[Bartlomiej Zolnierkiewicz]

On Wednesday 29 September 2004 03:50, Alan Cox wrote:
> On Mer, 2004-09-29 at 02:54, Bartlomiej Zolnierkiewicz wrote:
> > Your patch is a nice start but it don't solve main issues, not to even
> > mention minor stuff like leaving /proc/ide/<chipset> around.
> > Merging it now is asking for problems.
> 
> Oh I agree. There is a patch but it isn't the final answer. There is a
> small resource bug (harmless but a bug) in the 2.6.8.1-ac patch. I
> hadn't noticed proc/ide/<chipset> leaking but I'll take a look when I
> get time to sort that out. 

/proc/ide/<chipset> is the smallest problem

Don't waste your time on it, I've almost killed this bloat in my tree.

> > > like suspend mean the nasties in 2.4 for sequencing have gone away. No
> > > refcounting needed since the block and fs layer are doing it all for
> > 
> > It helps but you still get bunch of races.  Refcounting is _really_ needed.
> 
> Even in 2.4 ide drive hotplug was easy. The drive hotplug comes out
> trivially because your controllers are fairly constant. As we all know
> driver level hotplug is a bit trickier although the block layer has
> really made this vastly easier in 2.6
> 
> For drive level hotplug you don't actually need refcounting at all
> providing you've got a couple of locking issues dealt with.

These issues can't be solved without refcounting.

Feel free to probe me wrong, you can start with fixing
->open vs unregister race (drive->usage involved). :)

> Firstly the drive never goes away as a high level object (in fact you
> don't want it to as then you can't ioctl it to make it come back!). That
> means the upper layers don't know anything about it.

ioctls on not present devices are layering VIOLATION

> At the IDE layer the 2.4 code simply enforced the rule that you must be
> the only opener of the device in order to hot unplug it. That means we

"enforced" - there are a couple of races, sorry but ROTFL

> know its quiescent and not mounted. The only 2.4 race I know about is

- double unlock obvious mistake
- ->open() vs unregister
- /proc races (the same you fixed in your 2.6 patch)
- ioctl races

I'm sure there is more.

> suspend in parallel to hot unplug, and 2.6 has the mechanism to fix that
> properly because suspend is a command state machine.
> 
> Providing hot unplug is about making drive->present stuff vary and
> flipping to ide_default drivers the world is happy. The moment you want
> to make /dev/hda 'disappear' to the block layer its fun, and on 2.4 its
> as good as impossible.

gendisk layer and block layer enforces you to make /dev/hda disappear.
Does sysfs ring any bells?  Ask viro about static objects vs sysfs.
And yes not only gendisk and block enforces this, Patrick added basic,
premature sysfs support to IDE driver in the middle of 2.5 series.

We can get back to discussion when you get familiar with issues involved.

Bartlomiej

Re: IDE Hotswap

[Alan Cox]

On Mer, 2004-09-29 at 13:08, Bartlomiej Zolnierkiewicz wrote:
> > Even in 2.4 ide drive hotplug was easy. The drive hotplug comes out
> > trivially because your controllers are fairly constant. As we all know
> > driver level hotplug is a bit trickier although the block layer has
> > really made this vastly easier in 2.6
> > 
> > For drive level hotplug you don't actually need refcounting at all
> > providing you've got a couple of locking issues dealt with.
> 
> These issues can't be solved without refcounting.

So you keep saying, but you refcount objects that are going away, you
don't need to refcount objects that are staying put.

> Feel free to probe me wrong, you can start with fixing
> ->open vs unregister race (drive->usage involved). :)

Doesn't occur in the 2.4 situation or the 2.6 stuff with the locking in
the 2.6.8.1-ac patch.

> > Firstly the drive never goes away as a high level object (in fact you
> > don't want it to as then you can't ioctl it to make it come back!). That
> > means the upper layers don't know anything about it.
> 
> ioctls on not present devices are layering VIOLATION

Oh dear then I guess most of Linux is misdesigned. You aren't thinking
about the semantics at all. 

If /dev/hda is a CD-ROM drive I can issue commands to it with no CD 
present. Thats not a layering violation, and its how the IDE code works.
So whats the difference between hotplugging a drive and removing
a CD. Both are removing the media but leaving the controller behind.

On that item I think you are talking out of your backside.

> > At the IDE layer the 2.4 code simply enforced the rule that you must be
> > the only opener of the device in order to hot unplug it. That means we
> 
> "enforced" - there are a couple of races, sorry but ROTFL
> 
> > know its quiescent and not mounted. The only 2.4 race I know about is
> 
> - double unlock obvious mistake
Details ?
> - ->open() vs unregister
unregister is hot plug controller not drive and thats unfixable in 2.4
> - /proc races (the same you fixed in your 2.6 patch)
yeah that lot postdates the 2.4 work. hotplug drives is not the cause
however.
> - ioctl races
Details ?

> gendisk layer and block layer enforces you to make /dev/hda disappear.

No it does not. The block layer couldn't give a flying **** whether
/dev/hda disappears or not. SCSI devices that are offlined don't need to
disappear either you just hand back commands with an error. You know -
like every CD-ROM does...

> Does sysfs ring any bells?  Ask viro about static objects vs sysfs.
> And yes not only gendisk and block enforces this, Patrick added basic,
> premature sysfs support to IDE driver in the middle of 2.5 series.
> 
> We can get back to discussion when you get familiar with issues involved.

Ditto...

Alan

Re: IDE Hotswap

[Bartlomiej Zolnierkiewicz]

On Wednesday 29 September 2004 16:35, Alan Cox wrote:
> On Mer, 2004-09-29 at 13:08, Bartlomiej Zolnierkiewicz wrote:
> > > Even in 2.4 ide drive hotplug was easy. The drive hotplug comes out
> > > trivially because your controllers are fairly constant. As we all know
> > > driver level hotplug is a bit trickier although the block layer has
> > > really made this vastly easier in 2.6
> > > 
> > > For drive level hotplug you don't actually need refcounting at all
> > > providing you've got a couple of locking issues dealt with.
> > 
> > These issues can't be solved without refcounting.
> 
> So you keep saying, but you refcount objects that are going away, you
> don't need to refcount objects that are staying put.
> 
> > Feel free to probe me wrong, you can start with fixing
> > ->open vs unregister race (drive->usage involved). :)
> 
> Doesn't occur in the 2.4 situation or the 2.6 stuff with the locking in
> the 2.6.8.1-ac patch.

I will verify this in a few days, I have some real work to do first.

> > > Firstly the drive never goes away as a high level object (in fact you
> > > don't want it to as then you can't ioctl it to make it come back!). That
> > > means the upper layers don't know anything about it.
> > 
> > ioctls on not present devices are layering VIOLATION
> 
> Oh dear then I guess most of Linux is misdesigned. You aren't thinking
> about the semantics at all. 
> 
> If /dev/hda is a CD-ROM drive I can issue commands to it with no CD 
> present. Thats not a layering violation, and its how the IDE code works.
> So whats the difference between hotplugging a drive and removing
> a CD. Both are removing the media but leaving the controller behind.

Removing a CD leaves you with a drive between controller and media.

1st case: ioctl -> IDE driver -> drive
(lack of refcounting workarounded by ide-default)

2nd case: ioctl -> IDE driver
(lack of refcounting)

> On that item I think you are talking out of your backside.
> 
> > > At the IDE layer the 2.4 code simply enforced the rule that you must be
> > > the only opener of the device in order to hot unplug it. That means we
> > 
> > "enforced" - there are a couple of races, sorry but ROTFL
> > 
> > > know its quiescent and not mounted. The only 2.4 race I know about is
> > 
> > - double unlock obvious mistake
> Details ?

2003/08/16 alan               | 	/* Drive shutdown sequence done */
2003/08/16 alan               | 	/* Prevent new opens ?? */
2003/08/16 alan               | 	spin_unlock_irqrestore(&io_request_lock, flags);
2003/08/16 alan               | 	/*
2003/08/16 alan               | 	 * Flush kernel side caches, and dump the /proc files
2003/08/16 alan               | 	 */
2003/08/16 alan               | 	spin_unlock_irqrestore(&io_request_lock, flags);

> > - ->open() vs unregister
> unregister is hot plug controller not drive and thats unfixable in 2.4

Doesn't matter. race is the same AFAICS, drive->usage
access is not protected by any lock.

> > - /proc races (the same you fixed in your 2.6 patch)
> yeah that lot postdates the 2.4 work. hotplug drives is not the cause
> however.
> > - ioctl races
> Details ?

OK BKL protects us against i.e. concurrent HDIO_GETGEO
and hotplug ioctl.  There is however no protection for controller
hotplug.

> > gendisk layer and block layer enforces you to make /dev/hda disappear.
> 
> No it does not. The block layer couldn't give a flying **** whether
> /dev/hda disappears or not. SCSI devices that are offlined don't need to
> disappear either you just hand back commands with an error. You know -
> like every CD-ROM does...
>
> > Does sysfs ring any bells?  Ask viro about static objects vs sysfs.
> > And yes not only gendisk and block enforces this, Patrick added basic,
> > premature sysfs support to IDE driver in the middle of 2.5 series.
> > 
> > We can get back to discussion when you get familiar with issues involved.
> 
> Ditto...
> 
> Alan
> 
> 

Re: IDE Hotswap

[Alan Cox]

On Mer, 2004-09-29 at 19:20, Bartlomiej Zolnierkiewicz wrote:
> > Doesn't occur in the 2.4 situation or the 2.6 stuff with the locking in
> > the 2.6.8.1-ac patch.
> 
> I will verify this in a few days, I have some real work to do first.

2.6 is the important stuff sure.

> > > - double unlock obvious mistake
> > Details ?
> 
> 2003/08/16 alan               | 	/* Drive shutdown sequence done */
> 2003/08/16 alan               | 	/* Prevent new opens ?? */
> 2003/08/16 alan               | 	spin_unlock_irqrestore(&io_request_lock, flags);
> 2003/08/16 alan               | 	/*
> 2003/08/16 alan               | 	 * Flush kernel side caches, and dump the /proc files
> 2003/08/16 alan               | 	 */
> 2003/08/16 alan               | 	spin_unlock_irqrestore(&io_request_lock, flags);
> 

Thanks. I'll go over this when I dig out the one little bit 2.4.2x needs
for hotplug to behave right with hard disks that I should get to
Marcelo.

> OK BKL protects us against i.e. concurrent HDIO_GETGEO
> and hotplug ioctl.  There is however no protection for controller
> hotplug.

Agreed.

IDE hotswap

[Marc SCHAEFER]

I would like to implement real hotswapping support with IDE drives in the
following setup:

   Asus A7V motherboard (has 2 standard IDE bus, and 2 additional bus)
   cheap/standard hotswap IDE hardware

   00:04.1 IDE interface: VIA Technologies, Inc. VT82C586 IDE [Apollo] (rev 10)
   00:11.0 Unknown mass storage controller: Promise Technology, Inc.: Unknown device 0d30 (rev 02)

Simply removing and replacing the disk drive will cause hangs in normal
operation.

I have noticed, by creating a simple boot floppy with IDE modules
(ide_mod, ide_disk, ide_probe_mod), that if you remove ide_probe_mod, 
replace the disk, and reinsert ide_probe_mod, the disk is detected
correctly (geometry, size, etc).

My questions:

   - assuming one-disk-per-bus (ie 4 disks in the above IDE+promise
     configuration), is it at all feasible ?  (ie IDE masters only)

   - should I expect the IDE chips to hardware-fail after a few
     inserts/removes ?

   - would modifying drivers/ide/ide_probe.c to add a new function
     drop_bus(bus_id), scan_bus(bus_id) be a good idea, along with
     a /proc interface from ide_probe.c or another to be developped
     module ?

   - is there already work in progress in the topic ?  or a simpler
     way to do the above ?

The initial case is to assume that inserting/removing must be done in a
software-controlled way.

The next step will be to see if the IDE subsystem can detect a failing
drive / removed drive without warning. Reinsertion could be done by
polling every 30 seconds or manual user intervention. 

Thank you for any comments.

Re: IDE hotswap

[Marc SCHAEFER]

On Tue, 28 May 2002, Marc SCHAEFER wrote:

> I would like to implement real hotswapping support with IDE drives in the
> following setup:

Ok, after a few more searches, and very nice help from a linux-kernel
reader, I found notably a past thread which is full of very interesting
information:

   http://search.alphanet.ch/cgi-bin/search.cgi?subject=IDE+and+hot-swap+disk+caddies&max_results=10&type=long&domain=ml-linux-kernel

(put 31 instead of 10 if you want the whole thing in one page).

We learn there that:

   - someone else tried the module stuff, but failed (I had success)

        I have noticed, by creating a simple boot floppy with IDE modules
        (ide_mod, ide_disk, ide_probe_mod), that if you remove ide_probe_mod, 
        replace the disk, and reinsert ide_probe_mod, the disk is detected
        correctly (geometry, size, etc) after reinsert or even type
        change.

   - some chipset support tri-stating the bus (hdparm -b 2), I need
     to investigate if this is implemented/works on my hardware. It seems
     to work very well especially if you only put one device per bus,
     which is my goal.

   - some disks can be marked `hotswap' (-x option of hdparam), no idea
     what it means yet.

So I am going to investigate in this direction, maybe combining the
tristate with partial re-detection.

(I also got mail asking me to sign before an NDA before telling me
anything, that was a bit stupid. AFAIK saying `doing it without tristating
can fry your hardware' is not patented yet. Maybe it's this world which is
going insane because of patents).

Thank you for your time.