wrong /dev/sd... order with multiple adapters in kernel 2.4.4

wrong /dev/sd... order with multiple adapters in kernel 2.4.4

[Massimo Dal Zotto]

Hi,

I have recently upgraded the kernel from 2.2.19 to 2.4.4 and discovered
that it assigns the /dev/sd... devices in the wrong order with respect both
to the behavior of kernel 2.2.19 and to the `scsihosts' boot option which I
specified at the boot prompt.

I have a scsi-only machine with an Adaptec 7890 and an old Symbios 53c875.
The Adaptec mounts an LVD disk with the root and home partitions while the
Symbios mounts two additional disks used for other purposes:

# cat /proc/pci
  Bus  0, device   6, function  0:
    SCSI storage controller: Adaptec AHA-2940U2/W / 7890 (rev 0).
      IRQ 11.
      Master Capable.  Latency=32.  Min Gnt=39.Max Lat=25.
      I/O at 0xd000 [0xd0ff].
      Non-prefetchable 64 bit memory at 0xe0800000 [0xe0800fff].
  Bus  0, device   9, function  0:
    SCSI storage controller: Symbios Logic Inc. (formerly NCR) 53c875 (rev 3).
      IRQ 11.
      Master Capable.  Latency=144.  Min Gnt=17.Max Lat=64.
      I/O at 0xb800 [0xb8ff].
      Non-prefetchable 32 bit memory at 0xe0000000 [0xe00000ff].
      Non-prefetchable 32 bit memory at 0xdf800000 [0xdf800fff].

# cat /proc/scsi/scsi
Attached devices: 
Host: scsi0 Channel: 00 Id: 00 Lun: 00
  Vendor: FUJITSU  Model: MAE3091LP        Rev: 0112
  Type:   Direct-Access                    ANSI SCSI revision: 02
Host: scsi1 Channel: 00 Id: 00 Lun: 00
  Vendor: WDIGTL   Model: ENTERPRISE       Rev: 1.91
  Type:   Direct-Access                    ANSI SCSI revision: 02
Host: scsi1 Channel: 00 Id: 04 Lun: 00
  Vendor: FUJITSU  Model: MAE3091LP        Rev: 0112
  Type:   Direct-Access                    ANSI SCSI revision: 02

Since I want to use the kernel also in rescue diks I have compiled the needed
scsi divers statically:

CONFIG_SCSI=y
CONFIG_BLK_DEV_SD=y
CONFIG_BLK_DEV_SR=y
CONFIG_SCSI_AIC7XXX=y
CONFIG_SCSI_SYM53C8XX=y

With kernel 2.2.19 the Adaptec is initialized first and it assigns /dev/sda
to its only disk. With kernel 2.4.4 the Symbios is initialized first,
/dev/sda is assigned to the wrong (for my purposes) disk and I'm not able
to boot the system, unless I fix the fstab, which I don't want to do because
I need to use it also with the old kernel 2.2.19.
This is the relevant output from dmesg:

Linux version 2.4.4 (root@nikita) (gcc version 2.95.2 20000220 (Debian GNU/Linux)) #1 Tue May 15 10:12:38 MEST 2001
PCI: PCI BIOS revision 2.10 entry at 0xf0720, last bus=1
PCI: Using configuration type 1
PCI: Probing PCI hardware
PCI: Using IRQ router PIIX [8086/7110] at 00:04.0
SCSI subsystem driver Revision: 1.00
scsi: host order: aic7xxx
PCI: Found IRQ 11 for device 00:09.0
PCI: The same IRQ used for device 00:04.2
PCI: The same IRQ used for device 00:06.0
sym53c8xx: at PCI bus 0, device 9, function 0
scsi1 : sym53c8xx-1.7.3a-20010304
  Vendor: WDIGTL    Model: ENTERPRISE        Rev: 1.91
  Type:   Direct-Access                      ANSI SCSI revision: 02
  Vendor: FUJITSU   Model: MAE3091LP         Rev: 0112
  Type:   Direct-Access                      ANSI SCSI revision: 02
Detected scsi disk sda at scsi1, channel 0, id 0, lun 0
Detected scsi disk sdb at scsi1, channel 0, id 4, lun 0
SCSI device sda: 8515173 512-byte hdwr sectors (4360 MB)
 /dev/scsi/host1/bus0/target0/lun0: p1 p2
SCSI device sdb: 17826240 512-byte hdwr sectors (9127 MB)
 /dev/scsi/host1/bus0/target4/lun0: p1
PCI: Found IRQ 11 for device 00:06.0
PCI: The same IRQ used for device 00:04.2
PCI: The same IRQ used for device 00:09.0
scsi0 : Adaptec AIC7XXX EISA/VLB/PCI SCSI HBA DRIVER, Rev 6.1.5
        <Adaptec aic7890/91 Ultra2 SCSI adapter>
        aic7890/91: Wide Channel A, SCSI Id=7, 32/255 SCBs
  Vendor: FUJITSU   Model: MAE3091LP         Rev: 0112
  Type:   Direct-Access                      ANSI SCSI revision: 02
Detected scsi disk sdc at scsi0, channel 0, id 0, lun 0
SCSI device sdc: 17826240 512-byte hdwr sectors (9127 MB)
 /dev/scsi/host0/bus0/target0/lun0: p1 p2 p3 p4

There is nothing wrong in this, the kernel must choose some ordering when
initializing the devices and Symbios first is not worse than Adaptec first.

The problem is that the new kernel accepts a boot option (scsihosts) to
force the order in which the scsi devices are detected, but this option is
completely ignored while detecting the /dev/sd... devices. Even specifying
the boot option "scsihosts=aic7xxx" I get the following scsi assignments: 

    /dev/scsi/host0/bus0/target0/lun0   /dev/sdc
    /dev/scsi/host1/bus0/target0/lun0	/dev/sda
    /dev/scsi/host1/bus0/target4/lun0   /dev/sdb

which seems illogical to me because the device order is inconsistent between
the two device naming schemes. This is also incompatible with the behavior of
kernel-2.2.x.

Also trying the `pci=bios' option doesn't help because the initialization
order is determined not by the pci bus scan order but only by the ld order
of the drivers when the kernel was compiled.

The only way to solve my problem was to modify the makefiles and link the
aic7xxx driver before the sym53c8xx, but this is a solution only for my
specific case. With my patch the Adaptec is initialized first and I get
a more consistent order of the scsi devices: 

    /dev/scsi/host0/bus0/target0/lun0   /dev/sda
    /dev/scsi/host1/bus0/target0/lun0	/dev/sdb
    /dev/scsi/host1/bus0/target4/lun0   /dev/sdc

I suggest that the scsi device detection is done in the order specified by
the `scsihosts' boot option or, if not specified, in the pci bus scan order
as reported by /proc/pci.

-- 
Massimo Dal Zotto

+----------------------------------------------------------------------+
|  Massimo Dal Zotto               email: dz@cs.unitn.it               |
|  Via Marconi, 141                phone: ++39-0461534251              |
|  38057 Pergine Valsugana (TN)      www: http://www.cs.unitn.it/~dz/  |
|  Italy                             pgp: see my www home page         |
+----------------------------------------------------------------------+

Re: wrong /dev/sd... order with multiple adapters in kernel 2.4.4

[Michael Meissner]

On Wed, May 16, 2001 at 01:38:37PM +0200, Massimo Dal Zotto wrote:
> Hi,
> 
> I have recently upgraded the kernel from 2.2.19 to 2.4.4 and discovered
> that it assigns the /dev/sd... devices in the wrong order with respect both
> to the behavior of kernel 2.2.19 and to the `scsihosts' boot option which I
> specified at the boot prompt.
> 
> I have a scsi-only machine with an Adaptec 7890 and an old Symbios 53c875.
> The Adaptec mounts an LVD disk with the root and home partitions while the
> Symbios mounts two additional disks used for other purposes:

	...

> The only way to solve my problem was to modify the makefiles and link the
> aic7xxx driver before the sym53c8xx, but this is a solution only for my
> specific case. With my patch the Adaptec is initialized first and I get
> a more consistent order of the scsi devices: 
> 
>     /dev/scsi/host0/bus0/target0/lun0   /dev/sda
>     /dev/scsi/host1/bus0/target0/lun0	/dev/sdb
>     /dev/scsi/host1/bus0/target4/lun0   /dev/sdc

I had the same problem.  The fix that I use is to compile the Symbios
driver as a module, and add:

	alias scsi_hostadapter1 sym53c8xx

to my /etc/modules.conf.  That way, when the kernel boots, it will only see the
Adaptec driver.

-- 
Michael Meissner, Red Hat, Inc.  (GCC group)
PMB 198, 174 Littleton Road #3, Westford, Massachusetts 01886, USA
Work:	  meissner@redhat.com		phone: +1 978-486-9304
Non-work: meissner@spectacle-pond.org	fax:   +1 978-692-4482

Re: wrong /dev/sd... order with multiple adapters in kernel 2.4.4

[Richard B. Johnson]

On Wed, 16 May 2001, Massimo Dal Zotto wrote:

> Hi,
> 
> I have recently upgraded the kernel from 2.2.19 to 2.4.4 and discovered
> that it assigns the /dev/sd... devices in the wrong order with respect both
> to the behavior of kernel 2.2.19 and to the `scsihosts' boot option which I
> specified at the boot prompt.
[SNIPPED]

As a work-around, you can use modules and boot through 'initrd' where
you load the modules in the order you want.

When you have two SCSI disk controllers, the order at which the drives
are seen will "always be wrong" --Murphys Law. You can control the
order in which the controllers are detected and installed by using
modules.

Basically you cannot ever expect that multiple controllers will
be detected in any particular order. They usually end up being detected
in the device-order for which they exist on the PCI bus. This changes!

If both of your controllers are "pluggable", you can swap them
on the physical bus to change the order in which they are detected.

If only one is pluggable, you might try another PCI slot. It may
have a number above/below the one embedded on the board.

Cheers,
Dick Johnson

Penguin : Linux version 2.4.1 on an i686 machine (799.53 BogoMips).

"Memory is like gasoline. You use it up when you are running. Of
course you get it all back when you reboot..."; Actual explanation
obtained from the Micro$oft help desk.

Re: wrong /dev/sd... order with multiple adapters in kernel 2.4.4

[Massimo Dal Zotto]

> On Wed, 16 May 2001, Massimo Dal Zotto wrote:
> 
> > Hi,
> > 
> > I have recently upgraded the kernel from 2.2.19 to 2.4.4 and discovered
> > that it assigns the /dev/sd... devices in the wrong order with respect both
> > to the behavior of kernel 2.2.19 and to the `scsihosts' boot option which I
> > specified at the boot prompt.
> [SNIPPED]
> 
> As a work-around, you can use modules and boot through 'initrd' where
> you load the modules in the order you want.

I wanted to use a simpler method, a plain kernel with two static drivers
which can simply be copied to /boot or to a floppy. Of course any more
sophisticated method would work but I wanted a simple thing.

> When you have two SCSI disk controllers, the order at which the drives
> are seen will "always be wrong" --Murphys Law. You can control the
> order in which the controllers are detected and installed by using
> modules.

Yes, I understand that any particular order will be wrong for some users.

I'm not complaining about the current scan order. What I'm complaining about
is that we do *have* a specific method of forcing the desired order (the
scsihosts boot option) but this is ignored by some parts of the scsi code.

I suggest that the scsihosts order is taken into account also when detecting
the /dev/sd devices and not only when creating the /dev/scsi/ devices.

> Basically you cannot ever expect that multiple controllers will
> be detected in any particular order. They usually end up being detected
> in the device-order for which they exist on the PCI bus. This changes!

This is not the case with kernel 2.4.4. They are detected in `ld' order.
Changing the makefiles will change the scan order.

> If both of your controllers are "pluggable", you can swap them
> on the physical bus to change the order in which they are detected.

This is impossible given the density of cables and cards in the cabinet, and
useless anyway given the fact that devices are detected in `ld' order.

> If only one is pluggable, you might try another PCI slot. It may
> have a number above/below the one embedded on the board.
> 
> Cheers,
> Dick Johnson
> 
> Penguin : Linux version 2.4.1 on an i686 machine (799.53 BogoMips).
> 
> "Memory is like gasoline. You use it up when you are running. Of
> course you get it all back when you reboot..."; Actual explanation
> obtained from the Micro$oft help desk.
> 
> 

-- 
Massimo Dal Zotto

+----------------------------------------------------------------------+
|  Massimo Dal Zotto               email: dz@cs.unitn.it               |
|  Via Marconi, 141                phone: ++39-0461534251              |
|  38057 Pergine Valsugana (TN)      www: http://www.cs.unitn.it/~dz/  |
|  Italy                             pgp: see my www home page         |
+----------------------------------------------------------------------+