[dm-devel] bio too big device md1 (16 > 8)

[dm-devel] bio too big device md1 (16 > 8)

[syrius.ml]

Hi,

i'm using 2.6.21-rc5-git9 +
http://www.kernel.org/pub/linux/kernel/people/agk/patches/2.6/editing/dm-merge-max_hw_sector.patch 
( i've been testing with and without it, and first encountered it on
2.6.18-debian )

I've setup a raid1 array md1 (it was created in a degraded mode using
the debian installer)
(md0 is also a small raid1 array created in degraded mode, but i did
not have any issue with it)

md1 hold a lvm physical volume holding a vg and several lvs

mdadm -D /dev/md1:
/dev/md1:
        Version : 00.90.03
  Creation Time : Sun Mar 25 16:34:42 2007
     Raid Level : raid1
     Array Size : 290607744 (277.15 GiB 297.58 GB)
    Device Size : 290607744 (277.15 GiB 297.58 GB)
   Raid Devices : 2
  Total Devices : 1
Preferred Minor : 1
    Persistence : Superblock is persistent

    Update Time : Tue Apr  3 01:37:23 2007
          State : clean, degraded
 Active Devices : 1
Working Devices : 1
 Failed Devices : 0
  Spare Devices : 0

           UUID : af8d2807:e573935d:04be1e12:bc7defbb
         Events : 0.422096

    Number   Major   Minor   RaidDevice State
       0       3        3        0      active sync   /dev/hda3
       1       0        0        1      removed


the problem i'm encountering is when i add /dev/md2 to /dev/md1.

mdadm -D /dev/md2
/dev/md2:
        Version : 00.90.03
  Creation Time : Sun Apr  1 15:06:43 2007
     Raid Level : linear
     Array Size : 290607808 (277.15 GiB 297.58 GB)
   Raid Devices : 4
  Total Devices : 4
Preferred Minor : 2
    Persistence : Superblock is persistent

    Update Time : Sun Apr  1 15:06:43 2007
          State : clean
 Active Devices : 4
Working Devices : 4
 Failed Devices : 0
  Spare Devices : 0

       Rounding : 64K

           UUID : 887ecdeb:5f205eb6:4cd470d6:4cbda83c (local to host odo)
         Events : 0.1

    Number   Major   Minor   RaidDevice State
       0      34        4        0      active sync   /dev/hdg4
       1      57        2        1      active sync   /dev/hdk2
       2      91        3        2      active sync   /dev/hds3
       3      89        2        3      active sync   /dev/hdo2

I use mdadm --manage --add /dev/md1 /dev/md2
when I do so here is what happen:
md: bind<md2>
RAID1 conf printout:
 --- wd:1 rd:2
 disk 0, wo:0, o:1, dev:hda3
 disk 1, wo:1, o:1, dev:md2
md: syncing RAID array md1
md: minimum _guaranteed_ reconstruction speed: 1000 KB/sec/disc.
md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) 
for reconstruction.
md: using 128k window, over a total of 290607744 blocks.
bio too big device md1 (16 > 8)
Device dm-7, XFS metadata write error block 0x243ec0 in dm-7
bio too big device md1 (16 > 8)
I/O error in filesystem ("dm-8") meta-data dev dm-8 block 0x1b5b6550       ("xfs_trans_read_buf") error 5 buf count 8192
bio too big device md1 (16 > 8)
I/O error in filesystem ("dm-8") meta-data dev dm-8 block 0x1fb3b00       ("xfs_trans_read_buf") error 5 buf count 8192

every filesystems on md1 get corrupted.
I manually fail md2 then reboot and so i can boot the fs again.
(but md1 is still degraded)

Any idea ?
I can provide more information if needed. (the only weird thing is
/dev/hdo that doesn't seem to be lba48-ready, but i guess that
shouldn't be a geometry issue.)

-- 

Re: [dm-devel] bio too big device md1 (16 > 8)

[Neil Brown]

This is difficult.

Summary of problem is:
  Filesystem on LVM on md/raid1
  Add an md/linear to the md/raid1 and fs dies with 'bio too big'.

Where to begin....

The fs level builds bios to send down to the device, and it queries
the device to find out how big the bio can be.
There are two ways it can query the device
  1/ it can look at the max_sectors field in the queue structure.
  2/ it can call the merge_bvec_fn function in the queue structure.

It actually does both.  max_sectors sets an over-all maximum, and
merge_bvec_fn - if defined - can ACK or NAK each page being added.

It is currently very awkward for a stacked device to respond to a
merge_bvec_fn by calling the merge_bvec_fn of the underlying devices.
So they don't.
I'm not sure what dm does, but md simply sets max_sectors to 1 page if
there is a merge_bvec_fn in an underlying device.

So in this particular case, as md/linear defines a merge_bvec_fn, as
soon as the md/linear array is added to the md/raid1 array, the
max_sectors of the md/raid1 is set to 1 page.

This works ok when a filesystem is on an md/raid1 as the filesystem
check max_sectors every time before building the bio.

However in your case, dm (aka LVM) is in there too.

dm doesn't know that md/raid1 has just changed max_sectors and there
is no convenient way for it to find out.  So when the filesystem tries
to get the max_sectors for the dm device, it gets the value that dm set
up when it was created, which was somewhat larger than one page.
When the request gets down to the raid1 layer, it caused a problem.

You can probably make your array work by adding the md/linear array to
the md/raid1 *before* enabling the LVM device on top of it.  However
that isn't really a good long-term solution.

We really need better ways for stacked devices to communicate.

Alasdair Kergon (dm developer) has some patches to make it practical
to recursively call merge_bvec_fn so they might be part of the
solution, but more discussion is needed before that becomes a reality.

Sorry I cannot be more helpful at this stage.

NeilBrown

Re: [dm-devel] bio too big device md1 (16 > 8)

[syrius.ml]

Neil Brown <neilb@suse.de> writes:

> This is difficult.
>
> Summary of problem is:
>   Filesystem on LVM on md/raid1
>   Add an md/linear to the md/raid1 and fs dies with 'bio too big'.
>
> [...]
>
> You can probably make your array work by adding the md/linear array to
> the md/raid1 *before* enabling the LVM device on top of it.  However
> that isn't really a good long-term solution.

Thanks for the explanations !
That's the kind of solution i did use. But it seems I achieved to add
the md/linear array to the md/raid1 after lvm was enabled, I only have
to not to use an initrd.

the kernel I'm using at the moment is a 2.6.18-4-vserver-xen (debian
one) plus the following patches:
- dm-io-fix-bi_max_vecs.patch
- dm-merge-max_hw_sector.patch
also i'm using #define RESYNC_BLOCK_SIZE PAGE_SIZE rather than
(64*1024) in raid1.c.

Not sure if that could explain the fact i was able to add the md/linear
array to the md/raid1 after enabling lvm. (still i had not to use an
initrd)

When I have more time I'll try to reproduce this on a test machine.

> We really need better ways for stacked devices to communicate.
>
> Alasdair Kergon (dm developer) has some patches to make it practical
> to recursively call merge_bvec_fn so they might be part of the
> solution, but more discussion is needed before that becomes a reality.

Ok, i'd be curious to test them (if they're usable and publicly
available)

> Sorry I cannot be more helpful at this stage.

Thanks. (at least i understand(-ish) what's going on) ;-)

--