mirror of https://lore.kernel.org/lkml/
 help / color / mirror / Atom feed
From: Konstantin Kharlamov <Hi-Angel@yandex.ru>
To: Yu Kuai <yukuai1@huaweicloud.com>, Song Liu <song@kernel.org>,
	 linux-raid@vger.kernel.org, linux-kernel@vger.kernel.org,
	"yangerkun@huawei.com" <yangerkun@huawei.com>,
	"yukuai (C)" <yukuai3@huawei.com>
Cc: dm-devel@lists.linux.dev, Matthew Sakai <msakai@redhat.com>
Subject: Re: Lockup of (raid5 or raid6) + vdo after taking out a disk under load
Date: Wed, 31 Jul 2024 17:14:33 +0300	[thread overview]
Message-ID: <37df66ec9cf1a0570a86ec0b9f17ae18ed11b832.camel@yandex.ru> (raw)
In-Reply-To: <57241c91337e8fc3257b6d4a35c273af59875eff.camel@yandex.ru>

CC'ing VDO maintainers, because the problem is only reproducible with
VDO, so potentially they might have some ideas.

On Mon, 2024-07-22 at 20:56 +0300, Konstantin Kharlamov wrote:
> Hi, sorry for the delay, I had to give away the nodes and we had a
> week
> of teambuilding and company party, so for the past week I only
> managed
> to hack away stripping debug symbols, get another node and set it up.
> 
> Experiments below are based off of vanilla 6.9.8 kernel *without*
> your
> patch.
> 
> On Mon, 2024-07-15 at 09:56 +0800, Yu Kuai wrote:
> > Line number will be helpful.
> 
> So, after tinkering with building scripts I managed to build modules
> with debug symbols (not the kernel itself but should be good enough),
> but for some reason kernel doesn't show line numbers in stacktraces.
> No
> idea what could be causing it, so I had to decode line numbers
> manually, below is an output where I inserted line numbers for
> raid456
> manually after decoding them with `gdb`.
> 
>     […]
>     [ 1677.293366]  <TASK>
>     [ 1677.293661]  ? asm_sysvec_apic_timer_interrupt+0x16/0x20
>     [ 1677.293972]  ? _raw_spin_unlock_irq+0x10/0x30
>     [ 1677.294276]  ? _raw_spin_unlock_irq+0xa/0x30
>     [ 1677.294586]  raid5d at drivers/md/raid5.c:6572
>     [ 1677.294910]  md_thread+0xc1/0x170
>     [ 1677.295228]  ? __pfx_autoremove_wake_function+0x10/0x10
>     [ 1677.295545]  ? __pfx_md_thread+0x10/0x10
>     [ 1677.295870]  kthread+0xff/0x130
>     [ 1677.296189]  ? __pfx_kthread+0x10/0x10
>     [ 1677.296498]  ret_from_fork+0x30/0x50
>     [ 1677.296810]  ? __pfx_kthread+0x10/0x10
>     [ 1677.297112]  ret_from_fork_asm+0x1a/0x30
>     [ 1677.297424]  </TASK>
>     […]
>     [ 1705.296253]  <TASK>
>     [ 1705.296554]  ? asm_sysvec_apic_timer_interrupt+0x16/0x20
>     [ 1705.296864]  ? _raw_spin_unlock_irq+0x10/0x30
>     [ 1705.297172]  ? _raw_spin_unlock_irq+0xa/0x30
>     [ 1677.294586]  raid5d at drivers/md/raid5.c:6597
>     [ 1705.297794]  md_thread+0xc1/0x170
>     [ 1705.298099]  ? __pfx_autoremove_wake_function+0x10/0x10
>     [ 1705.298409]  ? __pfx_md_thread+0x10/0x10
>     [ 1705.298714]  kthread+0xff/0x130
>     [ 1705.299022]  ? __pfx_kthread+0x10/0x10
>     [ 1705.299333]  ret_from_fork+0x30/0x50
>     [ 1705.299641]  ? __pfx_kthread+0x10/0x10
>     [ 1705.299947]  ret_from_fork_asm+0x1a/0x30
>     [ 1705.300257]  </TASK>
>     […]
>     [ 1733.296255]  <TASK>
>     [ 1733.296556]  ? asm_sysvec_apic_timer_interrupt+0x16/0x20
>     [ 1733.296862]  ? _raw_spin_unlock_irq+0x10/0x30
>     [ 1733.297170]  ? _raw_spin_unlock_irq+0xa/0x30
>     [ 1677.294586]  raid5d at drivers/md/raid5.c:6572
>     [ 1733.297792]  md_thread+0xc1/0x170
>     [ 1733.298096]  ? __pfx_autoremove_wake_function+0x10/0x10
>     [ 1733.298403]  ? __pfx_md_thread+0x10/0x10
>     [ 1733.298711]  kthread+0xff/0x130
>     [ 1733.299018]  ? __pfx_kthread+0x10/0x10
>     [ 1733.299330]  ret_from_fork+0x30/0x50
>     [ 1733.299637]  ? __pfx_kthread+0x10/0x10
>     [ 1733.299943]  ret_from_fork_asm+0x1a/0x30
>     [ 1733.300251]  </TASK>
> 
> > Meanwhile, can you check if the underlying
> > disks has IO while raid5 stuck, by /sys/block/[device]/inflight.
> 
> The two devices that are left after the 3rd one is removed has these
> numbers that don't change with time:
> 
>     [Mon Jul 22 20:18:06 @ ~]:> for d in dm-19 dm-17; do echo -n $d;
> cat
>     /sys/block/$d/inflight; done
>     dm-19       9        1
>     dm-17      11        2
>     [Mon Jul 22 20:18:11 @ ~]:> for d in dm-19 dm-17; do echo -n $d;
> cat
>     /sys/block/$d/inflight; done
>     dm-19       9        1
>     dm-17      11        2
> 
> They also don't change after I return the disk back (which is to be
> expected I guess, given that the lockup doesn't go away).
> 
> > > 
> > > > At first, can the problem reporduce with raid1/raid10? If not,
> > > > this
> > > > is
> > > > probably a raid5 bug.
> > > 
> > > This is not reproducible with raid1 (i.e. no lockups for raid1),
> > > I
> > > tested that. I didn't test raid10, if you want I can try (but
> > > probably
> > > only after the weekend, because today I was asked to give the
> > > nodes
> > > away, for the weekend at least, to someone else).
> > 
> > Yes, please try raid10 as well. For now I'll say this is a raid5
> > problem.
> 
> Tested: raid10 works just fine, i.e. no lockup and fio continues
> having non-zero IOPS.
> 
> > > > The best will be that if I can reporduce this problem myself.
> > > > The problem is that I don't understand the step 4: turning off
> > > > jbod
> > > > slot's power, is this only possible for a real machine, or can
> > > > I
> > > > do
> > > > this in my VM?
> > > 
> > > Well, let's say that if it is possible, I don't know a way to do
> > > that.
> > > The `sg_ses` commands that I used
> > > 
> > > 	sg_ses --dev-slot-num=9 --set=3:4:1   /dev/sg26 #
> > > turning
> > > off
> > > 	sg_ses --dev-slot-num=9 --clear=3:4:1 /dev/sg26 #
> > > turning
> > > on
> > > 
> > > …sets and clears the value of the 3:4:1 bit, where the bit is
> > > defined
> > > by the JBOD's manufacturer datasheet. The 3:4:1 specifically is
> > > defined
> > > by "AIC" manufacturer. That means the command as is unlikely to
> > > work on
> > > a different hardware.
> > 
> > I never do this before, I'll try.
> > > 
> > > Well, while on it, do you have any thoughts why just using a
> > > `echo
> > > 1 >
> > > /sys/block/sdX/device/delete` doesn't reproduce it? Does perhaps
> > > kernel
> > > not emulate device disappearance too well?
> > 
> > echo 1 > delete just delete the disk from kernel, and scsi/dm-raid
> > will
> > know that this disk is deleted. However, the disk will stay in
> > kernel
> > for the other way, dm-raid does not aware that underlying disks are
> > problematic and IO will still be generated and issued.
> > 
> > Thanks,
> > Kuai


  parent reply	other threads:[~2024-07-31 14:21 UTC|newest]

Thread overview: 12+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2024-07-12  6:53 Konstantin Kharlamov
2024-07-12  8:35 ` Yu Kuai
2024-07-12 12:11   ` Konstantin Kharlamov
2024-07-13 11:06     ` Yu Kuai
2024-07-13 13:50       ` Konstantin Kharlamov
2024-07-15  1:56         ` Yu Kuai
2024-07-22 17:56           ` Konstantin Kharlamov
2024-07-23 14:34             ` Konstantin Kharlamov
2024-07-31 14:14             ` Konstantin Kharlamov [this message]
2024-07-31 20:41               ` Bryan Gurney
2024-08-01 14:46                 ` Konstantin Kharlamov
2024-07-31 21:33               ` Matthew Sakai

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=37df66ec9cf1a0570a86ec0b9f17ae18ed11b832.camel@yandex.ru \
    --to=hi-angel@yandex.ru \
    --cc=dm-devel@lists.linux.dev \
    --cc=linux-kernel@vger.kernel.org \
    --cc=linux-raid@vger.kernel.org \
    --cc=msakai@redhat.com \
    --cc=song@kernel.org \
    --cc=yangerkun@huawei.com \
    --cc=yukuai1@huaweicloud.com \
    --cc=yukuai3@huawei.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox

all inboxes | Powered by JetHome®