From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S935304AbYETPhV (ORCPT ); Tue, 20 May 2008 11:37:21 -0400 Received: (majordomo@vger.kernel.org) by vger.kernel.org id S933265AbYETPdQ (ORCPT ); Tue, 20 May 2008 11:33:16 -0400 Received: from wa-out-1112.google.com ([209.85.146.179]:37374 "EHLO wa-out-1112.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1753984AbYETPdO (ORCPT ); Tue, 20 May 2008 11:33:14 -0400 DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references; b=VDleMyk2xr0CetvQvJ5Kpmnku0r05zNrSjgAA3ScCQb8BHUYzi4Flj+dyxcR1Ysv/Ps7f179TiQ7KR02ewhGWTGwxTeX9HIe/aIjxrKoGmI8u/cMDkVdXIuXQRHRh4zPqPQoayrSGJql2Dw7dNzYkoDM62DefeyZUogfmqBLXqM= Message-ID: <170fa0d20805200833s30c0a856p5f08250879a182b9@mail.gmail.com> Date: Tue, 20 May 2008 11:33:13 -0400 From: "Mike Snitzer" To: "Neil Brown" Subject: Re: [RFC][PATCH] md: avoid fullsync if a faulty member missed a dirty transition Cc: linux-raid@vger.kernel.org, linux-kernel@vger.kernel.org, paul.clements@steeleye.com In-Reply-To: <170fa0d20805200830w184709dch221f8b9636f3bc0e@mail.gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline References: <1207174155-20090-1-git-send-email-snitzer@gmail.com> <170fa0d20805081311r6b726762ndc83c5f626f0956c@mail.gmail.com> <18467.43895.220708.254814@notabene.brown> <170fa0d20805082142o61462a7dnce7852a6b7c0d21d@mail.gmail.com> <18467.59565.676987.926988@notabene.brown> <170fa0d20805090800p45654901p36eea51e1ea7e9c7@mail.gmail.com> <18477.30199.992302.271647@notabene.brown> <170fa0d20805182133o46501cc4va81b087fb6b417bf@mail.gmail.com> <18481.4005.570022.160686@notabene.brown> <170fa0d20805200830w184709dch221f8b9636f3bc0e@mail.gmail.com> Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Tue, May 20, 2008 at 11:30 AM, Mike Snitzer wrote: > On Mon, May 19, 2008 at 1:27 AM, Neil Brown wrote: > > On Monday May 19, snitzer@gmail.com wrote: > > > > > > Hi Neil, > > > > > > Sorry about not getting back with you sooner. Thanks for putting > > > significant time to chasing this problem. > > > > > > I tested your most recent patch and unfortunately still hit the case > > > where the nbd member becomes degraded yet the array continues to clear > > > bits (events_cleared of the non-degraded member is higher than the > > > degraded member). Is this behavior somehow expected/correct? > > > > It shouldn't be..... ahhh. > > There is a delay between noting that the bit can be cleared, and > > actually writing the zero to disk. This is obviously intentional > > in case the bit gets set again quickly. > > I'm sampling the event count at the latter point instead of the > > former, and there is time for it to change. > > > > Maybe this patch on top of what I recently sent out? > > Hi Neil, > > We're much closer. The events_cleared is symmetric on both the failed > and active member of the raid1. But there have been some instances > where the md thread hits a deadlock during my testing. What follows > is the backtrace and live crash info: > > md0_raid1 D 000002c4b6483a7f 0 11249 2 (L-TLB) > ffff81005747dce0 0000000000000046 0000000000000000 ffff8100454c53c0 > 000000000000000a ffff810048fbd0c0 000000000000000a ffff810048fbd0c0 > ffff81007f853840 000000000000148e ffff810048fbd2b0 0000000362c10780 > Call Trace: > [] :md_mod:bitmap_daemon_work+0x249/0x4d3 > [] autoremove_wake_function+0x0/0x2e > [] :md_mod:md_check_recovery+0x20/0x4a5 > [] thread_return+0x0/0xf1 > [] :raid1:raid1d+0x25/0xd09 > [] lock_timer_base+0x26/0x4b > [] try_to_del_timer_sync+0x51/0x5a > [] del_timer_sync+0xc/0x16 > [] schedule_timeout+0x92/0xad > [] :md_mod:md_thread+0xeb/0x101 > [] autoremove_wake_function+0x0/0x2e > [] :md_mod:md_thread+0x0/0x101 > [] kthread+0x47/0x76 > [] child_rip+0xa/0x12 > [] kthread+0x0/0x76 > [] child_rip+0x0/0x12 > > crash> bt 11249 > PID: 11249 TASK: ffff810048fbd0c0 CPU: 3 COMMAND: "md0_raid1" > #0 [ffff81005747dbf0] schedule at ffffffff8044cb5c > #1 [ffff81005747dce8] bitmap_daemon_work at ffffffff88ba8503 > #2 [ffff81005747dd68] md_check_recovery at ffffffff88ba53b3 > #3 [ffff81005747ddb8] raid1d at ffffffff88bbe0eb > #4 [ffff81005747ded8] md_thread at ffffffff88ba6c6c > #5 [ffff81005747df28] kthread at ffffffff8024564d > #6 [ffff81005747df48] kernel_thread at ffffffff8020aa38 > > 0xffffffff88ba84ee : callq > 0xffffffff802458ec > 0xffffffff88ba84f3 : mov 0x18(%rbx),%rax > 0xffffffff88ba84f7 : mov 0x28(%rax),%eax > 0xffffffff88ba84fa : test $0x2,%al > 0xffffffff88ba84fc : je > 0xffffffff88ba8505 > 0xffffffff88ba84fe : callq > 0xffffffff8044c200 <__sched_text_start> > 0xffffffff88ba8503 : jmp > 0xffffffff88ba84d6 > 0xffffffff88ba8505 : mov 0x18(%rbx),%rdi > 0xffffffff88ba8509 : mov %rbp,%rsi > 0xffffffff88ba850c : add $0x200,%rdi > 0xffffffff88ba8513 : callq > 0xffffffff802457f6 > > So running with your latest patches seems to introduce a race in > bitmap_daemon_work's if (unlikely((*bmc & COUNTER_MAX) == > COUNTER_MAX)) { } block. Err, that block is in bitmap_startwrite()... Mike