From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1753655AbcIHRd6 (ORCPT ); Thu, 8 Sep 2016 13:33:58 -0400 Received: from mx0b-00082601.pphosted.com ([67.231.153.30]:49647 "EHLO mx0a-00082601.pphosted.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1753269AbcIHRdy (ORCPT ); Thu, 8 Sep 2016 13:33:54 -0400 Date: Thu, 8 Sep 2016 10:33:29 -0700 From: Shaohua Li To: Jens Axboe CC: Stefan Priebe - Profihost AG , , Subject: Re: kernel 4.8-rc5 kernel BUG at block/blk-core.c:2032! Message-ID: <20160908173328.GA58334@shli-mbp.local> References: <56e78c6a-5996-d106-2952-d985ceca9e34@profihost.ag> <7ad7efe2-ef46-e1d5-bb0d-c7ed0e8d47db@kernel.dk> MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Disposition: inline In-Reply-To: <7ad7efe2-ef46-e1d5-bb0d-c7ed0e8d47db@kernel.dk> User-Agent: Mutt/1.6.1 (2016-04-27) X-Originating-IP: [2620:10d:c090:200::3:8d55] X-ClientProxiedBy: MWHPR04CA0041.namprd04.prod.outlook.com (10.172.163.27) To CO1PR15MB0933.namprd15.prod.outlook.com (10.166.29.143) X-MS-Office365-Filtering-Correlation-Id: 46d1cdce-1dd3-4d79-7adb-08d3d80e4485 X-Microsoft-Exchange-Diagnostics: 1;CO1PR15MB0933;2:2EGnLDf2wW04rR4SZ95iyfoq4LgbkutbJvFNX8jY+/I4t3+16XlYbbEaqDKx8WDFWeU1L+gJKYBusdxY6AysJ9ElK9P0wyO8n1RQ1DwHcfKntNgPY00L3VDN+CFzBfWU8v2ZEzSNcYey3TODZ1j9TZUdvGXYPVQmVOko65o9ts7EJrLwtSf/elkalj3W/+Cx;3:XCiIv/Bss0abg94CECVnIns/g1mNQinpxzCc153CwPVKnBkdQQRp8uYCLGuYD3USOQlE+q+Fupnycscde1z/DA0jShF/kZLW5w/e4BezKbPoHomgci7sr6baZxKw8C8y;25:qie+Ypl8bqdfT3o5yZqxDPxZ6a4wAm3+Nr+DZZ2bepAI4zWKb2JRNYeCMLlnSiziLp4XNwIqPxealy02d7y6mqEch9NX8pJ/oLzXr+KLepdjobnvfXnUtH/fuhp0NNU12rp6suTYkwtrJumYHYUK1sfhQj22BxYX1fE80Vmu2Y2J7iFYPcn0clJ+2O4DXToo38tX1ScjsFdLmgRRz+FeIQULmVCU615hF242wgrrL9dMsLj3IUftEBk0SqolNvd8csb7pFyRIfUKzFUu5CbMUbx2JAtYbWZpabrKCdAxaMmSE/EO3ydKybTnknRNogeGS6iFoaJePv8h1otredBwkVigeMrq/Gw8uTUIx0VI/xDcdIMl+6DH/O6TvjrBVRLmVN5F05tf7ylLWTorVad9N9HwRjR6UkDGO8lqT8YflFI= X-Microsoft-Antispam: UriScan:;BCL:0;PCL:0;RULEID:;SRVR:CO1PR15MB0933; X-Microsoft-Exchange-Diagnostics: 1;CO1PR15MB0933;31:EHnd84qx+PG0VGD6uv/ZxfqlXmuS2X3BGmvE69R5bNOfGKCfVBr7+/xuo9vBEe/rxu2Bx5bENMe8a6Wtj/vuJdpQWOrWTcm94yGIsqELw8LueNfdGkt3lruwei3hTPLPwKr67hQyNAPIFJ+AMtr8yIRgWAmOZVYohyu5MFx6bFVuguAcGILTTgczXKvPvMRRykR1kCBWenwG6j9tnC28+QjL72DO0xOSUVZ11W1oJ+I=;20:P6bCk12EnH2aD3mqLTw5xDKd230nmWDnhsyS6kNP8qUiHTe1TuHhvmvNC3i/GNh+yqKxGCTo55NnGYu4XcpUTXNXhYrQg7jz0rOriM/J65IU9giufxuWAbehRcLREbhFi/iUhsz0/m/8/tK6whPNv6t9HXOE8NWQGqB18NnoLNo=;4:U1Nz7jGcOoPPC/QmGburc/cWP0ky81VcevzSrgevLFHL1rk+5AZjTOhpOO1NQ93sPJ/oXUKvb3YMTQqhZBI9lkr7NqW0W3bAiEP4a6d+K1enSCWk2CsipkqbFqamqeDgpunI0ZS5FaB4MFNs1WarBwJXnVjophSMD1ydl5zWrRauE7XAxUPngteqGz01QfS2qsH9agBvSyWwy2frlaePhHwNi4vgjF8dsj+/pGw7xuFkxA90Uk2pwcQ0g9EUkXxK2Brl/uvL/88jRxCybWPXua9+uNQeHew+WHOJ/YfTOkBpyB+3TP5BWAEUHU3Nhaps+cG5aEskDcyUWf9/cwy6cw0m8/RwMqYcz6a/rVnWJCeyRCzZz74lLTC/4ySEdhr1fROz+oVe4OheyMWvweUvRg== X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:; X-Exchange-Antispam-Report-CFA-Test: BCL:0;PCL:0;RULEID:(6040176)(601004)(2401047)(8121501046)(5005006)(10201501046)(3002001);SRVR:CO1PR15MB0933;BCL:0;PCL:0;RULEID:;SRVR:CO1PR15MB0933; X-Forefront-PRVS: 00594E8DBA X-Forefront-Antispam-Report: SFV:NSPM;SFS:(10019020)(4630300001)(6009001)(7916002)(189002)(377454003)(24454002)(199003)(50466002)(2950100001)(4326007)(305945005)(226693001)(76176999)(54356999)(7846002)(50986999)(5660300001)(106356001)(4001350100001)(97736004)(189998001)(110136002)(105586002)(7736002)(101416001)(42186005)(46406003)(98436002)(77096005)(92566002)(33656002)(23726003)(97756001)(9686002)(83506001)(1076002)(47776003)(6116002)(8676002)(19580395003)(586003)(81156014)(2906002)(81166006)(86362001)(575784001)(68736007)(18370500001)(3826002);DIR:OUT;SFP:1102;SCL:1;SRVR:CO1PR15MB0933;H:shli-mbp.local;FPR:;SPF:None;PTR:InfoNoRecords;MX:1;A:1;LANG:en; X-Microsoft-Exchange-Diagnostics: =?us-ascii?Q?1;CO1PR15MB0933;23:ljYTLPZTnCfoFvy++BGJP9osrg/a3Pt0sKwUmMaYm?= =?us-ascii?Q?ES0Q9+yTQBEUDXTjYnRI/QIvauqe4QfQfOeuy2WHZ0+S3m7jwwZ9OaG982mH?= =?us-ascii?Q?rMtG+GYqWNEsm9lKDyQ8A6vFR28boixr4mIxqCkUyQmN1E1d8Yv0iw9tGvhK?= =?us-ascii?Q?1LCRWY/isHlBRer0tsYH0NVMaZJU5uLMUNvjjULq45hGpRt07eQr6+BMi4Yu?= =?us-ascii?Q?PqJW7Ze77RLoONKKYaVo4Rtjmvi2BJ+dNcaANscDt3u6jxbSl9AwQY/Bi7Ji?= =?us-ascii?Q?0smvawuWGcgoE1vWeJ15L4+HezZ/TuuGMr0uVViq423jeOwvvjEEX8+mw3ga?= =?us-ascii?Q?ij0OP2QOEuHKvOXOw5pNt5Xmovo22jHroVc/kIr6VYehT+Nu7I18VvZ1CR8M?= =?us-ascii?Q?wLfZs6JPMkuGaszT5gQm5lyoNbA49QbScHRWhuo2RPEgSoI7uDxfxZURb8bO?= =?us-ascii?Q?yuePZiofutKX7bHr+KqL5nQDQsA6+78ms6zZ0MQzzyVkEtn3E/R8CzR+WBBX?= =?us-ascii?Q?bglIXhjKSPi598BAlWKTZHm78yn/I7wOhsiEQQJb5OF/B6KsWcofHTKIo02M?= =?us-ascii?Q?nqVb9t7Xd3IfSbv/dn8Hdy1A4VuqYPMdzqYxhi6fCWe6n1vqoXwO0N3aBtyV?= =?us-ascii?Q?jIvFrou0Oikjp/hYPBoocr0UaBC3xz4gSDl3Is6J+jLdr+fMFu7TvDoNizZw?= =?us-ascii?Q?yE3K1hbisfr632fIp+a2vnDc3HOkf2duLMFu2cY+YyRklHGUC77WmzTw2SEP?= =?us-ascii?Q?rFZpFGNsWFw8c3S/flVmhZRuVHs8hoMRYaQAqV2q4W83F9gRV2fcYSHKSneL?= =?us-ascii?Q?25cF9LWmPVBRj66cdj/CHAPzX6iXBDuOKovm287Mbp6AFQxOQ4y2SWcbM3Yc?= =?us-ascii?Q?M6uYOieeOqqlQAjI4wctzZp6mxRkBL2LlFn/h+yUFQNaC64A+FzY9GFudv38?= =?us-ascii?Q?CtQ1n+6Zc85kaW1Nv/tLEcVoidrHWNt/c40vVgY6gYdZkGQyoDJGcMaRp4we?= =?us-ascii?Q?lz7xUPH22dmJoNf7TwkV4Fu/t+JdzFRQFyjfuwNEWIQHJA6nUsPx8uealmRo?= =?us-ascii?Q?Z4QVI67NFKuDTHEePB1/0ouWF/7nVO0MAqoT1l59tzjcIl/VT9ccqhj4fsbI?= =?us-ascii?Q?yOjxPK5eNNiVXW9tPUUG4gv9eFS6s5/5d+bO6F7ZDJOhUh9XtUdle2xqJCZB?= =?us-ascii?Q?4J8Io/F6ZDswamwaguCFVJAvkBMYOABm9HtWFpJDlNFNleCm1Qn69ejMA=3D?= =?us-ascii?Q?=3D?= X-Microsoft-Exchange-Diagnostics: 1;CO1PR15MB0933;6:2fcHOl1pkmlSxXQ2C5yptwt22t0Wi/ixHvZU5Tu3PQvfoyYHjAxtAuLwbPWlVQvo2POErZNw0ckTuSUbN0nVIYELPgyTcaY7pDSDr9ycEpplp7+TPgxplu3n/A43m/caS98yfn4Yzy6yoP/adGX+UVvNNthsp7ui2qWheuVi9/p3gIZinCikp1EZIN7qGJ8FfRZy89cvU/nlDHx+WEeoikqjHiXOcFO11mud8LQV0uXWroUs6v0yigwq8ObTCSMeX2uNtI0K2GQQWoFHvveZbxyntkbFJaQ96aZe/SAIwZQ=;5:MZas3sWJpp90cFvkG7yTpiM5CrL1ocMh1VllWk6ocF0BNG4RfHEAQ4/642AoT7guSKvaneGnw+6kfZOckWcQcin54XkDrrDu2UBQunG3v0qNVI7S7cNDwRRFZbz3maDZFA9a5VTjhg6D7reADeRZjQ==;24:nyMqvPlNLbaRPW3gyGA3pKp5K5N/2FUY1VoTDIZyFl8+VVzhtZds3HzanNKfyzR432DJcQY7x2NusCaynO+YMsIrRVDhcIRochQyO0R4a0U=;7:pUZwI/xjjZkTFcZ4awhNV9RXFCsAqkI+0Um+qd5flsxLKOWruoSJiZTeSAOvlw2TsBVr2WlfxDPvt8x7M2YCzbp9sFzLUx7kKUJ5F63CzDx0cMQvq2OnMBW/dJd3fEzy4B2ickL9+3EtgaGBVGxB/881WI0bn1XXW4iZQnpZcY19FEclDqDNtHA3qap8zWJFmTk0z/83IfPyB2JpygX+RIcY+c+wcgAVSS7Mn9GnyVqfEMlFNb/tPtrkj4O3tpqt SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-Microsoft-Exchange-Diagnostics: 1;CO1PR15MB0933;20:BtdY8p9KJgx3Rh3ouiKr95hch7OpO0KkWAeqMh+7F12oPgXfgWdbL1slS7mW29qQo8YlqJPx0SVFsXEDxrAEl37a/vm1Cf1o9uXPDJU+IM81ziRSRiXPiGB8i5RBiTNLBOLxh9AipgFZilXk41wn+6COQQWMb7fsk5ZYcHfd5/I= X-MS-Exchange-CrossTenant-OriginalArrivalTime: 08 Sep 2016 17:33:37.7940 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-Transport-CrossTenantHeadersStamped: CO1PR15MB0933 X-OriginatorOrg: fb.com X-Proofpoint-Spam-Reason: safe X-FB-Internal: Safe X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10432:,, definitions=2016-09-08_08:,, signatures=0 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Thu, Sep 08, 2016 at 10:16:59AM -0600, Jens Axboe wrote: > On 09/08/2016 02:23 AM, Stefan Priebe - Profihost AG wrote: > > Hi, > > > > while trying Kernel 4.8-rc5 my raid5 breaks every few minutes. > > > > Trace: > > ------------[ cut here ]------------ > > kernel BUG at block/blk-core.c:2032! > > invalid opcode: 0000 [#1] SMP > > Modules linked in: netconsole ipt_REJECT nf_reject_ipv4 xt_multiport > > iptable_filter ip_tables x_tables 8021q garp bonding sb_edac edac_core > > x86_pkg_temp_thermal coretemp kvm_intel kvm i2c_i801 irqbypass i2c_smbus > > ipmi_si crc32_pclmul i2c_core ghash_clmulni_intel shpchp ipmi_msghandler > > button loop fuse btrfs dm_mod raid10 raid0 multipath linear raid456 > > async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq > > raid1 md_mod sg sd_mod ixgbe i40e mdio usbhid ehci_pci ehci_hcd ahci > > usbcore ptp libahci usb_common megaraid_sas pps_core > > CPU: 8 PID: 1105 Comm: md0_raid5 Not tainted 4.8.0-rc5-00003-g3abda5c #2 > > Hardware name: Supermicro X10DRH/X10DRH-iT, BIOS 1.0c 02/18/2015 > > task: ffff97de5e1e0000 task.stack: ffff97de597a0000 > > RIP: 0010:[] [] > > generic_make_request+0x1c0/0x1d0 > > RSP: 0018:ffff97de597a3aa0 EFLAGS: 00010286 > > RAX: ffff97de5e1e0000 RBX: ffff97dd227e5030 RCX: 0000000000000000 > > RDX: ffffffffc0000001 RSI: 0000000000000001 RDI: ffff97de5e7d9db8 > > RBP: ffff97de597a3ad8 R08: 0000000000000008 R09: 0000000000000000 > > R10: 0000000000000000 R11: 0000000000000001 R12: 00000000ffffffff > > R13: ffff97de5aa20c00 R14: 00000000000002f0 R15: ffff97e65dce0e00 > > FS: 0000000000000000(0000) GS:ffff97e67f200000(0000) knlGS:0000000000000000 > > CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > > CR2: 00007f0e4e1ec000 CR3: 0000000078c06000 CR4: 00000000001406e0Stack: > > ffff97de597a3b50 0000000000001000 0000000000000000 ffff97dd227e4c80 > > ffff97de5aa20c00 00000000000002f0 ffff97e65dce0e00 ffff97de597a3ba0 > > ffffffffc02595db ffffffffc025e04b 00000001597a3b01 0000000200000006 > > Call Trace: > > [] ops_run_io+0x3bb/0x990 [raid456] > > [] ? raid_run_ops+0xefb/0x1520 [raid456] > > [] handle_stripe+0x9a6/0x2280 [raid456] > > [] ? default_wake_function+0x12/0x20 > > [] ? autoremove_wake_function+0x12/0x40 > > [] handle_active_stripes.isra.54+0x193/0x4b0 [raid456] > > [] ? __release_stripe+0x15/0x20 [raid456] > > [] raid5d+0x4a9/0x740 [raid456] > > [] ? init_timer_key+0xa0/0xa0 > > [] md_thread+0x12b/0x130 [md_mod] > > [] ? wait_woken+0x90/0x90 > > [] ? find_pers+0x70/0x70 [md_mod] > > [] kthread+0xdb/0x100 > > [] ret_from_fork+0x1f/0x40 > > [] ? kthread_park+0x60/0x60 > > Code: bd 70 08 00 00 f0 49 83 ad 70 08 00 00 01 74 05 e9 5a ff ff ff 41 > > ff 95 80 08 00 00 e9 4e ff ff ff 48 c7 40 08 00 00 00 00 eb 8c <0f> 0b > > 66 66 66 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 > > RIP [] generic_make_request+0x1c0/0x1d0 > > RSP > > ---[ end trace 457dbe5e9cdd3473 ]--- > > CC'ing Shaohua - this is: > > BUG_ON(bio->bi_next); > > which doesn't look healthy. Hi Stefan, does below patch help? Looks there is a race condition introduced recently. diff --git a/drivers/md/raid5.c b/drivers/md/raid5.c index b95c54c..ee7fc37 100644 --- a/drivers/md/raid5.c +++ b/drivers/md/raid5.c @@ -2423,10 +2423,10 @@ static void raid5_end_read_request(struct bio * bi) } } rdev_dec_pending(rdev, conf->mddev); + bio_reset(bi); clear_bit(R5_LOCKED, &sh->dev[i].flags); set_bit(STRIPE_HANDLE, &sh->state); raid5_release_stripe(sh); - bio_reset(bi); } static void raid5_end_write_request(struct bio *bi) @@ -2498,6 +2498,7 @@ static void raid5_end_write_request(struct bio *bi) if (sh->batch_head && bi->bi_error && !replacement) set_bit(STRIPE_BATCH_ERR, &sh->batch_head->state); + bio_reset(bi); if (!test_and_clear_bit(R5_DOUBLE_LOCKED, &sh->dev[i].flags)) clear_bit(R5_LOCKED, &sh->dev[i].flags); set_bit(STRIPE_HANDLE, &sh->state); @@ -2505,7 +2506,6 @@ static void raid5_end_write_request(struct bio *bi) if (sh->batch_head && sh != sh->batch_head) raid5_release_stripe(sh->batch_head); - bio_reset(bi); } static void raid5_build_block(struct stripe_head *sh, int i, int previous)