From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S933582AbcALOxx (ORCPT ); Tue, 12 Jan 2016 09:53:53 -0500 Received: from arcturus.aphlor.org ([188.246.204.175]:33172 "EHLO arcturus.aphlor.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752926AbcALOxv (ORCPT ); Tue, 12 Jan 2016 09:53:51 -0500 Date: Tue, 12 Jan 2016 09:53:48 -0500 From: Dave Jones To: Linux Kernel Cc: linux-ext4@vger.kernel.org Subject: ext4 unkillable lseek. Message-ID: <20160112145348.GA15634@codemonkey.org.uk> Mail-Followup-To: Dave Jones , Linux Kernel , linux-ext4@vger.kernel.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.5.24 (2015-08-30) X-Spam-Score: -2.9 (--) X-Spam-Report: Spam detection software, running on the system "arcturus.aphlor.org", has NOT identified this incoming email as spam. The original message has been attached to this so you can view it or label similar future email. If you have any questions, see the administrator of that system for details. Content preview: I was investigating a case where it looked like Trinity was getting into a deadlock. The running task is doing an lseek(fd, , SEEK_DATA) on a sparse file that looks like this.. $ ll trinity-testfile4 --wxrwx--- 1 davej davej 4947802326691 Jan 12 09:14 trinity-testfile4* $ sudo filefrag trinity-testfile4 trinity-testfile4: 3 extents found [...] Content analysis details: (-2.9 points, 5.0 required) pts rule name description ---- ---------------------- -------------------------------------------------- -1.0 ALL_TRUSTED Passed through trusted hosts only via SMTP -1.9 BAYES_00 BODY: Bayes spam probability is 0 to 1% [score: 0.0000] Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org I was investigating a case where it looked like Trinity was getting into a deadlock. The running task is doing an lseek(fd, , SEEK_DATA) on a sparse file that looks like this.. $ ll trinity-testfile4 --wxrwx--- 1 davej davej 4947802326691 Jan 12 09:14 trinity-testfile4* $ sudo filefrag trinity-testfile4 trinity-testfile4: 3 extents found The kernel trace for that process looks like.. trinity-c11 R running task 22192 11483 2439 0x00080004 ffff8800428a7c98 ffff8800a2ef87dc ffff8800a3bdf758 ffff8800a3bdf730 ffff8800a2ef8008 ffff8800a2ef8340 ffff88009f8e9980 ffff8800a2ef8000 ffff8800428a0000 ffffed0008514001 ffff8800428a0008 ffff8800935499e0 Call Trace: [] preempt_schedule_common+0x42/0x70 [] preempt_schedule+0x1f/0x30 [] ___preempt_schedule+0x12/0x14 [] ? ext4_es_find_delayed_extent_range+0x2a0/0x780 [] ? _raw_read_unlock+0x31/0x50 [] ? _raw_read_unlock+0x44/0x50 [] ext4_es_find_delayed_extent_range+0x2a0/0x780 [] ext4_llseek+0x567/0x870 [] ? ext4_find_unwritten_pgoff.isra.12+0x790/0x790 [] ? mutex_lock_nested+0x51c/0x8e0 [] ? trace_hardirqs_on_caller+0x3f9/0x580 [] ? __fdget_pos+0xd5/0x110 [] ? trace_hardirqs_on+0xd/0x10 [] ? mutex_lock_interruptible_nested+0x9f0/0x9f0 [] ? enter_from_user_mode+0x1f/0x50 [] ? syscall_trace_enter_phase1+0x278/0x470 [] ? debug_lockdep_rcu_enabled+0x77/0x90 [] SyS_lseek+0x10d/0x180 [] entry_SYSCALL_64_fastpath+0x12/0x6b It's currently been running for a hour. Even though it's preempting back to userspace, it's ignoring all the SIGKILLs that trinity has been sending it for taking too long. Meanwhile all the other processes are backing up on the f_pos lock. trinity-c7 D ffff880066857d50 24240 11628 2439 0x00080004 ffff880066857d50 0000000000000007 ffff8800a3bdf758 ffff8800a3bdf730 ffff880045286608 ffff880045286940 ffff8800a0150000 ffff880045286600 ffff880066850000 ffffed000cd0a001 ffff880066850008 dffffc0000000000 Call Trace: [] schedule+0x9f/0x1c0 [] schedule_preempt_disabled+0x18/0x30 [] mutex_lock_nested+0x34d/0x8e0 [] ? __fdget_pos+0xd5/0x110 [] ? acct_account_cputime+0x63/0x80 [] ? __fdget_pos+0xd5/0x110 [] ? mutex_lock_interruptible_nested+0x9f0/0x9f0 [] ? debug_lockdep_rcu_enabled+0x77/0x90 [] __fdget_pos+0xd5/0x110 [] SyS_read+0x79/0x230 [] ? do_sendfile+0x1280/0x1280 [] ? trace_hardirqs_on_caller+0x3f9/0x580 [] ? trace_hardirqs_on_thunk+0x17/0x19 [] entry_SYSCALL_64_fastpath+0x12/0x6b Eventually it does complete, but waiting a half hour every time trinity picks lseek as a syscall is kinda crappy. Shouldn't lseek be a killable operation ? I notice this doesn't seem to happen with btrfs, suggesting it's an ext'ism. This has probably been there for a while, I've not been doing fuzz runs on ext4 enabled systems for a long time. Dave