mirror of https://lore.kernel.org/lkml/
 help / color / mirror / Atom feed
From: Luis Henriques <luis.henriques@canonical.com>
To: linux-kernel@vger.kernel.org, stable@vger.kernel.org,
	kernel-team@lists.ubuntu.com
Cc: Thomas Gleixner <tglx@linutronix.de>,
	Prarit Bhargava <prarit@redhat.com>,
	Richard Cochran <richardcochran@gmail.com>,
	Ingo Molnar <mingo@kernel.org>,
	Sasha Levin <sasha.levin@oracle.com>,
	John Stultz <john.stultz@linaro.org>,
	Luis Henriques <luis.henriques@canonical.com>
Subject: [PATCH 3.5 20/60] timekeeping: Avoid possible deadlock from clock_was_set_delayed
Date: Fri, 21 Feb 2014 13:00:47 +0000	[thread overview]
Message-ID: <1392987687-15367-21-git-send-email-luis.henriques@canonical.com> (raw)
In-Reply-To: <1392987687-15367-1-git-send-email-luis.henriques@canonical.com>

3.5.7.31 -stable review patch.  If anyone has any objections, please let me know.

------------------

From: John Stultz <john.stultz@linaro.org>

commit 6fdda9a9c5db367130cf32df5d6618d08b89f46a upstream.

As part of normal operaions, the hrtimer subsystem frequently calls
into the timekeeping code, creating a locking order of
  hrtimer locks -> timekeeping locks

clock_was_set_delayed() was suppoed to allow us to avoid deadlocks
between the timekeeping the hrtimer subsystem, so that we could
notify the hrtimer subsytem the time had changed while holding
the timekeeping locks. This was done by scheduling delayed work
that would run later once we were out of the timekeeing code.

But unfortunately the lock chains are complex enoguh that in
scheduling delayed work, we end up eventually trying to grab
an hrtimer lock.

Sasha Levin noticed this in testing when the new seqlock lockdep
enablement triggered the following (somewhat abrieviated) message:

[  251.100221] ======================================================
[  251.100221] [ INFO: possible circular locking dependency detected ]
[  251.100221] 3.13.0-rc2-next-20131206-sasha-00005-g8be2375-dirty #4053 Not tainted
[  251.101967] -------------------------------------------------------
[  251.101967] kworker/10:1/4506 is trying to acquire lock:
[  251.101967]  (timekeeper_seq){----..}, at: [<ffffffff81160e96>] retrigger_next_event+0x56/0x70
[  251.101967]
[  251.101967] but task is already holding lock:
[  251.101967]  (hrtimer_bases.lock#11){-.-...}, at: [<ffffffff81160e7c>] retrigger_next_event+0x3c/0x70
[  251.101967]
[  251.101967] which lock already depends on the new lock.
[  251.101967]
[  251.101967]
[  251.101967] the existing dependency chain (in reverse order) is:
[  251.101967]
-> #5 (hrtimer_bases.lock#11){-.-...}:
[snipped]
-> #4 (&rt_b->rt_runtime_lock){-.-...}:
[snipped]
-> #3 (&rq->lock){-.-.-.}:
[snipped]
-> #2 (&p->pi_lock){-.-.-.}:
[snipped]
-> #1 (&(&pool->lock)->rlock){-.-...}:
[  251.101967]        [<ffffffff81194803>] validate_chain+0x6c3/0x7b0
[  251.101967]        [<ffffffff81194d9d>] __lock_acquire+0x4ad/0x580
[  251.101967]        [<ffffffff81194ff2>] lock_acquire+0x182/0x1d0
[  251.101967]        [<ffffffff84398500>] _raw_spin_lock+0x40/0x80
[  251.101967]        [<ffffffff81153e69>] __queue_work+0x1a9/0x3f0
[  251.101967]        [<ffffffff81154168>] queue_work_on+0x98/0x120
[  251.101967]        [<ffffffff81161351>] clock_was_set_delayed+0x21/0x30
[  251.101967]        [<ffffffff811c4bd1>] do_adjtimex+0x111/0x160
[  251.101967]        [<ffffffff811e2711>] compat_sys_adjtimex+0x41/0x70
[  251.101967]        [<ffffffff843a4b49>] ia32_sysret+0x0/0x5
[  251.101967]
-> #0 (timekeeper_seq){----..}:
[snipped]
[  251.101967] other info that might help us debug this:
[  251.101967]
[  251.101967] Chain exists of:
  timekeeper_seq --> &rt_b->rt_runtime_lock --> hrtimer_bases.lock#11

[  251.101967]  Possible unsafe locking scenario:
[  251.101967]
[  251.101967]        CPU0                    CPU1
[  251.101967]        ----                    ----
[  251.101967]   lock(hrtimer_bases.lock#11);
[  251.101967]                                lock(&rt_b->rt_runtime_lock);
[  251.101967]                                lock(hrtimer_bases.lock#11);
[  251.101967]   lock(timekeeper_seq);
[  251.101967]
[  251.101967]  *** DEADLOCK ***
[  251.101967]
[  251.101967] 3 locks held by kworker/10:1/4506:
[  251.101967]  #0:  (events){.+.+.+}, at: [<ffffffff81154960>] process_one_work+0x200/0x530
[  251.101967]  #1:  (hrtimer_work){+.+...}, at: [<ffffffff81154960>] process_one_work+0x200/0x530
[  251.101967]  #2:  (hrtimer_bases.lock#11){-.-...}, at: [<ffffffff81160e7c>] retrigger_next_event+0x3c/0x70
[  251.101967]
[  251.101967] stack backtrace:
[  251.101967] CPU: 10 PID: 4506 Comm: kworker/10:1 Not tainted 3.13.0-rc2-next-20131206-sasha-00005-g8be2375-dirty #4053
[  251.101967] Workqueue: events clock_was_set_work

So the best solution is to avoid calling clock_was_set_delayed() while
holding the timekeeping lock, and instead using a flag variable to
decide if we should call clock_was_set() once we've released the locks.

This works for the case here, where the do_adjtimex() was the deadlock
trigger point. Unfortuantely, in update_wall_time() we still hold
the jiffies lock, which would deadlock with the ipi triggered by
clock_was_set(), preventing us from calling it even after we drop the
timekeeping lock. So instead call clock_was_set_delayed() at that point.

Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Prarit Bhargava <prarit@redhat.com>
Cc: Richard Cochran <richardcochran@gmail.com>
Cc: Ingo Molnar <mingo@kernel.org>
Cc: Sasha Levin <sasha.levin@oracle.com>
Reported-by: Sasha Levin <sasha.levin@oracle.com>
Tested-by: Sasha Levin <sasha.levin@oracle.com>
Signed-off-by: John Stultz <john.stultz@linaro.org>
[ luis: backported to 3.5: used John Stultz's backport to 3.4 kernel ]
Signed-off-by: Luis Henriques <luis.henriques@canonical.com>
---
 kernel/time/timekeeping.c | 12 ++++++++----
 1 file changed, 8 insertions(+), 4 deletions(-)

diff --git a/kernel/time/timekeeping.c b/kernel/time/timekeeping.c
index 8954990..a9ae4bd 100644
--- a/kernel/time/timekeeping.c
+++ b/kernel/time/timekeeping.c
@@ -986,7 +986,8 @@ static void timekeeping_adjust(s64 offset)
  *
  * Returns the unconsumed cycles.
  */
-static cycle_t logarithmic_accumulation(cycle_t offset, int shift)
+static cycle_t logarithmic_accumulation(cycle_t offset, int shift,
+							unsigned int *clock_set)
 {
 	u64 nsecps = (u64)NSEC_PER_SEC << timekeeper.shift;
 	u64 raw_nsecs;
@@ -1008,7 +1009,7 @@ static cycle_t logarithmic_accumulation(cycle_t offset, int shift)
 		timekeeper.xtime.tv_sec += leap;
 		timekeeper.wall_to_monotonic.tv_sec -= leap;
 		if (leap)
-			clock_was_set_delayed();
+			*clock_set = 1;
 	}
 
 	/* Accumulate raw time */
@@ -1040,6 +1041,7 @@ static void update_wall_time(void)
 	struct clocksource *clock;
 	cycle_t offset;
 	int shift = 0, maxshift;
+	unsigned int clock_set = 0;
 	unsigned long flags;
 
 	write_seqlock_irqsave(&timekeeper.lock, flags);
@@ -1076,7 +1078,7 @@ static void update_wall_time(void)
 	maxshift = (64 - (ilog2(ntp_tick_length())+1)) - 1;
 	shift = min(shift, maxshift);
 	while (offset >= timekeeper.cycle_interval) {
-		offset = logarithmic_accumulation(offset, shift);
+		offset = logarithmic_accumulation(offset, shift, &clock_set);
 		if(offset < timekeeper.cycle_interval<<shift)
 			shift--;
 	}
@@ -1130,7 +1132,7 @@ static void update_wall_time(void)
 		timekeeper.xtime.tv_sec += leap;
 		timekeeper.wall_to_monotonic.tv_sec -= leap;
 		if (leap)
-			clock_was_set_delayed();
+			clock_set = 1;
 	}
 
 	timekeeping_update(false);
@@ -1138,6 +1140,8 @@ static void update_wall_time(void)
 out:
 	write_sequnlock_irqrestore(&timekeeper.lock, flags);
 
+	if (clock_set)
+		clock_was_set_delayed();
 }
 
 /**
-- 
1.9.0


  parent reply	other threads:[~2014-02-21 13:02 UTC|newest]

Thread overview: 63+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2014-02-21 13:00 [3.5.y.z extended stable] Linux 3.5.7.31 stable review Luis Henriques
2014-02-21 13:00 ` [PATCH 3.5 01/60] PCI: Enable ARI if dev and upstream bridge support it; disable otherwise Luis Henriques
2014-02-21 13:00 ` [PATCH 3.5 02/60] crypto: s390 - fix concurrency issue in aes-ctr mode Luis Henriques
2014-02-21 13:00 ` [PATCH 3.5 03/60] crypto: s390 - fix des and des3_ede cbc concurrency issue Luis Henriques
2014-02-21 13:00 ` [PATCH 3.5 04/60] crypto: s390 - fix des and des3_ede ctr " Luis Henriques
2014-02-21 13:00 ` [PATCH 3.5 05/60] ALSA: usb-audio: Add missing kconfig dependecy Luis Henriques
2014-02-21 13:00 ` [PATCH 3.5 06/60] [media] mxl111sf: Fix unintentional garbage stack read Luis Henriques
2014-02-21 13:00 ` [PATCH 3.5 07/60] [media] mxl111sf: Fix compile when CONFIG_DVB_USB_MXL111SF is unset Luis Henriques
2014-02-21 13:00 ` [PATCH 3.5 08/60] drm/mgag200: fix typo causing bw limits to be ignored on some chips Luis Henriques
2014-02-21 13:00 ` [PATCH 3.5 09/60] ALSA: hda - Improve loopback path lookups for AD1983 Luis Henriques
2014-02-21 13:00 ` [PATCH 3.5 10/60] SELinux: Fix kernel BUG on empty security contexts Luis Henriques
2014-02-21 13:00 ` [PATCH 3.5 11/60] pinctrl: do not init debugfs entries for unimplemented functionalities Luis Henriques
2014-02-25  9:24   ` Linus Walleij
2014-02-25  9:38     ` Luís Henriques
2014-02-21 13:00 ` [PATCH 3.5 12/60] x86, hweight: Fix BUG when booting with CONFIG_GCOV_PROFILE_ALL=y Luis Henriques
2014-02-21 13:00 ` [PATCH 3.5 13/60] mm/swap: fix race on swap_info reuse between swapoff and swapon Luis Henriques
2014-02-21 13:00 ` [PATCH 3.5 14/60] mm: __set_page_dirty_nobuffers() uses spin_lock_irqsave() instead of spin_lock_irq() Luis Henriques
2014-02-21 13:00 ` [PATCH 3.5 15/60] mm: __set_page_dirty uses spin_lock_irqsave instead of spin_lock_irq Luis Henriques
2014-02-21 13:00 ` [PATCH 3.5 16/60] sched,rt: fix isolated CPUs leaving root_task_group indefinitely throttled Luis Henriques
2014-02-21 13:00 ` [PATCH 3.5 17/60] sched: Unthrottle rt runqueues in __disable_runtime() Luis Henriques
2014-02-21 13:00 ` [PATCH 3.5 18/60] sched/rt: Avoid updating RT entry timeout twice within one tick period Luis Henriques
2014-02-21 13:00 ` [PATCH 3.5 19/60] rtc-cmos: Add an alarm disable quirk Luis Henriques
2014-02-21 13:00 ` Luis Henriques [this message]
2014-02-21 13:00 ` [PATCH 3.5 21/60] timekeeping: fix 32-bit overflow in get_monotonic_boottime Luis Henriques
2014-02-21 13:00 ` [PATCH 3.5 22/60] ahci: Use PCI_VENDOR_ID_MARVELL_EXT for 0x1b4b Luis Henriques
2014-02-21 13:00 ` [PATCH 3.5 23/60] ahci: add an observed PCI ID for Marvell 88se9172 SATA controller Luis Henriques
2014-02-21 13:00 ` [PATCH 3.5 24/60] ftrace: Synchronize setting function_trace_op with ftrace_trace_function Luis Henriques
2014-02-21 13:00 ` [PATCH 3.5 25/60] ftrace: Fix synchronization location disabling and freeing ftrace_ops Luis Henriques
2014-02-21 13:00 ` [PATCH 3.5 26/60] ftrace: Have function graph only trace based on global_ops filters Luis Henriques
2014-02-21 13:00 ` [PATCH 3.5 27/60] printk: Fix scheduling-while-atomic problem in console_cpu_notify() Luis Henriques
2014-02-21 13:00 ` [PATCH 3.5 28/60] staging:iio:ad799x fix error_free_irq which was freeing an irq that may not have been requested Luis Henriques
2014-02-21 13:00 ` [PATCH 3.5 29/60] xhci: Fix resume issues on Renesas chips in Samsung laptops Luis Henriques
2014-02-21 13:00 ` [PATCH 3.5 30/60] KVM: return an error code in kvm_vm_ioctl_register_coalesced_mmio() Luis Henriques
2014-02-21 13:00 ` [PATCH 3.5 31/60] power: max17040: Fix NULL pointer dereference when there is no platform_data Luis Henriques
2014-02-21 13:00 ` [PATCH 3.5 32/60] s390/dump: Fix dump memory detection Luis Henriques
2014-02-21 13:01 ` [PATCH 3.5 33/60] ath9k_htc: Do not support PowerSave by default Luis Henriques
2014-02-21 13:01 ` [PATCH 3.5 34/60] ath9k: " Luis Henriques
2014-02-21 13:01 ` [PATCH 3.5 35/60] spi: nuc900: Set SPI_LSB_FIRST for master->mode_bits if hw->pdata->lsb is true Luis Henriques
2014-02-21 13:01 ` [PATCH 3.5 36/60] usb: ftdi_sio: add Mindstorms EV3 console adapter Luis Henriques
2014-02-21 13:01 ` [PATCH 3.5 37/60] usb-storage: restrict bcdDevice range for Super Top in Cypress ATACB Luis Henriques
2014-02-21 13:01 ` [PATCH 3.5 38/60] usb-storage: add unusual-devs entry for BlackBerry 9000 Luis Henriques
2014-02-21 13:01 ` [PATCH 3.5 39/60] usb-storage: enable multi-LUN scanning when needed Luis Henriques
2014-02-21 13:01 ` [PATCH 3.5 40/60] USB: ftdi_sio: add Tagsys RFID Reader IDs Luis Henriques
2014-02-21 13:01 ` [PATCH 3.5 41/60] mac80211: fix fragmentation code, particularly for encryption Luis Henriques
2014-02-21 13:01 ` [PATCH 3.5 42/60] time: Fix overflow when HZ is smaller than 60 Luis Henriques
2014-02-21 13:01 ` [PATCH 3.5 43/60] VME: Correct read/write alignment algorithm Luis Henriques
2014-02-21 13:01 ` [PATCH 3.5 44/60] raw: test against runtime value of max_raw_minors Luis Henriques
2014-02-21 13:01 ` [PATCH 3.5 45/60] tty: n_gsm: Fix for modems with brk in modem status control Luis Henriques
2014-02-21 13:01 ` [PATCH 3.5 46/60] staging: comedi: adv_pci1710: fix analog output readback value Luis Henriques
2014-02-21 13:01 ` [PATCH 3.5 47/60] xen-blkfront: handle backend CLOSED without CLOSING Luis Henriques
2014-02-21 13:01 ` [PATCH 3.5 48/60] Modpost: fixed USB alias generation for ranges including 0x9 and 0xA Luis Henriques
2014-02-21 13:01 ` [PATCH 3.5 49/60] iio: ak8975: Fix calculation formula for convert micro tesla to gauss unit Luis Henriques
2014-02-21 13:01 ` [PATCH 3.5 50/60] fs/file.c:fdtable: avoid triggering OOMs from alloc_fdmem Luis Henriques
2014-02-21 13:01 ` [PATCH 3.5 51/60] genirq: Add missing irq_to_desc export for CONFIG_SPARSE_IRQ=n Luis Henriques
2014-02-21 13:01 ` [PATCH 3.5 52/60] ring-buffer: Fix first commit on sub-buffer having non-zero delta Luis Henriques
2014-02-21 13:01 ` [PATCH 3.5 53/60] usb: option: blacklist ZTE MF667 net interface Luis Henriques
2014-02-21 13:01 ` [PATCH 3.5 54/60] md/raid5: Fix CPU hotplug callback registration Luis Henriques
2014-02-21 13:01 ` [PATCH 3.5 55/60] lockd: send correct lock when granting a delayed lock Luis Henriques
2014-02-21 13:01 ` [PATCH 3.5 56/60] IB/qib: Add missing serdes init sequence Luis Henriques
2014-02-21 13:01 ` [PATCH 3.5 57/60] EDAC: Correct workqueue setup path Luis Henriques
2014-02-21 13:01 ` [PATCH 3.5 58/60] dm sysfs: fix a module unload race Luis Henriques
2014-02-21 13:01 ` [PATCH 3.5 59/60] s390: fix kernel crash due to linkage stack instructions Luis Henriques
2014-02-21 13:01 ` [PATCH 3.5 60/60] block: add cond_resched() to potentially long running ioctl discard loop Luis Henriques

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=1392987687-15367-21-git-send-email-luis.henriques@canonical.com \
    --to=luis.henriques@canonical.com \
    --cc=john.stultz@linaro.org \
    --cc=kernel-team@lists.ubuntu.com \
    --cc=linux-kernel@vger.kernel.org \
    --cc=mingo@kernel.org \
    --cc=prarit@redhat.com \
    --cc=richardcochran@gmail.com \
    --cc=sasha.levin@oracle.com \
    --cc=stable@vger.kernel.org \
    --cc=tglx@linutronix.de \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox

all inboxes | Powered by JetHome®