Re: [PATCH 00/17] Backport rt/deadline crash and the ardous story of FUTEX_UNLOCK_PI to 4.4

[Henrik Austad <henrik@austad.us>]

[-- Attachment #1: Type: text/plain, Size: 5702 bytes --]

On Fri, Nov 09, 2018 at 11:07:28AM +0100, Henrik Austad wrote:
> From: Henrik Austad <haustad@cisco.com>
> 
> Short story:

Sorry for the spam, it looks like I was not very specific in /which/ 
version I targeted this to, as well as not providing a full Cc-list for the 
cover-letter.

The series is targeted at stable v4.4.162.

Expanding Cc-list to those missing from the first attempt.

-Henrik

> The following patches are needed on a 4.4 kernel to avoid
> Oops in the scheduler when a sched_rr and sched_deadline task contends
> on the same futex (with PI).
> 
> Longer story:
> 
> On one of our arm64 systems, we occasionally crash with an Oops in the
> scheduler with the following backtrace.
> 
> [<ffffffc0000ee398>] enqueue_task_dl+0x1f0/0x420
> [<ffffffc0000d0f14>] activate_task+0x7c/0x90
> [<ffffffc0000edbdc>] push_dl_task+0x164/0x1c8
> [<ffffffc0000edc60>] push_dl_tasks+0x20/0x30
> [<ffffffc0000cc00c>] __balance_callback+0x44/0x68
> [<ffffffc000d2c018>] __schedule+0x6f0/0x728
> [<ffffffc000d2c278>] schedule+0x78/0x98
> [<ffffffc000d2e76c>] __rt_mutex_slowlock+0x9c/0x108
> [<ffffffc000d2e9d0>] rt_mutex_slowlock+0xd8/0x198
> [<ffffffc0000f7f28>] rt_mutex_timed_futex_lock+0x30/0x40
> [<ffffffc00012c1a8>] futex_lock_pi+0x200/0x3b0
> [<ffffffc00012cf84>] do_futex+0x1c4/0x550
> [<ffffffc00012d92c>] compat_SyS_futex+0x10c/0x138
> [<ffffffc00008504c>] __sys_trace_return+0x0/0x4
> 
> This seems to be the same bug Xuneli Pang triggered and fixed in
> e96a7705e7d3 "sched/rtmutex/deadline: Fix a PI crash for deadline
> tasks". As noted by Peter Zijlstra in the previous attempt, this fix
> requires a few other patches, most notably the FUTEX_UNLOCK_PI series
> [1]
> 
> Testing this on a dual-core VM I have not been able to reproduce the
> same crash, but pi_stress (part of the rt-test suite) reveals that
> vanilla 4.4.162 behaves rather badly with a mix of deadline and
> sched_(rr|fifo) tasks:
> 
> time pi_stress --rr --mlockall --sched id=high,policy=deadline,runtime=100000,deadline=200000,period=200000
> Starting PI Stress Test
> Number of thread groups: 1
> Duration of test run: infinite
> Number of inversions per group: unlimited
>      Admin thread SCHED_RR priority 4
> 1 groups of 3 threads will be created
>       High thread SCHED_DEADLINE runtime 100000 deadline 200000 period 200000
>        Med thread SCHED_RR priority 2
>        Low thread SCHED_RR priority 1
> Current Inversions: 141627
> WATCHDOG triggered: group 0 is deadlocked!
> reporter stopping due to watchdog event
> Stopping test
> Terminated
> 
> real    0m26.291s
> user    0m0.148s
> sys     0m18.819s
> 
> With this series applied, the test ran for ~4.5 hours and again for 129
> minutes (when I remembered to time it) before crashing:
> 
> time pi_stress --rr --mlockall --sched id=high,policy=deadline,runtime=100000,deadline=200000,period=200000
> Starting PI Stress Test
> Number of thread groups: 1
> Duration of test run: infinite
> Number of inversions per group: unlimited
>      Admin thread SCHED_RR priority 4
> 1 groups of 3 threads will be created
>       High thread SCHED_DEADLINE runtime 100000 deadline 200000 period 200000
>        Med thread SCHED_RR priority 2
>        Low thread SCHED_RR priority 1
> Current Inversions: 51985223
> WATCHDOG triggered: group 0 is deadlocked!
> reporter stopping due to watchdog event
> Stopping test
> Terminated
> 
> real    129m38.807s
> user    0m59.084s
> sys     109m53.666s
> 
> 
> So clearly not perfect, but a *lot* better.
> 
> The same series on our vendor-4.4 kernel moves pi_stress up from ~30
> seconds before deadlock up to the same level as the VM (the test is
> still going as of this writing).
> 
> I suspect other users of 4.4 would benefit from having these patches
> backported, so tag them for stable. I assume 4.9 and 4.14 could benefit
> as well, but I have not had time to look into those.
> 
> 1) https://www.mail-archive.com/linux-kernel@vger.kernel.org/msg1359667.html
> 
> Peter Zijlstra (13):
>   futex: Cleanup variable names for futex_top_waiter()
>   futex: Use smp_store_release() in mark_wake_futex()
>   futex: Remove rt_mutex_deadlock_account_*()
>   futex,rt_mutex: Provide futex specific rt_mutex API
>   futex: Change locking rules
>   futex: Cleanup refcounting
>   futex: Rework inconsistent rt_mutex/futex_q state
>   futex: Pull rt_mutex_futex_unlock() out from under hb->lock
>   futex,rt_mutex: Introduce rt_mutex_init_waiter()
>   futex,rt_mutex: Restructure rt_mutex_finish_proxy_lock()
>   futex: Rework futex_lock_pi() to use rt_mutex_*_proxy_lock()
>   futex: Futex_unlock_pi() determinism
>   futex: Drop hb->lock before enqueueing on the rtmutex
> 
> Thomas Gleixner (2):
>   rtmutex: Make wait_lock irq safe
>   futex: Rename free_pi_state() to put_pi_state()
> 
> Xunlei Pang (2):
>   rtmutex: Deboost before waking up the top waiter
>   sched/rtmutex/deadline: Fix a PI crash for deadline tasks
> 
>  include/linux/init_task.h       |   1 +
>  include/linux/sched.h           |   2 +
>  include/linux/sched/rt.h        |   1 +
>  kernel/fork.c                   |   1 +
>  kernel/futex.c                  | 532 ++++++++++++++++++++++++++--------------
>  kernel/locking/rtmutex-debug.c  |   9 -
>  kernel/locking/rtmutex-debug.h  |   3 -
>  kernel/locking/rtmutex.c        | 406 ++++++++++++++++++------------
>  kernel/locking/rtmutex.h        |   2 -
>  kernel/locking/rtmutex_common.h |  24 +-
>  kernel/sched/core.c             |   2 +
>  11 files changed, 620 insertions(+), 363 deletions(-)
> 
> -- 
> 2.7.4
> 

-- 
Henrik Austad

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 181 bytes --]