From: Thomas Gleixner <tglx@linutronix.de>
To: LKML <linux-kernel@vger.kernel.org>
Cc: Peter Zijlstra <peterz@infradead.org>,
Ingo Molnar <mingo@kernel.org>,
Paul McKenney <paulmck@linux.vnet.ibm.com>,
Sebastian Siewior <bigeasy@linutronix.de>
Subject: [patch 7/8] sched: Make cond_resched_lock() variants RT aware
Date: Thu, 23 Sep 2021 18:54:44 +0200 (CEST) [thread overview]
Message-ID: <20210923165358.305969211@linutronix.de> (raw)
In-Reply-To: <20210923164145.466686140@linutronix.de>
The __might_resched() checks in the cond_resched_lock() variants use
PREEMPT_LOCK_OFFSET for preempt count offset checking which takes the
preemption disable by the spin_lock() which is still held at that point
into account.
On PREEMPT_RT enabled kernels spin/rw_lock held sections stay preemptible
which means PREEMPT_LOCK_OFFSET is 0, but that still triggers the
__might_resched() check because that takes RCU read side nesting into
account.
On RT enabled kernels spin/read/write_lock() issue rcu_read_lock() to
resemble the !RT semantics, which means in cond_resched_lock() the might
resched check will see preempt_count() == 0 and rcu_preempt_depth() == 1.
Introduce PREEMPT_LOCK_SCHED_OFFSET for those might resched checks and map
them depending on CONFIG_PREEMPT_RT.
Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
---
include/linux/preempt.h | 5 +++--
include/linux/sched.h | 34 +++++++++++++++++++++++++---------
2 files changed, 28 insertions(+), 11 deletions(-)
--- a/include/linux/preempt.h
+++ b/include/linux/preempt.h
@@ -122,9 +122,10 @@
* The preempt_count offset after spin_lock()
*/
#if !defined(CONFIG_PREEMPT_RT)
-#define PREEMPT_LOCK_OFFSET PREEMPT_DISABLE_OFFSET
+#define PREEMPT_LOCK_OFFSET PREEMPT_DISABLE_OFFSET
#else
-#define PREEMPT_LOCK_OFFSET 0
+/* Locks on RT do not disable preemption */
+#define PREEMPT_LOCK_OFFSET 0
#endif
/*
--- a/include/linux/sched.h
+++ b/include/linux/sched.h
@@ -2052,19 +2052,35 @@ extern int __cond_resched_rwlock_write(r
#define MIGHT_RESCHED_RCU_SHIFT 8
#define MIGHT_RESCHED_PREEMPT_MASK ((1U << MIGHT_RESCHED_RCU_SHIFT) - 1)
-#define cond_resched_lock(lock) ({ \
- __might_resched(__FILE__, __LINE__, PREEMPT_LOCK_OFFSET); \
- __cond_resched_lock(lock); \
+#ifndef CONFIG_PREEMPT_RT
+/*
+ * Non RT kernels have an elevated preempt count due to the held lock,
+ * but are not allowed to be inside a RCU read side critical section
+ */
+# define PREEMPT_LOCK_RESCHED_OFFSETS PREEMPT_LOCK_OFFSET
+#else
+/*
+ * spin/rw_lock() on RT implies rcu_read_lock(). The might_sleep() check in
+ * cond_resched*lock() has to take that into account because it checks for
+ * preempt_count() and rcu_preempt_depth().
+ */
+# define PREEMPT_LOCK_RESCHED_OFFSETS \
+ (PREEMPT_LOCK_OFFSET + (1U << MIGHT_RESCHED_RCU_SHIFT))
+#endif
+
+#define cond_resched_lock(lock) ({ \
+ __might_resched(__FILE__, __LINE__, PREEMPT_LOCK_RESCHED_OFFSETS); \
+ __cond_resched_lock(lock); \
})
-#define cond_resched_rwlock_read(lock) ({ \
- __might_resched(__FILE__, __LINE__, PREEMPT_LOCK_OFFSET); \
- __cond_resched_rwlock_read(lock); \
+#define cond_resched_rwlock_read(lock) ({ \
+ __might_resched(__FILE__, __LINE__, PREEMPT_LOCK_RESCHED_OFFSETS); \
+ __cond_resched_rwlock_read(lock); \
})
-#define cond_resched_rwlock_write(lock) ({ \
- __might_resched(__FILE__, __LINE__, PREEMPT_LOCK_OFFSET); \
- __cond_resched_rwlock_write(lock); \
+#define cond_resched_rwlock_write(lock) ({ \
+ __might_resched(__FILE__, __LINE__, PREEMPT_LOCK_RESCHED_OFFSETS); \
+ __cond_resched_rwlock_write(lock); \
})
static inline void cond_resched_rcu(void)
next prev parent reply other threads:[~2021-09-23 16:55 UTC|newest]
Thread overview: 17+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-09-23 16:54 [patch 0/8] sched: Clean up might_sleep() and make it " Thomas Gleixner
2021-09-23 16:54 ` [patch 1/8] sched: Clean up the might_sleep() underscore zoo Thomas Gleixner
2021-10-01 15:05 ` [tip: locking/core] " tip-bot2 for Thomas Gleixner
2021-09-23 16:54 ` [patch 2/8] sched: Make cond_resched_*lock() variants consistent vs. might_sleep() Thomas Gleixner
2021-10-01 15:05 ` [tip: locking/core] " tip-bot2 for Thomas Gleixner
2021-09-23 16:54 ` [patch 3/8] sched: Remove preempt_offset argument from __might_sleep() Thomas Gleixner
2021-10-01 15:05 ` [tip: locking/core] " tip-bot2 for Thomas Gleixner
2021-09-23 16:54 ` [patch 4/8] sched: Cleanup might_sleep() printks Thomas Gleixner
2021-10-01 15:05 ` [tip: locking/core] " tip-bot2 for Thomas Gleixner
2021-09-23 16:54 ` [patch 5/8] sched: Make might_sleep() output less confusing Thomas Gleixner
2021-10-01 15:05 ` [tip: locking/core] " tip-bot2 for Thomas Gleixner
2021-09-23 16:54 ` [patch 6/8] sched: Make RCU nest depth distinct in __might_resched() Thomas Gleixner
2021-10-01 15:05 ` [tip: locking/core] " tip-bot2 for Thomas Gleixner
2021-09-23 16:54 ` Thomas Gleixner [this message]
2021-10-01 15:05 ` [tip: locking/core] sched: Make cond_resched_lock() variants RT aware tip-bot2 for Thomas Gleixner
2021-09-23 16:54 ` [patch 8/8] locking/rt: Take RCU nesting into account for __might_resched() Thomas Gleixner
2021-10-01 15:05 ` [tip: locking/core] " tip-bot2 for Thomas Gleixner
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20210923165358.305969211@linutronix.de \
--to=tglx@linutronix.de \
--cc=bigeasy@linutronix.de \
--cc=linux-kernel@vger.kernel.org \
--cc=mingo@kernel.org \
--cc=paulmck@linux.vnet.ibm.com \
--cc=peterz@infradead.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®