From: "Paul E. McKenney" <paulmck@kernel.org>
To: rcu@vger.kernel.org
Cc: linux-kernel@vger.kernel.org, kernel-team@fb.com,
rostedt@goodmis.org, "Paul E. McKenney" <paulmck@kernel.org>,
Neeraj Upadhyay <quic_neeraju@quicinc.com>
Subject: [PATCH rcu 06/21] srcu: Add size-state transitioning code
Date: Mon, 18 Apr 2022 17:03:07 -0700 [thread overview]
Message-ID: <20220419000322.3948903-6-paulmck@kernel.org> (raw)
In-Reply-To: <20220419000315.GA3948789@paulmck-ThinkPad-P17-Gen-1>
This is just dead code at the moment, and will be used once
the state-transition code is activated.
Because srcu_barrier() must be aware of transition before call_srcu(), the
state machine waits for an SRCU grace period before callbacks are queued
to the non-CPU-0 queues. This requres that portions of srcu_barrier()
be enclosed in an SRCU read-side critical section.
Co-developed-by: Neeraj Upadhyay <quic_neeraju@quicinc.com>
Signed-off-by: Neeraj Upadhyay <quic_neeraju@quicinc.com>
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
---
kernel/rcu/srcutree.c | 13 +++++++++++++
1 file changed, 13 insertions(+)
diff --git a/kernel/rcu/srcutree.c b/kernel/rcu/srcutree.c
index e98cc218e42b..00f9aed6f7b9 100644
--- a/kernel/rcu/srcutree.c
+++ b/kernel/rcu/srcutree.c
@@ -562,6 +562,7 @@ static void srcu_gp_end(struct srcu_struct *ssp)
unsigned long mask;
struct srcu_data *sdp;
struct srcu_node *snp;
+ int ss_state;
/* Prevent more than one additional grace period. */
mutex_lock(&ssp->srcu_cb_mutex);
@@ -629,6 +630,15 @@ static void srcu_gp_end(struct srcu_struct *ssp)
} else {
spin_unlock_irq_rcu_node(ssp);
}
+
+ /* Transition to big if needed. */
+ ss_state = smp_load_acquire(&ssp->srcu_size_state);
+ if (ss_state != SRCU_SIZE_SMALL && ss_state != SRCU_SIZE_BIG) {
+ if (ss_state == SRCU_SIZE_ALLOC)
+ init_srcu_struct_nodes(ssp);
+ else
+ smp_store_release(&ssp->srcu_size_state, ss_state + 1);
+ }
}
/*
@@ -1178,6 +1188,7 @@ static void srcu_barrier_one_cpu(struct srcu_struct *ssp, struct srcu_data *sdp)
void srcu_barrier(struct srcu_struct *ssp)
{
int cpu;
+ int idx;
unsigned long s = rcu_seq_snap(&ssp->srcu_barrier_seq);
check_init_srcu_struct(ssp);
@@ -1193,11 +1204,13 @@ void srcu_barrier(struct srcu_struct *ssp)
/* Initial count prevents reaching zero until all CBs are posted. */
atomic_set(&ssp->srcu_barrier_cpu_cnt, 1);
+ idx = srcu_read_lock(ssp);
if (smp_load_acquire(&ssp->srcu_size_state) < SRCU_SIZE_WAIT_BARRIER)
srcu_barrier_one_cpu(ssp, per_cpu_ptr(ssp->sda, 0));
else
for_each_possible_cpu(cpu)
srcu_barrier_one_cpu(ssp, per_cpu_ptr(ssp->sda, cpu));
+ srcu_read_unlock(ssp, idx);
/* Remove the initial count, at which point reaching zero can happen. */
if (atomic_dec_and_test(&ssp->srcu_barrier_cpu_cnt))
--
2.31.1.189.g2e36527f23
next prev parent reply other threads:[~2022-04-19 0:03 UTC|newest]
Thread overview: 22+ messages / expand[flat|nested] mbox.gz Atom feed top
2022-04-19 0:03 [PATCH rcu 0/21] SRCU updates for v5.19 Paul E. McKenney
2022-04-19 0:03 ` [PATCH rcu 01/21] srcu: Tighten cleanup_srcu_struct() GP checks Paul E. McKenney
2022-04-19 0:03 ` [PATCH rcu 02/21] srcu: Fix s/is/if/ typo in srcu_node comment Paul E. McKenney
2022-04-19 0:03 ` [PATCH rcu 03/21] srcu: Make srcu_funnel_gp_start() cache ->mynode in snp_leaf Paul E. McKenney
2022-04-19 0:03 ` [PATCH rcu 04/21] srcu: Make Tree SRCU able to operate without snp_node array Paul E. McKenney
2022-04-19 0:03 ` [PATCH rcu 05/21] srcu: Dynamically allocate srcu_node array Paul E. McKenney
2022-04-19 0:03 ` Paul E. McKenney [this message]
2022-04-19 0:03 ` [PATCH rcu 07/21] srcu: Make rcutorture dump the SRCU size state Paul E. McKenney
2022-04-19 0:03 ` [PATCH rcu 08/21] srcu: Compute snp_seq earlier in srcu_funnel_gp_start() Paul E. McKenney
2022-04-19 0:03 ` [PATCH rcu 09/21] srcu: Use invalid initial value for srcu_node GP sequence numbers Paul E. McKenney
2022-04-19 0:03 ` [PATCH rcu 10/21] srcu: Ensure snp nodes tree is fully initialized before traversal Paul E. McKenney
2022-04-19 0:03 ` [PATCH rcu 11/21] srcu: Add boot-time control over srcu_node array allocation Paul E. McKenney
2022-04-19 0:03 ` [PATCH rcu 12/21] srcu: Use export for srcu_struct defined by DEFINE_STATIC_SRCU() Paul E. McKenney
2022-04-19 0:03 ` [PATCH rcu 13/21] srcu: Avoid NULL dereference in srcu_torture_stats_print() Paul E. McKenney
2022-04-19 0:03 ` [PATCH rcu 14/21] srcu: Prevent cleanup_srcu_struct() from freeing non-dynamic ->sda Paul E. McKenney
2022-04-19 0:03 ` [PATCH rcu 15/21] srcu: Explain srcu_funnel_gp_start() call to list_add() is safe Paul E. McKenney
2022-04-19 0:03 ` [PATCH rcu 16/21] srcu: Create concurrency-safe helper for initiating size transition Paul E. McKenney
2022-04-19 0:03 ` [PATCH rcu 17/21] srcu: Add contention-triggered addition of srcu_node tree Paul E. McKenney
2022-04-19 0:03 ` [PATCH rcu 18/21] srcu: Automatically determine size-transition strategy at boot Paul E. McKenney
2022-04-19 0:03 ` [PATCH rcu 19/21] srcu: Add contention check to call_srcu() srcu_data ->lock acquisition Paul E. McKenney
2022-04-19 0:03 ` [PATCH rcu 20/21] srcu: Prevent expedited GPs and blocking readers from consuming CPU Paul E. McKenney
2022-04-19 0:03 ` [PATCH rcu 21/21] srcu: Drop needless initialization of sdp in srcu_gp_start() Paul E. McKenney
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20220419000322.3948903-6-paulmck@kernel.org \
--to=paulmck@kernel.org \
--cc=kernel-team@fb.com \
--cc=linux-kernel@vger.kernel.org \
--cc=quic_neeraju@quicinc.com \
--cc=rcu@vger.kernel.org \
--cc=rostedt@goodmis.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®