[PATCH 0/1] printk: Yield console_sem to blocking waiters

[PATCH 0/1] printk: Yield console_sem to blocking waiters

[Aruna Ramakrishna]

Problem context:

After a reboot, NetworkManager failed to bring up interfaces because
systemd was stalled, due to a flood of printk warnings. Systemd was
blocked waiting on the console lock for over 122 seconds:

  __schedule+0x224/0x65b
  schedule+0x5f/0xdf
  schedule_timeout+0x110/0x145
  __down_common+0x14d/0x18c
  ? srso_alias_return_thunk+0x5/0x7
  down+0x43/0x54
  console_lock+0x16/0x3d
  show_cons_active+0x3d/0x1b6
  dev_attr_show+0x16/0x50
  sysfs_kf_seq_show+0x9c/0x124
  seq_read_iter+0x12b/0x4e3
  new_sync_read+0x140/0x1db
  vfs_read+0x106/0x199
  ksys_read+0x67/0xe8
  do_syscall_64+0x35/0x87
  entry_SYSCALL_64_after_hwframe+0x6e/0x0

The loglevel was set to 7, due to which there were a flood of messages
to the console, which in turn starved systemd due to the chain of direct
printk handoffs. Direct printk handoffs transfer printing ownership
between printk callers without releasing console_sem. A continuous
chain of these handoffs can therefore leave blocking console_lock()
callers waiting on the semaphore.

Proposed solution:

If there is a blocked console_sem waiter, then:
1. Prevent new printk-to-printk handoffs.
2. Stop the current owner from indefinitely flushing all records until
its done.

This unblocks console_sem waiters who are not doing printk.

Aruna Ramakrishna (1):
  printk: Yield console_sem to blocking waiters

 kernel/printk/printk.c | 39 ++++++++++++++++++++++++++++++++-------
 1 file changed, 32 insertions(+), 7 deletions(-)

-- 
2.43.7

[PATCH 1/1] printk: Yield console_sem to blocking waiters

[Aruna Ramakrishna]

A continuous chain of direct printk handoffs can keep console_sem held
while blocking console_lock() callers are starved.

If there is a blocked console_sem waiter, then:
1. Prevent new printk-to-printk handoffs.
2. Stop the current owner from indefinitely flushing all records until
its done.

This unblocks console_sem waiters who are not doing printk.

Fixes: dbdda842fe96 ("printk: Add console owner and waiter logic to load balance console writes")

Signed-off-by: Aruna Ramakrishna <aruna.ramakrishna@oracle.com>
---
 kernel/printk/printk.c | 39 ++++++++++++++++++++++++++++++++-------
 1 file changed, 32 insertions(+), 7 deletions(-)

diff --git a/kernel/printk/printk.c b/kernel/printk/printk.c
index 3fcdf4b4e2e5..b182bc980850 100644
--- a/kernel/printk/printk.c
+++ b/kernel/printk/printk.c
@@ -94,6 +94,13 @@ static DEFINE_MUTEX(console_mutex);
  * and also provides serialization for console printing.
  */
 static DEFINE_SEMAPHORE(console_sem, 1);
+
+/*
+ * Number of blocking console_sem waiters. This does not include spinning
+ * printk waiters - they bypass the semaphore wait list.
+ */
+static atomic_t console_sem_waiters = ATOMIC_INIT(0);
+
 HLIST_HEAD(console_list);
 EXPORT_SYMBOL_GPL(console_list);
 DEFINE_STATIC_SRCU(console_srcu);
@@ -311,7 +318,9 @@ EXPORT_SYMBOL(console_srcu_read_unlock);
  * macros instead of functions so that _RET_IP_ contains useful information.
  */
 #define down_console_sem() do { \
+	atomic_inc(&console_sem_waiters);\
 	down(&console_sem);\
+	atomic_dec(&console_sem_waiters);\
 	mutex_acquire(&console_lock_dep_map, 0, 0, _RET_IP_);\
 } while (0)
 
@@ -2002,7 +2011,8 @@ static int console_trylock_spinning(void)
 	raw_spin_lock(&console_owner_lock);
 	owner = READ_ONCE(console_owner);
 	waiter = READ_ONCE(console_waiter);
-	if (!waiter && owner && owner != current) {
+	if (!waiter && owner && owner != current &&
+	    !atomic_read(&console_sem_waiters)) {
 		WRITE_ONCE(console_waiter, true);
 		spin = true;
 	}
@@ -3329,16 +3339,22 @@ static bool console_flush_one_record(bool do_cond_resched, u64 *next_seq, bool *
  * console_lock, in which case the caller is no longer holding the
  * console_lock. Otherwise it is set to false.
  *
+ * @yield_to_waiter is set by a caller if there is a blocking console_sem
+ * waiter, in which case the console_lock will be released rather than
+ * being handed over to the next printk waiter.
+ *
  * Returns true when there was at least one usable console and all messages
  * were flushed to all usable consoles. A returned false informs the caller
  * that everything was not flushed (either there were no usable consoles or
  * another context has taken over printing or it is a panic situation and this
- * is not the panic CPU). Regardless the reason, the caller should assume it
- * is not useful to immediately try again.
+ * is not the panic CPU or there is a blocking console_lock waiter). Regardless
+ * the reason, the caller should assume it is not useful to immediately try
+ * again.
  *
  * Requires the console_lock.
  */
-static bool console_flush_all(bool do_cond_resched, u64 *next_seq, bool *handover)
+static bool console_flush_all(bool do_cond_resched, u64 *next_seq, bool *handover,
+			      bool yield_to_waiter)
 {
 	bool try_again;
 	bool ret;
@@ -3349,6 +3365,14 @@ static bool console_flush_all(bool do_cond_resched, u64 *next_seq, bool *handove
 	do {
 		ret = console_flush_one_record(do_cond_resched, next_seq,
 					       handover, &try_again);
+
+		/*
+		 * If there's a blocking console_lock waiter, and we're not
+		 * in panic, then yield to that waiter.
+		 */
+		if (!panic_in_progress() && try_again && yield_to_waiter &&
+		    atomic_read(&console_sem_waiters))
+			return false;
 	} while (try_again);
 
 	return ret;
@@ -3377,7 +3401,8 @@ static void __console_flush_and_unlock(void)
 	do {
 		console_may_schedule = 0;
 
-		flushed = console_flush_all(do_cond_resched, &next_seq, &handover);
+		flushed = console_flush_all(do_cond_resched, &next_seq, &handover,
+					    true);
 		if (!handover)
 			__console_unlock();
 
@@ -3563,7 +3588,7 @@ void console_flush_on_panic(enum con_flush_mode mode)
 
 	/* Flush legacy consoles once allowed, even when dangerous. */
 	if (legacy_allow_panic_sync)
-		console_flush_all(false, &next_seq, &handover);
+		console_flush_all(false, &next_seq, &handover, false);
 }
 
 /*
@@ -3990,7 +4015,7 @@ static u64 get_init_console_seq(struct console *newcon, bool bootcon_registered)
 			 * Flush all consoles and set the console to start at
 			 * the next unprinted sequence number.
 			 */
-			if (!console_flush_all(true, &init_seq, &handover)) {
+			if (!console_flush_all(true, &init_seq, &handover, false)) {
 				/*
 				 * Flushing failed. Just choose the lowest
 				 * sequence of the enabled boot consoles.
-- 
2.43.7

Re: [PATCH 0/1] printk: Yield console_sem to blocking waiters

[John Ogness]

On 2026-09-10, Aruna Ramakrishna <aruna.ramakrishna@oracle.com> wrote:
> Problem context:
>
> After a reboot, NetworkManager failed to bring up interfaces because
> systemd was stalled, due to a flood of printk warnings.

Note that kernel warnings are considered serious and will put the CPU
into emergency state.

> Systemd was
> blocked waiting on the console lock for over 122 seconds:

Also note that the console lock is related to legacy consoles. Rather
than trying to make the legacy console trainwreck more reliable and
non-interfering, efforts should be spent converting legacy consoles to
NBCON.

What console driver are you using?

For v7.3 the 8250 driver will be NBCON.

John

Re: [PATCH 0/1] printk: Yield console_sem to blocking waiters

[Aruna Ramakrishna]

> On Sep 11, 2026, at 1:08 AM, John Ogness <john.ogness@linutronix.de> wrote:
> 
> This Message Is From an External Sender
> This message came from outside your organization.
> Report Suspicious
> On 2026-09-10, Aruna Ramakrishna <aruna.ramakrishna@oracle.com> wrote:
> > Problem context:
> >
> > After a reboot, NetworkManager failed to bring up interfaces because
> > systemd was stalled, due to a flood of printk warnings.
> 
> Note that kernel warnings are considered serious and will put the CPU
> into emergency state.

I apologize - I misspoke. These were not KERN_WARNs, rather KERN_INFOs
being used by RDS to warn of connection failures etc. A flood of these caused
the console_lock starvation.

> 
> > Systemd was
> > blocked waiting on the console lock for over 122 seconds:
> 
> Also note that the console lock is related to legacy consoles. Rather
> than trying to make the legacy console trainwreck more reliable and
> non-interfering, efforts should be spent converting legacy consoles to
> NBCON.
> 
> What console driver are you using?
> 
> For v7.3 the 8250 driver will be NBCON.
> 
> John
> 

This is for the legacy 8250 serial console.

Thanks,
Aruna