[PATCH net-next v4 0/5] netconsole: Support messages ratelimit-ing

[PATCH net-next v4 0/5] netconsole: Support messages ratelimit-ing

[Breno Leitao]

netconsole hands every console message to every enabled target, with no
bound on the rate.

At Meta, a few hosts caused some DoS and netconsd was OOM killed in
some regions, fed by 3 billion RCU messages from thousand hosts and by
a swap fault that retries forever, which logged 6 million copies of one
line on a single host in a few hours.

Both floods are being fixed where they are printed, by Paul and me for RCU
Tasks [1][2] and by me for the swap fault fix [3], but that only ever
covers the flood already found, not new issues that might be happening.

Use the ratelimit in the kernel to ratelimit messages from netconsole.

Give each target a token bucket, consulted once per message so that a
message split into several ncfrag packets is sent whole or dropped
whole.

It starts with a zero interval, which struct ratelimit_state treats as
unlimited, so nothing changes until ratelimit_interval_ms and
ratelimit_burst are written.

What the bucket discards leaves nothing on the wire. An extended target
sees the loss as a gap in the sequence number the header carries:

	7.2.0-rc7-01460-g75848acaf136,13,628,252392647,-;netconsole selftest: netcons_AGujw 1
	7.2.0-rc7-01460-g75848acaf136,13,629,252392782,-;netconsole selftest: netcons_AGujw 2
	7.2.0-rc7-01460-g75848acaf136,13,678,254742125,-;netconsole selftest: netcons_AGujw 1

Crash output is exempt. The bucket is skipped once
netconsole_kernel_dying() is true, which the first patch adds:
oops_in_progress covers panic(), TAINT_DIE covers an oops or a BUG(),
whose records only reach netconsole from the printer thread, after
oops_end() cleared the flag.

A target that has seen the kernel die stays unlimited until reboot.

[1] https://lore.kernel.org/all/anw8Qw8lfeIykski@gmail.com/
[2] https://lore.kernel.org/all/20260810-rcu_task_shrink_lruvec-v1-1-4d9f7d5251cb@debian.org/
[3] https://lore.kernel.org/all/20260813-swap-v2-0-4a625ccabdae@debian.org/

Signed-off-by: Breno Leitao <leitao@debian.org>
---
Changes in v4:
- Rebased and resend, no change at all.
- Link to v3: https://patch.msgid.link/20260914-netcons_ratelimit-v3-0-8e81b789ab37@debian.org

Changes in v3:
- New first patch, a fix that stands on its own: oops_only=1 receives
  nothing on an oops today, because the backtrace reaches netconsole
  from the printer thread, after oops_end() cleared oops_in_progress.
  It tests TAINT_DIE too, behind netconsole_kernel_dying(), which the
  bucket then reuses so that a limit cannot truncate a crash either.
  (Sashiko)
- Restart the interval when either knob is written. A target is
  configured while it floods, with the bucket already empty, and the
  new limit only applied one interval later. (Sashiko)
- Reject an interval above INT_MAX ms. msecs_to_jiffies() saturates
  below INT_MAX on 32-bit, so checking the jiffies value alone took a
  write there that a 64-bit kernel turns down. (Sashiko)
- List both files in the target parameter table, and qualify the rule
  underneath it, which says that only a disabled target can be updated.
  (Sashiko)
- Link to v2: https://patch.msgid.link/20260910-netcons_ratelimit-v2-0-ebf0dd91e26e@debian.org

Changes in v2:
- Drop the patch that sent a "N messages dropped by rate limit" notice
  to the target. (Gustavo Luiz Duarte).
- Link to v1: https://patch.msgid.link/20260818-netcons_ratelimit-v1-0-8c5d2d17789c@debian.org

---
Breno Leitao (5):
      netconsole: send the oops when oops_only is set
      netconsole: add a per-target message rate limit
      netconsole: allow configuring the rate limit interval through configfs
      netconsole: allow configuring the rate limit burst through configfs
      docs: netconsole: document rate limit feature

 Documentation/networking/netconsole.rst |  71 ++++++++++++++----
 drivers/net/netconsole.c                | 126 +++++++++++++++++++++++++++++++-
 2 files changed, 181 insertions(+), 16 deletions(-)
---
base-commit: 2637bdec0535a2044cd830eba9a79363651d9043
base-tree: git://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git master
change-id: 20260817-netcons_ratelimit-629b04a73c57

Best regards,
--  
Breno Leitao <leitao@debian.org>

[PATCH net-next v4 1/5] netconsole: send the oops when oops_only is set

[Breno Leitao]

A target running with oops_only=1 receives nothing when the kernel
oopses, which is the one thing the option exists for. Only a panic
still gets through.

netconsole_write() bails out unless oops_in_progress is set, and that
flag is already gone by the time netconsole runs. netconsole is
CON_NBCON_ATOMIC_UNSAFE, so console_is_usable() keeps it out of the
emergency flush an oops prints in, and the records only leave the box
once the printer thread runs.

die() wakes that thread from oops_exit(), by which point
bust_spinlocks(0) has cleared oops_in_progress again. TAINT_DIE is set
in between, so test that as well, behind a helper.

With oops_only=1 on a netdevsim target, an lkdtm EXCEPTION produces 44
lines of oops. The receiver gets none of them before this change and
all of them after it, while ordinary messages stay suppressed.

The taint never clears, so from the first oops on, an oops_only target
sends everything instead of going quiet again. Sending the aftermath of
a crash beats sending nothing.

Fixes: 7eab73b18630 ("netconsole: convert to NBCON console infrastructure")
Reviewed-by: Gustavo Luiz Duarte <gustavold@gmail.com>
Signed-off-by: Breno Leitao <leitao@debian.org>
---
 drivers/net/netconsole.c | 16 ++++++++++++++--
 1 file changed, 14 insertions(+), 2 deletions(-)

diff --git a/drivers/net/netconsole.c b/drivers/net/netconsole.c
index 267254f046de9a..6637b4a16b6c98 100644
--- a/drivers/net/netconsole.c
+++ b/drivers/net/netconsole.c
@@ -83,7 +83,8 @@ MODULE_PARM_DESC(netconsole, " netconsole=[src-port]@[src-ip]/[dev],[tgt-port]@<
 
 static bool oops_only;
 module_param(oops_only, bool, 0600);
-MODULE_PARM_DESC(oops_only, "Only log oops messages");
+MODULE_PARM_DESC(oops_only,
+		 "Only log oops messages, everything once the kernel died");
 
 #define NETCONSOLE_PARAM_TARGET_PREFIX "cmdline"
 
@@ -96,6 +97,17 @@ static int __init option_setup(char *opt)
 __setup("netconsole=", option_setup);
 #endif	/* MODULE */
 
+/* The kernel is dying, or has died.
+ *
+ * oops_in_progress only spans the printing of the crash. netconsole is
+ * CON_NBCON_ATOMIC_UNSAFE, so the records reach the target later, from the
+ * printer thread, with the flag already cleared. TAINT_DIE outlives it.
+ */
+static bool netconsole_kernel_dying(void)
+{
+	return oops_in_progress || test_taint(TAINT_DIE);
+}
+
 /* Linked list of all configured targets */
 static LIST_HEAD(target_list);
 /* target_cleanup_list is used to track targets that need to be cleaned outside
@@ -2480,7 +2492,7 @@ static void netconsole_write(struct nbcon_write_context *wctxt, bool extended)
 {
 	struct netconsole_target *nt;
 
-	if (oops_only && !oops_in_progress)
+	if (oops_only && !netconsole_kernel_dying())
 		return;
 
 	list_for_each_entry(nt, &target_list, list) {

-- 
2.53.0-Meta

[PATCH net-next v4 2/5] netconsole: add a per-target message rate limit

[Breno Leitao]

Give each target a token bucket and consult it once per message in
netconsole_write().

The bucket is created with a zero interval, which struct ratelimit_state
treats as unlimited, and nothing can set a nonzero one yet, so no target
changes behaviour.

Skip the bucket while the kernel is dying, reusing the helper the
oops_only fix added, so a limit configured for steady-state logging
never truncates an oops, BUG() or panic().

The configfs files that expose it come next.

___ratelimit() only trylocks its own raw spinlock, so it is safe with
target_list_lock held and interrupts disabled, and safe from NMI. Set
RATELIMIT_MSG_ON_RELEASE so it does not report the suppressed count
itself, which would printk() from inside the console being serviced.

Reviewed-by: Gustavo Luiz Duarte <gustavold@gmail.com>
Signed-off-by: Breno Leitao <leitao@debian.org>
---
 drivers/net/netconsole.c | 37 +++++++++++++++++++++++++++++++++++++
 1 file changed, 37 insertions(+)

diff --git a/drivers/net/netconsole.c b/drivers/net/netconsole.c
index 6637b4a16b6c98..3e4495c1cab182 100644
--- a/drivers/net/netconsole.c
+++ b/drivers/net/netconsole.c
@@ -49,6 +49,7 @@
 #include <linux/rtnetlink.h>
 #include <linux/workqueue.h>
 #include <linux/delay.h>
+#include <linux/ratelimit.h>
 
 MODULE_AUTHOR("Matt Mackall <mpm@selenic.com>");
 MODULE_DESCRIPTION("Console driver for network interfaces");
@@ -196,6 +197,7 @@ struct netcons_userdata {
  * @sysdata:		Cached, formatted string of append
  * @sysdata_fields:	Sysdata features enabled.
  * @msgcounter:	Message sent counter.
+ * @ratelimit:	Token bucket used to rate limit messages
  * @stats:	Packet send stats for the target. Used for debugging.
  * @state:	State of the target.
  *		Visible from userspace (read-write).
@@ -239,6 +241,8 @@ struct netconsole_target {
 	u32			sysdata_fields;
 	/* protected by target_list_lock */
 	u32			msgcounter;
+	/* carries its own lock, writers hold dynamic_netconsole_mutex */
+	struct ratelimit_state	ratelimit;
 #endif
 	struct netconsole_target_stats stats;
 	enum target_state	state;
@@ -301,6 +305,26 @@ static void dynamic_netconsole_mutex_unlock(void)
 	mutex_unlock(&dynamic_netconsole_mutex);
 }
 
+static void netconsole_ratelimit_init(struct netconsole_target *nt)
+{
+	ratelimit_state_init(&nt->ratelimit, 0, DEFAULT_RATELIMIT_BURST);
+	/* The flag keeps ___ratelimit() from reporting the suppressed count
+	 * itself, which would printk() from inside the console being
+	 * serviced. Nothing calls ratelimit_state_exit(), so the count is
+	 * never reported on release either.
+	 */
+	ratelimit_set_flags(&nt->ratelimit, RATELIMIT_MSG_ON_RELEASE);
+}
+
+static bool netconsole_ratelimited(struct netconsole_target *nt)
+{
+	/* A limit meant for steady-state logging must not eat a crash dump. */
+	if (netconsole_kernel_dying())
+		return false;
+
+	return !__ratelimit(&nt->ratelimit);
+}
+
 #else	/* !CONFIG_NETCONSOLE_DYNAMIC */
 
 static int __init dynamic_netconsole_init(void)
@@ -337,6 +361,15 @@ static void dynamic_netconsole_mutex_unlock(void)
 {
 }
 
+static void netconsole_ratelimit_init(struct netconsole_target *nt)
+{
+}
+
+static bool netconsole_ratelimited(struct netconsole_target *nt)
+{
+	return false;
+}
+
 #endif	/* CONFIG_NETCONSOLE_DYNAMIC */
 
 /* Check if the target was bound by mac address. */
@@ -698,6 +731,7 @@ static struct netconsole_target *alloc_and_init(void)
 	nt->remote_port = 6666;
 	eth_broadcast_addr(nt->remote_mac);
 	nt->state = STATE_DISABLED;
+	netconsole_ratelimit_init(nt);
 	INIT_WORK(&nt->resume_wq, process_resume_target);
 	/* Set up the skb pool primitives once; enabling only refills it. */
 	skb_queue_head_init(&nt->skb_pool);
@@ -2500,6 +2534,9 @@ static void netconsole_write(struct nbcon_write_context *wctxt, bool extended)
 		    !netif_running(nt->np.dev))
 			continue;
 
+		if (netconsole_ratelimited(nt))
+			continue;
+
 		/* If nbcon_enter_unsafe() fails, just return given netconsole
 		 * lost the ownership, and iterating over the targets will not
 		 * be able to re-acquire.

-- 
2.53.0-Meta

[PATCH net-next v4 3/5] netconsole: allow configuring the rate limit interval through configfs

[Breno Leitao]

The per-target token bucket has no interface, so every target is still
unlimited.

Expose the interval as ratelimit_interval_ms through configfs. It
defaults to zero, so a target stays unlimited until an administrator
sets one.

The write restarts the interval instead of only publishing the new
value. A target is configured while it floods, by which point the
bucket is empty, and ___ratelimit() only refills it once the interval
it was set with has elapsed.

Reject more than INT_MAX milliseconds. msecs_to_jiffies() saturates at
MAX_JIFFY_OFFSET, which is below INT_MAX on 32-bit, so checking only
the jiffies value would take a write there that a 64-bit kernel turns
down.

Reviewed-by: Gustavo Luiz Duarte <gustavold@gmail.com>
Signed-off-by: Breno Leitao <leitao@debian.org>
---
 drivers/net/netconsole.c | 39 +++++++++++++++++++++++++++++++++++++++
 1 file changed, 39 insertions(+)

diff --git a/drivers/net/netconsole.c b/drivers/net/netconsole.c
index 3e4495c1cab182..69acf2c5c6b063 100644
--- a/drivers/net/netconsole.c
+++ b/drivers/net/netconsole.c
@@ -970,6 +970,14 @@ static ssize_t transmit_errors_show(struct config_item *item, char *buf)
 	return sysfs_emit(buf, "%llu\n", xmit_drop_count + enomem_count);
 }
 
+static ssize_t ratelimit_interval_ms_show(struct config_item *item, char *buf)
+{
+	struct netconsole_target *nt = to_target(item);
+
+	return sysfs_emit(buf, "%u\n",
+			  jiffies_to_msecs(READ_ONCE(nt->ratelimit.interval)));
+}
+
 /* configfs helper to display if cpu_nr sysdata feature is enabled */
 static ssize_t sysdata_cpu_nr_enabled_show(struct config_item *item, char *buf)
 {
@@ -1359,6 +1367,35 @@ static ssize_t remote_mac_store(struct config_item *item, const char *buf,
 	return ret;
 }
 
+static ssize_t ratelimit_interval_ms_store(struct config_item *item,
+					   const char *buf, size_t count)
+{
+	struct netconsole_target *nt = to_target(item);
+	unsigned int interval;
+	unsigned long jifs;
+	ssize_t ret;
+
+	ret = kstrtouint(buf, 10, &interval);
+	if (ret)
+		return ret;
+
+	/* msecs_to_jiffies() saturates below INT_MAX on 32-bit, so the
+	 * jiffies value alone does not bound what userspace wrote.
+	 */
+	jifs = msecs_to_jiffies(interval);
+	if (interval > INT_MAX || jifs > INT_MAX)
+		return -ERANGE;
+
+	/* Restart the interval, so a target that is already flooding picks
+	 * the new limit up now rather than at the next refill.
+	 */
+	dynamic_netconsole_mutex_lock();
+	ratelimit_state_reset_interval(&nt->ratelimit, jifs);
+	dynamic_netconsole_mutex_unlock();
+
+	return count;
+}
+
 struct userdatum {
 	struct config_item item;
 	char value[MAX_EXTRADATA_VALUE_LEN];
@@ -1723,6 +1760,7 @@ CONFIGFS_ATTR_RO(, local_mac);
 CONFIGFS_ATTR(, remote_mac);
 CONFIGFS_ATTR(, release);
 CONFIGFS_ATTR_RO(, transmit_errors);
+CONFIGFS_ATTR(, ratelimit_interval_ms);
 
 static struct configfs_attribute *netconsole_target_attrs[] = {
 	&attr_enabled,
@@ -1736,6 +1774,7 @@ static struct configfs_attribute *netconsole_target_attrs[] = {
 	&attr_local_mac,
 	&attr_remote_mac,
 	&attr_transmit_errors,
+	&attr_ratelimit_interval_ms,
 	NULL,
 };
 

-- 
2.53.0-Meta

[PATCH net-next v4 4/5] netconsole: allow configuring the rate limit burst through configfs

[Breno Leitao]

A target that sets ratelimit_interval_ms runs with the ratelimit default
of 10 messages per interval, which is either too coarse or too generous
depending on how chatty the target is.

Expose it as ratelimit_burst through configfs.

Restart the interval on write, as ratelimit_interval_ms_store() does.
Raising the burst of a target that has already drained the bucket
otherwise buys nothing until the interval in flight ends.

Reviewed-by: Gustavo Luiz Duarte <gustavold@gmail.com>
Signed-off-by: Breno Leitao <leitao@debian.org>
---
 drivers/net/netconsole.c | 34 ++++++++++++++++++++++++++++++++++
 1 file changed, 34 insertions(+)

diff --git a/drivers/net/netconsole.c b/drivers/net/netconsole.c
index 69acf2c5c6b063..2adf09adb9688c 100644
--- a/drivers/net/netconsole.c
+++ b/drivers/net/netconsole.c
@@ -978,6 +978,13 @@ static ssize_t ratelimit_interval_ms_show(struct config_item *item, char *buf)
 			  jiffies_to_msecs(READ_ONCE(nt->ratelimit.interval)));
 }
 
+static ssize_t ratelimit_burst_show(struct config_item *item, char *buf)
+{
+	struct netconsole_target *nt = to_target(item);
+
+	return sysfs_emit(buf, "%d\n", READ_ONCE(nt->ratelimit.burst));
+}
+
 /* configfs helper to display if cpu_nr sysdata feature is enabled */
 static ssize_t sysdata_cpu_nr_enabled_show(struct config_item *item, char *buf)
 {
@@ -1396,6 +1403,31 @@ static ssize_t ratelimit_interval_ms_store(struct config_item *item,
 	return count;
 }
 
+static ssize_t ratelimit_burst_store(struct config_item *item, const char *buf,
+				     size_t count)
+{
+	struct netconsole_target *nt = to_target(item);
+	unsigned int burst;
+	ssize_t ret;
+
+	ret = kstrtouint(buf, 10, &burst);
+	if (ret)
+		return ret;
+
+	if (burst > INT_MAX)
+		return -ERANGE;
+
+	/* Restart the interval as ratelimit_interval_ms_store() does, so the
+	 * new burst is spendable right away.
+	 */
+	dynamic_netconsole_mutex_lock();
+	WRITE_ONCE(nt->ratelimit.burst, burst);
+	ratelimit_state_reset_interval(&nt->ratelimit, nt->ratelimit.interval);
+	dynamic_netconsole_mutex_unlock();
+
+	return count;
+}
+
 struct userdatum {
 	struct config_item item;
 	char value[MAX_EXTRADATA_VALUE_LEN];
@@ -1761,6 +1793,7 @@ CONFIGFS_ATTR(, remote_mac);
 CONFIGFS_ATTR(, release);
 CONFIGFS_ATTR_RO(, transmit_errors);
 CONFIGFS_ATTR(, ratelimit_interval_ms);
+CONFIGFS_ATTR(, ratelimit_burst);
 
 static struct configfs_attribute *netconsole_target_attrs[] = {
 	&attr_enabled,
@@ -1775,6 +1808,7 @@ static struct configfs_attribute *netconsole_target_attrs[] = {
 	&attr_remote_mac,
 	&attr_transmit_errors,
 	&attr_ratelimit_interval_ms,
+	&attr_ratelimit_burst,
 	NULL,
 };
 

-- 
2.53.0-Meta

[PATCH net-next v4 5/5] docs: netconsole: document rate limit feature

[Breno Leitao]

Describe the per-target token bucket and the two configfs files that
drive it: ratelimit_interval_ms and ratelimit_burst.

Spell out the two properties that are not obvious from the file names.
The limit is accounted per message rather than per packet, so a message
split into several ncfrag packets is never truncated by the bucket
running dry halfway through.

List both files in the target parameter table too, and qualify the rule
underneath it. Only a disabled target can have its parameters updated,
these two aside.

Note in the message ID section that a message the bucket discards never
reaches the counter, so those drops leave no gap in the IDs.

Signed-off-by: Breno Leitao <leitao@debian.org>
Reviewed-by: Gustavo Luiz Duarte <gustavold@gmail.com>
---
 Documentation/networking/netconsole.rst | 71 ++++++++++++++++++++++++++-------
 1 file changed, 57 insertions(+), 14 deletions(-)

diff --git a/Documentation/networking/netconsole.rst b/Documentation/networking/netconsole.rst
index 694bca4803d50b..82b5d169d200aa 100644
--- a/Documentation/networking/netconsole.rst
+++ b/Documentation/networking/netconsole.rst
@@ -127,19 +127,21 @@ To remove a target::
 
 The interface exposes these parameters of a netconsole target to userspace:
 
-	=============== =================================       ============
-	enabled		Is this target currently enabled?	(read-write)
-	extended	Extended mode enabled			(read-write)
-	release		Prepend kernel release to message	(read-write)
-	dev_name	Local network interface name		(read-write)
-	local_port	Source UDP port to use			(read-write)
-	remote_port	Remote agent's UDP port			(read-write)
-	local_ip	Source IP address to use		(read-write)
-	remote_ip	Remote agent's IP address		(read-write)
-	local_mac	Local interface's MAC address		(read-only)
-	remote_mac	Remote agent's MAC address		(read-write)
-	transmit_errors	Number of packet send errors		(read-only)
-	=============== =================================       ============
+	===================== ================================= ============
+	enabled               Is this target currently enabled? (read-write)
+	extended              Extended mode enabled             (read-write)
+	release               Prepend kernel release to message (read-write)
+	dev_name              Local network interface name      (read-write)
+	local_port            Source UDP port to use            (read-write)
+	remote_port           Remote agent's UDP port           (read-write)
+	local_ip              Source IP address to use          (read-write)
+	remote_ip             Remote agent's IP address         (read-write)
+	local_mac             Local interface's MAC address     (read-only)
+	remote_mac            Remote agent's MAC address        (read-write)
+	transmit_errors       Number of packet send errors      (read-only)
+	ratelimit_interval_ms Rate limit interval, milliseconds (read-write)
+	ratelimit_burst       Messages allowed per interval     (read-write)
+	===================== ================================= ============
 
 local_ip is optional: if it is left unset, netconsole derives a source
 address from the egress interface when the target is enabled. Writing any
@@ -148,7 +150,8 @@ automatic selection.
 
 The "enabled" attribute is also used to control whether the parameters of
 a target can be updated or not -- you can modify the parameters of only
-disabled targets (i.e. if "enabled" is 0).
+disabled targets (i.e. if "enabled" is 0). The two rate limit parameters
+are the exception, see `Rate limiting`_.
 
 To update a target's parameters::
 
@@ -182,6 +185,43 @@ You can modify these targets in runtime by creating the following targets::
  cat cmdline1/remote_ip
  10.0.0.3
 
+Rate limiting
+-------------
+
+Netconsole hands every console message to every enabled target, so a host that
+logs continuously can saturate the receiving agent. Each target carries a token
+bucket that drops messages once the configured rate is exceeded, controlled by
+two files in the target directory:
+
+        ===================== ================================================
+        ratelimit_interval_ms Length of the accounting interval, in
+                              milliseconds. Zero, the default, sends
+                              everything.
+        ratelimit_burst       Messages allowed per interval. Defaults to
+                              10; zero drops every message once an
+                              interval is set.
+        ===================== ================================================
+
+Unlike most target parameters, both knobs can be written while the target is
+enabled, which is when a flooding target most likely needs them. Either write
+restarts the interval with a full burst, so a new limit applies from that
+moment on.
+
+The limit is applied per message, not per packet, so a message big enough to be
+split into several `ncfrag` packets is either sent whole or not at all.
+
+Crash output bypasses the bucket. Every message is sent while a panic is in
+progress, and an oops or a BUG() turns the limit off for the rest of the boot,
+so a small burst cannot cost you part of a crash dump.
+
+A drop leaves nothing on the wire. On an extended target it shows up as a gap
+in the sequence number the header carries; a basic target has no such marker.
+
+Capping a target at 500 messages a minute::
+
+  echo 60000 > ratelimit_interval_ms
+  echo 500 > ratelimit_burst
+
 Append User Data
 ----------------
 
@@ -364,6 +404,9 @@ indicate that a message was dropped during transmission, as it may never have
 been sent via netconsole. The message ID, on the other hand, is only assigned
 to messages that are actually transmitted via netconsole.
 
+A message the target's rate limit discards is dropped before the ID is
+assigned, so those drops leave no gap in the sequence of IDs either.
+
 Example::
 
   echo "This is message #1" > /dev/kmsg

-- 
2.53.0-Meta