[PATCH] genirq/proc: Fix missing CPU header in /proc/interrupts

[PATCH] genirq/proc: Fix missing CPU header in /proc/interrupts

[Haris Okanovic]

irq_seq_show() must be idempotent since it may be called multiple times
to render the same record, e.g. when seq_file's buffer is exceeded. It
emits the CPU header behind the one-shot print_header flag, which the
first pass clears, so the header is lost with the discarded buffer.

This can be seen on large systems, such as 8g.48xl (192 CPUs) and
8i.96xl (384 CPUs) AWS instance types:

  $ head -1 /proc/interrupts
     9:          0          0 ...  0  GICv3  25 Level  vgic

Replace print_header with the position of the first record and include
the header whenever that record is emitted.

Fixes: 171cc0d9eed1 ("genirq/proc: Speed up /proc/interrupts iteration")
Signed-off-by: Haris Okanovic <harisokn@amazon.com>
---
 kernel/irq/proc.c | 24 +++++++++++++++++-------
 1 file changed, 17 insertions(+), 7 deletions(-)

diff --git a/kernel/irq/proc.c b/kernel/irq/proc.c
index 1b835725f7b1c..0c0df16de5816 100644
--- a/kernel/irq/proc.c
+++ b/kernel/irq/proc.c
@@ -462,7 +462,7 @@ int __weak arch_show_interrupts(struct seq_file *p, int prec)
 static DEFINE_RAW_SPINLOCK(irq_proc_constraints_lock);
 
 static struct irq_proc_constraints {
-	bool		print_header;
+	loff_t		first_pos;
 	unsigned int	num_prec;
 	unsigned int	chip_width;
 } irq_proc_constraints __read_mostly = {
@@ -543,14 +543,13 @@ static int irq_seq_show(struct seq_file *p, void *v)
 	struct irqaction *action;
 
 	/* Print header for the first interrupt? */
-	if (constr->print_header) {
+	if (p->index == constr->first_pos) {
 		unsigned int cpu;
 
 		seq_printf(p, "%*s", constr->num_prec + 8, "");
 		for_each_online_cpu(cpu)
 			seq_printf(p, "CPU%-8d", cpu);
 		seq_putc(p, '\n');
-		constr->print_header = false;
 	}
 
 	if (desc == ARCH_PROC_IRQDESC)
@@ -635,14 +634,25 @@ static void *irq_seq_next_desc(loff_t *pos)
 
 static void *irq_seq_start(struct seq_file *f, loff_t *pos)
 {
-	if (!*pos) {
-		struct irq_proc_constraints *constr = f->private;
+	struct irq_proc_constraints *constr = f->private;
+	bool first = !*pos;
+	void *ret;
 
+	if (first) {
 		constr->num_prec = READ_ONCE(irq_proc_constraints.num_prec);
 		constr->chip_width = READ_ONCE(irq_proc_constraints.chip_width);
-		constr->print_header = true;
 	}
-	return irq_seq_next_desc(pos);
+
+	ret = irq_seq_next_desc(pos);
+
+	/*
+	 * Record position of the first output record so that irq_seq_show()
+	 * knows where to write the header.
+	 */
+	if (first)
+		constr->first_pos = *pos;
+
+	return ret;
 }
 
 static void *irq_seq_next(struct seq_file *f, void *v, loff_t *pos)

---
base-commit: 08df884136f1c1197bab2a27814404fd329d9aac
change-id: 20260911-dev-harisokn-proc-interrupts-fix-b4-7498f7e18db5

Best regards,
--  
Haris Okanovic
AWS Graviton

Re: [PATCH] genirq/proc: Fix missing CPU header in /proc/interrupts

[Okanovic, Haris]

Gentle ping on this one. Any concerns with the approach?

We'd like to get this in before more distros pick up 171cc0d9eed1. Losing the
CPU header is a user-visible format change that breaks tools parsing
/proc/interrupts, e.g. aperf:

  https://github.com/aws/aperf

Happy to respin if you'd prefer a different fix.

Regards,
-- 
Haris Okanovic
AWS Graviton

Re: [PATCH] genirq/proc: Fix missing CPU header in /proc/interrupts

[Radu Rendec]

On Fri, 2026-09-11 at 13:31 -0500, Haris Okanovic wrote:
> irq_seq_show() must be idempotent since it may be called multiple times
> to render the same record, e.g. when seq_file's buffer is exceeded. It
> emits the CPU header behind the one-shot print_header flag, which the
> first pass clears, so the header is lost with the discarded buffer.
> 
> This can be seen on large systems, such as 8g.48xl (192 CPUs) and
> 8i.96xl (384 CPUs) AWS instance types:
> 
>   $ head -1 /proc/interrupts
>      9:          0          0 ...  0  GICv3  25 Level  vgic
> 
> Replace print_header with the position of the first record and include
> the header whenever that record is emitted.
> 
> Fixes: 171cc0d9eed1 ("genirq/proc: Speed up /proc/interrupts iteration")
> Signed-off-by: Haris Okanovic <harisokn@amazon.com>
> ---
>  kernel/irq/proc.c | 24 +++++++++++++++++-------
>  1 file changed, 17 insertions(+), 7 deletions(-)
> 
> diff --git a/kernel/irq/proc.c b/kernel/irq/proc.c
> index 1b835725f7b1c..0c0df16de5816 100644
> --- a/kernel/irq/proc.c
> +++ b/kernel/irq/proc.c
> @@ -462,7 +462,7 @@ int __weak arch_show_interrupts(struct seq_file *p, int prec)
>  static DEFINE_RAW_SPINLOCK(irq_proc_constraints_lock);
>  
>  static struct irq_proc_constraints {
> -	bool		print_header;
> +	loff_t		first_pos;
>  	unsigned int	num_prec;
>  	unsigned int	chip_width;
>  } irq_proc_constraints __read_mostly = {
> @@ -543,14 +543,13 @@ static int irq_seq_show(struct seq_file *p, void *v)
>  	struct irqaction *action;
>  
>  	/* Print header for the first interrupt? */
> -	if (constr->print_header) {
> +	if (p->index == constr->first_pos) {
>  		unsigned int cpu;
>  
>  		seq_printf(p, "%*s", constr->num_prec + 8, "");
>  		for_each_online_cpu(cpu)
>  			seq_printf(p, "CPU%-8d", cpu);
>  		seq_putc(p, '\n');
> -		constr->print_header = false;
>  	}
>  
>  	if (desc == ARCH_PROC_IRQDESC)
> @@ -635,14 +634,25 @@ static void *irq_seq_next_desc(loff_t *pos)
>  
>  static void *irq_seq_start(struct seq_file *f, loff_t *pos)
>  {
> -	if (!*pos) {
> -		struct irq_proc_constraints *constr = f->private;
> +	struct irq_proc_constraints *constr = f->private;
> +	bool first = !*pos;
> +	void *ret;
>  
> +	if (first) {
>  		constr->num_prec = READ_ONCE(irq_proc_constraints.num_prec);
>  		constr->chip_width = READ_ONCE(irq_proc_constraints.chip_width);
> -		constr->print_header = true;
>  	}
> -	return irq_seq_next_desc(pos);
> +
> +	ret = irq_seq_next_desc(pos);
> +
> +	/*
> +	 * Record position of the first output record so that irq_seq_show()
> +	 * knows where to write the header.
> +	 */
> +	if (first)
> +		constr->first_pos = *pos;
> +
> +	return ret;
>  }
>  
>  static void *irq_seq_next(struct seq_file *f, void *v, loff_t *pos)
> 
> ---
> base-commit: 08df884136f1c1197bab2a27814404fd329d9aac
> change-id: 20260911-dev-harisokn-proc-interrupts-fix-b4-7498f7e18db5

Since I don't have access to a machine with so many CPUs, I tested
locally in a regular VM and simulated the large number of CPUs by
adding artificial buffer writes to both the header part and the IRQ
counter part of irq_seq_show(). I could reproduce the problem and
validate the fix. Also, semantically the fix looks good to me.
Thanks for catching this and for the patch!

Reviewed-by: Radu Rendec <radu@rendec.net>

Re: [PATCH] genirq/proc: Fix missing CPU header in /proc/interrupts

[Okanovic, Haris]

On Thu, 2026-09-17 at 22:55 -0400, Radu Rendec wrote:
> Since I don't have access to a machine with so many CPUs, I tested
> locally in a regular VM and simulated the large number of CPUs by
> adding artificial buffer writes to both the header part and the IRQ
> counter part of irq_seq_show(). I could reproduce the problem and
> validate the fix. Also, semantically the fix looks good to me.
> Thanks for catching this and for the patch!

Thanks for looking at this, and for finding a way to reproduce it without
the hardware. If a large machine would help in the future, let me know
off-list and I'll see what I can arrange.

-- 
Haris Okanovic
AWS Graviton

RE: [PATCH] genirq/proc: Fix missing CPU header in /proc/interrupts

[Michael Kelley]

From: Haris Okanovic <harisokn@amazon.com> Sent: Friday, September 11, 2026 11:32 AM
> 
> irq_seq_show() must be idempotent since it may be called multiple times
> to render the same record, e.g. when seq_file's buffer is exceeded. It
> emits the CPU header behind the one-shot print_header flag, which the
> first pass clears, so the header is lost with the discarded buffer.
> 
> This can be seen on large systems, such as 8g.48xl (192 CPUs) and
> 8i.96xl (384 CPUs) AWS instance types:
> 
>   $ head -1 /proc/interrupts
>      9:          0          0 ...  0  GICv3  25 Level  vgic
> 
> Replace print_header with the position of the first record and include
> the header whenever that record is emitted.
> 
> Fixes: 171cc0d9eed1 ("genirq/proc: Speed up /proc/interrupts iteration")
> Signed-off-by: Haris Okanovic <harisokn@amazon.com>
> ---
>  kernel/irq/proc.c | 24 +++++++++++++++++-------
>  1 file changed, 17 insertions(+), 7 deletions(-)
> 
> diff --git a/kernel/irq/proc.c b/kernel/irq/proc.c
> index 1b835725f7b1c..0c0df16de5816 100644
> --- a/kernel/irq/proc.c
> +++ b/kernel/irq/proc.c
> @@ -462,7 +462,7 @@ int __weak arch_show_interrupts(struct seq_file *p, int prec)
>  static DEFINE_RAW_SPINLOCK(irq_proc_constraints_lock);
> 
>  static struct irq_proc_constraints {
> -	bool		print_header;
> +	loff_t		first_pos;
>  	unsigned int	num_prec;
>  	unsigned int	chip_width;
>  } irq_proc_constraints __read_mostly = {
> @@ -543,14 +543,13 @@ static int irq_seq_show(struct seq_file *p, void *v)
>  	struct irqaction *action;
> 
>  	/* Print header for the first interrupt? */
> -	if (constr->print_header) {
> +	if (p->index == constr->first_pos) {
>  		unsigned int cpu;
> 
>  		seq_printf(p, "%*s", constr->num_prec + 8, "");
>  		for_each_online_cpu(cpu)
>  			seq_printf(p, "CPU%-8d", cpu);
>  		seq_putc(p, '\n');
> -		constr->print_header = false;
>  	}
> 
>  	if (desc == ARCH_PROC_IRQDESC)
> @@ -635,14 +634,25 @@ static void *irq_seq_next_desc(loff_t *pos)
> 
>  static void *irq_seq_start(struct seq_file *f, loff_t *pos)
>  {
> -	if (!*pos) {
> -		struct irq_proc_constraints *constr = f->private;
> +	struct irq_proc_constraints *constr = f->private;
> +	bool first = !*pos;
> +	void *ret;
> 
> +	if (first) {
>  		constr->num_prec = READ_ONCE(irq_proc_constraints.num_prec);
>  		constr->chip_width = READ_ONCE(irq_proc_constraints.chip_width);
> -		constr->print_header = true;
>  	}
> -	return irq_seq_next_desc(pos);
> +
> +	ret = irq_seq_next_desc(pos);
> +
> +	/*
> +	 * Record position of the first output record so that irq_seq_show()
> +	 * knows where to write the header.
> +	 */
> +	if (first)
> +		constr->first_pos = *pos;
> +
> +	return ret;
>  }
> 
>  static void *irq_seq_next(struct seq_file *f, void *v, loff_t *pos)
> 

I tested this patch in an Azure x86/x64 VM with 416 CPUs. The "CPU"
header line is missing without the patch, and it appears when the
patch is applied. I verified that the 416 "CPU<nnn>" column labels
are correctly shown.

I have not reviewed the code.

Tested-by: Michael Kelley <mhklinux@outlook.com>

Re: [PATCH] genirq/proc: Fix missing CPU header in /proc/interrupts

[Okanovic, Haris]

On Tue, 2026-09-22 at 00:52 +0000, Michael Kelley wrote:
> I tested this patch in an Azure x86/x64 VM with 416 CPUs. The "CPU"
> header line is missing without the patch, and it appears when the
> patch is applied. I verified that the 416 "CPU<nnn>" column labels
> are correctly shown.
>
> I have not reviewed the code.

Thanks Michael, good to hear it works on other platforms.

-- 
Haris Okanovic
AWS Graviton

Re: [PATCH] genirq/proc: Fix missing CPU header in /proc/interrupts

[Okanovic, Haris]

Hi,

Second ping on this one. It's been reviewed by Radu and tested by
Michael on an Azure VM. No outstanding feedback.

I've also tested the fix on aarch64 and x86_64, on AWS instances from
three major CPU vendors:

  8g.48xl   192 CPUs, Graviton4 (Neoverse V2)
  9g.48xl   192 CPUs, Graviton5 (Neoverse V3)
  8a.48xl   192 CPUs, AMD EPYC 5th gen (Zen 5)
  8i.96xl   384 CPUs, Intel Xeon 6 (Redwood Cove)

Let me know if you'd like a v2 with a different approach.

Regards,
-- 
Haris Okanovic
AWS Graviton

Re: [PATCH] genirq/proc: Fix missing CPU header in /proc/interrupts

[Radu Rendec]

On Mon, 2026-10-05 at 20:46 +0000, Okanovic, Haris wrote:
> Second ping on this one. It's been reviewed by Radu and tested by
> Michael on an Azure VM. No outstanding feedback.

Hi Haris,

This will have to be merged by Thomas, and AFAIK he's out at Linux
Plumbers, so I expect there will be some delay. FWIW, there are other
IRQ related patches/series that have been piling up lately, so I'm
pretty sure he's busy :)

-- 
Best regards,
Radu