[PATCH 1/3] Added new Proc entry to display per-cpu interrupts

[PATCH 1/3] Added new Proc entry to display per-cpu interrupts

[Sasikantha babu]

As the number of cores increases the output of the /proc/interrupts losing the readability.

I think instead of showing complete information of all cores in a single proc file spliting the 
information based on per-CPU will be more useful (especially for people who wants to track
per-cpu information in real-time systems).

It is just a base version and works only for X86  (Copy pasted the output below)

Please let me know your opinions on this (Based on that I will try to make it work for all arch).

On a 24-core machine output of /proc/interrupts is not readable.

root@localhost:~# cat /proc/interrupts
           CPU0       CPU1       CPU2       CPU3       CPU4       CPU5       CPU6       CPU7       CPU8       CPU9       CPU10      CPU11      CPU12      CPU13      CPU14      CPU15      CPU16      CPU17      CPU18      CPU19      CPU20      CPU21      CPU22      CPU23
  0:       1305          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   IO-APIC-edge      timer
  4:       3077          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   IO-APIC-edge      serial
  8:          1          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   IO-APIC-edge      rtc0
  9:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   IO-APIC-fasteoi   acpi
 14:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   IO-APIC-edge      ata_piix


Newly added proc entries output.

[sasikantha@localhost Desktop]$ cat /proc/cpu/
0/ 1/
[sasikantha@localhost Desktop]$ cat /proc/cpu/0/interrupts
           CPU0
  0:    1122723   IO-APIC-edge      timer
  1:      45534   IO-APIC-edge      i8042
  4:          2   IO-APIC-edge
  7:          0   IO-APIC-edge      parport0
  8:          0   IO-APIC-edge      rtc0
  9:          0   IO-APIC-fasteoi   acpi
 12:     444699   IO-APIC-edge      i8042
 14:     223211   IO-APIC-edge      ata_piix
 15:          0   IO-APIC-edge      ata_piix
 16:         10   IO-APIC-fasteoi   uhci_hcd:usb3, uhci_hcd:usb8, eth1
 18:          0   IO-APIC-fasteoi   ehci_hcd:usb1, uhci_hcd:usb7
 19:          0   IO-APIC-fasteoi   uhci_hcd:usb6
 21:          0   IO-APIC-fasteoi   uhci_hcd:usb4
 23:          0   IO-APIC-fasteoi   ehci_hcd:usb2, uhci_hcd:usb5
 42:      95394   PCI-MSI-edge      i915
 43:          0   PCI-MSI-edge      eth0
 44:        253   PCI-MSI-edge      snd_hda_intel
NMI:          0   Non-maskable interrupts
LOC:   10618394   Local timer interrupts
SPU:          0   Spurious interrupts
PMI:          0   Performance monitoring interrupts
IWI:          0   IRQ work interrupts
RTR:          0   APIC ICR read retries
RES:    1108550   Rescheduling interrupts
CAL:        223   Function call interrupts
TLB:      20763   TLB shootdowns
TRM:          0   Thermal event interrupts
THR:          0   Threshold APIC interrupts
MCE:          0   Machine check exceptions
MCP:         54   Machine check polls
ERR:          0
MIS:          0
[sasikantha@localhost Desktop]$ cat /proc/cpu/1/interrupts
           CPU1
  0:    1101722   IO-APIC-edge      timer
  1:         15   IO-APIC-edge      i8042
  4:          1   IO-APIC-edge
  7:          0   IO-APIC-edge      parport0
  8:          1   IO-APIC-edge      rtc0
  9:          0   IO-APIC-fasteoi   acpi
 12:         65   IO-APIC-edge      i8042
 14:     200909   IO-APIC-edge      ata_piix
 15:          0   IO-APIC-edge      ata_piix
 16:    2875246   IO-APIC-fasteoi   uhci_hcd:usb3, uhci_hcd:usb8, eth1
 18:          0   IO-APIC-fasteoi   ehci_hcd:usb1, uhci_hcd:usb7
 19:          0   IO-APIC-fasteoi   uhci_hcd:usb6
 21:          0   IO-APIC-fasteoi   uhci_hcd:usb4
 42:         26   PCI-MSI-edge      i915
 43:          0   PCI-MSI-edge      eth0
 44:      72904   PCI-MSI-edge      snd_hda_intel
NMI:          0   Non-maskable interrupts
LOC:    9635490   Local timer interrupts
SPU:          0   Spurious interrupts
PMI:          0   Performance monitoring interrupts
IWI:          0   IRQ work interrupts
RTR:          0   APIC ICR read retries
RES:    1216863   Rescheduling interrupts
CAL:        412   Function call interrupts
TLB:      21671   TLB shootdowns
TRM:          0   Thermal event interrupts
THR:          0   Threshold APIC interrupts
MCE:          0   Machine check exceptions
MCP:         54   Machine check polls
ERR:          0
MIS:          0


Did not touch the existing /proc/ininterrupts

[sasikantha@localhost Desktop]$ cat /proc/interrupts
           CPU0       CPU1
  0:    1122889    1101729   IO-APIC-edge      timer
  1:      45700         15   IO-APIC-edge      i8042
  4:          2          1   IO-APIC-edge
  7:          0          0   IO-APIC-edge      parport0
  8:          0          1   IO-APIC-edge      rtc0
  9:          0          0   IO-APIC-fasteoi   acpi
 12:     444699         65   IO-APIC-edge      i8042
 14:     223560     200967   IO-APIC-edge      ata_piix
 15:          0          0   IO-APIC-edge      ata_piix
 16:         10    2908377   IO-APIC-fasteoi   uhci_hcd:usb3, uhci_hcd:usb8, eth1
 18:          0          0   IO-APIC-fasteoi   ehci_hcd:usb1, uhci_hcd:usb7
 19:          0          0   IO-APIC-fasteoi   uhci_hcd:usb6
 21:          0          0   IO-APIC-fasteoi   uhci_hcd:usb4
 23:          0          0   IO-APIC-fasteoi   ehci_hcd:usb2, uhci_hcd:usb5
 42:      95409         26   PCI-MSI-edge      i915
 43:          0          0   PCI-MSI-edge      eth0
 44:        253      72929   PCI-MSI-edge      snd_hda_intel
NMI:          0          0   Non-maskable interrupts
LOC:   10627890    9640400   Local timer interrupts
SPU:          0          0   Spurious interrupts
PMI:          0          0   Performance monitoring interrupts
IWI:          0          0   IRQ work interrupts
RTR:          0          0   APIC ICR read retries
RES:    1113218    1220968   Rescheduling interrupts
CAL:        224        412   Function call interrupts
TLB:      20763      21671   TLB shootdowns
TRM:          0          0   Thermal event interrupts
THR:          0          0   Threshold APIC interrupts
MCE:          0          0   Machine check exceptions
MCP:         54         54   Machine check polls
ERR:          0
MIS:          0


Signed-off-by: Sasikantha babu <sasikanth.v19@gmail.com>
---
 fs/proc/Makefile |    1 +
 fs/proc/percpu.c |  115 ++++++++++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 116 insertions(+), 0 deletions(-)
 create mode 100644 fs/proc/percpu.c

diff --git a/fs/proc/Makefile b/fs/proc/Makefile
index c1c7293..3615c72 100644
--- a/fs/proc/Makefile
+++ b/fs/proc/Makefile
@@ -21,6 +21,7 @@ proc-y	+= uptime.o
 proc-y	+= version.o
 proc-y	+= softirqs.o
 proc-y	+= namespaces.o
+proc-y	+= percpu.o
 proc-$(CONFIG_PROC_SYSCTL)	+= proc_sysctl.o
 proc-$(CONFIG_NET)		+= proc_net.o
 proc-$(CONFIG_PROC_KCORE)	+= kcore.o
diff --git a/fs/proc/percpu.c b/fs/proc/percpu.c
new file mode 100644
index 0000000..da1d05c
--- /dev/null
+++ b/fs/proc/percpu.c
@@ -0,0 +1,115 @@
+/*
+ *   Per CPU-based information
+ *   Licensed under the GPL-2 or later
+ */
+
+#include <linux/fs.h>
+#include <linux/init.h>
+#include <linux/interrupt.h>
+#include <linux/irqnr.h>
+#include <linux/proc_fs.h>
+#include <linux/seq_file.h>
+
+
+static struct proc_dir_entry *cpu_base_dir;
+static struct proc_dir_entry *cpu_dir[NR_CPUS];
+
+/*
+   Interrupts proc is based on /proc/interrupts 
+*/
+
+/*
+ * /proc/cpu/<number>/interrupts 
+ */
+
+static void *int_seq_start(struct seq_file *f, loff_t *pos)
+{
+	return (*pos <= nr_irqs) ? pos : NULL;
+}
+
+static void *int_seq_next(struct seq_file *f, void *v, loff_t *pos)
+{
+	(*pos)++;
+	if (*pos > nr_irqs)
+		return NULL;
+	return pos;
+}
+
+static void int_seq_stop(struct seq_file *f, void *v)
+{
+	/* Nothing to do */
+}
+
+static const struct seq_operations int_seq_ops = {
+	.start = int_seq_start,
+	.next  = int_seq_next,
+	.stop  = int_seq_stop,
+	.show  = show_interrupts
+};
+
+static int interrupts_open(struct inode *inode, struct file *filp)
+{
+	int ret;
+	ret =  seq_open(filp, &int_seq_ops);
+
+	if (!ret) {
+		struct seq_file *m = filp->private_data;
+		m->private = PDE(inode)->data;
+	}
+	return ret;
+
+}
+
+static const struct file_operations proc_intr_ops = {
+	.open		= interrupts_open,
+	.read		= seq_read,
+	.llseek		= seq_lseek,
+	.release	= seq_release,
+};
+
+
+static void __exit proc_per_cpu_exit (void)
+{
+	int cpu = 0;
+        char name [10];
+
+        for_each_present_cpu (cpu) {
+		if  (cpu_dir[cpu]) {
+	                sprintf(name, "%d", cpu);
+			remove_proc_entry ("interrupts", cpu_dir[cpu]);
+			remove_proc_entry (name, cpu_base_dir);
+		}
+        }
+	remove_proc_entry ("cpu", NULL);
+}
+
+static int __init proc_per_cpu_init (void)
+{
+	int cpu;
+        char name [10];
+
+        cpu_base_dir = proc_mkdir("cpu", NULL);
+
+        if (!cpu_base_dir)
+		return -1;
+
+	for_each_present_cpu (cpu) {
+		sprintf(name, "%d", cpu);
+		cpu_dir[cpu] = proc_mkdir(name, cpu_base_dir);
+		if (!cpu_dir[cpu])
+			goto out_remove_dir;
+		/*XXX:Storing the CPU number as private data for simplicty,
+ 		 I may be wrong ?*/
+	        proc_create_data ("interrupts", 0, cpu_dir[cpu], &proc_intr_ops, 
+				  (void *)(cpu + 1));
+	}
+
+        return 0;
+
+out_remove_dir:
+	proc_per_cpu_exit ();
+	return -1;
+}
+
+module_exit(proc_per_cpu_exit);
+module_init(proc_per_cpu_init);
-- 
1.7.3.4

[PATCH 2/3] Added arch_show_interrupts_cpu to display CPU specific interrupts for Proc Fs

[Sasikantha babu]

Signed-off-by: Sasikantha babu <sasikanth.v19@gmail.com>
---
 arch/x86/kernel/irq.c     |   70 +++++++++++++++++++++++++++++++++++++++++++++
 include/linux/interrupt.h |    1 +
 2 files changed, 71 insertions(+), 0 deletions(-)

diff --git a/arch/x86/kernel/irq.c b/arch/x86/kernel/irq.c
index 3dafc60..cf64546 100644
--- a/arch/x86/kernel/irq.c
+++ b/arch/x86/kernel/irq.c
@@ -128,6 +128,76 @@ int arch_show_interrupts(struct seq_file *p, int prec)
 	return 0;
 }
 
+
+/*
+ * /proc/cpu/<node>/interrupts printing for CPU and arch  specific interrupts
+ * XXX: Could not figure out a way to re-use arch_show_interrupts
+ */
+int arch_show_interrupts_cpu (struct seq_file *p, int prec, int cpu)
+{
+	int j = cpu;
+
+	seq_printf(p, "%*s: ", prec, "NMI");
+	seq_printf(p, "%10u ", irq_stats(j)->__nmi_count);
+	seq_printf(p, "  Non-maskable interrupts\n");
+#ifdef CONFIG_X86_LOCAL_APIC
+	seq_printf(p, "%*s: ", prec, "LOC");
+	seq_printf(p, "%10u ", irq_stats(j)->apic_timer_irqs);
+	seq_printf(p, "  Local timer interrupts\n");
+
+	seq_printf(p, "%*s: ", prec, "SPU");
+	seq_printf(p, "%10u ", irq_stats(j)->irq_spurious_count);
+	seq_printf(p, "  Spurious interrupts\n");
+	seq_printf(p, "%*s: ", prec, "PMI");
+	seq_printf(p, "%10u ", irq_stats(j)->apic_perf_irqs);
+	seq_printf(p, "  Performance monitoring interrupts\n");
+	seq_printf(p, "%*s: ", prec, "IWI");
+	seq_printf(p, "%10u ", irq_stats(j)->apic_irq_work_irqs);
+	seq_printf(p, "  IRQ work interrupts\n");
+	seq_printf(p, "%*s: ", prec, "RTR");
+	seq_printf(p, "%10u ", irq_stats(j)->icr_read_retry_count);
+	seq_printf(p, "  APIC ICR read retries\n");
+#endif
+	if (x86_platform_ipi_callback) {
+		seq_printf(p, "%*s: ", prec, "PLT");
+		seq_printf(p, "%10u ", irq_stats(j)->x86_platform_ipis);
+		seq_printf(p, "  Platform interrupts\n");
+	}
+#ifdef CONFIG_SMP
+	seq_printf(p, "%*s: ", prec, "RES");
+	seq_printf(p, "%10u ", irq_stats(j)->irq_resched_count);
+	seq_printf(p, "  Rescheduling interrupts\n");
+	seq_printf(p, "%*s: ", prec, "CAL");
+	seq_printf(p, "%10u ", irq_stats(j)->irq_call_count);
+	seq_printf(p, "  Function call interrupts\n");
+	seq_printf(p, "%*s: ", prec, "TLB");
+	seq_printf(p, "%10u ", irq_stats(j)->irq_tlb_count);
+	seq_printf(p, "  TLB shootdowns\n");
+#endif
+#ifdef CONFIG_X86_THERMAL_VECTOR
+	seq_printf(p, "%*s: ", prec, "TRM");
+	seq_printf(p, "%10u ", irq_stats(j)->irq_thermal_count);
+	seq_printf(p, "  Thermal event interrupts\n");
+#endif
+#ifdef CONFIG_X86_MCE_THRESHOLD
+	seq_printf(p, "%*s: ", prec, "THR");
+	seq_printf(p, "%10u ", irq_stats(j)->irq_threshold_count);
+	seq_printf(p, "  Threshold APIC interrupts\n");
+#endif
+#ifdef CONFIG_X86_MCE
+	seq_printf(p, "%*s: ", prec, "MCE");
+	seq_printf(p, "%10u ", per_cpu(mce_exception_count, j));
+	seq_printf(p, "  Machine check exceptions\n");
+	seq_printf(p, "%*s: ", prec, "MCP");
+	seq_printf(p, "%10u ", per_cpu(mce_poll_count, j));
+	seq_printf(p, "  Machine check polls\n");
+#endif
+	seq_printf(p, "%*s: %10u\n", prec, "ERR", atomic_read(&irq_err_count));
+#if defined(CONFIG_X86_IO_APIC)
+	seq_printf(p, "%*s: %10u\n", prec, "MIS", atomic_read(&irq_mis_count));
+#endif
+	return 0;
+}
 /*
  * /proc/stat helpers
  */
diff --git a/include/linux/interrupt.h b/include/linux/interrupt.h
index 2aea5d2..27e15cc 100644
--- a/include/linux/interrupt.h
+++ b/include/linux/interrupt.h
@@ -695,6 +695,7 @@ static inline void init_irq_proc(void)
 struct seq_file;
 int show_interrupts(struct seq_file *p, void *v);
 int arch_show_interrupts(struct seq_file *p, int prec);
+int arch_show_interrupts_cpu(struct seq_file *p, int prec, int cpu);
 
 extern int early_irq_init(void);
 extern int arch_probe_nr_irqs(void);
-- 
1.7.3.4

[PATCH 3/3] Modified generic show_interrupts to display per-CPU interrupts information

[Sasikantha babu]

Signed-off-by: Sasikantha babu <sasikanth.v19@gmail.com>
---
 kernel/irq/proc.c |   39 ++++++++++++++++++++++++++++++---------
 1 files changed, 30 insertions(+), 9 deletions(-)

diff --git a/kernel/irq/proc.c b/kernel/irq/proc.c
index 4bd4faa..bc5eaeb 100644
--- a/kernel/irq/proc.c
+++ b/kernel/irq/proc.c
@@ -410,6 +410,10 @@ int __weak arch_show_interrupts(struct seq_file *p, int prec)
 {
 	return 0;
 }
+int __weak arch_show_interrupts_cpu (struct seq_file *p, int prec, int cpu)
+{
+	return 0;
+}
 
 #ifndef ACTUAL_NR_IRQS
 # define ACTUAL_NR_IRQS nr_irqs
@@ -423,12 +427,20 @@ int show_interrupts(struct seq_file *p, void *v)
 	int i = *(loff_t *) v, j;
 	struct irqaction *action;
 	struct irq_desc *desc;
-
+        int cpu = -1;
+	
 	if (i > ACTUAL_NR_IRQS)
 		return 0;
 
-	if (i == ACTUAL_NR_IRQS)
-		return arch_show_interrupts(p, prec);
+	if (p->private)
+		cpu += (int) p->private;
+
+	if (i == ACTUAL_NR_IRQS) {
+		if (cpu < 0)
+			return arch_show_interrupts (p, prec);
+		else
+			return arch_show_interrupts_cpu (p, prec, cpu);
+	}
 
 	/* print header and calculate the width of the first column */
 	if (i == 0) {
@@ -436,8 +448,11 @@ int show_interrupts(struct seq_file *p, void *v)
 			j *= 10;
 
 		seq_printf(p, "%*s", prec + 8, "");
-		for_each_online_cpu(j)
-			seq_printf(p, "CPU%-8d", j);
+		if (cpu < 0) {
+			for_each_online_cpu(j)
+				seq_printf(p, "CPU%-8d", j);
+		} else
+			seq_printf(p, "CPU%-8d", cpu);
 		seq_putc(p, '\n');
 	}
 
@@ -446,15 +461,21 @@ int show_interrupts(struct seq_file *p, void *v)
 		return 0;
 
 	raw_spin_lock_irqsave(&desc->lock, flags);
-	for_each_online_cpu(j)
-		any_count |= kstat_irqs_cpu(i, j);
+	if (cpu < 0) {
+		for_each_online_cpu(j)
+			any_count |= kstat_irqs_cpu(i, j);
+	} else
+		any_count |= kstat_irqs_cpu(i, cpu);
 	action = desc->action;
 	if (!action && !any_count)
 		goto out;
 
 	seq_printf(p, "%*d: ", prec, i);
-	for_each_online_cpu(j)
-		seq_printf(p, "%10u ", kstat_irqs_cpu(i, j));
+	if (cpu < 0) {
+		for_each_online_cpu(j)
+			seq_printf(p, "%10u ", kstat_irqs_cpu(i, j));
+	} else
+		seq_printf(p, "%10u ", kstat_irqs_cpu(i, cpu));
 
 	if (desc->irq_data.chip) {
 		if (desc->irq_data.chip->irq_print_chip)
-- 
1.7.3.4

Re: [PATCH 1/3] Added new Proc entry to display per-cpu interrupts

[Thomas Gleixner]

On Tue, 17 Apr 2012, Sasikantha babu wrote:

Sigh. I told you before, that a subject line of a patch which starts
with: "Added ..." is useless.

Please use the proper prefix for the subsystem and a reasonable
subject.

> As the number of cores increases the output of the /proc/interrupts losing the readability.
> 

And please use a proper mail client which does line wraps around 78 chars.

> As the number of cores increases the output of the /proc/interrupts
> losing the readability.

That's how it should look like.

> I think instead of showing complete information of all cores in a
> single proc file spliting the information based on per-CPU will be
> more useful (especially for people who wants to track per-cpu
> information in real-time systems).

That has nothing to do with real time systems. /proc/interrupts is
unreadable on any machine with a large number of cores.

What the heck is wrong with

cat /proc/interrupts | grep $YOUR_IRQ_OF_INTEREST | awk '{ print $CPU_ROW }'


Thanks,

	tglx