[PATCH v3 0/6] notifier error injection

[PATCH v3 0/6] notifier error injection

[Akinobu Mita]

This provides kernel modules that can be used to test the error handling
of notifier call chain failures by injecting artifical errors to the
following notifier chain callbacks.

 * CPU notifier
 * PM notifier
 * memory hotplug notifier
 * powerpc pSeries reconfig notifier

These modules create their own directory on debugfs and each of the files
in the directory represents an event which can be failed and contains the
error code.  If the notifier call chain should be failed with some events
notified, write the error code to the files.

Example: Inject CPU offline error (-1 == -EPERM)

	# cd /sys/kernel/debug/cpu-notifier-error-inject
	# echo -1 > CPU_UP_PREPARE
	# echo 0 > /sys/devices/system/cpu/cpu1/online
	bash: echo: write error: Operation not permitted

Changelog:

* v3
- rewrite to be kernel modules instead of initializing at late_initcall()s
  (it makes the diffstat look different but most code remains unchanged)
- export err_inject_notifier_block_{init,cleanup} for modules
- export pSeries_reconfig_notifier_{,un}register symbols for a module
- notifier priority can be specified as a module parameter
- add testing scripts in tools/testing/fault-injection

* v2
- "PM: Improve error code of pm_notifier_call_chain()" is now in -next
- "debugfs: add debugfs_create_int" is dropped
- put a comment in err_inject_notifier_block_init()
- only allow valid errno to be injected (-MAX_ERRNO <= errno <= 0)
- improve Kconfig help text
- make CONFIG_PM_NOTIFIER_ERROR_INJECTION visible even if PM_DEBUG is disabled
- make CONFIG_PM_NOTIFIER_ERROR_INJECTION default if PM_DEBUG is enabled

Akinobu Mita (6):
  fault-injection: notifier error injection
  cpu: rewrite cpu-notifier-error-inject module
  PM: PM notifier error injection module
  memory: memory notifier error injection module
  powerpc: pSeries reconfig notifier error injection module
  fault-injection: add notifier error injection testing scripts

 arch/powerpc/platforms/pseries/reconfig.c        |    2 +
 include/linux/notifier.h                         |   25 ++++
 kernel/notifier.c                                |   83 +++++++++++
 lib/Kconfig.debug                                |   84 +++++++++++-
 lib/Makefile                                     |    4 +
 lib/cpu-notifier-error-inject.c                  |   56 +++-----
 lib/memory-notifier-error-inject.c               |   45 ++++++
 lib/pSeries-reconfig-notifier-error-inject.c     |   48 +++++++
 lib/pm-notifier-error-inject.c                   |   46 ++++++
 tools/testing/fault-injection/cpu-notifier.sh    |  162 +++++++++++++++++++++
 tools/testing/fault-injection/memory-notifier.sh |  163 ++++++++++++++++++++++
 11 files changed, 680 insertions(+), 38 deletions(-)
 create mode 100644 lib/memory-notifier-error-inject.c
 create mode 100644 lib/pSeries-reconfig-notifier-error-inject.c
 create mode 100644 lib/pm-notifier-error-inject.c
 create mode 100755 tools/testing/fault-injection/cpu-notifier.sh
 create mode 100755 tools/testing/fault-injection/memory-notifier.sh

-- 
1.7.4.4

[PATCH v3 1/6] fault-injection: notifier error injection

[Akinobu Mita]

The notifier error injection provides the ability to inject artifical
errors to specified notifier chain callbacks.  It is useful to test the
error handling of notifier call chain failures.

This adds common basic functions to define which type of events can be
fail and to initialize the debugfs interface to control what error code
should be returned and which event should be failed.

Signed-off-by: Akinobu Mita <akinobu.mita@gmail.com>
Cc: Pavel Machek <pavel@ucw.cz>
Cc: "Rafael J. Wysocki" <rjw@sisk.pl>
Cc: linux-pm@lists.linux-foundation.org
Cc: Greg KH <greg@kroah.com>
Cc: linux-mm@kvack.org
Cc: Benjamin Herrenschmidt <benh@kernel.crashing.org>
Cc: Paul Mackerras <paulus@samba.org>
Cc: linuxppc-dev@lists.ozlabs.org
---
* v3
- export err_inject_notifier_block_{init,cleanup} for modules

 include/linux/notifier.h |   25 ++++++++++++++
 kernel/notifier.c        |   83 ++++++++++++++++++++++++++++++++++++++++++++++
 lib/Kconfig.debug        |   11 ++++++
 3 files changed, 119 insertions(+), 0 deletions(-)

diff --git a/include/linux/notifier.h b/include/linux/notifier.h
index c0688b0..51882d6 100644
--- a/include/linux/notifier.h
+++ b/include/linux/notifier.h
@@ -278,5 +278,30 @@ extern struct blocking_notifier_head reboot_notifier_list;
 #define VT_UPDATE		0x0004 /* A bigger update occurred */
 #define VT_PREWRITE		0x0005 /* A char is about to be written to the console */
 
+#ifdef CONFIG_NOTIFIER_ERROR_INJECTION
+
+struct err_inject_notifier_action {
+	unsigned long val;
+	int error;
+	const char *name;
+};
+
+#define ERR_INJECT_NOTIFIER_ACTION(action)	\
+	.name = #action, .val = (action),
+
+struct err_inject_notifier_block {
+	struct notifier_block nb;
+	struct dentry *dir;
+	struct err_inject_notifier_action actions[];
+	/* The last slot must be terminated with zero sentinel */
+};
+
+extern int err_inject_notifier_block_init(struct err_inject_notifier_block *enb,
+				const char *name, int priority);
+extern void err_inject_notifier_block_cleanup(
+				struct err_inject_notifier_block *enb);
+
+#endif /* CONFIG_NOTIFIER_ERROR_INJECTION */
+
 #endif /* __KERNEL__ */
 #endif /* _LINUX_NOTIFIER_H */
diff --git a/kernel/notifier.c b/kernel/notifier.c
index 2488ba7..8dcb2cc 100644
--- a/kernel/notifier.c
+++ b/kernel/notifier.c
@@ -5,6 +5,7 @@
 #include <linux/rcupdate.h>
 #include <linux/vmalloc.h>
 #include <linux/reboot.h>
+#include <linux/debugfs.h>
 
 /*
  *	Notifier list for kernel code which wants to be called
@@ -584,3 +585,85 @@ int unregister_die_notifier(struct notifier_block *nb)
 	return atomic_notifier_chain_unregister(&die_chain, nb);
 }
 EXPORT_SYMBOL_GPL(unregister_die_notifier);
+
+#ifdef CONFIG_NOTIFIER_ERROR_INJECTION
+
+static int debugfs_errno_set(void *data, u64 val)
+{
+	*(int *)data = clamp_t(int, val, -MAX_ERRNO, 0);
+	return 0;
+}
+
+static int debugfs_errno_get(void *data, u64 *val)
+{
+	*val = *(int *)data;
+	return 0;
+}
+
+DEFINE_SIMPLE_ATTRIBUTE(fops_errno, debugfs_errno_get, debugfs_errno_set,
+			"%lld\n");
+
+static struct dentry *debugfs_create_errno(const char *name, mode_t mode,
+				struct dentry *parent, int *value)
+{
+	return debugfs_create_file(name, mode, parent, value, &fops_errno);
+}
+
+static int err_inject_notifier_callback(struct notifier_block *nb,
+				unsigned long val, void *p)
+{
+	int err = 0;
+	struct err_inject_notifier_block *enb =
+		container_of(nb, struct err_inject_notifier_block, nb);
+	struct err_inject_notifier_action *action;
+
+	for (action = enb->actions; action->name; action++) {
+		if (action->val == val) {
+			err = action->error;
+			break;
+		}
+	}
+	if (err) {
+		printk(KERN_INFO "Injecting error (%d) to %s\n",
+			err, action->name);
+	}
+
+	return notifier_from_errno(err);
+}
+
+int err_inject_notifier_block_init(struct err_inject_notifier_block *enb,
+				const char *name, int priority)
+{
+	struct err_inject_notifier_action *action;
+	mode_t mode = S_IFREG | S_IRUSR | S_IWUSR;
+
+	enb->nb.notifier_call = err_inject_notifier_callback;
+	enb->nb.priority = priority;
+
+	enb->dir = debugfs_create_dir(name, NULL);
+	if (!enb->dir)
+		return -ENOMEM;
+
+	for (action = enb->actions; action->name; action++) {
+		/*
+		 * Create debugfs r/w file containing action->error. If
+		 * notifier call chain is called with action->val, it will
+		 * fail with the error code
+		 */
+		if (!debugfs_create_errno(action->name, mode, enb->dir,
+					&action->error)) {
+			debugfs_remove_recursive(enb->dir);
+			return -ENOMEM;
+		}
+	}
+	return 0;
+}
+EXPORT_SYMBOL_GPL(err_inject_notifier_block_init);
+
+void err_inject_notifier_block_cleanup(struct err_inject_notifier_block *enb)
+{
+	debugfs_remove_recursive(enb->dir);
+}
+EXPORT_SYMBOL_GPL(err_inject_notifier_block_cleanup);
+
+#endif /* CONFIG_NOTIFIER_ERROR_INJECTION */
diff --git a/lib/Kconfig.debug b/lib/Kconfig.debug
index c0cb9c4..2a62c5a 100644
--- a/lib/Kconfig.debug
+++ b/lib/Kconfig.debug
@@ -1021,6 +1021,17 @@ config LKDTM
 	Documentation on how to use the module can be found in
 	Documentation/fault-injection/provoke-crashes.txt
 
+config NOTIFIER_ERROR_INJECTION
+	bool "Notifier error injection"
+	depends on DEBUG_KERNEL
+	select DEBUG_FS
+	help
+	  This option provides the ability to inject artifical errors to
+	  specified notifier chain callbacks. It is useful to test the error
+	  handling of notifier call chain failures.
+
+	  Say N if unsure.
+
 config CPU_NOTIFIER_ERROR_INJECT
 	tristate "CPU notifier error injection module"
 	depends on HOTPLUG_CPU && DEBUG_KERNEL
-- 
1.7.4.4

[PATCH v3 2/6] cpu: rewrite cpu-notifier-error-inject module

[Akinobu Mita]

Rewrite existing cpu-notifier-error-inject module to use debugfs based
brand-new framework.

This change removes cpu_up_prepare_error and cpu_down_prepare_error module
parameters which were used to specify error code to be injected.  We could
keep these module parameters for backward compatibility by module_param_cb
but it seems overkill for this module.

This provides the ability to inject artifical errors to CPU notifier chain
callbacks.  It is controlled through debugfs interface under
/sys/kernel/debug/cpu-notifier-error-inject/

Each of the files in the directory represents an event which can be failed
and contains the error code.  If the notifier call chain should be failed
with some events notified, write the error code to the files.

Example1: inject CPU offline error (-1 == -EPERM)

	# echo -1 > /sys/kernel/debug/cpu-notifier-error-inject/CPU_UP_PREPARE
	# echo 0 > /sys/devices/system/cpu/cpu1/online
	bash: echo: write error: Operation not permitted

Example2: inject CPU online error (-2 == -ENOENT)

	# echo -2 > /sys/kernel/debug/cpu-notifier-error-inject/CPU_DOWN_PREPARE
	# echo 1 > /sys/devices/system/cpu/cpu1/online
	bash: echo: write error: No such file or directory

Signed-off-by: Akinobu Mita <akinobu.mita@gmail.com>
---
* v3
- rewrite to be kernel modules instead of initializing at late_initcall()s
- notifier priority can be specified as a module parameter

 lib/Kconfig.debug               |   18 +++++++++++-
 lib/cpu-notifier-error-inject.c |   56 ++++++++++++++-------------------------
 2 files changed, 36 insertions(+), 38 deletions(-)

diff --git a/lib/Kconfig.debug b/lib/Kconfig.debug
index 2a62c5a..55d56bb 100644
--- a/lib/Kconfig.debug
+++ b/lib/Kconfig.debug
@@ -1034,10 +1034,24 @@ config NOTIFIER_ERROR_INJECTION
 
 config CPU_NOTIFIER_ERROR_INJECT
 	tristate "CPU notifier error injection module"
-	depends on HOTPLUG_CPU && DEBUG_KERNEL
+	depends on HOTPLUG_CPU && NOTIFIER_ERROR_INJECTION
 	help
 	  This option provides a kernel module that can be used to test
-	  the error handling of the cpu notifiers
+	  the error handling of the cpu notifiers by injecting artifical
+	  errors to CPU notifier chain callbacks.  It is controlled through
+	  debugfs interface under /sys/kernel/debug/cpu-notifier-error-inject
+
+	  Each of the files in the directory represents an event which can be
+	  failed and contains the error code.  If the notifier call chain
+	  should be failed with some events notified, write the error code to
+	  the files.
+
+	  Example: Inject CPU offline error (-1 == -EPERM)
+
+	  # cd /sys/kernel/debug/cpu-notifier-error-inject
+	  # echo -1 > CPU_UP_PREPARE
+	  # echo 0 > /sys/devices/system/cpu/cpu1/online
+	  bash: echo: write error: Operation not permitted
 
 	  To compile this code as a module, choose M here: the module will
 	  be called cpu-notifier-error-inject.
diff --git a/lib/cpu-notifier-error-inject.c b/lib/cpu-notifier-error-inject.c
index 4dc2032..1601b72 100644
--- a/lib/cpu-notifier-error-inject.c
+++ b/lib/cpu-notifier-error-inject.c
@@ -4,55 +4,39 @@
 #include <linux/notifier.h>
 
 static int priority;
-static int cpu_up_prepare_error;
-static int cpu_down_prepare_error;
-
 module_param(priority, int, 0);
 MODULE_PARM_DESC(priority, "specify cpu notifier priority");
 
-module_param(cpu_up_prepare_error, int, 0644);
-MODULE_PARM_DESC(cpu_up_prepare_error,
-		"specify error code to inject CPU_UP_PREPARE action");
-
-module_param(cpu_down_prepare_error, int, 0644);
-MODULE_PARM_DESC(cpu_down_prepare_error,
-		"specify error code to inject CPU_DOWN_PREPARE action");
-
-static int err_inject_cpu_callback(struct notifier_block *nfb,
-				unsigned long action, void *hcpu)
-{
-	int err = 0;
-
-	switch (action) {
-	case CPU_UP_PREPARE:
-	case CPU_UP_PREPARE_FROZEN:
-		err = cpu_up_prepare_error;
-		break;
-	case CPU_DOWN_PREPARE:
-	case CPU_DOWN_PREPARE_FROZEN:
-		err = cpu_down_prepare_error;
-		break;
+static struct err_inject_notifier_block err_inject_cpu_notifier = {
+	.actions = {
+		{ ERR_INJECT_NOTIFIER_ACTION(CPU_UP_PREPARE) },
+		{ ERR_INJECT_NOTIFIER_ACTION(CPU_UP_PREPARE_FROZEN) },
+		{ ERR_INJECT_NOTIFIER_ACTION(CPU_DOWN_PREPARE) },
+		{ ERR_INJECT_NOTIFIER_ACTION(CPU_DOWN_PREPARE_FROZEN) },
+		{}
 	}
-	if (err)
-		printk(KERN_INFO "Injecting error (%d) at cpu notifier\n", err);
-
-	return notifier_from_errno(err);
-}
-
-static struct notifier_block err_inject_cpu_notifier = {
-	.notifier_call = err_inject_cpu_callback,
 };
 
 static int err_inject_init(void)
 {
-	err_inject_cpu_notifier.priority = priority;
+	int err;
+
+	err = err_inject_notifier_block_init(&err_inject_cpu_notifier,
+				"cpu-notifier-error-inject", priority);
+	if (err)
+		return err;
+
+	err = register_hotcpu_notifier(&err_inject_cpu_notifier.nb);
+	if (err)
+		err_inject_notifier_block_cleanup(&err_inject_cpu_notifier);
 
-	return register_hotcpu_notifier(&err_inject_cpu_notifier);
+	return err;
 }
 
 static void err_inject_exit(void)
 {
-	unregister_hotcpu_notifier(&err_inject_cpu_notifier);
+	unregister_hotcpu_notifier(&err_inject_cpu_notifier.nb);
+	err_inject_notifier_block_cleanup(&err_inject_cpu_notifier);
 }
 
 module_init(err_inject_init);
-- 
1.7.4.4

[PATCH v3 3/6] PM: PM notifier error injection module

[Akinobu Mita]

This provides the ability to inject artifical errors to PM notifier chain
callbacks.  It is controlled through debugfs interface under
/sys/kernel/debug/pm-notifier-error-inject/

Each of the files in the directory represents an event which can be
failed and contains the error code.  If the notifier call chain should
be failed with some events notified, write the error code to the files.

Example: Inject PM suspend error (-12 = -ENOMEM)

	# cd /sys/kernel/debug/pm-notifier-error-inject
	# echo -12 > PM_SUSPEND_PREPARE
	# echo mem > /sys/power/state
	bash: echo: write error: Cannot allocate memory

Signed-off-by: Akinobu Mita <akinobu.mita@gmail.com>
Cc: Pavel Machek <pavel@ucw.cz>
Cc: "Rafael J. Wysocki" <rjw@sisk.pl>
Cc: linux-pm@lists.linux-foundation.org
---
* v3
- rewrite to be kernel modules instead of initializing at late_initcall()s
- notifier priority can be specified as a module parameter

 lib/Kconfig.debug              |   21 ++++++++++++++++++
 lib/Makefile                   |    1 +
 lib/pm-notifier-error-inject.c |   46 ++++++++++++++++++++++++++++++++++++++++
 3 files changed, 68 insertions(+), 0 deletions(-)
 create mode 100644 lib/pm-notifier-error-inject.c

diff --git a/lib/Kconfig.debug b/lib/Kconfig.debug
index 55d56bb..69ecd50 100644
--- a/lib/Kconfig.debug
+++ b/lib/Kconfig.debug
@@ -1058,6 +1058,27 @@ config CPU_NOTIFIER_ERROR_INJECT
 
 	  If unsure, say N.
 
+config PM_NOTIFIER_ERROR_INJECT
+	tristate "PM notifier error injection module"
+	depends on PM && NOTIFIER_ERROR_INJECTION
+	default m if PM_DEBUG
+	help
+	  This option provides the ability to inject artifical errors to
+	  PM notifier chain callbacks.  It is controlled through debugfs
+	  interface.
+
+	  Example: Inject PM suspend error (-12 = -ENOMEM)
+
+	  # cd /sys/kernel/debug/pm-notifier-error-inject
+	  # echo -12 > PM_SUSPEND_PREPARE
+	  # echo mem > /sys/power/state
+	  bash: echo: write error: Cannot allocate memory
+
+	  To compile this code as a module, choose M here: the module will
+	  be called pm-notifier-error-inject.
+
+	  If unsure, say N.
+
 config FAULT_INJECTION
 	bool "Fault-injection framework"
 	depends on DEBUG_KERNEL
diff --git a/lib/Makefile b/lib/Makefile
index 892f4e2..4dca6c0 100644
--- a/lib/Makefile
+++ b/lib/Makefile
@@ -90,6 +90,7 @@ obj-$(CONFIG_SWIOTLB) += swiotlb.o
 obj-$(CONFIG_IOMMU_HELPER) += iommu-helper.o
 obj-$(CONFIG_FAULT_INJECTION) += fault-inject.o
 obj-$(CONFIG_CPU_NOTIFIER_ERROR_INJECT) += cpu-notifier-error-inject.o
+obj-$(CONFIG_PM_NOTIFIER_ERROR_INJECT) += pm-notifier-error-inject.o
 
 lib-$(CONFIG_GENERIC_BUG) += bug.o
 
diff --git a/lib/pm-notifier-error-inject.c b/lib/pm-notifier-error-inject.c
new file mode 100644
index 0000000..821b3db
--- /dev/null
+++ b/lib/pm-notifier-error-inject.c
@@ -0,0 +1,46 @@
+#include <linux/kernel.h>
+#include <linux/suspend.h>
+#include <linux/module.h>
+#include <linux/notifier.h>
+
+static int priority;
+module_param(priority, int, 0);
+MODULE_PARM_DESC(priority, "specify PM notifier priority");
+
+static struct err_inject_notifier_block err_inject_pm_notifier = {
+	.actions = {
+		{ ERR_INJECT_NOTIFIER_ACTION(PM_HIBERNATION_PREPARE) },
+		{ ERR_INJECT_NOTIFIER_ACTION(PM_SUSPEND_PREPARE) },
+		{ ERR_INJECT_NOTIFIER_ACTION(PM_RESTORE_PREPARE) },
+		{}
+	}
+};
+
+static int err_inject_init(void)
+{
+	int err;
+
+	err = err_inject_notifier_block_init(&err_inject_pm_notifier,
+				"pm-notifier-error-inject", priority);
+	if (err)
+		return err;
+
+	err = register_pm_notifier(&err_inject_pm_notifier.nb);
+	if (err)
+		err_inject_notifier_block_cleanup(&err_inject_pm_notifier);
+
+	return err;
+}
+
+static void err_inject_exit(void)
+{
+	unregister_pm_notifier(&err_inject_pm_notifier.nb);
+	err_inject_notifier_block_cleanup(&err_inject_pm_notifier);
+}
+
+module_init(err_inject_init);
+module_exit(err_inject_exit);
+
+MODULE_DESCRIPTION("PM notifier error injection module");
+MODULE_LICENSE("GPL");
+MODULE_AUTHOR("Akinobu Mita <akinobu.mita@gmail.com>");
-- 
1.7.4.4

[PATCH v3 4/6] memory: memory notifier error injection module

[Akinobu Mita]

This provides the ability to inject artifical errors to memory hotplug
notifier chain callbacks.  It is controlled through debugfs interface
under /sys/kernel/debug/memory-notifier-error-inject/

Each of the files in the directory represents an event which can be failed
and contains the error code.  If the notifier call chain should be failed
with some events notified, write the error code to the files.

Example: Inject memory hotplug offline error (-12 == -ENOMEM)

	# cd /sys/kernel/debug/memory-notifier-error-inject
	# echo -12 > MEM_GOING_OFFLINE
	# echo offline > /sys/devices/system/memory/memoryXXX/state
	bash: echo: write error: Cannot allocate memory

Signed-off-by: Akinobu Mita <akinobu.mita@gmail.com>
Cc: Greg KH <greg@kroah.com>
Cc: linux-mm@kvack.org
---
* v3
- rewrite to be kernel modules instead of initializing at late_initcall()s
- notifier priority can be specified as a module parameter

 lib/Kconfig.debug                  |   20 ++++++++++++++++
 lib/Makefile                       |    1 +
 lib/memory-notifier-error-inject.c |   45 ++++++++++++++++++++++++++++++++++++
 3 files changed, 66 insertions(+), 0 deletions(-)
 create mode 100644 lib/memory-notifier-error-inject.c

diff --git a/lib/Kconfig.debug b/lib/Kconfig.debug
index 69ecd50..a2b0856 100644
--- a/lib/Kconfig.debug
+++ b/lib/Kconfig.debug
@@ -1079,6 +1079,26 @@ config PM_NOTIFIER_ERROR_INJECT
 
 	  If unsure, say N.
 
+config MEMORY_NOTIFIER_ERROR_INJECT
+	tristate "Memory hotplug notifier error injection module"
+	depends on MEMORY_HOTPLUG_SPARSE && NOTIFIER_ERROR_INJECTION
+	help
+	  This option provides the ability to inject artifical errors to
+	  memory hotplug notifier chain callbacks.  It is controlled through
+	  debugfs interface.
+
+	  Example: Inject memory hotplug offline error (-12 == -ENOMEM)
+
+	  # cd /sys/kernel/debug/memory-notifier-error-inject
+	  # echo -12 > MEM_GOING_OFFLINE
+	  # echo offline > /sys/devices/system/memory/memoryXXX/state
+	  bash: echo: write error: Cannot allocate memory
+
+	  To compile this code as a module, choose M here: the module will
+	  be called memory-notifier-error-inject.
+
+	  If unsure, say N.
+
 config FAULT_INJECTION
 	bool "Fault-injection framework"
 	depends on DEBUG_KERNEL
diff --git a/lib/Makefile b/lib/Makefile
index 4dca6c0..f28914b 100644
--- a/lib/Makefile
+++ b/lib/Makefile
@@ -91,6 +91,7 @@ obj-$(CONFIG_IOMMU_HELPER) += iommu-helper.o
 obj-$(CONFIG_FAULT_INJECTION) += fault-inject.o
 obj-$(CONFIG_CPU_NOTIFIER_ERROR_INJECT) += cpu-notifier-error-inject.o
 obj-$(CONFIG_PM_NOTIFIER_ERROR_INJECT) += pm-notifier-error-inject.o
+obj-$(CONFIG_MEMORY_NOTIFIER_ERROR_INJECT) += memory-notifier-error-inject.o
 
 lib-$(CONFIG_GENERIC_BUG) += bug.o
 
diff --git a/lib/memory-notifier-error-inject.c b/lib/memory-notifier-error-inject.c
new file mode 100644
index 0000000..76f0b94
--- /dev/null
+++ b/lib/memory-notifier-error-inject.c
@@ -0,0 +1,45 @@
+#include <linux/kernel.h>
+#include <linux/memory.h>
+#include <linux/module.h>
+#include <linux/notifier.h>
+
+static int priority;
+module_param(priority, int, 0);
+MODULE_PARM_DESC(priority, "specify memory notifier priority");
+
+static struct err_inject_notifier_block err_inject_memory_notifier = {
+	.actions = {
+		{ ERR_INJECT_NOTIFIER_ACTION(MEM_GOING_ONLINE) },
+		{ ERR_INJECT_NOTIFIER_ACTION(MEM_GOING_OFFLINE) },
+		{}
+	}
+};
+
+static int err_inject_init(void)
+{
+	int err;
+
+	err = err_inject_notifier_block_init(&err_inject_memory_notifier,
+				"memory-notifier-error-inject", priority);
+	if (err)
+		return err;
+
+	err = register_memory_notifier(&err_inject_memory_notifier.nb);
+	if (err)
+		err_inject_notifier_block_cleanup(&err_inject_memory_notifier);
+
+	return err;
+}
+
+static void err_inject_exit(void)
+{
+	unregister_memory_notifier(&err_inject_memory_notifier.nb);
+	err_inject_notifier_block_cleanup(&err_inject_memory_notifier);
+}
+
+module_init(err_inject_init);
+module_exit(err_inject_exit);
+
+MODULE_DESCRIPTION("memory notifier error injection module");
+MODULE_LICENSE("GPL");
+MODULE_AUTHOR("Akinobu Mita <akinobu.mita@gmail.com>");
-- 
1.7.4.4

[PATCH v3 5/6] powerpc: pSeries reconfig notifier error injection module

[Akinobu Mita]

This provides the ability to inject artifical errors to pSeries reconfig
notifier chain callbacks.  It is controlled through debugfs interface
under /sys/kernel/debug/pSeries-reconfig-notifier-error-inject/

Each of the files in the directory represents an event which can be failed
and contains the error code.  If the notifier call chain should be failed
with some events notified, write the error code to the files.

This module needs pSeries_reconfig_notifier_{,un}register symbols to be
exported.

Signed-off-by: Akinobu Mita <akinobu.mita@gmail.com>
Cc: Benjamin Herrenschmidt <benh@kernel.crashing.org>
Cc: Paul Mackerras <paulus@samba.org>
Cc: linuxppc-dev@lists.ozlabs.org
---
* v3
- rewrite to be kernel modules instead of initializing at late_initcall()s
- export pSeries_reconfig_notifier_{,un}register symbols for this module
- notifier priority can be specified as a module parameter

 arch/powerpc/platforms/pseries/reconfig.c    |    2 +
 lib/Kconfig.debug                            |   14 +++++++
 lib/Makefile                                 |    2 +
 lib/pSeries-reconfig-notifier-error-inject.c |   48 ++++++++++++++++++++++++++
 4 files changed, 66 insertions(+), 0 deletions(-)
 create mode 100644 lib/pSeries-reconfig-notifier-error-inject.c

diff --git a/arch/powerpc/platforms/pseries/reconfig.c b/arch/powerpc/platforms/pseries/reconfig.c
index 168651a..b9808e9 100644
--- a/arch/powerpc/platforms/pseries/reconfig.c
+++ b/arch/powerpc/platforms/pseries/reconfig.c
@@ -103,11 +103,13 @@ int pSeries_reconfig_notifier_register(struct notifier_block *nb)
 {
 	return blocking_notifier_chain_register(&pSeries_reconfig_chain, nb);
 }
+EXPORT_SYMBOL_GPL(pSeries_reconfig_notifier_register);
 
 void pSeries_reconfig_notifier_unregister(struct notifier_block *nb)
 {
 	blocking_notifier_chain_unregister(&pSeries_reconfig_chain, nb);
 }
+EXPORT_SYMBOL_GPL(pSeries_reconfig_notifier_unregister);
 
 int pSeries_reconfig_notify(unsigned long action, void *p)
 {
diff --git a/lib/Kconfig.debug b/lib/Kconfig.debug
index a2b0856..46298d7 100644
--- a/lib/Kconfig.debug
+++ b/lib/Kconfig.debug
@@ -1099,6 +1099,20 @@ config MEMORY_NOTIFIER_ERROR_INJECT
 
 	  If unsure, say N.
 
+config PSERIES_RECONFIG_NOTIFIER_ERROR_INJECT
+	tristate "pSeries reconfig notifier error injection module"
+	depends on PPC_PSERIES && NOTIFIER_ERROR_INJECTION
+	help
+	  This option provides the ability to inject artifical errors to
+	  pSeries reconfig notifier chain callbacks.  It is controlled
+	  through debugfs interface under
+	  /sys/kernel/debug/pSeries-reconfig-notifier-error-inject/
+
+	  To compile this code as a module, choose M here: the module will
+	  be called memory-notifier-error-inject.
+
+	  If unsure, say N.
+
 config FAULT_INJECTION
 	bool "Fault-injection framework"
 	depends on DEBUG_KERNEL
diff --git a/lib/Makefile b/lib/Makefile
index f28914b..e68cea7 100644
--- a/lib/Makefile
+++ b/lib/Makefile
@@ -92,6 +92,8 @@ obj-$(CONFIG_FAULT_INJECTION) += fault-inject.o
 obj-$(CONFIG_CPU_NOTIFIER_ERROR_INJECT) += cpu-notifier-error-inject.o
 obj-$(CONFIG_PM_NOTIFIER_ERROR_INJECT) += pm-notifier-error-inject.o
 obj-$(CONFIG_MEMORY_NOTIFIER_ERROR_INJECT) += memory-notifier-error-inject.o
+obj-$(CONFIG_PSERIES_RECONFIG_NOTIFIER_ERROR_INJECT) += \
+	pSeries-reconfig-notifier-error-inject.o
 
 lib-$(CONFIG_GENERIC_BUG) += bug.o
 
diff --git a/lib/pSeries-reconfig-notifier-error-inject.c b/lib/pSeries-reconfig-notifier-error-inject.c
new file mode 100644
index 0000000..f4ed2b3
--- /dev/null
+++ b/lib/pSeries-reconfig-notifier-error-inject.c
@@ -0,0 +1,48 @@
+#include <linux/kernel.h>
+#include <linux/module.h>
+#include <linux/notifier.h>
+
+#include <asm/pSeries_reconfig.h>
+
+static int priority;
+module_param(priority, int, 0);
+MODULE_PARM_DESC(priority, "specify pSeries reconfig notifier priority");
+
+static struct err_inject_notifier_block err_inject_reconfig_nb = {
+	.actions = {
+		{ ERR_INJECT_NOTIFIER_ACTION(PSERIES_RECONFIG_ADD) },
+		{ ERR_INJECT_NOTIFIER_ACTION(PSERIES_RECONFIG_REMOVE) },
+		{ ERR_INJECT_NOTIFIER_ACTION(PSERIES_DRCONF_MEM_ADD) },
+		{ ERR_INJECT_NOTIFIER_ACTION(PSERIES_DRCONF_MEM_REMOVE) },
+		{}
+	}
+};
+
+static int err_inject_init(void)
+{
+	int err;
+
+	err = err_inject_notifier_block_init(&err_inject_reconfig_nb,
+			"pSeries-reconfig-notifier-error-inject", priority);
+	if (err)
+		return err;
+
+	err = pSeries_reconfig_notifier_register(&err_inject_reconfig_nb.nb);
+	if (err)
+		err_inject_notifier_block_cleanup(&err_inject_reconfig_nb);
+
+	return err;
+}
+
+static void err_inject_exit(void)
+{
+	pSeries_reconfig_notifier_unregister(&err_inject_reconfig_nb.nb);
+	err_inject_notifier_block_cleanup(&err_inject_reconfig_nb);
+}
+
+module_init(err_inject_init);
+module_exit(err_inject_exit);
+
+MODULE_DESCRIPTION("pSeries reconfig notifier error injection module");
+MODULE_LICENSE("GPL");
+MODULE_AUTHOR("Akinobu Mita <akinobu.mita@gmail.com>");
-- 
1.7.4.4

[PATCH v3 6/6] fault-injection: add notifier error injection testing scripts

[Akinobu Mita]

* tools/testing/fault-injection/cpu-notifier.sh is testing script for
CPU notifier error handling by using cpu-notifier-error-inject.ko.

1. Offline all hot-pluggable CPUs in preparation for testing
2. Test CPU hot-add error handling by injecting notifier errors
3. Online all hot-pluggable CPUs in preparation for testing
4. Test CPU hot-remove error handling by injecting notifier errors

* tools/testing/fault-injection/memory-notifier.sh is doing the same thing
for memory hotplug notifier.

Signed-off-by: Akinobu Mita <akinobu.mita@gmail.com>
Suggested-by: Andrew Morton <akpm@linux-foundation.org>
Cc: Pavel Machek <pavel@ucw.cz>
Cc: "Rafael J. Wysocki" <rjw@sisk.pl>
Cc: linux-pm@lists.linux-foundation.org
Cc: Greg KH <greg@kroah.com>
Cc: linux-mm@kvack.org
Cc: Benjamin Herrenschmidt <benh@kernel.crashing.org>
Cc: Paul Mackerras <paulus@samba.org>
Cc: linuxppc-dev@lists.ozlabs.org
Cc: Américo Wang <xiyou.wangcong@gmail.com>
---
* v3
- new patch

 tools/testing/fault-injection/cpu-notifier.sh    |  162 +++++++++++++++++++++
 tools/testing/fault-injection/memory-notifier.sh |  163 ++++++++++++++++++++++
 2 files changed, 325 insertions(+), 0 deletions(-)
 create mode 100755 tools/testing/fault-injection/cpu-notifier.sh
 create mode 100755 tools/testing/fault-injection/memory-notifier.sh

diff --git a/tools/testing/fault-injection/cpu-notifier.sh b/tools/testing/fault-injection/cpu-notifier.sh
new file mode 100755
index 0000000..be02a85
--- /dev/null
+++ b/tools/testing/fault-injection/cpu-notifier.sh
@@ -0,0 +1,162 @@
+#!/bin/bash
+
+#
+# list all hot-pluggable CPUs
+#
+hotpluggable_cpus()
+{
+	local state=${1:-.\*}
+
+	for cpu in /sys/devices/system/cpu/cpu*; do
+		if [ -f $cpu/online ] && grep -q $state $cpu/online; then
+			echo ${cpu##/*/cpu}
+		fi
+	done
+}
+
+hotplaggable_offline_cpus()
+{
+	hotpluggable_cpus 0
+}
+
+hotpluggable_online_cpus()
+{
+	hotpluggable_cpus 1
+}
+
+cpu_is_online()
+{
+	grep -q 1 /sys/devices/system/cpu/cpu$1/online
+}
+
+cpu_is_offline()
+{
+	grep -q 0 /sys/devices/system/cpu/cpu$1/online
+}
+
+add_cpu()
+{
+	echo 1 > /sys/devices/system/cpu/cpu$1/online
+}
+
+remove_cpu()
+{
+	echo 0 > /sys/devices/system/cpu/cpu$1/online
+}
+
+add_cpu_expect_success()
+{
+	local cpu=$1
+
+	if ! add_cpu $cpu; then
+		echo $FUNCNAME $cpu: unexpected fail >&2
+	elif ! cpu_is_online $cpu; then
+		echo $FUNCNAME $cpu: unexpected offline >&2
+	fi
+}
+
+add_cpu_expect_fail()
+{
+	local cpu=$1
+
+	if add_cpu $cpu 2> /dev/null; then
+		echo $FUNCNAME $cpu: unexpected success >&2
+	elif ! cpu_is_offline $cpu; then
+		echo $FUNCNAME $cpu: unexpected online >&2
+	fi
+}
+
+remove_cpu_expect_success()
+{
+	local cpu=$1
+
+	if ! remove_cpu $cpu; then
+		echo $FUNCNAME $cpu: unexpected fail >&2
+	elif ! cpu_is_offline $cpu; then
+		echo $FUNCNAME $cpu: unexpected offline >&2
+	fi
+}
+
+remove_cpu_expect_fail()
+{
+	local cpu=$1
+
+	if remove_cpu $cpu 2> /dev/null; then
+		echo $FUNCNAME $cpu: unexpected success >&2
+	elif ! cpu_is_online $cpu; then
+		echo $FUNCNAME $cpu: unexpected offline >&2
+	fi
+}
+
+if [ $UID != 0 ]; then
+	echo must be run as root >&2
+	exit 1
+fi
+
+error=-12
+priority=0
+
+while getopts e:p: opt; do
+	case $opt in
+	e)
+		error=$OPTARG
+		;;
+	p)
+		priority=$OPTARG
+		;;
+	esac
+done
+
+if ! [ "$error" -ge -4095 -a "$error" -lt 0 ]; then
+	echo "error code must be -4095 <= errno < 0" >&2
+	exit 1
+fi
+
+DEBUGFS=`mount -t debugfs | head -1 | awk '{ print $3 }'`
+
+if [ ! -d "$DEBUGFS" ]; then
+	echo debugfs is not mounted >&2
+	exit 1
+fi
+
+/sbin/modprobe -r cpu-notifier-error-inject
+/sbin/modprobe -q cpu-notifier-error-inject priority=$priority
+
+if [ ! -d $DEBUGFS/cpu-notifier-error-inject ]; then
+	echo cpu-notifier-error-inject module is not available >&2
+	exit 1
+fi
+
+#
+# Offline all hot-pluggable CPUs
+#
+echo 0 > $DEBUGFS/cpu-notifier-error-inject/CPU_DOWN_PREPARE
+for cpu in `hotpluggable_online_cpus`; do
+	remove_cpu_expect_success $cpu
+done
+
+#
+# Test CPU hot-add error handling (offline => online)
+#
+echo $error > $DEBUGFS/cpu-notifier-error-inject/CPU_UP_PREPARE
+for cpu in `hotplaggable_offline_cpus`; do
+	add_cpu_expect_fail $cpu
+done
+
+#
+# Online all hot-pluggable CPUs
+#
+echo 0 > $DEBUGFS/cpu-notifier-error-inject/CPU_UP_PREPARE
+for cpu in `hotplaggable_offline_cpus`; do
+	add_cpu_expect_success $cpu
+done
+
+#
+# Test CPU hot-remove error handling (online => offline)
+#
+echo $error > $DEBUGFS/cpu-notifier-error-inject/CPU_DOWN_PREPARE
+for cpu in `hotpluggable_online_cpus`; do
+	remove_cpu_expect_fail $cpu
+done
+
+/sbin/modprobe -r cpu-notifier-error-inject
diff --git a/tools/testing/fault-injection/memory-notifier.sh b/tools/testing/fault-injection/memory-notifier.sh
new file mode 100755
index 0000000..b7e7fa5
--- /dev/null
+++ b/tools/testing/fault-injection/memory-notifier.sh
@@ -0,0 +1,163 @@
+#!/bin/bash
+
+#
+# list all hot-pluggable memory
+#
+hotpluggable_memory()
+{
+	local state=${1:-.\*}
+
+	for memory in /sys/devices/system/memory/memory*; do
+		if grep -q 1 $memory/removable &&
+		   grep -q $state $memory/state; then
+			echo ${memory##/*/memory}
+		fi
+	done
+}
+
+hotplaggable_offline_memory()
+{
+	hotpluggable_memory offline
+}
+
+hotpluggable_online_memory()
+{
+	hotpluggable_memory online
+}
+
+memory_is_online()
+{
+	grep -q online /sys/devices/system/memory/memory$1/state
+}
+
+memory_is_offline()
+{
+	grep -q offline /sys/devices/system/memory/memory$1/state
+}
+
+add_memory()
+{
+	echo online > /sys/devices/system/memory/memory$1/state
+}
+
+remove_memory()
+{
+	echo offline > /sys/devices/system/memory/memory$1/state
+}
+
+add_memory_expect_success()
+{
+	local memory=$1
+
+	if ! add_memory $memory; then
+		echo $FUNCNAME $memory: unexpected fail >&2
+	elif ! memory_is_online $memory; then
+		echo $FUNCNAME $memory: unexpected offline >&2
+	fi
+}
+
+add_memory_expect_fail()
+{
+	local memory=$1
+
+	if add_memory $memory 2> /dev/null; then
+		echo $FUNCNAME $memory: unexpected success >&2
+	elif ! memory_is_offline $memory; then
+		echo $FUNCNAME $memory: unexpected online >&2
+	fi
+}
+
+remove_memory_expect_success()
+{
+	local memory=$1
+
+	if ! remove_memory $memory; then
+		echo $FUNCNAME $memory: unexpected fail >&2
+	elif ! memory_is_offline $memory; then
+		echo $FUNCNAME $memory: unexpected offline >&2
+	fi
+}
+
+remove_memory_expect_fail()
+{
+	local memory=$1
+
+	if remove_memory $memory 2> /dev/null; then
+		echo $FUNCNAME $memory: unexpected success >&2
+	elif ! memory_is_online $memory; then
+		echo $FUNCNAME $memory: unexpected offline >&2
+	fi
+}
+
+if [ $UID != 0 ]; then
+	echo must be run as root >&2
+	exit 1
+fi
+
+error=-12
+priority=0
+
+while getopts e:p: opt; do
+	case $opt in
+	e)
+		error=$OPTARG
+		;;
+	p)
+		priority=$OPTARG
+		;;
+	esac
+done
+
+if ! [ "$error" -ge -4095 -a "$error" -lt 0 ]; then
+	echo "error code must be -4095 <= errno < 0" >&2
+	exit 1
+fi
+
+DEBUGFS=`mount -t debugfs | head -1 | awk '{ print $3 }'`
+
+if [ ! -d "$DEBUGFS" ]; then
+	echo debugfs is not mounted >&2
+	exit 1
+fi
+
+/sbin/modprobe -r memory-notifier-error-inject
+/sbin/modprobe -q memory-notifier-error-inject priority=$priority
+
+if [ ! -d $DEBUGFS/memory-notifier-error-inject ]; then
+	echo memory-notifier-error-inject module is not available >&2
+	exit 1
+fi
+
+#
+# Offline all hot-pluggable memory
+#
+echo 0 > $DEBUGFS/memory-notifier-error-inject/MEM_GOING_OFFLINE
+for memory in `hotpluggable_online_memory`; do
+	remove_memory_expect_success $memory
+done
+
+#
+# Test memory hot-add error handling (offline => online)
+#
+echo $error > $DEBUGFS/memory-notifier-error-inject/MEM_GOING_ONLINE
+for memory in `hotplaggable_offline_memory`; do
+	add_memory_expect_fail $memory
+done
+
+#
+# Online all hot-pluggable memory
+#
+echo 0 > $DEBUGFS/memory-notifier-error-inject/MEM_GOING_ONLINE
+for memory in `hotplaggable_offline_memory`; do
+	add_memory_expect_success $memory
+done
+
+#
+# Test memory hot-remove error handling (online => offline)
+#
+echo $error > $DEBUGFS/memory-notifier-error-inject/MEM_GOING_OFFLINE
+for memory in `hotpluggable_online_memory`; do
+	remove_memory_expect_fail $memory
+done
+
+/sbin/modprobe -r memory-notifier-error-inject
-- 
1.7.4.4

Re: [PATCH v3 0/6] notifier error injection

[Pavel Machek]

Hi!

> This provides kernel modules that can be used to test the error handling
> of notifier call chain failures by injecting artifical errors to the
> following notifier chain callbacks.
> 
>  * CPU notifier
>  * PM notifier
>  * memory hotplug notifier
>  * powerpc pSeries reconfig notifier
...

>  11 files changed, 680 insertions(+), 38 deletions(-)

Do we really need this debugging code in mainline? How many bugs did
it find?

-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

Re: [PATCH v3 0/6] notifier error injection

[Akinobu Mita]

[-- Attachment #1: Type: text/plain, Size: 937 bytes --]

2011/8/1 Pavel Machek <pavel@ucw.cz>:
> Hi!
>
>> This provides kernel modules that can be used to test the error handling
>> of notifier call chain failures by injecting artifical errors to the
>> following notifier chain callbacks.
>>
>>  * CPU notifier
>>  * PM notifier
>>  * memory hotplug notifier
>>  * powerpc pSeries reconfig notifier
> ...
>
>>  11 files changed, 680 insertions(+), 38 deletions(-)
>
> Do we really need this debugging code in mainline? How many bugs did
> it find?

I'm seeing a memory hotplug related bug but it also can be reproducible
without this feature.  Repeatedly online/offline memory with attached
memory-hotplug.sh under disk activity like kernel build can easily cause
Oopses.

Aside from that, I didn't find any bugs but I still think this feature is
useful for testing and it just add a little complexity to exiting code.
(Maybe I need to prove it by finding more bugs)

[-- Attachment #2: memory-hotplug.sh --]
[-- Type: application/x-sh, Size: 1952 bytes --]

[-- Attachment #3: messages --]
[-- Type: application/octet-stream, Size: 14650 bytes --]

Aug  2 08:38:25 localhost kernel: [  420.373107] memory offlining 58000 to 5c000 failed
Aug  2 08:40:25 localhost kernel: [  540.310756] memory offlining 5c000 to 60000 failed
Aug  2 08:40:25 localhost kernel: [  540.392301] Offlined Pages 16384
Aug  2 08:40:25 localhost kernel: [  540.473104] Offlined Pages 16384
Aug  2 08:40:25 localhost kernel: [  540.545981] Offlined Pages 16384
Aug  2 08:40:25 localhost kernel: [  540.650516] Offlined Pages 16384
Aug  2 08:42:25 localhost kernel: [  660.587805] memory offlining 70000 to 74000 failed
Aug  2 08:42:25 localhost kernel: [  660.593054] BUG: unable to handle kernel NULL pointer dereference at 00000006
Aug  2 08:42:25 localhost kernel: [  660.593119] IP: [<c0548826>] search_dirblock+0x37/0xa8
Aug  2 08:42:25 localhost kernel: [  660.593165] *pde = 763df067 
Aug  2 08:42:25 localhost kernel: [  660.593190] Oops: 0000 [#1] SMP 
Aug  2 08:42:25 localhost kernel: [  660.593224] Modules linked in: memory_notifier_error_inject fuse ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat bridge stp llc tpm_infineon sunrpc cpufreq_ondemand acpi_cpufreq mperf ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 ip6table_filter ip6_tables ipv6 kvm_intel kvm uinput snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_intel snd_hda_codec snd_hwdep snd_seq snd_seq_device thinkpad_acpi rfkill snd_pcm snd_timer uvcvideo jmb38x_ms snd iTCO_wdt memstick pcspkr snd_page_alloc soundcore iTCO_vendor_support serio_raw r8169 mii videodev microcode wmi i2c_i801 joydev sdhci_pci sdhci mmc_core i915 drm_kms_helper drm i2c_algo_bit i2c_core video [last unloaded: cpu_notifier_error_inject]
Aug  2 08:42:25 localhost kernel: [  660.593830] 
Aug  2 08:42:25 localhost kernel: [  660.593846] Pid: 30759, comm: jd Not tainted 3.0.0+ #6 LENOVO 4444CTO/4444CTO
Aug  2 08:42:25 localhost kernel: [  660.593903] EIP: 0060:[<c0548826>] EFLAGS: 00210212 CPU: 2
Aug  2 08:42:25 localhost kernel: [  660.593941] EIP is at search_dirblock+0x37/0xa8
Aug  2 08:42:25 localhost kernel: [  660.593974] EAX: 00000008 EBX: 00000000 ECX: f0bbea2c EDX: ede998f0
Aug  2 08:42:25 localhost kernel: [  660.594016] ESI: ede998f0 EDI: 00000008 EBP: ef3c5d78 ESP: ef3c5d60
Aug  2 08:42:25 localhost kernel: [  660.594059]  DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068
Aug  2 08:42:25 localhost kernel: [  660.594097] Process jd (pid: 30759, ti=ef3c4000 task=ef39e2b0 task.ti=ef3c4000)
Aug  2 08:42:25 localhost kernel: [  660.594146] Stack:
Aug  2 08:42:25 localhost kernel: [  660.594161]  f0bbea3c ede9acc0 00001000 ede998f0 00000000 00000000 ef3c5e04 c05499b2
Aug  2 08:42:25 localhost kernel: [  660.594236]  00000000 ef3c5e10 ede9acc0 ef3c5e10 00000001 f687ec00 f0bbea2c 00000001
Aug  2 08:42:25 localhost kernel: [  660.594310]  00000001 00000001 ede9acc0 00000000 00000080 ef3c5e40 ef3c5dc0 c0429022
Aug  2 08:42:25 localhost kernel: [  660.594384] Call Trace:
Aug  2 08:42:25 localhost kernel: [  660.594406]  [<c05499b2>] ext4_find_entry+0x2d2/0x34f
Aug  2 08:42:25 localhost kernel: [  660.594444]  [<c0429022>] ? need_resched+0x19/0x23
Aug  2 08:42:25 localhost kernel: [  660.594478]  [<c0429039>] ? should_resched+0xd/0x28
Aug  2 08:42:25 localhost kernel: [  660.594517]  [<c04d4e0e>] ? kmem_cache_alloc+0x56/0xac
Aug  2 08:42:25 localhost kernel: [  660.594553]  [<c0549a60>] ext4_lookup+0x31/0xad
Aug  2 08:42:25 localhost kernel: [  660.594589]  [<c04eb87c>] ? list_add+0xf/0x11
Aug  2 08:42:25 localhost kernel: [  660.594621]  [<c04e4b18>] d_alloc_and_lookup+0x36/0x54
Aug  2 08:42:25 localhost kernel: [  660.594657]  [<c04e5db3>] walk_component+0x1a8/0x356
Aug  2 08:42:25 localhost kernel: [  660.594695]  [<c04e6efc>] do_last+0xe3/0x52d
Aug  2 08:42:25 localhost kernel: [  660.594727]  [<c04e7719>] path_openat+0xa2/0x2a2
Aug  2 08:42:25 localhost kernel: [  660.594762]  [<c04c202e>] ? handle_pte_fault+0x720/0x767
Aug  2 08:42:25 localhost kernel: [  660.594801]  [<c04e795b>] do_filp_open+0x26/0x62
Aug  2 08:42:25 localhost kernel: [  660.594837]  [<c05c7ba9>] ? might_fault+0xd/0xf
Aug  2 08:42:25 localhost kernel: [  660.594869]  [<c05c7d39>] ? strncpy_from_user+0x34/0x4e
Aug  2 08:42:25 localhost kernel: [  660.594908]  [<c04dc9ce>] do_sys_open+0x59/0xd2
Aug  2 08:42:25 localhost kernel: [  660.594941]  [<c04dca6a>] sys_open+0x23/0x2b
Aug  2 08:42:25 localhost kernel: [  660.594975]  [<c07c821f>] sysenter_do_call+0x12/0x28
Aug  2 08:42:25 localhost kernel: [  660.595009] Code: 26 00 8b 79 04 89 45 ec 8b 41 08 89 d6 89 45 e8 8b 45 ec 8b 58 18 8b 42 10 8b 40 10 01 d8 89 45 f0 eb 65 8d 04 3b 39 45 f0 72 50 <0f> b6 43 06 39 c7 75 48 83 3b 00 74 43 8b 45 e8 8d 53 08 89 f9 
Aug  2 08:42:25 localhost kernel: [  660.595380] EIP: [<c0548826>] search_dirblock+0x37/0xa8 SS:ESP 0068:ef3c5d60
Aug  2 08:42:25 localhost kernel: [  660.595434] CR2: 0000000000000006
Aug  2 08:42:25 localhost kernel: [  660.597842] type=1400 audit(1312242145.472:3): avc:  denied  { open } for  pid=30762 comm=72733A6D61696E20513A526567 name="0" dev=devpts ino=3 scontext=system_u:system_r:syslogd_t:s0 tcontext=unconfined_u:object_r:user_devpts_t:s0 tclass=chr_file
Aug  2 08:42:25 localhost kernel: [  660.663141] ---[ end trace 973aaf9210f930e6 ]---
Aug  2 08:44:08 localhost kernel: [  763.002205] BUG: unable to handle kernel NULL pointer dereference at 00000006
Aug  2 08:44:08 localhost kernel: [  763.004302] IP: [<c0548826>] search_dirblock+0x37/0xa8
Aug  2 08:44:08 localhost kernel: [  763.006380] *pde = 7687e067 
Aug  2 08:44:08 localhost kernel: [  763.008452] Oops: 0000 [#2] SMP 
Aug  2 08:44:08 localhost kernel: [  763.010520] Modules linked in: memory_notifier_error_inject fuse ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat bridge stp llc tpm_infineon sunrpc cpufreq_ondemand acpi_cpufreq mperf ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 ip6table_filter ip6_tables ipv6 kvm_intel kvm uinput snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_intel snd_hda_codec snd_hwdep snd_seq snd_seq_device thinkpad_acpi rfkill snd_pcm snd_timer uvcvideo jmb38x_ms snd iTCO_wdt memstick pcspkr snd_page_alloc soundcore iTCO_vendor_support serio_raw r8169 mii videodev microcode wmi i2c_i801 joydev sdhci_pci sdhci mmc_core i915 drm_kms_helper drm i2c_algo_bit i2c_core video [last unloaded: cpu_notifier_error_inject]
Aug  2 08:44:08 localhost kernel: [  763.022125] 
Aug  2 08:44:08 localhost kernel: [  763.024459] Pid: 6658, comm: gnome-session Tainted: G      D     3.0.0+ #6 LENOVO 4444CTO/4444CTO
Aug  2 08:44:08 localhost kernel: [  763.026871] EIP: 0060:[<c0548826>] EFLAGS: 00010216 CPU: 2
Aug  2 08:44:08 localhost kernel: [  763.029286] EIP is at search_dirblock+0x37/0xa8
Aug  2 08:44:08 localhost kernel: [  763.031704] EAX: 00000013 EBX: 00000000 ECX: f0aeba2c EDX: ede84d90
Aug  2 08:44:08 localhost kernel: [  763.034137] ESI: ede84d90 EDI: 00000013 EBP: efe47d5c ESP: efe47d44
Aug  2 08:44:08 localhost kernel: [  763.036575]  DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068
Aug  2 08:44:08 localhost kernel: [  763.039024] Process gnome-session (pid: 6658, ti=efe46000 task=efe1ad80 task.ti=efe46000)
Aug  2 08:44:08 localhost kernel: [  763.041499] Stack:
Aug  2 08:44:08 localhost kernel: [  763.043970]  f0aeba3c f10f16d8 00001000 ede84d90 f10f16d8 efe47dac efe47de8 c05497cb
Aug  2 08:44:08 localhost kernel: [  763.046501]  00002000 efe47df4 efe47d7c efe47df4 efe47d7c f687ec00 f0aeba2c 00000000
Aug  2 08:44:08 localhost kernel: [  763.048947]  f687ec00 00000002 00000000 00000000 00000000 00000000 efe47da4 c0429022
Aug  2 08:44:08 localhost kernel: [  763.051276] Call Trace:
Aug  2 08:44:08 localhost kernel: [  763.053564]  [<c05497cb>] ext4_find_entry+0xeb/0x34f
Aug  2 08:44:08 localhost kernel: [  763.055848]  [<c0429022>] ? need_resched+0x19/0x23
Aug  2 08:44:08 localhost kernel: [  763.058132]  [<c0429039>] ? should_resched+0xd/0x28
Aug  2 08:44:08 localhost kernel: [  763.060406]  [<c04d4e0e>] ? kmem_cache_alloc+0x56/0xac
Aug  2 08:44:08 localhost kernel: [  763.062672]  [<c0549a60>] ext4_lookup+0x31/0xad
Aug  2 08:44:08 localhost kernel: [  763.064887]  [<c04eb87c>] ? list_add+0xf/0x11
Aug  2 08:44:08 localhost kernel: [  763.067046]  [<c04e4b18>] d_alloc_and_lookup+0x36/0x54
Aug  2 08:44:08 localhost kernel: [  763.069192]  [<c04e5db3>] walk_component+0x1a8/0x356
Aug  2 08:44:08 localhost kernel: [  763.071327]  [<c04e6431>] path_lookupat+0x9e/0x2cb
Aug  2 08:44:08 localhost kernel: [  763.073434]  [<c0429022>] ? need_resched+0x19/0x23
Aug  2 08:44:08 localhost kernel: [  763.075532]  [<c07c1692>] ? _cond_resched+0xd/0x21
Aug  2 08:44:08 localhost kernel: [  763.077610]  [<c05c7ba9>] ? might_fault+0xd/0xf
Aug  2 08:44:08 localhost kernel: [  763.079670]  [<c04e6680>] do_path_lookup+0x22/0x80
Aug  2 08:44:08 localhost kernel: [  763.081739]  [<c04e6ac0>] user_path_at+0x3f/0x64
Aug  2 08:44:08 localhost kernel: [  763.083798]  [<c04e088b>] vfs_fstatat+0x56/0x7d
Aug  2 08:44:08 localhost kernel: [  763.085850]  [<c04e08cd>] vfs_lstat+0x1b/0x1d
Aug  2 08:44:08 localhost kernel: [  763.087889]  [<c04e0b12>] sys_lstat64+0x19/0x2d
Aug  2 08:44:08 localhost kernel: [  763.089930]  [<c0429039>] ? should_resched+0xd/0x28
Aug  2 08:44:08 localhost kernel: [  763.091848]  [<c05c7c41>] ? copy_to_user+0x44/0x4b
Aug  2 08:44:08 localhost kernel: [  763.093628]  [<c043cd68>] ? sys_gettimeofday+0x30/0x63
Aug  2 08:44:08 localhost kernel: [  763.095381]  [<c07c821f>] sysenter_do_call+0x12/0x28
Aug  2 08:44:08 localhost kernel: [  763.097128] Code: 26 00 8b 79 04 89 45 ec 8b 41 08 89 d6 89 45 e8 8b 45 ec 8b 58 18 8b 42 10 8b 40 10 01 d8 89 45 f0 eb 65 8d 04 3b 39 45 f0 72 50 <0f> b6 43 06 39 c7 75 48 83 3b 00 74 43 8b 45 e8 8d 53 08 89 f9 
Aug  2 08:44:08 localhost kernel: [  763.101203] EIP: [<c0548826>] search_dirblock+0x37/0xa8 SS:ESP 0068:efe47d44
Aug  2 08:44:08 localhost kernel: [  763.103187] CR2: 0000000000000006
Aug  2 08:44:08 localhost kernel: [  763.105169] ---[ end trace 973aaf9210f930e7 ]---
Aug  2 08:44:24 localhost abrt: Kerneloops: Reported 2 kernel oopses to Abrt
Aug  2 08:44:24 localhost abrtd: Directory 'kerneloops-1312242264-6300-2' creation detected
Aug  2 08:44:24 localhost kernel: [  779.486584] BUG: unable to handle kernel NULL pointer dereference at 00000e00
Aug  2 08:44:24 localhost kernel: [  779.490316] IP: [<c054278e>] ext4_iget+0x61/0x5f9
Aug  2 08:44:24 localhost kernel: [  779.494825] *pde = 75038067 
Aug  2 08:44:24 localhost kernel: [  779.499290] Oops: 0000 [#3] SMP 
Aug  2 08:44:24 localhost kernel: [  779.503759] Modules linked in: memory_notifier_error_inject fuse ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat bridge stp llc tpm_infineon sunrpc cpufreq_ondemand acpi_cpufreq mperf ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 ip6table_filter ip6_tables ipv6 kvm_intel kvm uinput snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_intel snd_hda_codec snd_hwdep snd_seq snd_seq_device thinkpad_acpi rfkill snd_pcm snd_timer uvcvideo jmb38x_ms snd iTCO_wdt memstick pcspkr snd_page_alloc soundcore iTCO_vendor_support serio_raw r8169 mii videodev microcode wmi i2c_i801 joydev sdhci_pci sdhci mmc_core i915 drm_kms_helper drm i2c_algo_bit i2c_core video [last unloaded: cpu_notifier_error_inject]
Aug  2 08:44:24 localhost kernel: [  779.527001] 
Aug  2 08:44:24 localhost kernel: [  779.531442] Pid: 6300, comm: abrtd Tainted: G      D     3.0.0+ #6 LENOVO 4444CTO/4444CTO
Aug  2 08:44:24 localhost kernel: [  779.531451] EIP: 0060:[<c054278e>] EFLAGS: 00210206 CPU: 1
Aug  2 08:44:24 localhost kernel: [  779.531457] EIP is at ext4_iget+0x61/0x5f9
Aug  2 08:44:24 localhost kernel: [  779.531462] EAX: ede9a7f0 EBX: edfac0c0 ECX: 00000000 EDX: 00000000
Aug  2 08:44:24 localhost kernel: [  779.531466] ESI: 00000e00 EDI: f3950e18 EBP: f18a5de8 ESP: f18a5dac
Aug  2 08:44:24 localhost kernel: [  779.531471]  DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068
Aug  2 08:44:24 localhost kernel: [  779.531477] Process abrtd (pid: 6300, ti=f18a4000 task=f6a4b140 task.ti=f18a4000)
Aug  2 08:44:24 localhost kernel: [  779.531480] Stack:
Aug  2 08:44:24 localhost kernel: [  779.531483]  f18a5db4 f4c03dc0 00200286 00200286 f687ec00 c04d4e0e 000000d0 f3eca000
Aug  2 08:44:24 localhost kernel: [  779.531492]  f13e0618 ede9a7f0 00000e00 000000b1 f11fb5a0 0016280f f3950e18 f18a5e04
Aug  2 08:44:24 localhost kernel: [  779.531500]  c0549aa2 f687ec00 f02e2024 c07e0540 f3950e18 f11fb5a0 f18a5e1c c04e4b18
Aug  2 08:44:24 localhost kernel: [  779.531508] Call Trace:
Aug  2 08:44:24 localhost kernel: [  779.531520]  [<c04d4e0e>] ? kmem_cache_alloc+0x56/0xac
Aug  2 08:44:24 localhost kernel: [  779.531528]  [<c0549aa2>] ext4_lookup+0x73/0xad
Aug  2 08:44:24 localhost kernel: [  779.531535]  [<c04e4b18>] d_alloc_and_lookup+0x36/0x54
Aug  2 08:44:24 localhost kernel: [  779.531541]  [<c04e5db3>] walk_component+0x1a8/0x356
Aug  2 08:44:24 localhost kernel: [  779.531547]  [<c04e6431>] path_lookupat+0x9e/0x2cb
Aug  2 08:44:24 localhost kernel: [  779.531555]  [<c0429022>] ? need_resched+0x19/0x23
Aug  2 08:44:24 localhost kernel: [  779.531564]  [<c07c1692>] ? _cond_resched+0xd/0x21
Aug  2 08:44:24 localhost kernel: [  779.531571]  [<c05c7ba9>] ? might_fault+0xd/0xf
Aug  2 08:44:24 localhost kernel: [  779.531577]  [<c04e6680>] do_path_lookup+0x22/0x80
Aug  2 08:44:24 localhost kernel: [  779.531583]  [<c04e6ac0>] user_path_at+0x3f/0x64
Aug  2 08:44:24 localhost kernel: [  779.531591]  [<c04e088b>] vfs_fstatat+0x56/0x7d
Aug  2 08:44:24 localhost kernel: [  779.531597]  [<c04e08cd>] vfs_lstat+0x1b/0x1d
Aug  2 08:44:24 localhost kernel: [  779.531603]  [<c04e0b12>] sys_lstat64+0x19/0x2d
Aug  2 08:44:24 localhost kernel: [  779.531611]  [<c04d81a0>] ? is_target_pte_for_mc+0xf7/0x222
Aug  2 08:44:24 localhost kernel: [  779.531620]  [<c07c821f>] sysenter_do_call+0x12/0x28
Aug  2 08:44:24 localhost kernel: [  779.531624] Code: 08 0f 84 b2 05 00 00 8d 55 e8 31 c9 c7 45 e8 00 00 00 00 e8 39 ea ff ff 85 c0 89 c2 0f 88 79 05 00 00 8b 45 e8 8b 70 18 03 75 ec <8b> 06 66 89 03 0f b7 4e 02 8b 43 10 89 4b 04 0f b7 56 18 89 53 
Aug  2 08:44:24 localhost kernel: [  779.531662] EIP: [<c054278e>] ext4_iget+0x61/0x5f9 SS:ESP 0068:f18a5dac
Aug  2 08:44:24 localhost kernel: [  779.531670] CR2: 0000000000000e00
Aug  2 08:44:24 localhost kernel: [  779.531701] ---[ end trace 973aaf9210f930e8 ]---
Aug  2 08:44:25 localhost kernel: [  780.522559] memory offlining c000 to 10000 failed
Aug  2 08:44:25 localhost kernel: [  780.629926] Offlined Pages 16384
Aug  2 08:44:25 localhost kernel: [  780.718095] Offlined Pages 16384
Aug  2 08:44:25 localhost kernel: [  780.721326] memory offlining 18000 to 1c000 failed
Aug  2 08:44:25 localhost kernel: [  780.722136] memory offlining 1c000 to 20000 failed

Re: [PATCH v3 0/6] notifier error injection

[Akinobu Mita]

2011/8/2 Akinobu Mita <akinobu.mita@gmail.com>:

> Aside from that, I didn't find any bugs but I still think this feature is
> useful for testing and it just add a little complexity to exiting code.
> (Maybe I need to prove it by finding more bugs)

As for the complexity, I can minimize it by moving the common functions
to lib/notifier-error-inject.c and lib/notifier-error-inject.h.
It is cleaner than adding them to kernel/notifier.c and
include/linux/notifier.h with #ifdef CONFIG_NOTIFIER_ERROR_INJECTION
as v3 patch set does.