[BUG] selftests/powerpc/eeh: eeh-basic breaks the device hosting rootfs, ext4 goes read-only and the test harness crashes

[BUG] selftests/powerpc/eeh: eeh-basic breaks the device hosting rootfs, ext4 goes read-only and the test harness crashes

[Venkat Rao Bagalkote]

Hello!!!

IBM CI has hit a failure that takes down the test harness, While running 
the powerpc EEH selftest on a bare-metal POWER system.

The selftest injects an error into the PCI device that hosts the root 
filesystem. The filesystem goes read-only and the run dies.

[ 1112.079091] EEH: Frozen PHB#3-PE#1fd detected
[ 1112.079905] aacraid 0003:01:00.0: aacraid: PCI error detected 2
[ 1112.140830] I/O error, dev sdb, sector 3906906472 op 0x1:(WRITE) ...
[ 1112.140957] Aborting journal on device sdb2-8.
[ 1112.140977] EXT4-fs error (device sdb2) in ext4_dirty_inode:7656: IO 
failure
[ 1112.176664] EXT4-fs error (device sdb2): ext4_journal_check_start:98: 
... Detected aborted journal
[ 1187.736385] aacraid 0003:01:00.0: aacraid: PCI error - slot reset
[ 1217.244306] EXT4-fs (sdb2): Remounting filesystem read-only
[ 1220.583495] aacraid 0003:01:00.0: aacraid: PCI error - resume

We need to make appropriate changes to selftest, so that it skips the 
adapter, if rootfs is hosted on it.


SetUp:

System: IBM 9006-22C (POWER9), PowerNV, bare metal
Firmware: skiboot-v6.8-45-g8246de863
Kernel: 7.4.0-rc2+, upstream master at 067c5c24cef2 ("Merge tag 
'page_alloc-for-7.4-rc2'", 2026-11-10)


Issue1: Selftest should skip the adapter that host the rootfs

eeh_can_break() already skips ahci, for similar reason, but has no check 
for a device that backs /. A friendly LLM has suggested

below change, which I tested and it works.

0003:01:00.0, Skipped: hosts the root filesystem

Other adapters are tested as before and the run completes

diff --git a/tools/testing/selftests/powerpc/eeh/eeh-functions.sh 
b/tools/testing/selftests/powerpc/eeh/eeh-functions.sh
index 70daa3925dcb..f9db1f9be05a 100644
--- a/tools/testing/selftests/powerpc/eeh/eeh-functions.sh
+++ b/tools/testing/selftests/powerpc/eeh/eeh-functions.sh
@@ -62,6 +62,42 @@ eeh_test_prep() {
      echo 5000 > /sys/kernel/debug/powerpc/eeh_max_freezes
  }

+# Print the resolved sysfs path of a block device followed by the paths of
+# every device stacked underneath it (dm, md, etc).
+eeh_blk_sysfs_paths() {
+    local sys="$1"
+    local s
+
+    realpath "$sys"
+    for s in "$sys"/slaves/* ; do
+        if [ -e "$s" ] ; then
+            eeh_blk_sysfs_paths "$s"
+        fi
+    done
+}
+
+# Succeeds if the root filesystem lives on a disk behind any function 
of the
+# slot containing the given PCI device.
+eeh_hosts_rootfs() {
+    local slot="${1%.*}"
+    local majmin src sys
+
+    # Use the last "/" entry so an initramfs rootfs entry is ignored.
+    majmin="$(awk '$5 == "/" { mm = $3 } END { print mm }' 
/proc/self/mountinfo)"
+    src="$(awk '$5 == "/" { for (i = 6; i < NF; i++) if ($i == "-") s = 
$(i + 2) } END { print s }' /proc/self/mountinfo)"
+
+    if [ -e "/sys/dev/block/$majmin" ] ; then
+        sys="/sys/dev/block/$majmin"
+    elif [ -b "$src" ] ; then
+        # e.g. btrfs reports an anonymous major:minor
+        sys="/sys/class/block/$(basename "$(realpath "$src")")"
+    else
+        return 1
+    fi
+
+    eeh_blk_sysfs_paths "$sys" 2>/dev/null | grep -q "/${slot}\.[0-7]/"
+}
+
  eeh_can_break() {
      # skip bridges since we can't recover them (yet...)
      if [ -e "/sys/bus/pci/devices/$dev/pci_bus" ] ; then
@@ -78,6 +114,14 @@ eeh_can_break() {
          return 1;
      fi

+    # Breaking the device that hosts the root filesystem makes I/O fail
+    # during recovery, which can abort the journal and leave / read-only
+    # (and kill the test harness). Leave it alone.
+    if eeh_hosts_rootfs "$1" ; then
+        log "$1, Skipped: hosts the root filesystem"
+        return 1;
+    fi
+
      # Don't inject errosr into an already-frozen PE. This happens with
      # PEs that contain multiple PCI devices (e.g. multi-function cards)
      # and injecting new errors during the recovery process will probably



Issue2: Some adapters take longer time to recover.


Once the rootfs device was skipped, the run completed, but with the 
default 60 s wait three adapters were reported as Failed to recover!,
and the summary was 3 devices failed to recover (6 tested). Checking the 
PE state afterwards showed them healthy .
I re-ran with EEH_MAX_WAIT=240, and all of them recovered:

Result with the 240 s limit: 0 devices failed to recover (6 tested).

BRCM/Mellonox/Intel Adapters took 78s/77s/75s respectively to recover. 
So I think its better to increase default timeout to 120s.

diff --git a/tools/testing/selftests/powerpc/eeh/eeh-functions.sh 
b/tools/testing/selftests/powerpc/eeh/eeh-functions.sh
index 70daa3925dcb..1f1d1c8d73bc 100644
--- a/tools/testing/selftests/powerpc/eeh/eeh-functions.sh
+++ b/tools/testing/selftests/powerpc/eeh/eeh-functions.sh
@@ -110,13 +110,13 @@ eeh_one_dev() {
      # is a no-op.
      echo $dev >/sys/kernel/debug/powerpc/eeh_dev_check

-    # Default to a 60s timeout when waiting for a device to recover. This
+    # Default to a 120s timeout when waiting for a device to recover. This
      # is an arbitrary default which can be overridden by setting the
      # EEH_MAX_WAIT environmental variable when required.

      # The current record holder for longest recovery time is:
-    #  "Adaptec Series 8 12G SAS/PCIe 3" at 39 seconds
-    max_wait=${EEH_MAX_WAIT:=60}
+    #  "Broadcom / LSI SAS3008" at 78 seconds
+    max_wait=${EEH_MAX_WAIT:=120}

      for i in `seq 0 ${max_wait}` ; do
          if pe_ok $dev ; then


Please let me know, if the fix for the two issues are good enough, I can 
send a formal patch.



Regards,

Venkat.