mirror of https://lore.kernel.org/lkml/
 help / color / mirror / Atom feed
* (no subject)
@ 2026-08-23  9:55 Mathieu Fluhr
  2026-08-23 10:12 ` Mathieu Fluhr
  0 siblings, 1 reply; 11+ messages in thread
From: Mathieu Fluhr @ 2026-08-23  9:55 UTC (permalink / raw)
  To: mathias.nyman; +Cc: stern, gregkh, linux-usb, regressions, stable, linux-kernel

Hi all,

I'm reporting a regression that causes a hard platform reset on my workstation,
But, before digging into the technical details, I would like to first re-trace
how I came to this particular commit.

As an AOSP developer I am compiling daily different versions of AOSP on it,
mostly building an Android (Automotive) emulator for quick "code-build-test"
runs. I use Ubuntu 24.04 LTS for that, updating to the latest HWE kernels that
Ubuntu provide there (6.8, 6.11, 6.14 etc. until 7.0 recently), with no manual
modifications applied.

3 weeks ago, I needed to analyze and issue happening inside Android 11,
building an emulator for a simple Android phone [1]. But when I started using
this emulator, first with the latest 7.0 kernel I noticed several hard crashes,
sometimes just freezing my workstation (with fans full on, but sometimes fans
full off), but also sometimes automatically rebooting it.

After a few days of deep investigations (To be honest, I first suspected an
issue with the nivida driver), I found out that a pattern to reproduce this
quickly was to let the computer idle with the emulator running. The crash was
always occurring under 20/25 minutes, most of the time letting it idle for less
than 10 was even sufficient.

This made me a bit curious, and looking a bit deeper (and with a little help
of AI) I was able to find a workaround. forbidding my CPU to enter C2 state,
using "processor.max_cstate=1" argument. Using this, I was not able to
reproduce the crash for more than an hour, but I did not pursue there very
much: As a developer I hate workarounds :)

I instead started to get "back in time" on a clean and up-to-date fresh Ubuntu
24.04 install, reverting back to "good old" kernel versions, since I could not
convince myself that first my CPU was dying and second that the issue has always
been there. I confirmed that 6.8 and 6.11 (again Ubuntu HWE packages) work
fine, but the crash was reproducible using 6.14 and above.

I then got my hands dirty, and started to test different mainline kernel
prebuilt packages up to a point where I confirmed that 6.12.35 worked fine, but
6.12.40 not. I then bisected both versions, ensuring a good case meant the
emulator was idling without any crash for 1 hour minimum. This lead at the end
to the following commit:

  aec11e5f9c452ef64e2c113637ab89a67a5ceb62
  ("usb: hub: fix detection of high tier USB3 devices behind suspended hubs")
  [mainline: 8f5b7e2bec1c36578fdaa74a6951833541103e27]

Being quite astonished that something related to C2 state was triggered by an
USB patch, I then tested the latest 7.0 kernel, this time using the
"usbcore.autosuspend=-1" argument instead. To my surprise, I could not
reproduce the crash, even with the exact same emulator idling for 2 hours.

Also, something very astonishing, that I still cannot fully understand today:
 1. "priming" my system with a 30 seconds (!) run of a modern Android
     emulator [2] cleared the issue: After closing the 15 emulator and starting
     the 11, I could let it idle for again more than an hour. It seems even not
     be related to the 'kvm' kernel modules, since removing the module and
     re-inserting it between both emulator did not change a thing.
 2. A few times (I did not really invest debugging this TBH), the crash even
    occurred shortly (2-3 minutes) after I closed the Android 11 emulator.

Now, the technical details of my setup...

Hardware / software
===================

CPU:        AMD Ryzen Threadripper 7970X (Storm Peak, family 19h)
Microcode:  0x0a10810c
Board:      ASUS Pro WS TRX50-SAGE WIFI, BIOS 1203 and 1317 (both affected)
Memory:     128 GB DDR5 ECC RDIMM @ 4800 MT/s (JEDEC, no XMP/EXPO)
GPU:        NVIDIA RTX 3080 Ti (also reproduced with RTX 5080)
Distro:     Ubuntu 24.04 and 26.04 with 595-open NVIDIA driver
USB:        8 onboard xHCI controllers; only a USB keyboard and mouse
            attached

Error signature
===============

On the boot following each crash I could always see the following lines in
the dmesg logs:
---8<-------------------------------------------------------------------------
  x86/amd: Previous system reset reason [0x88000800]: an uncorrected
           error caused a data fabric sync flood event
  x86/amd: Previous system reset reason [0x88000800]: a software sync
           flood event occurred
---8<-------------------------------------------------------------------------

When I was fortunate enough and had an automatic reboot, this was also inside:
---8<-------------------------------------------------------------------------
  [Hardware Error]: event severity: fatal
  [Hardware Error]:   section_type: IA32/X64 processor error
  [Hardware Error]:    Error Structure Type: cache error
  [Hardware Error]:    Check Information: 0x000000000602001f
  [Hardware Error]:     Transaction Type: 2, Generic
  [Hardware Error]:     Level: 0
  [Hardware Error]:     Processor Context Corrupt: true
  [Hardware Error]:     Uncorrected: true
  mce: [Hardware Error]: CPU 34: Machine Check: 0 Bank 5: aea0000000000108
  mce: [Hardware Error]: TSC 0 ADDR 1ffffff98e8ee38 MISC d0150fff00000000
       SYND 4d000000 IPID 500b020049b00
---8<-------------------------------------------------------------------------

The signature is bit-identical across every occurrence except for the
reporting CPU, which varies (observed on CPUs 1, 3, 7, 16, 32, 34, 35).


Reproducer
==========

1. Boot an affected kernel with default idle settings (C2 available,
   no max_cstate restriction).
2. Start an old and/or 32/64-bit Android emulator (QEMU/KVM guest) and leave it
   idle on the launcher screen. Nothing else running.
3. System hard-resets within 20 minutes.

Under sustained CPU load the fault never occurs; it requires the system to be
idle. turbostat confirms ~99% C2 residency across all cores in the crashing
condition.


Bisection
=========

---8<-------------------------------------------------------------------------
git bisect start
# status: waiting for both good and bad commits
# good: [783cd2c3dca8b6c434e955b84c20c8940588dc68] Linux 6.12.35
git bisect good 783cd2c3dca8b6c434e955b84c20c8940588dc68
# status: waiting for bad commit, 1 good commit known
# bad: [d90ecb2b1308b3e362ec4c21ff7cf0a051b445df] Linux 6.12.40
git bisect bad d90ecb2b1308b3e362ec4c21ff7cf0a051b445df
# good: [3ce57d493dd81ecc27033c7a918cc0d0ddaa8edc] ata: libata-acpi:
Do not assume 40 wire cable if no devices are enabled
git bisect good 3ce57d493dd81ecc27033c7a918cc0d0ddaa8edc
# good: [bbd385b65f9e56cab1243e753510a99a59110083] net/mlx5e: Add new
prio for promiscuous mode
git bisect good bbd385b65f9e56cab1243e753510a99a59110083
# good: [2a76bc2b24ed889a689fb1c9015307bf16aafb5b] smb: client: fix
use-after-free in crypt_message when using async crypto
git bisect good 2a76bc2b24ed889a689fb1c9015307bf16aafb5b
# good: [95a13b0a6b042ac5333b67f59af1dffe7b71137b] riscv:
traps_misaligned: properly sign extend value in misaligned load
handler
git bisect good 95a13b0a6b042ac5333b67f59af1dffe7b71137b
# good: [40b5b4ba8ed87c0bfb6268c10589777652ebde4c] drm/mediatek: Add
wait_event_timeout when disabling plane
git bisect good 40b5b4ba8ed87c0bfb6268c10589777652ebde4c
# bad: [affb46db59f908474a211f23953c3b9109f0d647] net: libwx: fix
multicast packets received count
git bisect bad affb46db59f908474a211f23953c3b9109f0d647
# good: [ee56da95f8962b86fec4ef93f866e64c8d025a58] btrfs: fix block
group refcount race in btrfs_create_pending_block_groups()
git bisect good ee56da95f8962b86fec4ef93f866e64c8d025a58
# bad: [bf71baa3cfe754c6e99ad62711b9fb3e3da33517] usb: hub: Fix
flushing of delayed work used for post resume purposes
git bisect bad bf71baa3cfe754c6e99ad62711b9fb3e3da33517
# bad: [e11359640090ab26e4153cce583f50e0b8979f0d] usb: hub: Fix
flushing and scheduling of delayed work that tunes runtime pm
git bisect bad e11359640090ab26e4153cce583f50e0b8979f0d
# bad: [aec11e5f9c452ef64e2c113637ab89a67a5ceb62] usb: hub: fix
detection of high tier USB3 devices behind suspended hubs
git bisect bad aec11e5f9c452ef64e2c113637ab89a67a5ceb62
# first bad commit: [aec11e5f9c452ef64e2c113637ab89a67a5ceb62] usb:
hub: fix detection of high tier USB3 devices behind suspended hubs
---8<-------------------------------------------------------------------------

(Please note that I did not perform a full "revert test", since the aec11e5f9c45
commit could not be cleanly reverted on both 6.12.40 and 7.0.0)


Workarounds
===========

Either of these prevents the crash on an affected kernel:
  usbcore.autosuspend=-1     (disables USB runtime PM)
  processor.max_cstate=1     (prevents C2 entry)


Ruled out
=========

- GPU/driver: reproduced with RTX 5080 (595-open module) and RTX 3080 Ti
              (both 595-open and 595 proprietary modules)
- ASUS USB4 PCIe add-in card: reproduces in ~5 min with the card physically
                              removed
- AVIC: kvm_amd avic=N on both good and bad kernels
- TSA mitigation: tsa=off verified applied
  (/sys/devices/system/cpu/vulnerabilities/tsa = "Vulnerable"), still crashes
- Memory: stock JEDEC 4800, no XMP/EXPO, no ECC errors logged
  (ras-mc-ctl reports zero CE/UE)
- Thermal/power: ~47-50 C at time of crash, system idle; 1600W Seasonic PSU
- Hardware health: multi-hour AOSP compiles (all 64 threads, heavy cache
  load) and repeated GPU stress runs complete without error


I hope this gives you enough details to start looking at what could cause this
weird behavior. Just FYI, when asked about hardware damage, AI suggested more
something like "a CPU-level microcode erratum in the deep-idle path on this
platform", but being old-school, I tend to always triple-check what AI tells
before claiming it myself :)

Thanks and Kind Regards,
Mathieu

------
[1] using android-11.0.0_r46 release tag and "sdk_phone_x86_64" as lunch target
[3] using android-15.0.0_r32 release tag and "sdk_phone64_x86_64" as lunch
    target this time, since Google only releases 64-bit only today

^ permalink raw reply	[flat|nested] 11+ messages in thread

* Re:
  2026-08-23  9:55 Mathieu Fluhr
@ 2026-08-23 10:12 ` Mathieu Fluhr
  2026-08-23 10:15   ` [REGRESSION] 6.12.36+: usb: hub: post-resume delayed work triggers > uncorrected MCE / data fabric sync flood on Threadripper 7970X > (bisected to aec11e5f9c45) Mathieu Fluhr
  0 siblings, 1 reply; 11+ messages in thread
From: Mathieu Fluhr @ 2026-08-23 10:12 UTC (permalink / raw)
  To: mathias.nyman; +Cc: stern, gregkh, linux-usb, regressions, stable, linux-kernel

[REGRESSION] 6.12.36+: usb: hub: post-resume delayed work triggers
uncorrected MCE / data fabric sync flood on Threadripper 7970X
(bisected to aec11e5f9c45)

.... this time with the correct subject too.
(apologies for that stupid mistake)

On Sun, Aug 23, 2026 at 11:55 AM Mathieu Fluhr <mathieu.fluhr@gmail.com> wrote:
>
> Hi all,
>
> I'm reporting a regression that causes a hard platform reset on my workstation,
> But, before digging into the technical details, I would like to first re-trace
> how I came to this particular commit.
>
> As an AOSP developer I am compiling daily different versions of AOSP on it,
> mostly building an Android (Automotive) emulator for quick "code-build-test"
> runs. I use Ubuntu 24.04 LTS for that, updating to the latest HWE kernels that
> Ubuntu provide there (6.8, 6.11, 6.14 etc. until 7.0 recently), with no manual
> modifications applied.
>
> 3 weeks ago, I needed to analyze and issue happening inside Android 11,
> building an emulator for a simple Android phone [1]. But when I started using
> this emulator, first with the latest 7.0 kernel I noticed several hard crashes,
> sometimes just freezing my workstation (with fans full on, but sometimes fans
> full off), but also sometimes automatically rebooting it.
>
> After a few days of deep investigations (To be honest, I first suspected an
> issue with the nivida driver), I found out that a pattern to reproduce this
> quickly was to let the computer idle with the emulator running. The crash was
> always occurring under 20/25 minutes, most of the time letting it idle for less
> than 10 was even sufficient.
>
> This made me a bit curious, and looking a bit deeper (and with a little help
> of AI) I was able to find a workaround. forbidding my CPU to enter C2 state,
> using "processor.max_cstate=1" argument. Using this, I was not able to
> reproduce the crash for more than an hour, but I did not pursue there very
> much: As a developer I hate workarounds :)
>
> I instead started to get "back in time" on a clean and up-to-date fresh Ubuntu
> 24.04 install, reverting back to "good old" kernel versions, since I could not
> convince myself that first my CPU was dying and second that the issue has always
> been there. I confirmed that 6.8 and 6.11 (again Ubuntu HWE packages) work
> fine, but the crash was reproducible using 6.14 and above.
>
> I then got my hands dirty, and started to test different mainline kernel
> prebuilt packages up to a point where I confirmed that 6.12.35 worked fine, but
> 6.12.40 not. I then bisected both versions, ensuring a good case meant the
> emulator was idling without any crash for 1 hour minimum. This lead at the end
> to the following commit:
>
>   aec11e5f9c452ef64e2c113637ab89a67a5ceb62
>   ("usb: hub: fix detection of high tier USB3 devices behind suspended hubs")
>   [mainline: 8f5b7e2bec1c36578fdaa74a6951833541103e27]
>
> Being quite astonished that something related to C2 state was triggered by an
> USB patch, I then tested the latest 7.0 kernel, this time using the
> "usbcore.autosuspend=-1" argument instead. To my surprise, I could not
> reproduce the crash, even with the exact same emulator idling for 2 hours.
>
> Also, something very astonishing, that I still cannot fully understand today:
>  1. "priming" my system with a 30 seconds (!) run of a modern Android
>      emulator [2] cleared the issue: After closing the 15 emulator and starting
>      the 11, I could let it idle for again more than an hour. It seems even not
>      be related to the 'kvm' kernel modules, since removing the module and
>      re-inserting it between both emulator did not change a thing.
>  2. A few times (I did not really invest debugging this TBH), the crash even
>     occurred shortly (2-3 minutes) after I closed the Android 11 emulator.
>
> Now, the technical details of my setup...
>
> Hardware / software
> ===================
>
> CPU:        AMD Ryzen Threadripper 7970X (Storm Peak, family 19h)
> Microcode:  0x0a10810c
> Board:      ASUS Pro WS TRX50-SAGE WIFI, BIOS 1203 and 1317 (both affected)
> Memory:     128 GB DDR5 ECC RDIMM @ 4800 MT/s (JEDEC, no XMP/EXPO)
> GPU:        NVIDIA RTX 3080 Ti (also reproduced with RTX 5080)
> Distro:     Ubuntu 24.04 and 26.04 with 595-open NVIDIA driver
> USB:        8 onboard xHCI controllers; only a USB keyboard and mouse
>             attached
>
> Error signature
> ===============
>
> On the boot following each crash I could always see the following lines in
> the dmesg logs:
> ---8<-------------------------------------------------------------------------
>   x86/amd: Previous system reset reason [0x88000800]: an uncorrected
>            error caused a data fabric sync flood event
>   x86/amd: Previous system reset reason [0x88000800]: a software sync
>            flood event occurred
> ---8<-------------------------------------------------------------------------
>
> When I was fortunate enough and had an automatic reboot, this was also inside:
> ---8<-------------------------------------------------------------------------
>   [Hardware Error]: event severity: fatal
>   [Hardware Error]:   section_type: IA32/X64 processor error
>   [Hardware Error]:    Error Structure Type: cache error
>   [Hardware Error]:    Check Information: 0x000000000602001f
>   [Hardware Error]:     Transaction Type: 2, Generic
>   [Hardware Error]:     Level: 0
>   [Hardware Error]:     Processor Context Corrupt: true
>   [Hardware Error]:     Uncorrected: true
>   mce: [Hardware Error]: CPU 34: Machine Check: 0 Bank 5: aea0000000000108
>   mce: [Hardware Error]: TSC 0 ADDR 1ffffff98e8ee38 MISC d0150fff00000000
>        SYND 4d000000 IPID 500b020049b00
> ---8<-------------------------------------------------------------------------
>
> The signature is bit-identical across every occurrence except for the
> reporting CPU, which varies (observed on CPUs 1, 3, 7, 16, 32, 34, 35).
>
>
> Reproducer
> ==========
>
> 1. Boot an affected kernel with default idle settings (C2 available,
>    no max_cstate restriction).
> 2. Start an old and/or 32/64-bit Android emulator (QEMU/KVM guest) and leave it
>    idle on the launcher screen. Nothing else running.
> 3. System hard-resets within 20 minutes.
>
> Under sustained CPU load the fault never occurs; it requires the system to be
> idle. turbostat confirms ~99% C2 residency across all cores in the crashing
> condition.
>
>
> Bisection
> =========
>
> ---8<-------------------------------------------------------------------------
> git bisect start
> # status: waiting for both good and bad commits
> # good: [783cd2c3dca8b6c434e955b84c20c8940588dc68] Linux 6.12.35
> git bisect good 783cd2c3dca8b6c434e955b84c20c8940588dc68
> # status: waiting for bad commit, 1 good commit known
> # bad: [d90ecb2b1308b3e362ec4c21ff7cf0a051b445df] Linux 6.12.40
> git bisect bad d90ecb2b1308b3e362ec4c21ff7cf0a051b445df
> # good: [3ce57d493dd81ecc27033c7a918cc0d0ddaa8edc] ata: libata-acpi:
> Do not assume 40 wire cable if no devices are enabled
> git bisect good 3ce57d493dd81ecc27033c7a918cc0d0ddaa8edc
> # good: [bbd385b65f9e56cab1243e753510a99a59110083] net/mlx5e: Add new
> prio for promiscuous mode
> git bisect good bbd385b65f9e56cab1243e753510a99a59110083
> # good: [2a76bc2b24ed889a689fb1c9015307bf16aafb5b] smb: client: fix
> use-after-free in crypt_message when using async crypto
> git bisect good 2a76bc2b24ed889a689fb1c9015307bf16aafb5b
> # good: [95a13b0a6b042ac5333b67f59af1dffe7b71137b] riscv:
> traps_misaligned: properly sign extend value in misaligned load
> handler
> git bisect good 95a13b0a6b042ac5333b67f59af1dffe7b71137b
> # good: [40b5b4ba8ed87c0bfb6268c10589777652ebde4c] drm/mediatek: Add
> wait_event_timeout when disabling plane
> git bisect good 40b5b4ba8ed87c0bfb6268c10589777652ebde4c
> # bad: [affb46db59f908474a211f23953c3b9109f0d647] net: libwx: fix
> multicast packets received count
> git bisect bad affb46db59f908474a211f23953c3b9109f0d647
> # good: [ee56da95f8962b86fec4ef93f866e64c8d025a58] btrfs: fix block
> group refcount race in btrfs_create_pending_block_groups()
> git bisect good ee56da95f8962b86fec4ef93f866e64c8d025a58
> # bad: [bf71baa3cfe754c6e99ad62711b9fb3e3da33517] usb: hub: Fix
> flushing of delayed work used for post resume purposes
> git bisect bad bf71baa3cfe754c6e99ad62711b9fb3e3da33517
> # bad: [e11359640090ab26e4153cce583f50e0b8979f0d] usb: hub: Fix
> flushing and scheduling of delayed work that tunes runtime pm
> git bisect bad e11359640090ab26e4153cce583f50e0b8979f0d
> # bad: [aec11e5f9c452ef64e2c113637ab89a67a5ceb62] usb: hub: fix
> detection of high tier USB3 devices behind suspended hubs
> git bisect bad aec11e5f9c452ef64e2c113637ab89a67a5ceb62
> # first bad commit: [aec11e5f9c452ef64e2c113637ab89a67a5ceb62] usb:
> hub: fix detection of high tier USB3 devices behind suspended hubs
> ---8<-------------------------------------------------------------------------
>
> (Please note that I did not perform a full "revert test", since the aec11e5f9c45
> commit could not be cleanly reverted on both 6.12.40 and 7.0.0)
>
>
> Workarounds
> ===========
>
> Either of these prevents the crash on an affected kernel:
>   usbcore.autosuspend=-1     (disables USB runtime PM)
>   processor.max_cstate=1     (prevents C2 entry)
>
>
> Ruled out
> =========
>
> - GPU/driver: reproduced with RTX 5080 (595-open module) and RTX 3080 Ti
>               (both 595-open and 595 proprietary modules)
> - ASUS USB4 PCIe add-in card: reproduces in ~5 min with the card physically
>                               removed
> - AVIC: kvm_amd avic=N on both good and bad kernels
> - TSA mitigation: tsa=off verified applied
>   (/sys/devices/system/cpu/vulnerabilities/tsa = "Vulnerable"), still crashes
> - Memory: stock JEDEC 4800, no XMP/EXPO, no ECC errors logged
>   (ras-mc-ctl reports zero CE/UE)
> - Thermal/power: ~47-50 C at time of crash, system idle; 1600W Seasonic PSU
> - Hardware health: multi-hour AOSP compiles (all 64 threads, heavy cache
>   load) and repeated GPU stress runs complete without error
>
>
> I hope this gives you enough details to start looking at what could cause this
> weird behavior. Just FYI, when asked about hardware damage, AI suggested more
> something like "a CPU-level microcode erratum in the deep-idle path on this
> platform", but being old-school, I tend to always triple-check what AI tells
> before claiming it myself :)
>
> Thanks and Kind Regards,
> Mathieu
>
> ------
> [1] using android-11.0.0_r46 release tag and "sdk_phone_x86_64" as lunch target
> [3] using android-15.0.0_r32 release tag and "sdk_phone64_x86_64" as lunch
>     target this time, since Google only releases 64-bit only today

^ permalink raw reply	[flat|nested] 11+ messages in thread

* [REGRESSION] 6.12.36+: usb: hub: post-resume delayed work triggers > uncorrected MCE / data fabric sync flood on Threadripper 7970X > (bisected to aec11e5f9c45)
  2026-08-23 10:12 ` Mathieu Fluhr
@ 2026-08-23 10:15   ` Mathieu Fluhr
  2026-08-23 10:36     ` Thorsten Leemhuis
  0 siblings, 1 reply; 11+ messages in thread
From: Mathieu Fluhr @ 2026-08-23 10:15 UTC (permalink / raw)
  Cc: linux-usb, regressions, stable, linux-kernel

On Sun, Aug 23, 2026 at 12:12 PM Mathieu Fluhr <mathieu.fluhr@gmail.com> wrote:
>
> [REGRESSION] 6.12.36+: usb: hub: post-resume delayed work triggers
> uncorrected MCE / data fabric sync flood on Threadripper 7970X
> (bisected to aec11e5f9c45)
>
> .... this time with the correct subject too.
> (apologies for that stupid mistake)
>
> On Sun, Aug 23, 2026 at 11:55 AM Mathieu Fluhr <mathieu.fluhr@gmail.com> wrote:
> >
> > Hi all,
> >
> > I'm reporting a regression that causes a hard platform reset on my workstation,
> > But, before digging into the technical details, I would like to first re-trace
> > how I came to this particular commit.
> >
> > As an AOSP developer I am compiling daily different versions of AOSP on it,
> > mostly building an Android (Automotive) emulator for quick "code-build-test"
> > runs. I use Ubuntu 24.04 LTS for that, updating to the latest HWE kernels that
> > Ubuntu provide there (6.8, 6.11, 6.14 etc. until 7.0 recently), with no manual
> > modifications applied.
> >
> > 3 weeks ago, I needed to analyze and issue happening inside Android 11,
> > building an emulator for a simple Android phone [1]. But when I started using
> > this emulator, first with the latest 7.0 kernel I noticed several hard crashes,
> > sometimes just freezing my workstation (with fans full on, but sometimes fans
> > full off), but also sometimes automatically rebooting it.
> >
> > After a few days of deep investigations (To be honest, I first suspected an
> > issue with the nivida driver), I found out that a pattern to reproduce this
> > quickly was to let the computer idle with the emulator running. The crash was
> > always occurring under 20/25 minutes, most of the time letting it idle for less
> > than 10 was even sufficient.
> >
> > This made me a bit curious, and looking a bit deeper (and with a little help
> > of AI) I was able to find a workaround. forbidding my CPU to enter C2 state,
> > using "processor.max_cstate=1" argument. Using this, I was not able to
> > reproduce the crash for more than an hour, but I did not pursue there very
> > much: As a developer I hate workarounds :)
> >
> > I instead started to get "back in time" on a clean and up-to-date fresh Ubuntu
> > 24.04 install, reverting back to "good old" kernel versions, since I could not
> > convince myself that first my CPU was dying and second that the issue has always
> > been there. I confirmed that 6.8 and 6.11 (again Ubuntu HWE packages) work
> > fine, but the crash was reproducible using 6.14 and above.
> >
> > I then got my hands dirty, and started to test different mainline kernel
> > prebuilt packages up to a point where I confirmed that 6.12.35 worked fine, but
> > 6.12.40 not. I then bisected both versions, ensuring a good case meant the
> > emulator was idling without any crash for 1 hour minimum. This lead at the end
> > to the following commit:
> >
> >   aec11e5f9c452ef64e2c113637ab89a67a5ceb62
> >   ("usb: hub: fix detection of high tier USB3 devices behind suspended hubs")
> >   [mainline: 8f5b7e2bec1c36578fdaa74a6951833541103e27]
> >
> > Being quite astonished that something related to C2 state was triggered by an
> > USB patch, I then tested the latest 7.0 kernel, this time using the
> > "usbcore.autosuspend=-1" argument instead. To my surprise, I could not
> > reproduce the crash, even with the exact same emulator idling for 2 hours.
> >
> > Also, something very astonishing, that I still cannot fully understand today:
> >  1. "priming" my system with a 30 seconds (!) run of a modern Android
> >      emulator [2] cleared the issue: After closing the 15 emulator and starting
> >      the 11, I could let it idle for again more than an hour. It seems even not
> >      be related to the 'kvm' kernel modules, since removing the module and
> >      re-inserting it between both emulator did not change a thing.
> >  2. A few times (I did not really invest debugging this TBH), the crash even
> >     occurred shortly (2-3 minutes) after I closed the Android 11 emulator.
> >
> > Now, the technical details of my setup...
> >
> > Hardware / software
> > ===================
> >
> > CPU:        AMD Ryzen Threadripper 7970X (Storm Peak, family 19h)
> > Microcode:  0x0a10810c
> > Board:      ASUS Pro WS TRX50-SAGE WIFI, BIOS 1203 and 1317 (both affected)
> > Memory:     128 GB DDR5 ECC RDIMM @ 4800 MT/s (JEDEC, no XMP/EXPO)
> > GPU:        NVIDIA RTX 3080 Ti (also reproduced with RTX 5080)
> > Distro:     Ubuntu 24.04 and 26.04 with 595-open NVIDIA driver
> > USB:        8 onboard xHCI controllers; only a USB keyboard and mouse
> >             attached
> >
> > Error signature
> > ===============
> >
> > On the boot following each crash I could always see the following lines in
> > the dmesg logs:
> > ---8<-------------------------------------------------------------------------
> >   x86/amd: Previous system reset reason [0x88000800]: an uncorrected
> >            error caused a data fabric sync flood event
> >   x86/amd: Previous system reset reason [0x88000800]: a software sync
> >            flood event occurred
> > ---8<-------------------------------------------------------------------------
> >
> > When I was fortunate enough and had an automatic reboot, this was also inside:
> > ---8<-------------------------------------------------------------------------
> >   [Hardware Error]: event severity: fatal
> >   [Hardware Error]:   section_type: IA32/X64 processor error
> >   [Hardware Error]:    Error Structure Type: cache error
> >   [Hardware Error]:    Check Information: 0x000000000602001f
> >   [Hardware Error]:     Transaction Type: 2, Generic
> >   [Hardware Error]:     Level: 0
> >   [Hardware Error]:     Processor Context Corrupt: true
> >   [Hardware Error]:     Uncorrected: true
> >   mce: [Hardware Error]: CPU 34: Machine Check: 0 Bank 5: aea0000000000108
> >   mce: [Hardware Error]: TSC 0 ADDR 1ffffff98e8ee38 MISC d0150fff00000000
> >        SYND 4d000000 IPID 500b020049b00
> > ---8<-------------------------------------------------------------------------
> >
> > The signature is bit-identical across every occurrence except for the
> > reporting CPU, which varies (observed on CPUs 1, 3, 7, 16, 32, 34, 35).
> >
> >
> > Reproducer
> > ==========
> >
> > 1. Boot an affected kernel with default idle settings (C2 available,
> >    no max_cstate restriction).
> > 2. Start an old and/or 32/64-bit Android emulator (QEMU/KVM guest) and leave it
> >    idle on the launcher screen. Nothing else running.
> > 3. System hard-resets within 20 minutes.
> >
> > Under sustained CPU load the fault never occurs; it requires the system to be
> > idle. turbostat confirms ~99% C2 residency across all cores in the crashing
> > condition.
> >
> >
> > Bisection
> > =========
> >
> > ---8<-------------------------------------------------------------------------
> > git bisect start
> > # status: waiting for both good and bad commits
> > # good: [783cd2c3dca8b6c434e955b84c20c8940588dc68] Linux 6.12.35
> > git bisect good 783cd2c3dca8b6c434e955b84c20c8940588dc68
> > # status: waiting for bad commit, 1 good commit known
> > # bad: [d90ecb2b1308b3e362ec4c21ff7cf0a051b445df] Linux 6.12.40
> > git bisect bad d90ecb2b1308b3e362ec4c21ff7cf0a051b445df
> > # good: [3ce57d493dd81ecc27033c7a918cc0d0ddaa8edc] ata: libata-acpi:
> > Do not assume 40 wire cable if no devices are enabled
> > git bisect good 3ce57d493dd81ecc27033c7a918cc0d0ddaa8edc
> > # good: [bbd385b65f9e56cab1243e753510a99a59110083] net/mlx5e: Add new
> > prio for promiscuous mode
> > git bisect good bbd385b65f9e56cab1243e753510a99a59110083
> > # good: [2a76bc2b24ed889a689fb1c9015307bf16aafb5b] smb: client: fix
> > use-after-free in crypt_message when using async crypto
> > git bisect good 2a76bc2b24ed889a689fb1c9015307bf16aafb5b
> > # good: [95a13b0a6b042ac5333b67f59af1dffe7b71137b] riscv:
> > traps_misaligned: properly sign extend value in misaligned load
> > handler
> > git bisect good 95a13b0a6b042ac5333b67f59af1dffe7b71137b
> > # good: [40b5b4ba8ed87c0bfb6268c10589777652ebde4c] drm/mediatek: Add
> > wait_event_timeout when disabling plane
> > git bisect good 40b5b4ba8ed87c0bfb6268c10589777652ebde4c
> > # bad: [affb46db59f908474a211f23953c3b9109f0d647] net: libwx: fix
> > multicast packets received count
> > git bisect bad affb46db59f908474a211f23953c3b9109f0d647
> > # good: [ee56da95f8962b86fec4ef93f866e64c8d025a58] btrfs: fix block
> > group refcount race in btrfs_create_pending_block_groups()
> > git bisect good ee56da95f8962b86fec4ef93f866e64c8d025a58
> > # bad: [bf71baa3cfe754c6e99ad62711b9fb3e3da33517] usb: hub: Fix
> > flushing of delayed work used for post resume purposes
> > git bisect bad bf71baa3cfe754c6e99ad62711b9fb3e3da33517
> > # bad: [e11359640090ab26e4153cce583f50e0b8979f0d] usb: hub: Fix
> > flushing and scheduling of delayed work that tunes runtime pm
> > git bisect bad e11359640090ab26e4153cce583f50e0b8979f0d
> > # bad: [aec11e5f9c452ef64e2c113637ab89a67a5ceb62] usb: hub: fix
> > detection of high tier USB3 devices behind suspended hubs
> > git bisect bad aec11e5f9c452ef64e2c113637ab89a67a5ceb62
> > # first bad commit: [aec11e5f9c452ef64e2c113637ab89a67a5ceb62] usb:
> > hub: fix detection of high tier USB3 devices behind suspended hubs
> > ---8<-------------------------------------------------------------------------
> >
> > (Please note that I did not perform a full "revert test", since the aec11e5f9c45
> > commit could not be cleanly reverted on both 6.12.40 and 7.0.0)
> >
> >
> > Workarounds
> > ===========
> >
> > Either of these prevents the crash on an affected kernel:
> >   usbcore.autosuspend=-1     (disables USB runtime PM)
> >   processor.max_cstate=1     (prevents C2 entry)
> >
> >
> > Ruled out
> > =========
> >
> > - GPU/driver: reproduced with RTX 5080 (595-open module) and RTX 3080 Ti
> >               (both 595-open and 595 proprietary modules)
> > - ASUS USB4 PCIe add-in card: reproduces in ~5 min with the card physically
> >                               removed
> > - AVIC: kvm_amd avic=N on both good and bad kernels
> > - TSA mitigation: tsa=off verified applied
> >   (/sys/devices/system/cpu/vulnerabilities/tsa = "Vulnerable"), still crashes
> > - Memory: stock JEDEC 4800, no XMP/EXPO, no ECC errors logged
> >   (ras-mc-ctl reports zero CE/UE)
> > - Thermal/power: ~47-50 C at time of crash, system idle; 1600W Seasonic PSU
> > - Hardware health: multi-hour AOSP compiles (all 64 threads, heavy cache
> >   load) and repeated GPU stress runs complete without error
> >
> >
> > I hope this gives you enough details to start looking at what could cause this
> > weird behavior. Just FYI, when asked about hardware damage, AI suggested more
> > something like "a CPU-level microcode erratum in the deep-idle path on this
> > platform", but being old-school, I tend to always triple-check what AI tells
> > before claiming it myself :)
> >
> > Thanks and Kind Regards,
> > Mathieu
> >
> > ------
> > [1] using android-11.0.0_r46 release tag and "sdk_phone_x86_64" as lunch target
> > [3] using android-15.0.0_r32 release tag and "sdk_phone64_x86_64" as lunch
> >     target this time, since Google only releases 64-bit only today

^ permalink raw reply	[flat|nested] 11+ messages in thread

* Re: [REGRESSION] 6.12.36+: usb: hub: post-resume delayed work triggers > uncorrected MCE / data fabric sync flood on Threadripper 7970X > (bisected to aec11e5f9c45)
  2026-08-23 10:15   ` [REGRESSION] 6.12.36+: usb: hub: post-resume delayed work triggers > uncorrected MCE / data fabric sync flood on Threadripper 7970X > (bisected to aec11e5f9c45) Mathieu Fluhr
@ 2026-08-23 10:36     ` Thorsten Leemhuis
  2026-08-23 11:44       ` Mathieu Fluhr
  2026-08-23 15:17       ` Lovekesh Solanki
  0 siblings, 2 replies; 11+ messages in thread
From: Thorsten Leemhuis @ 2026-08-23 10:36 UTC (permalink / raw)
  To: Mathias Nyman
  Cc: linux-usb, regressions, stable, linux-kernel, Mario Limonciello,
	Forest, Slavik Dev, Michal Pecio, Mathieu Fluhr

On 8/23/26 12:15, Mathieu Fluhr wrote:
> On Sun, Aug 23, 2026 at 12:12 PM Mathieu Fluhr <mathieu.fluhr@gmail.com> wrote:
>> On Sun, Aug 23, 2026 at 11:55 AM Mathieu Fluhr <mathieu.fluhr@gmail.com> wrote:
>>>
>>> I'm reporting a regression that causes a hard platform reset on my workstation,
>>> But, before digging into the technical details, I would like to first re-trace
>>> how I came to this particular commit.

Thx for the report. Side note: mentioning the essence at the top before
explaining backstory and details is usually the way better approach to
get people interested in a bug report, especially if it's more that one
or two paras.

Anyway, as you mentioned later and quoted below, your problems are
caused by a 6.12.y backport of 8f5b7e2bec1c36 ("usb: hub: fix detection
of high tier USB3 devices behind suspended hubs") [v6.16-rc5 (2025-07-06
23:10:26)], authored by Mathias Nyman (now CCed).

There were at least three earlier reports about problems caused by that
commit, all mentioning adb (which I assume you use, too):

https://lore.kernel.org/all/qc0nhk9c6l0a08bkfeplrm3qjssgrjkvkp@sonic.net/
https://lore.kernel.org/all/CAGQo=TJXef-QzbkFEpTcJ138j0k6WYnV4=tNY8Q7s3U7DUOY+Q@mail.gmail.com/
https://bugzilla.kernel.org/show_bug.cgi?id=221103

FWIW, https://bugzilla.kernel.org/show_bug.cgi?id=221103#c16 mentions
the workaround you apparently found, too. The patch Mario (now CCed,
just like the authors of the two mailed reports linked above; and
Michael, too, who seems to be involved) proposed in the last comment of
that ticket apparently didn't help. And no other fix is in sight, or am
I missing something?

Mathias reverting the culprit in mainline a option to resolve this (I
assume it is not, but I'm asking just to be sure)?

Ideally this would have been fixed months ago, but seems to have fallen
through the cracks (partly my fault for not prodding people when this
stalled).

Ciao, Thorsten

>>> As an AOSP developer I am compiling daily different versions of AOSP on it,
>>> mostly building an Android (Automotive) emulator for quick "code-build-test"
>>> runs. I use Ubuntu 24.04 LTS for that, updating to the latest HWE kernels that
>>> Ubuntu provide there (6.8, 6.11, 6.14 etc. until 7.0 recently), with no manual
>>> modifications applied.
>>>
>>> 3 weeks ago, I needed to analyze and issue happening inside Android 11,
>>> building an emulator for a simple Android phone [1]. But when I started using
>>> this emulator, first with the latest 7.0 kernel I noticed several hard crashes,
>>> sometimes just freezing my workstation (with fans full on, but sometimes fans
>>> full off), but also sometimes automatically rebooting it.
>>>
>>> After a few days of deep investigations (To be honest, I first suspected an
>>> issue with the nivida driver), I found out that a pattern to reproduce this
>>> quickly was to let the computer idle with the emulator running. The crash was
>>> always occurring under 20/25 minutes, most of the time letting it idle for less
>>> than 10 was even sufficient.
>>>
>>> This made me a bit curious, and looking a bit deeper (and with a little help
>>> of AI) I was able to find a workaround. forbidding my CPU to enter C2 state,
>>> using "processor.max_cstate=1" argument. Using this, I was not able to
>>> reproduce the crash for more than an hour, but I did not pursue there very
>>> much: As a developer I hate workarounds :)
>>>
>>> I instead started to get "back in time" on a clean and up-to-date fresh Ubuntu
>>> 24.04 install, reverting back to "good old" kernel versions, since I could not
>>> convince myself that first my CPU was dying and second that the issue has always
>>> been there. I confirmed that 6.8 and 6.11 (again Ubuntu HWE packages) work
>>> fine, but the crash was reproducible using 6.14 and above.
>>>
>>> I then got my hands dirty, and started to test different mainline kernel
>>> prebuilt packages up to a point where I confirmed that 6.12.35 worked fine, but
>>> 6.12.40 not. I then bisected both versions, ensuring a good case meant the
>>> emulator was idling without any crash for 1 hour minimum. This lead at the end
>>> to the following commit:
>>>
>>>   aec11e5f9c452ef64e2c113637ab89a67a5ceb62
>>>   ("usb: hub: fix detection of high tier USB3 devices behind suspended hubs")
>>>   [mainline: 8f5b7e2bec1c36578fdaa74a6951833541103e27]
>>>
>>> Being quite astonished that something related to C2 state was triggered by an
>>> USB patch, I then tested the latest 7.0 kernel, this time using the
>>> "usbcore.autosuspend=-1" argument instead. To my surprise, I could not
>>> reproduce the crash, even with the exact same emulator idling for 2 hours.
>>>
>>> Also, something very astonishing, that I still cannot fully understand today:
>>>  1. "priming" my system with a 30 seconds (!) run of a modern Android
>>>      emulator [2] cleared the issue: After closing the 15 emulator and starting
>>>      the 11, I could let it idle for again more than an hour. It seems even not
>>>      be related to the 'kvm' kernel modules, since removing the module and
>>>      re-inserting it between both emulator did not change a thing.
>>>  2. A few times (I did not really invest debugging this TBH), the crash even
>>>     occurred shortly (2-3 minutes) after I closed the Android 11 emulator.
>>>
>>> Now, the technical details of my setup...
>>>
>>> Hardware / software
>>> ===================
>>>
>>> CPU:        AMD Ryzen Threadripper 7970X (Storm Peak, family 19h)
>>> Microcode:  0x0a10810c
>>> Board:      ASUS Pro WS TRX50-SAGE WIFI, BIOS 1203 and 1317 (both affected)
>>> Memory:     128 GB DDR5 ECC RDIMM @ 4800 MT/s (JEDEC, no XMP/EXPO)
>>> GPU:        NVIDIA RTX 3080 Ti (also reproduced with RTX 5080)
>>> Distro:     Ubuntu 24.04 and 26.04 with 595-open NVIDIA driver
>>> USB:        8 onboard xHCI controllers; only a USB keyboard and mouse
>>>             attached
>>>
>>> Error signature
>>> ===============
>>>
>>> On the boot following each crash I could always see the following lines in
>>> the dmesg logs:
>>> ---8<-------------------------------------------------------------------------
>>>   x86/amd: Previous system reset reason [0x88000800]: an uncorrected
>>>            error caused a data fabric sync flood event
>>>   x86/amd: Previous system reset reason [0x88000800]: a software sync
>>>            flood event occurred
>>> ---8<-------------------------------------------------------------------------
>>>
>>> When I was fortunate enough and had an automatic reboot, this was also inside:
>>> ---8<-------------------------------------------------------------------------
>>>   [Hardware Error]: event severity: fatal
>>>   [Hardware Error]:   section_type: IA32/X64 processor error
>>>   [Hardware Error]:    Error Structure Type: cache error
>>>   [Hardware Error]:    Check Information: 0x000000000602001f
>>>   [Hardware Error]:     Transaction Type: 2, Generic
>>>   [Hardware Error]:     Level: 0
>>>   [Hardware Error]:     Processor Context Corrupt: true
>>>   [Hardware Error]:     Uncorrected: true
>>>   mce: [Hardware Error]: CPU 34: Machine Check: 0 Bank 5: aea0000000000108
>>>   mce: [Hardware Error]: TSC 0 ADDR 1ffffff98e8ee38 MISC d0150fff00000000
>>>        SYND 4d000000 IPID 500b020049b00
>>> ---8<-------------------------------------------------------------------------
>>>
>>> The signature is bit-identical across every occurrence except for the
>>> reporting CPU, which varies (observed on CPUs 1, 3, 7, 16, 32, 34, 35).
>>>
>>>
>>> Reproducer
>>> ==========
>>>
>>> 1. Boot an affected kernel with default idle settings (C2 available,
>>>    no max_cstate restriction).
>>> 2. Start an old and/or 32/64-bit Android emulator (QEMU/KVM guest) and leave it
>>>    idle on the launcher screen. Nothing else running.
>>> 3. System hard-resets within 20 minutes.
>>>
>>> Under sustained CPU load the fault never occurs; it requires the system to be
>>> idle. turbostat confirms ~99% C2 residency across all cores in the crashing
>>> condition.
>>>
>>>
>>> Bisection
>>> =========
>>>
>>> ---8<-------------------------------------------------------------------------
>>> git bisect start
>>> # status: waiting for both good and bad commits
>>> # good: [783cd2c3dca8b6c434e955b84c20c8940588dc68] Linux 6.12.35
>>> git bisect good 783cd2c3dca8b6c434e955b84c20c8940588dc68
>>> # status: waiting for bad commit, 1 good commit known
>>> # bad: [d90ecb2b1308b3e362ec4c21ff7cf0a051b445df] Linux 6.12.40
>>> git bisect bad d90ecb2b1308b3e362ec4c21ff7cf0a051b445df
>>> # good: [3ce57d493dd81ecc27033c7a918cc0d0ddaa8edc] ata: libata-acpi:
>>> Do not assume 40 wire cable if no devices are enabled
>>> git bisect good 3ce57d493dd81ecc27033c7a918cc0d0ddaa8edc
>>> # good: [bbd385b65f9e56cab1243e753510a99a59110083] net/mlx5e: Add new
>>> prio for promiscuous mode
>>> git bisect good bbd385b65f9e56cab1243e753510a99a59110083
>>> # good: [2a76bc2b24ed889a689fb1c9015307bf16aafb5b] smb: client: fix
>>> use-after-free in crypt_message when using async crypto
>>> git bisect good 2a76bc2b24ed889a689fb1c9015307bf16aafb5b
>>> # good: [95a13b0a6b042ac5333b67f59af1dffe7b71137b] riscv:
>>> traps_misaligned: properly sign extend value in misaligned load
>>> handler
>>> git bisect good 95a13b0a6b042ac5333b67f59af1dffe7b71137b
>>> # good: [40b5b4ba8ed87c0bfb6268c10589777652ebde4c] drm/mediatek: Add
>>> wait_event_timeout when disabling plane
>>> git bisect good 40b5b4ba8ed87c0bfb6268c10589777652ebde4c
>>> # bad: [affb46db59f908474a211f23953c3b9109f0d647] net: libwx: fix
>>> multicast packets received count
>>> git bisect bad affb46db59f908474a211f23953c3b9109f0d647
>>> # good: [ee56da95f8962b86fec4ef93f866e64c8d025a58] btrfs: fix block
>>> group refcount race in btrfs_create_pending_block_groups()
>>> git bisect good ee56da95f8962b86fec4ef93f866e64c8d025a58
>>> # bad: [bf71baa3cfe754c6e99ad62711b9fb3e3da33517] usb: hub: Fix
>>> flushing of delayed work used for post resume purposes
>>> git bisect bad bf71baa3cfe754c6e99ad62711b9fb3e3da33517
>>> # bad: [e11359640090ab26e4153cce583f50e0b8979f0d] usb: hub: Fix
>>> flushing and scheduling of delayed work that tunes runtime pm
>>> git bisect bad e11359640090ab26e4153cce583f50e0b8979f0d
>>> # bad: [aec11e5f9c452ef64e2c113637ab89a67a5ceb62] usb: hub: fix
>>> detection of high tier USB3 devices behind suspended hubs
>>> git bisect bad aec11e5f9c452ef64e2c113637ab89a67a5ceb62
>>> # first bad commit: [aec11e5f9c452ef64e2c113637ab89a67a5ceb62] usb:
>>> hub: fix detection of high tier USB3 devices behind suspended hubs
>>> ---8<-------------------------------------------------------------------------
>>>
>>> (Please note that I did not perform a full "revert test", since the aec11e5f9c45
>>> commit could not be cleanly reverted on both 6.12.40 and 7.0.0)
>>>
>>>
>>> Workarounds
>>> ===========
>>>
>>> Either of these prevents the crash on an affected kernel:
>>>   usbcore.autosuspend=-1     (disables USB runtime PM)
>>>   processor.max_cstate=1     (prevents C2 entry)
>>>
>>>
>>> Ruled out
>>> =========
>>>
>>> - GPU/driver: reproduced with RTX 5080 (595-open module) and RTX 3080 Ti
>>>               (both 595-open and 595 proprietary modules)
>>> - ASUS USB4 PCIe add-in card: reproduces in ~5 min with the card physically
>>>                               removed
>>> - AVIC: kvm_amd avic=N on both good and bad kernels
>>> - TSA mitigation: tsa=off verified applied
>>>   (/sys/devices/system/cpu/vulnerabilities/tsa = "Vulnerable"), still crashes
>>> - Memory: stock JEDEC 4800, no XMP/EXPO, no ECC errors logged
>>>   (ras-mc-ctl reports zero CE/UE)
>>> - Thermal/power: ~47-50 C at time of crash, system idle; 1600W Seasonic PSU
>>> - Hardware health: multi-hour AOSP compiles (all 64 threads, heavy cache
>>>   load) and repeated GPU stress runs complete without error
>>>
>>>
>>> I hope this gives you enough details to start looking at what could cause this
>>> weird behavior. Just FYI, when asked about hardware damage, AI suggested more
>>> something like "a CPU-level microcode erratum in the deep-idle path on this
>>> platform", but being old-school, I tend to always triple-check what AI tells
>>> before claiming it myself :)
>>>
>>> Thanks and Kind Regards,
>>> Mathieu
>>>
>>> ------
>>> [1] using android-11.0.0_r46 release tag and "sdk_phone_x86_64" as lunch target
>>> [3] using android-15.0.0_r32 release tag and "sdk_phone64_x86_64" as lunch
>>>     target this time, since Google only releases 64-bit only today
> 


^ permalink raw reply	[flat|nested] 11+ messages in thread

* Re: [REGRESSION] 6.12.36+: usb: hub: post-resume delayed work triggers > uncorrected MCE / data fabric sync flood on Threadripper 7970X > (bisected to aec11e5f9c45)
  2026-08-23 10:36     ` Thorsten Leemhuis
@ 2026-08-23 11:44       ` Mathieu Fluhr
  2026-08-23 16:10         ` Michal Pecio
  2026-08-23 15:17       ` Lovekesh Solanki
  1 sibling, 1 reply; 11+ messages in thread
From: Mathieu Fluhr @ 2026-08-23 11:44 UTC (permalink / raw)
  To: Thorsten Leemhuis
  Cc: Mathias Nyman, linux-usb, regressions, stable, linux-kernel,
	Mario Limonciello, Forest, Slavik Dev, Michal Pecio

On Sun, Aug 23, 2026 at 12:37 PM Thorsten Leemhuis
<regressions@leemhuis.info> wrote:
>
> On 8/23/26 12:15, Mathieu Fluhr wrote:
> Thx for the report. Side note: mentioning the essence at the top before
> explaining backstory and details is usually the way better approach to
> get people interested in a bug report, especially if it's more that one
> or two paras.

Point taken :)

> Anyway, as you mentioned later and quoted below, your problems are
> caused by a 6.12.y backport of 8f5b7e2bec1c36 ("usb: hub: fix detection
> of high tier USB3 devices behind suspended hubs") [v6.16-rc5 (2025-07-06
> 23:10:26)], authored by Mathias Nyman (now CCed).
>
> There were at least three earlier reports about problems caused by that
> commit, all mentioning adb (which I assume you use, too):

Yes, looking at the reports, this looks definitely related.
What is happening when the emulator starts is that it also starts adbd
if not running on the Linux side, to communicate with the emulator
later on.

So, this could perfectly explain the persistent state after the
"Android 15 priming": Since the Android 15 repo contains a more recent
ADB, where the issue has been eventually wored around, it is started
firs, and later, when 11 starts, it does not kill it, restarting its
older adbd, but instead keep the new one running.

So, long story short, on my side, I was still looking at the "wrong
side" of the problem, trying to find what QEMU was doing wrong, where
in the end, it was more the interactions that adb was doing with the
USB bus during C2 state that was causing the issue.

>
> https://lore.kernel.org/all/qc0nhk9c6l0a08bkfeplrm3qjssgrjkvkp@sonic.net/
> https://lore.kernel.org/all/CAGQo=TJXef-QzbkFEpTcJ138j0k6WYnV4=tNY8Q7s3U7DUOY+Q@mail.gmail.com/
> https://bugzilla.kernel.org/show_bug.cgi?id=221103
>
> FWIW, https://bugzilla.kernel.org/show_bug.cgi?id=221103#c16 mentions
> the workaround you apparently found, too. The patch Mario (now CCed,
> just like the authors of the two mailed reports linked above; and
> Michael, too, who seems to be involved) proposed in the last comment of
> that ticket apparently didn't help. And no other fix is in sight, or am
> I missing something?
>
> Mathias reverting the culprit in mainline a option to resolve this (I
> assume it is not, but I'm asking just to be sure)?

Ok, here you lost me a little: Looking to all the links provided, yes,
this looks very similar to what I am experiencing. But (and correct me
if I am wrong)
1. reverting the "bad commit" is not really an option, since it was
originally authored for a specific purpose (other than annoying AMD
owners :)
2. the patch that Mario proposed is very recent, and would address the
issue not on the USB but on the PCI level?

Thanks again for the swift answer here and all the informative links!
I must admit that I researched a lot debugging this issue, but I never
thought about looking to the USB side of the problem, focusing more on
the CPU side.

Kind Regards,
Mathieu

>
> Ciao, Thorsten
>
> >>> As an AOSP developer I am compiling daily different versions of AOSP on it,
> >>> mostly building an Android (Automotive) emulator for quick "code-build-test"
> >>> runs. I use Ubuntu 24.04 LTS for that, updating to the latest HWE kernels that
> >>> Ubuntu provide there (6.8, 6.11, 6.14 etc. until 7.0 recently), with no manual
> >>> modifications applied.
> >>>
> >>> 3 weeks ago, I needed to analyze and issue happening inside Android 11,
> >>> building an emulator for a simple Android phone [1]. But when I started using
> >>> this emulator, first with the latest 7.0 kernel I noticed several hard crashes,
> >>> sometimes just freezing my workstation (with fans full on, but sometimes fans
> >>> full off), but also sometimes automatically rebooting it.
> >>>
> >>> After a few days of deep investigations (To be honest, I first suspected an
> >>> issue with the nivida driver), I found out that a pattern to reproduce this
> >>> quickly was to let the computer idle with the emulator running. The crash was
> >>> always occurring under 20/25 minutes, most of the time letting it idle for less
> >>> than 10 was even sufficient.
> >>>
> >>> This made me a bit curious, and looking a bit deeper (and with a little help
> >>> of AI) I was able to find a workaround. forbidding my CPU to enter C2 state,
> >>> using "processor.max_cstate=1" argument. Using this, I was not able to
> >>> reproduce the crash for more than an hour, but I did not pursue there very
> >>> much: As a developer I hate workarounds :)
> >>>
> >>> I instead started to get "back in time" on a clean and up-to-date fresh Ubuntu
> >>> 24.04 install, reverting back to "good old" kernel versions, since I could not
> >>> convince myself that first my CPU was dying and second that the issue has always
> >>> been there. I confirmed that 6.8 and 6.11 (again Ubuntu HWE packages) work
> >>> fine, but the crash was reproducible using 6.14 and above.
> >>>
> >>> I then got my hands dirty, and started to test different mainline kernel
> >>> prebuilt packages up to a point where I confirmed that 6.12.35 worked fine, but
> >>> 6.12.40 not. I then bisected both versions, ensuring a good case meant the
> >>> emulator was idling without any crash for 1 hour minimum. This lead at the end
> >>> to the following commit:
> >>>
> >>>   aec11e5f9c452ef64e2c113637ab89a67a5ceb62
> >>>   ("usb: hub: fix detection of high tier USB3 devices behind suspended hubs")
> >>>   [mainline: 8f5b7e2bec1c36578fdaa74a6951833541103e27]
> >>>
> >>> Being quite astonished that something related to C2 state was triggered by an
> >>> USB patch, I then tested the latest 7.0 kernel, this time using the
> >>> "usbcore.autosuspend=-1" argument instead. To my surprise, I could not
> >>> reproduce the crash, even with the exact same emulator idling for 2 hours.
> >>>
> >>> Also, something very astonishing, that I still cannot fully understand today:
> >>>  1. "priming" my system with a 30 seconds (!) run of a modern Android
> >>>      emulator [2] cleared the issue: After closing the 15 emulator and starting
> >>>      the 11, I could let it idle for again more than an hour. It seems even not
> >>>      be related to the 'kvm' kernel modules, since removing the module and
> >>>      re-inserting it between both emulator did not change a thing.
> >>>  2. A few times (I did not really invest debugging this TBH), the crash even
> >>>     occurred shortly (2-3 minutes) after I closed the Android 11 emulator.
> >>>
> >>> Now, the technical details of my setup...
> >>>
> >>> Hardware / software
> >>> ===================
> >>>
> >>> CPU:        AMD Ryzen Threadripper 7970X (Storm Peak, family 19h)
> >>> Microcode:  0x0a10810c
> >>> Board:      ASUS Pro WS TRX50-SAGE WIFI, BIOS 1203 and 1317 (both affected)
> >>> Memory:     128 GB DDR5 ECC RDIMM @ 4800 MT/s (JEDEC, no XMP/EXPO)
> >>> GPU:        NVIDIA RTX 3080 Ti (also reproduced with RTX 5080)
> >>> Distro:     Ubuntu 24.04 and 26.04 with 595-open NVIDIA driver
> >>> USB:        8 onboard xHCI controllers; only a USB keyboard and mouse
> >>>             attached
> >>>
> >>> Error signature
> >>> ===============
> >>>
> >>> On the boot following each crash I could always see the following lines in
> >>> the dmesg logs:
> >>> ---8<-------------------------------------------------------------------------
> >>>   x86/amd: Previous system reset reason [0x88000800]: an uncorrected
> >>>            error caused a data fabric sync flood event
> >>>   x86/amd: Previous system reset reason [0x88000800]: a software sync
> >>>            flood event occurred
> >>> ---8<-------------------------------------------------------------------------
> >>>
> >>> When I was fortunate enough and had an automatic reboot, this was also inside:
> >>> ---8<-------------------------------------------------------------------------
> >>>   [Hardware Error]: event severity: fatal
> >>>   [Hardware Error]:   section_type: IA32/X64 processor error
> >>>   [Hardware Error]:    Error Structure Type: cache error
> >>>   [Hardware Error]:    Check Information: 0x000000000602001f
> >>>   [Hardware Error]:     Transaction Type: 2, Generic
> >>>   [Hardware Error]:     Level: 0
> >>>   [Hardware Error]:     Processor Context Corrupt: true
> >>>   [Hardware Error]:     Uncorrected: true
> >>>   mce: [Hardware Error]: CPU 34: Machine Check: 0 Bank 5: aea0000000000108
> >>>   mce: [Hardware Error]: TSC 0 ADDR 1ffffff98e8ee38 MISC d0150fff00000000
> >>>        SYND 4d000000 IPID 500b020049b00
> >>> ---8<-------------------------------------------------------------------------
> >>>
> >>> The signature is bit-identical across every occurrence except for the
> >>> reporting CPU, which varies (observed on CPUs 1, 3, 7, 16, 32, 34, 35).
> >>>
> >>>
> >>> Reproducer
> >>> ==========
> >>>
> >>> 1. Boot an affected kernel with default idle settings (C2 available,
> >>>    no max_cstate restriction).
> >>> 2. Start an old and/or 32/64-bit Android emulator (QEMU/KVM guest) and leave it
> >>>    idle on the launcher screen. Nothing else running.
> >>> 3. System hard-resets within 20 minutes.
> >>>
> >>> Under sustained CPU load the fault never occurs; it requires the system to be
> >>> idle. turbostat confirms ~99% C2 residency across all cores in the crashing
> >>> condition.
> >>>
> >>>
> >>> Bisection
> >>> =========
> >>>
> >>> ---8<-------------------------------------------------------------------------
> >>> git bisect start
> >>> # status: waiting for both good and bad commits
> >>> # good: [783cd2c3dca8b6c434e955b84c20c8940588dc68] Linux 6.12.35
> >>> git bisect good 783cd2c3dca8b6c434e955b84c20c8940588dc68
> >>> # status: waiting for bad commit, 1 good commit known
> >>> # bad: [d90ecb2b1308b3e362ec4c21ff7cf0a051b445df] Linux 6.12.40
> >>> git bisect bad d90ecb2b1308b3e362ec4c21ff7cf0a051b445df
> >>> # good: [3ce57d493dd81ecc27033c7a918cc0d0ddaa8edc] ata: libata-acpi:
> >>> Do not assume 40 wire cable if no devices are enabled
> >>> git bisect good 3ce57d493dd81ecc27033c7a918cc0d0ddaa8edc
> >>> # good: [bbd385b65f9e56cab1243e753510a99a59110083] net/mlx5e: Add new
> >>> prio for promiscuous mode
> >>> git bisect good bbd385b65f9e56cab1243e753510a99a59110083
> >>> # good: [2a76bc2b24ed889a689fb1c9015307bf16aafb5b] smb: client: fix
> >>> use-after-free in crypt_message when using async crypto
> >>> git bisect good 2a76bc2b24ed889a689fb1c9015307bf16aafb5b
> >>> # good: [95a13b0a6b042ac5333b67f59af1dffe7b71137b] riscv:
> >>> traps_misaligned: properly sign extend value in misaligned load
> >>> handler
> >>> git bisect good 95a13b0a6b042ac5333b67f59af1dffe7b71137b
> >>> # good: [40b5b4ba8ed87c0bfb6268c10589777652ebde4c] drm/mediatek: Add
> >>> wait_event_timeout when disabling plane
> >>> git bisect good 40b5b4ba8ed87c0bfb6268c10589777652ebde4c
> >>> # bad: [affb46db59f908474a211f23953c3b9109f0d647] net: libwx: fix
> >>> multicast packets received count
> >>> git bisect bad affb46db59f908474a211f23953c3b9109f0d647
> >>> # good: [ee56da95f8962b86fec4ef93f866e64c8d025a58] btrfs: fix block
> >>> group refcount race in btrfs_create_pending_block_groups()
> >>> git bisect good ee56da95f8962b86fec4ef93f866e64c8d025a58
> >>> # bad: [bf71baa3cfe754c6e99ad62711b9fb3e3da33517] usb: hub: Fix
> >>> flushing of delayed work used for post resume purposes
> >>> git bisect bad bf71baa3cfe754c6e99ad62711b9fb3e3da33517
> >>> # bad: [e11359640090ab26e4153cce583f50e0b8979f0d] usb: hub: Fix
> >>> flushing and scheduling of delayed work that tunes runtime pm
> >>> git bisect bad e11359640090ab26e4153cce583f50e0b8979f0d
> >>> # bad: [aec11e5f9c452ef64e2c113637ab89a67a5ceb62] usb: hub: fix
> >>> detection of high tier USB3 devices behind suspended hubs
> >>> git bisect bad aec11e5f9c452ef64e2c113637ab89a67a5ceb62
> >>> # first bad commit: [aec11e5f9c452ef64e2c113637ab89a67a5ceb62] usb:
> >>> hub: fix detection of high tier USB3 devices behind suspended hubs
> >>> ---8<-------------------------------------------------------------------------
> >>>
> >>> (Please note that I did not perform a full "revert test", since the aec11e5f9c45
> >>> commit could not be cleanly reverted on both 6.12.40 and 7.0.0)
> >>>
> >>>
> >>> Workarounds
> >>> ===========
> >>>
> >>> Either of these prevents the crash on an affected kernel:
> >>>   usbcore.autosuspend=-1     (disables USB runtime PM)
> >>>   processor.max_cstate=1     (prevents C2 entry)
> >>>
> >>>
> >>> Ruled out
> >>> =========
> >>>
> >>> - GPU/driver: reproduced with RTX 5080 (595-open module) and RTX 3080 Ti
> >>>               (both 595-open and 595 proprietary modules)
> >>> - ASUS USB4 PCIe add-in card: reproduces in ~5 min with the card physically
> >>>                               removed
> >>> - AVIC: kvm_amd avic=N on both good and bad kernels
> >>> - TSA mitigation: tsa=off verified applied
> >>>   (/sys/devices/system/cpu/vulnerabilities/tsa = "Vulnerable"), still crashes
> >>> - Memory: stock JEDEC 4800, no XMP/EXPO, no ECC errors logged
> >>>   (ras-mc-ctl reports zero CE/UE)
> >>> - Thermal/power: ~47-50 C at time of crash, system idle; 1600W Seasonic PSU
> >>> - Hardware health: multi-hour AOSP compiles (all 64 threads, heavy cache
> >>>   load) and repeated GPU stress runs complete without error
> >>>
> >>>
> >>> I hope this gives you enough details to start looking at what could cause this
> >>> weird behavior. Just FYI, when asked about hardware damage, AI suggested more
> >>> something like "a CPU-level microcode erratum in the deep-idle path on this
> >>> platform", but being old-school, I tend to always triple-check what AI tells
> >>> before claiming it myself :)
> >>>
> >>> Thanks and Kind Regards,
> >>> Mathieu
> >>>
> >>> ------
> >>> [1] using android-11.0.0_r46 release tag and "sdk_phone_x86_64" as lunch target
> >>> [3] using android-15.0.0_r32 release tag and "sdk_phone64_x86_64" as lunch
> >>>     target this time, since Google only releases 64-bit only today
> >
>

^ permalink raw reply	[flat|nested] 11+ messages in thread

* Re: [REGRESSION] 6.12.36+: usb: hub: post-resume delayed work triggers > uncorrected MCE / data fabric sync flood on Threadripper 7970X > (bisected to aec11e5f9c45)
  2026-08-23 10:36     ` Thorsten Leemhuis
  2026-08-23 11:44       ` Mathieu Fluhr
@ 2026-08-23 15:17       ` Lovekesh Solanki
  2026-08-23 15:40         ` Michal Pecio
  1 sibling, 1 reply; 11+ messages in thread
From: Lovekesh Solanki @ 2026-08-23 15:17 UTC (permalink / raw)
  To: Thorsten Leemhuis
  Cc: Mathias Nyman, linux-usb, regressions, stable, linux-kernel,
	Mario Limonciello, Forest, Slavik Dev, Michal Pecio,
	Mathieu Fluhr

On Sun, Aug 23, 2026 at 12:36:55PM +0200, Thorsten Leemhuis wrote:
> that ticket apparently didn't help. And no other fix is in sight, or am
> I missing something?
> 
> Mathias reverting the culprit in mainline a option to resolve this (I
> assume it is not, but I'm asking just to be sure)?
I don't think a revert is needed here, we're just trading one regression
for another.
Commit 8f5b7e2bec1c introduced the 200ms hold for all superspeed hubs,
but it's only useful for external ones. Root hubs are superspeed hubs
too so they go through the same code, but there's nothing useful for
the hold to do there:
A root hub has no upstream suspended hub whose wake propagation we need
to wait for and xhci already handles late USB3 link training itself.

Also hubs have a 0 second autosuspend delay (596d789a211d), so this hold
is the only thing stretching the awake window.

and anything opening or closing /dev/bus/usb nodes auto resumes and auto
suspends the whole host controller, adb's periodic enumeration does that
so on the reported affected systems every SS roothub cycle grows from
~30 to ~235ms (from the dynamic debug traces in
https://lore.kernel.org/all/qc0nhk9c6l0a08bkfeplrm3qjssgrjkvkp@sonic.net/)
which makes suspend move from close() call into delayed work and results
in the ~1Hz stress loop described upthread.

I think skipping the hold for hubs without a parent device, i.e. keep TB
dock behaviour everywhere it matters, should fix this.

Patch below could any of the reporters test it?

Thanks,
DrEggCake

diff --git a/drivers/usb/core/hub.c b/drivers/usb/core/hub.c
index d92bf887739d..f642ea0c31dc 100644
--- a/drivers/usb/core/hub.c
+++ b/drivers/usb/core/hub.c
@@ -1346,7 +1346,8 @@ static void hub_activate(struct usb_hub *hub, enum hub_activation_type type)
 		device_unlock(&hdev->dev);
 	}
 
-	if (type == HUB_RESUME && hub_is_superspeed(hub->hdev)) {
+	if (type == HUB_RESUME && hub->hdev->parent &&
+	    hub_is_superspeed(hub->hdev)) {
 		/* give usb3 downstream links training time after hub resume */
 		usb_autopm_get_interface_no_resume(
 			to_usb_interface(hub->intfdev));

^ permalink raw reply	[flat|nested] 11+ messages in thread

* Re: [REGRESSION] 6.12.36+: usb: hub: post-resume delayed work triggers > uncorrected MCE / data fabric sync flood on Threadripper 7970X > (bisected to aec11e5f9c45)
  2026-08-23 15:17       ` Lovekesh Solanki
@ 2026-08-23 15:40         ` Michal Pecio
  2026-08-23 17:05           ` Lovekesh Solanki
  0 siblings, 1 reply; 11+ messages in thread
From: Michal Pecio @ 2026-08-23 15:40 UTC (permalink / raw)
  To: Lovekesh Solanki
  Cc: Thorsten Leemhuis, Mathias Nyman, linux-usb, regressions, stable,
	linux-kernel, Mario Limonciello, Forest, Slavik Dev,
	Mathieu Fluhr

On Sun, 23 Aug 2026 20:47:55 +0530, Lovekesh Solanki wrote:
> I don't think a revert is needed here, we're just trading one
> regression for another.
> Commit 8f5b7e2bec1c introduced the 200ms hold for all superspeed hubs,
> but it's only useful for external ones. Root hubs are superspeed hubs
> too so they go through the same code, but there's nothing useful for
> the hold to do there:
> A root hub has no upstream suspended hub whose wake propagation we
> need to wait for and xhci already handles late USB3 link training
> itself.
> 
> Also hubs have a 0 second autosuspend delay (596d789a211d), so this
> hold is the only thing stretching the awake window.
> 
> and anything opening or closing /dev/bus/usb nodes auto resumes and
> auto suspends the whole host controller, adb's periodic enumeration
> does that so on the reported affected systems every SS roothub cycle
> grows from ~30 to ~235ms (from the dynamic debug traces in
> https://lore.kernel.org/all/qc0nhk9c6l0a08bkfeplrm3qjssgrjkvkp@sonic.net/)
> which makes suspend move from close() call into delayed work and
> results in the ~1Hz stress loop described upthread.
> 
> I think skipping the hold for hubs without a parent device, i.e. keep
> TB dock behaviour everywhere it matters, should fix this.

It likely will, as it effectively reverts the problematic commit for
those particular affected devices (root hubs).

But I'm not sure what you mean by "1Hz stress loop" and why is slowing
down the suspend/resume cycles or moving suspend from close() call into
a work supposed to create problems?

I would naively think that doing things *too fast* is more likely to
trigger races and break the HW. The whole issue smells like a HW bug,
or maybe one of those "undefined behaviors" that the xHCI spec warns
about if SW dares to do something out of spec.

So why was the culprit patch even a problem for those root hubs?
Can it not become a problem for external hubs, under other workloads?

Also, what if we connect a downstream SS hub to the root hub? Will this
not cause the root hub to stay awake for 200ms again? Problem is back?

Regards,
Michal

^ permalink raw reply	[flat|nested] 11+ messages in thread

* Re: [REGRESSION] 6.12.36+: usb: hub: post-resume delayed work triggers > uncorrected MCE / data fabric sync flood on Threadripper 7970X > (bisected to aec11e5f9c45)
  2026-08-23 11:44       ` Mathieu Fluhr
@ 2026-08-23 16:10         ` Michal Pecio
  0 siblings, 0 replies; 11+ messages in thread
From: Michal Pecio @ 2026-08-23 16:10 UTC (permalink / raw)
  To: Mathieu Fluhr
  Cc: Thorsten Leemhuis, Mathias Nyman, linux-usb, regressions, stable,
	linux-kernel, Mario Limonciello, Forest, Slavik Dev

On Sun, 23 Aug 2026 13:44:53 +0200, Mathieu Fluhr wrote:
> Yes, looking at the reports, this looks definitely related.
> What is happening when the emulator starts is that it also starts adbd
> if not running on the Linux side, to communicate with the emulator
> later on.
> 
> [...]
>
> Ok, here you lost me a little: Looking to all the links provided, yes,
> this looks very similar to what I am experiencing. But (and correct me
> if I am wrong)
> 1. reverting the "bad commit" is not really an option, since it was
> originally authored for a specific purpose (other than annoying AMD
> owners :)

It does seem so, even if the specific issue report isn't known.

It also isn't clear (at least to me) whether the intended behavior of
the patch is causing this, or if the patch simply has some bug which
causes unintended side effects that can and should be fixed.

> 2. the patch that Mario proposed is very recent, and would address the
> issue not on the USB but on the PCI level?
> 
> Thanks again for the swift answer here and all the informative links!
> I must admit that I researched a lot debugging this issue, but I never
> thought about looking to the USB side of the problem, focusing more on
> the CPU side.

The issue is obviously a severe HW malfunction (you mentioned MCEs, the
Ryzen CPUs simply totally locked up), triggered by poking certain xHCI
controllers on the I/O die of these CPUs in some wrong way.

The minimum known trigger is this loop from bugzilla comment 8:

	while sleep .1 ; do true </dev/bus/usb/010/001 ; done

Opinions seem to vary on whether CPU load must be present or absent.

The minimum known workaround is disabling xHCI PCI function suspend,
see bugzilla comment 16. This knowledge has spread on distribution
forums and many users stopped caring about this bug, but clearly new
people are still running into it.

It seems USB devices and even the root hub can be suspended safely, as
long as the PCI function remains on full power. The influence of CPU
C-states was previously unknown. May be specific to Threadripper.

Regards,
Michal

^ permalink raw reply	[flat|nested] 11+ messages in thread

* Re: [REGRESSION] 6.12.36+: usb: hub: post-resume delayed work triggers > uncorrected MCE / data fabric sync flood on Threadripper 7970X > (bisected to aec11e5f9c45)
  2026-08-23 15:40         ` Michal Pecio
@ 2026-08-23 17:05           ` Lovekesh Solanki
  2026-08-24 19:37             ` Mathieu Fluhr
  0 siblings, 1 reply; 11+ messages in thread
From: Lovekesh Solanki @ 2026-08-23 17:05 UTC (permalink / raw)
  To: Michal Pecio
  Cc: Thorsten Leemhuis, Mathias Nyman, linux-usb, regressions, stable,
	linux-kernel, Mario Limonciello, Forest, Slavik Dev,
	Mathieu Fluhr

On Sun, Aug 23, 2026 at 05:40:59PM +0200, Michal Pecio wrote:
> But I'm not sure what you mean by "1Hz stress loop" and why is slowing
> down the suspend/resume cycles or moving suspend from close() call into
> a work supposed to create problems?
> 
> I would naively think that doing things *too fast* is more likely to
> trigger races and break the HW. The whole issue smells like a HW bug,
USB2 roothubs still do 30 ms cycles and are harmless, only stretched SS
ones kill the box. So if speed were the issue the faster ones would be
dying first and pre regression kernels ran fine.

About 1Hz, adb scans the bus once a second,
every open resumes the root hub and host controller out of D3 and close
puts it back this existed before regression too but with 30 ms of
close() and slept for rest of the second and now it keeps it
in D0 well after close(), the suspend itself happening later from the
delayed work, so this d3 - d0 - d3 trip of ~230ms repeats on
each scan. That's the loop I meant.

> Can it not become a problem for external hubs, under other workloads?
Possibly yes, but nobody reported that so I'd rather scope where the
harm is proven.

> Also, what if we connect a downstream SS hub to the root hub? Will this
> not cause the root hub to stay awake for 200ms again? Problem is back?
The root hub itself skips the hold since the check is on its own parent,
but it will be kept awake anyway while the downstream hub or devices are
in use which is normal activity based PM.
If heavy polling behind a real hub ever causes trouble that's probably a
xhci/platform level fix anyways since real hubs can't really just drop
the hold.

Regards,
Lovekesh

^ permalink raw reply	[flat|nested] 11+ messages in thread

* Re: [REGRESSION] 6.12.36+: usb: hub: post-resume delayed work triggers > uncorrected MCE / data fabric sync flood on Threadripper 7970X > (bisected to aec11e5f9c45)
  2026-08-23 17:05           ` Lovekesh Solanki
@ 2026-08-24 19:37             ` Mathieu Fluhr
  2026-08-24 22:09               ` Mario Limonciello
  0 siblings, 1 reply; 11+ messages in thread
From: Mathieu Fluhr @ 2026-08-24 19:37 UTC (permalink / raw)
  To: Lovekesh Solanki
  Cc: Michal Pecio, Thorsten Leemhuis, Mathias Nyman, linux-usb,
	regressions, stable, linux-kernel, Mario Limonciello, Forest,
	Slavik Dev

> The issue is obviously a severe HW malfunction (you mentioned MCEs, the
> Ryzen CPUs simply totally locked up), triggered by poking certain xHCI
> controllers on the I/O die of these CPUs in some wrong way.

Yes. As mentioned, I first thought that the emulator itself triggered that by
doing something that the CPU did not like.To be honest, I barely play with old
Android versions anymore, but seeing that I could reproduce it even with
Android 13 or 14 made me suspicious.

I _guess_ Google implemented a workaround inside adb for version Android
15 since using this version, it remains stable for more than 2 hours.

But, in the end, the situation is that from a simple user account having access
to some usb plugged in devices (I usually add my user account to the plugdev
group and use some known udev rules to access my Android tests devices),
you have a way to crash the complete system.

> Opinions seem to vary on whether CPU load must be present or absent.

On my side (and I am here only speaking about my TR. I don't know about
other Ryzen CPUs), I can't reproduce it under load, and one condition to
reproduce it is my CPU going in C2 state.
-> I did a 2:30 hour test using several youtube videos playing at the same
time on my desktop, also stressing the emulator with some 3D Mark runs
(as mentioned, I first suspected the nvidia driver to be faulty). As long as my
computer was busy everything went fine. But then I let it stand still for a few
minutes, and it just crashed.

If you need me to do some further tests or experiments, let me know. I will
be more than happy to play the guinea pig here.

On Sun, Aug 23, 2026 at 7:05 PM Lovekesh Solanki
<lovekeshsolanki00@gmail.com> wrote:
>
> On Sun, Aug 23, 2026 at 05:40:59PM +0200, Michal Pecio wrote:
> > But I'm not sure what you mean by "1Hz stress loop" and why is slowing
> > down the suspend/resume cycles or moving suspend from close() call into
> > a work supposed to create problems?
> >
> > I would naively think that doing things *too fast* is more likely to
> > trigger races and break the HW. The whole issue smells like a HW bug,
> USB2 roothubs still do 30 ms cycles and are harmless, only stretched SS
> ones kill the box. So if speed were the issue the faster ones would be
> dying first and pre regression kernels ran fine.
>
> About 1Hz, adb scans the bus once a second,
> every open resumes the root hub and host controller out of D3 and close
> puts it back this existed before regression too but with 30 ms of
> close() and slept for rest of the second and now it keeps it
> in D0 well after close(), the suspend itself happening later from the
> delayed work, so this d3 - d0 - d3 trip of ~230ms repeats on
> each scan. That's the loop I meant.
>
> > Can it not become a problem for external hubs, under other workloads?
> Possibly yes, but nobody reported that so I'd rather scope where the
> harm is proven.
>
> > Also, what if we connect a downstream SS hub to the root hub? Will this
> > not cause the root hub to stay awake for 200ms again? Problem is back?
> The root hub itself skips the hold since the check is on its own parent,
> but it will be kept awake anyway while the downstream hub or devices are
> in use which is normal activity based PM.
> If heavy polling behind a real hub ever causes trouble that's probably a
> xhci/platform level fix anyways since real hubs can't really just drop
> the hold.
>
> Regards,
> Lovekesh

^ permalink raw reply	[flat|nested] 11+ messages in thread

* Re: [REGRESSION] 6.12.36+: usb: hub: post-resume delayed work triggers > uncorrected MCE / data fabric sync flood on Threadripper 7970X > (bisected to aec11e5f9c45)
  2026-08-24 19:37             ` Mathieu Fluhr
@ 2026-08-24 22:09               ` Mario Limonciello
  0 siblings, 0 replies; 11+ messages in thread
From: Mario Limonciello @ 2026-08-24 22:09 UTC (permalink / raw)
  To: Mathieu Fluhr, Lovekesh Solanki
  Cc: Michal Pecio, Thorsten Leemhuis, Mathias Nyman, linux-usb,
	regressions, stable, linux-kernel, Forest, Slavik Dev



On 8/24/26 14:37, Mathieu Fluhr wrote:
>> The issue is obviously a severe HW malfunction (you mentioned MCEs, the
>> Ryzen CPUs simply totally locked up), triggered by poking certain xHCI
>> controllers on the I/O die of these CPUs in some wrong way.
> 
> Yes. As mentioned, I first thought that the emulator itself triggered that by
> doing something that the CPU did not like.To be honest, I barely play with old
> Android versions anymore, but seeing that I could reproduce it even with
> Android 13 or 14 made me suspicious.
> 
> I _guess_ Google implemented a workaround inside adb for version Android
> 15 since using this version, it remains stable for more than 2 hours.
> 
> But, in the end, the situation is that from a simple user account having access
> to some usb plugged in devices (I usually add my user account to the plugdev
> group and use some known udev rules to access my Android tests devices),
> you have a way to crash the complete system.
> 
>> Opinions seem to vary on whether CPU load must be present or absent.
> 
> On my side (and I am here only speaking about my TR. I don't know about
> other Ryzen CPUs), I can't reproduce it under load, and one condition to
> reproduce it is my CPU going in C2 state.
> -> I did a 2:30 hour test using several youtube videos playing at the same
> time on my desktop, also stressing the emulator with some 3D Mark runs
> (as mentioned, I first suspected the nvidia driver to be faulty). As long as my
> computer was busy everything went fine. But then I let it stand still for a few
> minutes, and it just crashed.
> 
> If you need me to do some further tests or experiments, let me know. I will
> be more than happy to play the guinea pig here.
> 
The behavior that is described here sounds like a platform firmware bug 
to me.  Are you on the latest BIOS available from your OEM?

Can you please confirm:

1. Your CPU model number/codename
2. Your OEM (from /sys/class/dmi/id)
3. OEM BIOS version (from /sys/class/dmi/id)
4. AGESA version (see commit bc91133e260c8113c1119073c03b93c12aa41738 if 
the OEM didn't tear it out.  Otherwise look in BIOS menus)?


^ permalink raw reply	[flat|nested] 11+ messages in thread

end of thread, other threads:[~2026-08-24 22:09 UTC | newest]

Thread overview: 11+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2026-08-23  9:55 Mathieu Fluhr
2026-08-23 10:12 ` Mathieu Fluhr
2026-08-23 10:15   ` [REGRESSION] 6.12.36+: usb: hub: post-resume delayed work triggers > uncorrected MCE / data fabric sync flood on Threadripper 7970X > (bisected to aec11e5f9c45) Mathieu Fluhr
2026-08-23 10:36     ` Thorsten Leemhuis
2026-08-23 11:44       ` Mathieu Fluhr
2026-08-23 16:10         ` Michal Pecio
2026-08-23 15:17       ` Lovekesh Solanki
2026-08-23 15:40         ` Michal Pecio
2026-08-23 17:05           ` Lovekesh Solanki
2026-08-24 19:37             ` Mathieu Fluhr
2026-08-24 22:09               ` Mario Limonciello

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox

all inboxes | Powered by JetHome®