[BUG] misc/mei: Race between mei_release() disconnect and mei_ioctl_connect_vtag() causes CSME reset storm and i915 freeze

[BUG] misc/mei: Race between mei_release() disconnect and mei_ioctl_connect_vtag() causes CSME reset storm and i915 freeze

[nirbhayykumarr]

Hi all,

This issue was discovered using a custom multi-threaded C fuzzer
designed to stress-test MEI Virtual Tag (vtag) client lifecycles and
multiplexing over /dev/mei0. By concurrently racing rapid vtag
connections against file descriptor closures and streaming I/O, a
race condition is triggered during client teardown.

System Information & Environment:
- Kernel Version: 7.1.8-zen1-3-zen x86_64 (drivers/misc/mei is identical to upstream)
- Kernel Taint: 0 (Not tainted)
- CPU / Platform: 12th Gen Intel(R) Core(TM) i5-12500H (Alder Lake-P)
- MEI Controller: 00:16.0 Intel Alder Lake PCH HECI Controller (/dev/mei0)
- Graphics / DRM: Intel Iris Xe Graphics (i915 driver with mei_hdcp and mei_pxp components)
- Subsystems:     drivers/misc/mei, drivers/gpu/drm/i915

In mei_release(), closing the last file descriptor holding a virtual tag
invokes mei_cl_disconnect(). Inside __mei_cl_disconnect(), dev->device_lock
is dropped while awaiting the firmware disconnect ACK on cl->wait.

During this lock-drop window, a concurrent IOCTL_MEI_CONNECT_CLIENT_VTAG
call on the same UUID (e.g. MKHI) scans dev->file_list, matches the tearing-
down client 'pos' (in MEI_FILE_DISCONNECTING), repoints file->private_data
to pos, and adds its new vtag to pos->vtag_map.

When the disconnect ACK arrives, __mei_cl_disconnect() calls
mei_cl_set_disconnected(cl), setting cl->me_cl = NULL and
cl->state = MEI_FILE_DISCONNECTED. Because pos->vtag_map now contains the
second thread's tag, mei_release() skips unlinking/freeing cl. The second
thread then wakes up and attempts to reconnect via mei_ioctl_connect_client().

Additionally, shared clients lack tag based demuxing on cl->rd_pending in
drivers/misc/mei/interrupt.c:
  cb = list_first_entry_or_null(&cl->rd_pending, struct mei_cl_cb, list);

Incoming packets are matched to the head of the FIFO queue regardless of
vtag, causing -EPROTO errors, dropped packets, and out-of-order HBM command
sequences. This triggers continuous CSME hardware link resets:
  mei mei0: FW not ready: resetting: dev_state = 3
  mei mei0: unexpected reset: dev_state = ENABLED fw status = ...

During each reset, child client drivers (mei_hdcp, mei_pxp) unbind and rebind
with i915 DRM. With resets looping at hundreds of cycles per second (>9,800
events in 27s), mei_cldev_enable() repeatedly fails with -EFAULT / -ENODEV,
deadlocking i915 display worker mutexes in TASK_UNINTERRUPTIBLE and causing
an unrecoverable full system freeze.

Journalctl logs:
  mei mei0: FW not ready: resetting: dev_state = 3
  mei mei0: unexpected reset: dev_state = ENABLED fw status = 90000245 89110106 00000020 00004000 00021F03 446003CB
  mei_hdcp 0000:00:16.0-b638ab7e-94e2-4ea2-a552-d1c54b627f04: mei_cldev_enable Failed. -14
  mei_hdcp 0000:00:16.0-b638ab7e-94e2-4ea2-a552-d1c54b627f04: probe with driver mei_hdcp failed with error -14
  mei_hdcp 0000:00:16.0-b638ab7e-94e2-4ea2-a552-d1c54b627f04: bound 0000:00:02.0 (ops i915_hdcp_ops [i915])
  mei_pxp 0000:00:16.0-fbf6fcf1-96cf-4e2e-a6a6-1bab8cbe36b1: bound 0000:00:02.0 (ops i915_pxp_tee_component_ops [i915])
  mei mei0: FW not ready: resetting: dev_state = 3
  mei mei0: unexpected reset: dev_state = ENABLED fw status = 90000245 89110106 00000020 00004000 00021F03 446003CB

Proposed Fix:
- Prevent vtag reuse during teardown: in mei_ioctl_connect_vtag(),
   ignore existing clients on dev->file_list if they are in
   MEI_FILE_DISCONNECTING or MEI_FILE_DISCONNECTED states or undergoing
   teardown.
- Implement proper reference counting / lifecycle synchronization on
   shared struct mei_cl instances.
- In drivers/misc/mei/interrupt.c, demux cl->rd_pending by matching the
   incoming packet's vtag header to the corresponding callback rather
   than assuming FIFO order.
- Add rate-limiting / backoff to MEI client reprobing during hardware
   link resets to prevent cascading bus storms into DRM / i915.

A minimal standalone C reproducer (repro_mei_vtag.c) is attached below.

WARNING: Running this reproducer triggers an unrecoverable hard freeze
and display deadlock requiring a forced restart.

===================================================================
Standalone C Reproducer (repro_mei_vtag.c)
===================================================================
/*
 * repro_mei_vtag.c - Minimal Standalone Reproducer for MEI Vtag Reconnect Race
 * Compilation: gcc -Wall -Wextra -O2 -pthread repro_mei_vtag.c -o repro_mei_vtag
 * Execution: sudo ./repro_mei_vtag /dev/mei0
 */
#define _GNU_SOURCE
#include <errno.h>
#include <fcntl.h>
#include <poll.h>
#include <pthread.h>
#include <signal.h>
#include <stdatomic.h>
#include <stdint.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/ioctl.h>
#include <unistd.h>
#pragma pack(push, 1)
typedef struct {
	uint8_t b[16];
} uuid_le;
struct mei_client {
	uint32_t max_msg_length;
	uint8_t protocol_version;
	uint8_t reserved[3];
};
struct mei_connect_client_data_vtag {
	union {
		struct {
			uuid_le in_client_uuid;
			uint8_t vtag;
			uint8_t reserved[3];
		} connect;
		struct mei_client out_client_properties;
	};
};
#pragma pack(pop)
#ifndef IOCTL_MEI_CONNECT_CLIENT_VTAG
#define IOCTL_MEI_CONNECT_CLIENT_VTAG \
	_IOWR('H', 0x04, struct mei_connect_client_data_vtag)
#endif
static const uuid_le MKHI_UUID = {.b = {0x15, 0x67, 0x6a, 0x8e, 0xbc, 0x9a,
					0x43, 0x40, 0x88, 0xef, 0x9e, 0x39,
					0xc6, 0xf6, 0x3e, 0x0f}};
static volatile sig_atomic_t g_running = 1;
static void sig_handler(int sig) { (void)sig; g_running = 0; }
static atomic_uint_fast64_t g_t1_ops = 0;
static atomic_uint_fast64_t g_t2_ops = 0;
static atomic_uint_fast64_t g_t3_ops = 0;
static int connect_vtag(const char *dev_path, uint8_t vtag) {
	int fd = open(dev_path, O_RDWR | O_NONBLOCK);
	if (fd < 0)
		return -1;
	struct mei_connect_client_data_vtag data;
	memset(&data, 0, sizeof(data));
	data.connect.in_client_uuid = MKHI_UUID;
	data.connect.vtag = vtag;
	if (ioctl(fd, IOCTL_MEI_CONNECT_CLIENT_VTAG, &data) < 0) {
		close(fd);
		return -1;
	}
	return fd;
}
static void *thread_vtag1(void *arg) {
	const char *dev = (const char *)arg;
	uint8_t cmd[4] = {0x07, 0x01, 0x00, 0x00};
	while (g_running) {
		int fd = connect_vtag(dev, 1);
		if (fd >= 0) {
			atomic_fetch_add(&g_t1_ops, 1);
			write(fd, cmd, sizeof(cmd));
			usleep(rand() % 40);
			close(fd);
		} else {
			usleep(50);
		}
	}
	return NULL;
}
static void *thread_vtag2(void *arg) {
	const char *dev = (const char *)arg;
	uint8_t cmd[4] = {0x07, 0x02, 0x00, 0x00};
	while (g_running) {
		int fd = connect_vtag(dev, 2);
		if (fd >= 0) {
			atomic_fetch_add(&g_t2_ops, 1);
			write(fd, cmd, sizeof(cmd));
			usleep(rand() % 60);
			close(fd);
		} else {
			usleep(50);
		}
	}
	return NULL;
}
static void *thread_vtag3(void *arg) {
	const char *dev = (const char *)arg;
	uint8_t cmd[4] = {0x07, 0x01, 0x00, 0x00};
	uint8_t buf[128];
	while (g_running) {
		int fd = connect_vtag(dev, 3);
		if (fd >= 0) {
			struct pollfd pfd = {.fd = fd, .events = POLLIN};
			for (int i = 0; i < 15 && g_running; i++) {
				if (write(fd, cmd, sizeof(cmd)) > 0) {
					atomic_fetch_add(&g_t3_ops, 1);
					if (poll(&pfd, 1, 50) > 0)
						read(fd, buf, sizeof(buf));
				}
				usleep(300);
			}
			close(fd);
		} else {
			usleep(500);
		}
	}
	return NULL;
}
int main(int argc, char **argv) {
	const char *dev = (argc > 1) ? argv[1] : "/dev/mei0";
	signal(SIGINT, sig_handler);
	signal(SIGTERM, sig_handler);
	int test_fd = connect_vtag(dev, 1);
	if (test_fd < 0) {
		fprintf(stderr, "[-] Failed to connect with vtag on %s: %s\n", dev, strerror(errno));
		return 1;
	}
	close(test_fd);
	pthread_t t1, t2, t3;
	pthread_create(&t1, NULL, thread_vtag1, (void *)dev);
	pthread_create(&t2, NULL, thread_vtag2, (void *)dev);
	pthread_create(&t3, NULL, thread_vtag3, (void *)dev);
	while (g_running) {
		printf("\r[+] T1 (vtag=1 close): %lu | T2 (vtag=2 race): %lu | T3 (vtag=3 io): %lu",
		       atomic_load(&g_t1_ops), atomic_load(&g_t2_ops), atomic_load(&g_t3_ops));
		fflush(stdout);
		usleep(250000);
	}
	pthread_join(t1, NULL);
	pthread_join(t2, NULL);
	pthread_join(t3, NULL);
	return 0;
}
===================================================================

Best regards,
Nirbhay Kumar

Re: [BUG] misc/mei: Race between mei_release() disconnect and mei_ioctl_connect_vtag() causes CSME reset storm and i915 freeze

[gregkh]

On Sat, Aug 29, 2026 at 07:14:10AM +0000, nirbhayykumarr@proton.me wrote:
> Hi all,
> 
> This issue was discovered using a custom multi-threaded C fuzzer
> designed to stress-test MEI Virtual Tag (vtag) client lifecycles and
> multiplexing over /dev/mei0. By concurrently racing rapid vtag
> connections against file descriptor closures and streaming I/O, a
> race condition is triggered during client teardown.

But that's not a normal use case, right?

> System Information & Environment:
> - Kernel Version: 7.1.8-zen1-3-zen x86_64 (drivers/misc/mei is identical to upstream)
> - Kernel Taint: 0 (Not tainted)
> - CPU / Platform: 12th Gen Intel(R) Core(TM) i5-12500H (Alder Lake-P)
> - MEI Controller: 00:16.0 Intel Alder Lake PCH HECI Controller (/dev/mei0)
> - Graphics / DRM: Intel Iris Xe Graphics (i915 driver with mei_hdcp and mei_pxp components)
> - Subsystems:     drivers/misc/mei, drivers/gpu/drm/i915
> 
> In mei_release(), closing the last file descriptor holding a virtual tag
> invokes mei_cl_disconnect(). Inside __mei_cl_disconnect(), dev->device_lock
> is dropped while awaiting the firmware disconnect ACK on cl->wait.
> 
> During this lock-drop window, a concurrent IOCTL_MEI_CONNECT_CLIENT_VTAG
> call on the same UUID (e.g. MKHI) scans dev->file_list, matches the tearing-
> down client 'pos' (in MEI_FILE_DISCONNECTING), repoints file->private_data
> to pos, and adds its new vtag to pos->vtag_map.
> 
> When the disconnect ACK arrives, __mei_cl_disconnect() calls
> mei_cl_set_disconnected(cl), setting cl->me_cl = NULL and
> cl->state = MEI_FILE_DISCONNECTED. Because pos->vtag_map now contains the
> second thread's tag, mei_release() skips unlinking/freeing cl. The second
> thread then wakes up and attempts to reconnect via mei_ioctl_connect_client().
> 
> Additionally, shared clients lack tag based demuxing on cl->rd_pending in
> drivers/misc/mei/interrupt.c:
>   cb = list_first_entry_or_null(&cl->rd_pending, struct mei_cl_cb, list);
> 
> Incoming packets are matched to the head of the FIFO queue regardless of
> vtag, causing -EPROTO errors, dropped packets, and out-of-order HBM command
> sequences. This triggers continuous CSME hardware link resets:
>   mei mei0: FW not ready: resetting: dev_state = 3
>   mei mei0: unexpected reset: dev_state = ENABLED fw status = ...
> 
> During each reset, child client drivers (mei_hdcp, mei_pxp) unbind and rebind
> with i915 DRM. With resets looping at hundreds of cycles per second (>9,800
> events in 27s), mei_cldev_enable() repeatedly fails with -EFAULT / -ENODEV,
> deadlocking i915 display worker mutexes in TASK_UNINTERRUPTIBLE and causing
> an unrecoverable full system freeze.
> 
> Journalctl logs:
>   mei mei0: FW not ready: resetting: dev_state = 3
>   mei mei0: unexpected reset: dev_state = ENABLED fw status = 90000245 89110106 00000020 00004000 00021F03 446003CB
>   mei_hdcp 0000:00:16.0-b638ab7e-94e2-4ea2-a552-d1c54b627f04: mei_cldev_enable Failed. -14
>   mei_hdcp 0000:00:16.0-b638ab7e-94e2-4ea2-a552-d1c54b627f04: probe with driver mei_hdcp failed with error -14
>   mei_hdcp 0000:00:16.0-b638ab7e-94e2-4ea2-a552-d1c54b627f04: bound 0000:00:02.0 (ops i915_hdcp_ops [i915])
>   mei_pxp 0000:00:16.0-fbf6fcf1-96cf-4e2e-a6a6-1bab8cbe36b1: bound 0000:00:02.0 (ops i915_pxp_tee_component_ops [i915])
>   mei mei0: FW not ready: resetting: dev_state = 3
>   mei mei0: unexpected reset: dev_state = ENABLED fw status = 90000245 89110106 00000020 00004000 00021F03 446003CB
> 
> Proposed Fix:
> - Prevent vtag reuse during teardown: in mei_ioctl_connect_vtag(),
>    ignore existing clients on dev->file_list if they are in
>    MEI_FILE_DISCONNECTING or MEI_FILE_DISCONNECTED states or undergoing
>    teardown.
> - Implement proper reference counting / lifecycle synchronization on
>    shared struct mei_cl instances.
> - In drivers/misc/mei/interrupt.c, demux cl->rd_pending by matching the
>    incoming packet's vtag header to the corresponding callback rather
>    than assuming FIFO order.
> - Add rate-limiting / backoff to MEI client reprobing during hardware
>    link resets to prevent cascading bus storms into DRM / i915.

Please send patches for this if you wish to see these issues addressed.
As you have a reproducer, it should be simple for you to do so.

And how does this differ from the patch you have already sent but has
not yet been merged?

thanks,

greg k-h

Re: [BUG] misc/mei: Race between mei_release() disconnect and mei_ioctl_connect_vtag() causes CSME reset storm and i915 freeze

[nirbhayykumarr]

On Saturday, August 29th, 2026 at 1:00 PM, gregkh@linuxfoundation.org <gregkh@linuxfoundation.org> wrote:

> On Sat, Aug 29, 2026 at 07:14:10AM +0000, nirbhayykumarr@proton.me wrote:
> > Hi all,
> >
> > This issue was discovered using a custom multi-threaded C fuzzer
> > designed to stress-test MEI Virtual Tag (vtag) client lifecycles and
> > multiplexing over /dev/mei0. By concurrently racing rapid vtag
> > connections against file descriptor closures and streaming I/O, a
> > race condition is triggered during client teardown.
> 
> But that's not a normal use case, right?

Yes, you are right. It is definitely not a normal, everyday
workload. it is an extreme edge case that I caught unintentionally while
stress-testing vtag multiplexing during a broader CSME research project.
But because it results in an unrecoverable full system kernel and
DRM/i915 display freeze when hit, I thought it was important to document
and report it.

> > In mei_release(), closing the last file descriptor holding a virtual tag
> > invokes mei_cl_disconnect(). Inside __mei_cl_disconnect(), dev->device_lock
> > is dropped while awaiting the firmware disconnect ACK on cl->wait.
> >
> > During this lock-drop window, a concurrent IOCTL_MEI_CONNECT_CLIENT_VTAG
> > call on the same UUID (e.g. MKHI) scans dev->file_list, matches the tearing-
> > down client 'pos' (in MEI_FILE_DISCONNECTING), repoints file->private_data
> > to pos, and adds its new vtag to pos->vtag_map.
> >
> > When the disconnect ACK arrives, __mei_cl_disconnect() calls
> > mei_cl_set_disconnected(cl), setting cl->me_cl = NULL and
> > cl->state = MEI_FILE_DISCONNECTED. Because pos->vtag_map now contains the
> > second thread's tag, mei_release() skips unlinking/freeing cl. The second
> > thread then wakes up and attempts to reconnect via mei_ioctl_connect_client().
> >
> > Additionally, shared clients lack tag based demuxing on cl->rd_pending in
> > drivers/misc/mei/interrupt.c:
> >   cb = list_first_entry_or_null(&cl->rd_pending, struct mei_cl_cb, list);
> >
> > Incoming packets are matched to the head of the FIFO queue regardless of
> > vtag, causing -EPROTO errors, dropped packets, and out-of-order HBM command
> > sequences. This triggers continuous CSME hardware link resets:
> >   mei mei0: FW not ready: resetting: dev_state = 3
> >   mei mei0: unexpected reset: dev_state = ENABLED fw status = ...
> >
> > During each reset, child client drivers (mei_hdcp, mei_pxp) unbind and rebind
> > with i915 DRM. With resets looping at hundreds of cycles per second (>9,800
> > events in 27s), mei_cldev_enable() repeatedly fails with -EFAULT / -ENODEV,
> > deadlocking i915 display worker mutexes in TASK_UNINTERRUPTIBLE and causing
> > an unrecoverable full system freeze.
> >
> > Proposed Fix:
> > - Prevent vtag reuse during teardown: in mei_ioctl_connect_vtag(),
> >    ignore existing clients on dev->file_list if they are in
> >    MEI_FILE_DISCONNECTING or MEI_FILE_DISCONNECTED states or undergoing
> >    teardown.
> > - Implement proper reference counting / lifecycle synchronization on
> >    shared struct mei_cl instances.
> > - In drivers/misc/mei/interrupt.c, demux cl->rd_pending by matching the
> >    incoming packet's vtag header to the corresponding callback rather
> >    than assuming FIFO order.
> > - Add rate-limiting / backoff to MEI client reprobing during hardware
> >    link resets to prevent cascading bus storms into DRM / i915.
> 
> Please send patches for this if you wish to see these issues addressed.
> As you have a reproducer, it should be simple for you to do so.

I appreciate the nudge. While the reproducer is straightforward, safely 
resolving the cross-subsystem interaction between MEI and i915 without 
introducing regressions seemed complex enough that I initially deferred 
to the Intel maintainers. 

However, since I've already mapped out the race in mei_ioctl_connect_vtag(), 
I am happy to take a shot at it. I will work on implementing the localized 
state checks to prevent vtag reuse during teardown, and I'll submit a 
patch for review in the next few days.

> And how does this differ from the patch you have already sent but has
> not yet been merged?

The previous patch addressed a single-client teardown ordering issue where 
incoming IRQ completions populated cl->rd_completed after mei_cl_flush_queues() 
was called, triggering a WARN_ON() in mei_cl_unlink().

This issue is a separate, multi-client race specific to Virtual Tag (vtag)
multiplexing in mei_ioctl_connect_vtag(). While mei_release() drops
dev->device_lock to wait for firmware disconnect, a concurrent thread
requesting a new vtag matches the tearing-down client on dev->file_list.
When the disconnect finishes, cl->me_cl is cleared/NULL'd while the client
is actively reused, resulting in CSME hardware reset loops and cascading
DRM/i915 display freezes.


Thanks,
Nirbhay Kumar