[BUG] AMX tile data lost after nanosleep in a Linux guest

[BUG] AMX tile data lost after nanosleep in a Linux guest

[yoch melka]

[-- Attachment #1.1: Type: text/plain, Size: 1712 bytes --]

Hi,

I'm seeing AMX tile data silently turn to zero after a 1 ms sleep on a
Linux guest.

The reproducer requests AMX permission, loads a known nonzero pattern into
all eight tiles, calls `nanosleep` through a direct syscall, then stores
and checks the contents. There are no function calls while the tiles are
live; I checked the disassembly with both GCC and Clang.

Build and run on an AMX-capable CPU (replace CPU 0 with an allowed CPU if
necessary):

```
gcc -O2 -g -Wall -Wextra -Werror -mamx-tile -fno-plt amx_min.c -o amx_min

timeout --kill-after=2s 20s taskset -c 0 ./amx_min
timeout --kill-after=2s 20s taskset -c 0 ./amx_min getpid
```

Expected: 0/300 mismatches in both modes.

Observed with GCC 13.3.0: 5/300, 5/300 and 4/300 mismatches in three
`nanosleep` runs. Clang 18.1.3 also reproduces it (3/300). The `getpid`
control was clean in all three control runs (two GCC, one Clang).

The first mismatch reports:
First mismatch: 8192/8192 bytes are zero; TILECFG unchanged

`nanosleep` returns successfully. The failure is intermittent, so a clean
run may need repeating.

Environment:
Ubuntu 24.04, custom Linux 6.12.94+
#1 SMP PREEMPT_DYNAMIC Mon Sep 7 16:15:06 UTC 2026
Intel Xeon, family 6, model 207, stepping 2; 4 vCPUs
KVM reported by the guest; host kernel and VMM unknown

A separate test also loses tile data in a busy loop under CPU contention,
without any syscall while the tiles are live. This suggests an AMX
state-preservation issue around scheduling, but I cannot tell whether the
fault is in the custom guest kernel or the host environment.

Does this resemble a known issue?
Any suggestions for narrowing it down from inside the guest would be
appreciated.

Thanks,
Joshua Melka

[-- Attachment #1.2: Type: text/html, Size: 1846 bytes --]

[-- Attachment #2: amx_min.c --]
[-- Type: text/x-csrc, Size: 2693 bytes --]

/* SPDX-License-Identifier: MIT */
#define _GNU_SOURCE
#include <cpuid.h>
#include <errno.h>
#include <immintrin.h>
#include <stdio.h>
#include <string.h>
#include <sys/syscall.h>
#include <time.h>
#include <unistd.h>

static _Alignas(64) unsigned char cfg[64], cfg_after[64];
static _Alignas(64) unsigned char src[8][1024], dst[8][1024];
static const struct timespec delay = { .tv_nsec = 1000000 };
#define TILES(OP) OP(0); OP(1); OP(2); OP(3); OP(4); OP(5); OP(6); OP(7)
#define LOAD(t) _tile_loadd(t, src[t], 64)
#define STORE(t) _tile_stored(t, dst[t], 64)

/* No function calls from LDTILECFG until TILERELEASE. */
__attribute__((noinline)) long roundtrip(long nr)
{
    _tile_loadconfig(cfg);
    TILES(LOAD);
    long ret;
    __asm__ volatile("syscall" : "=a"(ret)
        : "0"(nr), "D"(&delay), "S"(0L)
        : "rcx", "r11", "memory", "cc");
    _tile_storeconfig(cfg_after);
    for (int i = 0; i < 64; i++) {
        if (cfg[i] != ((volatile unsigned char *)cfg_after)[i]) {
            _tile_release();
            return -4096; /* Configuration changed; do not store tiles. */
        }
    }
    TILES(STORE);
    _mm_mfence();
    _tile_release();
    return ret;
}

int main(int argc, char **argv)
{
    if (argc > 2 || (argc == 2 && strcmp(argv[1], "getpid"))) {
        fprintf(stderr, "Usage: %s [getpid]\n", argv[0]);
        return 2;
    }
    unsigned a, b, c, d;
    if (!__get_cpuid_count(7, 0, &a, &b, &c, &d) || !(d & (1u << 24))) {
        puts("SKIP: AMX_TILE not exposed");
        return 77;
    }
    if (syscall(SYS_arch_prctl, 0x1023 /* ARCH_REQ_XCOMP_PERM */, 18L)) {
        perror("ARCH_REQ_XCOMP_PERM");
        return 77;
    }
    cfg[0] = 1;
    for (int t = 0; t < 8; t++) {
        cfg[16 + 2*t] = 64;
        cfg[48 + t] = 16;
        for (int i = 0; i < 1024; i++)
            src[t][i] = 1 + ((t*1024 + i) % 251);
    }
    int bad = 0;
    for (int n = 0; n < 300; n++) {
        memset(dst, 0xcc, sizeof dst);
        memset(cfg_after, 0xcc, sizeof cfg_after);
        long ret = roundtrip(argc == 2 ? SYS_getpid : SYS_nanosleep);
        if (ret == -4096) { puts("FAIL: TILECFG changed"); return 2; }
        if (ret < 0) { errno = (int)-ret; perror("syscall"); return 2; }
        if (memcmp(src, dst, sizeof src)) {
            if (bad++ == 0) {
                int zeros = 0;
                for (int t = 0; t < 8; t++)
                    for (int i = 0; i < 1024; i++) zeros += dst[t][i] == 0;
                printf("First mismatch: %d/8192 bytes are zero; TILECFG unchanged\n", zeros);
            }
        }
    }
    printf("%d/300 trials lost tile data (%s)\n", bad, argc == 2 ? "getpid" : "nanosleep");
    return bad ? 1 : 0;
}

Re: [BUG] AMX tile data lost after nanosleep in a Linux guest

[Sean Christopherson]

On Thu, Sep 10, 2026, yoch melka wrote:
> Hi,
> 
> I'm seeing AMX tile data silently turn to zero after a 1 ms sleep on a
> Linux guest.
> 
> The reproducer requests AMX permission, loads a known nonzero pattern into
> all eight tiles, calls `nanosleep` through a direct syscall, then stores
> and checks the contents. There are no function calls while the tiles are
> live; I checked the disassembly with both GCC and Clang.
> 
> Build and run on an AMX-capable CPU (replace CPU 0 with an allowed CPU if
> necessary):
> 
> ```
> gcc -O2 -g -Wall -Wextra -Werror -mamx-tile -fno-plt amx_min.c -o amx_min
> 
> timeout --kill-after=2s 20s taskset -c 0 ./amx_min
> timeout --kill-after=2s 20s taskset -c 0 ./amx_min getpid
> ```
> 
> Expected: 0/300 mismatches in both modes.
> 
> Observed with GCC 13.3.0: 5/300, 5/300 and 4/300 mismatches in three
> `nanosleep` runs. Clang 18.1.3 also reproduces it (3/300). The `getpid`
> control was clean in all three control runs (two GCC, one Clang).
> 
> The first mismatch reports:
> First mismatch: 8192/8192 bytes are zero; TILECFG unchanged
> 
> `nanosleep` returns successfully. The failure is intermittent, so a clean
> run may need repeating.
> 
> Environment:
> Ubuntu 24.04, custom Linux 6.12.94+
> #1 SMP PREEMPT_DYNAMIC Mon Sep 7 16:15:06 UTC 2026
> Intel Xeon, family 6, model 207, stepping 2; 4 vCPUs
> KVM reported by the guest; host kernel and VMM unknown

This suggests that your VM is being hosted by a CSP/third-party.  Is that correct?

> A separate test also loses tile data in a busy loop under CPU contention,
> without any syscall while the tiles are live. This suggests an AMX
> state-preservation issue around scheduling, but I cannot tell whether the
> fault is in the custom guest kernel or the host environment.
> 
> Does this resemble a known issue?

I can't think of anything off the top of my head.

> Any suggestions for narrowing it down from inside the guest would be
> appreciated.

I would first try reproducing the failure with an upstream guest kernel.  If you
can repro with an upstream guest kernel, and your VM is being hosted by a third-
party, then I would contact their support (file a bug?) and provide them your
reproducer and guest kernel version.

If you have access to bare metal with AMX, I would definitely try reproing there.
E.g. if it repros on bare metal with an upstream kernel, then you can get support
directly from upstream.  Or if it repros only with your custom kernel and on bare
metal, then you can be pretty darn confident it's a bug in that custom kernel.
And if it doesn't repro on bare metal, then you can go yell at your CSP (and hope
it wasn't a false negative). :-)

In other words, I would first do what you can to narrow down where in the stack
the bug likely resides before diving straight into debug.
 
I ran your reproducer with a v7.2 guest kernel and a v6.12-based host kernel as
well as a vanilla v7.3-rc2 host kernel, and didn't see any failures across 50k
"trials" of each host kernel (100k trials total).  That doesn't guarantee upstream
isn't buggy, but without knowing the host kernel, and without having a reproducer
that is relevant to upstream, there's not really anything us upstream folks can
do to help.  Sorry :-(