From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-oi2-f1.google.com (mail-oi2-f1.google.com [74.125.231.193]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C3E4C43C7D7 for ; Thu, 30 Jul 2026 14:34:18 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=74.125.231.193 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1785422062; cv=none; b=hL6k3rxogyRdNE+ygyFrjIxdx664maPHndwvIHWwua5t/qAINEc0y6aoDt3hkG0tFRYYteb+/r16D7ev3Uo4AitWjARweP7JRTQtCOfgkkQA1NTWbJN2ihI2V+L4BMf4ITWtutHIFuj+jqlJK2s7xqVwWziU2zjq4DQ1TJ12yDw= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1785422062; c=relaxed/simple; bh=TB9COi3LZwqdeW20gV3IacB42nQh2XS8QzExjKq/IG4=; h=Message-ID:Date:MIME-Version:Subject:To:Cc:References:From: In-Reply-To:Content-Type; b=bEwnBhHhk9XM6lKDZSzsyYRm9sGhw+yUJzTvm+bZhC7fT+BJvoCZvZX+FbBkN4k7WVfTiaaUTaDE2Z/So5a9nwGy+sjnyatJD/xJV7SR+JnlfkqONl26muN4tdBk77FqXcKls1EjO7UG6ElnY0jeU4yfRzFmx/z89fMj/ImTZuI= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20251104.gappssmtp.com header.i=@kernel-dk.20251104.gappssmtp.com header.b=W4C662Sm; arc=none smtp.client-ip=74.125.231.193 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20251104.gappssmtp.com header.i=@kernel-dk.20251104.gappssmtp.com header.b="W4C662Sm" Received: by mail-oi2-f1.google.com with SMTP id 5614622812f47-4960cd51443so223273b6e.1 for ; Thu, 30 Jul 2026 07:34:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20251104.gappssmtp.com; s=20251104; t=1785422057; x=1786026857; darn=vger.kernel.org; h=content-transfer-encoding:content-type:in-reply-to:from :content-language:references:cc:to:subject:user-agent:mime-version :date:message-id:from:to:cc:subject:date:message-id:reply-to :content-type; bh=sL74mQw0WBxRoWRghx17Ng3Z07+sXaKozIlj/fsplB0=; b=W4C662Sm0V5xzscBhax4gHQ1noaQJaD7a3IHjaqCKtoiiKriQXyNcxzFAqoRdwnSzU BwWzvhbewCoSOSRZJgAzEiETEyqGxceOOh5siMpTm8HzDHR84wLdNrgHAqj8y+RGSjBB kThppv/NKSB/sWw+vUYJ+lJk6BAtQDujBw57hjNKY6q/ClJm+HcGtnpGByJyloi7Vgnd EMkLHI+/0P+lF1y2/V5B+Q4v2dkpW34Ky8X8d2+XpamteNqwydFFX4aCPh2yvn0qyPRJ 5IVOIT4CCtSXqd/loSlg/3qSXBFIcO/GATkMI0KWOxDO93G4uoHnudq+rCBuQxnOXBwW +ghw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1785422057; x=1786026857; h=content-transfer-encoding:content-type:in-reply-to:from :content-language:references:cc:to:subject:user-agent:mime-version :date:message-id:x-gm-gg:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to:content-type; bh=sL74mQw0WBxRoWRghx17Ng3Z07+sXaKozIlj/fsplB0=; b=d5WyJlKgqbnW6I2oVdiN/FvXYzZVcxDdWRUpfb+wUw8ss1CmPfAfEwzrFgM8OZUTm8 uNEr8PH3MYK2PGXvaHsJBa/+amRG3FY2Ee3chUArXYs12QJV96gKr/hcljgutlle0iWn UShzU+/eF+T2pvQWUxQjnR4EoTW70zhJW9LH2mf8ULIvoZMdp+dO0muEjzT85PoQ+RbT C7dCc6wVxSKitOOHsOQMMOU6Kz8zIyGxmkh49Me6TOO4garoDcd0l7Qyz2k4Negyb9pL wcCArkRRIV2O+O98GLKuumy4wzk90bw5ApXhX0YuSvY8bKTT6JmZqXLBuAm2IztJMdh0 WzkA== X-Forwarded-Encrypted: i=1; AHgh+RqH/BhkAWNeQYBqc97bZnKHVbnZW+LwxWmh2CFS//4lSnZENz6+NiXlOrDUduvweleHJVedqVnjrMu0BJI=@vger.kernel.org X-Gm-Message-State: AOJu0YwKpofcfjOS0wBMTHKFqgmjlSz/jwdxurQDb66OVw/wiNr6rOs2 +IaHrGQAyGpYwz182wfFeegBAi3Uu9BJgMCAw0sgknTVc3WX+A9jyBQ67QSYo9BDE7ABIhXvDue dAL03KoUaeR2a X-Gm-Gg: AR+sD1163kE1Hfu7f4hiRvBpzIbKIBIsEFJ8FtGcq3Lwk1/tk8MRe020rYqKIOG+vwf pnSTxsW5nMMQnQiY2O8IvyXf//XxwykWYEjKYCKwOAfgZp7BvOaXFQsnjkbjE/ao8WQMssFfd1Y n4GWKH4/KViQIjlY5PK5S249dVc8kA80yTFbgrm03NxMAzwBqYHzTtjMyJV44nN7Ou0wzKLRKrC fmu8lmPcTZKd5p77XexEjIk6FdFZT612mrJ9phLD5b+b1MrswCu0n+MERv/ZKoEtQlMaDGdxcYP jyIYLUBA1+K+MeAhn02V+a0ve5JTOM2/oLU9qrR2YYTHBdLTKMmkrRdxY8E2j4Elw62cLKnTam+ xECEgaFiw/NSMmCWU7RCl65qo/LXDTfsXCaTffaPkZtpTVoIDE5xmeFwTJb6dK3Y9q+/p/L3e5z 1yvUU9/PVhWcx/yeDsBkHhJ7ghreym9h2+pONhITT2Jm9xSgORdmN/6PRDOg+elaWLs6a0ZJ/VF UrUtao0/O/sEBHCi/5KRLGFFBVZ5tRrj3c/C09Dlt6o80NZOLLzyt8v X-Received: by 2002:a05:6808:159a:b0:492:9e13:d5a7 with SMTP id 5614622812f47-4ad876c9e1amr2001691b6e.6.1785422057412; Thu, 30 Jul 2026 07:34:17 -0700 (PDT) Received: from [192.168.1.102] ([96.43.243.2]) by smtp.gmail.com with ESMTPSA id 5614622812f47-4ad6ecb8040sm4033444b6e.1.2026.07.30.07.34.16 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Thu, 30 Jul 2026 07:34:16 -0700 (PDT) Message-ID: Date: Thu, 30 Jul 2026 08:34:15 -0600 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [REGRESSION] io_uring/futex: scalar wait/wake slowdown after 079afb081c42 To: Chengfeng Lin Cc: Pavel Begunkov , Robert Morris , io-uring@vger.kernel.org, linux-kernel@vger.kernel.org, regressions@lists.linux.dev References: Content-Language: en-US From: Jens Axboe In-Reply-To: Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit On 7/30/26 4:42 AM, Chengfeng Lin wrote: > Hi Jens, > > I tested 079afb081c42 against its direct parent on bare metal. In a narrow > scalar io_uring futex wait/wake workload, the child was 9.27% slower. A > separate 338-line standalone reproduced the result at 10.02%. All compared > kernels actually ran with preempt=full. > > #regzbot introduced: 079afb081c4288e94d5e4223d3eb6306d853c68b > #regzbot title: io_uring scalar futex wait/wake slowdown > > This is a focused synthetic microbenchmark, not an application benchmark. It > uses one raw-UAPI ring and 32 cacheline-separated private futex words on one > pinned P-core. Each timed cycle submits 32 scalar IORING_OP_FUTEX_WAIT > requests, then 32 scalar IORING_OP_FUTEX_WAKE requests, and drains exactly 64 > CQEs. Every wait must return 0 and every wake must return 1. > > I used a fresh boot for each point: > > 6a8118a77eec parent A -> 079afb081c42 child -> 6a8118a77eec parent B > > Each point had 3 warm-up rounds and 15 measured rounds. Every measured round > ran 512 cycles, or 16,384 wait/wake pairs. The results in ns/pair were: > > implementation parent A child parent B child vs midpoint > formal 180.079 196.647 179.856 +9.268% > standalone 180.171 197.856 179.502 +10.020% > > For the formal source, dropping the first measured round gave +9.269%. Parent > drift was -0.124%, and the maximum CV was 0.169%. The standalone drop-first > result was +9.996%, with -0.371% parent drift. All 90 scalar timing rows > passed the CQE, result, timeout, overflow, outstanding-request, and CPU > checks. > > An untimed child trace also hit io_futex_prep(), io_futex_wait(), > io_futex_wake(), and io_futex_complete() with the expected request counts. > > A matched WAITV -> WAKE profile changed by only +1.385%, below my preregistered > 5% signal gate, so my claim is limited to scalar wait/wake. > > I understand that 079afb fixes the exit-time use-after-free by keeping pending > private futex waits visible to cancellation before their mm state disappears. > Scalar WAIT and WAKE both use io_futex_prep(), so in the child both sides of > each measured pair execute the added tracking call. I am not suggesting a > revert. > > Is this per-request cost an expected trade-off for the lifetime fix, or could > the same exit/mm-lifetime guarantee be retained with cheaper tracking? > > Evidence bundle: > > https://github.com/lcf0399/linux-regression-evidence/tree/65c8cbf86f40cbe759e3f7db1d29d152ba03a8f2/io-uring-futex-inflight-wait-wake > > Standalone reproducer: > > https://github.com/lcf0399/linux-regression-evidence/tree/65c8cbf86f40cbe759e3f7db1d29d152ba03a8f2/io-uring-futex-inflight-wait-wake/reproducer Great report, thanks for that! I'll take a look at this. The inflight tracking is a bit of a big hammer for sure for this, and it isn't even needed on the wake side. Can you tell me what parameters you're using for the reproducers? -- Jens Axboe