From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-wr1-f54.google.com (mail-wr1-f54.google.com [209.85.221.54]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B23641A23B6 for ; Tue, 28 Jan 2025 13:39:37 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.221.54 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738071580; cv=none; b=d4jxighK99kwgt57zPMUOTWhhtV1KwD5Cxdvwc8FqtEHmKyHsou5RLQwaI/VtGUDAb5N1f9gyO1RDqhcKXBQiab7GlpnZc8sODNyjUFaVOCxhZs87zhzKwLv6WEtdVf3JL5fuNHOAsUtyTlM1+TN3x62mJbd7cSjNRgfobJqHyA= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738071580; c=relaxed/simple; bh=K0chwOk8j+gfblQqMSn8zaQQwXrnqJlL2AW5smVHg8Q=; h=From:To:Cc:Subject:Date:Message-ID:MIME-Version; b=bW0wAS/vMmNzCwPTuVQZevQNHVuwD+Yt7BYFN0JmQ95axRjB3FmpTbllNO1C2JeKGSbBGN1OX8iN0M0ZU0SuKIuYY8ymbdogOMPBoGOGRp3oenAKN4IFTlBR7sCwbRnCOfyfSS5uhP+qibg5mFDdAUIAVyL7xok73U7ms4WoMmY= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=ionos.com; spf=pass smtp.mailfrom=ionos.com; dkim=pass (2048-bit key) header.d=ionos.com header.i=@ionos.com header.b=Q+JIiPXn; arc=none smtp.client-ip=209.85.221.54 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=ionos.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=ionos.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=ionos.com header.i=@ionos.com header.b="Q+JIiPXn" Received: by mail-wr1-f54.google.com with SMTP id ffacd0b85a97d-385dece873cso3057264f8f.0 for ; Tue, 28 Jan 2025 05:39:37 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ionos.com; s=google; t=1738071576; x=1738676376; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date:message-id:reply-to; bh=hK6gm4w3aip1p6X/TmteESgDCFv0ITdAvMjv3TFp//Y=; b=Q+JIiPXnE8L7CrxQrda/qLx8Kh57ysVoRMzz12yQfNw5Dk0gIIwodbAFoxtZh3TQab hl9L0lZkj7e0N8+RN+zo7U1b/dCmh6hY08HjPMoa5/oEX6ZE5Y3QiGuT34ywfVdvidMC G7bTuBY5D4vOOP6v8MibWARv9yU81I//Nj1r+eZIWfUFHpgHheKVflXkvoixbfs3zz85 vNFI2LAf5Z1jdJMxBXXWMjkyUVVHWmbgY1z5JbfQvCOLTA2bjiiCh68xAEWBlau5kZKR oZaERXVOWINVKYa6H/alMV15ekvre5JPRt21e43q8xa+aNgYZd1340cctFKgOpXmAHYK WdyQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1738071576; x=1738676376; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=hK6gm4w3aip1p6X/TmteESgDCFv0ITdAvMjv3TFp//Y=; b=pSt65zbEB8v/t6gR1MM7sVI9UhZNm9azAOaeHzyPaR6Lv5dbuvVbNhSvp87fkA3Tyc zCghIDNoexDDopUf9ZLPDgytzVm1nW0zIiSyjEqlqUd1aZ6lPhGyUyoX+TxG5W4MNwiV W2nrLD4q4v4+TZByelyDybWqJiHuxypF/0FayZiznzf6LqTfxDp7MTscHalshPayR97P 5iGZBJcBIO4CNG20afEQaPGEBKUiK8Vfdl0ljt72LAhpMLbS/R4oUZ7o9cFmLNQ7unNK 1PW66bWh0OMW/fjcDaoQzu96XH3x77YvvoVa0ez1HBWcuUseP2krO6Yu0vmqlwtzyeJC pefA== X-Forwarded-Encrypted: i=1; AJvYcCWI9fvtc64l+KPgmbheFNilaxCYj7ALm0FlypRUo8DJyPtHUi5lNZjMNCHZzA4dEqgqrLbCAjYE7mKBDDE=@vger.kernel.org X-Gm-Message-State: AOJu0YyLuJ1H633jYSkp1tLfP6v8OezJOW0GhU/I7cv0lf5yk24sVmDU Lkzg/myIe76diXgGJ8XyUEcfLBVndqXcE0ipM7q94GKpbLFgmDpDde01C56EunQ= X-Gm-Gg: ASbGnctylcsYE5dWqkzvFSut7dn+l8JcSF4e/x3yLwwB03T/njKqhXvKm0gAGavvZnw 7Dsc6BWE6QNSKoJhd1PLjANUijWdz+XwOdD3hFjtNzSUMXF49wuEzaxQ5XKgCFtkDGBYhEe0dtd /mYGQtBT1WIoXyn8J3mpnmwOmUzNxaRmeEEgTgJLOlIUkvLCWvWhsZOyA8W1czrBmvOghM2q4s1 dIXZ8WMfapdNh80mgjAluOYVW+5w1tjEhFhRoMs3XSRyP3QT/mTbqxnUAi92D4tv1O1vpJW84lo seYZti0P9r98DT+kTAWbnu1M2oGonprT1Urb2Z5hj6dvdAao10kjAuK0GdsHRq1ypmhJyMJfePV kioaF90ekBnjyU8g= X-Google-Smtp-Source: AGHT+IEa5+xB0pJnhusdcQsnmAOsVGLJFO6EMVx0QymFEmPYEr1LFSe9P9cZFAeEoCBOIOfrrx0nvQ== X-Received: by 2002:adf:a1cc:0:b0:38a:a074:9f3c with SMTP id ffacd0b85a97d-38bf5663dbcmr35768742f8f.16.1738071576010; Tue, 28 Jan 2025 05:39:36 -0800 (PST) Received: from raven.intern.cm-ag (p200300dc6f2b6900023064fffe740809.dip0.t-ipconnect.de. [2003:dc:6f2b:6900:230:64ff:fe74:809]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-38c2a1bb02dsm14160780f8f.70.2025.01.28.05.39.35 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 28 Jan 2025 05:39:35 -0800 (PST) From: Max Kellermann To: axboe@kernel.dk, asml.silence@gmail.com, io-uring@vger.kernel.org, linux-kernel@vger.kernel.org Cc: Max Kellermann Subject: [PATCH 0/8] Various io_uring micro-optimizations (reducing lock contention) Date: Tue, 28 Jan 2025 14:39:19 +0100 Message-ID: <20250128133927.3989681-1-max.kellermann@ionos.com> X-Mailer: git-send-email 2.45.2 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit While optimizing my io_uring-based web server, I found that the kernel spends 35% of the CPU time waiting for `io_wq_acct.lock`. This patch set reduces contention of this lock, though I believe much more should be done in order to allow more worker concurrency. I measured these patches with my HTTP server (serving static files and running a tiny PHP script) and with a micro-benchmark that submits millions of `IORING_OP_NOP` entries (with `IOSQE_ASYNC` to force offloading the operation to a worker, so this offload overhead can be measured). Some of the optimizations eliminate memory accesses, e.g. by passing values that are already known to (inlined) functions and by caching values in local variables. These are useful optimizations, but they are too small to measure them in a benchmark (too much noise). Some of the patches have a measurable effect and they contain benchmark numbers that I could reproduce in repeated runs, despite the noise. I'm not confident about the correctness of the last patch ("io_uring: skip redundant poll wakeups"). This seemed like low-hanging fruit, so low that it seemed suspicious to me. If this is a useful optimization, the idea could probably be ported to other wait_queue users, or even into the wait_queue library. What I'm not confident about is whether the optimization is valid or whether it may miss wakeups, leading to stalls. Please advise! Total "perf diff" for `IORING_OP_NOP`: 42.25% -9.24% [kernel.kallsyms] [k] queued_spin_lock_slowpath 4.79% +2.83% [kernel.kallsyms] [k] io_worker_handle_work 7.23% -1.41% [kernel.kallsyms] [k] io_wq_submit_work 6.80% +1.23% [kernel.kallsyms] [k] io_wq_free_work 3.19% +1.10% [kernel.kallsyms] [k] io_req_task_complete 2.45% +0.94% [kernel.kallsyms] [k] try_to_wake_up +0.81% [kernel.kallsyms] [k] io_acct_activate_free_worker 0.79% +0.64% [kernel.kallsyms] [k] __schedule Serving static files with HTTP (send+receive on local+TCP,splice file->pipe->TCP): 42.92% -7.84% [kernel.kallsyms] [k] queued_spin_lock_slowpath 1.53% -1.51% [kernel.kallsyms] [k] ep_poll_callback 1.18% +1.49% [kernel.kallsyms] [k] io_wq_free_work 0.61% +0.60% [kernel.kallsyms] [k] try_to_wake_up 0.76% -0.43% [kernel.kallsyms] [k] _raw_spin_lock_irqsave 2.22% -0.33% [kernel.kallsyms] [k] io_wq_submit_work Running PHP script (send+receive on local+TCP, splice pipe->TCP): 33.01% -4.13% [kernel.kallsyms] [k] queued_spin_lock_slowpath 1.57% -1.56% [kernel.kallsyms] [k] ep_poll_callback 1.36% +1.19% [kernel.kallsyms] [k] io_wq_free_work 0.94% -0.61% [kernel.kallsyms] [k] _raw_spin_lock_irqsave 2.56% -0.36% [kernel.kallsyms] [k] io_wq_submit_work 2.06% +0.36% [kernel.kallsyms] [k] io_worker_handle_work 1.00% +0.35% [kernel.kallsyms] [k] try_to_wake_up (The `IORING_OP_NOP` benchmark finishes after a hardcoded number of operations; the two HTTP benchmarks finish after a certain wallclock duration, and therefore more HTTP requests were handled.) Max Kellermann (8): io_uring/io-wq: eliminate redundant io_work_get_acct() calls io_uring/io-wq: add io_worker.acct pointer io_uring/io-wq: move worker lists to struct io_wq_acct io_uring/io-wq: cache work->flags in variable io_uring/io-wq: do not use bogus hash value io_uring/io-wq: pass io_wq to io_get_next_work() io_uring: cache io_kiocb->flags in variable io_uring: skip redundant poll wakeups include/linux/io_uring_types.h | 10 ++ io_uring/io-wq.c | 230 +++++++++++++++++++-------------- io_uring/io-wq.h | 7 +- io_uring/io_uring.c | 63 +++++---- io_uring/io_uring.h | 2 +- 5 files changed, 187 insertions(+), 125 deletions(-) -- 2.45.2