From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-il1-f178.google.com (mail-il1-f178.google.com [209.85.166.178]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 09B0A18FC65 for ; Wed, 29 Jan 2025 17:45:46 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.166.178 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738172749; cv=none; b=qwwf3nJlOUc0SmvoAaJ/vxBZHylZbHj5lVW6VNbqDz4WH+b8hBrjzb67qp6U0NN+PXJrEnkt9eJYD1Ma+F1SbMfB2y50eKCvcM6S0fIdnGF8tW9XGtxvM2MrxKcvU3Bwscbhr9leX7+71PLlmu4L0jpjSirdRwkSRz9CFmx5s78= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738172749; c=relaxed/simple; bh=SLeqwYno/5jSjNKjpjCOzjY6UT7jbbPK5g022EczFGo=; h=Message-ID:Date:MIME-Version:Subject:To:Cc:References:From: In-Reply-To:Content-Type; b=gHwSAJZaRkeewAyAXW6NhQvxI4zDB6xe+MwyvqkB84jMytlhJVZ7T0hO2imKASpuMIkYhQ0D6kk2WF2/UJ4yIIGywVpNWW9Vxsl7VNU1nlwj5rbljWIIE5HzPxCOf12QNW8YdgjGdsok4xCAtqgtQa33TFHvWAzZN8yxEE4TokA= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=ZrBpWfyv; arc=none smtp.client-ip=209.85.166.178 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="ZrBpWfyv" Received: by mail-il1-f178.google.com with SMTP id e9e14a558f8ab-3cfc8a2415fso19020015ab.2 for ; Wed, 29 Jan 2025 09:45:46 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1738172746; x=1738777546; darn=vger.kernel.org; h=content-transfer-encoding:in-reply-to:from:content-language :references:cc:to:subject:user-agent:mime-version:date:message-id :from:to:cc:subject:date:message-id:reply-to; bh=0cX5eV5zH8zQT7dGq+IpzEAUG7LZB8+/TXgDA2I5kwM=; b=ZrBpWfyvWItSWlvwfEinU+ho/SwOkatYvWBr9prnR6hJmb1/otStA3t8GG5cw4INwn t786hiCMsqip+PKTZNEwg+A2B86cY2sAkpcvPM7yn/HCfr1vBEKXcsDxxO45lCrdw+B6 jq/GJt8m/PP1u++lTtkLWrE3PcFC3pEQlZn47JeLm+BvWcYeuWMfNi1Z1FY1oxQHEGI+ g76uLDg4deRcXUaEIHYarm2j85wwcThS9PUTzfRNYN5cZao2OY47Ufg3XXhSD5fwZLev IdrPPYwNb3babwHLNfXcaS4qFHtVClfGWp3kUHI5BA/tmu2rejfjtwYl0JmmwVxxvgkM mgaw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1738172746; x=1738777546; h=content-transfer-encoding:in-reply-to:from:content-language :references:cc:to:subject:user-agent:mime-version:date:message-id :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=0cX5eV5zH8zQT7dGq+IpzEAUG7LZB8+/TXgDA2I5kwM=; b=RZ5RoW8NCFCWlnKbl9cCl/GzKYPY0JOeFFvEGT5F54NKyV/Ah/KIzMMiIimCBSjBH0 mU/qS8z7TeSUTOog87/Kvrt+bqq4i2vWvt/OaIK+HJJYlQ2p8g1MhU5vsKCrR8ZzNQ1H xin37FiFfCL10U50yAZFtn6wQzyd36PdNwXILUNPUiVXz+ajVUZsykqbABQoaihZpMdm 9uHQ3uZAyFV8VVAMyUwZxoAepJkdk4gXbaEENnQpxXv1LrYmLkFmqDoodM7y6Yrxvh+G XjQl7YpNcKlyrPtwoaeE+LZS9Mq6gGlDug1y1JoVw3bc/5JUbaYIEqgFxOeZlZ6E/oHV cnWQ== X-Forwarded-Encrypted: i=1; AJvYcCU25MQ6PYiKdrssN1Cp6DBe1OSq4lagSHGa/2+C9uhH62K9hXJ6NMYjQhQlk1i4JbCAL66PAXKRGSWAVBs=@vger.kernel.org X-Gm-Message-State: AOJu0YwNvm6kZbnxAQmmN1AhALb3t6DEcyl0rRTyK+8Vj9lw2zzQ6FQZ LBL2aJ30sb/sd6AJ2P52qwsOdsf4LjT8dQFT4Dk939aDhkf3a/bCVrM1Tnp7pXA= X-Gm-Gg: ASbGncthf9ooTiRUhre94hT/AhP7ICwnlddxPVMnThZTVerEsJSKzNBqaH5Kn2vjonh R6yYNpQwGkhHPdmx1BD1KEGWpGvKaxkNNDMvV6vC5Oa0KgtzFrTB+s5HmvVYZggbWahoMy03Qzs WMIDZUQSpzEBuNuxq9UKCBUg/RxQnibpknkpChHv4xURFMrYcCeYtYiYaFf3fIATSj1mZE7Ps7Q HDfoOXMIYYqdpdavmyoMmDMlcEAPzvovLbq6YURuui/9GKuvNSTUvhmpbiYGJ2kb6INLB+YYAOp VXdpLqUn1ws= X-Google-Smtp-Source: AGHT+IE5mR4efwEP14GeO5jDtJc2e5N4jJYuXLCM4aHVGobgckQ8tei4eb6isuAKg48a2RKhuQJGVg== X-Received: by 2002:a05:6e02:1fe4:b0:3cf:fa94:cad with SMTP id e9e14a558f8ab-3cffe3a6a12mr34535845ab.8.1738172746065; Wed, 29 Jan 2025 09:45:46 -0800 (PST) Received: from [192.168.1.116] ([96.43.243.2]) by smtp.gmail.com with ESMTPSA id 8926c6da1cb9f-4ec1da31b81sm3892197173.52.2025.01.29.09.45.45 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Wed, 29 Jan 2025 09:45:45 -0800 (PST) Message-ID: Date: Wed, 29 Jan 2025 10:45:43 -0700 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [PATCH 0/8] Various io_uring micro-optimizations (reducing lock contention) To: Max Kellermann Cc: asml.silence@gmail.com, io-uring@vger.kernel.org, linux-kernel@vger.kernel.org References: <20250128133927.3989681-1-max.kellermann@ionos.com> <7f046e6e-bb9d-4e72-9683-2cdfeabf51bc@kernel.dk> Content-Language: en-US From: Jens Axboe In-Reply-To: Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit On 1/29/25 10:39 AM, Max Kellermann wrote: > On Wed, Jan 29, 2025 at 6:19?PM Jens Axboe wrote: >> The other patches look pretty straight forward to me. Only thing that >> has me puzzled a bit is why you have so much io-wq activity with your >> application, in general I'd expect 0 activity there. But Then I saw the >> forced ASYNC flag, and it makes sense. In general, forcing that isn't a >> great idea, but for a benchmark for io-wq it certainly makes sense. > > I was experimenting with io_uring and wanted to see how much > performance I can squeeze out of my web server running > single-threaded. The overhead of io_uring_submit() grew very large, > because the "send" operation would do a lot of synchronous work in the > kernel. I tried SQPOLL but it was actually a big performance > regression; this just shifted my CPU usage to epoll_wait(). Forcing > ASYNC gave me large throughput improvements (moving the submission > overhead to iowq), but then the iowq lock contention was the next > limit, thus this patch series. > > I'm still experimenting, and I will certainly revisit SQPOLL to learn > more about why it didn't help and how to fix it. Why are you combining it with epoll in the first place? It's a lot more efficient to wait on a/multiple events in io_uring_enter() rather than go back to a serialize one-event-per-notification by using epoll to wait on completions on the io_uring side. -- Jens Axboe