From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mx0a-00364e01.pphosted.com (mx0a-00364e01.pphosted.com [148.163.135.74]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9E8B93FBA7 for ; Mon, 22 Jun 2026 17:04:36 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=148.163.135.74 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1782147878; cv=none; b=WMhjyxQm7CiGCCErTlKxVcNdR67TnLjTaifsa7h8N4X6wxCJKP9bGZPU90IxT4u9n9GPKXMp1fqdgsNwSv6Ppp9+myCn+aoxVbU2T9jQU2rhTOr3tHyRGcitF7vqEiHWjSr+CXcyP6r4XJGBv2KYkPtDGyC6b8hpGMgr0uTWeLc= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1782147878; c=relaxed/simple; bh=YPns9BMqruoPxxfbzv9I+MeJ+qdUDzUn5e+bYYe7QOs=; h=Message-ID:Date:MIME-Version:Subject:To:Cc:References:From: In-Reply-To:Content-Type; b=fmGMDkRcWInBMjQUTsu4nIUlcKWBCbytGg/AFEjLfdklHwOxvRrLWSUFKDueP1SQgJoMk1257NW9+n3HMNRXm2T3S5ZEexNMY3wvKASL32hBjkvln9heV2RMZ5DI2m1Atc+ficsnQTKNu5+yWHi4GGHDERsDa8rnANXkm1YJOE0= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=columbia.edu; spf=pass smtp.mailfrom=columbia.edu; dkim=pass (2048-bit key) header.d=columbia.edu header.i=@columbia.edu header.b=OgqViIyl; arc=none smtp.client-ip=148.163.135.74 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=columbia.edu Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=columbia.edu Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=columbia.edu header.i=@columbia.edu header.b="OgqViIyl" Received: from pps.filterd (m0167070.ppops.net [127.0.0.1]) by mx0a-00364e01.pphosted.com (8.18.1.11/8.18.1.11) with ESMTP id 65MGjHST2146934 for ; Mon, 22 Jun 2026 12:45:29 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=columbia.edu; h= cc:content-transfer-encoding:content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to; s=pps01; bh=xdMm a0yFZmweYohmNWNZydRh/c/N/8MYIz6Kz390Uw8=; b=OgqViIylG77wjYnZddWf 88YfH8KC/P+mTnqfevI0Too4wjycZ3p6FDhKIuH4gSeU6WIqYUin9SIwCTR6PaKZ 045ca89rm3XjdDbD88fhES2ssfW6qKxDwQin7jbln+eDXhZkvt3rzuDRgmWjHMRt bXYHTOkWmIAFxgnpV4zpM7IRvRFLSqvlRe9rO0LhHgRiw57Lo6kU9bc2l1P547kk e9qbhjV/IUmE4cBLJp0ph0MjvifIWO9Cbl1cu3blssXdXcGOmioJG41eIAP7Lg8m yUF+mqa00i0REXrcqjro2yGFuw1WO8p8bDvLjCkeQhbdW59jQV35RatkKdld/Er1 Cw== Received: from mail-qt1-f198.google.com (mail-qt1-f198.google.com [209.85.160.198]) by mx0a-00364e01.pphosted.com (PPS) with ESMTPS id 4exmrhnv8r-1 (version=TLSv1.3 cipher=TLS_AES_128_GCM_SHA256 bits=128 verify=NOT) for ; Mon, 22 Jun 2026 12:45:28 -0400 (EDT) Received: by mail-qt1-f198.google.com with SMTP id d75a77b69052e-519fdfbd340so1219601cf.1 for ; Mon, 22 Jun 2026 09:45:28 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1782146728; x=1782751528; h=content-transfer-encoding:in-reply-to:from:content-language :references:cc:to:subject:user-agent:mime-version:date:message-id :x-gm-gg:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=xdMma0yFZmweYohmNWNZydRh/c/N/8MYIz6Kz390Uw8=; b=d8gPJds2OEvrESTkBQ9loiyeH3gqnLsubo16AlU+GzP0KN3nJIkh2789hpseWEfOXg 4Kl2fzCoZXgesEbw36bAD7nu7IfPny8MGfak+A9AYhHJnCsKgTwex30sdbyhrqX7YvDw eeb2fArfSolozpINgTFNxtsZhZc7UALInz51PFcmRAdmMxLg0oyShiswdovLtDfV4sFL O/jtPPAAc5d80kgPXGr6cHXRXuG+NTK93Uzqhbr/vkenMTU2PKH4Zp5G66Hg0V1fDuQm Wqu3zcp4qtx3uGxGJ2XNuHTznLezAp3yAvF0CAfeKEcFMcL28s9ay6+SXh2JIeuSG3UY 0PAQ== X-Forwarded-Encrypted: i=1; AFNElJ96sVFDJKi4zhbQ6hc2dOJunvXQbvAmf6j7uoaUmkAghRSky3oIedd7WaUHkX8wOaEWXx9EC2gUhdRtU4U=@vger.kernel.org X-Gm-Message-State: AOJu0YxQGGzW33dPVMFGy1Tn30nFthAzcSpSp26vrUQtPbg8jpf/ffBc Qk/E/9cpF4gFla4nNiws8bppjjWo2jZAk37N9CyRK7i2rF56VCnRJx8XtwvPMtBFUGccisFW1Bl nqpRY3wkH1w2PTDM2EONIqkDOUK/z0KdjaMzfurAMMiNfkwaJRMVIQ5xO6n55mQ== X-Gm-Gg: AfdE7cltfoI7MFZ4I9xPrD2CUwnINNBMNyXHNZVGKs7HgpYC1lsVEQvJbr8sSnrhJe7 eim9hYUiUzmWOIh6qFzKzDuA5a0+ufoJv/+WwYGwUuXMQfothVn0PwQdeaWOKMyLZ3ISBRUWRFw MubPx9zD1NiiWePiPNHtdlch1wHTM/aPtvwpfTsgs/7zm6UBYm+cEQCl1KI+BM7yOBRTUY7TInQ mFSANPUobaM7Ou592unAPlySAl0PuYebdJtA2aP+P4UY4+2y9XVPi1xU3l9AQfoHgbg0JUSE8ey qS2q9XncS+MM4spktD0jK2GRnN6GlIQZVpTQFh/qpIKmaTo5UpSsxmI7LfiSbn/QYQHyUbzbEXE L271bkbwPfkRJIpnKOfvtIEiIoliy X-Received: by 2002:ac8:5703:0:b0:516:ed2c:9c75 with SMTP id d75a77b69052e-51a51a9be7bmr4274291cf.16.1782146727917; Mon, 22 Jun 2026 09:45:27 -0700 (PDT) X-Received: by 2002:ac8:5703:0:b0:516:ed2c:9c75 with SMTP id d75a77b69052e-51a51a9be7bmr4273601cf.16.1782146727437; Mon, 22 Jun 2026 09:45:27 -0700 (PDT) Received: from [10.206.129.146] ([129.236.229.236]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-51a515ccdbbsm2022461cf.11.2026.06.22.09.45.25 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Mon, 22 Jun 2026 09:45:26 -0700 (PDT) Message-ID: Date: Mon, 22 Jun 2026 12:45:25 -0400 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [PATCH v6 1/4] block: add task-context bio completion infrastructure To: Jan Kara Cc: Christoph Hellwig , Jens Axboe , "Matthew Wilcox (Oracle)" , Christian Brauner , "Darrick J. Wong" , Carlos Maiolino , Alexander Viro , Dave Chinner , Bart Van Assche , linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-xfs@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, Gao Xiang References: <20260514-blk-dontcache-v6-0-782e2fa7477b@columbia.edu> <20260514-blk-dontcache-v6-1-782e2fa7477b@columbia.edu> <80cb53f3-e7e5-4f96-bacc-f9fb7661d976@columbia.edu> <4jtsjd2sbsn2w7fzfwb7wwowz72r4kc6345ckkkcjoxjbbwjwn@rkl44x3o3sgy> Content-Language: en-US From: Tal Zussman In-Reply-To: <4jtsjd2sbsn2w7fzfwb7wwowz72r4kc6345ckkkcjoxjbbwjwn@rkl44x3o3sgy> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Proofpoint-GUID: gFhCe3dS5fUpZao3E9-Rb24riU-BL49O X-Proofpoint-Spam-Details-Enc: AW1haW4tMjYwNjIyMDE2NCBTYWx0ZWRfX5Wr3IMVN2e04 ni9W8OG7njNmQD2/lMnrTQepq1NGxjqm8UU9iezplmZy5ScnNqju7zKahyOUdJps5898aIqfxNu NXNTDUIwKMbmYA2ZPfO1QiLg8gtlokYFfBedYHmFyZsxTFDbwgao5pAL2DMFlMIP04iyQDAqtBy reg4RtIAYTEQkkfq9M50QQBPSdOYDFfG4Out3D+cYM0s5PBaW1Xnjz47DP4nnd+Dm0O9+T0INoB GhC1VbjYCYboaeYvMpESYKhpQc2Rd0iVLOuNarDGlSEElcYmKcPlfPt2hTCUdhll4A3n9z5URZ5 J1saLh4PnJCFw05Qo20Ap+i0oPRimkFb+1mBxWXjm6f2fsDDofoon2xZwAObIZAN9zu6v5YIKnw 1MJ1EtkDBcDvgIs922H+GQxuixkyaqpj4kuF6KGYwYtZt/ff/xkxuubnCrY/1FCEgQ2fKIgUkOL MQZMvBnjlk+Bp9DCgsQ== X-Authority-Analysis: v=2.4 cv=E9H9Y6dl c=1 sm=1 tr=0 ts=6a3966a9 cx=c_pps a=mPf7EqFMSY9/WdsSgAYMbA==:117 a=Ne+/oOzQhjyqTAlfBCchDA==:17 a=IkcTkHD0fZMA:10 a=FelO9ux0wxsA:10 a=x7bEGLp0ZPQA:10 a=VkNPw1HP01LnGYTKEx00:22 a=Da8U98TiO7q1upZEImrf:22 a=svvvyxlR1OQQkelhaPoB:22 a=xMg8OGkpyC76Z8KfPXcA:9 a=3ZKOabzyN94A:10 a=QEXdDO2ut3YA:10 a=dawVfQjAaf238kedN5IG:22 X-Proofpoint-ORIG-GUID: gFhCe3dS5fUpZao3E9-Rb24riU-BL49O X-Proofpoint-Spam-Info: AW1haW4tMjYwNjIyMDE2NCBTYWx0ZWRfX2+XNrN4shulT Df9fG1M46q2lcczEJ7TfDPWpVihFjktl5F7bggCiYceHpxwhSdiCWxcfkUQzbe7TVKd4lZ+VyD0 eMfwN7EbZTR79BPC1vbF9UnaVXhRXMd5aZ0CSfBTO1UrdCRmNGIZ X-Proofpoint-Virus-Version: vendor=nai engine=6900 definitions=11825 signatures=596817 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 priorityscore=1501 lowpriorityscore=10 bulkscore=10 malwarescore=0 adultscore=0 phishscore=0 suspectscore=0 spamscore=0 clxscore=1015 impostorscore=10 classifier=typeunknown authscore=0 authtc= authcc= route=outbound adjust=0 reason=mlx scancount=1 engine=8.22.0-2606150000 definitions=main-2606220164 On 6/18/26 10:26 AM, Jan Kara wrote: > On Mon 01-06-26 13:04:41, Jan Kara wrote: >> On Fri 29-05-26 16:46:15, Tal Zussman wrote: >> > On 5/27/26 9:00 AM, Christoph Hellwig wrote: >> > > On Wed, May 27, 2026 at 11:42:28AM +0200, Jan Kara wrote: >> > >> > I ran some experiments with fio on both XFS and a raw block device. Five >> > >> > iterations each for 60s. Results below. >> > >> > >> > >> > TLDR: Removing the delay doesn't significantly decrease user-visible >> > >> > latency or otherwise improve performance, but does significantly reduce >> > >> > throughput and increase context switches in some workloads (e.g. C). >> > >> > I think it makes sense to leave the delay as-is. Thoughts? >> > >> >> > >> Thanks for the test! One question below: >> > > >> > > Thanks from me as well! >> > > >> > >> >> > >> > Results: >> > >> > >> > >> > Workloads (all `uncached=1`): >> > >> > A: rw=write bs=128k iodepth=1 ioengine=pvsync2 # XFS >> > >> > B: rw=write bs=128k iodepth=128 ioengine=io_uring # XFS >> > >> > C: rw=randwrite bs=4k iodepth=32 ioengine=io_uring # XFS >> > >> > D: rw=rw 50/50 bs=64k iodepth=32 ioengine=io_uring # XFS >> > >> > E: rw=write bs=128k iodepth=128 ioengine=io_uring # raw /dev/nvmeXn1 >> > >> > F: rw=write bs=128k iodepth=128 numjobs=4 >> > >> > + vm.dirty_bytes=64MB, vm.dirty_background_bytes=32MB # XFS >> > >> > >> > >> > Mean ± stddev across 5 iterations: >> > >> > >> > >> > metric delay=1 delay=0 delta >> > >> > -------------------------------------------------------------- >> > >> > >> > >> > A seq 128k qd1 >> > >> > BW (MB/s) 4333 ± 27 4374 ± 34 +0.9% >> > >> > p99 (us) 36.2 ± 0.8 35.8 ± 0.4 -1.1% >> > >> > p999 (us) 3260 ± 75 3228 ± 29 -1.0% >> > >> > ctx-switches 184 k ± 59 k 3.68 M ± 65 k +1903% >> > >> > cs / io 0.09 ± 0.03 1.86 ± 0.03 +1888% >> > >> > avg bios/run 80.4 ± 0.6 1.1 ± 0.0 -98.7% >> > >> >> > >> So 1 jiffie delay is (with default HZ=1000) 1ms. That means for this load >> > >> the completion latency should be at least 1000us but your results show p99 >> > >> latency of 36. What am I missing? >> > > >> > > Yes, this looks a bit odd. Unless there's multiple threads submitting >> > > and somehow the completions get batched this should complete one >> > > bio at a time and be the worst case for the delay scheme. >> > >> > Sorry, I should've clarified - the latency here is the userspace-visible >> > I/O completion latency (i.e. fio's clat value). >> > >> > I ran again and traced to get the actual time from __bio_complete_in_task() >> > to calling ->bi_end_io(). The results match the 1 jiffie delay now: >> > >> > metric delay=1 delay=0 >> > >> > A seq 128k qd1 >> > fio clat p99 38us 36us >> > bio cb p50 1.23ms 2.5us >> > bio cb p99 4.13ms 1.44ms >> > bio cb p999 5.01ms 2.63ms >> >> So I'm clearly missing something fundamental as I don't see how can fio >> reported IO completion time be lower than the end_io callback latency... >> Ahh, it is the strange meaning of clat in fio in combination with sync >> engine where clat means: "how long after the syscall has returned the data >> is ready". Which for sync engine is immediately so the clat number is >> meaningless. I think reporting 'lat' numbers from fio would make more >> sense but whatever. >> >> The bio cb latency indeed looks like what I'd roughly expect now. And >> notice how the median latency of IO completion is 1.23ms in delay=1 case >> and your throughput isn't abbysmal only because writes end up accumulating >> in the page cache and writeback infrastructure ends up submitting a lot of >> writeback IOs in parallel (you have ~80 bios to complete per run which >> amortizes the latency to decent level). >> >> However if you'd have IO that were to use BIO_COMPLETE_IN_TASK >> infrastructure which doesn't have so many IOs in flight (like direct IO >> with lower queue depth which has to do extent conversion on completion), >> you would very much see the latency hit on your throughput as well. In the >> extreme case of qd=1 direct IO you'd reduce the throughput to ~4MB/s. >> >> Now I'm not saying the delay is bad - it is a tradeoff with clear wins in >> CPU overhead your benchmarks are showing. I just wanted to point out >> there's also the cost side which your benchmarks don't show very clearly. >> So we might need to keep some stats showing how many IO completions we are >> offloading per second on each CPU and switch to delaying the work only once >> it crosses a threshold like 1000000/HZ per second or so (so we at most >> double the IO latency by delaying the end_io callback). > > Any progress here? The patchset looks really promising so I'd love to have > it completed :) > Sorry for the delay - got caught up with some other work and had to set this aside for a couple weeks, but haven't forgotten about this. Planning to pick it back up some time this week. Thanks, Tal