From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mx0a-0031df01.pphosted.com (mx0a-0031df01.pphosted.com [205.220.168.131]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1A9023A1A2D for ; Fri, 10 Jul 2026 06:38:27 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=205.220.168.131 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1783665509; cv=none; b=dd94J1SqTbP7TKL2rLvFNb9cRqoRQVqb+OebHrKrph7YBscYmSfT4SCMFRNz+VH68u4zo46sW/gqUuoxz3w2i0ScZCiHn7ttSNtwdNwtiKI2n7fmqFrMuo2Qlw4bUaa4aGc1QVwmPD9A8669nD37ytTUP/BcFL0cTQEF0VueNt8= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1783665509; c=relaxed/simple; bh=VKRIt/7yRmxAz+S9gKURZ4WyeFPoIUvlkDUS5dWUi6k=; h=Message-ID:Date:MIME-Version:Subject:To:Cc:References:From: In-Reply-To:Content-Type; b=YSBd5SPaEZEVZEdM9AIH4NnOlNZJOBPWZ7oVhxRDdhFbcmPTMberXrhyxYK/yK2t6A/p7PVzSw/mSeUKXBK96YNP1ZwCiuF3Sb1MqJiWv3E27wA725aMUlBB1U4dzWOiTA6H2Eb6jjOhyqQjVIi8EFXTUDfA8TQnai8I+USkHus= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=oss.qualcomm.com; spf=pass smtp.mailfrom=oss.qualcomm.com; dkim=pass (2048-bit key) header.d=qualcomm.com header.i=@qualcomm.com header.b=An11jyea; dkim=pass (2048-bit key) header.d=oss.qualcomm.com header.i=@oss.qualcomm.com header.b=jtWjAIvc; arc=none smtp.client-ip=205.220.168.131 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=oss.qualcomm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=oss.qualcomm.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=qualcomm.com header.i=@qualcomm.com header.b="An11jyea"; dkim=pass (2048-bit key) header.d=oss.qualcomm.com header.i=@oss.qualcomm.com header.b="jtWjAIvc" Received: from pps.filterd (m0279867.ppops.net [127.0.0.1]) by mx0a-0031df01.pphosted.com (8.18.1.11/8.18.1.11) with ESMTP id 66A3mWFa3838001 for ; Fri, 10 Jul 2026 06:38:27 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=qualcomm.com; h= cc:content-transfer-encoding:content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to; s=qcppdkim1; bh= 0K5Opuo8W6vFKvh+gdX8BbeA6S2QAhhORoWXkYpGKec=; b=An11jyea+IvpFq77 RvyfijqFOM1GQc75OTNGUf2BHZXCyUWplzVX5aEvM6EHLptMMrsfNu1mvMo+rTY7 C1BFzMsCv8ds8Ff5vzjn7zHge+GVICsoYya1/rxoJQz0M3SCX00lLk+zlaaHif0+ PjouPLZY81+I0wilKDTy00akcR2TdgU8sBZkqcAFlyPxeAtcAxtzRhjNC5yWIvSA VvysduvlLdkC5Vm13vnzGgSmW4Cv+TRQeObmcimleXRkQP7/w7KbvjIYUHwX1BgV oG3Y0be2lbXtqMawfhrZQXVIiGKD+OsIfyrNL9jWIA9LD795SI2BU1jmXSUkzheW uKiLKg== Received: from mail-pg1-f197.google.com (mail-pg1-f197.google.com [209.85.215.197]) by mx0a-0031df01.pphosted.com (PPS) with ESMTPS id 4fanwe96ar-1 (version=TLSv1.3 cipher=TLS_AES_128_GCM_SHA256 bits=128 verify=NOT) for ; Fri, 10 Jul 2026 06:38:27 +0000 (GMT) Received: by mail-pg1-f197.google.com with SMTP id 41be03b00d2f7-c8895156101so1029359a12.1 for ; Thu, 09 Jul 2026 23:38:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oss.qualcomm.com; s=google; t=1783665506; x=1784270306; darn=vger.kernel.org; h=content-transfer-encoding:content-type:in-reply-to:from :content-language:references:cc:to:subject:user-agent:mime-version :date:message-id:from:to:cc:subject:date:message-id:reply-to :content-type; bh=0K5Opuo8W6vFKvh+gdX8BbeA6S2QAhhORoWXkYpGKec=; b=jtWjAIvcphDWd2aXpGfIE4L3RMtBaOz6Wly+HEMcq7N1etPSDpIuwVRmRvXADoFLoA H7KiVMId6AQWOdslDBIKCl+/NR3Y27+2f4hN+5p5Fxo4oDy7Ao5y/F2y2D3wzNu+lI65 VG7GZ5wugfPm+XTTWvjMsI5r/DuVR+Ho7f/Bg8PW/JrH2qdkqYRE9Zb5BTic/LcxPh8E waB8sNK8sujzp+JPXM6atbai0OcIOLAcLls3JLn6PNlbwpOAsuYNRc8kIh1Ql2g6Hw8y VKG+n89/3qq0e50h4OJMGBjz2jvLE049zKOsGwTIijFVwjefVMKr5S50XDXkOM5yUjEH P27Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1783665506; x=1784270306; h=content-transfer-encoding:content-type:in-reply-to:from :content-language:references:cc:to:subject:user-agent:mime-version :date:message-id:x-gm-gg:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to:content-type; bh=0K5Opuo8W6vFKvh+gdX8BbeA6S2QAhhORoWXkYpGKec=; b=BQWx6jPs4Xqr6ZAJdDHizqwH3hwuHwEK6cwky4Sr9xtiOpfMk0dL7ZS88pNlzXMds4 iXW2rk0ScKMvyZ5Z7XQLxC/flFaYb0Hv5AvmfnXHza0vBmiRL2J4CG01AZgLa+BY2aAm 3sgx4l4LymOXH1/5y9PZJYtLHdt5vngQiW8+f99qWRTCLMT7hFHbfS48BzUzf1WifYOL tR7wlB9vnvFzXZlHs348l6Q7cTTEiopEJn3984rKQ6ZVhOT6ckWAGuy1Mn5UCrm89iSe V/EF6VfW5GV4UMDto+46L7jw2c2nkKOZidC+DXF3CgcYWG91aktobGUkgJ4MbesVDE90 ivfg== X-Forwarded-Encrypted: i=1; AHgh+RoeLnKU+5P/5jDBb5oVt2FVchIOeSjUnQCrqepf9VGh5/1GwhyBkGRQNKVnFsIBhhiC0Xk3TYPsmc0Y0no=@vger.kernel.org X-Gm-Message-State: AOJu0Yw1DpHDeljwAYz9nVYE7GpUBzZo1G5TZXdCeZCQpHbLrHanl4+e PXaNI3qvs4abYIACm9DUfnZF1vwaUbi1irD/KhBZUdpla9YZ2v1kyPy3iDCDkmw9xz2bJBK85E/ T+6E+m+S8uN4qO/Rw7JltuiyGTCTSKyJdNzh3WBXhgGFdgeUOoeeEA+UIcGU2YA59JTw= X-Gm-Gg: AfdE7cnC/F6pNDBtrTDDZ579faaZHsocTxfI9LWWxCThj6GMNlRRpRmUoQGR49BuBOT oQstXeZJ+WCBxBzkgHkOTKI2RU0wrm+76ksUOgk26CBP/wKzCzytV2U3d2qjkSxpAsBe8Uyp+dg Ce4mreIpwTwiIxt0SHg2YRX8MzBpUM6F50EBk98d8rWt/eadk1EswsWJhUtMN0xZVzh7ckfFSRf yCS1sRnJkeeHfjsKEwiiKneA960JI2Y/XdlO3h188bzxM7WJIWrj7ixNSayZCtfyHLAeG6wDU8d MAmHngTGw3GOdK8bOQRbLA3+C7pKKQdU9XejIKssrGNQDqZdnj3Z6N9yLA4l6FWGUMbMpsQdC5a cxsg9NF6XsPZO6KyKbjqQXRzI+G8u3k+qeBNVnFBJmyUgoSM0TU/ywD0P4SLafzoLPoCqlJtv2n zfpQOIvKLbEd8tE/gfhuAlOlIFa+o6 X-Received: by 2002:a17:903:32c9:b0:2ca:e9dc:7357 with SMTP id d9443c01a7336-2ccea2d8029mr109938095ad.9.1783665506279; Thu, 09 Jul 2026 23:38:26 -0700 (PDT) X-Received: by 2002:a17:903:32c9:b0:2ca:e9dc:7357 with SMTP id d9443c01a7336-2ccea2d8029mr109937745ad.9.1783665505631; Thu, 09 Jul 2026 23:38:25 -0700 (PDT) Received: from [10.193.24.221] (blr-bdr-fw-01_GlobalNAT_AllZones-Outside.qualcomm.com. [103.229.18.19]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-2ccc9bfb7aesm54996575ad.29.2026.07.09.23.38.18 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Thu, 09 Jul 2026 23:38:25 -0700 (PDT) Message-ID: <0d1398f3-c2d5-4717-adf6-bdbc6055606b@oss.qualcomm.com> Date: Fri, 10 Jul 2026 12:08:17 +0530 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [RFC PATCH] x86/pci-dma: add "any" keyword to swiotlb= kernel parameter To: Robin Murphy , m.szyprowski@samsung.com Cc: corbet@lwn.net, skhan@linuxfoundation.org, akpm@linux-foundation.org, bp@alien8.de, rdunlap@infradead.org, peterz@infradead.org, feng.tang@linux.alibaba.com, dapeng1.mi@linux.intel.com, elver@google.com, enelsonmoore@gmail.com, kuba@kernel.org, lirongqing@baidu.com, ebiggers@kernel.org, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, aneesh.kumar@kernel.org, alexey.kardashevskiy@amd.com, thomas.lendacky@amd.com, jeff.hugo@oss.qualcomm.com, thanson@qti.qualcomm.com References: <20260708114244.246176-1-aakarsh.jain@oss.qualcomm.com> <7c21e121-9bce-469e-a983-adbdf3781842@arm.com> Content-Language: en-US From: Aakarsh Jain In-Reply-To: <7c21e121-9bce-469e-a983-adbdf3781842@arm.com> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Authority-Analysis: v=2.4 cv=LbIMLDfi c=1 sm=1 tr=0 ts=6a509363 cx=c_pps a=rz3CxIlbcmazkYymdCej/Q==:117 a=Ou0eQOY4+eZoSc0qltEV5Q==:17 a=IkcTkHD0fZMA:10 a=RAioF0-LDSMA:10 a=s4-Qcg_JpJYA:10 a=VkNPw1HP01LnGYTKEx00:22 a=u7WPNUs3qKkmUXheDGA7:22 a=eoimf2acIAo5FJnRuUoq:22 a=VwQbUJbxAAAA:8 a=QyXUC8HyAAAA:8 a=zd2uoN0lAAAA:8 a=EUspDBNiAAAA:8 a=LWuNMJU_oDth5YTcOp8A:9 a=3ZKOabzyN94A:10 a=QEXdDO2ut3YA:10 a=bFCP_H2QrGi7Okbo017w:22 X-Proofpoint-Spam-Details-Enc: AW1haW4tMjYwNzEwMDA2MiBTYWx0ZWRfX/1D42T9alR8Q zHuM7/28VBIe5/sagdLy+seD+yffsRwIslfj3kEy4k5126lTbRJzJgv/qdwOcmDp8c1Z/pEb9AS IGJEyG3gZiGa0t2pdP3Id92nymRS3GsmnXJ7/xgJfIsC8FddqtxNzQsK6rQS64GYrFeyJgc8EvF JGU34f8xg0ppIJ8c62yDYOjNDNV01elvbxkOhF0+LSRV3026dKrbpm7Feaz+oPOwlQix+WIRrrU Gp3QShmwmkJY28/VeMoHez37sUXdwUJw9oe/J/Epcj7Tfrg80d6AaChhELmNuhaDsAca66wyt0Y Y0zBCwyHGW8WVeEoaF7zc2vnsHoNE+bm8i3KrhSVxvzBoDcXM0f9zVs8bI2XaxYFfoqW2F3Ba71 b8BV5QeSUd8rsYNItykDajVbHnVSuILtPt4LXzAb6P35e24h9KbRe/M3C0vJM9EliZ6mPwx7Xti ELLUqggP4MJ7bFM4y4g== X-Proofpoint-ORIG-GUID: wmv7OD5JcyYf4mGfDurbg47u4xtbaYPI X-Proofpoint-Spam-Info: AW1haW4tMjYwNzEwMDA2MiBTYWx0ZWRfX0THG7rh+BoK/ WxNZcakS9o/6QQTkoOehfo3RCXIZ+QFmfDIbcnkSSgv8NStOlJIHJmwcz/d4xE+Kd2da07FUx6k bfcqkUf2Fw8G1nL1i+U4AN+NsqVmLzk= X-Proofpoint-GUID: wmv7OD5JcyYf4mGfDurbg47u4xtbaYPI X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1143,Hydra:6.1.134,FMLib:17.12.100.49 definitions=2026-07-10_01,2026-07-09_04,2025-10-01_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 malwarescore=0 spamscore=0 suspectscore=0 lowpriorityscore=0 bulkscore=0 clxscore=1015 impostorscore=0 priorityscore=1501 phishscore=0 adultscore=0 classifier=typeunknown authscore=0 authtc= authcc= route=outbound adjust=0 reason=mlx scancount=1 engine=8.22.0-2606150000 definitions=main-2607100062 Thanks Robin, Marek for the review. On 7/9/2026 5:58 PM, Robin Murphy wrote: > On 08/07/2026 12:42 pm, Aakarsh Jain wrote: >> CoCo guests (AMD SEV-SNP, Intel TDX) require large swiotlb pools for >> streaming DMA workloads such as high-speed NIC and AI accelerator >> inference. The existing swiotlb pool allocator restricts placement to >> low memory (below 4GB by default), capping usable pool size at ~1GB even >> when a larger pool is requested via swiotlb=. >> >> The SWIOTLB_ANY flag already exists to lift this restriction, and >> swiotlb_init_remap() already handles it correctly via the flags >> parameter (see CONFIG_SWIOTLB_DYNAMIC path: io_tlb_default_mem.phys_limit >> is set to virt_to_phys(high_memory-1) when SWIOTLB_ANY is set). >> >> However, there is no way to set SWIOTLB_ANY from the command line. The >> only existing mechanism was via arch-specific code (e.g. powerpc SVM sets >> SWIOTLB_ANY in pci_iommu_init). x86 CoCo guests have no such path. >> >> After Aneesh  series ("dma-mapping: Track shared DMA state through >> direct, pool and swiotlb paths", https://patchwork.kernel.org/project/ >> linux-arm-kernel/cover/20260701054926.825925-1-aneesh.kumar@kernel.org/) >> removes SWIOTLB_FORCE, x86 pci_swiotlb_detect() leaves >> x86_swiotlb_flags = 0 for >> CoCo guests. The pool falls back to low memory and caps at ~1GB: > > This is entirely irrelevant; SWIOTLB_FORCE has no impact on allocation > behaviour anyway. x86 wasn't passing SWIOTLB_ANY before and it still > isn't, although there doesn't seem to be any particular reason why the > well-reviewed patch for that hasn't been picked up: > > https://lore.kernel.org/lkml/20260625012616.2992535-1-jun.miao@intel.com/ > You are right my commit message incorrectly tied SWIOTLB_FORCE removal to pool placement. SWIOTLB_FORCE only controls whether bounce buffering is forced at runtime, it has no bearing on where the pool is allocated. Placement is governed solely by SWIOTLB_ANY. I'll drop that paragraph entirely. I see both you and Marek are pointing to this patch, which fixes the issue architecturally. I'm happy to help get that landed instead of adding a user facing knob. My only hesitation is the 32-bit concern below: auto-enabling SWIOTLB_ANY for *all* CoCo guests carries the same breakage risk for limited-mask devices, which was flagged here: https://lore.kernel.org/lkml/0a08cea7-62a2-4b33-8d8b-dd14b3c74235@amd.com/ >>    Without "any": pool at 0x35a9c000 (~900MB, below 4GB boundary) >>    With    "any": pool at 0x1df9c00000 (~120GB, anywhere in RAM) >>    [Tested on AMD SEV-SNP guest, swiotlb=4194304] >> >> Add "any" as a new keyword to the swiotlb= kernel parameter. This is an >> explicit, opt-in mechanism that sets SWIOTLB_ANY for the default pool at >> boot time, without touching any arch-specific code. >> >> Devices with 32-bit DMA masks are not affected, they still use the normal >> low-memory bounce buffer path. The "any" option is only meaningful for >> workloads where all active DMA devices have 64-bit masks. > > That doesn't make any sense - if a user passes this option then any > devices with DMA addressing limitations definitely *are* going to be > affected, and quite likely broken altogether. If anything, the > documentation should be even more explicit that this should only be used > if you do know for sure that no devices have DMA addressing limitations. > You are correct here too. My statement that "32-bit devices are not affected" is wrong. There is a single default pool, once it moves above 4GB via "any", a device with a <64-bit DMA mask has no reachable bounce buffer and will break. The opt-in nature only means the admin accepted that risk, it does not make the risk go away. If this option survives at all, the documentation must state the opposite of what I wrote: use it ONLY when you are certain no device in the system has DMA addressing limitations. > As a general SWIOTLB-behaviour-debugging option for orthogonality with > force/noforce I'm not opposed to the idea, but it is definitely not > something that real CoCo use-cases should rely on - if the general > consensus if that CoCo environments want a different setup by default > then the arch/CoCo code should be taking care of that. > So my takeaway, this patch, if kept, is at best a debugging knob for orthogonality with force/noforce, and not something real CoCo deployments should rely on. I'm fine positioning it that way, or dropping it. Which brings me to the actual motivation behind posting this patch. what should be recommended approach for large streaming DMA workloads in CoCo guests(high-speed NIC / AI accelerator inference)? Is the intended direction to move such drivers to the coherent DMA path rather than growing or relocating the SWIOTLB pool? Thanks, Aakarsh > Thanks, > Robin. > >> Signed-off-by: Aakarsh Jain >> --- >>   Documentation/admin-guide/kernel-parameters.txt | 5 ++++- >>   kernel/dma/swiotlb.c                            | 5 +++++ >>   2 files changed, 9 insertions(+), 1 deletion(-) >> >> diff --git a/Documentation/admin-guide/kernel-parameters.txt b/ >> Documentation/admin-guide/kernel-parameters.txt >> index b5493a7f8f22..8a1fccbd9b25 100644 >> --- a/Documentation/admin-guide/kernel-parameters.txt >> +++ b/Documentation/admin-guide/kernel-parameters.txt >> @@ -7477,7 +7477,7 @@ Kernel parameters >>               Execution Facility on pSeries. >>       swiotlb=    [ARM,PPC,MIPS,X86,S390,EARLY] >> -            Format: { [,] | force | noforce } >> +            Format: { [,] | force | noforce | any} >>               -- Number of I/O TLB slabs >>               -- Second integer after comma. Number of swiotlb >>                    areas with their own lock. Will be rounded up >> @@ -7485,6 +7485,9 @@ Kernel parameters >>               force -- force using of bounce buffers even if they >>                        wouldn't be automatically used by the kernel >>               noforce -- Never use bounce buffers (for debugging) >> +            any --  Allow the swiotlb pool to be placed anywhere in >> +                system RAM, lifting the default low-memory (4GB) >> +                restriction. >>       switches=    [HW,M68k,EARLY] >> diff --git a/kernel/dma/swiotlb.c b/kernel/dma/swiotlb.c >> index 1abd3e6146f4..34773ae7c770 100644 >> --- a/kernel/dma/swiotlb.c >> +++ b/kernel/dma/swiotlb.c >> @@ -80,6 +80,7 @@ struct io_tlb_slot { >>   static bool swiotlb_force_bounce; >>   static bool swiotlb_force_disable; >> +static unsigned int swiotlb_param_flags __initdata; >>   #ifdef CONFIG_SWIOTLB_DYNAMIC >> @@ -198,6 +199,8 @@ setup_io_tlb_npages(char *str) >>           swiotlb_force_bounce = true; >>       else if (!strcmp(str, "noforce")) >>           swiotlb_force_disable = true; >> +    else if (!strcmp(str, "any")) >> +        swiotlb_param_flags |= SWIOTLB_ANY; >>       return 0; >>   } >> @@ -445,6 +448,8 @@ int swiotlb_init_late(size_t size, gfp_t gfp_mask, >>       io_tlb_default_mem.force_bounce = swiotlb_force_bounce; >> +    flags |= swiotlb_param_flags; >> + >>   #ifdef CONFIG_SWIOTLB_DYNAMIC >>       if (!remap) >>           io_tlb_default_mem.can_grow = true; >