From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-qk1-f171.google.com (mail-qk1-f171.google.com [209.85.222.171]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C1DA62F5A05 for ; Thu, 12 Mar 2026 12:34:38 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.222.171 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1773318880; cv=none; b=RuNgECIIQjALWMaBXmdJ+yniDeFczfhJLbKmT5ePCGEj+P57gdTNqohaxGaCkbxc8On4CtuyEz9ENZHRTfass5zT9PTC/EmwxoTE2Hd8a+sLMbriKc1Ven5zLdHNJ7pGyAVgPiAdM2kgydRw+OD8WO/nDqkHf9KLLdKRiKzFxuo= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1773318880; c=relaxed/simple; bh=APBW+HdbDJGC0qi66ebGpgzi6a/5pJAPmLFyIpoPoUs=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=h4dZQFz3XRnrwxZwbTuF6HbMOeIwqyvyARh1DaYk7J4g23j7KqL90iziVea51af7XKb2xeJoLCD9OEJs3PVycwG2Hn8hYkUTrzzDzEnCZcV/SstIZ33AHhPkWBztFSz0nlEs5TQqTA+STMbKGUNj0+YCq4Vu0gILSbeptwRxvKY= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=ziepe.ca; spf=pass smtp.mailfrom=ziepe.ca; dkim=pass (2048-bit key) header.d=ziepe.ca header.i=@ziepe.ca header.b=ownQHuOT; arc=none smtp.client-ip=209.85.222.171 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=ziepe.ca Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=ziepe.ca Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=ziepe.ca header.i=@ziepe.ca header.b="ownQHuOT" Received: by mail-qk1-f171.google.com with SMTP id af79cd13be357-8cd858e860aso134835885a.2 for ; Thu, 12 Mar 2026 05:34:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ziepe.ca; s=google; t=1773318878; x=1773923678; darn=vger.kernel.org; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=ULg7grqreMOX2MMOX7z3yI5zs+73GrCTo1UP0N+hYkE=; b=ownQHuOTOoTG7P5CNB7JnKZ3YS6g/Kj+BPvHDxppydLUGu0P9FeCYCozpvrx+M0CvO 6zQiTL2JTFDUIfRCZ9f6CCGls7LKo13jawgowaQLhPAh/yZrtsyaWL2k7d7XXUH460n2 0Uxloj5c2y7Pb//ALmD0enI1ADfzsbNbpXJBzR+TpxF0z3V0pCXM/1wTaCy7HoJUxCTn uxAXPrvwy7I/0WQExRWhrFpLrCOjdwDbT2BnZitDU9ZthhtuIq+DOd+95U2kp/6iYjuD ut/cf4QpMNxMFY6pqgBQBfZXAuwC5O/8KD+P0aKFGwlu5yOwXjQ0xIYL9iJyV+JfI+g2 hYag== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1773318878; x=1773923678; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-gg:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=ULg7grqreMOX2MMOX7z3yI5zs+73GrCTo1UP0N+hYkE=; b=Y5g0HeDL4Uwsyzv0mhzRFdRe+AxIBHXpps8qP3LH/bQdfyq7XZN+amorYj0HYPUiFl eG+0KqQW4wTNYEgVDoT6h2/IJ48wyGn0KthC7vqoSk1SF9YtvaVLlF8YWu2hkGEdcqhc G5Vxm/DfGCK6Pu5aInf8w9dvyU3cmtmmniFNiYWGJ+JeqC6mAm1oBVc/e8xSS9ptUrLC Ku+4RlQBSWC3anq44hxx959tdmNCMzFTEQz/jE7XkOZa8WzT8zAtUuS4MRyvR/5ceyoI PULTgT5tDxS2TypL1eIbnTB2MBfzWKi5FLrIW+P07/M2aERDFy8i3XycqLW/kw1UVdrq 2Ftg== X-Forwarded-Encrypted: i=1; AJvYcCVuMfp89lzrXnUJck1pzBXCkG5QT4/p88dp2uy+PedP9j9Zc2GiADq1q/WM1NhiL1b+f8/ND60YQd2QGHk=@vger.kernel.org X-Gm-Message-State: AOJu0YwCXFj10ttj/qst++J38ZiKm+y21N2aupKJ9UZirkWLsfLGtSsN 1ms+1EryqB/wROMlr+rZ4vRkasHpONOzn1aWBHHzaEWIzrDIRhTY2zylCfbf3sEpDFg= X-Gm-Gg: ATEYQzwmV1OZisUmW6ixvOcgLHf97c7CiveY4e2Eanq8VVVCKucKXDSthDDNgFSGDea EFD7MjLoAYIEROolSNIVEUZTrmiz44m+MPZhLBD7aPjBZj8OsgHLIzUgoCTAKhDd08R6VPKAG7z rNI5BgW9tY8OzOvg7prTAt3hqeWfnRLhVN01P5VczuaHP/zo1eEO6pbLUIIElFP+IfK857kri04 ZEL17Ua8GrBTOThUS8x4RVgEPoKr22DPvkm8LOwub+j8Nc8b+q0RiqtIOXL5J+c+4nxb7ZmAOcT SH5whlDAw45ojHfpQp1fiwC7bhHc1j73GIFYl1ExQoYbmp2+znP/g4H1/D8mg3V0asVaKN3hpcD prvYRxan4moQBVHDCP34oXZtyrO9kIH00H0tDML1c2lDaXxcl2wn/GIqog3p81buepd9ApAz84N 7Xab147xdhUkO8zFfVYOEU7A0e/XU1YyXH+4C81HLNP+ARFbED1Vtf9L51dP8Zw1pr8XzrSj93i 3T2GcMG X-Received: by 2002:a05:620a:46a8:b0:8cd:8938:eff9 with SMTP id af79cd13be357-8cda1936a90mr737651285a.1.1773318877767; Thu, 12 Mar 2026 05:34:37 -0700 (PDT) Received: from ziepe.ca (hlfxns017vw-142-162-112-119.dhcp-dynamic.fibreop.ns.bellaliant.net. [142.162.112.119]) by smtp.gmail.com with ESMTPSA id af79cd13be357-8cda21346a0sm323887785a.34.2026.03.12.05.34.36 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 12 Mar 2026 05:34:36 -0700 (PDT) Received: from jgg by wakko with local (Exim 4.97) (envelope-from ) id 1w0fFL-00000006ezF-1zix; Thu, 12 Mar 2026 09:34:35 -0300 Date: Thu, 12 Mar 2026 09:34:35 -0300 From: Jason Gunthorpe To: Leon Romanovsky Cc: Marek Szyprowski , Robin Murphy , "Michael S. Tsirkin" , Petr Tesarik , Jonathan Corbet , Shuah Khan , Jason Wang , Xuan Zhuo , Eugenio =?utf-8?B?UMOpcmV6?= , Steven Rostedt , Masami Hiramatsu , Mathieu Desnoyers , Joerg Roedel , Will Deacon , Andrew Morton , iommu@lists.linux.dev, linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, virtualization@lists.linux.dev, linux-rdma@vger.kernel.org, linux-trace-kernel@vger.kernel.org, linux-mm@kvack.org Subject: Re: [PATCH v2 7/8] RDMA/umem: Tell DMA mapping that UMEM requires coherency Message-ID: <20260312123435.GH1469476@ziepe.ca> References: <20260311-dma-debug-overlap-v2-0-e00bc2ca346d@nvidia.com> <20260311-dma-debug-overlap-v2-7-e00bc2ca346d@nvidia.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20260311-dma-debug-overlap-v2-7-e00bc2ca346d@nvidia.com> On Wed, Mar 11, 2026 at 09:08:50PM +0200, Leon Romanovsky wrote: > From: Leon Romanovsky > > The RDMA subsystem exposes DMA regions through the verbs interface, which > assumes a coherent system. Use the DMA_ATTR_REQUIRE_COHERENCE attribute to > ensure coherency and avoid taking the SWIOTLB path. Lets elaborate a bit more so people understand why verbs is like this: The RDMA verbs programming model is like HMM and assumes concurrent DMA and CPU access to userspace memory in a process. The HW device and programming model has so-called "one-sided" operations which are initiated over the network by a remote CPU without notification or involvement of the local CPU. These include things like ATOMIC compare/swap, READ, and WRITE. Using these operations a remote CPU can traverse data structures, form locks, and so on without awareness of the host CPU. Having SWIOTLB substitute the memory or the DMA be cache incoherent completely breaks these use cases. RDMA in-kernel is OK with incoherence because none of the kernel use cases make use of one-sided operations that would cause problems. Jason