From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-qt1-f195.google.com (mail-qt1-f195.google.com [209.85.160.195]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 90E7B318EDD for ; Wed, 7 Jan 2026 20:46:09 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.160.195 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1767818772; cv=none; b=LToRghBusxauSGGmbEI/5XsXHz4fnnHDCwH0NM/vwi3hvVq7a5QPGXZ5HqsmYqO4GiN/napwW09fNCtRs2jrXKpIZiQkzyY6I1bJuNmyhJ+YAgCz+jrLKlNEd0+5uqeAlmknij8Qwz08E8D8L8wwiBKYI57txIbdPEi+WcHQ3uc= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1767818772; c=relaxed/simple; bh=V1GVCS9K1Qf//sa902zfVHBVWTNmYA6vlf7ykEXvGGI=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=X6l/gF8dhsrlqW3DjIK7Kqu7qH8yi7+su4pH6WA5XZMZYF4mwLFLBl7pTO27GbyQYgeDvuTtXBgTH+GygP7eVeBnlmsg1GRsN6NafUDCsv7NUqpTxlOOfALegli2efYv8TFUqCcsQaIb0+8FO0NWj+c6xvkccKts/bw/MjI6/XY= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=ziepe.ca; spf=pass smtp.mailfrom=ziepe.ca; dkim=pass (2048-bit key) header.d=ziepe.ca header.i=@ziepe.ca header.b=Z41Yc77O; arc=none smtp.client-ip=209.85.160.195 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=ziepe.ca Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=ziepe.ca Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=ziepe.ca header.i=@ziepe.ca header.b="Z41Yc77O" Received: by mail-qt1-f195.google.com with SMTP id d75a77b69052e-4ee1a3ef624so16184031cf.0 for ; Wed, 07 Jan 2026 12:46:09 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ziepe.ca; s=google; t=1767818768; x=1768423568; darn=vger.kernel.org; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=DqLGiR8rj/edjA3ZmthNk/CuW8dhDUcDoy+Z3OjEZpY=; b=Z41Yc77O+Z2GpUMXvb8qcnyXKdrVGjhmiLyBEvN6Ro4zCT3pbNG1JTMC3wEHjqPCjv Tt2Pps2M/sMIIWwiazQv9AM/oWh1R4fSkEoLqa/gfBmCZTBYDlkRXEcY3je8Wa4ogHx9 i6Q7o13i776LeHlq1VOqv60/8mtjJtBGIOf61G0UKBDP/p7bYl+wL0G7+fda7NWHKa/X 7teF33trMHadnQcm7ftDKHTLLevO1FbjGG4FcA6dMiHWFme3G5xnkcq2M43P5d5hvz1C 79GkQ+n4cmusVLI0my1NshDnZgBhGHIjr4sKztdUTYTwhhOR6jGtoSLNpnBIfH+mop53 hiRQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1767818768; x=1768423568; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-gg:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=DqLGiR8rj/edjA3ZmthNk/CuW8dhDUcDoy+Z3OjEZpY=; b=RelqIC1qKg4cuhwdx75VYTqh7mSHfJRpk4LAEEvF7BTryKeg9dKbikFfWeRw1FBuse Omlu3Lg7YkQp2Q/OKxoG4KrRBJwP3nBQ8z92ALL2eRKFcs0tdHv0K0k5/O4PCBKk3DJv qzkzLP6V9lZsBgMSx/hI+WE2FIGqjsNn9nyLzNHJyqwO5O5t7UbVIjzoNiGTnyRZHZMo Uh7ZwCzI5id9NTSAfBd92EJsrGynbt95NIuJm+3MpNihN99Nl04w16gt3623w2VwV/9H reUmga0IbIVxthnfD157Kg3a/RO4R/dowHlkSp7ql0K1aP4bMblKrB1ThGUQo6xLDsNZ vmmA== X-Forwarded-Encrypted: i=1; AJvYcCVkxfEcITtX7/ztNK+58Ccv72GU9nnQr4fhLuFuQ/rNmnOO91LHhb0LGb5TpIR3Tko8tPKP/713GIkYHYc=@vger.kernel.org X-Gm-Message-State: AOJu0YxVneTkOPTp5FqZCIBH+tK+gub+0uE+Xix87zpt9MQhgpxo3g8q Sjjq5Ar4+65XBpmBei8s/Jclmp3KshcnI/VCZn09EH97XbRhGYU7QHWk6mfZBTssh98= X-Gm-Gg: AY/fxX5gCP0T6OZiY3BCanQqg4l/Q6IY6eEjxwUXTYpU09Jv7om8uEYxultBGeG/8ki gjj1TcD4zvGT2g728Kt+bffEP2M3LvSS1oKPBE9W7TAHy+0+XGC+bSZr8d1LWlD7FrEUFiy1GTv NhnoSBbApgkKVYX02bJHTuywVuHjN0V3Ko0U4RAiOQrptKQLLgfIk110zhEb+f4zGOhLUz+4sbZ Ut2xNmGUNrAg6jbPEzNG5C/E1nsr4o+OeH+pE2ZKrOvJhK5M4cLzwrR/lwwUa5xRRUptKJVrvyB SljABWASKiHKuQKoshoYTFyQ6ou/qBNh2PAWb3wZghCnkhg12CXQu/dkzNxQ0a+ub4ieCDLGNgV 9b01jyGW+KqJuJl3Lw7UYSTCwZHvjCuvmBzi8b1UuIWk8TcHiIQZhly5oeYTOeWSO9tLTB05Uvs wwdVg0hPsyD5dt1K1g72C5bBdbKUMvQiWk5byx4sw1lXDRM2+25Exq5KYRo7RItiTz0ks= X-Google-Smtp-Source: AGHT+IGDmyd7kjxIp/jIm0kXF9tq0fAn0CQ3sKUlFYdgvJub9VEiYkmE5sOV0MLJ1rA3hSKtj1n7+A== X-Received: by 2002:a05:622a:155:b0:4f1:ba0b:90 with SMTP id d75a77b69052e-4ffb48bc30fmr42029251cf.8.1767818768314; Wed, 07 Jan 2026 12:46:08 -0800 (PST) Received: from ziepe.ca (hlfxns017vw-142-162-112-119.dhcp-dynamic.fibreop.ns.bellaliant.net. [142.162.112.119]) by smtp.gmail.com with ESMTPSA id 6a1803df08f44-89077234c96sm39240416d6.27.2026.01.07.12.46.07 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 07 Jan 2026 12:46:07 -0800 (PST) Received: from jgg by wakko with local (Exim 4.97) (envelope-from ) id 1vdaPv-00000002FiH-0jqm; Wed, 07 Jan 2026 16:46:07 -0400 Date: Wed, 7 Jan 2026 16:46:07 -0400 From: Jason Gunthorpe To: Samiullah Khawaja Cc: David Woodhouse , Lu Baolu , Joerg Roedel , Will Deacon , Pasha Tatashin , David Matlack , Robin Murphy , Pratyush Yadav , Kevin Tian , Alex Williamson , Shuah Khan , iommu@lists.linux.dev, linux-kernel@vger.kernel.org, kvm@vger.kernel.org, Saeed Mahameed , Adithya Jayachandran , Parav Pandit , Leon Romanovsky , William Tu Subject: Re: [PATCH 0/3] iommu/vt-d: Add support to hitless replace IOMMU domain Message-ID: <20260107204607.GE340082@ziepe.ca> References: <20260107201800.2486137-1-skhawaja@google.com> <20260107202812.GD340082@ziepe.ca> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20260107202812.GD340082@ziepe.ca> On Wed, Jan 07, 2026 at 04:28:12PM -0400, Jason Gunthorpe wrote: > On Wed, Jan 07, 2026 at 08:17:57PM +0000, Samiullah Khawaja wrote: > > Intel IOMMU Driver already supports replacing IOMMU domain hitlessly in > > scalable mode. > > It does? We were just talking about how it doesn't work because it > makes the PASID entry non-present while loading the new domain. If you tried your tests in scalable mode they are probably only working because the HW is holding the entry in cache while the CPU is completely mangling it: int intel_pasid_replace_first_level(struct intel_iommu *iommu, struct device *dev, phys_addr_t fsptptr, u32 pasid, u16 did, u16 old_did, int flags) { [..] *pte = new_pte; That just doesn't work for "replace", it isn't hitless unless the entry stays in the cache. Since your test effectively will hold the context entry in the cache while testing for "hitless" it doesn't really test if it is really working without races.. All of this needs to be reworked to always use the stack to build the entry, like the replace path does, and have a ARM-like algorithm to update the live memory in just the right order to guarentee the HW does not see a corrupted entry. It is a little bit tricky, but it should start with reworking everything to consistently use the stack to create the new entry and calling a centralized function to set the new entry to the live memory. This replace/not replace split should be purged completely. Some discussion is here https://lore.kernel.org/all/20260106142301.GS125261@ziepe.ca/ It also needs to be very careful that the invalidation is doing both the old and new context entry concurrently while it is being replaced. For instance the placement of cache_tag_assign_domain() looks wrong to me, it can't be *after* the HW has been programmed to use the new tags :\ I also didn't note where the currently active cache_tag is removed from the linked list during attach, is that another bug? In short, this needs alot of work to actually properly implement hitless replace the way ARM can. Fortunately I think it is mostly mechanical and should be fairly straightfoward. Refer to the ARM driver and try to structure vtd to have the same essential flow.. Jason