From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id A37C0C4167B for ; Mon, 4 Dec 2023 13:12:18 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233810AbjLDNMK (ORCPT ); Mon, 4 Dec 2023 08:12:10 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39514 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233512AbjLDNMI (ORCPT ); Mon, 4 Dec 2023 08:12:08 -0500 Received: from mail-oo1-xc2a.google.com (mail-oo1-xc2a.google.com [IPv6:2607:f8b0:4864:20::c2a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 68E33D8 for ; Mon, 4 Dec 2023 05:12:13 -0800 (PST) Received: by mail-oo1-xc2a.google.com with SMTP id 006d021491bc7-58d54612d9cso2957612eaf.1 for ; Mon, 04 Dec 2023 05:12:13 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ziepe.ca; s=google; t=1701695532; x=1702300332; darn=vger.kernel.org; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=7RWsKxXhxAMEDfXPpZR35TsYFyeNBhVQ2t6NvxM6Pec=; b=aJjsobrC4xLthF60nykm5M+H1I0/mqY2kQShWN/a+M5HbS/Z3fT6bB/TJQISmTi5zK GIHD5hIlWUw5ZwAIYKNpBXbq5qWb9JU/8cRyc23hqavr5D71t74NPpeCBKHJ7D2JyyZe ZpE8dmq6rVmDQjy7qCPrbqFNNgmXTZ7g98InqFexzeU0PfMQPjOvEGSgrqvFgQDe8z8p xJAqFS2OogpZINvaI3a1x49sCL1vT1mJDPhQfyua4wSfN/9dRDrsqT69XuZqhx0UvOGN +z56kOjo44eP0eRaXZUDZJu6NwnJisdJ1hT2FimB0CpbLips1hzYS+rFhFA9QtIBxdBw 4MXQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1701695532; x=1702300332; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=7RWsKxXhxAMEDfXPpZR35TsYFyeNBhVQ2t6NvxM6Pec=; b=U3sKAGd1sO/VZ6BON10KNSp+vSP6aUSzrpgIPD5+O7a7GPiOY7mvXBBYWftjtLHfWR SWMzn+47e+BBi13bugTPJckHwmWvqC56HHRtM9ACh7UjtOeBTZ6OzloGbwaqRVPpbk0l DHNx728YnCwNHodYSXjU2H3CQjKtyVznfjY3AEPTy+9Rgo+LhHoSqx/2UU2kN7wN2/vW FwX6OXP6EoQzC0onsqHf41fLmA6iVLsEooVn1ocbT/6LAFsE1T39UrUvvZYZOVMF0TyV 6rKhyzjeiyIpzqBE4GB/DA+8kd3BOZBWcQ8+5MzXqkIYXqfqvhQjENhLZVnXF5aaIjmU 4tiQ== X-Gm-Message-State: AOJu0Yz3mMZEPvaePnwdIycoMfd4TNNgjDzvGQlOkHfyVZ2NhhPg6PiI v+5941R6lZkMrZg1qN6C/QeFcQ== X-Google-Smtp-Source: AGHT+IGoQOzPPQKNgLswcbzAqC7rBuTYY1wEC/9v3DXJqgA/wrMtqAvrBw+cZGFjMr3uAfVfX68LDg== X-Received: by 2002:a05:6358:299:b0:170:2f86:392e with SMTP id w25-20020a056358029900b001702f86392emr1553337rwj.60.1701695532580; Mon, 04 Dec 2023 05:12:12 -0800 (PST) Received: from ziepe.ca (hlfxns017vw-142-134-23-187.dhcp-dynamic.fibreop.ns.bellaliant.net. [142.134.23.187]) by smtp.gmail.com with ESMTPSA id z21-20020ae9c115000000b0077d8ad77069sm4248899qki.26.2023.12.04.05.12.12 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 04 Dec 2023 05:12:12 -0800 (PST) Received: from jgg by wakko with local (Exim 4.95) (envelope-from ) id 1rA8k7-00ArNK-HA; Mon, 04 Dec 2023 09:12:11 -0400 Date: Mon, 4 Dec 2023 09:12:11 -0400 From: Jason Gunthorpe To: Baolu Lu Cc: Joerg Roedel , Will Deacon , Robin Murphy , Kevin Tian , Jean-Philippe Brucker , Nicolin Chen , Yi Liu , Jacob Pan , Yan Zhao , iommu@lists.linux.dev, kvm@vger.kernel.org, linux-kernel@vger.kernel.org Subject: Re: [PATCH v7 12/12] iommu: Improve iopf_queue_flush_dev() Message-ID: <20231204131211.GK1489931@ziepe.ca> References: <20231115030226.16700-1-baolu.lu@linux.intel.com> <20231115030226.16700-13-baolu.lu@linux.intel.com> <20231201203536.GG1489931@ziepe.ca> <20231203141414.GJ1489931@ziepe.ca> <2354dd69-0179-4689-bc35-f4bf4ea5a886@linux.intel.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <2354dd69-0179-4689-bc35-f4bf4ea5a886@linux.intel.com> Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Mon, Dec 04, 2023 at 09:32:37AM +0800, Baolu Lu wrote: > > > > I know that is why it does, but it doesn't explain at all why. > > > > > 1. Clears the pasid translation entry. Thus, all subsequent DMA > > > transactions (translation requests, translated requests or page > > > requests) targeting the iommu domain will be blocked. > > > > > > 2. Waits until all pending page requests for the device's PASID have > > > been reported to upper layers via the iommu_report_device_fault(). > > > However, this does not guarantee that all page requests have been > > > responded. > > > > > > 3. Free all partial page requests for this pasid since the page request > > > response is only needed for a complete request group. There's no > > > action required for the page requests which are not last of a request > > > group. > > > > But we expect the last to come eventually since everything should be > > grouped properly, so why bother doing this? > > > > Indeed if 2 worked, how is this even possible to have partials? > > Step 1 clears the pasid table entry, hence all subsequent page requests > are blocked (hardware auto-respond the request but not put it in the > queue). OK, that part makes sense, but it should be clearly documented that is why this stuff is going on with the partial list. "We have to clear the parial list as the new domain may not generate a SW visible LAST. If it does generate a SW visible last then we simply incompletely fault it and restart the device which will fix things on retry" > > Requests simply have to continue to be acked, it doesn't matter if > > they are acked against the wrong domain because the device will simply > > re-issue them.. > > Ah! I start to get your point now. > > Even a page fault response is postponed to a new address space, which > possibly be another address space or hardware blocking state, the > hardware just retries. > > As long as we flushes all caches (IOTLB and device TLB) during switching, > the mappings of the old domain won't leak. So it's safe to keep page > requests there. > > Do I get you correctly? Yes It seems much simpler to me than trying to make this synchronous and it is compatible with hitless replace of a PASID. The lifetime and locking rules are also far more understandable Jason