From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from DM5PR21CU001.outbound.protection.outlook.com (mail-centralusazon11011052.outbound.protection.outlook.com [52.101.62.52]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6B03C1D90AD for ; Tue, 7 Oct 2025 20:37:09 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=52.101.62.52 ARC-Seal:i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1759869433; cv=fail; b=nnW8ag5Bhq52ZimNXS9rN8BswvChxjaPyfcK3Ui9CMVlJr0oBr94rgjO0+r2uBM8+Y33c0vsP+QTW9EselcCK8I+TY9fHVEPmZz/jaX5MSCIqScNqfwMlwUkXUU72W9nwTg1i4dzdthh6/nZcZpvMO5dz8mOK+pRJLXulg3A57M= ARC-Message-Signature:i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1759869433; c=relaxed/simple; bh=1grdaqmN2VucA68m/29YHovqvakzQ5R9sHlGsDiktg0=; h=Message-ID:Date:Subject:To:Cc:References:From:In-Reply-To: Content-Type:MIME-Version; b=Z5QMN1iavWAIX4LlWA8bC4nWyNd9qvmsoksWqk9kWXs+zzLBbNG3l6Rlth2YXS/H3upWDPc7ZmzHQ+KMB+04dfbd8GSHeXBmJ6ixh6yfyaLkxGUkRVQHWO0LYXRqCNYUj9cvuMJ/dZ8zmGbpeuNNOWwMTBOjWp5KEdi4AxfHoFQ= ARC-Authentication-Results:i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=e0iISmFO; arc=fail smtp.client-ip=52.101.62.52 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="e0iISmFO" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=cO+wrNPDQHV3hGufeTkeUhFcAk5Y64Fga/wqAQ9mRjNGgS6VGmMQzWrZ5HBwdgEQGKqhMGQDq1NbFH0liU4uT86nHXKKqR32NfVSzNQkEaytHvTeXfbj/AJuzjIMuYW8CWOTSAkTfrcg+fbyEsQND5JV3QPgfpqSd9seyAFvXFPGicZl/otxCFs+69iop+AoQSa6Ff7tD0droJ5NY6coQFwr5r+mNQPcHFMsUW56zPKG2qWCERg0wJGj39fAjNd6qV7XaZUzPJP6wnSMYnC2e0KRrQwIPmCI+ocSW90+EUOOwFIUJrUVFJEbPemQ62B1SCocPTV/e7crwyVxQd/T1w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=P+FVIStw2xKo9gfnQrd3tvWoMJMHSGGsR7NdSSU7/Yk=; b=ZiG3saV6nI1yk48DVW7upnLWXUu09pZdRvKgecd7scsQyzBH4uzIzZJEpU/Yt4+mir1qRDvr1Sz629EelObXmS++FRiCazlqk1CV0oek5h9z2PutD7z+tYMqGnJsStqWYzkk25qJq/GshC75yjD3kzPkvK7687TwIlxte6qtNxcFb6g560NbSRv0l8gu6lG/2xgvc0H6DUMtrWz+3romqQGoY02k5LTiwJzjAvBIqfGyah2UbX6mugLBopS9R5dG2YWqu+t2ktzC/H3gC3KxrSMrROfIbh5YYOxSSitSY8ZLpDRRfX8JL9rkIB4TMe95ftUuxKuQ1lB7tjAEbBB6Jg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=amd.com; dmarc=pass action=none header.from=amd.com; dkim=pass header.d=amd.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=P+FVIStw2xKo9gfnQrd3tvWoMJMHSGGsR7NdSSU7/Yk=; b=e0iISmFOdQrcwhWC6xH1WJZFapF+YDc45vskakUzNrCgD56Qqzq3RqchJvoIYYu3fvi9qPF52/krBzS8kkwn6oTD5I6iFHX7WN6UaflHpKFh9fUlsFyp48gMa1FCk2HIp8g7powjGsYHcgvpLXF/LbMeBP4puJSRfOqR/jVjmuw= Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=amd.com; Received: from DM8PR12MB5445.namprd12.prod.outlook.com (2603:10b6:8:24::7) by SJ2PR12MB9212.namprd12.prod.outlook.com (2603:10b6:a03:563::19) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.9182.20; Tue, 7 Oct 2025 20:37:01 +0000 Received: from DM8PR12MB5445.namprd12.prod.outlook.com ([fe80::a544:caf8:b505:5db6]) by DM8PR12MB5445.namprd12.prod.outlook.com ([fe80::a544:caf8:b505:5db6%6]) with mapi id 15.20.9203.007; Tue, 7 Oct 2025 20:37:01 +0000 Message-ID: Date: Tue, 7 Oct 2025 15:36:58 -0500 User-Agent: Mozilla Thunderbird Subject: Re: [PATCH v2 10/12] iommu/amd: Add support for nested domain allocation Content-Language: en-US To: Jason Gunthorpe Cc: nicolinc@nvidia.com, linux-kernel@vger.kernel.org, robin.murphy@arm.com, will@kernel.org, joro@8bytes.org, kevin.tian@intel.com, jsnitsel@redhat.com, vasant.hegde@amd.com, iommu@lists.linux.dev, santosh.shukla@amd.com, sairaj.arunkodilkar@amd.com, jon.grimm@amd.com, prashanthpra@google.com, wvw@google.com, wnliu@google.com, gptran@google.com, kpsingh@google.com, joao.m.martins@oracle.com, alejandro.j.jimenez@oracle.com References: <20251001060954.5030-1-suravee.suthikulpanit@amd.com> <20251001060954.5030-11-suravee.suthikulpanit@amd.com> <20251006144936.GS3360665@nvidia.com> From: "Suthikulpanit, Suravee" In-Reply-To: <20251006144936.GS3360665@nvidia.com> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-ClientProxiedBy: SN7P220CA0017.NAMP220.PROD.OUTLOOK.COM (2603:10b6:806:123::22) To DM8PR12MB5445.namprd12.prod.outlook.com (2603:10b6:8:24::7) Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DM8PR12MB5445:EE_|SJ2PR12MB9212:EE_ X-MS-Office365-Filtering-Correlation-Id: bd9dc2d4-7419-46e1-b6c4-08de05e1446f X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|366016|376014|7416014|1800799024; X-Microsoft-Antispam-Message-Info: =?utf-8?B?M2F4a3VIL3VLdFhUS0NobFhyNTliRitsV1FoeC9xaEE3VFU0STlSbVpxV2Nw?= =?utf-8?B?byt0RUdXa2xmWGtsWTRzV1cza2RFZ21rb2RQYVlGejZuWEJDNy9wOStXcHo5?= =?utf-8?B?QjJDdkdlUlUza0pKV2p0ZUNLWUV0VlRsS3dMUDI0ZVRJd3FBc3BEenVHVTlQ?= =?utf-8?B?eE0zWkFodE5ieXJxQVVFVWVuMS9sRU5qdjFkVzFTUThQREtPSENGbXk1V1U4?= =?utf-8?B?eWR5TkZtRDNjNFVVNWZibU80K1FRd1BjRjcxaWF2NTRhQVM3YmdiTEJjM3hz?= =?utf-8?B?bTJXWHcrOWIxd202dHR1TUJDOURzYjBmaGRtWXV3N0wvL1BNd1pZMWtVNDFk?= =?utf-8?B?N3dTQ2o1TklmaHhta3JUZHNSWmF5dWl4TWdzTzQ2eVV1TjAxaU9QMDFiMWZq?= =?utf-8?B?bTZjZkUyNUoyazhOOFZqcTJFeGhSVkczbnh4LzIyQmtLQUhDZWkzbVdrMlMz?= =?utf-8?B?a2dGR3A0RDUrUXZDbjQ1UXhWa1M3YU5OamNSTTNQS2NQSkUwdzZGTUFNQmNt?= =?utf-8?B?MzAveGcrZFpoanY2b2MxNndjZC9qYURDRUt3Nmtic3VNT3J0bk5OcTlZYXlK?= =?utf-8?B?RHdiNHVJcThSTnFLUU9EZ2daOWg4SnBEWS9pTTQ1SVpob0pMNHRNTWJkdzEz?= =?utf-8?B?SlE2eTc4RnZISnozOW5YdWJIbGd0MFJnQTBjeHhKVHRnMkE0OGtRd3liY1Vi?= =?utf-8?B?T0p0Myt4NGZLU3VGRXltKzFEMFJoNUF4dERUeFpuMWlEM041Qk1RREZlbzRi?= =?utf-8?B?dWQ3eTk1VGM0aWNtdWtMZmhtcmo2ZGxxaW82dytaaThkZithMFZ1YjlzZDEx?= =?utf-8?B?eG5mbFBMYjcxaUJkMkZlNkcvZ2pQUHJLR0txb1R5SUFibjRRSm9XTFV3Y1VE?= =?utf-8?B?L2VxY2pXWDBZQ2VIN05USkpZMHpqZmRRS0FMbmpVQkpGc1g1KzdNQVZZSnN4?= =?utf-8?B?SDZ1UUtwOXg1SVVYUnpwaEllZUh0cG0zSVc5UnVlVldkTHNVRkY1RnozRU90?= =?utf-8?B?OE5YdHdOZ3BJblZYaitma043R2RlOUdtS1IwRGtHL2NjUC9zTURnNkJOVHox?= =?utf-8?B?U1AzOTZWSSs0TVpjc2JId2NjODRWYklDVCsremNYdHJWT2dUSXFLaDR1MlJq?= =?utf-8?B?ejd6dnJTbEpPS1JOQXp1Q2Vkbk9YSDBnRXZJM1hZaU5LT0Y5LzNPM0NSWStK?= =?utf-8?B?UWlMa1RyQjgrMGlwTEFlY2xSSDRIMkhKU3FGOW5zckZoc2ZTMEE0SlFrS2Fp?= =?utf-8?B?a3lXUjk1RG5IOTliY2pxMXhzcVFCcnp5U2NJNU9ZWklNbGw4blFvQUxFWThX?= =?utf-8?B?OCs3VVI0dTY0NGZQeHZmN3o1VXQ1YWFHb04xQ2NtZ1RVYUE5YzVoVUw0M0hY?= =?utf-8?B?dlRwaDdwYlNUWGpNUElFMUJlQTZadzNLdnZGdlpydmphS000ZWNHM3FHMklH?= =?utf-8?B?WCtoUXdEWGFxK3ZTQmNUU2RVR0NSWVlReW9FWU51TTFERXZIMEh4clVYdGJF?= =?utf-8?B?NXRrVjcxOVZpY1RVMjlkMnhVcjM1aWk3cDBtemhYUVg4ZEZOK2luclJrRGdB?= =?utf-8?B?S0FmaEdhN3UxTURCeWdTK1VCRjNqTkVDVG04aG1ibWc2SGVTR0Z5NktLMStv?= =?utf-8?B?WlRaVklkd2xMdW9wK1R0ejNGU2FxTUhVMmNTT0FuQStyQVZia0J5bXhPQmty?= =?utf-8?B?NXZTWlhiRFk5V256VEluNUpSSi8zQ0d4cVBZZTBhbUlyNUJHYzBEZWZQOThG?= =?utf-8?B?RnFLTzY5N3BCUzQvT24zbXVUbkpheC9WZ01yWWpMNEhYMU4vRTZVOWx6dlRr?= =?utf-8?B?eUtDcFAzQit3TXlrQmpVeUhNeVlZN1g3S2V5M2hNcWt4M1loTDBKZmthbEor?= =?utf-8?B?SzcrVzVXYmRvU1JaZDVPZUFkOTFWSmF6MzRpd0pucDB0MWVnMEZHRHNmSW1u?= =?utf-8?Q?O12sfLWFkp1fMnwtBv4K5L6hSX25SGk+?= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DM8PR12MB5445.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(366016)(376014)(7416014)(1800799024);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?eDZnNC9Sc2h6aTBRKzhaMGtiRkRISHBkd1ZHbU82SWlNRnR3ejNnblZmR09Q?= =?utf-8?B?UUNIM3RrcWFyaG5Gdy9Fb3Y5RURUZDBPcThjWW5JMHJCaFY0SzY1NXBheXZt?= =?utf-8?B?ZjdJK0pHMjZKL0xReUdhWUh2THExNEpJZko3R0t2VHZMcysrYnhWaDBnVnpV?= =?utf-8?B?WG4vYno5TFQvMVZkSHNSWnZPR3RVTWk0SnVaaXZkOFZ2QktiWmdMWDFGRll6?= =?utf-8?B?bUlWRndRamh0RW9ZcC9ZVzdZYjFMZlRnOFpXdVUrY3UzaWpSWC9KMm84WmxM?= =?utf-8?B?cDVsMkZYeWdRZ0g5L0VHUG56V3NWeWlWV3ltN3UxZ21BRjdteUhqRmk2YUpF?= =?utf-8?B?cDJaU2RoUUpRYVVzTUkzRnloOTh5eTdCa2s3TDR4QVU4MHFSTkFQVVdYK21i?= =?utf-8?B?Q01tS08vcm9kck9VdTlvLzFyRzY3YjJQMUp6YVNWSzFUSDdvc3RzQitjTDls?= =?utf-8?B?MlJ0eVpFU0RDK3dHRkErbjVFMzJOWC9aOEJYMW9Hc1VqZHFiWEdpUk1ZckJv?= =?utf-8?B?UEVvTXdFbWw1THQ2WXB2VGpvSVlXNlBuWHJtZzlneXdBbkRlWnhxSE9GaGNT?= =?utf-8?B?dVB1T2lMU3FOWFMzamF1MGg0MHJoemg5Z3dNU2ZJUjRML1ZBMjFaUkl2YVl1?= =?utf-8?B?Q2ZTSi9IQUs2NERSZWNZbGJjWm1QWVpsYjNKU2VqZE10bFRTcXY3TzI1aEhG?= =?utf-8?B?cy9tZWJQek5YcXJmVTJiYnY0M1dmc1MzYnJIMU4xeXBOLzhHTmEwdy8rRC8v?= =?utf-8?B?cWg2SlhtVjBXMEdTZ3BZdDdjVWxMYmdQVk1abDFJTFd4UUg0eTV6OEVlR3ll?= =?utf-8?B?ZGtTWW16eEZWQXVDN2dZQkFiblBuUmZWZC9Xb1dEd2JoNmlQUHpNc2tWcFds?= =?utf-8?B?bnFGcWNQUUNFUTU1ME1Ud2gyUVl4UmtKWVhDdldMeml1RDRMUnNQRzR0U1Zk?= =?utf-8?B?amhCTDhWMDdoVndsYjJ5TWgzaHdiK2lWVkNmTmVaRlJqZ29ybTBMM3k0VkNw?= =?utf-8?B?bzZ3a1ZTRzRLRmszTVBsdHRFRDdsN0NVSENaVkFJZkdVNEcweUtTakQyWTJi?= =?utf-8?B?bzY0cUlVcWsycFBlVWJLVDJpWWRzQXp4OWkrWVV4TXByQlRKNTA5alJsemtY?= =?utf-8?B?VUQ4Rmp0a3Y2RExaWmJNYmpjdzNqbUN1TFpLVkJnczY2OHBaOXJsYTFzdFMw?= =?utf-8?B?ajI1Q1VYS21qZ3IrcENrekprSzJFS2s3OEVEMXUrQ2R4TVVZZHJZc2YzRmE0?= =?utf-8?B?SUFyQ21zb0hBa1lremk5WWJSeHBDRldFTklrRkRObkpiTW5UNzdEbUNCd3F3?= =?utf-8?B?b00xUGMvVjRvcmNaaDVDVzc3RjRNbTRtSUR2dHg2TzMzNWN1eTZlTFluNUV2?= =?utf-8?B?MVNXNWpORTNVbmRvRHJnUEJ6TXhCVjRueldPaEdxQnkzRWZmUlZkRlF2Zktr?= =?utf-8?B?TUJmeURuY056YzV1Qkk4LzM4akFCSnBlZW90UVpzMThGY2E3QkwyalpDS1BG?= =?utf-8?B?WUVZNEh5WnhqbTQzUGZwMCtRM3F0dWlCd05xVHgxNmY5a2g2OHFVT25sdXI3?= =?utf-8?B?NG1PQmdJbG9pODFlL3BiZm1nbVBLNVQ5dkM1UFJ0Z2doaUtiUjQzdHBZYit5?= =?utf-8?B?d2JqQm9sc3FpZldTU0VmZXEwV0ZCTCsrS3Q5V1RSVDJydGx4d2JKMzRrcTNQ?= =?utf-8?B?Yzd0Zk5heXYzN245SXBNdGhtSDJ2QUJlK1RLaUJ2ajhrMndHalVUQ2dza1pN?= =?utf-8?B?Sjg0S0xhVERSUkNmUVZMZUdocHZ6ZmdVQnUySWR5dk5vb3U3aFlOS3dEU1ZY?= =?utf-8?B?SEJCL3RhTVMwa0liWFZEVmVKV011WFJ0bWx3Tnd0S01IV0crbEFzdHRFc0kx?= =?utf-8?B?SEFYZTF5aHRlNDRmSU9PUTBnNlJHdFI5bkZ6dXpkU3lLMWsrTVV1T0FqV2Jw?= =?utf-8?B?RFhyNFpSbW90T2JwMUVqK1JYemNYNlh0Sit2TGZWWjNpaHFBVXpqeW1VWWhR?= =?utf-8?B?VUNQS1VuZ0pudDI4QWhtZ2xFK1JLNEtjVjFFUC8vdkNvK0F0UEJNTGI1cmRP?= =?utf-8?B?b0hQODhHY3dTSVp1MFVZNENPN1VkR3V3bzZIV0lYbGpnYlA0N1RMQ1RyQ3ls?= =?utf-8?Q?CUhx6eTzKBAJIyxFebIlRM53O?= X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-Network-Message-Id: bd9dc2d4-7419-46e1-b6c4-08de05e1446f X-MS-Exchange-CrossTenant-AuthSource: DM8PR12MB5445.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 07 Oct 2025 20:37:00.9432 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: UTlPdcdDno3RokwdcqF8SpYBNYea3pJLrdDG2iHcgzDdSMKhNpz7GMT6ZkCsjEKGepK+Lq/7cyuxOhJwB6f4Jg== X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ2PR12MB9212 On 10/6/2025 9:49 AM, Jason Gunthorpe wrote: > On Wed, Oct 01, 2025 at 06:09:52AM +0000, Suravee Suthikulpanit wrote: > .... >> + >> + ret = iommu_copy_struct_from_user(&gdte, user_data, >> + IOMMU_HWPT_DATA_AMD_GUEST, >> + dte); >> + if (ret) >> + return ERR_PTR(ret); >> + >> + ndom = kzalloc(sizeof(*ndom), GFP_KERNEL); >> + if (IS_ERR(ndom)) >> + return ERR_PTR(-ENOMEM); >> + >> + ndom->domain.ops = &nested_domain_ops; >> + ndom->domain.type = IOMMU_DOMAIN_NESTED; >> + memcpy(&ndom->guest_dte, &gdte, sizeof(struct dev_table_entry)); >> + >> + /* Due to possible aliasing issue use per-device nested domain ID */ >> + ndom->id = amd_iommu_pdom_id_alloc(); > > I've forgotten the details, but doesn't the gdet have the virtual > domain ID in side it? Shouldn't this be going to the viommu struct and > mapping virtual domain IDs to physical ones so they can be shared if > the guest says it is safe to share them? I guess that is an > optimization, but it should have a note here explaining it. The gDTE[DomainID] field contains guest Domain ID (gDomID). The host IOMMU driver uses the gDomId and guest ID (gid) to index the Domain ID mapping table, and store the host Domain ID (hDomID) in the table entry. This data structure is required by hw to translation gDomID->hDomID to virtualize guest invalidation command. This will be part of the upcoming series to enable hw-vIOMMU. This ndom->id is the hDomID, which is currently allocated per-device to avoid TLB aliasing i.e. A guest w/ multiple pass-through devices w/ the same hDomID (same stage 2 table) and different stage-1 tables with same PASID. IOMMU would use the same TLB tag, which results in TLB aliasing issue. Therefore, we workaround the issue by allocating per-device hDomID for nested domain. Thanks, Suravee Thanks, Suravee