From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from BL0PR03CU003.outbound.protection.outlook.com (mail-eastusazon11012064.outbound.protection.outlook.com [52.101.53.64]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7BB31370D56 for ; Thu, 2 Jul 2026 19:26:06 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=52.101.53.64 ARC-Seal:i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1783020368; cv=fail; b=CGei1lq5S/bRja+Y9GZsB3eiDx8NuDRRk64rv2ycvxSNYssmnA2cr58sRDBQopJky7VyqjmsPe/P8aBDuPhNt5j3VpYyihbc2/PYgshbT/N16rjmZEsM0txnxqRqp17EXkxlvUwT/sxQr3BNAymrW/c3QuQZz1P2K6jzp/J2uwI= ARC-Message-Signature:i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1783020368; c=relaxed/simple; bh=giKr9WJKGvBcpscCY/3o/tzl9V44AQzuQA/C3bbdyuw=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=KGbWrShOWE5pMZ9U0fO072lgxNKWFGS+sTIrW3tdQICU1bJdOzCXRM6tGXYLBY7Z+8GJP8ybLxOIIJ5hcS7b7wO9sBjmPislH0L5QN+JWCy38aLYpVYnQblaR0Z2dPD1nBpSNHPmQgahaxutfJlY+S9aVdgCIEFZYcLsossFy24= ARC-Authentication-Results:i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com; spf=fail smtp.mailfrom=nvidia.com; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b=Wj4hJNOO; arc=fail smtp.client-ip=52.101.53.64 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="Wj4hJNOO" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=k+ijoGnlxaK4ivkEWLQiQp4HBcQJM33qCcmg82h0Tas1XPMfu3yXFgfCZozcWfxDaFreZJQXyTQQX8uWWn6TeBo3qsXzNeb+UtQJ+VqU7b6rKOvCyDs8sGdVZvVN1Ddu/AfWj2RMr+GWM9/UtmYzCa6RvzUweKTjx5N0LN/3rFZAtBw2nZtcIWiIhgg+2dLxANjGXdzWhg6uZjB+7BUiOXFw/cZ2B0UZj9ZadXwN6+haMIyDuNlOEGgwnltFn1VNdpy2b7Y9a1YvYjG2jXQrTCFn8aw/OrEoCRLFvZiQajGYu946hJvujL8tRxVUILQ1mF9UTz8cov0rAq2W82CQTg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=thmPwd1AqBnCZfdkjnSgCwkL+gVZyseAvzLYWihBeII=; b=JiuGgM0VhA06LMiqEEEnSlDaEFrF9FOi+B4gO8fbh6Il6GKqN9dj0x3Wn+CJiwec9DIa4tUGoO8NYyIwCaJO9q6IBU72gzUzI+hsxkDE0HbZX5HMEG7lBmj5Yc/pXjhvvrJqMfdYzuXe590wihfk7fO04X6zoVjE7NOyN7RKsfdgwQncRDFO6ay5GdkEJFgZaCzcX4T9UiGMlirvkbONREcqN0SAuu0oSsDcw58J+xhjMm8bQWU7pCg6/+A5r1/f19tSzCI8vLRNroNXFUv0xpAhyK6ULRGaBWipDocdCEC6tQeZ8R+K0oi45p6ZelCJLf0LFaqjvEjkozhqCGDR+Q== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.161) smtp.rcpttodomain=linuxfoundation.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=thmPwd1AqBnCZfdkjnSgCwkL+gVZyseAvzLYWihBeII=; b=Wj4hJNOOSXh++nJWT7zKgWSyAR8xS/uba+C1a6zd/SUyJSAI6sfvpCe4ClhygiidG/mi61HaziDtKFt9r/wFkOnYiZE8OOXeL228iUtgHiLSmntZwdEPUkBmrdkv9OkKJB5WTSh7afhGYRjcvp68keCJsOjshkIn80hqb4J7CqS5gxeq5BzGsJBEsyJBR+/CLq8Tf1gmSv1h9l0RFW4FBW2ciscC1uysf0w7BbMiYoPKBtwadCk+4uI8HmSGk7H3z+WSpK8d+s4RtfIfvbSPOsONdFhy5ZrV1NvV67G42FV6rTz7PGkG1Oknf8UyufG0QQL1cbxyVRkMnMUDgO2IZQ== Received: from PH7PR02CA0008.namprd02.prod.outlook.com (2603:10b6:510:33d::27) by DS2PR12MB9776.namprd12.prod.outlook.com (2603:10b6:8:2bd::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.181.10; Thu, 2 Jul 2026 19:25:58 +0000 Received: from SN1PEPF000252A2.namprd05.prod.outlook.com (2603:10b6:510:33d:cafe::6b) by PH7PR02CA0008.outlook.office365.com (2603:10b6:510:33d::27) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.21.181.10 via Frontend Transport; Thu, 2 Jul 2026 19:25:58 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.117.161) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.161 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.161; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.117.161) by SN1PEPF000252A2.mail.protection.outlook.com (10.167.242.9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.181.6 via Frontend Transport; Thu, 2 Jul 2026 19:25:58 +0000 Received: from rnnvmail202.nvidia.com (10.129.68.7) by mail.nvidia.com (10.129.200.67) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.20; Thu, 2 Jul 2026 12:25:34 -0700 Received: from rnnvmail203.nvidia.com (10.129.68.9) by rnnvmail202.nvidia.com (10.129.68.7) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.20; Thu, 2 Jul 2026 12:25:34 -0700 Received: from localhost.nvidia.com (10.127.8.12) by mail.nvidia.com (10.129.68.9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.20 via Frontend Transport; Thu, 2 Jul 2026 12:25:33 -0700 From: Ankit Agrawal To: , , CC: , , , , , , , , Subject: [PATCH 1/4] platform/nvidia: Introduce nvgrace-egm driver and enumerate EGM regions Date: Thu, 2 Jul 2026 19:25:29 +0000 Message-ID: <20260702192532.455400-2-ankita@nvidia.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20260702192532.455400-1-ankita@nvidia.com> References: <20260702192532.455400-1-ankita@nvidia.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Content-Type: text/plain X-NV-OnPremToCloud: ExternallySecured X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: SN1PEPF000252A2:EE_|DS2PR12MB9776:EE_ X-MS-Office365-Filtering-Correlation-Id: bfba0de6-63b1-45c5-e753-08ded86fbe93 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|36860700016|1800799024|82310400026|376014|23010399003|18002099003|22082099003|3023799007|6133799003|11063799006|5023799004|56012099006; X-Microsoft-Antispam-Message-Info: hhwfBBvXuHe58XIkaOdUkWjLC+E3d76+2euecFVnu2z+rJJY0FP8Nh4JbTZVYDrAZnaZKLaUEDtGzZZuglt0jICdONTMfwQ9RBTadBSypZjbG85NEUlURdE5+o6eST92hoNU+eMo2YyHQ8mtXu1fxc6yWzrybiGGFCRzTvMBmOKhO/LqyBn0uFbzqcD5uE9XX6aJ6MxRo4eVJPVHkR8flS0wfRkGHMpeQ+y/YIRjJtRD1NQH7CzIOUDvVS7pS44OIgTHGzicq2gSiVjtze4Or2ybtQoJODFOPq070ax+QcMhSqLpVwbYObgxQfYsvwCuPAAKVZWMAm1FdJW3AABozIoSzx6B65YMfnzU7qVedRMpTCUGWrHGRJpxpe7BrtlYYidEJoE8gwmDzwpoT9CQKIZ0Ithu6VvT82gO47p9mUAEQWonbp2f0BCJTdyCY3m5R9tqlmDO1ahAM+PJ4IP1D4ZmuT05etWR9CT8iaisjgiATAWs3J4rbCRl2ibV82tWpw2QuthxA4cXkFQpRFkpU/S+ooqEgZ2AFZ/Rm6Zf50SzRQikK1krGwxGo8rRqWdk/AqNv40deUo3d0N+YB+8vG3Nbisil6vzvbY3EvYfjcj0GDQrDmK7L15wwm4iyng7WfaJYu2w/5hbc5i18jGyoljPBZtESTSpjl/fJEX9SRKsVnsy96DAFaQqf+UWFxT4SZQep9Sk1PpfiwKQJevpcw== X-Forefront-Antispam-Report: CIP:216.228.117.161;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc6edge2.nvidia.com;CAT:NONE;SFS:(13230040)(36860700016)(1800799024)(82310400026)(376014)(23010399003)(18002099003)(22082099003)(3023799007)(6133799003)(11063799006)(5023799004)(56012099006);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: i8z+ytRuC0jbSVWSn/pl4HlgoEUC0XzSEQBQW+CE3eZpQOIf+JCCYGLeGQqsW9DEjpG2h3+k4y72JcxrZuUndK01JGA52kRhKUGyx44RGlqbcZw7zkWhY0D3q87veWinyBpwiIxxuB5qd/0Cy8TR6iqfLGhLZEXo0TZ/1LRTNZR9bWqRlo6veXLOOL7ykFp9UUJDDT3bPrdyi33/LGqCgYqaVqMpJN9+UU34nFaAzO+jSlxDpRubEcieDI9roH6qN8MWP4QLiC7oGlZH1H0KRXlP4GlqNqPqeAmI/+v7iSTMdzmase35Idt6/rvejjR+1OM9XqUaezABPMJd+OQQv4/ZtF+/8BgO3TA1v6ZgUKgw9W7UpJVWefVadJM3y9oENG4Hgs+ubbNjKvsMJspmNw03eKmtiWnXoIPhLKAiwwl5/5/UrR/UgD1XbhdrCWLF X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 02 Jul 2026 19:25:58.2028 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: bfba0de6-63b1-45c5-e753-08ded86fbe93 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.117.161];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: SN1PEPF000252A2.namprd05.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS2PR12MB9776 The Extended GPU Memory (EGM) feature enables the GPU to access system memory allocations within and across nodes through a high-bandwidth path on Grace-based systems. The GPU can utilize system memory on the same socket, a different socket or a different node in a multi-node setup. When EGM mode is enabled through SBIOS, host system memory is partitioned into two parts: a Hypervisor region for the host OS and a Hypervisor-Invisible (HI) region for the VM. Only the hypervisor region is part of the host EFI map and visible to the host OS on bootup. The HI partition is interchangeably called the EGM region. Its base SPA and size are exposed through ACPI DSDT properties. Whilst the EGM region is accessible on the host, it is not added/hot-plugged to the kernel. The HI region is assigned to a VM by mapping the QEMU VMA to the SPA using remap_pfn_range(). So it looks like the following: |---- Sysmem ----| VM view | | |IPA <-> SPA map | | | |--- HI / EGM ---|-- Host Mem --| Host view Introduce the nvgrace-egm platform driver module and implement full EGM region discovery, char device exposure and sysfs metadata. Build infrastructure: - Kconfig, Makefile entries, MAINTAINERS entry. - An 'egm' device class and a range of char device minor numbers. MAX_EGM_NODES is set to 4 to matches the largest 4-socket Grace configuration. Discovery: - Walk all PCI devices at module init time. For each one advertising "nvidia,egm-pxm", read the EGM region properties (base SPA, size and PXM domain) from the ACPI DSD table via the GPU device handle, using "nvidia,egm-base-pa" and "nvidia,egm-size" properties. - Maintain a module-level list (egm_chardevs) of discovered regions, de-duplicating by PXM so each socket gets exactly one EGM device regardless of how many GPUs it has (2 GPUs per EGM device in GB200). Char device creation: - Each EGM region is exposed as /dev/egmX, where X is the PXM number. Embed struct device and struct cdev inside nvgrace_egm_dev so the device lifecycle follows the standard kernel device model. - Implement setup_egm_chardev() and del_egm_chardev() for setup and teardown. The devnode callback sets device permissions to 0600. - Stub file operations (open, release and mmap) are registered. GPU tracking and sysfs: - Track which PCI devices are associated with each EGM region in a per-device GPU list (struct gpu_node). - Expose sysfs symlinks from each EGM device to its associated GPU PCI devices to allow the userspace to determine GPU-to-socket affinity and replicate the host EGM topology in the VM. Helps admin understand GPU association. - Expose the EGM region size via an egm_size sysfs attribute so QEMU can determine the memory-backend-file size. On a 2-socket Grace Blackwell setup this appears as: Socket0: /sys/devices/virtual/egm/egm4/egm_size Socket1: /sys/devices/virtual/egm/egm5/egm_size Introduce nvgrace_egm_create_pci_egm_devs() and its counterpart nvgrace_egm_destroy_pci_devs() that are called from module init/exit. Suggested-by: Alex Williamson Suggested-by: Jason Gunthorpe Suggested-by: Aniket Agashe Suggested-by: Matthew R. Ochs Assisted-by: Claude:claude-opus-4.8 Signed-off-by: Ankit Agrawal --- MAINTAINERS | 5 + drivers/platform/Kconfig | 1 + drivers/platform/Makefile | 1 + drivers/platform/nvidia/Kconfig | 10 + drivers/platform/nvidia/Makefile | 3 + drivers/platform/nvidia/egm.c | 459 +++++++++++++++++++++++++++++++ 6 files changed, 479 insertions(+) create mode 100644 drivers/platform/nvidia/Kconfig create mode 100644 drivers/platform/nvidia/Makefile create mode 100644 drivers/platform/nvidia/egm.c diff --git a/MAINTAINERS b/MAINTAINERS index 15011f5752a9..2edf903a7746 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -19344,6 +19344,11 @@ L: linux-acpi@vger.kernel.org S: Maintained F: drivers/acpi/apei/ghes-nvidia.c +NVIDIA GRACE EGM PLATFORM DRIVER +M: Ankit Agrawal +S: Supported +F: drivers/platform/nvidia/ + NVIDIA VRS RTC DRIVER M: Shubhi Garg L: linux-tegra@vger.kernel.org diff --git a/drivers/platform/Kconfig b/drivers/platform/Kconfig index 312788f249c9..21f2dc901301 100644 --- a/drivers/platform/Kconfig +++ b/drivers/platform/Kconfig @@ -22,3 +22,4 @@ source "drivers/platform/arm64/Kconfig" source "drivers/platform/raspberrypi/Kconfig" source "drivers/platform/wmi/Kconfig" +source "drivers/platform/nvidia/Kconfig" diff --git a/drivers/platform/Makefile b/drivers/platform/Makefile index fa322e7f8716..96de8fa26ed5 100644 --- a/drivers/platform/Makefile +++ b/drivers/platform/Makefile @@ -15,3 +15,4 @@ obj-$(CONFIG_SURFACE_PLATFORMS) += surface/ obj-$(CONFIG_ARM64_PLATFORM_DEVICES) += arm64/ obj-$(CONFIG_BCM2835_VCHIQ) += raspberrypi/ obj-$(CONFIG_ACPI_WMI) += wmi/ +obj-$(CONFIG_NVGRACE_EGM) += nvidia/ diff --git a/drivers/platform/nvidia/Kconfig b/drivers/platform/nvidia/Kconfig new file mode 100644 index 000000000000..8554a58b1141 --- /dev/null +++ b/drivers/platform/nvidia/Kconfig @@ -0,0 +1,10 @@ +# SPDX-License-Identifier: GPL-2.0-only +config NVGRACE_EGM + tristate "EGM driver for NVIDIA Grace Hopper and Blackwell Superchip" + depends on ARM64 || (COMPILE_TEST && 64BIT) + help + Extended GPU Memory (EGM) support for the GPU in the NVIDIA Grace + based chips required to avail the CPU memory as additional + cross-node/cross-socket memory for GPU using KVM/qemu. + + If you don't know what to do here, say N. diff --git a/drivers/platform/nvidia/Makefile b/drivers/platform/nvidia/Makefile new file mode 100644 index 000000000000..a89f820908fb --- /dev/null +++ b/drivers/platform/nvidia/Makefile @@ -0,0 +1,3 @@ +# SPDX-License-Identifier: GPL-2.0-only +obj-$(CONFIG_NVGRACE_EGM) += nvgrace-egm.o +nvgrace-egm-y := egm.o diff --git a/drivers/platform/nvidia/egm.c b/drivers/platform/nvidia/egm.c new file mode 100644 index 000000000000..a340c4fcbc7a --- /dev/null +++ b/drivers/platform/nvidia/egm.c @@ -0,0 +1,459 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved + */ + +#include + +#define NVGRACE_EGM_DEV_NAME "egm" +#define MAX_EGM_NODES 4 + +static dev_t dev; +static struct class *class; + +struct gpu_node { + struct list_head list; + struct pci_dev *pdev; +}; + +struct nvgrace_egm_dev { + struct device device; + struct cdev cdev; + phys_addr_t egmphys; + size_t egmlength; + u64 egmpxm; + struct list_head gpus; +}; + +struct nvgrace_egm_dev_entry { + struct list_head list; + struct nvgrace_egm_dev *egm_dev; +}; + +/* + * Track egm device lists. Note that there is one device per socket. + * All the GPUs belonging to the same sockets are associated with + * the EGM device for that socket. + */ +static LIST_HEAD(egm_chardevs); + +static int nvgrace_egm_open(struct inode *inode, struct file *file) +{ + return 0; +} + +static int nvgrace_egm_release(struct inode *inode, struct file *file) +{ + return 0; +} + +static int nvgrace_egm_mmap(struct file *file, struct vm_area_struct *vma) +{ + /* + * Mapping the EGM region into userspace is implemented by a later + * patch. Until then refuse the mmap. + */ + return -EOPNOTSUPP; +} + +static const struct file_operations file_ops = { + .owner = THIS_MODULE, + .open = nvgrace_egm_open, + .release = nvgrace_egm_release, + .mmap = nvgrace_egm_mmap, +}; + +static int nvgrace_egm_create_gpu_links(struct nvgrace_egm_dev *egm_dev, + struct pci_dev *pdev) +{ + int ret; + + ret = sysfs_create_link(&egm_dev->device.kobj, + &pdev->dev.kobj, + dev_name(&pdev->dev)); + + if (ret && ret != -EEXIST) + return ret; + + return 0; +} + +static void remove_egm_symlinks(struct nvgrace_egm_dev *egm_dev, + struct pci_dev *pdev) +{ + sysfs_remove_link(&egm_dev->device.kobj, dev_name(&pdev->dev)); +} + +static int add_gpu(struct nvgrace_egm_dev *egm_dev, struct pci_dev *pdev) +{ + struct gpu_node *node; + + node = kzalloc_obj(*node, GFP_KERNEL); + if (!node) + return -ENOMEM; + + /* + * The pdev is stored for the lifetime of the EGM device but it is + * obtained from a for_each_pci_dev() walk that drops its reference as + * it advances. Take a reference here so the stored pointer stays valid + * even if the GPU is later unbound; remove_gpus() drops it. + */ + node->pdev = pci_dev_get(pdev); + + list_add_tail(&node->list, &egm_dev->gpus); + + return nvgrace_egm_create_gpu_links(egm_dev, pdev); +} + +static void remove_gpus(struct nvgrace_egm_dev *egm_dev) +{ + struct gpu_node *node, *tmp; + + list_for_each_entry_safe(node, tmp, &egm_dev->gpus, list) { + remove_egm_symlinks(egm_dev, node->pdev); + list_del(&node->list); + pci_dev_put(node->pdev); + kfree(node); + } +} + +static void egm_chardev_release(struct device *dev) +{ + struct nvgrace_egm_dev *egm_chardev = container_of(dev, struct nvgrace_egm_dev, device); + + remove_gpus(egm_chardev); + kfree(egm_chardev); +} + +static struct nvgrace_egm_dev *setup_egm_chardev(u64 egmphys, u64 egmlength, + u64 egmpxm) +{ + struct nvgrace_egm_dev *egm_chardev; + unsigned int baseminor = MINOR(dev); + int ret; + + /* + * Only MAX_EGM_NODES minors from baseminor were reserved. Reject a PXM + * outside that window. + */ + if (egmpxm < baseminor || egmpxm - baseminor >= MAX_EGM_NODES) { + pr_err("nvgrace-egm: EGM proximity domain %llu outside reserved minor window [%u, %u)\n", + egmpxm, baseminor, baseminor + MAX_EGM_NODES); + goto create_err; + } + + egm_chardev = kzalloc_obj(*egm_chardev, GFP_KERNEL); + if (!egm_chardev) + goto create_err; + + device_initialize(&egm_chardev->device); + + /* + * Use the proximity domain number as the device minor number. + * So the EGM corresponding to node X would be /dev/egmX. + */ + egm_chardev->egmphys = egmphys; + egm_chardev->egmlength = egmlength; + egm_chardev->egmpxm = egmpxm; + INIT_LIST_HEAD(&egm_chardev->gpus); + + egm_chardev->device.devt = MKDEV(MAJOR(dev), egm_chardev->egmpxm); + egm_chardev->device.class = class; + egm_chardev->device.release = egm_chardev_release; + cdev_init(&egm_chardev->cdev, &file_ops); + egm_chardev->cdev.owner = THIS_MODULE; + + ret = dev_set_name(&egm_chardev->device, "egm%llu", egm_chardev->egmpxm); + if (ret) + goto error_exit; + + ret = cdev_device_add(&egm_chardev->cdev, &egm_chardev->device); + if (ret) + goto error_exit; + + return egm_chardev; + +error_exit: + put_device(&egm_chardev->device); +create_err: + return NULL; +} + +static void del_egm_chardev(struct nvgrace_egm_dev *egm_chardev) +{ + cdev_device_del(&egm_chardev->cdev, &egm_chardev->device); + put_device(&egm_chardev->device); +} + +static char *egm_devnode(const struct device *device, umode_t *mode) +{ + if (mode) + *mode = 0600; + + return NULL; +} + +static ssize_t egm_size_show(struct device *dev, struct device_attribute *attr, + char *buf) +{ + struct nvgrace_egm_dev *egm_dev = + container_of(dev, struct nvgrace_egm_dev, device); + + return sysfs_emit(buf, "0x%zx\n", egm_dev->egmlength); +} + +static DEVICE_ATTR_RO(egm_size); + +static struct attribute *attrs[] = { + &dev_attr_egm_size.attr, + NULL, +}; + +static const struct attribute_group attr_group = { + .attrs = attrs, +}; + +static const struct attribute_group *attr_groups[] = { + &attr_group, + NULL +}; + +/* + * Determine if the EGM feature is enabled. If disabled, there + * will be no EGM properties populated in the ACPI tables and this + * fetch would fail. + */ +static int has_egm_property(struct pci_dev *pdev, u64 *pegmpxm) +{ + return device_property_read_u64(&pdev->dev, "nvidia,egm-pxm", + pegmpxm); +} + +static int fetch_egm_property(struct pci_dev *pdev, u64 *pegmphys, + u64 *pegmlength) +{ + int ret; + + /* + * The memory information is present in the system ACPI tables as DSD + * properties nvidia,egm-base-pa and nvidia,egm-size. + */ + ret = device_property_read_u64(&pdev->dev, "nvidia,egm-size", + pegmlength); + if (ret) + return ret; + + ret = device_property_read_u64(&pdev->dev, "nvidia,egm-base-pa", + pegmphys); + + return ret; +} + +static struct nvgrace_egm_dev *get_egm_dev(u64 egmpxm) +{ + struct nvgrace_egm_dev_entry *egm_entry; + + list_for_each_entry(egm_entry, &egm_chardevs, list) { + /* + * A system could have multiple GPUs associated with an + * EGM region and will have the same set of EGM region + * information. Return the existing EGM device if already + * registered through a different GPU on the same socket. + */ + if (egm_entry->egm_dev->egmpxm == egmpxm) + return egm_entry->egm_dev; + } + + return NULL; +} + +static void nvgrace_egm_destroy_pci_devs(void) +{ + struct nvgrace_egm_dev_entry *entry, *tmp; + + list_for_each_entry_safe(entry, tmp, &egm_chardevs, list) { + list_del(&entry->list); + del_egm_chardev(entry->egm_dev); + kfree(entry); + } +} + +/* + * Walk all PCI devices and for each one that has a companion ACPI object + * advertising EGM properties, create an auxiliary device for that EGM range. + * Multiple GPUs on the same socket share a single EGM region (same proximity + * domain). So de-duplicate by skipping a pxm that is already registered. + */ +static int nvgrace_egm_create_pci_egm_devs(void) +{ + struct nvgrace_egm_dev_entry *egm_entry; + struct nvgrace_egm_dev *egm_dev; + struct pci_dev *pdev = NULL; + int ret; + + for_each_pci_dev(pdev) { + u64 egmphys, egmlength, egmpxm; + + if (has_egm_property(pdev, &egmpxm)) + continue; + + ret = fetch_egm_property(pdev, &egmphys, &egmlength); + if (ret) + continue; + + egm_dev = get_egm_dev(egmpxm); + if (egm_dev) { + ret = add_gpu(egm_dev, pdev); + if (ret) + goto free_dev; + continue; + } + + egm_entry = kzalloc_obj(*egm_entry, GFP_KERNEL); + if (!egm_entry) { + pci_dev_put(pdev); + return -ENOMEM; + } + + egm_dev = setup_egm_chardev(egmphys, egmlength, egmpxm); + if (!egm_dev) { + kfree(egm_entry); + pci_dev_put(pdev); + return -EINVAL; + } + + egm_entry->egm_dev = egm_dev; + + ret = add_gpu(egm_entry->egm_dev, pdev); + if (ret) { + del_egm_chardev(egm_dev); + kfree(egm_entry); + goto free_dev; + } + + list_add_tail(&egm_entry->list, &egm_chardevs); + } + + return 0; + +free_dev: + /* + * Reached only via goto from inside the loop where the pdev still + * holds the reference from the last pci_get_device(). So drop it here. + */ + pci_dev_put(pdev); + nvgrace_egm_destroy_pci_devs(); + return ret; +} + +/* + * The char device minor is the EGM proximity domain number which is sparse + * and need not start at 0. Find the lowest proximity domain among the EGM + * devices so the minor range can be reserved starting there. Returns -ENODEV + * if the system has no EGM devices. + * + * EGM PXMs are assumed contiguous within [lowest, lowest + MAX_EGM_NODES), + * which holds for the supported Grace configurations. + */ +static int nvgrace_egm_lowest_pxm(u64 *pmin_pxm) +{ + struct pci_dev *pdev = NULL; + bool found = false; + + for_each_pci_dev(pdev) { + u64 egmphys, egmlength, egmpxm; + + if (has_egm_property(pdev, &egmpxm)) + continue; + + if (fetch_egm_property(pdev, &egmphys, &egmlength)) + continue; + + if (!found || egmpxm < *pmin_pxm) { + *pmin_pxm = egmpxm; + found = true; + } + } + + if (!found) + return -ENODEV; + + /* + * The PXM is used verbatim as the minor (only MINORBITS wide). Validate + * the window would not overflow MINORMASK and be truncated. + */ + if (*pmin_pxm + (MAX_EGM_NODES - 1) > MINORMASK) { + pr_err("nvgrace-egm: lowest EGM proximity domain %llu\n" + "exceeds the minor range\n", *pmin_pxm); + return -ERANGE; + } + + return 0; +} + +static int __init nvgrace_egm_init(void) +{ + u64 min_pxm; + int ret; + + /* + * Each EGM region is exposed as /dev/egmX with the proximity domain X + * as its minor number. Reserve MAX_EGM_NODES minors starting at + * the lowest proximity domain so that every egmX device falls within + * the reserved range. With no EGM devices present, load without + * reserving any resources. + */ + ret = nvgrace_egm_lowest_pxm(&min_pxm); + if (ret == -ENODEV) + return 0; + else if (ret) + return ret; + + ret = alloc_chrdev_region(&dev, min_pxm, MAX_EGM_NODES, + NVGRACE_EGM_DEV_NAME); + if (ret < 0) + return ret; + + class = class_create(NVGRACE_EGM_DEV_NAME); + if (IS_ERR(class)) { + unregister_chrdev_region(dev, MAX_EGM_NODES); + return PTR_ERR(class); + } + + class->devnode = egm_devnode; + class->dev_groups = attr_groups; + + ret = nvgrace_egm_create_pci_egm_devs(); + if (ret) + goto cleanup_pci_devs; + + return 0; + +cleanup_pci_devs: + nvgrace_egm_destroy_pci_devs(); + class_destroy(class); + unregister_chrdev_region(dev, MAX_EGM_NODES); + + return ret; +} + +static void __exit nvgrace_egm_cleanup(void) +{ + /* + * When no EGM devices were present, the module loaded inertly and + * reserved nothing. @class stays NULL and there is nothing to undo. + */ + if (!class) + return; + + nvgrace_egm_destroy_pci_devs(); + class_destroy(class); + unregister_chrdev_region(dev, MAX_EGM_NODES); +} + +module_init(nvgrace_egm_init); +module_exit(nvgrace_egm_cleanup); + +MODULE_LICENSE("GPL"); +MODULE_AUTHOR("Ankit Agrawal "); +MODULE_DESCRIPTION("NVGRACE EGM - Module to support Extended GPU Memory on NVIDIA Grace Based systems"); -- 2.34.1