From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from SN4PR2101CU001.outbound.protection.outlook.com (mail-southcentralusazon11012048.outbound.protection.outlook.com [40.93.195.48]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7902D3DEAF6 for ; Wed, 10 Jun 2026 12:28:19 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.93.195.48 ARC-Seal:i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1781094501; cv=fail; b=jaYCY1l1LsEg+Gxap9+ENHCDg+7SZs78JCuJ/JtIc0kUHp7/3PCWP3FQd46DTabvCdDVd6MaFiGT9MCy3Cjs8X+SiYK0VyscEjUNUUd2cUAR0G9DkgqDmGzAv6dPh65Meezmk55jXszaHZe/Z+s0421QYYACB04ArEfe8EjOGe0= ARC-Message-Signature:i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1781094501; c=relaxed/simple; bh=feFaapsUNbNTab49fnL+S3ksAMkACQCVubpTvdx1XJQ=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=iv3cROKipkjR5SYsPnuI7/DsFK9ntNg/7S2g4oXXAeHFJvraN++0y94y1Eui8GRaUMqyhEornD1r2RcUkKMWpilgtwDxrPVH/sK8JYnDgyFDK68P9gENOJXfAfUsEETmmvpPeUtYZ7Nd3M64kum82rYt9TQF1tyt7XyiwU0uRTw= ARC-Authentication-Results:i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=PB04Gie6; arc=fail smtp.client-ip=40.93.195.48 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="PB04Gie6" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=vibl+r+JRgBPAr171PbqM0suHmaaAGaa7TaacP2cVAeeyo/Vbhd7p+prilmihPwQ+7YdrgVshkow2l5ZGaRjLzFc/O4ZxFuGnPsz90KNLDf2gVF9dnG2AJ7Yv6h6vUpVfDZoOaNN0VZqrii3cIh4jrPixaOfGnQOY5vNRl7RhA8HBa8LiOCS5bM2aKoqZYVTge/lXmv9rhR+XgJhTGU1edq4+2hzjwYpGf4Xf1GOrlivWdsh3MQsxSbQ27NYr8eERJgaLBIQO/e1kJ0PLOpgwcAou8ohUTfGxGHe0mfxlgqvke8W9zBm3oEgZiVFK8aVGGUjf9MmzpcbL4RrXsSYGw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=n5M9n6qDhAxsk+X7Y1Hj6xb3m+sIGhZYHr8FSdbaKfs=; b=lkccx2jzGnoK+uyVX6Lt8T8bxMcV/abAfBWC4wfO6NeCsQJfA6kRyb2iQzyrAuY0nEAgSFpPfXZaAbYFhhEocPSq8OY0vVNaqKzNN1DlURm3v8YGwxq+0/6gfyfYYynlg5U0YRH6Cr4xKIK6X7/Bpe2ImcvWhi97/JyqXd2XoG2pL57VpGo3lBaygU0CDJNwKw9XadDNVcotQ63nEX81Ub4lDN1fT8ZbHjPPHDE7m+n7xsuoYRnPg7U1tf+UvF2f5/K1K+GzG5UJixjlEkOPpQMLkXksawB1rLvmQPE4ykDaOsuxgSVspwDw1ptNdjNmYXo3KBj0AwYNBhvRq5Tgog== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=gmail.com smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=n5M9n6qDhAxsk+X7Y1Hj6xb3m+sIGhZYHr8FSdbaKfs=; b=PB04Gie6lvKLNlxFAxy/XkGfCtt1w6XTsguKzuJ20weMK6mrYJRx91d8GNUJItoqHYZ/qTmTc/b7fqRdXUJPkvAdZwMFXy/vnCTU/4ZDoUzZyeMfWKDmh0ZL4X/dklz32jCMPeV7gMyd568D/crBUGcAZrMrDy3QkAQ5msnzbG4= Received: from MN2PR15CA0042.namprd15.prod.outlook.com (2603:10b6:208:237::11) by DM4PR12MB8452.namprd12.prod.outlook.com (2603:10b6:8:184::18) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.92.13; Wed, 10 Jun 2026 12:28:07 +0000 Received: from BL02EPF0001A104.namprd05.prod.outlook.com (2603:10b6:208:237:cafe::8) by MN2PR15CA0042.outlook.office365.com (2603:10b6:208:237::11) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.21.113.12 via Frontend Transport; Wed, 10 Jun 2026 12:28:07 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=satlexmb07.amd.com; pr=C Received: from satlexmb07.amd.com (165.204.84.17) by BL02EPF0001A104.mail.protection.outlook.com (10.167.241.135) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.113.7 via Frontend Transport; Wed, 10 Jun 2026 12:28:07 +0000 Received: from kaveri.amd.com (10.180.168.240) by satlexmb07.amd.com (10.181.42.216) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.41; Wed, 10 Jun 2026 07:27:55 -0500 From: Shivank Garg To: CC: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , Subject: Date: Wed, 10 Jun 2026 12:26:54 +0000 Message-ID: <20260610122652.129407-3-shivankg@amd.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20260609161742.phipg7eid3yy4j7f@wrangler> References: <20260609161742.phipg7eid3yy4j7f@wrangler> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Content-Type: text/plain X-ClientProxiedBy: satlexmb08.amd.com (10.181.42.217) To satlexmb07.amd.com (10.181.42.216) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BL02EPF0001A104:EE_|DM4PR12MB8452:EE_ X-MS-Office365-Filtering-Correlation-Id: c3097c9d-f895-4089-a8c5-08dec6ebb9fb X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|23010399003|36860700016|7416014|376014|82310400026|1800799024|11063799006|56012099006|18002099003|6133799003|22082099003|3023799007; X-Microsoft-Antispam-Message-Info: /6CcQT15UMYu6eUWLSpzzgqEVKPI5+SIKYl7nl5iqTAXZHXUSYuntQlw8AsWMqVN6+z1QE6Fa3o2rSC2FjkW7gTihSoVo3CZufayrEm7bR7jFfEqoKH/xHYpXBObmHnIf4d+qMYYg+Iy478Utunxi0PSGjvTJ8HQwtj7eOYx+Ba9gGfcC7GBNdZa9UXIZFcx0hRnSxk+Ez/7TtH3VenikakgfnzT6Z/yE+kFY7G4hUwZNVMDExz5f8NpWBW/8Kd+SwSntsywEPyyWk+3E2SRaYSKB0FyeZkuSClqmXuD/oFnnWGtWNSQRaA7+E15xRFNccimFP/vSjYPKRPlT0KbcDIDilcdm5ggv8rStluKmcYBwMKzXwfV6gQt+oCA/Hfm6OAg4/lwKXrJw04y9pyXy9+H4snYLw8Enq8ZYi5szB8Uu1Umn5JS2vDGHCl8nV9jvZ0xqBBCSQVV+MtDhJYJgVbXEahOF1vYrrBeHcQBXJa8otCQTgsYoFQ9E7CGIjwS0tThETRjCJ5/j+1K9q7v/sSKmmOMu4wWBO7x5fz4WHhMGAl9KSihn6VBt9I8bi6kqvMnkWqA3OjG2S0b27u00ronn2hlIztmGtbkrca8WcPOiJUR7FQPWKqWj0cbew4/0e9q++6HdIURYJzdKwwjwI9X01KxVzsZ/Bi3nYblN2Ps1I2evt2Dbyxhc8vrxNfuDfa6QDm+4oWhYBY7aiPpikFY/NeJPHKuALuO4tjR7L8= X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:satlexmb07.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(23010399003)(36860700016)(7416014)(376014)(82310400026)(1800799024)(11063799006)(56012099006)(18002099003)(6133799003)(22082099003)(3023799007);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: ewXR1Fgfw5e/uuPGb0i5IPu7Jq4eUo/3unhIYhNP7BZ38qX1bcHmFk6M1nphsYFMwvb+GignrpkOUS0YGlOMwfC2PQtHelDN1BCSrw/wxqk+CTAl7mbJbjZ8efW5b2OC7UEJFfAOyn2Xuu+u8JLCX4A3Hs4W0WIjQaAZH9bS9ROvtmXCquD7dSDs+EcQ4IDH4QxaMFATWuovbDFWF8UKk1tkrMQ1QvLeV/2M7k/8HsGMG1erFn38cxIamrMB9fIall/5OXo9GpilYqGMJ4+WS+QrbFoxGkdAHGFZgj/HZQvj+jNnAOfXLRgVRsB8pnaQRiStX94l5m60swKZJOd0HhgzEN1g4QqiglGgcvR+LD3FGefkvbs3onNDuYeGqut4ASsiyr+D6mrliAvCWWnuBcD2j+tsLCCTtRoAyDr3C8x8PGOcYtSoA/PKYitNorHf X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 10 Jun 2026 12:28:07.2652 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: c3097c9d-f895-4089-a8c5-08dec6ebb9fb X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[satlexmb07.amd.com] X-MS-Exchange-CrossTenant-AuthSource: BL02EPF0001A104.namprd05.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM4PR12MB8452 ... > I'm still testing, but the initial implementation I wrote with > DMAEngine had too much overhead because of the sgtable allocations > and the conversion between kernel scatterlists to device descriptors. > So I entirely bypassed the DMAEngine API by directly passing the folios > lists to the driver. > > I know it depends on the use case. If you just want to offload with no > latency requirements, then DMAEngine is fine, but if the goal is to > achieve high bandwidth with minimal latency, then it's a problem. > > Another example, if you have to do several independent copies of 256 or > 512 4KiB pages in a short period of time, there will to much stress on > sgtable allocations. > > Another problem for low latency is DMA mapping. > > Anyway, I need to collect more numbers. I will try to share my insights > with idxd asap. Thanks, looking forward to those insights and numbers. An IDXD specific implementation is good for experimentation, but for upstream path, I think this would be hard to maintain and add duplicate logic. The cleanest approch is the DMA_MEMCPY_SG API. So, a single offload driver can drive any engine that implements it. dmaengine_prep_dma_memcpy_sg() submits a whole src/dst scatterlist as one transaction, which cuts the per-descriptor setup overhead that dominates for 4KB pages. I've added a patch for dmaengine_prep_dma_memcpy_sg(), Could you look into wiring up device_prep_dma_memcpy_sg hook in the IDXD? This will keep it generic and address the bandwidth/latency problem for small transfers. Best Regards, Shivank --- >From d60e4dcd58552cbc6c7ecee37df9e216ab0c5a4f Mon Sep 17 00:00:00 2001 From: Shivank Garg Date: Sun, 17 May 2026 18:19:10 +0000 Subject: [PATCH] dmaengine: add DMA_MEMCPY_SG transaction type Currently, a client that wants to copy N (src, dst, len) tuples call dmaengine_prep_dma_memcpy() N times, allocating N independent dma_async_tx_descriptor. The provider has to treat them as independent transactions, even when the underlying hardware can program them as one descriptor group with a single completion. This overheads are dominant for page size like 4KB. To reduce this overheads, add DMA_MEMCPY_SG and the matching device_prep_dma_memcpy_sg() callback taking paired src and dst scatterlists, plus the dmaengine_prep_dma_memcpy_sg() inline wrapper. The provider walks both lists in lockstep and retires the whole batch as one async tx. This API was removed by commit 0cae04373b77 ("dmaengine: remove DMA_MEMCPY_SG once again") for lack of in-tree users. The user this time is the page-migration copy offload driver. Suggested-by: Vinod Koul Signed-off-by: Shivank Garg --- .../driver-api/dmaengine/provider.rst | 7 ++++ drivers/dma/dmaengine.c | 1 + include/linux/dmaengine.h | 36 +++++++++++++++++++ 3 files changed, 44 insertions(+) diff --git a/Documentation/driver-api/dmaengine/provider.rst b/Documentation/driver-api/dmaengine/provider.rst index f4ed98f701c9..fc3cab78e61a 100644 --- a/Documentation/driver-api/dmaengine/provider.rst +++ b/Documentation/driver-api/dmaengine/provider.rst @@ -175,6 +175,13 @@ Currently, the types available are: ``glReadPixels()``, which might require a verbatim copy of a huge framebuffer from local device memory onto host memory. +- DMA_MEMCPY_SG + + - The device is able to do memory to memory scatter-gather transfers. + + - This takes pair of src and dst scatterlists and retires the whole batch + as one async tx. + - DMA_XOR - The device is able to perform XOR operations on memory areas diff --git a/drivers/dma/dmaengine.c b/drivers/dma/dmaengine.c index 405bd2fbb4a3..665e00a96c8f 100644 --- a/drivers/dma/dmaengine.c +++ b/drivers/dma/dmaengine.c @@ -1197,6 +1197,7 @@ int dma_async_device_register(struct dma_device *device) } CHECK_CAP(dma_memcpy, DMA_MEMCPY); + CHECK_CAP(dma_memcpy_sg, DMA_MEMCPY_SG); CHECK_CAP(dma_xor, DMA_XOR); CHECK_CAP(dma_xor_val, DMA_XOR_VAL); CHECK_CAP(dma_pq, DMA_PQ); diff --git a/include/linux/dmaengine.h b/include/linux/dmaengine.h index b3d251c9734e..9378e7a7803a 100644 --- a/include/linux/dmaengine.h +++ b/include/linux/dmaengine.h @@ -50,6 +50,7 @@ enum dma_status { */ enum dma_transaction_type { DMA_MEMCPY, + DMA_MEMCPY_SG, DMA_XOR, DMA_PQ, DMA_XOR_VAL, @@ -824,6 +825,7 @@ struct dma_filter { * @device_router_config: optional callback for DMA router configuration * @device_free_chan_resources: release DMA channel's resources * @device_prep_dma_memcpy: prepares a memcpy operation + * @device_prep_dma_memcpy_sg: prepares a memcpy operation over a scatter list * @device_prep_dma_xor: prepares a xor operation * @device_prep_dma_xor_val: prepares a xor validation operation * @device_prep_dma_pq: prepares a pq operation @@ -903,6 +905,11 @@ struct dma_device { struct dma_async_tx_descriptor *(*device_prep_dma_memcpy)( struct dma_chan *chan, dma_addr_t dst, dma_addr_t src, size_t len, unsigned long flags); + struct dma_async_tx_descriptor *(*device_prep_dma_memcpy_sg)( + struct dma_chan *chan, + struct scatterlist *dst_sg, unsigned int dst_nents, + struct scatterlist *src_sg, unsigned int src_nents, + unsigned long flags); struct dma_async_tx_descriptor *(*device_prep_dma_xor)( struct dma_chan *chan, dma_addr_t dst, dma_addr_t *src, unsigned int src_cnt, size_t len, unsigned long flags); @@ -1091,6 +1098,35 @@ static inline struct dma_async_tx_descriptor *dmaengine_prep_dma_memcpy( len, flags); } +/** + * dmaengine_prep_dma_memcpy_sg - Prepare a scatter-gather memcpy + * @chan: The channel to prepare on + * @dst_sg: Destination scatterlist + * @dst_nents: Number of mapped entries in @dst_sg + * @src_sg: Source scatterlist + * @src_nents: Number of mapped entries in @src_sg + * @flags: Dmaengine flags (e.g. DMA_PREP_INTERRUPT, DMA_CTRL_ACK) + * + * Submit a batch of memcpy operations described by two scatterlists as + * a single async transaction. All segments retire as one tx with one + * completion. + * + * Returns NULL if the channel does not implement this operation. + */ +static inline struct dma_async_tx_descriptor *dmaengine_prep_dma_memcpy_sg( + struct dma_chan *chan, + struct scatterlist *dst_sg, unsigned int dst_nents, + struct scatterlist *src_sg, unsigned int src_nents, + unsigned long flags) +{ + if (!chan || !chan->device || + !chan->device->device_prep_dma_memcpy_sg) + return NULL; + + return chan->device->device_prep_dma_memcpy_sg(chan, + dst_sg, dst_nents, src_sg, src_nents, flags); +} + static inline bool dmaengine_is_metadata_mode_supported(struct dma_chan *chan, enum dma_desc_metadata_mode mode) { -- 2.43.0