From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from out30-112.freemail.mail.aliyun.com (out30-112.freemail.mail.aliyun.com [115.124.30.112]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C04BC34B1B0 for ; Tue, 14 Jul 2026 11:29:50 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=115.124.30.112 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784028593; cv=none; b=Ep4EfUXrp3qwwPeFeid0ZPgyHXrwloUsGxLT81rE0nnqf6x9Ae3I5a4Eh6PJv+zGMNdb9wj+3r3FOYtB5q6SgKgjf3vku7vO5SjXoLiuBXg89qIZOKN7LDcarezZgKRLzYm9p3q9M5GwoJnaUPm40woOimVXCWUtOvW0gWdgCvg= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784028593; c=relaxed/simple; bh=rGvDtMeIzjj+m77k/TTfKxsorRhTVJk72L/W0Tku/x8=; h=From:To:Cc:Subject:In-Reply-To:References:Date:Message-ID: MIME-Version:Content-Type; b=YoiOpJNUaRMBT+CRi7gTIqKm2drfYj8JilUFnCLuiD1+E126EwCtXxbOJujeQLQQkAr5TRM1K0uHhoVUKc0mImMYk22BYtoIr7HwkNP2WNkLaQ5whCu+HnfUE8LZNkqjVDIJspl1yLxB4thhjcr1kRw1/9TQplzNTc8tdsxpyvw= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.alibaba.com; spf=pass smtp.mailfrom=linux.alibaba.com; dkim=pass (1024-bit key) header.d=linux.alibaba.com header.i=@linux.alibaba.com header.b=Z3IQc0cG; arc=none smtp.client-ip=115.124.30.112 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.alibaba.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.alibaba.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.alibaba.com header.i=@linux.alibaba.com header.b="Z3IQc0cG" DKIM-Signature:v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.alibaba.com; s=default; t=1784028586; h=From:To:Subject:Date:Message-ID:MIME-Version:Content-Type; bh=bJOi5rEy563Y/DC2Gpf7vppljoWhPl3nNbGQu7+o5Dw=; b=Z3IQc0cGH47wFlxkrSStcZMS0uxWVc/HjG/QcbfbETGYM/cGhpj8vNqWEoWehAJ3DIOaqD6/s3JiQDIsqujIlW45c6mrd+24xOg88KrK1E8AnCj2CO8hZBgtn8GDs/03jOHgJ4pL3yFn5V3CwoiQY489pO2dY6S8gfSvcTiuL4Y= X-Alimail-AntiSpam:AC=PASS;BC=-1|-1;BR=01201311R201e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=maildocker-contentspam033037009110;MF=ying.huang@linux.alibaba.com;NM=1;PH=DS;RN=46;SR=0;TI=SMTPD_---0X74J2yL_1784028582; Received: from DESKTOP-5N7EMDA(mailfrom:ying.huang@linux.alibaba.com fp:SMTPD_---0X74J2yL_1784028582 cluster:ay36) by smtp.aliyun-inc.com; Tue, 14 Jul 2026 19:29:43 +0800 From: "Huang, Ying" To: Shivank Garg Cc: Andrew Morton , David Hildenbrand , Zi Yan , Matthew Brost , Joshua Hahn , Rakie Kim , Byungchul Park , Gregory Price , "Alistair Popple" , Lorenzo Stoakes , "Liam R. Howlett" , Vlastimil Babka , "Mike Rapoport" , Suren Baghdasaryan , "Michal Hocko" , Karim Manaouil , Frank van der Linden , Teja Vojjala , Pravin Tamkhane , Kinsey Ho , Wei Xu , Matthew Wilcox , Davidlohr Bueso , Vinod Koul , Bharata B Rao , SeongJae Park , David Rientjes , Xuezheng Chu , "Yiannis Nikolakopoulos" , Dave Hansen , Johannes Weiner , John Hubbard , Peter Xu , Rik van Riel , Shakeel Butt , Tejun Heo , Fan Ni , Jonathan Cameron , Aneesh Kumar K.V , Nathan Lynch , Frank Li , Dan Williams , , , Mike Day Subject: Re: [PATCH RFC v6 0/5] Accelerate page migration with batch copying and hardware offload In-Reply-To: <20260630-shivank-batch-migrate-offload-v6-0-da95d7e8b8a2@amd.com> (Shivank Garg's message of "Tue, 30 Jun 2026 07:28:36 +0000") References: <20260630-shivank-batch-migrate-offload-v6-0-da95d7e8b8a2@amd.com> Date: Tue, 14 Jul 2026 19:29:40 +0800 Message-ID: <87wlux6c7v.fsf@DESKTOP-5N7EMDA> User-Agent: Gnus/5.13 (Gnus v5.13) Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Hi, Garg, Thanks for updated patch. Shivank Garg writes: [snip] > > PERFORMANCE RESULTS: > -------------------- > > AMD EPYC 7713 (Zen 3), 2 sockets, 32 cores, SMT on,=20 > 1 NUMA node per socket, 256 GB/node, v7.2-rc1, DVFS=3DPerformance, PTDMA > (16 DMA channels). > > Benchmark: move_pages() syscall to move pages between two NUMA nodes. > > 1). Moving different sized folios such that total transfer size is consta= nt > (1GB), with different number of DMA channels. Throughput in GB/s. > > a. Baseline (vanilla kernel, single-threaded, serial folio_copy): > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D > 4K | 16K | 64K | 256K | 1M | 2M = | > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D > 3.28=C2=B10.14 | 4.98=C2=B10.18 | 6.19=C2=B10.08 | 6.77=C2=B10.08 | = 7.02=C2=B10.11 | 10.80=C2=B10.13 | > > b. DMA offload (Patched Kernel, dcbm driver, N DMA channels): > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > N channel| 4K | 16K | 64K | 256K | 1M = | 2M | > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > 1 | 2.38=C2=B10.17 | 2.77=C2=B10.03 | 3.21=C2=B10.03 | 5.00=C2= =B10.02 | 5.09=C2=B10.64 | 12.62=C2=B10.07 | > 2 | 2.87=C2=B10.11 | 4.06=C2=B10.05 | 5.09=C2=B10.04 | 6.97=C2= =B10.08 | 8.43=C2=B10.06 | 14.32=C2=B10.10 | > 4 | 3.32=C2=B10.07 | 5.30=C2=B10.06 | 7.21=C2=B10.09 | 9.69=C2= =B10.15 | 11.36=C2=B10.13 | 26.98=C2=B10.19 | > 8 | 3.68=C2=B10.09 | 6.28=C2=B10.10 | 9.16=C2=B10.13 | 12.05= =C2=B10.16 | 15.33=C2=B12.80 | 46.06=C2=B10.55 | > 12 | 3.83=C2=B10.05 | 6.65=C2=B10.17 | 10.00=C2=B10.16 | 12.98= =C2=B10.18 | 15.87=C2=B10.19 | 61.31=C2=B11.28 | > 16 | 3.94=C2=B10.09 | 6.78=C2=B10.10 | 10.48=C2=B10.13 | 13.48= =C2=B10.20 | 16.90=C2=B10.24 | 65.06=C2=B12.46 | > > 2). First-folio latency: custom tracepoints (in migrate_pages_batch enter= /exit, > migrate_folio_done) measure latency per migrate_pages_batch() call. > > Throughput (GB/s) and first-folio latency (us), median of 10 runs. > > a. Vanilla Kernel: > > NR_MAX_BATCHED_MIGRATION upstream default value is 512. > --- Order 0 (4K folios) --- --- Order 9 (2M folios) --- > n vanilla/cpu n vanilla/cpu > (folios) GB/s | first(us) (folios) GB/s | first(us) > -------------------------- -------------------------- > 1 0.03 | 24 1 6.86 | 204 > 4 0.13 | 30 4 8.68 | 191 > 8 0.27 | 27 8 7.92 | 207 > 16 0.43 | 34 16 6.77 | 234 > 64 1.12 | 51 64 10.44 | 179 > 256 1.67 | 166 256 10.43 | 181 > 512 1.98 | 255 512 10.55 | 179 > 2048 2.38 | 233 > 4096 2.42 | 168 > 16384 2.72 | 167 > 65536 3.00 | 156 > 262144 3.10 | 151 > > b. Patched kernel: > N =3D NR_MAX_BATCHED_MIGRATION (in pages), Total migrated data fixed at > 1 GB. Change N with knob (just for testing) to measure impact of > different max batched size. > > --- ORDER 0 (4K folios) --- > > N offload/dma1 offload/dma4 offload/dma16 > GB/s | first(us) GB/s | first(us) GB/s | first(u= s) > ------------------------------------------------------------------------ > 512 2.21 | 628 3.29 | 275 3.25 | 245 > 1024 2.06 | 1271 3.21 | 601 3.36 | 518 > 2048 2.02 | 2646 3.00 | 1388 3.20 | 1110 > 4096 2.08 | 4832 3.17 | 2514 3.41 | 2175 > 8192 2.16 | 9253 3.14 | 4839 3.62 | 3592 > 16384 2.24 | 17543 3.23 | 9680 3.58 | 7144 > 32768 2.22 | 36408 3.26 | 19301 3.67 | 14524 > 65536 2.12 | 82572 3.24 | 38091 3.62 | 29835 > 131072 2.08 | 153669 3.17 | 79744 3.48 | 62157 > 262144 2.05 | 332297 2.97 | 175315 3.33 | 134774 > > --- ORDER 9 (2M folios) --- > > N offload/dma1 offload/dma4 offload/dma16 > GB/s | first(us) GB/s | first(us) GB/s | first(u= s) > ------------------------------------------------------------------------ > 512 11.74 | 160 11.71 | 160 11.75 | 159 > 1024 12.18 | 310 13.82 | 274 13.76 | 275 > 2048 12.39 | 612 25.55 | 290 25.69 | 289 > 4096 12.54 | 1211 26.25 | 564 42.36 | 334 > 8192 12.54 | 2421 26.82 | 1111 51.85 | 485 > 16384 12.61 | 4824 26.91 | 2209 54.26 | 925 > 32768 12.62 | 9652 27.04 | 4404 54.72 | 1942 > 65536 12.64 | 19287 26.95 | 8835 57.30 | 3535 > 131072 12.64 | 38824 26.95 | 17900 58.58 | 7747 > 262144 12.66 | 77610 26.95 | 35743 66.31 | 13801 > > OPEN QUESTION: > -------------- > > The best batch size depends on the hardware, and bigger isn't always bett= er. > NR_MAX_BATCHED_MIGRATION decides how many pages we move at once. > Higher batch size can help amortize the setup cost of migrator but > increases the first-folio latency (the folio is inaccessible for this > window). Yes. This is an important parameter for the batched migration. We really need more information from people who have workload information to make a decision. Can you try to reach them? Also, can we find a sweet point where we can archive higher throughput without increasing latency too much (e.g., < 1ms)? > Goals could be workload dependent, e.g. higher throughput versus > same throughput under a bounded latency. > > Should this be tunable to accommodate different hardware and goals? [snip] --- Best Regards, Huang, Ying