From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-qv1-f54.google.com (mail-qv1-f54.google.com [209.85.219.54]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3BEB51F2C59 for ; Sat, 21 Dec 2024 14:49:00 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.219.54 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1734792543; cv=none; b=QdQ8PVpdBUKTphTVZqRlcju/mzaLtKF4q6QtwxBC14LyNYGEcWNxWexUekDVoMNU2WauGxhqN1qng0YsmYEWy0oKOeDLis9e+glfozchZiRg2fGVxEIw4joiqh9qGRd9C7m0LXMwR1cuk43pHd3+xQcUbTJPdIkzK7rJ0EiovXA= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1734792543; c=relaxed/simple; bh=meMFFqNoiSi2RraG6xMc9ivdpcc9wPBJQ9fuHyJi4lc=; h=From:Date:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=dBHl1S5Kb5ngd4jWw+WMDUQW/mw5NKVJU2MC1M9IvKRB6bq1zQDavBvVICH18FiVHt8IMlskT1e4nuFgKdGJw2zz3xiDzM+/Dg+IfGisx1iNCOPhF5fKZudAVjyVo8Vx5DOtUr0c2ewxYcCrAdhg6NlEGwr6zzfrhgdfFdauWLo= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=gourry.net; spf=pass smtp.mailfrom=gourry.net; dkim=pass (2048-bit key) header.d=gourry.net header.i=@gourry.net header.b=rxRkk/4R; arc=none smtp.client-ip=209.85.219.54 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=gourry.net Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gourry.net Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gourry.net header.i=@gourry.net header.b="rxRkk/4R" Received: by mail-qv1-f54.google.com with SMTP id 6a1803df08f44-6dd15d03eacso26624476d6.0 for ; Sat, 21 Dec 2024 06:49:00 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gourry.net; s=google; t=1734792540; x=1735397340; darn=vger.kernel.org; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:date:from:from:to:cc:subject:date:message-id:reply-to; bh=5i66wyNJ4GlJ5wDCHDiVB8aDJ2Bcpwt+j2OanevERxg=; b=rxRkk/4RcSB6VsHFkzfSqJ9FXqkKxwF2PH5SIQEn+5z4EYUHC3TCV0Uj5pBy3h/AvH W/vdumugR41PSrN/x/AHctAx6siv129pk9a2WZNYfJKkga9ljoOMH29C5KX4Qjc+mAtE u/68xdpXPwfYw65mo9W07Utr8bOH9ALW8bVz6sMhm05AvX4Lfa18CeUJ2r6hdeBxIVUv Rwmqi/sjhjxHk5Q9piiGe7vTva/Yp8jcRW2t5OCsiECrpj1GzR2H3OhQcDamHQh4WtNL ZZVXQHXniQcJA3veVV2/zkOdoyIIo/naqqvnhkLHRe6iricQiSOcmqq3KOfm2vH3zR6c MUVg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1734792540; x=1735397340; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:date:from:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=5i66wyNJ4GlJ5wDCHDiVB8aDJ2Bcpwt+j2OanevERxg=; b=lvPipHi7W2RwDD0Wb7feosz0W7lJNg98pP5KgIOC+/qD0VaGRBMpfweT7We48xHgbt mcSvPVif9j+E4ceSUj+qoN129fFZjWCDT3b3hPocSRDGm6MUTWIeIyDBddvi+s4sUmLw IHG1iVF9vEVrk8yVhMWjNJWiPjiyTAaqjomhUAy6yZ2HGNy+5cLsIuFWImXdxBVxi9IV vggou3OnVRlhwOjyUxsEGxDchsm7es4pGV0s8oVXaD5xqCDZXqSfeO0FESiGzf7koDfO FS0vAMRou0wSnhzprMnoL+eLDA0BIMT0Lr7e5APHHg9pphZWj/yb9EI5tCmXA4bzxLJm X33w== X-Forwarded-Encrypted: i=1; AJvYcCUSqcmCu2a80kGlpr0Kr1zbmm6u6dow5XDoiUb1BtAoyawA+vGmpJQsXosjlvJUhh+DBRhZFAQosxVd7DQ=@vger.kernel.org X-Gm-Message-State: AOJu0Yy8j4IKnfIevELTGEpxgaF7OKl1sQ8mDvEUmxFBsuULyxXItCYf osmt8dP4do1XCn2Phk5u0UQ5jI4OspkCJrDp5ADkDY1ez465RDaLmrii/2oucU0= X-Gm-Gg: ASbGncu3rvKaPINh569wyyKffF+22S3H4mS7rcG4GVASecR/UQyav5itApDpZc/ry4V xuCo+jiebqZT+7WM3WW9aW1CLgEhIGEgLNvk9xmtg7AvsDfamgjW/VFyPtC8B+BF1K9Lh1yrq5C fZKGp9QrKCm7aglvmgVWX4ETMaifGQSGwBuv4XTJnD1uRPD1GIYqxQU9DmNMgWU8spJTvhYji51 PXcJjX45WV00n+JueYBGwicHHvgy7DmvJEXx+hng45W6Eist9FsXHQegcuBEapMOKk6O6GPREw6 IzZ8fXPBKBCvJ8MkydkS27Hv4luH7xWt/6lgkUefKrCHffd3Cd2AxwY= X-Google-Smtp-Source: AGHT+IE08WY46z05aeVGGZfbqV6PpIY33PC1rYA5PmRSYOOUtDhRdu7FjN8tcxXoII4nrAfaU8n7TA== X-Received: by 2002:a05:6214:428c:b0:6d8:7a85:59ca with SMTP id 6a1803df08f44-6dd2360e187mr121468676d6.16.1734792540122; Sat, 21 Dec 2024 06:49:00 -0800 (PST) Received: from gourry-fedora-PF4VCD3F (pool-173-79-56-208.washdc.fios.verizon.net. [173.79.56.208]) by smtp.gmail.com with ESMTPSA id 6a1803df08f44-6dd1810d5casm26920556d6.39.2024.12.21.06.48.58 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 21 Dec 2024 06:48:59 -0800 (PST) From: Gregory Price X-Google-Original-From: Gregory Price Date: Sat, 21 Dec 2024 09:48:57 -0500 To: "Huang, Ying" Cc: Gregory Price , linux-mm@kvack.org, linux-kernel@vger.kernel.org, nehagholkar@meta.com, abhishekd@meta.com, kernel-team@meta.com, david@redhat.com, nphamcs@gmail.com, akpm@linux-foundation.org, hannes@cmpxchg.org, kbusch@meta.com Subject: Re: [RFC v2 PATCH 0/5] Promotion of Unmapped Page Cache Folios. Message-ID: References: <20241210213744.2968-1-gourry@gourry.net> <87o715r4vn.fsf@DESKTOP-5N7EMDA> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <87o715r4vn.fsf@DESKTOP-5N7EMDA> On Sat, Dec 21, 2024 at 01:18:04PM +0800, Huang, Ying wrote: > Gregory Price writes: > > > > > Single-reader DRAM: ~16.0-16.4s > > Single-reader CXL (after demotion): ~16.8-17s > > The difference is trivial. This makes me thought that why we need this > patchset? > That's 3-6% performance in this contrived case. We're working to testing a real workload we know suffers from this problem as it is long-running. Should be early in the new year hopefully. > > Next we turned promotion on with only a single reader running. > > > > Before promotions: > > Node 0 MemFree: 636478112 kB > > Node 0 FilePages: 59009156 kB > > Node 1 MemFree: 250336004 kB > > Node 1 FilePages: 14979628 kB > > Why are there some many file pages on node 1 even if there're a lot of > free pages on node 0? You moved some file pages from node 0 to node 1? > This was explicit and explained in the test notes: First we ran with promotion disabled to show consistent overhead as a result of forcing a file out to CXL memory. We first ran a single reader to see uncontended performance, launched many readers to force demotions, then dropped back to a single reader to observe. The goal here was to simply demonstrate functionality and stability. > > After promotions: > > Node 0 MemFree: 632267268 kB > > Node 0 FilePages: 72204968 kB > > Node 1 MemFree: 262567056 kB > > Node 1 FilePages: 2918768 kB > > > > Single-reader (after_promotion): ~16.5s This represents a 2.5-6% speedup depending on the spread. > > > > numa_migrate_prep: 93 - time(3969867917) count(42576860) > > migrate_misplaced_folio_prepare: 491 - time(3433174319) count(6985523) > > migrate_misplaced_folio: 1635 - time(11426529980) count(6985523) > > > > Thoughts on a good throttling heuristic would be appreciated here. > > We do have a throttle mechanism already, for example, you can used > > $ echo 100 > /proc/sys/kernel/numa_balancing_promote_rate_limit_MBps > > to rate limit the promotion throughput under 100 MB/s for each DRAM > node. > Can easily piggyback on that, just wasn't sure if overloading it was an acceptable idea. Although since that promotion rate limit is also per-task (as far as I know, will need to read into it a bit more) this is probably fine. ~Gregory