From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-qv1-f47.google.com (mail-qv1-f47.google.com [209.85.219.47]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B9E402D05E for ; Sat, 28 Dec 2024 03:38:49 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.219.47 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1735357132; cv=none; b=fsBRTCVlKihc1p+SCt68flhQ+uixZ1Cgr+tR6rQgTM0KZ2fCumPdqSQx3yToeboilaqJWBbb/0a/e2n16cotJhTiXkzDO0SmEYUaDOUlIFGiOXaqETVeWzAuel5IBTB5UvglunjbpfxikihVNJidtU1VPzAgurftb3QWMJJHT3M= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1735357132; c=relaxed/simple; bh=fUGrIgvsS+O5yTNtO6d1hTXOW9cVexNbb8NzKZYFVd0=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=sHCjnEuK1H0Gr5RqlrM6mwD7N7t6pyBoiuxjBx+Cg50lpAtqQN/U4Vo5c2e95E3Ei9zUxfrZR+g+biniGBrX+DXizRr0/qg+QsVZskuPRzEU7RM/EbyfroeN5GGpwOkfRQCXj0/TtDJExrXpccKA1gc5XaugMXoC5QSs+5R6Yss= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=gourry.net; spf=pass smtp.mailfrom=gourry.net; dkim=pass (2048-bit key) header.d=gourry.net header.i=@gourry.net header.b=ngk2C6sn; arc=none smtp.client-ip=209.85.219.47 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=gourry.net Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gourry.net Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gourry.net header.i=@gourry.net header.b="ngk2C6sn" Received: by mail-qv1-f47.google.com with SMTP id 6a1803df08f44-6dd049b5428so63187056d6.2 for ; Fri, 27 Dec 2024 19:38:49 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gourry.net; s=google; t=1735357128; x=1735961928; darn=vger.kernel.org; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=ZHoWyMeD0simD6rzZZ407DZSH4sL8HtAX2h8hHCq/do=; b=ngk2C6snCaxnXFCwDO7So6rAZiMEufxyoawOYhFTCJ/b0VHh9aS8HhzyybY8QB5Ymp 4W+QU/n2nXDF57hSd2XZY1lxbuXAvzEdgkUg/L2RvS+TTSx8Nkh8UytZPAHwmNd+kZC7 VmJauAnw2TrXzGBqEAI2Te9smcr+IY8jUDoN6JI9auEnPWeGZrnM2gubCjC50KmRfAOJ WEiQljd2pFz4gkkI138SG2eOqaRTzZr0CCskL6y3PSQ9SQu1jJHFlN7Cf7LSinlzcOSL UHtC+FizTVLNcNlP+USnYqvkBCn07kaG2aBEN6YvlnKFogsbZmFxvf+Y4l7Tzh7C3unK rpmw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1735357128; x=1735961928; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=ZHoWyMeD0simD6rzZZ407DZSH4sL8HtAX2h8hHCq/do=; b=Wf9hyqSImMbscT1cgvJ/EllIzYAXv1hevoUUurPHj6/wE7d558r9xOA6ZKFg+ihbmk YTWhMCZwC/ycHz5C28iOu7kd9FJ+KuLnrwtJkqpD6r7FR1IuJT8F77wERHlPDymGIZZK 4LPvHuh1uz/OFeKnvYGRJivymCh+Ckgpp9pCkr8HKQw92u8He4nk1nyiASQWRLu+qvvF EFURwKegzOKRxRxUWaOWoWEX8iDCw6j3JmYNZOhWaG1d7xkS6lAwyj3Ap7UbChcfot6K s05JN0UXa4EZ8EhWbmvFpPXdOnsO2QFpnbJFPcEilNz6H02EOthbuisT9gl1P/P9B8cn EwJQ== X-Forwarded-Encrypted: i=1; AJvYcCWQeVak6qw0A8QVGwu/AqmAEL6nKZfcz3tgiSYfsdNqDRRWs/cL/dYUReLpRYtf6jWrX8ywMAI9WoPgYfE=@vger.kernel.org X-Gm-Message-State: AOJu0YyrjPst2dkg+IUTy0Atmm0PbMS9CMjTBbhVILQPm11XWgWBkFyO bpX6Hb/zPNcXsLOLa1ndUBqtFPI20oWcI6TAbk3kxz19GaCPH14QWfKvqxHLxpyGm+QYsrBWs5F 6 X-Gm-Gg: ASbGncv0wM10/dudsFYCRh+UotK70WvXlacBoDypAdtWQfHsI/uzHaZyGt4PLjdUu9P i8x9U6BqBEGrO79FgBxNrLVC6/teNog9aVLsCvXtdfVJc2wp2TuVTM1SvlCgJzF/hVS/3RVJRTW /5BxTjC3fnqWUVSGSPHrcVE+8rd7sx4xAI1i773doGhHWaq8volDvgyTAvEmSjqeAyh/57VK25l 6FNQrGoc89JAK1hYqBQJdMXM1B8xCmuN/3u0EhMUTKtXZURFsRjAjZfIsgkAI1MGhW0sA9u0gtH I9VqFwCXZiOwqVmUy8K7ZAwcC2cStBuirPbH9Oc= X-Google-Smtp-Source: AGHT+IGKz6QK+/uf1/3a6peL/xApYCX6a2P3FtsHH4J1utqrJ1T20aaf3i7Y4eSSTMB1ws2RSpp0pQ== X-Received: by 2002:a05:6214:19c2:b0:6d8:d5f6:8c52 with SMTP id 6a1803df08f44-6dd2339d943mr512258066d6.38.1735357128581; Fri, 27 Dec 2024 19:38:48 -0800 (PST) Received: from gourry-fedora-PF4VCD3F (pool-173-79-56-208.washdc.fios.verizon.net. [173.79.56.208]) by smtp.gmail.com with ESMTPSA id 6a1803df08f44-6dd18110112sm84089166d6.33.2024.12.27.19.38.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 27 Dec 2024 19:38:47 -0800 (PST) Date: Fri, 27 Dec 2024 22:38:45 -0500 From: Gregory Price To: "Huang, Ying" Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, nehagholkar@meta.com, abhishekd@meta.com, kernel-team@meta.com, david@redhat.com, nphamcs@gmail.com, akpm@linux-foundation.org, hannes@cmpxchg.org, kbusch@meta.com Subject: Re: [RFC v2 PATCH 0/5] Promotion of Unmapped Page Cache Folios. Message-ID: References: <20241210213744.2968-1-gourry@gourry.net> <87o715r4vn.fsf@DESKTOP-5N7EMDA> <87wmfsi47b.fsf@DESKTOP-5N7EMDA> <87v7v5g99x.fsf@DESKTOP-5N7EMDA> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: On Fri, Dec 27, 2024 at 02:09:50PM -0500, Gregory Price wrote: > On Fri, Dec 27, 2024 at 10:40:36AM -0500, Gregory Price wrote: just adding some follow-up data test is essentially membind(1) - node1 is cxl read() - filecache is initialized on cxl set_mempolicy(MPOL_DEFAULT) - allow migrations while true: start = time() read() print(time()-start) // external events cause migration/drop cache while running baseline: .93-1s/read() from cxl: ~1.15-1.2s/read() So we are seeing anywhere from 20-25% overhead from the filecache living on CXL right out of the box. At least we have good clear signal, right? tests: echo 3 > drop_cache - filecache refills into node 1 result => ~.95-1s/read() we return back to the baseline, which is expected enable promotion - numactl shows promotion occurs result => ~1.15-1.2s/read() No effect?! Even offlining the dax devices does nothing. enable promotion, wait for it to complete, drop cache after promotion => 1.15-1.2s/read after drop cache => .95-1s/read() Back to baseline! This seems to imply that the overhead we're seeing from read() even when filecache is on the remote node isn't actually related to the memory speed, but instead likely related to some kind of stale metadata in the filesystem or filecache layers. This is going to take me a bit to figure out. I need to isolate the filesystem influence (we are using btrfs, i want to make sure this behavior is consistent on other file systems). ~Gregory