From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3A0B5C001B0 for ; Wed, 26 Jul 2023 10:14:37 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232844AbjGZKOf (ORCPT ); Wed, 26 Jul 2023 06:14:35 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48392 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230013AbjGZKOc (ORCPT ); Wed, 26 Jul 2023 06:14:32 -0400 Received: from mblankhorst.nl (lankhorst.se [141.105.120.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id ABF871995 for ; Wed, 26 Jul 2023 03:14:30 -0700 (PDT) Message-ID: Date: Wed, 26 Jul 2023 12:14:24 +0200 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Thunderbird/102.11.0 Subject: Re: [PATCH 16/17] cgroup/drm: Expose memory stats Content-Language: en-US To: Tejun Heo , Tvrtko Ursulin Cc: Intel-gfx@lists.freedesktop.org, dri-devel@lists.freedesktop.org, cgroups@vger.kernel.org, linux-kernel@vger.kernel.org, Johannes Weiner , Zefan Li , Dave Airlie , Daniel Vetter , Rob Clark , =?UTF-8?Q?St=c3=a9phane_Marchesin?= , "T . J . Mercier" , Kenny.Ho@amd.com, =?UTF-8?Q?Christian_K=c3=b6nig?= , Brian Welty , Tvrtko Ursulin , Eero Tamminen References: <20230712114605.519432-1-tvrtko.ursulin@linux.intel.com> <20230712114605.519432-17-tvrtko.ursulin@linux.intel.com> From: Maarten Lankhorst In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Hey, On 2023-07-22 00:21, Tejun Heo wrote: > On Wed, Jul 12, 2023 at 12:46:04PM +0100, Tvrtko Ursulin wrote: >> $ cat drm.memory.stat >> card0 region=system total=12898304 shared=0 active=0 resident=12111872 purgeable=167936 >> card0 region=stolen-system total=0 shared=0 active=0 resident=0 purgeable=0 >> >> Data is generated on demand for simplicty of implementation ie. no running >> totals are kept or accounted during migrations and such. Various >> optimisations such as cheaper collection of data are possible but >> deliberately left out for now. >> >> Overall, the feature is deemed to be useful to container orchestration >> software (and manual management). >> >> Limits, either soft or hard, are not envisaged to be implemented on top of >> this approach due on demand nature of collecting the stats. > > So, yeah, if you want to add memory controls, we better think through how > the fd ownership migration should work. I've taken a look at the series, since I have been working on cgroup memory eviction. The scheduling stuff will work for i915, since it has a purely software execlist scheduler, but I don't think it will work for GuC (firmware) scheduling or other drivers that use the generic drm scheduler. For something like this, you would probably want it to work inside the drm scheduler first. Presumably, this can be done by setting a weight on each runqueue, and perhaps adding a callback to update one for a running queue. Calculating the weights hierarchically might be fun.. I have taken a look at how the rest of cgroup controllers change ownership when moved to a different cgroup, and the answer was: not at all. If we attempt to create the scheduler controls only on the first time the fd is used, you could probably get rid of all the tracking. This can be done very easily with the drm scheduler. WRT memory, I think the consensus is to track system memory like normal memory. Stolen memory doesn't need to be tracked. It's kernel only memory, used for internal bookkeeping only. The only time userspace can directly manipulate stolen memory, is by mapping the pinned initial framebuffer to its own address space. The only allocation it can do is when a framebuffer is displayed, and framebuffer compression creates some stolen memory. Userspace is not aware of this though, and has no way to manipulate those contents. Cheers, ~Maarten