From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-pl1-f173.google.com (mail-pl1-f173.google.com [209.85.214.173]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7B17E4D5AB for ; Mon, 20 Jan 2025 05:48:48 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.173 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1737352130; cv=none; b=X4XK3o0Ririhae3SJqdQuLDw8W/Jq7ld6z0Ddjc5NgAM4QaIwb7KXeTeBSv3tUE0SMdw8ftSeIa1adK2mrZUNceyNOb6vYQxXWdLMHyRYwgRpkpON4ex9pi81bsF2v7iDhP1u+M1f1gnzZOdf+EAfYWlHcbfE79rYYEZqrbJVeo= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1737352130; c=relaxed/simple; bh=KuFk3XZ9QLoXRTqWHp732llNYHV3q4Lx3PSzSA3WoQE=; h=Message-ID:Date:MIME-Version:Subject:To:Cc:References:From: In-Reply-To:Content-Type; b=UxO+waRvQ4PYa1kKDwPw2neQuEQvajvlXyIWgI6fs2HnhTfYCpUyNAIJdhBtoxoRDhrVzTe0W5UX1AF7nDri4cEDplH71Mqsaje8G3zjx32q3UdQHM4BU23PCNgPqGisqTGMpwHi/zYHvg+O8+TYUIE9kiz+ZioTABHqsAbyr5M= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=B3LaOMpn; arc=none smtp.client-ip=209.85.214.173 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="B3LaOMpn" Received: by mail-pl1-f173.google.com with SMTP id d9443c01a7336-21636268e43so92962975ad.2 for ; Sun, 19 Jan 2025 21:48:48 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1737352128; x=1737956928; darn=vger.kernel.org; h=content-transfer-encoding:in-reply-to:from:references:cc:to:subject :user-agent:mime-version:date:message-id:from:to:cc:subject:date :message-id:reply-to; bh=AuoU/qShB8LcxcJgkCx/TpSS5rW8Ia7w61BRXqiZAuM=; b=B3LaOMpnVke4vmho3GrD9IjrTnMI+gZ+ry3niWflzbQKKqZJB1o9NNp7Z+v20fcq9Y M1IdPOkIy45adAHrUi3ZeHgGy3whR6OIPeiyl0rfLzBVp/WQlpISX5snQ3zL8NAcxVrg 9U4blNJE3oQmashniKDuWE8kC4UI01J7orzzV6w/CEci2hh1sl78ym+PGCYnRxgk9p4M jpJAqcxRgcEeWGdouAJh7Y+9tsDF9ZyvojG/+okuDvCW/yDmV+RbElgqW3YzzJz/+OIY IDd22IczKBPAN/7TUB3sbwmqj+I1uXS6a+hKkNLOwUMohm9PYDOMd7UNnTcgs86fUFOT XVvA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1737352128; x=1737956928; h=content-transfer-encoding:in-reply-to:from:references:cc:to:subject :user-agent:mime-version:date:message-id:x-gm-message-state:from:to :cc:subject:date:message-id:reply-to; bh=AuoU/qShB8LcxcJgkCx/TpSS5rW8Ia7w61BRXqiZAuM=; b=QQt3KUYKbS73nAy5s/keUuFu063ScxPfdjHYTDspT4RYEhealqWRRZyZn3JUUExNFS kIUaHdwxBS7mdqA1GK3EvEWnnGTEhF44aQS6anLOz1CmTAXKuPsiIKXzj0/mWbk2bF0n IQ2Vz0krbQ5oQPEh4oE34tUy7ZUw5oaSOuELhm7R19h8YE741+unu3V1PdX2LukKZoJ+ FD9BBKzUqK9BuKAPQGCShBJ9L/mHGBozYOsUaRQD7CHThTuqEkfz5Ocg2GzQslm0ggMq 4CvosOD7UGXsA75rp9aMq9UgTeofdKAjfltSZfszzQFB9zBdY4jyCFBZM/7P0pih1voO 7ffg== X-Forwarded-Encrypted: i=1; AJvYcCXi5dNWDs3rCN7RlKS+2dlUfYYdXc6Q0yFCS8zl0SSioxdGmFThDj7eKwJR335kRIPt6o9UWqhr3FTxMtI=@vger.kernel.org X-Gm-Message-State: AOJu0Yxh/7TlNbc9kpzuoL90JegzF4t5ck+ld29ymUhHtmnhwa4chWbf 85KHbrJ5LApBuqfryrRmy2llMP5jNE6LX0pjhEeynhN76cSV0hHh X-Gm-Gg: ASbGncu8zX3eASkYqnk7yzZKUtvUgxF177oLbMmCWHVkOorYwVHqu0EFvFKAIh1lWUR pE9MdkTgF3sPBX6E87hNNblOwawhPNlj4uxTOg3jYRHOJ1qbU8sB/GRsYN3fYUip4atQ55i4OC4 4UYCGiBQXD/rbTHw+F+4cS2UxRo2UDqvd2j2q+M0Y6CLNuHACmWCWc2Btad5V2YlJMh01pL13H6 1JPgQptsms+4EZQ4nsVaj/41hs05RkMm4CoigfYIExYi/NII+8u4snjsea6zURv9IDwjxslQGRX kv1JAmqWLPFBtA== X-Google-Smtp-Source: AGHT+IGVJBzHZvsc/tEVHZ0Ybr/JlYxFA0QBMQNyRYzcuEGW85kUf2/IFyCnWJ74KVvwTlsQcMznWg== X-Received: by 2002:a17:903:2b06:b0:215:50fb:ae4a with SMTP id d9443c01a7336-21c34cd3f0cmr190035185ad.0.1737352127549; Sun, 19 Jan 2025 21:48:47 -0800 (PST) Received: from [10.125.112.20] ([103.165.80.178]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-21c2d40278dsm52144955ad.223.2025.01.19.21.48.43 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Sun, 19 Jan 2025 21:48:47 -0800 (PST) Message-ID: <53082ec4-d862-6135-8d22-44dd2742156a@gmail.com> Date: Mon, 20 Jan 2025 13:48:39 +0800 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:102.0) Gecko/20100101 Thunderbird/102.15.0 Subject: Re: [PATCH v2] sched/core: Prioritize migrating eligible tasks in sched_balance_rq() To: Vincent Guittot Cc: mingo@redhat.com, peterz@infradead.org, mingo@kernel.org, juri.lelli@redhat.com, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de, vschneid@redhat.com, linux-kernel@vger.kernel.org, Hao Jia References: <20241223091446.90208-1-jiahao.kernel@gmail.com> <597384e3-6519-b10e-081b-30c3f89b6e3f@gmail.com> <56d9d6cc-021e-4bd5-7254-aa39c9845500@gmail.com> From: Hao Jia In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit On 2025/1/16 19:26, Vincent Guittot wrote: > On Wed, 15 Jan 2025 at 12:55, Hao Jia wrote: >> >> >> >> On 2025/1/15 17:28, Vincent Guittot wrote: >>> On Wed, 15 Jan 2025 at 09:55, Hao Jia wrote: >>>> >>>> >>>> >>>> On 2025/1/14 16:07, Vincent Guittot wrote: >>>>> On Tue, 14 Jan 2025 at 04:18, Hao Jia wrote: >>>>>> >>>>>> >>>>>> >>>>>> On 2025/1/14 00:40, Vincent Guittot wrote: >>>>>>> On Mon, 13 Jan 2025 at 10:21, Hao Jia wrote: >>>>>>>> >>>>>>>> Friendly ping... >>>>>>>> >>>>>>>> >>>>>>>> On 2024/12/23 17:14, Hao Jia wrote: >>>>>>>>> From: Hao Jia >>>>>>>>> >>>>>>>>> When the PLACE_LAG scheduling feature is enabled and >>>>>>>>> dst_cfs_rq->nr_queued is greater than 1, if a task is >>>>>>>>> ineligible (lag < 0) on the source cpu runqueue, it will >>>>>>>>> also be ineligible when it is migrated to the destination >>>>>>>>> cpu runqueue. Because we will keep the original equivalent >>>>>>>>> lag of the task in place_entity(). So if the task was >>>>>>>>> ineligible before, it will still be ineligible after >>>>>>>>> migration. >>>>>>>>> >>>>>>>>> So in sched_balance_rq(), we prioritize migrating eligible >>>>>>>>> tasks, and we soft-limit ineligible tasks, allowing them >>>>>>>>> to migrate only when nr_balance_failed is non-zero to >>>>>>>>> avoid load-balancing trying very hard to balance the load. >>>>>>> >>>>>>> Could you explain why you think it's better to balance eligible tasks >>>>>>> in priority and potentially skip a load balance ? >>>>>> >>>>>> In place_entity(), we maintain the task's original equivalent lag, even >>>>>> if we migrate the task to dst_rq, this does not change its eligibility >>>>>> attribute. >>>>> >>>>> Yes, but you don't answer the question why it's better to select an >>>>> eligible task vs a non eligible task. >>>>> >>>>>> >>>>>> When there are multiple tasks on src_rq, and the dst_cpu has some >>>>>> runnable tasks, migrating ineligible tasks to dst_rq will not allow them >>>>>> to run. Therefore, such task migration is inefficient. We should >>>>> >>>>> Why is it inefficient ? load balance is about evenly balancing the >>>>> number of tasks or the load between CPUs, it never says that the newly >>>>> migrated task should run immediately >>>> >>>> >>>> My initial thought is that when we need to migrate some tasks during >>>> load balancing, at the current point in time, migrating ineligible tasks >>>> to dst_cpu means they definitely cannot run there. Therefore, I prefer >>>> to keep them on src_cpu to reduce the overhead of dequeueing and >>>> enqueueing ineligible tasks. >>> >>> Sorry but I still don't get why it's important and would make a >>> difference. They are all runnable but ineligible tasks got more >>> runtime than other at that point in time so there is no real >>> difference >> >> >> I adopt a lazy strategy for ineligible tasks. At the current point in >> time, even if we migrate ineligible tasks to the dst CPU, they still >> have to wait on the dst CPU until they become eligible. We do not see >> clear benefits from migrating ineligible tasks, but their dequeueing and >> enqueueing would instead incur overhead. > > But your explanation doesn't make sense. > Not migrating an ineligible task only make sense for delayed_dequeue > tasks because they don't really want to run but only exhaust their lag > but this is already taken into account by > 61b82dfb6b7e ("sched/fair: Do not try to migrate delayed dequeue task") > Thank you for your suggestion. Yes, as you mentioned, this commit 61b82dfb6b7e ("sched/fair: Do not try to migrate delayed dequeue task") reduces the migration of delayed_dequeue tasks, but it doesn't work for ineligible RUNNING tasks and when the migration_type is migrate_load. > Did you run your benchmark on top of this change ? My previous benchmark tests were based on the torvalds/linux/master branch, which does not include commit 61b82dfb6b7e ("sched/fair: Do not try to migrate delayed dequeue task"). I will include this commit and retest on my machine after my leave ends. Thanks, Hao > >> >> Let them wait on the src CPU until they become eligible before migrating >> them. this can reduce the number of task migrations. >> >> >> >> Thanks, >> Hao >> >> >>> >>>> >>>> Migrating eligible tasks to dst_cpu does not guarantee that they will >>>> run earlier than on src_cpu. it depends on too many factors. >>>> >>>> >>>> >>>>> >>>>>> prioritize migrating tasks that can run on dst_rq. >>>>>> >>>>>> In other words, migrating ineligible tasks is merely moving them to >>>>>> another runqueue to wait until they become eligible. >>>>> >>>>> But I don't get why it's a problem. Migrating an eligible task might >>>>> delay its scheduling because of its deadline vs other tasks already >>>>> eligible on the dst_rq. Eligible and non eligible tasks are all >>>>> runnable, it's just how much they have already run. In addition, >>>>> migrating an eligible task will clear its positive vlag with >>>>> DELAY_ZERO which is unfair IMO >>>> >>>> >>>> Sorry, I'd like to ask you a question that confuses me: Why does >>>> migrating eligible task will clear the positive vlag? >>> >>> sorry I mess up everything that only for delayed dequeue task >>> >>>> >>>> In detach_task(), the ENQUEUE_DELAYED and DEQUEUE_SLEEP flags are not >>>> set, and in dequeue_entity(), eligible tasks will not set sched_delayed, >>>> so they will be dequeued normally with se->on_rq being 0. >>>> >>>> Similarly, attach_task() does not set the ENQUEUE_DELAYED and >>>> DEQUEUE_SLEEP flags, and since se->on_rq is 0, it will not call >>>> requeue_delayed_entity(). >>>> >>>> >>>> Thanks, >>>> Hao >>