From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from fanzine2.igalia.com (fanzine.igalia.com [178.60.130.6]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id EA3C6218838 for ; Thu, 9 Jan 2025 13:15:18 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=178.60.130.6 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1736428523; cv=none; b=BF63x1jZKiPssJAwO1Z23YUnf947aTSvq1vWmXWaul8tD7yW4FSe6iaz285XeWNtHlJpEexQW98pLseBhig+RxPPYvjPCtekOcKH+9Hdj3m+hadBqa0j0NbqidzfZaD8KduONwyZckW/oE4ev/RABCcrtnmIgbqdxAsVs1HY++U= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1736428523; c=relaxed/simple; bh=hFK7Mm5QY5GMyU+StsX0AGYVbpVUGJlOm10yNPt5OAw=; h=From:To:Cc:Subject:Date:Message-ID:MIME-Version; b=qPJ8PoHnZP61ID6/1QMSFecH+P3ef1iyDxgSgEoMe+OQsg7FRF6DAhuhRvkaMvVYOe9zTxUJlj0LR+GwE4/31CyCJ/NM8+kHCTwV6ecaBNRgoh/k++X7xkcIM5q4VyOJTl5WN5U8zma0JgsBRypoUds/wYSGmSSoYPnWKXYLRTI= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=igalia.com; spf=pass smtp.mailfrom=igalia.com; dkim=pass (2048-bit key) header.d=igalia.com header.i=@igalia.com header.b=A1/Bcxy1; arc=none smtp.client-ip=178.60.130.6 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=igalia.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=igalia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=igalia.com header.i=@igalia.com header.b="A1/Bcxy1" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=igalia.com; s=20170329; h=Content-Transfer-Encoding:MIME-Version:Message-ID:Date:Subject: Cc:To:From:Sender:Reply-To:Content-Type:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: In-Reply-To:References:List-Id:List-Help:List-Unsubscribe:List-Subscribe: List-Post:List-Owner:List-Archive; bh=Hl1O730y+nbvZKwo0kcl6Q1tXKOQ5PoBzDx3ERXzi2A=; b=A1/Bcxy1BwF3jADSHNmSB/5LlQ E/2jQZIBSY0JECpIGZTAeWu2ZANeyeZzaAq9kumfBZEqHuj6EJyzODtef9FZzBTujMCDpYOCHTn78 5jo3BBhWJ/JCArK1zuSUpu6qLbe3sW85YAq062O7CkZ+BDyi7eHWJ6CvR7L8BGaJNAr2Eh4YKfvQk thbKxQ2QzLlm0+5xJ4I2p3yN58Ugl2DQQRD6+97PR6IGTrF2Vk21qY0+ZOsjiFlT3myCq7iUU7cu9 DEXjvvxmCSgMV29p9JP4Q+jET6qv8RzGIYHCRBo3LGvMAspPli29FreUbPfXeDQVkUdFEHULVbAj5 zgMrfYVw==; Received: from [58.29.143.236] (helo=localhost) by fanzine2.igalia.com with esmtpsa (Cipher TLS1.3:ECDHE_SECP256R1__RSA_PSS_RSAE_SHA256__AES_256_GCM:256) (Exim) id 1tVsNK-00DZ9F-C6; Thu, 09 Jan 2025 14:15:02 +0100 From: Changwoo Min To: tj@kernel.org, void@manifault.com, arighi@nvidia.com, mingo@redhat.com, peterz@infradead.org Cc: kernel-dev@igalia.com, linux-kernel@vger.kernel.org, Changwoo Min Subject: [PATCH v8 0/6] sched_ext: Support high-performance monotonically non-decreasing clock Date: Thu, 9 Jan 2025 22:14:50 +0900 Message-ID: <20250109131456.7055-1-changwoo@igalia.com> X-Mailer: git-send-email 2.47.1 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Many BPF schedulers (such as scx_central, scx_lavd, scx_rusty, scx_bpfland, and scx_flash) frequently call bpf_ktime_get_ns() for tracking tasks' runtime properties. If supported, bpf_ktime_get_ns() eventually reads a hardware timestamp counter (TSC). However, reading a hardware TSC is not performant in some hardware platforms, degrading IPC. This patchset addresses the performance problem of reading hardware TSC by leveraging the rq clock in the scheduler core, introducing a scx_bpf_now() function for BPF schedulers. Whenever the rq clock is fresh and valid, scx_bpf_now() provides the rq clock, which is already updated by the scheduler core (update_rq_clock), so it can reduce reading the hardware TSC. When the rq lock is released (rq_unpin_lock), the rq clock is invalidated, so a subsequent scx_bpf_now() call gets the fresh sched_clock for the caller. In addition, scx_bpf_now() guarantees the clock is monotonically non-decreasing for the same CPU, so the clock cannot go backward in the same CPU. Using scx_bpf_now() reduces the number of reading hardware TSC by 50-80% (76% for scx_lavd, 82% for scx_bpfland, and 51% for scx_rusty) for the following benchmark: perf bench -f simple sched messaging -t -g 20 -l 6000 ChangeLog v7 -> v8: - Use smp_store_release() and smp_load_acquire() to ensure the ordering between rq->scx.clock and rq->scx.flags. ChangeLog v6 -> v7: - Do not update rq_lock when the rq_clock is invalidated. - Rename scx_bpf_now_ns() to scx_bpf_now(). - Rename vtime_*() helper functions to time_*(). - Fix a typo. ChangeLog v5 -> v6: - Drop prev_clock because a race between a process context and an interrupt context (e.g., timer interrupt) is not observable from a caller (discussed with Tejun offline), so it is not necessary to check prev_clock. ChangeLog v4 -> v5: - Merge patches 2, 3, and 4 into one for readability. - Use a time helper (time_after_eq64) for time comparison at scx_bpf_now_ns(). - Do not validate the rq clock outside the rq critical section for more predictable behavior. - Improve the comment at scx_bpf_now_ns() for readability. - Rename scx_rq_clock_stale() into scx_rq_clock_invalidate(). - Invalidate all the rq clocks upon unloading to prevent getting outdated rq clocks from a previous scx scheduler. - Use READ/WRITE_ONCE() when accessing rq->scx.{clock, prev_clock, flags} to properly handle concurrent accesses from an interrupt contex. - Add time helpers for BPF schedulers. - Update the rdtsc reduction numbers in the cover letter with the latest scx_scheds (1.0.7) and the updated scx_bpf_now_ns(). ChangeLog v3 -> v4: - Separate the code relocation related to scx_enabled() into a separate patch. - Remove scx_rq_clock_stale() after (or before) ops.running() and ops.update_idle() calls - Rename scx_bpf_clock_get_ns() into scx_bpf_now_ns() and revise it to address the comments - Move the per-CPU variable holding a prev clock into scx_rq (rq->scx.prev_clock) - Add a comment describing when the clock could go backward in scx_bpf_now_ns() - Rebase the code to the tip of Tejun's sched_ext repo (for-next branch) ChangeLog v2 -> v3: - To avoid unnecessarily modifying cache lines, scx_rq_clock_update() and scx_rq_clock_stale() update the clock and flags only when a sched_ext scheduler is enabled. ChangeLog v1 -> v2: - Rename SCX_RQ_CLK_UPDATED to SCX_RQ_CLK_VALID to denote the validity of an rq clock clearly. - Rearrange the clock and flags fields in struct scx_rq to make sure they are in the same cacheline to minimize the cache misses - Add an additional explanation to the commit message in the 2/5 patch describing when the rq clock will be reused with an example. - Fix typos - Rebase the code to the tip of Tejun's sched_ext repo Changwoo Min (6): sched_ext: Relocate scx_enabled() related code sched_ext: Implement scx_bpf_now() sched_ext: Add scx_bpf_now() for BPF scheduler sched_ext: Add time helpers for BPF schedulers sched_ext: Replace bpf_ktime_get_ns() to scx_bpf_now() sched_ext: Use time helpers in BPF schedulers kernel/sched/core.c | 6 +- kernel/sched/ext.c | 74 +++++++++++++++++- kernel/sched/sched.h | 51 +++++++++---- tools/sched_ext/include/scx/common.bpf.h | 95 ++++++++++++++++++++++++ tools/sched_ext/include/scx/compat.bpf.h | 5 ++ tools/sched_ext/scx_central.bpf.c | 11 +-- tools/sched_ext/scx_flatcg.bpf.c | 23 +++--- tools/sched_ext/scx_simple.bpf.c | 9 +-- 8 files changed, 228 insertions(+), 46 deletions(-) -- 2.47.1