From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6A1BE30E823 for ; Tue, 25 Nov 2025 13:58:39 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.129.124 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1764079121; cv=none; b=TpBxjE/NahBocTlonWgEg5Pq3dDZQsQCFzqN817EMXHTZ/2HZRYjtBH3gldg6F/DwufDTGv45hcnNSKLIwJBgzT74CKkk4///BQPlRW0YQ4G/xTYaw5hx4uHfKTb4UsrVzMg2Tr0wPZX3mK1t6hMzFgU359FUccElr6tBcPOq/w= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1764079121; c=relaxed/simple; bh=ZDH314136pyD/YVSg1YaQOwkACs0qiAVsGotI60qPLM=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=meBN8/+q51gIvj94bR/Gg77IO2ZLx0YQWYlwufZOkwYzeH0c3kQKOrBUsgnX83CPQKjbLJIb2F1k0oIqmMT9UssY1g9IpxFSc1hD6NCsde9FoyQM8XRxWFchhJCbK900TLLGfV/d/JMjSkQiUV0rMczgwXEhR91VZzY2mAOoSQY= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=fMcPM756; dkim=pass (2048-bit key) header.d=redhat.com header.i=@redhat.com header.b=LHl8YXPU; arc=none smtp.client-ip=170.10.129.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="fMcPM756"; dkim=pass (2048-bit key) header.d=redhat.com header.i=@redhat.com header.b="LHl8YXPU" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1764079118; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=ZDH314136pyD/YVSg1YaQOwkACs0qiAVsGotI60qPLM=; b=fMcPM756DkzFbUHbgH2vsrsnYQmK6y9kM6mEU3dziuulHQ1i6M0llYXb8pcDliHHiKhZJ3 XhMAiDRxtleY4MzVxTlNy6KIQ0iPjuJRQ2aH7jQywDvJ9y2oq2kmZ8IK71pXEvU2jVUiqu +WL/RtCrQ4kC6I4/YQ7VtOgRquNUcj4= Received: from mail-wm1-f70.google.com (mail-wm1-f70.google.com [209.85.128.70]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-359-APa8SDbsNXe-Ne8-JHiG_A-1; Tue, 25 Nov 2025 08:58:30 -0500 X-MC-Unique: APa8SDbsNXe-Ne8-JHiG_A-1 X-Mimecast-MFC-AGG-ID: APa8SDbsNXe-Ne8-JHiG_A_1764079109 Received: by mail-wm1-f70.google.com with SMTP id 5b1f17b1804b1-4779b3749a8so47918995e9.1 for ; Tue, 25 Nov 2025 05:58:30 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=google; t=1764079108; x=1764683908; darn=vger.kernel.org; h=in-reply-to:content-transfer-encoding:content-disposition :mime-version:references:message-id:subject:cc:to:from:date:from:to :cc:subject:date:message-id:reply-to; bh=ZDH314136pyD/YVSg1YaQOwkACs0qiAVsGotI60qPLM=; b=LHl8YXPUY39/gMPkmbCm6vnc3s1hBMxl7iJyc4rJ8h+Zn0mo433XoSPcBb0i0gUj2u B3dlnW+L4goed4XcSxj0/dHx/vOonhfPDiDW6A+6qfvOwX5dm1u1Nro5quRf1xtYxQdC rsxJkNn961Vz99oENd7vrZrSjmzYAiWeWkSG6nnDkbWEj60ToRUEBGJ0xP7aRiZxuMj+ A4RJ/81t7SntBTD3WeSrsXl4k0bnqHsQ/7Lef1WO+LX8og+mAyy60gu0jWkVdqPO7mji cOvoRr+v0qRzYgLY5KkHpRfs5FiVzAb4JHL4bejmwTN4k17+ZtEz7YrZJZkVvIKUQRIz CJpg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1764079108; x=1764683908; h=in-reply-to:content-transfer-encoding:content-disposition :mime-version:references:message-id:subject:cc:to:from:date:x-gm-gg :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=ZDH314136pyD/YVSg1YaQOwkACs0qiAVsGotI60qPLM=; b=GzrRD1XnQyFfEkJfQaYa46fXH5eHCcRZLIaQvaLKWrnWnFj46Rb7JmZUOQE6dZkcri n29iOyJsi8nz8UQETPy4YorHA7JvSAeuySEACe9lKKtGIYE7ZyA6InNLvu9x0KJYQsE/ wxr8h6ecBgASjfv5K7HlVen34sLUmONQfC1vVmaCdTyHlad9MQBd1sizKIR/IRPz7QOD oMZC7sWomlp64frTsHw/kFfAFEgN04cCNwve7cKnefj6iWjkgjcPaMtaNZIWXxg818BZ hM/jG35YBsxHLV4LkmvV2Wq+8cm0l45BSuWNzPdbDIkmX4Q0YbPWKdVdrXH1TRrvLY7s hmow== X-Gm-Message-State: AOJu0YyhSJ/8qp7BzD5E5HjN4DhWq38+pgIzYahDLlVnuWahSkqtBJjw TQfLn7rm+6C31JP0AKdH9bEBHcyEDeEmDt3AEy41f/KEXnA5IJFOxJg9d1pmrlg1jhp7lw0gR6l zM6Kk1J2Om+pBUtls4rnK3X/cWiyptRCEM0M5X5BeB7t2TopIaFGic+bLxTgCnJ3YUaDiteh7nv rX X-Gm-Gg: ASbGncvFvOnv2qnu0Bzuu0JdclTydgciud6x5yMF1kSgqdepOP5Q2/Dvm/w6WE1RHOc Oy7LU6a4WIdlWFkXLexshu12b/s0hD9avt4xSiSkx7xkcFlGg8tU/lP1D7LSuMa1Vof7ZxD1xOa CzHL8YfAjy50QKsNfcS8lEKHtlSZ+BVknqdU0MANtAYABlAY8dJtozmeFBoMK9O5EDsG3faPHBE lJ4g+OpwFe42I+5gt0sEGLUpb/2SN4Ab2xa6rtV/bI2jxz3bOxS9/8jrZB42avECvNGjTk6/hAi CoxfYbXZTbYq8hfjUUZxBhIQejdDYGj6jx5kGiW9CwtK5Rzx2U7zOGPF1RMMW/EOhxktrM/wxeu TOOMVCZ4UD/wECmZgxtqxr3rKlUN3Kr0SQ5JKRrlUJoAKmmXoByWeIzQVy0lJZLa49c5bQw== X-Received: by 2002:a05:600c:354a:b0:477:97ca:b727 with SMTP id 5b1f17b1804b1-477c01b21bcmr136192915e9.19.1764079108399; Tue, 25 Nov 2025 05:58:28 -0800 (PST) X-Google-Smtp-Source: AGHT+IEODg8XtdeTCZScSJOjCJ2ZzP/NpPPwu6Zy7b8+gP0DLp9rMkJo3hkwhNGzyfSjtC9QmPqBdw== X-Received: by 2002:a05:600c:354a:b0:477:97ca:b727 with SMTP id 5b1f17b1804b1-477c01b21bcmr136192715e9.19.1764079107984; Tue, 25 Nov 2025 05:58:27 -0800 (PST) Received: from jlelli-thinkpadt14gen4.remote.csb (host-2-99-207-158.as13285.net. [2.99.207.158]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-479052def4bsm16308045e9.13.2025.11.25.05.58.26 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 25 Nov 2025 05:58:27 -0800 (PST) Date: Tue, 25 Nov 2025 13:58:24 +0000 From: Juri Lelli To: David Haufe Cc: linux-kernel@vger.kernel.org Subject: Re: Kernel 6.14.11 dl_server_timer(...) causing IPI/Function Call Interrupts on isolcpu/nohz_full cores, performance regression Message-ID: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: Hi David, On 24/11/25 14:30, David Haufe wrote: > Hi Juri, > Working with 6.17.7 (code appears the same in .8 and 6.18rc), we are > still having isolated/nohz cores interrupted by deadline > functionality. The dl_task_timer is firing once a second with a single > SCHED_OTHER process spinning on the core. I am no longer seeing the > IPI interrupts, but the hrtime activity is still causing a performance > regression compared to kernels prior to the deadline merge. Where you able to check if the changes on the branch below made things any better? It is hopefully not that hard to apply to newer/different baselines. I have to apologize in advance, but I'm going to be on pto for a few days and then I have to travel for LPC26. So, unfortunately I am going to have very limited availability until end of the year. Best, Juri > On Mon, Aug 4, 2025 at 11:59 AM Juri Lelli wrote: > > > > On 04/08/25 10:44, David Haufe wrote: > > > My apologies, I see what you mean now. add_nr_running() is not being > > > invoked if it is the dl_server. We are still trying to get this branch > > > to boot to verify ourselves. We will be on the lookout for this to be > > > merged for release. > > > > No worries, guess I wasn't clear the first time. :) > > > > I added a very much experimental commit on > > > > https://github.com/jlelli/linux/tree/upstream/fix-dlserver-1 > > > > that seems to be able to remove the one per second dl_server_timer and > > start it back as needed. But, I just played briefly with it, so I am not > > fully convinced is what we want. Anyway, if you could test with it as > > well it would be a useful data point. In principle you could try porting > > the following commits to your current tree and check if they do improve > > things (in reverse order starting from the bottom from the branch above): > > > > f237e524f3c7 ("sched/deadline: Make dl-server nohz full aware") > > 219a63335b67 ("sched/deadline: Don't count nr_running twice for dl_server proxy tasks") > > 7620177e8108 ("sched/deadline: Fix RT task potential starvation when expiry time passed") > > cccb45d7c429 ("sched/deadline: Less agressive dl_server handling")