From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7FCAD1EBFE4 for ; Tue, 7 Jan 2025 11:26:10 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=90.155.92.199 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1736249172; cv=none; b=mQ2zvdsdXMx0u15BVOEpOe6QLzfcfXeA3ucpBr5BTvkKzIZlgYqO/3RbUH1mhVsMpQRwAPEeKQPQeE3YtWU9xjzLN2yloHl20j06B57haOCUbUS9KXWoX6m9c4+E5M43OAr4K01vQXLBNyFgxqIUGeHtSMb6I7J3UVGl+sxY+v0= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1736249172; c=relaxed/simple; bh=HXpFGwzRwhbohbtdrdRpjufubZyn/f6E5uTrrCO3Qkg=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=UYP6x+RBdWQVz/xMQR13E9Gy6KxlCpuq9q7qQCwCWXjOsMmO27Kg7P3MJvRRp4Uclydzz2g3yENg0EfPOswx4FBwj6NfcGze5PJfKofG1Rg6A0+iB200VR3YZHX1tzRqZuz2x3TlNBKTVWb/2V1LvEkG8R6mwMVRJF4UXbsjaVM= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org; spf=none smtp.mailfrom=infradead.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b=g/oFFhrZ; arc=none smtp.client-ip=90.155.92.199 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=infradead.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="g/oFFhrZ" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=In-Reply-To:Content-Type:MIME-Version: References:Message-ID:Subject:Cc:To:From:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=qAvYrwCyg1mQjquQSK/aYHeX81dzZ+MM97lS+NpZpKM=; b=g/oFFhrZRQs6JrSxncdVVZUevv PHD5QInFABoWZwHXKjBQB7O/dSMhMFKiQvxy9r31/xCyhiTm5gSCHnVJ/Cc7tgdcTLp4TTY/VBcJV ZfqAib/K8KnE5QO0FsES1/a5ppwRRNcIczSTgEz5IHC7bgomsCZJEHa/j+6FtN/VRhsYEsfZUjpyc RHMtvMOQBuvuCicvYK+xN0lozGFHTrzImLpU58BEm0C5gPjMgGInrjtqCzAVVq059Z1xUULVpK7ZL mKU7qkyLLXV4l+zVpH6Jpib4CxKGGC76ZnT5fi5ZUIQj7TVqw/LBwXBE7DKluXIwJZHvO7rNP4exk 6oak48Ng==; Received: from 77-249-17-89.cable.dynamic.v4.ziggo.nl ([77.249.17.89] helo=noisy.programming.kicks-ass.net) by desiato.infradead.org with esmtpsa (Exim 4.98 #2 (Red Hat Linux)) id 1tV7ip-0000000989K-3V77; Tue, 07 Jan 2025 11:26:08 +0000 Received: by noisy.programming.kicks-ass.net (Postfix, from userid 1000) id 1BFB43005AF; Tue, 7 Jan 2025 12:26:07 +0100 (CET) Date: Tue, 7 Jan 2025 12:26:06 +0100 From: Peter Zijlstra To: Doug Smythies Cc: linux-kernel@vger.kernel.org, vincent.guittot@linaro.org Subject: Re: [REGRESSION] Re: [PATCH 00/24] Complete EEVDF Message-ID: <20250107112606.GN20870@noisy.programming.kicks-ass.net> References: <005f01db5a44$3bb698e0$b323caa0$@telus.net> <20250106115732.GE20870@noisy.programming.kicks-ass.net> <000801db604b$e0f6b580$a2e42080$@telus.net> <20250106165932.GG20870@noisy.programming.kicks-ass.net> <20250106170455.GB22191@noisy.programming.kicks-ass.net> <001b01db608a$56d3dc40$047b94c0$@telus.net> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <001b01db608a$56d3dc40$047b94c0$@telus.net> On Mon, Jan 06, 2025 at 02:28:40PM -0800, Doug Smythies wrote: > Which will show when a CPU migration took over 10 milliseconds. > If you want to go further, for example to only display ones that took > over a second and to include the target CPU, then patch turbostat: > > doug@s19:~/kernel/linux/tools/power/x86/turbostat$ git diff > diff --git a/tools/power/x86/turbostat/turbostat.c b/tools/power/x86/turbostat/turbostat.c > index 58a487c225a7..f8a73cc8fbfc 100644 > --- a/tools/power/x86/turbostat/turbostat.c > +++ b/tools/power/x86/turbostat/turbostat.c > @@ -2704,7 +2704,7 @@ int format_counters(struct thread_data *t, struct core_data *c, struct pkg_data > struct timeval tv; > > timersub(&t->tv_end, &t->tv_begin, &tv); > - outp += sprintf(outp, "%5ld\t", tv.tv_sec * 1000000 + tv.tv_usec); > + outp += sprintf(outp, "%7ld\t", tv.tv_sec * 1000000 + tv.tv_usec); > } > > /* Time_Of_Day_Seconds: on each row, print sec.usec last timestamp taken */ > @@ -4570,12 +4570,14 @@ int get_counters(struct thread_data *t, struct core_data *c, struct pkg_data *p) > int i; > int status; > > + gettimeofday(&t->tv_begin, (struct timezone *)NULL); /* doug test */ > + > if (cpu_migrate(cpu)) { > fprintf(outf, "%s: Could not migrate to CPU %d\n", __func__, cpu); > return -1; > } > > - gettimeofday(&t->tv_begin, (struct timezone *)NULL); > +// gettimeofday(&t->tv_begin, (struct timezone *)NULL); > > if (first_counter_read) > get_apic_id(t); > > So I've taken the second node offline, running with 10 cores (20 threads) now. usec Time_Of_Day_Seconds CPU Busy% IRQ 106783 1736248404.951438 - 100.00 20119 46 1736248404.844701 0 100.00 1005 41 1736248404.844742 20 100.00 1007 42 1736248404.844784 1 100.00 1005 40 1736248404.844824 21 100.00 1006 41 1736248404.844865 2 100.00 1005 40 1736248404.844905 22 100.00 1006 41 1736248404.844946 3 100.00 1006 40 1736248404.844986 23 100.00 1005 41 1736248404.845027 4 100.00 1005 40 1736248404.845067 24 100.00 1006 41 1736248404.845108 5 100.00 1011 40 1736248404.845149 25 100.00 1005 41 1736248404.845190 6 100.00 1005 40 1736248404.845230 26 100.00 1005 42 1736248404.845272 7 100.00 1007 41 1736248404.845313 27 100.00 1005 41 1736248404.845355 8 100.00 1005 42 1736248404.845397 28 100.00 1006 46 1736248404.845443 9 100.00 1009 105995 1736248404.951438 29 100.00 1005 Is by far the worst I've had in the past few minutes playing with this. If I get a blimp (>10000) then it is always on the last CPU, are you seeing the same thing? > In this short example all captures were for the CPU 5 to 11 migration. > 2 at 6 seconds, 1 at 1.33 seconds and 1 at 2 seconds. This seems to suggest you are, always on CPU 11. Weird! Anyway, let me see if I can capture a trace of this..