From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1753926AbcDDIoU (ORCPT ); Mon, 4 Apr 2016 04:44:20 -0400 Received: from casper.infradead.org ([85.118.1.10]:58916 "EHLO casper.infradead.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1750741AbcDDIoT (ORCPT ); Mon, 4 Apr 2016 04:44:19 -0400 Date: Mon, 4 Apr 2016 10:44:16 +0200 From: Peter Zijlstra To: Jiri Olsa Cc: Ingo Molnar , James Hartsock , Rik van Riel , Srivatsa Vaddagiri , Kirill Tkhai , linux-kernel@vger.kernel.org Subject: Re: [RFC] sched: unused cpu in affine workload Message-ID: <20160404084416.GV3448@twins.programming.kicks-ass.net> References: <20160404082302.GB2137@krava.local> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20160404082302.GB2137@krava.local> User-Agent: Mutt/1.5.21 (2012-12-30) Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Mon, Apr 04, 2016 at 10:23:02AM +0200, Jiri Olsa wrote: > hi, > we've noticed following issue in one of our workloads. > > I have 24 CPUs server with following sched domains: > domain 0: (pairs) > domain 1: 0-5,12-17 (group1) 6-11,18-23 (group2) > domain 2: 0-23 level NUMA > > I run CPU hogging workload on following CPUs: > 4,6,14,18,19,20,23 > > that is: > 4,14 CPUs from group1 > 6,18,19,20,23 CPUs from group2 > > the workload process gets affinity setup via 'taskset -c ${CPUs workload ...' > and forks child for every CPU > > very often we notice CPUs 4 and 14 running 3 processes of the workload > while CPUs 6,18,19,20,23 running just 4 processes, leaving one of the > CPU from group2 idle > > AFAICS from the code the reason for this is that the load balancing > follows domains setup (topology) and does not regard affinity setups > like this. The code in find_busiest_group running under idle cpu from > group2 will find group1 as bussiest, but its average load will be > smaller than the one on the local group, so there's no task pulling. > > It's obvious, that load balancer follows sched domain topology. > However is there some sched feature I'm missing that could help > with this? Or do we need to follow sched domains topology when > we select CPUs for workload to get even balancing? Yeah, this is 'hard', there is some code that tries not to totally blow with this but its all a bit of a mess. See kernel/sched/fair.c:sg_imbalanced(). The easiest solution is to simply not do this and stick with the topo like you suggest. So far I've not come up with a sane/stable solution for this problem.