From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752132Ab2LKAuF (ORCPT ); Mon, 10 Dec 2012 19:50:05 -0500 Received: from ch1ehsobe002.messaging.microsoft.com ([216.32.181.182]:21161 "EHLO ch1outboundpool.messaging.microsoft.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751470Ab2LKAuD (ORCPT ); Mon, 10 Dec 2012 19:50:03 -0500 X-Forefront-Antispam-Report: CIP:160.33.194.228;KIP:(null);UIP:(null);IPV:NLI;H:usculsndmail01v.am.sony.com;RD:mail01.sonyusa.com;EFVD:NLI X-SpamScore: -3 X-BigFish: VPS-3(zzbb2dI98dI1432Izz1de0h1202h1e76h1d1ah1d2ahzzz2fh2a8h668h839h947hd25hf0ah10d2h1288h12a5h12a9h12bdh137ah13b6h1441h1537h153bh162dh1631h1758h1765h1155h) Message-ID: <50C682F6.5030709@am.sony.com> Date: Mon, 10 Dec 2012 16:48:54 -0800 From: Frank Rowand Reply-To: User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.17) Gecko/20110428 Fedora/3.1.10-1.fc14 Thunderbird/3.1.10 MIME-Version: 1.0 To: Steven Rostedt CC: "linux-kernel@vger.kernel.org" , linux-rt-users , Thomas Gleixner , Carsten Emde , John Kacur , Peter Zijlstra , Clark Williams , Ingo Molnar Subject: Re: [RFC][PATCH RT 3/4] sched/rt: Use IPI to trigger RT task push migration instead of pulling References: <20121207235615.206108556@goodmis.org> <20121208000900.613917378@goodmis.org> In-Reply-To: <20121208000900.613917378@goodmis.org> Content-Type: text/plain; charset="ISO-8859-1" Content-Transfer-Encoding: 7bit X-OriginatorOrg: am.sony.com Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On 12/07/12 15:56, Steven Rostedt wrote: > When debugging the latencies on a 40 core box, where we hit 300 to > 500 microsecond latencies, I found there was a huge contention on the > runqueue locks. > > Investigating it further, running ftrace, I found that it was due to > the pulling of RT tasks. > > The test that was run was the following: > > cyclictest --numa -p95 -m -d0 -i100 > > This created a thread on each CPU, that would set its wakeup in interations > of 100 microseconds. The -d0 means that all the threads had the same > interval (100us). Each thread sleeps for 100us and wakes up and measures > its latencies. > > What happened was another RT task would be scheduled on one of the CPUs > that was running our test, when the other CPUS test went to sleep and > scheduled idle. This cause the "pull" operation to execute on all > these CPUs. Each one of these saw the RT task that was overloaded on > the CPU of the test that was still running, and each one tried > to grab that task in a thundering herd way. > > To grab the task, each thread would do a double rq lock grab, grabbing > its own lock as well as the rq of the overloaded CPU. As the sched > domains on this box was rather flat for its size, I saw up to 12 CPUs > block on this lock at once. This caused a ripple affect with the > rq locks. As these locks were blocked, any wakeups on these CPUs > would also block on these locks, and the wait time escalated. > > I've tried various methods to lesson the load, but things like an > atomic counter to only let one CPU grab the task wont work, because > the task may have a limited affinity, and we may pick the wrong > CPU to take that lock and do the pull, to only find out that the > CPU we picked isn't in the task's affinity. You are saying that the pulling CPU might not be in the pulled task's affinity? But isn't that checked: pull_rt_task() pick_next_highest_task_rt() pick_rt_task() if ( ... || cpumask_test_cpu(cpu, tsk_cpus_allowed(p) ... > > Instead of doing the PULL, I now have the CPUs that want the pull to > send over an IPI to the overloaded CPU, and let that CPU pick what > CPU to push the task to. No more need to grab the rq lock, and the > push/pull algorithm still works fine. That gives me the opposite of a warm fuzzy feeling. Processing an IPI on the overloaded CPU is not free (I'm being ARM-centric), and this is putting more load on the already overloaded CPU. I do recognize that you have actual measurements below that show goodness for the pathological case you debugged. I'm still mulling this all over... > > With this patch, the latency dropped to just 150us over a 20 hour run. > Without the patch, the huge latencies would trigger in seconds. -Frank