From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1754164AbcAVTMA (ORCPT ); Fri, 22 Jan 2016 14:12:00 -0500 Received: from mx0a-00082601.pphosted.com ([67.231.145.42]:49168 "EHLO mx0a-00082601.pphosted.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752209AbcAVTL5 (ORCPT ); Fri, 22 Jan 2016 14:11:57 -0500 Date: Fri, 22 Jan 2016 11:11:45 -0800 From: Shaohua Li To: Tejun Heo CC: , , , , Subject: Re: [RFC 0/3] block: proportional based blk-throttling Message-ID: <20160122191144.GA2151859@devbig084.prn1.facebook.com> References: <20160121211002.GH5157@mtj.duckdns.org> <20160121222449.GA3770911@devbig084.prn1.facebook.com> <20160121224157.GL5157@mtj.duckdns.org> <20160122000015.GA4066045@devbig084.prn1.facebook.com> <20160122144822.GA32380@htj.duckdns.org> <20160122175653.GA2023129@devbig084.prn1.facebook.com> <20160122180844.GM5157@mtj.duckdns.org> MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Disposition: inline In-Reply-To: <20160122180844.GM5157@mtj.duckdns.org> User-Agent: Mutt/1.5.20 (2009-12-10) X-Originating-IP: [192.168.52.123] X-Proofpoint-Spam-Reason: safe X-FB-Internal: Safe X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10432:,, definitions=2016-01-22_07:,, signatures=0 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Fri, Jan 22, 2016 at 01:08:44PM -0500, Tejun Heo wrote: > Hello, Shaohua. > > On Fri, Jan 22, 2016 at 09:57:10AM -0800, Shaohua Li wrote: > > > Let's say per-cgroup buffer budget B is calculated as, say, 100ms > > > worth of IO cost (or bandwidth or iops) available to the cgroup. In > > > practice, this may have to be adjusted down depending on the number of > > > cgroups performing active IOs. For a given cgroup, B can be > > > distributed among the CPUs that are actively issuing IOs in that > > > cgroup. It will degenerate to round robin of small budget if there > > > are too many active for the budget available but for most cases this > > > will cut down most of cross-CPU traffic. > > > > The cgroup could be a single thread. It uses cpu0's per-cpu budget B-1, > > move to cpu1 and use another B - 1, and so on > > Sure, just ensure that the total cached is bound by B and expire if > not used over a certain amount of time. The thing is as long as we > can go through percpu cache most of the time, it's all fine. We can > spend a lot of processing budget for corner cases. > > > > cost = F + R * size > > > > F could be IOPS. and the real cost becomes R. How do you get R? We can't > > simply use R(4k) = 1, R(8k) = 2 .... I tried the idea several years ago: > > https://urldefense.proofpoint.com/v2/url?u=https-3A__lwn.net_Articles_474164_&d=CwIBAg&c=5VD0RTtNlTh3ycd41b3MUw&r=X13hAPkxmvBro1Ug8vcKHw&m=4X56EQmXhfF82BH-eQkQL08afWwbrOErtEVkn5xKsWA&s=_IkvDWMM7AXgh840OrQKndkJpBVcKrGhgLnHkA_aYNg&e= > > The idea is the same. But the reality is we can't get R. I don't want to > > have a random math working for one SSD but not for another. > > Yeah, it'll have to be adaptive. We can't use fixed values; however, > note that using bandwidth means that we assume F == 0 and R == 1, > which wouldn't be appropriate for most devices. It's true bandwidth means R == 1. But it has a kind of adaptive. The cgroup bandwidth == share * disk_bandwidth. disk_bandwidth is adaptive. It might not work well if cgroups have completely different IO pattern though. Thanks, Shaohua