From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5278DC46467 for ; Sat, 7 Jan 2023 04:44:50 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236720AbjAGEor (ORCPT ); Fri, 6 Jan 2023 23:44:47 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:53900 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229542AbjAGEon (ORCPT ); Fri, 6 Jan 2023 23:44:43 -0500 Received: from mail-pj1-x102f.google.com (mail-pj1-x102f.google.com [IPv6:2607:f8b0:4864:20::102f]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C2A2084BD6 for ; Fri, 6 Jan 2023 20:44:40 -0800 (PST) Received: by mail-pj1-x102f.google.com with SMTP id v23so3453420pju.3 for ; Fri, 06 Jan 2023 20:44:40 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20210112.gappssmtp.com; s=20210112; h=content-transfer-encoding:in-reply-to:from:references:cc:to:subject :user-agent:mime-version:date:message-id:from:to:cc:subject:date :message-id:reply-to; bh=ZeUPS2iKNE5OrN4vW8ZJQleZwh3WJYgGdDBNvXHEJYc=; b=OGcIprLg/imeJCuDYwJBLKaKHytu+oosVJno5w7Yr3wgiFgmnzXnr1Nl+0w6dZkuyj bVWz/aMUwD5CuH5eRpIOyK9e60S5jbQ8rDUhqFOksAKbNucMI6FMlI6PuVkyS847mbQS lYrSoXAVHGN8q9pRHISuHJO2Dmf1gDsJ/8tILU2M10Jm3/73zvNFOJ61PUtHJX5c7tF7 B6YLPAHPVtXAsti0eDHGXuyKa4kCHvyeoLqkTBM/b5YXCi+SMlSSvdIPh96dTo02SLVq hsWCrzmn0M0liCwxM2oJDLgOLTk9d0X2uG4kYtSCQftSFfueOy3Q65jwAGwlUV7sH4SP 8LMw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:in-reply-to:from:references:cc:to:subject :user-agent:mime-version:date:message-id:x-gm-message-state:from:to :cc:subject:date:message-id:reply-to; bh=ZeUPS2iKNE5OrN4vW8ZJQleZwh3WJYgGdDBNvXHEJYc=; b=nDaVHzS0jupfeULIZ+EFA238efNPXqV554fCy0GBX79558N0qg5M+wnDXfNRtABy8P Pt/fHBcP1esETSn9JJNjbUmsG/0iRJ4HnbBCQPh9IbzuIXq6Y0zze08losHNS97rkD+R V2Z2hdsZ3uCI1bFewEzjbyCswAh7wKbmW00J2ZyMNKE4BIbtA605AjT103G69VY9KAhk MCadGgxK0WOupU/MwzwN6oeTvcqV0LEHApd50zZXw7fu123V6+qpxzlXA2oKpWtZok60 fwSFVahreJdZde+LYYcYh93+9nwkz/ui1iTucOZONiKWysphnWACF2lkgYA65M5judhz 0Hyg== X-Gm-Message-State: AFqh2kqbRnSuEEHQ9m4PZoyQZp4zzx+Hv4ROZOO5BSrtambkLDbwBarf b1U4e4zWX4NuvHte6ZKCg3gC542ZB9JeJf+/Slv5hQ== X-Google-Smtp-Source: AMrXdXtXiqi91oaCK8p4McWzuikKF9pIEcPcAY6x5BQyBDS6NQ0R2lbIDYHL/bt6PClbzNw4gptUOw== X-Received: by 2002:a05:6a20:3d16:b0:9d:efbf:48c7 with SMTP id y22-20020a056a203d1600b0009defbf48c7mr84529944pzi.11.1673066680306; Fri, 06 Jan 2023 20:44:40 -0800 (PST) Received: from [10.255.208.163] ([139.177.225.230]) by smtp.gmail.com with ESMTPSA id t1-20020a170902d14100b00186b3c3e2dasm1722418plt.155.2023.01.06.20.44.37 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Fri, 06 Jan 2023 20:44:40 -0800 (PST) Message-ID: Date: Sat, 7 Jan 2023 12:44:35 +0800 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:102.0) Gecko/20100101 Thunderbird/102.2.2 Subject: Re: [External] Re: [PATCH v3] blk-throtl: Introduce sync and async queues for blk-throtl To: Tejun Heo Cc: Jan Kara , =?UTF-8?Q?Michal_Koutn=c3=bd?= , josef@toxicpanda.com, axboe@kernel.dk, cgroups@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, yinxin.x@bytedance.com References: <20221226130505.7186-1-hanjinke.666@bytedance.com> <20230105161854.GA1259@blackbody.suse.cz> <20230106153813.4ttyuikzaagkk2sc@quack3> From: hanjinke In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org 在 2023/1/7 上午2:15, Tejun Heo 写道: > Hello, > > On Sat, Jan 07, 2023 at 02:07:38AM +0800, hanjinke wrote: >> In our internal scenario, iocost has been deployed as the main io isolation >> method and is gradually spreading。 > > Ah, glad to hear. If you don't mind sharing, how are you configuring iocost > currently? How do you derive the parameters? > For cost.model setting, We first use the tools iocost provided to test the benchmark model parameters of different types of disks online, and then save these benchmark parameters to a parametric Model Table. During the deployment process, pull and set the corresponding model parameters according to the type of disk. The setting of cost.qos should be considered slightly more，we need to make some compromises between overall disk throughput and io latency. The average disk utilization of the entire disk on a specific business and the RLA（if it is io sensitive） of key businesses will be taken as important input considerations. The cost.qos will be dynamically fine-tuned according to the health status monitoring of key businesses. For cost.weight setting, high-priority services will gain greater advantages through weight settings to deal with a large number of io requests in a short period of time. It works fine as work-conservation of iocost works well according to our observation. These practices can be done better and I look forward to your better suggestions. > blk-throttle has a lot of issues which may be difficult to address. Even the > way it's configured is pretty difficult to scale across different hardware / > application combinations and we've neglected its control performance and > behavior (like handling of shared IOs) for quite a while. > > While iocost's work-conserving control does address a lot of the use cases > we see today, it's likely that we'll need hard limits more in the future > too. I've been thinking about implementing io.max on top of iocost. There > are some challenges around dynamic vrate adj semantics but it's kinda > attractive because iocost already has the concept of total device capacity. Indeed in our multi-tenancy scenario, the hard limits are necessary. Jinke Thanks.