From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <linux-kernel-owner@kernel.org>
X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on
	aws-us-west-2-korg-lkml-1.web.codeaurora.org
Received: from vger.kernel.org (vger.kernel.org [23.128.96.18])
	by smtp.lore.kernel.org (Postfix) with ESMTP id AF068C4332F
	for <linux-kernel@archiver.kernel.org>; Thu, 13 Oct 2022 22:45:05 +0000 (UTC)
Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand
        id S229747AbiJMWpD (ORCPT <rfc822;linux-kernel@archiver.kernel.org>);
        Thu, 13 Oct 2022 18:45:03 -0400
Received: from lindbergh.monkeyblade.net ([23.128.96.19]:32782 "EHLO
        lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org
        with ESMTP id S229460AbiJMWpB (ORCPT
        <rfc822;linux-kernel@vger.kernel.org>);
        Thu, 13 Oct 2022 18:45:01 -0400
X-Greylist: delayed 150 seconds by postgrey-1.37 at lindbergh.monkeyblade.net; Thu, 13 Oct 2022 15:44:56 PDT
Received: from resqmta-c1p-023463.sys.comcast.net (resqmta-c1p-023463.sys.comcast.net [IPv6:2001:558:fd00:56::4])
        by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 5683F45F44
        for <linux-kernel@vger.kernel.org>; Thu, 13 Oct 2022 15:44:55 -0700 (PDT)
Received: from resomta-c1p-023267.sys.comcast.net ([96.102.18.232])
        by resqmta-c1p-023463.sys.comcast.net with ESMTP
        id j6RpojqbljS9Rj6uGop7ul; Thu, 13 Oct 2022 22:42:24 +0000
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=comcastmailservice.net; s=20211018a; t=1665700944;
        bh=LV8/JcMHb+MmlhgjlVRQXW/dcI3Ijaatxb0kxbiP4wE=;
        h=Received:Received:From:To:Subject:Date:Message-Id:MIME-Version;
        b=Bf/IvOWLEP/fzD0l0knDUzF8vrErgOOJD7mWZY2FHQhI6LBMR/fPjf+eFdEltgIfL
         Vyg3e0db4ZRkJnbukq6EGFMsWcl32u2BPQxWFHjJkkDU43jKOYZuWSYV4mkF820JI0
         xSbH5xIyOmXyN+ZIB86lmccpn6HbMMBy9XultRXTUIDTJpDng6yWI2lmlP5oCo97ZQ
         Pd843NeFvHzxM6Kr0LoDNxemzP4+nMd7LM1T4FnSahmm2pT+7TqSVkNq9c+ltu1EBl
         9YbtibG3+8zXI96C9t/e2hBJ77q1y/UgxK21ogksiTsX0G3gXwfB3/slJxswyil1N3
         F92l4sdjXwMQw==
Received: from jderrick-mobl4.amr.corp.intel.com ([71.205.181.50])
        by resomta-c1p-023267.sys.comcast.net with ESMTPA
        id j6toofOVmA6uYj6tuozg6t; Thu, 13 Oct 2022 22:42:03 +0000
X-Xfinity-VAAS:  
 hsohhlihguihhgmhdrtghomhdprhgtphhtthhopehjohhnrghthhgrnhigrdhskhdruggvrhhrihgtkhesihhnthgvlhdrtghomhdprhgtphhtthhopehmrghrihhushiirdhtkhgrtgiihihksehlihhnuhigrdhinhhtvghlrdgtohhmpdhrtghpthhtohepjhhonhgrthhhrghnrdguvghrrhhitghksehlihhnuhigrdguvghv
X-Xfinity-VMeta: sc=-100.00;st=legit
From:   Jonathan Derrick <jonathan.derrick@linux.dev>
To:     Song Liu <song@kernel.org>
Cc:     <linux-raid@vger.kernel.org>, <linux-kernel@vger.kernel.org>,
        jonathan.derrick@solidigm.com, jonathanx.sk.derrick@intel.com,
        Mariusz Tkaczyk <mariusz.tkaczyk@linux.intel.com>,
        Jonathan Derrick <jonathan.derrick@linux.dev>
Subject: [PATCH v2 2/3] md/bitmap: Add sysfs interface for flush threshold
Date:   Thu, 13 Oct 2022 16:41:50 -0600
Message-Id: <20221013224151.300-3-jonathan.derrick@linux.dev>
X-Mailer: git-send-email 2.30.2
In-Reply-To: <20221013224151.300-1-jonathan.derrick@linux.dev>
References: <20221013224151.300-1-jonathan.derrick@linux.dev>
MIME-Version: 1.0
Content-Transfer-Encoding: 8bit
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

Adds a sysfs interface in the bitmap device for setting the chunk flush
threshold. This is an unsigned integer value which defines the amount of
dirty chunks allowed to be pending between bitmap flushes.

Signed-off-by: Jonathan Derrick <jonathan.derrick@linux.dev>
---
 Documentation/admin-guide/md.rst |  5 +++++
 drivers/md/md-bitmap.c           | 33 ++++++++++++++++++++++++++++++++
 2 files changed, 38 insertions(+)

diff --git a/Documentation/admin-guide/md.rst b/Documentation/admin-guide/md.rst
index d8fc9a59c086..d688ae4065cf 100644
--- a/Documentation/admin-guide/md.rst
+++ b/Documentation/admin-guide/md.rst
@@ -401,6 +401,11 @@ All md devices contain:
      once the array becomes non-degraded, and this fact has been
      recorded in the metadata.
 
+  bitmap/flush_threshold
+     The number of outstanding dirty chunks that are allowed to be pending
+     before unplugging the bitmap queue. The default behavior is to always
+     unplugging the queue when requested.
+
   consistency_policy
      This indicates how the array maintains consistency in case of unexpected
      shutdown. It can be:
diff --git a/drivers/md/md-bitmap.c b/drivers/md/md-bitmap.c
index c5c77f8371a8..cd8250368860 100644
--- a/drivers/md/md-bitmap.c
+++ b/drivers/md/md-bitmap.c
@@ -2652,6 +2652,38 @@ static struct md_sysfs_entry max_backlog_used =
 __ATTR(max_backlog_used, S_IRUGO | S_IWUSR,
        behind_writes_used_show, behind_writes_used_reset);
 
+static ssize_t
+bitmap_flush_threshold_show(struct mddev *mddev, char *page)
+{
+	ssize_t ret;
+	spin_lock(&mddev->lock);
+	if (mddev->bitmap == NULL)
+		ret = sprintf(page, "0\n");
+	else
+		ret = sprintf(page, "%u\n",
+			      mddev->bitmap_info.flush_threshold);
+	spin_unlock(&mddev->lock);
+	return ret;
+}
+
+static ssize_t
+bitmap_flush_threshold_store(struct mddev *mddev, const char *buf, size_t len)
+{
+	unsigned int thresh;
+	int ret;
+	if (!mddev->bitmap)
+		return -ENOENT;
+	ret = kstrtouint(buf, 10, &thresh);
+	if (ret)
+		return ret;
+	mddev->bitmap_info.flush_threshold = thresh;
+	return len;
+}
+
+static struct md_sysfs_entry bitmap_flush_threshold =
+__ATTR(flush_threshold, S_IRUGO | S_IWUSR,
+       bitmap_flush_threshold_show, bitmap_flush_threshold_store);
+
 static struct attribute *md_bitmap_attrs[] = {
 	&bitmap_location.attr,
 	&bitmap_space.attr,
@@ -2661,6 +2693,7 @@ static struct attribute *md_bitmap_attrs[] = {
 	&bitmap_metadata.attr,
 	&bitmap_can_clear.attr,
 	&max_backlog_used.attr,
+	&bitmap_flush_threshold.attr,
 	NULL
 };
 const struct attribute_group md_bitmap_group = {
-- 
2.31.1


From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <linux-kernel-owner@kernel.org>
X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on
	aws-us-west-2-korg-lkml-1.web.codeaurora.org
Received: from vger.kernel.org (vger.kernel.org [23.128.96.18])
	by smtp.lore.kernel.org (Postfix) with ESMTP id 8EC58C43217
	for <linux-kernel@archiver.kernel.org>; Thu, 13 Oct 2022 22:45:11 +0000 (UTC)
Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand
        id S229823AbiJMWpJ (ORCPT <rfc822;linux-kernel@archiver.kernel.org>);
        Thu, 13 Oct 2022 18:45:09 -0400
Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33308 "EHLO
        lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org
        with ESMTP id S229771AbiJMWpE (ORCPT
        <rfc822;linux-kernel@vger.kernel.org>);
        Thu, 13 Oct 2022 18:45:04 -0400
Received: from resqmta-c1p-023465.sys.comcast.net (resqmta-c1p-023465.sys.comcast.net [IPv6:2001:558:fd00:56::5])
        by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B46755F234
        for <linux-kernel@vger.kernel.org>; Thu, 13 Oct 2022 15:44:56 -0700 (PDT)
Received: from resomta-c1p-023267.sys.comcast.net ([96.102.18.232])
        by resqmta-c1p-023465.sys.comcast.net with ESMTP
        id j2WSoDUqzf0y0j6uGoIgBW; Thu, 13 Oct 2022 22:42:24 +0000
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=comcastmailservice.net; s=20211018a; t=1665700944;
        bh=uV9jv29H6aFjQ0ZSoY1eBV2TaHPRxfvEQYTsQ4HbTmo=;
        h=Received:Received:From:To:Subject:Date:Message-Id:MIME-Version;
        b=pkixH37in8eAtc+ANM7ONsju8O0nl69xc2fbW3o9jM11zq3Z4fkA0wCOI59y1/RCl
         Hpzt8pT/j8XmFzjczzffsJuYoiHEVK4gI3mMJ64PS1rkHq1mQ7WLlb0Jq8XfqcuWDB
         eMKgePlLoZP+aPHhm7Z7pW/IMg0oo4BKcP2bqGh3V8xbPwqQMmVBNltvrIUWeik2mU
         Wf9zXB1kEkRmtl5YdVH0qLGIbwSdV/aEjH4bDCLl/aMNV9u4pploA3xOd0WhDegXeU
         xna7IeSPPhE5P2rQd6e/N0o+gw5F/aWYPndf9oJk8dmmcfb9xGWcL7DWrMFYVr+DgX
         krdoBC0fbd4gw==
Received: from jderrick-mobl4.amr.corp.intel.com ([71.205.181.50])
        by resomta-c1p-023267.sys.comcast.net with ESMTPA
        id j6toofOVmA6uYj6ttozg6o; Thu, 13 Oct 2022 22:42:02 +0000
X-Xfinity-VAAS:  
 hlihguihhgmhdrtghomhdprhgtphhtthhopehjohhnrghthhgrnhigrdhskhdruggvrhhrihgtkhesihhnthgvlhdrtghomhdprhgtphhtthhopehmrghrihhushiirdhtkhgrtgiihihksehlihhnuhigrdhinhhtvghlrdgtohhmpdhrtghpthhtohepjhhonhgrthhhrghnrdguvghrrhhitghksehlihhnuhigrdguvghv
X-Xfinity-VMeta: sc=-100.00;st=legit
From:   Jonathan Derrick <jonathan.derrick@linux.dev>
To:     Song Liu <song@kernel.org>
Cc:     <linux-raid@vger.kernel.org>, <linux-kernel@vger.kernel.org>,
        jonathan.derrick@solidigm.com, jonathanx.sk.derrick@intel.com,
        Mariusz Tkaczyk <mariusz.tkaczyk@linux.intel.com>,
        Jonathan Derrick <jonathan.derrick@linux.dev>
Subject: [PATCH v2 0/3] Bitmap percentage flushing
Date:   Thu, 13 Oct 2022 16:41:48 -0600
Message-Id: <20221013224151.300-1-jonathan.derrick@linux.dev>
X-Mailer: git-send-email 2.30.2
MIME-Version: 1.0
Content-Transfer-Encoding: 8bit
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

This introduces a percentage-flushing mechanism that works in-tandem to the
mdadm delay timer. The percentage argument is based on the number of chunks
dirty (rather than percentage), due to large drives requiring smaller and
smaller percentages (eg, 32TB drives-> 1% is 320GB).

This set hopes to provide a way to make the bitmap flushing more consistent. It
was observed that a synchronous, random write qd1 workload, could make bitmap
writes easily become almost half of the I/O. And in similar workloads with
different timing, it was several minutes between bitmap updates. This is too
inconsistent to be reliable.

This first and second patches adds the flush_threshold parameter. The default
value of 0 defines the default behavior: unplugging immediately just as before.
With a flush-threshold value of 1, it becomes more consistent and paranoid,
flushing on nearly every I/O, leading to a 40% or greater situation. From
there, the flush_threshold can be defined higher for those situations where
power loss is rare and full resync can be tolerated.

The third patch converts the daemon worker to an actual timer. This makes it
more consistent and removes some ugly code.

Jonathan Derrick (3):
  md/bitmap: Add chunk-threshold unplugging
  md/bitmap: Add sysfs interface for flush threshold
  md/bitmap: Convert daemon_work to proper timer

 Documentation/admin-guide/md.rst |  5 ++
 drivers/md/md-bitmap.c           | 98 +++++++++++++++++++++++++-------
 drivers/md/md-bitmap.h           |  4 +-
 drivers/md/md.c                  |  9 ++-
 drivers/md/md.h                  |  2 +
 5 files changed, 93 insertions(+), 25 deletions(-)

-- 
2.31.1


From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <linux-kernel-owner@kernel.org>
X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on
	aws-us-west-2-korg-lkml-1.web.codeaurora.org
Received: from vger.kernel.org (vger.kernel.org [23.128.96.18])
	by smtp.lore.kernel.org (Postfix) with ESMTP id AFA0EC4332F
	for <linux-kernel@archiver.kernel.org>; Thu, 13 Oct 2022 22:45:14 +0000 (UTC)
Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand
        id S229836AbiJMWpN (ORCPT <rfc822;linux-kernel@archiver.kernel.org>);
        Thu, 13 Oct 2022 18:45:13 -0400
Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33300 "EHLO
        lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org
        with ESMTP id S229773AbiJMWpE (ORCPT
        <rfc822;linux-kernel@vger.kernel.org>);
        Thu, 13 Oct 2022 18:45:04 -0400
Received: from resqmta-c1p-023462.sys.comcast.net (resqmta-c1p-023462.sys.comcast.net [IPv6:2001:558:fd00:56::2])
        by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B454A5B05E
        for <linux-kernel@vger.kernel.org>; Thu, 13 Oct 2022 15:44:56 -0700 (PDT)
Received: from resomta-c1p-023267.sys.comcast.net ([96.102.18.232])
        by resqmta-c1p-023462.sys.comcast.net with ESMTP
        id j6RpomdrT03bPj6uGoD3ZD; Thu, 13 Oct 2022 22:42:24 +0000
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=comcastmailservice.net; s=20211018a; t=1665700944;
        bh=Vod9bfBwBD3lRXXsfZrFXG7EXQYLJ2cHVyFKENUAAzU=;
        h=Received:Received:From:To:Subject:Date:Message-Id:MIME-Version;
        b=J690zIjBgP0kPbDXXI+Neca+cVOheOOsmD6EbSP2jjoIDXf24bi9A40PBQMNAfYAH
         f3ELOCc4QbG3JzZTRCawGr1f4VqxcbPv8bWvI7LdOQ7NA4pW4kr+cYBR1qRd7e8N7h
         SWxTvgvbnZGxVGIIY+xzhNAgIglF793NBeWjsZ18UNXvMfes1vR6aBOYJNKA3Iaujc
         PpS/wBZTN3TJYdw6xQCk4McWlLRyqQRnvLh05sSWN9eWE7D7SDFhx4OtFX8QfAJxEf
         +jjzrTZfjKbqKUe2JuW3OwHxsMZSxj6KNnSpKdqkVLQTuDCtotW3xdvj9qzwVL8NQF
         Kg7TmB2D1jFqA==
Received: from jderrick-mobl4.amr.corp.intel.com ([71.205.181.50])
        by resomta-c1p-023267.sys.comcast.net with ESMTPA
        id j6toofOVmA6uYj6tvozg6y; Thu, 13 Oct 2022 22:42:04 +0000
X-Xfinity-VAAS:  
 hsohhlihguihhgmhdrtghomhdprhgtphhtthhopehjohhnrghthhgrnhigrdhskhdruggvrhhrihgtkhesihhnthgvlhdrtghomhdprhgtphhtthhopehmrghrihhushiirdhtkhgrtgiihihksehlihhnuhigrdhinhhtvghlrdgtohhmpdhrtghpthhtohepjhhonhgrthhhrghnrdguvghrrhhitghksehlihhnuhigrdguvghv
X-Xfinity-VMeta: sc=-100.00;st=legit
From:   Jonathan Derrick <jonathan.derrick@linux.dev>
To:     Song Liu <song@kernel.org>
Cc:     <linux-raid@vger.kernel.org>, <linux-kernel@vger.kernel.org>,
        jonathan.derrick@solidigm.com, jonathanx.sk.derrick@intel.com,
        Mariusz Tkaczyk <mariusz.tkaczyk@linux.intel.com>,
        Jonathan Derrick <jonathan.derrick@linux.dev>
Subject: [PATCH v2 3/3] md/bitmap: Convert daemon_work to proper timer
Date:   Thu, 13 Oct 2022 16:41:51 -0600
Message-Id: <20221013224151.300-4-jonathan.derrick@linux.dev>
X-Mailer: git-send-email 2.30.2
In-Reply-To: <20221013224151.300-1-jonathan.derrick@linux.dev>
References: <20221013224151.300-1-jonathan.derrick@linux.dev>
MIME-Version: 1.0
Content-Transfer-Encoding: 8bit
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

It was observed that with certain high I/O workloads that the daemon
work may never run, preventing the bitmap from fully flushing. Ensure
the bitmap fully flushes by converting the daemon worker to a proper
timer.

Signed-off-by: Jonathan Derrick <jonathan.derrick@linux.dev>
---
 drivers/md/md-bitmap.c | 30 +++++++++++++-----------------
 drivers/md/md-bitmap.h |  3 +--
 drivers/md/md.c        |  9 +++++++--
 drivers/md/md.h        |  1 +
 4 files changed, 22 insertions(+), 21 deletions(-)

diff --git a/drivers/md/md-bitmap.c b/drivers/md/md-bitmap.c
index cd8250368860..34feb906243f 100644
--- a/drivers/md/md-bitmap.c
+++ b/drivers/md/md-bitmap.c
@@ -1250,8 +1250,9 @@ static bitmap_counter_t *md_bitmap_get_counter(struct bitmap_counts *bitmap,
  *			out to disk
  */
 
-void md_bitmap_daemon_work(struct mddev *mddev)
+void md_bitmap_daemon_work(struct timer_list *t)
 {
+	struct mddev *mddev = from_timer(mddev, t, daemon_timer);
 	struct bitmap *bitmap;
 	unsigned long j;
 	unsigned long nextpage;
@@ -1267,11 +1268,7 @@ void md_bitmap_daemon_work(struct mddev *mddev)
 		mutex_unlock(&mddev->bitmap_info.mutex);
 		return;
 	}
-	if (time_before(jiffies, bitmap->daemon_lastrun
-			+ mddev->bitmap_info.daemon_sleep))
-		goto done;
 
-	bitmap->daemon_lastrun = jiffies;
 	if (bitmap->allclean) {
 		mddev->thread->timeout = MAX_SCHEDULE_TIMEOUT;
 		goto done;
@@ -1372,6 +1369,7 @@ void md_bitmap_daemon_work(struct mddev *mddev)
 	if (bitmap->allclean == 0)
 		mddev->thread->timeout =
 			mddev->bitmap_info.daemon_sleep;
+	mod_timer(&mddev->daemon_timer, jiffies + mddev->bitmap_info.daemon_sleep);
 	mutex_unlock(&mddev->bitmap_info.mutex);
 }
 
@@ -1735,21 +1733,16 @@ void md_bitmap_dirty_bits(struct bitmap *bitmap, unsigned long s, unsigned long
 void md_bitmap_flush(struct mddev *mddev)
 {
 	struct bitmap *bitmap = mddev->bitmap;
-	long sleep;
 
 	if (!bitmap) /* there was no bitmap */
 		return;
 
 	/* run the daemon_work three time to ensure everything is flushed
-	 * that can be
-	 */
-	sleep = mddev->bitmap_info.daemon_sleep * 2;
-	bitmap->daemon_lastrun -= sleep;
-	md_bitmap_daemon_work(mddev);
-	bitmap->daemon_lastrun -= sleep;
-	md_bitmap_daemon_work(mddev);
-	bitmap->daemon_lastrun -= sleep;
-	md_bitmap_daemon_work(mddev);
+	* that can be
+	*/
+	md_bitmap_daemon_work(&mddev->daemon_timer);
+	md_bitmap_daemon_work(&mddev->daemon_timer);
+	md_bitmap_daemon_work(&mddev->daemon_timer);
 	if (mddev->bitmap_info.external)
 		md_super_wait(mddev);
 	md_bitmap_update_sb(bitmap);
@@ -1826,7 +1819,7 @@ void md_bitmap_destroy(struct mddev *mddev)
 	mutex_unlock(&mddev->bitmap_info.mutex);
 	if (mddev->thread)
 		mddev->thread->timeout = MAX_SCHEDULE_TIMEOUT;
-
+	del_timer_sync(&mddev->daemon_timer);
 	md_bitmap_free(bitmap);
 }
 
@@ -1904,7 +1897,10 @@ struct bitmap *md_bitmap_create(struct mddev *mddev, int slot)
 	if (err)
 		goto error;
 
-	bitmap->daemon_lastrun = jiffies;
+	timer_setup(&mddev->daemon_timer, md_bitmap_daemon_work, 0);
+	mddev->daemon_timer.expires = jiffies + mddev->bitmap_info.daemon_sleep;
+	add_timer(&mddev->daemon_timer);
+
 	err = md_bitmap_resize(bitmap, blocks, mddev->bitmap_info.chunksize, 1);
 	if (err)
 		goto error;
diff --git a/drivers/md/md-bitmap.h b/drivers/md/md-bitmap.h
index 49a93d8ff307..b7e8f2266bf2 100644
--- a/drivers/md/md-bitmap.h
+++ b/drivers/md/md-bitmap.h
@@ -220,7 +220,6 @@ struct bitmap {
 	 * the bitmap daemon - periodically wakes up and sweeps the bitmap
 	 * file, cleaning up bits and flushing out pages to disk as necessary
 	 */
-	unsigned long daemon_lastrun; /* jiffies of last run */
 	unsigned long last_end_sync; /* when we lasted called end_sync to
 				      * update bitmap with resync progress */
 	unsigned long unplugged_count; /* last dirty count from md_bitmap_unplug */
@@ -265,7 +264,7 @@ void md_bitmap_sync_with_cluster(struct mddev *mddev,
 				 sector_t new_lo, sector_t new_hi);
 
 void md_bitmap_unplug(struct bitmap *bitmap);
-void md_bitmap_daemon_work(struct mddev *mddev);
+void md_bitmap_daemon_work(struct timer_list *t);
 
 int md_bitmap_resize(struct bitmap *bitmap, sector_t blocks,
 		     int chunksize, int init);
diff --git a/drivers/md/md.c b/drivers/md/md.c
index afaf36b2f6ab..9f8a9f62b3db 100644
--- a/drivers/md/md.c
+++ b/drivers/md/md.c
@@ -9265,8 +9265,13 @@ void md_check_recovery(struct mddev *mddev)
 	if (mddev->suspended)
 		return;
 
-	if (mddev->bitmap)
-		md_bitmap_daemon_work(mddev);
+	if (mddev->bitmap) {
+		spin_lock(&pers_lock);
+		if (mddev->bitmap->allclean == 0)
+			mddev->thread->timeout =
+				mddev->bitmap_info.daemon_sleep;
+		spin_unlock(&pers_lock);
+	}
 
 	if (signal_pending(current)) {
 		if (mddev->pers->sync_request && !mddev->external) {
diff --git a/drivers/md/md.h b/drivers/md/md.h
index 1a558cb18bd4..578cc496c325 100644
--- a/drivers/md/md.h
+++ b/drivers/md/md.h
@@ -503,6 +503,7 @@ struct mddev {
 		char                    cluster_name[64]; /* Name of the cluster */
 		unsigned int		flush_threshold; /* how many dirty chunks between updates */
 	} bitmap_info;
+	struct timer_list		daemon_timer;
 
 	atomic_t			max_corr_read_errors; /* max read retries */
 	struct list_head		all_mddevs;
-- 
2.31.1


From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <linux-kernel-owner@kernel.org>
X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on
	aws-us-west-2-korg-lkml-1.web.codeaurora.org
Received: from vger.kernel.org (vger.kernel.org [23.128.96.18])
	by smtp.lore.kernel.org (Postfix) with ESMTP id 6EDF7C4332F
	for <linux-kernel@archiver.kernel.org>; Thu, 13 Oct 2022 22:45:23 +0000 (UTC)
Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand
        id S229852AbiJMWpS (ORCPT <rfc822;linux-kernel@archiver.kernel.org>);
        Thu, 13 Oct 2022 18:45:18 -0400
Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33302 "EHLO
        lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org
        with ESMTP id S229776AbiJMWpE (ORCPT
        <rfc822;linux-kernel@vger.kernel.org>);
        Thu, 13 Oct 2022 18:45:04 -0400
Received: from resqmta-c1p-024063.sys.comcast.net (resqmta-c1p-024063.sys.comcast.net [IPv6:2001:558:fd00:56::8])
        by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B45F65C96E
        for <linux-kernel@vger.kernel.org>; Thu, 13 Oct 2022 15:44:56 -0700 (PDT)
Received: from resomta-c1p-023267.sys.comcast.net ([96.102.18.232])
        by resqmta-c1p-024063.sys.comcast.net with ESMTP
        id j5u4o8WmbopRHj6uGozIg7; Thu, 13 Oct 2022 22:42:24 +0000
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=comcastmailservice.net; s=20211018a; t=1665700944;
        bh=RmsPdp6XB2/XHXf0fJ7VeuVLgpyyn/dl4lgDaA8Nfi4=;
        h=Received:Received:From:To:Subject:Date:Message-Id:MIME-Version;
        b=o+aFDZmt1HIJr8SdJmyUSe1F6OhddikpPnTNE/7TWKuY8ZeOlKKMXW8mb6qExDBmg
         Ttlz1va1XslPHu/tqqsY2297fd1RWO0KlPDdjY8ql8JSq4zd9Vo8mmoBaDRHA2gfEF
         QyUIWDPw6ru4Pb81MLiDsC5QHYxnPXpg+p5Mvl10BT90kN5pL244H06/i0DjOO5lDa
         BSfSPGFGgKEFr3SQc73vO9wlddOHLttZ3jDC2szN8r1N6f3C31ZXIJqA6CZ1RxmkrI
         vF2aWwE0qOB8BR1TSu3x0O+QL8kvymFjzy2GJVcqWQD2ZL9EfJGL+kW8NLGHqDAwoF
         ct0H9LC9SYUxw==
Received: from jderrick-mobl4.amr.corp.intel.com ([71.205.181.50])
        by resomta-c1p-023267.sys.comcast.net with ESMTPA
        id j6toofOVmA6uYj6tuozg6q; Thu, 13 Oct 2022 22:42:02 +0000
X-Xfinity-VAAS:  
 hsohhlihguihhgmhdrtghomhdprhgtphhtthhopehjohhnrghthhgrnhigrdhskhdruggvrhhrihgtkhesihhnthgvlhdrtghomhdprhgtphhtthhopehmrghrihhushiirdhtkhgrtgiihihksehlihhnuhigrdhinhhtvghlrdgtohhmpdhrtghpthhtohepjhhonhgrthhhrghnrdguvghrrhhitghksehlihhnuhigrdguvghv
X-Xfinity-VMeta: sc=-100.00;st=legit
From:   Jonathan Derrick <jonathan.derrick@linux.dev>
To:     Song Liu <song@kernel.org>
Cc:     <linux-raid@vger.kernel.org>, <linux-kernel@vger.kernel.org>,
        jonathan.derrick@solidigm.com, jonathanx.sk.derrick@intel.com,
        Mariusz Tkaczyk <mariusz.tkaczyk@linux.intel.com>,
        Jonathan Derrick <jonathan.derrick@linux.dev>
Subject: [PATCH v2 1/3] md/bitmap: Add chunk-threshold unplugging
Date:   Thu, 13 Oct 2022 16:41:49 -0600
Message-Id: <20221013224151.300-2-jonathan.derrick@linux.dev>
X-Mailer: git-send-email 2.30.2
In-Reply-To: <20221013224151.300-1-jonathan.derrick@linux.dev>
References: <20221013224151.300-1-jonathan.derrick@linux.dev>
MIME-Version: 1.0
Content-Transfer-Encoding: 8bit
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

Add a mechanism to allow bitmap unplugging and flushing to wait until it
has surpassed a defined threshold of dirty chunks. This allows certain
high I/O write workloads to make good forward progress between bitmap
updates or provide reliable bitmap consistency. The default behavior is
previous behavior of always unplugging when called.

Signed-off-by: Jonathan Derrick <jonathan.derrick@linux.dev>
---
 drivers/md/md-bitmap.c | 35 +++++++++++++++++++++++++++++++----
 drivers/md/md-bitmap.h |  1 +
 drivers/md/md.h        |  1 +
 3 files changed, 33 insertions(+), 4 deletions(-)

diff --git a/drivers/md/md-bitmap.c b/drivers/md/md-bitmap.c
index bf6dffadbe6f..c5c77f8371a8 100644
--- a/drivers/md/md-bitmap.c
+++ b/drivers/md/md-bitmap.c
@@ -1004,7 +1004,7 @@ static int md_bitmap_file_test_bit(struct bitmap *bitmap, sector_t block)
 /* this gets called when the md device is ready to unplug its underlying
  * (slave) device queues -- before we let any writes go down, we need to
  * sync the dirty pages of the bitmap file to disk */
-void md_bitmap_unplug(struct bitmap *bitmap)
+static void __md_bitmap_unplug(struct bitmap *bitmap)
 {
 	unsigned long i;
 	int dirty, need_write;
@@ -1038,6 +1038,33 @@ void md_bitmap_unplug(struct bitmap *bitmap)
 	if (test_bit(BITMAP_WRITE_ERROR, &bitmap->flags))
 		md_bitmap_file_kick(bitmap);
 }
+
+/*
+ * Conditional unplug based on user-defined parameter
+ * Defaults to unconditional behavior
+ */
+void md_bitmap_unplug(struct bitmap *bitmap)
+{
+	unsigned int flush_threshold = bitmap->mddev->bitmap_info.flush_threshold;
+
+	if (!flush_threshold) {
+		__md_bitmap_unplug(bitmap);
+	} else {
+		struct bitmap_page *bp = bitmap->counts.bp;
+		unsigned long pages = bitmap->counts.pages;
+		unsigned long k, count = 0;
+
+		for (k = 0; k < pages; k++)
+			if (bp[k].map && !bp[k].hijacked)
+				count += bp[k].count;
+
+		if (count - bitmap->unplugged_count > flush_threshold) {
+			bitmap->unplugged_count = count;
+			md_bitmap_daemon_work(&bitmap->mddev->daemon_timer);
+			__md_bitmap_unplug(bitmap);
+		}
+	}
+}
 EXPORT_SYMBOL(md_bitmap_unplug);
 
 static void md_bitmap_set_memory_bits(struct bitmap *bitmap, sector_t offset, int needed);
@@ -2012,9 +2039,9 @@ int md_bitmap_copy_from_slot(struct mddev *mddev, int slot,
 		for (i = 0; i < bitmap->storage.file_pages; i++)
 			if (test_page_attr(bitmap, i, BITMAP_PAGE_PENDING))
 				set_page_attr(bitmap, i, BITMAP_PAGE_NEEDWRITE);
-		md_bitmap_unplug(bitmap);
+		__md_bitmap_unplug(bitmap);
 	}
-	md_bitmap_unplug(mddev->bitmap);
+	__md_bitmap_unplug(mddev->bitmap);
 	*low = lo;
 	*high = hi;
 	md_bitmap_free(bitmap);
@@ -2246,7 +2273,7 @@ int md_bitmap_resize(struct bitmap *bitmap, sector_t blocks,
 	spin_unlock_irq(&bitmap->counts.lock);
 
 	if (!init) {
-		md_bitmap_unplug(bitmap);
+		__md_bitmap_unplug(bitmap);
 		bitmap->mddev->pers->quiesce(bitmap->mddev, 0);
 	}
 	ret = 0;
diff --git a/drivers/md/md-bitmap.h b/drivers/md/md-bitmap.h
index cfd7395de8fd..49a93d8ff307 100644
--- a/drivers/md/md-bitmap.h
+++ b/drivers/md/md-bitmap.h
@@ -223,6 +223,7 @@ struct bitmap {
 	unsigned long daemon_lastrun; /* jiffies of last run */
 	unsigned long last_end_sync; /* when we lasted called end_sync to
 				      * update bitmap with resync progress */
+	unsigned long unplugged_count; /* last dirty count from md_bitmap_unplug */
 
 	atomic_t pending_writes; /* pending writes to the bitmap file */
 	wait_queue_head_t write_wait;
diff --git a/drivers/md/md.h b/drivers/md/md.h
index b4e2d8b87b61..1a558cb18bd4 100644
--- a/drivers/md/md.h
+++ b/drivers/md/md.h
@@ -501,6 +501,7 @@ struct mddev {
 		int			external;
 		int			nodes; /* Maximum number of nodes in the cluster */
 		char                    cluster_name[64]; /* Name of the cluster */
+		unsigned int		flush_threshold; /* how many dirty chunks between updates */
 	} bitmap_info;
 
 	atomic_t			max_corr_read_errors; /* max read retries */
-- 
2.31.1


From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <linux-kernel-owner@kernel.org>
X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on
	aws-us-west-2-korg-lkml-1.web.codeaurora.org
Received: from vger.kernel.org (vger.kernel.org [23.128.96.18])
	by smtp.lore.kernel.org (Postfix) with ESMTP id 480B6C433FE
	for <linux-kernel@archiver.kernel.org>; Fri, 14 Oct 2022 01:11:25 +0000 (UTC)
Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand
        id S229733AbiJNBLT (ORCPT <rfc822;linux-kernel@archiver.kernel.org>);
        Thu, 13 Oct 2022 21:11:19 -0400
Received: from lindbergh.monkeyblade.net ([23.128.96.19]:56948 "EHLO
        lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org
        with ESMTP id S229491AbiJNBLP (ORCPT
        <rfc822;linux-kernel@vger.kernel.org>);
        Thu, 13 Oct 2022 21:11:15 -0400
Received: from out2.migadu.com (out2.migadu.com [IPv6:2001:41d0:2:aacc::])
        by lindbergh.monkeyblade.net (Postfix) with ESMTPS id DAE7912B35B;
        Thu, 13 Oct 2022 18:11:12 -0700 (PDT)
Message-ID: <e8509e11-3a40-09ba-330e-1e8371eac323@linux.dev>
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1;
        t=1665709870;
        h=from:from:reply-to:subject:subject:date:date:message-id:message-id:
         to:to:cc:cc:mime-version:mime-version:content-type:content-type:
         content-transfer-encoding:content-transfer-encoding:
         in-reply-to:in-reply-to:references:references;
        bh=qxNGLneyLXiEIftrmWJ7UR9IFJikkY+SVPP4K4jC414=;
        b=AQcOVBmmnWzup8jej9bMok8KskVHNuJTxaJ5yoetWfZH6A1UApc8jvAxMXhruVHWkTERJJ
        vErOe0yRFjEkQbGatCDV58uqb3MI3FXFUAO58yf/+dMGYHjve1ts/3EMOesOkyP1Zjb/2C
        B3tFEnJpjT/3KWiMgNDRsyaEsIYLcrE=
Date:   Thu, 13 Oct 2022 19:11:06 -0600
MIME-Version: 1.0
Subject: Re: [PATCH v2 1/3] md/bitmap: Add chunk-threshold unplugging
Content-Language: en-US
To:     Song Liu <song@kernel.org>
Cc:     linux-raid@vger.kernel.org, linux-kernel@vger.kernel.org,
        jonathan.derrick@solidigm.com, jonathanx.sk.derrick@intel.com,
        Mariusz Tkaczyk <mariusz.tkaczyk@linux.intel.com>
References: <20221013224151.300-1-jonathan.derrick@linux.dev>
 <20221013224151.300-2-jonathan.derrick@linux.dev>
X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers.
From:   Jonathan Derrick <jonathan.derrick@linux.dev>
In-Reply-To: <20221013224151.300-2-jonathan.derrick@linux.dev>
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit
X-Migadu-Flow: FLOW_OUT
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org



On 10/13/2022 4:41 PM, Jonathan Derrick wrote:
> Add a mechanism to allow bitmap unplugging and flushing to wait until it
> has surpassed a defined threshold of dirty chunks. This allows certain
> high I/O write workloads to make good forward progress between bitmap
> updates or provide reliable bitmap consistency. The default behavior is
> previous behavior of always unplugging when called.
> 
> Signed-off-by: Jonathan Derrick <jonathan.derrick@linux.dev>
> ---
>  drivers/md/md-bitmap.c | 35 +++++++++++++++++++++++++++++++----
>  drivers/md/md-bitmap.h |  1 +
>  drivers/md/md.h        |  1 +
>  3 files changed, 33 insertions(+), 4 deletions(-)
> 
> diff --git a/drivers/md/md-bitmap.c b/drivers/md/md-bitmap.c
> index bf6dffadbe6f..c5c77f8371a8 100644
> --- a/drivers/md/md-bitmap.c
> +++ b/drivers/md/md-bitmap.c
> @@ -1004,7 +1004,7 @@ static int md_bitmap_file_test_bit(struct bitmap *bitmap, sector_t block)
>  /* this gets called when the md device is ready to unplug its underlying
>   * (slave) device queues -- before we let any writes go down, we need to
>   * sync the dirty pages of the bitmap file to disk */
> -void md_bitmap_unplug(struct bitmap *bitmap)
> +static void __md_bitmap_unplug(struct bitmap *bitmap)
>  {
>  	unsigned long i;
>  	int dirty, need_write;
> @@ -1038,6 +1038,33 @@ void md_bitmap_unplug(struct bitmap *bitmap)
>  	if (test_bit(BITMAP_WRITE_ERROR, &bitmap->flags))
>  		md_bitmap_file_kick(bitmap);
>  }
> +
> +/*
> + * Conditional unplug based on user-defined parameter
> + * Defaults to unconditional behavior
> + */
> +void md_bitmap_unplug(struct bitmap *bitmap)
> +{
> +	unsigned int flush_threshold = bitmap->mddev->bitmap_info.flush_threshold;
> +
> +	if (!flush_threshold) {
> +		__md_bitmap_unplug(bitmap);
> +	} else {
> +		struct bitmap_page *bp = bitmap->counts.bp;
> +		unsigned long pages = bitmap->counts.pages;
> +		unsigned long k, count = 0;
> +
> +		for (k = 0; k < pages; k++)
> +			if (bp[k].map && !bp[k].hijacked)
> +				count += bp[k].count;
> +
> +		if (count - bitmap->unplugged_count > flush_threshold) {
> +			bitmap->unplugged_count = count;
> +			md_bitmap_daemon_work(&bitmap->mddev->daemon_timer);
I just noticed I call daemon_timer before adding it in 3/3
I'll fix that in v3

> +			__md_bitmap_unplug(bitmap);
> +		}
> +	}
> +}
>  EXPORT_SYMBOL(md_bitmap_unplug);
>  
>  static void md_bitmap_set_memory_bits(struct bitmap *bitmap, sector_t offset, int needed);
> @@ -2012,9 +2039,9 @@ int md_bitmap_copy_from_slot(struct mddev *mddev, int slot,
>  		for (i = 0; i < bitmap->storage.file_pages; i++)
>  			if (test_page_attr(bitmap, i, BITMAP_PAGE_PENDING))
>  				set_page_attr(bitmap, i, BITMAP_PAGE_NEEDWRITE);
> -		md_bitmap_unplug(bitmap);
> +		__md_bitmap_unplug(bitmap);
>  	}
> -	md_bitmap_unplug(mddev->bitmap);
> +	__md_bitmap_unplug(mddev->bitmap);
>  	*low = lo;
>  	*high = hi;
>  	md_bitmap_free(bitmap);
> @@ -2246,7 +2273,7 @@ int md_bitmap_resize(struct bitmap *bitmap, sector_t blocks,
>  	spin_unlock_irq(&bitmap->counts.lock);
>  
>  	if (!init) {
> -		md_bitmap_unplug(bitmap);
> +		__md_bitmap_unplug(bitmap);
>  		bitmap->mddev->pers->quiesce(bitmap->mddev, 0);
>  	}
>  	ret = 0;
> diff --git a/drivers/md/md-bitmap.h b/drivers/md/md-bitmap.h
> index cfd7395de8fd..49a93d8ff307 100644
> --- a/drivers/md/md-bitmap.h
> +++ b/drivers/md/md-bitmap.h
> @@ -223,6 +223,7 @@ struct bitmap {
>  	unsigned long daemon_lastrun; /* jiffies of last run */
>  	unsigned long last_end_sync; /* when we lasted called end_sync to
>  				      * update bitmap with resync progress */
> +	unsigned long unplugged_count; /* last dirty count from md_bitmap_unplug */
>  
>  	atomic_t pending_writes; /* pending writes to the bitmap file */
>  	wait_queue_head_t write_wait;
> diff --git a/drivers/md/md.h b/drivers/md/md.h
> index b4e2d8b87b61..1a558cb18bd4 100644
> --- a/drivers/md/md.h
> +++ b/drivers/md/md.h
> @@ -501,6 +501,7 @@ struct mddev {
>  		int			external;
>  		int			nodes; /* Maximum number of nodes in the cluster */
>  		char                    cluster_name[64]; /* Name of the cluster */
> +		unsigned int		flush_threshold; /* how many dirty chunks between updates */
>  	} bitmap_info;
>  
>  	atomic_t			max_corr_read_errors; /* max read retries */

From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <linux-kernel-owner@kernel.org>
X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on
	aws-us-west-2-korg-lkml-1.web.codeaurora.org
Received: from vger.kernel.org (vger.kernel.org [23.128.96.18])
	by smtp.lore.kernel.org (Postfix) with ESMTP id AF434C43217
	for <linux-kernel@archiver.kernel.org>; Fri, 14 Oct 2022 21:19:21 +0000 (UTC)
Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand
        id S229832AbiJNVTT (ORCPT <rfc822;linux-kernel@archiver.kernel.org>);
        Fri, 14 Oct 2022 17:19:19 -0400
Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39644 "EHLO
        lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org
        with ESMTP id S229826AbiJNVTR (ORCPT
        <rfc822;linux-kernel@vger.kernel.org>);
        Fri, 14 Oct 2022 17:19:17 -0400
X-Greylist: delayed 518 seconds by postgrey-1.37 at lindbergh.monkeyblade.net; Fri, 14 Oct 2022 14:19:10 PDT
Received: from mail.stoffel.org (mail.stoffel.org [172.104.24.175])
        by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 647D71D29AC;
        Fri, 14 Oct 2022 14:19:09 -0700 (PDT)
Received: from quad.stoffel.org (068-116-170-226.res.spectrum.com [68.116.170.226])
        (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
         key-exchange X25519 server-signature RSA-PSS (2048 bits))
        (No client certificate requested)
        by mail.stoffel.org (Postfix) with ESMTPSA id 1A24F1E668;
        Fri, 14 Oct 2022 17:10:31 -0400 (EDT)
Received: by quad.stoffel.org (Postfix, from userid 1000)
        id B8C27A7F96; Fri, 14 Oct 2022 17:10:30 -0400 (EDT)
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
Message-ID: <25417.53318.731340.683540@quad.stoffel.home>
Date:   Fri, 14 Oct 2022 17:10:30 -0400
From:   "John Stoffel" <john@stoffel.org>
To:     Jonathan Derrick <jonathan.derrick@linux.dev>
Cc:     Song Liu <song@kernel.org>, <linux-raid@vger.kernel.org>,
        <linux-kernel@vger.kernel.org>, jonathan.derrick@solidigm.com,
        jonathanx.sk.derrick@intel.com,
        Mariusz Tkaczyk <mariusz.tkaczyk@linux.intel.com>
Subject: Re: [PATCH v2 0/3] Bitmap percentage flushing
In-Reply-To: <20221013224151.300-1-jonathan.derrick@linux.dev>
References: <20221013224151.300-1-jonathan.derrick@linux.dev>
X-Mailer: VM 8.2.0b under 27.1 (x86_64-pc-linux-gnu)
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

>>>>> "Jonathan" == Jonathan Derrick <jonathan.derrick@linux.dev> writes:

> This introduces a percentage-flushing mechanism that works in-tandem to the
> mdadm delay timer. The percentage argument is based on the number of chunks
> dirty (rather than percentage), due to large drives requiring smaller and
> smaller percentages (eg, 32TB drives-> 1% is 320GB).

I've been reading and re-reading this and I still don't understand
what you're saying here.  You say you're adding a percentage based
mechanism, but then you say it's based on chunk counts, not
percentages.  I think you need to clean this up and re-word it.

Maybe you're trying to say that you only take a percentage of the
available write bandwidth per second or something like that? 


> This set hopes to provide a way to make the bitmap flushing more consistent. It
> was observed that a synchronous, random write qd1 workload, could make bitmap
> writes easily become almost half of the I/O. And in similar workloads with
> different timing, it was several minutes between bitmap updates. This is too
> inconsistent to be reliable.

> This first and second patches adds the flush_threshold parameter. The default
> value of 0 defines the default behavior: unplugging immediately just as before.
> With a flush-threshold value of 1, it becomes more consistent and paranoid,
> flushing on nearly every I/O, leading to a 40% or greater situation. From

What situation?  Please be more clear here.  

> there, the flush_threshold can be defined higher for those situations where
> power loss is rare and full resync can be tolerated.

> The third patch converts the daemon worker to an actual timer. This makes it
> more consistent and removes some ugly code.

> Jonathan Derrick (3):
>   md/bitmap: Add chunk-threshold unplugging
>   md/bitmap: Add sysfs interface for flush threshold
>   md/bitmap: Convert daemon_work to proper timer

>  Documentation/admin-guide/md.rst |  5 ++
>  drivers/md/md-bitmap.c           | 98 +++++++++++++++++++++++++-------
>  drivers/md/md-bitmap.h           |  4 +-
>  drivers/md/md.c                  |  9 ++-
>  drivers/md/md.h                  |  2 +
>  5 files changed, 93 insertions(+), 25 deletions(-)

> -- 
> 2.31.1


From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <linux-kernel-owner@kernel.org>
X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on
	aws-us-west-2-korg-lkml-1.web.codeaurora.org
Received: from vger.kernel.org (vger.kernel.org [23.128.96.18])
	by smtp.lore.kernel.org (Postfix) with ESMTP id C665AC4332F
	for <linux-kernel@archiver.kernel.org>; Sat, 15 Oct 2022 22:27:26 +0000 (UTC)
Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand
        id S229583AbiJOW1T (ORCPT <rfc822;linux-kernel@archiver.kernel.org>);
        Sat, 15 Oct 2022 18:27:19 -0400
Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35982 "EHLO
        lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org
        with ESMTP id S229510AbiJOW1R (ORCPT
        <rfc822;linux-kernel@vger.kernel.org>);
        Sat, 15 Oct 2022 18:27:17 -0400
Received: from out0.migadu.com (out0.migadu.com [IPv6:2001:41d0:2:267::])
        by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C1C494003D;
        Sat, 15 Oct 2022 15:27:15 -0700 (PDT)
Message-ID: <5b3c2ccb-4e77-5707-d41a-5a91bd677c8f@linux.dev>
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1;
        t=1665872833;
        h=from:from:reply-to:subject:subject:date:date:message-id:message-id:
         to:to:cc:cc:mime-version:mime-version:content-type:content-type:
         content-transfer-encoding:content-transfer-encoding:
         in-reply-to:in-reply-to:references:references;
        bh=jyH/LzULF59OH5GGYkSCeFkBRluBAuYmG36QDeuLDGA=;
        b=ItVPqsA8nPWYRcgRztZnKGeigQevIb9NNMeUq1fNX+x4UrWR9/a/r4xf9hM1/M/UN/A0FE
        J0jjez2AWgJo3JWhZKhTKzlqPa+y42b15ojkvYOEoa+ulIkQTOuTTRZJIw0bPYje9gkI6f
        tYhoEvKW6MGKkq4vtGWr/i4EaGVLM5Y=
Date:   Sat, 15 Oct 2022 16:27:12 -0600
MIME-Version: 1.0
Subject: Re: [PATCH v2 0/3] Bitmap percentage flushing
Content-Language: en-US
To:     John Stoffel <john@stoffel.org>
Cc:     Song Liu <song@kernel.org>, linux-raid@vger.kernel.org,
        linux-kernel@vger.kernel.org, jonathan.derrick@solidigm.com,
        jonathanx.sk.derrick@intel.com,
        Mariusz Tkaczyk <mariusz.tkaczyk@linux.intel.com>
References: <20221013224151.300-1-jonathan.derrick@linux.dev>
 <25417.53318.731340.683540@quad.stoffel.home>
X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers.
From:   Jonathan Derrick <jonathan.derrick@linux.dev>
In-Reply-To: <25417.53318.731340.683540@quad.stoffel.home>
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit
X-Migadu-Flow: FLOW_OUT
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org



On 10/14/2022 3:10 PM, John Stoffel wrote:
>>>>>> "Jonathan" == Jonathan Derrick <jonathan.derrick@linux.dev> writes:
> 
>> This introduces a percentage-flushing mechanism that works in-tandem to the
>> mdadm delay timer. The percentage argument is based on the number of chunks
>> dirty (rather than percentage), due to large drives requiring smaller and
>> smaller percentages (eg, 32TB drives-> 1% is 320GB).
> 
> I've been reading and re-reading this and I still don't understand
> what you're saying here.  You say you're adding a percentage based
> mechanism, but then you say it's based on chunk counts, not
> percentages.  I think you need to clean this up and re-word it.> 
> Maybe you're trying to say that you only take a percentage of the
> available write bandwidth per second or something like that? 
I'll adjust it to chunk-count-based in the cover letter and make sure it
specifies bandwidth. I figured the chunk-count-based was a good way to
cover the desired percentage-based feature [1]. 

[1] https://elixir.bootlin.com/linux/latest/source/drivers/md/md-bitmap.c#L16


> 
> 
>> This set hopes to provide a way to make the bitmap flushing more consistent. It
>> was observed that a synchronous, random write qd1 workload, could make bitmap
>> writes easily become almost half of the I/O. And in similar workloads with
>> different timing, it was several minutes between bitmap updates. This is too
>> inconsistent to be reliable.
> 
>> This first and second patches adds the flush_threshold parameter. The default
>> value of 0 defines the default behavior: unplugging immediately just as before.
>> With a flush-threshold value of 1, it becomes more consistent and paranoid,
>> flushing on nearly every I/O, leading to a 40% or greater situation. From
> 
> What situation?  Please be more clear here.  

40% or more of given workload I/Os being bitmap flushes.
Will be more clear in v3

> 
>> there, the flush_threshold can be defined higher for those situations where
>> power loss is rare and full resync can be tolerated.
> 
>> The third patch converts the daemon worker to an actual timer. This makes it
>> more consistent and removes some ugly code.
> 
>> Jonathan Derrick (3):
>>   md/bitmap: Add chunk-threshold unplugging
>>   md/bitmap: Add sysfs interface for flush threshold
>>   md/bitmap: Convert daemon_work to proper timer
> 
>>  Documentation/admin-guide/md.rst |  5 ++
>>  drivers/md/md-bitmap.c           | 98 +++++++++++++++++++++++++-------
>>  drivers/md/md-bitmap.h           |  4 +-
>>  drivers/md/md.c                  |  9 ++-
>>  drivers/md/md.h                  |  2 +
>>  5 files changed, 93 insertions(+), 25 deletions(-)
> 
>> -- 
>> 2.31.1
> 

