[PATCH 0/2] mm, iomap: account dirty at page granularity for per-block dirty folios

[PATCH 0/2] mm, iomap: account dirty at page granularity for per-block dirty folios

[Salvatore Dipietro]

Since iomap gained per-block dirty tracking, a large folio can be
partially dirty: struct iomap_folio_state records exactly which
blocks are dirty, and writeback only writes those ranges.  The mm
dirty accounting, however, still charges the whole folio even if a
single block is dirty (up to 512x over-accounting for a 2 MB folio
over 4 KB blocks).

balance_dirty_pages() then throttles writers against a number that
is unrelated to how much data actually needs writing back.  On
workloads that repeatedly dirty small ranges of large folios --
e.g. PostgreSQL, which writes mostly 8/16 KB pages -- dirty_thresh
is crossed and writers are throttled long before the real dirty data
justifies it.  On the benchmark below, run on v7.3-rc7, the
unpatched kernel charges ~159 TB of dirtied pages while writing
~6 TB to the array, a factor of 27; with this series the dirtied
pages charged match the data actually written.

This series makes iomap charge and discharge the mm dirty counters
per dirty page for folios that carry per-block dirty state, so a
partially dirty large folio is accounted for the dirty part only:

  - Patch 1 adds the mm infrastructure: an AS_DIRTY_PER_BLOCK
    mapping flag, an @accounted argument to __folio_mark_dirty()
    and __filemap_dirty_folio() to skip the generic per-folio
    charge, and an address_space_operations::accounted_dirty_pages()
    callback used by the discharge and migration paths instead of
    folio_nr_pages().  No filesystem opts in, so it is a no-op on
    its own.

  - Patch 2 makes iomap charge per dirty page from the per-block
    dirty bitmap and wires iomap_accounted_dirty_pages() into every
    iomap-based filesystem.

This was benchmarked with PostgreSQL using the pgbench simple-update
workload on v7.3-rc7.  Average throughput rises 46.8% (139.8k ->
205.2k tps) and run-to-run variability drops from a 23.60% to a
3.98% coefficient of variation: the worst iteration improves from
97.9k to 191.5k tps, while the best gains 8%, 199.4k -> 215.9k.

Ten iterations per kernel, same database and workload (tps):

  Kernel                     Avg     Min     Max      CV   iters   iters
                                                             1-5    6-10
  ______________________  ______  ______  ______  ______  ______  ______
  v7.3-rc7                139.8k   97.9k  199.4k  23.60%  164.4k  115.2k
  v7.3-rc7 + this series  205.2k  191.5k  215.9k   3.98%  200.1k  210.2k

On unpatched v7.3-rc7 the worst iteration lands ~51% below the best,
and throughput decays as the run progresses: iterations 6-10 average
30% below iterations 1-5.  With this series the decay is gone -- the
second half is similar to the first -- and the coefficient of
variation stays at 3.98%.

The over-accounting has existed since commit 4ce02c679722 ("iomap: Add
per-block dirty state tracking to improve performance"), merged in
v6.6, where the ifs bitmap started recording partial dirty state while
the mm counters kept charging whole folios.

The throughput collapse bisects to the v6.17 readahead series, first
bad commit c4602f9fa77f ("mm/readahead: store folio order in struct
file_ra_state"), which lets readahead stay at the largest folio order
once it has ramped up, so a single dirty block charges far more than
is written.  Patch 2 therefore carries:

  Fixes: 4ce02c679722 ("iomap: Add per-block dirty state tracking to improve performance")
  Cc: stable@vger.kernel.org # v6.17+

Test environment:

  Hardware:  2x AWS EC2 m8g.24xlarge
             (SUT with 12x 1TB io2, 32000 IOPS each, RAID0, XFS)
  OS:        AL2023
  Kernels:   v7.3-rc7, v7.3-rc7 + this series
  Database:  PostgreSQL 17 (with huge_pages=on)
  Workload:  pgbench simple-update
             1024 clients, 96 threads, 600s duration
             scale factor 8470, fillfactor=90, prepared protocol
  Runs:      10 iterations per configuration

How to replicate results:

This script starts two AWS EC2 m8g.24xlarge instances (SUT + driver),
then installs, loads and benchmarks PostgreSQL using the
repro-collection repository.  To run it:

  # Reproducer code (dirty-folio-regression branch):
  git clone -b dirty-folio-regression \
    https://github.com/aws/repro-collection.git ~/repro-collection

  # Concatenate patches 1 and 2 of this series, in order, into one file:
  cat 0001-*.patch 0002-*.patch > dirty-per-block.patch

  # Setup and start PostgreSQL benchmark:
  KERNEL_REPO="https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git" \
  KERNEL_BRANCH="v7.3-rc7" \
  ITERS=10 \
  PLACEMENT_GROUP="pg-test" \
  KEY="$HOME/.ssh/id_rsa" \
  KEYPAIR="<aws keypair>" \
  ./repros/repro-postgresql-dirty-folios/folio-reproduce.sh <benchmark name> \
  --patch="<path to dirty-per-block.patch>"

Salvatore Dipietro (2):
  mm: add infrastructure for per-dirty-page accounting
  mm, iomap: account dirty at page granularity for per-block dirty
    folios

 fs/buffer.c             |   4 +-
 fs/exfat/inode.c        |   1 +
 fs/fuse/file.c          |   1 +
 fs/gfs2/aops.c          |   1 +
 fs/iomap/buffered-io.c  | 167 ++++++++++++++++++++++++++++++++++++++--
 fs/ntfs/aops.c          |   2 +
 fs/ntfs3/inode.c        |   2 +
 fs/xfs/xfs_aops.c       |   1 +
 fs/zonefs/file.c        |   1 +
 include/linux/fs.h      |   7 ++
 include/linux/iomap.h   |   1 +
 include/linux/pagemap.h |  20 ++++-
 mm/internal.h           |   3 +
 mm/migrate.c            |  17 +++-
 mm/page-writeback.c     |  91 +++++++++++++++++-----
 15 files changed, 286 insertions(+), 33 deletions(-)


base-commit: 24a19b953e2a3d0ea5a8c4951a0ab8f32720c907
-- 
2.50.1




AMAZON DEVELOPMENT CENTER ITALY SRL, viale Monte Grappa 3/5, 20124 Milano, Italia, Registro delle Imprese di Milano Monza Brianza Lodi REA n. 2504859, Capitale Sociale: 10.000 EUR i.v., Cod. Fisc. e P.IVA 10100050961, Societa con Socio Unico

[PATCH 1/2] mm: add infrastructure for per-dirty-page accounting

[Salvatore Dipietro]

Add the mm-side machinery that lets a filesystem charge the dirty
counters (NR_FILE_DIRTY, NR_ZONE_WRITE_PENDING, WB_RECLAIMABLE) per
dirty page instead of per whole folio, without yet enabling it for
any filesystem.

Add an AS_DIRTY_PER_BLOCK mapping flag with
mapping_set_dirty_per_block()/mapping_dirty_per_block() helpers, an
accounted_dirty_pages() address_space_operations callback, and an
@accounted argument to __folio_mark_dirty() (with filemap_dirty_folio()
split into __filemap_dirty_folio()) so the generic per-folio charge can
be suppressed for folios a filesystem has already accounted.

folio_accounted_dirty_pages() dispatches through the new a_ops callback,
falling back to folio_nr_pages() when no filesystem opts in, and
folio_account_cleaned() and folio migration use it in place of
folio_nr_pages().  folio_clear_dirty_for_io() leaves the charge of
per-block folios to the filesystem.  No filesystem sets the flag or
registers the callback yet, so this commit is a no-op on its own.

This is a prerequisite for the following iomap patch, which fixes the
over-accounting; both are needed for stable.

Cc: stable@vger.kernel.org # v6.17+
Signed-off-by: Salvatore Dipietro <dipiets@amazon.it>
Reviewed-by: Hazem Mohamed Abuelfotoh <abuehaze@amazon.com>
---
 fs/buffer.c             |  4 +-
 include/linux/fs.h      |  7 ++++
 include/linux/pagemap.h | 20 ++++++++-
 mm/internal.h           |  3 ++
 mm/migrate.c            | 17 ++++++--
 mm/page-writeback.c     | 91 ++++++++++++++++++++++++++++++++---------
 6 files changed, 115 insertions(+), 27 deletions(-)

diff --git a/fs/buffer.c b/fs/buffer.c
index ed966fa73b1b..7ed5250dda9d 100644
--- a/fs/buffer.c
+++ b/fs/buffer.c
@@ -693,7 +693,7 @@ bool block_dirty_folio(struct address_space *mapping, struct folio *folio)
 	spin_unlock(&mapping->i_private_lock);
 
 	if (newly_dirty)
-		__folio_mark_dirty(folio, mapping, 1);
+		__folio_mark_dirty(folio, mapping, 1, false);
 
 	if (newly_dirty)
 		__mark_inode_dirty(mapping->host, I_DIRTY_PAGES);
@@ -1011,7 +1011,7 @@ void mark_buffer_dirty(struct buffer_head *bh)
 		if (!folio_test_set_dirty(folio)) {
 			mapping = folio->mapping;
 			if (mapping)
-				__folio_mark_dirty(folio, mapping, 0);
+				__folio_mark_dirty(folio, mapping, 0, false);
 		}
 		if (mapping)
 			__mark_inode_dirty(mapping->host, I_DIRTY_PAGES);
diff --git a/include/linux/fs.h b/include/linux/fs.h
index deb411e86661..5abd2af50b59 100644
--- a/include/linux/fs.h
+++ b/include/linux/fs.h
@@ -407,6 +407,13 @@ struct address_space_operations {
 	/* Mark a folio dirty.  Return true if this dirtied it */
 	bool (*dirty_folio)(struct address_space *, struct folio *);
 
+	/*
+	 * Pages of the folio currently charged to the dirty counters, for
+	 * filesystems that account dirty state at sub-folio (per-block)
+	 * granularity.
+	 */
+	long (*accounted_dirty_pages)(struct folio *folio);
+
 	void (*readahead)(struct readahead_control *);
 
 	int (*write_begin)(const struct kiocb *, struct address_space *mapping,
diff --git a/include/linux/pagemap.h b/include/linux/pagemap.h
index 0adfa6605653..5bbcd9da1a6b 100644
--- a/include/linux/pagemap.h
+++ b/include/linux/pagemap.h
@@ -210,6 +210,7 @@ enum mapping_flags {
 	AS_WRITEBACK_MAY_DEADLOCK_ON_RECLAIM = 9,
 	AS_KERNEL_FILE = 10,	/* mapping for a fake kernel file that shouldn't
 				   account usage to user cgroups */
+	AS_DIRTY_PER_BLOCK = 11, /* fs accounts dirty pages, not whole folios */
 	/* Bits 16-25 are used for FOLIO_ORDER */
 	AS_FOLIO_ORDER_BITS = 5,
 	AS_FOLIO_ORDER_MIN = 16,
@@ -304,6 +305,20 @@ static inline void mapping_clear_release_always(struct address_space *mapping)
 	clear_bit(AS_RELEASE_ALWAYS, &mapping->flags);
 }
 
+/*
+ * Set by filesystems that charge the mm dirty counters per dirty page rather
+ * than per folio, so the generic paths must not also account whole folios.
+ */
+static inline void mapping_set_dirty_per_block(struct address_space *mapping)
+{
+	set_bit(AS_DIRTY_PER_BLOCK, &mapping->flags);
+}
+
+static inline bool mapping_dirty_per_block(const struct address_space *mapping)
+{
+	return test_bit(AS_DIRTY_PER_BLOCK, &mapping->flags);
+}
+
 static inline bool mapping_stable_writes(const struct address_space *mapping)
 {
 	return test_bit(AS_STABLE_WRITES, &mapping->flags);
@@ -1311,7 +1326,10 @@ void folio_end_writeback(struct folio *folio);
 void folio_end_writeback_no_dropbehind(struct folio *folio);
 void folio_end_dropbehind(struct folio *folio);
 void folio_wait_stable(struct folio *folio);
-void __folio_mark_dirty(struct folio *folio, struct address_space *, int warn);
+void __folio_mark_dirty(struct folio *folio, struct address_space *mapping,
+		int warn, bool accounted);
+bool __filemap_dirty_folio(struct address_space *mapping, struct folio *folio,
+		bool accounted);
 void folio_account_cleaned(struct folio *folio, struct bdi_writeback *wb);
 void __folio_cancel_dirty(struct folio *folio);
 static inline void folio_cancel_dirty(struct folio *folio)
diff --git a/mm/internal.h b/mm/internal.h
index 38b1165212c9..4647aca5fe8d 100644
--- a/mm/internal.h
+++ b/mm/internal.h
@@ -40,6 +40,9 @@ void *workingset_eviction(struct folio *folio,
 void workingset_refault(struct folio *folio, void *shadow);
 void workingset_activation(struct folio *folio);
 
+/* mm/page-writeback.c */
+long folio_accounted_dirty_pages(struct folio *folio);
+
 /* mm/folio.c */
 void folio_add_lru_vma(struct folio *folio, struct vm_area_struct *vma);
 
diff --git a/mm/migrate.c b/mm/migrate.c
index 15b45832bcfa..0e7b7834a9fd 100644
--- a/mm/migrate.c
+++ b/mm/migrate.c
@@ -588,6 +588,7 @@ static int __folio_migrate_mapping(struct address_space *mapping,
 	struct zone *oldzone, *newzone;
 	int dirty;
 	long nr = folio_nr_pages(folio);
+	long nr_dirty = nr;
 
 	if (!mapping) {
 		/* Take off deferred split queue while frozen and memcg set */
@@ -614,6 +615,14 @@ static int __folio_migrate_mapping(struct address_space *mapping,
 	oldzone = folio_zone(folio);
 	newzone = folio_zone(newfolio);
 
+	/*
+	 * Only what was charged moves between the zones below, which may be less
+	 * than the whole folio.  Sample it outside the i_pages lock, as it walks
+	 * the per-block state, and while the folio is still dirty.
+	 */
+	if (folio_test_dirty(folio))
+		nr_dirty = folio_accounted_dirty_pages(folio);
+
 	if (folio_test_swapcache(folio))
 		ci = swap_cluster_get_and_lock_irq(folio);
 	else
@@ -708,10 +717,10 @@ static int __folio_migrate_mapping(struct address_space *mapping,
 		}
 #endif
 		if (dirty && mapping_can_writeback(mapping)) {
-			mod_lruvec_state(old_lruvec, NR_FILE_DIRTY, -nr);
-			__mod_zone_page_state(oldzone, NR_ZONE_WRITE_PENDING, -nr);
-			mod_lruvec_state(new_lruvec, NR_FILE_DIRTY, nr);
-			__mod_zone_page_state(newzone, NR_ZONE_WRITE_PENDING, nr);
+			mod_lruvec_state(old_lruvec, NR_FILE_DIRTY, -nr_dirty);
+			__mod_zone_page_state(oldzone, NR_ZONE_WRITE_PENDING, -nr_dirty);
+			mod_lruvec_state(new_lruvec, NR_FILE_DIRTY, nr_dirty);
+			__mod_zone_page_state(newzone, NR_ZONE_WRITE_PENDING, nr_dirty);
 		}
 		rcu_read_unlock();
 	}
diff --git a/mm/page-writeback.c b/mm/page-writeback.c
index eeab25d6ce36..6b4a8b6ad34c 100644
--- a/mm/page-writeback.c
+++ b/mm/page-writeback.c
@@ -2601,7 +2601,7 @@ EXPORT_SYMBOL(noop_dirty_folio);
  * NOTE: This relies on being atomic wrt interrupts.
  */
 static void folio_account_dirtied(struct folio *folio,
-		struct address_space *mapping)
+		struct address_space *mapping, bool accounted)
 {
 	struct inode *inode = mapping->host;
 
@@ -2614,14 +2614,21 @@ static void folio_account_dirtied(struct folio *folio,
 		inode_attach_wb(inode, folio);
 		wb = inode_to_wb(inode);
 
-		lruvec_stat_mod_folio(folio, NR_FILE_DIRTY, nr);
-		if (folio_test_dropbehind(folio))
-			wb_stat_mod(wb, WB_DONTCACHE_DIRTY, nr);
-		__zone_stat_mod_folio(folio, NR_ZONE_WRITE_PENDING, nr);
-		__node_stat_mod_folio(folio, NR_DIRTIED, nr);
-		wb_stat_mod(wb, WB_RECLAIMABLE, nr);
-		wb_stat_mod(wb, WB_DIRTIED, nr);
-		task_io_account_write(nr * PAGE_SIZE);
+		/*
+		 * @accounted means the filesystem charged these per dirty page
+		 * already; the rate limiters below stay per-folio.  Keep in sync
+		 * with folio_account_cleaned() and folio_clear_dirty_for_io().
+		 */
+		if (!accounted) {
+			lruvec_stat_mod_folio(folio, NR_FILE_DIRTY, nr);
+			if (folio_test_dropbehind(folio))
+				wb_stat_mod(wb, WB_DONTCACHE_DIRTY, nr);
+			__zone_stat_mod_folio(folio, NR_ZONE_WRITE_PENDING, nr);
+			wb_stat_mod(wb, WB_RECLAIMABLE, nr);
+			__node_stat_mod_folio(folio, NR_DIRTIED, nr);
+			wb_stat_mod(wb, WB_DIRTIED, nr);
+			task_io_account_write(nr * PAGE_SIZE);
+		}
 		current->nr_dirtied += nr;
 		__this_cpu_add(bdp_ratelimits, nr);
 
@@ -2629,13 +2636,41 @@ static void folio_account_dirtied(struct folio *folio,
 	}
 }
 
+/**
+ * folio_accounted_dirty_pages - pages of @folio charged to the dirty counters
+ * @folio: folio to query
+ *
+ * Use this rather than folio_nr_pages() when moving or discharging a folio's dirty
+ * accounting, as a folio is not necessarily charged for all of its pages.  Walks
+ * per-block state that truncation frees, so the caller must hold the folio lock.
+ *
+ * Return: the whole folio for the usual per-folio accounting, otherwise only the
+ * pages holding a dirty block, and zero once those have been discharged.
+ */
+long folio_accounted_dirty_pages(struct folio *folio)
+{
+	struct address_space *mapping = folio->mapping;
+
+	if (mapping && mapping_dirty_per_block(mapping) && folio->private &&
+	    mapping->a_ops->accounted_dirty_pages)
+		return mapping->a_ops->accounted_dirty_pages(folio);
+	return folio_nr_pages(folio);
+}
+
 /*
  * Helper function for deaccounting dirty page without writeback.
  *
  */
 void folio_account_cleaned(struct folio *folio, struct bdi_writeback *wb)
 {
-	long nr = folio_nr_pages(folio);
+	/*
+	 * Discharge exactly what was charged; zero if the fs already did it.
+	 * Counters below mirror folio_account_dirtied().
+	 */
+	long nr = folio_accounted_dirty_pages(folio);
+
+	if (!nr)
+		return;
 
 	lruvec_stat_mod_folio(folio, NR_FILE_DIRTY, -nr);
 	if (folio_test_dropbehind(folio))
@@ -2661,7 +2696,7 @@ void folio_account_cleaned(struct folio *folio, struct bdi_writeback *wb)
  * try_to_free_buffers() to fail.
  */
 void __folio_mark_dirty(struct folio *folio, struct address_space *mapping,
-			     int warn)
+			     int warn, bool accounted)
 {
 	unsigned long flags;
 
@@ -2674,7 +2709,7 @@ void __folio_mark_dirty(struct folio *folio, struct address_space *mapping,
 	xa_lock_irqsave(&mapping->i_pages, flags);
 	if (folio->mapping) {	/* Race with truncate? */
 		WARN_ON_ONCE(warn && !folio_test_uptodate(folio));
-		folio_account_dirtied(folio, mapping);
+		folio_account_dirtied(folio, mapping, accounted);
 		__xa_set_mark(&mapping->i_pages, folio->index,
 			      PAGECACHE_TAG_DIRTY);
 	}
@@ -2700,12 +2735,13 @@ void __folio_mark_dirty(struct folio *folio, struct address_space *mapping,
  * simply hold the folio lock, but e.g. zap_pte_range() calls with the
  * folio mapped and the pte lock held, which also locks out truncation.
  */
-bool filemap_dirty_folio(struct address_space *mapping, struct folio *folio)
+bool __filemap_dirty_folio(struct address_space *mapping, struct folio *folio,
+		bool accounted)
 {
 	if (folio_test_set_dirty(folio))
 		return false;
 
-	__folio_mark_dirty(folio, mapping, !folio_test_private(folio));
+	__folio_mark_dirty(folio, mapping, !folio_test_private(folio), accounted);
 
 	if (mapping->host) {
 		/* !PageAnon && !swapper_space */
@@ -2713,6 +2749,12 @@ bool filemap_dirty_folio(struct address_space *mapping, struct folio *folio)
 	}
 	return true;
 }
+EXPORT_SYMBOL_GPL(__filemap_dirty_folio);
+
+bool filemap_dirty_folio(struct address_space *mapping, struct folio *folio)
+{
+	return __filemap_dirty_folio(mapping, folio, false);
+}
 EXPORT_SYMBOL(filemap_dirty_folio);
 
 /**
@@ -2912,12 +2954,21 @@ bool folio_clear_dirty_for_io(struct folio *folio)
 		 */
 		wb = unlocked_inode_to_wb_begin(inode, &cookie);
 		if (folio_test_clear_dirty(folio)) {
-			long nr = folio_nr_pages(folio);
-			lruvec_stat_mod_folio(folio, NR_FILE_DIRTY, -nr);
-			if (folio_test_dropbehind(folio))
-				wb_stat_mod(wb, WB_DONTCACHE_DIRTY, -nr);
-			zone_stat_mod_folio(folio, NR_ZONE_WRITE_PENDING, -nr);
-			wb_stat_mod(wb, WB_RECLAIMABLE, -nr);
+			/*
+			 * Per-block folios keep their dirty bits, and the charge,
+			 * until the fs writes them back.  Folios of such a mapping
+			 * that never got per-block state were charged whole, so
+			 * they discharge here as usual.
+			 */
+			if (!mapping_dirty_per_block(mapping) || !folio->private) {
+				long nr = folio_nr_pages(folio);
+
+				lruvec_stat_mod_folio(folio, NR_FILE_DIRTY, -nr);
+				if (folio_test_dropbehind(folio))
+					wb_stat_mod(wb, WB_DONTCACHE_DIRTY, -nr);
+				zone_stat_mod_folio(folio, NR_ZONE_WRITE_PENDING, -nr);
+				wb_stat_mod(wb, WB_RECLAIMABLE, -nr);
+			}
 			ret = true;
 		}
 		unlocked_inode_to_wb_end(inode, &cookie);
-- 
2.50.1




AMAZON DEVELOPMENT CENTER ITALY SRL, viale Monte Grappa 3/5, 20124 Milano, Italia, Registro delle Imprese di Milano Monza Brianza Lodi REA n. 2504859, Capitale Sociale: 10.000 EUR i.v., Cod. Fisc. e P.IVA 10100050961, Societa con Socio Unico

[PATCH 2/2] mm, iomap: account dirty at page granularity for per-block dirty folios

[Salvatore Dipietro]

The mm dirty counters (NR_FILE_DIRTY, NR_ZONE_WRITE_PENDING,
WB_RECLAIMABLE) are charged per folio, which charges up to 512
pages for a single dirty 4 kB block of a 2 MB folio.  iomap knows
exactly which blocks are dirty and writeback only writes those
ranges, so balance_dirty_pages() ends up throttling writers
against a dirty count far larger than the data that actually
needs writeback.

Charge the counters per dirty page for folios carrying per-block
dirty state, using the infrastructure from the previous patch:
struct iomap_folio_state counts the pages holding a dirty block, and
each dirty bitmap change charges or discharges the difference.  The
charge survives folio_clear_dirty_for_io() and is dropped when
writeback or invalidation clears the blocks.  The monotonic
NR_DIRTIED / WB_DIRTIED / task_io totals are charged only for newly
dirtied data, and the dirty rate limiters stay per-folio.

Every iomap-based filesystem sets .accounted_dirty_pages alongside
.dirty_folio = iomap_dirty_folio.

On pgbench simple-update (1024 clients, XFS, 96-vCPU arm64,
10 iterations) on v7.3-rc7, the unpatched kernel charges 159 TB
of dirtied pages while writing 6 TB to the array, a factor of 27;
with this patch the dirtied pages charged match the data written.
Unpatched, nr_dirty is above dirty_threshold in ~17% of 5s samples
and its peak exceeds it in 9 of 10 iterations; patched, it never
reaches the threshold.

On v7.3-rc7, average throughput rises 46.8% (139.8k -> 205.2k tps)
and run-to-run variability drops from 23.60% to 3.98% coefficient
of variation.  The best iteration improves by 8% (199.4k -> 215.9k
tps); most of the gain comes from the worst iteration, up from 97.9k
to 191.5k tps, i.e. from throttling episodes that no longer happen.

Fixes: 4ce02c679722 ("iomap: Add per-block dirty state tracking to improve performance")
Cc: stable@vger.kernel.org # v6.17+
Signed-off-by: Salvatore Dipietro <dipiets@amazon.it>
Reviewed-by: Hazem Mohamed Abuelfotoh <abuehaze@amazon.com>
---
 fs/exfat/inode.c       |   1 +
 fs/fuse/file.c         |   1 +
 fs/gfs2/aops.c         |   1 +
 fs/iomap/buffered-io.c | 167 +++++++++++++++++++++++++++++++++++++++--
 fs/ntfs/aops.c         |   2 +
 fs/ntfs3/inode.c       |   2 +
 fs/xfs/xfs_aops.c      |   1 +
 fs/zonefs/file.c       |   1 +
 include/linux/iomap.h  |   1 +
 9 files changed, 171 insertions(+), 6 deletions(-)

diff --git a/fs/exfat/inode.c b/fs/exfat/inode.c
index ccd13630187e..48704fe5d59b 100644
--- a/fs/exfat/inode.c
+++ b/fs/exfat/inode.c
@@ -302,6 +302,7 @@ static const struct address_space_operations exfat_aops = {
 	.readahead		= exfat_readahead,
 	.writepages		= exfat_writepages,
 	.dirty_folio		= iomap_dirty_folio,
+	.accounted_dirty_pages	= iomap_accounted_dirty_pages,
 	.bmap			= exfat_aop_bmap,
 	.migrate_folio		= filemap_migrate_folio,
 	.is_partially_uptodate	= iomap_is_partially_uptodate,
diff --git a/fs/fuse/file.c b/fs/fuse/file.c
index 9a36d0329e22..eb7179f854ee 100644
--- a/fs/fuse/file.c
+++ b/fs/fuse/file.c
@@ -3101,6 +3101,7 @@ static const struct address_space_operations fuse_file_aops  = {
 	.writepages	= fuse_writepages,
 	.launder_folio	= fuse_launder_folio,
 	.dirty_folio	= iomap_dirty_folio,
+	.accounted_dirty_pages = iomap_accounted_dirty_pages,
 	.release_folio	= iomap_release_folio,
 	.invalidate_folio = iomap_invalidate_folio,
 	.is_partially_uptodate = iomap_is_partially_uptodate,
diff --git a/fs/gfs2/aops.c b/fs/gfs2/aops.c
index 0a7b8076af3a..ea77af00b111 100644
--- a/fs/gfs2/aops.c
+++ b/fs/gfs2/aops.c
@@ -713,6 +713,7 @@ static const struct address_space_operations gfs2_aops = {
 	.read_folio = gfs2_read_folio,
 	.readahead = gfs2_readahead,
 	.dirty_folio = iomap_dirty_folio,
+	.accounted_dirty_pages = iomap_accounted_dirty_pages,
 	.release_folio = iomap_release_folio,
 	.invalidate_folio = iomap_invalidate_folio,
 	.bmap = gfs2_bmap,
diff --git a/fs/iomap/buffered-io.c b/fs/iomap/buffered-io.c
index 0a5ebfda90f1..75671b29f7cc 100644
--- a/fs/iomap/buffered-io.c
+++ b/fs/iomap/buffered-io.c
@@ -5,6 +5,8 @@
  */
 #include <linux/iomap.h>
 #include <linux/buffer_head.h>
+#include <linux/backing-dev.h>
+#include <linux/task_io_accounting_ops.h>
 #include <linux/writeback.h>
 #include <linux/swap.h>
 #include <linux/migrate.h>
@@ -24,6 +26,13 @@ struct iomap_folio_state {
 	unsigned int		read_bytes_pending;
 	atomic_t		write_bytes_pending;
 
+	/*
+	 * Pages holding at least one dirty block, the unit the mm dirty counters
+	 * work in.  Maintained under @state_lock alongside the dirty bits below,
+	 * so it is what those counters were charged for.
+	 */
+	unsigned int		dirty_pages;
+
 	/*
 	 * Each block has two bits in this bitmap:
 	 * Bits [0..blocks_per_folio) has the uptodate status.
@@ -180,6 +189,89 @@ static unsigned iomap_find_dirty_range(struct folio *folio, u64 *range_start,
 	return range_end - *range_start;
 }
 
+/*
+ * Count pages in [first_blk, last_blk] holding at least one dirty block.  Callers
+ * take this before and after flipping the bitmap for an exact delta, so each page
+ * is scanned over its full block span to land in the same page in both counts.
+ * state_lock must be held.
+ */
+static unsigned int ifs_count_dirty_pages(struct folio *folio,
+		struct iomap_folio_state *ifs, unsigned int first_blk,
+		unsigned int last_blk)
+{
+	struct inode *inode = folio->mapping->host;
+	unsigned int blks_per_folio = i_blocks_per_folio(inode, folio);
+	unsigned int blkbits = inode->i_blkbits;
+	size_t fsize = folio_size(folio);
+	size_t start = (size_t)first_blk << blkbits;
+	size_t end = min(((size_t)last_blk << blkbits) + (1UL << blkbits) - 1,
+			 fsize - 1);
+	unsigned long first_pg = start >> PAGE_SHIFT;
+	unsigned long last_pg = end >> PAGE_SHIFT;
+	unsigned int dirty_pages = 0;
+	unsigned long pg;
+
+	for (pg = first_pg; pg <= last_pg; pg++) {
+		size_t pg_start = pg << PAGE_SHIFT;
+		size_t pg_end = min(pg_start + PAGE_SIZE - 1, fsize - 1);
+		unsigned int b_lo = pg_start >> blkbits;
+		unsigned int b_hi = pg_end >> blkbits;
+		unsigned int b;
+
+		for (b = b_lo; b <= b_hi; b++) {
+			if (test_bit(b + blks_per_folio, ifs->state)) {
+				dirty_pages++;
+				break;
+			}
+		}
+	}
+	return dirty_pages;
+}
+
+/*
+ * Mirror folio_account_dirtied() in page-sized increments; @nr_pages is positive
+ * to charge and negative to discharge.
+ *
+ * The NR_DIRTIED/WB_DIRTIED/task_io totals are monotonic, so only @new_dirty data
+ * may charge them; writeback re-marks an already charged folio.
+ *
+ * Only the folio lock is held, not i_pages, hence the IRQ-safe stat helpers.
+ */
+static void iomap_account_dirty_pages(struct folio *folio, long nr_pages,
+		bool new_dirty)
+{
+	struct address_space *mapping = folio->mapping;
+	struct inode *inode = mapping->host;
+	struct wb_lock_cookie cookie = {};
+	struct bdi_writeback *wb;
+	bool dirtied = nr_pages > 0 && new_dirty;
+
+	if (!nr_pages || !mapping_can_writeback(mapping))
+		return;
+
+	lruvec_stat_mod_folio(folio, NR_FILE_DIRTY, nr_pages);
+	zone_stat_mod_folio(folio, NR_ZONE_WRITE_PENDING, nr_pages);
+	if (dirtied)
+		node_stat_mod_folio(folio, NR_DIRTIED, nr_pages);
+
+	/* Pin the wb against a concurrent inode_do_switch_wbs(). */
+	inode_attach_wb(inode, folio);
+	wb = unlocked_inode_to_wb_begin(inode, &cookie);
+	wb_stat_mod(wb, WB_RECLAIMABLE, nr_pages);
+	if (folio_test_dropbehind(folio))
+		wb_stat_mod(wb, WB_DONTCACHE_DIRTY, nr_pages);
+	if (dirtied)
+		wb_stat_mod(wb, WB_DIRTIED, nr_pages);
+	unlocked_inode_to_wb_end(inode, &cookie);
+
+	if (dirtied) {
+		task_io_account_write(nr_pages * PAGE_SIZE);
+	} else if (nr_pages < 0 && !folio_test_writeback(folio)) {
+		/* truncated or invalidated before reaching disk, so refund */
+		task_io_account_cancelled_write((-nr_pages) * PAGE_SIZE);
+	}
+}
+
 /*
  * Clear the per-block dirty bits for the range [@off, @off + @len) within a
  * folio.  The range is rounded inwards so that only blocks fully covered by
@@ -195,15 +287,22 @@ static void ifs_clear_range_dirty(struct folio *folio,
 	unsigned int first_blk = round_up(off, i_blocksize(inode)) >>
 				 inode->i_blkbits;
 	unsigned int last_blk = (off + len) >> inode->i_blkbits;
+	unsigned int before, after;
 	unsigned long flags;
 
 	if (first_blk >= last_blk)
 		return;
 
 	spin_lock_irqsave(&ifs->state_lock, flags);
+	before = ifs_count_dirty_pages(folio, ifs, first_blk, last_blk - 1);
 	bitmap_clear(ifs->state, first_blk + blks_per_folio,
 		     last_blk - first_blk);
+	after = ifs_count_dirty_pages(folio, ifs, first_blk, last_blk - 1);
+	ifs->dirty_pages -= before - after;
 	spin_unlock_irqrestore(&ifs->state_lock, flags);
+
+	/* Pages that lost their last dirty block, so before >= after. */
+	iomap_account_dirty_pages(folio, -(long)(before - after), false);
 }
 
 static void iomap_clear_range_dirty(struct folio *folio, size_t off, size_t len)
@@ -222,11 +321,13 @@ static void iomap_clear_range_dirty(struct folio *folio, size_t off, size_t len)
  * data loss when partial blocks are written.
  */
 static void ifs_set_range_dirty(struct folio *folio,
-		struct iomap_folio_state *ifs, size_t off, size_t len)
+		struct iomap_folio_state *ifs, size_t off, size_t len,
+		bool new_dirty)
 {
 	struct inode *inode = folio->mapping->host;
 	unsigned int blks_per_folio = i_blocks_per_folio(inode, folio);
 	unsigned int first_blk, last_blk;
+	unsigned int before, after;
 	unsigned long flags;
 
 	if (!len)
@@ -235,9 +336,15 @@ static void ifs_set_range_dirty(struct folio *folio,
 	first_blk = off >> inode->i_blkbits;
 	last_blk = (off + len - 1) >> inode->i_blkbits;
 	spin_lock_irqsave(&ifs->state_lock, flags);
+	before = ifs_count_dirty_pages(folio, ifs, first_blk, last_blk);
 	bitmap_set(ifs->state, first_blk + blks_per_folio,
 		   last_blk - first_blk + 1);
+	after = ifs_count_dirty_pages(folio, ifs, first_blk, last_blk);
+	ifs->dirty_pages += after - before;
 	spin_unlock_irqrestore(&ifs->state_lock, flags);
+
+	/* Pages that gained their first dirty block, so after >= before. */
+	iomap_account_dirty_pages(folio, (long)(after - before), new_dirty);
 }
 
 static void iomap_set_range_dirty(struct folio *folio, size_t off, size_t len)
@@ -245,7 +352,20 @@ static void iomap_set_range_dirty(struct folio *folio, size_t off, size_t len)
 	struct iomap_folio_state *ifs = folio->private;
 
 	if (ifs)
-		ifs_set_range_dirty(folio, ifs, off, len);
+		ifs_set_range_dirty(folio, ifs, off, len, true);
+}
+
+/*
+ * Set the dirty bits for blocks that were already dirty, and already counted,
+ * before this folio had per-block state: the reclaimable counters still move,
+ * but the dirtied totals must not be charged twice.
+ */
+static void iomap_redirty_range(struct folio *folio, size_t off, size_t len)
+{
+	struct iomap_folio_state *ifs = folio->private;
+
+	if (ifs)
+		ifs_set_range_dirty(folio, ifs, off, len, false);
 }
 
 static struct iomap_folio_state *ifs_alloc(struct inode *inode,
@@ -276,9 +396,13 @@ static struct iomap_folio_state *ifs_alloc(struct inode *inode,
 	spin_lock_init(&ifs->state_lock);
 	if (folio_test_uptodate(folio))
 		bitmap_set(ifs->state, 0, nr_blocks);
-	if (folio_test_dirty(folio))
+	if (folio_test_dirty(folio)) {
 		bitmap_set(ifs->state, nr_blocks, nr_blocks);
+		ifs->dirty_pages = folio_nr_pages(folio);
+	}
 	folio_attach_private(folio, ifs);
+	/* The mm dirty counters are now charged per dirty page for this file. */
+	mapping_set_dirty_per_block(inode->i_mapping);
 
 	return ifs;
 }
@@ -839,6 +963,11 @@ void iomap_invalidate_folio(struct folio *folio, size_t offset, size_t len)
 	 */
 	if (offset == 0 && len == folio_size(folio)) {
 		WARN_ON_ONCE(folio_test_writeback(folio));
+		/*
+		 * Discharge while the bitmap is still attached; ifs_free() below
+		 * drops the state that records how many pages were charged.
+		 */
+		iomap_clear_range_dirty(folio, 0, folio_size(folio));
 		folio_cancel_dirty(folio);
 		ifs_free(folio);
 	} else {
@@ -854,10 +983,31 @@ bool iomap_dirty_folio(struct address_space *mapping, struct folio *folio)
 
 	ifs_alloc(inode, folio, 0);
 	iomap_set_range_dirty(folio, 0, len);
-	return filemap_dirty_folio(mapping, folio);
+	return __filemap_dirty_folio(mapping, folio, folio->private);
 }
 EXPORT_SYMBOL_GPL(iomap_dirty_folio);
 
+/**
+ * iomap_accounted_dirty_pages - pages of @folio charged to the dirty counters
+ * @folio: folio to query
+ *
+ * For callers that discharge what iomap charged.  The caller must hold the folio
+ * lock, and @folio must have per-block state attached.
+ *
+ * Return: pages of @folio holding at least one dirty block, zero once they have
+ * been cleared, so a later discharge does not double count.
+ */
+long iomap_accounted_dirty_pages(struct folio *folio)
+{
+	struct iomap_folio_state *ifs = folio->private;
+
+	if (WARN_ON_ONCE(!ifs))
+		return 0;
+
+	return ifs->dirty_pages;
+}
+EXPORT_SYMBOL_GPL(iomap_accounted_dirty_pages);
+
 static void
 iomap_write_failed(struct inode *inode, loff_t pos, unsigned len)
 {
@@ -1137,7 +1287,7 @@ static bool __iomap_write_end(struct inode *inode, loff_t pos, size_t len,
 		return false;
 	iomap_set_range_uptodate(folio, offset_in_folio(folio, pos), len);
 	iomap_set_range_dirty(folio, offset_in_folio(folio, pos), copied);
-	filemap_dirty_folio(inode->i_mapping, folio);
+	__filemap_dirty_folio(inode->i_mapping, folio, folio->private);
 	return true;
 }
 
@@ -2001,8 +2151,13 @@ int iomap_writeback_folio(struct iomap_writepage_ctx *wpc, struct folio *folio)
 
 	if (i_blocks_per_folio(inode, folio) > 1) {
 		if (!ifs) {
+			/*
+			 * Dirtied before it had per-block state, so charged
+			 * per-folio.  Re-mark the blocks for the walk below,
+			 * paired with the clear at the end of this function.
+			 */
 			ifs = ifs_alloc(inode, folio, 0);
-			iomap_set_range_dirty(folio, 0, end_pos - pos);
+			iomap_redirty_range(folio, 0, end_pos - pos);
 		}
 
 		iomap_writeback_init(inode, folio);
diff --git a/fs/ntfs/aops.c b/fs/ntfs/aops.c
index 517d9a1563e7..d26ddd5ae118 100644
--- a/fs/ntfs/aops.c
+++ b/fs/ntfs/aops.c
@@ -308,6 +308,7 @@ const struct address_space_operations ntfs_aops = {
 	.readahead		= ntfs_readahead,
 	.writepages		= ntfs_writepages,
 	.dirty_folio		= iomap_dirty_folio,
+	.accounted_dirty_pages	= iomap_accounted_dirty_pages,
 	.bmap			= ntfs_bmap,
 	.migrate_folio		= filemap_migrate_folio,
 	.is_partially_uptodate	= iomap_is_partially_uptodate,
@@ -322,6 +323,7 @@ const struct address_space_operations ntfs_mft_aops = {
 	.readahead		= ntfs_readahead,
 	.writepages		= ntfs_mft_writepages,
 	.dirty_folio		= iomap_dirty_folio,
+	.accounted_dirty_pages	= iomap_accounted_dirty_pages,
 	.bmap			= ntfs_bmap,
 	.migrate_folio		= filemap_migrate_folio,
 	.is_partially_uptodate	= iomap_is_partially_uptodate,
diff --git a/fs/ntfs3/inode.c b/fs/ntfs3/inode.c
index 4ac26c80bd34..509570091e48 100644
--- a/fs/ntfs3/inode.c
+++ b/fs/ntfs3/inode.c
@@ -2301,6 +2301,7 @@ const struct address_space_operations ntfs_aops = {
 	.writepages	= ntfs_writepages,
 	.bmap		= ntfs_bmap,
 	.dirty_folio	= iomap_dirty_folio,
+	.accounted_dirty_pages = iomap_accounted_dirty_pages,
 	.migrate_folio	= filemap_migrate_folio,
 	.release_folio	= iomap_release_folio,
 	.invalidate_folio = iomap_invalidate_folio,
@@ -2309,6 +2310,7 @@ const struct address_space_operations ntfs_aops = {
 const struct address_space_operations ntfs_aops_cmpr = {
 	.read_folio	= ntfs_read_folio,
 	.dirty_folio	= iomap_dirty_folio,
+	.accounted_dirty_pages = iomap_accounted_dirty_pages,
 	.release_folio	= iomap_release_folio,
 	.invalidate_folio = iomap_invalidate_folio,
 };
diff --git a/fs/xfs/xfs_aops.c b/fs/xfs/xfs_aops.c
index 8b6119776fb3..869248241f3b 100644
--- a/fs/xfs/xfs_aops.c
+++ b/fs/xfs/xfs_aops.c
@@ -682,6 +682,7 @@ const struct address_space_operations xfs_address_space_operations = {
 	.readahead		= xfs_vm_readahead,
 	.writepages		= xfs_vm_writepages,
 	.dirty_folio		= iomap_dirty_folio,
+	.accounted_dirty_pages	= iomap_accounted_dirty_pages,
 	.release_folio		= iomap_release_folio,
 	.invalidate_folio	= iomap_invalidate_folio,
 	.bmap			= xfs_vm_bmap,
diff --git a/fs/zonefs/file.c b/fs/zonefs/file.c
index 5b34849be7a2..9847ce9cd68b 100644
--- a/fs/zonefs/file.c
+++ b/fs/zonefs/file.c
@@ -192,6 +192,7 @@ const struct address_space_operations zonefs_file_aops = {
 	.readahead		= zonefs_readahead,
 	.writepages		= zonefs_writepages,
 	.dirty_folio		= iomap_dirty_folio,
+	.accounted_dirty_pages	= iomap_accounted_dirty_pages,
 	.release_folio		= iomap_release_folio,
 	.invalidate_folio	= iomap_invalidate_folio,
 	.migrate_folio		= filemap_migrate_folio,
diff --git a/include/linux/iomap.h b/include/linux/iomap.h
index bc7ae6327dbf..fbf6480ac928 100644
--- a/include/linux/iomap.h
+++ b/include/linux/iomap.h
@@ -444,6 +444,7 @@ bool iomap_release_folio(struct folio *folio, gfp_t gfp_flags);
 void iomap_invalidate_folio(struct folio *folio, size_t offset, size_t len);
 bool iomap_dirty_folio(struct address_space *mapping, struct folio *folio);
 void iomap_folio_mark_uptodate(struct folio *folio);
+long iomap_accounted_dirty_pages(struct folio *folio);
 int iomap_file_unshare(struct inode *inode, loff_t pos, loff_t len,
 		const struct iomap_ops *ops,
 		const struct iomap_write_ops *write_ops);
-- 
2.50.1




AMAZON DEVELOPMENT CENTER ITALY SRL, viale Monte Grappa 3/5, 20124 Milano, Italia, Registro delle Imprese di Milano Monza Brianza Lodi REA n. 2504859, Capitale Sociale: 10.000 EUR i.v., Cod. Fisc. e P.IVA 10100050961, Societa con Socio Unico

Re: [PATCH 0/2] mm, iomap: account dirty at page granularity for per-block dirty folios

[David Hildenbrand (Arm)]

On 10/27/26 12:27, Salvatore Dipietro wrote:
> Since iomap gained per-block dirty tracking, a large folio can be
> partially dirty: struct iomap_folio_state records exactly which
> blocks are dirty, and writeback only writes those ranges.  The mm
> dirty accounting, however, still charges the whole folio even if a
> single block is dirty (up to 512x over-accounting for a 2 MB folio
> over 4 KB blocks).
> 
> balance_dirty_pages() then throttles writers against a number that
> is unrelated to how much data actually needs writing back.  On
> workloads that repeatedly dirty small ranges of large folios --
> e.g. PostgreSQL, which writes mostly 8/16 KB pages -- dirty_thresh
> is crossed and writers are throttled long before the real dirty data
> justifies it.  On the benchmark below, run on v7.3-rc7, the
> unpatched kernel charges ~159 TB of dirtied pages while writing
> ~6 TB to the array, a factor of 27; with this series the dirtied
> pages charged match the data actually written.
> 
> This series makes iomap charge and discharge the mm dirty counters
> per dirty page for folios that carry per-block dirty state, so a
> partially dirty large folio is accounted for the dirty part only:
> 
>   - Patch 1 adds the mm infrastructure: an AS_DIRTY_PER_BLOCK
>     mapping flag, an @accounted argument to __folio_mark_dirty()
>     and __filemap_dirty_folio() to skip the generic per-folio
>     charge, and an address_space_operations::accounted_dirty_pages()
>     callback used by the discharge and migration paths instead of
>     folio_nr_pages().  No filesystem opts in, so it is a no-op on
>     its own.
> 
>   - Patch 2 makes iomap charge per dirty page from the per-block
>     dirty bitmap and wires iomap_accounted_dirty_pages() into every
>     iomap-based filesystem.
> 
> This was benchmarked with PostgreSQL using the pgbench simple-update
> workload on v7.3-rc7.  Average throughput rises 46.8% (139.8k ->
> 205.2k tps) and run-to-run variability drops from a 23.60% to a
> 3.98% coefficient of variation: the worst iteration improves from
> 97.9k to 191.5k tps, while the best gains 8%, 199.4k -> 215.9k.
> 
> Ten iterations per kernel, same database and workload (tps):
> 
>   Kernel                     Avg     Min     Max      CV   iters   iters
>                                                              1-5    6-10
>   ______________________  ______  ______  ______  ______  ______  ______
>   v7.3-rc7                139.8k   97.9k  199.4k  23.60%  164.4k  115.2k
>   v7.3-rc7 + this series  205.2k  191.5k  215.9k   3.98%  200.1k  210.2k
> 
> On unpatched v7.3-rc7 the worst iteration lands ~51% below the best,
> and throughput decays as the run progresses: iterations 6-10 average
> 30% below iterations 1-5.  With this series the decay is gone -- the
> second half is similar to the first -- and the coefficient of
> variation stays at 3.98%.
> 
> The over-accounting has existed since commit 4ce02c679722 ("iomap: Add
> per-block dirty state tracking to improve performance"), merged in
> v6.6, where the ifs bitmap started recording partial dirty state while
> the mm counters kept charging whole folios.
> 
> The throughput collapse bisects to the v6.17 readahead series, first
> bad commit c4602f9fa77f ("mm/readahead: store folio order in struct
> file_ra_state"), which lets readahead stay at the largest folio order
> once it has ramped up, so a single dirty block charges far more than
> is written.  Patch 2 therefore carries:


How does this relate to

[RFC PATCH 0/5] mm: sub-folio dirty tracking for PTE-mapped mmap writes

https://lore.kernel.org/all/20260903182943.662461-1-kirill@shutemov.name/

?

I assume they try to solve different problems, but would more fine-grained folio
dirtying also make your solution benefit?

-- 
Cheers,

David

Re: [PATCH 0/2] mm, iomap: account dirty at page granularity for per-block dirty folios

[Kiryl Shutsemau]

On Tue, Oct 27, 2026 at 01:22:22PM +0100, David Hildenbrand (Arm) wrote:
> On 10/27/26 12:27, Salvatore Dipietro wrote:
> > Since iomap gained per-block dirty tracking, a large folio can be
> > partially dirty: struct iomap_folio_state records exactly which
> > blocks are dirty, and writeback only writes those ranges.  The mm
> > dirty accounting, however, still charges the whole folio even if a
> > single block is dirty (up to 512x over-accounting for a 2 MB folio
> > over 4 KB blocks).
> > 
> > balance_dirty_pages() then throttles writers against a number that
> > is unrelated to how much data actually needs writing back.  On
> > workloads that repeatedly dirty small ranges of large folios --
> > e.g. PostgreSQL, which writes mostly 8/16 KB pages -- dirty_thresh
> > is crossed and writers are throttled long before the real dirty data
> > justifies it.  On the benchmark below, run on v7.3-rc7, the
> > unpatched kernel charges ~159 TB of dirtied pages while writing
> > ~6 TB to the array, a factor of 27; with this series the dirtied
> > pages charged match the data actually written.
> > 
> > This series makes iomap charge and discharge the mm dirty counters
> > per dirty page for folios that carry per-block dirty state, so a
> > partially dirty large folio is accounted for the dirty part only:
> > 
> >   - Patch 1 adds the mm infrastructure: an AS_DIRTY_PER_BLOCK
> >     mapping flag, an @accounted argument to __folio_mark_dirty()
> >     and __filemap_dirty_folio() to skip the generic per-folio
> >     charge, and an address_space_operations::accounted_dirty_pages()
> >     callback used by the discharge and migration paths instead of
> >     folio_nr_pages().  No filesystem opts in, so it is a no-op on
> >     its own.
> > 
> >   - Patch 2 makes iomap charge per dirty page from the per-block
> >     dirty bitmap and wires iomap_accounted_dirty_pages() into every
> >     iomap-based filesystem.
> > 
> > This was benchmarked with PostgreSQL using the pgbench simple-update
> > workload on v7.3-rc7.  Average throughput rises 46.8% (139.8k ->
> > 205.2k tps) and run-to-run variability drops from a 23.60% to a
> > 3.98% coefficient of variation: the worst iteration improves from
> > 97.9k to 191.5k tps, while the best gains 8%, 199.4k -> 215.9k.
> > 
> > Ten iterations per kernel, same database and workload (tps):
> > 
> >   Kernel                     Avg     Min     Max      CV   iters   iters
> >                                                              1-5    6-10
> >   ______________________  ______  ______  ______  ______  ______  ______
> >   v7.3-rc7                139.8k   97.9k  199.4k  23.60%  164.4k  115.2k
> >   v7.3-rc7 + this series  205.2k  191.5k  215.9k   3.98%  200.1k  210.2k
> > 
> > On unpatched v7.3-rc7 the worst iteration lands ~51% below the best,
> > and throughput decays as the run progresses: iterations 6-10 average
> > 30% below iterations 1-5.  With this series the decay is gone -- the
> > second half is similar to the first -- and the coefficient of
> > variation stays at 3.98%.
> > 
> > The over-accounting has existed since commit 4ce02c679722 ("iomap: Add
> > per-block dirty state tracking to improve performance"), merged in
> > v6.6, where the ifs bitmap started recording partial dirty state while
> > the mm counters kept charging whole folios.
> > 
> > The throughput collapse bisects to the v6.17 readahead series, first
> > bad commit c4602f9fa77f ("mm/readahead: store folio order in struct
> > file_ra_state"), which lets readahead stay at the largest folio order
> > once it has ramped up, so a single dirty block charges far more than
> > is written.  Patch 2 therefore carries:
> 
> 
> How does this relate to
> 
> [RFC PATCH 0/5] mm: sub-folio dirty tracking for PTE-mapped mmap writes
> 
> https://lore.kernel.org/all/20260903182943.662461-1-kirill@shutemov.name/
> 
> ?
> 
> I assume they try to solve different problems, but would more fine-grained folio
> dirtying also make your solution benefit?

Yeah, it solves different problem.

And I abandoned my patchset. The right way to do this is to make
page_mkwrite() per-page, not folio.

-- 
  Kiryl Shutsemau / Kirill A. Shutemov

Re: [PATCH 0/2] mm, iomap: account dirty at page granularity for per-block dirty folios

[Pedro Falcato]

On Tue, Oct 27, 2026 at 11:27:00AM +0000, Salvatore Dipietro wrote:
> Since iomap gained per-block dirty tracking, a large folio can be
> partially dirty: struct iomap_folio_state records exactly which
> blocks are dirty, and writeback only writes those ranges.  The mm
> dirty accounting, however, still charges the whole folio even if a
> single block is dirty (up to 512x over-accounting for a 2 MB folio
> over 4 KB blocks).
> 
> balance_dirty_pages() then throttles writers against a number that
> is unrelated to how much data actually needs writing back.  On
> workloads that repeatedly dirty small ranges of large folios --
> e.g. PostgreSQL, which writes mostly 8/16 KB pages -- dirty_thresh
> is crossed and writers are throttled long before the real dirty data
> justifies it.  On the benchmark below, run on v7.3-rc7, the
> unpatched kernel charges ~159 TB of dirtied pages while writing
> ~6 TB to the array, a factor of 27; with this series the dirtied
> pages charged match the data actually written.
> 
> This series makes iomap charge and discharge the mm dirty counters
> per dirty page for folios that carry per-block dirty state, so a
> partially dirty large folio is accounted for the dirty part only:

Why? This looks wrong. Yes, you may only have dirtied e.g 4K out of a 2MB
folio, but the whole 2MB folio is still effectively pinned by the VM, awaiting
cleaning + writeback. And that's precisely what we want to measure and
throttle in certain cases, so that e.g the VM system never finds itself
short of clean pages it can reclaim, and no easy way to start writeback
on them.

-- 
Pedro