Re: [PATCH] erofs: don't bother with s_stack_depth increasing for now

[Gao Xiang <hsiangkao@linux.alibaba.com>]

On 2026/1/4 18:01, Amir Goldstein wrote:
> [+fsdevel][+overlayfs]
> 
> On Sun, Jan 4, 2026 at 4:56 AM Gao Xiang <hsiangkao@linux.alibaba.com> wrote:
>>
>> Hi Amir,
>>
>> On 2026/1/1 23:52, Amir Goldstein wrote:
>>> On Wed, Dec 31, 2025 at 9:42 PM Gao Xiang <hsiangkao@linux.alibaba.com> wrote:
>>>>
>>>> Previously, commit d53cd891f0e4 ("erofs: limit the level of fs stacking
>>>> for file-backed mounts") bumped `s_stack_depth` by one to avoid kernel
>>>> stack overflow, but it breaks composefs mounts, which need erofs+ovl^2
>>>> sometimes (and such setups are already used in production for quite long
>>>> time) since `s_stack_depth` can be 3 (i.e., FILESYSTEM_MAX_STACK_DEPTH
>>>> needs to change from 2 to 3).
>>>>
>>>> After a long discussion on GitHub issues [1] about possible solutions,
>>>> it seems there is no need to support nesting file-backed mounts as one
>>>> conclusion (especially when increasing FILESYSTEM_MAX_STACK_DEPTH to 3).
>>>> So let's disallow this right now, since there is always a way to use
>>>> loopback devices as a fallback.
>>>>
>>>> Then, I started to wonder about an alternative EROFS quick fix to
>>>> address the composefs mounts directly for this cycle: since EROFS is the
>>>> only fs to support file-backed mounts and other stacked fses will just
>>>> bump up `FILESYSTEM_MAX_STACK_DEPTH`, just check that `s_stack_depth`
>>>> != 0 and the backing inode is not from EROFS instead.
>>>>
>>>> At least it works for all known file-backed mount use cases (composefs,
>>>> containerd, and Android APEX for some Android vendors), and the fix is
>>>> self-contained.
>>>>
>>>> Let's defer increasing FILESYSTEM_MAX_STACK_DEPTH for now.
>>>>
>>>> Fixes: d53cd891f0e4 ("erofs: limit the level of fs stacking for file-backed mounts")
>>>> Closes: https://github.com/coreos/fedora-coreos-tracker/issues/2087 [1]
>>>> Closes: https://lore.kernel.org/r/CAFHtUiYv4+=+JP_-JjARWjo6OwcvBj1wtYN=z0QXwCpec9sXtg@mail.gmail.com
>>>> Cc: Amir Goldstein <amir73il@gmail.com>
>>>> Cc: Alexander Larsson <alexl@redhat.com>
>>>> Cc: Christian Brauner <brauner@kernel.org>
>>>> Cc: Miklos Szeredi <mszeredi@redhat.com>
>>>> Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
>>>> ---
>>>
>>> Acked-by: Amir Goldstein <amir73il@gmail.com>
>>>
>>> But you forgot to include details of the stack usage analysis you ran
>>> with erofs+ovl^2 setup.
>>>
>>> I am guessing people will want to see this information before relaxing
>>> s_stack_depth in this case.
>>
>> Sorry I didn't check emails these days, I'm not sure if posting
>> detailed stack traces are useful, how about adding the following
>> words:
> 
> Didn't mean detailed stack traces, but you did some tests with the
> new possible setup and you reached stack usage < 8K so  I think this is

The issue is that my limited stress test setup cannot cover
every cases:

  - I cannot find a way to make direct reclaim reliably in the
    deep memory allocation, is there some suggestion on this?

  - I'm not sure what's the perfered way to evaluate the worst
    stack usage below the block layer, but we should care more
    about increasing delta just out of one more overlayfs I
    guess?

I can only say what I've seen is the peak stack usage of my
fsstress for an erofs+ovl^2 setup on x86_64 is < 8K (7184 bytes,
but I don't think the peak value absolutely useful), which
evaluates RW workloads in the upperdir, and for such workloads,
the stack depth won't be impacted by FILESYSTEM_MAX_STACK_DEPTH,
I don't see such workload is harmful.

And then I manually copyup some files (because I didn't find any
available tool to stress overlayfs copyups) and I could see the
delta is (I think "ovl_copy_up_" is the only one path to do
copyups):

   0)     6688      48   mempool_alloc_slab+0x9/0x20
   1)     6640      56   mempool_alloc_noprof+0x65/0xd0
   2)     6584      72   __sg_alloc_table+0x128/0x190
   3)     6512      40   sg_alloc_table_chained+0x46/0xa0
   4)     6472      64   scsi_alloc_sgtables+0x91/0x2c0
   5)     6408      72   sd_init_command+0x263/0x930
   6)     6336      88   scsi_queue_rq+0x54a/0xb70
   7)     6248     144   blk_mq_dispatch_rq_list+0x265/0x6c0
   8)     6104     144   __blk_mq_sched_dispatch_requests+0x399/0x5c0
   9)     5960      16   blk_mq_sched_dispatch_requests+0x2d/0x70
  10)     5944      56   blk_mq_run_hw_queue+0x208/0x290
  11)     5888      96   blk_mq_dispatch_list+0x13f/0x460
  12)     5792      48   blk_mq_flush_plug_list+0x4b/0x180
  13)     5744      32   blk_add_rq_to_plug+0x3d/0x160
  14)     5712     136   blk_mq_submit_bio+0x4f4/0x760
  15)     5576     120   __submit_bio+0x9b/0x240
  16)     5456      88   submit_bio_noacct_nocheck+0x271/0x330
  17)     5368      72   iomap_bio_read_folio_range+0xde/0x1d0
  18)     5296     112   iomap_read_folio_iter+0x1ee/0x2d0
  19)     5184     264   iomap_readahead+0xb9/0x290
  20)     4920      48   xfs_vm_readahead+0x4a/0x70
  21)     4872     112   read_pages+0x6c/0x1b0
  22)     4760     104   page_cache_ra_unbounded+0x12c/0x210
  23)     4656      80   filemap_readahead.isra.0+0x78/0xb0
  24)     4576     192   filemap_get_pages+0x3a6/0x820
  25)     4384     376   filemap_read+0xde/0x380
  26)     4008      32   xfs_file_buffered_read+0xa6/0xd0
  27)     3976      16   xfs_file_read_iter+0x6a/0xd0
  28)     3960      48   vfs_iocb_iter_read+0xdb/0x140
  29)     3912      88   erofs_fileio_rq_submit+0x136/0x190
  30)     3824     368   z_erofs_runqueue+0x1ce/0x9f0
  31)     3456     232   z_erofs_readahead+0x16c/0x220
  32)     3224     112   read_pages+0x6c/0x1b0
  33)     3112     104   page_cache_ra_unbounded+0x12c/0x210
  34)     3008      80   filemap_readahead.isra.0+0x78/0xb0
  35)     2928     192   filemap_get_pages+0x3a6/0x820
  36)     2736     400   filemap_splice_read+0x12c/0x2f0
  37)     2336      48   backing_file_splice_read+0x3f/0x90
  38)     2288     128   ovl_splice_read+0xef/0x170
  39)     2160     104   splice_direct_to_actor+0xb9/0x260
  40)     2056      88   do_splice_direct+0x76/0xc0
  41)     1968     120   ovl_copy_up_file+0x1a8/0x2b0
  42)     1848     840   ovl_copy_up_one+0x14b0/0x1610
  43)     1008      72   ovl_copy_up_flags+0xd7/0x110
  44)      936      56   ovl_open+0x72/0x110
  45)      880      56   do_dentry_open+0x16c/0x480
  46)      824      40   vfs_open+0x2e/0xf0
  47)      784     152   path_openat+0x80a/0x12e0
  48)      632     296   do_filp_open+0xb8/0x160
  49)      336      80   do_sys_openat2+0x72/0xf0
  50)      256      40   __x64_sys_openat+0x57/0xa0
  51)      216      40   do_syscall_64+0xa4/0x310
  52)      176     176   entry_SYSCALL_64_after_hwframe+0x77/0x7f

And it's still far from the stack overflow of 16k stacks,
because the difference seems only how many (
ovl_splice_read + backing_file_splice_read), and there only takes
hundreds of bytes for each layer.

Finally I used my own rostress to stress RO workloads, and the
deepest stack so far is as below (5456 bytes):

   0)     5456      48   arch_scale_cpu_capacity+0x9/0x30
   1)     5408      16   cpu_util.constprop.0+0x7e/0xe0
   2)     5392     392   sched_balance_find_src_group+0x29f/0xd30
   3)     5000     280   sched_balance_rq+0x1b2/0xf10
   4)     4720     120   pick_next_task_fair+0x23b/0x7b0
   5)     4600     104   __schedule+0x2bc/0xda0
   6)     4496      16   schedule+0x27/0xd0
   7)     4480      24   io_schedule+0x46/0x70
   8)     4456     120   blk_mq_get_tag+0x11b/0x280
   9)     4336      96   __blk_mq_alloc_requests+0x2a1/0x410
  10)     4240     136   blk_mq_submit_bio+0x59c/0x760
  11)     4104     120   __submit_bio+0x9b/0x240
  12)     3984      88   submit_bio_noacct_nocheck+0x271/0x330
  13)     3896      72   iomap_bio_read_folio_range+0xde/0x1d0
  14)     3824     112   iomap_read_folio_iter+0x1ee/0x2d0
  15)     3712     264   iomap_readahead+0xb9/0x290
  16)     3448      48   xfs_vm_readahead+0x4a/0x70
  17)     3400     112   read_pages+0x6c/0x1b0
  18)     3288     104   page_cache_ra_unbounded+0x12c/0x210
  19)     3184      80   filemap_readahead.isra.0+0x78/0xb0
  20)     3104     192   filemap_get_pages+0x3a6/0x820
  21)     2912     376   filemap_read+0xde/0x380
  22)     2536      32   xfs_file_buffered_read+0xa6/0xd0
  23)     2504      16   xfs_file_read_iter+0x6a/0xd0
  24)     2488      48   vfs_iocb_iter_read+0xdb/0x140
  25)     2440      88   erofs_fileio_rq_submit+0x136/0x190
  26)     2352     368   z_erofs_runqueue+0x1ce/0x9f0
  27)     1984     232   z_erofs_readahead+0x16c/0x220
  28)     1752     112   read_pages+0x6c/0x1b0
  29)     1640     104   page_cache_ra_unbounded+0x12c/0x210
  30)     1536      40   force_page_cache_ra+0x96/0xc0
  31)     1496     192   filemap_get_pages+0x123/0x820
  32)     1304     376   filemap_read+0xde/0x380
  33)      928      72   do_iter_readv_writev+0x1b9/0x220
  34)      856      56   vfs_iter_read+0xde/0x140
  35)      800      64   backing_file_read_iter+0x193/0x1e0
  36)      736      56   ovl_read_iter+0x98/0xa0
  37)      680      72   do_iter_readv_writev+0x1b9/0x220
  38)      608      56   vfs_iter_read+0xde/0x140
  39)      552      64   backing_file_read_iter+0x193/0x1e0
  40)      488      56   ovl_read_iter+0x98/0xa0
  41)      432     152   vfs_read+0x21a/0x350
  42)      280      64   __x64_sys_pread64+0x92/0xc0
  43)      216      40   do_syscall_64+0xa4/0x310
  44)      176     176   entry_SYSCALL_64_after_hwframe+0x77/0x7f

> something worth mentioning.
> 
>>
>> Note: There are some observations while evaluating the erofs + ovl^2
>> setup with an XFS backing fs:
>>
>>    - Regular RW workloads traverse only one overlayfs layer regardless of
>>      the value of FILESYSTEM_MAX_STACK_DEPTH, because `upperdir=` cannot
>>      point to another overlayfs.  Therefore, for pure RW workloads, the
>>      typical stack is always just:
>>        overlayfs + upper fs + underlay storage
>>
>>    - For read-only workloads and the copy-up read part (ovl_splice_read),
>>      the difference can lie in how many overlays are nested.
>>      The stack just looks like either:
>>        ovl + ovl [+ erofs] + backing fs + underlay storage
>>      or
>>        ovl [+ erofs] + ext4/xfs + underlay storage
>>
>>    - The fs reclaim path should be entered only once, so the writeback
>>      path will not re-enter.
>>
>> Sorry about my English, and I'm not sure if it's enough (e.g. FUSE
>> passthrough part).  I will look for your further inputs (and other
>> acks) before sending this patch upstream.
>>
> 
> I think that most people will have problems understanding this
> rationale not because of the English, but because of the tech ;)
> this is a bit too hand wavy IMO.

Honestly, I don't have better way to describe it, I think we'd
better just to focus more on the increment of one more overlayfs:

FILESYSTEM_MAX_STACK_DEPTH 2 already works for 8k kstacks on
32-bit arches, so I don't think FILESYSTEM_MAX_STACK_DEPTH from
2 to 3, which causes hundreds-more-byte additional stack usage
out of mediate overlayfs on 16k kstacks on 64-bit arches is
harmful (and only RO workloads and copyups are impacted).

And if hundreds-more-byte additional stack usage can overflow
the 16k kstack, I do think then the kernel stack can be
overflowed randomly everywhere in the storage stack, not just
because this FILESYSTEM_MAX_STACK_DEPTH modification.

Thanks,
Gao Xiang

> 
>> (Also btw, i'm not sure if it's possible to optimize read_iter and
>>    splice_read stack usage even further in overlayfs, e.g. just
>>    recursive handling real file/path directly in the top overlayfs
>>    since the permission check is already done when opening the file.)
> 
> Maybe so, but LSM permission to open hook is not the same hook
> as permission to read/write.
> 
> Thanks,
> Amir.