* [PATCH v4] ring-buffer: Do not check si_mem_available() before deferred init completes
@ 2026-10-23 14:54 Amit Machhiwal
2026-10-26 16:51 ` Vincent Donnefort
` (2 more replies)
0 siblings, 3 replies; 5+ messages in thread
From: Amit Machhiwal @ 2026-10-23 14:54 UTC (permalink / raw)
To: Steven Rostedt, Masami Hiramatsu, linux-trace-kernel, linuxppc-dev
Cc: Amit Machhiwal, Michal Suchánek, Vincent Donnefort,
Mathieu Desnoyers, Madhavan Srinivasan, Ritesh Harjani (IBM),
Harsh Prateek Bora, linux-kernel, kvm, kvm-ppc, stable
During early boot (early_trace_init() called from start_kernel()),
tracing allocates initial ring buffers (temp_buffer, array_buffer, and
snapshot_buffer) for CPU 0. Later, as secondary CPUs come online during
smp_init(), each CPU's ring buffer is allocated via trace_rb_cpu_prepare().
Before attempting page allocation, __rb_allocate_pages() performs a
heuristic check using si_mem_available() to return early with -ENOMEM if
memory appears insufficient.
However, on kernels built with CONFIG_DEFERRED_STRUCT_PAGE_INIT=y,
NR_FREE_PAGES only reflects the initial non-deferred pool across the entire
period from early_trace_init() through smp_init(). defer_init() leaves
only a single section per node (e.g. 16 MiB with 64 KB pages) initialized
up-front, and on kernels with large static binary footprints (debug
configurations enabling PAGE_OWNER, DEBUG_PAGEALLOC, KFENCE, or SLUB_DEBUG),
early core initialisations (SLUB caches, vmalloc, static ftrace records)
consume virtually all managed pages in this pool.
si_mem_available() therefore returns a false negative (often 0-1 pages)
across both SYSTEM_BOOTING (for the initial CPU 0 buffers) and
SYSTEM_SCHEDULING (for secondary CPU buffers via trace_rb_cpu_prepare()),
causing __rb_allocate_pages() to prematurely abort with -ENOMEM.
This failure is false: the buddy allocator is designed to replenish
ZONE_NORMAL on demand via deferred_grow_zone() when an actual allocation
is attempted. Checking si_mem_available() before calling alloc_pages_node()
short-circuits this on-demand growth entirely.
deferred_grow_zone() is only permanently disabled after page_alloc_init_late()
completes (it spawns per-node kthreads and blocks until done), which happens
inside kernel_init_freeable() — before system_state is set to
SYSTEM_FREEING_INITMEM. Skip the si_mem_available() check until
SYSTEM_FREEING_INITMEM, at which point all deferred pages are initialized
and si_mem_available() is accurate.
Use IS_ENABLED(CONFIG_DEFERRED_STRUCT_PAGE_INIT) to set i = LONG_MAX during
this window, unconditionally bypassing the threshold check. On kernels
without deferred init IS_ENABLED() compiles the branch away and the original
check is preserved unchanged.
Fixes: 2a872fa4e9c8 ("ring-buffer: Check if memory is available before allocation")
Cc: stable@vger.kernel.org
Reported-by: Michal Suchánek <msuchanek@suse.de>
Closes: https://lore.kernel.org/all/arYskzbiaNzBR9MD@kunlun.suse.cz/
Link: https://lore.kernel.org/all/arYskzbiaNzBR9MD@kunlun.suse.cz/
Signed-off-by: Amit Machhiwal <amachhiw@linux.ibm.com>
---
Changes since v3:
- Replace #ifdef CONFIG_DEFERRED_STRUCT_PAGE_INIT / #else / #endif with
IS_ENABLED() + i = LONG_MAX to avoid duplicating the threshold check.
(Vincent Donnefort)
- Trim the in-code comment to one sentence. (Vincent Donnefort)
Changes since v2:
- Rebase on top of upstream ring_buffer.c changes from Vincent Donnefort;
update threshold to i < (nr_pages << subbuf_order). (Michal Suchánek)
Changes since v1:
- Extend the guard to cover SYSTEM_SCHEDULING as well: smp_init() calls
trace_rb_cpu_prepare() -> __rb_allocate_pages() before page_alloc_init_late()
completes, so the false negative window spans both SYSTEM_BOOTING and
SYSTEM_SCHEDULING. Use system_state >= SYSTEM_FREEING_INITMEM as the correct
boundary. (Sashiko, Vincent Donnefort)
- Wrap in #ifdef CONFIG_DEFERRED_STRUCT_PAGE_INIT to preserve the original
check unchanged on non-deferred kernels. (Vincent Donnefort)
- Drop Tested-by: guard condition changed, requesting fresh test.
kernel/trace/ring_buffer.c | 8 ++++++++
1 file changed, 8 insertions(+)
diff --git a/kernel/trace/ring_buffer.c b/kernel/trace/ring_buffer.c
index 9c4e40847e43..09c02e838c4d 100644
--- a/kernel/trace/ring_buffer.c
+++ b/kernel/trace/ring_buffer.c
@@ -2463,8 +2463,16 @@ static int __rb_allocate_pages(struct ring_buffer_per_cpu *cpu_buffer,
* memory. It may not be accurate. But we don't care, we just want
* to prevent doing any allocation when it is obvious that it is
* not going to succeed.
+ *
+ * With CONFIG_DEFERRED_STRUCT_PAGE_INIT, si_mem_available()
+ * will reflect the actual available memory only after
+ * SYSTEM_FREEING_INITMEM.
*/
i = si_mem_available();
+ if (IS_ENABLED(CONFIG_DEFERRED_STRUCT_PAGE_INIT) &&
+ system_state < SYSTEM_FREEING_INITMEM)
+ i = LONG_MAX;
+
if (i < (nr_pages << subbuf_order))
return -ENOMEM;
base-commit: 734bef548a96e8774d6eef5dfa2026c6edc4e80b
--
2.54.0 (Apple Git-157)
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [PATCH v4] ring-buffer: Do not check si_mem_available() before deferred init completes
2026-10-23 14:54 [PATCH v4] ring-buffer: Do not check si_mem_available() before deferred init completes Amit Machhiwal
@ 2026-10-26 16:51 ` Vincent Donnefort
2026-11-17 18:45 ` Amit Machhiwal
2026-11-17 21:28 ` Steven Rostedt
2 siblings, 0 replies; 5+ messages in thread
From: Vincent Donnefort @ 2026-10-26 16:51 UTC (permalink / raw)
To: Amit Machhiwal
Cc: Steven Rostedt, Masami Hiramatsu, linux-trace-kernel,
linuxppc-dev, Michal Suchánek, Mathieu Desnoyers,
Madhavan Srinivasan, Ritesh Harjani (IBM),
Harsh Prateek Bora, linux-kernel, kvm, kvm-ppc, stable
On Fri, Oct 23, 2026 at 08:24:59PM +0530, Amit Machhiwal wrote:
> During early boot (early_trace_init() called from start_kernel()),
> tracing allocates initial ring buffers (temp_buffer, array_buffer, and
> snapshot_buffer) for CPU 0. Later, as secondary CPUs come online during
> smp_init(), each CPU's ring buffer is allocated via trace_rb_cpu_prepare().
>
> Before attempting page allocation, __rb_allocate_pages() performs a
> heuristic check using si_mem_available() to return early with -ENOMEM if
> memory appears insufficient.
>
> However, on kernels built with CONFIG_DEFERRED_STRUCT_PAGE_INIT=y,
> NR_FREE_PAGES only reflects the initial non-deferred pool across the entire
> period from early_trace_init() through smp_init(). defer_init() leaves
> only a single section per node (e.g. 16 MiB with 64 KB pages) initialized
> up-front, and on kernels with large static binary footprints (debug
> configurations enabling PAGE_OWNER, DEBUG_PAGEALLOC, KFENCE, or SLUB_DEBUG),
> early core initialisations (SLUB caches, vmalloc, static ftrace records)
> consume virtually all managed pages in this pool.
>
> si_mem_available() therefore returns a false negative (often 0-1 pages)
> across both SYSTEM_BOOTING (for the initial CPU 0 buffers) and
> SYSTEM_SCHEDULING (for secondary CPU buffers via trace_rb_cpu_prepare()),
> causing __rb_allocate_pages() to prematurely abort with -ENOMEM.
>
> This failure is false: the buddy allocator is designed to replenish
> ZONE_NORMAL on demand via deferred_grow_zone() when an actual allocation
> is attempted. Checking si_mem_available() before calling alloc_pages_node()
> short-circuits this on-demand growth entirely.
>
> deferred_grow_zone() is only permanently disabled after page_alloc_init_late()
> completes (it spawns per-node kthreads and blocks until done), which happens
> inside kernel_init_freeable() — before system_state is set to
> SYSTEM_FREEING_INITMEM. Skip the si_mem_available() check until
> SYSTEM_FREEING_INITMEM, at which point all deferred pages are initialized
> and si_mem_available() is accurate.
>
> Use IS_ENABLED(CONFIG_DEFERRED_STRUCT_PAGE_INIT) to set i = LONG_MAX during
> this window, unconditionally bypassing the threshold check. On kernels
> without deferred init IS_ENABLED() compiles the branch away and the original
> check is preserved unchanged.
Commit messages are not LLMs' strong suit.
Otherwise.
Reviewed-by: Vincent Donnefort <vdonnefort@google.com>
>
> Fixes: 2a872fa4e9c8 ("ring-buffer: Check if memory is available before allocation")
> Cc: stable@vger.kernel.org
> Reported-by: Michal Suchánek <msuchanek@suse.de>
> Closes: https://lore.kernel.org/all/arYskzbiaNzBR9MD@kunlun.suse.cz/
> Link: https://lore.kernel.org/all/arYskzbiaNzBR9MD@kunlun.suse.cz/
> Signed-off-by: Amit Machhiwal <amachhiw@linux.ibm.com>
> ---
> Changes since v3:
> - Replace #ifdef CONFIG_DEFERRED_STRUCT_PAGE_INIT / #else / #endif with
> IS_ENABLED() + i = LONG_MAX to avoid duplicating the threshold check.
> (Vincent Donnefort)
> - Trim the in-code comment to one sentence. (Vincent Donnefort)
>
> Changes since v2:
> - Rebase on top of upstream ring_buffer.c changes from Vincent Donnefort;
> update threshold to i < (nr_pages << subbuf_order). (Michal Suchánek)
>
> Changes since v1:
> - Extend the guard to cover SYSTEM_SCHEDULING as well: smp_init() calls
> trace_rb_cpu_prepare() -> __rb_allocate_pages() before page_alloc_init_late()
> completes, so the false negative window spans both SYSTEM_BOOTING and
> SYSTEM_SCHEDULING. Use system_state >= SYSTEM_FREEING_INITMEM as the correct
> boundary. (Sashiko, Vincent Donnefort)
> - Wrap in #ifdef CONFIG_DEFERRED_STRUCT_PAGE_INIT to preserve the original
> check unchanged on non-deferred kernels. (Vincent Donnefort)
> - Drop Tested-by: guard condition changed, requesting fresh test.
>
> kernel/trace/ring_buffer.c | 8 ++++++++
> 1 file changed, 8 insertions(+)
>
> diff --git a/kernel/trace/ring_buffer.c b/kernel/trace/ring_buffer.c
> index 9c4e40847e43..09c02e838c4d 100644
> --- a/kernel/trace/ring_buffer.c
> +++ b/kernel/trace/ring_buffer.c
> @@ -2463,8 +2463,16 @@ static int __rb_allocate_pages(struct ring_buffer_per_cpu *cpu_buffer,
> * memory. It may not be accurate. But we don't care, we just want
> * to prevent doing any allocation when it is obvious that it is
> * not going to succeed.
> + *
> + * With CONFIG_DEFERRED_STRUCT_PAGE_INIT, si_mem_available()
> + * will reflect the actual available memory only after
> + * SYSTEM_FREEING_INITMEM.
> */
> i = si_mem_available();
> + if (IS_ENABLED(CONFIG_DEFERRED_STRUCT_PAGE_INIT) &&
> + system_state < SYSTEM_FREEING_INITMEM)
> + i = LONG_MAX;
> +
> if (i < (nr_pages << subbuf_order))
> return -ENOMEM;
>
>
> base-commit: 734bef548a96e8774d6eef5dfa2026c6edc4e80b
> --
> 2.54.0 (Apple Git-157)
>
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [PATCH v4] ring-buffer: Do not check si_mem_available() before deferred init completes
2026-10-23 14:54 [PATCH v4] ring-buffer: Do not check si_mem_available() before deferred init completes Amit Machhiwal
2026-10-26 16:51 ` Vincent Donnefort
@ 2026-11-17 18:45 ` Amit Machhiwal
2026-11-17 21:28 ` Steven Rostedt
2 siblings, 0 replies; 5+ messages in thread
From: Amit Machhiwal @ 2026-11-17 18:45 UTC (permalink / raw)
To: Steven Rostedt, Masami Hiramatsu, linux-trace-kernel, linuxppc-dev
Cc: Amit Machhiwal, Michal Suchánek, Vincent Donnefort,
Mathieu Desnoyers, Madhavan Srinivasan, Ritesh Harjani (IBM),
Harsh Prateek Bora, linux-kernel, kvm, kvm-ppc, stable
Hi Steven,
Gentle ping on this patch. This has Vincent's R-b and fixes a boot-time
ring-buffer allocation failure on DEFERRED_STRUCT_PAGE_INIT kernels. Is there
anything else needed from my side? I'm happy to trim the commit message or send
a v5 if that would help.
Thanks,
Amit
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [PATCH v4] ring-buffer: Do not check si_mem_available() before deferred init completes
2026-10-23 14:54 [PATCH v4] ring-buffer: Do not check si_mem_available() before deferred init completes Amit Machhiwal
2026-10-26 16:51 ` Vincent Donnefort
2026-11-17 18:45 ` Amit Machhiwal
@ 2026-11-17 21:28 ` Steven Rostedt
2026-11-18 18:32 ` Amit Machhiwal
2 siblings, 1 reply; 5+ messages in thread
From: Steven Rostedt @ 2026-11-17 21:28 UTC (permalink / raw)
To: Amit Machhiwal
Cc: Masami Hiramatsu, linux-trace-kernel, linuxppc-dev,
Michal Suchánek, Vincent Donnefort, Mathieu Desnoyers,
Madhavan Srinivasan, Ritesh Harjani (IBM),
Harsh Prateek Bora, linux-kernel, kvm, kvm-ppc, stable
On Fri, 23 Oct 2026 20:24:59 +0530
Amit Machhiwal <amachhiw@linux.ibm.com> wrote:
> During early boot (early_trace_init() called from start_kernel()),
> tracing allocates initial ring buffers (temp_buffer, array_buffer, and
> snapshot_buffer) for CPU 0. Later, as secondary CPUs come online during
> smp_init(), each CPU's ring buffer is allocated via trace_rb_cpu_prepare().
>
> Before attempting page allocation, __rb_allocate_pages() performs a
> heuristic check using si_mem_available() to return early with -ENOMEM if
> memory appears insufficient.
>
> However, on kernels built with CONFIG_DEFERRED_STRUCT_PAGE_INIT=y,
> NR_FREE_PAGES only reflects the initial non-deferred pool across the entire
> period from early_trace_init() through smp_init(). defer_init() leaves
> only a single section per node (e.g. 16 MiB with 64 KB pages) initialized
> up-front, and on kernels with large static binary footprints (debug
> configurations enabling PAGE_OWNER, DEBUG_PAGEALLOC, KFENCE, or SLUB_DEBUG),
> early core initialisations (SLUB caches, vmalloc, static ftrace records)
> consume virtually all managed pages in this pool.
>
> si_mem_available() therefore returns a false negative (often 0-1 pages)
> across both SYSTEM_BOOTING (for the initial CPU 0 buffers) and
> SYSTEM_SCHEDULING (for secondary CPU buffers via trace_rb_cpu_prepare()),
> causing __rb_allocate_pages() to prematurely abort with -ENOMEM.
>
> This failure is false: the buddy allocator is designed to replenish
> ZONE_NORMAL on demand via deferred_grow_zone() when an actual allocation
> is attempted. Checking si_mem_available() before calling alloc_pages_node()
> short-circuits this on-demand growth entirely.
>
> deferred_grow_zone() is only permanently disabled after page_alloc_init_late()
> completes (it spawns per-node kthreads and blocks until done), which happens
> inside kernel_init_freeable() — before system_state is set to
> SYSTEM_FREEING_INITMEM. Skip the si_mem_available() check until
> SYSTEM_FREEING_INITMEM, at which point all deferred pages are initialized
> and si_mem_available() is accurate.
>
> Use IS_ENABLED(CONFIG_DEFERRED_STRUCT_PAGE_INIT) to set i = LONG_MAX during
> this window, unconditionally bypassing the threshold check. On kernels
> without deferred init IS_ENABLED() compiles the branch away and the original
> check is preserved unchanged.
As Vincent mentioned, the above is just word salad. Please make it human
readable. That is, state what the actual issue is and how it's fixed and
remove all the AI fluff.
Also note, just because we no longer require an "Assisted-by" tag, it
should still be mentioned how the patch was created. I want to know.
>
> Fixes: 2a872fa4e9c8 ("ring-buffer: Check if memory is available before allocation")
> Cc: stable@vger.kernel.org
> Reported-by: Michal Suchánek <msuchanek@suse.de>
> Closes: https://lore.kernel.org/all/arYskzbiaNzBR9MD@kunlun.suse.cz/
> Link: https://lore.kernel.org/all/arYskzbiaNzBR9MD@kunlun.suse.cz/
> Signed-off-by: Amit Machhiwal <amachhiw@linux.ibm.com>
> ---
> diff --git a/kernel/trace/ring_buffer.c b/kernel/trace/ring_buffer.c
> index 9c4e40847e43..09c02e838c4d 100644
> --- a/kernel/trace/ring_buffer.c
> +++ b/kernel/trace/ring_buffer.c
> @@ -2463,8 +2463,16 @@ static int __rb_allocate_pages(struct ring_buffer_per_cpu *cpu_buffer,
> * memory. It may not be accurate. But we don't care, we just want
> * to prevent doing any allocation when it is obvious that it is
> * not going to succeed.
> + *
> + * With CONFIG_DEFERRED_STRUCT_PAGE_INIT, si_mem_available()
> + * will reflect the actual available memory only after
> + * SYSTEM_FREEING_INITMEM.
> */
> i = si_mem_available();
> + if (IS_ENABLED(CONFIG_DEFERRED_STRUCT_PAGE_INIT) &&
> + system_state < SYSTEM_FREEING_INITMEM)
> + i = LONG_MAX;
Make the above an if/else statement.
if (IS_ENABLED(CONFIG_DEFERRED_STRUCT_PAGE_INIT) &&
unlikely(system_state < SYSTEM_FREEING_INITMEM))
i = LONG_MAX;
else
i = si_mem_available();
-- Steve
> +
> if (i < (nr_pages << subbuf_order))
> return -ENOMEM;
>
>
> base-commit: 734bef548a96e8774d6eef5dfa2026c6edc4e80b
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [PATCH v4] ring-buffer: Do not check si_mem_available() before deferred init completes
2026-11-17 21:28 ` Steven Rostedt
@ 2026-11-18 18:32 ` Amit Machhiwal
0 siblings, 0 replies; 5+ messages in thread
From: Amit Machhiwal @ 2026-11-18 18:32 UTC (permalink / raw)
To: Steven Rostedt
Cc: Amit Machhiwal, Masami Hiramatsu, linux-trace-kernel,
linuxppc-dev, Michal Suchánek, Vincent Donnefort,
Mathieu Desnoyers, Madhavan Srinivasan, Ritesh Harjani (IBM),
Harsh Prateek Bora, linux-kernel, kvm, kvm-ppc, stable
On 2026/11/17 04:28 PM, Steven Rostedt wrote:
> On Fri, 23 Oct 2026 20:24:59 +0530
> Amit Machhiwal <amachhiw@linux.ibm.com> wrote:
>
> > During early boot (early_trace_init() called from start_kernel()),
> > tracing allocates initial ring buffers (temp_buffer, array_buffer, and
> > snapshot_buffer) for CPU 0. Later, as secondary CPUs come online during
> > smp_init(), each CPU's ring buffer is allocated via trace_rb_cpu_prepare().
> >
> > Before attempting page allocation, __rb_allocate_pages() performs a
> > heuristic check using si_mem_available() to return early with -ENOMEM if
> > memory appears insufficient.
> >
> > However, on kernels built with CONFIG_DEFERRED_STRUCT_PAGE_INIT=y,
> > NR_FREE_PAGES only reflects the initial non-deferred pool across the entire
> > period from early_trace_init() through smp_init(). defer_init() leaves
> > only a single section per node (e.g. 16 MiB with 64 KB pages) initialized
> > up-front, and on kernels with large static binary footprints (debug
> > configurations enabling PAGE_OWNER, DEBUG_PAGEALLOC, KFENCE, or SLUB_DEBUG),
> > early core initialisations (SLUB caches, vmalloc, static ftrace records)
> > consume virtually all managed pages in this pool.
> >
> > si_mem_available() therefore returns a false negative (often 0-1 pages)
> > across both SYSTEM_BOOTING (for the initial CPU 0 buffers) and
> > SYSTEM_SCHEDULING (for secondary CPU buffers via trace_rb_cpu_prepare()),
> > causing __rb_allocate_pages() to prematurely abort with -ENOMEM.
> >
> > This failure is false: the buddy allocator is designed to replenish
> > ZONE_NORMAL on demand via deferred_grow_zone() when an actual allocation
> > is attempted. Checking si_mem_available() before calling alloc_pages_node()
> > short-circuits this on-demand growth entirely.
> >
> > deferred_grow_zone() is only permanently disabled after page_alloc_init_late()
> > completes (it spawns per-node kthreads and blocks until done), which happens
> > inside kernel_init_freeable() — before system_state is set to
> > SYSTEM_FREEING_INITMEM. Skip the si_mem_available() check until
> > SYSTEM_FREEING_INITMEM, at which point all deferred pages are initialized
> > and si_mem_available() is accurate.
> >
> > Use IS_ENABLED(CONFIG_DEFERRED_STRUCT_PAGE_INIT) to set i = LONG_MAX during
> > this window, unconditionally bypassing the threshold check. On kernels
> > without deferred init IS_ENABLED() compiles the branch away and the original
> > check is preserved unchanged.
>
> As Vincent mentioned, the above is just word salad. Please make it human
> readable. That is, state what the actual issue is and how it's fixed and
> remove all the AI fluff.
>
> Also note, just because we no longer require an "Assisted-by" tag, it
> should still be mentioned how the patch was created. I want to know.
Hi Steven,
To answer how this was created: I investigated the issue, carried out the RCA
and debug runs on a Power11 KVM setup, and authored the patch logic. When
writing the commit message, in a (clearly failed) attempt to explain the
details thoroughly, I ended up introducing the wordiness and fluff. Later I did
use an LLM for a quick rewording and spell checking, which only made it worse I
guess.
For v5, I have completely rewritten the commit message myself to be concise
and to the point: stating the actual issue (deferred page init leaves
NR_FREE_PAGES near zero, making si_mem_available() under-report), the
spurious -ENOMEM failure during early_trace_init() and smp_init(), and the
fix mechanism.
<snip>
> > i = si_mem_available();
> > + if (IS_ENABLED(CONFIG_DEFERRED_STRUCT_PAGE_INIT) &&
> > + system_state < SYSTEM_FREEING_INITMEM)
> > + i = LONG_MAX;
>
> Make the above an if/else statement.
>
> if (IS_ENABLED(CONFIG_DEFERRED_STRUCT_PAGE_INIT) &&
> unlikely(system_state < SYSTEM_FREEING_INITMEM))
> i = LONG_MAX;
> else
> i = si_mem_available();
Sure, this makes more sense and avoids calling si_mem_available() when it is not
reliable anyway. I have incorporated this in v5.
Thanks,
Amit
^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2026-11-18 18:24 UTC | newest]
Thread overview: 5+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2026-10-23 14:54 [PATCH v4] ring-buffer: Do not check si_mem_available() before deferred init completes Amit Machhiwal
2026-10-26 16:51 ` Vincent Donnefort
2026-11-17 18:45 ` Amit Machhiwal
2026-11-17 21:28 ` Steven Rostedt
2026-11-18 18:32 ` Amit Machhiwal
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®