From: Peter Zijlstra <peterz@infradead.org>
To: Alexei Starovoitov <alexei.starovoitov@gmail.com>
Cc: x86@kernel.org, joao@overdrivepizza.com, hjl.tools@gmail.com,
jpoimboe@redhat.com, andrew.cooper3@citrix.com,
linux-kernel@vger.kernel.org, ndesaulniers@google.com,
keescook@chromium.org, samitolvanen@google.com,
mark.rutland@arm.com, alyssa.milburn@intel.com, mbenes@suse.cz,
rostedt@goodmis.org, mhiramat@kernel.org, daniel@iogearbox.net,
andrii@kernel.org, bpf@vger.kernel.org
Subject: Re: [PATCH v4 00/45] x86: Kernel IBT
Date: Thu, 10 Mar 2022 10:35:32 +0100 [thread overview]
Message-ID: <YinGZObp37b27LjK@hirez.programming.kicks-ass.net> (raw)
In-Reply-To: <20220309190917.w3tq72alughslanq@ast-mbp.dhcp.thefacebook.com>
On Wed, Mar 09, 2022 at 11:09:17AM -0800, Alexei Starovoitov wrote:
> Pulled above and it got even worse.
> With kasan and lockdep during qemu boot I see:
> [ 1.147498] rcu_scheduler_active = 1, debug_locks = 1
> [ 1.147498] 2 locks held by kthreadd/2:
> [ 1.147498] #0: ffff888100362b80 (&p->pi_lock){....}-{2:2}, at: task_rq_lock+0x71/0x380
> [ 1.147498] #1: ffff8881f6a3a218 (&rq->__lock){-...}-{2:2}, at: raw_spin_rq_lock_nested+0x2b/0x40
> [ 1.147498]
> [ 1.147498] stack backtrace:
> [ 1.147498] CPU: 0 PID: 2 Comm: kthreadd Not tainted 5.17.0-rc7-02289-gc958c6aae879 #1
> [ 1.147498] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.12.0-59-gc9ba5276e321-prebuilt.qemu.org 04/01/2014
> [ 1.147498] Call Trace:
> [ 1.147498] <TASK>
> [ 1.147498] dump_stack_lvl+0x48/0x5b
> [ 1.147498] cpuacct_charge+0x2b3/0x390
> [ 1.147498] update_curr+0x33e/0x7d0
> [ 1.147498] dequeue_entity+0x28/0xdf0
> [ 1.147498] ? rcu_read_lock_bh_held+0xa0/0xa0
> [ 1.147498] dequeue_task_fair+0x1fa/0xd60
> [ 1.147498] __do_set_cpus_allowed+0x253/0x620
> [ 1.147498] __set_cpus_allowed_ptr_locked+0x25f/0x450
> [ 1.147498] __set_cpus_allowed_ptr+0x7c/0xa0
> [ 1.147498] ? __set_cpus_allowed_ptr_locked+0x450/0x450
> [ 1.147498] ? _raw_spin_unlock_irqrestore+0x34/0x60
> [ 1.147498] ? lockdep_hardirqs_on+0x7d/0x100
> [ 1.147498] kthreadd+0x48/0x610
> [ 1.147498] ? _raw_spin_unlock_irq+0x28/0x50
> [ 1.147498] ? kthread_is_per_cpu+0xc0/0xc0
> [ 1.147498] ret_from_fork+0x1f/0x30
Yeah, sorry about that, currently arguing with Paul about that one.
Should go away if you disable the RCU lockdep thing. The warning itself
is a false positive and more harmful than anything else due to it
generating a possible printk deadlock.
> Most of the time it hangs during the boot.
> I'm using gcc 8.5 and qemu -smp 8
> With qemu -smp 1 it luckly boots.
> Then I run test_progs and see:
> Summary: 215/1115 PASSED, 4 SKIPPED, 18 FAILED
> All trampoline tests fail.
> Here is one:
> $ test_progs -t fentry
> test_fentry_fexit:PASS:fentry_skel_load 0 nsec
> test_fentry_fexit:PASS:fexit_skel_load 0 nsec
> test_fentry_fexit:PASS:fentry_attach 0 nsec
> test_fentry_fexit:FAIL:fexit_attach unexpected error: -1 (errno 19)
> #54 fentry_fexit:FAIL
>
> or
>
> ./test_progs -t xdp_bpf
> test_xdp_bpf2bpf:PASS:test_xdp__open_and_load 0 nsec
> test_xdp_bpf2bpf:PASS:test_xdp_bpf2bpf__open 0 nsec
> test_xdp_bpf2bpf:PASS:test_xdp_bpf2bpf__load 0 nsec
> libbpf: prog 'trace_on_entry': failed to attach: Device or resource busy
> libbpf: prog 'trace_on_entry': failed to auto-attach: -16
> test_xdp_bpf2bpf:FAIL:test_xdp_bpf2bpf__attach unexpected error: -16 (errno 16)
> #225 xdp_bpf2bpf:FAIL
Urgh.. I totally missed that in flood of output. Let me go try and
figure out what's happening there.
next prev parent reply other threads:[~2022-03-10 9:36 UTC|newest]
Thread overview: 193+ messages / expand[flat|nested] mbox.gz Atom feed top
2022-03-08 15:30 Peter Zijlstra
2022-03-08 15:30 ` [PATCH v4 01/45] static_call: Avoid building empty .static_call_sites Peter Zijlstra
2022-03-09 7:55 ` [tip: x86/core] " tip-bot2 for Peter Zijlstra
2022-03-15 10:44 ` tip-bot2 for Peter Zijlstra
2022-03-08 15:30 ` [PATCH v4 02/45] objtool: Add --dry-run Peter Zijlstra
2022-03-09 7:55 ` [tip: x86/core] " tip-bot2 for Peter Zijlstra
2022-03-15 10:44 ` tip-bot2 for Peter Zijlstra
2022-03-08 15:30 ` [PATCH v4 03/45] objtool: Default ignore INT3 for unreachable Peter Zijlstra
2022-03-09 7:55 ` [tip: x86/core] " tip-bot2 for Peter Zijlstra
2022-03-15 10:44 ` tip-bot2 for Peter Zijlstra
2022-03-08 15:30 ` [PATCH v4 04/45] objtool,efi: Update __efi64_thunk annotation Peter Zijlstra
2022-03-09 7:55 ` [tip: x86/core] " tip-bot2 for Peter Zijlstra
2022-03-09 8:35 ` [PATCH v4 04/45] " Miroslav Benes
2022-03-15 10:44 ` [tip: x86/core] " tip-bot2 for Peter Zijlstra
2022-03-08 15:30 ` [PATCH v4 05/45] objtool: Have WARN_FUNC fall back to sym+off Peter Zijlstra
2022-03-09 7:55 ` [tip: x86/core] " tip-bot2 for Peter Zijlstra
2022-03-15 10:44 ` tip-bot2 for Peter Zijlstra
2022-03-08 15:30 ` [PATCH v4 06/45] x86/ibt: Base IBT bits Peter Zijlstra
2022-03-09 7:55 ` [tip: x86/core] " tip-bot2 for Peter Zijlstra
2022-03-15 10:44 ` tip-bot2 for Peter Zijlstra
2022-03-08 15:30 ` [PATCH v4 07/45] x86/ibt: Add ANNOTATE_NOENDBR Peter Zijlstra
2022-03-09 7:55 ` [tip: x86/core] " tip-bot2 for Peter Zijlstra
2022-03-15 10:44 ` tip-bot2 for Peter Zijlstra
2022-03-08 15:30 ` [PATCH v4 08/45] x86/text-patching: Make text_gen_insn() play nice with ANNOTATE_NOENDBR Peter Zijlstra
2022-03-09 7:55 ` [tip: x86/core] " tip-bot2 for Peter Zijlstra
2022-03-15 10:44 ` tip-bot2 for Peter Zijlstra
2022-03-08 15:30 ` [PATCH v4 09/45] x86/ibt,paravirt: Use text_gen_insn() for paravirt_patch() Peter Zijlstra
2022-03-09 7:55 ` [tip: x86/core] " tip-bot2 for Peter Zijlstra
2022-03-15 10:44 ` tip-bot2 for Peter Zijlstra
2022-03-08 15:30 ` [PATCH v4 10/45] x86/entry: Cleanup PARAVIRT Peter Zijlstra
2022-03-09 7:55 ` [tip: x86/core] " tip-bot2 for Peter Zijlstra
2022-03-15 10:44 ` tip-bot2 for Peter Zijlstra
2022-03-08 15:30 ` [PATCH v4 11/45] x86/entry,xen: Early rewrite of restore_regs_and_return_to_kernel() Peter Zijlstra
2022-03-09 7:55 ` [tip: x86/core] " tip-bot2 for Peter Zijlstra
2022-03-15 10:44 ` tip-bot2 for Peter Zijlstra
2022-03-08 15:30 ` [PATCH v4 12/45] x86/ibt,xen: Sprinkle the ENDBR Peter Zijlstra
2022-03-09 7:55 ` [tip: x86/core] " tip-bot2 for Peter Zijlstra
2022-03-15 10:44 ` tip-bot2 for Peter Zijlstra
2022-03-08 15:30 ` [PATCH v4 13/45] x86/ibt,entry: Sprinkle ENDBR dust Peter Zijlstra
2022-03-09 7:55 ` [tip: x86/core] " tip-bot2 for Peter Zijlstra
2022-03-15 10:44 ` tip-bot2 for Peter Zijlstra
2022-03-08 15:30 ` [PATCH v4 14/45] x86/linkage: Add ENDBR to SYM_FUNC_START*() Peter Zijlstra
2022-03-09 7:55 ` [tip: x86/core] " tip-bot2 for Peter Zijlstra
2022-03-15 10:44 ` tip-bot2 for Peter Zijlstra
2022-03-08 15:30 ` [PATCH v4 15/45] x86/ibt,paravirt: Sprinkle ENDBR Peter Zijlstra
2022-03-09 7:55 ` [tip: x86/core] " tip-bot2 for Peter Zijlstra
2022-03-15 10:44 ` tip-bot2 for Peter Zijlstra
2022-03-08 15:30 ` [PATCH v4 16/45] x86/ibt,crypto: Add ENDBR for the jump-table entries Peter Zijlstra
2022-03-09 7:55 ` [tip: x86/core] " tip-bot2 for Peter Zijlstra
2022-03-15 10:44 ` tip-bot2 for Peter Zijlstra
2022-03-08 15:30 ` [PATCH v4 17/45] x86/ibt,kvm: Add ENDBR to fastops Peter Zijlstra
2022-03-09 7:55 ` [tip: x86/core] " tip-bot2 for Peter Zijlstra
2022-03-15 10:44 ` tip-bot2 for Peter Zijlstra
2022-03-08 15:30 ` [PATCH v4 18/45] x86/ibt,ftrace: Search for __fentry__ location Peter Zijlstra
2022-03-09 7:55 ` [tip: x86/core] " tip-bot2 for Peter Zijlstra
2022-03-15 10:44 ` tip-bot2 for Peter Zijlstra
2022-03-08 15:30 ` [PATCH v4 19/45] x86/livepatch: Validate " Peter Zijlstra
2022-03-09 7:55 ` [tip: x86/core] " tip-bot2 for Peter Zijlstra
2022-03-09 8:58 ` [PATCH v4 19/45] " Miroslav Benes
2022-03-15 10:44 ` [tip: x86/core] " tip-bot2 for Peter Zijlstra
2022-03-08 15:30 ` [PATCH v4 20/45] x86/ibt,ftrace: Make function-graph play nice Peter Zijlstra
2022-03-09 7:55 ` [tip: x86/core] " tip-bot2 for Peter Zijlstra
2022-03-15 10:44 ` tip-bot2 for Peter Zijlstra
2022-03-08 15:30 ` [PATCH v4 21/45] x86/ibt,kprobes: Cure sym+0 equals fentry woes Peter Zijlstra
2022-03-09 7:55 ` [tip: x86/core] " tip-bot2 for Peter Zijlstra
2022-03-15 10:44 ` tip-bot2 for Peter Zijlstra
2022-03-08 15:30 ` [PATCH v4 22/45] x86/ibt,bpf: Add ENDBR instructions to prologue and trampoline Peter Zijlstra
2022-03-09 7:54 ` [tip: x86/core] " tip-bot2 for Peter Zijlstra
2022-03-15 10:44 ` tip-bot2 for Peter Zijlstra
2022-03-08 15:30 ` [PATCH v4 23/45] x86/ibt,ftrace: Add ENDBR to samples/ftrace Peter Zijlstra
2022-03-09 7:54 ` [tip: x86/core] " tip-bot2 for Peter Zijlstra
2022-03-15 10:43 ` tip-bot2 for Peter Zijlstra
2022-03-08 15:30 ` [PATCH v4 24/45] x86/ibt: Add IBT feature, MSR and #CP handling Peter Zijlstra
2022-03-09 7:54 ` [tip: x86/core] " tip-bot2 for Peter Zijlstra
2022-03-09 13:56 ` [PATCH v4 24/45] " Andrew Cooper
2022-03-15 10:43 ` [tip: x86/core] " tip-bot2 for Peter Zijlstra
2022-03-08 15:30 ` [PATCH v4 25/45] x86/ibt,kexec: Disable CET on kexec Peter Zijlstra
2022-03-09 7:54 ` [tip: x86/core] " tip-bot2 for Peter Zijlstra
2022-03-15 10:43 ` tip-bot2 for Peter Zijlstra
2022-03-08 15:30 ` [PATCH v4 26/45] x86/alternative: Simplify int3_selftest_ip Peter Zijlstra
2022-03-09 7:54 ` [tip: x86/core] " tip-bot2 for Peter Zijlstra
2022-03-15 10:43 ` tip-bot2 for Peter Zijlstra
2022-03-08 15:30 ` [PATCH v4 27/45] x86/ibt: Disable IBT around firmware Peter Zijlstra
2022-03-09 7:54 ` [tip: x86/core] " tip-bot2 for Peter Zijlstra
2022-03-15 10:43 ` tip-bot2 for Peter Zijlstra
2022-03-08 15:30 ` [PATCH v4 28/45] x86/bugs: Disable Retpoline when IBT Peter Zijlstra
2022-03-08 15:30 ` [PATCH v4 29/45] x86/ibt: Annotate text references Peter Zijlstra
2022-03-09 7:54 ` [tip: x86/core] " tip-bot2 for Peter Zijlstra
2022-03-15 10:43 ` tip-bot2 for Peter Zijlstra
2022-03-22 4:43 ` [PATCH v4 29/45] " Masami Hiramatsu
2022-03-22 4:45 ` Alexei Starovoitov
2022-03-08 15:30 ` [PATCH v4 30/45] x86/ibt,ftrace: Annotate ftrace code patching Peter Zijlstra
2022-03-09 7:54 ` [tip: x86/core] " tip-bot2 for Peter Zijlstra
2022-03-15 10:43 ` tip-bot2 for Peter Zijlstra
2022-03-08 15:30 ` [PATCH v4 31/45] x86/ibt,sev: Annotations Peter Zijlstra
2022-03-09 7:54 ` [tip: x86/core] " tip-bot2 for Peter Zijlstra
2022-03-15 10:43 ` tip-bot2 for Peter Zijlstra
2022-03-08 15:30 ` [PATCH v4 32/45] x86/ibt: Dont generate ENDBR in .discard.text Peter Zijlstra
2022-03-09 7:54 ` [tip: x86/core] " tip-bot2 for Peter Zijlstra
2022-03-15 10:43 ` tip-bot2 for Peter Zijlstra
2022-03-08 15:30 ` [PATCH v4 33/45] x86/ibt: Ensure module init/exit points have references Peter Zijlstra
2022-03-09 7:54 ` [tip: x86/core] " tip-bot2 for Peter Zijlstra
2022-03-15 10:43 ` tip-bot2 for Peter Zijlstra
2022-03-08 15:30 ` [PATCH v4 34/45] objtool: Rename --duplicate to --lto Peter Zijlstra
2022-03-09 7:54 ` [tip: x86/core] " tip-bot2 for Peter Zijlstra
2022-03-15 10:43 ` tip-bot2 for Peter Zijlstra
2022-03-08 15:30 ` [PATCH v4 35/45] objtool: Ignore extra-symbol code Peter Zijlstra
2022-03-09 7:54 ` [tip: x86/core] " tip-bot2 for Peter Zijlstra
2022-03-15 10:43 ` tip-bot2 for Peter Zijlstra
2022-03-08 15:30 ` [PATCH v4 36/45] x86: Mark stop_this_cpu() __noreturn Peter Zijlstra
2022-03-09 7:54 ` [tip: x86/core] " tip-bot2 for Peter Zijlstra
2022-03-15 10:43 ` tip-bot2 for Peter Zijlstra
2022-03-08 15:30 ` [PATCH v4 37/45] exit: Mark do_group_exit() __noreturn Peter Zijlstra
2022-03-09 7:54 ` [tip: x86/core] " tip-bot2 for Peter Zijlstra
2022-03-15 10:43 ` tip-bot2 for Peter Zijlstra
2022-03-08 15:30 ` [PATCH v4 38/45] objtool: Rework ASM_REACHABLE Peter Zijlstra
2022-03-09 7:54 ` [tip: x86/core] " tip-bot2 for Peter Zijlstra
2022-03-15 10:43 ` tip-bot2 for Peter Zijlstra
2022-03-08 15:30 ` [PATCH v4 39/45] x86: Annotate call_on_stack() Peter Zijlstra
2022-03-09 7:54 ` [tip: x86/core] " tip-bot2 for Peter Zijlstra
2022-03-15 10:43 ` tip-bot2 for Peter Zijlstra
2022-03-08 15:30 ` [PATCH v4 40/45] Kbuild: Allow whole module objtool runs Peter Zijlstra
2022-03-09 7:54 ` [tip: x86/core] " tip-bot2 for Peter Zijlstra
2022-03-08 15:30 ` [PATCH v4 41/45] objtool: Read the NOENDBR annotation Peter Zijlstra
2022-03-09 7:54 ` [tip: x86/core] " tip-bot2 for Peter Zijlstra
2022-03-15 10:43 ` tip-bot2 for Peter Zijlstra
2022-03-08 15:30 ` [PATCH v4 42/45] objtool: Add IBT/ENDBR decoding Peter Zijlstra
2022-03-09 7:54 ` [tip: x86/core] " tip-bot2 for Peter Zijlstra
2022-03-15 10:43 ` tip-bot2 for Peter Zijlstra
2022-03-08 15:30 ` [PATCH v4 43/45] objtool: Validate IBT assumptions Peter Zijlstra
2022-03-09 7:54 ` [tip: x86/core] " tip-bot2 for Peter Zijlstra
2022-03-15 10:43 ` tip-bot2 for Peter Zijlstra
2022-03-08 15:30 ` [PATCH v4 44/45] objtool: Find unused ENDBR instructions Peter Zijlstra
2022-03-09 7:54 ` [tip: x86/core] " tip-bot2 for Peter Zijlstra
2022-03-15 10:43 ` tip-bot2 for Peter Zijlstra
2022-03-15 15:39 ` David Laight
2022-03-17 22:22 ` Josh Poimboeuf
2022-03-18 10:49 ` Peter Zijlstra
2022-03-08 15:30 ` [PATCH v4 45/45] x86/alternative: Use .ibt_endbr_seal to seal indirect calls Peter Zijlstra
2022-03-09 7:54 ` [tip: x86/core] " tip-bot2 for Peter Zijlstra
2022-03-15 10:43 ` tip-bot2 for Peter Zijlstra
2022-03-08 20:00 ` [PATCH v4 00/45] x86: Kernel IBT Alexei Starovoitov
2022-03-08 22:01 ` Peter Zijlstra
2022-03-08 22:32 ` Peter Zijlstra
2022-03-09 1:02 ` Peter Zijlstra
2022-03-09 19:09 ` Alexei Starovoitov
2022-03-10 9:35 ` Peter Zijlstra [this message]
2022-03-10 13:47 ` Peter Zijlstra
2022-03-10 14:37 ` Steven Rostedt
2022-03-11 15:23 ` Peter Zijlstra
2022-03-10 16:29 ` Peter Zijlstra
2022-03-11 10:40 ` Peter Zijlstra
2022-03-11 17:09 ` Alexei Starovoitov
2022-03-12 15:44 ` Peter Zijlstra
2022-03-13 1:33 ` Alexei Starovoitov
2022-03-13 8:52 ` Peter Zijlstra
2022-03-14 14:59 ` Peter Zijlstra
2022-03-15 8:15 ` Peter Zijlstra
2022-03-15 16:28 ` Masahiro Yamada
2022-03-17 19:44 ` Peter Zijlstra
2022-03-18 2:07 ` David Laight
2022-03-17 18:15 ` Masahiro Yamada
2022-03-17 19:52 ` Peter Zijlstra
2022-03-18 15:36 ` [tip: x86/core] kbuild: Fixup the IBT kbuild changes tip-bot2 for Peter Zijlstra
2022-03-18 16:15 ` Masahiro Yamada
2022-03-22 20:24 ` tip-bot2 for Peter Zijlstra
2022-03-15 16:26 ` [PATCH v4 00/45] x86: Kernel IBT Masahiro Yamada
2022-03-17 19:36 ` Peter Zijlstra
2022-03-14 15:33 ` Peter Zijlstra
2022-03-15 10:43 ` [tip: x86/core] x86: Annotate idtentry_df() tip-bot2 for Peter Zijlstra
2022-03-15 10:43 ` [tip: x86/core] x86,objtool: Move the ASM_REACHABLE annotation to objtool.h tip-bot2 for Peter Zijlstra
2022-03-15 10:43 ` [tip: x86/core] x86: Mark __invalid_creds() __noreturn tip-bot2 for Peter Zijlstra
2022-03-14 20:44 ` [PATCH v4 00/45] x86: Kernel IBT Kumar Kartikeya Dwivedi
2022-03-15 9:00 ` Peter Zijlstra
2022-03-15 10:05 ` Kumar Kartikeya Dwivedi
2022-03-15 10:07 ` Peter Zijlstra
2022-03-15 10:39 ` Peter Zijlstra
2022-03-16 9:35 ` Peter Zijlstra
2022-03-16 11:12 ` Kumar Kartikeya Dwivedi
2022-03-15 18:26 ` Alexei Starovoitov
2022-03-17 20:27 ` Peter Zijlstra
2022-03-11 14:40 ` [tip: x86/core] x86,ftrace: Fix modify_ftrace_direct() tip-bot2 for Peter Zijlstra
2022-03-10 0:30 ` [PATCH v4 00/45] x86: Kernel IBT Nick Desaulniers
2022-03-10 9:05 ` Peter Zijlstra
2022-03-10 9:22 ` David Laight
2022-03-10 10:16 ` Peter Zijlstra
2022-03-10 20:49 ` Nick Desaulniers
2022-03-11 14:40 ` [tip: x86/core] x86: Fix {int3,ibt}_selftest() vs LTO tip-bot2 for Peter Zijlstra
2022-03-08 20:06 ` [PATCH v4 00/45] x86: Kernel IBT Josh Poimboeuf
2022-03-09 6:57 ` Josh Poimboeuf
2022-03-09 13:37 ` David Laight
2022-03-10 14:27 ` Peter Zijlstra
2022-03-09 12:51 ` Miroslav Benes
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=YinGZObp37b27LjK@hirez.programming.kicks-ass.net \
--to=peterz@infradead.org \
--cc=alexei.starovoitov@gmail.com \
--cc=alyssa.milburn@intel.com \
--cc=andrew.cooper3@citrix.com \
--cc=andrii@kernel.org \
--cc=bpf@vger.kernel.org \
--cc=daniel@iogearbox.net \
--cc=hjl.tools@gmail.com \
--cc=joao@overdrivepizza.com \
--cc=jpoimboe@redhat.com \
--cc=keescook@chromium.org \
--cc=linux-kernel@vger.kernel.org \
--cc=mark.rutland@arm.com \
--cc=mbenes@suse.cz \
--cc=mhiramat@kernel.org \
--cc=ndesaulniers@google.com \
--cc=rostedt@goodmis.org \
--cc=samitolvanen@google.com \
--cc=x86@kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®