From: Chris Metcalf <cmetcalf@ezchip.com>
To: Gilad Ben Yossef <giladb@ezchip.com>,
Steven Rostedt <rostedt@goodmis.org>,
Ingo Molnar <mingo@redhat.com>,
Peter Zijlstra <peterz@infradead.org>,
Andrew Morton <akpm@linux-foundation.org>,
"Rik van Riel" <riel@redhat.com>, Tejun Heo <tj@kernel.org>,
Thomas Gleixner <tglx@linutronix.de>,
Frederic Weisbecker <fweisbec@gmail.com>,
"Paul E. McKenney" <paulmck@linux.vnet.ibm.com>,
Christoph Lameter <cl@linux.com>,
"Srivatsa S. Bhat" <srivatsa.bhat@linux.vnet.ibm.com>,
<linux-doc@vger.kernel.org>, <linux-api@vger.kernel.org>,
<linux-kernel@vger.kernel.org>
Cc: Chris Metcalf <cmetcalf@ezchip.com>
Subject: [PATCH 5/6] nohz: support PR_DATAPLANE_STRICT mode
Date: Fri, 8 May 2015 13:58:46 -0400 [thread overview]
Message-ID: <1431107927-13998-6-git-send-email-cmetcalf@ezchip.com> (raw)
In-Reply-To: <1431107927-13998-1-git-send-email-cmetcalf@ezchip.com>
With QUIESCE mode, the task is in principle guaranteed not to be
interrupted by the kernel, but only if it behaves. In particular,
if it enters the kernel via system call, page fault, or any of
a number of other synchronous traps, it may be unexpectedly
exposed to long latencies. Add a simple flag that puts the process
into a state where any such kernel entry is fatal.
To allow the state to be entered and exited, we add an internal
bit to current->dataplane_flags that is set when prctl() sets the
flags. That way, when we are exiting the kernel after calling
prctl() to forbid future kernel exits, we don't get immediately
killed.
Signed-off-by: Chris Metcalf <cmetcalf@ezchip.com>
---
include/uapi/linux/prctl.h | 2 ++
kernel/sys.c | 2 +-
kernel/time/tick-sched.c | 17 +++++++++++++++++
3 files changed, 20 insertions(+), 1 deletion(-)
diff --git a/include/uapi/linux/prctl.h b/include/uapi/linux/prctl.h
index 8b735651304a..9cf79aa1e73f 100644
--- a/include/uapi/linux/prctl.h
+++ b/include/uapi/linux/prctl.h
@@ -195,5 +195,7 @@ struct prctl_mm_map {
#define PR_GET_DATAPLANE 48
# define PR_DATAPLANE_ENABLE (1 << 0)
# define PR_DATAPLANE_QUIESCE (1 << 1)
+# define PR_DATAPLANE_STRICT (1 << 2)
+# define PR_DATAPLANE_PRCTL (1U << 31) /* kernel internal */
#endif /* _LINUX_PRCTL_H */
diff --git a/kernel/sys.c b/kernel/sys.c
index 930b750aefde..8102433c9edd 100644
--- a/kernel/sys.c
+++ b/kernel/sys.c
@@ -2245,7 +2245,7 @@ SYSCALL_DEFINE5(prctl, int, option, unsigned long, arg2, unsigned long, arg3,
break;
#ifdef CONFIG_NO_HZ_FULL
case PR_SET_DATAPLANE:
- me->dataplane_flags = arg2;
+ me->dataplane_flags = arg2 | PR_DATAPLANE_PRCTL;
break;
case PR_GET_DATAPLANE:
error = me->dataplane_flags;
diff --git a/kernel/time/tick-sched.c b/kernel/time/tick-sched.c
index 69d908c6cef8..22ed0decb363 100644
--- a/kernel/time/tick-sched.c
+++ b/kernel/time/tick-sched.c
@@ -436,6 +436,20 @@ static void dataplane_quiesce(void)
(jiffies - start));
dump_stack();
}
+
+ /*
+ * Kill the process if it violates STRICT mode. Note that this
+ * code also results in killing the task if a kernel bug causes an
+ * irq to be delivered to this core.
+ */
+ if ((task->dataplane_flags & (PR_DATAPLANE_STRICT|PR_DATAPLANE_PRCTL))
+ == PR_DATAPLANE_STRICT) {
+ pr_warn("Dataplane STRICT mode violated; process killed.\n");
+ dump_stack();
+ task->dataplane_flags &= ~PR_DATAPLANE_QUIESCE;
+ local_irq_enable();
+ do_group_exit(SIGKILL);
+ }
}
/*
@@ -464,6 +478,9 @@ void tick_nohz_dataplane_enter(void)
if ((current->dataplane_flags & PR_DATAPLANE_QUIESCE) != 0)
dataplane_quiesce();
+ /* Clear the bit set by prctl() when it updates the flags. */
+ current->dataplane_flags &= ~PR_DATAPLANE_PRCTL;
+
/*
* Disable interrupts again since other code running in this
* function may have enabled them, and the caller expects
--
2.1.2
next prev parent reply other threads:[~2015-05-08 17:59 UTC|newest]
Thread overview: 236+ messages / expand[flat|nested] mbox.gz Atom feed top
2015-05-08 17:58 [PATCH 0/6] support "dataplane" mode for nohz_full Chris Metcalf
2015-05-08 17:58 ` [PATCH 1/6] nohz_full: add support for "dataplane" mode Chris Metcalf
2015-05-08 17:58 ` [PATCH 2/6] nohz: dataplane: allow tick to be fully disabled for dataplane Chris Metcalf
2015-05-12 9:26 ` Peter Zijlstra
2015-05-12 13:12 ` Paul E. McKenney
2015-05-14 20:55 ` Chris Metcalf
2015-05-08 17:58 ` [PATCH 3/6] dataplane nohz: run softirqs synchronously on user entry Chris Metcalf
2015-05-09 7:04 ` Mike Galbraith
2015-05-11 20:13 ` Chris Metcalf
2015-05-12 2:21 ` Mike Galbraith
2015-05-12 9:28 ` Peter Zijlstra
2015-05-12 9:32 ` Peter Zijlstra
2015-05-12 13:08 ` Paul E. McKenney
2015-05-08 17:58 ` [PATCH 4/6] nohz: support PR_DATAPLANE_QUIESCE Chris Metcalf
2015-05-12 9:33 ` Peter Zijlstra
2015-05-12 9:50 ` Ingo Molnar
2015-05-12 10:38 ` Peter Zijlstra
2015-05-12 12:52 ` Ingo Molnar
2015-05-13 4:35 ` Andy Lutomirski
2015-05-13 17:51 ` Paul E. McKenney
2015-05-14 20:55 ` Chris Metcalf
2015-05-14 20:54 ` Chris Metcalf
2015-05-08 17:58 ` Chris Metcalf [this message]
2015-05-09 7:28 ` [PATCH 5/6] nohz: support PR_DATAPLANE_STRICT mode Andy Lutomirski
2015-05-09 10:37 ` Gilad Ben Yossef
2015-05-11 19:13 ` Chris Metcalf
2015-05-11 22:28 ` Andy Lutomirski
2015-05-12 21:06 ` Chris Metcalf
2015-05-12 22:23 ` Andy Lutomirski
2015-05-15 21:25 ` Chris Metcalf
2015-05-12 9:38 ` Peter Zijlstra
2015-05-12 13:20 ` Paul E. McKenney
2015-05-08 17:58 ` [PATCH 6/6] nohz: add dataplane_debug boot flag Chris Metcalf
2015-05-08 21:18 ` [PATCH 0/6] support "dataplane" mode for nohz_full Andrew Morton
2015-05-08 21:22 ` Steven Rostedt
2015-05-08 23:11 ` Chris Metcalf
2015-05-08 23:19 ` Andrew Morton
2015-05-09 7:05 ` Ingo Molnar
2015-05-09 7:19 ` Andy Lutomirski
2015-05-11 19:54 ` Chris Metcalf
2015-05-11 22:15 ` Andy Lutomirski
[not found] ` <55510885.9070101@ezchip.com>
2015-05-12 13:18 ` Paul E. McKenney
2015-05-09 7:19 ` Mike Galbraith
2015-05-09 10:18 ` Gilad Ben Yossef
2015-05-11 12:57 ` Steven Rostedt
2015-05-11 15:36 ` Frederic Weisbecker
2015-05-11 19:19 ` Mike Galbraith
2015-05-11 19:25 ` Chris Metcalf
2015-05-12 1:47 ` Mike Galbraith
2015-05-12 4:35 ` Mike Galbraith
2015-05-15 15:05 ` Chris Metcalf
2015-05-15 18:44 ` Mike Galbraith
2015-05-26 19:51 ` Chris Metcalf
2015-05-27 3:28 ` Mike Galbraith
2015-05-11 17:19 ` Paul E. McKenney
2015-05-11 17:27 ` Andrew Morton
2015-05-11 17:33 ` Frederic Weisbecker
2015-05-11 18:00 ` Steven Rostedt
2015-05-11 18:09 ` Chris Metcalf
2015-05-11 18:36 ` Steven Rostedt
2015-05-12 9:10 ` CONFIG_ISOLATION=y (was: [PATCH 0/6] support "dataplane" mode for nohz_full) Ingo Molnar
2015-05-12 11:48 ` Peter Zijlstra
2015-05-12 12:34 ` Ingo Molnar
2015-05-12 12:39 ` Peter Zijlstra
2015-05-12 12:43 ` Ingo Molnar
2015-05-12 15:36 ` Frederic Weisbecker
2015-05-12 21:05 ` CONFIG_ISOLATION=y Chris Metcalf
2015-05-12 10:46 ` [PATCH 0/6] support "dataplane" mode for nohz_full Peter Zijlstra
2015-05-15 15:10 ` Chris Metcalf
2015-05-15 21:26 ` [PATCH v2 0/5] support "cpu_isolated" " Chris Metcalf
2015-05-15 21:27 ` [PATCH v2 1/5] nohz_full: add support for "cpu_isolated" mode Chris Metcalf
2015-05-15 21:27 ` [PATCH v2 2/5] nohz: support PR_CPU_ISOLATED_STRICT mode Chris Metcalf
2015-05-15 21:27 ` [PATCH v2 3/5] nohz: cpu_isolated strict mode configurable signal Chris Metcalf
2015-05-15 21:27 ` [PATCH v2 4/5] nohz: add cpu_isolated_debug boot flag Chris Metcalf
2015-05-15 21:27 ` [PATCH v2 5/5] nohz: cpu_isolated: allow tick to be fully disabled Chris Metcalf
2015-05-15 22:17 ` [PATCH v2 1/5] nohz_full: add support for "cpu_isolated" mode Thomas Gleixner
2015-05-28 20:38 ` Chris Metcalf
2015-06-03 15:29 ` [PATCH v3 0/5] support "cpu_isolated" mode for nohz_full Chris Metcalf
2015-06-03 15:29 ` [PATCH v3 1/5] nohz_full: add support for "cpu_isolated" mode Chris Metcalf
2015-06-03 15:29 ` [PATCH v3 2/5] nohz: support PR_CPU_ISOLATED_STRICT mode Chris Metcalf
2015-06-03 15:29 ` [PATCH v3 3/5] nohz: cpu_isolated strict mode configurable signal Chris Metcalf
2015-06-03 15:29 ` [PATCH v3 4/5] nohz: add cpu_isolated_debug boot flag Chris Metcalf
2015-06-03 15:29 ` [PATCH v3 5/5] nohz: cpu_isolated: allow tick to be fully disabled Chris Metcalf
2015-07-13 19:57 ` [PATCH v4 0/5] support "cpu_isolated" mode for nohz_full Chris Metcalf
2015-07-13 19:57 ` [PATCH v4 1/5] nohz_full: add support for "cpu_isolated" mode Chris Metcalf
2015-07-13 20:40 ` Andy Lutomirski
2015-07-13 21:01 ` Chris Metcalf
2015-07-13 21:45 ` Andy Lutomirski
2015-07-21 19:10 ` Chris Metcalf
2015-07-21 19:26 ` Andy Lutomirski
2015-07-21 20:36 ` Paul E. McKenney
2015-07-22 13:57 ` Christoph Lameter
2015-07-22 19:28 ` Paul E. McKenney
2015-07-22 20:02 ` Christoph Lameter
2015-07-24 20:21 ` Chris Metcalf
2015-07-24 20:22 ` Chris Metcalf
2015-07-24 14:03 ` Frederic Weisbecker
2015-07-24 20:19 ` Chris Metcalf
2015-07-24 13:27 ` Frederic Weisbecker
2015-07-24 20:21 ` Chris Metcalf
2015-07-13 19:57 ` [PATCH v4 2/5] nohz: support PR_CPU_ISOLATED_STRICT mode Chris Metcalf
2015-07-13 21:47 ` Andy Lutomirski
2015-07-21 19:34 ` Chris Metcalf
2015-07-21 19:42 ` Andy Lutomirski
2015-07-24 20:29 ` Chris Metcalf
2015-07-13 19:57 ` [PATCH v4 3/5] nohz: cpu_isolated strict mode configurable signal Chris Metcalf
2015-07-13 19:58 ` [PATCH v4 4/5] nohz: add cpu_isolated_debug boot flag Chris Metcalf
2015-07-13 19:58 ` [PATCH v4 5/5] nohz: cpu_isolated: allow tick to be fully disabled Chris Metcalf
2015-07-28 19:49 ` [PATCH v5 0/6] support "cpu_isolated" mode for nohz_full Chris Metcalf
2015-07-28 19:49 ` [PATCH v5 1/6] vmstat: provide a function to quiet down the diff processing Chris Metcalf
2015-07-28 19:49 ` [PATCH v5 2/6] cpu_isolated: add initial support Chris Metcalf
2015-08-12 16:00 ` Frederic Weisbecker
2015-08-12 18:22 ` Chris Metcalf
2015-08-26 15:26 ` Frederic Weisbecker
2015-08-26 15:55 ` Chris Metcalf
2015-07-28 19:49 ` [PATCH v5 3/6] cpu_isolated: support PR_CPU_ISOLATED_STRICT mode Chris Metcalf
2015-07-28 19:49 ` [PATCH v5 4/6] cpu_isolated: provide strict mode configurable signal Chris Metcalf
2015-07-28 19:49 ` [PATCH v5 5/6] cpu_isolated: add debug boot flag Chris Metcalf
2015-07-28 19:49 ` [PATCH v5 6/6] nohz: cpu_isolated: allow tick to be fully disabled Chris Metcalf
2015-08-25 19:55 ` [PATCH v6 0/6] support "task_isolated" mode for nohz_full Chris Metcalf
2015-08-25 19:55 ` [PATCH v6 1/6] vmstat: provide a function to quiet down the diff processing Chris Metcalf
2015-08-25 19:55 ` [PATCH v6 2/6] task_isolation: add initial support Chris Metcalf
2015-08-25 19:55 ` [PATCH v6 3/6] task_isolation: support PR_TASK_ISOLATION_STRICT mode Chris Metcalf
2015-08-26 10:36 ` Will Deacon
2015-08-26 15:10 ` Chris Metcalf
2015-09-02 10:13 ` Will Deacon
2015-08-28 15:31 ` [PATCH v6.1 " Chris Metcalf
2015-08-25 19:55 ` [PATCH v6 4/6] task_isolation: provide strict mode configurable signal Chris Metcalf
2015-08-28 19:22 ` Andy Lutomirski
2015-09-02 18:38 ` [PATCH v6.2 3/6] task_isolation: support PR_TASK_ISOLATION_STRICT mode Chris Metcalf
2015-08-25 19:55 ` [PATCH v6 5/6] task_isolation: add debug boot flag Chris Metcalf
2015-08-25 19:55 ` [PATCH v6 6/6] nohz: task_isolation: allow tick to be fully disabled Chris Metcalf
2015-09-28 15:17 ` [PATCH v7 00/11] support "task_isolated" mode for nohz_full Chris Metcalf
2015-09-28 15:17 ` [PATCH v7 01/11] vmstat: provide a function to quiet down the diff processing Chris Metcalf
2015-09-28 15:17 ` [PATCH v7 02/11] task_isolation: add initial support Chris Metcalf
2015-10-01 12:14 ` Frederic Weisbecker
2015-10-01 12:18 ` Thomas Gleixner
2015-10-01 12:23 ` Frederic Weisbecker
2015-10-01 12:31 ` Thomas Gleixner
2015-10-01 17:02 ` Chris Metcalf
2015-10-01 21:20 ` Thomas Gleixner
2015-10-02 17:15 ` Chris Metcalf
2015-10-02 19:02 ` Thomas Gleixner
2015-10-01 19:25 ` Chris Metcalf
2015-09-28 15:17 ` [PATCH v7 03/11] task_isolation: support PR_TASK_ISOLATION_STRICT mode Chris Metcalf
2015-09-28 20:51 ` Andy Lutomirski
2015-09-28 21:54 ` Chris Metcalf
2015-09-28 22:38 ` Andy Lutomirski
2015-09-29 17:35 ` Chris Metcalf
2015-09-29 17:46 ` Andy Lutomirski
2015-09-29 17:57 ` Chris Metcalf
2015-09-29 18:00 ` Andy Lutomirski
2015-10-01 19:25 ` Chris Metcalf
2015-09-28 15:17 ` [PATCH v7 04/11] task_isolation: provide strict mode configurable signal Chris Metcalf
2015-09-28 20:54 ` Andy Lutomirski
2015-09-28 21:54 ` Chris Metcalf
2015-09-28 15:17 ` [PATCH v7 05/11] task_isolation: add debug boot flag Chris Metcalf
2015-09-28 20:59 ` Andy Lutomirski
2015-09-28 21:55 ` Chris Metcalf
2015-09-28 22:40 ` Andy Lutomirski
2015-09-29 17:35 ` Chris Metcalf
2015-10-05 17:07 ` Luiz Capitulino
2015-10-08 0:33 ` Chris Metcalf
2015-10-08 20:28 ` Luiz Capitulino
2015-09-28 15:17 ` [PATCH v7 06/11] nohz: task_isolation: allow tick to be fully disabled Chris Metcalf
2015-09-28 20:40 ` Andy Lutomirski
2015-10-01 13:07 ` Frederic Weisbecker
2015-10-01 14:13 ` Thomas Gleixner
2015-09-28 15:17 ` [PATCH v7 07/11] arch/x86: enable task isolation functionality Chris Metcalf
2015-09-28 20:59 ` Andy Lutomirski
2015-09-28 21:57 ` Chris Metcalf
2015-09-28 22:43 ` Andy Lutomirski
2015-09-29 17:42 ` Chris Metcalf
2015-09-29 17:57 ` Andy Lutomirski
2015-09-30 20:25 ` Thomas Gleixner
2015-09-30 20:58 ` Chris Metcalf
2015-09-30 22:02 ` Thomas Gleixner
2015-09-30 22:11 ` Andy Lutomirski
2015-10-01 8:12 ` Thomas Gleixner
2015-10-01 9:08 ` Christoph Lameter
2015-10-01 10:10 ` Thomas Gleixner
2015-10-01 19:25 ` Chris Metcalf
2015-09-28 15:17 ` [PATCH v7 08/11] arch/arm64: adopt prepare_exit_to_usermode() model from x86 Chris Metcalf
2015-09-28 15:17 ` [PATCH v7 09/11] arch/arm64: enable task isolation functionality Chris Metcalf
2015-09-28 15:17 ` [PATCH v7 10/11] arch/tile: adopt prepare_exit_to_usermode() model from x86 Chris Metcalf
2015-09-28 15:17 ` [PATCH v7 11/11] arch/tile: enable task isolation functionality Chris Metcalf
2015-10-20 20:35 ` [PATCH v8 00/14] support "task_isolation" mode for nohz_full Chris Metcalf
2015-10-20 20:35 ` [PATCH v8 01/14] vmstat: provide a function to quiet down the diff processing Chris Metcalf
2015-10-20 20:36 ` [PATCH v8 02/14] vmstat: add vmstat_idle function Chris Metcalf
2015-10-20 20:45 ` Christoph Lameter
2015-10-20 20:36 ` [PATCH v8 03/14] lru_add_drain_all: factor out lru_add_drain_needed Chris Metcalf
2015-10-20 20:36 ` [PATCH v8 04/14] task_isolation: add initial support Chris Metcalf
2015-10-20 20:56 ` Andy Lutomirski
2015-10-20 21:20 ` Chris Metcalf
2015-10-20 21:26 ` Andy Lutomirski
2015-10-21 0:29 ` Steven Rostedt
2015-10-26 20:19 ` Chris Metcalf
2015-10-26 21:13 ` Chris Metcalf
2015-10-26 20:32 ` Chris Metcalf
2015-10-21 16:12 ` Frederic Weisbecker
2015-10-27 16:40 ` Chris Metcalf
2016-01-28 16:38 ` Frederic Weisbecker
2016-02-11 19:58 ` Chris Metcalf
2015-10-20 20:36 ` [PATCH v8 05/14] task_isolation: support PR_TASK_ISOLATION_STRICT mode Chris Metcalf
2015-10-20 20:36 ` [PATCH v8 06/14] task_isolation: provide strict mode configurable signal Chris Metcalf
2015-10-21 0:56 ` Steven Rostedt
2015-10-21 1:30 ` Chris Metcalf
2015-10-21 1:41 ` Steven Rostedt
2015-10-21 1:42 ` Andy Lutomirski
2015-10-21 6:41 ` Gilad Ben Yossef
2015-10-21 18:53 ` Andy Lutomirski
2015-10-22 20:44 ` Chris Metcalf
2015-10-22 21:00 ` Andy Lutomirski
2015-10-27 19:37 ` Chris Metcalf
2015-10-24 9:16 ` Gilad Ben Yossef
2015-10-20 20:36 ` [PATCH v8 07/14] task_isolation: add debug boot flag Chris Metcalf
2015-10-20 20:36 ` [PATCH v8 08/14] nohz_full: allow disabling the 1Hz minimum tick at boot Chris Metcalf
2015-10-20 21:03 ` Frederic Weisbecker
2015-10-20 21:18 ` Chris Metcalf
2015-10-21 0:59 ` Steven Rostedt
2015-10-21 6:56 ` Gilad Ben Yossef
2015-10-21 14:28 ` Christoph Lameter
2015-10-21 15:35 ` Frederic Weisbecker
2015-10-20 20:36 ` [PATCH v8 09/14] arch/x86: enable task isolation functionality Chris Metcalf
2015-10-20 20:36 ` [PATCH v8 10/14] arch/arm64: adopt prepare_exit_to_usermode() model from x86 Chris Metcalf
2015-10-20 20:36 ` [PATCH v8 11/14] arch/arm64: enable task isolation functionality Chris Metcalf
2015-10-20 20:36 ` [PATCH v8 12/14] arch/tile: adopt prepare_exit_to_usermode() model from x86 Chris Metcalf
2015-10-20 20:36 ` [PATCH v8 13/14] arch/tile: turn off timer tick for oneshot_stopped state Chris Metcalf
2015-10-20 20:36 ` [PATCH v8 14/14] arch/tile: enable task isolation functionality Chris Metcalf
2015-10-21 12:39 ` [PATCH v8 00/14] support "task_isolation" mode for nohz_full Peter Zijlstra
2015-10-22 20:31 ` Chris Metcalf
2015-10-23 2:33 ` Frederic Weisbecker
2015-10-23 8:49 ` Peter Zijlstra
2015-10-23 13:29 ` Frederic Weisbecker
2015-10-23 9:04 ` Peter Zijlstra
2015-10-23 11:52 ` Theodore Ts'o
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=1431107927-13998-6-git-send-email-cmetcalf@ezchip.com \
--to=cmetcalf@ezchip.com \
--cc=akpm@linux-foundation.org \
--cc=cl@linux.com \
--cc=fweisbec@gmail.com \
--cc=giladb@ezchip.com \
--cc=linux-api@vger.kernel.org \
--cc=linux-doc@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=mingo@redhat.com \
--cc=paulmck@linux.vnet.ibm.com \
--cc=peterz@infradead.org \
--cc=riel@redhat.com \
--cc=rostedt@goodmis.org \
--cc=srivatsa.bhat@linux.vnet.ibm.com \
--cc=tglx@linutronix.de \
--cc=tj@kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®