From: Mike Galbraith <efault@gmx.de>
To: K Prateek Nayak <kprateek.nayak@amd.com>, Phil Auld <pauld@redhat.com>
Cc: Peter Zijlstra <peterz@infradead.org>,
mingo@redhat.com, juri.lelli@redhat.com,
vincent.guittot@linaro.org, dietmar.eggemann@arm.com,
rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de,
vschneid@redhat.com, linux-kernel@vger.kernel.org,
wuyun.abel@bytedance.com, youssefesmat@chromium.org,
tglx@linutronix.de
Subject: Re: [PATCH V2] sched/fair: Dequeue sched_delayed tasks when waking to a busy CPU
Date: Tue, 26 Nov 2024 07:30:52 +0100 [thread overview]
Message-ID: <924b853c01a5466c0bebe3bd324c46e9980dba1c.camel@gmx.de> (raw)
In-Reply-To: <8df808ca-186d-41f8-845c-c42fd2fd4d45@amd.com>
On Tue, 2024-11-26 at 11:02 +0530, K Prateek Nayak wrote:
> Hello Mike,
>
> On 11/23/2024 2:14 PM, Mike Galbraith wrote:
> > [..snip..]
> >
> > Question: did wiping off the evil leave any meaningful goodness behind?
> >
> > ---
> >
> > sched/fair: Dequeue sched_delayed tasks when waking to a busy CPU
> >
> > Phil Auld (Redhat) reported an fio benchmark regression having been found
> > to have been caused by addition of the DELAY_DEQUEUE feature, suggested it
> > may be related to wakees losing the ability to migrate, and confirmed that
> > restoration of same indeed did restore previous performance.
> >
> > V2: do not rip buddies apart, convenient on/off switch
> >
> > Fixes: 152e11f6df29 ("sched/fair: Implement delayed dequeue")
> > Signed-off-by: Mike Galbraith <efault@gmx.de>
> > ---
> > kernel/sched/core.c | 51 ++++++++++++++++++++++++++++++------------------
> > kernel/sched/features.h | 5 ++++
> > kernel/sched/sched.h | 5 ++++
> > 3 files changed, 42 insertions(+), 19 deletions(-)
> >
> > --- a/kernel/sched/core.c
> > +++ b/kernel/sched/core.c
> > @@ -3783,28 +3783,41 @@ ttwu_do_activate(struct rq *rq, struct t
> > */
> > static int ttwu_runnable(struct task_struct *p, int wake_flags)
> > {
> > - struct rq_flags rf;
> > - struct rq *rq;
> > - int ret = 0;
> > -
> > - rq = __task_rq_lock(p, &rf);
> > - if (task_on_rq_queued(p)) {
> > - update_rq_clock(rq);
> > - if (p->se.sched_delayed)
> > - enqueue_task(rq, p, ENQUEUE_NOCLOCK | ENQUEUE_DELAYED);
> > - if (!task_on_cpu(rq, p)) {
> > - /*
> > - * When on_rq && !on_cpu the task is preempted, see if
> > - * it should preempt the task that is current now.
> > - */
> > - wakeup_preempt(rq, p, wake_flags);
> > + CLASS(__task_rq_lock, rq_guard)(p);
> > + struct rq *rq = rq_guard.rq;
> > +
> > + if (!task_on_rq_queued(p))
> > + return 0;
> > +
> > + update_rq_clock(rq);
> > + if (p->se.sched_delayed) {
> > + int queue_flags = ENQUEUE_DELAYED | ENQUEUE_NOCLOCK;
> > + int dequeue = sched_feat(DEQUEUE_DELAYED);
> > +
> > + /*
> > + * Since sched_delayed means we cannot be current anywhere,
> > + * dequeue it here and have it fall through to the
> > + * select_task_rq() case further along in ttwu() path.
> > + * Note: Do not rip buddies apart else chaos follows.
> > + */
> > + if (dequeue && rq->nr_running > 1 && p->nr_cpus_allowed > 1 &&
>
> Do we really care if DEQUEUE_DELAYED is enabled / disabled here when we
> encounter a delayed task?
The switch is for test convenience.
> > + !(rq->curr->last_wakee == p || p->last_wakee == rq->curr)) {
>
> Technically, we are still looking at the last wakeup here since
> record_wakee() is only called later. If we care about 1:1 buddies,
> should we just see "current == p->last_wakee", otherwise, there is a
> good chance "p" has a m:n waker-wakee relationship in which case
> perhaps a "want_affine" like heuristic can help?
The intent is to blunt the instrument a bit. Paul should have a highly
active interrupt source, which will give wakeup credit to whatever is
sitting on that CPU, breaking 1:1 connections.. a little bit. That's
why it still migrates tbench buddies, but NOT at a rate that turns a
tbench progression into a new low regression. The hope is that the
load shift caused by that active interrupt source is enough to give
Paul's regression some of the help it demonstrated wanting, without the
collateral damage. It might now be so weak as to not meet the
"meaningful" in my question, in which case it lands on the ginormous
pile of meh, sorta works, but why would anyone care.
> For science, I was wondering if the decision to dequeue + migrate or
> requeue the delayed task can be put off until after the whole
> select_task_rq() target selection (note: without the h_nr_delayed
> stuff, some of that wake_affine_idle() logic falls apart). Hackbench
> (which saw some regression with EEVDF Complete) seem to like it
> somewhat, but it still falls behind NO_DELAY_DEQUEUE.
You can, with a few more fast path cycles and some duplication, none of
which looks very desirable.
> ==================================================================
> Test : hackbench
> Units : Normalized time in seconds
> Interpretation: Lower is better
> Statistic : AMean
> ==================================================================
> NO_DELAY_DEQUEUE Mike's v2 Full ttwu + requeue/migrate
> 5.76 5.72 ( 1% ) 5.82 ( -1% )
> 6.53 6.56 ( 0% ) 6.65 ( -2% )
> 6.79 7.04 ( -4% ) 7.02 ( -3% )
> 6.91 7.04 ( -2% ) 7.03 ( -2% )
> 7.63 8.05 ( -6% ) 7.88 ( -3% )
>
> Only subtle changes in IBS profiles; there aren't any obvious shift
> in hotspots with hackbench at least. Not sure if it is just the act of
> needing to do a dequeue + enqueue from the wakeup context that adds to
> the overall regression.
Those numbers say say to me that hackbench doesn't care deeply. That
works for me, because I don't care deeply about nutty fork bombs ;-)
-Mike
next prev parent reply other threads:[~2024-11-26 6:31 UTC|newest]
Thread overview: 241+ messages / expand[flat|nested] mbox.gz Atom feed top
2024-07-27 10:27 [PATCH 00/24] Complete EEVDF Peter Zijlstra
2024-07-27 10:27 ` [PATCH 01/24] sched/eevdf: Add feature comments Peter Zijlstra
2024-08-18 6:23 ` [tip: sched/core] " tip-bot2 for Peter Zijlstra
2024-07-27 10:27 ` [PATCH 02/24] sched/eevdf: Remove min_vruntime_copy Peter Zijlstra
2024-08-18 6:23 ` [tip: sched/core] " tip-bot2 for Peter Zijlstra
2024-07-27 10:27 ` [PATCH 03/24] sched/fair: Cleanup pick_task_fair() vs throttle Peter Zijlstra
2024-08-18 6:23 ` [tip: sched/core] " tip-bot2 for Peter Zijlstra
2024-07-27 10:27 ` [PATCH 04/24] sched/fair: Cleanup pick_task_fair()s curr Peter Zijlstra
2024-08-18 6:23 ` [tip: sched/core] sched/fair: Cleanup pick_task_fair()'s curr tip-bot2 for Peter Zijlstra
2024-07-27 10:27 ` [PATCH 05/24] sched/fair: Unify pick_{,next_}_task_fair() Peter Zijlstra
2024-08-18 6:23 ` [tip: sched/core] " tip-bot2 for Peter Zijlstra
2024-07-27 10:27 ` [PATCH 06/24] sched: Allow sched_class::dequeue_task() to fail Peter Zijlstra
2024-08-18 6:23 ` [tip: sched/core] " tip-bot2 for Peter Zijlstra
2024-07-27 10:27 ` [PATCH 07/24] sched/fair: Re-organize dequeue_task_fair() Peter Zijlstra
2024-08-09 16:53 ` Valentin Schneider
2024-08-10 22:17 ` Peter Zijlstra
2024-08-12 10:02 ` Valentin Schneider
2024-08-18 6:23 ` [tip: sched/core] " tip-bot2 for Peter Zijlstra
2024-07-27 10:27 ` [PATCH 08/24] sched: Split DEQUEUE_SLEEP from deactivate_task() Peter Zijlstra
2024-08-18 6:23 ` [tip: sched/core] " tip-bot2 for Peter Zijlstra
2024-07-27 10:27 ` [PATCH 09/24] sched: Prepare generic code for delayed dequeue Peter Zijlstra
2024-08-18 6:23 ` [tip: sched/core] " tip-bot2 for Peter Zijlstra
2024-07-27 10:27 ` [PATCH 10/24] sched/uclamg: Handle " Peter Zijlstra
2024-08-18 6:23 ` [tip: sched/core] " tip-bot2 for Peter Zijlstra
2024-08-19 9:14 ` Christian Loehle
2024-08-20 16:23 ` [PATCH 10/24] " Hongyan Xia
2024-08-21 13:34 ` Hongyan Xia
2024-08-22 8:19 ` Vincent Guittot
2024-08-22 8:21 ` Vincent Guittot
2024-08-22 9:21 ` Luis Machado
2024-08-22 9:53 ` Vincent Guittot
2024-08-22 10:20 ` Vincent Guittot
2024-08-22 10:28 ` Luis Machado
2024-08-22 12:07 ` Luis Machado
2024-08-22 12:10 ` Vincent Guittot
2024-08-22 14:58 ` Vincent Guittot
2024-08-29 15:42 ` Hongyan Xia
2024-09-05 13:02 ` Dietmar Eggemann
2024-09-05 13:33 ` Vincent Guittot
2024-09-05 14:07 ` Dietmar Eggemann
2024-09-05 14:29 ` Vincent Guittot
2024-09-05 14:50 ` Dietmar Eggemann
2024-09-05 14:53 ` Peter Zijlstra
2024-09-06 6:14 ` Vincent Guittot
2024-09-06 10:45 ` Peter Zijlstra
2024-09-08 7:43 ` Mike Galbraith
2024-09-10 8:09 ` [tip: sched/core] sched/eevdf: More PELT vs DELAYED_DEQUEUE tip-bot2 for Peter Zijlstra
2024-11-27 4:17 ` K Prateek Nayak
2024-11-27 9:34 ` Luis Machado
2024-11-28 6:35 ` K Prateek Nayak
2024-09-10 11:04 ` [PATCH 10/24] sched/uclamg: Handle delayed dequeue Luis Machado
2024-09-10 14:05 ` Peter Zijlstra
2024-09-11 8:35 ` Luis Machado
2024-09-11 8:45 ` Peter Zijlstra
2024-09-11 8:55 ` Luis Machado
2024-09-11 9:10 ` Mike Galbraith
2024-09-11 9:13 ` Peter Zijlstra
2024-09-11 9:27 ` Mike Galbraith
2024-09-12 14:00 ` Mike Galbraith
2024-09-13 16:39 ` Mike Galbraith
2024-09-14 3:40 ` Mike Galbraith
2024-09-24 15:16 ` Luis Machado
2024-09-24 17:35 ` Mike Galbraith
2024-09-25 5:14 ` Mike Galbraith
2024-09-11 11:49 ` Dietmar Eggemann
2024-09-11 9:38 ` Luis Machado
2024-09-12 12:58 ` Luis Machado
2024-09-12 20:44 ` Dietmar Eggemann
2024-09-11 10:46 ` Luis Machado
2024-09-06 9:55 ` Dietmar Eggemann
2024-09-05 14:18 ` Peter Zijlstra
2024-09-10 8:09 ` [tip: sched/core] kernel/sched: Fix util_est accounting for DELAY_DEQUEUE tip-bot2 for Dietmar Eggemann
2024-07-27 10:27 ` [PATCH 11/24] sched/fair: Assert {set_next,put_prev}_entity() are properly balanced Peter Zijlstra
2024-08-18 6:23 ` [tip: sched/core] " tip-bot2 for Peter Zijlstra
2024-07-27 10:27 ` [PATCH 12/24] sched/fair: Prepare exit/cleanup paths for delayed_dequeue Peter Zijlstra
2024-08-13 12:43 ` Valentin Schneider
2024-08-13 21:54 ` Peter Zijlstra
2024-08-13 22:07 ` Peter Zijlstra
2024-08-14 5:53 ` Peter Zijlstra
2024-08-27 9:35 ` Chen Yu
2024-08-27 20:29 ` Valentin Schneider
2024-08-28 2:55 ` Chen Yu
2024-08-18 6:23 ` [tip: sched/core] " tip-bot2 for Peter Zijlstra
2024-08-27 9:17 ` [PATCH 12/24] " Chen Yu
2024-08-28 3:06 ` Chen Yu
2024-07-27 10:27 ` [PATCH 13/24] sched/fair: Prepare pick_next_task() for delayed dequeue Peter Zijlstra
2024-08-18 6:23 ` [tip: sched/core] " tip-bot2 for Peter Zijlstra
2024-09-10 9:16 ` [PATCH 13/24] " Luis Machado
2024-07-27 10:27 ` [PATCH 14/24] sched/fair: Implement ENQUEUE_DELAYED Peter Zijlstra
2024-08-18 6:23 ` [tip: sched/core] " tip-bot2 for Peter Zijlstra
2024-07-27 10:27 ` [PATCH 15/24] sched,freezer: Mark TASK_FROZEN special Peter Zijlstra
2024-08-18 6:23 ` [tip: sched/core] " tip-bot2 for Peter Zijlstra
2024-07-27 10:27 ` [PATCH 16/24] sched: Teach dequeue_task() about special task states Peter Zijlstra
2024-08-18 6:23 ` [tip: sched/core] " tip-bot2 for Peter Zijlstra
2024-07-27 10:27 ` [PATCH 17/24] sched/fair: Implement delayed dequeue Peter Zijlstra
2024-08-02 14:39 ` Valentin Schneider
2024-08-02 14:59 ` Peter Zijlstra
2024-08-02 16:32 ` Valentin Schneider
2024-08-18 6:23 ` [tip: sched/core] " tip-bot2 for Peter Zijlstra
2024-08-19 10:01 ` [PATCH 17/24] " Luis Machado
[not found] ` <CGME20240828223802eucas1p16755f4531ed0611dc4871649746ea774@eucas1p1.samsung.com>
2024-08-28 22:38 ` Marek Szyprowski
2024-10-10 2:49 ` Sean Christopherson
2024-10-10 7:57 ` Mike Galbraith
2024-10-10 16:18 ` Sean Christopherson
2024-10-10 17:12 ` Mike Galbraith
2024-10-10 8:19 ` Peter Zijlstra
2024-10-10 9:18 ` Peter Zijlstra
2024-10-10 18:23 ` Sean Christopherson
2024-10-12 14:15 ` [tip: sched/urgent] sched: Fix external p->on_rq users tip-bot2 for Peter Zijlstra
2024-10-14 7:28 ` [tip: sched/urgent] sched/fair: " tip-bot2 for Peter Zijlstra
2024-11-01 12:47 ` [PATCH 17/24] sched/fair: Implement delayed dequeue Phil Auld
2024-11-01 12:56 ` Peter Zijlstra
2024-11-01 13:38 ` Phil Auld
2024-11-01 14:26 ` Peter Zijlstra
2024-11-01 14:42 ` Phil Auld
2024-11-01 18:08 ` Mike Galbraith
2024-11-01 20:07 ` Phil Auld
2024-11-02 4:32 ` Mike Galbraith
2024-11-04 13:05 ` Phil Auld
2024-11-05 4:05 ` Mike Galbraith
2024-11-05 4:22 ` K Prateek Nayak
2024-11-05 6:46 ` Mike Galbraith
2024-11-06 3:02 ` K Prateek Nayak
2024-11-05 15:20 ` Phil Auld
2024-11-05 19:05 ` Phil Auld
2024-11-06 2:45 ` Mike Galbraith
2024-11-06 13:53 ` Peter Zijlstra
2024-11-06 14:14 ` Peter Zijlstra
2024-11-06 14:38 ` Peter Zijlstra
2024-11-06 15:22 ` Mike Galbraith
2024-11-07 4:03 ` Mike Galbraith
2024-11-07 9:46 ` Mike Galbraith
2024-11-07 14:02 ` Mike Galbraith
2024-11-07 14:09 ` Peter Zijlstra
2024-11-08 0:24 ` [PATCH] sched/fair: Dequeue sched_delayed tasks when waking to a busy CPU Mike Galbraith
2024-11-08 13:34 ` Phil Auld
2024-11-11 2:46 ` Xuewen Yan
2024-11-11 3:53 ` Mike Galbraith
2024-11-12 7:05 ` Mike Galbraith
2024-11-12 12:41 ` Phil Auld
2024-11-12 14:23 ` Peter Zijlstra
2024-11-12 14:23 ` Mike Galbraith
2024-11-12 15:41 ` Phil Auld
2024-11-12 16:15 ` Mike Galbraith
2024-11-14 11:07 ` Mike Galbraith
2024-11-14 11:28 ` Phil Auld
2024-11-19 11:30 ` Phil Auld
2024-11-19 11:51 ` Mike Galbraith
2024-11-20 18:37 ` Mike Galbraith
2024-11-21 11:56 ` Phil Auld
2024-11-21 12:07 ` Phil Auld
2024-11-21 21:21 ` Phil Auld
2024-11-23 8:44 ` [PATCH V2] " Mike Galbraith
2024-11-26 5:32 ` K Prateek Nayak
2024-11-26 6:30 ` Mike Galbraith [this message]
2024-11-26 9:42 ` Mike Galbraith
2024-12-02 19:15 ` Phil Auld
2024-11-27 14:13 ` Mike Galbraith
2024-12-02 16:24 ` Phil Auld
2024-12-02 16:55 ` Mike Galbraith
2024-12-02 19:12 ` Phil Auld
2024-12-09 13:11 ` Phil Auld
2024-12-09 15:06 ` Mike Galbraith
2024-11-06 14:14 ` [PATCH 17/24] sched/fair: Implement delayed dequeue Mike Galbraith
2024-11-06 14:33 ` Peter Zijlstra
2024-11-04 9:28 ` Dietmar Eggemann
2024-11-04 11:55 ` Dietmar Eggemann
2024-11-04 12:50 ` Phil Auld
2024-11-05 9:53 ` Christian Loehle
2024-11-05 15:55 ` Phil Auld
2024-11-08 14:53 ` Dietmar Eggemann
2024-11-08 18:16 ` Phil Auld
2024-11-11 11:29 ` Dietmar Eggemann
2024-07-27 10:27 ` [PATCH 18/24] sched/fair: Implement DELAY_ZERO Peter Zijlstra
2024-08-18 6:23 ` [tip: sched/core] " tip-bot2 for Peter Zijlstra
2024-07-27 10:27 ` [PATCH 19/24] sched/eevdf: Fixup PELT vs DELAYED_DEQUEUE Peter Zijlstra
2024-08-13 12:43 ` Valentin Schneider
2024-08-13 22:18 ` Peter Zijlstra
2024-08-14 7:25 ` Peter Zijlstra
2024-08-14 7:28 ` Peter Zijlstra
2024-08-14 10:23 ` Valentin Schneider
2024-08-14 12:59 ` Vincent Guittot
2024-08-17 23:06 ` Peter Zijlstra
2024-08-19 12:50 ` Vincent Guittot
2024-08-18 6:23 ` [tip: sched/core] " tip-bot2 for Peter Zijlstra
2024-07-27 10:27 ` [PATCH 20/24] sched/fair: Avoid re-setting virtual deadline on migrations Peter Zijlstra
2024-08-18 6:23 ` [tip: sched/core] sched/fair: Avoid re-setting virtual deadline on 'migrations' tip-bot2 for Peter Zijlstra
2024-07-27 10:27 ` [PATCH 21/24] sched/eevdf: Allow shorter slices to wakeup-preempt Peter Zijlstra
2024-08-05 12:24 ` Chunxin Zang
2024-08-07 17:54 ` Peter Zijlstra
2024-08-13 10:44 ` Chunxin Zang
2024-08-08 10:15 ` Chen Yu
2024-08-08 10:22 ` Peter Zijlstra
2024-08-08 12:31 ` Chen Yu
2024-08-09 7:35 ` Peter Zijlstra
2024-08-18 6:23 ` [tip: sched/core] " tip-bot2 for Peter Zijlstra
2024-07-27 10:27 ` [PATCH 22/24] sched/eevdf: Use sched_attr::sched_runtime to set request/slice suggestion Peter Zijlstra
2024-08-18 6:23 ` [tip: sched/core] " tip-bot2 for Peter Zijlstra
2024-07-27 10:27 ` [PATCH 23/24] sched/eevdf: Propagate min_slice up the cgroup hierarchy Peter Zijlstra
2024-08-18 6:23 ` [tip: sched/core] " tip-bot2 for Peter Zijlstra
2024-09-29 2:02 ` [PATCH 23/24] " Tianchen Ding
2024-07-27 10:27 ` [RFC PATCH 24/24] sched/time: Introduce CLOCK_THREAD_DVFS_ID Peter Zijlstra
2024-07-28 21:30 ` Thomas Gleixner
2024-07-29 7:53 ` Juri Lelli
2024-08-02 11:29 ` Peter Zijlstra
2024-08-19 11:11 ` Christian Loehle
2024-08-01 12:08 ` [PATCH 00/24] Complete EEVDF Luis Machado
2024-08-14 14:34 ` Vincent Guittot
2024-08-14 16:45 ` Mike Galbraith
2024-08-14 16:59 ` Vincent Guittot
2024-08-14 17:18 ` Mike Galbraith
2024-08-14 17:25 ` Vincent Guittot
2024-08-14 17:35 ` K Prateek Nayak
2024-08-16 15:22 ` Valentin Schneider
2024-08-20 16:43 ` Hongyan Xia
2024-08-21 9:46 ` Hongyan Xia
2024-08-21 16:25 ` Mike Galbraith
2024-08-22 15:55 ` Peter Zijlstra
2024-08-27 9:43 ` Hongyan Xia
2024-08-29 17:02 ` Aleksandr Nogikh
2024-09-10 11:45 ` Sven Schnelle
2024-09-10 12:21 ` Sven Schnelle
2024-09-10 14:07 ` Peter Zijlstra
2024-09-10 14:52 ` Sven Schnelle
2024-11-06 1:07 ` Saravana Kannan
2024-11-06 6:19 ` K Prateek Nayak
2024-11-06 11:09 ` Peter Zijlstra
2024-11-06 12:06 ` Luis Machado
2024-11-08 7:07 ` Saravana Kannan
2024-11-08 23:17 ` Samuel Wu
2024-11-11 4:07 ` K Prateek Nayak
2024-11-26 23:32 ` Saravana Kannan
2024-11-28 10:32 ` [REGRESSION] " Marcel Ziswiler
2024-11-28 10:58 ` Peter Zijlstra
2024-11-28 11:37 ` Marcel Ziswiler
2024-11-29 9:08 ` Peter Zijlstra
2024-12-02 18:46 ` Marcel Ziswiler
2024-12-09 9:49 ` Peter Zijlstra
2024-12-10 16:05 ` Marcel Ziswiler
2024-12-10 16:13 ` Steven Rostedt
2024-12-10 8:45 ` Luis Machado
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=924b853c01a5466c0bebe3bd324c46e9980dba1c.camel@gmx.de \
--to=efault@gmx.de \
--cc=bsegall@google.com \
--cc=dietmar.eggemann@arm.com \
--cc=juri.lelli@redhat.com \
--cc=kprateek.nayak@amd.com \
--cc=linux-kernel@vger.kernel.org \
--cc=mgorman@suse.de \
--cc=mingo@redhat.com \
--cc=pauld@redhat.com \
--cc=peterz@infradead.org \
--cc=rostedt@goodmis.org \
--cc=tglx@linutronix.de \
--cc=vincent.guittot@linaro.org \
--cc=vschneid@redhat.com \
--cc=wuyun.abel@bytedance.com \
--cc=youssefesmat@chromium.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®