mirror of https://lore.kernel.org/lkml/
 help / color / mirror / Atom feed
From: Hannes Reinecke <hare@suse.de>
To: Mohamed Khalfella <mkhalfella@purestorage.com>,
	Keith Busch <kbusch@kernel.org>, Jens Axboe <axboe@kernel.dk>,
	Christoph Hellwig <hch@lst.de>, Sagi Grimberg <sagi@grimberg.me>
Cc: Justin Tee <justin.tee@broadcom.com>,
	Naresh Gottumukkala <nareshgottumukkala83@gmail.com>,
	Paul Ely <paul.ely@broadcom.com>,
	Hannes Reinecke <hare@kernel.org>,
	Chaitanya Kulkarni <kch@nvidia.com>,
	James Smart <jsmart833426@gmail.com>,
	Randy Jennings <randyj@purestorage.com>,
	Daniel Wagner <dwagner@suse.de>,
	linux-nvme@lists.infradead.org, linux-kernel@vger.kernel.org
Subject: Re: [PATCH v7 15/17] nvme-fc: Hold inflight requests while in FENCING state
Date: Mon, 16 Nov 2026 08:52:37 +0100	[thread overview]
Message-ID: <508eed68-2633-45dd-b503-65c62b5fe16c@suse.de> (raw)
In-Reply-To: <20261114030454.2836543-16-mkhalfella@purestorage.com>

On 11/14/26 4:04 AM, Mohamed Khalfella wrote:
> When a controller enters FENCING, its inflight requests must not be
> completed until CCR or CQT guarantees that the target can no longer
> execute them. Completing them earlier lets multipath resubmit them on
> another path while the original command may still run on this one.
> 
> nvme-fc currently cannot hold requests across the fence.
> nvme_fc_fencing_work() tears the association down first, so the LLDD
> can finish unregistering the remoteport/localport, and
> nvme_fc_delete_association() completes all requests as part of the
> teardown. nvme_fc_fcpio_done() also completes requests that finish with
> an error, which happens to every request aborted by the teardown.
> 
> Split nvme_fc_delete_association() in two:
> 
>   - __nvme_fc_teardown_association() aborts outstanding ios, waits for
>     the aborts to drain, sends Disconnect, deletes the hw queues and
>     marks the controller inactive on the rport. It completes no
>     requests.
> 
>   - nvme_fc_flush_held_requests() completes the held requests and
>     unquiesces the queues.
> 
> nvme_fc_delete_association() calls both, as before.
> nvme_fc_fencing_work() calls the teardown up front and the flush only
> after nvme_fence_ctrl() returns, before moving to FENCED. The queues
> stay quiesced in between, so nothing new reaches the dead association.
> 
> In nvme_fc_fcpio_done(), start recovery first and hold the request
> only if the controller is FENCING afterwards, since a held request is
> completed only by the flush that fencing_work runs. If the claim fails
> because a reset, a delete, or a connect in progress owns the controller,
> the request is completed at once and that owner finishes the teardown.
> A held request cannot be completed behind the fence's back because
> nvme_fc_timeout() returns BLK_EH_RESET_TIMER while FENCING, and
> nvme_fc_start_ioerr_recovery() does nothing in FENCING.
> 
> Track held requests explicitly in op->state with a new
> FCPOP_STATE_HELD, and have the flush complete only those. An op enters
> HELD only through cmpxchg(COMPLETE -> HELD) in nvme_fc_fcpio_done().
> For each busy request, the flush does cmpxchg(COMPLETE -> IDLE) and
> cancels the request only if it saw HELD. This settles the race with an
> nvme_fc_fcpio_done() that has already moved its op to COMPLETE but has
> not yet decided whether to hold it. Teardown does not wait for such an
> op, since it was no longer ACTIVE when the aborts were issued:
> 
>   - nvme_fc_fcpio_done() wins: the op is HELD and the flush cancels it.
>   - The flush wins: the op is IDLE, the hold cmpxchg fails and
>     nvme_fc_fcpio_done() completes the request itself.
> 
> The flush must use a single cmpxchg. Reading the state and then acting
> on it would let the op become HELD in between and leave the request
> with nothing to complete it.
> 
> Both cases rely on the flush being the only thing that can move an op
> out of COMPLETE behind nvme_fc_fcpio_done(), so make
> __nvme_fc_abort_op() use cmpxchg(ACTIVE -> ABORTED). It used
> xchg(ABORTED) and then restored the old value if the op was not ACTIVE,
> so for a moment such an op read as ABORTED. A hold cmpxchg that ran in
> that moment would fail and complete the request in the middle of the
> fence.
> 
> Count an aborted op down only after nvme_fc_fcpio_done() has completed
> or held its request, so that teardown, once it stops waiting, knows
> every aborted request has been decided. The count-down now runs outside
> ctrl->lock, so make ctrl->iocnt an atomic_t and wait for it with
> wait_event().
> 
> Signed-off-by: Mohamed Khalfella <mkhalfella@purestorage.com>
> ---
>   drivers/nvme/host/fc.c | 171 +++++++++++++++++++++++++++++++++--------
>   1 file changed, 139 insertions(+), 32 deletions(-)
> 
> diff --git a/drivers/nvme/host/fc.c b/drivers/nvme/host/fc.c
> index 095adc9011cc..5e8977ad51cf 100644
> --- a/drivers/nvme/host/fc.c
> +++ b/drivers/nvme/host/fc.c
> @@ -83,6 +83,7 @@ enum nvme_fcpop_state {
>   	FCPOP_STATE_ACTIVE	= 2,
>   	FCPOP_STATE_ABORTED	= 3,
>   	FCPOP_STATE_COMPLETE	= 4,
> +	FCPOP_STATE_HELD	= 5,
>   };
>   
>   struct nvme_fc_fcp_op {
> @@ -172,7 +173,7 @@ struct nvme_fc_ctrl {
>   
>   	struct kref		ref;
>   	unsigned long		flags;
> -	u32			iocnt;
> +	atomic_t		iocnt;
>   	wait_queue_head_t	ioabort_wait;
>   
>   	struct nvme_fc_fcp_op	aen_ops[NVME_NR_AEN_COMMANDS];
> @@ -990,6 +991,8 @@ fc_dma_unmap_sg(struct device *dev, struct scatterlist *sg, int nents,
>   static void nvme_fc_ctrl_put(struct nvme_fc_ctrl *);
>   static int nvme_fc_ctrl_get(struct nvme_fc_ctrl *);
>   
> +static void __nvme_fc_teardown_association(struct nvme_fc_ctrl *ctrl);
> +static void nvme_fc_flush_held_requests(struct nvme_fc_ctrl *ctrl);
>   static void nvme_fc_delete_association(struct nvme_fc_ctrl *ctrl);
>   static void nvme_fc_reconnect_or_delete(struct nvme_fc_ctrl *ctrl, int status);
>   
> @@ -1828,12 +1831,11 @@ __nvme_fc_abort_op(struct nvme_fc_ctrl *ctrl, struct nvme_fc_fcp_op *op)
>   	 * path, must not still be in progress in the LLDD then.
>   	 */
>   	spin_lock_irqsave(&ctrl->lock, flags);
> -	opstate = atomic_xchg(&op->state, FCPOP_STATE_ABORTED);
> -	if (opstate != FCPOP_STATE_ACTIVE) {
> -		atomic_set(&op->state, opstate);
> -	} else {
> +	opstate = atomic_cmpxchg(&op->state, FCPOP_STATE_ACTIVE,
> +				 FCPOP_STATE_ABORTED);
> +	if (opstate == FCPOP_STATE_ACTIVE) {
>   		op->flags |= FCOP_FLAGS_TERMIO;
> -		ctrl->iocnt++;
> +		atomic_inc(&ctrl->iocnt);
>   	}
>   	spin_unlock_irqrestore(&ctrl->lock, flags);
>   

Nit: You could've moved the change from atomic_xchg() to 
atomic_cmpxchg() to patch 12, to keep this change cleaner.

> @@ -1863,19 +1865,27 @@ nvme_fc_abort_aen_ops(struct nvme_fc_ctrl *ctrl)
>   }
>   
>   static inline void
> +__nvme_fc_fcpop_count_one_down(struct nvme_fc_ctrl *ctrl)
> +{
> +	if (atomic_dec_return(&ctrl->iocnt) == 0)
> +		wake_up(&ctrl->ioabort_wait);
> +}
> +
> +static inline bool
>   __nvme_fc_fcpop_chk_teardowns(struct nvme_fc_ctrl *ctrl,
>   		struct nvme_fc_fcp_op *op, int opstate)
>   {
>   	unsigned long flags;
> +	bool ret = false;
>   
>   	if (opstate == FCPOP_STATE_ABORTED) {
>   		spin_lock_irqsave(&ctrl->lock, flags);
> -		if (op->flags & FCOP_FLAGS_TERMIO) {
> -			if (!--ctrl->iocnt)
> -				wake_up(&ctrl->ioabort_wait);
> -		}
> +		if (op->flags & FCOP_FLAGS_TERMIO)
> +			ret = true;
>   		spin_unlock_irqrestore(&ctrl->lock, flags);
>   	}
> +
> +	return ret;
>   }
>   
>   static void nvme_fc_fencing_work(struct work_struct *work)
> @@ -1894,12 +1904,14 @@ static void nvme_fc_fencing_work(struct work_struct *work)
>   	 */
>   	nvme_stop_keep_alive(ctrl);
>   	cancel_work_sync(&ctrl->async_event_work);
> -	nvme_fc_delete_association(fc_ctrl);
> +	if (test_and_clear_bit(ASSOC_ACTIVE, &fc_ctrl->flags))
> +		__nvme_fc_teardown_association(fc_ctrl);
>   
>   	rem = nvme_fence_ctrl(ctrl);
>   	if (rem)
>   		dev_info(ctrl->device, "CCR failed, starting error recovery\n");
>   
> +	nvme_fc_flush_held_requests(fc_ctrl);
>   	nvme_change_ctrl_state(ctrl, NVME_CTRL_FENCED);
>   	if (nvme_change_ctrl_state(ctrl, NVME_CTRL_RESETTING))
>   		queue_work(nvme_reset_wq, &fc_ctrl->ioerr_work);
> @@ -1982,7 +1994,7 @@ nvme_fc_fcpio_done(struct nvmefc_fcp_req *req)
>   	struct nvme_command *sqe = &op->cmd_iu.sqe;
>   	__le16 status = cpu_to_le16(NVME_SC_SUCCESS << 1);
>   	union nvme_result result;
> -	bool terminate_assoc = true;
> +	bool op_term, hold = false, terminate_assoc = true;
>   	int opstate;
>   
>   	/*
> @@ -2113,22 +2125,58 @@ nvme_fc_fcpio_done(struct nvmefc_fcp_req *req)
>   	terminate_assoc = false;
>   
>   done:
> +	/*
> +	 * Start recovery before completing or holding anything. A held
> +	 * request is only completed by nvme_fc_flush_held_requests(), which
> +	 * fencing_work runs once the fence is over, so a request may only be
> +	 * held once FENCING has been claimed and fencing_work is guaranteed
> +	 * to run. If the claim fails because a reset or a delete already owns
> +	 * the controller, the request is completed below and that owner
> +	 * tears the association down.
> +	 */
> +	if (terminate_assoc) {
> +		nvme_fc_start_ioerr_recovery(ctrl, "io error");
> +		hold = nvme_ctrl_state(&ctrl->ctrl) == NVME_CTRL_FENCING;
> +	}
> +
>   	if (op->flags & FCOP_FLAGS_AEN) {
>   		nvme_complete_async_event(&queue->ctrl->ctrl, status, &result);
> -		__nvme_fc_fcpop_chk_teardowns(ctrl, op, opstate);
> +		if (__nvme_fc_fcpop_chk_teardowns(ctrl, op, opstate))
> +			__nvme_fc_fcpop_count_one_down(ctrl);
>   		atomic_set(&op->state, FCPOP_STATE_IDLE);
>   		op->flags = FCOP_FLAGS_AEN;	/* clear other flags */
>   		nvme_fc_ctrl_put(ctrl);
> -		goto check_error;
> +		return;
>   	}
>   
> -	__nvme_fc_fcpop_chk_teardowns(ctrl, op, opstate);
> +	/*
> +	 * We can not access op after the request is completed because it can
> +	 * be reused immediately. At the same time we want to wakeup the thread
> +	 * waiting for ongoing IOs _after_ requests are completed. This is
> +	 * necessary because that thread will start canceling inflight IOs
> +	 * and we want to avoid request completion racing with cancellation.
> +	 */
> +	op_term = __nvme_fc_fcpop_chk_teardowns(ctrl, op, opstate);
> +
> +	/*
> +	 * If the controller is FENCING, do not complete this request now.
> +	 * Move the op to HELD and let nvme_fc_cancel_held_requests() cancel
> +	 * it when it is safe to do so. HELD is only reachable from COMPLETE
> +	 * through this cmpxchg: if the cancellation has already walked the
> +	 * tagset it moved the op out of COMPLETE, the cmpxchg fails and the
> +	 * request is completed here instead. Either way exactly one path
> +	 * completes it.
> +	 */
> +	if (hold && atomic_cmpxchg(&op->state, FCPOP_STATE_COMPLETE,
> +				   FCPOP_STATE_HELD) == FCPOP_STATE_COMPLETE)
> +		goto check_op_term;
> +
>   	if (!nvme_try_complete_req(rq, status, result))
>   		nvme_fc_complete_rq(rq);
>   
> -check_error:
> -	if (terminate_assoc)
> -		nvme_fc_start_ioerr_recovery(ctrl, "io error");
> +check_op_term:
> +	if (op_term)
> +		__nvme_fc_fcpop_count_one_down(ctrl);
>   }
>   
>   static int
> @@ -2803,7 +2851,8 @@ nvme_fc_start_fcp_op(struct nvme_fc_ctrl *ctrl, struct nvme_fc_queue *queue,
>   		 * cmd with the csn was supposed to arrive.
>   		 */
>   		opstate = atomic_xchg(&op->state, FCPOP_STATE_COMPLETE);
> -		__nvme_fc_fcpop_chk_teardowns(ctrl, op, opstate);
> +		if (__nvme_fc_fcpop_chk_teardowns(ctrl, op, opstate))
> +			__nvme_fc_fcpop_count_one_down(ctrl);
>   
>   		if (!(op->flags & FCOP_FLAGS_AEN)) {
>   			nvme_fc_unmap_data(ctrl, op->rq, op);
> @@ -3261,22 +3310,22 @@ nvme_fc_create_association(struct nvme_fc_ctrl *ctrl)
>   	return ret;
>   }
>   
> -
>   /*
> - * This routine stops operation of the controller on the host side.
> - * On the host os stack side: Admin and IO queues are stopped,
> - *   outstanding ios on them terminated via FC ABTS.
> - * On the link side: the association is terminated.
> + * This routine tears down the association on the link side and releases
> + * all LLDD resources: outstanding ios are terminated via FC ABTS, hw
> + * queues are deleted and the controller is marked inactive on the
> + * rport. Requests that nvme_fc_fcpio_done() decided to hold are NOT
> + * completed here, the caller must follow up with
> + * nvme_fc_flush_held_requests() once it is safe to release them.
> + *
> + * The caller must own the ASSOC_ACTIVE bit (test_and_clear_bit()).
>    */
>   static void
> -nvme_fc_delete_association(struct nvme_fc_ctrl *ctrl)
> +__nvme_fc_teardown_association(struct nvme_fc_ctrl *ctrl)
>   {
>   	struct nvmefc_ls_rcv_op *disls = NULL;
>   	unsigned long flags;
>   
> -	if (!test_and_clear_bit(ASSOC_ACTIVE, &ctrl->flags))
> -		return;
> -
>   	spin_lock_irqsave(&ctrl->lock, flags);
>   	set_bit(FCCTRL_TERMIO, &ctrl->flags);
>   	spin_unlock_irqrestore(&ctrl->lock, flags);
> @@ -3287,8 +3336,8 @@ nvme_fc_delete_association(struct nvme_fc_ctrl *ctrl)
>   	nvme_fc_abort_aen_ops(ctrl);
>   
>   	/* wait for all io that had to be aborted */
> +	wait_event(ctrl->ioabort_wait, atomic_read(&ctrl->iocnt) == 0);
>   	spin_lock_irq(&ctrl->lock);
> -	wait_event_lock_irq(ctrl->ioabort_wait, ctrl->iocnt == 0, ctrl->lock);
>   	clear_bit(FCCTRL_TERMIO, &ctrl->flags);
>   	spin_unlock_irq(&ctrl->lock);
>   

But this makes the spin_lock() here a bit pointless, protecting as it is
only a single 'flags' value.

> @@ -3323,13 +3372,71 @@ nvme_fc_delete_association(struct nvme_fc_ctrl *ctrl)
>   	__nvme_fc_delete_hw_queue(ctrl, &ctrl->queues[0], 0);
>   	nvme_fc_free_queue(&ctrl->queues[0]);
>   
> +	nvme_fc_ctlr_inactive_on_rport(ctrl);
> +}
> +
> +static bool
> +nvme_fc_cancel_held_request(struct request *req, void *data)
> +{
> +	struct nvme_fc_fcp_op *op = blk_mq_rq_to_pdu(req);
> +	struct nvme_ctrl *ctrl = data;
> +
> +	/*
> +	 * A single cmpxchg decides who completes this request. If the op is
> +	 * HELD, nvme_fc_fcpio_done() is done with it and we cancel it. If it
> +	 * is still COMPLETE, nvme_fc_fcpio_done() has not decided yet: move
> +	 * it to IDLE so its own cmpxchg to HELD fails and it completes the
> +	 * request itself (nvme_fc_complete_rq() sets IDLE anyway). Doing this
> +	 * as read-then-cmpxchg would let the op become HELD in between and
> +	 * leave it with no one to complete it.
> +	 */
> +	if (atomic_cmpxchg(&op->state, FCPOP_STATE_COMPLETE,
> +			   FCPOP_STATE_IDLE) == FCPOP_STATE_HELD)
> +		nvme_cancel_request(req, ctrl);

That is _soo_ odd. So you're doing a compare-exchange, but decode to act 
on it only if the compare-exchange fails for one specific reason.
What happens if 'op-state' has any other value?
Is it still okay to move on?

> +
> +	return true;
> +}
> +
> +static void
> +nvme_fc_cancel_held_requests(struct nvme_ctrl *ctrl, struct blk_mq_tag_set *set)
> +{
> +	if (!set)
> +		return;
> +
> +	blk_mq_tagset_busy_iter(set, nvme_fc_cancel_held_request, ctrl);
> +	blk_mq_tagset_wait_completed_request(set);
> +}
> +
> +/* Complete the requests nvme_fc_fcpio_done() held during the fence */
> +static void
> +nvme_fc_flush_held_requests(struct nvme_fc_ctrl *fc_ctrl)
> +{
> +	struct nvme_ctrl *ctrl = &fc_ctrl->ctrl;
> +
> +	nvme_fc_cancel_held_requests(ctrl, ctrl->tagset);
> +	nvme_fc_cancel_held_requests(ctrl, ctrl->admin_tagset);
> +
>   	/* re-enable the admin_q so anything new can fast fail */
> -	nvme_unquiesce_admin_queue(&ctrl->ctrl);
> +	nvme_unquiesce_admin_queue(ctrl);
>   
>   	/* resume the io queues so that things will fast fail */
> -	nvme_unquiesce_io_queues(&ctrl->ctrl);
> +	nvme_unquiesce_io_queues(ctrl);
> +}
>   
> -	nvme_fc_ctlr_inactive_on_rport(ctrl);
> +/*
> + * This routine stops operation of the controller on the host side.
> + * On the host os stack side: Admin and IO queues are stopped,
> + *   outstanding ios on them terminated via FC ABTS.
> + * On the link side: the association is terminated.
> + */
> +static void
> +nvme_fc_delete_association(struct nvme_fc_ctrl *ctrl)
> +{
> +	if (!test_and_clear_bit(ASSOC_ACTIVE, &ctrl->flags))
> +		return;
> +
> +	__nvme_fc_teardown_association(ctrl);
> +	nvme_fc_flush_held_requests(ctrl);
>   }
>   
>   static void

Cheers,

Hannes
-- 
Dr. Hannes Reinecke                  Kernel Storage Architect
hare@suse.de                                +49 911 74053 688
SUSE Software Solutions GmbH, Frankenstr. 146, 90461 Nürnberg
HRB 36809 (AG Nürnberg), GF: I. Totev, A. McDonald, W. Knoblich

  reply	other threads:[~2026-11-16  7:52 UTC|newest]

Thread overview: 27+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-11-14  3:04 [PATCH v7 00/17] TP8028 Rapid Path Failure Recovery Mohamed Khalfella
2026-11-14  3:04 ` [PATCH v7 01/17] nvmet: Rapid Path Failure Recovery set controller identify fields Mohamed Khalfella
2026-11-14  3:04 ` [PATCH v7 02/17] nvmet/debugfs: Export controller CIU and CIRN via debugfs Mohamed Khalfella
2026-11-14  3:04 ` [PATCH v7 03/17] nvmet: Implement CCR nvme command Mohamed Khalfella
2026-11-16  8:45   ` Dmitry Bogdanov
2026-11-14  3:04 ` [PATCH v7 04/17] nvmet: Implement CCR logpage Mohamed Khalfella
2026-11-14  3:04 ` [PATCH v7 05/17] nvmet: Send an AEN on CCR completion Mohamed Khalfella
2026-11-14  3:04 ` [PATCH v7 06/17] nvme: Rapid Path Failure Recovery read controller identify fields Mohamed Khalfella
2026-11-14  3:04 ` [PATCH v7 07/17] nvme: Introduce FENCING and FENCED controller states Mohamed Khalfella
2026-11-16  7:16   ` Hannes Reinecke
2026-11-14  3:04 ` [PATCH v7 08/17] nvme: Implement cross-controller reset recovery Mohamed Khalfella
2026-11-14  3:04 ` [PATCH v7 09/17] nvme: Implement cross-controller reset completion Mohamed Khalfella
2026-11-16  7:17   ` Hannes Reinecke
2026-11-14  3:04 ` [PATCH v7 10/17] nvme-tcp: Use CCR to recover controller that hits an error Mohamed Khalfella
2026-11-14  3:04 ` [PATCH v7 11/17] nvme-rdma: " Mohamed Khalfella
2026-11-16  7:19   ` Hannes Reinecke
2026-11-14  3:04 ` [PATCH v7 12/17] nvme-fc: start error recovery instead of aborting timed out IOs Mohamed Khalfella
2026-11-16  7:31   ` Hannes Reinecke
2026-11-14  3:04 ` [PATCH v7 13/17] nvme-fc: perform error recovery directly from ioerr_work Mohamed Khalfella
2026-11-16  7:37   ` Hannes Reinecke
2026-11-14  3:04 ` [PATCH v7 14/17] nvme-fc: Use CCR to recover controller that hits an error Mohamed Khalfella
2026-11-16  7:38   ` Hannes Reinecke
2026-11-14  3:04 ` [PATCH v7 15/17] nvme-fc: Hold inflight requests while in FENCING state Mohamed Khalfella
2026-11-16  7:52   ` Hannes Reinecke [this message]
2026-11-14  3:04 ` [PATCH v7 16/17] nvmet: Add support for CQT to nvme target Mohamed Khalfella
2026-11-14  3:04 ` [PATCH v7 17/17] nvme: Add support for CQT to nvme host Mohamed Khalfella
2026-11-16  7:55   ` Hannes Reinecke

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=508eed68-2633-45dd-b503-65c62b5fe16c@suse.de \
    --to=hare@suse.de \
    --cc=axboe@kernel.dk \
    --cc=dwagner@suse.de \
    --cc=hare@kernel.org \
    --cc=hch@lst.de \
    --cc=jsmart833426@gmail.com \
    --cc=justin.tee@broadcom.com \
    --cc=kbusch@kernel.org \
    --cc=kch@nvidia.com \
    --cc=linux-kernel@vger.kernel.org \
    --cc=linux-nvme@lists.infradead.org \
    --cc=mkhalfella@purestorage.com \
    --cc=nareshgottumukkala83@gmail.com \
    --cc=paul.ely@broadcom.com \
    --cc=randyj@purestorage.com \
    --cc=sagi@grimberg.me \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox

all inboxes | Powered by JetHome®