mirror of https://lore.kernel.org/lkml/
 help / color / mirror / Atom feed
* Re: [PATCH net-next v8 3/4] net: mana: support concurrent HWC requests
@ 2026-10-20 13:36 netdev-bot+sashiko
  0 siblings, 0 replies; 2+ messages in thread
From: netdev-bot+sashiko @ 2026-10-20 13:36 UTC (permalink / raw)
  To: weh
  Cc: longli, kotaranov, kuba, davem, pabeni, edumazet, andrew+netdev,
	jgg, leon, haiyangz, wei.liu, decui, shradhagupta, horms, ernis,
	stephen, netdev, linux-rdma, linux-hyperv, linux-kernel,
	gargaditya, kees+treewide, kees, mawasthi, weh, edumazet

Thank you for your contribution! Sashiko AI review found 2 potential
issue(s) to consider.

Critical: 0 · High: 1 · Medium: 1 (1) · Low: 0

If you believe none of the issues are legitimate, please reply to each
issue of Medium severity or higher and explain why the AI is wrong.
If any of them are legitimate and you plan to respin, please let patchwork
know by including "pw-bot: cr" as a separate line at the end of your reply
(one such reply per series is enough).

- [High] The patch replaces the unbounded down(&hwc->sema) with
  down_timeout(..., hwc_timeout).
- [Medium] mana_hwc_get_msg_index() replaces the unbounded down() with
  down_timeout().

Pre-existing issues:
- [Medium] This problem wasn't introduced by this patch, but
  mana_hwc_rx_event_handler() returns without reposting the RX WQE when…

--

net: mana: support concurrent HWC requests

This patch gives each HWC message slot separate sender and response
references, serializes SQ posting, and replaces the unbounded admission
down() with down_timeout(), which returns -EBUSY or -ETIMEDOUT. It also
publishes the channel under a new hwc_lock, drains active senders on
teardown, and moves hwc_timeout updates to atomic helpers.

Link: https://lore.kernel.org/r/20260908035201.402424-4-longli@microsoft.com
Link: https://lore.kernel.org/r/20260914175044.2a26bb46@kernel.org
Link: https://lore.kernel.org/r/179196575640.434549.9102950927041409151@kernel.org

> diff --git a/drivers/net/ethernet/microsoft/mana/hw_channel.c b/drivers/net/ethernet/microsoft/mana/hw_channel.c
> index 03a3a0e9d3f84..9ea67c076ebda 100644
> --- a/drivers/net/ethernet/microsoft/mana/hw_channel.c
> +++ b/drivers/net/ethernet/microsoft/mana/hw_channel.c
> @@ -6,34 +6,105 @@
>  #include <net/mana/hw_channel.h>
>  #include <linux/vmalloc.h>
>  
> -static int mana_hwc_get_msg_index(struct hw_channel_context *hwc, u16 *msg_id)
> +u32 mana_hwc_timeout_read(const struct hw_channel_context *hwc)

[ ... ]

> +static int mana_hwc_get_msg_index(struct hw_channel_context *hwc, void *resp,
> +				  u32 resp_len,
> +				  struct hwc_caller_ctx **caller_ctx)
>  {
>  	struct gdma_resource *r = &hwc->inflight_msg_res;
>  	struct hwc_caller_ctx *ctx;
>  	unsigned long flags;
> +	u32 wait_ms;
>  	u32 index;
> +	int err;
>  
> -	down(&hwc->sema);
> +	wait_ms = mana_hwc_timeout_read(hwc);
> +	if (down_timeout(&hwc->sema, msecs_to_jiffies(wait_ms))) {

[Severity: High]
Before this patch, admission could only block here. Every command that
reached mana_hwc_send_request() was eventually posted. With
down_timeout(), a channel that is healthy but busy can now return -EBUSY
without ever sending the command.

At HW_CHANNEL_VF_BOOTSTRAP_QUEUE_DEPTH of 1, every sender shares one
permit in FIFO order. That includes RDMA verbs that users can reach
through mana_ib. The wait is hwc_timeout, which stays at 1 ms for good
after any mana_hwc_timeout_reduce().

Several teardown callers do not handle a command that was never sent.
mana_gd_destroy_eq() ignores the result:

	if (queue->eq.disable_needed)
		mana_gd_disable_queue(queue);

mana_gd_create_hw_eq() has already set dma_region_handle to
GDMA_INVALID_DMA_REGION, so the later mana_gd_destroy_dma_region() call
in mana_gd_destroy_queue() cannot replace the missed disable.
mana_gd_destroy_queue() then calls mana_gd_free_memory() without
condition. mana_destroy_wq_obj() returns void, so its callers free the
WQ/CQ memory either way.

If GDMA_DISABLE_QUEUE, GDMA_DESTROY_DMA_REGION or MANA_DESTROY_WQ_OBJ is
rejected at admission like this, can the device, which is still healthy,
keep writing EQEs/CQEs into pages the kernel has already freed and
reused?

> +		spin_lock_irqsave(&r->lock, flags);
> +		if (!hwc->channel_up)
> +			err = -ENODEV;
> +		else if (atomic_read(&hwc->abandoned_msgs))
> +			err = -ETIMEDOUT;
> +		else
> +			err = -EBUSY;

[Severity: Medium]
Can this choose -EBUSY when the channel is hung? Only the permit holder
raises abandoned_msgs, in mana_hwc_wait_for_response(), and only after
its own completion wait has expired:

mana_hwc_wait_for_response() {
    ...
	if (err == -EINPROGRESS) {
		ctx->error = -ETIMEDOUT;
		ctx->responded = true;
		atomic_inc(&hwc->abandoned_msgs);
    ...
}

The bootstrap depth gives one permit. Suppose a waiter enters
down_timeout() after holder A takes the permit but before A enters
wait_for_completion_timeout(), or both timers expire in the same jiffy.
The waiter then reads abandoned_msgs == 0 and returns -EBUSY, even
though the device is not responding.

There is a second case after mana_serv_reset() calls
mana_gd_cancel_hwc_timeout(). A request posted before the cancel still
holds the permit with its old wait. The suspend-path commands would then
fail at once in down_timeout(0) with -EBUSY. Before, they blocked in
down() and failed quietly with -ETIMEDOUT.

The existing consumers check for -ETIMEDOUT.

mana_need_log() only suppresses -ETIMEDOUT on a published hwc whose
timeout is 0. Two new errors slip past it:

  - the -EBUSY returned here
  - the -ENODEV that mana_hwc_stop_channel() sets after
    mana_hwc_unpublish_channel() has cleared gc->hwc.driver_data

Both would now print dev_err() messages on reset and teardown paths,
for example from mana_gd_deregister_device().

mana_gd_probe() schedules recovery only for -ETIMEDOUT or -EPROTO, and
mana_serv_reset() rescans after a failed resume only for those two
codes:

	if (err == -ETIMEDOUT || err == -EPROTO) {

If a probe or resume gets -EBUSY from a hung HWC, will it fail with no
recovery scheduled?

> +		spin_unlock_irqrestore(&r->lock, flags);
> +
> +		return err;
> +	}

[ ... ]

> @@ -90,40 +183,53 @@ static void mana_hwc_handle_resp(struct hw_channel_context *hwc, u32 resp_len,
>  				 struct hwc_work_request *rx_req, u16 msg_id)
>  {
>  	const struct gdma_resp_hdr *resp_msg = rx_req->buf_va;
> +	struct gdma_resource *r = &hwc->inflight_msg_res;
>  	struct hwc_caller_ctx *ctx;
> +	bool release;
>  	int err;
>  
> -	if (!test_bit(msg_id, hwc->inflight_msg_res.map)) {
> +	spin_lock(&r->lock);
> +	if (!test_bit(msg_id, r->map)) {
> +		spin_unlock(&r->lock);
>  		dev_err(hwc->dev, "hwc_rx: invalid msg_id = %u\n", msg_id);
>  		mana_hwc_post_rx_wqe(hwc->rxq, rx_req);
>  		return;
>  	}

[Severity: Medium]
This is a pre-existing issue and was not introduced by this patch. This
rejection path reposts rx_req, but the range check before it in
mana_hwc_rx_event_handler() does not:

mana_hwc_rx_event_handler() {
    ...
	msg_id = READ_ONCE(resp->response.hwc_msg_id);
	if (msg_id >= hwc->num_inflight_msg) {
		dev_err(hwc->dev, "HWC RX: wrong msg_id=%u\n", msg_id);
		return;
	}
    ...
}

Does each response with an out-of-range hwc_msg_id lose one RQ buffer
for good? At depth 1, a single such response from the device or a
hostile CVM host would leave the RQ empty. Every later response could
then not be delivered until a reset.

[ ... ]

-- 
Sashiko AI review · https://netdev-ai.bots.linux.dev/sashiko/#/patchset/cover.1791998462.git.weh%40linux.microsoft.com

^ permalink raw reply	[flat|nested] 2+ messages in thread
* [PATCH net-next v8 0/4] net: mana: concurrent HWC requests and dynamic queue depth
@ 2026-10-16 12:06 Wei Hu
  2026-10-16 12:06 ` [PATCH net-next v8 3/4] net: mana: support concurrent HWC requests Wei Hu
  0 siblings, 1 reply; 2+ messages in thread
From: Wei Hu @ 2026-10-16 12:06 UTC (permalink / raw)
  To: longli, kotaranov, kuba, davem, pabeni, edumazet, andrew+netdev,
	jgg, leon, haiyangz, wei.liu, decui, shradhagupta, horms, ernis,
	stephen
  Cc: netdev, linux-rdma, linux-hyperv, linux-kernel, gargaditya,
	kees+treewide, kees, mawasthi, weh, edumazet

This series adds concurrent HWC request ownership and negotiated HWC
queue depth to MANA. It starts from depth-one bootstrap queues, keeps
setup private until the final channel is tested, and preserves the
distinction between CPU lifetime and PF ownership of DMA mappings.

I am continuing Long Li's four-patch series. Original per-patch author
identity, dates and sign-offs are retained.

Changes in v8 (v7 -> v8):
 - Rebase onto net-next 05b5019b01fb9443b09ae4ed34af1135a80a84c8.
 - Withdraw the table and drain RCU readers before freeing the HWC
   bundle, in patch 1. Add final PCI/CDX software-only disposal of
   retained state without reclaiming unacknowledged PF-owned DMA.
 - Initialize caller state under the per-slot lock. Do not introduce
   a temporary channel-wide timeout latch in intermediate patch 2.
 - Preserve recovery signaling when abandoned responses exhaust
   admission, while healthy saturation remains ordinary pressure.
 - Include the candidate functional test in bounded bootstrap fallback
   and preserve a live retry budget without reviving cancellation.
 - Share the effective TX budget with GDMA/RDMA request producers;
   bound RDMA batches without changing RX allocation/indexing strides.
 - Preserve stopped-channel errors before admission and fail malformed
   doorbell initialization without depending on a later INIT_DONE.
 - Correct cleanup comments and commit-message rationale. Review both
   Sashiko report streams rather than treating either as exhaustive.

This series remains independent of queue-set replacement. That work
addresses individual Ethernet CQ publication/retirement. The shared
CQ-table hunks must be reconciled while preserving both contracts,
whichever series lands second. Neither series is silently stacked.
https://lore.kernel.org/all/cover.1790795005.git.weh@linux.microsoft.com/

Known separate concerns include individual RDMA/datapath CQ lifetime,
malformed-response RX reposting and wider DMA failure/device-lifecycle
policy. This revision does not claim to fix them. Standalone net/stable
backports of older-kernel fixes require separate extraction/validation.

Changes in v7 (v6 -> v7):
 - Rebase onto net-next d8674294aefef02266c4d47ad10131f1bffbe534,
   following Simon Horman's request after the v6 CI application failure.
 - Preserve the upstream GDMA core/PCI transport split and CDX support.
   Initialize the HWC lifetime lock on both transport probe paths and
   perform PCI recovery's timeout cancellation through a GDMA core helper
   which protects the published HWC pointer across the module boundary.
 - Retain the four-patch lineage and the v6 ownership, request, timeout,
   publication, depth-validation and fallback policies.

Rebase request:
https://lore.kernel.org/all/20261009151906.GF83879@horms.kernel.org/

Changes in v6 (v5 -> v6):
 - Keep the four-patch v5 structure while folding review-required HWC
   ownership, teardown, timeout, publication and fallback safety into
   their owning feature patches.
 - Serialize complete ESTABLISH_HWC and DESTROY_HWC shared-memory
   transactions. Record possible PF ownership before MMIO submission and
   clear it only after DESTROY_HWC is acknowledged.
 - Retain PF-visible queues after an indeterminate destroy. Fence the
   dedicated HWC EQ and withdraw its CQ from dispatch without freeing the
   mappings, then retry teardown before assigning fresh queues.
 - Destroy the HWC EQ before CQ and completion state. Release-publish the
   CQ table and wait for IRQ-side RCU readers before freeing it, without
   adding per-completion refcount operations.
 - Withdraw the caller's response buffer before return and quarantine a
   timed-out slot until its late response arrives or teardown cancels it.
 - Return -EBUSY for healthy slot pressure and reserve -ETIMEDOUT for an
   admitted request which receives no response.
 - Publish slots under the inflight lock, protect response state with the
   per-slot lock, and retain sender/response references until both sides
   finish.
 - Synchronize hwc_timeout updates and publish the HWC only after queues,
   admission state, caller contexts and the direct EQ test are ready.
 - Replace the goto-heavy request and channel-creation flows called out in
   review with focused admission, submission, wait, build, validation,
   rebuild and fallback helpers.
 - Treat reported depth as a maximum: rebuild exactly for 2 through 128,
   retain untouched bootstrap queues for 1 or above 128, and require the
   rebuilt report to repeat the selected depth.
 - Enforce directional request/response maxima against both the report
   and allocated buffers. Rebuild only for exact 4096-byte reports while
   permitting bootstrap operation for other nonzero maxima.
 - Validate BAR page geometry and every stored doorbell. Invalidate the
   old HWC doorbell before re-establishment and suppress MMIO while the
   invalid sentinel is present, without adding full validation to each
   data-path doorbell ring.
 - Remove the proposed capability bit because driver-version
   advertisement occurs after HWC establishment and cannot negotiate
   creation-time behavior.
 - Preserve current net-next behavior which always uses the
   hardware-reported dest_vrq_id and dest_vrcq_id.

Changes in v5 (v4 -> v5):
 - Shortened comments and documented the existing error paths in patches
   1 and 2.
 - Rejected apparent success from a response accepted before request
   submission.
 - Guarded CQ unpublication when establishment failed before cq_table
   allocation.
 - Required successful retry teardown before creating fresh bootstrap
   queues after an initial destroy failure.

Changes in v4 (return to net-next after the v3 split):
 - Reworked the feature series into four standalone patches for handover,
   per-slot completion state, concurrency and dynamic depth.
 - Added bounded admission, guarded sender accounting, timeout quarantine,
   queue-depth bounds, message-buffer sizing and rebuild validation.
 - Reset routing IDs for each establish and rejected a missing doorbell.

Changes in v3 (v2 -> net v3, historical fixes-only posting):
 - Split the combined series after maintainer feedback, posting fixes to
   net and deferring concurrency and dynamic depth.
 - Added response-buffer withdrawal and stale-response handling.
 - Removed pcie_flr() teardown fallback and retained resources after an
   unacknowledged destroy.

Changes in v2 (v1 -> v2):
 - Bounds-checked the SGE pointer derived from inline OOB size.
 - Protected active-sender accounting and drained senders during
   teardown and failure exits.
 - Rebased on net-next e354f7d60f14.

v1:
 - Combined CQ-table lifetime, queue sizing, RX validation, teardown,
   concurrent-request and dynamic-depth work in seven patches.

Validation:
Source and build qualification:
Per-intermediate GCC/Clang W=1 builds, patch style checks, focused
actual-source C cases and paired full kernel/image/module qualification
are complete. Sparse retains a baseline warning; Smatch retains baseline
frontend diagnostics and is not counted as a clean pass.

Ordinary baseline/release coverage on VM5 and VM6 is complete.
Each VM/fixture has 1,600 type-2 PD/MR and 320 type-3 PD/CQ/MR operations
across the initial and three post-PCI rounds, native channels 8->4->8,
native/upper MTUs 1500->1400->1500, three PCI remove/rescan cycles,
bounded ping and three-second TCP/1Mbit UDP with positive receivers.
Both VMs returned to their exact SAFE baselines. Owned fixtures/helpers
and processes were removed; guest and central reservations were released.

Evidence is explicitly composed, not one uninterrupted four-trial run.
The completed VM5 baseline uses retained exact runtime receipts and a
corrected evaluation of iperf3's duplicate, identical UDP bitrate metadata.
That original harness attempt remains PROFILE_FAILED in its history.
VM5 release and both VM6 fixtures completed in a new three-trial run.
Fixture identity and SAFE continuity are checked across the two scopes.

No measured simultaneous outstanding firmware-command count, negotiated
depth proof, performance comparison, connected RDMA workload, CDX hardware,
fault injection, sanitizer, DIM or queue-set qualification is claimed.

The final intermediate-history correction leaves the entire final
source tree identical to the release build snapshot a1454f411cdc.
Evidence retains the literal build commit and kernelrelease rather
than relabeling an image with a later history-only commit ID.

v7:
https://lore.kernel.org/all/cover.1791561995.git.weh@linux.microsoft.com/
v6:
https://lore.kernel.org/all/cover.1790665894.git.weh@linux.microsoft.com/
v5:
https://lore.kernel.org/all/20260908035201.402424-1-longli@microsoft.com/
v4:
https://lore.kernel.org/all/20260901200018.3194525-1-longli@microsoft.com/
v3:
https://lore.kernel.org/all/20260803234355.636038-1-longli@microsoft.com/
v2:
https://lore.kernel.org/all/20260721234339.1476932-1-longli@microsoft.com/
v1:
https://lore.kernel.org/all/20260715032942.3945317-1-longli@microsoft.com/

Long Li (4):
  net: mana: prepare HWC ownership for safe reinitialization
  net: mana: give each HWC message slot its own completion state
  net: mana: support concurrent HWC requests
  net: mana: add dynamic HWC queue depth with reinit path

 drivers/infiniband/hw/mana/main.c             |   19 +-
 .../net/ethernet/microsoft/mana/gdma_cdx.c    |    5 +-
 .../net/ethernet/microsoft/mana/gdma_main.c   |  154 ++-
 .../net/ethernet/microsoft/mana/gdma_pci.c    |   15 +-
 .../net/ethernet/microsoft/mana/hw_channel.c  | 1019 ++++++++++++++---
 .../net/ethernet/microsoft/mana/shm_channel.c |   58 +-
 include/net/mana/gdma.h                       |   17 +
 include/net/mana/hw_channel.h                 |   69 +-
 include/net/mana/shm_channel.h                |   10 +-
 9 files changed, 1150 insertions(+), 216 deletions(-)


base-commit: 05b5019b01fb9443b09ae4ed34af1135a80a84c8
-- 
2.43.0


^ permalink raw reply	[flat|nested] 2+ messages in thread
end of thread, other threads:[~2026-10-20 13:36 UTC | newest]

Thread overview: 2+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2026-10-20 13:36 [PATCH net-next v8 3/4] net: mana: support concurrent HWC requests netdev-bot+sashiko
  -- strict thread matches above, loose matches on Subject: below --
2026-10-16 12:06 [PATCH net-next v8 0/4] net: mana: concurrent HWC requests and dynamic queue depth Wei Hu
2026-10-16 12:06 ` [PATCH net-next v8 3/4] net: mana: support concurrent HWC requests Wei Hu

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®