Re: [PATCH net 3/3] enic: require DEVCMD2 and set the V2 VF mailbox source

Re: [PATCH net 3/3] enic: require DEVCMD2 and set the V2 VF mailbox source

[netdev-bot+sashiko]

Thank you for your contribution! Sashiko AI review found 1 potential
issue(s) to consider.

Critical: 0 · High: 0 · Medium: 1 · Low: 0

If you believe none of the issues are legitimate, please reply to each
issue of Medium severity or higher and explain why the AI is wrong.
If any of them are legitimate and you plan to respin, please let patchwork
know by including "pw-bot: cr" as a separate line at the end of your reply
(one such reply per series is enough).

- [Medium] The fence for the statistics-command DMA is cleared only when
  the command succeeds, not whenever it completes.

--

enic: require DEVCMD2 and set the V2 VF mailbox source

This commit has V2 VFs query CMD_SRIOV_STATS_GET after opening the admin
channel. The returned logical VF index becomes the mailbox src_vnic_id,
and SR-IOV V2 now requires DEVCMD2. A timeout of the statistics command
is tracked with sriov_stats_dma_unresolved. A later CMD_QP_TYPE_SET is
expected to fence it, and if nothing does, the probe resources are kept
until reboot.

> diff --git a/drivers/net/ethernet/cisco/enic/enic.h b/drivers/net/ethernet/cisco/enic/enic.h
> index cf2dee8fa78ff..886f3cba543dc 100644
> --- a/drivers/net/ethernet/cisco/enic/enic.h
> +++ b/drivers/net/ethernet/cisco/enic/enic.h
> @@ -273,6 +273,11 @@ struct enic {
>  	u16 num_vfs;
>  	enum enic_vf_type vf_type;
>  	bool vf_registered;
> +	u16 vf_id;
> +	/* A CMD_SRIOV_STATS_GET timeout leaves its persistent coherent target
> +	 * device-owned until a later serialized command completes.
> +	 */
> +	bool sriov_stats_dma_unresolved;
>  	u32 pf_cap_version;
>  	unsigned int enable_count;
>  	spinlock_t enic_api_lock;
> diff --git a/drivers/net/ethernet/cisco/enic/enic_admin.c b/drivers/net/ethernet/cisco/enic/enic_admin.c
> index a5c08cbcb3391..2f6688a73b41a 100644
> --- a/drivers/net/ethernet/cisco/enic/enic_admin.c
> +++ b/drivers/net/ethernet/cisco/enic/enic_admin.c
> @@ -406,6 +406,12 @@ static int enic_admin_qp_type_set(struct enic *enic, u32 enable)
>  
>  	spin_lock_bh(&enic->devcmd_lock);
>  	err = vnic_dev_cmd(enic->vdev, CMD_QP_TYPE_SET, &a0, &a1, wait);
> +	/* DEVCMD2 runs one command at a time for a vNIC. Completion of this
> +	 * blocking command proves that earlier statistics-command DMA has
> +	 * finished. Admin WQ/RQ ownership still requires RUNNING == 0.
> +	 */
> +	if (!err)
> +		WRITE_ONCE(enic->sriov_stats_dma_unresolved, false);

[Severity: Medium]
Should sriov_stats_dma_unresolved be cleared whenever CMD_QP_TYPE_SET
completes, not only when it returns 0?

The comment above, the enic.h field comment ("until a later serialized
command completes") and the commit message all treat completion as the
fence. On the DEVCMD2 path, _vnic_dev_cmd2() returns -result->error only
after firmware has written this command's result slot:

drivers/net/ethernet/cisco/enic/vnic_dev.c:_vnic_dev_cmd2() {
    ...
	for (delay = 0; delay < wait; delay++) {
		if (result->color == color) {
			if (result->error) {
				err = result->error;
				...
				return -err;
			}
    ...
}

Only -ETIMEDOUT leaves completion unknown after the command is posted.
-ENODEV and -EBUSY are returned before anything is posted.

In enic_probe(), a timeout in enic_dev_sriov_stats() sets the flag, and
the unwind goes to err_out_admin_close. There, enic_admin_channel_close()
calls enic_admin_qp_type_set(QP_DISABLE). If firmware completes that
command with an error status, the flag stays set. This happens even
though enic_admin_channel_close() treats the rejection as harmless:

	/* Firmware rejection does not undermine the WQ/RQ proof. */
	return qp_err;

At err_out_admin_fence, enic_dma_quarantined() still returns true. Probe
then calls enic_terminal_dma_quarantine() and returns early, skipping
enic_dev_deinit(), vnic_dev_close(), vnic_dev_unregister(),
enic_iounmap(), pci_release_regions() and free_netdev().

Would that keep the netdev, vdev, coherent buffers, BAR mappings and PCI
regions held until reboot, and prevent the VF from being probed again?

If clearing only on success is intended, could the comments and the
commit message say so?

>  	spin_unlock_bh(&enic->devcmd_lock);
>  
>  	return err;

[ ... ]

-- 
Sashiko AI review · https://netdev-ai.bots.linux.dev/sashiko/#/patchset/20261016-b4-enic-dma-vf-index-net-generic-first-v1-20261010-v1-0-d27b55f69883%40cisco.com

[PATCH net 0/3] enic: fence DMA ownership and identify V2 VFs

[Satish Kharat]

An automated Sashiko review of the V2 SR-IOV reset paths prompted a
broader audit of ENIC queue teardown. That audit found three prerequisite
correctness problems in code which is already upstream.

First, a datapath WQ or RQ disable can time out while hardware may still
own descriptors and their DMA mappings. Patch 1 replaces iteration-counted
polling with a bounded 500 ms RUNNING poll for each queue. The RQ keeps its
required two disable-and-poll passes. The datapath graph is released only
after every applicable queue reports RUNNING == 0. Because rings, mapped
buffers, page pools, and shared state form a coupled ownership graph, it is
not partially released when only some queues stop.

The queue-disable helpers can return -ETIMEDOUT without observing
RUNNING == 0. Firmware defines RUNNING == 0 as conclusive evidence that
queue DMA has stopped. A nonzero RUNNING value does not establish that
DMA is active, but it leaves DMA ownership unresolved. The existing error
path can nevertheless release DMA-backed resources without obtaining the
required ownership proof.

This series deliberately retains the complete datapath graph if any
datapath queue does not report RUNNING == 0. This can retain resources
belonging to queues which did stop, but keeps the ownership rule simple
in an exceptional failure path. Reclaiming packet buffers individually for
queues which successfully stop is a possible follow-up optimization; it
is not required to prevent DMA into freed memory.

If a queue does not report RUNNING == 0, the whole datapath graph is
quarantined. A final direct queue recheck runs during device removal,
before terminal containment. The graph is released if every queue reports
RUNNING == 0 by then; otherwise it is retained until reboot. Firmware soft
reset, hang reset, and PCI function reset are not treated as independent
ownership proof for a queue whose stop could not be confirmed.

The terminal path detaches CPU entry points, clears bus mastering, disables
the PCI function, and intentionally retains the netdev, vnic_dev, BAR
mappings and reservations, rings, mapped buffers, page pools, and graph
metadata until reboot. Clearing bus mastering blocks new device-initiated
PCI transactions; it does not cancel DMA already in progress or prevent a
queue from draining to RUNNING == 0. The final remove-time recheck
therefore remains meaningful, but bus-master state alone does not prove
that a queue with unresolved DMA ownership relinquished its old targets.
The BAR reservation may outlive driver unbind and module unload while the
PCI device remains enumerated. The resource subsystem stores the supplied
owner-name pointer rather than copying it, so the reservation uses the PCI
device name instead of module rodata.

A PCI driver's remove callback cannot reject driver unbind. With DMA API
debugging enabled, the driver's deliberate retention therefore produces the
expected "pending DMA allocations while released from device" diagnostic
if the module is unloaded after terminal quarantine. The retained BAR
reservation prevents a normal ENIC rebind; reboot is the supported
recovery.

Ring resizing needs an explicit stop because dev_close() cannot propagate
an ndo_stop() error. Patch 1 returns a queue-disable failure to ethtool and
rejects later reconfiguration after quarantine. It also makes reset, IRQ,
NAPI, notification-buffer, and final-remove ownership transitions
idempotent.

Second, an admin-WQ send can time out while hardware may still own the
descriptor and mapping. Patch 2 applies the same bounded polling,
observed-queue-stop rule, and final remove-time recheck to the admin WQ/RQ
graph. The datapath and admin channel are independent complete graphs: a
clean graph may be released even when the other is quarantined, but an
individual graph is never partially released. Shared objects needed by an
unresolved DMA target remain retained. Putting the generic repair first
makes every committed state independently safe.

Third, the in-tree V2 VF always transmits mailbox source zero. Deployed
async PFs compare that field with sender identity from CQ metadata, so a
nonzero VF cannot register. Patch 3 obtains firmware's logical VF index
with CMD_SRIOV_STATS_GET after the admin channel is live and uses it as the
source of subsequent V2 VF mailbox messages.

Patch 3 also requires DEVCMD2 for SR-IOV V2. A V2 VF without DEVCMD2
fails probe before opening its admin channel. A PF which fell back to
DEVCMD1 keeps ordinary Ethernet operation but does not expose V2 SR-IOV;
V1 and USNIC keep their existing DEVCMD1 fallback. With DEVCMD2, -ENODEV
is detected before a descriptor is posted, so only a timeout can leave
ownership of the persistent statistics-command DMA target uncertain. A
later successful blocking command for the same vNIC proves that target
completed; otherwise it remains retained with the shared vNIC lifetime
envelope until reboot.

The companion V2 VF receive-control v3 series carries explicit prerequisite
patch IDs for all three patches.

AI assistance:

An LLM was used for design review, source review, commit-message drafting,
test automation, and review triage. Sashiko was used as an additional
review tool. All findings and generated changes were manually reviewed, and
the author takes responsibility for the series.

Validation:

- Tests used Cisco VIC PF and V2 VF functions. PF/VF probe, open/close,
  repeated clean module unload/reload, link cycling, ring resizing, jumbo
  MTU, and bidirectional ICMP, TCP, and UDP traffic passed.
- A nonzero V2 VF registered and passed traffic with the deployed async PF.
  Before patch 3, the PF rejected the VF's zero mailbox source because it
  did not match the sender identity obtained from CQ metadata.
- The full no-reboot suite exercised exactly 32 secondary unicast
  addresses, 33-address fallback, address churn, station-address
  multicast-list changes, administrative-MAC policy, 25 module reloads,
  10 ring resize cycles, 10 reopen cycles, and five VF destroy/recreate
  cycles.
- A test-only admin-channel overlay injected one false first-attempt WQ
  disable failure and then one false first-attempt RQ disable failure after
  the corresponding real hardware disable had succeeded. Each failed helper
  alone was retried, each retry succeeded, and the production module
  then passed bidirectional traffic.
- A test-only V2 VF overlay forced exactly one posted mailbox send timeout.
  Serialized reset recovery rebuilt the admin channel, refreshed the
  administrative MAC, restored bidirectional traffic, and completed a later
  exact-filter transaction.
- A disposable-VF-guest test overlay held the selected WQ or RQ's observed
  RUNNING value high without preventing the real hardware disable. A 300 ms
  hold verified that both queue types wait rather than time out early. A
  hold through the 500 ms bound quarantined each datapath and admin queue
  path. Clearing the synthetic condition before removal let the final
  recheck release each complete graph. Leaving it set exercised terminal
  retention. The uninstrumented candidate then bound and passed
  bidirectional ENIC traffic after guest reboot.

Queue-stop failure handling was exercised using instrumentation which
kept the RUNNING value observed by the driver nonzero after the underlying
hardware queue had stopped. This exercised the bounded polling,
quarantine, retained-resource, and final remove-time recheck paths. It
validates the driver's handling when RUNNING == 0 is not observed; it does
not attempt to reproduce or characterize the underlying hardware condition
which produces that result. If ownership remains unresolved at removal,
reboot is the supported recovery.

The DEVCMD2 requirement and pre-admin-channel V2 VF rejection were
verified by source review and exact per-commit builds. The physical test
adapter uses DEVCMD2, so the no-DEVCMD2 rejection path was not exercised
on hardware.

Link: https://patch.msgid.link/20260929-b4-enic-sriov-v2-vf-receive-control-v2-0-5962c10061b2@cisco.com
Signed-off-by: Satish Kharat <satishkh@cisco.com>
---
Satish Kharat (3):
      enic: retain datapath DMA resources until queues stop
      enic: retain admin-channel DMA until queues stop
      enic: require DEVCMD2 and set the V2 VF mailbox source

 drivers/net/ethernet/cisco/enic/enic.h         |  36 +-
 drivers/net/ethernet/cisco/enic/enic_admin.c   | 182 ++++++---
 drivers/net/ethernet/cisco/enic/enic_admin.h   |   4 +-
 drivers/net/ethernet/cisco/enic/enic_dev.c     |  15 +
 drivers/net/ethernet/cisco/enic/enic_dev.h     |   2 +
 drivers/net/ethernet/cisco/enic/enic_ethtool.c |  10 +-
 drivers/net/ethernet/cisco/enic/enic_main.c    | 508 +++++++++++++++++++++----
 drivers/net/ethernet/cisco/enic/enic_mbox.c    |  13 +-
 drivers/net/ethernet/cisco/enic/vnic_dev.c     |  42 ++
 drivers/net/ethernet/cisco/enic/vnic_dev.h     |   5 +
 drivers/net/ethernet/cisco/enic/vnic_devcmd.h  |  34 ++
 drivers/net/ethernet/cisco/enic/vnic_rq.c      |  18 +-
 drivers/net/ethernet/cisco/enic/vnic_wq.c      |  18 +-
 13 files changed, 735 insertions(+), 152 deletions(-)
---
base-commit: 6d25ffca055a77787c21a36b66c253f76239411b
change-id: 20261010-b4-enic-dma-vf-index-net-generic-first-v1-20261010-d9abf4510335

Best regards,
--  
Satish Kharat <satishkh@cisco.com>

[PATCH net 3/3] enic: require DEVCMD2 and set the V2 VF mailbox source

[Satish Kharat]

enic_mbox_fill_hdr() clears the mailbox header, leaving src_vnic_id zero.
That is correct only for VF index zero.

The deployed async PF compares the mailbox source with sender identity from
CQ metadata and rejects mismatches. A nonzero VF therefore cannot complete
mailbox registration when the in-tree VF always sends source zero.

CMD_SRIOV_STATS_GET is part of the deployed V2 contract and returns this
function's logical VF index. Query it after opening the admin channel but
before the first outbound mailbox request. If the command is unavailable
or fails, fail probe cleanly rather than sending an ambiguous source.
Annotate its multibyte DMA fields as little endian, order reads after
command completion, convert and snapshot the version and VF index once,
validate version 1, and reject the PF sentinel.

Require DEVCMD2 for SR-IOV V2. A V2 VF without DEVCMD2 fails probe before
opening its admin channel. A PF which has fallen back to DEVCMD1 keeps its
normal Ethernet function but does not expose V2 SR-IOV. Legacy V1 and USNIC
operation retain their existing DEVCMD1 fallback.

DEVCMD2 returns -ENODEV only while checking the command queue, before it
posts a descriptor. A timeout after posting is therefore the only error
which leaves ownership of the statistics-command DMA target uncertain.
DEVCMD2 runs one command at a time for a vNIC, and the statistics command
is completed only after the DMA write to its target finishes. Completion
of a later blocking command for the same vNIC therefore fences that target.
In the current probe unwind, QP_TYPE_SET during admin-channel teardown is
that fence. This is separate from WQ/RQ ownership: queue resources still
require their own RUNNING == 0 observations.

Keep the persistent target allocated while its ownership is uncertain. If
no later command establishes the completion fence, retain the target and
the shared vNIC lifetime envelope until reboot. Clearing bus mastering
contains the function but does not itself prove that already-issued DMA
has completed.

Use the firmware VF index as the little-endian source of subsequent VF
messages. Initialize the stored index to the protocol PF sentinel so an
unexpected message before the statistics command completes cannot use zero
as an apparently valid VF0 identity. PF and non-V2 senders use the protocol
PF sentinel directly rather than the old implicit zero.

The defect was found during interoperability testing of the in-tree V2 VF
against the deployed async PF: VF0 could register, while a nonzero VF's
source field disagreed with the sender index derived by the PF from CQ
metadata.

The statistics-command identity, nonzero-VF registration, and mailbox
interoperability were exercised against the deployed async PF. The composed
driver also passed the no-reboot SR-IOV receive-control and stress suite.

Fixes: 1f0c856b5963 ("enic: add MBOX core send and receive for admin channel")
Assisted-by: LLM
Signed-off-by: Satish Kharat <satishkh@cisco.com>
Reviewed-by: Aravind Dhanasekaran <aradhana@cisco.com>
Reviewed-by: Reese Faucette <rfaucett@cisco.com>
---
 drivers/net/ethernet/cisco/enic/enic.h        |  5 +++
 drivers/net/ethernet/cisco/enic/enic_admin.c  | 11 +++++++
 drivers/net/ethernet/cisco/enic/enic_dev.c    | 15 +++++++++
 drivers/net/ethernet/cisco/enic/enic_dev.h    |  2 ++
 drivers/net/ethernet/cisco/enic/enic_main.c   | 47 ++++++++++++++++++++++++---
 drivers/net/ethernet/cisco/enic/enic_mbox.c   | 12 +++++--
 drivers/net/ethernet/cisco/enic/vnic_dev.c    | 42 ++++++++++++++++++++++++
 drivers/net/ethernet/cisco/enic/vnic_dev.h    |  5 +++
 drivers/net/ethernet/cisco/enic/vnic_devcmd.h | 34 +++++++++++++++++++
 9 files changed, 167 insertions(+), 6 deletions(-)

diff --git a/drivers/net/ethernet/cisco/enic/enic.h b/drivers/net/ethernet/cisco/enic/enic.h
index cf2dee8fa78f..886f3cba543d 100644
--- a/drivers/net/ethernet/cisco/enic/enic.h
+++ b/drivers/net/ethernet/cisco/enic/enic.h
@@ -273,6 +273,11 @@ struct enic {
 	u16 num_vfs;
 	enum enic_vf_type vf_type;
 	bool vf_registered;
+	u16 vf_id;
+	/* A CMD_SRIOV_STATS_GET timeout leaves its persistent coherent target
+	 * device-owned until a later serialized command completes.
+	 */
+	bool sriov_stats_dma_unresolved;
 	u32 pf_cap_version;
 	unsigned int enable_count;
 	spinlock_t enic_api_lock;
diff --git a/drivers/net/ethernet/cisco/enic/enic_admin.c b/drivers/net/ethernet/cisco/enic/enic_admin.c
index a5c08cbcb339..2f6688a73b41 100644
--- a/drivers/net/ethernet/cisco/enic/enic_admin.c
+++ b/drivers/net/ethernet/cisco/enic/enic_admin.c
@@ -406,6 +406,12 @@ static int enic_admin_qp_type_set(struct enic *enic, u32 enable)
 
 	spin_lock_bh(&enic->devcmd_lock);
 	err = vnic_dev_cmd(enic->vdev, CMD_QP_TYPE_SET, &a0, &a1, wait);
+	/* DEVCMD2 runs one command at a time for a vNIC. Completion of this
+	 * blocking command proves that earlier statistics-command DMA has
+	 * finished. Admin WQ/RQ ownership still requires RUNNING == 0.
+	 */
+	if (!err)
+		WRITE_ONCE(enic->sriov_stats_dma_unresolved, false);
 	spin_unlock_bh(&enic->devcmd_lock);
 
 	return err;
@@ -546,6 +552,11 @@ int enic_admin_channel_open(struct enic *enic)
 			   "Refusing to reopen poisoned admin channel\n");
 		return -EIO;
 	}
+	if (READ_ONCE(enic->sriov_stats_dma_unresolved)) {
+		netdev_err(enic->netdev,
+			   "Refusing to reopen admin channel before SR-IOV statistics-command DMA is fenced\n");
+		return -EIO;
+	}
 	/* Keep MBOX sends disabled for the entire open sequence.  It is
 	 * cleared only after every resource is allocated and enabled below,
 	 * so any early error return here leaves sends disabled and a
diff --git a/drivers/net/ethernet/cisco/enic/enic_dev.c b/drivers/net/ethernet/cisco/enic/enic_dev.c
index 659787f73cf1..695e1db625c0 100644
--- a/drivers/net/ethernet/cisco/enic/enic_dev.c
+++ b/drivers/net/ethernet/cisco/enic/enic_dev.c
@@ -171,6 +171,21 @@ int enic_dev_intr_coal_timer_info(struct enic *enic)
 	return err;
 }
 
+int enic_dev_sriov_stats(struct enic *enic,
+			 struct vnic_sriov_stats **stats)
+{
+	int err;
+
+	*stats = NULL;
+	spin_lock_bh(&enic->devcmd_lock);
+	err = vnic_dev_sriov_stats(enic->vdev, stats);
+	if (err == -ETIMEDOUT)
+		WRITE_ONCE(enic->sriov_stats_dma_unresolved, true);
+	spin_unlock_bh(&enic->devcmd_lock);
+
+	return err;
+}
+
 /* rtnl lock is held */
 int enic_vlan_rx_add_vid(struct net_device *netdev, __be16 proto, u16 vid)
 {
diff --git a/drivers/net/ethernet/cisco/enic/enic_dev.h b/drivers/net/ethernet/cisco/enic/enic_dev.h
index 698d0cb02064..4bd67578801f 100644
--- a/drivers/net/ethernet/cisco/enic/enic_dev.h
+++ b/drivers/net/ethernet/cisco/enic/enic_dev.h
@@ -40,6 +40,8 @@ int enic_dev_set_ig_vlan_rewrite_mode(struct enic *enic);
 int enic_dev_enable(struct enic *enic);
 int enic_dev_disable(struct enic *enic);
 int enic_dev_intr_coal_timer_info(struct enic *enic);
+int enic_dev_sriov_stats(struct enic *enic,
+			 struct vnic_sriov_stats **stats);
 int enic_dev_status_to_errno(int devcmd_status);
 
 #endif /* _ENIC_DEV_H_ */
diff --git a/drivers/net/ethernet/cisco/enic/enic_main.c b/drivers/net/ethernet/cisco/enic/enic_main.c
index 06d76648b420..5574d71aa9a5 100644
--- a/drivers/net/ethernet/cisco/enic/enic_main.c
+++ b/drivers/net/ethernet/cisco/enic/enic_main.c
@@ -3024,6 +3024,12 @@ static void enic_sriov_detect_vf_type(struct enic *enic)
 
 	if (enic->vf_type != ENIC_VF_TYPE_V2)
 		return;
+	if (!vnic_dev_uses_devcmd2(enic->vdev)) {
+		dev_warn(&pdev->dev,
+			 "SR-IOV V2 requires DEVCMD2; disabling V2 support\n");
+		enic->vf_type = ENIC_VF_TYPE_NONE;
+		return;
+	}
 
 	/* A successful command means firmware recognizes
 	 * VIC_FEATURE_SRIOV; supported_versions is available
@@ -3199,6 +3205,7 @@ enic_sriov_configure(struct pci_dev *pdev, int num_vfs)
 
 static int enic_probe(struct pci_dev *pdev, const struct pci_device_id *ent)
 {
+	struct vnic_sriov_stats *sriov_stats;
 	struct device *dev = &pdev->dev;
 	struct net_device *netdev;
 	struct enic *enic;
@@ -3226,6 +3233,7 @@ static int enic_probe(struct pci_dev *pdev, const struct pci_device_id *ent)
 	enic = netdev_priv(netdev);
 	enic->netdev = netdev;
 	enic->pdev = pdev;
+	WRITE_ONCE(enic->vf_id, ENIC_MBOX_DST_PF);
 	spin_lock_init(&enic->vf_link_state_lock);
 
 	/* Setup PCI resources
@@ -3295,6 +3303,12 @@ static int enic_probe(struct pci_dev *pdev, const struct pci_device_id *ent)
 
 	if (err)
 		goto err_out_vnic_unregister;
+	if (enic_is_sriov_vf_v2(enic) &&
+	    !vnic_dev_uses_devcmd2(enic->vdev)) {
+		dev_err(dev, "SR-IOV V2 VF requires DEVCMD2\n");
+		err = -EOPNOTSUPP;
+		goto err_out_vnic_unregister;
+	}
 
 #ifdef CONFIG_PCI_IOV
 	enic_sriov_detect_vf_type(enic);
@@ -3404,6 +3418,9 @@ static int enic_probe(struct pci_dev *pdev, const struct pci_device_id *ent)
 	 * unmasks the admin interrupt so an early completion is not dropped.
 	 */
 	if (enic_is_sriov_vf_v2(enic)) {
+		u32 stats_ver;
+		u16 vf_id;
+
 		enic_mbox_init(enic);
 		err = enic_admin_channel_open(enic);
 		if (err) {
@@ -3411,6 +3428,27 @@ static int enic_probe(struct pci_dev *pdev, const struct pci_device_id *ent)
 				"Failed to open admin channel: %d\n", err);
 			goto err_out_admin_fence;
 		}
+
+		err = enic_dev_sriov_stats(enic, &sriov_stats);
+		if (err) {
+			dev_err(dev, "Failed to get SR-IOV statistics: %d\n",
+				err);
+			goto err_out_admin_close;
+		}
+		/* enic_dev_sriov_stats() orders reads from the coherent target.
+		 * Snapshot each multibyte field once.
+		 */
+		stats_ver = le32_to_cpu(sriov_stats->ver);
+		vf_id = le16_to_cpu(sriov_stats->vf_index);
+		if (stats_ver != VNIC_SRIOV_STATS_VERSION ||
+		    vf_id == ENIC_MBOX_DST_PF) {
+			dev_err(dev,
+				"Invalid SR-IOV statistics version %u VF index %#x\n",
+				stats_ver, vf_id);
+			err = -EINVAL;
+			goto err_out_admin_close;
+		}
+		WRITE_ONCE(enic->vf_id, vf_id);
 		err = enic_mbox_vf_capability_check(enic);
 		if (err) {
 			dev_err(dev,
@@ -3573,7 +3611,7 @@ static int enic_probe(struct pci_dev *pdev, const struct pci_device_id *ent)
 err_out_admin_fence:
 	if (enic_dma_quarantined(enic)) {
 		dev_err(dev,
-			"Queue DMA ownership unresolved; retaining probe resources until reboot\n");
+			"DMA ownership unresolved; retaining probe resources until reboot\n");
 		enic_terminal_dma_quarantine(enic);
 		pci_disable_device(pdev);
 		return err;
@@ -3608,7 +3646,8 @@ static int enic_probe(struct pci_dev *pdev, const struct pci_device_id *ent)
 static bool enic_dma_quarantined(struct enic *enic)
 {
 	return enic_datapath_is_quarantined(enic) ||
-	       enic_admin_channel_is_quarantined(enic);
+	       enic_admin_channel_is_quarantined(enic) ||
+	       READ_ONCE(enic->sriov_stats_dma_unresolved);
 }
 
 static int enic_datapath_recheck_on_remove(struct enic *enic)
@@ -3667,7 +3706,7 @@ static void enic_terminal_dma_quarantine(struct enic *enic)
 	enic_free_affinity_hint(enic);
 
 	/* Preserve netdev, vdev, BAR mappings, regions, rings, and their buffer
-	 * metadata because queue ownership remains unresolved until reboot.
+	 * metadata because DMA ownership remains unresolved until reboot.
 	 */
 	pci_clear_master(enic->pdev);
 }
@@ -3727,7 +3766,7 @@ static void enic_remove(struct pci_dev *pdev)
 			enic_datapath_recheck_on_remove(enic);
 		if (enic_dma_quarantined(enic)) {
 			dev_err(&pdev->dev,
-				"Queue DMA ownership unresolved; retaining resources until reboot\n");
+				"DMA ownership unresolved; retaining resources until reboot\n");
 			enic_terminal_dma_quarantine(enic);
 			retain_dma = true;
 			goto disable_device;
diff --git a/drivers/net/ethernet/cisco/enic/enic_mbox.c b/drivers/net/ethernet/cisco/enic/enic_mbox.c
index e106ee7d411d..1716da29eee7 100644
--- a/drivers/net/ethernet/cisco/enic/enic_mbox.c
+++ b/drivers/net/ethernet/cisco/enic/enic_mbox.c
@@ -18,10 +18,17 @@
 #define ENIC_MBOX_POLL_TIMEOUT_US	5000000
 #define ENIC_MBOX_POLL_INTERVAL_US	100
 
-static void enic_mbox_fill_hdr(struct enic_mbox_hdr *hdr, u8 msg_type,
+static void enic_mbox_fill_hdr(struct enic *enic, struct enic_mbox_hdr *hdr,
+			       u8 msg_type,
 			       u16 dst_vnic_id, u16 msg_len, u64 msg_num)
 {
+	u16 src_vnic_id = ENIC_MBOX_DST_PF;
+
+	if (enic_is_sriov_vf_v2(enic))
+		src_vnic_id = READ_ONCE(enic->vf_id);
+
 	memset(hdr, 0, sizeof(*hdr));
+	hdr->src_vnic_id = cpu_to_le16(src_vnic_id);
 	hdr->dst_vnic_id = cpu_to_le16(dst_vnic_id);
 	hdr->msg_type = msg_type;
 	hdr->msg_len = cpu_to_le16(msg_len);
@@ -85,7 +92,8 @@ static int enic_mbox_send_msg_id(struct enic *enic, u8 msg_type,
 		reply_expected = true;
 	}
 
-	enic_mbox_fill_hdr(buf, msg_type, dst_vnic_id, total_len, msg_num);
+	enic_mbox_fill_hdr(enic, buf, msg_type, dst_vnic_id, total_len,
+			   msg_num);
 	if (payload_len) {
 		void *dst = buf + sizeof(struct enic_mbox_hdr);
 
diff --git a/drivers/net/ethernet/cisco/enic/vnic_dev.c b/drivers/net/ethernet/cisco/enic/vnic_dev.c
index c8d657e97094..1a00f55658d4 100644
--- a/drivers/net/ethernet/cisco/enic/vnic_dev.c
+++ b/drivers/net/ethernet/cisco/enic/vnic_dev.c
@@ -1046,6 +1046,11 @@ void vnic_dev_unregister(struct vnic_dev *vdev)
 			dma_free_coherent(&vdev->pdev->dev,
 					  sizeof(struct vnic_stats),
 					  vdev->stats, vdev->stats_pa);
+		if (vdev->sriov_stats)
+			dma_free_coherent(&vdev->pdev->dev,
+					  sizeof(struct vnic_sriov_stats),
+					  vdev->sriov_stats,
+					  vdev->sriov_stats_pa);
 		if (vdev->fw_info)
 			dma_free_coherent(&vdev->pdev->dev,
 					  sizeof(struct vnic_devcmd_fw_info),
@@ -1088,6 +1093,11 @@ struct pci_dev *vnic_dev_get_pdev(struct vnic_dev *vdev)
 }
 EXPORT_SYMBOL(vnic_dev_get_pdev);
 
+bool vnic_dev_uses_devcmd2(struct vnic_dev *vdev)
+{
+	return vdev->devcmd2;
+}
+
 int vnic_devcmd_init(struct vnic_dev *vdev)
 {
 	void __iomem *res;
@@ -1295,3 +1305,35 @@ int vnic_dev_capable_rss_hash_type(struct vnic_dev *vdev, u8 *rss_hash_type)
 
 	return 0;
 }
+
+int vnic_dev_sriov_stats(struct vnic_dev *vdev,
+			 struct vnic_sriov_stats **stats)
+{
+	u64 a0, a1;
+	int err;
+
+	*stats = NULL;
+	if (!vdev->sriov_stats) {
+		vdev->sriov_stats =
+			dma_alloc_coherent(&vdev->pdev->dev,
+					   sizeof(*vdev->sriov_stats),
+					   &vdev->sriov_stats_pa,
+					   GFP_ATOMIC);
+		if (!vdev->sriov_stats)
+			return -ENOMEM;
+	}
+
+	memset(vdev->sriov_stats, 0, sizeof(*vdev->sriov_stats));
+	a0 = vdev->sriov_stats_pa;
+	a1 = sizeof(*vdev->sriov_stats);
+	err = vnic_dev_cmd(vdev, CMD_SRIOV_STATS_GET, &a0, &a1, 1000);
+	if (!err) {
+		/* Order CPU reads from the coherent target after command
+		 * completion.
+		 */
+		dma_rmb();
+		*stats = vdev->sriov_stats;
+	}
+
+	return err;
+}
diff --git a/drivers/net/ethernet/cisco/enic/vnic_dev.h b/drivers/net/ethernet/cisco/enic/vnic_dev.h
index 7fdd8c661c99..467435f57b92 100644
--- a/drivers/net/ethernet/cisco/enic/vnic_dev.h
+++ b/drivers/net/ethernet/cisco/enic/vnic_dev.h
@@ -93,6 +93,8 @@ struct vnic_dev {
 	dma_addr_t linkstatus_pa;
 	struct vnic_stats *stats;
 	dma_addr_t stats_pa;
+	struct vnic_sriov_stats *sriov_stats;
+	dma_addr_t sriov_stats_pa;
 	struct vnic_devcmd_fw_info *fw_info;
 	dma_addr_t fw_info_pa;
 	enum vnic_proxy_type proxy;
@@ -163,6 +165,7 @@ struct vnic_dev *vnic_dev_register(struct vnic_dev *vdev,
 	void *priv, struct pci_dev *pdev, struct vnic_dev_bar *bar,
 	unsigned int num_bars);
 struct pci_dev *vnic_dev_get_pdev(struct vnic_dev *vdev);
+bool vnic_dev_uses_devcmd2(struct vnic_dev *vdev);
 int vnic_dev_init_prov2(struct vnic_dev *vdev, u8 *buf, u32 len);
 int vnic_dev_enable2(struct vnic_dev *vdev, int active);
 int vnic_dev_enable2_done(struct vnic_dev *vdev, int *status);
@@ -177,5 +180,7 @@ int vnic_dev_overlay_offload_cfg(struct vnic_dev *vdev, u8 overlay,
 int vnic_dev_get_supported_feature_ver(struct vnic_dev *vdev, u8 feature,
 				       u64 *supported_versions, u64 *a1);
 int vnic_dev_capable_rss_hash_type(struct vnic_dev *vdev, u8 *rss_hash_type);
+int vnic_dev_sriov_stats(struct vnic_dev *vdev,
+			 struct vnic_sriov_stats **stats);
 
 #endif /* _VNIC_DEV_H_ */
diff --git a/drivers/net/ethernet/cisco/enic/vnic_devcmd.h b/drivers/net/ethernet/cisco/enic/vnic_devcmd.h
index 90ca06691ebd..9f397df88c50 100644
--- a/drivers/net/ethernet/cisco/enic/vnic_devcmd.h
+++ b/drivers/net/ethernet/cisco/enic/vnic_devcmd.h
@@ -461,6 +461,14 @@ enum vnic_devcmd_cmd {
 	 * in: (u32) a1 = enable (1) / disable (0)
 	 */
 	CMD_QP_TYPE_SET = _CMDC(_CMD_DIR_WRITE, _CMD_VTYPE_ENET, 97),
+
+	/*
+	 * Get SR-IOV capabilities, resource usage, and this function's logical
+	 * VF index.
+	 * in: (u64) a0 = DMA address of struct vnic_sriov_stats
+	 * in: (u32) a1 = length of the buffer
+	 */
+	CMD_SRIOV_STATS_GET = _CMDC(_CMD_DIR_WRITE, _CMD_VTYPE_ENET, 98),
 };
 
 #define QP_TYPE_ADMIN	0
@@ -468,6 +476,32 @@ enum vnic_devcmd_cmd {
 #define QP_ENABLE	1
 #define QP_DISABLE	0
 
+#define VNIC_SRIOV_STATS_VERSION	1
+
+struct vnic_sriov_stats {
+	__le32 ver;
+	u8 sriov_vlan_membership_cap;
+	u8 sriov_vlan_membership_enabled;
+	u8 sriov_rss_vf_full_cap;
+	u8 sriov_host_rx_stats;
+	__le32 ig_classifier0_tcam_cfg;
+	__le32 ig_classifier0_tcam_free;
+	__le32 eg_classifier0_tcam_cfg;
+	__le32 eg_classifier0_tcam_free;
+	__le32 ig_classifier1_tcam_cfg;
+	__le32 ig_classifier1_tcam_free;
+	__le32 eg_classifier1_tcam_cfg;
+	__le32 eg_classifier1_tcam_free;
+	__le32 sriov_ig_flow_table_cfg;
+	__le32 sriov_ig_flow_table_free;
+	__le32 sriov_eg_flow_table_cfg;
+	__le32 sriov_eg_flow_table_free;
+	u8 admin_qp_ready[32];
+	__le16 vf_index;
+	__le16 reserved1;
+	__le32 reserved2[256 - 23];
+};
+
 /* CMD_ENABLE2 flags */
 #define CMD_ENABLE2_STANDBY 0x0
 #define CMD_ENABLE2_ACTIVE  0x1

-- 
2.43.0