[PATCH v3 0/6] Axiado AX3000 and AX3005 USB Device Controller (UDC)

[PATCH v3 0/6] Axiado AX3000 and AX3005 USB Device Controller (UDC)

[Petar Stepanovic]

Dear maintainers,

This series continues the work of adding support for the USB device
controller (UDC) integrated into the Axiado AX3000 and AX3005 SoCs.

The controller is based on the Corigine USB 3.1 IP core, which is
dual-role capable (device and host) at the hardware level. Which role
is active depends on which driver is bound to it: this series binds
it in device role, while host role would instead use a standard xHCI
host driver - the hardware itself does not fix the role. Endpoint 0
shares the same command-ring / event-ring / per-endpoint transfer-ring
TRB hardware as every other endpoint, so the driver tracks control
transfer stages (SETUP/DATA/STATUS) explicitly instead of relying on a
dedicated hardware FIFO. Other upstream TRB/ring based UDC drivers,
such as cdns2 (whose file layout this series' structure is now modeled
on), share this same shape for the same reason.

Since v2, the driver has been restructured to follow the file layout
and conventions of drivers/usb/gadget/udc/cdns2, split into
core/gadget/ep0/ring source files with ftrace tracepoint support, and
brought in line with cdns2's static/global function ratio. See the
detailed changelog below.

The devicetree binding also picks up one item left over from the v2
review: the description now notes the underlying core is dual-role
capable (device and host), used here in device role.

Tested on an Axiado AX3005 EVK with a mass-storage + composite HID
(keyboard/mouse) gadget, including repeated bind/unbind/rebind cycles
and host-driven bus resets.

Signed-off-by: Petar Stepanovic <pstepanovic@axiado.com>
---
Changes in v3:

Devicetree binding:
- Note that the underlying Corigine core is dual-role capable (device
  and host), used here in device role.
- Drop the explicit maximum-speed property declaration. Krzysztof noted
  in the v2 review that compatible already implies SuperSpeed
capability,
  and testing confirms the property is not required.
- Reference the generic usb.yaml schema and replace additionalProperties
  with unevaluatedProperties to permit standard USB properties while
  rejecting unsupported properties.
- Validated the updated binding with dt_binding_check.

Driver structure (matching drivers/usb/gadget/udc/cdns2):
- Split crg_udc.c/crg_udc.h into crg-udc-core.c, crg-udc-gadget.c,
  crg-udc-ep0.c, crg-udc-ring.c and crg-udc.h, one concern per file.
- Add ftrace tracepoint support (crg-udc-trace.h/.c) covering pullup,
  doorbell, TRB queueing, event processing, control requests, and
  request alloc/free/giveback, plus a decode-helper header
  (crg-udc-debug.h) for the PORTSC/completion-code/TRB-type strings
  used by the trace TP_printk()s.
- Make internal-only functions static and normalize the remaining
  exported ones to a consistent crg_udc_ prefix.
- Reorganize #include lists so each file pulls in only the headers it
  actually uses, instead of funneling every kernel header the driver
  needs through the shared crg-udc.h.
- Match the SPDX/license comment style used elsewhere in the tree:
  a line comment for the SPDX tag, block comment for the description
  and copyright.

Bug fixes:
- Restore the VBUS-detect vendor-register write on udc_start(),
  udc_stop(), remove() and shutdown(). Without it the controller never
  considers itself powered and enumeration is unreliable.
- Restore the DATA_STAGE_XFER/DATA_STAGE_RECV cases dropped from the
  EP0 status-stage completion handler. Without them, any control read
  (e.g. GET_DESCRIPTOR) never arms its STATUS stage and the host times
  out waiting for it.
- Fix the transfer-ring TRB-deferral check to defer building a TRB
  only when the ring is actually full, not merely because another
  request is already queued on the same endpoint. The stricter check
  silently broke back-to-back completions on one endpoint, such as
  mass storage's CSW immediately following its data stage.
- Restore the RUN-gated completion check in the hardware command
  helper, so a command issued after the controller has already been
  stopped (e.g. during udc_stop()) does not spin waiting on a
  completion the hardware will never post.
- Call usb_gadget_udc_reset() on a bus reset so the composite/function
  drivers correctly re-run set_alt() and re-enable their endpoints
  after a reset, instead of composite silently treating a repeated
  SET_CONFIGURATION as a no-op. This is deferred to a workqueue rather
  than called directly from the port-status interrupt path, since the
  synchronous call re-enters the function drivers and would otherwise
  require dropping the driver lock in the middle of event processing.
- Clear the generic usb_ep.enabled flag and endpoint descriptors during
  internal endpoint teardown and controller reset, preventing stale
  endpoint state across sessions and allowing endpoints to be correctly
  re-enabled after a new bind or reset.

Additional fixes from Sashiko review:
- Use dma_addr_t for DMA offset calculations and validate DMA address
  translation failures before programming hardware registers.
- Reserve transfer-ring capacity and publish multi-TRB descriptors
  only after all TRBs have been initialized.
- Track remaining TRBs and buffer length for requests spanning
  multiple transfer descriptors.
- Advance the hardware dequeue pointer before completing canceled
  requests, and restore the producer cycle state when rewinding rings.
- Detach pending requests before completing them during endpoint
  teardown, and complete requests that fail during deferred queueing.
- Reduce the hardware reset timeout from 500 ms to 10 ms and release
  the spinlock while polling for completion.
- Protect pullup register updates with the controller spinlock.
- Update the event-ring dequeue pointer before yielding the CPU.
- Unregister the interrupt handler before freeing DMA resources.

- Link to v2: https://patch.msgid.link/20260817-axiado-ax3000-usb-device-controller-v2-0-0ef033dd0a68@axiado.com

Changes in v2:
- Drop the dual (GPL-2.0+ OR MIT) license; the driver and header are now
  GPL-2.0+ only.
- Register via module_platform_driver() and add MODULE_AUTHOR/MODULE_ALIAS
  instead of the hand-written module_init/module_exit pair.
- Acquire clock, reset and PHY through the driver model
  (devm_clk_get_optional, devm_reset_control_get_optional_exclusive,
  devm_phy_optional_get), with teardown wired up via
  devm_add_action_or_reset.
- Switch to devm_request_irq() and drop the manual free_irq() paths.
- Support multiple instances using an IDA (DEFINE_IDA/ida_alloc/ida_free)
  instead of a file-static device counter.
- Read match data with device_get_match_data() and move the former
  compile-time feature switches to DT properties (dis-u1u2-entry-quirk,
  dis-u1u2-initiation-quirk, reinit-ep0-on-bus-reset).
- Set the DMA mask via dma_set_mask_and_coherent() with a 32-bit fallback,
  and convert PM to SIMPLE_DEV_PM_OPS.
- Fix endianness handling for TRBs, events and endpoint contexts using
  cpu_to_le32()/le32_to_cpu(), and decode setup packets without casting.
- Locking: pass the caller's IRQ flags through the completion paths so the
  interrupt state is correctly restored across the unlock/lock window around
  gadget callbacks.
- Properly free all coherent DMA allocations and add
  dma_mapping_error() checks after dma_map_single()
- Fix DMA directions on the GET_STATUS/SET_SEL control transfers.
- Replace open-coded register busy-wait loops with
  readl_poll_timeout_atomic() so they time out instead of spinning forever,
  and return proper errnos (-EBUSY/-ETIMEDOUT/-EIO) instead of -1.
- Replace the custom SETF_VAR/GETF/MAKEF_VAR shift-and-mask macros with
  FIELD_PREP/FIELD_GET (include linux/bitfield.h), and use BIT()/GENMASK()
  and dma_wmb()/dma_rmb().
- Use DMA_MAPPING_ERROR instead of a private DMA_ADDR_INVALID.
- Remove the U-Boot-derived VBUS cable-detect routine, the VBUS-detect
  kthread and its wait queue, dead defines and struct fields, the unused
  interrupt-register-array macros, the mdelay()-based reset delays and the
  bulk of the dev_dbg() tracing.
- Implement the pullup(is_on == 0) path and route suspend/shutdown through
  pullup(0); mark crg_udc_ep_ops const and return IRQ_NONE from the shared
  IRQ handler when the interrupt is not ours.
- Use dev_err_probe() with structured goto-based unwinding in probe.
- Reword the Kconfig help text for USB_CRG_UDC to name the crg_udc module.
- Link to v1: https://lore.kernel.org/r/20260202-axiado-ax3000-usb-device-controller-v1-0-45ce0a8b014f@axiado.com

To: Naveen Kumar Rajgiri Bassappa <rbnaveenkumar@axiado.com>
To: Prasad Bolisetty <pbolisetty@axiado.com>
To: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
To: Rob Herring <robh@kernel.org>
To: Krzysztof Kozlowski <krzk+dt@kernel.org>
To: Conor Dooley <conor+dt@kernel.org>
To: Tzu-Hao Wei <twei@axiado.com>
To: Karthikeyan Mitran <kmitran@axiado.com>
To: Philipp Zabel <p.zabel@pengutronix.de>
To: Petar Stepanovic <pstepanovic@axiado.com>
Cc: linux-usb@vger.kernel.org
Cc: devicetree@vger.kernel.org
Cc: linux-arm-kernel@lists.infradead.org
Cc: linux-kernel@vger.kernel.org

---
Petar Stepanovic (6):
      dt-bindings: usb: gadget: udc: add Axiado AX3000 and AX3005 UDC
      usb: gadget: udc: crg: Add hardware definitions and structs
      usb: gadget: udc: crg: Add ring and transfer management
      usb: gadget: udc: crg: Add endpoint 0 handling
      usb: gadget: udc: crg: Add gadget and endpoint operations
      usb: gadget: udc: crg: Add core driver and build integration

 .../devicetree/bindings/usb/axiado,ax3000-udc.yaml |   69 +
 MAINTAINERS                                        |    9 +
 drivers/usb/gadget/udc/Kconfig                     |    2 +
 drivers/usb/gadget/udc/Makefile                    |    1 +
 drivers/usb/gadget/udc/crg/Kconfig                 |   16 +
 drivers/usb/gadget/udc/crg/Makefile                |    9 +
 drivers/usb/gadget/udc/crg/crg-udc-core.c          |  706 ++++++
 drivers/usb/gadget/udc/crg/crg-udc-debug.h         |  143 ++
 drivers/usb/gadget/udc/crg/crg-udc-ep0.c           | 1106 ++++++++++
 drivers/usb/gadget/udc/crg/crg-udc-gadget.c        | 2299 ++++++++++++++++++++
 drivers/usb/gadget/udc/crg/crg-udc-ring.c          |  664 ++++++
 drivers/usb/gadget/udc/crg/crg-udc-trace.c         |    9 +
 drivers/usb/gadget/udc/crg/crg-udc-trace.h         |  164 ++
 drivers/usb/gadget/udc/crg/crg-udc.h               |  722 ++++++
 14 files changed, 5919 insertions(+)
---
base-commit: c68641afacd9ea5707a489b30f204d5e89b324e0
change-id: 20260128-axiado-ax3000-usb-device-controller-156d3f1840c9

Best regards,
--  
Petar Stepanovic <pstepanovic@axiado.com>

[PATCH v3 1/6] dt-bindings: usb: gadget: udc: add Axiado AX3000 and AX3005 UDC

[Petar Stepanovic]

Add a Devicetree binding for the USB device controller integrated into
the Axiado AX3000 and AX3005 SoCs.

The controller is based on a Corigine USB 3.1 IP core that is
dual-role capable (device and host), used here in device role. It
provides SuperSpeed (5 Gb/s) or High-Speed (480 Mb/s) operation, and
supports control, bulk, interrupt, and isochronous transfer types
across multiple configurable endpoints.

Co-developed-by: Naveen Kumar Rajgiri Bassappa <rbnaveenkumar@axiado.com>
Signed-off-by: Naveen Kumar Rajgiri Bassappa <rbnaveenkumar@axiado.com>
Co-developed-by: Krutik Shah <krutikshah@axiado.com>
Signed-off-by: Krutik Shah <krutikshah@axiado.com>
Signed-off-by: Petar Stepanovic <pstepanovic@axiado.com>
---
 .../devicetree/bindings/usb/axiado,ax3000-udc.yaml | 69 ++++++++++++++++++++++
 MAINTAINERS                                        |  8 +++
 2 files changed, 77 insertions(+)

diff --git a/Documentation/devicetree/bindings/usb/axiado,ax3000-udc.yaml b/Documentation/devicetree/bindings/usb/axiado,ax3000-udc.yaml
new file mode 100644
index 000000000000..b9f7f4879ee0
--- /dev/null
+++ b/Documentation/devicetree/bindings/usb/axiado,ax3000-udc.yaml
@@ -0,0 +1,69 @@
+# SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause)
+%YAML 1.2
+---
+$id: http://devicetree.org/schemas/usb/axiado,ax3000-udc.yaml#
+$schema: http://devicetree.org/meta-schemas/core.yaml#
+
+title: Axiado AX3000/AX3005 USB Device Controller
+
+maintainers:
+  - Petar Stepanovic <pstepanovic@axiado.com>
+  - Naveen Kumar Rajgiri Bassappa <rbnaveenkumar@axiado.com>
+  - Prasad Bolisetty <pbolisetty@axiado.com>
+
+description:
+  Axiado USB dual-role controller based on the Corigine USB 3.1 IP core.
+  Supports USB 3.1 SuperSpeed and USB 2.0 High-Speed device operation
+  with control, bulk, interrupt, and isochronous transfers.
+
+allOf:
+  - $ref: usb.yaml#
+
+properties:
+  compatible:
+    oneOf:
+      - const: axiado,ax3000-udc
+      - items:
+          - const: axiado,ax3005-udc
+          - const: axiado,ax3000-udc
+
+  reg:
+    maxItems: 1
+
+  interrupts:
+    maxItems: 1
+
+  clocks:
+    maxItems: 1
+
+  resets:
+    maxItems: 1
+
+  phys:
+    maxItems: 1
+
+required:
+  - compatible
+  - reg
+  - interrupts
+
+unevaluatedProperties: false
+
+examples:
+  - |
+    #include <dt-bindings/interrupt-controller/arm-gic.h>
+    #include <dt-bindings/interrupt-controller/irq.h>
+
+    soc {
+        #address-cells = <2>;
+        #size-cells = <2>;
+
+        usb@81200000 {
+            compatible = "axiado,ax3000-udc";
+            reg = <0x0 0x81200000 0x0 0x100000>;
+            interrupt-parent = <&gic500>;
+            interrupts = <GIC_SPI 21 IRQ_TYPE_LEVEL_HIGH>;
+            resets = <&reset 0>;
+            phys = <&usb_phy>;
+        };
+    };
diff --git a/MAINTAINERS b/MAINTAINERS
index 65e8a4b5c90b..58c5c8e00870 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -4508,6 +4508,14 @@ F:	Documentation/devicetree/bindings/spi/axiado,ax3000-spi.yaml
 F:	drivers/spi/spi-axiado.c
 F:	drivers/spi/spi-axiado.h
 
+AXIADO USB UDC DRIVER
+M:	Petar Stepanovic <pstepanovic@axiado.com>
+M:	Naveen Kumar Rajgiri Bassappa <rbnaveenkumar@axiado.com>
+M:	Prasad Bolisetty <pbolisetty@axiado.com>
+L:	linux-usb@vger.kernel.org
+S:	Supported
+F:	Documentation/devicetree/bindings/usb/axiado,ax3000-udc.yaml
+
 AYANEO PLATFORM EC DRIVER
 M:	Antheas Kapenekakis <lkml@antheas.dev>
 L:	platform-driver-x86@vger.kernel.org

-- 
2.34.1

[PATCH v3 2/6] usb: gadget: udc: crg: Add hardware definitions and structs

[Petar Stepanovic]

This patch introduces the hardware definitions, register offsets, and
fundamental data structures for the Axiado AX3000/AX3005 USB Device
Controller driver.

Co-developed-by: Naveen Kumar Rajgiri Bassappa <rbnaveenkumar@axiado.com>
Signed-off-by: Naveen Kumar Rajgiri Bassappa <rbnaveenkumar@axiado.com>
Co-developed-by: Krutik Shah <krutikshah@axiado.com>
Signed-off-by: Krutik Shah <krutikshah@axiado.com>
Signed-off-by: Petar Stepanovic <pstepanovic@axiado.com>
---
 drivers/usb/gadget/udc/crg/crg-udc-debug.h | 143 ++++++
 drivers/usb/gadget/udc/crg/crg-udc-trace.h | 164 +++++++
 drivers/usb/gadget/udc/crg/crg-udc.h       | 722 +++++++++++++++++++++++++++++
 3 files changed, 1029 insertions(+)

diff --git a/drivers/usb/gadget/udc/crg/crg-udc-debug.h b/drivers/usb/gadget/udc/crg/crg-udc-debug.h
new file mode 100644
index 000000000000..f73c60a79ade
--- /dev/null
+++ b/drivers/usb/gadget/udc/crg/crg-udc-debug.h
@@ -0,0 +1,143 @@
+/* SPDX-License-Identifier: GPL-2.0+ */
+/*
+ * Axiado AX3000/AX3005 USB Device Controller driver.
+ * Debug/trace decode helpers.
+ *
+ * Copyright (c) 2021-2026 Axiado Corporation.
+ */
+
+#ifndef __CRG_UDC_DEBUG_H
+#define __CRG_UDC_DEBUG_H
+
+#include "crg-udc.h"
+
+static inline const char *crg_udc_trb_type_string(u32 type)
+{
+	switch (type) {
+	case TRB_TYPE_XFER_NORMAL:
+		return "Normal";
+	case TRB_TYPE_XFER_DATA_STAGE:
+		return "Data Stage";
+	case TRB_TYPE_XFER_STATUS_STAGE:
+		return "Status Stage";
+	case TRB_TYPE_XFER_DATA_ISOCH:
+		return "Isoch";
+	case TRB_TYPE_LINK:
+		return "Link";
+	case TRB_TYPE_NO_OP:
+		return "No-Op";
+	case TRB_TYPE_EVT_TRANSFER:
+		return "Transfer Event";
+	case TRB_TYPE_EVT_CMD_COMPLETION:
+		return "Command Completion Event";
+	case TRB_TYPE_EVT_PORT_STATUS_CHANGE:
+		return "Port Status Change Event";
+	case TRB_TYPE_EVT_MFINDEX_WRAP:
+		return "MFINDEX Wrap Event";
+	case TRB_TYPE_EVT_SETUP_PKT:
+		return "Setup Packet Event";
+	default:
+		return "Reserved";
+	}
+}
+
+static inline const char *crg_udc_cmpl_code_string(u32 code)
+{
+	switch (code) {
+	case CMPL_CODE_INVALID:
+		return "Invalid";
+	case CMPL_CODE_SUCCESS:
+		return "Success";
+	case CMPL_CODE_DATA_BUFFER_ERR:
+		return "Data Buffer Error";
+	case CMPL_CODE_BABBLE_DETECTED_ERR:
+		return "Babble Detected";
+	case CMPL_CODE_USB_TRANS_ERR:
+		return "USB Transaction Error";
+	case CMPL_CODE_TRB_ERR:
+		return "TRB Error";
+	case CMPL_CODE_TRB_STALL:
+		return "Stall";
+	case CMPL_CODE_INVALID_STREAM_TYPE_ERR:
+		return "Invalid Stream Type";
+	case CMPL_CODE_SHORT_PKT:
+		return "Short Packet";
+	case CMPL_CODE_RING_UNDERRUN:
+		return "Ring Underrun";
+	case CMPL_CODE_RING_OVERRUN:
+		return "Ring Overrun";
+	case CMPL_CODE_EVENT_RING_FULL_ERR:
+		return "Event Ring Full";
+	case CMPL_CODE_STOPPED:
+		return "Stopped";
+	case CMPL_CODE_STOPPED_LENGTH_INVALID:
+		return "Stopped - Length Invalid";
+	case CMPL_CODE_ISOCH_BUFFER_OVERRUN:
+		return "Isoch Buffer Overrun";
+	case CMPL_CODE_PROTOCOL_STALL:
+		return "Protocol Stall";
+	case CMPL_CODE_SETUP_TAG_MISMATCH:
+		return "Setup Tag Mismatch";
+	case CMPL_CODE_HALTED:
+		return "Halted";
+	case CMPL_CODE_HALTED_LENGTH_INVALID:
+		return "Halted - Length Invalid";
+	case CMPL_CODE_DISABLED:
+		return "Disabled";
+	case CMPL_CODE_DISABLED_LENGTH_INVALID:
+		return "Disabled - Length Invalid";
+	default:
+		return "Unknown";
+	}
+}
+
+static inline const char *crg_udc_decode_portsc(char *str, size_t size,
+						u32 portsc)
+{
+	scnprintf(str, size, "0x%08x, PLS: %lu, Speed: %lu, %s%s%s%s",
+		 portsc, FIELD_GET(CRG_U3DC_PORTSC_PLS_MASK, portsc),
+		 FIELD_GET(CRG_U3DC_PORTSC_SPEED_MASK, portsc),
+		 (portsc & CRG_U3DC_PORTSC_CCS) ? "CCS " : "",
+		 (portsc & CRG_U3DC_PORTSC_PED) ? "PED " : "",
+		 (portsc & CRG_U3DC_PORTSC_PR) ? "PR " : "",
+		 (portsc & CRG_U3DC_PORTSC_PP) ? "PP " : "");
+
+	return str;
+}
+
+static inline const char *crg_udc_decode_trb(char *str, size_t size,
+					     struct crg_transfer_trb *trb)
+{
+	u32 dw2 = le32_to_cpu(trb->dw2);
+	u32 dw3 = le32_to_cpu(trb->dw3);
+	u32 type = FIELD_GET(TRB_TYPE_MASK, dw3);
+
+	scnprintf(str, size,
+		  "type: %s, len: %lu, cycle: %lu, chain: %lu, ioc: %lu",
+		  crg_udc_trb_type_string(type),
+		  FIELD_GET(TRB_TRANSFER_LEN_MASK, dw2),
+		  dw3 & TRB_CYCLE_BIT_MASK ? 1UL : 0UL,
+		  dw3 & TRB_CHAIN_BIT_MASK ? 1UL : 0UL,
+		  dw3 & TRB_INTR_ON_COMPLETION_MASK ? 1UL : 0UL);
+
+	return str;
+}
+
+static inline const char *crg_udc_decode_event_trb(char *str, size_t size,
+						   struct crg_event_trb *trb)
+{
+	u32 dw2 = le32_to_cpu(trb->dw2);
+	u32 dw3 = le32_to_cpu(trb->dw3);
+	u32 type = FIELD_GET(EVE_TRB_TYPE_MASK, dw3);
+
+	scnprintf(str, size, "type: %s, cmpl: %s, len: %lu, dci: %lu",
+		  crg_udc_trb_type_string(type),
+		  crg_udc_cmpl_code_string(
+			  FIELD_GET(EVE_TRB_COMPL_CODE_MASK, dw2)),
+		  FIELD_GET(EVE_TRB_TRAN_LEN_MASK, dw2),
+		  FIELD_GET(EVE_TRB_ENDPOINT_ID_MASK, dw3));
+
+	return str;
+}
+
+#endif /* __CRG_UDC_DEBUG_H */
diff --git a/drivers/usb/gadget/udc/crg/crg-udc-trace.h b/drivers/usb/gadget/udc/crg/crg-udc-trace.h
new file mode 100644
index 000000000000..672a719a041b
--- /dev/null
+++ b/drivers/usb/gadget/udc/crg/crg-udc-trace.h
@@ -0,0 +1,164 @@
+/* SPDX-License-Identifier: GPL-2.0+ */
+/*
+ * Axiado AX3000/AX3005 USB Device Controller driver.
+ * Trace support header file.
+ *
+ * Copyright (c) 2021-2026 Axiado Corporation.
+ */
+
+#undef TRACE_SYSTEM
+#define TRACE_SYSTEM crg-udc
+
+#undef TRACE_SYSTEM_VAR
+#define TRACE_SYSTEM_VAR crg_udc
+
+#if !defined(__CRG_UDC_TRACE_H) || defined(TRACE_HEADER_MULTI_READ)
+#define __CRG_UDC_TRACE_H
+
+#include <linux/types.h>
+#include <linux/tracepoint.h>
+#include <linux/usb/ch9.h>
+#include "crg-udc.h"
+#include "crg-udc-debug.h"
+
+#define CRG_MSG_MAX 200
+
+DECLARE_EVENT_CLASS(crg_udc_log_enable_disable, TP_PROTO(int set), TP_ARGS(set),
+		    TP_STRUCT__entry(__field(int, set)),
+		    TP_fast_assign(__entry->set = set;),
+		    TP_printk("%s", __entry->set ? "enabled" : "disabled"));
+
+DEFINE_EVENT(crg_udc_log_enable_disable, crg_udc_pullup, TP_PROTO(int set),
+	     TP_ARGS(set));
+
+TRACE_EVENT(crg_udc_no_room_on_ring, TP_PROTO(char *msg), TP_ARGS(msg),
+	    TP_STRUCT__entry(__string(text, msg)),
+	    TP_fast_assign(__assign_str(text);),
+	    TP_printk("%s", __get_str(text)));
+
+TRACE_EVENT(crg_udc_ep_halt,
+	    TP_PROTO(struct crg_udc_ep *udc_ep, u8 halt, u8 wedge),
+	    TP_ARGS(udc_ep, halt, wedge),
+	    TP_STRUCT__entry(__string(name, udc_ep->usb_ep.name)
+				     __field(u8, halt) __field(u8, wedge)),
+	    TP_fast_assign(__assign_str(name); __entry->halt = halt;
+			   __entry->wedge = wedge;),
+	    TP_printk("%s: %s%s", __get_str(name),
+		      __entry->halt ? "halted" : "cleared",
+		      __entry->wedge ? " (wedged)" : ""));
+
+TRACE_EVENT(crg_udc_portsc, TP_PROTO(u32 portsc), TP_ARGS(portsc),
+	    TP_STRUCT__entry(__field(u32, portsc)),
+	    TP_fast_assign(__entry->portsc = portsc;),
+	    TP_printk("%s",
+		      crg_udc_decode_portsc(__get_buf(CRG_MSG_MAX), CRG_MSG_MAX,
+					    __entry->portsc)));
+
+TRACE_EVENT(crg_udc_doorbell, TP_PROTO(struct crg_gadget_dev *crg_udc, int dci),
+	    TP_ARGS(crg_udc, dci),
+	    TP_STRUCT__entry(__string(name, crg_udc->udc_ep[dci].usb_ep.name)
+				     __field(int, dci)),
+	    TP_fast_assign(__assign_str(name); __entry->dci = dci;),
+	    TP_printk("%s, dci %d", __get_str(name), __entry->dci));
+
+DECLARE_EVENT_CLASS(
+	crg_udc_log_ctrl, TP_PROTO(struct usb_ctrlrequest *ctrl), TP_ARGS(ctrl),
+	TP_STRUCT__entry(__field(u8, bRequestType) __field(u8, bRequest)
+				 __field(u16, wValue) __field(u16, wIndex)
+					 __field(u16, wLength)),
+	TP_fast_assign(__entry->bRequestType = ctrl->bRequestType;
+		       __entry->bRequest = ctrl->bRequest;
+		       __entry->wValue = le16_to_cpu(ctrl->wValue);
+		       __entry->wIndex = le16_to_cpu(ctrl->wIndex);
+		       __entry->wLength = le16_to_cpu(ctrl->wLength);),
+	TP_printk("%s", usb_decode_ctrl(__get_buf(CRG_MSG_MAX), CRG_MSG_MAX,
+					__entry->bRequestType,
+					__entry->bRequest, __entry->wValue,
+					__entry->wIndex, __entry->wLength)));
+
+DEFINE_EVENT(crg_udc_log_ctrl, crg_udc_ctrl_req,
+	     TP_PROTO(struct usb_ctrlrequest *ctrl), TP_ARGS(ctrl));
+
+DECLARE_EVENT_CLASS(
+	crg_udc_log_trb,
+	TP_PROTO(struct crg_udc_ep *udc_ep, struct crg_transfer_trb *trb),
+	TP_ARGS(udc_ep, trb),
+	TP_STRUCT__entry(__string(name, udc_ep->usb_ep.name)
+				 __field(struct crg_transfer_trb *, trb)
+					 __dynamic_array(char, decoded,
+							 CRG_MSG_MAX)),
+	TP_fast_assign(__assign_str(name); __entry->trb = trb;
+		       crg_udc_decode_trb(__get_dynamic_array(decoded),
+					  CRG_MSG_MAX, trb);),
+	TP_printk("%s: trb %p, %s", __get_str(name), __entry->trb,
+		  __get_str(decoded)));
+
+DEFINE_EVENT(crg_udc_log_trb, crg_udc_queue_trb,
+	     TP_PROTO(struct crg_udc_ep *udc_ep, struct crg_transfer_trb *trb),
+	     TP_ARGS(udc_ep, trb));
+
+TRACE_EVENT(
+	crg_udc_event_trb, TP_PROTO(struct crg_event_trb *trb), TP_ARGS(trb),
+	TP_STRUCT__entry(__field(struct crg_event_trb *, trb)
+				 __dynamic_array(char, decoded, CRG_MSG_MAX)),
+	TP_fast_assign(__entry->trb = trb;
+		       crg_udc_decode_event_trb(__get_dynamic_array(decoded),
+						CRG_MSG_MAX, trb);),
+	TP_printk("trb %p, %s", __entry->trb, __get_str(decoded)));
+
+DECLARE_EVENT_CLASS(
+	crg_udc_log_request,
+	TP_PROTO(struct crg_udc_ep *udc_ep, struct crg_udc_request *udc_req),
+	TP_ARGS(udc_ep, udc_req),
+	TP_STRUCT__entry(
+		__string(name, udc_ep->usb_ep.name) __field(
+			struct usb_request *, request) __field(void *, buf)
+			__field(unsigned int, actual) __field(unsigned int,
+							      length)
+				__field(int, status) __field(dma_addr_t, dma)
+					__field(int, zero) __field(int,
+								   short_not_ok)
+						__field(int, no_interrupt)),
+	TP_fast_assign(__assign_str(name); __entry->request = &udc_req->usb_req;
+		       __entry->buf = udc_req->usb_req.buf;
+		       __entry->actual = udc_req->usb_req.actual;
+		       __entry->length = udc_req->usb_req.length;
+		       __entry->status = udc_req->usb_req.status;
+		       __entry->dma = udc_req->usb_req.dma;
+		       __entry->zero = udc_req->usb_req.zero;
+		       __entry->short_not_ok = udc_req->usb_req.short_not_ok;
+		       __entry->no_interrupt = udc_req->usb_req.no_interrupt;),
+	TP_printk(
+		"%s: req: %p, buf: %p, length: %u/%u, status: %d, dma: %pad, %s%s%s",
+		__get_str(name), __entry->request, __entry->buf,
+		__entry->actual, __entry->length, __entry->status,
+		&__entry->dma, __entry->zero ? "Z" : "z",
+		__entry->short_not_ok ? "S" : "s",
+		__entry->no_interrupt ? "I" : "i"));
+
+DEFINE_EVENT(crg_udc_log_request, crg_udc_alloc_request,
+	     TP_PROTO(struct crg_udc_ep *udc_ep,
+		      struct crg_udc_request *udc_req),
+	     TP_ARGS(udc_ep, udc_req));
+
+DEFINE_EVENT(crg_udc_log_request, crg_udc_free_request,
+	     TP_PROTO(struct crg_udc_ep *udc_ep,
+		      struct crg_udc_request *udc_req),
+	     TP_ARGS(udc_ep, udc_req));
+
+DEFINE_EVENT(crg_udc_log_request, crg_udc_request_giveback,
+	     TP_PROTO(struct crg_udc_ep *udc_ep,
+		      struct crg_udc_request *udc_req),
+	     TP_ARGS(udc_ep, udc_req));
+
+#endif /* __CRG_UDC_TRACE_H */
+
+/* This part must be outside header guard. */
+
+#undef TRACE_INCLUDE_PATH
+#define TRACE_INCLUDE_PATH .
+
+#undef TRACE_INCLUDE_FILE
+#define TRACE_INCLUDE_FILE crg-udc-trace
+
+#include <trace/define_trace.h>
diff --git a/drivers/usb/gadget/udc/crg/crg-udc.h b/drivers/usb/gadget/udc/crg/crg-udc.h
new file mode 100644
index 000000000000..f3b545db5599
--- /dev/null
+++ b/drivers/usb/gadget/udc/crg/crg-udc.h
@@ -0,0 +1,722 @@
+/* SPDX-License-Identifier: GPL-2.0+ */
+/*
+ * Copyright (c) 2019 Corigine Inc.
+ * Copyright (c) 2021-2026 Axiado Corporation.
+ */
+
+#ifndef CRG_UDC_H
+#define CRG_UDC_H
+
+#include <linux/bitfield.h>
+#include <linux/bits.h>
+#include <linux/dma-mapping.h>
+#include <linux/io.h>
+#include <linux/kernel.h>
+#include <linux/types.h>
+#include <linux/workqueue.h>
+
+#include <linux/usb/ch9.h>
+#include <linux/usb/gadget.h>
+
+#define CRG_UCCR_OFFSET 0x2400
+#define CRG_UICR_OFFSET 0x2500
+#define CRG_UICR_STRIDE 0x20
+
+/* Corigine USB 3.1 device core register layout. */
+struct crg_uccr {
+	u32 capability;
+	u32 resv0[3];
+
+	u32 config0;
+	u32 config1;
+	u32 resv1[2];
+
+	u32 control;
+	u32 status;
+	u32 dcbaplo;
+	u32 dcbaphi;
+	u32 portsc;
+	u32 u3portpmsc;
+	u32 u2portpmsc;
+	u32 u3portli;
+
+	u32 doorbell;
+	u32 mfindex;
+	u32 ptmctrl;
+	u32 resv3[5];
+
+	u32 ep_enable;
+	u32 ep_running;
+	u32 resv4[2];
+
+	u32 cmd_param0;
+	u32 cmd_param1;
+	u32 cmd_control;
+	u32 resv5[1];
+
+	u32 odb_capability;
+	u32 resv6[3];
+	u32 odb_config[8];
+
+	u32 debug0;
+};
+
+struct crg_uicr {
+	u32 iman;
+	u32 imod;
+	u32 erstsz;
+	u32 resv0;
+
+	u32 erstbalo;
+	u32 erstbahi;
+	u32 erdplo;
+	u32 erdphi;
+};
+
+/* config0/config1 registers */
+#define CRG_U3DC_CFG0_MAXSPEED_MASK GENMASK(3, 0)
+#define CRG_U3DC_CFG0_MAXSPEED_FS 1
+#define CRG_U3DC_CFG0_MAXSPEED_HS 3
+#define CRG_U3DC_CFG0_MAXSPEED_SS 4
+#define CRG_U3DC_CFG0_MAXSPEED_SSP 5
+
+#define CRG_U3DC_CFG1_CSC_EVENT_EN BIT(0)
+#define CRG_U3DC_CFG1_PEC_EVENT_EN BIT(1)
+#define CRG_U3DC_CFG1_PPC_EVENT_EN BIT(3)
+#define CRG_U3DC_CFG1_PRC_EVENT_EN BIT(4)
+#define CRG_U3DC_CFG1_PLC_EVENT_EN BIT(5)
+#define CRG_U3DC_CFG1_CEC_EVENT_EN BIT(6)
+
+#define CRG_U3DC_CFG1_U3_ENTRY_EN BIT(8)
+#define CRG_U3DC_CFG1_L1_ENTRY_EN BIT(9)
+#define CRG_U3DC_CFG1_U3_RESUME_EN BIT(10)
+#define CRG_U3DC_CFG1_L1_RESUME_EN BIT(11)
+#define CRG_U3DC_CFG1_INACTIVE_PLC_EN BIT(12)
+#define CRG_U3DC_CFG1_U3_RESUME_NORESP_PLC_EN BIT(13)
+#define CRG_U3DC_CFG1_U2_RESUME_NORESP_PLC_EN BIT(14)
+
+#define CRG_U3DC_CFG1_SETUP_EVENT_EN BIT(16)
+
+/* control register */
+#define CRG_U3DC_CTRL_RUN BIT(0)
+#define CRG_U3DC_CTRL_STOP 0
+#define CRG_U3DC_CTRL_SWRST BIT(1)
+#define CRG_U3DC_CTRL_INT_EN BIT(2)
+#define CRG_U3DC_CTRL_SYSERR_EN BIT(3)
+#define CRG_U3DC_CTRL_EWE BIT(10)
+#define CRG_U3DC_CTRL_KP_CNCT BIT(11)
+
+/* status register */
+#define CRG_U3DC_STATUS_DEV_CTRL_HALT BIT(0)
+#define CRG_U3DC_STATUS_SYS_ERR BIT(2)
+#define CRG_U3DC_STATUS_EINT BIT(3)
+
+/* portsc register */
+#define CRG_U3DC_PORTSC_CCS BIT(0)
+#define CRG_U3DC_PORTSC_PED BIT(1)
+#define CRG_U3DC_PORTSC_PP BIT(3)
+#define CRG_U3DC_PORTSC_PR BIT(4)
+#define CRG_U3DC_PORTSC_PLS_MASK GENMASK(8, 5)
+#define CRG_U3DC_PORTSC_PLS_U3 3
+
+#define CRG_U3DC_PORTSC_SPEED_MASK GENMASK(13, 10)
+#define CRG_U3DC_PORTSC_SPEED_FS 0x1
+#define CRG_U3DC_PORTSC_SPEED_LS 0x2
+#define CRG_U3DC_PORTSC_SPEED_HS 0x3
+#define CRG_U3DC_PORTSC_SPEED_SS 0x4
+#define CRG_U3DC_PORTSC_SPEED_SSP 0x5
+
+#define CRG_U3DC_PORTSC_LWS BIT(16)
+#define CRG_U3DC_PORTSC_CSC BIT(17)
+#define CRG_U3DC_PORTSC_PEC BIT(18)
+#define CRG_U3DC_PORTSC_PPC BIT(20)
+#define CRG_U3DC_PORTSC_PRC BIT(21)
+#define CRG_U3DC_PORTSC_PLC BIT(22)
+#define CRG_U3DC_PORTSC_CEC BIT(23)
+#define CRG_U3DC_PORTSC_WCE BIT(25)
+#define CRG_U3DC_PORTSC_WDE BIT(26)
+#define CRG_U3DC_PORTSC_WPR BIT(31)
+
+#define CRG_U3DC_PORTSC_W1C_MASK                                           \
+	(CRG_U3DC_PORTSC_CSC | CRG_U3DC_PORTSC_PEC | CRG_U3DC_PORTSC_PPC | \
+	 CRG_U3DC_PORTSC_PRC | CRG_U3DC_PORTSC_PLC | CRG_U3DC_PORTSC_CEC)
+
+/* mfindex register */
+#define CRG_U3DC_MFINDEX_CURRENT_MASK GENMASK(17, 4)
+#define CRG_U3DC_MFINDEX_FRAME_MASK GENMASK(17, 7)
+
+/* ptmctrl register */
+#define CRG_U3DC_PTMCTRL_ISOCH_DELAY_MASK GENMASK(13, 0)
+
+/* u3portpmsc */
+#define CRG_U3DC_U3PORTPM_U1TMOUT_MASK GENMASK(7, 0)
+#define CRG_U3DC_U3PORTPM_U2TMOUT_MASK GENMASK(15, 8)
+#define CRG_U3DC_U3PORTPM_FLA BIT(16)
+#define CRG_U3DC_U3PORTPM_U1IEN BIT(20)
+#define CRG_U3DC_U3PORTPM_U2IEN BIT(21)
+#define CRG_U3DC_U3PORTPM_U1AEN BIT(22)
+#define CRG_U3DC_U3PORTPM_U2AEN BIT(23)
+
+#define CRG_U3DC_U3PORTPM_U1U2TMOUT_MASK GENMASK(31, 24)
+
+/* u2portpmsc */
+#define CRG_U3DC_U2PORTPM_RJ_TH_MASK GENMASK(3, 0)
+#define CRG_U3DC_U2PORTPM_DS_TH_MASK GENMASK(7, 4)
+#define CRG_U3DC_U2PORTPM_LPM_EN BIT(8)
+#define CRG_U3DC_U2PORTPM_RJ_TH_EN BIT(9)
+#define CRG_U3DC_U2PORTPM_DS_EN BIT(10)
+#define CRG_U3DC_U2PORTPM_SLP_EN BIT(11)
+#define CRG_U3DC_U2PORTPM_LPM_FACK BIT(12)
+#define CRG_U3DC_U2PORTPM_L1_AEX BIT(13)
+#define CRG_U3DC_U2PORTPM_H_B_MASK GENMASK(19, 16)
+#define CRG_U3DC_U2PORTPM_RWE BIT(20)
+#define CRG_U3DC_U2PORTPM_TM_MASK GENMASK(31, 28)
+
+/* doorbell register */
+#define CRG_U3DC_DB_TARGET_MASK GENMASK(4, 0)
+
+/* odb registers */
+#define CRG_U3DC_ODBCFG_2N_OFFSET_MASK GENMASK(9, 0)
+#define CRG_U3DC_ODBCFG_2N_SIZE_MASK GENMASK(12, 10)
+#define CRG_U3DC_ODBCFG_2N1_OFFSET_MASK GENMASK(25, 16)
+#define CRG_U3DC_ODBCFG_2N1_SIZE_MASK GENMASK(28, 26)
+
+/* command control register */
+#define CRG_U3DC_CMD_CTRL_ACTIVE BIT(0)
+#define CRG_U3DC_CMD_CTRL_IOC_EN BIT(1)
+#define CRG_U3DC_CMD_CTRL_TYPE_MASK GENMASK(7, 4)
+#define CRG_U3DC_CMD_CTRL_STATUS_MASK GENMASK(19, 16)
+
+/* interrupt register / IMAN bits */
+#define CRG_U3DC_IMAN_INT_PEND BIT(0)
+#define CRG_U3DC_IMAN_INT_EN BIT(1)
+
+/* erdp bits */
+#define CRG_U3DC_ERDPLO_EHB BIT(3)
+
+/* command parameters: init EP0 */
+#define CRG_CMD0_0_DQPTRLO_MASK GENMASK(31, 4)
+#define CRG_CMD0_0_DCS_MASK BIT(0)
+
+/* command parameters: update EP0 */
+#define CRG_CMD1_0_MPS_MASK GENMASK(31, 16)
+
+/* command parameters: set address */
+#define CRG_CMD2_0_DEV_ADDR_MASK GENMASK(7, 0)
+
+#define CRG_UDC_MODE_REG 0x20fc
+#define CRG_UDC_ROLE_DEVICE 0x1
+
+#define CRG_UDC_VENDOR_REG 0x210c
+#define CRG_UDC_VBUS_DETECT BIT(26)
+
+/*
+ * Device-visible data structures. The controller reads and writes these
+ * over DMA, so the layouts and field encodings below are fixed by the
+ * hardware in the same way as the register definitions above.
+ */
+
+/* Transfer and event TRB types */
+enum crg_trb_type {
+	TRB_TYPE_RSVD = 0,
+	TRB_TYPE_XFER_NORMAL,
+	TRB_TYPE_RSVD2,
+	TRB_TYPE_XFER_DATA_STAGE,
+	TRB_TYPE_XFER_STATUS_STAGE,
+	TRB_TYPE_XFER_DATA_ISOCH,
+	TRB_TYPE_LINK,
+	TRB_TYPE_RSVD7,
+	TRB_TYPE_NO_OP,
+
+	TRB_TYPE_EVT_TRANSFER = 32,
+	TRB_TYPE_EVT_CMD_COMPLETION = 33,
+	TRB_TYPE_EVT_PORT_STATUS_CHANGE = 34,
+	TRB_TYPE_EVT_MFINDEX_WRAP = 39,
+	TRB_TYPE_EVT_SETUP_PKT = 40,
+};
+
+/* Event TRB completion codes */
+enum crg_trb_cmpl_code {
+	CMPL_CODE_INVALID = 0,
+	CMPL_CODE_SUCCESS,
+	CMPL_CODE_DATA_BUFFER_ERR,
+	CMPL_CODE_BABBLE_DETECTED_ERR,
+	CMPL_CODE_USB_TRANS_ERR,
+	CMPL_CODE_TRB_ERR,
+	CMPL_CODE_TRB_STALL,
+	CMPL_CODE_INVALID_STREAM_TYPE_ERR = 10,
+	CMPL_CODE_SHORT_PKT = 13,
+	CMPL_CODE_RING_UNDERRUN,
+	CMPL_CODE_RING_OVERRUN,
+	CMPL_CODE_EVENT_RING_FULL_ERR = 21,
+	CMPL_CODE_STOPPED = 26,
+	CMPL_CODE_STOPPED_LENGTH_INVALID = 27,
+	CMPL_CODE_ISOCH_BUFFER_OVERRUN = 31,
+
+	CMPL_CODE_PROTOCOL_STALL = 192,
+	CMPL_CODE_SETUP_TAG_MISMATCH = 193,
+	CMPL_CODE_HALTED = 194,
+	CMPL_CODE_HALTED_LENGTH_INVALID = 195,
+	CMPL_CODE_DISABLED = 196,
+	CMPL_CODE_DISABLED_LENGTH_INVALID = 197,
+};
+
+/* Endpoint type encoding in the endpoint context */
+enum crg_ep_type {
+	EP_TYPE_INVALID = 0,
+	EP_TYPE_ISOCH_OUTBOUND,
+	EP_TYPE_BULK_OUTBOUND,
+	EP_TYPE_INTR_OUTBOUND,
+	EP_TYPE_INVALID2,
+	EP_TYPE_ISOCH_INBOUND,
+	EP_TYPE_BULK_INBOUND,
+	EP_TYPE_INTR_INBOUND,
+};
+
+/* crg_transfer_trb dw2 fields */
+#define TRB_TRANSFER_LEN_MASK GENMASK(16, 0)
+#define TRB_TD_SIZE_MASK GENMASK(21, 17)
+#define TRB_INTR_TARGET_MASK GENMASK(31, 22)
+
+/* crg_transfer_trb dw3 fields */
+#define TRB_CYCLE_BIT_MASK BIT(0)
+#define TRB_LINK_TOGGLE_CYCLE_MASK BIT(1)
+#define TRB_INTR_ON_SHORT_PKT_MASK BIT(2)
+#define TRB_NO_SNOOP_MASK BIT(3)
+#define TRB_CHAIN_BIT_MASK BIT(4)
+#define TRB_INTR_ON_COMPLETION_MASK BIT(5)
+#define TRB_APPEND_ZLP_MASK BIT(7)
+#define TRB_BLOCK_EVENT_INT_MASK BIT(9)
+#define TRB_TYPE_MASK GENMASK(15, 10)
+#define DATA_STAGE_TRB_DIR_MASK BIT(16)
+#define TRB_SETUP_TAG_MASK GENMASK(18, 17)
+#define STATUS_STAGE_TRB_STALL_MASK BIT(19)
+#define STATUS_STAGE_TRB_SET_ADDR_MASK BIT(20)
+#define ISOC_TRB_FRAME_ID_MASK GENMASK(30, 20)
+#define ISOC_TRB_SIA_MASK BIT(31)
+
+struct crg_transfer_trb {
+	__le32 dw0;
+	__le32 dw1;
+	__le32 dw2;
+	__le32 dw3;
+};
+
+/* crg_event_trb dw2 fields */
+#define EVE_TRB_TRAN_LEN_MASK GENMASK(16, 0)
+#define EVE_TRB_COMPL_CODE_MASK GENMASK(31, 24)
+
+/* crg_event_trb dw3 fields */
+#define EVE_TRB_CYCLE_BIT_MASK BIT(0)
+#define EVE_TRB_TYPE_MASK GENMASK(15, 10)
+#define EVE_TRB_ENDPOINT_ID_MASK GENMASK(20, 16)
+#define EVE_TRB_SETUP_TAG_MASK GENMASK(22, 21)
+
+struct crg_event_trb {
+	__le32 dw0;
+	__le32 dw1;
+	__le32 dw2;
+	__le32 dw3;
+};
+
+/* crg_ep_cx dw0 fields */
+#define EP_CX_LOGICAL_EP_NUM_MASK GENMASK(6, 3)
+#define EP_CX_INTERVAL_MASK GENMASK(23, 16)
+
+/* crg_ep_cx dw1 fields */
+#define EP_CX_EP_TYPE_MASK GENMASK(5, 3)
+#define EP_CX_MAX_BURST_SIZE_MASK GENMASK(15, 8)
+#define EP_CX_MAX_PACKET_SIZE_MASK GENMASK(31, 16)
+
+/* crg_ep_cx dw2 fields */
+#define EP_CX_DEQ_CYC_STATE_MASK BIT(0)
+#define EP_CX_TR_DQPT_LO_MASK GENMASK(31, 4)
+
+struct crg_ep_cx {
+	__le32 dw0;
+	__le32 dw1;
+	__le32 dw2;
+	__le32 dw3;
+};
+
+struct crg_erst {
+	/* 64-bit event ring segment address */
+	__le32 seg_addr_lo;
+	__le32 seg_addr_hi;
+	__le32 seg_size;
+	/* Set to zero */
+	__le32 rsvd;
+};
+
+/* command type */
+enum crg_cmd_type {
+	CRG_CMD_INIT_EP0 = 0,
+	CRG_CMD_UPDATE_EP0_CFG = 1,
+	CRG_CMD_SET_ADDR = 2,
+	CRG_CMD_SEND_DEV_NOTIFICATION = 3,
+	CRG_CMD_CONFIG_EP = 4,
+	CRG_CMD_SET_HALT = 5,
+	CRG_CMD_CLEAR_HALT = 6,
+	CRG_CMD_RESET_SEQNUM = 7,
+	CRG_CMD_STOP_EP = 8,
+	CRG_CMD_SET_TR_DQPTR = 9,
+	CRG_CMD_FORCE_FLOW_CONTROL = 10,
+	CRG_CMD_REQ_LDM_EXCHANGE = 11,
+};
+
+#define CRG_MAX_PACKET_SIZE 1024
+
+#define CRG_ERST_SIZE 1
+#define CRG_EVENT_RING_SIZE 256
+#define CRG_NUM_EP_CX 32
+
+/*
+ * Cap on events drained per udc_lock acquisition in crg_udc_process_event_ring().
+ * Command polling and crg_udc_reset()'s hardware-reset poll both run from
+ * crg_udc_handle_event() with udc_lock held, so without a cap a busy event
+ * ring could keep the lock, and local IRQs, held for as long as events
+ * keep arriving.
+ */
+#define CRG_EVENT_RING_BATCH 32
+
+#define CRG_UDC_TRB_MAX_BUFFER_SIZE 65536
+#define CRG_UDC_CONTROL_EP_TD_RING_SIZE 16
+#define CRG_UDC_BULK_EP_TD_RING_SIZE 1024
+#define CRG_UDC_ISOC_EP_TD_RING_SIZE 32
+#define CRG_UDC_INT_EP_TD_RING_SIZE 8
+
+#define CRG_UDC_U1_TIMEOUT_VAL 0x70
+#define CRG_UDC_U2_TIMEOUT_VAL 0x70
+
+enum crg_ep_state {
+	EP_STATE_DISABLED = 0,
+	EP_STATE_RUNNING = 1,
+	EP_STATE_HALTED = 2,
+	EP_STATE_STOPPED = 3
+};
+
+/* control-transfer setup state machine */
+enum crg_setup_status {
+	WAIT_FOR_SETUP = 0,
+	SETUP_PKT_PROCESS_IN_PROGRESS,
+	DATA_STAGE_XFER,
+	DATA_STAGE_RECV,
+	STATUS_STAGE_XFER,
+	STATUS_STAGE_RECV,
+};
+
+/* interrupt-on-completion moderation rate (in TRBs) per endpoint type */
+enum {
+	CRG_UDC_BULK_EP_INTERRUPT_RATE = 5,
+	CRG_UDC_ISOC_EP_INTERRUPT_RATE = 1,
+};
+
+struct crg_buffer_info {
+	void *cpu_addr;
+	dma_addr_t dma_addr;
+	u32 len;
+};
+
+struct crg_sel_value {
+	u16 u2_pel_value;
+	u16 u2_sel_value;
+	u8 u1_pel_value;
+	u8 u1_sel_value;
+};
+
+struct crg_udc_request {
+	struct usb_request usb_req;
+	struct list_head queue;
+	bool mapped;
+	/*
+	 * Direction used for the DMA mapping call that produced the mapping
+	 * tracked above. ep0 is bidirectional and its
+	 * descriptor address is always 0, so usb_endpoint_dir_in() on the
+	 * endpoint cannot tell an unmap apart from a control IN or OUT
+	 * transfer; the direction actually used to map is recorded here
+	 * instead and reused as-is at unmap time.
+	 */
+	enum dma_data_direction map_dir;
+	struct scatterlist *sg;
+	unsigned int sg_index;
+	u32 sg_offset;
+	u64 buff_len_left;
+	u32 trbs_needed;
+	struct crg_transfer_trb *first_trb;
+	struct crg_transfer_trb *last_trb;
+	bool all_trbs_queued;
+};
+
+struct crg_udc_ep {
+	struct usb_ep usb_ep;
+
+	struct crg_buffer_info tran_ring_info;
+	struct crg_transfer_trb *first_trb;
+	struct crg_transfer_trb *last_trb;
+
+	struct crg_transfer_trb *enq_pt;
+	struct crg_transfer_trb *deq_pt;
+	u8 pcs;
+
+	char name[10];
+	u8 dci;
+	struct list_head queue;
+	const struct usb_endpoint_descriptor *desc;
+	const struct usb_ss_ep_comp_descriptor *comp_desc;
+	bool tran_ring_full;
+	struct crg_gadget_dev *crg_udc;
+
+	enum crg_ep_state ep_state;
+
+	bool wedge;
+};
+
+#define CRG_RING_NUM 1
+
+struct crg_udc_event {
+	struct crg_buffer_info erst;
+	struct crg_erst *p_erst;
+	struct crg_buffer_info event_ring;
+	struct crg_event_trb *evt_dq_pt;
+	u8 ccs;
+	struct crg_event_trb *evt_seg0_last_trb;
+};
+
+enum crg_ep0_state {
+	EP0_STATE_IDLE = 0,
+	EP0_STATE_ABORTING,
+	EP0_STATE_ERROR,
+};
+
+struct crg_gadget_dev {
+	void __iomem *mmio_virt_base;
+	struct crg_uccr __iomem *uccr;
+	struct crg_uicr __iomem *uicr[CRG_RING_NUM];
+
+	/* udc_lock device lock */
+	spinlock_t udc_lock;
+
+	struct device *dev;
+	struct clk *clk;
+	struct phy *phy;
+	struct reset_control *rst;
+
+	struct usb_gadget gadget;
+	struct usb_gadget_driver *gadget_driver;
+	/* Speed selected for the currently bound gadget driver. */
+	enum usb_device_speed max_speed;
+
+	int irq;
+
+	struct usb_endpoint_descriptor ep0_desc;
+
+	struct crg_udc_ep udc_ep[CRG_NUM_EP_CX];
+	struct crg_buffer_info ep_cx;
+	struct crg_ep_cx *p_epcx;
+
+	struct crg_udc_event udc_event[CRG_RING_NUM];
+
+	struct crg_udc_request *status_req;
+	struct crg_sel_value sel_value;
+	void (*setup_fn_call_back)(struct crg_gadget_dev *a);
+
+	enum crg_setup_status setup_status;
+	/* EP0 rejects new requests while the previous control transfer is
+	 * being aborted and its completion callbacks are being given back.
+	 */
+	enum crg_ep0_state ep0_state;
+
+	u8 device_state;
+	u16 dev_addr;
+	u8 setup_tag;
+	u8 set_tm;
+
+	u32 num_enabled_eps;
+
+	bool connected;
+	bool pullup_on;
+	bool async_callbacks;
+	/*
+	 * Set while crg_udc_reset() has udc_lock dropped and is polling for
+	 * SWRST to clear. Checked by crg_udc_ep_queue() so a request queued
+	 * on another endpoint during that window is rejected instead of
+	 * being built into a ring and doorbell-rung while the controller is
+	 * still mid-reset.
+	 */
+	bool resetting;
+
+	bool u2_rwe;
+	bool feature_u1_enable;
+	bool feature_u2_enable;
+
+	bool portsc_on_reconnecting;
+	bool u12_forbidden;
+	bool u12_initiate_forbidden;
+	bool reinit_ep0_on_bus_reset;
+
+	/*
+	 * usb_gadget_udc_reset() re-enters the gadget/function drivers
+	 * synchronously. Calling it directly from the port-status IRQ path
+	 * would require dropping udc_lock mid-event-processing, opening a
+	 * window for a second port-status event to run crg_udc_reinit()
+	 * again while the first event's SET_CONFIGURATION/set_alt() work is
+	 * still landing on another thread, tearing the just-enabled
+	 * endpoints back down. Defer the notification to this work item so
+	 * the IRQ-thread's event processing never has to drop the lock.
+	 */
+	struct work_struct udc_reset_work;
+};
+
+/* Shared prototypes */
+void crg_udc_build_ep0_status(struct crg_udc_ep *udc_ep_ptr, bool default_value,
+			      u32 status, struct crg_udc_request *udc_req_ptr,
+			      u8 set_addr, u8 stall);
+/*
+ * Abort the current EP0 control transfer when a new SETUP packet arrives.
+ *
+ * A SETUP packet is allowed to abort any in-progress control transfer. Do
+ * not defer the new SETUP until the old transfer completes: doing so leaves
+ * the old request on the EP0 software queue and subsequent gadget requests
+ * are rejected as "queue not empty".
+ *
+ * The controller's EP0 ring is reinitialized at the ring start, making all
+ * TRBs belonging to the aborted control transfer stale. The new SETUP is
+ * then handled immediately with a fresh setup tag.
+ */
+int crg_udc_abort_ep0_for_setup(struct crg_gadget_dev *crg_udc);
+int crg_udc_enable_setup(struct crg_gadget_dev *crg_udc);
+void crg_udc_ep0_req_complete(struct crg_udc_ep *udc_ep_ptr,
+			      unsigned long *flags);
+int crg_udc_prepare_for_setup(struct crg_gadget_dev *crg_udc);
+int crg_udc_set_ep0_halt(struct crg_gadget_dev *crg_udc);
+int crg_udc_update_ep0_maxpacketsize(struct crg_gadget_dev *crg_udc);
+void crg_udc_handle_setup_pkt(struct crg_gadget_dev *crg_udc,
+			      struct usb_ctrlrequest *setup_pkt, u8 setup_tag,
+			      unsigned long *flags);
+int crg_udc_init_ep0(struct crg_gadget_dev *crg_udc);
+int crg_udc_queue_ctrl(struct crg_udc_ep *udc_ep_ptr,
+		       struct crg_udc_request *udc_req_ptr,
+		       u32 num_of_trbs_needed);
+void crg_udc_advance_dequeue_pt(struct crg_udc_ep *udc_ep);
+unsigned int crg_udc_count_trbs_linear(dma_addr_t addr, u64 len);
+unsigned int crg_udc_count_trbs_sg(struct crg_udc_request *udc_req);
+int crg_udc_init_event_ring(struct crg_gadget_dev *crg_udc, int index);
+bool crg_udc_is_event_rings_empty(struct crg_gadget_dev *crg_udc);
+bool crg_udc_is_pointer_less_than(struct crg_transfer_trb *a,
+				  struct crg_transfer_trb *b,
+				  struct crg_udc_ep *udc_ep);
+int crg_udc_process_event_ring(struct crg_gadget_dev *crg_udc, int index,
+			       unsigned long *flags);
+/* This function will go through the list of the USB requests for the
+ * given endpoint and schedule any unscheduled trb's to the xfer ring
+ */
+int crg_udc_queue_pending_trbs(struct crg_udc_ep *udc_ep_ptr);
+/* num_trbs here is the size of the ring. */
+u32 crg_udc_room_on_ring(struct crg_gadget_dev *crg_udc, u32 num_trbs,
+			 struct crg_transfer_trb *p_ring,
+			 struct crg_transfer_trb *enq_pt,
+			 struct crg_transfer_trb *dq_pt);
+void crg_udc_setup_datastage_trb(struct crg_transfer_trb *p_trb, dma_addr_t dma,
+				 u8 pcs, u32 transfer_length, u32 td_size,
+				 bool ioc, bool azp, u8 dir, u8 setup_tag);
+void crg_udc_setup_link_trb(struct crg_transfer_trb *link_trb, bool toggle,
+			    dma_addr_t next_trb);
+void crg_udc_setup_status_trb(struct crg_gadget_dev *crg_udc,
+			      struct crg_transfer_trb *p_trb,
+			      struct usb_request *usb_req, u8 pcs, u8 set_addr,
+			      u8 stall);
+void crg_udc_setup_trb(struct crg_transfer_trb *p_trb, u32 xfer_len,
+		       dma_addr_t xfer_buf_addr, u8 td_size, u8 pcs,
+		       u8 trb_type, bool short_pkt, bool chain_bit,
+		       bool intr_on_compl, bool b_setup_stage, u8 usb_dir,
+		       bool b_isoc, u16 frame_id, bool sia, bool azp);
+void crg_udc_squeeze_xfer_ring(struct crg_udc_ep *udc_ep_ptr,
+			       struct crg_udc_request *udc_req_ptr);
+struct crg_transfer_trb *crg_udc_tran_trb_dma_to_virt(struct crg_udc_ep *udc_ep,
+						    dma_addr_t address);
+dma_addr_t crg_udc_tran_trb_virt_to_dma(struct crg_udc_ep *udc_ep,
+					struct crg_transfer_trb *trb);
+void crg_udc_update_dequeue_pt(struct crg_event_trb *event,
+			       struct crg_udc_ep *udc_ep);
+u32 crg_udc_trb_chunk_len(dma_addr_t dma, u64 remaining);
+u32 crg_udc_actual_before(struct crg_udc_ep *udc_ep,
+			  struct crg_udc_request *udc_req,
+			  struct crg_transfer_trb *pause);
+/*
+ * command related ops
+ */
+int crg_udc_issue_command(struct crg_gadget_dev *crg_udc,
+			  enum crg_cmd_type type, u32 param0, u32 param1);
+void crg_udc_free_resources(struct crg_gadget_dev *crg_udc);
+int crg_udc_program_speed(struct crg_gadget_dev *crg_udc);
+int crg_udc_reset_data_struct(struct crg_gadget_dev *crg_udc);
+int crg_udc_reset(struct crg_gadget_dev *crg_udc, unsigned long *flags);
+struct usb_request *crg_udc_alloc_request(struct usb_ep *_ep, gfp_t gfp_flags);
+int crg_udc_build_td(struct crg_udc_ep *udc_ep_ptr,
+		     struct crg_udc_request *udc_req_ptr);
+void crg_udc_clear_portpm(struct crg_gadget_dev *crg_udc);
+void crg_udc_ep_nuke(struct crg_udc_ep *udc_ep, int status);
+void crg_udc_free_request(struct usb_ep *_ep, struct usb_request *_req);
+void crg_udc_set_pullup(struct crg_gadget_dev *crg_udc, bool is_on);
+void crg_udc_reset_work(struct work_struct *work);
+void crg_udc_set_vbus_detect(struct crg_gadget_dev *crg_udc, bool enable);
+/*
+ * Undo whatever crg_udc_ep_queue() mapped for this request: the merged
+ * SG list, the single buffer, or neither if the request carried a
+ * pre-mapped DMA address.
+ */
+void crg_udc_unmap_request(struct crg_udc_ep *udc_ep,
+			   struct crg_udc_request *udc_req);
+int crg_udc_handle_port_status(struct crg_gadget_dev *crg_udc,
+			       unsigned long *flags);
+int crg_udc_handle_xfer_event(struct crg_gadget_dev *crg_udc,
+			      struct crg_event_trb *event, unsigned long *flags);
+int crg_udc_handle_event(struct crg_gadget_dev *crg_udc,
+			 struct crg_event_trb *event, unsigned long *flags);
+/*
+ * Invoke the bound gadget driver's ->disconnect() callback, if any.
+ *
+ * The gadget disconnect callback may be called in interrupt context.
+ * Release only udc_lock while keeping local IRQs disabled.
+ */
+void crg_udc_notify_disconnect(struct crg_gadget_dev *crg_udc,
+			       unsigned long *flags);
+int crg_udc_ep_halt(struct crg_udc_ep *udc_ep_ptr, int halt, int ignore_wedge,
+		    unsigned long *flags);
+enum crg_ep_state crg_udc_get_ep_state(struct crg_gadget_dev *crg_udc, int dci);
+u32 crg_udc_index2dci(u16 index);
+int crg_udc_init_device_context(struct crg_gadget_dev *crg_udc);
+void crg_udc_init_ep_info(struct crg_gadget_dev *crg_udc);
+void crg_udc_knock_doorbell(struct crg_gadget_dev *crg_udc, int dci);
+/*
+ * Complete a request and give it back to the gadget layer via the
+ * standard usb_gadget_giveback_request() helper.
+ *
+ * Must be called with udc_lock held and interrupts disabled.
+ *
+ * The request's ->complete() callback must not sleep. crg_udc_req_done() drops
+ * udc_lock before the giveback and re-acquires it before returning.
+ * Callers therefore cannot assume state protected by udc_lock is
+ * unchanged across this call.
+ */
+void crg_udc_req_done(struct crg_udc_ep *udc_ep,
+		      struct crg_udc_request *udc_req, int status);
+void crg_udc_clear_req_container(struct crg_udc_request *udc_req_ptr);
+
+#define CRG_QUIRK_DIS_U1U2_ENTRY BIT(0)
+#define CRG_QUIRK_DIS_U1U2_INITIATION BIT(1)
+#define CRG_QUIRK_REINIT_EP0_ON_RESET BIT(2)
+
+#define CRG_UDC_QUIRKS_DEFAULT                                      \
+	(CRG_QUIRK_DIS_U1U2_ENTRY | CRG_QUIRK_DIS_U1U2_INITIATION | \
+	 CRG_QUIRK_REINIT_EP0_ON_RESET)
+
+struct crg_udc_priv {
+	u32 quirks;
+};
+
+extern const struct usb_gadget_ops crg_gadget_ops;
+extern const struct usb_ep_ops crg_udc_ep_ops;
+#define gadget_to_udc(g) (container_of(g, struct crg_gadget_dev, gadget))
+
+#endif /* CRG_UDC_H */

-- 
2.34.1

[PATCH v3 3/6] usb: gadget: udc: crg: Add ring and transfer management

[Petar Stepanovic]

This patch introduces the memory management and ring buffer logic
(TRBs and event rings) required by the Axiado AX3000/AX3005 UDC.

Co-developed-by: Naveen Kumar Rajgiri Bassappa <rbnaveenkumar@axiado.com>
Signed-off-by: Naveen Kumar Rajgiri Bassappa <rbnaveenkumar@axiado.com>
Co-developed-by: Krutik Shah <krutikshah@axiado.com>
Signed-off-by: Krutik Shah <krutikshah@axiado.com>
Signed-off-by: Petar Stepanovic <pstepanovic@axiado.com>
---
 drivers/usb/gadget/udc/crg/crg-udc-ring.c | 664 ++++++++++++++++++++++++++++++
 1 file changed, 664 insertions(+)

diff --git a/drivers/usb/gadget/udc/crg/crg-udc-ring.c b/drivers/usb/gadget/udc/crg/crg-udc-ring.c
new file mode 100644
index 000000000000..1c57dc1787d4
--- /dev/null
+++ b/drivers/usb/gadget/udc/crg/crg-udc-ring.c
@@ -0,0 +1,664 @@
+// SPDX-License-Identifier: GPL-2.0+
+/*
+ * Axiado AX3000/AX3005 USB Device Controller driver.
+ *
+ * Copyright (c) 2019 Corigine Inc.
+ * Copyright (c) 2021-2026 Axiado Corporation.
+ */
+#include <linux/sched.h>
+#include <linux/scatterlist.h>
+
+#include "crg-udc.h"
+
+void crg_udc_advance_dequeue_pt(struct crg_udc_ep *udc_ep)
+{
+	struct crg_udc_request *udc_req;
+
+	if (!list_empty(&udc_ep->queue)) {
+		udc_req = list_entry(udc_ep->queue.next, struct crg_udc_request,
+				     queue);
+
+		if (udc_req->first_trb)
+			udc_ep->deq_pt = udc_req->first_trb;
+		else
+			udc_ep->deq_pt = udc_ep->enq_pt;
+	} else {
+		udc_ep->deq_pt = udc_ep->enq_pt;
+	}
+}
+
+unsigned int crg_udc_count_trbs_linear(dma_addr_t addr, u64 len)
+{
+	unsigned int num_trbs = 0;
+
+	while (len) {
+		u32 chunk = crg_udc_trb_chunk_len(addr, len);
+
+		if (!chunk)
+			return 0;
+		num_trbs++;
+		addr += chunk;
+		len -= chunk;
+	}
+
+	return num_trbs;
+}
+
+unsigned int crg_udc_count_trbs_sg(struct crg_udc_request *udc_req)
+{
+	struct scatterlist *sg;
+	u64 remaining = udc_req->usb_req.length;
+	unsigned int i;
+	unsigned int num_trbs = 0;
+
+	for_each_sg(udc_req->usb_req.sg, sg, udc_req->usb_req.num_sgs, i) {
+		dma_addr_t addr;
+		u64 len;
+
+		if (!sg_dma_len(sg))
+			continue;
+		if (!remaining)
+			break;
+
+		len = min_t(u64, sg_dma_len(sg), remaining);
+		addr = sg_dma_address(sg);
+		while (len) {
+			u32 chunk = crg_udc_trb_chunk_len(addr, len);
+
+			if (!chunk)
+				return 0;
+			num_trbs++;
+			addr += chunk;
+			len -= chunk;
+			remaining -= chunk;
+		}
+	}
+
+	return remaining ? 0 : num_trbs;
+}
+
+static dma_addr_t event_trb_virt_to_dma(struct crg_udc_event *udc_event,
+					struct crg_event_trb *event)
+{
+	dma_addr_t dma_addr = 0;
+	unsigned long seg_offset;
+
+	if (!udc_event || !event)
+		return DMA_MAPPING_ERROR;
+
+	if ((void *)event < udc_event->event_ring.cpu_addr ||
+	    (void *)event >=
+		    udc_event->event_ring.cpu_addr + udc_event->event_ring.len)
+		return DMA_MAPPING_ERROR;
+
+	seg_offset = (void *)event - udc_event->event_ring.cpu_addr;
+	dma_addr = udc_event->event_ring.dma_addr + seg_offset;
+
+	return dma_addr;
+}
+
+int crg_udc_init_event_ring(struct crg_gadget_dev *crg_udc, int index)
+{
+	struct crg_uicr __iomem *uicr = crg_udc->uicr[index];
+	struct crg_udc_event *udc_event = &crg_udc->udc_event[index];
+
+	int ret;
+	u32 buff_length;
+	dma_addr_t mapping;
+	bool erst_allocated = false;
+
+	buff_length = CRG_ERST_SIZE * sizeof(struct crg_erst);
+	if (!udc_event->erst.cpu_addr) {
+		udc_event->erst.cpu_addr = dma_alloc_coherent(
+			crg_udc->dev, buff_length, &mapping, GFP_ATOMIC);
+
+		if (!udc_event->erst.cpu_addr)
+			return -ENOMEM;
+		erst_allocated = true;
+	} else {
+		mapping = udc_event->erst.dma_addr;
+	}
+
+	udc_event->erst.len = buff_length;
+	udc_event->erst.dma_addr = mapping;
+	udc_event->p_erst = udc_event->erst.cpu_addr;
+
+	buff_length = CRG_EVENT_RING_SIZE * sizeof(struct crg_event_trb);
+	if (!udc_event->event_ring.cpu_addr) {
+		udc_event->event_ring.cpu_addr = dma_alloc_coherent(
+			crg_udc->dev, buff_length, &mapping, GFP_ATOMIC);
+
+		if (!udc_event->event_ring.cpu_addr) {
+			ret = -ENOMEM;
+			goto err_free_erst;
+		}
+	} else {
+		mapping = udc_event->event_ring.dma_addr;
+	}
+
+	udc_event->event_ring.len = buff_length;
+	udc_event->event_ring.dma_addr = mapping;
+	udc_event->evt_dq_pt = udc_event->event_ring.cpu_addr;
+
+	udc_event->evt_seg0_last_trb =
+		(struct crg_event_trb *)(udc_event->event_ring.cpu_addr) +
+		(CRG_EVENT_RING_SIZE - 1);
+
+	udc_event->ccs = 1;
+
+	udc_event->p_erst->seg_addr_lo =
+		cpu_to_le32(lower_32_bits(udc_event->event_ring.dma_addr));
+	udc_event->p_erst->seg_addr_hi =
+		cpu_to_le32(upper_32_bits(udc_event->event_ring.dma_addr));
+	udc_event->p_erst->seg_size = cpu_to_le32(CRG_EVENT_RING_SIZE);
+	udc_event->p_erst->rsvd = cpu_to_le32(0);
+	/* Ensure that event ring is updated */
+	dma_wmb();
+	/* clear the event ring, to avoid hw unexpected ops
+	 * because of dirty data
+	 */
+	memset(udc_event->event_ring.cpu_addr, 0, buff_length);
+
+	/* Ensure that event ring is clear before programming
+	 * ERSTBA/ERSTSZ/ERDP
+	 */
+	dma_wmb();
+
+	writel(CRG_ERST_SIZE, &uicr->erstsz);
+	writel(lower_32_bits(udc_event->erst.dma_addr), &uicr->erstbalo);
+	writel(upper_32_bits(udc_event->erst.dma_addr), &uicr->erstbahi);
+	writel(lower_32_bits(udc_event->event_ring.dma_addr) |
+		       CRG_U3DC_ERDPLO_EHB,
+	       &uicr->erdplo);
+	writel(upper_32_bits(udc_event->event_ring.dma_addr), &uicr->erdphi);
+
+	writel((CRG_U3DC_IMAN_INT_EN | CRG_U3DC_IMAN_INT_PEND), &uicr->iman);
+	/* Interrupt moderation: 4000 * 250ns = 1ms */
+	writel(4000, &uicr->imod);
+	return 0;
+
+err_free_erst:
+	/* Only free what this call allocated; a pre-existing erst is owned
+	 * by an earlier successful init and released by crg_udc_free_resources().
+	 */
+	if (erst_allocated) {
+		dma_free_coherent(crg_udc->dev, udc_event->erst.len,
+				  udc_event->erst.cpu_addr,
+				  udc_event->erst.dma_addr);
+		udc_event->erst.cpu_addr = NULL;
+	}
+	return ret;
+}
+
+static bool is_event_ring_x_empty(struct crg_gadget_dev *crg_udc, int index)
+{
+	struct crg_event_trb *event;
+	struct crg_udc_event *udc_event;
+
+	udc_event = &crg_udc->udc_event[index];
+	if (udc_event->evt_dq_pt) {
+		event = (struct crg_event_trb *)udc_event->evt_dq_pt;
+
+		if (FIELD_GET(EVE_TRB_CYCLE_BIT_MASK,
+			      le32_to_cpu(event->dw3)) != udc_event->ccs)
+			return true;
+	}
+
+	return false;
+}
+
+bool crg_udc_is_event_rings_empty(struct crg_gadget_dev *crg_udc)
+{
+	int i;
+
+	for (i = 0; i < CRG_RING_NUM; i++) {
+		if (!is_event_ring_x_empty(crg_udc, i)) {
+			dev_err(crg_udc->dev, "event ring not empty\n");
+			return false;
+		}
+	}
+	return true;
+}
+
+bool crg_udc_is_pointer_less_than(struct crg_transfer_trb *a,
+				  struct crg_transfer_trb *b,
+				  struct crg_udc_ep *udc_ep)
+{
+	if (!a || !b || !udc_ep->first_trb || !udc_ep->last_trb)
+		return false;
+	if (b > a && (udc_ep->enq_pt >= b || udc_ep->enq_pt < a))
+		return true;
+	if (b < a && (udc_ep->enq_pt >= b && udc_ep->enq_pt < a))
+		return true;
+	return false;
+}
+
+int crg_udc_process_event_ring(struct crg_gadget_dev *crg_udc, int index,
+			       unsigned long *flags)
+{
+	struct crg_event_trb *event;
+	struct crg_udc_event *udc_event;
+	struct crg_uicr __iomem *uicr = crg_udc->uicr[index];
+	u32 val;
+	dma_addr_t erdp;
+	int ret;
+	unsigned int processed = 0;
+
+	if (!uicr)
+		return -ENODEV;
+
+	val = readl(&uicr->iman);
+	val |= CRG_U3DC_IMAN_INT_PEND;
+	writel(val, &uicr->iman);
+
+	udc_event = &crg_udc->udc_event[index];
+	while (udc_event->evt_dq_pt) {
+		event = (struct crg_event_trb *)udc_event->evt_dq_pt;
+
+		if (FIELD_GET(EVE_TRB_CYCLE_BIT_MASK,
+			      le32_to_cpu(event->dw3)) != udc_event->ccs)
+			break;
+		/* Read the payload only after observing device ownership. */
+		dma_rmb();
+
+		ret = crg_udc_handle_event(crg_udc, event, flags);
+		if (ret == -ECONNRESET) {
+			/* The disconnect path rebuilt and reprogrammed the ring. */
+			return 0;
+		}
+		if (ret < 0)
+			dev_dbg(crg_udc->dev, "failed to process event: %d\n",
+				ret);
+
+		/* Consume malformed events so they cannot pin the ring head. */
+		if (event == udc_event->evt_seg0_last_trb) {
+			udc_event->ccs = udc_event->ccs ? 0 : 1;
+			udc_event->evt_dq_pt = udc_event->event_ring.cpu_addr;
+		} else {
+			udc_event->evt_dq_pt++;
+		}
+
+		/*
+		 * udc_event state is only re-read from crg_udc, never cached
+		 * across this drop, so releasing and reacquiring the lock
+		 * here is safe and lets other udc_lock users, and this CPU's
+		 * other interrupts, make progress during a busy event ring.
+		 */
+		if (++processed % CRG_EVENT_RING_BATCH == 0) {
+			erdp = event_trb_virt_to_dma(udc_event,
+						     udc_event->evt_dq_pt);
+			if (!dma_mapping_error(crg_udc->dev, erdp)) {
+				writel(upper_32_bits(erdp), &uicr->erdphi);
+				writel(lower_32_bits(erdp) |
+					       CRG_U3DC_ERDPLO_EHB,
+				       &uicr->erdplo);
+			}
+			spin_unlock_irqrestore(&crg_udc->udc_lock, *flags);
+			cond_resched();
+			spin_lock_irqsave(&crg_udc->udc_lock, *flags);
+		}
+	}
+
+	/* update dequeue pointer */
+	erdp = event_trb_virt_to_dma(udc_event, udc_event->evt_dq_pt);
+	if (dma_mapping_error(crg_udc->dev, erdp))
+		return -EINVAL;
+	writel(upper_32_bits(erdp), &uicr->erdphi);
+	writel(lower_32_bits(erdp) | CRG_U3DC_ERDPLO_EHB, &uicr->erdplo);
+
+	return 0;
+}
+
+/* This function will go through the list of the USB requests for the
+ * given endpoint and schedule any unscheduled trb's to the xfer ring
+ */
+int crg_udc_queue_pending_trbs(struct crg_udc_ep *udc_ep_ptr)
+{
+	struct crg_udc_request *udc_req_ptr, *tmp;
+	int ret;
+
+	list_for_each_entry_safe(udc_req_ptr, tmp, &udc_ep_ptr->queue, queue) {
+		if (udc_req_ptr->all_trbs_queued)
+			continue;
+
+		ret = crg_udc_build_td(udc_ep_ptr, udc_req_ptr);
+		if (ret == -ENOSPC || ret == -EBUSY) {
+			/*
+			 * Ring is full; leave this request queued and retry
+			 * it (and anything behind it) on the next TRB
+			 * completion via crg_udc_queue_pending_trbs().
+			 */
+			return ret;
+		} else if (ret) {
+			/*
+			 * A non-capacity failure is not transient: fail this
+			 * request instead of leaving it stuck forever, and
+			 * keep rebuilding the rest of the queue.
+			 */
+			dev_err(udc_ep_ptr->crg_udc->dev,
+				"failed to build pending transfer: %d\n", ret);
+			list_del_init(&udc_req_ptr->queue);
+			crg_udc_req_done(udc_ep_ptr, udc_req_ptr, ret);
+			continue;
+		}
+		break;
+	}
+
+	return 0;
+
+}
+
+/* num_trbs here is the size of the ring. */
+u32 crg_udc_room_on_ring(struct crg_gadget_dev *crg_udc, u32 num_trbs,
+			 struct crg_transfer_trb *p_ring,
+			 struct crg_transfer_trb *enq_pt,
+			 struct crg_transfer_trb *dq_pt)
+{
+	u32 i = 0;
+
+	if (enq_pt == dq_pt) {
+		/* ring is empty */
+		return num_trbs - 2;
+	}
+
+	while (enq_pt != dq_pt) {
+		i++;
+
+		enq_pt++;
+
+		if (FIELD_GET(TRB_TYPE_MASK, le32_to_cpu(enq_pt->dw3)) ==
+		    TRB_TYPE_LINK)
+			enq_pt = p_ring;
+
+		if (i > num_trbs)
+			break;
+	}
+
+	return i - 1;
+}
+
+void crg_udc_setup_datastage_trb(struct crg_transfer_trb *p_trb, dma_addr_t dma,
+				 u8 pcs, u32 transfer_length, u32 td_size,
+				 bool ioc, bool azp, u8 dir, u8 setup_tag)
+{
+	u32 tmp;
+
+	p_trb->dw0 = cpu_to_le32(lower_32_bits(dma));
+	p_trb->dw1 = cpu_to_le32(upper_32_bits(dma));
+
+	/* TRB_Transfer_Length
+	 * For USB_DIR_OUT, this field is the number of data bytes expected
+	 * from xhc. For USB_DIR_IN, this field is the number of data bytes
+	 * the device will send.
+	 */
+	tmp = 0;
+	FIELD_MODIFY(TRB_TRANSFER_LEN_MASK, &tmp, transfer_length);
+	FIELD_MODIFY(TRB_TD_SIZE_MASK, &tmp, td_size);
+	FIELD_MODIFY(TRB_INTR_TARGET_MASK, &tmp, 0);
+	p_trb->dw2 = cpu_to_le32(tmp);
+
+	tmp = 0;
+	FIELD_MODIFY(TRB_CYCLE_BIT_MASK, &tmp, pcs);
+	FIELD_MODIFY(TRB_INTR_ON_SHORT_PKT_MASK, &tmp, 1);
+	FIELD_MODIFY(TRB_INTR_ON_COMPLETION_MASK, &tmp, ioc);
+	FIELD_MODIFY(TRB_TYPE_MASK, &tmp, TRB_TYPE_XFER_DATA_STAGE);
+	FIELD_MODIFY(TRB_APPEND_ZLP_MASK, &tmp, azp);
+	FIELD_MODIFY(DATA_STAGE_TRB_DIR_MASK, &tmp, dir);
+	FIELD_MODIFY(TRB_SETUP_TAG_MASK, &tmp, setup_tag);
+
+	/* Publish ownership only after the TRB payload is visible. */
+	dma_wmb();
+	p_trb->dw3 = cpu_to_le32(tmp);
+}
+
+void crg_udc_setup_link_trb(struct crg_transfer_trb *link_trb, bool toggle,
+			    dma_addr_t next_trb)
+{
+	u32 dw = 0;
+
+	link_trb->dw0 = cpu_to_le32(lower_32_bits(next_trb));
+	link_trb->dw1 = cpu_to_le32(upper_32_bits(next_trb));
+	link_trb->dw2 = cpu_to_le32(0);
+
+	FIELD_MODIFY(TRB_TYPE_MASK, &dw, TRB_TYPE_LINK);
+	if (toggle)
+		FIELD_MODIFY(TRB_LINK_TOGGLE_CYCLE_MASK, &dw, 1);
+	else
+		FIELD_MODIFY(TRB_LINK_TOGGLE_CYCLE_MASK, &dw, 0);
+
+	/* Publish ownership only after the link address is visible. */
+	dma_wmb();
+	link_trb->dw3 = cpu_to_le32(dw);
+}
+
+void crg_udc_setup_status_trb(struct crg_gadget_dev *crg_udc,
+			      struct crg_transfer_trb *p_trb,
+			      struct usb_request *usb_req, u8 pcs, u8 set_addr,
+			      u8 stall)
+{
+	u32 tmp, dir = 0;
+
+	/* There are some cases where crg_udc_setup_status_trb() is called with
+	 * usb_req set to NULL.
+	 */
+	p_trb->dw0 = cpu_to_le32(0);
+	p_trb->dw1 = cpu_to_le32(0);
+
+	tmp = 0;
+	FIELD_MODIFY(TRB_INTR_TARGET_MASK, &tmp, 0);
+	p_trb->dw2 = cpu_to_le32(tmp);
+
+	tmp = 0;
+	FIELD_MODIFY(TRB_CYCLE_BIT_MASK, &tmp, pcs);
+	FIELD_MODIFY(TRB_INTR_ON_COMPLETION_MASK, &tmp, 1);
+	FIELD_MODIFY(TRB_TYPE_MASK, &tmp, TRB_TYPE_XFER_STATUS_STAGE);
+
+	dir = (crg_udc->setup_status == STATUS_STAGE_XFER) ? 0 : 1;
+	FIELD_MODIFY(DATA_STAGE_TRB_DIR_MASK, &tmp, dir);
+
+	FIELD_MODIFY(TRB_SETUP_TAG_MASK, &tmp, crg_udc->setup_tag);
+	FIELD_MODIFY(STATUS_STAGE_TRB_STALL_MASK, &tmp, stall);
+	FIELD_MODIFY(STATUS_STAGE_TRB_SET_ADDR_MASK, &tmp, set_addr);
+
+	/* Publish ownership only after the TRB payload is visible. */
+	dma_wmb();
+	p_trb->dw3 = cpu_to_le32(tmp);
+}
+
+void crg_udc_setup_trb(struct crg_transfer_trb *p_trb, u32 xfer_len,
+		       dma_addr_t xfer_buf_addr, u8 td_size, u8 pcs,
+		       u8 trb_type, bool short_pkt, bool chain_bit,
+		       bool intr_on_compl, bool b_setup_stage, u8 usb_dir,
+		       bool b_isoc, u16 frame_id, bool sia, bool azp)
+{
+	u32 tmp;
+
+	p_trb->dw0 = cpu_to_le32(lower_32_bits(xfer_buf_addr));
+	p_trb->dw1 = cpu_to_le32(upper_32_bits(xfer_buf_addr));
+
+	tmp = 0;
+	FIELD_MODIFY(TRB_TRANSFER_LEN_MASK, &tmp, xfer_len);
+	FIELD_MODIFY(TRB_TD_SIZE_MASK, &tmp, td_size);
+	FIELD_MODIFY(TRB_INTR_TARGET_MASK, &tmp, 0);
+
+	p_trb->dw2 = cpu_to_le32(tmp);
+
+	tmp = 0;
+	FIELD_MODIFY(TRB_CYCLE_BIT_MASK, &tmp, pcs);
+	FIELD_MODIFY(TRB_INTR_ON_SHORT_PKT_MASK, &tmp, short_pkt);
+	FIELD_MODIFY(TRB_CHAIN_BIT_MASK, &tmp, chain_bit);
+	FIELD_MODIFY(TRB_INTR_ON_COMPLETION_MASK, &tmp, intr_on_compl);
+	FIELD_MODIFY(TRB_APPEND_ZLP_MASK, &tmp, azp);
+	FIELD_MODIFY(TRB_TYPE_MASK, &tmp, trb_type);
+
+	if (b_setup_stage)
+		FIELD_MODIFY(DATA_STAGE_TRB_DIR_MASK, &tmp, usb_dir);
+
+	if (b_isoc) {
+		FIELD_MODIFY(ISOC_TRB_FRAME_ID_MASK, &tmp, frame_id);
+		FIELD_MODIFY(ISOC_TRB_SIA_MASK, &tmp, sia);
+	}
+
+	/* Publish ownership only after the TRB payload is visible. */
+	dma_wmb();
+	p_trb->dw3 = cpu_to_le32(tmp);
+}
+
+void crg_udc_squeeze_xfer_ring(struct crg_udc_ep *udc_ep_ptr,
+			       struct crg_udc_request *udc_req_ptr)
+{
+	struct crg_transfer_trb *trb = udc_req_ptr->first_trb;
+	struct crg_udc_request *next_req, *tmp;
+	u8 new_pcs;
+
+	if (!trb)
+		return;
+
+	/* The canceled request becomes the new enqueue point. Its original
+	 * cycle bit is the cycle expected at that point after the tail is
+	 * removed.
+	 */
+	new_pcs = FIELD_GET(TRB_CYCLE_BIT_MASK, le32_to_cpu(trb->dw3));
+
+	while (trb != udc_ep_ptr->enq_pt) {
+		memset(trb, 0, sizeof(*trb));
+		trb++;
+		if (trb == udc_ep_ptr->last_trb)
+			trb = udc_ep_ptr->first_trb;
+	}
+
+	udc_ep_ptr->enq_pt = udc_req_ptr->first_trb;
+	udc_ep_ptr->pcs = new_pcs;
+	udc_ep_ptr->tran_ring_full = false;
+	crg_udc_setup_link_trb(udc_ep_ptr->last_trb, true,
+			udc_ep_ptr->tran_ring_info.dma_addr);
+	dma_wmb();
+
+	if (list_is_last(&udc_req_ptr->queue, &udc_ep_ptr->queue))
+		return;
+
+	next_req = udc_req_ptr;
+	list_for_each_entry_safe_continue(next_req, tmp, &udc_ep_ptr->queue,
+					  queue) {
+		int ret;
+
+		next_req->usb_req.status = -EINPROGRESS;
+		next_req->usb_req.actual = 0;
+		crg_udc_clear_req_container(next_req);
+		ret = crg_udc_build_td(udc_ep_ptr, next_req);
+		if (ret == -ENOSPC) {
+			/* Ring is full; a later TRB completion will retry
+			 * this and any remaining requests via
+			 * crg_udc_queue_pending_trbs().
+			 */
+			udc_ep_ptr->tran_ring_full = true;
+			break;
+		} else if (ret) {
+			/*
+			 * A non-capacity failure is not transient: fail this
+			 * request instead of leaving it stuck forever, and
+			 * keep rebuilding the rest of the queue.
+			 */
+			dev_err(udc_ep_ptr->crg_udc->dev,
+				"failed to rebuild queued request: %d\n", ret);
+			crg_udc_req_done(udc_ep_ptr, next_req, ret);
+			continue;
+		}
+		if (udc_ep_ptr->tran_ring_full)
+			break;
+	}
+}
+
+struct crg_transfer_trb *crg_udc_tran_trb_dma_to_virt(struct crg_udc_ep *udc_ep,
+						    dma_addr_t address)
+{
+	dma_addr_t offset;
+	struct crg_transfer_trb *trb_virt;
+
+	if (lower_32_bits(address) & 0xf)
+		return NULL;
+
+	offset = address - udc_ep->tran_ring_info.dma_addr;
+	if (unlikely(offset >= udc_ep->tran_ring_info.len))
+		return NULL;
+	offset = offset / sizeof(struct crg_transfer_trb);
+	trb_virt = udc_ep->first_trb + offset;
+	return trb_virt;
+}
+
+dma_addr_t crg_udc_tran_trb_virt_to_dma(struct crg_udc_ep *udc_ep,
+					struct crg_transfer_trb *trb)
+{
+	unsigned long offset;
+	int trb_idx;
+	dma_addr_t dma_addr = 0;
+
+	trb_idx = trb - udc_ep->first_trb;
+	if (unlikely(trb_idx < 0))
+		return DMA_MAPPING_ERROR;
+
+	offset = trb_idx * sizeof(*trb);
+	if (unlikely(offset >= udc_ep->tran_ring_info.len))
+		return DMA_MAPPING_ERROR;
+	dma_addr = udc_ep->tran_ring_info.dma_addr + offset;
+	return dma_addr;
+}
+
+void crg_udc_update_dequeue_pt(struct crg_event_trb *event,
+			       struct crg_udc_ep *udc_ep)
+{
+	u32 deq_pt_lo = le32_to_cpu(event->dw0);
+	u32 deq_pt_hi = le32_to_cpu(event->dw1);
+	u64 dq_pt_addr = (u64)deq_pt_lo | ((u64)deq_pt_hi << 32);
+	struct crg_transfer_trb *deq_pt;
+
+	deq_pt = crg_udc_tran_trb_dma_to_virt(udc_ep, dq_pt_addr);
+	if (!deq_pt)
+		return;
+	deq_pt++;
+	if (deq_pt == udc_ep->last_trb)
+		deq_pt = udc_ep->first_trb;
+
+	if (FIELD_GET(TRB_TYPE_MASK, le32_to_cpu(deq_pt->dw3)) == TRB_TYPE_LINK)
+		deq_pt = udc_ep->first_trb;
+
+	udc_ep->deq_pt = deq_pt;
+}
+
+u32 crg_udc_trb_chunk_len(dma_addr_t dma, u64 remaining)
+{
+	u32 boundary = CRG_UDC_TRB_MAX_BUFFER_SIZE -
+		       (lower_32_bits(dma) & (CRG_UDC_TRB_MAX_BUFFER_SIZE - 1));
+
+	return min_t(u64, remaining, boundary);
+}
+
+u32 crg_udc_actual_before(struct crg_udc_ep *udc_ep,
+			  struct crg_udc_request *udc_req,
+			  struct crg_transfer_trb *pause)
+{
+	struct crg_transfer_trb *trb = udc_req->first_trb;
+	u32 actual = 0;
+	u32 guard = 0;
+	u32 max_iter = udc_ep->tran_ring_info.len / sizeof(*trb);
+
+	if (!trb || !pause)
+		return 0;
+
+	while (trb != pause && trb != udc_ep->enq_pt && guard++ < max_iter) {
+		u32 len;
+
+		if (FIELD_GET(TRB_TYPE_MASK, le32_to_cpu(trb->dw3)) ==
+		    TRB_TYPE_LINK) {
+			trb = udc_ep->first_trb;
+			continue;
+		}
+		len = FIELD_GET(TRB_TRANSFER_LEN_MASK, le32_to_cpu(trb->dw2));
+		actual += len;
+		trb++;
+		if (trb == udc_ep->last_trb)
+			trb = udc_ep->first_trb;
+	}
+
+	return min(actual, udc_req->usb_req.length);
+}

-- 
2.34.1

[PATCH v3 4/6] usb: gadget: udc: crg: Add endpoint 0 handling

[Petar Stepanovic]

This patch adds the endpoint 0 setup packet, status, and control
transfer handling for the Axiado AX3000/AX3005 UDC.

Co-developed-by: Naveen Kumar Rajgiri Bassappa <rbnaveenkumar@axiado.com>
Signed-off-by: Naveen Kumar Rajgiri Bassappa <rbnaveenkumar@axiado.com>
Co-developed-by: Krutik Shah <krutikshah@axiado.com>
Signed-off-by: Krutik Shah <krutikshah@axiado.com>
Signed-off-by: Petar Stepanovic <pstepanovic@axiado.com>
---
 drivers/usb/gadget/udc/crg/crg-udc-ep0.c | 1106 ++++++++++++++++++++++++++++++
 1 file changed, 1106 insertions(+)

diff --git a/drivers/usb/gadget/udc/crg/crg-udc-ep0.c b/drivers/usb/gadget/udc/crg/crg-udc-ep0.c
new file mode 100644
index 000000000000..dce36d23d3ec
--- /dev/null
+++ b/drivers/usb/gadget/udc/crg/crg-udc-ep0.c
@@ -0,0 +1,1106 @@
+// SPDX-License-Identifier: GPL-2.0+
+/*
+ * Axiado AX3000/AX3005 USB Device Controller driver.
+ *
+ * Copyright (c) 2019 Corigine Inc.
+ * Copyright (c) 2021-2026 Axiado Corporation.
+ */
+#include <linux/iopoll.h>
+
+#include "crg-udc.h"
+#include "crg-udc-trace.h"
+
+static void enable_setup_event(struct crg_gadget_dev *crg_udc);
+
+void crg_udc_build_ep0_status(struct crg_udc_ep *udc_ep_ptr, bool default_value,
+			      u32 status, struct crg_udc_request *udc_req_ptr,
+			      u8 set_addr, u8 stall)
+{
+	struct crg_gadget_dev *crg_udc = udc_ep_ptr->crg_udc;
+	struct crg_transfer_trb *enq_pt = udc_ep_ptr->enq_pt;
+
+	if (default_value) {
+		udc_req_ptr = crg_udc->status_req;
+		udc_req_ptr->usb_req.length = 0;
+		udc_req_ptr->usb_req.status = status;
+		udc_req_ptr->usb_req.actual = 0;
+		udc_req_ptr->usb_req.complete = NULL;
+	} else {
+		udc_req_ptr->usb_req.status = status;
+		udc_req_ptr->usb_req.actual = 0;
+	}
+
+	crg_udc_setup_status_trb(crg_udc, enq_pt, &udc_req_ptr->usb_req,
+				 udc_ep_ptr->pcs, set_addr, stall);
+
+	enq_pt++;
+
+	/* check if we are at end of trb segment.  If so, update
+	 * pcs and enq for next segment
+	 */
+	if (FIELD_GET(TRB_TYPE_MASK, le32_to_cpu(enq_pt->dw3)) ==
+	    TRB_TYPE_LINK) {
+		if (FIELD_GET(TRB_LINK_TOGGLE_CYCLE_MASK,
+			      le32_to_cpu(enq_pt->dw3))) {
+			dma_wmb();
+			le32p_replace_bits(&enq_pt->dw3, udc_ep_ptr->pcs,
+					   TRB_CYCLE_BIT_MASK);
+			udc_ep_ptr->pcs ^= 0x1;
+		}
+		enq_pt = udc_ep_ptr->first_trb;
+	}
+	udc_ep_ptr->enq_pt = enq_pt;
+
+	crg_udc_knock_doorbell(crg_udc, 0);
+
+	list_add_tail(&udc_req_ptr->queue, &udc_ep_ptr->queue);
+}
+
+/*
+ * Abort the current EP0 control transfer when a new SETUP packet arrives.
+ *
+ * A SETUP packet is allowed to abort any in-progress control transfer. Do
+ * not defer the new SETUP until the old transfer completes: doing so leaves
+ * the old request on the EP0 software queue and subsequent gadget requests
+ * are rejected as "queue not empty".
+ *
+ * The controller's EP0 ring is reinitialized at the ring start, making all
+ * TRBs belonging to the aborted control transfer stale. The new SETUP is
+ * then handled immediately with a fresh setup tag.
+ */
+int crg_udc_abort_ep0_for_setup(struct crg_gadget_dev *crg_udc)
+{
+	struct crg_udc_ep *udc_ep0 = &crg_udc->udc_ep[0];
+	struct crg_uccr __iomem *uccr = crg_udc->uccr;
+	dma_addr_t dqptaddr;
+	u32 cmd_param0;
+	u32 cmd_param1;
+	u32 tmp;
+	int ret;
+
+	/*
+	 * A new SETUP is allowed to abort the current control transfer.
+	 * Keep EP0 in ABORTING for the complete stop/nuke/reinitialize
+	 * sequence. crg_udc_req_done() drops udc_lock while invoking the gadget
+	 * callback, so the callback may re-enter the UDC during the nuke.
+	 */
+	crg_udc->setup_status = WAIT_FOR_SETUP;
+	crg_udc->ep0_state = EP0_STATE_ABORTING;
+
+	/*
+	 * Stop EP0 before touching the transfer ring. The controller may still
+	 * own the old TRBs even when no EP0 request is running from the gadget's
+	 * point of view. Do not clear the DMA-visible ring until the hardware
+	 * reports that EP0 is no longer running.
+	 */
+	ret = crg_udc_issue_command(crg_udc, CRG_CMD_STOP_EP, BIT(0), 0);
+	if (ret) {
+		dev_err(crg_udc->dev,
+			"failed to stop EP0 for SETUP abort: %d\n", ret);
+		crg_udc->ep0_state = EP0_STATE_ERROR;
+		return ret;
+	}
+
+	ret = readl_poll_timeout_atomic(&uccr->ep_running, tmp, !(tmp & BIT(0)),
+					10, 10000);
+	if (ret) {
+		dev_err(crg_udc->dev, "timeout stopping EP0 for SETUP abort\n");
+		crg_udc->ep0_state = EP0_STATE_ERROR;
+		return ret;
+	}
+
+	udc_ep0->ep_state = EP_STATE_STOPPED;
+	/* Complete requests belonging to the aborted control transfer. */
+	crg_udc_ep_nuke(udc_ep0, -ECONNRESET);
+
+	if (!udc_ep0->first_trb || !udc_ep0->tran_ring_info.cpu_addr) {
+		crg_udc->ep0_state = EP0_STATE_ERROR;
+		return -ENODEV;
+	}
+
+	/*
+	 * The controller has stopped consuming EP0 TRBs. It is now safe to
+	 * discard the old transfer descriptors and restart the ring from its
+	 * beginning. No ring memory is freed here.
+	 */
+	memset(udc_ep0->first_trb, 0, udc_ep0->tran_ring_info.len);
+	udc_ep0->enq_pt = udc_ep0->first_trb;
+	udc_ep0->deq_pt = udc_ep0->first_trb;
+	udc_ep0->pcs = 1;
+	udc_ep0->tran_ring_full = false;
+	crg_udc_setup_link_trb(udc_ep0->last_trb, true,
+			       udc_ep0->tran_ring_info.dma_addr);
+
+	/* Make the new ring contents visible before handing the ring to HW. */
+	dma_wmb();
+
+	dqptaddr = crg_udc_tran_trb_virt_to_dma(udc_ep0, udc_ep0->first_trb);
+	if (dma_mapping_error(crg_udc->dev, dqptaddr)) {
+		crg_udc->ep0_state = EP0_STATE_ERROR;
+		return -EIO;
+	}
+
+	cmd_param0 = (lower_32_bits(dqptaddr) & CRG_CMD0_0_DQPTRLO_MASK) |
+		     FIELD_PREP(CRG_CMD0_0_DCS_MASK, udc_ep0->pcs);
+	cmd_param1 = upper_32_bits(dqptaddr);
+
+	ret = crg_udc_issue_command(crg_udc, CRG_CMD_INIT_EP0, cmd_param0,
+				    cmd_param1);
+	if (ret) {
+		dev_err(crg_udc->dev,
+			"failed to reinitialize EP0 for SETUP: %d\n", ret);
+		crg_udc->ep0_state = EP0_STATE_ERROR;
+		return ret;
+	}
+
+	udc_ep0->ep_state = EP_STATE_RUNNING;
+	crg_udc->ep0_state = EP0_STATE_IDLE;
+	return 0;
+}
+
+int crg_udc_enable_setup(struct crg_gadget_dev *crg_udc)
+{
+	enable_setup_event(crg_udc);
+	crg_udc->device_state = USB_STATE_DEFAULT;
+	usb_gadget_set_state(&crg_udc->gadget, USB_STATE_DEFAULT);
+	crg_udc->setup_status = WAIT_FOR_SETUP;
+
+	return 0;
+}
+
+static void enable_setup_event(struct crg_gadget_dev *crg_udc)
+{
+	struct crg_uccr __iomem *uccr = crg_udc->uccr;
+	u32 val;
+
+	val = readl(&uccr->config1);
+	val |= CRG_U3DC_CFG1_SETUP_EVENT_EN;
+	writel(val, &uccr->config1);
+}
+
+void crg_udc_ep0_req_complete(struct crg_udc_ep *udc_ep_ptr,
+			      unsigned long *flags)
+{
+	struct crg_gadget_dev *crg_udc = udc_ep_ptr->crg_udc;
+
+	switch (crg_udc->setup_status) {
+	case DATA_STAGE_XFER:
+		crg_udc->setup_status = STATUS_STAGE_RECV;
+		crg_udc_build_ep0_status(udc_ep_ptr, true, -EINPROGRESS, NULL,
+					 0, 0);
+		break;
+	case DATA_STAGE_RECV:
+		crg_udc->setup_status = STATUS_STAGE_XFER;
+		crg_udc_build_ep0_status(udc_ep_ptr, true, -EINPROGRESS, NULL,
+					 0, 0);
+		break;
+	default:
+		if (crg_udc->setup_fn_call_back) {
+			crg_udc->setup_fn_call_back(crg_udc);
+			crg_udc->setup_fn_call_back = NULL;
+		}
+		crg_udc->setup_status = WAIT_FOR_SETUP;
+		break;
+	}
+}
+
+static void get_status_cmpl(struct usb_ep *ep, struct usb_request *req)
+{
+	struct crg_udc_ep *udc_ep = container_of(ep, struct crg_udc_ep, usb_ep);
+	struct crg_gadget_dev *crg_udc = udc_ep->crg_udc;
+	unsigned long flags;
+
+	kfree(req->buf);
+	req->buf = NULL;
+
+	/*
+	 * A non-zero status means this request was completed early (e.g.
+	 * crg_udc_ep_nuke(..., -ECONNRESET) from crg_udc_abort_ep0_for_setup()
+	 * for a new SETUP). EP0 is being reinitialized for that new control
+	 * transfer; queuing a status stage here would race with it.
+	 */
+	if (req->status)
+		return;
+
+	spin_lock_irqsave(&crg_udc->udc_lock, flags);
+	crg_udc->setup_status = STATUS_STAGE_RECV;
+	crg_udc_build_ep0_status(udc_ep, true, -EINPROGRESS, NULL, 0, 0);
+	spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+}
+
+static void getstatusrequest(struct crg_gadget_dev *crg_udc, u8 request_type,
+			     u16 value, u16 index, u16 length)
+{
+	struct crg_udc_ep *udc_ep_ptr = &crg_udc->udc_ep[0];
+	struct crg_udc_request *udc_req_ptr = crg_udc->status_req;
+	u16 status_val = 0;
+	u32 dci;
+	int ret;
+
+	if (!udc_req_ptr || !list_empty(&udc_req_ptr->queue) || value ||
+	    length != sizeof(u16)) {
+		crg_udc_set_ep0_halt(crg_udc);
+		return;
+	}
+
+	switch (request_type & USB_RECIP_MASK) {
+	case USB_RECIP_DEVICE:
+		if (index)
+			goto invalid;
+		if (crg_udc->gadget.speed == USB_SPEED_HIGH ||
+		    crg_udc->gadget.speed == USB_SPEED_FULL) {
+			if (crg_udc->u2_rwe)
+				status_val |= BIT(USB_DEVICE_REMOTE_WAKEUP);
+		} else if (crg_udc->gadget.speed >= USB_SPEED_SUPER) {
+			if (crg_udc->feature_u1_enable)
+				status_val |= BIT(USB_DEV_STAT_U1_ENABLED);
+			if (crg_udc->feature_u2_enable)
+				status_val |= BIT(USB_DEV_STAT_U2_ENABLED);
+		}
+		if (crg_udc->gadget.is_selfpowered)
+			status_val |= BIT(USB_DEVICE_SELF_POWERED);
+		break;
+
+	case USB_RECIP_INTERFACE:
+		if (crg_udc->device_state != USB_STATE_CONFIGURED ||
+		    (index & 0xff00))
+			goto invalid;
+		break;
+
+	case USB_RECIP_ENDPOINT:
+		if (index & ~(USB_ENDPOINT_NUMBER_MASK | USB_DIR_IN))
+			goto invalid;
+		dci = crg_udc_index2dci(index);
+		if (dci == 1 || dci >= CRG_NUM_EP_CX)
+			goto invalid;
+		if (crg_udc->device_state < USB_STATE_ADDRESS && dci != 0)
+			goto invalid;
+		if (crg_udc->device_state == USB_STATE_ADDRESS && dci != 0)
+			goto invalid;
+		if (crg_udc->device_state == USB_STATE_CONFIGURED &&
+		    crg_udc_get_ep_state(crg_udc, dci) == EP_STATE_DISABLED)
+			goto invalid;
+		if (crg_udc_get_ep_state(crg_udc, dci) == EP_STATE_HALTED)
+			status_val = BIT(USB_ENDPOINT_HALT);
+		break;
+
+	default:
+		goto invalid;
+	}
+
+	kfree(udc_req_ptr->usb_req.buf);
+
+	udc_req_ptr->usb_req.buf = kzalloc_obj(status_val, GFP_ATOMIC);
+	if (!udc_req_ptr->usb_req.buf)
+		goto invalid;
+
+	*(__le16 *)udc_req_ptr->usb_req.buf = cpu_to_le16(status_val);
+	udc_req_ptr->usb_req.length = sizeof(status_val);
+	udc_req_ptr->usb_req.status = -EINPROGRESS;
+	udc_req_ptr->usb_req.actual = 0;
+	udc_req_ptr->usb_req.complete = get_status_cmpl;
+
+	udc_req_ptr->map_dir = DMA_TO_DEVICE;
+	ret = usb_gadget_map_request_by_dev(crg_udc->dev, &udc_req_ptr->usb_req,
+					    true);
+	if (ret) {
+		kfree(udc_req_ptr->usb_req.buf);
+		udc_req_ptr->usb_req.buf = NULL;
+		goto invalid;
+	}
+
+	crg_udc->setup_status = DATA_STAGE_XFER;
+
+	ret = crg_udc_build_td(udc_ep_ptr, udc_req_ptr);
+	if (ret) {
+		crg_udc_unmap_request(udc_ep_ptr, udc_req_ptr);
+		kfree(udc_req_ptr->usb_req.buf);
+		udc_req_ptr->usb_req.buf = NULL;
+		crg_udc_set_ep0_halt(crg_udc);
+		return;
+	}
+	list_add_tail(&udc_req_ptr->queue, &udc_ep_ptr->queue);
+	return;
+
+invalid:
+	crg_udc_set_ep0_halt(crg_udc);
+}
+
+int crg_udc_prepare_for_setup(struct crg_gadget_dev *crg_udc)
+{
+	struct crg_udc_ep *udc_ep0_ptr;
+
+	dma_addr_t dqptaddr;
+	u32 cmd_param0;
+	u32 cmd_param1;
+
+	if (!crg_udc_is_event_rings_empty(crg_udc) ||
+	    crg_udc->portsc_on_reconnecting == 1)
+		return -EBUSY;
+
+	udc_ep0_ptr = &crg_udc->udc_ep[0];
+	/* If we reinit ep0 on bus reset, we just make ep0 dequeue pointer
+	 * align with enqueue pointer, all remaining xfer trbs became dumb once
+	 * which will not produce xfer event anymore.
+	 *
+	 * If we considering the opposite solution, we should wait all ep0 xfer
+	 * trbs be completed(with some err complete code)
+	 */
+	if (crg_udc->reinit_ep0_on_bus_reset) {
+		/* Complete any reqs on EP0 queue */
+		crg_udc_ep_nuke(udc_ep0_ptr, -ESHUTDOWN);
+
+		/* context related ops */
+		dqptaddr = crg_udc_tran_trb_virt_to_dma(udc_ep0_ptr,
+							udc_ep0_ptr->enq_pt);
+		if (dma_mapping_error(crg_udc->dev, dqptaddr))
+			return -EIO;
+		cmd_param0 =
+			(lower_32_bits(dqptaddr) & CRG_CMD0_0_DQPTRLO_MASK) |
+			FIELD_PREP(CRG_CMD0_0_DCS_MASK, udc_ep0_ptr->pcs);
+		cmd_param1 = upper_32_bits(dqptaddr);
+		if (crg_udc_issue_command(crg_udc, CRG_CMD_INIT_EP0, cmd_param0,
+					  cmd_param1))
+			return -EIO;
+
+		udc_ep0_ptr->deq_pt = udc_ep0_ptr->enq_pt;
+		udc_ep0_ptr->tran_ring_full = false;
+	} else {
+		if (!list_empty(&udc_ep0_ptr->queue)) {
+			dev_err(crg_udc->dev,
+				"remaining ep0 xfer trbs on reset\n");
+			return -EBUSY;
+		}
+	}
+
+	crg_udc_enable_setup(crg_udc);
+
+	return 0;
+}
+
+static void set_address_cmpl(struct crg_gadget_dev *crg_udc)
+{
+	if (crg_udc->device_state == USB_STATE_DEFAULT &&
+	    crg_udc->dev_addr != 0) {
+		crg_udc->device_state = USB_STATE_ADDRESS;
+		usb_gadget_set_state(&crg_udc->gadget, USB_STATE_ADDRESS);
+	} else if (crg_udc->device_state == USB_STATE_ADDRESS) {
+		if (crg_udc->dev_addr == 0) {
+			crg_udc->device_state = USB_STATE_DEFAULT;
+			usb_gadget_set_state(&crg_udc->gadget,
+					     USB_STATE_DEFAULT);
+		}
+
+	}
+}
+
+int crg_udc_set_ep0_halt(struct crg_gadget_dev *crg_udc)
+{
+	struct crg_udc_ep *udc_ep_ptr = &crg_udc->udc_ep[0];
+	enum crg_ep_state ep_state;
+
+	ep_state = crg_udc_get_ep_state(crg_udc, udc_ep_ptr->dci);
+	if (ep_state == EP_STATE_HALTED || ep_state == EP_STATE_DISABLED)
+		return 0;
+
+	crg_udc_build_ep0_status(udc_ep_ptr, true, -EINVAL, NULL, 0, 1);
+
+	udc_ep_ptr->ep_state = EP_STATE_HALTED;
+
+	return 0;
+}
+
+static void set_sel_cmpl(struct usb_ep *ep, struct usb_request *req)
+{
+	struct crg_udc_ep *udc_ep;
+	struct crg_gadget_dev *crg_udc;
+	struct crg_sel_value *sel_value;
+
+	udc_ep = container_of(ep, struct crg_udc_ep, usb_ep);
+	crg_udc = udc_ep->crg_udc;
+
+	if (req->buf) {
+		if (!req->status && req->actual >= sizeof(*sel_value)) {
+			sel_value = req->buf;
+			crg_udc->sel_value.u2_pel_value =
+				le16_to_cpu(sel_value->u2_pel_value);
+			crg_udc->sel_value.u2_sel_value =
+				le16_to_cpu(sel_value->u2_sel_value);
+			crg_udc->sel_value.u1_pel_value = sel_value->u1_pel_value;
+			crg_udc->sel_value.u1_sel_value = sel_value->u1_sel_value;
+		}
+
+		kfree(req->buf);
+		req->buf = NULL;
+	}
+}
+
+static void set_test_mode_cmpl(struct crg_gadget_dev *crg_udc)
+{
+	if (crg_udc->set_tm != 0) {
+		u32 tmp;
+		struct crg_uccr __iomem *uccr = crg_udc->uccr;
+
+		tmp = readl(&uccr->u2portpmsc);
+		FIELD_MODIFY(CRG_U3DC_U2PORTPM_TM_MASK, &tmp, crg_udc->set_tm);
+		writel(tmp, &uccr->u2portpmsc);
+
+		crg_udc->set_tm = 0;
+	}
+}
+
+static void setaddressrequest(struct crg_gadget_dev *crg_udc, u16 value,
+			      u16 index, u16 length)
+{
+	int status = -EINPROGRESS;
+	u8 status_set_addr = 0;
+
+	if (value > 127 || index != 0 || length != 0) {
+		status = -EINVAL;
+		goto set_address_error;
+	}
+
+	if ((crg_udc->device_state == USB_STATE_DEFAULT && value != 0) ||
+	    crg_udc->device_state == USB_STATE_ADDRESS) {
+		u32 param0;
+
+		param0 = FIELD_PREP(CRG_CMD2_0_DEV_ADDR_MASK, value);
+		if (crg_udc_issue_command(crg_udc, CRG_CMD_SET_ADDR, param0,
+					  0)) {
+			status = -EIO;
+		} else {
+			crg_udc->dev_addr = value;
+			status_set_addr = 1;
+		}
+	} else {
+		status = -EINVAL;
+	}
+
+set_address_error:
+	dev_dbg(crg_udc->dev, "crg_udc_build_ep0_status for Address Device\n");
+
+	crg_udc->setup_status = STATUS_STAGE_XFER;
+	crg_udc->setup_fn_call_back = &set_address_cmpl;
+	crg_udc_build_ep0_status(&crg_udc->udc_ep[0], true, status, NULL,
+				 status_set_addr, status != -EINPROGRESS);
+}
+
+static bool setfeaturesrequest(struct crg_gadget_dev *crg_udc, u8 request_type,
+			       u8 b_request, u16 value, u16 index, u16 length,
+			       unsigned long *flags)
+{
+	int status = -EINPROGRESS;
+	u8 dci;
+	struct crg_udc_ep *udc_ep_ptr;
+	u32 tmp;
+	u32 set_feat = 0;
+	struct crg_uccr __iomem *uccr = crg_udc->uccr;
+
+	if (length != 0) {
+		status = -EINVAL;
+		goto set_feature_error;
+	}
+
+	if (crg_udc->device_state == USB_STATE_DEFAULT) {
+		status = -EINVAL;
+		goto set_feature_error;
+	}
+
+	set_feat = (b_request == USB_REQ_SET_FEATURE) ? 1 : 0;
+	if ((request_type & (USB_RECIP_MASK | USB_TYPE_MASK)) ==
+	    (USB_RECIP_ENDPOINT | USB_TYPE_STANDARD)) {
+		dev_dbg(crg_udc->dev, "Halt/Unhalt EP\n");
+		if (crg_udc->device_state == USB_STATE_ADDRESS) {
+			if (index != 0) {
+				status = -EINVAL;
+				goto set_feature_error;
+			}
+		}
+
+		if (value != USB_ENDPOINT_HALT)
+			goto set_feature_error;
+
+		if (index & ~(USB_ENDPOINT_NUMBER_MASK | USB_DIR_IN))
+			goto set_feature_error;
+
+		dci = crg_udc_index2dci(index);
+		if (dci == 1 || dci >= CRG_NUM_EP_CX)
+			goto set_feature_error;
+		if (dci != 0 && crg_udc->device_state == USB_STATE_CONFIGURED &&
+		    crg_udc_get_ep_state(crg_udc, dci) == EP_STATE_DISABLED)
+			goto set_feature_error;
+
+		udc_ep_ptr = &crg_udc->udc_ep[dci];
+		status = crg_udc_ep_halt(
+			udc_ep_ptr, (b_request == USB_REQ_SET_FEATURE) ? 1 : 0,
+			0, flags);
+
+		if (status < 0)
+			goto set_feature_error;
+	} else if ((request_type & (USB_RECIP_MASK | USB_TYPE_MASK)) ==
+		   (USB_RECIP_DEVICE | USB_TYPE_STANDARD)) {
+		switch (value) {
+		case USB_DEVICE_REMOTE_WAKEUP:
+			/* REMOTE_WAKEUP selector is not used by USB3.0 */
+			if (index ||
+			    crg_udc->device_state < USB_STATE_DEFAULT ||
+			    crg_udc->gadget.speed >= USB_SPEED_SUPER) {
+				status = -EINVAL;
+				goto set_feature_error;
+			}
+
+			crg_udc->u2_rwe = set_feat;
+			crg_udc->gadget.wakeup_armed = set_feat;
+
+			break;
+		case USB_DEVICE_U1_ENABLE:
+		case USB_DEVICE_U2_ENABLE: {
+			u32 timeout_val;
+
+			if (crg_udc->device_state != USB_STATE_CONFIGURED) {
+				dev_err(crg_udc->dev,
+					"u12 enable fail, usb state=%d\n",
+					crg_udc->device_state);
+				status = -EINVAL;
+				goto set_feature_error;
+			}
+
+			if (index != 0 ||
+			    crg_udc->gadget.speed < USB_SPEED_SUPER) {
+				status = -EINVAL;
+				goto set_feature_error;
+			}
+
+			if (set_feat == 1 && crg_udc->u12_forbidden) {
+				status = -EINVAL;
+				goto set_feature_error;
+			}
+
+			tmp = readl(&uccr->u3portpmsc);
+			if (value == USB_DEVICE_U1_ENABLE) {
+				if (!crg_udc->u12_initiate_forbidden) {
+					tmp &= (~CRG_U3DC_U3PORTPM_U1IEN);
+					tmp |= FIELD_PREP(
+						CRG_U3DC_U3PORTPM_U1IEN,
+						set_feat);
+				}
+				if (!crg_udc->u12_forbidden)
+					timeout_val =
+						set_feat ? CRG_UDC_U1_TIMEOUT_VAL : 0;
+				else
+					timeout_val = 0;
+
+				FIELD_MODIFY(CRG_U3DC_U3PORTPM_U1TMOUT_MASK,
+					     &tmp, timeout_val);
+
+				crg_udc->feature_u1_enable = set_feat;
+			}
+
+			if (value == USB_DEVICE_U2_ENABLE) {
+				if (!crg_udc->u12_initiate_forbidden) {
+					tmp &= (~CRG_U3DC_U3PORTPM_U2IEN);
+					tmp |= FIELD_PREP(
+						CRG_U3DC_U3PORTPM_U2IEN,
+						set_feat);
+				}
+				if (!crg_udc->u12_forbidden)
+					timeout_val =
+						set_feat ? CRG_UDC_U2_TIMEOUT_VAL : 0;
+				else
+					timeout_val = 0;
+
+				FIELD_MODIFY(CRG_U3DC_U3PORTPM_U2TMOUT_MASK,
+					     &tmp, timeout_val);
+
+				crg_udc->feature_u2_enable = set_feat;
+			}
+			writel(tmp, &uccr->u3portpmsc);
+
+			break;
+		}
+		case USB_DEVICE_TEST_MODE: {
+			u32 u_pattern;
+
+			if (b_request != USB_REQ_SET_FEATURE ||
+			    crg_udc->gadget.speed != USB_SPEED_HIGH ||
+			    crg_udc->device_state < USB_STATE_DEFAULT ||
+			    (index & 0xff) || (index >> 8) == 0 ||
+			    (index >> 8) > 6)
+				goto set_feature_error;
+			u_pattern = index >> 8;
+			crg_udc->set_tm = u_pattern;
+			crg_udc->setup_fn_call_back = &set_test_mode_cmpl;
+
+			break;
+		}
+
+		default:
+			goto set_feature_error;
+		}
+
+	} else if ((request_type & (USB_RECIP_MASK | USB_TYPE_MASK)) ==
+		   (USB_RECIP_INTERFACE | USB_TYPE_STANDARD)) {
+		if (crg_udc->device_state != USB_STATE_CONFIGURED) {
+			dev_err(crg_udc->dev,
+				"interface u12 enable fail, usb state=%d\n",
+				crg_udc->device_state);
+			status = -EINVAL;
+			goto set_feature_error;
+		}
+
+		/* Suspend Option */
+		if (value != USB_INTRF_FUNC_SUSPEND) {
+			status = -EINVAL;
+			goto set_feature_error;
+		}
+
+		if (!(index & USB_INTR_FUNC_SUSPEND_OPT_MASK &
+		      USB_INTRF_FUNC_SUSPEND_LP)) {
+			status = -EINVAL;
+			goto set_feature_error;
+		}
+
+		if (index & USB_INTRF_FUNC_SUSPEND_RW)
+			dev_dbg(crg_udc->dev, "Interface En Remote Wakeup\n");
+		else
+			dev_dbg(crg_udc->dev, "Interface Dis RemoteWakeup\n");
+
+		/* Do not need to return status stage here
+		 * Pass to composite gadget driver to process
+		 * the request
+		 */
+		return false;
+	}
+
+	crg_udc->setup_status = STATUS_STAGE_XFER;
+	crg_udc_build_ep0_status(&crg_udc->udc_ep[0], true, status, NULL, 0, 0);
+	return true;
+
+set_feature_error:
+	crg_udc_set_ep0_halt(crg_udc);
+	return true;
+}
+
+static void setselrequest(struct crg_gadget_dev *crg_udc, u16 value, u16 index,
+			  u16 length)
+{
+	struct crg_udc_request *udc_req_ptr = crg_udc->status_req;
+	struct crg_udc_ep *udc_ep_ptr = &crg_udc->udc_ep[0];
+	struct crg_sel_value *sel;
+	int ret;
+
+	if (!udc_req_ptr || !list_empty(&udc_req_ptr->queue) ||
+	    crg_udc->device_state == USB_STATE_DEFAULT || value || index ||
+	    length != sizeof(struct crg_sel_value)) {
+		crg_udc_set_ep0_halt(crg_udc);
+		return;
+	}
+
+	kfree(udc_req_ptr->usb_req.buf);
+
+	udc_req_ptr->usb_req.buf = kzalloc_obj(*sel, GFP_ATOMIC);
+	if (!udc_req_ptr->usb_req.buf) {
+		crg_udc_set_ep0_halt(crg_udc);
+		return;
+	}
+
+	sel = udc_req_ptr->usb_req.buf;
+	udc_req_ptr->usb_req.length = sizeof(*sel);
+	udc_req_ptr->usb_req.status = -EINPROGRESS;
+	udc_req_ptr->usb_req.actual = 0;
+	udc_req_ptr->usb_req.complete = set_sel_cmpl;
+	udc_req_ptr->map_dir = DMA_FROM_DEVICE;
+	ret = usb_gadget_map_request_by_dev(crg_udc->dev, &udc_req_ptr->usb_req,
+					    false);
+	if (ret) {
+		kfree(udc_req_ptr->usb_req.buf);
+		udc_req_ptr->usb_req.buf = NULL;
+		crg_udc_set_ep0_halt(crg_udc);
+		return;
+	}
+
+	crg_udc->setup_status = DATA_STAGE_RECV;
+
+	ret = crg_udc_build_td(udc_ep_ptr, udc_req_ptr);
+	if (ret) {
+		crg_udc_unmap_request(udc_ep_ptr, udc_req_ptr);
+		kfree(udc_req_ptr->usb_req.buf);
+		udc_req_ptr->usb_req.buf = NULL;
+		crg_udc_set_ep0_halt(crg_udc);
+		return;
+	}
+	list_add_tail(&udc_req_ptr->queue, &udc_ep_ptr->queue);
+}
+
+int crg_udc_update_ep0_maxpacketsize(struct crg_gadget_dev *crg_udc)
+{
+	u16 maxpacketsize = 0;
+	struct crg_udc_ep *udc_ep0 = &crg_udc->udc_ep[0];
+	u32 param0;
+
+	if (crg_udc->gadget.speed >= USB_SPEED_SUPER)
+		maxpacketsize = 512;
+	else
+		maxpacketsize = 64;
+
+	param0 = FIELD_PREP(CRG_CMD1_0_MPS_MASK, maxpacketsize);
+	if (crg_udc_issue_command(crg_udc, CRG_CMD_UPDATE_EP0_CFG, param0, 0))
+		return -EIO;
+
+	crg_udc->ep0_desc.wMaxPacketSize = cpu_to_le16(maxpacketsize);
+	udc_ep0->usb_ep.maxpacket = maxpacketsize;
+
+	return 0;
+}
+
+void crg_udc_handle_setup_pkt(struct crg_gadget_dev *crg_udc,
+			      struct usb_ctrlrequest *setup_pkt, u8 setup_tag,
+			      unsigned long *flags)
+{
+	u16 w_value = le16_to_cpu(setup_pkt->wValue);
+	u16 w_index = le16_to_cpu(setup_pkt->wIndex);
+	u16 w_length = le16_to_cpu(setup_pkt->wLength);
+	struct usb_gadget_driver *driver;
+	u32 tmp;
+
+	trace_crg_udc_ctrl_req(setup_pkt);
+
+	/* EP0 come backs to running when new setup packet comes */
+	crg_udc->udc_ep[0].ep_state = EP_STATE_RUNNING;
+
+	crg_udc->setup_tag = setup_tag;
+	crg_udc->setup_status = SETUP_PKT_PROCESS_IN_PROGRESS;
+	crg_udc->setup_fn_call_back = NULL;
+
+	if ((setup_pkt->bRequestType & USB_TYPE_MASK) == USB_TYPE_STANDARD) {
+		switch (setup_pkt->bRequest) {
+		case USB_REQ_GET_STATUS:
+			if (setup_pkt->bRequestType !=
+				    (USB_DIR_IN | USB_RECIP_DEVICE |
+				     USB_TYPE_STANDARD) &&
+			    setup_pkt->bRequestType !=
+				    (USB_DIR_IN | USB_RECIP_INTERFACE |
+				     USB_TYPE_STANDARD) &&
+			    setup_pkt->bRequestType !=
+				    (USB_DIR_IN | USB_RECIP_ENDPOINT |
+				     USB_TYPE_STANDARD)) {
+				crg_udc_set_ep0_halt(crg_udc);
+				return;
+			}
+
+			getstatusrequest(crg_udc, setup_pkt->bRequestType,
+					 w_value, w_index, w_length);
+			return;
+		case USB_REQ_SET_ADDRESS:
+			if (setup_pkt->bRequestType !=
+			    (USB_DIR_OUT | USB_RECIP_DEVICE |
+			     USB_TYPE_STANDARD)) {
+				crg_udc_set_ep0_halt(crg_udc);
+				return;
+			}
+
+			setaddressrequest(crg_udc, w_value, w_index, w_length);
+			return;
+		case USB_REQ_SET_SEL:
+			if (setup_pkt->bRequestType !=
+			    (USB_DIR_OUT | USB_RECIP_DEVICE |
+			     USB_TYPE_STANDARD)) {
+				crg_udc_set_ep0_halt(crg_udc);
+				return;
+			}
+			crg_udc->setup_status = DATA_STAGE_RECV;
+			setselrequest(crg_udc, w_value, w_index, w_length);
+			return;
+		case USB_REQ_SET_ISOCH_DELAY:
+			if (setup_pkt->bRequestType !=
+				    (USB_DIR_OUT | USB_RECIP_DEVICE |
+				     USB_TYPE_STANDARD) ||
+			    w_index || w_length ||
+			    w_value >
+				    FIELD_MAX(
+					    CRG_U3DC_PTMCTRL_ISOCH_DELAY_MASK)) {
+				crg_udc_set_ep0_halt(crg_udc);
+				return;
+			}
+
+			tmp = readl(&crg_udc->uccr->ptmctrl);
+			FIELD_MODIFY(CRG_U3DC_PTMCTRL_ISOCH_DELAY_MASK, &tmp,
+				     w_value);
+			writel(tmp, &crg_udc->uccr->ptmctrl);
+			crg_udc->setup_status = STATUS_STAGE_XFER;
+			crg_udc_build_ep0_status(&crg_udc->udc_ep[0], true,
+						 -EINPROGRESS, NULL, 0, 0);
+			return;
+
+		case USB_REQ_CLEAR_FEATURE:
+		case USB_REQ_SET_FEATURE:
+			/* Need composite gadget driver
+			 * to process the function remote wakeup request
+			 */
+			if (setfeaturesrequest(crg_udc, setup_pkt->bRequestType,
+					       setup_pkt->bRequest, w_value,
+					       w_index, w_length, flags)) {
+				/* Get here if request has been processed.*/
+				return;
+			}
+			break;
+		case USB_REQ_SET_CONFIGURATION:
+			if (setup_pkt->bRequestType !=
+				    (USB_DIR_OUT | USB_RECIP_DEVICE |
+				     USB_TYPE_STANDARD) ||
+			    w_length || w_value > 0xff) {
+				crg_udc_set_ep0_halt(crg_udc);
+				return;
+			}
+			/* In theory we need to clear RUN bit before
+			 * status stage of deconfig request sent.
+			 * But seeing problem if we do it before all the
+			 * endpoints belonging to the configuration
+			 * disabled.
+			 */
+			if (crg_udc->device_state <= USB_STATE_DEFAULT) {
+				crg_udc_set_ep0_halt(crg_udc);
+				return;
+			}
+
+			break;
+		default:
+			dev_dbg(crg_udc->dev,
+				"USB_REQ default bRequest=%d, bRequestType=%d\n",
+				setup_pkt->bRequest, setup_pkt->bRequestType);
+		}
+	}
+
+	if (w_length) {
+		/* data phase from gadget like GET_CONFIGURATION
+		 * call the setup routine of gadget driver.
+		 * remember the request direction.
+		 */
+		crg_udc->setup_status = (setup_pkt->bRequestType & USB_DIR_IN) ?
+						DATA_STAGE_XFER :
+						DATA_STAGE_RECV;
+	}
+	driver = crg_udc->async_callbacks ? crg_udc->gadget_driver : NULL;
+	spin_unlock_irqrestore(&crg_udc->udc_lock, *flags);
+
+	/*
+	 * Make sure any reset notification queued by
+	 * crg_udc_handle_port_status() has actually run before this SETUP
+	 * (which may be SET_CONFIGURATION) reaches the function driver.
+	 * Otherwise udc_reset_work's usb_gadget_udc_reset() could run after
+	 * the new configuration is already active and tear it back down.
+	 */
+	flush_work(&crg_udc->udc_reset_work);
+
+	if (!driver || driver->setup(&crg_udc->gadget, setup_pkt) < 0) {
+		spin_lock_irqsave(&crg_udc->udc_lock, *flags);
+		crg_udc_set_ep0_halt(crg_udc);
+		return;
+	}
+	spin_lock_irqsave(&crg_udc->udc_lock, *flags);
+	if (setup_pkt->bRequest == USB_REQ_SET_CONFIGURATION &&
+	    (setup_pkt->bRequestType & (USB_TYPE_MASK | USB_RECIP_MASK)) ==
+		    (USB_TYPE_STANDARD | USB_RECIP_DEVICE)) {
+		if (w_value) {
+			crg_udc->device_state = USB_STATE_CONFIGURED;
+			usb_gadget_set_state(&crg_udc->gadget,
+					     USB_STATE_CONFIGURED);
+		} else {
+			/* Endpoint disable may already have updated device_state. */
+			crg_udc->device_state = USB_STATE_ADDRESS;
+			usb_gadget_set_state(&crg_udc->gadget,
+					     USB_STATE_ADDRESS);
+		}
+	}
+}
+
+int crg_udc_init_ep0(struct crg_gadget_dev *crg_udc)
+{
+	struct crg_udc_ep *udc_ep_ptr = &crg_udc->udc_ep[0];
+	u32 cmd_param0;
+	u32 cmd_param1;
+
+	/* setup transfer ring */
+	if (!udc_ep_ptr->tran_ring_info.cpu_addr) {
+		dma_addr_t dma;
+		u32 ring_size = CRG_UDC_CONTROL_EP_TD_RING_SIZE;
+		void *vaddr;
+		size_t len;
+
+		len = ring_size * sizeof(struct crg_transfer_trb);
+		vaddr = dma_alloc_coherent(crg_udc->dev, len, &dma, GFP_KERNEL);
+		if (!vaddr)
+			return -ENOMEM;
+
+		udc_ep_ptr->tran_ring_info.cpu_addr = vaddr;
+		udc_ep_ptr->tran_ring_info.dma_addr = dma;
+		udc_ep_ptr->tran_ring_info.len = len;
+		udc_ep_ptr->first_trb = vaddr;
+		udc_ep_ptr->last_trb = udc_ep_ptr->first_trb + ring_size - 1;
+	}
+
+	memset(udc_ep_ptr->first_trb, 0, udc_ep_ptr->tran_ring_info.len);
+	/* Ensure that transfer ring is updated */
+	dma_wmb();
+	udc_ep_ptr->enq_pt = udc_ep_ptr->first_trb;
+	udc_ep_ptr->deq_pt = udc_ep_ptr->first_trb;
+	udc_ep_ptr->pcs = 1;
+	udc_ep_ptr->tran_ring_full = false;
+
+	crg_udc_setup_link_trb(udc_ep_ptr->last_trb, true,
+			       udc_ep_ptr->tran_ring_info.dma_addr);
+
+	/* context related ops */
+	cmd_param0 = (lower_32_bits(udc_ep_ptr->tran_ring_info.dma_addr) &
+		      CRG_CMD0_0_DQPTRLO_MASK) |
+		     FIELD_PREP(CRG_CMD0_0_DCS_MASK, udc_ep_ptr->pcs);
+	cmd_param1 = upper_32_bits(udc_ep_ptr->tran_ring_info.dma_addr);
+
+	if (crg_udc_issue_command(crg_udc, CRG_CMD_INIT_EP0, cmd_param0,
+				  cmd_param1))
+		return -EIO;
+
+	udc_ep_ptr->desc = &crg_udc->ep0_desc;
+	udc_ep_ptr->ep_state = EP_STATE_RUNNING;
+	udc_ep_ptr->pcs = 1;
+
+	return 0;
+}
+
+int crg_udc_queue_ctrl(struct crg_udc_ep *udc_ep_ptr,
+		       struct crg_udc_request *udc_req_ptr,
+		       u32 num_of_trbs_needed)
+{
+	struct crg_gadget_dev *crg_udc = udc_ep_ptr->crg_udc;
+	enum crg_ep_state ep_state;
+	struct crg_transfer_trb *enq_pt = udc_ep_ptr->enq_pt;
+	struct crg_transfer_trb *dq_pt = udc_ep_ptr->deq_pt;
+	struct usb_request *usb_req = &udc_req_ptr->usb_req;
+	struct crg_transfer_trb *p_trb;
+	u32 transfer_length;
+	bool ioc;
+	bool azp;
+	u8 dir = 0;
+	u8 setup_tag = crg_udc->setup_tag;
+
+	/* A new SETUP aborts the previous EP0 transaction. The completion
+	 * callback of the aborted request is allowed to run, but it must not
+	 * requeue the old request before the new SETUP has been handed to the
+	 * gadget driver.
+	 */
+	if (crg_udc->ep0_state == EP0_STATE_ABORTING)
+		return -ECONNRESET;
+
+	ep_state = crg_udc_get_ep_state(crg_udc, 0);
+	/* Need to queue the request even ep is paused or halted */
+	if (ep_state != EP_STATE_RUNNING) {
+		dev_dbg(crg_udc->dev, "EP State = 0x%x\n", ep_state);
+		return -EINVAL;
+	}
+	if (list_empty(&udc_ep_ptr->queue)) {
+		/* For control endpoint, we can handle one setup request at a
+		 * time. so if there are TD pending in the transfer ring.
+		 * wait for the sequence number error event. Then put the new
+		 * request to transfer ring
+		 */
+		if (enq_pt == dq_pt) {
+			u32 i;
+			bool need_zlp = false;
+			dma_addr_t trb_dma = usb_req->dma;
+			u64 remaining = usb_req->length;
+			/* Transfer ring is empty
+			 * setup data stage TRBs
+			 */
+			udc_req_ptr->first_trb = udc_ep_ptr->enq_pt;
+			if (crg_udc->setup_status == DATA_STAGE_XFER)
+				dir = 0;
+			else if (crg_udc->setup_status == DATA_STAGE_RECV)
+				dir = 1;
+			else
+				dev_dbg(crg_udc->dev,
+					"unexpected setup_status!%d\n",
+					crg_udc->setup_status);
+			if (udc_req_ptr->usb_req.zero == 1 &&
+			    udc_req_ptr->usb_req.length != 0 &&
+			    ((udc_req_ptr->usb_req.length %
+			      udc_ep_ptr->usb_ep.maxpacket) == 0))
+				need_zlp = true; /* zero length packet */
+			if (num_of_trbs_needed > 32)
+				return -EMSGSIZE;
+
+			if (num_of_trbs_needed >
+			    crg_udc_room_on_ring(
+				    crg_udc, CRG_UDC_CONTROL_EP_TD_RING_SIZE,
+				    udc_ep_ptr->first_trb, enq_pt, dq_pt))
+				return -ENOSPC;
+
+			u8 original_pcs = udc_ep_ptr->pcs;
+
+			for (i = 0; i < num_of_trbs_needed; i++) {
+				p_trb = enq_pt;
+				transfer_length = crg_udc_trb_chunk_len(
+					trb_dma, remaining);
+				ioc = (i == num_of_trbs_needed - 1);
+				azp = ioc && need_zlp;
+				crg_udc_setup_datastage_trb(
+					p_trb, trb_dma,
+					i == 0 ? (original_pcs ^ 1) :
+						 udc_ep_ptr->pcs,
+					transfer_length,
+					num_of_trbs_needed - i - 1, ioc, azp,
+					dir, setup_tag);
+				udc_req_ptr->all_trbs_queued = 1;
+				trb_dma += transfer_length;
+				remaining -= transfer_length;
+				enq_pt++;
+				if (FIELD_GET(TRB_TYPE_MASK,
+					      le32_to_cpu(enq_pt->dw3)) ==
+				    TRB_TYPE_LINK) {
+					if (FIELD_GET(
+						    TRB_LINK_TOGGLE_CYCLE_MASK,
+						    le32_to_cpu(enq_pt->dw3))) {
+						dma_wmb();
+						le32p_replace_bits(
+							&enq_pt->dw3,
+							udc_ep_ptr->pcs,
+							TRB_CYCLE_BIT_MASK);
+						udc_ep_ptr->pcs ^= 0x1;
+					}
+					enq_pt = udc_ep_ptr->first_trb;
+				}
+			}
+			dma_wmb();
+			le32p_replace_bits(&udc_req_ptr->first_trb->dw3,
+					   original_pcs, TRB_CYCLE_BIT_MASK);
+			udc_ep_ptr->enq_pt = enq_pt;
+			crg_udc_knock_doorbell(crg_udc, 0);
+			if (udc_ep_ptr->enq_pt == udc_ep_ptr->first_trb)
+				udc_req_ptr->last_trb =
+					udc_ep_ptr->last_trb - 1;
+			else
+				udc_req_ptr->last_trb = udc_ep_ptr->enq_pt - 1;
+		} else {
+			/*
+			 * The software queue is empty but the transfer ring
+			 * still holds pending TDs from a previous control
+			 * transfer (the host issued a new setup before the
+			 * old one drained). We process one setup at a time,
+			 * so we intentionally do not queue TRBs here.
+			 * Returning success leaves this request on the
+			 * endpoint queue; the stale TDs and this request are
+			 * flushed and completed (with -EINVAL) by the
+			 * sequence-number-error / setup-tag-mismatch event in
+			 * crg_udc_handle_xfer_event().
+			 */
+			dev_err(crg_udc->dev,
+				"ep0 ring not idle on new setup\n");
+		}
+	} else {
+		dev_err(crg_udc->dev, "udc_ep_ptr->queue not empty\n");
+		/* New setup packet came
+		 * Drop this request.
+		 */
+		return -EINVAL;
+	}
+	return 0;
+}

-- 
2.34.1

[PATCH v3 5/6] usb: gadget: udc: crg: Add gadget and endpoint operations

[Petar Stepanovic]

This patch implements the gadget and endpoint operations for the
Axiado AX3000/AX3005 UDC driver.

Co-developed-by: Naveen Kumar Rajgiri Bassappa <rbnaveenkumar@axiado.com>
Signed-off-by: Naveen Kumar Rajgiri Bassappa <rbnaveenkumar@axiado.com>
Co-developed-by: Krutik Shah <krutikshah@axiado.com>
Signed-off-by: Krutik Shah <krutikshah@axiado.com>
Signed-off-by: Petar Stepanovic <pstepanovic@axiado.com>
---
 drivers/usb/gadget/udc/crg/crg-udc-gadget.c | 2299 +++++++++++++++++++++++++++
 drivers/usb/gadget/udc/crg/crg-udc-trace.c  |    9 +
 2 files changed, 2308 insertions(+)

diff --git a/drivers/usb/gadget/udc/crg/crg-udc-gadget.c b/drivers/usb/gadget/udc/crg/crg-udc-gadget.c
new file mode 100644
index 000000000000..a330e857a950
--- /dev/null
+++ b/drivers/usb/gadget/udc/crg/crg-udc-gadget.c
@@ -0,0 +1,2299 @@
+// SPDX-License-Identifier: GPL-2.0+
+/*
+ * Axiado AX3000/AX3005 USB Device Controller driver.
+ *
+ * Copyright (c) 2019 Corigine Inc.
+ * Copyright (c) 2021-2026 Axiado Corporation.
+ */
+#include <linux/iopoll.h>
+#include <linux/scatterlist.h>
+
+#include "crg-udc.h"
+#include "crg-udc-trace.h"
+
+/* Forward declarations for functions used before their definition. */
+static int crg_udc_epcx_setup(struct crg_udc_ep *udc_ep);
+static int crg_udc_epcx_update_dqptr(struct crg_udc_ep *udc_ep);
+static int crg_udc_queue_trbs(struct crg_udc_ep *udc_ep_ptr,
+			      struct crg_udc_request *udc_req_ptr, bool b_isoc,
+			      u32 xfer_ring_size, u32 num_trbs_needed,
+			      u64 buffer_length);
+static int crg_udc_reinit(struct crg_gadget_dev *crg_udc);
+static bool crg_udc_sg_advance(struct crg_udc_request *udc_req, u32 bytes);
+static bool crg_udc_sg_current(struct crg_udc_request *udc_req, dma_addr_t *dma,
+			       u32 *len);
+static int crg_udc_sg_init(struct crg_udc_request *udc_req);
+static int crg_udc_validate_ep_desc(struct crg_udc_ep *udc_ep,
+				    const struct usb_endpoint_descriptor *desc);
+static bool is_request_dequeued(struct crg_gadget_dev *crg_udc,
+				struct crg_udc_ep *udc_ep,
+				struct crg_event_trb *event);
+static int set_ep_halt(struct crg_gadget_dev *crg_udc, int dci,
+		       unsigned long *flags);
+
+static int crg_gadget_get_frame(struct usb_gadget *g)
+{
+	struct crg_gadget_dev *crg_udc = gadget_to_udc(g);
+
+	return FIELD_GET(CRG_U3DC_MFINDEX_FRAME_MASK,
+			 readl(&crg_udc->uccr->mfindex));
+}
+
+static int crg_gadget_pullup(struct usb_gadget *g, int is_on)
+{
+	struct crg_gadget_dev *crg_udc = gadget_to_udc(g);
+	unsigned long flags;
+
+	spin_lock_irqsave(&crg_udc->udc_lock, flags);
+	crg_udc->pullup_on = !!is_on;
+	crg_udc_set_pullup(crg_udc, crg_udc->pullup_on);
+	spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+	trace_crg_udc_pullup(crg_udc->pullup_on);
+	return 0;
+}
+
+static int crg_gadget_set_remote_wakeup(struct usb_gadget *g, int set)
+{
+	struct crg_gadget_dev *crg_udc = gadget_to_udc(g);
+	unsigned long flags;
+
+	spin_lock_irqsave(&crg_udc->udc_lock, flags);
+	crg_udc->u2_rwe = !!set;
+	g->wakeup_armed = !!set;
+	spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+
+	return 0;
+}
+
+static int crg_gadget_set_selfpowered(struct usb_gadget *g, int set)
+{
+	struct crg_gadget_dev *crg_udc = gadget_to_udc(g);
+	unsigned long flags;
+
+	spin_lock_irqsave(&crg_udc->udc_lock, flags);
+	g->is_selfpowered = !!set;
+	spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+
+	return 0;
+}
+
+static void crg_udc_set_speed(struct usb_gadget *gadget,
+			      enum usb_device_speed speed)
+{
+	struct crg_gadget_dev *crg_udc = gadget_to_udc(gadget);
+
+	crg_udc->max_speed = speed;
+}
+
+static int crg_gadget_start(struct usb_gadget *g,
+			    struct usb_gadget_driver *driver)
+{
+	struct crg_gadget_dev *crg_udc = gadget_to_udc(g);
+	int ret;
+
+	/* The UDC core calls udc_set_speed() before udc_start(). */
+	ret = crg_udc_program_speed(crg_udc);
+	if (ret)
+		return ret;
+
+	crg_udc->gadget_driver = driver;
+	crg_udc_set_vbus_detect(crg_udc, true);
+	return 0;
+}
+
+static int crg_gadget_stop(struct usb_gadget *g)
+{
+	struct crg_gadget_dev *crg_udc;
+	unsigned long flags = 0;
+
+	crg_udc = gadget_to_udc(g);
+
+	/*
+	 * Clear gadget_driver first so crg_udc_handle_port_status() can no
+	 * longer schedule udc_reset_work (it only does so while
+	 * gadget_driver is set), then cancel_work_sync() is guaranteed to
+	 * be the last word: no new reset notification can be queued after
+	 * it returns.
+	 */
+	spin_lock_irqsave(&crg_udc->udc_lock, flags);
+	crg_udc->gadget_driver = NULL;
+	spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+
+	cancel_work_sync(&crg_udc->udc_reset_work);
+
+	crg_udc->device_state = USB_STATE_ATTACHED;
+	crg_udc_set_vbus_detect(crg_udc, false);
+
+	spin_lock_irqsave(&crg_udc->udc_lock, flags);
+
+	/*
+	 * crg_udc_reset() completes any queued requests via
+	 * crg_udc_ep_nuke() -> crg_udc_req_done(). crg_udc_req_done() drops udc_lock
+	 * (IRQ-safe) around each ->complete() callback and re-acquires it
+	 * afterwards, so gadget completion callbacks are never invoked with
+	 * the spinlock held. flags is threaded through for that purpose.
+	 */
+	if (crg_udc_reset(crg_udc, &flags)) {
+		dev_err(crg_udc->dev, "controller reset failed during stop\n");
+		spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+		return -EIO;
+	}
+	if (crg_udc_reset_data_struct(crg_udc)) {
+		dev_err(crg_udc->dev,
+			"controller data reset failed during stop\n");
+		spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+		return -EIO;
+	}
+
+	crg_udc->connected = 0;
+	crg_udc->gadget.speed = USB_SPEED_UNKNOWN;
+
+	if (crg_udc_init_ep0(crg_udc)) {
+		spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+		return -EIO;
+	}
+
+	spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+
+	return 0;
+}
+
+static int crg_gadget_wakeup(struct usb_gadget *g)
+{
+	struct crg_gadget_dev *crg_udc = gadget_to_udc(g);
+	struct crg_uccr __iomem *uccr = crg_udc->uccr;
+	unsigned long flags;
+	u32 portsc;
+	int ret = 0;
+
+	spin_lock_irqsave(&crg_udc->udc_lock, flags);
+	if (!crg_udc->connected || g->speed > USB_SPEED_HIGH) {
+		ret = -EOPNOTSUPP;
+		goto out;
+	}
+	if (!crg_udc->u2_rwe) {
+		ret = -EACCES;
+		goto out;
+	}
+
+	portsc = readl(&uccr->portsc);
+	if (FIELD_GET(CRG_U3DC_PORTSC_PLS_MASK, portsc) !=
+	    CRG_U3DC_PORTSC_PLS_U3) {
+		ret = -EINVAL;
+		goto out;
+	}
+
+	portsc &= ~(CRG_U3DC_PORTSC_W1C_MASK | CRG_U3DC_PORTSC_PLS_MASK);
+	portsc |= CRG_U3DC_PORTSC_LWS;
+	writel(portsc, &uccr->portsc);
+out:
+	spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+	return ret;
+}
+
+static void crg_gadget_async_callbacks(struct usb_gadget *g, bool enable)
+{
+	struct crg_gadget_dev *crg_udc = gadget_to_udc(g);
+	unsigned long flags;
+
+	spin_lock_irqsave(&crg_udc->udc_lock, flags);
+	crg_udc->async_callbacks = enable;
+	spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+}
+
+struct usb_request *crg_udc_alloc_request(struct usb_ep *_ep, gfp_t gfp_flags)
+{
+	struct crg_udc_request *udc_req_ptr;
+
+	udc_req_ptr = kzalloc_obj(*udc_req_ptr, gfp_flags);
+
+	if (!udc_req_ptr)
+		return NULL;
+
+	udc_req_ptr->usb_req.dma = DMA_MAPPING_ERROR;
+	INIT_LIST_HEAD(&udc_req_ptr->queue);
+
+	trace_crg_udc_alloc_request(
+		container_of(_ep, struct crg_udc_ep, usb_ep), udc_req_ptr);
+
+	return &udc_req_ptr->usb_req;
+}
+
+int crg_udc_build_td(struct crg_udc_ep *udc_ep_ptr,
+		     struct crg_udc_request *udc_req_ptr)
+{
+	int status = 0;
+	struct crg_gadget_dev *crg_udc = udc_ep_ptr->crg_udc;
+	u32 num_trbs_needed;
+	u64 buffer_length;
+
+	if (udc_req_ptr->trbs_needed) {
+		/* If this is called to complete pending TRB transfers
+		 * of previous Request
+		 */
+		buffer_length = udc_req_ptr->buff_len_left;
+		num_trbs_needed = udc_req_ptr->trbs_needed;
+	} else {
+		buffer_length = udc_req_ptr->usb_req.length;
+		if (udc_req_ptr->usb_req.num_mapped_sgs)
+			num_trbs_needed = crg_udc_count_trbs_sg(udc_req_ptr);
+		else
+			num_trbs_needed = crg_udc_count_trbs_linear(
+				udc_req_ptr->usb_req.dma, buffer_length);
+		if (!num_trbs_needed && buffer_length)
+			return -EINVAL;
+		if (!num_trbs_needed)
+			num_trbs_needed = 1;
+	}
+
+	if (usb_endpoint_xfer_control(udc_ep_ptr->desc))
+		status = crg_udc_queue_ctrl(udc_ep_ptr, udc_req_ptr,
+					    num_trbs_needed);
+	else if (usb_endpoint_xfer_isoc(udc_ep_ptr->desc))
+		status = crg_udc_queue_trbs(udc_ep_ptr, udc_req_ptr, true,
+					    CRG_UDC_ISOC_EP_TD_RING_SIZE,
+					    num_trbs_needed, buffer_length);
+	else if (usb_endpoint_xfer_bulk(udc_ep_ptr->desc))
+		status = crg_udc_queue_trbs(udc_ep_ptr, udc_req_ptr, false,
+					    CRG_UDC_BULK_EP_TD_RING_SIZE,
+					    num_trbs_needed, buffer_length);
+	else
+		/* interrupt endpoint */
+		status = crg_udc_queue_trbs(udc_ep_ptr, udc_req_ptr, false,
+					    CRG_UDC_INT_EP_TD_RING_SIZE,
+					    num_trbs_needed, buffer_length);
+
+	if (!status && !usb_endpoint_xfer_control(udc_ep_ptr->desc))
+		crg_udc_knock_doorbell(crg_udc, udc_ep_ptr->dci);
+
+	return status;
+}
+
+void crg_udc_clear_portpm(struct crg_gadget_dev *crg_udc)
+{
+	struct crg_uccr __iomem *uccr = crg_udc->uccr;
+	u32 tmp;
+
+	tmp = readl(&uccr->u3portpmsc);
+
+	tmp &= (~CRG_U3DC_U3PORTPM_U1IEN);
+	FIELD_MODIFY(CRG_U3DC_U3PORTPM_U1TMOUT_MASK, &tmp, 0);
+	tmp &= (~CRG_U3DC_U3PORTPM_U2IEN);
+	FIELD_MODIFY(CRG_U3DC_U3PORTPM_U2TMOUT_MASK, &tmp, 0);
+
+	writel(tmp, &uccr->u3portpmsc);
+
+	crg_udc->feature_u1_enable = 0;
+	crg_udc->feature_u2_enable = 0;
+}
+
+static int crg_udc_ep_dequeue(struct usb_ep *_ep, struct usb_request *_req)
+{
+	struct crg_udc_ep *udc_ep_ptr;
+	struct crg_gadget_dev *crg_udc;
+	struct crg_uccr __iomem *uccr;
+	struct crg_udc_request *udc_req = NULL;
+	struct crg_udc_request *iter;
+	struct crg_ep_cx *p_ep_cx;
+	struct crg_transfer_trb *pause_pt;
+	enum crg_ep_state old_ep_state;
+	u32 tmp, deq_pt_lo, deq_pt_hi;
+	dma_addr_t dq_pt_addr;
+	u8 dci;
+	bool found = false;
+	unsigned long flags;
+
+	if (!_ep || !_req)
+		return -EINVAL;
+
+	udc_ep_ptr = container_of(_ep, struct crg_udc_ep, usb_ep);
+	crg_udc = udc_ep_ptr->crg_udc;
+	uccr = crg_udc->uccr;
+	dci = udc_ep_ptr->dci;
+
+	spin_lock_irqsave(&crg_udc->udc_lock, flags);
+	old_ep_state = udc_ep_ptr->ep_state;
+
+	list_for_each_entry(iter, &udc_ep_ptr->queue, queue) {
+		if (&iter->usb_req == _req) {
+			udc_req = iter;
+			found = true;
+			break;
+		}
+	}
+
+	if (!found) {
+		spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+		return -EINVAL;
+	}
+
+	if (!udc_req->first_trb) {
+		crg_udc_req_done(udc_ep_ptr, udc_req, -ECONNRESET);
+		spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+		return 0;
+	}
+
+	if (dci == 0) {
+		spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+		return -EINVAL;
+	}
+
+	if (old_ep_state == EP_STATE_RUNNING) {
+		int ret = crg_udc_issue_command(crg_udc, CRG_CMD_STOP_EP,
+						BIT(dci), 0);
+
+		if (ret) {
+			spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+			return ret;
+		}
+		if (readl_poll_timeout_atomic(&uccr->ep_running, tmp,
+					      !(tmp & BIT(dci)), 10, 10000)) {
+			spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+			return -ETIMEDOUT;
+		}
+		udc_ep_ptr->ep_state = EP_STATE_STOPPED;
+	}
+
+	p_ep_cx = &crg_udc->p_epcx[dci - 2];
+	deq_pt_lo = le32_to_cpu(p_ep_cx->dw2) & EP_CX_TR_DQPT_LO_MASK;
+	deq_pt_hi = le32_to_cpu(p_ep_cx->dw3);
+	dq_pt_addr = (dma_addr_t)deq_pt_lo | ((dma_addr_t)deq_pt_hi << 32);
+	pause_pt = crg_udc_tran_trb_dma_to_virt(udc_ep_ptr, dq_pt_addr);
+	if (!pause_pt) {
+		if (old_ep_state == EP_STATE_RUNNING) {
+			udc_ep_ptr->ep_state = EP_STATE_RUNNING;
+			crg_udc_knock_doorbell(crg_udc, dci);
+		}
+		spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+		return -EIO;
+	}
+
+	if (crg_udc_is_pointer_less_than(pause_pt, udc_req->first_trb,
+					 udc_ep_ptr)) {
+		crg_udc_squeeze_xfer_ring(udc_ep_ptr, udc_req);
+		crg_udc_req_done(udc_ep_ptr, udc_req, -ECONNRESET);
+	} else if (udc_req->last_trb &&
+		   crg_udc_is_pointer_less_than(udc_req->last_trb, pause_pt,
+						udc_ep_ptr)) {
+		crg_udc_req_done(udc_ep_ptr, udc_req, -ECONNRESET);
+	} else {
+		struct crg_udc_request *next_req = NULL;
+
+		udc_req->usb_req.actual =
+			crg_udc_actual_before(udc_ep_ptr, udc_req, pause_pt);
+
+		if (!list_is_last(&udc_req->queue, &udc_ep_ptr->queue))
+			next_req = list_entry(udc_req->queue.next,
+					      struct crg_udc_request, queue);
+
+		if (next_req && next_req->first_trb)
+			udc_ep_ptr->deq_pt = next_req->first_trb;
+		else
+			udc_ep_ptr->deq_pt = udc_ep_ptr->enq_pt;
+
+		if (crg_udc_epcx_update_dqptr(udc_ep_ptr)) {
+			spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+			return -EIO;
+		}
+		crg_udc_req_done(udc_ep_ptr, udc_req, -ECONNRESET);
+	}
+
+	if (old_ep_state == EP_STATE_RUNNING) {
+		udc_ep_ptr->ep_state = EP_STATE_RUNNING;
+		crg_udc_knock_doorbell(crg_udc, dci);
+	}
+
+	spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+	return 0;
+}
+
+/* endpoint operations */
+
+static int crg_udc_ep_disable(struct usb_ep *ep)
+{
+	struct crg_udc_ep *udc_ep;
+	struct crg_gadget_dev *crg_udc;
+	struct crg_ep_cx *p_ep_cx;
+	enum crg_ep_state ep_state;
+	struct crg_uccr __iomem *uccr;
+	unsigned long flags = 0;
+
+	if (!ep)
+		return -EINVAL;
+
+	udc_ep = container_of(ep, struct crg_udc_ep, usb_ep);
+	crg_udc = udc_ep->crg_udc;
+
+	if (udc_ep->dci == 0)
+		return 0;
+
+	spin_lock_irqsave(&crg_udc->udc_lock, flags);
+
+	uccr = crg_udc->uccr;
+	p_ep_cx = &crg_udc->p_epcx[udc_ep->dci - 2];
+
+	ep_state = crg_udc_get_ep_state(crg_udc, udc_ep->dci);
+	if (ep_state == EP_STATE_DISABLED) {
+		/* get here if ep is already disabled */
+		spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+		return -EINVAL;
+	}
+
+	/* Maybe we need to halt ep before ep disable */
+
+	/*
+	 * ep_enable is a write-1-to-disable register: endpoints are enabled
+	 * via the CONFIG_EP command, but writing a set bit here disables the
+	 * corresponding endpoint.
+	 */
+	writel(BIT(udc_ep->dci), &uccr->ep_enable);
+
+	udc_ep->desc = NULL;
+	udc_ep->comp_desc = NULL;
+
+	/* clean up the request queue */
+	crg_udc_ep_nuke(udc_ep, -ESHUTDOWN);
+
+	/* decrement ep counters */
+	crg_udc->num_enabled_eps--;
+	/* clean up the endpoint context */
+	memset(p_ep_cx, 0, sizeof(struct crg_ep_cx));
+
+	/* If device state was changed to default by port
+	 * reset, should not overwrite it again
+	 */
+	if (crg_udc->num_enabled_eps == 0 &&
+	    crg_udc->device_state == USB_STATE_CONFIGURED) {
+		crg_udc->device_state = USB_STATE_ADDRESS;
+	}
+
+	udc_ep->ep_state = EP_STATE_DISABLED;
+
+	spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+
+	return 0;
+}
+
+static int crg_udc_ep_enable(struct usb_ep *ep,
+			     const struct usb_endpoint_descriptor *desc)
+{
+	struct crg_udc_ep *udc_ep;
+	struct crg_gadget_dev *crg_udc;
+	u32 param0;
+	int status;
+	unsigned long flags = 0;
+
+	if (!ep || !desc || desc->bDescriptorType != USB_DT_ENDPOINT)
+		return -EINVAL;
+	udc_ep = container_of(ep, struct crg_udc_ep, usb_ep);
+	/* ep0 is always running */
+	if (udc_ep->dci == 0)
+		return 0;
+	crg_udc = udc_ep->crg_udc;
+	if (!crg_udc->gadget_driver)
+		return -ESHUTDOWN;
+	spin_lock_irqsave(&crg_udc->udc_lock, flags);
+	/* crg ep context start from ep1 */
+	if (crg_udc_get_ep_state(crg_udc, udc_ep->dci) != EP_STATE_DISABLED) {
+		int ret;
+
+		spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+		ret = crg_udc_ep_disable(ep);
+		if (ret)
+			return ret;
+		spin_lock_irqsave(&crg_udc->udc_lock, flags);
+	}
+	udc_ep->desc = desc;
+	udc_ep->comp_desc = ep->comp_desc;
+	status = crg_udc_validate_ep_desc(udc_ep, desc);
+	if (status) {
+		udc_ep->desc = NULL;
+		udc_ep->comp_desc = NULL;
+		spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+		return status;
+	}
+	/* setup endpoint context for regular endpoint
+	 * the endpoint context for control endpoint has been
+	 * set up in probe function
+	 */
+	if (udc_ep->dci) {
+		u32 ring_size = 0;
+		size_t len;
+
+		if (usb_endpoint_xfer_bulk(desc))
+			ring_size = CRG_UDC_BULK_EP_TD_RING_SIZE;
+		else if (usb_endpoint_xfer_isoc(desc))
+			ring_size = CRG_UDC_ISOC_EP_TD_RING_SIZE;
+		else if (usb_endpoint_xfer_int(desc))
+			ring_size = CRG_UDC_INT_EP_TD_RING_SIZE;
+
+		if (!ring_size) {
+			spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+			return -EINVAL;
+		}
+		len = ring_size * sizeof(struct crg_transfer_trb);
+
+		if (udc_ep->tran_ring_info.cpu_addr &&
+		    udc_ep->tran_ring_info.len != len) {
+			void *old_vaddr = udc_ep->tran_ring_info.cpu_addr;
+			dma_addr_t old_dma = udc_ep->tran_ring_info.dma_addr;
+			size_t old_len = udc_ep->tran_ring_info.len;
+
+			udc_ep->tran_ring_info.cpu_addr = NULL;
+			udc_ep->enq_pt = NULL;
+			udc_ep->deq_pt = NULL;
+
+			spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+			dma_free_coherent(crg_udc->dev, old_len, old_vaddr,
+					  old_dma);
+			spin_lock_irqsave(&crg_udc->udc_lock, flags);
+
+			if (udc_ep->desc != desc ||
+			    udc_ep->ep_state != EP_STATE_DISABLED) {
+				spin_unlock_irqrestore(&crg_udc->udc_lock,
+						       flags);
+				return -EAGAIN;
+			}
+		}
+
+		/* setup transfer ring */
+		if (!udc_ep->tran_ring_info.cpu_addr) {
+			dma_addr_t dma;
+			void *vaddr;
+
+			spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+			vaddr = dma_alloc_coherent(crg_udc->dev, len, &dma,
+						   GFP_KERNEL);
+			if (!vaddr)
+				return -ENOMEM;
+			spin_lock_irqsave(&crg_udc->udc_lock, flags);
+			if (udc_ep->desc != desc ||
+			    udc_ep->ep_state != EP_STATE_DISABLED) {
+				spin_unlock_irqrestore(&crg_udc->udc_lock,
+						       flags);
+				dma_free_coherent(crg_udc->dev, len, vaddr,
+						  dma);
+				return -EAGAIN;
+			}
+			udc_ep->tran_ring_info.cpu_addr = vaddr;
+			udc_ep->tran_ring_info.dma_addr = dma;
+			udc_ep->tran_ring_info.len = len;
+			udc_ep->first_trb = vaddr;
+			udc_ep->last_trb = udc_ep->first_trb + ring_size - 1;
+		}
+		memset(udc_ep->first_trb, 0, udc_ep->tran_ring_info.len);
+		/* Ensure that transfer ring is updated */
+		dma_wmb();
+		crg_udc_setup_link_trb(udc_ep->last_trb, true,
+				       udc_ep->tran_ring_info.dma_addr);
+		udc_ep->enq_pt = udc_ep->first_trb;
+		udc_ep->deq_pt = udc_ep->first_trb;
+		udc_ep->pcs = 1;
+		udc_ep->tran_ring_full = false;
+		crg_udc->num_enabled_eps++;
+		status = crg_udc_epcx_setup(udc_ep);
+		if (status)
+			goto err_ep_enable;
+	}
+	param0 = BIT(udc_ep->dci);
+	status = crg_udc_issue_command(crg_udc, CRG_CMD_CONFIG_EP, param0, 0);
+	if (status)
+		goto err_ep_enable;
+	if (crg_udc->device_state == USB_STATE_ADDRESS)
+		crg_udc->device_state = USB_STATE_CONFIGURED;
+	udc_ep->wedge = false;
+	udc_ep->usb_ep.enabled = 1;
+	udc_ep->ep_state = EP_STATE_RUNNING;
+	spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+	return 0;
+
+err_ep_enable:
+	if (udc_ep->dci && crg_udc->num_enabled_eps)
+		crg_udc->num_enabled_eps--;
+	udc_ep->desc = NULL;
+	udc_ep->comp_desc = NULL;
+	udc_ep->ep_state = EP_STATE_DISABLED;
+	spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+
+	if (udc_ep->tran_ring_info.cpu_addr) {
+		dma_free_coherent(crg_udc->dev, udc_ep->tran_ring_info.len,
+				  udc_ep->tran_ring_info.cpu_addr,
+				  udc_ep->tran_ring_info.dma_addr);
+		udc_ep->tran_ring_info.cpu_addr = NULL;
+		udc_ep->tran_ring_info.dma_addr = DMA_MAPPING_ERROR;
+		udc_ep->tran_ring_info.len = 0;
+		udc_ep->first_trb = NULL;
+		udc_ep->last_trb = NULL;
+	}
+	return status;
+}
+
+void crg_udc_ep_nuke(struct crg_udc_ep *udc_ep, int status)
+{
+	struct crg_udc_request *req;
+	struct list_head nuke_list;
+
+	list_replace_init(&udc_ep->queue, &nuke_list);
+
+	while (!list_empty(&nuke_list)) {
+		req = list_first_entry(&nuke_list, struct crg_udc_request,
+				       queue);
+
+		crg_udc_req_done(udc_ep, req, status);
+	}
+}
+
+static int crg_udc_ep_queue(struct usb_ep *_ep, struct usb_request *_req,
+			    gfp_t gfp_flags)
+{
+	struct crg_udc_request *udc_req_ptr;
+	struct crg_udc_ep *udc_ep_ptr;
+	struct crg_gadget_dev *crg_udc;
+	enum dma_data_direction dma_data_dir;
+	const struct usb_endpoint_descriptor *desc;
+	unsigned long flags;
+	int status;
+	bool is_sg;
+
+	if (!_ep || !_req)
+		return -EINVAL;
+
+	udc_req_ptr = container_of(_req, struct crg_udc_request, usb_req);
+	udc_ep_ptr = container_of(_ep, struct crg_udc_ep, usb_ep);
+	crg_udc = udc_ep_ptr->crg_udc;
+	is_sg = _req->num_sgs != 0 && _req->length != 0;
+
+	if (is_sg && !_req->sg)
+		return -EINVAL;
+	if (_req->stream_id)
+		return -EOPNOTSUPP;
+
+	spin_lock_irqsave(&crg_udc->udc_lock, flags);
+	/*
+	 * A completion callback for an aborted EP0 transfer may requeue the
+	 * same request. Reject EP0 requests while the previous control
+	 * transfer is being aborted or EP0 is in an error state. EP0 becomes
+	 * available again only after the new SETUP has been accepted and
+	 * the EP0 transfer ring has been successfully reinitialized.
+	 */
+	if (udc_ep_ptr->dci == 0 && crg_udc->ep0_state != EP0_STATE_IDLE) {
+		spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+		return -ECONNRESET;
+	}
+	if (!udc_ep_ptr->first_trb || !_req->complete ||
+	    (!is_sg && !_req->buf) || !list_empty(&udc_req_ptr->queue)) {
+		spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+		return -EINVAL;
+	}
+	if (crg_udc->resetting) {
+		spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+		return -ESHUTDOWN;
+	}
+	if (!udc_ep_ptr->desc) {
+		spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+		return -EINVAL;
+	}
+	desc = udc_ep_ptr->desc;
+
+	if (usb_endpoint_xfer_control(udc_ep_ptr->desc) && is_sg) {
+		spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+		return -EOPNOTSUPP;
+	}
+
+	if (usb_endpoint_xfer_control(udc_ep_ptr->desc) && !_req->length) {
+		if (crg_udc->setup_status == DATA_STAGE_XFER)
+			crg_udc->setup_status = STATUS_STAGE_RECV;
+		else
+			crg_udc->setup_status = STATUS_STAGE_XFER;
+		crg_udc_build_ep0_status(&crg_udc->udc_ep[0], false,
+					 -EINPROGRESS, udc_req_ptr, 0, 0);
+		spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+		return 0;
+	}
+
+	if (usb_endpoint_xfer_control(udc_ep_ptr->desc)) {
+		if (crg_udc->setup_status == DATA_STAGE_XFER ||
+		    crg_udc->setup_status == STATUS_STAGE_XFER) {
+			dma_data_dir = DMA_TO_DEVICE;
+		} else if (crg_udc->setup_status == DATA_STAGE_RECV ||
+			   crg_udc->setup_status == STATUS_STAGE_RECV) {
+			dma_data_dir = DMA_FROM_DEVICE;
+		} else {
+			spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+			return -EINVAL;
+		}
+	} else {
+		dma_data_dir = usb_endpoint_dir_in(udc_ep_ptr->desc) ?
+				       DMA_TO_DEVICE :
+				       DMA_FROM_DEVICE;
+	}
+	spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+
+	crg_udc_clear_req_container(udc_req_ptr);
+	udc_req_ptr->mapped = false;
+
+	if (_req->length) {
+		udc_req_ptr->map_dir = dma_data_dir;
+		status = usb_gadget_map_request_by_dev(
+			crg_udc->dev, _req, dma_data_dir == DMA_TO_DEVICE);
+		if (status)
+			goto err_unmap;
+
+		if (is_sg) {
+			if (!_req->num_mapped_sgs ||
+			    crg_udc_sg_init(udc_req_ptr)) {
+				status = -EIO;
+				goto err_unmap;
+			}
+		} else if (_req->dma_mapped) {
+			udc_req_ptr->mapped = true;
+		}
+	}
+
+	spin_lock_irqsave(&crg_udc->udc_lock, flags);
+	if (udc_ep_ptr->desc != desc || !udc_req_ptr->usb_req.complete ||
+	    !list_empty(&udc_req_ptr->queue) || crg_udc->resetting) {
+		status = -ESHUTDOWN;
+		spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+		crg_udc_unmap_request(udc_ep_ptr, udc_req_ptr);
+		return status;
+	}
+
+	_req->status = -EINPROGRESS;
+	_req->actual = 0;
+
+	if (udc_ep_ptr->tran_ring_full)
+		status = 0;
+	else
+		status = crg_udc_build_td(udc_ep_ptr, udc_req_ptr);
+
+	if (!status)
+		list_add_tail(&udc_req_ptr->queue, &udc_ep_ptr->queue);
+	spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+
+	if (status)
+		crg_udc_unmap_request(udc_ep_ptr, udc_req_ptr);
+	return status;
+
+err_unmap:
+	crg_udc_unmap_request(udc_ep_ptr, udc_req_ptr);
+	return status;
+}
+
+static int crg_udc_ep_set_halt(struct usb_ep *_ep, int value)
+{
+	struct crg_udc_ep *udc_ep_ptr;
+	int status;
+	unsigned long flags = 0;
+	struct crg_gadget_dev *crg_udc;
+
+	if (!_ep)
+		return -EINVAL;
+
+	udc_ep_ptr = container_of(_ep, struct crg_udc_ep, usb_ep);
+	crg_udc = udc_ep_ptr->crg_udc;
+
+	spin_lock_irqsave(&crg_udc->udc_lock, flags);
+	if (!udc_ep_ptr->desc) {
+		spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+		return -EINVAL;
+	}
+	if (value && usb_endpoint_dir_in(udc_ep_ptr->desc) &&
+	    !list_empty(&udc_ep_ptr->queue)) {
+		dev_err(crg_udc->dev, "set_halt: list not empty\n");
+		spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+		return -EAGAIN;
+	}
+
+	status = crg_udc_ep_halt(udc_ep_ptr, value, 1, &flags);
+
+	spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+
+	return status;
+}
+
+static int crg_udc_ep_set_wedge(struct usb_ep *_ep)
+{
+	struct crg_udc_ep *udc_ep_ptr;
+	int status;
+	unsigned long flags = 0;
+	struct crg_gadget_dev *crg_udc;
+
+	if (!_ep)
+		return -EINVAL;
+
+	udc_ep_ptr = container_of(_ep, struct crg_udc_ep, usb_ep);
+	crg_udc = udc_ep_ptr->crg_udc;
+
+	spin_lock_irqsave(&crg_udc->udc_lock, flags);
+
+	if (!udc_ep_ptr->desc) {
+		spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+		return -EINVAL;
+	}
+
+	udc_ep_ptr->wedge = 1;
+
+	status = crg_udc_ep_halt(udc_ep_ptr, 1, 1, &flags);
+
+	spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+
+	return status;
+}
+
+static int crg_udc_epcx_setup(struct crg_udc_ep *udc_ep)
+{
+	struct crg_gadget_dev *crg_udc = udc_ep->crg_udc;
+	const struct usb_endpoint_descriptor *desc = udc_ep->desc;
+	const struct usb_ss_ep_comp_descriptor *comp_desc = udc_ep->comp_desc;
+	u8 dci = udc_ep->dci;
+	struct crg_ep_cx *epcx = &crg_udc->p_epcx[dci - 2];
+	enum crg_ep_type ep_type;
+	u16 maxburst = 0;
+	u8 maxstreams = 0;
+	u16 maxsize;
+	u32 dw;
+
+	/* corigine gadget dir should be opposite to host dir */
+	if (usb_endpoint_dir_out(desc))
+		ep_type = usb_endpoint_type(desc) + EP_TYPE_INVALID2;
+	else
+		ep_type = usb_endpoint_type(desc);
+
+	maxsize = usb_endpoint_maxp(desc) & 0x07ff;
+
+	if (crg_udc->gadget.speed >= USB_SPEED_SUPER) {
+		if (!comp_desc)
+			return -EINVAL;
+		maxburst = comp_desc->bMaxBurst;
+
+		if (usb_endpoint_xfer_bulk(udc_ep->desc))
+			maxstreams = comp_desc->bmAttributes & 0x1f;
+
+	} else if ((crg_udc->gadget.speed == USB_SPEED_HIGH ||
+		    crg_udc->gadget.speed == USB_SPEED_FULL) &&
+		   (usb_endpoint_xfer_int(udc_ep->desc) ||
+		    usb_endpoint_xfer_isoc(udc_ep->desc))) {
+		if (crg_udc->gadget.speed == USB_SPEED_HIGH)
+			maxburst = (usb_endpoint_maxp(desc) >> 11) & 0x3;
+		if (maxburst == 0x3) {
+			dev_err(crg_udc->dev, "invalid maxburst\n");
+			maxburst = 0x2;
+		}
+	}
+
+	/* fill ep_dw0 */
+	dw = 0;
+	FIELD_MODIFY(EP_CX_LOGICAL_EP_NUM_MASK, &dw, udc_ep->dci / 2);
+	FIELD_MODIFY(EP_CX_INTERVAL_MASK, &dw, desc->bInterval);
+	if (maxstreams) {
+		dev_err(crg_udc->dev, "maxstream=%d is not expected\n",
+			maxstreams);
+	}
+	epcx->dw0 = cpu_to_le32(dw);
+
+	/* fill ep_dw1 */
+	dw = 0;
+	FIELD_MODIFY(EP_CX_EP_TYPE_MASK, &dw, ep_type);
+	FIELD_MODIFY(EP_CX_MAX_PACKET_SIZE_MASK, &dw, maxsize);
+	FIELD_MODIFY(EP_CX_MAX_BURST_SIZE_MASK, &dw, maxburst);
+	epcx->dw1 = cpu_to_le32(dw);
+
+	/* fill ep_dw2 */
+	dw = lower_32_bits(udc_ep->tran_ring_info.dma_addr) &
+	     EP_CX_TR_DQPT_LO_MASK;
+	FIELD_MODIFY(EP_CX_DEQ_CYC_STATE_MASK, &dw, udc_ep->pcs);
+	epcx->dw2 = cpu_to_le32(dw);
+
+	/* fill ep_dw3 */
+	dw = upper_32_bits(udc_ep->tran_ring_info.dma_addr);
+	epcx->dw3 = cpu_to_le32(dw);
+	/* Ensure that epcx is updated */
+	dma_wmb();
+	return 0;
+}
+
+static int crg_udc_epcx_update_dqptr(struct crg_udc_ep *udc_ep)
+{
+	struct crg_gadget_dev *crg_udc = udc_ep->crg_udc;
+	u8 dci = udc_ep->dci;
+	struct crg_ep_cx *epcx;
+	u32 dw;
+	dma_addr_t dqptaddr;
+	u32 cmd_param0;
+
+	if (dci < 2 || dci >= CRG_NUM_EP_CX) {
+		dev_err(crg_udc->dev, "invalid DCI %u for dequeue pointer\n", dci);
+		return -EINVAL;
+	}
+
+	epcx = &crg_udc->p_epcx[dci - 2];
+	dqptaddr = crg_udc_tran_trb_virt_to_dma(udc_ep, udc_ep->deq_pt);
+
+	if (dqptaddr == DMA_MAPPING_ERROR)
+		return -EINVAL;
+
+	/* fill ep_dw2 */
+	dw = lower_32_bits(dqptaddr) & EP_CX_TR_DQPT_LO_MASK;
+	FIELD_MODIFY(EP_CX_DEQ_CYC_STATE_MASK, &dw, udc_ep->pcs);
+	epcx->dw2 = cpu_to_le32(dw);
+
+	/* fill ep_dw3 */
+	dw = upper_32_bits(dqptaddr);
+	epcx->dw3 = cpu_to_le32(dw);
+
+	cmd_param0 = BIT(udc_ep->dci);
+	/* Ensure that dqptr is updated */
+	dma_wmb();
+
+	return crg_udc_issue_command(crg_udc, CRG_CMD_SET_TR_DQPTR, cmd_param0, 0);
+}
+
+void crg_udc_free_request(struct usb_ep *_ep, struct usb_request *_req)
+{
+	struct crg_udc_request *udc_req_ptr;
+
+	if (!_ep || !_req)
+		return;
+
+	udc_req_ptr = container_of(_req, struct crg_udc_request, usb_req);
+
+	trace_crg_udc_free_request(container_of(_ep, struct crg_udc_ep, usb_ep),
+				   udc_req_ptr);
+
+	if (udc_req_ptr->mapped || udc_req_ptr->usb_req.num_mapped_sgs)
+		crg_udc_unmap_request(container_of(_ep, struct crg_udc_ep,
+						   usb_ep),
+				      udc_req_ptr);
+	kfree(udc_req_ptr);
+}
+
+static int crg_udc_queue_trbs(struct crg_udc_ep *udc_ep_ptr,
+			      struct crg_udc_request *udc_req_ptr, bool b_isoc,
+			      u32 xfer_ring_size, u32 num_trbs_needed,
+			      u64 buffer_length)
+{
+	struct crg_gadget_dev *crg_udc = udc_ep_ptr->crg_udc;
+	struct crg_transfer_trb *p_xfer_ring = udc_ep_ptr->first_trb;
+	struct crg_transfer_trb *enq_pt = udc_ep_ptr->enq_pt;
+	struct usb_request *usb_req = &udc_req_ptr->usb_req;
+	u32 num_trbs_ava, count, i;
+	u32 td_trbs;
+	u32 td_size;
+	u32 j = 1;
+	u32 intr_rate = b_isoc ? CRG_UDC_ISOC_EP_INTERRUPT_RATE :
+				 CRG_UDC_BULK_EP_INTERRUPT_RATE;
+	bool full_td = true;
+	bool need_zlp = false;
+	bool is_sg = usb_req->num_mapped_sgs;
+
+	if (!b_isoc && usb_req->zero && usb_req->length &&
+	    !(usb_req->length % udc_ep_ptr->usb_ep.maxpacket))
+		need_zlp = true;
+
+	if (!num_trbs_needed)
+		return -EINVAL;
+
+	/* TD_SIZE is five bits and encodes the number of TRBs remaining
+	 * in this TD minus one. A request may contain multiple TDs.
+	 */
+	td_trbs = min_t(u32, num_trbs_needed, 32);
+
+	num_trbs_ava = crg_udc_room_on_ring(crg_udc, xfer_ring_size,
+					    p_xfer_ring, udc_ep_ptr->enq_pt,
+					    udc_ep_ptr->deq_pt);
+
+	if (num_trbs_ava < td_trbs) {
+		if (b_isoc)
+			return -EBUSY;
+		trace_crg_udc_no_room_on_ring(
+			"deferring TRBs to next doorbell");
+		count = num_trbs_ava;
+		full_td = false;
+		udc_ep_ptr->tran_ring_full = true;
+		need_zlp = false;
+	} else {
+		count = td_trbs;
+	}
+
+	if (!count)
+		return -ENOSPC;
+
+	td_size = count - 1;
+
+	if (is_sg && !udc_req_ptr->sg && crg_udc_sg_init(udc_req_ptr))
+		return -EINVAL;
+
+	u8 original_pcs = udc_ep_ptr->pcs;
+
+	for (i = 0; i < count; i++) {
+		dma_addr_t trb_buf_addr;
+		u32 trb_len, sg_len = 0;
+		bool chain_bit = true;
+		bool ioc = false;
+		bool short_pkt = usb_endpoint_dir_out(udc_ep_ptr->desc);
+
+		if (is_sg) {
+			if (!crg_udc_sg_current(udc_req_ptr, &trb_buf_addr,
+						&sg_len))
+				return -EINVAL;
+			if (!sg_len ||
+			    dma_mapping_error(crg_udc->dev, trb_buf_addr))
+				return -EINVAL;
+			trb_len = min_t(
+				u64, buffer_length,
+				min_t(u32, sg_len,
+				      crg_udc_trb_chunk_len(trb_buf_addr,
+							    buffer_length)));
+		} else {
+			if (buffer_length)
+				trb_buf_addr = usb_req->dma + (usb_req->length -
+							       buffer_length);
+			else
+				trb_buf_addr = 0;
+			trb_len = crg_udc_trb_chunk_len(trb_buf_addr,
+							buffer_length);
+		}
+
+		if (!trb_len && buffer_length)
+			return -EINVAL;
+
+		buffer_length -= trb_len;
+
+		if (i == count - 1) {
+			chain_bit = false;
+			ioc = true;
+			if (!buffer_length)
+				udc_req_ptr->all_trbs_queued = true;
+		}
+
+		if (!full_td && j == intr_rate) {
+			ioc = true;
+			j = 0;
+		}
+
+		if (b_isoc) {
+			crg_udc_setup_trb(
+				enq_pt, trb_len, trb_buf_addr, td_size,
+				i == 0 ? (original_pcs ^ 1) : udc_ep_ptr->pcs,
+				TRB_TYPE_XFER_DATA_ISOCH, short_pkt, chain_bit,
+				ioc, false, 0, true, 0, 1, 0);
+		} else {
+			bool azp = ioc && !chain_bit && need_zlp;
+
+			crg_udc_setup_trb(
+				enq_pt, trb_len, trb_buf_addr, td_size,
+				i == 0 ? (original_pcs ^ 1) : udc_ep_ptr->pcs,
+				TRB_TYPE_XFER_NORMAL, short_pkt, chain_bit, ioc,
+				false, 0, false, 0, 0, azp);
+		}
+
+		trace_crg_udc_queue_trb(udc_ep_ptr, enq_pt);
+
+		if (is_sg && !crg_udc_sg_advance(udc_req_ptr, trb_len))
+			return -EINVAL;
+
+		td_size--;
+		enq_pt++;
+		j++;
+
+		if (enq_pt == udc_ep_ptr->last_trb) {
+			dma_wmb();
+			le32p_replace_bits(&enq_pt->dw3, udc_ep_ptr->pcs,
+					   TRB_CYCLE_BIT_MASK);
+			udc_ep_ptr->pcs ^= 1;
+			enq_pt = udc_ep_ptr->first_trb;
+		}
+	}
+
+	dma_wmb();
+	le32p_replace_bits(&udc_ep_ptr->enq_pt->dw3, original_pcs,
+			   TRB_CYCLE_BIT_MASK);
+
+	if (!udc_req_ptr->trbs_needed)
+		udc_req_ptr->first_trb = udc_ep_ptr->enq_pt;
+
+	udc_ep_ptr->enq_pt = enq_pt;
+	if (count == num_trbs_ava)
+		udc_ep_ptr->tran_ring_full = true;
+	udc_req_ptr->buff_len_left = buffer_length;
+	if (buffer_length)
+		udc_req_ptr->trbs_needed = num_trbs_needed - count;
+	else
+		udc_req_ptr->trbs_needed = 0;
+
+	if (!udc_req_ptr->buff_len_left) {
+		if (udc_ep_ptr->enq_pt == udc_req_ptr->first_trb)
+			udc_req_ptr->last_trb = udc_ep_ptr->last_trb - 1;
+		else
+			udc_req_ptr->last_trb = udc_ep_ptr->enq_pt - 1;
+	}
+
+	return 0;
+}
+
+static int crg_udc_reinit(struct crg_gadget_dev *crg_udc)
+{
+	struct crg_uccr __iomem *uccr = crg_udc->uccr;
+	u32 i, tmp;
+	struct crg_udc_ep *udc_ep_ptr;
+
+	crg_udc->setup_status = WAIT_FOR_SETUP;
+	/* Base on Figure 9-1, default USB_STATE is attached */
+	crg_udc->device_state = USB_STATE_RECONNECTING;
+
+	/* halt all the endpoints */
+
+	/* disable all the endpoints */
+	tmp = readl(&uccr->ep_enable);
+	writel(tmp, &uccr->ep_enable);
+	if (readl_poll_timeout_atomic(&uccr->ep_enable, tmp, !tmp, 1, 50)) {
+		dev_dbg(crg_udc->dev,
+			"timeout waiting for all endpoints to disable\n");
+		return -ETIMEDOUT;
+	}
+
+	for (i = 2; i < CRG_NUM_EP_CX; i++) {
+		udc_ep_ptr = &crg_udc->udc_ep[i];
+		udc_ep_ptr->usb_ep.enabled = 0;
+		if (udc_ep_ptr->desc) {
+			udc_ep_ptr->desc = NULL;
+			udc_ep_ptr->comp_desc = NULL;
+			crg_udc_ep_nuke(udc_ep_ptr, -ESHUTDOWN);
+		}
+		udc_ep_ptr->tran_ring_full = false;
+		udc_ep_ptr->ep_state = EP_STATE_DISABLED;
+	}
+	crg_udc->num_enabled_eps = 0;
+
+	/* we don't handle ep0 here, we crg_udc_init_ep0 when event ring is empty */
+
+	if (crg_udc->dev_addr != 0) {
+		u32 param0;
+
+		param0 = FIELD_PREP(CRG_CMD2_0_DEV_ADDR_MASK, 0);
+		if (crg_udc_issue_command(crg_udc, CRG_CMD_SET_ADDR, param0, 0))
+			return -EIO;
+		crg_udc->dev_addr = 0;
+	}
+
+	crg_udc_clear_portpm(crg_udc);
+
+	return 0;
+}
+
+void crg_udc_set_pullup(struct crg_gadget_dev *crg_udc, bool is_on)
+{
+	struct crg_uccr __iomem *uccr = crg_udc->uccr;
+	u32 val;
+
+	if (is_on) {
+		/* Enable event reporting and interrupts, then assert RUN */
+		val = readl(&uccr->config1);
+		val |= (CRG_U3DC_CFG1_CSC_EVENT_EN |
+			CRG_U3DC_CFG1_PEC_EVENT_EN |
+			CRG_U3DC_CFG1_PPC_EVENT_EN |
+			CRG_U3DC_CFG1_PRC_EVENT_EN |
+			CRG_U3DC_CFG1_PLC_EVENT_EN |
+			CRG_U3DC_CFG1_CEC_EVENT_EN);
+		writel(val, &uccr->config1);
+
+		val = readl(&uccr->control);
+		val |= (CRG_U3DC_CTRL_SYSERR_EN | CRG_U3DC_CTRL_INT_EN |
+			CRG_U3DC_CTRL_RUN);
+		writel(val, &uccr->control);
+	} else {
+		val = readl(&uccr->control);
+		val &= ~CRG_U3DC_CTRL_RUN;
+		writel(val, &uccr->control);
+	}
+}
+
+void crg_udc_set_vbus_detect(struct crg_gadget_dev *crg_udc, bool enable)
+{
+	u32 val = readl(crg_udc->mmio_virt_base + CRG_UDC_VENDOR_REG);
+
+	if (enable)
+		val |= CRG_UDC_VBUS_DETECT;
+	else
+		val &= ~CRG_UDC_VBUS_DETECT;
+
+	writel(val, crg_udc->mmio_virt_base + CRG_UDC_VENDOR_REG);
+}
+
+static bool crg_udc_sg_advance(struct crg_udc_request *udc_req, u32 bytes)
+{
+	struct scatterlist *sg = udc_req->sg;
+	unsigned int i = udc_req->sg_index;
+
+	while (bytes) {
+		u32 len;
+
+		if (!sg || i >= udc_req->usb_req.num_sgs)
+			return false;
+
+		len = sg_dma_len(sg);
+		if (!len || udc_req->sg_offset == len) {
+			sg = sg_next(sg);
+			i++;
+			udc_req->sg_offset = 0;
+			continue;
+		}
+
+		len -= udc_req->sg_offset;
+		if (bytes < len) {
+			udc_req->sg_offset += bytes;
+			bytes = 0;
+			break;
+		}
+
+		bytes -= len;
+		sg = sg_next(sg);
+		i++;
+		udc_req->sg_offset = 0;
+	}
+
+	udc_req->sg = sg;
+	udc_req->sg_index = i;
+	return true;
+}
+
+static bool crg_udc_sg_current(struct crg_udc_request *udc_req, dma_addr_t *dma,
+			       u32 *len)
+{
+	struct scatterlist *sg = udc_req->sg;
+
+	while (sg && udc_req->sg_index < udc_req->usb_req.num_sgs &&
+	       !sg_dma_len(sg)) {
+		sg = sg_next(sg);
+		udc_req->sg_index++;
+		udc_req->sg_offset = 0;
+	}
+	udc_req->sg = sg;
+
+	if (!sg || udc_req->sg_index >= udc_req->usb_req.num_sgs)
+		return false;
+
+	if (udc_req->sg_offset >= sg_dma_len(sg))
+		return false;
+
+	*dma = sg_dma_address(sg) + udc_req->sg_offset;
+	*len = sg_dma_len(sg) - udc_req->sg_offset;
+	return true;
+}
+
+static int crg_udc_sg_init(struct crg_udc_request *udc_req)
+{
+	struct scatterlist *sg;
+	unsigned int i;
+
+	for_each_sg(udc_req->usb_req.sg, sg, udc_req->usb_req.num_sgs, i) {
+		if (sg_dma_len(sg)) {
+			udc_req->sg = sg;
+			udc_req->sg_index = i;
+			udc_req->sg_offset = 0;
+			return 0;
+		}
+	}
+
+	return -EINVAL;
+}
+
+/*
+ * Undo whatever crg_udc_ep_queue() mapped for this request: the merged
+ * SG list, the single buffer, or neither if the request carried a
+ * pre-mapped DMA address.
+ */
+void crg_udc_unmap_request(struct crg_udc_ep *udc_ep,
+			   struct crg_udc_request *udc_req)
+{
+	struct crg_gadget_dev *crg_udc = udc_ep->crg_udc;
+	struct usb_request *req = &udc_req->usb_req;
+	bool is_in = udc_req->map_dir == DMA_TO_DEVICE;
+
+	if (req->num_mapped_sgs || req->dma_mapped)
+		usb_gadget_unmap_request_by_dev(crg_udc->dev, req, is_in);
+
+	if (udc_req->mapped) {
+		req->dma = DMA_MAPPING_ERROR;
+		udc_req->mapped = false;
+	}
+
+	udc_req->sg = NULL;
+	udc_req->sg_index = 0;
+	udc_req->sg_offset = 0;
+}
+
+static int crg_udc_validate_ep_desc(struct crg_udc_ep *udc_ep,
+				    const struct usb_endpoint_descriptor *desc)
+{
+	const struct usb_ss_ep_comp_descriptor *comp = udc_ep->comp_desc;
+	u16 maxpacket = usb_endpoint_maxp(desc);
+
+	if (usb_endpoint_xfer_control(desc) || !maxpacket ||
+	    maxpacket > CRG_MAX_PACKET_SIZE)
+		return -EINVAL;
+
+	if (crg_udc_index2dci(desc->bEndpointAddress) != udc_ep->dci)
+		return -EINVAL;
+
+	if (usb_endpoint_xfer_bulk(desc) && comp && usb_ss_max_streams(comp))
+		return -EOPNOTSUPP;
+
+	if (usb_endpoint_xfer_isoc(desc) && desc->bInterval == 0)
+		return -EINVAL;
+
+	if (udc_ep->crg_udc->gadget.speed >= USB_SPEED_SUPER) {
+		if (!comp || comp->bMaxBurst > 15)
+			return -EINVAL;
+		if (usb_endpoint_xfer_isoc(desc) &&
+		    ((comp->bmAttributes & 0x3) > 2))
+			return -EINVAL;
+	} else if (udc_ep->crg_udc->gadget.speed == USB_SPEED_HIGH) {
+		if (desc->bInterval > 16 && (usb_endpoint_xfer_int(desc) ||
+					     usb_endpoint_xfer_isoc(desc)))
+			return -EINVAL;
+		if (usb_endpoint_xfer_isoc(desc) &&
+		    usb_endpoint_maxp_mult(desc) > 3)
+			return -EINVAL;
+	} else if (udc_ep->crg_udc->gadget.speed == USB_SPEED_FULL) {
+		if (maxpacket > 64 && !usb_endpoint_xfer_isoc(desc))
+			return -EINVAL;
+		if (usb_endpoint_xfer_isoc(desc) && maxpacket > 1023)
+			return -EINVAL;
+	}
+
+	return 0;
+}
+
+static bool crg_udc_vbus_connected(struct crg_gadget_dev *crg_udc)
+{
+	return !!(readl(&crg_udc->uccr->portsc) & CRG_U3DC_PORTSC_PP);
+}
+
+static void crg_ep_struct_setup(struct crg_gadget_dev *crg_udc, u32 dci,
+				const char *name)
+{
+	struct crg_udc_ep *ep = &crg_udc->udc_ep[dci];
+
+	ep->dci = dci;
+
+	if (ep->dci > 1) {
+		strscpy_pad(ep->name, name, sizeof(ep->name));
+		ep->usb_ep.name = ep->name;
+
+		ep->usb_ep.caps.type_iso = 1;
+		ep->usb_ep.caps.type_bulk = 1;
+		ep->usb_ep.caps.type_int = 1;
+		if (ep->dci % 2)
+			ep->usb_ep.caps.dir_out = 1;
+		else
+			ep->usb_ep.caps.dir_in = 1;
+		usb_ep_set_maxpacket_limit(&ep->usb_ep, CRG_MAX_PACKET_SIZE);
+	} else {
+		strscpy(ep->name, "ep0", sizeof(ep->name));
+		ep->usb_ep.name = ep->name;
+		ep->usb_ep.maxpacket = 512;
+		ep->usb_ep.caps.type_control = 1;
+		ep->usb_ep.caps.dir_in = 1;
+		ep->usb_ep.caps.dir_out = 1;
+	}
+
+	ep->usb_ep.ops = &crg_udc_ep_ops;
+	ep->crg_udc = crg_udc;
+
+	INIT_LIST_HEAD(&ep->queue);
+	if (ep->dci > 1)
+		list_add_tail(&ep->usb_ep.ep_list, &crg_udc->gadget.ep_list);
+}
+
+void crg_udc_reset_work(struct work_struct *work)
+{
+	struct crg_gadget_dev *crg_udc =
+		container_of(work, struct crg_gadget_dev, udc_reset_work);
+
+	if (crg_udc->gadget_driver)
+		usb_gadget_udc_reset(&crg_udc->gadget, crg_udc->gadget_driver);
+}
+
+int crg_udc_handle_port_status(struct crg_gadget_dev *crg_udc,
+			       unsigned long *flags)
+{
+	struct crg_uccr __iomem *uccr = crg_udc->uccr;
+	bool was_connected = crg_udc->connected;
+	u32 portsc_val;
+	u32 tmp;
+	u32 speed;
+
+	/* handle Port Reset */
+	portsc_val = readl(&uccr->portsc);
+	trace_crg_udc_portsc(portsc_val);
+	writel(portsc_val & CRG_U3DC_PORTSC_W1C_MASK, &uccr->portsc);
+
+	tmp = readl(&uccr->portsc);
+
+	if (portsc_val & CRG_U3DC_PORTSC_PRC) {
+		tmp = readl(&uccr->portsc);
+		if (tmp & CRG_U3DC_PORTSC_PRC) {
+			dev_dbg(crg_udc->dev, "PRC is still set\n");
+		} else if (tmp & CRG_U3DC_PORTSC_PR) {
+			/* first port status change event for port reset */
+			dev_dbg(crg_udc->dev,
+				"PRC is not set, but PR is set\n");
+		} else {
+			if (FIELD_GET(CRG_U3DC_PORTSC_PLS_MASK, tmp) != 0 ||
+			    (!(tmp & CRG_U3DC_PORTSC_PED))) {
+				dev_dbg(crg_udc->dev, "portsc=0x%x no PED\n",
+					tmp);
+				return 0;
+			}
+
+			switch (FIELD_GET(CRG_U3DC_PORTSC_SPEED_MASK, tmp)) {
+			case CRG_U3DC_PORTSC_SPEED_SSP:
+				speed = USB_SPEED_SUPER_PLUS;
+				break;
+			case CRG_U3DC_PORTSC_SPEED_SS:
+				speed = USB_SPEED_SUPER;
+				break;
+			case CRG_U3DC_PORTSC_SPEED_HS:
+				speed = USB_SPEED_HIGH;
+				break;
+
+			case CRG_U3DC_PORTSC_SPEED_FS:
+				speed = USB_SPEED_FULL;
+				break;
+
+			case CRG_U3DC_PORTSC_SPEED_LS:
+				speed = USB_SPEED_LOW;
+				break;
+			default:
+				speed = USB_SPEED_UNKNOWN;
+				return 0;
+			}
+
+			if (crg_udc->device_state >= USB_STATE_DEFAULT) {
+				/*
+				 * A single enumeration commonly involves two
+				 * bus resets (an initial one, then another
+				 * after SET_ADDRESS before SET_CONFIGURATION).
+				 * By the second reset device_state is already
+				 * USB_STATE_ADDRESS, which is >= DEFAULT but
+				 * has no active configuration to tear down.
+				 * Only notify composite when a configuration
+				 * was actually active: otherwise this races
+				 * with the SET_CONFIGURATION that's about to
+				 * arrive, since the deferred work below can
+				 * run after fsg_set_alt() has already raised
+				 * its own FSG_STATE_CONFIG_CHANGE exception
+				 * and reset_config()'s ->disable() would
+				 * overwrite that exception's argument back to
+				 * NULL before the function's thread consumes
+				 * it, silently skipping the real enable.
+				 */
+				bool was_configured = crg_udc->device_state ==
+						       USB_STATE_CONFIGURED;
+				int reinit_ret;
+
+				reinit_ret = crg_udc_reinit(crg_udc);
+				if (reinit_ret) {
+					dev_dbg(crg_udc->dev,
+						"bus reset reinit failed: %d\n",
+						reinit_ret);
+					return reinit_ret;
+				}
+
+				/*
+				 * Notify the function/composite driver of the
+				 * bus reset so it tears down the active
+				 * configuration (usb_gadget_udc_reset() ->
+				 * ->reset()). Without this, a host that
+				 * re-sends the same SET_CONFIGURATION value
+				 * after a reset can see composite treat it as
+				 * a no-op and skip re-running set_alt(), even
+				 * though every endpoint was just disabled in
+				 * software above.
+				 *
+				 * This must not be called synchronously here:
+				 * usb_gadget_udc_reset() re-enters the
+				 * function drivers, which requires dropping
+				 * udc_lock. Doing that mid-event-processing
+				 * would let a second port-status event run
+				 * crg_udc_reinit() again while this event's
+				 * SET_CONFIGURATION/set_alt() work is still
+				 * landing on another thread, tearing the
+				 * just-enabled endpoints back down before
+				 * they can be used. Defer it to a work item
+				 * so this function never has to drop the
+				 * lock.
+				 */
+				if (was_configured && crg_udc->gadget_driver)
+					schedule_work(&crg_udc->udc_reset_work);
+			}
+
+			crg_udc->gadget.speed = speed;
+
+			crg_udc_update_ep0_maxpacketsize(crg_udc);
+
+			crg_udc->connected = 1;
+
+			if (crg_udc->device_state < USB_STATE_RECONNECTING)
+				crg_udc_enable_setup(crg_udc);
+		}
+	}
+	/* handle Port Reset end */
+
+	/* handle Port Connection Change */
+	tmp = readl(&uccr->portsc);
+	if ((tmp & (CRG_U3DC_PORTSC_CCS | CRG_U3DC_PORTSC_PP)) ==
+	    (CRG_U3DC_PORTSC_CCS | CRG_U3DC_PORTSC_PP)) {
+		if (FIELD_GET(CRG_U3DC_PORTSC_PLS_MASK, tmp) != 0 ||
+		    (!(tmp & CRG_U3DC_PORTSC_PED)))
+			return 0;
+
+		switch (FIELD_GET(CRG_U3DC_PORTSC_SPEED_MASK, tmp)) {
+		case CRG_U3DC_PORTSC_SPEED_SSP:
+			speed = USB_SPEED_SUPER_PLUS;
+			break;
+		case CRG_U3DC_PORTSC_SPEED_SS:
+			speed = USB_SPEED_SUPER;
+			break;
+		case CRG_U3DC_PORTSC_SPEED_HS:
+			speed = USB_SPEED_HIGH;
+			break;
+
+		case CRG_U3DC_PORTSC_SPEED_FS:
+			speed = USB_SPEED_FULL;
+			break;
+
+		case CRG_U3DC_PORTSC_SPEED_LS:
+			speed = USB_SPEED_LOW;
+			break;
+		default:
+			speed = USB_SPEED_UNKNOWN;
+			return 0;
+		}
+		crg_udc->gadget.speed = speed;
+
+		crg_udc_update_ep0_maxpacketsize(crg_udc);
+
+		crg_udc->connected = 1;
+
+		if (crg_udc->device_state < USB_STATE_RECONNECTING)
+			crg_udc_enable_setup(crg_udc);
+
+	} else if (!(tmp & CRG_U3DC_PORTSC_CCS)) {
+		bool cable_connected;
+		int ccs_drop_ignore = 0;
+
+		if ((FIELD_GET(CRG_U3DC_PORTSC_PLS_MASK, tmp) == 0x0) &&
+		    (FIELD_GET(CRG_U3DC_PORTSC_SPEED_MASK, tmp) <
+		     CRG_U3DC_PORTSC_SPEED_SS)) {
+			ccs_drop_ignore = 1;
+			dev_err(crg_udc->dev, "ccs glitch detect on HS/FS!\n");
+		}
+
+		if (!ccs_drop_ignore)
+			crg_udc->gadget.speed = USB_SPEED_UNKNOWN;
+
+		cable_connected = crg_udc_vbus_connected(crg_udc);
+		if (cable_connected && !ccs_drop_ignore) {
+			crg_udc->device_state = USB_STATE_POWERED;
+			crg_udc_reinit(crg_udc);
+			crg_udc->connected = 0;
+		} else if (!cable_connected) {
+			dev_dbg(crg_udc->dev,
+				"cable disconnected, rst controller\n");
+			if (was_connected)
+				crg_udc_notify_disconnect(crg_udc, flags);
+			if (crg_udc_reset(crg_udc, flags))
+				dev_err(crg_udc->dev,
+					"controller reset failed on disconnect\n");
+			crg_udc_reset_data_struct(crg_udc);
+			crg_udc->connected = 0;
+			crg_udc_init_ep0(crg_udc);
+
+			crg_udc->device_state = USB_STATE_ATTACHED;
+			return -ECONNRESET;
+		}
+	}
+	/* handle Port Connection Change end*/
+
+	if (portsc_val & CRG_U3DC_PORTSC_PLC) {
+		tmp = readl(&uccr->portsc);
+		if (FIELD_GET(CRG_U3DC_PORTSC_PLS_MASK, tmp) == 0xf) {
+			FIELD_MODIFY(CRG_U3DC_PORTSC_PLS_MASK, &tmp, 0x0);
+			writel(tmp, &uccr->portsc);
+		}
+	}
+
+	return 0;
+}
+
+int crg_udc_handle_xfer_event(struct crg_gadget_dev *crg_udc,
+			      struct crg_event_trb *event, unsigned long *flags)
+{
+	u32 dw3 = le32_to_cpu(event->dw3);
+	u8 dci = FIELD_GET(EVE_TRB_ENDPOINT_ID_MASK, dw3);
+	u16 comp_code =
+		FIELD_GET(EVE_TRB_COMPL_CODE_MASK, le32_to_cpu(event->dw2));
+	struct crg_udc_ep *udc_ep_ptr;
+	struct crg_udc_request *udc_req_ptr;
+	struct crg_transfer_trb *trb;
+	bool trbs_dequeued = false;
+	u32 residual;
+
+	if (dci >= CRG_NUM_EP_CX || dci == 1)
+		return -EINVAL;
+
+	udc_ep_ptr = &crg_udc->udc_ep[dci];
+	if (!udc_ep_ptr->first_trb ||
+	    crg_udc_get_ep_state(crg_udc, dci) == EP_STATE_DISABLED)
+		return -ENODEV;
+
+	if (comp_code != CMPL_CODE_STOPPED &&
+	    comp_code != CMPL_CODE_STOPPED_LENGTH_INVALID &&
+	    comp_code != CMPL_CODE_DISABLED &&
+	    comp_code != CMPL_CODE_DISABLED_LENGTH_INVALID &&
+	    comp_code != CMPL_CODE_HALTED &&
+	    comp_code != CMPL_CODE_HALTED_LENGTH_INVALID)
+		crg_udc_update_dequeue_pt(event, udc_ep_ptr);
+
+	if (is_request_dequeued(crg_udc, udc_ep_ptr, event)) {
+		trbs_dequeued = true;
+		goto queue_more_trbs;
+	}
+
+	switch (comp_code) {
+	case CMPL_CODE_SUCCESS:
+		trb = crg_udc_tran_trb_dma_to_virt(
+			udc_ep_ptr,
+			(dma_addr_t)le32_to_cpu(event->dw0) |
+				((dma_addr_t)le32_to_cpu(event->dw1) << 32));
+		if (!trb || list_empty(&udc_ep_ptr->queue))
+			break;
+
+		/*
+		 * A control transfer has separate data-stage and status-stage
+		 * TRBs.  The status request is an internal zero-length request
+		 * and its successful completion must not go through
+		 * crg_udc_build_td().  Doing that attempts to queue another
+		 * control TD while the status request is already on the EP0
+		 * software queue, leaving it stuck in -EINPROGRESS.  The next
+		 * SETUP then sees a non-empty queue and enumeration fails.
+		 */
+		if (FIELD_GET(TRB_TYPE_MASK, le32_to_cpu(trb->dw3)) ==
+		    TRB_TYPE_XFER_STATUS_STAGE) {
+			udc_req_ptr = list_first_entry(&udc_ep_ptr->queue,
+						       struct crg_udc_request,
+						       queue);
+			if (dci == 0) {
+				udc_req_ptr->usb_req.actual = 0;
+				trbs_dequeued = true;
+				crg_udc_req_done(udc_ep_ptr, udc_req_ptr, 0);
+				crg_udc_ep0_req_complete(udc_ep_ptr, flags);
+			} else {
+				dev_err(crg_udc->dev,
+					"status-stage TRB on endpoint %u\n",
+					dci);
+			}
+			break;
+		}
+
+		if (!FIELD_GET(TRB_CHAIN_BIT_MASK, le32_to_cpu(trb->dw3))) {
+			udc_req_ptr = list_first_entry(&udc_ep_ptr->queue,
+						       struct crg_udc_request,
+						       queue);
+			trbs_dequeued = true;
+
+			if (udc_req_ptr->all_trbs_queued) {
+				udc_req_ptr->usb_req.actual =
+					udc_req_ptr->usb_req.length -
+					udc_req_ptr->buff_len_left;
+				residual = FIELD_GET(EVE_TRB_TRAN_LEN_MASK,
+						     le32_to_cpu(event->dw2));
+				if (residual > udc_req_ptr->usb_req.actual)
+					residual = 0;
+				udc_req_ptr->usb_req.actual -= residual;
+				crg_udc_req_done(udc_ep_ptr, udc_req_ptr, 0);
+
+				if (dci == 0 && udc_ep_ptr->desc &&
+				    usb_endpoint_xfer_control(udc_ep_ptr->desc))
+					crg_udc_ep0_req_complete(udc_ep_ptr,
+								 flags);
+			} else {
+				int ret;
+
+				ret = crg_udc_build_td(udc_ep_ptr, udc_req_ptr);
+				if (ret) {
+					udc_req_ptr->usb_req.actual =
+						udc_req_ptr->usb_req.length -
+						udc_req_ptr->buff_len_left;
+					crg_udc_req_done(udc_ep_ptr, udc_req_ptr,
+							 -EIO);
+				} else {
+					crg_udc_knock_doorbell(crg_udc, dci);
+				}
+			}
+		}
+		break;
+
+	case CMPL_CODE_SHORT_PKT:
+		if (list_empty(&udc_ep_ptr->queue))
+			break;
+
+		udc_req_ptr = list_first_entry(&udc_ep_ptr->queue,
+					       struct crg_udc_request, queue);
+		udc_req_ptr->usb_req.actual = udc_req_ptr->usb_req.length -
+					      udc_req_ptr->buff_len_left;
+		residual = FIELD_GET(EVE_TRB_TRAN_LEN_MASK,
+				     le32_to_cpu(event->dw2));
+		if (residual > udc_req_ptr->usb_req.actual)
+			residual = 0;
+		udc_req_ptr->usb_req.actual -= residual;
+
+		if (udc_req_ptr->usb_req.short_not_ok &&
+		    usb_endpoint_dir_out(udc_ep_ptr->desc))
+			crg_udc_req_done(udc_ep_ptr, udc_req_ptr, -EREMOTEIO);
+		else
+			crg_udc_req_done(udc_ep_ptr, udc_req_ptr, 0);
+
+		trbs_dequeued = true;
+		crg_udc_advance_dequeue_pt(udc_ep_ptr);
+		break;
+
+	case CMPL_CODE_PROTOCOL_STALL:
+	case CMPL_CODE_TRB_STALL:
+		if (!list_empty(&udc_ep_ptr->queue)) {
+			udc_req_ptr = list_first_entry(&udc_ep_ptr->queue,
+						       struct crg_udc_request,
+						       queue);
+			crg_udc_req_done(udc_ep_ptr, udc_req_ptr, -EPIPE);
+			trbs_dequeued = true;
+		}
+		if (dci == 0)
+			crg_udc->setup_status = WAIT_FOR_SETUP;
+		break;
+
+	case CMPL_CODE_SETUP_TAG_MISMATCH:
+		/*
+		 * EP0 SETUP changes are serialized by the SETUP event handler.
+		 * When a new SETUP arrives, crg_udc_abort_ep0_for_setup() stops
+		 * EP0, gives back the old request, resets the EP0 ring, and then
+		 * accepts the new SETUP. A SETUP_TAG_MISMATCH event generated
+		 * afterwards therefore belongs to the aborted control transfer.
+		 *
+		 * Do not complete the current EP0 request or change setup_status
+		 * here: doing so could operate on the new control transfer and
+		 * recreate the EP0 queue/state race this path is intended to avoid.
+		 */
+		if (dci == 0) {
+			dev_dbg(crg_udc->dev,
+				"ignoring stale EP0 SETUP_TAG_MISMATCH\n");
+			break;
+		}
+
+		/* This completion code is meaningful only for EP0. */
+		dev_warn(crg_udc->dev,
+			 "SETUP_TAG_MISMATCH on non-EP0 endpoint %u\n", dci);
+		break;
+
+	case CMPL_CODE_BABBLE_DETECTED_ERR:
+	case CMPL_CODE_DATA_BUFFER_ERR:
+	case CMPL_CODE_INVALID_STREAM_TYPE_ERR:
+	case CMPL_CODE_RING_UNDERRUN:
+	case CMPL_CODE_RING_OVERRUN:
+	case CMPL_CODE_EVENT_RING_FULL_ERR:
+	case CMPL_CODE_ISOCH_BUFFER_OVERRUN:
+	case CMPL_CODE_USB_TRANS_ERR:
+	case CMPL_CODE_TRB_ERR:
+		dev_err(crg_udc->dev, "transfer error, completion code 0x%x\n",
+			comp_code);
+		if (!list_empty(&udc_ep_ptr->queue)) {
+			udc_req_ptr = list_first_entry(&udc_ep_ptr->queue,
+						       struct crg_udc_request,
+						       queue);
+			crg_udc_req_done(udc_ep_ptr, udc_req_ptr, -EIO);
+			trbs_dequeued = true;
+		}
+		if (dci)
+			set_ep_halt(crg_udc, dci, flags);
+		break;
+
+	case CMPL_CODE_STOPPED:
+	case CMPL_CODE_STOPPED_LENGTH_INVALID:
+	case CMPL_CODE_HALTED:
+	case CMPL_CODE_HALTED_LENGTH_INVALID:
+	case CMPL_CODE_DISABLED:
+	case CMPL_CODE_DISABLED_LENGTH_INVALID:
+		break;
+
+	default:
+		dev_err(crg_udc->dev,
+			"unsupported transfer completion code 0x%x\n",
+			comp_code);
+		if (!list_empty(&udc_ep_ptr->queue)) {
+			udc_req_ptr = list_first_entry(&udc_ep_ptr->queue,
+						       struct crg_udc_request,
+						       queue);
+			crg_udc_req_done(udc_ep_ptr, udc_req_ptr, -EIO);
+			trbs_dequeued = true;
+		}
+		break;
+	}
+
+queue_more_trbs:
+	if (trbs_dequeued && udc_ep_ptr->tran_ring_full) {
+		udc_ep_ptr->tran_ring_full = false;
+		if (crg_udc_queue_pending_trbs(udc_ep_ptr))
+			udc_ep_ptr->tran_ring_full = true;
+	}
+
+	return 0;
+}
+
+int crg_udc_handle_event(struct crg_gadget_dev *crg_udc,
+			 struct crg_event_trb *event, unsigned long *flags)
+{
+	int ret;
+
+	trace_crg_udc_event_trb(event);
+
+	switch (FIELD_GET(EVE_TRB_TYPE_MASK, le32_to_cpu(event->dw3))) {
+	case TRB_TYPE_EVT_PORT_STATUS_CHANGE:
+		if (crg_udc->device_state == USB_STATE_RECONNECTING) {
+			crg_udc->portsc_on_reconnecting = 1;
+			break;
+		}
+		/* 1.Port Reset  2.Port Connection Change 3.Port Link Change */
+		ret = crg_udc_handle_port_status(crg_udc, flags);
+		if (ret)
+			return ret;
+
+		break;
+	case TRB_TYPE_EVT_TRANSFER:
+		if (crg_udc->device_state < USB_STATE_RECONNECTING) {
+			dev_err(crg_udc->dev,
+				"Xfer compl event rcved when dev state=%d !\n",
+				crg_udc->device_state);
+			break;
+		}
+
+		ret = crg_udc_handle_xfer_event(crg_udc, event, flags);
+		if (ret)
+			return ret;
+		break;
+	case TRB_TYPE_EVT_SETUP_PKT: {
+		struct usb_ctrlrequest setup;
+		u32 dw0 = le32_to_cpu(event->dw0);
+		u32 dw1 = le32_to_cpu(event->dw1);
+		u8 setup_tag;
+
+		BUILD_BUG_ON(sizeof(struct usb_ctrlrequest) != 8);
+		setup.bRequestType = dw0 & 0xff;
+		setup.bRequest = (dw0 >> 8) & 0xff;
+		setup.wValue = cpu_to_le16((dw0 >> 16) & 0xffff);
+		setup.wIndex = cpu_to_le16(dw1 & 0xffff);
+		setup.wLength = cpu_to_le16((dw1 >> 16) & 0xffff);
+
+		setup_tag = FIELD_GET(EVE_TRB_SETUP_TAG_MASK,
+				      le32_to_cpu(event->dw3));
+		if (crg_udc->setup_status != WAIT_FOR_SETUP) {
+			/* A new SETUP aborts the previous control transfer. The old
+			 * EP0 request must be completed and its TRBs discarded before
+			 * the new setup is handed to the gadget driver.
+			 */
+			if (crg_udc_abort_ep0_for_setup(crg_udc)) {
+				crg_udc_set_ep0_halt(crg_udc);
+				break;
+			}
+		}
+
+		crg_udc_handle_setup_pkt(crg_udc, &setup, setup_tag, flags);
+
+		break;
+	}
+	default:
+		dev_dbg(crg_udc->dev, "unexpected TRB_TYPE = 0x%lx",
+			FIELD_GET(EVE_TRB_TYPE_MASK, le32_to_cpu(event->dw3)));
+		break;
+	}
+
+	return 0;
+}
+
+/*
+ * Invoke the bound gadget driver's ->disconnect() callback, if any.
+ *
+ * The gadget disconnect callback may be called in interrupt context.
+ * Release only udc_lock while keeping local IRQs disabled.
+ */
+void crg_udc_notify_disconnect(struct crg_gadget_dev *crg_udc,
+			       unsigned long *flags)
+{
+	/* Report link loss even when there is no gadget callback. */
+	usb_gadget_set_state(&crg_udc->gadget, USB_STATE_NOTATTACHED);
+
+	if (!crg_udc->async_callbacks || !crg_udc->gadget_driver ||
+	    !crg_udc->gadget_driver->disconnect)
+		return;
+
+	spin_unlock_irqrestore(&crg_udc->udc_lock, *flags);
+	crg_udc->gadget_driver->disconnect(&crg_udc->gadget);
+	spin_lock_irqsave(&crg_udc->udc_lock, *flags);
+}
+
+int crg_udc_ep_halt(struct crg_udc_ep *udc_ep_ptr, int halt, int ignore_wedge,
+		    unsigned long *flags)
+{
+	struct crg_gadget_dev *crg_udc = udc_ep_ptr->crg_udc;
+	struct crg_uccr __iomem *uccr = crg_udc->uccr;
+	enum crg_ep_state ep_state;
+	bool reset_seq_only = false;
+	int do_halt;
+	u32 param0;
+	u32 tmp;
+
+	if (!udc_ep_ptr->desc) {
+		dev_err(crg_udc->dev, "NULL desc\n");
+		return -EINVAL;
+	}
+
+	if (usb_endpoint_xfer_isoc(udc_ep_ptr->desc)) {
+		dev_err(crg_udc->dev, "Isoc ep, halt not supported\n");
+		return -EOPNOTSUPP;
+	}
+
+	if (udc_ep_ptr->dci == 0)
+		return 0;
+
+	ep_state = crg_udc_get_ep_state(crg_udc, udc_ep_ptr->dci);
+
+	if (ep_state == EP_STATE_DISABLED)
+		return 0;
+
+	if (ep_state == EP_STATE_HALTED) {
+		if (halt != 0)
+			return 0;
+
+		/* want unhalt an halted ep */
+		if (udc_ep_ptr->wedge && !ignore_wedge) {
+			do_halt = -1;
+			reset_seq_only = true;
+		} else {
+			do_halt = 0;
+		}
+
+	} else {
+		/* ep state == running or stopped */
+		if (halt != 0) {
+			/* want halt a running ep */
+			do_halt = 1;
+		} else {
+			/* reset a running ep */
+			do_halt = 0;
+			reset_seq_only = true;
+		}
+	}
+
+	trace_crg_udc_ep_halt(udc_ep_ptr, do_halt == 1, udc_ep_ptr->wedge);
+
+	param0 = BIT(udc_ep_ptr->dci);
+	if (do_halt == 1) {
+		/* setting ep to halt */
+		if (crg_udc_issue_command(crg_udc, CRG_CMD_SET_HALT, param0, 0))
+			return -EIO;
+		if (readl_poll_timeout_atomic(&uccr->ep_running, tmp,
+					      !(tmp & param0), 10, 10000))
+			dev_warn(crg_udc->dev, "ep %d halt wait timed out\n",
+				 udc_ep_ptr->dci);
+
+		if (crg_udc->gadget.speed >= USB_SPEED_SUPER) {
+			/* clean up the request queue */
+			crg_udc_ep_nuke(udc_ep_ptr, -ECONNRESET);
+
+			udc_ep_ptr->deq_pt = udc_ep_ptr->enq_pt;
+			udc_ep_ptr->tran_ring_full = false;
+		}
+		/* clean up the request queue */
+		udc_ep_ptr->ep_state = EP_STATE_HALTED;
+	} else if (do_halt == 0) {
+		/* clearing ep halt state */
+		/* reset sequence number */
+		if (crg_udc_issue_command(crg_udc, CRG_CMD_RESET_SEQNUM, param0,
+					  0))
+			return -EIO;
+
+		if (!reset_seq_only) {
+			/* Clear halt for a halted EP. */
+			/* NOTE: we must CLEAR_HALT first,
+			 * then SET_TR_DQPTR
+			 */
+			if (crg_udc_issue_command(crg_udc, CRG_CMD_CLEAR_HALT,
+						  param0, 0))
+				return -EIO;
+			if (crg_udc_epcx_update_dqptr(udc_ep_ptr))
+				return -EIO;
+		}
+
+		udc_ep_ptr->wedge = 0;
+		udc_ep_ptr->ep_state = EP_STATE_RUNNING;
+		/* set endpoint to running state */
+		/* clear pause for the endpoint */
+		if (!list_empty(&udc_ep_ptr->queue)) {
+			struct crg_transfer_trb *next = udc_ep_ptr->deq_pt + 1;
+
+			if (FIELD_GET(TRB_TYPE_MASK, le32_to_cpu(next->dw3)) ==
+			    TRB_TYPE_LINK) {
+				udc_ep_ptr->pcs ^= 1;
+				if (crg_udc_epcx_update_dqptr(udc_ep_ptr)) {
+					udc_ep_ptr->pcs ^= 1;
+					return -EIO;
+				}
+				udc_ep_ptr->pcs ^= 1;
+			}
+			crg_udc_knock_doorbell(crg_udc, udc_ep_ptr->dci);
+		}
+	} else {
+		/* wedged EP deny CLEAR HALT */
+		dev_dbg(crg_udc->dev, "wedged EP deny CLEAR HALT dci = %d\n",
+			udc_ep_ptr->dci);
+		/* reset sequence number */
+		if (reset_seq_only &&
+		    crg_udc_issue_command(crg_udc, CRG_CMD_RESET_SEQNUM, param0,
+					  0))
+			return -EIO;
+	}
+
+	return 0;
+}
+
+enum crg_ep_state crg_udc_get_ep_state(struct crg_gadget_dev *crg_udc, int dci)
+{
+	struct crg_udc_ep *udc_ep_ptr;
+
+	/*
+	 * DCI 0 is ep0; DCI 1 is unused in the Corigine ep-context layout
+	 * (ep contexts start from DCI 2 for ep1in/ep1out). This is why the
+	 * ep-context array is indexed as p_epcx[dci - 2] elsewhere.
+	 */
+	if (dci < 0 || dci >= CRG_NUM_EP_CX || dci == 1)
+		return EP_STATE_DISABLED;
+
+	udc_ep_ptr = &crg_udc->udc_ep[dci];
+
+	return udc_ep_ptr->ep_state;
+}
+
+u32 crg_udc_index2dci(u16 index)
+{
+	if (index == 0)
+		return 0;
+
+	return (index & USB_ENDPOINT_NUMBER_MASK) * 2 +
+	       ((index & USB_DIR_IN) ? 0 : 1);
+}
+
+int crg_udc_init_device_context(struct crg_gadget_dev *crg_udc)
+{
+	struct crg_uccr __iomem *uccr = crg_udc->uccr;
+	int ret;
+	u32 buff_length;
+	dma_addr_t mapping;
+
+	/* ep0 is not included in ep contexts in crg udc */
+	buff_length = (CRG_NUM_EP_CX - 2) * sizeof(struct crg_ep_cx);
+
+	if (!crg_udc->ep_cx.cpu_addr) {
+		crg_udc->ep_cx.cpu_addr = dma_alloc_coherent(
+			crg_udc->dev, buff_length, &mapping, GFP_ATOMIC);
+		if (!crg_udc->ep_cx.cpu_addr) {
+			ret = -ENOMEM;
+			return ret;
+		}
+	} else {
+		mapping = crg_udc->ep_cx.dma_addr;
+	}
+
+	crg_udc->p_epcx = crg_udc->ep_cx.cpu_addr;
+	crg_udc->ep_cx.len = buff_length;
+	crg_udc->ep_cx.dma_addr = mapping;
+	memset(crg_udc->p_epcx, 0, buff_length);
+	/*
+	 * Ensure the endpoint contexts are initialized before making their
+	 * DMA address visible to the controller.
+	 */
+	dma_wmb();
+	/* hw ops DCBAPLO DCBAPHI */
+	writel(lower_32_bits(crg_udc->ep_cx.dma_addr), &uccr->dcbaplo);
+	writel(upper_32_bits(crg_udc->ep_cx.dma_addr), &uccr->dcbaphi);
+
+	return 0;
+}
+
+void crg_udc_init_ep_info(struct crg_gadget_dev *crg_udc)
+{
+	int i;
+
+	/* udc_ep[0] is reserved */
+	crg_ep_struct_setup(crg_udc, 0, NULL);
+
+	for (i = 1; i < CRG_NUM_EP_CX / 2; i++) {
+		char name[14];
+
+		snprintf(name, sizeof(name), "ep%din", i);
+		crg_ep_struct_setup(crg_udc, i * 2, name);
+		snprintf(name, sizeof(name), "ep%dout", i);
+		crg_ep_struct_setup(crg_udc, i * 2 + 1, name);
+	}
+}
+
+static bool is_request_dequeued(struct crg_gadget_dev *crg_udc,
+				struct crg_udc_ep *udc_ep,
+				struct crg_event_trb *event)
+{
+	struct crg_udc_request *udc_req;
+	u32 trb_pt_lo = le32_to_cpu(event->dw0);
+	u32 trb_pt_hi = le32_to_cpu(event->dw1);
+	u64 trb_addr = (u64)trb_pt_lo | ((u64)trb_pt_hi << 32);
+	struct crg_transfer_trb *trb_pt;
+	bool status = true;
+
+	if (udc_ep->dci == 0 || list_empty(&udc_ep->queue))
+		return false;
+
+	trb_pt = crg_udc_tran_trb_dma_to_virt(udc_ep, trb_addr);
+	if (!trb_pt)
+		return false;
+	list_for_each_entry(udc_req, &udc_ep->queue, queue) {
+		if (trb_pt == udc_req->last_trb ||
+		    trb_pt == udc_req->first_trb) {
+			status = false;
+			break;
+		}
+
+		if (crg_udc_is_pointer_less_than(trb_pt, udc_req->last_trb,
+						 udc_ep) &&
+		    crg_udc_is_pointer_less_than(udc_req->first_trb, trb_pt,
+						 udc_ep)) {
+			status = false;
+			break;
+		}
+	}
+
+	return status;
+}
+
+void crg_udc_knock_doorbell(struct crg_gadget_dev *crg_udc, int dci)
+{
+	u32 tmp;
+	struct crg_uccr __iomem *uccr;
+
+	uccr = crg_udc->uccr;
+	/* Ensure everything is written before notifying the HW */
+	dma_wmb();
+
+	tmp = FIELD_PREP(CRG_U3DC_DB_TARGET_MASK, dci);
+	writel(tmp, &uccr->doorbell);
+
+	trace_crg_udc_doorbell(crg_udc, dci);
+}
+
+/*
+ * Complete a request and give it back to the gadget layer via the
+ * standard usb_gadget_giveback_request() helper.
+ *
+ * Must be called with udc_lock held and interrupts disabled.
+ *
+ * The request's ->complete() callback must not sleep. crg_udc_req_done() drops
+ * udc_lock before the giveback and re-acquires it before returning.
+ * Callers therefore cannot assume state protected by udc_lock is
+ * unchanged across this call.
+ */
+void crg_udc_req_done(struct crg_udc_ep *udc_ep,
+		      struct crg_udc_request *udc_req, int status)
+{
+	struct crg_gadget_dev *crg_udc = udc_ep->crg_udc;
+
+	if (likely(udc_req->usb_req.status == -EINPROGRESS))
+		udc_req->usb_req.status = status;
+
+	trace_crg_udc_request_giveback(udc_ep, udc_req);
+
+	list_del_init(&udc_req->queue);
+
+	crg_udc_unmap_request(udc_ep, udc_req);
+
+	if (udc_req->usb_req.complete) {
+		spin_unlock(&crg_udc->udc_lock);
+		usb_gadget_giveback_request(&udc_ep->usb_ep, &udc_req->usb_req);
+		spin_lock(&crg_udc->udc_lock);
+	}
+}
+
+static int set_ep_halt(struct crg_gadget_dev *crg_udc, int dci,
+		       unsigned long *flags)
+{
+	struct crg_uccr __iomem *uccr = crg_udc->uccr;
+	struct crg_udc_ep *udc_ep_ptr = &crg_udc->udc_ep[dci];
+	u32 param0;
+	u32 tmp;
+
+	if (dci == 0)
+		return 0;
+
+	if (udc_ep_ptr->ep_state == EP_STATE_DISABLED ||
+	    udc_ep_ptr->ep_state == EP_STATE_HALTED)
+		return 0;
+	if (udc_ep_ptr->desc && usb_endpoint_xfer_isoc(udc_ep_ptr->desc))
+		return -EOPNOTSUPP;
+
+	param0 = BIT(dci);
+	if (crg_udc_issue_command(crg_udc, CRG_CMD_SET_HALT, param0, 0))
+		return -EIO;
+	if (readl_poll_timeout_atomic(&uccr->ep_running, tmp, !(tmp & param0),
+				      10, 10000))
+		return -ETIMEDOUT;
+	/* clean up the request queue */
+	crg_udc_ep_nuke(udc_ep_ptr, -ECONNRESET);
+
+	udc_ep_ptr->deq_pt = udc_ep_ptr->enq_pt;
+	udc_ep_ptr->tran_ring_full = false;
+	udc_ep_ptr->ep_state = EP_STATE_HALTED;
+
+	return 0;
+}
+
+void crg_udc_clear_req_container(struct crg_udc_request *udc_req_ptr)
+{
+	udc_req_ptr->buff_len_left = 0;
+	udc_req_ptr->trbs_needed = 0;
+	udc_req_ptr->all_trbs_queued = 0;
+	udc_req_ptr->first_trb = NULL;
+	udc_req_ptr->last_trb = NULL;
+	udc_req_ptr->sg = NULL;
+	udc_req_ptr->sg_index = 0;
+	udc_req_ptr->sg_offset = 0;
+}
+
+const struct usb_ep_ops crg_udc_ep_ops = {
+	.enable = crg_udc_ep_enable,
+	.disable = crg_udc_ep_disable,
+	.alloc_request = crg_udc_alloc_request,
+	.free_request = crg_udc_free_request,
+	.queue = crg_udc_ep_queue,
+	.dequeue = crg_udc_ep_dequeue,
+	.set_halt = crg_udc_ep_set_halt,
+	.set_wedge = crg_udc_ep_set_wedge,
+};
+
+const struct usb_gadget_ops crg_gadget_ops = {
+	.get_frame = crg_gadget_get_frame,
+	.wakeup = crg_gadget_wakeup,
+	.set_remote_wakeup = crg_gadget_set_remote_wakeup,
+	.set_selfpowered = crg_gadget_set_selfpowered,
+	.pullup = crg_gadget_pullup,
+	.udc_start = crg_gadget_start,
+	.udc_stop = crg_gadget_stop,
+	.udc_set_speed = crg_udc_set_speed,
+	.udc_async_callbacks = crg_gadget_async_callbacks,
+};
diff --git a/drivers/usb/gadget/udc/crg/crg-udc-trace.c b/drivers/usb/gadget/udc/crg/crg-udc-trace.c
new file mode 100644
index 000000000000..ee091708dbe0
--- /dev/null
+++ b/drivers/usb/gadget/udc/crg/crg-udc-trace.c
@@ -0,0 +1,9 @@
+// SPDX-License-Identifier: GPL-2.0+
+/*
+ * Axiado AX3000/AX3005 USB Device Controller driver.
+ * Trace Support.
+ *
+ * Copyright (c) 2021-2026 Axiado Corporation.
+ */
+#define CREATE_TRACE_POINTS
+#include "crg-udc-trace.h"

-- 
2.34.1

[PATCH v3 6/6] usb: gadget: udc: crg: Add core driver and build integration

[Petar Stepanovic]

This patch introduces the core probe/remove functions for the Axiado
AX3000/AX3005 UDC driver, and wires up the directory into the build
system via Kconfig and Makefile.

Co-developed-by: Naveen Kumar Rajgiri Bassappa <rbnaveenkumar@axiado.com>
Signed-off-by: Naveen Kumar Rajgiri Bassappa <rbnaveenkumar@axiado.com>
Co-developed-by: Krutik Shah <krutikshah@axiado.com>
Signed-off-by: Krutik Shah <krutikshah@axiado.com>
Signed-off-by: Petar Stepanovic <pstepanovic@axiado.com>
---
 MAINTAINERS                               |   1 +
 drivers/usb/gadget/udc/Kconfig            |   2 +
 drivers/usb/gadget/udc/Makefile           |   1 +
 drivers/usb/gadget/udc/crg/Kconfig        |  16 +
 drivers/usb/gadget/udc/crg/Makefile       |   9 +
 drivers/usb/gadget/udc/crg/crg-udc-core.c | 706 ++++++++++++++++++++++++++++++
 6 files changed, 735 insertions(+)

diff --git a/MAINTAINERS b/MAINTAINERS
index 58c5c8e00870..d5b6a04a95ad 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -4515,6 +4515,7 @@ M:	Prasad Bolisetty <pbolisetty@axiado.com>
 L:	linux-usb@vger.kernel.org
 S:	Supported
 F:	Documentation/devicetree/bindings/usb/axiado,ax3000-udc.yaml
+F:	drivers/usb/gadget/udc/crg/
 
 AYANEO PLATFORM EC DRIVER
 M:	Antheas Kapenekakis <lkml@antheas.dev>
diff --git a/drivers/usb/gadget/udc/Kconfig b/drivers/usb/gadget/udc/Kconfig
index 7ddbc2606686..72a5a8a04982 100644
--- a/drivers/usb/gadget/udc/Kconfig
+++ b/drivers/usb/gadget/udc/Kconfig
@@ -421,6 +421,8 @@ source "drivers/usb/gadget/udc/aspeed-vhub/Kconfig"
 
 source "drivers/usb/gadget/udc/cdns2/Kconfig"
 
+source "drivers/usb/gadget/udc/crg/Kconfig"
+
 #
 # LAST -- dummy/emulated controller
 #
diff --git a/drivers/usb/gadget/udc/Makefile b/drivers/usb/gadget/udc/Makefile
index 1b9b1a4f9c57..227d2a0b525b 100644
--- a/drivers/usb/gadget/udc/Makefile
+++ b/drivers/usb/gadget/udc/Makefile
@@ -38,3 +38,4 @@ obj-$(CONFIG_USB_ASPEED_UDC)	+= aspeed_udc.o
 obj-$(CONFIG_USB_BDC_UDC)	+= bdc/
 obj-$(CONFIG_USB_MAX3420_UDC)	+= max3420_udc.o
 obj-$(CONFIG_USB_CDNS2_UDC)	+= cdns2/
+obj-$(CONFIG_USB_CRG_UDC)	+= crg/
diff --git a/drivers/usb/gadget/udc/crg/Kconfig b/drivers/usb/gadget/udc/crg/Kconfig
new file mode 100644
index 000000000000..7167c4943389
--- /dev/null
+++ b/drivers/usb/gadget/udc/crg/Kconfig
@@ -0,0 +1,16 @@
+# SPDX-License-Identifier: GPL-2.0
+config USB_CRG_UDC
+	tristate "Corigine / Axiado USB Device Controller"
+	depends on USB_GADGET
+	depends on HAS_DMA
+	depends on ARCH_AXIADO || COMPILE_TEST
+	help
+	  Say Y here to support the USB device controller found on Axiado AX3000
+	  and AX3005 SoCs, which is based on the Corigine USB 3.1 IP.
+
+	  This controller supports High-Speed and SuperSpeed.
+
+	  Say "y" to link the driver statically, or "m" to build a
+	  dynamically linked module called "crg-udc.ko".
+
+	  If unsure, say N.
diff --git a/drivers/usb/gadget/udc/crg/Makefile b/drivers/usb/gadget/udc/crg/Makefile
new file mode 100644
index 000000000000..96b209fb59fc
--- /dev/null
+++ b/drivers/usb/gadget/udc/crg/Makefile
@@ -0,0 +1,9 @@
+# SPDX-License-Identifier: GPL-2.0
+
+# define_trace.h needs to know how to find our trace header
+CFLAGS_crg-udc-trace.o		:= -I$(src)
+
+obj-$(CONFIG_USB_CRG_UDC)	+= crg-udc.o
+
+crg-udc-y			:= crg-udc-core.o crg-udc-ring.o crg-udc-ep0.o crg-udc-gadget.o
+crg-udc-$(CONFIG_TRACING)	+= crg-udc-trace.o
diff --git a/drivers/usb/gadget/udc/crg/crg-udc-core.c b/drivers/usb/gadget/udc/crg/crg-udc-core.c
new file mode 100644
index 000000000000..80a4fc6a61e8
--- /dev/null
+++ b/drivers/usb/gadget/udc/crg/crg-udc-core.c
@@ -0,0 +1,706 @@
+// SPDX-License-Identifier: GPL-2.0+
+/*
+ * Axiado AX3000/AX3005 USB Device Controller driver.
+ *
+ * Copyright (c) 2019 Corigine Inc.
+ * Copyright (c) 2021-2026 Axiado Corporation.
+ */
+#include <linux/clk.h>
+#include <linux/err.h>
+#include <linux/interrupt.h>
+#include <linux/iopoll.h>
+#include <linux/module.h>
+#include <linux/of.h>
+#include <linux/phy/phy.h>
+#include <linux/platform_device.h>
+#include <linux/pm.h>
+#include <linux/property.h>
+#include <linux/reset.h>
+#include <linux/slab.h>
+
+#include "crg-udc.h"
+
+static int crg_gadget_irq_init(struct platform_device *pdev,
+			       struct crg_gadget_dev *crg_udc);
+static void crg_udc_phy_exit(void *data);
+
+static void crg_udc_gadget_release(struct device *dev)
+{
+	struct crg_gadget_dev *crg_udc;
+
+	crg_udc = container_of(dev, struct crg_gadget_dev, gadget.dev);
+	kfree(crg_udc);
+}
+
+/*
+ * command related ops
+ */
+int crg_udc_issue_command(struct crg_gadget_dev *crg_udc,
+			  enum crg_cmd_type type, u32 param0, u32 param1)
+{
+	struct crg_uccr __iomem *uccr = crg_udc->uccr;
+	u32 status;
+	bool check_complete = false;
+	u32 tmp;
+
+	/*
+	 * The command state machine only runs while the controller itself is
+	 * running. Once RUN is cleared (e.g. during udc_stop()/reset()), the
+	 * hardware never processes queued commands, so ACTIVE never clears.
+	 * Skip the busy/completion checks in that case instead of getting
+	 * stuck on a command the hardware will never service.
+	 */
+	tmp = readl(&uccr->control);
+	if (tmp & CRG_U3DC_CTRL_RUN)
+		check_complete = true;
+
+	if (check_complete) {
+		tmp = readl(&uccr->cmd_control);
+		if (tmp & CRG_U3DC_CMD_CTRL_ACTIVE) {
+			dev_err(crg_udc->dev,
+				"previous command is not complete\n");
+			return -EBUSY;
+		}
+	}
+	/* Ensure that everything is written before issuing new command */
+	dma_wmb();
+
+	writel(param0, &uccr->cmd_param0);
+	writel(param1, &uccr->cmd_param1);
+
+	tmp = CRG_U3DC_CMD_CTRL_ACTIVE |
+	      FIELD_PREP(CRG_U3DC_CMD_CTRL_TYPE_MASK, type);
+	writel(tmp, &uccr->cmd_control);
+
+	if (!check_complete)
+		return 0;
+
+	if (readl_poll_timeout_atomic(&uccr->cmd_control, tmp,
+				      !(tmp & CRG_U3DC_CMD_CTRL_ACTIVE), 1,
+				      1000)) {
+		dev_dbg(crg_udc->dev, "command timed out\n");
+		return -ETIMEDOUT;
+	}
+
+	status = FIELD_GET(CRG_U3DC_CMD_CTRL_STATUS_MASK, tmp);
+	if (status) {
+		dev_dbg(crg_udc->dev, "command failed with status %u\n",
+			status);
+		return -EIO;
+	}
+
+	return 0;
+}
+
+static void crg_udc_clk_disable(void *data)
+{
+	clk_disable_unprepare(data);
+}
+
+void crg_udc_free_resources(struct crg_gadget_dev *crg_udc)
+{
+	int i;
+
+	if (crg_udc->uccr) {
+		u32 val = readl(&crg_udc->uccr->control);
+
+		val &= ~(CRG_U3DC_CTRL_INT_EN | CRG_U3DC_CTRL_RUN);
+		writel(val, &crg_udc->uccr->control);
+	}
+	if (crg_udc->irq >= 0) {
+		devm_free_irq(crg_udc->dev, crg_udc->irq, crg_udc);
+		crg_udc->irq = -1;
+		crg_udc->gadget.irq = 0;
+	}
+
+	for (i = 0; i < CRG_RING_NUM; i++) {
+		struct crg_udc_event *udc_event = &crg_udc->udc_event[i];
+
+		if (udc_event->event_ring.cpu_addr) {
+			dma_free_coherent(crg_udc->dev,
+					  udc_event->event_ring.len,
+					  udc_event->event_ring.cpu_addr,
+					  udc_event->event_ring.dma_addr);
+			udc_event->event_ring.cpu_addr = NULL;
+			udc_event->event_ring.dma_addr = DMA_MAPPING_ERROR;
+			udc_event->event_ring.len = 0;
+			udc_event->evt_dq_pt = NULL;
+		}
+		if (udc_event->erst.cpu_addr) {
+			dma_free_coherent(crg_udc->dev, udc_event->erst.len,
+					  udc_event->erst.cpu_addr,
+					  udc_event->erst.dma_addr);
+			udc_event->erst.cpu_addr = NULL;
+			udc_event->erst.dma_addr = DMA_MAPPING_ERROR;
+			udc_event->erst.len = 0;
+			udc_event->p_erst = NULL;
+		}
+	}
+
+	if (crg_udc->ep_cx.cpu_addr) {
+		dma_free_coherent(crg_udc->dev, crg_udc->ep_cx.len,
+				  crg_udc->ep_cx.cpu_addr,
+				  crg_udc->ep_cx.dma_addr);
+		crg_udc->ep_cx.cpu_addr = NULL;
+		crg_udc->ep_cx.dma_addr = DMA_MAPPING_ERROR;
+		crg_udc->ep_cx.len = 0;
+		crg_udc->p_epcx = NULL;
+	}
+
+	for (i = 0; i < CRG_NUM_EP_CX; i++) {
+		struct crg_udc_ep *udc_ep = &crg_udc->udc_ep[i];
+
+		if (udc_ep->tran_ring_info.cpu_addr) {
+			dma_free_coherent(crg_udc->dev,
+					  udc_ep->tran_ring_info.len,
+					  udc_ep->tran_ring_info.cpu_addr,
+					  udc_ep->tran_ring_info.dma_addr);
+			udc_ep->tran_ring_info.cpu_addr = NULL;
+			udc_ep->tran_ring_info.dma_addr = DMA_MAPPING_ERROR;
+			udc_ep->tran_ring_info.len = 0;
+			udc_ep->first_trb = NULL;
+			udc_ep->last_trb = NULL;
+			udc_ep->enq_pt = NULL;
+			udc_ep->deq_pt = NULL;
+		}
+	}
+
+	if (crg_udc->status_req) {
+		crg_udc_free_request(&crg_udc->udc_ep[0].usb_ep,
+				     &crg_udc->status_req->usb_req);
+		crg_udc->status_req = NULL;
+	}
+}
+
+static int crg_udc_init_hw(struct crg_gadget_dev *crg_udc)
+{
+	struct device *dev = crg_udc->dev;
+	int ret;
+
+	crg_udc->clk = devm_clk_get_optional(dev, NULL);
+	if (IS_ERR(crg_udc->clk))
+		return dev_err_probe(dev, PTR_ERR(crg_udc->clk),
+				     "failed to get clock\n");
+
+	ret = clk_prepare_enable(crg_udc->clk);
+	if (ret)
+		return dev_err_probe(dev, ret, "failed to enable clock\n");
+
+	ret = devm_add_action_or_reset(dev, crg_udc_clk_disable, crg_udc->clk);
+	if (ret)
+		return ret;
+
+	crg_udc->rst =
+		devm_reset_control_get_optional_exclusive_deasserted(dev, NULL);
+	if (IS_ERR(crg_udc->rst))
+		return dev_err_probe(dev, PTR_ERR(crg_udc->rst),
+				     "failed to get reset\n");
+
+	crg_udc->phy = devm_phy_optional_get(dev, NULL);
+	if (IS_ERR(crg_udc->phy))
+		return dev_err_probe(dev, PTR_ERR(crg_udc->phy),
+				     "failed to get phy\n");
+
+	ret = phy_init(crg_udc->phy);
+	if (ret)
+		return dev_err_probe(dev, ret, "failed to init phy\n");
+
+	ret = phy_power_on(crg_udc->phy);
+	if (ret) {
+		phy_exit(crg_udc->phy);
+		return dev_err_probe(dev, ret, "failed to power on phy\n");
+	}
+
+	return devm_add_action_or_reset(dev, crg_udc_phy_exit, crg_udc->phy);
+}
+
+static void crg_udc_phy_exit(void *data)
+{
+	struct phy *phy = data;
+
+	phy_power_off(phy);
+	phy_exit(phy);
+}
+
+static int crg_udc_probe(struct platform_device *pdev)
+{
+	unsigned long flags = 0;
+	int ret;
+	int i;
+	struct crg_gadget_dev *crg_udc;
+	const struct crg_udc_priv *priv;
+
+	priv = device_get_match_data(&pdev->dev);
+	if (!priv)
+		return -ENODEV;
+
+	crg_udc = kzalloc_obj(*crg_udc, GFP_KERNEL);
+	if (!crg_udc)
+		return -ENOMEM;
+	usb_initialize_gadget(&pdev->dev, &crg_udc->gadget,
+			      crg_udc_gadget_release);
+
+	crg_udc->dev = &pdev->dev;
+	crg_udc->irq = -1;
+
+	ret = dma_set_mask_and_coherent(&pdev->dev, DMA_BIT_MASK(64));
+	if (ret) {
+		ret = dma_set_mask_and_coherent(&pdev->dev, DMA_BIT_MASK(32));
+		if (ret) {
+			dev_err(&pdev->dev, "no usable DMA mask\n");
+			goto err_free_resources;
+		}
+	}
+
+	ret = crg_udc_init_hw(crg_udc);
+	if (ret)
+		goto err_free_resources;
+
+	crg_udc->ep0_desc.bLength = USB_DT_ENDPOINT_SIZE;
+	crg_udc->ep0_desc.bDescriptorType = USB_DT_ENDPOINT;
+	crg_udc->ep0_desc.bEndpointAddress = 0;
+	crg_udc->ep0_desc.bmAttributes = USB_ENDPOINT_XFER_CONTROL;
+	crg_udc->ep0_desc.wMaxPacketSize = cpu_to_le16(64);
+
+	crg_udc->u12_forbidden = priv->quirks & CRG_QUIRK_DIS_U1U2_ENTRY;
+	crg_udc->u12_initiate_forbidden = priv->quirks &
+					  CRG_QUIRK_DIS_U1U2_INITIATION;
+	crg_udc->reinit_ep0_on_bus_reset = priv->quirks &
+					   CRG_QUIRK_REINIT_EP0_ON_RESET;
+
+	spin_lock_init(&crg_udc->udc_lock);
+	INIT_WORK(&crg_udc->udc_reset_work, crg_udc_reset_work);
+	platform_set_drvdata(pdev, crg_udc);
+
+	crg_udc->gadget.ops = &crg_gadget_ops;
+	crg_udc->gadget.ep0 = &crg_udc->udc_ep[0].usb_ep;
+	INIT_LIST_HEAD(&crg_udc->gadget.ep_list);
+	/*
+	 * The controller supports up to SuperSpeed. Honour the standard
+	 * "maximum-speed" property to cap it; default to the hardware
+	 * maximum when the property is absent.
+	 */
+	crg_udc->gadget.max_speed = usb_get_maximum_speed(&pdev->dev);
+	if (crg_udc->gadget.max_speed == USB_SPEED_UNKNOWN)
+		crg_udc->gadget.max_speed = USB_SPEED_SUPER;
+	crg_udc->max_speed = crg_udc->gadget.max_speed;
+	crg_udc->gadget.speed = USB_SPEED_UNKNOWN;
+	crg_udc->gadget.name = KBUILD_MODNAME;
+	crg_udc->gadget.sg_supported = true;
+	crg_udc->gadget.wakeup_capable = true;
+	crg_udc->connected = 0;
+	crg_udc->dev_addr = 0;
+
+	crg_udc->mmio_virt_base = devm_platform_ioremap_resource(pdev, 0);
+	if (IS_ERR(crg_udc->mmio_virt_base)) {
+		ret = dev_err_probe(&pdev->dev,
+				    PTR_ERR(crg_udc->mmio_virt_base),
+				    "failed to map registers\n");
+		goto err_put_gadget;
+	}
+
+	/* set controller device role */
+	writel((readl(crg_udc->mmio_virt_base + CRG_UDC_MODE_REG) |
+		CRG_UDC_ROLE_DEVICE),
+	       crg_udc->mmio_virt_base + CRG_UDC_MODE_REG);
+	for (i = 0; i < CRG_RING_NUM; i++) {
+		crg_udc->uicr[i] = crg_udc->mmio_virt_base + CRG_UICR_OFFSET +
+				   i * CRG_UICR_STRIDE;
+	}
+	crg_udc->uccr = crg_udc->mmio_virt_base + CRG_UCCR_OFFSET;
+
+	spin_lock_irqsave(&crg_udc->udc_lock, flags);
+	ret = crg_udc_reset(crg_udc, &flags);
+	spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+	if (ret)
+		goto err_free_resources;
+
+	crg_udc_clear_portpm(crg_udc);
+
+	ret = crg_gadget_irq_init(pdev, crg_udc);
+	if (ret)
+		goto err_free_resources;
+
+	ret = crg_udc_reset_data_struct(crg_udc);
+	if (ret) {
+		dev_err(crg_udc->dev, "crg_udc_reset_data_struct error\n");
+		goto err_free_resources;
+	}
+
+	crg_udc_init_ep_info(crg_udc);
+	ret = crg_udc_init_ep0(crg_udc);
+	if (ret)
+		goto err_free_resources;
+
+	crg_udc->udc_ep[0].desc = &crg_udc->ep0_desc;
+
+	ret = usb_add_gadget(&crg_udc->gadget);
+	if (ret)
+		goto err_free_resources;
+
+	return 0;
+
+err_free_resources:
+	crg_udc_free_resources(crg_udc);
+err_put_gadget:
+	usb_put_gadget(&crg_udc->gadget);
+	return ret;
+}
+
+static void crg_udc_remove(struct platform_device *pdev)
+{
+	struct crg_gadget_dev *crg_udc = platform_get_drvdata(pdev);
+	u32 tmp;
+
+	crg_udc->device_state = USB_STATE_ATTACHED;
+	crg_udc_set_vbus_detect(crg_udc, false);
+
+	usb_del_gadget(&crg_udc->gadget);
+	cancel_work_sync(&crg_udc->udc_reset_work);
+	crg_udc_free_resources(crg_udc);
+
+	/* set controller host role */
+	tmp = readl(crg_udc->mmio_virt_base + CRG_UDC_MODE_REG) & ~0x1;
+	writel(tmp, crg_udc->mmio_virt_base + CRG_UDC_MODE_REG);
+
+	platform_set_drvdata(pdev, NULL);
+	usb_put_gadget(&crg_udc->gadget);
+}
+
+static void crg_udc_shutdown(struct platform_device *pdev)
+{
+	struct crg_gadget_dev *crg_udc = platform_get_drvdata(pdev);
+	unsigned long flags;
+
+	spin_lock_irqsave(&crg_udc->udc_lock, flags);
+	crg_udc->device_state = USB_STATE_ATTACHED;
+	crg_udc_set_pullup(crg_udc, false);
+	spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+	crg_udc_set_vbus_detect(crg_udc, false);
+	if (crg_udc->uccr) {
+		u32 val = readl(&crg_udc->uccr->control);
+
+		val &= ~CRG_U3DC_CTRL_INT_EN;
+		writel(val, &crg_udc->uccr->control);
+	}
+	if (crg_udc->irq >= 0)
+		synchronize_irq(crg_udc->irq);
+}
+
+static int __maybe_unused crg_udc_suspend(struct device *dev)
+{
+	struct crg_gadget_dev *crg_udc = dev_get_drvdata(dev);
+	unsigned long flags;
+
+	/* Stop the pullup while the controller is suspended. */
+	spin_lock_irqsave(&crg_udc->udc_lock, flags);
+	crg_udc_set_pullup(crg_udc, false);
+	spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+	return 0;
+}
+
+static int __maybe_unused crg_udc_resume(struct device *dev)
+{
+	struct crg_gadget_dev *crg_udc = dev_get_drvdata(dev);
+	unsigned long flags;
+
+	spin_lock_irqsave(&crg_udc->udc_lock, flags);
+	if (crg_udc->gadget_driver && crg_udc->pullup_on)
+		crg_udc_set_pullup(crg_udc, true);
+	spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+
+	return 0;
+}
+
+/* Program the speed limit while the controller is stopped. */
+int crg_udc_program_speed(struct crg_gadget_dev *crg_udc)
+{
+	u32 cfg, speed;
+
+	switch (crg_udc->max_speed) {
+	case USB_SPEED_FULL:
+		speed = CRG_U3DC_CFG0_MAXSPEED_FS;
+		break;
+	case USB_SPEED_HIGH:
+		speed = CRG_U3DC_CFG0_MAXSPEED_HS;
+		break;
+	case USB_SPEED_SUPER:
+		speed = CRG_U3DC_CFG0_MAXSPEED_SS;
+		break;
+	case USB_SPEED_SUPER_PLUS:
+		speed = CRG_U3DC_CFG0_MAXSPEED_SSP;
+		break;
+	default:
+		return -EINVAL;
+	}
+
+	cfg = readl(&crg_udc->uccr->config0);
+	cfg &= ~CRG_U3DC_CFG0_MAXSPEED_MASK;
+	cfg |= FIELD_PREP(CRG_U3DC_CFG0_MAXSPEED_MASK, speed);
+	writel(cfg, &crg_udc->uccr->config0);
+
+	return 0;
+}
+
+int crg_udc_reset_data_struct(struct crg_gadget_dev *crg_udc)
+{
+	int i;
+	int ret;
+	struct crg_uccr __iomem *uccr = crg_udc->uccr;
+	u32 val;
+
+	crg_udc->ep0_state = EP0_STATE_IDLE;
+
+	val = readl(&uccr->control);
+	val &= (~(CRG_U3DC_CTRL_INT_EN | CRG_U3DC_CTRL_RUN));
+	writel(val, &uccr->control);
+
+	ret = crg_udc_program_speed(crg_udc);
+	if (ret)
+		return ret;
+
+	for (i = 0; i < CRG_RING_NUM; i++) {
+		ret = crg_udc_init_event_ring(crg_udc, i);
+		if (ret)
+			return ret;
+	}
+
+	ret = crg_udc_init_device_context(crg_udc);
+	if (ret)
+		return ret;
+
+	if (!crg_udc->status_req) {
+		struct usb_request *req;
+
+		req = crg_udc_alloc_request(&crg_udc->udc_ep[0].usb_ep,
+					    GFP_ATOMIC);
+		if (!req)
+			return -ENOMEM;
+
+		crg_udc->status_req =
+			container_of(req, struct crg_udc_request, usb_req);
+	}
+
+	/* other hw ops */
+	return 0;
+}
+
+static void crg_gadget_handle_interrupt(struct crg_gadget_dev *crg_udc)
+{
+	struct crg_uccr __iomem *uccr = crg_udc->uccr;
+	unsigned long flags = 0;
+	u32 tmp_status;
+	int ret;
+
+	spin_lock_irqsave(&crg_udc->udc_lock, flags);
+
+	tmp_status = readl(&uccr->status);
+
+	if (tmp_status & CRG_U3DC_STATUS_SYS_ERR) {
+		bool was_connected = crg_udc->connected;
+
+		dev_err(crg_udc->dev, "controller system error\n");
+		writel(CRG_U3DC_STATUS_SYS_ERR, &uccr->status);
+		ret = crg_udc_reset(crg_udc, &flags);
+		if (!ret)
+			ret = crg_udc_reset_data_struct(crg_udc);
+		if (!ret)
+			ret = crg_udc_init_ep0(crg_udc);
+		if (ret)
+			dev_err(crg_udc->dev,
+				"failed to recover from controller error: %d\n",
+				ret);
+		crg_udc->gadget.speed = USB_SPEED_UNKNOWN;
+		crg_udc->connected = 0;
+		if (was_connected)
+			crg_udc_notify_disconnect(crg_udc, &flags);
+		if (!ret && crg_udc->pullup_on)
+			crg_udc_set_pullup(crg_udc, true);
+
+		/*
+		 * tmp_status was captured before the reset above, which tore
+		 * down and rebuilt the event rings. It no longer reflects
+		 * the controller's current state, and if recovery failed the
+		 * rings may not be consistent at all. Skip the rest of this
+		 * interrupt's processing; any event still pending will raise
+		 * a fresh interrupt once the controller is running again.
+		 */
+		spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+		return;
+	}
+
+	if (tmp_status & CRG_U3DC_STATUS_EINT) {
+		int i;
+
+		writel(CRG_U3DC_STATUS_EINT, &uccr->status);
+
+		for (i = 0; i < CRG_RING_NUM; i++) {
+			ret = crg_udc_process_event_ring(crg_udc, i, &flags);
+			if (ret)
+				dev_err(crg_udc->dev,
+					"process event ring error: %d\n", ret);
+		}
+	}
+
+	if (crg_udc->device_state == USB_STATE_RECONNECTING &&
+	    crg_udc->portsc_on_reconnecting == 1 &&
+	    crg_udc_is_event_rings_empty(crg_udc)) {
+		crg_udc->portsc_on_reconnecting = 0;
+		crg_udc_handle_port_status(crg_udc, &flags);
+	}
+
+	if (crg_udc->device_state == USB_STATE_RECONNECTING &&
+	    crg_udc->connected == 1)
+		crg_udc_prepare_for_setup(crg_udc);
+
+	spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+}
+
+/*
+ * Primary handler only checks whether the shared interrupt belongs to the
+ * controller. All controller work, including reset polling, runs in the
+ * threaded handler and therefore never busy-waits in hard IRQ context.
+ */
+static irqreturn_t crg_udc_common_irq(int irq, void *dev)
+{
+	struct crg_gadget_dev *crg_udc = dev;
+	struct crg_uccr __iomem *uccr = crg_udc->uccr;
+	u32 status = readl(&uccr->status);
+
+	if (!(status & (CRG_U3DC_STATUS_EINT | CRG_U3DC_STATUS_SYS_ERR)))
+		return IRQ_NONE;
+
+	return IRQ_WAKE_THREAD;
+}
+
+static irqreturn_t crg_udc_irq_thread(int irq, void *dev)
+{
+	struct crg_gadget_dev *crg_udc = dev;
+
+	crg_gadget_handle_interrupt(crg_udc);
+
+	return IRQ_HANDLED;
+}
+
+static int crg_gadget_irq_init(struct platform_device *pdev,
+			       struct crg_gadget_dev *crg_udc)
+{
+	int irq;
+	int ret;
+
+	irq = platform_get_irq(pdev, 0);
+	if (irq < 0)
+		return irq;
+
+	/*
+	 * IRQF_ONESHOT keeps this (level-triggered) line masked until
+	 * crg_udc_irq_thread() has cleared the status bits, which is the
+	 * correct pairing for IRQF_SHARED with a threaded handler.
+	 */
+	ret = devm_request_threaded_irq(&pdev->dev, irq, crg_udc_common_irq,
+					crg_udc_irq_thread,
+					IRQF_SHARED | IRQF_ONESHOT, "crg_udc",
+					crg_udc);
+	if (ret) {
+		dev_err(crg_udc->dev, "request of irq%d failed\n", irq);
+	} else {
+		crg_udc->irq = irq;
+		crg_udc->gadget.irq = irq;
+	}
+
+	return ret;
+}
+
+int crg_udc_reset(struct crg_gadget_dev *crg_udc, unsigned long *flags)
+{
+	struct crg_uccr __iomem *uccr = crg_udc->uccr;
+	u32 i, tmp;
+	struct crg_udc_ep *udc_ep_ptr;
+	u32 val;
+
+	val = readl(&uccr->control);
+	val |= CRG_U3DC_CTRL_SWRST;
+	writel(val, &uccr->control);
+	crg_udc->resetting = true;
+	spin_unlock_irqrestore(&crg_udc->udc_lock, *flags);
+	/* Poll until SWRST clears; hardware reset takes up to 10 ms */
+	if (readl_poll_timeout_atomic(&uccr->control, tmp,
+			       !(tmp & CRG_U3DC_CTRL_SWRST), 10, 10000)) {
+		spin_lock_irqsave(&crg_udc->udc_lock, *flags);
+		crg_udc->resetting = false;
+		dev_err(crg_udc->dev, "reset error\n");
+		return -ETIMEDOUT;
+	}
+	spin_lock_irqsave(&crg_udc->udc_lock, *flags);
+	crg_udc->resetting = false;
+
+	crg_udc_clear_portpm(crg_udc);
+
+	crg_udc->setup_status = WAIT_FOR_SETUP;
+	/* Base on Figure 9-1, default USB_STATE is attached */
+	crg_udc->device_state = USB_STATE_ATTACHED;
+	crg_udc->dev_addr = 0;
+
+	/* Complete any reqs on EP1-EP15 queue */
+	for (i = 2; i < CRG_NUM_EP_CX; i++) {
+		udc_ep_ptr = &crg_udc->udc_ep[i];
+
+		if (udc_ep_ptr->desc) {
+			udc_ep_ptr->desc = NULL;
+			udc_ep_ptr->comp_desc = NULL;
+			crg_udc_ep_nuke(udc_ep_ptr, -ESHUTDOWN);
+		}
+		udc_ep_ptr->tran_ring_full = false;
+		udc_ep_ptr->ep_state = EP_STATE_DISABLED;
+		/*
+		 * This bypasses the generic usb_ep_disable() API, so the
+		 * framework's own usb_ep.enabled bookkeeping needs to be
+		 * cleared directly here too (crg_udc_reinit() already does
+		 * this for the live bus-reset case). Otherwise a later
+		 * usb_ep_enable() call for this endpoint short-circuits on
+		 * the stale "already enabled" flag without ever calling our
+		 * .enable callback, leaving desc/the ring never set up for
+		 * the new session.
+		 */
+		udc_ep_ptr->usb_ep.enabled = 0;
+	}
+	crg_udc->num_enabled_eps = 0;
+
+	/* Complete any reqs on EP0 queue */
+	udc_ep_ptr = &crg_udc->udc_ep[0];
+	if (udc_ep_ptr->desc) {
+		udc_ep_ptr->desc = NULL;
+		udc_ep_ptr->comp_desc = NULL;
+		crg_udc_ep_nuke(udc_ep_ptr, -ESHUTDOWN);
+	}
+
+	return 0;
+}
+
+static const struct crg_udc_priv crg_udc_data = {
+	.quirks = CRG_UDC_QUIRKS_DEFAULT,
+};
+
+static const struct of_device_id of_crg_udc_match[] = {
+	{ .compatible = "axiado,ax3000-udc", .data = &crg_udc_data },
+	{ },
+};
+MODULE_DEVICE_TABLE(of, of_crg_udc_match);
+
+static SIMPLE_DEV_PM_OPS(crg_udc_pm_ops, crg_udc_suspend, crg_udc_resume);
+
+static struct platform_driver crg_udc_driver = {
+	.probe = crg_udc_probe,
+	.remove = crg_udc_remove,
+	.shutdown = crg_udc_shutdown,
+	.driver = {
+		.name = KBUILD_MODNAME,
+		.of_match_table = of_crg_udc_match,
+		.pm = &crg_udc_pm_ops,
+	},
+};
+module_platform_driver(crg_udc_driver);
+
+MODULE_AUTHOR("Axiado Corporation");
+MODULE_DESCRIPTION("Axiado AX3000/AX3005 USB Device Controller");
+MODULE_LICENSE("GPL");

-- 
2.34.1