From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-ed1-f41.google.com (mail-ed1-f41.google.com [209.85.208.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id ADA3A2E7378 for ; Mon, 8 Jun 2026 09:11:30 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=pass smtp.client-ip=209.85.208.41 ARC-Seal:i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780909892; cv=pass; b=pC9L6/WhQT/X7C3FcjEpC8sQYj2jXogT1E17DcrEG0arnA54LEsKL1IPE/Y7ubBDpV8mb6aTiOHEpr5o9Fj1eOd4ddsj5B6tRmjQB+/kd6RU23r8XvCHbSNRGP4usZiwxhWsKqzjKZDBFZt28+c2YE8St45v9/bwYC0N9LTJc6w= ARC-Message-Signature:i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780909892; c=relaxed/simple; bh=z8TRp/eansM7NQFOUeoehuya+exenRZRyWpFPzqPkDI=; h=MIME-Version:References:In-Reply-To:From:Date:Message-ID:Subject: To:Cc:Content-Type; b=kmfYsyOpUevUrHSYsgIrppB+QZtShwBUo3ocKW8REnFPxXk516CnChju4Hc5tAyjNyAJKcrtOpjwZKW7IxzXsBxOpqbxX+McpJJK7hzmyk3u8yz72HwTZ7peaQ19JZYO+SLc++Pcr+hGzVBwag+oskBTF0HwRwC5FB2jkiLo5wk= ARC-Authentication-Results:i=2; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=fKFk5CGK; arc=pass smtp.client-ip=209.85.208.41 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="fKFk5CGK" Received: by mail-ed1-f41.google.com with SMTP id 4fb4d7f45d1cf-68b482888c3so6854973a12.0 for ; Mon, 08 Jun 2026 02:11:30 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1780909889; cv=none; d=google.com; s=arc-20240605; b=aEAR3RF+cwaCIV7I9QwUmKvEG8S6YMAqjA+qWgTYSU8LR6JQP0hhU9JLGwTPqKJ+MG s33vShUJ+H8Ebd6kYDpisAYKaH0CxbBLPL2Zvv9oeBYbdeaOm0zg8CBGnd/u+fQQ+UFF dWLX5/CPPiWW2lIB9+/lO98buztRqNmiBdDBsUAXQKceCt1FltaCUSr4ugtRycm6rhEF QVUOtmbSOjc6EwCbWiyFmhBaT1gu1oFnw66Vci0N+Q02npMIjUEqY040AyrJqkipbo1F uBaVVcVlaepzFjpWqJzP0dD1V3eLTCB4788yfh6+SjwE5/pTZgk+DUlRxXTvsaBGBqYT flnw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20240605; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:dkim-signature; bh=WelUToYaunngdS4iiekhxlbrmFbbBfrAT5j9UWPzR4A=; fh=zqS3EXXGti/X+nh9lxQC5oT1jTKO3+g0RSDXONwbSXM=; b=YtKnqn4hy7CXcKKa18yCkubv0mjf9vEzXBm/FThD7/tkc5U7Dz8UQM6VRhvSykMe5w BMI5xCEHCwRAINQC4NKSXmxcn4spWIJodMbuTs/OhPzio+Rhjf0AzJkoPHbaiFbaVkO0 HenECMXEywFnr2pFv5ZacEfo2YdzVExNDpjEqtpqp6J8rlh5665t2aoUVTmKW9Et3hgm dfbiobYSdlUfpXuJpN+Nj3WKyjbiHgYxFmtCLUuDDyIIRAibjCF83Gd7ccK65/B167l0 Z0/9E9U5hXQ6gtDpHCml/LN2m7h6/NeQ+9y/DjFYSTD+HORPpMJNjVYUKZUxpOf/sJQt vVzA==; darn=vger.kernel.org ARC-Authentication-Results: i=1; mx.google.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1780909889; x=1781514689; darn=vger.kernel.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=WelUToYaunngdS4iiekhxlbrmFbbBfrAT5j9UWPzR4A=; b=fKFk5CGK+16WTVKCrEDMSAAv4uWxj+hoK1unEpLNU+Ql/qbz+ZBE/2LFqssrxc7ELa 25Ao27Evj1jXo5ry+u2Uig5D9E3sjBNF3GmPOrE/sf/Hxt7eejibebwd4J9GZY0Ckv3A 8RSEb9n1V/OoN1FzNeFuYgiilizmINj5A23RwV5wSTaoLcTTHxZHB5l5UAbxmF5XzQWH j9BJDJG84DsBskFONiigdIFPc71sg6rHBJkvKhFhy7flQxHe5RkMWiqLaWIR8oD+T6P/ OCNaLFUZymGFMvp2aohuLC9QKskkjDyBVSxficPUZ1CGEid0yQMCF0LCeGAVc91nVidl TBAw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1780909889; x=1781514689; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=WelUToYaunngdS4iiekhxlbrmFbbBfrAT5j9UWPzR4A=; b=XwmGjKA6Tf7IahWsXgmfvR147rlCu9QAmQ+5JfkgYpz8dxQcrwqb5xn0JiIo0jsx5P vzNC1Pf21awb9+JPiLozQVWeLDUW4JpGXl71t964CBMY8oPoqDJRcpNNkc5ht2VNWS2X iOl5xHWqdYCr+5p/HSxZyErZYfHJaRK9XKAnv43XJL3X56jvOum3Argz+TwyiRKzs9i7 56GS2Tm4Y1I5zLRJxVRYO2gf+M/1L/otjcxwGphufssAMwcK+s91pYTBDveHlwWTqA1h zjjTfGZ08Al/HVZyt7D03zaVsSiJjqBudg/FhSPRSSzqwA9VfQV6GClzSce9VJ/jqzRz zkXQ== X-Forwarded-Encrypted: i=1; AFNElJ/Pi8ePc+alp+S51KCnx9euLKBgxrOkfkkHx8/dkeckvFg094r/trRfEiaqg3oGSU0A3WdO8eftuz6of0Q=@vger.kernel.org X-Gm-Message-State: AOJu0Yy9vpSomDJ8q3bFjxyYFCl8ctG7fepQw9BwAKXvw1GT5yn4Nfaw q9QNAfebCQR0nNZ+Y04cVUl/5DlQXuySrkBdIuSJidHJp8hQGqj9ylJJF3GWdIaHrXcQrGrUKcM D5cAoUecr3mBj+tG4OqJ4otZovaE+xns= X-Gm-Gg: Acq92OEcYOtqywAPHVR7sCQ1LZqlwGzZa97BSsLLyq2Ji0/68z0l7mmJ7CdTZSEhdkn MGTe5UmqvjJmmgOyG14g6+1mMTk1sLLQg6yht5liK6eBb8HKnceGW0lONzaiORb3Y7P9p5QQF+t f4PH+IbHxUT3voa/zEdGgrHrSNFyKTHouv2DCvccFGOl+CTDODb3qzSxPF4XzGXevdeCvMEkFt+ 26PnwXjLbR94MQQcmO8irx2bkj7aMQT/BXQ0WU5uGMhigsOTRuhcwyZo/+L3v17V+JoTkErPra6 PnmjFuADbgEd9y/x4g== X-Received: by 2002:a17:907:2d12:b0:beb:2980:e2d2 with SMTP id a640c23a62f3a-bf3a6b13b16mr580109766b.8.1780909888634; Mon, 08 Jun 2026 02:11:28 -0700 (PDT) Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 References: <20260604135255.62682-1-midgy971@gmail.com> <3d99569e-9c3a-49d1-93fb-1335382523e9@rock-chips.com> In-Reply-To: From: Midgy Balon Date: Mon, 8 Jun 2026 11:14:17 +0200 X-Gm-Features: AVVi8CefQKT7ky3Sqeq6MkiJ04hcvCvZiupw30w_yNvYYxCVtDBxVUNhbRaFSOk Message-ID: Subject: Re: [RFC PATCH v3 0/9] accel: rocket: Add RK3568 NPU support To: Chaoyi Chen Cc: tomeu@tomeuvizoso.net, ogabbay@kernel.org, heiko@sntech.de, robh@kernel.org, krzk+dt@kernel.org, conor+dt@kernel.org, joro@8bytes.org, will@kernel.org, robin.murphy@arm.com, dri-devel@lists.freedesktop.org, linux-rockchip@lists.infradead.org, devicetree@vger.kernel.org, linux-arm-kernel@lists.infradead.org, iommu@lists.linux.dev, linux-kernel@vger.kernel.org, Simon Xue , Finley Xiao Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Hello Chaoyi, Following up on the need_regulator suggestion -- I implemented and tested it on the board, and unfortunately it doesn't avoid the deadlock on RK3568; it moves it from boot to the NPU job submit. What I did: gave the RK3568 NPU power domain a regulator (a DOMAIN_M_R variant with need_regulator =3D true), wired domain-supply =3D <&vdd_npu>, and dropped t= he regulator-always-on workaround. Boot is now clean and the NPU probes, but there is a warning during boot: rockchip-pm-domain ...: Failed to create device link (0x180) with supplie= r 0-0020 for .../power-domain@6 (0-0020 is the rk809 PMIC that supplies vdd_npu.) Then on the first NPU job submit the board hard-hangs with an RCU stall: rcu: INFO: rcu_preempt detected stalls on CPUs/tasks: rcu: 3-...!: (1 GPs behind) ... rcu: rcu_preempt kthread starved for 5115 jiffies! ... RCU_GP_WAIT_FQS(5) rcu: Unless rcu_preempt kthread gets sufficient CPU time, OOM is now expe= cted My reading: vdd_npu is on the rk809 *I2C* PMIC, so when genpd enables/disables the regulator during the NPU's runtime-PM power transition, the I2C transfer runs in a context that starves RCU and the box freezes. (I suspect need_regulator is fine on the RK3588 NPU because its supply isn't behind an I2C PMIC.) The always-on workaround avoids this precisely because genpd never touches the I2C regulator in that path. So: for an NPU domain whose supply is an I2C PMIC, is there a supported way to let genpd own the regulator without performing the I2C op in the power-transition path (a deferred/async regulator enable, or a flag), or should RK3568 keep vdd_n= pu as regulator-always-on? For v4 I'll keep always-on unless there's a cleaner pa= th. Thanks, Midgy Le lun. 8 juin 2026 =C3=A0 10:05, Midgy Balon a =C3=A9= crit : > > Hello Chaoyi, > > Thanks -- this is exactly what I needed. > > - v2/DTE: will do. I'll keep building on Simon's per-device-ops series --= with > that in place the NPU MMU can use the 32-bit-DTE ops (the per-ops GFP_D= MA32 > that's already in mainline) without the global rk_ops conflict. I'll > keep it as > a stated dependency of the v4 cover letter. > > - vdd_npu: I'll switch the RK3568 NPU > power domain to need_regulator + domain-supply =3D <&vdd_npu> and drop = the > regulator-always-on workaround. I suspect that's also the right fix for= the > power-off/on de-idle issue I described -- the always-on was really > just papering > over the domain not being modelled with a regulator. I'll confirm on > the board. > > - AUTO_GATING: thanks for the commit references -- I'll keep the bit-31 > read-modify-write form with your Suggested-by and write the comment > from those. > For the record: on v7.1-rc6 the NPU MMU also completes translations > on the reset > value (I couldn't reproduce a page-walk stall without the write), so I'= ll note > in the commit that it matches the vendor clock-gating handling rather t= han > fixing a failure I can reproduce here -- happy to drop it if the iommu > maintainers would prefer. > > - PVTPLL/NoC: I'll follow up with Finley. First I'll check whether the > need_regulator change resolves the NoC re-power de-idle on its own; > if it still > I'll bring him the details (the genpd power-on de-idle ack and the > BUS_IDLE_ST state). > > I'll send a v4 with these. Thanks again for the quick, detailed answers. > > Kind regards, > Midgy > > Le lun. 8 juin 2026 =C3=A0 03:40, Chaoyi Chen a =C3=A9crit : > > > > Hi Midgy, > > > > On 6/8/2026 5:03 AM, Midgy Balon wrote: > > > Hi Chaoyi, > > > > > > Thanks a lot for looking at this -- input from Rockchip is exactly wh= at this > > > series needs. > > > > > >> Hmmm. If I understand correctly, the NPU IOMMU should be v2 rather t= han v1, > > >> implying it should support 40-bit PAs. Nevertheless, please note tha= t the > > >> upper limit for DTE is 32 bits. > > > > > > Understood, and that 32-bit-DTE note is the crux of the trouble I had= , so let > > > me lay out what I see and ask how you'd prefer to solve it. > > > > > > The mainline node is already v2 (rockchip,rk3568-iommu in rk356x-base= .dtsi). > > > The problem on this 8 GiB board: with the v2 ops the page-table alloc= ations > > > (gfp_flags =3D=3D 0) can land above 4 GiB, so the DTE ends up > 32 bi= ts and the > > > NPU's first translation faults with DMA_READ_ERROR. To work around th= at I had > > > switched the NPU MMU to the v1 compatible (rockchip,iommu), whose ops= set > > > GFP_DMA32 and keep the DTE sub-4 GiB. That works in isolation, but be= cause the > > > driver keeps a single global rk_ops, a v1 NPU MMU then trips > > > WARN_ON(rk_ops !=3D ops) against the SoC's v2 instances (VOP/VDEC), w= hich is why > > > I based the series on Simon's per-device-ops work. > > > > > > So my question: with per-device ops in place, what's the intended way= to keep > > > the NPU MMU on v2 *and* cap its DTE at 32 bits on boards with >4 GiB = of RAM? > > > A v2 ops variant carrying GFP_DMA32 for this device, or is there a re= gister/ > > > config bit that constrains the DTE address? I'd rather follow the Roc= kchip > > > intent here than carry the v1 workaround. (Simon, cc'd -- this is rig= ht next to > > > your per-device-ops series.) > > > > > > > If Simon's method works, please use it :) > > > > >> Can these operations not be completed via the pmdomain driver? > > >> If some operations are controlled by TF-A, are you using open source= TF-A? > > > > > > Most of it is in pmdomain already. Power-on and NoC de-idle are done = by the > > > RK3568 NPU power domain (genpd) at power-on -- the driver no longer p= okes the > > > PMU directly. Two things remain outside it: > > > > > > - vdd_npu: I mark it regulator-always-on in DT rather than wiring it= as the > > > domain's domain-supply, because as a domain-supply it created a de= vice-link > > > to the I2C PMIC (rk809) and genpd's power-off QoS-save path then h= ung > > > reading the NPU QoS registers behind the (gated) NoC. If there's a= clean way > > > to let genpd own vdd_npu without that I2C ordering deadlock I'd mu= ch prefer > > > that -- pointers welcome. > > > > > > > Please refer to the patch below regarding the RK3588 NPU pmdomain. > > In short, you need to set a "need_regulator" for the RK3568 NPU pmdomai= n. > > > > https://lore.kernel.org/all/20251216055247.13150-1-rmxpzlb@gmail.com/ > > > > > - the NPU compute clock (PVTPLL): set from the driver via SCMI, and = only > > > needed for actual compute, not for bring-up. > > > > > > One more pmdomain observation from testing, possibly relevant to how = the NPU > > > domain should be modelled: the domain's power-off/on cycle doesn't re= liably > > > re-de-idle the NoC. If the NPU is probed after genpd has already powe= red the > > > (unused) domain off, the power-on de-idle fails ("failed to set idle = on domain > > > 'npu'") and the NPU IOMMU then takes an external abort on its first M= MIO access. > > > Probing the NPU before the unused-domain power-off, or marking the do= main > > > always-on, both avoid it. Is the NoC de-idle expected to work on a ge= npd > > > re-power here, or should this domain effectively stay on? > > > > > > > Not quite sure what's going on with PVTPLL and NOC. > > Maybe @Finley knows about this? > > > > > On TF-A: yes -- bl31 is built from upstream arm-trusted-firmware > > > (github.com/ARM-software/arm-trusted-firmware, RK3568 platform), prov= iding PSCI > > > and the SCMI clock service. The only closed blob in the boot chain is= Rockchip's > > > DDR init (rkbin), which is the standard situation for mainline RK356x= . > > > > -- > > Best, > > Chaoyi