[PATCH 0/2] drm/radeon: fix VCE1 encoding on SI by handling the VCE interrupt

[PATCH 0/2] drm/radeon: fix VCE1 encoding on SI by handling the VCE interrupt

[Matheus Tagliari]

Hardware H.264 encoding with the VCE block on SI GPUs (radeon driver)
hangs the GPU after a few dozen frames.  I hit it with the KDE Plasma
RDP server, which encodes through VAAPI, and reproduced it with a
plain "ffmpeg -vaapi_device ... -c:v h264_vaapi" test on a Radeon HD
8870M (Cape Verde).

The VCE1 code never enables the VCE trap interrupt and the SI (and
Trinity) interrupt handlers do not process it, unlike the CIK code and
amdgpu.  Patch 1 adds the handlers, patch 2 enables the interrupt.

Results on Debian 6.12.113 (see patch 2 for numbers): hang before,
none after, and 34 fps instead of ~4 fps for the workaround that avoids
the hang.  The KDE RDP server (KRDP) itself, driven by a local FreeRDP
client, now encodes on the VCE ring for several minutes (about 40
fences per second on ring 6 in the radeon_fence_emit tracepoint) with a
clean kernel log.

Two caveats:

 - I have not worked out why the missing interrupt turns into ring
   lockups instead of only added latency.  What I know is that the
   hang needs the VCE ring to wait on a hardware semaphore (disabling
   only the waiting side, or both sides, of radeon_vce_semaphore_emit()
   avoids it) and that handling the interrupt makes it go away with
   the semaphores left enabled.  If someone knows why, I would be glad
   to improve the commit message.
 - The Trinity/Richland change is compile tested only.

The series was developed and tested on Debian's 6.12.113 with the
radeon module built out of tree.  It applies cleanly on drm-next
(fad06cf), where drivers/gpu/drm/radeon builds without warnings (W=1,
allmodconfig) both after patch 1 and with the whole series.

Matheus Tagliari (2):
  drm/radeon: process the VCE interrupt on SI and Trinity
  drm/radeon/vce1: enable the VCE trap interrupt

 drivers/gpu/drm/radeon/evergreen.c | 14 ++++++++++++++
 drivers/gpu/drm/radeon/si.c        | 14 ++++++++++++++
 drivers/gpu/drm/radeon/sid.h       |  2 ++
 drivers/gpu/drm/radeon/vce_v1_0.c  |  4 ++++
 4 files changed, 34 insertions(+)


base-commit: fad06cfaf385787b9ffa2f73f049d42dbe1de57f
-- 
2.47.3

[PATCH 1/2] drm/radeon: process the VCE interrupt on SI and Trinity

[Matheus Tagliari]

radeon_vce_fence_emit() ends every VCE fence with VCE_CMD_TRAP, which
raises the VCE trap interrupt (IH src_id 167).  cik_irq_process()
handles it by calling radeon_fence_process() for the matching VCE ring,
but si_irq_process() and evergreen_irq_process(), which serve the other
ASICs with VCE1 (SI and Trinity/Richland), have no case for it.

Add the same handling.  The trap interrupt is not enabled on these
ASICs yet, so this does not change behaviour on its own; the next patch
enables it.

The SI part was tested together with the next patch.  The Trinity part
mirrors the CIK and SI code and has not been tested on Trinity/Richland
hardware.

Fixes: a918efab631a ("drm/radeon: add VCE 1.0 support v4")
Cc: stable@vger.kernel.org
Signed-off-by: Matheus Tagliari <mptagliari@gmail.com>
---
 drivers/gpu/drm/radeon/evergreen.c | 14 ++++++++++++++
 drivers/gpu/drm/radeon/si.c        | 14 ++++++++++++++
 2 files changed, 28 insertions(+)

diff --git a/drivers/gpu/drm/radeon/evergreen.c b/drivers/gpu/drm/radeon/evergreen.c
index 3cbc6eedb..eb94916c8 100644
--- a/drivers/gpu/drm/radeon/evergreen.c
+++ b/drivers/gpu/drm/radeon/evergreen.c
@@ -4847,6 +4847,20 @@ int evergreen_irq_process(struct radeon_device *rdev)
 			drm_dbg(ddev, "IH: UVD int: 0x%08x\n", src_data);
 			radeon_fence_process(rdev, R600_RING_TYPE_UVD_INDEX);
 			break;
+		case 167: /* VCE */
+			drm_dbg(ddev, "IH: VCE int: 0x%08x\n", src_data);
+			switch (src_data) {
+			case 0:
+				radeon_fence_process(rdev, TN_RING_TYPE_VCE1_INDEX);
+				break;
+			case 1:
+				radeon_fence_process(rdev, TN_RING_TYPE_VCE2_INDEX);
+				break;
+			default:
+				drm_err(ddev, "Unhandled interrupt: %d %d\n", src_id, src_data);
+				break;
+			}
+			break;
 		case 146:
 		case 147:
 			addr = RREG32(VM_CONTEXT1_PROTECTION_FAULT_ADDR);
diff --git a/drivers/gpu/drm/radeon/si.c b/drivers/gpu/drm/radeon/si.c
index 4681ff286..af3c4026d 100644
--- a/drivers/gpu/drm/radeon/si.c
+++ b/drivers/gpu/drm/radeon/si.c
@@ -6347,6 +6347,20 @@ int si_irq_process(struct radeon_device *rdev)
 			DRM_DEBUG("IH: UVD int: 0x%08x\n", src_data);
 			radeon_fence_process(rdev, R600_RING_TYPE_UVD_INDEX);
 			break;
+		case 167: /* VCE */
+			DRM_DEBUG("IH: VCE int: 0x%08x\n", src_data);
+			switch (src_data) {
+			case 0:
+				radeon_fence_process(rdev, TN_RING_TYPE_VCE1_INDEX);
+				break;
+			case 1:
+				radeon_fence_process(rdev, TN_RING_TYPE_VCE2_INDEX);
+				break;
+			default:
+				DRM_ERROR("Unhandled interrupt: %d %d\n", src_id, src_data);
+				break;
+			}
+			break;
 		case 146:
 		case 147:
 			addr = RREG32(VM_CONTEXT1_PROTECTION_FAULT_ADDR);
-- 
2.47.3

[PATCH 2/2] drm/radeon/vce1: enable the VCE trap interrupt

[Matheus Tagliari]

vce_v2_0 sets VCE_SYS_INT_TRAP_INTERRUPT_EN in VCE_SYS_INT_EN so that
the VCE_CMD_TRAP emitted after each fence raises an interrupt.  The
VCE1 code, used by SI and Trinity/Richland, never does, and amdgpu's
vce_v1_0 does (mmVCE_SYS_INT_EN, 0x8340, i.e. byte offset 0x20d00 on
SI).  VCE fences are therefore never processed from the interrupt
handler and are only noticed when a fence wait times out.

Enable the trap interrupt at the end of vce_v1_0_start().

Observed on a Radeon HD 8870M (Cape Verde, 1002:6821) with VCE firmware
50.0.1, Mesa 25.0.7 radeonsi and ffmpeg h264_vaapi on Debian 6.12.113:

 - Stock driver: the encode hangs after a few dozen frames ("ring 0
   stalled", "ring 3 stalled", "GPU lockup"), and the reset does not
   recover ("failed VCE resume (-22)", "ring 0 test failed").
 - With the hardware ring semaphores of the VCE ring disabled as a
   workaround, the hang is gone but encoding runs at 3.9 fps
   (~256 ms per frame), consistent with fences being noticed only by
   timeouts.
 - With this series: 640x360 for 1800 frames, 1280x720 for 900 frames
   and 1920x1080 for 600 frames, all in real time with no decode
   errors, and 191 fps at 720p / 91 fps at 1080p unthrottled.  Four
   runtime suspend/resume cycles with an encode in between also
   worked.  No lockup, reset, WARNING or error in the kernel log, with
   DPM and runtime PM enabled.

Trinity/Richland shares this code and is covered by the previous
patch, but I have no such hardware to test.

Fixes: a918efab631a ("drm/radeon: add VCE 1.0 support v4")
Cc: stable@vger.kernel.org
Signed-off-by: Matheus Tagliari <mptagliari@gmail.com>
---
 drivers/gpu/drm/radeon/sid.h      | 2 ++
 drivers/gpu/drm/radeon/vce_v1_0.c | 4 ++++
 2 files changed, 6 insertions(+)

diff --git a/drivers/gpu/drm/radeon/sid.h b/drivers/gpu/drm/radeon/sid.h
index f9267b026..ba322de84 100644
--- a/drivers/gpu/drm/radeon/sid.h
+++ b/drivers/gpu/drm/radeon/sid.h
@@ -1878,6 +1878,8 @@
 #define	DMA_PACKET_NOP					  0xf
 
 #define VCE_STATUS					0x20004
+#define VCE_SYS_INT_EN					0x20d00
+#define		VCE_SYS_INT_TRAP_INTERRUPT_EN		(1 << 3)
 #define VCE_VCPU_CNTL					0x20014
 #define		VCE_CLK_EN				(1 << 0)
 #define VCE_VCPU_CACHE_OFFSET0				0x20024
diff --git a/drivers/gpu/drm/radeon/vce_v1_0.c b/drivers/gpu/drm/radeon/vce_v1_0.c
index bdfbcf14b..0866e7fb0 100644
--- a/drivers/gpu/drm/radeon/vce_v1_0.c
+++ b/drivers/gpu/drm/radeon/vce_v1_0.c
@@ -350,6 +350,10 @@ int vce_v1_0_start(struct radeon_device *rdev)
 		return r;
 	}
 
+	/* enable the trap interrupt so that VCE fences are processed */
+	WREG32_P(VCE_SYS_INT_EN, VCE_SYS_INT_TRAP_INTERRUPT_EN,
+		 ~VCE_SYS_INT_TRAP_INTERRUPT_EN);
+
 	return 0;
 }
 
-- 
2.47.3

Re: [PATCH 0/2] drm/radeon: fix VCE1 encoding on SI by handling the VCE interrupt

[Christian König]

[Adding Timur on CC]

On 10/21/26 13:41, Matheus Tagliari wrote:
> Hardware H.264 encoding with the VCE block on SI GPUs (radeon driver)
> hangs the GPU after a few dozen frames.  I hit it with the KDE Plasma
> RDP server, which encodes through VAAPI, and reproduced it with a
> plain "ffmpeg -vaapi_device ... -c:v h264_vaapi" test on a Radeon HD
> 8870M (Cape Verde).
> 
> The VCE1 code never enables the VCE trap interrupt and the SI (and
> Trinity) interrupt handlers do not process it, unlike the CIK code and
> amdgpu.  Patch 1 adds the handlers, patch 2 enables the interrupt.

Oh, I do remember that I intentionally never enabled interrupts for VCE on radeon. But I honestly absolutely doesn't remember why (that was ~12-13 years ago!).

Timur hammered out most (all?) of the UVD/VCE bugs on SI, CIK and partially Polaris hardware as well for amdgpu and we are strongly considering just dropping SI/CIK support for radeon.

The fence handling on radeon should work completely without interrupts since those have been rather flaky on really old HW.

So no idea why that actually crashes for you if the interrupt is not enabled, my educated guess is that enabling interrupts actually doesn't fix the crash but makes it less likely because of timing change.

Would it be possible to just switch over to amdgpu instead? Or is there any support still missing there?

Regards,
Christian.

> 
> Results on Debian 6.12.113 (see patch 2 for numbers): hang before,
> none after, and 34 fps instead of ~4 fps for the workaround that avoids
> the hang.  The KDE RDP server (KRDP) itself, driven by a local FreeRDP
> client, now encodes on the VCE ring for several minutes (about 40
> fences per second on ring 6 in the radeon_fence_emit tracepoint) with a
> clean kernel log.
> 
> Two caveats:
> 
>  - I have not worked out why the missing interrupt turns into ring
>    lockups instead of only added latency.  What I know is that the
>    hang needs the VCE ring to wait on a hardware semaphore (disabling
>    only the waiting side, or both sides, of radeon_vce_semaphore_emit()
>    avoids it) and that handling the interrupt makes it go away with
>    the semaphores left enabled.  If someone knows why, I would be glad
>    to improve the commit message.
>  - The Trinity/Richland change is compile tested only.
> 
> The series was developed and tested on Debian's 6.12.113 with the
> radeon module built out of tree.  It applies cleanly on drm-next
> (fad06cf), where drivers/gpu/drm/radeon builds without warnings (W=1,
> allmodconfig) both after patch 1 and with the whole series.
> 
> Matheus Tagliari (2):
>   drm/radeon: process the VCE interrupt on SI and Trinity
>   drm/radeon/vce1: enable the VCE trap interrupt
> 
>  drivers/gpu/drm/radeon/evergreen.c | 14 ++++++++++++++
>  drivers/gpu/drm/radeon/si.c        | 14 ++++++++++++++
>  drivers/gpu/drm/radeon/sid.h       |  2 ++
>  drivers/gpu/drm/radeon/vce_v1_0.c  |  4 ++++
>  4 files changed, 34 insertions(+)
> 
> 
> base-commit: fad06cfaf385787b9ffa2f73f049d42dbe1de57f
> --
> 2.47.3
> 

Re: [PATCH 0/2] drm/radeon: fix VCE1 encoding on SI by handling the VCE interrupt

[Alex Deucher]

On Wed, Oct 21, 2026 at 10:10 AM Christian König
<christian.koenig@amd.com> wrote:
>
> [Adding Timur on CC]
>
> On 10/21/26 13:41, Matheus Tagliari wrote:
> > Hardware H.264 encoding with the VCE block on SI GPUs (radeon driver)
> > hangs the GPU after a few dozen frames.  I hit it with the KDE Plasma
> > RDP server, which encodes through VAAPI, and reproduced it with a
> > plain "ffmpeg -vaapi_device ... -c:v h264_vaapi" test on a Radeon HD
> > 8870M (Cape Verde).
> >
> > The VCE1 code never enables the VCE trap interrupt and the SI (and
> > Trinity) interrupt handlers do not process it, unlike the CIK code and
> > amdgpu.  Patch 1 adds the handlers, patch 2 enables the interrupt.
>
> Oh, I do remember that I intentionally never enabled interrupts for VCE on radeon. But I honestly absolutely doesn't remember why (that was ~12-13 years ago!).
>
> Timur hammered out most (all?) of the UVD/VCE bugs on SI, CIK and partially Polaris hardware as well for amdgpu and we are strongly considering just dropping SI/CIK support for radeon.
>
> The fence handling on radeon should work completely without interrupts since those have been rather flaky on really old HW.
>
> So no idea why that actually crashes for you if the interrupt is not enabled, my educated guess is that enabling interrupts actually doesn't fix the crash but makes it less likely because of timing change.
>
> Would it be possible to just switch over to amdgpu instead? Or is there any support still missing there?

VCE1 was also on some older APUs that are radeon only.

Alex

>
> Regards,
> Christian.
>
> >
> > Results on Debian 6.12.113 (see patch 2 for numbers): hang before,
> > none after, and 34 fps instead of ~4 fps for the workaround that avoids
> > the hang.  The KDE RDP server (KRDP) itself, driven by a local FreeRDP
> > client, now encodes on the VCE ring for several minutes (about 40
> > fences per second on ring 6 in the radeon_fence_emit tracepoint) with a
> > clean kernel log.
> >
> > Two caveats:
> >
> >  - I have not worked out why the missing interrupt turns into ring
> >    lockups instead of only added latency.  What I know is that the
> >    hang needs the VCE ring to wait on a hardware semaphore (disabling
> >    only the waiting side, or both sides, of radeon_vce_semaphore_emit()
> >    avoids it) and that handling the interrupt makes it go away with
> >    the semaphores left enabled.  If someone knows why, I would be glad
> >    to improve the commit message.
> >  - The Trinity/Richland change is compile tested only.
> >
> > The series was developed and tested on Debian's 6.12.113 with the
> > radeon module built out of tree.  It applies cleanly on drm-next
> > (fad06cf), where drivers/gpu/drm/radeon builds without warnings (W=1,
> > allmodconfig) both after patch 1 and with the whole series.
> >
> > Matheus Tagliari (2):
> >   drm/radeon: process the VCE interrupt on SI and Trinity
> >   drm/radeon/vce1: enable the VCE trap interrupt
> >
> >  drivers/gpu/drm/radeon/evergreen.c | 14 ++++++++++++++
> >  drivers/gpu/drm/radeon/si.c        | 14 ++++++++++++++
> >  drivers/gpu/drm/radeon/sid.h       |  2 ++
> >  drivers/gpu/drm/radeon/vce_v1_0.c  |  4 ++++
> >  4 files changed, 34 insertions(+)
> >
> >
> > base-commit: fad06cfaf385787b9ffa2f73f049d42dbe1de57f
> > --
> > 2.47.3
> >
>

Re: [PATCH 0/2] drm/radeon: fix VCE1 encoding on SI by handling the VCE interrupt

[Matheus Tagliari]

On Wed, Oct 21, 2026 at 04:10:04PM +0200, Christian König wrote:
> The fence handling on radeon should work completely without interrupts
> since those have been rather flaky on really old HW.
>
> So no idea why that actually crashes for you if the interrupt is not
> enabled, my educated guess is that enabling interrupts actually doesn't
> fix the crash but makes it less likely because of timing change.
>
> Would it be possible to just switch over to amdgpu instead? Or is there
> any support still missing there?

Thanks for taking a look.

On the interrupt: I agree I have not shown why the hang happens, and a
timing change is entirely possible.  What I can say is limited to this:

 - stock radeon: every encode hung within about 100 frames (many runs);
 - with the VCE ring semaphores disabled: no hang, but fences are only
   noticed by the wait timeout, about 4 fps (~256 ms per frame);
 - with the interrupt enabled and handled, semaphores left on: no hang
   in roughly ten minutes of total encode time (up to 1080p, plus
   several minutes of KDE's RDP server).

That is correlation, not a root cause.  If it helps I can test enabling
VCE_SYS_INT_EN without processing the fences from the handler, and run a
long soak test with two concurrent encodes.

One difference that may be relevant: on CIK radeon does enable and
process the VCE interrupt (vce_v2_0_resume() sets VCE_SYS_INT_EN and
cik_irq_process() handles src_id 167).  Only VCE1 (SI and Trinity)
never did.

On amdgpu: it works for me, I found nothing missing for this use.  I
tested it on 7.2.9 and on Debian's 6.12.113 with the amdgpu VCE1 series
backported: h264_vaapi encodes up to 1080p with DPM and runtime PM
enabled, no hangs.

The reason I looked at radeon at all is the distribution situation.
Debian 13 stable ships 6.12, where SI is driven by radeon by default
and amdgpu has no VCE1 support (that needs a newer kernel, which Debian
only offers through backports).  So anyone on the stable kernel who
uses VAAPI encoding on an SI card, here through KDE's RDP server, ends
up with a wedged GPU.

Since you are considering dropping SI/CIK from radeon, I do not want to
push a change you don't want.  If radeon keeps SI for the 6.x kernels,
what would you prefer?

 a) this series;
 b) disable the VCE semaphores like uvd_v1_0 did in 013ead48a843 (works
    here, but slow);
 c) leave radeon alone and just document VCE encoding on SI with radeon
    as a known issue that amdgpu's VCE1 support in newer kernels
    supersedes (even a one-line warning pointing to amdgpu.si_support=1
    would help people on 6.x).

I am fine with any of them, and if it is (c) please feel free to ignore
the series.  A v2 of patch 2 that fixes the CIK register issue found by
the automated review is ready in case you want (a).

Regards,
Matheus