[REGRESSION 7.3] drm/amd/display: single-frame scanout corruption on DCN32 at 120 Hz since a637a140ab86

[REGRESSION 7.3] drm/amd/display: single-frame scanout corruption on DCN32 at 120 Hz since a637a140ab86

[Mikhail Gavrilov]

Hi,

Since the 7.3 merge window my desktop shows a band of pixel noise flashing
across the screen at random on a 3840x2160@120 display driven by an
RX 7900 XTX (Navi 31, DCN32).  Each occurrence lasts exactly one frame -
8.3 ms - which is why it reads as a flicker rather than as corruption.  It
is frequent enough to be painful to look at during ordinary work.

The band is not new.  It existed before, rarely enough to ignore, and back
then a modeset (switching to 60 Hz and back) cleared it for a while.  What
changed this merge window is that it became roughly two orders of magnitude
more frequent, and the modeset workaround stopped helping.

What I would like is help finding the underlying bug.  I am not asking for a
revert: reverting the commit below does not fix the artefact, it only takes
the rate back down to what it was before.  I could not work out how the
commit leads to the symptom, so most of this mail is measurements and a list
of mechanisms I was able to rule out.

A photo of the screen, taken with a phone because the artefact never appears
in a screenshot:

  https://gitlab.freedesktop.org/drm/amd/-/work_items/5714


Measuring it
------------

The artefact is invisible to screen capture but visible to the CRTC CRC
engine.  A frame in which it occurs has a CRC triple that differs from its
neighbours while the neighbours are byte-identical to each other: displayed
content did not change, but one scanned-out frame did.  A torn readback
cannot produce that, because inside such a window the adjacent frames are
identical and there is nothing to mix.

The colour band moves all three CRC registers.  Single-frame content
flicker in grey areas moves only CRC1 (Y), because on this YCbCr link the
chroma of an achromatic region is unchanged.  That gives a built-in control:
the CRC1-only population is content, not corruption, and its rate should not
depend on the kernel.

All figures below are per *eligible* frame - frames where the quiet window
existed - so runs against differently busy screens stay comparable.


The regression
--------------

I bisected the change in frequency to

  commit a637a140ab86 ("drm/amd/display: Allocate DCN32/1 dmub memory to GTT")

Each bisect step was judged by eye over a kernel build's worth of ordinary
desktop use at 120 Hz.  That is a weak criterion for a probabilistic bug, so
I re-tested the endpoints with the CRC detector instead.  Both runs were
ordinary desktop work, same machine, same display, no modeset before either
run:

                              a637a140ab86      a637a140ab86
                                reverted           present
  observation                    8.06 h            6.43 h
  eligible frames              1,621,663         1,382,934
  events, all three CRCs              12               924
    per 1000 eligible               0.0074            0.6681
  events, CRC1 only (control)         30                32
    per 1000 eligible               0.0185            0.0231

The artefact signature is 90x more frequent.  The content-flicker control is
flat at 1.25x, so the detector, the screen activity and the sensitivity were
comparable across the two runs.

The composition changes too: with the commit reverted, 24% of detected
events are all-three; with it present, 94% are.


Refresh rate
------------

  4K120, YCbCr 4:2:0, 8 bpc, TMDS 594 MHz   artefact present
  4K100, YCbCr 4:2:0, 8 bpc, TMDS 495 MHz   artefact present
  4K60,  YCbCr 4:4:4, 8 bpc, TMDS 594 MHz   0 events in 70,691 eligible
  4K60,  YCbCr 4:2:0, 12 bpc (forced)       0 events in 1,290,588 eligible

The last row was measured with force_yuv_pixel_format=4 to separate refresh
rate from chroma subsampling, since the two change together on this display.
At 60 Hz with the same 4:2:0 encoding that shows the artefact at 120 Hz,
1.29 million eligible frames produced nothing.  Link bandwidth is not the
variable either: 594 MHz appears in a configuration that shows the artefact
and in one that does not.

So the dependency is on frame rate.


What I could rule out
---------------------

Twenty DTN snapshots taken within seconds of confirmed events, on the kernel
with the commit reverted:

  - HUBP underflow and OTG underflow read 0 in all twenty.
  - dcfclk, dispclk, dppclk and fclk are identical in all twenty; no clock
    transition happens near an event.
  - vmax = vmin = 2249 with both select bits set, in all twenty: DRR is armed
    but pinned to nominal, so no vblank stretching is available.
  - The watermark sets, including dram_clk_change, are identical throughout.

Caveat: the snapshots are taken seconds after the event, so a momentary
transition would be missed.  But vmax = vmin and the identical watermark
sets are configuration, not instantaneous state.

Reading the commit, the buffer is pinned by amdgpu_bo_create_kernel(), so
TTM eviction is not a factor either.  What the commit moves out of VRAM is
every DMUB FB window at once - instruction constants, stack, BSS data,
VBIOS, mailbox, trace buffer, firmware state, scratch, IB, shared state,
LSDMA buffer and cursor offload.  I do not know how that leads to a single
corrupted scanout frame; that is the part I am hoping someone from DC can
see.

Two details from the diff that I am not able to judge myself.

AMDGPU_GEM_CREATE_VRAM_CONTIGUOUS is still set unconditionally in
amdgpu_bo_create_reserved(), and with the domain now GTT-only it no longer
does anything.

The commit immediately before this one in the same file is 5b69770e945c
("drm/amd/display: Generalize DMUB memory flushes from host"), which adds a
dmub_srv context parameter to the flush helper; the Linux implementation
still ignores it and remains the generic read-back loop. That helper is
called on scratch_mem_fb, on ib_mem_gart and on inst_fb during backdoor
load. Before a637a140ab86 the remaining windows sat in VRAM; now all twelve
are in system memory. Is that flush coverage still sufficient for the
windows that moved? I am asking, not claiming - I do not know the coherency
model for these buffers well enough to say.

The commit's own comment says the restriction to DCN32/321 is temporary and
that it will be made general to other dGPUs later, which is why I would
rather this were understood before it spreads.


Configuration
-------------

  GPU        : AMD Radeon RX 7900 XTX (Navi 31, DCN32), 1002:744c rev c8
  iGPU       : Raphael 1002:164e, bound to amdgpu, drives no display
  Board      : ASUS ROG STRIX B650E-I GAMING WIFI, BIOS 3854 04/03/2026
  CPU        : AMD Ryzen 9 7950X
  Display    : LG OLED42C3 (42" OLED TV), HDMI, 3840x2160@120
  Timing     : 3840 4016 4104 4400  2160 2168 2178 2250, 1188000 kHz
  Encoding   : YCbCr 4:2:0 8 bpc at 120/100 Hz, YCbCr 4:4:4 8 bpc at 60 Hz
  FreeSync   : enabled, range 40-120, reported FIXED by the display
  DSC        : off
  Kernel     : 7.2.0-66498c75b4f8-with-fixes-v1-m62-quirk+
  Userspace  : Fedora Rawhide, GNOME on Wayland

The kernel is tainted W: opening the CRTC CRC debugfs file triggers a
WARN in dc_helper.c on DCN32.  That is a separate bug in the same merge
window, with a patch already posted:

  https://lore.kernel.org/all/20260829004608.25252-1-mikhail.v.gavrilov@gmail.com/

It does not affect CRC0, which is what the measurements above use.

Both the capture tool and the raw logs are available if they would help.

Thanks,
Mikhail Gavrilov

#regzbot introduced: a637a140ab86
#regzbot title: drm/amd/display: single-frame scanout corruption on DCN32 at 120 Hz
#regzbot link: https://gitlab.freedesktop.org/drm/amd/-/work_items/5714

Re: [REGRESSION 7.3] drm/amd/display: single-frame scanout corruption on DCN32 at 120 Hz since a637a140ab86

[Mikhail Gavrilov]

I have to correct my own report.

The 90x figure I gave for a637a140ab86 does not hold up.  Since sending it
I have measured five more sessions, and the rate turns out to be controlled
by something other than the kernel.  Please do not spend time on the commit
on the strength of that number.

What the artefact actually does is sit in one of two states, and the state
changes during a session.  All figures are events per 1000 eligible frames,
where an eligible frame is one the detector could have judged - the method
is unchanged from my first mail.

  60 Hz                       0.0000   0.0045   0.0252
  120 Hz, quiet state         0.0308   0.0372   0.0400
  120 Hz, excited state       0.5208   0.5739   0.7115   1.3027   1.9153

The three groups are about a decade apart from each other, and the kernel
does not sort them.  0.0308 is the run on the kernel with a637a140ab86
reverted; 0.0372 and 0.0400 are runs on unreverted kernels in the quiet
state.  Those three are the same measurement.  My original pair simply
caught one kernel quiet and the other excited.

Worse, every kernel comparison I made began with a reboot, and a reboot
resets the state.  So the bisect was measuring how soon after boot I
happened to load the machine.  I no longer consider the attribution to
a637a140ab86 supported, and I cannot say from my data whether this is a
7.3 regression at all.


The two transitions
-------------------

I can now move the artefact in both directions, but only as bundles of
variables - I have not separated what inside each bundle does the work.

Excited by: a long session of a heavy 3D game (Dying Light: The Beast).
Measured twice on the same kernel and refresh rate: 0.0372 -> 1.3027 (35x)
and 0.0400 -> 0.5208 (13x).  The effect outlasts the game by hours and
decays gradually.  That bundle contains sustained shader load, large VRAM
allocation and eviction, GTT traffic, clock and power-state churn, heating,
and fullscreen presentation.

Calmed by: about three hours spent at 60 Hz, after which returning to
120 Hz gave 1.9153 -> 0.0400 (48x), on an unreverted kernel.  That bundle
contains two modesets, three hours of elapsed time, half the pixel clock,
a different pixel encoding (this display uses YCbCr 4:2:0 at 120 Hz and
4:4:4 at 60 Hz), different watermarks, and an idle GPU.

Also calmed by: a reboot.

Which single variable in each bundle matters, I do not know.  A short
120 -> 60 -> 120 toggle would separate "a modeset" from "time at 60 Hz",
and a compute-only load would separate "GPU busy" from "memory pressure".
I am happy to run whichever of those would be most useful.


What did survive
----------------

The dependency on frame rate is unchanged and now rests on four
independent sessions: 120 Hz and 100 Hz show the artefact, 60 Hz is at
least an order of magnitude below the quiet 120 Hz state.  Forcing YCbCr
4:2:0 at 60 Hz, so that the encoding matches 120 Hz, gave 0 events in
1,290,588 eligible frames, so this is frame rate and not chroma
subsampling or link bandwidth.

Swapping the HDMI cable made no difference.

And one negative result that may be the most useful thing in this mail:
DTN snapshots taken seconds after real events in the excited state are
byte-identical to snapshots taken in the quiet state - 195 lines against
195, differing only in the scanout buffer address.  Watermarks, dcfclk,
dispclk, dppclk, fclk, HUBP and OTG underflow, vtotal and the DRR select
bits, DSC, RQ, DLG and TTU are all the same in both states, at a fifty-fold
difference in artefact rate.

So whatever holds this state is not in anything DTN exposes.  Is there
something in DMCUB, in the memory path, or in link state that could persist
across a game, decay over hours, be cleared by a reboot, and leave every
register in the DTN dump identical?  That is the question I am stuck on.

Thorsten: the regzbot entry points at a637a140ab86 on the strength of the
bisect above, which I no longer trust.  I do not know yet whether there is
a regression here at all, so I would rather not close it as invalid; please
handle it however is appropriate for an entry whose attribution has fallen
apart.

The GitLab issue has been updated with the same correction:

  https://gitlab.freedesktop.org/drm/amd/-/work_items/5714

Sorry for the noise on the first pass.

-- 
Thanks,
Mikhail Gavrilov

Re: [REGRESSION 7.3] drm/amd/display: single-frame scanout corruption on DCN32 at 120 Hz since a637a140ab86

[Mikhail Gavrilov]

Following up on my own correction: the attribution to a637a140ab86 is back,
and this time on a comparison that does not have the flaw the first one had.

In my previous mail I withdrew the 90x figure because the rate turned out to
depend on the machine's recent load and my two runs had not controlled for
it.  I have now measured the commit against its immediate parent, in matched
conditions, and the difference is there without that confound.

  parent  1d8327353de6 ("drm/amd/display: Fix VABC Init and OLED Support")
  commit  a637a140ab86 ("drm/amd/display: Allocate DCN32/1 dmub memory to GTT")

Same machine, same display, same 3840x2160@120 mode, same .config, ordinary
desktop use, no game in either run, about eleven hours each:

                          parent            commit
  observation             11.0 h            11.6 h
  eligible frames       3,019,571         3,632,444
  events                        7               293
  per 1000 eligible        0.0023            0.0807
  95% interval      0.0006-0.0040     0.0714-0.0899

That is 35x, with the intervals more than a decade apart.

"Events" counts only frames where all three CRC registers moved, which is
what the artefact does.  The single-frame content flicker that moves only
the luma CRC serves as a built-in control, and it did not move with the
kernel: 0.0020 per 1000 on the parent against 0.0008 on the commit.


The load confound does not reach this
-------------------------------------

Both runs began immediately after a reboot, which resets the state, and
neither had any load yet in its first hour:

  parent : 0 events in 434,204 eligible frames
  commit : 52 events in 349,718 eligible frames  (0.149 per 1000)

So the difference is present before anything could have excited it.

The shape differs too, not only the rate.  On the parent the worst
five-minute bucket in eleven hours contained 2 events, and no bucket
reached 3.  On the commit the worst contained 30, and 28 of 138 buckets
reached 3 or more: the excited state comes and goes by itself there, and
on the parent it never appeared at all.

For completeness: the commit run does contain three windows where almost no
frame was eligible - roughly 14:30-15:25, 21:10-21:25 and 23:45-23:55 -
which is what fullscreen content looks like in this data, and the parent run
has no such window.  The first-hour figures above are free of that, and the
rate is normalised per eligible frame throughout, but I would rather point
it out than have it found.


What this does and does not settle
----------------------------------

It restores a637a140ab86 as the point where this starts on DCN3.2.  What I
said in the correction about the state machine still holds and is
independent of it: heavy GPU load excites the artefact, a long spell at
60 Hz or a reboot calms it, and DTN snapshots in the excited state are
byte-identical to quiet-state ones.  Those were measured on kernels that
already contain the commit; the parent simply never leaves the quiet state.

I still have no mechanism.  The commit moves every DMUB FB window out of
VRAM into GTT, and I cannot see how that produces one corrupted scanout
frame while leaving every register in the DTN dump unchanged.

The GitLab issue has been updated with the same numbers:

  https://gitlab.freedesktop.org/drm/amd/-/work_items/5714

#regzbot introduced: a637a140ab86

Thanks,
Mikhail Gavrilov

Re: [REGRESSION 7.3] drm/amd/display: single-frame scanout corruption on DCN32 at 120 Hz since a637a140ab86

[Mikhail Gavrilov]

The band appears while DC switches the memory clock through
sub-viewport (SubVP).  When SubVP is not in use, a kernel that contains
a637a140ab86 shows the same low rate as that commit's parent.  On my
display, whether SubVP is used at 120 Hz is decided by a TV setting,
which also gives affected users a workaround.

Two runs on the same kernel (7.3-rc6), the same TV setting and ordinary
desktop use, differing only in the refresh rate.  SubVP state and the
memory clock were read once a minute from amdgpu_dm_capabilities and
pp_dpm_mclk.  SubVP never changed state within a run; the memory clock
stayed at 1249 MHz for the whole 120 Hz run and moved through all four
levels at 100 Hz:

  mode     SubVP  memory clock    events   eligible  per 1000 (95%)
  4K@120   no     1249 MHz            5   1,582,072  0.0032 (0.0010-0.0074)
  4K@100   yes    96..1249 MHz      139     712,333  0.195  (0.164-0.230)

138 of the 139 events at 100 Hz carry CRC triples that occur nowhere
else in the capture; the remaining one recurs seven times.  All twenty
DTN snapshots taken after events show watermark set A at the register
maximum and a calculated DCFCLK of 400 MHz: the SubVP override in
dcn20_optimize_bandwidth() and MIN_SUBVP_DCFCLK_KHZ.

With SubVP in use the commit matters.  The parent/commit pair I
reported earlier ran at 120 Hz in HDMI VRR mode, where DC uses SubVP
(the revert kernel's DTN snapshots carry the same two values), and
measured 0.0023 against 0.0807 per 1000.  Without SubVP, the kernel
with the commit is at 0.0032.


Why a TV setting decides it
---------------------------

The LG OLED42C3 offers "VRR & G-Sync" (HDMI Forum VRR) and "AMD FreeSync
Premium".  With the first, the EDID carries the HF-VSDB VRR range 40-120
Hz and no AMD VSDB; with the second, the HF-VSDB VRR range is zero and
an AMD VSDB v2 with 40-120 Hz appears.  amdgpu_dm_update_freesync_caps()
takes FreeSync on HDMI only from the AMD VSDB, so:

  - HDMI VRR mode: the stream has allow_freesync = false, and
    dcn32_allow_subvp_high_refresh_rate() permits SubVP for a single
    display at 120 Hz.  amdgpu_dm_capabilities reports sub-viewport
    enabled at 120 and 100 Hz; visual confirm 16 draws red at 120 Hz.

  - FreeSync mode: SubVP is not permitted at 120 Hz.  DC finds no
    p-state method (watermark set A p-state 0, no visual confirm colour)
    and keeps UCLK at its maximum, which pp_dpm_mclk showed for six
    hours.  At 100 Hz, below the 120 Hz threshold in
    dcn32_assign_subvp_pipe(), SubVP is used in this mode too.

  - 60 Hz, either mode: the switch happens in vactive without DMUB
    (visual confirm 16: magenta).  My 60 Hz runs measured between 0
    and 0.025 per 1000, one of them 0 in 1,290,588 eligible frames.

All my earlier measurements were taken in HDMI VRR mode.  The line
"FreeSync: enabled, range 40-120" in my original report was wrong: in
that mode amdgpu does not enable FreeSync.

For affected users with such a TV, selecting FreeSync instead of HDMI
VRR avoids SubVP at 120 Hz, at the cost of the memory clock staying at
its maximum.  I have not tried amdgpu.dcdebugmask=0x20000
(DC_DISABLE_SUBVP_FAMS) in HDMI VRR mode; I can if that would help.


The question
------------

SubVP is carried out by the DMUB firmware, and DC budgets the
firmware's reaction time when it sizes the phantom pipe's MALL region:
subvp_fw_processing_delay_us = 15 and subvp_pstate_allow_width_us = 20
in dcn32_resource.c, used in dcn32_fpu.c.  a637a140ab86 moves every DMUB
window -- instruction constants, stack, BSS data, mailbox, firmware
state and the rest -- from VRAM to GTT.  Is the firmware's SubVP timing
sensitive to where its memory lives?  Asking, not claiming.

DMCUB firmware here is 0x07003300 (7.0.51).


Also
----

A second user reports the same on an RX 7900 XT with two TVs over HDMI
at 120 Hz and above: from 7.3 on, not on a DisplayPort monitor, and gone
when he locks the GPU clocks.  If the lock includes the memory clock, it
stops memory clock switching, which fits the above; I have asked which
VRR mode his TVs use:

  https://gitlab.freedesktop.org/drm/amd/-/work_items/5714#note_3692576

Two more corrections to my original report.  The DTN "CALCULATED
Clocks" line prints DC's calculated requirements, not the clocks the SMU
is running at, so it says nothing about clock transitions.  And a
snapshot taken seconds after an event cannot exclude what the firmware
does around a switch: with SubVP it lowers watermark set A when the
switch starts, so "the watermark sets are configuration, not
instantaneous state" was wrong as well.

The GitLab issue has the same data:

  https://gitlab.freedesktop.org/drm/amd/-/work_items/5714

-- 
Thanks,
Mikhail

Re: [REGRESSION 7.3] drm/amd/display: single-frame scanout corruption on DCN32 at 120 Hz since a637a140ab86

[Mikhail Gavrilov]

The workaround from my previous mail does not survive a suspend.  After
resume from S3 the band was back (seen, not measured) with the TV still
in FreeSync mode, because DC had turned SubVP back on.

Snapshots of amdgpu_dm_capabilities, pp_dpm_mclk and the DTN log, all
at 3840x2160@120 with FreeSync enabled (vrr_range 40-120, the EDID
byte-identical in every snapshot):

  after boot                    sub-viewport disabled, UCLK at 1249 MHz
  20 s after rtcwake -m mem     sub-viewport enabled, UCLK switching,
                                watermark set A at the register maximum
  after toggling Night Light    sub-viewport disabled again

dcn32_allow_subvp_high_refresh_rate() keeps SubVP off a single FreeSync
display by reading stream->allow_freesync, but amdgpu_dm sets that
field only on the first flip after a commit, so the modeset forced on
resume is validated without it.  A gamma change makes DC validate the
state again, this time with the field set.  The patch:

  https://lore.kernel.org/all/20261018221726.27259-1-mikhail.v.gavrilov@gmail.com/

With it, SubVP stays disabled across rtcwake -m mem.  Until it is
merged, toggling Night Light after resume restores the workaround.

None of this changes the band itself: in HDMI VRR mode, and at 100 Hz
in either mode, DC uses SubVP anyway.

-- 
Thanks,
Mikhail

Re: [REGRESSION 7.3] drm/amd/display: single-frame scanout corruption on DCN32 at 120 Hz since a637a140ab86

[Mikhail Gavrilov]

Calvin Owens bisected the same artefact to a637a140ab86 on his RX 7900,
with the same parent good, and posted a revert.  On his machine the
band appears within 30 minutes, and the revert removes it:

  https://lore.kernel.org/all/25995707c39c0c191e3ac46c4813f55a7f928403.1793131583.git.calvin@wbinvd.org/

#regzbot monitor: https://lore.kernel.org/all/25995707c39c0c191e3ac46c4813f55a7f928403.1793131583.git.calvin@wbinvd.org/

A correction to my earlier mails, which listed the instruction
constants and the stack among the DMUB windows that a637a140ab86 moves
to GTT.  When the firmware is loaded through the PSP, the default on
these cards, the driver neither copies the instruction constants into
its window nor programs the instruction and stack windows
(dm_dmub_hw_init(), dmub_srv_hw_init()), so the firmware does not run
from the driver's copies of those two.  The windows the driver does set
up -- BSS data, VBIOS, mailbox, trace buffer, firmware state, shared
state and the rest -- did move.

-- 
Thanks,
Mikhail