[PATCH v2 0/3] drm/ci: enable lockdep detection

[PATCH v2 0/3] drm/ci: enable lockdep detection

[Vignesh Raman]

This patch series enables lockdep detection in drm-ci. Any lockdep
failures will be shown as warnings in the pipeline. This series
also enables CONFIG_DEBUG_WW_MUTEX_SLOWPATH for mutex slowpath
debugging and refactors software-driver stage jobs.

Test run with this series,
https://gitlab.freedesktop.org/vigneshraman/linux/-/pipelines/1362134

Link to v1:
https://lore.kernel.org/all/20240812112030.81774-1-vignesh.raman@collabora.com/

Vignesh Raman (3):
  drm/ci: refactor software-driver stage jobs
  drm/ci: enable CONFIG_DEBUG_WW_MUTEX_SLOWPATH
  drm/ci: enable lockdep detection

 Documentation/gpu/automated_testing.rst |  4 ++
 drivers/gpu/drm/ci/build.yml            |  6 +-
 drivers/gpu/drm/ci/igt_runner.sh        | 11 ++++
 drivers/gpu/drm/ci/test.yml             | 78 +++++++++++++------------
 4 files changed, 58 insertions(+), 41 deletions(-)

-- 
2.43.0

[PATCH v2 1/3] drm/ci: refactor software-driver stage jobs

[Vignesh Raman]

Move common job configuration for software-driver
stage jobs to separate job.

Signed-off-by: Vignesh Raman <vignesh.raman@collabora.com>
---

v2:
  - New patch in the series.

---
 drivers/gpu/drm/ci/test.yml | 59 +++++++++++++++----------------------
 1 file changed, 24 insertions(+), 35 deletions(-)

diff --git a/drivers/gpu/drm/ci/test.yml b/drivers/gpu/drm/ci/test.yml
index 6a1e059858e5..0eab020a33b9 100644
--- a/drivers/gpu/drm/ci/test.yml
+++ b/drivers/gpu/drm/ci/test.yml
@@ -89,6 +89,26 @@
   tags:
     - $RUNNER_TAG
 
+.software-driver:
+  stage: software-driver
+  timeout: "1h30m"
+  rules:
+    - !reference [.scheduled_pipeline-rules, rules]
+    - when: on_success
+  extends:
+    - .test-gl
+  tags:
+    - kvm
+  script:
+    - ln -sf $CI_PROJECT_DIR/install /install
+    - mv install/bzImage /lava-files/bzImage
+    - mkdir -p /lib/modules
+    - install/crosvm-runner.sh install/igt_runner.sh
+  needs:
+    - debian/x86_64_test-gl
+    - testing:x86_64
+    - igt:x86_64
+
 .msm-sc7180:
   extends:
     - .lava-igt:arm64
@@ -440,47 +460,16 @@ panfrost:g12b:
     - .panfrost-gpu
 
 virtio_gpu:none:
-  stage: software-driver
-  timeout: "1h30m"
-  rules:
-    - !reference [.scheduled_pipeline-rules, rules]
-    - when: on_success
+  extends:
+    - .software-driver
   variables:
     CROSVM_GALLIUM_DRIVER: llvmpipe
     DRIVER_NAME: virtio_gpu
     GPU_VERSION: none
-  extends:
-    - .test-gl
-  tags:
-    - kvm
-  script:
-    - ln -sf $CI_PROJECT_DIR/install /install
-    - mv install/bzImage /lava-files/bzImage
-    - install/crosvm-runner.sh install/igt_runner.sh
-  needs:
-    - debian/x86_64_test-gl
-    - testing:x86_64
-    - igt:x86_64
 
 vkms:none:
-  stage: software-driver
-  timeout: "1h30m"
-  rules:
-    - !reference [.scheduled_pipeline-rules, rules]
-    - when: on_success
+  extends:
+    - .software-driver
   variables:
     DRIVER_NAME: vkms
     GPU_VERSION: none
-  extends:
-    - .test-gl
-  tags:
-    - kvm
-  script:
-    - ln -sf $CI_PROJECT_DIR/install /install
-    - mv install/bzImage /lava-files/bzImage
-    - mkdir -p /lib/modules
-    - ./install/crosvm-runner.sh ./install/igt_runner.sh
-  needs:
-    - debian/x86_64_test-gl
-    - testing:x86_64
-    - igt:x86_64
-- 
2.43.0

[PATCH v2 2/3] drm/ci: enable CONFIG_DEBUG_WW_MUTEX_SLOWPATH

[Vignesh Raman]

Enable CONFIG_DEBUG_WW_MUTEX_SLOWPATH for mutex
slowpath debugging.

Signed-off-by: Vignesh Raman <vignesh.raman@collabora.com>
---

v2:
  - New patch in the series.

---
 drivers/gpu/drm/ci/build.yml | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/drivers/gpu/drm/ci/build.yml b/drivers/gpu/drm/ci/build.yml
index 274f118533a7..6c0dc10b547c 100644
--- a/drivers/gpu/drm/ci/build.yml
+++ b/drivers/gpu/drm/ci/build.yml
@@ -67,7 +67,7 @@ testing:arm32:
     #
     # db410c and db820c don't boot with KASAN_INLINE, probably due to the kernel
     # becoming too big for their bootloaders.
-    ENABLE_KCONFIGS: "PROVE_LOCKING DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT"
+    ENABLE_KCONFIGS: "PROVE_LOCKING DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT DEBUG_WW_MUTEX_SLOWPATH"
     UPLOAD_TO_MINIO: 1
     MERGE_FRAGMENT: arm.config
 
@@ -79,7 +79,7 @@ testing:arm64:
     #
     # db410c and db820c don't boot with KASAN_INLINE, probably due to the kernel
     # becoming too big for their bootloaders.
-    ENABLE_KCONFIGS: "PROVE_LOCKING DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT"
+    ENABLE_KCONFIGS: "PROVE_LOCKING DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT DEBUG_WW_MUTEX_SLOWPATH"
     UPLOAD_TO_MINIO: 1
     MERGE_FRAGMENT: arm64.config
 
@@ -91,7 +91,7 @@ testing:x86_64:
     #
     # db410c and db820c don't boot with KASAN_INLINE, probably due to the kernel
     # becoming too big for their bootloaders.
-    ENABLE_KCONFIGS: "PROVE_LOCKING DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT"
+    ENABLE_KCONFIGS: "PROVE_LOCKING DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT DEBUG_WW_MUTEX_SLOWPATH"
     UPLOAD_TO_MINIO: 1
     MERGE_FRAGMENT: x86_64.config
 
-- 
2.43.0

[PATCH v2 3/3] drm/ci: enable lockdep detection

[Vignesh Raman]

We have enabled PROVE_LOCKING (which enables LOCKDEP) in drm-ci.
This will output warnings when kernel locking errors are encountered
and will continue executing tests. To detect if lockdep has been
triggered, check the debug_locks value in /proc/lockdep_stats after
the tests have run. When debug_locks is 0, it indicates that lockdep
has detected issues and turned itself off. Check this value, and if
lockdep is detected, exit with an error and configure it as a warning
in GitLab CI.

GitLab CI ignores exit codes other than 1 by default. Pass the correct
exit code with variable FF_USE_NEW_BASH_EVAL_STRATEGY set to true or
exit on failure.

Also update the documentation.

Signed-off-by: Vignesh Raman <vignesh.raman@collabora.com>
---

v2:
  - Lockdep failures are reported as pipeline warnings,
    and the documentation is updated.

---
 Documentation/gpu/automated_testing.rst |  4 ++++
 drivers/gpu/drm/ci/igt_runner.sh        | 11 +++++++++++
 drivers/gpu/drm/ci/test.yml             | 19 ++++++++++++++++---
 3 files changed, 31 insertions(+), 3 deletions(-)

diff --git a/Documentation/gpu/automated_testing.rst b/Documentation/gpu/automated_testing.rst
index 6d7c6086034d..62aa3ede02a5 100644
--- a/Documentation/gpu/automated_testing.rst
+++ b/Documentation/gpu/automated_testing.rst
@@ -115,6 +115,10 @@ created (eg. https://gitlab.freedesktop.org/janedoe/linux/-/pipelines)
 5. The various jobs will be run and when the pipeline is finished, all jobs
 should be green unless a regression has been found.
 
+6. Warnings in the pipeline indicate that lockdep
+(see Documentation/locking/lockdep-design.rst) issues have been detected
+during the tests.
+
 
 How to update test expectations
 ===============================
diff --git a/drivers/gpu/drm/ci/igt_runner.sh b/drivers/gpu/drm/ci/igt_runner.sh
index 68b042e43b7f..2a0599f12c58 100755
--- a/drivers/gpu/drm/ci/igt_runner.sh
+++ b/drivers/gpu/drm/ci/igt_runner.sh
@@ -85,5 +85,16 @@ deqp-runner junit \
    --limit 50 \
    --template "See $ARTIFACTS_BASE_URL/results/{{testcase}}.xml"
 
+# Check if /proc/lockdep_stats exists
+if [ -f /proc/lockdep_stats ]; then
+    # If debug_locks is 0, it indicates lockdep is detected and it turns itself off.
+    debug_locks=$(grep 'debug_locks:' /proc/lockdep_stats | awk '{print $2}')
+    if [ "$debug_locks" -eq 0 ] && [ "$ret" -eq 0 ]; then
+        echo "Warning: LOCKDEP issue detected. Please check dmesg logs for more information."
+        cat /proc/lockdep_stats
+        ret=101
+    fi
+fi
+
 cd $oldpath
 exit $ret
diff --git a/drivers/gpu/drm/ci/test.yml b/drivers/gpu/drm/ci/test.yml
index 0eab020a33b9..3af735dbf6bd 100644
--- a/drivers/gpu/drm/ci/test.yml
+++ b/drivers/gpu/drm/ci/test.yml
@@ -1,6 +1,8 @@
 .lava-test:
   extends:
     - .container+build-rules
+  variables:
+    FF_USE_NEW_BASH_EVAL_STRATEGY: 'true'
   timeout: "1h30m"
   rules:
     - !reference [.scheduled_pipeline-rules, rules]
@@ -13,6 +15,9 @@
     - mv -n install/* artifacts/.
     # Override it with our lava-submit.sh script
     - ./artifacts/lava-submit.sh
+  allow_failure:
+    exit_codes:
+      - 101
 
 .lava-igt:arm32:
   extends:
@@ -88,9 +93,14 @@
     - igt:arm64
   tags:
     - $RUNNER_TAG
+  allow_failure:
+    exit_codes:
+      - 101
 
 .software-driver:
   stage: software-driver
+  variables:
+    FF_USE_NEW_BASH_EVAL_STRATEGY: 'true'
   timeout: "1h30m"
   rules:
     - !reference [.scheduled_pipeline-rules, rules]
@@ -108,6 +118,9 @@
     - debian/x86_64_test-gl
     - testing:x86_64
     - igt:x86_64
+  allow_failure:
+    exit_codes:
+      - 101
 
 .msm-sc7180:
   extends:
@@ -153,7 +166,7 @@ msm:apq8016:
     BM_KERNEL_EXTRA_ARGS: clk_ignore_unused
     RUNNER_TAG: google-freedreno-db410c
   script:
-    - ./install/bare-metal/fastboot.sh
+    - ./install/bare-metal/fastboot.sh || exit $?
 
 msm:apq8096:
   extends:
@@ -167,7 +180,7 @@ msm:apq8096:
     GPU_VERSION: apq8096
     RUNNER_TAG: google-freedreno-db820c
   script:
-    - ./install/bare-metal/fastboot.sh
+    - ./install/bare-metal/fastboot.sh || exit $?
 
 msm:sdm845:
   extends:
@@ -181,7 +194,7 @@ msm:sdm845:
     GPU_VERSION: sdm845
     RUNNER_TAG: google-freedreno-cheza
   script:
-    - ./install/bare-metal/cros-servo.sh
+    - ./install/bare-metal/cros-servo.sh || exit $?
 
 msm:sm8350-hdk:
   extends:
-- 
2.43.0

Re: [PATCH v2 3/3] drm/ci: enable lockdep detection

[Helen Mae Koike Fornazier]

Hi Vignesh,

thanks for your patch.

Em ter., 11 de fev. de 2025 às 12:29, Vignesh Raman
<vignesh.raman@collabora.com> escreveu:
>
> We have enabled PROVE_LOCKING (which enables LOCKDEP) in drm-ci.
> This will output warnings when kernel locking errors are encountered
> and will continue executing tests. To detect if lockdep has been
> triggered, check the debug_locks value in /proc/lockdep_stats after
> the tests have run. When debug_locks is 0, it indicates that lockdep
> has detected issues and turned itself off. Check this value, and if
> lockdep is detected, exit with an error and configure it as a warning
> in GitLab CI.
>
> GitLab CI ignores exit codes other than 1 by default. Pass the correct
> exit code with variable FF_USE_NEW_BASH_EVAL_STRATEGY set to true or
> exit on failure.
>
> Also update the documentation.
>
> Signed-off-by: Vignesh Raman <vignesh.raman@collabora.com>
> ---
>
> v2:
>   - Lockdep failures are reported as pipeline warnings,
>     and the documentation is updated.
>
> ---
>  Documentation/gpu/automated_testing.rst |  4 ++++
>  drivers/gpu/drm/ci/igt_runner.sh        | 11 +++++++++++
>  drivers/gpu/drm/ci/test.yml             | 19 ++++++++++++++++---
>  3 files changed, 31 insertions(+), 3 deletions(-)
>
> diff --git a/Documentation/gpu/automated_testing.rst b/Documentation/gpu/automated_testing.rst
> index 6d7c6086034d..62aa3ede02a5 100644
> --- a/Documentation/gpu/automated_testing.rst
> +++ b/Documentation/gpu/automated_testing.rst
> @@ -115,6 +115,10 @@ created (eg. https://gitlab.freedesktop.org/janedoe/linux/-/pipelines)
>  5. The various jobs will be run and when the pipeline is finished, all jobs
>  should be green unless a regression has been found.
>
> +6. Warnings in the pipeline indicate that lockdep
> +(see Documentation/locking/lockdep-design.rst) issues have been detected
> +during the tests.
> +
>
>  How to update test expectations
>  ===============================
> diff --git a/drivers/gpu/drm/ci/igt_runner.sh b/drivers/gpu/drm/ci/igt_runner.sh
> index 68b042e43b7f..2a0599f12c58 100755
> --- a/drivers/gpu/drm/ci/igt_runner.sh
> +++ b/drivers/gpu/drm/ci/igt_runner.sh
> @@ -85,5 +85,16 @@ deqp-runner junit \
>     --limit 50 \
>     --template "See $ARTIFACTS_BASE_URL/results/{{testcase}}.xml"
>
> +# Check if /proc/lockdep_stats exists
> +if [ -f /proc/lockdep_stats ]; then
> +    # If debug_locks is 0, it indicates lockdep is detected and it turns itself off.
> +    debug_locks=$(grep 'debug_locks:' /proc/lockdep_stats | awk '{print $2}')
> +    if [ "$debug_locks" -eq 0 ] && [ "$ret" -eq 0 ]; then
> +        echo "Warning: LOCKDEP issue detected. Please check dmesg logs for more information."
> +        cat /proc/lockdep_stats
> +        ret=101
> +    fi
> +fi
> +
>  cd $oldpath
>  exit $ret
> diff --git a/drivers/gpu/drm/ci/test.yml b/drivers/gpu/drm/ci/test.yml
> index 0eab020a33b9..3af735dbf6bd 100644
> --- a/drivers/gpu/drm/ci/test.yml
> +++ b/drivers/gpu/drm/ci/test.yml
> @@ -1,6 +1,8 @@
>  .lava-test:
>    extends:
>      - .container+build-rules
> +  variables:
> +    FF_USE_NEW_BASH_EVAL_STRATEGY: 'true'
>    timeout: "1h30m"
>    rules:
>      - !reference [.scheduled_pipeline-rules, rules]
> @@ -13,6 +15,9 @@
>      - mv -n install/* artifacts/.
>      # Override it with our lava-submit.sh script
>      - ./artifacts/lava-submit.sh
> +  allow_failure:
> +    exit_codes:
> +      - 101

Maybe we could have this rule more generically instead of just in lava,
so we can re-use it in other jobs as well and we don't need to repeat it.


Regards,
Helen

>
>  .lava-igt:arm32:
>    extends:
> @@ -88,9 +93,14 @@
>      - igt:arm64
>    tags:
>      - $RUNNER_TAG
> +  allow_failure:
> +    exit_codes:
> +      - 101
>
>  .software-driver:
>    stage: software-driver
> +  variables:
> +    FF_USE_NEW_BASH_EVAL_STRATEGY: 'true'
>    timeout: "1h30m"
>    rules:
>      - !reference [.scheduled_pipeline-rules, rules]
> @@ -108,6 +118,9 @@
>      - debian/x86_64_test-gl
>      - testing:x86_64
>      - igt:x86_64
> +  allow_failure:
> +    exit_codes:
> +      - 101
>
>  .msm-sc7180:
>    extends:
> @@ -153,7 +166,7 @@ msm:apq8016:
>      BM_KERNEL_EXTRA_ARGS: clk_ignore_unused
>      RUNNER_TAG: google-freedreno-db410c
>    script:
> -    - ./install/bare-metal/fastboot.sh
> +    - ./install/bare-metal/fastboot.sh || exit $?
>
>  msm:apq8096:
>    extends:
> @@ -167,7 +180,7 @@ msm:apq8096:
>      GPU_VERSION: apq8096
>      RUNNER_TAG: google-freedreno-db820c
>    script:
> -    - ./install/bare-metal/fastboot.sh
> +    - ./install/bare-metal/fastboot.sh || exit $?
>
>  msm:sdm845:
>    extends:
> @@ -181,7 +194,7 @@ msm:sdm845:
>      GPU_VERSION: sdm845
>      RUNNER_TAG: google-freedreno-cheza
>    script:
> -    - ./install/bare-metal/cros-servo.sh
> +    - ./install/bare-metal/cros-servo.sh || exit $?
>
>  msm:sm8350-hdk:
>    extends:
> --
> 2.43.0
>


-- 
Helen Koike

Re: [PATCH v2 3/3] drm/ci: enable lockdep detection

[Vignesh Raman]

Hi Helen,

On 13/02/25 18:13, Helen Mae Koike Fornazier wrote:
> Hi Vignesh,
> 
> thanks for your patch.
> 
> Em ter., 11 de fev. de 2025 às 12:29, Vignesh Raman
> <vignesh.raman@collabora.com> escreveu:
>>
>> We have enabled PROVE_LOCKING (which enables LOCKDEP) in drm-ci.
>> This will output warnings when kernel locking errors are encountered
>> and will continue executing tests. To detect if lockdep has been
>> triggered, check the debug_locks value in /proc/lockdep_stats after
>> the tests have run. When debug_locks is 0, it indicates that lockdep
>> has detected issues and turned itself off. Check this value, and if
>> lockdep is detected, exit with an error and configure it as a warning
>> in GitLab CI.
>>
>> GitLab CI ignores exit codes other than 1 by default. Pass the correct
>> exit code with variable FF_USE_NEW_BASH_EVAL_STRATEGY set to true or
>> exit on failure.
>>
>> Also update the documentation.
>>
>> Signed-off-by: Vignesh Raman <vignesh.raman@collabora.com>
>> ---
>>
>> v2:
>>    - Lockdep failures are reported as pipeline warnings,
>>      and the documentation is updated.
>>
>> ---
>>   Documentation/gpu/automated_testing.rst |  4 ++++
>>   drivers/gpu/drm/ci/igt_runner.sh        | 11 +++++++++++
>>   drivers/gpu/drm/ci/test.yml             | 19 ++++++++++++++++---
>>   3 files changed, 31 insertions(+), 3 deletions(-)
>>
>> diff --git a/Documentation/gpu/automated_testing.rst b/Documentation/gpu/automated_testing.rst
>> index 6d7c6086034d..62aa3ede02a5 100644
>> --- a/Documentation/gpu/automated_testing.rst
>> +++ b/Documentation/gpu/automated_testing.rst
>> @@ -115,6 +115,10 @@ created (eg. https://gitlab.freedesktop.org/janedoe/linux/-/pipelines)
>>   5. The various jobs will be run and when the pipeline is finished, all jobs
>>   should be green unless a regression has been found.
>>
>> +6. Warnings in the pipeline indicate that lockdep
>> +(see Documentation/locking/lockdep-design.rst) issues have been detected
>> +during the tests.
>> +
>>
>>   How to update test expectations
>>   ===============================
>> diff --git a/drivers/gpu/drm/ci/igt_runner.sh b/drivers/gpu/drm/ci/igt_runner.sh
>> index 68b042e43b7f..2a0599f12c58 100755
>> --- a/drivers/gpu/drm/ci/igt_runner.sh
>> +++ b/drivers/gpu/drm/ci/igt_runner.sh
>> @@ -85,5 +85,16 @@ deqp-runner junit \
>>      --limit 50 \
>>      --template "See $ARTIFACTS_BASE_URL/results/{{testcase}}.xml"
>>
>> +# Check if /proc/lockdep_stats exists
>> +if [ -f /proc/lockdep_stats ]; then
>> +    # If debug_locks is 0, it indicates lockdep is detected and it turns itself off.
>> +    debug_locks=$(grep 'debug_locks:' /proc/lockdep_stats | awk '{print $2}')
>> +    if [ "$debug_locks" -eq 0 ] && [ "$ret" -eq 0 ]; then
>> +        echo "Warning: LOCKDEP issue detected. Please check dmesg logs for more information."
>> +        cat /proc/lockdep_stats
>> +        ret=101
>> +    fi
>> +fi
>> +
>>   cd $oldpath
>>   exit $ret
>> diff --git a/drivers/gpu/drm/ci/test.yml b/drivers/gpu/drm/ci/test.yml
>> index 0eab020a33b9..3af735dbf6bd 100644
>> --- a/drivers/gpu/drm/ci/test.yml
>> +++ b/drivers/gpu/drm/ci/test.yml
>> @@ -1,6 +1,8 @@
>>   .lava-test:
>>     extends:
>>       - .container+build-rules
>> +  variables:
>> +    FF_USE_NEW_BASH_EVAL_STRATEGY: 'true'
>>     timeout: "1h30m"
>>     rules:
>>       - !reference [.scheduled_pipeline-rules, rules]
>> @@ -13,6 +15,9 @@
>>       - mv -n install/* artifacts/.
>>       # Override it with our lava-submit.sh script
>>       - ./artifacts/lava-submit.sh
>> +  allow_failure:
>> +    exit_codes:
>> +      - 101
> 
> Maybe we could have this rule more generically instead of just in lava,
> so we can re-use it in other jobs as well and we don't need to repeat it.

Yes agreed. I will post a patch with this update.

Regards,
Vignesh

> 
> 
> Regards,
> Helen
> 
>>
>>   .lava-igt:arm32:
>>     extends:
>> @@ -88,9 +93,14 @@
>>       - igt:arm64
>>     tags:
>>       - $RUNNER_TAG
>> +  allow_failure:
>> +    exit_codes:
>> +      - 101
>>
>>   .software-driver:
>>     stage: software-driver
>> +  variables:
>> +    FF_USE_NEW_BASH_EVAL_STRATEGY: 'true'
>>     timeout: "1h30m"
>>     rules:
>>       - !reference [.scheduled_pipeline-rules, rules]
>> @@ -108,6 +118,9 @@
>>       - debian/x86_64_test-gl
>>       - testing:x86_64
>>       - igt:x86_64
>> +  allow_failure:
>> +    exit_codes:
>> +      - 101
>>
>>   .msm-sc7180:
>>     extends:
>> @@ -153,7 +166,7 @@ msm:apq8016:
>>       BM_KERNEL_EXTRA_ARGS: clk_ignore_unused
>>       RUNNER_TAG: google-freedreno-db410c
>>     script:
>> -    - ./install/bare-metal/fastboot.sh
>> +    - ./install/bare-metal/fastboot.sh || exit $?
>>
>>   msm:apq8096:
>>     extends:
>> @@ -167,7 +180,7 @@ msm:apq8096:
>>       GPU_VERSION: apq8096
>>       RUNNER_TAG: google-freedreno-db820c
>>     script:
>> -    - ./install/bare-metal/fastboot.sh
>> +    - ./install/bare-metal/fastboot.sh || exit $?
>>
>>   msm:sdm845:
>>     extends:
>> @@ -181,7 +194,7 @@ msm:sdm845:
>>       GPU_VERSION: sdm845
>>       RUNNER_TAG: google-freedreno-cheza
>>     script:
>> -    - ./install/bare-metal/cros-servo.sh
>> +    - ./install/bare-metal/cros-servo.sh || exit $?
>>
>>   msm:sm8350-hdk:
>>     extends:
>> --
>> 2.43.0
>>
> 
>