mirror of https://lore.kernel.org/lkml/
 help / color / mirror / Atom feed
* [PATCH v1 0/2] drm/ci: uprev mesa and modify gitlab rules
@ 2024-11-28  4:20 Vignesh Raman
  2024-11-28  4:20 ` [PATCH v1 1/2] drm/ci: uprev mesa Vignesh Raman
                   ` (2 more replies)
  0 siblings, 3 replies; 10+ messages in thread
From: Vignesh Raman @ 2024-11-28  4:20 UTC (permalink / raw)
  To: dri-devel
  Cc: daniels, helen.koike, airlied, daniel, robdclark,
	guilherme.gallo, sergi.blanch.torne, jani.nikula,
	dmitry.baryshkov, mripard, linux-kernel

Uprev mesa to adapt to the latest changes in mesa-ci,
including new container jobs and stages. Update the
lava-submit script to align with recent mesa-ci changes
for using LAVA rootfs overlays. Modify gitLab rules
to include scheduled pipelines.

Pipeline link,
https://gitlab.freedesktop.org/vigneshraman/linux/-/pipelines/1318024

Vignesh Raman (2):
  drm/ci: uprev mesa
  drm/ci: update gitlab rules

 drivers/gpu/drm/ci/build.sh       |   2 +-
 drivers/gpu/drm/ci/build.yml      | 104 ++++++++++++++++-
 drivers/gpu/drm/ci/container.yml  |  22 ++--
 drivers/gpu/drm/ci/gitlab-ci.yml  | 185 ++++++++++++++++--------------
 drivers/gpu/drm/ci/image-tags.yml |  11 +-
 drivers/gpu/drm/ci/lava-submit.sh |  99 +++++++++++-----
 drivers/gpu/drm/ci/test.yml       |  33 +++---
 7 files changed, 314 insertions(+), 142 deletions(-)

-- 
2.43.0


^ permalink raw reply	[flat|nested] 10+ messages in thread

* [PATCH v1 1/2] drm/ci: uprev mesa
  2024-11-28  4:20 [PATCH v1 0/2] drm/ci: uprev mesa and modify gitlab rules Vignesh Raman
@ 2024-11-28  4:20 ` Vignesh Raman
  2024-12-03 14:14   ` Helen Mae Koike Fornazier
  2024-11-28  4:20 ` [PATCH v1 2/2] drm/ci: update gitlab rules Vignesh Raman
  2024-12-03 14:12 ` [PATCH v1 0/2] drm/ci: uprev mesa and modify " Helen Mae Koike Fornazier
  2 siblings, 1 reply; 10+ messages in thread
From: Vignesh Raman @ 2024-11-28  4:20 UTC (permalink / raw)
  To: dri-devel
  Cc: daniels, helen.koike, airlied, daniel, robdclark,
	guilherme.gallo, sergi.blanch.torne, jani.nikula,
	dmitry.baryshkov, mripard, linux-kernel

Uprev mesa to adapt to the latest changes in mesa ci
which includes new container jobs and stages. Also update
lava-submit script to adapt to the recent changes in mesa
to use LAVA rootfs overlays.

Signed-off-by: Vignesh Raman <vignesh.raman@collabora.com>
---
 drivers/gpu/drm/ci/build.sh       |   2 +-
 drivers/gpu/drm/ci/build.yml      | 103 +++++++++++++++++++++++++++++-
 drivers/gpu/drm/ci/container.yml  |  22 +++----
 drivers/gpu/drm/ci/gitlab-ci.yml  |  71 ++++++++++++++------
 drivers/gpu/drm/ci/image-tags.yml |  11 +++-
 drivers/gpu/drm/ci/lava-submit.sh |  99 ++++++++++++++++++++--------
 drivers/gpu/drm/ci/test.yml       |  13 ++--
 7 files changed, 255 insertions(+), 66 deletions(-)

diff --git a/drivers/gpu/drm/ci/build.sh b/drivers/gpu/drm/ci/build.sh
index 139b81db6312..19fe01257ab9 100644
--- a/drivers/gpu/drm/ci/build.sh
+++ b/drivers/gpu/drm/ci/build.sh
@@ -132,7 +132,7 @@ fi
 # Pass needed files to the test stage
 mkdir -p install
 cp -rfv .gitlab-ci/* install/.
-cp -rfv ci/*  install/.
+cp -rfv bin/ci/*  install/.
 cp -rfv install/common install/ci-common
 cp -rfv drivers/gpu/drm/ci/* install/.
 
diff --git a/drivers/gpu/drm/ci/build.yml b/drivers/gpu/drm/ci/build.yml
index 9c198239033d..771ecfc5008d 100644
--- a/drivers/gpu/drm/ci/build.yml
+++ b/drivers/gpu/drm/ci/build.yml
@@ -2,7 +2,7 @@
   extends:
     - .build-rules
     - .container+build-rules
-  stage: build
+  stage: build-only
   artifacts:
     paths:
       - artifacts
@@ -110,3 +110,104 @@ build-nodebugfs:arm64:
 
 build:x86_64:
   extends: .build:x86_64
+
+# Disable build jobs that we won't use
+alpine-build-testing:
+  rules:
+    - when: never
+
+debian-android:
+  rules:
+    - when: never
+
+debian-arm32:
+  rules:
+    - when: never
+
+debian-arm32-asan:
+  rules:
+    - when: never
+
+debian-arm64:
+  rules:
+    - when: never
+
+debian-arm64-asan:
+  rules:
+    - when: never
+
+debian-arm64-build-test:
+  rules:
+    - when: never
+
+debian-arm64-release:
+  rules:
+    - when: never
+
+debian-build-testing:
+  rules:
+    - when: never
+
+debian-clang:
+  rules:
+    - when: never
+
+debian-clang-release:
+  rules:
+    - when: never
+
+debian-no-libdrm:
+  rules:
+    - when: never
+
+debian-ppc64el:
+  rules:
+    - when: never
+
+debian-release:
+  rules:
+    - when: never
+
+debian-s390x:
+  rules:
+    - when: never
+
+debian-testing:
+  rules:
+    - when: never
+
+debian-testing-asan:
+  rules:
+    - when: never
+
+debian-testing-msan:
+  rules:
+    - when: never
+
+debian-vulkan:
+  rules:
+    - when: never
+
+debian-x86_32:
+  rules:
+    - when: never
+
+fedora-release:
+  rules:
+    - when: never
+
+rustfmt:
+  rules:
+    - when: never
+
+shader-db:
+  rules:
+    - when: never
+
+windows-msvc:
+  rules:
+    - when: never
+
+yaml-toml-shell-test:
+  rules:
+    - when: never
diff --git a/drivers/gpu/drm/ci/container.yml b/drivers/gpu/drm/ci/container.yml
index 2a94f54ce4cf..07dc13ff865d 100644
--- a/drivers/gpu/drm/ci/container.yml
+++ b/drivers/gpu/drm/ci/container.yml
@@ -24,7 +24,7 @@ alpine/x86_64_build:
   rules:
     - when: never
 
-debian/x86_64_test-vk:
+debian/arm64_test-gl:
   rules:
     - when: never
 
@@ -32,7 +32,15 @@ debian/arm64_test-vk:
   rules:
     - when: never
 
-debian/arm64_test-gl:
+debian/ppc64el_build:
+  rules:
+    - when: never
+
+debian/s390x_build:
+  rules:
+    - when: never
+
+debian/x86_64_test-vk:
   rules:
     - when: never
 
@@ -56,14 +64,6 @@ windows_test_msvc:
   rules:
     - when: never
 
-.debian/x86_64_build-mingw:
-   rules:
-    - when: never
-
-rustfmt:
-   rules:
-    - when: never
-
 windows_msvc:
    rules:
-    - when: never
\ No newline at end of file
+    - when: never
diff --git a/drivers/gpu/drm/ci/gitlab-ci.yml b/drivers/gpu/drm/ci/gitlab-ci.yml
index 90bde9f00cc3..86d8c5d8ce3b 100644
--- a/drivers/gpu/drm/ci/gitlab-ci.yml
+++ b/drivers/gpu/drm/ci/gitlab-ci.yml
@@ -1,6 +1,6 @@
 variables:
   DRM_CI_PROJECT_PATH: &drm-ci-project-path mesa/mesa
-  DRM_CI_COMMIT_SHA: &drm-ci-commit-sha c6a9a9c3bce90923f7700219354e0b6e5a3c9ba6
+  DRM_CI_COMMIT_SHA: &drm-ci-commit-sha e1a8fd80d411a5ff8fa19ffcf09516ac5099a25c
 
   UPSTREAM_REPO: https://gitlab.freedesktop.org/drm/kernel.git
   TARGET_BRANCH: drm-next
@@ -31,11 +31,7 @@ variables:
   PIPELINE_ARTIFACTS_BASE: ${S3_HOST}/${S3_ARTIFACTS_BUCKET}/${CI_PROJECT_PATH}/${CI_PIPELINE_ID}
   # per-job artifact storage on MinIO
   JOB_ARTIFACTS_BASE: ${PIPELINE_ARTIFACTS_BASE}/${CI_JOB_ID}
-  # default kernel for rootfs before injecting the current kernel tree
-  KERNEL_REPO: "gfx-ci/linux"
-  KERNEL_TAG: "v6.6.21-mesa-f8ea"
   KERNEL_IMAGE_BASE: https://${S3_HOST}/${S3_KERNEL_BUCKET}/${KERNEL_REPO}/${KERNEL_TAG}
-  PKG_REPO_REV: "3cc12a2a"
   LAVA_TAGS: subset-1-gfx
   LAVA_JOB_PRIORITY: 30
   ARTIFACTS_BASE_URL: https://${CI_PROJECT_ROOT_NAMESPACE}.${CI_PAGES_DOMAIN}/-/${CI_PROJECT_NAME}/-/jobs/${CI_JOB_ID}/artifacts
@@ -59,7 +55,7 @@ default:
     - cd $CI_PROJECT_DIR
     - curl --output - $DRM_CI_PROJECT_URL/-/archive/$DRM_CI_COMMIT_SHA/mesa-$DRM_CI_COMMIT_SHA.tar.gz | tar -xz
     - mv mesa-$DRM_CI_COMMIT_SHA/.gitlab-ci* .
-    - mv mesa-$DRM_CI_COMMIT_SHA/bin/ci .
+    - mv mesa-$DRM_CI_COMMIT_SHA/bin .
     - rm -rf mesa-$DRM_CI_COMMIT_SHA/
     - echo -e "\e[0Ksection_end:$(date +%s):drm_ci_download_section\r\e[0K"
 
@@ -85,6 +81,7 @@ include:
   - project: *drm-ci-project-path
     ref: *drm-ci-commit-sha
     file:
+      - '/.gitlab-ci/build/gitlab-ci.yml'
       - '/.gitlab-ci/container/gitlab-ci.yml'
       - '/.gitlab-ci/farm-rules.yml'
       - '/.gitlab-ci/lava/lava-gitlab-ci.yml'
@@ -115,9 +112,10 @@ include:
 stages:
   - sanity
   - container
-  - code-validation
   - git-archive
-  - build
+  - build-for-tests
+  - build-only
+  - code-validation
   - amdgpu
   - i915
   - mediatek
@@ -264,30 +262,63 @@ sanity:
   rules:
     - if: *is-pre-merge
       when: on_success
-    # Other cases default to never
+    - when: never
   variables:
     GIT_STRATEGY: none
   script:
     # ci-fairy check-commits --junit-xml=check-commits.xml
     - ci-fairy check-merge-request --require-allow-collaboration --junit-xml=check-merge-request.xml
+    - |
+      set -eu
+      image_tags=(
+        ALPINE_X86_64_LAVA_SSH_TAG
+        CONTAINER_TAG
+        DEBIAN_BASE_TAG
+        DEBIAN_BUILD_TAG
+        DEBIAN_PYUTILS_TAG
+        DEBIAN_TEST_GL_TAG
+        KERNEL_ROOTFS_TAG
+        KERNEL_TAG
+        PKG_REPO_REV
+      )
+      for var in "${image_tags[@]}"
+      do
+        if [ "$(echo -n "${!var}" | wc -c)" -gt 20 ]
+        then
+          echo "$var is too long; please make sure it is at most 20 chars."
+          exit 1
+        fi
+      done
   artifacts:
     when: on_failure
     reports:
       junit: check-*.xml
+  tags:
+    - placeholder-job
 
-# Rules for tests that should not block merging, but should be available to
-# optionally run with the "play" button in the UI in pre-merge non-marge
-# pipelines.  This should appear in "extends:" after any includes of
-# test-source-dep.yml rules, so that these rules replace those.
-.test-manual-mr:
+
+mr-label-maker-test:
+  extends:
+    - .fdo.ci-fairy
+  stage: sanity
   rules:
-    - !reference [.no_scheduled_pipelines-rules, rules]
-    - if: *is-forked-branch-or-pre-merge-not-for-marge
-      when: manual
+    - !reference [.mr-label-maker-rules, rules]
   variables:
-    JOB_TIMEOUT: 80
-
+    GIT_STRATEGY: fetch
+  timeout: 10m
+  script:
+    - set -eu
+    - python3 -m venv .venv
+    - source .venv/bin/activate
+    - pip install git+https://gitlab.freedesktop.org/freedesktop/mr-label-maker
+    - mr-label-maker --dry-run --mr $CI_MERGE_REQUEST_IID
 
 # Jobs that need to pass before spending hardware resources on further testing
 .required-for-hardware-jobs:
-  needs: []
+  needs:
+    - job: clang-format
+      optional: true
+    - job: rustfmt
+      optional: true
+    - job: toml-lint
+      optional: true
diff --git a/drivers/gpu/drm/ci/image-tags.yml b/drivers/gpu/drm/ci/image-tags.yml
index 8d8b9e71852e..528ef44285bb 100644
--- a/drivers/gpu/drm/ci/image-tags.yml
+++ b/drivers/gpu/drm/ci/image-tags.yml
@@ -1,5 +1,5 @@
 variables:
-   CONTAINER_TAG: "2024-09-09-uprevs"
+   CONTAINER_TAG: "20241127-mesa-uprev"
    DEBIAN_X86_64_BUILD_BASE_IMAGE: "debian/x86_64_build-base"
    DEBIAN_BASE_TAG: "${CONTAINER_TAG}"
 
@@ -7,9 +7,16 @@ variables:
    DEBIAN_BUILD_TAG: "${CONTAINER_TAG}"
 
    KERNEL_ROOTFS_TAG: "${CONTAINER_TAG}"
+   # default kernel for rootfs before injecting the current kernel tree
+   KERNEL_TAG: "v6.6.21-mesa-f8ea"
+   KERNEL_REPO: "gfx-ci/linux"
+   PKG_REPO_REV: "bca9635d"
 
    DEBIAN_X86_64_TEST_BASE_IMAGE: "debian/x86_64_test-base"
    DEBIAN_X86_64_TEST_IMAGE_GL_PATH: "debian/x86_64_test-gl"
    DEBIAN_TEST_GL_TAG: "${CONTAINER_TAG}"
 
-   ALPINE_X86_64_LAVA_SSH_TAG: "${CONTAINER_TAG}"
\ No newline at end of file
+   DEBIAN_PYUTILS_IMAGE: "debian/x86_64_pyutils"
+   DEBIAN_PYUTILS_TAG: "${CONTAINER_TAG}"
+
+   ALPINE_X86_64_LAVA_SSH_TAG: "${CONTAINER_TAG}"
diff --git a/drivers/gpu/drm/ci/lava-submit.sh b/drivers/gpu/drm/ci/lava-submit.sh
index 6add15083c78..a1404a2d8cee 100755
--- a/drivers/gpu/drm/ci/lava-submit.sh
+++ b/drivers/gpu/drm/ci/lava-submit.sh
@@ -1,58 +1,105 @@
-#!/bin/bash
+#!/usr/bin/env bash
 # SPDX-License-Identifier: MIT
+# shellcheck disable=SC2086 # we want word splitting
+# shellcheck disable=SC1091 # paths only become valid at runtime
 
-set -e
-set -x
+# If we run in the fork (not from mesa or Marge-bot), reuse mainline kernel and rootfs, if exist.
+_check_artifact_path() {
+	_url="https://${1}/${2}"
+	if curl -s -o /dev/null -I -L -f --retry 4 --retry-delay 15 "${_url}"; then
+		echo -n "${_url}"
+	fi
+}
 
-# Try to use the kernel and rootfs built in mainline first, so we're more
-# likely to hit cache
-if curl -L --retry 4 -f --retry-all-errors --retry-delay 60 -s "https://${BASE_SYSTEM_MAINLINE_HOST_PATH}/done"; then
-	BASE_SYSTEM_HOST_PATH="${BASE_SYSTEM_MAINLINE_HOST_PATH}"
-else
-	BASE_SYSTEM_HOST_PATH="${BASE_SYSTEM_FORK_HOST_PATH}"
-fi
+get_path_to_artifact() {
+	_mainline_artifact="$(_check_artifact_path ${BASE_SYSTEM_MAINLINE_HOST_PATH} ${1})"
+	if [ -n "${_mainline_artifact}" ]; then
+		echo -n "${_mainline_artifact}"
+		return
+	fi
+	_fork_artifact="$(_check_artifact_path ${BASE_SYSTEM_FORK_HOST_PATH} ${1})"
+	if [ -n "${_fork_artifact}" ]; then
+		echo -n "${_fork_artifact}"
+		return
+	fi
+	set +x
+	error "Sorry, I couldn't find a viable built path for ${1} in either mainline or a fork." >&2
+	echo "" >&2
+	echo "If you're working on CI, this probably means that you're missing a dependency:" >&2
+	echo "this job ran ahead of the job which was supposed to upload that artifact." >&2
+	echo "" >&2
+	echo "If you aren't working on CI, please ping @mesa/ci-helpers to see if we can help." >&2
+	echo "" >&2
+	echo "This job is going to fail, because I can't find the resources I need. Sorry." >&2
+	set -x
+	exit 1
+}
+
+. "${SCRIPTS_DIR}/setup-test-env.sh"
+
+section_start prepare_rootfs "Preparing root filesystem"
+
+set -ex
+
+section_switch rootfs "Assembling root filesystem"
+ROOTFS_URL="$(get_path_to_artifact lava-rootfs.tar.zst)"
+[ $? != 1 ] || exit 1
 
 rm -rf results
 mkdir -p results/job-rootfs-overlay/
 
+artifacts/ci-common/generate-env.sh > results/job-rootfs-overlay/set-job-env-vars.sh
 cp artifacts/ci-common/capture-devcoredump.sh results/job-rootfs-overlay/
 cp artifacts/ci-common/init-*.sh results/job-rootfs-overlay/
 cp artifacts/ci-common/intel-gpu-freq.sh results/job-rootfs-overlay/
+cp artifacts/ci-common/kdl.sh results/job-rootfs-overlay/
 cp "$SCRIPTS_DIR"/setup-test-env.sh results/job-rootfs-overlay/
 
-# Prepare env vars for upload.
-section_start variables "Variables passed through:"
-KERNEL_IMAGE_BASE="https://${BASE_SYSTEM_HOST_PATH}" \
-	artifacts/ci-common/generate-env.sh | tee results/job-rootfs-overlay/set-job-env-vars.sh
-section_end variables
-
 tar zcf job-rootfs-overlay.tar.gz -C results/job-rootfs-overlay/ .
 ci-fairy s3cp --token-file "${S3_JWT_FILE}" job-rootfs-overlay.tar.gz "https://${JOB_ROOTFS_OVERLAY_PATH}"
 
+# Prepare env vars for upload.
+section_switch variables "Environment variables passed through to device:"
+cat results/job-rootfs-overlay/set-job-env-vars.sh
+
+section_switch lava_submit "Submitting job for scheduling"
+
 touch results/lava.log
 tail -f results/lava.log &
-
 PYTHONPATH=artifacts/ artifacts/lava/lava_job_submitter.py \
-	submit \
+	--farm "${FARM}" \
+	--device-type "${DEVICE_TYPE}" \
+	--boot-method "${BOOT_METHOD}" \
+	--job-timeout-min ${JOB_TIMEOUT:-30} \
 	--dump-yaml \
 	--pipeline-info "$CI_JOB_NAME: $CI_PIPELINE_URL on $CI_COMMIT_REF_NAME ${CI_NODE_INDEX}/${CI_NODE_TOTAL}" \
-	--rootfs-url-prefix "https://${BASE_SYSTEM_HOST_PATH}" \
+	--rootfs-url "${ROOTFS_URL}" \
 	--kernel-url-prefix "https://${PIPELINE_ARTIFACTS_BASE}/${DEBIAN_ARCH}" \
-	--build-url "${FDO_HTTP_CACHE_URI:-}https://${PIPELINE_ARTIFACTS_BASE}/${DEBIAN_ARCH}/kernel-files.tar.zst" \
-	--job-rootfs-overlay-url "${FDO_HTTP_CACHE_URI:-}https://${JOB_ROOTFS_OVERLAY_PATH}" \
-	--job-timeout-min ${JOB_TIMEOUT:-80} \
+	--kernel-external "${EXTERNAL_KERNEL_TAG}" \
 	--first-stage-init artifacts/ci-common/init-stage1.sh \
-	--ci-project-dir "${CI_PROJECT_DIR}" \
-	--device-type "${DEVICE_TYPE}" \
-	--farm "${FARM}" \
 	--dtb-filename "${DTB}" \
 	--jwt-file "${S3_JWT_FILE}" \
 	--kernel-image-name "${KERNEL_IMAGE_NAME}" \
 	--kernel-image-type "${KERNEL_IMAGE_TYPE}" \
-	--boot-method "${BOOT_METHOD}" \
 	--visibility-group "${VISIBILITY_GROUP}" \
 	--lava-tags "${LAVA_TAGS}" \
 	--mesa-job-name "$CI_JOB_NAME" \
 	--structured-log-file "results/lava_job_detail.json" \
 	--ssh-client-image "${LAVA_SSH_CLIENT_IMAGE}" \
+	--project-name "${CI_PROJECT_NAME}" \
+	--starting-section "${CURRENT_SECTION}" \
+	--job-submitted-at "${CI_JOB_STARTED_AT}" \
+	- append-overlay \
+		--name=kernel-build \
+		--url="${FDO_HTTP_CACHE_URI:-}https://${PIPELINE_ARTIFACTS_BASE}/${DEBIAN_ARCH}/kernel-files.tar.zst" \
+		--compression=zstd \
+		--path="${CI_PROJECT_DIR}" \
+		--format=tar \
+	- append-overlay \
+		--name=job-overlay \
+		--url="https://${JOB_ROOTFS_OVERLAY_PATH}" \
+		--compression=gz \
+		--path="/" \
+		--format=tar \
+	- submit \
 	>> results/lava.log
diff --git a/drivers/gpu/drm/ci/test.yml b/drivers/gpu/drm/ci/test.yml
index f0ef60c8f56d..42a8e7f93714 100644
--- a/drivers/gpu/drm/ci/test.yml
+++ b/drivers/gpu/drm/ci/test.yml
@@ -1,21 +1,21 @@
 .test-rules:
   rules:
-    - if: '$FD_FARM == "offline" && $RUNNER_TAG =~ /^google-freedreno-/'
-      when: never
-    - if: '$COLLABORA_FARM == "offline" && $RUNNER_TAG =~ /^mesa-ci-x86-64-lava-/'
-      when: never
     - !reference [.no_scheduled_pipelines-rules, rules]
+    - !reference [.collabora-farm-rules, rules]
+    - !reference [.google-freedreno-farm-rules, rules]
     - when: on_success
 
 .lava-test:
   extends:
     - .test-rules
+    - .build-rules
+    - .container+build-rules
   timeout: "1h30m"
   script:
     # Note: Build dir (and thus install) may be dirty due to GIT_STRATEGY
     - rm -rf install
     - tar -xf artifacts/install.tar
-    - mv install/* artifacts/.
+    - mv -n install/* artifacts/.
     # Override it with our lava-submit.sh script
     - ./artifacts/lava-submit.sh
 
@@ -32,6 +32,7 @@
     - alpine/x86_64_lava_ssh_client
     - kernel+rootfs_arm32
     - debian/x86_64_build
+    - python-test
     - testing:arm32
     - igt:arm32
 
@@ -48,6 +49,7 @@
     - alpine/x86_64_lava_ssh_client
     - kernel+rootfs_arm64
     - debian/x86_64_build
+    - python-test
     - testing:arm64
     - igt:arm64
 
@@ -64,6 +66,7 @@
     - alpine/x86_64_lava_ssh_client
     - kernel+rootfs_x86_64
     - debian/x86_64_build
+    - python-test
     - testing:x86_64
     - igt:x86_64
 
-- 
2.43.0


^ permalink raw reply	[flat|nested] 10+ messages in thread

* [PATCH v1 2/2] drm/ci: update gitlab rules
  2024-11-28  4:20 [PATCH v1 0/2] drm/ci: uprev mesa and modify gitlab rules Vignesh Raman
  2024-11-28  4:20 ` [PATCH v1 1/2] drm/ci: uprev mesa Vignesh Raman
@ 2024-11-28  4:20 ` Vignesh Raman
  2024-12-03 14:12 ` [PATCH v1 0/2] drm/ci: uprev mesa and modify " Helen Mae Koike Fornazier
  2 siblings, 0 replies; 10+ messages in thread
From: Vignesh Raman @ 2024-11-28  4:20 UTC (permalink / raw)
  To: dri-devel
  Cc: daniels, helen.koike, airlied, daniel, robdclark,
	guilherme.gallo, sergi.blanch.torne, jani.nikula,
	dmitry.baryshkov, mripard, linux-kernel

Update gitlab rules to include scheduled pipelines.

Signed-off-by: Vignesh Raman <vignesh.raman@collabora.com>
---
 drivers/gpu/drm/ci/build.yml     |   1 -
 drivers/gpu/drm/ci/gitlab-ci.yml | 116 +++++++++++++------------------
 drivers/gpu/drm/ci/test.yml      |  26 +++----
 3 files changed, 63 insertions(+), 80 deletions(-)

diff --git a/drivers/gpu/drm/ci/build.yml b/drivers/gpu/drm/ci/build.yml
index 771ecfc5008d..4ea54aee3dcd 100644
--- a/drivers/gpu/drm/ci/build.yml
+++ b/drivers/gpu/drm/ci/build.yml
@@ -1,6 +1,5 @@
 .build:
   extends:
-    - .build-rules
     - .container+build-rules
   stage: build-only
   artifacts:
diff --git a/drivers/gpu/drm/ci/gitlab-ci.yml b/drivers/gpu/drm/ci/gitlab-ci.yml
index 86d8c5d8ce3b..49d6930ba87f 100644
--- a/drivers/gpu/drm/ci/gitlab-ci.yml
+++ b/drivers/gpu/drm/ci/gitlab-ci.yml
@@ -38,6 +38,7 @@ variables:
   # Python scripts for structured logger
   PYTHONPATH: "$PYTHONPATH:$CI_PROJECT_DIR/install"
 
+
 default:
   id_tokens:
     S3_JWT:
@@ -67,6 +68,7 @@ default:
       export S3_JWT="$(<${S3_JWT_FILE})" &&
       rm "${S3_JWT_FILE}"
 
+
 include:
   - project: 'freedesktop/ci-templates'
     ref: 16bc29078de5e0a067ff84a1a199a3760d3b3811
@@ -126,33 +128,27 @@ stages:
   - rockchip
   - software-driver
 
+
 # YAML anchors for rule conditions
 # --------------------------------
 .rules-anchors:
   rules:
-    # Pipeline for forked project branch
-    - if: &is-forked-branch '$CI_COMMIT_BRANCH && $CI_PROJECT_NAMESPACE != "mesa"'
-      when: manual
-    # Forked project branch / pre-merge pipeline not for Marge bot
-    - if: &is-forked-branch-or-pre-merge-not-for-marge '$CI_PROJECT_NAMESPACE != "mesa" || ($GITLAB_USER_LOGIN != "marge-bot" && $CI_PIPELINE_SOURCE == "merge_request_event")'
-      when: manual
-    # Pipeline runs for the main branch of the upstream Mesa project
-    - if: &is-mesa-main '$CI_PROJECT_NAMESPACE == "mesa" && $CI_COMMIT_REF_NAME == $CI_DEFAULT_BRANCH && $CI_COMMIT_BRANCH'
-      when: always
-    # Post-merge pipeline
-    - if: &is-post-merge '$CI_PROJECT_NAMESPACE == "mesa" && $CI_COMMIT_BRANCH'
-      when: on_success
-    # Post-merge pipeline, not for Marge Bot
-    - if: &is-post-merge-not-for-marge '$CI_PROJECT_NAMESPACE == "mesa" && $GITLAB_USER_LOGIN != "marge-bot" && $CI_COMMIT_BRANCH'
-      when: on_success
+    # do not duplicate pipelines on merge pipelines
+    - if: $CI_COMMIT_BRANCH && $CI_OPEN_MERGE_REQUESTS && $CI_PIPELINE_SOURCE == "push"
+      when: never
+    # merge pipeline
+    - if: &is-merge-attempt $GITLAB_USER_LOGIN == "marge-bot" && $CI_PIPELINE_SOURCE == "merge_request_event"
+    # post-merge pipeline
+    - if: &is-post-merge $GITLAB_USER_LOGIN == "marge-bot" && $CI_PIPELINE_SOURCE == "push"
     # Pre-merge pipeline
-    - if: &is-pre-merge '$CI_PIPELINE_SOURCE == "merge_request_event"'
-      when: on_success
-    # Pre-merge pipeline for Marge Bot
-    - if: &is-pre-merge-for-marge '$GITLAB_USER_LOGIN == "marge-bot" && $CI_PIPELINE_SOURCE == "merge_request_event"'
-      when: on_success
+    - if: &is-pre-merge $CI_PIPELINE_SOURCE == "merge_request_event"
     # Push to a branch on a fork
-    - &is-fork-push '$CI_PROJECT_NAMESPACE != "mesa" && $CI_PIPELINE_SOURCE == "push"'
+    - if: &is-fork-push $CI_PROJECT_NAMESPACE != "mesa" && $CI_PIPELINE_SOURCE == "push"
+    # nightly pipeline
+    - if: &is-scheduled-pipeline $CI_PIPELINE_SOURCE == "schedule"
+    # pipeline for direct pushes that bypassed the CI
+    - if: &is-direct-push $CI_PROJECT_NAMESPACE == "mesa" && $CI_PIPELINE_SOURCE == "push" && $GITLAB_USER_LOGIN != "marge-bot"
+
 
 # Rules applied to every job in the pipeline
 .common-rules:
@@ -160,39 +156,48 @@ stages:
     - if: *is-fork-push
       when: manual
 
+
 .never-post-merge-rules:
   rules:
     - if: *is-post-merge
       when: never
 
-# Rule to filter for only scheduled pipelines.
-.scheduled_pipeline-rules:
-  rules:
-    - if: &is-scheduled-pipeline '$CI_PIPELINE_SOURCE == "schedule"'
-      when: on_success
-
-# Generic rule to not run the job during scheduled pipelines. Jobs that aren't
-# something like a nightly run should include this rule.
-.no_scheduled_pipelines-rules:
-  rules:
-    - if: *is-scheduled-pipeline
-      when: never
 
-# When to automatically run the CI for build jobs
-.build-rules:
-  rules:
-    - !reference [.no_scheduled_pipelines-rules, rules]
-    - !reference [.never-post-merge-rules, rules]
-    # Run automatically once all dependency jobs have passed
-    - when: on_success
-
-# When to automatically run the CI for container jobs
 .container+build-rules:
   rules:
-    - !reference [.no_scheduled_pipelines-rules, rules]
+    - !reference [.common-rules, rules]
+    # Run when re-enabling a disabled farm, but not when disabling it
+    - !reference [.disable-farm-mr-rules, rules]
+    # Never run immediately after merging, as we just ran everything
     - !reference [.never-post-merge-rules, rules]
+    # Build everything in merge pipelines, if any files affecting the pipeline
+    # were changed
+    - if: *is-merge-attempt
+      changes: &all_paths
+      - drivers/gpu/drm/ci/**/*
+      when: on_success
+    # Same as above, but for pre-merge pipelines
+    - if: *is-pre-merge
+      changes:
+        *all_paths
+      when: manual
+    # Skip everything for pre-merge and merge pipelines which don't change
+    # anything in the build
+    - if: *is-merge-attempt
+      when: never
+    - if: *is-pre-merge
+      when: never
+    # Build everything after someone bypassed the CI
+    - if: *is-direct-push
+      when: on_success
+    # Build everything in scheduled pipelines
+    - if: *is-scheduled-pipeline
+      when: on_success
+    # Allow building everything in fork pipelines, but build nothing unless
+    # manually triggered
     - when: manual
 
+
 .ci-deqp-artifacts:
   artifacts:
     name: "mesa_${CI_JOB_NAME}"
@@ -206,31 +211,7 @@ stages:
       - _build/meson-logs/strace
 
 
-.container-rules:
-  rules:
-    - !reference [.no_scheduled_pipelines-rules, rules]
-    - !reference [.never-post-merge-rules, rules]
-    # Run pipeline by default in the main project if any CI pipeline
-    # configuration files were changed, to ensure docker images are up to date
-    - if: *is-post-merge
-      changes:
-      - drivers/gpu/drm/ci/**/*
-      when: on_success
-    # Run pipeline by default if it was triggered by Marge Bot, is for a
-    # merge request, and any files affecting the pipeline were changed
-    - if: *is-pre-merge-for-marge
-      when: on_success
-    # Run pipeline by default in the main project if it was not triggered by
-    # Marge Bot, and any files affecting the pipeline were changed
-    - if: *is-post-merge-not-for-marge
-      when: on_success
-    # Allow triggering jobs manually in other cases
-    - when: manual
-
-
-
 # Git archive
-
 make git archive:
   extends:
     - .fdo.ci-fairy
@@ -313,6 +294,7 @@ mr-label-maker-test:
     - pip install git+https://gitlab.freedesktop.org/freedesktop/mr-label-maker
     - mr-label-maker --dry-run --mr $CI_MERGE_REQUEST_IID
 
+
 # Jobs that need to pass before spending hardware resources on further testing
 .required-for-hardware-jobs:
   needs:
diff --git a/drivers/gpu/drm/ci/test.yml b/drivers/gpu/drm/ci/test.yml
index 42a8e7f93714..6bfcbd4c0350 100644
--- a/drivers/gpu/drm/ci/test.yml
+++ b/drivers/gpu/drm/ci/test.yml
@@ -1,16 +1,11 @@
-.test-rules:
-  rules:
-    - !reference [.no_scheduled_pipelines-rules, rules]
-    - !reference [.collabora-farm-rules, rules]
-    - !reference [.google-freedreno-farm-rules, rules]
-    - when: on_success
-
 .lava-test:
   extends:
-    - .test-rules
-    - .build-rules
     - .container+build-rules
   timeout: "1h30m"
+  rules:
+    - !reference [.scheduled_pipeline-rules, rules]
+    - !reference [.collabora-farm-rules, rules]
+    - when: on_success
   script:
     # Note: Build dir (and thus install) may be dirty due to GIT_STRATEGY
     - rm -rf install
@@ -74,8 +69,11 @@
   extends:
     - .baremetal-test-arm64
     - .use-debian/baremetal_arm64_test
-    - .test-rules
   timeout: "1h30m"
+  rules:
+    - !reference [.scheduled_pipeline-rules, rules]
+    - !reference [.google-freedreno-farm-rules, rules]
+    - when: on_success
   variables:
     FDO_CI_CONCURRENT: 10
     HWCI_TEST_SCRIPT: "/install/igt_runner.sh"
@@ -444,13 +442,15 @@ panfrost:g12b:
 virtio_gpu:none:
   stage: software-driver
   timeout: "1h30m"
+  rules:
+    - !reference [.scheduled_pipeline-rules, rules]
+    - when: on_success
   variables:
     CROSVM_GALLIUM_DRIVER: llvmpipe
     DRIVER_NAME: virtio_gpu
     GPU_VERSION: none
   extends:
     - .test-gl
-    - .test-rules
   tags:
     - kvm
   script:
@@ -467,12 +467,14 @@ virtio_gpu:none:
 vkms:none:
   stage: software-driver
   timeout: "1h30m"
+  rules:
+    - !reference [.scheduled_pipeline-rules, rules]
+    - when: on_success
   variables:
     DRIVER_NAME: vkms
     GPU_VERSION: none
   extends:
     - .test-gl
-    - .test-rules
   tags:
     - kvm
   script:
-- 
2.43.0


^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: [PATCH v1 0/2] drm/ci: uprev mesa and modify gitlab rules
  2024-11-28  4:20 [PATCH v1 0/2] drm/ci: uprev mesa and modify gitlab rules Vignesh Raman
  2024-11-28  4:20 ` [PATCH v1 1/2] drm/ci: uprev mesa Vignesh Raman
  2024-11-28  4:20 ` [PATCH v1 2/2] drm/ci: update gitlab rules Vignesh Raman
@ 2024-12-03 14:12 ` Helen Mae Koike Fornazier
  2024-12-04  2:23   ` Vignesh Raman
  2 siblings, 1 reply; 10+ messages in thread
From: Helen Mae Koike Fornazier @ 2024-12-03 14:12 UTC (permalink / raw)
  To: Vignesh Raman
  Cc: dri-devel, daniels, airlied, daniel, robdclark, guilherme.gallo,
	sergi.blanch.torne, jani.nikula, dmitry.baryshkov, mripard,
	linux-kernel

Hi Vignesh,

Thanks for your patchset.



---- On Thu, 28 Nov 2024 01:20:19 -0300 Vignesh Raman  wrote ---

 > Uprev mesa to adapt to the latest changes in mesa-ci, 
 > including new container jobs and stages. Update the 
 > lava-submit script to align with recent mesa-ci changes 
 > for using LAVA rootfs overlays. Modify gitLab rules 
 > to include scheduled pipelines. 
 >  
 > Pipeline link, 
 > https://gitlab.freedesktop.org/vigneshraman/linux/-/pipelines/1318024 

I see this pipeline has a few failures, most of them are because outdated xfails files and timeouts.
Could we fix that first? So when we test this patchset we can verify if it introduces any issues or not.

Thanks
Helen

 >  
 > Vignesh Raman (2): 
 >  drm/ci: uprev mesa 
 >  drm/ci: update gitlab rules 
 >  
 >  drivers/gpu/drm/ci/build.sh       |   2 +- 
 >  drivers/gpu/drm/ci/build.yml      | 104 ++++++++++++++++- 
 >  drivers/gpu/drm/ci/container.yml  |  22 ++-- 
 >  drivers/gpu/drm/ci/gitlab-ci.yml  | 185 ++++++++++++++++-------------- 
 >  drivers/gpu/drm/ci/image-tags.yml |  11 +- 
 >  drivers/gpu/drm/ci/lava-submit.sh |  99 +++++++++++----- 
 >  drivers/gpu/drm/ci/test.yml       |  33 +++--- 
 >  7 files changed, 314 insertions(+), 142 deletions(-) 
 >  
 > -- 
 > 2.43.0 
 >  
 > 


^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: [PATCH v1 1/2] drm/ci: uprev mesa
  2024-11-28  4:20 ` [PATCH v1 1/2] drm/ci: uprev mesa Vignesh Raman
@ 2024-12-03 14:14   ` Helen Mae Koike Fornazier
  0 siblings, 0 replies; 10+ messages in thread
From: Helen Mae Koike Fornazier @ 2024-12-03 14:14 UTC (permalink / raw)
  To: Vignesh Raman
  Cc: dri-devel, daniels, airlied, daniel, robdclark, guilherme.gallo,
	sergi.blanch.torne, jani.nikula, dmitry.baryshkov, mripard,
	linux-kernel





---- On Thu, 28 Nov 2024 01:20:20 -0300 Vignesh Raman  wrote ---

 > Uprev mesa to adapt to the latest changes in mesa ci 
 > which includes new container jobs and stages. Also update 
 > lava-submit script to adapt to the recent changes in mesa 
 > to use LAVA rootfs overlays. 
 >  
 > Signed-off-by: Vignesh Raman vignesh.raman@collabora.com> 
 > --- 
 >  drivers/gpu/drm/ci/build.sh       |   2 +- 
 >  drivers/gpu/drm/ci/build.yml      | 103 +++++++++++++++++++++++++++++- 
 >  drivers/gpu/drm/ci/container.yml  |  22 +++---- 
 >  drivers/gpu/drm/ci/gitlab-ci.yml  |  71 ++++++++++++++------ 
 >  drivers/gpu/drm/ci/image-tags.yml |  11 +++- 
 >  drivers/gpu/drm/ci/lava-submit.sh |  99 ++++++++++++++++++++-------- 
 >  drivers/gpu/drm/ci/test.yml       |  13 ++-- 
 >  7 files changed, 255 insertions(+), 66 deletions(-) 
 >  
 > diff --git a/drivers/gpu/drm/ci/build.sh b/drivers/gpu/drm/ci/build.sh 
 > index 139b81db6312..19fe01257ab9 100644 
 > --- a/drivers/gpu/drm/ci/build.sh 
 > +++ b/drivers/gpu/drm/ci/build.sh 
 > @@ -132,7 +132,7 @@ fi 
 >  # Pass needed files to the test stage 
 >  mkdir -p install 
 >  cp -rfv .gitlab-ci/* install/. 
 > -cp -rfv ci/*  install/. 
 > +cp -rfv bin/ci/*  install/. 
 >  cp -rfv install/common install/ci-common 
 >  cp -rfv drivers/gpu/drm/ci/* install/. 
 >  
 > diff --git a/drivers/gpu/drm/ci/build.yml b/drivers/gpu/drm/ci/build.yml 
 > index 9c198239033d..771ecfc5008d 100644 
 > --- a/drivers/gpu/drm/ci/build.yml 
 > +++ b/drivers/gpu/drm/ci/build.yml 
 > @@ -2,7 +2,7 @@ 
 >  extends: 
 >  - .build-rules 
 >  - .container+build-rules 
 > -  stage: build 
 > +  stage: build-only 
 >  artifacts: 
 >  paths: 
 >  - artifacts 
 > @@ -110,3 +110,104 @@ build-nodebugfs:arm64: 
 >  
 >  build:x86_64: 
 >  extends: .build:x86_64 
 > + 
 > +# Disable build jobs that we won't use 
 > +alpine-build-testing: 
 > +  rules: 
 > +    - when: never 
 > + 
 > +debian-android: 
 > +  rules: 
 > +    - when: never 
 > + 
 > +debian-arm32: 
 > +  rules: 
 > +    - when: never 
 > + 
 > +debian-arm32-asan: 
 > +  rules: 
 > +    - when: never 
 > + 
 > +debian-arm64: 
 > +  rules: 
 > +    - when: never 
 > + 
 > +debian-arm64-asan: 
 > +  rules: 
 > +    - when: never 
 > + 
 > +debian-arm64-build-test: 
 > +  rules: 
 > +    - when: never 
 > + 
 > +debian-arm64-release: 
 > +  rules: 
 > +    - when: never 
 > + 
 > +debian-build-testing: 
 > +  rules: 
 > +    - when: never 
 > + 
 > +debian-clang: 
 > +  rules: 
 > +    - when: never 
 > + 
 > +debian-clang-release: 
 > +  rules: 
 > +    - when: never 
 > + 
 > +debian-no-libdrm: 
 > +  rules: 
 > +    - when: never 
 > + 
 > +debian-ppc64el: 
 > +  rules: 
 > +    - when: never 
 > + 
 > +debian-release: 
 > +  rules: 
 > +    - when: never 
 > + 
 > +debian-s390x: 
 > +  rules: 
 > +    - when: never 
 > + 
 > +debian-testing: 
 > +  rules: 
 > +    - when: never 
 > + 
 > +debian-testing-asan: 
 > +  rules: 
 > +    - when: never 
 > + 
 > +debian-testing-msan: 
 > +  rules: 
 > +    - when: never 
 > + 
 > +debian-vulkan: 
 > +  rules: 
 > +    - when: never 
 > + 
 > +debian-x86_32: 
 > +  rules: 
 > +    - when: never 
 > + 
 > +fedora-release: 
 > +  rules: 
 > +    - when: never 
 > + 
 > +rustfmt: 
 > +  rules: 
 > +    - when: never 
 > + 
 > +shader-db: 
 > +  rules: 
 > +    - when: never 
 > + 
 > +windows-msvc: 
 > +  rules: 
 > +    - when: never 
 > + 
 > +yaml-toml-shell-test: 
 > +  rules: 
 > +    - when: never 

Well, I don't really like this list, but I guess we don't have a choice for now...

 > diff --git a/drivers/gpu/drm/ci/container.yml b/drivers/gpu/drm/ci/container.yml 
 > index 2a94f54ce4cf..07dc13ff865d 100644 
 > --- a/drivers/gpu/drm/ci/container.yml 
 > +++ b/drivers/gpu/drm/ci/container.yml 
 > @@ -24,7 +24,7 @@ alpine/x86_64_build: 
 >  rules: 
 >  - when: never 
 >  
 > -debian/x86_64_test-vk: 
 > +debian/arm64_test-gl: 
 >  rules: 
 >  - when: never 
 >  
 > @@ -32,7 +32,15 @@ debian/arm64_test-vk: 
 >  rules: 
 >  - when: never 
 >  
 > -debian/arm64_test-gl: 
 > +debian/ppc64el_build: 
 > +  rules: 
 > +    - when: never 
 > + 
 > +debian/s390x_build: 
 > +  rules: 
 > +    - when: never 
 > + 
 > +debian/x86_64_test-vk: 
 >  rules: 
 >  - when: never 
 >  
 > @@ -56,14 +64,6 @@ windows_test_msvc: 
 >  rules: 
 >  - when: never 
 >  
 > -.debian/x86_64_build-mingw: 
 > -   rules: 
 > -    - when: never 
 > - 
 > -rustfmt: 
 > -   rules: 
 > -    - when: never 
 > - 
 >  windows_msvc: 
 >  rules: 
 > -    - when: never 
 > \ No newline at end of file 
 > +    - when: never 
 > diff --git a/drivers/gpu/drm/ci/gitlab-ci.yml b/drivers/gpu/drm/ci/gitlab-ci.yml 
 > index 90bde9f00cc3..86d8c5d8ce3b 100644 
 > --- a/drivers/gpu/drm/ci/gitlab-ci.yml 
 > +++ b/drivers/gpu/drm/ci/gitlab-ci.yml 
 > @@ -1,6 +1,6 @@ 
 >  variables: 
 >  DRM_CI_PROJECT_PATH: &drm-ci-project-path mesa/mesa 
 > -  DRM_CI_COMMIT_SHA: &drm-ci-commit-sha c6a9a9c3bce90923f7700219354e0b6e5a3c9ba6 
 > +  DRM_CI_COMMIT_SHA: &drm-ci-commit-sha e1a8fd80d411a5ff8fa19ffcf09516ac5099a25c 
 >  
 >  UPSTREAM_REPO: https://gitlab.freedesktop.org/drm/kernel.git 
 >  TARGET_BRANCH: drm-next 
 > @@ -31,11 +31,7 @@ variables: 
 >  PIPELINE_ARTIFACTS_BASE: ${S3_HOST}/${S3_ARTIFACTS_BUCKET}/${CI_PROJECT_PATH}/${CI_PIPELINE_ID} 
 >  # per-job artifact storage on MinIO 
 >  JOB_ARTIFACTS_BASE: ${PIPELINE_ARTIFACTS_BASE}/${CI_JOB_ID} 
 > -  # default kernel for rootfs before injecting the current kernel tree 
 > -  KERNEL_REPO: "gfx-ci/linux" 
 > -  KERNEL_TAG: "v6.6.21-mesa-f8ea" 
 >  KERNEL_IMAGE_BASE: https://${S3_HOST}/${S3_KERNEL_BUCKET}/${KERNEL_REPO}/${KERNEL_TAG} 
 > -  PKG_REPO_REV: "3cc12a2a" 
 >  LAVA_TAGS: subset-1-gfx 
 >  LAVA_JOB_PRIORITY: 30 
 >  ARTIFACTS_BASE_URL: https://${CI_PROJECT_ROOT_NAMESPACE}.${CI_PAGES_DOMAIN}/-/${CI_PROJECT_NAME}/-/jobs/${CI_JOB_ID}/artifacts 
 > @@ -59,7 +55,7 @@ default: 
 >  - cd $CI_PROJECT_DIR 
 >  - curl --output - $DRM_CI_PROJECT_URL/-/archive/$DRM_CI_COMMIT_SHA/mesa-$DRM_CI_COMMIT_SHA.tar.gz | tar -xz 
 >  - mv mesa-$DRM_CI_COMMIT_SHA/.gitlab-ci* . 
 > -    - mv mesa-$DRM_CI_COMMIT_SHA/bin/ci . 
 > +    - mv mesa-$DRM_CI_COMMIT_SHA/bin . 
 >  - rm -rf mesa-$DRM_CI_COMMIT_SHA/ 
 >  - echo -e "\e[0Ksection_end:$(date +%s):drm_ci_download_section\r\e[0K" 
 >  
 > @@ -85,6 +81,7 @@ include: 
 >  - project: *drm-ci-project-path 
 >  ref: *drm-ci-commit-sha 
 >  file: 
 > +      - '/.gitlab-ci/build/gitlab-ci.yml' 
 >  - '/.gitlab-ci/container/gitlab-ci.yml' 
 >  - '/.gitlab-ci/farm-rules.yml' 
 >  - '/.gitlab-ci/lava/lava-gitlab-ci.yml' 
 > @@ -115,9 +112,10 @@ include: 
 >  stages: 
 >  - sanity 
 >  - container 
 > -  - code-validation 
 >  - git-archive 
 > -  - build 
 > +  - build-for-tests 
 > +  - build-only 
 > +  - code-validation 
 >  - amdgpu 
 >  - i915 
 >  - mediatek 
 > @@ -264,30 +262,63 @@ sanity: 
 >  rules: 
 >  - if: *is-pre-merge 
 >  when: on_success 
 > -    # Other cases default to never 
 > +    - when: never 
 >  variables: 
 >  GIT_STRATEGY: none 
 >  script: 
 >  # ci-fairy check-commits --junit-xml=check-commits.xml 
 >  - ci-fairy check-merge-request --require-allow-collaboration --junit-xml=check-merge-request.xml 
 > +    - | 
 > +      set -eu 
 > +      image_tags=( 
 > +        ALPINE_X86_64_LAVA_SSH_TAG 
 > +        CONTAINER_TAG 
 > +        DEBIAN_BASE_TAG 
 > +        DEBIAN_BUILD_TAG 
 > +        DEBIAN_PYUTILS_TAG 
 > +        DEBIAN_TEST_GL_TAG 
 > +        KERNEL_ROOTFS_TAG 
 > +        KERNEL_TAG 
 > +        PKG_REPO_REV 
 > +      ) 
 > +      for var in "${image_tags[@]}" 
 > +      do 
 > +        if [ "$(echo -n "${!var}" | wc -c)" -gt 20 ] 
 > +        then 
 > +          echo "$var is too long; please make sure it is at most 20 chars." 
 > +          exit 1 
 > +        fi 
 > +      done 
 >  artifacts: 
 >  when: on_failure 
 >  reports: 
 >  junit: check-*.xml 
 > +  tags: 
 > +    - placeholder-job 
 >  
 > -# Rules for tests that should not block merging, but should be available to 
 > -# optionally run with the "play" button in the UI in pre-merge non-marge 
 > -# pipelines.  This should appear in "extends:" after any includes of 
 > -# test-source-dep.yml rules, so that these rules replace those. 
 > -.test-manual-mr: 
 > + 
 > +mr-label-maker-test: 
 > +  extends: 
 > +    - .fdo.ci-fairy 
 > +  stage: sanity 
 >  rules: 
 > -    - !reference [.no_scheduled_pipelines-rules, rules] 
 > -    - if: *is-forked-branch-or-pre-merge-not-for-marge 
 > -      when: manual 
 > +    - !reference [.mr-label-maker-rules, rules] 
 >  variables: 
 > -    JOB_TIMEOUT: 80 
 > - 
 > +    GIT_STRATEGY: fetch 
 > +  timeout: 10m 
 > +  script: 
 > +    - set -eu 
 > +    - python3 -m venv .venv 
 > +    - source .venv/bin/activate 
 > +    - pip install git+https://gitlab.freedesktop.org/freedesktop/mr-label-maker 
 > +    - mr-label-maker --dry-run --mr $CI_MERGE_REQUEST_IID 
 >  
 >  # Jobs that need to pass before spending hardware resources on further testing 
 >  .required-for-hardware-jobs: 
 > -  needs: [] 
 > +  needs: 
 > +    - job: clang-format 
 > +      optional: true 
 > +    - job: rustfmt 
 > +      optional: true 
 > +    - job: toml-lint 
 > +      optional: true 
 > diff --git a/drivers/gpu/drm/ci/image-tags.yml b/drivers/gpu/drm/ci/image-tags.yml 
 > index 8d8b9e71852e..528ef44285bb 100644 
 > --- a/drivers/gpu/drm/ci/image-tags.yml 
 > +++ b/drivers/gpu/drm/ci/image-tags.yml 
 > @@ -1,5 +1,5 @@ 
 >  variables: 
 > -   CONTAINER_TAG: "2024-09-09-uprevs" 
 > +   CONTAINER_TAG: "20241127-mesa-uprev" 
 >  DEBIAN_X86_64_BUILD_BASE_IMAGE: "debian/x86_64_build-base" 
 >  DEBIAN_BASE_TAG: "${CONTAINER_TAG}" 
 >  
 > @@ -7,9 +7,16 @@ variables: 
 >  DEBIAN_BUILD_TAG: "${CONTAINER_TAG}" 
 >  
 >  KERNEL_ROOTFS_TAG: "${CONTAINER_TAG}" 
 > +   # default kernel for rootfs before injecting the current kernel tree 
 > +   KERNEL_TAG: "v6.6.21-mesa-f8ea" 
 > +   KERNEL_REPO: "gfx-ci/linux" 
 > +   PKG_REPO_REV: "bca9635d" 
 >  
 >  DEBIAN_X86_64_TEST_BASE_IMAGE: "debian/x86_64_test-base" 
 >  DEBIAN_X86_64_TEST_IMAGE_GL_PATH: "debian/x86_64_test-gl" 
 >  DEBIAN_TEST_GL_TAG: "${CONTAINER_TAG}" 
 >  
 > -   ALPINE_X86_64_LAVA_SSH_TAG: "${CONTAINER_TAG}" 
 > \ No newline at end of file 
 > +   DEBIAN_PYUTILS_IMAGE: "debian/x86_64_pyutils" 
 > +   DEBIAN_PYUTILS_TAG: "${CONTAINER_TAG}" 
 > + 
 > +   ALPINE_X86_64_LAVA_SSH_TAG: "${CONTAINER_TAG}" 
 > diff --git a/drivers/gpu/drm/ci/lava-submit.sh b/drivers/gpu/drm/ci/lava-submit.sh 
 > index 6add15083c78..a1404a2d8cee 100755 
 > --- a/drivers/gpu/drm/ci/lava-submit.sh 
 > +++ b/drivers/gpu/drm/ci/lava-submit.sh 
 > @@ -1,58 +1,105 @@ 
 > -#!/bin/bash 
 > +#!/usr/bin/env bash 
 >  # SPDX-License-Identifier: MIT 
 > +# shellcheck disable=SC2086 # we want word splitting 
 > +# shellcheck disable=SC1091 # paths only become valid at runtime 
 >  
 > -set -e 
 > -set -x 
 > +# If we run in the fork (not from mesa or Marge-bot), reuse mainline kernel and rootfs, if exist. 
 > +_check_artifact_path() { 
 > +    _url="https://${1}/${2}" 
 > +    if curl -s -o /dev/null -I -L -f --retry 4 --retry-delay 15 "${_url}"; then 
 > +        echo -n "${_url}" 
 > +    fi 
 > +} 
 >  
 > -# Try to use the kernel and rootfs built in mainline first, so we're more 
 > -# likely to hit cache 
 > -if curl -L --retry 4 -f --retry-all-errors --retry-delay 60 -s "https://${BASE_SYSTEM_MAINLINE_HOST_PATH}/done"; then 
 > -    BASE_SYSTEM_HOST_PATH="${BASE_SYSTEM_MAINLINE_HOST_PATH}" 
 > -else 
 > -    BASE_SYSTEM_HOST_PATH="${BASE_SYSTEM_FORK_HOST_PATH}" 
 > -fi 
 > +get_path_to_artifact() { 
 > +    _mainline_artifact="$(_check_artifact_path ${BASE_SYSTEM_MAINLINE_HOST_PATH} ${1})" 
 > +    if [ -n "${_mainline_artifact}" ]; then 
 > +        echo -n "${_mainline_artifact}" 
 > +        return 
 > +    fi 
 > +    _fork_artifact="$(_check_artifact_path ${BASE_SYSTEM_FORK_HOST_PATH} ${1})" 
 > +    if [ -n "${_fork_artifact}" ]; then 
 > +        echo -n "${_fork_artifact}" 
 > +        return 
 > +    fi 
 > +    set +x 
 > +    error "Sorry, I couldn't find a viable built path for ${1} in either mainline or a fork." >&2 
 > +    echo "" >&2 
 > +    echo "If you're working on CI, this probably means that you're missing a dependency:" >&2 
 > +    echo "this job ran ahead of the job which was supposed to upload that artifact." >&2 
 > +    echo "" >&2 
 > +    echo "If you aren't working on CI, please ping @mesa/ci-helpers to see if we can help." >&2 
 > +    echo "" >&2 
 > +    echo "This job is going to fail, because I can't find the resources I need. Sorry." >&2 
 > +    set -x 
 > +    exit 1 
 > +} 
 > + 
 > +. "${SCRIPTS_DIR}/setup-test-env.sh" 
 > + 
 > +section_start prepare_rootfs "Preparing root filesystem" 
 > + 
 > +set -ex 
 > + 
 > +section_switch rootfs "Assembling root filesystem" 
 > +ROOTFS_URL="$(get_path_to_artifact lava-rootfs.tar.zst)" 
 > +[ $? != 1 ] || exit 1 
 >  
 >  rm -rf results 
 >  mkdir -p results/job-rootfs-overlay/ 
 >  
 > +artifacts/ci-common/generate-env.sh > results/job-rootfs-overlay/set-job-env-vars.sh 
 >  cp artifacts/ci-common/capture-devcoredump.sh results/job-rootfs-overlay/ 
 >  cp artifacts/ci-common/init-*.sh results/job-rootfs-overlay/ 
 >  cp artifacts/ci-common/intel-gpu-freq.sh results/job-rootfs-overlay/ 
 > +cp artifacts/ci-common/kdl.sh results/job-rootfs-overlay/ 
 >  cp "$SCRIPTS_DIR"/setup-test-env.sh results/job-rootfs-overlay/ 
 >  
 > -# Prepare env vars for upload. 
 > -section_start variables "Variables passed through:" 
 > -KERNEL_IMAGE_BASE="https://${BASE_SYSTEM_HOST_PATH}" \ 
 > -    artifacts/ci-common/generate-env.sh | tee results/job-rootfs-overlay/set-job-env-vars.sh 
 > -section_end variables 
 > - 
 >  tar zcf job-rootfs-overlay.tar.gz -C results/job-rootfs-overlay/ . 
 >  ci-fairy s3cp --token-file "${S3_JWT_FILE}" job-rootfs-overlay.tar.gz "https://${JOB_ROOTFS_OVERLAY_PATH}" 
 >  
 > +# Prepare env vars for upload. 
 > +section_switch variables "Environment variables passed through to device:" 
 > +cat results/job-rootfs-overlay/set-job-env-vars.sh 
 > + 
 > +section_switch lava_submit "Submitting job for scheduling" 
 > + 
 >  touch results/lava.log 
 >  tail -f results/lava.log & 
 > - 
 >  PYTHONPATH=artifacts/ artifacts/lava/lava_job_submitter.py \ 
 > -    submit \ 
 > +    --farm "${FARM}" \ 
 > +    --device-type "${DEVICE_TYPE}" \ 
 > +    --boot-method "${BOOT_METHOD}" \ 
 > +    --job-timeout-min ${JOB_TIMEOUT:-30} \ 
 >      --dump-yaml \ 
 >      --pipeline-info "$CI_JOB_NAME: $CI_PIPELINE_URL on $CI_COMMIT_REF_NAME ${CI_NODE_INDEX}/${CI_NODE_TOTAL}" \ 
 > -    --rootfs-url-prefix "https://${BASE_SYSTEM_HOST_PATH}" \ 
 > +    --rootfs-url "${ROOTFS_URL}" \ 
 >      --kernel-url-prefix "https://${PIPELINE_ARTIFACTS_BASE}/${DEBIAN_ARCH}" \ 
 > -    --build-url "${FDO_HTTP_CACHE_URI:-}https://${PIPELINE_ARTIFACTS_BASE}/${DEBIAN_ARCH}/kernel-files.tar.zst" \ 
 > -    --job-rootfs-overlay-url "${FDO_HTTP_CACHE_URI:-}https://${JOB_ROOTFS_OVERLAY_PATH}" \ 
 > -    --job-timeout-min ${JOB_TIMEOUT:-80} \ 
 > +    --kernel-external "${EXTERNAL_KERNEL_TAG}" \ 
 >      --first-stage-init artifacts/ci-common/init-stage1.sh \ 
 > -    --ci-project-dir "${CI_PROJECT_DIR}" \ 
 > -    --device-type "${DEVICE_TYPE}" \ 
 > -    --farm "${FARM}" \ 
 >      --dtb-filename "${DTB}" \ 
 >      --jwt-file "${S3_JWT_FILE}" \ 
 >      --kernel-image-name "${KERNEL_IMAGE_NAME}" \ 
 >      --kernel-image-type "${KERNEL_IMAGE_TYPE}" \ 
 > -    --boot-method "${BOOT_METHOD}" \ 
 >      --visibility-group "${VISIBILITY_GROUP}" \ 
 >      --lava-tags "${LAVA_TAGS}" \ 
 >      --mesa-job-name "$CI_JOB_NAME" \ 
 >      --structured-log-file "results/lava_job_detail.json" \ 
 >      --ssh-client-image "${LAVA_SSH_CLIENT_IMAGE}" \ 
 > +    --project-name "${CI_PROJECT_NAME}" \ 
 > +    --starting-section "${CURRENT_SECTION}" \ 
 > +    --job-submitted-at "${CI_JOB_STARTED_AT}" \ 
 > +    - append-overlay \ 
 > +        --name=kernel-build \ 
 > +        --url="${FDO_HTTP_CACHE_URI:-}https://${PIPELINE_ARTIFACTS_BASE}/${DEBIAN_ARCH}/kernel-files.tar.zst" \ 
 > +        --compression=zstd \ 
 > +        --path="${CI_PROJECT_DIR}" \ 
 > +        --format=tar \ 
 > +    - append-overlay \ 
 > +        --name=job-overlay \ 
 > +        --url="https://${JOB_ROOTFS_OVERLAY_PATH}" \ 
 > +        --compression=gz \ 
 > +        --path="/" \ 
 > +        --format=tar \ 
 > +    - submit \ 
 >      >> results/lava.log 
 > diff --git a/drivers/gpu/drm/ci/test.yml b/drivers/gpu/drm/ci/test.yml 
 > index f0ef60c8f56d..42a8e7f93714 100644 
 > --- a/drivers/gpu/drm/ci/test.yml 
 > +++ b/drivers/gpu/drm/ci/test.yml 
 > @@ -1,21 +1,21 @@ 
 >  .test-rules: 
 >  rules: 
 > -    - if: '$FD_FARM == "offline" && $RUNNER_TAG =~ /^google-freedreno-/' 
 > -      when: never 
 > -    - if: '$COLLABORA_FARM == "offline" && $RUNNER_TAG =~ /^mesa-ci-x86-64-lava-/' 
 > -      when: never 
 >  - !reference [.no_scheduled_pipelines-rules, rules] 
 > +    - !reference [.collabora-farm-rules, rules] 
 > +    - !reference [.google-freedreno-farm-rules, rules] 
 >  - when: on_success 
 >  
 >  .lava-test: 
 >  extends: 
 >  - .test-rules 
 > +    - .build-rules 
 > +    - .container+build-rules 
 >  timeout: "1h30m" 
 >  script: 
 >  # Note: Build dir (and thus install) may be dirty due to GIT_STRATEGY 
 >  - rm -rf install 
 >  - tar -xf artifacts/install.tar 
 > -    - mv install/* artifacts/. 
 > +    - mv -n install/* artifacts/. 
 >  # Override it with our lava-submit.sh script 
 >  - ./artifacts/lava-submit.sh 
 >  
 > @@ -32,6 +32,7 @@ 
 >  - alpine/x86_64_lava_ssh_client 
 >  - kernel+rootfs_arm32 
 >  - debian/x86_64_build 
 > +    - python-test 
 >  - testing:arm32 
 >  - igt:arm32 
 >  
 > @@ -48,6 +49,7 @@ 
 >  - alpine/x86_64_lava_ssh_client 
 >  - kernel+rootfs_arm64 
 >  - debian/x86_64_build 
 > +    - python-test 
 >  - testing:arm64 
 >  - igt:arm64 
 >  
 > @@ -64,6 +66,7 @@ 
 >  - alpine/x86_64_lava_ssh_client 
 >  - kernel+rootfs_x86_64 
 >  - debian/x86_64_build 
 > +    - python-test 
 >  - testing:x86_64 
 >  - igt:x86_64 
 >  
 > -- 
 > 2.43.0 
 >  
 > 

lgtm
Acked-by: Helen Koike <helen.koike@collabora.com>

Thanks
Helen




^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: [PATCH v1 0/2] drm/ci: uprev mesa and modify gitlab rules
  2024-12-03 14:12 ` [PATCH v1 0/2] drm/ci: uprev mesa and modify " Helen Mae Koike Fornazier
@ 2024-12-04  2:23   ` Vignesh Raman
  0 siblings, 0 replies; 10+ messages in thread
From: Vignesh Raman @ 2024-12-04  2:23 UTC (permalink / raw)
  To: Helen Mae Koike Fornazier
  Cc: dri-devel, daniels, airlied, daniel, robdclark, guilherme.gallo,
	sergi.blanch.torne, jani.nikula, dmitry.baryshkov, mripard,
	linux-kernel

Hi Helen,

On 03/12/24 19:42, Helen Mae Koike Fornazier wrote:
> Hi Vignesh,
> 
> Thanks for your patchset.
> 
> 
> 
> ---- On Thu, 28 Nov 2024 01:20:19 -0300 Vignesh Raman  wrote ---
> 
>   > Uprev mesa to adapt to the latest changes in mesa-ci,
>   > including new container jobs and stages. Update the
>   > lava-submit script to align with recent mesa-ci changes
>   > for using LAVA rootfs overlays. Modify gitLab rules
>   > to include scheduled pipelines.
>   >
>   > Pipeline link,
>   > https://gitlab.freedesktop.org/vigneshraman/linux/-/pipelines/1318024
> 
> I see this pipeline has a few failures, most of them are because outdated xfails files and timeouts.
> Could we fix that first? So when we test this patchset we can verify if it introduces any issues or not.

Yes. The expectation files are not updated. I will uprev IGT and rerun 
the tests and update xfails.

Regards,
Vignesh

> 
> Thanks
> Helen
> 
>   >
>   > Vignesh Raman (2):
>   >  drm/ci: uprev mesa
>   >  drm/ci: update gitlab rules
>   >
>   >  drivers/gpu/drm/ci/build.sh       |   2 +-
>   >  drivers/gpu/drm/ci/build.yml      | 104 ++++++++++++++++-
>   >  drivers/gpu/drm/ci/container.yml  |  22 ++--
>   >  drivers/gpu/drm/ci/gitlab-ci.yml  | 185 ++++++++++++++++--------------
>   >  drivers/gpu/drm/ci/image-tags.yml |  11 +-
>   >  drivers/gpu/drm/ci/lava-submit.sh |  99 +++++++++++-----
>   >  drivers/gpu/drm/ci/test.yml       |  33 +++---
>   >  7 files changed, 314 insertions(+), 142 deletions(-)
>   >
>   > --
>   > 2.43.0
>   >
>   >
> 

^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: [PATCH v1 1/2] drm/ci: uprev mesa
  2025-11-15  3:55     ` Dmitry Baryshkov
@ 2025-11-16 15:32       ` Dmitry Baryshkov
  0 siblings, 0 replies; 10+ messages in thread
From: Dmitry Baryshkov @ 2025-11-16 15:32 UTC (permalink / raw)
  To: Vignesh Raman
  Cc: dri-devel, daniels, helen.fornazier, airlied, simona.vetter,
	lumag, robdclark, robin.clark, guilherme.gallo,
	sergi.blanch.torne, valentine.burley, linux-mediatek,
	linux-amlogic, linux-rockchip, amd-gfx, linux-arm-msm, intel-gfx,
	virtualization, linux-kernel

On Sat, Nov 15, 2025 at 05:55:17AM +0200, Dmitry Baryshkov wrote:
> On Sat, Nov 15, 2025 at 02:44:56AM +0200, Dmitry Baryshkov wrote:
> > On Fri, Nov 14, 2025 at 08:30:50AM +0530, Vignesh Raman wrote:
> > > Uprev mesa to adapt to the latest changes in Mesa CI, including support
> > > for firmware via LAVA overlays, removal of the python-artifacts job,
> > > split container and build rules, use lava-job-submitter container,
> > > and various misc fixes.
> > > 
> > > Co-developed-by: Dmitry Baryshkov <dmitry.baryshkov@oss.qualcomm.com>
> > > Signed-off-by: Dmitry Baryshkov <dmitry.baryshkov@oss.qualcomm.com>
> > > Signed-off-by: Vignesh Raman <vignesh.raman@collabora.com>
> > > ---
> > >  drivers/gpu/drm/ci/build.yml         |  22 +++---
> > >  drivers/gpu/drm/ci/container.yml     |  28 ++++++--
> > >  drivers/gpu/drm/ci/gitlab-ci.yml     |  95 ++++++++++++++++++++-----
> > >  drivers/gpu/drm/ci/igt_runner.sh     |   4 +-
> > >  drivers/gpu/drm/ci/image-tags.yml    |  22 +++---
> > >  drivers/gpu/drm/ci/lava-submit.sh    | 101 +++++++++++++--------------
> > >  drivers/gpu/drm/ci/static-checks.yml |   1 +
> > >  drivers/gpu/drm/ci/test.yml          |  18 +++--
> > >  8 files changed, 186 insertions(+), 105 deletions(-)
> > > 
> > 
> > Thanks a lot for taking care of it!
> > 
> > 
> > Reviewed-by: Dmitry Baryshkov <dmitry.baryshkov@oss.qualcomm.com>
> 
> Hmm, this causes a lot of failures, see [1] (note, the run is not
> finished yet):
> - On some platforms (msm, rk3399, mt8173) the
>   core_setmaster@master-drop-set-root test fails. On other it seems to
>   work correctly
> - sm8350 seems to have some issues with this uprev, it cant' finish the
>   jobs at all
> - mt8173 and rockchip:rk3399 seem to have other test failures too

I have fixed most of the fallouts, see [2]. But ideally this should be
split into two parts: fix before mesa uprev (I think some of them are
already there), uprev mesa, applying necessary fixes and then enable apq
jobs.

At this point I have no idea what's wrong with MT8183. Most likely it
needs either a DT fix or a driver fix. The driver fails to probe because
it can't build a sensible component chain.

On i915 several tests kill the hardware, making it timeout all remaining
tests in a bad way.

BTW: I see that for some of the drivers we skip the absolute majority of
the tests. What are we actually testing on those?

> 
> Could you please update fails / flakes tor all the affected platforms?
> Also I'm really interested in the master-drop-set tests. Why do they
> fail on some of the platforms, but not on all? This seems to be some
> infrastructure-related issue, since exactly the same kernel with the
> same IGT passes those tests (at least on msm hardware).
> 
> [1] https://gitlab.freedesktop.org/drm/msm/-/pipelines/1547684

[2] https://gitlab.freedesktop.org/lumag/msm/-/commits/msm-next-lumag-db820c


-- 
With best wishes
Dmitry

^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: [PATCH v1 1/2] drm/ci: uprev mesa
  2025-11-15  0:44   ` Dmitry Baryshkov
@ 2025-11-15  3:55     ` Dmitry Baryshkov
  2025-11-16 15:32       ` Dmitry Baryshkov
  0 siblings, 1 reply; 10+ messages in thread
From: Dmitry Baryshkov @ 2025-11-15  3:55 UTC (permalink / raw)
  To: Vignesh Raman
  Cc: dri-devel, daniels, helen.fornazier, airlied, simona.vetter,
	lumag, robdclark, robin.clark, guilherme.gallo,
	sergi.blanch.torne, valentine.burley, linux-mediatek,
	linux-amlogic, linux-rockchip, amd-gfx, linux-arm-msm, intel-gfx,
	virtualization, linux-kernel

On Sat, Nov 15, 2025 at 02:44:56AM +0200, Dmitry Baryshkov wrote:
> On Fri, Nov 14, 2025 at 08:30:50AM +0530, Vignesh Raman wrote:
> > Uprev mesa to adapt to the latest changes in Mesa CI, including support
> > for firmware via LAVA overlays, removal of the python-artifacts job,
> > split container and build rules, use lava-job-submitter container,
> > and various misc fixes.
> > 
> > Co-developed-by: Dmitry Baryshkov <dmitry.baryshkov@oss.qualcomm.com>
> > Signed-off-by: Dmitry Baryshkov <dmitry.baryshkov@oss.qualcomm.com>
> > Signed-off-by: Vignesh Raman <vignesh.raman@collabora.com>
> > ---
> >  drivers/gpu/drm/ci/build.yml         |  22 +++---
> >  drivers/gpu/drm/ci/container.yml     |  28 ++++++--
> >  drivers/gpu/drm/ci/gitlab-ci.yml     |  95 ++++++++++++++++++++-----
> >  drivers/gpu/drm/ci/igt_runner.sh     |   4 +-
> >  drivers/gpu/drm/ci/image-tags.yml    |  22 +++---
> >  drivers/gpu/drm/ci/lava-submit.sh    | 101 +++++++++++++--------------
> >  drivers/gpu/drm/ci/static-checks.yml |   1 +
> >  drivers/gpu/drm/ci/test.yml          |  18 +++--
> >  8 files changed, 186 insertions(+), 105 deletions(-)
> > 
> 
> Thanks a lot for taking care of it!
> 
> 
> Reviewed-by: Dmitry Baryshkov <dmitry.baryshkov@oss.qualcomm.com>

Hmm, this causes a lot of failures, see [1] (note, the run is not
finished yet):
- On some platforms (msm, rk3399, mt8173) the
  core_setmaster@master-drop-set-root test fails. On other it seems to
  work correctly
- sm8350 seems to have some issues with this uprev, it cant' finish the
  jobs at all
- mt8173 and rockchip:rk3399 seem to have other test failures too

Could you please update fails / flakes tor all the affected platforms?
Also I'm really interested in the master-drop-set tests. Why do they
fail on some of the platforms, but not on all? This seems to be some
infrastructure-related issue, since exactly the same kernel with the
same IGT passes those tests (at least on msm hardware).

[1] https://gitlab.freedesktop.org/drm/msm/-/pipelines/1547684

-- 
With best wishes
Dmitry

^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: [PATCH v1 1/2] drm/ci: uprev mesa
  2025-11-14  3:00 ` [PATCH v1 1/2] drm/ci: uprev mesa Vignesh Raman
@ 2025-11-15  0:44   ` Dmitry Baryshkov
  2025-11-15  3:55     ` Dmitry Baryshkov
  0 siblings, 1 reply; 10+ messages in thread
From: Dmitry Baryshkov @ 2025-11-15  0:44 UTC (permalink / raw)
  To: Vignesh Raman
  Cc: dri-devel, daniels, helen.fornazier, airlied, simona.vetter,
	lumag, robdclark, robin.clark, guilherme.gallo,
	sergi.blanch.torne, valentine.burley, linux-mediatek,
	linux-amlogic, linux-rockchip, amd-gfx, linux-arm-msm, intel-gfx,
	virtualization, linux-kernel

On Fri, Nov 14, 2025 at 08:30:50AM +0530, Vignesh Raman wrote:
> Uprev mesa to adapt to the latest changes in Mesa CI, including support
> for firmware via LAVA overlays, removal of the python-artifacts job,
> split container and build rules, use lava-job-submitter container,
> and various misc fixes.
> 
> Co-developed-by: Dmitry Baryshkov <dmitry.baryshkov@oss.qualcomm.com>
> Signed-off-by: Dmitry Baryshkov <dmitry.baryshkov@oss.qualcomm.com>
> Signed-off-by: Vignesh Raman <vignesh.raman@collabora.com>
> ---
>  drivers/gpu/drm/ci/build.yml         |  22 +++---
>  drivers/gpu/drm/ci/container.yml     |  28 ++++++--
>  drivers/gpu/drm/ci/gitlab-ci.yml     |  95 ++++++++++++++++++++-----
>  drivers/gpu/drm/ci/igt_runner.sh     |   4 +-
>  drivers/gpu/drm/ci/image-tags.yml    |  22 +++---
>  drivers/gpu/drm/ci/lava-submit.sh    | 101 +++++++++++++--------------
>  drivers/gpu/drm/ci/static-checks.yml |   1 +
>  drivers/gpu/drm/ci/test.yml          |  18 +++--
>  8 files changed, 186 insertions(+), 105 deletions(-)
> 

Thanks a lot for taking care of it!


Reviewed-by: Dmitry Baryshkov <dmitry.baryshkov@oss.qualcomm.com>



-- 
With best wishes
Dmitry

^ permalink raw reply	[flat|nested] 10+ messages in thread

* [PATCH v1 1/2] drm/ci: uprev mesa
  2025-11-14  3:00 [PATCH v1 0/2] drm/ci: uprev IGT and enable apq8016, apq8096 Vignesh Raman
@ 2025-11-14  3:00 ` Vignesh Raman
  2025-11-15  0:44   ` Dmitry Baryshkov
  0 siblings, 1 reply; 10+ messages in thread
From: Vignesh Raman @ 2025-11-14  3:00 UTC (permalink / raw)
  To: dri-devel
  Cc: daniels, helen.fornazier, airlied, simona.vetter,
	dmitry.baryshkov, lumag, robdclark, robin.clark, guilherme.gallo,
	sergi.blanch.torne, valentine.burley, linux-mediatek,
	linux-amlogic, linux-rockchip, amd-gfx, linux-arm-msm, intel-gfx,
	virtualization, linux-kernel

Uprev mesa to adapt to the latest changes in Mesa CI, including support
for firmware via LAVA overlays, removal of the python-artifacts job,
split container and build rules, use lava-job-submitter container,
and various misc fixes.

Co-developed-by: Dmitry Baryshkov <dmitry.baryshkov@oss.qualcomm.com>
Signed-off-by: Dmitry Baryshkov <dmitry.baryshkov@oss.qualcomm.com>
Signed-off-by: Vignesh Raman <vignesh.raman@collabora.com>
---
 drivers/gpu/drm/ci/build.yml         |  22 +++---
 drivers/gpu/drm/ci/container.yml     |  28 ++++++--
 drivers/gpu/drm/ci/gitlab-ci.yml     |  95 ++++++++++++++++++++-----
 drivers/gpu/drm/ci/igt_runner.sh     |   4 +-
 drivers/gpu/drm/ci/image-tags.yml    |  22 +++---
 drivers/gpu/drm/ci/lava-submit.sh    | 101 +++++++++++++--------------
 drivers/gpu/drm/ci/static-checks.yml |   1 +
 drivers/gpu/drm/ci/test.yml          |  18 +++--
 8 files changed, 186 insertions(+), 105 deletions(-)

diff --git a/drivers/gpu/drm/ci/build.yml b/drivers/gpu/drm/ci/build.yml
index af27ff5de369..efbcaae49807 100644
--- a/drivers/gpu/drm/ci/build.yml
+++ b/drivers/gpu/drm/ci/build.yml
@@ -1,6 +1,6 @@
 .build:
   extends:
-    - .container+build-rules
+    - .build-rules
   stage: build-only
   artifacts:
     paths:
@@ -133,6 +133,10 @@ debian-arm32-asan:
   rules:
     - when: never
 
+debian-x86_64-msan:
+  rules:
+    - when: never
+
 debian-arm64:
   rules:
     - when: never
@@ -153,7 +157,7 @@ debian-arm64-ubsan:
   rules:
     - when: never
 
-debian-build-testing:
+debian-build-x86_64:
   rules:
     - when: never
 
@@ -177,31 +181,31 @@ debian-release:
   rules:
     - when: never
 
-debian-s390x:
+debian-riscv64:
   rules:
     - when: never
 
-debian-testing:
+debian-s390x:
   rules:
     - when: never
 
-debian-testing-asan:
+debian-vulkan:
   rules:
     - when: never
 
-debian-testing-msan:
+debian-x86_32:
   rules:
     - when: never
 
-debian-testing-ubsan:
+debian-x86_64:
   rules:
     - when: never
 
-debian-vulkan:
+debian-x86_64-asan:
   rules:
     - when: never
 
-debian-x86_32:
+debian-x86_64-ubsan:
   rules:
     - when: never
 
diff --git a/drivers/gpu/drm/ci/container.yml b/drivers/gpu/drm/ci/container.yml
index 5f90508578a3..ce6007a45a07 100644
--- a/drivers/gpu/drm/ci/container.yml
+++ b/drivers/gpu/drm/ci/container.yml
@@ -5,21 +5,29 @@
 
 debian/x86_64_build-base:
   variables:
-    EXTRA_LOCAL_PACKAGES: "libcairo-dev libdw-dev libjson-c-dev libkmod2 libkmod-dev libpciaccess-dev libproc2-dev libudev-dev libunwind-dev python3-docutils bc python3-ply libssl-dev bc"
+    EXTRA_LOCAL_PACKAGES: "libcairo-dev libdw-dev libjson-c-dev libkmod-dev libpciaccess-dev libproc2-dev libudev-dev libunwind-dev python3-docutils bc python3-ply libssl-dev bc"
+
+debian/arm64_build:
+  variables:
+    EXTRA_LOCAL_PACKAGES: "libcairo-dev libdw-dev libjson-c-dev libproc2-dev libkmod-dev libpciaccess-dev libudev-dev libunwind-dev python3-docutils libssl-dev crossbuild-essential-armhf libkmod-dev:armhf libproc2-dev:armhf libunwind-dev:armhf libdw-dev:armhf libpixman-1-dev:armhf libcairo-dev:armhf libudev-dev:armhf libjson-c-dev:armhf"
 
 debian/x86_64_test-gl:
   variables:
-    EXTRA_LOCAL_PACKAGES: "jq libasound2 libcairo2 libdw1 libglib2.0-0 libjson-c5 libkmod-dev libkmod2 libgles2 libproc2-dev"
+    EXTRA_LOCAL_PACKAGES: "jq libasound2t64 libcairo2 libdw1t64 libglib2.0-0t64 libjson-c5 libkmod2 libgles2 libdrm-nouveau2 libdrm-amdgpu1"
 
-debian/arm64_build:
+debian/arm64_test-gl:
   variables:
-    EXTRA_LOCAL_PACKAGES: "libcairo-dev libdw-dev libjson-c-dev libproc2-dev libkmod2 libkmod-dev libpciaccess-dev libudev-dev libunwind-dev python3-docutils libssl-dev crossbuild-essential-armhf libkmod-dev:armhf libproc2-dev:armhf libunwind-dev:armhf libdw-dev:armhf libpixman-1-dev:armhf libcairo-dev:armhf libudev-dev:armhf libjson-c-dev:armhf"
+    EXTRA_LOCAL_PACKAGES: "jq libasound2t64 libcairo2 libdw1t64 libglib2.0-0t64 libjson-c5 libkmod2 libgles2 libdrm-nouveau2 libdrm-amdgpu1"
 
-.kernel+rootfs:
+debian/arm32_test-gl:
   variables:
-    EXTRA_LOCAL_PACKAGES: "jq libasound2 libcairo2 libdw1 libglib2.0-0 libjson-c5"
+    EXTRA_LOCAL_PACKAGES: "jq libasound2t64 libcairo2 libdw1t64 libglib2.0-0t64 libjson-c5 libkmod2 libgles2 libdrm-nouveau2 libdrm-amdgpu1 libunwind8"
 
 # Disable container jobs that we won't use
+alpine/x86_64_build:
+  rules:
+    - when: never
+
 debian/arm64_test-vk:
   rules:
     - when: never
@@ -28,6 +36,10 @@ debian/baremetal_arm32_test-gl:
   rules:
     - when: never
 
+debian/baremetal_arm64_test-gl:
+  rules:
+    - when: never
+
 debian/baremetal_arm64_test-vk:
   rules:
     - when: never
@@ -36,6 +48,10 @@ debian/ppc64el_build:
   rules:
     - when: never
 
+debian/riscv64_build:
+  rules:
+    - when: never
+
 debian/s390x_build:
   rules:
     - when: never
diff --git a/drivers/gpu/drm/ci/gitlab-ci.yml b/drivers/gpu/drm/ci/gitlab-ci.yml
index 56638814bb28..e23a9110008c 100644
--- a/drivers/gpu/drm/ci/gitlab-ci.yml
+++ b/drivers/gpu/drm/ci/gitlab-ci.yml
@@ -1,6 +1,6 @@
 variables:
   DRM_CI_PROJECT_PATH: &drm-ci-project-path mesa/mesa
-  DRM_CI_COMMIT_SHA: &drm-ci-commit-sha 02337aec715c25dae7ff2479d986f831c77fe536
+  DRM_CI_COMMIT_SHA: &drm-ci-commit-sha f07dab189928ee65865ef9cefcaa4eff05da99b5
 
   UPSTREAM_REPO: https://gitlab.freedesktop.org/drm/kernel.git
   TARGET_BRANCH: drm-next
@@ -11,7 +11,7 @@ variables:
   DEQP_RUNNER_GIT_TAG: v0.20.0
 
   FDO_UPSTREAM_REPO: helen.fornazier/linux   # The repo where the git-archive daily runs
-  MESA_TEMPLATES_COMMIT: &ci-templates-commit c6aeb16f86e32525fa630fb99c66c4f3e62fc3cb
+  MESA_TEMPLATES_COMMIT: &ci-templates-commit aec7a6ce7bb38902c70641526f6611e27141784a
   DRM_CI_PROJECT_URL: https://gitlab.freedesktop.org/${DRM_CI_PROJECT_PATH}
   CI_PRE_CLONE_SCRIPT: |-
           set -o xtrace
@@ -30,6 +30,8 @@ variables:
   S3_GITCACHE_BUCKET: git-cache
   # Bucket for the pipeline artifacts pushed to S3
   S3_ARTIFACTS_BUCKET: artifacts
+  # Base path used for various artifacts
+  S3_BASE_PATH: "${S3_HOST}/${S3_KERNEL_BUCKET}"
   # per-pipeline artifact storage on MinIO
   PIPELINE_ARTIFACTS_BASE: ${S3_HOST}/${S3_ARTIFACTS_BUCKET}/${CI_PROJECT_PATH}/${CI_PIPELINE_ID}
   # per-job artifact storage on MinIO
@@ -44,6 +46,8 @@ variables:
   ARTIFACTS_BASE_URL: https://${CI_PROJECT_ROOT_NAMESPACE}.${CI_PAGES_DOMAIN}/-/${CI_PROJECT_NAME}/-/jobs/${CI_JOB_ID}/artifacts
   # Python scripts for structured logger
   PYTHONPATH: "$PYTHONPATH:$CI_PROJECT_DIR/install"
+  # Mesa-specific variables that shouldn't be forwarded to DUTs and crosvm
+  CI_EXCLUDE_ENV_VAR_REGEX: 'SCRIPTS_DIR|RESULTS_DIR'
 
 
 default:
@@ -84,10 +88,11 @@ include:
   - project: *drm-ci-project-path
     ref: *drm-ci-commit-sha
     file:
+      - '/.gitlab-ci/bare-metal/gitlab-ci.yml'
       - '/.gitlab-ci/build/gitlab-ci.yml'
       - '/.gitlab-ci/container/gitlab-ci.yml'
       - '/.gitlab-ci/farm-rules.yml'
-      - '/.gitlab-ci/lava/lava-gitlab-ci.yml'
+      - '/.gitlab-ci/lava/gitlab-ci.yml'
       - '/.gitlab-ci/test-source-dep.yml'
       - '/.gitlab-ci/test/gitlab-ci.yml'
       - '/src/amd/ci/gitlab-ci-inc.yml'
@@ -147,10 +152,10 @@ stages:
     - if: &is-merge-attempt $GITLAB_USER_LOGIN == "marge-bot" && $CI_PIPELINE_SOURCE == "merge_request_event"
     # post-merge pipeline
     - if: &is-post-merge $GITLAB_USER_LOGIN == "marge-bot" && $CI_PIPELINE_SOURCE == "push"
-    # Pre-merge pipeline
-    - if: &is-pre-merge $CI_PIPELINE_SOURCE == "merge_request_event"
+    # Pre-merge pipeline (because merge pipelines are already caught above)
+    - if: &is-merge-request $CI_PIPELINE_SOURCE == "merge_request_event"
     # Push to a branch on a fork
-    - if: &is-fork-push $CI_PIPELINE_SOURCE == "push"
+    - if: &is-push-to-fork $CI_PIPELINE_SOURCE == "push"
     # nightly pipeline
     - if: &is-scheduled-pipeline $CI_PIPELINE_SOURCE == "schedule"
     # pipeline for direct pushes that bypassed the CI
@@ -160,17 +165,59 @@ stages:
 # Rules applied to every job in the pipeline
 .common-rules:
   rules:
-    - if: *is-fork-push
+    - if: *is-push-to-fork
       when: manual
 
-
 .never-post-merge-rules:
   rules:
     - if: *is-post-merge
       when: never
 
 
-.container+build-rules:
+# Note: make sure the branches in this list are the same as in
+# `.build-only-delayed-rules` below.
+.container-rules:
+  rules:
+    - !reference [.common-rules, rules]
+    # Run when re-enabling a disabled farm, but not when disabling it
+    - !reference [.disable-farm-mr-rules, rules]
+    # Never run immediately after merging, as we just ran everything
+    - !reference [.never-post-merge-rules, rules]
+    # Only rebuild containers in merge pipelines if any tags have been
+    # changed, else we'll just use the already-built containers
+    - if: *is-merge-attempt
+      changes: &image_tags_path
+        - drivers/gpu/drm/ci/image-tags.yml
+      when: on_success
+    # Skip everything for pre-merge and merge pipelines which don't change
+    # anything in the build; we only do this for marge-bot and not user
+    # pipelines in a MR, because we might still need to run it to copy the
+    # container into the user's namespace.
+    - if: *is-merge-attempt
+      when: never
+    # Any MR pipeline which changes image-tags.yml needs to be able to
+    # rebuild the containers
+    - if: *is-merge-request
+      changes: *image_tags_path
+      when: manual
+    # ... however for MRs running inside the user namespace, we may need to
+    # run these jobs to copy the container images from upstream
+    - if: *is-merge-request
+      when: manual
+    # Build everything after someone bypassed the CI
+    - if: *is-direct-push
+      when: manual
+    # Scheduled pipelines reuse already-built containers
+    - if: *is-scheduled-pipeline
+      when: never
+    # Allow building everything in fork pipelines, but build nothing unless
+    # manually triggered
+    - when: manual
+
+
+# Note: make sure the branches in this list are the same as in
+# `.build-only-delayed-rules` below.
+.build-rules:
   rules:
     - !reference [.common-rules, rules]
     # Run when re-enabling a disabled farm, but not when disabling it
@@ -181,7 +228,7 @@ stages:
     - if: *is-merge-attempt
       when: on_success
     # Same as above, but for pre-merge pipelines
-    - if: *is-pre-merge
+    - if: *is-merge-request
       when: manual
     # Build everything after someone bypassed the CI
     - if: *is-direct-push
@@ -197,7 +244,7 @@ stages:
 # Repeat of the above but with `when: on_success` replaced with
 # `when: delayed` + `start_in:`, for build-only jobs.
 # Note: make sure the branches in this list are the same as in
-# `.container+build-rules` above.
+# `.build-rules` above.
 .build-only-delayed-rules:
   rules:
     - !reference [.common-rules, rules]
@@ -210,7 +257,7 @@ stages:
       when: delayed
       start_in: &build-delay 5 minutes
     # Same as above, but for pre-merge pipelines
-    - if: *is-pre-merge
+    - if: *is-merge-request
       when: manual
     # Build everything after someone bypassed the CI
     - if: *is-direct-push
@@ -237,11 +284,6 @@ stages:
       - _build/meson-logs/strace
 
 
-python-artifacts:
-  variables:
-    GIT_DEPTH: 10
-
-
 # Git archive
 make-git-archive:
   extends:
@@ -273,7 +315,7 @@ sanity:
   tags:
     - $FDO_RUNNER_JOB_PRIORITY_TAG_X86_64
   rules:
-    - if: *is-pre-merge
+    - if: *is-merge-request
       when: on_success
     - when: never
   variables:
@@ -284,7 +326,6 @@ sanity:
     - |
       set -eu
       image_tags=(
-        ALPINE_X86_64_LAVA_SSH_TAG
         CONTAINER_TAG
         DEBIAN_BASE_TAG
         DEBIAN_BUILD_TAG
@@ -347,3 +388,19 @@ linkcheck-docs:
 test-docs:
    rules:
     - when: never
+
+.ci-tron-x86_64-test-vk:
+   rules:
+    - when: never
+
+.ci-tron-x86_64-test-gl-manual:
+   rules:
+    - when: never
+
+.ci-tron-arm64-test-gl:
+   rules:
+    - when: never
+
+.ci-tron-x86_64-test-gl:
+   rules:
+    - when: never
diff --git a/drivers/gpu/drm/ci/igt_runner.sh b/drivers/gpu/drm/ci/igt_runner.sh
index b24d4bc53cda..1c01bda52237 100755
--- a/drivers/gpu/drm/ci/igt_runner.sh
+++ b/drivers/gpu/drm/ci/igt_runner.sh
@@ -1,6 +1,8 @@
-#!/bin/sh
+#!/usr/bin/env bash
 # SPDX-License-Identifier: MIT
 
+. "${SCRIPTS_DIR}/setup-test-env.sh"
+
 set -ex
 
 export IGT_FORCE_DRIVER=${DRIVER_NAME}
diff --git a/drivers/gpu/drm/ci/image-tags.yml b/drivers/gpu/drm/ci/image-tags.yml
index 7acc2e2a8eaa..1e6bcb166790 100644
--- a/drivers/gpu/drm/ci/image-tags.yml
+++ b/drivers/gpu/drm/ci/image-tags.yml
@@ -1,18 +1,22 @@
 variables:
-   CONTAINER_TAG: "20250502-mesa-uprev"
-   DEBIAN_BASE_TAG: "${CONTAINER_TAG}"
+   CONTAINER_TAG: "20251113-mesa-uprev"
+
+   DEBIAN_BUILD_BASE_TAG: "${CONTAINER_TAG}"
    DEBIAN_BUILD_TAG: "${CONTAINER_TAG}"
+   DEBIAN_TEST_BASE_TAG: "${CONTAINER_TAG}"
 
    DEBIAN_TEST_GL_TAG: "${CONTAINER_TAG}"
    # default kernel for rootfs before injecting the current kernel tree
-   KERNEL_TAG: "v6.14-mesa-0bdd"
+   KERNEL_TAG: "v6.16-mesa-9d85"
    KERNEL_REPO: "gfx-ci/linux"
-   PKG_REPO_REV: "95bf62c"
-
-   DEBIAN_PYUTILS_TAG: "${CONTAINER_TAG}"
+   PKG_REPO_REV: "0d2527f6"
+   FIRMWARE_TAG: "36f9bbfa"
+   FIRMWARE_REPO: "gfx-ci/firmware"
 
    ALPINE_X86_64_BUILD_TAG: "${CONTAINER_TAG}"
-   ALPINE_X86_64_LAVA_SSH_TAG: "${CONTAINER_TAG}"
 
-   CONDITIONAL_BUILD_ANGLE_TAG: 384145a4023315dae658259bee07c43a
-   CONDITIONAL_BUILD_PIGLIT_TAG: a19e424b8a3f020dbf1b9dd29f220a4f
+   CONDITIONAL_BUILD_ANGLE_TAG: 2388089f6225e4b5542d395cdf23de2f
+   CONDITIONAL_BUILD_CROSVM_TAG: 4079babd375b09761d59eacb25a0598a
+   CONDITIONAL_BUILD_PIGLIT_TAG: 21ab2c66f54777163dd038dc4cfcfde6
+
+   CROSVM_TAG: ${CONDITIONAL_BUILD_CROSVM_TAG}
diff --git a/drivers/gpu/drm/ci/lava-submit.sh b/drivers/gpu/drm/ci/lava-submit.sh
index a295102c3468..405055aa9cc4 100755
--- a/drivers/gpu/drm/ci/lava-submit.sh
+++ b/drivers/gpu/drm/ci/lava-submit.sh
@@ -3,27 +3,18 @@
 # shellcheck disable=SC2086 # we want word splitting
 # shellcheck disable=SC1091 # paths only become valid at runtime
 
-# If we run in the fork (not from mesa or Marge-bot), reuse mainline kernel and rootfs, if exist.
-_check_artifact_path() {
-	_url="https://${1}/${2}"
-	if curl -s -o /dev/null -I -L -f --retry 4 --retry-delay 15 "${_url}"; then
-		echo -n "${_url}"
-	fi
-}
+# shellcheck disable=SC1090
+source "${FDO_CI_BASH_HELPERS}"
 
-get_path_to_artifact() {
-	_mainline_artifact="$(_check_artifact_path ${BASE_SYSTEM_MAINLINE_HOST_PATH} ${1})"
-	if [ -n "${_mainline_artifact}" ]; then
-		echo -n "${_mainline_artifact}"
-		return
-	fi
-	_fork_artifact="$(_check_artifact_path ${BASE_SYSTEM_FORK_HOST_PATH} ${1})"
-	if [ -n "${_fork_artifact}" ]; then
-		echo -n "${_fork_artifact}"
-		return
-	fi
+fdo_log_section_start_collapsed prepare_rootfs "Preparing root filesystem"
+
+set -ex
+
+# If we run in the fork (not from mesa or Marge-bot), reuse mainline kernel and rootfs, if exist.
+ROOTFS_URL="$(fdo_find_s3_path "$LAVA_ROOTFS_PATH")" ||
+{
 	set +x
-	error "Sorry, I couldn't find a viable built path for ${1} in either mainline or a fork." >&2
+	fdo_log_section_error "Sorry, I couldn't find a viable built path for ${LAVA_ROOTFS_PATH} in either mainline or a fork." >&2
 	echo "" >&2
 	echo "If you're working on CI, this probably means that you're missing a dependency:" >&2
 	echo "this job ran ahead of the job which was supposed to upload that artifact." >&2
@@ -35,38 +26,51 @@ get_path_to_artifact() {
 	exit 1
 }
 
-. "${SCRIPTS_DIR}/setup-test-env.sh"
-
-section_start prepare_rootfs "Preparing root filesystem"
-
-set -ex
-
-ROOTFS_URL="$(get_path_to_artifact lava-rootfs.tar.zst)"
-[ $? != 1 ] || exit 1
-
 rm -rf results
-mkdir -p results/job-rootfs-overlay/
+mkdir results
 
-artifacts/ci-common/export-gitlab-job-env-for-dut.sh \
-    > results/job-rootfs-overlay/set-job-env-vars.sh
-cp artifacts/ci-common/init-*.sh results/job-rootfs-overlay/
-cp "$SCRIPTS_DIR"/setup-test-env.sh results/job-rootfs-overlay/
+fdo_filter_env_vars > dut-env-vars.sh
+# Set SCRIPTS_DIR to point to the Mesa install we download for the DUT
+echo "export SCRIPTS_DIR='$CI_PROJECT_DIR/install'" >> dut-env-vars.sh
 
-tar zcf job-rootfs-overlay.tar.gz -C results/job-rootfs-overlay/ .
-ci-fairy s3cp --token-file "${S3_JWT_FILE}" job-rootfs-overlay.tar.gz "https://${JOB_ROOTFS_OVERLAY_PATH}"
+fdo_log_section_end prepare_rootfs
 
 # Prepare env vars for upload.
-section_switch variables "Environment variables passed through to device:"
-cat results/job-rootfs-overlay/set-job-env-vars.sh
+fdo_log_section_start_collapsed variables "Environment variables passed through to device:"
+cat dut-env-vars.sh
+fdo_log_section_end variables
 
-section_switch lava_submit "Submitting job for scheduling"
+fdo_log_section_start_collapsed lava_submit "Submitting job for scheduling"
 
 touch results/lava.log
 tail -f results/lava.log &
 # Ensure that we are printing the commands that are being executed,
 # making it easier to debug the job in case it fails.
 set -x
-PYTHONPATH=artifacts/ artifacts/lava/lava_job_submitter.py \
+
+# List of optional overlays
+LAVA_EXTRA_OVERLAYS=()
+if [ -n "${LAVA_FIRMWARE:-}" ]; then
+    for fw in $LAVA_FIRMWARE; do
+        LAVA_EXTRA_OVERLAYS+=(
+            - append-overlay
+              --name=linux-firmware
+              --url="https://${S3_BASE_PATH}/${FIRMWARE_REPO}/${fw}-${FIRMWARE_TAG}.tar"
+              --path="/"
+              --format=tar
+        )
+    done
+fi
+LAVA_EXTRA_OVERLAYS+=(
+    - append-overlay \
+      --name=kernel-build \
+      --url="${FDO_HTTP_CACHE_URI:-}https://${PIPELINE_ARTIFACTS_BASE}/${DEBIAN_ARCH}/kernel-files.tar.zst" \
+      --compression=zstd \
+      --path="${CI_PROJECT_DIR}" \
+      --format=tar \
+)
+
+lava-job-submitter \
 	--farm "${FARM}" \
 	--device-type "${DEVICE_TYPE}" \
 	--boot-method "${BOOT_METHOD}" \
@@ -75,9 +79,8 @@ PYTHONPATH=artifacts/ artifacts/lava/lava_job_submitter.py \
 	--pipeline-info "$CI_JOB_NAME: $CI_PIPELINE_URL on $CI_COMMIT_REF_NAME ${CI_NODE_INDEX}/${CI_NODE_TOTAL}" \
 	--rootfs-url "${ROOTFS_URL}" \
 	--kernel-url-prefix "https://${PIPELINE_ARTIFACTS_BASE}/${DEBIAN_ARCH}" \
-	--kernel-external "${EXTERNAL_KERNEL_TAG}" \
-	--first-stage-init artifacts/ci-common/init-stage1.sh \
 	--dtb-filename "${DTB}" \
+	--env-file dut-env-vars.sh \
 	--jwt-file "${S3_JWT_FILE}" \
 	--kernel-image-name "${KERNEL_IMAGE_NAME}" \
 	--kernel-image-type "${KERNEL_IMAGE_TYPE}" \
@@ -86,20 +89,10 @@ PYTHONPATH=artifacts/ artifacts/lava/lava_job_submitter.py \
 	--mesa-job-name "$CI_JOB_NAME" \
 	--structured-log-file "results/lava_job_detail.json" \
 	--ssh-client-image "${LAVA_SSH_CLIENT_IMAGE}" \
+	--project-dir "${CI_PROJECT_DIR}" \
 	--project-name "${CI_PROJECT_NAME}" \
-	--starting-section "${CURRENT_SECTION}" \
+	--starting-section lava_submit \
 	--job-submitted-at "${CI_JOB_STARTED_AT}" \
-	- append-overlay \
-		--name=kernel-build \
-		--url="${FDO_HTTP_CACHE_URI:-}https://${PIPELINE_ARTIFACTS_BASE}/${DEBIAN_ARCH}/kernel-files.tar.zst" \
-		--compression=zstd \
-		--path="${CI_PROJECT_DIR}" \
-		--format=tar \
-	- append-overlay \
-		--name=job-overlay \
-		--url="https://${JOB_ROOTFS_OVERLAY_PATH}" \
-		--compression=gz \
-		--path="/" \
-		--format=tar \
+	"${LAVA_EXTRA_OVERLAYS[@]}" \
 	- submit \
 	>> results/lava.log
diff --git a/drivers/gpu/drm/ci/static-checks.yml b/drivers/gpu/drm/ci/static-checks.yml
index 13ffa827b7fa..712d3efe99c6 100644
--- a/drivers/gpu/drm/ci/static-checks.yml
+++ b/drivers/gpu/drm/ci/static-checks.yml
@@ -1,4 +1,5 @@
 check-patch:
+  stage: static-checks
   extends:
     - .build
     - .use-debian/x86_64_build
diff --git a/drivers/gpu/drm/ci/test.yml b/drivers/gpu/drm/ci/test.yml
index 81147e86bfd0..e56895626500 100644
--- a/drivers/gpu/drm/ci/test.yml
+++ b/drivers/gpu/drm/ci/test.yml
@@ -7,13 +7,16 @@
 
 .lava-test:
   extends:
-    - .container+build-rules
+    - .build-rules
     - .allow_failure_lockdep
   timeout: "1h30m"
   rules:
     - !reference [.scheduled_pipeline-rules, rules]
     - !reference [.collabora-farm-rules, rules]
     - when: on_success
+  before_script:
+    #  lava-submit.sh is a part of the archive, unlike Mesa CI
+    - eval "$S3_JWT_FILE_SCRIPT"
   script:
     # Note: Build dir (and thus install) may be dirty due to GIT_STRATEGY
     - rm -rf install
@@ -32,9 +35,7 @@
   dependencies:
     - testing:arm32
   needs:
-    - alpine/x86_64_lava_ssh_client
     - debian/arm32_test-gl
-    - python-artifacts
     - testing:arm32
     - igt:arm32
 
@@ -48,9 +49,7 @@
   dependencies:
     - testing:arm64
   needs:
-    - alpine/x86_64_lava_ssh_client
     - debian/arm64_test-gl
-    - python-artifacts
     - testing:arm64
     - igt:arm64
 
@@ -64,9 +63,7 @@
   dependencies:
     - testing:x86_64
   needs:
-    - alpine/x86_64_lava_ssh_client
     - debian/x86_64_test-gl
-    - python-artifacts
     - testing:x86_64
     - igt:x86_64
 
@@ -110,6 +107,7 @@
     - !reference [default, before_script]
     - rm -rf install
     - tar -xf artifacts/install.tar
+    - mkdir -p /kernel
   script:
     - ln -sf $CI_PROJECT_DIR/install /install
     - mv install/bzImage /kernel/bzImage
@@ -127,6 +125,7 @@
     DRIVER_NAME: msm
     BOOT_METHOD: depthcharge
     KERNEL_IMAGE_TYPE: ""
+    LAVA_FIRMWARE: qcom-lava
 
 msm:sc7180-trogdor-lazor-limozeen:
   extends:
@@ -194,11 +193,13 @@ msm:sm8350-hdk:
     KERNEL_IMAGE_NAME: "Image.gz"
     KERNEL_IMAGE_TYPE: ""
     RUNNER_TAG: mesa-ci-x86-64-lava-sm8350-hdk
+    LAVA_FIRMWARE: qcom-lava
 
 .rockchip-device:
   variables:
     DTB: ${DEVICE_TYPE}
     BOOT_METHOD: depthcharge
+    LAVA_FIRMWARE: arm-lava
 
 .rockchip-display:
   stage: rockchip
@@ -255,6 +256,7 @@ panfrost:rk3399:
     DTB: ""
     BOOT_METHOD: depthcharge
     KERNEL_IMAGE_TYPE: ""
+    LAVA_FIRMWARE: i915
 
 i915:apl:
   extends:
@@ -337,6 +339,7 @@ i915:jsl:
     DTB: ""
     BOOT_METHOD: depthcharge
     KERNEL_IMAGE_TYPE: ""
+    LAVA_FIRMWARE: amdgpu-lava
 
 amdgpu:stoney:
   extends:
@@ -355,6 +358,7 @@ amdgpu:stoney:
     DTB: ${DEVICE_TYPE}
     BOOT_METHOD: depthcharge
     KERNEL_IMAGE_TYPE: ""
+    LAVA_FIRMWARE: arm-lava
 
 .mediatek-display:
   stage: mediatek
-- 
2.51.0


^ permalink raw reply	[flat|nested] 10+ messages in thread

end of thread, other threads:[~2025-11-16 15:32 UTC | newest]

Thread overview: 10+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2024-11-28  4:20 [PATCH v1 0/2] drm/ci: uprev mesa and modify gitlab rules Vignesh Raman
2024-11-28  4:20 ` [PATCH v1 1/2] drm/ci: uprev mesa Vignesh Raman
2024-12-03 14:14   ` Helen Mae Koike Fornazier
2024-11-28  4:20 ` [PATCH v1 2/2] drm/ci: update gitlab rules Vignesh Raman
2024-12-03 14:12 ` [PATCH v1 0/2] drm/ci: uprev mesa and modify " Helen Mae Koike Fornazier
2024-12-04  2:23   ` Vignesh Raman
2025-11-14  3:00 [PATCH v1 0/2] drm/ci: uprev IGT and enable apq8016, apq8096 Vignesh Raman
2025-11-14  3:00 ` [PATCH v1 1/2] drm/ci: uprev mesa Vignesh Raman
2025-11-15  0:44   ` Dmitry Baryshkov
2025-11-15  3:55     ` Dmitry Baryshkov
2025-11-16 15:32       ` Dmitry Baryshkov

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox

all inboxes | Powered by JetHome®