* [PATCH v1 0/2] drm/ci: uprev IGT and enable apq8016, apq8096 @ 2025-11-14 3:00 Vignesh Raman 2025-11-14 3:00 ` [PATCH v1 1/2] drm/ci: uprev mesa Vignesh Raman 2025-11-14 3:00 ` [PATCH v1 2/2] drm/ci: Move qualcomm baremetal jobs to lava Vignesh Raman 0 siblings, 2 replies; 9+ messages in thread From: Vignesh Raman @ 2025-11-14 3:00 UTC (permalink / raw) To: dri-devel Cc: daniels, helen.fornazier, airlied, simona.vetter, dmitry.baryshkov, lumag, robdclark, robin.clark, guilherme.gallo, sergi.blanch.torne, valentine.burley, linux-mediatek, linux-amlogic, linux-rockchip, amd-gfx, linux-arm-msm, intel-gfx, virtualization, linux-kernel This series includes, - Adapting to recent changes in Mesa CI, such as: - LAVA overlay-based firmware handling - Container/job rule separation - Removal of the python-artifacts job - Use lava-job-submitter for submitting jobs to LAVA - Various other CI improvements - Move bare-metal jobs for apq8016 and apq8096 to LAVA, as these devices are migrated to Collabora LAVA farm. Pipeline: https://gitlab.freedesktop.org/vigneshraman/msm/-/pipelines/1546424 MR: https://gitlab.freedesktop.org/drm/msm/-/merge_requests/200 There will be a follow up series to update expectation files for the other jobs. Vignesh Raman (2): drm/ci: uprev mesa drm/ci: Move qualcomm baremetal jobs to lava drivers/gpu/drm/ci/arm64.config | 1 + drivers/gpu/drm/ci/build.yml | 22 ++-- drivers/gpu/drm/ci/container.yml | 28 +++-- drivers/gpu/drm/ci/gitlab-ci.yml | 95 ++++++++++++---- drivers/gpu/drm/ci/igt_runner.sh | 4 +- drivers/gpu/drm/ci/image-tags.yml | 22 ++-- drivers/gpu/drm/ci/lava-submit.sh | 101 ++++++++---------- drivers/gpu/drm/ci/static-checks.yml | 1 + drivers/gpu/drm/ci/test.yml | 80 ++++++-------- .../gpu/drm/ci/xfails/msm-apq8016-fails.txt | 4 + .../gpu/drm/ci/xfails/msm-apq8096-fails.txt | 2 + 11 files changed, 213 insertions(+), 147 deletions(-) -- 2.51.0 ^ permalink raw reply [flat|nested] 9+ messages in thread
* [PATCH v1 1/2] drm/ci: uprev mesa 2025-11-14 3:00 [PATCH v1 0/2] drm/ci: uprev IGT and enable apq8016, apq8096 Vignesh Raman @ 2025-11-14 3:00 ` Vignesh Raman 2025-11-15 0:44 ` Dmitry Baryshkov 2025-11-14 3:00 ` [PATCH v1 2/2] drm/ci: Move qualcomm baremetal jobs to lava Vignesh Raman 1 sibling, 1 reply; 9+ messages in thread From: Vignesh Raman @ 2025-11-14 3:00 UTC (permalink / raw) To: dri-devel Cc: daniels, helen.fornazier, airlied, simona.vetter, dmitry.baryshkov, lumag, robdclark, robin.clark, guilherme.gallo, sergi.blanch.torne, valentine.burley, linux-mediatek, linux-amlogic, linux-rockchip, amd-gfx, linux-arm-msm, intel-gfx, virtualization, linux-kernel Uprev mesa to adapt to the latest changes in Mesa CI, including support for firmware via LAVA overlays, removal of the python-artifacts job, split container and build rules, use lava-job-submitter container, and various misc fixes. Co-developed-by: Dmitry Baryshkov <dmitry.baryshkov@oss.qualcomm.com> Signed-off-by: Dmitry Baryshkov <dmitry.baryshkov@oss.qualcomm.com> Signed-off-by: Vignesh Raman <vignesh.raman@collabora.com> --- drivers/gpu/drm/ci/build.yml | 22 +++--- drivers/gpu/drm/ci/container.yml | 28 ++++++-- drivers/gpu/drm/ci/gitlab-ci.yml | 95 ++++++++++++++++++++----- drivers/gpu/drm/ci/igt_runner.sh | 4 +- drivers/gpu/drm/ci/image-tags.yml | 22 +++--- drivers/gpu/drm/ci/lava-submit.sh | 101 +++++++++++++-------------- drivers/gpu/drm/ci/static-checks.yml | 1 + drivers/gpu/drm/ci/test.yml | 18 +++-- 8 files changed, 186 insertions(+), 105 deletions(-) diff --git a/drivers/gpu/drm/ci/build.yml b/drivers/gpu/drm/ci/build.yml index af27ff5de369..efbcaae49807 100644 --- a/drivers/gpu/drm/ci/build.yml +++ b/drivers/gpu/drm/ci/build.yml @@ -1,6 +1,6 @@ .build: extends: - - .container+build-rules + - .build-rules stage: build-only artifacts: paths: @@ -133,6 +133,10 @@ debian-arm32-asan: rules: - when: never +debian-x86_64-msan: + rules: + - when: never + debian-arm64: rules: - when: never @@ -153,7 +157,7 @@ debian-arm64-ubsan: rules: - when: never -debian-build-testing: +debian-build-x86_64: rules: - when: never @@ -177,31 +181,31 @@ debian-release: rules: - when: never -debian-s390x: +debian-riscv64: rules: - when: never -debian-testing: +debian-s390x: rules: - when: never -debian-testing-asan: +debian-vulkan: rules: - when: never -debian-testing-msan: +debian-x86_32: rules: - when: never -debian-testing-ubsan: +debian-x86_64: rules: - when: never -debian-vulkan: +debian-x86_64-asan: rules: - when: never -debian-x86_32: +debian-x86_64-ubsan: rules: - when: never diff --git a/drivers/gpu/drm/ci/container.yml b/drivers/gpu/drm/ci/container.yml index 5f90508578a3..ce6007a45a07 100644 --- a/drivers/gpu/drm/ci/container.yml +++ b/drivers/gpu/drm/ci/container.yml @@ -5,21 +5,29 @@ debian/x86_64_build-base: variables: - EXTRA_LOCAL_PACKAGES: "libcairo-dev libdw-dev libjson-c-dev libkmod2 libkmod-dev libpciaccess-dev libproc2-dev libudev-dev libunwind-dev python3-docutils bc python3-ply libssl-dev bc" + EXTRA_LOCAL_PACKAGES: "libcairo-dev libdw-dev libjson-c-dev libkmod-dev libpciaccess-dev libproc2-dev libudev-dev libunwind-dev python3-docutils bc python3-ply libssl-dev bc" + +debian/arm64_build: + variables: + EXTRA_LOCAL_PACKAGES: "libcairo-dev libdw-dev libjson-c-dev libproc2-dev libkmod-dev libpciaccess-dev libudev-dev libunwind-dev python3-docutils libssl-dev crossbuild-essential-armhf libkmod-dev:armhf libproc2-dev:armhf libunwind-dev:armhf libdw-dev:armhf libpixman-1-dev:armhf libcairo-dev:armhf libudev-dev:armhf libjson-c-dev:armhf" debian/x86_64_test-gl: variables: - EXTRA_LOCAL_PACKAGES: "jq libasound2 libcairo2 libdw1 libglib2.0-0 libjson-c5 libkmod-dev libkmod2 libgles2 libproc2-dev" + EXTRA_LOCAL_PACKAGES: "jq libasound2t64 libcairo2 libdw1t64 libglib2.0-0t64 libjson-c5 libkmod2 libgles2 libdrm-nouveau2 libdrm-amdgpu1" -debian/arm64_build: +debian/arm64_test-gl: variables: - EXTRA_LOCAL_PACKAGES: "libcairo-dev libdw-dev libjson-c-dev libproc2-dev libkmod2 libkmod-dev libpciaccess-dev libudev-dev libunwind-dev python3-docutils libssl-dev crossbuild-essential-armhf libkmod-dev:armhf libproc2-dev:armhf libunwind-dev:armhf libdw-dev:armhf libpixman-1-dev:armhf libcairo-dev:armhf libudev-dev:armhf libjson-c-dev:armhf" + EXTRA_LOCAL_PACKAGES: "jq libasound2t64 libcairo2 libdw1t64 libglib2.0-0t64 libjson-c5 libkmod2 libgles2 libdrm-nouveau2 libdrm-amdgpu1" -.kernel+rootfs: +debian/arm32_test-gl: variables: - EXTRA_LOCAL_PACKAGES: "jq libasound2 libcairo2 libdw1 libglib2.0-0 libjson-c5" + EXTRA_LOCAL_PACKAGES: "jq libasound2t64 libcairo2 libdw1t64 libglib2.0-0t64 libjson-c5 libkmod2 libgles2 libdrm-nouveau2 libdrm-amdgpu1 libunwind8" # Disable container jobs that we won't use +alpine/x86_64_build: + rules: + - when: never + debian/arm64_test-vk: rules: - when: never @@ -28,6 +36,10 @@ debian/baremetal_arm32_test-gl: rules: - when: never +debian/baremetal_arm64_test-gl: + rules: + - when: never + debian/baremetal_arm64_test-vk: rules: - when: never @@ -36,6 +48,10 @@ debian/ppc64el_build: rules: - when: never +debian/riscv64_build: + rules: + - when: never + debian/s390x_build: rules: - when: never diff --git a/drivers/gpu/drm/ci/gitlab-ci.yml b/drivers/gpu/drm/ci/gitlab-ci.yml index 56638814bb28..e23a9110008c 100644 --- a/drivers/gpu/drm/ci/gitlab-ci.yml +++ b/drivers/gpu/drm/ci/gitlab-ci.yml @@ -1,6 +1,6 @@ variables: DRM_CI_PROJECT_PATH: &drm-ci-project-path mesa/mesa - DRM_CI_COMMIT_SHA: &drm-ci-commit-sha 02337aec715c25dae7ff2479d986f831c77fe536 + DRM_CI_COMMIT_SHA: &drm-ci-commit-sha f07dab189928ee65865ef9cefcaa4eff05da99b5 UPSTREAM_REPO: https://gitlab.freedesktop.org/drm/kernel.git TARGET_BRANCH: drm-next @@ -11,7 +11,7 @@ variables: DEQP_RUNNER_GIT_TAG: v0.20.0 FDO_UPSTREAM_REPO: helen.fornazier/linux # The repo where the git-archive daily runs - MESA_TEMPLATES_COMMIT: &ci-templates-commit c6aeb16f86e32525fa630fb99c66c4f3e62fc3cb + MESA_TEMPLATES_COMMIT: &ci-templates-commit aec7a6ce7bb38902c70641526f6611e27141784a DRM_CI_PROJECT_URL: https://gitlab.freedesktop.org/${DRM_CI_PROJECT_PATH} CI_PRE_CLONE_SCRIPT: |- set -o xtrace @@ -30,6 +30,8 @@ variables: S3_GITCACHE_BUCKET: git-cache # Bucket for the pipeline artifacts pushed to S3 S3_ARTIFACTS_BUCKET: artifacts + # Base path used for various artifacts + S3_BASE_PATH: "${S3_HOST}/${S3_KERNEL_BUCKET}" # per-pipeline artifact storage on MinIO PIPELINE_ARTIFACTS_BASE: ${S3_HOST}/${S3_ARTIFACTS_BUCKET}/${CI_PROJECT_PATH}/${CI_PIPELINE_ID} # per-job artifact storage on MinIO @@ -44,6 +46,8 @@ variables: ARTIFACTS_BASE_URL: https://${CI_PROJECT_ROOT_NAMESPACE}.${CI_PAGES_DOMAIN}/-/${CI_PROJECT_NAME}/-/jobs/${CI_JOB_ID}/artifacts # Python scripts for structured logger PYTHONPATH: "$PYTHONPATH:$CI_PROJECT_DIR/install" + # Mesa-specific variables that shouldn't be forwarded to DUTs and crosvm + CI_EXCLUDE_ENV_VAR_REGEX: 'SCRIPTS_DIR|RESULTS_DIR' default: @@ -84,10 +88,11 @@ include: - project: *drm-ci-project-path ref: *drm-ci-commit-sha file: + - '/.gitlab-ci/bare-metal/gitlab-ci.yml' - '/.gitlab-ci/build/gitlab-ci.yml' - '/.gitlab-ci/container/gitlab-ci.yml' - '/.gitlab-ci/farm-rules.yml' - - '/.gitlab-ci/lava/lava-gitlab-ci.yml' + - '/.gitlab-ci/lava/gitlab-ci.yml' - '/.gitlab-ci/test-source-dep.yml' - '/.gitlab-ci/test/gitlab-ci.yml' - '/src/amd/ci/gitlab-ci-inc.yml' @@ -147,10 +152,10 @@ stages: - if: &is-merge-attempt $GITLAB_USER_LOGIN == "marge-bot" && $CI_PIPELINE_SOURCE == "merge_request_event" # post-merge pipeline - if: &is-post-merge $GITLAB_USER_LOGIN == "marge-bot" && $CI_PIPELINE_SOURCE == "push" - # Pre-merge pipeline - - if: &is-pre-merge $CI_PIPELINE_SOURCE == "merge_request_event" + # Pre-merge pipeline (because merge pipelines are already caught above) + - if: &is-merge-request $CI_PIPELINE_SOURCE == "merge_request_event" # Push to a branch on a fork - - if: &is-fork-push $CI_PIPELINE_SOURCE == "push" + - if: &is-push-to-fork $CI_PIPELINE_SOURCE == "push" # nightly pipeline - if: &is-scheduled-pipeline $CI_PIPELINE_SOURCE == "schedule" # pipeline for direct pushes that bypassed the CI @@ -160,17 +165,59 @@ stages: # Rules applied to every job in the pipeline .common-rules: rules: - - if: *is-fork-push + - if: *is-push-to-fork when: manual - .never-post-merge-rules: rules: - if: *is-post-merge when: never -.container+build-rules: +# Note: make sure the branches in this list are the same as in +# `.build-only-delayed-rules` below. +.container-rules: + rules: + - !reference [.common-rules, rules] + # Run when re-enabling a disabled farm, but not when disabling it + - !reference [.disable-farm-mr-rules, rules] + # Never run immediately after merging, as we just ran everything + - !reference [.never-post-merge-rules, rules] + # Only rebuild containers in merge pipelines if any tags have been + # changed, else we'll just use the already-built containers + - if: *is-merge-attempt + changes: &image_tags_path + - drivers/gpu/drm/ci/image-tags.yml + when: on_success + # Skip everything for pre-merge and merge pipelines which don't change + # anything in the build; we only do this for marge-bot and not user + # pipelines in a MR, because we might still need to run it to copy the + # container into the user's namespace. + - if: *is-merge-attempt + when: never + # Any MR pipeline which changes image-tags.yml needs to be able to + # rebuild the containers + - if: *is-merge-request + changes: *image_tags_path + when: manual + # ... however for MRs running inside the user namespace, we may need to + # run these jobs to copy the container images from upstream + - if: *is-merge-request + when: manual + # Build everything after someone bypassed the CI + - if: *is-direct-push + when: manual + # Scheduled pipelines reuse already-built containers + - if: *is-scheduled-pipeline + when: never + # Allow building everything in fork pipelines, but build nothing unless + # manually triggered + - when: manual + + +# Note: make sure the branches in this list are the same as in +# `.build-only-delayed-rules` below. +.build-rules: rules: - !reference [.common-rules, rules] # Run when re-enabling a disabled farm, but not when disabling it @@ -181,7 +228,7 @@ stages: - if: *is-merge-attempt when: on_success # Same as above, but for pre-merge pipelines - - if: *is-pre-merge + - if: *is-merge-request when: manual # Build everything after someone bypassed the CI - if: *is-direct-push @@ -197,7 +244,7 @@ stages: # Repeat of the above but with `when: on_success` replaced with # `when: delayed` + `start_in:`, for build-only jobs. # Note: make sure the branches in this list are the same as in -# `.container+build-rules` above. +# `.build-rules` above. .build-only-delayed-rules: rules: - !reference [.common-rules, rules] @@ -210,7 +257,7 @@ stages: when: delayed start_in: &build-delay 5 minutes # Same as above, but for pre-merge pipelines - - if: *is-pre-merge + - if: *is-merge-request when: manual # Build everything after someone bypassed the CI - if: *is-direct-push @@ -237,11 +284,6 @@ stages: - _build/meson-logs/strace -python-artifacts: - variables: - GIT_DEPTH: 10 - - # Git archive make-git-archive: extends: @@ -273,7 +315,7 @@ sanity: tags: - $FDO_RUNNER_JOB_PRIORITY_TAG_X86_64 rules: - - if: *is-pre-merge + - if: *is-merge-request when: on_success - when: never variables: @@ -284,7 +326,6 @@ sanity: - | set -eu image_tags=( - ALPINE_X86_64_LAVA_SSH_TAG CONTAINER_TAG DEBIAN_BASE_TAG DEBIAN_BUILD_TAG @@ -347,3 +388,19 @@ linkcheck-docs: test-docs: rules: - when: never + +.ci-tron-x86_64-test-vk: + rules: + - when: never + +.ci-tron-x86_64-test-gl-manual: + rules: + - when: never + +.ci-tron-arm64-test-gl: + rules: + - when: never + +.ci-tron-x86_64-test-gl: + rules: + - when: never diff --git a/drivers/gpu/drm/ci/igt_runner.sh b/drivers/gpu/drm/ci/igt_runner.sh index b24d4bc53cda..1c01bda52237 100755 --- a/drivers/gpu/drm/ci/igt_runner.sh +++ b/drivers/gpu/drm/ci/igt_runner.sh @@ -1,6 +1,8 @@ -#!/bin/sh +#!/usr/bin/env bash # SPDX-License-Identifier: MIT +. "${SCRIPTS_DIR}/setup-test-env.sh" + set -ex export IGT_FORCE_DRIVER=${DRIVER_NAME} diff --git a/drivers/gpu/drm/ci/image-tags.yml b/drivers/gpu/drm/ci/image-tags.yml index 7acc2e2a8eaa..1e6bcb166790 100644 --- a/drivers/gpu/drm/ci/image-tags.yml +++ b/drivers/gpu/drm/ci/image-tags.yml @@ -1,18 +1,22 @@ variables: - CONTAINER_TAG: "20250502-mesa-uprev" - DEBIAN_BASE_TAG: "${CONTAINER_TAG}" + CONTAINER_TAG: "20251113-mesa-uprev" + + DEBIAN_BUILD_BASE_TAG: "${CONTAINER_TAG}" DEBIAN_BUILD_TAG: "${CONTAINER_TAG}" + DEBIAN_TEST_BASE_TAG: "${CONTAINER_TAG}" DEBIAN_TEST_GL_TAG: "${CONTAINER_TAG}" # default kernel for rootfs before injecting the current kernel tree - KERNEL_TAG: "v6.14-mesa-0bdd" + KERNEL_TAG: "v6.16-mesa-9d85" KERNEL_REPO: "gfx-ci/linux" - PKG_REPO_REV: "95bf62c" - - DEBIAN_PYUTILS_TAG: "${CONTAINER_TAG}" + PKG_REPO_REV: "0d2527f6" + FIRMWARE_TAG: "36f9bbfa" + FIRMWARE_REPO: "gfx-ci/firmware" ALPINE_X86_64_BUILD_TAG: "${CONTAINER_TAG}" - ALPINE_X86_64_LAVA_SSH_TAG: "${CONTAINER_TAG}" - CONDITIONAL_BUILD_ANGLE_TAG: 384145a4023315dae658259bee07c43a - CONDITIONAL_BUILD_PIGLIT_TAG: a19e424b8a3f020dbf1b9dd29f220a4f + CONDITIONAL_BUILD_ANGLE_TAG: 2388089f6225e4b5542d395cdf23de2f + CONDITIONAL_BUILD_CROSVM_TAG: 4079babd375b09761d59eacb25a0598a + CONDITIONAL_BUILD_PIGLIT_TAG: 21ab2c66f54777163dd038dc4cfcfde6 + + CROSVM_TAG: ${CONDITIONAL_BUILD_CROSVM_TAG} diff --git a/drivers/gpu/drm/ci/lava-submit.sh b/drivers/gpu/drm/ci/lava-submit.sh index a295102c3468..405055aa9cc4 100755 --- a/drivers/gpu/drm/ci/lava-submit.sh +++ b/drivers/gpu/drm/ci/lava-submit.sh @@ -3,27 +3,18 @@ # shellcheck disable=SC2086 # we want word splitting # shellcheck disable=SC1091 # paths only become valid at runtime -# If we run in the fork (not from mesa or Marge-bot), reuse mainline kernel and rootfs, if exist. -_check_artifact_path() { - _url="https://${1}/${2}" - if curl -s -o /dev/null -I -L -f --retry 4 --retry-delay 15 "${_url}"; then - echo -n "${_url}" - fi -} +# shellcheck disable=SC1090 +source "${FDO_CI_BASH_HELPERS}" -get_path_to_artifact() { - _mainline_artifact="$(_check_artifact_path ${BASE_SYSTEM_MAINLINE_HOST_PATH} ${1})" - if [ -n "${_mainline_artifact}" ]; then - echo -n "${_mainline_artifact}" - return - fi - _fork_artifact="$(_check_artifact_path ${BASE_SYSTEM_FORK_HOST_PATH} ${1})" - if [ -n "${_fork_artifact}" ]; then - echo -n "${_fork_artifact}" - return - fi +fdo_log_section_start_collapsed prepare_rootfs "Preparing root filesystem" + +set -ex + +# If we run in the fork (not from mesa or Marge-bot), reuse mainline kernel and rootfs, if exist. +ROOTFS_URL="$(fdo_find_s3_path "$LAVA_ROOTFS_PATH")" || +{ set +x - error "Sorry, I couldn't find a viable built path for ${1} in either mainline or a fork." >&2 + fdo_log_section_error "Sorry, I couldn't find a viable built path for ${LAVA_ROOTFS_PATH} in either mainline or a fork." >&2 echo "" >&2 echo "If you're working on CI, this probably means that you're missing a dependency:" >&2 echo "this job ran ahead of the job which was supposed to upload that artifact." >&2 @@ -35,38 +26,51 @@ get_path_to_artifact() { exit 1 } -. "${SCRIPTS_DIR}/setup-test-env.sh" - -section_start prepare_rootfs "Preparing root filesystem" - -set -ex - -ROOTFS_URL="$(get_path_to_artifact lava-rootfs.tar.zst)" -[ $? != 1 ] || exit 1 - rm -rf results -mkdir -p results/job-rootfs-overlay/ +mkdir results -artifacts/ci-common/export-gitlab-job-env-for-dut.sh \ - > results/job-rootfs-overlay/set-job-env-vars.sh -cp artifacts/ci-common/init-*.sh results/job-rootfs-overlay/ -cp "$SCRIPTS_DIR"/setup-test-env.sh results/job-rootfs-overlay/ +fdo_filter_env_vars > dut-env-vars.sh +# Set SCRIPTS_DIR to point to the Mesa install we download for the DUT +echo "export SCRIPTS_DIR='$CI_PROJECT_DIR/install'" >> dut-env-vars.sh -tar zcf job-rootfs-overlay.tar.gz -C results/job-rootfs-overlay/ . -ci-fairy s3cp --token-file "${S3_JWT_FILE}" job-rootfs-overlay.tar.gz "https://${JOB_ROOTFS_OVERLAY_PATH}" +fdo_log_section_end prepare_rootfs # Prepare env vars for upload. -section_switch variables "Environment variables passed through to device:" -cat results/job-rootfs-overlay/set-job-env-vars.sh +fdo_log_section_start_collapsed variables "Environment variables passed through to device:" +cat dut-env-vars.sh +fdo_log_section_end variables -section_switch lava_submit "Submitting job for scheduling" +fdo_log_section_start_collapsed lava_submit "Submitting job for scheduling" touch results/lava.log tail -f results/lava.log & # Ensure that we are printing the commands that are being executed, # making it easier to debug the job in case it fails. set -x -PYTHONPATH=artifacts/ artifacts/lava/lava_job_submitter.py \ + +# List of optional overlays +LAVA_EXTRA_OVERLAYS=() +if [ -n "${LAVA_FIRMWARE:-}" ]; then + for fw in $LAVA_FIRMWARE; do + LAVA_EXTRA_OVERLAYS+=( + - append-overlay + --name=linux-firmware + --url="https://${S3_BASE_PATH}/${FIRMWARE_REPO}/${fw}-${FIRMWARE_TAG}.tar" + --path="/" + --format=tar + ) + done +fi +LAVA_EXTRA_OVERLAYS+=( + - append-overlay \ + --name=kernel-build \ + --url="${FDO_HTTP_CACHE_URI:-}https://${PIPELINE_ARTIFACTS_BASE}/${DEBIAN_ARCH}/kernel-files.tar.zst" \ + --compression=zstd \ + --path="${CI_PROJECT_DIR}" \ + --format=tar \ +) + +lava-job-submitter \ --farm "${FARM}" \ --device-type "${DEVICE_TYPE}" \ --boot-method "${BOOT_METHOD}" \ @@ -75,9 +79,8 @@ PYTHONPATH=artifacts/ artifacts/lava/lava_job_submitter.py \ --pipeline-info "$CI_JOB_NAME: $CI_PIPELINE_URL on $CI_COMMIT_REF_NAME ${CI_NODE_INDEX}/${CI_NODE_TOTAL}" \ --rootfs-url "${ROOTFS_URL}" \ --kernel-url-prefix "https://${PIPELINE_ARTIFACTS_BASE}/${DEBIAN_ARCH}" \ - --kernel-external "${EXTERNAL_KERNEL_TAG}" \ - --first-stage-init artifacts/ci-common/init-stage1.sh \ --dtb-filename "${DTB}" \ + --env-file dut-env-vars.sh \ --jwt-file "${S3_JWT_FILE}" \ --kernel-image-name "${KERNEL_IMAGE_NAME}" \ --kernel-image-type "${KERNEL_IMAGE_TYPE}" \ @@ -86,20 +89,10 @@ PYTHONPATH=artifacts/ artifacts/lava/lava_job_submitter.py \ --mesa-job-name "$CI_JOB_NAME" \ --structured-log-file "results/lava_job_detail.json" \ --ssh-client-image "${LAVA_SSH_CLIENT_IMAGE}" \ + --project-dir "${CI_PROJECT_DIR}" \ --project-name "${CI_PROJECT_NAME}" \ - --starting-section "${CURRENT_SECTION}" \ + --starting-section lava_submit \ --job-submitted-at "${CI_JOB_STARTED_AT}" \ - - append-overlay \ - --name=kernel-build \ - --url="${FDO_HTTP_CACHE_URI:-}https://${PIPELINE_ARTIFACTS_BASE}/${DEBIAN_ARCH}/kernel-files.tar.zst" \ - --compression=zstd \ - --path="${CI_PROJECT_DIR}" \ - --format=tar \ - - append-overlay \ - --name=job-overlay \ - --url="https://${JOB_ROOTFS_OVERLAY_PATH}" \ - --compression=gz \ - --path="/" \ - --format=tar \ + "${LAVA_EXTRA_OVERLAYS[@]}" \ - submit \ >> results/lava.log diff --git a/drivers/gpu/drm/ci/static-checks.yml b/drivers/gpu/drm/ci/static-checks.yml index 13ffa827b7fa..712d3efe99c6 100644 --- a/drivers/gpu/drm/ci/static-checks.yml +++ b/drivers/gpu/drm/ci/static-checks.yml @@ -1,4 +1,5 @@ check-patch: + stage: static-checks extends: - .build - .use-debian/x86_64_build diff --git a/drivers/gpu/drm/ci/test.yml b/drivers/gpu/drm/ci/test.yml index 81147e86bfd0..e56895626500 100644 --- a/drivers/gpu/drm/ci/test.yml +++ b/drivers/gpu/drm/ci/test.yml @@ -7,13 +7,16 @@ .lava-test: extends: - - .container+build-rules + - .build-rules - .allow_failure_lockdep timeout: "1h30m" rules: - !reference [.scheduled_pipeline-rules, rules] - !reference [.collabora-farm-rules, rules] - when: on_success + before_script: + # lava-submit.sh is a part of the archive, unlike Mesa CI + - eval "$S3_JWT_FILE_SCRIPT" script: # Note: Build dir (and thus install) may be dirty due to GIT_STRATEGY - rm -rf install @@ -32,9 +35,7 @@ dependencies: - testing:arm32 needs: - - alpine/x86_64_lava_ssh_client - debian/arm32_test-gl - - python-artifacts - testing:arm32 - igt:arm32 @@ -48,9 +49,7 @@ dependencies: - testing:arm64 needs: - - alpine/x86_64_lava_ssh_client - debian/arm64_test-gl - - python-artifacts - testing:arm64 - igt:arm64 @@ -64,9 +63,7 @@ dependencies: - testing:x86_64 needs: - - alpine/x86_64_lava_ssh_client - debian/x86_64_test-gl - - python-artifacts - testing:x86_64 - igt:x86_64 @@ -110,6 +107,7 @@ - !reference [default, before_script] - rm -rf install - tar -xf artifacts/install.tar + - mkdir -p /kernel script: - ln -sf $CI_PROJECT_DIR/install /install - mv install/bzImage /kernel/bzImage @@ -127,6 +125,7 @@ DRIVER_NAME: msm BOOT_METHOD: depthcharge KERNEL_IMAGE_TYPE: "" + LAVA_FIRMWARE: qcom-lava msm:sc7180-trogdor-lazor-limozeen: extends: @@ -194,11 +193,13 @@ msm:sm8350-hdk: KERNEL_IMAGE_NAME: "Image.gz" KERNEL_IMAGE_TYPE: "" RUNNER_TAG: mesa-ci-x86-64-lava-sm8350-hdk + LAVA_FIRMWARE: qcom-lava .rockchip-device: variables: DTB: ${DEVICE_TYPE} BOOT_METHOD: depthcharge + LAVA_FIRMWARE: arm-lava .rockchip-display: stage: rockchip @@ -255,6 +256,7 @@ panfrost:rk3399: DTB: "" BOOT_METHOD: depthcharge KERNEL_IMAGE_TYPE: "" + LAVA_FIRMWARE: i915 i915:apl: extends: @@ -337,6 +339,7 @@ i915:jsl: DTB: "" BOOT_METHOD: depthcharge KERNEL_IMAGE_TYPE: "" + LAVA_FIRMWARE: amdgpu-lava amdgpu:stoney: extends: @@ -355,6 +358,7 @@ amdgpu:stoney: DTB: ${DEVICE_TYPE} BOOT_METHOD: depthcharge KERNEL_IMAGE_TYPE: "" + LAVA_FIRMWARE: arm-lava .mediatek-display: stage: mediatek -- 2.51.0 ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [PATCH v1 1/2] drm/ci: uprev mesa 2025-11-14 3:00 ` [PATCH v1 1/2] drm/ci: uprev mesa Vignesh Raman @ 2025-11-15 0:44 ` Dmitry Baryshkov 2025-11-15 3:55 ` Dmitry Baryshkov 0 siblings, 1 reply; 9+ messages in thread From: Dmitry Baryshkov @ 2025-11-15 0:44 UTC (permalink / raw) To: Vignesh Raman Cc: dri-devel, daniels, helen.fornazier, airlied, simona.vetter, lumag, robdclark, robin.clark, guilherme.gallo, sergi.blanch.torne, valentine.burley, linux-mediatek, linux-amlogic, linux-rockchip, amd-gfx, linux-arm-msm, intel-gfx, virtualization, linux-kernel On Fri, Nov 14, 2025 at 08:30:50AM +0530, Vignesh Raman wrote: > Uprev mesa to adapt to the latest changes in Mesa CI, including support > for firmware via LAVA overlays, removal of the python-artifacts job, > split container and build rules, use lava-job-submitter container, > and various misc fixes. > > Co-developed-by: Dmitry Baryshkov <dmitry.baryshkov@oss.qualcomm.com> > Signed-off-by: Dmitry Baryshkov <dmitry.baryshkov@oss.qualcomm.com> > Signed-off-by: Vignesh Raman <vignesh.raman@collabora.com> > --- > drivers/gpu/drm/ci/build.yml | 22 +++--- > drivers/gpu/drm/ci/container.yml | 28 ++++++-- > drivers/gpu/drm/ci/gitlab-ci.yml | 95 ++++++++++++++++++++----- > drivers/gpu/drm/ci/igt_runner.sh | 4 +- > drivers/gpu/drm/ci/image-tags.yml | 22 +++--- > drivers/gpu/drm/ci/lava-submit.sh | 101 +++++++++++++-------------- > drivers/gpu/drm/ci/static-checks.yml | 1 + > drivers/gpu/drm/ci/test.yml | 18 +++-- > 8 files changed, 186 insertions(+), 105 deletions(-) > Thanks a lot for taking care of it! Reviewed-by: Dmitry Baryshkov <dmitry.baryshkov@oss.qualcomm.com> -- With best wishes Dmitry ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [PATCH v1 1/2] drm/ci: uprev mesa 2025-11-15 0:44 ` Dmitry Baryshkov @ 2025-11-15 3:55 ` Dmitry Baryshkov 2025-11-16 15:32 ` Dmitry Baryshkov 0 siblings, 1 reply; 9+ messages in thread From: Dmitry Baryshkov @ 2025-11-15 3:55 UTC (permalink / raw) To: Vignesh Raman Cc: dri-devel, daniels, helen.fornazier, airlied, simona.vetter, lumag, robdclark, robin.clark, guilherme.gallo, sergi.blanch.torne, valentine.burley, linux-mediatek, linux-amlogic, linux-rockchip, amd-gfx, linux-arm-msm, intel-gfx, virtualization, linux-kernel On Sat, Nov 15, 2025 at 02:44:56AM +0200, Dmitry Baryshkov wrote: > On Fri, Nov 14, 2025 at 08:30:50AM +0530, Vignesh Raman wrote: > > Uprev mesa to adapt to the latest changes in Mesa CI, including support > > for firmware via LAVA overlays, removal of the python-artifacts job, > > split container and build rules, use lava-job-submitter container, > > and various misc fixes. > > > > Co-developed-by: Dmitry Baryshkov <dmitry.baryshkov@oss.qualcomm.com> > > Signed-off-by: Dmitry Baryshkov <dmitry.baryshkov@oss.qualcomm.com> > > Signed-off-by: Vignesh Raman <vignesh.raman@collabora.com> > > --- > > drivers/gpu/drm/ci/build.yml | 22 +++--- > > drivers/gpu/drm/ci/container.yml | 28 ++++++-- > > drivers/gpu/drm/ci/gitlab-ci.yml | 95 ++++++++++++++++++++----- > > drivers/gpu/drm/ci/igt_runner.sh | 4 +- > > drivers/gpu/drm/ci/image-tags.yml | 22 +++--- > > drivers/gpu/drm/ci/lava-submit.sh | 101 +++++++++++++-------------- > > drivers/gpu/drm/ci/static-checks.yml | 1 + > > drivers/gpu/drm/ci/test.yml | 18 +++-- > > 8 files changed, 186 insertions(+), 105 deletions(-) > > > > Thanks a lot for taking care of it! > > > Reviewed-by: Dmitry Baryshkov <dmitry.baryshkov@oss.qualcomm.com> Hmm, this causes a lot of failures, see [1] (note, the run is not finished yet): - On some platforms (msm, rk3399, mt8173) the core_setmaster@master-drop-set-root test fails. On other it seems to work correctly - sm8350 seems to have some issues with this uprev, it cant' finish the jobs at all - mt8173 and rockchip:rk3399 seem to have other test failures too Could you please update fails / flakes tor all the affected platforms? Also I'm really interested in the master-drop-set tests. Why do they fail on some of the platforms, but not on all? This seems to be some infrastructure-related issue, since exactly the same kernel with the same IGT passes those tests (at least on msm hardware). [1] https://gitlab.freedesktop.org/drm/msm/-/pipelines/1547684 -- With best wishes Dmitry ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [PATCH v1 1/2] drm/ci: uprev mesa 2025-11-15 3:55 ` Dmitry Baryshkov @ 2025-11-16 15:32 ` Dmitry Baryshkov 0 siblings, 0 replies; 9+ messages in thread From: Dmitry Baryshkov @ 2025-11-16 15:32 UTC (permalink / raw) To: Vignesh Raman Cc: dri-devel, daniels, helen.fornazier, airlied, simona.vetter, lumag, robdclark, robin.clark, guilherme.gallo, sergi.blanch.torne, valentine.burley, linux-mediatek, linux-amlogic, linux-rockchip, amd-gfx, linux-arm-msm, intel-gfx, virtualization, linux-kernel On Sat, Nov 15, 2025 at 05:55:17AM +0200, Dmitry Baryshkov wrote: > On Sat, Nov 15, 2025 at 02:44:56AM +0200, Dmitry Baryshkov wrote: > > On Fri, Nov 14, 2025 at 08:30:50AM +0530, Vignesh Raman wrote: > > > Uprev mesa to adapt to the latest changes in Mesa CI, including support > > > for firmware via LAVA overlays, removal of the python-artifacts job, > > > split container and build rules, use lava-job-submitter container, > > > and various misc fixes. > > > > > > Co-developed-by: Dmitry Baryshkov <dmitry.baryshkov@oss.qualcomm.com> > > > Signed-off-by: Dmitry Baryshkov <dmitry.baryshkov@oss.qualcomm.com> > > > Signed-off-by: Vignesh Raman <vignesh.raman@collabora.com> > > > --- > > > drivers/gpu/drm/ci/build.yml | 22 +++--- > > > drivers/gpu/drm/ci/container.yml | 28 ++++++-- > > > drivers/gpu/drm/ci/gitlab-ci.yml | 95 ++++++++++++++++++++----- > > > drivers/gpu/drm/ci/igt_runner.sh | 4 +- > > > drivers/gpu/drm/ci/image-tags.yml | 22 +++--- > > > drivers/gpu/drm/ci/lava-submit.sh | 101 +++++++++++++-------------- > > > drivers/gpu/drm/ci/static-checks.yml | 1 + > > > drivers/gpu/drm/ci/test.yml | 18 +++-- > > > 8 files changed, 186 insertions(+), 105 deletions(-) > > > > > > > Thanks a lot for taking care of it! > > > > > > Reviewed-by: Dmitry Baryshkov <dmitry.baryshkov@oss.qualcomm.com> > > Hmm, this causes a lot of failures, see [1] (note, the run is not > finished yet): > - On some platforms (msm, rk3399, mt8173) the > core_setmaster@master-drop-set-root test fails. On other it seems to > work correctly > - sm8350 seems to have some issues with this uprev, it cant' finish the > jobs at all > - mt8173 and rockchip:rk3399 seem to have other test failures too I have fixed most of the fallouts, see [2]. But ideally this should be split into two parts: fix before mesa uprev (I think some of them are already there), uprev mesa, applying necessary fixes and then enable apq jobs. At this point I have no idea what's wrong with MT8183. Most likely it needs either a DT fix or a driver fix. The driver fails to probe because it can't build a sensible component chain. On i915 several tests kill the hardware, making it timeout all remaining tests in a bad way. BTW: I see that for some of the drivers we skip the absolute majority of the tests. What are we actually testing on those? > > Could you please update fails / flakes tor all the affected platforms? > Also I'm really interested in the master-drop-set tests. Why do they > fail on some of the platforms, but not on all? This seems to be some > infrastructure-related issue, since exactly the same kernel with the > same IGT passes those tests (at least on msm hardware). > > [1] https://gitlab.freedesktop.org/drm/msm/-/pipelines/1547684 [2] https://gitlab.freedesktop.org/lumag/msm/-/commits/msm-next-lumag-db820c -- With best wishes Dmitry ^ permalink raw reply [flat|nested] 9+ messages in thread
* [PATCH v1 2/2] drm/ci: Move qualcomm baremetal jobs to lava 2025-11-14 3:00 [PATCH v1 0/2] drm/ci: uprev IGT and enable apq8016, apq8096 Vignesh Raman 2025-11-14 3:00 ` [PATCH v1 1/2] drm/ci: uprev mesa Vignesh Raman @ 2025-11-14 3:00 ` Vignesh Raman 2025-11-15 0:50 ` Dmitry Baryshkov 1 sibling, 1 reply; 9+ messages in thread From: Vignesh Raman @ 2025-11-14 3:00 UTC (permalink / raw) To: dri-devel Cc: daniels, helen.fornazier, airlied, simona.vetter, dmitry.baryshkov, lumag, robdclark, robin.clark, guilherme.gallo, sergi.blanch.torne, valentine.burley, linux-mediatek, linux-amlogic, linux-rockchip, amd-gfx, linux-arm-msm, intel-gfx, virtualization, linux-kernel Qualcomm apq8016 and apq8096 DUTS are moved to Collabora lava farm. So enable these jobs to use lava and update expectation files. Co-developed-by: Dmitry Baryshkov <dmitry.baryshkov@oss.qualcomm.com> Signed-off-by: Dmitry Baryshkov <dmitry.baryshkov@oss.qualcomm.com> Signed-off-by: Vignesh Raman <vignesh.raman@collabora.com> --- drivers/gpu/drm/ci/arm64.config | 1 + drivers/gpu/drm/ci/test.yml | 62 ++++++------------- .../gpu/drm/ci/xfails/msm-apq8016-fails.txt | 4 ++ .../gpu/drm/ci/xfails/msm-apq8096-fails.txt | 2 + 4 files changed, 27 insertions(+), 42 deletions(-) diff --git a/drivers/gpu/drm/ci/arm64.config b/drivers/gpu/drm/ci/arm64.config index fddfbd4d2493..b850b88787ad 100644 --- a/drivers/gpu/drm/ci/arm64.config +++ b/drivers/gpu/drm/ci/arm64.config @@ -83,6 +83,7 @@ CONFIG_SC_DISPCC_7180=y CONFIG_SC_GPUCC_7180=y CONFIG_SM_GPUCC_8350=y CONFIG_QCOM_SPMI_ADC5=y +CONFIG_QCOM_SPMI_VADC=y CONFIG_DRM_PARADE_PS8640=y CONFIG_DRM_LONTIUM_LT9611UXC=y CONFIG_PHY_QCOM_USB_HS=y diff --git a/drivers/gpu/drm/ci/test.yml b/drivers/gpu/drm/ci/test.yml index e56895626500..42f75993e190 100644 --- a/drivers/gpu/drm/ci/test.yml +++ b/drivers/gpu/drm/ci/test.yml @@ -67,31 +67,6 @@ - testing:x86_64 - igt:x86_64 -.baremetal-igt-arm64: - extends: - - .baremetal-test-arm64-gl - - .use-debian/baremetal_arm64_test-gl - - .allow_failure_lockdep - timeout: "1h30m" - rules: - - !reference [.scheduled_pipeline-rules, rules] - - !reference [.google-freedreno-farm-rules, rules] - - when: on_success - variables: - FDO_CI_CONCURRENT: 10 - HWCI_TEST_SCRIPT: "/install/igt_runner.sh" - S3_ARTIFACT_NAME: "arm64/kernel-files" - BM_KERNEL: https://${PIPELINE_ARTIFACTS_BASE}/arm64/Image.gz - BM_CMDLINE: "ip=dhcp console=ttyMSM0,115200n8 $BM_KERNEL_EXTRA_ARGS root=/dev/nfs rw nfsrootdebug nfsroot=,tcp,nfsvers=4.2 init=/init $BM_KERNELARGS" - FARM: google - needs: - - debian/baremetal_arm64_test-gl - - job: testing:arm64 - artifacts: false - - igt:arm64 - tags: - - $RUNNER_TAG - .software-driver: stage: software-driver extends: @@ -149,34 +124,37 @@ msm:sc7180-trogdor-kingoftown: msm:apq8016: extends: - - .baremetal-igt-arm64 + - .lava-igt:arm64 stage: msm + parallel: 3 variables: - DEVICE_TYPE: apq8016-sbc-usb-host + BOOT_METHOD: fastboot + DEVICE_TYPE: dragonboard-410c DRIVER_NAME: msm - BM_DTB: https://${PIPELINE_ARTIFACTS_BASE}/arm64/${DEVICE_TYPE}.dtb + DTB: apq8016-sbc-usb-host + FARM: collabora GPU_VERSION: apq8016 - # disabling unused clocks congests with the MDSS runtime PM trying to - # disable those clocks and causes boot to fail. - # Reproducer: DRM_MSM=y, DRM_I2C_ADV7511=m - BM_KERNEL_EXTRA_ARGS: clk_ignore_unused - RUNNER_TAG: google-freedreno-db410c - script: - - ./install/bare-metal/fastboot.sh || exit $? + KERNEL_IMAGE_NAME: "Image.gz" + KERNEL_IMAGE_TYPE: "" + RUNNER_TAG: mesa-ci-x86-64-lava-dragonboard-410c + LAVA_FIRMWARE: qcom-lava msm:apq8096: extends: - - .baremetal-igt-arm64 + - .lava-igt:arm64 stage: msm + parallel: 3 variables: - DEVICE_TYPE: apq8096-db820c + BOOT_METHOD: fastboot + DEVICE_TYPE: dragonboard-820c DRIVER_NAME: msm - BM_KERNEL_EXTRA_ARGS: maxcpus=2 - BM_DTB: https://${PIPELINE_ARTIFACTS_BASE}/arm64/${DEVICE_TYPE}.dtb + DTB: apq8096-db820c + FARM: collabora GPU_VERSION: apq8096 - RUNNER_TAG: google-freedreno-db820c - script: - - ./install/bare-metal/fastboot.sh || exit $? + KERNEL_IMAGE_NAME: "Image.gz" + KERNEL_IMAGE_TYPE: "" + RUNNER_TAG: mesa-ci-x86-64-lava-dragonboard-820c + LAVA_FIRMWARE: qcom-lava msm:sm8350-hdk: extends: diff --git a/drivers/gpu/drm/ci/xfails/msm-apq8016-fails.txt b/drivers/gpu/drm/ci/xfails/msm-apq8016-fails.txt index 72c469021b66..4546363447ff 100644 --- a/drivers/gpu/drm/ci/xfails/msm-apq8016-fails.txt +++ b/drivers/gpu/drm/ci/xfails/msm-apq8016-fails.txt @@ -1,5 +1,9 @@ +core_setmaster@master-drop-set-user,Fail kms_3d,Fail +kms_cursor_legacy@forked-move,Fail +kms_cursor_legacy@single-bo,Fail kms_force_connector_basic@force-edid,Fail kms_hdmi_inject@inject-4k,Fail kms_lease@lease-uevent,Fail +msm/msm_mapping@memptrs,Fail msm/msm_mapping@ring,Fail diff --git a/drivers/gpu/drm/ci/xfails/msm-apq8096-fails.txt b/drivers/gpu/drm/ci/xfails/msm-apq8096-fails.txt index 2893f98a6b97..0d5cb2a87e67 100644 --- a/drivers/gpu/drm/ci/xfails/msm-apq8096-fails.txt +++ b/drivers/gpu/drm/ci/xfails/msm-apq8096-fails.txt @@ -1,2 +1,4 @@ +core_setmaster@master-drop-set-user,Fail kms_3d,Fail kms_lease@lease-uevent,Fail +msm/msm_mapping@memptrs,Fail -- 2.51.0 ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [PATCH v1 2/2] drm/ci: Move qualcomm baremetal jobs to lava 2025-11-14 3:00 ` [PATCH v1 2/2] drm/ci: Move qualcomm baremetal jobs to lava Vignesh Raman @ 2025-11-15 0:50 ` Dmitry Baryshkov 0 siblings, 0 replies; 9+ messages in thread From: Dmitry Baryshkov @ 2025-11-15 0:50 UTC (permalink / raw) To: Vignesh Raman Cc: dri-devel, daniels, helen.fornazier, airlied, simona.vetter, lumag, robdclark, robin.clark, guilherme.gallo, sergi.blanch.torne, valentine.burley, linux-mediatek, linux-amlogic, linux-rockchip, amd-gfx, linux-arm-msm, intel-gfx, virtualization, linux-kernel On Fri, Nov 14, 2025 at 08:30:51AM +0530, Vignesh Raman wrote: > Qualcomm apq8016 and apq8096 DUTS are moved to Collabora lava > farm. So enable these jobs to use lava and update expectation > files. > > Co-developed-by: Dmitry Baryshkov <dmitry.baryshkov@oss.qualcomm.com> > Signed-off-by: Dmitry Baryshkov <dmitry.baryshkov@oss.qualcomm.com> > Signed-off-by: Vignesh Raman <vignesh.raman@collabora.com> > --- > drivers/gpu/drm/ci/arm64.config | 1 + > drivers/gpu/drm/ci/test.yml | 62 ++++++------------- > .../gpu/drm/ci/xfails/msm-apq8016-fails.txt | 4 ++ > .../gpu/drm/ci/xfails/msm-apq8096-fails.txt | 2 + > 4 files changed, 27 insertions(+), 42 deletions(-) > Reviewed-by: Dmitry Baryshkov <dmitry.baryshkov@oss.qualcomm.com> I'll try picking up these patches through the drm/msm tree, if noone objects. -- With best wishes Dmitry ^ permalink raw reply [flat|nested] 9+ messages in thread
* [PATCH v1 0/2] drm/ci: uprev mesa and modify gitlab rules @ 2024-11-28 4:20 Vignesh Raman 2024-11-28 4:20 ` [PATCH v1 1/2] drm/ci: uprev mesa Vignesh Raman 0 siblings, 1 reply; 9+ messages in thread From: Vignesh Raman @ 2024-11-28 4:20 UTC (permalink / raw) To: dri-devel Cc: daniels, helen.koike, airlied, daniel, robdclark, guilherme.gallo, sergi.blanch.torne, jani.nikula, dmitry.baryshkov, mripard, linux-kernel Uprev mesa to adapt to the latest changes in mesa-ci, including new container jobs and stages. Update the lava-submit script to align with recent mesa-ci changes for using LAVA rootfs overlays. Modify gitLab rules to include scheduled pipelines. Pipeline link, https://gitlab.freedesktop.org/vigneshraman/linux/-/pipelines/1318024 Vignesh Raman (2): drm/ci: uprev mesa drm/ci: update gitlab rules drivers/gpu/drm/ci/build.sh | 2 +- drivers/gpu/drm/ci/build.yml | 104 ++++++++++++++++- drivers/gpu/drm/ci/container.yml | 22 ++-- drivers/gpu/drm/ci/gitlab-ci.yml | 185 ++++++++++++++++-------------- drivers/gpu/drm/ci/image-tags.yml | 11 +- drivers/gpu/drm/ci/lava-submit.sh | 99 +++++++++++----- drivers/gpu/drm/ci/test.yml | 33 +++--- 7 files changed, 314 insertions(+), 142 deletions(-) -- 2.43.0 ^ permalink raw reply [flat|nested] 9+ messages in thread
* [PATCH v1 1/2] drm/ci: uprev mesa 2024-11-28 4:20 [PATCH v1 0/2] drm/ci: uprev mesa and modify gitlab rules Vignesh Raman @ 2024-11-28 4:20 ` Vignesh Raman 2024-12-03 14:14 ` Helen Mae Koike Fornazier 0 siblings, 1 reply; 9+ messages in thread From: Vignesh Raman @ 2024-11-28 4:20 UTC (permalink / raw) To: dri-devel Cc: daniels, helen.koike, airlied, daniel, robdclark, guilherme.gallo, sergi.blanch.torne, jani.nikula, dmitry.baryshkov, mripard, linux-kernel Uprev mesa to adapt to the latest changes in mesa ci which includes new container jobs and stages. Also update lava-submit script to adapt to the recent changes in mesa to use LAVA rootfs overlays. Signed-off-by: Vignesh Raman <vignesh.raman@collabora.com> --- drivers/gpu/drm/ci/build.sh | 2 +- drivers/gpu/drm/ci/build.yml | 103 +++++++++++++++++++++++++++++- drivers/gpu/drm/ci/container.yml | 22 +++---- drivers/gpu/drm/ci/gitlab-ci.yml | 71 ++++++++++++++------ drivers/gpu/drm/ci/image-tags.yml | 11 +++- drivers/gpu/drm/ci/lava-submit.sh | 99 ++++++++++++++++++++-------- drivers/gpu/drm/ci/test.yml | 13 ++-- 7 files changed, 255 insertions(+), 66 deletions(-) diff --git a/drivers/gpu/drm/ci/build.sh b/drivers/gpu/drm/ci/build.sh index 139b81db6312..19fe01257ab9 100644 --- a/drivers/gpu/drm/ci/build.sh +++ b/drivers/gpu/drm/ci/build.sh @@ -132,7 +132,7 @@ fi # Pass needed files to the test stage mkdir -p install cp -rfv .gitlab-ci/* install/. -cp -rfv ci/* install/. +cp -rfv bin/ci/* install/. cp -rfv install/common install/ci-common cp -rfv drivers/gpu/drm/ci/* install/. diff --git a/drivers/gpu/drm/ci/build.yml b/drivers/gpu/drm/ci/build.yml index 9c198239033d..771ecfc5008d 100644 --- a/drivers/gpu/drm/ci/build.yml +++ b/drivers/gpu/drm/ci/build.yml @@ -2,7 +2,7 @@ extends: - .build-rules - .container+build-rules - stage: build + stage: build-only artifacts: paths: - artifacts @@ -110,3 +110,104 @@ build-nodebugfs:arm64: build:x86_64: extends: .build:x86_64 + +# Disable build jobs that we won't use +alpine-build-testing: + rules: + - when: never + +debian-android: + rules: + - when: never + +debian-arm32: + rules: + - when: never + +debian-arm32-asan: + rules: + - when: never + +debian-arm64: + rules: + - when: never + +debian-arm64-asan: + rules: + - when: never + +debian-arm64-build-test: + rules: + - when: never + +debian-arm64-release: + rules: + - when: never + +debian-build-testing: + rules: + - when: never + +debian-clang: + rules: + - when: never + +debian-clang-release: + rules: + - when: never + +debian-no-libdrm: + rules: + - when: never + +debian-ppc64el: + rules: + - when: never + +debian-release: + rules: + - when: never + +debian-s390x: + rules: + - when: never + +debian-testing: + rules: + - when: never + +debian-testing-asan: + rules: + - when: never + +debian-testing-msan: + rules: + - when: never + +debian-vulkan: + rules: + - when: never + +debian-x86_32: + rules: + - when: never + +fedora-release: + rules: + - when: never + +rustfmt: + rules: + - when: never + +shader-db: + rules: + - when: never + +windows-msvc: + rules: + - when: never + +yaml-toml-shell-test: + rules: + - when: never diff --git a/drivers/gpu/drm/ci/container.yml b/drivers/gpu/drm/ci/container.yml index 2a94f54ce4cf..07dc13ff865d 100644 --- a/drivers/gpu/drm/ci/container.yml +++ b/drivers/gpu/drm/ci/container.yml @@ -24,7 +24,7 @@ alpine/x86_64_build: rules: - when: never -debian/x86_64_test-vk: +debian/arm64_test-gl: rules: - when: never @@ -32,7 +32,15 @@ debian/arm64_test-vk: rules: - when: never -debian/arm64_test-gl: +debian/ppc64el_build: + rules: + - when: never + +debian/s390x_build: + rules: + - when: never + +debian/x86_64_test-vk: rules: - when: never @@ -56,14 +64,6 @@ windows_test_msvc: rules: - when: never -.debian/x86_64_build-mingw: - rules: - - when: never - -rustfmt: - rules: - - when: never - windows_msvc: rules: - - when: never \ No newline at end of file + - when: never diff --git a/drivers/gpu/drm/ci/gitlab-ci.yml b/drivers/gpu/drm/ci/gitlab-ci.yml index 90bde9f00cc3..86d8c5d8ce3b 100644 --- a/drivers/gpu/drm/ci/gitlab-ci.yml +++ b/drivers/gpu/drm/ci/gitlab-ci.yml @@ -1,6 +1,6 @@ variables: DRM_CI_PROJECT_PATH: &drm-ci-project-path mesa/mesa - DRM_CI_COMMIT_SHA: &drm-ci-commit-sha c6a9a9c3bce90923f7700219354e0b6e5a3c9ba6 + DRM_CI_COMMIT_SHA: &drm-ci-commit-sha e1a8fd80d411a5ff8fa19ffcf09516ac5099a25c UPSTREAM_REPO: https://gitlab.freedesktop.org/drm/kernel.git TARGET_BRANCH: drm-next @@ -31,11 +31,7 @@ variables: PIPELINE_ARTIFACTS_BASE: ${S3_HOST}/${S3_ARTIFACTS_BUCKET}/${CI_PROJECT_PATH}/${CI_PIPELINE_ID} # per-job artifact storage on MinIO JOB_ARTIFACTS_BASE: ${PIPELINE_ARTIFACTS_BASE}/${CI_JOB_ID} - # default kernel for rootfs before injecting the current kernel tree - KERNEL_REPO: "gfx-ci/linux" - KERNEL_TAG: "v6.6.21-mesa-f8ea" KERNEL_IMAGE_BASE: https://${S3_HOST}/${S3_KERNEL_BUCKET}/${KERNEL_REPO}/${KERNEL_TAG} - PKG_REPO_REV: "3cc12a2a" LAVA_TAGS: subset-1-gfx LAVA_JOB_PRIORITY: 30 ARTIFACTS_BASE_URL: https://${CI_PROJECT_ROOT_NAMESPACE}.${CI_PAGES_DOMAIN}/-/${CI_PROJECT_NAME}/-/jobs/${CI_JOB_ID}/artifacts @@ -59,7 +55,7 @@ default: - cd $CI_PROJECT_DIR - curl --output - $DRM_CI_PROJECT_URL/-/archive/$DRM_CI_COMMIT_SHA/mesa-$DRM_CI_COMMIT_SHA.tar.gz | tar -xz - mv mesa-$DRM_CI_COMMIT_SHA/.gitlab-ci* . - - mv mesa-$DRM_CI_COMMIT_SHA/bin/ci . + - mv mesa-$DRM_CI_COMMIT_SHA/bin . - rm -rf mesa-$DRM_CI_COMMIT_SHA/ - echo -e "\e[0Ksection_end:$(date +%s):drm_ci_download_section\r\e[0K" @@ -85,6 +81,7 @@ include: - project: *drm-ci-project-path ref: *drm-ci-commit-sha file: + - '/.gitlab-ci/build/gitlab-ci.yml' - '/.gitlab-ci/container/gitlab-ci.yml' - '/.gitlab-ci/farm-rules.yml' - '/.gitlab-ci/lava/lava-gitlab-ci.yml' @@ -115,9 +112,10 @@ include: stages: - sanity - container - - code-validation - git-archive - - build + - build-for-tests + - build-only + - code-validation - amdgpu - i915 - mediatek @@ -264,30 +262,63 @@ sanity: rules: - if: *is-pre-merge when: on_success - # Other cases default to never + - when: never variables: GIT_STRATEGY: none script: # ci-fairy check-commits --junit-xml=check-commits.xml - ci-fairy check-merge-request --require-allow-collaboration --junit-xml=check-merge-request.xml + - | + set -eu + image_tags=( + ALPINE_X86_64_LAVA_SSH_TAG + CONTAINER_TAG + DEBIAN_BASE_TAG + DEBIAN_BUILD_TAG + DEBIAN_PYUTILS_TAG + DEBIAN_TEST_GL_TAG + KERNEL_ROOTFS_TAG + KERNEL_TAG + PKG_REPO_REV + ) + for var in "${image_tags[@]}" + do + if [ "$(echo -n "${!var}" | wc -c)" -gt 20 ] + then + echo "$var is too long; please make sure it is at most 20 chars." + exit 1 + fi + done artifacts: when: on_failure reports: junit: check-*.xml + tags: + - placeholder-job -# Rules for tests that should not block merging, but should be available to -# optionally run with the "play" button in the UI in pre-merge non-marge -# pipelines. This should appear in "extends:" after any includes of -# test-source-dep.yml rules, so that these rules replace those. -.test-manual-mr: + +mr-label-maker-test: + extends: + - .fdo.ci-fairy + stage: sanity rules: - - !reference [.no_scheduled_pipelines-rules, rules] - - if: *is-forked-branch-or-pre-merge-not-for-marge - when: manual + - !reference [.mr-label-maker-rules, rules] variables: - JOB_TIMEOUT: 80 - + GIT_STRATEGY: fetch + timeout: 10m + script: + - set -eu + - python3 -m venv .venv + - source .venv/bin/activate + - pip install git+https://gitlab.freedesktop.org/freedesktop/mr-label-maker + - mr-label-maker --dry-run --mr $CI_MERGE_REQUEST_IID # Jobs that need to pass before spending hardware resources on further testing .required-for-hardware-jobs: - needs: [] + needs: + - job: clang-format + optional: true + - job: rustfmt + optional: true + - job: toml-lint + optional: true diff --git a/drivers/gpu/drm/ci/image-tags.yml b/drivers/gpu/drm/ci/image-tags.yml index 8d8b9e71852e..528ef44285bb 100644 --- a/drivers/gpu/drm/ci/image-tags.yml +++ b/drivers/gpu/drm/ci/image-tags.yml @@ -1,5 +1,5 @@ variables: - CONTAINER_TAG: "2024-09-09-uprevs" + CONTAINER_TAG: "20241127-mesa-uprev" DEBIAN_X86_64_BUILD_BASE_IMAGE: "debian/x86_64_build-base" DEBIAN_BASE_TAG: "${CONTAINER_TAG}" @@ -7,9 +7,16 @@ variables: DEBIAN_BUILD_TAG: "${CONTAINER_TAG}" KERNEL_ROOTFS_TAG: "${CONTAINER_TAG}" + # default kernel for rootfs before injecting the current kernel tree + KERNEL_TAG: "v6.6.21-mesa-f8ea" + KERNEL_REPO: "gfx-ci/linux" + PKG_REPO_REV: "bca9635d" DEBIAN_X86_64_TEST_BASE_IMAGE: "debian/x86_64_test-base" DEBIAN_X86_64_TEST_IMAGE_GL_PATH: "debian/x86_64_test-gl" DEBIAN_TEST_GL_TAG: "${CONTAINER_TAG}" - ALPINE_X86_64_LAVA_SSH_TAG: "${CONTAINER_TAG}" \ No newline at end of file + DEBIAN_PYUTILS_IMAGE: "debian/x86_64_pyutils" + DEBIAN_PYUTILS_TAG: "${CONTAINER_TAG}" + + ALPINE_X86_64_LAVA_SSH_TAG: "${CONTAINER_TAG}" diff --git a/drivers/gpu/drm/ci/lava-submit.sh b/drivers/gpu/drm/ci/lava-submit.sh index 6add15083c78..a1404a2d8cee 100755 --- a/drivers/gpu/drm/ci/lava-submit.sh +++ b/drivers/gpu/drm/ci/lava-submit.sh @@ -1,58 +1,105 @@ -#!/bin/bash +#!/usr/bin/env bash # SPDX-License-Identifier: MIT +# shellcheck disable=SC2086 # we want word splitting +# shellcheck disable=SC1091 # paths only become valid at runtime -set -e -set -x +# If we run in the fork (not from mesa or Marge-bot), reuse mainline kernel and rootfs, if exist. +_check_artifact_path() { + _url="https://${1}/${2}" + if curl -s -o /dev/null -I -L -f --retry 4 --retry-delay 15 "${_url}"; then + echo -n "${_url}" + fi +} -# Try to use the kernel and rootfs built in mainline first, so we're more -# likely to hit cache -if curl -L --retry 4 -f --retry-all-errors --retry-delay 60 -s "https://${BASE_SYSTEM_MAINLINE_HOST_PATH}/done"; then - BASE_SYSTEM_HOST_PATH="${BASE_SYSTEM_MAINLINE_HOST_PATH}" -else - BASE_SYSTEM_HOST_PATH="${BASE_SYSTEM_FORK_HOST_PATH}" -fi +get_path_to_artifact() { + _mainline_artifact="$(_check_artifact_path ${BASE_SYSTEM_MAINLINE_HOST_PATH} ${1})" + if [ -n "${_mainline_artifact}" ]; then + echo -n "${_mainline_artifact}" + return + fi + _fork_artifact="$(_check_artifact_path ${BASE_SYSTEM_FORK_HOST_PATH} ${1})" + if [ -n "${_fork_artifact}" ]; then + echo -n "${_fork_artifact}" + return + fi + set +x + error "Sorry, I couldn't find a viable built path for ${1} in either mainline or a fork." >&2 + echo "" >&2 + echo "If you're working on CI, this probably means that you're missing a dependency:" >&2 + echo "this job ran ahead of the job which was supposed to upload that artifact." >&2 + echo "" >&2 + echo "If you aren't working on CI, please ping @mesa/ci-helpers to see if we can help." >&2 + echo "" >&2 + echo "This job is going to fail, because I can't find the resources I need. Sorry." >&2 + set -x + exit 1 +} + +. "${SCRIPTS_DIR}/setup-test-env.sh" + +section_start prepare_rootfs "Preparing root filesystem" + +set -ex + +section_switch rootfs "Assembling root filesystem" +ROOTFS_URL="$(get_path_to_artifact lava-rootfs.tar.zst)" +[ $? != 1 ] || exit 1 rm -rf results mkdir -p results/job-rootfs-overlay/ +artifacts/ci-common/generate-env.sh > results/job-rootfs-overlay/set-job-env-vars.sh cp artifacts/ci-common/capture-devcoredump.sh results/job-rootfs-overlay/ cp artifacts/ci-common/init-*.sh results/job-rootfs-overlay/ cp artifacts/ci-common/intel-gpu-freq.sh results/job-rootfs-overlay/ +cp artifacts/ci-common/kdl.sh results/job-rootfs-overlay/ cp "$SCRIPTS_DIR"/setup-test-env.sh results/job-rootfs-overlay/ -# Prepare env vars for upload. -section_start variables "Variables passed through:" -KERNEL_IMAGE_BASE="https://${BASE_SYSTEM_HOST_PATH}" \ - artifacts/ci-common/generate-env.sh | tee results/job-rootfs-overlay/set-job-env-vars.sh -section_end variables - tar zcf job-rootfs-overlay.tar.gz -C results/job-rootfs-overlay/ . ci-fairy s3cp --token-file "${S3_JWT_FILE}" job-rootfs-overlay.tar.gz "https://${JOB_ROOTFS_OVERLAY_PATH}" +# Prepare env vars for upload. +section_switch variables "Environment variables passed through to device:" +cat results/job-rootfs-overlay/set-job-env-vars.sh + +section_switch lava_submit "Submitting job for scheduling" + touch results/lava.log tail -f results/lava.log & - PYTHONPATH=artifacts/ artifacts/lava/lava_job_submitter.py \ - submit \ + --farm "${FARM}" \ + --device-type "${DEVICE_TYPE}" \ + --boot-method "${BOOT_METHOD}" \ + --job-timeout-min ${JOB_TIMEOUT:-30} \ --dump-yaml \ --pipeline-info "$CI_JOB_NAME: $CI_PIPELINE_URL on $CI_COMMIT_REF_NAME ${CI_NODE_INDEX}/${CI_NODE_TOTAL}" \ - --rootfs-url-prefix "https://${BASE_SYSTEM_HOST_PATH}" \ + --rootfs-url "${ROOTFS_URL}" \ --kernel-url-prefix "https://${PIPELINE_ARTIFACTS_BASE}/${DEBIAN_ARCH}" \ - --build-url "${FDO_HTTP_CACHE_URI:-}https://${PIPELINE_ARTIFACTS_BASE}/${DEBIAN_ARCH}/kernel-files.tar.zst" \ - --job-rootfs-overlay-url "${FDO_HTTP_CACHE_URI:-}https://${JOB_ROOTFS_OVERLAY_PATH}" \ - --job-timeout-min ${JOB_TIMEOUT:-80} \ + --kernel-external "${EXTERNAL_KERNEL_TAG}" \ --first-stage-init artifacts/ci-common/init-stage1.sh \ - --ci-project-dir "${CI_PROJECT_DIR}" \ - --device-type "${DEVICE_TYPE}" \ - --farm "${FARM}" \ --dtb-filename "${DTB}" \ --jwt-file "${S3_JWT_FILE}" \ --kernel-image-name "${KERNEL_IMAGE_NAME}" \ --kernel-image-type "${KERNEL_IMAGE_TYPE}" \ - --boot-method "${BOOT_METHOD}" \ --visibility-group "${VISIBILITY_GROUP}" \ --lava-tags "${LAVA_TAGS}" \ --mesa-job-name "$CI_JOB_NAME" \ --structured-log-file "results/lava_job_detail.json" \ --ssh-client-image "${LAVA_SSH_CLIENT_IMAGE}" \ + --project-name "${CI_PROJECT_NAME}" \ + --starting-section "${CURRENT_SECTION}" \ + --job-submitted-at "${CI_JOB_STARTED_AT}" \ + - append-overlay \ + --name=kernel-build \ + --url="${FDO_HTTP_CACHE_URI:-}https://${PIPELINE_ARTIFACTS_BASE}/${DEBIAN_ARCH}/kernel-files.tar.zst" \ + --compression=zstd \ + --path="${CI_PROJECT_DIR}" \ + --format=tar \ + - append-overlay \ + --name=job-overlay \ + --url="https://${JOB_ROOTFS_OVERLAY_PATH}" \ + --compression=gz \ + --path="/" \ + --format=tar \ + - submit \ >> results/lava.log diff --git a/drivers/gpu/drm/ci/test.yml b/drivers/gpu/drm/ci/test.yml index f0ef60c8f56d..42a8e7f93714 100644 --- a/drivers/gpu/drm/ci/test.yml +++ b/drivers/gpu/drm/ci/test.yml @@ -1,21 +1,21 @@ .test-rules: rules: - - if: '$FD_FARM == "offline" && $RUNNER_TAG =~ /^google-freedreno-/' - when: never - - if: '$COLLABORA_FARM == "offline" && $RUNNER_TAG =~ /^mesa-ci-x86-64-lava-/' - when: never - !reference [.no_scheduled_pipelines-rules, rules] + - !reference [.collabora-farm-rules, rules] + - !reference [.google-freedreno-farm-rules, rules] - when: on_success .lava-test: extends: - .test-rules + - .build-rules + - .container+build-rules timeout: "1h30m" script: # Note: Build dir (and thus install) may be dirty due to GIT_STRATEGY - rm -rf install - tar -xf artifacts/install.tar - - mv install/* artifacts/. + - mv -n install/* artifacts/. # Override it with our lava-submit.sh script - ./artifacts/lava-submit.sh @@ -32,6 +32,7 @@ - alpine/x86_64_lava_ssh_client - kernel+rootfs_arm32 - debian/x86_64_build + - python-test - testing:arm32 - igt:arm32 @@ -48,6 +49,7 @@ - alpine/x86_64_lava_ssh_client - kernel+rootfs_arm64 - debian/x86_64_build + - python-test - testing:arm64 - igt:arm64 @@ -64,6 +66,7 @@ - alpine/x86_64_lava_ssh_client - kernel+rootfs_x86_64 - debian/x86_64_build + - python-test - testing:x86_64 - igt:x86_64 -- 2.43.0 ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [PATCH v1 1/2] drm/ci: uprev mesa 2024-11-28 4:20 ` [PATCH v1 1/2] drm/ci: uprev mesa Vignesh Raman @ 2024-12-03 14:14 ` Helen Mae Koike Fornazier 0 siblings, 0 replies; 9+ messages in thread From: Helen Mae Koike Fornazier @ 2024-12-03 14:14 UTC (permalink / raw) To: Vignesh Raman Cc: dri-devel, daniels, airlied, daniel, robdclark, guilherme.gallo, sergi.blanch.torne, jani.nikula, dmitry.baryshkov, mripard, linux-kernel ---- On Thu, 28 Nov 2024 01:20:20 -0300 Vignesh Raman wrote --- > Uprev mesa to adapt to the latest changes in mesa ci > which includes new container jobs and stages. Also update > lava-submit script to adapt to the recent changes in mesa > to use LAVA rootfs overlays. > > Signed-off-by: Vignesh Raman vignesh.raman@collabora.com> > --- > drivers/gpu/drm/ci/build.sh | 2 +- > drivers/gpu/drm/ci/build.yml | 103 +++++++++++++++++++++++++++++- > drivers/gpu/drm/ci/container.yml | 22 +++---- > drivers/gpu/drm/ci/gitlab-ci.yml | 71 ++++++++++++++------ > drivers/gpu/drm/ci/image-tags.yml | 11 +++- > drivers/gpu/drm/ci/lava-submit.sh | 99 ++++++++++++++++++++-------- > drivers/gpu/drm/ci/test.yml | 13 ++-- > 7 files changed, 255 insertions(+), 66 deletions(-) > > diff --git a/drivers/gpu/drm/ci/build.sh b/drivers/gpu/drm/ci/build.sh > index 139b81db6312..19fe01257ab9 100644 > --- a/drivers/gpu/drm/ci/build.sh > +++ b/drivers/gpu/drm/ci/build.sh > @@ -132,7 +132,7 @@ fi > # Pass needed files to the test stage > mkdir -p install > cp -rfv .gitlab-ci/* install/. > -cp -rfv ci/* install/. > +cp -rfv bin/ci/* install/. > cp -rfv install/common install/ci-common > cp -rfv drivers/gpu/drm/ci/* install/. > > diff --git a/drivers/gpu/drm/ci/build.yml b/drivers/gpu/drm/ci/build.yml > index 9c198239033d..771ecfc5008d 100644 > --- a/drivers/gpu/drm/ci/build.yml > +++ b/drivers/gpu/drm/ci/build.yml > @@ -2,7 +2,7 @@ > extends: > - .build-rules > - .container+build-rules > - stage: build > + stage: build-only > artifacts: > paths: > - artifacts > @@ -110,3 +110,104 @@ build-nodebugfs:arm64: > > build:x86_64: > extends: .build:x86_64 > + > +# Disable build jobs that we won't use > +alpine-build-testing: > + rules: > + - when: never > + > +debian-android: > + rules: > + - when: never > + > +debian-arm32: > + rules: > + - when: never > + > +debian-arm32-asan: > + rules: > + - when: never > + > +debian-arm64: > + rules: > + - when: never > + > +debian-arm64-asan: > + rules: > + - when: never > + > +debian-arm64-build-test: > + rules: > + - when: never > + > +debian-arm64-release: > + rules: > + - when: never > + > +debian-build-testing: > + rules: > + - when: never > + > +debian-clang: > + rules: > + - when: never > + > +debian-clang-release: > + rules: > + - when: never > + > +debian-no-libdrm: > + rules: > + - when: never > + > +debian-ppc64el: > + rules: > + - when: never > + > +debian-release: > + rules: > + - when: never > + > +debian-s390x: > + rules: > + - when: never > + > +debian-testing: > + rules: > + - when: never > + > +debian-testing-asan: > + rules: > + - when: never > + > +debian-testing-msan: > + rules: > + - when: never > + > +debian-vulkan: > + rules: > + - when: never > + > +debian-x86_32: > + rules: > + - when: never > + > +fedora-release: > + rules: > + - when: never > + > +rustfmt: > + rules: > + - when: never > + > +shader-db: > + rules: > + - when: never > + > +windows-msvc: > + rules: > + - when: never > + > +yaml-toml-shell-test: > + rules: > + - when: never Well, I don't really like this list, but I guess we don't have a choice for now... > diff --git a/drivers/gpu/drm/ci/container.yml b/drivers/gpu/drm/ci/container.yml > index 2a94f54ce4cf..07dc13ff865d 100644 > --- a/drivers/gpu/drm/ci/container.yml > +++ b/drivers/gpu/drm/ci/container.yml > @@ -24,7 +24,7 @@ alpine/x86_64_build: > rules: > - when: never > > -debian/x86_64_test-vk: > +debian/arm64_test-gl: > rules: > - when: never > > @@ -32,7 +32,15 @@ debian/arm64_test-vk: > rules: > - when: never > > -debian/arm64_test-gl: > +debian/ppc64el_build: > + rules: > + - when: never > + > +debian/s390x_build: > + rules: > + - when: never > + > +debian/x86_64_test-vk: > rules: > - when: never > > @@ -56,14 +64,6 @@ windows_test_msvc: > rules: > - when: never > > -.debian/x86_64_build-mingw: > - rules: > - - when: never > - > -rustfmt: > - rules: > - - when: never > - > windows_msvc: > rules: > - - when: never > \ No newline at end of file > + - when: never > diff --git a/drivers/gpu/drm/ci/gitlab-ci.yml b/drivers/gpu/drm/ci/gitlab-ci.yml > index 90bde9f00cc3..86d8c5d8ce3b 100644 > --- a/drivers/gpu/drm/ci/gitlab-ci.yml > +++ b/drivers/gpu/drm/ci/gitlab-ci.yml > @@ -1,6 +1,6 @@ > variables: > DRM_CI_PROJECT_PATH: &drm-ci-project-path mesa/mesa > - DRM_CI_COMMIT_SHA: &drm-ci-commit-sha c6a9a9c3bce90923f7700219354e0b6e5a3c9ba6 > + DRM_CI_COMMIT_SHA: &drm-ci-commit-sha e1a8fd80d411a5ff8fa19ffcf09516ac5099a25c > > UPSTREAM_REPO: https://gitlab.freedesktop.org/drm/kernel.git > TARGET_BRANCH: drm-next > @@ -31,11 +31,7 @@ variables: > PIPELINE_ARTIFACTS_BASE: ${S3_HOST}/${S3_ARTIFACTS_BUCKET}/${CI_PROJECT_PATH}/${CI_PIPELINE_ID} > # per-job artifact storage on MinIO > JOB_ARTIFACTS_BASE: ${PIPELINE_ARTIFACTS_BASE}/${CI_JOB_ID} > - # default kernel for rootfs before injecting the current kernel tree > - KERNEL_REPO: "gfx-ci/linux" > - KERNEL_TAG: "v6.6.21-mesa-f8ea" > KERNEL_IMAGE_BASE: https://${S3_HOST}/${S3_KERNEL_BUCKET}/${KERNEL_REPO}/${KERNEL_TAG} > - PKG_REPO_REV: "3cc12a2a" > LAVA_TAGS: subset-1-gfx > LAVA_JOB_PRIORITY: 30 > ARTIFACTS_BASE_URL: https://${CI_PROJECT_ROOT_NAMESPACE}.${CI_PAGES_DOMAIN}/-/${CI_PROJECT_NAME}/-/jobs/${CI_JOB_ID}/artifacts > @@ -59,7 +55,7 @@ default: > - cd $CI_PROJECT_DIR > - curl --output - $DRM_CI_PROJECT_URL/-/archive/$DRM_CI_COMMIT_SHA/mesa-$DRM_CI_COMMIT_SHA.tar.gz | tar -xz > - mv mesa-$DRM_CI_COMMIT_SHA/.gitlab-ci* . > - - mv mesa-$DRM_CI_COMMIT_SHA/bin/ci . > + - mv mesa-$DRM_CI_COMMIT_SHA/bin . > - rm -rf mesa-$DRM_CI_COMMIT_SHA/ > - echo -e "\e[0Ksection_end:$(date +%s):drm_ci_download_section\r\e[0K" > > @@ -85,6 +81,7 @@ include: > - project: *drm-ci-project-path > ref: *drm-ci-commit-sha > file: > + - '/.gitlab-ci/build/gitlab-ci.yml' > - '/.gitlab-ci/container/gitlab-ci.yml' > - '/.gitlab-ci/farm-rules.yml' > - '/.gitlab-ci/lava/lava-gitlab-ci.yml' > @@ -115,9 +112,10 @@ include: > stages: > - sanity > - container > - - code-validation > - git-archive > - - build > + - build-for-tests > + - build-only > + - code-validation > - amdgpu > - i915 > - mediatek > @@ -264,30 +262,63 @@ sanity: > rules: > - if: *is-pre-merge > when: on_success > - # Other cases default to never > + - when: never > variables: > GIT_STRATEGY: none > script: > # ci-fairy check-commits --junit-xml=check-commits.xml > - ci-fairy check-merge-request --require-allow-collaboration --junit-xml=check-merge-request.xml > + - | > + set -eu > + image_tags=( > + ALPINE_X86_64_LAVA_SSH_TAG > + CONTAINER_TAG > + DEBIAN_BASE_TAG > + DEBIAN_BUILD_TAG > + DEBIAN_PYUTILS_TAG > + DEBIAN_TEST_GL_TAG > + KERNEL_ROOTFS_TAG > + KERNEL_TAG > + PKG_REPO_REV > + ) > + for var in "${image_tags[@]}" > + do > + if [ "$(echo -n "${!var}" | wc -c)" -gt 20 ] > + then > + echo "$var is too long; please make sure it is at most 20 chars." > + exit 1 > + fi > + done > artifacts: > when: on_failure > reports: > junit: check-*.xml > + tags: > + - placeholder-job > > -# Rules for tests that should not block merging, but should be available to > -# optionally run with the "play" button in the UI in pre-merge non-marge > -# pipelines. This should appear in "extends:" after any includes of > -# test-source-dep.yml rules, so that these rules replace those. > -.test-manual-mr: > + > +mr-label-maker-test: > + extends: > + - .fdo.ci-fairy > + stage: sanity > rules: > - - !reference [.no_scheduled_pipelines-rules, rules] > - - if: *is-forked-branch-or-pre-merge-not-for-marge > - when: manual > + - !reference [.mr-label-maker-rules, rules] > variables: > - JOB_TIMEOUT: 80 > - > + GIT_STRATEGY: fetch > + timeout: 10m > + script: > + - set -eu > + - python3 -m venv .venv > + - source .venv/bin/activate > + - pip install git+https://gitlab.freedesktop.org/freedesktop/mr-label-maker > + - mr-label-maker --dry-run --mr $CI_MERGE_REQUEST_IID > > # Jobs that need to pass before spending hardware resources on further testing > .required-for-hardware-jobs: > - needs: [] > + needs: > + - job: clang-format > + optional: true > + - job: rustfmt > + optional: true > + - job: toml-lint > + optional: true > diff --git a/drivers/gpu/drm/ci/image-tags.yml b/drivers/gpu/drm/ci/image-tags.yml > index 8d8b9e71852e..528ef44285bb 100644 > --- a/drivers/gpu/drm/ci/image-tags.yml > +++ b/drivers/gpu/drm/ci/image-tags.yml > @@ -1,5 +1,5 @@ > variables: > - CONTAINER_TAG: "2024-09-09-uprevs" > + CONTAINER_TAG: "20241127-mesa-uprev" > DEBIAN_X86_64_BUILD_BASE_IMAGE: "debian/x86_64_build-base" > DEBIAN_BASE_TAG: "${CONTAINER_TAG}" > > @@ -7,9 +7,16 @@ variables: > DEBIAN_BUILD_TAG: "${CONTAINER_TAG}" > > KERNEL_ROOTFS_TAG: "${CONTAINER_TAG}" > + # default kernel for rootfs before injecting the current kernel tree > + KERNEL_TAG: "v6.6.21-mesa-f8ea" > + KERNEL_REPO: "gfx-ci/linux" > + PKG_REPO_REV: "bca9635d" > > DEBIAN_X86_64_TEST_BASE_IMAGE: "debian/x86_64_test-base" > DEBIAN_X86_64_TEST_IMAGE_GL_PATH: "debian/x86_64_test-gl" > DEBIAN_TEST_GL_TAG: "${CONTAINER_TAG}" > > - ALPINE_X86_64_LAVA_SSH_TAG: "${CONTAINER_TAG}" > \ No newline at end of file > + DEBIAN_PYUTILS_IMAGE: "debian/x86_64_pyutils" > + DEBIAN_PYUTILS_TAG: "${CONTAINER_TAG}" > + > + ALPINE_X86_64_LAVA_SSH_TAG: "${CONTAINER_TAG}" > diff --git a/drivers/gpu/drm/ci/lava-submit.sh b/drivers/gpu/drm/ci/lava-submit.sh > index 6add15083c78..a1404a2d8cee 100755 > --- a/drivers/gpu/drm/ci/lava-submit.sh > +++ b/drivers/gpu/drm/ci/lava-submit.sh > @@ -1,58 +1,105 @@ > -#!/bin/bash > +#!/usr/bin/env bash > # SPDX-License-Identifier: MIT > +# shellcheck disable=SC2086 # we want word splitting > +# shellcheck disable=SC1091 # paths only become valid at runtime > > -set -e > -set -x > +# If we run in the fork (not from mesa or Marge-bot), reuse mainline kernel and rootfs, if exist. > +_check_artifact_path() { > + _url="https://${1}/${2}" > + if curl -s -o /dev/null -I -L -f --retry 4 --retry-delay 15 "${_url}"; then > + echo -n "${_url}" > + fi > +} > > -# Try to use the kernel and rootfs built in mainline first, so we're more > -# likely to hit cache > -if curl -L --retry 4 -f --retry-all-errors --retry-delay 60 -s "https://${BASE_SYSTEM_MAINLINE_HOST_PATH}/done"; then > - BASE_SYSTEM_HOST_PATH="${BASE_SYSTEM_MAINLINE_HOST_PATH}" > -else > - BASE_SYSTEM_HOST_PATH="${BASE_SYSTEM_FORK_HOST_PATH}" > -fi > +get_path_to_artifact() { > + _mainline_artifact="$(_check_artifact_path ${BASE_SYSTEM_MAINLINE_HOST_PATH} ${1})" > + if [ -n "${_mainline_artifact}" ]; then > + echo -n "${_mainline_artifact}" > + return > + fi > + _fork_artifact="$(_check_artifact_path ${BASE_SYSTEM_FORK_HOST_PATH} ${1})" > + if [ -n "${_fork_artifact}" ]; then > + echo -n "${_fork_artifact}" > + return > + fi > + set +x > + error "Sorry, I couldn't find a viable built path for ${1} in either mainline or a fork." >&2 > + echo "" >&2 > + echo "If you're working on CI, this probably means that you're missing a dependency:" >&2 > + echo "this job ran ahead of the job which was supposed to upload that artifact." >&2 > + echo "" >&2 > + echo "If you aren't working on CI, please ping @mesa/ci-helpers to see if we can help." >&2 > + echo "" >&2 > + echo "This job is going to fail, because I can't find the resources I need. Sorry." >&2 > + set -x > + exit 1 > +} > + > +. "${SCRIPTS_DIR}/setup-test-env.sh" > + > +section_start prepare_rootfs "Preparing root filesystem" > + > +set -ex > + > +section_switch rootfs "Assembling root filesystem" > +ROOTFS_URL="$(get_path_to_artifact lava-rootfs.tar.zst)" > +[ $? != 1 ] || exit 1 > > rm -rf results > mkdir -p results/job-rootfs-overlay/ > > +artifacts/ci-common/generate-env.sh > results/job-rootfs-overlay/set-job-env-vars.sh > cp artifacts/ci-common/capture-devcoredump.sh results/job-rootfs-overlay/ > cp artifacts/ci-common/init-*.sh results/job-rootfs-overlay/ > cp artifacts/ci-common/intel-gpu-freq.sh results/job-rootfs-overlay/ > +cp artifacts/ci-common/kdl.sh results/job-rootfs-overlay/ > cp "$SCRIPTS_DIR"/setup-test-env.sh results/job-rootfs-overlay/ > > -# Prepare env vars for upload. > -section_start variables "Variables passed through:" > -KERNEL_IMAGE_BASE="https://${BASE_SYSTEM_HOST_PATH}" \ > - artifacts/ci-common/generate-env.sh | tee results/job-rootfs-overlay/set-job-env-vars.sh > -section_end variables > - > tar zcf job-rootfs-overlay.tar.gz -C results/job-rootfs-overlay/ . > ci-fairy s3cp --token-file "${S3_JWT_FILE}" job-rootfs-overlay.tar.gz "https://${JOB_ROOTFS_OVERLAY_PATH}" > > +# Prepare env vars for upload. > +section_switch variables "Environment variables passed through to device:" > +cat results/job-rootfs-overlay/set-job-env-vars.sh > + > +section_switch lava_submit "Submitting job for scheduling" > + > touch results/lava.log > tail -f results/lava.log & > - > PYTHONPATH=artifacts/ artifacts/lava/lava_job_submitter.py \ > - submit \ > + --farm "${FARM}" \ > + --device-type "${DEVICE_TYPE}" \ > + --boot-method "${BOOT_METHOD}" \ > + --job-timeout-min ${JOB_TIMEOUT:-30} \ > --dump-yaml \ > --pipeline-info "$CI_JOB_NAME: $CI_PIPELINE_URL on $CI_COMMIT_REF_NAME ${CI_NODE_INDEX}/${CI_NODE_TOTAL}" \ > - --rootfs-url-prefix "https://${BASE_SYSTEM_HOST_PATH}" \ > + --rootfs-url "${ROOTFS_URL}" \ > --kernel-url-prefix "https://${PIPELINE_ARTIFACTS_BASE}/${DEBIAN_ARCH}" \ > - --build-url "${FDO_HTTP_CACHE_URI:-}https://${PIPELINE_ARTIFACTS_BASE}/${DEBIAN_ARCH}/kernel-files.tar.zst" \ > - --job-rootfs-overlay-url "${FDO_HTTP_CACHE_URI:-}https://${JOB_ROOTFS_OVERLAY_PATH}" \ > - --job-timeout-min ${JOB_TIMEOUT:-80} \ > + --kernel-external "${EXTERNAL_KERNEL_TAG}" \ > --first-stage-init artifacts/ci-common/init-stage1.sh \ > - --ci-project-dir "${CI_PROJECT_DIR}" \ > - --device-type "${DEVICE_TYPE}" \ > - --farm "${FARM}" \ > --dtb-filename "${DTB}" \ > --jwt-file "${S3_JWT_FILE}" \ > --kernel-image-name "${KERNEL_IMAGE_NAME}" \ > --kernel-image-type "${KERNEL_IMAGE_TYPE}" \ > - --boot-method "${BOOT_METHOD}" \ > --visibility-group "${VISIBILITY_GROUP}" \ > --lava-tags "${LAVA_TAGS}" \ > --mesa-job-name "$CI_JOB_NAME" \ > --structured-log-file "results/lava_job_detail.json" \ > --ssh-client-image "${LAVA_SSH_CLIENT_IMAGE}" \ > + --project-name "${CI_PROJECT_NAME}" \ > + --starting-section "${CURRENT_SECTION}" \ > + --job-submitted-at "${CI_JOB_STARTED_AT}" \ > + - append-overlay \ > + --name=kernel-build \ > + --url="${FDO_HTTP_CACHE_URI:-}https://${PIPELINE_ARTIFACTS_BASE}/${DEBIAN_ARCH}/kernel-files.tar.zst" \ > + --compression=zstd \ > + --path="${CI_PROJECT_DIR}" \ > + --format=tar \ > + - append-overlay \ > + --name=job-overlay \ > + --url="https://${JOB_ROOTFS_OVERLAY_PATH}" \ > + --compression=gz \ > + --path="/" \ > + --format=tar \ > + - submit \ > >> results/lava.log > diff --git a/drivers/gpu/drm/ci/test.yml b/drivers/gpu/drm/ci/test.yml > index f0ef60c8f56d..42a8e7f93714 100644 > --- a/drivers/gpu/drm/ci/test.yml > +++ b/drivers/gpu/drm/ci/test.yml > @@ -1,21 +1,21 @@ > .test-rules: > rules: > - - if: '$FD_FARM == "offline" && $RUNNER_TAG =~ /^google-freedreno-/' > - when: never > - - if: '$COLLABORA_FARM == "offline" && $RUNNER_TAG =~ /^mesa-ci-x86-64-lava-/' > - when: never > - !reference [.no_scheduled_pipelines-rules, rules] > + - !reference [.collabora-farm-rules, rules] > + - !reference [.google-freedreno-farm-rules, rules] > - when: on_success > > .lava-test: > extends: > - .test-rules > + - .build-rules > + - .container+build-rules > timeout: "1h30m" > script: > # Note: Build dir (and thus install) may be dirty due to GIT_STRATEGY > - rm -rf install > - tar -xf artifacts/install.tar > - - mv install/* artifacts/. > + - mv -n install/* artifacts/. > # Override it with our lava-submit.sh script > - ./artifacts/lava-submit.sh > > @@ -32,6 +32,7 @@ > - alpine/x86_64_lava_ssh_client > - kernel+rootfs_arm32 > - debian/x86_64_build > + - python-test > - testing:arm32 > - igt:arm32 > > @@ -48,6 +49,7 @@ > - alpine/x86_64_lava_ssh_client > - kernel+rootfs_arm64 > - debian/x86_64_build > + - python-test > - testing:arm64 > - igt:arm64 > > @@ -64,6 +66,7 @@ > - alpine/x86_64_lava_ssh_client > - kernel+rootfs_x86_64 > - debian/x86_64_build > + - python-test > - testing:x86_64 > - igt:x86_64 > > -- > 2.43.0 > > lgtm Acked-by: Helen Koike <helen.koike@collabora.com> Thanks Helen ^ permalink raw reply [flat|nested] 9+ messages in thread
end of thread, other threads:[~2025-11-16 15:32 UTC | newest] Thread overview: 9+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2025-11-14 3:00 [PATCH v1 0/2] drm/ci: uprev IGT and enable apq8016, apq8096 Vignesh Raman 2025-11-14 3:00 ` [PATCH v1 1/2] drm/ci: uprev mesa Vignesh Raman 2025-11-15 0:44 ` Dmitry Baryshkov 2025-11-15 3:55 ` Dmitry Baryshkov 2025-11-16 15:32 ` Dmitry Baryshkov 2025-11-14 3:00 ` [PATCH v1 2/2] drm/ci: Move qualcomm baremetal jobs to lava Vignesh Raman 2025-11-15 0:50 ` Dmitry Baryshkov -- strict thread matches above, loose matches on Subject: below -- 2024-11-28 4:20 [PATCH v1 0/2] drm/ci: uprev mesa and modify gitlab rules Vignesh Raman 2024-11-28 4:20 ` [PATCH v1 1/2] drm/ci: uprev mesa Vignesh Raman 2024-12-03 14:14 ` Helen Mae Koike Fornazier
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®