[PATCH v1 0/3] drm/ci: mesa uprev and python-artifacts/check-patch fixes

[PATCH v1 0/3] drm/ci: mesa uprev and python-artifacts/check-patch fixes

[Vignesh Raman]

The python-artifacts job has a timeout of 10 minutes, which causes
build failures as it was unable to clone the repository within the
specified limits. This patch sets GIT_DEPTH to 10 to speed up cloning
and avoid build failures due to timeouts when fetching the full
repository.

The check-patch job is also updated to ensure the repository is not
shallow before fetching branches. This prevents issues where git
merge-base fails due to incomplete history. Additionally, the timeout
for the check-patch job is set to 1 hour.

The current s3cp implementation does not work anymore after the migration.
Uprev mesa to adapt these changes. Also replace broken s3cp command with a
curl wrapper call in drm-ci.

check-patch testing,
MR - https://gitlab.freedesktop.org/vigneshraman/linux/-/merge_requests/18
pipeline - https://gitlab.freedesktop.org/vigneshraman/linux/-/pipelines/1391545

mesa uprev testing,
MR - https://gitlab.freedesktop.org/vigneshraman/linux/-/merge_requests/18
pipeline - https://gitlab.freedesktop.org/vigneshraman/linux/-/pipelines/1391440

Google farm is down, so the baremetal jobs failed.
i915:cml jobs are stuck due to some devices in the lab being under maintenance.

Vignesh Raman (3):
  drm/ci: use shallow clone to avoid timeouts
  drm/ci: check-patch: unshallow repository before fetching
  drm/ci: uprev mesa

 drivers/gpu/drm/ci/build-igt.sh      |  2 +-
 drivers/gpu/drm/ci/build.sh          |  6 +++---
 drivers/gpu/drm/ci/check-patch.py    | 16 ++++++++++++----
 drivers/gpu/drm/ci/gitlab-ci.yml     |  8 ++++++--
 drivers/gpu/drm/ci/image-tags.yml    |  2 +-
 drivers/gpu/drm/ci/lava-submit.sh    |  2 +-
 drivers/gpu/drm/ci/static-checks.yml |  1 +
 7 files changed, 25 insertions(+), 12 deletions(-)

-- 
2.47.2

[PATCH v1 1/3] drm/ci: use shallow clone to avoid timeouts

[Vignesh Raman]

The python-artifacts job has a timeout of 10 minutes, which causes
build failures as it was unable to clone the repository within the
specified limits. Set GIT_DEPTH to 10 to speed up cloning and avoid
build failures due to timeouts when fetching the full repository.

Signed-off-by: Vignesh Raman <vignesh.raman@collabora.com>
---
 drivers/gpu/drm/ci/gitlab-ci.yml | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/drivers/gpu/drm/ci/gitlab-ci.yml b/drivers/gpu/drm/ci/gitlab-ci.yml
index 65adcd97e06b..6850ce99a673 100644
--- a/drivers/gpu/drm/ci/gitlab-ci.yml
+++ b/drivers/gpu/drm/ci/gitlab-ci.yml
@@ -40,6 +40,8 @@ variables:
   ARTIFACTS_BASE_URL: https://${CI_PROJECT_ROOT_NAMESPACE}.${CI_PAGES_DOMAIN}/-/${CI_PROJECT_NAME}/-/jobs/${CI_JOB_ID}/artifacts
   # Python scripts for structured logger
   PYTHONPATH: "$PYTHONPATH:$CI_PROJECT_DIR/install"
+  # Set to 0 to disable shallow cloning
+  GIT_DEPTH: 10
 
 
 default:
-- 
2.47.2

[PATCH v1 2/3] drm/ci: check-patch: unshallow repository before fetching

[Vignesh Raman]

Ensure the repository is not shallow before fetching branches in
check-patch job. This prevents issues where git merge-base fails
due to incomplete history. Set the timeout of check-patch job to 1h.

Signed-off-by: Vignesh Raman <vignesh.raman@collabora.com>
---
 drivers/gpu/drm/ci/check-patch.py    | 16 ++++++++++++----
 drivers/gpu/drm/ci/static-checks.yml |  1 +
 2 files changed, 13 insertions(+), 4 deletions(-)

diff --git a/drivers/gpu/drm/ci/check-patch.py b/drivers/gpu/drm/ci/check-patch.py
index a5f399a20e25..b206f12feb64 100755
--- a/drivers/gpu/drm/ci/check-patch.py
+++ b/drivers/gpu/drm/ci/check-patch.py
@@ -18,12 +18,20 @@ repourl = "https://gitlab.freedesktop.org/%s.git" % os.environ["CI_MERGE_REQUEST
 # GitLab CI environment does not give us any direct info about the
 # base for the user's branch. We thus need to figure out a common
 # ancestor between the user's branch and current git master.
-os.environ["GIT_DEPTH"] = "1000"
 subprocess.call(["git", "remote", "remove", "check-patch"], stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
 subprocess.check_call(["git", "remote", "add", "check-patch", repourl])
-subprocess.check_call(["git", "fetch", "check-patch", os.environ["CI_MERGE_REQUEST_TARGET_BRANCH_NAME"]],
-                      stdout=subprocess.DEVNULL,
-                      stderr=subprocess.DEVNULL)
+
+# Check if the repository is shallow
+shallow = subprocess.run(["git", "rev-parse", "--is-shallow-repository"],
+                         stdout=subprocess.PIPE, universal_newlines=True).stdout.strip()
+
+if shallow == "true":
+    print("Repository is shallow, unshallow git history")
+    subprocess.check_call(["git", "fetch", "--unshallow", "check-patch", os.environ["CI_MERGE_REQUEST_TARGET_BRANCH_NAME"]],
+                          stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
+else:
+    subprocess.check_call(["git", "fetch", "check-patch", os.environ["CI_MERGE_REQUEST_TARGET_BRANCH_NAME"]],
+                          stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
 
 ancestor = subprocess.check_output(["git", "merge-base",
                                     "check-patch/%s" % os.environ["CI_MERGE_REQUEST_TARGET_BRANCH_NAME"], "HEAD"],
diff --git a/drivers/gpu/drm/ci/static-checks.yml b/drivers/gpu/drm/ci/static-checks.yml
index 13ffa827b7fa..734d6055fa65 100644
--- a/drivers/gpu/drm/ci/static-checks.yml
+++ b/drivers/gpu/drm/ci/static-checks.yml
@@ -1,4 +1,5 @@
 check-patch:
+  timeout: "1h"
   extends:
     - .build
     - .use-debian/x86_64_build
-- 
2.47.2

Re: [PATCH v1 2/3] drm/ci: check-patch: unshallow repository before fetching

[Daniel Stone]

Hi Vignesh,

On Fri, 28 Mar 2025 at 11:03, Vignesh Raman <vignesh.raman@collabora.com> wrote:
> Ensure the repository is not shallow before fetching branches in
> check-patch job. This prevents issues where git merge-base fails
> due to incomplete history. Set the timeout of check-patch job to 1h.

Ouch - an hour is pretty brutal. Is there a way to unshallow only back
to the merge base?

Cheers,
Daniel

Re: [PATCH v1 2/3] drm/ci: check-patch: unshallow repository before fetching

[Vignesh Raman]

Hi Daniel,

On 28/03/25 17:05, Daniel Stone wrote:
> Hi Vignesh,
> 
> On Fri, 28 Mar 2025 at 11:03, Vignesh Raman <vignesh.raman@collabora.com> wrote:
>> Ensure the repository is not shallow before fetching branches in
>> check-patch job. This prevents issues where git merge-base fails
>> due to incomplete history. Set the timeout of check-patch job to 1h.
> 
> Ouch - an hour is pretty brutal. Is there a way to unshallow only back
> to the merge base?

I set it to 1h, but the job is completed in ~15min for
https://gitlab.freedesktop.org/vigneshraman/linux/-/merge_requests/18 
which has 486 commits.

I will check if we can unshallow only up to the merge base.

Regards,
Vignesh

> 
> Cheers,
> Daniel

Re: [PATCH v1 2/3] drm/ci: check-patch: unshallow repository before fetching

[Vignesh Raman]

Hi Daniel,

On 28/03/25 17:40, Vignesh Raman wrote:
> Hi Daniel,
> 
> On 28/03/25 17:05, Daniel Stone wrote:
>> Hi Vignesh,
>>
>> On Fri, 28 Mar 2025 at 11:03, Vignesh Raman 
>> <vignesh.raman@collabora.com> wrote:
>>> Ensure the repository is not shallow before fetching branches in
>>> check-patch job. This prevents issues where git merge-base fails
>>> due to incomplete history. Set the timeout of check-patch job to 1h.
>>
>> Ouch - an hour is pretty brutal. Is there a way to unshallow only back
>> to the merge base?
> 
> I set it to 1h, but the job is completed in ~15min for
> https://gitlab.freedesktop.org/vigneshraman/linux/-/merge_requests/18 
> which has 486 commits.
> 
> I will check if we can unshallow only up to the merge base.

I tried this and still checkpatch failed. Below is the sequence.

- GitLab starts with a shallow fetch (depth=10).
- Script fetches full commit history (--unshallow --filter=blob:none). 
We need this to calculate the merge-base commit.
- Calculates how much history to fetch using the merge-base commit.
- Refetch with depth (--depth=N) until the merge-base commit
- checkpatch.pl fails because the earlier blobless fetch (--unshallow 
--filter=blob:none) skipped file contents.

Please see the commit and pipeline,
https://gitlab.freedesktop.org/vigneshraman/linux/-/commit/40a3fc31c2405f90f3fc3177a575a66a10b33324
https://gitlab.freedesktop.org/vigneshraman/linux/-/jobs/73884148

Looks like the reliable solution is to fully unshallow the repository 
(without any --filter) and set a 30m timeout? Would this be acceptable?

Regards,
Vignesh

> 
> Regards,
> Vignesh
> 
>>
>> Cheers,
>> Daniel
> 

[PATCH v1 3/3] drm/ci: uprev mesa

[Vignesh Raman]

The current s3cp implementation does not work anymore after the
migration, and instead of fixing it and propagating the fix down to us,
it's simpler to directly use curl. Uprev mesa [1][2] to adapt these
changes. Also replace broken s3cp command with a curl wrapper call in
drm-ci.

[1] https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/34120
[2] https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/34244

Signed-off-by: Vignesh Raman <vignesh.raman@collabora.com>
---
 drivers/gpu/drm/ci/build-igt.sh   | 2 +-
 drivers/gpu/drm/ci/build.sh       | 6 +++---
 drivers/gpu/drm/ci/gitlab-ci.yml  | 6 ++++--
 drivers/gpu/drm/ci/image-tags.yml | 2 +-
 drivers/gpu/drm/ci/lava-submit.sh | 2 +-
 5 files changed, 10 insertions(+), 8 deletions(-)

diff --git a/drivers/gpu/drm/ci/build-igt.sh b/drivers/gpu/drm/ci/build-igt.sh
index eddb5f782a5e..caa2f4804ed5 100644
--- a/drivers/gpu/drm/ci/build-igt.sh
+++ b/drivers/gpu/drm/ci/build-igt.sh
@@ -71,4 +71,4 @@ tar -cf artifacts/igt.tar /igt
 # Pass needed files to the test stage
 S3_ARTIFACT_NAME="igt.tar.gz"
 gzip -c artifacts/igt.tar > ${S3_ARTIFACT_NAME}
-ci-fairy s3cp --token-file "${S3_JWT_FILE}" ${S3_ARTIFACT_NAME} https://${PIPELINE_ARTIFACTS_BASE}/${KERNEL_ARCH}/${S3_ARTIFACT_NAME}
+s3_upload ${S3_ARTIFACT_NAME} https://${PIPELINE_ARTIFACTS_BASE}/${KERNEL_ARCH}/
diff --git a/drivers/gpu/drm/ci/build.sh b/drivers/gpu/drm/ci/build.sh
index 284873e94d8d..6fb74c51abe2 100644
--- a/drivers/gpu/drm/ci/build.sh
+++ b/drivers/gpu/drm/ci/build.sh
@@ -148,13 +148,13 @@ if [[ "$UPLOAD_TO_MINIO" = "1" ]]; then
 
     ls -l "${S3_JWT_FILE}"
     for f in $FILES_TO_UPLOAD; do
-        ci-fairy s3cp --token-file "${S3_JWT_FILE}" /kernel/$f \
-                https://${PIPELINE_ARTIFACTS_BASE}/${DEBIAN_ARCH}/$f
+        s3_upload /kernel/$f \
+                https://${PIPELINE_ARTIFACTS_BASE}/${DEBIAN_ARCH}/
     done
 
     S3_ARTIFACT_NAME="kernel-files.tar.zst"
     tar --zstd -cf $S3_ARTIFACT_NAME install
-    ci-fairy s3cp --token-file "${S3_JWT_FILE}" ${S3_ARTIFACT_NAME} https://${PIPELINE_ARTIFACTS_BASE}/${DEBIAN_ARCH}/${S3_ARTIFACT_NAME}
+    s3_upload ${S3_ARTIFACT_NAME} https://${PIPELINE_ARTIFACTS_BASE}/${DEBIAN_ARCH}/
 
     echo "Download vmlinux.xz from https://${PIPELINE_ARTIFACTS_BASE}/${DEBIAN_ARCH}/vmlinux.xz"
 fi
diff --git a/drivers/gpu/drm/ci/gitlab-ci.yml b/drivers/gpu/drm/ci/gitlab-ci.yml
index 6850ce99a673..e54e382bf5fb 100644
--- a/drivers/gpu/drm/ci/gitlab-ci.yml
+++ b/drivers/gpu/drm/ci/gitlab-ci.yml
@@ -1,6 +1,6 @@
 variables:
   DRM_CI_PROJECT_PATH: &drm-ci-project-path mesa/mesa
-  DRM_CI_COMMIT_SHA: &drm-ci-commit-sha 82ab58f6c6f94fa80ca7e1615146f08356e3ba69
+  DRM_CI_COMMIT_SHA: &drm-ci-commit-sha f73132f1215a37ce8ffc711a0136c90649aaf128
 
   UPSTREAM_REPO: https://gitlab.freedesktop.org/drm/kernel.git
   TARGET_BRANCH: drm-next
@@ -20,8 +20,10 @@ variables:
           rm download-git-cache.sh
           set +o xtrace
   S3_JWT_FILE: /s3_jwt
+  S3_JWT_HEADER_FILE: /s3_jwt_header
   S3_JWT_FILE_SCRIPT: |-
       echo -n '${S3_JWT}' > '${S3_JWT_FILE}' &&
+      echo -n "Authorization: Bearer ${S3_JWT}" > '${S3_JWT_HEADER_FILE}' &&
       unset CI_JOB_JWT S3_JWT  # Unsetting vulnerable env variables
   S3_HOST: s3.freedesktop.org
   # This bucket is used to fetch the kernel image
@@ -253,7 +255,7 @@ make git archive:
     - tar -cvzf ../$CI_PROJECT_NAME.tar.gz .
 
     # Use id_tokens for JWT auth
-    - ci-fairy s3cp --token-file "${S3_JWT_FILE}" ../$CI_PROJECT_NAME.tar.gz https://$S3_HOST/${S3_GITCACHE_BUCKET}/$CI_PROJECT_NAMESPACE/$CI_PROJECT_NAME/$CI_PROJECT_NAME.tar.gz
+    - s3_upload ../$CI_PROJECT_NAME.tar.gz https://$S3_HOST/${S3_GITCACHE_BUCKET}/$CI_PROJECT_NAMESPACE/$CI_PROJECT_NAME/
 
 
 # Sanity checks of MR settings and commit logs
diff --git a/drivers/gpu/drm/ci/image-tags.yml b/drivers/gpu/drm/ci/image-tags.yml
index c04ba0e69935..53fe34b86578 100644
--- a/drivers/gpu/drm/ci/image-tags.yml
+++ b/drivers/gpu/drm/ci/image-tags.yml
@@ -1,5 +1,5 @@
 variables:
-   CONTAINER_TAG: "20250307-mesa-uprev"
+   CONTAINER_TAG: "20250328-mesa-uprev"
    DEBIAN_X86_64_BUILD_BASE_IMAGE: "debian/x86_64_build-base"
    DEBIAN_BASE_TAG: "${CONTAINER_TAG}"
 
diff --git a/drivers/gpu/drm/ci/lava-submit.sh b/drivers/gpu/drm/ci/lava-submit.sh
index f22720359b33..a1e8b34fb2d4 100755
--- a/drivers/gpu/drm/ci/lava-submit.sh
+++ b/drivers/gpu/drm/ci/lava-submit.sh
@@ -54,7 +54,7 @@ cp artifacts/ci-common/init-*.sh results/job-rootfs-overlay/
 cp "$SCRIPTS_DIR"/setup-test-env.sh results/job-rootfs-overlay/
 
 tar zcf job-rootfs-overlay.tar.gz -C results/job-rootfs-overlay/ .
-ci-fairy s3cp --token-file "${S3_JWT_FILE}" job-rootfs-overlay.tar.gz "https://${JOB_ROOTFS_OVERLAY_PATH}"
+s3_upload job-rootfs-overlay.tar.gz "https://${JOB_ARTIFACTS_BASE}"
 
 # Prepare env vars for upload.
 section_switch variables "Environment variables passed through to device:"
-- 
2.47.2

Re: [PATCH v1 3/3] drm/ci: uprev mesa

[Daniel Stone]

Hi Vignesh,

On Fri, 28 Mar 2025 at 11:03, Vignesh Raman <vignesh.raman@collabora.com> wrote:
> The current s3cp implementation does not work anymore after the
> migration, and instead of fixing it and propagating the fix down to us,
> it's simpler to directly use curl. Uprev mesa [1][2] to adapt these
> changes. Also replace broken s3cp command with a curl wrapper call in
> drm-ci.

Thanks a lot for fixing this. Sorry the fallout has been so bad.

You can also upgrade ci-templates to get an s3cp which works again.

Cheers,
Daniel

Re: [PATCH v1 3/3] drm/ci: uprev mesa

[Vignesh Raman]

Hi Daniel,

On 28/03/25 17:06, Daniel Stone wrote:
> Hi Vignesh,
> 
> On Fri, 28 Mar 2025 at 11:03, Vignesh Raman <vignesh.raman@collabora.com> wrote:
>> The current s3cp implementation does not work anymore after the
>> migration, and instead of fixing it and propagating the fix down to us,
>> it's simpler to directly use curl. Uprev mesa [1][2] to adapt these
>> changes. Also replace broken s3cp command with a curl wrapper call in
>> drm-ci.
> 
> Thanks a lot for fixing this. Sorry the fallout has been so bad.
> 
> You can also upgrade ci-templates to get an s3cp which works again.

Thanks for fixing this. Will use the latest ci-templates and test it.

Regards,
Vignesh

> 
> Cheers,
> Daniel

Re: [PATCH v1 3/3] drm/ci: uprev mesa

[Vignesh Raman]

Hi Daniel,

On 28/03/25 17:29, Vignesh Raman wrote:
> Hi Daniel,
> 
> On 28/03/25 17:06, Daniel Stone wrote:
>> Hi Vignesh,
>>
>> On Fri, 28 Mar 2025 at 11:03, Vignesh Raman 
>> <vignesh.raman@collabora.com> wrote:
>>> The current s3cp implementation does not work anymore after the
>>> migration, and instead of fixing it and propagating the fix down to us,
>>> it's simpler to directly use curl. Uprev mesa [1][2] to adapt these
>>> changes. Also replace broken s3cp command with a curl wrapper call in
>>> drm-ci.
>>
>> Thanks a lot for fixing this. Sorry the fallout has been so bad.
>>
>> You can also upgrade ci-templates to get an s3cp which works again.
> 
> Thanks for fixing this. Will use the latest ci-templates and test it.

We need to update mesa to use the latest ci-templates and then uprev 
mesa in drm-ci. I will send this in a separate series after fixing it in 
mesa.

Regards,
Vignesh

> 
> Regards,
> Vignesh
> 
>>
>> Cheers,
>> Daniel
> 

Re: [PATCH v1 3/3] drm/ci: uprev mesa

[Dmitry Baryshkov]

On Thu, Apr 03, 2025 at 11:29:55AM +0530, Vignesh Raman wrote:
> Hi Daniel,
> 
> On 28/03/25 17:29, Vignesh Raman wrote:
> > Hi Daniel,
> > 
> > On 28/03/25 17:06, Daniel Stone wrote:
> > > Hi Vignesh,
> > > 
> > > On Fri, 28 Mar 2025 at 11:03, Vignesh Raman
> > > <vignesh.raman@collabora.com> wrote:
> > > > The current s3cp implementation does not work anymore after the
> > > > migration, and instead of fixing it and propagating the fix down to us,
> > > > it's simpler to directly use curl. Uprev mesa [1][2] to adapt these
> > > > changes. Also replace broken s3cp command with a curl wrapper call in
> > > > drm-ci.
> > > 
> > > Thanks a lot for fixing this. Sorry the fallout has been so bad.
> > > 
> > > You can also upgrade ci-templates to get an s3cp which works again.
> > 
> > Thanks for fixing this. Will use the latest ci-templates and test it.
> 
> We need to update mesa to use the latest ci-templates and then uprev mesa in
> drm-ci. I will send this in a separate series after fixing it in mesa.

Vignesh, Daniel, any updates on this? Currently drm/ci is broken both in
master and in drm-misc.

-- 
With best wishes
Dmitry

Re: [PATCH v1 3/3] drm/ci: uprev mesa

[Vignesh Raman]

Hi Dmitry,

On 29/04/25 23:51, Dmitry Baryshkov wrote:
> On Thu, Apr 03, 2025 at 11:29:55AM +0530, Vignesh Raman wrote:
>> Hi Daniel,
>>
>> On 28/03/25 17:29, Vignesh Raman wrote:
>>> Hi Daniel,
>>>
>>> On 28/03/25 17:06, Daniel Stone wrote:
>>>> Hi Vignesh,
>>>>
>>>> On Fri, 28 Mar 2025 at 11:03, Vignesh Raman
>>>> <vignesh.raman@collabora.com> wrote:
>>>>> The current s3cp implementation does not work anymore after the
>>>>> migration, and instead of fixing it and propagating the fix down to us,
>>>>> it's simpler to directly use curl. Uprev mesa [1][2] to adapt these
>>>>> changes. Also replace broken s3cp command with a curl wrapper call in
>>>>> drm-ci.
>>>>
>>>> Thanks a lot for fixing this. Sorry the fallout has been so bad.
>>>>
>>>> You can also upgrade ci-templates to get an s3cp which works again.
>>>
>>> Thanks for fixing this. Will use the latest ci-templates and test it.
>>
>> We need to update mesa to use the latest ci-templates and then uprev mesa in
>> drm-ci. I will send this in a separate series after fixing it in mesa.
> 
> Vignesh, Daniel, any updates on this? Currently drm/ci is broken both in
> master and in drm-misc.
> 

mesa-ci is updated to use s3cp and drop the s3_upload, which is a curl 
wrapper. These patches were merged this week and I will uprev mesa in 
drm-ci to get s3cp working. I will send update to this series.

Regards,
Vignesh

Re: [PATCH v1 0/3] drm/ci: mesa uprev and python-artifacts/check-patch fixes

[Dmitry Baryshkov]

On Fri, 28 Mar 2025 16:32:30 +0530, Vignesh Raman wrote:
> The python-artifacts job has a timeout of 10 minutes, which causes
> build failures as it was unable to clone the repository within the
> specified limits. This patch sets GIT_DEPTH to 10 to speed up cloning
> and avoid build failures due to timeouts when fetching the full
> repository.
> 
> The check-patch job is also updated to ensure the repository is not
> shallow before fetching branches. This prevents issues where git
> merge-base fails due to incomplete history. Additionally, the timeout
> for the check-patch job is set to 1 hour.
> 
> [...]

Applied to b4/fix-dp-mst-interfaces, thanks!

[1/3] drm/ci: use shallow clone to avoid timeouts
      (no commit info)
[2/3] drm/ci: check-patch: unshallow repository before fetching
      (no commit info)
[3/3] drm/ci: uprev mesa
      https://gitlab.freedesktop.org/lumag/msm/-/commit/0bebb1773d61

Best regards,
-- 
With best wishes
Dmitry