1788201 - gecko decision tasks broken (Nightlies, performance tasks, searchfox etc blocked): revision is not specified for checkout

Status: RESOLVED FIXED

(Release Engineering :: Firefox-CI Administration, defect)

Description

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

ahal, could you investigate this?

All gecko decision tasks (e.g. to schedule nightly builds, performance tests, searchfox) fail:

Task config: https://firefox-ci-tc.services.mozilla.com/tasks/CnvNaSZhTQCgYEfW3nKf9w/runs/0
Log: https://firefox-ci-tc.services.mozilla.com/tasks/CnvNaSZhTQCgYEfW3nKf9w/runs/0/logs/public/logs/live.log

0 matches
[taskcluster 2022-08-31 10:26:20.802Z] Task ID: CnvNaSZhTQCgYEfW3nKf9w
[taskcluster 2022-08-31 10:26:20.802Z] Worker ID: i-0ba8a84af68a8256f
[taskcluster 2022-08-31 10:26:20.802Z] Worker Group: us-west-1
[taskcluster 2022-08-31 10:26:20.802Z] Worker Node Type: m5d.xlarge
[taskcluster 2022-08-31 10:26:20.802Z] Worker Type: decision
[taskcluster 2022-08-31 10:26:20.802Z] Public IP: 3.101.16.122
[taskcluster 2022-08-31 10:26:20.802Z] Hostname: ip-10-143-23-137
[taskcluster 2022-08-31 10:26:20.802Z] using cache "gecko-level-3-checkouts-sparse-v3" -> /builds/worker/checkouts
[taskcluster 2022-08-31 10:26:21.327Z] === Task Starting ===
[setup 2022-08-31T10:26:21.689Z] run-task started in /builds/worker
[cache 2022-08-31T10:26:21.690Z] cache /builds/worker/checkouts exists; requirements: gid=1000 uid=1000 version=1
[setup 2022-08-31T10:26:21.690Z] running as worker:worker
[vcs 2022-08-31T10:26:21.690Z] fetching hgmointernal config from http://taskcluster/secrets/v1/secret/project/taskcluster/gecko/hgmointernal
[vcs 2022-08-31T10:26:21.746Z] hgmointernal rate miss; using public hg.mozilla.org service
revision is not specified for checkout
[taskcluster 2022-08-31 10:26:21.989Z] === Task Finished ===
[taskcluster 2022-08-31 10:26:22.106Z] Artifact "public/docker-contexts" not found at "/builds/worker/checkouts/gecko/docker-contexts"
[taskcluster 2022-08-31 10:26:22.195Z] Unsuccessful task run with exit code: 1 completed in 1.394 seconds

References of error messages: 1, 2
The repositories are set while the revision values are empty in the payload:

    "GECKO_HEAD_REF": "",
    "GECKO_HEAD_REV": "",

Michelle mentioned the schedule image update hasn't started yet. The latest changes to mozilla-central look unrelated.

Comment 1

[Julien Cristau [:jcristau]]

Attachment: Bug 1788201 - Backed out changeset 88b8c2378d27 on suspicion of breaking gecko decision tasks.

Comment 2

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/ci/ci-configuration/rev/03037c56a257
Backed out changeset 88b8c2378d27 on suspicion of breaking gecko decision tasks.

Comment 3

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

This was a regression from bug 1784232 which got fixed by https://hg.mozilla.org/ci/ci-configuration/rev/03037c56a2570060cff1ee18e58c75c6939dcfca pushed by jcristau. I will manually trigger the hooks of failed cron tasks where possible.

Comment 4

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

The gecko-3/decision worker pool got terminated but at least the searchfox tasks still run into this issue, e.g. https://firefox-ci-tc.services.mozilla.com/tasks/Dk8J5MjkSG6hB-2gkz-F6g

Comment 5

[Julien Cristau [:jcristau]]

This is a rerun of a task that was created during the bustage window, so it has a broken definition. We need new tasks not just reruns.

Comment 6

[Rob Lemley [:rjl]]

This broke comm-central dailies this morning as well. With the backout I was able to trigger them manually.

Comment 7

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

I don't have the permission to trigger most of the hooks or this is not possible in general. Nightlies have been started and the searchfox team has been contacted about the failed cron jobs.

Comment 8

[BugBot [:suhaib / :marco/ :calixte]]

Set release status flags based on info from the regressing bug 1784232