Closed Bug 1557331 Opened 1 year ago Closed 1 year ago

Nightly cron failure related to merge

Categories

(Release Engineering :: Release Automation: Other, defect)

defect
Not set
normal

Tracking

(firefox69 fixed)

RESOLVED FIXED
Tracking Status
firefox69 --- fixed

People

(Reporter: sfraser, Unassigned)

Details

Attachments

(1 file)

This morning's nightly didn't happen on mozilla-central.
Looking at old cron firings in https://hooks.taskcluster.net/v1/hooks/project-releng/cron-task-mozilla-central/last-fires we find https://taskcluster-artifacts.net/Cc-Y1crrRJOlnu2YM14VJg/0/public/logs/live_backing.log

[task 2019-06-06T10:00:30.154Z] retry: calling query_pushlog with args: (u'https://hg.mozilla.org/mozilla-central/json-pushes?version=2&changeset=fee989d27558bdbf83351633f09bd7f80d16f2d4&tipsonly=1&full=1',), kwargs: {}, attempt #1
[task 2019-06-06T10:00:30.155Z] Starting new HTTPS connection (1): hg.mozilla.org
[task 2019-06-06T10:00:30.318Z] "GET /mozilla-central/json-pushes?version=2&changeset=fee989d27558bdbf83351633f09bd7f80d16f2d4&tipsonly=1&full=1 HTTP/1.1" 200 None
[task 2019-06-06T10:00:30.347Z] Traceback (most recent call last):
[task 2019-06-06T10:00:30.347Z]   File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/cron/__init__.py", line 67, in run_job
[task 2019-06-06T10:00:30.348Z]     tasks = JOB_TYPES[job_type](job['job'], params, root=root)
[task 2019-06-06T10:00:30.348Z]   File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/cron/decision.py", line 38, in run_decision_task
[task 2019-06-06T10:00:30.348Z]     root=root),
[task 2019-06-06T10:00:30.348Z]   File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/cron/decision.py", line 48, in make_decision_task
[task 2019-06-06T10:00:30.349Z]     params['head_rev'])
[task 2019-06-06T10:00:30.349Z]   File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/util/hg.py", line 33, in find_hg_revision_push_info
[task 2019-06-06T10:00:30.349Z]     repository, revision, pushes
[task 2019-06-06T10:00:30.349Z] RuntimeError: Unable to find a single pushlog_id for https://hg.mozilla.org/mozilla-central revision fee989d27558bdbf83351633f09bd7f80d16f2d4: {}
[task 2019-06-06T10:00:30.349Z] cron job nightly-desktop run failed; continuing to next job

It wasn't retried since the actual http operation was a success, it just didn't have the expected content.
The code currently checks to see if the length of the pushes is != 1 in https://hg.mozilla.org/mozilla-central/file/tip/taskcluster/taskgraph/util/hg.py#l30

There was a merge at 10:00:05 UTC - for the revision mentioned above - and so current hypothesis is that the length of the pushes was zero. If this is confirmed perhaps we should be retrying on that error, and aborting if >1

This affected
nightly-desktop
nightly-geckoview
periodic-update
searchfox-index

If the length of the pushes from the pushlog is wrong, raise exception and retry.

Pushed by sfraser@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/dcb20740187a
Cover missing-data scenario with retries in util.hg r=dustin
Status: NEW → RESOLVED
Closed: 1 year ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.