1557331 - Nightly cron failure related to merge

Status: RESOLVED FIXED

(Release Engineering :: Release Automation, defect)

Description

[Simon Fraser [:sfraser] ⌚️GMT]

This morning's nightly didn't happen on mozilla-central.
Looking at old cron firings in https://hooks.taskcluster.net/v1/hooks/project-releng/cron-task-mozilla-central/last-fires we find https://taskcluster-artifacts.net/Cc-Y1crrRJOlnu2YM14VJg/0/public/logs/live_backing.log

[task 2019-06-06T10:00:30.154Z] retry: calling query_pushlog with args: (u'https://hg.mozilla.org/mozilla-central/json-pushes?version=2&changeset=fee989d27558bdbf83351633f09bd7f80d16f2d4&tipsonly=1&full=1',), kwargs: {}, attempt #1
[task 2019-06-06T10:00:30.155Z] Starting new HTTPS connection (1): hg.mozilla.org
[task 2019-06-06T10:00:30.318Z] "GET /mozilla-central/json-pushes?version=2&changeset=fee989d27558bdbf83351633f09bd7f80d16f2d4&tipsonly=1&full=1 HTTP/1.1" 200 None
[task 2019-06-06T10:00:30.347Z] Traceback (most recent call last):
[task 2019-06-06T10:00:30.347Z]   File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/cron/__init__.py", line 67, in run_job
[task 2019-06-06T10:00:30.348Z]     tasks = JOB_TYPES[job_type](job['job'], params, root=root)
[task 2019-06-06T10:00:30.348Z]   File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/cron/decision.py", line 38, in run_decision_task
[task 2019-06-06T10:00:30.348Z]     root=root),
[task 2019-06-06T10:00:30.348Z]   File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/cron/decision.py", line 48, in make_decision_task
[task 2019-06-06T10:00:30.349Z]     params['head_rev'])
[task 2019-06-06T10:00:30.349Z]   File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/util/hg.py", line 33, in find_hg_revision_push_info
[task 2019-06-06T10:00:30.349Z]     repository, revision, pushes
[task 2019-06-06T10:00:30.349Z] RuntimeError: Unable to find a single pushlog_id for https://hg.mozilla.org/mozilla-central revision fee989d27558bdbf83351633f09bd7f80d16f2d4: {}
[task 2019-06-06T10:00:30.349Z] cron job nightly-desktop run failed; continuing to next job

It wasn't retried since the actual http operation was a success, it just didn't have the expected content.
The code currently checks to see if the length of the pushes is != 1 in https://hg.mozilla.org/mozilla-central/file/tip/taskcluster/taskgraph/util/hg.py#l30

There was a merge at 10:00:05 UTC - for the revision mentioned above - and so current hypothesis is that the length of the pushes was zero. If this is confirmed perhaps we should be retrying on that error, and aborting if >1

Comment 1

[Simon Fraser [:sfraser] ⌚️GMT]

This affected
nightly-desktop
nightly-geckoview
periodic-update
searchfox-index

Comment 2

[Simon Fraser [:sfraser] ⌚️GMT]

Attachment: Bug 1557331 Cover missing-data scenario with retries in util.hg r=dustin

If the length of the pushes from the pushlog is wrong, raise exception and retry.

Comment 3

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/dcb20740187a
Cover missing-data scenario with retries in util.hg r=dustin

Comment 4

[Noemi Erli[:noemi_erli]]

https://hg.mozilla.org/mozilla-central/rev/dcb20740187a