Nightly cron failure related to merge
Categories
(Release Engineering :: Release Automation: Other, defect)
Tracking
(firefox69 fixed)
| Tracking | Status | |
|---|---|---|
| firefox69 | --- | fixed |
People
(Reporter: sfraser, Unassigned)
Details
Attachments
(1 file)
This morning's nightly didn't happen on mozilla-central.
Looking at old cron firings in https://hooks.taskcluster.net/v1/hooks/project-releng/cron-task-mozilla-central/last-fires we find https://taskcluster-artifacts.net/Cc-Y1crrRJOlnu2YM14VJg/0/public/logs/live_backing.log
[task 2019-06-06T10:00:30.154Z] retry: calling query_pushlog with args: (u'https://hg.mozilla.org/mozilla-central/json-pushes?version=2&changeset=fee989d27558bdbf83351633f09bd7f80d16f2d4&tipsonly=1&full=1',), kwargs: {}, attempt #1
[task 2019-06-06T10:00:30.155Z] Starting new HTTPS connection (1): hg.mozilla.org
[task 2019-06-06T10:00:30.318Z] "GET /mozilla-central/json-pushes?version=2&changeset=fee989d27558bdbf83351633f09bd7f80d16f2d4&tipsonly=1&full=1 HTTP/1.1" 200 None
[task 2019-06-06T10:00:30.347Z] Traceback (most recent call last):
[task 2019-06-06T10:00:30.347Z] File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/cron/__init__.py", line 67, in run_job
[task 2019-06-06T10:00:30.348Z] tasks = JOB_TYPES[job_type](job['job'], params, root=root)
[task 2019-06-06T10:00:30.348Z] File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/cron/decision.py", line 38, in run_decision_task
[task 2019-06-06T10:00:30.348Z] root=root),
[task 2019-06-06T10:00:30.348Z] File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/cron/decision.py", line 48, in make_decision_task
[task 2019-06-06T10:00:30.349Z] params['head_rev'])
[task 2019-06-06T10:00:30.349Z] File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/util/hg.py", line 33, in find_hg_revision_push_info
[task 2019-06-06T10:00:30.349Z] repository, revision, pushes
[task 2019-06-06T10:00:30.349Z] RuntimeError: Unable to find a single pushlog_id for https://hg.mozilla.org/mozilla-central revision fee989d27558bdbf83351633f09bd7f80d16f2d4: {}
[task 2019-06-06T10:00:30.349Z] cron job nightly-desktop run failed; continuing to next job
It wasn't retried since the actual http operation was a success, it just didn't have the expected content.
The code currently checks to see if the length of the pushes is != 1 in https://hg.mozilla.org/mozilla-central/file/tip/taskcluster/taskgraph/util/hg.py#l30
There was a merge at 10:00:05 UTC - for the revision mentioned above - and so current hypothesis is that the length of the pushes was zero. If this is confirmed perhaps we should be retrying on that error, and aborting if >1
|
Reporter | |
Comment 1•5 years ago
|
||
This affected
nightly-desktop
nightly-geckoview
periodic-update
searchfox-index
|
|
Reporter | |
Comment 2•5 years ago
|
||
If the length of the pushes from the pushlog is wrong, raise exception and retry.
Pushed by sfraser@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/dcb20740187a Cover missing-data scenario with retries in util.hg r=dustin
|
||
Comment 4•5 years ago
|
||
| bugherder | ||
Description
•