1395563 - taskcluster backfill action task (BK) fails to run

Status: RESOLVED FIXED

Description

[Joel Maher ( :jmaher ) (UTC-4)]

I tried to backfill (tc only stuff, no BBB):
https://treeherder.mozilla.org/#/jobs?repo=autoland&revision=6b9d06ba6f769234530ae67d8353377d58a93fd0&filter-searchStr=action

the Bk job is red, so I did edit and recreate, and s/builds/home/, then it failed again.

Comment 1

[Intermittent Failures Robot]

1 failures in 924 pushes (0.001 failures/push) were associated with this bug in the last 7 days.   

Repository breakdown:
* mozilla-inbound: 1

Platform breakdown:
* gecko-3-decision: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1395563&startday=2017-09-04&endday=2017-09-10&tree=all

Comment 2

[Intermittent Failures Robot]

1 failures in 1032 pushes (0.001 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* autoland: 1

Platform breakdown:
* gecko-3-decision: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1395563&startday=2017-09-11&endday=2017-09-17&tree=all

Comment 3

[Joel Maher ( :jmaher ) (UTC-4)]

this is still broken:
https://public-artifacts.taskcluster.net/NXkB_-L1QUmeEVbrzljLVQ/0/public/logs/live_backing.log

[task 2017-09-20T16:47:27.634783Z] + cd /builds/worker/checkouts/gecko
[task 2017-09-20T16:47:27.634853Z] + ln -s /builds/worker/artifacts artifacts
[task 2017-09-20T16:47:27.635771Z] + ./mach --log-no-times taskgraph action-callback
[task 2017-09-20T16:47:28.279700Z] Starting new HTTPS connection (1): hg.mozilla.org
[task 2017-09-20T16:47:28.714968Z] "GET /integration/mozilla-inbound/json-pushes?version=2&startID=95179&endID=95184 HTTP/1.1" 200 None
[task 2017-09-20T16:47:28.718509Z] Starting new HTTPS connection (1): index.taskcluster.net
[task 2017-09-20T16:47:29.182339Z] "GET /v1/task/gecko.v2.mozilla-inbound.pushlog-id.95180.decision/artifacts/public/full-task-graph.json HTTP/1.1" 404 498
[task 2017-09-20T16:47:29.182878Z] Traceback (most recent call last):
[task 2017-09-20T16:47:29.183090Z]   File "/builds/worker/checkouts/gecko/taskcluster/mach_commands.py", line 315, in action_callback
[task 2017-09-20T16:47:29.183203Z]     test=False)
[task 2017-09-20T16:47:29.183264Z]   File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/actions/registry.py", line 321, in trigger_action_callback
[task 2017-09-20T16:47:29.183306Z]     cb(Parameters(**parameters), input, task_group_id, task_id, task)
[task 2017-09-20T16:47:29.183372Z]   File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/actions/backfill.py", line 75, in backfill_action
[task 2017-09-20T16:47:29.183404Z]     'public/full-task-graph.json')
[task 2017-09-20T16:47:29.183458Z]   File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/util/taskcluster.py", line 92, in get_artifact_from_index
[task 2017-09-20T16:47:29.183502Z]     response = _do_request(get_index_url(full_path, use_proxy))
[task 2017-09-20T16:47:29.183553Z]   File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/util/taskcluster.py", line 37, in _do_request
[task 2017-09-20T16:47:29.183576Z]     response.raise_for_status()
[task 2017-09-20T16:47:29.183629Z]   File "/builds/worker/checkouts/gecko/third_party/python/requests/requests/models.py", line 840, in raise_for_status
[task 2017-09-20T16:47:29.183714Z]     raise HTTPError(http_error_msg, response=self)
[task 2017-09-20T16:47:29.183796Z] HTTPError: 404 Client Error: Not Found for url: https://index.taskcluster.net/v1/task/gecko.v2.mozilla-inbound.pushlog-id.95180.decision/artifacts/public/full-task-graph.json

:garndt, can you ensure that these services are working and we have tests available to keep them working?

Comment 4

[Brian Stack [:bstack]]

Investigation:

* https://hg.mozilla.org/integration/mozilla-inbound/json-pushes?version=2&startID=95179&endID=95184 is the set of pushes it is attempting to backfill.

* All of those other than 95180 exist in the index.

* https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&revision=331bcdb4167c725609e9fadf41f7815a42dd7e3f is what should have been 95180 in the index

* It does not appear to have a decision task at all? I didn't think that was possible on inbound.

What should we do in the case where there is a push with no decision task for backfilling? Just skip it?

Comment 5

[Joel Maher ( :jmaher ) (UTC-4)]

thanks for looking into this :bstack.  odd that there is no decision task, that would clearly explain it.

Comment 6

[Brian Stack [:bstack]]

Created attachment 8910402 [details]
Bug 1395563 - Handle case of missing decision task for backfills

Review commit: https://reviewboard.mozilla.org/r/181846/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/181846/

Comment 7

[Brian Stack [:bstack]]

Comment on attachment 8910402 [details]
Bug 1395563 - Handle case of missing decision task for backfills

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/181846/diff/1-2/

Comment 8

[Dustin J. Mitchell [:dustin] pronoun: he]

Comment on attachment 8910402 [details]
Bug 1395563 - Handle case of missing decision task for backfills

https://reviewboard.mozilla.org/r/181846/#review187598

Comment 9

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/7f987fe0f425
Handle case of missing decision task for backfills r=dustin

Comment 10

[Phil Ringnalda (:philor)]

https://hg.mozilla.org/mozilla-central/rev/7f987fe0f425