1667482 - Intermittent requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/GJLIrPE4Sm-G9TzK7-3hQw/artifacts/public/parameters.yml

Status: RESOLVED FIXED

(Firefox Build System :: Task Configuration, defect, P5)

Description

[Treeherder Bug Filer]

Filed by: nbeleuzu [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer.html#?job_id=316747715&repo=autoland
Full log: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/OUN3Pb9aQ_mIwPPG5ayLJA/runs/0/artifacts/public/logs/live_backing.log

---

[task 2020-09-25T21:06:37.874Z] Traceback (most recent call last):
[task 2020-09-25T21:06:37.874Z]   File "/builds/worker/checkouts/gecko/taskcluster/mach_commands.py", line 208, in taskgraph_decision
[task 2020-09-25T21:06:37.874Z]     ret = taskgraph.decision.taskgraph_decision(options)
[task 2020-09-25T21:06:37.874Z]   File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/decision.py", line 232, in taskgraph_decision
[task 2020-09-25T21:06:37.874Z]     set_decision_indexes(decision_task_id, tgg.parameters, tgg.graph_config)
[task 2020-09-25T21:06:37.874Z]   File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/generator.py", line 145, in parameters
[task 2020-09-25T21:06:37.874Z]     return self._run_until('parameters')
[task 2020-09-25T21:06:37.874Z]   File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/generator.py", line 376, in _run_until
[task 2020-09-25T21:06:37.874Z]     k, v = next(self._run)
[task 2020-09-25T21:06:37.874Z]   File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/generator.py", line 241, in _run
[task 2020-09-25T21:06:37.874Z]     parameters = self._parameters(graph_config)
[task 2020-09-25T21:06:37.874Z]   File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/decision.py", line 218, in <lambda>
[task 2020-09-25T21:06:37.874Z]     lambda graph_config: get_decision_parameters(graph_config, options)
[task 2020-09-25T21:06:37.874Z]   File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/decision.py", line 405, in get_decision_parameters
[task 2020-09-25T21:06:37.874Z]     parameters['backstop'] = is_backstop(parameters)
[task 2020-09-25T21:06:37.874Z]   File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/util/backstop.py", line 64, in is_backstop
[task 2020-09-25T21:06:37.874Z]     last_pushdate = get_artifact(last_backstop_id, 'public/parameters.yml')["pushdate"]
[task 2020-09-25T21:06:37.874Z]   File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/util/taskcluster.py", line 154, in get_artifact
[task 2020-09-25T21:06:37.874Z]     response = _do_request(get_artifact_url(task_id, path, use_proxy))
[task 2020-09-25T21:06:37.874Z]   File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/util/taskcluster.py", line 113, in _do_request
[task 2020-09-25T21:06:37.874Z]     response.raise_for_status()
[task 2020-09-25T21:06:37.874Z]   File "/builds/worker/checkouts/gecko/third_party/python/requests/requests/models.py", line 840, in raise_for_status
[task 2020-09-25T21:06:37.874Z]     raise HTTPError(http_error_msg, response=self)
[task 2020-09-25T21:06:37.874Z] requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/GJLIrPE4Sm-G9TzK7-3hQw/artifacts/public/parameters.yml
[taskcluster 2020-09-25 21:06:38.497Z] === Task Finished ===
[taskcluster 2020-09-25 21:06:38.598Z] Artifact "public/docker-contexts" not found at "/builds/worker/checkouts/gecko/docker-contexts"
[taskcluster 2020-09-25 21:06:38.705Z] Unsuccessful task run with exit code: 1 completed in 179.279 seconds

Comment 1

[Aki Sasaki (not active)]

Attachment: Bug 1667482 - retry taskcluster requests. r=#releng-reviewers,#taskgraph-reviewers

Comment 2

[Aki Sasaki (not active)]

Try push https://treeherder.mozilla.org/#/jobs?repo=try&revision=f2aa43b012c06e7d2a9f881309c347e75b455fea

Comment 3

[Intermittent Failures Robot]

4 failures in 4019 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 4

Platform breakdown:

• gecko-decision: 4

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1667482&startday=2020-09-21&endday=2020-09-27&tree=all

Comment 4

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/725f6e6d8aaf
retry taskcluster requests. r=taskgraph-reviewers,jmaher DONTBUILD

Comment 5

[Intermittent Failures Robot]

20 failures in 856 pushes (0.023 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 20

Platform breakdown:

• gecko-decision: 20

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1667482&startday=2020-09-30&endday=2020-09-30&tree=all

Comment 6

[Cosmin Sabou [:CosminS]]

https://hg.mozilla.org/mozilla-central/rev/725f6e6d8aaf

Comment 7

[Bogdan Tara[:bogdan_tara | bogdant]]

Backed out changeset 725f6e6d8aaf (bug 1667482) by aki's request.

Backout link: https://hg.mozilla.org/integration/autoland/rev/127cd76bff369fecb81dc99cde6e10bd2620b6e1

Comment 10

[Aki Sasaki (not active)]

Retries are just making this worse: we continue to 404, but it takes longer to get the result, and sometimes we even hit claim-expired. I think we need to figure out why the queue is giving us 404s for valid artifact URLs.

Comment 11

[Dustin J. Mitchell [:dustin] (he/him)]

[task 2020-09-25T21:06:37.874Z] requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/GJLIrPE4Sm-G9TzK7-3hQw/artifacts/public/parameters.yml

But https://firefox-ci-tc.services.mozilla.com/tasks/GJLIrPE4Sm-G9TzK7-3hQw has resolved time 2020-09-25T21:07:16.122Z. That was, I think, when the artifacts were finished uploading. The decision task's log has

[taskcluster 2020-09-25 21:06:59.366Z] === Task Finished ===

which is still 22 seconds after OUN3Pb9aQ_mIwPPG5ayLJA logged the 404.

So, I think those artifact URLs are not in fact valid at the time they are being requested. It looks like a dependency of some sort is missing.

Comment 12

[Aki Sasaki (not active)]

Oh. This might be fallout from :ahal adding indexes mid-decision task.

Comment 13

[Aki Sasaki (not active)]

Sheriffs: I suspect https://hg.mozilla.org/mozilla-central/rev/8289b6083a6143b2bd712084343967ec9ab01d5e .

Comment 14

[Andrew Halberstadt [:ahal]]

Ah, we moved the index insertion to the start of the decision task to avoid double backstops.. but then immediately afterwards rely on parameters.yml which is only available after the decision task anyway ><.

We only need the parameters.yml to compute the time component, so we can just assume that the current push is not a backstop in this case.

Comment 15

[Andrew Halberstadt [:ahal]]

For context, any decision task that started while a backstop decision task was still ongoing, would hit this issue. So that's why we were seeing strings of decision tasks that hit this in a row. I'll have a fix up soon.

Comment 16

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1667482 - [taskgraph] Don't fail if last backstop decision task is still running, r?marco

Comment 17

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/dd07adf3f7bd
[taskgraph] Don't fail if last backstop decision task is still running, r=marco

Comment 18

[Atila Butkovits]

https://hg.mozilla.org/mozilla-central/rev/dd07adf3f7bd

Comment 19

[Intermittent Failures Robot]

30 failures in 4195 pushes (0.007 failures/push) were associated with this bug in the last 7 days.

This is the #35 most frequent failure this week.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **

Repository breakdown:

• autoland: 30

Platform breakdown:

• gecko-decision: 30

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1667482&startday=2020-09-28&endday=2020-10-04&tree=all