1407279 - Docker image changes on mozilla-beta result in failed decision tasks

Status: RESOLVED FIXED

(Firefox Build System :: Task Configuration, task)

Description

[Rail Aliiev [:rail]]

I tried to uplift a patch to mozilla-beta, which also touches one of the docker images under taskcluster/docker. The corresponding decision taks failed https://tools.taskcluster.net/groups/XugxWvNaRFOf3SR1hJHlsg/tasks/XugxWvNaRFOf3SR1hJHlsg/runs/0/logs/public%2Flogs%2Flive.log:

[task 2017-10-10T15:00:08.268737Z] "GET /index/v1/task/docker.images.v2.level-3.funsize-update-generator.hash.600cc7c926a7c83498b1dd523c8df4a848417220bf6ee0a61a9c14786ca6dec0 HTTP/1.1" 404 456
[task 2017-10-10T15:00:08.269916Z] Traceback (most recent call last):
[task 2017-10-10T15:00:08.269986Z]   File "/builds/worker/checkouts/gecko/taskcluster/mach_commands.py", line 154, in taskgraph_decision
[task 2017-10-10T15:00:08.270034Z]     return taskgraph.decision.taskgraph_decision(options)
[task 2017-10-10T15:00:08.270095Z]   File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/decision.py", line 138, in taskgraph_decision
[task 2017-10-10T15:00:08.270151Z]     write_artifact('task-graph.json', tgg.morphed_task_graph.to_json())
[task 2017-10-10T15:00:08.270207Z]   File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/generator.py", line 184, in morphed_task_graph
[task 2017-10-10T15:00:08.270255Z]     return self._run_until('morphed_task_graph')
[task 2017-10-10T15:00:08.270313Z]   File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/generator.py", line 289, in _run_until
[task 2017-10-10T15:00:08.270350Z]     k, v = self._run.next()
[task 2017-10-10T15:00:08.270399Z]   File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/generator.py", line 276, in _run
[task 2017-10-10T15:00:08.270432Z]     do_not_optimize)
[task 2017-10-10T15:00:08.270491Z]   File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/optimize.py", line 42, in optimize_task_graph
[task 2017-10-10T15:00:08.270528Z]     existing_tasks=existing_tasks)
[task 2017-10-10T15:00:08.270588Z]   File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/optimize.py", line 96, in annotate_task_graph
[task 2017-10-10T15:00:08.270629Z]     opt_result = optimize_task(task, params)
[task 2017-10-10T15:00:08.270685Z]   File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/optimize.py", line 54, in optimize_task
[task 2017-10-10T15:00:08.270725Z]     opt_result = opt_fn(task, params, *args)
[task 2017-10-10T15:00:08.270780Z]   File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/optimize.py", line 175, in opt_index_search
[task 2017-10-10T15:00:08.270819Z]     use_proxy=bool(os.environ.get('TASK_ID')))
[task 2017-10-10T15:00:08.270874Z]   File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/util/taskcluster.py", line 90, in find_task_id
[task 2017-10-10T15:00:08.270919Z]     raise KeyError("index path {} not found".format(index_path))
[task 2017-10-10T15:00:08.270991Z] KeyError: u'index path docker.images.v2.level-3.funsize-update-generator.hash.600cc7c926a7c83498b1dd523c8df4a848417220bf6ee0a61a9c14786ca6dec0 not found'

Looks like https://hg.mozilla.org/mozilla-central/rev/83fdd1638d8e has made to beta (57 atm), but the corresponding function usage still uses a different exception https://dxr.mozilla.org/mozilla-beta/rev/b92b69f3503e71ea7a7184f3f56526aafa57d999/taskcluster/taskgraph/optimize.py#178

Mozilla-central has changed it in https://hg.mozilla.org/mozilla-central/rev/79bb52091750#l2.394. It's not the smallest thing to uplift, so I wonder if it's just easier to backout https://hg.mozilla.org/mozilla-central/rev/83fdd1638d8e from beta instead and let the huge change ride the trains.

Comment 1

[Rail Aliiev [:rail]]

Dustin, what do you think about backing out https://hg.mozilla.org/mozilla-central/rev/83fdd1638d8e from beta?

Comment 2

[Dustin J. Mitchell [:dustin] (he/him)]

Seems like a good fix.  I'm confused how things became dissociated here, but .. ok :)

Comment 3

[Rail Aliiev [:rail]]

Attachment: Bug 1407279 - Docker image changes on mozilla-beta result in failed decision tasks, Backed out changeset 83fdd1638d8e a=release

Review commit: https://reviewboard.mozilla.org/r/188110/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/188110/

Comment 4

[Rail Aliiev [:rail]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=603d87b1f9c27697008b86d94cd1913395ebc9d9&selectedJob=136008093 looks fine. I cancelled all other tasks.

Comment 5

[Dustin J. Mitchell [:dustin] (he/him)]

Comment on attachment 8917099 [details]
Bug 1407279 - Docker image changes on mozilla-beta result in failed decision tasks, Backed out changeset 83fdd1638d8e  a=release

https://reviewboard.mozilla.org/r/188110/#review193264

I assume we'll need to put this back in when the next uplift occurs?

Comment 6

[Rail Aliiev [:rail]]

(In reply to Dustin J. Mitchell [:dustin] from comment #5)
> I assume we'll need to put this back in when the next uplift occurs?

No, no extra work is required. These beta-only changes will be discarded by the next uplift from mozilla-central to mozilla-beta.

Comment 7

[Rail Aliiev [:rail]]

Comment on attachment 8917099 [details]
Bug 1407279 - Docker image changes on mozilla-beta result in failed decision tasks, Backed out changeset 83fdd1638d8e  a=release

I'd like to land this on beta (only) to unblock LZMA/BZ2 related work. 

Unfortunately some TC related optimization work in bug 1401183 and bug 1383880 have landed on the same day, but several hours apart. That was enough to get one of the changes in gecko 57, and the other one in 58. Backing out the first part (this patch) from beta sound like the safest solution here. The backout will be discarded as a part of the next uplift.

[Feature/Bug causing the regression]: Bug 1401183 and bug 1383880
[User impact if declined]: None
[Needs manual test from QE? If yes, steps to reproduce]: No
[List of other uplifts needed for the feature/fix]: None
[Is the change risky?]: No, it affects decision tasks only
[Why is the change risky/not risky?]: It doesn't affect the browser code, automation (scheduling) only

Comment 8

[Ritu Kothari (:ritu) (Inactive, please n-i to RyanVM, jcristau, or pascal)]

Comment on attachment 8917099 [details]
Bug 1407279 - Docker image changes on mozilla-beta result in failed decision tasks, Backed out changeset 83fdd1638d8e  a=release

NPOTB, Beta57+

Comment 9

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-beta/rev/4c2e84704183