1401725 - Use sparse checkout for toolchain tasks

Status: RESOLVED FIXED

(Firefox Build System :: General, enhancement)

Description

[Gregory Szorc [:gps]]

We're currently doing a full checkout for toolchain tasks even though they need very few files. They are a prime candidate for sparse checkouts.

And, I've written this patch a few times because I was annoyed with how long Try pushes were taking. So I'm just going to submit it :)

Comment 1

[Gregory Szorc [:gps]]

Attachment: Bug 1401725 - Use sparse checkouts for toolchain tasks;

Strictly speaking we don't need all files in the directories listed
in the profile. But the checkout is still small enough and it is far
less effort than cherry-picking every file needed by every toolchain
task.

Review commit: https://reviewboard.mozilla.org/r/181906/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/181906/

Comment 2

[Gregory Szorc [:gps]]

Comment on attachment 8910476 [details]
Bug 1401725 - Use sparse checkouts for toolchain tasks;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/181906/diff/1-2/

Comment 3

[Gregory Szorc [:gps]]

Comment on attachment 8910476 [details]
Bug 1401725 - Use sparse checkouts for toolchain tasks;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/181906/diff/2-3/

Comment 4

[Gregory Szorc [:gps]]

Comment on attachment 8910476 [details]
Bug 1401725 - Use sparse checkouts for toolchain tasks;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/181906/diff/3-4/

Comment 5

[Mike Hommey [:glandium]]

Comment on attachment 8910476 [details]
Bug 1401725 - Use sparse checkouts for toolchain tasks;

https://reviewboard.mozilla.org/r/181906/#review187430

::: build/sparse-profiles/toolchain-build:4
(Diff revision 4)
> +%include build/sparse-profiles/mach
> +
> +[include]
> +path:browser/config/tooltool-manifests/
> +
> +path:build/
> +path:taskcluster/
> +

I wonder if it wouldn't be better to just %include build/sparse-profiles/taskgraph, which should be a superset (and if not, there's a problem in the taskgraph)

Comment 6

[Gregory Szorc [:gps]]

Comment on attachment 8910476 [details]
Bug 1401725 - Use sparse checkouts for toolchain tasks;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/181906/diff/4-5/

Comment 7

[Gregory Szorc [:gps]]

Comment on attachment 8910476 [details]
Bug 1401725 - Use sparse checkouts for toolchain tasks;

https://reviewboard.mozilla.org/r/181906/#review187430

> I wonder if it wouldn't be better to just %include build/sparse-profiles/taskgraph, which should be a superset (and if not, there's a problem in the taskgraph)

I agree. I've made the change.

The toolchain profile still adds tools/rewriting/ because of that one toolchain task. That's kinda wonky to me and is possibly deserving of its own bug. I'll file something.

Comment 8

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/af6c2544903e
Use sparse checkouts for toolchain tasks; r=glandium

Comment 9

[Gregory Szorc [:gps]]

I had to back this out for the following decision task failure:

https://public-artifacts.taskcluster.net/PcpguSSvR2qaFMuhPLwQBA/0/public/logs/live_backing.log

[task 2017-09-21T14:39:23.168306Z] "GET /integration/autoland/json-automationrelevance/af6c2544903e3281baf169b682daeef515f77ee3 HTTP/1.1" 200 None
[task 2017-09-21T14:39:23.174459Z] 1 commits influencing task scheduling:
[task 2017-09-21T14:39:23.174537Z]  af6c2544903e Bug 1401725 - Use sparse checkouts for toolchain tasks; r=glandium
[task 2017-09-21T14:39:23.546879Z] "GET /index/v1/task/gecko.cache.level-3.toolchains.v1.linux64-binutils.2856d9293652b9192c77948dfb65c9a55fc465d4d13e99b509a43d665c39e28a HTTP/1.1" 404 452
[task 2017-09-21T14:39:23.548141Z] Traceback (most recent call last):
[task 2017-09-21T14:39:23.548250Z]   File "/builds/worker/checkouts/gecko/taskcluster/mach_commands.py", line 154, in taskgraph_decision
[task 2017-09-21T14:39:23.548347Z]     return taskgraph.decision.taskgraph_decision(options)
[task 2017-09-21T14:39:23.548522Z]   File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/decision.py", line 138, in taskgraph_decision
[task 2017-09-21T14:39:23.548657Z]     write_artifact('task-graph.json', tgg.morphed_task_graph.to_json())
[task 2017-09-21T14:39:23.548813Z]   File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/generator.py", line 184, in morphed_task_graph
[task 2017-09-21T14:39:23.549009Z]     return self._run_until('morphed_task_graph')
[task 2017-09-21T14:39:23.549142Z]   File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/generator.py", line 289, in _run_until
[task 2017-09-21T14:39:23.549275Z]     k, v = self._run.next()
[task 2017-09-21T14:39:23.549387Z]   File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/generator.py", line 276, in _run
[task 2017-09-21T14:39:23.549495Z]     do_not_optimize)
[task 2017-09-21T14:39:23.549667Z]   File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/optimize.py", line 42, in optimize_task_graph
[task 2017-09-21T14:39:23.549785Z]     existing_tasks=existing_tasks)
[task 2017-09-21T14:39:23.549934Z]   File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/optimize.py", line 96, in annotate_task_graph
[task 2017-09-21T14:39:23.550046Z]     opt_result = optimize_task(task, params)
[task 2017-09-21T14:39:23.550174Z]   File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/optimize.py", line 54, in optimize_task
[task 2017-09-21T14:39:23.550274Z]     opt_result = opt_fn(task, params, *args)
[task 2017-09-21T14:39:23.550354Z]   File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/optimize.py", line 175, in opt_index_search
[task 2017-09-21T14:39:23.550472Z]     use_proxy=bool(os.environ.get('TASK_ID')))
[task 2017-09-21T14:39:23.550542Z]   File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/util/taskcluster.py", line 90, in find_task_id
[task 2017-09-21T14:39:23.550637Z]     raise KeyError("index path {} not found".format(index_path))
[task 2017-09-21T14:39:23.550732Z] KeyError: u'index path gecko.cache.level-3.toolchains.v1.linux64-binutils.2856d9293652b9192c77948dfb65c9a55fc465d4d13e99b509a43d665c39e28a not found'

Comment 10

[Gregory Szorc [:gps]]

Attachment: Bug 1401725 - Catch HTTP 404 errors during taskgraph optimization;

find_task_id() converts HTTP 404 to KeyError. So we need to trap
an additional exception to handle this failure.

Review commit: https://reviewboard.mozilla.org/r/182240/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/182240/

Comment 11

[Gregory Szorc [:gps]]

Comment on attachment 8910476 [details]
Bug 1401725 - Use sparse checkouts for toolchain tasks;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/181906/diff/5-6/

Comment 12

[Pulsebot]

Backout by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/e66f516aa655
Backed out changeset af6c2544903e for uncovering a taskgraph bug

Comment 13

[Dustin J. Mitchell [:dustin] (he/him)]

Comment on attachment 8910765 [details]
Bug 1401725 - Catch HTTP 404 errors during taskgraph optimization;

https://reviewboard.mozilla.org/r/182240/#review187562

See the last cset in bug 1383880.

Comment 14

[Gregory Szorc [:gps]]

Comment on attachment 8910476 [details]
Bug 1401725 - Use sparse checkouts for toolchain tasks;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/181906/diff/6-7/

Comment 15

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/bffb6b9d7212
Use sparse checkouts for toolchain tasks; r=glandium

Comment 16

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

https://hg.mozilla.org/mozilla-central/rev/bffb6b9d7212