Closed Bug 1401725 Opened 2 years ago Closed 2 years ago

Use sparse checkout for toolchain tasks

Categories

(Firefox Build System :: General, enhancement)

enhancement
Not set

Tracking

(firefox58 fixed)

RESOLVED FIXED
mozilla58
Tracking Status
firefox58 --- fixed

People

(Reporter: gps, Assigned: gps)

References

(Blocks 1 open bug)

Details

Attachments

(1 file, 1 obsolete file)

We're currently doing a full checkout for toolchain tasks even though they need very few files. They are a prime candidate for sparse checkouts.

And, I've written this patch a few times because I was annoyed with how long Try pushes were taking. So I'm just going to submit it :)
Comment on attachment 8910476 [details]
Bug 1401725 - Use sparse checkouts for toolchain tasks;

https://reviewboard.mozilla.org/r/181906/#review187430

::: build/sparse-profiles/toolchain-build:4
(Diff revision 4)
> +%include build/sparse-profiles/mach
> +
> +[include]
> +path:browser/config/tooltool-manifests/
> +
> +path:build/
> +path:taskcluster/
> +

I wonder if it wouldn't be better to just %include build/sparse-profiles/taskgraph, which should be a superset (and if not, there's a problem in the taskgraph)
Attachment #8910476 - Flags: review?(mh+mozilla) → review+
Comment on attachment 8910476 [details]
Bug 1401725 - Use sparse checkouts for toolchain tasks;

https://reviewboard.mozilla.org/r/181906/#review187430

> I wonder if it wouldn't be better to just %include build/sparse-profiles/taskgraph, which should be a superset (and if not, there's a problem in the taskgraph)

I agree. I've made the change.

The toolchain profile still adds tools/rewriting/ because of that one toolchain task. That's kinda wonky to me and is possibly deserving of its own bug. I'll file something.
Pushed by gszorc@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/af6c2544903e
Use sparse checkouts for toolchain tasks; r=glandium
I had to back this out for the following decision task failure:

https://public-artifacts.taskcluster.net/PcpguSSvR2qaFMuhPLwQBA/0/public/logs/live_backing.log

[task 2017-09-21T14:39:23.168306Z] "GET /integration/autoland/json-automationrelevance/af6c2544903e3281baf169b682daeef515f77ee3 HTTP/1.1" 200 None
[task 2017-09-21T14:39:23.174459Z] 1 commits influencing task scheduling:
[task 2017-09-21T14:39:23.174537Z]  af6c2544903e Bug 1401725 - Use sparse checkouts for toolchain tasks; r=glandium
[task 2017-09-21T14:39:23.546879Z] "GET /index/v1/task/gecko.cache.level-3.toolchains.v1.linux64-binutils.2856d9293652b9192c77948dfb65c9a55fc465d4d13e99b509a43d665c39e28a HTTP/1.1" 404 452
[task 2017-09-21T14:39:23.548141Z] Traceback (most recent call last):
[task 2017-09-21T14:39:23.548250Z]   File "/builds/worker/checkouts/gecko/taskcluster/mach_commands.py", line 154, in taskgraph_decision
[task 2017-09-21T14:39:23.548347Z]     return taskgraph.decision.taskgraph_decision(options)
[task 2017-09-21T14:39:23.548522Z]   File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/decision.py", line 138, in taskgraph_decision
[task 2017-09-21T14:39:23.548657Z]     write_artifact('task-graph.json', tgg.morphed_task_graph.to_json())
[task 2017-09-21T14:39:23.548813Z]   File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/generator.py", line 184, in morphed_task_graph
[task 2017-09-21T14:39:23.549009Z]     return self._run_until('morphed_task_graph')
[task 2017-09-21T14:39:23.549142Z]   File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/generator.py", line 289, in _run_until
[task 2017-09-21T14:39:23.549275Z]     k, v = self._run.next()
[task 2017-09-21T14:39:23.549387Z]   File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/generator.py", line 276, in _run
[task 2017-09-21T14:39:23.549495Z]     do_not_optimize)
[task 2017-09-21T14:39:23.549667Z]   File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/optimize.py", line 42, in optimize_task_graph
[task 2017-09-21T14:39:23.549785Z]     existing_tasks=existing_tasks)
[task 2017-09-21T14:39:23.549934Z]   File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/optimize.py", line 96, in annotate_task_graph
[task 2017-09-21T14:39:23.550046Z]     opt_result = optimize_task(task, params)
[task 2017-09-21T14:39:23.550174Z]   File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/optimize.py", line 54, in optimize_task
[task 2017-09-21T14:39:23.550274Z]     opt_result = opt_fn(task, params, *args)
[task 2017-09-21T14:39:23.550354Z]   File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/optimize.py", line 175, in opt_index_search
[task 2017-09-21T14:39:23.550472Z]     use_proxy=bool(os.environ.get('TASK_ID')))
[task 2017-09-21T14:39:23.550542Z]   File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/util/taskcluster.py", line 90, in find_task_id
[task 2017-09-21T14:39:23.550637Z]     raise KeyError("index path {} not found".format(index_path))
[task 2017-09-21T14:39:23.550732Z] KeyError: u'index path gecko.cache.level-3.toolchains.v1.linux64-binutils.2856d9293652b9192c77948dfb65c9a55fc465d4d13e99b509a43d665c39e28a not found'
Backout by gszorc@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/e66f516aa655
Backed out changeset af6c2544903e for uncovering a taskgraph bug
Comment on attachment 8910765 [details]
Bug 1401725 - Catch HTTP 404 errors during taskgraph optimization;

https://reviewboard.mozilla.org/r/182240/#review187562

See the last cset in bug 1383880.
Attachment #8910765 - Flags: review?(dustin)
Attachment #8910765 - Attachment is obsolete: true
Pushed by gszorc@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/bffb6b9d7212
Use sparse checkouts for toolchain tasks; r=glandium
https://hg.mozilla.org/mozilla-central/rev/bffb6b9d7212
Status: ASSIGNED → RESOLVED
Closed: 2 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla58
Product: Core → Firefox Build System
You need to log in before you can comment on or make changes to this bug.