taskgraph generation is too slow after declarative artifacts landed
Categories
(Release Engineering :: General, enhancement)
Tracking
(firefox68 fixed)
Tracking | Status | |
---|---|---|
firefox68 | --- | fixed |
People
(Reporter: catlee, Assigned: catlee)
References
Details
Attachments
(1 file, 1 obsolete file)
The work in bug 1535679 added support for declarative artifacts for Firefox builds, but unfortunately regressed mach taskgraph
times. It looks like this is due to calling load_yaml multiple times for the same file. If we can cache the results of load_yaml, then mach taskgraph
speeds up by around 60s.
Assignee | ||
Comment 1•6 years ago
|
||
Assignee | ||
Updated•6 years ago
|
Comment 3•6 years ago
|
||
My understanding is that even with this caching patch, task generation can take ~30s longer than it did before. Would it be possible to somehow rig things such that declarative artifacts are disabled when taskgraph.fast
is True?
Assignee | ||
Comment 4•6 years ago
•
|
||
On my machine, mach taskgraph target --fast
takes about 20s comparing e3762382f790 (2019-03-02) to 31334b642715 (2019-03-29)
Comment 5•6 years ago
|
||
bugherder |
Comment 6•6 years ago
|
||
(In reply to Oana Pop-Rus from comment #5)
Grafted to beta: https://hg.mozilla.org/releases/mozilla-beta/rev/7a9123da87cc
Comment 7•6 years ago
|
||
This has caused some of the beetmover jobs to attempt file uploads to the wrong location. We take the data returned by load yaml and pass it through resolve_keyed_by, which changes it. Unfortunately, this is the same object in memory that cached_load_yaml is holding on to, so we never get to resolve those keys a different way in subsequent tasks.
This bit us with mozilla-beta today and uploading the firefox sources
https://taskcluster-artifacts.net/Y7USRDMjQWut9wdwoVeWEg/1/public/logs/live_backing.log
Comment 8•6 years ago
|
||
Backed out changeset 7a9123da87cc (bug 1539905)for causing beetmover and update verify failures a=backout
Backout revision https://hg.mozilla.org/releases/mozilla-beta/rev/13b7d4d4df59f7f2e50f66644daf655bad40bec8
Log failure https://treeherder.mozilla.org/logviewer.html#?job_id=238116702&repo=mozilla-beta
Simon can you please take a look?
Comment 9•6 years ago
|
||
We'll see what happens in the new beta - for now, clearing the NI to my namesake.
Comment 10•6 years ago
|
||
What is happening!
Comment 11•6 years ago
|
||
Comment 12•6 years ago
|
||
Comment 14•6 years ago
|
||
(In reply to Pulsebot from comment #12)
Pushed by sfraser@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/648001848ed2
Ensure a copy is taken of memoized values r=mtabara
Sheriffs will take this to central and close the bug.
In the meantime, I took the backed-out patch 7a9123da87cc + new patch 648001848ed2 from Simon and baked them together on beta: https://hg.mozilla.org/releases/mozilla-beta/rev/864e16abd2f7
Comment 15•6 years ago
|
||
bugherder |
Updated•6 years ago
|
Description
•