Closed Bug 1539905 Opened 6 years ago Closed 6 years ago

taskgraph generation is too slow after declarative artifacts landed

Categories

(Release Engineering :: General, enhancement)

enhancement
Not set
normal

Tracking

(firefox68 fixed)

RESOLVED FIXED
Tracking Status
firefox68 --- fixed

People

(Reporter: catlee, Assigned: catlee)

References

Details

Attachments

(1 file, 1 obsolete file)

The work in bug 1535679 added support for declarative artifacts for Firefox builds, but unfortunately regressed mach taskgraph times. It looks like this is due to calling load_yaml multiple times for the same file. If we can cache the results of load_yaml, then mach taskgraph speeds up by around 60s.

Attached file Bug 1539905: Cache yaml loading (obsolete) —
Assignee: nobody → catlee

My understanding is that even with this caching patch, task generation can take ~30s longer than it did before. Would it be possible to somehow rig things such that declarative artifacts are disabled when taskgraph.fast is True?

Flags: needinfo?(catlee)

On my machine, mach taskgraph target --fast takes about 20s comparing e3762382f790 (2019-03-02) to 31334b642715 (2019-03-29)

Flags: needinfo?(catlee)
Status: NEW → RESOLVED
Closed: 6 years ago
Resolution: --- → FIXED

This has caused some of the beetmover jobs to attempt file uploads to the wrong location. We take the data returned by load yaml and pass it through resolve_keyed_by, which changes it. Unfortunately, this is the same object in memory that cached_load_yaml is holding on to, so we never get to resolve those keys a different way in subsequent tasks.

This bit us with mozilla-beta today and uploading the firefox sources

https://taskcluster-artifacts.net/Y7USRDMjQWut9wdwoVeWEg/1/public/logs/live_backing.log

Status: RESOLVED → REOPENED
Resolution: FIXED → ---

We'll see what happens in the new beta - for now, clearing the NI to my namesake.

Flags: needinfo?(sfraser_bugs)

What is happening!

Pushed by sfraser@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/648001848ed2 Ensure a copy is taken of memoized values r=mtabara
See Also: → 1541840

(In reply to Pulsebot from comment #12)

Pushed by sfraser@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/648001848ed2
Ensure a copy is taken of memoized values r=mtabara

Sheriffs will take this to central and close the bug.

In the meantime, I took the backed-out patch 7a9123da87cc + new patch 648001848ed2 from Simon and baked them together on beta: https://hg.mozilla.org/releases/mozilla-beta/rev/864e16abd2f7

Status: REOPENED → RESOLVED
Closed: 6 years ago6 years ago
Resolution: --- → FIXED
Attachment #9054277 - Attachment is obsolete: true
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: