taskgraph generation is too slow after declarative artifacts landed

RESOLVED FIXED

Status

enhancement
RESOLVED FIXED
3 months ago
3 months ago

People

(Reporter: catlee, Assigned: catlee)

Tracking

unspecified
Dependency tree / graph

Firefox Tracking Flags

(firefox68 fixed)

Details

Attachments

(2 attachments)

Assignee

Description

3 months ago

The work in bug 1535679 added support for declarative artifacts for Firefox builds, but unfortunately regressed mach taskgraph times. It looks like this is due to calling load_yaml multiple times for the same file. If we can cache the results of load_yaml, then mach taskgraph speeds up by around 60s.

Assignee

Updated

3 months ago
Assignee: nobody → catlee

My understanding is that even with this caching patch, task generation can take ~30s longer than it did before. Would it be possible to somehow rig things such that declarative artifacts are disabled when taskgraph.fast is True?

Flags: needinfo?(catlee)
Assignee

Comment 4

3 months ago

On my machine, mach taskgraph target --fast takes about 20s comparing e3762382f790 (2019-03-02) to 31334b642715 (2019-03-29)

Flags: needinfo?(catlee)

Comment 5

3 months ago
bugherder
Status: NEW → RESOLVED
Closed: 3 months ago
Resolution: --- → FIXED

This has caused some of the beetmover jobs to attempt file uploads to the wrong location. We take the data returned by load yaml and pass it through resolve_keyed_by, which changes it. Unfortunately, this is the same object in memory that cached_load_yaml is holding on to, so we never get to resolve those keys a different way in subsequent tasks.

This bit us with mozilla-beta today and uploading the firefox sources

https://taskcluster-artifacts.net/Y7USRDMjQWut9wdwoVeWEg/1/public/logs/live_backing.log

Status: RESOLVED → REOPENED
Resolution: FIXED → ---

We'll see what happens in the new beta - for now, clearing the NI to my namesake.

Flags: needinfo?(sfraser_bugs)

What is happening!

Comment 12

3 months ago
Pushed by sfraser@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/648001848ed2
Ensure a copy is taken of memoized values r=mtabara
See Also: → 1541840
Duplicate of this bug: 1541840

(In reply to Pulsebot from comment #12)

Pushed by sfraser@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/648001848ed2
Ensure a copy is taken of memoized values r=mtabara

Sheriffs will take this to central and close the bug.

In the meantime, I took the backed-out patch 7a9123da87cc + new patch 648001848ed2 from Simon and baked them together on beta: https://hg.mozilla.org/releases/mozilla-beta/rev/864e16abd2f7

Comment 15

3 months ago
bugherder
Status: REOPENED → RESOLVED
Closed: 3 months ago3 months ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.