1652086 - task generation performance has significantly regressed

Status: RESOLVED FIXED

(Firefox Build System :: Task Configuration, defect)

Description

[Nathan Froyd [:froydnj]]

Somewhere between 5766d99b88f379d3eb631085387cc9cbae438b6a and 4e9d6619c9d5a7306f66ca8d6f9d97579ba77b4d, task generation performance (e.g. generate the list of tasks for mach try fuzzy) has moved from "mildly annoying" to "borderline unusable".

Comment 1

[Tom Prince [:tomprince]]

Attachment: Bug 1652086: Make `kind_dependencies_tasks` a dictionary based on the label; r?Callek

I was looking at py-spy for generating the taskgraph, and found that a bunch
of time was spent in taskgraph.transforms.job.use_fetches[1]. Use a dictionary
there instead saves about 20-30s on my machine.

[1] https://searchfox.org/mozilla-central/rev/622dbd3409610ad3f71b56c9a6a92da905dab0aa/taskcluster/taskgraph/transforms/job/__init__.py#243-247

Comment 2

[Justin Wood (:Callek)]

I just ran this through hg bisect and:

The first bad revision is:
changeset: 617179:c90d36eecc9e
user: Gregory Mierzwinski <[e-mail]>
date: Wed Jul 08 07:48:16 2020 +0000
summary: Bug 1650871 - Add all browsertime desktop tests. r=perftest-reviewers,AlexandruIonescu

Comment 3

[Mike Hommey [:glandium]]

Decision tasks are now taking > 10 minutes instead of 3~4. Just for this, until this is sorted out, I think the revision listed in comment 2 should be backed out.

Comment 4

[Mike Hommey [:glandium]]

Huh... so bug 1650871 only made decision tasks take 6~7 minutes... what bumped to double that is... https://treeherder.mozilla.org/#/jobs?repo=autoland&revision=5cff164097d9d5eb441d7908ceaf370b865258c1 (a backout!) which makes no sense.
The log for a slow decision task shows something like:

[task 2020-07-10T20:41:47.997Z] optimize: system-symbols-win-upload-symbols kept because of 'never' strategy
[task 2020-07-10T20:49:50.154Z] Retrieving low-value jobs list from SETA

So the bad state of decision tasks would be unrelated.

Comment 5

[Tom Prince [:tomprince]]

Attachment: Bug 1652086: [taskgraph] Use a simple deepcopy in taskgraph; r?Callek

Python's copy.deepcopy does a bunch of work to handle recusive and otherwise
non-graph-like strucutres. The way we use it most of the time in taskgraph, we
are only care about nested dictionaries and lists, so we can use a simpler
implementation. On my machine, this saves about 15s.

Comment 6

[BugBot [:suhaib / :marco/ :calixte]]

Set release status flags based on info from the regressing bug 1650871

Comment 7

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/ab433cee6227
Make `kind_dependencies_tasks` a dictionary based on the label; r=Callek

Comment 8

[Andreea Pavel [:apavel]]

https://hg.mozilla.org/mozilla-central/rev/ab433cee6227

Comment 9

[Tom Prince [:tomprince]]

Attachment: Bug 1652086: [taskgraph] Use integers rather than strings in chunk attributes; r?Callek

Comment 10

[Tom Prince [:tomprince]]

Attachment: Bug 1652086: [taskgraph] Make `taskgraph.util.templates.merge` return an unshared object; r?Callek

The docstring for merge claims it returns an new object, without modifying
any of the arguments, but will (depending on the shape) return an object
with shared subobjects in some cases.

Comment 11

[Tom Prince [:tomprince]]

Attachment: Bug 1652086: [taskgraph] Remove some gratuituous uses of deepcopy; r?Callek

Comment 12

[Tom Prince [:tomprince]]

Attachment: Bug 1652086: [taskgraph] Don't compile schemas if we aren't going to check them; r?Callek

I'm planning on replacing voluptuous with something faster in Bug 1652123, but
in the meantime, compile the schema is slow, so there is no reason to do so if
we aren't going to check them anyway.

Comment 13

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/e1c5c97858c8
[taskgraph] Don't compile schemas if we aren't going to check them; r=Callek

Comment 14

[Cristina Coroiu [:ccoroiu]]

https://hg.mozilla.org/mozilla-central/rev/e1c5c97858c8

Comment 15

[BugBot [:suhaib / :marco/ :calixte]]

The leave-open keyword is there and there is no activity for 6 months.
:ahal, maybe it's time to close this bug?

Comment 16

[Andrew Halberstadt [:ahal]]

We should file bugs blocking bug 1617598 for future performance work.