1795502 - mach try fuzzy is slow (45s) when it has to regenerate the task graph

Status: RESOLVED DUPLICATE of bug 1617598

(Firefox Build System :: Task Configuration, defect)

Description

[Michael Comella (:mcomella) [NI reported issues only: ex-Mozilla]]

Attachment: mach_profile_try.cProfile

When I run /.mach try fuzzy locally and have to regenerate the task graph, it can take a while, e.g. this execution took ~46 seconds (note: I left the interactive prompt ASAP & this might have some file system caching because it usually feels slower):

$ time ./mach try fuzzy
Task configuration changed, generating target task set
no tasks selected
./mach try fuzzy  45.69s user 2.25s system 97% cpu 49.018 total

I took a profile using ./mach --profile-command (where it took 104s to generate_tasks()) and tried to find any obvious opportunities for speed ups. The only thing I found is that we spend 26 seconds or 1/4 of total execution time in copy.deepcopy, which seems excessive.

Note: I've attached the profile where I may have lingered in the prompt for a little while. You can read it with snakeviz (i.e. pip install it and execute python3 -m snakeviz <profile>. It's hard to read the profile though because the generator functions get nested in the call stack.

I'm on macOS 12.5.1. The commit I took the timing from is 8454bb0c09fe and the profile was taken there or on a nearby commit.

Comment 1

[Andrew Halberstadt [:ahal]]

Thanks for the bug! I'm going to dupe this to a perf meta bug, will paste your comment to a comment over there.

In case you haven't seen it, one way to reduce the perf overhead is to setup a watchman target as per:
https://firefox-source-docs.mozilla.org/tools/try/tasks.html#configuring-watchman

(caveat: I haven't tried this myself recently)

Comment 2

[[:sergesanspaille]]

I proposed an optimisation of deepcopy in the context of taskcluster: https://bugzilla.mozilla.org/show_bug.cgi?id=1805842