2079630 - add resource monitor markers to task graph generation

Status: ASSIGNED

(Firefox Build System :: Task Configuration, task)

Description

[Julien Cristau [:jcristau]]

taskgraph decision sets up a resource monitor and decision tasks output a profile, but it's missing a lot of useful markers to get insights into the various steps.
We should also set up the resource monitor for the task graph generation step of mach try.

Marking as dependent on bug 1425785 because that adds a mechanism to forward markers from subprocesses.

Comment 1

[Julien Cristau [:jcristau]]

Attachment: Bug 2079630 - Record a resource usage profile while generating the taskgraph in mach try. r?#releng

The profile is written to the logs/try directory of the mach state directory.

Comment 2

[Julien Cristau [:jcristau]]

Attachment: Bug 2079630 - Let resource monitors in child processes forward to their parent's profile without inheriting its monitor. r?#releng

Add a forward_to argument to SystemResourceMonitor so child processes can
forward to a file given to them other than through the environment, and a
children_forward_path property giving the file children should forward to.

Forked children no longer see the parent's monitor as theirs, so they can't
stop the parent's resource usage collection when exiting.

Comment 3

[Julien Cristau [:jcristau]]

Attachment: Bug 2079630 - Record a resource usage profile marker for each kind loaded while generating the taskgraph. r?#releng

Kinds loaded in worker processes forward their marker to the monitor in the
parent process. The workers find ProfiledKind through the kind instances they
are sent, so this works whatever the multiprocessing start method is.

Comment 4

[Julien Cristau [:jcristau]]

Attachment: Bug 2079630 - Record resource usage profile markers for kind results and taskgraph verifications. r?#releng

Getting the tasks of a kind loaded in another process back to the parent
process takes over a second for the largest kinds, and verifications take
most of the time of some generation phases, neither of which was covered by
markers.

Comment 5

[Julien Cristau [:jcristau]]

Attachment: Bug 2079630 - Record a resource usage profile marker for writing the taskgraph cache in mach try. r?#releng

Comment 6

[Julien Cristau [:jcristau]]

Attachment: Bug 2079630 - Record a resource usage profile marker for each taskgraph generation phase. r?#releng

taskgraph calls generator.phase_finished after each phase it runs, with no name
for a phase that raised. The graph_config phase isn't recorded: it ends before
register() replaces the hook.