1850647 - Output build resource use data in a format the Firefox Profiler can understand

Status: RESOLVED FIXED

(Testing :: Mozbase, task)

Comment 1

[Florian Quèze [:florian]]

Attachment: Bug 1850647 - teach SystemResourceMonitor to output its data in the Firefox Profiler format, r=glandium.

Comment 2

[Florian Quèze [:florian]]

Attachment: Bug 1850647 - Show python build tasks as markers, r=glandium.

Depends on D187076

Comment 3

[Florian Quèze [:florian]]

Attachment: Bug 1850647 - Show markers for processes started by make files, r=glandium.

Depends on D187077

Comment 4

[Florian Quèze [:florian]]

Attachment: Bug 1850647 - Show 'Line' markers for build system lines that are not associated, r=glandium.

Depends on D187078

Comment 5

[Florian Quèze [:florian]]

Attachment: Bug 1850647 - add an optional argument to py_action to record the target file name for most actions, r=glandium.

Depends on D187079

Comment 6

[Florian Quèze [:florian]]

Attachment: Bug 1850647 - prepare py_action callers to py_action expanding to multiple lines, r=glandium.

Depends on D187078

Comment 7

[Florian Quèze [:florian]]

Attachment: Bug 1850647 - make py_action emit BUILDSTATUS START and END lines for every action, r=glandium.

Depends on D190513

Comment 8

[Florian Quèze [:florian]]

Attachment: Bug 1850647 - make 'mach resource-usage' open the profile of the most recent build in the Firefox Profiler, r=glandium.

Depends on D187402

Comment 9

[Florian Quèze [:florian]]

Attachment: Bug 1850647 - stop generating the build_resources.json file, r=glandium.

Depends on D190567

Comment 10

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/ce7ac5e23c1c
teach SystemResourceMonitor to output its data in the Firefox Profiler format, r=glandium,ahal.
https://hg.mozilla.org/integration/autoland/rev/6c15af4bc2c6
Show markers for processes started by make files, r=glandium.
https://hg.mozilla.org/integration/autoland/rev/e9277971c260
avoid using py_action with addprefix to enable wrapping the actual command in BUILDSTATUS messages, r=glandium.
https://hg.mozilla.org/integration/autoland/rev/0128ca1c9d18
make py_action emit BUILDSTATUS START and END lines for every action, r=glandium.
https://hg.mozilla.org/integration/autoland/rev/c95204654c91
add an optional argument to py_action to record the target file name for most actions, r=glandium.
https://hg.mozilla.org/integration/autoland/rev/7465055478d9
make 'mach resource-usage' open the profile of the most recent build in the Firefox Profiler, r=glandium.
https://hg.mozilla.org/integration/autoland/rev/ab9d4ca38d83
stop generating the build_resources.json file, r=glandium.

Comment 11

[Sandor Molnar[:smolnar]]

Backed out for causing build bustages

Backout link: https://hg.mozilla.org/integration/autoland/rev/2124c3eebcb12eb72e8ced7bfb3c8521a028af8d

Push with failures

Failure log -> NotADirectoryError: [WinError 267] The directory name is invalid

Comment 12

[Florian Quèze [:florian]]

The build failure happens on Windows machines using msys (not on Windows cross compilations from Linux).

In D:/task_169701372024339/.mozbuild/srcdirs/src-fc6a3ae757e7/_virtualenvs/build/Scripts/python.exe -m mozbuild.action.file_generate D:/task_169701372024339/build/src/python/mozbuild/mozbuild/action/file_generate_wrapper.py action third_party/libwebrtc/gen/experiments/registered_field_trials.h third_party/libwebrtc/gen/experiments/.deps/registered_field_trials.h.pp third_party/libwebrtc/gen/experiments/.deps/registered_field_trials.h.stub /third_party/libwebrtc/experiments/field_trials.py /third_party/libwebrtc/ header --output gen/experiments/registered_field_trials.h
the arguments starting with / get expended to C:\\mozilla-build\\msys2\\third_party\\libwebrtc\\experiments\\field_trials.py and C:/mozilla-build/msys2/third_party/libwebrtc/, ie. msys interprets them as absolute Unix paths.

Changing https://searchfox.org/mozilla-central/rev/5ad226c7379b0564c76dc3b54b44985356f94c5a/third_party/libwebrtc/experiments/registered_field_trials_header_gn/moz.build#19-20 to have a double slash prefix (as suggested in https://stackoverflow.com/a/14189687) makes the build pass locally for me, but I'm not convinced this is a good solution, and even if it was, I got a bit lost in https://searchfox.org/mozilla-central/source/third_party/libwebrtc/experiments/BUILD.gn and https://searchfox.org/mozilla-central/source/python/mozbuild/mozbuild/gn_processor.py

Mike, any suggestion about how to resolve this?

Comment 13

[Mike Hommey [:glandium]]

This fixes the problem, and hopefully shouldn't break comm-central: https://treeherder.mozilla.org/jobs?repo=try&revision=a4313043816333a92214bf9120c776c6b4a5decf (ignore the mac bustage, the workers are busted on try)

Comment 14

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/07fa901acf21
teach SystemResourceMonitor to output its data in the Firefox Profiler format, r=glandium,ahal.
https://hg.mozilla.org/integration/autoland/rev/6b35d787f630
Show markers for processes started by make files, r=glandium.
https://hg.mozilla.org/integration/autoland/rev/e7fd8fc6d6e1
prepare py_action callers to py_action expanding to multiple lines, r=glandium.
https://hg.mozilla.org/integration/autoland/rev/991f0a911ff4
make py_action emit BUILDSTATUS START and END lines for every action, r=glandium.
https://hg.mozilla.org/integration/autoland/rev/730bb9fd0b23
add an optional argument to py_action to record the target file name for most actions, r=glandium.
https://hg.mozilla.org/integration/autoland/rev/087ef6db6255
make 'mach resource-usage' open the profile of the most recent build in the Firefox Profiler, r=glandium.
https://hg.mozilla.org/integration/autoland/rev/ae879e486e0e
stop generating the build_resources.json file, r=glandium.

Comment 15

[Cristian Tuns]

https://hg.mozilla.org/mozilla-central/rev/07fa901acf21
https://hg.mozilla.org/mozilla-central/rev/6b35d787f630
https://hg.mozilla.org/mozilla-central/rev/e7fd8fc6d6e1
https://hg.mozilla.org/mozilla-central/rev/991f0a911ff4
https://hg.mozilla.org/mozilla-central/rev/730bb9fd0b23
https://hg.mozilla.org/mozilla-central/rev/087ef6db6255
https://hg.mozilla.org/mozilla-central/rev/ae879e486e0e