1713173 - Add a flag like `--profile-command` for profiling `mach` commands

Status: RESOLVED FIXED

(Firefox Build System :: Mach Core, enhancement)

Description

[Nick Alexander :nalexander [he/him]]

mach has a very nice --debug-command flag that invokes the actual command under pdb. I would like something like a --profile-command flag that runs the command while capturing a profile. I want to investigate some slow mach invocation frequently enough to make this worth the effort.

Comment 1

[Mitchell Hentges [:mhentges] 🦀]

We could lean into functiontrace for this.

Comment 2

[Nick Alexander :nalexander [he/him]]

(In reply to Mitchell Hentges [:mhentges] 🦀 from comment #1)

We could lean into functiontrace for this.

Sadly, it doesn't support Windows -- which is actually what's motivating me investigating mach lint right now.

Comment 3

[Nick Alexander :nalexander [he/him]]

Locally, here's how I'm doing this:

# HG changeset patch
# User Nick Alexander <[e-mail]>
# Date 1615497686 28800
#      Thu Mar 11 13:21:26 2021 -0800
# Node ID e257b7e479f29cfcd6f2d584277536afe14a7940
# Parent  9ba32bd932d4f089a0eaf73c09784e165fc0bf0c
WIP on profiling Python mach commands.  View profiles with snakeviz.

diff --git a/mach b/mach
--- a/mach
+++ b/mach
@@ -19,13 +19,15 @@ nativecmds="
     install-moz-phab
 "
 
+set -o xtrace
+
 run_py() {
     # Try to run a specific Python interpreter.
     py_executable="$1"
     shift
     if command -v "$py_executable" > /dev/null
     then
-        exec "$py_executable" "$0" "$@"
+        exec "$py_executable" "${py_profile_command_args[@]}" "$0" "$@"
     else
         echo "This mach command requires $py_executable, which wasn't found on the system!"
         case "$py_executable" in
@@ -78,6 +80,12 @@ get_command() {
 state_dir=${MOZBUILD_STATE_PATH:-~/.mozbuild}
 command=$(get_command "$@")
 
+py_profile_command_args=("-m" "cProfile" "-o" "mach_profile_${command}.txt")
+
+# To visualize, use snakeviz:
+# $HOME/.mozbuild/_virtualenvs/mach/bin/pip install snakeviz
+# $HOME/.mozbuild/_virtualenvs/mach/bin/python -m snakeviz mach_profile_command.txt
+
 # If MACH_USE_SYSTEM_PYTHON or MOZ_AUTOMATION are set, always use the
 # python 3 executables and not the virtualenv locations.
 if [ -z ${MACH_USE_SYSTEM_PYTHON} ] && [ -z ${MOZ_AUTOMATION} ]

I tried and failed to add --profile-command to make this conditional, but I can't recall why I failed.

Comment 4

[Nick Alexander :nalexander [he/him]]

Attachment: Bug 1713173 - Add `--profile-command` flag for profiling `mach` commands. r?#build

I chose to do this at the level of the outer Python invocation because:

1. python -m cProfile ... handles writing the file and some other
   details. It's possible to rebuild the functionality -- the tools
   are there -- but the APIs are awkward.

2. this allows to profile mach internals, instead of just the
   invoked command's implementation.

This uses the return code of the get_command subshell to transmit
the single bit of information "is the flag present".

The Python-level argument is required in order to have --help know
about the option and to avoid the mach shell script having to filter
arguments.

Comment 5

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/a1a67efda98e
Add `--profile-command` flag for profiling `mach` commands. r=firefox-build-system-reviewers,mhentges

Comment 6

[Sandor Molnar[:smolnar]]

https://hg.mozilla.org/mozilla-central/rev/a1a67efda98e